From llvmlistbot at llvm.org  Wed Jul  1 00:05:43 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Wed, 01 Jul 2026 00:05:43 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a44bc47.23ea12de.369d14.ca46@mx.google.com>


nikic wrote:

@efriedma-quic Are you fine with the new way the properties get encoded?

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Wed Jul  1 00:27:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 00:27:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a44c16a.7436a696.10d138.d2bb@mx.google.com>


================
@@ -699,6 +699,138 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
----------------
yifeihe-007 wrote:

 Originally we wanted the NVGPU conversion ops to have tensor-level expressiveness like arith.truncf/arith.extf. The current pipeline doesn't really use the tensor form yet. Down the line it could be nice to actually use nvgpu at the tensor layer, so fp conversion has one unified representation from tensor down to the lower levels. But open on this, happy to hear your thoughts on the layering here.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Wed Jul  1 00:34:48 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 00:34:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a44c318.30759430.1a950e.7f7d@mx.google.com>


================
@@ -840,6 +840,23 @@ LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
   return success();
 }
 
+FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
+                                                  Type dest,
+                                                  Value value) const {
+  // If the value already has the correct type, no cast is needed.
+  if (value.getType() == dest)
+    return value;
+
+  if (castFn)
+    return (*castFn)(b, loc, dest, value);
+
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
----------------
andrey-golubev wrote:

yes. indeed, this is why I wrote initially:
> I aligned cast's API / implementation to alloc + memcpy so that the three things are cohesive

in general, it feels like having default lambda values is cleaner (we can get rid of `std::optional<>` around and not check for them being set)
I'll make a follow-up probably to convert the three to the same form as the rest.

https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 00:38:09 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 01 Jul 2026 00:38:09 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] Reland "[mlir][nvvm] extract
 NVVM inliner interface from LLVM dialect" with flang fixes (PR #206782)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206782@github.com>
Message-ID: <6a44c3e1.30759430.1a950e.f90b@mx.google.com>


joker-eph wrote:

I don't know how. It would be nice to have a way to do it with a / command here though!

https://github.com/llvm/llvm-project/pull/206782

From llvmlistbot at llvm.org  Wed Jul  1 00:38:14 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 01 Jul 2026 00:38:14 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] Reland "[mlir][nvvm] extract
 NVVM inliner interface from LLVM dialect" with flang fixes (PR #206782)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206782@github.com>
Message-ID: <6a44c3e6.a151105c.15f7b1.d7be@mx.google.com>


https://github.com/joker-eph approved this pull request.


https://github.com/llvm/llvm-project/pull/206782

From llvmlistbot at llvm.org  Wed Jul  1 00:44:34 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 01 Jul 2026 00:44:34 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Lower scan
 directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a44c562.ff037943.172e14.f17a@mx.google.com>


https://github.com/chandraghale updated https://github.com/llvm/llvm-project/pull/206747

>From 82416bc775c4befdbcce1f999a5f5b97ef954cd8 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 10:00:56 -0500
Subject: [PATCH 1/2] Lower scan directive and inscan reduction modifier

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             |  57 ++-
 .../Lower/OpenMP/Todo/nested-wsloop-scan.f90  |  34 ++
 .../OpenMP/Todo/wsloop-scan-collapse.f90      |  29 ++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 476 +++++++++++++++---
 .../Target/LLVMIR/openmp-reduction-scan.mlir  | 123 +++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  41 +-
 openmp/runtime/test/scan/scan.f90             |  38 ++
 7 files changed, 726 insertions(+), 72 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
 create mode 100644 openmp/runtime/test/scan/scan.f90

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..e77584a142eea 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2972,12 +2972,59 @@ genParallelOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
 static mlir::omp::ScanOp
 genScanOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-          semantics::SemanticsContext &semaCtx, mlir::Location loc,
-          const ConstructQueue &queue, ConstructQueue::const_iterator item) {
+          semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
+          mlir::Location loc, const ConstructQueue &queue,
+          ConstructQueue::const_iterator item) {
   mlir::omp::ScanOperands clauseOps;
   genScanClauses(converter, semaCtx, item->clauses, loc, clauseOps);
-  return mlir::omp::ScanOp::create(converter.getFirOpBuilder(),
-                                   converter.getCurrentLocation(), clauseOps);
+  mlir::omp::ScanOp scanOp = mlir::omp::ScanOp::create(
+      converter.getFirOpBuilder(), converter.getCurrentLocation(), clauseOps);
+
+  /// Scan reduction is not implemented with nested workshare loops, linear
+  /// clause, tiling
+  mlir::omp::LoopNestOp loopNestOp =
+      scanOp->getParentOfType<mlir::omp::LoopNestOp>();
+  llvm::SmallVector<mlir::omp::LoopWrapperInterface> loopWrappers;
+  loopNestOp.gatherWrappers(loopWrappers);
+  mlir::Operation *loopWrapperOp = loopWrappers.front().getOperation();
+  if (llvm::isa<mlir::omp::SimdOp>(loopWrapperOp))
+    TODO(loc, "unsupported simd");
+  if (loopWrappers.size() > 1)
+    TODO(loc, "unsupported composite");
+  mlir::omp::WsloopOp wsLoopOp = llvm::cast<mlir::omp::WsloopOp>(loopWrapperOp);
+  bool isNested =
+      (loopNestOp.getNumLoops() > 1) ||
+      (wsLoopOp && (wsLoopOp->getParentOfType<mlir::omp::WsloopOp>()));
+  if (isNested)
+    TODO(loc, "Scan directive inside nested workshare loops");
+  if (wsLoopOp && !wsLoopOp.getLinearVars().empty())
+    TODO(loc, "Scan directive with linear clause");
+  if (loopNestOp.getTileSizes())
+    TODO(loc, "Scan directive with loop tiling");
+
+  // All loop indices should be loaded after the scan construct as otherwise,
+  // it would result in using the index variable across scan directive.
+  // (`Intra-iteration dependences from a statement in the structured
+  // block sequence that precede a scan directive to a statement in the
+  // structured block sequence that follows a scan directive must not exist,
+  // except for dependences for the list items specified in an inclusive or
+  // exclusive clause.`).
+  // TODO: Nested loops are not handled.
+  mlir::Region &region = loopNestOp->getRegion(0);
+  mlir::Value indexVal = fir::getBase(region.getArgument(0));
+  lower::pft::Evaluation *doConstructEval = eval.parentConstruct;
+  fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
+  lower::pft::Evaluation *doLoop = &doConstructEval->getFirstNestedEvaluation();
+  auto *doStmt = doLoop->getIf<parser::NonLabelDoStmt>();
+  assert(doStmt && "Expected do loop to be in the nested evaluation");
+  const auto &loopControl =
+      std::get<std::optional<parser::LoopControl>>(doStmt->t);
+  const parser::LoopControl::Bounds *bounds =
+      std::get_if<parser::LoopControl::Bounds>(&loopControl->u);
+  mlir::Operation *storeOp =
+      setLoopVar(converter, loc, indexVal, bounds->Name().thing.symbol);
+  firOpBuilder.setInsertionPointAfter(storeOp);
+  return scanOp;
 }
 
 static mlir::omp::SectionsOp
@@ -4305,7 +4352,7 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
                                   loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_scan:
-    newOp = genScanOp(converter, symTable, semaCtx, loc, queue, item);
+    newOp = genScanOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_section:
     llvm_unreachable("genOMPDispatch: OMPD_section");
diff --git a/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90 b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
new file mode 100644
index 0000000000000..414e2ef2d5aed
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
@@ -0,0 +1,34 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_scan_example
+  implicit none
+  integer, parameter :: n = 4, m = 5
+  integer :: a(n, m), b(n, m)
+  integer :: i, j
+  integer :: row_sum, col_sum
+
+  do i = 1, n
+     do j = 1, m
+        a(i, j) = i + j
+     end do
+  end do
+
+  !$omp parallel do reduction(inscan, +: row_sum) private(col_sum, j)
+  do i = 1, n
+     row_sum = row_sum + i
+     !$omp scan inclusive(row_sum)
+
+     col_sum = 0
+     !$omp parallel do reduction(inscan, +: col_sum)
+     do j = 1, m
+        col_sum = col_sum + a(i, j)
+        !CHECK: not yet implemented: Scan directive inside nested workshare loops
+        !$omp scan inclusive(col_sum)
+        b(i, j) = col_sum + row_sum 
+     end do
+     !$omp end parallel do
+  end do
+  !$omp end parallel do
+end program nested_scan_example
diff --git a/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90 b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
new file mode 100644
index 0000000000000..b8e6e831884ab
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
@@ -0,0 +1,29 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_loop_example
+  implicit none
+  integer :: i, j, x
+  integer, parameter :: N = 100, M = 200
+  real :: A(N, M), B(N, M)
+  x = 0
+
+  do i = 1, N
+    do j = 1, M
+      A(i, j) = i * j
+    end do
+  end do
+  
+  !$omp parallel do collapse(2) reduction(inscan, +:x)
+  do i = 1, N
+    do j = 1, M
+      x = x + A(i,j)
+      !CHECK: not yet implemented: Scan directive inside nested workshare loops
+      !$omp scan inclusive(x)
+      B(i, j) = x
+    end do
+  end do
+  !$omp end parallel do
+
+end program nested_loop_example
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d860c408e0fd2..297ebac51de88 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -41,6 +41,8 @@
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 
+#include <cassert>
+#include <cstddef>
 #include <cstdint>
 #include <iterator>
 #include <numeric>
@@ -84,6 +86,10 @@ class OpenMPAllocStackFrame
       : allocInsertPoint(allocaIP), deallocBlocks(deallocBlocks) {}
   llvm::OpenMPIRBuilder::InsertPointTy allocInsertPoint;
   llvm::SmallVector<llvm::BasicBlock *> deallocBlocks;
+  /// Set to true when this alloca frame encloses an omp.parallel operation.
+  /// The alloca insertion point of a function in which a parallel op is
+  /// defined may be used to allocate the temporary buffer for scan reductions.
+  bool containsParallelOp = false;
 };
 
 /// Stack frame to hold a \see llvm::CanonicalLoopInfo representing the
@@ -93,7 +99,17 @@ class OpenMPLoopInfoStackFrame
     : public StateStackFrameBase<OpenMPLoopInfoStackFrame> {
 public:
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(OpenMPLoopInfoStackFrame)
+  /// For constructs like `scan`, a single `omp.loop_nest` is split into an
+  /// input loop and a scan loop. In that case `loopInfo` holds the input loop
+  /// info and `scanloopInfo` holds the scan loop info.
   llvm::CanonicalLoopInfo *loopInfo = nullptr;
+  llvm::CanonicalLoopInfo *scanloopInfo = nullptr;
+  llvm::ScanInfo *scanInfo = nullptr;
+  /// Map reduction variables to their LLVM types. Populated when the reduction
+  /// clause is processed and used when a `scan` directive is encountered in the
+  /// loop body.
+  std::unique_ptr<llvm::DenseMap<llvm::Value *, llvm::Type *>>
+      reductionVarToType = nullptr;
 };
 
 /// Custom error class to signal translation errors that don't need reporting,
@@ -359,6 +375,10 @@ static LogicalResult checkImplementationStatus(Operation &op) {
     if (!op.getDependVars().empty() || op.getDependKinds())
       result = todo("depend");
   };
+  auto checkExclusive = [&todo](auto op, LogicalResult &result) {
+    if (!op.getExclusiveVars().empty())
+      result = todo("exclusive");
+  };
   auto checkHint = [](auto op, LogicalResult &) {
     if (op.getHint())
       op.emitWarning("hint clause discarded");
@@ -389,21 +409,27 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         op.getReductionMod().value() != omp::ReductionModifier::defaultmod) {
       omp::ReductionModifier mod = op.getReductionMod().value();
       // The `task` reduction modifier is supported on the parallel and
-      // worksharing (do/for and sections) constructs. Other modifiers, and the
-      // `task` modifier on other constructs, are not yet implemented.
+      // worksharing (do/for and sections) constructs. The `inscan` modifier is
+      // supported on the worksharing-loop construct (it is translated as a scan
+      // reduction). Other modifiers, and these modifiers on other constructs,
+      // are not yet implemented.
       bool taskModifierSupported =
           mod == omp::ReductionModifier::task &&
           isa<omp::ParallelOp, omp::WsloopOp, omp::SectionsOp>(op);
-      if (!taskModifierSupported) {
+      bool inscanModifierSupported =
+          mod == omp::ReductionModifier::inscan && isa<omp::WsloopOp>(op);
+      if (!taskModifierSupported && !inscanModifierSupported) {
         result = todo("reduction with modifier");
-      } else if (auto byref = op.getReductionByref()) {
-        // The task reduction modifier lowering only handles non-byref
-        // reductions for now.
-        for (bool isByRef : *byref)
-          if (isByRef) {
-            result = todo("task reduction modifier with by-ref reduction");
-            break;
-          }
+      } else if (taskModifierSupported) {
+        if (auto byref = op.getReductionByref()) {
+          // The task reduction modifier lowering only handles non-byref
+          // reductions for now.
+          for (bool isByRef : *byref)
+            if (isByRef) {
+              result = todo("task reduction modifier with by-ref reduction");
+              break;
+            }
+        }
       }
     }
   };
@@ -460,6 +486,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkAllocate(op, result);
         checkOrder(op, result);
       })
+      .Case([&](omp::ScanOp op) { checkExclusive(op, result); })
       .Case([&](omp::SectionsOp op) {
         checkAllocate(op, result);
         checkPrivate(op, result);
@@ -650,6 +677,97 @@ findCurrentLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
   return loopInfo;
 }
 
+/// Find the scan loop information structure for the scan loop nest being
+/// translated. It will return a `null` value unless called from the
+/// translation function for a loop wrapper operation after successfully
+/// translating its body.
+static llvm::CanonicalLoopInfo *
+findCurrentScanLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::CanonicalLoopInfo *scanLoopInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanLoopInfo = frame.scanloopInfo;
+        return WalkResult::interrupt();
+      });
+  return scanLoopInfo;
+}
+
+/// Find the `ScanInfo` stored on the loop stack frame. Upon encountering an
+/// `inscan` reduction modifier, `scanInfoInitialize` initializes the
+/// `ScanInfo`, which is then used when a `scan` directive is encountered in the
+/// body of the loop nest.
+static llvm::ScanInfo *
+findScanInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::ScanInfo *scanInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanInfo = frame.scanInfo;
+        return WalkResult::interrupt();
+      });
+  return scanInfo;
+}
+
+/// The types of reduction variables are used for lowering a `scan` directive
+/// that appears in the body of the loop. The types are stored in the loop frame
+/// when the reduction clause is encountered and used when the `scan` directive
+/// is encountered.
+static llvm::DenseMap<llvm::Value *, llvm::Type *> *
+findReductionVarTypes(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        if (!frame.reductionVarToType)
+          frame.reductionVarToType =
+              std::make_unique<llvm::DenseMap<llvm::Value *, llvm::Type *>>();
+        reductionVarToType = frame.reductionVarToType.get();
+        return WalkResult::interrupt();
+      });
+  return reductionVarToType;
+}
+
+/// Scan reduction requires a shared buffer to be allocated to perform the
+/// reduction. The allocation needs to be done outside the parallel region in
+/// which the scan operation is used.
+static llvm::OpenMPIRBuilder::InsertPointTy
+findParallelAllocaIP(llvm::IRBuilderBase &builder,
+                     LLVM::ModuleTranslation &moduleTranslation) {
+  // If there is an alloca insertion point on the stack belonging to a frame
+  // that encloses a parallel op, use it.
+  llvm::OpenMPIRBuilder::InsertPointTy allocaInsertPoint;
+  WalkResult walkResult = moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (frame.containsParallelOp) {
+          allocaInsertPoint = frame.allocInsertPoint;
+          return WalkResult::interrupt();
+        }
+        return WalkResult::skip();
+      });
+  if (walkResult.wasInterrupted())
+    return allocaInsertPoint;
+  // Otherwise, insert into the entry block of the surrounding function.
+  // If the current IRBuilder InsertPoint is the function's entry, it cannot
+  // also be used for alloca insertion which would result in insertion order
+  // confusion. Create a new BasicBlock for the Builder and use the entry block
+  // for the allocs.
+  // TODO: Create a dedicated alloca BasicBlock at function creation such that
+  // we do not need to move the current InsertPoint here.
+  if (builder.GetInsertBlock() ==
+      &builder.GetInsertBlock()->getParent()->getEntryBlock()) {
+    assert(builder.GetInsertPoint() == builder.GetInsertBlock()->end() &&
+           "Assuming end of basic block");
+    llvm::BasicBlock *entryBB = llvm::BasicBlock::Create(
+        builder.getContext(), "entry", builder.GetInsertBlock()->getParent(),
+        builder.GetInsertBlock()->getNextNode());
+    builder.CreateBr(entryBB);
+    builder.SetInsertPoint(entryBB);
+  }
+
+  llvm::BasicBlock &funcEntryBlock =
+      builder.GetInsertBlock()->getParent()->getEntryBlock();
+  return llvm::OpenMPIRBuilder::InsertPointTy(
+      &funcEntryBlock, funcEntryBlock.getFirstInsertionPt());
+}
+
 /// Converts the given region that appears within an OpenMP dialect operation to
 /// LLVM IR, creating a branch from the `sourceBlock` to the entry block of the
 /// region, and a branch from any block with an successor-less OpenMP terminator
@@ -1414,7 +1532,8 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
                   SmallVectorImpl<llvm::Value *> &privateReductionVariables,
                   DenseMap<Value, llvm::Value *> &reductionVariableMap,
                   llvm::ArrayRef<bool> isByRef,
-                  SmallVectorImpl<DeferredStore> &deferredStores) {
+                  SmallVectorImpl<DeferredStore> &deferredStores,
+                  bool isInScanRegion = false) {
   if (op.getNumReductionVars() == 0)
     return success();
 
@@ -1451,11 +1570,17 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
   for (auto [data, addr] : deferredStores)
     builder.CreateStore(data, addr);
 
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
   // Before the loop, store the initial values of reductions into reduction
   // variables. Although this could be done after allocas, we don't want to mess
   // up with the alloca insertion point.
   for (unsigned i = 0; i < op.getNumReductionVars(); ++i) {
     SmallVector<llvm::Value *, 1> phis;
+    llvm::Type *reductionType =
+        moduleTranslation.convertType(reductionDecls[i].getType());
+    if (isInScanRegion && reductionVarToType != nullptr)
+      (*reductionVarToType)[privateReductionVariables[i]] = reductionType;
 
     // map block argument to initializer region
     mapInitializationArgs(op, moduleTranslation, builder, reductionDecls,
@@ -4406,11 +4531,15 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     return failure();
 
   assert(afterAllocas.get()->getSinglePredecessor());
+  bool isInScanRegion =
+      wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                     mlir::omp::ReductionModifier::inscan);
   if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
                                moduleTranslation,
                                afterAllocas.get()->getSinglePredecessor(),
                                reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores)))
+                               reductionVariableMap, isByRef, deferredStores,
+                               isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -4432,6 +4561,10 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   std::optional<omp::ScheduleModifier> scheduleMod = wsloopOp.getScheduleMod();
   bool isSimd = wsloopOp.getScheduleSimd();
   bool loopNeedsBarrier = !wsloopOp.getNowait();
+  // TODO: Linear clause support needs to be enabled for scan reduction.
+  if (isInScanRegion)
+    assert(wsloopOp.getLinearVars().empty() &&
+           "Linear clause support is not enabled with scan reduction");
 
   // The only legal way for the direct parent to be omp.distribute is that this
   // represents 'distribute parallel do'. Otherwise, this is a regular
@@ -4469,21 +4602,93 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   if (failed(handleError(regionBlock, opInst)))
     return failure();
 
+  // Generates the loop body for a worksharing loop, including linear-variable
+  // handling and the call into the OpenMPIRBuilder's worksharing-loop helper.
+  // For scan reductions this lambda is invoked twice: once for the input loop
+  // and once for the scan loop.
+  const auto &&wsloopCodeGen = [&](llvm::CanonicalLoopInfo *loopInfo,
+                                   bool noLoopMode,
+                                   bool inputScanLoop) -> LogicalResult {
+    // Emit Initialization and Update IR for linear variables
+    if (!wsloopOp.getLinearVars().empty()) {
+      linearClauseProcessor.initLinearVar(builder, moduleTranslation,
+                                          loopInfo->getPreheader());
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          moduleTranslation.getOpenMPBuilder()->createBarrier(
+              builder.saveIP(), llvm::omp::OMPD_barrier);
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+      builder.restoreIP(*afterBarrierIP);
+      linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                            loopInfo->getIndVar());
+      linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+    }
+
+    builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
+
+    for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
+      linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
+                                           loopInfo->getLatch(), index);
+
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
+        ompBuilder->applyWorkshareLoop(
+            ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
+            convertToScheduleKind(schedule), chunk, isSimd,
+            scheduleMod == omp::ScheduleModifier::monotonic,
+            scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
+            workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
+
+    if (failed(handleError(wsloopIP, opInst)))
+      return failure();
+
+    // Emit finalization for linear vars.
+    if (!wsloopOp.getLinearVars().empty()) {
+      llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
+      assert(loopInfo->getLastIter() &&
+             "`lastiter` in CanonicalLoopInfo is nullptr");
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
+                                                  loopInfo->getLastIter());
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+
+      builder.restoreIP(oldIP);
+    }
+
+    // For scan reductions, the cancellation finalization callback is popped
+    // only after the scan (second) loop has been generated.
+    if (!inputScanLoop || !isInScanRegion)
+      popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
+
+    return success();
+  };
+
   llvm::CanonicalLoopInfo *loopInfo = findCurrentLoopInfo(moduleTranslation);
 
-  // Emit Initialization and Update IR for linear variables
-  if (!wsloopOp.getLinearVars().empty()) {
-    linearClauseProcessor.initLinearVar(builder, moduleTranslation,
-                                        loopInfo->getPreheader());
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    // Emit the scan reduction combiner between the input loop and the scan
+    // loop, operating on the per-iteration partial values stored in the shared
+    // buffer.
+    auto inputLoopFinishIp = loopInfo->getAfterIP();
+    builder.restoreIP(inputLoopFinishIp);
+    SmallVector<OwningReductionGen> owningReductionGens;
+    SmallVector<OwningAtomicReductionGen> owningAtomicReductionGens;
+    SmallVector<llvm::OpenMPIRBuilder::ReductionInfo, 2> reductionInfos;
+    SmallVector<OwningDataPtrPtrReductionGen> owningReductionGenRefDataPtrGens;
+    collectReductionInfo(wsloopOp, builder, moduleTranslation, reductionDecls,
+                         owningReductionGens, owningAtomicReductionGens,
+                         owningReductionGenRefDataPtrGens,
+                         privateReductionVariables, reductionInfos, isByRef);
+    llvm::BasicBlock *cont = splitBB(builder, false, "omp.scan.loop.cont");
+    llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy redIP =
+        ompBuilder->emitScanReduction(builder.saveIP(), reductionInfos,
+                                      scanInfo);
+    if (failed(handleError(redIP, opInst)))
       return failure();
-    builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
-    linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+
+    builder.restoreIP(*redIP);
+    builder.CreateBr(cont);
   }
 
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
@@ -4502,50 +4707,41 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
       noLoopMode = true;
   }
 
-  for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
-    linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
-                                         loopInfo->getLatch(), index);
-
-  llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
-      ompBuilder->applyWorkshareLoop(
-          ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
-          convertToScheduleKind(schedule), chunk, isSimd,
-          scheduleMod == omp::ScheduleModifier::monotonic,
-          scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
-          workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
-
-  if (failed(handleError(wsloopIP, opInst)))
+  // For scan loops, the input loop does not pop the cancellation finalization
+  // callback; that happens after the scan loop is generated below.
+  bool inputScanLoop = isInScanRegion;
+  if (failed(wsloopCodeGen(loopInfo, noLoopMode, inputScanLoop)))
     return failure();
+  inputScanLoop = false;
 
-  // Emit finalization and in-place rewrites for linear vars.
-  if (!wsloopOp.getLinearVars().empty()) {
-    llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
-    assert(loopInfo->getLastIter() &&
-           "`lastiter` in CanonicalLoopInfo is nullptr");
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
-                                                loopInfo->getLastIter());
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    llvm::CanonicalLoopInfo *scanLoopInfo =
+        findCurrentScanLoopInfo(moduleTranslation);
+    if (failed(wsloopCodeGen(scanLoopInfo, noLoopMode, inputScanLoop)))
+      return failure();
+    SmallVector<Region *> reductionRegions;
+    llvm::transform(reductionDecls, std::back_inserter(reductionRegions),
+                    [](omp::DeclareReductionOp reductionDecl) {
+                      return &reductionDecl.getCleanupRegion();
+                    });
+    if (failed(inlineOmpRegionCleanup(
+            reductionRegions, privateReductionVariables, moduleTranslation,
+            builder, "omp.reduction.cleanup")))
       return failure();
+  } else {
+    // Close the task-reduction scope before the worksharing reduction combine.
+    if (isTaskReductionMod)
+      emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
+                                    moduleTranslation);
 
-    builder.restoreIP(oldIP);
+    // Process the reductions if required.
+    if (failed(createReductionsAndCleanup(
+            wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
+            privateReductionVariables, isByRef, wsloopOp.getNowait(),
+            /*isTeamsReduction=*/false)))
+      return failure();
   }
 
-  // Set the correct branch target for task cancellation
-  popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
-
-  // Close the task-reduction scope before the worksharing reduction combine.
-  if (isTaskReductionMod)
-    emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
-                                  moduleTranslation);
-
-  // Process the reductions if required.
-  if (failed(createReductionsAndCleanup(
-          wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
-          privateReductionVariables, isByRef, wsloopOp.getNowait(),
-          /*isTeamsReduction=*/false)))
-    return failure();
-
   return cleanupPrivateVars(wsloopOp, builder, moduleTranslation,
                             wsloopOp.getLoc(), privateVarsInfo);
 }
@@ -4579,6 +4775,20 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       opInst.getReductionMod() == omp::ReductionModifier::task &&
       opInst.getNumReductionVars() > 0;
 
+  // Mark the enclosing alloca stack frame as containing a parallel op. Scan
+  // reductions use the alloca insertion point of the function enclosing the
+  // parallel region to allocate their shared temporary buffer.
+  bool foundParallelOp = false;
+  moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (foundParallelOp) {
+          frame.containsParallelOp = true;
+          return WalkResult::interrupt();
+        }
+        foundParallelOp = true;
+        return WalkResult::skip();
+      });
+
   auto bodyGenCB =
       [&](InsertPointTy allocaIP, InsertPointTy codeGenIP,
           llvm::ArrayRef<llvm::BasicBlock *> deallocBlocks) -> llvm::Error {
@@ -4972,6 +5182,99 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             privateVarsInfo);
 }
 
+static LogicalResult
+convertOmpScan(Operation &opInst, llvm::IRBuilderBase &builder,
+               LLVM::ModuleTranslation &moduleTranslation) {
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+  auto scanOp = cast<omp::ScanOp>(opInst);
+  bool isInclusive = scanOp.hasInclusiveVars();
+  SmallVector<llvm::Value *> llvmScanVars;
+  SmallVector<llvm::Type *> llvmScanVarsType;
+  mlir::OperandRange mlirScanVars = scanOp.getInclusiveVars();
+  if (!isInclusive)
+    mlirScanVars = scanOp.getExclusiveVars();
+
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
+  for (auto val : mlirScanVars) {
+    llvm::Value *llvmVal = moduleTranslation.lookupValue(val);
+    llvmScanVars.push_back(llvmVal);
+    llvmScanVarsType.push_back((*reductionVarToType)[llvmVal]);
+  }
+  llvm::OpenMPIRBuilder::InsertPointTy allocaIP =
+      findParallelAllocaIP(builder, moduleTranslation);
+  llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+      moduleTranslation.getOpenMPBuilder()->createScan(
+          ompLoc, allocaIP, llvmScanVars, llvmScanVarsType, isInclusive,
+          scanInfo);
+  if (failed(handleError(afterIP, opInst)))
+    return failure();
+  builder.restoreIP(*afterIP);
+  return success();
+}
+
+/// Re-initialize the scan reduction variables to their identity at the start of
+/// each input-loop iteration of a scan reduction. This is required because the
+/// OpenMPIRBuilder stores the per-iteration reduction value into the scan
+/// buffer (`buffer[i] = red`) and `emitScanReduction` then computes the prefix
+/// sum across the buffer. Without resetting `red` to the reduction identity
+/// each iteration, the buffer would hold a running sum and the prefix sum would
+/// be computed twice. This reset only applies to the input loop; in the scan
+/// loop the reduction variable is loaded back from the buffer.
+static LogicalResult
+initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation) {
+  auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>();
+  if (!wsloopOp || !wsloopOp.getReductionMod() ||
+      wsloopOp.getReductionMod().value() != omp::ReductionModifier::inscan)
+    return success();
+
+  // Only reset in the input loop; in the scan loop the reduction variable holds
+  // the value loaded from the buffer and must not be overwritten.
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  if (!scanInfo || !scanInfo->OMPFirstScanLoop)
+    return success();
+
+  unsigned numReductions = wsloopOp.getNumReductionVars();
+  if (numReductions == 0)
+    return success();
+
+  SmallVector<omp::DeclareReductionOp> reductionDecls;
+  collectReductionDecls(wsloopOp, reductionDecls);
+  ArrayRef<bool> isByRef = getIsByRef(wsloopOp.getReductionByref());
+  MutableArrayRef<BlockArgument> reductionArgs =
+      cast<omp::BlockArgOpenMPOpInterface>(wsloopOp.getOperation())
+          .getReductionBlockArgs();
+
+  DenseMap<Value, llvm::Value *> reductionVariableMap;
+  for (unsigned i = 0; i < numReductions; ++i)
+    reductionVariableMap.try_emplace(
+        wsloopOp.getReductionVars()[i],
+        moduleTranslation.lookupValue(reductionArgs[i]));
+
+  for (unsigned i = 0; i < numReductions; ++i) {
+    // Scan reductions only support by-value (scalar) reductions.
+    if (isByRef[i])
+      continue;
+    SmallVector<llvm::Value *, 1> phis;
+    mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
+                          reductionVariableMap, i);
+    if (failed(inlineConvertOmpRegions(
+            reductionDecls[i].getInitializerRegion(),
+            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+      return failure();
+    assert(phis.size() == 1 &&
+           "expected one value to be yielded from the reduction init region");
+    setInsertPointForPossiblyEmptyBlock(builder);
+    builder.CreateStore(phis[0],
+                        moduleTranslation.lookupValue(reductionArgs[i]));
+  }
+  return success();
+}
+
 /// Converts an OpenMP loop nest into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -5004,6 +5307,10 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
 
     // Convert the body of the loop.
     builder.restoreIP(ip);
+    // For scan reductions, reset the reduction variables to their identity at
+    // the start of each input-loop iteration, before the input phase runs.
+    if (failed(initScanReductionVars(loopOp, builder, moduleTranslation)))
+      return llvm::make_error<PreviouslyReportedError>();
     llvm::Expected<llvm::BasicBlock *> regionBlock = convertOmpOpRegions(
         loopOp.getRegion(), "omp.loop_nest.region", builder, moduleTranslation);
     if (!regionBlock)
@@ -5036,6 +5343,46 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
       computeIP = loopInfos.front()->getPreheaderIP();
     }
 
+    // If this loop is the worksharing loop of a scan reduction, generate a
+    // pair of canonical loops (an input loop and a scan loop) instead of a
+    // single loop. The `scan` directive in the body is translated against the
+    // `ScanInfo` recorded here.
+    bool isInScanRegion = false;
+    if (auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>())
+      isInScanRegion =
+          wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                         mlir::omp::ReductionModifier::inscan);
+    if (isInScanRegion) {
+      llvm::Expected<llvm::ScanInfo *> res = ompBuilder->scanInfoInitialize();
+      if (failed(handleError(res, *loopOp)))
+        return failure();
+      llvm::ScanInfo *scanInfo = res.get();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.scanInfo = scanInfo;
+            return WalkResult::interrupt();
+          });
+      llvm::Expected<llvm::SmallVector<llvm::CanonicalLoopInfo *>> loopResults =
+          ompBuilder->createCanonicalScanLoops(
+              loc, bodyGen, lowerBound, upperBound, step,
+              /*IsSigned=*/true, loopOp.getLoopInclusive(), computeIP, "loop",
+              scanInfo);
+
+      if (failed(handleError(loopResults, *loopOp)))
+        return failure();
+      llvm::CanonicalLoopInfo *inputLoop = loopResults.get().front();
+      llvm::CanonicalLoopInfo *scanLoop = loopResults.get().back();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.loopInfo = inputLoop;
+            frame.scanloopInfo = scanLoop;
+            return WalkResult::interrupt();
+          });
+      builder.restoreIP(scanLoop->getAfterIP());
+      // TODO: tiling and collapse are not yet implemented for scan reduction.
+      return success();
+    }
+
     llvm::Expected<llvm::CanonicalLoopInfo *> loopResult =
         ompBuilder->createCanonicalLoop(
             loc, bodyGen, lowerBound, upperBound, step,
@@ -9454,6 +9801,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::WsloopOp) {
             return convertOmpWsloop(*op, builder, moduleTranslation);
           })
+          .Case([&](omp::ScanOp) {
+            return convertOmpScan(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::SimdOp) {
             return convertOmpSimd(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
new file mode 100644
index 0000000000000..c6b8daccad313
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
@@ -0,0 +1,123 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+omp.declare_reduction @add_reduction_i32 : i32 init {
+^bb0(%arg0: i32):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%0 : i32)
+} combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %0 = llvm.add %arg0, %arg1 : i32
+  omp.yield(%0 : i32)
+}
+// CHECK-LABEL: @scan_reduction
+llvm.func @scan_reduction() {
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 {bindc_name = "z"} : (i64) -> !llvm.ptr
+  %3 = llvm.alloca %0 x i32 {bindc_name = "y"} : (i64) -> !llvm.ptr
+  %5 = llvm.alloca %0 x i32 {bindc_name = "x"} : (i64) -> !llvm.ptr
+  %7 = llvm.alloca %0 x i32 {bindc_name = "k"} : (i64) -> !llvm.ptr
+  %10 = llvm.mlir.constant(100 : i32) : i32
+  %11 = llvm.mlir.constant(1 : i32) : i32
+  %12 = llvm.mlir.constant(0 : i32) : i32
+  %13 = llvm.mlir.constant(100 : index) : i64
+  %14 = llvm.mlir.addressof @_QFEa : !llvm.ptr
+  %15 = llvm.mlir.addressof @_QFEb : !llvm.ptr
+  omp.parallel {
+    %37 = llvm.mlir.constant(1 : i64) : i64
+    %38 = llvm.alloca %37 x i32 {bindc_name = "k", pinned} : (i64) -> !llvm.ptr
+    %39 = llvm.mlir.constant(1 : i64) : i64
+    omp.wsloop reduction(mod: inscan, @add_reduction_i32 %5 -> %arg0 : !llvm.ptr) {
+      omp.loop_nest (%arg1) : i32 = (%11) to (%10) inclusive step (%11) {
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %40 = llvm.load %arg0 : !llvm.ptr -> i32
+        %41 = llvm.load %38 : !llvm.ptr -> i32
+        %42 = llvm.sext %41 : i32 to i64
+        %50 = llvm.getelementptr %14[%42] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        %51 = llvm.load %50 : !llvm.ptr -> i32
+        %52 = llvm.add %40, %51 : i32
+        llvm.store %52, %arg0 : i32, !llvm.ptr
+        omp.scan inclusive(%arg0 : !llvm.ptr)
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %53 = llvm.load %arg0 : !llvm.ptr -> i32
+        %54 = llvm.load %38 : !llvm.ptr -> i32
+        %55 = llvm.sext %54 : i32 to i64
+        %63 = llvm.getelementptr %15[%55] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        llvm.store %53, %63 : i32, !llvm.ptr
+        omp.yield
+      }
+    }
+    omp.terminator
+  }
+  llvm.return
+}
+llvm.mlir.global internal @_QFEa() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+//CHECK: %vla = alloca ptr, align 8
+//CHECK: omp_parallel
+//CHECK: store ptr %vla, ptr %gep_vla, align 8
+//CHECK: @__kmpc_fork_call
+//CHECK: void @scan_reduction..omp_par
+//CHECK: %[[BUFF_PTR:.+]] = load ptr, ptr %gep_vla
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: omp.scan.loop.cont:
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: %[[FREE_VAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:  %[[ARRLAST:.+]] = getelementptr inbounds i32, ptr %[[FREE_VAR]], i32 100
+//CHECK:  %[[RES:.+]] = load i32, ptr %[[ARRLAST]], align 4
+//CHECK:  store i32 %[[RES]], ptr %loadgep{{.*}}, align 4
+//CHECK: tail call void @free(ptr %[[FREE_VAR]])
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inscan.dispatch{{.*}}:                            ; preds = %omp_loop.body{{.*}}
+//CHECK:   %[[BUFFVAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:   %[[arrayOffset1:.+]] = getelementptr inbounds i32, ptr %[[BUFFVAR]], i32 %{{.*}}
+//CHECK:   %[[BUFFVAL1:.+]] = load i32, ptr %[[arrayOffset1]], align 4
+//CHECK:   store i32 %[[BUFFVAL1]], ptr %{{.*}}, align 4
+//CHECK:   %[[LOG:.+]] = call double @llvm.log2.f64(double 1.000000e+02) #0
+//CHECK:   %[[CEIL:.+]] = call double @llvm.ceil.f64(double %[[LOG]]) #0
+//CHECK:   %[[UB:.+]] = fptoui double %[[CEIL]] to i32
+//CHECK:   br label %omp.outer.log.scan.body
+//CHECK: omp.outer.log.scan.body:
+//CHECK:   %[[K:.+]] = phi i32 [ 0, %{{.*}} ], [ %[[NEXTK:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[I:.+]] = phi i32 [ 1, %{{.*}} ], [ %[[NEXTI:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[CMP1:.+]] = icmp uge i32 99, %[[I]]
+//CHECK:   br i1 %[[CMP1]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inner.log.scan.exit:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[NEXTK]] = add nuw i32 %[[K]], 1
+//CHECK:   %[[NEXTI]] = shl nuw i32 %[[I]], 1
+//CHECK:   %[[CMP2:.+]] = icmp ne i32 %[[NEXTK]], %[[UB]]
+//CHECK:   br i1 %[[CMP2]], label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
+//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inner.log.scan.body:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[CNT:.+]] = phi i32 [ 99, %omp.outer.log.scan.body ], [ %[[CNTNXT:.+]], %omp.inner.log.scan.body ]
+//CHECK:   %[[BUFF:.+]] = load ptr, ptr %[[BUFF_PTR]]
+//CHECK:   %[[IND1:.+]] = add i32 %[[CNT]], 1
+//CHECK:   %[[IND1PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND1]]
+//CHECK:   %[[IND2:.+]] = sub nuw i32 %[[IND1]], %[[I]]
+//CHECK:   %[[IND2PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND2]]
+//CHECK:   %[[IND1VAL:.+]] = load i32, ptr %[[IND1PTR]], align 4
+//CHECK:   %[[IND2VAL:.+]] = load i32, ptr %[[IND2PTR]], align 4
+//CHECK:   %[[REDVAL:.+]] = add i32 %[[IND1VAL]], %[[IND2VAL]]
+//CHECK:   store i32 %[[REDVAL]], ptr %[[IND1PTR]], align 4
+//CHECK:   %[[CNTNXT]] = sub nuw i32 %[[CNT]], 1
+//CHECK:   %[[CMP3:.+]] = icmp uge i32 %[[CNTNXT]], %[[I]]
+//CHECK:   br i1 %[[CMP3]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inscan.dispatch:                              ; preds = %omp_loop.body
+//CHECK:   br i1 true, label %omp.before.scan.bb, label %omp.after.scan.bb
+//CHECK: omp.before.scan.bb:
+//CHECK:   store i32 0, ptr %[[REDPRIV:.+]], align 4
+//CHECK: omp.loop_nest.region:                             ; preds = %omp.before.scan.bb
+//CHECK:   %[[BUFFER:.+]] = load ptr, ptr %loadgep_vla, align 8
+//CHECK:   %[[ARRAYOFFSET2:.+]] = getelementptr inbounds i32, ptr %[[BUFFER]], i32 %{{.*}}
+//CHECK-NEXT:   %[[REDPRIVVAL:.+]] = load i32, ptr %{{.*}}, align 4
+//CHECK:   store i32 %[[REDPRIVVAL]], ptr %[[ARRAYOFFSET2]], align 4
+//CHECK:   br label %omp.scan.loop.exit
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..7edf88b5ea559 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -120,10 +120,10 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.wsloop operation}}
-  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
-  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+llvm.func @simd_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.simd operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       omp.scan inclusive(%prv : !llvm.ptr)
       omp.yield
@@ -162,6 +162,39 @@ llvm.func @parallel_task_reduction_modifier_byref(%x : !llvm.ptr) {
 
 // -----
 
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> f32
+  llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
+  omp.yield
+}
+llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
+  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+    // expected-error at below {{LLVM Translation failed for operation: omp.loop_nest}}
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      // expected-error at below {{not yet implemented: Unhandled clause exclusive in omp.scan operation}}
+      // expected-error at below {{LLVM Translation failed for operation: omp.scan}}
+      omp.scan exclusive(%prv : !llvm.ptr)
+      omp.yield
+    }
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @single_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.single operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.single}}
diff --git a/openmp/runtime/test/scan/scan.f90 b/openmp/runtime/test/scan/scan.f90
new file mode 100644
index 0000000000000..76263508105d0
--- /dev/null
+++ b/openmp/runtime/test/scan/scan.f90
@@ -0,0 +1,38 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+program inclusive_scan
+ implicit none
+ integer, parameter :: n = 100
+ integer a(n), b(n)
+ integer x, k, y, z
+
+ ! initialization
+ x = 0
+ do k = 1, n
+  a(k) = k
+ end do
+
+ ! a(k) is included in the computation of producing results in b(k)
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, n
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+
+ print *,'x =', x
+ do k = 1, 10
+  print *, 'b(', k, ') =', b(k)
+ end do
+end program
+!CHECK: x = 5050
+!CHECK: b( 1 ) = 1
+!CHECK: b( 2 ) = 3
+!CHECK: b( 3 ) = 6
+!CHECK: b( 4 ) = 10
+!CHECK: b( 5 ) = 15
+!CHECK: b( 6 ) = 21
+!CHECK: b( 7 ) = 28
+!CHECK: b( 8 ) = 36
+!CHECK: b( 9 ) = 45
+!CHECK: b( 10 ) = 55

>From 4011dd537d9830ee4a0379cba80413d1e4cdf9c3 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 02:44:02 -0500
Subject: [PATCH 2/2] formatting correction

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp        | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 297ebac51de88..e18cc0df79860 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4534,12 +4534,11 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isInScanRegion =
       wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
                                      mlir::omp::ReductionModifier::inscan);
-  if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
-                               moduleTranslation,
-                               afterAllocas.get()->getSinglePredecessor(),
-                               reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores,
-                               isInScanRegion)))
+  if (failed(initReductionVars(
+          wsloopOp, reductionArgs, builder, moduleTranslation,
+          afterAllocas.get()->getSinglePredecessor(), reductionDecls,
+          privateReductionVariables, reductionVariableMap, isByRef,
+          deferredStores, isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -5262,9 +5261,9 @@ initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
     SmallVector<llvm::Value *, 1> phis;
     mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
                           reductionVariableMap, i);
-    if (failed(inlineConvertOmpRegions(
-            reductionDecls[i].getInitializerRegion(),
-            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+    if (failed(inlineConvertOmpRegions(reductionDecls[i].getInitializerRegion(),
+                                       "omp.scan.reduction.init", builder,
+                                       moduleTranslation, &phis)))
       return failure();
     assert(phis.size() == 1 &&
            "expected one value to be yielded from the reduction init region");


From llvmlistbot at llvm.org  Wed Jul  1 01:32:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 01:32:38 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a44d0a6.70920dee.23960.b9f3@mx.google.com>


Ritanya-B-Bharadwaj wrote:

I'm still new to reviewing in this area, so please don't take my review as sufficient on its own. That said, I went through the PR and left some findings below. Let me know if anything's unclear or if you have questions.


https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Wed Jul  1 01:50:54 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 01:50:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a44d4ee.1a4df1a5.3108a4.10b3@mx.google.com>


https://github.com/andrey-golubev updated https://github.com/llvm/llvm-project/pull/205080

>From 588b9902bb74a182580cedd901fca4e0318882e9 Mon Sep 17 00:00:00 2001
From: "Golubev, Andrey" <andrey.golubev at intel.com>
Date: Fri, 5 Jun 2026 15:18:07 +0000
Subject: [PATCH 1/2] [mlir][bufferization] Introduce castBufferTypeFn hook and
 API

This PR is the second part of the work that aims to allow customizations
in resolving mismatching buffer types.
(see 0b1fe74f079fd330e4d4d85ce33b7f5702da1be7 for part one)

Add a new bufferization hook that lets downstream bufferization
implementations materialize an operation to resolve a pre-reconciled
buffer type mismatch.

The hook + API is used as a fallback mechanism in several upstream operations,
as well as in the general bufferization infrastructure. Semantically,
this is a "cast" creation akin to `createAlloc()` and `createMemCpy()`
but for the cases where a memref::CastOp-like operation is acceptable.

A valid value is returned whenever a cast-like operation is created,
failure signifies a failed attempt to materialize a cast (in theory,
there could be a fall back to alloc + copy either externally, but this
right now is not considered in most contexts). By default, a
`memref.cast` operation is created that casts a bufferized value to a
reconciled type.
---
 .../IR/BufferizableOpInterface.h              |  11 ++
 .../BufferizableOpInterfaceImpl.cpp           |   9 +-
 .../IR/BufferizableOpInterface.cpp            |  17 +++
 .../Bufferization/IR/BufferizationOps.cpp     |  16 ++-
 .../Bufferization/Transforms/Bufferize.cpp    |   7 +-
 .../BufferizableOpInterfaceImpl.cpp           |  38 ++----
 .../test-one-shot-module-bufferize.mlir       | 111 ++++++++++++++++--
 .../TestOneShotModuleBufferize.cpp            |  35 ++++--
 mlir/test/lib/Dialect/Test/TestOpDefs.cpp     |  60 ++++++----
 mlir/test/lib/Dialect/Test/TestOps.td         |  16 ++-
 mlir/test/lib/Dialect/Test/TestTypeDefs.td    |   6 +-
 11 files changed, 241 insertions(+), 85 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f7a6a638f307..9f49a31c1583b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -258,6 +258,12 @@ struct BufferizationOptions {
   /// Memcpy function: Generate a memcpy between two buffers.
   using MemCpyFn =
       std::function<LogicalResult(OpBuilder &, Location, Value, Value)>;
+  /// Cast function: Convert a buffer value to a new value with the specified
+  /// type. This method is typically used when a simple cast-like operation is
+  /// sufficient to convert the buffer value, for example, when layout maps
+  /// between buffer value and resulting type do not match.
+  using CastFn =
+      std::function<FailureOr<Value>(OpBuilder &, Location, Type, Value)>;
   /// Initializer function for analysis state.
   using AnalysisStateInitFn = std::function<void(AnalysisState &)>;
   /// Tensor-like -> Buffer-like type conversion.
@@ -300,6 +306,7 @@ struct BufferizationOptions {
   /// Helper functions for allocation and memory copying.
   std::optional<AllocationFn> allocationFn;
   std::optional<MemCpyFn> memCpyFn;
+  std::optional<CastFn> castFn;
 
   /// Create a memref allocation with the given type and dynamic extents.
   FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
@@ -309,6 +316,10 @@ struct BufferizationOptions {
   LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
                              Value to) const;
 
+  /// Creates a cast function from a buffer value to a new type.
+  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
+                              Value value) const;
+
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index 35e472470560a..7ef115b7ef003 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -172,12 +172,9 @@ struct SelectOpInterface
         return selectOp->emitError(
             "incompatible buffer types on true/false operands");
       }
-      if (trueBuffer.getType() != *targetType)
-        trueBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, trueBuffer);
-      if (falseBuffer.getType() != *targetType)
-        falseBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, falseBuffer);
+      trueBuffer = *options.createCast(rewriter, loc, *targetType, trueBuffer);
+      falseBuffer =
+          *options.createCast(rewriter, loc, *targetType, falseBuffer);
     }
 
     replaceOpWithNewBufferizedOp<arith::SelectOp>(
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index e0a35e1932f3d..79ab923864871 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -840,6 +840,23 @@ LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
   return success();
 }
 
+FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
+                                                  Type dest,
+                                                  Value value) const {
+  // If the value already has the correct type, no cast is needed.
+  if (value.getType() == dest)
+    return value;
+
+  if (castFn)
+    return (*castFn)(b, loc, dest, value);
+
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 2a1dcc89a83ed..320f426985750 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = memref::CastOp::create(b, value.getLoc(), destType, value);
+    Value casted = *options.createCast(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -100,6 +100,20 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
     return success();
   }
 
+  if (!llvm::isa<BaseMemRefType>(srcType) ||
+      !llvm::isa<BaseMemRefType>(destType)) {
+    // Non-builtin case: the best is to try the user-provided cast.
+    assert(options.castFn.has_value() &&
+           "user-provided cast is required for non-builtin types");
+    auto replacement =
+        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
+                           destType, bufferToTensor.getBuffer());
+    if (failed(replacement))
+      return failure();
+    rewriter.replaceOp(toBuffer, *replacement);
+    return success();
+  }
+
   auto rankedSrcType = llvm::dyn_cast<MemRefType>(srcType);
   auto rankedDestType = llvm::dyn_cast<MemRefType>(destType);
   auto unrankedSrcType = llvm::dyn_cast<UnrankedMemRefType>(srcType);
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index 8bdee09d0d6e7..e0262abf313b8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -476,9 +476,10 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
           rewriter, operand.getLoc(), *operandBufferType, operand);
       // A cast is needed if the operand and the block argument have different
       // bufferized types.
-      if (type != *operandBufferType)
-        bufferizedOperand = memref::CastOp::create(rewriter, operand.getLoc(),
-                                                   type, bufferizedOperand);
+      if (type != *operandBufferType) {
+        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
+                                                type, bufferizedOperand);
+      }
       newOperands.push_back(bufferizedOperand);
     }
     operands.getMutableForwardedOperands().assign(newOperands);
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index b9662751aa7f4..db26bf2f82b6a 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -29,25 +29,6 @@ namespace mlir {
 namespace scf {
 namespace {
 
-/// Helper function for loop bufferization. Cast the given buffer to the given
-/// memref type.
-static Value castBuffer(OpBuilder &b, Value buffer, Type type) {
-  // If the buffer already has the correct type, no cast is needed.
-  if (buffer.getType() == type)
-    return buffer;
-
-  // TODO: Properly support with options, for now it is hardcoded MemRef type
-  // based approach
-  assert(isa<BaseMemRefType>(type) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(buffer.getType()) && "expected BaseMemRefType");
-  // TODO: In case `type` has a layout map that is not the fully dynamic
-  // one, we may not be able to cast the buffer. In that case, the loop
-  // iter_arg's layout map must be changed (see uses of `castBuffer`).
-  assert(memref::CastOp::areCastCompatible(buffer.getType(), type) &&
-         "scf.while op bufferization: cast incompatible");
-  return memref::CastOp::create(b, buffer.getLoc(), type, buffer).getResult();
-}
-
 /// Helper function for loop bufferization. Return "true" if the given value
 /// is guaranteed to not alias with an external tensor apart from values in
 /// `exceptions`. A value is external if it is defined outside of the given
@@ -114,7 +95,8 @@ struct ConditionOpInterface
             whileOp.getAfterArguments()[it.index()], options, state);
         if (failed(resultType))
           return failure();
-        Value buffer = castBuffer(rewriter, *maybeBuffer, *resultType);
+        Value buffer = *options.createCast(rewriter, maybeBuffer->getLoc(),
+                                           *resultType, *maybeBuffer);
         newArgs.push_back(buffer);
       } else {
         newArgs.push_back(value);
@@ -206,7 +188,7 @@ struct ExecuteRegionOpInterface
     rewriter.setInsertionPointAfter(newOp);
     SmallVector<Value> newResults;
     for (const auto &it : llvm::enumerate(executeRegionOp->getResultTypes())) {
-      if (isa<TensorType>(it.value())) {
+      if (isa<TensorLikeType>(it.value())) {
         newResults.push_back(bufferization::ToTensorOp::create(
             rewriter, executeRegionOp.getLoc(), it.value(),
             newOp->getResult(it.index())));
@@ -361,7 +343,7 @@ struct IndexSwitchOpInterface
     // Compute bufferized result types.
     SmallVector<Type> newTypes;
     for (Value result : switchOp.getResults()) {
-      if (!isa<TensorType>(result.getType())) {
+      if (!isa<TensorLikeType>(result.getType())) {
         newTypes.push_back(result.getType());
         continue;
       }
@@ -762,7 +744,8 @@ struct ForOpInterface
       auto targetType = bufferization::getBufferType(result, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(castBuffer(rewriter, initArg, *targetType));
+      castedInitArgs.push_back(*options.createCast(rewriter, initArg.getLoc(),
+                                                   *targetType, initArg));
     }
 
     // Construct a new scf.for op with memref instead of tensor values.
@@ -986,7 +969,8 @@ struct WhileOpInterface
       auto targetType = bufferization::getBufferType(beforeArg, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(castBuffer(rewriter, initArg, *targetType));
+      castedInitArgs.push_back(*options.createCast(rewriter, initArg.getLoc(),
+                                                   *targetType, initArg));
     }
 
     // The result types of a WhileOp are the same as the "after" bbArg types.
@@ -1186,14 +1170,16 @@ struct YieldOpInterface
               yieldOp->getParentOp()->getResult(it.index()), options, state);
           if (failed(resultType))
             return failure();
-          buffer = castBuffer(rewriter, buffer, *resultType);
+          buffer = *options.createCast(rewriter, buffer.getLoc(), *resultType,
+                                       buffer);
         } else if (auto whileOp =
                        dyn_cast<scf::WhileOp>(yieldOp->getParentOp())) {
           FailureOr<BufferLikeType> resultType = bufferization::getBufferType(
               whileOp.getBeforeArguments()[it.index()], options, state);
           if (failed(resultType))
             return failure();
-          buffer = castBuffer(rewriter, buffer, *resultType);
+          buffer = *options.createCast(rewriter, buffer.getLoc(), *resultType,
+                                       buffer);
         }
         newResults.push_back(buffer);
       } else {
diff --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
index b2674e9497927..ccc60d42ca23a 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
@@ -312,7 +312,7 @@ module @BufferizeLayoutForFunction {
     %memref = "test.tensor_with_future_layout"() {layout = #layout1}
       : () -> tensor<10xf32>
 
-    // CHECK: %[[out:.*]] = memref.cast %[[memref]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[memref]]
     // CHECK-SAME:  to memref<10xf32>
     // CHECK: return %[[out]]
     return %memref : tensor<10xf32>
@@ -341,7 +341,7 @@ module @BufferizeLayoutMismatchInsideScfIf {
     } else {
       // CHECK: %[[another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %another = "test.tensor_with_future_layout"() {layout = #layout2}
         : () -> tensor<10xf32>
@@ -349,7 +349,7 @@ module @BufferizeLayoutMismatchInsideScfIf {
       scf.yield %another : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[if]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[if]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %ret : tensor<10xf32>
@@ -381,7 +381,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
     case 1 {
       // CHECK: %[[another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %another = "test.tensor_with_future_layout"() {layout = #layout2}
         : () -> tensor<10xf32>
@@ -391,7 +391,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
     default {
       // CHECK: %[[yet_another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_c">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[yet_another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[yet_another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %yet_another = "test.tensor_with_future_layout"() {layout = #layout3}
         : () -> tensor<10xf32>
@@ -399,7 +399,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
       scf.yield %yet_another : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[switch]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[switch]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %ret : tensor<10xf32>
@@ -430,7 +430,7 @@ module @BufferizeLayoutMismatchInsideScfFor {
         iter_args(%iter = %init) -> (tensor<10xf32>) {
       // CHECK: %[[conflict:.*]] = "test.dummy_memref_op"(%[[iter]])
       // CHECK-SAME:  -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[conflict]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[conflict]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %conflict = "test.force_new_layout"(%iter) {layout = #layout2}
         : (tensor<10xf32>) -> tensor<10xf32>
@@ -438,7 +438,7 @@ module @BufferizeLayoutMismatchInsideScfFor {
       scf.yield %conflict : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[loop]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %loop : tensor<10xf32>
@@ -450,6 +450,45 @@ module @BufferizeLayoutMismatchInsideScfFor {
 #layout1 = #test.memref_layout<"layout_a">
 #layout2 = #test.memref_layout<"layout_b">
 
+// CHECK-LABEL: @BufferizeLayoutMismatchInsideScfForWithUserType
+module @BufferizeLayoutMismatchInsideScfForWithUserType {
+  // CHECK: func.func @mismatch_in_for
+  // CHECK-SAME:  -> !test.test_memref<[10], f32>
+  func.func @mismatch_in_for(
+      %lb: index, %ub: index, %step: index)
+      -> !test.test_tensor<[10], f32> {
+    // CHECK: %[[init:.*]] = "test.create_memref_op"
+    // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+    %init = "test.tensor_with_future_layout"() {layout = #layout1}
+      : () -> !test.test_tensor<[10], f32>
+
+    // CHECK: %[[loop:.+]] = scf.for
+    // CHECK-SAME: iter_args(%[[iter:.*]] = %[[init]])
+    // CHECK-SAME: -> (!test.test_memref<[10], f32, #test.memref_layout<"layout_a">>)
+    %loop = scf.for %i = %lb to %ub step %step
+        iter_args(%iter = %init) -> (!test.test_tensor<[10], f32>) {
+      // CHECK: %[[conflict:.*]] = "test.dummy_memref_op"(%[[iter]])
+      // CHECK-SAME:  -> !test.test_memref<[10], f32, #test.memref_layout<"layout_b">>
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[conflict]]
+      // CHECK-SAME: to !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+      %conflict = "test.force_new_layout"(%iter) {layout = #layout2}
+        : (!test.test_tensor<[10], f32>) -> !test.test_tensor<[10], f32>
+      // CHECK: scf.yield %[[cast]]
+      scf.yield %conflict : !test.test_tensor<[10], f32 >
+    }
+
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
+    // CHECK-SAME: to !test.test_memref<[10], f32>
+    // CHECK: return %[[out]]
+    return %loop : !test.test_tensor<[10], f32>
+  }
+}
+
+// -----
+
+#layout1 = #test.memref_layout<"layout_a">
+#layout2 = #test.memref_layout<"layout_b">
+
 // Test that custom layout can co-exist in principle within an "end-to-end"
 // SCF example (`for { extract slice -> custom op -> insert slice }`) without
 // bufferization failing completely due to a layout mismatch. The fact that the
@@ -495,7 +534,7 @@ module @BufferizeLayoutMismatchInsideScfForWithSubviews {
       scf.yield %out : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[loop]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %loop : tensor<10xf32>
@@ -528,7 +567,7 @@ module @BufferizeLayoutMismatchInScfExecuteRegion {
       ^bb2:
         // CHECK: %[[another:.*]] = "test.create_memref_op"
         // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-        // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+        // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
         // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
         %another = "test.tensor_with_future_layout"() {layout = #layout2}
           : () -> tensor<10xf32>
@@ -540,13 +579,59 @@ module @BufferizeLayoutMismatchInScfExecuteRegion {
         scf.yield %res : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[region]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[region]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %out : tensor<10xf32>
   }
 }
 
+// -----
+
+#layout1 = #test.memref_layout<"layout_a">
+#layout2 = #test.memref_layout<"layout_b">
+
+// CHECK-LABEL: @BufferizeLayoutMismatchInScfExecuteRegionWithUserType
+module @BufferizeLayoutMismatchInScfExecuteRegionWithUserType {
+  // CHECK: func.func @mismatch_in_scf_execute_region
+  // CHECK-SAME: -> !test.test_memref<[10], f32>
+  func.func @mismatch_in_scf_execute_region(%cond: i1) -> !test.test_tensor<[10], f32> {
+    // CHECK: %[[region:.*]] = scf.execute_region
+    // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+    %out = scf.execute_region -> !test.test_tensor<[10], f32> {
+      cf.cond_br %cond, ^bb1, ^bb2
+
+      ^bb1:
+        // CHECK: %[[one:.*]] = "test.create_memref_op"
+        // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        %one = "test.tensor_with_future_layout"() {layout = #layout1}
+          : () -> !test.test_tensor<[10], f32>
+        // CHECK: cf.br ^bb3(%[[one]]
+        cf.br ^bb3(%one : !test.test_tensor<[10], f32>)
+
+      ^bb2:
+        // CHECK: %[[another:.*]] = "test.create_memref_op"
+        // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_b">>
+        // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
+        // CHECK-SAME: to !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        %another = "test.tensor_with_future_layout"() {layout = #layout2}
+          : () -> !test.test_tensor<[10], f32>
+        // CHECK: cf.br ^bb3(%[[cast]]
+        cf.br ^bb3(%another : !test.test_tensor<[10], f32>)
+
+      ^bb3(%res: !test.test_tensor<[10], f32>):
+        // CHECK: scf.yield {{%.*}} : !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        scf.yield %res : !test.test_tensor<[10], f32>
+    }
+
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[region]]
+    // CHECK-SAME: to !test.test_memref<[10], f32>
+    // CHECK: return %[[out]]
+    return %out : !test.test_tensor<[10], f32>
+  }
+}
+
+
 // -----
 
 #layout1 = #test.memref_layout<"layout_a">
@@ -566,13 +651,13 @@ module @BufferizeLayoutMismatchInArithSelect {
     %tensor2 = "test.tensor_with_future_layout"() {layout = #layout2}
       : () -> tensor<10xf32>
 
-    // CHECK: %[[cast:.*]] = memref.cast %[[memref2]]
+    // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[memref2]]
     // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
 
     // CHECK: %[[select:.*]] = arith.select %[[cond]], %[[memref1]], %[[cast]]
     %select = arith.select %cond, %tensor1, %tensor2 : i1, tensor<10xf32>
 
-    // CHECK: %[[out:.*]] = memref.cast %[[select]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[select]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %select : tensor<10xf32>
diff --git a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
index 00eb2dc92e14f..036e05bfc2e02 100644
--- a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
+++ b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
@@ -16,6 +16,7 @@
 
 #include "TestAttributes.h" // TestTensorEncodingAttr, TestMemRefLayoutAttr
 #include "TestDialect.h"
+#include "TestOps.h"
 
 using namespace mlir;
 
@@ -30,6 +31,21 @@ getMemRefLayoutForTensorEncoding(RankedTensorType tensorType) {
   return {};
 }
 
+test::TestMemRefLayoutAttr
+getLayoutFromBuffer(bufferization::BufferLikeType buffer) {
+  auto layout =
+      llvm::TypeSwitch<bufferization::BufferLikeType,
+                       MemRefLayoutAttrInterface>(buffer)
+          .Case([&](MemRefType memref) { return memref.getLayout(); })
+          .Case([&](test::TestMemrefType testMemref) {
+            return cast<MemRefLayoutAttrInterface>(testMemref.getLayout());
+          })
+          .Default([](bufferization::BufferLikeType) {
+            return MemRefLayoutAttrInterface();
+          });
+  return dyn_cast_or_null<test::TestMemRefLayoutAttr>(layout);
+}
+
 struct TestOneShotModuleBufferizePass
     : public PassWrapper<TestOneShotModuleBufferizePass, OperationPass<>> {
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestOneShotModuleBufferizePass)
@@ -82,9 +98,14 @@ struct TestOneShotModuleBufferizePass
               })
               .Case([&](test::TestTensorType testTensorType)
                         -> bufferization::BufferLikeType {
+                assert(memSpace == nullptr &&
+                       "memory space is not supported for test types");
+                // Note: unknown type conversion cannot create layout because
+                // test tensor does not have an encoding, this is probably
+                // enough for a test.
                 return test::TestMemrefType::get(
                     testTensorType.getContext(), testTensorType.getShape(),
-                    testTensorType.getElementType(), memSpace);
+                    testTensorType.getElementType(), /*layout=*/nullptr);
               })
               .Default([&](bufferization::TensorLikeType tensor) {
                 llvm_unreachable("unexpected tensor type");
@@ -97,13 +118,8 @@ struct TestOneShotModuleBufferizePass
         [](bufferization::BufferLikeType x, bufferization::BufferLikeType y,
            const bufferization::BufferizationOptions &)
         -> FailureOr<bufferization::BufferLikeType> {
-      auto getLayout = [](bufferization::BufferLikeType t) {
-        auto m = dyn_cast<MemRefType>(t);
-        return m ? dyn_cast<test::TestMemRefLayoutAttr>(m.getLayout())
-                 : test::TestMemRefLayoutAttr();
-      };
-      auto lhsLayout = getLayout(x);
-      auto rhsLayout = getLayout(y);
+      auto lhsLayout = getLayoutFromBuffer(x);
+      auto rhsLayout = getLayoutFromBuffer(y);
       if (lhsLayout && rhsLayout) {
         return lhsLayout.getDummy().getValue() <=
                        rhsLayout.getDummy().getValue()
@@ -112,6 +128,9 @@ struct TestOneShotModuleBufferizePass
       }
       return rhsLayout ? y : x;
     };
+    opt.castFn = [&](OpBuilder &b, Location loc, Type dest, Value value) {
+      return test::TestDummyCastOp::create(b, loc, dest, value).getResult();
+    };
     // Function signature update only works with memref.cast. Disable it to
     // align behaviour for upstream and user casts.
     opt.inferFunctionResultLayout = false;
diff --git a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
index 20d72a2fd09ca..75dae36fe9ce8 100644
--- a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
@@ -2068,19 +2068,27 @@ test::TestTensorWithFutureLayoutOp::getBufferType(
     mlir::Value value, const mlir::bufferization::BufferizationOptions &,
     const mlir::bufferization::BufferizationState &,
     llvm::SmallVector<::mlir::Value> &) {
-  auto tensorType = dyn_cast<RankedTensorType>(value.getType());
-  if (!tensorType)
-    return failure();
-
-  // Set the memref layout to the op's 'layout' attribute, ignoring
-  // any pre-existing tensor encoding. This is what lets two
-  // `test.tensor_with_layout` ops produce *bufferized* memrefs with different
-  // layouts while keeping their *tensor* result types identical -- which is
-  // required to construct SCF iter_arg/branch mismatches that the verifier
-  // still accepts.
-  auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
-  return cast<bufferization::BufferLikeType>(MemRefType::get(
-      tensorType.getShape(), tensorType.getElementType(), layout));
+  return llvm::TypeSwitch<mlir::Type,
+                          mlir::FailureOr<mlir::bufferization::BufferLikeType>>(
+             value.getType())
+      .Case([&](RankedTensorType tensorType) {
+        // Set the memref layout to the op's 'layout' attribute, ignoring
+        // any pre-existing tensor encoding. This is what lets two
+        // `test.tensor_with_layout` ops produce *bufferized* memrefs with
+        // different layouts while keeping their *tensor* result types identical
+        // -- which is required to construct SCF iter_arg/branch mismatches that
+        // the verifier still accepts.
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(MemRefType::get(
+            tensorType.getShape(), tensorType.getElementType(), layout));
+      })
+      .Case([&](TestTensorType tensorType) {
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(
+            TestMemrefType::get(tensorType.getContext(), tensorType.getShape(),
+                                tensorType.getElementType(), layout));
+      })
+      .Default([&](Type) { return emitError("unknown type"); });
 }
 
 LogicalResult test::TestForceNewLayoutOp::bufferize(
@@ -2107,15 +2115,23 @@ test::TestForceNewLayoutOp::getBufferType(
     mlir::Value value, const bufferization::BufferizationOptions &options,
     const bufferization::BufferizationState &,
     llvm::SmallVector<::mlir::Value> &) {
-  auto tensorType = dyn_cast<RankedTensorType>(value.getType());
-  if (!tensorType)
-    return failure();
-
-  // Set the memref layout to the op's 'layout' attribute, ignoring any
-  // pre-existing tensor encoding.
-  auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
-  return cast<bufferization::BufferLikeType>(MemRefType::get(
-      tensorType.getShape(), tensorType.getElementType(), layout));
+  return llvm::TypeSwitch<mlir::Type,
+                          mlir::FailureOr<mlir::bufferization::BufferLikeType>>(
+             value.getType())
+      .Case([&](RankedTensorType tensorType) {
+        // Set the memref layout to the op's 'layout' attribute, ignoring any
+        // pre-existing tensor encoding.
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(MemRefType::get(
+            tensorType.getShape(), tensorType.getElementType(), layout));
+      })
+      .Case([&](TestTensorType tensorType) {
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(
+            TestMemrefType::get(tensorType.getContext(), tensorType.getShape(),
+                                tensorType.getElementType(), layout));
+      })
+      .Default([&](Type) { return emitError("unknown type"); });
 }
 
 // Define a custom builder for ManyRegionsOp declared in TestOps.td.
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index 29f456d1ec7eb..c12e2abb3ab75 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -4239,7 +4239,7 @@ def TestTensorWithFutureLayoutOp
   }];
 
   let arguments = (ins TestMemRefLayoutAttr:$layout);
-  let results = (outs AnyRankedTensor:$output);
+  let results = (outs Bufferization_TensorLikeTypeInterface:$output);
   let extraClassDefinition = [{
     bool test::TestTensorWithFutureLayoutOp::bufferizesToMemoryRead(
         ::mlir::OpOperand&,
@@ -4285,10 +4285,10 @@ def TestForceNewLayoutOp
   }];
 
   let arguments = (ins
-    AnyRankedTensor:$input,
+    Bufferization_TensorLikeTypeInterface:$input,
     TestMemRefLayoutAttr:$layout
   );
-  let results = (outs AnyRankedTensor:$output);
+  let results = (outs Bufferization_TensorLikeTypeInterface:$output);
   let extraClassDefinition = [{
     bool test::TestForceNewLayoutOp::bufferizesToMemoryRead(::mlir::OpOperand&,
         const ::mlir::bufferization::AnalysisState&) {
@@ -4322,6 +4322,16 @@ def TestForceNewLayoutOp
   }];
 }
 
+def TestDummyCastOp : TEST_Op<"dummy_cast_op", []> {
+  let arguments = (ins
+    Arg<Bufferization_BufferLikeTypeInterface>:$source
+  );
+  let results = (outs
+    Arg<Bufferization_BufferLikeTypeInterface>:$dest
+  );
+  let assemblyFormat = "$source attr-dict `:` type($source) `to` type($dest)";
+}
+
 //===----------------------------------------------------------------------===//
 // Test assembly format references
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Dialect/Test/TestTypeDefs.td b/mlir/test/lib/Dialect/Test/TestTypeDefs.td
index f405940358ca0..58c340a935bdb 100644
--- a/mlir/test/lib/Dialect/Test/TestTypeDefs.td
+++ b/mlir/test/lib/Dialect/Test/TestTypeDefs.td
@@ -477,9 +477,9 @@ def TestMemrefType : Test_Type<"TestMemref",
   let parameters = (ins
     ArrayRefParameter<"int64_t">:$shape,
     "mlir::Type":$elementType,
-    DefaultValuedParameter<"mlir::Attribute", "nullptr">:$memSpace
+    DefaultValuedParameter<"mlir::MemRefLayoutAttrInterface", "nullptr">:$layout
   );
-  let assemblyFormat = "`<` `[` $shape `]` `,` $elementType (`,` $memSpace^)? `>`";
+  let assemblyFormat = "`<` `[` $shape `]` `,` $elementType (`,` $layout^)? `>`";
 
   let extraClassDeclaration = [{
     // ShapedTypeInterface:
@@ -489,7 +489,7 @@ def TestMemrefType : Test_Type<"TestMemref",
     test::TestMemrefType cloneWith(std::optional<llvm::ArrayRef<int64_t>> shape,
                                    mlir::Type elementType) const {
       return test::TestMemrefType::get(
-        getContext(), shape.value_or(getShape()), elementType, getMemSpace());
+        getContext(), shape.value_or(getShape()), elementType, getLayout());
     }
   }];
 }

>From bf0491143f858b9f74dbd81f3a41f6467e7f88aa Mon Sep 17 00:00:00 2001
From: "Golubev, Andrey" <andrey.golubev at intel.com>
Date: Wed, 1 Jul 2026 08:09:30 +0000
Subject: [PATCH 2/2] Move check for type equality outside of createCast()

---
 .../BufferizableOpInterfaceImpl.cpp           | 11 ++++++--
 .../IR/BufferizableOpInterface.cpp            |  4 ---
 .../BufferizableOpInterfaceImpl.cpp           | 28 ++++++++++++-------
 3 files changed, 26 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index 7ef115b7ef003..f810000dd082f 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -172,9 +172,14 @@ struct SelectOpInterface
         return selectOp->emitError(
             "incompatible buffer types on true/false operands");
       }
-      trueBuffer = *options.createCast(rewriter, loc, *targetType, trueBuffer);
-      falseBuffer =
-          *options.createCast(rewriter, loc, *targetType, falseBuffer);
+      if (trueBuffer.getType() != *targetType) {
+        trueBuffer =
+            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+      }
+      if (falseBuffer.getType() != *targetType) {
+        falseBuffer =
+            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+      }
     }
 
     replaceOpWithNewBufferizedOp<arith::SelectOp>(
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 79ab923864871..7f656164f3a15 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -843,10 +843,6 @@ LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
 FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
                                                   Type dest,
                                                   Value value) const {
-  // If the value already has the correct type, no cast is needed.
-  if (value.getType() == dest)
-    return value;
-
   if (castFn)
     return (*castFn)(b, loc, dest, value);
 
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index db26bf2f82b6a..401be7440f5a2 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -29,6 +29,17 @@ namespace mlir {
 namespace scf {
 namespace {
 
+/// Helper function for loop bufferization. Cast the given buffer to the given
+/// memref type.
+static Value castBuffer(OpBuilder &b, Value buffer, Type type,
+                        const BufferizationOptions &options) {
+  // If the buffer already has the correct type, no cast is needed.
+  if (buffer.getType() == type)
+    return buffer;
+
+  return *options.createCast(b, buffer.getLoc(), type, buffer);
+}
+
 /// Helper function for loop bufferization. Return "true" if the given value
 /// is guaranteed to not alias with an external tensor apart from values in
 /// `exceptions`. A value is external if it is defined outside of the given
@@ -95,8 +106,7 @@ struct ConditionOpInterface
             whileOp.getAfterArguments()[it.index()], options, state);
         if (failed(resultType))
           return failure();
-        Value buffer = *options.createCast(rewriter, maybeBuffer->getLoc(),
-                                           *resultType, *maybeBuffer);
+        Value buffer = castBuffer(rewriter, *maybeBuffer, *resultType, options);
         newArgs.push_back(buffer);
       } else {
         newArgs.push_back(value);
@@ -744,8 +754,8 @@ struct ForOpInterface
       auto targetType = bufferization::getBufferType(result, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(*options.createCast(rewriter, initArg.getLoc(),
-                                                   *targetType, initArg));
+      castedInitArgs.push_back(
+          castBuffer(rewriter, initArg, *targetType, options));
     }
 
     // Construct a new scf.for op with memref instead of tensor values.
@@ -969,8 +979,8 @@ struct WhileOpInterface
       auto targetType = bufferization::getBufferType(beforeArg, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(*options.createCast(rewriter, initArg.getLoc(),
-                                                   *targetType, initArg));
+      castedInitArgs.push_back(
+          castBuffer(rewriter, initArg, *targetType, options));
     }
 
     // The result types of a WhileOp are the same as the "after" bbArg types.
@@ -1170,16 +1180,14 @@ struct YieldOpInterface
               yieldOp->getParentOp()->getResult(it.index()), options, state);
           if (failed(resultType))
             return failure();
-          buffer = *options.createCast(rewriter, buffer.getLoc(), *resultType,
-                                       buffer);
+          buffer = castBuffer(rewriter, buffer, *resultType, options);
         } else if (auto whileOp =
                        dyn_cast<scf::WhileOp>(yieldOp->getParentOp())) {
           FailureOr<BufferLikeType> resultType = bufferization::getBufferType(
               whileOp.getBeforeArguments()[it.index()], options, state);
           if (failed(resultType))
             return failure();
-          buffer = *options.createCast(rewriter, buffer.getLoc(), *resultType,
-                                       buffer);
+          buffer = castBuffer(rewriter, buffer, *resultType, options);
         }
         newResults.push_back(buffer);
       } else {


From llvmlistbot at llvm.org  Wed Jul  1 01:52:43 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 01:52:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a44d55b.97bb2efc.2241ed.32ed@mx.google.com>


================
@@ -172,12 +172,9 @@ struct SelectOpInterface
         return selectOp->emitError(
             "incompatible buffer types on true/false operands");
       }
-      if (trueBuffer.getType() != *targetType)
-        trueBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, trueBuffer);
-      if (falseBuffer.getType() != *targetType)
-        falseBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, falseBuffer);
+      trueBuffer = *options.createCast(rewriter, loc, *targetType, trueBuffer);
----------------
andrey-golubev wrote:

done! it does look a bit clunkier now but i guess this is something that I'll revise in the follow-up: when cleaning up the `create*()` functions.

https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 02:02:58 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 01 Jul 2026 02:02:58 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Lower scan
 directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a44d7c2.dd2b4bd3.10c1cf.3a1b@mx.google.com>


https://github.com/chandraghale ready_for_review https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Wed Jul  1 02:07:50 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 01 Jul 2026 02:07:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for non-trivial elementwise operations (PR #202574)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202574@github.com>
Message-ID: <6a44d8e6.70920dee.23960.576b@mx.google.com>


IgWod wrote:

Sorry, I've been quite busy recently. If you could split it that would make reviewing easier. I'll try to catch up on reviews in the next few days.

https://github.com/llvm/llvm-project/pull/202574

From llvmlistbot at llvm.org  Wed Jul  1 02:10:44 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 01 Jul 2026 02:10:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix crash on empty composite
 in replicated constants pass (PR #206647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206647@github.com>
Message-ID: <6a44d994.9b510543.2bbb51.281f@mx.google.com>


https://github.com/IgWod approved this pull request.

Looks good, thanks!

https://github.com/llvm/llvm-project/pull/206647

From llvmlistbot at llvm.org  Wed Jul  1 02:14:46 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 01 Jul 2026 02:14:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add spirv.GL.Step and
 spirv.GL.SmoothStep ops (PR #206037)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206037@github.com>
Message-ID: <6a44da86.a87eb5dc.379910.be0c@mx.google.com>


https://github.com/IgWod commented:

Looks good, but it is missing target tests.

https://github.com/llvm/llvm-project/pull/206037

From llvmlistbot at llvm.org  Wed Jul  1 02:24:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 02:24:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 713d95e - [mlir][SPIR-V] Fix crash on empty
 composite in replicated constants pass (#206647)
Message-ID: <6a44dcd9.58bdd12c.2e0810.e4e5@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-01T11:24:37+02:00
New Revision: 713d95e8f30ddca93c0eb737a0f6b2ad0f492a2c

URL: https://github.com/llvm/llvm-project/commit/713d95e8f30ddca93c0eb737a0f6b2ad0f492a2c
DIFF: https://github.com/llvm/llvm-project/commit/713d95e8f30ddca93c0eb737a0f6b2ad0f492a2c.diff

LOG: [mlir][SPIR-V] Fix crash on empty composite in replicated constants pass (#206647)

Added: 
    

Modified: 
    mlir/lib/Dialect/SPIRV/Transforms/ConvertToReplicatedConstantCompositePass.cpp
    mlir/test/Dialect/SPIRV/Transforms/replicated-const-composites.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/SPIRV/Transforms/ConvertToReplicatedConstantCompositePass.cpp b/mlir/lib/Dialect/SPIRV/Transforms/ConvertToReplicatedConstantCompositePass.cpp
index 2f5588c7f8cc7..739c3092d36a7 100644
--- a/mlir/lib/Dialect/SPIRV/Transforms/ConvertToReplicatedConstantCompositePass.cpp
+++ b/mlir/lib/Dialect/SPIRV/Transforms/ConvertToReplicatedConstantCompositePass.cpp
@@ -28,6 +28,8 @@ static Type getArrayElemType(Attribute attr) {
   }
 
   if (auto arrayAttr = dyn_cast<ArrayAttr>(attr)) {
+    if (arrayAttr.empty())
+      return nullptr;
     return ArrayType::get(getArrayElemType(arrayAttr[0]), arrayAttr.size());
   }
 
@@ -45,7 +47,7 @@ getSplatAttrAndNumElements(Attribute valueAttr, Type valueType) {
   }
 
   if (auto arrayAttr = dyn_cast<ArrayAttr>(valueAttr)) {
-    if (llvm::all_equal(arrayAttr)) {
+    if (!arrayAttr.empty() && llvm::all_equal(arrayAttr)) {
       Attribute attr = arrayAttr[0];
       uint32_t numElements = arrayAttr.size();
 
@@ -94,9 +96,9 @@ struct SpecConstantCompositeOpConversion final
       return rewriter.notifyMatchFailure(op, "not a composite constant");
 
     ArrayAttr constituents = op.getConstituents();
-    if (constituents.size() == 1)
-      return rewriter.notifyMatchFailure(op,
-                                         "composite has only one consituent");
+    if (constituents.size() <= 1)
+      return rewriter.notifyMatchFailure(
+          op, "composite has zero or one consituent");
 
     if (!llvm::all_equal(constituents))
       return rewriter.notifyMatchFailure(op, "composite is not splat");

diff  --git a/mlir/test/Dialect/SPIRV/Transforms/replicated-const-composites.mlir b/mlir/test/Dialect/SPIRV/Transforms/replicated-const-composites.mlir
index 0a413e5036be9..f0c0248bf694d 100644
--- a/mlir/test/Dialect/SPIRV/Transforms/replicated-const-composites.mlir
+++ b/mlir/test/Dialect/SPIRV/Transforms/replicated-const-composites.mlir
@@ -186,6 +186,12 @@ spirv.module Logical GLSL450 requires #spirv.vce<v1.0, [Shader, ReplicatedCompos
     %0 = spirv.Constant [[dense<[1, 2]> : vector<2xi32>]] : !spirv.array<1 x !spirv.array<1 x vector<2xi32>>>
     spirv.ReturnValue %0 : !spirv.array<1 x !spirv.array<1 x vector<2xi32>>>
   }
+
+  spirv.func @empty_struct() -> (!spirv.struct<()>) "None" {
+    // CHECK-NOT: spirv.EXT.ConstantCompositeReplicate
+    %0 = spirv.Constant [] : !spirv.struct<()>
+    spirv.ReturnValue %0 : !spirv.struct<()>
+  }
 }
 
 // -----


        

From llvmlistbot at llvm.org  Wed Jul  1 02:24:44 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 02:24:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix crash on empty composite
 in replicated constants pass (PR #206647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206647@github.com>
Message-ID: <6a44dcdc.fdc477d6.2b77d8.8fb1@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/206647

From llvmlistbot at llvm.org  Wed Jul  1 02:35:13 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 02:35:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add spirv.GL.Step and
 spirv.GL.SmoothStep ops (PR #206037)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206037@github.com>
Message-ID: <6a44df51.fdc477d6.2b77d8.42ee@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206037

>From 6b51508dd36cfbcea9d852257f70ec5d5c189004 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 26 Jun 2026 13:32:08 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add spirv.GL.Step and spirv.GL.SmoothStep
 ops

---
 .../mlir/Dialect/SPIRV/IR/SPIRVGLOps.td       | 52 +++++++++++++++++++
 mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp        | 12 +++++
 mlir/test/Dialect/SPIRV/IR/gl-ops.mlir        | 44 ++++++++++++++++
 3 files changed, 108 insertions(+)

diff --git a/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td b/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
index e0b85e73cad99..34b49567deafe 100644
--- a/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
+++ b/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
@@ -1488,6 +1488,58 @@ def SPIRV_GLReflectOp : SPIRV_GLBinaryArithmeticOp<"Reflect", 71, SPIRV_Float, [
 
 // ----
 
+def SPIRV_GLStepOp : SPIRV_GLBinaryArithmeticOp<"Step", 48, SPIRV_Float, [AlwaysSpeculatable]> {
+  let summary = "Return 0.0 if x < edge, else 1.0";
+
+  let description = [{
+    Result is 0.0 if x < edge; otherwise result is 1.0.
+
+    The operands must all be a scalar or vector whose component type is
+    floating-point.
+
+    Result Type and the type of all operands must be the same type. Results
+    are computed per component.
+
+    #### Example:
+
+    ```mlir
+    %2 = spirv.GL.Step %edge, %x : f32
+    %3 = spirv.GL.Step %edge, %x : vector<3xf32>
+    ```
+  }];
+}
+
+// ----
+
+def SPIRV_GLSmoothStepOp : SPIRV_GLTernaryArithmeticOp<"SmoothStep", 49, SPIRV_Float> {
+  let summary = "Return smooth Hermite interpolation between 0 and 1";
+
+  let description = [{
+    Result is 0.0 if x <= edge0 and 1.0 if x >= edge1 and performs smooth
+    Hermite interpolation between 0 and 1 when edge0 < x < edge1. This is
+    equivalent to:
+      t = clamp((x - edge0) / (edge1 - edge0), 0, 1)
+      t * t * (3 - 2 * t)
+
+    The resulting value is undefined if edge0 >= edge1.
+
+    The operands must all be a scalar or vector whose component type is
+    floating-point.
+
+    Result Type and the type of all operands must be the same type. Results
+    are computed per component.
+
+    #### Example:
+
+    ```mlir
+    %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : f32
+    %3 = spirv.GL.SmoothStep %edge0, %edge1, %x : vector<3xf32>
+    ```
+  }];
+}
+
+// ----
+
 def SPIRV_GLFindILsbOp : SPIRV_GLUnaryArithmeticOp<"FindILsb", 73, SPIRV_Integer, [AlwaysSpeculatable]> {
   let summary = "Integer least-significant bit";
 
diff --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index f699055b5502f..c0393888af5c3 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -1294,6 +1294,18 @@ ParseResult spirv::GLNClampOp::parse(OpAsmParser &parser,
 }
 void spirv::GLNClampOp::print(OpAsmPrinter &p) { printOneResultOp(*this, p); }
 
+//===----------------------------------------------------------------------===//
+// spirv.GLSmoothStepOp
+//===----------------------------------------------------------------------===//
+
+ParseResult spirv::GLSmoothStepOp::parse(OpAsmParser &parser,
+                                         OperationState &result) {
+  return parseOneResultSameOperandTypeOp(parser, result);
+}
+void spirv::GLSmoothStepOp::print(OpAsmPrinter &p) {
+  printOneResultOp(*this, p);
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GLFmaOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir b/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
index 5665ae158d2e4..627c764feb6e6 100644
--- a/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
@@ -839,6 +839,50 @@ func.func @reflect_invalid_type(%arg0 : i32, %arg1 : i32) {
 
 // -----
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.Step
+//===----------------------------------------------------------------------===//
+
+func.func @step_scalar(%edge : f32, %x : f32) {
+  %2 = spirv.GL.Step %edge, %x : f32
+  // CHECK: %{{.+}} = spirv.GL.Step %{{.+}}, %{{.+}} : f32
+  return
+}
+
+func.func @step_vector(%edge : vector<3xf32>, %x : vector<3xf32>) {
+  %2 = spirv.GL.Step %edge, %x : vector<3xf32>
+  // CHECK: %{{.+}} = spirv.GL.Step %{{.+}}, %{{.+}} : vector<3xf32>
+  return
+}
+
+// -----
+
+func.func @step_invalid_type(%edge : i32, %x : i32) {
+  // expected-error @+1 {{'spirv.GL.Step' op operand #0 must be 16/32/64-bit float or fixed-length vector of 16/32/64-bit float values}}
+  %0 = spirv.GL.Step %edge, %x : i32
+  return
+}
+
+// -----
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SmoothStep
+//===----------------------------------------------------------------------===//
+
+func.func @smoothstep_scalar(%edge0 : f32, %edge1 : f32, %x : f32) {
+  // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : f32
+  %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : f32
+  return
+}
+
+func.func @smoothstep_vector(%edge0 : vector<3xf32>, %edge1 : vector<3xf32>, %x : vector<3xf32>) {
+  // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : vector<3xf32>
+  %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : vector<3xf32>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Fract
 //===----------------------------------------------------------------------===//

>From 54efafcd83920a7964186ae4f3c68470fd0905f6 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:34:34 +0200
Subject: [PATCH 2/2] Add target tests

---
 mlir/test/Target/SPIRV/gl-ops.mlir | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/mlir/test/Target/SPIRV/gl-ops.mlir b/mlir/test/Target/SPIRV/gl-ops.mlir
index bc17ff82cc441..41364dca31af3 100644
--- a/mlir/test/Target/SPIRV/gl-ops.mlir
+++ b/mlir/test/Target/SPIRV/gl-ops.mlir
@@ -174,4 +174,16 @@ spirv.module Logical GLSL450 requires #spirv.vce<v1.0, [Shader, Linkage], []> {
     %1 = spirv.GL.PackSnorm4x8 %0 : vector<4xf32> -> i32
     spirv.Return
   }
+
+  spirv.func @step(%arg0 : f32, %arg1 : f32) "None" {
+    // CHECK: spirv.GL.Step {{%[^,]*}}, {{%[^,]*}} : f32
+    %0 = spirv.GL.Step %arg0, %arg1 : f32
+    spirv.Return
+  }
+
+  spirv.func @smoothstep(%arg0 : f32, %arg1 : f32, %arg2 : f32) "None" {
+    // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : f32
+    %0 = spirv.GL.SmoothStep %arg0, %arg1, %arg2 : f32
+    spirv.Return
+  }
 }


From llvmlistbot at llvm.org  Wed Jul  1 02:35:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 02:35:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ABI] Fix scalable vector mapping in
	ABITypeMapper (PR #206617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206617@github.com>
Message-ID: <6a44df57.510f576d.1df92b.3c2c@mx.google.com>


banach-space wrote:

@felichita , am I correct that you don't have commit access?

https://github.com/llvm/llvm-project/pull/206617

From llvmlistbot at llvm.org  Wed Jul  1 02:40:44 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 02:40:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e09c.14d9b9dc.7e5e3.e660@mx.google.com>


https://github.com/pechy updated https://github.com/llvm/llvm-project/pull/206310

>From 0032210bc80986c4fd3a61e2f193a1be24ef55c0 Mon Sep 17 00:00:00 2001
From: Martin Petracek <martin at petracek.net>
Date: Mon, 29 Jun 2026 23:27:05 +0700
Subject: [PATCH] [MLIR] Verify that all blocks are reachable before CFG to SCF
 transform

CFG to SCF transform cannot handle unreachable code. Properly verify
that all blocks are reachable and document this limitation in the pass
description.

Fixes #206086
---
 mlir/include/mlir/Transforms/CFGToSCF.h       |  7 +++---
 mlir/lib/Transforms/Utils/CFGToSCF.cpp        | 22 ++++++++++++++++---
 .../ControlFlowToSCF/unreachable-blocks.mlir  | 18 +++++++++++++++
 3 files changed, 41 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir

diff --git a/mlir/include/mlir/Transforms/CFGToSCF.h b/mlir/include/mlir/Transforms/CFGToSCF.h
index 517e116f3c58e..2ca019ddfe23d 100644
--- a/mlir/include/mlir/Transforms/CFGToSCF.h
+++ b/mlir/include/mlir/Transforms/CFGToSCF.h
@@ -149,9 +149,10 @@ class CFGToSCFInterface {
 /// Otherwise a single control flow graph operation branching to one block
 /// per return-like operation kind remains.
 ///
-/// The transformation currently requires that all control flow graph operations
-/// have no side effects, implement the BranchOpInterface and does not have any
-/// operation produced successor operands.
+/// The transformation currently requires that the region has no unreachable
+/// blocks and that all control flow graph operations have no side effects,
+/// implement the BranchOpInterface and does not have any operation produced
+/// successor operands.
 /// Returns failure if any of the preconditions are violated or if any of the
 /// methods of `interface` failed. The IR is left in an unspecified state.
 ///
diff --git a/mlir/lib/Transforms/Utils/CFGToSCF.cpp b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
index 0cc6bb43ddd38..28362abf3300d 100644
--- a/mlir/lib/Transforms/Utils/CFGToSCF.cpp
+++ b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
@@ -950,8 +950,10 @@ static FailureOr<SmallVector<Block *>> transformToStructuredCFBranches(
     return SmallVector<Block *>{};
 
   if (regionEntry->getNumSuccessors() == 1) {
-    // Single successor we can just splice together.
+    // Single successor we can splice together, if they are not self-loops and
+    // the successor's only predecessor is `regionEntry`.
     Block *successor = regionEntry->getSuccessor(0);
+
     for (auto &&[oldValue, newValue] : llvm::zip(
              successor->getArguments(), getSuccessorOperands(regionEntry, 0)))
       oldValue.replaceAllUsesWith(newValue);
@@ -1234,10 +1236,24 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
-  for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+  auto reachableBlocks = [&region] {
+    SmallVector<Block *, 16> workList = {&region.front()};
+    SmallPtrSet<Block *, 16> visited = {&region.front()};
+    while (!workList.empty()) {
+      Block *current = workList.pop_back_val();
+      for (Block *succ : current->getSuccessors()) {
+        if (visited.insert(succ).second) {
+          workList.push_back(succ);
+        }
+      }
+    }
+    return visited;
+  }();
+  for (Block &block : region.getBlocks()) {
+    if (!reachableBlocks.contains(&block))
       return block.front().emitOpError(
           "transformation does not support unreachable blocks");
+  }
 
   WalkResult result = region.walk([](Operation *operation) {
     if (operation->getNumSuccessors() == 0)
diff --git a/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
new file mode 100644
index 0000000000000..4317e5a5a8102
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt --lift-cf-to-scf -verify-diagnostics -split-input-file %s
+
+// Verify that --lift-cf-to-scf does not crash when it encounters an
+// unreachable blocks (dead code).  Instead it should emit a clean error.
+// See: https://github.com/llvm/llvm-project/issues/206086
+
+module {
+  func.func @single_return_statement(%arg0: i32) -> i32 {
+    return %arg0 : i32
+  ^bb1:  // pred: ^bb1
+    // expected-error at below {{'arith.constant' op transformation does not support unreachable blocks}}
+    %true = arith.constant true
+    cf.cond_br %true, ^bb2, ^bb1
+  ^bb2:  // pred: ^bb1
+    return %arg0 : i32
+  }
+}
+


From llvmlistbot at llvm.org  Wed Jul  1 02:43:23 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 02:43:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e13b.97bb2efc.2241ed.2b89@mx.google.com>


https://github.com/pechy updated https://github.com/llvm/llvm-project/pull/206310

>From 5922b05e230282299355360cec8e704c244ef79d Mon Sep 17 00:00:00 2001
From: Martin Petracek <martin at petracek.net>
Date: Mon, 29 Jun 2026 23:27:05 +0700
Subject: [PATCH] [MLIR] Verify that all blocks are reachable before CFG to SCF
 transform

CFG to SCF transform cannot handle unreachable code. Properly verify
that all blocks are reachable and document this limitation in the pass
description.

Fixes #206086
---
 mlir/include/mlir/Transforms/CFGToSCF.h       |  7 ++++---
 mlir/lib/Transforms/Utils/CFGToSCF.cpp        | 19 +++++++++++++++++--
 .../ControlFlowToSCF/unreachable-blocks.mlir  | 18 ++++++++++++++++++
 3 files changed, 39 insertions(+), 5 deletions(-)
 create mode 100644 mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir

diff --git a/mlir/include/mlir/Transforms/CFGToSCF.h b/mlir/include/mlir/Transforms/CFGToSCF.h
index 517e116f3c58e..2ca019ddfe23d 100644
--- a/mlir/include/mlir/Transforms/CFGToSCF.h
+++ b/mlir/include/mlir/Transforms/CFGToSCF.h
@@ -149,9 +149,10 @@ class CFGToSCFInterface {
 /// Otherwise a single control flow graph operation branching to one block
 /// per return-like operation kind remains.
 ///
-/// The transformation currently requires that all control flow graph operations
-/// have no side effects, implement the BranchOpInterface and does not have any
-/// operation produced successor operands.
+/// The transformation currently requires that the region has no unreachable
+/// blocks and that all control flow graph operations have no side effects,
+/// implement the BranchOpInterface and does not have any operation produced
+/// successor operands.
 /// Returns failure if any of the preconditions are violated or if any of the
 /// methods of `interface` failed. The IR is left in an unspecified state.
 ///
diff --git a/mlir/lib/Transforms/Utils/CFGToSCF.cpp b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
index 0cc6bb43ddd38..e2ef9d5076d0d 100644
--- a/mlir/lib/Transforms/Utils/CFGToSCF.cpp
+++ b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
@@ -952,6 +952,7 @@ static FailureOr<SmallVector<Block *>> transformToStructuredCFBranches(
   if (regionEntry->getNumSuccessors() == 1) {
     // Single successor we can just splice together.
     Block *successor = regionEntry->getSuccessor(0);
+
     for (auto &&[oldValue, newValue] : llvm::zip(
              successor->getArguments(), getSuccessorOperands(regionEntry, 0)))
       oldValue.replaceAllUsesWith(newValue);
@@ -1234,10 +1235,24 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
-  for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+  const auto reachableBlocks = [&region] {
+    SmallVector<Block *, 16> workList = {&region.front()};
+    SmallPtrSet<Block *, 16> visited = {&region.front()};
+    while (!workList.empty()) {
+      Block *current = workList.pop_back_val();
+      for (Block *succ : current->getSuccessors()) {
+        if (visited.insert(succ).second) {
+          workList.push_back(succ);
+        }
+      }
+    }
+    return visited;
+  }();
+  for (Block &block : region.getBlocks()) {
+    if (!reachableBlocks.contains(&block))
       return block.front().emitOpError(
           "transformation does not support unreachable blocks");
+  }
 
   WalkResult result = region.walk([](Operation *operation) {
     if (operation->getNumSuccessors() == 0)
diff --git a/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
new file mode 100644
index 0000000000000..4317e5a5a8102
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt --lift-cf-to-scf -verify-diagnostics -split-input-file %s
+
+// Verify that --lift-cf-to-scf does not crash when it encounters an
+// unreachable blocks (dead code).  Instead it should emit a clean error.
+// See: https://github.com/llvm/llvm-project/issues/206086
+
+module {
+  func.func @single_return_statement(%arg0: i32) -> i32 {
+    return %arg0 : i32
+  ^bb1:  // pred: ^bb1
+    // expected-error at below {{'arith.constant' op transformation does not support unreachable blocks}}
+    %true = arith.constant true
+    cf.cond_br %true, ^bb2, ^bb1
+  ^bb2:  // pred: ^bb1
+    return %arg0 : i32
+  }
+}
+


From llvmlistbot at llvm.org  Wed Jul  1 02:43:26 2026
From: llvmlistbot at llvm.org (Fedor Nikolaev)
Date: Wed, 01 Jul 2026 02:43:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ABI] Fix scalable vector mapping in
	ABITypeMapper (PR #206617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206617@github.com>
Message-ID: <6a44e13e.61d696b2.2be3bd.7068@mx.google.com>


felichita wrote:

> @felichita , am I correct that you don't have commit access?

Yeap, you're right. I'm a newcomer

https://github.com/llvm/llvm-project/pull/206617

From llvmlistbot at llvm.org  Wed Jul  1 02:44:03 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 02:44:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e163.d5797644.130832.07a4@mx.google.com>


https://github.com/pechy updated https://github.com/llvm/llvm-project/pull/206310

>From 4cbe8205cdc43ed1a1f801a2dd37e4cb45fa9c54 Mon Sep 17 00:00:00 2001
From: Martin Petracek <martin at petracek.net>
Date: Mon, 29 Jun 2026 23:27:05 +0700
Subject: [PATCH] [MLIR] Verify that all blocks are reachable before CFG to SCF
 transform

CFG to SCF transform cannot handle unreachable code. Properly verify
that all blocks are reachable and document this limitation in the pass
description.

Fixes #206086
---
 mlir/include/mlir/Transforms/CFGToSCF.h        |  7 ++++---
 mlir/lib/Transforms/Utils/CFGToSCF.cpp         | 18 ++++++++++++++++--
 .../ControlFlowToSCF/unreachable-blocks.mlir   | 18 ++++++++++++++++++
 3 files changed, 38 insertions(+), 5 deletions(-)
 create mode 100644 mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir

diff --git a/mlir/include/mlir/Transforms/CFGToSCF.h b/mlir/include/mlir/Transforms/CFGToSCF.h
index 517e116f3c58e..2ca019ddfe23d 100644
--- a/mlir/include/mlir/Transforms/CFGToSCF.h
+++ b/mlir/include/mlir/Transforms/CFGToSCF.h
@@ -149,9 +149,10 @@ class CFGToSCFInterface {
 /// Otherwise a single control flow graph operation branching to one block
 /// per return-like operation kind remains.
 ///
-/// The transformation currently requires that all control flow graph operations
-/// have no side effects, implement the BranchOpInterface and does not have any
-/// operation produced successor operands.
+/// The transformation currently requires that the region has no unreachable
+/// blocks and that all control flow graph operations have no side effects,
+/// implement the BranchOpInterface and does not have any operation produced
+/// successor operands.
 /// Returns failure if any of the preconditions are violated or if any of the
 /// methods of `interface` failed. The IR is left in an unspecified state.
 ///
diff --git a/mlir/lib/Transforms/Utils/CFGToSCF.cpp b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
index 0cc6bb43ddd38..e5c6e202c5b80 100644
--- a/mlir/lib/Transforms/Utils/CFGToSCF.cpp
+++ b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
@@ -1234,10 +1234,24 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
-  for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+  const auto reachableBlocks = [&region] {
+    SmallVector<Block *, 16> workList = {&region.front()};
+    SmallPtrSet<Block *, 16> visited = {&region.front()};
+    while (!workList.empty()) {
+      Block *current = workList.pop_back_val();
+      for (Block *succ : current->getSuccessors()) {
+        if (visited.insert(succ).second) {
+          workList.push_back(succ);
+        }
+      }
+    }
+    return visited;
+  }();
+  for (Block &block : region.getBlocks()) {
+    if (!reachableBlocks.contains(&block))
       return block.front().emitOpError(
           "transformation does not support unreachable blocks");
+  }
 
   WalkResult result = region.walk([](Operation *operation) {
     if (operation->getNumSuccessors() == 0)
diff --git a/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
new file mode 100644
index 0000000000000..4317e5a5a8102
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt --lift-cf-to-scf -verify-diagnostics -split-input-file %s
+
+// Verify that --lift-cf-to-scf does not crash when it encounters an
+// unreachable blocks (dead code).  Instead it should emit a clean error.
+// See: https://github.com/llvm/llvm-project/issues/206086
+
+module {
+  func.func @single_return_statement(%arg0: i32) -> i32 {
+    return %arg0 : i32
+  ^bb1:  // pred: ^bb1
+    // expected-error at below {{'arith.constant' op transformation does not support unreachable blocks}}
+    %true = arith.constant true
+    cf.cond_br %true, ^bb2, ^bb1
+  ^bb2:  // pred: ^bb1
+    return %arg0 : i32
+  }
+}
+


From llvmlistbot at llvm.org  Wed Jul  1 02:47:18 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 02:47:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e226.30759430.1a950e.d6d0@mx.google.com>


pechy wrote:

Ok, makes sense. I didn't see that check before.

I changed the PR as you suggested, properly checking the reachability.

https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 03:05:24 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 01 Jul 2026 03:05:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add spirv.GL.Step and
 spirv.GL.SmoothStep ops (PR #206037)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206037@github.com>
Message-ID: <6a44e664.b47a2ac0.265e67.53e7@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/206037

From llvmlistbot at llvm.org  Wed Jul  1 03:07:04 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Wed, 01 Jul 2026 03:07:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e6c8.63d39acb.be069.7483@mx.google.com>


https://github.com/zero9178 approved this pull request.

LGTM % one comment

https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 03:07:04 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Wed, 01 Jul 2026 03:07:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e6c8.dd7c1686.32daab.38c4@mx.google.com>


https://github.com/zero9178 edited https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 03:07:04 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Wed, 01 Jul 2026 03:07:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44e6c8.cb385abd.24f5ee.c61e@mx.google.com>


================
@@ -1234,10 +1234,24 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
-  for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+  const auto reachableBlocks = [&region] {
+    SmallVector<Block *, 16> workList = {&region.front()};
+    SmallPtrSet<Block *, 16> visited = {&region.front()};
+    while (!workList.empty()) {
+      Block *current = workList.pop_back_val();
+      for (Block *succ : current->getSuccessors()) {
+        if (visited.insert(succ).second) {
+          workList.push_back(succ);
+        }
+      }
+    }
+    return visited;
+  }();
----------------
zero9178 wrote:

I think you ought to be able to use `llvm::depth_first_ext` here to generate the reachability set

https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 03:07:11 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 03:07:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for FMod and SMod (PR #206933)
Message-ID: <llvm.org/llvm/llvm-project/pull/206933@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206933

None

>From 2244a3cbcb0294fc1e9adb865ef974b6ff802a26 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:03:16 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for FMod and SMod

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 80 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 54 +++++++++++++
 2 files changed, 133 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..5be4cc6bc4ebd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,83 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.FMod` to `x - y * floor(x / y)`. The SPIR-V op requires the
+/// result to take the sign of the divisor, whereas `llvm.frem` keeps the sign
+/// of the dividend, so `frem` cannot be used directly.
+class FModPattern : public SPIRVToLLVMConversion<spirv::FModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::FModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::FModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Value div = LLVM::FDivOp::create(rewriter, loc, dstType, lhs, rhs);
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, div);
+    Value scaled = LLVM::FMulOp::create(rewriter, loc, dstType, rhs, floored);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, lhs, scaled);
+    return success();
+  }
+};
+
+/// Converts `spirv.SMod` to a signed remainder corrected to take the sign of
+/// the divisor. `llvm.srem` keeps the sign of the dividend, so the result is
+/// adjusted by adding the divisor when the remainder is non-zero and its sign
+/// differs from the divisor's.
+class SModPattern : public SPIRVToLLVMConversion<spirv::SModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Type i1Type = rewriter.getI1Type();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value rem = LLVM::SRemOp::create(rewriter, loc, dstType, lhs, rhs);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+
+    Value remNonZero = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                            LLVM::ICmpPredicate::ne, rem, zero);
+    Value remNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rem, zero);
+    Value rhsNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rhs, zero);
+    Value signMismatch =
+        LLVM::XOrOp::create(rewriter, loc, cmpType, remNeg, rhsNeg);
+    Value needsAdjust =
+        LLVM::AndOp::create(rewriter, loc, cmpType, remNonZero, signMismatch);
+
+    Value adjusted = LLVM::AddOp::create(rewriter, loc, dstType, rem, rhs);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, needsAdjust,
+                                                adjusted, rem);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1901,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, FModPattern,
+      SModPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..36d4c5facb057 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,57 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.FMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmod_scalar
+spirv.func @fmod_scalar(%arg0: f32, %arg1: f32) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : f32
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (f32) -> f32
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : f32
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : f32
+  %0 = spirv.FMod %arg0, %arg1 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmod_vector
+spirv.func @fmod_vector(%arg0: vector<4xf32>, %arg1: vector<4xf32>) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (vector<4xf32>) -> vector<4xf32>
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : vector<4xf32>
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : vector<4xf32>
+  %0 = spirv.FMod %arg0, %arg1 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.SMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @smod_scalar
+spirv.func @smod_scalar(%arg0: i32, %arg1: i32) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : i32
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[RNEG:.*]] = llvm.icmp "slt" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[DNEG:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[XOR:.*]] = llvm.xor %[[RNEG]], %[[DNEG]] : i1
+  // CHECK: %[[ADJ:.*]] = llvm.and %[[NZ]], %[[XOR]] : i1
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : i32
+  // CHECK: llvm.select %[[ADJ]], %[[ADD]], %[[REM]] : i1, i32
+  %0 = spirv.SMod %arg0, %arg1 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @smod_vector
+spirv.func @smod_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : vector<4xi32>
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : vector<4xi32>
+  // CHECK: llvm.select %{{.*}}, %[[ADD]], %[[REM]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.SMod %arg0, %arg1 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 03:07:12 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 03:07:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
Message-ID: <llvm.org/llvm/llvm-project/pull/206934@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206934

None

>From 2e1952b741d34c2b0dfc1887493498283a21a93c Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:06:39 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FSign and
 GL.SSign

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 64 +++++++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 54 ++++++++++++++++
 2 files changed, 118 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 03:07:13 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 03:07:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
Message-ID: <llvm.org/llvm/llvm-project/pull/206935@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206935

None

>From 4fa5c4ca4a5a4333743f1fc3afa38979d20a176a Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:54:35 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix and
 CL.mix

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 27 +++++++++++++++++++
 .../SPIRVToLLVM/cl-ops-to-llvm.mlir           | 20 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 20 ++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 03:07:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 03:07:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a44e6f3.ca5545e3.1a87f7.8f60@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-spirv

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206935.diff


3 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+27) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir (+20) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir (+20) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Wed Jul  1 03:07:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 03:07:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206934@github.com>
Message-ID: <6a44e6f7.720fb68d.3971aa.6997@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-spirv

@llvm/pr-subscribers-mlir

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206934.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+64) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir (+54) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206934

From llvmlistbot at llvm.org  Wed Jul  1 03:07:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 03:07:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for FMod and SMod (PR #206933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206933@github.com>
Message-ID: <6a44e6f9.1551f180.318ac5.8fb5@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206933.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+79-1) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir (+54) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..5be4cc6bc4ebd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,83 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.FMod` to `x - y * floor(x / y)`. The SPIR-V op requires the
+/// result to take the sign of the divisor, whereas `llvm.frem` keeps the sign
+/// of the dividend, so `frem` cannot be used directly.
+class FModPattern : public SPIRVToLLVMConversion<spirv::FModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::FModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::FModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Value div = LLVM::FDivOp::create(rewriter, loc, dstType, lhs, rhs);
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, div);
+    Value scaled = LLVM::FMulOp::create(rewriter, loc, dstType, rhs, floored);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, lhs, scaled);
+    return success();
+  }
+};
+
+/// Converts `spirv.SMod` to a signed remainder corrected to take the sign of
+/// the divisor. `llvm.srem` keeps the sign of the dividend, so the result is
+/// adjusted by adding the divisor when the remainder is non-zero and its sign
+/// differs from the divisor's.
+class SModPattern : public SPIRVToLLVMConversion<spirv::SModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Type i1Type = rewriter.getI1Type();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value rem = LLVM::SRemOp::create(rewriter, loc, dstType, lhs, rhs);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+
+    Value remNonZero = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                            LLVM::ICmpPredicate::ne, rem, zero);
+    Value remNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rem, zero);
+    Value rhsNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rhs, zero);
+    Value signMismatch =
+        LLVM::XOrOp::create(rewriter, loc, cmpType, remNeg, rhsNeg);
+    Value needsAdjust =
+        LLVM::AndOp::create(rewriter, loc, cmpType, remNonZero, signMismatch);
+
+    Value adjusted = LLVM::AddOp::create(rewriter, loc, dstType, rem, rhs);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, needsAdjust,
+                                                adjusted, rem);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1901,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, FModPattern,
+      SModPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..36d4c5facb057 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,57 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.FMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmod_scalar
+spirv.func @fmod_scalar(%arg0: f32, %arg1: f32) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : f32
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (f32) -> f32
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : f32
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : f32
+  %0 = spirv.FMod %arg0, %arg1 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmod_vector
+spirv.func @fmod_vector(%arg0: vector<4xf32>, %arg1: vector<4xf32>) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (vector<4xf32>) -> vector<4xf32>
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : vector<4xf32>
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : vector<4xf32>
+  %0 = spirv.FMod %arg0, %arg1 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.SMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @smod_scalar
+spirv.func @smod_scalar(%arg0: i32, %arg1: i32) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : i32
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[RNEG:.*]] = llvm.icmp "slt" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[DNEG:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[XOR:.*]] = llvm.xor %[[RNEG]], %[[DNEG]] : i1
+  // CHECK: %[[ADJ:.*]] = llvm.and %[[NZ]], %[[XOR]] : i1
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : i32
+  // CHECK: llvm.select %[[ADJ]], %[[ADD]], %[[REM]] : i1, i32
+  %0 = spirv.SMod %arg0, %arg1 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @smod_vector
+spirv.func @smod_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : vector<4xi32>
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : vector<4xi32>
+  // CHECK: llvm.select %{{.*}}, %[[ADD]], %[[REM]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.SMod %arg0, %arg1 : vector<4xi32>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206933

From llvmlistbot at llvm.org  Wed Jul  1 03:08:15 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 03:08:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for non-trivial elementwise operations (PR #202574)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202574@github.com>
Message-ID: <6a44e70f.d3ef658f.28a2a5.85f5@mx.google.com>


aobolensk wrote:

> Sorry, I've been quite busy recently. If you could split it that would make reviewing easier. I'll try to catch up on reviews in the next few days.

- FMod, SMod: https://github.com/llvm/llvm-project/pull/206933
- FSign, SSign: https://github.com/llvm/llvm-project/pull/206934
- FMix: https://github.com/llvm/llvm-project/pull/206935

https://github.com/llvm/llvm-project/pull/202574

From llvmlistbot at llvm.org  Wed Jul  1 03:24:03 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 03:24:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for non-trivial elementwise operations (PR #202574)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202574@github.com>
Message-ID: <6a44eac3.af6ba3ed.127cbd.6c22@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/202574

From llvmlistbot at llvm.org  Wed Jul  1 03:24:57 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 01 Jul 2026 03:24:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LowerToLLVM] Pass DataLayoutAnalysis
 to LowerToLLVMOptions in `ArithToLLVM`, `ControlFlowToLLVM`,
 and `VectorToLLVM` passes (PR #206380)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206380@github.com>
Message-ID: <6a44eaf9.dd2b4bd3.10c1cf.0d56@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/206380

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.481afadf.29a24b.dfb4@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
----------------
banach-space wrote:

In this example, `scf.for` and `linalg.unpack` tiles should be different, right? It would be good to hightlight that via `CHECK` lines.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.481afadf.29a24b.dfc3@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_unaligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // No hint is passed: the static tile sizes (7, 5) are not aligned to the scalable
+      // inner tiles (8 * vscale, 4 * vscale), so the tiled unpack keeps a remainder slice.
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [7, 5] : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
----------------
banach-space wrote:

Why not re-use `@NCnc_to_NC_scalable_aligned`. It would make more sense to me to use that test - simply remove the hints. Also, it's not clear from the `CHECK` lines where the remainder is computed.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.eb165e5c.5b8dc.34d4@mx.google.com>


================

----------------
banach-space wrote:

Can you remind me why you don't test pure tiling in the case of `linalg.pack`?

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.4a342cbc.22496e.9ca6@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
----------------
banach-space wrote:

Please add `step <>` to all tests in this file :)

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.9b510543.2bbb51.065c@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_unaligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // No hint is passed: the static tile sizes (7, 5) are not aligned to the scalable
+      // inner tiles (8 * vscale, 4 * vscale), so the tiled unpack keeps a remainder slice.
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [7, 5] : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Producer fusion - linalg.unpack with scalable inner tiles fused as a producer
+// into an elementwise consumer that is tiled with scalable tile
+// sizes that are an integer multiple of the inner tiles (16*vscale of 8*vscale,
+// 8*vscale of 4*vscale). `Multiple` hint is passed accordingly.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @unpack_elemwise_scalable_multiple
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+// CHECK:             %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+// CHECK:             %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             linalg.exp ins(%[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @unpack_elemwise_scalable_multiple(%arg0: tensor<4x8x?x?xf32>, %arg1: tensor<?x?xf32>, %arg2 : index, %arg3 : index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c16 = arith.constant 16 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %t0 = arith.muli %c16, %vscale : index
+  %t1 = arith.muli %c8, %vscale : index
+  %0 = tensor.empty(%arg2, %arg3) : tensor<?x?xf32>
+  %1 = linalg.unpack %arg0 inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %0
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  %2 = linalg.exp ins(%1: tensor<?x?xf32>)
+                       outs(%arg1: tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %2 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %exp = transform.structured.match ops{["linalg.exp"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %mulis = transform.structured.match ops{["arith.muli"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %i0, %i1, %t0h, %t1h = transform.split_handle %mulis : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
----------------
banach-space wrote:

Is this required here? Normally you'd pass tile sizes as e.g. `[[16], [4]]`, no?

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.c56fb102.1c15b6.682f@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
----------------
banach-space wrote:

Is this a negative test? How does the output differ from what's produced for `@NCnc_to_NC_scalable_aligned`? That's currently not clear.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:29:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 03:29:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ec0e.2020f6c2.2129d2.2c02@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
----------------
banach-space wrote:

[nit] `OUTER0` -> `OUTER_DIM0`

Similar comment for `OUTER1` and other tests.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:31:07 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 01 Jul 2026 03:31:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] add consistent stride
 verification to masked load/store and gather/scatter ops (PR #205869)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205869@github.com>
Message-ID: <6a44ec6b.d4361ba8.1154fd.903e@mx.google.com>


FedericoBruzzone wrote:

Kind ping @banach-space @dcaballe 🫶

You had already reviewed it in #204842. I made a minor fix in the last commit.


https://github.com/llvm/llvm-project/pull/205869

From llvmlistbot at llvm.org  Wed Jul  1 03:36:01 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Wed, 01 Jul 2026 03:36:01 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a44ed91.27b3b003.9dfd8.8c94@mx.google.com>


================

----------------
egebeysel wrote:

I didn't implement it :D 

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 03:41:31 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 01 Jul 2026 03:41:31 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a44eedb.492e23e4.6e404.9ea4@mx.google.com>


FedericoBruzzone wrote:

Hi @LouisLu060211!
Thank you for your contribution!

Please, could you reply to https://github.com/llvm/llvm-project/pull/206683#issuecomment-4841671527?
You could take the opportunity to familiarize with LLVM policies in general :D


https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 03:54:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 03:54:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 633344d - [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (#205080)
Message-ID: <6a44f203.a151105c.15f7b1.3879@mx.google.com>


Author: Andrei Golubev
Date: 2026-07-01T12:54:54+02:00
New Revision: 633344de1ba723503ae3489933d1f318c0bb4473

URL: https://github.com/llvm/llvm-project/commit/633344de1ba723503ae3489933d1f318c0bb4473
DIFF: https://github.com/llvm/llvm-project/commit/633344de1ba723503ae3489933d1f318c0bb4473.diff

LOG: [mlir][bufferization] Introduce castBufferTypeFn hook and API (#205080)

This PR is the second part of the work that aims to allow customizations
in resolving mismatching buffer types.
(see 0b1fe74f079fd330e4d4d85ce33b7f5702da1be7 for part one)

Add a new bufferization hook that lets downstream bufferization
implementations materialize an operation to resolve a pre-reconciled
buffer type mismatch.

The hook + API is used as a fallback mechanism in several upstream
operations, as well as in the general bufferization infrastructure.
Semantically, this is a "cast" creation akin to `createAlloc()` and
`createMemCpy()` but for the cases where a memref::CastOp-like operation
is acceptable.

A valid value is returned whenever a cast-like operation is created,
failure signifies a failed attempt to materialize a cast (in theory,
there could be a fall back to alloc + copy either externally, but this
right now is not considered in most contexts). By default, a
`memref.cast` operation is created that casts a bufferized value to a
reconciled type.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
    mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
    mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
    mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
    mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
    mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
    mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
    mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
    mlir/test/lib/Dialect/Test/TestOpDefs.cpp
    mlir/test/lib/Dialect/Test/TestOps.td
    mlir/test/lib/Dialect/Test/TestTypeDefs.td

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f7a6a638f307..9f49a31c1583b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -258,6 +258,12 @@ struct BufferizationOptions {
   /// Memcpy function: Generate a memcpy between two buffers.
   using MemCpyFn =
       std::function<LogicalResult(OpBuilder &, Location, Value, Value)>;
+  /// Cast function: Convert a buffer value to a new value with the specified
+  /// type. This method is typically used when a simple cast-like operation is
+  /// sufficient to convert the buffer value, for example, when layout maps
+  /// between buffer value and resulting type do not match.
+  using CastFn =
+      std::function<FailureOr<Value>(OpBuilder &, Location, Type, Value)>;
   /// Initializer function for analysis state.
   using AnalysisStateInitFn = std::function<void(AnalysisState &)>;
   /// Tensor-like -> Buffer-like type conversion.
@@ -300,6 +306,7 @@ struct BufferizationOptions {
   /// Helper functions for allocation and memory copying.
   std::optional<AllocationFn> allocationFn;
   std::optional<MemCpyFn> memCpyFn;
+  std::optional<CastFn> castFn;
 
   /// Create a memref allocation with the given type and dynamic extents.
   FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
@@ -309,6 +316,10 @@ struct BufferizationOptions {
   LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
                              Value to) const;
 
+  /// Creates a cast function from a buffer value to a new type.
+  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
+                              Value value) const;
+
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.

diff  --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index 35e472470560a..f810000dd082f 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -172,12 +172,14 @@ struct SelectOpInterface
         return selectOp->emitError(
             "incompatible buffer types on true/false operands");
       }
-      if (trueBuffer.getType() != *targetType)
+      if (trueBuffer.getType() != *targetType) {
         trueBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, trueBuffer);
-      if (falseBuffer.getType() != *targetType)
+            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+      }
+      if (falseBuffer.getType() != *targetType) {
         falseBuffer =
-            memref::CastOp::create(rewriter, loc, *targetType, falseBuffer);
+            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+      }
     }
 
     replaceOpWithNewBufferizedOp<arith::SelectOp>(

diff  --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index e0a35e1932f3d..7f656164f3a15 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -840,6 +840,19 @@ LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
   return success();
 }
 
+FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
+                                                  Type dest,
+                                                  Value value) const {
+  if (castFn)
+    return (*castFn)(b, loc, dest, value);
+
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 2a1dcc89a83ed..320f426985750 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = memref::CastOp::create(b, value.getLoc(), destType, value);
+    Value casted = *options.createCast(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -100,6 +100,20 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
     return success();
   }
 
+  if (!llvm::isa<BaseMemRefType>(srcType) ||
+      !llvm::isa<BaseMemRefType>(destType)) {
+    // Non-builtin case: the best is to try the user-provided cast.
+    assert(options.castFn.has_value() &&
+           "user-provided cast is required for non-builtin types");
+    auto replacement =
+        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
+                           destType, bufferToTensor.getBuffer());
+    if (failed(replacement))
+      return failure();
+    rewriter.replaceOp(toBuffer, *replacement);
+    return success();
+  }
+
   auto rankedSrcType = llvm::dyn_cast<MemRefType>(srcType);
   auto rankedDestType = llvm::dyn_cast<MemRefType>(destType);
   auto unrankedSrcType = llvm::dyn_cast<UnrankedMemRefType>(srcType);

diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index 8bdee09d0d6e7..e0262abf313b8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -476,9 +476,10 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
           rewriter, operand.getLoc(), *operandBufferType, operand);
       // A cast is needed if the operand and the block argument have 
diff erent
       // bufferized types.
-      if (type != *operandBufferType)
-        bufferizedOperand = memref::CastOp::create(rewriter, operand.getLoc(),
-                                                   type, bufferizedOperand);
+      if (type != *operandBufferType) {
+        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
+                                                type, bufferizedOperand);
+      }
       newOperands.push_back(bufferizedOperand);
     }
     operands.getMutableForwardedOperands().assign(newOperands);

diff  --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index b9662751aa7f4..401be7440f5a2 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -31,21 +31,13 @@ namespace {
 
 /// Helper function for loop bufferization. Cast the given buffer to the given
 /// memref type.
-static Value castBuffer(OpBuilder &b, Value buffer, Type type) {
+static Value castBuffer(OpBuilder &b, Value buffer, Type type,
+                        const BufferizationOptions &options) {
   // If the buffer already has the correct type, no cast is needed.
   if (buffer.getType() == type)
     return buffer;
 
-  // TODO: Properly support with options, for now it is hardcoded MemRef type
-  // based approach
-  assert(isa<BaseMemRefType>(type) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(buffer.getType()) && "expected BaseMemRefType");
-  // TODO: In case `type` has a layout map that is not the fully dynamic
-  // one, we may not be able to cast the buffer. In that case, the loop
-  // iter_arg's layout map must be changed (see uses of `castBuffer`).
-  assert(memref::CastOp::areCastCompatible(buffer.getType(), type) &&
-         "scf.while op bufferization: cast incompatible");
-  return memref::CastOp::create(b, buffer.getLoc(), type, buffer).getResult();
+  return *options.createCast(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value
@@ -114,7 +106,7 @@ struct ConditionOpInterface
             whileOp.getAfterArguments()[it.index()], options, state);
         if (failed(resultType))
           return failure();
-        Value buffer = castBuffer(rewriter, *maybeBuffer, *resultType);
+        Value buffer = castBuffer(rewriter, *maybeBuffer, *resultType, options);
         newArgs.push_back(buffer);
       } else {
         newArgs.push_back(value);
@@ -206,7 +198,7 @@ struct ExecuteRegionOpInterface
     rewriter.setInsertionPointAfter(newOp);
     SmallVector<Value> newResults;
     for (const auto &it : llvm::enumerate(executeRegionOp->getResultTypes())) {
-      if (isa<TensorType>(it.value())) {
+      if (isa<TensorLikeType>(it.value())) {
         newResults.push_back(bufferization::ToTensorOp::create(
             rewriter, executeRegionOp.getLoc(), it.value(),
             newOp->getResult(it.index())));
@@ -361,7 +353,7 @@ struct IndexSwitchOpInterface
     // Compute bufferized result types.
     SmallVector<Type> newTypes;
     for (Value result : switchOp.getResults()) {
-      if (!isa<TensorType>(result.getType())) {
+      if (!isa<TensorLikeType>(result.getType())) {
         newTypes.push_back(result.getType());
         continue;
       }
@@ -762,7 +754,8 @@ struct ForOpInterface
       auto targetType = bufferization::getBufferType(result, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(castBuffer(rewriter, initArg, *targetType));
+      castedInitArgs.push_back(
+          castBuffer(rewriter, initArg, *targetType, options));
     }
 
     // Construct a new scf.for op with memref instead of tensor values.
@@ -986,7 +979,8 @@ struct WhileOpInterface
       auto targetType = bufferization::getBufferType(beforeArg, options, state);
       if (failed(targetType))
         return failure();
-      castedInitArgs.push_back(castBuffer(rewriter, initArg, *targetType));
+      castedInitArgs.push_back(
+          castBuffer(rewriter, initArg, *targetType, options));
     }
 
     // The result types of a WhileOp are the same as the "after" bbArg types.
@@ -1186,14 +1180,14 @@ struct YieldOpInterface
               yieldOp->getParentOp()->getResult(it.index()), options, state);
           if (failed(resultType))
             return failure();
-          buffer = castBuffer(rewriter, buffer, *resultType);
+          buffer = castBuffer(rewriter, buffer, *resultType, options);
         } else if (auto whileOp =
                        dyn_cast<scf::WhileOp>(yieldOp->getParentOp())) {
           FailureOr<BufferLikeType> resultType = bufferization::getBufferType(
               whileOp.getBeforeArguments()[it.index()], options, state);
           if (failed(resultType))
             return failure();
-          buffer = castBuffer(rewriter, buffer, *resultType);
+          buffer = castBuffer(rewriter, buffer, *resultType, options);
         }
         newResults.push_back(buffer);
       } else {

diff  --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
index b2674e9497927..ccc60d42ca23a 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
@@ -312,7 +312,7 @@ module @BufferizeLayoutForFunction {
     %memref = "test.tensor_with_future_layout"() {layout = #layout1}
       : () -> tensor<10xf32>
 
-    // CHECK: %[[out:.*]] = memref.cast %[[memref]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[memref]]
     // CHECK-SAME:  to memref<10xf32>
     // CHECK: return %[[out]]
     return %memref : tensor<10xf32>
@@ -341,7 +341,7 @@ module @BufferizeLayoutMismatchInsideScfIf {
     } else {
       // CHECK: %[[another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %another = "test.tensor_with_future_layout"() {layout = #layout2}
         : () -> tensor<10xf32>
@@ -349,7 +349,7 @@ module @BufferizeLayoutMismatchInsideScfIf {
       scf.yield %another : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[if]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[if]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %ret : tensor<10xf32>
@@ -381,7 +381,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
     case 1 {
       // CHECK: %[[another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %another = "test.tensor_with_future_layout"() {layout = #layout2}
         : () -> tensor<10xf32>
@@ -391,7 +391,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
     default {
       // CHECK: %[[yet_another:.*]] = "test.create_memref_op"
       // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_c">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[yet_another]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[yet_another]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %yet_another = "test.tensor_with_future_layout"() {layout = #layout3}
         : () -> tensor<10xf32>
@@ -399,7 +399,7 @@ module @BufferizeLayoutMismatchInsideScfSwitch {
       scf.yield %yet_another : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[switch]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[switch]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %ret : tensor<10xf32>
@@ -430,7 +430,7 @@ module @BufferizeLayoutMismatchInsideScfFor {
         iter_args(%iter = %init) -> (tensor<10xf32>) {
       // CHECK: %[[conflict:.*]] = "test.dummy_memref_op"(%[[iter]])
       // CHECK-SAME:  -> memref<10xf32, #test.memref_layout<"layout_b">>
-      // CHECK: %[[cast:.*]] = memref.cast %[[conflict]]
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[conflict]]
       // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
       %conflict = "test.force_new_layout"(%iter) {layout = #layout2}
         : (tensor<10xf32>) -> tensor<10xf32>
@@ -438,7 +438,7 @@ module @BufferizeLayoutMismatchInsideScfFor {
       scf.yield %conflict : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[loop]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %loop : tensor<10xf32>
@@ -450,6 +450,45 @@ module @BufferizeLayoutMismatchInsideScfFor {
 #layout1 = #test.memref_layout<"layout_a">
 #layout2 = #test.memref_layout<"layout_b">
 
+// CHECK-LABEL: @BufferizeLayoutMismatchInsideScfForWithUserType
+module @BufferizeLayoutMismatchInsideScfForWithUserType {
+  // CHECK: func.func @mismatch_in_for
+  // CHECK-SAME:  -> !test.test_memref<[10], f32>
+  func.func @mismatch_in_for(
+      %lb: index, %ub: index, %step: index)
+      -> !test.test_tensor<[10], f32> {
+    // CHECK: %[[init:.*]] = "test.create_memref_op"
+    // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+    %init = "test.tensor_with_future_layout"() {layout = #layout1}
+      : () -> !test.test_tensor<[10], f32>
+
+    // CHECK: %[[loop:.+]] = scf.for
+    // CHECK-SAME: iter_args(%[[iter:.*]] = %[[init]])
+    // CHECK-SAME: -> (!test.test_memref<[10], f32, #test.memref_layout<"layout_a">>)
+    %loop = scf.for %i = %lb to %ub step %step
+        iter_args(%iter = %init) -> (!test.test_tensor<[10], f32>) {
+      // CHECK: %[[conflict:.*]] = "test.dummy_memref_op"(%[[iter]])
+      // CHECK-SAME:  -> !test.test_memref<[10], f32, #test.memref_layout<"layout_b">>
+      // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[conflict]]
+      // CHECK-SAME: to !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+      %conflict = "test.force_new_layout"(%iter) {layout = #layout2}
+        : (!test.test_tensor<[10], f32>) -> !test.test_tensor<[10], f32>
+      // CHECK: scf.yield %[[cast]]
+      scf.yield %conflict : !test.test_tensor<[10], f32 >
+    }
+
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
+    // CHECK-SAME: to !test.test_memref<[10], f32>
+    // CHECK: return %[[out]]
+    return %loop : !test.test_tensor<[10], f32>
+  }
+}
+
+// -----
+
+#layout1 = #test.memref_layout<"layout_a">
+#layout2 = #test.memref_layout<"layout_b">
+
 // Test that custom layout can co-exist in principle within an "end-to-end"
 // SCF example (`for { extract slice -> custom op -> insert slice }`) without
 // bufferization failing completely due to a layout mismatch. The fact that the
@@ -495,7 +534,7 @@ module @BufferizeLayoutMismatchInsideScfForWithSubviews {
       scf.yield %out : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[loop]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[loop]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %loop : tensor<10xf32>
@@ -528,7 +567,7 @@ module @BufferizeLayoutMismatchInScfExecuteRegion {
       ^bb2:
         // CHECK: %[[another:.*]] = "test.create_memref_op"
         // CHECK-SAME: -> memref<10xf32, #test.memref_layout<"layout_b">>
-        // CHECK: %[[cast:.*]] = memref.cast %[[another]]
+        // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
         // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
         %another = "test.tensor_with_future_layout"() {layout = #layout2}
           : () -> tensor<10xf32>
@@ -540,13 +579,59 @@ module @BufferizeLayoutMismatchInScfExecuteRegion {
         scf.yield %res : tensor<10xf32>
     }
 
-    // CHECK: %[[out:.*]] = memref.cast %[[region]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[region]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %out : tensor<10xf32>
   }
 }
 
+// -----
+
+#layout1 = #test.memref_layout<"layout_a">
+#layout2 = #test.memref_layout<"layout_b">
+
+// CHECK-LABEL: @BufferizeLayoutMismatchInScfExecuteRegionWithUserType
+module @BufferizeLayoutMismatchInScfExecuteRegionWithUserType {
+  // CHECK: func.func @mismatch_in_scf_execute_region
+  // CHECK-SAME: -> !test.test_memref<[10], f32>
+  func.func @mismatch_in_scf_execute_region(%cond: i1) -> !test.test_tensor<[10], f32> {
+    // CHECK: %[[region:.*]] = scf.execute_region
+    // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+    %out = scf.execute_region -> !test.test_tensor<[10], f32> {
+      cf.cond_br %cond, ^bb1, ^bb2
+
+      ^bb1:
+        // CHECK: %[[one:.*]] = "test.create_memref_op"
+        // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        %one = "test.tensor_with_future_layout"() {layout = #layout1}
+          : () -> !test.test_tensor<[10], f32>
+        // CHECK: cf.br ^bb3(%[[one]]
+        cf.br ^bb3(%one : !test.test_tensor<[10], f32>)
+
+      ^bb2:
+        // CHECK: %[[another:.*]] = "test.create_memref_op"
+        // CHECK-SAME: -> !test.test_memref<[10], f32, #test.memref_layout<"layout_b">>
+        // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[another]]
+        // CHECK-SAME: to !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        %another = "test.tensor_with_future_layout"() {layout = #layout2}
+          : () -> !test.test_tensor<[10], f32>
+        // CHECK: cf.br ^bb3(%[[cast]]
+        cf.br ^bb3(%another : !test.test_tensor<[10], f32>)
+
+      ^bb3(%res: !test.test_tensor<[10], f32>):
+        // CHECK: scf.yield {{%.*}} : !test.test_memref<[10], f32, #test.memref_layout<"layout_a">>
+        scf.yield %res : !test.test_tensor<[10], f32>
+    }
+
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[region]]
+    // CHECK-SAME: to !test.test_memref<[10], f32>
+    // CHECK: return %[[out]]
+    return %out : !test.test_tensor<[10], f32>
+  }
+}
+
+
 // -----
 
 #layout1 = #test.memref_layout<"layout_a">
@@ -566,13 +651,13 @@ module @BufferizeLayoutMismatchInArithSelect {
     %tensor2 = "test.tensor_with_future_layout"() {layout = #layout2}
       : () -> tensor<10xf32>
 
-    // CHECK: %[[cast:.*]] = memref.cast %[[memref2]]
+    // CHECK: %[[cast:.*]] = test.dummy_cast_op %[[memref2]]
     // CHECK-SAME: to memref<10xf32, #test.memref_layout<"layout_a">>
 
     // CHECK: %[[select:.*]] = arith.select %[[cond]], %[[memref1]], %[[cast]]
     %select = arith.select %cond, %tensor1, %tensor2 : i1, tensor<10xf32>
 
-    // CHECK: %[[out:.*]] = memref.cast %[[select]]
+    // CHECK: %[[out:.*]] = test.dummy_cast_op %[[select]]
     // CHECK-SAME: to memref<10xf32>
     // CHECK: return %[[out]]
     return %select : tensor<10xf32>

diff  --git a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
index 00eb2dc92e14f..036e05bfc2e02 100644
--- a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
+++ b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
@@ -16,6 +16,7 @@
 
 #include "TestAttributes.h" // TestTensorEncodingAttr, TestMemRefLayoutAttr
 #include "TestDialect.h"
+#include "TestOps.h"
 
 using namespace mlir;
 
@@ -30,6 +31,21 @@ getMemRefLayoutForTensorEncoding(RankedTensorType tensorType) {
   return {};
 }
 
+test::TestMemRefLayoutAttr
+getLayoutFromBuffer(bufferization::BufferLikeType buffer) {
+  auto layout =
+      llvm::TypeSwitch<bufferization::BufferLikeType,
+                       MemRefLayoutAttrInterface>(buffer)
+          .Case([&](MemRefType memref) { return memref.getLayout(); })
+          .Case([&](test::TestMemrefType testMemref) {
+            return cast<MemRefLayoutAttrInterface>(testMemref.getLayout());
+          })
+          .Default([](bufferization::BufferLikeType) {
+            return MemRefLayoutAttrInterface();
+          });
+  return dyn_cast_or_null<test::TestMemRefLayoutAttr>(layout);
+}
+
 struct TestOneShotModuleBufferizePass
     : public PassWrapper<TestOneShotModuleBufferizePass, OperationPass<>> {
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestOneShotModuleBufferizePass)
@@ -82,9 +98,14 @@ struct TestOneShotModuleBufferizePass
               })
               .Case([&](test::TestTensorType testTensorType)
                         -> bufferization::BufferLikeType {
+                assert(memSpace == nullptr &&
+                       "memory space is not supported for test types");
+                // Note: unknown type conversion cannot create layout because
+                // test tensor does not have an encoding, this is probably
+                // enough for a test.
                 return test::TestMemrefType::get(
                     testTensorType.getContext(), testTensorType.getShape(),
-                    testTensorType.getElementType(), memSpace);
+                    testTensorType.getElementType(), /*layout=*/nullptr);
               })
               .Default([&](bufferization::TensorLikeType tensor) {
                 llvm_unreachable("unexpected tensor type");
@@ -97,13 +118,8 @@ struct TestOneShotModuleBufferizePass
         [](bufferization::BufferLikeType x, bufferization::BufferLikeType y,
            const bufferization::BufferizationOptions &)
         -> FailureOr<bufferization::BufferLikeType> {
-      auto getLayout = [](bufferization::BufferLikeType t) {
-        auto m = dyn_cast<MemRefType>(t);
-        return m ? dyn_cast<test::TestMemRefLayoutAttr>(m.getLayout())
-                 : test::TestMemRefLayoutAttr();
-      };
-      auto lhsLayout = getLayout(x);
-      auto rhsLayout = getLayout(y);
+      auto lhsLayout = getLayoutFromBuffer(x);
+      auto rhsLayout = getLayoutFromBuffer(y);
       if (lhsLayout && rhsLayout) {
         return lhsLayout.getDummy().getValue() <=
                        rhsLayout.getDummy().getValue()
@@ -112,6 +128,9 @@ struct TestOneShotModuleBufferizePass
       }
       return rhsLayout ? y : x;
     };
+    opt.castFn = [&](OpBuilder &b, Location loc, Type dest, Value value) {
+      return test::TestDummyCastOp::create(b, loc, dest, value).getResult();
+    };
     // Function signature update only works with memref.cast. Disable it to
     // align behaviour for upstream and user casts.
     opt.inferFunctionResultLayout = false;

diff  --git a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
index 20d72a2fd09ca..75dae36fe9ce8 100644
--- a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
@@ -2068,19 +2068,27 @@ test::TestTensorWithFutureLayoutOp::getBufferType(
     mlir::Value value, const mlir::bufferization::BufferizationOptions &,
     const mlir::bufferization::BufferizationState &,
     llvm::SmallVector<::mlir::Value> &) {
-  auto tensorType = dyn_cast<RankedTensorType>(value.getType());
-  if (!tensorType)
-    return failure();
-
-  // Set the memref layout to the op's 'layout' attribute, ignoring
-  // any pre-existing tensor encoding. This is what lets two
-  // `test.tensor_with_layout` ops produce *bufferized* memrefs with 
diff erent
-  // layouts while keeping their *tensor* result types identical -- which is
-  // required to construct SCF iter_arg/branch mismatches that the verifier
-  // still accepts.
-  auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
-  return cast<bufferization::BufferLikeType>(MemRefType::get(
-      tensorType.getShape(), tensorType.getElementType(), layout));
+  return llvm::TypeSwitch<mlir::Type,
+                          mlir::FailureOr<mlir::bufferization::BufferLikeType>>(
+             value.getType())
+      .Case([&](RankedTensorType tensorType) {
+        // Set the memref layout to the op's 'layout' attribute, ignoring
+        // any pre-existing tensor encoding. This is what lets two
+        // `test.tensor_with_layout` ops produce *bufferized* memrefs with
+        // 
diff erent layouts while keeping their *tensor* result types identical
+        // -- which is required to construct SCF iter_arg/branch mismatches that
+        // the verifier still accepts.
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(MemRefType::get(
+            tensorType.getShape(), tensorType.getElementType(), layout));
+      })
+      .Case([&](TestTensorType tensorType) {
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(
+            TestMemrefType::get(tensorType.getContext(), tensorType.getShape(),
+                                tensorType.getElementType(), layout));
+      })
+      .Default([&](Type) { return emitError("unknown type"); });
 }
 
 LogicalResult test::TestForceNewLayoutOp::bufferize(
@@ -2107,15 +2115,23 @@ test::TestForceNewLayoutOp::getBufferType(
     mlir::Value value, const bufferization::BufferizationOptions &options,
     const bufferization::BufferizationState &,
     llvm::SmallVector<::mlir::Value> &) {
-  auto tensorType = dyn_cast<RankedTensorType>(value.getType());
-  if (!tensorType)
-    return failure();
-
-  // Set the memref layout to the op's 'layout' attribute, ignoring any
-  // pre-existing tensor encoding.
-  auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
-  return cast<bufferization::BufferLikeType>(MemRefType::get(
-      tensorType.getShape(), tensorType.getElementType(), layout));
+  return llvm::TypeSwitch<mlir::Type,
+                          mlir::FailureOr<mlir::bufferization::BufferLikeType>>(
+             value.getType())
+      .Case([&](RankedTensorType tensorType) {
+        // Set the memref layout to the op's 'layout' attribute, ignoring any
+        // pre-existing tensor encoding.
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(MemRefType::get(
+            tensorType.getShape(), tensorType.getElementType(), layout));
+      })
+      .Case([&](TestTensorType tensorType) {
+        auto layout = cast<MemRefLayoutAttrInterface>(getLayout());
+        return cast<bufferization::BufferLikeType>(
+            TestMemrefType::get(tensorType.getContext(), tensorType.getShape(),
+                                tensorType.getElementType(), layout));
+      })
+      .Default([&](Type) { return emitError("unknown type"); });
 }
 
 // Define a custom builder for ManyRegionsOp declared in TestOps.td.

diff  --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index 29f456d1ec7eb..c12e2abb3ab75 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -4239,7 +4239,7 @@ def TestTensorWithFutureLayoutOp
   }];
 
   let arguments = (ins TestMemRefLayoutAttr:$layout);
-  let results = (outs AnyRankedTensor:$output);
+  let results = (outs Bufferization_TensorLikeTypeInterface:$output);
   let extraClassDefinition = [{
     bool test::TestTensorWithFutureLayoutOp::bufferizesToMemoryRead(
         ::mlir::OpOperand&,
@@ -4285,10 +4285,10 @@ def TestForceNewLayoutOp
   }];
 
   let arguments = (ins
-    AnyRankedTensor:$input,
+    Bufferization_TensorLikeTypeInterface:$input,
     TestMemRefLayoutAttr:$layout
   );
-  let results = (outs AnyRankedTensor:$output);
+  let results = (outs Bufferization_TensorLikeTypeInterface:$output);
   let extraClassDefinition = [{
     bool test::TestForceNewLayoutOp::bufferizesToMemoryRead(::mlir::OpOperand&,
         const ::mlir::bufferization::AnalysisState&) {
@@ -4322,6 +4322,16 @@ def TestForceNewLayoutOp
   }];
 }
 
+def TestDummyCastOp : TEST_Op<"dummy_cast_op", []> {
+  let arguments = (ins
+    Arg<Bufferization_BufferLikeTypeInterface>:$source
+  );
+  let results = (outs
+    Arg<Bufferization_BufferLikeTypeInterface>:$dest
+  );
+  let assemblyFormat = "$source attr-dict `:` type($source) `to` type($dest)";
+}
+
 //===----------------------------------------------------------------------===//
 // Test assembly format references
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/test/lib/Dialect/Test/TestTypeDefs.td b/mlir/test/lib/Dialect/Test/TestTypeDefs.td
index f405940358ca0..58c340a935bdb 100644
--- a/mlir/test/lib/Dialect/Test/TestTypeDefs.td
+++ b/mlir/test/lib/Dialect/Test/TestTypeDefs.td
@@ -477,9 +477,9 @@ def TestMemrefType : Test_Type<"TestMemref",
   let parameters = (ins
     ArrayRefParameter<"int64_t">:$shape,
     "mlir::Type":$elementType,
-    DefaultValuedParameter<"mlir::Attribute", "nullptr">:$memSpace
+    DefaultValuedParameter<"mlir::MemRefLayoutAttrInterface", "nullptr">:$layout
   );
-  let assemblyFormat = "`<` `[` $shape `]` `,` $elementType (`,` $memSpace^)? `>`";
+  let assemblyFormat = "`<` `[` $shape `]` `,` $elementType (`,` $layout^)? `>`";
 
   let extraClassDeclaration = [{
     // ShapedTypeInterface:
@@ -489,7 +489,7 @@ def TestMemrefType : Test_Type<"TestMemref",
     test::TestMemrefType cloneWith(std::optional<llvm::ArrayRef<int64_t>> shape,
                                    mlir::Type elementType) const {
       return test::TestMemrefType::get(
-        getContext(), shape.value_or(getShape()), elementType, getMemSpace());
+        getContext(), shape.value_or(getShape()), elementType, getLayout());
     }
   }];
 }


        

From llvmlistbot at llvm.org  Wed Jul  1 03:55:02 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 03:55:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a44f206.c56fb102.1c15b6.9601@mx.google.com>


https://github.com/andrey-golubev closed https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 03:59:25 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 01 Jul 2026 03:59:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a44f30d.cb30c3e4.37526.192a@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11404

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8591/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInlining.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8593/8676] Linking CXX executable bin/fir-lsp-server
[8594/8676] Linking CXX shared library lib/libHLFIRTransforms.so.23.0git
[8595/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8596/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
[8597/8676] Linking CXX shared library lib/libFortranLower.so.23.0git
[8598/8676] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8599/8676] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8600/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8601/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
[8602/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8603/8676] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8591/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInlining.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8593/8676] Linking CXX executable bin/fir-lsp-server
[8594/8676] Linking CXX shared library lib/libHLFIRTransforms.so.23.0git
[8595/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8596/8676] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
[8597/8676] Linking CXX shared library lib/libFortranLower.so.23.0git
[8598/8676] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8599/8676] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8600/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8601/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
[8602/8676] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8603/8676] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=132.508597

```

</details>

https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 04:03:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:03:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f2fd0d6 - [mlir][SPIR-V] Add spirv.GL.Step
 and spirv.GL.SmoothStep ops (#206037)
Message-ID: <6a44f400.07a3e4bc.32ad2b.a050@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-01T13:03:23+02:00
New Revision: f2fd0d648987f0187a7e57bca17bf10ef2d289d8

URL: https://github.com/llvm/llvm-project/commit/f2fd0d648987f0187a7e57bca17bf10ef2d289d8
DIFF: https://github.com/llvm/llvm-project/commit/f2fd0d648987f0187a7e57bca17bf10ef2d289d8.diff

LOG: [mlir][SPIR-V] Add spirv.GL.Step and spirv.GL.SmoothStep ops (#206037)

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
    mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
    mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
    mlir/test/Target/SPIRV/gl-ops.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td b/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
index e0b85e73cad99..34b49567deafe 100644
--- a/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
+++ b/mlir/include/mlir/Dialect/SPIRV/IR/SPIRVGLOps.td
@@ -1488,6 +1488,58 @@ def SPIRV_GLReflectOp : SPIRV_GLBinaryArithmeticOp<"Reflect", 71, SPIRV_Float, [
 
 // ----
 
+def SPIRV_GLStepOp : SPIRV_GLBinaryArithmeticOp<"Step", 48, SPIRV_Float, [AlwaysSpeculatable]> {
+  let summary = "Return 0.0 if x < edge, else 1.0";
+
+  let description = [{
+    Result is 0.0 if x < edge; otherwise result is 1.0.
+
+    The operands must all be a scalar or vector whose component type is
+    floating-point.
+
+    Result Type and the type of all operands must be the same type. Results
+    are computed per component.
+
+    #### Example:
+
+    ```mlir
+    %2 = spirv.GL.Step %edge, %x : f32
+    %3 = spirv.GL.Step %edge, %x : vector<3xf32>
+    ```
+  }];
+}
+
+// ----
+
+def SPIRV_GLSmoothStepOp : SPIRV_GLTernaryArithmeticOp<"SmoothStep", 49, SPIRV_Float> {
+  let summary = "Return smooth Hermite interpolation between 0 and 1";
+
+  let description = [{
+    Result is 0.0 if x <= edge0 and 1.0 if x >= edge1 and performs smooth
+    Hermite interpolation between 0 and 1 when edge0 < x < edge1. This is
+    equivalent to:
+      t = clamp((x - edge0) / (edge1 - edge0), 0, 1)
+      t * t * (3 - 2 * t)
+
+    The resulting value is undefined if edge0 >= edge1.
+
+    The operands must all be a scalar or vector whose component type is
+    floating-point.
+
+    Result Type and the type of all operands must be the same type. Results
+    are computed per component.
+
+    #### Example:
+
+    ```mlir
+    %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : f32
+    %3 = spirv.GL.SmoothStep %edge0, %edge1, %x : vector<3xf32>
+    ```
+  }];
+}
+
+// ----
+
 def SPIRV_GLFindILsbOp : SPIRV_GLUnaryArithmeticOp<"FindILsb", 73, SPIRV_Integer, [AlwaysSpeculatable]> {
   let summary = "Integer least-significant bit";
 

diff  --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index f699055b5502f..c0393888af5c3 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -1294,6 +1294,18 @@ ParseResult spirv::GLNClampOp::parse(OpAsmParser &parser,
 }
 void spirv::GLNClampOp::print(OpAsmPrinter &p) { printOneResultOp(*this, p); }
 
+//===----------------------------------------------------------------------===//
+// spirv.GLSmoothStepOp
+//===----------------------------------------------------------------------===//
+
+ParseResult spirv::GLSmoothStepOp::parse(OpAsmParser &parser,
+                                         OperationState &result) {
+  return parseOneResultSameOperandTypeOp(parser, result);
+}
+void spirv::GLSmoothStepOp::print(OpAsmPrinter &p) {
+  printOneResultOp(*this, p);
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GLFmaOp
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir b/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
index 5665ae158d2e4..627c764feb6e6 100644
--- a/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/gl-ops.mlir
@@ -839,6 +839,50 @@ func.func @reflect_invalid_type(%arg0 : i32, %arg1 : i32) {
 
 // -----
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.Step
+//===----------------------------------------------------------------------===//
+
+func.func @step_scalar(%edge : f32, %x : f32) {
+  %2 = spirv.GL.Step %edge, %x : f32
+  // CHECK: %{{.+}} = spirv.GL.Step %{{.+}}, %{{.+}} : f32
+  return
+}
+
+func.func @step_vector(%edge : vector<3xf32>, %x : vector<3xf32>) {
+  %2 = spirv.GL.Step %edge, %x : vector<3xf32>
+  // CHECK: %{{.+}} = spirv.GL.Step %{{.+}}, %{{.+}} : vector<3xf32>
+  return
+}
+
+// -----
+
+func.func @step_invalid_type(%edge : i32, %x : i32) {
+  // expected-error @+1 {{'spirv.GL.Step' op operand #0 must be 16/32/64-bit float or fixed-length vector of 16/32/64-bit float values}}
+  %0 = spirv.GL.Step %edge, %x : i32
+  return
+}
+
+// -----
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SmoothStep
+//===----------------------------------------------------------------------===//
+
+func.func @smoothstep_scalar(%edge0 : f32, %edge1 : f32, %x : f32) {
+  // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : f32
+  %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : f32
+  return
+}
+
+func.func @smoothstep_vector(%edge0 : vector<3xf32>, %edge1 : vector<3xf32>, %x : vector<3xf32>) {
+  // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : vector<3xf32>
+  %2 = spirv.GL.SmoothStep %edge0, %edge1, %x : vector<3xf32>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Fract
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/test/Target/SPIRV/gl-ops.mlir b/mlir/test/Target/SPIRV/gl-ops.mlir
index bc17ff82cc441..41364dca31af3 100644
--- a/mlir/test/Target/SPIRV/gl-ops.mlir
+++ b/mlir/test/Target/SPIRV/gl-ops.mlir
@@ -174,4 +174,16 @@ spirv.module Logical GLSL450 requires #spirv.vce<v1.0, [Shader, Linkage], []> {
     %1 = spirv.GL.PackSnorm4x8 %0 : vector<4xf32> -> i32
     spirv.Return
   }
+
+  spirv.func @step(%arg0 : f32, %arg1 : f32) "None" {
+    // CHECK: spirv.GL.Step {{%[^,]*}}, {{%[^,]*}} : f32
+    %0 = spirv.GL.Step %arg0, %arg1 : f32
+    spirv.Return
+  }
+
+  spirv.func @smoothstep(%arg0 : f32, %arg1 : f32, %arg2 : f32) "None" {
+    // CHECK: spirv.GL.SmoothStep {{%[^,]*}}, {{%[^,]*}}, {{%[^,]*}} : f32
+    %0 = spirv.GL.SmoothStep %arg0, %arg1, %arg2 : f32
+    spirv.Return
+  }
 }


        

From llvmlistbot at llvm.org  Wed Jul  1 04:03:31 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 04:03:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add spirv.GL.Step and
 spirv.GL.SmoothStep ops (PR #206037)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206037@github.com>
Message-ID: <6a44f403.9ffd0497.1ca4c1.6a52@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/206037

From llvmlistbot at llvm.org  Wed Jul  1 04:16:18 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 04:16:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 VectorTimesScalar (PR #206949)
Message-ID: <llvm.org/llvm/llvm-project/pull/206949@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206949

None

>From 86c221950fd16910955f66c6be5c59c7f76b99f0 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:10:12 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 VectorTimesScalar

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 26 +++++++++++++++++++
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 15 +++++++++++
 2 files changed, 41 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..23007a58713bd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,31 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.VectorTimesScalar` to a broadcast of the scalar followed by
+/// an `llvm.fmul`.
+class VectorTimesScalarPattern
+    : public SPIRVToLLVMConversion<spirv::VectorTimesScalarOp> {
+public:
+  using SPIRVToLLVMConversion<
+      spirv::VectorTimesScalarOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::VectorTimesScalarOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    unsigned numElements = op.getVector().getType().getNumElements();
+    Value broadcasted = broadcast(op.getLoc(), adaptor.getScalar(), numElements,
+                                  *getTypeConverter(), rewriter);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getVector(),
+                                              broadcasted);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1825,6 +1850,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
       DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      VectorTimesScalarPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..0739f377c008f 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,18 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.VectorTimesScalar
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @vector_times_scalar
+spirv.func @vector_times_scalar(%vector: vector<4xf32>, %scalar: f32) "None" {
+  // CHECK: %[[UNDEF:.*]] = llvm.mlir.poison : vector<4xf32>
+  // CHECK: llvm.insertelement %{{.*}}, %[[UNDEF]]
+  // CHECK-COUNT-2: llvm.insertelement
+  // CHECK: %[[BCAST:.*]] = llvm.insertelement
+  // CHECK: llvm.fmul %{{.*}}, %[[BCAST]] : vector<4xf32>
+  %0 = spirv.VectorTimesScalar %vector, %scalar : (vector<4xf32>, f32) -> vector<4xf32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 04:16:19 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 04:16:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
Message-ID: <llvm.org/llvm/llvm-project/pull/206950@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206950

None

>From 187cd538f98c32b897f1d8839a38a3b6915917a1 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:12:30 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for SNegate

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 31 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 20 ++++++++++++
 2 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..610c052b64ea7 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1853,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 04:16:20 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 01 Jul 2026 04:16:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
Message-ID: <llvm.org/llvm/llvm-project/pull/206951@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/206951

None

>From 0c27ffa07eb9b334cbc1bd5e4df2a94abd4e2643 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:15:31 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for GL.Fract

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 22 ++++++++++++++++++-
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 15 +++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c9eba35cf0939 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, operand);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, operand, floored);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1936,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..042b68bef9dcd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,18 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Fract
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fract
+spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%{{.*}}) : (f32) -> f32
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOOR]] : f32
+  %0 = spirv.GL.Fract %arg0 : f32
+  // CHECK: %[[FLOORV:.*]] = llvm.intr.floor(%{{.*}}) : (vector<3xf16>) -> vector<3xf16>
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOORV]] : vector<3xf16>
+  %1 = spirv.GL.Fract %arg1 : vector<3xf16>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Wed Jul  1 04:16:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:16:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a44f725.b8c88a87.f3618.e416@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206950.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+30-1) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir (+20) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..610c052b64ea7 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1853,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Wed Jul  1 04:16:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:16:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 VectorTimesScalar (PR #206949)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206949@github.com>
Message-ID: <6a44f726.e72b82d4.141ee.1397@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-spirv

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206949.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+26) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir (+15) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..23007a58713bd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,31 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.VectorTimesScalar` to a broadcast of the scalar followed by
+/// an `llvm.fmul`.
+class VectorTimesScalarPattern
+    : public SPIRVToLLVMConversion<spirv::VectorTimesScalarOp> {
+public:
+  using SPIRVToLLVMConversion<
+      spirv::VectorTimesScalarOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::VectorTimesScalarOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    unsigned numElements = op.getVector().getType().getNumElements();
+    Value broadcasted = broadcast(op.getLoc(), adaptor.getScalar(), numElements,
+                                  *getTypeConverter(), rewriter);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getVector(),
+                                              broadcasted);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1825,6 +1850,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
       DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      VectorTimesScalarPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..0739f377c008f 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,18 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.VectorTimesScalar
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @vector_times_scalar
+spirv.func @vector_times_scalar(%vector: vector<4xf32>, %scalar: f32) "None" {
+  // CHECK: %[[UNDEF:.*]] = llvm.mlir.poison : vector<4xf32>
+  // CHECK: llvm.insertelement %{{.*}}, %[[UNDEF]]
+  // CHECK-COUNT-2: llvm.insertelement
+  // CHECK: %[[BCAST:.*]] = llvm.insertelement
+  // CHECK: llvm.fmul %{{.*}}, %[[BCAST]] : vector<4xf32>
+  %0 = spirv.VectorTimesScalar %vector, %scalar : (vector<4xf32>, f32) -> vector<4xf32>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206949

From llvmlistbot at llvm.org  Wed Jul  1 04:16:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:16:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a44f727.e0e11d61.1fabe9.0a9b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-spirv

@llvm/pr-subscribers-mlir

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/206951.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+21-1) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir (+15) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c9eba35cf0939 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, operand);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, operand, floored);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1936,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..042b68bef9dcd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,18 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Fract
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fract
+spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%{{.*}}) : (f32) -> f32
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOOR]] : f32
+  %0 = spirv.GL.Fract %arg0 : f32
+  // CHECK: %[[FLOORV:.*]] = llvm.intr.floor(%{{.*}}) : (vector<3xf16>) -> vector<3xf16>
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOORV]] : vector<3xf16>
+  %1 = spirv.GL.Fract %arg1 : vector<3xf16>
+  spirv.Return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/206951

From llvmlistbot at llvm.org  Wed Jul  1 04:23:34 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 04:23:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44f8b6.14d9b9dc.7e5e3.1daa@mx.google.com>


https://github.com/pechy updated https://github.com/llvm/llvm-project/pull/206310

>From 6a37fc159eead0e85f6821867d741a2d4c2423c2 Mon Sep 17 00:00:00 2001
From: Martin Petracek <martin at petracek.net>
Date: Mon, 29 Jun 2026 23:27:05 +0700
Subject: [PATCH] [MLIR] Verify that all blocks are reachable before CFG to SCF
 transform

CFG to SCF transform cannot handle unreachable code. Properly verify
that all blocks are reachable and document this limitation in the pass
description.

Fixes #206086
---
 mlir/include/mlir/Transforms/CFGToSCF.h        |  7 ++++---
 mlir/lib/Transforms/Utils/CFGToSCF.cpp         |  7 ++++++-
 .../ControlFlowToSCF/unreachable-blocks.mlir   | 18 ++++++++++++++++++
 3 files changed, 28 insertions(+), 4 deletions(-)
 create mode 100644 mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir

diff --git a/mlir/include/mlir/Transforms/CFGToSCF.h b/mlir/include/mlir/Transforms/CFGToSCF.h
index 517e116f3c58e..2ca019ddfe23d 100644
--- a/mlir/include/mlir/Transforms/CFGToSCF.h
+++ b/mlir/include/mlir/Transforms/CFGToSCF.h
@@ -149,9 +149,10 @@ class CFGToSCFInterface {
 /// Otherwise a single control flow graph operation branching to one block
 /// per return-like operation kind remains.
 ///
-/// The transformation currently requires that all control flow graph operations
-/// have no side effects, implement the BranchOpInterface and does not have any
-/// operation produced successor operands.
+/// The transformation currently requires that the region has no unreachable
+/// blocks and that all control flow graph operations have no side effects,
+/// implement the BranchOpInterface and does not have any operation produced
+/// successor operands.
 /// Returns failure if any of the preconditions are violated or if any of the
 /// methods of `interface` failed. The IR is left in an unspecified state.
 ///
diff --git a/mlir/lib/Transforms/Utils/CFGToSCF.cpp b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
index 0cc6bb43ddd38..2cda4a26543bd 100644
--- a/mlir/lib/Transforms/Utils/CFGToSCF.cpp
+++ b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
@@ -1234,8 +1234,13 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
+  llvm::df_iterator_default_set<Block *, 16> reachable;
+  // Find all blocks reachable from the entry.
+  for (Block *block : depth_first_ext(&region.front(), reachable))
+    (void)block;
+
   for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+    if (!reachable.contains(&block))
       return block.front().emitOpError(
           "transformation does not support unreachable blocks");
 
diff --git a/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
new file mode 100644
index 0000000000000..4317e5a5a8102
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt --lift-cf-to-scf -verify-diagnostics -split-input-file %s
+
+// Verify that --lift-cf-to-scf does not crash when it encounters an
+// unreachable blocks (dead code).  Instead it should emit a clean error.
+// See: https://github.com/llvm/llvm-project/issues/206086
+
+module {
+  func.func @single_return_statement(%arg0: i32) -> i32 {
+    return %arg0 : i32
+  ^bb1:  // pred: ^bb1
+    // expected-error at below {{'arith.constant' op transformation does not support unreachable blocks}}
+    %true = arith.constant true
+    cf.cond_br %true, ^bb2, ^bb1
+  ^bb2:  // pred: ^bb1
+    return %arg0 : i32
+  }
+}
+


From llvmlistbot at llvm.org  Wed Jul  1 04:26:06 2026
From: llvmlistbot at llvm.org (Martin P)
Date: Wed, 01 Jul 2026 04:26:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a44f94e.c20dae4b.1a23b.5af0@mx.google.com>


================
@@ -1234,10 +1234,24 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
-  for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+  const auto reachableBlocks = [&region] {
+    SmallVector<Block *, 16> workList = {&region.front()};
+    SmallPtrSet<Block *, 16> visited = {&region.front()};
+    while (!workList.empty()) {
+      Block *current = workList.pop_back_val();
+      for (Block *succ : current->getSuccessors()) {
+        if (visited.insert(succ).second) {
+          workList.push_back(succ);
+        }
+      }
+    }
+    return visited;
+  }();
----------------
pechy wrote:

neat

https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 04:33:11 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Wed, 01 Jul 2026 04:33:11 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
Message-ID: <llvm.org/llvm/llvm-project/pull/206953@github.com>


https://github.com/skatrak created https://github.com/llvm/llvm-project/pull/206953

This patch modifies the handling of the `-fopenmp-assume-teams-oversubscription` and `-fopenmp-assume-threads-oversubscription` compiler options in Flang and the corresponding `omp.flags` MLIR attribute to have this information present on both host and target device MLIR modules. This is necessary to ensure SPMD to SPMD-No-Loop kernel promotion happens uniformly and no divergence between host and device representations of the same target region is introduced.

The identification of the kernel type in Flang lowering is updated to address some broken edge cases. In particular, no SPMD to SPMD-No-Loop transformation is performed on `target parallel do [simd]` kernels. A split `target + parallel loop` is now marked SPMD, which was a case missed previously, and the trip count on `target teams ompx_bare` kernels isn't host evaluated even if removing the `ompx_bare` clause would result in a valid SPMD kernel.

>From 853def0a44eb53a2edd6178972481af008d97468 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Wed, 1 Jul 2026 12:11:06 +0100
Subject: [PATCH] [Flang][OpenMP][MLIR] Improve SPMD-No-Loop handling

This patch modifies the handling of the -fopenmp-assume-teams-oversubscription
and -fopenmp-assume-threads-oversubscription compiler options in Flang
and the corresponding `omp.flags` MLIR attribute to have this
information present on both host and target device MLIR modules. This is
necessary to ensure SPMD to SPMD-No-Loop kernel promotion happens
uniformly and no divergence between host and device representations of
the same target region is introduced.

The identification of the kernel type in Flang lowering is updated to
address some broken edge cases. In particular, no SPMD to SPMD-No-Loop
transformation is performed on `target parallel do [simd]` kernels. A
split `target + parallel loop` is now marked SPMD, which was a case missed
previously, and the trip count on `target teams ompx_bare` kernels isn't host
evaluated even if removing the `ompx_bare` clause would result in a
valid SPMD kernel.
---
 clang/lib/Driver/ToolChains/Flang.cpp         |  24 +-
 flang/lib/Frontend/CompilerInvocation.cpp     |  22 +-
 flang/lib/Lower/OpenMP/OpenMP.cpp             |  72 ++--
 flang/test/Driver/omp-driver-offload.f90      |   8 +-
 .../test/Lower/OpenMP/target-kernel-types.f90 | 336 ++++++++++++++++++
 .../mlir/Dialect/OpenMP/OpenMPOffloadUtils.h  |  15 +-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |  15 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  15 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  23 +-
 mlir/test/Dialect/OpenMP/ops.mlir             |  19 +-
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  14 +-
 .../offloading/fortran/target-no-loop.f90     |   4 +-
 12 files changed, 482 insertions(+), 85 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/target-kernel-types.f90

diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index e4aa1c9456d9a..ee67c570fb96c 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -739,6 +739,19 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     }
   }
 
+  // When in OpenMP offloading mode, forward assumptions information about
+  // thread and team counts in the target device. The host needs to know about
+  // this to prevent the SPMD to SPMD-no-loop promotion being done differently
+  // for host and device on the same target region.
+  if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
+                   options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
+  if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
+                   options::OPT_fno_openmp_assume_threads_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
+
   if (IsOpenMPDevice) {
     // -fopenmp-is-target-device is passed along to tell the frontend that it is
     // generating code for a device, so that only the relevant code is emitted.
@@ -749,17 +762,6 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     if (Args.hasFlag(options::OPT_fopenmp_target_debug,
                      options::OPT_fno_openmp_target_debug, /*Default=*/false))
       CmdArgs.push_back("-fopenmp-target-debug");
-
-    // When in OpenMP offloading mode, forward assumptions information about
-    // thread and team counts in the device.
-    if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
-                     options::OPT_fno_openmp_assume_teams_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
-    if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
-                     options::OPT_fno_openmp_assume_threads_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_thread_state))
       CmdArgs.push_back("-fopenmp-assume-no-thread-state");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_nested_parallelism))
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 003b815bef678..8e13fc9f05324 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -1301,6 +1301,16 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
     }
   }
 
+  if (args.hasFlag(clang::options::OPT_fopenmp_assume_teams_oversubscription,
+                   clang::options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    res.getLangOpts().OpenMPTeamSubscription = true;
+  if (args.hasFlag(
+          clang::options::OPT_fopenmp_assume_threads_oversubscription,
+          clang::options::OPT_fno_openmp_assume_threads_oversubscription,
+          /*Default=*/false))
+    res.getLangOpts().OpenMPThreadSubscription = true;
+
   if (args.hasArg(clang::options::OPT_fopenmp_force_usm)) {
     res.getLangOpts().OpenMPForceUSM = 1;
   }
@@ -1317,24 +1327,12 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
             << res.getLangOpts().OMPHostIRFile;
     }
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_teams_oversubscription,
-            clang::options::OPT_fno_openmp_assume_teams_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPTeamSubscription = true;
-
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_thread_state))
       res.getLangOpts().OpenMPNoThreadState = 1;
 
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_nested_parallelism))
       res.getLangOpts().OpenMPNoNestedParallelism = 1;
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_threads_oversubscription,
-            clang::options::OPT_fno_openmp_assume_threads_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPThreadSubscription = true;
-
     if ((args.hasArg(clang::options::OPT_fopenmp_target_debug) ||
          args.hasArg(clang::options::OPT_fopenmp_target_debug_EQ))) {
       res.getLangOpts().OpenMPTargetDebug =
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..d630d825062b7 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -304,17 +304,17 @@ class HostEvalInfo {
 ///
 /// The main entry point visit() will call visitDirective() for the OpenMP
 /// directive associated to the initial given evaluation based on whether it is
-/// part of the initialDirectives() set. A nested OpenMP evaluation might
-/// optionally be also visited by the pattern recursively if it meets all of the
-/// following conditions:
+/// part of the initial set of directives of interest. A nested OpenMP
+/// evaluation might optionally be also visited by the pattern recursively if it
+/// meets all of the following conditions:
 ///   - It is the only nested evaluation, apart from an optional END statement
 ///     associated to the same directive.
 ///   - The OpenMP directive is part of the directive set returned by the
 ///     `visitDirective` call for the parent.
 ///
-/// Subclasses define the expected pattern by implementing the
-/// initialDirectives() and visitDirective() methods, and users are expected to
-/// use visit() to trigger the complete pattern visit.
+/// Subclasses define the expected pattern by implementing the initialize() and
+/// visitDirective() methods, and users are expected to use visit() to trigger
+/// the complete pattern visit.
 class DirectivePatternVisitor {
 public:
   DirectivePatternVisitor(semantics::SemanticsContext &semaCtx)
@@ -323,13 +323,14 @@ class DirectivePatternVisitor {
 
   /// Run the pattern from the given evaluation.
   void visit(lower::pft::Evaluation &eval) {
-    directivesOfInterest = initialDirectives();
+    directivesOfInterest = initialize();
     visitEval(eval);
   }
 
 protected:
-  /// Returns the set of directives of interest at the beginning of the pattern.
-  virtual OmpDirectiveSet initialDirectives() const = 0;
+  /// Initializes the visitor and returns the set of initial directives of
+  /// interest to be matched the beginning of the pattern.
+  virtual OmpDirectiveSet initialize() = 0;
 
   /// Visits a single directive and, based on it, returns the set of other
   /// directives of interest that would be part of the pattern if nested inside.
@@ -419,11 +420,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
   virtual ~TargetSPMDVisitor() = default;
 
 protected:
-  virtual OmpDirectiveSet initialDirectives() const override {
+  virtual OmpDirectiveSet initialize() override {
+    teamsVisited = false;
     return llvm::omp::allTargetSet;
   }
 
-  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &,
+  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &eval,
                                          llvm::omp::Directive dir) override {
     using namespace llvm::omp;
 
@@ -440,7 +442,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
     case OMPD_target:
       return topTeamsSet | topParallelSet;
     case OMPD_target_teams:
+      // The 'bare' kernel type prevents the SPMD pattern from matching.
+      if (hasOmpxBareClause(eval))
+        return {};
+      [[fallthrough]];
     case OMPD_teams:
+      teamsVisited = true;
       return topDistributeSet | topLoopSet;
     case OMPD_target_parallel:
     case OMPD_parallel:
@@ -449,6 +456,19 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
       return {};
     }
   }
+
+  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
+    List<lower::omp::Clause> clauses;
+    extractClauses(eval, clauses);
+
+    return llvm::find_if(clauses, [](const Clause &clause) {
+             return std::holds_alternative<clause::OmpxBare>(clause.u);
+           }) != clauses.end();
+  }
+
+protected:
+  /// Whether a `teams` construct has been visited by visitDirective().
+  bool teamsVisited;
 };
 
 /// Populates the given HostEvalInfo structure after processing clauses for
@@ -628,20 +648,31 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
     case OMPD_target_teams_distribute_parallel_do:
     case OMPD_target_teams_distribute_parallel_do_simd:
     case OMPD_target_teams_loop:
-    case OMPD_target_parallel_do:
-    case OMPD_target_parallel_do_simd:
-    case OMPD_target_parallel_loop:
     case OMPD_teams_distribute_parallel_do:
     case OMPD_teams_distribute_parallel_do_simd:
     case OMPD_teams_loop:
     case OMPD_distribute_parallel_do:
     case OMPD_distribute_parallel_do_simd:
-    case OMPD_loop:
+      execMode = canPromoteSPMDToNoLoop(eval)
+                     ? mlir::omp::TargetExecMode::spmd_no_loop
+                     : mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_target_parallel_do:
+    case OMPD_target_parallel_do_simd:
+    case OMPD_target_parallel_loop:
     case OMPD_parallel_do:
     case OMPD_parallel_do_simd:
+    case OMPD_parallel_loop:
     case OMPD_do:
     case OMPD_do_simd:
-      execMode = canPromoteSPMDToNoLoop(eval)
+      // SPMD kernels without a `teams` construct cannot be promoted to no-loop
+      // mode.
+      execMode = mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_loop:
+      // Prevent `target parallel loop` or equivalent nests to be promoted to
+      // no-loop mode.
+      execMode = teamsVisited && canPromoteSPMDToNoLoop(eval)
                      ? mlir::omp::TargetExecMode::spmd_no_loop
                      : mlir::omp::TargetExecMode::spmd;
       return {};
@@ -674,15 +705,6 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
            }) == clauses.end();
   }
 
-  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
-    List<lower::omp::Clause> clauses;
-    extractClauses(eval, clauses);
-
-    return llvm::find_if(clauses, [](const Clause &clause) {
-             return std::holds_alternative<clause::OmpxBare>(clause.u);
-           }) != clauses.end();
-  }
-
 private:
   mlir::ModuleOp moduleOp;
   mlir::omp::TargetExecMode execMode;
diff --git a/flang/test/Driver/omp-driver-offload.f90 b/flang/test/Driver/omp-driver-offload.f90
index 1d1d4f6f1dd4b..fc9bbd7f968cb 100644
--- a/flang/test/Driver/omp-driver-offload.f90
+++ b/flang/test/Driver/omp-driver-offload.f90
@@ -77,7 +77,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-threads-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-THREADS-OVS
-! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-threads-oversubscription" {{.*}}.f90"
+! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-threads-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -89,7 +89,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-teams-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-TEAMS-OVS
-! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-teams-oversubscription" {{.*}}.f90"
+! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -153,8 +153,8 @@
 ! RUN: -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism \
 ! RUN: -fopenmp-assume-no-thread-state -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-RTL-ALL
-! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-teams-oversubscription"
-! CHECK-RTL-ALL: "-fopenmp-assume-threads-oversubscription" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
+! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-assume-threads-oversubscription"
+! CHECK-RTL-ALL: "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
 ! CHECK-RTL-ALL: {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
diff --git a/flang/test/Lower/OpenMP/target-kernel-types.f90 b/flang/test/Lower/OpenMP/target-kernel-types.f90
new file mode 100644
index 0000000000000..50d715ea61486
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-kernel-types.f90
@@ -0,0 +1,336 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-assume-threads-oversubscription -o - %s | FileCheck %s
+
+! ------------------------------------------------------------------------------
+! GENERIC KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}generic
+subroutine generic(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i, j
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target
+    call foo()
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    !$omp parallel do
+    do j = 1, n
+      call foo()
+    end do
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! BARE KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}bare
+subroutine bare(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do_simd
+subroutine target_teams_distribute_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do
+subroutine target_teams_distribute_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_loop
+subroutine target_teams_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  !$omp target teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS NOT PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do_simd
+subroutine target_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do
+subroutine target_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_loop
+subroutine target_parallel_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h
index 59b20d1b50ee1..f1f2950d2325b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h
@@ -70,14 +70,13 @@ setOffloadModuleInterfaceAttributes(ModuleOp module, OffloadModuleOpts opts) {
     offloadMod.setIsGPU(opts.OpenMPIsGPU);
     if (opts.OpenMPForceUSM)
       offloadMod.setRequires(ClauseRequires::unified_shared_memory);
-    if (opts.OpenMPIsTargetDevice) {
-      offloadMod.setFlags(
-          opts.OpenMPTargetDebug, opts.OpenMPTeamSubscription,
-          opts.OpenMPThreadSubscription, opts.OpenMPNoThreadState,
-          opts.OpenMPNoNestedParallelism, opts.OpenMPVersion, opts.NoGPULib);
-      if (!opts.OMPHostIRFile.empty())
-        offloadMod.setHostIRFilePath(opts.OMPHostIRFile);
-    }
+    offloadMod.setFlags(opts.OpenMPTargetDebug, opts.OpenMPTeamSubscription,
+                        opts.OpenMPThreadSubscription, opts.OpenMPNoThreadState,
+                        opts.OpenMPNoNestedParallelism, opts.OpenMPVersion,
+                        opts.NoGPULib);
+    if (opts.OpenMPIsTargetDevice && !opts.OMPHostIRFile.empty())
+      offloadMod.setHostIRFilePath(opts.OMPHostIRFile);
+
     auto strTriples = llvm::to_vector(
         llvm::map_range(opts.OMPTargetTriples, [](llvm::Triple triple) {
           return triple.normalize();
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index d29d221dfed14..41d3c547ca618 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2664,9 +2664,18 @@ LogicalResult TargetOp::verifyRegions() {
   if (numNestedTeams > 1)
     return emitError("target containing multiple 'omp.teams' nested ops");
 
-  if (getKernelType() == TargetExecMode::bare && numNestedTeams == 0)
-    return emitOpError()
-           << "bare kernel must contain a nested 'omp.teams' operation";
+  if (numNestedTeams == 0) {
+    switch (getKernelType()) {
+    case TargetExecMode::bare:
+      return emitOpError()
+             << "bare kernel must contain a nested 'omp.teams' operation";
+    case TargetExecMode::spmd_no_loop:
+      return emitOpError() << "spmd_no_loop kernel must contain a nested "
+                              "'omp.teams' operation";
+    default:
+      break;
+    }
+  }
 
   Operation *capturedOp =
       cast<ComposableOpInterface>(getOperation()).findCapturedOp();
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f1569f857b3b7..eb34cabdf20f8 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7784,20 +7784,25 @@ convertOmpDistribute(Operation &opInst, llvm::IRBuilderBase &builder,
   return success();
 }
 
-/// Lowers the FlagsAttr which is applied to the module on the device
-/// pass when offloading, this attribute contains OpenMP RTL globals that can
-/// be passed as flags to the frontend, otherwise they are set to default
+/// Lowers the FlagsAttr which is applied to the module when offloading. This
+/// attribute contains OpenMP RTL globals that can be passed as flags to the
+/// frontend, otherwise they are set to default
 static LogicalResult
 convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
                  LLVM::ModuleTranslation &moduleTranslation) {
-  if (!cast<mlir::ModuleOp>(op))
-    return failure();
+  auto offloadMod = dyn_cast<omp::OffloadModuleInterface>(op);
+  if (!offloadMod)
+    return op->emitOpError() << "omp flags attached to non offload module op";
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
   ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
                               attribute.getOpenmpDeviceVersion());
 
+  // The flags below are only intended to be emitted for GPU offload targets.
+  if (!offloadMod.getIsGPU())
+    return success();
+
   if (attribute.getNoGpuLib())
     return success();
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 7f04d73b0ddbd..2db7f953fa3bb 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3196,9 +3196,30 @@ func.func @omp_target_spmd() {
 
 // -----
 
-func.func @omp_target_no_loop() {
+func.func @omp_target_no_loop1(%n : i32) {
+  // expected-error @below {{op spmd_no_loop kernel must contain a nested 'omp.teams' operation}}
+  omp.target kernel_type(spmd_no_loop) host_eval(%n -> %arg0 : i32) {
+    omp.parallel {
+      omp.wsloop {
+        omp.loop_nest (%iv) : i32 = (%arg0) to (%arg0) step (%arg0) {
+          omp.yield
+        }
+      }
+      omp.terminator
+    } {omp.combined}
+    omp.terminator
+  } {omp.combined}
+  return
+}
+
+// -----
+
+func.func @omp_target_no_loop2() {
   // expected-error @below {{op SPMD kernel must capture an 'omp.loop_nest' operation}}
   omp.target kernel_type(spmd_no_loop) {
+    omp.teams {
+      omp.terminator
+    }
     omp.terminator
   }
   return
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 7621dd87eb152..9a1874379d2fb 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -918,16 +918,23 @@ func.func @omp_target(%if_cond : i1, %device : si32,  %num_threads : i32, %devic
       omp.terminator
     } {omp.combined}
     // CHECK: omp.target kernel_type(spmd_no_loop) host_eval({{.*}}) {
+    // CHECK: omp.teams {
     // CHECK: omp.parallel {
+    // CHECK: omp.distribute {
     // CHECK: omp.wsloop {
     // CHECK: omp.loop_nest
     omp.target kernel_type(spmd_no_loop) host_eval(%n -> %arg0 : i32) {
-      omp.parallel {
-        omp.wsloop {
-          omp.loop_nest (%iv) : i32 = (%arg0) to (%arg0) step (%arg0) {
-            omp.yield
-          }
-        }
+      omp.teams {
+        omp.parallel {
+          omp.distribute {
+            omp.wsloop {
+              omp.loop_nest (%iv) : i32 = (%arg0) to (%arg0) step (%arg0) {
+                omp.yield
+              }
+            } {omp.composite}
+          } {omp.composite}
+          omp.terminator
+        } {omp.composite}
         omp.terminator
       } {omp.combined}
       omp.terminator
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 1890eaa6a3f0b..790c0dfe9b12e 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3547,7 +3547,7 @@ llvm.func @omp_opaque_pointers(%arg0 : !llvm.ptr, %arg1: !llvm.ptr, %expr: i32)
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 1
 module attributes {omp.flags = #omp.flags<debug_kind = 1, assume_teams_oversubscription = true, 
                                           assume_threads_oversubscription = true, assume_no_thread_state = true, 
-                                          assume_no_nested_parallelism = true>} {}
+                                          assume_no_nested_parallelism = true>, omp.is_gpu = true} {}
 // -----
 
 // CHECK: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0
@@ -3556,7 +3556,7 @@ module attributes {omp.flags = #omp.flags<debug_kind = 1, assume_teams_oversubsc
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
-module attributes {omp.flags = #omp.flags<>} {}
+module attributes {omp.flags = #omp.flags<>, omp.is_gpu = true} {}
 
 // -----
 
@@ -3566,7 +3566,7 @@ module attributes {omp.flags = #omp.flags<>} {}
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 51}
-module attributes {omp.flags = #omp.flags<openmp_device_version = 51>} {}
+module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_gpu = true} {}
 
 // -----
 
@@ -3577,7 +3577,7 @@ module attributes {omp.flags = #omp.flags<openmp_device_version = 51>} {}
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 50}
-module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>} {}
+module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>, omp.is_gpu = true} {}
 
 // -----
 
@@ -3593,7 +3593,7 @@ module attributes {omp.version = #omp.version<version = 51>} {}
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 module attributes {omp.flags = #omp.flags<debug_kind = 0, assume_teams_oversubscription = false, 
                                           assume_threads_oversubscription = false, assume_no_thread_state = false, 
-                                          assume_no_nested_parallelism = false>} {}
+                                          assume_no_nested_parallelism = false>, omp.is_gpu = true} {}
 
 // -----
 
@@ -3602,7 +3602,7 @@ module attributes {omp.flags = #omp.flags<debug_kind = 0, assume_teams_oversubsc
 // CHECK: @__omp_rtl_assume_threads_oversubscription = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 1
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
-module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true>} {}
+module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true>, omp.is_gpu = true} {}
 
 // -----
 
@@ -3612,7 +3612,7 @@ module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true,
 // CHECK-NOT: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 1
 // CHECK-NOT: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 module attributes {omp.flags = #omp.flags<assume_teams_oversubscription = true, assume_no_thread_state = true,
-                                          no_gpu_lib=true>} {}
+                                          no_gpu_lib=true>, omp.is_gpu = true} {}
 
 // -----
 
diff --git a/offload/test/offloading/fortran/target-no-loop.f90 b/offload/test/offloading/fortran/target-no-loop.f90
index bd174806ff449..93b4a1bba7bc6 100644
--- a/offload/test/offloading/fortran/target-no-loop.f90
+++ b/offload/test/offloading/fortran/target-no-loop.f90
@@ -88,9 +88,7 @@ end program main
 ! CHECK:  info: #Args: 3 Teams x Thrds:   3x  16 {{.*}}
 ! CHECK:  PluginInterface device {{[0-9]+}} info: Launching kernel {{.*}} SPMD-No-Loop mode
 ! CHECK:  info: #Args: 3 Teams x Thrds:   64x  16 {{.*}}
-! TODO: This kernel is currently incorrectly tagged as no-loop. This is a known issue in the
-!       Flang frontend that currently does not result in any known runtime problems.
-! CHECK:  PluginInterface device {{[0-9]+}} info: Launching kernel {{.*}} SPMD-No-Loop mode
+! CHECK:  PluginInterface device {{[0-9]+}} info: Launching kernel {{.*}} SPMD mode
 ! CHECK:  info: #Args: 3 Teams x Thrds:   1x  16
 ! CHECK:  PluginInterface device {{[0-9]+}} info: Launching kernel {{.*}} Generic mode
 ! CHECK:  info: #Args: 3 Teams x Thrds:   16x  16 {{.*}}


From llvmlistbot at llvm.org  Wed Jul  1 04:33:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:33:49 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a44fb1d.6f9aa017.2b48cf.ee7e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-offload

Author: Sergio Afonso (skatrak)

<details>
<summary>Changes</summary>

This patch modifies the handling of the `-fopenmp-assume-teams-oversubscription` and `-fopenmp-assume-threads-oversubscription` compiler options in Flang and the corresponding `omp.flags` MLIR attribute to have this information present on both host and target device MLIR modules. This is necessary to ensure SPMD to SPMD-No-Loop kernel promotion happens uniformly and no divergence between host and device representations of the same target region is introduced.

The identification of the kernel type in Flang lowering is updated to address some broken edge cases. In particular, no SPMD to SPMD-No-Loop transformation is performed on `target parallel do [simd]` kernels. A split `target + parallel loop` is now marked SPMD, which was a case missed previously, and the trip count on `target teams ompx_bare` kernels isn't host evaluated even if removing the `ompx_bare` clause would result in a valid SPMD kernel.

---

Patch is 31.59 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/206953.diff


12 Files Affected:

- (modified) clang/lib/Driver/ToolChains/Flang.cpp (+13-11) 
- (modified) flang/lib/Frontend/CompilerInvocation.cpp (+10-12) 
- (modified) flang/lib/Lower/OpenMP/OpenMP.cpp (+47-25) 
- (modified) flang/test/Driver/omp-driver-offload.f90 (+4-4) 
- (added) flang/test/Lower/OpenMP/target-kernel-types.f90 (+336) 
- (modified) mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h (+7-8) 
- (modified) mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp (+12-3) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+10-5) 
- (modified) mlir/test/Dialect/OpenMP/invalid.mlir (+22-1) 
- (modified) mlir/test/Dialect/OpenMP/ops.mlir (+13-6) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+7-7) 
- (modified) offload/test/offloading/fortran/target-no-loop.f90 (+1-3) 


``````````diff
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index e4aa1c9456d9a..ee67c570fb96c 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -739,6 +739,19 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     }
   }
 
+  // When in OpenMP offloading mode, forward assumptions information about
+  // thread and team counts in the target device. The host needs to know about
+  // this to prevent the SPMD to SPMD-no-loop promotion being done differently
+  // for host and device on the same target region.
+  if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
+                   options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
+  if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
+                   options::OPT_fno_openmp_assume_threads_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
+
   if (IsOpenMPDevice) {
     // -fopenmp-is-target-device is passed along to tell the frontend that it is
     // generating code for a device, so that only the relevant code is emitted.
@@ -749,17 +762,6 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     if (Args.hasFlag(options::OPT_fopenmp_target_debug,
                      options::OPT_fno_openmp_target_debug, /*Default=*/false))
       CmdArgs.push_back("-fopenmp-target-debug");
-
-    // When in OpenMP offloading mode, forward assumptions information about
-    // thread and team counts in the device.
-    if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
-                     options::OPT_fno_openmp_assume_teams_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
-    if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
-                     options::OPT_fno_openmp_assume_threads_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_thread_state))
       CmdArgs.push_back("-fopenmp-assume-no-thread-state");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_nested_parallelism))
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 003b815bef678..8e13fc9f05324 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -1301,6 +1301,16 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
     }
   }
 
+  if (args.hasFlag(clang::options::OPT_fopenmp_assume_teams_oversubscription,
+                   clang::options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    res.getLangOpts().OpenMPTeamSubscription = true;
+  if (args.hasFlag(
+          clang::options::OPT_fopenmp_assume_threads_oversubscription,
+          clang::options::OPT_fno_openmp_assume_threads_oversubscription,
+          /*Default=*/false))
+    res.getLangOpts().OpenMPThreadSubscription = true;
+
   if (args.hasArg(clang::options::OPT_fopenmp_force_usm)) {
     res.getLangOpts().OpenMPForceUSM = 1;
   }
@@ -1317,24 +1327,12 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
             << res.getLangOpts().OMPHostIRFile;
     }
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_teams_oversubscription,
-            clang::options::OPT_fno_openmp_assume_teams_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPTeamSubscription = true;
-
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_thread_state))
       res.getLangOpts().OpenMPNoThreadState = 1;
 
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_nested_parallelism))
       res.getLangOpts().OpenMPNoNestedParallelism = 1;
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_threads_oversubscription,
-            clang::options::OPT_fno_openmp_assume_threads_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPThreadSubscription = true;
-
     if ((args.hasArg(clang::options::OPT_fopenmp_target_debug) ||
          args.hasArg(clang::options::OPT_fopenmp_target_debug_EQ))) {
       res.getLangOpts().OpenMPTargetDebug =
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..d630d825062b7 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -304,17 +304,17 @@ class HostEvalInfo {
 ///
 /// The main entry point visit() will call visitDirective() for the OpenMP
 /// directive associated to the initial given evaluation based on whether it is
-/// part of the initialDirectives() set. A nested OpenMP evaluation might
-/// optionally be also visited by the pattern recursively if it meets all of the
-/// following conditions:
+/// part of the initial set of directives of interest. A nested OpenMP
+/// evaluation might optionally be also visited by the pattern recursively if it
+/// meets all of the following conditions:
 ///   - It is the only nested evaluation, apart from an optional END statement
 ///     associated to the same directive.
 ///   - The OpenMP directive is part of the directive set returned by the
 ///     `visitDirective` call for the parent.
 ///
-/// Subclasses define the expected pattern by implementing the
-/// initialDirectives() and visitDirective() methods, and users are expected to
-/// use visit() to trigger the complete pattern visit.
+/// Subclasses define the expected pattern by implementing the initialize() and
+/// visitDirective() methods, and users are expected to use visit() to trigger
+/// the complete pattern visit.
 class DirectivePatternVisitor {
 public:
   DirectivePatternVisitor(semantics::SemanticsContext &semaCtx)
@@ -323,13 +323,14 @@ class DirectivePatternVisitor {
 
   /// Run the pattern from the given evaluation.
   void visit(lower::pft::Evaluation &eval) {
-    directivesOfInterest = initialDirectives();
+    directivesOfInterest = initialize();
     visitEval(eval);
   }
 
 protected:
-  /// Returns the set of directives of interest at the beginning of the pattern.
-  virtual OmpDirectiveSet initialDirectives() const = 0;
+  /// Initializes the visitor and returns the set of initial directives of
+  /// interest to be matched the beginning of the pattern.
+  virtual OmpDirectiveSet initialize() = 0;
 
   /// Visits a single directive and, based on it, returns the set of other
   /// directives of interest that would be part of the pattern if nested inside.
@@ -419,11 +420,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
   virtual ~TargetSPMDVisitor() = default;
 
 protected:
-  virtual OmpDirectiveSet initialDirectives() const override {
+  virtual OmpDirectiveSet initialize() override {
+    teamsVisited = false;
     return llvm::omp::allTargetSet;
   }
 
-  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &,
+  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &eval,
                                          llvm::omp::Directive dir) override {
     using namespace llvm::omp;
 
@@ -440,7 +442,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
     case OMPD_target:
       return topTeamsSet | topParallelSet;
     case OMPD_target_teams:
+      // The 'bare' kernel type prevents the SPMD pattern from matching.
+      if (hasOmpxBareClause(eval))
+        return {};
+      [[fallthrough]];
     case OMPD_teams:
+      teamsVisited = true;
       return topDistributeSet | topLoopSet;
     case OMPD_target_parallel:
     case OMPD_parallel:
@@ -449,6 +456,19 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
       return {};
     }
   }
+
+  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
+    List<lower::omp::Clause> clauses;
+    extractClauses(eval, clauses);
+
+    return llvm::find_if(clauses, [](const Clause &clause) {
+             return std::holds_alternative<clause::OmpxBare>(clause.u);
+           }) != clauses.end();
+  }
+
+protected:
+  /// Whether a `teams` construct has been visited by visitDirective().
+  bool teamsVisited;
 };
 
 /// Populates the given HostEvalInfo structure after processing clauses for
@@ -628,20 +648,31 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
     case OMPD_target_teams_distribute_parallel_do:
     case OMPD_target_teams_distribute_parallel_do_simd:
     case OMPD_target_teams_loop:
-    case OMPD_target_parallel_do:
-    case OMPD_target_parallel_do_simd:
-    case OMPD_target_parallel_loop:
     case OMPD_teams_distribute_parallel_do:
     case OMPD_teams_distribute_parallel_do_simd:
     case OMPD_teams_loop:
     case OMPD_distribute_parallel_do:
     case OMPD_distribute_parallel_do_simd:
-    case OMPD_loop:
+      execMode = canPromoteSPMDToNoLoop(eval)
+                     ? mlir::omp::TargetExecMode::spmd_no_loop
+                     : mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_target_parallel_do:
+    case OMPD_target_parallel_do_simd:
+    case OMPD_target_parallel_loop:
     case OMPD_parallel_do:
     case OMPD_parallel_do_simd:
+    case OMPD_parallel_loop:
     case OMPD_do:
     case OMPD_do_simd:
-      execMode = canPromoteSPMDToNoLoop(eval)
+      // SPMD kernels without a `teams` construct cannot be promoted to no-loop
+      // mode.
+      execMode = mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_loop:
+      // Prevent `target parallel loop` or equivalent nests to be promoted to
+      // no-loop mode.
+      execMode = teamsVisited && canPromoteSPMDToNoLoop(eval)
                      ? mlir::omp::TargetExecMode::spmd_no_loop
                      : mlir::omp::TargetExecMode::spmd;
       return {};
@@ -674,15 +705,6 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
            }) == clauses.end();
   }
 
-  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
-    List<lower::omp::Clause> clauses;
-    extractClauses(eval, clauses);
-
-    return llvm::find_if(clauses, [](const Clause &clause) {
-             return std::holds_alternative<clause::OmpxBare>(clause.u);
-           }) != clauses.end();
-  }
-
 private:
   mlir::ModuleOp moduleOp;
   mlir::omp::TargetExecMode execMode;
diff --git a/flang/test/Driver/omp-driver-offload.f90 b/flang/test/Driver/omp-driver-offload.f90
index 1d1d4f6f1dd4b..fc9bbd7f968cb 100644
--- a/flang/test/Driver/omp-driver-offload.f90
+++ b/flang/test/Driver/omp-driver-offload.f90
@@ -77,7 +77,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-threads-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-THREADS-OVS
-! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-threads-oversubscription" {{.*}}.f90"
+! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-threads-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -89,7 +89,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-teams-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-TEAMS-OVS
-! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-teams-oversubscription" {{.*}}.f90"
+! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -153,8 +153,8 @@
 ! RUN: -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism \
 ! RUN: -fopenmp-assume-no-thread-state -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-RTL-ALL
-! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-teams-oversubscription"
-! CHECK-RTL-ALL: "-fopenmp-assume-threads-oversubscription" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
+! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-assume-threads-oversubscription"
+! CHECK-RTL-ALL: "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
 ! CHECK-RTL-ALL: {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
diff --git a/flang/test/Lower/OpenMP/target-kernel-types.f90 b/flang/test/Lower/OpenMP/target-kernel-types.f90
new file mode 100644
index 0000000000000..50d715ea61486
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-kernel-types.f90
@@ -0,0 +1,336 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-assume-threads-oversubscription -o - %s | FileCheck %s
+
+! ------------------------------------------------------------------------------
+! GENERIC KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}generic
+subroutine generic(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i, j
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target
+    call foo()
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    !$omp parallel do
+    do j = 1, n
+      call foo()
+    end do
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! BARE KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}bare
+subroutine bare(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do_simd
+subroutine target_teams_distribute_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do
+subroutine target_teams_distribute_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_loop
+subroutine target_teams_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  !$omp target teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS NOT PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do_simd
+subroutine target_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do
+subroutine target_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_loop
+subroutine target_parallel_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Wed Jul  1 04:33:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:33:49 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a44fb1d.283fd794.1f10ec.b320@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-openmp

Author: Sergio Afonso (skatrak)

<details>
<summary>Changes</summary>

This patch modifies the handling of the `-fopenmp-assume-teams-oversubscription` and `-fopenmp-assume-threads-oversubscription` compiler options in Flang and the corresponding `omp.flags` MLIR attribute to have this information present on both host and target device MLIR modules. This is necessary to ensure SPMD to SPMD-No-Loop kernel promotion happens uniformly and no divergence between host and device representations of the same target region is introduced.

The identification of the kernel type in Flang lowering is updated to address some broken edge cases. In particular, no SPMD to SPMD-No-Loop transformation is performed on `target parallel do [simd]` kernels. A split `target + parallel loop` is now marked SPMD, which was a case missed previously, and the trip count on `target teams ompx_bare` kernels isn't host evaluated even if removing the `ompx_bare` clause would result in a valid SPMD kernel.

---

Patch is 31.59 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/206953.diff


12 Files Affected:

- (modified) clang/lib/Driver/ToolChains/Flang.cpp (+13-11) 
- (modified) flang/lib/Frontend/CompilerInvocation.cpp (+10-12) 
- (modified) flang/lib/Lower/OpenMP/OpenMP.cpp (+47-25) 
- (modified) flang/test/Driver/omp-driver-offload.f90 (+4-4) 
- (added) flang/test/Lower/OpenMP/target-kernel-types.f90 (+336) 
- (modified) mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h (+7-8) 
- (modified) mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp (+12-3) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+10-5) 
- (modified) mlir/test/Dialect/OpenMP/invalid.mlir (+22-1) 
- (modified) mlir/test/Dialect/OpenMP/ops.mlir (+13-6) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+7-7) 
- (modified) offload/test/offloading/fortran/target-no-loop.f90 (+1-3) 


``````````diff
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index e4aa1c9456d9a..ee67c570fb96c 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -739,6 +739,19 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     }
   }
 
+  // When in OpenMP offloading mode, forward assumptions information about
+  // thread and team counts in the target device. The host needs to know about
+  // this to prevent the SPMD to SPMD-no-loop promotion being done differently
+  // for host and device on the same target region.
+  if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
+                   options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
+  if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
+                   options::OPT_fno_openmp_assume_threads_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
+
   if (IsOpenMPDevice) {
     // -fopenmp-is-target-device is passed along to tell the frontend that it is
     // generating code for a device, so that only the relevant code is emitted.
@@ -749,17 +762,6 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     if (Args.hasFlag(options::OPT_fopenmp_target_debug,
                      options::OPT_fno_openmp_target_debug, /*Default=*/false))
       CmdArgs.push_back("-fopenmp-target-debug");
-
-    // When in OpenMP offloading mode, forward assumptions information about
-    // thread and team counts in the device.
-    if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
-                     options::OPT_fno_openmp_assume_teams_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
-    if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
-                     options::OPT_fno_openmp_assume_threads_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_thread_state))
       CmdArgs.push_back("-fopenmp-assume-no-thread-state");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_nested_parallelism))
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 003b815bef678..8e13fc9f05324 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -1301,6 +1301,16 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
     }
   }
 
+  if (args.hasFlag(clang::options::OPT_fopenmp_assume_teams_oversubscription,
+                   clang::options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    res.getLangOpts().OpenMPTeamSubscription = true;
+  if (args.hasFlag(
+          clang::options::OPT_fopenmp_assume_threads_oversubscription,
+          clang::options::OPT_fno_openmp_assume_threads_oversubscription,
+          /*Default=*/false))
+    res.getLangOpts().OpenMPThreadSubscription = true;
+
   if (args.hasArg(clang::options::OPT_fopenmp_force_usm)) {
     res.getLangOpts().OpenMPForceUSM = 1;
   }
@@ -1317,24 +1327,12 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
             << res.getLangOpts().OMPHostIRFile;
     }
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_teams_oversubscription,
-            clang::options::OPT_fno_openmp_assume_teams_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPTeamSubscription = true;
-
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_thread_state))
       res.getLangOpts().OpenMPNoThreadState = 1;
 
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_nested_parallelism))
       res.getLangOpts().OpenMPNoNestedParallelism = 1;
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_threads_oversubscription,
-            clang::options::OPT_fno_openmp_assume_threads_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPThreadSubscription = true;
-
     if ((args.hasArg(clang::options::OPT_fopenmp_target_debug) ||
          args.hasArg(clang::options::OPT_fopenmp_target_debug_EQ))) {
       res.getLangOpts().OpenMPTargetDebug =
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..d630d825062b7 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -304,17 +304,17 @@ class HostEvalInfo {
 ///
 /// The main entry point visit() will call visitDirective() for the OpenMP
 /// directive associated to the initial given evaluation based on whether it is
-/// part of the initialDirectives() set. A nested OpenMP evaluation might
-/// optionally be also visited by the pattern recursively if it meets all of the
-/// following conditions:
+/// part of the initial set of directives of interest. A nested OpenMP
+/// evaluation might optionally be also visited by the pattern recursively if it
+/// meets all of the following conditions:
 ///   - It is the only nested evaluation, apart from an optional END statement
 ///     associated to the same directive.
 ///   - The OpenMP directive is part of the directive set returned by the
 ///     `visitDirective` call for the parent.
 ///
-/// Subclasses define the expected pattern by implementing the
-/// initialDirectives() and visitDirective() methods, and users are expected to
-/// use visit() to trigger the complete pattern visit.
+/// Subclasses define the expected pattern by implementing the initialize() and
+/// visitDirective() methods, and users are expected to use visit() to trigger
+/// the complete pattern visit.
 class DirectivePatternVisitor {
 public:
   DirectivePatternVisitor(semantics::SemanticsContext &semaCtx)
@@ -323,13 +323,14 @@ class DirectivePatternVisitor {
 
   /// Run the pattern from the given evaluation.
   void visit(lower::pft::Evaluation &eval) {
-    directivesOfInterest = initialDirectives();
+    directivesOfInterest = initialize();
     visitEval(eval);
   }
 
 protected:
-  /// Returns the set of directives of interest at the beginning of the pattern.
-  virtual OmpDirectiveSet initialDirectives() const = 0;
+  /// Initializes the visitor and returns the set of initial directives of
+  /// interest to be matched the beginning of the pattern.
+  virtual OmpDirectiveSet initialize() = 0;
 
   /// Visits a single directive and, based on it, returns the set of other
   /// directives of interest that would be part of the pattern if nested inside.
@@ -419,11 +420,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
   virtual ~TargetSPMDVisitor() = default;
 
 protected:
-  virtual OmpDirectiveSet initialDirectives() const override {
+  virtual OmpDirectiveSet initialize() override {
+    teamsVisited = false;
     return llvm::omp::allTargetSet;
   }
 
-  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &,
+  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &eval,
                                          llvm::omp::Directive dir) override {
     using namespace llvm::omp;
 
@@ -440,7 +442,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
     case OMPD_target:
       return topTeamsSet | topParallelSet;
     case OMPD_target_teams:
+      // The 'bare' kernel type prevents the SPMD pattern from matching.
+      if (hasOmpxBareClause(eval))
+        return {};
+      [[fallthrough]];
     case OMPD_teams:
+      teamsVisited = true;
       return topDistributeSet | topLoopSet;
     case OMPD_target_parallel:
     case OMPD_parallel:
@@ -449,6 +456,19 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
       return {};
     }
   }
+
+  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
+    List<lower::omp::Clause> clauses;
+    extractClauses(eval, clauses);
+
+    return llvm::find_if(clauses, [](const Clause &clause) {
+             return std::holds_alternative<clause::OmpxBare>(clause.u);
+           }) != clauses.end();
+  }
+
+protected:
+  /// Whether a `teams` construct has been visited by visitDirective().
+  bool teamsVisited;
 };
 
 /// Populates the given HostEvalInfo structure after processing clauses for
@@ -628,20 +648,31 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
     case OMPD_target_teams_distribute_parallel_do:
     case OMPD_target_teams_distribute_parallel_do_simd:
     case OMPD_target_teams_loop:
-    case OMPD_target_parallel_do:
-    case OMPD_target_parallel_do_simd:
-    case OMPD_target_parallel_loop:
     case OMPD_teams_distribute_parallel_do:
     case OMPD_teams_distribute_parallel_do_simd:
     case OMPD_teams_loop:
     case OMPD_distribute_parallel_do:
     case OMPD_distribute_parallel_do_simd:
-    case OMPD_loop:
+      execMode = canPromoteSPMDToNoLoop(eval)
+                     ? mlir::omp::TargetExecMode::spmd_no_loop
+                     : mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_target_parallel_do:
+    case OMPD_target_parallel_do_simd:
+    case OMPD_target_parallel_loop:
     case OMPD_parallel_do:
     case OMPD_parallel_do_simd:
+    case OMPD_parallel_loop:
     case OMPD_do:
     case OMPD_do_simd:
-      execMode = canPromoteSPMDToNoLoop(eval)
+      // SPMD kernels without a `teams` construct cannot be promoted to no-loop
+      // mode.
+      execMode = mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_loop:
+      // Prevent `target parallel loop` or equivalent nests to be promoted to
+      // no-loop mode.
+      execMode = teamsVisited && canPromoteSPMDToNoLoop(eval)
                      ? mlir::omp::TargetExecMode::spmd_no_loop
                      : mlir::omp::TargetExecMode::spmd;
       return {};
@@ -674,15 +705,6 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
            }) == clauses.end();
   }
 
-  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
-    List<lower::omp::Clause> clauses;
-    extractClauses(eval, clauses);
-
-    return llvm::find_if(clauses, [](const Clause &clause) {
-             return std::holds_alternative<clause::OmpxBare>(clause.u);
-           }) != clauses.end();
-  }
-
 private:
   mlir::ModuleOp moduleOp;
   mlir::omp::TargetExecMode execMode;
diff --git a/flang/test/Driver/omp-driver-offload.f90 b/flang/test/Driver/omp-driver-offload.f90
index 1d1d4f6f1dd4b..fc9bbd7f968cb 100644
--- a/flang/test/Driver/omp-driver-offload.f90
+++ b/flang/test/Driver/omp-driver-offload.f90
@@ -77,7 +77,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-threads-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-THREADS-OVS
-! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-threads-oversubscription" {{.*}}.f90"
+! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-threads-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -89,7 +89,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-teams-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-TEAMS-OVS
-! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-teams-oversubscription" {{.*}}.f90"
+! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -153,8 +153,8 @@
 ! RUN: -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism \
 ! RUN: -fopenmp-assume-no-thread-state -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-RTL-ALL
-! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-teams-oversubscription"
-! CHECK-RTL-ALL: "-fopenmp-assume-threads-oversubscription" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
+! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-assume-threads-oversubscription"
+! CHECK-RTL-ALL: "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
 ! CHECK-RTL-ALL: {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
diff --git a/flang/test/Lower/OpenMP/target-kernel-types.f90 b/flang/test/Lower/OpenMP/target-kernel-types.f90
new file mode 100644
index 0000000000000..50d715ea61486
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-kernel-types.f90
@@ -0,0 +1,336 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-assume-threads-oversubscription -o - %s | FileCheck %s
+
+! ------------------------------------------------------------------------------
+! GENERIC KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}generic
+subroutine generic(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i, j
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target
+    call foo()
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    !$omp parallel do
+    do j = 1, n
+      call foo()
+    end do
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! BARE KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}bare
+subroutine bare(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do_simd
+subroutine target_teams_distribute_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do
+subroutine target_teams_distribute_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_loop
+subroutine target_teams_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  !$omp target teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS NOT PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do_simd
+subroutine target_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do
+subroutine target_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_loop
+subroutine target_parallel_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Wed Jul  1 04:33:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 04:33:50 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a44fb1e.21153faf.3c5219.55d3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openmp

Author: Sergio Afonso (skatrak)

<details>
<summary>Changes</summary>

This patch modifies the handling of the `-fopenmp-assume-teams-oversubscription` and `-fopenmp-assume-threads-oversubscription` compiler options in Flang and the corresponding `omp.flags` MLIR attribute to have this information present on both host and target device MLIR modules. This is necessary to ensure SPMD to SPMD-No-Loop kernel promotion happens uniformly and no divergence between host and device representations of the same target region is introduced.

The identification of the kernel type in Flang lowering is updated to address some broken edge cases. In particular, no SPMD to SPMD-No-Loop transformation is performed on `target parallel do [simd]` kernels. A split `target + parallel loop` is now marked SPMD, which was a case missed previously, and the trip count on `target teams ompx_bare` kernels isn't host evaluated even if removing the `ompx_bare` clause would result in a valid SPMD kernel.

---

Patch is 31.59 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/206953.diff


12 Files Affected:

- (modified) clang/lib/Driver/ToolChains/Flang.cpp (+13-11) 
- (modified) flang/lib/Frontend/CompilerInvocation.cpp (+10-12) 
- (modified) flang/lib/Lower/OpenMP/OpenMP.cpp (+47-25) 
- (modified) flang/test/Driver/omp-driver-offload.f90 (+4-4) 
- (added) flang/test/Lower/OpenMP/target-kernel-types.f90 (+336) 
- (modified) mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h (+7-8) 
- (modified) mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp (+12-3) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+10-5) 
- (modified) mlir/test/Dialect/OpenMP/invalid.mlir (+22-1) 
- (modified) mlir/test/Dialect/OpenMP/ops.mlir (+13-6) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+7-7) 
- (modified) offload/test/offloading/fortran/target-no-loop.f90 (+1-3) 


``````````diff
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index e4aa1c9456d9a..ee67c570fb96c 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -739,6 +739,19 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     }
   }
 
+  // When in OpenMP offloading mode, forward assumptions information about
+  // thread and team counts in the target device. The host needs to know about
+  // this to prevent the SPMD to SPMD-no-loop promotion being done differently
+  // for host and device on the same target region.
+  if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
+                   options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
+  if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
+                   options::OPT_fno_openmp_assume_threads_oversubscription,
+                   /*Default=*/false))
+    CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
+
   if (IsOpenMPDevice) {
     // -fopenmp-is-target-device is passed along to tell the frontend that it is
     // generating code for a device, so that only the relevant code is emitted.
@@ -749,17 +762,6 @@ void Flang::addOffloadOptions(Compilation &C, const InputInfoList &Inputs,
     if (Args.hasFlag(options::OPT_fopenmp_target_debug,
                      options::OPT_fno_openmp_target_debug, /*Default=*/false))
       CmdArgs.push_back("-fopenmp-target-debug");
-
-    // When in OpenMP offloading mode, forward assumptions information about
-    // thread and team counts in the device.
-    if (Args.hasFlag(options::OPT_fopenmp_assume_teams_oversubscription,
-                     options::OPT_fno_openmp_assume_teams_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-teams-oversubscription");
-    if (Args.hasFlag(options::OPT_fopenmp_assume_threads_oversubscription,
-                     options::OPT_fno_openmp_assume_threads_oversubscription,
-                     /*Default=*/false))
-      CmdArgs.push_back("-fopenmp-assume-threads-oversubscription");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_thread_state))
       CmdArgs.push_back("-fopenmp-assume-no-thread-state");
     if (Args.hasArg(options::OPT_fopenmp_assume_no_nested_parallelism))
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index 003b815bef678..8e13fc9f05324 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -1301,6 +1301,16 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
     }
   }
 
+  if (args.hasFlag(clang::options::OPT_fopenmp_assume_teams_oversubscription,
+                   clang::options::OPT_fno_openmp_assume_teams_oversubscription,
+                   /*Default=*/false))
+    res.getLangOpts().OpenMPTeamSubscription = true;
+  if (args.hasFlag(
+          clang::options::OPT_fopenmp_assume_threads_oversubscription,
+          clang::options::OPT_fno_openmp_assume_threads_oversubscription,
+          /*Default=*/false))
+    res.getLangOpts().OpenMPThreadSubscription = true;
+
   if (args.hasArg(clang::options::OPT_fopenmp_force_usm)) {
     res.getLangOpts().OpenMPForceUSM = 1;
   }
@@ -1317,24 +1327,12 @@ static bool parseOpenMPArgs(CompilerInvocation &res, llvm::opt::ArgList &args,
             << res.getLangOpts().OMPHostIRFile;
     }
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_teams_oversubscription,
-            clang::options::OPT_fno_openmp_assume_teams_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPTeamSubscription = true;
-
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_thread_state))
       res.getLangOpts().OpenMPNoThreadState = 1;
 
     if (args.hasArg(clang::options::OPT_fopenmp_assume_no_nested_parallelism))
       res.getLangOpts().OpenMPNoNestedParallelism = 1;
 
-    if (args.hasFlag(
-            clang::options::OPT_fopenmp_assume_threads_oversubscription,
-            clang::options::OPT_fno_openmp_assume_threads_oversubscription,
-            /*Default=*/false))
-      res.getLangOpts().OpenMPThreadSubscription = true;
-
     if ((args.hasArg(clang::options::OPT_fopenmp_target_debug) ||
          args.hasArg(clang::options::OPT_fopenmp_target_debug_EQ))) {
       res.getLangOpts().OpenMPTargetDebug =
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..d630d825062b7 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -304,17 +304,17 @@ class HostEvalInfo {
 ///
 /// The main entry point visit() will call visitDirective() for the OpenMP
 /// directive associated to the initial given evaluation based on whether it is
-/// part of the initialDirectives() set. A nested OpenMP evaluation might
-/// optionally be also visited by the pattern recursively if it meets all of the
-/// following conditions:
+/// part of the initial set of directives of interest. A nested OpenMP
+/// evaluation might optionally be also visited by the pattern recursively if it
+/// meets all of the following conditions:
 ///   - It is the only nested evaluation, apart from an optional END statement
 ///     associated to the same directive.
 ///   - The OpenMP directive is part of the directive set returned by the
 ///     `visitDirective` call for the parent.
 ///
-/// Subclasses define the expected pattern by implementing the
-/// initialDirectives() and visitDirective() methods, and users are expected to
-/// use visit() to trigger the complete pattern visit.
+/// Subclasses define the expected pattern by implementing the initialize() and
+/// visitDirective() methods, and users are expected to use visit() to trigger
+/// the complete pattern visit.
 class DirectivePatternVisitor {
 public:
   DirectivePatternVisitor(semantics::SemanticsContext &semaCtx)
@@ -323,13 +323,14 @@ class DirectivePatternVisitor {
 
   /// Run the pattern from the given evaluation.
   void visit(lower::pft::Evaluation &eval) {
-    directivesOfInterest = initialDirectives();
+    directivesOfInterest = initialize();
     visitEval(eval);
   }
 
 protected:
-  /// Returns the set of directives of interest at the beginning of the pattern.
-  virtual OmpDirectiveSet initialDirectives() const = 0;
+  /// Initializes the visitor and returns the set of initial directives of
+  /// interest to be matched the beginning of the pattern.
+  virtual OmpDirectiveSet initialize() = 0;
 
   /// Visits a single directive and, based on it, returns the set of other
   /// directives of interest that would be part of the pattern if nested inside.
@@ -419,11 +420,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
   virtual ~TargetSPMDVisitor() = default;
 
 protected:
-  virtual OmpDirectiveSet initialDirectives() const override {
+  virtual OmpDirectiveSet initialize() override {
+    teamsVisited = false;
     return llvm::omp::allTargetSet;
   }
 
-  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &,
+  virtual OmpDirectiveSet visitDirective(lower::pft::Evaluation &eval,
                                          llvm::omp::Directive dir) override {
     using namespace llvm::omp;
 
@@ -440,7 +442,12 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
     case OMPD_target:
       return topTeamsSet | topParallelSet;
     case OMPD_target_teams:
+      // The 'bare' kernel type prevents the SPMD pattern from matching.
+      if (hasOmpxBareClause(eval))
+        return {};
+      [[fallthrough]];
     case OMPD_teams:
+      teamsVisited = true;
       return topDistributeSet | topLoopSet;
     case OMPD_target_parallel:
     case OMPD_parallel:
@@ -449,6 +456,19 @@ class TargetSPMDVisitor : public DirectivePatternVisitor {
       return {};
     }
   }
+
+  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
+    List<lower::omp::Clause> clauses;
+    extractClauses(eval, clauses);
+
+    return llvm::find_if(clauses, [](const Clause &clause) {
+             return std::holds_alternative<clause::OmpxBare>(clause.u);
+           }) != clauses.end();
+  }
+
+protected:
+  /// Whether a `teams` construct has been visited by visitDirective().
+  bool teamsVisited;
 };
 
 /// Populates the given HostEvalInfo structure after processing clauses for
@@ -628,20 +648,31 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
     case OMPD_target_teams_distribute_parallel_do:
     case OMPD_target_teams_distribute_parallel_do_simd:
     case OMPD_target_teams_loop:
-    case OMPD_target_parallel_do:
-    case OMPD_target_parallel_do_simd:
-    case OMPD_target_parallel_loop:
     case OMPD_teams_distribute_parallel_do:
     case OMPD_teams_distribute_parallel_do_simd:
     case OMPD_teams_loop:
     case OMPD_distribute_parallel_do:
     case OMPD_distribute_parallel_do_simd:
-    case OMPD_loop:
+      execMode = canPromoteSPMDToNoLoop(eval)
+                     ? mlir::omp::TargetExecMode::spmd_no_loop
+                     : mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_target_parallel_do:
+    case OMPD_target_parallel_do_simd:
+    case OMPD_target_parallel_loop:
     case OMPD_parallel_do:
     case OMPD_parallel_do_simd:
+    case OMPD_parallel_loop:
     case OMPD_do:
     case OMPD_do_simd:
-      execMode = canPromoteSPMDToNoLoop(eval)
+      // SPMD kernels without a `teams` construct cannot be promoted to no-loop
+      // mode.
+      execMode = mlir::omp::TargetExecMode::spmd;
+      return {};
+    case OMPD_loop:
+      // Prevent `target parallel loop` or equivalent nests to be promoted to
+      // no-loop mode.
+      execMode = teamsVisited && canPromoteSPMDToNoLoop(eval)
                      ? mlir::omp::TargetExecMode::spmd_no_loop
                      : mlir::omp::TargetExecMode::spmd;
       return {};
@@ -674,15 +705,6 @@ class KernelTypeVisitor : protected TargetSPMDVisitor {
            }) == clauses.end();
   }
 
-  bool hasOmpxBareClause(lower::pft::Evaluation &eval) {
-    List<lower::omp::Clause> clauses;
-    extractClauses(eval, clauses);
-
-    return llvm::find_if(clauses, [](const Clause &clause) {
-             return std::holds_alternative<clause::OmpxBare>(clause.u);
-           }) != clauses.end();
-  }
-
 private:
   mlir::ModuleOp moduleOp;
   mlir::omp::TargetExecMode execMode;
diff --git a/flang/test/Driver/omp-driver-offload.f90 b/flang/test/Driver/omp-driver-offload.f90
index 1d1d4f6f1dd4b..fc9bbd7f968cb 100644
--- a/flang/test/Driver/omp-driver-offload.f90
+++ b/flang/test/Driver/omp-driver-offload.f90
@@ -77,7 +77,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-threads-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-THREADS-OVS
-! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-threads-oversubscription" {{.*}}.f90"
+! CHECK-THREADS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-threads-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -89,7 +89,7 @@
 ! RUN: -fopenmp-targets=nvptx64-nvidia-cuda \
 ! RUN: -fopenmp-assume-teams-oversubscription -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-TEAMS-OVS
-! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-assume-teams-oversubscription" {{.*}}.f90"
+! CHECK-TEAMS-OVS: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-is-target-device" {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
 ! RUN: -fopenmp --offload-arch=gfx90a \
@@ -153,8 +153,8 @@
 ! RUN: -fopenmp-assume-teams-oversubscription -fopenmp-assume-no-nested-parallelism \
 ! RUN: -fopenmp-assume-no-thread-state -nogpulib \
 ! RUN: | FileCheck %s --check-prefixes=CHECK-RTL-ALL
-! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-teams-oversubscription"
-! CHECK-RTL-ALL: "-fopenmp-assume-threads-oversubscription" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
+! CHECK-RTL-ALL: "{{[^"]*}}flang{{[^"]*}}" "-fc1" {{.*}} "-fopenmp" {{.*}} "-fopenmp-assume-teams-oversubscription" "-fopenmp-assume-threads-oversubscription"
+! CHECK-RTL-ALL: "-fopenmp-is-target-device" "-fopenmp-target-debug" "-fopenmp-assume-no-thread-state" "-fopenmp-assume-no-nested-parallelism"
 ! CHECK-RTL-ALL: {{.*}}.f90"
 
 ! RUN: %flang -### %s -o %t 2>&1 \
diff --git a/flang/test/Lower/OpenMP/target-kernel-types.f90 b/flang/test/Lower/OpenMP/target-kernel-types.f90
new file mode 100644
index 0000000000000..50d715ea61486
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-kernel-types.f90
@@ -0,0 +1,336 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-assume-teams-oversubscription -fopenmp-assume-threads-oversubscription -o - %s | FileCheck %s
+
+! ------------------------------------------------------------------------------
+! GENERIC KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}generic
+subroutine generic(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i, j
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target
+    call foo()
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams distribute
+  do i = 1, n
+    !$omp parallel do
+    do j = 1, n
+      call foo()
+    end do
+  end do
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(generic)
+  !$omp target teams
+  !$omp parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! BARE KERNELS
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}bare
+subroutine bare(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+    call foo()
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(bare)
+  !$omp target teams ompx_bare num_teams(1) thread_limit(1)
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do_simd
+subroutine target_teams_distribute_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_distribute_parallel_do
+subroutine target_teams_distribute_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp distribute parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_teams_loop
+subroutine target_teams_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target teams loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  !$omp target teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target teams
+
+  ! CHECK: omp.target kernel_type(spmd_no_loop)
+  !$omp target
+  !$omp teams
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end teams
+  !$omp end target
+end subroutine
+
+! ------------------------------------------------------------------------------
+! SPMD KERNELS NOT PROMOTABLE TO NO-LOOP MODE
+! ------------------------------------------------------------------------------
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do_simd
+subroutine target_parallel_do_simd(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do simd
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_do
+subroutine target_parallel_do(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel do
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel
+  !$omp do
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end parallel
+  !$omp end target
+end subroutine
+
+! CHECK-LABEL: func.func @{{.*}}target_parallel_loop
+subroutine target_parallel_loop(n)
+  implicit none
+  integer, intent(in) :: n
+  integer :: i
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel loop
+  do i = 1, n
+    call foo()
+  end do
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target
+  !$omp parallel loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target
+
+  ! CHECK: omp.target kernel_type(spmd)
+  !$omp target parallel
+  !$omp loop
+  do i = 1, n
+    call foo()
+  end do
+  !$omp end target parallel
+...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Wed Jul  1 05:03:30 2026
From: llvmlistbot at llvm.org (Dominik Adamski)
Date: Wed, 01 Jul 2026 05:03:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a450212.63d39acb.be069.d604@mx.google.com>


https://github.com/DominikAdamski approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Wed Jul  1 05:07:35 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Wed, 01 Jul 2026 05:07:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a450307.86bccb3e.1e739d.0c58@mx.google.com>


https://github.com/zero9178 auto_merge_enabled https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 05:14:24 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:14:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4504a0.b7ecdeec.190680.9213@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 1/9] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 2/9] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 3/9] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 4/9] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 5/9] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 6/9] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 7/9] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 8/9] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From 53f1e7ccb9bbda237101269d3a0fba71ee31d571 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 9/9] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  92 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 155 insertions(+), 232 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..30d1af44f043d 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
+  // For non-scalar strided memrefs, only support result strides identical to
+  // the identity strides of the source. This enables direct indexing into the
+  // same source dimensions, without linearization.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  assert(resType.hasStaticShape() && "expected static shape");
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   // Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
-
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast"
+           "source indices");
+    // Non-scalar copies: additionally assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -254,13 +254,17 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Wed Jul  1 05:16:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:16:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 53107fb - [MLIR] Prevent CF to SCF transform
 from erasing blocks still in use (#206310)
Message-ID: <6a450527.b7ecdeec.190680.a02e@mx.google.com>


Author: Martin P
Date: 2026-07-01T12:16:34Z
New Revision: 53107fb19af49724ae810bf8fb0c8356759124ed

URL: https://github.com/llvm/llvm-project/commit/53107fb19af49724ae810bf8fb0c8356759124ed
DIFF: https://github.com/llvm/llvm-project/commit/53107fb19af49724ae810bf8fb0c8356759124ed.diff

LOG: [MLIR] Prevent CF to SCF transform from erasing blocks still in use (#206310)

While merging blocks, `transformToStructuredCFBranches` may try to erase
blocks that still have uses. This may happen when the block has
self-loop or when it has 1 successor, but the successor has multiple
predecessors. This probably happens only with unreachable blocks in the
function.

Note that running a DCE pass before CF to SCF transform would remove the
triggering assert in the lit test.

Fixes #206086.

Added: 
    mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir

Modified: 
    mlir/include/mlir/Transforms/CFGToSCF.h
    mlir/lib/Transforms/Utils/CFGToSCF.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Transforms/CFGToSCF.h b/mlir/include/mlir/Transforms/CFGToSCF.h
index 517e116f3c58e..2ca019ddfe23d 100644
--- a/mlir/include/mlir/Transforms/CFGToSCF.h
+++ b/mlir/include/mlir/Transforms/CFGToSCF.h
@@ -149,9 +149,10 @@ class CFGToSCFInterface {
 /// Otherwise a single control flow graph operation branching to one block
 /// per return-like operation kind remains.
 ///
-/// The transformation currently requires that all control flow graph operations
-/// have no side effects, implement the BranchOpInterface and does not have any
-/// operation produced successor operands.
+/// The transformation currently requires that the region has no unreachable
+/// blocks and that all control flow graph operations have no side effects,
+/// implement the BranchOpInterface and does not have any operation produced
+/// successor operands.
 /// Returns failure if any of the preconditions are violated or if any of the
 /// methods of `interface` failed. The IR is left in an unspecified state.
 ///

diff  --git a/mlir/lib/Transforms/Utils/CFGToSCF.cpp b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
index 0cc6bb43ddd38..2cda4a26543bd 100644
--- a/mlir/lib/Transforms/Utils/CFGToSCF.cpp
+++ b/mlir/lib/Transforms/Utils/CFGToSCF.cpp
@@ -1234,8 +1234,13 @@ static ReturnLikeExitCombiner createSingleExitBlocksForReturnLike(
 /// Returns failure if any precondition is violated.
 static LogicalResult
 checkTransformationPreconditions(Region &region, CFGToSCFInterface &interface) {
+  llvm::df_iterator_default_set<Block *, 16> reachable;
+  // Find all blocks reachable from the entry.
+  for (Block *block : depth_first_ext(&region.front(), reachable))
+    (void)block;
+
   for (Block &block : region.getBlocks())
-    if (block.hasNoPredecessors() && !block.isEntryBlock())
+    if (!reachable.contains(&block))
       return block.front().emitOpError(
           "transformation does not support unreachable blocks");
 

diff  --git a/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
new file mode 100644
index 0000000000000..4317e5a5a8102
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToSCF/unreachable-blocks.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt --lift-cf-to-scf -verify-diagnostics -split-input-file %s
+
+// Verify that --lift-cf-to-scf does not crash when it encounters an
+// unreachable blocks (dead code).  Instead it should emit a clean error.
+// See: https://github.com/llvm/llvm-project/issues/206086
+
+module {
+  func.func @single_return_statement(%arg0: i32) -> i32 {
+    return %arg0 : i32
+  ^bb1:  // pred: ^bb1
+    // expected-error at below {{'arith.constant' op transformation does not support unreachable blocks}}
+    %true = arith.constant true
+    cf.cond_br %true, ^bb2, ^bb1
+  ^bb2:  // pred: ^bb1
+    return %arg0 : i32
+  }
+}
+


        

From llvmlistbot at llvm.org  Wed Jul  1 05:16:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Wed, 01 Jul 2026 05:16:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a450529.fa459026.36f57e.f6d6@mx.google.com>


https://github.com/zero9178 closed https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 05:17:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:17:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Prevent CF to SCF transform from
 erasing blocks still in use (PR #206310)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206310@github.com>
Message-ID: <6a45053f.7201836d.e1e02.2a9b@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@pechy Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/206310

From llvmlistbot at llvm.org  Wed Jul  1 05:24:17 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:24:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4506f1.ddad13b0.18b57b.791f@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 1/9] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 2/9] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 3/9] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 4/9] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 5/9] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 6/9] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 7/9] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 8/9] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 9/9] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Wed Jul  1 05:27:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:27:37 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a4507b9.c84e7546.5f94c.76d1@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/202315

>From 216cbd2e710ce3c725bb51c5606bea4ba84a33a3 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Sat, 6 Jun 2026 14:38:54 -0500
Subject: [PATCH 1/8] Support for "atomic compare capture".

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  88 ++++---
 .../Interfaces/AtomicInterfaces.td            |   9 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  10 +
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |   6 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 216 ++++++++++++++----
 5 files changed, 250 insertions(+), 79 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 9d711b92bb520..5e02d3fb10dbb 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -558,16 +558,45 @@ void Fortran::lower::omp::lowerAtomic(
   int action1 = analysis.op1.what & analysis.Action;
   memOrder = makeValidForAction(memOrder, action0, action1, version);
 
+  // --- Shared capture scaffolding ---
+  mlir::Operation *captureOp = nullptr;
+  fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
+  fir::FirOpBuilder::InsertPoint atomicAt, postAt;
+
+  if (construct.IsCapture()) {
+    assert(action0 != analysis.None && action1 != analysis.None &&
+           "Expexcing two actions");
+    (void)action0;
+    (void)action1;
+    captureOp = mlir::omp::AtomicCaptureOp::create(
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+    // Set the non-atomic insertion point to before the atomic.capture.
+    preAt = getInsertionPointBefore(captureOp);
+
+    mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
+    builder.setInsertionPointToEnd(block);
+    // Set the atomic insertion point to before the terminator inside
+    // atomic.capture.
+    mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
+    atomicAt = getInsertionPointBefore(term);
+    postAt = getInsertionPointAfter(captureOp);
+    hint = nullptr;
+    memOrder = std::nullopt;
+  }
+
   if (auto *cond = get(analysis.cond)) {
     // atomic compare: if (x == e) x = d
     // e : expecteVal
     // d : desiredVal
 
-    // Check for compound clauses (fail, capture, weak) that are not yet
+    // Restore insertion point so pre-processing code (e.g. computing
+    // expectedVal) is emitted before the capture op, not after the terminator.
+    builder.restoreInsertionPoint(preAt);
+
+    // Check for compound clauses (fail, weak) that are not yet
     // supported with atomic compare.
     if (llvm::any_of(clauses, [](const omp::Clause &clause) {
           return clause.id == llvm::omp::Clause::OMPC_fail ||
-                 clause.id == llvm::omp::Clause::OMPC_capture ||
                  clause.id == llvm::omp::Clause::OMPC_weak;
         })) {
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
@@ -616,6 +645,17 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // If this is a compare+capture, generate the read op first.
+    if (construct.IsCapture()) {
+      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
+             "Expected a read operation for compare capture");
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
+
     mlir::UnitAttr weakAttr = nullptr;
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
@@ -679,34 +719,9 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
-
     // END omp atomic compare
   } else {
-    mlir::Operation *captureOp = nullptr;
-    fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
-    fir::FirOpBuilder::InsertPoint atomicAt, postAt;
-
-    if (construct.IsCapture()) {
-      // Capturing operation.
-      assert(action0 != analysis.None && action1 != analysis.None &&
-             "Expexcing two actions");
-      (void)action0;
-      (void)action1;
-      captureOp = mlir::omp::AtomicCaptureOp::create(
-          builder, loc, hint, makeMemOrderAttr(converter, memOrder));
-      // Set the non-atomic insertion point to before the atomic.capture.
-      preAt = getInsertionPointBefore(captureOp);
-
-      mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
-      builder.setInsertionPointToEnd(block);
-      // Set the atomic insertion point to before the terminator inside
-      // atomic.capture.
-      mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
-      atomicAt = getInsertionPointBefore(term);
-      postAt = getInsertionPointAfter(captureOp);
-      hint = nullptr;
-      memOrder = std::nullopt;
-    } else {
+    if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
              "Expexcing single action");
@@ -729,16 +744,13 @@ void Fortran::lower::omp::lowerAtomic(
           *get(analysis.op1.assign), hint, memOrder, preAt, atomicAt, postAt);
     }
 
-    if (construct.IsCapture()) {
-      // If this is a capture operation, the first/second ops will be inside
-      // of it. Set the insertion point to past the capture op itself.
-      builder.restoreInsertionPoint(postAt);
-    } else {
-      if (secondOp) {
-        builder.setInsertionPointAfter(secondOp);
-      } else {
-        builder.setInsertionPointAfter(firstOp);
-      }
+    if (!construct.IsCapture()) {
+      builder.setInsertionPointAfter(secondOp ? secondOp : firstOp);
     }
   }
+
+  // Shared capture cleanup.
+  if (construct.IsCapture()) {
+    builder.restoreInsertionPoint(postAt);
+  }
 }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index abb21705b3c1c..8c9015f05bb72 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -289,10 +289,12 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
+        auto secondCompareStmt = dyn_cast<AtomicCompareOpInterface>(secondOp);
 
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
-              (firstReadStmt && secondWriteStmt)))
+              (firstReadStmt && secondWriteStmt) ||
+              (firstReadStmt && secondCompareStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -310,6 +312,11 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
           return firstReadStmt.emitError()
                 << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstReadStmt && secondCompareStmt &&
+            firstReadStmt.getX() != secondCompareStmt.getX())
+          return firstReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "second operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..1241abc10298f 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1928,6 +1928,12 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
         omp.atomic.write ...
         omp.terminator
       }
+
+      omp.atomic.capture {
+        omp.atomic.read ...
+        omp.atomic.compare ...
+        omp.terminator
+      }
     ```
   }] # clausesDescription;
 
@@ -1946,6 +1952,10 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     /// Returns the `atomic.update` operation inside the region, if any.
     /// Otherwise, it returns nullptr.
     AtomicUpdateOp getAtomicUpdateOp();
+
+    /// Returns the `atomic.compare` operation inside the region, if any.
+    /// Otherwise, it returns nullptr.
+    AtomicCompareOp getAtomicCompareOp();
   }] # clausesExtraClassDeclaration;
 
   let hasRegionVerifier = 1;
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..0eafd0a267b97 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4615,6 +4615,12 @@ AtomicUpdateOp AtomicCaptureOp::getAtomicUpdateOp() {
   return dyn_cast<AtomicUpdateOp>(getSecondOp());
 }
 
+AtomicCompareOp AtomicCaptureOp::getAtomicCompareOp() {
+  if (auto op = dyn_cast<AtomicCompareOp>(getFirstOp()))
+    return op;
+  return dyn_cast<AtomicCompareOp>(getSecondOp());
+}
+
 LogicalResult AtomicCaptureOp::verify() {
   return verifySynchronizationHint(*this, getHint());
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 07fa92bdabe50..4928b288adb65 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4795,6 +4795,43 @@ convertOmpAtomicUpdate(omp::AtomicUpdateOp &opInst,
   return success();
 }
 
+/// Helper to extract the OMPAtomicCompareOp from an integer comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::ICmpPredicate::eq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::ICmpPredicate::slt:
+  case LLVM::ICmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::ICmpPredicate::sgt:
+  case LLVM::ICmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
+/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::FCmpPredicate::oeq:
+  case LLVM::FCmpPredicate::ueq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::FCmpPredicate::olt:
+  case LLVM::FCmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::FCmpPredicate::ogt:
+  case LLVM::FCmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -4803,13 +4840,149 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   if (failed(checkImplementationStatus(*atomicCaptureOp)))
     return failure();
 
+  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
+  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
+  omp::AtomicCompareOp atomicCompareOp = atomicCaptureOp.getAtomicCompareOp();
+
+  // If the capture contains an atomic.compare, delegate to
+  // createAtomicCompare with the capture variable (V) set.
+  if (atomicCompareOp) {
+    omp::AtomicReadOp atomicReadOp = atomicCaptureOp.getAtomicReadOp();
+    assert(atomicReadOp && "expected atomic.read in capture+compare");
+
+    Region &region = atomicCompareOp.getRegion();
+    Block &block = region.front();
+
+    llvm::Type *llvmXElementType =
+        moduleTranslation.convertType(block.getArgument(0).getType());
+    llvm::Value *llvmX = moduleTranslation.lookupValue(atomicCompareOp.getX());
+    llvm::Value *llvmV = moduleTranslation.lookupValue(atomicReadOp.getV());
+
+    bool isSigned = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicX = {
+        llvmX, llvmXElementType, isSigned, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicV = {
+        llvmV, llvmXElementType, /*isSigned=*/false, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicR = {nullptr, nullptr, false,
+                                                        false};
+
+    llvm::AtomicOrdering atomicOrdering =
+        convertAtomicOrdering(atomicCaptureOp.getMemoryOrder());
+
+    // Pre-translate non-pattern operations inside the compare region.
+    auto isAtomicComparePatternOp = [](Operation &op) {
+      return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
+                       LLVM::OrOp>(op);
+    };
+    for (Operation &op : block.without_terminator()) {
+      if (isAtomicComparePatternOp(op))
+        continue;
+      bool allOperandsMapped =
+          llvm::all_of(op.getOperands(), [&](mlir::Value v) {
+            return moduleTranslation.lookupValue(v) != nullptr;
+          });
+      if (!allOperandsMapped)
+        continue;
+      if (failed(moduleTranslation.convertOperation(op, builder)))
+        return atomicCompareOp.emitError(
+            "failed to translate operation inside atomic compare region");
+    }
+
+    auto materializeValue = [&](mlir::Value val) -> llvm::Value * {
+      if (llvm::Value *existing = moduleTranslation.lookupValue(val))
+        return existing;
+      if (auto loadOp = val.getDefiningOp<LLVM::LoadOp>()) {
+        if (loadOp->getParentRegion() == &region) {
+          llvm::Value *loadAddr =
+              moduleTranslation.lookupValue(loadOp.getAddr());
+          if (!loadAddr)
+            return nullptr;
+          llvm::Type *loadType =
+              moduleTranslation.convertType(loadOp.getResult().getType());
+          return builder.CreateLoad(loadType, loadAddr);
+        }
+      }
+      return nullptr;
+    };
+
+    // Extract comparison predicate, eVal, and dVal from the region.
+    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    llvm::Value *eVal = nullptr;
+    llvm::Value *dVal = nullptr;
+    bool isXBinopExpr = false;
+
+    for (Operation &op : block.getOperations()) {
+      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
+        auto maybeOp =
+            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+        isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                    pred == LLVM::ICmpPredicate::sgt ||
+                    pred == LLVM::ICmpPredicate::sle ||
+                    pred == LLVM::ICmpPredicate::sge);
+
+        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
+        auto maybeOp =
+            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        if (!dVal)
+          dVal = materializeValue(selectOp.getTrueValue());
+      }
+    }
+
+    if (!eVal)
+      return atomicCompareOp.emitError(
+          "failed to extract expected value (e) from atomic compare region");
+    if (!dVal) {
+      auto yieldOp = cast<omp::YieldOp>(block.getTerminator());
+      if (yieldOp.getResults().empty())
+        return atomicCompareOp.emitError(
+            "failed to extract desired value (d) from atomic compare region");
+      dVal = materializeValue(yieldOp.getResults()[0]);
+    }
+
+    llvmAtomicX.IsSigned = isSigned;
+
+    llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    bool isPostfixUpdate = true;
+
+    bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, /*IsFailOnly=*/false);
+    ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
+
+    if (failed(handleError(afterIP, *atomicCaptureOp)))
+      return failure();
+
+    builder.restoreIP(*afterIP);
+    return success();
+  }
+
   mlir::Value mlirExpr;
   bool isXBinopExpr = false, isPostfixUpdate = false;
   llvm::AtomicRMWInst::BinOp binop = llvm::AtomicRMWInst::BinOp::BAD_BINOP;
 
-  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
-  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
-
   assert((atomicUpdateOp || atomicWriteOp) &&
          "internal op must be an atomic.update or atomic.write op");
 
@@ -4896,43 +5069,6 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   return success();
 }
 
-/// Helper to extract the OMPAtomicCompareOp from an integer comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::ICmpPredicate::eq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::ICmpPredicate::slt:
-  case LLVM::ICmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::ICmpPredicate::sgt:
-  case LLVM::ICmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
-/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::FCmpPredicate::oeq:
-  case LLVM::FCmpPredicate::ueq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::FCmpPredicate::olt:
-  case LLVM::FCmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::FCmpPredicate::ogt:
-  case LLVM::FCmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
 /// Converts an omp.atomic.compare operation to LLVM IR.
 ///
 ///      if (x == e) x = d

>From 7d3293944598e8090a64cdec2ac3c791a64b73f1 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Mon, 8 Jun 2026 05:12:22 -0500
Subject: [PATCH 2/8] Adding tests for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     | 32 +++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 56 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 29 ++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  9 +--
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 41 ++++++++++++++
 6 files changed, 211 insertions(+), 4 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 249fb0dd8fa64..650f64b80af12 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,3 +260,35 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
+! Integer equality compare+capture: cmpxchg + store old value
+!CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
+! Compare+capture with clause order reversed: capture compare
+!CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_capture_compare_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic capture compare
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ac70edbed4e60..752a221aa538d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -161,3 +161,59 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
   !$omp atomic compare weak
   if (x .eq. e) x = d
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_eq(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_gt(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           } {{.*}}weak{{.*}}
+! CHECK:         }
+subroutine atomic_compare_capture_int_gt(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+    v = x
+    if (x > e) x = d
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 4f3cf0eac0338..dc0cc1a62f6c2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -192,6 +192,26 @@ program main
       i = j
    end if
 
+!COMPARE CAPTURE
+!$omp atomic compare capture
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare weak
+   k = i
+   if (i < j) then
+      i = k
+   end if
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -296,6 +316,15 @@ end program main
 !CHECK: !$OMP ATOMIC WEAK COMPARE
 !CHECK: !$OMP ATOMIC COMPARE SEQ_CST WEAK
 
+!COMPARE CAPTURE
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 17f99a8c86d52..c5a07a7dc6cb2 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4963,13 +4963,14 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
     bool isPostfixUpdate = true;
+    bool isWeak = atomicCompareOp.getWeak();
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, /*IsFailOnly=*/false);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            /*IsFailOnly=*/false, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index acf6879906e74..289ec60551cb7 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2047,6 +2047,54 @@ func.func @omp_atomic_compare(%x: memref<i32>, %e: i32, %d: i32) {
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_weak
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: } {weak}
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi sgt, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 1890eaa6a3f0b..6ce57b304bfd9 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2795,6 +2795,47 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From f84987dea68a1dedc6b61bf0c9f5a2e24a0a1b40 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 9 Jun 2026 04:42:48 -0500
Subject: [PATCH 3/8] Minor changes incorporating feedback.

---
 flang/lib/Lower/OpenMP/Atomic.cpp                             | 4 ++--
 .../Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index c0f97aada637d..aa497b838e5b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -567,7 +567,7 @@ void Fortran::lower::omp::lowerAtomic(
 
   if (construct.IsCapture()) {
     assert(action0 != analysis.None && action1 != analysis.None &&
-           "Expexcing two actions");
+           "Expecting two actions");
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
@@ -730,7 +730,7 @@ void Fortran::lower::omp::lowerAtomic(
     if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
-             "Expexcing single action");
+             "Expecting single action");
       assert(!(analysis.op0.what & analysis.Condition));
       postAt = atomicAt = preAt;
     }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 8c9015f05bb72..dc5b04126b757 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -315,7 +315,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be updated in "
+                << "captured variable in atomic.read must be compared/updated in "
                     "second operation";
 
         return mlir::success();

>From ca24a147920d24c188127dbbe37ca62bcaeab792 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 14:18:17 -0500
Subject: [PATCH 4/8] Handling test cases mentionend in OpenMP spec.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 61 ++++++++++++++++---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 ++-
 .../Interfaces/AtomicInterfaces.td            | 11 +++-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  2 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 12 ++--
 5 files changed, 74 insertions(+), 19 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index aa497b838e5b6..209871abd52b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -571,7 +571,8 @@ void Fortran::lower::omp::lowerAtomic(
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
-        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder),
+        /*fail_only=*/nullptr);
     // Set the non-atomic insertion point to before the atomic.capture.
     preAt = getInsertionPointBefore(captureOp);
 
@@ -646,15 +647,45 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
-    // If this is a compare+capture, generate the read op first.
+    // If this is a compare+capture, determine the ordering of ops.
+    // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
+    // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
+    // Pattern 3 (fail-only): if (x == e) x = d; else v = x → read on failure
+    bool isPostfixCapture = false;
+    bool isFailOnly = false;
+    const evaluate::Assignment *readAssign = nullptr;
     if (construct.IsCapture()) {
-      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
-             "Expected a read operation for compare capture");
-      mlir::Operation *readOp = genAtomicRead(
-          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
-          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
-      assert(readOp && "Should have created an atomic read operation");
-      builder.setInsertionPointAfter(readOp);
+      // Determine which op is the read and check for fail-only (IfFalse).
+      int readWhat = 0;
+      if (analysis.op0.what & analysis.Read) {
+        readAssign = get(analysis.op0.assign);
+        readWhat = analysis.op0.what;
+      } else if (analysis.op1.what & analysis.Read) {
+        readAssign = get(analysis.op1.assign);
+        readWhat = analysis.op1.what;
+        isPostfixCapture = true;
+      }
+      assert(readAssign && "Expected a read assignment for compare capture");
+
+      // Check if the read is conditioned on comparison failure (else branch).
+      if (readWhat & analysis.IfFalse)
+        isFailOnly = true;
+
+      if (!isPostfixCapture && !isFailOnly) {
+        // Pattern 1 (prefix): read is first, generate it before compare.
+        mlir::Operation *readOp =
+            genAtomicRead(converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+                          *readAssign, hint, memOrder, preAt, atomicAt, postAt);
+        assert(readOp && "Should have created an atomic read operation");
+        builder.setInsertionPointAfter(readOp);
+      } else {
+        // Pattern 2 (postfix) or 3 (fail-only): compare first, read after.
+        builder.restoreInsertionPoint(atomicAt);
+      }
+
+      // Set the fail_only attribute on the capture op.
+      if (isFailOnly && captureOp)
+        mlir::cast<mlir::omp::AtomicCaptureOp>(captureOp).setFailOnly(true);
     }
 
     mlir::UnitAttr weakAttr = nullptr;
@@ -725,6 +756,18 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
+
+    // Pattern 2 (postfix) or 3 (fail-only): compare first, read second.
+    // Generate read after compare for postfix or fail-only patterns.
+    if (construct.IsCapture() && (isPostfixCapture || isFailOnly)) {
+      fir::FirOpBuilder::InsertPoint afterCompareAt =
+          builder.saveInsertionPoint();
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom, *readAssign, hint,
+          memOrder, preAt, afterCompareAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
     // END omp atomic compare
   } else {
     if (!construct.IsCapture()) {
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 92eb7de0d882f..b2ccc05ad6770 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11321,7 +11321,9 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11377,7 +11379,8 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index dc5b04126b757..5b1527bcb84ec 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -286,6 +286,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto &secondOp = *ops.getNextNode(firstOp);
         auto firstReadStmt = dyn_cast<AtomicReadOpInterface>(firstOp);
         auto firstUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(firstOp);
+        auto firstCompareStmt = dyn_cast<AtomicCompareOpInterface>(firstOp);
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
@@ -294,7 +295,8 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
               (firstReadStmt && secondWriteStmt) ||
-              (firstReadStmt && secondCompareStmt)))
+              (firstReadStmt && secondCompareStmt) ||
+              (firstCompareStmt && secondReadStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -315,8 +317,13 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be compared/updated in "
+                << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstCompareStmt && secondReadStmt &&
+            secondReadStmt.getX() != firstCompareStmt.getX())
+          return secondReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "first operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index ebee887f2afd1..69c22641df6a2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1938,6 +1938,8 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     ```
   }] # clausesDescription;
 
+  let arguments = !con(clausesArgs, (ins UnitAttr:$fail_only));
+
   // Override region definition.
   let regions = (region SizedRegion<1>:$region);
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index dff1ae30e20aa..76616c7fba27a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,15 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    bool isPostfixUpdate = true;
-    bool isWeak = atomicCompareOp.getWeak();
+    bool isFailOnly = atomicCaptureOp.getFailOnly();
+    bool isPostfixUpdate = !isFailOnly;
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(
-            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
-            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
-            /*IsFailOnly=*/false, isWeak);
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, isFailOnly);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))

>From afaf2025463e9e8cf15974747fa9e2d8b8e5fa95 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 18:02:52 -0500
Subject: [PATCH 5/8] Adding tests into flang and mlir for the grammar
 mentionend in the spec.

---
 .../Integration/OpenMP/atomic-compare.f90     | 56 ++++++++++++++++--
 flang/test/Lower/OpenMP/atomic-compare.f90    | 59 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 17 ++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 +++++-
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 58 ++++++++++++++++--
 6 files changed, 245 insertions(+), 10 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 650f64b80af12..da86f3ff6ef16 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,14 +260,17 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
-! Integer equality compare+capture: cmpxchg + store old value
+! Integer equality compare+capture (prefix): v=x; if(x==e) x=d
+! v captures old value of x
 !CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -276,14 +279,16 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
-! Compare+capture with clause order reversed: capture compare
+! Compare+capture with clause order reversed: capture compare (still prefix read)
 !CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
@@ -292,3 +297,46 @@ subroutine atomic_capture_compare_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
+! Postfix compare+capture: if (x == e) x = d; v = x
+! v captures new value (d if swapped, old x if not)
+!CHECK-LABEL: define void @atomic_compare_capture_postfix_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Fail-only compare+capture: if (x == e) x = d; else v = x
+! v is only written when the comparison fails
+!CHECK-LABEL: define void @atomic_compare_capture_fail_only_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+!CHECK: {{.*}}:
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: br label %[[EXIT2:.*]]
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 752a221aa538d..c770605954539 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -217,3 +217,62 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
     if (x > e) x = d
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_fail_only(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index dc0cc1a62f6c2..521c1bb2811e2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -211,6 +211,19 @@ program main
       i = k
    end if
 !$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   end if
+   k = i
+!$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   else
+      k = i
+   end if
+!$omp end atomic
 
 !ATOMIC
 !$omp atomic
@@ -324,6 +337,10 @@ end program main
 !CHECK: !$OMP END ATOMIC
 !CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
 !CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
 
 !ATOMIC
 !CHECK: !$OMP ATOMIC
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 76616c7fba27a..b6fc6d6dc3038 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,28 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    // In OMPIRBuilder::createAtomicCompare:
+    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
+    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
+    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
+    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
+    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
+    bool isPostfixUpdate =
+        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
     bool isFailOnly = atomicCaptureOp.getFailOnly();
-    bool isPostfixUpdate = !isFailOnly;
 
+    // For fail-only, the OMPIRBuilder's conditional store logic is under
+    // the !IsPostfixUpdate path, so force it to false.
+    if (isFailOnly)
+      isPostfixUpdate = false;
+
+    bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
         ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly);
+                                        isPostfixUpdate, isFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 87f9a6f8e4119..6b13be481e0e0 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2113,6 +2113,54 @@ func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e:
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_postfix
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_postfix(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: } {fail_only}
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  } {fail_only}
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 6ce57b304bfd9..978450d598b9b 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,10 +2798,12 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2819,10 +2821,12 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2836,6 +2840,52 @@ llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_postfix
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Postfix compare+capture: v captures new value (d if swapped, old x if not)
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[D]], i32 %[[OLD]]
+  // CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Fail-only compare+capture: v is only written when comparison fails
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+  // CHECK: [[CONT]]:
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: br label %[[EXIT2:.*]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From e8462a267b6d010ebc1e3ed881c1f38a9faca6be Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 24 Jun 2026 14:44:43 -0500
Subject: [PATCH 6/8] Simplifying the IR generator for one of the cases. This
 also fixes the failure in clang for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     |  8 +---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 43 +++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 12 ++----
 4 files changed, 38 insertions(+), 32 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index da86f3ff6ef16..4ae1b9a1f1e52 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -268,9 +268,7 @@ subroutine atomic_compare_weak(x, e, d)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -286,9 +284,7 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 51f210e189e22..f8c1999fe1b89 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11400,9 +11400,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11458,8 +11456,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 85389caa9ac42..e1fb08d780753 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5499,25 +5499,23 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    // In OMPIRBuilder::createAtomicCompare:
-    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
-    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
-    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
-    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
-    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
-    bool isPostfixUpdate =
-        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isReadFirst = isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
-    // For fail-only, the OMPIRBuilder's conditional store logic is under
-    // the !IsPostfixUpdate path, so force it to false.
-    if (isFailOnly)
-      isPostfixUpdate = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
+    // Postfix: bypass V in OMPIRBuilder, handled manually below.
+    if (isPostfixCapture && !isFailOnly)
+      llvmAtomicVForCall = {nullptr, nullptr, false, false};
+
+    // Prefix: IsPostfixUpdate=true → direct store of old value.
+    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
+    bool isPostfixUpdate = !isFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
                                         isPostfixUpdate, isFailOnly, isWeak);
@@ -5527,6 +5525,25 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
       return failure();
 
     builder.restoreIP(*afterIP);
+
+    // Postfix: v = select(success, D, old) — captures new value of x.
+    if (isPostfixCapture && !isFailOnly) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::Instruction *cmpxchgInst = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
+          cmpxchgInst = &inst;
+          break;
+        }
+      }
+      assert(cmpxchgInst && "expected cmpxchg instruction");
+      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
+      llvm::Value *successVal =
+          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+      llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
+      builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+    }
+
     return success();
   }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 978450d598b9b..377bd88894854 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,12 +2798,10 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2821,12 +2819,10 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality weak prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {

>From 101b9fa02882686a21d346ef12e541c717ca605b Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 14:00:46 -0500
Subject: [PATCH 7/8] Handling real and logical types.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 117 ++++++--
 .../Integration/OpenMP/atomic-compare.f90     |  58 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    |  83 +++++-
 flang/test/Parser/OpenMP/atomic-unparse.f90   |  12 +
 .../Interfaces/AtomicInterfaces.td            |  13 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 252 +++++++++++-------
 mlir/test/Dialect/OpenMP/ops.mlir             |  24 ++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++
 8 files changed, 470 insertions(+), 137 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 209871abd52b6..3811ea3d40bbd 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -375,6 +375,8 @@ genAtomicRead(lower::AbstractConverter &converter,
         fir::getBase(converter.genExprValue(assign.rhs, stmtCtx, &loc));
     converter.resetExprOverrides();
 
+    if (value.getType() != storeType)
+      value = builder.createConvert(loc, storeType, value);
     fir::StoreOp::create(builder, loc, value, storeAddr);
   }
   return op;
@@ -634,6 +636,45 @@ void Fortran::lower::omp::lowerAtomic(
           },
           rel->u);
     }
+
+    // Fortran uses .eqv./.neqv. for logical equality/inequality, which are
+    // LogicalOperation expressions rather than Relational expressions.
+    if (!expectedExprStorage) {
+      if (const auto *someLogical =
+              evaluate::UnwrapExpr<evaluate::Expr<evaluate::SomeLogical>>(
+                  *cond)) {
+        common::visit(
+            [&](const auto &kindLogical) {
+              using LogicalExpr = std::decay_t<decltype(kindLogical)>;
+              constexpr int K = LogicalExpr::Result::kind;
+              if (const auto *logOp =
+                      std::get_if<evaluate::LogicalOperation<K>>(
+                          &kindLogical.u)) {
+                if (logOp->logicalOperator == common::LogicalOperator::Eqv ||
+                    logOp->logicalOperator == common::LogicalOperator::Neqv) {
+                  relOpr =
+                      (logOp->logicalOperator == common::LogicalOperator::Eqv)
+                          ? common::RelationalOperator::EQ
+                          : common::RelationalOperator::NE;
+                  using Operand =
+                      typename evaluate::LogicalOperation<K>::Operand;
+                  auto leftExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->left()});
+                  auto rightExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->right()});
+                  if (evaluate::IsSameOrConvertOf(rightExpr, atom)) {
+                    expectedExprStorage = std::move(leftExpr);
+                    relOpr = reverseRelOp(relOpr);
+                  } else {
+                    expectedExprStorage = std::move(rightExpr);
+                  }
+                }
+              }
+            },
+            someLogical->u);
+      }
+    }
+
     if (!expectedExprStorage) {
       mlir::emitError(loc, "internal error: atomic compare condition is not a "
                            "recognized relational expression");
@@ -647,6 +688,20 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // For logical types, convert address and expected value to integer
+    // type here (above the atomic compare region) so that the region
+    // only contains arith.cmpi eq on integers. If done inside the region,
+    // fir.convert logical<4> -> i32 would lower to `icmp ne %val, 0`
+    // which violates the atomic compare verifier's expectations.
+    if (mlir::isa<fir::LogicalType>(elemTypeOfX)) {
+      unsigned kind = mlir::cast<fir::LogicalType>(elemTypeOfX).getFKind();
+      mlir::Type intTy = builder.getIntegerType(kind * 8);
+      mlir::Type intRefTy = builder.getRefType(intTy);
+      atomAddr = builder.createConvert(loc, intRefTy, atomAddr);
+      expectedVal = builder.createConvert(loc, intTy, expectedVal);
+      elemTypeOfX = intTy;
+    }
+
     // If this is a compare+capture, determine the ordering of ops.
     // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
     // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
@@ -694,6 +749,39 @@ void Fortran::lower::omp::lowerAtomic(
         })) {
       weakAttr = builder.getUnitAttr();
     }
+
+    // Extract write assignment (x = d) and generate desired value (d)
+    // before creating the compare region, so that d is defined outside
+    // the region and any intermediate conversions (e.g., logical-to-integer
+    // truthiness normalization) don't appear inside the atomic compare block.
+    [[maybe_unused]] int writeActionCond = 0;
+    const evaluate::Assignment *writeAssign = nullptr;
+    if (analysis.op0.what & analysis.Write) {
+      writeAssign = get(analysis.op0.assign);
+      writeActionCond = analysis.op0.what;
+    }
+    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
+      writeAssign = get(analysis.op1.assign);
+      writeActionCond = analysis.op1.what;
+    }
+    if (!writeAssign) {
+      mlir::emitError(loc,
+                      "internal error: atomic compare has no write assignment");
+      return;
+    }
+    assert((writeActionCond & analysis.IfTrue) &&
+           "atomic compare write should be conditioned on IfTrue");
+
+    // Generate desiredVal before the capture/compare region so any
+    // intermediate ops (loads, conversions) don't pollute the atomic blocks.
+    fir::FirOpBuilder::InsertPoint savedIP = builder.saveInsertionPoint();
+    builder.restoreInsertionPoint(preAt);
+    mlir::Value desiredVal =
+        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
+    if (desiredVal.getType() != elemTypeOfX)
+      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
+    builder.restoreInsertionPoint(savedIP);
+
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
         makeMemOrderAttr(converter, memOrder));
@@ -721,35 +809,6 @@ void Fortran::lower::omp::lowerAtomic(
       return;
     }
 
-    // Check for presence of Assignment (x = d) and wether it is being invoked
-    // only for IfTrue condition.
-
-    // writeActionCond is a bitmask combining the following flags:
-    //  1) the action type (Read/Write/Update)
-    //  2) condition (IfTrue/IfFalse)
-    [[maybe_unused]] int writeActionCond = 0;
-    const evaluate::Assignment *writeAssign = nullptr;
-    if (analysis.op0.what & analysis.Write) {
-      writeAssign = get(analysis.op0.assign);
-      writeActionCond = analysis.op0.what;
-    }
-    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
-      writeAssign = get(analysis.op1.assign);
-      writeActionCond = analysis.op1.what;
-    }
-    if (!writeAssign) {
-      mlir::emitError(loc,
-                      "internal error: atomic compare has no write assignment");
-      return;
-    }
-    assert((writeActionCond & analysis.IfTrue) &&
-           "atomic compare write should be conditioned on IfTrue");
-
-    // Generate new/desired value of x e.g. x = d
-    mlir::Value desiredVal =
-        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
-    if (desiredVal.getType() != elemTypeOfX)
-      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
     mlir::Value newVal = mlir::arith::SelectOp::create(builder, loc, cmpResult,
                                                        desiredVal, blockArg);
 
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 4ae1b9a1f1e52..f30349c42e33e 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -336,3 +336,61 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
   !$omp end atomic
 end
 
+! Real equality compare+capture (postfix): if(x==e) x=d; v=x
+! v captures new value of x (d if success, old if fail)
+!CHECK-LABEL: define void @atomic_compare_capture_real_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load float, ptr %[[E]]{{.*}}
+!CHECK: %[[DVAL:.*]] = load float, ptr %[[D]]{{.*}}
+!CHECK: %[[EBITS:.*]] = bitcast float %[[EVAL]] to i32
+!CHECK: %[[DBITS:.*]] = bitcast float %[[DVAL]] to i32
+!CHECK: %[[XLOAD:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+!CHECK: %[[XFP:.*]] = bitcast i32 %[[XLOAD]] to float
+! Part 1: NaN check - if either x or e is NaN, comparison fails
+!CHECK: %[[E_ISNAN:.*]] = fcmp uno float %[[EVAL]], %[[EVAL]]
+!CHECK: %[[X_ISNAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+!CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_ISNAN]], %[[X_ISNAN]]
+!CHECK: br i1 %[[EITHER_NAN]], label %[[NAN_BB:.*]], label %[[NOTNAN_BB:.*]]
+!CHECK: [[NAN_BB]]:
+!CHECK: br label %[[EXIT_BB:.*]]
+! Part 2: Both-zero check - handles +0.0 vs -0.0 (same value, different bits)
+!CHECK: [[NOTNAN_BB]]:
+!CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+!CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[EVAL]], 0.000000e+00
+!CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+!CHECK: br i1 %[[BOTHZERO]], label %[[ZERO_BB:.*]], label %[[NORMAL_BB:.*]]
+!CHECK: [[ZERO_BB]]:
+!CHECK: %[[ZERORES:.*]] = cmpxchg ptr %[[X]], i32 %[[XLOAD]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Part 3: Normal compare - standard cmpxchg with bitcasted expected value
+!CHECK: [[NORMAL_BB]]:
+!CHECK: %[[NORMRES:.*]] = cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Exit: select v = (success ? d : old_x)
+!CHECK: [[EXIT_BB]]:
+!CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+!CHECK: %[[OK:.*]] = phi i1 {{.*}}
+!CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+!CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[DVAL]], float %[[OLDFP]]
+!CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Logical .eqv. compare+capture (postfix): if(x.eqv.e) x=d; v=x
+! Logicals are compared as integers after truthiness normalization
+!CHECK-LABEL: define void @atomic_compare_capture_logical_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: cmpxchg ptr %[[X]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index c770605954539..1219d05296e4d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -10,10 +10,10 @@
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -31,10 +31,10 @@ subroutine atomic_compare_int_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -52,10 +52,10 @@ subroutine atomic_compare_float_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<complex<f32>>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<complex<f32>> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: complex<f32>):
 ! CHECK:           %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %[[EVAL]] {fastmath = #arith.fastmath<contract>} : complex<f32>
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : complex<f32>
 ! CHECK:           omp.yield(%[[SEL]] : complex<f32>)
 ! CHECK:         }
@@ -71,10 +71,10 @@ subroutine atomic_compare_complex_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi slt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -90,10 +90,10 @@ subroutine atomic_compare_int_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -109,10 +109,10 @@ subroutine atomic_compare_int_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -128,10 +128,10 @@ subroutine atomic_compare_float_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -149,10 +149,10 @@ subroutine atomic_compare_float_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         } {{.*}}weak{{.*}}
@@ -172,12 +172,12 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -200,12 +200,12 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           } {{.*}}weak{{.*}}
@@ -228,11 +228,11 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -256,11 +256,11 @@ subroutine atomic_compare_capture_postfix(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -276,3 +276,62 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
     end if
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<f32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<f32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<f32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<f32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:         %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         %[[DLOAD:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[DCONV:.*]] = fir.convert %[[DLOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DCONV]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 521c1bb2811e2..0d30cd48035ea 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -4,6 +4,7 @@ program main
    implicit none
    integer :: i, j = 10
    integer :: k
+   logical :: l1, l2, l3, l4
 !READ
 !$omp atomic read
    i = j
@@ -225,6 +226,12 @@ program main
    end if
 !$omp end atomic
 
+!COMPARE CAPTURE LOGICAL
+!$omp atomic compare capture
+   if (l1 .eqv. l2) l1 = l3
+   l4 = l1
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -342,6 +349,11 @@ end program main
 !CHECK: !$OMP ATOMIC COMPARE CAPTURE
 !CHECK: !$OMP END ATOMIC
 
+!COMPARE CAPTURE LOGICAL
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 5b1527bcb84ec..a61c43e2fd9cd 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -488,6 +488,19 @@ def AtomicCompareOpInterface : OpInterface<"AtomicCompareOpInterface"> {
               }
             }
             break;
+          } else if (llvm::is_contained(
+                         {"arith.maxsi", "arith.minsi", "arith.maxui",
+                          "arith.minui", "arith.maximumf", "arith.minimumf",
+                          "llvm.intr.smax", "llvm.intr.smin",
+                          "llvm.intr.umax", "llvm.intr.umin",
+                          "llvm.intr.maxnum", "llvm.intr.minnum"},
+                         opName)) {
+            // The canonicalizer may fold select(cmpi slt, e, x) into
+            // maxsi/minsi (or the float equivalents). After fir-to-llvm-ir
+            // these become llvm.intr.smax/smin etc. All are valid
+            // representations of atomic compare min/max operations.
+            foundComparison = true;
+            break;
           }
         }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e1fb08d780753..8e49571cd2461 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 #include "mlir/Analysis/TopologicalSortUtils.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMTypes.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
@@ -5369,6 +5370,102 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Holds the extracted comparison pattern information from an atomic compare
+/// region.
+struct AtomicComparePatternInfo {
+  llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+  llvm::Value *eVal = nullptr;
+  llvm::Value *dVal = nullptr;
+  bool isXBinopExpr = false;
+  bool isSigned = false;
+};
+
+/// Extract comparison predicate, expected value (e), desired value (d), and
+/// related flags from an atomic compare region block by scanning for
+/// icmp/fcmp/select/min/max operations.
+static LogicalResult extractAtomicComparePattern(
+    Block &block,
+    llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
+    omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  for (Operation &op : block.getOperations()) {
+    // Pre-filter: skip icmps that don't involve the block argument
+    // (e.g., truthiness extractions from logical-to-integer conversion).
+    if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op);
+        icmpOp && icmpOp.getOperand(0) != block.getArgument(0) &&
+        icmpOp.getOperand(1) != block.getArgument(0))
+      continue;
+
+    LogicalResult result =
+        llvm::TypeSwitch<Operation *, LogicalResult>(&op)
+            .Case<LLVM::ICmpOp>([&](LLVM::ICmpOp icmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+              info.isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                               pred == LLVM::ICmpPredicate::sgt ||
+                               pred == LLVM::ICmpPredicate::sle ||
+                               pred == LLVM::ICmpPredicate::sge);
+              info.isXBinopExpr =
+                  (icmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? icmpOp.getOperand(1)
+                                                       : icmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::FCmpOp>([&](LLVM::FCmpOp fcmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              info.isXBinopExpr =
+                  (fcmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? fcmpOp.getOperand(1)
+                                                       : fcmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::SelectOp>([&](LLVM::SelectOp selectOp) {
+              if (!info.dVal)
+                info.dVal = materializeValue(selectOp.getTrueValue());
+              return success();
+            })
+            .Case<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                  mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                  mlir::arith::MaximumFOp, mlir::arith::MinimumFOp,
+                  LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp, LLVM::UMinOp,
+                  LLVM::MaxNumOp, LLVM::MinNumOp>([&](Operation *) {
+              // Canonicalized min/max ops (arith or LLVM intrinsic form).
+              // max(x,e) came from slt/ult/olt -> OMPAtomicCompareOp::MIN
+              // min(x,e) came from sgt/ugt/ogt -> OMPAtomicCompareOp::MAX
+              // (OMPIRBuilder inverts: MIN->atomicrmw max, MAX->atomicrmw min)
+              bool isMax = isa<mlir::arith::MaxSIOp, mlir::arith::MaxUIOp,
+                               mlir::arith::MaximumFOp, LLVM::SMaxOp,
+                               LLVM::UMaxOp, LLVM::MaxNumOp>(op);
+              info.compareOp = isMax ? llvm::omp::OMPAtomicCompareOp::MIN
+                                     : llvm::omp::OMPAtomicCompareOp::MAX;
+              info.isSigned = isa<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                                  LLVM::SMaxOp, LLVM::SMinOp>(op);
+              info.isXBinopExpr = (op.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand =
+                  info.isXBinopExpr ? op.getOperand(1) : op.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              info.dVal = info.eVal;
+              return success();
+            })
+            .Default([](Operation *) { return success(); });
+
+    if (failed(result))
+      return result;
+  }
+  return success();
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -5409,7 +5506,11 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Pre-translate non-pattern operations inside the compare region.
     auto isAtomicComparePatternOp = [](Operation &op) {
       return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                       LLVM::OrOp>(op);
+                       LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                       LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                       mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                       mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                       mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
     };
     for (Operation &op : block.without_terminator()) {
       if (isAtomicComparePatternOp(op))
@@ -5443,47 +5544,16 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     };
 
     // Extract comparison predicate, eVal, and dVal from the region.
-    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    llvm::Value *eVal = nullptr;
-    llvm::Value *dVal = nullptr;
-    bool isXBinopExpr = false;
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
 
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
+    llvm::omp::OMPAtomicCompareOp compareOp = patternInfo.compareOp;
+    llvm::Value *eVal = patternInfo.eVal;
+    llvm::Value *dVal = patternInfo.dVal;
+    bool isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
 
     if (!eVal)
       return atomicCompareOp.emitError(
@@ -5529,17 +5599,42 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Postfix: v = select(success, D, old) — captures new value of x.
     if (isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
-      llvm::Instruction *cmpxchgInst = nullptr;
+      llvm::Value *oldVal = nullptr;
+      llvm::Value *successVal = nullptr;
+
+      // Integer path (and non-HandleFPNegZero FP path): a single cmpxchg
+      // lives in the current block.
       for (auto &inst : llvm::reverse(*curBB)) {
         if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
-          cmpxchgInst = &inst;
+          oldVal = builder.CreateExtractValue(&inst, /*Idxs=*/0);
+          successVal = builder.CreateExtractValue(&inst, /*Idxs=*/1);
           break;
         }
       }
-      assert(cmpxchgInst && "expected cmpxchg instruction");
-      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
-      llvm::Value *successVal =
-          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+
+      // FP HandleFPNegZero path: the OMPIRBuilder emits a multi-block
+      // structure (NaN / ±0.0 handling) with cmpxchg in predecessor
+      // blocks. Results are merged via PHI nodes in the current (exit)
+      // block: an i1 PHI for success and a bitcast of an integer PHI
+      // for the old FP value.
+      if (!oldVal) {
+        for (auto &inst : *curBB) {
+          auto *phi = dyn_cast<llvm::PHINode>(&inst);
+          if (!phi)
+            break;
+          if (phi->getType()->isIntegerTy(1))
+            successVal = phi;
+        }
+        for (auto &inst : *curBB) {
+          if (auto *bc = dyn_cast<llvm::BitCastInst>(&inst)) {
+            oldVal = bc;
+            break;
+          }
+        }
+      }
+
+      assert(oldVal && "expected cmpxchg or PHI+bitcast for compare capture");
+      assert(successVal && "expected success flag for compare capture");
       llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
       builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
     }
@@ -5686,7 +5781,11 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
 
   auto isAtomicComparePatternOp = [](Operation &op) {
     return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                     LLVM::OrOp>(op);
+                     LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                     LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                     mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                     mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                     mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
   };
 
   // Pre-translate operations inside the region that compute e and d (e.g.,
@@ -5859,54 +5958,15 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
     }
     return success();
   } else {
-
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        // Identify which operand is the block argument (x) and which is e.
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
-  }
-
-  // For non-complex patterns, also extract dVal from SelectOp.
-  if (!dVal) {
-    for (Operation &op : block.getOperations()) {
-      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        dVal = materializeValue(selectOp.getTrueValue());
-        break;
-      }
-    }
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
+    compareOp = patternInfo.compareOp;
+    eVal = patternInfo.eVal;
+    dVal = patternInfo.dVal;
+    isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
   }
 
   if (!eVal)
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 6b13be481e0e0..cc40506b909b1 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2161,6 +2161,30 @@ func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_real
+// CHECK-SAME: (%[[V:.*]]: memref<f32>, %[[X:.*]]: memref<f32>, %[[E:.*]]: f32, %[[D:.*]]: f32)
+func.func @omp_atomic_compare_capture_real(%v: memref<f32>, %x: memref<f32>, %e: f32, %d: f32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<f32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: f32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[E]] : f32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : f32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : f32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<f32>, memref<f32>, f32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<f32> {
+    ^bb0(%xval: f32):
+      %cmp = arith.cmpf oeq, %xval, %e : f32
+      %sel = arith.select %cmp, %d, %xval : f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : memref<f32>, memref<f32>, f32
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 377bd88894854..b779d20f42d0c 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2882,6 +2882,54 @@ llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr,
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_real
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], float %[[E:.*]], float %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_real(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32, %d : f32) {
+  // Real compare+capture: uses HandleFPNegZero multi-block path
+  // CHECK: %[[EBITS:.*]] = bitcast float %[[E]] to i32
+  // CHECK: %[[DBITS:.*]] = bitcast float %[[D]] to i32
+  // CHECK: %[[XI:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+  // CHECK: %[[XFP:.*]] = bitcast i32 %[[XI]] to float
+  // Part 1: NaN check
+  // CHECK: %[[E_NAN:.*]] = fcmp uno float %[[E]], %[[E]]
+  // CHECK: %[[X_NAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+  // CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_NAN]], %[[X_NAN]]
+  // CHECK: br i1 %[[EITHER_NAN]], label %[[NAN:.*]], label %[[NOTNAN:.*]]
+  // CHECK: [[NAN]]:
+  // CHECK: br label %[[EXIT:.*]]
+  // Part 2: Both-zero check
+  // CHECK: [[NOTNAN]]:
+  // CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+  // CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[E]], 0.000000e+00
+  // CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+  // CHECK: br i1 %[[BOTHZERO]], label %[[ZERO:.*]], label %[[NORMAL:.*]]
+  // CHECK: [[ZERO]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[XI]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Part 3: Normal cmpxchg
+  // CHECK: [[NORMAL]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Exit: select v = (success ? d : old_x)
+  // CHECK: [[EXIT]]:
+  // CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+  // CHECK: %[[OK:.*]] = phi i1 {{.*}}
+  // CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+  // CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[D]], float %[[OLDFP]]
+  // CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "oeq" %xval, %e : f32
+      %sel = llvm.select %cmp, %d, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From d42a6412d158c581c76cadf7d4626b4235e60725 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 07:25:53 -0500
Subject: [PATCH 8/8] Handling a crash in logical "omp atomic compare-capture"
 inside "omp parallel". Adding some extra tests.

---
 flang/lib/Semantics/check-omp-atomic.cpp      |  19 ++-
 .../Integration/OpenMP/atomic-compare.f90     |  27 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 117 ++++++++++++++++++
 .../test/Semantics/OpenMP/atomic-compare.f90  |  12 ++
 4 files changed, 174 insertions(+), 1 deletion(-)

diff --git a/flang/lib/Semantics/check-omp-atomic.cpp b/flang/lib/Semantics/check-omp-atomic.cpp
index ec307be469982..88997df26e4d0 100644
--- a/flang/lib/Semantics/check-omp-atomic.cpp
+++ b/flang/lib/Semantics/check-omp-atomic.cpp
@@ -419,6 +419,22 @@ static bool IsMaybeAtomicWrite(const evaluate::Assignment &assign) {
   return HasStorageOverlap(assign.lhs, assign.rhs) == nullptr;
 }
 
+// Determine whether two designators refer to the same atomic variable.
+// The comparison tolerates OpenMP host-association: a shared variable
+// referenced inside a parallel region may be designated through a
+// host-associated symbol in one statement and through its ultimate symbol
+// in another, even though both denote the same storage. Comparing the
+// (ultimate) symbol vectors together with the source form distinguishes
+// different variables and different constant subscripts (e.g. a(1) vs a(2))
+// while treating host-associated and ultimate references as equal.
+static bool IsSameAtomicVariable(const SomeExpr &x, const SomeExpr &y) {
+  if (x == y) {
+    return true;
+  }
+  return evaluate::GetSymbolVector(x) == evaluate::GetSymbolVector(y) &&
+      x.AsFortran() == y.AsFortran();
+}
+
 static void SetExpr(parser::TypedExpr &expr, MaybeExpr value) {
   if (value) {
     expr.Reset(new evaluate::GenericExprWrapper(std::move(value)),
@@ -822,7 +838,8 @@ void OmpStructureChecker::CheckAtomicCaptureAssignment(
     CheckAtomicVariable(
         atom, rsrc, /*checkTypeOnPointer=*/!IsPointerAssignment(capture));
     // This part should have been checked prior to calling this function.
-    assert(*GetConvertInput(capture.rhs) == atom &&
+    assert(GetConvertInput(capture.rhs) &&
+        IsSameAtomicVariable(*GetConvertInput(capture.rhs), atom) &&
         "This cannot be a capture assignment");
     CheckStorageOverlap(atom, {cap}, source);
   }
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index f30349c42e33e..e98b25e4b4539 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -394,3 +394,30 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
   !$omp end atomic
 end
 
+! Logical compare+capture inside a parallel region.
+!CHECK-LABEL: define void @atomic_compare_capture_logical_parallel_(
+!CHECK-SAME: ptr noalias %[[X:.*]])
+!CHECK: %[[CGEP:.*]] = getelementptr { ptr }, ptr %structArg, i32 0, i32 0
+!CHECK: store ptr %[[X]], ptr %[[CGEP]]
+!CHECK: call void {{.*}}@__kmpc_fork_call{{.*}}@atomic_compare_capture_logical_parallel_..omp_par
+!CHECK-LABEL: define internal void @atomic_compare_capture_logical_parallel_..omp_par(
+!CHECK-SAME: ptr noalias %{{.*}}, ptr noalias %{{.*}}, ptr %[[STRUCTARG:.*]])
+!CHECK: %[[GEP:.*]] = getelementptr { ptr }, ptr %[[STRUCTARG]], i32 0, i32 0
+!CHECK: %[[SHARED:.*]] = load ptr, ptr %[[GEP]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[SHARED]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %{{.*}}
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 1219d05296e4d..ccc44a370be79 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -305,6 +305,83 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Compare-capture with an explicit seq_cst memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
+! CHECK:         omp.atomic.capture memory_order(seq_cst) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_seq_cst(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture seq_cst
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acquire memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acquire(
+! CHECK:         omp.atomic.capture memory_order(acquire) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acquire(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acquire
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit release memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_release(
+! CHECK:         omp.atomic.capture memory_order(release) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_release(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture release
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acq_rel memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acq_rel(
+! CHECK:         omp.atomic.capture memory_order(acq_rel) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acq_rel(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acq_rel
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
@@ -335,3 +412,43 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
     v = x
   !$omp end atomic
 end
+
+! Logical compare-capture inside a parallel region. The shared atom 'x' is
+! host-associated, and the capture reads it while the conditional update writes
+! it. Verify the atom is converted to i32 once (above the atomic region) and
+! that the capture reads then compares the same converted reference.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical_parallel(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"})
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}Ex"
+! CHECK:         omp.parallel private(
+! CHECK:           %[[OLD_DECL:.*]]:2 = hlfir.declare %{{.*}}Eold_value"
+! CHECK:           %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Eexpected"
+! CHECK:           %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:           %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:           omp.atomic.capture memory_order(relaxed) {
+! CHECK:             omp.atomic.read %[[ATOMVAL:.*]] = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:             omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:             ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:               %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:               %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:               omp.yield(%[[SEL]] : i32)
+! CHECK:             }
+! CHECK:           }
+! CHECK:           %[[RES:.*]] = fir.load %[[ATOMVAL]] : !fir.ref<i32>
+! CHECK:           %[[RESCONV:.*]] = fir.convert %[[RES]] : (i32) -> !fir.logical<4>
+! CHECK:           fir.store %[[RESCONV]] to %[[OLD_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           omp.terminator
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
diff --git a/flang/test/Semantics/OpenMP/atomic-compare.f90 b/flang/test/Semantics/OpenMP/atomic-compare.f90
index 0e53729c2f02a..5cee2359f52be 100644
--- a/flang/test/Semantics/OpenMP/atomic-compare.f90
+++ b/flang/test/Semantics/OpenMP/atomic-compare.f90
@@ -9,6 +9,7 @@
 
   real a, b, c
   logical :: r, s
+  logical :: lx, le, ld, lo
   a = 1.0
   b = 2.0
   c = 3.0
@@ -58,6 +59,17 @@
   if (r) b = c
   !$omp end atomic
 
+  ! Compare-capture with a LOGICAL atom using .eqv. inside a parallel region.
+  ! Regression: the host-associated atom (from the update) must be recognized
+  ! as the same variable as the captured one, even though OpenMP resolves them
+  ! through different (but ultimately identical) symbols.
+  !$omp atomic compare capture
+  lo = lx
+  if (lx .eqv. le) then
+    lx = ld
+  end if
+  !$omp end atomic
+
   ! Check for error conditions:
   !ERROR: At most one SEQ_CST clause can appear on the ATOMIC directive
   !$omp atomic seq_cst seq_cst compare


From llvmlistbot at llvm.org  Wed Jul  1 05:28:21 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:28:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a4507e5.976e2a4f.30a67f.e1f0@mx.google.com>


================
@@ -1,5 +1,5 @@
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s -split-input-file | FileCheck %s --check-prefixes=CPP,CHECK
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s -split-input-file | FileCheck %s --check-prefixes=NOCPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s | FileCheck %s --check-prefixes=CPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s | FileCheck %s --check-prefixes=NOCPP,CHECK
----------------
ioghiban wrote:

I felt they are unnecessary, but you're right. The changes are not related and may be in a separate PR.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Wed Jul  1 05:41:05 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 05:41:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
Message-ID: <llvm.org/llvm/llvm-project/pull/206966@github.com>


https://github.com/andrey-golubev created https://github.com/llvm/llvm-project/pull/206966

There are three hooks that have a different, more complicated API than the other ones: allocationFn, memCpyFn, castFn. All three are called via "helper" functions that check whether the hooks are set to call them and fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options have a different "API": they are always set to some default implementation and can be overwritten by the user. This is a simpler API overall and seems it can be universal.

>From ffa05840b8c5f3d3aad166b0a0467ab498c04760 Mon Sep 17 00:00:00 2001
From: "Golubev, Andrey" <andrey.golubev at intel.com>
Date: Wed, 1 Jul 2026 12:33:24 +0000
Subject: [PATCH] [NFC][mlir][bufferization] Align alloc/memcpy/cast options
 hooks to the rest

There are three hooks that have a different, more complicated API than
the other ones: allocationFn, memCpyFn, castFn. All three are called via
"helper" functions that check whether the hooks are set to call them and
fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options
have a different "API": they are always set to some default
implementation and can be overwritten by the user. This is a simpler
API overall and seems it can be universal.
---
 .../IR/BufferizableOpInterface.h              | 26 +++----
 .../BufferizableOpInterfaceImpl.cpp           |  6 +-
 .../IR/BufferizableOpInterface.cpp            | 74 ++++++++-----------
 .../Bufferization/IR/BufferizationOps.cpp     | 23 +++---
 .../Bufferization/Transforms/Bufferize.cpp    |  4 +-
 .../BufferizableOpInterfaceImpl.cpp           |  2 +-
 .../BufferizableOpInterfaceImpl.cpp           |  2 +-
 .../BufferizableOpInterfaceImpl.cpp           |  8 +-
 8 files changed, 61 insertions(+), 84 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f49a31c1583b..203680796a289 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -303,23 +303,6 @@ struct BufferizationOptions {
   /// Return `true` if the given op should be bufferized.
   bool isOpAllowed(Operation *op) const;
 
-  /// Helper functions for allocation and memory copying.
-  std::optional<AllocationFn> allocationFn;
-  std::optional<MemCpyFn> memCpyFn;
-  std::optional<CastFn> castFn;
-
-  /// Create a memref allocation with the given type and dynamic extents.
-  FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
-                               ValueRange dynShape) const;
-
-  /// Creates a memcpy between two given buffers.
-  LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
-                             Value to) const;
-
-  /// Creates a cast function from a buffer value to a new type.
-  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
-                              Value value) const;
-
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
@@ -353,6 +336,15 @@ struct BufferizationOptions {
   /// predictable.
   void setFunctionBoundaryTypeConversion(LayoutMapOption layoutMapOption);
 
+  /// Create a memref allocation with the given type and dynamic extents.
+  AllocationFn allocationFn = nullptr;
+
+  /// Creates a memcpy between two given buffers.
+  MemCpyFn memCpyFn = nullptr;
+
+  /// Creates a cast function from a buffer value to a new type.
+  CastFn castFn = nullptr;
+
   /// Type conversion from tensors to buffers. This type conversion is used to
   /// determine bufferized function argument and result types.
   ///
diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index f810000dd082f..0fe7be82959a3 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -173,12 +173,10 @@ struct SelectOpInterface
             "incompatible buffer types on true/false operands");
       }
       if (trueBuffer.getType() != *targetType) {
-        trueBuffer =
-            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+        trueBuffer = *options.castFn(rewriter, loc, *targetType, trueBuffer);
       }
       if (falseBuffer.getType() != *targetType) {
-        falseBuffer =
-            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+        falseBuffer = *options.castFn(rewriter, loc, *targetType, falseBuffer);
       }
     }
 
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 7f656164f3a15..e39b20492e5ed 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,6 +358,34 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+/// Create a memref allocation with the given type and dynamic extents.
+FailureOr<Value> defaultCreateAlloc(OpBuilder &b, Location loc, MemRefType type,
+                                    ValueRange dynShape,
+                                    unsigned int bufferAlignment) {
+  // Default buffer allocation via AllocOp.
+  if (bufferAlignment != 0)
+    return memref::AllocOp::create(b, loc, type, dynShape,
+                                   b.getI64IntegerAttr(bufferAlignment))
+        .getResult();
+  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
+}
+
+/// Create a memory copy between two memref buffers.
+LogicalResult defaultCreateMemCpy(OpBuilder &b, Location loc, Value from,
+                                  Value to) {
+  memref::CopyOp::create(b, loc, from, to);
+  return success();
+}
+
+FailureOr<Value> defaultCreateCast(OpBuilder &b, Location loc, Type dest,
+                                   Value value) {
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
@@ -405,7 +433,9 @@ defaultReconcileBufferTypeMismatch(BufferLikeType x, BufferLikeType y,
 
 // Default constructor for BufferizationOptions.
 BufferizationOptions::BufferizationOptions()
-    : functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
+    : allocationFn(defaultCreateAlloc), memCpyFn(defaultCreateMemCpy),
+      castFn(defaultCreateCast),
+      functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
       unknownTypeConverterFn(defaultUnknownTypeConverter),
       reconcileBufferTypeMismatchFn(defaultReconcileBufferTypeMismatch) {}
 
@@ -811,48 +841,6 @@ void bufferization::replaceOpWithBufferizedValues(RewriterBase &rewriter,
   rewriter.replaceOp(op, replacements);
 }
 
-//===----------------------------------------------------------------------===//
-// Bufferization-specific scoped alloc insertion support.
-//===----------------------------------------------------------------------===//
-
-/// Create a memref allocation with the given type and dynamic extents.
-FailureOr<Value> BufferizationOptions::createAlloc(OpBuilder &b, Location loc,
-                                                   MemRefType type,
-                                                   ValueRange dynShape) const {
-  if (allocationFn)
-    return (*allocationFn)(b, loc, type, dynShape, bufferAlignment);
-
-  // Default bufferallocation via AllocOp.
-  if (bufferAlignment != 0)
-    return memref::AllocOp::create(b, loc, type, dynShape,
-                                   b.getI64IntegerAttr(bufferAlignment))
-        .getResult();
-  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
-}
-
-/// Create a memory copy between two memref buffers.
-LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
-                                                 Value from, Value to) const {
-  if (memCpyFn)
-    return (*memCpyFn)(b, loc, from, to);
-
-  memref::CopyOp::create(b, loc, from, to);
-  return success();
-}
-
-FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
-                                                  Type dest,
-                                                  Value value) const {
-  if (castFn)
-    return (*castFn)(b, loc, dest, value);
-
-  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
-  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
-         "cast incompatible");
-  return memref::CastOp::create(b, loc, dest, value).getResult();
-}
-
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 320f426985750..45f5b3eaa5aea 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = *options.createCast(b, value.getLoc(), destType, value);
+    Value casted = *options.castFn(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -73,11 +73,11 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
     dynamicOperands.push_back(size);
   }
 
-  FailureOr<Value> copy =
-      options.createAlloc(b, loc, destType, dynamicOperands);
+  FailureOr<Value> copy = options.allocationFn(
+      b, loc, destType, dynamicOperands, options.bufferAlignment);
   if (failed(copy))
     return failure();
-  if (failed(options.createMemCpy(b, loc, value, *copy)))
+  if (failed(options.memCpyFn(b, loc, value, *copy)))
     return failure();
   return copy;
 }
@@ -103,11 +103,9 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (!llvm::isa<BaseMemRefType>(srcType) ||
       !llvm::isa<BaseMemRefType>(destType)) {
     // Non-builtin case: the best is to try the user-provided cast.
-    assert(options.castFn.has_value() &&
-           "user-provided cast is required for non-builtin types");
     auto replacement =
-        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
-                           destType, bufferToTensor.getBuffer());
+        options.castFn(rewriter, bufferToTensor.getBuffer().getLoc(), destType,
+                       bufferToTensor.getBuffer());
     if (failed(replacement))
       return failure();
     rewriter.replaceOp(toBuffer, *replacement);
@@ -194,14 +192,15 @@ LogicalResult AllocTensorOp::bufferize(RewriterBase &rewriter,
     assert(dynamicDims.empty() && "expected either `copy` or `dynamicDims`");
     populateDynamicDimSizes(rewriter, loc, copyBuffer, dynamicDims);
   }
-  FailureOr<Value> alloc = options.createAlloc(
-      rewriter, loc, llvm::cast<MemRefType>(*allocType), dynamicDims);
+  FailureOr<Value> alloc =
+      options.allocationFn(rewriter, loc, llvm::cast<MemRefType>(*allocType),
+                           dynamicDims, options.bufferAlignment);
   if (failed(alloc))
     return failure();
 
   // Create memory copy (if any).
   if (getCopy()) {
-    if (failed(options.createMemCpy(rewriter, loc, copyBuffer, *alloc)))
+    if (failed(options.memCpyFn(rewriter, loc, copyBuffer, *alloc)))
       return failure();
   }
 
@@ -619,7 +618,7 @@ MaterializeInDestinationOp::bufferize(RewriterBase &rewriter,
   auto srcBuffer = getBuffer(rewriter, getSource(), options, state);
   if (failed(srcBuffer))
     return failure();
-  if (failed(options.createMemCpy(rewriter, getLoc(), *srcBuffer, buffer)))
+  if (failed(options.memCpyFn(rewriter, getLoc(), *srcBuffer, buffer)))
     return failure();
   replaceOpWithBufferizedValues(rewriter, getOperation(),
                                 tensorDest ? ValueRange(buffer) : ValueRange());
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index e0262abf313b8..d6e0b12733976 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -477,8 +477,8 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
       // A cast is needed if the operand and the block argument have different
       // bufferized types.
       if (type != *operandBufferType) {
-        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
-                                                type, bufferizedOperand);
+        bufferizedOperand = *options.castFn(rewriter, operand.getLoc(), type,
+                                            bufferizedOperand);
       }
       newOperands.push_back(bufferizedOperand);
     }
diff --git a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
index c2cbc95e94343..a7c5934132b35 100644
--- a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -146,7 +146,7 @@ struct GlobalStoreOpInterface
     }
 
     auto memcpy =
-        options.createMemCpy(rewriter, loc, sourceMemref.value(), targetMemref);
+        options.memCpyFn(rewriter, loc, sourceMemref.value(), targetMemref);
     if (failed(memcpy)) {
       return failure();
     }
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index 401be7440f5a2..f8302126a4a4e 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -37,7 +37,7 @@ static Value castBuffer(OpBuilder &b, Value buffer, Type type,
   if (buffer.getType() == type)
     return buffer;
 
-  return *options.createCast(b, buffer.getLoc(), type, buffer);
+  return *options.castFn(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value
diff --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 41a3d96b832d1..525d57341cee5 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -724,7 +724,7 @@ struct InsertSliceOpInterface
         getBuffer(rewriter, insertSliceOp.getSource(), options, state);
     if (failed(srcMemref))
       return failure();
-    if (failed(options.createMemCpy(rewriter, loc, *srcMemref, subView)))
+    if (failed(options.memCpyFn(rewriter, loc, *srcMemref, subView)))
       return failure();
 
     replaceOpWithBufferizedValues(rewriter, op, *dstMemref);
@@ -1002,8 +1002,8 @@ struct ParallelInsertSliceOpInterface
         parallelInsertSliceOp.getMixedStrides());
 
     // This memcpy will fold away if everything bufferizes in-place.
-    if (failed(options.createMemCpy(rewriter, parallelInsertSliceOp.getLoc(),
-                                    *srcBuffer, subview)))
+    if (failed(options.memCpyFn(rewriter, parallelInsertSliceOp.getLoc(),
+                                *srcBuffer, subview)))
       return failure();
 
     // In case the source was allocated in the same block, make sure that the
@@ -1175,7 +1175,7 @@ struct ConcatOpInterface
           rewriter, loc, subviewMemRefType, dstBuffer, offsets, sizes, strides);
 
       // Copy the source buffer into the destination subview.
-      if (failed(options.createMemCpy(rewriter, loc, *srcBuffer, subview)))
+      if (failed(options.memCpyFn(rewriter, loc, *srcBuffer, subview)))
         return failure();
 
       concatDimOffset = sum(concatDimOffset, concatDimSize);


From llvmlistbot at llvm.org  Wed Jul  1 05:41:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:41:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a450b07.d4361ba8.1154fd.89b2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andrei Golubev (andrey-golubev)

<details>
<summary>Changes</summary>

There are three hooks that have a different, more complicated API than the other ones: allocationFn, memCpyFn, castFn. All three are called via "helper" functions that check whether the hooks are set to call them and fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options have a different "API": they are always set to some default implementation and can be overwritten by the user. This is a simpler API overall and seems it can be universal.

---
Full diff: https://github.com/llvm/llvm-project/pull/206966.diff


8 Files Affected:

- (modified) mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h (+9-17) 
- (modified) mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp (+2-4) 
- (modified) mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp (+31-43) 
- (modified) mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp (+11-12) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp (+2-2) 
- (modified) mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp (+1-1) 
- (modified) mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp (+1-1) 
- (modified) mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp (+4-4) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f49a31c1583b..203680796a289 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -303,23 +303,6 @@ struct BufferizationOptions {
   /// Return `true` if the given op should be bufferized.
   bool isOpAllowed(Operation *op) const;
 
-  /// Helper functions for allocation and memory copying.
-  std::optional<AllocationFn> allocationFn;
-  std::optional<MemCpyFn> memCpyFn;
-  std::optional<CastFn> castFn;
-
-  /// Create a memref allocation with the given type and dynamic extents.
-  FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
-                               ValueRange dynShape) const;
-
-  /// Creates a memcpy between two given buffers.
-  LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
-                             Value to) const;
-
-  /// Creates a cast function from a buffer value to a new type.
-  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
-                              Value value) const;
-
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
@@ -353,6 +336,15 @@ struct BufferizationOptions {
   /// predictable.
   void setFunctionBoundaryTypeConversion(LayoutMapOption layoutMapOption);
 
+  /// Create a memref allocation with the given type and dynamic extents.
+  AllocationFn allocationFn = nullptr;
+
+  /// Creates a memcpy between two given buffers.
+  MemCpyFn memCpyFn = nullptr;
+
+  /// Creates a cast function from a buffer value to a new type.
+  CastFn castFn = nullptr;
+
   /// Type conversion from tensors to buffers. This type conversion is used to
   /// determine bufferized function argument and result types.
   ///
diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index f810000dd082f..0fe7be82959a3 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -173,12 +173,10 @@ struct SelectOpInterface
             "incompatible buffer types on true/false operands");
       }
       if (trueBuffer.getType() != *targetType) {
-        trueBuffer =
-            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+        trueBuffer = *options.castFn(rewriter, loc, *targetType, trueBuffer);
       }
       if (falseBuffer.getType() != *targetType) {
-        falseBuffer =
-            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+        falseBuffer = *options.castFn(rewriter, loc, *targetType, falseBuffer);
       }
     }
 
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 7f656164f3a15..e39b20492e5ed 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,6 +358,34 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+/// Create a memref allocation with the given type and dynamic extents.
+FailureOr<Value> defaultCreateAlloc(OpBuilder &b, Location loc, MemRefType type,
+                                    ValueRange dynShape,
+                                    unsigned int bufferAlignment) {
+  // Default buffer allocation via AllocOp.
+  if (bufferAlignment != 0)
+    return memref::AllocOp::create(b, loc, type, dynShape,
+                                   b.getI64IntegerAttr(bufferAlignment))
+        .getResult();
+  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
+}
+
+/// Create a memory copy between two memref buffers.
+LogicalResult defaultCreateMemCpy(OpBuilder &b, Location loc, Value from,
+                                  Value to) {
+  memref::CopyOp::create(b, loc, from, to);
+  return success();
+}
+
+FailureOr<Value> defaultCreateCast(OpBuilder &b, Location loc, Type dest,
+                                   Value value) {
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
@@ -405,7 +433,9 @@ defaultReconcileBufferTypeMismatch(BufferLikeType x, BufferLikeType y,
 
 // Default constructor for BufferizationOptions.
 BufferizationOptions::BufferizationOptions()
-    : functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
+    : allocationFn(defaultCreateAlloc), memCpyFn(defaultCreateMemCpy),
+      castFn(defaultCreateCast),
+      functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
       unknownTypeConverterFn(defaultUnknownTypeConverter),
       reconcileBufferTypeMismatchFn(defaultReconcileBufferTypeMismatch) {}
 
@@ -811,48 +841,6 @@ void bufferization::replaceOpWithBufferizedValues(RewriterBase &rewriter,
   rewriter.replaceOp(op, replacements);
 }
 
-//===----------------------------------------------------------------------===//
-// Bufferization-specific scoped alloc insertion support.
-//===----------------------------------------------------------------------===//
-
-/// Create a memref allocation with the given type and dynamic extents.
-FailureOr<Value> BufferizationOptions::createAlloc(OpBuilder &b, Location loc,
-                                                   MemRefType type,
-                                                   ValueRange dynShape) const {
-  if (allocationFn)
-    return (*allocationFn)(b, loc, type, dynShape, bufferAlignment);
-
-  // Default bufferallocation via AllocOp.
-  if (bufferAlignment != 0)
-    return memref::AllocOp::create(b, loc, type, dynShape,
-                                   b.getI64IntegerAttr(bufferAlignment))
-        .getResult();
-  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
-}
-
-/// Create a memory copy between two memref buffers.
-LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
-                                                 Value from, Value to) const {
-  if (memCpyFn)
-    return (*memCpyFn)(b, loc, from, to);
-
-  memref::CopyOp::create(b, loc, from, to);
-  return success();
-}
-
-FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
-                                                  Type dest,
-                                                  Value value) const {
-  if (castFn)
-    return (*castFn)(b, loc, dest, value);
-
-  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
-  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
-         "cast incompatible");
-  return memref::CastOp::create(b, loc, dest, value).getResult();
-}
-
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 320f426985750..45f5b3eaa5aea 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = *options.createCast(b, value.getLoc(), destType, value);
+    Value casted = *options.castFn(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -73,11 +73,11 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
     dynamicOperands.push_back(size);
   }
 
-  FailureOr<Value> copy =
-      options.createAlloc(b, loc, destType, dynamicOperands);
+  FailureOr<Value> copy = options.allocationFn(
+      b, loc, destType, dynamicOperands, options.bufferAlignment);
   if (failed(copy))
     return failure();
-  if (failed(options.createMemCpy(b, loc, value, *copy)))
+  if (failed(options.memCpyFn(b, loc, value, *copy)))
     return failure();
   return copy;
 }
@@ -103,11 +103,9 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (!llvm::isa<BaseMemRefType>(srcType) ||
       !llvm::isa<BaseMemRefType>(destType)) {
     // Non-builtin case: the best is to try the user-provided cast.
-    assert(options.castFn.has_value() &&
-           "user-provided cast is required for non-builtin types");
     auto replacement =
-        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
-                           destType, bufferToTensor.getBuffer());
+        options.castFn(rewriter, bufferToTensor.getBuffer().getLoc(), destType,
+                       bufferToTensor.getBuffer());
     if (failed(replacement))
       return failure();
     rewriter.replaceOp(toBuffer, *replacement);
@@ -194,14 +192,15 @@ LogicalResult AllocTensorOp::bufferize(RewriterBase &rewriter,
     assert(dynamicDims.empty() && "expected either `copy` or `dynamicDims`");
     populateDynamicDimSizes(rewriter, loc, copyBuffer, dynamicDims);
   }
-  FailureOr<Value> alloc = options.createAlloc(
-      rewriter, loc, llvm::cast<MemRefType>(*allocType), dynamicDims);
+  FailureOr<Value> alloc =
+      options.allocationFn(rewriter, loc, llvm::cast<MemRefType>(*allocType),
+                           dynamicDims, options.bufferAlignment);
   if (failed(alloc))
     return failure();
 
   // Create memory copy (if any).
   if (getCopy()) {
-    if (failed(options.createMemCpy(rewriter, loc, copyBuffer, *alloc)))
+    if (failed(options.memCpyFn(rewriter, loc, copyBuffer, *alloc)))
       return failure();
   }
 
@@ -619,7 +618,7 @@ MaterializeInDestinationOp::bufferize(RewriterBase &rewriter,
   auto srcBuffer = getBuffer(rewriter, getSource(), options, state);
   if (failed(srcBuffer))
     return failure();
-  if (failed(options.createMemCpy(rewriter, getLoc(), *srcBuffer, buffer)))
+  if (failed(options.memCpyFn(rewriter, getLoc(), *srcBuffer, buffer)))
     return failure();
   replaceOpWithBufferizedValues(rewriter, getOperation(),
                                 tensorDest ? ValueRange(buffer) : ValueRange());
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index e0262abf313b8..d6e0b12733976 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -477,8 +477,8 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
       // A cast is needed if the operand and the block argument have different
       // bufferized types.
       if (type != *operandBufferType) {
-        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
-                                                type, bufferizedOperand);
+        bufferizedOperand = *options.castFn(rewriter, operand.getLoc(), type,
+                                            bufferizedOperand);
       }
       newOperands.push_back(bufferizedOperand);
     }
diff --git a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
index c2cbc95e94343..a7c5934132b35 100644
--- a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -146,7 +146,7 @@ struct GlobalStoreOpInterface
     }
 
     auto memcpy =
-        options.createMemCpy(rewriter, loc, sourceMemref.value(), targetMemref);
+        options.memCpyFn(rewriter, loc, sourceMemref.value(), targetMemref);
     if (failed(memcpy)) {
       return failure();
     }
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index 401be7440f5a2..f8302126a4a4e 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -37,7 +37,7 @@ static Value castBuffer(OpBuilder &b, Value buffer, Type type,
   if (buffer.getType() == type)
     return buffer;
 
-  return *options.createCast(b, buffer.getLoc(), type, buffer);
+  return *options.castFn(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value
diff --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 41a3d96b832d1..525d57341cee5 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -724,7 +724,7 @@ struct InsertSliceOpInterface
         getBuffer(rewriter, insertSliceOp.getSource(), options, state);
     if (failed(srcMemref))
       return failure();
-    if (failed(options.createMemCpy(rewriter, loc, *srcMemref, subView)))
+    if (failed(options.memCpyFn(rewriter, loc, *srcMemref, subView)))
       return failure();
 
     replaceOpWithBufferizedValues(rewriter, op, *dstMemref);
@@ -1002,8 +1002,8 @@ struct ParallelInsertSliceOpInterface
         parallelInsertSliceOp.getMixedStrides());
 
     // This memcpy will fold away if everything bufferizes in-place.
-    if (failed(options.createMemCpy(rewriter, parallelInsertSliceOp.getLoc(),
-                                    *srcBuffer, subview)))
+    if (failed(options.memCpyFn(rewriter, parallelInsertSliceOp.getLoc(),
+                                *srcBuffer, subview)))
       return failure();
 
     // In case the source was allocated in the same block, make sure that the
@@ -1175,7 +1175,7 @@ struct ConcatOpInterface
           rewriter, loc, subviewMemRefType, dstBuffer, offsets, sizes, strides);
 
       // Copy the source buffer into the destination subview.
-      if (failed(options.createMemCpy(rewriter, loc, *srcBuffer, subview)))
+      if (failed(options.memCpyFn(rewriter, loc, *srcBuffer, subview)))
         return failure();
 
       concatDimOffset = sum(concatDimOffset, concatDimSize);

``````````

</details>


https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Wed Jul  1 05:41:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:41:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a450b07.cbc3f03a.3cbca0.3012@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-mlprogram

Author: Andrei Golubev (andrey-golubev)

<details>
<summary>Changes</summary>

There are three hooks that have a different, more complicated API than the other ones: allocationFn, memCpyFn, castFn. All three are called via "helper" functions that check whether the hooks are set to call them and fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options have a different "API": they are always set to some default implementation and can be overwritten by the user. This is a simpler API overall and seems it can be universal.

---
Full diff: https://github.com/llvm/llvm-project/pull/206966.diff


8 Files Affected:

- (modified) mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h (+9-17) 
- (modified) mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp (+2-4) 
- (modified) mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp (+31-43) 
- (modified) mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp (+11-12) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp (+2-2) 
- (modified) mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp (+1-1) 
- (modified) mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp (+1-1) 
- (modified) mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp (+4-4) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f49a31c1583b..203680796a289 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -303,23 +303,6 @@ struct BufferizationOptions {
   /// Return `true` if the given op should be bufferized.
   bool isOpAllowed(Operation *op) const;
 
-  /// Helper functions for allocation and memory copying.
-  std::optional<AllocationFn> allocationFn;
-  std::optional<MemCpyFn> memCpyFn;
-  std::optional<CastFn> castFn;
-
-  /// Create a memref allocation with the given type and dynamic extents.
-  FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
-                               ValueRange dynShape) const;
-
-  /// Creates a memcpy between two given buffers.
-  LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
-                             Value to) const;
-
-  /// Creates a cast function from a buffer value to a new type.
-  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
-                              Value value) const;
-
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
@@ -353,6 +336,15 @@ struct BufferizationOptions {
   /// predictable.
   void setFunctionBoundaryTypeConversion(LayoutMapOption layoutMapOption);
 
+  /// Create a memref allocation with the given type and dynamic extents.
+  AllocationFn allocationFn = nullptr;
+
+  /// Creates a memcpy between two given buffers.
+  MemCpyFn memCpyFn = nullptr;
+
+  /// Creates a cast function from a buffer value to a new type.
+  CastFn castFn = nullptr;
+
   /// Type conversion from tensors to buffers. This type conversion is used to
   /// determine bufferized function argument and result types.
   ///
diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index f810000dd082f..0fe7be82959a3 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -173,12 +173,10 @@ struct SelectOpInterface
             "incompatible buffer types on true/false operands");
       }
       if (trueBuffer.getType() != *targetType) {
-        trueBuffer =
-            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+        trueBuffer = *options.castFn(rewriter, loc, *targetType, trueBuffer);
       }
       if (falseBuffer.getType() != *targetType) {
-        falseBuffer =
-            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+        falseBuffer = *options.castFn(rewriter, loc, *targetType, falseBuffer);
       }
     }
 
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 7f656164f3a15..e39b20492e5ed 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,6 +358,34 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+/// Create a memref allocation with the given type and dynamic extents.
+FailureOr<Value> defaultCreateAlloc(OpBuilder &b, Location loc, MemRefType type,
+                                    ValueRange dynShape,
+                                    unsigned int bufferAlignment) {
+  // Default buffer allocation via AllocOp.
+  if (bufferAlignment != 0)
+    return memref::AllocOp::create(b, loc, type, dynShape,
+                                   b.getI64IntegerAttr(bufferAlignment))
+        .getResult();
+  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
+}
+
+/// Create a memory copy between two memref buffers.
+LogicalResult defaultCreateMemCpy(OpBuilder &b, Location loc, Value from,
+                                  Value to) {
+  memref::CopyOp::create(b, loc, from, to);
+  return success();
+}
+
+FailureOr<Value> defaultCreateCast(OpBuilder &b, Location loc, Type dest,
+                                   Value value) {
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
@@ -405,7 +433,9 @@ defaultReconcileBufferTypeMismatch(BufferLikeType x, BufferLikeType y,
 
 // Default constructor for BufferizationOptions.
 BufferizationOptions::BufferizationOptions()
-    : functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
+    : allocationFn(defaultCreateAlloc), memCpyFn(defaultCreateMemCpy),
+      castFn(defaultCreateCast),
+      functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
       unknownTypeConverterFn(defaultUnknownTypeConverter),
       reconcileBufferTypeMismatchFn(defaultReconcileBufferTypeMismatch) {}
 
@@ -811,48 +841,6 @@ void bufferization::replaceOpWithBufferizedValues(RewriterBase &rewriter,
   rewriter.replaceOp(op, replacements);
 }
 
-//===----------------------------------------------------------------------===//
-// Bufferization-specific scoped alloc insertion support.
-//===----------------------------------------------------------------------===//
-
-/// Create a memref allocation with the given type and dynamic extents.
-FailureOr<Value> BufferizationOptions::createAlloc(OpBuilder &b, Location loc,
-                                                   MemRefType type,
-                                                   ValueRange dynShape) const {
-  if (allocationFn)
-    return (*allocationFn)(b, loc, type, dynShape, bufferAlignment);
-
-  // Default bufferallocation via AllocOp.
-  if (bufferAlignment != 0)
-    return memref::AllocOp::create(b, loc, type, dynShape,
-                                   b.getI64IntegerAttr(bufferAlignment))
-        .getResult();
-  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
-}
-
-/// Create a memory copy between two memref buffers.
-LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
-                                                 Value from, Value to) const {
-  if (memCpyFn)
-    return (*memCpyFn)(b, loc, from, to);
-
-  memref::CopyOp::create(b, loc, from, to);
-  return success();
-}
-
-FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
-                                                  Type dest,
-                                                  Value value) const {
-  if (castFn)
-    return (*castFn)(b, loc, dest, value);
-
-  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
-  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
-         "cast incompatible");
-  return memref::CastOp::create(b, loc, dest, value).getResult();
-}
-
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 320f426985750..45f5b3eaa5aea 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = *options.createCast(b, value.getLoc(), destType, value);
+    Value casted = *options.castFn(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -73,11 +73,11 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
     dynamicOperands.push_back(size);
   }
 
-  FailureOr<Value> copy =
-      options.createAlloc(b, loc, destType, dynamicOperands);
+  FailureOr<Value> copy = options.allocationFn(
+      b, loc, destType, dynamicOperands, options.bufferAlignment);
   if (failed(copy))
     return failure();
-  if (failed(options.createMemCpy(b, loc, value, *copy)))
+  if (failed(options.memCpyFn(b, loc, value, *copy)))
     return failure();
   return copy;
 }
@@ -103,11 +103,9 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (!llvm::isa<BaseMemRefType>(srcType) ||
       !llvm::isa<BaseMemRefType>(destType)) {
     // Non-builtin case: the best is to try the user-provided cast.
-    assert(options.castFn.has_value() &&
-           "user-provided cast is required for non-builtin types");
     auto replacement =
-        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
-                           destType, bufferToTensor.getBuffer());
+        options.castFn(rewriter, bufferToTensor.getBuffer().getLoc(), destType,
+                       bufferToTensor.getBuffer());
     if (failed(replacement))
       return failure();
     rewriter.replaceOp(toBuffer, *replacement);
@@ -194,14 +192,15 @@ LogicalResult AllocTensorOp::bufferize(RewriterBase &rewriter,
     assert(dynamicDims.empty() && "expected either `copy` or `dynamicDims`");
     populateDynamicDimSizes(rewriter, loc, copyBuffer, dynamicDims);
   }
-  FailureOr<Value> alloc = options.createAlloc(
-      rewriter, loc, llvm::cast<MemRefType>(*allocType), dynamicDims);
+  FailureOr<Value> alloc =
+      options.allocationFn(rewriter, loc, llvm::cast<MemRefType>(*allocType),
+                           dynamicDims, options.bufferAlignment);
   if (failed(alloc))
     return failure();
 
   // Create memory copy (if any).
   if (getCopy()) {
-    if (failed(options.createMemCpy(rewriter, loc, copyBuffer, *alloc)))
+    if (failed(options.memCpyFn(rewriter, loc, copyBuffer, *alloc)))
       return failure();
   }
 
@@ -619,7 +618,7 @@ MaterializeInDestinationOp::bufferize(RewriterBase &rewriter,
   auto srcBuffer = getBuffer(rewriter, getSource(), options, state);
   if (failed(srcBuffer))
     return failure();
-  if (failed(options.createMemCpy(rewriter, getLoc(), *srcBuffer, buffer)))
+  if (failed(options.memCpyFn(rewriter, getLoc(), *srcBuffer, buffer)))
     return failure();
   replaceOpWithBufferizedValues(rewriter, getOperation(),
                                 tensorDest ? ValueRange(buffer) : ValueRange());
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index e0262abf313b8..d6e0b12733976 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -477,8 +477,8 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
       // A cast is needed if the operand and the block argument have different
       // bufferized types.
       if (type != *operandBufferType) {
-        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
-                                                type, bufferizedOperand);
+        bufferizedOperand = *options.castFn(rewriter, operand.getLoc(), type,
+                                            bufferizedOperand);
       }
       newOperands.push_back(bufferizedOperand);
     }
diff --git a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
index c2cbc95e94343..a7c5934132b35 100644
--- a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -146,7 +146,7 @@ struct GlobalStoreOpInterface
     }
 
     auto memcpy =
-        options.createMemCpy(rewriter, loc, sourceMemref.value(), targetMemref);
+        options.memCpyFn(rewriter, loc, sourceMemref.value(), targetMemref);
     if (failed(memcpy)) {
       return failure();
     }
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index 401be7440f5a2..f8302126a4a4e 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -37,7 +37,7 @@ static Value castBuffer(OpBuilder &b, Value buffer, Type type,
   if (buffer.getType() == type)
     return buffer;
 
-  return *options.createCast(b, buffer.getLoc(), type, buffer);
+  return *options.castFn(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value
diff --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 41a3d96b832d1..525d57341cee5 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -724,7 +724,7 @@ struct InsertSliceOpInterface
         getBuffer(rewriter, insertSliceOp.getSource(), options, state);
     if (failed(srcMemref))
       return failure();
-    if (failed(options.createMemCpy(rewriter, loc, *srcMemref, subView)))
+    if (failed(options.memCpyFn(rewriter, loc, *srcMemref, subView)))
       return failure();
 
     replaceOpWithBufferizedValues(rewriter, op, *dstMemref);
@@ -1002,8 +1002,8 @@ struct ParallelInsertSliceOpInterface
         parallelInsertSliceOp.getMixedStrides());
 
     // This memcpy will fold away if everything bufferizes in-place.
-    if (failed(options.createMemCpy(rewriter, parallelInsertSliceOp.getLoc(),
-                                    *srcBuffer, subview)))
+    if (failed(options.memCpyFn(rewriter, parallelInsertSliceOp.getLoc(),
+                                *srcBuffer, subview)))
       return failure();
 
     // In case the source was allocated in the same block, make sure that the
@@ -1175,7 +1175,7 @@ struct ConcatOpInterface
           rewriter, loc, subviewMemRefType, dstBuffer, offsets, sizes, strides);
 
       // Copy the source buffer into the destination subview.
-      if (failed(options.createMemCpy(rewriter, loc, *srcBuffer, subview)))
+      if (failed(options.memCpyFn(rewriter, loc, *srcBuffer, subview)))
         return failure();
 
       concatDimOffset = sum(concatDimOffset, concatDimSize);

``````````

</details>


https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Wed Jul  1 05:42:44 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:42:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a450b44.14d9b9dc.7e5e3.0519@mx.google.com>


================
@@ -60,6 +59,16 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
+Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
+  Type resultTy;
+  if (opTy.getRank() == 0) {
+    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
----------------
ioghiban wrote:

I tested it on the examples in `mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir`. For rank 0 the function returned `!emitc.array<...>` type and for rank 0 the element type.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Wed Jul  1 05:43:30 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 05:43:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a450b72.f14cf1a5.9019b.d140@mx.google.com>


================
@@ -840,6 +840,23 @@ LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
   return success();
 }
 
+FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
+                                                  Type dest,
+                                                  Value value) const {
+  // If the value already has the correct type, no cast is needed.
+  if (value.getType() == dest)
+    return value;
+
+  if (castFn)
+    return (*castFn)(b, loc, dest, value);
+
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
----------------
andrey-golubev wrote:

fyi - https://github.com/llvm/llvm-project/pull/206966

https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 05:43:51 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Wed, 01 Jul 2026 05:43:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Introduce
 castBufferTypeFn hook and API (PR #205080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205080@github.com>
Message-ID: <6a450b87.f732ad81.1d6a4d.a147@mx.google.com>


https://github.com/andrey-golubev edited https://github.com/llvm/llvm-project/pull/205080

From llvmlistbot at llvm.org  Wed Jul  1 05:52:36 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:52:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a450d94.65dfdb71.32f23b.bb92@mx.google.com>


================
@@ -142,9 +154,12 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-// If `v` is defined through an unrealized cast and the source of that cast
-// is `emitc.ptr`, return the pointer.
-static Value stripPointerUnrealizedCast(Value v) {
+static Value getPointerOrStripUnrealizedCast(Value v) {
----------------
ioghiban wrote:

I believa that may not be accurate, since this function may return:
`<<NULL VALUE>>`
`emitc.cast %p : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>`
OR
`unrealized_conversion_cast %arg0 : memref<i32> to !emitc.ptr<i32> {__pure_type_conversion__}`

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Wed Jul  1 05:55:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 05:55:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2146dab - [mlir][affine] Add trip count check
 for affine-pipeline-data-transfer pass (#206755)
Message-ID: <6a450e2f.eb165e5c.5b8dc.0b1b@mx.google.com>


Author: lonely eagle
Date: 2026-07-01T20:55:07+08:00
New Revision: 2146dab17c298b296e16c22cac64bf9ccd9f4e2f

URL: https://github.com/llvm/llvm-project/commit/2146dab17c298b296e16c22cac64bf9ccd9f4e2f
DIFF: https://github.com/llvm/llvm-project/commit/2146dab17c298b296e16c22cac64bf9ccd9f4e2f.diff

LOG: [mlir][affine] Add trip count check for affine-pipeline-data-transfer pass (#206755)

Fix `affine-pipeline-data-transfer` to skip loops with a constant trip
count ≤0, avoiding invalid DMA generation that leads to wrong code.

Added: 
    

Modified: 
    mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
    mlir/test/Dialect/Affine/pipeline-data-transfer.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp b/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
index 575b529658127..9a1c731b5a97e 100644
--- a/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/PipelineDataTransfer.cpp
@@ -245,10 +245,16 @@ static void findMatchingStartFinishInsts(
 /// 'forOp' is deleted, and a prologue, a new pipelined loop, and epilogue are
 /// inserted right before where it was.
 void PipelineDataTransfer::runOnAffineForOp(AffineForOp forOp) {
-  if (!forOp.getStaticTripCount()) {
+  std::optional<llvm::APInt> tripCount = forOp.getStaticTripCount();
+  if (!tripCount) {
     LLVM_DEBUG(forOp.emitRemark("won't pipeline due to unknown trip count"));
     return;
   }
+  if (tripCount->getSExtValue() <= 0) {
+    LLVM_DEBUG(
+        forOp.emitRemark("won't pipeline due to trip count is non-positive"));
+    return;
+  }
 
   SmallVector<std::pair<Operation *, Operation *>, 4> startWaitPairs;
   findMatchingStartFinishInsts(forOp, startWaitPairs);

diff  --git a/mlir/test/Dialect/Affine/pipeline-data-transfer.mlir b/mlir/test/Dialect/Affine/pipeline-data-transfer.mlir
index 35507c37be79b..2abf28a7a9249 100644
--- a/mlir/test/Dialect/Affine/pipeline-data-transfer.mlir
+++ b/mlir/test/Dialect/Affine/pipeline-data-transfer.mlir
@@ -396,3 +396,30 @@ func.func @same_memref_source_and_tag(%arg0: index, %arg1: index) {
   return
 }
 // CHECK: affine.for
+
+// -----
+
+// CHECK-LABEL: func @loop_trip_count_non_positive
+func.func @loop_trip_count_non_positive() {
+  %A = memref.alloc() : memref<256 x f32, affine_map<(d0) -> (d0)>, 0>
+  %Ah = memref.alloc() : memref<32 x f32, affine_map<(d0) -> (d0)>, 1>
+  %tag = memref.alloc() : memref<1 x f32>
+  %zero = arith.constant 0 : index
+  %num_elts = arith.constant 32 : index
+  affine.for %i = 0 to -1 {
+    affine.dma_start %A[%i], %Ah[%i], %tag[%zero], %num_elts : memref<256 x f32>, memref<32 x f32, 1>, memref<1 x f32>
+    affine.dma_wait %tag[%zero], %num_elts : memref<1 x f32>
+    %v = affine.load %Ah[%i] : memref<32 x f32, affine_map<(d0) -> (d0)>, 1>
+    %r = "compute"(%v) : (f32) -> (f32)
+    affine.store %r, %Ah[%i] : memref<32 x f32, affine_map<(d0) -> (d0)>, 1>
+  }
+  memref.dealloc %tag : memref<1 x f32>
+  memref.dealloc %Ah : memref<32 x f32, affine_map<(d0) -> (d0)>, 1>
+  return
+}
+
+//  CHECK-NOT: affine.dma_start
+//  CHECK-NOT: affine.dma_wait
+//      CHECK: affine.for %{{.*}} = 0 to -1
+// CHECK-NEXT:   affine.dma_start
+// CHECK-NEXT:   affine.dma_wait


        

From llvmlistbot at llvm.org  Wed Jul  1 05:55:14 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Wed, 01 Jul 2026 05:55:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add trip count check for
 affine-pipeline-data-transfer pass (PR #206755)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206755@github.com>
Message-ID: <6a450e32.fdc477d6.2b77d8.fc15@mx.google.com>


https://github.com/linuxlonelyeagle closed https://github.com/llvm/llvm-project/pull/206755

From llvmlistbot at llvm.org  Wed Jul  1 05:58:11 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:58:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a450ee3.bf38ae28.12214b.fd1a@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Wed Jul  1 05:58:18 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Wed, 01 Jul 2026 05:58:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (PR #199911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199911@github.com>
Message-ID: <6a450eea.a87eb5dc.379910.ae2a@mx.google.com>


linuxlonelyeagle wrote:

ping for review: cc @superty @Groverkss. Thanks.

https://github.com/llvm/llvm-project/pull/199911

From llvmlistbot at llvm.org  Wed Jul  1 05:59:31 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 01 Jul 2026 05:59:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a450f33.6cfa33d2.26a96c.d8e7@mx.google.com>


================
@@ -165,6 +180,9 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
+  if (shape.empty())
+    return linearIndex;
+
----------------
ioghiban wrote:

will use `indices` instead, for consistency

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Wed Jul  1 06:00:59 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Wed, 01 Jul 2026 06:00:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a450f8b.a70a48cb.68e9d.4ae3@mx.google.com>


https://github.com/sahas3 edited https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:00:59 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Wed, 01 Jul 2026 06:00:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a450f8b.1173ae6c.139100.8326@mx.google.com>


================

----------------
sahas3 wrote:

pre-existing but can you move this after the nullptr check?

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:00:59 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Wed, 01 Jul 2026 06:00:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a450f8b.c06a7583.2413fa.1fc8@mx.google.com>


================

----------------
sahas3 wrote:

A comment will be nice for readability, some thing like, 
"when transposing the reshape output yields exactly the reshape input shape, return the reshape input directly."

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:00:59 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Wed, 01 Jul 2026 06:00:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a450f8b.4793980a.3a77f9.aa5f@mx.google.com>


https://github.com/sahas3 approved this pull request.

LG, apologies for the late review. I'll try to review the other PRs this week.

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:00:59 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Wed, 01 Jul 2026 06:00:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a450f8b.cb03ad15.18ebfa.66d0@mx.google.com>


================

----------------
sahas3 wrote:

I think a LIT test for this early return path is missing?

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:35:42 2026
From: llvmlistbot at llvm.org (Caroline Newcombe)
Date: Wed, 01 Jul 2026 06:35:42 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a4517ae.a151105c.15f7b1.3f9e@mx.google.com>


cenewcombe wrote:

> I'm still new to reviewing in this area, so please don't take my review as sufficient on its own. That said, I went through the PR and left some findings below. Let me know if anything's unclear or if you have questions.

@Ritanya-B-Bharadwaj thanks so much for taking the time to review this PR! Unfortunately, I don't see any of your review comments. Could you try posting them again? You can also reach out to me offline.

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Wed Jul  1 06:42:37 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:42:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a45194d.7d41f3a1.2f50d3.76f6@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/203529

>From 7dfb40eb9154c3c75843e6fbeb68caa8743025e8 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 10 Jun 2026 21:33:36 +0100
Subject: [PATCH 1/2] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes

Extend tosa-reduce-transposes so transpose hosting can fold through
unit-expanded vector reshapes such as 1x...x1xC. In particular,
this improves transpose folding when hosting through broadcast binary
elementwise operations.

Change-Id: If2eb010ac322d5b9ef73f1fdda9de6049f2d2b2d
---
 .../Tosa/Transforms/TosaReduceTransposes.cpp  | 38 +++++++++++++------
 .../Dialect/Tosa/tosa-reduce-transposes.mlir  | 16 ++++++++
 2 files changed, 43 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 65ef49bdc3077..0b4c0f04085e4 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -174,6 +174,23 @@ struct TosaReduceTransposes final
   transposeDenseAttribute(DenseElementsAttr input, ArrayRef<int32_t> perms);
 };
 
+// Check if shape is of the form 1x1x...x1xNx1x...x1x1 -> 1x1x...x1xNx1x...x1x1
+// Valid examples include:
+// - N -> 1x1xNx1
+// - Nx1x1x1 -> 1x1xNx1
+// - 1x1xNx1 -> 1x1xNx1
+static LogicalResult verifyUnitExpandedVectorShape(ArrayRef<int64_t> shape) {
+  bool nonUnitDimDetected = false;
+  for (const int64_t dim : shape) {
+    if (dim != 1) {
+      if (nonUnitDimDetected)
+        return failure();
+      nonUnitDimDetected = true;
+    }
+  }
+  return success();
+}
+
 std::optional<DenseElementsAttr>
 TosaReduceTransposes::transposeDenseAttribute(DenseElementsAttr input,
                                               ArrayRef<int32_t> perms) {
@@ -393,24 +410,24 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   auto reshapeInputType =
       llvm::dyn_cast<RankedTensorType>(reshapeOp.getInput1().getType());
   auto reshapeInputShape = reshapeInputType.getShape();
-  // want reshape N -> 1x1x...x1xNx1x...x1x1
-  if (!reshapeInputType || reshapeInputShape.size() != 1)
+  if (!reshapeInputType)
     return std::nullopt;
   auto reshapeOutputType =
       llvm::cast<RankedTensorType>(reshapeOutput.getType());
-
+  const ArrayRef<int64_t> reshapeOutputShape = reshapeOutputType.getShape();
   // Instead of inserting a TransposeOp here, we check if we can fold it into
   // the ReshapeOp. There is more complex cases where this is possible, and
   // this check can be extended.
 
-  // Checking if reshape is N -> 1x1x...x1xNx1x...x1x1
-  auto shape = reshapeOutputType.getShape();
-  size_t ones = llvm::count(shape, 1);
-  // N == 1 and N != 1
-  if (ones != shape.size() - 1 &&
-      (ones != shape.size() || reshapeInputShape[0] != 1))
+  if (failed(verifyUnitExpandedVectorShape(reshapeInputShape)) ||
+      failed(verifyUnitExpandedVectorShape(reshapeOutputShape)))
     return std::nullopt;
 
+  SmallVector<int64_t> mappedShape =
+      applyTOSAPermutation(reshapeOutputShape, hoistedPerms);
+  if (llvm::equal(mappedShape, reshapeInputShape))
+    return reshapeOp.getInput1();
+
   // Do not insert a TransposeOp, instead we fold the reshape and its attribute.
   llvm::SmallVector<int64_t> newShape;
   if (!tosa::getConstShapeValues(reshapeOp.getShape().getDefiningOp(),
@@ -421,8 +438,7 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(applyTOSAPermutation(shape, hoistedPerms),
-                            reshapeOutputType.getElementType()),
+      RankedTensorType::get(mappedShape, reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));
diff --git a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
index b3f4260ede2f5..0ad3068011f7f 100644
--- a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
@@ -174,6 +174,22 @@ func.func @test_reshape_for_broadcast(%arg0: tensor<4x3x2xi32>) -> tensor<4x3x2x
 
 // -----
 
+// CHECK-LABEL: @test_multi_dim_reshape_for_broadcast
+// CHECK-DAG: %[[SHAPE:.*]] = tosa.const_shape  {values = dense<[4, 1, 1]> : tensor<3xindex>}
+// CHECK: %[[RESHAPE:.*]] = tosa.reshape %arg0, %[[SHAPE]] : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<4x1x1xi32>
+// CHECK: %[[ADD:.*]] = tosa.add %arg1, %[[RESHAPE]]
+// CHECK: return %[[ADD]]
+func.func @test_multi_dim_reshape_for_broadcast(%arg0: tensor<1x4x1xi32>, %arg1: tensor<4x3x2xi32>) -> tensor<4x3x2xi32> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 4]> : tensor<3xindex>} : () -> !tosa.shape<3>
+  %reshape = tosa.reshape %arg0, %shape : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<1x1x4xi32>
+  %transpose0 = tosa.transpose %arg1 {perms = array<i32: 2, 1, 0>}: (tensor<4x3x2xi32>) -> tensor<2x3x4xi32>
+  %add = tosa.add %transpose0, %reshape : (tensor<2x3x4xi32>, tensor<1x1x4xi32>) -> tensor<2x3x4xi32>
+  %transpose1 = tosa.transpose %add {perms = array<i32: 2, 1, 0>}: (tensor<2x3x4xi32>) -> tensor<4x3x2xi32>
+  return %transpose1 : tensor<4x3x2xi32>
+}
+
+// -----
+
 // COM: taken directly from ResNet18 translation.
 // COM: changes: %74 as argument instead of result of conv2d
 

>From 87f51a82f73c4460670b267e3085c57a635c1073 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 1 Jul 2026 14:41:07 +0100
Subject: [PATCH 2/2] Address review comments

Change-Id: I571c79239a48824b5b816203465182d0f8fc3c3b
---
 .../Dialect/Tosa/Transforms/TosaReduceTransposes.cpp  | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 0b4c0f04085e4..f025526d52d67 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -409,25 +409,20 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   auto reshapeOutput = reshapeOp.getOutput();
   auto reshapeInputType =
       llvm::dyn_cast<RankedTensorType>(reshapeOp.getInput1().getType());
-  auto reshapeInputShape = reshapeInputType.getShape();
   if (!reshapeInputType)
     return std::nullopt;
+  auto reshapeInputShape = reshapeInputType.getShape();
   auto reshapeOutputType =
       llvm::cast<RankedTensorType>(reshapeOutput.getType());
   const ArrayRef<int64_t> reshapeOutputShape = reshapeOutputType.getShape();
+
   // Instead of inserting a TransposeOp here, we check if we can fold it into
   // the ReshapeOp. There is more complex cases where this is possible, and
   // this check can be extended.
-
   if (failed(verifyUnitExpandedVectorShape(reshapeInputShape)) ||
       failed(verifyUnitExpandedVectorShape(reshapeOutputShape)))
     return std::nullopt;
 
-  SmallVector<int64_t> mappedShape =
-      applyTOSAPermutation(reshapeOutputShape, hoistedPerms);
-  if (llvm::equal(mappedShape, reshapeInputShape))
-    return reshapeOp.getInput1();
-
   // Do not insert a TransposeOp, instead we fold the reshape and its attribute.
   llvm::SmallVector<int64_t> newShape;
   if (!tosa::getConstShapeValues(reshapeOp.getShape().getDefiningOp(),
@@ -438,7 +433,7 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(mappedShape, reshapeOutputType.getElementType()),
+      RankedTensorType::get(applyTOSAPermutation(reshapeOutputShape, hoistedPerms), reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));


From llvmlistbot at llvm.org  Wed Jul  1 06:43:22 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:43:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a45197a.92ea0e2a.99846.bc21@mx.google.com>


================

----------------
lhutton1 wrote:

See comment above

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:43:22 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:43:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a45197a.eea6ab6e.29c4b3.5158@mx.google.com>


================

----------------
lhutton1 wrote:

This short-cut actually seems unnecessary on second glance. The canonicalizer will clean up any identity reshapes. Therefore I've removed it.

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:43:22 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:43:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a45197a.14d9b9dc.7e5e3.134d@mx.google.com>


================

----------------
lhutton1 wrote:

good catch, thanks!

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:44:03 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:44:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a4519a3.14d9b9dc.7e5e3.2a50@mx.google.com>


lhutton1 wrote:

> LG, apologies for the late review. I'll try to review the other PRs this week.

No worries, thanks for taking the time to review!

https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 06:48:00 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 01 Jul 2026 06:48:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a451a90.a2377b91.1fccb4.43b4@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/203529

>From 7dfb40eb9154c3c75843e6fbeb68caa8743025e8 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 10 Jun 2026 21:33:36 +0100
Subject: [PATCH 1/2] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes

Extend tosa-reduce-transposes so transpose hosting can fold through
unit-expanded vector reshapes such as 1x...x1xC. In particular,
this improves transpose folding when hosting through broadcast binary
elementwise operations.

Change-Id: If2eb010ac322d5b9ef73f1fdda9de6049f2d2b2d
---
 .../Tosa/Transforms/TosaReduceTransposes.cpp  | 38 +++++++++++++------
 .../Dialect/Tosa/tosa-reduce-transposes.mlir  | 16 ++++++++
 2 files changed, 43 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 65ef49bdc3077..0b4c0f04085e4 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -174,6 +174,23 @@ struct TosaReduceTransposes final
   transposeDenseAttribute(DenseElementsAttr input, ArrayRef<int32_t> perms);
 };
 
+// Check if shape is of the form 1x1x...x1xNx1x...x1x1 -> 1x1x...x1xNx1x...x1x1
+// Valid examples include:
+// - N -> 1x1xNx1
+// - Nx1x1x1 -> 1x1xNx1
+// - 1x1xNx1 -> 1x1xNx1
+static LogicalResult verifyUnitExpandedVectorShape(ArrayRef<int64_t> shape) {
+  bool nonUnitDimDetected = false;
+  for (const int64_t dim : shape) {
+    if (dim != 1) {
+      if (nonUnitDimDetected)
+        return failure();
+      nonUnitDimDetected = true;
+    }
+  }
+  return success();
+}
+
 std::optional<DenseElementsAttr>
 TosaReduceTransposes::transposeDenseAttribute(DenseElementsAttr input,
                                               ArrayRef<int32_t> perms) {
@@ -393,24 +410,24 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   auto reshapeInputType =
       llvm::dyn_cast<RankedTensorType>(reshapeOp.getInput1().getType());
   auto reshapeInputShape = reshapeInputType.getShape();
-  // want reshape N -> 1x1x...x1xNx1x...x1x1
-  if (!reshapeInputType || reshapeInputShape.size() != 1)
+  if (!reshapeInputType)
     return std::nullopt;
   auto reshapeOutputType =
       llvm::cast<RankedTensorType>(reshapeOutput.getType());
-
+  const ArrayRef<int64_t> reshapeOutputShape = reshapeOutputType.getShape();
   // Instead of inserting a TransposeOp here, we check if we can fold it into
   // the ReshapeOp. There is more complex cases where this is possible, and
   // this check can be extended.
 
-  // Checking if reshape is N -> 1x1x...x1xNx1x...x1x1
-  auto shape = reshapeOutputType.getShape();
-  size_t ones = llvm::count(shape, 1);
-  // N == 1 and N != 1
-  if (ones != shape.size() - 1 &&
-      (ones != shape.size() || reshapeInputShape[0] != 1))
+  if (failed(verifyUnitExpandedVectorShape(reshapeInputShape)) ||
+      failed(verifyUnitExpandedVectorShape(reshapeOutputShape)))
     return std::nullopt;
 
+  SmallVector<int64_t> mappedShape =
+      applyTOSAPermutation(reshapeOutputShape, hoistedPerms);
+  if (llvm::equal(mappedShape, reshapeInputShape))
+    return reshapeOp.getInput1();
+
   // Do not insert a TransposeOp, instead we fold the reshape and its attribute.
   llvm::SmallVector<int64_t> newShape;
   if (!tosa::getConstShapeValues(reshapeOp.getShape().getDefiningOp(),
@@ -421,8 +438,7 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(applyTOSAPermutation(shape, hoistedPerms),
-                            reshapeOutputType.getElementType()),
+      RankedTensorType::get(mappedShape, reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));
diff --git a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
index b3f4260ede2f5..0ad3068011f7f 100644
--- a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
@@ -174,6 +174,22 @@ func.func @test_reshape_for_broadcast(%arg0: tensor<4x3x2xi32>) -> tensor<4x3x2x
 
 // -----
 
+// CHECK-LABEL: @test_multi_dim_reshape_for_broadcast
+// CHECK-DAG: %[[SHAPE:.*]] = tosa.const_shape  {values = dense<[4, 1, 1]> : tensor<3xindex>}
+// CHECK: %[[RESHAPE:.*]] = tosa.reshape %arg0, %[[SHAPE]] : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<4x1x1xi32>
+// CHECK: %[[ADD:.*]] = tosa.add %arg1, %[[RESHAPE]]
+// CHECK: return %[[ADD]]
+func.func @test_multi_dim_reshape_for_broadcast(%arg0: tensor<1x4x1xi32>, %arg1: tensor<4x3x2xi32>) -> tensor<4x3x2xi32> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 4]> : tensor<3xindex>} : () -> !tosa.shape<3>
+  %reshape = tosa.reshape %arg0, %shape : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<1x1x4xi32>
+  %transpose0 = tosa.transpose %arg1 {perms = array<i32: 2, 1, 0>}: (tensor<4x3x2xi32>) -> tensor<2x3x4xi32>
+  %add = tosa.add %transpose0, %reshape : (tensor<2x3x4xi32>, tensor<1x1x4xi32>) -> tensor<2x3x4xi32>
+  %transpose1 = tosa.transpose %add {perms = array<i32: 2, 1, 0>}: (tensor<2x3x4xi32>) -> tensor<4x3x2xi32>
+  return %transpose1 : tensor<4x3x2xi32>
+}
+
+// -----
+
 // COM: taken directly from ResNet18 translation.
 // COM: changes: %74 as argument instead of result of conv2d
 

>From e1ec59c8100fa4bd8d3cbdd5b193a3057a24c505 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 1 Jul 2026 14:41:07 +0100
Subject: [PATCH 2/2] Address review comments

Change-Id: I571c79239a48824b5b816203465182d0f8fc3c3b
---
 .../Tosa/Transforms/TosaReduceTransposes.cpp        | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 0b4c0f04085e4..0bd415040f22f 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -409,25 +409,20 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   auto reshapeOutput = reshapeOp.getOutput();
   auto reshapeInputType =
       llvm::dyn_cast<RankedTensorType>(reshapeOp.getInput1().getType());
-  auto reshapeInputShape = reshapeInputType.getShape();
   if (!reshapeInputType)
     return std::nullopt;
+  auto reshapeInputShape = reshapeInputType.getShape();
   auto reshapeOutputType =
       llvm::cast<RankedTensorType>(reshapeOutput.getType());
   const ArrayRef<int64_t> reshapeOutputShape = reshapeOutputType.getShape();
+
   // Instead of inserting a TransposeOp here, we check if we can fold it into
   // the ReshapeOp. There is more complex cases where this is possible, and
   // this check can be extended.
-
   if (failed(verifyUnitExpandedVectorShape(reshapeInputShape)) ||
       failed(verifyUnitExpandedVectorShape(reshapeOutputShape)))
     return std::nullopt;
 
-  SmallVector<int64_t> mappedShape =
-      applyTOSAPermutation(reshapeOutputShape, hoistedPerms);
-  if (llvm::equal(mappedShape, reshapeInputShape))
-    return reshapeOp.getInput1();
-
   // Do not insert a TransposeOp, instead we fold the reshape and its attribute.
   llvm::SmallVector<int64_t> newShape;
   if (!tosa::getConstShapeValues(reshapeOp.getShape().getDefiningOp(),
@@ -438,7 +433,9 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(mappedShape, reshapeOutputType.getElementType()),
+      RankedTensorType::get(
+          applyTOSAPermutation(reshapeOutputShape, hoistedPerms),
+          reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));


From llvmlistbot at llvm.org  Wed Jul  1 07:07:31 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Wed, 01 Jul 2026 07:07:31 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a451f23.510f576d.1df92b.e19b@mx.google.com>


================

----------------
egebeysel wrote:

I checked this again to refresh my memory, and the reason is that we did not need it. For pure tiling,  the tile size is set by destination (packed) outer dimensions. In fact, the tiling interface doesn't even allow you to tile across the inner dims.

So I might as well drop my TODO on the pack tiling interface for this.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 07:08:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 07:08:29 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a451f5d.8e75e0cb.3d2f0.b76c@mx.google.com>


================
@@ -2626,6 +2626,38 @@ void OmpStructureChecker::Enter(const parser::OpenMPDispatchConstruct &x) {
   }
 }
 
+void OmpStructureChecker::Enter(const parser::OmpErrorDirective &x) {
+  // Defaults per spec: AT(compilation), SEVERITY(fatal)
+  auto atKind{parser::OmpAtClause::ActionTime::Compilation};
+  auto sevKind{parser::OmpSeverityClause::SevLevel::Fatal};
+  std::optional<std::string> message;
+
+  for (const parser::OmpClause &clause : x.v.Clauses().v) {
----------------
Ritanya-B-Bharadwaj wrote:

I noticed the atKind/sevKind/message extraction loop is basically duplicated between genErrorDirective and here. Don't you think it'd be worth pulling this into a shared helper so the two don't drift apart over time?

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Wed Jul  1 07:08:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 07:08:29 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a451f5d.f732ad81.1d6a4d.28b1@mx.google.com>


================
@@ -4452,6 +4452,18 @@ LogicalResult CriticalOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Error directive (2.5.4)
----------------
Ritanya-B-Bharadwaj wrote:

Should we maintain a single openmp spec reference throughout (either 5.1 or 5.2)? 

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Wed Jul  1 07:08:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 07:08:29 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a451f5d.58bdd12c.2e0810.6de6@mx.google.com>


================
@@ -1730,6 +1730,37 @@ def BarrierOp : OpenMP_Op<"barrier"> {
   let assemblyFormat = "attr-dict";
 }
 
+//===----------------------------------------------------------------------===//
+// [5.2] 10.4 error directive
----------------
Ritanya-B-Bharadwaj wrote:

I believe it should be OpenMP 5.2 - section 8.5 here?

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Wed Jul  1 07:19:24 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 01 Jul 2026 07:19:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Use `SymbolUserMap`
 to avoid quadratic symbol lookups (PR #205448)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205448@github.com>
Message-ID: <6a4521ec.294e96f5.91a50.8439@mx.google.com>


================
@@ -770,6 +766,13 @@ void RemoveDeadValues::runOnOperation() {
   auto &la = getAnalysis<RunLivenessAnalysis>();
   Operation *module = getOperation();
 
+  // Build a symbol user map once up front so that processFuncOp can look up the
+  // callers of each function in O(1). Otherwise, each call would walk the
+  // entire module to find the callers, making the pass O(numFunctions *
+  // numOperations).
+  SymbolTableCollection symbolTableCollection;
+  SymbolUserMap symbolUserMap(symbolTableCollection, module);
+
----------------
joker-eph wrote:

Isn't the pass possibly removing call sites, thus affecting the symbol table? Do we need to update it as we go?

https://github.com/llvm/llvm-project/pull/205448

From llvmlistbot at llvm.org  Wed Jul  1 07:19:59 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 01 Jul 2026 07:19:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Use `SymbolUserMap`
 to avoid quadratic symbol lookups (PR #205448)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205448@github.com>
Message-ID: <6a45220f.c2422094.1ec368.efa0@mx.google.com>


================
@@ -770,6 +766,13 @@ void RemoveDeadValues::runOnOperation() {
   auto &la = getAnalysis<RunLivenessAnalysis>();
   Operation *module = getOperation();
 
+  // Build a symbol user map once up front so that processFuncOp can look up the
+  // callers of each function in O(1). Otherwise, each call would walk the
+  // entire module to find the callers, making the pass O(numFunctions *
+  // numOperations).
+  SymbolTableCollection symbolTableCollection;
+  SymbolUserMap symbolUserMap(symbolTableCollection, module);
+
----------------
joker-eph wrote:

Mmm, I see your note in the description:

> This is behavior-preserving: the map is read only during the mutation-free collection walk, and all IR erasure happens afterwards in cleanUpDeadVals, so the map cannot become stale while it is in use.

https://github.com/llvm/llvm-project/pull/205448

From llvmlistbot at llvm.org  Wed Jul  1 07:20:29 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 01 Jul 2026 07:20:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Use `SymbolUserMap`
 to avoid quadratic symbol lookups (PR #205448)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205448@github.com>
Message-ID: <6a45222d.288e0ebf.17d3f2.4d7d@mx.google.com>


https://github.com/joker-eph approved this pull request.


https://github.com/llvm/llvm-project/pull/205448

From llvmlistbot at llvm.org  Wed Jul  1 07:22:15 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Wed, 01 Jul 2026 07:22:15 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a452297.8f540080.396cf4.773e@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_unaligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // No hint is passed: the static tile sizes (7, 5) are not aligned to the scalable
+      // inner tiles (8 * vscale, 4 * vscale), so the tiled unpack keeps a remainder slice.
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [7, 5] : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Producer fusion - linalg.unpack with scalable inner tiles fused as a producer
+// into an elementwise consumer that is tiled with scalable tile
+// sizes that are an integer multiple of the inner tiles (16*vscale of 8*vscale,
+// 8*vscale of 4*vscale). `Multiple` hint is passed accordingly.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @unpack_elemwise_scalable_multiple
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+// CHECK:             %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+// CHECK:             %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             linalg.exp ins(%[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @unpack_elemwise_scalable_multiple(%arg0: tensor<4x8x?x?xf32>, %arg1: tensor<?x?xf32>, %arg2 : index, %arg3 : index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c16 = arith.constant 16 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %t0 = arith.muli %c16, %vscale : index
+  %t1 = arith.muli %c8, %vscale : index
+  %0 = tensor.empty(%arg2, %arg3) : tensor<?x?xf32>
+  %1 = linalg.unpack %arg0 inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %0
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  %2 = linalg.exp ins(%1: tensor<?x?xf32>)
+                       outs(%arg1: tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %2 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %exp = transform.structured.match ops{["linalg.exp"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %mulis = transform.structured.match ops{["arith.muli"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %i0, %i1, %t0h, %t1h = transform.split_handle %mulis : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
----------------
egebeysel wrote:

The transform fuse op does not have the `scalable_sizes` and the custom parsing available. I can add that if you wish, but I'd rather leave that to a follow-up if you don't mind.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 07:27:30 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 07:27:30 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a4523d2.e72b82d4.141ee.353b@mx.google.com>


banach-space wrote:

Hi @LouisLu060211,

Thank you for the patch! Unfortunately, I do not think Vector should move in this direction.

>From your summary:

> For example, vector<16xi3> to vector<3xi16> has 48 bits on both sides, but 16 is not divisible by 3.

It sounds like you are looking for logic to model sub-byte types in Vector. There is already [VectorEmulateNarrowType](https://github.com/llvm/llvm-project/blob/0b5c006208cf7b0fee3709f468152e7de81607ef/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp) for this purpose. While that pass works, it is admittedly somewhat fragile. In my opinion, this is because we never fully settled whether it should support arbitrary integer widths such as `i3`.

In hindsight, I think `VectorEmulateNarrowType` should not support such exotic types. The underlying logic is very complex, difficult to support, and there does not seem to be enough interest in maintaining it beyond very narrow use cases. In other words, the current support is not very general.

I also question whether `i3` is a type we should support beyond the most basic examples. I appreciate that some users may want this, but supporting it generally would come at a significant maintenance cost. IMHO, we should instead prioritise the more common cases that have more users and reviewers.

Is this a major blocker for you? If we start unblocking cases like this, I am worried that we will also need to review many other Vector transformations, such as unrolling and flattening, to make sure they bail out cleanly. That would be a substantial effort, and I would prefer to avoid it at this point.

If you have broader plans for sub-byte types, I would be happy to hear them.

Thank you!

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 07:40:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 07:40:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a4526d8.e9d484b9.34b6c.b63c@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index f025526d5..0bd415040 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -433,7 +433,9 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(applyTOSAPermutation(reshapeOutputShape, hoistedPerms), reshapeOutputType.getElementType()),
+      RankedTensorType::get(
+          applyTOSAPermutation(reshapeOutputShape, hoistedPerms),
+          reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));

``````````

</details>


https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Wed Jul  1 07:42:24 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Wed, 01 Jul 2026 07:42:24 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a452750.f14cf1a5.9019b.53c2@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
----------------
egebeysel wrote:

I'm adding these as comments currently, but the main difference is:

- if it's equal: you get outer size(s) 1
- if it's aligned: you do NOT get an extra extract_slice after the unpack
- if it's unaligned: you do get an extra extract_slice after the unpack before the insert_slice, i.e. the unpack cannot be consumed directly, it has extra computation, and the correct slice from that should be extracted.

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Wed Jul  1 07:44:48 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 01 Jul 2026 07:44:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a4527e0.fdc477d6.2b77d8.e84f@mx.google.com>


https://github.com/Jianhui-Li edited https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Wed Jul  1 07:44:49 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 01 Jul 2026 07:44:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a4527e1.7d41f3a1.2f50d3.2cab@mx.google.com>


https://github.com/Jianhui-Li commented:

Confirmed the issue, thanks! Please address the comments

https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Wed Jul  1 07:44:49 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 01 Jul 2026 07:44:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a4527e1.eea6ab6e.29c4b3.cb87@mx.google.com>


================
@@ -288,7 +292,7 @@ computeMemrefMeta(OpType xferOp, PatternRewriter &rewriter) {
 //   %offsets =  memref_offset + orig_offset + local_offsets
 static Value computeOffsets(VectorTransferOpInterface xferOp,
----------------
Jianhui-Li wrote:

Instead of change two function signature and pass two strides to computeOffsets(), you could do the following: 
  1. computeMemrefMeta() no longer permutes
  2. Inside computeOffsets(), it builds a local permutedStrides copy (via
  adjustStridesForPermutation) used only for the local-offset step-vector loop; the base-offset loop keeps the original memref-order strides.

https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Wed Jul  1 07:44:49 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 01 Jul 2026 07:44:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a4527e1.f8b79514.3daeff.60f0@mx.google.com>


================
@@ -0,0 +1,46 @@
+// RUN: mlir-opt %s -convert-vector-to-xegpu -split-input-file | FileCheck %s
+
+// -----
+gpu.module @xevm_module {
----------------
Jianhui-Li wrote:

please combine this test to the existing test file: transfer-read-to-xegpu.mlir

https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Wed Jul  1 07:55:12 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 07:55:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a452a50.d5797644.130832.9159@mx.google.com>


================
@@ -477,3 +477,52 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+// A contiguous `tensor.extract` from a source whose rank is *smaller* than the
+// iteration space (here: a 1-D source inside a 2-D loop nest). The contiguous
+// `vector.transfer_read` only reads the trailing (contiguous) dim - the leading
+// dim is broadcast via the permutation map - so its mask must be rank-reduced
+// (`vector<4xi1>`) rather than the full iteration-space mask (`vector<1x4xi1>`).
+// Regression test: previously the full iteration mask was attached, producing
+// an invalid `vector.transfer_read` ("inferred mask type ... don't match").
+
+func.func @masked_contiguous_extract_rank_reducing_mask(
+    %src: tensor<16xf32>,
+    %output : tensor<1x3xf32>,
+    %idx: index) -> tensor<1x3xf32> {
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%output : tensor<1x3xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%2, %idx)
+    %extracted = tensor.extract %src[%3] : tensor<16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x3xf32>
+  return %1 : tensor<1x3xf32>
----------------
banach-space wrote:

Would will happen if the leading dim is != 1?
```suggestion
  return %1 : tensor<4x3xf32>
```

https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Wed Jul  1 07:56:46 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 01 Jul 2026 07:56:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
Message-ID: <llvm.org/llvm/llvm-project/pull/207005@github.com>


https://github.com/nujaa created https://github.com/llvm/llvm-project/pull/207005

Applying FlattenElementwiseLinalgOp on Broadcasted linalg would break. This transformation is not valid for those usecase, so I added a condition to exit gracefully instead.

>From d453196fa77a41fa2c54d13a98da731353c10b0f Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Wed, 1 Jul 2026 21:08:28 +0800
Subject: [PATCH] [MLIR][Vector] Fix FlattenElementwiseOp on broadcasted
 linalgs

---
 .../TransformOps/LinalgTransformOps.cpp       |  8 +++
 .../Dialect/Linalg/flatten-elementwise.mlir   | 53 +++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index f44693096b26b..d39c1cb0e4a20 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4297,6 +4297,14 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return DiagnosedSilenceableFailure::success();
   }
 
+  // Bail out on linalgs with broadcasting semantics
+  if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
+        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+      })) {
+    results.push_back(target);
+    return DiagnosedSilenceableFailure::success();
+  }
+
   // Attempt to flatten all dims to one.
   ReassociationIndices reassociation(target.getNumLoops());
   std::iota(reassociation.begin(), reassociation.end(), 0);
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 9fe50a521d2d8..2ca08215a166a 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -118,3 +118,56 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<1x2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Wed Jul  1 07:57:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 07:57:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4ddf664 - [mlir][ABI] Fix scalable vector
 mapping in ABITypeMapper (#206617)
Message-ID: <6a452acd.492e23e4.6e404.9d99@mx.google.com>


Author: Fedor Nikolaev
Date: 2026-07-01T15:57:12+01:00
New Revision: 4ddf664c6c3d6487c7fca1fd2149c0f2356db321

URL: https://github.com/llvm/llvm-project/commit/4ddf664c6c3d6487c7fca1fd2149c0f2356db321
DIFF: https://github.com/llvm/llvm-project/commit/4ddf664c6c3d6487c7fca1fd2149c0f2356db321.diff

LOG: [mlir][ABI] Fix scalable vector mapping in ABITypeMapper (#206617)

mapVectorType in ABITypeMapper incorrectly used ElementCount::getFixed
for all vector types, including scalable ones (e.g. vector<[4]xf32>).
This results in scalable vectors being mapped as fixed-size vectors,
which produces incorrect ABI lowering for SVE and RISC-V V targets.

Added: 
    

Modified: 
    mlir/lib/ABI/ABITypeMapper.cpp
    mlir/unittests/ABI/ABITypeMapperTest.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/ABI/ABITypeMapper.cpp b/mlir/lib/ABI/ABITypeMapper.cpp
index 83dc6990ec5bb..fe63df659302a 100644
--- a/mlir/lib/ABI/ABITypeMapper.cpp
+++ b/mlir/lib/ABI/ABITypeMapper.cpp
@@ -70,12 +70,19 @@ const llvm::abi::Type *ABITypeMapper::mapVectorType(mlir::VectorType type) {
   if (!elementTy)
     return nullptr;
 
+  // ElementCount supports at most one scalable dimension, hence
+  // vectors with more than one scalable dimension (e.g. vector<[2]x[4]xf32>)
+  // cannot be represented and mapVectorType must return nullptr in this case.
+  if (llvm::count(type.getScalableDims(), true) > 1)
+    return nullptr;
+
   auto shape = type.getShape();
   uint64_t totalElements = 1;
   for (int64_t dim : shape)
     totalElements *= dim;
 
-  llvm::ElementCount ec = llvm::ElementCount::getFixed(totalElements);
+  llvm::ElementCount ec =
+      llvm::ElementCount::get(totalElements, type.isScalable());
   uint64_t abiAlign = dl.getTypeABIAlignment(type);
   return builder.getVectorType(elementTy, ec, llvm::Align(abiAlign));
 }

diff  --git a/mlir/unittests/ABI/ABITypeMapperTest.cpp b/mlir/unittests/ABI/ABITypeMapperTest.cpp
index 4a7989298a149..c702cf387b58a 100644
--- a/mlir/unittests/ABI/ABITypeMapperTest.cpp
+++ b/mlir/unittests/ABI/ABITypeMapperTest.cpp
@@ -170,4 +170,34 @@ TEST_F(ABITypeMapperTest, MapUnsignedI32) {
   EXPECT_FALSE(intTy->isSigned());
 }
 
+TEST_F(ABITypeMapperTest, MapScalableVectorOf4xF32) {
+  DataLayout dl(module);
+  ABITypeMapper mapper(dl);
+
+  auto f32 = Float32Type::get(&ctx);
+  auto vec = VectorType::get({4}, f32, /* scalableDims=*/true);
+  const llvm::abi::Type *result = mapper.map(vec);
+
+  ASSERT_NE(result, nullptr);
+  EXPECT_TRUE(result->isVector());
+
+  auto *vecTy = llvm::cast<llvm::abi::VectorType>(result);
+  EXPECT_TRUE(vecTy->getNumElements().isScalable());
+  EXPECT_EQ(vecTy->getNumElements().getKnownMinValue(), 4u);
+  EXPECT_TRUE(vecTy->getElementType()->isFloat());
+}
+
+TEST_F(ABITypeMapperTest, MapWithTwoScalableDimsReturnsNull) {
+  DataLayout dl(module);
+  ABITypeMapper mapper(dl);
+
+  auto f32 = Float32Type::get(&ctx);
+  auto vec = VectorType::get({2, 4}, f32, /*scalableDims=*/{true, true});
+  const llvm::abi::Type *result = mapper.map(vec);
+
+  // LLVM supports at most 1 scalable dimension, hence this case should
+  // return nullptr.
+  EXPECT_EQ(result, nullptr);
+}
+
 } // namespace


        

From llvmlistbot at llvm.org  Wed Jul  1 07:57:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 07:57:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ABI] Fix scalable vector mapping in
	ABITypeMapper (PR #206617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206617@github.com>
Message-ID: <6a452acf.6c8668f2.2eb228.9a33@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/206617

From llvmlistbot at llvm.org  Wed Jul  1 08:01:33 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 08:01:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Generalize broadcast lowering
 for single-element vector to nD (PR #206501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206501@github.com>
Message-ID: <6a452bcd.97bb2efc.2241ed.5fe3@mx.google.com>


================
@@ -95,6 +95,42 @@ func.func @broadcast_stretch(%arg0: vector<1xf32>) -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
+// CHECK-LABEL: func @broadcast_vec2d_from_vec1d_single_element
+// CHECK-SAME: %[[A:.*0]]: vector<1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0] : f32 from vector<1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<16x1xf32>
+// CHECK:      return %[[T1]] : vector<16x1xf32>
+
+func.func @broadcast_vec2d_from_vec1d_single_element(%arg0: vector<1xf32>) -> vector<16x1xf32> {
+  %0 = vector.broadcast %arg0 : vector<1xf32> to vector<16x1xf32>
+  return %0 : vector<16x1xf32>
+}
+
+// CHECK-LABEL: func @broadcast_vec2d_from_vec2d_single_element
+// CHECK-SAME: %[[A:.*0]]: vector<1x1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0, 0] : f32 from vector<1x1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<4x3xf32>
+// CHECK:      return %[[T1]] : vector<4x3xf32>
+
+func.func @broadcast_vec2d_from_vec2d_single_element(%arg0: vector<1x1xf32>) -> vector<4x3xf32> {
+  %0 = vector.broadcast %arg0 : vector<1x1xf32> to vector<4x3xf32>
+  return %0 : vector<4x3xf32>
+}
+
+// CHECK-LABEL: func @broadcast_single_elem_scalable_src
+// CHECK-SAME: %[[A:.*0]]: vector<[1]xf32>
+// CHECK:      %[[U0:.*]] = ub.poison : vector<4x[1]xf32>
+// CHECK:      %[[T0:.*]] = vector.insert %[[A]], %[[U0]] [0] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T1:.*]] = vector.insert %[[A]], %[[T0]] [1] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T2:.*]] = vector.insert %[[A]], %[[T1]] [2] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T3:.*]] = vector.insert %[[A]], %[[T2]] [3] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      return %[[T3]] : vector<4x[1]xf32>
+
+func.func @broadcast_single_elem_scalable_src(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
----------------
banach-space wrote:

[ultra-nit] We tend to use "unit scalable" for `[1]`
```suggestion
func.func @broadcast_vec2d_from_vec1d_unit_scalable_dim(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
```

https://github.com/llvm/llvm-project/pull/206501

From llvmlistbot at llvm.org  Wed Jul  1 08:02:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 08:02:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Generalize broadcast lowering
 for single-element vector to nD (PR #206501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206501@github.com>
Message-ID: <6a452c0c.fdc477d6.2b77d8.a42b@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you!

I left one ultra-nit, feel free to ignore.

https://github.com/llvm/llvm-project/pull/206501

From llvmlistbot at llvm.org  Wed Jul  1 08:09:47 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Wed, 01 Jul 2026 08:09:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Generalize broadcast lowering
 for single-element vector to nD (PR #206501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206501@github.com>
Message-ID: <6a452dbb.576b3af0.3bb11f.a61c@mx.google.com>


https://github.com/akroviakov updated https://github.com/llvm/llvm-project/pull/206501

>From 29026889aebcb0fadb8766d0c70983eef5f7d04c Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Mon, 29 Jun 2026 14:05:40 +0000
Subject: [PATCH 1/4] [MLIR][Vector] Generalize broadcast lowering for unit
 vector to nD

---
 .../Transforms/LowerVectorBroadcast.cpp       |  6 ++--
 .../VectorToLLVM/vector-to-llvm.mlir          | 23 ++++++------
 .../vector-broadcast-lowering-transforms.mlir | 36 +++++++++++++++++++
 3 files changed, 50 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
index 66dd7c8f36e6b..d2da24b366e83 100644
--- a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
@@ -52,9 +52,9 @@ class BroadcastOpLowering : public OpRewritePattern<vector::BroadcastOp> {
     int64_t srcRank = srcType.getRank();
     int64_t dstRank = dstType.getRank();
 
-    // Here we are broadcasting to a rank-1 vector. Ensure that the source is a
-    // scalar.
-    if (srcRank <= 1 && dstRank == 1) {
+    // A broadcast from a single-element source vector is equivalent to scalar
+    // broadcasting to an nD shape.
+    if (srcType.getNumElements() == 1 && !srcType.isScalable()) {
       SmallVector<int64_t> fullRankPosition(srcRank, 0);
       Value ext = vector::ExtractOp::create(rewriter, loc, op.getSource(),
                                             fullRankPosition);
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
index 77f60b3172296..f2e9b606acd4d 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
@@ -197,17 +197,16 @@ func.func @broadcast_vec2d_from_vec0d(%arg0: vector<f32>) -> vector<3x2xf32> {
 // CHECK-LABEL: @broadcast_vec2d_from_vec0d(
 // CHECK-SAME:  %[[A:.*]]: vector<f32>)
 //       CHECK: %[[T0:.*]] = builtin.unrealized_conversion_cast %[[A]] : vector<f32> to vector<1xf32>
-//       CHECK: %[[T1:.*]] = ub.poison : vector<3x2xf32>
-//       CHECK: %[[T2:.*]] = builtin.unrealized_conversion_cast %[[T1]] : vector<3x2xf32> to !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T4:.*]] = llvm.mlir.constant(0 : i64) : i64
-//       CHECK: %[[T5:.*]] = llvm.extractelement %[[T0]][%[[T4]] : i64] : vector<1xf32>
-//       CHECK: %[[T6Insert:.*]] = llvm.insertelement %[[T5]]
-//       CHECK: %[[T6:.*]] = llvm.shufflevector %[[T6Insert]]
-//       CHECK: %[[T7:.*]] = llvm.insertvalue %[[T6]], %[[T2]][0] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T8:.*]] = llvm.insertvalue %[[T6]], %[[T7]][1] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T9:.*]] = llvm.insertvalue %[[T6]], %[[T8]][2] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T10:.*]] = builtin.unrealized_conversion_cast %[[T9]] : !llvm.array<3 x vector<2xf32>> to vector<3x2xf32>
-//       CHECK: return %[[T10]] : vector<3x2xf32>
+//       CHECK: %[[T1:.*]] = llvm.mlir.constant(0 : i64) : i64
+//       CHECK: %[[T2:.*]] = llvm.extractelement %[[T0]][%[[T1]] : i64] : vector<1xf32>
+//       CHECK: %[[T3:.*]] = llvm.mlir.poison : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T4Insert:.*]] = llvm.insertelement %[[T2]]
+//       CHECK: %[[T4:.*]] = llvm.shufflevector %[[T4Insert]]
+//       CHECK: %[[T5:.*]] = llvm.insertvalue %[[T4]], %[[T3]][0] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T6:.*]] = llvm.insertvalue %[[T4]], %[[T5]][1] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T7:.*]] = llvm.insertvalue %[[T4]], %[[T6]][2] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T8:.*]] = builtin.unrealized_conversion_cast %[[T7]] : !llvm.array<3 x vector<2xf32>> to vector<3x2xf32>
+//       CHECK: return %[[T8]] : vector<3x2xf32>
 
 // -----
 
@@ -1499,7 +1498,7 @@ func.func @constant_mask_2d() -> vector<4x4xi1> {
 }
 
 // CHECK-LABEL: func @constant_mask_2d
-// CHECK: %[[VAL_0:.*]] = arith.constant 
+// CHECK: %[[VAL_0:.*]] = arith.constant
 // CHECK-SAME{LITERAL}: dense<[[true, true, false, false], [true, true, false, false], [false, false, false, false], [false, false, false, false]]> : vector<4x4xi1>
 // CHECK: return %[[VAL_0]] : vector<4x4xi1>
 
diff --git a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
index d5e344393b217..8de08e7e26417 100644
--- a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
@@ -95,6 +95,42 @@ func.func @broadcast_stretch(%arg0: vector<1xf32>) -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
+// CHECK-LABEL: func @broadcast_single_elem_vec1d_to_vec2d
+// CHECK-SAME: %[[A:.*0]]: vector<1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0] : f32 from vector<1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<16x1xf32>
+// CHECK:      return %[[T1]] : vector<16x1xf32>
+
+func.func @broadcast_single_elem_vec1d_to_vec2d(%arg0: vector<1xf32>) -> vector<16x1xf32> {
+  %0 = vector.broadcast %arg0 : vector<1xf32> to vector<16x1xf32>
+  return %0 : vector<16x1xf32>
+}
+
+// CHECK-LABEL: func @broadcast_single_elem_vec2d_to_vec2d
+// CHECK-SAME: %[[A:.*0]]: vector<1x1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0, 0] : f32 from vector<1x1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<4x3xf32>
+// CHECK:      return %[[T1]] : vector<4x3xf32>
+
+func.func @broadcast_single_elem_vec2d_to_vec2d(%arg0: vector<1x1xf32>) -> vector<4x3xf32> {
+  %0 = vector.broadcast %arg0 : vector<1x1xf32> to vector<4x3xf32>
+  return %0 : vector<4x3xf32>
+}
+
+// CHECK-LABEL: func @broadcast_single_elem_scalable_src
+// CHECK-SAME: %[[A:.*0]]: vector<[1]xf32>
+// CHECK:      %[[U0:.*]] = ub.poison : vector<4x[1]xf32>
+// CHECK:      %[[T0:.*]] = vector.insert %[[A]], %[[U0]] [0] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T1:.*]] = vector.insert %[[A]], %[[T0]] [1] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T2:.*]] = vector.insert %[[A]], %[[T1]] [2] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T3:.*]] = vector.insert %[[A]], %[[T2]] [3] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      return %[[T3]] : vector<4x[1]xf32>
+
+func.func @broadcast_single_elem_scalable_src(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
+  %0 = vector.broadcast %arg0 : vector<[1]xf32> to vector<4x[1]xf32>
+  return %0 : vector<4x[1]xf32>
+}
+
 // CHECK-LABEL: func @broadcast_stretch_at_start
 // CHECK-SAME: %[[A:.*0]]: vector<1x4xf32>
 // CHECK:      %[[U0:.*]] = ub.poison : vector<3x4xf32>

>From 6d225601f0c576c329df64ddc83bfb8e272f11ab Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Tue, 30 Jun 2026 09:10:25 +0000
Subject: [PATCH 2/4] Shorter comment

---
 mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
index d2da24b366e83..a78c31367ee28 100644
--- a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
@@ -52,8 +52,7 @@ class BroadcastOpLowering : public OpRewritePattern<vector::BroadcastOp> {
     int64_t srcRank = srcType.getRank();
     int64_t dstRank = dstType.getRank();
 
-    // A broadcast from a single-element source vector is equivalent to scalar
-    // broadcasting to an nD shape.
+    // Single-element fixed-size source: extract the scalar and broadcast it.
     if (srcType.getNumElements() == 1 && !srcType.isScalable()) {
       SmallVector<int64_t> fullRankPosition(srcRank, 0);
       Value ext = vector::ExtractOp::create(rewriter, loc, op.getSource(),

>From 2ed250a72ee15982f7a349ae4e35eb9574a63309 Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Tue, 30 Jun 2026 15:40:42 +0000
Subject: [PATCH 3/4] Rename tests

---
 .../Vector/vector-broadcast-lowering-transforms.mlir      | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
index 8de08e7e26417..d7fb0d95bf1e7 100644
--- a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
@@ -95,24 +95,24 @@ func.func @broadcast_stretch(%arg0: vector<1xf32>) -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
-// CHECK-LABEL: func @broadcast_single_elem_vec1d_to_vec2d
+// CHECK-LABEL: func @broadcast_vec2d_from_vec1d_single_element
 // CHECK-SAME: %[[A:.*0]]: vector<1xf32>
 // CHECK:      %[[T0:.*]] = vector.extract %[[A]][0] : f32 from vector<1xf32>
 // CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<16x1xf32>
 // CHECK:      return %[[T1]] : vector<16x1xf32>
 
-func.func @broadcast_single_elem_vec1d_to_vec2d(%arg0: vector<1xf32>) -> vector<16x1xf32> {
+func.func @broadcast_vec2d_from_vec1d_single_element(%arg0: vector<1xf32>) -> vector<16x1xf32> {
   %0 = vector.broadcast %arg0 : vector<1xf32> to vector<16x1xf32>
   return %0 : vector<16x1xf32>
 }
 
-// CHECK-LABEL: func @broadcast_single_elem_vec2d_to_vec2d
+// CHECK-LABEL: func @broadcast_vec2d_from_vec2d_single_element
 // CHECK-SAME: %[[A:.*0]]: vector<1x1xf32>
 // CHECK:      %[[T0:.*]] = vector.extract %[[A]][0, 0] : f32 from vector<1x1xf32>
 // CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<4x3xf32>
 // CHECK:      return %[[T1]] : vector<4x3xf32>
 
-func.func @broadcast_single_elem_vec2d_to_vec2d(%arg0: vector<1x1xf32>) -> vector<4x3xf32> {
+func.func @broadcast_vec2d_from_vec2d_single_element(%arg0: vector<1x1xf32>) -> vector<4x3xf32> {
   %0 = vector.broadcast %arg0 : vector<1x1xf32> to vector<4x3xf32>
   return %0 : vector<4x3xf32>
 }

>From 53d35b607b90c2e67504f01dc5f5f257b72bc833 Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Wed, 1 Jul 2026 15:09:20 +0000
Subject: [PATCH 4/4] Rename test

---
 .../Dialect/Vector/vector-broadcast-lowering-transforms.mlir  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
index d7fb0d95bf1e7..5b193c5e25ca0 100644
--- a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
@@ -117,7 +117,7 @@ func.func @broadcast_vec2d_from_vec2d_single_element(%arg0: vector<1x1xf32>) ->
   return %0 : vector<4x3xf32>
 }
 
-// CHECK-LABEL: func @broadcast_single_elem_scalable_src
+// CHECK-LABEL: func @broadcast_vec2d_from_vec1d_unit_scalable_dim
 // CHECK-SAME: %[[A:.*0]]: vector<[1]xf32>
 // CHECK:      %[[U0:.*]] = ub.poison : vector<4x[1]xf32>
 // CHECK:      %[[T0:.*]] = vector.insert %[[A]], %[[U0]] [0] : vector<[1]xf32> into vector<4x[1]xf32>
@@ -126,7 +126,7 @@ func.func @broadcast_vec2d_from_vec2d_single_element(%arg0: vector<1x1xf32>) ->
 // CHECK:      %[[T3:.*]] = vector.insert %[[A]], %[[T2]] [3] : vector<[1]xf32> into vector<4x[1]xf32>
 // CHECK:      return %[[T3]] : vector<4x[1]xf32>
 
-func.func @broadcast_single_elem_scalable_src(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
+func.func @broadcast_vec2d_from_vec1d_unit_scalable_dim(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
   %0 = vector.broadcast %arg0 : vector<[1]xf32> to vector<4x[1]xf32>
   return %0 : vector<4x[1]xf32>
 }


From llvmlistbot at llvm.org  Wed Jul  1 08:05:12 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 01 Jul 2026 08:05:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] add consistent stride
 verification to masked load/store and gather/scatter ops (PR #205869)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205869@github.com>
Message-ID: <6a452ca8.f14cf1a5.9019b.3551@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM!

Thank you for the ping and sorry for the delay - the review backlog is very dense :(

https://github.com/llvm/llvm-project/pull/205869

From llvmlistbot at llvm.org  Wed Jul  1 08:11:34 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Wed, 01 Jul 2026 08:11:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Generalize broadcast lowering
 for single-element vector to nD (PR #206501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206501@github.com>
Message-ID: <6a452e26.576b3af0.3bb11f.d049@mx.google.com>


https://github.com/akroviakov auto_merge_enabled https://github.com/llvm/llvm-project/pull/206501

From llvmlistbot at llvm.org  Wed Jul  1 08:11:38 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 01 Jul 2026 08:11:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] add consistent stride
 verification to masked load/store and gather/scatter ops (PR #205869)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205869@github.com>
Message-ID: <6a452e2a.481afadf.29a24b.f60d@mx.google.com>


FedericoBruzzone wrote:

> Thank you for the ping and sorry for the delay - the review backlog is very dense :(

You don't need to apologize! It's partly my fault too that it's full :'D
I'll try to reduce the number of PRs.

https://github.com/llvm/llvm-project/pull/205869

From llvmlistbot at llvm.org  Wed Jul  1 08:12:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 08:12:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a452e64.e9d484b9.34b6c.32a9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Hugo Trachino (nujaa)

<details>
<summary>Changes</summary>

Applying FlattenElementwiseLinalgOp on Broadcasted linalg would break. This transformation is not valid for those usecase, so I added a condition to exit gracefully instead.

---
Full diff: https://github.com/llvm/llvm-project/pull/207005.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp (+8) 
- (modified) mlir/test/Dialect/Linalg/flatten-elementwise.mlir (+53) 


``````````diff
diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index f44693096b26b..d39c1cb0e4a20 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4297,6 +4297,14 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return DiagnosedSilenceableFailure::success();
   }
 
+  // Bail out on linalgs with broadcasting semantics
+  if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
+        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+      })) {
+    results.push_back(target);
+    return DiagnosedSilenceableFailure::success();
+  }
+
   // Attempt to flatten all dims to one.
   ReassociationIndices reassociation(target.getNumLoops());
   std::iota(reassociation.begin(), reassociation.end(), 0);
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 9fe50a521d2d8..2ca08215a166a 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -118,3 +118,56 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<1x2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Wed Jul  1 08:21:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 08:21:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 0c2d34d - Reland "[mlir][nvvm] extract NVVM
 inliner interface from LLVM dialect" with flang fixes (#206782)
Message-ID: <6a453081.7d41f3a1.2f50d3.368e@mx.google.com>


Author: Jeremy Kun
Date: 2026-07-01T08:21:32-07:00
New Revision: 0c2d34d221b74aa5a8d3afbd17aee2d0e8c1c256

URL: https://github.com/llvm/llvm-project/commit/0c2d34d221b74aa5a8d3afbd17aee2d0e8c1c256
DIFF: https://github.com/llvm/llvm-project/commit/0c2d34d221b74aa5a8d3afbd17aee2d0e8c1c256.diff

LOG: Reland "[mlir][nvvm] extract NVVM inliner interface from LLVM dialect" with flang fixes (#206782)

Added: 
    

Modified: 
    flang/lib/Optimizer/Transforms/CMakeLists.txt
    mlir/include/mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h
    mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
    mlir/lib/Dialect/LLVMIR/Transforms/CMakeLists.txt
    mlir/lib/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.cpp
    mlir/lib/RegisterAllDialects.cpp
    utils/bazel/llvm-project-overlay/mlir/BUILD.bazel

Removed: 
    


################################################################################
diff  --git a/flang/lib/Optimizer/Transforms/CMakeLists.txt b/flang/lib/Optimizer/Transforms/CMakeLists.txt
index 5a3059ebbd97f..2eba2ac314fd5 100644
--- a/flang/lib/Optimizer/Transforms/CMakeLists.txt
+++ b/flang/lib/Optimizer/Transforms/CMakeLists.txt
@@ -87,6 +87,7 @@ add_flang_library(FIRTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
   MLIRMathTransforms
+  MLIRNVVMDialect
   MLIROpenACCDialect
   MLIROpenACCToLLVMIRTranslation
   MLIROpenMPDialect

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h b/mlir/include/mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h
index 69cc2e32285b6..472a049d567fd 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h
+++ b/mlir/include/mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h
@@ -24,12 +24,6 @@ void registerInlinerInterface(DialectRegistry &registry);
 
 } // namespace LLVM
 
-namespace NVVM {
-/// Register the `NVVMInlinerInterface` implementation of
-/// `DialectInlinerInterface` with the NVVM dialect.
-void registerInlinerInterface(DialectRegistry &registry);
-} // namespace NVVM
-
 } // namespace mlir
 
 #endif // MLIR_DIALECT_LLVMIR_TRANSFORMS_INLINERINTERFACEIMPL_H

diff  --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 13de1e7f2b2ad..14406ab55f138 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -28,6 +28,7 @@
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/Types.h"
+#include "mlir/Transforms/InliningUtils.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/IRBuilder.h"
@@ -6330,6 +6331,15 @@ LogicalResult Tcgen05LdRedOp::verify() {
 // NVVMDialect initialization, type parsing, and registration.
 //===----------------------------------------------------------------------===//
 
+namespace {
+struct NVVMInlinerInterface final : DialectInlinerInterface {
+  using DialectInlinerInterface::DialectInlinerInterface;
+  bool isLegalToInline(Operation *, Region *, bool, IRMapping &) const final {
+    return true;
+  }
+};
+} // namespace
+
 // TODO: This should be the llvm.nvvm dialect once this is supported.
 void NVVMDialect::initialize() {
   addOperations<
@@ -6344,6 +6354,7 @@ void NVVMDialect::initialize() {
   // Support unknown operations because not all NVVM operations are
   // registered.
   allowUnknownOperations();
+  addInterfaces<NVVMInlinerInterface>();
   declarePromisedInterface<ConvertToLLVMPatternInterface, NVVMDialect>();
   declarePromisedInterface<gpu::TargetAttrInterface, NVVMTargetAttr>();
 }

diff  --git a/mlir/lib/Dialect/LLVMIR/Transforms/CMakeLists.txt b/mlir/lib/Dialect/LLVMIR/Transforms/CMakeLists.txt
index 63046b1ced914..fff166d39dad9 100644
--- a/mlir/lib/Dialect/LLVMIR/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/LLVMIR/Transforms/CMakeLists.txt
@@ -17,6 +17,5 @@ add_mlir_dialect_library(MLIRLLVMIRTransforms
   MLIRLLVMDialect
   MLIRPass
   MLIRTransforms
-  MLIRNVVMDialect
   MLIROpenMPDialect
   )

diff  --git a/mlir/lib/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.cpp b/mlir/lib/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.cpp
index 680fafcc099f2..657835ae54149 100644
--- a/mlir/lib/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.cpp
@@ -14,7 +14,6 @@
 #include "mlir/Dialect/LLVMIR/Transforms/InlinerInterfaceImpl.h"
 #include "mlir/Analysis/SliceWalk.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
-#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/Interfaces/DataLayoutInterfaces.h"
 #include "mlir/Interfaces/ViewLikeInterface.h"
@@ -861,9 +860,3 @@ void mlir::LLVM::registerInlinerInterface(DialectRegistry &registry) {
     dialect->addInterfaces<LLVMInlinerInterface>();
   });
 }
-
-void mlir::NVVM::registerInlinerInterface(DialectRegistry &registry) {
-  registry.addExtension(+[](MLIRContext *ctx, NVVM::NVVMDialect *dialect) {
-    dialect->addInterfaces<LLVMInlinerInterface>();
-  });
-}

diff  --git a/mlir/lib/RegisterAllDialects.cpp b/mlir/lib/RegisterAllDialects.cpp
index 2f55296f424cd..974b5f533860a 100644
--- a/mlir/lib/RegisterAllDialects.cpp
+++ b/mlir/lib/RegisterAllDialects.cpp
@@ -176,7 +176,6 @@ void mlir::registerAllDialects(DialectRegistry &registry) {
   gpu::registerIndexedAccessOpInterfaceExternalModels(registry);
   gpu::registerValueBoundsOpInterfaceExternalModels(registry);
   LLVM::registerInlinerInterface(registry);
-  NVVM::registerInlinerInterface(registry);
   linalg::registerAllDialectInterfaceImplementations(registry);
   linalg::registerRuntimeVerifiableOpInterfaceExternalModels(registry);
   memref::registerAllocationOpInterfaceExternalModels(registry);

diff  --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
index 12b8ee89ffb04..e9bb52e88ace7 100644
--- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
@@ -5570,7 +5570,6 @@ cc_library(
         ":LLVMDialect",
         ":LLVMIRTransformsLegalizeForExport",
         ":LLVMPassIncGen",
-        ":NVVMDialect",
         ":Pass",
         ":TransformUtils",
         ":ViewLikeInterface",
@@ -6412,6 +6411,7 @@ cc_library(
         ":GPUDialect",
         ":IR",
         ":InferIntRangeInterface",
+        ":InliningUtils",
         ":LLVMDialect",
         ":NVVMOpsIncGen",
         ":NVVMRequiresSMTraitsIncGen",


        

From llvmlistbot at llvm.org  Wed Jul  1 08:21:40 2026
From: llvmlistbot at llvm.org (Jeremy Kun)
Date: Wed, 01 Jul 2026 08:21:40 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] Reland "[mlir][nvvm] extract
 NVVM inliner interface from LLVM dialect" with flang fixes (PR #206782)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206782@github.com>
Message-ID: <6a453084.e9d484b9.34b6c.269f@mx.google.com>


https://github.com/j2kun closed https://github.com/llvm/llvm-project/pull/206782

From llvmlistbot at llvm.org  Wed Jul  1 08:21:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 08:21:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] eff826e - [MLIR][Vector] Generalize broadcast
 lowering for single-element vector to nD (#206501)
Message-ID: <6a453085.3157459c.1bf8d3.64b6@mx.google.com>


Author: Artem Kroviakov
Date: 2026-07-01T15:21:35Z
New Revision: eff826e91dc7321eb38821da600dd411707e67fa

URL: https://github.com/llvm/llvm-project/commit/eff826e91dc7321eb38821da600dd411707e67fa
DIFF: https://github.com/llvm/llvm-project/commit/eff826e91dc7321eb38821da600dd411707e67fa.diff

LOG: [MLIR][Vector] Generalize broadcast lowering for single-element vector to nD (#206501)

Added: 
    

Modified: 
    mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
    mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
    mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
index 66dd7c8f36e6b..a78c31367ee28 100644
--- a/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/LowerVectorBroadcast.cpp
@@ -52,9 +52,8 @@ class BroadcastOpLowering : public OpRewritePattern<vector::BroadcastOp> {
     int64_t srcRank = srcType.getRank();
     int64_t dstRank = dstType.getRank();
 
-    // Here we are broadcasting to a rank-1 vector. Ensure that the source is a
-    // scalar.
-    if (srcRank <= 1 && dstRank == 1) {
+    // Single-element fixed-size source: extract the scalar and broadcast it.
+    if (srcType.getNumElements() == 1 && !srcType.isScalable()) {
       SmallVector<int64_t> fullRankPosition(srcRank, 0);
       Value ext = vector::ExtractOp::create(rewriter, loc, op.getSource(),
                                             fullRankPosition);

diff  --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
index 1fc80ffb5cc50..094353119e493 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
@@ -197,17 +197,16 @@ func.func @broadcast_vec2d_from_vec0d(%arg0: vector<f32>) -> vector<3x2xf32> {
 // CHECK-LABEL: @broadcast_vec2d_from_vec0d(
 // CHECK-SAME:  %[[A:.*]]: vector<f32>)
 //       CHECK: %[[T0:.*]] = builtin.unrealized_conversion_cast %[[A]] : vector<f32> to vector<1xf32>
-//       CHECK: %[[T1:.*]] = ub.poison : vector<3x2xf32>
-//       CHECK: %[[T2:.*]] = builtin.unrealized_conversion_cast %[[T1]] : vector<3x2xf32> to !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T4:.*]] = llvm.mlir.constant(0 : i64) : i64
-//       CHECK: %[[T5:.*]] = llvm.extractelement %[[T0]][%[[T4]] : i64] : vector<1xf32>
-//       CHECK: %[[T6Insert:.*]] = llvm.insertelement %[[T5]]
-//       CHECK: %[[T6:.*]] = llvm.shufflevector %[[T6Insert]]
-//       CHECK: %[[T7:.*]] = llvm.insertvalue %[[T6]], %[[T2]][0] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T8:.*]] = llvm.insertvalue %[[T6]], %[[T7]][1] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T9:.*]] = llvm.insertvalue %[[T6]], %[[T8]][2] : !llvm.array<3 x vector<2xf32>>
-//       CHECK: %[[T10:.*]] = builtin.unrealized_conversion_cast %[[T9]] : !llvm.array<3 x vector<2xf32>> to vector<3x2xf32>
-//       CHECK: return %[[T10]] : vector<3x2xf32>
+//       CHECK: %[[T1:.*]] = llvm.mlir.constant(0 : i64) : i64
+//       CHECK: %[[T2:.*]] = llvm.extractelement %[[T0]][%[[T1]] : i64] : vector<1xf32>
+//       CHECK: %[[T3:.*]] = llvm.mlir.poison : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T4Insert:.*]] = llvm.insertelement %[[T2]]
+//       CHECK: %[[T4:.*]] = llvm.shufflevector %[[T4Insert]]
+//       CHECK: %[[T5:.*]] = llvm.insertvalue %[[T4]], %[[T3]][0] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T6:.*]] = llvm.insertvalue %[[T4]], %[[T5]][1] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T7:.*]] = llvm.insertvalue %[[T4]], %[[T6]][2] : !llvm.array<3 x vector<2xf32>>
+//       CHECK: %[[T8:.*]] = builtin.unrealized_conversion_cast %[[T7]] : !llvm.array<3 x vector<2xf32>> to vector<3x2xf32>
+//       CHECK: return %[[T8]] : vector<3x2xf32>
 
 // -----
 
@@ -1499,7 +1498,7 @@ func.func @constant_mask_2d() -> vector<4x4xi1> {
 }
 
 // CHECK-LABEL: func @constant_mask_2d
-// CHECK: %[[VAL_0:.*]] = arith.constant 
+// CHECK: %[[VAL_0:.*]] = arith.constant
 // CHECK-SAME{LITERAL}: dense<[[true, true, false, false], [true, true, false, false], [false, false, false, false], [false, false, false, false]]> : vector<4x4xi1>
 // CHECK: return %[[VAL_0]] : vector<4x4xi1>
 

diff  --git a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
index d5e344393b217..5b193c5e25ca0 100644
--- a/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-broadcast-lowering-transforms.mlir
@@ -95,6 +95,42 @@ func.func @broadcast_stretch(%arg0: vector<1xf32>) -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
+// CHECK-LABEL: func @broadcast_vec2d_from_vec1d_single_element
+// CHECK-SAME: %[[A:.*0]]: vector<1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0] : f32 from vector<1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<16x1xf32>
+// CHECK:      return %[[T1]] : vector<16x1xf32>
+
+func.func @broadcast_vec2d_from_vec1d_single_element(%arg0: vector<1xf32>) -> vector<16x1xf32> {
+  %0 = vector.broadcast %arg0 : vector<1xf32> to vector<16x1xf32>
+  return %0 : vector<16x1xf32>
+}
+
+// CHECK-LABEL: func @broadcast_vec2d_from_vec2d_single_element
+// CHECK-SAME: %[[A:.*0]]: vector<1x1xf32>
+// CHECK:      %[[T0:.*]] = vector.extract %[[A]][0, 0] : f32 from vector<1x1xf32>
+// CHECK:      %[[T1:.*]] = vector.broadcast %[[T0]] : f32 to vector<4x3xf32>
+// CHECK:      return %[[T1]] : vector<4x3xf32>
+
+func.func @broadcast_vec2d_from_vec2d_single_element(%arg0: vector<1x1xf32>) -> vector<4x3xf32> {
+  %0 = vector.broadcast %arg0 : vector<1x1xf32> to vector<4x3xf32>
+  return %0 : vector<4x3xf32>
+}
+
+// CHECK-LABEL: func @broadcast_vec2d_from_vec1d_unit_scalable_dim
+// CHECK-SAME: %[[A:.*0]]: vector<[1]xf32>
+// CHECK:      %[[U0:.*]] = ub.poison : vector<4x[1]xf32>
+// CHECK:      %[[T0:.*]] = vector.insert %[[A]], %[[U0]] [0] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T1:.*]] = vector.insert %[[A]], %[[T0]] [1] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T2:.*]] = vector.insert %[[A]], %[[T1]] [2] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      %[[T3:.*]] = vector.insert %[[A]], %[[T2]] [3] : vector<[1]xf32> into vector<4x[1]xf32>
+// CHECK:      return %[[T3]] : vector<4x[1]xf32>
+
+func.func @broadcast_vec2d_from_vec1d_unit_scalable_dim(%arg0: vector<[1]xf32>) -> vector<4x[1]xf32> {
+  %0 = vector.broadcast %arg0 : vector<[1]xf32> to vector<4x[1]xf32>
+  return %0 : vector<4x[1]xf32>
+}
+
 // CHECK-LABEL: func @broadcast_stretch_at_start
 // CHECK-SAME: %[[A:.*0]]: vector<1x4xf32>
 // CHECK:      %[[U0:.*]] = ub.poison : vector<3x4xf32>


        

From llvmlistbot at llvm.org  Wed Jul  1 08:21:46 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Wed, 01 Jul 2026 08:21:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Generalize broadcast lowering
 for single-element vector to nD (PR #206501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206501@github.com>
Message-ID: <6a45308a.bf38ae28.12214b.9121@mx.google.com>


https://github.com/akroviakov closed https://github.com/llvm/llvm-project/pull/206501

From llvmlistbot at llvm.org  Wed Jul  1 08:24:03 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 01 Jul 2026 08:24:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a453113.1a4df1a5.3108a4.7505@mx.google.com>


https://github.com/nujaa updated https://github.com/llvm/llvm-project/pull/207005

>From 28fcbd2515983a70bd149a09daad1bca3b4ca77c Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Wed, 1 Jul 2026 21:08:28 +0800
Subject: [PATCH] [MLIR][Linalg] Fix FlattenElementwiseOp on broadcasted
 linalgs

---
 .../TransformOps/LinalgTransformOps.cpp       |  8 +++
 .../Dialect/Linalg/flatten-elementwise.mlir   | 53 +++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index f44693096b26b..d39c1cb0e4a20 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4297,6 +4297,14 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return DiagnosedSilenceableFailure::success();
   }
 
+  // Bail out on linalgs with broadcasting semantics
+  if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
+        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+      })) {
+    results.push_back(target);
+    return DiagnosedSilenceableFailure::success();
+  }
+
   // Attempt to flatten all dims to one.
   ReassociationIndices reassociation(target.getNumLoops());
   std::iota(reassociation.begin(), reassociation.end(), 0);
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 9fe50a521d2d8..2ca08215a166a 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -118,3 +118,56 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<1x2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:         %[[RESULT:.*]] = linalg.generic
+// CHECK-NEXT:         return %[[RESULT]]
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Wed Jul  1 08:51:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 08:51:50 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a453796.f06f4994.2ae0bf.265f@mx.google.com>


================
@@ -5527,6 +5525,25 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
       return failure();
 
     builder.restoreIP(*afterIP);
+
+    // Postfix: v = select(success, D, old) — captures new value of x.
+    if (isPostfixCapture && !isFailOnly) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::Instruction *cmpxchgInst = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
+          cmpxchgInst = &inst;
+          break;
+        }
+      }
+      assert(cmpxchgInst && "expected cmpxchg instruction");
----------------
SunilKuravinakop wrote:

I have addressed this in the latest push. Can you please review the code?

https://github.com/llvm/llvm-project/pull/202315

From llvmlistbot at llvm.org  Wed Jul  1 08:54:13 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Wed, 01 Jul 2026 08:54:13 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a453825.eb24c6ea.3c92b7.2a53@mx.google.com>


https://github.com/skatrak closed https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Wed Jul  1 09:46:00 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Wed, 01 Jul 2026 09:46:00 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add opt-in
 query caching; use it in flang LICM (PR #207030)
Message-ID: <llvm.org/llvm/llvm-project/pull/207030@github.com>


https://github.com/vzakhari created https://github.com/llvm/llvm-project/pull/207030

Add a generic, opt-in query-caching capability to the mlir::AliasAnalysis aggregator. The aggregator gains a public RAII QueryCacheScope and private enable/disableQueryCaching() fan-out; the Concept gains enableQueryCaching()/ disableQueryCaching() and the Model forwards them to implementations that provide enableSourceCache()/disableSourceCache() (detected via llvm::is_detected) and no-ops otherwise, so existing implementations need no changes and transformation passes never name a concrete implementation.

The FIR AliasAnalysis implements the hook by memoizing getSource(): getSource() becomes a thin wrapper over getSourceImpl() that consults a frozen-snapshot cache keyed on (value, flags) when caching is enabled.

Flang's LICM enables caching for the duration of the pass via QueryCacheScope. This is sound because LICM only moves operations and getSource()'s position-dependent inputs are already frozen for the pass run by the AliasAnalysis's own dominance/scope caches, making caching observationally equivalent to no caching across the hoists.

Assisted by Cursor

>From 585f5854516df94c0ee343486b184e7868df4d8f Mon Sep 17 00:00:00 2001
From: Slava Zakharin <szakharin at nvidia.com>
Date: Tue, 30 Jun 2026 12:04:20 -0700
Subject: [PATCH] [mlir][AliasAnalysis] Add opt-in query caching; use it in
 flang LICM

Add a generic, opt-in query-caching capability to the mlir::AliasAnalysis
aggregator. The aggregator gains a public RAII QueryCacheScope and private
enable/disableQueryCaching() fan-out; the Concept gains enableQueryCaching()/
disableQueryCaching() and the Model forwards them to implementations that
provide enableSourceCache()/disableSourceCache() (detected via
llvm::is_detected) and no-ops otherwise, so existing implementations need no
changes and transformation passes never name a concrete implementation.

The FIR AliasAnalysis implements the hook by memoizing getSource(): getSource()
becomes a thin wrapper over getSourceImpl() that consults a frozen-snapshot
cache keyed on (value, flags) when caching is enabled.

Flang's LICM enables caching for the duration of the pass via QueryCacheScope.
This is sound because LICM only moves operations and getSource()'s
position-dependent inputs are already frozen for the pass run by the
AliasAnalysis's own dominance/scope caches, making caching observationally
equivalent to no caching across the hoists.

Assisted by Cursor
---
 .../flang/Optimizer/Analysis/AliasAnalysis.h  | 38 ++++++++++++
 .../lib/Optimizer/Analysis/AliasAnalysis.cpp  | 29 +++++++++
 .../Transforms/LoopInvariantCodeMotion.cpp    | 10 ++++
 mlir/include/mlir/Analysis/AliasAnalysis.h    | 60 +++++++++++++++++++
 mlir/lib/Analysis/AliasAnalysis.cpp           | 10 ++++
 5 files changed, 147 insertions(+)

diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index 832634a708dba..a40632c23efce 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -20,6 +20,7 @@
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
+#include <utility>
 
 namespace fir {
 
@@ -351,10 +352,25 @@ struct AliasAnalysis {
   /// snapshots are not collected, and getSource performs only the
   /// SourceKind/origin classification without that bookkeeping side
   /// effect.
+  ///
+  /// When source caching is enabled (see enableSourceCache()), the result is
+  /// memoized keyed on (value, flags), and recursive sub-queries share the
+  /// same cache. The cache is a frozen snapshot: it is only valid while the IR
+  /// reachable from the queried values is not mutated in a way that would
+  /// change the source. Callers must scope caching no wider than such a region
+  /// (see mlir::AliasAnalysis::QueryCacheScope).
   fir::AliasAnalysis::Source getSource(mlir::Value,
                                        bool getLastInstantiationPoint = false,
                                        bool collectScopedOrigins = true);
 
+  /// Enable/disable memoization of getSource() results. These are invoked by
+  /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
+  /// QueryCacheScope) and are not meant to be called directly by passes.
+  /// enableSourceCache() turns on the frozen snapshot cache; disableSourceCache
+  /// clears it and turns it off.
+  void enableSourceCache();
+  void disableSourceCache();
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -370,6 +386,16 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  /// Compute the memory source of a value. This is the uncached
+  /// implementation of getSource(); getSource() is a thin wrapper that
+  /// memoizes the result when source caching is enabled.
+  fir::AliasAnalysis::Source getSourceImpl(mlir::Value v,
+                                           bool getLastInstantiationPoint,
+                                           bool collectScopedOrigins);
+
+  /// Clear the getSource() memoization cache.
+  void clearSourceCache() { getSourceCache.clear(); }
+
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
   /// classification (with collectScopedOrigins=false), then overrides
@@ -446,6 +472,18 @@ struct AliasAnalysis {
   /// functionHasMultipleScopes(); both true and false are cached so that
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
+
+  /// Opt-in memoization of getSource() results, keyed on the queried value
+  /// and the two boolean flags (getLastInstantiationPoint,
+  /// collectScopedOrigins) packed into the unsigned. Only consulted/populated
+  /// while \c sourceCacheEnabled is set. This is a frozen snapshot cache: it
+  /// has no automatic invalidation, so it is only enabled (via
+  /// enableSourceCache(), driven by mlir::AliasAnalysis::QueryCacheScope) for
+  /// the duration of a region in which the relevant IR is not mutated.
+  llvm::DenseMap<std::pair<mlir::Value, unsigned>, Source> getSourceCache;
+
+  /// Whether getSource() should consult/populate getSourceCache.
+  bool sourceCacheEnabled = false;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index 02a628968ddc4..f3e77a4fc0826 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
+#include <utility>
 
 using namespace mlir;
 
@@ -1108,9 +1109,37 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
+void AliasAnalysis::enableSourceCache() { sourceCacheEnabled = true; }
+
+void AliasAnalysis::disableSourceCache() {
+  sourceCacheEnabled = false;
+  clearSourceCache();
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
+  if (!sourceCacheEnabled)
+    return getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+
+  // Key on the queried value and the two boolean flags. Recursive sub-queries
+  // go through this same wrapper, so the whole walk is memoized.
+  std::pair<mlir::Value, unsigned> key{v,
+                                       (getLastInstantiationPoint ? 1u : 0u) |
+                                           (collectScopedOrigins ? 2u : 0u)};
+  auto it = getSourceCache.find(key);
+  if (it != getSourceCache.end())
+    return it->second;
+
+  Source source =
+      getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+  getSourceCache.try_emplace(key, source);
+  return source;
+}
+
+AliasAnalysis::Source
+AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
+                             bool collectScopedOrigins) {
   // If v is a pass-through block argument (see walkBlockArgPassThroughs),
   // continue from the underlying operand so the tracking loop below has a
   // defining op to chew on. Without this, a recursive query like the one in
diff --git a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
index 14fd5c6dd7fbc..2f640199252e4 100644
--- a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
+++ b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
@@ -268,6 +268,16 @@ void LoopInvariantCodeMotion::runOnOperation() {
   auto &aliasAnalysis = getAnalysis<AliasAnalysis>();
   aliasAnalysis.addAnalysisImplementation(fir::AliasAnalysis{});
 
+  // Enable alias-query caching for the duration of this pass. The scope
+  // enables caching on every registered implementation that supports it
+  // (the FIR AliasAnalysis memoizes getSource()) and no-ops on the rest, and
+  // reliably disables it on every exit. This is a frozen-snapshot cache with
+  // no listener: it is sound here because LICM only moves operations, and
+  // getSource()'s position-dependent inputs are already frozen for the pass
+  // run by the AliasAnalysis's own dominance/scope caches, so caching is
+  // observationally equivalent to no caching across the hoists.
+  mlir::AliasAnalysis::QueryCacheScope cacheScope(aliasAnalysis);
+
   std::function<bool(Operation *, LoopLikeOpInterface, bool)>
       shouldMoveOutOfLoop = [&](Operation *op, LoopLikeOpInterface loopLike,
                                 bool maybeConditionallyExecuted) {
diff --git a/mlir/include/mlir/Analysis/AliasAnalysis.h b/mlir/include/mlir/Analysis/AliasAnalysis.h
index 0fdcf01d9b6a3..71a08780c48a6 100644
--- a/mlir/include/mlir/Analysis/AliasAnalysis.h
+++ b/mlir/include/mlir/Analysis/AliasAnalysis.h
@@ -15,6 +15,7 @@
 #define MLIR_ANALYSIS_ALIASANALYSIS_H_
 
 #include "mlir/IR/Operation.h"
+#include "llvm/ADT/STLForwardCompat.h"
 
 namespace mlir {
 
@@ -187,8 +188,22 @@ struct AliasAnalysisTraits {
 
     /// Return the modify-reference behavior of `op` on `location`.
     virtual ModRefResult getModRef(Operation *op, Value location) = 0;
+
+    /// Enable opt-in caching of alias query results on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void enableQueryCaching() = 0;
+
+    /// Disable and clear any query caching on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void disableQueryCaching() = 0;
   };
 
+  /// Detection trait: true if `ImplT` provides `enableSourceCache()` /
+  /// `disableSourceCache()`. Mirrors the `has_is_invalidated` idiom used by
+  /// the pass analysis manager.
+  template <typename T>
+  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache());
+
   /// This class represents the `Model` of an alias analysis implementation
   /// `ImplT`. A model is instantiated for each alias analysis implementation
   /// to implement the `Concept` without the need for the derived
@@ -209,6 +224,17 @@ struct AliasAnalysisTraits {
       return impl.getModRef(op, location);
     }
 
+    /// Forward query-caching control to the implementation when it provides
+    /// the cache hooks; for implementations without them these are no-ops.
+    void enableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.enableSourceCache();
+    }
+    void disableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.disableSourceCache();
+    }
+
   private:
     ImplT impl;
   };
@@ -279,7 +305,41 @@ class AliasAnalysis {
   /// Return the modify-reference behavior of `op` on `location`.
   ModRefResult getModRef(Operation *op, Value location);
 
+  //===--------------------------------------------------------------------===//
+  // Query Caching
+  //===--------------------------------------------------------------------===//
+
+  /// RAII scope that enables opt-in query caching on all registered
+  /// implementations that support it for the duration of the scope, and
+  /// disables (and clears) it on destruction.
+  ///
+  /// This is the only entry point to query caching: enabling and disabling
+  /// are private so that caching is always paired with a guaranteed teardown.
+  /// A frozen (snapshot) cache is only valid while the IR is not mutated in a
+  /// way that affects the cached queries, so callers must keep the scope no
+  /// wider than such a no-mutation (or move-only) region.
+  class QueryCacheScope {
+  public:
+    explicit QueryCacheScope(AliasAnalysis &aa) : aa(aa) {
+      aa.enableQueryCaching();
+    }
+    ~QueryCacheScope() { aa.disableQueryCaching(); }
+
+    QueryCacheScope(const QueryCacheScope &) = delete;
+    QueryCacheScope &operator=(const QueryCacheScope &) = delete;
+    QueryCacheScope(QueryCacheScope &&) = delete;
+    QueryCacheScope &operator=(QueryCacheScope &&) = delete;
+
+  private:
+    AliasAnalysis &aa;
+  };
+
 private:
+  /// Enable/disable query caching on every registered implementation that
+  /// supports it. Private: use QueryCacheScope to guarantee teardown.
+  void enableQueryCaching();
+  void disableQueryCaching();
+
   /// A set of internal alias analysis implementations.
   SmallVector<std::unique_ptr<Concept>, 4> aliasImpls;
 };
diff --git a/mlir/lib/Analysis/AliasAnalysis.cpp b/mlir/lib/Analysis/AliasAnalysis.cpp
index 1382060f4dd90..9db48e18042cc 100644
--- a/mlir/lib/Analysis/AliasAnalysis.cpp
+++ b/mlir/lib/Analysis/AliasAnalysis.cpp
@@ -99,3 +99,13 @@ ModRefResult AliasAnalysis::getModRef(Operation *op, Value location) {
   }
   return result;
 }
+
+void AliasAnalysis::enableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->enableQueryCaching();
+}
+
+void AliasAnalysis::disableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->disableQueryCaching();
+}


From llvmlistbot at llvm.org  Wed Jul  1 09:46:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 09:46:36 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add opt-in
 query caching; use it in flang LICM (PR #207030)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207030@github.com>
Message-ID: <6a45446c.331483e7.17c361.1bb9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-fir-hlfir

Author: Slava Zakharin (vzakhari)

<details>
<summary>Changes</summary>

Add a generic, opt-in query-caching capability to the mlir::AliasAnalysis aggregator. The aggregator gains a public RAII QueryCacheScope and private enable/disableQueryCaching() fan-out; the Concept gains enableQueryCaching()/ disableQueryCaching() and the Model forwards them to implementations that provide enableSourceCache()/disableSourceCache() (detected via llvm::is_detected) and no-ops otherwise, so existing implementations need no changes and transformation passes never name a concrete implementation.

The FIR AliasAnalysis implements the hook by memoizing getSource(): getSource() becomes a thin wrapper over getSourceImpl() that consults a frozen-snapshot cache keyed on (value, flags) when caching is enabled.

Flang's LICM enables caching for the duration of the pass via QueryCacheScope. This is sound because LICM only moves operations and getSource()'s position-dependent inputs are already frozen for the pass run by the AliasAnalysis's own dominance/scope caches, making caching observationally equivalent to no caching across the hoists.

Assisted by Cursor

---
Full diff: https://github.com/llvm/llvm-project/pull/207030.diff


5 Files Affected:

- (modified) flang/include/flang/Optimizer/Analysis/AliasAnalysis.h (+38) 
- (modified) flang/lib/Optimizer/Analysis/AliasAnalysis.cpp (+29) 
- (modified) flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp (+10) 
- (modified) mlir/include/mlir/Analysis/AliasAnalysis.h (+60) 
- (modified) mlir/lib/Analysis/AliasAnalysis.cpp (+10) 


``````````diff
diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index 832634a708dba..a40632c23efce 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -20,6 +20,7 @@
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
+#include <utility>
 
 namespace fir {
 
@@ -351,10 +352,25 @@ struct AliasAnalysis {
   /// snapshots are not collected, and getSource performs only the
   /// SourceKind/origin classification without that bookkeeping side
   /// effect.
+  ///
+  /// When source caching is enabled (see enableSourceCache()), the result is
+  /// memoized keyed on (value, flags), and recursive sub-queries share the
+  /// same cache. The cache is a frozen snapshot: it is only valid while the IR
+  /// reachable from the queried values is not mutated in a way that would
+  /// change the source. Callers must scope caching no wider than such a region
+  /// (see mlir::AliasAnalysis::QueryCacheScope).
   fir::AliasAnalysis::Source getSource(mlir::Value,
                                        bool getLastInstantiationPoint = false,
                                        bool collectScopedOrigins = true);
 
+  /// Enable/disable memoization of getSource() results. These are invoked by
+  /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
+  /// QueryCacheScope) and are not meant to be called directly by passes.
+  /// enableSourceCache() turns on the frozen snapshot cache; disableSourceCache
+  /// clears it and turns it off.
+  void enableSourceCache();
+  void disableSourceCache();
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -370,6 +386,16 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  /// Compute the memory source of a value. This is the uncached
+  /// implementation of getSource(); getSource() is a thin wrapper that
+  /// memoizes the result when source caching is enabled.
+  fir::AliasAnalysis::Source getSourceImpl(mlir::Value v,
+                                           bool getLastInstantiationPoint,
+                                           bool collectScopedOrigins);
+
+  /// Clear the getSource() memoization cache.
+  void clearSourceCache() { getSourceCache.clear(); }
+
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
   /// classification (with collectScopedOrigins=false), then overrides
@@ -446,6 +472,18 @@ struct AliasAnalysis {
   /// functionHasMultipleScopes(); both true and false are cached so that
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
+
+  /// Opt-in memoization of getSource() results, keyed on the queried value
+  /// and the two boolean flags (getLastInstantiationPoint,
+  /// collectScopedOrigins) packed into the unsigned. Only consulted/populated
+  /// while \c sourceCacheEnabled is set. This is a frozen snapshot cache: it
+  /// has no automatic invalidation, so it is only enabled (via
+  /// enableSourceCache(), driven by mlir::AliasAnalysis::QueryCacheScope) for
+  /// the duration of a region in which the relevant IR is not mutated.
+  llvm::DenseMap<std::pair<mlir::Value, unsigned>, Source> getSourceCache;
+
+  /// Whether getSource() should consult/populate getSourceCache.
+  bool sourceCacheEnabled = false;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index 02a628968ddc4..f3e77a4fc0826 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
+#include <utility>
 
 using namespace mlir;
 
@@ -1108,9 +1109,37 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
+void AliasAnalysis::enableSourceCache() { sourceCacheEnabled = true; }
+
+void AliasAnalysis::disableSourceCache() {
+  sourceCacheEnabled = false;
+  clearSourceCache();
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
+  if (!sourceCacheEnabled)
+    return getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+
+  // Key on the queried value and the two boolean flags. Recursive sub-queries
+  // go through this same wrapper, so the whole walk is memoized.
+  std::pair<mlir::Value, unsigned> key{v,
+                                       (getLastInstantiationPoint ? 1u : 0u) |
+                                           (collectScopedOrigins ? 2u : 0u)};
+  auto it = getSourceCache.find(key);
+  if (it != getSourceCache.end())
+    return it->second;
+
+  Source source =
+      getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+  getSourceCache.try_emplace(key, source);
+  return source;
+}
+
+AliasAnalysis::Source
+AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
+                             bool collectScopedOrigins) {
   // If v is a pass-through block argument (see walkBlockArgPassThroughs),
   // continue from the underlying operand so the tracking loop below has a
   // defining op to chew on. Without this, a recursive query like the one in
diff --git a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
index 14fd5c6dd7fbc..2f640199252e4 100644
--- a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
+++ b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
@@ -268,6 +268,16 @@ void LoopInvariantCodeMotion::runOnOperation() {
   auto &aliasAnalysis = getAnalysis<AliasAnalysis>();
   aliasAnalysis.addAnalysisImplementation(fir::AliasAnalysis{});
 
+  // Enable alias-query caching for the duration of this pass. The scope
+  // enables caching on every registered implementation that supports it
+  // (the FIR AliasAnalysis memoizes getSource()) and no-ops on the rest, and
+  // reliably disables it on every exit. This is a frozen-snapshot cache with
+  // no listener: it is sound here because LICM only moves operations, and
+  // getSource()'s position-dependent inputs are already frozen for the pass
+  // run by the AliasAnalysis's own dominance/scope caches, so caching is
+  // observationally equivalent to no caching across the hoists.
+  mlir::AliasAnalysis::QueryCacheScope cacheScope(aliasAnalysis);
+
   std::function<bool(Operation *, LoopLikeOpInterface, bool)>
       shouldMoveOutOfLoop = [&](Operation *op, LoopLikeOpInterface loopLike,
                                 bool maybeConditionallyExecuted) {
diff --git a/mlir/include/mlir/Analysis/AliasAnalysis.h b/mlir/include/mlir/Analysis/AliasAnalysis.h
index 0fdcf01d9b6a3..71a08780c48a6 100644
--- a/mlir/include/mlir/Analysis/AliasAnalysis.h
+++ b/mlir/include/mlir/Analysis/AliasAnalysis.h
@@ -15,6 +15,7 @@
 #define MLIR_ANALYSIS_ALIASANALYSIS_H_
 
 #include "mlir/IR/Operation.h"
+#include "llvm/ADT/STLForwardCompat.h"
 
 namespace mlir {
 
@@ -187,8 +188,22 @@ struct AliasAnalysisTraits {
 
     /// Return the modify-reference behavior of `op` on `location`.
     virtual ModRefResult getModRef(Operation *op, Value location) = 0;
+
+    /// Enable opt-in caching of alias query results on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void enableQueryCaching() = 0;
+
+    /// Disable and clear any query caching on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void disableQueryCaching() = 0;
   };
 
+  /// Detection trait: true if `ImplT` provides `enableSourceCache()` /
+  /// `disableSourceCache()`. Mirrors the `has_is_invalidated` idiom used by
+  /// the pass analysis manager.
+  template <typename T>
+  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache());
+
   /// This class represents the `Model` of an alias analysis implementation
   /// `ImplT`. A model is instantiated for each alias analysis implementation
   /// to implement the `Concept` without the need for the derived
@@ -209,6 +224,17 @@ struct AliasAnalysisTraits {
       return impl.getModRef(op, location);
     }
 
+    /// Forward query-caching control to the implementation when it provides
+    /// the cache hooks; for implementations without them these are no-ops.
+    void enableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.enableSourceCache();
+    }
+    void disableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.disableSourceCache();
+    }
+
   private:
     ImplT impl;
   };
@@ -279,7 +305,41 @@ class AliasAnalysis {
   /// Return the modify-reference behavior of `op` on `location`.
   ModRefResult getModRef(Operation *op, Value location);
 
+  //===--------------------------------------------------------------------===//
+  // Query Caching
+  //===--------------------------------------------------------------------===//
+
+  /// RAII scope that enables opt-in query caching on all registered
+  /// implementations that support it for the duration of the scope, and
+  /// disables (and clears) it on destruction.
+  ///
+  /// This is the only entry point to query caching: enabling and disabling
+  /// are private so that caching is always paired with a guaranteed teardown.
+  /// A frozen (snapshot) cache is only valid while the IR is not mutated in a
+  /// way that affects the cached queries, so callers must keep the scope no
+  /// wider than such a no-mutation (or move-only) region.
+  class QueryCacheScope {
+  public:
+    explicit QueryCacheScope(AliasAnalysis &aa) : aa(aa) {
+      aa.enableQueryCaching();
+    }
+    ~QueryCacheScope() { aa.disableQueryCaching(); }
+
+    QueryCacheScope(const QueryCacheScope &) = delete;
+    QueryCacheScope &operator=(const QueryCacheScope &) = delete;
+    QueryCacheScope(QueryCacheScope &&) = delete;
+    QueryCacheScope &operator=(QueryCacheScope &&) = delete;
+
+  private:
+    AliasAnalysis &aa;
+  };
+
 private:
+  /// Enable/disable query caching on every registered implementation that
+  /// supports it. Private: use QueryCacheScope to guarantee teardown.
+  void enableQueryCaching();
+  void disableQueryCaching();
+
   /// A set of internal alias analysis implementations.
   SmallVector<std::unique_ptr<Concept>, 4> aliasImpls;
 };
diff --git a/mlir/lib/Analysis/AliasAnalysis.cpp b/mlir/lib/Analysis/AliasAnalysis.cpp
index 1382060f4dd90..9db48e18042cc 100644
--- a/mlir/lib/Analysis/AliasAnalysis.cpp
+++ b/mlir/lib/Analysis/AliasAnalysis.cpp
@@ -99,3 +99,13 @@ ModRefResult AliasAnalysis::getModRef(Operation *op, Value location) {
   }
   return result;
 }
+
+void AliasAnalysis::enableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->enableQueryCaching();
+}
+
+void AliasAnalysis::disableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->disableQueryCaching();
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207030

From llvmlistbot at llvm.org  Wed Jul  1 09:47:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 09:47:05 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add opt-in
 query caching; use it in flang LICM (PR #207030)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207030@github.com>
Message-ID: <6a454489.4a342cbc.22496e.9c2d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Slava Zakharin (vzakhari)

<details>
<summary>Changes</summary>

Add a generic, opt-in query-caching capability to the mlir::AliasAnalysis aggregator. The aggregator gains a public RAII QueryCacheScope and private enable/disableQueryCaching() fan-out; the Concept gains enableQueryCaching()/ disableQueryCaching() and the Model forwards them to implementations that provide enableSourceCache()/disableSourceCache() (detected via llvm::is_detected) and no-ops otherwise, so existing implementations need no changes and transformation passes never name a concrete implementation.

The FIR AliasAnalysis implements the hook by memoizing getSource(): getSource() becomes a thin wrapper over getSourceImpl() that consults a frozen-snapshot cache keyed on (value, flags) when caching is enabled.

Flang's LICM enables caching for the duration of the pass via QueryCacheScope. This is sound because LICM only moves operations and getSource()'s position-dependent inputs are already frozen for the pass run by the AliasAnalysis's own dominance/scope caches, making caching observationally equivalent to no caching across the hoists.

Assisted by Cursor

---
Full diff: https://github.com/llvm/llvm-project/pull/207030.diff


5 Files Affected:

- (modified) flang/include/flang/Optimizer/Analysis/AliasAnalysis.h (+38) 
- (modified) flang/lib/Optimizer/Analysis/AliasAnalysis.cpp (+29) 
- (modified) flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp (+10) 
- (modified) mlir/include/mlir/Analysis/AliasAnalysis.h (+60) 
- (modified) mlir/lib/Analysis/AliasAnalysis.cpp (+10) 


``````````diff
diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index 832634a708dba..a40632c23efce 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -20,6 +20,7 @@
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
+#include <utility>
 
 namespace fir {
 
@@ -351,10 +352,25 @@ struct AliasAnalysis {
   /// snapshots are not collected, and getSource performs only the
   /// SourceKind/origin classification without that bookkeeping side
   /// effect.
+  ///
+  /// When source caching is enabled (see enableSourceCache()), the result is
+  /// memoized keyed on (value, flags), and recursive sub-queries share the
+  /// same cache. The cache is a frozen snapshot: it is only valid while the IR
+  /// reachable from the queried values is not mutated in a way that would
+  /// change the source. Callers must scope caching no wider than such a region
+  /// (see mlir::AliasAnalysis::QueryCacheScope).
   fir::AliasAnalysis::Source getSource(mlir::Value,
                                        bool getLastInstantiationPoint = false,
                                        bool collectScopedOrigins = true);
 
+  /// Enable/disable memoization of getSource() results. These are invoked by
+  /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
+  /// QueryCacheScope) and are not meant to be called directly by passes.
+  /// enableSourceCache() turns on the frozen snapshot cache; disableSourceCache
+  /// clears it and turns it off.
+  void enableSourceCache();
+  void disableSourceCache();
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -370,6 +386,16 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  /// Compute the memory source of a value. This is the uncached
+  /// implementation of getSource(); getSource() is a thin wrapper that
+  /// memoizes the result when source caching is enabled.
+  fir::AliasAnalysis::Source getSourceImpl(mlir::Value v,
+                                           bool getLastInstantiationPoint,
+                                           bool collectScopedOrigins);
+
+  /// Clear the getSource() memoization cache.
+  void clearSourceCache() { getSourceCache.clear(); }
+
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
   /// classification (with collectScopedOrigins=false), then overrides
@@ -446,6 +472,18 @@ struct AliasAnalysis {
   /// functionHasMultipleScopes(); both true and false are cached so that
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
+
+  /// Opt-in memoization of getSource() results, keyed on the queried value
+  /// and the two boolean flags (getLastInstantiationPoint,
+  /// collectScopedOrigins) packed into the unsigned. Only consulted/populated
+  /// while \c sourceCacheEnabled is set. This is a frozen snapshot cache: it
+  /// has no automatic invalidation, so it is only enabled (via
+  /// enableSourceCache(), driven by mlir::AliasAnalysis::QueryCacheScope) for
+  /// the duration of a region in which the relevant IR is not mutated.
+  llvm::DenseMap<std::pair<mlir::Value, unsigned>, Source> getSourceCache;
+
+  /// Whether getSource() should consult/populate getSourceCache.
+  bool sourceCacheEnabled = false;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index 02a628968ddc4..f3e77a4fc0826 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
+#include <utility>
 
 using namespace mlir;
 
@@ -1108,9 +1109,37 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
+void AliasAnalysis::enableSourceCache() { sourceCacheEnabled = true; }
+
+void AliasAnalysis::disableSourceCache() {
+  sourceCacheEnabled = false;
+  clearSourceCache();
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
+  if (!sourceCacheEnabled)
+    return getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+
+  // Key on the queried value and the two boolean flags. Recursive sub-queries
+  // go through this same wrapper, so the whole walk is memoized.
+  std::pair<mlir::Value, unsigned> key{v,
+                                       (getLastInstantiationPoint ? 1u : 0u) |
+                                           (collectScopedOrigins ? 2u : 0u)};
+  auto it = getSourceCache.find(key);
+  if (it != getSourceCache.end())
+    return it->second;
+
+  Source source =
+      getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+  getSourceCache.try_emplace(key, source);
+  return source;
+}
+
+AliasAnalysis::Source
+AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
+                             bool collectScopedOrigins) {
   // If v is a pass-through block argument (see walkBlockArgPassThroughs),
   // continue from the underlying operand so the tracking loop below has a
   // defining op to chew on. Without this, a recursive query like the one in
diff --git a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
index 14fd5c6dd7fbc..2f640199252e4 100644
--- a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
+++ b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
@@ -268,6 +268,16 @@ void LoopInvariantCodeMotion::runOnOperation() {
   auto &aliasAnalysis = getAnalysis<AliasAnalysis>();
   aliasAnalysis.addAnalysisImplementation(fir::AliasAnalysis{});
 
+  // Enable alias-query caching for the duration of this pass. The scope
+  // enables caching on every registered implementation that supports it
+  // (the FIR AliasAnalysis memoizes getSource()) and no-ops on the rest, and
+  // reliably disables it on every exit. This is a frozen-snapshot cache with
+  // no listener: it is sound here because LICM only moves operations, and
+  // getSource()'s position-dependent inputs are already frozen for the pass
+  // run by the AliasAnalysis's own dominance/scope caches, so caching is
+  // observationally equivalent to no caching across the hoists.
+  mlir::AliasAnalysis::QueryCacheScope cacheScope(aliasAnalysis);
+
   std::function<bool(Operation *, LoopLikeOpInterface, bool)>
       shouldMoveOutOfLoop = [&](Operation *op, LoopLikeOpInterface loopLike,
                                 bool maybeConditionallyExecuted) {
diff --git a/mlir/include/mlir/Analysis/AliasAnalysis.h b/mlir/include/mlir/Analysis/AliasAnalysis.h
index 0fdcf01d9b6a3..71a08780c48a6 100644
--- a/mlir/include/mlir/Analysis/AliasAnalysis.h
+++ b/mlir/include/mlir/Analysis/AliasAnalysis.h
@@ -15,6 +15,7 @@
 #define MLIR_ANALYSIS_ALIASANALYSIS_H_
 
 #include "mlir/IR/Operation.h"
+#include "llvm/ADT/STLForwardCompat.h"
 
 namespace mlir {
 
@@ -187,8 +188,22 @@ struct AliasAnalysisTraits {
 
     /// Return the modify-reference behavior of `op` on `location`.
     virtual ModRefResult getModRef(Operation *op, Value location) = 0;
+
+    /// Enable opt-in caching of alias query results on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void enableQueryCaching() = 0;
+
+    /// Disable and clear any query caching on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void disableQueryCaching() = 0;
   };
 
+  /// Detection trait: true if `ImplT` provides `enableSourceCache()` /
+  /// `disableSourceCache()`. Mirrors the `has_is_invalidated` idiom used by
+  /// the pass analysis manager.
+  template <typename T>
+  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache());
+
   /// This class represents the `Model` of an alias analysis implementation
   /// `ImplT`. A model is instantiated for each alias analysis implementation
   /// to implement the `Concept` without the need for the derived
@@ -209,6 +224,17 @@ struct AliasAnalysisTraits {
       return impl.getModRef(op, location);
     }
 
+    /// Forward query-caching control to the implementation when it provides
+    /// the cache hooks; for implementations without them these are no-ops.
+    void enableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.enableSourceCache();
+    }
+    void disableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.disableSourceCache();
+    }
+
   private:
     ImplT impl;
   };
@@ -279,7 +305,41 @@ class AliasAnalysis {
   /// Return the modify-reference behavior of `op` on `location`.
   ModRefResult getModRef(Operation *op, Value location);
 
+  //===--------------------------------------------------------------------===//
+  // Query Caching
+  //===--------------------------------------------------------------------===//
+
+  /// RAII scope that enables opt-in query caching on all registered
+  /// implementations that support it for the duration of the scope, and
+  /// disables (and clears) it on destruction.
+  ///
+  /// This is the only entry point to query caching: enabling and disabling
+  /// are private so that caching is always paired with a guaranteed teardown.
+  /// A frozen (snapshot) cache is only valid while the IR is not mutated in a
+  /// way that affects the cached queries, so callers must keep the scope no
+  /// wider than such a no-mutation (or move-only) region.
+  class QueryCacheScope {
+  public:
+    explicit QueryCacheScope(AliasAnalysis &aa) : aa(aa) {
+      aa.enableQueryCaching();
+    }
+    ~QueryCacheScope() { aa.disableQueryCaching(); }
+
+    QueryCacheScope(const QueryCacheScope &) = delete;
+    QueryCacheScope &operator=(const QueryCacheScope &) = delete;
+    QueryCacheScope(QueryCacheScope &&) = delete;
+    QueryCacheScope &operator=(QueryCacheScope &&) = delete;
+
+  private:
+    AliasAnalysis &aa;
+  };
+
 private:
+  /// Enable/disable query caching on every registered implementation that
+  /// supports it. Private: use QueryCacheScope to guarantee teardown.
+  void enableQueryCaching();
+  void disableQueryCaching();
+
   /// A set of internal alias analysis implementations.
   SmallVector<std::unique_ptr<Concept>, 4> aliasImpls;
 };
diff --git a/mlir/lib/Analysis/AliasAnalysis.cpp b/mlir/lib/Analysis/AliasAnalysis.cpp
index 1382060f4dd90..9db48e18042cc 100644
--- a/mlir/lib/Analysis/AliasAnalysis.cpp
+++ b/mlir/lib/Analysis/AliasAnalysis.cpp
@@ -99,3 +99,13 @@ ModRefResult AliasAnalysis::getModRef(Operation *op, Value location) {
   }
   return result;
 }
+
+void AliasAnalysis::enableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->enableQueryCaching();
+}
+
+void AliasAnalysis::disableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->disableQueryCaching();
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207030

From llvmlistbot at llvm.org  Wed Jul  1 09:47:25 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Wed, 01 Jul 2026 09:47:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add precise
 listener-driven cache invalidation (PR #207032)
Message-ID: <llvm.org/llvm/llvm-project/pull/207032@github.com>


https://github.com/vzakhari created https://github.com/llvm/llvm-project/pull/207032

Extend the opt-in query cache so it can be used safely by passes that mutate
the IR through a rewriter. The query-caching hooks gain a RewriterBase*
parameter (defaulted, so QueryCacheScope callers are unaffected): in frozen
mode (null rewriter) the cache is a snapshot as before; with a rewriter the
FIR AliasAnalysis installs a ForwardingListener on it and evicts cached
getSource() results precisely as operations are erased, modified, or replaced.

Each cached entry now records the operations it depends on (the def-use chain
and the dominating dummy_scope op visited while computing it), maintained via a
reverse-dependency index. Nested queries thread a dependency sink and union a
cache hit's dependencies into the enclosing entry, so transitive dependencies
are tracked. Erasure notifications also walk nested operations and fire before
the storage is freed, closing the window where a freed-and-reused pointer could
yield a stale hit.

Adds AliasAnalysisCacheTest covering precise eviction on modify, eviction on
erase, and frozen-mode memoization.

Assisted by Cursor


>From 585f5854516df94c0ee343486b184e7868df4d8f Mon Sep 17 00:00:00 2001
From: Slava Zakharin <szakharin at nvidia.com>
Date: Tue, 30 Jun 2026 12:04:20 -0700
Subject: [PATCH 1/2] [mlir][AliasAnalysis] Add opt-in query caching; use it in
 flang LICM

Add a generic, opt-in query-caching capability to the mlir::AliasAnalysis
aggregator. The aggregator gains a public RAII QueryCacheScope and private
enable/disableQueryCaching() fan-out; the Concept gains enableQueryCaching()/
disableQueryCaching() and the Model forwards them to implementations that
provide enableSourceCache()/disableSourceCache() (detected via
llvm::is_detected) and no-ops otherwise, so existing implementations need no
changes and transformation passes never name a concrete implementation.

The FIR AliasAnalysis implements the hook by memoizing getSource(): getSource()
becomes a thin wrapper over getSourceImpl() that consults a frozen-snapshot
cache keyed on (value, flags) when caching is enabled.

Flang's LICM enables caching for the duration of the pass via QueryCacheScope.
This is sound because LICM only moves operations and getSource()'s
position-dependent inputs are already frozen for the pass run by the
AliasAnalysis's own dominance/scope caches, making caching observationally
equivalent to no caching across the hoists.

Assisted by Cursor
---
 .../flang/Optimizer/Analysis/AliasAnalysis.h  | 38 ++++++++++++
 .../lib/Optimizer/Analysis/AliasAnalysis.cpp  | 29 +++++++++
 .../Transforms/LoopInvariantCodeMotion.cpp    | 10 ++++
 mlir/include/mlir/Analysis/AliasAnalysis.h    | 60 +++++++++++++++++++
 mlir/lib/Analysis/AliasAnalysis.cpp           | 10 ++++
 5 files changed, 147 insertions(+)

diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index 832634a708dba..a40632c23efce 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -20,6 +20,7 @@
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
+#include <utility>
 
 namespace fir {
 
@@ -351,10 +352,25 @@ struct AliasAnalysis {
   /// snapshots are not collected, and getSource performs only the
   /// SourceKind/origin classification without that bookkeeping side
   /// effect.
+  ///
+  /// When source caching is enabled (see enableSourceCache()), the result is
+  /// memoized keyed on (value, flags), and recursive sub-queries share the
+  /// same cache. The cache is a frozen snapshot: it is only valid while the IR
+  /// reachable from the queried values is not mutated in a way that would
+  /// change the source. Callers must scope caching no wider than such a region
+  /// (see mlir::AliasAnalysis::QueryCacheScope).
   fir::AliasAnalysis::Source getSource(mlir::Value,
                                        bool getLastInstantiationPoint = false,
                                        bool collectScopedOrigins = true);
 
+  /// Enable/disable memoization of getSource() results. These are invoked by
+  /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
+  /// QueryCacheScope) and are not meant to be called directly by passes.
+  /// enableSourceCache() turns on the frozen snapshot cache; disableSourceCache
+  /// clears it and turns it off.
+  void enableSourceCache();
+  void disableSourceCache();
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -370,6 +386,16 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  /// Compute the memory source of a value. This is the uncached
+  /// implementation of getSource(); getSource() is a thin wrapper that
+  /// memoizes the result when source caching is enabled.
+  fir::AliasAnalysis::Source getSourceImpl(mlir::Value v,
+                                           bool getLastInstantiationPoint,
+                                           bool collectScopedOrigins);
+
+  /// Clear the getSource() memoization cache.
+  void clearSourceCache() { getSourceCache.clear(); }
+
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
   /// classification (with collectScopedOrigins=false), then overrides
@@ -446,6 +472,18 @@ struct AliasAnalysis {
   /// functionHasMultipleScopes(); both true and false are cached so that
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
+
+  /// Opt-in memoization of getSource() results, keyed on the queried value
+  /// and the two boolean flags (getLastInstantiationPoint,
+  /// collectScopedOrigins) packed into the unsigned. Only consulted/populated
+  /// while \c sourceCacheEnabled is set. This is a frozen snapshot cache: it
+  /// has no automatic invalidation, so it is only enabled (via
+  /// enableSourceCache(), driven by mlir::AliasAnalysis::QueryCacheScope) for
+  /// the duration of a region in which the relevant IR is not mutated.
+  llvm::DenseMap<std::pair<mlir::Value, unsigned>, Source> getSourceCache;
+
+  /// Whether getSource() should consult/populate getSourceCache.
+  bool sourceCacheEnabled = false;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index 02a628968ddc4..f3e77a4fc0826 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -30,6 +30,7 @@
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
+#include <utility>
 
 using namespace mlir;
 
@@ -1108,9 +1109,37 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
+void AliasAnalysis::enableSourceCache() { sourceCacheEnabled = true; }
+
+void AliasAnalysis::disableSourceCache() {
+  sourceCacheEnabled = false;
+  clearSourceCache();
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
+  if (!sourceCacheEnabled)
+    return getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+
+  // Key on the queried value and the two boolean flags. Recursive sub-queries
+  // go through this same wrapper, so the whole walk is memoized.
+  std::pair<mlir::Value, unsigned> key{v,
+                                       (getLastInstantiationPoint ? 1u : 0u) |
+                                           (collectScopedOrigins ? 2u : 0u)};
+  auto it = getSourceCache.find(key);
+  if (it != getSourceCache.end())
+    return it->second;
+
+  Source source =
+      getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+  getSourceCache.try_emplace(key, source);
+  return source;
+}
+
+AliasAnalysis::Source
+AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
+                             bool collectScopedOrigins) {
   // If v is a pass-through block argument (see walkBlockArgPassThroughs),
   // continue from the underlying operand so the tracking loop below has a
   // defining op to chew on. Without this, a recursive query like the one in
diff --git a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
index 14fd5c6dd7fbc..2f640199252e4 100644
--- a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
+++ b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
@@ -268,6 +268,16 @@ void LoopInvariantCodeMotion::runOnOperation() {
   auto &aliasAnalysis = getAnalysis<AliasAnalysis>();
   aliasAnalysis.addAnalysisImplementation(fir::AliasAnalysis{});
 
+  // Enable alias-query caching for the duration of this pass. The scope
+  // enables caching on every registered implementation that supports it
+  // (the FIR AliasAnalysis memoizes getSource()) and no-ops on the rest, and
+  // reliably disables it on every exit. This is a frozen-snapshot cache with
+  // no listener: it is sound here because LICM only moves operations, and
+  // getSource()'s position-dependent inputs are already frozen for the pass
+  // run by the AliasAnalysis's own dominance/scope caches, so caching is
+  // observationally equivalent to no caching across the hoists.
+  mlir::AliasAnalysis::QueryCacheScope cacheScope(aliasAnalysis);
+
   std::function<bool(Operation *, LoopLikeOpInterface, bool)>
       shouldMoveOutOfLoop = [&](Operation *op, LoopLikeOpInterface loopLike,
                                 bool maybeConditionallyExecuted) {
diff --git a/mlir/include/mlir/Analysis/AliasAnalysis.h b/mlir/include/mlir/Analysis/AliasAnalysis.h
index 0fdcf01d9b6a3..71a08780c48a6 100644
--- a/mlir/include/mlir/Analysis/AliasAnalysis.h
+++ b/mlir/include/mlir/Analysis/AliasAnalysis.h
@@ -15,6 +15,7 @@
 #define MLIR_ANALYSIS_ALIASANALYSIS_H_
 
 #include "mlir/IR/Operation.h"
+#include "llvm/ADT/STLForwardCompat.h"
 
 namespace mlir {
 
@@ -187,8 +188,22 @@ struct AliasAnalysisTraits {
 
     /// Return the modify-reference behavior of `op` on `location`.
     virtual ModRefResult getModRef(Operation *op, Value location) = 0;
+
+    /// Enable opt-in caching of alias query results on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void enableQueryCaching() = 0;
+
+    /// Disable and clear any query caching on this implementation.
+    /// Implementations that do not support caching ignore this request.
+    virtual void disableQueryCaching() = 0;
   };
 
+  /// Detection trait: true if `ImplT` provides `enableSourceCache()` /
+  /// `disableSourceCache()`. Mirrors the `has_is_invalidated` idiom used by
+  /// the pass analysis manager.
+  template <typename T>
+  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache());
+
   /// This class represents the `Model` of an alias analysis implementation
   /// `ImplT`. A model is instantiated for each alias analysis implementation
   /// to implement the `Concept` without the need for the derived
@@ -209,6 +224,17 @@ struct AliasAnalysisTraits {
       return impl.getModRef(op, location);
     }
 
+    /// Forward query-caching control to the implementation when it provides
+    /// the cache hooks; for implementations without them these are no-ops.
+    void enableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.enableSourceCache();
+    }
+    void disableQueryCaching() final {
+      if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
+        impl.disableSourceCache();
+    }
+
   private:
     ImplT impl;
   };
@@ -279,7 +305,41 @@ class AliasAnalysis {
   /// Return the modify-reference behavior of `op` on `location`.
   ModRefResult getModRef(Operation *op, Value location);
 
+  //===--------------------------------------------------------------------===//
+  // Query Caching
+  //===--------------------------------------------------------------------===//
+
+  /// RAII scope that enables opt-in query caching on all registered
+  /// implementations that support it for the duration of the scope, and
+  /// disables (and clears) it on destruction.
+  ///
+  /// This is the only entry point to query caching: enabling and disabling
+  /// are private so that caching is always paired with a guaranteed teardown.
+  /// A frozen (snapshot) cache is only valid while the IR is not mutated in a
+  /// way that affects the cached queries, so callers must keep the scope no
+  /// wider than such a no-mutation (or move-only) region.
+  class QueryCacheScope {
+  public:
+    explicit QueryCacheScope(AliasAnalysis &aa) : aa(aa) {
+      aa.enableQueryCaching();
+    }
+    ~QueryCacheScope() { aa.disableQueryCaching(); }
+
+    QueryCacheScope(const QueryCacheScope &) = delete;
+    QueryCacheScope &operator=(const QueryCacheScope &) = delete;
+    QueryCacheScope(QueryCacheScope &&) = delete;
+    QueryCacheScope &operator=(QueryCacheScope &&) = delete;
+
+  private:
+    AliasAnalysis &aa;
+  };
+
 private:
+  /// Enable/disable query caching on every registered implementation that
+  /// supports it. Private: use QueryCacheScope to guarantee teardown.
+  void enableQueryCaching();
+  void disableQueryCaching();
+
   /// A set of internal alias analysis implementations.
   SmallVector<std::unique_ptr<Concept>, 4> aliasImpls;
 };
diff --git a/mlir/lib/Analysis/AliasAnalysis.cpp b/mlir/lib/Analysis/AliasAnalysis.cpp
index 1382060f4dd90..9db48e18042cc 100644
--- a/mlir/lib/Analysis/AliasAnalysis.cpp
+++ b/mlir/lib/Analysis/AliasAnalysis.cpp
@@ -99,3 +99,13 @@ ModRefResult AliasAnalysis::getModRef(Operation *op, Value location) {
   }
   return result;
 }
+
+void AliasAnalysis::enableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->enableQueryCaching();
+}
+
+void AliasAnalysis::disableQueryCaching() {
+  for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
+    aliasImpl->disableQueryCaching();
+}

>From 13632fdf7ed39af3e0c6403d51d7c7ee61c0aba3 Mon Sep 17 00:00:00 2001
From: Slava Zakharin <szakharin at nvidia.com>
Date: Tue, 30 Jun 2026 12:23:03 -0700
Subject: [PATCH 2/2] [mlir][AliasAnalysis] Add precise listener-driven cache
 invalidation

Extend the opt-in query cache so it can be used safely by passes that mutate
the IR through a rewriter. The query-caching hooks gain a RewriterBase*
parameter (defaulted, so QueryCacheScope callers are unaffected): in frozen
mode (null rewriter) the cache is a snapshot as before; with a rewriter the
FIR AliasAnalysis installs a ForwardingListener on it and evicts cached
getSource() results precisely as operations are erased, modified, or replaced.

Each cached entry now records the operations it depends on (the def-use chain
and the dominating dummy_scope op visited while computing it), maintained via a
reverse-dependency index. Nested queries thread a dependency sink and union a
cache hit's dependencies into the enclosing entry, so transitive dependencies
are tracked. Erasure notifications also walk nested operations and fire before
the storage is freed, closing the window where a freed-and-reused pointer could
yield a stale hit.

Adds AliasAnalysisCacheTest covering precise eviction on modify, eviction on
erase, and frozen-mode memoization.

Assisted by Cursor
---
 .../flang/Optimizer/Analysis/AliasAnalysis.h  | 107 +++++++++++--
 .../lib/Optimizer/Analysis/AliasAnalysis.cpp  | 150 ++++++++++++++++-
 .../Optimizer/AliasAnalysisCacheTest.cpp      | 151 ++++++++++++++++++
 flang/unittests/Optimizer/CMakeLists.txt      |   1 +
 mlir/include/mlir/Analysis/AliasAnalysis.h    |  35 ++--
 mlir/lib/Analysis/AliasAnalysis.cpp           |   4 +-
 6 files changed, 416 insertions(+), 32 deletions(-)
 create mode 100644 flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp

diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index a40632c23efce..913270d4586ac 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -24,10 +24,25 @@
 
 namespace fir {
 
+/// Listener used to evict getSource() cache entries as the IR is mutated
+/// through a rewriter. Defined in AliasAnalysis.cpp.
+class AliasAnalysisCacheListener;
+
 //===----------------------------------------------------------------------===//
 // AliasAnalysis
 //===----------------------------------------------------------------------===//
 struct AliasAnalysis {
+  // Special members are user-declared (and mostly defined out of line) because
+  // the source cache owns a std::unique_ptr to the incomplete listener type
+  // AliasAnalysisCacheListener. A move constructor is required so instances can
+  // be registered via mlir::AliasAnalysis::addAnalysisImplementation().
+  AliasAnalysis();
+  ~AliasAnalysis();
+  AliasAnalysis(AliasAnalysis &&);
+  AliasAnalysis &operator=(AliasAnalysis &&);
+  AliasAnalysis(const AliasAnalysis &) = delete;
+  AliasAnalysis &operator=(const AliasAnalysis &) = delete;
+
   // Structures to describe the memory source of a value.
 
   /// Kind of the memory source referenced by a value.
@@ -366,11 +381,25 @@ struct AliasAnalysis {
   /// Enable/disable memoization of getSource() results. These are invoked by
   /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
   /// QueryCacheScope) and are not meant to be called directly by passes.
-  /// enableSourceCache() turns on the frozen snapshot cache; disableSourceCache
-  /// clears it and turns it off.
-  void enableSourceCache();
+  ///
+  /// If \p rewriter is null, the cache is a frozen snapshot with no automatic
+  /// invalidation: it is only valid while the IR reachable from the queried
+  /// values is not mutated. If \p rewriter is non-null, a listener is installed
+  /// on it (chained ahead of any existing listener) so that cached entries are
+  /// evicted precisely as operations are erased / modified / replaced through
+  /// that rewriter; the cache then stays valid as long as all such mutations
+  /// flow through the rewriter.
+  ///
+  /// disableSourceCache() clears the cache (and reverse index), uninstalls the
+  /// listener if any, and turns caching off.
+  void enableSourceCache(mlir::RewriterBase *rewriter = nullptr);
   void disableSourceCache();
 
+  /// Testing only: number of entries currently held in the getSource() cache.
+  std::size_t getSourceCacheSizeForTesting() const {
+    return getSourceCache.size();
+  }
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -386,6 +415,22 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  friend class fir::AliasAnalysisCacheListener;
+
+  /// Cache key for getSource(): the queried value together with the two
+  /// boolean flags (getLastInstantiationPoint, collectScopedOrigins) packed
+  /// into the unsigned.
+  using SourceCacheKey = std::pair<mlir::Value, unsigned>;
+
+  /// A memoized getSource() result together with the operations it depends on
+  /// (the def-use chain and scope ops visited while computing it). The deps
+  /// drive precise eviction: if any of them is erased / modified / replaced,
+  /// this entry must be dropped.
+  struct CachedSource {
+    Source result;
+    llvm::SmallVector<mlir::Operation *, 4> deps;
+  };
+
   /// Compute the memory source of a value. This is the uncached
   /// implementation of getSource(); getSource() is a thin wrapper that
   /// memoizes the result when source caching is enabled.
@@ -393,8 +438,24 @@ struct AliasAnalysis {
                                            bool getLastInstantiationPoint,
                                            bool collectScopedOrigins);
 
-  /// Clear the getSource() memoization cache.
-  void clearSourceCache() { getSourceCache.clear(); }
+  /// Record \p op as a dependency of the getSource() computation currently in
+  /// progress (no-op when not collecting dependencies). Used to build the
+  /// reverse index that drives precise cache eviction.
+  void recordSourceDep(mlir::Operation *op) {
+    if (currentDepSink && op)
+      currentDepSink->push_back(op);
+  }
+
+  /// Evict every cached getSource() entry that depends on \p op (and, for
+  /// erasure, on any op nested within its regions), dropping the corresponding
+  /// reverse-index entries.
+  void evictSourceDependents(mlir::Operation *op);
+
+  /// Clear the getSource() memoization cache and its reverse index.
+  void clearSourceCache() {
+    getSourceCache.clear();
+    opDependents.clear();
+  }
 
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
@@ -473,17 +534,37 @@ struct AliasAnalysis {
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
 
-  /// Opt-in memoization of getSource() results, keyed on the queried value
-  /// and the two boolean flags (getLastInstantiationPoint,
-  /// collectScopedOrigins) packed into the unsigned. Only consulted/populated
-  /// while \c sourceCacheEnabled is set. This is a frozen snapshot cache: it
-  /// has no automatic invalidation, so it is only enabled (via
-  /// enableSourceCache(), driven by mlir::AliasAnalysis::QueryCacheScope) for
-  /// the duration of a region in which the relevant IR is not mutated.
-  llvm::DenseMap<std::pair<mlir::Value, unsigned>, Source> getSourceCache;
+  /// Opt-in memoization of getSource() results, keyed on the queried value and
+  /// the two boolean flags (see SourceCacheKey). Only consulted/populated while
+  /// \c sourceCacheEnabled is set. Each entry also records the operations it
+  /// depends on so it can be evicted precisely. Without a listener (frozen
+  /// mode) this is a snapshot with no automatic invalidation, valid only while
+  /// the relevant IR is not mutated; with a listener installed (see
+  /// enableSourceCache(rewriter)) entries are evicted as the IR is mutated
+  /// through that rewriter. Driven by mlir::AliasAnalysis::QueryCacheScope.
+  llvm::DenseMap<SourceCacheKey, CachedSource> getSourceCache;
+
+  /// Reverse dependency index: for each operation, the set of getSourceCache
+  /// keys whose result depends on it. Maintained alongside getSourceCache and
+  /// used to evict only the affected entries when an op is mutated/erased.
+  llvm::DenseMap<mlir::Operation *, llvm::SmallVector<SourceCacheKey, 2>>
+      opDependents;
+
+  /// When non-null, getSourceImpl() and nested getSource() hits append the
+  /// operations they depend on here, so the in-progress getSource() result can
+  /// be stored with its dependencies. Saved/restored around nested queries.
+  llvm::SmallVectorImpl<mlir::Operation *> *currentDepSink = nullptr;
 
   /// Whether getSource() should consult/populate getSourceCache.
   bool sourceCacheEnabled = false;
+
+  /// The rewriter on which \c cacheListener is installed (listener mode), or
+  /// null in frozen mode. Used to uninstall the listener on disable.
+  mlir::RewriterBase *cacheRewriter = nullptr;
+
+  /// Listener installed on \c cacheRewriter that evicts cache entries as the
+  /// IR is mutated. Null in frozen mode. Owns a back-pointer to this analysis.
+  std::unique_ptr<AliasAnalysisCacheListener> cacheListener;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index f3e77a4fc0826..3fb73fcc78280 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -22,9 +22,11 @@
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/Value.h"
 #include "mlir/Interfaces/ControlFlowInterfaces.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
@@ -1109,13 +1111,116 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
-void AliasAnalysis::enableSourceCache() { sourceCacheEnabled = true; }
+//===----------------------------------------------------------------------===//
+// Source cache invalidation listener
+//===----------------------------------------------------------------------===//
+
+/// Listener installed on a rewriter while source caching is enabled in listener
+/// mode. It evicts the affected getSource() cache entries on each mutation and
+/// then forwards the notification to any previously installed listener so the
+/// cache composes with greedy/conversion driver listeners.
+class AliasAnalysisCacheListener
+    : public mlir::RewriterBase::ForwardingListener {
+public:
+  AliasAnalysisCacheListener(AliasAnalysis &aa,
+                             mlir::OpBuilder::Listener *previous)
+      : mlir::RewriterBase::ForwardingListener(previous), aa(aa),
+        previous(previous) {}
+
+  /// The listener that was installed on the rewriter before this one, so it
+  /// can be restored when caching is disabled.
+  mlir::OpBuilder::Listener *getPreviousListener() const { return previous; }
+
+  void notifyOperationErased(mlir::Operation *op) override {
+    // Runs before the operation's storage is freed, so evicting here closes
+    // the window where a freed-and-reused Operation* could produce a stale hit.
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationErased(op);
+  }
+
+  void notifyOperationModified(mlir::Operation *op) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationModified(op);
+  }
+
+  void notifyOperationReplaced(mlir::Operation *op,
+                               mlir::Operation *newOp) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationReplaced(op, newOp);
+  }
+
+  void notifyOperationReplaced(mlir::Operation *op,
+                               mlir::ValueRange replacement) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationReplaced(op, replacement);
+  }
+
+private:
+  AliasAnalysis &aa;
+  mlir::OpBuilder::Listener *previous;
+};
+
+//===----------------------------------------------------------------------===//
+// AliasAnalysis special members
+//===----------------------------------------------------------------------===//
+
+// Defined out of line because cacheListener is a std::unique_ptr to the
+// (here-complete) AliasAnalysisCacheListener.
+AliasAnalysis::AliasAnalysis() = default;
+AliasAnalysis::~AliasAnalysis() = default;
+AliasAnalysis::AliasAnalysis(AliasAnalysis &&) = default;
+AliasAnalysis &AliasAnalysis::operator=(AliasAnalysis &&) = default;
+
+//===----------------------------------------------------------------------===//
+// Source cache
+//===----------------------------------------------------------------------===//
+
+void AliasAnalysis::enableSourceCache(mlir::RewriterBase *rewriter) {
+  sourceCacheEnabled = true;
+  if (rewriter && !cacheListener) {
+    cacheRewriter = rewriter;
+    cacheListener = std::make_unique<AliasAnalysisCacheListener>(
+        *this, rewriter->getListener());
+    rewriter->setListener(cacheListener.get());
+  }
+}
 
 void AliasAnalysis::disableSourceCache() {
   sourceCacheEnabled = false;
+  if (cacheListener) {
+    // Restore the listener that was installed before ours.
+    cacheRewriter->setListener(cacheListener->getPreviousListener());
+    cacheListener.reset();
+    cacheRewriter = nullptr;
+  }
   clearSourceCache();
 }
 
+void AliasAnalysis::evictSourceDependents(mlir::Operation *op) {
+  if (getSourceCache.empty())
+    return;
+
+  // Collect the ops whose dependents must be evicted. For erased ops, this
+  // includes ops nested in their regions, since cached entries may depend on
+  // those nested ops too and their storage is freed along with the parent.
+  llvm::SmallVector<mlir::Operation *, 8> worklist{op};
+  while (!worklist.empty()) {
+    mlir::Operation *cur = worklist.pop_back_val();
+    auto it = opDependents.find(cur);
+    if (it != opDependents.end()) {
+      // Copy the keys out before erasing, then drop the cache entries.
+      llvm::SmallVector<SourceCacheKey, 2> keys = std::move(it->second);
+      opDependents.erase(it);
+      for (const SourceCacheKey &key : keys)
+        getSourceCache.erase(key);
+    }
+    for (mlir::Region &region : cur->getRegions())
+      for (mlir::Block &block : region)
+        for (mlir::Operation &nested : block)
+          worklist.push_back(&nested);
+  }
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
@@ -1124,16 +1229,40 @@ AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
 
   // Key on the queried value and the two boolean flags. Recursive sub-queries
   // go through this same wrapper, so the whole walk is memoized.
-  std::pair<mlir::Value, unsigned> key{v,
-                                       (getLastInstantiationPoint ? 1u : 0u) |
-                                           (collectScopedOrigins ? 2u : 0u)};
+  SourceCacheKey key{v, (getLastInstantiationPoint ? 1u : 0u) |
+                            (collectScopedOrigins ? 2u : 0u)};
   auto it = getSourceCache.find(key);
-  if (it != getSourceCache.end())
-    return it->second;
+  if (it != getSourceCache.end()) {
+    // Propagate this entry's dependencies into the enclosing query (if any) so
+    // transitive dependencies are tracked: if a dep of this entry is later
+    // erased, the enclosing entry that incorporated it is evicted too.
+    if (currentDepSink)
+      currentDepSink->append(it->second.deps.begin(), it->second.deps.end());
+    return it->second.result;
+  }
 
+  // Miss: compute with a fresh dependency sink, then propagate to the parent.
+  llvm::SmallVector<mlir::Operation *, 4> deps;
+  llvm::SmallVectorImpl<mlir::Operation *> *parentSink = currentDepSink;
+  currentDepSink = &deps;
   Source source =
       getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
-  getSourceCache.try_emplace(key, source);
+  currentDepSink = parentSink;
+
+  // Deduplicate before storing / indexing (the walk may visit an op multiple
+  // times). Sorting by pointer only affects internal bookkeeping, not emitted
+  // IR, so dump determinism is preserved.
+  llvm::sort(deps);
+  deps.erase(llvm::unique(deps), deps.end());
+
+  if (parentSink)
+    parentSink->append(deps.begin(), deps.end());
+
+  bool inserted =
+      getSourceCache.try_emplace(key, CachedSource{source, deps}).second;
+  if (inserted)
+    for (mlir::Operation *depOp : deps)
+      opDependents[depOp].push_back(key);
   return source;
 }
 
@@ -1176,6 +1305,9 @@ AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
   // buildSourceAtDeclare reuses getSource purely for declare classification).
   llvm::SmallVector<Source::ScopedOrigin, 4> scopedOrigins;
   while (defOp && !breakFromLoop) {
+    // Record each operation visited along the def-use chain as a dependency
+    // of this query, so the cached result is evicted if any of them changes.
+    recordSourceDep(defOp);
     // Operations may have multiple results, so we need to analyze
     // the result for which the source is queried.
     auto opResult = mlir::cast<OpResult>(v);
@@ -1443,6 +1575,10 @@ AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
           if (collectScopedOrigins) {
             Source::ScopedOrigin scopedOrigin;
             scopedOrigin.scope = getDeclarationScope(op);
+            // The chosen scope depends on the dominating fir.dummy_scope op;
+            // record it so the entry is evicted if that scope op changes.
+            if (scopedOrigin.scope)
+              recordSourceDep(scopedOrigin.scope.getDefiningOp());
             scopedOrigin.declValue = opResult;
             scopedOrigin.accessPath.steps.assign(pathSteps.rbegin(),
                                                  pathSteps.rend());
diff --git a/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp b/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp
new file mode 100644
index 0000000000000..b199adc4191b6
--- /dev/null
+++ b/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp
@@ -0,0 +1,151 @@
+//===- AliasAnalysisCacheTest.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Tests for the listener-driven invalidation of fir::AliasAnalysis's getSource
+// cache: mutating the IR through a rewriter must evict precisely the cached
+// entries that depend on the mutated operation, and an erase must drop the
+// affected entries before the operation's storage is freed (so a freed,
+// possibly reused, pointer can never produce a stale cache hit).
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "flang/Optimizer/Analysis/AliasAnalysis.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Support/InitFIR.h"
+
+struct AliasAnalysisCacheTest : public testing::Test {
+public:
+  void SetUp() override {
+    fir::support::loadDialects(context);
+    builder = std::make_unique<mlir::OpBuilder>(&context);
+    mlir::Location loc = builder->getUnknownLoc();
+
+    moduleOp = mlir::ModuleOp::create(*builder, loc);
+    builder->setInsertionPointToStart(moduleOp->getBody());
+    mlir::func::FuncOp func = mlir::func::FuncOp::create(
+        *builder, loc, "test", builder->getFunctionType({}, {}));
+    builder->setInsertionPointToStart(func.addEntryBlock());
+  }
+
+  mlir::Location getLoc() { return builder->getUnknownLoc(); }
+
+  // Build an `alloca -> declare` chain for a scalar i32 variable and return the
+  // declared variable address.
+  mlir::Value createScalarVariable(llvm::StringRef name) {
+    mlir::Location loc = getLoc();
+    mlir::Type eleType = mlir::IntegerType::get(&context, 32);
+    mlir::Value addr = fir::AllocaOp::create(*builder, loc, eleType);
+    auto declare = fir::DeclareOp::create(*builder, loc, addr.getType(), addr,
+        /*shape=*/mlir::Value{}, /*typeParams=*/mlir::ValueRange{},
+        /*dummy_scope=*/nullptr, /*storage=*/nullptr, /*storage_offset=*/0,
+        mlir::StringAttr::get(&context, name),
+        /*fortran_attrs=*/fir::FortranVariableFlagsAttr{},
+        /*data_attr=*/cuf::DataAttributeAttr{},
+        /*dummy_arg_no=*/mlir::IntegerAttr{});
+    return declare.getResult();
+  }
+
+  mlir::MLIRContext context;
+  std::unique_ptr<mlir::OpBuilder> builder;
+  mlir::OwningOpRef<mlir::ModuleOp> moduleOp;
+};
+
+// Modifying an operation that a cached query transitively depends on (here the
+// alloca underlying a declared variable) must evict that query's entry while
+// leaving unrelated entries intact.
+TEST_F(AliasAnalysisCacheTest, PreciseEvictionOnModify) {
+  mlir::Value var1 = createScalarVariable("x1");
+  mlir::Value var2 = createScalarVariable("x2");
+
+  fir::AliasAnalysis aa;
+  mlir::IRRewriter rewriter(&context);
+  aa.enableSourceCache(&rewriter);
+
+  fir::AliasAnalysis::Source s1 = aa.getSource(var1);
+  fir::AliasAnalysis::Source s2 = aa.getSource(var2);
+  std::size_t before = aa.getSourceCacheSizeForTesting();
+  EXPECT_GE(before, 2u);
+
+  // var1's source depends on its alloca; modifying the alloca must evict the
+  // var1 entry (a transitive dependency, not the queried op itself).
+  mlir::Operation *alloca1 =
+      var1.getDefiningOp()->getOperand(0).getDefiningOp();
+  ASSERT_TRUE(alloca1);
+  rewriter.modifyOpInPlace(alloca1, [] {});
+
+  std::size_t after = aa.getSourceCacheSizeForTesting();
+  EXPECT_LT(after, before); // some entry was evicted
+  EXPECT_GT(after, 0u); // but not everything (var2 is unaffected)
+
+  // Re-querying var2 must still return the same source (it was not evicted).
+  fir::AliasAnalysis::Source s2again = aa.getSource(var2);
+  EXPECT_EQ(s2again.origin, s2.origin);
+  EXPECT_EQ(s2again.kind, s2.kind);
+
+  // Sanity: var1 and var2 resolve to distinct origins.
+  EXPECT_NE(s1.origin, s2.origin);
+
+  aa.disableSourceCache();
+  EXPECT_EQ(aa.getSourceCacheSizeForTesting(), 0u);
+}
+
+// Erasing a cached query's defining operation must drop the affected entries
+// (the notification fires before the storage is freed), so no stale entry can
+// survive to alias with a later, pointer-reused operation.
+TEST_F(AliasAnalysisCacheTest, EvictionOnEraseClosesReuseHole) {
+  mlir::Value var1 = createScalarVariable("x1");
+  mlir::Value var2 = createScalarVariable("x2");
+
+  // A load whose result we query; it has no uses, so it can be erased.
+  mlir::Value load1 = fir::LoadOp::create(*builder, getLoc(), var1);
+
+  fir::AliasAnalysis aa;
+  mlir::IRRewriter rewriter(&context);
+  aa.enableSourceCache(&rewriter);
+
+  aa.getSource(load1);
+  aa.getSource(var2);
+  std::size_t before = aa.getSourceCacheSizeForTesting();
+  EXPECT_GE(before, 2u);
+
+  // Erase the load: its cache entry must be gone afterwards, while var2's
+  // entry survives.
+  mlir::Operation *loadOp = load1.getDefiningOp();
+  rewriter.eraseOp(loadOp);
+
+  std::size_t after = aa.getSourceCacheSizeForTesting();
+  EXPECT_LT(after, before);
+  EXPECT_GT(after, 0u);
+
+  // var2 is still cached and valid.
+  fir::AliasAnalysis::Source s2 = aa.getSource(var2);
+  EXPECT_EQ(s2.kind, fir::AliasAnalysis::SourceKind::Allocate);
+
+  aa.disableSourceCache();
+}
+
+// Without a rewriter the cache is a frozen snapshot: it still memoizes results,
+// but installs no listener (so the caller is responsible for scoping).
+TEST_F(AliasAnalysisCacheTest, FrozenModeMemoizes) {
+  mlir::Value var1 = createScalarVariable("x1");
+
+  fir::AliasAnalysis aa;
+  aa.enableSourceCache(/*rewriter=*/nullptr);
+  EXPECT_EQ(aa.getSourceCacheSizeForTesting(), 0u);
+
+  fir::AliasAnalysis::Source first = aa.getSource(var1);
+  EXPECT_GT(aa.getSourceCacheSizeForTesting(), 0u);
+  fir::AliasAnalysis::Source second = aa.getSource(var1);
+  EXPECT_EQ(first.origin, second.origin);
+
+  aa.disableSourceCache();
+  EXPECT_EQ(aa.getSourceCacheSizeForTesting(), 0u);
+}
diff --git a/flang/unittests/Optimizer/CMakeLists.txt b/flang/unittests/Optimizer/CMakeLists.txt
index 3af6f9d2a3724..6991d67191dd6 100644
--- a/flang/unittests/Optimizer/CMakeLists.txt
+++ b/flang/unittests/Optimizer/CMakeLists.txt
@@ -37,6 +37,7 @@ add_flang_unittest(FlangOptimizerTests
   Builder/Runtime/StopTest.cpp
   Builder/Runtime/TransformationalTest.cpp
   OpenACC/FIROpenACCPointerLikeTypeInterfaceTest.cpp
+  AliasAnalysisCacheTest.cpp
   FIRCallInterfaceTest.cpp
   FIRContextTest.cpp
   FIRTypesTest.cpp
diff --git a/mlir/include/mlir/Analysis/AliasAnalysis.h b/mlir/include/mlir/Analysis/AliasAnalysis.h
index 71a08780c48a6..b52d5a9a6f146 100644
--- a/mlir/include/mlir/Analysis/AliasAnalysis.h
+++ b/mlir/include/mlir/Analysis/AliasAnalysis.h
@@ -19,6 +19,8 @@
 
 namespace mlir {
 
+class RewriterBase;
+
 //===----------------------------------------------------------------------===//
 // AliasResult
 //===----------------------------------------------------------------------===//
@@ -191,7 +193,11 @@ struct AliasAnalysisTraits {
 
     /// Enable opt-in caching of alias query results on this implementation.
     /// Implementations that do not support caching ignore this request.
-    virtual void enableQueryCaching() = 0;
+    /// If `rewriter` is non-null, the implementation may install a listener on
+    /// it to invalidate cached results precisely as the IR is mutated; if null
+    /// the cache is a frozen snapshot valid only while the IR is not mutated in
+    /// a way that affects the cached queries.
+    virtual void enableQueryCaching(RewriterBase *rewriter) = 0;
 
     /// Disable and clear any query caching on this implementation.
     /// Implementations that do not support caching ignore this request.
@@ -202,7 +208,8 @@ struct AliasAnalysisTraits {
   /// `disableSourceCache()`. Mirrors the `has_is_invalidated` idiom used by
   /// the pass analysis manager.
   template <typename T>
-  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache());
+  using has_query_caching_t = decltype(std::declval<T &>().enableSourceCache(
+      std::declval<RewriterBase *>()));
 
   /// This class represents the `Model` of an alias analysis implementation
   /// `ImplT`. A model is instantiated for each alias analysis implementation
@@ -226,9 +233,9 @@ struct AliasAnalysisTraits {
 
     /// Forward query-caching control to the implementation when it provides
     /// the cache hooks; for implementations without them these are no-ops.
-    void enableQueryCaching() final {
+    void enableQueryCaching(RewriterBase *rewriter) final {
       if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
-        impl.enableSourceCache();
+        impl.enableSourceCache(rewriter);
     }
     void disableQueryCaching() final {
       if constexpr (llvm::is_detected<has_query_caching_t, ImplT>::value)
@@ -315,13 +322,21 @@ class AliasAnalysis {
   ///
   /// This is the only entry point to query caching: enabling and disabling
   /// are private so that caching is always paired with a guaranteed teardown.
-  /// A frozen (snapshot) cache is only valid while the IR is not mutated in a
-  /// way that affects the cached queries, so callers must keep the scope no
-  /// wider than such a no-mutation (or move-only) region.
+  ///
+  /// If `rewriter` is null (the default), the cache is a frozen snapshot that
+  /// is only valid while the IR is not mutated in a way that affects the
+  /// cached queries, so callers must keep the scope no wider than such a
+  /// no-mutation (or move-only) region. If `rewriter` is non-null,
+  /// implementations that support it install a listener on the rewriter and
+  /// invalidate cached results precisely as the IR is mutated through it; the
+  /// cache then stays valid as long as all mutations flow through that
+  /// rewriter.
   class QueryCacheScope {
   public:
-    explicit QueryCacheScope(AliasAnalysis &aa) : aa(aa) {
-      aa.enableQueryCaching();
+    explicit QueryCacheScope(AliasAnalysis &aa,
+                             RewriterBase *rewriter = nullptr)
+        : aa(aa) {
+      aa.enableQueryCaching(rewriter);
     }
     ~QueryCacheScope() { aa.disableQueryCaching(); }
 
@@ -337,7 +352,7 @@ class AliasAnalysis {
 private:
   /// Enable/disable query caching on every registered implementation that
   /// supports it. Private: use QueryCacheScope to guarantee teardown.
-  void enableQueryCaching();
+  void enableQueryCaching(RewriterBase *rewriter = nullptr);
   void disableQueryCaching();
 
   /// A set of internal alias analysis implementations.
diff --git a/mlir/lib/Analysis/AliasAnalysis.cpp b/mlir/lib/Analysis/AliasAnalysis.cpp
index 9db48e18042cc..79d7ced564a33 100644
--- a/mlir/lib/Analysis/AliasAnalysis.cpp
+++ b/mlir/lib/Analysis/AliasAnalysis.cpp
@@ -100,9 +100,9 @@ ModRefResult AliasAnalysis::getModRef(Operation *op, Value location) {
   return result;
 }
 
-void AliasAnalysis::enableQueryCaching() {
+void AliasAnalysis::enableQueryCaching(RewriterBase *rewriter) {
   for (const std::unique_ptr<Concept> &aliasImpl : aliasImpls)
-    aliasImpl->enableQueryCaching();
+    aliasImpl->enableQueryCaching(rewriter);
 }
 
 void AliasAnalysis::disableQueryCaching() {


From llvmlistbot at llvm.org  Wed Jul  1 09:47:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 09:47:51 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add precise
 listener-driven cache invalidation (PR #207032)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207032@github.com>
Message-ID: <6a4544b7.ff037943.172e14.3544@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-fir-hlfir

Author: Slava Zakharin (vzakhari)

<details>
<summary>Changes</summary>

Extend the opt-in query cache so it can be used safely by passes that mutate
the IR through a rewriter. The query-caching hooks gain a RewriterBase*
parameter (defaulted, so QueryCacheScope callers are unaffected): in frozen
mode (null rewriter) the cache is a snapshot as before; with a rewriter the
FIR AliasAnalysis installs a ForwardingListener on it and evicts cached
getSource() results precisely as operations are erased, modified, or replaced.

Each cached entry now records the operations it depends on (the def-use chain
and the dominating dummy_scope op visited while computing it), maintained via a
reverse-dependency index. Nested queries thread a dependency sink and union a
cache hit's dependencies into the enclosing entry, so transitive dependencies
are tracked. Erasure notifications also walk nested operations and fire before
the storage is freed, closing the window where a freed-and-reused pointer could
yield a stale hit.

Adds AliasAnalysisCacheTest covering precise eviction on modify, eviction on
erase, and frozen-mode memoization.

Assisted by Cursor


---

Patch is 30.16 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207032.diff


7 Files Affected:

- (modified) flang/include/flang/Optimizer/Analysis/AliasAnalysis.h (+119) 
- (modified) flang/lib/Optimizer/Analysis/AliasAnalysis.cpp (+165) 
- (modified) flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp (+10) 
- (added) flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp (+151) 
- (modified) flang/unittests/Optimizer/CMakeLists.txt (+1) 
- (modified) mlir/include/mlir/Analysis/AliasAnalysis.h (+75) 
- (modified) mlir/lib/Analysis/AliasAnalysis.cpp (+10) 


``````````diff
diff --git a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
index 832634a708dba..913270d4586ac 100644
--- a/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
+++ b/flang/include/flang/Optimizer/Analysis/AliasAnalysis.h
@@ -20,13 +20,29 @@
 #include "llvm/ADT/PointerUnion.h"
 #include "llvm/ADT/SmallVector.h"
 #include <memory>
+#include <utility>
 
 namespace fir {
 
+/// Listener used to evict getSource() cache entries as the IR is mutated
+/// through a rewriter. Defined in AliasAnalysis.cpp.
+class AliasAnalysisCacheListener;
+
 //===----------------------------------------------------------------------===//
 // AliasAnalysis
 //===----------------------------------------------------------------------===//
 struct AliasAnalysis {
+  // Special members are user-declared (and mostly defined out of line) because
+  // the source cache owns a std::unique_ptr to the incomplete listener type
+  // AliasAnalysisCacheListener. A move constructor is required so instances can
+  // be registered via mlir::AliasAnalysis::addAnalysisImplementation().
+  AliasAnalysis();
+  ~AliasAnalysis();
+  AliasAnalysis(AliasAnalysis &&);
+  AliasAnalysis &operator=(AliasAnalysis &&);
+  AliasAnalysis(const AliasAnalysis &) = delete;
+  AliasAnalysis &operator=(const AliasAnalysis &) = delete;
+
   // Structures to describe the memory source of a value.
 
   /// Kind of the memory source referenced by a value.
@@ -351,10 +367,39 @@ struct AliasAnalysis {
   /// snapshots are not collected, and getSource performs only the
   /// SourceKind/origin classification without that bookkeeping side
   /// effect.
+  ///
+  /// When source caching is enabled (see enableSourceCache()), the result is
+  /// memoized keyed on (value, flags), and recursive sub-queries share the
+  /// same cache. The cache is a frozen snapshot: it is only valid while the IR
+  /// reachable from the queried values is not mutated in a way that would
+  /// change the source. Callers must scope caching no wider than such a region
+  /// (see mlir::AliasAnalysis::QueryCacheScope).
   fir::AliasAnalysis::Source getSource(mlir::Value,
                                        bool getLastInstantiationPoint = false,
                                        bool collectScopedOrigins = true);
 
+  /// Enable/disable memoization of getSource() results. These are invoked by
+  /// the mlir::AliasAnalysis aggregator (via mlir::AliasAnalysis::
+  /// QueryCacheScope) and are not meant to be called directly by passes.
+  ///
+  /// If \p rewriter is null, the cache is a frozen snapshot with no automatic
+  /// invalidation: it is only valid while the IR reachable from the queried
+  /// values is not mutated. If \p rewriter is non-null, a listener is installed
+  /// on it (chained ahead of any existing listener) so that cached entries are
+  /// evicted precisely as operations are erased / modified / replaced through
+  /// that rewriter; the cache then stays valid as long as all such mutations
+  /// flow through the rewriter.
+  ///
+  /// disableSourceCache() clears the cache (and reverse index), uninstalls the
+  /// listener if any, and turns caching off.
+  void enableSourceCache(mlir::RewriterBase *rewriter = nullptr);
+  void disableSourceCache();
+
+  /// Testing only: number of entries currently held in the getSource() cache.
+  std::size_t getSourceCacheSizeForTesting() const {
+    return getSourceCache.size();
+  }
+
   /// Return true, if `ty` is a reference type to a boxed
   /// POINTER object or a raw fir::PointerType.
   static bool isPointerReference(mlir::Type ty);
@@ -370,6 +415,48 @@ struct AliasAnalysis {
   bool functionHasMultipleScopes(mlir::Value v);
 
 private:
+  friend class fir::AliasAnalysisCacheListener;
+
+  /// Cache key for getSource(): the queried value together with the two
+  /// boolean flags (getLastInstantiationPoint, collectScopedOrigins) packed
+  /// into the unsigned.
+  using SourceCacheKey = std::pair<mlir::Value, unsigned>;
+
+  /// A memoized getSource() result together with the operations it depends on
+  /// (the def-use chain and scope ops visited while computing it). The deps
+  /// drive precise eviction: if any of them is erased / modified / replaced,
+  /// this entry must be dropped.
+  struct CachedSource {
+    Source result;
+    llvm::SmallVector<mlir::Operation *, 4> deps;
+  };
+
+  /// Compute the memory source of a value. This is the uncached
+  /// implementation of getSource(); getSource() is a thin wrapper that
+  /// memoizes the result when source caching is enabled.
+  fir::AliasAnalysis::Source getSourceImpl(mlir::Value v,
+                                           bool getLastInstantiationPoint,
+                                           bool collectScopedOrigins);
+
+  /// Record \p op as a dependency of the getSource() computation currently in
+  /// progress (no-op when not collecting dependencies). Used to build the
+  /// reverse index that drives precise cache eviction.
+  void recordSourceDep(mlir::Operation *op) {
+    if (currentDepSink && op)
+      currentDepSink->push_back(op);
+  }
+
+  /// Evict every cached getSource() entry that depends on \p op (and, for
+  /// erasure, on any op nested within its regions), dropping the corresponding
+  /// reverse-index entries.
+  void evictSourceDependents(mlir::Operation *op);
+
+  /// Clear the getSource() memoization cache and its reverse index.
+  void clearSourceCache() {
+    getSourceCache.clear();
+    opDependents.clear();
+  }
+
   /// Build an intermediate Source rooted at the declare captured by the
   /// snapshot. Reuses getSource(declValue) for the SourceKind / origin
   /// classification (with collectScopedOrigins=false), then overrides
@@ -446,6 +533,38 @@ struct AliasAnalysis {
   /// functionHasMultipleScopes(); both true and false are cached so that
   /// repeated queries are O(1) without re-walking the function body.
   llvm::DenseMap<mlir::Operation *, bool> multiScopeCache;
+
+  /// Opt-in memoization of getSource() results, keyed on the queried value and
+  /// the two boolean flags (see SourceCacheKey). Only consulted/populated while
+  /// \c sourceCacheEnabled is set. Each entry also records the operations it
+  /// depends on so it can be evicted precisely. Without a listener (frozen
+  /// mode) this is a snapshot with no automatic invalidation, valid only while
+  /// the relevant IR is not mutated; with a listener installed (see
+  /// enableSourceCache(rewriter)) entries are evicted as the IR is mutated
+  /// through that rewriter. Driven by mlir::AliasAnalysis::QueryCacheScope.
+  llvm::DenseMap<SourceCacheKey, CachedSource> getSourceCache;
+
+  /// Reverse dependency index: for each operation, the set of getSourceCache
+  /// keys whose result depends on it. Maintained alongside getSourceCache and
+  /// used to evict only the affected entries when an op is mutated/erased.
+  llvm::DenseMap<mlir::Operation *, llvm::SmallVector<SourceCacheKey, 2>>
+      opDependents;
+
+  /// When non-null, getSourceImpl() and nested getSource() hits append the
+  /// operations they depend on here, so the in-progress getSource() result can
+  /// be stored with its dependencies. Saved/restored around nested queries.
+  llvm::SmallVectorImpl<mlir::Operation *> *currentDepSink = nullptr;
+
+  /// Whether getSource() should consult/populate getSourceCache.
+  bool sourceCacheEnabled = false;
+
+  /// The rewriter on which \c cacheListener is installed (listener mode), or
+  /// null in frozen mode. Used to uninstall the listener on disable.
+  mlir::RewriterBase *cacheRewriter = nullptr;
+
+  /// Listener installed on \c cacheRewriter that evicts cache entries as the
+  /// IR is mutated. Null in frozen mode. Owns a back-pointer to this analysis.
+  std::unique_ptr<AliasAnalysisCacheListener> cacheListener;
 };
 
 inline bool operator==(const AliasAnalysis::Source::SourceOrigin &lhs,
diff --git a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
index 02a628968ddc4..3fb73fcc78280 100644
--- a/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
+++ b/flang/lib/Optimizer/Analysis/AliasAnalysis.cpp
@@ -22,14 +22,17 @@
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/Value.h"
 #include "mlir/Interfaces/ControlFlowInterfaces.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include <optional>
+#include <utility>
 
 using namespace mlir;
 
@@ -1108,9 +1111,164 @@ static mlir::Value walkBlockArgPassThroughs(mlir::Value v) {
   return v;
 }
 
+//===----------------------------------------------------------------------===//
+// Source cache invalidation listener
+//===----------------------------------------------------------------------===//
+
+/// Listener installed on a rewriter while source caching is enabled in listener
+/// mode. It evicts the affected getSource() cache entries on each mutation and
+/// then forwards the notification to any previously installed listener so the
+/// cache composes with greedy/conversion driver listeners.
+class AliasAnalysisCacheListener
+    : public mlir::RewriterBase::ForwardingListener {
+public:
+  AliasAnalysisCacheListener(AliasAnalysis &aa,
+                             mlir::OpBuilder::Listener *previous)
+      : mlir::RewriterBase::ForwardingListener(previous), aa(aa),
+        previous(previous) {}
+
+  /// The listener that was installed on the rewriter before this one, so it
+  /// can be restored when caching is disabled.
+  mlir::OpBuilder::Listener *getPreviousListener() const { return previous; }
+
+  void notifyOperationErased(mlir::Operation *op) override {
+    // Runs before the operation's storage is freed, so evicting here closes
+    // the window where a freed-and-reused Operation* could produce a stale hit.
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationErased(op);
+  }
+
+  void notifyOperationModified(mlir::Operation *op) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationModified(op);
+  }
+
+  void notifyOperationReplaced(mlir::Operation *op,
+                               mlir::Operation *newOp) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationReplaced(op, newOp);
+  }
+
+  void notifyOperationReplaced(mlir::Operation *op,
+                               mlir::ValueRange replacement) override {
+    aa.evictSourceDependents(op);
+    ForwardingListener::notifyOperationReplaced(op, replacement);
+  }
+
+private:
+  AliasAnalysis &aa;
+  mlir::OpBuilder::Listener *previous;
+};
+
+//===----------------------------------------------------------------------===//
+// AliasAnalysis special members
+//===----------------------------------------------------------------------===//
+
+// Defined out of line because cacheListener is a std::unique_ptr to the
+// (here-complete) AliasAnalysisCacheListener.
+AliasAnalysis::AliasAnalysis() = default;
+AliasAnalysis::~AliasAnalysis() = default;
+AliasAnalysis::AliasAnalysis(AliasAnalysis &&) = default;
+AliasAnalysis &AliasAnalysis::operator=(AliasAnalysis &&) = default;
+
+//===----------------------------------------------------------------------===//
+// Source cache
+//===----------------------------------------------------------------------===//
+
+void AliasAnalysis::enableSourceCache(mlir::RewriterBase *rewriter) {
+  sourceCacheEnabled = true;
+  if (rewriter && !cacheListener) {
+    cacheRewriter = rewriter;
+    cacheListener = std::make_unique<AliasAnalysisCacheListener>(
+        *this, rewriter->getListener());
+    rewriter->setListener(cacheListener.get());
+  }
+}
+
+void AliasAnalysis::disableSourceCache() {
+  sourceCacheEnabled = false;
+  if (cacheListener) {
+    // Restore the listener that was installed before ours.
+    cacheRewriter->setListener(cacheListener->getPreviousListener());
+    cacheListener.reset();
+    cacheRewriter = nullptr;
+  }
+  clearSourceCache();
+}
+
+void AliasAnalysis::evictSourceDependents(mlir::Operation *op) {
+  if (getSourceCache.empty())
+    return;
+
+  // Collect the ops whose dependents must be evicted. For erased ops, this
+  // includes ops nested in their regions, since cached entries may depend on
+  // those nested ops too and their storage is freed along with the parent.
+  llvm::SmallVector<mlir::Operation *, 8> worklist{op};
+  while (!worklist.empty()) {
+    mlir::Operation *cur = worklist.pop_back_val();
+    auto it = opDependents.find(cur);
+    if (it != opDependents.end()) {
+      // Copy the keys out before erasing, then drop the cache entries.
+      llvm::SmallVector<SourceCacheKey, 2> keys = std::move(it->second);
+      opDependents.erase(it);
+      for (const SourceCacheKey &key : keys)
+        getSourceCache.erase(key);
+    }
+    for (mlir::Region &region : cur->getRegions())
+      for (mlir::Block &block : region)
+        for (mlir::Operation &nested : block)
+          worklist.push_back(&nested);
+  }
+}
+
 AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
                                                bool getLastInstantiationPoint,
                                                bool collectScopedOrigins) {
+  if (!sourceCacheEnabled)
+    return getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+
+  // Key on the queried value and the two boolean flags. Recursive sub-queries
+  // go through this same wrapper, so the whole walk is memoized.
+  SourceCacheKey key{v, (getLastInstantiationPoint ? 1u : 0u) |
+                            (collectScopedOrigins ? 2u : 0u)};
+  auto it = getSourceCache.find(key);
+  if (it != getSourceCache.end()) {
+    // Propagate this entry's dependencies into the enclosing query (if any) so
+    // transitive dependencies are tracked: if a dep of this entry is later
+    // erased, the enclosing entry that incorporated it is evicted too.
+    if (currentDepSink)
+      currentDepSink->append(it->second.deps.begin(), it->second.deps.end());
+    return it->second.result;
+  }
+
+  // Miss: compute with a fresh dependency sink, then propagate to the parent.
+  llvm::SmallVector<mlir::Operation *, 4> deps;
+  llvm::SmallVectorImpl<mlir::Operation *> *parentSink = currentDepSink;
+  currentDepSink = &deps;
+  Source source =
+      getSourceImpl(v, getLastInstantiationPoint, collectScopedOrigins);
+  currentDepSink = parentSink;
+
+  // Deduplicate before storing / indexing (the walk may visit an op multiple
+  // times). Sorting by pointer only affects internal bookkeeping, not emitted
+  // IR, so dump determinism is preserved.
+  llvm::sort(deps);
+  deps.erase(llvm::unique(deps), deps.end());
+
+  if (parentSink)
+    parentSink->append(deps.begin(), deps.end());
+
+  bool inserted =
+      getSourceCache.try_emplace(key, CachedSource{source, deps}).second;
+  if (inserted)
+    for (mlir::Operation *depOp : deps)
+      opDependents[depOp].push_back(key);
+  return source;
+}
+
+AliasAnalysis::Source
+AliasAnalysis::getSourceImpl(mlir::Value v, bool getLastInstantiationPoint,
+                             bool collectScopedOrigins) {
   // If v is a pass-through block argument (see walkBlockArgPassThroughs),
   // continue from the underlying operand so the tracking loop below has a
   // defining op to chew on. Without this, a recursive query like the one in
@@ -1147,6 +1305,9 @@ AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
   // buildSourceAtDeclare reuses getSource purely for declare classification).
   llvm::SmallVector<Source::ScopedOrigin, 4> scopedOrigins;
   while (defOp && !breakFromLoop) {
+    // Record each operation visited along the def-use chain as a dependency
+    // of this query, so the cached result is evicted if any of them changes.
+    recordSourceDep(defOp);
     // Operations may have multiple results, so we need to analyze
     // the result for which the source is queried.
     auto opResult = mlir::cast<OpResult>(v);
@@ -1414,6 +1575,10 @@ AliasAnalysis::Source AliasAnalysis::getSource(mlir::Value v,
           if (collectScopedOrigins) {
             Source::ScopedOrigin scopedOrigin;
             scopedOrigin.scope = getDeclarationScope(op);
+            // The chosen scope depends on the dominating fir.dummy_scope op;
+            // record it so the entry is evicted if that scope op changes.
+            if (scopedOrigin.scope)
+              recordSourceDep(scopedOrigin.scope.getDefiningOp());
             scopedOrigin.declValue = opResult;
             scopedOrigin.accessPath.steps.assign(pathSteps.rbegin(),
                                                  pathSteps.rend());
diff --git a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
index 14fd5c6dd7fbc..2f640199252e4 100644
--- a/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
+++ b/flang/lib/Optimizer/Transforms/LoopInvariantCodeMotion.cpp
@@ -268,6 +268,16 @@ void LoopInvariantCodeMotion::runOnOperation() {
   auto &aliasAnalysis = getAnalysis<AliasAnalysis>();
   aliasAnalysis.addAnalysisImplementation(fir::AliasAnalysis{});
 
+  // Enable alias-query caching for the duration of this pass. The scope
+  // enables caching on every registered implementation that supports it
+  // (the FIR AliasAnalysis memoizes getSource()) and no-ops on the rest, and
+  // reliably disables it on every exit. This is a frozen-snapshot cache with
+  // no listener: it is sound here because LICM only moves operations, and
+  // getSource()'s position-dependent inputs are already frozen for the pass
+  // run by the AliasAnalysis's own dominance/scope caches, so caching is
+  // observationally equivalent to no caching across the hoists.
+  mlir::AliasAnalysis::QueryCacheScope cacheScope(aliasAnalysis);
+
   std::function<bool(Operation *, LoopLikeOpInterface, bool)>
       shouldMoveOutOfLoop = [&](Operation *op, LoopLikeOpInterface loopLike,
                                 bool maybeConditionallyExecuted) {
diff --git a/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp b/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp
new file mode 100644
index 0000000000000..b199adc4191b6
--- /dev/null
+++ b/flang/unittests/Optimizer/AliasAnalysisCacheTest.cpp
@@ -0,0 +1,151 @@
+//===- AliasAnalysisCacheTest.cpp -----------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Tests for the listener-driven invalidation of fir::AliasAnalysis's getSource
+// cache: mutating the IR through a rewriter must evict precisely the cached
+// entries that depend on the mutated operation, and an erase must drop the
+// affected entries before the operation's storage is freed (so a freed,
+// possibly reused, pointer can never produce a stale cache hit).
+//
+//===----------------------------------------------------------------------===//
+
+#include "gtest/gtest.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "flang/Optimizer/Analysis/AliasAnalysis.h"
+#include "flang/Optimizer/Dialect/FIROps.h"
+#include "flang/Optimizer/Support/InitFIR.h"
+
+struct AliasAnalysisCacheTest : public testing::Test {
+public:
+  void SetUp() override {
+    fir::support::loadDialects(context);
+    builder = std::make_unique<mlir::OpBuilder>(&context);
+    mlir::Location loc = builder->getUnknownLoc();
+
+    moduleOp = mlir::ModuleOp::create(*builder, loc);
+    builder->setInsertionPointToStart(moduleOp->getBody());
+    mlir::func::FuncOp func = mlir::func::FuncOp::create(
+        *builder, loc, "test", builder->getFunctionType({}, {}));
+    builder->setInsertionPointToStart(func.addEntryBlock());
+  }
+
+  mlir::Location getLoc() { return builder->getUnknownLoc(); }
+
+  // Build an `alloca -> declare` chain for a scalar i32 variable and return the
+  // declared variable address...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207032

From llvmlistbot at llvm.org  Wed Jul  1 10:05:54 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 01 Jul 2026 10:05:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] append triples when materializing
 reduction destroy recipes (PR #207034)
Message-ID: <llvm.org/llvm/llvm-project/pull/207034@github.com>


https://github.com/rscottmanley created https://github.com/llvm/llvm-project/pull/207034

Append the triples if they exist when materializing the destroy region.

>From 98fba352c0d84c0266a85c5e867ef5c1a745bba6 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Wed, 1 Jul 2026 09:50:02 -0700
Subject: [PATCH] [OpenACC] append triples when materializing reduction destroy
 recipes

Append the triples if they exist when materializing the destroy region.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index e99cdaa15dae6..398f4d4bbfdf6 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -392,6 +392,7 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
 
     if (!recipe.getDestroyRegion().empty()) {
       SmallVector<Value> results{origPtr, reductionOp.getResult()};
+      results.append(triples);
       Block::iterator ip = std::next(Block::iterator(combineRegionOp));
       cloneDestroy(loc, recipe, combineRegionOp->getBlock(), ip, results);
     }


From llvmlistbot at llvm.org  Wed Jul  1 10:06:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 10:06:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] append triples when materializing
 reduction destroy recipes (PR #207034)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207034@github.com>
Message-ID: <6a454916.fc0deea3.239ca8.9b6c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-openacc

@llvm/pr-subscribers-mlir

Author: Scott Manley (rscottmanley)

<details>
<summary>Changes</summary>

Append the triples if they exist when materializing the destroy region.

---
Full diff: https://github.com/llvm/llvm-project/pull/207034.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp (+1) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index e99cdaa15dae6..398f4d4bbfdf6 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -392,6 +392,7 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
 
     if (!recipe.getDestroyRegion().empty()) {
       SmallVector<Value> results{origPtr, reductionOp.getResult()};
+      results.append(triples);
       Block::iterator ip = std::next(Block::iterator(combineRegionOp));
       cloneDestroy(loc, recipe, combineRegionOp->getBlock(), ip, results);
     }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207034

From llvmlistbot at llvm.org  Wed Jul  1 10:06:51 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 01 Jul 2026 10:06:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] append triples when materializing
 reduction destroy recipes (PR #207034)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207034@github.com>
Message-ID: <6a45492b.31d1b862.a81c9.c8b1@mx.google.com>


rscottmanley wrote:

No test here because it seems like their is a TODO to not use the triple but to use a acc::BoundsOp instead from FIR. I don't see much reason to add tests for memref lowering for something that's not *really* supported

https://github.com/llvm/llvm-project/pull/207034

From llvmlistbot at llvm.org  Wed Jul  1 10:53:47 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 01 Jul 2026 10:53:47 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] Reland "[mlir][nvvm] extract
 NVVM inliner interface from LLVM dialect" with flang fixes (PR #206782)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206782@github.com>
Message-ID: <6a45542b.a151105c.30c1b9.20b6@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `ppc64le-flang-rhel-clang` running on `ppc64le-flang-rhel-test` while building `flang,mlir,utils` at step 6 "test-build-unified-tree-check-flang".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/157/builds/47983

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 6 (test-build-unified-tree-check-flang) failure: 1200 seconds without output running [b'ninja', b'check-flang'], attempting to kill
...
PASS: Flang :: Transforms/DoConcurrent/use_loop_bounds_in_body.f90 (4646 of 4656)
PASS: Flang :: Transforms/debug-line-table.fir (4647 of 4656)
PASS: flang-Unit :: Optimizer/./FlangOptimizerTests/165/168 (4648 of 4656)
PASS: Flang :: Lower/zero_init.f90 (4649 of 4656)
PASS: Flang :: Transforms/OpenACC/acc-implicit-data-fortran.F90 (4650 of 4656)
PASS: Flang :: Lower/zero_init_default_init.f90 (4651 of 4656)
PASS: Flang :: Lower/debug-use-stmt.f90 (4652 of 4656)
PASS: Flang :: Semantics/numeric_storage_size.f90 (4653 of 4656)
PASS: Flang :: Transforms/debug-dwarf-version.fir (4654 of 4656)
PASS: Flang :: Transforms/vscale-attr.fir (4655 of 4656)
command timed out: 1200 seconds without output running [b'ninja', b'check-flang'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=4326.249857

```

</details>

https://github.com/llvm/llvm-project/pull/206782

From llvmlistbot at llvm.org  Wed Jul  1 11:53:18 2026
From: llvmlistbot at llvm.org (Caroline Newcombe)
Date: Wed, 01 Jul 2026 11:53:18 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a45621e.c56fb102.11841f.fc5c@mx.google.com>


cenewcombe wrote:

@dreachem FYI I am not able to add you as a reviewer, but you may be able to add yourself if you comment on the PR.

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul  1 12:06:36 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 01 Jul 2026 12:06:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix blocking pass for scf.if
	distribution (PR #207060)
Message-ID: <llvm.org/llvm/llvm-project/pull/207060@github.com>


https://github.com/nbpatel created https://github.com/llvm/llvm-project/pull/207060

None

>From d81f090b2cb072e29317ebac50070010344492bd Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 30 Jun 2026 17:59:49 +0000
Subject: [PATCH 1/2] Fix blocking for scf.if with 1:N expanded results

---
 .../XeGPU/Transforms/XeGPUBlocking.cpp        |  2 +-
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   | 17 +++++++++
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   | 35 +++++++++++++++++++
 3 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
index 4f81085c21e86..57b8ac73f41ac 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
@@ -430,7 +430,7 @@ void XeGPUBlockingPass::runOnOperation() {
     // `layout_result_N` lands on the wrong (renumbered) result, corrupting the
     // count invariant and leaving the loop illegal.
     op->walk([](Operation *loopOp) {
-      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp>(loopOp))
+      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp, scf::IfOp>(loopOp))
         return;
       SmallVector<StringRef> toRemove;
       for (const NamedAttribute &attr : loopOp->getAttrs()) {
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 1aad2aa77741b..9ab0d55f51be4 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -924,6 +924,23 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
         recordTypes(init, {arg, res, yieldVal});
       return;
     }
+    if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
+      // scf.if is not loop-carried, but its results still expand 1:N during
+      // blocking. Record each result together with the then/else yield operands
+      // that feed it so they all convert to the same distributed type by Value
+      // identity. The yield operand's layout is the authoritative source (the
+      // result's own per-position layout attr is stripped before conversion).
+      scf::YieldOp thenYield = ifOp.thenYield();
+      scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
+      for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
+        Value thenVal = thenYield.getOperand(idx);
+        SmallVector<Value> dests{res, thenVal};
+        if (elseYield)
+          dests.push_back(elseYield.getOperand(idx));
+        recordTypes(thenVal, dests);
+      }
+      return;
+    }
   });
   return loopArgTypes;
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index f2c25f45f45bc..b94119fa33121 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -815,3 +815,38 @@ gpu.module @test_kernel {
     gpu.return
   }
 }
+
+// -----
+// Test that an scf.if with results that expand 1:N during blocking is
+// legalized. The two results use different layouts (a regular 2D layout that
+// expands 1:N and a slice layout), and the slice result is consumed by a
+// broadcast/transpose chain. This exercises the scf.if structural type
+// conversion, which previously failed to legalize because the per-position
+// layout attrs were not stripped and the results were not recorded in the
+// loop-arg type map.
+#if_a = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#if_b = #xegpu.slice<#xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>
+gpu.module @test_kernel {
+  // CHECK-LABEL: func @if_one_to_n_results
+  gpu.func @if_one_to_n_results(%c: i1, %p: i64) {
+    %c0 = arith.constant 0 : index
+    // CHECK: scf.if
+    // CHECK-COUNT-8: vector<8x16xf32>
+    %r:2 = scf.if %c -> (vector<16x64xf32>, vector<16xf32>) {
+      %a = arith.constant {layout_result_0 = #if_a} dense<1.0> : vector<16x64xf32>
+      %b = arith.constant {layout_result_0 = #if_b} dense<1.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    } else {
+      %a = arith.constant {layout_result_0 = #if_a} dense<0.0> : vector<16x64xf32>
+      %b = arith.constant {layout_result_0 = #if_b} dense<0.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    } {layout_result_0 = #if_a, layout_result_1 = #if_b}
+    %bc = vector.broadcast %r#1 {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<64x16xf32>
+    %tp = vector.transpose %bc, [1, 0] {layout_result_0 = #if_a} : vector<64x16xf32> to vector<16x64xf32>
+    %add = arith.addf %r#0, %tp {layout_result_0 = #if_a} : vector<16x64xf32>
+    %tr = arith.truncf %add {layout_result_0 = #if_a} : vector<16x64xf32> to vector<16x64xf16>
+    %td = xegpu.create_nd_tdesc %p, shape:[16, 64], strides:[64, 1] : i64 -> !xegpu.tensor_desc<16x64xf16, #if_a>
+    xegpu.store_nd %tr, %td[%c0, %c0] <{layout = #if_a}> : vector<16x64xf16>, !xegpu.tensor_desc<16x64xf16, #if_a>
+    gpu.return
+  }
+}

>From c5ceb9208dd9f0f99b42dd8d2e6d009d43a3753f Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Wed, 1 Jul 2026 17:18:20 +0000
Subject: [PATCH 2/2] Fix comments

---
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp |  8 +++---
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir | 27 ++++++++-------------
 2 files changed, 13 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9ab0d55f51be4..7faada2b8223f 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -925,11 +925,9 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
       return;
     }
     if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
-      // scf.if is not loop-carried, but its results still expand 1:N during
-      // blocking. Record each result together with the then/else yield operands
-      // that feed it so they all convert to the same distributed type by Value
-      // identity. The yield operand's layout is the authoritative source (the
-      // result's own per-position layout attr is stripped before conversion).
+      // scf.if results expand 1:N during blocking. Record each result with the
+      // then/else yield operands that feed it so they convert to the same
+      // distributed type by Value identity, using the yield operand's layout.
       scf::YieldOp thenYield = ifOp.thenYield();
       scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
       for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b94119fa33121..b0b16c2adba6e 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -817,15 +817,8 @@ gpu.module @test_kernel {
 }
 
 // -----
-// Test that an scf.if with results that expand 1:N during blocking is
-// legalized. The two results use different layouts (a regular 2D layout that
-// expands 1:N and a slice layout), and the slice result is consumed by a
-// broadcast/transpose chain. This exercises the scf.if structural type
-// conversion, which previously failed to legalize because the per-position
-// layout attrs were not stripped and the results were not recorded in the
-// loop-arg type map.
+// Test that an scf.if whose results expand 1:N during blocking is legalized.
 #if_a = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
-#if_b = #xegpu.slice<#xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>
 gpu.module @test_kernel {
   // CHECK-LABEL: func @if_one_to_n_results
   gpu.func @if_one_to_n_results(%c: i1, %p: i64) {
@@ -833,18 +826,18 @@ gpu.module @test_kernel {
     // CHECK: scf.if
     // CHECK-COUNT-8: vector<8x16xf32>
     %r:2 = scf.if %c -> (vector<16x64xf32>, vector<16xf32>) {
-      %a = arith.constant {layout_result_0 = #if_a} dense<1.0> : vector<16x64xf32>
-      %b = arith.constant {layout_result_0 = #if_b} dense<1.0> : vector<16xf32>
+      %a = arith.constant dense<1.0> : vector<16x64xf32>
+      %b = arith.constant dense<1.0> : vector<16xf32>
       scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
     } else {
-      %a = arith.constant {layout_result_0 = #if_a} dense<0.0> : vector<16x64xf32>
-      %b = arith.constant {layout_result_0 = #if_b} dense<0.0> : vector<16xf32>
+      %a = arith.constant dense<0.0> : vector<16x64xf32>
+      %b = arith.constant dense<0.0> : vector<16xf32>
       scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
-    } {layout_result_0 = #if_a, layout_result_1 = #if_b}
-    %bc = vector.broadcast %r#1 {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<64x16xf32>
-    %tp = vector.transpose %bc, [1, 0] {layout_result_0 = #if_a} : vector<64x16xf32> to vector<16x64xf32>
-    %add = arith.addf %r#0, %tp {layout_result_0 = #if_a} : vector<16x64xf32>
-    %tr = arith.truncf %add {layout_result_0 = #if_a} : vector<16x64xf32> to vector<16x64xf16>
+    }
+    %bc = vector.broadcast %r#1 : vector<16xf32> to vector<64x16xf32>
+    %tp = vector.transpose %bc, [1, 0] : vector<64x16xf32> to vector<16x64xf32>
+    %add = arith.addf %r#0, %tp : vector<16x64xf32>
+    %tr = arith.truncf %add : vector<16x64xf32> to vector<16x64xf16>
     %td = xegpu.create_nd_tdesc %p, shape:[16, 64], strides:[64, 1] : i64 -> !xegpu.tensor_desc<16x64xf16, #if_a>
     xegpu.store_nd %tr, %td[%c0, %c0] <{layout = #if_a}> : vector<16x64xf16>, !xegpu.tensor_desc<16x64xf16, #if_a>
     gpu.return


From llvmlistbot at llvm.org  Wed Jul  1 12:49:24 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Wed, 01 Jul 2026 12:49:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add attribute indicating source
 for created `FieldOp`s (PR #207069)
Message-ID: <llvm.org/llvm/llvm-project/pull/207069@github.com>


https://github.com/beamandala created https://github.com/llvm/llvm-project/pull/207069

Adds an informative `source` attr to created `FieldOp`s based on whether the `FieldOp` was a function arg or a `GlobalOp`.

This is motivated by the reflection map pass (#205464) which will be run after the `wrap-emitc-func-in-class` pass and needs attributes to include and exclude `FieldOp`s from the reflection map.

>From f50e972826879216f9db150df4bce16a480d2234 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 30 Jun 2026 13:07:00 -0700
Subject: [PATCH] [mlir][EmitC] Add attribute indicating source for created
 `FieldOp`s

Adds an informative `source` attr to created `FieldOp`s based on whether the
`FieldOp` was a function arg or a `GlobalOp`. This is motivated by the
reflection map pass (PR #205464) which will be run after the
`wrap-emitc-func-in-class` pass and needs attributes to include and
exclude `FieldOp`s from the reflection map.
---
 .../EmitC/Transforms/WrapFuncInClass.cpp      | 14 ++++++++++---
 .../Dialect/EmitC/wrap-func-in-class.mlir     | 20 +++++++++----------
 2 files changed, 21 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
index aaceddb1e1b61..c9ece98029825 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
@@ -93,16 +93,24 @@ class WrapFuncInClass : public OpRewritePattern<FuncOp> {
       FieldOp fieldop = FieldOp::create(rewriter, funcOp->getLoc(), fieldName,
                                         typeAttr, nullptr);
 
+      NamedAttrList attrs(funcOp.getArgAttrDict(idx));
+      attrs.set("was_arg", rewriter.getUnitAttr());
       if (argAttrs && idx < argAttrs->size()) {
-        fieldop->setDiscardableAttrs(funcOp.getArgAttrDict(idx));
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
     auto globalsIt = globalsToMove.find(funcOp);
     if (globalsIt != globalsToMove.end()) {
       for (auto global : globalsIt->second) {
-        FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
-                        global.getTypeAttr(), global.getInitialValueAttr());
+        FieldOp fieldop =
+            FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
+                            global.getTypeAttr(), global.getInitialValueAttr());
+
+        NamedAttrList attrs;
+        attrs.append(global->getDiscardableAttrs());
+        attrs.set("was_global", rewriter.getUnitAttr());
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
diff --git a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
index 7d5b3d30a64ba..ed62ca7928e2c 100644
--- a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
+++ b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
@@ -37,9 +37,9 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @modelClass {
-// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature"}
-// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.name_hint = "some_feature"}
-// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32>  {emitc.name_hint = "output_0"}
+// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature", was_arg}
+// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.name_hint = "some_feature", was_arg}
+// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32> {emitc.name_hint = "output_0", was_arg}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK:       get_field @fieldName1 : !emitc.array<1xf32>
@@ -71,7 +71,7 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @fooClass {
-// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK:       return
@@ -97,7 +97,7 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr2 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -129,14 +129,14 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr3 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr1 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr2 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -165,14 +165,14 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -197,7 +197,7 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         %1 = get_field @global_arr : !emitc.array<1xi8>


From llvmlistbot at llvm.org  Wed Jul  1 12:49:56 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Wed, 01 Jul 2026 12:49:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add attribute indicating source
 for created `FieldOp`s (PR #207069)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207069@github.com>
Message-ID: <6a456f64.f8b79514.3b441.3bd4@mx.google.com>


beamandala wrote:

@mtrofin @ilovepi @banach-space @ioghiban 

https://github.com/llvm/llvm-project/pull/207069

From llvmlistbot at llvm.org  Wed Jul  1 12:50:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 12:50:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add attribute indicating source
 for created `FieldOp`s (PR #207069)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207069@github.com>
Message-ID: <6a456f76.14d9b9dc.140829.d193@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-emitc

Author: Bhavesh M (beamandala)

<details>
<summary>Changes</summary>

Adds an informative `source` attr to created `FieldOp`s based on whether the `FieldOp` was a function arg or a `GlobalOp`.

This is motivated by the reflection map pass (#<!-- -->205464) which will be run after the `wrap-emitc-func-in-class` pass and needs attributes to include and exclude `FieldOp`s from the reflection map.

---
Full diff: https://github.com/llvm/llvm-project/pull/207069.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp (+11-3) 
- (modified) mlir/test/Dialect/EmitC/wrap-func-in-class.mlir (+10-10) 


``````````diff
diff --git a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
index aaceddb1e1b61..c9ece98029825 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
@@ -93,16 +93,24 @@ class WrapFuncInClass : public OpRewritePattern<FuncOp> {
       FieldOp fieldop = FieldOp::create(rewriter, funcOp->getLoc(), fieldName,
                                         typeAttr, nullptr);
 
+      NamedAttrList attrs(funcOp.getArgAttrDict(idx));
+      attrs.set("was_arg", rewriter.getUnitAttr());
       if (argAttrs && idx < argAttrs->size()) {
-        fieldop->setDiscardableAttrs(funcOp.getArgAttrDict(idx));
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
     auto globalsIt = globalsToMove.find(funcOp);
     if (globalsIt != globalsToMove.end()) {
       for (auto global : globalsIt->second) {
-        FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
-                        global.getTypeAttr(), global.getInitialValueAttr());
+        FieldOp fieldop =
+            FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
+                            global.getTypeAttr(), global.getInitialValueAttr());
+
+        NamedAttrList attrs;
+        attrs.append(global->getDiscardableAttrs());
+        attrs.set("was_global", rewriter.getUnitAttr());
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
diff --git a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
index 7d5b3d30a64ba..ed62ca7928e2c 100644
--- a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
+++ b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
@@ -37,9 +37,9 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @modelClass {
-// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature"}
-// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.name_hint = "some_feature"}
-// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32>  {emitc.name_hint = "output_0"}
+// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature", was_arg}
+// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.name_hint = "some_feature", was_arg}
+// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32> {emitc.name_hint = "output_0", was_arg}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK:       get_field @fieldName1 : !emitc.array<1xf32>
@@ -71,7 +71,7 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @fooClass {
-// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK:       return
@@ -97,7 +97,7 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr2 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -129,14 +129,14 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr3 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr1 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr2 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -165,14 +165,14 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -197,7 +197,7 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         %1 = get_field @global_arr : !emitc.array<1xi8>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207069

From llvmlistbot at llvm.org  Wed Jul  1 12:50:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 12:50:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add attribute indicating source
 for created `FieldOp`s (PR #207069)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207069@github.com>
Message-ID: <6a456f76.576b3af0.2cb9b7.a9d1@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Bhavesh M (beamandala)

<details>
<summary>Changes</summary>

Adds an informative `source` attr to created `FieldOp`s based on whether the `FieldOp` was a function arg or a `GlobalOp`.

This is motivated by the reflection map pass (#<!-- -->205464) which will be run after the `wrap-emitc-func-in-class` pass and needs attributes to include and exclude `FieldOp`s from the reflection map.

---
Full diff: https://github.com/llvm/llvm-project/pull/207069.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp (+11-3) 
- (modified) mlir/test/Dialect/EmitC/wrap-func-in-class.mlir (+10-10) 


``````````diff
diff --git a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
index aaceddb1e1b61..c9ece98029825 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/WrapFuncInClass.cpp
@@ -93,16 +93,24 @@ class WrapFuncInClass : public OpRewritePattern<FuncOp> {
       FieldOp fieldop = FieldOp::create(rewriter, funcOp->getLoc(), fieldName,
                                         typeAttr, nullptr);
 
+      NamedAttrList attrs(funcOp.getArgAttrDict(idx));
+      attrs.set("was_arg", rewriter.getUnitAttr());
       if (argAttrs && idx < argAttrs->size()) {
-        fieldop->setDiscardableAttrs(funcOp.getArgAttrDict(idx));
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
     auto globalsIt = globalsToMove.find(funcOp);
     if (globalsIt != globalsToMove.end()) {
       for (auto global : globalsIt->second) {
-        FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
-                        global.getTypeAttr(), global.getInitialValueAttr());
+        FieldOp fieldop =
+            FieldOp::create(rewriter, funcOp->getLoc(), global.getSymNameAttr(),
+                            global.getTypeAttr(), global.getInitialValueAttr());
+
+        NamedAttrList attrs;
+        attrs.append(global->getDiscardableAttrs());
+        attrs.set("was_global", rewriter.getUnitAttr());
+        fieldop->setDiscardableAttrs(attrs);
       }
     }
 
diff --git a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
index 7d5b3d30a64ba..ed62ca7928e2c 100644
--- a/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
+++ b/mlir/test/Dialect/EmitC/wrap-func-in-class.mlir
@@ -37,9 +37,9 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @modelClass {
-// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature"}
-// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.name_hint = "some_feature"}
-// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32>  {emitc.name_hint = "output_0"}
+// CHECK:     emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.name_hint = "another_feature", was_arg}
+// CHECK:     emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.name_hint = "some_feature", was_arg}
+// CHECK:     emitc.field @fieldName2 : !emitc.array<1xf32> {emitc.name_hint = "output_0", was_arg}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK:       get_field @fieldName1 : !emitc.array<1xf32>
@@ -71,7 +71,7 @@ module attributes { } {
 }
 
 // CHECK:   emitc.class @fooClass {
-// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK:     emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK:     emitc.func @"operator()"() {
 // CHECK:       %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK:       return
@@ -97,7 +97,7 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr2 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -129,14 +129,14 @@ module attributes { } {
 // CHECK:   module {
 // CHECK-NEXT:     emitc.global static const @global_arr3 : !emitc.array<1xi8> = dense<0>
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr1 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr1 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr2 : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr2 : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -165,14 +165,14 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
 // CHECK-NEXT:       }
 // CHECK-NEXT:     }
 // CHECK-NEXT:     emitc.class @barClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         return
@@ -197,7 +197,7 @@ module attributes { } {
 
 // CHECK:   module {
 // CHECK-NEXT:     emitc.class @fooClass {
-// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0>
+// CHECK-NEXT:       emitc.field @global_arr : !emitc.array<1xi8> = dense<0> {was_global}
 // CHECK-NEXT:       emitc.func @"operator()"() {
 // CHECK-NEXT:         %0 = get_field @global_arr : !emitc.array<1xi8>
 // CHECK-NEXT:         %1 = get_field @global_arr : !emitc.array<1xi8>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207069

From llvmlistbot at llvm.org  Wed Jul  1 15:47:02 2026
From: llvmlistbot at llvm.org (Andrew Lazarev)
Date: Wed, 01 Jul 2026 15:47:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add ConversionTarget dynamic
	legality C API (PR #206161)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206161@github.com>
Message-ID: <6a4598e6.30759430.9c78.6331@mx.google.com>


alazarev wrote:

Looks like it broke https://lab.llvm.org/buildbot/#/builders/169/builds/24081 and https://lab.llvm.org/buildbot/#/builders/52/builds/18263

https://github.com/llvm/llvm-project/pull/206161

From llvmlistbot at llvm.org  Wed Jul  1 18:52:03 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Wed, 01 Jul 2026 18:52:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reimplement
 SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
 (PR #204971)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204971@github.com>
Message-ID: <6a45c443.cbc3f03a.3be201.17f9@mx.google.com>


================
@@ -394,3 +397,196 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// CHECK-LABEL: func.func @pack_3d_to_5d(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x32x64xf32>)
+// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2, 3], [4]] output_shape [3, 1, 1, 32, 64] : tensor<3x32x64xf32> into tensor<3x1x1x32x64xf32>
+// CHECK:         return %[[EXPANDED]] : tensor<3x1x1x32x64xf32>
+func.func @pack_3d_to_5d(%arg0: tensor<3x32x64xf32>) -> tensor<3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [1, 2] inner_tiles = [32, 64] into %empty : tensor<3x32x64xf32> -> tensor<3x1x1x32x64xf32>
+  return %0 : tensor<3x1x1x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @pack_3d_to_5d_with_outer_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x32x64xf32>)
+// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2, 3], [4]] output_shape [3, 1, 1, 32, 64] : tensor<3x32x64xf32> into tensor<3x1x1x32x64xf32>
+// CHECK:         return %[[EXPANDED]] : tensor<3x1x1x32x64xf32>
+func.func @pack_3d_to_5d_with_outer_dims_perm(%arg0: tensor<3x32x64xf32>) -> tensor<3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [0, 2, 1] inner_dims_pos = [1, 2] inner_tiles = [32, 64] into %empty : tensor<3x32x64xf32> -> tensor<3x1x1x32x64xf32>
+  return %0 : tensor<3x1x1x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @pack_3d_to_5d_dynamic_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x?x64xf32>)
+// CHECK:         %[[C1:.+]] = arith.constant 1 : index
+// CHECK:         %[[DIM1:.+]] = tensor.dim %[[ARG0]], %[[C1]]
+// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2], [3, 4]] output_shape [32, 1, %[[DIM1]], 1, 64] : tensor<32x?x64xf32> into tensor<32x1x?x1x64xf32>
+// CHECK:         return %[[EXPANDED]] : tensor<32x1x?x1x64xf32>
+func.func @pack_3d_to_5d_dynamic_shape(%arg0: tensor<32x?x64xf32>) -> tensor<32x1x?x1x64xf32> {
+  %c1 = arith.constant 1 : index
+  %dim1 = tensor.dim %arg0, %c1 : tensor<32x?x64xf32>
+  %empty = tensor.empty(%dim1) : tensor<32x1x?x1x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [0, 2, 1] inner_dims_pos = [1, 2] inner_tiles = [1, 64] into %empty : tensor<32x?x64xf32> -> tensor<32x1x?x1x64xf32>
+  return %0 : tensor<32x1x?x1x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @pack_nd_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @pack_nd_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
----------------
JerryShih wrote:

@banach-space 

case1:
```
linalg.pack %arg0 inner_dims_pos = [1, 2] inner_tiles = [32, 64] into %empty : tensor<3x32x64xf32> -> tensor<3x1x1x32x64xf32>
```
steps:
```
expand for packed dims:
3x32x64
=>
3x1x32x1x64

transpose to make inner tile to the last dims:
3x1x32x1x64
=>
3x1x1x32x64
since the outer dims are all 1, we could do nothing for transpose
```

case2:
```
linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
```
steps:
```
expand for packed dims:
4x4
=>
2x2x2x2

transpose to make inner tile to the last dims:
2_outer0*2_inner0*2_outer1*2_inner1*
=>
2_outer0*2_outer1*2_inner0*2_inner1*
We actually need transpose here.
The expand_shape op can't handle the transpose. So, this is the negative case.
```

https://github.com/llvm/llvm-project/pull/204971

From llvmlistbot at llvm.org  Wed Jul  1 18:53:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:53:10 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c486.70920dee.3514b2.e0b4@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 18:55:53 2026
From: llvmlistbot at llvm.org (Daniel Christian Mandolang)
Date: Wed, 01 Jul 2026 18:55:53 -0700 (PDT)
Subject: [Mlir-commits]  [mlir] [mlir][MemRef] Add guard,
	only pass for int or float (PR #206692)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206692@github.com>
Message-ID: <6a45c529.576b3af0.2cb9b7.77ef@mx.google.com>


danielcm585 wrote:

hi, could you pls review @dcaballe 
thanks

https://github.com/llvm/llvm-project/pull/206692

From llvmlistbot at llvm.org  Wed Jul  1 18:57:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:57:01 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c56d.eea6ab6e.1256b6.9288@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 18:57:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:57:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] "[mlir][linalg] Fix masking for rank-reducing
 contiguous extracts" (PR #206637)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206637@github.com>
Message-ID: <6a45c573.01f0a1da.2016e9.7041@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206637

From llvmlistbot at llvm.org  Wed Jul  1 18:57:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:57:54 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c5a2.ac469dc7.e3aa6.2e72@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 18:58:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:58:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c5cf.14d9b9dc.140829.1677@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 18:59:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 18:59:43 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c60f.92ea0e2a.21f97.e5e4@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 19:01:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 19:01:27 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a45c677.fdc477d6.118e5e.52cc@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Wed Jul  1 19:24:21 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Wed, 01 Jul 2026 19:24:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reimplement
 SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
 (PR #204971)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204971@github.com>
Message-ID: <6a45cbd5.6c8668f2.74a35.faf3@mx.google.com>


JerryShih wrote:

> > Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
> 
> Could you summarise the newly supported cases? Thank you!

@banach-space 
Do I misunderstand your sentence? Should I rewrite the commit message `title` to show the cases?
Maybe:
```
Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for no transposition/padding semantic pack/unpack op.
```

original commit msg:
```
Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.

If there is no transposition/padding semantic for pack/unpack,
we could try to use `mlir::getReassociationIndicesForReshape()`
to get the CollapseShapeOp/ExpandShapeOp form.
```

https://github.com/llvm/llvm-project/pull/204971

From llvmlistbot at llvm.org  Wed Jul  1 19:32:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 19:32:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
Message-ID: <llvm.org/llvm/llvm-project/pull/207111@github.com>


https://github.com/xys-syx created https://github.com/llvm/llvm-project/pull/207111

`LLVM_IntrOpBase` assumes a fixed 1:1 mapping between an `llvm::Intrinsic::ID` and an MLIR op: `emitIntrMLIRBuilders` builds its predicate from a single `llvmEnumName`, and `emitOneMLIRBuilder` relies on a fixed `llvmArgIndices` permutation to operands. This cannot express variants with different operand layouts that all collapse into one op, for example the four PTX `@llvm.nvvm.barrier.cta.sync.{all,count}{,.aligned}` intrinsics that all import into `nvvm.barrier`.

This is an alternative to #202862. Both address the same import gap:
- 202862 keeps the dispatch NVVM-local (no framework changes).
- This PR moves the dispatch into ODS (more declarative and reusable by future N:1 ops).

>From 4096416035a27206ad3f8ac2c5105aef084f7307 Mon Sep 17 00:00:00 2001
From: Yuansui Xu <xuyuansui at outlook.com>
Date: Wed, 1 Jul 2026 21:23:34 -0500
Subject: [PATCH] import llvm barrier intrinics

---
 .../include/mlir/Dialect/LLVMIR/LLVMOpBase.td | 14 ++++++
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td   | 21 ++++++++-
 .../Dialect/NVVM/LLVMIRToNVVMTranslation.cpp  | 29 ++++++++++++
 mlir/test/Target/LLVMIR/Import/nvvmir.ll      | 44 ++++++++++++++++++-
 .../tools/mlir-tblgen/LLVMIRConversionGen.cpp | 37 ++++++++++++++++
 5 files changed, 142 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
index a787840663171..7c9fe89535430 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
@@ -422,6 +422,20 @@ class LLVM_IntrOpBase<Dialect dialect, string opName, string enumName,
   }];
 }
 
+// Base class for an MLIR op that maps to multiple LLVM intrinsics on import.
+// The "enumNames" list contains the LLVM intrinsic IDs that import to this op,
+// and "converter" names the function that builds the op from the LLVM call.
+// Unlike `LLVM_IntrOpBase`, this class sets no "mlirBuilder"; the converter
+// handles import. Ops set their own "llvmBuilder" for export.
+class LLVM_NToOneIntrOpBase<Dialect dialect, string opName,
+                            list<string> enumNames, string converter,
+                            list<Trait> traits = [], int numResults = 0>
+    : LLVM_OpBase<dialect, opName, traits>,
+      Results<!if(!gt(numResults, 0), (outs LLVM_Type:$res), (outs))> {
+  list<string> llvmEnumNames = enumNames;
+  string llvmIRConverter = converter;
+}
+
 // Base class for LLVM intrinsic operations, should not be used directly. Places
 // the intrinsic into the LLVM dialect and prefixes its name with "intr.".
 class LLVM_IntrOp<string mnem, list<int> overloadedResults,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..3fb812f78c59a 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -1123,8 +1123,13 @@ def BarrierReductionAttr
   let assemblyFormat = "`<` $value `>`";
 }
 
-def NVVM_BarrierOp : NVVM_VoidIntrinsicOp<"barrier",
-    [AttrSizedOperandSegments]> {
+def NVVM_BarrierOp : LLVM_NToOneIntrOpBase<NVVM_Dialect, "barrier",
+    /*enumNames=*/["nvvm_barrier_cta_sync_all",
+                   "nvvm_barrier_cta_sync_count",
+                   "nvvm_barrier_cta_sync_aligned_all",
+                   "nvvm_barrier_cta_sync_aligned_count"],
+    /*converter=*/"convertBarrierSyncIntrinsic",
+    /*traits=*/[AttrSizedOperandSegments]> {
   let summary = "CTA Barrier Synchronization Op";
   let description = [{
     The `nvvm.barrier` operation performs barrier synchronization and communication 
@@ -1170,6 +1175,18 @@ def NVVM_BarrierOp : NVVM_VoidIntrinsicOp<"barrier",
   let builders = [OpBuilder<(ins), [{
       return build($_builder, $_state, Value{}, Value{});
     }]>];
+
+  let extraClassDeclaration = [{
+    static NVVM::IDArgPair
+      getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                            llvm::IRBuilderBase &builder);
+  }];
+
+  string llvmBuilder = [{
+    auto [id, args] = NVVM::BarrierOp::getIntrinsicIDAndArgs(
+                          *op, moduleTranslation, builder);
+    createIntrinsicCall(builder, id, builder.getVoidTy(), args);
+  }];
 }
 
 def NVVM_BarrierReductionOp :
diff --git a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
index 55e73e839afcb..82eafedd5b281 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
@@ -36,6 +36,35 @@ static ArrayRef<unsigned> getSupportedIntrinsicsImpl() {
   return convertibleIntrinsics;
 }
 
+/// Imports one of the four `bar.sync` LLVM intrinsic variants into a single
+/// `nvvm.barrier` op, deriving the `aligned` attribute and the optional
+/// `numberOfThreads` operand from the specific intrinsic ID.
+static LogicalResult
+convertBarrierSyncIntrinsic(OpBuilder &odsBuilder, llvm::CallInst *inst,
+                            LLVM::ModuleImport &moduleImport,
+                            ArrayRef<llvm::Value *> llvmOperands,
+                            ArrayRef<llvm::OperandBundleUse> llvmOpBundles) {
+  llvm::Intrinsic::ID id = inst->getIntrinsicID();
+  bool aligned = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_all ||
+                  id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+  bool hasCount = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_count ||
+                   id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+
+  SmallVector<Value> mlirOperands;
+  SmallVector<NamedAttribute> mlirAttrs;
+  if (failed(moduleImport.convertIntrinsicArguments(
+          llvmOperands, llvmOpBundles, /*requiresOpBundles=*/false, {}, {},
+          mlirOperands, mlirAttrs)))
+    return failure();
+
+  auto op = NVVM::BarrierOp::create(
+      odsBuilder, moduleImport.translateLoc(inst->getDebugLoc()),
+      mlirOperands.front(), hasCount ? mlirOperands.back() : Value{},
+      odsBuilder.getBoolAttr(aligned));
+  moduleImport.mapNoResultOp(inst, op);
+  return success();
+}
+
 /// Converts the LLVM intrinsic to an MLIR NVVM dialect operation if a
 /// conversion exits. Returns failure otherwise.
 static LogicalResult convertIntrinsicImpl(OpBuilder &odsBuilder,
diff --git a/mlir/test/Target/LLVMIR/Import/nvvmir.ll b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
index 1430f9a44eba1..b2529cc8fac6c 100644
--- a/mlir/test/Target/LLVMIR/Import/nvvmir.ll
+++ b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
@@ -73,11 +73,45 @@ define float @nvvm_rcp(float %0) {
 
 ; CHECK-LABEL: @llvm_nvvm_barrier0()
 define void @llvm_nvvm_barrier0() {
-  ; CHECK: llvm.nvvm.barrier.cta.sync.aligned.all
+  ; CHECK: %[[c0:.*]] = llvm.mlir.constant(0 : i32) : i32
+  ; CHECK: nvvm.barrier id = %[[c0]]
+  ; CHECK-NOT: aligned
   call void @llvm.nvvm.barrier0()
   ret void
 }
 
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_all
+define void @llvm_nvvm_barrier_sync_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}} {aligned = false}
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_all
+define void @llvm_nvvm_barrier_sync_aligned_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}}
+  ; CHECK-NOT: aligned
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.aligned.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_count
+define void @llvm_nvvm_barrier_sync_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}} {aligned = false}
+  call void @llvm.nvvm.barrier.cta.sync.count(i32 %bar, i32 %n)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_count
+define void @llvm_nvvm_barrier_sync_aligned_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}}
+  ; CHECK-NOT: aligned
+  call void @llvm.nvvm.barrier.cta.sync.aligned.count(i32 %bar, i32 %n)
+  ret void
+}
+
 ; CHECK-LABEL: @llvm_nvvm_bar_warp_sync
 define void @llvm_nvvm_bar_warp_sync(i32 %mask) {
   ; CHECK: nvvm.bar.warp.sync %{{.*}} : i32
@@ -276,6 +310,14 @@ declare float @llvm.nvvm.rcp.approx.ftz.f(float)
 
 declare void @llvm.nvvm.barrier0()
 
+declare void @llvm.nvvm.barrier.cta.sync.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.count(i32, i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.count(i32, i32)
+
 declare void @llvm.nvvm.bar.warp.sync(i32)
 
 declare i32 @llvm.nvvm.shfl.sync.bfly.i32(i32, i32, i32, i32)
diff --git a/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp b/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
index 11a2db4a1cc67..9a2d8bb1d07ca 100644
--- a/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
+++ b/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
@@ -303,6 +303,30 @@ static LogicalResult emitOneMLIRBuilder(const Record &record, raw_ostream &os,
   return success();
 }
 
+// Emit a dispatch block for an N-to-1 intrinsic op: an `if` matching any of
+// the listed intrinsic IDs that returns the named converter's result.
+static LogicalResult emitOneNToOneIntrMLIRBuilder(const Record &record,
+                                                  raw_ostream &os) {
+  std::vector<StringRef> enumNames =
+      record.getValueAsListOfStrings("llvmEnumNames");
+  StringRef converter = record.getValueAsString("llvmIRConverter");
+  if (enumNames.empty())
+    return emitError(record, "expected non-empty 'llvmEnumNames'");
+  if (converter.empty())
+    return emitError(record, "expected non-empty 'llvmIRConverter'");
+
+  os << "if (";
+  llvm::interleave(
+      enumNames, os,
+      [&](StringRef id) { os << "intrinsicID == llvm::Intrinsic::" << id; },
+      " ||\n    ");
+  os << ") {\n";
+  os << "  return " << converter
+     << "(odsBuilder, inst, moduleImport, llvmOperands, llvmOpBundles);\n";
+  os << "}\n";
+  return success();
+}
+
 // Emit all intrinsic MLIR builders. Returns false on success because of the
 // generator registration requirements.
 static bool emitIntrMLIRBuilders(const RecordKeeper &records, raw_ostream &os) {
@@ -316,6 +340,11 @@ static bool emitIntrMLIRBuilders(const RecordKeeper &records, raw_ostream &os) {
     if (failed(emitOneMLIRBuilder(*def, os, emitIntrCond)))
       return true;
   }
+  for (const Record *def :
+       records.getAllDerivedDefinitions("LLVM_NToOneIntrOpBase")) {
+    if (failed(emitOneNToOneIntrMLIRBuilder(*def, os)))
+      return true;
+  }
   return false;
 }
 
@@ -554,12 +583,20 @@ static void emitOneIntrinsic(const Record &record, raw_ostream &os) {
   os << "llvm::Intrinsic::" << record.getValueAsString("llvmEnumName") << ",\n";
 }
 
+static void emitOneNToOneIntrinsic(const Record &record, raw_ostream &os) {
+  for (StringRef id : record.getValueAsListOfStrings("llvmEnumNames"))
+    os << "llvm::Intrinsic::" << id << ",\n";
+}
+
 // Emit the list of LLVM IR intrinsics identifiers that are convertible to a
 // matching MLIR LLVM dialect intrinsic operation.
 static bool emitConvertibleIntrinsics(const RecordKeeper &records,
                                       raw_ostream &os) {
   for (const Record *def : records.getAllDerivedDefinitions("LLVM_IntrOpBase"))
     emitOneIntrinsic(*def, os);
+  for (const Record *def :
+       records.getAllDerivedDefinitions("LLVM_NToOneIntrOpBase"))
+    emitOneNToOneIntrinsic(*def, os);
 
   return false;
 }


From llvmlistbot at llvm.org  Wed Jul  1 19:32:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 19:32:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207111@github.com>
Message-ID: <6a45cdcb.f06f4994.1e3a60.670a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-core

Author: xys-syx

<details>
<summary>Changes</summary>

`LLVM_IntrOpBase` assumes a fixed 1:1 mapping between an `llvm::Intrinsic::ID` and an MLIR op: `emitIntrMLIRBuilders` builds its predicate from a single `llvmEnumName`, and `emitOneMLIRBuilder` relies on a fixed `llvmArgIndices` permutation to operands. This cannot express variants with different operand layouts that all collapse into one op, for example the four PTX `@<!-- -->llvm.nvvm.barrier.cta.sync.{all,count}{,.aligned}` intrinsics that all import into `nvvm.barrier`.

This is an alternative to #<!-- -->202862. Both address the same import gap:
- 202862 keeps the dispatch NVVM-local (no framework changes).
- This PR moves the dispatch into ODS (more declarative and reusable by future N:1 ops).

---
Full diff: https://github.com/llvm/llvm-project/pull/207111.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td (+14) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+19-2) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp (+29) 
- (modified) mlir/test/Target/LLVMIR/Import/nvvmir.ll (+43-1) 
- (modified) mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp (+37) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
index a787840663171..7c9fe89535430 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMOpBase.td
@@ -422,6 +422,20 @@ class LLVM_IntrOpBase<Dialect dialect, string opName, string enumName,
   }];
 }
 
+// Base class for an MLIR op that maps to multiple LLVM intrinsics on import.
+// The "enumNames" list contains the LLVM intrinsic IDs that import to this op,
+// and "converter" names the function that builds the op from the LLVM call.
+// Unlike `LLVM_IntrOpBase`, this class sets no "mlirBuilder"; the converter
+// handles import. Ops set their own "llvmBuilder" for export.
+class LLVM_NToOneIntrOpBase<Dialect dialect, string opName,
+                            list<string> enumNames, string converter,
+                            list<Trait> traits = [], int numResults = 0>
+    : LLVM_OpBase<dialect, opName, traits>,
+      Results<!if(!gt(numResults, 0), (outs LLVM_Type:$res), (outs))> {
+  list<string> llvmEnumNames = enumNames;
+  string llvmIRConverter = converter;
+}
+
 // Base class for LLVM intrinsic operations, should not be used directly. Places
 // the intrinsic into the LLVM dialect and prefixes its name with "intr.".
 class LLVM_IntrOp<string mnem, list<int> overloadedResults,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..3fb812f78c59a 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -1123,8 +1123,13 @@ def BarrierReductionAttr
   let assemblyFormat = "`<` $value `>`";
 }
 
-def NVVM_BarrierOp : NVVM_VoidIntrinsicOp<"barrier",
-    [AttrSizedOperandSegments]> {
+def NVVM_BarrierOp : LLVM_NToOneIntrOpBase<NVVM_Dialect, "barrier",
+    /*enumNames=*/["nvvm_barrier_cta_sync_all",
+                   "nvvm_barrier_cta_sync_count",
+                   "nvvm_barrier_cta_sync_aligned_all",
+                   "nvvm_barrier_cta_sync_aligned_count"],
+    /*converter=*/"convertBarrierSyncIntrinsic",
+    /*traits=*/[AttrSizedOperandSegments]> {
   let summary = "CTA Barrier Synchronization Op";
   let description = [{
     The `nvvm.barrier` operation performs barrier synchronization and communication 
@@ -1170,6 +1175,18 @@ def NVVM_BarrierOp : NVVM_VoidIntrinsicOp<"barrier",
   let builders = [OpBuilder<(ins), [{
       return build($_builder, $_state, Value{}, Value{});
     }]>];
+
+  let extraClassDeclaration = [{
+    static NVVM::IDArgPair
+      getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                            llvm::IRBuilderBase &builder);
+  }];
+
+  string llvmBuilder = [{
+    auto [id, args] = NVVM::BarrierOp::getIntrinsicIDAndArgs(
+                          *op, moduleTranslation, builder);
+    createIntrinsicCall(builder, id, builder.getVoidTy(), args);
+  }];
 }
 
 def NVVM_BarrierReductionOp :
diff --git a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
index 55e73e839afcb..82eafedd5b281 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
@@ -36,6 +36,35 @@ static ArrayRef<unsigned> getSupportedIntrinsicsImpl() {
   return convertibleIntrinsics;
 }
 
+/// Imports one of the four `bar.sync` LLVM intrinsic variants into a single
+/// `nvvm.barrier` op, deriving the `aligned` attribute and the optional
+/// `numberOfThreads` operand from the specific intrinsic ID.
+static LogicalResult
+convertBarrierSyncIntrinsic(OpBuilder &odsBuilder, llvm::CallInst *inst,
+                            LLVM::ModuleImport &moduleImport,
+                            ArrayRef<llvm::Value *> llvmOperands,
+                            ArrayRef<llvm::OperandBundleUse> llvmOpBundles) {
+  llvm::Intrinsic::ID id = inst->getIntrinsicID();
+  bool aligned = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_all ||
+                  id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+  bool hasCount = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_count ||
+                   id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+
+  SmallVector<Value> mlirOperands;
+  SmallVector<NamedAttribute> mlirAttrs;
+  if (failed(moduleImport.convertIntrinsicArguments(
+          llvmOperands, llvmOpBundles, /*requiresOpBundles=*/false, {}, {},
+          mlirOperands, mlirAttrs)))
+    return failure();
+
+  auto op = NVVM::BarrierOp::create(
+      odsBuilder, moduleImport.translateLoc(inst->getDebugLoc()),
+      mlirOperands.front(), hasCount ? mlirOperands.back() : Value{},
+      odsBuilder.getBoolAttr(aligned));
+  moduleImport.mapNoResultOp(inst, op);
+  return success();
+}
+
 /// Converts the LLVM intrinsic to an MLIR NVVM dialect operation if a
 /// conversion exits. Returns failure otherwise.
 static LogicalResult convertIntrinsicImpl(OpBuilder &odsBuilder,
diff --git a/mlir/test/Target/LLVMIR/Import/nvvmir.ll b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
index 1430f9a44eba1..b2529cc8fac6c 100644
--- a/mlir/test/Target/LLVMIR/Import/nvvmir.ll
+++ b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
@@ -73,11 +73,45 @@ define float @nvvm_rcp(float %0) {
 
 ; CHECK-LABEL: @llvm_nvvm_barrier0()
 define void @llvm_nvvm_barrier0() {
-  ; CHECK: llvm.nvvm.barrier.cta.sync.aligned.all
+  ; CHECK: %[[c0:.*]] = llvm.mlir.constant(0 : i32) : i32
+  ; CHECK: nvvm.barrier id = %[[c0]]
+  ; CHECK-NOT: aligned
   call void @llvm.nvvm.barrier0()
   ret void
 }
 
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_all
+define void @llvm_nvvm_barrier_sync_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}} {aligned = false}
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_all
+define void @llvm_nvvm_barrier_sync_aligned_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}}
+  ; CHECK-NOT: aligned
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.aligned.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_count
+define void @llvm_nvvm_barrier_sync_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}} {aligned = false}
+  call void @llvm.nvvm.barrier.cta.sync.count(i32 %bar, i32 %n)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_count
+define void @llvm_nvvm_barrier_sync_aligned_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}}
+  ; CHECK-NOT: aligned
+  call void @llvm.nvvm.barrier.cta.sync.aligned.count(i32 %bar, i32 %n)
+  ret void
+}
+
 ; CHECK-LABEL: @llvm_nvvm_bar_warp_sync
 define void @llvm_nvvm_bar_warp_sync(i32 %mask) {
   ; CHECK: nvvm.bar.warp.sync %{{.*}} : i32
@@ -276,6 +310,14 @@ declare float @llvm.nvvm.rcp.approx.ftz.f(float)
 
 declare void @llvm.nvvm.barrier0()
 
+declare void @llvm.nvvm.barrier.cta.sync.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.count(i32, i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.count(i32, i32)
+
 declare void @llvm.nvvm.bar.warp.sync(i32)
 
 declare i32 @llvm.nvvm.shfl.sync.bfly.i32(i32, i32, i32, i32)
diff --git a/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp b/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
index 11a2db4a1cc67..9a2d8bb1d07ca 100644
--- a/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
+++ b/mlir/tools/mlir-tblgen/LLVMIRConversionGen.cpp
@@ -303,6 +303,30 @@ static LogicalResult emitOneMLIRBuilder(const Record &record, raw_ostream &os,
   return success();
 }
 
+// Emit a dispatch block for an N-to-1 intrinsic op: an `if` matching any of
+// the listed intrinsic IDs that returns the named converter's result.
+static LogicalResult emitOneNToOneIntrMLIRBuilder(const Record &record,
+                                                  raw_ostream &os) {
+  std::vector<StringRef> enumNames =
+      record.getValueAsListOfStrings("llvmEnumNames");
+  StringRef converter = record.getValueAsString("llvmIRConverter");
+  if (enumNames.empty())
+    return emitError(record, "expected non-empty 'llvmEnumNames'");
+  if (converter.empty())
+    return emitError(record, "expected non-empty 'llvmIRConverter'");
+
+  os << "if (";
+  llvm::interleave(
+      enumNames, os,
+      [&](StringRef id) { os << "intrinsicID == llvm::Intrinsic::" << id; },
+      " ||\n    ");
+  os << ") {\n";
+  os << "  return " << converter
+     << "(odsBuilder, inst, moduleImport, llvmOperands, llvmOpBundles);\n";
+  os << "}\n";
+  return success();
+}
+
 // Emit all intrinsic MLIR builders. Returns false on success because of the
 // generator registration requirements.
 static bool emitIntrMLIRBuilders(const RecordKeeper &records, raw_ostream &os) {
@@ -316,6 +340,11 @@ static bool emitIntrMLIRBuilders(const RecordKeeper &records, raw_ostream &os) {
     if (failed(emitOneMLIRBuilder(*def, os, emitIntrCond)))
       return true;
   }
+  for (const Record *def :
+       records.getAllDerivedDefinitions("LLVM_NToOneIntrOpBase")) {
+    if (failed(emitOneNToOneIntrMLIRBuilder(*def, os)))
+      return true;
+  }
   return false;
 }
 
@@ -554,12 +583,20 @@ static void emitOneIntrinsic(const Record &record, raw_ostream &os) {
   os << "llvm::Intrinsic::" << record.getValueAsString("llvmEnumName") << ",\n";
 }
 
+static void emitOneNToOneIntrinsic(const Record &record, raw_ostream &os) {
+  for (StringRef id : record.getValueAsListOfStrings("llvmEnumNames"))
+    os << "llvm::Intrinsic::" << id << ",\n";
+}
+
 // Emit the list of LLVM IR intrinsics identifiers that are convertible to a
 // matching MLIR LLVM dialect intrinsic operation.
 static bool emitConvertibleIntrinsics(const RecordKeeper &records,
                                       raw_ostream &os) {
   for (const Record *def : records.getAllDerivedDefinitions("LLVM_IntrOpBase"))
     emitOneIntrinsic(*def, os);
+  for (const Record *def :
+       records.getAllDerivedDefinitions("LLVM_NToOneIntrOpBase"))
+    emitOneNToOneIntrinsic(*def, os);
 
   return false;
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207111

From llvmlistbot at llvm.org  Wed Jul  1 20:45:51 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Wed, 01 Jul 2026 20:45:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Add missing dependent dialect
 to TestVectorUnrollingPatterns (PR #207113)
Message-ID: <llvm.org/llvm/llvm-project/pull/207113@github.com>


https://github.com/CoTinker created https://github.com/llvm/llvm-project/pull/207113

Add `vector::VectorDialect` to `getDependentDialects()` for `TestVectorUnrollingPatterns`. Previously, `arith.addf` test passed only because the vector dialect was indirectly loaded via other vector tests in the same `MLIRContext`. With `split-input-file`, each test runs in an isolated context, causing failures due to the missing vector dialect. Fixes #206921.

>From bd1c3b047568364576eb8cc36b5d61726bd4a772 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Thu, 2 Jul 2026 11:40:01 +0800
Subject: [PATCH 1/2] [mlir][vector] Add missing dependent dialect to
 TestVectorUnrollingPatterns

Add `vector::VectorDialect` to `getDependentDialects()` for
`TestVectorUnrollingPatterns`. Previously, `arith.addf` test passed only because the vector dialect was indirectly loaded via other vector tests in the same `MLIRContext`. With `split-input-file`, each test runs in an isolated context, causing failures due to the missing vector dialect.
---
 mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..1a4bef664fbe1 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -150,6 +150,11 @@ struct TestVectorUnrollingPatterns
     return "Test lowering patterns to unroll contract ops in the vector "
            "dialect";
   }
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<vector::VectorDialect>();
+  }
+
   TestVectorUnrollingPatterns() = default;
   TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass)
       : PassWrapper(pass) {}

>From caa23c28c8c3e9a9122e8b501b9eca1925be17b7 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Thu, 2 Jul 2026 11:44:27 +0800
Subject: [PATCH 2/2] split file

---
 .../Dialect/Vector/vector-unroll-options.mlir | 43 +++++++++++++++++--
 1 file changed, 39 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..0ec71620c5324 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -1,6 +1,6 @@
-// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type | FileCheck %s
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1"  | FileCheck %s --check-prefix=ORDER
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" | FileCheck %s --check-prefix=BATCHED
+// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1" -split-input-file | FileCheck %s --check-prefix=ORDER
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" -split-input-file | FileCheck %s --check-prefix=BATCHED
 
 func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
                           %init : vector<8x8xf32>) -> vector<8x8xf32> {
@@ -150,7 +150,7 @@ func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
 
 //       ORDER:   return
 
-
+// -----
 
 func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
                           %init : vector<8x8xf16>) -> vector<8x8xf16> {
@@ -181,6 +181,8 @@ func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
 //  CHECK-SAME:     vector<4x4xf16>, vector<4x4xf16> into vector<4x4xf16>
 //       CHECK:   return
 
+// -----
+
 func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf32>) -> vector<4x4xf32> {
   %0 = vector.fma %a, %b, %c: vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -188,6 +190,8 @@ func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf
 //   CHECK-LABEL: func @vector_fma
 // CHECK-COUNT-4: vector.fma %{{.+}}, %{{.+}}, %{{.+}} : vector<2x2xf32>
 
+// -----
+
 func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
   %0 = vector.fma %a, %a, %a : vector<3x2x2xf32>
   return %0 : vector<3x2x2xf32>
@@ -221,6 +225,8 @@ func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
 //       CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[FMA2]], %[[I1]] {offsets = [2, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<3x2x2xf32>
 //       CHECK:   return %[[I2]] : vector<3x2x2xf32>
 
+// -----
+
 func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) -> vector<4xf32> {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [1] : vector<4x6xf32> to vector<4xf32>
   return %0 : vector<4xf32>
@@ -245,6 +251,8 @@ func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) ->
 //       CHECK:   %[[V2:.*]] = vector.insert_strided_slice %[[R5]], %[[V1]] {offsets = [2], strides = [1]} : vector<2xf32> into vector<4xf32>
 //       CHECK:   return %[[V2]] : vector<4xf32>
 
+// -----
+
 func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32 {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [0, 1] : vector<4x2xf32> to f32
   return %0 : f32
@@ -257,6 +265,8 @@ func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32
 //       CHECK:   %[[R1:.*]] = vector.multi_reduction <add>, %[[S1]], %[[R0]] [0, 1] : vector<2x2xf32> to f32
 //       CHECK:   return %[[R1]] : f32
 
+// -----
+
 func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
   %0 = vector.reduction <add>, %v : vector<8xf32> into f32
   return %0 : f32
@@ -276,6 +286,8 @@ func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
 //       CHECK:   %[[add3:.*]] = arith.addf %[[add2]], %[[r3]]
 //       CHECK:   return %[[add3]]
 
+// -----
+
 func.func @vector_transpose(%v : vector<2x4x3x8xf32>) -> vector<2x3x8x4xf32> {
   %t = vector.transpose %v, [0, 2, 3, 1] : vector<2x4x3x8xf32> to vector<2x3x8x4xf32>
   return %t : vector<2x3x8x4xf32>
@@ -337,6 +349,7 @@ func.func @vector_contract_batched(%lhs: vector<8x8x4xf32>, %rhs: vector<8x8x4xf
 //      BATCHED-NOT: vector.contract
 //          BATCHED: return
 
+// -----
 
 func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4xf32> to vector<4x4xf32>
@@ -360,6 +373,8 @@ func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<1x4xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -382,6 +397,8 @@ func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4x1xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -404,6 +421,7 @@ func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
 
 func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   %c0 = arith.constant 0 : index
@@ -426,6 +444,7 @@ func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   // CHECK: %[[V7:.*]] = vector.insert_strided_slice %[[V6]], %[[V5]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf16> into vector<4x4xf16>
   // CHECK: return %[[V7]] : vector<4x4xf16>
 
+// -----
 
 func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   %c0 = arith.constant 0 : index
@@ -446,6 +465,7 @@ func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   // CHECK: %[[V3:.*]] = vector.extract_strided_slice %[[ARG1]] {offsets = [2, 2], sizes = [2, 2], strides = [1, 1]} : vector<4x4xf16> to vector<2x2xf16>
   // CHECK: vector.store %[[V3]], %[[ARG0]][%[[C2]], %[[C2]]] : memref<4x4xf16>, vector<2x2xf16>
 
+// -----
 
 func.func @vector_step() -> vector<32xindex> {
     %0 = vector.step : vector<32xindex>
@@ -466,6 +486,7 @@ func.func @vector_step() -> vector<32xindex> {
 // CHECK: %[[INS3:.*]] = vector.insert_strided_slice %[[ADD3]], %[[INS2]] {offsets = [24], strides = [1]} : vector<8xindex> into vector<32xindex>
 // CHECK: return %[[INS3]] : vector<32xindex>
 
+// -----
 
 func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>) -> vector<2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2xf32>
@@ -488,6 +509,7 @@ func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>)
 //       CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[ADD1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<2x2x2xf32>
 //       CHECK:   return %[[I1]] : vector<2x2x2xf32>
 
+// -----
 
 func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf32>) -> vector<2x2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2x2xf32>
@@ -499,6 +521,8 @@ func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf3
 // CHECK-NOT: arith.addf
 // CHECK: return
 
+// -----
+
 func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1> {
   %0 = vector.create_mask %size1, %size2 : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -539,6 +563,8 @@ func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1>
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
   %cst16 = arith.constant 16 : index
   %0 = vector.create_mask %cst16, %cst16 : vector<16x16xi1>
@@ -554,6 +580,8 @@ func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
 // CHECK:   %[[S3:.*]] = vector.insert_strided_slice %[[CST_0]], %[[S2]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 // CHECK:   return %[[S3]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_constant_mask() -> vector<16x16xi1> {
   %0 = vector.constant_mask [12, 10] : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -572,6 +600,8 @@ func.func @vector_constant_mask() -> vector<16x16xi1> {
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
   %0 = vector.shape_cast %v : vector<16xf32> to vector<2x2x4xf32>
   return %0 : vector<2x2x4xf32>
@@ -588,6 +618,7 @@ func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x4xf32> into vector<2x2x4xf32>
 // CHECK:   return %[[I1]] : vector<2x2x4xf32>
 
+// -----
 
 func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
   %0 = vector.shape_cast %v : vector<8x2xf32> to vector<4x4xf32>
@@ -605,6 +636,7 @@ func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [2, 0], strides = [1, 1]} : vector<2x4xf32> into vector<4x4xf32>
 // CHECK:   return %[[I1]] : vector<4x4xf32>
 
+// -----
 
 // This is a negative test case to ensure that such shape casts are not unrolled
 // because the targetShape (2x4) is not contiguous in result vector
@@ -618,6 +650,7 @@ func.func @negative_shape_cast_target_shape_not_contiguous(%v: vector<64xf32>) -
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<64xf32> to vector<8x8xf32>
 // CHECK:   return %[[SC]] : vector<8x8xf32>
 
+// -----
 
 // This is negative test case to ensure that such shape casts are not unrolled
 // because it cannot determine the extractShape from source vector (8x3)
@@ -632,6 +665,7 @@ func.func @negative_shape_cast_source_shape_not_determinable(%v: vector<8x3xf32>
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<8x3xf32> to vector<6x4xf32>
 // CHECK:   return %[[SC]] : vector<6x4xf32>
 
+// -----
 
 // TargetShape is [1x16]
 func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
@@ -650,6 +684,7 @@ func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 16], strides = [1, 1]} : vector<1x16xf32> into vector<1x32xf32>
 // CHECK:   return %[[I1]] : vector<1x32xf32>
 
+// -----
 
 // TargetShape is [1x1]
 func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x1xf32> {


From llvmlistbot at llvm.org  Wed Jul  1 20:46:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 20:46:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Add missing dependent dialect
 to TestVectorUnrollingPatterns (PR #207113)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207113@github.com>
Message-ID: <6a45df15.f14cf1a5.2c3aaa.b014@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-vector

Author: Longsheng Mou (CoTinker)

<details>
<summary>Changes</summary>

Add `vector::VectorDialect` to `getDependentDialects()` for `TestVectorUnrollingPatterns`. Previously, `arith.addf` test passed only because the vector dialect was indirectly loaded via other vector tests in the same `MLIRContext`. With `split-input-file`, each test runs in an isolated context, causing failures due to the missing vector dialect. Fixes #<!-- -->206921.

---
Full diff: https://github.com/llvm/llvm-project/pull/207113.diff


2 Files Affected:

- (modified) mlir/test/Dialect/Vector/vector-unroll-options.mlir (+39-4) 
- (modified) mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp (+5) 


``````````diff
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..0ec71620c5324 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -1,6 +1,6 @@
-// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type | FileCheck %s
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1"  | FileCheck %s --check-prefix=ORDER
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" | FileCheck %s --check-prefix=BATCHED
+// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1" -split-input-file | FileCheck %s --check-prefix=ORDER
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" -split-input-file | FileCheck %s --check-prefix=BATCHED
 
 func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
                           %init : vector<8x8xf32>) -> vector<8x8xf32> {
@@ -150,7 +150,7 @@ func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
 
 //       ORDER:   return
 
-
+// -----
 
 func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
                           %init : vector<8x8xf16>) -> vector<8x8xf16> {
@@ -181,6 +181,8 @@ func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
 //  CHECK-SAME:     vector<4x4xf16>, vector<4x4xf16> into vector<4x4xf16>
 //       CHECK:   return
 
+// -----
+
 func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf32>) -> vector<4x4xf32> {
   %0 = vector.fma %a, %b, %c: vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -188,6 +190,8 @@ func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf
 //   CHECK-LABEL: func @vector_fma
 // CHECK-COUNT-4: vector.fma %{{.+}}, %{{.+}}, %{{.+}} : vector<2x2xf32>
 
+// -----
+
 func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
   %0 = vector.fma %a, %a, %a : vector<3x2x2xf32>
   return %0 : vector<3x2x2xf32>
@@ -221,6 +225,8 @@ func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
 //       CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[FMA2]], %[[I1]] {offsets = [2, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<3x2x2xf32>
 //       CHECK:   return %[[I2]] : vector<3x2x2xf32>
 
+// -----
+
 func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) -> vector<4xf32> {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [1] : vector<4x6xf32> to vector<4xf32>
   return %0 : vector<4xf32>
@@ -245,6 +251,8 @@ func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) ->
 //       CHECK:   %[[V2:.*]] = vector.insert_strided_slice %[[R5]], %[[V1]] {offsets = [2], strides = [1]} : vector<2xf32> into vector<4xf32>
 //       CHECK:   return %[[V2]] : vector<4xf32>
 
+// -----
+
 func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32 {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [0, 1] : vector<4x2xf32> to f32
   return %0 : f32
@@ -257,6 +265,8 @@ func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32
 //       CHECK:   %[[R1:.*]] = vector.multi_reduction <add>, %[[S1]], %[[R0]] [0, 1] : vector<2x2xf32> to f32
 //       CHECK:   return %[[R1]] : f32
 
+// -----
+
 func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
   %0 = vector.reduction <add>, %v : vector<8xf32> into f32
   return %0 : f32
@@ -276,6 +286,8 @@ func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
 //       CHECK:   %[[add3:.*]] = arith.addf %[[add2]], %[[r3]]
 //       CHECK:   return %[[add3]]
 
+// -----
+
 func.func @vector_transpose(%v : vector<2x4x3x8xf32>) -> vector<2x3x8x4xf32> {
   %t = vector.transpose %v, [0, 2, 3, 1] : vector<2x4x3x8xf32> to vector<2x3x8x4xf32>
   return %t : vector<2x3x8x4xf32>
@@ -337,6 +349,7 @@ func.func @vector_contract_batched(%lhs: vector<8x8x4xf32>, %rhs: vector<8x8x4xf
 //      BATCHED-NOT: vector.contract
 //          BATCHED: return
 
+// -----
 
 func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4xf32> to vector<4x4xf32>
@@ -360,6 +373,8 @@ func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<1x4xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -382,6 +397,8 @@ func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4x1xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -404,6 +421,7 @@ func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
 
 func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   %c0 = arith.constant 0 : index
@@ -426,6 +444,7 @@ func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   // CHECK: %[[V7:.*]] = vector.insert_strided_slice %[[V6]], %[[V5]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf16> into vector<4x4xf16>
   // CHECK: return %[[V7]] : vector<4x4xf16>
 
+// -----
 
 func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   %c0 = arith.constant 0 : index
@@ -446,6 +465,7 @@ func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   // CHECK: %[[V3:.*]] = vector.extract_strided_slice %[[ARG1]] {offsets = [2, 2], sizes = [2, 2], strides = [1, 1]} : vector<4x4xf16> to vector<2x2xf16>
   // CHECK: vector.store %[[V3]], %[[ARG0]][%[[C2]], %[[C2]]] : memref<4x4xf16>, vector<2x2xf16>
 
+// -----
 
 func.func @vector_step() -> vector<32xindex> {
     %0 = vector.step : vector<32xindex>
@@ -466,6 +486,7 @@ func.func @vector_step() -> vector<32xindex> {
 // CHECK: %[[INS3:.*]] = vector.insert_strided_slice %[[ADD3]], %[[INS2]] {offsets = [24], strides = [1]} : vector<8xindex> into vector<32xindex>
 // CHECK: return %[[INS3]] : vector<32xindex>
 
+// -----
 
 func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>) -> vector<2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2xf32>
@@ -488,6 +509,7 @@ func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>)
 //       CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[ADD1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<2x2x2xf32>
 //       CHECK:   return %[[I1]] : vector<2x2x2xf32>
 
+// -----
 
 func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf32>) -> vector<2x2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2x2xf32>
@@ -499,6 +521,8 @@ func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf3
 // CHECK-NOT: arith.addf
 // CHECK: return
 
+// -----
+
 func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1> {
   %0 = vector.create_mask %size1, %size2 : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -539,6 +563,8 @@ func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1>
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
   %cst16 = arith.constant 16 : index
   %0 = vector.create_mask %cst16, %cst16 : vector<16x16xi1>
@@ -554,6 +580,8 @@ func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
 // CHECK:   %[[S3:.*]] = vector.insert_strided_slice %[[CST_0]], %[[S2]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 // CHECK:   return %[[S3]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_constant_mask() -> vector<16x16xi1> {
   %0 = vector.constant_mask [12, 10] : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -572,6 +600,8 @@ func.func @vector_constant_mask() -> vector<16x16xi1> {
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
   %0 = vector.shape_cast %v : vector<16xf32> to vector<2x2x4xf32>
   return %0 : vector<2x2x4xf32>
@@ -588,6 +618,7 @@ func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x4xf32> into vector<2x2x4xf32>
 // CHECK:   return %[[I1]] : vector<2x2x4xf32>
 
+// -----
 
 func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
   %0 = vector.shape_cast %v : vector<8x2xf32> to vector<4x4xf32>
@@ -605,6 +636,7 @@ func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [2, 0], strides = [1, 1]} : vector<2x4xf32> into vector<4x4xf32>
 // CHECK:   return %[[I1]] : vector<4x4xf32>
 
+// -----
 
 // This is a negative test case to ensure that such shape casts are not unrolled
 // because the targetShape (2x4) is not contiguous in result vector
@@ -618,6 +650,7 @@ func.func @negative_shape_cast_target_shape_not_contiguous(%v: vector<64xf32>) -
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<64xf32> to vector<8x8xf32>
 // CHECK:   return %[[SC]] : vector<8x8xf32>
 
+// -----
 
 // This is negative test case to ensure that such shape casts are not unrolled
 // because it cannot determine the extractShape from source vector (8x3)
@@ -632,6 +665,7 @@ func.func @negative_shape_cast_source_shape_not_determinable(%v: vector<8x3xf32>
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<8x3xf32> to vector<6x4xf32>
 // CHECK:   return %[[SC]] : vector<6x4xf32>
 
+// -----
 
 // TargetShape is [1x16]
 func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
@@ -650,6 +684,7 @@ func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 16], strides = [1, 1]} : vector<1x16xf32> into vector<1x32xf32>
 // CHECK:   return %[[I1]] : vector<1x32xf32>
 
+// -----
 
 // TargetShape is [1x1]
 func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x1xf32> {
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..1a4bef664fbe1 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -150,6 +150,11 @@ struct TestVectorUnrollingPatterns
     return "Test lowering patterns to unroll contract ops in the vector "
            "dialect";
   }
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<vector::VectorDialect>();
+  }
+
   TestVectorUnrollingPatterns() = default;
   TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass)
       : PassWrapper(pass) {}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207113

From llvmlistbot at llvm.org  Wed Jul  1 20:46:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 01 Jul 2026 20:46:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Add missing dependent dialect
 to TestVectorUnrollingPatterns (PR #207113)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207113@github.com>
Message-ID: <6a45df15.92ea0e2a.21f97.28e5@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Longsheng Mou (CoTinker)

<details>
<summary>Changes</summary>

Add `vector::VectorDialect` to `getDependentDialects()` for `TestVectorUnrollingPatterns`. Previously, `arith.addf` test passed only because the vector dialect was indirectly loaded via other vector tests in the same `MLIRContext`. With `split-input-file`, each test runs in an isolated context, causing failures due to the missing vector dialect. Fixes #<!-- -->206921.

---
Full diff: https://github.com/llvm/llvm-project/pull/207113.diff


2 Files Affected:

- (modified) mlir/test/Dialect/Vector/vector-unroll-options.mlir (+39-4) 
- (modified) mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp (+5) 


``````````diff
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..0ec71620c5324 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -1,6 +1,6 @@
-// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type | FileCheck %s
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1"  | FileCheck %s --check-prefix=ORDER
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" | FileCheck %s --check-prefix=BATCHED
+// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1" -split-input-file | FileCheck %s --check-prefix=ORDER
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" -split-input-file | FileCheck %s --check-prefix=BATCHED
 
 func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
                           %init : vector<8x8xf32>) -> vector<8x8xf32> {
@@ -150,7 +150,7 @@ func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
 
 //       ORDER:   return
 
-
+// -----
 
 func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
                           %init : vector<8x8xf16>) -> vector<8x8xf16> {
@@ -181,6 +181,8 @@ func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
 //  CHECK-SAME:     vector<4x4xf16>, vector<4x4xf16> into vector<4x4xf16>
 //       CHECK:   return
 
+// -----
+
 func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf32>) -> vector<4x4xf32> {
   %0 = vector.fma %a, %b, %c: vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -188,6 +190,8 @@ func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf
 //   CHECK-LABEL: func @vector_fma
 // CHECK-COUNT-4: vector.fma %{{.+}}, %{{.+}}, %{{.+}} : vector<2x2xf32>
 
+// -----
+
 func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
   %0 = vector.fma %a, %a, %a : vector<3x2x2xf32>
   return %0 : vector<3x2x2xf32>
@@ -221,6 +225,8 @@ func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
 //       CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[FMA2]], %[[I1]] {offsets = [2, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<3x2x2xf32>
 //       CHECK:   return %[[I2]] : vector<3x2x2xf32>
 
+// -----
+
 func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) -> vector<4xf32> {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [1] : vector<4x6xf32> to vector<4xf32>
   return %0 : vector<4xf32>
@@ -245,6 +251,8 @@ func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) ->
 //       CHECK:   %[[V2:.*]] = vector.insert_strided_slice %[[R5]], %[[V1]] {offsets = [2], strides = [1]} : vector<2xf32> into vector<4xf32>
 //       CHECK:   return %[[V2]] : vector<4xf32>
 
+// -----
+
 func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32 {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [0, 1] : vector<4x2xf32> to f32
   return %0 : f32
@@ -257,6 +265,8 @@ func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32
 //       CHECK:   %[[R1:.*]] = vector.multi_reduction <add>, %[[S1]], %[[R0]] [0, 1] : vector<2x2xf32> to f32
 //       CHECK:   return %[[R1]] : f32
 
+// -----
+
 func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
   %0 = vector.reduction <add>, %v : vector<8xf32> into f32
   return %0 : f32
@@ -276,6 +286,8 @@ func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
 //       CHECK:   %[[add3:.*]] = arith.addf %[[add2]], %[[r3]]
 //       CHECK:   return %[[add3]]
 
+// -----
+
 func.func @vector_transpose(%v : vector<2x4x3x8xf32>) -> vector<2x3x8x4xf32> {
   %t = vector.transpose %v, [0, 2, 3, 1] : vector<2x4x3x8xf32> to vector<2x3x8x4xf32>
   return %t : vector<2x3x8x4xf32>
@@ -337,6 +349,7 @@ func.func @vector_contract_batched(%lhs: vector<8x8x4xf32>, %rhs: vector<8x8x4xf
 //      BATCHED-NOT: vector.contract
 //          BATCHED: return
 
+// -----
 
 func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4xf32> to vector<4x4xf32>
@@ -360,6 +373,8 @@ func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<1x4xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -382,6 +397,8 @@ func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4x1xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -404,6 +421,7 @@ func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
 
 func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   %c0 = arith.constant 0 : index
@@ -426,6 +444,7 @@ func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   // CHECK: %[[V7:.*]] = vector.insert_strided_slice %[[V6]], %[[V5]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf16> into vector<4x4xf16>
   // CHECK: return %[[V7]] : vector<4x4xf16>
 
+// -----
 
 func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   %c0 = arith.constant 0 : index
@@ -446,6 +465,7 @@ func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   // CHECK: %[[V3:.*]] = vector.extract_strided_slice %[[ARG1]] {offsets = [2, 2], sizes = [2, 2], strides = [1, 1]} : vector<4x4xf16> to vector<2x2xf16>
   // CHECK: vector.store %[[V3]], %[[ARG0]][%[[C2]], %[[C2]]] : memref<4x4xf16>, vector<2x2xf16>
 
+// -----
 
 func.func @vector_step() -> vector<32xindex> {
     %0 = vector.step : vector<32xindex>
@@ -466,6 +486,7 @@ func.func @vector_step() -> vector<32xindex> {
 // CHECK: %[[INS3:.*]] = vector.insert_strided_slice %[[ADD3]], %[[INS2]] {offsets = [24], strides = [1]} : vector<8xindex> into vector<32xindex>
 // CHECK: return %[[INS3]] : vector<32xindex>
 
+// -----
 
 func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>) -> vector<2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2xf32>
@@ -488,6 +509,7 @@ func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>)
 //       CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[ADD1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<2x2x2xf32>
 //       CHECK:   return %[[I1]] : vector<2x2x2xf32>
 
+// -----
 
 func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf32>) -> vector<2x2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2x2xf32>
@@ -499,6 +521,8 @@ func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf3
 // CHECK-NOT: arith.addf
 // CHECK: return
 
+// -----
+
 func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1> {
   %0 = vector.create_mask %size1, %size2 : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -539,6 +563,8 @@ func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1>
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
   %cst16 = arith.constant 16 : index
   %0 = vector.create_mask %cst16, %cst16 : vector<16x16xi1>
@@ -554,6 +580,8 @@ func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
 // CHECK:   %[[S3:.*]] = vector.insert_strided_slice %[[CST_0]], %[[S2]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 // CHECK:   return %[[S3]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_constant_mask() -> vector<16x16xi1> {
   %0 = vector.constant_mask [12, 10] : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -572,6 +600,8 @@ func.func @vector_constant_mask() -> vector<16x16xi1> {
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
   %0 = vector.shape_cast %v : vector<16xf32> to vector<2x2x4xf32>
   return %0 : vector<2x2x4xf32>
@@ -588,6 +618,7 @@ func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x4xf32> into vector<2x2x4xf32>
 // CHECK:   return %[[I1]] : vector<2x2x4xf32>
 
+// -----
 
 func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
   %0 = vector.shape_cast %v : vector<8x2xf32> to vector<4x4xf32>
@@ -605,6 +636,7 @@ func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [2, 0], strides = [1, 1]} : vector<2x4xf32> into vector<4x4xf32>
 // CHECK:   return %[[I1]] : vector<4x4xf32>
 
+// -----
 
 // This is a negative test case to ensure that such shape casts are not unrolled
 // because the targetShape (2x4) is not contiguous in result vector
@@ -618,6 +650,7 @@ func.func @negative_shape_cast_target_shape_not_contiguous(%v: vector<64xf32>) -
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<64xf32> to vector<8x8xf32>
 // CHECK:   return %[[SC]] : vector<8x8xf32>
 
+// -----
 
 // This is negative test case to ensure that such shape casts are not unrolled
 // because it cannot determine the extractShape from source vector (8x3)
@@ -632,6 +665,7 @@ func.func @negative_shape_cast_source_shape_not_determinable(%v: vector<8x3xf32>
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<8x3xf32> to vector<6x4xf32>
 // CHECK:   return %[[SC]] : vector<6x4xf32>
 
+// -----
 
 // TargetShape is [1x16]
 func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
@@ -650,6 +684,7 @@ func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 16], strides = [1, 1]} : vector<1x16xf32> into vector<1x32xf32>
 // CHECK:   return %[[I1]] : vector<1x32xf32>
 
+// -----
 
 // TargetShape is [1x1]
 func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x1xf32> {
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..1a4bef664fbe1 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -150,6 +150,11 @@ struct TestVectorUnrollingPatterns
     return "Test lowering patterns to unroll contract ops in the vector "
            "dialect";
   }
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<vector::VectorDialect>();
+  }
+
   TestVectorUnrollingPatterns() = default;
   TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass)
       : PassWrapper(pass) {}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207113

From llvmlistbot at llvm.org  Wed Jul  1 21:27:42 2026
From: llvmlistbot at llvm.org (Sebastian Pop)
Date: Wed, 01 Jul 2026 21:27:42 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang] Emit llvm.assume for array
 bounds constraints (PR #178811)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/178811@github.com>
Message-ID: <6a45e8be.a151105c.30c1b9.0f6a@mx.google.com>


https://github.com/sebpop updated https://github.com/llvm/llvm-project/pull/178811

>From 5eabf504e20c205dc91376d1c9e7ded04ccabc03 Mon Sep 17 00:00:00 2001
From: Sebastian Pop <spop at nvidia.com>
Date: Wed, 28 Jan 2026 10:03:01 -0600
Subject: [PATCH] [flang] Emit llvm.assume for array bounds constraints

Emit llvm.assume intrinsics for array bounds checking during the FIR to
LLVM conversion. This implements array bounds constraints per Fortran
2018 standard section 9.5.3.3.2: "The value of each subscript expression
shall be within the bounds for its dimension unless the array section
has size zero."

For non-boxed arrays with known shape, the conversion emits two assumes
per dimension:
- assume(index >= lower_bound)
- assume(index <= lower_bound + extent - 1)

This enables LLVM loop nest optimizations to take advantage of the
guaranteed bounds information.
---
 .../include/flang/Optimizer/CodeGen/CodeGen.h |  6 ++
 .../flang/Optimizer/Passes/CommandLineOpts.h  |  5 ++
 flang/lib/Optimizer/CodeGen/CodeGen.cpp       | 44 +++++++++++
 .../lib/Optimizer/Passes/CommandLineOpts.cpp  |  6 ++
 flang/lib/Optimizer/Passes/Pipelines.cpp      |  5 ++
 flang/test/Fir/array-bounds-assume.fir        | 76 +++++++++++++++++++
 .../mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td   |  8 ++
 .../LLVMIR/LLVMToLLVMIRTranslation.cpp        | 12 +++
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 12 +++
 9 files changed, 174 insertions(+)
 create mode 100644 flang/test/Fir/array-bounds-assume.fir

diff --git a/flang/include/flang/Optimizer/CodeGen/CodeGen.h b/flang/include/flang/Optimizer/CodeGen/CodeGen.h
index b7a9397edfe6d..dd4a4b4838f17 100644
--- a/flang/include/flang/Optimizer/CodeGen/CodeGen.h
+++ b/flang/include/flang/Optimizer/CodeGen/CodeGen.h
@@ -65,6 +65,12 @@ struct FIRToLLVMPassOptions {
   // Conversion pass of the MLIR complex dialect.
   Fortran::frontend::CodeGenOptions::ComplexRangeKind ComplexRange =
       Fortran::frontend::CodeGenOptions::ComplexRangeKind::CX_Full;
+
+  // Emit llvm.assume intrinsics describing array bounds constraints for
+  // non-boxed array accesses. These provide extra information to loop
+  // optimizations but also increase IR size, so they are only emitted at
+  // higher optimization levels (see getFIRToLLVMPassOptions).
+  bool emitArrayBoundsAssumes = false;
 };
 
 /// Convert FIR to the LLVM IR dialect with default options.
diff --git a/flang/include/flang/Optimizer/Passes/CommandLineOpts.h b/flang/include/flang/Optimizer/Passes/CommandLineOpts.h
index 882f02032a3b8..6ee2e41936840 100644
--- a/flang/include/flang/Optimizer/Passes/CommandLineOpts.h
+++ b/flang/include/flang/Optimizer/Passes/CommandLineOpts.h
@@ -45,6 +45,11 @@ extern llvm::cl::opt<bool> ignoreMissingTypeDescriptors;
 /// differs from most compilers).
 extern llvm::cl::opt<bool> skipExternalRttiDefinition;
 
+/// Shared option in tools to control the emission of llvm.assume intrinsics
+/// describing array bounds constraints for non-boxed array accesses. Even when
+/// enabled, these assumes are only emitted at higher optimization levels.
+extern llvm::cl::opt<bool> enableArrayBoundsAssumes;
+
 /// Default optimization level used to create Flang pass pipeline is O0.
 extern llvm::OptimizationLevel defaultOptLevel;
 
diff --git a/flang/lib/Optimizer/CodeGen/CodeGen.cpp b/flang/lib/Optimizer/CodeGen/CodeGen.cpp
index 74ad23a050fae..6b6c2612b9b52 100644
--- a/flang/lib/Optimizer/CodeGen/CodeGen.cpp
+++ b/flang/lib/Optimizer/CodeGen/CodeGen.cpp
@@ -2883,6 +2883,13 @@ struct XArrayCoorOpConversion
     mlir::LLVM::GEPNoWrapFlags gepFlags =
         mlir::LLVM::GEPNoWrapFlags::nusw | mlir::LLVM::GEPNoWrapFlags::nuw;
 
+    // An assumed-size array's last dimension has an unknown extent that is
+    // encoded as -1, so no meaningful upper-bound assume can be emitted for it.
+    const bool lastDimIsAssumedSize =
+        !coor.getShape().empty() &&
+        llvm::isa_and_nonnull<fir::AssumedSizeExtentOp>(
+            coor.getShape().back().getDefiningOp());
+
     // For each dimension of the array, generate the offset calculation.
     for (unsigned i = 0; i < rank; ++i, ++indexOffset, ++shapeOffset,
                   ++shiftOffset, sliceOffset += 3, sliceOps += 3) {
@@ -2891,6 +2898,43 @@ struct XArrayCoorOpConversion
       mlir::Value lb =
           isShifted ? integerCast(loc, rewriter, idxTy, operands[shiftOffset])
                     : one;
+
+      // Emit array bounds assumes per Fortran 2018 standard section 9.5.3.3.2:
+      // "The value of each subscript expression shall be within the bounds for
+      // its dimension unless the array section has size zero."
+      // For non-boxed arrays with known shape, emit: assume(index >= lb) and
+      // assume(index <= lb + extent - 1).
+      // The llvm.array_bounds attribute marks these assumes for removal before
+      // vectorization to prevent IR bloat and avoid impacting cost models. This
+      // is only emitted at higher optimization levels (see FIRToLLVMPassOptions
+      // and getFIRToLLVMPassOptions).
+      if (options.emitArrayBoundsAssumes && !baseIsBoxed &&
+          !coor.getShape().empty()) {
+        // Lower bound assume: index >= lb.
+        mlir::Value lbCheck = mlir::LLVM::ICmpOp::create(
+            rewriter, loc, mlir::LLVM::ICmpPredicate::sge, index, lb);
+        auto lbAssume = mlir::LLVM::AssumeOp::create(rewriter, loc, lbCheck);
+        lbAssume->setAttr(mlir::LLVM::AssumeOp::getArrayBoundsAttrName(),
+                          rewriter.getUnitAttr());
+
+        // Upper bound assume: index <= lb + extent - 1. The extent of an
+        // assumed-size array's last dimension is unknown, so skip the
+        // upper-bound assume in that case to avoid a bogus assumption.
+        if (!(i == rank - 1 && lastDimIsAssumedSize)) {
+          mlir::Value extent =
+              integerCast(loc, rewriter, idxTy, operands[shapeOffset]);
+          mlir::Value lbPlusExtent =
+              mlir::LLVM::AddOp::create(rewriter, loc, idxTy, lb, extent, nsw);
+          mlir::Value ub = mlir::LLVM::SubOp::create(rewriter, loc, idxTy,
+                                                     lbPlusExtent, one, nsw);
+          mlir::Value ubCheck = mlir::LLVM::ICmpOp::create(
+              rewriter, loc, mlir::LLVM::ICmpPredicate::sle, index, ub);
+          auto ubAssume = mlir::LLVM::AssumeOp::create(rewriter, loc, ubCheck);
+          ubAssume->setAttr(mlir::LLVM::AssumeOp::getArrayBoundsAttrName(),
+                            rewriter.getUnitAttr());
+        }
+      }
+
       mlir::Value step = one;
       bool normalSlice = isSliced;
       // Compute zero based index in dimension i of the element, applying
diff --git a/flang/lib/Optimizer/Passes/CommandLineOpts.cpp b/flang/lib/Optimizer/Passes/CommandLineOpts.cpp
index d461c1b9757b5..946c66435118f 100644
--- a/flang/lib/Optimizer/Passes/CommandLineOpts.cpp
+++ b/flang/lib/Optimizer/Passes/CommandLineOpts.cpp
@@ -45,6 +45,12 @@ cl::opt<bool> skipExternalRttiDefinition(
                    "other compilation units"),
     cl::Hidden);
 
+cl::opt<bool> enableArrayBoundsAssumes(
+    "enable-array-bounds-assumes",
+    cl::desc("emit llvm.assume intrinsics for array bounds constraints of "
+             "non-boxed array accesses (only at -O2 and above)"),
+    cl::init(true), cl::Hidden);
+
 OptimizationLevel defaultOptLevel{OptimizationLevel::O0};
 
 codegenoptions::DebugInfoKind noDebugInfo{codegenoptions::NoDebugInfo};
diff --git a/flang/lib/Optimizer/Passes/Pipelines.cpp b/flang/lib/Optimizer/Passes/Pipelines.cpp
index a4e9d0c227817..40d5f97f2893e 100644
--- a/flang/lib/Optimizer/Passes/Pipelines.cpp
+++ b/flang/lib/Optimizer/Passes/Pipelines.cpp
@@ -130,6 +130,11 @@ getFIRToLLVMPassOptions(const MLIRToLLVMPassPipelineConfig &config) {
   options.typeDescriptorsRenamedForAssembly =
       !disableCompilerGeneratedNamesConversion;
   options.ComplexRange = config.ComplexRange;
+  // Only emit array bounds assumes at -O2 and above, as they increase IR size
+  // for a benefit that mainly helps loop optimizations at higher optimization
+  // levels.
+  options.emitArrayBoundsAssumes =
+      enableArrayBoundsAssumes && config.OptLevel.getSpeedupLevel() >= 2;
   return options;
 }
 
diff --git a/flang/test/Fir/array-bounds-assume.fir b/flang/test/Fir/array-bounds-assume.fir
new file mode 100644
index 0000000000000..ac0b87b9ffd11
--- /dev/null
+++ b/flang/test/Fir/array-bounds-assume.fir
@@ -0,0 +1,76 @@
+// Test that array bounds assumes are generated for fir.array_coor operations.
+// This tests that Flang emits llvm.assume intrinsics for array bounds constraints.
+
+// RUN: tco %s | FileCheck %s
+
+// Test 1D array with constant shape.
+func.func @array_coor_1d(%addr : !fir.ref<!fir.array<10xi32>>, %idx : index) -> i32 {
+  %c10 = arith.constant 10 : index
+  %shape = fir.shape %c10 : (index) -> !fir.shape<1>
+  %ref = fir.array_coor %addr(%shape) %idx : (!fir.ref<!fir.array<10xi32>>, !fir.shape<1>, index) -> !fir.ref<i32>
+  %val = fir.load %ref : !fir.ref<i32>
+  return %val : i32
+}
+
+// CHECK-LABEL: define i32 @array_coor_1d
+// CHECK: %[[LB_CHECK:.*]] = icmp sge i64 %{{.*}}, 1
+// CHECK: call void @llvm.assume(i1 %[[LB_CHECK]]){{.*}}!llvm.array.bounds
+// CHECK: %[[UB_CHECK:.*]] = icmp sle i64 %{{.*}}, 10
+// CHECK: call void @llvm.assume(i1 %[[UB_CHECK]]){{.*}}!llvm.array.bounds
+
+// Test 2D array with constant shape.
+func.func @array_coor_2d(%addr : !fir.ref<!fir.array<10x20xi32>>, %i : index, %j : index) -> i32 {
+  %c10 = arith.constant 10 : index
+  %c20 = arith.constant 20 : index
+  %shape = fir.shape %c10, %c20 : (index, index) -> !fir.shape<2>
+  %ref = fir.array_coor %addr(%shape) %i, %j : (!fir.ref<!fir.array<10x20xi32>>, !fir.shape<2>, index, index) -> !fir.ref<i32>
+  %val = fir.load %ref : !fir.ref<i32>
+  return %val : i32
+}
+
+// CHECK-LABEL: define i32 @array_coor_2d
+// First dimension (i) bounds check: 1 <= i <= 10.
+// CHECK: %[[I_LB_CHECK:.*]] = icmp sge i64 %{{.*}}, 1
+// CHECK: call void @llvm.assume(i1 %[[I_LB_CHECK]]){{.*}}!llvm.array.bounds
+// CHECK: %[[I_UB_CHECK:.*]] = icmp sle i64 %{{.*}}, 10
+// CHECK: call void @llvm.assume(i1 %[[I_UB_CHECK]]){{.*}}!llvm.array.bounds
+// Second dimension (j) bounds check: 1 <= j <= 20.
+// CHECK: %[[J_LB_CHECK:.*]] = icmp sge i64 %{{.*}}, 1
+// CHECK: call void @llvm.assume(i1 %[[J_LB_CHECK]]){{.*}}!llvm.array.bounds
+// CHECK: %[[J_UB_CHECK:.*]] = icmp sle i64 %{{.*}}, 20
+// CHECK: call void @llvm.assume(i1 %[[J_UB_CHECK]]){{.*}}!llvm.array.bounds
+
+// Test array with shifted lower bounds.
+func.func @array_coor_shifted(%addr : !fir.ref<!fir.array<10xi32>>, %idx : index) -> i32 {
+  %c5 = arith.constant 5 : index
+  %c10 = arith.constant 10 : index
+  %shape = fir.shape_shift %c5, %c10 : (index, index) -> !fir.shapeshift<1>
+  %ref = fir.array_coor %addr(%shape) %idx : (!fir.ref<!fir.array<10xi32>>, !fir.shapeshift<1>, index) -> !fir.ref<i32>
+  %val = fir.load %ref : !fir.ref<i32>
+  return %val : i32
+}
+
+// CHECK-LABEL: define i32 @array_coor_shifted
+// Bounds check with lower bound 5: 5 <= idx <= 14.
+// CHECK: %[[LB_CHECK:.*]] = icmp sge i64 %{{.*}}, 5
+// CHECK: call void @llvm.assume(i1 %[[LB_CHECK]]){{.*}}!llvm.array.bounds
+// CHECK: %[[UB_CHECK:.*]] = icmp sle i64 %{{.*}}, 14
+// CHECK: call void @llvm.assume(i1 %[[UB_CHECK]]){{.*}}!llvm.array.bounds
+
+// Test assumed-size array: the extent of the last dimension is unknown
+// (encoded as -1), so only the lower-bound assume is emitted for it; emitting
+// an upper-bound assume would be a bogus assumption.
+func.func @array_coor_assumed_size(%addr : !fir.ref<!fir.array<?xi32>>, %idx : index) -> i32 {
+  %ext = fir.assumed_size_extent : index
+  %shape = fir.shape %ext : (index) -> !fir.shape<1>
+  %ref = fir.array_coor %addr(%shape) %idx : (!fir.ref<!fir.array<?xi32>>, !fir.shape<1>, index) -> !fir.ref<i32>
+  %val = fir.load %ref : !fir.ref<i32>
+  return %val : i32
+}
+
+// CHECK-LABEL: define i32 @array_coor_assumed_size
+// CHECK: %[[LB_CHECK:.*]] = icmp sge i64 %{{.*}}, 1
+// CHECK: call void @llvm.assume(i1 %[[LB_CHECK]]){{.*}}!llvm.array.bounds
+// No upper-bound assume for the assumed-size last dimension.
+// CHECK-NOT: call void @llvm.assume
+// CHECK: ret i32
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index 73629b95a06e5..1d1e86c4bd077 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -655,6 +655,14 @@ def LLVM_AssumeOp
                    "Value":$ptr1, "Value":$ptr2)>
   ];
 
+  let extraClassDeclaration = [{
+    /// Name of the discardable attribute marking an `llvm.assume` as an array
+    /// bounds assumption. It is translated to the `llvm.array.bounds` metadata
+    /// on the corresponding LLVM IR intrinsic call, which passes such as
+    /// DropUnnecessaryAssumes use to selectively drop these assumptions.
+    static StringRef getArrayBoundsAttrName() { return "llvm.array_bounds"; }
+  }];
+
   let hasVerifier = 1;
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp
index 7cd8c3c77c15f..fd89fcdec33f7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp
@@ -903,6 +903,18 @@ amendOperationImpl(Operation &op, ArrayRef<llvm::Instruction *> instructions,
       inst->setMetadata(llvm::LLVMContext::MD_mmra, mmraMd);
     return success();
   }
+
+  // Handle the array bounds attribute on llvm.assume: it marks assumes as
+  // array bounds checks that should be dropped before vectorization. Translate
+  // it to the "llvm.array.bounds" metadata on the LLVM IR intrinsic call.
+  if (name == LLVM::AssumeOp::getArrayBoundsAttrName()) {
+    llvm::LLVMContext &ctx = moduleTranslation.getLLVMContext();
+    llvm::MDNode *md = llvm::MDNode::get(ctx, {});
+    for (llvm::Instruction *inst : instructions)
+      inst->setMetadata("llvm.array.bounds", md);
+    return success();
+  }
+
   return success();
 }
 
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index ea393dd445eda..df356d9ef45c4 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1415,6 +1415,15 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
   llvm.return %0 : vector<4 x i32>
 }
 
+// The `llvm.array_bounds` unit attribute on `llvm.intr.assume` is translated to
+// the `llvm.array.bounds` metadata (an empty node) on the LLVM IR call.
+// CHECK-LABEL: @assume_array_bounds
+llvm.func @assume_array_bounds(%cond: i1) {
+  // CHECK: call void @llvm.assume(i1 %{{.+}}){{.*}}!llvm.array.bounds ![[BOUNDS:[0-9]+]]
+  llvm.intr.assume %cond : i1 {llvm.array_bounds}
+  llvm.return
+}
+
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
 // CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
@@ -1622,3 +1631,6 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare range(i2 -1, -2) i2 @llvm.scmp.i2.i32(i32, i32)
 // CHECK-DAG: declare range(i32 -1, 2) <4 x i32> @llvm.scmp.v4i32.v4i32(<4 x i32>, <4 x i32>)
 // CHECK-DAG: declare void @llvm.fake.use(...)
+
+// The array bounds metadata node is empty.
+// CHECK-DAG: ![[BOUNDS]] = !{}


From llvmlistbot at llvm.org  Thu Jul  2 00:34:43 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Thu, 02 Jul 2026 00:34:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Add missing dependent dialect
 to TestVectorUnrollingPatterns (PR #207113)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207113@github.com>
Message-ID: <6a461493.491d90f5.1f0d58.838c@mx.google.com>


https://github.com/hockyy approved this pull request.


https://github.com/llvm/llvm-project/pull/207113

From llvmlistbot at llvm.org  Thu Jul  2 00:50:30 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Thu, 02 Jul 2026 00:50:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a461846.abb10101.208f3e.cdeb@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/10] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/10] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/10] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/10] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/10] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/10] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/10] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/10] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/10] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 9833b469158286fe795dbe657b681b7056a75242 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/10] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 307 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 161 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..d8e985f133e70 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,12 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
-  // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
-  // Delinearized offsets to in-bounds reinterpret_cast source indices.
-  // Optional since it is only supported for static offsets.
+struct ResultNonUnitDimsAndOffsetsForRC {
+  SmallVector<unsigned> nonUnitDimsPos;
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +105,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +213,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +251,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +282,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +334,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +364,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +376,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +411,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +734,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..f427b96e5eae9 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 3, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 3, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 3, 1]>>
   return
 }
 


From llvmlistbot at llvm.org  Thu Jul  2 00:56:32 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Thu, 02 Jul 2026 00:56:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4619b0.331483e7.5bc1e.4383@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/10] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/10] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/10] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/10] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/10] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/10] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/10] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/10] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/10] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 3fe012b6f054db7666f502a2b07bd822a39fd1cf Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/10] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..f427b96e5eae9 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 3, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 3, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 3, 1]>>
   return
 }
 


From llvmlistbot at llvm.org  Thu Jul  2 01:09:23 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Thu, 02 Jul 2026 01:09:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM]Import llvm intrinsics for
	nvvm.barrier (PR #202862)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202862@github.com>
Message-ID: <6a461cb3.ac469dc7.e3aa6.01f9@mx.google.com>


https://github.com/grypp approved this pull request.


https://github.com/llvm/llvm-project/pull/202862

From llvmlistbot at llvm.org  Thu Jul  2 01:14:24 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Thu, 02 Jul 2026 01:14:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207111@github.com>
Message-ID: <6a461de0.92ea0e2a.21f97.ee1e@mx.google.com>


================

----------------
grypp wrote:

We're currently using `static constexpr` tables in the code. They're 100% ugly but easy to read, and it's clear what's happening. We definitely want to change this, but we need a generic solution.

Your idea here is interesting — we have other OPs where we do intrinsic selection on larger lists. What would this look like there? 

I'm reluctant to have a different way of doing intrinsic selection.

https://github.com/llvm/llvm-project/pull/207111

From llvmlistbot at llvm.org  Thu Jul  2 01:16:36 2026
From: llvmlistbot at llvm.org (Momchil Velikov)
Date: Thu, 02 Jul 2026 01:16:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Install missing standalone helper
	tools (PR #205066)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205066@github.com>
Message-ID: <6a461e64.f14cf1a5.2c3aaa.ec4c@mx.google.com>


momchil-velikov wrote:

Ping?

https://github.com/llvm/llvm-project/pull/205066

From llvmlistbot at llvm.org  Thu Jul  2 01:42:34 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Thu, 02 Jul 2026 01:42:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207111@github.com>
Message-ID: <6a46247a.65dfdb71.1ca534.b946@mx.google.com>


================

----------------
durga4github wrote:

Thank you for thinking about the import line! Like Guray mentioned, we would prefer a general design/solution that can apply to all the Ops, though we may not implement everything in this PR.

https://github.com/llvm/llvm-project/pull/207111

From llvmlistbot at llvm.org  Thu Jul  2 01:43:47 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Thu, 02 Jul 2026 01:43:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4624c3.d1496011.3d49cb.f358@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/10] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/10] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/10] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/10] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/10] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/10] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/10] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/10] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/10] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/10] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 


From llvmlistbot at llvm.org  Thu Jul  2 01:51:06 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Thu, 02 Jul 2026 01:51:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207111@github.com>
Message-ID: <6a46267a.a87eb5dc.2cf3a1.dac7@mx.google.com>


================

----------------
durga4github wrote:

I wonder if it's possible to share the same table across both the import/export flows..?
That would reduce duplication and make future maintenance and updates easier.

https://github.com/llvm/llvm-project/pull/207111

From llvmlistbot at llvm.org  Thu Jul  2 01:52:51 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 01:52:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4626e3.f8b79514.3b441.dce8@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
    to offset: [0], sizes: [1, 1], strides: [54, 2]
    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
```

Keep `[2, 54]` as an example of the weird stuff that `memref.reinterpret_cast` allows:
```mlir
 %reinterpret_cast = memref.reinterpret_cast %dst
    to offset: [0], sizes: [1, 1], strides: [2, 54]
    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 01:59:29 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 02 Jul 2026 01:59:29 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a462871.331483e7.5bc1e.e8c6@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Multiple` hints are passed to hint the alignment between the loop tile sizes
+      // (16 * vscale, 8 * vscale) and the inner tile sizes (8 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[16], [8]]
+          inner_tile_alignments = [Multiple, Multiple]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Unaligned scalable tiling - static tile sizes not aligned to scalable
+// inner tiles.
+
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_unaligned
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK:             %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[EXTRACT]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_unaligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // No hint is passed: the static tile sizes (7, 5) are not aligned to the scalable
+      // inner tiles (8 * vscale, 4 * vscale), so the tiled unpack keeps a remainder slice.
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [7, 5] : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
----------------
egebeysel wrote:

I restructured the tests quite heavily and increased re-use by a lot. Please take another look :)

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 01:59:34 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 02 Jul 2026 01:59:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a462876.f14cf1a5.2c3aaa.f1f1@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
----------------
egebeysel wrote:

done

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 01:59:39 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 02 Jul 2026 01:59:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46287b.f06f4994.1e3a60.e1c8@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
----------------
egebeysel wrote:

done

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 02:01:07 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Thu, 02 Jul 2026 02:01:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LLVM][NVVM] ADD LLVM_NToOneIntrOpBase
 for N:1 intrinsic import (PR #207111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207111@github.com>
Message-ID: <6a4628d3.576b3af0.2cb9b7.958c@mx.google.com>


durga4github wrote:

Also, if possible, could you share a bit more context? Is there a particular flow or use case you have in mind using the import? 

Or are you interested in the importer for this barrier Op alone?

https://github.com/llvm/llvm-project/pull/207111

From llvmlistbot at llvm.org  Thu Jul  2 02:02:22 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 02 Jul 2026 02:02:22 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46291e.dd2b4bd3.35cd5e.3fd3@mx.google.com>


================

----------------
egebeysel wrote:

Update here, the pure tiling case did not need it, but the TODO was still necessary, since producer fusion of pack needed it. I initially left that as a TODO, but it was rather a short fix, so I implemented it (in `PackOpTiling::generateResultTileValue`, `mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp:1074`) and added the corresponding tests. Please take a look :) 

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 02:02:35 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 02 Jul 2026 02:02:35 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46292b.ac469dc7.e3aa6.750a@mx.google.com>


================
@@ -0,0 +1,784 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// The transform ops below carry an optional `inner_tile_alignments` hint,
+// written as a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// Perfect scalable tiling - scalable tile sizes equal scalable inner
+// tiles. Outer sizes of the tiled unpack should be 1's.
+
+// CHECK-LABEL: func.func @perfect_CKkc_to_KC_scalable
+// CHECK:         %[[RES:.*]] = scf.for
+// CHECK:           scf.for
+// CHECK:             %[[UNPACK:.*]] = linalg.unpack
+// CHECK-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @perfect_CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+      %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+      // The `Equal` hints assert the loop tile sizes (2 * vscale, 4 * vscale) equal the
+      // inner tile sizes (2 * vscale, 4 * vscale).
+      %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+          inner_tile_alignments = [Equal, Equal]
+          : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+      transform.yield
+  }
+}
+
+// -----
+
+// Aligned scalable tiling - scalable tile sizes ([16] = 16*vscale, [8] =
+// 8*vscale) that are integer multiples of the scalable inner tiles (2x 8*vscale
+// and 2x 4*vscale), so tiling is aligned and the tiled unpack needs no trailing
+// remainder slice.
+
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @NCnc_to_NC_scalable_aligned
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//      CHECK:    %[[RES:.*]] = scf.for
+//      CHECK:      %[[OUTER0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:      %[[OUTER1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+//      CHECK:      %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER0]], %[[OUTER1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+//      CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+// CHECK-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+// CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+// CHECK:             tensor.insert_slice %[[UNPACK]]
+// CHECK:         return %[[RES]]
+func.func @NCnc_to_NC_scalable_aligned(%source: tensor<4x8x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
----------------
egebeysel wrote:

done

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 02:25:16 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 02:25:16 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM
 conversion for GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a462e7c.fdc477d6.118e5e.c7f0@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206951

>From 0c27ffa07eb9b334cbc1bd5e4df2a94abd4e2643 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:15:31 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM conversion for GL.Fract

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 22 ++++++++++++++++++-
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 15 +++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c9eba35cf0939 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, operand);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, operand, floored);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1936,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..042b68bef9dcd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,18 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Fract
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fract
+spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%{{.*}}) : (f32) -> f32
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOOR]] : f32
+  %0 = spirv.GL.Fract %arg0 : f32
+  // CHECK: %[[FLOORV:.*]] = llvm.intr.floor(%{{.*}}) : (vector<3xf16>) -> vector<3xf16>
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOORV]] : vector<3xf16>
+  %1 = spirv.GL.Fract %arg1 : vector<3xf16>
+  spirv.Return
+}

>From c7a22b3372540610ec185a488467988809792f07 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Thu, 2 Jul 2026 11:25:00 +0200
Subject: [PATCH 2/2] [SPIR-V] Replace isPipeOrAddressSpaceCastBI hardcoded
 list with table lookup

---
 llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp |  4 ++++
 llvm/lib/Target/SPIRV/SPIRVBuiltins.h   |  3 +++
 llvm/lib/Target/SPIRV/SPIRVBuiltins.td  |  2 ++
 llvm/lib/Target/SPIRV/SPIRVUtils.cpp    | 29 ++-----------------------
 4 files changed, 11 insertions(+), 27 deletions(-)

diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
index f97e0060c67e4..d7adc5e108172 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.cpp
@@ -3984,5 +3984,9 @@ lowerBuiltinType(const Type *OpaqueType,
 
   return TargetType;
 }
+
+bool isPipeOrAddressSpaceCastBuiltin(StringRef Name) {
+  return lookupBuiltin(Name, OpenCL_std) != nullptr;
+}
 } // namespace SPIRV
 } // namespace llvm
diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.h b/llvm/lib/Target/SPIRV/SPIRVBuiltins.h
index 81a6cc697ead7..88afb38a7c5aa 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.h
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.h
@@ -83,6 +83,9 @@ SPIRVTypeInst lowerBuiltinType(const Type *Type,
                                AccessQualifier::AccessQualifier AccessQual,
                                MachineIRBuilder &MIRBuilder,
                                SPIRVGlobalRegistry *GR);
+
+/// Returns true if \p Name is a pipe or address-space-cast OpenCL builtin.
+bool isPipeOrAddressSpaceCastBuiltin(StringRef Name);
 } // namespace SPIRV
 } // namespace llvm
 #endif // LLVM_LIB_TARGET_SPIRV_SPIRVBUILTINS_H
diff --git a/llvm/lib/Target/SPIRV/SPIRVBuiltins.td b/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
index 64d5695ef2167..d8aa8d63f921f 100644
--- a/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
+++ b/llvm/lib/Target/SPIRV/SPIRVBuiltins.td
@@ -1256,6 +1256,8 @@ defm : DemangledNativeBuiltin<"clock_read_hilo_sub_group", OpenCL_std, KernelClo
 //SPV_ALTERA_blocking_pipes
 defm : DemangledNativeBuiltin<"__spirv_WritePipeBlockingINTEL", OpenCL_std, BlockingPipes, 0, 0, OpWritePipeBlockingALTERA>;
 defm : DemangledNativeBuiltin<"__spirv_ReadPipeBlockingINTEL", OpenCL_std, BlockingPipes, 0, 0, OpReadPipeBlockingALTERA>;
+defm : DemangledNativeBuiltin<"__read_pipe_2_bl", OpenCL_std, BlockingPipes, 4, 4, OpReadPipeBlockingALTERA>;
+defm : DemangledNativeBuiltin<"__write_pipe_2_bl", OpenCL_std, BlockingPipes, 4, 4, OpWritePipeBlockingALTERA>;
 defm : DemangledNativeBuiltin<"__spirv_ReadClockKHR", OpenCL_std, KernelClock, 1, 1, OpReadClockKHR>;
 
 //SPV_ALTERA_arbitrary_precision_fixed_point
diff --git a/llvm/lib/Target/SPIRV/SPIRVUtils.cpp b/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
index 7ffd6d1e86c21..4b158a1145959 100644
--- a/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVUtils.cpp
@@ -13,6 +13,7 @@
 #include "SPIRVUtils.h"
 #include "MCTargetDesc/SPIRVBaseInfo.h"
 #include "SPIRV.h"
+#include "SPIRVBuiltins.h"
 #include "SPIRVGlobalRegistry.h"
 #include "SPIRVInstrInfo.h"
 #include "SPIRVSubtarget.h"
@@ -535,32 +536,6 @@ Type *getMDOperandAsType(const MDNode *N, unsigned I) {
   return toTypedPointer(ElementTy);
 }
 
-// The set of names is borrowed from the SPIR-V translator.
-// TODO: may be implemented in SPIRVBuiltins.td.
-static bool isPipeOrAddressSpaceCastBI(const StringRef MangledName) {
-  return MangledName == "write_pipe_2" || MangledName == "read_pipe_2" ||
-         MangledName == "write_pipe_2_bl" || MangledName == "read_pipe_2_bl" ||
-         MangledName == "write_pipe_4" || MangledName == "read_pipe_4" ||
-         MangledName == "reserve_write_pipe" ||
-         MangledName == "reserve_read_pipe" ||
-         MangledName == "commit_write_pipe" ||
-         MangledName == "commit_read_pipe" ||
-         MangledName == "work_group_reserve_write_pipe" ||
-         MangledName == "work_group_reserve_read_pipe" ||
-         MangledName == "work_group_commit_write_pipe" ||
-         MangledName == "work_group_commit_read_pipe" ||
-         MangledName == "get_pipe_num_packets_ro" ||
-         MangledName == "get_pipe_max_packets_ro" ||
-         MangledName == "get_pipe_num_packets_wo" ||
-         MangledName == "get_pipe_max_packets_wo" ||
-         MangledName == "sub_group_reserve_write_pipe" ||
-         MangledName == "sub_group_reserve_read_pipe" ||
-         MangledName == "sub_group_commit_write_pipe" ||
-         MangledName == "sub_group_commit_read_pipe" ||
-         MangledName == "to_global" || MangledName == "to_local" ||
-         MangledName == "to_private";
-}
-
 static bool isEnqueueKernelBI(const StringRef MangledName) {
   return MangledName == "__enqueue_kernel_basic" ||
          MangledName == "__enqueue_kernel_basic_events" ||
@@ -580,7 +555,7 @@ static bool isNonMangledOCLBuiltin(StringRef Name) {
     return false;
 
   return isEnqueueKernelBI(Name) || isKernelQueryBI(Name) ||
-         isPipeOrAddressSpaceCastBI(Name.drop_front(2)) ||
+         SPIRV::isPipeOrAddressSpaceCastBuiltin(Name) ||
          Name == "__translate_sampler_initializer";
 }
 


From llvmlistbot at llvm.org  Thu Jul  2 02:25:43 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 02:25:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a462e97.bb839515.23e6d3.295b@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206951

>From 0c27ffa07eb9b334cbc1bd5e4df2a94abd4e2643 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:15:31 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for GL.Fract

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 22 ++++++++++++++++++-
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 15 +++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c9eba35cf0939 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, operand);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, operand, floored);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1936,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..042b68bef9dcd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,18 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Fract
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fract
+spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%{{.*}}) : (f32) -> f32
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOOR]] : f32
+  %0 = spirv.GL.Fract %arg0 : f32
+  // CHECK: %[[FLOORV:.*]] = llvm.intr.floor(%{{.*}}) : (vector<3xf16>) -> vector<3xf16>
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOORV]] : vector<3xf16>
+  %1 = spirv.GL.Fract %arg1 : vector<3xf16>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Thu Jul  2 03:09:54 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f2.4a342cbc.134cb.6db9@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f3.c56fb102.11841f.1faa@mx.google.com>


https://github.com/banach-space commented:

Fresh round of comments - not associated with any specific commit.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f3.6cfa33d2.19518c.f5ce@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(%src : memref<1x1xf32>,
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f3.cb30c3e4.6f38b.3fd7@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
// CHECK-LABEL: func.func private @copy_scalar_into_1D_zero_offset_strided_dynamic_stride(
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f4.492e23e4.e71ca.aa50@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f4.4a342cbc.134cb.6e6d@mx.google.com>


================

----------------
banach-space wrote:

Does the FIXME comment apply to the 1st `assert` or the 1st and the 2nd? ATM the formatting implies the latter.
```suggestion
  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
  // only a single offset. That should be fixed at the op definition level.
  assert(rcOffsets.size() == 1 && "Expecting single offset");

    assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
  assert(rcOffsets[0] >= 0 &&
         "static reinterpret_cast offset must be non-negative");
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f4.92ea0e2a.21f97.d5ff@mx.google.com>


================

----------------
banach-space wrote:

Could this be adapted and moved near `@copy_scalar_into_1D_strided_zero_offset`? Also, I would rename it as (after adapting) `@negative_copy_scalar_into_1D_strided_non_identity_layout`.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f4.c56fb102.11841f.21ae@mx.google.com>


================

----------------
banach-space wrote:

[nit] I would move it near where `rcOffsets` are define - just to avoid mixing `asserts` and regular code so much.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f4.eb24c6ea.1f2e26.889f@mx.google.com>


================

----------------
banach-space wrote:

I've scanned this method and I see no mention of `memref.copy` in the implementation. This implies that it should be possible to document this method without mentioning `memref.copy`. Also, this high-level comment is longer than the implementation. A simple helper hook should not require this much documentation. Please select 1 or 2 most representative examples and remove everything else. Instead, you can use the test to document the functionality.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f5.2020f6c2.171568.16f3@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
  bool isScalarCopy = !llvm::all_of(resType.getShape(),
                    [](int64_t size) { return size == 1; }));
  
  // For scalar copies, result strides are irrelevant, including dynamic ones.
  // For non-scalar copies, require static result strides identical to the
  // identity strides of the reinterpret_cast source.
  if (isScalarCopy) {
    SmallVector<int64_t> srcIdentityStrides =
        computeStrides(srcType.getShape());
    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();

    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
           "Expecting same number of strides for rank-preserving "
           "reinterpret_casts.");

    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                      [](auto pair) {
                        auto [srcStride, resultStride] = pair;
                        return !ShapedType::isDynamic(resultStride) &&
                               srcStride == resultStride;
}))
      return std::nullopt;
      }
   
   // TODO - Add comment explaining _why_ non-unit sizes are tracked in this case   
   if (isScalarCopy) {
    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
      if (resultSize != 1)
        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
    }
  }
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f5.abb10101.208f3e.7801@mx.google.com>


================

----------------
banach-space wrote:

It would make more sense to have `@negative_copy_into_ND_strided_dynamic_offset` here, renamed (and adapted) as `@negative_scalar_into_2D_strided_dynamic_offset`. Basically, `@negative_copy_into_ND_strided_dynamic_offset` is a more complex version of `@copy_scalar_into_2D_strided_non_identity_stride`, right?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 03:09:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 03:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4638f5.af6ba3ed.2c2f30.313d@mx.google.com>


================

----------------
banach-space wrote:

Doesn't this comment apply to all 3 examples below? Rather than repeat the comment, you can insert a block comment.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 04:24:13 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 02 Jul 2026 04:24:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enable end-to-end integration
 test compilation (PR #207158)
Message-ID: <llvm.org/llvm/llvm-project/pull/207158@github.com>


https://github.com/akroviakov created https://github.com/llvm/llvm-project/pull/207158

This PR enables the complex integration test compilation (not execution) to catch compilation regressions in our pipeline.

>From b3e7bace7693bc8e998cb88cf402992e4a0b566b Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Thu, 2 Jul 2026 11:22:26 +0000
Subject: [PATCH] [MLIR][XeGPU] Enable end-to-end integration test compilation

---
 .../WG/simple_mxfp_gemm_dequantizeB_F4.mlir    | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
index 8db649b600350..cb5c5a7422e1b 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
@@ -1,12 +1,12 @@
-// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri" \
-// RUN: | mlir-runner \
-// RUN:   --shared-libs=%mlir_levelzero_runtime \
-// RUN:   --shared-libs=%mlir_runner_utils \
-// RUN:   --shared-libs=%mlir_c_runner_utils \
-// RUN:   --entry-point-result=void \
-// RUN: | FileCheck %s
-
-// XFAIL: *
+// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri"
+// RUN-DISABLED: | mlir-runner \
+// RUN-DISABLED:   --shared-libs=%mlir_levelzero_runtime \
+// RUN-DISABLED:   --shared-libs=%mlir_runner_utils \
+// RUN-DISABLED:   --shared-libs=%mlir_c_runner_utils \
+// RUN-DISABLED:   --entry-point-result=void \
+// RUN-DISABLED: | FileCheck %s
+
+
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
 #a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>


From llvmlistbot at llvm.org  Thu Jul  2 04:24:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 04:24:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enable end-to-end integration
 test compilation (PR #207158)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207158@github.com>
Message-ID: <6a464a7f.283fd794.7ac42.58c2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Artem Kroviakov (akroviakov)

<details>
<summary>Changes</summary>

This PR enables the complex integration test compilation (not execution) to catch compilation regressions in our pipeline.

---
Full diff: https://github.com/llvm/llvm-project/pull/207158.diff


1 Files Affected:

- (modified) mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir (+9-9) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
index 8db649b600350..cb5c5a7422e1b 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
@@ -1,12 +1,12 @@
-// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri" \
-// RUN: | mlir-runner \
-// RUN:   --shared-libs=%mlir_levelzero_runtime \
-// RUN:   --shared-libs=%mlir_runner_utils \
-// RUN:   --shared-libs=%mlir_c_runner_utils \
-// RUN:   --entry-point-result=void \
-// RUN: | FileCheck %s
-
-// XFAIL: *
+// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri"
+// RUN-DISABLED: | mlir-runner \
+// RUN-DISABLED:   --shared-libs=%mlir_levelzero_runtime \
+// RUN-DISABLED:   --shared-libs=%mlir_runner_utils \
+// RUN-DISABLED:   --shared-libs=%mlir_c_runner_utils \
+// RUN-DISABLED:   --entry-point-result=void \
+// RUN-DISABLED: | FileCheck %s
+
+
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
 #a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207158

From llvmlistbot at llvm.org  Thu Jul  2 04:28:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 04:28:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Fix crash on 0D vector in
 vector-to-xegpu transfer lowering (PR #205812)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205812@github.com>
Message-ID: <6a464b4b.331483e7.5bc1e.7a51@mx.google.com>


Chennesxu wrote:

 @Jianhui-Li The checks are green now. Could you please help land this when you have a chance? Thanks!

https://github.com/llvm/llvm-project/pull/205812

From llvmlistbot at llvm.org  Thu Jul  2 04:46:14 2026
From: llvmlistbot at llvm.org (Pradeep Kumar)
Date: Thu, 02 Jul 2026 04:46:14 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [SDAG][NVPTX] Enable custom
 legalization of v1 type for Intrinsic results and operands (PR #203237)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203237@github.com>
Message-ID: <6a464f86.a87eb5dc.2cf3a1.54f0@mx.google.com>


https://github.com/schwarzschild-radius updated https://github.com/llvm/llvm-project/pull/203237

>From 87eaad76a187f9c4f3e0c9b5c7fa16b2b467f873 Mon Sep 17 00:00:00 2001
From: Pradeep Kumar <pradeepku at nvidia.com>
Date: Mon, 23 Mar 2026 09:25:43 +0000
Subject: [PATCH] [SDAG][NVPTX] Enable custom legalization of v1 type for
 Intrinsic results and operands

This commit enables custom legalization for v1 types by calling CustomLowerNode
which gives the target a chance to custom handle the node before scalarizing

To demonstrate, I have used tcgen05.ld/st intrinsics which previously used a
scalar i32 type while all wider variants used vector types

Assisted by: Claude Code
---
 llvm/include/llvm/IR/IntrinsicsNVVM.td        | 13 ++----
 .../SelectionDAG/LegalizeVectorTypes.cpp      |  8 ++++
 llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp   | 23 ++++++----
 llvm/lib/Target/NVPTX/NVPTXIntrinsics.td      | 11 +++--
 llvm/test/CodeGen/NVPTX/tcgen05-ld.ll         | 12 ++---
 llvm/test/CodeGen/NVPTX/tcgen05-st.ll         | 32 +++++++-------
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td   | 18 +++-----
 .../Dialect/LLVMIR/nvvm_check_target_sm.mlir  |  6 +--
 mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir  | 24 +++++-----
 mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir  | 44 +++++++++----------
 10 files changed, 100 insertions(+), 91 deletions(-)

diff --git a/llvm/include/llvm/IR/IntrinsicsNVVM.td b/llvm/include/llvm/IR/IntrinsicsNVVM.td
index fc7abc50dce01..d96f94cf4889f 100644
--- a/llvm/include/llvm/IR/IntrinsicsNVVM.td
+++ b/llvm/include/llvm/IR/IntrinsicsNVVM.td
@@ -1145,7 +1145,8 @@ class SHFL_INFO<bit sync, string mode, string type, bit return_pred> {
     [OpType, llvm_i32_ty, llvm_i32_ty]);
 }
 
-class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num, string ElemType = "i32"> {
+class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num,
+                                    string ElemType = "i32"> {
   int shift = !cond(!eq(Shape, "16x128b"): 1,
                     !eq(Shape, "16x256b"): 2,
                     true : 0);
@@ -1153,15 +1154,7 @@ class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num, string ElemType = "i3
   int veclen = !shl(1, !add(Num, shift));
 
   int valid = !le(veclen, 128);
-  LLVMType type = !cond(!eq(veclen,   1): LLVMType<!cast<ValueType>(ElemType)>,
-                        !eq(veclen,   2): LLVMType<!cast<ValueType>("v"#2#ElemType)>,
-                        !eq(veclen,   4): LLVMType<!cast<ValueType>("v"#4#ElemType)>,
-                        !eq(veclen,   8): LLVMType<!cast<ValueType>("v"#8#ElemType)>,
-                        !eq(veclen,  16): LLVMType<!cast<ValueType>("v"#16#ElemType)>,
-                        !eq(veclen,  32): LLVMType<!cast<ValueType>("v"#32#ElemType)>,
-                        !eq(veclen,  64): LLVMType<!cast<ValueType>("v"#64#ElemType)>,
-                        !eq(veclen, 128): LLVMType<!cast<ValueType>("v"#128#ElemType)>,
-                        true : llvm_void_ty);
+  LLVMType type = LLVMType<!cast<ValueType>("v"#veclen#ElemType)>;
 }
 
 class NVVM_TCGEN05_MMA_BASE<string Space, bit IsSparse, string Kind = ""> {
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index 547ee11db45e2..3a511b5d022c9 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -43,6 +43,10 @@ void DAGTypeLegalizer::ScalarizeVectorResult(SDNode *N, unsigned ResNo) {
              N->dump(&DAG));
   SDValue R = SDValue();
 
+  // See if the target wants to custom expand this node.
+  if (CustomLowerNode(N, N->getValueType(ResNo), true))
+    return;
+
   switch (N->getOpcode()) {
   default:
 #ifndef NDEBUG
@@ -860,6 +864,10 @@ bool DAGTypeLegalizer::ScalarizeVectorOperand(SDNode *N, unsigned OpNo) {
              N->dump(&DAG));
   SDValue Res = SDValue();
 
+  // See if the target wants to custom scalarize this node.
+  if (CustomLowerNode(N, N->getOperand(OpNo).getValueType(), false))
+    return false;
+
   switch (N->getOpcode()) {
   default:
 #ifndef NDEBUG
diff --git a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
index fa0b9fe5f7051..bacd857b19e8b 100644
--- a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
@@ -1124,19 +1124,19 @@ NVPTXTargetLowering::NVPTXTargetLowering(const NVPTXTargetMachine &TM,
 
   // Custom lowering for tcgen05.ld vector operands
   setOperationAction(ISD::INTRINSIC_W_CHAIN,
-                     {MVT::v2i32, MVT::v4i32, MVT::v8i32, MVT::v16i32,
-                      MVT::v32i32, MVT::v64i32, MVT::v128i32, MVT::v2f32,
-                      MVT::v4f32, MVT::v8f32, MVT::v16f32, MVT::v32f32,
-                      MVT::v64f32, MVT::v128f32},
+                     {MVT::v1i32, MVT::v2i32, MVT::v4i32, MVT::v8i32,
+                      MVT::v16i32, MVT::v32i32, MVT::v64i32, MVT::v128i32,
+                      MVT::v2f32, MVT::v4f32, MVT::v8f32, MVT::v16f32,
+                      MVT::v32f32, MVT::v64f32, MVT::v128f32},
                      Custom);
 
   // Custom lowering for tcgen05.st vector operands and the st.async
   // i128 (.b128) operand. MVT::i8 is needed for the st.async.{sys,gpu} b8
   // variant.
   setOperationAction(ISD::INTRINSIC_VOID,
-                     {MVT::i8, MVT::v2i32, MVT::v4i32, MVT::v8i32, MVT::v16i32,
-                      MVT::v32i32, MVT::v64i32, MVT::v128i32, MVT::i128,
-                      MVT::Other},
+                     {MVT::i8, MVT::v1i32, MVT::v2i32, MVT::v4i32, MVT::v8i32,
+                      MVT::v16i32, MVT::v32i32, MVT::v64i32, MVT::v128i32,
+                      MVT::i128, MVT::Other},
                      Custom);
 
   // Enable custom lowering for the following:
@@ -2796,6 +2796,8 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_st_async_gpu:
   case Intrinsic::nvvm_st_async_mmio_sys:
     return lowerStAsyncRelease(Op, DAG);
+
+  case Intrinsic::nvvm_tcgen05_st_16x64b_x1:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x2:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x4:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x8:
@@ -2815,6 +2817,7 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_tcgen05_st_16x256b_x8:
   case Intrinsic::nvvm_tcgen05_st_16x256b_x16:
   case Intrinsic::nvvm_tcgen05_st_16x256b_x32:
+  case Intrinsic::nvvm_tcgen05_st_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x2:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x4:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x8:
@@ -2824,6 +2827,7 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_tcgen05_st_32x32b_x64:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x128:
     return lowerTcgen05St(Op, DAG);
+  case Intrinsic::nvvm_tcgen05_st_16x32bx2_x1:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x2:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x4:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x8:
@@ -5357,7 +5361,7 @@ void NVPTXTargetLowering::getTgtMemIntrinsic(
   case Intrinsic::nvvm_tcgen05_st_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x1: {
     Info.opc = ISD::INTRINSIC_VOID;
-    Info.memVT = MVT::i32;
+    Info.memVT = MVT::v1i32;
     Info.ptrVal = I.getArgOperand(0);
     Info.offset = 0;
     Info.flags = MachineMemOperand::MOStore;
@@ -7249,12 +7253,14 @@ static void ReplaceINTRINSIC_W_CHAIN(SDNode *N, SelectionDAG &DAG,
     return;
   }
 
+  case Intrinsic::nvvm_tcgen05_ld_16x64b_x1:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x4:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x8:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x16:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x32:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x64:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x128:
+  case Intrinsic::nvvm_tcgen05_ld_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x4:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x8:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x16:
@@ -7279,6 +7285,7 @@ static void ReplaceINTRINSIC_W_CHAIN(SDNode *N, SelectionDAG &DAG,
     }
     return;
 
+  case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x1:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x4:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x8:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x16:
diff --git a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
index 3819b3143f447..5af5ae2fe9e52 100644
--- a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
+++ b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
@@ -5769,7 +5769,10 @@ class TCGEN05_LDST_REGINFO<int Veclen> {
   list<NVPTXRegClass> regs = !listsplat(B32, Veclen);
   // generate list of regnames for load/store operands
   list<string> reg_names = !foreach(x, !range(0, Veclen), "r" # x);
-  string regstring = "{{" # !interleave(!foreach(n, !range(0, Veclen), "$r" # n), ", ") # "}}";
+  string regstring = "{{" #
+                      !interleave(
+                        !foreach(n, !range(0, Veclen), "$r" # n), ", ") #
+                     "}}";
   dag Ins = !dag(ins, regs, reg_names);
   dag Outs = !dag(outs, regs, reg_names);
 }
@@ -5785,7 +5788,8 @@ class TCGEN05_LD_INST<string Shape, int Num, bit Pack> :
                                 NVVM_TCGEN05_LDST_ACCESS_SIZE<Shape, Num>.veclen>;
 
   let InOperandList = !con((ins B32:$taddr),
-                           !if(!eq(Shape, "16x32bx2"), (ins i64imm:$offset), (ins)));
+                           !if(!eq(Shape, "16x32bx2"),
+                                  (ins i64imm:$offset), (ins)));
   let OutOperandList = Info.Outs;
   let AsmString = "tcgen05.ld.sync.aligned"
                   # "." # Shape
@@ -5809,7 +5813,8 @@ class TCGEN05_ST_INST<string Shape, int Num, bit Unpack> :
                                 NVVM_TCGEN05_LDST_ACCESS_SIZE<Shape, Num>.veclen>;
 
   let InOperandList = !con((ins B32:$taddr),
-                           !if(!eq(Shape, "16x32bx2"), (ins i64imm:$offset), (ins)),
+                           !if(!eq(Shape, "16x32bx2"),
+                                  (ins i64imm:$offset), (ins)),
                            Info.Ins);
   let OutOperandList = (outs);
   let AsmString = "tcgen05.st.sync.aligned"
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll b/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
index 22eb7298133bb..33eb39ac7701e 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
@@ -27,7 +27,7 @@ define void @nvvm_tcgen05_ld_16x64b(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) %taddr, i1 0)
 
@@ -62,7 +62,7 @@ define void @nvvm_tcgen05_ld_16x64b_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) %taddr, i1 1)
 
@@ -219,7 +219,7 @@ define void @nvvm_tcgen05_ld_32x32b(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) %taddr, i1 0)
 
@@ -253,7 +253,7 @@ define void @nvvm_tcgen05_ld_32x32b_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) %taddr, i1 1)
 
@@ -288,7 +288,7 @@ define void @nvvm_tcgen05_ld_16x32bx2(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1], 2;
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1], 2;
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, i1 0)
 
@@ -322,7 +322,7 @@ define void @nvvm_tcgen05_ld_16x32bx2_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1], 2;
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1], 2;
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, i1 1)
 
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-st.ll b/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
index ccf6541d01973..952e1cf39b6fc 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
@@ -11,7 +11,7 @@
 ; RUN: %if ptxas-sm_110f && ptxas-isa-9.0 %{ llc < %s -march=nvptx64 -mcpu=sm_110f -mattr=+ptx90 | %ptxas-verify -arch=sm_110f %}
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b
-define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -92,7 +92,7 @@ define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x64b_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x64b.x128.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 0)
 
@@ -111,7 +111,7 @@ define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b_unpack
-define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -192,7 +192,7 @@ define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x64b_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x64b.x128.unpack::16b.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 1)
 
@@ -211,7 +211,7 @@ define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b
-define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<256>;
@@ -307,7 +307,7 @@ define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i3
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b_unpack
-define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<256>;
@@ -403,7 +403,7 @@ define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b
-define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<254>;
@@ -495,7 +495,7 @@ define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i3
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b_unpack
-define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<254>;
@@ -587,7 +587,7 @@ define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b
-define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -668,7 +668,7 @@ define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_32x32b_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.32x32b.x128.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 0)
 
@@ -687,7 +687,7 @@ define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b_unpack
-define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -768,7 +768,7 @@ define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_32x32b_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.32x32b.x128.unpack::16b.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 1)
 
@@ -787,7 +787,7 @@ define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2
-define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -868,7 +868,7 @@ define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x32bx2_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x32bx2.x128.b32 [%r1], 2, {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, <2 x i32> %stv2, i1 0)
 
@@ -887,7 +887,7 @@ define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2_unpack
-define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -968,7 +968,7 @@ define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x32bx2_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x32bx2.x128.unpack::16b.b32 [%r1], 2, {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, <2 x i32> %stv2, i1 1)
 
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..89950d95e647f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5538,8 +5538,8 @@ def NVVM_Tcgen05LdOp : NVVM_Op<"tcgen05.ld", [NVVMRequiresSMf<[100, 101, 110]>]>
     Optional<I64>:$offset
   );
 
-  let results = (outs AnyTypeOf<[I32, VectorOfLengthAndType<
-                                  [2, 4, 8, 16, 32, 64, 128], [I32]>]>:$res);
+  let results = (outs VectorOfLengthAndType<
+                        [1, 2, 4, 8, 16, 32, 64, 128], [I32]>:$res);
 
   let assemblyFormat = [{
     $tmemAddr (`,` $offset^)? (`pack` $pack^)? attr-dict `:` type($res)
@@ -5598,11 +5598,9 @@ def NVVM_Tcgen05LdOp : NVVM_Op<"tcgen05.ld", [NVVMRequiresSMf<[100, 101, 110]>]>
     llvm::LLVMContext &Context = moduleTranslation.getLLVMContext();
     auto Pack = llvm::ConstantInt::get(Context, llvm::APInt(1, $pack));
 
-    unsigned num = $_resultType->isVectorTy()
-                       ? llvm::cast<llvm::VectorType>($_resultType)
+    unsigned num = llvm::cast<llvm::VectorType>($_resultType)
                              ->getElementCount()
-                             .getFixedValue()
-                       : 1;
+                             .getFixedValue();
 
     auto ID = getTcgen05LdIntrinsicID($shape, num);
     if (ID == llvm::Intrinsic::not_intrinsic)
@@ -5717,8 +5715,7 @@ def NVVM_Tcgen05StOp : NVVM_Op<"tcgen05.st", [NVVMRequiresSMf<[100, 101, 110]>]>
     Tcgen05LdStShapeAttr:$shape,
     // Arguments
     LLVM_PointerTensor:$tmemAddr,
-    AnyTypeOf<[I32, VectorOfLengthAndType<
-                      [2, 4, 8, 16, 32, 64, 128], [I32]>]>:$val,
+    VectorOfLengthAndType<[1, 2, 4, 8, 16, 32, 64, 128], [I32]>:$val,
     Optional<I64>:$offset
   );
 
@@ -5777,10 +5774,9 @@ def NVVM_Tcgen05StOp : NVVM_Op<"tcgen05.st", [NVVMRequiresSMf<[100, 101, 110]>]>
     auto Unpack = llvm::ConstantInt::get(Context, llvm::APInt(1, $unpack));
 
     auto valTy = $val->getType();
-    uint32_t num = valTy->isVectorTy() ? llvm::cast<llvm::VectorType>(valTy)
+    uint32_t num = llvm::cast<llvm::VectorType>(valTy)
                                              ->getElementCount()
-                                             .getFixedValue()
-                                       : 1;
+                                             .getFixedValue();
 
     auto ID = getTcgen05StIntrinsicID($shape, num);
     if (ID == llvm::Intrinsic::not_intrinsic)
diff --git a/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir b/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
index ff90ad47ba410..34f7f5b87e72b 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
@@ -83,7 +83,7 @@ gpu.module @tcgen05_cp_sm90a [#nvvm.target<chip = "sm_90a">] {
 gpu.module @tcgen05_ld_sm90 [#nvvm.target<chip = "sm_90">] {
   func.func @tcgen05_ld_sm90(%taddr: !llvm.ptr<6>) {
     // expected-error @below {{'nvvm.tcgen05.ld' op is not supported on sm_90}}
-    %0 = nvvm.tcgen05.ld %taddr {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+    %0 = nvvm.tcgen05.ld %taddr {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
     return
   }
 }
@@ -91,9 +91,9 @@ gpu.module @tcgen05_ld_sm90 [#nvvm.target<chip = "sm_90">] {
 // -----
 
 gpu.module @tcgen05_st_sm120f [#nvvm.target<chip = "sm_120f">] {
-  func.func @tcgen05_st_sm120f(%taddr: !llvm.ptr<6>, %val: i32) {
+  func.func @tcgen05_st_sm120f(%taddr: !llvm.ptr<6>, %val: vector<1 x i32>) {
     // expected-error @below {{'nvvm.tcgen05.st' op is not supported on sm_120f}}
-    nvvm.tcgen05.st %taddr, %val {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+    nvvm.tcgen05.st %taddr, %val {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
     return
   }
 }
diff --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
index b1266b0e8151d..bcd342e2e6e94 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
@@ -3,8 +3,8 @@
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b
 llvm.func @nvvm_tcgen05_ld_16x64b(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<2 x i32>
@@ -33,8 +33,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b_pack
 llvm.func @nvvm_tcgen05_ld_16x64b_pack(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<2 x i32>
@@ -165,8 +165,8 @@ llvm.func @nvvm_tcgen05_ld_16x256b_pack(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b
 llvm.func @nvvm_tcgen05_ld_32x32b(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<2 x i32>
@@ -195,8 +195,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b_pack
 llvm.func @nvvm_tcgen05_ld_32x32b_pack(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<2 x i32>
@@ -227,8 +227,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(%tmemAddr : !llvm.ptr<6>) {
 
   %halfSplitOffset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<2 x i32>
@@ -259,8 +259,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(%tmemAddr : !llvm.ptr<6>) {
 
   %halfSplitOffset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<2 x i32>
diff --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
index 119746133625d..123c6308794a6 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
@@ -3,7 +3,7 @@
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b
 llvm.func @nvvm_tcgen05_ld_16x64b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -12,8 +12,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=2:i32 } : vector<2xi32>
@@ -42,7 +42,7 @@ llvm.func @nvvm_tcgen05_ld_16x64b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b_pack
 llvm.func @nvvm_tcgen05_ld_16x64b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -51,8 +51,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b_pack(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=2:i32 } : vector<2xi32>
@@ -81,7 +81,7 @@ llvm.func @nvvm_tcgen05_ld_16x64b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x128b
 llvm.func @nvvm_tcgen05_ld_16x128b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -117,7 +117,7 @@ llvm.func @nvvm_tcgen05_ld_16x128b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x128b_pack
 llvm.func @nvvm_tcgen05_ld_16x128b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -153,7 +153,7 @@ llvm.func @nvvm_tcgen05_ld_16x128b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x256b
 llvm.func @nvvm_tcgen05_ld_16x256b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -186,7 +186,7 @@ llvm.func @nvvm_tcgen05_ld_16x256b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x256b_pack
 llvm.func @nvvm_tcgen05_ld_16x256b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -219,7 +219,7 @@ llvm.func @nvvm_tcgen05_ld_16x256b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b
 llvm.func @nvvm_tcgen05_ld_32x32b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -228,8 +228,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=2:i32 } : vector<2xi32>
@@ -258,7 +258,7 @@ llvm.func @nvvm_tcgen05_ld_32x32b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b_pack
 llvm.func @nvvm_tcgen05_ld_32x32b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -267,8 +267,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b_pack(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=2:i32 } : vector<2xi32>
@@ -297,7 +297,7 @@ llvm.func @nvvm_tcgen05_ld_32x32b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x32bx2
 llvm.func @nvvm_tcgen05_ld_16x32bx2(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -308,8 +308,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(
 
   %offset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=2:i32 } : vector<2xi32>
@@ -338,7 +338,7 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x32bx2_pack
 llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -349,8 +349,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(
 
   %offset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=2:i32 } : vector<2xi32>


From llvmlistbot at llvm.org  Thu Jul  2 04:52:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 04:52:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 5b70c3e - [mlir][vector] Add missing
 dependent dialect to TestVectorUnrollingPatterns (#207113)
Message-ID: <6a465110.af6ba3ed.2c2f30.f4d3@mx.google.com>


Author: Longsheng Mou
Date: 2026-07-02T19:52:43+08:00
New Revision: 5b70c3eeb79f634fd57ce19e21c678de2636ffa2

URL: https://github.com/llvm/llvm-project/commit/5b70c3eeb79f634fd57ce19e21c678de2636ffa2
DIFF: https://github.com/llvm/llvm-project/commit/5b70c3eeb79f634fd57ce19e21c678de2636ffa2.diff

LOG: [mlir][vector] Add missing dependent dialect to TestVectorUnrollingPatterns (#207113)

Add `vector::VectorDialect` to `getDependentDialects()` for
`TestVectorUnrollingPatterns`. Previously, `arith.addf` test passed only
because the vector dialect was indirectly loaded via other vector tests
in the same `MLIRContext`. With `split-input-file`, each test runs in an
isolated context, causing failures due to the missing vector dialect.
Fixes #206921.

Added: 
    

Modified: 
    mlir/test/Dialect/Vector/vector-unroll-options.mlir
    mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..0ec71620c5324 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -1,6 +1,6 @@
-// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type | FileCheck %s
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1"  | FileCheck %s --check-prefix=ORDER
-// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" | FileCheck %s --check-prefix=BATCHED
+// RUN: mlir-opt %s -test-vector-unrolling-patterns=unroll-based-on-type -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=2,0,1" -split-input-file | FileCheck %s --check-prefix=ORDER
+// RUN: mlir-opt %s -test-vector-unrolling-patterns="unroll-based-on-type unroll-order=0,3,1,2" -split-input-file | FileCheck %s --check-prefix=BATCHED
 
 func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
                           %init : vector<8x8xf32>) -> vector<8x8xf32> {
@@ -150,7 +150,7 @@ func.func @vector_contract_f32(%lhs : vector<8x4xf32>, %rhs : vector<8x4xf32>,
 
 //       ORDER:   return
 
-
+// -----
 
 func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
                           %init : vector<8x8xf16>) -> vector<8x8xf16> {
@@ -181,6 +181,8 @@ func.func @vector_contract_f16(%lhs : vector<8x8xf16>, %rhs : vector<8x8xf16>,
 //  CHECK-SAME:     vector<4x4xf16>, vector<4x4xf16> into vector<4x4xf16>
 //       CHECK:   return
 
+// -----
+
 func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf32>) -> vector<4x4xf32> {
   %0 = vector.fma %a, %b, %c: vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -188,6 +190,8 @@ func.func @vector_fma(%a: vector<4x4xf32>, %b: vector<4x4xf32>, %c: vector<4x4xf
 //   CHECK-LABEL: func @vector_fma
 // CHECK-COUNT-4: vector.fma %{{.+}}, %{{.+}}, %{{.+}} : vector<2x2xf32>
 
+// -----
+
 func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
   %0 = vector.fma %a, %a, %a : vector<3x2x2xf32>
   return %0 : vector<3x2x2xf32>
@@ -221,6 +225,8 @@ func.func @vector_fma_3d(%a: vector<3x2x2xf32>) -> vector<3x2x2xf32>{
 //       CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[FMA2]], %[[I1]] {offsets = [2, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<3x2x2xf32>
 //       CHECK:   return %[[I2]] : vector<3x2x2xf32>
 
+// -----
+
 func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) -> vector<4xf32> {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [1] : vector<4x6xf32> to vector<4xf32>
   return %0 : vector<4xf32>
@@ -245,6 +251,8 @@ func.func @vector_multi_reduction(%v : vector<4x6xf32>, %acc: vector<4xf32>) ->
 //       CHECK:   %[[V2:.*]] = vector.insert_strided_slice %[[R5]], %[[V1]] {offsets = [2], strides = [1]} : vector<2xf32> into vector<4xf32>
 //       CHECK:   return %[[V2]] : vector<4xf32>
 
+// -----
+
 func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32 {
   %0 = vector.multi_reduction #vector.kind<add>, %v, %acc [0, 1] : vector<4x2xf32> to f32
   return %0 : f32
@@ -257,6 +265,8 @@ func.func @vector_multi_reduction_scalar(%v: vector<4x2xf32>, %acc: f32) -> f32
 //       CHECK:   %[[R1:.*]] = vector.multi_reduction <add>, %[[S1]], %[[R0]] [0, 1] : vector<2x2xf32> to f32
 //       CHECK:   return %[[R1]] : f32
 
+// -----
+
 func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
   %0 = vector.reduction <add>, %v : vector<8xf32> into f32
   return %0 : f32
@@ -276,6 +286,8 @@ func.func @vector_reduction(%v : vector<8xf32>) -> f32 {
 //       CHECK:   %[[add3:.*]] = arith.addf %[[add2]], %[[r3]]
 //       CHECK:   return %[[add3]]
 
+// -----
+
 func.func @vector_transpose(%v : vector<2x4x3x8xf32>) -> vector<2x3x8x4xf32> {
   %t = vector.transpose %v, [0, 2, 3, 1] : vector<2x4x3x8xf32> to vector<2x3x8x4xf32>
   return %t : vector<2x3x8x4xf32>
@@ -337,6 +349,7 @@ func.func @vector_contract_batched(%lhs: vector<8x8x4xf32>, %rhs: vector<8x8x4xf
 //      BATCHED-NOT: vector.contract
 //          BATCHED: return
 
+// -----
 
 func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4xf32> to vector<4x4xf32>
@@ -360,6 +373,8 @@ func.func @vector_broadcast(%v: vector<4xf32>) -> vector<4x4xf32> {
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<1x4xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -382,6 +397,8 @@ func.func @vector_broadcast_with_leading_unit_dim(%v: vector<1x4xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
+
 func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector<4x4xf32> {
   %0 = vector.broadcast %v : vector<4x1xf32> to vector<4x4xf32>
   return %0 : vector<4x4xf32>
@@ -404,6 +421,7 @@ func.func @vector_broadcast_with_tailing_unit_dim(%v: vector<4x1xf32>) -> vector
 //       CHECK: [[r3:%.+]] = vector.insert_strided_slice [[b3]], [[r2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 //       CHECK: return [[r3]] : vector<4x4xf32>
 
+// -----
 
 func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   %c0 = arith.constant 0 : index
@@ -426,6 +444,7 @@ func.func @vector_load_2D(%mem: memref<4x4xf16>) -> vector<4x4xf16> {
   // CHECK: %[[V7:.*]] = vector.insert_strided_slice %[[V6]], %[[V5]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf16> into vector<4x4xf16>
   // CHECK: return %[[V7]] : vector<4x4xf16>
 
+// -----
 
 func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   %c0 = arith.constant 0 : index
@@ -446,6 +465,7 @@ func.func @vector_store_2D(%mem: memref<4x4xf16>, %v: vector<4x4xf16>) {
   // CHECK: %[[V3:.*]] = vector.extract_strided_slice %[[ARG1]] {offsets = [2, 2], sizes = [2, 2], strides = [1, 1]} : vector<4x4xf16> to vector<2x2xf16>
   // CHECK: vector.store %[[V3]], %[[ARG0]][%[[C2]], %[[C2]]] : memref<4x4xf16>, vector<2x2xf16>
 
+// -----
 
 func.func @vector_step() -> vector<32xindex> {
     %0 = vector.step : vector<32xindex>
@@ -466,6 +486,7 @@ func.func @vector_step() -> vector<32xindex> {
 // CHECK: %[[INS3:.*]] = vector.insert_strided_slice %[[ADD3]], %[[INS2]] {offsets = [24], strides = [1]} : vector<8xindex> into vector<32xindex>
 // CHECK: return %[[INS3]] : vector<32xindex>
 
+// -----
 
 func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>) -> vector<2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2xf32>
@@ -488,6 +509,7 @@ func.func @elementwise_3D_to_2D(%v1: vector<2x2x2xf32>, %v2: vector<2x2x2xf32>)
 //       CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[ADD1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<2x2x2xf32>
 //       CHECK:   return %[[I1]] : vector<2x2x2xf32>
 
+// -----
 
 func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf32>) -> vector<2x2x2x2xf32> {
   %0 = arith.addf %v1, %v2 : vector<2x2x2x2xf32>
@@ -499,6 +521,8 @@ func.func @elementwise_4D_to_2D(%v1: vector<2x2x2x2xf32>, %v2: vector<2x2x2x2xf3
 // CHECK-NOT: arith.addf
 // CHECK: return
 
+// -----
+
 func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1> {
   %0 = vector.create_mask %size1, %size2 : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -539,6 +563,8 @@ func.func @vector_create_mask(%size1: index, %size2: index) -> vector<16x16xi1>
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
   %cst16 = arith.constant 16 : index
   %0 = vector.create_mask %cst16, %cst16 : vector<16x16xi1>
@@ -554,6 +580,8 @@ func.func @vector_create_mask_constant_dim_sizes() -> vector<16x16xi1> {
 // CHECK:   %[[S3:.*]] = vector.insert_strided_slice %[[CST_0]], %[[S2]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 // CHECK:   return %[[S3]] : vector<16x16xi1>
 
+// -----
+
 func.func @vector_constant_mask() -> vector<16x16xi1> {
   %0 = vector.constant_mask [12, 10] : vector<16x16xi1>
   return %0 : vector<16x16xi1>
@@ -572,6 +600,8 @@ func.func @vector_constant_mask() -> vector<16x16xi1> {
 //       CHECK:   %[[INS11:.*]] = vector.insert_strided_slice %[[MASK11]], %[[INS10]] {offsets = [8, 8], strides = [1, 1]} : vector<8x8xi1> into vector<16x16xi1>
 //       CHECK:   return %[[INS11]] : vector<16x16xi1>
 
+// -----
+
 func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
   %0 = vector.shape_cast %v : vector<16xf32> to vector<2x2x4xf32>
   return %0 : vector<2x2x4xf32>
@@ -588,6 +618,7 @@ func.func @shape_cast_1D(%v: vector<16xf32>) -> vector<2x2x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0, 0], strides = [1, 1]} : vector<2x4xf32> into vector<2x2x4xf32>
 // CHECK:   return %[[I1]] : vector<2x2x4xf32>
 
+// -----
 
 func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
   %0 = vector.shape_cast %v : vector<8x2xf32> to vector<4x4xf32>
@@ -605,6 +636,7 @@ func.func @shape_cast_2D(%v: vector<8x2xf32>) -> vector<4x4xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [2, 0], strides = [1, 1]} : vector<2x4xf32> into vector<4x4xf32>
 // CHECK:   return %[[I1]] : vector<4x4xf32>
 
+// -----
 
 // This is a negative test case to ensure that such shape casts are not unrolled
 // because the targetShape (2x4) is not contiguous in result vector
@@ -618,6 +650,7 @@ func.func @negative_shape_cast_target_shape_not_contiguous(%v: vector<64xf32>) -
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<64xf32> to vector<8x8xf32>
 // CHECK:   return %[[SC]] : vector<8x8xf32>
 
+// -----
 
 // This is negative test case to ensure that such shape casts are not unrolled
 // because it cannot determine the extractShape from source vector (8x3)
@@ -632,6 +665,7 @@ func.func @negative_shape_cast_source_shape_not_determinable(%v: vector<8x3xf32>
 // CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<8x3xf32> to vector<6x4xf32>
 // CHECK:   return %[[SC]] : vector<6x4xf32>
 
+// -----
 
 // TargetShape is [1x16]
 func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
@@ -650,6 +684,7 @@ func.func @shape_cast_leading_unit_dim(%v: vector<32xf32>) -> vector<1x32xf32> {
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 16], strides = [1, 1]} : vector<1x16xf32> into vector<1x32xf32>
 // CHECK:   return %[[I1]] : vector<1x32xf32>
 
+// -----
 
 // TargetShape is [1x1]
 func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x1xf32> {

diff  --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..1a4bef664fbe1 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -150,6 +150,11 @@ struct TestVectorUnrollingPatterns
     return "Test lowering patterns to unroll contract ops in the vector "
            "dialect";
   }
+
+  void getDependentDialects(DialectRegistry &registry) const override {
+    registry.insert<vector::VectorDialect>();
+  }
+
   TestVectorUnrollingPatterns() = default;
   TestVectorUnrollingPatterns(const TestVectorUnrollingPatterns &pass)
       : PassWrapper(pass) {}


        

From llvmlistbot at llvm.org  Thu Jul  2 04:52:51 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Thu, 02 Jul 2026 04:52:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Add missing dependent dialect
 to TestVectorUnrollingPatterns (PR #207113)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207113@github.com>
Message-ID: <6a465113.d5797644.146403.36b7@mx.google.com>


https://github.com/CoTinker closed https://github.com/llvm/llvm-project/pull/207113

From llvmlistbot at llvm.org  Thu Jul  2 04:56:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 04:56:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Move GET_ATTR/TYPEDEF_CLASSES after
	declarations (PR #171444)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/171444@github.com>
Message-ID: <6a4651f2.f8b79514.3b441.3b8c@mx.google.com>


https://github.com/Harald-R closed https://github.com/llvm/llvm-project/pull/171444

From llvmlistbot at llvm.org  Thu Jul  2 04:56:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 04:56:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Move GET_ATTR/TYPEDEF_CLASSES after
	declarations (PR #171444)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/171444@github.com>
Message-ID: <6a4651f2.27b3b003.1610ae.5640@mx.google.com>


Harald-R wrote:

Closing due to lack of interest.

https://github.com/llvm/llvm-project/pull/171444

From llvmlistbot at llvm.org  Thu Jul  2 06:22:33 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Thu, 02 Jul 2026 06:22:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fold vector.to_elements from
	constants (PR #207098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207098@github.com>
Message-ID: <6a466619.f8b79514.3b441.389e@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/207098

From llvmlistbot at llvm.org  Thu Jul  2 06:22:39 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Thu, 02 Jul 2026 06:22:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fold vector.to_elements from
	constants (PR #207098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207098@github.com>
Message-ID: <6a46661f.491d90f5.1f0d58.b142@mx.google.com>


https://github.com/matthias-springer edited https://github.com/llvm/llvm-project/pull/207098

From llvmlistbot at llvm.org  Thu Jul  2 06:25:29 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 02 Jul 2026 06:25:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Builtin] Fix crash folding
 self-referential unrealized_conversion_cast in graph regions (PR #207185)
Message-ID: <llvm.org/llvm/llvm-project/pull/207185@github.com>


https://github.com/anutosh491 created https://github.com/llvm/llvm-project/pull/207185

Fix #205064

Graph regions permit use-def cycles. When an `unrealized_conversion_cast`uses its own result as an operand and the input/output types match, the fold returned success with the operand (= the result itself) as the replacement value. `replaceOp` then called r`eplaceAllUsesWith(%0, %0)` (a no-op) followed by eraseOp, which asserted `use_empty()` because the self-use was still live.

Fix by bailing out of the fold when any operand is produced by this op itself. The greedy rewriter then correctly DCEs the op by dropping the self-use before erasing. As a result we have the following 
```
 anutosh491 at Anutoshs-MacBook-Air mlir-build % cat bin/a.mlir                                     
module {
  test.graph_region {
    %0 = builtin.unrealized_conversion_cast %0 : i32 to i32
    "test.return"() : () -> ()
  }
}%
                                                                                                                                          
 anutosh491 at Anutoshs-MacBook-Air mlir-build % ./bin/mlir-opt -canonicalize bin/a.mlir            
module {
  test.graph_region {
    "test.return"() : () -> ()
  }
}

 anutosh491 at Anutoshs-MacBook-Air mlir-build % ./bin/llvm-lit ../mlir/test/Dialect/Builtin/canonicalize.mlir -v
-- Testing: 1 tests, 1 workers --
PASS: MLIR :: Dialect/Builtin/canonicalize.mlir (1 of 1)

Testing Time: 0.12s

Total Discovered Tests: 1
  Passed: 1 (100.00%)
```

>From 9c7000d2fcf1d424823b58e97adb45cfbe1fca7a Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Thu, 2 Jul 2026 18:50:24 +0530
Subject: [PATCH] [mlir][Builtin] Fix crash folding self-referential
 unrealized_conversion_cast

---
 mlir/lib/IR/BuiltinDialect.cpp              |  5 +++++
 mlir/test/Dialect/Builtin/canonicalize.mlir | 11 +++++++++++
 2 files changed, 16 insertions(+)

diff --git a/mlir/lib/IR/BuiltinDialect.cpp b/mlir/lib/IR/BuiltinDialect.cpp
index c88b328282275..00cad06122dd8 100644
--- a/mlir/lib/IR/BuiltinDialect.cpp
+++ b/mlir/lib/IR/BuiltinDialect.cpp
@@ -201,6 +201,11 @@ UnrealizedConversionCastOp::fold(FoldAdaptor adaptor,
   ResultRange results = getOutputs();
 
   if (operands.getType() == results.getType()) {
+    // Skip if any operand is this op's own result (self-referential cycle).
+    if (llvm::any_of(operands, [this](Value v) {
+          return v.getDefiningOp() == getOperation();
+        }))
+      return failure();
     foldResults.append(operands.begin(), operands.end());
     return success();
   }
diff --git a/mlir/test/Dialect/Builtin/canonicalize.mlir b/mlir/test/Dialect/Builtin/canonicalize.mlir
index 2e36b7ee371c3..537868e990d78 100644
--- a/mlir/test/Dialect/Builtin/canonicalize.mlir
+++ b/mlir/test/Dialect/Builtin/canonicalize.mlir
@@ -23,3 +23,14 @@ func.func @multiple_conversion_casts_failure(%arg0: i32, %arg1: i32, %arg2: i64)
   %outputs:2 = builtin.unrealized_conversion_cast %arg2, %inputs#1 : i64, i64 to i32, i32
   return %outputs#0, %outputs#1 : i32, i32
 }
+
+// Test a self-referential cast in a graph region must not crash.
+// CHECK-LABEL: func @self_referential_cast
+// CHECK-NOT: unrealized_conversion_cast
+func.func @self_referential_cast() {
+  test.graph_region {
+    %0 = builtin.unrealized_conversion_cast %0 : i32 to i32
+    "test.return"() : () -> ()
+  }
+  return
+}


From llvmlistbot at llvm.org  Thu Jul  2 06:28:36 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 02 Jul 2026 06:28:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Builtin] Fix crash folding
 self-referential unrealized_conversion_cast in graph regions (PR #207185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207185@github.com>
Message-ID: <6a466784.eb24c6ea.1f2e26.4d6d@mx.google.com>


================
@@ -201,6 +201,11 @@ UnrealizedConversionCastOp::fold(FoldAdaptor adaptor,
   ResultRange results = getOutputs();
 
   if (operands.getType() == results.getType()) {
+    // Skip if any operand is this op's own result (self-referential cycle).
----------------
anutosh491 wrote:

Added a comment here just in case.

https://github.com/llvm/llvm-project/pull/207185

From llvmlistbot at llvm.org  Thu Jul  2 06:33:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 06:33:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Builtin] Fix crash folding
 self-referential unrealized_conversion_cast in graph regions (PR #207185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207185@github.com>
Message-ID: <6a46689e.6cfa33d2.19518c.91ab@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Anutosh Bhat (anutosh491)

<details>
<summary>Changes</summary>

Fix #<!-- -->205064

Graph regions permit use-def cycles. When an `unrealized_conversion_cast`uses its own result as an operand and the input/output types match, the fold returned success with the operand (= the result itself) as the replacement value. `replaceOp` then called `replaceAllUsesWith(%0, %0)` (a no-op) followed by `eraseOp`, which asserted `use_empty()` because the self-use was still live.

This can be fixed by bailing out of the fold when any operand is produced by this op itself. The greedy rewriter then correctly DCEs the op by dropping the self-use before erasing. As a result we have the following 
```
 anutosh491@<!-- -->Anutoshs-MacBook-Air mlir-build % cat bin/a.mlir                                     
module {
  test.graph_region {
    %0 = builtin.unrealized_conversion_cast %0 : i32 to i32
    "test.return"() : () -&gt; ()
  }
}%
                                                                                                                                          
 anutosh491@<!-- -->Anutoshs-MacBook-Air mlir-build % ./bin/mlir-opt -canonicalize bin/a.mlir            
module {
  test.graph_region {
    "test.return"() : () -&gt; ()
  }
}

 anutosh491@<!-- -->Anutoshs-MacBook-Air mlir-build % ./bin/llvm-lit ../mlir/test/Dialect/Builtin/canonicalize.mlir -v
-- Testing: 1 tests, 1 workers --
PASS: MLIR :: Dialect/Builtin/canonicalize.mlir (1 of 1)

Testing Time: 0.12s

Total Discovered Tests: 1
  Passed: 1 (100.00%)
```

---
Full diff: https://github.com/llvm/llvm-project/pull/207185.diff


2 Files Affected:

- (modified) mlir/lib/IR/BuiltinDialect.cpp (+5) 
- (modified) mlir/test/Dialect/Builtin/canonicalize.mlir (+11) 


``````````diff
diff --git a/mlir/lib/IR/BuiltinDialect.cpp b/mlir/lib/IR/BuiltinDialect.cpp
index c88b328282275..00cad06122dd8 100644
--- a/mlir/lib/IR/BuiltinDialect.cpp
+++ b/mlir/lib/IR/BuiltinDialect.cpp
@@ -201,6 +201,11 @@ UnrealizedConversionCastOp::fold(FoldAdaptor adaptor,
   ResultRange results = getOutputs();
 
   if (operands.getType() == results.getType()) {
+    // Skip if any operand is this op's own result (self-referential cycle).
+    if (llvm::any_of(operands, [this](Value v) {
+          return v.getDefiningOp() == getOperation();
+        }))
+      return failure();
     foldResults.append(operands.begin(), operands.end());
     return success();
   }
diff --git a/mlir/test/Dialect/Builtin/canonicalize.mlir b/mlir/test/Dialect/Builtin/canonicalize.mlir
index 2e36b7ee371c3..537868e990d78 100644
--- a/mlir/test/Dialect/Builtin/canonicalize.mlir
+++ b/mlir/test/Dialect/Builtin/canonicalize.mlir
@@ -23,3 +23,14 @@ func.func @multiple_conversion_casts_failure(%arg0: i32, %arg1: i32, %arg2: i64)
   %outputs:2 = builtin.unrealized_conversion_cast %arg2, %inputs#1 : i64, i64 to i32, i32
   return %outputs#0, %outputs#1 : i32, i32
 }
+
+// Test a self-referential cast in a graph region must not crash.
+// CHECK-LABEL: func @self_referential_cast
+// CHECK-NOT: unrealized_conversion_cast
+func.func @self_referential_cast() {
+  test.graph_region {
+    %0 = builtin.unrealized_conversion_cast %0 : i32 to i32
+    "test.return"() : () -> ()
+  }
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207185

From llvmlistbot at llvm.org  Thu Jul  2 06:35:13 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Thu, 02 Jul 2026 06:35:13 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a466911.7d41f3a1.d1cf7.ddd0@mx.google.com>


joker-eph wrote:

> While that pass works, it is admittedly somewhat fragile. In my opinion, this is because we never fully settled whether it should support arbitrary integer widths such as i3.

Seems to me that this goes beyond the pass itself, it's really a semantic question about whether:
`  %1 = vector.bitcast %0 : vector<16xi3> to vector<3xi16>` is well-formed or not (that is: this is a verifier issue). That is what is the layout of vector including subbyte types.


https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Thu Jul  2 06:36:03 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 02 Jul 2026 06:36:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Builtin] Fix crash folding
 self-referential unrealized_conversion_cast in graph regions (PR #207185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207185@github.com>
Message-ID: <6a466943.cb30c3e4.6f38b.76fe@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/207185

>From 92a760069ab67b5d4fdaab65f711aff53619573b Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Thu, 2 Jul 2026 18:50:24 +0530
Subject: [PATCH] [mlir][Builtin] Fix crash folding self-referential
 unrealized_conversion_cast

---
 mlir/lib/IR/BuiltinDialect.cpp              |  5 +++++
 mlir/test/Dialect/Builtin/canonicalize.mlir | 11 +++++++++++
 2 files changed, 16 insertions(+)

diff --git a/mlir/lib/IR/BuiltinDialect.cpp b/mlir/lib/IR/BuiltinDialect.cpp
index c88b328282275..00cad06122dd8 100644
--- a/mlir/lib/IR/BuiltinDialect.cpp
+++ b/mlir/lib/IR/BuiltinDialect.cpp
@@ -201,6 +201,11 @@ UnrealizedConversionCastOp::fold(FoldAdaptor adaptor,
   ResultRange results = getOutputs();
 
   if (operands.getType() == results.getType()) {
+    // Skip if any operand is this op's own result (self-referential cycle).
+    if (llvm::any_of(operands, [this](Value v) {
+          return v.getDefiningOp() == getOperation();
+        }))
+      return failure();
     foldResults.append(operands.begin(), operands.end());
     return success();
   }
diff --git a/mlir/test/Dialect/Builtin/canonicalize.mlir b/mlir/test/Dialect/Builtin/canonicalize.mlir
index 2e36b7ee371c3..537868e990d78 100644
--- a/mlir/test/Dialect/Builtin/canonicalize.mlir
+++ b/mlir/test/Dialect/Builtin/canonicalize.mlir
@@ -23,3 +23,14 @@ func.func @multiple_conversion_casts_failure(%arg0: i32, %arg1: i32, %arg2: i64)
   %outputs:2 = builtin.unrealized_conversion_cast %arg2, %inputs#1 : i64, i64 to i32, i32
   return %outputs#0, %outputs#1 : i32, i32
 }
+
+// Test a self-referential cast in a graph region must not crash.
+// CHECK-LABEL: func @self_referential_cast
+// CHECK-NOT: unrealized_conversion_cast
+func.func @self_referential_cast() {
+  test.graph_region {
+    %0 = builtin.unrealized_conversion_cast %0 : i32 to i32
+    "test.return"() : () -> ()
+  }
+  return
+}


From llvmlistbot at llvm.org  Thu Jul  2 06:55:45 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Thu, 02 Jul 2026 06:55:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a466de1.eb24c6ea.1f2e26.eaec@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/11] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/11] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/11] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/11] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/11] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/11] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/11] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/11] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/11] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/11] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 122569519584840f9dcd19e73a954fbc049718e9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/11] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Thu Jul  2 07:00:09 2026
From: llvmlistbot at llvm.org (Dmitry Chigarev)
Date: Thu, 02 Jul 2026 07:00:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enable end-to-end integration
 test compilation (PR #207158)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207158@github.com>
Message-ID: <6a466ee9.1a4df1a5.118deb.026c@mx.google.com>


https://github.com/dchigarev approved this pull request.


https://github.com/llvm/llvm-project/pull/207158

From llvmlistbot at llvm.org  Thu Jul  2 07:35:15 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Thu, 02 Jul 2026 07:35:15 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a467723.3157459c.24b6e5.7ea6@mx.google.com>


https://github.com/Meinersbur edited https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Thu Jul  2 07:35:16 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Thu, 02 Jul 2026 07:35:16 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a467724.eb24c6ea.1f2e26.b100@mx.google.com>


https://github.com/Meinersbur commented:

[critical] With this change, there will be an hard error with metadata with the old format. We will need the AutoUpgrade otherwise older .bc files will be rejected, violating the IR backwards compatibility guarantee

> The current LLVM version supports loading any bitcode since version 3.0.

> Non-debug metadata is defined to be safe to drop, so a valid way to upgrade it is to drop it. That is not very user friendly and a bit more effort is expected, but no promises are made.

https://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Thu Jul  2 07:35:16 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Thu, 02 Jul 2026 07:35:16 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a467724.f06f4994.1e3a60.4388@mx.google.com>


================

----------------
Meinersbur wrote:

```suggestion
; RUN: cat %s > %t
```
is the established pattern

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Thu Jul  2 07:35:16 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Thu, 02 Jul 2026 07:35:16 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a467724.f14cf1a5.2c3aaa.b94d@mx.google.com>


================

----------------
Meinersbur wrote:

`opt -passes=verify` runs the verifier multiple times -- before the pass, the pass itself (which is the verifier), after the pass.

However, it seems to be ane established pattern in other verifier tests for some reason, but we should avoid unnecessary work. Please change. 

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Thu Jul  2 07:35:16 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Thu, 02 Jul 2026 07:35:16 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a467724.491d90f5.1f0d58.9bd5@mx.google.com>


================

----------------
Meinersbur wrote:

The point of this PR is to have a common format for boolean named metdata. disable/enable is used by other loop passes and `enable=0` to switch off was found confusing (could be interpreted as "not explicitly enable" instead of "disable" which it actually means.)

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.f732ad81.f30e7.3092@mx.google.com>


https://github.com/chbessonova edited https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.510f576d.185953.9712@mx.google.com>


================
@@ -546,85 +546,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
----------------
chbessonova wrote:

Consider replacing with a loop. Filter express intent better when no side effects involved, here we simultaneosly filter and fill SPToEntities which is misleading.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.1a4df1a5.118deb.27a9@mx.google.com>


================
@@ -3236,9 +3240,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
----------------
chbessonova wrote:

Same as above: the temporary is redundant.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.a151105c.30c1b9.bab1@mx.google.com>


https://github.com/chbessonova commented:

Thanks, Vladislav, for driving this work forward and for all the effort you've put into these activities! It's great to see this patch getting close to the finish line.

I've reviewed the changes and left a few comments and suggestions that may help improve readability, but overall this looks good to me. Thanks again for taking this on and pushing it through.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.6c8668f2.74a35.2ace@mx.google.com>


================
@@ -2124,6 +2124,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
----------------
chbessonova wrote:

I'd rather not do this, it looks misleading to me. It's not clear what the purpose of these globals is or which metadata kind they belong to. It also feels inconsistent even within DICompileUnit, since, for example, retainedTypes does not have a corresponding operand number global. Given that operand numbers are still hardcoded explicitly in other places, I'd prefer to leave these three as they were.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:36:59 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:36:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778b.492e23e4.e71ca.a383@mx.google.com>


================
@@ -1553,6 +1553,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
----------------
chbessonova wrote:

The predicate usually carries what elements to retain, see https://llvm.org/doxygen/classllvm_1_1filter__iterator__base.html

The logic of the function can be simplified with filtering iterator or filtering range use.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 07:37:00 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Thu, 02 Jul 2026 07:37:00 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a46778c.01f0a1da.2016e9.ee0a@mx.google.com>


================
@@ -3236,9 +3240,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
----------------
chbessonova wrote:

```
if (!GlobalMap.count(GVE) && DIE->isConstant())
  GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
```

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Thu Jul  2 08:42:04 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Thu, 02 Jul 2026 08:42:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Use `SymbolUserMap`
 to avoid quadratic symbol lookups (PR #205448)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205448@github.com>
Message-ID: <6a4686cc.576b3af0.2cb9b7.d678@mx.google.com>


victor-eds wrote:

Thanks for the review! @joker-eph, can you also land? I still haven't recovered my land privilege

https://github.com/llvm/llvm-project/pull/205448

From llvmlistbot at llvm.org  Thu Jul  2 08:47:25 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Thu, 02 Jul 2026 08:47:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a46880d.f732ad81.f30e7.37cf@mx.google.com>


https://github.com/tblah commented:

Thanks for the patch. I think this needs:

  1. A `mlir/test/Dialect/OpenMP/` test for `omp.unroll_partial` parser/printer/verifier coverage, analogous to the existing `cli-unroll-
  heuristic.mlir` coverage. This should cover the custom assembly form and invalid `unroll_factor` cases.

  2. A `mlir/test/Target/LLVMIR/` test for lowering `omp.unroll_partial` through `mlir-translate -mlir-to-llvmir`, analogous to the existing
  `openmp-cli-unroll-heuristic*.mlir` tests. The current Flang test only checks that lowering emits the MLIR op; it does not exercise the
  new LLVM IR translation path.


https://github.com/llvm/llvm-project/pull/206642

From llvmlistbot at llvm.org  Thu Jul  2 09:24:55 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Thu, 02 Jul 2026 09:24:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#207221)
Message-ID: <llvm.org/llvm/llvm-project/pull/207221@github.com>


https://github.com/smada3 created https://github.com/llvm/llvm-project/pull/207221

MmaOp::getIntrinsicID can trigger an uninitialized read with valgrind enabled: intOverflowBehavior is optional, so when it's omitted it can be read without being initialized. It is set to a default value, which follows the PTX spec / existing lowering. Documentation updated to not be stale.

>From 603c5478a8b83d5e040cde10839be5663203d460 Mon Sep 17 00:00:00 2001
From: Stefan Mada <smada at nvidia.com>
Date: Tue, 30 Jun 2026 23:29:21 +0000
Subject: [PATCH] Fixed valgrind leak in MMAOp

---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  8 ++++----
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  | 12 +++---------
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..553219e5ed382 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3384,9 +3384,9 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     `intOverflowBehavior` is only relevant when the `multiplicandType` attribute
     is one of `u8, s8, u4, s4`, this attribute describes how overflow is handled
     in the accumulator. When the attribute is `satfinite`, the accumulator values
-    are clamped in the int32 range on overflow. This is the default behavior.
-    Alternatively, accumulator behavior `wrapped` can also be specified, in
-    which case overflow wraps from one end of the range to the other.
+    are clamped in the int32 range on overflow. Alternatively, accumulator
+    behavior `wrapped` can be specified (this is the default), in which case
+    overflow wraps from one end of the range to the other.
 
     `layoutA` and `layoutB` are required and should generally be set to
     `#nvvm.mma_layout<row>` and `#nvvm.mma_layout<col>` respectively, but other
@@ -3431,7 +3431,7 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
   let results = (outs LLVM_AnyStruct:$res);
   let arguments = (ins NVVM_MMAShapeAttr:$shape,
              OptionalAttr<MMAB1OpAttr>:$b1Op,
-             OptionalAttr<MMAIntOverflowAttr>:$intOverflowBehavior,
+             DefaultValuedAttr<MMAIntOverflowAttr, "MMAIntOverflow::wrapped">:$intOverflowBehavior,
              MMALayoutAttr:$layoutA,
              MMALayoutAttr:$layoutB,
              OptionalAttr<MMATypesAttr>:$multiplicandAPtxType,
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..d9ef70dc1c12d 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1129,15 +1129,9 @@ LogicalResult MmaOp::verify() {
                        " attribute");
   }
 
-  // Ensure int4/int8 MMA variants specify the accum overflow behavior
-  // attribute.
-  if (isInt4PtxType(*getMultiplicandAPtxType()) ||
-      isInt8PtxType(*getMultiplicandAPtxType())) {
-    if (!getIntOverflowBehavior())
-      return emitOpError("op requires " +
-                         getIntOverflowBehaviorAttrName().strref() +
-                         " attribute");
-  }
+  // `intOverflowBehavior` is a default-valued attribute (`wrapped`), so it
+  // always carries a value; no presence check is required for int4/int8
+  // variants.
 
   // Validate layout combinations. According to the operation description, most
   // MMA operations require layoutA=row and layoutB=col. Only m8n8k4 with f16


From llvmlistbot at llvm.org  Thu Jul  2 09:25:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 09:25:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#207221)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207221@github.com>
Message-ID: <6a4690fe.f6bcba31.bd69d.1339@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Stefan Mada (smada3)

<details>
<summary>Changes</summary>

MmaOp::getIntrinsicID can trigger an uninitialized read with valgrind enabled: intOverflowBehavior is optional, so when it's omitted it can be read without being initialized. It is set to a default value, which follows the PTX spec / existing lowering. Documentation updated to not be stale.

---
Full diff: https://github.com/llvm/llvm-project/pull/207221.diff


2 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+4-4) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+3-9) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..553219e5ed382 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3384,9 +3384,9 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     `intOverflowBehavior` is only relevant when the `multiplicandType` attribute
     is one of `u8, s8, u4, s4`, this attribute describes how overflow is handled
     in the accumulator. When the attribute is `satfinite`, the accumulator values
-    are clamped in the int32 range on overflow. This is the default behavior.
-    Alternatively, accumulator behavior `wrapped` can also be specified, in
-    which case overflow wraps from one end of the range to the other.
+    are clamped in the int32 range on overflow. Alternatively, accumulator
+    behavior `wrapped` can be specified (this is the default), in which case
+    overflow wraps from one end of the range to the other.
 
     `layoutA` and `layoutB` are required and should generally be set to
     `#nvvm.mma_layout<row>` and `#nvvm.mma_layout<col>` respectively, but other
@@ -3431,7 +3431,7 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
   let results = (outs LLVM_AnyStruct:$res);
   let arguments = (ins NVVM_MMAShapeAttr:$shape,
              OptionalAttr<MMAB1OpAttr>:$b1Op,
-             OptionalAttr<MMAIntOverflowAttr>:$intOverflowBehavior,
+             DefaultValuedAttr<MMAIntOverflowAttr, "MMAIntOverflow::wrapped">:$intOverflowBehavior,
              MMALayoutAttr:$layoutA,
              MMALayoutAttr:$layoutB,
              OptionalAttr<MMATypesAttr>:$multiplicandAPtxType,
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..d9ef70dc1c12d 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1129,15 +1129,9 @@ LogicalResult MmaOp::verify() {
                        " attribute");
   }
 
-  // Ensure int4/int8 MMA variants specify the accum overflow behavior
-  // attribute.
-  if (isInt4PtxType(*getMultiplicandAPtxType()) ||
-      isInt8PtxType(*getMultiplicandAPtxType())) {
-    if (!getIntOverflowBehavior())
-      return emitOpError("op requires " +
-                         getIntOverflowBehaviorAttrName().strref() +
-                         " attribute");
-  }
+  // `intOverflowBehavior` is a default-valued attribute (`wrapped`), so it
+  // always carries a value; no presence check is required for int4/int8
+  // variants.
 
   // Validate layout combinations. According to the operation description, most
   // MMA operations require layoutA=row and layoutB=col. Only m8n8k4 with f16

``````````

</details>


https://github.com/llvm/llvm-project/pull/207221

From llvmlistbot at llvm.org  Thu Jul  2 09:38:11 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Thu, 02 Jul 2026 09:38:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#207221)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207221@github.com>
Message-ID: <6a4693f3.ff037943.3bf0e2.eee3@mx.google.com>


https://github.com/smada3 closed https://github.com/llvm/llvm-project/pull/207221

From llvmlistbot at llvm.org  Thu Jul  2 09:41:27 2026
From: llvmlistbot at llvm.org (Ivan Ho)
Date: Thu, 02 Jul 2026 09:41:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][IRDL] Instantiate "." in IRDL-defined
 names as nested namespaces (PR #207035)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207035@github.com>
Message-ID: <6a4694b7.abb10101.208f3e.3f9d@mx.google.com>


https://github.com/hhkit updated https://github.com/llvm/llvm-project/pull/207035

>From 01d8081c6fd6155074c7874733fb703931fbcad0 Mon Sep 17 00:00:00 2001
From: Ivan Ho <38537881+hhkit at users.noreply.github.com>
Date: Wed, 1 Jul 2026 18:02:15 +0100
Subject: [PATCH 1/2] unit tests for the changes we want to make

---
 .../TestIRDLToCpp/test_irdl_to_cpp.irdl.mlir  | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/mlir/test/lib/Dialect/TestIRDLToCpp/test_irdl_to_cpp.irdl.mlir b/mlir/test/lib/Dialect/TestIRDLToCpp/test_irdl_to_cpp.irdl.mlir
index 770cd36cdee33..2a37cfaa3557c 100644
--- a/mlir/test/lib/Dialect/TestIRDLToCpp/test_irdl_to_cpp.irdl.mlir
+++ b/mlir/test/lib/Dialect/TestIRDLToCpp/test_irdl_to_cpp.irdl.mlir
@@ -18,6 +18,32 @@ irdl.dialect @test_irdl_to_cpp {
         irdl.results(res: %0)
     }
 
+    // CHECK: // ::mlir::test_irdl_to_cpp::nested::NamespacedOp declarations
+    // CHECK: namespace nested {
+    // CHECK: class NamespacedOp : public ::mlir::Op<NamespacedOp
+    // CHECK: static constexpr ::llvm::StringLiteral getOperationName()
+    // CHECK-NEXT: return ::llvm::StringLiteral("test_irdl_to_cpp.nested.namespaced");
+    // CHECK: } // namespace nested
+    // CHECK: MLIR_DECLARE_EXPLICIT_TYPE_ID(::mlir::test_irdl_to_cpp::nested::NamespacedOp)
+    irdl.operation @nested.namespaced {
+        %0 = irdl.any
+        irdl.results(res: %0)
+    }
+
+    // CHECK: // ::mlir::test_irdl_to_cpp::nested::namespaced::MoreOp declarations
+    // CHECK: namespace nested {
+    // CHECK: namespace namespaced {
+    // CHECK: class MoreOp : public ::mlir::Op<MoreOp
+    // CHECK: static constexpr ::llvm::StringLiteral getOperationName()
+    // CHECK-NEXT: return ::llvm::StringLiteral("test_irdl_to_cpp.nested.namespaced.more");
+    // CHECK: } // namespace namespaced
+    // CHECK: } // namespace nested
+    // CHECK: MLIR_DECLARE_EXPLICIT_TYPE_ID(::mlir::test_irdl_to_cpp::nested::namespaced::MoreOp)
+    irdl.operation @nested.namespaced.more {
+        %0 = irdl.any
+        irdl.results(res: %0)
+    }
+
     // CHECK: class BeefOp
     // CHECK: ::mlir::Value getLhs()
     // CHECK: ::mlir::Value getRhs()
@@ -49,6 +75,20 @@ irdl.dialect @test_irdl_to_cpp {
     // CHECK:  ::mlir::Region &getThen() { return (*this)->getRegion(1); }
     // CHECK:  ::mlir::Region &getElse() { return (*this)->getRegion(2); }
 
+    // CHECK: // ::mlir::test_irdl_to_cpp::nested::NamespacedOp definitions
+    // CHECK: namespace nested {
+    // CHECK: NamespacedOp::build
+    // CHECK: } // namespace nested
+    // CHECK: MLIR_DEFINE_EXPLICIT_TYPE_ID(::mlir::test_irdl_to_cpp::nested::NamespacedOp)
+
+    // CHECK: // ::mlir::test_irdl_to_cpp::nested::namespaced::MoreOp definitions
+    // CHECK: namespace nested {
+    // CHECK: namespace namespaced {
+    // CHECK: MoreOp::build
+    // CHECK: } // namespace namespaced
+    // CHECK: } // namespace nested
+    // CHECK: MLIR_DEFINE_EXPLICIT_TYPE_ID(::mlir::test_irdl_to_cpp::nested::namespaced::MoreOp)
+
     // CHECK: ConditionalOp definitions
     // CHECK: __mlir_irdl_local_region_constraint_ConditionalOp_cond
     // CHECK: if (!(region.getNumArguments() == 1)) {
@@ -73,6 +113,10 @@ irdl.dialect @test_irdl_to_cpp {
     // CHECK: __mlir_irdl_local_region_constraint_ConditionalOp_else
     // CHECK: failure
     // CHECK: success
+
+    // CHECK: void TestIrdlToCppDialect::initialize()
+    // CHECK: ::mlir::test_irdl_to_cpp::nested::NamespacedOp
+    // CHECK: ::mlir::test_irdl_to_cpp::nested::namespaced::MoreOp
     irdl.operation @conditional {
         %r0 = irdl.region      // Unconstrained region
         %r1 = irdl.region()    // Region with no entry block arguments

>From f78a519928fc14d49c4daa42dd93c995a0cc54d7 Mon Sep 17 00:00:00 2001
From: Ivan Ho <38537881+hhkit at users.noreply.github.com>
Date: Thu, 2 Jul 2026 17:41:13 +0100
Subject: [PATCH 2/2] fix that works but i don't like it

---
 mlir/lib/Dialect/IRDL/IR/IRDL.cpp       |  9 ++++
 mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp | 61 +++++++++++++++++++------
 2 files changed, 57 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/IRDL/IR/IRDL.cpp b/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
index 278113fc6e966..b1005b01af86c 100644
--- a/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
+++ b/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
@@ -78,12 +78,19 @@ static llvm::LogicalResult isValidName(llvm::StringRef in, mlir::Operation *loc,
     return loc->emitError("name of ") << label << " is empty";
 
   bool allowUnderscore = false;
+  auto prev = '\0';
   for (auto &elem : in) {
     if (elem == '_') {
       if (!allowUnderscore)
         return loc->emitError("name of ")
                << label << " should not contain leading or double underscores";
     } else {
+      if (elem == '.') {
+        if (prev == '.')
+          return loc->emitError("empty namespace not allowed");
+      }
+      else
+      {
       if (!isalnum(elem))
         return loc->emitError("name of ")
                << label
@@ -93,9 +100,11 @@ static llvm::LogicalResult isValidName(llvm::StringRef in, mlir::Operation *loc,
       if (llvm::isUpper(elem))
         return loc->emitError("name of ")
                << label << " should not contain uppercase letters";
+      }
     }
 
     allowUnderscore = elem != '_';
+    prev = elem;
   }
 
   return success();
diff --git a/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp b/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
index 046c7dd0fccff..e8773937fc313 100644
--- a/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
+++ b/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
@@ -13,6 +13,7 @@
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/raw_ostream.h"
@@ -52,6 +53,8 @@ struct TypeStrings {
 struct OpStrings {
   StringRef opName;
   std::string opCppName;
+  std::string opScopedCppName;
+  SmallVector<std::string> opNameSpaces;
   SmallVector<std::string> opResultNames;
   SmallVector<std::string> opOperandNames;
   SmallVector<std::string> opRegionNames;
@@ -73,8 +76,19 @@ static std::string typeToCppName(irdl::TypeOp type) {
 
 /// Generates the C++ class name for an OperationOp
 static std::string opToCppName(irdl::OperationOp op) {
+  const auto opName = op.getSymName();
+  const auto periodIndex = opName.find_last_of(".");
+  const auto nameSubstr = periodIndex == std::string::npos ? opName : opName.substr(periodIndex + 1);
   return llvm::formatv("{0}Op",
-                       convertToCamelFromSnakeCase(op.getSymName(), true));
+                       convertToCamelFromSnakeCase(nameSubstr, true));
+}
+
+// Generates the C++ class name for an OperationOp, scoped to the namespace
+static std::string opToScopedCppName(irdl::OperationOp op) {
+  auto names = llvm::SmallVector<std::string>(llvm::split(op.getSymName(), "."));
+  names.pop_back();
+  names.push_back(opToCppName(op));
+  return llvm::join(names, "::");
 }
 
 /// Generates TypeStrings from a TypeOp
@@ -92,8 +106,12 @@ static OpStrings getStrings(irdl::OperationOp op) {
   auto regionsOp = op.getOp<irdl::RegionsOp>();
 
   OpStrings strings;
+  auto opNameParts = SmallVector<StringRef>(llvm::split(op.getSymName(), "."));
   strings.opName = op.getSymName();
+  opNameParts.pop_back();
+  strings.opNameSpaces = llvm::map_to_vector(opNameParts, &StringRef::str);
   strings.opCppName = opToCppName(op);
+  strings.opScopedCppName = opToScopedCppName(op);
 
   if (operandOp) {
     strings.opOperandNames = SmallVector<std::string>(
@@ -134,6 +152,7 @@ static void fillDict(irdl::detail::dictionary &dict, const OpStrings &strings) {
 
   dict["OP_NAME"] = strings.opName;
   dict["OP_CPP_NAME"] = strings.opCppName;
+  dict["OP_SCOPED_CPP_NAME"] = strings.opScopedCppName;
   dict["OP_OPERAND_COUNT"] = std::to_string(strings.opOperandNames.size());
   dict["OP_RESULT_COUNT"] = std::to_string(strings.opResultNames.size());
   dict["OP_OPERAND_INITIALIZER_LIST"] =
@@ -141,6 +160,20 @@ static void fillDict(irdl::detail::dictionary &dict, const OpStrings &strings) {
   dict["OP_RESULT_INITIALIZER_LIST"] =
       resultCount ? joinNameList(strings.opResultNames) : "{\"\"}";
   dict["OP_REGION_COUNT"] = std::to_string(regionCount);
+  dict["NAMESPACE_OPEN"] = (dict["NAMESPACE_OPEN"] + llvm::join(llvm::map_range(strings.opNameSpaces, [](llvm::StringRef ref) -> std::string {
+    return llvm::formatv("namespace {0} {{", ref);
+  }), "\n")).str();
+  dict["NAMESPACE_PATH"] =
+      (dict["NAMESPACE_PATH"] +
+       llvm::join(llvm::map_range(strings.opNameSpaces,
+                                  [](llvm::StringRef ref) -> std::string {
+                                    return llvm::formatv("::{0}", ref);
+                                  }),
+                  ""))
+          .str();
+  dict["NAMESPACE_CLOSE"] = (llvm::join(llvm::map_range(llvm::reverse(strings.opNameSpaces), [](llvm::StringRef ref) -> std::string {
+    return llvm::formatv("} // namespace {0}\n", ref);
+  }), "") + dict["NAMESPACE_CLOSE"]).str();
 }
 
 /// Fills a dictionary with values from DialectStrings
@@ -166,7 +199,7 @@ static LogicalResult generateTypedefList(irdl::DialectOp &dialect,
 static LogicalResult generateOpList(irdl::DialectOp &dialect,
                                     SmallVector<std::string> &opNames) {
   auto operationOps = dialect.getOps<irdl::OperationOp>();
-  auto range = llvm::map_range(operationOps, opToCppName);
+  auto range = llvm::map_range(operationOps, opToScopedCppName);
   opNames = SmallVector<std::string>(range);
   return success();
 }
@@ -283,24 +316,25 @@ static SmallVector<std::string> generateTraits(irdl::OperationOp op,
 
 static LogicalResult generateOperationInclude(irdl::OperationOp op,
                                               raw_ostream &output,
-                                              irdl::detail::dictionary &dict) {
+                                              const irdl::detail::dictionary &dict) {
   static const auto perOpDeclTemplate = irdl::detail::Template(
 #include "Templates/PerOperationDecl.txt"
   );
   const auto opStrings = getStrings(op);
-  fillDict(dict, opStrings);
+  auto opDict = dict;
+  fillDict(opDict, opStrings);
 
   SmallVector<std::string> traitNames = generateTraits(op, opStrings);
   if (traitNames.empty())
-    dict["OP_TEMPLATE_ARGS"] = opStrings.opCppName;
+    opDict["OP_TEMPLATE_ARGS"] = opStrings.opCppName;
   else
-    dict["OP_TEMPLATE_ARGS"] = llvm::formatv("{0}, {1}", opStrings.opCppName,
+    opDict["OP_TEMPLATE_ARGS"] = llvm::formatv("{0}, {1}", opStrings.opCppName,
                                              llvm::join(traitNames, ", "));
 
-  generateOpGetterDeclarations(dict, opStrings);
-  generateOpBuilderDeclarations(dict, opStrings);
+  generateOpGetterDeclarations(opDict, opStrings);
+  generateOpBuilderDeclarations(opDict, opStrings);
 
-  perOpDeclTemplate.render(output, dict);
+  perOpDeclTemplate.render(output, opDict);
   return success();
 }
 
@@ -462,7 +496,8 @@ static std::string generateOpDefinition(irdl::detail::dictionary &dict,
   };
 
   auto opStrings = getStrings(op);
-  fillDict(dict, opStrings);
+  auto opDict = dict;
+  fillDict(opDict, opStrings);
 
   auto resultTypes = llvm::join(
       llvm::map_range(opStrings.opResultNames,
@@ -516,13 +551,13 @@ void {0}::build(::mlir::OpBuilder &opBuilder, ::mlir::OperationState &opState, {
       llvm::join(opStrings.opOperandNames, ",") +
           (!opStrings.opOperandNames.empty() ? "," : ""));
 
-  dict["OP_BUILD_DEFS"] = buildDefinition;
+  opDict["OP_BUILD_DEFS"] = buildDefinition;
 
-  generateVerifiers(dict, op, opStrings);
+  generateVerifiers(opDict, op, opStrings);
 
   std::string str;
   llvm::raw_string_ostream stream{str};
-  perOpDefTemplate.render(stream, dict);
+  perOpDefTemplate.render(stream, opDict);
   return str;
 }
 


From llvmlistbot at llvm.org  Thu Jul  2 09:43:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 09:43:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][IRDL] Instantiate "." in IRDL-defined
 names as nested namespaces (PR #207035)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207035@github.com>
Message-ID: <6a469526.331483e7.5bc1e.d081@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/IRDL/IR/IRDL.cpp mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/IRDL/IR/IRDL.cpp b/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
index b1005b01a..170acb97d 100644
--- a/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
+++ b/mlir/lib/Dialect/IRDL/IR/IRDL.cpp
@@ -88,18 +88,16 @@ static llvm::LogicalResult isValidName(llvm::StringRef in, mlir::Operation *loc,
       if (elem == '.') {
         if (prev == '.')
           return loc->emitError("empty namespace not allowed");
-      }
-      else
-      {
-      if (!isalnum(elem))
-        return loc->emitError("name of ")
-               << label
-               << " must contain only lowercase letters, digits and "
-                  "underscores";
-
-      if (llvm::isUpper(elem))
-        return loc->emitError("name of ")
-               << label << " should not contain uppercase letters";
+      } else {
+        if (!isalnum(elem))
+          return loc->emitError("name of ")
+                 << label
+                 << " must contain only lowercase letters, digits and "
+                    "underscores";
+
+        if (llvm::isUpper(elem))
+          return loc->emitError("name of ")
+                 << label << " should not contain uppercase letters";
       }
     }
 
diff --git a/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp b/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
index e8773937f..ed52dd42c 100644
--- a/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
+++ b/mlir/lib/Target/IRDLToCpp/IRDLToCpp.cpp
@@ -78,14 +78,16 @@ static std::string typeToCppName(irdl::TypeOp type) {
 static std::string opToCppName(irdl::OperationOp op) {
   const auto opName = op.getSymName();
   const auto periodIndex = opName.find_last_of(".");
-  const auto nameSubstr = periodIndex == std::string::npos ? opName : opName.substr(periodIndex + 1);
-  return llvm::formatv("{0}Op",
-                       convertToCamelFromSnakeCase(nameSubstr, true));
+  const auto nameSubstr = periodIndex == std::string::npos
+                              ? opName
+                              : opName.substr(periodIndex + 1);
+  return llvm::formatv("{0}Op", convertToCamelFromSnakeCase(nameSubstr, true));
 }
 
 // Generates the C++ class name for an OperationOp, scoped to the namespace
 static std::string opToScopedCppName(irdl::OperationOp op) {
-  auto names = llvm::SmallVector<std::string>(llvm::split(op.getSymName(), "."));
+  auto names =
+      llvm::SmallVector<std::string>(llvm::split(op.getSymName(), "."));
   names.pop_back();
   names.push_back(opToCppName(op));
   return llvm::join(names, "::");
@@ -160,9 +162,15 @@ static void fillDict(irdl::detail::dictionary &dict, const OpStrings &strings) {
   dict["OP_RESULT_INITIALIZER_LIST"] =
       resultCount ? joinNameList(strings.opResultNames) : "{\"\"}";
   dict["OP_REGION_COUNT"] = std::to_string(regionCount);
-  dict["NAMESPACE_OPEN"] = (dict["NAMESPACE_OPEN"] + llvm::join(llvm::map_range(strings.opNameSpaces, [](llvm::StringRef ref) -> std::string {
-    return llvm::formatv("namespace {0} {{", ref);
-  }), "\n")).str();
+  dict["NAMESPACE_OPEN"] =
+      (dict["NAMESPACE_OPEN"] +
+       llvm::join(llvm::map_range(strings.opNameSpaces,
+                                  [](llvm::StringRef ref) -> std::string {
+                                    return llvm::formatv("namespace {0} {{",
+                                                         ref);
+                                  }),
+                  "\n"))
+          .str();
   dict["NAMESPACE_PATH"] =
       (dict["NAMESPACE_PATH"] +
        llvm::join(llvm::map_range(strings.opNameSpaces,
@@ -171,9 +179,15 @@ static void fillDict(irdl::detail::dictionary &dict, const OpStrings &strings) {
                                   }),
                   ""))
           .str();
-  dict["NAMESPACE_CLOSE"] = (llvm::join(llvm::map_range(llvm::reverse(strings.opNameSpaces), [](llvm::StringRef ref) -> std::string {
-    return llvm::formatv("} // namespace {0}\n", ref);
-  }), "") + dict["NAMESPACE_CLOSE"]).str();
+  dict["NAMESPACE_CLOSE"] =
+      (llvm::join(llvm::map_range(llvm::reverse(strings.opNameSpaces),
+                                  [](llvm::StringRef ref) -> std::string {
+                                    return llvm::formatv("} // namespace {0}\n",
+                                                         ref);
+                                  }),
+                  "") +
+       dict["NAMESPACE_CLOSE"])
+          .str();
 }
 
 /// Fills a dictionary with values from DialectStrings
@@ -314,9 +328,9 @@ static SmallVector<std::string> generateTraits(irdl::OperationOp op,
   return cppTraitNames;
 }
 
-static LogicalResult generateOperationInclude(irdl::OperationOp op,
-                                              raw_ostream &output,
-                                              const irdl::detail::dictionary &dict) {
+static LogicalResult
+generateOperationInclude(irdl::OperationOp op, raw_ostream &output,
+                         const irdl::detail::dictionary &dict) {
   static const auto perOpDeclTemplate = irdl::detail::Template(
 #include "Templates/PerOperationDecl.txt"
   );
@@ -329,7 +343,7 @@ static LogicalResult generateOperationInclude(irdl::OperationOp op,
     opDict["OP_TEMPLATE_ARGS"] = opStrings.opCppName;
   else
     opDict["OP_TEMPLATE_ARGS"] = llvm::formatv("{0}, {1}", opStrings.opCppName,
-                                             llvm::join(traitNames, ", "));
+                                               llvm::join(traitNames, ", "));
 
   generateOpGetterDeclarations(opDict, opStrings);
   generateOpBuilderDeclarations(opDict, opStrings);

``````````

</details>


https://github.com/llvm/llvm-project/pull/207035

From llvmlistbot at llvm.org  Thu Jul  2 09:44:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 09:44:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2116728 - [MLIR][XeGPU] Enable end-to-end
 integration test compilation (#207158)
Message-ID: <6a46956c.e9d484b9.11ad58.22cd@mx.google.com>


Author: Artem Kroviakov
Date: 2026-07-02T18:44:23+02:00
New Revision: 21167283a0d3687db672541b58dbf44c9ad6056a

URL: https://github.com/llvm/llvm-project/commit/21167283a0d3687db672541b58dbf44c9ad6056a
DIFF: https://github.com/llvm/llvm-project/commit/21167283a0d3687db672541b58dbf44c9ad6056a.diff

LOG: [MLIR][XeGPU] Enable end-to-end integration test compilation (#207158)

Added: 
    

Modified: 
    mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
index 8db649b600350..cb5c5a7422e1b 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_dequantizeB_F4.mlir
@@ -1,12 +1,12 @@
-// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri" \
-// RUN: | mlir-runner \
-// RUN:   --shared-libs=%mlir_levelzero_runtime \
-// RUN:   --shared-libs=%mlir_runner_utils \
-// RUN:   --shared-libs=%mlir_c_runner_utils \
-// RUN:   --entry-point-result=void \
-// RUN: | FileCheck %s
-
-// XFAIL: *
+// RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri"
+// RUN-DISABLED: | mlir-runner \
+// RUN-DISABLED:   --shared-libs=%mlir_levelzero_runtime \
+// RUN-DISABLED:   --shared-libs=%mlir_runner_utils \
+// RUN-DISABLED:   --shared-libs=%mlir_c_runner_utils \
+// RUN-DISABLED:   --entry-point-result=void \
+// RUN-DISABLED: | FileCheck %s
+
+
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
 #a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>


        

From llvmlistbot at llvm.org  Thu Jul  2 09:44:30 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 02 Jul 2026 09:44:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enable end-to-end integration
 test compilation (PR #207158)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207158@github.com>
Message-ID: <6a46956e.70920dee.3514b2.5600@mx.google.com>


https://github.com/akroviakov closed https://github.com/llvm/llvm-project/pull/207158

From llvmlistbot at llvm.org  Thu Jul  2 09:50:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 09:50:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 025c4c1 - [mlir][tosa] Fold unit-expanded
 reshapes in reduce-transposes (#203529)
Message-ID: <6a4696d7.d5797644.146403.5e6c@mx.google.com>


Author: Luke Hutton
Date: 2026-07-02T17:50:27+01:00
New Revision: 025c4c1d6bd017749e71ad3dc0b0d7b253606b4f

URL: https://github.com/llvm/llvm-project/commit/025c4c1d6bd017749e71ad3dc0b0d7b253606b4f
DIFF: https://github.com/llvm/llvm-project/commit/025c4c1d6bd017749e71ad3dc0b0d7b253606b4f.diff

LOG: [mlir][tosa] Fold unit-expanded reshapes in reduce-transposes (#203529)

Extend tosa-reduce-transposes so transpose hosting can fold through
unit-expanded vector reshapes such as 1x...x1xC. In particular, this
improves transpose folding when hosting through broadcast binary
elementwise operations.

Added: 
    

Modified: 
    mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
    mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 65ef49bdc3077..0bd415040f22f 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -174,6 +174,23 @@ struct TosaReduceTransposes final
   transposeDenseAttribute(DenseElementsAttr input, ArrayRef<int32_t> perms);
 };
 
+// Check if shape is of the form 1x1x...x1xNx1x...x1x1 -> 1x1x...x1xNx1x...x1x1
+// Valid examples include:
+// - N -> 1x1xNx1
+// - Nx1x1x1 -> 1x1xNx1
+// - 1x1xNx1 -> 1x1xNx1
+static LogicalResult verifyUnitExpandedVectorShape(ArrayRef<int64_t> shape) {
+  bool nonUnitDimDetected = false;
+  for (const int64_t dim : shape) {
+    if (dim != 1) {
+      if (nonUnitDimDetected)
+        return failure();
+      nonUnitDimDetected = true;
+    }
+  }
+  return success();
+}
+
 std::optional<DenseElementsAttr>
 TosaReduceTransposes::transposeDenseAttribute(DenseElementsAttr input,
                                               ArrayRef<int32_t> perms) {
@@ -392,23 +409,18 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   auto reshapeOutput = reshapeOp.getOutput();
   auto reshapeInputType =
       llvm::dyn_cast<RankedTensorType>(reshapeOp.getInput1().getType());
-  auto reshapeInputShape = reshapeInputType.getShape();
-  // want reshape N -> 1x1x...x1xNx1x...x1x1
-  if (!reshapeInputType || reshapeInputShape.size() != 1)
+  if (!reshapeInputType)
     return std::nullopt;
+  auto reshapeInputShape = reshapeInputType.getShape();
   auto reshapeOutputType =
       llvm::cast<RankedTensorType>(reshapeOutput.getType());
+  const ArrayRef<int64_t> reshapeOutputShape = reshapeOutputType.getShape();
 
   // Instead of inserting a TransposeOp here, we check if we can fold it into
   // the ReshapeOp. There is more complex cases where this is possible, and
   // this check can be extended.
-
-  // Checking if reshape is N -> 1x1x...x1xNx1x...x1x1
-  auto shape = reshapeOutputType.getShape();
-  size_t ones = llvm::count(shape, 1);
-  // N == 1 and N != 1
-  if (ones != shape.size() - 1 &&
-      (ones != shape.size() || reshapeInputShape[0] != 1))
+  if (failed(verifyUnitExpandedVectorShape(reshapeInputShape)) ||
+      failed(verifyUnitExpandedVectorShape(reshapeOutputShape)))
     return std::nullopt;
 
   // Do not insert a TransposeOp, instead we fold the reshape and its attribute.
@@ -421,8 +433,9 @@ std::optional<Value> TosaReduceTransposes::buildMappedToValue(
   ImplicitLocOpBuilder builder(reshapeOp.getLoc(), rewriter);
   auto foldedReshape = ReshapeOp::create(
       rewriter, reshapeOp.getLoc(),
-      RankedTensorType::get(applyTOSAPermutation(shape, hoistedPerms),
-                            reshapeOutputType.getElementType()),
+      RankedTensorType::get(
+          applyTOSAPermutation(reshapeOutputShape, hoistedPerms),
+          reshapeOutputType.getElementType()),
       reshapeOp.getInput1(),
       getTosaConstShape(builder, applyTOSAPermutation(llvm::ArrayRef(newShape),
                                                       hoistedPerms)));

diff  --git a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
index b3f4260ede2f5..0ad3068011f7f 100644
--- a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
@@ -174,6 +174,22 @@ func.func @test_reshape_for_broadcast(%arg0: tensor<4x3x2xi32>) -> tensor<4x3x2x
 
 // -----
 
+// CHECK-LABEL: @test_multi_dim_reshape_for_broadcast
+// CHECK-DAG: %[[SHAPE:.*]] = tosa.const_shape  {values = dense<[4, 1, 1]> : tensor<3xindex>}
+// CHECK: %[[RESHAPE:.*]] = tosa.reshape %arg0, %[[SHAPE]] : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<4x1x1xi32>
+// CHECK: %[[ADD:.*]] = tosa.add %arg1, %[[RESHAPE]]
+// CHECK: return %[[ADD]]
+func.func @test_multi_dim_reshape_for_broadcast(%arg0: tensor<1x4x1xi32>, %arg1: tensor<4x3x2xi32>) -> tensor<4x3x2xi32> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 4]> : tensor<3xindex>} : () -> !tosa.shape<3>
+  %reshape = tosa.reshape %arg0, %shape : (tensor<1x4x1xi32>, !tosa.shape<3>) -> tensor<1x1x4xi32>
+  %transpose0 = tosa.transpose %arg1 {perms = array<i32: 2, 1, 0>}: (tensor<4x3x2xi32>) -> tensor<2x3x4xi32>
+  %add = tosa.add %transpose0, %reshape : (tensor<2x3x4xi32>, tensor<1x1x4xi32>) -> tensor<2x3x4xi32>
+  %transpose1 = tosa.transpose %add {perms = array<i32: 2, 1, 0>}: (tensor<2x3x4xi32>) -> tensor<4x3x2xi32>
+  return %transpose1 : tensor<4x3x2xi32>
+}
+
+// -----
+
 // COM: taken directly from ResNet18 translation.
 // COM: changes: %74 as argument instead of result of conv2d
 


        

From llvmlistbot at llvm.org  Thu Jul  2 09:50:35 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 02 Jul 2026 09:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold unit-expanded reshapes in
 reduce-transposes (PR #203529)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203529@github.com>
Message-ID: <6a4696db.8e75e0cb.2e0812.0316@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/203529

From llvmlistbot at llvm.org  Thu Jul  2 10:15:45 2026
From: llvmlistbot at llvm.org (Robert Imschweiler)
Date: Thu, 02 Jul 2026 10:15:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
Message-ID: <llvm.org/llvm/llvm-project/pull/207234@github.com>


https://github.com/ro-i created https://github.com/llvm/llvm-project/pull/207234

During OpenMP lowering, globally visible device functions are emitted. These functions might not be kernels themselves, but are designed to only be called in a kernel context. However, if they are unused, and not inlined, and reference LDS, the AMDGPU ISel emits lots of misleading warnings related to "local memory global used by non-kernel function". Fix by changing visibility from external+default to external+hidden, which allows DCE to just remove the functions.

Claude assisted with this patch.

>From 67f0c1e7ca0ae51ae84ba93f6cb50f1cd760c827 Mon Sep 17 00:00:00 2001
From: Robert Imschweiler <robert.imschweiler at amd.com>
Date: Thu, 2 Jul 2026 08:54:02 -0500
Subject: [PATCH] [mlir][OpenMP] Change device declare target functions to
 hidden visibility

During OpenMP lowering, globally visible device functions are emitted.
These functions might not be kernels themselves, but are designed to
only be called in a kernel context. However, if they are unused, and not
inlined, and reference LDS, the AMDGPU ISel emits lots of misleading
warnings related to "local memory global used by non-kernel function".
Fix by changing visibility from external+default to external+hidden,
which allows DCE to just remove the functions.

Claude assisted with this patch.
---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 15 ++++++++
 ...target-declare-target-func-visibility.mlir | 38 +++++++++++++++++++
 .../LLVMIR/omptarget-device-shared-mem.mlir   |  2 +-
 .../LLVMIR/omptarget-wsloop-collapsed.mlir    |  2 +-
 mlir/test/Target/LLVMIR/omptarget-wsloop.mlir |  4 +-
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  6 +--
 6 files changed, 60 insertions(+), 7 deletions(-)
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index eb34cabdf20f8..a287644207724 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8819,6 +8819,21 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
         // a deleted block.
         ompBuilder->Builder.ClearInsertionPoint();
         ompBuilder->Builder.SetCurrentDebugLocation(llvm::DebugLoc());
+      } else if (llvm::Function *llvmFunc =
+                     moduleTranslation.lookupFunction(funcOp.getName())) {
+        // Device-side declare target functions are externally visible by
+        // default so they can be referenced from other device translation
+        // units. That also prevents the offload LTO from internalizing and
+        // deleting them when they end up unused in the final device image.
+        // Such dead functions can still reference internal LDS and trigger
+        // spurious "local memory global used by non-kernel function" backend
+        // warnings. Marking them hidden keeps the symbol usable within the
+        // device image's linkage unit while letting LTO drop it when nothing
+        // references it; symbols that must stay reachable (e.g. via an offload
+        // entry that takes their address) are kept alive by that reference.
+        if (!llvmFunc->isDeclaration() && llvmFunc->hasExternalLinkage() &&
+            llvmFunc->getVisibility() == llvm::GlobalValue::DefaultVisibility)
+          llvmFunc->setVisibility(llvm::GlobalValue::HiddenVisibility);
       }
     }
     return success();
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
new file mode 100644
index 0000000000000..f5f3c9f419f0b
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
@@ -0,0 +1,38 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Device-side `declare target` functions are externally visible by default so
+// that they can be referenced from other device translation units. They are
+// emitted with hidden visibility so that the offload LTO can internalize and
+// delete them when they turn out to be unused in the final device image. This
+// avoids leaving dead functions behind that may, for example, reference LDS and
+// trigger spurious backend diagnostics.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: define hidden void @device_any()
+  llvm.func @device_any() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // CHECK: define hidden void @device_nohost()
+  llvm.func @device_nohost() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function with an explicitly requested (non-default) visibility is left
+  // untouched.
+  // CHECK: define protected void @device_protected()
+  llvm.func protected @device_protected() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function that is not declare target is unaffected.
+  // CHECK: define void @not_declare_target()
+  llvm.func @not_declare_target() {
+    llvm.return
+  }
+
+  // A declaration (no definition) is left untouched: there is nothing to
+  // internalize, and hiding it could over-constrain the symbol's visibility.
+  // CHECK: declare void @device_decl()
+  llvm.func @device_decl() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>}
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
index cdebebc3ed233..6fccc0127b347 100644
--- a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
 module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memory_space", 5 : ui32>>, llvm.data_layout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9", llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
-  // CHECK-LABEL: define void @device_shared_mem(
+  // CHECK-LABEL: define hidden void @device_shared_mem(
   // CHECK-SAME:  i32 %[[N0:.*]], i64 %[[N1:.*]])
   llvm.func @device_shared_mem(%n0: i32, %n1: i64) attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
     // CHECK:      %[[CAST_N0:.*]] = zext i32 %[[N0]] to i64
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
index d84641ff9c99b..f120dde8131d6 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
@@ -22,7 +22,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr),
 // CHECK-SAME: ptr @[[COLLAPSED_WSLOOP_BODY_FN:.*]], ptr %[[STRUCT_ARG:.*]], i32 10000,
 // CHECK-SAME: i32 %[[NUM_THREADS:.*]], i8 0)
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
index 7be635f46111b..04458af9654c3 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
@@ -31,7 +31,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
 // CHECK:   %[[STRUCTARG:.*]] = alloca { ptr }, align 8, addrspace(5)
 // CHECK:   %[[STRUCTARG_ASCAST:.*]] = addrspacecast ptr addrspace(5) %[[STRUCTARG]] to ptr
 // CHECK:   %[[GEP:.*]] = getelementptr { ptr }, ptr addrspace(5) %[[STRUCTARG]], i32 0, i32 0
@@ -45,7 +45,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
 // CHECK:   %[[GEP3:.*]] = getelementptr [10 x i32], ptr %[[LOADGEP]], i32 0, i32 %[[TMP2:.*]]
 // CHECK:   store i32 %[[VAL0:.*]], ptr %[[GEP3]], align 4
 
-// CHECK: define void @[[FUNC_EMPTY_WSLOOP:.*]]()
+// CHECK: define hidden void @[[FUNC_EMPTY_WSLOOP:.*]]()
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr), ptr @[[LOOP_EMPTY_BODY_FN:.*]], ptr null, i32 10, i32 %[[NUM_THREADS:.*]], i32 0, i8 0)
 
 // CHECK: define internal void @[[LOOP_EMPTY_BODY_FN]](i32 %[[LOOP_CNT:.*]])
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 790c0dfe9b12e..444918fb1a9e3 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3661,7 +3661,7 @@ module attributes {omp.is_target_device = false} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3683,7 +3683,7 @@ module attributes {omp.is_target_device = true} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3993,7 +3993,7 @@ llvm.func @omp_groupprivate_device() attributes {
 // CHECK-DAG: @nohost = internal global i32 undef
 // CHECK-DAG: @[[SHARED_ANY:any.*]] = internal addrspace(3) global i32 poison
 // CHECK-DAG: @[[SHARED_NOHOST:nohost.*]] = internal addrspace(3) global i32 poison
-// CHECK: define void @omp_groupprivate_device()
+// CHECK: define hidden void @omp_groupprivate_device()
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_ANY]], align 4
 // CHECK: store i32 1, ptr @host, align 4
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_NOHOST]], align 4


From llvmlistbot at llvm.org  Thu Jul  2 10:16:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 10:16:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a469cf1.a151105c.30c1b9.3cfd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Robert Imschweiler (ro-i)

<details>
<summary>Changes</summary>

During OpenMP lowering, globally visible device functions are emitted. These functions might not be kernels themselves, but are designed to only be called in a kernel context. However, if they are unused, and not inlined, and reference LDS, the AMDGPU ISel emits lots of misleading warnings related to "local memory global used by non-kernel function". Fix by changing visibility from external+default to external+hidden, which allows DCE to just remove the functions.

Claude assisted with this patch.

---
Full diff: https://github.com/llvm/llvm-project/pull/207234.diff


6 Files Affected:

- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+15) 
- (added) mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir (+38) 
- (modified) mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop.mlir (+2-2) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+3-3) 


``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index eb34cabdf20f8..a287644207724 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8819,6 +8819,21 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
         // a deleted block.
         ompBuilder->Builder.ClearInsertionPoint();
         ompBuilder->Builder.SetCurrentDebugLocation(llvm::DebugLoc());
+      } else if (llvm::Function *llvmFunc =
+                     moduleTranslation.lookupFunction(funcOp.getName())) {
+        // Device-side declare target functions are externally visible by
+        // default so they can be referenced from other device translation
+        // units. That also prevents the offload LTO from internalizing and
+        // deleting them when they end up unused in the final device image.
+        // Such dead functions can still reference internal LDS and trigger
+        // spurious "local memory global used by non-kernel function" backend
+        // warnings. Marking them hidden keeps the symbol usable within the
+        // device image's linkage unit while letting LTO drop it when nothing
+        // references it; symbols that must stay reachable (e.g. via an offload
+        // entry that takes their address) are kept alive by that reference.
+        if (!llvmFunc->isDeclaration() && llvmFunc->hasExternalLinkage() &&
+            llvmFunc->getVisibility() == llvm::GlobalValue::DefaultVisibility)
+          llvmFunc->setVisibility(llvm::GlobalValue::HiddenVisibility);
       }
     }
     return success();
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
new file mode 100644
index 0000000000000..f5f3c9f419f0b
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
@@ -0,0 +1,38 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Device-side `declare target` functions are externally visible by default so
+// that they can be referenced from other device translation units. They are
+// emitted with hidden visibility so that the offload LTO can internalize and
+// delete them when they turn out to be unused in the final device image. This
+// avoids leaving dead functions behind that may, for example, reference LDS and
+// trigger spurious backend diagnostics.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: define hidden void @device_any()
+  llvm.func @device_any() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // CHECK: define hidden void @device_nohost()
+  llvm.func @device_nohost() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function with an explicitly requested (non-default) visibility is left
+  // untouched.
+  // CHECK: define protected void @device_protected()
+  llvm.func protected @device_protected() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function that is not declare target is unaffected.
+  // CHECK: define void @not_declare_target()
+  llvm.func @not_declare_target() {
+    llvm.return
+  }
+
+  // A declaration (no definition) is left untouched: there is nothing to
+  // internalize, and hiding it could over-constrain the symbol's visibility.
+  // CHECK: declare void @device_decl()
+  llvm.func @device_decl() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>}
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
index cdebebc3ed233..6fccc0127b347 100644
--- a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
 module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memory_space", 5 : ui32>>, llvm.data_layout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9", llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
-  // CHECK-LABEL: define void @device_shared_mem(
+  // CHECK-LABEL: define hidden void @device_shared_mem(
   // CHECK-SAME:  i32 %[[N0:.*]], i64 %[[N1:.*]])
   llvm.func @device_shared_mem(%n0: i32, %n1: i64) attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
     // CHECK:      %[[CAST_N0:.*]] = zext i32 %[[N0]] to i64
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
index d84641ff9c99b..f120dde8131d6 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
@@ -22,7 +22,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr),
 // CHECK-SAME: ptr @[[COLLAPSED_WSLOOP_BODY_FN:.*]], ptr %[[STRUCT_ARG:.*]], i32 10000,
 // CHECK-SAME: i32 %[[NUM_THREADS:.*]], i8 0)
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
index 7be635f46111b..04458af9654c3 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
@@ -31,7 +31,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
 // CHECK:   %[[STRUCTARG:.*]] = alloca { ptr }, align 8, addrspace(5)
 // CHECK:   %[[STRUCTARG_ASCAST:.*]] = addrspacecast ptr addrspace(5) %[[STRUCTARG]] to ptr
 // CHECK:   %[[GEP:.*]] = getelementptr { ptr }, ptr addrspace(5) %[[STRUCTARG]], i32 0, i32 0
@@ -45,7 +45,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
 // CHECK:   %[[GEP3:.*]] = getelementptr [10 x i32], ptr %[[LOADGEP]], i32 0, i32 %[[TMP2:.*]]
 // CHECK:   store i32 %[[VAL0:.*]], ptr %[[GEP3]], align 4
 
-// CHECK: define void @[[FUNC_EMPTY_WSLOOP:.*]]()
+// CHECK: define hidden void @[[FUNC_EMPTY_WSLOOP:.*]]()
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr), ptr @[[LOOP_EMPTY_BODY_FN:.*]], ptr null, i32 10, i32 %[[NUM_THREADS:.*]], i32 0, i8 0)
 
 // CHECK: define internal void @[[LOOP_EMPTY_BODY_FN]](i32 %[[LOOP_CNT:.*]])
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 790c0dfe9b12e..444918fb1a9e3 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3661,7 +3661,7 @@ module attributes {omp.is_target_device = false} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3683,7 +3683,7 @@ module attributes {omp.is_target_device = true} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3993,7 +3993,7 @@ llvm.func @omp_groupprivate_device() attributes {
 // CHECK-DAG: @nohost = internal global i32 undef
 // CHECK-DAG: @[[SHARED_ANY:any.*]] = internal addrspace(3) global i32 poison
 // CHECK-DAG: @[[SHARED_NOHOST:nohost.*]] = internal addrspace(3) global i32 poison
-// CHECK: define void @omp_groupprivate_device()
+// CHECK: define hidden void @omp_groupprivate_device()
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_ANY]], align 4
 // CHECK: store i32 1, ptr @host, align 4
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_NOHOST]], align 4

``````````

</details>


https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Thu Jul  2 10:16:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 10:16:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a469cf2.bb839515.23e6d3.971c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openmp

Author: Robert Imschweiler (ro-i)

<details>
<summary>Changes</summary>

During OpenMP lowering, globally visible device functions are emitted. These functions might not be kernels themselves, but are designed to only be called in a kernel context. However, if they are unused, and not inlined, and reference LDS, the AMDGPU ISel emits lots of misleading warnings related to "local memory global used by non-kernel function". Fix by changing visibility from external+default to external+hidden, which allows DCE to just remove the functions.

Claude assisted with this patch.

---
Full diff: https://github.com/llvm/llvm-project/pull/207234.diff


6 Files Affected:

- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+15) 
- (added) mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir (+38) 
- (modified) mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop.mlir (+2-2) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+3-3) 


``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index eb34cabdf20f8..a287644207724 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8819,6 +8819,21 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
         // a deleted block.
         ompBuilder->Builder.ClearInsertionPoint();
         ompBuilder->Builder.SetCurrentDebugLocation(llvm::DebugLoc());
+      } else if (llvm::Function *llvmFunc =
+                     moduleTranslation.lookupFunction(funcOp.getName())) {
+        // Device-side declare target functions are externally visible by
+        // default so they can be referenced from other device translation
+        // units. That also prevents the offload LTO from internalizing and
+        // deleting them when they end up unused in the final device image.
+        // Such dead functions can still reference internal LDS and trigger
+        // spurious "local memory global used by non-kernel function" backend
+        // warnings. Marking them hidden keeps the symbol usable within the
+        // device image's linkage unit while letting LTO drop it when nothing
+        // references it; symbols that must stay reachable (e.g. via an offload
+        // entry that takes their address) are kept alive by that reference.
+        if (!llvmFunc->isDeclaration() && llvmFunc->hasExternalLinkage() &&
+            llvmFunc->getVisibility() == llvm::GlobalValue::DefaultVisibility)
+          llvmFunc->setVisibility(llvm::GlobalValue::HiddenVisibility);
       }
     }
     return success();
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
new file mode 100644
index 0000000000000..f5f3c9f419f0b
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
@@ -0,0 +1,38 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Device-side `declare target` functions are externally visible by default so
+// that they can be referenced from other device translation units. They are
+// emitted with hidden visibility so that the offload LTO can internalize and
+// delete them when they turn out to be unused in the final device image. This
+// avoids leaving dead functions behind that may, for example, reference LDS and
+// trigger spurious backend diagnostics.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: define hidden void @device_any()
+  llvm.func @device_any() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // CHECK: define hidden void @device_nohost()
+  llvm.func @device_nohost() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function with an explicitly requested (non-default) visibility is left
+  // untouched.
+  // CHECK: define protected void @device_protected()
+  llvm.func protected @device_protected() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function that is not declare target is unaffected.
+  // CHECK: define void @not_declare_target()
+  llvm.func @not_declare_target() {
+    llvm.return
+  }
+
+  // A declaration (no definition) is left untouched: there is nothing to
+  // internalize, and hiding it could over-constrain the symbol's visibility.
+  // CHECK: declare void @device_decl()
+  llvm.func @device_decl() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>}
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
index cdebebc3ed233..6fccc0127b347 100644
--- a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
 module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memory_space", 5 : ui32>>, llvm.data_layout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9", llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
-  // CHECK-LABEL: define void @device_shared_mem(
+  // CHECK-LABEL: define hidden void @device_shared_mem(
   // CHECK-SAME:  i32 %[[N0:.*]], i64 %[[N1:.*]])
   llvm.func @device_shared_mem(%n0: i32, %n1: i64) attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
     // CHECK:      %[[CAST_N0:.*]] = zext i32 %[[N0]] to i64
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
index d84641ff9c99b..f120dde8131d6 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
@@ -22,7 +22,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr),
 // CHECK-SAME: ptr @[[COLLAPSED_WSLOOP_BODY_FN:.*]], ptr %[[STRUCT_ARG:.*]], i32 10000,
 // CHECK-SAME: i32 %[[NUM_THREADS:.*]], i8 0)
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
index 7be635f46111b..04458af9654c3 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
@@ -31,7 +31,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
 // CHECK:   %[[STRUCTARG:.*]] = alloca { ptr }, align 8, addrspace(5)
 // CHECK:   %[[STRUCTARG_ASCAST:.*]] = addrspacecast ptr addrspace(5) %[[STRUCTARG]] to ptr
 // CHECK:   %[[GEP:.*]] = getelementptr { ptr }, ptr addrspace(5) %[[STRUCTARG]], i32 0, i32 0
@@ -45,7 +45,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
 // CHECK:   %[[GEP3:.*]] = getelementptr [10 x i32], ptr %[[LOADGEP]], i32 0, i32 %[[TMP2:.*]]
 // CHECK:   store i32 %[[VAL0:.*]], ptr %[[GEP3]], align 4
 
-// CHECK: define void @[[FUNC_EMPTY_WSLOOP:.*]]()
+// CHECK: define hidden void @[[FUNC_EMPTY_WSLOOP:.*]]()
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr), ptr @[[LOOP_EMPTY_BODY_FN:.*]], ptr null, i32 10, i32 %[[NUM_THREADS:.*]], i32 0, i8 0)
 
 // CHECK: define internal void @[[LOOP_EMPTY_BODY_FN]](i32 %[[LOOP_CNT:.*]])
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 790c0dfe9b12e..444918fb1a9e3 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3661,7 +3661,7 @@ module attributes {omp.is_target_device = false} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3683,7 +3683,7 @@ module attributes {omp.is_target_device = true} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3993,7 +3993,7 @@ llvm.func @omp_groupprivate_device() attributes {
 // CHECK-DAG: @nohost = internal global i32 undef
 // CHECK-DAG: @[[SHARED_ANY:any.*]] = internal addrspace(3) global i32 poison
 // CHECK-DAG: @[[SHARED_NOHOST:nohost.*]] = internal addrspace(3) global i32 poison
-// CHECK: define void @omp_groupprivate_device()
+// CHECK: define hidden void @omp_groupprivate_device()
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_ANY]], align 4
 // CHECK: store i32 1, ptr @host, align 4
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_NOHOST]], align 4

``````````

</details>


https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Thu Jul  2 10:16:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 10:16:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a469cf4.1a4df1a5.118deb.ac18@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-openmp

Author: Robert Imschweiler (ro-i)

<details>
<summary>Changes</summary>

During OpenMP lowering, globally visible device functions are emitted. These functions might not be kernels themselves, but are designed to only be called in a kernel context. However, if they are unused, and not inlined, and reference LDS, the AMDGPU ISel emits lots of misleading warnings related to "local memory global used by non-kernel function". Fix by changing visibility from external+default to external+hidden, which allows DCE to just remove the functions.

Claude assisted with this patch.

---
Full diff: https://github.com/llvm/llvm-project/pull/207234.diff


6 Files Affected:

- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+15) 
- (added) mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir (+38) 
- (modified) mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/omptarget-wsloop.mlir (+2-2) 
- (modified) mlir/test/Target/LLVMIR/openmp-llvm.mlir (+3-3) 


``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index eb34cabdf20f8..a287644207724 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8819,6 +8819,21 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
         // a deleted block.
         ompBuilder->Builder.ClearInsertionPoint();
         ompBuilder->Builder.SetCurrentDebugLocation(llvm::DebugLoc());
+      } else if (llvm::Function *llvmFunc =
+                     moduleTranslation.lookupFunction(funcOp.getName())) {
+        // Device-side declare target functions are externally visible by
+        // default so they can be referenced from other device translation
+        // units. That also prevents the offload LTO from internalizing and
+        // deleting them when they end up unused in the final device image.
+        // Such dead functions can still reference internal LDS and trigger
+        // spurious "local memory global used by non-kernel function" backend
+        // warnings. Marking them hidden keeps the symbol usable within the
+        // device image's linkage unit while letting LTO drop it when nothing
+        // references it; symbols that must stay reachable (e.g. via an offload
+        // entry that takes their address) are kept alive by that reference.
+        if (!llvmFunc->isDeclaration() && llvmFunc->hasExternalLinkage() &&
+            llvmFunc->getVisibility() == llvm::GlobalValue::DefaultVisibility)
+          llvmFunc->setVisibility(llvm::GlobalValue::HiddenVisibility);
       }
     }
     return success();
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
new file mode 100644
index 0000000000000..f5f3c9f419f0b
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
@@ -0,0 +1,38 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Device-side `declare target` functions are externally visible by default so
+// that they can be referenced from other device translation units. They are
+// emitted with hidden visibility so that the offload LTO can internalize and
+// delete them when they turn out to be unused in the final device image. This
+// avoids leaving dead functions behind that may, for example, reference LDS and
+// trigger spurious backend diagnostics.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: define hidden void @device_any()
+  llvm.func @device_any() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // CHECK: define hidden void @device_nohost()
+  llvm.func @device_nohost() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function with an explicitly requested (non-default) visibility is left
+  // untouched.
+  // CHECK: define protected void @device_protected()
+  llvm.func protected @device_protected() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function that is not declare target is unaffected.
+  // CHECK: define void @not_declare_target()
+  llvm.func @not_declare_target() {
+    llvm.return
+  }
+
+  // A declaration (no definition) is left untouched: there is nothing to
+  // internalize, and hiding it could over-constrain the symbol's visibility.
+  // CHECK: declare void @device_decl()
+  llvm.func @device_decl() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>}
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
index cdebebc3ed233..6fccc0127b347 100644
--- a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
 module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memory_space", 5 : ui32>>, llvm.data_layout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9", llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
-  // CHECK-LABEL: define void @device_shared_mem(
+  // CHECK-LABEL: define hidden void @device_shared_mem(
   // CHECK-SAME:  i32 %[[N0:.*]], i64 %[[N1:.*]])
   llvm.func @device_shared_mem(%n0: i32, %n1: i64) attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
     // CHECK:      %[[CAST_N0:.*]] = zext i32 %[[N0]] to i64
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
index d84641ff9c99b..f120dde8131d6 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
@@ -22,7 +22,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr),
 // CHECK-SAME: ptr @[[COLLAPSED_WSLOOP_BODY_FN:.*]], ptr %[[STRUCT_ARG:.*]], i32 10000,
 // CHECK-SAME: i32 %[[NUM_THREADS:.*]], i8 0)
diff --git a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
index 7be635f46111b..04458af9654c3 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
@@ -31,7 +31,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
 // CHECK:   %[[STRUCTARG:.*]] = alloca { ptr }, align 8, addrspace(5)
 // CHECK:   %[[STRUCTARG_ASCAST:.*]] = addrspacecast ptr addrspace(5) %[[STRUCTARG]] to ptr
 // CHECK:   %[[GEP:.*]] = getelementptr { ptr }, ptr addrspace(5) %[[STRUCTARG]], i32 0, i32 0
@@ -45,7 +45,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
 // CHECK:   %[[GEP3:.*]] = getelementptr [10 x i32], ptr %[[LOADGEP]], i32 0, i32 %[[TMP2:.*]]
 // CHECK:   store i32 %[[VAL0:.*]], ptr %[[GEP3]], align 4
 
-// CHECK: define void @[[FUNC_EMPTY_WSLOOP:.*]]()
+// CHECK: define hidden void @[[FUNC_EMPTY_WSLOOP:.*]]()
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr), ptr @[[LOOP_EMPTY_BODY_FN:.*]], ptr null, i32 10, i32 %[[NUM_THREADS:.*]], i32 0, i8 0)
 
 // CHECK: define internal void @[[LOOP_EMPTY_BODY_FN]](i32 %[[LOOP_CNT:.*]])
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 790c0dfe9b12e..444918fb1a9e3 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3661,7 +3661,7 @@ module attributes {omp.is_target_device = false} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3683,7 +3683,7 @@ module attributes {omp.is_target_device = true} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3993,7 +3993,7 @@ llvm.func @omp_groupprivate_device() attributes {
 // CHECK-DAG: @nohost = internal global i32 undef
 // CHECK-DAG: @[[SHARED_ANY:any.*]] = internal addrspace(3) global i32 poison
 // CHECK-DAG: @[[SHARED_NOHOST:nohost.*]] = internal addrspace(3) global i32 poison
-// CHECK: define void @omp_groupprivate_device()
+// CHECK: define hidden void @omp_groupprivate_device()
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_ANY]], align 4
 // CHECK: store i32 1, ptr @host, align 4
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_NOHOST]], align 4

``````````

</details>


https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Thu Jul  2 10:19:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db8.510f576d.185953.1669@mx.google.com>


================

----------------
banach-space wrote:

Note, this comment refers to "copy", but you have no access to `memref.copy` in this method. 

What this is checking is that you never end-up with an RC like this:
```mlir
 %rc = memref.reinterpret_cast %dst
    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
    : memref<1x3x11xf32>
      to memref<1x3x2xf32, strided<[33, 11, 1], offset: 10>>
```
(Note, with `offset == 10`, there's only `1` element available in the trailing dimension).

I think that you can rephrase the comment and assert as follows:

> Sanity check that the reinterpret_cast doesn't create an out-of-bounds MemRef. Such cases should probably be rejected by Op verifier.

(please take a note of the example)

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db8.fdc477d6.118e5e.d2c9@mx.google.com>


================

----------------
banach-space wrote:

What if K == 1? Also, instead of `N * M`, I would use some other letter (e.g. `L`) - the actual value of this top stride is irrelevant, IIUC.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.fdc477d6.118e5e.d2dc@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
    // Track result dimensions that produce varying indices; unit dimensions are
    // always indexed at 0.
    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
      if (resultSize != 1)
        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
    }

    bool isScalarRes =
        llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });

    // For non-scalar results, verify that strides do not introduce
    // non-contiguity that would require extra logic.
    if (!isScalarRes) {
      SmallVector<int64_t> srcIdentityStrides =
          computeStrides(srcType.getShape());
      ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();

      if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                        [](auto pair) {
                          auto [srcStride, resultStride] = pair;
                          return !ShapedType::isDynamic(resultStride) &&
                                 srcStride == resultStride;
                        }))
        return std::nullopt;
    }
```

Explanation:
* The loop to compute non-unit dimension can be safely hoisted outside ` if (!isScalarCopy) {}`.
* ` assert((srcIdentityStrides.size() == rcResultStrides.size()) ` is not required - further up you verify that `if (srcType.getRank() != resType.getRank())`. If ranks match, so do strides. Otherwise MemRefs would be ill-formed.
* `!isScalarCopy)` -> `isScalarRes`. We shouldn't mention `Copy` in this method. Apologies for that poor suggestion in my previous review.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db8.481afadf.3c64c3.7268@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
/// for effectively-1D sources. Returns nullopt for unsupported reinterpret_casts.
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.1a4df1a5.118deb.267d@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
  // With an effectively-1D source, a dynamic offset can be mapped to the unique non-unit dim.
  // For other cases, bail out.
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.576b3af0.2cb9b7.91df@mx.google.com>


================

----------------
banach-space wrote:

I'm not sure I follow why this assert holds :(

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.bb839515.23e6d3.1453@mx.google.com>


================

----------------
banach-space wrote:

This example also, in general, could be OOB.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.da8bafd3.1eab2b.83d2@mx.google.com>


================

----------------
banach-space wrote:

This is a variation of `@copy_1D_into_2D_strided_nonzero_offset`. I would rename them as:
* `@copy_1D_into_2D_strided_nonzero_offset` -> `@copy_1D_into_2D_strided_nonzero_offset_delinierized_v1`
* `@copy_1D_into_2D_strided_delinearized_offset` -> `@copy_1D_into_2D_strided_nonzero_offset_delinearized_v2`

And then, add comments to explain the actual difference between the two. 

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469db9.1a4df1a5.118deb.267e@mx.google.com>


================

----------------
banach-space wrote:

IIUC, if the offset was `8`, the copy below would be OOB. Is that correct? If yes, could you add a comment for that? Also, perhaps a FIXME:
```
// FIXME: Add run-time verification for cases like this.
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:19:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:19:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a469dbb.492e23e4.e71ca.9406@mx.google.com>


================

----------------
banach-space wrote:

Expand - positive tests for which pattern?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Thu Jul  2 10:48:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:48:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46a463.27b3b003.1610ae.739f@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 10:48:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:48:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46a463.cb30c3e4.6f38b.afee@mx.google.com>


================
@@ -0,0 +1,366 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// Consumer fusion - linalg.pack with scalable inner tiles. Producer step (8*vscale)
+// equals the pack inner tile size(8*vscale) on the tiled source dimension, so the
+// outer dim of the fused pack tile is statically 1. This information is passed as
+// an inner tile alignment hint `Equal`.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_equal
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_equal(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // Loop tile size is equal to the inner tile size of the consumer `linalg.pack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //      CHECK:    %[[C8:.*]] = arith.constant 8 : index
+  //      CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //      CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %{{.*}}]
+  // CHECK-SAME:          -> tensor<1x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion with a static producer step (64) and a scalable pack inner
+// tile (8*vscale), hinted `Multiple`. Fusion honors the hint and takes the aligned 
+// (non-equal) path and the outer dim of the fused pack tile is dynamic (`64 ceildiv 8*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<()[s0] -> (64 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c64 = arith.constant 64 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c64 iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %ext_out = tensor.extract_slice %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<64x128xf32>, tensor<64x128xf32>)
+        outs(%ext_out : tensor<64x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<64x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<64x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size 64
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[C64:.*]] = arith.constant 64 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C64]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]]()[%[[C8_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
----------------
banach-space wrote:

What's the tile size?

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 10:48:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:48:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46a463.abb10101.208f3e.bc6a@mx.google.com>


================
@@ -0,0 +1,122 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=equal -canonicalize -cse | FileCheck %s --check-prefixes=CHECK,EQUAL
----------------
banach-space wrote:

[nit] `CHECK` --> `ALL` 

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 10:48:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:48:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46a463.e9d484b9.11ad58.7f21@mx.google.com>


================
@@ -0,0 +1,366 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// Consumer fusion - linalg.pack with scalable inner tiles. Producer step (8*vscale)
+// equals the pack inner tile size(8*vscale) on the tiled source dimension, so the
+// outer dim of the fused pack tile is statically 1. This information is passed as
+// an inner tile alignment hint `Equal`.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_equal
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_equal(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // Loop tile size is equal to the inner tile size of the consumer `linalg.pack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //      CHECK:    %[[C8:.*]] = arith.constant 8 : index
+  //      CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //      CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %{{.*}}]
+  // CHECK-SAME:          -> tensor<1x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion with a static producer step (64) and a scalable pack inner
+// tile (8*vscale), hinted `Multiple`. Fusion honors the hint and takes the aligned 
+// (non-equal) path and the outer dim of the fused pack tile is dynamic (`64 ceildiv 8*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<()[s0] -> (64 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c64 = arith.constant 64 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c64 iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %ext_out = tensor.extract_slice %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<64x128xf32>, tensor<64x128xf32>)
+        outs(%ext_out : tensor<64x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<64x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<64x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size 64
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[C64:.*]] = arith.constant 64 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C64]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]]()[%[[C8_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
+  // CHECK-SAME:          -> tensor<?x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion - both producer step and pack inner tile are scalable, step (8*vscale)
+// is an integer multiple of the inner tile (4*vscale) but not equal to it. The corresponding
+// `Multiple` hint is passed to the tiling interface. Fusion succeeds via the aligned (non-equal)
+// path, so the outer dim of the fused pack tile stays dynamic (`8*vscale ceildiv 4*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_MIN:.+]] = affine_map<(d0)[s0] -> (-d0 + 256, s0)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned_scalable
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned_scalable(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+  
+  // Loop tile size (8 * vscale) is a multiple of the inner tile size of the consumer `linalg.pack` (4 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c4_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size
+    // 8 * vscale and the inner tile size 4 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[SZ:.*]] = affine.min #[[$MAP_MIN]](%{{.*}})[%[[C8_VSCALE]]]
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]](%[[SZ]])[%[[C4_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
----------------
banach-space wrote:

What's the tile size?

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 10:48:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 10:48:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a46a463.7d41f3a1.d1cf7.4245@mx.google.com>


https://github.com/banach-space approved this pull request.

Thank you for the very comprehensive testing — quite impressive, and a good example to follow. It will be a great source of documentation for us in the future.

LGTM!

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Thu Jul  2 10:49:19 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Thu, 02 Jul 2026 10:49:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve source encoding when
 folding insert_slice canonicalizers (PR #207239)
Message-ID: <llvm.org/llvm/llvm-project/pull/207239@github.com>


https://github.com/Devjiu created https://github.com/llvm/llvm-project/pull/207239


InsertSliceOpConstantArgumentFolder currently re-derives the refined source type via ExtractSliceOp::inferCanonicalRankReducedResultType, which copies the encoding of the passed-in "source template" — which in this pattern is insertSliceOp.getDestType(). For a static, encoding-less destination this silently drops any encoding the actual source carried. Downstream dialects that stash mandatory metadata in the encoding attribute (upper bounds, layout, sparsity descriptors) lose that metadata during --canonicalize.

Preserve the original source's encoding by rebuilding the refined RankedTensorType with srcType.getEncoding() explicitly. Shape refinement (the `?` -> static direction the pattern already performs) is unchanged and still satisfies preservesStaticInformation.

lit created with Claude Opus 4.7

>From 58b03acba8abd581f08554c4600b03d94031330a Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Thu, 2 Jul 2026 16:28:11 +0000
Subject: [PATCH] [mlir][tensor] Preserve source encoding when folding
 insert_slice canonicalizers
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

InsertSliceOpConstantArgumentFolder currently re-derives the refined source
type via ExtractSliceOp::inferCanonicalRankReducedResultType, which copies
the encoding of the passed-in "source template" — which in this pattern is
insertSliceOp.getDestType(). For a static, encoding-less destination this
silently drops any encoding the actual source carried. Downstream dialects
that stash mandatory metadata in the encoding attribute (upper bounds,
layout, sparsity descriptors) lose that metadata during --canonicalize.

Preserve the original source's encoding by rebuilding the refined
RankedTensorType with srcType.getEncoding() explicitly. Shape refinement
(the `?` -> static direction the pattern already performs) is unchanged
and still satisfies preservesStaticInformation.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 12 +++++-
 mlir/test/Dialect/Tensor/canonicalize.mlir | 48 ++++++++++++++++++++++
 2 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..8dadc83fdc548 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -3014,10 +3014,18 @@ class InsertSliceOpConstantArgumentFolder final
     if (!sliceResult.isValid)
       return failure();
 
-    // Create the new op in canonical form.
-    auto sourceType = ExtractSliceOp::inferCanonicalRankReducedResultType(
+    // Create the new op in canonical form. The refined shape is inferred from
+    // the destination type, but the encoding is a per-value property of the
+    // source and must be preserved: insert_slice does not convert between
+    // encodings, so the source's encoding is what the produced cast/op must
+    // carry (dropping it would silently discard downstream metadata such as
+    // bounds, layout, or sparsity descriptors).
+    auto sourceTypeBase = ExtractSliceOp::inferCanonicalRankReducedResultType(
         insertSliceOp.getSourceType().getRank(), insertSliceOp.getDestType(),
         mixedSizes);
+    auto sourceType = RankedTensorType::get(
+        sourceTypeBase.getShape(), sourceTypeBase.getElementType(),
+        insertSliceOp.getSourceType().getEncoding());
     Value toInsert = insertSliceOp.getSource();
     if (sourceType != insertSliceOp.getSourceType()) {
       OpBuilder::InsertionGuard g(rewriter);
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..df92d3ccfd82b 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -937,6 +937,54 @@ func.func @insert_slice_cast_no_fold(%arg0 : tensor<1x?xf32>, %arg1 : tensor<?x?
 
 // -----
 
+// Verify that the constant-argument folder for insert_slice preserves the
+// source's encoding on the inserted cast, rather than silently picking up the
+// destination's encoding (which is `none` here) via the shape template used by
+// ExtractSliceOp::inferCanonicalRankReducedResultType.
+// CHECK-LABEL: func @preserve_source_encoding_on_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.insert_slice %[[C]] into %[[DST]]
+func.func @preserve_source_encoding_on_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %ivC = arith.constant 0 : index
+  %ivH = arith.constant 0 : index
+  %r = tensor.insert_slice %src into %dst[0, %ivC, %ivH, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+      : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
+// Same invariant for the parallel_insert_slice variant.
+// CHECK-LABEL: func @preserve_source_encoding_on_parallel_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.parallel_insert_slice %[[C]] into
+func.func @preserve_source_encoding_on_parallel_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>,
+    %num_threads: index) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %r = scf.forall (%tid) in (%num_threads) shared_outs(%o = %dst) -> (tensor<1x1280x32x32xf16>) {
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %src into %o[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+          : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+    }
+  }
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @insert_tensor_cast_on_insert_slice_src(
 // CHECK-SAME:      %[[arg0:.*]]: tensor<?x5x?xf32>, %[[arg1:.*]]: tensor<?x?x?xf32>
 //      CHECK:    %[[cast:.*]] = tensor.cast %[[arg0]] : tensor<?x5x?xf32> to tensor<64x5x64xf32>


From llvmlistbot at llvm.org  Thu Jul  2 10:49:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 10:49:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve source encoding when
 folding insert_slice canonicalizers (PR #207239)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207239@github.com>
Message-ID: <6a46a4c7.bb839515.23e6d3.88ed@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-tensor

Author: Dmitrii Makarenko (Devjiu)

<details>
<summary>Changes</summary>


InsertSliceOpConstantArgumentFolder currently re-derives the refined source type via ExtractSliceOp::inferCanonicalRankReducedResultType, which copies the encoding of the passed-in "source template" — which in this pattern is insertSliceOp.getDestType(). For a static, encoding-less destination this silently drops any encoding the actual source carried. Downstream dialects that stash mandatory metadata in the encoding attribute (upper bounds, layout, sparsity descriptors) lose that metadata during --canonicalize.

Preserve the original source's encoding by rebuilding the refined RankedTensorType with srcType.getEncoding() explicitly. Shape refinement (the `?` -&gt; static direction the pattern already performs) is unchanged and still satisfies preservesStaticInformation.

lit created with Claude Opus 4.7

---
Full diff: https://github.com/llvm/llvm-project/pull/207239.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Tensor/IR/TensorOps.cpp (+10-2) 
- (modified) mlir/test/Dialect/Tensor/canonicalize.mlir (+48) 


``````````diff
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..8dadc83fdc548 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -3014,10 +3014,18 @@ class InsertSliceOpConstantArgumentFolder final
     if (!sliceResult.isValid)
       return failure();
 
-    // Create the new op in canonical form.
-    auto sourceType = ExtractSliceOp::inferCanonicalRankReducedResultType(
+    // Create the new op in canonical form. The refined shape is inferred from
+    // the destination type, but the encoding is a per-value property of the
+    // source and must be preserved: insert_slice does not convert between
+    // encodings, so the source's encoding is what the produced cast/op must
+    // carry (dropping it would silently discard downstream metadata such as
+    // bounds, layout, or sparsity descriptors).
+    auto sourceTypeBase = ExtractSliceOp::inferCanonicalRankReducedResultType(
         insertSliceOp.getSourceType().getRank(), insertSliceOp.getDestType(),
         mixedSizes);
+    auto sourceType = RankedTensorType::get(
+        sourceTypeBase.getShape(), sourceTypeBase.getElementType(),
+        insertSliceOp.getSourceType().getEncoding());
     Value toInsert = insertSliceOp.getSource();
     if (sourceType != insertSliceOp.getSourceType()) {
       OpBuilder::InsertionGuard g(rewriter);
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..df92d3ccfd82b 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -937,6 +937,54 @@ func.func @insert_slice_cast_no_fold(%arg0 : tensor<1x?xf32>, %arg1 : tensor<?x?
 
 // -----
 
+// Verify that the constant-argument folder for insert_slice preserves the
+// source's encoding on the inserted cast, rather than silently picking up the
+// destination's encoding (which is `none` here) via the shape template used by
+// ExtractSliceOp::inferCanonicalRankReducedResultType.
+// CHECK-LABEL: func @preserve_source_encoding_on_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.insert_slice %[[C]] into %[[DST]]
+func.func @preserve_source_encoding_on_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %ivC = arith.constant 0 : index
+  %ivH = arith.constant 0 : index
+  %r = tensor.insert_slice %src into %dst[0, %ivC, %ivH, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+      : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
+// Same invariant for the parallel_insert_slice variant.
+// CHECK-LABEL: func @preserve_source_encoding_on_parallel_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.parallel_insert_slice %[[C]] into
+func.func @preserve_source_encoding_on_parallel_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>,
+    %num_threads: index) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %r = scf.forall (%tid) in (%num_threads) shared_outs(%o = %dst) -> (tensor<1x1280x32x32xf16>) {
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %src into %o[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+          : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+    }
+  }
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @insert_tensor_cast_on_insert_slice_src(
 // CHECK-SAME:      %[[arg0:.*]]: tensor<?x5x?xf32>, %[[arg1:.*]]: tensor<?x?x?xf32>
 //      CHECK:    %[[cast:.*]] = tensor.cast %[[arg0]] : tensor<?x5x?xf32> to tensor<64x5x64xf32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207239

From llvmlistbot at llvm.org  Thu Jul  2 10:49:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 10:49:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve source encoding when
 folding insert_slice canonicalizers (PR #207239)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207239@github.com>
Message-ID: <6a46a4c7.481afadf.3c64c3.7213@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Dmitrii Makarenko (Devjiu)

<details>
<summary>Changes</summary>


InsertSliceOpConstantArgumentFolder currently re-derives the refined source type via ExtractSliceOp::inferCanonicalRankReducedResultType, which copies the encoding of the passed-in "source template" — which in this pattern is insertSliceOp.getDestType(). For a static, encoding-less destination this silently drops any encoding the actual source carried. Downstream dialects that stash mandatory metadata in the encoding attribute (upper bounds, layout, sparsity descriptors) lose that metadata during --canonicalize.

Preserve the original source's encoding by rebuilding the refined RankedTensorType with srcType.getEncoding() explicitly. Shape refinement (the `?` -&gt; static direction the pattern already performs) is unchanged and still satisfies preservesStaticInformation.

lit created with Claude Opus 4.7

---
Full diff: https://github.com/llvm/llvm-project/pull/207239.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Tensor/IR/TensorOps.cpp (+10-2) 
- (modified) mlir/test/Dialect/Tensor/canonicalize.mlir (+48) 


``````````diff
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..8dadc83fdc548 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -3014,10 +3014,18 @@ class InsertSliceOpConstantArgumentFolder final
     if (!sliceResult.isValid)
       return failure();
 
-    // Create the new op in canonical form.
-    auto sourceType = ExtractSliceOp::inferCanonicalRankReducedResultType(
+    // Create the new op in canonical form. The refined shape is inferred from
+    // the destination type, but the encoding is a per-value property of the
+    // source and must be preserved: insert_slice does not convert between
+    // encodings, so the source's encoding is what the produced cast/op must
+    // carry (dropping it would silently discard downstream metadata such as
+    // bounds, layout, or sparsity descriptors).
+    auto sourceTypeBase = ExtractSliceOp::inferCanonicalRankReducedResultType(
         insertSliceOp.getSourceType().getRank(), insertSliceOp.getDestType(),
         mixedSizes);
+    auto sourceType = RankedTensorType::get(
+        sourceTypeBase.getShape(), sourceTypeBase.getElementType(),
+        insertSliceOp.getSourceType().getEncoding());
     Value toInsert = insertSliceOp.getSource();
     if (sourceType != insertSliceOp.getSourceType()) {
       OpBuilder::InsertionGuard g(rewriter);
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..df92d3ccfd82b 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -937,6 +937,54 @@ func.func @insert_slice_cast_no_fold(%arg0 : tensor<1x?xf32>, %arg1 : tensor<?x?
 
 // -----
 
+// Verify that the constant-argument folder for insert_slice preserves the
+// source's encoding on the inserted cast, rather than silently picking up the
+// destination's encoding (which is `none` here) via the shape template used by
+// ExtractSliceOp::inferCanonicalRankReducedResultType.
+// CHECK-LABEL: func @preserve_source_encoding_on_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.insert_slice %[[C]] into %[[DST]]
+func.func @preserve_source_encoding_on_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %ivC = arith.constant 0 : index
+  %ivH = arith.constant 0 : index
+  %r = tensor.insert_slice %src into %dst[0, %ivC, %ivH, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+      : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
+// Same invariant for the parallel_insert_slice variant.
+// CHECK-LABEL: func @preserve_source_encoding_on_parallel_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.parallel_insert_slice %[[C]] into
+func.func @preserve_source_encoding_on_parallel_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>,
+    %num_threads: index) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %r = scf.forall (%tid) in (%num_threads) shared_outs(%o = %dst) -> (tensor<1x1280x32x32xf16>) {
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %src into %o[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+          : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+    }
+  }
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @insert_tensor_cast_on_insert_slice_src(
 // CHECK-SAME:      %[[arg0:.*]]: tensor<?x5x?xf32>, %[[arg1:.*]]: tensor<?x?x?xf32>
 //      CHECK:    %[[cast:.*]] = tensor.cast %[[arg0]] : tensor<?x5x?xf32> to tensor<64x5x64xf32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207239

From llvmlistbot at llvm.org  Thu Jul  2 10:59:22 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Thu, 02 Jul 2026 10:59:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
Message-ID: <llvm.org/llvm/llvm-project/pull/207241@github.com>


https://github.com/Devjiu created https://github.com/llvm/llvm-project/pull/207241

This commit fixes encoding drop in similar with InsertSliceOpConstantArgumentFolder cases. The same anti-pattern — silently producing a `RankedTensorType` without an encoding while a source value with an encoding flows through the refined type — is present in several other canonicalizers/inferResultType implementations:

* ConcatOp::inferResultType built the result type with no encoding. When all inputs share an encoding, propagate it; when they differ, drop it (concat semantics are undefined across differing encodings, so we do not pick one).

* InferConcatOperandTypes built per-operand refined types with the result element type but no encoding. The cast targets an operand value, so it must carry that operand's own encoding.

* CollapseShapeOp::inferCollapsedType dropped the encoding. Since collapse changes the tensor rank, `VerifiableTensorEncoding` attrs (e.g. sparse tensor encoding) would become invalid on the new shape; those are dropped as before. Encodings that do not implement `VerifiableTensorEncoding` (opaque dict/string/bounds attrs used by downstream dialects) are rank-agnostic and are now preserved.

* ConvertToStaticExpandShape built the inserted `tensor.cast` target and the refined `expand_shape` result with no encoding. Both types preserve the rank of the value flowing through them, so they now carry the source's and result's encodings respectively.

* PadOp::inferResultType dropped the encoding. Pad preserves rank, so the source encoding stays valid. FoldSourceTensorCast and FoldStaticPadding inherit the fix through inferResultType; FoldStaticPadding additionally rebuilt the type directly and is updated to propagate the pad's result encoding.

>From 2d3a314f3d670b262df33319ce0cb6b7e223d469 Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Thu, 2 Jul 2026 16:32:59 +0000
Subject: [PATCH] [mlir][tensor] Preserve encoding in more canonicalizers
 (concat, reshape, pad)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This commit fixes encoding drop in similar with InsertSliceOpConstantArgumentFolder cases.
The same anti-pattern — silently producing a `RankedTensorType` without an
encoding while a source value with an encoding flows through the
refined type — is present in several other canonicalizers/inferResultType
implementations:

* ConcatOp::inferResultType built the result type with no encoding. When all
  inputs share an encoding, propagate it; when they differ, drop it (concat
  semantics are undefined across differing encodings, so we do not pick one).

* InferConcatOperandTypes built per-operand refined types with the result
  element type but no encoding. The cast targets an operand value, so it
  must carry that operand's own encoding.

* CollapseShapeOp::inferCollapsedType dropped the encoding. Since collapse
  changes the tensor rank, `VerifiableTensorEncoding` attrs (e.g. sparse
  tensor encoding) would become invalid on the new shape; those are dropped
  as before. Encodings that do not implement `VerifiableTensorEncoding`
  (opaque dict/string/bounds attrs used by downstream dialects) are
  rank-agnostic and are now preserved.

* ConvertToStaticExpandShape built the inserted `tensor.cast` target and
  the refined `expand_shape` result with no encoding. Both types preserve
  the rank of the value flowing through them, so they now carry the source's
  and result's encodings respectively.

* PadOp::inferResultType dropped the encoding. Pad preserves rank, so the
  source encoding stays valid. FoldSourceTensorCast and FoldStaticPadding
  inherit the fix through inferResultType; FoldStaticPadding additionally
  rebuilt the type directly and is updated to propagate the pad's result
  encoding.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 45 ++++++++---
 mlir/test/Dialect/Tensor/canonicalize.mlir | 93 +++++++++++++++++++++-
 2 files changed, 125 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..2406df1db81d0 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -24,6 +24,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TensorEncoding.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/DestinationStyleOpInterface.h"
 #include "mlir/Interfaces/InferIntRangeInterface.h"
@@ -577,7 +578,17 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  return RankedTensorType::get(sizes, tensorTypes[0].getElementType());
+  // Preserve the encoding when all inputs share it; otherwise drop it (the
+  // semantics of concatenating tensors with differing encodings are undefined
+  // at this level, so we don't try to pick one).
+  Attribute encoding = tensorTypes[0].getEncoding();
+  for (auto tensorType : llvm::drop_begin(tensorTypes)) {
+    if (tensorType.getEncoding() != encoding) {
+      encoding = Attribute();
+      break;
+    }
+  }
+  return RankedTensorType::get(sizes, tensorTypes[0].getElementType(), encoding);
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -811,11 +822,14 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
     SmallVector<int64_t> inferredOperandShape(inferredResultType.getShape());
     for (auto [operandIdx, operandType] :
          llvm::enumerate(concatOp->getOperandTypes())) {
-      // Compute inferred type for operand.
-      inferredOperandShape[dim] =
-          cast<RankedTensorType>(operandType).getDimSize(dim);
+      // Compute inferred type for operand. The refined type is applied to the
+      // operand itself, so it must carry the operand's own encoding rather
+      // than the (potentially different or missing) result encoding.
+      auto operandRankedType = cast<RankedTensorType>(operandType);
+      inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, inferredResultType.getElementType());
+          inferredOperandShape, inferredResultType.getElementType(),
+          operandRankedType.getEncoding());
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2023,7 +2037,10 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  return RankedTensorType::get(newShape, type.getElementType());
+  Attribute encoding = type.getEncoding();
+  if (llvm::isa_and_present<VerifiableTensorEncoding>(encoding))
+    encoding = {};
+  return RankedTensorType::get(newShape, type.getElementType(), encoding);
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
@@ -2273,10 +2290,13 @@ struct ConvertToStaticExpandShape : public OpRewritePattern<ExpandShapeOp> {
 
     SmallVector<OpFoldResult> outputOfr =
         getMixedValues(newOutputShape, dynamicOutputShape, rewriter);
+    // The refined types keep the ranks of the src / result respectively
     auto inputType = RankedTensorType::get(
-        newInputShape, expandOp.getSrcType().getElementType());
+        newInputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getSrcType().getEncoding());
     auto outputType = RankedTensorType::get(
-        newOutputShape, expandOp.getSrcType().getElementType());
+        newOutputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getResultType().getEncoding());
     auto inputCast = CastOp::create(rewriter, expandOp.getLoc(), inputType,
                                     expandOp.getSrc());
     auto newExpand = ExpandShapeOp::create(
@@ -2385,7 +2405,7 @@ RankedTensorType ExtractSliceOp::inferCanonicalRankReducedResultType(
       if (!dimsToProject.test(pos))
         projectedShape.push_back(shape[pos]);
     inferredType =
-        RankedTensorType::get(projectedShape, inferredType.getElementType());
+        RankedTensorType::get(projectedShape, inferredType.getElementType(), inferredType.getEncoding());
   }
   return inferredType;
 }
@@ -3323,7 +3343,8 @@ RankedTensorType PadOp::inferResultType(RankedTensorType sourceType,
     }
   }
 
-  return RankedTensorType::get(inferredShape, sourceType.getElementType());
+  return RankedTensorType::get(inferredShape, sourceType.getElementType(),
+                               sourceType.getEncoding());
 }
 
 void PadOp::build(OpBuilder &b, OperationState &result, Type resultType,
@@ -3730,9 +3751,9 @@ struct FoldStaticPadding : public OpRewritePattern<PadOp> {
                      [&](int64_t x) { return x == ShapedType::kDynamic; }))
       return failure();
 
-    // Rewrite the op using the new static type.
     auto newResultType = RankedTensorType::get(
-        newOutDims, padTensorOp.getType().getElementType());
+        newOutDims, padTensorOp.getType().getElementType(),
+        padTensorOp.getType().getEncoding());
     auto newOp = PadOp::create(
         rewriter, padTensorOp->getLoc(), newResultType, input, staticLow,
         staticHigh, newLows, newHighs, padTensorOp.getNofold(),
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..c5fc71e6c7bd5 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -162,6 +162,22 @@ func.func @infer_concat_return_type(%arg0: tensor<5x12xi32>, %arg1: tensor<?x12x
 
 // -----
 
+// ConcatOp::inferResultType must carry the (uniformly-shared) operand encoding
+// onto both the refined operand cast and the refined ConcatOp result.
+// CHECK-LABEL: concat_preserves_uniform_encoding
+//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?xi32, "abc">
+//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?xi32, "abc">
+//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?xi32, "abc"> to tensor<3x?xi32, "abc">
+//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]] : (tensor<3x?xi32, "abc">, tensor<3x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+func.func @concat_preserves_uniform_encoding(
+    %a: tensor<3x?xi32, "abc">, %b: tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc"> {
+  %r = tensor.concat dim(1) %a, %b
+      : (tensor<3x?xi32, "abc">, tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+  return %r : tensor<3x?xi32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1092,6 +1108,22 @@ func.func @collapse_of_cast(%t: tensor<8x12x32xf32>) -> tensor<?x32xf32> {
 
 // -----
 
+// A user-defined (non-VerifiableTensorEncoding) encoding must be preserved
+// through the collapse_of_cast folder; inferCollapsedType propagates it
+// alongside the refined shape.
+// CHECK-LABEL: func.func @collapse_of_cast_preserves_encoding(
+//  CHECK-SAME:     %[[IN:.*]]: tensor<8x12x32xf32, "abc">
+//       CHECK:   %[[COLLAPSE:.*]] = tensor.collapse_shape %[[IN]] {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, "abc"> into tensor<96x32xf32, "abc">
+//       CHECK:   tensor.cast %[[COLLAPSE]] : tensor<96x32xf32, "abc"> to tensor<?x32xf32>
+func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -> tensor<?x32xf32> {
+  %0 = tensor.cast %t : tensor<8x12x32xf32, "abc"> to tensor<?x?x?xf32, "abc">
+  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, "abc"> into tensor<?x?xf32, "abc">
+  %2 = tensor.cast %1 : tensor<?x?xf32, "abc"> to tensor<?x32xf32>
+  return %2 : tensor<?x32xf32>
+}
+
+// -----
+
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>
@@ -1773,6 +1805,28 @@ func.func @pad_nofold_same_static_shape(%arg0: tensor<5x6xf32>, %a: index)
 
 // -----
 
+// FoldSourceTensorCast (via PadOp::inferResultType) must preserve the source
+// encoding onto the refined pad result and the inserted result cast.
+// CHECK-LABEL:   func @pad_after_cast_preserves_encoding(
+//  CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[PADDED:.*]] = tensor.pad %[[INPUT]]
+//       CHECK:       : tensor<?x64x?x?xf32, "abc"> to tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[CAST:.*]] = tensor.cast %[[PADDED]] : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+//       CHECK:     return %[[CAST]]
+func.func @pad_after_cast_preserves_encoding(
+    %arg0: tensor<?x64x?x?xf32, "abc">) -> tensor<?x?x?x?xf32, "abc"> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %dynamic = tensor.cast %arg0
+      : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  %padded = tensor.pad %dynamic low[0, 0, 1, 1] high[0, 0, 1, 1] {
+    ^bb0(%a: index, %b: index, %c: index, %d: index):
+      tensor.yield %cst: f32
+  } : tensor<?x?x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  return %padded: tensor<?x?x?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL:   func @pad_after_cast_different_shape(
 // CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32>) -> tensor<?x?x?x?xf32> {
 // CHECK:           %[[CST:.*]] = arith.constant 0.000000e+00 : f32
@@ -1921,6 +1975,26 @@ func.func @pad_static_zero_cast(%arg0: tensor<?x?x?xf32>, %pad_value: f32) -> te
 
 // -----
 
+// FoldStaticPadding must preserve the pad's result encoding on the refined
+// (more-static) pad and the inserted result cast.
+// CHECK-LABEL: func @fold_static_padding_preserves_encoding(
+//  CHECK-SAME:     %[[SRC:.*]]: tensor<8x?xf32, "abc">
+//       CHECK:   %[[PADDED:.*]] = tensor.pad %[[SRC]] low[1, 2] high[1, 2]
+//       CHECK:     : tensor<8x?xf32, "abc"> to tensor<10x?xf32, "abc">
+//       CHECK:   tensor.cast %[[PADDED]] : tensor<10x?xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @fold_static_padding_preserves_encoding(
+    %arg0: tensor<8x?xf32, "abc">, %pv: f32) -> tensor<?x?xf32, "abc"> {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %r = tensor.pad %arg0 low[%c1, %c2] high[%c1, %c2] {
+    ^bb0(%a: index, %b: index):
+      tensor.yield %pv: f32
+  } : tensor<8x?xf32, "abc"> to tensor<?x?xf32, "abc">
+  return %r : tensor<?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @pad_nofold_static_zero(
 //  CHECK-SAME:                  %[[ARG0:.*]]: tensor<?x?x?xf32>
 //       CHECK:   %[[PAD:.*]] = tensor.pad
@@ -2641,8 +2715,25 @@ func.func @partial_sink_expand_of_cast(%arg0 : tensor<10x10xf32>, %arg1 : index,
 // CHECK-LABEL:  func.func @partial_sink_expand_of_cast
 //       CHECK:   %[[CAST:.+]] = tensor.cast
 //  CHECK-SAME:     tensor<10x10xf32> to tensor<?x10xf32>
-//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]] 
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]]
 //  CHECK-SAME:     output_shape [%{{.*}}, %{{.*}}, 10]
 //       CHECK:   %[[RES:.+]] = tensor.cast %[[EXPAND]]
 //  CHECK-SAME:     tensor<?x?x10xf32> to tensor<?x?x?xf32>
 //       CHECK:   return %[[RES]]
+
+// -----
+
+// ConvertToStaticExpandShape must carry the source's encoding onto the refined
+// cast-target and the (source-side) encoding of the new ExpandShapeOp result.
+// CHECK-LABEL:  func.func @sink_expand_of_cast_preserves_encoding
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape
+//  CHECK-SAME:     tensor<64xf32, "abc"> into tensor<8x8xf32, "abc">
+//       CHECK:   tensor.cast %[[EXPAND]] : tensor<8x8xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @sink_expand_of_cast_preserves_encoding(%t: tensor<64xf32, "abc">) -> tensor<?x?xf32, "abc"> {
+  %c = tensor.cast %t : tensor<64xf32, "abc"> to tensor<?xf32, "abc">
+  %c8 = arith.constant 8 : index
+  %c8b = arith.constant 8 : index
+  %e = tensor.expand_shape %c [[0, 1]] output_shape [%c8, %c8b]
+      : tensor<?xf32, "abc"> into tensor<?x?xf32, "abc">
+  return %e : tensor<?x?xf32, "abc">
+}


From llvmlistbot at llvm.org  Thu Jul  2 10:59:58 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Thu, 02 Jul 2026 10:59:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a46a71e.70920dee.3514b2.cc85@mx.google.com>


https://github.com/Devjiu edited https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Thu Jul  2 11:47:02 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 02 Jul 2026 11:47:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reimplement
 SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
 (PR #204971)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204971@github.com>
Message-ID: <6a46b226.eb165e5c.113c2a.65e0@mx.google.com>


banach-space wrote:

> > > Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
> > 
> > 
> > Could you summarise the newly supported cases? Thank you!
> 
> @banach-space Do I misunderstand your sentence? Should I rewrite the commit message `title` to show the cases? Maybe:
> 
> ```
> Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for no transposition/padding semantic pack/unpack op.
> ```
> 
> original commit msg:
> 
> ```
> Reimplement SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
> 
> If there is no transposition/padding semantic for pack/unpack,
> we could try to use `mlir::getReassociationIndicesForReshape()`
> to get the CollapseShapeOp/ExpandShapeOp form.
> ```

Yes, something along those lines. Basically, I am struggling to see what is currently supported and what new functionality is added. I've not looked at those transformations for a while and some context/reminder would help 😅 

In general, I find the tests in mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir quite poorly structured - it's hard to tell negative from positive cases. Ideally, we should refactor that file following:
* https://mlir.llvm.org/getting_started/TestingGuide/#test-formatting-best-practices and
* https://mlir.llvm.org/getting_started/TestingGuide/#test-documentation-best-practices.

Specifically, any negative tests added here that would already fail without this change, should be submitted in a separate PR. Would you be open iterating a bit here? At the very least, we should clearly label the negative tests. If not, I can update the tests and then you can submit this as a follow-up. I would be more than happy to review your refactoring PR if you decided to that. Please let me know your preference.

https://github.com/llvm/llvm-project/pull/204971

From llvmlistbot at llvm.org  Thu Jul  2 11:56:21 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 11:56:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Fix memory leak in ConversionTarget
 dynamic legality test (PR #207253)
Message-ID: <llvm.org/llvm/llvm-project/pull/207253@github.com>


https://github.com/makslevental created https://github.com/llvm/llvm-project/pull/207253

## Summary
- Fixes LeakSanitizer failure from #206161 (reverted in #207104)
- mlirFreezeRewritePattern moves contents out of the RewritePatternSet but does not free the container (passed by value in the C API), so the allocation from mlirRewritePatternSetCreate was never freed
- Adds mlirRewritePatternSetDestroy(patterns) after freezing

## Test plan
- [x] Reproduced LSan failure on Linux with clang-23 -DLLVM_USE_SANITIZER=Address;Undefined
- [x] Verified fix eliminates the leak on Linux
- [x] Verified test still passes on macOS with ASAN/UBSAN

>From 8bf15918aa72044c927ce58c2698def85f7f4d31 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Thu, 2 Jul 2026 11:09:26 -0700
Subject: [PATCH 1/2] Reapply "[mlir-c] Add ConversionTarget dynamic legality C
 API" (#207104)

This reverts commit 92313eff069c3148153aa8097ea3780ebc8dc057.
---
 mlir/include/mlir-c/Rewrite.h        |  44 +++++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  60 +++++++++
 mlir/test/CAPI/rewrite.c             | 180 +++++++++++++++++++++++++++
 3 files changed, 284 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index ac243a4c9d8f9..3356e6f445e47 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -533,6 +533,50 @@ MLIR_CAPI_EXPORTED void
 mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
                                       MlirStringRef dialectName);
 
+/// Result of a dynamic legality callback.
+typedef enum {
+  /// The operation instance is legal.
+  MLIR_CONVERSION_TARGET_LEGALITY_LEGAL,
+  /// The operation instance is illegal.
+  MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL,
+  /// The callback has no opinion on this instance. The decision is deferred to
+  /// other registered callbacks (legality callbacks are composed) or, failing
+  /// that, to the operation's static legality action.
+  MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION
+} MlirConversionTargetLegality;
+
+/// Callback for dynamic legality checks. Returns the legality of the given
+/// operation instance (see MlirConversionTargetLegality).
+typedef MlirConversionTargetLegality (
+    *MlirConversionTargetDynamicLegalityCallback)(MlirOperation op,
+                                                  void *userData);
+
+/// Register the given operation as dynamically legal, with a callback to
+/// determine per-instance legality. The callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Register the given dialect as dynamically legal, with a callback to
+/// determine per-instance legality for all operations in the dialect. The
+/// callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark the given operation as recursively legal. The optional callback (may
+/// be NULL) determines whether a specific instance is recursively legal; a NULL
+/// callback marks the operation as unconditionally recursively legal.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark unknown operations as dynamically legal, with a callback. The callback
+/// must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 56ce9212f4811..083ed6f999ae3 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -23,6 +23,8 @@
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
 
+#include <cassert>
+
 using namespace mlir;
 
 //===----------------------------------------------------------------------===//
@@ -575,6 +577,64 @@ void mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
   unwrap(target)->addIllegalDialect(unwrap(dialectName));
 }
 
+namespace {
+/// Wraps a C dynamic-legality callback as a C++ DynamicLegalityCallbackFn,
+/// translating the tri-state MlirConversionTargetLegality result into the
+/// std::optional<bool> expected by ConversionTarget (NO_OPINION -> nullopt).
+ConversionTarget::DynamicLegalityCallbackFn
+wrapLegalityCallback(MlirConversionTargetDynamicLegalityCallback callback,
+                     void *userData) {
+  return [callback, userData](Operation *op) -> std::optional<bool> {
+    switch (callback(wrap(op), userData)) {
+    case MLIR_CONVERSION_TARGET_LEGALITY_LEGAL:
+      return true;
+    case MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL:
+      return false;
+    case MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION:
+      return std::nullopt;
+    }
+    llvm_unreachable("unknown MlirConversionTargetLegality");
+  };
+}
+} // namespace
+
+void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  unwrap(target)->addDynamicallyLegalOp(
+      name, wrapLegalityCallback(callback, userData));
+}
+
+void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->addDynamicallyLegalDialect(
+      wrapLegalityCallback(callback, userData), unwrap(dialectName));
+}
+
+void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  ConversionTarget::DynamicLegalityCallbackFn fn;
+  if (callback)
+    fn = wrapLegalityCallback(callback, userData);
+  unwrap(target)->markOpRecursivelyLegal(name, fn);
+}
+
+void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->markUnknownOpDynamicallyLegal(
+      wrapLegalityCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 3809dd6a7843f..439d1355af822 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -623,6 +623,185 @@ void testCloneWithMapping(MlirContext ctx) {
   fprintf(stderr, "testCloneWithMapping: PASSED\n");
 }
 
+static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
+                                                               void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
+}
+
+static MlirConversionTargetLegality
+dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
+}
+
+static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
+                                                             void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
+}
+
+// Runs a partial conversion of `moduleString` against `target` with an empty
+// pattern set and returns whether it succeeded. This is what actually drives
+// the registered dynamic-legality callbacks.
+static bool runPartialConversion(MlirContext ctx, const char *moduleString,
+                                 MlirConversionTarget target) {
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  assert(!mlirModuleIsNull(module) && "expected module to parse");
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+  MlirConversionConfig config = mlirConversionConfigCreate();
+
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+
+  mlirConversionConfigDestroy(config);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirModuleDestroy(module);
+
+  return mlirLogicalResultIsSuccess(result);
+}
+
+void testConversionTargetDynamicLegality(MlirContext ctx) {
+  // CHECK-LABEL: @testConversionTargetDynamicLegality
+  fprintf(stderr, "@testConversionTargetDynamicLegality\n");
+
+  const char *opModule = "\"dialect.op1\"() : () -> ()\n";
+
+  // addDynamicallyLegalOp: callback returning true makes the op legal, so the
+  // (pattern-free) partial conversion succeeds and the callback is invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp: callback returning false makes the op illegal. With
+  // no pattern to legalize it, the partial conversion fails -- proving the
+  // callback's return value actually drives the result.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &counter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp composition: callbacks registered for the same op are
+  // chained, most-recent first. A callback returning NoOpinion abstains and
+  // defers to the previously-registered callback. Here the first callback marks
+  // the op illegal and the second abstains, so the op stays illegal (conversion
+  // fails) and BOTH callbacks are invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t illegalCounter = 0;
+    intptr_t noOpinionCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &illegalCounter);
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityNoOpinion, &noOpinionCounter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(noOpinionCounter > 0 && "abstaining callback must be invoked");
+    assert(illegalCounter > 0 && "deferred-to callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalDialect: the callback applies to every op in the
+  // dialect. Returning true keeps `dialect.op1` legal -> success.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalDialect(
+        target, mlirStringRefCreateFromCString("dialect"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "dialect legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markUnknownOpDynamicallyLegal: `dialect.op1` is unregistered and otherwise
+  // unmarked, so the unknown-op callback decides its legality.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+        target, dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "unknown-op legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal: an op marked recursively legal short-circuits the
+  // walk so nested ops are never checked. Here `dialect.inner` is illegal, but
+  // because `dialect.outer` is recursively legal the conversion still succeeds
+  // and the inner op's (illegal) callback is never invoked.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    intptr_t recursiveCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"),
+        dynamicLegalityAlwaysLegal, &recursiveCounter);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(recursiveCounter > 0 && "recursive legality callback must run");
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal with a NULL callback: the op is unconditionally
+  // recursively legal (no per-instance check), so the nested illegal op is
+  // still skipped and the conversion succeeds.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"), NULL, NULL);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // CHECK: testConversionTargetDynamicLegality: PASSED
+  fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -638,6 +817,7 @@ int main(void) {
   testReplaceUses(ctx);
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
+  testConversionTargetDynamicLegality(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 74fdfc610665bde5f9c07f065bb898d7f3c0de39 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Thu, 2 Jul 2026 11:54:43 -0700
Subject: [PATCH 2/2] [mlir-c] Fix memory leak in ConversionTarget dynamic
 legality test

mlirFreezeRewritePattern moves contents out of the RewritePatternSet
but does not free the container (set is passed by value in the C API).
Add mlirRewritePatternSetDestroy after freezing to fix LeakSanitizer
failures on Linux.
---
 mlir/test/CAPI/rewrite.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..de7e78720050e 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -659,6 +659,7 @@ static bool runPartialConversion(MlirContext ctx, const char *moduleString,
 
   MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
   MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+  mlirRewritePatternSetDestroy(patterns);
   MlirConversionConfig config = mlirConversionConfigCreate();
 
   MlirLogicalResult result =


From llvmlistbot at llvm.org  Thu Jul  2 11:58:11 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 11:58:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Reapply "[mlir-c] Add ConversionTarget
 dynamic legality C API" (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46b4c3.492e23e4.e71ca.3a99@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 11:58:48 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 11:58:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Reapply "[mlir-c] Add ConversionTarget
 dynamic legality C API" (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46b4e8.7d41f3a1.d1cf7.1b34@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 12:00:36 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 12:00:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46b554.f14cf1a5.2c3aaa.20cc@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 12:01:00 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 12:01:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46b56c.576b3af0.2cb9b7.7ba3@mx.google.com>


makslevental wrote:

https://lab.llvm.org/buildbot/#/buildrequests/8843739?redirect_to_build=true

https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 12:01:31 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 12:01:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46b58b.6c8668f2.74a35.651c@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 12:02:32 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Thu, 02 Jul 2026 12:02:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a46b5c8.ac469dc7.e3aa6.ecba@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/206951

From llvmlistbot at llvm.org  Thu Jul  2 12:02:32 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Thu, 02 Jul 2026 12:02:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a46b5c8.8e75e0cb.2e0812.73c5@mx.google.com>


================
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
----------------
IgWod wrote:

```suggestion
    Type dstType = getTypeConverter()->convertType(op.getType());
```

https://github.com/llvm/llvm-project/pull/206951

From llvmlistbot at llvm.org  Thu Jul  2 12:04:35 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 12:04:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	GL.Fract (PR #206951)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206951@github.com>
Message-ID: <6a46b643.2afc2531.82a15.ed8f@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206951

>From 0c27ffa07eb9b334cbc1bd5e4df2a94abd4e2643 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:15:31 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM conversion for GL.Fract

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 22 ++++++++++++++++++-
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 15 +++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c9eba35cf0939 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,26 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.Fract` to `x - floor(x)`.
+class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFractOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, operand);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, operand, floored);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1936,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..042b68bef9dcd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,18 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Fract
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fract
+spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%{{.*}}) : (f32) -> f32
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOOR]] : f32
+  %0 = spirv.GL.Fract %arg0 : f32
+  // CHECK: %[[FLOORV:.*]] = llvm.intr.floor(%{{.*}}) : (vector<3xf16>) -> vector<3xf16>
+  // CHECK: llvm.fsub %{{.*}}, %[[FLOORV]] : vector<3xf16>
+  %1 = spirv.GL.Fract %arg1 : vector<3xf16>
+  spirv.Return
+}

>From d25627693f8e7f2f43d1a9e1d0da6d1fae1b8205 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <gooddoog at student.su>
Date: Thu, 2 Jul 2026 21:04:25 +0200
Subject: [PATCH 2/2] Update mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp

Co-authored-by: Igor Wodiany <dev at wodiany.com>
---
 mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c9eba35cf0939..2682e1d9e26ce 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1572,7 +1572,7 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   LogicalResult
   matchAndRewrite(spirv::GLFractOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto dstType = getTypeConverter()->convertType(op.getType());
+    Type dstType = getTypeConverter()->convertType(op.getType());
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 


From llvmlistbot at llvm.org  Thu Jul  2 12:06:54 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Thu, 02 Jul 2026 12:06:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a46b6ce.ff037943.3bf0e2.46a5@mx.google.com>


================
@@ -1564,6 +1564,50 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+class RadiansPattern : public SPIRVToLLVMConversion<spirv::GLRadiansOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLRadiansOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLRadiansOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
----------------
IgWod wrote:

```suggestion
    Type srcType = op.getType();
```

https://github.com/llvm/llvm-project/pull/205967

From llvmlistbot at llvm.org  Thu Jul  2 12:06:54 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Thu, 02 Jul 2026 12:06:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a46b6ce.491d90f5.1f0d58.3065@mx.google.com>


================
@@ -1564,6 +1564,50 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+class RadiansPattern : public SPIRVToLLVMConversion<spirv::GLRadiansOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLRadiansOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLRadiansOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    // pi / 180
+    Value factor =
+        createFPConstant(loc, srcType, dstType, rewriter, 0.017453292519943295);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
+                                              factor);
+    return success();
+  }
+};
+
+class DegreesPattern : public SPIRVToLLVMConversion<spirv::GLDegreesOp> {
----------------
IgWod wrote:

Could we merge this with the function above and template it? They only seem to differ by the constant used.

https://github.com/llvm/llvm-project/pull/205967

From llvmlistbot at llvm.org  Thu Jul  2 12:06:54 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Thu, 02 Jul 2026 12:06:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a46b6ce.f06f4994.1e3a60.3a30@mx.google.com>


================
@@ -1564,6 +1564,50 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+class RadiansPattern : public SPIRVToLLVMConversion<spirv::GLRadiansOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLRadiansOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLRadiansOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
----------------
IgWod wrote:

```suggestion
    Type dstType = getTypeConverter()->convertType(srcType);
```

https://github.com/llvm/llvm-project/pull/205967

From llvmlistbot at llvm.org  Thu Jul  2 13:04:46 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Thu, 02 Jul 2026 13:04:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a46c45e.92ea0e2a.21f97.f216@mx.google.com>


joker-eph wrote:

> ConcatOp::inferResultType built the result type with no encoding. When all inputs share an encoding, propagate it; when they differ, drop it (concat semantics are undefined across differing encodings, so we do not pick one).

It's not clear to me that this is correct: the encoding could relate to the inputs dimensions and be totally inconsistent with the concatenated ones.

Overall there are similar issues with most of the "fixes" at hand with encoding: these "opaque metadata" poses fundamental problems for transformations.

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Thu Jul  2 13:49:32 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 13:49:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46cedc.6cfa33d2.19518c.d7e1@mx.google.com>


makslevental wrote:

cc @alazarev

https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 13:49:34 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 13:49:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46cede.01f0a1da.2016e9.d031@mx.google.com>


https://github.com/makslevental ready_for_review https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 13:50:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 13:50:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a46ceff.92ea0e2a.21f97.2f8e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Maksim Levental (makslevental)

<details>
<summary>Changes</summary>

Fixes LeakSanitizer failure from #<!-- -->206161 (reverted in #<!-- -->207104); `mlirFreezeRewritePattern` moves contents out of the `RewritePatternSet` but does not free the container (passed by value in the C API), so the allocation from `mlirRewritePatternSetCreate` was never freed (add `mlirRewritePatternSetDestroy(patterns)` after freezing).

---
Full diff: https://github.com/llvm/llvm-project/pull/207253.diff


3 Files Affected:

- (modified) mlir/include/mlir-c/Rewrite.h (+44) 
- (modified) mlir/lib/CAPI/Transforms/Rewrite.cpp (+60) 
- (modified) mlir/test/CAPI/rewrite.c (+181) 


``````````diff
diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index ac243a4c9d8f9..3356e6f445e47 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -533,6 +533,50 @@ MLIR_CAPI_EXPORTED void
 mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
                                       MlirStringRef dialectName);
 
+/// Result of a dynamic legality callback.
+typedef enum {
+  /// The operation instance is legal.
+  MLIR_CONVERSION_TARGET_LEGALITY_LEGAL,
+  /// The operation instance is illegal.
+  MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL,
+  /// The callback has no opinion on this instance. The decision is deferred to
+  /// other registered callbacks (legality callbacks are composed) or, failing
+  /// that, to the operation's static legality action.
+  MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION
+} MlirConversionTargetLegality;
+
+/// Callback for dynamic legality checks. Returns the legality of the given
+/// operation instance (see MlirConversionTargetLegality).
+typedef MlirConversionTargetLegality (
+    *MlirConversionTargetDynamicLegalityCallback)(MlirOperation op,
+                                                  void *userData);
+
+/// Register the given operation as dynamically legal, with a callback to
+/// determine per-instance legality. The callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Register the given dialect as dynamically legal, with a callback to
+/// determine per-instance legality for all operations in the dialect. The
+/// callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark the given operation as recursively legal. The optional callback (may
+/// be NULL) determines whether a specific instance is recursively legal; a NULL
+/// callback marks the operation as unconditionally recursively legal.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark unknown operations as dynamically legal, with a callback. The callback
+/// must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 56ce9212f4811..083ed6f999ae3 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -23,6 +23,8 @@
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
 
+#include <cassert>
+
 using namespace mlir;
 
 //===----------------------------------------------------------------------===//
@@ -575,6 +577,64 @@ void mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
   unwrap(target)->addIllegalDialect(unwrap(dialectName));
 }
 
+namespace {
+/// Wraps a C dynamic-legality callback as a C++ DynamicLegalityCallbackFn,
+/// translating the tri-state MlirConversionTargetLegality result into the
+/// std::optional<bool> expected by ConversionTarget (NO_OPINION -> nullopt).
+ConversionTarget::DynamicLegalityCallbackFn
+wrapLegalityCallback(MlirConversionTargetDynamicLegalityCallback callback,
+                     void *userData) {
+  return [callback, userData](Operation *op) -> std::optional<bool> {
+    switch (callback(wrap(op), userData)) {
+    case MLIR_CONVERSION_TARGET_LEGALITY_LEGAL:
+      return true;
+    case MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL:
+      return false;
+    case MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION:
+      return std::nullopt;
+    }
+    llvm_unreachable("unknown MlirConversionTargetLegality");
+  };
+}
+} // namespace
+
+void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  unwrap(target)->addDynamicallyLegalOp(
+      name, wrapLegalityCallback(callback, userData));
+}
+
+void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->addDynamicallyLegalDialect(
+      wrapLegalityCallback(callback, userData), unwrap(dialectName));
+}
+
+void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  ConversionTarget::DynamicLegalityCallbackFn fn;
+  if (callback)
+    fn = wrapLegalityCallback(callback, userData);
+  unwrap(target)->markOpRecursivelyLegal(name, fn);
+}
+
+void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->markUnknownOpDynamicallyLegal(
+      wrapLegalityCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 3809dd6a7843f..de7e78720050e 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -623,6 +623,186 @@ void testCloneWithMapping(MlirContext ctx) {
   fprintf(stderr, "testCloneWithMapping: PASSED\n");
 }
 
+static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
+                                                               void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
+}
+
+static MlirConversionTargetLegality
+dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
+}
+
+static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
+                                                             void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
+}
+
+// Runs a partial conversion of `moduleString` against `target` with an empty
+// pattern set and returns whether it succeeded. This is what actually drives
+// the registered dynamic-legality callbacks.
+static bool runPartialConversion(MlirContext ctx, const char *moduleString,
+                                 MlirConversionTarget target) {
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  assert(!mlirModuleIsNull(module) && "expected module to parse");
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+  mlirRewritePatternSetDestroy(patterns);
+  MlirConversionConfig config = mlirConversionConfigCreate();
+
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+
+  mlirConversionConfigDestroy(config);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirModuleDestroy(module);
+
+  return mlirLogicalResultIsSuccess(result);
+}
+
+void testConversionTargetDynamicLegality(MlirContext ctx) {
+  // CHECK-LABEL: @testConversionTargetDynamicLegality
+  fprintf(stderr, "@testConversionTargetDynamicLegality\n");
+
+  const char *opModule = "\"dialect.op1\"() : () -> ()\n";
+
+  // addDynamicallyLegalOp: callback returning true makes the op legal, so the
+  // (pattern-free) partial conversion succeeds and the callback is invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp: callback returning false makes the op illegal. With
+  // no pattern to legalize it, the partial conversion fails -- proving the
+  // callback's return value actually drives the result.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &counter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp composition: callbacks registered for the same op are
+  // chained, most-recent first. A callback returning NoOpinion abstains and
+  // defers to the previously-registered callback. Here the first callback marks
+  // the op illegal and the second abstains, so the op stays illegal (conversion
+  // fails) and BOTH callbacks are invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t illegalCounter = 0;
+    intptr_t noOpinionCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &illegalCounter);
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityNoOpinion, &noOpinionCounter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(noOpinionCounter > 0 && "abstaining callback must be invoked");
+    assert(illegalCounter > 0 && "deferred-to callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalDialect: the callback applies to every op in the
+  // dialect. Returning true keeps `dialect.op1` legal -> success.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalDialect(
+        target, mlirStringRefCreateFromCString("dialect"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "dialect legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markUnknownOpDynamicallyLegal: `dialect.op1` is unregistered and otherwise
+  // unmarked, so the unknown-op callback decides its legality.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+        target, dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "unknown-op legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal: an op marked recursively legal short-circuits the
+  // walk so nested ops are never checked. Here `dialect.inner` is illegal, but
+  // because `dialect.outer` is recursively legal the conversion still succeeds
+  // and the inner op's (illegal) callback is never invoked.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    intptr_t recursiveCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"),
+        dynamicLegalityAlwaysLegal, &recursiveCounter);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(recursiveCounter > 0 && "recursive legality callback must run");
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal with a NULL callback: the op is unconditionally
+  // recursively legal (no per-instance check), so the nested illegal op is
+  // still skipped and the conversion succeeds.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"), NULL, NULL);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // CHECK: testConversionTargetDynamicLegality: PASSED
+  fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -638,6 +818,7 @@ int main(void) {
   testReplaceUses(ctx);
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
+  testConversionTargetDynamicLegality(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

``````````

</details>


https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 13:50:14 2026
From: llvmlistbot at llvm.org (Andrey Pavlenko)
Date: Thu, 02 Jul 2026 13:50:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a46cf06.6c8668f2.74a35.1c25@mx.google.com>


https://github.com/AndreyPavlenko updated https://github.com/llvm/llvm-project/pull/197281

>From 63501e958730dc81c5d28c5404838954500a8eb9 Mon Sep 17 00:00:00 2001
From: Andrey Pavlenko <andrey.a.pavlenko at gmail.com>
Date: Tue, 12 May 2026 19:35:41 +0000
Subject: [PATCH] Fixed transposed transfer_read/write

The problem was that adjustStridesForPermutation() permutes strides,
but they are required to compute base offset from transfer read indices.
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 18 ++++----
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 45 +++++++++++++++++++
 2 files changed, 54 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 8dcc9771d8db4..ba5f010cdd862 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -254,13 +254,8 @@ computeMemrefMeta(OpType xferOp, PatternRewriter &rewriter) {
       offsetVal = meta.getOffset();
   }
 
-  if constexpr (llvm::is_one_of<std::decay_t<OpType>, vector::TransferReadOp,
-                                vector::TransferWriteOp>::value) {
-    AffineMap permMap = xferOp.getPermutationMap();
-    // Adjust strides according to the permutation map (e.g., for transpose)
-    adjustStridesForPermutation(permMap, strides);
-  }
-
+  // Strides are returned in original memref order; permutation is applied in
+  // computeOffsets only where offsets are indexed in vector order.
   return {strides, offsetVal};
 }
 
@@ -310,13 +305,18 @@ static Value computeOffsets(VectorTransferOpInterface xferOp,
     return stepOp;
   });
 
+  // Local offsets are indexed in vector order, so permute strides; the base
+  // offset below uses the original memref-order strides.
+  SmallVector<Value> permutedStrides(strides.begin(), strides.end());
+  adjustStridesForPermutation(xferOp.getPermutationMap(), permutedStrides);
+
   // Multiply step vectors by corresponding strides
-  size_t memrefRank = strides.size();
+  size_t memrefRank = permutedStrides.size();
   size_t vectorRank = vectorShape.size();
   SmallVector<Value> strideMultiplied;
   for (size_t i = 0; i < vectorRank; ++i) {
     size_t memrefDim = memrefRank - vectorRank + i;
-    Value strideValue = strides[memrefDim];
+    Value strideValue = permutedStrides[memrefDim];
     auto mulType = dyn_cast<VectorType>(stepVectors[i].getType());
     auto bcastOp =
         vector::BroadcastOp::create(rewriter, loc, mulType, strideValue);
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index f64bcc27940ea..b319efa97554f 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -679,4 +679,49 @@ gpu.func @load_0D_memref_unsupported(%source: memref<f16>) -> vector<f16> {
 // CHECK-LABEL: @load_0D_memref_unsupported
 // CHECK: vector.transfer_read
 
+}
+
+// -----
+gpu.module @xevm_module {
+gpu.func @transpose_1x1024x24x64(
+    %arg0: memref<1x1024x24x64xf16>,
+    %arg1: memref<1x24x1024x64xf16>) kernel
+    attributes {known_block_size = array<i32: 256, 1, 1>} {
+  %pad = ub.poison : f16
+  %c0 = arith.constant 0 : index
+  %block_id_x = gpu.block_id x
+  %block_id_y = gpu.block_id y
+  %block_id_z = gpu.block_id z
+  %seq_off = affine.apply affine_map<()[s0] -> (s0 * 16)>()[%block_id_y]
+  %hid_off = affine.apply affine_map<()[s0] -> (s0 * 8)>()[%block_id_z]
+  %vec = vector.transfer_read %arg0[%c0, %seq_off, %block_id_x, %hid_off], %pad
+    {in_bounds = [true, true, true, true]}
+    : memref<1x1024x24x64xf16>, vector<1x16x1x8xf16>
+  %transposed = vector.transpose %vec, [0, 2, 1, 3]
+    : vector<1x16x1x8xf16> to vector<1x1x16x8xf16>
+  vector.transfer_write %transposed, %arg1[%c0, %block_id_x, %seq_off, %hid_off]
+    {in_bounds = [true, true, true, true]}
+    : vector<1x1x16x8xf16>, memref<1x24x1024x64xf16>
+  gpu.return
+}
+
+// CHECK-LABEL: @transpose_1x1024x24x64
+// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
+// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+
+// Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
+// Scalar base offset: seq_off * 1536 (original dim1 stride),
+//                    block_id_x * 64  (original dim2 stride).
+// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
+// CHECK:     arith.muli %block_id_x, %[[C64]] : index
+// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+
+// Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
+// Scalar base offset: block_id_x * 65536 (original dim1 stride),
+//                    seq_off * 64        (original dim2 stride).
+// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
+// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
+// CHECK:     xegpu.store {{.*}}
+
 }
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul  2 16:54:26 2026
From: llvmlistbot at llvm.org (Eli Friedman)
Date: Thu, 02 Jul 2026 16:54:26 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a46fa32.2020f6c2.1fdf61.85f0@mx.google.com>


https://github.com/efriedma-quic approved this pull request.

The new approach for properties seems fine.  LGTM

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Thu Jul  2 17:46:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 17:46:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a470681.2020f6c2.1fdf61.a850@mx.google.com>


https://github.com/marquisburg edited https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Thu Jul  2 17:47:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 17:47:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4706b0.fdc477d6.118e5e.948d@mx.google.com>


https://github.com/marquisburg edited https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Thu Jul  2 17:48:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 17:48:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4706e8.510f576d.185953.a2e3@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From bb179a3054918920bf0f77f7d8b141d176e630de Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload


From llvmlistbot at llvm.org  Thu Jul  2 17:49:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 17:49:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a47071f.58bdd12c.21857.830e@mx.google.com>


marquisburg wrote:

Had to fix a spelling mistake in the commit, apologies on my end! Keyboard has a tendency to spam 'h'.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Thu Jul  2 19:31:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 19:31:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
Message-ID: <llvm.org/llvm/llvm-project/pull/207307@github.com>


https://github.com/weimin023 created https://github.com/llvm/llvm-project/pull/207307

## Description:

`nvgpu.mma.sync` / `nvvm.mma.sync` (warp-level, dense) don't support FP8 (e4m3/e5m2) operands, even though the underlying PTX/NVPTX intrinsics have supported FP8 `mma.sync.m16n8k32` since sm_89. FP8 currently only exists on the sparse mma path and on warpgroup-level `wgmma`; the warp-level dense path was never updated.

This adds FP8 (e4m3/e5m2) support to `nvgpu.mma.sync` and its lowering to `nvvm.mma.sync`, covering `m16n8k16` and `m16n8k32`.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type<e4m3>,
      multiplicandBPtxType = #nvvm.mma_type<e4m3>,
      shape = #nvvm.shape<m = 16, n = 8, k = 32>}
     : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>

```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).



>From ced9cb1536542a9884b1dffcf5abf50c18e6e423 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 02:23:22 +0000
Subject: [PATCH] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td          |  6 +++++-
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp           |  9 +++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 5 files changed, 34 insertions(+), 4 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..2a93a3ba3d235 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,f8E4M3FN,"
+              "f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Thu Jul  2 19:31:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 19:31:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a471f15.cbc3f03a.3be201.d83d@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @weimin023 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 19:32:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 19:32:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a471f36.d1496011.3d49cb.89c7@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-nvgpu

Author: WMC (weimin023)

<details>
<summary>Changes</summary>

## Description:

`nvgpu.mma.sync` / `nvvm.mma.sync` (warp-level, dense) don't support FP8 (e4m3/e5m2) operands, even though the underlying PTX/NVPTX intrinsics have supported FP8 `mma.sync.m16n8k32` since sm_89. FP8 currently only exists on the sparse mma path and on warpgroup-level `wgmma`; the warp-level dense path was never updated.

This adds FP8 (e4m3/e5m2) support to `nvgpu.mma.sync` and its lowering to `nvvm.mma.sync`, covering `m16n8k16` and `m16n8k32`.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      multiplicandBPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      shape = #nvvm.shape&lt;m = 16, n = 8, k = 32&gt;}
     : (i32, i32, f32) -&gt; !llvm.struct&lt;(f32, f32, f32, f32)&gt;

```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).



---
Full diff: https://github.com/llvm/llvm-project/pull/207307.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+5-1) 
- (modified) mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp (+4) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+9) 
- (modified) mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp (+4-3) 
- (added) mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir (+12) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..2a93a3ba3d235 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,f8E4M3FN,"
+              "f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 19:32:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 19:32:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a471f37.ff037943.3bf0e2.4fbd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: WMC (weimin023)

<details>
<summary>Changes</summary>

## Description:

`nvgpu.mma.sync` / `nvvm.mma.sync` (warp-level, dense) don't support FP8 (e4m3/e5m2) operands, even though the underlying PTX/NVPTX intrinsics have supported FP8 `mma.sync.m16n8k32` since sm_89. FP8 currently only exists on the sparse mma path and on warpgroup-level `wgmma`; the warp-level dense path was never updated.

This adds FP8 (e4m3/e5m2) support to `nvgpu.mma.sync` and its lowering to `nvvm.mma.sync`, covering `m16n8k16` and `m16n8k32`.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      multiplicandBPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      shape = #nvvm.shape&lt;m = 16, n = 8, k = 32&gt;}
     : (i32, i32, f32) -&gt; !llvm.struct&lt;(f32, f32, f32, f32)&gt;

```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).



---
Full diff: https://github.com/llvm/llvm-project/pull/207307.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+5-1) 
- (modified) mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp (+4) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+9) 
- (modified) mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp (+4-3) 
- (added) mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir (+12) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..2a93a3ba3d235 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,f8E4M3FN,"
+              "f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 20:00:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 20:00:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4725e3.30759430.1908d3.6d1c@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 20:01:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 20:01:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a47261a.abb10101.208f3e.29b6@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 21:14:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 21:14:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a473728.58bdd12c.21857.3692@mx.google.com>


https://github.com/PragmaTwice approved this pull request.

Hmm I don't have a diff between this and prev changes but I guess it's in the test case?

https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 21:14:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 21:14:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a473733.70920dee.3514b2.fac2@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 21:30:14 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 21:30:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a473ad6.1a4df1a5.118deb.7a30@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/205967

>From 4a57422646469ef455fcd5efe91b1072d04dd11d Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 26 Jun 2026 05:53:39 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM lowering for GL Radians
 and Degrees ops

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 47 ++++++++++++++++++-
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 30 ++++++++++++
 2 files changed, 76 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..9ef26c01c3321 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,50 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+class RadiansPattern : public SPIRVToLLVMConversion<spirv::GLRadiansOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLRadiansOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLRadiansOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    // pi / 180
+    Value factor =
+        createFPConstant(loc, srcType, dstType, rewriter, 0.017453292519943295);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
+                                              factor);
+    return success();
+  }
+};
+
+class DegreesPattern : public SPIRVToLLVMConversion<spirv::GLDegreesOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLDegreesOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLDegreesOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    // 180 / pi
+    Value factor =
+        createFPConstant(loc, srcType, dstType, rewriter, 57.29577951308232);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
+                                              factor);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1916,7 +1960,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, RadiansPattern,
+      DegreesPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f26754248a6f2 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -334,6 +334,36 @@ spirv.func @inverse_sqrt(%arg0: f32) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.Radians
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @radians
+spirv.func @radians(%arg0: f32, %arg1: vector<3xf32>) "None" {
+  // CHECK: %[[FACTOR:.*]] = llvm.mlir.constant(0.0174532924 : f32) : f32
+  // CHECK: llvm.fmul %{{.*}}, %[[FACTOR]] : f32
+  %0 = spirv.GL.Radians %arg0 : f32
+  // CHECK: %[[VFACTOR:.*]] = llvm.mlir.constant(dense<0.0174532924> : vector<3xf32>) : vector<3xf32>
+  // CHECK: llvm.fmul %{{.*}}, %[[VFACTOR]] : vector<3xf32>
+  %1 = spirv.GL.Radians %arg1 : vector<3xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Degrees
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @degrees
+spirv.func @degrees(%arg0: f32, %arg1: vector<3xf32>) "None" {
+  // CHECK: %[[FACTOR:.*]] = llvm.mlir.constant(57.2957802 : f32) : f32
+  // CHECK: llvm.fmul %{{.*}}, %[[FACTOR]] : f32
+  %0 = spirv.GL.Degrees %arg0 : f32
+  // CHECK: %[[VFACTOR:.*]] = llvm.mlir.constant(dense<57.2957802> : vector<3xf32>) : vector<3xf32>
+  // CHECK: llvm.fmul %{{.*}}, %[[VFACTOR]] : vector<3xf32>
+  %1 = spirv.GL.Degrees %arg1 : vector<3xf32>
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Trunc
 //===----------------------------------------------------------------------===//

>From be1645f077ccc89070dd0a3550b9aa9286d56058 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 3 Jul 2026 06:30:03 +0200
Subject: [PATCH 2/2] Address comments

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 50 ++++++++-----------
 1 file changed, 20 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 46762d632b5c1..1cd3d79c13e20 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1584,48 +1584,33 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
-class RadiansPattern : public SPIRVToLLVMConversion<spirv::GLRadiansOp> {
+// Converts spirv.GL.Radians (scale = pi/180) and spirv.GL.Degrees
+// (scale = 180/pi) by multiplying the operand by a compile-time constant.
+template <typename SPIRVOp>
+class ScalePattern : public SPIRVToLLVMConversion<SPIRVOp> {
 public:
-  using SPIRVToLLVMConversion<spirv::GLRadiansOp>::SPIRVToLLVMConversion;
+  template <typename... Args>
+  ScalePattern(double scale, Args &&...args)
+      : SPIRVToLLVMConversion<SPIRVOp>(std::forward<Args>(args)...),
+        scale(scale) {}
 
   LogicalResult
-  matchAndRewrite(spirv::GLRadiansOp op, OpAdaptor adaptor,
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto srcType = op.getType();
-    auto dstType = getTypeConverter()->convertType(srcType);
+    Type srcType = op.getType();
+    Type dstType = this->getTypeConverter()->convertType(srcType);
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
     Location loc = op.getLoc();
-    // pi / 180
-    Value factor =
-        createFPConstant(loc, srcType, dstType, rewriter, 0.017453292519943295);
+    Value factor = createFPConstant(loc, srcType, dstType, rewriter, scale);
     rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
                                               factor);
     return success();
   }
-};
 
-class DegreesPattern : public SPIRVToLLVMConversion<spirv::GLDegreesOp> {
-public:
-  using SPIRVToLLVMConversion<spirv::GLDegreesOp>::SPIRVToLLVMConversion;
-
-  LogicalResult
-  matchAndRewrite(spirv::GLDegreesOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto srcType = op.getType();
-    auto dstType = getTypeConverter()->convertType(srcType);
-    if (!dstType)
-      return rewriter.notifyMatchFailure(op, "type conversion failed");
-
-    Location loc = op.getLoc();
-    // 180 / pi
-    Value factor =
-        createFPConstant(loc, srcType, dstType, rewriter, 57.29577951308232);
-    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
-                                              factor);
-    return success();
-  }
+private:
+  double scale;
 };
 
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
@@ -1981,7 +1966,6 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
-      RadiansPattern, DegreesPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -2091,6 +2075,12 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
 
   patterns.add<GlobalVariablePattern>(clientAPI, patterns.getContext(),
                                       typeConverter);
+  // pi / 180
+  patterns.add<ScalePattern<spirv::GLRadiansOp>>(
+      0.017453292519943295, patterns.getContext(), typeConverter);
+  // 180 / pi
+  patterns.add<ScalePattern<spirv::GLDegreesOp>>(
+      57.29577951308232, patterns.getContext(), typeConverter);
 }
 
 void mlir::populateSPIRVToLLVMFunctionConversionPatterns(


From llvmlistbot at llvm.org  Thu Jul  2 21:30:33 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 21:30:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a473ae9.92ea0e2a.21f97.4f22@mx.google.com>


makslevental wrote:

> Hmm I don't have a diff between this and previous changes but I guess it's in the test case?

It's the second commit that just adds the destroy call

https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 21:51:49 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 21:51:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a473fe5.abb10101.208f3e.317d@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206935

>From 4fa5c4ca4a5a4333743f1fc3afa38979d20a176a Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:54:35 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix and
 CL.mix

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 27 +++++++++++++++++++
 .../SPIRVToLLVM/cl-ops-to-llvm.mlir           | 20 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 20 ++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Thu Jul  2 21:54:29 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 21:54:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206934@github.com>
Message-ID: <6a474085.1a4df1a5.118deb.b37d@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206934

>From 2e1952b741d34c2b0dfc1887493498283a21a93c Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:06:39 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FSign and
 GL.SSign

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 64 +++++++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 54 ++++++++++++++++
 2 files changed, 118 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Thu Jul  2 22:38:47 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 22:38:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Guard getSizeInBytes()
 optionals in UnifyAliasedResourcePass rewriters (PR #207325)
Message-ID: <llvm.org/llvm/llvm-project/pull/207325@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/207325

None

>From c07f158daeb688288e079d93cfe5727ef99c949d Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 3 Jul 2026 07:36:46 +0200
Subject: [PATCH] [mlir][SPIR-V] Guard getSizeInBytes() optionals in
 UnifyAliasedResourcePass rewriters

---
 .../Transforms/UnifyAliasedResourcePass.cpp   | 24 ++++++++++++++-----
 .../Transforms/unify-aliased-resource.mlir    | 18 ++++++++++++++
 2 files changed, 36 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
index 692f2e7616e5a..ecfce68db036b 100644
--- a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
+++ b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
@@ -371,8 +371,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // them into a buffer with vector element types. We need to scale the last
       // index for the vector as a whole, then add one level of index for inside
       // the vector.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(dstNumBytes >= srcNumBytes && dstNumBytes % srcNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -398,8 +402,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // The source indices are for a buffer with larger bitwidth scalar/vector
       // element types. Rewrite them into a buffer with smaller bitwidth element
       // types. We only need to scale the last index.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes >= dstNumBytes && srcNumBytes % dstNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -455,8 +463,12 @@ struct ConvertLoad : public ConvertAliasResource<spirv::LoadOp> {
       // vector types of different component counts. For such cases, we load
       // multiple smaller bitwidth values and construct a larger bitwidth one.
 
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(loadOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes > dstNumBytes && srcNumBytes % dstNumBytes == 0);
       int ratio = srcNumBytes / dstNumBytes;
       if (ratio > 4)
diff --git a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
index f5cd490c164d7..d7cc1d0cc1f84 100644
--- a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
+++ b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
@@ -549,6 +549,24 @@ spirv.module Logical GLSL450 {
 
 // -----
 
+// Make sure we do not crash on element types without a defined byte size (i1).
+
+spirv.module Logical GLSL450 {
+  spirv.GlobalVariable @var01_i1 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i1, stride=1> [0])>, StorageBuffer>
+  spirv.GlobalVariable @var01_i32 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+
+  spirv.func @unknown_byte_size(%index: i32) -> i32 "None" {
+    %c0 = spirv.Constant 0 : i32
+    %addr = spirv.mlir.addressof @var01_i32 : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+    // expected-error at +1 {{failed to legalize operation 'spirv.AccessChain'}}
+    %ac = spirv.AccessChain %addr[%c0, %index] : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>, i32, i32 -> !spirv.ptr<i32, StorageBuffer>
+    %value = spirv.Load "StorageBuffer" %ac : i32
+    spirv.ReturnValue %value : i32
+  }
+}
+
+// -----
+
 // Make sure we do not crash on function arguments.
 
 spirv.module Logical GLSL450 {


From llvmlistbot at llvm.org  Thu Jul  2 22:39:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 22:39:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Guard getSizeInBytes()
 optionals in UnifyAliasedResourcePass rewriters (PR #207325)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207325@github.com>
Message-ID: <6a474b07.d1496011.3d49cb.e711@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Arseniy Obolenskiy (aobolensk)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/207325.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp (+18-6) 
- (modified) mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir (+18) 


``````````diff
diff --git a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
index 692f2e7616e5a..ecfce68db036b 100644
--- a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
+++ b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
@@ -371,8 +371,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // them into a buffer with vector element types. We need to scale the last
       // index for the vector as a whole, then add one level of index for inside
       // the vector.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(dstNumBytes >= srcNumBytes && dstNumBytes % srcNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -398,8 +402,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // The source indices are for a buffer with larger bitwidth scalar/vector
       // element types. Rewrite them into a buffer with smaller bitwidth element
       // types. We only need to scale the last index.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes >= dstNumBytes && srcNumBytes % dstNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -455,8 +463,12 @@ struct ConvertLoad : public ConvertAliasResource<spirv::LoadOp> {
       // vector types of different component counts. For such cases, we load
       // multiple smaller bitwidth values and construct a larger bitwidth one.
 
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(loadOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes > dstNumBytes && srcNumBytes % dstNumBytes == 0);
       int ratio = srcNumBytes / dstNumBytes;
       if (ratio > 4)
diff --git a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
index f5cd490c164d7..d7cc1d0cc1f84 100644
--- a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
+++ b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
@@ -549,6 +549,24 @@ spirv.module Logical GLSL450 {
 
 // -----
 
+// Make sure we do not crash on element types without a defined byte size (i1).
+
+spirv.module Logical GLSL450 {
+  spirv.GlobalVariable @var01_i1 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i1, stride=1> [0])>, StorageBuffer>
+  spirv.GlobalVariable @var01_i32 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+
+  spirv.func @unknown_byte_size(%index: i32) -> i32 "None" {
+    %c0 = spirv.Constant 0 : i32
+    %addr = spirv.mlir.addressof @var01_i32 : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+    // expected-error at +1 {{failed to legalize operation 'spirv.AccessChain'}}
+    %ac = spirv.AccessChain %addr[%c0, %index] : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>, i32, i32 -> !spirv.ptr<i32, StorageBuffer>
+    %value = spirv.Load "StorageBuffer" %ac : i32
+    spirv.ReturnValue %value : i32
+  }
+}
+
+// -----
+
 // Make sure we do not crash on function arguments.
 
 spirv.module Logical GLSL450 {

``````````

</details>


https://github.com/llvm/llvm-project/pull/207325

From llvmlistbot at llvm.org  Thu Jul  2 22:43:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 22:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] d3b6ad1 - [mlir-c] Reapply Add
 ConversionTarget dynamic legality C API (#207104) (#207253)
Message-ID: <6a474c19.f8b79514.3b441.3932@mx.google.com>


Author: Maksim Levental
Date: 2026-07-02T22:43:48-07:00
New Revision: d3b6ad1c11c24362c7f94b41e5a26652047582c8

URL: https://github.com/llvm/llvm-project/commit/d3b6ad1c11c24362c7f94b41e5a26652047582c8
DIFF: https://github.com/llvm/llvm-project/commit/d3b6ad1c11c24362c7f94b41e5a26652047582c8.diff

LOG: [mlir-c] Reapply Add ConversionTarget dynamic legality C API (#207104) (#207253)

Fixes LeakSanitizer failure from #206161 (reverted in #207104);
`mlirFreezeRewritePattern` moves contents out of the `RewritePatternSet`
but does not free the container (passed by value in the C API), so the
allocation from `mlirRewritePatternSetCreate` was never freed (add
`mlirRewritePatternSetDestroy(patterns)` after freezing).

Added: 
    

Modified: 
    mlir/include/mlir-c/Rewrite.h
    mlir/lib/CAPI/Transforms/Rewrite.cpp
    mlir/test/CAPI/rewrite.c

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index ac243a4c9d8f9..3356e6f445e47 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -533,6 +533,50 @@ MLIR_CAPI_EXPORTED void
 mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
                                       MlirStringRef dialectName);
 
+/// Result of a dynamic legality callback.
+typedef enum {
+  /// The operation instance is legal.
+  MLIR_CONVERSION_TARGET_LEGALITY_LEGAL,
+  /// The operation instance is illegal.
+  MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL,
+  /// The callback has no opinion on this instance. The decision is deferred to
+  /// other registered callbacks (legality callbacks are composed) or, failing
+  /// that, to the operation's static legality action.
+  MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION
+} MlirConversionTargetLegality;
+
+/// Callback for dynamic legality checks. Returns the legality of the given
+/// operation instance (see MlirConversionTargetLegality).
+typedef MlirConversionTargetLegality (
+    *MlirConversionTargetDynamicLegalityCallback)(MlirOperation op,
+                                                  void *userData);
+
+/// Register the given operation as dynamically legal, with a callback to
+/// determine per-instance legality. The callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Register the given dialect as dynamically legal, with a callback to
+/// determine per-instance legality for all operations in the dialect. The
+/// callback must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark the given operation as recursively legal. The optional callback (may
+/// be NULL) determines whether a specific instance is recursively legal; a NULL
+/// callback marks the operation as unconditionally recursively legal.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
+/// Mark unknown operations as dynamically legal, with a callback. The callback
+/// must not be NULL.
+MLIR_CAPI_EXPORTED void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 56ce9212f4811..083ed6f999ae3 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -23,6 +23,8 @@
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
 
+#include <cassert>
+
 using namespace mlir;
 
 //===----------------------------------------------------------------------===//
@@ -575,6 +577,64 @@ void mlirConversionTargetAddIllegalDialect(MlirConversionTarget target,
   unwrap(target)->addIllegalDialect(unwrap(dialectName));
 }
 
+namespace {
+/// Wraps a C dynamic-legality callback as a C++ DynamicLegalityCallbackFn,
+/// translating the tri-state MlirConversionTargetLegality result into the
+/// std::optional<bool> expected by ConversionTarget (NO_OPINION -> nullopt).
+ConversionTarget::DynamicLegalityCallbackFn
+wrapLegalityCallback(MlirConversionTargetDynamicLegalityCallback callback,
+                     void *userData) {
+  return [callback, userData](Operation *op) -> std::optional<bool> {
+    switch (callback(wrap(op), userData)) {
+    case MLIR_CONVERSION_TARGET_LEGALITY_LEGAL:
+      return true;
+    case MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL:
+      return false;
+    case MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION:
+      return std::nullopt;
+    }
+    llvm_unreachable("unknown MlirConversionTargetLegality");
+  };
+}
+} // namespace
+
+void mlirConversionTargetAddDynamicallyLegalOp(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  unwrap(target)->addDynamicallyLegalOp(
+      name, wrapLegalityCallback(callback, userData));
+}
+
+void mlirConversionTargetAddDynamicallyLegalDialect(
+    MlirConversionTarget target, MlirStringRef dialectName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->addDynamicallyLegalDialect(
+      wrapLegalityCallback(callback, userData), unwrap(dialectName));
+}
+
+void mlirConversionTargetMarkOpRecursivelyLegal(
+    MlirConversionTarget target, MlirStringRef opName,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  MLIRContext *ctx = &unwrap(target)->getContext();
+  OperationName name(unwrap(opName), ctx);
+  ConversionTarget::DynamicLegalityCallbackFn fn;
+  if (callback)
+    fn = wrapLegalityCallback(callback, userData);
+  unwrap(target)->markOpRecursivelyLegal(name, fn);
+}
+
+void mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+    MlirConversionTarget target,
+    MlirConversionTargetDynamicLegalityCallback callback, void *userData) {
+  assert(callback && "expected non-null legality callback");
+  unwrap(target)->markUnknownOpDynamicallyLegal(
+      wrapLegalityCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// TypeConverter API
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 3809dd6a7843f..de7e78720050e 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -623,6 +623,186 @@ void testCloneWithMapping(MlirContext ctx) {
   fprintf(stderr, "testCloneWithMapping: PASSED\n");
 }
 
+static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
+                                                               void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
+}
+
+static MlirConversionTargetLegality
+dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
+}
+
+static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
+                                                             void *userData) {
+  (void)op;
+  intptr_t *counter = (intptr_t *)userData;
+  (*counter)++;
+  return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
+}
+
+// Runs a partial conversion of `moduleString` against `target` with an empty
+// pattern set and returns whether it succeeded. This is what actually drives
+// the registered dynamic-legality callbacks.
+static bool runPartialConversion(MlirContext ctx, const char *moduleString,
+                                 MlirConversionTarget target) {
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  assert(!mlirModuleIsNull(module) && "expected module to parse");
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+  mlirRewritePatternSetDestroy(patterns);
+  MlirConversionConfig config = mlirConversionConfigCreate();
+
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+
+  mlirConversionConfigDestroy(config);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirModuleDestroy(module);
+
+  return mlirLogicalResultIsSuccess(result);
+}
+
+void testConversionTargetDynamicLegality(MlirContext ctx) {
+  // CHECK-LABEL: @testConversionTargetDynamicLegality
+  fprintf(stderr, "@testConversionTargetDynamicLegality\n");
+
+  const char *opModule = "\"dialect.op1\"() : () -> ()\n";
+
+  // addDynamicallyLegalOp: callback returning true makes the op legal, so the
+  // (pattern-free) partial conversion succeeds and the callback is invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp: callback returning false makes the op illegal. With
+  // no pattern to legalize it, the partial conversion fails -- proving the
+  // callback's return value actually drives the result.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &counter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalOp composition: callbacks registered for the same op are
+  // chained, most-recent first. A callback returning NoOpinion abstains and
+  // defers to the previously-registered callback. Here the first callback marks
+  // the op illegal and the second abstains, so the op stays illegal (conversion
+  // fails) and BOTH callbacks are invoked.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t illegalCounter = 0;
+    intptr_t noOpinionCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityAlwaysIllegal, &illegalCounter);
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.op1"),
+        dynamicLegalityNoOpinion, &noOpinionCounter);
+    assert(!runPartialConversion(ctx, opModule, target));
+    assert(noOpinionCounter > 0 && "abstaining callback must be invoked");
+    assert(illegalCounter > 0 && "deferred-to callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // addDynamicallyLegalDialect: the callback applies to every op in the
+  // dialect. Returning true keeps `dialect.op1` legal -> success.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetAddDynamicallyLegalDialect(
+        target, mlirStringRefCreateFromCString("dialect"),
+        dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "dialect legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markUnknownOpDynamicallyLegal: `dialect.op1` is unregistered and otherwise
+  // unmarked, so the unknown-op callback decides its legality.
+  {
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t counter = 0;
+    mlirConversionTargetMarkUnknownOpDynamicallyLegal(
+        target, dynamicLegalityAlwaysLegal, &counter);
+    assert(runPartialConversion(ctx, opModule, target));
+    assert(counter > 0 && "unknown-op legality callback must be invoked");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal: an op marked recursively legal short-circuits the
+  // walk so nested ops are never checked. Here `dialect.inner` is illegal, but
+  // because `dialect.outer` is recursively legal the conversion still succeeds
+  // and the inner op's (illegal) callback is never invoked.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    intptr_t recursiveCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"),
+        dynamicLegalityAlwaysLegal, &recursiveCounter);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(recursiveCounter > 0 && "recursive legality callback must run");
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // markOpRecursivelyLegal with a NULL callback: the op is unconditionally
+  // recursively legal (no per-instance check), so the nested illegal op is
+  // still skipped and the conversion succeeds.
+  {
+    const char *nestedModule = "\"dialect.outer\"() ({\n"
+                               "  \"dialect.inner\"() : () -> ()\n"
+                               "}) : () -> ()\n";
+    MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+    intptr_t innerCounter = 0;
+    mlirConversionTargetAddDynamicallyLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.inner"),
+        dynamicLegalityAlwaysIllegal, &innerCounter);
+    mlirConversionTargetAddLegalOp(
+        target, mlirStringRefCreateFromCString("dialect.outer"));
+    mlirConversionTargetMarkOpRecursivelyLegal(
+        target, mlirStringRefCreateFromCString("dialect.outer"), NULL, NULL);
+    assert(runPartialConversion(ctx, nestedModule, target));
+    assert(innerCounter == 0 &&
+           "nested op must not be visited under recursive legality");
+    mlirConversionTargetDestroy(target);
+  }
+
+  // CHECK: testConversionTargetDynamicLegality: PASSED
+  fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -638,6 +818,7 @@ int main(void) {
   testReplaceUses(ctx);
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
+  testConversionTargetDynamicLegality(ctx);
 
   mlirContextDestroy(ctx);
   return 0;


        

From llvmlistbot at llvm.org  Thu Jul  2 22:43:55 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 02 Jul 2026 22:43:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Reapply Add ConversionTarget dynamic
 legality C API (#207104) (PR #207253)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207253@github.com>
Message-ID: <6a474c1b.8e75e0cb.2e0812.3be9@mx.google.com>


https://github.com/makslevental closed https://github.com/llvm/llvm-project/pull/207253

From llvmlistbot at llvm.org  Thu Jul  2 23:12:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:12:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
Message-ID: <llvm.org/llvm/llvm-project/pull/207327@github.com>


https://github.com/Qixi-1 created https://github.com/llvm/llvm-project/pull/207327

This fixes a crash in -convert-arith-to-amdgpu=chipset=gfx950 when the input contains tensor arith.scaling_truncf operations.

The scaling rewrite patterns are written for scalar/vector values. Tensor values should not be rewritten by these patterns, so this patch returns failure for tensor operands and leaves those ops unchanged.

A regression test is added for the tensor reproducer from #207311.

>From 14840d36550b5d9c7fe64e5a0fb41d50dfa274f1 Mon Sep 17 00:00:00 2001
From: tongjinxuan <tongjinxuan at longcheer.com>
Date: Fri, 3 Jul 2026 14:08:52 +0800
Subject: [PATCH] [mlir][ArithToAMDGPU] Skip tensor scaling ops

---
 .../ArithToAMDGPU/ArithToAMDGPU.cpp           | 10 ++++++++++
 .../ArithToAMDGPU/scaling-truncf-tensor.mlir  | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir

diff --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index 3372faf4b16cd..d61eeaf3d1e8d 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -464,6 +464,11 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+    isa<RankedTensorType>(in.getType()) ||
+    isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
@@ -577,6 +582,11 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+    isa<RankedTensorType>(in.getType()) ||
+    isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
diff --git a/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
new file mode 100644
index 0000000000000..e92bfafc0babc
--- /dev/null
+++ b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+module {
+  // CHECK-LABEL: func.func @m0
+  // CHECK: arith.scaling_truncf
+  func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+    %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+    return %0 : tensor<16xf4E2M1FN>
+  }
+
+  // CHECK-LABEL: func.func @m1
+  // CHECK: arith.constant
+  // CHECK: arith.scaling_truncf
+  func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+    %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+    %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+    return %0 : tensor<4xf4E2M1FN>
+  }
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul  2 23:12:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:12:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a4752d6.e9d484b9.11ad58.9c1b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-amdgpu

Author: Qixi (Qixi-1)

<details>
<summary>Changes</summary>

This fixes a crash in -convert-arith-to-amdgpu=chipset=gfx950 when the input contains tensor arith.scaling_truncf operations.

The scaling rewrite patterns are written for scalar/vector values. Tensor values should not be rewritten by these patterns, so this patch returns failure for tensor operands and leaves those ops unchanged.

A regression test is added for the tensor reproducer from #<!-- -->207311.

---
Full diff: https://github.com/llvm/llvm-project/pull/207327.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp (+10) 
- (added) mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir (+19) 


``````````diff
diff --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index 3372faf4b16cd..d61eeaf3d1e8d 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -464,6 +464,11 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+    isa<RankedTensorType>(in.getType()) ||
+    isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
@@ -577,6 +582,11 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+    isa<RankedTensorType>(in.getType()) ||
+    isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
diff --git a/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
new file mode 100644
index 0000000000000..e92bfafc0babc
--- /dev/null
+++ b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+module {
+  // CHECK-LABEL: func.func @m0
+  // CHECK: arith.scaling_truncf
+  func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+    %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+    return %0 : tensor<16xf4E2M1FN>
+  }
+
+  // CHECK-LABEL: func.func @m1
+  // CHECK: arith.constant
+  // CHECK: arith.scaling_truncf
+  func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+    %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+    %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+    return %0 : tensor<4xf4E2M1FN>
+  }
+}
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Thu Jul  2 23:13:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:13:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a475311.3157459c.24b6e5.db33@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index d61eeaf3d..cf44c2a00 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -465,8 +465,8 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
     return failure();
 
   if (isa<RankedTensorType>(out.getType()) ||
-    isa<RankedTensorType>(in.getType()) ||
-    isa<RankedTensorType>(scale.getType()))
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
     return failure();
 
   Type scaleF32Type =
@@ -583,8 +583,8 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
     return failure();
 
   if (isa<RankedTensorType>(out.getType()) ||
-    isa<RankedTensorType>(in.getType()) ||
-    isa<RankedTensorType>(scale.getType()))
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
     return failure();
 
   Type scaleF32Type =

``````````

</details>


https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Thu Jul  2 23:16:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:16:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a4753cc.70920dee.3514b2.7ef7@mx.google.com>


https://github.com/Qixi-1 updated https://github.com/llvm/llvm-project/pull/207327

>From 7eb1de2cdf18033ebb636590dd73ecf459457be9 Mon Sep 17 00:00:00 2001
From: tongjinxuan <tongjinxuan at longcheer.com>
Date: Fri, 3 Jul 2026 14:08:52 +0800
Subject: [PATCH] [mlir][ArithToAMDGPU] Skip tensor scaling ops

---
 .../ArithToAMDGPU/ArithToAMDGPU.cpp           | 10 ++++++++++
 .../ArithToAMDGPU/scaling-truncf-tensor.mlir  | 19 +++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir

diff --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index 3372faf4b16cd..cf44c2a0033ac 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -464,6 +464,11 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
@@ -577,6 +582,11 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
diff --git a/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
new file mode 100644
index 0000000000000..e92bfafc0babc
--- /dev/null
+++ b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+module {
+  // CHECK-LABEL: func.func @m0
+  // CHECK: arith.scaling_truncf
+  func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+    %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+    return %0 : tensor<16xf4E2M1FN>
+  }
+
+  // CHECK-LABEL: func.func @m1
+  // CHECK: arith.constant
+  // CHECK: arith.scaling_truncf
+  func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+    %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+    %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+    return %0 : tensor<4xf4E2M1FN>
+  }
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul  2 23:20:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:20:02 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Support for "!$omp
	dispatch". (PR #203320)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203320@github.com>
Message-ID: <6a475492.fdc477d6.118e5e.24e4@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/203320

>From d007fc287982ed638ef3900307ca700f0d529f87 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 10:56:08 -0500
Subject: [PATCH 1/4] Support for "!$omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 33 ++++++-
 flang/test/Integration/OpenMP/dispatch.f90    | 76 ++++++++++++++++
 flang/test/Lower/OpenMP/Todo/dispatch.f90     | 12 ---
 flang/test/Lower/OpenMP/dispatch.f90          | 89 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 15 ++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 ++++
 mlir/test/Dialect/OpenMP/dispatch.mlir        | 77 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  | 81 +++++++++++++++++
 8 files changed, 385 insertions(+), 15 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/dispatch.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90
 create mode 100644 flang/test/Lower/OpenMP/dispatch.f90
 create mode 100644 mlir/test/Dialect/OpenMP/dispatch.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-dispatch.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index be876c563433a..758d8eb8148e5 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2592,6 +2592,18 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -3940,6 +3952,9 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
   case llvm::omp::Directive::OMPD_barrier:
     newOp = genBarrierOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
+  case llvm::omp::Directive::OMPD_dispatch:
+    newOp = genDispatchOp(converter, symTable, semaCtx, eval, loc, queue, item);
+    break;
   case llvm::omp::Directive::OMPD_distribute:
     newOp = genStandaloneDistribute(converter, symTable, stmtCtx, semaCtx, eval,
                                     loc, queue, item);
@@ -5312,9 +5327,21 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
-                   const parser::OpenMPDispatchConstruct &) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPDispatchConstruct");
+                   const parser::OpenMPDispatchConstruct &dispatchConstruct) {
+  const parser::OmpDirectiveSpecification &beginSpec =
+      dispatchConstruct.BeginDir();
+  List<Clause> clauses = makeClauses(beginSpec.Clauses(), semaCtx);
+  if (auto &endSpec = dispatchConstruct.EndDir())
+    clauses.append(makeClauses(endSpec->Clauses(), semaCtx));
+
+  llvm::omp::Directive directive = beginSpec.DirId();
+  mlir::Location currentLocation = converter.genLocation(beginSpec.source);
+
+  ConstructQueue queue{
+      buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
+                          eval, beginSpec.source, directive, clauses)};
+  genOMPDispatch(converter, symTable, semaCtx, eval, currentLocation, queue,
+                 queue.begin());
 }
 
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..c4d16690c8e29
--- /dev/null
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -0,0 +1,76 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
+!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
+!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
+!CHECK: [[IF_TRUE]]:
+!CHECK: call void @_QMfuncsPfoo_variant2()
+!CHECK: [[IF_FALSE]]:
+!CHECK: call void @_QMfuncsPfoo_variant1()
+
+!CHECK-LABEL: define void @_QQmain()
+!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region
+!CHECK: omp.dispatch.region:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont
+!CHECK: omp.region.cont:
+!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region2
+!CHECK: omp.dispatch.region2:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont1
+!CHECK: omp.region.cont1:
+
+module funcs
+  implicit none
+  logical :: foo_sub
+
+contains
+
+  subroutine foo_variant1()
+    print *, "in foo_variant1"
+  end subroutine
+
+  subroutine foo_variant2()
+    print *, "in foo_variant2"
+  end subroutine
+
+  subroutine foo_dispatch()
+    if (foo_sub) then
+      call foo_variant2()
+    else
+      call foo_variant1()
+    end if
+  end subroutine
+
+end module funcs
+
+program dispatch_test
+  use funcs
+  implicit none
+
+  foo_sub = .false.
+  !$omp dispatch
+  call foo_dispatch()
+
+  foo_sub = .true.
+  !$omp dispatch
+  call foo_dispatch()
+
+end program
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
deleted file mode 100644
index 380dfa14eaae1..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ /dev/null
@@ -1,12 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPDispatchConstruct
-program p
-  integer r
-  r = 1
-!$omp dispatch nowait
-  call foo()
-contains
-  subroutine foo
-  end subroutine
-end program p
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..df86503817163
--- /dev/null
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -0,0 +1,89 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
+
+!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
+!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
+!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
+!HLFIR: fir.if %[[CMP_H]] {
+!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!HLFIR: } else {
+!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!HLFIR: }
+
+!FIR-LABEL: func @_QMfuncsPfoo_dispatch
+!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
+!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
+!FIR: fir.if %[[CMP_F]] {
+!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!FIR: } else {
+!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!FIR: }
+
+module funcs
+  implicit none
+
+contains
+
+  subroutine variant1()
+    print *, "in variant1"
+  end subroutine
+
+  subroutine variant2()
+    print *, "in variant2"
+  end subroutine
+
+  subroutine foo_dispatch(x)
+    integer, intent(in) :: x
+    if (x == 1) then
+      call variant1()
+    else
+      call variant2()
+    end if
+  end subroutine
+
+end module funcs
+
+!HLFIR-LABEL: func @_QQmain
+!FIR-LABEL: func @_QQmain
+!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
+!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
+!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+program dispatch_test
+  use funcs
+  implicit none
+  integer :: x
+
+  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
+  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
+  x = 1
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+
+  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
+  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
+  x = 2
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 51e7080db5b29..6170e40a835c6 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2286,6 +2286,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
   ];
 }
 
+//===----------------------------------------------------------------------===//
+// [Spec 5.1] 12.3 dispatch Construct
+//===----------------------------------------------------------------------===//
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+  let summary = "dispatch construct";
+  let description = [{
+    The dispatch construct enables the invocation of a variant of a
+    base procedure. The structured block of a dispatch construct is a
+    single expression statement that contains a function call or a
+    subroutine call.
+  }];
+
+  let assemblyFormat = "$region attr-dict";
+}
+
 //===----------------------------------------------------------------------===//
 // [Spec 5.2] 6.5 allocate Directive
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..12818a5e56ac6 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -761,6 +761,20 @@ static llvm::omp::ProcBindKind getProcBindKind(omp::ClauseProcBindKind kind) {
   llvm_unreachable("Unknown ClauseProcBindKind kind");
 }
 
+/// Convert 'dispatch' operation into LLVM IR.
+static LogicalResult
+convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  auto dispatchOp = cast<omp::DispatchOp>(opInst);
+  auto &region = dispatchOp.getRegion();
+  auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
+                                    moduleTranslation);
+  if (!result)
+    return handleError(result.takeError(), opInst);
+  builder.SetInsertPoint(*result);
+  return success();
+}
+
 /// Converts an OpenMP 'masked' operation into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpMasked(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -8815,6 +8829,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })
+          .Case([&](omp::DispatchOp) {
+            return convertOmpDispatch(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::MaskedOp) {
             return convertOmpMasked(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
new file mode 100644
index 0000000000000..c935f56836185
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: func @foo_dispatch
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @foo_dispatch(%x : memref<i32>) -> () {
+  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
+  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
+  %v = memref.load %x[] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  %cmp = arith.cmpi eq, %v, %c1 : i32
+  cf.cond_br %cmp, ^bb1, ^bb2
+// CHECK: ^[[BB1]]:
+// CHECK: call @variant1() : () -> ()
+^bb1:
+  func.call @variant1() : () -> ()
+  cf.br ^bb3
+// CHECK: ^[[BB2]]:
+// CHECK: call @variant2() : () -> ()
+^bb2:
+  func.call @variant2() : () -> ()
+  cf.br ^bb3
+^bb3:
+  return
+}
+
+// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
+// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// Test multiple dispatch regions with stores selecting different variants.
+// CHECK-LABEL: func @omp_dispatch_multiple
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  memref.store %c1, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
+  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
+  %c2 = arith.constant 2 : i32
+  memref.store %c2, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// CHECK-LABEL: func private @variant1()
+// CHECK-LABEL: func private @variant2()
+func.func private @variant1() -> ()
+func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
new file mode 100644
index 0000000000000..8a3a7a6b5d8e6
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -0,0 +1,81 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.return %0 : i32
+}
+
+llvm.func @variant1() -> ()
+llvm.func @variant2() -> ()
+
+// CHECK-LABEL: define void @foo_dispatch()
+llvm.func @foo_dispatch() {
+  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
+  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
+  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %1 = llvm.load %0 : !llvm.ptr -> i32
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  %cmp = llvm.icmp "eq" %1, %c1 : i32
+  llvm.cond_br %cmp, ^bb1, ^bb2
+// CHECK: [[BB1]]:
+// CHECK: call void @variant1()
+^bb1:
+  llvm.call @variant1() : () -> ()
+  llvm.br ^bb3
+// CHECK: [[BB2]]:
+// CHECK: call void @variant2()
+^bb2:
+  llvm.call @variant2() : () -> ()
+  llvm.br ^bb3
+^bb3:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch()
+llvm.func @test_omp_dispatch() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch_multiple()
+llvm.func @test_omp_dispatch_multiple() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  // CHECK: store i32 2, ptr @x
+  %c2 = llvm.mlir.constant(2 : i32) : i32
+  llvm.store %c2, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region2
+  omp.dispatch {
+    // CHECK: omp.dispatch.region2:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont1
+    omp.terminator
+  }
+  // CHECK: omp.region.cont1:
+  llvm.return
+}

>From b50c11a322615d4e189303f9b4b73e1c6246c7ae Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 12:19:36 -0500
Subject: [PATCH 2/4] Checking for no caluses support with "omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 3 +++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 758d8eb8148e5..71bc392d0e8d9 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -5337,6 +5337,9 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
+  if (!clauses.empty())
+    TODO(currentLocation, "OpenMP Dispatch clauses");
+
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 6170e40a835c6..47c7587c5d08b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2293,9 +2293,10 @@ def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure. The structured block of a dispatch construct is a
-    single expression statement that contains a function call or a
-    subroutine call.
+    base procedure (without any construct's clauses: device, is_device_ptr,
+    nowait, nocontext, novariants). The structured block of a dispatch
+    construct is a single expression statement that contains a function
+    call or a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";

>From 3ca936ebb5aff08e318fcd4abede5951e77fcb07 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 18 Jun 2026 11:50:04 -0500
Subject: [PATCH 3/4] Handling feedbacks suggested by Sergio Afonso.

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 37 ++++++++++++-------
 flang/test/Lower/OpenMP/dispatch.f90          | 26 +------------
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 11 +++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  4 ++
 mlir/test/Dialect/OpenMP/dispatch.mlir        |  1 +
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  |  1 +
 6 files changed, 36 insertions(+), 44 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 33e67ed2c3ab7..fe7d17807727a 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1661,6 +1661,16 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
   cp.processOrder(clauseOps);
 }
 
+static void genDispatchClauses(lower::AbstractConverter &converter,
+                               semantics::SemanticsContext &semaCtx,
+                               const List<Clause> &clauses,
+                               mlir::Location loc) {
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
+                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+      loc, llvm::omp::Directive::OMPD_dispatch);
+}
+
 static void genFlushClauses(lower::AbstractConverter &converter,
                             semantics::SemanticsContext &semaCtx,
                             const ObjectList &objects,
@@ -2154,6 +2164,18 @@ genCriticalOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item, nameAttr);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::FlushOp
 genFlushOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
            semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -2592,18 +2614,6 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
-static mlir::omp::DispatchOp
-genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-              semantics::SemanticsContext &semaCtx,
-              lower::pft::Evaluation &eval, mlir::Location loc,
-              const ConstructQueue &queue,
-              ConstructQueue::const_iterator item) {
-  return genOpWithBody<mlir::omp::DispatchOp>(
-      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
-                        llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
-}
-
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -5294,8 +5304,7 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  if (!clauses.empty())
-    TODO(currentLocation, "OpenMP Dispatch clauses");
+  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
 
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index df86503817163..ecc7b1b31454f 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,5 +1,4 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
-!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
 
 !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
 !HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
@@ -12,16 +11,6 @@
 !HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
 !HLFIR: }
 
-!FIR-LABEL: func @_QMfuncsPfoo_dispatch
-!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
-!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
-!FIR: fir.if %[[CMP_F]] {
-!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!FIR: } else {
-!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!FIR: }
-
 module funcs
   implicit none
 
@@ -35,6 +24,7 @@ subroutine variant2()
     print *, "in variant2"
   end subroutine
 
+  !TODO: replace with declare_variant when the support is merged.
   subroutine foo_dispatch(x)
     integer, intent(in) :: x
     if (x == 1) then
@@ -47,10 +37,6 @@ subroutine foo_dispatch(x)
 end module funcs
 
 !HLFIR-LABEL: func @_QQmain
-!FIR-LABEL: func @_QQmain
-!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
-!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
-!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
 program dispatch_test
   use funcs
   implicit none
@@ -59,31 +45,21 @@ program dispatch_test
   !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
   !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
   !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
   x = 1
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 
   !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
   !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
   x = 2
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 47c7587c5d08b..490bee24799c2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,14 +2289,15 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
+   ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure (without any construct's clauses: device, is_device_ptr,
-    nowait, nocontext, novariants). The structured block of a dispatch
-    construct is a single expression statement that contains a function
-    call or a subroutine call.
+    base procedure. The structured block of a dispatch construct is
+    a single expression statement that contains a function call or
+    a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 12818a5e56ac6..04c36b87d73ad 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -766,6 +766,10 @@ static LogicalResult
 convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
                    LLVM::ModuleTranslation &moduleTranslation) {
   auto dispatchOp = cast<omp::DispatchOp>(opInst);
+
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+
   auto &region = dispatchOp.getRegion();
   auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
                                     moduleTranslation);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c935f56836185..b8d4b93a6c8e3 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -41,6 +41,7 @@ func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
 }
 
 // Test multiple dispatch regions with stores selecting different variants.
+// TODO: Use declare_variant when the support is merged.
 // CHECK-LABEL: func @omp_dispatch_multiple
 // CHECK-SAME: (%[[X:.*]]: memref<i32>)
 func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index 8a3a7a6b5d8e6..d947a871ca30c 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -8,6 +8,7 @@ llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
 llvm.func @variant1() -> ()
 llvm.func @variant2() -> ()
 
+// TODO: Add support for declare_variant, when support to it is merged.
 // CHECK-LABEL: define void @foo_dispatch()
 llvm.func @foo_dispatch() {
   // CHECK: %[[ADDR:.*]] = load i32, ptr @x

>From 6eacbc3cd3d8e278d7b51ea898fe27e009de2228 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 01:09:09 -0500
Subject: [PATCH 4/4] 1) Handling -fc1 -fopenmp-simd 2) Todo testcase for
 clauses not supported in dispatch

---
 flang/lib/Semantics/rewrite-parse-tree.cpp |  5 +++
 flang/test/Lower/OpenMP/Todo/dispatch.f90  | 43 ++++++++++++++++++++++
 flang/test/Semantics/OpenMP/simd-only.f90  | 12 ++++++
 3 files changed, 60 insertions(+)
 create mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90

diff --git a/flang/lib/Semantics/rewrite-parse-tree.cpp b/flang/lib/Semantics/rewrite-parse-tree.cpp
index 4e1c9bae9c153..76a60c75488e3 100644
--- a/flang/lib/Semantics/rewrite-parse-tree.cpp
+++ b/flang/lib/Semantics/rewrite-parse-tree.cpp
@@ -191,6 +191,11 @@ void RewriteMutator::OpenMPSimdOnly(
                        &omp->value().u)}) {
           it = replaceInlineBlock(std::get<parser::Block>(ompBlock->t), it);
           continue;
+        } else if (auto *ompDispatch{
+                       std::get_if<parser::OpenMPDispatchConstruct>(
+                           &omp->value().u)}) {
+          it = replaceInlineBlock(std::get<parser::Block>(ompDispatch->t), it);
+          continue;
         } else if (auto *ompLoop{std::get_if<parser::OpenMPLoopConstruct>(
                        &omp->value().u)}) {
           if (LoopConstructIsSIMD(ompLoop)) {
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
new file mode 100644
index 0000000000000..f100cb40ba3d7
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -0,0 +1,43 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEPEND -o - %s 2>&1 | FileCheck %s --check-prefix=DEPEND
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEVICE -o - %s 2>&1 | FileCheck %s --check-prefix=DEVICE
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
+
+! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
+! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
+! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
+! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
+! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
+! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
+
+subroutine sub()
+#ifdef IS_DEVICE_PTR
+  use iso_c_binding
+  type(c_ptr) :: x
+#endif
+  integer :: r
+#ifdef DEPEND
+!$omp dispatch depend(inout: r)
+#endif
+#ifdef DEVICE
+!$omp dispatch device(0)
+#endif
+#ifdef IS_DEVICE_PTR
+!$omp dispatch is_device_ptr(x)
+#endif
+#ifdef NOCONTEXT
+!$omp dispatch nocontext(.false.)
+#endif
+#ifdef NOVARIANTS
+!$omp dispatch novariants(.true.)
+#endif
+#ifdef NOWAIT
+!$omp dispatch nowait
+#endif
+  call foo()
+contains
+  subroutine foo
+  end subroutine
+end subroutine sub
diff --git a/flang/test/Semantics/OpenMP/simd-only.f90 b/flang/test/Semantics/OpenMP/simd-only.f90
index 01370af0a00a7..860befaf43247 100644
--- a/flang/test/Semantics/OpenMP/simd-only.f90
+++ b/flang/test/Semantics/OpenMP/simd-only.f90
@@ -414,3 +414,15 @@ module test_declare_mapper
   ! CHECK-NOT: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpDeclareMapperDirective
   !$omp declare mapper(myvec_t :: v) map(v, v%data(1:v%len))
 end module
+
+! CHECK-LABEL: Name = 'test_dispatch'
+subroutine test_dispatch()
+  ! CHECK-NOT: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OpenMPDispatchConstruct
+  ! CHECK-NOT: OmpDirectiveName -> llvm::omp::Directive = dispatch
+  ! CHECK: ExecutionPartConstruct -> ExecutableConstruct -> ActionStmt -> CallStmt
+  !$omp dispatch
+  call foo()
+contains
+  subroutine foo()
+  end subroutine
+end subroutine


From llvmlistbot at llvm.org  Thu Jul  2 23:20:56 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 23:20:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-v] Fix null deref in
 SpecConstantOperationOp::verifyRegions (PR #207328)
Message-ID: <llvm.org/llvm/llvm-project/pull/207328@github.com>


https://github.com/aobolensk created https://github.com/llvm/llvm-project/pull/207328

None

>From 59e39b767b7e778c2aff4fb36554ae7675d8c83d Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 3 Jul 2026 08:20:16 +0200
Subject: [PATCH] [mlir][SPIR-v] Fix null deref in
 SpecConstantOperationOp::verifyRegions

---
 mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp        |  5 +++--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir | 11 +++++++++++
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index c0393888af5c3..99bc3913c56c6 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -2083,8 +2083,9 @@ LogicalResult spirv::SpecConstantOperationOp::verifyRegions() {
     return emitOpError("invalid enclosed op");
 
   for (auto operand : enclosedOp.getOperands())
-    if (!isa<spirv::ConstantOp, spirv::ReferenceOfOp,
-             spirv::SpecConstantOperationOp>(operand.getDefiningOp()))
+    if (!isa_and_present<spirv::ConstantOp, spirv::ReferenceOfOp,
+                         spirv::SpecConstantOperationOp>(
+            operand.getDefiningOp()))
       return emitOpError(
           "invalid operand, must be defined by a constant operation");
 
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index c0b73b926c764..37a9e97106de9 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -1220,3 +1220,14 @@ spirv.module Logical GLSL450 {
     spirv.Return
   }
 }
+
+// -----
+
+spirv.module Logical GLSL450 {
+  spirv.func @foo(%arg0: i32) -> () "None" {
+    // expected-error @+1 {{invalid operand, must be defined by a constant operation}}
+    %0 = spirv.SpecConstantOperation wraps "spirv.IAdd"(%arg0, %arg0) : (i32, i32) -> i32
+
+    spirv.Return
+  }
+}


From llvmlistbot at llvm.org  Thu Jul  2 23:21:01 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 02 Jul 2026 23:21:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix null deref in
 SpecConstantOperationOp::verifyRegions (PR #207328)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207328@github.com>
Message-ID: <6a4754cd.f14cf1a5.2c3aaa.18ad@mx.google.com>


https://github.com/aobolensk edited https://github.com/llvm/llvm-project/pull/207328

From llvmlistbot at llvm.org  Thu Jul  2 23:53:37 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Thu, 02 Jul 2026 23:53:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a475c71.dd2b4bd3.35cd5e.105b@mx.google.com>


================
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
----------------
grypp wrote:

Thanks for your work. I would split this PR into 2 pieces. One for nvvm and another for nvgpu. 

we also need to write tests for nvvm case. 

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Thu Jul  2 23:58:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 02 Jul 2026 23:58:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a475d87.58bdd12c.21857.e0de@mx.google.com>


================
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
----------------
weimin023 wrote:

Sure, I'll do this.

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 00:03:28 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 00:03:28 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a475ec0.af6ba3ed.2c2f30.fd6a@mx.google.com>


https://github.com/nikic auto_merge_disabled https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 00:03:55 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edb.f06f4994.1e3a60.2617@mx.google.com>


https://github.com/Saieiei edited https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:03:56 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edc.6c8668f2.74a35.37d2@mx.google.com>


https://github.com/Saieiei commented:

Thanks for working on the OpenMP interop support. I left a few comments on lowering correctness, diagnostics, and `omp_lib.h` parity.

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:03:56 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edc.a87eb5dc.2cf3a1.e69f@mx.google.com>


================
@@ -5872,7 +5872,12 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     context_.Say(GetContext().directiveSource,
         "Each interop-type may be specified at most once."_err_en_US);
   }
-  if (isDependClauseOccurred && !targetSyncCount) {
+  // Only enforce the depend/targetsync constraint when an init clause is
+  // present on this directive. For use/destroy-only directives, the interop
+  // object's type was established at its init site, which may be in a
+  // different statement — we cannot verify it statically here.
+  bool hasInitClause = (targetCount > 0 || targetSyncCount > 0);
----------------
Saieiei wrote:

I think this should track whether an `init` action clause is actually present, rather than using `targetCount || targetSyncCount` as a proxy.

At OpenMP 6.0, an untyped init clause is accepted:

```fortran
integer(omp_interop_kind) :: obj
integer :: x
!$omp interop init(obj) depend(inout: x)
```

With this change, both counts are zero, so `hasInitClause` becomes false and the `depend`/`targetsync` diagnostic is skipped. The typed control case still diagnoses correctly:

```fortran
!$omp interop init(target: obj) depend(inout: x)
```


https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:03:56 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edc.abb10101.208f3e.ab6c@mx.google.com>


================
@@ -3711,6 +3711,98 @@ convertOmpTaskgroupOp(omp::TaskgroupOp tgOp, llvm::IRBuilderBase &builder,
   return success();
 }
 
+static LogicalResult
+convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
+                        LLVM::ModuleTranslation &moduleTranslation) {
+  if (!initOp.getDependVars().empty() || initOp.getDependKinds() ||
+      !initOp.getDependIterated().empty() || initOp.getDependIteratedKinds())
+    return initOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropInitOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(initOp.getInteropVar());
+  llvm::Value *device = initOp.getDevice()
+                            ? moduleTranslation.lookupValue(initOp.getDevice())
+                            : nullptr;
+
+  // TODO: Handle depend clauses when supported.
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = initOp.getNowait();
+
+  // Emit one call per interop type. The runtime expects a single type per call.
+  for (auto typeAttr : initOp.getInteropTypes()) {
----------------
Saieiei wrote:

Could this avoid emitting multiple runtime init calls for a single `init` clause?

For example:

```fortran
integer(omp_interop_kind) :: obj
!$omp interop init(targetsync, target: obj)
```

currently emits two `__tgt_interop_init` calls on the same interop handle, first for `targetsync` and then for `target`. That makes the final initialized object order-dependent, since the last runtime init call wins.

This also seems to diverge from Clang, which lowers one `init` clause to a single `createOMPInteropInit` call.


https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:03:57 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:57 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edd.58bdd12c.21857.08a2@mx.google.com>


================
@@ -3711,6 +3711,98 @@ convertOmpTaskgroupOp(omp::TaskgroupOp tgOp, llvm::IRBuilderBase &builder,
   return success();
 }
 
+static LogicalResult
+convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
+                        LLVM::ModuleTranslation &moduleTranslation) {
+  if (!initOp.getDependVars().empty() || initOp.getDependKinds() ||
+      !initOp.getDependIterated().empty() || initOp.getDependIteratedKinds())
+    return initOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropInitOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(initOp.getInteropVar());
+  llvm::Value *device = initOp.getDevice()
+                            ? moduleTranslation.lookupValue(initOp.getDevice())
+                            : nullptr;
+
+  // TODO: Handle depend clauses when supported.
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = initOp.getNowait();
+
+  // Emit one call per interop type. The runtime expects a single type per call.
+  for (auto typeAttr : initOp.getInteropTypes()) {
+    auto interopTypeEnum = cast<omp::InteropTypeAttr>(typeAttr).getValue();
+    llvm::omp::OMPInteropType interopType;
+    switch (interopTypeEnum) {
+    case omp::InteropType::target:
+      interopType = llvm::omp::OMPInteropType::Target;
+      break;
+    case omp::InteropType::targetsync:
+      interopType = llvm::omp::OMPInteropType::TargetSync;
+      break;
+    }
+    ompBuilder->createOMPInteropInit(builder, interopVar, interopType, device,
----------------
Saieiei wrote:

Could we cast the `device` value to the runtime device-id type before passing it to `createOMPInteropInit`?

This currently crashes for a wider Fortran integer device expression:

```fortran
integer(8) :: dev
integer(omp_interop_kind) :: obj
!$omp interop device(dev) init(target: obj)
```

The runtime call expects an `i32` device argument, but the lowered value remains `i64`, which triggers a bad function-call signature assertion during LLVM IR generation. The regular `omp target device(i64)` path compiles successfully, so this looks specific to the interop translation path.


https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:03:57 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:03:57 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openmp] [Flang][OpenMP] Add support
 for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a475edd.7d41f3a1.d1cf7.2e7b@mx.google.com>


================
@@ -218,6 +222,28 @@
 
         integer (kind=omp_interop_kind), parameter, public :: omp_interop_none = 0
 
+!       interop property constants
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_id = -1
----------------
Saieiei wrote:

Could the matching fixed-form include file be updated as well?

This PR adds the new interop kinds, constants, and query-routine interfaces to `omp_lib.F90.var`, but not to `omp_lib.h.var`. Since `omp_lib.h.var` is configured separately into the shipped `omp_lib.h`, fixed-form users using:

```fortran
include 'omp_lib.h'
```

would not see the new `omp_ipr_*`, `omp_irc_*`, `omp_interop_property_kind`, `omp_interop_rc_kind`, or the new interop query routines.


https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul  3 00:11:21 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 00:11:21 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a476099.dd2b4bd3.35cd5e.f39a@mx.google.com>


https://github.com/nikic updated https://github.com/llvm/llvm-project/pull/204548

>From d2684d1fce721e0c4fe1d23b36b4fc5524ad6495 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Wed, 17 Jun 2026 15:15:56 +0200
Subject: [PATCH 1/6] [IR] Explicitly specify target feature for module asm

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can
now be specified as follows:

    module asm(target_features="+foo", target_cpu="bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO
scenarios we don't know what target features to use for parsing
the module-level inline assembly. Now they can be faithfully
preserved, even when merging inline assembly from different
modules with different features.
---
 clang/lib/CodeGen/CGObjCMac.cpp               |  6 +-
 clang/lib/CodeGen/CodeGenModule.cpp           |  6 +-
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c  | 11 +--
 clang/test/CodeGen/asm.c                      |  7 +-
 clang/test/CodeGen/asm_incbin.c               |  3 +-
 clang/test/CodeGen/cfi-salt.c                 | 11 +--
 clang/test/CodeGen/kcfi.c                     |  7 +-
 clang/test/CodeGenCUDA/filter-decl.cu         |  5 +-
 clang/test/CodeGenCXX/gnu-asm-constexpr.cpp   |  3 +-
 clang/test/CodeGenObjC/category-class.m       |  7 +-
 clang/test/CodeGenObjC/objc-runtime-name.m    |  7 +-
 clang/test/Frontend/ast-codegen.c             |  3 +-
 clang/test/Modules/codegen.test               |  3 +-
 llvm/include/llvm/AsmParser/LLToken.h         |  2 +
 llvm/include/llvm/Bitcode/LLVMBitCodes.h      |  3 +
 llvm/include/llvm/IR/Module.h                 | 79 +++++++++++++---
 llvm/lib/AsmParser/LLLexer.cpp                |  2 +
 llvm/lib/AsmParser/LLParser.cpp               | 37 +++++++-
 llvm/lib/Bitcode/Reader/BitcodeReader.cpp     | 18 +++-
 llvm/lib/Bitcode/Writer/BitcodeWriter.cpp     | 13 ++-
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    | 25 ++++--
 llvm/lib/IR/AsmWriter.cpp                     | 47 +++++++---
 llvm/lib/IR/Core.cpp                          | 13 ++-
 llvm/lib/LTO/LTO.cpp                          |  4 +-
 llvm/lib/Linker/IRMover.cpp                   |  9 +-
 llvm/lib/Object/ModuleSymbolTable.cpp         | 90 ++++++++++---------
 llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp     |  2 +-
 llvm/lib/Transforms/IPO/ExtractGV.cpp         |  2 +-
 .../Transforms/IPO/ThinLTOBitcodeWriter.cpp   |  2 +-
 .../Instrumentation/DataFlowSanitizer.cpp     | 26 +++---
 llvm/lib/Transforms/Utils/SplitModule.cpp     |  2 +-
 llvm/test/Assembler/module-asm-invalid.ll     | 25 ++++++
 llvm/test/Bitcode/compatibility-3.6.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.7.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.8.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.9.ll        |  3 +-
 llvm/test/Bitcode/compatibility-4.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-5.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-6.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility.ll            |  3 +-
 llvm/test/Bitcode/highLevelStructure.3.2.ll   |  3 +-
 llvm/test/Bitcode/module-asm.ll               | 29 ++++++
 .../test/CodeGen/RISCV/module-asm-features.ll | 12 +++
 .../DataFlowSanitizer/prefix-rename.ll        |  3 +-
 llvm/test/LTO/RISCV/module-asm.ll             | 18 ++++
 llvm/test/LTO/X86/inline-asm-lto-discard.ll   | 24 ++---
 llvm/test/Linker/Inputs/module-asm.ll         |  2 +
 .../link-arm-and-thumb-module-inline-asm.ll   | 17 ++--
 llvm/test/Linker/module-asm.ll                |  9 ++
 llvm/test/ThinLTO/X86/import-symver.ll        |  3 +-
 .../LowerTypeTests/export-symver.ll           |  3 +-
 .../cfi-icall-static-inline-asm.ll            |  3 +-
 .../ThinLTOBitcodeWriter/x86/module-asm.ll    |  3 +-
 .../llvm-reduce/deltas/ReduceModuleData.cpp   |  2 +-
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 11 ++-
 mlir/test/Target/LLVMIR/module-asm.mlir       |  5 +-
 56 files changed, 474 insertions(+), 177 deletions(-)
 create mode 100644 llvm/test/Assembler/module-asm-invalid.ll
 create mode 100644 llvm/test/Bitcode/module-asm.ll
 create mode 100644 llvm/test/CodeGen/RISCV/module-asm-features.ll
 create mode 100644 llvm/test/LTO/RISCV/module-asm.ll
 create mode 100644 llvm/test/Linker/Inputs/module-asm.ll
 create mode 100644 llvm/test/Linker/module-asm.ll

diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index 69c5e88f3c768..e518b875f44bb 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,10 +6610,6 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
-    Asm += CGM.getModule().getModuleInlineAsm();
-    if (!Asm.empty() && Asm.back() != '\n')
-      Asm += '\n';
-
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6624,7 +6620,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().setModuleInlineAsm(OS.str());
+    CGM.getModule().appendModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 813cd67e4dd51..41696a0001b4f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,7 +8057,11 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-    getModule().appendModuleInlineAsm(AD->getAsmString());
+
+    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
+    std::string Features = llvm::join(TargetOpts.Features, ",");
+    getModule().appendModuleInlineAsm(llvm::Module::GlobalAsmFragment(
+        AD->getAsmString(), Features, TargetOpts.CPU));
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 877da4fc669e3..eee40654240f7 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,13 +1,14 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
 // RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// CHECK: module asm "foo1"
+// CHECK: module asm(target_features="{{.*}}")
+// CHECK-NEXT: "foo1"
 __asm__ ("foo1");
-// CHECK: module asm "foo2"
+// CHECK-NEXT: "foo2"
 __asm__ ("foo2");
-// CHECK: module asm "foo3"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo3");
-// CHECK: module asm "foo4"
+// CHECK-NEXT: "foo4"
 __asm__ ("foo4");
-// CHECK: module asm "foo5"
+// CHECK-NEXT: "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index d7465b22fbbf6..bc54eb017ad12 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,9 +2,10 @@
 
 // PR10415:
 //
-// CHECK:      module asm "foo1"
-// CHECK-NEXT: module asm "foo2"
-// CHECK-NEXT: module asm "foo3"
+// CHECK: module asm(target_features="{{.*}}")
+// CHECK-NEXT: "foo1"
+// CHECK-NEXT: "foo2"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index 6234eea1d6738..bbc5d3af55ea2 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,4 +6,5 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm ".incbin \22foo.h\22"
+// CHECK: module asm
+// CHECK-NEXT: ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 2a90dc31d939b..18156dc014280 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,13 +26,14 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
+// CHECK-NOT: ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index da32859dd3d88..f08cf7635fb5a 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,10 +6,11 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index 02dacd0ad8ef4..f87d62a56631e 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,8 +5,9 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm "file scope asm is host only"
-// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
+// CHECK-HOST: module asm
+// CHECK-HOST-NEXT: "file scope asm is host only"
+// CHECK-DEVICE-NOT: "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 175a3b7bc588c..b7e0ba389c7dd 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,8 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm "nop; nop; nop; nop"
+    // CHECK: module asm(target_features="{{.*}}")
+    // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index 92fd36cbe4066..febe94ce5b9f5 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,9 +1,10 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
-// CHECK: module asm "\09.objc_category_name_A_foo=0"
-// CHECK: module asm "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm
+// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
+// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
+// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index c838b8c2c6f1a..a7f59caced390 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,9 +3,10 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm {{.*}}objc_class_name_XYZ=0
-// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
-// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm
+// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
+// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
+// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index 2e2e3d360dd17..bf35da069772d 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,8 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm "foo"
+// CHECK: module asm(target_features="{{.*}}")
+// CHECK-NEXT: "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 0af630a754805..5cc5b3c1cda2a 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,7 +18,8 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm "narf"
+BOTH: module asm
+BOTH-NEXT: "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index d2766a05ce9ba..95d3e67ff3849 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -86,6 +86,8 @@ enum Kind {
   kw_musttail,
   kw_notail,
   kw_target,
+  kw_target_cpu,
+  kw_target_features,
   kw_triple,
   kw_source_filename,
   kw_unwind,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index af4cac996a78c..e480dd0c35945 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,6 +120,9 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
+
+  MODULE_CODE_ASM_TARGET_FEATURES = 19, // [strchr x N]
+  MODULE_CODE_ASM_TARGET_CPU = 20,      // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 4f7e33969f16f..8a873a361ff06 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,6 +169,28 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
+  struct GlobalAsmFragment {
+    std::string Asm;
+    std::string TargetFeatures;
+    std::string TargetCPU;
+
+    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
+    GlobalAsmFragment(std::string AsmArg, std::string TargetFeatures = "",
+                      std::string TargetCPU = "")
+        : Asm(std::move(AsmArg)), TargetFeatures(std::move(TargetFeatures)),
+          TargetCPU(std::move(TargetCPU)) {
+      if (!Asm.empty() && Asm.back() != '\n')
+        Asm += '\n';
+    }
+
+    bool empty() const { return Asm.empty(); }
+
+    bool hasSameProperties(const GlobalAsmFragment &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+  };
+
 /// @}
 /// @name Member Variables
 /// @{
@@ -180,7 +202,8 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
+  /// Inline Asm at the global scope.
+  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -287,8 +310,17 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  /// @returns a string containing the module-scope inline assembly blocks.
-  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
+  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
+    return GlobalScopeAsm;
+  }
+
+  /// Get any module-scope inline assembly blocks.
+  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
+    return GlobalScopeAsm;
+  }
+
+  /// Return whether there is any module-scope inline assembly.
+  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -325,20 +357,45 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
+  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
+
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm = std::string(Asm);
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
+    GlobalScopeAsm.clear();
+    appendModuleInlineAsm(std::move(Fragment));
+  }
+
+  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
+    GlobalScopeAsm.clear();
+    append_range(GlobalScopeAsm, Fragments);
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm += Asm;
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.back().Asm += Fragment.Asm;
+    } else {
+      GlobalScopeAsm.emplace_back(std::move(Fragment));
+    }
+  }
+
+  /// Prepend to the module-scope inline assembly blocks.
+  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
+    } else {
+      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
+    }
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index 069a180056488..a978867e1ad14 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -595,6 +595,8 @@ lltok::Kind LLLexer::LexIdentifier() {
   KEYWORD(tail);
   KEYWORD(musttail);
   KEYWORD(notail);
+  KEYWORD(target_cpu);
+  KEYWORD(target_features);
   KEYWORD(target);
   KEYWORD(triple);
   KEYWORD(source_filename);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 75e6add0ec76f..cfd52c98d5b88 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,16 +641,47 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
+///   ::= 'module' 'asm' '(' 'target_features' '=' STRINGCONSTANT ','
+///                          'target_cpu' '=' STRINGCONSTANT ')'
+///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
-      parseStringConstant(AsmStr))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  M->appendModuleInlineAsm(AsmStr);
+  std::string TargetFeatures, TargetCPU;
+  if (EatIfPresent(lltok::lparen)) {
+    while (true) {
+      if (EatIfPresent(lltok::kw_target_features)) {
+        if (parseToken(lltok::equal, "expected '='") ||
+            parseStringConstant(TargetFeatures))
+          return true;
+      } else if (EatIfPresent(lltok::kw_target_cpu)) {
+        if (parseToken(lltok::equal, "expected '='") ||
+            parseStringConstant(TargetCPU))
+          return true;
+      } else {
+        return error(Lex.getLoc(),
+                     "expected one of 'target_features' or 'target_cpu'");
+      }
+      if (EatIfPresent(lltok::rparen))
+        break;
+      if (parseToken(lltok::comma, "expected ',' or ')'"))
+        return true;
+    }
+  }
+
+  do {
+    std::string AsmStrPart;
+    if (parseStringConstant(AsmStrPart))
+      return true;
+    AsmStr += AsmStrPart + "\n";
+  } while (Lex.getKind() == lltok::StringConstant);
+
+  M->appendModuleInlineAsm({AsmStr, TargetFeatures, TargetCPU});
   return false;
 }
 
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 2bd251efd05ce..aaf2116d3140b 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,6 +4580,9 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
+  // Apply to the following module asm.
+  std::string ModuleAsmTargetFeatures, ModuleAsmTargetCPU;
+
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4788,11 +4791,24 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
+    case bitc::MODULE_CODE_ASM_TARGET_FEATURES:
+      ModuleAsmTargetFeatures.clear();
+      if (convertToString(Record, 0, ModuleAsmTargetFeatures))
+        return error("Invalid asm target features record");
+      break;
+    case bitc::MODULE_CODE_ASM_TARGET_CPU:
+      ModuleAsmTargetCPU.clear();
+      if (convertToString(Record, 0, ModuleAsmTargetCPU))
+        return error("Invalid asm target cpu record");
+      break;
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->setModuleInlineAsm(S);
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(
+          S, ModuleAsmTargetFeatures, ModuleAsmTargetCPU));
+      ModuleAsmTargetFeatures.clear();
+      ModuleAsmTargetCPU.clear();
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 4aa203b0d12d1..d37cacb69f24b 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,9 +1551,16 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-  if (!M.getModuleInlineAsm().empty())
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
-                      0 /*TODO*/);
+
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    if (!Frag.TargetFeatures.empty())
+      writeStringRecord(Stream, bitc::MODULE_CODE_ASM_TARGET_FEATURES,
+                        Frag.TargetFeatures, 0);
+    if (!Frag.TargetCPU.empty())
+      writeStringRecord(Stream, bitc::MODULE_CODE_ASM_TARGET_CPU,
+                        Frag.TargetCPU, 0);
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
+  }
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index abb20a7919fa1..c16cc98339e56 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,6 +110,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -127,7 +128,6 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,13 +610,26 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    emitInlineAsm(
-        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
-        TM.Options.MCOptions, nullptr,
-        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+      if (!Frag.TargetFeatures.empty() || !Frag.TargetCPU.empty()) {
+        std::unique_ptr<MCSubtargetInfo> AsmSTI(
+            TM.getTarget().createMCSubtargetInfo(
+                TM.getTargetTriple(), Frag.TargetCPU, Frag.TargetFeatures));
+        emitInlineAsm(
+            Frag.Asm + "\n", *AsmSTI, TM.Options.MCOptions, nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      } else {
+        // If the module asm does not explicitly specify target features,
+        // fall back to default subtargetinfo.
+        emitInlineAsm(
+            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
+            nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      }
+    }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 665d7122af2c7..579898cf594ed 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,21 +3126,42 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (!M->getModuleInlineAsm().empty()) {
+  if (M->hasModuleInlineAsm()) {
     Out << '\n';
 
-    // Split the string into lines, to make it easier to read the .ll file.
-    StringRef Asm = M->getModuleInlineAsm();
-    do {
-      StringRef Front;
-      std::tie(Front, Asm) = Asm.split('\n');
-
-      // We found a newline, print the portion of the asm string from the
-      // last newline up to this newline.
-      Out << "module asm \"";
-      printEscapedString(Front, Out);
-      Out << "\"\n";
-    } while (!Asm.empty());
+    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
+      Out << "module asm";
+      if (!Frag.TargetFeatures.empty() || !Frag.TargetCPU.empty()) {
+        ListSeparator LS;
+        Out << "(";
+        if (!Frag.TargetFeatures.empty()) {
+          Out << LS;
+          Out << "target_features=\"";
+          printEscapedString(Frag.TargetFeatures, Out);
+          Out << "\"";
+        }
+        if (!Frag.TargetCPU.empty()) {
+          Out << LS;
+          Out << "target_cpu=\"";
+          printEscapedString(Frag.TargetCPU, Out);
+          Out << "\"";
+        }
+        Out << ")";
+      }
+      Out << "\n";
+      // Split the string into lines, to make it easier to read the .ll file.
+      StringRef Asm = Frag.Asm;
+      do {
+        StringRef Front;
+        std::tie(Front, Asm) = Asm.split('\n');
+
+        // We found a newline, print the portion of the asm string from the
+        // last newline up to this newline.
+        Out << "    \"";
+        printEscapedString(Front, Out);
+        Out << "\"\n";
+      } while (!Asm.empty());
+    }
   }
 
   printTypeIdentities();
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index 6c6c3b5c84ede..a7abd3eed31c3 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,7 +510,18 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  auto &Str = unwrap(M)->getModuleInlineAsm();
+  Module *Mod = unwrap(M);
+  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
+  if (Frags.empty()) {
+    *Len = 0;
+    return nullptr;
+  }
+
+  if (Frags.size() != 1)
+    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
+                          "more than one module inline assembly fragment");
+
+  auto &Str = Frags.begin()->Asm;
   *Len = Str.length();
   return Str.c_str();
 }
diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 6a754f95092bb..b672f4e5fac14 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
+    M.prependModuleInlineAsm(NewIA);
   }
 
   assert(MsymI == MsymE);
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde1cec..3b72b412d0b2e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,11 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
+      Module::GlobalAsmFragment NewFrag(Frag);
+      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
+      DstM.appendModuleInlineAsm(std::move(NewFrag));
+    }
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1573,7 +1576,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(S);
+        DstM.appendModuleInlineAsm(std::string(S));
       }
     });
   }
diff --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 1da5fa9c10a0b..b60e35fba1507 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,8 +72,7 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  StringRef InlineAsm = M.getModuleInlineAsm();
-  if (InlineAsm.empty())
+  if (!M.hasModuleInlineAsm())
     return;
 
   std::string Err;
@@ -90,52 +89,55 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
-  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
-  if (!STI)
-    return;
-
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  std::unique_ptr<MemoryBuffer> Buffer(
-      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
-  SourceMgr SrcMgr;
-  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-  std::unique_ptr<MCObjectFileInfo> MOFI(
-      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-  MCCtx.setObjectFileInfo(MOFI.get());
-  RecordStreamer Streamer(MCCtx, M);
-  T->createNullTargetStreamer(Streamer);
-
-  std::unique_ptr<MCAsmParser> Parser(
-      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-  std::unique_ptr<MCTargetAsmParser> TAP(
-      T->createMCAsmParser(*STI, *Parser, *MCII));
-  if (!TAP)
-    return;
-
-  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                 const SourceMgr &SrcMgr,
-                                 std::vector<const MDNode *> &LocInfos) {
-    M.getContext().diagnose(
-        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-  });
-
-  // Module-level inline asm is assumed to use At&t syntax (see
-  // AsmPrinter::doInitialization()).
-  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-  Parser->setSymbolScanningMode(true);
-
-  Parser->setTargetParser(*TAP);
-  if (Parser->Run(false))
-    return;
-
-  Init(Streamer);
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    std::unique_ptr<MCSubtargetInfo> STI(
+        T->createMCSubtargetInfo(TT, Frag.TargetCPU, Frag.TargetFeatures));
+    if (!STI)
+      return;
+
+    std::unique_ptr<MemoryBuffer> Buffer(
+        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
+    SourceMgr SrcMgr;
+    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+    std::unique_ptr<MCObjectFileInfo> MOFI(
+        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+    MCCtx.setObjectFileInfo(MOFI.get());
+    RecordStreamer Streamer(MCCtx, M);
+    T->createNullTargetStreamer(Streamer);
+
+    std::unique_ptr<MCAsmParser> Parser(
+        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+    std::unique_ptr<MCTargetAsmParser> TAP(
+        T->createMCAsmParser(*STI, *Parser, *MCII));
+    if (!TAP)
+      return;
+
+    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                   const SourceMgr &SrcMgr,
+                                   std::vector<const MDNode *> &LocInfos) {
+      M.getContext().diagnose(
+          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+    });
+
+    // Module-level inline asm is assumed to use At&t syntax (see
+    // AsmPrinter::doInitialization()).
+    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+    Parser->setSymbolScanningMode(true);
+
+    Parser->setTargetParser(*TAP);
+    if (Parser->Run(false))
+      return;
+
+    Init(Streamer);
+  }
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(
diff --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 2a5d16a470c4a..22d0d97090746 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
   }
 
   // Register the NSDI handler before calling the base class so that
diff --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 16fdb93e2b21b..4b76b02c32fe4 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually
diff --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index a486979a58c2d..e857c94330bad 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->setModuleInlineAsm("");
+  MergedM->removeModuleInlineAsm();
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.
diff --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index c349fe33dd237..375ac069272eb 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,18 +1283,20 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  std::string Asm = GV->getParent()->getModuleInlineAsm();
-  std::string SearchStr = ".symver " + GVName + ",";
-  size_t Pos = Asm.find(SearchStr);
-  if (Pos != std::string::npos) {
-    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
-    Pos = Asm.find('@');
-
-    if (Pos == std::string::npos)
-      report_fatal_error(Twine("unsupported .symver: ", Asm));
-
-    Asm.replace(Pos, 1, Suffix + "@");
-    GV->getParent()->setModuleInlineAsm(Asm);
+  for (Module::GlobalAsmFragment &Frag :
+       GV->getParent()->getModuleInlineAsm()) {
+    std::string SearchStr = ".symver " + GVName + ",";
+    size_t Pos = Frag.Asm.find(SearchStr);
+    if (Pos != std::string::npos) {
+      Frag.Asm.replace(Pos, SearchStr.size(),
+                       ".symver " + GVName + Suffix + ",");
+      Pos = Frag.Asm.find('@');
+
+      if (Pos == std::string::npos)
+        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
+
+      Frag.Asm.replace(Pos, 1, Suffix + "@");
+    }
   }
 }
 
diff --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 64910b4d1ce99..8e70902c4dac2 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->setModuleInlineAsm("");
+      MPart->removeModuleInlineAsm();
     ModuleCallback(std::move(MPart));
   }
 }
diff --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
new file mode 100644
index 0000000000000..de2a1c32ca6b5
--- /dev/null
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -0,0 +1,25 @@
+; RUN: split-file %s %t
+; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
+; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
+; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
+; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
+
+;--- invalid_prop.ll
+; CHECK: expected one of 'target_features' or 'target_cpu'
+module asm(foo="bar")
+    "asm"
+
+;--- missing_value1.ll
+; CHECK: expected '='
+module asm(target_features)
+    "asm"
+
+;--- missing_value2.ll
+; CHECK: expected string constant
+module asm(target_features=)
+    "asm"
+
+;--- missing_paren.ll
+; CHECK: expected ',' or ')'
+module asm(target_features="bar"
+    "asm"
diff --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index 288e0fc50a9ba..c79086bad1ba5 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index ef9490310fe5d..00b2bb9b30b62 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 538d6101b3994..5b86d5ffa1fee 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index 3bfef73e485dd..eca4bbdf03766 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index aaf9fe5492171..8551b98815b1a 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index b820417107e8b..883ac3eaaa293 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 67e96c7277cc3..624d5bd7703c6 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index d55aa1dc496df..97889d4687516 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,7 +16,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index f2770ea4e4d34..cd9a34c810d57 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,7 +9,8 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm "some assembly"
+; CHECK: module asm
+; CHECK-NEXT: "some assembly"
 module asm "some assembly"
 
 ; Named Types Test
diff --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
new file mode 100644
index 0000000000000..4d2144c0cba33
--- /dev/null
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -0,0 +1,29 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; CHECK: module asm
+; CHECK:     "asm line 0"
+; CHECK: module asm(target_features="+foo", target_cpu="foo_cpu")
+; CHECK:     "asm line 1"
+; CHECK:     "asm line 2"
+; CHECK: module asm(target_features="+bar")
+; CHECK:     "asm line 3"
+; CHECK: module asm(target_cpu="bar_cpu")
+; CHECK:     "asm line 4"
+; CHECK: module asm
+; CHECK:     "asm line 5"
+
+module asm
+    "asm line 0"
+
+module asm(target_features="+foo", target_cpu="foo_cpu")
+    "asm line 1"
+    "asm line 2"
+
+module asm(target_features="+bar")
+    "asm line 3"
+
+module asm(target_cpu="bar_cpu")
+    "asm line 4"
+
+module asm
+    "asm line 5"
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
new file mode 100644
index 0000000000000..eecc967734ad0
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -0,0 +1,12 @@
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s
+
+; This should work fine, because the module asm specifies the necessary
+; target features
+
+; CHECK: fld ft0, 0(sp)
+
+module asm(target_features="+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 454c496d1d4bb..829dd829d51f9 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,7 +2,8 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan
diff --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
new file mode 100644
index 0000000000000..c92343b457ff7
--- /dev/null
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -0,0 +1,18 @@
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
+; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
+; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
+; RUN: FileCheck %s --input-file %t.s.0
+
+; NM: T func
+
+; CHECK: fld ft0, 0(sp)
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+target triple = "riscv64-unknown-linux-gnu"
+
+module asm(target_features="+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 88492e4366489..6c5173c3f9553 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,17 +32,19 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1:      module asm ".lto_discard foo"
-; ASM1-NEXT: module asm ".weak foo"
-; ASM1-NEXT: module asm ".equ foo,bar"
-
-; ASM2:      module asm ".lto_discard foo"
-; ASM2-NEXT: module asm ".weak foo"
-; ASM2-NEXT: module asm ".equ foo,bar"
-; ASM2-NEXT: module asm ".lto_discard"
-; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  module asm ".lto_discard foo"
+; ASM1: module asm
+; ASM1-NEXT: ".lto_discard foo"
+; ASM1-NEXT: ".weak foo"
+; ASM1-NEXT: ".equ foo,bar"
+
+; ASM2: module asm
+; ASM2-NEXT: ".lto_discard foo"
+; ASM2-NEXT: ".weak foo"
+; ASM2-NEXT: ".equ foo,bar"
+; ASM2-NEXT: ".lto_discard"
+; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  ".lto_discard foo"
 
 ; SYM: T foo
 
diff --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
new file mode 100644
index 0000000000000..01c55a875af4f
--- /dev/null
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -0,0 +1,2 @@
+module asm(target_features="+bar")
+    "asm 2"
diff --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 5e12a8c88d90a..612bce5be18f1 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,11 +9,12 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK:      module asm ".text"
-; CHECK-NEXT: module asm ".balign 4"
-; CHECK-NEXT: module asm ".arm"
-; CHECK-NEXT: module asm "add r1, r2, r2"
-; CHECK-NEXT: module asm ".text"
-; CHECK-NEXT: module asm ".balign 2"
-; CHECK-NEXT: module asm ".thumb"
-; CHECK-NEXT: module asm "orn r1, r2, r2"
+; CHECK: module asm
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 4"
+; CHECK-NEXT: ".arm"
+; CHECK-NEXT: "add r1, r2, r2"
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 2"
+; CHECK-NEXT: ".thumb"
+; CHECK-NEXT: "orn r1, r2, r2"
diff --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
new file mode 100644
index 0000000000000..9f43c7a22514d
--- /dev/null
+++ b/llvm/test/Linker/module-asm.ll
@@ -0,0 +1,9 @@
+; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
+
+; CHECK: module asm(target_features="+foo")
+; CHECK-NEXT: "asm 1"
+; CHECK: module asm(target_features="+bar")
+; CHECK-NEXT: "asm 2"
+
+module asm(target_features="+foo")
+    "asm 1"
diff --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 556c4fd992f0d..9525527b4531c 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,7 +9,8 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm
+; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.
diff --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index ea4594a359cc0..7c3ed3da3537e 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,6 +1,7 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm ".symver external_addrtaken, alias1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index d8ebae17d4693..61141c71335bf 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,7 +3,8 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm
+; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 587ab3fbb5020..5796537661e60 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,6 +7,7 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm "ret"
+; M0: module asm
+; M0-NEXT: "ret"
 ; M1-NOT: module asm
 module asm "ret"
diff --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index 4aeaef6d8d676..b5b68404e8690 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.setModuleInlineAsm("");
+    Program.removeModuleInlineAsm();
 }
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index 3038a6a2ed986..bb55678f91005 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,12 +1150,15 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
-  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
-    if (!line.empty())
-      asmArrayAttr.push_back(builder.getStringAttr(line));
+  for (const llvm::Module::GlobalAsmFragment &Frag :
+       llvmModule->getModuleInlineAsm()) {
+    // TODO: Preserve module asm properties.
+    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
+      if (!line.empty())
+        asmArrayAttr.push_back(builder.getStringAttr(line));
+  }
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));
diff --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 2afb37cf005bf..813eeeafdeb44 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,5 +2,6 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm "foo"
-// CHECK: module asm "bar"
+// CHECK: module asm
+// CHECK-NEXT: "foo"
+// CHECK-NEXT: "bar"

>From bc61f44473b4b5c3c7cb3f724f762528d4a8db9c Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Mon, 22 Jun 2026 08:48:23 +0200
Subject: [PATCH 2/6] Add LangRef

---
 llvm/docs/LangRef.rst | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5ce6019660e0b..5f8f02e25c803 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,19 +3357,34 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks. These blocks are internally
-concatenated by LLVM and treated as a single unit, but may be separated
-in the ``.ll`` file if desired. The syntax is very simple:
+to the GCC "file scope inline asm" blocks:
 
 .. code-block:: llvm
 
-    module asm "inline asm code goes here"
-    module asm "more can go here"
+    module asm
+        "inline asm code goes here"
+        "more can go here"
 
-The strings can contain any character by escaping non-printable
+The sequence of string literals is internally concatenated using newlines as
+separators. The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
+Additionally, module-level inline assembly can specify an optional list of
+properties:
+
+.. code-block:: llvm
+
+    module asm(target_features="+foo", target_cpu="bar")
+        "inline asm code goes here"
+        "more can go here"
+
+Currently, the only supported properties are ``target_features`` and
+``target_cpu``, with the same meaning as the ``"target-features"`` and
+``"target-cpu"`` function attributes.
+
+Consecutive blocks with identical properties will be concatenated into one.
+
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 

>From 1048842fe4b32dd3bc0579711f98aa9053b5650a Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Mon, 22 Jun 2026 12:35:53 +0200
Subject: [PATCH 3/6] Emit option push/pop on RISCV

---
 llvm/include/llvm/CodeGen/AsmPrinter.h        | 13 ++++++++++
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    |  4 ++-
 llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp     | 25 +++++++++++--------
 .../test/CodeGen/RISCV/module-asm-features.ll |  7 +++++-
 4 files changed, 37 insertions(+), 12 deletions(-)

diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 78e64440657cc..335bf9de9a071 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,6 +930,19 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
+  /// Emit necessary directives to allow use of instructions that are permitted
+  /// by target features enabled by STI, but are not permitted by target
+  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
+  /// Returns whether anything was emitted.
+  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+    return false;
+  }
+
+  /// Emit necessary directives to restore target feature state.
+  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
+  /// call.
+  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
+
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index c16cc98339e56..44deefb9b5530 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -618,9 +618,11 @@ bool AsmPrinter::doInitialization(Module &M) {
         std::unique_ptr<MCSubtargetInfo> AsmSTI(
             TM.getTarget().createMCSubtargetInfo(
                 TM.getTargetTriple(), Frag.TargetCPU, Frag.TargetFeatures));
+        bool DidPush = emitTargetFeaturePush(*AsmSTI);
         emitInlineAsm(
-            Frag.Asm + "\n", *AsmSTI, TM.Options.MCOptions, nullptr,
+            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
             InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+        emitTargetFeaturePop(*AsmSTI, DidPush);
       } else {
         // If the module asm does not explicitly specify target features,
         // fall back to default subtargetinfo.
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index dbda25c34cee0..c308f4412d1e3 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,7 +113,8 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitDirectiveOptionArch();
+  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
+  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -523,20 +524,20 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitDirectiveOptionArch() {
+bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                                : RISCVOptionArchArgType::Minus;
-    StringRef ExtName = Feature.key();
+    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                               : RISCVOptionArchArgType::Minus;
+    StringRef ExtName = Feature.Key;
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
   }
@@ -549,11 +550,16 @@ bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   return false;
 }
 
+void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
+                                           bool DidPush) {
+  if (DidPush)
+    getTargetStreamer().emitDirectiveOptionPop();
+}
+
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
-  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitDirectiveOptionArch();
+  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -561,8 +567,7 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  if (EmittedOptionArch)
-    RTS.emitDirectiveOptionPop();
+  emitTargetFeaturePop(*STI, EmittedOptionArch);
   return false;
 }
 
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
index eecc967734ad0..3e9483cbf4461 100644
--- a/llvm/test/CodeGen/RISCV/module-asm-features.ll
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -1,9 +1,14 @@
-; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
 
 ; This should work fine, because the module asm specifies the necessary
 ; target features
 
+; SAME-FEATURES-NOT: .option arch
+; EXTRA-FEATURES: .option push
+; EXTRA-FEATURES: .option arch, +d
 ; CHECK: fld ft0, 0(sp)
+; EXTRA-FEATURES: .option pop
 
 module asm(target_features="+d")
     ".globl func"

>From 66610af4638149d499afea38bfe4b7c9d3cde524 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Thu, 25 Jun 2026 14:37:12 +0200
Subject: [PATCH 4/6] Make the implementation easier to extend

---
 clang/lib/CodeGen/CodeGenModule.cpp           |  8 +++--
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c  |  2 +-
 clang/test/CodeGen/asm.c                      |  2 +-
 clang/test/CodeGenCXX/gnu-asm-constexpr.cpp   |  2 +-
 clang/test/Frontend/ast-codegen.c             |  2 +-
 llvm/docs/LangRef.rst                         |  2 +-
 llvm/docs/ReleaseNotes.md                     |  5 +++
 llvm/include/llvm/AsmParser/LLToken.h         |  2 --
 llvm/include/llvm/Bitcode/LLVMBitCodes.h      |  3 +-
 llvm/include/llvm/IR/Module.h                 | 33 +++++++++++++-----
 llvm/lib/AsmParser/LLLexer.cpp                |  2 --
 llvm/lib/AsmParser/LLParser.cpp               | 34 ++++++++++---------
 llvm/lib/Bitcode/Reader/BitcodeReader.cpp     | 27 +++++++--------
 llvm/lib/Bitcode/Writer/BitcodeWriter.cpp     | 15 ++++----
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    |  7 ++--
 llvm/lib/IR/AsmWriter.cpp                     | 16 ++++-----
 llvm/lib/IR/Module.cpp                        | 20 +++++++++++
 llvm/lib/Object/ModuleSymbolTable.cpp         |  4 +--
 llvm/test/Assembler/module-asm-invalid.ll     | 10 +++---
 llvm/test/Bitcode/module-asm.ll               | 12 +++----
 .../test/CodeGen/RISCV/module-asm-features.ll |  2 +-
 llvm/test/LTO/RISCV/module-asm.ll             |  2 +-
 llvm/test/Linker/Inputs/module-asm.ll         |  2 +-
 llvm/test/Linker/module-asm.ll                |  6 ++--
 24 files changed, 130 insertions(+), 90 deletions(-)

diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 41696a0001b4f..e155fdd752d7f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8059,9 +8059,11 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     auto *AD = cast<FileScopeAsmDecl>(D);
 
     const TargetOptions &TargetOpts = getTarget().getTargetOpts();
-    std::string Features = llvm::join(TargetOpts.Features, ",");
-    getModule().appendModuleInlineAsm(llvm::Module::GlobalAsmFragment(
-        AD->getAsmString(), Features, TargetOpts.CPU));
+    llvm::Module::GlobalAsmProperties Props;
+    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
+    Props.TargetCPU = TargetOpts.CPU;
+    getModule().appendModuleInlineAsm(
+        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index eee40654240f7..cc1f4b585f209 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,7 +1,7 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
 // RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// CHECK: module asm(target_features="{{.*}}")
+// CHECK: module asm(target_features: "{{.*}}")
 // CHECK-NEXT: "foo1"
 __asm__ ("foo1");
 // CHECK-NEXT: "foo2"
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index bc54eb017ad12..ea69c7a17d1cb 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,7 +2,7 @@
 
 // PR10415:
 //
-// CHECK: module asm(target_features="{{.*}}")
+// CHECK: module asm(target_features: "{{.*}}")
 // CHECK-NEXT: "foo1"
 // CHECK-NEXT: "foo2"
 // CHECK-NEXT: "foo3"
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index b7e0ba389c7dd..2b4abee5dc05f 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,7 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm(target_features="{{.*}}")
+    // CHECK: module asm(target_features: "{{.*}}")
     // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index bf35da069772d..c26b0a1908379 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,7 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm(target_features="{{.*}}")
+// CHECK: module asm(target_features: "{{.*}}")
 // CHECK-NEXT: "foo"
 __asm__("foo");
 
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5f8f02e25c803..2c8276d04b668 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3375,7 +3375,7 @@ properties:
 
 .. code-block:: llvm
 
-    module asm(target_features="+foo", target_cpu="bar")
+    module asm(target_features: "+foo", target_cpu: "bar")
         "inline asm code goes here"
         "more can go here"
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index e42a959ab3101..9b2208b34da82 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -102,6 +102,7 @@ Makes programs 10x faster by doing Special New Thing.
   outlining. Add the `noinline` and `nooutline` attributes as well in cases
   where inlining and outlining should additionally be disabled.
 
+<<<<<<< HEAD
 * Added support for ``callgraph`` metadata. The `!callgraph` metadata
   associates a function definition with its type identifier and is used for call
   graph section generation. See the [callgraph Metadata](https://llvm.org/docs/LangRef.html#callgraph-metadata)
@@ -111,6 +112,10 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
+* Module-level inline assembly now accepts optional `target_features` and
+  `target_cpu` properties. This resolves errors during LTO on some
+  architectures.
+
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/AsmParser/LLToken.h b/llvm/include/llvm/AsmParser/LLToken.h
index 95d3e67ff3849..d2766a05ce9ba 100644
--- a/llvm/include/llvm/AsmParser/LLToken.h
+++ b/llvm/include/llvm/AsmParser/LLToken.h
@@ -86,8 +86,6 @@ enum Kind {
   kw_musttail,
   kw_notail,
   kw_target,
-  kw_target_cpu,
-  kw_target_features,
   kw_triple,
   kw_source_filename,
   kw_unwind,
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index e480dd0c35945..1761e7a542a73 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -121,8 +121,7 @@ enum ModuleCodes {
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
 
-  MODULE_CODE_ASM_TARGET_FEATURES = 19, // [strchr x N]
-  MODULE_CODE_ASM_TARGET_CPU = 20,      // [strchr x N]
+  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 8a873a361ff06..a82ac363dee65 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,16 +169,34 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
-  struct GlobalAsmFragment {
-    std::string Asm;
+  struct GlobalAsmProperties {
     std::string TargetFeatures;
     std::string TargetCPU;
 
+    /// Set a property using a string name.
+    /// Returns whether the property name was valid.
+    bool set(StringRef Name, std::string Value);
+
+    /// Get a list of set properties as pairs of key and value.
+    SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+
+    bool operator==(const GlobalAsmProperties &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+
+    bool operator!=(const GlobalAsmProperties &Other) const {
+      return !(*this == Other);
+    }
+  };
+
+  struct GlobalAsmFragment {
+    std::string Asm;
+    GlobalAsmProperties Props;
+
     GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
-    GlobalAsmFragment(std::string AsmArg, std::string TargetFeatures = "",
-                      std::string TargetCPU = "")
-        : Asm(std::move(AsmArg)), TargetFeatures(std::move(TargetFeatures)),
-          TargetCPU(std::move(TargetCPU)) {
+    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
+        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
       if (!Asm.empty() && Asm.back() != '\n')
         Asm += '\n';
     }
@@ -186,8 +204,7 @@ class LLVM_ABI Module {
     bool empty() const { return Asm.empty(); }
 
     bool hasSameProperties(const GlobalAsmFragment &Other) const {
-      return TargetFeatures == Other.TargetFeatures &&
-             TargetCPU == Other.TargetCPU;
+      return Props == Other.Props;
     }
   };
 
diff --git a/llvm/lib/AsmParser/LLLexer.cpp b/llvm/lib/AsmParser/LLLexer.cpp
index a978867e1ad14..069a180056488 100644
--- a/llvm/lib/AsmParser/LLLexer.cpp
+++ b/llvm/lib/AsmParser/LLLexer.cpp
@@ -595,8 +595,6 @@ lltok::Kind LLLexer::LexIdentifier() {
   KEYWORD(tail);
   KEYWORD(musttail);
   KEYWORD(notail);
-  KEYWORD(target_cpu);
-  KEYWORD(target_features);
   KEYWORD(target);
   KEYWORD(triple);
   KEYWORD(source_filename);
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index cfd52c98d5b88..0a5b4c6c91879 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,8 +641,8 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
-///   ::= 'module' 'asm' '(' 'target_features' '=' STRINGCONSTANT ','
-///                          'target_cpu' '=' STRINGCONSTANT ')'
+///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
+///                          'property_name2:' STRINGCONSTANT ')'
 ///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
@@ -652,21 +652,23 @@ bool LLParser::parseModuleAsm() {
   if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  std::string TargetFeatures, TargetCPU;
+  Module::GlobalAsmProperties Props;
   if (EatIfPresent(lltok::lparen)) {
     while (true) {
-      if (EatIfPresent(lltok::kw_target_features)) {
-        if (parseToken(lltok::equal, "expected '='") ||
-            parseStringConstant(TargetFeatures))
-          return true;
-      } else if (EatIfPresent(lltok::kw_target_cpu)) {
-        if (parseToken(lltok::equal, "expected '='") ||
-            parseStringConstant(TargetCPU))
-          return true;
-      } else {
-        return error(Lex.getLoc(),
-                     "expected one of 'target_features' or 'target_cpu'");
-      }
+      std::string Key, Value;
+      SMLoc Loc = Lex.getLoc();
+      if (Lex.getKind() != lltok::LabelStr)
+        return error(Loc, "expected property name followed by ':'");
+
+      Key = Lex.getStrVal();
+      Lex.Lex();
+
+      if (parseStringConstant(Value))
+        return true;
+
+      if (!Props.set(Key, Value))
+        return error(Loc, "unknown property name");
+
       if (EatIfPresent(lltok::rparen))
         break;
       if (parseToken(lltok::comma, "expected ',' or ')'"))
@@ -681,7 +683,7 @@ bool LLParser::parseModuleAsm() {
     AsmStr += AsmStrPart + "\n";
   } while (Lex.getKind() == lltok::StringConstant);
 
-  M->appendModuleInlineAsm({AsmStr, TargetFeatures, TargetCPU});
+  M->appendModuleInlineAsm({AsmStr, Props});
   return false;
 }
 
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index aaf2116d3140b..06e9aea4fd3ac 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4581,7 +4581,7 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
   // Apply to the following module asm.
-  std::string ModuleAsmTargetFeatures, ModuleAsmTargetCPU;
+  Module::GlobalAsmProperties Props;
 
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
@@ -4791,24 +4791,23 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
-    case bitc::MODULE_CODE_ASM_TARGET_FEATURES:
-      ModuleAsmTargetFeatures.clear();
-      if (convertToString(Record, 0, ModuleAsmTargetFeatures))
-        return error("Invalid asm target features record");
-      break;
-    case bitc::MODULE_CODE_ASM_TARGET_CPU:
-      ModuleAsmTargetCPU.clear();
-      if (convertToString(Record, 0, ModuleAsmTargetCPU))
-        return error("Invalid asm target cpu record");
+    case bitc::MODULE_CODE_ASM_PROPERTY: {
+      std::string Str;
+      if (convertToString(Record, 0, Str))
+        return error("Invalid module asm record");
+      size_t SepPos = Str.find('\0');
+      if (SepPos == std::string::npos)
+        return error("Invalid module asm record");
+      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
+        return error("Unknown module asm property");
       break;
+    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(
-          S, ModuleAsmTargetFeatures, ModuleAsmTargetCPU));
-      ModuleAsmTargetFeatures.clear();
-      ModuleAsmTargetCPU.clear();
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
+      Props = {};
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index d37cacb69f24b..493cbec7506f2 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1553,12 +1553,15 @@ void ModuleBitcodeWriter::writeModuleInfo() {
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
 
   for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    if (!Frag.TargetFeatures.empty())
-      writeStringRecord(Stream, bitc::MODULE_CODE_ASM_TARGET_FEATURES,
-                        Frag.TargetFeatures, 0);
-    if (!Frag.TargetCPU.empty())
-      writeStringRecord(Stream, bitc::MODULE_CODE_ASM_TARGET_CPU,
-                        Frag.TargetCPU, 0);
+    SmallVector<std::pair<StringRef, StringRef>> Props =
+        Frag.Props.getAsStrings();
+    for (auto [Key, Value] : Props) {
+      SmallVector<unsigned, 64> Record;
+      Record.append(Key.begin(), Key.end());
+      Record.push_back(0);
+      Record.append(Value.begin(), Value.end());
+      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
+    }
     writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
   }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 44deefb9b5530..8248a05f1a35f 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -614,10 +614,11 @@ bool AsmPrinter::doInitialization(Module &M) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
     for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-      if (!Frag.TargetFeatures.empty() || !Frag.TargetCPU.empty()) {
+      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
         std::unique_ptr<MCSubtargetInfo> AsmSTI(
-            TM.getTarget().createMCSubtargetInfo(
-                TM.getTargetTriple(), Frag.TargetCPU, Frag.TargetFeatures));
+            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
+                                                 Frag.Props.TargetCPU,
+                                                 Frag.Props.TargetFeatures));
         bool DidPush = emitTargetFeaturePush(*AsmSTI);
         emitInlineAsm(
             Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 579898cf594ed..02d6bcd90add0 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3131,19 +3131,15 @@ void AssemblyWriter::printModule(const Module *M) {
 
     for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
       Out << "module asm";
-      if (!Frag.TargetFeatures.empty() || !Frag.TargetCPU.empty()) {
+      SmallVector<std::pair<StringRef, StringRef>> Props =
+          Frag.Props.getAsStrings();
+      if (!Props.empty()) {
         ListSeparator LS;
         Out << "(";
-        if (!Frag.TargetFeatures.empty()) {
+        for (auto [Key, Value] : Props) {
           Out << LS;
-          Out << "target_features=\"";
-          printEscapedString(Frag.TargetFeatures, Out);
-          Out << "\"";
-        }
-        if (!Frag.TargetCPU.empty()) {
-          Out << LS;
-          Out << "target_cpu=\"";
-          printEscapedString(Frag.TargetCPU, Out);
+          Out << Key << ": \"";
+          printEscapedString(Value, Out);
           Out << "\"";
         }
         Out << ")";
diff --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 40ef7e354da72..34e44e26d5db5 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,3 +977,23 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
+
+bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
+  if (Name == "target_features")
+    TargetFeatures = std::move(Value);
+  else if (Name == "target_cpu")
+    TargetCPU = std::move(Value);
+  else
+    return false;
+  return true;
+}
+
+SmallVector<std::pair<StringRef, StringRef>>
+Module::GlobalAsmProperties::getAsStrings() const {
+  SmallVector<std::pair<StringRef, StringRef>> Props;
+  if (!TargetFeatures.empty())
+    Props.emplace_back("target_features", TargetFeatures);
+  if (!TargetCPU.empty())
+    Props.emplace_back("target_cpu", TargetCPU);
+  return Props;
+}
diff --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index b60e35fba1507..44f717ec12323 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -94,8 +94,8 @@ initializeRecordStreamer(const Module &M,
     return;
 
   for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    std::unique_ptr<MCSubtargetInfo> STI(
-        T->createMCSubtargetInfo(TT, Frag.TargetCPU, Frag.TargetFeatures));
+    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
+        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
     if (!STI)
       return;
 
diff --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
index de2a1c32ca6b5..34e5b71e683a9 100644
--- a/llvm/test/Assembler/module-asm-invalid.ll
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -5,21 +5,21 @@
 ; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
 
 ;--- invalid_prop.ll
-; CHECK: expected one of 'target_features' or 'target_cpu'
-module asm(foo="bar")
+; CHECK: unknown property name
+module asm(foo: "bar")
     "asm"
 
 ;--- missing_value1.ll
-; CHECK: expected '='
+; CHECK: expected property name followed by ':'
 module asm(target_features)
     "asm"
 
 ;--- missing_value2.ll
 ; CHECK: expected string constant
-module asm(target_features=)
+module asm(target_features:)
     "asm"
 
 ;--- missing_paren.ll
 ; CHECK: expected ',' or ')'
-module asm(target_features="bar"
+module asm(target_features: "bar"
     "asm"
diff --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
index 4d2144c0cba33..406f25c5dd28c 100644
--- a/llvm/test/Bitcode/module-asm.ll
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -2,12 +2,12 @@
 
 ; CHECK: module asm
 ; CHECK:     "asm line 0"
-; CHECK: module asm(target_features="+foo", target_cpu="foo_cpu")
+; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
 ; CHECK:     "asm line 1"
 ; CHECK:     "asm line 2"
-; CHECK: module asm(target_features="+bar")
+; CHECK: module asm(target_features: "+bar")
 ; CHECK:     "asm line 3"
-; CHECK: module asm(target_cpu="bar_cpu")
+; CHECK: module asm(target_cpu: "bar_cpu")
 ; CHECK:     "asm line 4"
 ; CHECK: module asm
 ; CHECK:     "asm line 5"
@@ -15,14 +15,14 @@
 module asm
     "asm line 0"
 
-module asm(target_features="+foo", target_cpu="foo_cpu")
+module asm(target_features: "+foo", target_cpu: "foo_cpu")
     "asm line 1"
     "asm line 2"
 
-module asm(target_features="+bar")
+module asm(target_features: "+bar")
     "asm line 3"
 
-module asm(target_cpu="bar_cpu")
+module asm(target_cpu: "bar_cpu")
     "asm line 4"
 
 module asm
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
index 3e9483cbf4461..ab16acab48688 100644
--- a/llvm/test/CodeGen/RISCV/module-asm-features.ll
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -10,7 +10,7 @@
 ; CHECK: fld ft0, 0(sp)
 ; EXTRA-FEATURES: .option pop
 
-module asm(target_features="+d")
+module asm(target_features: "+d")
     ".globl func"
     "func:"
     "fld f0, 0(sp)"
diff --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
index c92343b457ff7..73320185e778a 100644
--- a/llvm/test/LTO/RISCV/module-asm.ll
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -11,7 +11,7 @@
 target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
 target triple = "riscv64-unknown-linux-gnu"
 
-module asm(target_features="+d")
+module asm(target_features: "+d")
     ".globl func"
     "func:"
     "fld f0, 0(sp)"
diff --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
index 01c55a875af4f..b097c7067ccc1 100644
--- a/llvm/test/Linker/Inputs/module-asm.ll
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -1,2 +1,2 @@
-module asm(target_features="+bar")
+module asm(target_features: "+bar")
     "asm 2"
diff --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
index 9f43c7a22514d..6c600981d72a5 100644
--- a/llvm/test/Linker/module-asm.ll
+++ b/llvm/test/Linker/module-asm.ll
@@ -1,9 +1,9 @@
 ; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
 
-; CHECK: module asm(target_features="+foo")
+; CHECK: module asm(target_features: "+foo")
 ; CHECK-NEXT: "asm 1"
-; CHECK: module asm(target_features="+bar")
+; CHECK: module asm(target_features: "+bar")
 ; CHECK-NEXT: "asm 2"
 
-module asm(target_features="+foo")
+module asm(target_features: "+foo")
     "asm 1"

>From f5d52683f76b5a1ca24a60c34ad6b1d420ccef27 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Thu, 25 Jun 2026 14:56:53 +0200
Subject: [PATCH 5/6] Add LLVM_ABI annotations

---
 llvm/include/llvm/IR/Module.h             | 4 ++--
 llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index a82ac363dee65..63d1013896b47 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -175,10 +175,10 @@ class LLVM_ABI Module {
 
     /// Set a property using a string name.
     /// Returns whether the property name was valid.
-    bool set(StringRef Name, std::string Value);
+    LLVM_ABI bool set(StringRef Name, std::string Value);
 
     /// Get a list of set properties as pairs of key and value.
-    SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
 
     bool operator==(const GlobalAsmProperties &Other) const {
       return TargetFeatures == Other.TargetFeatures &&
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index c308f4412d1e3..e37ea27364e6e 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -537,7 +537,7 @@ bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
 
     auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
                                                : RISCVOptionArchArgType::Minus;
-    StringRef ExtName = Feature.Key;
+    StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
   }

>From 34afe76619fe10f44b7f14ad483e017d5d89f9ed Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Fri, 3 Jul 2026 09:05:58 +0200
Subject: [PATCH 6/6] Fix test failure on AArch64

The triple might not set any features by default, in which case
just `module asm` gets printed. Only test for that for the default
target, and add separate tests for RISCV with different features
instead.
---
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index cc1f4b585f209..8a212a0b61905 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,7 +1,11 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
+// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
 
-// CHECK: module asm(target_features: "{{.*}}")
+// RISCV64: module asm(target_features: "+64bit,+i")
+// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
+// DEFAULT: module asm
 // CHECK-NEXT: "foo1"
 __asm__ ("foo1");
 // CHECK-NEXT: "foo2"


From llvmlistbot at llvm.org  Fri Jul  3 00:13:17 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 00:13:17 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47610d.d5797644.146403.b517@mx.google.com>


https://github.com/nikic auto_merge_enabled https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 00:13:54 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:13:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a476132.bd93e731.b068b.36d9@mx.google.com>


Saieiei wrote:

LGTM. The change is well-gated and the test covers the key cases.

https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Fri Jul  3 00:15:59 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Fri, 03 Jul 2026 00:15:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a4761af.f8b79514.3b441.aa69@mx.google.com>


https://github.com/Saieiei commented:

LGTM. The change is well-gated and the test covers the key cases.

https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Fri Jul  3 00:17:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:17:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU][NVVM] Add FP8 (e4m3/e5m2) support to
 dense mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a476206.6c8668f2.74a35.98da@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207307

>From 0b477bb6afe7f9cb470a5b07f6cc94c86f9bc7b3 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 +++++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 24 +++++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {


From llvmlistbot at llvm.org  Fri Jul  3 00:21:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:21:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4762de.58bdd12c.21857.a371@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 00:26:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:26:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a476420.eea6ab6e.1256b6.4078@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 00:28:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:28:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
Message-ID: <llvm.org/llvm/llvm-project/pull/207336@github.com>


https://github.com/qyingwu created https://github.com/llvm/llvm-project/pull/207336

Fixes #197175.

`MLIRConfig.cmake` does not currently propagate `MLIR_LINK_MLIR_DYLIB`
to out-of-tree MLIR users. Standalone projects using
`find_package(MLIR CONFIG)` and `include(AddMLIR)` therefore do not
observe the value used by `mlir_target_link_libraries(...)` when MLIR
was configured with `-DMLIR_LINK_MLIR_DYLIB=ON`.

This mirrors LLVM's existing `LLVM_LINK_LLVM_DYLIB` export pattern by:
- defining `MLIR_CONFIG_LINK_MLIR_DYLIB` in `mlir/cmake/modules/CMakeLists.txt`
- emitting it from `mlir/cmake/modules/MLIRConfig.cmake.in`

Verification:
- configured with `-DMLIR_LINK_MLIR_DYLIB=ON` and confirmed both generated
  build-tree and install-tree `MLIRConfig.cmake` files contain
  `set(MLIR_LINK_MLIR_DYLIB ON)`
- configured with `-DMLIR_LINK_MLIR_DYLIB=OFF` and confirmed neither generated
  config contains `MLIR_LINK_MLIR_DYLIB`

>From 77325b2b497fd5f30c622b48bdb812eb140853e3 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 3 Jul 2026 00:20:18 -0700
Subject: [PATCH] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake

---
 mlir/cmake/modules/CMakeLists.txt      | 5 +++++
 mlir/cmake/modules/MLIRConfig.cmake.in | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..8b49e96bf2531 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -60,6 +60,11 @@ set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
 
+if (MLIR_LINK_MLIR_DYLIB)
+  set(MLIR_CONFIG_LINK_MLIR_DYLIB
+      "set(MLIR_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})")
+endif()
+
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
   ${mlir_cmake_builddir}/MLIRConfig.cmake
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..ae426b43382c1 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -17,6 +17,8 @@ set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
 
+ at MLIR_CONFIG_LINK_MLIR_DYLIB@
+
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_CONVERSION_LIBS "@MLIR_CONVERSION_LIBS@")


From llvmlistbot at llvm.org  Fri Jul  3 00:28:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:28:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4764b1.576b3af0.2cb9b7.bbb1@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @qyingwu :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Fri Jul  3 00:29:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:29:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4764db.bd93e731.b068b.98af@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: qyingwu

<details>
<summary>Changes</summary>

Fixes #<!-- -->197175.

`MLIRConfig.cmake` does not currently propagate `MLIR_LINK_MLIR_DYLIB`
to out-of-tree MLIR users. Standalone projects using
`find_package(MLIR CONFIG)` and `include(AddMLIR)` therefore do not
observe the value used by `mlir_target_link_libraries(...)` when MLIR
was configured with `-DMLIR_LINK_MLIR_DYLIB=ON`.

This mirrors LLVM's existing `LLVM_LINK_LLVM_DYLIB` export pattern by:
- defining `MLIR_CONFIG_LINK_MLIR_DYLIB` in `mlir/cmake/modules/CMakeLists.txt`
- emitting it from `mlir/cmake/modules/MLIRConfig.cmake.in`

Verification:
- configured with `-DMLIR_LINK_MLIR_DYLIB=ON` and confirmed both generated
  build-tree and install-tree `MLIRConfig.cmake` files contain
  `set(MLIR_LINK_MLIR_DYLIB ON)`
- configured with `-DMLIR_LINK_MLIR_DYLIB=OFF` and confirmed neither generated
  config contains `MLIR_LINK_MLIR_DYLIB`

---
Full diff: https://github.com/llvm/llvm-project/pull/207336.diff


2 Files Affected:

- (modified) mlir/cmake/modules/CMakeLists.txt (+5) 
- (modified) mlir/cmake/modules/MLIRConfig.cmake.in (+2) 


``````````diff
diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..8b49e96bf2531 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -60,6 +60,11 @@ set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
 
+if (MLIR_LINK_MLIR_DYLIB)
+  set(MLIR_CONFIG_LINK_MLIR_DYLIB
+      "set(MLIR_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})")
+endif()
+
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
   ${mlir_cmake_builddir}/MLIRConfig.cmake
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..ae426b43382c1 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -17,6 +17,8 @@ set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
 
+ at MLIR_CONFIG_LINK_MLIR_DYLIB@
+
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_CONVERSION_LIBS "@MLIR_CONVERSION_LIBS@")

``````````

</details>


https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Fri Jul  3 00:31:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:31:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a47655c.6c8668f2.74a35.3dcd@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Fri Jul  3 00:35:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:35:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a47663b.510f576d.185953.78be@mx.google.com>


qyingwu wrote:

I have read the linked LLVM contribution policies, including the LLVM AI Tool Use Policy.

I used ChatGPT/Codex to help understand the issue, identify the analogous LLVM CMake pattern, and draft the PR description. I have noted this AI tool usage in the PR description and manually reviewed the final patch and verification.

https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Fri Jul  3 00:36:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:36:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a47669a.ff037943.3bf0e2.dc36@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/207336

>From ae563973b270187b3be4ca86d7f070f7a9aac711 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 3 Jul 2026 00:20:18 -0700
Subject: [PATCH] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake

---
 mlir/cmake/modules/CMakeLists.txt      | 5 +++++
 mlir/cmake/modules/MLIRConfig.cmake.in | 2 ++
 2 files changed, 7 insertions(+)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..8b49e96bf2531 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -60,6 +60,11 @@ set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
 
+if (MLIR_LINK_MLIR_DYLIB)
+  set(MLIR_CONFIG_LINK_MLIR_DYLIB
+      "set(MLIR_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})")
+endif()
+
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
   ${mlir_cmake_builddir}/MLIRConfig.cmake
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..ae426b43382c1 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -17,6 +17,8 @@ set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
 
+ at MLIR_CONFIG_LINK_MLIR_DYLIB@
+
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_CONVERSION_LIBS "@MLIR_CONVERSION_LIBS@")


From llvmlistbot at llvm.org  Fri Jul  3 00:46:31 2026
From: llvmlistbot at llvm.org (Rahul Joshi)
Date: Fri, 03 Jul 2026 00:46:31 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a4768d7.4a342cbc.134cb.a277@mx.google.com>


https://github.com/jurahul updated https://github.com/llvm/llvm-project/pull/207300

>From a8fa26814dfc0636e81a0f8bda0ad98e0f293eb7 Mon Sep 17 00:00:00 2001
From: Rahul Joshi <rjoshi at nvidia.com>
Date: Thu, 2 Jul 2026 17:47:20 -0700
Subject: [PATCH] [NFC][LLVM] Define range attribute for `llvm.prefetch` args

Define valid range for `llvm.prefetch` immarg values and eliminate
C++ verification code.

Additionally, change the range verification failure message to
include the argument number and simplify formatting using `formatv`.
---
 llvm/include/llvm/IR/Intrinsics.td            |  6 ++--
 llvm/lib/IR/Verifier.cpp                      | 14 ++-------
 llvm/test/Assembler/invalid-immarg5.ll        |  2 +-
 .../AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll  | 30 +++++++++----------
 .../Verifier/NVPTX/fence-proxy.tensormap.ll   |  8 ++---
 llvm/test/Verifier/NVPTX/setmaxnreg.ll        |  4 +--
 llvm/test/Verifier/intrinsic-immarg.ll        | 12 ++++++++
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir |  2 +-
 8 files changed, 42 insertions(+), 36 deletions(-)

diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),
diff --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }
diff --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
   ret <2 x half> %result
 }
 
-; CHECK: immarg value 15 out of range [0, 4)
+; CHECK: immarg value 15 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
@@ -87,14 +87,14 @@ define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f32 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 8 out of range [0, 4)
+; CHECK: immarg value 8 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
@@ -105,14 +105,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f16 - dest_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 12 out of range [0, 4)
+; CHECK: immarg value 12 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
@@ -123,14 +123,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %ol
 ; llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 9 out of range [0, 4)
+; CHECK: immarg value 9 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_way_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)
diff --git a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
index 4fa7a7ae71001..652990a700a6e 100644
--- a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
+++ b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
@@ -1,16 +1,16 @@
 ; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cluster(ptr addrspace(0) %addr, i32 129);
 
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.gpu(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.sys(ptr addrspace(0) %addr, i32 129);
 
   ret void
diff --git a/llvm/test/Verifier/NVPTX/setmaxnreg.ll b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
index 1afebeab4742c..79714a0521c36 100644
--- a/llvm/test/Verifier/NVPTX/setmaxnreg.ll
+++ b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
@@ -7,10 +7,10 @@ define void @test_set_maxn_reg() {
   ; CHECK: reg_count argument to nvvm.setmaxnreg must be in multiples of 8
   call void @llvm.nvvm.setmaxnreg.inc.sync.aligned.u32(i32 95)
 
-  ; CHECK: immarg value 16 out of range [24, 257)
+  ; CHECK: immarg value 16 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 16)
 
-  ; CHECK: immarg value 264 out of range [24, 257)
+  ; CHECK: immarg value 264 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 264)
   ret void
 }
diff --git a/llvm/test/Verifier/intrinsic-immarg.ll b/llvm/test/Verifier/intrinsic-immarg.ll
index c95c7214fde71..39ba3cb480b4d 100644
--- a/llvm/test/Verifier/intrinsic-immarg.ll
+++ b/llvm/test/Verifier/intrinsic-immarg.ll
@@ -154,6 +154,18 @@ define void @test_prefetch(ptr %ptr, i32 %arg0, i32 %arg1) {
   ; CHECK-NEXT:  i32 %arg1
   call void @llvm.prefetch(ptr %ptr, i32 %arg0, i32 0, i32 0)
   call void @llvm.prefetch(ptr %ptr, i32 0, i32 %arg1, i32 0)
+
+  ; CHECK: immarg value 2 for arg 1 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 2, i32 0, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 2, i32 0, i32 0)
+
+  ; CHECK: immarg value 8 for arg 2 out of range [0,4)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 8, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 8, i32 0)
+
+  ; CHECK: immarg value 4 for arg 3 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 0, i32 4)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 0, i32 4)
   ret void
 }
 
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index ea393dd445eda..5266470891b4d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1420,7 +1420,7 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
 // CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
-// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg, i32 immarg, i32 immarg)
+// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0


From llvmlistbot at llvm.org  Fri Jul  3 00:57:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:57:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
Message-ID: <llvm.org/llvm/llvm-project/pull/207342@github.com>


https://github.com/weimin023 created https://github.com/llvm/llvm-project/pull/207342

This PR is stacked on #207307 

## Motivation
I'm adding NVIDIA FP8 mma.sync support in IREE for sm_89/sm_120, and IREE's codegen lowers through nvgpu.mma.sync on this path.

## Description
`nvgpu.mma.sync` (warp-level, dense) doesn't support FP8 (e4m3/e5m2) operands. This builds on the NVVM-level FP8 support added in #207307 to add the corresponding support on the NVGPU side, plus a lit test covering the lowering.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file
`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type<e4m3>,
      multiplicandBPtxType = #nvvm.mma_type<e4m3>,
      shape = #nvvm.shape<m = 16, n = 8, k = 32>}
     : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).

Co-authored with Claude Sonnet 5

>From 0b477bb6afe7f9cb470a5b07f6cc94c86f9bc7b3 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH 1/2] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 +++++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 24 +++++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {

>From d2ea4e55a8937cb30d426670a59ce99e6bec3c1f Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:43:32 +0000
Subject: [PATCH 2/2] [NVGPU] Add FP8 (e4m3/e5m2) support to nvgpu.mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..198fd958693fb 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Fri Jul  3 00:58:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:58:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a476b8b.eb24c6ea.1f2e26.85da@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @weimin023 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Fri Jul  3 00:58:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:58:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a476b9e.eea6ab6e.1256b6.a421@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Fri Jul  3 00:58:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:58:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a476baf.dd2b4bd3.35cd5e.7e8c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: WMC (weimin023)

<details>
<summary>Changes</summary>

This PR is stacked on #<!-- -->207307 

## Motivation
I'm adding NVIDIA FP8 mma.sync support in IREE for sm_89/sm_120, and IREE's codegen lowers through `nvgpu.mma.sync` on this path.

## Description
`nvgpu.mma.sync` (warp-level, dense) doesn't support FP8 (e4m3/e5m2) operands. This builds on the NVVM-level FP8 support added in #<!-- -->207307 to add the corresponding support on the NVGPU side, plus a lit test covering the lowering.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file
`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      multiplicandBPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      shape = #nvvm.shape&lt;m = 16, n = 8, k = 32&gt;}
     : (i32, i32, f32) -&gt; !llvm.struct&lt;(f32, f32, f32, f32)&gt;
```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).

Co-authored with Claude Sonnet 5

---
Full diff: https://github.com/llvm/llvm-project/pull/207342.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+5-1) 
- (modified) mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp (+4) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+9) 
- (modified) mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp (+4-3) 
- (added) mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir (+12) 
- (modified) mlir/test/Dialect/LLVMIR/nvvm.mlir (+24) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..198fd958693fb 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {

``````````

</details>


https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Fri Jul  3 00:58:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:58:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a476bb0.e9d484b9.11ad58.29bd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-nvgpu

Author: WMC (weimin023)

<details>
<summary>Changes</summary>

This PR is stacked on #<!-- -->207307 

## Motivation
I'm adding NVIDIA FP8 mma.sync support in IREE for sm_89/sm_120, and IREE's codegen lowers through `nvgpu.mma.sync` on this path.

## Description
`nvgpu.mma.sync` (warp-level, dense) doesn't support FP8 (e4m3/e5m2) operands. This builds on the NVVM-level FP8 support added in #<!-- -->207307 to add the corresponding support on the NVGPU side, plus a lit test covering the lowering.

## Verification
`$ mlir-opt mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir -convert-nvgpu-to-nvvm -split-input-file
`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      multiplicandBPtxType = #nvvm.mma_type&lt;e4m3&gt;,
      shape = #nvvm.shape&lt;m = 16, n = 8, k = 32&gt;}
     : (i32, i32, f32) -&gt; !llvm.struct&lt;(f32, f32, f32, f32)&gt;
```

Existing FP16/BF16 tests in `nvgpu-to-nvvm.mlir` still pass (no regression).

Co-authored with Claude Sonnet 5

---
Full diff: https://github.com/llvm/llvm-project/pull/207342.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+5-1) 
- (modified) mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp (+4) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+9) 
- (modified) mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp (+4-3) 
- (added) mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir (+12) 
- (modified) mlir/test/Dialect/LLVMIR/nvvm.mlir (+24) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..198fd958693fb 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {

``````````

</details>


https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Fri Jul  3 00:59:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 00:59:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 0341dd5 - [IR] Explicitly specify target
 feature for module asm (#204548)
Message-ID: <6a476bd4.bb839515.23e6d3.abe1@mx.google.com>


Author: Nikita Popov
Date: 2026-07-03T07:59:08Z
New Revision: 0341dd51303fc13cafdd21a33e2985cbf603c66b

URL: https://github.com/llvm/llvm-project/commit/0341dd51303fc13cafdd21a33e2985cbf603c66b
DIFF: https://github.com/llvm/llvm-project/commit/0341dd51303fc13cafdd21a33e2985cbf603c66b.diff

LOG: [IR] Explicitly specify target feature for module asm (#204548)

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can now be
specified as follows:

    module asm(target_features: "+foo", target_cpu: "bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO scenarios
we don't know what target features to use for parsing the module-level
inline assembly. Now they can be faithfully preserved, even when merging
inline assembly from different modules with different features.

If target_features and target_cpu are empty, we fall back to the old
behavior (which, in the backend, would be to use the default subtarget
of the TargetMachine).

Fixes https://github.com/llvm/llvm-project/issues/61991.
Fixes https://github.com/llvm/llvm-project/issues/67698.

Added: 
    llvm/test/Assembler/module-asm-invalid.ll
    llvm/test/Bitcode/module-asm.ll
    llvm/test/CodeGen/RISCV/module-asm-features.ll
    llvm/test/LTO/RISCV/module-asm.ll
    llvm/test/Linker/Inputs/module-asm.ll
    llvm/test/Linker/module-asm.ll

Modified: 
    clang/lib/CodeGen/CGObjCMac.cpp
    clang/lib/CodeGen/CodeGenModule.cpp
    clang/test/CodeGen/2006-01-23-FileScopeAsm.c
    clang/test/CodeGen/asm.c
    clang/test/CodeGen/asm_incbin.c
    clang/test/CodeGen/cfi-salt.c
    clang/test/CodeGen/kcfi.c
    clang/test/CodeGenCUDA/filter-decl.cu
    clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
    clang/test/CodeGenObjC/category-class.m
    clang/test/CodeGenObjC/objc-runtime-name.m
    clang/test/Frontend/ast-codegen.c
    clang/test/Modules/codegen.test
    llvm/docs/LangRef.rst
    llvm/docs/ReleaseNotes.md
    llvm/include/llvm/Bitcode/LLVMBitCodes.h
    llvm/include/llvm/CodeGen/AsmPrinter.h
    llvm/include/llvm/IR/Module.h
    llvm/lib/AsmParser/LLParser.cpp
    llvm/lib/Bitcode/Reader/BitcodeReader.cpp
    llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
    llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
    llvm/lib/IR/AsmWriter.cpp
    llvm/lib/IR/Core.cpp
    llvm/lib/IR/Module.cpp
    llvm/lib/LTO/LTO.cpp
    llvm/lib/Linker/IRMover.cpp
    llvm/lib/Object/ModuleSymbolTable.cpp
    llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
    llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
    llvm/lib/Transforms/IPO/ExtractGV.cpp
    llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
    llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
    llvm/lib/Transforms/Utils/SplitModule.cpp
    llvm/test/Bitcode/compatibility-3.6.ll
    llvm/test/Bitcode/compatibility-3.7.ll
    llvm/test/Bitcode/compatibility-3.8.ll
    llvm/test/Bitcode/compatibility-3.9.ll
    llvm/test/Bitcode/compatibility-4.0.ll
    llvm/test/Bitcode/compatibility-5.0.ll
    llvm/test/Bitcode/compatibility-6.0.ll
    llvm/test/Bitcode/compatibility.ll
    llvm/test/Bitcode/highLevelStructure.3.2.ll
    llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
    llvm/test/LTO/X86/inline-asm-lto-discard.ll
    llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
    llvm/test/ThinLTO/X86/import-symver.ll
    llvm/test/Transforms/LowerTypeTests/export-symver.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
    llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
    mlir/lib/Target/LLVMIR/ModuleImport.cpp
    mlir/test/Target/LLVMIR/module-asm.mlir

Removed: 
    


################################################################################
diff  --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index 69c5e88f3c768..e518b875f44bb 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,10 +6610,6 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
-    Asm += CGM.getModule().getModuleInlineAsm();
-    if (!Asm.empty() && Asm.back() != '\n')
-      Asm += '\n';
-
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6624,7 +6620,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().setModuleInlineAsm(OS.str());
+    CGM.getModule().appendModuleInlineAsm(OS.str());
   }
 }
 

diff  --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 813cd67e4dd51..e155fdd752d7f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,7 +8057,13 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-    getModule().appendModuleInlineAsm(AD->getAsmString());
+
+    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
+    llvm::Module::GlobalAsmProperties Props;
+    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
+    Props.TargetCPU = TargetOpts.CPU;
+    getModule().appendModuleInlineAsm(
+        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
     break;
   }
 

diff  --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 877da4fc669e3..8a212a0b61905 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,13 +1,18 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
+// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
 
-// CHECK: module asm "foo1"
+// RISCV64: module asm(target_features: "+64bit,+i")
+// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
+// DEFAULT: module asm
+// CHECK-NEXT: "foo1"
 __asm__ ("foo1");
-// CHECK: module asm "foo2"
+// CHECK-NEXT: "foo2"
 __asm__ ("foo2");
-// CHECK: module asm "foo3"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo3");
-// CHECK: module asm "foo4"
+// CHECK-NEXT: "foo4"
 __asm__ ("foo4");
-// CHECK: module asm "foo5"
+// CHECK-NEXT: "foo5"
 __asm__ ("foo5");

diff  --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index d7465b22fbbf6..ea69c7a17d1cb 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,9 +2,10 @@
 
 // PR10415:
 //
-// CHECK:      module asm "foo1"
-// CHECK-NEXT: module asm "foo2"
-// CHECK-NEXT: module asm "foo3"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo1"
+// CHECK-NEXT: "foo2"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");

diff  --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index 6234eea1d6738..bbc5d3af55ea2 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,4 +6,5 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm ".incbin \22foo.h\22"
+// CHECK: module asm
+// CHECK-NEXT: ".incbin \22foo.h\22"

diff  --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 2a90dc31d939b..18156dc014280 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,13 +26,14 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
+// CHECK-NOT: ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;

diff  --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index da32859dd3d88..f08cf7635fb5a 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,10 +6,11 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2

diff  --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index 02dacd0ad8ef4..f87d62a56631e 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,8 +5,9 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm "file scope asm is host only"
-// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
+// CHECK-HOST: module asm
+// CHECK-HOST-NEXT: "file scope asm is host only"
+// CHECK-DEVICE-NOT: "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global

diff  --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 175a3b7bc588c..2b4abee5dc05f 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,8 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm "nop; nop; nop; nop"
+    // CHECK: module asm(target_features: "{{.*}}")
+    // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
 int func() {return 0;};

diff  --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index 92fd36cbe4066..febe94ce5b9f5 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,9 +1,10 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
-// CHECK: module asm "\09.objc_category_name_A_foo=0"
-// CHECK: module asm "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm
+// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
+// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
+// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end

diff  --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index c838b8c2c6f1a..a7f59caced390 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,9 +3,10 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm {{.*}}objc_class_name_XYZ=0
-// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
-// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm
+// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
+// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
+// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"

diff  --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index 2e2e3d360dd17..c26b0a1908379 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,8 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm "foo"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4

diff  --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 0af630a754805..5cc5b3c1cda2a 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,7 +18,8 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm "narf"
+BOTH: module asm
+BOTH-NEXT: "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any

diff  --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5ce6019660e0b..2c8276d04b668 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,19 +3357,34 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks. These blocks are internally
-concatenated by LLVM and treated as a single unit, but may be separated
-in the ``.ll`` file if desired. The syntax is very simple:
+to the GCC "file scope inline asm" blocks:
 
 .. code-block:: llvm
 
-    module asm "inline asm code goes here"
-    module asm "more can go here"
+    module asm
+        "inline asm code goes here"
+        "more can go here"
 
-The strings can contain any character by escaping non-printable
+The sequence of string literals is internally concatenated using newlines as
+separators. The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
+Additionally, module-level inline assembly can specify an optional list of
+properties:
+
+.. code-block:: llvm
+
+    module asm(target_features: "+foo", target_cpu: "bar")
+        "inline asm code goes here"
+        "more can go here"
+
+Currently, the only supported properties are ``target_features`` and
+``target_cpu``, with the same meaning as the ``"target-features"`` and
+``"target-cpu"`` function attributes.
+
+Consecutive blocks with identical properties will be concatenated into one.
+
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 

diff  --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index e42a959ab3101..9b2208b34da82 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -102,6 +102,7 @@ Makes programs 10x faster by doing Special New Thing.
   outlining. Add the `noinline` and `nooutline` attributes as well in cases
   where inlining and outlining should additionally be disabled.
 
+<<<<<<< HEAD
 * Added support for ``callgraph`` metadata. The `!callgraph` metadata
   associates a function definition with its type identifier and is used for call
   graph section generation. See the [callgraph Metadata](https://llvm.org/docs/LangRef.html#callgraph-metadata)
@@ -111,6 +112,10 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
+* Module-level inline assembly now accepts optional `target_features` and
+  `target_cpu` properties. This resolves errors during LTO on some
+  architectures.
+
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to

diff  --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index af4cac996a78c..1761e7a542a73 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,6 +120,8 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
+
+  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.

diff  --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 78e64440657cc..335bf9de9a071 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,6 +930,19 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
+  /// Emit necessary directives to allow use of instructions that are permitted
+  /// by target features enabled by STI, but are not permitted by target
+  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
+  /// Returns whether anything was emitted.
+  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+    return false;
+  }
+
+  /// Emit necessary directives to restore target feature state.
+  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
+  /// call.
+  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
+
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,

diff  --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 4f7e33969f16f..63d1013896b47 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,6 +169,45 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
+  struct GlobalAsmProperties {
+    std::string TargetFeatures;
+    std::string TargetCPU;
+
+    /// Set a property using a string name.
+    /// Returns whether the property name was valid.
+    LLVM_ABI bool set(StringRef Name, std::string Value);
+
+    /// Get a list of set properties as pairs of key and value.
+    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+
+    bool operator==(const GlobalAsmProperties &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+
+    bool operator!=(const GlobalAsmProperties &Other) const {
+      return !(*this == Other);
+    }
+  };
+
+  struct GlobalAsmFragment {
+    std::string Asm;
+    GlobalAsmProperties Props;
+
+    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
+    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
+        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
+      if (!Asm.empty() && Asm.back() != '\n')
+        Asm += '\n';
+    }
+
+    bool empty() const { return Asm.empty(); }
+
+    bool hasSameProperties(const GlobalAsmFragment &Other) const {
+      return Props == Other.Props;
+    }
+  };
+
 /// @}
 /// @name Member Variables
 /// @{
@@ -180,7 +219,8 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
+  /// Inline Asm at the global scope.
+  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -287,8 +327,17 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  /// @returns a string containing the module-scope inline assembly blocks.
-  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
+  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
+    return GlobalScopeAsm;
+  }
+
+  /// Get any module-scope inline assembly blocks.
+  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
+    return GlobalScopeAsm;
+  }
+
+  /// Return whether there is any module-scope inline assembly.
+  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -325,20 +374,45 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
+  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
+
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm = std::string(Asm);
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
+    GlobalScopeAsm.clear();
+    appendModuleInlineAsm(std::move(Fragment));
+  }
+
+  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
+    GlobalScopeAsm.clear();
+    append_range(GlobalScopeAsm, Fragments);
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm += Asm;
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.back().Asm += Fragment.Asm;
+    } else {
+      GlobalScopeAsm.emplace_back(std::move(Fragment));
+    }
+  }
+
+  /// Prepend to the module-scope inline assembly blocks.
+  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
+    } else {
+      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
+    }
   }
 
 /// @}

diff  --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 75e6add0ec76f..0a5b4c6c91879 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,16 +641,49 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
+///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
+///                          'property_name2:' STRINGCONSTANT ')'
+///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
-      parseStringConstant(AsmStr))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  M->appendModuleInlineAsm(AsmStr);
+  Module::GlobalAsmProperties Props;
+  if (EatIfPresent(lltok::lparen)) {
+    while (true) {
+      std::string Key, Value;
+      SMLoc Loc = Lex.getLoc();
+      if (Lex.getKind() != lltok::LabelStr)
+        return error(Loc, "expected property name followed by ':'");
+
+      Key = Lex.getStrVal();
+      Lex.Lex();
+
+      if (parseStringConstant(Value))
+        return true;
+
+      if (!Props.set(Key, Value))
+        return error(Loc, "unknown property name");
+
+      if (EatIfPresent(lltok::rparen))
+        break;
+      if (parseToken(lltok::comma, "expected ',' or ')'"))
+        return true;
+    }
+  }
+
+  do {
+    std::string AsmStrPart;
+    if (parseStringConstant(AsmStrPart))
+      return true;
+    AsmStr += AsmStrPart + "\n";
+  } while (Lex.getKind() == lltok::StringConstant);
+
+  M->appendModuleInlineAsm({AsmStr, Props});
   return false;
 }
 

diff  --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 2bd251efd05ce..06e9aea4fd3ac 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,6 +4580,9 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
+  // Apply to the following module asm.
+  Module::GlobalAsmProperties Props;
+
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4788,11 +4791,23 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
+    case bitc::MODULE_CODE_ASM_PROPERTY: {
+      std::string Str;
+      if (convertToString(Record, 0, Str))
+        return error("Invalid module asm record");
+      size_t SepPos = Str.find('\0');
+      if (SepPos == std::string::npos)
+        return error("Invalid module asm record");
+      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
+        return error("Unknown module asm property");
+      break;
+    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->setModuleInlineAsm(S);
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
+      Props = {};
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]

diff  --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 4aa203b0d12d1..493cbec7506f2 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,9 +1551,19 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-  if (!M.getModuleInlineAsm().empty())
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
-                      0 /*TODO*/);
+
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    SmallVector<std::pair<StringRef, StringRef>> Props =
+        Frag.Props.getAsStrings();
+    for (auto [Key, Value] : Props) {
+      SmallVector<unsigned, 64> Record;
+      Record.append(Key.begin(), Key.end());
+      Record.push_back(0);
+      Record.append(Value.begin(), Value.end());
+      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
+    }
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
+  }
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.

diff  --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index abb20a7919fa1..8248a05f1a35f 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,6 +110,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -127,7 +128,6 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,13 +610,29 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    emitInlineAsm(
-        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
-        TM.Options.MCOptions, nullptr,
-        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
+        std::unique_ptr<MCSubtargetInfo> AsmSTI(
+            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
+                                                 Frag.Props.TargetCPU,
+                                                 Frag.Props.TargetFeatures));
+        bool DidPush = emitTargetFeaturePush(*AsmSTI);
+        emitInlineAsm(
+            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+        emitTargetFeaturePop(*AsmSTI, DidPush);
+      } else {
+        // If the module asm does not explicitly specify target features,
+        // fall back to default subtargetinfo.
+        emitInlineAsm(
+            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
+            nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      }
+    }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }

diff  --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 665d7122af2c7..02d6bcd90add0 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,21 +3126,38 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (!M->getModuleInlineAsm().empty()) {
+  if (M->hasModuleInlineAsm()) {
     Out << '\n';
 
-    // Split the string into lines, to make it easier to read the .ll file.
-    StringRef Asm = M->getModuleInlineAsm();
-    do {
-      StringRef Front;
-      std::tie(Front, Asm) = Asm.split('\n');
-
-      // We found a newline, print the portion of the asm string from the
-      // last newline up to this newline.
-      Out << "module asm \"";
-      printEscapedString(Front, Out);
-      Out << "\"\n";
-    } while (!Asm.empty());
+    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
+      Out << "module asm";
+      SmallVector<std::pair<StringRef, StringRef>> Props =
+          Frag.Props.getAsStrings();
+      if (!Props.empty()) {
+        ListSeparator LS;
+        Out << "(";
+        for (auto [Key, Value] : Props) {
+          Out << LS;
+          Out << Key << ": \"";
+          printEscapedString(Value, Out);
+          Out << "\"";
+        }
+        Out << ")";
+      }
+      Out << "\n";
+      // Split the string into lines, to make it easier to read the .ll file.
+      StringRef Asm = Frag.Asm;
+      do {
+        StringRef Front;
+        std::tie(Front, Asm) = Asm.split('\n');
+
+        // We found a newline, print the portion of the asm string from the
+        // last newline up to this newline.
+        Out << "    \"";
+        printEscapedString(Front, Out);
+        Out << "\"\n";
+      } while (!Asm.empty());
+    }
   }
 
   printTypeIdentities();

diff  --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index 6c6c3b5c84ede..a7abd3eed31c3 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,7 +510,18 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  auto &Str = unwrap(M)->getModuleInlineAsm();
+  Module *Mod = unwrap(M);
+  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
+  if (Frags.empty()) {
+    *Len = 0;
+    return nullptr;
+  }
+
+  if (Frags.size() != 1)
+    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
+                          "more than one module inline assembly fragment");
+
+  auto &Str = Frags.begin()->Asm;
   *Len = Str.length();
   return Str.c_str();
 }

diff  --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 40ef7e354da72..34e44e26d5db5 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,3 +977,23 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
+
+bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
+  if (Name == "target_features")
+    TargetFeatures = std::move(Value);
+  else if (Name == "target_cpu")
+    TargetCPU = std::move(Value);
+  else
+    return false;
+  return true;
+}
+
+SmallVector<std::pair<StringRef, StringRef>>
+Module::GlobalAsmProperties::getAsStrings() const {
+  SmallVector<std::pair<StringRef, StringRef>> Props;
+  if (!TargetFeatures.empty())
+    Props.emplace_back("target_features", TargetFeatures);
+  if (!TargetCPU.empty())
+    Props.emplace_back("target_cpu", TargetCPU);
+  return Props;
+}

diff  --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 6a754f95092bb..b672f4e5fac14 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
+    M.prependModuleInlineAsm(NewIA);
   }
 
   assert(MsymI == MsymE);

diff  --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde1cec..3b72b412d0b2e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,11 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
+      Module::GlobalAsmFragment NewFrag(Frag);
+      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
+      DstM.appendModuleInlineAsm(std::move(NewFrag));
+    }
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1573,7 +1576,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(S);
+        DstM.appendModuleInlineAsm(std::string(S));
       }
     });
   }

diff  --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 1da5fa9c10a0b..44f717ec12323 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,8 +72,7 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  StringRef InlineAsm = M.getModuleInlineAsm();
-  if (InlineAsm.empty())
+  if (!M.hasModuleInlineAsm())
     return;
 
   std::string Err;
@@ -90,52 +89,55 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
-  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
-  if (!STI)
-    return;
-
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  std::unique_ptr<MemoryBuffer> Buffer(
-      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
-  SourceMgr SrcMgr;
-  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-  std::unique_ptr<MCObjectFileInfo> MOFI(
-      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-  MCCtx.setObjectFileInfo(MOFI.get());
-  RecordStreamer Streamer(MCCtx, M);
-  T->createNullTargetStreamer(Streamer);
-
-  std::unique_ptr<MCAsmParser> Parser(
-      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-  std::unique_ptr<MCTargetAsmParser> TAP(
-      T->createMCAsmParser(*STI, *Parser, *MCII));
-  if (!TAP)
-    return;
-
-  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                 const SourceMgr &SrcMgr,
-                                 std::vector<const MDNode *> &LocInfos) {
-    M.getContext().diagnose(
-        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-  });
-
-  // Module-level inline asm is assumed to use At&t syntax (see
-  // AsmPrinter::doInitialization()).
-  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-  Parser->setSymbolScanningMode(true);
-
-  Parser->setTargetParser(*TAP);
-  if (Parser->Run(false))
-    return;
-
-  Init(Streamer);
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
+        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
+    if (!STI)
+      return;
+
+    std::unique_ptr<MemoryBuffer> Buffer(
+        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
+    SourceMgr SrcMgr;
+    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+    std::unique_ptr<MCObjectFileInfo> MOFI(
+        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+    MCCtx.setObjectFileInfo(MOFI.get());
+    RecordStreamer Streamer(MCCtx, M);
+    T->createNullTargetStreamer(Streamer);
+
+    std::unique_ptr<MCAsmParser> Parser(
+        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+    std::unique_ptr<MCTargetAsmParser> TAP(
+        T->createMCAsmParser(*STI, *Parser, *MCII));
+    if (!TAP)
+      return;
+
+    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                   const SourceMgr &SrcMgr,
+                                   std::vector<const MDNode *> &LocInfos) {
+      M.getContext().diagnose(
+          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+    });
+
+    // Module-level inline asm is assumed to use At&t syntax (see
+    // AsmPrinter::doInitialization()).
+    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+    Parser->setSymbolScanningMode(true);
+
+    Parser->setTargetParser(*TAP);
+    if (Parser->Run(false))
+      return;
+
+    Init(Streamer);
+  }
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(

diff  --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index dbda25c34cee0..e37ea27364e6e 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,7 +113,8 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitDirectiveOptionArch();
+  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
+  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -523,19 +524,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitDirectiveOptionArch() {
+bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                                : RISCVOptionArchArgType::Minus;
+    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                               : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -549,11 +550,16 @@ bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   return false;
 }
 
+void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
+                                           bool DidPush) {
+  if (DidPush)
+    getTargetStreamer().emitDirectiveOptionPop();
+}
+
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
-  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitDirectiveOptionArch();
+  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -561,8 +567,7 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  if (EmittedOptionArch)
-    RTS.emitDirectiveOptionPop();
+  emitTargetFeaturePop(*STI, EmittedOptionArch);
   return false;
 }
 

diff  --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 2a5d16a470c4a..22d0d97090746 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
   }
 
   // Register the NSDI handler before calling the base class so that

diff  --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 16fdb93e2b21b..4b76b02c32fe4 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually

diff  --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index a486979a58c2d..e857c94330bad 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->setModuleInlineAsm("");
+  MergedM->removeModuleInlineAsm();
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.

diff  --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index c349fe33dd237..375ac069272eb 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,18 +1283,20 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  std::string Asm = GV->getParent()->getModuleInlineAsm();
-  std::string SearchStr = ".symver " + GVName + ",";
-  size_t Pos = Asm.find(SearchStr);
-  if (Pos != std::string::npos) {
-    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
-    Pos = Asm.find('@');
-
-    if (Pos == std::string::npos)
-      report_fatal_error(Twine("unsupported .symver: ", Asm));
-
-    Asm.replace(Pos, 1, Suffix + "@");
-    GV->getParent()->setModuleInlineAsm(Asm);
+  for (Module::GlobalAsmFragment &Frag :
+       GV->getParent()->getModuleInlineAsm()) {
+    std::string SearchStr = ".symver " + GVName + ",";
+    size_t Pos = Frag.Asm.find(SearchStr);
+    if (Pos != std::string::npos) {
+      Frag.Asm.replace(Pos, SearchStr.size(),
+                       ".symver " + GVName + Suffix + ",");
+      Pos = Frag.Asm.find('@');
+
+      if (Pos == std::string::npos)
+        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
+
+      Frag.Asm.replace(Pos, 1, Suffix + "@");
+    }
   }
 }
 

diff  --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 64910b4d1ce99..8e70902c4dac2 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->setModuleInlineAsm("");
+      MPart->removeModuleInlineAsm();
     ModuleCallback(std::move(MPart));
   }
 }

diff  --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
new file mode 100644
index 0000000000000..34e5b71e683a9
--- /dev/null
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -0,0 +1,25 @@
+; RUN: split-file %s %t
+; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
+; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
+; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
+; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
+
+;--- invalid_prop.ll
+; CHECK: unknown property name
+module asm(foo: "bar")
+    "asm"
+
+;--- missing_value1.ll
+; CHECK: expected property name followed by ':'
+module asm(target_features)
+    "asm"
+
+;--- missing_value2.ll
+; CHECK: expected string constant
+module asm(target_features:)
+    "asm"
+
+;--- missing_paren.ll
+; CHECK: expected ',' or ')'
+module asm(target_features: "bar"
+    "asm"

diff  --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index 288e0fc50a9ba..c79086bad1ba5 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index ef9490310fe5d..00b2bb9b30b62 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 538d6101b3994..5b86d5ffa1fee 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index 3bfef73e485dd..eca4bbdf03766 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index aaf9fe5492171..8551b98815b1a 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index b820417107e8b..883ac3eaaa293 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 67e96c7277cc3..624d5bd7703c6 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index d55aa1dc496df..97889d4687516 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,7 +16,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index f2770ea4e4d34..cd9a34c810d57 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,7 +9,8 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm "some assembly"
+; CHECK: module asm
+; CHECK-NEXT: "some assembly"
 module asm "some assembly"
 
 ; Named Types Test

diff  --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
new file mode 100644
index 0000000000000..406f25c5dd28c
--- /dev/null
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -0,0 +1,29 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; CHECK: module asm
+; CHECK:     "asm line 0"
+; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
+; CHECK:     "asm line 1"
+; CHECK:     "asm line 2"
+; CHECK: module asm(target_features: "+bar")
+; CHECK:     "asm line 3"
+; CHECK: module asm(target_cpu: "bar_cpu")
+; CHECK:     "asm line 4"
+; CHECK: module asm
+; CHECK:     "asm line 5"
+
+module asm
+    "asm line 0"
+
+module asm(target_features: "+foo", target_cpu: "foo_cpu")
+    "asm line 1"
+    "asm line 2"
+
+module asm(target_features: "+bar")
+    "asm line 3"
+
+module asm(target_cpu: "bar_cpu")
+    "asm line 4"
+
+module asm
+    "asm line 5"

diff  --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
new file mode 100644
index 0000000000000..ab16acab48688
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -0,0 +1,17 @@
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
+
+; This should work fine, because the module asm specifies the necessary
+; target features
+
+; SAME-FEATURES-NOT: .option arch
+; EXTRA-FEATURES: .option push
+; EXTRA-FEATURES: .option arch, +d
+; CHECK: fld ft0, 0(sp)
+; EXTRA-FEATURES: .option pop
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"

diff  --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 454c496d1d4bb..829dd829d51f9 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,7 +2,8 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan

diff  --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
new file mode 100644
index 0000000000000..73320185e778a
--- /dev/null
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -0,0 +1,18 @@
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
+; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
+; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
+; RUN: FileCheck %s --input-file %t.s.0
+
+; NM: T func
+
+; CHECK: fld ft0, 0(sp)
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+target triple = "riscv64-unknown-linux-gnu"
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"

diff  --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 88492e4366489..6c5173c3f9553 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,17 +32,19 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1:      module asm ".lto_discard foo"
-; ASM1-NEXT: module asm ".weak foo"
-; ASM1-NEXT: module asm ".equ foo,bar"
-
-; ASM2:      module asm ".lto_discard foo"
-; ASM2-NEXT: module asm ".weak foo"
-; ASM2-NEXT: module asm ".equ foo,bar"
-; ASM2-NEXT: module asm ".lto_discard"
-; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  module asm ".lto_discard foo"
+; ASM1: module asm
+; ASM1-NEXT: ".lto_discard foo"
+; ASM1-NEXT: ".weak foo"
+; ASM1-NEXT: ".equ foo,bar"
+
+; ASM2: module asm
+; ASM2-NEXT: ".lto_discard foo"
+; ASM2-NEXT: ".weak foo"
+; ASM2-NEXT: ".equ foo,bar"
+; ASM2-NEXT: ".lto_discard"
+; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  ".lto_discard foo"
 
 ; SYM: T foo
 

diff  --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
new file mode 100644
index 0000000000000..b097c7067ccc1
--- /dev/null
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -0,0 +1,2 @@
+module asm(target_features: "+bar")
+    "asm 2"

diff  --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 5e12a8c88d90a..612bce5be18f1 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,11 +9,12 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK:      module asm ".text"
-; CHECK-NEXT: module asm ".balign 4"
-; CHECK-NEXT: module asm ".arm"
-; CHECK-NEXT: module asm "add r1, r2, r2"
-; CHECK-NEXT: module asm ".text"
-; CHECK-NEXT: module asm ".balign 2"
-; CHECK-NEXT: module asm ".thumb"
-; CHECK-NEXT: module asm "orn r1, r2, r2"
+; CHECK: module asm
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 4"
+; CHECK-NEXT: ".arm"
+; CHECK-NEXT: "add r1, r2, r2"
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 2"
+; CHECK-NEXT: ".thumb"
+; CHECK-NEXT: "orn r1, r2, r2"

diff  --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
new file mode 100644
index 0000000000000..6c600981d72a5
--- /dev/null
+++ b/llvm/test/Linker/module-asm.ll
@@ -0,0 +1,9 @@
+; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
+
+; CHECK: module asm(target_features: "+foo")
+; CHECK-NEXT: "asm 1"
+; CHECK: module asm(target_features: "+bar")
+; CHECK-NEXT: "asm 2"
+
+module asm(target_features: "+foo")
+    "asm 1"

diff  --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 556c4fd992f0d..9525527b4531c 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,7 +9,8 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm
+; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.

diff  --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index ea4594a359cc0..7c3ed3da3537e 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,6 +1,7 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm ".symver external_addrtaken, alias1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index d8ebae17d4693..61141c71335bf 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,7 +3,8 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm
+; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 587ab3fbb5020..5796537661e60 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,6 +7,7 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm "ret"
+; M0: module asm
+; M0-NEXT: "ret"
 ; M1-NOT: module asm
 module asm "ret"

diff  --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index 4aeaef6d8d676..b5b68404e8690 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.setModuleInlineAsm("");
+    Program.removeModuleInlineAsm();
 }

diff  --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index 3038a6a2ed986..bb55678f91005 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,12 +1150,15 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
-  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
-    if (!line.empty())
-      asmArrayAttr.push_back(builder.getStringAttr(line));
+  for (const llvm::Module::GlobalAsmFragment &Frag :
+       llvmModule->getModuleInlineAsm()) {
+    // TODO: Preserve module asm properties.
+    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
+      if (!line.empty())
+        asmArrayAttr.push_back(builder.getStringAttr(line));
+  }
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));

diff  --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 2afb37cf005bf..813eeeafdeb44 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,5 +2,6 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm "foo"
-// CHECK: module asm "bar"
+// CHECK: module asm
+// CHECK-NEXT: "foo"
+// CHECK-NEXT: "bar"


        

From llvmlistbot at llvm.org  Fri Jul  3 00:59:18 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 00:59:18 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a476bd6.70920dee.3514b2.b0f4@mx.google.com>


https://github.com/nikic closed https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 01:18:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 01:18:50 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47706a.6c8668f2.74a35.0e77@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

* 136606 tests passed
* 3522 tests skipped
* 5 tests failed

## Failed Tests
(click on a test name to see its output)

### LLVM
<details>
<summary>LLVM.CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
not c:\_work\llvm-project\llvm-project\build\bin\llc.exe < C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\cp-async-bulk-tensor-g2s-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\cp-async-bulk-tensor-g2s-invalid.ll
# executed command: not 'c:\_work\llvm-project\llvm-project\build\bin\llc.exe' -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\cp-async-bulk-tensor-g2s-invalid.ll'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\cp-async-bulk-tensor-g2s-invalid.ll:8:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 3 out of range [0, 3)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 3 for arg 8 out of range [0,3)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 3 for arg 8 out of range [0,3)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\cp-async-bulk-tensor-g2s-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 3 for arg 8 out of range [0,3) 
# | check:8'0    {                                              search range start (exclusive)
# | check:8'1                                                   error: no match found in search range
# | check:8'2                  ?                                possible intended match
# |            2:  tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 false, i1 false, i32 3) 
# |            3: immarg value -1 for arg 8 out of range [0,3) 
# |            4:  tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 false, i1 false, i32 -1) 
# |            5: llc: error: '<stdin>': input module cannot be verified 
# | check:8'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
not c:\_work\llvm-project\llvm-project\build\bin\llc.exe < C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\fence-proxy-tensormap-invalid.ll -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\fence-proxy-tensormap-invalid.ll
# executed command: not 'c:\_work\llvm-project\llvm-project\build\bin\llc.exe' -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\fence-proxy-tensormap-invalid.ll'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\fence-proxy-tensormap-invalid.ll:4:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 130 out of range [128, 129)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 130 for arg 1 out of range [128,129)
# | ^
# | <stdin>:1:16: note: possible intended match here
# | immarg value 130 for arg 1 out of range [128,129)
# |                ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\fence-proxy-tensormap-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 130 for arg 1 out of range [128,129) 
# | check:4'0    {                                                    search range start (exclusive)
# | check:4'1                                                         error: no match found in search range
# | check:4'2                    ?                                    possible intended match
# |            2:  call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr %addr, i32 130) 
# |            3: llc: error: '<stdin>': input module cannot be verified 
# | check:4'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not c:\_work\llvm-project\llvm-project\build\bin\llc.exe < C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-block-scale-invalid.ll -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-block-scale-invalid.ll
# executed command: not 'c:\_work\llvm-project\llvm-project\build\bin\llc.exe' -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-block-scale-invalid.ll'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-block-scale-invalid.ll:7:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 0 out of range [1, 3)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 0 for arg 7 out of range [1,3)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 0 for arg 7 out of range [1,3)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-block-scale-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 0 for arg 7 out of range [1,3) 
# | check:7'0    {                                              search range start (exclusive)
# | check:7'1                                                   error: no match found in search range
# | check:7'2                  ?                                possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |            3: immarg value 5 for arg 8 out of range [0,4) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |            5: immarg value 0 for arg 7 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |            .
# |            .
# |            .
# |           12:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |           13: immarg value 0 for arg 7 out of range [1,3) 
# |           14:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |           15: immarg value 5 for arg 8 out of range [0,4) 
# |           16:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |           17: llc: error: '<stdin>': input module cannot be verified 
# | check:7'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not c:\_work\llvm-project\llvm-project\build\bin\llc.exe < C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-invalid.ll
# executed command: not 'c:\_work\llvm-project\llvm-project\build\bin\llc.exe' -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-invalid.ll'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-invalid.ll:8:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 5 out of range [0, 4)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 5 for arg 5 out of range [0,4)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 5 for arg 5 out of range [0,4)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 5 for arg 5 out of range [0,4) 
# | check:8'0    {                                              search range start (exclusive)
# | check:8'1                                                   error: no match found in search range
# | check:8'2                  ?                                possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3) 
# |            3: immarg value 0 for arg 6 out of range [1,3) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3) 
# |            5: immarg value 3 for arg 6 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3) 
# |            .
# |            .
# |            .
# |           16:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0) 
# |           17: immarg value 5 for arg 6 out of range [0,4) 
# |           18:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0) 
# |           19: immarg value 5 for arg 7 out of range [0,4) 
# |           20:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5) 
# |           21: llc: error: '<stdin>': input module cannot be verified 
# | check:8'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not c:\_work\llvm-project\llvm-project\build\bin\llc.exe < C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-scale-d-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-scale-d-invalid.ll
# executed command: not 'c:\_work\llvm-project\llvm-project\build\bin\llc.exe' -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-scale-d-invalid.ll'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-scale-d-invalid.ll:6:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 16 out of range [0, 16)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 16 for arg 5 out of range [0,16)
# | ^
# | <stdin>:1:15: note: possible intended match here
# | immarg value 16 for arg 5 out of range [0,16)
# |               ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\llvm\test\CodeGen\NVPTX\tcgen05-mma-scale-d-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 16 for arg 5 out of range [0,16) 
# | check:6'0    {                                                search range start (exclusive)
# | check:6'1                                                     error: no match found in search range
# | check:6'2                   ?                                 possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0) 
# |            3: immarg value 3 for arg 6 out of range [0,2) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0) 
# |            5: immarg value 0 for arg 7 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0) 
# |            7: immarg value 5 for arg 8 out of range [0,4) 
# |            8:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5) 
# |            9: immarg value 16 for arg 5 out of range [0,16) 
# |           10:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0) 
# |           11: immarg value 3 for arg 7 out of range [0,2) 
# |           12:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0) 
# |           13: immarg value 5 for arg 8 out of range [0,4) 
# |           14:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5) 
# |           15: llc: error: '<stdin>': input module cannot be verified 
# | check:6'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 01:18:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 01:18:50 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47706a.eea6ab6e.1256b6.640e@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 176678 tests passed
* 3570 tests skipped
* 5 tests failed

## Failed Tests
(click on a test name to see its output)

### LLVM
<details>
<summary>LLVM.CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
# executed command: not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll:8:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 3 out of range [0, 3)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 3 for arg 8 out of range [0,3)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 3 for arg 8 out of range [0,3)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 3 for arg 8 out of range [0,3) 
# | check:8'0    {                                              search range start (exclusive)
# | check:8'1                                                   error: no match found in search range
# | check:8'2                  ?                                possible intended match
# |            2:  tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 false, i1 false, i32 3) 
# |            3: immarg value -1 for arg 8 out of range [0,3) 
# |            4:  tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 false, i1 false, i32 -1) 
# |            5: llc: error: '<stdin>': input module cannot be verified 
# | check:8'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
# executed command: not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll:4:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 130 out of range [128, 129)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 130 for arg 1 out of range [128,129)
# | ^
# | <stdin>:1:16: note: possible intended match here
# | immarg value 130 for arg 1 out of range [128,129)
# |                ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 130 for arg 1 out of range [128,129) 
# | check:4'0    {                                                    search range start (exclusive)
# | check:4'1                                                         error: no match found in search range
# | check:4'2                    ?                                    possible intended match
# |            2:  call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr %addr, i32 130) 
# |            3: llc: error: '<stdin>': input module cannot be verified 
# | check:4'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
# executed command: not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll:7:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 0 out of range [1, 3)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 0 for arg 7 out of range [1,3)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 0 for arg 7 out of range [1,3)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 0 for arg 7 out of range [1,3) 
# | check:7'0    {                                              search range start (exclusive)
# | check:7'1                                                   error: no match found in search range
# | check:7'2                  ?                                possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |            3: immarg value 5 for arg 8 out of range [0,4) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |            5: immarg value 0 for arg 7 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |            .
# |            .
# |            .
# |           12:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |           13: immarg value 0 for arg 7 out of range [1,3) 
# |           14:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0) 
# |           15: immarg value 5 for arg 8 out of range [0,4) 
# |           16:  call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5) 
# |           17: llc: error: '<stdin>': input module cannot be verified 
# | check:7'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
# executed command: not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll:8:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 5 out of range [0, 4)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 5 for arg 5 out of range [0,4)
# | ^
# | <stdin>:1:14: note: possible intended match here
# | immarg value 5 for arg 5 out of range [0,4)
# |              ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 5 for arg 5 out of range [0,4) 
# | check:8'0    {                                              search range start (exclusive)
# | check:8'1                                                   error: no match found in search range
# | check:8'2                  ?                                possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3) 
# |            3: immarg value 0 for arg 6 out of range [1,3) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3) 
# |            5: immarg value 3 for arg 6 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3) 
# |            .
# |            .
# |            .
# |           16:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0) 
# |           17: immarg value 5 for arg 6 out of range [0,4) 
# |           18:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0) 
# |           19: immarg value 5 for arg 7 out of range [0,4) 
# |           20:  call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5) 
# |           21: llc: error: '<stdin>': input module cannot be verified 
# | check:8'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>LLVM.CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc < /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
# executed command: not /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/llc -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll:6:11: error: CHECK: expected string not found in input
# |  ; CHECK: immarg value 16 out of range [0, 16)
# |           ^
# | <stdin>:1:1: note: scanning from here
# | immarg value 16 for arg 5 out of range [0,16)
# | ^
# | <stdin>:1:15: note: possible intended match here
# | immarg value 16 for arg 5 out of range [0,16)
# |               ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: immarg value 16 for arg 5 out of range [0,16) 
# | check:6'0    {                                                search range start (exclusive)
# | check:6'1                                                     error: no match found in search range
# | check:6'2                   ?                                 possible intended match
# |            2:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0) 
# |            3: immarg value 3 for arg 6 out of range [0,2) 
# |            4:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0) 
# |            5: immarg value 0 for arg 7 out of range [1,3) 
# |            6:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0) 
# |            7: immarg value 5 for arg 8 out of range [0,4) 
# |            8:  call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5) 
# |            9: immarg value 16 for arg 5 out of range [0,16) 
# |           10:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0) 
# |           11: immarg value 3 for arg 7 out of range [0,2) 
# |           12:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0) 
# |           13: immarg value 5 for arg 8 out of range [0,4) 
# |           14:  call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5) 
# |           15: llc: error: '<stdin>': input module cannot be verified 
# | check:6'3                                                            } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 02:18:08 2026
From: llvmlistbot at llvm.org (Yuvaraj Venkatesh)
Date: Fri, 03 Jul 2026 02:18:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a477e50.e9d484b9.11ad58.58e5@mx.google.com>


Yuvaraj-Venkatesh wrote:

LGTM, Thanks

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 02:23:15 2026
From: llvmlistbot at llvm.org (Rahul Joshi)
Date: Fri, 03 Jul 2026 02:23:15 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a477f83.f732ad81.1d8ca.3b94@mx.google.com>


https://github.com/jurahul updated https://github.com/llvm/llvm-project/pull/207300

>From c4ca246823df7394962ce1e347677e207983c939 Mon Sep 17 00:00:00 2001
From: Rahul Joshi <rjoshi at nvidia.com>
Date: Thu, 2 Jul 2026 17:47:20 -0700
Subject: [PATCH] [NFC][LLVM] Define range attribute for `llvm.prefetch` args

Define valid range for `llvm.prefetch` immarg values and eliminate
C++ verification code.

Additionally, change the range verification failure message to
include the argument number and simplify formatting using `formatv`.
---
 llvm/include/llvm/IR/Intrinsics.td            |  6 ++--
 llvm/lib/IR/Verifier.cpp                      | 14 ++-------
 llvm/test/Assembler/invalid-immarg5.ll        |  2 +-
 .../NVPTX/cp-async-bulk-tensor-g2s-invalid.ll |  8 ++---
 .../NVPTX/fence-proxy-tensormap-invalid.ll    |  4 +--
 .../NVPTX/tcgen05-mma-block-scale-invalid.ll  | 19 +++++-------
 .../test/CodeGen/NVPTX/tcgen05-mma-invalid.ll | 26 +++++++---------
 .../NVPTX/tcgen05-mma-scale-d-invalid.ll      | 18 +++++------
 .../AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll  | 30 +++++++++----------
 .../Verifier/NVPTX/fence-proxy.tensormap.ll   |  8 ++---
 llvm/test/Verifier/NVPTX/setmaxnreg.ll        |  4 +--
 llvm/test/Verifier/intrinsic-immarg.ll        | 12 ++++++++
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir |  2 +-
 13 files changed, 74 insertions(+), 79 deletions(-)

diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),
diff --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
index 1c35fbead389e..0cef9c1d9405c 100644
--- a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
@@ -1,14 +1,12 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) writeonly, ptr addrspace(3), ptr readonly, i32, i16, i64, i1 immarg, i1 immarg, i32 immarg range(i32 0, 3))
 
 define void @test_cp_async_bulk_tensor_g2s_tile_1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch) {
-  ; CHECK: immarg value 3 out of range [0, 3)
+  ; CHECK: immarg value 3 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 3)
 
-  ; CHECK: immarg value -1 out of range [0, 3)
+  ; CHECK: immarg value -1 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 -1)
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
index ab35e4fb396d6..52f494ae590b6 100644
--- a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
@@ -1,7 +1,7 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 130 out of range [128, 129)
+  ; CHECK: immarg value 130 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 130);
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
index c0f6f4c7c46bd..bd3ed2e447c0b 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
@@ -1,23 +1,20 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: not llc < %s -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_101a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_110a -march=nvptx64 -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_block_scale_invalid_flags(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b) {
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
index dff829ecf5321..c80755b6711d8 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
@@ -1,37 +1,33 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_101a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_110a -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3)
-  ; CHECK: immarg value 3 out of range [1, 3)
+  ; CHECK: immarg value 3 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 5)
-  ; CHECK: immarg value 2 out of range [0, 2)
+  ; CHECK: immarg value 2 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.ashift(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 2)
   ret void
 }
 
 define void @tcgen05_mma_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_ws_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
index 7c884a70b7530..5987668b10add 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
@@ -1,25 +1,23 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_scale_d_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 6 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_scale_d_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
   ret <2 x half> %result
 }
 
-; CHECK: immarg value 15 out of range [0, 4)
+; CHECK: immarg value 15 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
@@ -87,14 +87,14 @@ define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f32 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 8 out of range [0, 4)
+; CHECK: immarg value 8 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
@@ -105,14 +105,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f16 - dest_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 12 out of range [0, 4)
+; CHECK: immarg value 12 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
@@ -123,14 +123,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %ol
 ; llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 9 out of range [0, 4)
+; CHECK: immarg value 9 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_way_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)
diff --git a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
index 4fa7a7ae71001..652990a700a6e 100644
--- a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
+++ b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
@@ -1,16 +1,16 @@
 ; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cluster(ptr addrspace(0) %addr, i32 129);
 
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.gpu(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.sys(ptr addrspace(0) %addr, i32 129);
 
   ret void
diff --git a/llvm/test/Verifier/NVPTX/setmaxnreg.ll b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
index 1afebeab4742c..79714a0521c36 100644
--- a/llvm/test/Verifier/NVPTX/setmaxnreg.ll
+++ b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
@@ -7,10 +7,10 @@ define void @test_set_maxn_reg() {
   ; CHECK: reg_count argument to nvvm.setmaxnreg must be in multiples of 8
   call void @llvm.nvvm.setmaxnreg.inc.sync.aligned.u32(i32 95)
 
-  ; CHECK: immarg value 16 out of range [24, 257)
+  ; CHECK: immarg value 16 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 16)
 
-  ; CHECK: immarg value 264 out of range [24, 257)
+  ; CHECK: immarg value 264 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 264)
   ret void
 }
diff --git a/llvm/test/Verifier/intrinsic-immarg.ll b/llvm/test/Verifier/intrinsic-immarg.ll
index c95c7214fde71..39ba3cb480b4d 100644
--- a/llvm/test/Verifier/intrinsic-immarg.ll
+++ b/llvm/test/Verifier/intrinsic-immarg.ll
@@ -154,6 +154,18 @@ define void @test_prefetch(ptr %ptr, i32 %arg0, i32 %arg1) {
   ; CHECK-NEXT:  i32 %arg1
   call void @llvm.prefetch(ptr %ptr, i32 %arg0, i32 0, i32 0)
   call void @llvm.prefetch(ptr %ptr, i32 0, i32 %arg1, i32 0)
+
+  ; CHECK: immarg value 2 for arg 1 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 2, i32 0, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 2, i32 0, i32 0)
+
+  ; CHECK: immarg value 8 for arg 2 out of range [0,4)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 8, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 8, i32 0)
+
+  ; CHECK: immarg value 4 for arg 3 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 0, i32 4)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 0, i32 4)
   ret void
 }
 
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index ea393dd445eda..5266470891b4d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1420,7 +1420,7 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
 // CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
-// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg, i32 immarg, i32 immarg)
+// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0


From llvmlistbot at llvm.org  Fri Jul  3 02:42:52 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:42:52 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a47841c.c56fb102.11841f.bb77@mx.google.com>


https://github.com/egebeysel updated https://github.com/llvm/llvm-project/pull/204007

>From c456cae1e6861224d54b901f740293935655e01e Mon Sep 17 00:00:00 2001
From: Ege Beysel <beyselege at gmail.com>
Date: Wed, 10 Jun 2026 17:07:05 +0000
Subject: [PATCH 1/4] feat(mlir/TilingInterface): add InnerTileAlignment hint
 as overloaded tiling methods

Add an optional, caller-supplied per-dimension InnerTileAlignment
{Unknown,Multiple,Equal} hint describing how each loop tile size relates to a
linalg.pack/linalg.unpack inner tile, so the pack/unpack TilingInterface impls
need not re-derive it from the materialized IR (which fails for scalable inner
tiles). The caller performed the tiling and knows both the tile sizes and the
op's inner tiles.

Rather than changing the existing method signatures, the hint is added as
same-name OVERLOADS of getTiledImplementation, generateResultTileValue,
getTiledImplementationFromOperandTiles and getIterationDomainTileFromOperandTiles.
The hint-bearing overloads default to forwarding to the hint-less ones, so only
linalg.pack/unpack consult them; every other implementer inherits the defaulted
forwarder and need not reason about the hint. This is a caller assertion and is
always honored by pack/unpack; when the relationship is statically decidable,
it is asserted that the hint agrees with them.

Downstream integration: because the hint methods are overloads of existing
interface methods, an out-of-tree TilingInterface external model that overrides
only a hint-less overload hides the inherited defaulted hint-bearing overload
(C++ name hiding) and fails to compile when the interface forwards to it. Such
implementers must add a `using Base::<method>;` declaration (or define the
overload explicitly to forward) for each affected method; all in-tree
implementers are updated here.

Signed-off-by: Ege Beysel <beyselege at gmail.com>
---
 .../include/mlir/Interfaces/TilingInterface.h |  33 +++
 .../mlir/Interfaces/TilingInterface.td        | 104 ++++++++
 mlir/lib/Dialect/Linalg/IR/CMakeLists.txt     |   1 +
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      |  36 +++
 .../Linalg/Transforms/TilingInterfaceImpl.cpp | 232 +++++++++++++++---
 .../Tensor/IR/TensorTilingInterfaceImpl.cpp   |   5 +
 mlir/lib/Interfaces/TilingInterface.cpp       |  22 ++
 mlir/test/lib/Dialect/Test/TestOpDefs.cpp     |   6 +
 .../llvm-project-overlay/mlir/BUILD.bazel     |   1 +
 9 files changed, 408 insertions(+), 32 deletions(-)

diff --git a/mlir/include/mlir/Interfaces/TilingInterface.h b/mlir/include/mlir/Interfaces/TilingInterface.h
index 8693cbea7f0b0..8b685a30868e8 100644
--- a/mlir/include/mlir/Interfaces/TilingInterface.h
+++ b/mlir/include/mlir/Interfaces/TilingInterface.h
@@ -66,6 +66,39 @@ struct MergeResult {
   SmallVector<Value> replacements;
 };
 
+/// Per-dimension alignment of a loop tile size to a `linalg.pack` /
+/// `linalg.unpack` inner tile size, supplied by the caller (which performed the
+/// tiling and knows both the tile sizes and the inner tiles) so that
+/// pack/unpack TilingInterface implementations need not re-derive it from the
+/// materialized IR. An absent entry (or `Unknown`) means "no information": the
+/// implementation must fall back to its prior behavior for that dimension.
+///   - `Multiple`: the loop tile size is an integer multiple of the pack/unpack
+///   inner tile.
+///   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+enum class InnerTileAlignment : int64_t { Unknown = 0, Multiple, Equal };
+
+/// Returns true iff `value` is a valid `InnerTileAlignment` enumerator.
+inline bool isValidInnerTileAlignment(int64_t value) {
+  switch (static_cast<InnerTileAlignment>(value)) {
+  case InnerTileAlignment::Unknown:
+  case InnerTileAlignment::Multiple:
+  case InnerTileAlignment::Equal:
+    return true;
+  }
+  return false;
+}
+
+/// Verifies that every entry of a raw `inner_tile_alignments` integer array is
+/// a valid `InnerTileAlignment`, emitting the standard op error on `op`
+/// otherwise.
+LogicalResult verifyInnerTileAlignments(Operation *op,
+                                        ArrayRef<int64_t> alignments);
+
+/// Maps a validated `inner_tile_alignments` integer array onto the
+/// per-dimension `InnerTileAlignment` hints consumed by the tiling driver.
+SmallVector<InnerTileAlignment>
+convertInnerTileAlignments(ArrayRef<int64_t> alignments);
+
 } // namespace mlir
 
 /// Include the ODS generated interface header files.
diff --git a/mlir/include/mlir/Interfaces/TilingInterface.td b/mlir/include/mlir/Interfaces/TilingInterface.td
index e14ea926f4d00..5845545ce686a 100644
--- a/mlir/include/mlir/Interfaces/TilingInterface.td
+++ b/mlir/include/mlir/Interfaces/TilingInterface.td
@@ -59,6 +59,16 @@ def TilingInterface : OpInterface<"TilingInterface"> {
     description below):
       - `getTiledImplementationFromOperandTiles`
       - `getIterationDomainTileFromOperandTiles`.
+
+    `getTiledImplementation`, `generateResultTileValue`,
+    `getTiledImplementationFromOperandTiles` and
+    `getIterationDomainTileFromOperandTiles` each have a second, overloaded form
+    taking an extra `ArrayRef<InnerTileAlignment>` caller hint (see
+    `InnerTileAlignment` in TilingInterface.h). The hint-bearing overload
+    defaults to forwarding to the hint-less one, so only ops that opt in consult
+    it. Because the two forms share a source name, an implementer that defines
+    only the hint-less overload hides the inherited defaulted one (C++ name
+    hiding) and fails to compile when the interface forwards to it.
   }];
   let cppNamespace = "::mlir";
   let methods = [
@@ -115,6 +125,28 @@ def TilingInterface : OpInterface<"TilingInterface"> {
           return {};
         }]
       >,
+      InterfaceMethod<
+        /*desc=*/[{
+          Variant of `getTiledImplementation` that additionally takes a
+          per-iteration-domain `InnerTileAlignment` hint (see
+          `InnerTileAlignment`) asserting how each loop tile size relates to a
+          pack/unpack inner tile. The hint is consulted only by pack/unpack
+          implementations and may be empty (all `Unknown`). The default
+          implementation ignores it and forwards to the hint-less overload.
+        }],
+        /*retType=*/"::mlir::FailureOr<::mlir::TilingResult>",
+        /*methodName=*/"getTiledImplementation",
+        /*args=*/(ins
+            "::mlir::OpBuilder &":$b,
+            "::mlir::ArrayRef<::mlir::OpFoldResult> ":$offsets,
+            "::mlir::ArrayRef<::mlir::OpFoldResult> ":$sizes,
+            "::mlir::ArrayRef<::mlir::InnerTileAlignment> ":$innerTileAlignments),
+        /*methodBody=*/"",
+        /*defaultImplementation=*/[{
+          return ::mlir::cast<::mlir::TilingInterface>($_op.getOperation())
+              .getTiledImplementation(b, offsets, sizes);
+        }]
+      >,
       InterfaceMethod<
         /*desc=*/[{
           Method to return the position of the result tile computed by the
@@ -199,6 +231,28 @@ def TilingInterface : OpInterface<"TilingInterface"> {
           return failure();
         }]
       >,
+      InterfaceMethod<
+        /*desc=*/[{
+          Variant of `generateResultTileValue` that additionally takes a
+          per-iteration-domain `InnerTileAlignment` hint (see
+          `InnerTileAlignment`). The hint is consulted only by pack/unpack
+          implementations and may be empty (all `Unknown`). The default
+          implementation ignores it and forwards to the hint-less overload.
+        }],
+        /*retType=*/"::mlir::FailureOr<::mlir::TilingResult>",
+        /*methodName=*/"generateResultTileValue",
+        /*args=*/(ins
+          "::mlir::OpBuilder &":$b,
+          "unsigned":$resultNumber,
+          "::mlir::ArrayRef<::mlir::OpFoldResult>":$offsets,
+          "::mlir::ArrayRef<::mlir::OpFoldResult>":$sizes,
+          "::mlir::ArrayRef<::mlir::InnerTileAlignment>":$innerTileAlignments),
+        /*methodBody=*/"",
+        /*defaultImplementation=*/[{
+          return ::mlir::cast<::mlir::TilingInterface>($_op.getOperation())
+              .generateResultTileValue(b, resultNumber, offsets, sizes);
+        }]
+      >,
       InterfaceMethod<
         /*desc=*/[{
           Method to generate the tiled implementation of an operation that uses
@@ -229,6 +283,29 @@ def TilingInterface : OpInterface<"TilingInterface"> {
           return failure();
         }]
       >,
+      InterfaceMethod<
+        /*desc=*/[{
+          Variant of `getTiledImplementationFromOperandTiles` that additionally
+          takes a per-iteration-domain `InnerTileAlignment` hint (see
+          `InnerTileAlignment`). The hint is consulted only by pack/unpack
+          implementations and may be empty (all `Unknown`). The default
+          implementation ignores it and forwards to the hint-less overload.
+        }],
+        /*retType=*/"::mlir::FailureOr<::mlir::TilingResult>",
+        /*methodName=*/"getTiledImplementationFromOperandTiles",
+        /*args=*/(ins
+          "::mlir::OpBuilder &":$b,
+          "::mlir::ArrayRef<unsigned>":$operandNumbers,
+          "::mlir::ArrayRef<::mlir::SmallVector<::mlir::OpFoldResult>>":$allOffsets,
+          "::mlir::ArrayRef<::mlir::SmallVector<::mlir::OpFoldResult>>":$allSizes,
+          "::mlir::ArrayRef<::mlir::InnerTileAlignment>":$innerTileAlignments),
+        /*methodBody=*/"",
+        /*defaultImplementation=*/[{
+          return ::mlir::cast<::mlir::TilingInterface>($_op.getOperation())
+              .getTiledImplementationFromOperandTiles(b, operandNumbers,
+                                                      allOffsets, allSizes);
+        }]
+      >,
       InterfaceMethod<
         /*desc=*/[{
           Method to return the tile of the iteration domain that uses a given
@@ -304,6 +381,33 @@ def TilingInterface : OpInterface<"TilingInterface"> {
           return failure();
         }]
       >,
+      InterfaceMethod<
+        /*desc=*/[{
+          Variant of `getIterationDomainTileFromOperandTiles` that additionally
+          takes a per-iteration-domain `InnerTileAlignment` hint (see
+          `InnerTileAlignment`). The hint is consulted only by pack/unpack
+          implementations and may be empty (all `Unknown`). The default
+          implementation ignores it and forwards to the hint-less overload.
+        }],
+        /*retType=*/"::llvm::LogicalResult",
+        /*methodName=*/"getIterationDomainTileFromOperandTiles",
+        /*args=*/(ins
+          "::mlir::OpBuilder &":$b,
+          "::mlir::ArrayRef<unsigned>":$operandNumbers,
+          "::mlir::ArrayRef<::mlir::SmallVector<::mlir::OpFoldResult>> ":$allOffsets,
+          "::mlir::ArrayRef<::mlir::SmallVector<::mlir::OpFoldResult>> ":$allSizes,
+          "::mlir::SmallVectorImpl<::mlir::OpFoldResult> &":$iterDomainOffsets,
+          "::mlir::SmallVectorImpl<::mlir::OpFoldResult> &":$iterDomainSizes,
+          "::mlir::ArrayRef<::mlir::InnerTileAlignment> ":$innerTileAlignments),
+        /*methodBody=*/"",
+        /*defaultImplementation=*/[{
+          return ::mlir::cast<::mlir::TilingInterface>($_op.getOperation())
+              .getIterationDomainTileFromOperandTiles(b, operandNumbers,
+                                                      allOffsets, allSizes,
+                                                      iterDomainOffsets,
+                                                      iterDomainSizes);
+        }]
+      >,
       InterfaceMethod<
         /*desc=*/[{
           Method to return the tile of the iteration domain based
diff --git a/mlir/lib/Dialect/Linalg/IR/CMakeLists.txt b/mlir/lib/Dialect/Linalg/IR/CMakeLists.txt
index ec433284e17ad..ec4fd82468dbb 100644
--- a/mlir/lib/Dialect/Linalg/IR/CMakeLists.txt
+++ b/mlir/lib/Dialect/Linalg/IR/CMakeLists.txt
@@ -34,6 +34,7 @@ add_mlir_dialect_library(MLIRLinalgDialect
   MLIRMathDialect
   MLIRMemRefDialect
   MLIRTensorDialect
+  MLIRTilingInterface
   MLIRValueBoundsOpInterface
   MLIRViewLikeInterface
   )
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 1a56c5a483e73..8b2f5064e78bd 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -2940,6 +2940,15 @@ SmallVector<utils::IteratorType> SoftmaxOp::getLoopIteratorTypes() {
   return iteratorTypes;
 }
 
+/// The inner tile alignment hint is only used by `linalg.pack` and
+/// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
+/// overload.
+FailureOr<TilingResult> SoftmaxOp::getTiledImplementation(
+    OpBuilder &builder, ArrayRef<OpFoldResult> offsets,
+    ArrayRef<OpFoldResult> sizes, ArrayRef<InnerTileAlignment>) {
+  return getTiledImplementation(builder, offsets, sizes);
+}
+
 FailureOr<TilingResult>
 SoftmaxOp::getTiledImplementation(OpBuilder &builder,
                                   ArrayRef<OpFoldResult> offsets,
@@ -3293,6 +3302,15 @@ LogicalResult WinogradFilterTransformOp::getResultTilePosition(
   return success();
 }
 
+/// The inner tile alignment hint is only used by `linalg.pack` and
+/// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
+/// overload.
+FailureOr<TilingResult> WinogradFilterTransformOp::getTiledImplementation(
+    OpBuilder &builder, ArrayRef<OpFoldResult> offsets,
+    ArrayRef<OpFoldResult> sizes, ArrayRef<InnerTileAlignment>) {
+  return getTiledImplementation(builder, offsets, sizes);
+}
+
 /// Implement tiling for winograd_filter_transform
 /// The input of winograd_filter_transform is (F, KH, KW, C).
 /// The output of winograd_filter_transform is (alphaH, alphaW, C, F)
@@ -3446,6 +3464,15 @@ LogicalResult WinogradInputTransformOp::getResultTilePosition(
   return success();
 }
 
+/// The inner tile alignment hint is only used by `linalg.pack` and
+/// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
+/// overload.
+FailureOr<TilingResult> WinogradInputTransformOp::getTiledImplementation(
+    OpBuilder &builder, ArrayRef<OpFoldResult> offsets,
+    ArrayRef<OpFoldResult> sizes, ArrayRef<InnerTileAlignment>) {
+  return getTiledImplementation(builder, offsets, sizes);
+}
+
 /// Implement tiling for winograd_input_transform
 /// The input of winograd_input_transform is (N, H, W, C).
 /// The output of winograd_input_transform is (alphaH, alphaW, tileH, tileW, N,
@@ -3641,6 +3668,15 @@ LogicalResult WinogradOutputTransformOp::getResultTilePosition(
   return success();
 }
 
+/// The inner tile alignment hint is only used by `linalg.pack` and
+/// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
+/// overload.
+FailureOr<TilingResult> WinogradOutputTransformOp::getTiledImplementation(
+    OpBuilder &builder, ArrayRef<OpFoldResult> offsets,
+    ArrayRef<OpFoldResult> sizes, ArrayRef<InnerTileAlignment>) {
+  return getTiledImplementation(builder, offsets, sizes);
+}
+
 /// Implement tiling for winograd_output_transform
 /// The input of winograd_output_transform is (alphaH, alphaW, tileH, tileW, N,
 /// F). The output of winograd_output_transform is (N, H, W, F) Users can
diff --git a/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp b/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
index 4eaa7bf0233c6..13b959fc7b0cc 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/TilingInterfaceImpl.cpp
@@ -96,6 +96,16 @@ template <typename LinalgOpTy>
 struct LinalgOpTilingInterface
     : public TilingInterface::ExternalModel<LinalgOpTilingInterface<LinalgOpTy>,
                                             LinalgOpTy> {
+  using Base =
+      TilingInterface::ExternalModel<LinalgOpTilingInterface<LinalgOpTy>,
+                                     LinalgOpTy>;
+  // Inherit the defaulted hint-bearing overloads; these ops do not require the
+  // hint (no inner tiles).
+  using Base::generateResultTileValue;
+  using Base::getIterationDomainTileFromOperandTiles;
+  using Base::getTiledImplementation;
+  using Base::getTiledImplementationFromOperandTiles;
+
   /// Return the loop iterator type.
   SmallVector<utils::IteratorType> getLoopIteratorTypes(Operation *op) const {
     LinalgOpTy concreteOp = cast<LinalgOpTy>(op);
@@ -924,6 +934,8 @@ static void generatePackOpScalarImplementationBody(PackOp packOp,
 
 struct PackOpTiling
     : public TilingInterface::ExternalModel<PackOpTiling, linalg::PackOp> {
+  using Base = TilingInterface::ExternalModel<PackOpTiling, linalg::PackOp>;
+  using Base::getTiledImplementation;
 
   SmallVector<utils::IteratorType> getLoopIteratorTypes(Operation *op) const {
     // Note that here we only consider untiled dimensions and outer tiled data
@@ -1055,20 +1067,39 @@ struct PackOpTiling
   generateResultTileValue(Operation *op, OpBuilder &b, unsigned resultNumber,
                           ArrayRef<OpFoldResult> offsets,
                           ArrayRef<OpFoldResult> sizes) const {
+    return generateResultTileValue(op, b, resultNumber, offsets, sizes,
+                                   /*innerTileAlignments=*/{});
+  }
+
+  FailureOr<TilingResult> generateResultTileValue(
+      Operation *op, OpBuilder &b, unsigned resultNumber,
+      ArrayRef<OpFoldResult> offsets, ArrayRef<OpFoldResult> sizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     auto packOp = cast<PackOp>(op);
     int64_t numTiles = packOp.getInnerDimsPos().size();
 
-    // tensor.pack op is fusible (as a producer) only if full inner tiles are
+    // linalg.pack op is fusible (as a producer) only if full inner tiles are
     // iterated or inner dims are not tiled. Otherwise, it will generate a
     // sequence of non-trivial ops (for partial tiles).
     for (auto offset : offsets.take_back(numTiles))
       if (!isZeroInteger(offset))
         return failure();
 
-    for (auto iter :
-         llvm::zip_equal(packOp.getMixedTiles(), sizes.take_back(numTiles)))
-      if (!isEqualConstantIntOrValue(std::get<0>(iter), std::get<1>(iter)))
+    // Each requested inner-dim size must cover a full inner tile. A caller may
+    // instead assert this via an `Equal` alignment hint. The hint is indexed by
+    // source dim, matching the consumer-fusion path.
+    ArrayRef<int64_t> innerDimsPos = packOp.getInnerDimsPos();
+    SmallVector<OpFoldResult> mixedTiles = packOp.getMixedTiles();
+    ArrayRef<OpFoldResult> innerSizes = sizes.take_back(numTiles);
+    for (auto [i, pos] : llvm::enumerate(innerDimsPos)) {
+      InnerTileAlignment alignment =
+          pos < static_cast<int64_t>(innerTileAlignments.size())
+              ? innerTileAlignments[pos]
+              : InnerTileAlignment::Unknown;
+      if (alignment != InnerTileAlignment::Equal &&
+          !isEqualConstantIntOrValue(mixedTiles[i], innerSizes[i]))
         return failure();
+    }
 
     FailureOr<TilingResult> tilingResult = getTiledImplementation(
         op, b, offsets.drop_back(numTiles), sizes.drop_back(numTiles));
@@ -1123,15 +1154,27 @@ struct PackOpTiling
     return success();
   }
 
+  LogicalResult getIterationDomainTileFromOperandTiles(
+      Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
+      ArrayRef<SmallVector<OpFoldResult>> allOffsets,
+      ArrayRef<SmallVector<OpFoldResult>> allSizes,
+      SmallVectorImpl<OpFoldResult> &resultOffsets,
+      SmallVectorImpl<OpFoldResult> &resultSizes) const {
+    return getIterationDomainTileFromOperandTiles(
+        op, b, operandNumbers, allOffsets, allSizes, resultOffsets, resultSizes,
+        /*innerTileAlignments=*/{});
+  }
+
   /// Method to return the position of iteration domain tile computed by the
-  /// tiled operation. In current `tensor.pack` context, the `resultOffsets` and
+  /// tiled operation. In current `linalg.pack` context, the `resultOffsets` and
   /// `resultSizes` only cover outer dimensions.
   LogicalResult getIterationDomainTileFromOperandTiles(
       Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
       ArrayRef<SmallVector<OpFoldResult>> allOffsets,
       ArrayRef<SmallVector<OpFoldResult>> allSizes,
       SmallVectorImpl<OpFoldResult> &resultOffsets,
-      SmallVectorImpl<OpFoldResult> &resultSizes) const {
+      SmallVectorImpl<OpFoldResult> &resultSizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     if (operandNumbers.size() != 1 || operandNumbers[0] != 0) {
       LLVM_DEBUG(
           { llvm::dbgs() << "unsupported operands for consumer fusion"; });
@@ -1162,6 +1205,14 @@ struct PackOpTiling
         std::optional<int64_t> cstInnerSize =
             getConstantIntValue(dimAndTileMapping[dim]);
 
+        // A caller-supplied alignment hint (see InnerTileAlignment) asserts
+        // that this packed dimension is tiled and how its loop tile size
+        // relates to the pack op inner tile size.
+        InnerTileAlignment innerTileAlignment =
+            dim < static_cast<int64_t>(innerTileAlignments.size())
+                ? innerTileAlignments[dim]
+                : InnerTileAlignment::Unknown;
+
         // If a dimension is not tiled, it is always valid to fuse the pack op,
         // even if the op has padding semantics. Because it always generates a
         // full slice along the dimension. The tile sizes are for unpacked
@@ -1169,9 +1220,14 @@ struct PackOpTiling
         // dimension is tiled.
         // TODO: It could be untiled if the `srcDimSize` is dynamic. It is a
         // hard check to determine if a dimension is tiled or not.
+        // A non-`Unknown` hint also means the caller asserts the dimension is
+        // tiled: `cstTileSize` is an upper bound, so a scalable/`min`-shaped
+        // tile (whose bound equals `srcDimSize`) would otherwise be mistaken
+        // for untiled and bypass the hint below.
         int64_t srcDimSize = packOp.getSourceType().getDimSize(dim);
         int64_t destDimSize = outerShapeWithoutTranspose[dim];
-        bool isTiled = failed(cstTileSize) ||
+        bool isTiled = innerTileAlignment != InnerTileAlignment::Unknown ||
+                       failed(cstTileSize) ||
                        ShapedType::isDynamic(srcDimSize) ||
                        cstTileSize.value() < srcDimSize;
         if (!isTiled) {
@@ -1199,9 +1255,30 @@ struct PackOpTiling
         // another word, we can only support tiling with consumer if the tile
         // size for the producer is a multiple of the inner tile size for the
         // packed dimensions at this moment.
-        if ((failed(cstTileSize) || !cstInnerSize ||
-             *cstTileSize % *cstInnerSize != 0))
-          return failure();
+
+        // The caller may assert how this packed dimension's loop tile size
+        // relates to the inner tile size via `innerTileAlignments` (see
+        // InnerTileAlignment). The hint is the source of truth and is honored
+        // when present. When both sizes are also statically known we assert the
+        // hint agrees with them (a contradicting hint is a caller bug). When
+        // the hint is `Unknown`, fall back to requiring a statically-provable
+        // multiple.
+        bool assumeInnerTileSizesMatchTiles =
+            innerTileAlignment == InnerTileAlignment::Equal;
+        bool staticallyDecidable =
+            !failed(cstTileSize) && cstInnerSize.has_value();
+        if (innerTileAlignment == InnerTileAlignment::Unknown) {
+          if (!staticallyDecidable || *cstTileSize % *cstInnerSize != 0)
+            return failure();
+        } else if (staticallyDecidable) {
+          assert(*cstTileSize % *cstInnerSize == 0 &&
+                 "InnerTileAlignment hint contradicts statically known tile "
+                 "sizes");
+          assert((innerTileAlignment != InnerTileAlignment::Equal ||
+                  *cstTileSize == *cstInnerSize) &&
+                 "InnerTileAlignment::Equal contradicts statically known tile "
+                 "sizes");
+        }
 
         using AV = affine::AffineValueExpr;
         affine::AffineBuilder ab(b, loc);
@@ -1212,7 +1289,11 @@ struct PackOpTiling
         auto avSize = AV(dim0).bind(sizes[dim]);
         auto avTileSize = AV(sym).bind(dimAndTileMapping[dim]);
         outerDimOffsets.push_back(ab.floor(avOffset, avTileSize));
-        outerDimSizes.push_back(ab.ceil(avSize, avTileSize));
+        // If the tile size equals the inner tile size, the outer dims are
+        // always 1.
+        outerDimSizes.push_back(assumeInnerTileSizesMatchTiles
+                                    ? b.getIndexAttr(1)
+                                    : ab.ceil(avSize, avTileSize));
       } else {
         outerDimOffsets.push_back(offsets[dim]);
         outerDimSizes.push_back(sizes[dim]);
@@ -1224,14 +1305,23 @@ struct PackOpTiling
     return success();
   }
 
-  /// Method to return the tiled implementation of tensor.pack as a consumer.
   FailureOr<TilingResult> getTiledImplementationFromOperandTiles(
       Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
       ArrayRef<SmallVector<OpFoldResult>> allOffsets,
       ArrayRef<SmallVector<OpFoldResult>> allSizes) const {
+    return getTiledImplementationFromOperandTiles(op, b, operandNumbers,
+                                                  allOffsets, allSizes,
+                                                  /*innerTileAlignments=*/{});
+  }
+
+  /// Method to return the tiled implementation of linalg.pack as a consumer.
+  FailureOr<TilingResult> getTiledImplementationFromOperandTiles(
+      Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
+      ArrayRef<SmallVector<OpFoldResult>> allOffsets,
+      ArrayRef<SmallVector<OpFoldResult>> allSizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     if (operandNumbers.size() != 1 || operandNumbers[0] != 0) {
-      LLVM_DEBUG(
-          { llvm ::dbgs() << "unhandled operands for consumer fusion"; });
+      LLVM_DEBUG({ llvm::dbgs() << "unhandled operands for consumer fusion"; });
       return failure();
     }
 
@@ -1257,7 +1347,7 @@ struct PackOpTiling
     SmallVector<OpFoldResult> outerDimOffsets, outerDimSizes;
     if (failed(getIterationDomainTileFromOperandTiles(
             op, b, operandNumbers, allOffsets, allSizes, outerDimOffsets,
-            outerDimSizes)))
+            outerDimSizes, innerTileAlignments)))
       return failure();
 
     SmallVector<OpFoldResult> outputOffsets, outputSizes;
@@ -1296,10 +1386,10 @@ struct UnpackTileDimInfo {
 /// Returns the needed information for tiling unpack op on `tileDim` with given
 /// `tileOffset` and `tileSize`. For more details, see the comment of the
 /// `getTiledImplementation`.
-static UnpackTileDimInfo getUnpackTileDimInfo(OpBuilder &b, UnPackOp unpackOp,
-                                              int64_t tileDim,
-                                              OpFoldResult tileOffset,
-                                              OpFoldResult tileSize) {
+static UnpackTileDimInfo
+getUnpackTileDimInfo(OpBuilder &b, UnPackOp unpackOp, int64_t tileDim,
+                     OpFoldResult tileOffset, OpFoldResult tileSize,
+                     InnerTileAlignment innerTileAlignment) {
   UnpackTileDimInfo info;
   Attribute zeroAttr = b.getIndexAttr(0);
   Attribute oneAttr = b.getIndexAttr(1);
@@ -1329,13 +1419,36 @@ static UnpackTileDimInfo getUnpackTileDimInfo(OpBuilder &b, UnPackOp unpackOp,
       presburger::BoundType::UB, tileSize,
       /*stopCondition=*/nullptr, ValueBoundsOptions{/*closedUB=*/true});
   std::optional<int64_t> cstInnerSize = getConstantIntValue(innerTileSize);
-  if (!failed(cstSize) && cstInnerSize) {
-    if (*cstSize % *cstInnerSize == 0)
+  // The caller may assert how this dimension's loop tile size relates to the
+  // op's inner tile size via `innerTileAlignment` (see InnerTileAlignment). The
+  // hint is the source of truth and is honored when present: `Equal`/`Multiple`
+  // both mean the tile is aligned to (a multiple of) the inner tile, and
+  // `Equal` additionally collapses the source slice to a single inner tile.
+  // When both sizes are also statically known we assert the hint agrees with
+  // them (a contradicting hint is a caller bug). When `Unknown`, fall back to
+  // the static upper-bound path below.
+  bool assumeInnerTileSizesMatchTiles =
+      innerTileAlignment == InnerTileAlignment::Equal;
+  bool staticallyDecidable = !failed(cstSize) && cstInnerSize.has_value();
+  if (innerTileAlignment != InnerTileAlignment::Unknown) {
+    info.isAlignedToInnerTileSize = true;
+    if (staticallyDecidable) {
+      assert(*cstSize % *cstInnerSize == 0 &&
+             "InnerTileAlignment hint contradicts statically known tile sizes");
+      assert((innerTileAlignment != InnerTileAlignment::Equal ||
+              *cstSize == *cstInnerSize) &&
+             "InnerTileAlignment::Equal contradicts statically known tile "
+             "sizes");
+    }
+  }
+  if (info.isAlignedToInnerTileSize || (!failed(cstSize) && cstInnerSize)) {
+    if (!info.isAlignedToInnerTileSize && *cstSize % *cstInnerSize == 0)
       info.isAlignedToInnerTileSize = true;
 
     // If the tiling size equals to the inner tiling size, the outer dims are
     // always 1.
-    if (*cstInnerSize == *cstSize) {
+    if (assumeInnerTileSizesMatchTiles ||
+        (cstInnerSize && !failed(cstSize) && *cstInnerSize == *cstSize)) {
       auto lhs = AV(dim0).bind(tileOffset);
       auto rhs = AV(dim1).bind(innerTileSize);
       info.sourceOffset = ab.floor(lhs, rhs);
@@ -1391,6 +1504,8 @@ static UnpackTileDimInfo getUnpackTileDimInfo(OpBuilder &b, UnPackOp unpackOp,
 
 struct UnPackOpTiling
     : public TilingInterface::ExternalModel<UnPackOpTiling, linalg::UnPackOp> {
+  using Base = TilingInterface::ExternalModel<UnPackOpTiling, linalg::UnPackOp>;
+  using Base::getIterationDomainTileFromOperandTiles;
 
   SmallVector<utils::IteratorType> getLoopIteratorTypes(Operation *op) const {
     auto unpackOp = cast<UnPackOp>(op);
@@ -1421,6 +1536,14 @@ struct UnPackOpTiling
   getTiledImplementation(Operation *op, OpBuilder &b,
                          ArrayRef<OpFoldResult> offsets,
                          ArrayRef<OpFoldResult> sizes) const {
+    return getTiledImplementation(op, b, offsets, sizes,
+                                  /*innerTileAlignments=*/{});
+  }
+
+  FailureOr<TilingResult> getTiledImplementation(
+      Operation *op, OpBuilder &b, ArrayRef<OpFoldResult> offsets,
+      ArrayRef<OpFoldResult> sizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     auto unpackOp = cast<UnPackOp>(op);
     // TODO: Support Memref UnPackOp. Temporarily return failure.
     if (!unpackOp.hasPureTensorSemantics())
@@ -1440,8 +1563,11 @@ struct UnPackOpTiling
     SmallVector<OpFoldResult> sliceSrcIndices, sliceSrcSizes;
     SmallVector<OpFoldResult> destExpandedSizes, resultOffsetsFromDest;
     for (auto dim : llvm::seq<int64_t>(0, destRank)) {
-      UnpackTileDimInfo info =
-          getUnpackTileDimInfo(b, unpackOp, dim, offsets[dim], sizes[dim]);
+      UnpackTileDimInfo info = getUnpackTileDimInfo(
+          b, unpackOp, dim, offsets[dim], sizes[dim],
+          dim < static_cast<int64_t>(innerTileAlignments.size())
+              ? innerTileAlignments[dim]
+              : InnerTileAlignment::Unknown);
       if (!info.isAlignedToInnerTileSize)
         isPerfectTilingCase = false;
       sliceSrcIndices.push_back(info.sourceOffset);
@@ -1510,8 +1636,16 @@ struct UnPackOpTiling
   generateResultTileValue(Operation *op, OpBuilder &b, unsigned resultNumber,
                           ArrayRef<OpFoldResult> offsets,
                           ArrayRef<OpFoldResult> sizes) const {
+    return generateResultTileValue(op, b, resultNumber, offsets, sizes,
+                                   /*innerTileAlignments=*/{});
+  }
+
+  FailureOr<TilingResult> generateResultTileValue(
+      Operation *op, OpBuilder &b, unsigned resultNumber,
+      ArrayRef<OpFoldResult> offsets, ArrayRef<OpFoldResult> sizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     FailureOr<TilingResult> tilingResult =
-        getTiledImplementation(op, b, offsets, sizes);
+        getTiledImplementation(op, b, offsets, sizes, innerTileAlignments);
     if (failed(tilingResult))
       return failure();
     return tilingResult.value();
@@ -1642,11 +1776,21 @@ struct UnPackOpTiling
     return success();
   }
 
-  /// Method to return the tiled implementation of tensor.unpack as a consumer.
   FailureOr<TilingResult> getTiledImplementationFromOperandTiles(
       Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
       ArrayRef<SmallVector<OpFoldResult>> allOffsets,
       ArrayRef<SmallVector<OpFoldResult>> allSizes) const {
+    return getTiledImplementationFromOperandTiles(op, b, operandNumbers,
+                                                  allOffsets, allSizes,
+                                                  /*innerTileAlignments=*/{});
+  }
+
+  /// Method to return the tiled implementation of linalg.unpack as a consumer.
+  FailureOr<TilingResult> getTiledImplementationFromOperandTiles(
+      Operation *op, OpBuilder &b, ArrayRef<unsigned> operandNumbers,
+      ArrayRef<SmallVector<OpFoldResult>> allOffsets,
+      ArrayRef<SmallVector<OpFoldResult>> allSizes,
+      ArrayRef<InnerTileAlignment> innerTileAlignments) const {
     if (operandNumbers.size() != 1 || operandNumbers[0] != 0) {
       LLVM_DEBUG({ llvm::dbgs() << "unhandled operands for consumer fusion"; });
       return failure();
@@ -1659,13 +1803,37 @@ struct UnPackOpTiling
     ArrayRef<OpFoldResult> offsets(allOffsets[0]);
     ArrayRef<OpFoldResult> sizes(allSizes[0]);
 
-    // tensor.unpack op is fusible (as a consumer) only if inner dims are not
-    // tiled.
+    // linalg.unpack op is fusible (as a consumer) only if the inner dims are
+    // not tiled, i.e. each inner-dim loop tile size equals the inner tile size.
+    // The caller may assert this per inner dim via InnerTileAlignment::Equal;
+    // otherwise we require a statically-provable equality.
     int64_t numTiles = unPackOp.getInnerDimsPos().size();
-    for (auto iter :
-         llvm::zip_equal(unPackOp.getMixedTiles(), sizes.take_back(numTiles))) {
-      if (!isEqualConstantIntOrValue(std::get<0>(iter), std::get<1>(iter)))
-        return failure();
+    ArrayRef<int64_t> innerDimsPos = unPackOp.getInnerDimsPos();
+    SmallVector<OpFoldResult> mixedTiles = unPackOp.getMixedTiles();
+    ArrayRef<OpFoldResult> innerSizes = sizes.take_back(numTiles);
+    for (int64_t i = 0; i < numTiles; ++i) {
+      // `innerTileAlignments` is indexed by the unpack iteration domain (the
+      // dest dims); the i-th inner tile lives on dest dim `innerDimsPos[i]`.
+      int64_t destDim = innerDimsPos[i];
+      bool hintedEqual =
+          destDim < static_cast<int64_t>(innerTileAlignments.size()) &&
+          innerTileAlignments[destDim] == InnerTileAlignment::Equal;
+      // The hint is the source of truth: honor a caller `Equal` assertion. When
+      // both sizes are also statically known, assert the hint agrees with them
+      // (a contradicting hint is a caller bug) rather than silently ignoring
+      // it. Without an `Equal` hint, require a statically-provable equality
+      // (the inner dim must not be tiled).
+      if (hintedEqual) {
+        assert((!getConstantIntValue(mixedTiles[i]) ||
+                !getConstantIntValue(innerSizes[i]) ||
+                isEqualConstantIntOrValue(mixedTiles[i], innerSizes[i])) &&
+               "InnerTileAlignment::Equal contradicts statically known tile "
+               "sizes");
+        continue;
+      }
+      if (isEqualConstantIntOrValue(mixedTiles[i], innerSizes[i]))
+        continue;
+      return failure();
     }
 
     Location loc = unPackOp.getLoc();
diff --git a/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
index 124a63281a37c..782412557fe27 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorTilingInterfaceImpl.cpp
@@ -22,6 +22,11 @@ using namespace mlir::tensor;
 namespace {
 
 struct PadOpTiling : public TilingInterface::ExternalModel<PadOpTiling, PadOp> {
+  using Base = TilingInterface::ExternalModel<PadOpTiling, PadOp>;
+  // Inherit the defaulted hint-bearing overloads; this op does not require the
+  // hint.
+  using Base::generateResultTileValue;
+  using Base::getTiledImplementation;
 
   SmallVector<utils::IteratorType> getLoopIteratorTypes(Operation *op) const {
     auto padOp = cast<PadOp>(op);
diff --git a/mlir/lib/Interfaces/TilingInterface.cpp b/mlir/lib/Interfaces/TilingInterface.cpp
index 67ddb5b1c1c50..4183d4fd59cd6 100644
--- a/mlir/lib/Interfaces/TilingInterface.cpp
+++ b/mlir/lib/Interfaces/TilingInterface.cpp
@@ -12,6 +12,28 @@
 
 #include "mlir/Interfaces/TilingInterface.h"
 
+#include "llvm/ADT/SmallVectorExtras.h"
+
 using namespace mlir;
 
+LogicalResult mlir::verifyInnerTileAlignments(Operation *op,
+                                              ArrayRef<int64_t> alignments) {
+  for (int64_t a : alignments)
+    if (!isValidInnerTileAlignment(a))
+      return op->emitOpError()
+             << "expected inner_tile_alignments entries to be one of 0 "
+                "(Unknown), 1 (Multiple) or 2 (Equal), but got "
+             << a;
+  return success();
+}
+
+SmallVector<InnerTileAlignment>
+mlir::convertInnerTileAlignments(ArrayRef<int64_t> alignments) {
+  return llvm::map_to_vector(alignments, [](int64_t v) {
+    assert(isValidInnerTileAlignment(v) &&
+           "invalid InnerTileAlignment; should be rejected by the verifier");
+    return static_cast<InnerTileAlignment>(v);
+  });
+}
+
 #include "mlir/Interfaces/TilingInterface.cpp.inc"
diff --git a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
index 20d72a2fd09ca..f241e6f05185b 100644
--- a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
@@ -1237,6 +1237,12 @@ SmallVector<utils::IteratorType> TilingNoDpsOp::getLoopIteratorTypes() {
   return {};
 }
 
+FailureOr<TilingResult> TilingNoDpsOp::getTiledImplementation(
+    OpBuilder &builder, ArrayRef<OpFoldResult> offsets,
+    ArrayRef<OpFoldResult> sizes, ArrayRef<InnerTileAlignment>) {
+  return getTiledImplementation(builder, offsets, sizes);
+}
+
 FailureOr<TilingResult>
 TilingNoDpsOp::getTiledImplementation(OpBuilder &builder,
                                       ArrayRef<OpFoldResult> offsets,
diff --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
index f6c845fefc915..0ce66a086186d 100644
--- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
@@ -11731,6 +11731,7 @@ cc_library(
         ":Support",
         ":TilingInterfaceIncGen",
         ":ViewLikeInterface",
+        "//llvm:Support",
     ],
 )
 

>From 998af51e748cac344d4c88c28baecf8b1d268ea4 Mon Sep 17 00:00:00 2001
From: Ege Beysel <beyselege at gmail.com>
Date: Wed, 10 Jun 2026 17:07:06 +0000
Subject: [PATCH 2/4] feat(mlir/scf): thread InnerTileAlignment through the SCF
 tiling/fusion driver

Forward the optional InnerTileAlignment hint to the hint-bearing TilingInterface
overloads from the SCF tiling/fusion driver. It is passed as a defaulted
argument to tileUsingSCF and tileConsumerAndFuseProducersUsingSCF (and the
existing tileAndFuseProducerOfSlice / tileAndFuseConsumer entry points). As a
pack/unpack-specific caller assertion the hint is kept off the general
SCFTilingOptions struct and passed explicitly where pack/unpack tiling/fusion
happens.

Signed-off-by: Ege Beysel <beyselege at gmail.com>
---
 .../SCF/Transforms/TileUsingInterface.h       | 60 ++++++++++++++++---
 .../Dialect/Tensor/Transforms/Transforms.h    | 12 ++--
 .../SCF/Transforms/TileUsingInterface.cpp     | 60 ++++++++++++++-----
 .../SwapExtractSliceWithProducerPatterns.cpp  | 11 ++--
 4 files changed, 111 insertions(+), 32 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 0005fad3d5c01..0bd5d19b136d0 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -31,6 +31,17 @@ namespace scf {
 using SCFTileSizeComputationFunction =
     std::function<SmallVector<OpFoldResult>(OpBuilder &, Operation *)>;
 
+/// Computation function returning, for the op currently being tiled or fused,
+/// the per-iteration-domain-dimension `InnerTileAlignment` array.
+///`tileSizes` holds the given tile sizes for pure tiling
+/// and is empty for fusion; `slices` holds the `tensor.extract_slice` (producer
+/// fusion) or `tensor.insert_slice`/`tensor.parallel_insert_slice`s (consumer
+/// fusion) being fused, and is empty for pure tiling. Only pack/unpack
+/// implementations consult the result; every other op ignores it.
+using InnerTileAlignmentFnTy = std::function<SmallVector<InnerTileAlignment>(
+    TilingInterface op, ArrayRef<OpFoldResult> tileSizes,
+    ArrayRef<Operation *> slices)>;
+
 /// Options to use to control tiling.
 struct SCFTilingOptions {
   /// Specify which loop construct to use for tile and fuse.
@@ -65,6 +76,26 @@ struct SCFTilingOptions {
     return *this;
   }
 
+  /// Optional control function returning, per tiled/fused op, the inner tile
+  /// alignment hints.
+  InnerTileAlignmentFnTy innerTileAlignmentFn = nullptr;
+  SCFTilingOptions &setInnerTileAlignmentFn(InnerTileAlignmentFnTy fn) {
+    innerTileAlignmentFn = std::move(fn);
+    return *this;
+  }
+
+  /// Sets a constant `innerTileAlignmentFn` that returns `alignments` for every
+  /// op, ignoring which op is being tiled or fused. Use when a single fixed
+  /// array is correct for the whole tiling/fusion (e.g. tiling a single op).
+  SCFTilingOptions &
+  setInnerTileAlignments(ArrayRef<InnerTileAlignment> alignments) {
+    SmallVector<InnerTileAlignment> fixed = llvm::to_vector(alignments);
+    innerTileAlignmentFn =
+        [fixed = std::move(fixed)](TilingInterface, ArrayRef<OpFoldResult>,
+                                   ArrayRef<Operation *>) { return fixed; };
+    return *this;
+  }
+
   //-------------------------------------------------------------------------//
   // Options related to tiling using `scf.forall`.
   //-------------------------------------------------------------------------//
@@ -287,20 +318,26 @@ struct SCFTileAndFuseOptions {
   std::optional<FrozenRewritePatternSet> cleanupPatterns = std::nullopt;
 };
 
-/// Fuse the producer of the source of `candidateSliceOp` by computing the
-/// required slice of the producer in-place.  Note that the method
-/// replaces the uses of `candidateSliceOp` with the tiled and fused producer
-/// value but does not delete the slice operation.
+/// Result of fusing the producer of the source of a `tensor.extract_slice`.
 struct SCFFuseProducerOfSliceResult {
   OpResult origProducer;       // Original untiled producer.
   Value tiledAndFusedProducer; // Tile and fused producer value.
   SmallVector<Operation *> tiledOps;
   SmallVector<Operation *> generatedSlices;
 };
+/// Fuse the producer of the source of `candidateSliceOp` by computing the
+/// required slice of the producer in-place.  Note that the method
+/// replaces the uses of `candidateSliceOp` with the tiled and fused producer
+/// value but does not delete the slice operation.
+///
+/// When the fused producer is a `linalg.pack`/`linalg.unpack`, `fn` (if
+/// non-null) is invoked with the producer and `candidateSliceOp` to obtain the
+/// inner-tile alignment hint in the producer's own iteration domain.
 std::optional<SCFFuseProducerOfSliceResult>
 tileAndFuseProducerOfSlice(RewriterBase &rewriter,
                            tensor::ExtractSliceOp candidateSliceOp,
-                           MutableArrayRef<LoopLikeOpInterface> loops);
+                           MutableArrayRef<LoopLikeOpInterface> loops,
+                           const InnerTileAlignmentFnTy &fn = nullptr);
 
 /// Reconstruct the fused producer from within the tiled-and-fused code. Based
 /// on the slice of the producer computed in place it is possible that within
@@ -426,18 +463,27 @@ struct SCFFuseConsumerOfSliceResult {
   SmallVector<OpOperand *> tiledAndFusedConsumerOperands;
   SmallVector<Operation *> tiledOps;
 };
+/// When the consumer is a `linalg.pack`/`linalg.unpack`, `fn` (if non-null) is
+/// invoked with the consumer and `candidateSlices` to obtain the inner-tile
+/// alignment hint in the consumer's own iteration domain.
 FailureOr<scf::SCFFuseConsumerOfSliceResult>
 tileAndFuseConsumerOfSlices(RewriterBase &rewriter,
                             ArrayRef<Operation *> candidateSlices,
-                            MutableArrayRef<LoopLikeOpInterface> loops);
+                            MutableArrayRef<LoopLikeOpInterface> loops,
+                            const InnerTileAlignmentFnTy &fn = nullptr);
 
 /// Fuse the `consumer` operation into the loop nest provided by `loops`.
 /// The transformation looks for operands in the `consumer` that are defined
 /// by the outermost loop of the loop nest in `loops`. The nested loop is
 /// expected to have the structure of the loops generated through tiling.
+///
+/// When the consumer is a `linalg.pack`/`linalg.unpack`, `fn` (if non-null) is
+/// invoked with the `consumer` and the fused slices to obtain the inner-tile
+/// alignment hint in the consumer's own iteration domain.
 FailureOr<scf::SCFFuseConsumerOfSliceResult>
 tileAndFuseConsumer(RewriterBase &rewriter, Operation *consumer,
-                    MutableArrayRef<LoopLikeOpInterface> loops);
+                    MutableArrayRef<LoopLikeOpInterface> loops,
+                    const InnerTileAlignmentFnTy &fn = nullptr);
 
 /// Method to lower an `op` that implements the `TilingInterface` to
 /// loops/scalars.
diff --git a/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
index 093393eca7436..341b3b1dc3472 100644
--- a/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Tensor/Transforms/Transforms.h
@@ -11,6 +11,7 @@
 
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/Interfaces/TilingInterface.h"
 #include "mlir/Interfaces/ViewLikeInterface.h"
 
 namespace mlir {
@@ -29,7 +30,8 @@ namespace tensor {
 /// transform dialect that control is done within the transform dialect. Other
 /// use cases can inherit from this pattern and add necessary controls.
 FailureOr<TilingResult> replaceExtractSliceWithTiledProducer(
-    OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producerOp);
+    OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producerOp,
+    ArrayRef<InnerTileAlignment> innerTileAlignments = {});
 
 /// Method to swap `tensor.insert_slice`s with their consumers when the
 /// consumer implements the `TilingInterface`. The size of `sliceOps` and
@@ -37,10 +39,10 @@ FailureOr<TilingResult> replaceExtractSliceWithTiledProducer(
 /// `consumerOperands` represents a use of the the corresponding
 /// entry in `sliceOps` in the consumer. All entries of `consumerOperands` is
 /// expected to be uses in the same consumer.
-FailureOr<TilingResult>
-replaceInsertSlicesWithTiledConsumer(OpBuilder &builder,
-                                     ArrayRef<tensor::InsertSliceOp> sliceOps,
-                                     ArrayRef<OpOperand *> consumerOperands);
+FailureOr<TilingResult> replaceInsertSlicesWithTiledConsumer(
+    OpBuilder &builder, ArrayRef<tensor::InsertSliceOp> sliceOps,
+    ArrayRef<OpOperand *> consumerOperands,
+    ArrayRef<InnerTileAlignment> innerTileAlignments = {});
 
 //===----------------------------------------------------------------------===//
 // Populate functions.
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index 963e2d6d4f37b..ad4aff893a03a 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -824,9 +824,11 @@ getTiledImplementation(RewriterBase &rewriter, TilingInterface op,
                        ArrayRef<OpFoldResult> sizes, ValueRange ivs,
                        ArrayRef<OpFoldResult> numThreads,
                        ArrayRef<OpFoldResult> givenTileSizes,
+                       ArrayRef<InnerTileAlignment> innerTileAlignments,
                        const SetVector<unsigned> &reductionDims) {
   if (reductionStrategy == ReductionTilingStrategy::FullReduction) {
-    return op.getTiledImplementation(rewriter, offsets, sizes);
+    return op.getTiledImplementation(rewriter, offsets, sizes,
+                                     innerTileAlignments);
   }
 
   auto redOp = dyn_cast<PartialReductionOpInterface>(op.getOperation());
@@ -1190,10 +1192,15 @@ mlir::scf::tileUsingSCF(RewriterBase &rewriter, TilingInterface op,
     }
 
     // 5c. Tile the cloned operation.
-    tilingResult =
-        getTiledImplementation(rewriter, clonedOp, options.reductionStrategy,
-                               regionIterArgs, tileOffsetsVec, tileSizesVec,
-                               ivs, numThreads, givenTileSizes, reductionDims);
+    SmallVector<InnerTileAlignment> innerTileAlignments =
+        options.innerTileAlignmentFn
+            ? options.innerTileAlignmentFn(clonedOp, givenTileSizes,
+                                           /*slices=*/{})
+            : SmallVector<InnerTileAlignment>{};
+    tilingResult = getTiledImplementation(
+        rewriter, clonedOp, options.reductionStrategy, regionIterArgs,
+        tileOffsetsVec, tileSizesVec, ivs, numThreads, givenTileSizes,
+        innerTileAlignments, reductionDims);
     if (failed(tilingResult)) {
       rewriter.eraseOp(clonedOp);
       return op.emitOpError("failed to tile operation");
@@ -1346,7 +1353,8 @@ getUntiledProducerFromSliceSource(OpOperand *source,
 std::optional<scf::SCFFuseProducerOfSliceResult>
 mlir::scf::tileAndFuseProducerOfSlice(
     RewriterBase &rewriter, tensor::ExtractSliceOp candidateSliceOp,
-    MutableArrayRef<LoopLikeOpInterface> loops) {
+    MutableArrayRef<LoopLikeOpInterface> loops,
+    const InnerTileAlignmentFnTy &fn) {
   // 1. Get the producer of the source (potentially walking through
   // `iter_args` of nested `scf.for`)
   auto [fusableProducer, destinationInitArg] =
@@ -1356,6 +1364,14 @@ mlir::scf::tileAndFuseProducerOfSlice(
     return std::nullopt;
   unsigned resultNumber = fusableProducer.getResultNumber();
 
+  // Resolve the inner-tile alignment hint for the producer in its own iteration
+  // domain via the control function (consulted only by pack/unpack).
+  SmallVector<InnerTileAlignment> innerTileAlignments;
+  if (fn)
+    if (auto producer = dyn_cast<TilingInterface>(fusableProducer.getOwner()))
+      innerTileAlignments =
+          fn(producer, /*tileSizes=*/{}, {candidateSliceOp.getOperation()});
+
   OpBuilder::InsertionGuard g(rewriter);
   rewriter.setInsertionPoint(candidateSliceOp);
 
@@ -1394,7 +1410,7 @@ mlir::scf::tileAndFuseProducerOfSlice(
   FailureOr<TilingResult> tileAndFuseResult =
       tensor::replaceExtractSliceWithTiledProducer(
           rewriter, clonedCandidateSliceOp,
-          clonedProducerOp->getResult(resultNumber));
+          clonedProducerOp->getResult(resultNumber), innerTileAlignments);
   if (failed(tileAndFuseResult))
     return std::nullopt;
   // Note: Do not delete the candidateSliceOp, since its passed in from the
@@ -1819,8 +1835,8 @@ mlir::scf::tileConsumerAndFuseProducersUsingSCF(
     // values produced by operations that implement the `TilingInterface`.
     // Add these operations to the worklist.
     std::optional<scf::SCFFuseProducerOfSliceResult> fusedResult =
-        tileAndFuseProducerOfSlice(rewriter, worklistItem.candidateSlice,
-                                   loops);
+        tileAndFuseProducerOfSlice(rewriter, worklistItem.candidateSlice, loops,
+                                   options.tilingOptions.innerTileAlignmentFn);
     if (!fusedResult)
       continue;
 
@@ -2206,9 +2222,17 @@ static FailureOr<scf::SCFFuseConsumerOfSliceResult>
 tileAndFuseConsumerOfSlicesImpl(RewriterBase &rewriter, Operation *consumerOp,
                                 ArrayRef<OpOperand *> consumerOpOperands,
                                 ArrayRef<Operation *> candidateSlices,
-                                MutableArrayRef<LoopLikeOpInterface> loops) {
+                                MutableArrayRef<LoopLikeOpInterface> loops,
+                                const mlir::scf::InnerTileAlignmentFnTy &fn) {
   assert(!loops.empty() && "expected loops to be not empty");
 
+  // Resolve the inner-tile alignment hint for the consumer in its own iteration
+  // domain via the control function (consulted only by pack/unpack).
+  SmallVector<InnerTileAlignment> innerTileAlignments;
+  if (fn)
+    if (auto consumer = dyn_cast<TilingInterface>(consumerOp))
+      innerTileAlignments = fn(consumer, /*tileSizes=*/{}, candidateSlices);
+
   // 1. Check assumption for loop with `reorderOperations` disabled.
   if (failed(checkAssumptionForLoop(loops.front(), consumerOp, false))) {
     return rewriter.notifyMatchFailure(
@@ -2282,7 +2306,8 @@ tileAndFuseConsumerOfSlicesImpl(RewriterBase &rewriter, Operation *consumerOp,
   // `operandNumber` with the source of the cloned tensor.insert_slice op.
   FailureOr<TilingResult> tileAndFuseResult =
       tensor::replaceInsertSlicesWithTiledConsumer(rewriter, clonedInsertSlices,
-                                                   clonedOpFusedOperandsList);
+                                                   clonedOpFusedOperandsList,
+                                                   innerTileAlignments);
   if (failed(tileAndFuseResult)) {
     return failure();
   }
@@ -2329,7 +2354,7 @@ tileAndFuseConsumerOfSlicesImpl(RewriterBase &rewriter, Operation *consumerOp,
     SmallVector<OpFoldResult> iterDomainOffsets, iterDomainSizes;
     if (failed(clonedConsumerOp.getIterationDomainTileFromOperandTiles(
             rewriter, operandNumbers, allOffsets, allSizes, iterDomainOffsets,
-            iterDomainSizes))) {
+            iterDomainSizes, innerTileAlignments))) {
       return rewriter.notifyMatchFailure(
           clonedConsumerOp,
           "can't get iter domain position from input position");
@@ -2420,7 +2445,8 @@ tileAndFuseConsumerOfSlicesImpl(RewriterBase &rewriter, Operation *consumerOp,
 FailureOr<scf::SCFFuseConsumerOfSliceResult>
 mlir::scf::tileAndFuseConsumerOfSlices(
     RewriterBase &rewriter, ArrayRef<Operation *> candidateSlices,
-    MutableArrayRef<LoopLikeOpInterface> loops) {
+    MutableArrayRef<LoopLikeOpInterface> loops,
+    const InnerTileAlignmentFnTy &fn) {
   if (candidateSlices.empty()) {
     return rewriter.notifyMatchFailure(
         rewriter.getUnknownLoc(),
@@ -2455,7 +2481,7 @@ mlir::scf::tileAndFuseConsumerOfSlices(
 
   return tileAndFuseConsumerOfSlicesImpl(rewriter, consumerOp,
                                          maybeConsumerOpOperands.value(),
-                                         candidateSlices, loops);
+                                         candidateSlices, loops, fn);
 }
 
 /// For a given `result` of a `forallOp` return the
@@ -2520,7 +2546,8 @@ getProducingInsertSliceLikeOp(OpResult result,
 
 FailureOr<scf::SCFFuseConsumerOfSliceResult>
 mlir::scf::tileAndFuseConsumer(RewriterBase &rewriter, Operation *consumer,
-                               MutableArrayRef<LoopLikeOpInterface> loops) {
+                               MutableArrayRef<LoopLikeOpInterface> loops,
+                               const InnerTileAlignmentFnTy &fn) {
   if (!isa<TilingInterface>(consumer)) {
     return rewriter.notifyMatchFailure(
         consumer, "unhandled consumer that does not implement TilingInterface");
@@ -2565,8 +2592,9 @@ mlir::scf::tileAndFuseConsumer(RewriterBase &rewriter, Operation *consumer,
     }
     candidateSlices.push_back(slice.value());
   }
+
   return tileAndFuseConsumerOfSlicesImpl(
-      rewriter, consumer, consumerFusableOperands, candidateSlices, loops);
+      rewriter, consumer, consumerFusableOperands, candidateSlices, loops, fn);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp b/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
index 7903f3c51b73b..0e0262037f08d 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/SwapExtractSliceWithProducerPatterns.cpp
@@ -23,7 +23,8 @@
 using namespace mlir;
 
 FailureOr<TilingResult> tensor::replaceExtractSliceWithTiledProducer(
-    OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producer) {
+    OpBuilder &builder, tensor::ExtractSliceOp sliceOp, OpResult producer,
+    ArrayRef<InnerTileAlignment> innerTileAlignments) {
   auto producerOp = dyn_cast<TilingInterface>(producer.getOwner());
   if (!producerOp)
     return failure();
@@ -34,7 +35,7 @@ FailureOr<TilingResult> tensor::replaceExtractSliceWithTiledProducer(
 
   FailureOr<TilingResult> tiledResult = producerOp.generateResultTileValue(
       builder, producer.getResultNumber(), sliceOp.getMixedOffsets(),
-      sliceOp.getMixedSizes());
+      sliceOp.getMixedSizes(), innerTileAlignments);
   if (failed(tiledResult))
     return failure();
 
@@ -61,7 +62,8 @@ FailureOr<TilingResult> tensor::replaceExtractSliceWithTiledProducer(
 
 FailureOr<TilingResult> tensor::replaceInsertSlicesWithTiledConsumer(
     OpBuilder &builder, ArrayRef<tensor::InsertSliceOp> sliceOps,
-    ArrayRef<OpOperand *> consumerOperands) {
+    ArrayRef<OpOperand *> consumerOperands,
+    ArrayRef<InnerTileAlignment> innerTileAlignments) {
   if (sliceOps.empty()) {
     LLVM_DEBUG(
         { llvm::dbgs() << "expected candidate slices list to be non-empty"; });
@@ -107,7 +109,8 @@ FailureOr<TilingResult> tensor::replaceInsertSlicesWithTiledConsumer(
   }
   FailureOr<TilingResult> tiledResult =
       consumerOp.getTiledImplementationFromOperandTiles(
-          builder, consumerOperandNums, allOffsets, allSizes);
+          builder, consumerOperandNums, allOffsets, allSizes,
+          innerTileAlignments);
   if (failed(tiledResult))
     return failure();
 

>From 7ec7f9d0a5c9b3746f2f2a8026dd98f7127e81c8 Mon Sep 17 00:00:00 2001
From: Ege Beysel <beyselege at gmail.com>
Date: Wed, 10 Jun 2026 17:07:06 +0000
Subject: [PATCH 3/4] feat(mlir/transform): expose inner_tile_alignments on
 tiling/fusion transform ops

Expose the per-dimension InnerTileAlignment hint as an optional
inner_tile_alignments attribute on transform.structured.tile_using_for,
transform.structured.fuse and transform.structured.fuse_into_containing_op (and
the test consumer-fusion op), forwarding it to the SCF driver.

Signed-off-by: Ege Beysel <beyselege at gmail.com>
---
 .../Linalg/TransformOps/LinalgTransformOps.td | 36 ++++++++++--
 .../include/mlir/Interfaces/TilingInterface.h | 19 +++++++
 .../TransformOps/LinalgTransformOps.cpp       | 36 +++++++++---
 mlir/lib/Interfaces/TilingInterface.cpp       | 56 +++++++++++++++++++
 .../Dialect/Linalg/transform-ops-invalid.mlir | 36 ++++++++++++
 mlir/test/Dialect/Linalg/transform-ops.mlir   | 22 ++++++++
 .../TestTilingInterfaceTransformOps.cpp       | 21 ++++++-
 .../TestTilingInterfaceTransformOps.td        | 12 +++-
 8 files changed, 220 insertions(+), 18 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Linalg/TransformOps/LinalgTransformOps.td b/mlir/include/mlir/Dialect/Linalg/TransformOps/LinalgTransformOps.td
index 7f155c7f75d75..09a6c4c4d652f 100644
--- a/mlir/include/mlir/Dialect/Linalg/TransformOps/LinalgTransformOps.td
+++ b/mlir/include/mlir/Dialect/Linalg/TransformOps/LinalgTransformOps.td
@@ -432,6 +432,12 @@ def FuseOp : Op<Transform_Dialect, "structured.fuse",
     If `apply_cleanup` is true then slice canonicalization is applied between
     fusion steps. If `use_forall` is true then tiling method generates a
     `scf.forall` loop instead of `scf.for` loops.
+
+    The optional `inner_tile_alignments` attribute forwards a per-dimension
+    `InnerTileAlignment` hint to the tiling driver for pack/unpack tiling.
+    These are only used to hint alignment or equality between the pack/unpack
+    operations inner tile sizes, and the loop tile sizes. No other operation
+    needs them.
   }];
 
   let arguments =
@@ -441,6 +447,7 @@ def FuseOp : Op<Transform_Dialect, "structured.fuse",
         Optional<TransformAnyParamTypeOrAnyHandle> : $packed_tile_sizes,
         DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$static_tile_sizes,
         DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$static_tile_interchange,
+        DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$inner_tile_alignments,
         UnitAttr:$apply_cleanup,
         UnitAttr:$use_forall);
   let results = (outs TransformHandleTypeInterface:$transformed,
@@ -477,6 +484,8 @@ def FuseOp : Op<Transform_Dialect, "structured.fuse",
                                                     $static_tile_sizes) |
       `interchange` custom<DynamicIndexList>($tile_interchange, $static_tile_interchange)
     )
+    (`inner_tile_alignments` `=`
+        custom<InnerTileAlignmentArray>($inner_tile_alignments)^)?
     attr-dict `:` functional-type(operands, results)
   }];
   let hasVerifier = 1;
@@ -536,14 +545,24 @@ def FuseIntoContainingOp :
 
     This operation consumes the producer handle.
     This operation only reads the containing op handle.
+
+    The optional `inner_tile_alignments` attribute forwards a per-dimension
+    `InnerTileAlignment` hint to the tiling driver for pack/unpack tiling.
+    These are only used to hint alignment or equality between the pack/unpack
+    operations inner tile sizes, and the loop tile sizes. No other operation
+    needs them.
   }];
 
   let arguments = (ins TransformHandleTypeInterface:$producer_op,
-                       TransformHandleTypeInterface:$containing_op);
+                       TransformHandleTypeInterface:$containing_op,
+                       DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$inner_tile_alignments);
   let results = (outs TransformHandleTypeInterface:$fused_op,
                       TransformHandleTypeInterface:$new_containing_op);
-  let assemblyFormat = "$producer_op `into` $containing_op attr-dict "
-                       " `:` functional-type(operands, results)";
+  let assemblyFormat = "$producer_op `into` $containing_op "
+                       "(`inner_tile_alignments` `=` "
+                       "custom<InnerTileAlignmentArray>($inner_tile_alignments)^)? "
+                       "attr-dict `:` functional-type(operands, results)";
+  let hasVerifier = 1;
 
   let builders = [
     OpBuilder<(ins "Value":$producerOp, "Value":$containingOp)>
@@ -2266,13 +2285,20 @@ def TileUsingForOp : Op<Transform_Dialect, "structured.tile_using_for",
 
     If the internal implementation of tiling for any of the operations fails,
     produces a definite failure.
+
+    The optional `inner_tile_alignments` attribute forwards a per-dimension
+    `InnerTileAlignment` hint to the tiling driver for pack/unpack tiling.
+    These are only used to hint alignment or equality between the pack/unpack
+    operations inner tile sizes, and the loop tile sizes. No other operation
+    needs them.
   }];
 
   let arguments = (ins TransformHandleTypeInterface:$target,
                    Variadic<TransformAnyParamTypeOrAnyHandle>:$dynamic_sizes,
                    DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$static_sizes,
                    DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$interchange,
-                   DefaultValuedOptionalAttr<DenseBoolArrayAttr, "{}">:$scalable_sizes);
+                   DefaultValuedOptionalAttr<DenseBoolArrayAttr, "{}">:$scalable_sizes,
+                   DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$inner_tile_alignments);
   let results = (outs TransformHandleTypeInterface:$tiled_linalg_op,
                       Variadic<TransformHandleTypeInterface>:$loops);
   let builders = [
@@ -2307,6 +2333,8 @@ def TileUsingForOp : Op<Transform_Dialect, "structured.tile_using_for",
         $static_sizes,
         $scalable_sizes)
       (`interchange` `=` $interchange^)?
+      (`inner_tile_alignments` `=`
+          custom<InnerTileAlignmentArray>($inner_tile_alignments)^)?
     attr-dict
     `:` functional-type(operands, results)
   }];
diff --git a/mlir/include/mlir/Interfaces/TilingInterface.h b/mlir/include/mlir/Interfaces/TilingInterface.h
index 8b685a30868e8..d432d926647a2 100644
--- a/mlir/include/mlir/Interfaces/TilingInterface.h
+++ b/mlir/include/mlir/Interfaces/TilingInterface.h
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/OpImplementation.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/Interfaces/ViewLikeInterface.h"
 #include "mlir/Support/LLVM.h"
@@ -99,6 +100,24 @@ LogicalResult verifyInnerTileAlignments(Operation *op,
 SmallVector<InnerTileAlignment>
 convertInnerTileAlignments(ArrayRef<int64_t> alignments);
 
+/// Returns the keyword spelling of an `InnerTileAlignment` (`Unknown`,
+/// `Multiple` or `Equal`) used by the `inner_tile_alignments` assembly syntax.
+StringRef stringifyInnerTileAlignment(InnerTileAlignment alignment);
+
+/// Returns the `InnerTileAlignment` for a keyword spelling, or `std::nullopt`
+/// if `keyword` is not one of `Unknown`, `Multiple` or `Equal`.
+std::optional<InnerTileAlignment>
+symbolizeInnerTileAlignment(StringRef keyword);
+
+/// Custom directive parser/printer for an `inner_tile_alignments` attribute,
+/// rendering the `DenseI64ArrayAttr` as a keyword list, e.g.
+/// `[Equal, Multiple, Unknown]` (see `InnerTileAlignment`). Shared by the
+/// transform ops that carry the hint.
+ParseResult parseInnerTileAlignmentArray(OpAsmParser &parser,
+                                         DenseI64ArrayAttr &alignments);
+void printInnerTileAlignmentArray(OpAsmPrinter &printer, Operation *op,
+                                  DenseI64ArrayAttr alignments);
+
 } // namespace mlir
 
 /// Include the ODS generated interface header files.
diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index f44693096b26b..61b81db1e6a74 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -657,6 +657,7 @@ void transform::FuseOp::build(OpBuilder &builder, OperationState &result,
         /*packed_tile_sizes=*/Value(),
         /*static_tile_sizes=*/staticTileSizesAttr,
         /*static_tile_interchange=*/staticTileInterchangeAttr,
+        /*inner_tile_alignments=*/ArrayRef<int64_t>{},
         /*apply_cleanup=*/applyCleanup,
         /*use_forall=*/useForall);
 }
@@ -757,6 +758,10 @@ transform::FuseOp::apply(transform::TransformRewriter &rewriter,
   tilingOptions = tilingOptions.setTileSizes(mixedTileSizes);
   scf::SCFTileAndFuseOptions tileAndFuseOptions;
   tileAndFuseOptions.tilingOptions = tilingOptions;
+  // Optional caller-asserted pack/unpack inner-tile alignment (see
+  // InnerTileAlignment).
+  tileAndFuseOptions.tilingOptions.setInnerTileAlignments(
+      convertInnerTileAlignments(getInnerTileAlignments()));
 
   if (getApplyCleanup()) {
     MLIRContext *context = rewriter.getContext();
@@ -821,7 +826,7 @@ LogicalResult transform::FuseOp::verify() {
   if (numExpectedLoops != getNumResults() - 1)
     return emitOpError() << "expects " << numExpectedLoops << " loop results";
 
-  return success();
+  return verifyInnerTileAlignments(getOperation(), getInnerTileAlignments());
 }
 
 SmallVector<OpFoldResult> transform::FuseOp::getMixedTileSizes() {
@@ -991,7 +996,8 @@ static bool sameOrEquivalentIterArg(Value src, Value dst) {
 /// results of the `containingOp` or nullptr if there are no dominated uses.
 static std::tuple<SmallVector<Operation *>, Operation *>
 tileAndFuseFirstExtractUse(RewriterBase &rewriter, Diagnostic &diag,
-                           Operation *producerOp, Operation *containingOp) {
+                           Operation *producerOp, Operation *containingOp,
+                           ArrayRef<InnerTileAlignment> innerTileAlignments) {
   LDBG() << "Try to fuse a direct extract use";
   auto tileableProducer = dyn_cast<TilingInterface>(producerOp);
   if (!tileableProducer) {
@@ -1067,7 +1073,7 @@ tileAndFuseFirstExtractUse(RewriterBase &rewriter, Diagnostic &diag,
 
   FailureOr<TilingResult> tileAndFuseResult =
       tileableProducer.generateResultTileValue(rewriter, resultNumber, offsets,
-                                               sizes);
+                                               sizes, innerTileAlignments);
 
   if (failed(tileAndFuseResult)) {
     diag.attachNote(tileableProducer->getLoc())
@@ -1115,7 +1121,7 @@ tileAndFuseFirstExtractUse(RewriterBase &rewriter, Diagnostic &diag,
 static SmallVector<Operation *>
 tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
     RewriterBase &rewriter, Diagnostic &diag, Operation *producerOp,
-    Operation *containingOp) {
+    Operation *containingOp, ArrayRef<InnerTileAlignment> innerTileAlignments) {
   LDBG() << "Try to fuse an extract use through block argument";
 
   auto tileableProducer = dyn_cast<TilingInterface>(producerOp);
@@ -1192,7 +1198,7 @@ tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
   FailureOr<TilingResult> tileAndFuseResult =
       tileableProducerClone.generateResultTileValue(
           rewriter, resultNumber, sliceOpToTile.getMixedOffsets(),
-          sliceOpToTile.getMixedSizes());
+          sliceOpToTile.getMixedSizes(), innerTileAlignments);
   if (failed(tileAndFuseResult)) {
     diag.attachNote(tileableProducer->getLoc())
         << "failed to tile producer op: " << *tileableProducer;
@@ -1268,6 +1274,10 @@ bool transform::FuseIntoContainingOp::allowsRepeatedHandleOperands() {
   return true;
 }
 
+LogicalResult transform::FuseIntoContainingOp::verify() {
+  return verifyInnerTileAlignments(getOperation(), getInnerTileAlignments());
+}
+
 DiagnosedSilenceableFailure
 transform::FuseIntoContainingOp::apply(transform::TransformRewriter &rewriter,
                                        transform::TransformResults &results,
@@ -1282,6 +1292,12 @@ transform::FuseIntoContainingOp::apply(transform::TransformRewriter &rewriter,
   }
   Operation *containingOp = *containingOps.begin();
 
+  // Forward the optional, caller-asserted alignment of a fused pack/unpack op's
+  // inner tiles relative to the loop tile sizes (see InnerTileAlignment) to
+  // each fused producer.
+  SmallVector<InnerTileAlignment> innerTileAlignments =
+      convertInnerTileAlignments(getInnerTileAlignments());
+
   // If nothing to fuse, propagate success.
   if (std::empty(producerOps)) {
     results.set(cast<OpResult>(getFusedOp()), SmallVector<mlir::Operation *>{});
@@ -1332,8 +1348,8 @@ transform::FuseIntoContainingOp::apply(transform::TransformRewriter &rewriter,
     // cases, we can tile/clone once and reuse the value for each use.
     // Futhermore, producers should then be traversed according to a
     // topological sorting.
-    auto [tiledOps, newContainingOp] =
-        tileAndFuseFirstExtractUse(rewriter, diag, producerOp, containingOp);
+    auto [tiledOps, newContainingOp] = tileAndFuseFirstExtractUse(
+        rewriter, diag, producerOp, containingOp, innerTileAlignments);
     if (!tiledOps.empty()) {
       LDBG() << "\nFused a direct extract use\n" << *containingOp;
       fusedOps.append(tiledOps);
@@ -1359,7 +1375,7 @@ transform::FuseIntoContainingOp::apply(transform::TransformRewriter &rewriter,
 
     SmallVector<Operation *> tiledContainingOpOperand =
         tileAndFuseFirstExtractUseThroughContainingOpBlockArgument(
-            rewriter, diag, producerOp, containingOp);
+            rewriter, diag, producerOp, containingOp, innerTileAlignments);
     if (!tiledContainingOpOperand.empty()) {
       LDBG() << "\nFused an extract use through block argument\n"
              << *containingOp;
@@ -3549,7 +3565,7 @@ LogicalResult transform::TileUsingForOp::verify() {
     return emitOpError("expected number of loops to tile (")
            << numExpectedLoops << ") to match number of `loops` results ("
            << getLoops().size() << ")";
-  return success();
+  return verifyInnerTileAlignments(getOperation(), getInnerTileAlignments());
 }
 
 DiagnosedSilenceableFailure
@@ -3679,6 +3695,8 @@ transform::TileUsingForOp::apply(transform::TransformRewriter &rewriter,
     }
 
     tilingOptions.setInterchange(getInterchange());
+    tilingOptions.setInnerTileAlignments(
+        convertInnerTileAlignments(getInnerTileAlignments()));
     FailureOr<scf::SCFTilingResult> maybeTilingResult =
         tileUsingSCF(rewriter, tilingInterface, tilingOptions);
     if (failed(maybeTilingResult))
diff --git a/mlir/lib/Interfaces/TilingInterface.cpp b/mlir/lib/Interfaces/TilingInterface.cpp
index 4183d4fd59cd6..87e0af3b5cbc0 100644
--- a/mlir/lib/Interfaces/TilingInterface.cpp
+++ b/mlir/lib/Interfaces/TilingInterface.cpp
@@ -12,7 +12,9 @@
 
 #include "mlir/Interfaces/TilingInterface.h"
 
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVectorExtras.h"
+#include "llvm/ADT/StringSwitch.h"
 
 using namespace mlir;
 
@@ -36,4 +38,58 @@ mlir::convertInnerTileAlignments(ArrayRef<int64_t> alignments) {
   });
 }
 
+StringRef mlir::stringifyInnerTileAlignment(InnerTileAlignment alignment) {
+  switch (alignment) {
+  case InnerTileAlignment::Unknown:
+    return "Unknown";
+  case InnerTileAlignment::Multiple:
+    return "Multiple";
+  case InnerTileAlignment::Equal:
+    return "Equal";
+  }
+  llvm_unreachable("unknown InnerTileAlignment");
+}
+
+std::optional<InnerTileAlignment>
+mlir::symbolizeInnerTileAlignment(StringRef keyword) {
+  return llvm::StringSwitch<std::optional<InnerTileAlignment>>(keyword)
+      .Case("Unknown", InnerTileAlignment::Unknown)
+      .Case("Multiple", InnerTileAlignment::Multiple)
+      .Case("Equal", InnerTileAlignment::Equal)
+      .Default(std::nullopt);
+}
+
+ParseResult mlir::parseInnerTileAlignmentArray(OpAsmParser &parser,
+                                               DenseI64ArrayAttr &alignments) {
+  SmallVector<int64_t> values;
+  auto parseEntry = [&]() -> ParseResult {
+    StringRef keyword;
+    llvm::SMLoc loc = parser.getCurrentLocation();
+    if (parser.parseKeyword(&keyword))
+      return failure();
+    std::optional<InnerTileAlignment> alignment =
+        symbolizeInnerTileAlignment(keyword);
+    if (!alignment)
+      return parser.emitError(loc)
+             << "expected one of 'Unknown', 'Multiple' or 'Equal', but got '"
+             << keyword << "'";
+    values.push_back(static_cast<int64_t>(*alignment));
+    return success();
+  };
+  if (parser.parseCommaSeparatedList(AsmParser::Delimiter::Square, parseEntry))
+    return failure();
+  alignments = DenseI64ArrayAttr::get(parser.getContext(), values);
+  return success();
+}
+
+void mlir::printInnerTileAlignmentArray(OpAsmPrinter &printer, Operation *,
+                                        DenseI64ArrayAttr alignments) {
+  printer << "[";
+  llvm::interleaveComma(alignments.asArrayRef(), printer, [&](int64_t value) {
+    printer << stringifyInnerTileAlignment(
+        static_cast<InnerTileAlignment>(value));
+  });
+  printer << "]";
+}
+
 #include "mlir/Interfaces/TilingInterface.cpp.inc"
diff --git a/mlir/test/Dialect/Linalg/transform-ops-invalid.mlir b/mlir/test/Dialect/Linalg/transform-ops-invalid.mlir
index 6584596cdfdb2..5a6e0cc62b95c 100644
--- a/mlir/test/Dialect/Linalg/transform-ops-invalid.mlir
+++ b/mlir/test/Dialect/Linalg/transform-ops-invalid.mlir
@@ -100,3 +100,39 @@ transform.sequence failures(propagate) {
   // expected-error at below {{expected '('}}
   %res = transform.structured.generalize %arg0 : !transform.any_op -> !transform.any_op 
 }
+
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !transform.any_op):
+  // expected-error at below {{expected one of 'Unknown', 'Multiple' or 'Equal', but got 'Foo'}}
+  %1, %loop = transform.structured.tile_using_for %arg0 tile_sizes [8] inner_tile_alignments = [Foo] : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
+
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !transform.any_op):
+  // expected-error at below {{expected one of 'Unknown', 'Multiple' or 'Equal', but got 'Bogus'}}
+  %1, %loop = transform.structured.fuse %arg0 tile_sizes [8] inner_tile_alignments = [Bogus] : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
+
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !transform.any_op):
+  %0 = transform.structured.match ops{["linalg.generic"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  %1 = transform.structured.match ops{["tensor.empty"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  // expected-error at below {{expected one of 'Unknown', 'Multiple' or 'Equal', but got 'Nope'}}
+  %fused, %new = transform.structured.fuse_into_containing_op %0 into %1 inner_tile_alignments = [Nope] : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
+
+// -----
+
+transform.sequence failures(propagate) {
+^bb0(%arg0: !transform.any_op):
+  %0 = transform.structured.match ops{["scf.for"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  %1 = transform.structured.match ops{["linalg.unpack"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  // expected-error at below {{expected one of 'Unknown', 'Multiple' or 'Equal', but got 'Xyz'}}
+  %a, %b = transform.test.fuse_consumer %1 into (%0) inner_tile_alignments = [Xyz] : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
diff --git a/mlir/test/Dialect/Linalg/transform-ops.mlir b/mlir/test/Dialect/Linalg/transform-ops.mlir
index 06a89fccd5c38..c92f29b441f72 100644
--- a/mlir/test/Dialect/Linalg/transform-ops.mlir
+++ b/mlir/test/Dialect/Linalg/transform-ops.mlir
@@ -16,6 +16,28 @@ transform.sequence failures(propagate) {
   %2, %3:2 = transform.structured.tile_using_for %0 tile_sizes [0, 5, 3] {test_attr3 = 1 : i64, test_attr4}: (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 }
 
+// Check that the `inner_tile_alignments` hint round-trips as a keyword list
+// (Unknown / Multiple / Equal); see `InnerTileAlignment`.
+transform.sequence failures(propagate) {
+^bb1(%arg0: !transform.any_op):
+  // CHECK: transform.structured.tile_using_for %arg0 tile_sizes [8, 4] inner_tile_alignments = [Equal, Multiple]
+  %0, %1:2 = transform.structured.tile_using_for %arg0 tile_sizes [8, 4] inner_tile_alignments = [Equal, Multiple] : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+}
+
+transform.sequence failures(propagate) {
+^bb1(%arg0: !transform.any_op):
+  // CHECK: transform.structured.fuse %arg0 tile_sizes [8] inner_tile_alignments = [Multiple, Unknown]
+  %0, %1 = transform.structured.fuse %arg0 tile_sizes [8] inner_tile_alignments = [Multiple, Unknown] : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
+
+transform.sequence failures(propagate) {
+^bb1(%arg0: !transform.any_op):
+  %0 = transform.structured.match ops{["linalg.generic"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  %1 = transform.structured.match ops{["tensor.empty"]} in %arg0 : (!transform.any_op) -> !transform.any_op
+  // CHECK: transform.structured.fuse_into_containing_op %{{.*}} into %{{.*}} inner_tile_alignments = [Equal, Equal]
+  %fused, %new = transform.structured.fuse_into_containing_op %0 into %1 inner_tile_alignments = [Equal, Equal] : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+}
+
 transform.sequence failures(propagate) {
 ^bb1(%arg0: !transform.any_op):
   %t = transform.structured.split %arg0 after 42 { dimension = 0 } : !transform.any_op
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index b7cb810cda0e6..9467c925e543c 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -177,12 +177,20 @@ static LogicalResult
 applyFuseConsumer(RewriterBase &rewriter, Operation *transformOp,
                   Operation *consumer,
                   MutableArrayRef<LoopLikeOpInterface> loops,
+                  ArrayRef<InnerTileAlignment> innerTileAlignments,
                   TransformResults &transformResults) {
   SmallVector<Operation *> fusedConsumerOps;
   rewriter.setInsertionPoint(consumer);
 
+  // Drive the SCF helper with a constant control function returning the fixed
+  // `inner_tile_alignments` array for every op (consulted only by pack/unpack).
+  scf::InnerTileAlignmentFnTy alignmentFn =
+      [alignments = llvm::to_vector(innerTileAlignments)](
+          TilingInterface, ArrayRef<OpFoldResult>, ArrayRef<Operation *>) {
+        return alignments;
+      };
   FailureOr<scf::SCFFuseConsumerOfSliceResult> fuseConsumerResults =
-      scf::tileAndFuseConsumer(rewriter, consumer, loops);
+      scf::tileAndFuseConsumer(rewriter, consumer, loops, alignmentFn);
   if (failed(fuseConsumerResults))
     return consumer->emitOpError("failed to fuse consumer of slice");
 
@@ -198,6 +206,10 @@ applyFuseConsumer(RewriterBase &rewriter, Operation *transformOp,
   return success();
 }
 
+LogicalResult transform::TestFuseConsumerOp::verify() {
+  return verifyInnerTileAlignments(getOperation(), getInnerTileAlignments());
+}
+
 DiagnosedSilenceableFailure
 transform::TestFuseConsumerOp::apply(TransformRewriter &rewriter,
                                      TransformResults &transformResults,
@@ -215,8 +227,11 @@ transform::TestFuseConsumerOp::apply(TransformRewriter &rewriter,
     }
     loops.push_back(loopLikeOp);
   }
-  LogicalResult result = applyFuseConsumer(rewriter, getOperation(), consumer,
-                                           loops, transformResults);
+  SmallVector<InnerTileAlignment> innerTileAlignments =
+      convertInnerTileAlignments(getInnerTileAlignments());
+  LogicalResult result =
+      applyFuseConsumer(rewriter, getOperation(), consumer, loops,
+                        innerTileAlignments, transformResults);
   return failed(result) ? DiagnosedSilenceableFailure::definiteFailure()
                         : DiagnosedSilenceableFailure::success();
 }
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
index 8c4f64de47795..efa16212f0fef 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
@@ -89,18 +89,26 @@ def TestFuseConsumerOp : Op<Transform_Dialect, "test.fuse_consumer",
 
     Returns a handle to the consumer operation after fusion and the loops that might be
     modified.
+
+    The optional `inner_tile_alignments` array is supplied to the SCF driver as a
+    constant `scf::InnerTileAlignmentFnTy` control function (returning it for
+    every op, see `InnerTileAlignment`).
   }];
 
-  let arguments = (ins 
+  let arguments = (ins
       TransformHandleTypeInterface:$consumer,
-      Variadic<TransformHandleTypeInterface>:$loops);
+      Variadic<TransformHandleTypeInterface>:$loops,
+      DefaultValuedOptionalAttr<DenseI64ArrayAttr, "{}">:$inner_tile_alignments);
   let results = (outs TransformHandleTypeInterface:$fused_consumer,
       Variadic<TransformHandleTypeInterface>:$result_loops);
 
   let assemblyFormat = [{
     $consumer `into` `(` $loops `)`
+    (`inner_tile_alignments` `=`
+        custom<InnerTileAlignmentArray>($inner_tile_alignments)^)?
     attr-dict `:` functional-type(operands, results)
   }];
+  let hasVerifier = 1;
 }
 
 

>From 87107fff669ebdb51b30abed273decd7dd980837 Mon Sep 17 00:00:00 2001
From: Ege Beysel <beyselege at gmail.com>
Date: Wed, 10 Jun 2026 17:07:06 +0000
Subject: [PATCH 4/4] test(mlir/linalg): cover scalable pack/unpack
 tiling+fusion via InnerTileAlignment

Add tests for standalone tiling and producer/consumer fusion of scalable
linalg.pack/linalg.unpack driven by the inner_tile_alignments hint.

Signed-off-by: Ege Beysel <beyselege at gmail.com>
---
 .../Linalg/scalable-pack-consumer-fusion.mlir | 373 ++++++++++++++++++
 .../Linalg/scalable-pack-producer-fusion.mlir | 111 ++++++
 .../Dialect/Linalg/scalable-pack-tiling.mlir  |  87 ++++
 .../scalable-unpack-consumer-fusion.mlir      | 209 ++++++++++
 .../scalable-unpack-producer-fusion.mlir      | 364 +++++++++++++++++
 .../Linalg/scalable-unpack-tiling.mlir        | 122 ++++++
 6 files changed, 1266 insertions(+)
 create mode 100644 mlir/test/Dialect/Linalg/scalable-pack-consumer-fusion.mlir
 create mode 100644 mlir/test/Dialect/Linalg/scalable-pack-producer-fusion.mlir
 create mode 100644 mlir/test/Dialect/Linalg/scalable-pack-tiling.mlir
 create mode 100644 mlir/test/Dialect/Linalg/scalable-unpack-consumer-fusion.mlir
 create mode 100644 mlir/test/Dialect/Linalg/scalable-unpack-producer-fusion.mlir
 create mode 100644 mlir/test/Dialect/Linalg/scalable-unpack-tiling.mlir

diff --git a/mlir/test/Dialect/Linalg/scalable-pack-consumer-fusion.mlir b/mlir/test/Dialect/Linalg/scalable-pack-consumer-fusion.mlir
new file mode 100644
index 0000000000000..de619d1fb9aa4
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-pack-consumer-fusion.mlir
@@ -0,0 +1,373 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// Consumer fusion - linalg.pack with scalable inner tiles. Producer step (8*vscale)
+// equals the pack inner tile size(8*vscale) on the tiled source dimension, so the
+// outer dim of the fused pack tile is statically 1. This information is passed as
+// an inner tile alignment hint `Equal`.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_equal
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_equal(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // Loop tile size is equal to the inner tile size of the consumer `linalg.pack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //      CHECK:    %[[C8:.*]] = arith.constant 8 : index
+  //      CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //      CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %{{.*}}]
+  // CHECK-SAME:          -> tensor<1x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion with a static producer step (64) and a scalable pack inner
+// tile (8*vscale), hinted `Multiple`. Fusion honors the hint and takes the aligned 
+// (non-equal) path and the outer dim of the fused pack tile is dynamic (`64 ceildiv 8*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<()[s0] -> (64 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c64 = arith.constant 64 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c64 iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %ext_out = tensor.extract_slice %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<64x128xf32>, tensor<64x128xf32>)
+        outs(%ext_out : tensor<64x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<64x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<64x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size 64
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[C64:.*]] = arith.constant 64 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C64]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]]()[%[[C8_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %[[C4_VSCALE]]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
+  // CHECK-SAME:          -> tensor<?x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion - both producer step and pack inner tile are scalable, step (8*vscale)
+// is an integer multiple of the inner tile (4*vscale) but not equal to it. The corresponding
+// `Multiple` hint is passed to the tiling interface. Fusion succeeds via the aligned (non-equal)
+// path, so the outer dim of the fused pack tile stays dynamic (`8*vscale ceildiv 4*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_MIN:.+]] = affine_map<(d0)[s0] -> (-d0 + 256, s0)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned_scalable
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned_scalable(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+  
+  // Loop tile size (8 * vscale) is a multiple of the inner tile size of the consumer `linalg.pack` (4 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c4_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size
+    // 8 * vscale and the inner tile size 4 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[SZ:.*]] = affine.min #[[$MAP_MIN]](%{{.*}})[%[[C8_VSCALE]]]
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]](%[[SZ]])[%[[C4_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C4_VSCALE]], %[[C4_VSCALE]]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
+  // CHECK-SAME:          -> tensor<?x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion (negative): linalg.pack with scalable inner tiles and no alignment hint.
+// The relationship between the loop tile size (8 * vscale) and the inner tile size (8 * vscale)
+// cannot be decided statically. Without a user hint that asserts this, fusion fails.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+func.func @negative_fuse_scalable_pack_consumer_no_hint(
+    %arg0: tensor<?x128xf32>, %arg1: tensor<?x128xf32>,
+    %arg2: tensor<?x128xf32>, %dest: tensor<?x?x?x?xf32>, %ub: index)
+    -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %ub step %c8_vscale iter_args(%out = %arg2) -> (tensor<?x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0, s1] -> (s1 - d0, s0)>(%iv)[%c8_vscale, %ub]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<?x128xf32>
+    scf.yield %inserted : tensor<?x128xf32>
+  }
+
+  // expected-error @below {{'linalg.pack' op failed to fuse consumer of slice}}
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<?x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // No inner_tile_alignments hint passed to signal the equality between the loop tile size
+    // 8 * vscale and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop)
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Consumer fusion with a transposing outer_dims_perm. The hint is read in source
+// (operand) order, so `Equal` on source dim 0 (loop step 8*vscale == inner tile
+// 8*vscale) collapses that outer tile to 1; outer_dims_perm = [1, 0] then places
+// it at result position 1 (`tensor<?x1x?x?xf32>`), not position 0.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_pack_consumer_transposed_outer
+func.func @fuse_pack_consumer_transposed_outer(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    // 256 is not a static multiple of 8*vscale, so a real tiling clamps the
+    // per-iteration tile to affine.min(256 - iv, 8*vscale).
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [1, 0]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //      CHECK:    scf.for {{.*}} step %[[C8_VSCALE]]
+  //      CHECK:      linalg.pack
+  // CHECK-SAME:          outer_dims_perm = [1, 0]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %[[C4_VSCALE]]]
+  // CHECK-SAME:          -> tensor<?x1x?x?xf32>
+  //      CHECK:      scf.yield
+}
diff --git a/mlir/test/Dialect/Linalg/scalable-pack-producer-fusion.mlir b/mlir/test/Dialect/Linalg/scalable-pack-producer-fusion.mlir
new file mode 100644
index 0000000000000..12544b04d8478
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-pack-producer-fusion.mlir
@@ -0,0 +1,111 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_equal -canonicalize -cse | FileCheck %s --check-prefixes=ONES_FUSED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_multiple -canonicalize -cse | FileCheck %s --check-prefixes=ONES_UNFUSED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_nohint -canonicalize -cse | FileCheck %s --check-prefixes=ONES_UNFUSED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=inner_equal -canonicalize -cse | FileCheck %s --check-prefixes=INNER
+// RUN: mlir-opt %s --transform-interpreter=entry-point=inner_multiple -canonicalize -cse | FileCheck %s --check-prefixes=INNER
+// RUN: mlir-opt %s --transform-interpreter=entry-point=inner_nohint -canonicalize -cse | FileCheck %s --check-prefixes=INNER
+
+// Producer fusion of a scalable `linalg.pack` into a tiled elementwise consumer.
+// A pack producer can only be fused when the consumer requests full inner tiles.
+//
+// The `inner_tile_alignments` hint is a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>
+func.func @pack_elemwise(%src: tensor<?x?xf32>, %pack_dest: tensor<?x?x?x?xf32>,
+    %out: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %pack = linalg.pack %src inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %pack_dest
+      : tensor<?x?xf32> -> tensor<?x?x?x?xf32>
+  %generic = linalg.generic {indexing_maps = [#map, #map],
+                             iterator_types = ["parallel", "parallel", "parallel", "parallel"]}
+      ins(%pack : tensor<?x?x?x?xf32>) outs(%out : tensor<?x?x?x?xf32>) {
+    ^bb0(%in: f32, %o: f32):
+      %e = math.exp %in : f32
+      linalg.yield %e : f32
+  } -> tensor<?x?x?x?xf32>
+  return %generic : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Tile the consumer's outer dims by 1. `Equal` -> the pack fuses.
+  transform.named_sequence @ones_equal(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [1, 1, 0, 0] inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // ONES_FUSED-LABEL: func.func @pack_elemwise
+  //   ONES_FUSED-DAG:    %[[C1:.*]] = arith.constant 1 : index
+  //       ONES_FUSED:    %[[RES:.*]] = scf.for {{.*}} step %[[C1]]
+  //       ONES_FUSED:      scf.for {{.*}} step %[[C1]]
+  // The pack is fused into the loop: it reads a source slice and produces a
+  // collapsed 1x1 outer tile that the generic consumes directly.
+  //       ONES_FUSED:        %[[SRC:.*]] = tensor.extract_slice %{{.*}} : tensor<?x?xf32> to tensor<?x?xf32>
+  //       ONES_FUSED:        %[[PACK:.*]] = linalg.pack %[[SRC]]
+  //  ONES_FUSED-SAME:            -> tensor<1x1x?x?xf32>
+  //       ONES_FUSED:        linalg.generic {{.*}} ins(%[[PACK]]
+  //       ONES_FUSED:    return %[[RES]]
+
+  // Same input, `Multiple` hint: not sufficient, the pack is not fused.
+  transform.named_sequence @ones_multiple(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [1, 1, 0, 0] inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // Same input, no hint: the pack is not fused.
+  transform.named_sequence @ones_nohint(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [1, 1, 0, 0]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // ONES_UNFUSED-LABEL: func.func @pack_elemwise
+  // Without `Equal` the full-inner-tile requirement cannot be proven (scalable),
+  // so the pack stays outside the loop as a full pack and the tiled generic reads
+  // slices of its result.
+  //       ONES_UNFUSED:    %[[PACK:.*]] = linalg.pack
+  //  ONES_UNFUSED-SAME:        -> tensor<?x?x?x?xf32>
+  //       ONES_UNFUSED:    %[[RES:.*]] = scf.for
+  //       ONES_UNFUSED:      scf.for
+  //       ONES_UNFUSED:        tensor.extract_slice %[[PACK]]
+  //       ONES_UNFUSED:        linalg.generic
+  //       ONES_UNFUSED:    return %[[RES]]
+
+  // Tile the packed inner-tile dims. The pack is never fused (its inner tiles are not tileable),
+  // regardless of the hint - only the generic is tiled. All three sequences produce the same IR.
+
+  transform.named_sequence @inner_equal(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [0, 0, 8, 4] inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @inner_multiple(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [0, 0, 8, 4] inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @inner_nohint(%arg1: !transform.any_op {transform.readonly}) {
+    %g = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %t, %loops:2 = transform.structured.fuse %g tile_sizes [0, 0, 8, 4]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // INNER-LABEL: func.func @pack_elemwise
+  //       INNER:    %[[PACK:.*]] = linalg.pack
+  //  INNER-SAME:        -> tensor<?x?x?x?xf32>
+  //       INNER:    scf.for
+  //       INNER:      scf.for
+  //       INNER:        tensor.extract_slice %[[PACK]]
+  //       INNER:        linalg.generic
+}
diff --git a/mlir/test/Dialect/Linalg/scalable-pack-tiling.mlir b/mlir/test/Dialect/Linalg/scalable-pack-tiling.mlir
new file mode 100644
index 0000000000000..bc1e98fbc91c7
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-pack-tiling.mlir
@@ -0,0 +1,87 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_equal -canonicalize -cse | FileCheck %s --check-prefixes=ONES
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_multiple -canonicalize -cse | FileCheck %s --check-prefixes=ONES
+// RUN: mlir-opt %s --transform-interpreter=entry-point=ones_nohint -canonicalize -cse | FileCheck %s --check-prefixes=ONES
+// RUN: mlir-opt %s --transform-interpreter=entry-point=twofour_equal -canonicalize -cse | FileCheck %s --check-prefixes=ALIGNED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=twofour_multiple -canonicalize -cse | FileCheck %s --check-prefixes=ALIGNED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=twofour_nohint -canonicalize -cse | FileCheck %s --check-prefixes=ALIGNED
+
+// Tiling of a scalable `linalg.pack`. A pack's tiling interface iterates over the
+// packed (destination) OUTER dims only,  the inner tiles are not part of the
+// iteration domain - so tiling is completely independent of any
+// `inner_tile_alignments` hint.
+
+func.func @pack(%src: tensor<128x256xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c8 = arith.constant 8 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %pack = linalg.pack %src inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %dest
+      : tensor<128x256xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Tile the two outer dims by 1. Hint-independent, the three sequences below produce identical IR.
+  transform.named_sequence @ones_equal(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [1, 1] inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @ones_multiple(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [1, 1] inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @ones_nohint(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [1, 1]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // ONES-LABEL: func.func @pack
+  //   ONES-DAG:    %[[C1:.*]] = arith.constant 1 : index
+  //       ONES:    %[[RES:.*]] = scf.for {{.*}} step %[[C1]]
+  //       ONES:      scf.for {{.*}} step %[[C1]]
+  //       ONES:        %[[PACK:.*]] = linalg.pack
+  //  ONES-SAME:            tensor<?x?xf32> -> tensor<1x1x?x?xf32>
+  //       ONES:        tensor.insert_slice %[[PACK]]
+  //       ONES:      scf.yield
+  //       ONES:    scf.yield
+  //       ONES:    return %[[RES]]
+
+  // Tile the two outer dims by 2 and 4. Hint-independent.
+  transform.named_sequence @twofour_equal(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [2, 4] inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @twofour_multiple(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [2, 4] inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  transform.named_sequence @twofour_nohint(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.pack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [2, 4]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // ALIGNED-LABEL: func.func @pack
+  // ALIGNED-DAG:    %[[C2:.*]] = arith.constant 2 : index
+  // ALIGNED-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //     ALIGNED:    %[[RES:.*]] = scf.for {{.*}} step %[[C2]]
+  //     ALIGNED:      scf.for {{.*}} step %[[C4]]
+  //     ALIGNED:        %[[SRC:.*]] = tensor.extract_slice %{{.*}} : tensor<128x256xf32> to tensor<?x?xf32>
+  //     ALIGNED:        %[[PACK:.*]] = linalg.pack %[[SRC]]
+  // ALIGNED-SAME:            tensor<?x?xf32> -> tensor<?x?x?x?xf32>
+  //     ALIGNED:        tensor.insert_slice %[[PACK]]
+  //     ALIGNED:      scf.yield
+  //     ALIGNED:    scf.yield
+  //     ALIGNED:    return %[[RES]]
+}
diff --git a/mlir/test/Dialect/Linalg/scalable-unpack-consumer-fusion.mlir b/mlir/test/Dialect/Linalg/scalable-unpack-consumer-fusion.mlir
new file mode 100644
index 0000000000000..f0605d36780ba
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-unpack-consumer-fusion.mlir
@@ -0,0 +1,209 @@
+// RUN: mlir-opt %s -transform-interpreter -split-input-file -canonicalize -cse | FileCheck %s --check-prefixes=CHECK
+// RUN: not mlir-opt %s --transform-interpreter=entry-point=no_hint -split-input-file 2>&1 | FileCheck %s --check-prefixes=NO_HINT
+// RUN: not mlir-opt %s --transform-interpreter=entry-point=multiple -split-input-file 2>&1 | FileCheck %s --check-prefixes=MULTIPLE
+
+// Consumer fusion of a scalable `linalg.unpack` into an `scf.for` loop.
+// The `inner_tile_alignments` hint is a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// The loop tile size on the tiled (inner-tile) dimension is 8 * vscale, the same
+// scalable value as the unpack inner tile. The SAME input IR is fused three ways:
+// with the `Equal` hint, with the `Multiple` hint, and without a hint.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+func.func @fuse_scalable_unpack_consumer(
+    %arg0: tensor<32x?xf32>, %arg1: tensor<32x?xf32>,
+    %arg2: tensor<32x?xf32>) -> tensor<?xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c8 = arith.constant 8 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %dim1 = tensor.dim %arg2, %c1 : tensor<32x?xf32>
+
+  // Loop tile size is equal to the inner tile size of the consumer
+  // `linalg.unpack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %dim1 step %c8_vscale iter_args(%out = %arg2) -> (tensor<32x?xf32>) {
+    %sz = affine.min affine_map<(d0)[s0, s1] -> (s1 - d0, s0)>(%iv)[%c8_vscale, %dim1]
+    %ext_a = tensor.extract_slice %arg0[0, %iv] [32, %sz] [1, 1]
+        : tensor<32x?xf32> to tensor<32x?xf32>
+    %ext_b = tensor.extract_slice %arg1[0, %iv] [32, %sz] [1, 1]
+        : tensor<32x?xf32> to tensor<32x?xf32>
+    %ext_out = tensor.extract_slice %out[0, %iv] [32, %sz] [1, 1]
+        : tensor<32x?xf32> to tensor<32x?xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<32x?xf32>, tensor<32x?xf32>)
+        outs(%ext_out : tensor<32x?xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<32x?xf32>
+    %inserted = tensor.insert_slice %computed into %out[0, %iv] [32, %sz] [1, 1]
+        : tensor<32x?xf32> into tensor<32x?xf32>
+    scf.yield %inserted : tensor<32x?xf32>
+  }
+
+  %output = tensor.empty(%dim1) : tensor<?xf32>
+  %unpack = linalg.unpack %0 outer_dims_perm = [0]
+      inner_dims_pos = [0] inner_tiles = [%c8_vscale]
+      into %output : tensor<32x?xf32> -> tensor<?xf32>
+  return %unpack : tensor<?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Intended fusion: the loop tile equals the inner tile, asserted via `Equal`.
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %a, %b = transform.test.fuse_consumer %unpack into (%loop) inner_tile_alignments = [Equal]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   CHECK-LABEL: func.func @fuse_scalable_unpack_consumer
+  //    CHECK-SAME:     %[[ARG0:.+]]: tensor<32x?xf32>, %[[ARG1:.+]]: tensor<32x?xf32>, %[[ARG2:.+]]: tensor<32x?xf32>
+  //         CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //         CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %{{.*}} : index
+  //         CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  //    CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %{{.*}})
+  //         CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //         CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[GENERIC]]
+  //    CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]]]
+  //         CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //         CHECK:    return %[[RES]]#1
+
+  // No hint: fusion fails (scalable loop tile vs scalable inner tile, undecidable).
+  transform.named_sequence @no_hint(%arg1: !transform.any_op {transform.readonly}) {
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %a, %b = transform.test.fuse_consumer %unpack into (%loop)
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // NO_HINT: 'linalg.unpack' op failed to fuse consumer of slice
+
+  // `Multiple` hint: insufficient for an unpack consumer's inner dim, fusion fails.
+  transform.named_sequence @multiple(%arg1: !transform.any_op {transform.readonly}) {
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %a, %b = transform.test.fuse_consumer %unpack into (%loop) inner_tile_alignments = [Multiple]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // MULTIPLE: 'linalg.unpack' op failed to fuse consumer of slice
+}
+
+// -----
+
+// A 3-op dispatch tiled from the mmt4d root, fusing its consumers, with an
+// inner-tile alignment hint driving the scalable unpack fusion:
+//
+//   linalg.mmt4d              (root, produces a packed [M, N, M0, N0] layout)
+//   -> linalg.generic         (transposing bias add: [M, N, M0, N0] -> [N, M, N0, M0])
+//   -> linalg.unpack          ([N, M, N0, M0] -> [N*N0, M*M0])
+//
+// The mmt4d is tiled along its scalable inner dim N0 (iteration dim 4) by
+// 8 * vscale; the generic and unpack are fused as consumers. The SAME input IR
+// is fused two ways: with the equal hint and without.
+
+#id4 = affine_map<(m, n, m0, n0) -> (m, n, m0, n0)>
+#tr4 = affine_map<(m, n, m0, n0) -> (n, m, n0, m0)>
+
+// CHECK: #[[$TR:.+]] = affine_map<(d0, d1, d2, d3) -> (d1, d0, d3, d2)>
+func.func @mmt4d_transpose_unpack(
+    %lhs: tensor<2x2x4x2xf32>, %rhs: tensor<2x2x?x2xf32>,
+    %acc: tensor<2x2x4x?xf32>, %bias: tensor<2x2x?x4xf32>,
+    %trinit: tensor<2x2x?x4xf32>, %out: tensor<?x8xf32>) -> tensor<?x8xf32> {
+  %c8 = arith.constant 8 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // 1. mmt4d root: lhs[M,K,M0,K0] x rhs[N,K,N0,K0] -> out[M,N,M0,N0].
+  %mm = linalg.mmt4d ins(%lhs, %rhs : tensor<2x2x4x2xf32>, tensor<2x2x?x2xf32>)
+      outs(%acc : tensor<2x2x4x?xf32>) -> tensor<2x2x4x?xf32>
+
+  // 2. transposing bias add: [M,N,M0,N0] -> [N,M,N0,M0].
+  %tr = linalg.generic {
+      indexing_maps = [#id4, #tr4, #tr4],
+      iterator_types = ["parallel", "parallel", "parallel", "parallel"]}
+      ins(%mm, %bias : tensor<2x2x4x?xf32>, tensor<2x2x?x4xf32>)
+      outs(%trinit : tensor<2x2x?x4xf32>) {
+    ^bb0(%a: f32, %b: f32, %o: f32):
+      %s = arith.addf %a, %b : f32
+      linalg.yield %s : f32
+  } -> tensor<2x2x?x4xf32>
+
+  // 3. unpack: [N,M,N0,M0] -> [N*N0, M*M0] = [?, 8].
+  %unpack = linalg.unpack %tr outer_dims_perm = [0, 1] inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, 4] into %out
+      : tensor<2x2x?x4xf32> -> tensor<?x8xf32>
+  return %unpack : tensor<?x8xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Intended fusion: tile the mmt4d root, fuse the transposing generic and the
+  // unpack, asserting `Equal` on the unpack's tiled (transposed) inner dim.
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %mmt4d = transform.structured.match ops{["linalg.mmt4d"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // Tile the mmt4d root along its scalable inner dim N0 (iteration dim 4).
+    %tiled, %loop = transform.structured.tile_using_for %mmt4d tile_sizes [0, 0, 0, 0, [8], 0]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    %gen = transform.structured.match ops{["linalg.generic"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %fg, %loop2 = transform.test.fuse_consumer %gen into (%loop)
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    %unp = transform.structured.match ops{["linalg.unpack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %fu, %loop3 = transform.test.fuse_consumer %unp into (%loop2) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   CHECK-LABEL: func.func @mmt4d_transpose_unpack
+  //    CHECK-SAME:     %[[LHS:.+]]: tensor<2x2x4x2xf32>, %[[RHS:.+]]: tensor<2x2x?x2xf32>, %[[ACC:.+]]: tensor<2x2x4x?xf32>, %[[BIAS:.+]]: tensor<2x2x?x4xf32>, %[[TRINIT:.+]]: tensor<2x2x?x4xf32>, %[[OUT:.+]]: tensor<?x8xf32>
+  //         CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //         CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %{{.*}} : index
+  //         CHECK:    %[[RES:.*]]:3 = scf.for {{.*}} step %[[C8_VSCALE]]
+  //    CHECK-SAME:        iter_args(%{{.*}} = %[[ACC]], %{{.*}} = %[[TRINIT]], %{{.*}} = %[[OUT]])
+  //         CHECK:      %[[MM:.*]] = linalg.mmt4d
+  //         CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //    CHECK-SAME:          indexing_maps = [#{{.+}}, #[[$TR]], #[[$TR]]]
+  //    CHECK-SAME:          ins(%[[MM]],
+  //         CHECK:      %[[UNPACK:.*]] = linalg.unpack %[[GENERIC]]
+  //    CHECK-SAME:          outer_dims_perm = [0, 1] inner_dims_pos = [0, 1]
+  //    CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], 4]
+  //         CHECK:      scf.yield {{.*}}, {{.*}}, %{{.*}} :
+  //         CHECK:    return %[[RES]]#2
+
+  // No hint on the unpack: unpack fusion fails.
+  transform.named_sequence @no_hint(%arg1: !transform.any_op {transform.readonly}) {
+    %mmt4d = transform.structured.match ops{["linalg.mmt4d"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %mmt4d tile_sizes [0, 0, 0, 0, [8], 0]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    %gen = transform.structured.match ops{["linalg.generic"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %fg, %loop2 = transform.test.fuse_consumer %gen into (%loop)
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    %unp = transform.structured.match ops{["linalg.unpack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %fu, %loop3 = transform.test.fuse_consumer %unp into (%loop2)
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // NO_HINT: 'linalg.unpack' op failed to fuse consumer of slice
+
+  // Dummy entry point so the `multiple` RUN line resolves here.
+  transform.named_sequence @multiple(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/scalable-unpack-producer-fusion.mlir b/mlir/test/Dialect/Linalg/scalable-unpack-producer-fusion.mlir
new file mode 100644
index 0000000000000..96dc4937176fc
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-unpack-producer-fusion.mlir
@@ -0,0 +1,364 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=aligned -split-input-file -canonicalize -cse | FileCheck %s --check-prefixes=ALIGNED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=equal -split-input-file -canonicalize -cse | FileCheck %s --check-prefixes=EQUAL
+// RUN: mlir-opt %s --transform-interpreter=entry-point=unaligned -split-input-file -canonicalize -cse | FileCheck %s --check-prefixes=UNALIGNED
+// RUN: mlir-opt %s -transform-interpreter -split-input-file -canonicalize -cse | FileCheck %s --check-prefixes=CHECK
+
+// Producer fusion of a scalable `linalg.unpack` into a tiled consumer.
+// The `inner_tile_alignments` hint is a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+// The elemwise consumer is tiled three ways from the same input IR: aligned, equal, and unaligned.
+
+// ALIGNED-DAG: #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+func.func @unpack_elemwise_scalable(%arg0: tensor<4x8x?x?xf32>, %arg1: tensor<?x?xf32>, %arg2 : index, %arg3 : index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c16 = arith.constant 16 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  // 16 * vscale is only consumed as a loop tile size by `@aligned`; `@equal` and
+  // `@unaligned` leave it dead and it is folded away.
+  %c16_vscale = arith.muli %c16, %vscale : index
+  %0 = tensor.empty(%arg2, %arg3) : tensor<?x?xf32>
+  %1 = linalg.unpack %arg0 inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %0
+      : tensor<4x8x?x?xf32> -> tensor<?x?xf32>
+  %2 = linalg.exp ins(%1: tensor<?x?xf32>)
+                       outs(%arg1: tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %2 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Aligned: loop tile sizes (16 * vscale, 8 * vscale) are integer multiples of
+  // the inner tiles, asserted via `Multiple`. The outer dims become
+  // `ceilDiv(loop tile, inner tile)`.
+  transform.named_sequence @aligned(%arg1: !transform.any_op {transform.readonly}) {
+    %exp = transform.structured.match ops{["linalg.exp"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %mulis = transform.structured.match ops{["arith.muli"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %h8, %h4, %h16 = transform.split_handle %mulis : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled, %loops:2 = transform.structured.fuse %exp tile_sizes [%h16, %h8] interchange [0, 1]
+      inner_tile_alignments = [Multiple, Multiple]
+      : (!transform.any_op, !transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // ALIGNED-LABEL: func.func @unpack_elemwise_scalable
+  //   ALIGNED-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //   ALIGNED-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //   ALIGNED-DAG:    %[[C16:.*]] = arith.constant 16 : index
+  //   ALIGNED-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //   ALIGNED-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //   ALIGNED-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //   ALIGNED-DAG:    %[[C16_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C16]] : index
+  //       ALIGNED:    %[[RES:.*]] = scf.for {{.*}} step %[[C16_VSCALE]]
+  //       ALIGNED:      scf.for {{.*}} step %[[C8_VSCALE]]
+  //       ALIGNED:        %[[OUTER_DIM0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+  //       ALIGNED:        %[[OUTER_DIM1:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+  //       ALIGNED:        %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER_DIM0]], %[[OUTER_DIM1]], %[[C8_VSCALE]], %[[C4_VSCALE]]]
+  //       ALIGNED:        %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+  //  ALIGNED-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  //   ALIGNED-NOT:         tensor.extract_slice %[[UNPACK]]
+  //       ALIGNED:        linalg.exp ins(%[[UNPACK]]
+  //       ALIGNED:      scf.yield
+  //       ALIGNED:    scf.yield
+  //       ALIGNED:    return %[[RES]]
+
+  // Equal: loop tile sizes (8 * vscale, 4 * vscale) equal the inner tiles,
+  // asserted via `Equal`; the fused unpack's outer dims collapse to 1.
+  transform.named_sequence @equal(%arg1: !transform.any_op {transform.readonly}) {
+    %exp = transform.structured.match ops{["linalg.exp"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %mulis = transform.structured.match ops{["arith.muli"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %h8, %h4, %h16 = transform.split_handle %mulis : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled, %loops:2 = transform.structured.fuse %exp tile_sizes [%h8, %h4] interchange [0, 1]
+      inner_tile_alignments = [Equal, Equal]
+      : (!transform.any_op, !transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   EQUAL-LABEL: func.func @unpack_elemwise_scalable
+  //   EQUAL-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //   EQUAL-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //   EQUAL-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //   EQUAL-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //   EQUAL-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //       EQUAL:    %[[RES:.*]] = scf.for {{.*}} step %[[C8_VSCALE]]
+  //       EQUAL:      scf.for {{.*}} step %[[C4_VSCALE]]
+  //       EQUAL:        %[[UNPACK:.*]] = linalg.unpack
+  //  EQUAL-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+  //   EQUAL-NOT:         tensor.extract_slice %[[UNPACK]]
+  //       EQUAL:        linalg.exp ins(%[[UNPACK]]
+  //       EQUAL:      scf.yield
+  //       EQUAL:    scf.yield
+  //       EQUAL:    return %[[RES]]
+
+  // Unaligned: static tile sizes (7, 5) are not aligned to the scalable inner
+  // tiles and no hint is passed, so the fused unpack over-computes and a trailing
+  // extract_slice recovers the needed slice.
+  transform.named_sequence @unaligned(%arg1: !transform.any_op {transform.readonly}) {
+    %exp = transform.structured.match ops{["linalg.exp"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %tiled, %loops:2 = transform.structured.fuse %exp tile_sizes [7, 5] interchange [0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   UNALIGNED-LABEL: func.func @unpack_elemwise_scalable
+  //   UNALIGNED-DAG:    %[[C7:.*]] = arith.constant 7 : index
+  //   UNALIGNED-DAG:    %[[C5:.*]] = arith.constant 5 : index
+  //       UNALIGNED:    %[[RES:.*]] = scf.for {{.*}} step %[[C7]]
+  //       UNALIGNED:      scf.for {{.*}} step %[[C5]]
+  //       UNALIGNED:        %[[UNPACK:.*]] = linalg.unpack
+  //  UNALIGNED-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  //       UNALIGNED:        %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+  //   UNALIGNED-NOT:        linalg.exp ins(%[[UNPACK]]
+  //       UNALIGNED:        linalg.exp ins(%[[EXTRACT]]
+  //       UNALIGNED:      scf.yield
+  //       UNALIGNED:    scf.yield
+  //       UNALIGNED:    return %[[RES]]
+
+  // Dummy entry point so the default RUN line resolves here.
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+}
+
+// -----
+
+// Fusing a scalable `linalg.unpack` producer into a containing `scf.forall`. The
+// same input IR is fused three ways: aligned, equal, and unaligned.
+
+func.func @fuse_unpack_into_containing(
+    %src: tensor<?x?x?x?xf32>, %unpack_empty: tensor<?x?xf32>,
+    %out: tensor<?x?xf32>, %ub0: index, %ub1: index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %vscale = vector.vscale
+  // Inner tile sizes of the unpack.
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  // Containing-loop tile sizes: numerically the same as the inner tiles, but
+  // deliberately distinct SSA values so the `@aligned` (`Multiple`) path cannot
+  // statically fold `ceilDiv(tile, inner)` to 1 and keeps a dynamic outer dim.
+  %c8_vscale_step = arith.muli %c8, %vscale : index
+  %c4_vscale_step = arith.muli %c4, %vscale : index
+  %unpack = linalg.unpack %src inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %unpack_empty
+      : tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  // Loop tile sizes (8 * vscale, 4 * vscale) equal the inner tile sizes of the
+  // consumer `linalg.unpack` in value.
+  %res = scf.forall (%i, %j) = (0, 0) to (%ub0, %ub1) step (%c8_vscale_step, %c4_vscale_step)
+      shared_outs(%o = %out) -> (tensor<?x?xf32>) {
+    %slice = tensor.extract_slice %unpack[%i, %j] [%c8_vscale_step, %c4_vscale_step] [1, 1]
+        : tensor<?x?xf32> to tensor<?x?xf32>
+    %oslice = tensor.extract_slice %o[%i, %j] [%c8_vscale_step, %c4_vscale_step] [1, 1]
+        : tensor<?x?xf32> to tensor<?x?xf32>
+    %0 = linalg.exp ins(%slice : tensor<?x?xf32>) outs(%oslice : tensor<?x?xf32>) -> tensor<?x?xf32>
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %0 into %o[%i, %j] [%c8_vscale_step, %c4_vscale_step] [1, 1]
+          : tensor<?x?xf32> into tensor<?x?xf32>
+    }
+  }
+  return %res : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Equal tiles: the fused unpack collapses to a 1x1 outer tile.
+  transform.named_sequence @equal(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1 = transform.structured.match ops{["scf.forall"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %fused, %newc = transform.structured.fuse_into_containing_op %0 into %1
+        inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   EQUAL-LABEL: func.func @fuse_unpack_into_containing
+  //       EQUAL:   scf.forall
+  //       EQUAL:     %[[UNPACK:.+]] = linalg.unpack
+  //  EQUAL-SAME:         : tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+  //   EQUAL-NOT:     tensor.extract_slice %[[UNPACK]]
+  //       EQUAL:     linalg.exp ins(%[[UNPACK]]
+
+  // No hint: general (unaligned) tiling with a trailing result slice.
+  transform.named_sequence @unaligned(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1 = transform.structured.match ops{["scf.forall"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %fused, %newc = transform.structured.fuse_into_containing_op %0 into %1
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // UNALIGNED-LABEL: func.func @fuse_unpack_into_containing
+  //     UNALIGNED:   scf.forall
+  //     UNALIGNED:     %[[UNPACK:.+]] = linalg.unpack
+  // UNALIGNED-SAME:         : tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  //     UNALIGNED:     %[[EXTRACTED:.+]] = tensor.extract_slice %[[UNPACK]]
+  // UNALIGNED-NOT:     linalg.exp ins(%[[UNPACK]]
+  //     UNALIGNED:     linalg.exp ins(%[[EXTRACTED]]
+
+  // Aligned: the `Multiple` hints assert the containing loop's tile sizes are
+  // multiples of the inner tiles. Although the loop tile size and inner tile sizes
+  // are equal but distinct SSA values, the equivalence cannot be inferred in the
+  // absence of the `Equal` hint.
+  transform.named_sequence @aligned(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1 = transform.structured.match ops{["scf.forall"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %fused, %newc = transform.structured.fuse_into_containing_op %0 into %1
+        inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //   ALIGNED-LABEL: func.func @fuse_unpack_into_containing
+  //       ALIGNED:   scf.forall
+  // The Multiple hint keeps a dynamic ceilDiv outer dim (step and inner tile are
+  // distinct SSA values, so it does not fold to 1) but needs no result slice.
+  //       ALIGNED:     %[[OUTER:.+]] = affine.apply {{.*}}[%{{.*}}, %{{.*}}]
+  //   ALIGNED-NOT:     linalg.unpack {{.*}} : tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+  //       ALIGNED:     %[[UNPACK:.+]] = linalg.unpack
+  //  ALIGNED-SAME:         : tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  //   ALIGNED-NOT:     tensor.extract_slice %[[UNPACK]]
+  //       ALIGNED:     linalg.exp ins(%[[UNPACK]]
+
+  // Dummy entry point so the default RUN line resolves here.
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+}
+
+// -----
+
+// Producer fusion with a transposing consumer, `inner_tile_alignments` is
+// a caller-asserted hint indexed in the unpack's dest-dim order, so the
+// caller must arrange it for the transpose. Source dim 0 is `Multiple`
+// so  -> outer size is `ceilDiv(loop tile,8 * vscale)` (dynamic); 
+// source dim 1 is `Equal` -> outer size collapses to 1.
+
+// CHECK-DAG: #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+func.func @unpack_transposed_consumer_scalable(%arg0: tensor<2x4x?x?xf32>, %out: tensor<?x?xf32>, %d0: index, %d1: index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c16 = arith.constant 16 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %t0 = arith.muli %c4, %vscale : index
+  %t1 = arith.muli %c16, %vscale : index
+  %0 = tensor.empty(%d0, %d1) : tensor<?x?xf32>
+  %unpack = linalg.unpack %arg0 inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %0
+      : tensor<2x4x?x?xf32> -> tensor<?x?xf32>
+  // Consumer reads %unpack transposed.
+  %1 = linalg.generic {indexing_maps = [affine_map<(d0, d1) -> (d1, d0)>,
+                                        affine_map<(d0, d1) -> (d0, d1)>],
+                       iterator_types = ["parallel", "parallel"]}
+       ins(%unpack : tensor<?x?xf32>) outs(%out : tensor<?x?xf32>) {
+    ^bb0(%in: f32, %o: f32):
+      linalg.yield %in : f32
+  } -> tensor<?x?xf32>
+  return %1 : tensor<?x?xf32>
+}
+// CHECK-LABEL: func.func @unpack_transposed_consumer_scalable
+//  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+//  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+//  CHECK-DAG:    %[[C16:.*]] = arith.constant 16 : index
+//  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+//  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+//  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+//  CHECK-DAG:    %[[C16_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C16]] : index
+//      CHECK:    %[[RES:.*]] = scf.for {{.*}} step %[[C4_VSCALE]]
+//      CHECK:      scf.for {{.*}} step %[[C16_VSCALE]]
+//      CHECK:        %[[OUTER_DIM0:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C8_VSCALE]]]
+//      CHECK:        %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[OUTER_DIM0]], 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+// CHECK-SAME:            tensor<2x4x?x?xf32> to tensor<?x1x?x?xf32>
+//      CHECK:        %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+//  CHECK-NOT:         tensor.extract_slice %[[UNPACK]]
+//      CHECK:        linalg.generic
+//      CHECK:      scf.yield
+//      CHECK:    scf.yield
+//      CHECK:    return %[[RES]]
+
+module attributes {transform.with_named_sequence} {
+  // Hint indexed in the unpack's dest-dim order: dim0 Multiple, dim1 Equal.
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %gen = transform.structured.match ops{["linalg.generic"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %mulis = transform.structured.match ops{["arith.muli"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %i0, %i1, %h0, %h1 = transform.split_handle %mulis
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op,
+                                  !transform.any_op, !transform.any_op)
+    %tiled, %loops:2 = transform.structured.fuse %gen tile_sizes [%h0, %h1]
+        inner_tile_alignments = [Multiple, Equal]
+        : (!transform.any_op, !transform.any_op, !transform.any_op)
+        -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+
+  // Dummy entry points so the aligned/equal/unaligned RUN lines resolve here.
+  transform.named_sequence @aligned(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+  transform.named_sequence @equal(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+  transform.named_sequence @unaligned(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+}
+
+// -----
+
+// The hint also reaches the block-argument fusion path, when the producer is the
+// `scf.forall` init (used through the block argument rather than via a direct
+// extract use), `inner_tile_alignments` still yields the aligned tiling.
+
+func.func @fuse_unpack_through_block_arg(
+    %src: tensor<?x?x?x?xf32>, %unpack_empty: tensor<?x?xf32>,
+    %ub0: index, %ub1: index) -> tensor<?x?xf32> {
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %vscale = vector.vscale
+  %c8_vscale = arith.muli %c8, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %unpack = linalg.unpack %src inner_dims_pos = [0, 1]
+      inner_tiles = [%c8_vscale, %c4_vscale] into %unpack_empty
+      : tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  %res = scf.forall (%i, %j) = (0, 0) to (%ub0, %ub1) step (%c8_vscale, %c4_vscale)
+      shared_outs(%o = %unpack) -> (tensor<?x?xf32>) {
+    %slice = tensor.extract_slice %o[%i, %j] [%c8_vscale, %c4_vscale] [1, 1]
+        : tensor<?x?xf32> to tensor<?x?xf32>
+    %0 = linalg.exp ins(%slice : tensor<?x?xf32>) outs(%slice : tensor<?x?xf32>) -> tensor<?x?xf32>
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %0 into %o[%i, %j] [%c8_vscale, %c4_vscale] [1, 1]
+          : tensor<?x?xf32> into tensor<?x?xf32>
+    }
+  }
+  return %res : tensor<?x?xf32>
+}
+// CHECK-LABEL: func.func @fuse_unpack_through_block_arg
+//       CHECK:   scf.forall
+//       CHECK:     %[[UNPACK:.+]] = linalg.unpack
+//  CHECK-SAME:         : tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+//   CHECK-NOT:     tensor.extract_slice %[[UNPACK]]
+//       CHECK:     linalg.exp ins(%[[UNPACK]]
+
+module attributes {transform.with_named_sequence} {
+  // The `Equal` hints assert the containing loop's tile sizes equal the unpack
+  // inner tile sizes (8 * vscale, 4 * vscale).
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1 = transform.structured.match ops{["scf.forall"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %fused, %newc = transform.structured.fuse_into_containing_op %0 into %1
+        inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+
+  // Dummy entry points so the aligned/equal/unaligned RUN lines resolve here.
+  transform.named_sequence @aligned(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+  transform.named_sequence @equal(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+  transform.named_sequence @unaligned(%arg1: !transform.any_op {transform.readonly}) {
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/scalable-unpack-tiling.mlir b/mlir/test/Dialect/Linalg/scalable-unpack-tiling.mlir
new file mode 100644
index 0000000000000..57bbc3af79499
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/scalable-unpack-tiling.mlir
@@ -0,0 +1,122 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=equal -canonicalize -cse | FileCheck %s --check-prefixes=ALL,EQUAL
+// RUN: mlir-opt %s --transform-interpreter=entry-point=aligned -canonicalize -cse | FileCheck %s --check-prefixes=ALL,ALIGNED
+// RUN: mlir-opt %s --transform-interpreter=entry-point=unaligned -canonicalize -cse | FileCheck %s --check-prefixes=ALL,UNALIGNED
+
+// Pure tiling of a scalable `linalg.unpack` (no producer/consumer fusion). The
+// same input IR is tiled three ways, each selected by a different transform
+// entry point (see the named sequences and RUN lines):
+//   - `@equal`     tiles with [2, 4] * vscale, equal to the inner tiles, and
+//                  asserts that via `[Equal, Equal]`. Perfect tiling: the tiled
+//                  unpack's outer dims collapse to 1 and no remainder is needed.
+//   - `@aligned`   tiles with [4, 8] * vscale, integer multiples of the inner
+//                  tiles, asserted via `[Multiple, Multiple]`. The outer dims
+//                  become `ceilDiv(loop tile, inner tile)` and no remainder is
+//                  needed.
+//   - `@unaligned` tiles with static [7, 5], not aligned to the scalable inner
+//                  tiles and with no hint, so the tiled unpack keeps a trailing
+//                  remainder `tensor.extract_slice`.
+//
+// The `inner_tile_alignments` hint is a per-dimension keyword list:
+//   - `Equal`:    the loop tile size equals the pack/unpack inner tile size.
+//   - `Multiple`: the loop tile size is an integer multiple of the inner tile.
+//   - `Unknown`:  the default; nothing is asserted for that dimension.
+
+//   ALIGNED-DAG: #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// ALL-LABEL: func.func @CKkc_to_KC_scalable
+
+func.func @CKkc_to_KC_scalable(%source: tensor<32x4x?x?xf32>, %dest: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %vscale = vector.vscale
+  %c2_vscale = arith.muli %c2, %vscale : index
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %0 = linalg.unpack %source outer_dims_perm = [1, 0] inner_dims_pos = [0, 1]
+      inner_tiles = [%c2_vscale, %c4_vscale] into %dest
+      : tensor<32x4x?x?xf32> -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  // Perfect tiling: the scalable tile sizes ([2] = 2 * vscale, [4] = 4 * vscale)
+  // equal the scalable inner tiles, asserted via `Equal`.
+  transform.named_sequence @equal(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[2], [4]]
+        inner_tile_alignments = [Equal, Equal]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // --- @equal: perfect tiling, outer dims collapse to 1. ---
+  //   EQUAL-DAG:    %[[C2:.*]] = arith.constant 2 : index
+  //   EQUAL-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //   EQUAL-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //   EQUAL-DAG:    %[[C2_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C2]] : index
+  //   EQUAL-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //       EQUAL:    %[[RES:.*]] = scf.for {{.*}} step %[[C2_VSCALE]]
+  //       EQUAL:      scf.for {{.*}} step %[[C4_VSCALE]]
+  //       EQUAL:        %[[UNPACK:.*]] = linalg.unpack
+  //  EQUAL-SAME:            tensor<1x1x?x?xf32> -> tensor<?x?xf32>
+  // For the perfect tiling case, the outer dims equal 1 and the unpack can be
+  // consumed directly - no extra extract_slice on the result is needed.
+  //   EQUAL-NOT:         tensor.extract_slice %[[UNPACK]]
+  //       EQUAL:        tensor.insert_slice %[[UNPACK]]
+  //       EQUAL:      scf.yield
+  //       EQUAL:    scf.yield
+  //       EQUAL:    return %[[RES]]
+
+  // Aligned tiling: the scalable tile sizes ([4] = 4 * vscale, [8] = 8 * vscale)
+  // are integer multiples of the scalable inner tiles (2 * vscale, 4 * vscale),
+  // asserted via `Multiple`.
+  transform.named_sequence @aligned(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [[4], [8]]
+        inner_tile_alignments = [Multiple, Multiple]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // --- @aligned: multiple tiling, outer dims are ceilDiv(loop tile, inner tile). ---
+  // ALIGNED-DAG:    %[[C2:.*]] = arith.constant 2 : index
+  // ALIGNED-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  // ALIGNED-DAG:    %[[VSCALE:.*]] = vector.vscale
+  // ALIGNED-DAG:    %[[C2_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C2]] : index
+  // ALIGNED-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  // ALIGNED-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %{{.*}} : index
+  //     ALIGNED:    %[[RES:.*]] = scf.for {{.*}} step %[[C4_VSCALE]]
+  //     ALIGNED:      scf.for {{.*}} step %[[C8_VSCALE]]
+  // The outer dims are ceilDiv(loop tile, inner tile); the transpose (outer_dims_perm
+  // = [1, 0]) swaps them back into source-dim order in the extracted slice.
+  //     ALIGNED:        %[[CEIL_C2:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C2_VSCALE]]]
+  //     ALIGNED:        %[[CEIL_C4:.*]] = affine.apply #[[$MAP_CEILDIV]](%{{.*}})[%[[C4_VSCALE]]]
+  //     ALIGNED:        %[[SRC:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, %{{.*}}, 0, 0] [%[[CEIL_C4]], %[[CEIL_C2]], %[[C2_VSCALE]], %[[C4_VSCALE]]]
+  //     ALIGNED:        %[[UNPACK:.*]] = linalg.unpack %[[SRC]]
+  // ALIGNED-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  // For the aligned tiling case, the unpack can also be consumed directly.
+  // ALIGNED-NOT:         tensor.extract_slice %[[UNPACK]]
+  //     ALIGNED:        tensor.insert_slice %[[UNPACK]]
+  //     ALIGNED:      scf.yield
+  //     ALIGNED:    scf.yield
+  //     ALIGNED:    return %[[RES]]
+
+  // Unaligned tiling: static tile sizes (7, 5) are not aligned to the scalable
+  // inner tiles and no hint is passed, so the tiled unpack keeps a remainder.
+  transform.named_sequence @unaligned(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.unpack"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1, %loops:2 = transform.structured.tile_using_for %0 tile_sizes [7, 5]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  // --- @unaligned: static tiling, keeps a trailing remainder slice. ---
+  // UNALIGNED-DAG:    %[[C7:.*]] = arith.constant 7 : index
+  // UNALIGNED-DAG:    %[[C5:.*]] = arith.constant 5 : index
+  //     UNALIGNED:    %[[RES:.*]] = scf.for {{.*}} step %[[C7]]
+  //     UNALIGNED:      scf.for {{.*}} step %[[C5]]
+  //     UNALIGNED:        %[[UNPACK:.*]] = linalg.unpack
+  // UNALIGNED-SAME:            tensor<?x?x?x?xf32> -> tensor<?x?xf32>
+  // For the unaligned tiling case the unpack over-computes, so a trailing
+  // extract_slice recovers the correct slice from the result.
+  //     UNALIGNED:        %[[EXTRACT:.*]] = tensor.extract_slice %[[UNPACK]]
+  //     UNALIGNED:        tensor.insert_slice %[[EXTRACT]]
+  //     UNALIGNED:      scf.yield
+  //     UNALIGNED:    scf.yield
+  //     UNALIGNED:    return %[[RES]]
+}


From llvmlistbot at llvm.org  Fri Jul  3 02:43:00 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:43:00 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a478424.eb24c6ea.1f2e26.6cc1@mx.google.com>


================
@@ -0,0 +1,122 @@
+// RUN: mlir-opt %s --transform-interpreter=entry-point=equal -canonicalize -cse | FileCheck %s --check-prefixes=CHECK,EQUAL
----------------
egebeysel wrote:

done

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 02:43:05 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:43:05 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a478429.4a342cbc.134cb.0d55@mx.google.com>


================
@@ -0,0 +1,366 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// Consumer fusion - linalg.pack with scalable inner tiles. Producer step (8*vscale)
+// equals the pack inner tile size(8*vscale) on the tiled source dimension, so the
+// outer dim of the fused pack tile is statically 1. This information is passed as
+// an inner tile alignment hint `Equal`.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_equal
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_equal(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // Loop tile size is equal to the inner tile size of the consumer `linalg.pack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //      CHECK:    %[[C8:.*]] = arith.constant 8 : index
+  //      CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //      CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %{{.*}}]
+  // CHECK-SAME:          -> tensor<1x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion with a static producer step (64) and a scalable pack inner
+// tile (8*vscale), hinted `Multiple`. Fusion honors the hint and takes the aligned 
+// (non-equal) path and the outer dim of the fused pack tile is dynamic (`64 ceildiv 8*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<()[s0] -> (64 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c64 = arith.constant 64 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c64 iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %ext_out = tensor.extract_slice %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<64x128xf32>, tensor<64x128xf32>)
+        outs(%ext_out : tensor<64x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<64x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<64x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size 64
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[C64:.*]] = arith.constant 64 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C64]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]]()[%[[C8_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
----------------
egebeysel wrote:

added

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 02:43:09 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:43:09 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a47842d.7d41f3a1.d1cf7.5ddc@mx.google.com>


================
@@ -0,0 +1,366 @@
+// RUN: mlir-opt %s -transform-interpreter -canonicalize -cse -split-input-file --verify-diagnostics | FileCheck %s
+
+// Consumer fusion - linalg.pack with scalable inner tiles. Producer step (8*vscale)
+// equals the pack inner tile size(8*vscale) on the tiled source dimension, so the
+// outer dim of the fused pack tile is statically 1. This information is passed as
+// an inner tile alignment hint `Equal`.
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_equal
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_equal(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  // Loop tile size is equal to the inner tile size of the consumer `linalg.pack` (8 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Equal` hint is passed to hint the equality between the loop tile size 8 * vscale
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Equal, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //      CHECK:    %[[C8:.*]] = arith.constant 8 : index
+  //      CHECK:    %[[VSCALE:.*]] = vector.vscale
+  //      CHECK:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          inner_tiles = [%[[C8_VSCALE]], %{{.*}}]
+  // CHECK-SAME:          -> tensor<1x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion with a static producer step (64) and a scalable pack inner
+// tile (8*vscale), hinted `Multiple`. Fusion honors the hint and takes the aligned 
+// (non-equal) path and the outer dim of the fused pack tile is dynamic (`64 ceildiv 8*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<()[s0] -> (64 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c64 = arith.constant 64 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+
+  %0 = scf.for %iv = %c0 to %c256 step %c64 iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %ext_out = tensor.extract_slice %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [64, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<64x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<64x128xf32>, tensor<64x128xf32>)
+        outs(%ext_out : tensor<64x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<64x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [64, 128] [1, 1]
+        : tensor<64x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c8_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size 64
+    // and the inner tile size 8 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[C64:.*]] = arith.constant 64 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C64]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]]()[%[[C8_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
+  // CHECK-SAME:          -> tensor<?x?x?x?xf32>
+  //      CHECK:      scf.yield {{.*}}, %{{.*}} :
+  //      CHECK:    return %[[RES]]#1
+}
+
+// -----
+
+// Consumer fusion - both producer step and pack inner tile are scalable, step (8*vscale)
+// is an integer multiple of the inner tile (4*vscale) but not equal to it. The corresponding
+// `Multiple` hint is passed to the tiling interface. Fusion succeeds via the aligned (non-equal)
+// path, so the outer dim of the fused pack tile stays dynamic (`8*vscale ceildiv 4*vscale`).
+
+#map = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK:       #[[$MAP_MIN:.+]] = affine_map<(d0)[s0] -> (-d0 + 256, s0)>
+// CHECK:       #[[$MAP_CEILDIV:.+]] = affine_map<(d0)[s0] -> (d0 ceildiv s0)>
+// CHECK-LABEL: func.func @fuse_scalable_pack_consumer_aligned_scalable
+// CHECK-SAME:      %[[ARG0:.+]]: tensor<256x128xf32>, %[[ARG1:.+]]: tensor<256x128xf32>, %[[ARG2:.+]]: tensor<256x128xf32>, %[[DEST:.+]]: tensor<?x?x?x?xf32>
+func.func @fuse_scalable_pack_consumer_aligned_scalable(
+    %arg0: tensor<256x128xf32>, %arg1: tensor<256x128xf32>,
+    %arg2: tensor<256x128xf32>, %dest: tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %c256 = arith.constant 256 : index
+  %vscale = vector.vscale
+  %c4_vscale = arith.muli %c4, %vscale : index
+  %c8_vscale = arith.muli %c8, %vscale : index
+  
+  // Loop tile size (8 * vscale) is a multiple of the inner tile size of the consumer `linalg.pack` (4 * vscale).
+  %0 = scf.for %iv = %c0 to %c256 step %c8_vscale iter_args(%out = %arg2) -> (tensor<256x128xf32>) {
+    %sz = affine.min affine_map<(d0)[s0] -> (-d0 + 256, s0)>(%iv)[%c8_vscale]
+    %ext_out = tensor.extract_slice %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_a = tensor.extract_slice %arg0[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %ext_b = tensor.extract_slice %arg1[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<256x128xf32> to tensor<?x128xf32>
+    %computed = linalg.generic {
+        indexing_maps = [#map, #map, #map],
+        iterator_types = ["parallel", "parallel"]}
+        ins(%ext_a, %ext_b : tensor<?x128xf32>, tensor<?x128xf32>)
+        outs(%ext_out : tensor<?x128xf32>) {
+      ^bb0(%in0: f32, %in1: f32, %out_elem: f32):
+        %mul = arith.mulf %in0, %in1 : f32
+        linalg.yield %mul : f32
+    } -> tensor<?x128xf32>
+    %inserted = tensor.insert_slice %computed into %out[%iv, 0] [%sz, 128] [1, 1]
+        : tensor<?x128xf32> into tensor<256x128xf32>
+    scf.yield %inserted : tensor<256x128xf32>
+  }
+
+  %pack = linalg.pack %0 outer_dims_perm = [0, 1]
+      inner_dims_pos = [0, 1] inner_tiles = [%c4_vscale, %c4_vscale]
+      into %dest : tensor<256x128xf32> -> tensor<?x?x?x?xf32>
+  return %pack : tensor<?x?x?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %pack = transform.structured.match ops{["linalg.pack"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    %loop = transform.structured.match ops{["scf.for"]} in %arg1
+        : (!transform.any_op) -> !transform.any_op
+    // The `Multiple` hint is passed to hint the alignment between the loop tile size
+    // 8 * vscale and the inner tile size 4 * vscale.
+    %a, %b = transform.test.fuse_consumer %pack into (%loop) inner_tile_alignments = [Multiple, Unknown]
+        : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+  //  CHECK-DAG:    %[[C4:.*]] = arith.constant 4 : index
+  //  CHECK-DAG:    %[[C8:.*]] = arith.constant 8 : index
+  //  CHECK-DAG:    %[[VSCALE:.*]] = vector.vscale
+  //  CHECK-DAG:    %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+  //  CHECK-DAG:    %[[C8_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C8]] : index
+  //      CHECK:    %[[RES:.*]]:2 = scf.for {{.*}} step %[[C8_VSCALE]]
+  // CHECK-SAME:        iter_args(%{{.*}} = %[[ARG2]], %{{.*}} = %[[DEST]])
+  //      CHECK:      %[[SZ:.*]] = affine.min #[[$MAP_MIN]](%{{.*}})[%[[C8_VSCALE]]]
+  //      CHECK:      %[[GENERIC:.*]] = linalg.generic
+  //      CHECK:      %[[OUTER:.*]] = affine.apply #[[$MAP_CEILDIV]](%[[SZ]])[%[[C4_VSCALE]]]
+  //      CHECK:      %[[PACK_DEST:.*]] = tensor.extract_slice %{{.*}}[%{{.*}}, 0, 0, 0] [%[[OUTER]], %{{.*}}, %{{.*}}, %{{.*}}] [1, 1, 1, 1]
+  //      CHECK:      %[[PACK:.*]] = linalg.pack %[[GENERIC]]
+  // CHECK-SAME:          into %[[PACK_DEST]]
----------------
egebeysel wrote:

added

https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 02:45:24 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:45:24 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a4784b4.bb839515.23e6d3.cc83@mx.google.com>


https://github.com/egebeysel edited https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 02:46:01 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:46:01 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a4784d9.6b2462aa.255021.726a@mx.google.com>


https://github.com/egebeysel auto_merge_enabled https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 02:53:36 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 03 Jul 2026 02:53:36 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][linalg/scf/transform] scalable
 tiling and fusion for pack/unpack ops (PR #204007)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204007@github.com>
Message-ID: <6a4786a0.14d9b9dc.140829.42eb@mx.google.com>


https://github.com/egebeysel closed https://github.com/llvm/llvm-project/pull/204007

From llvmlistbot at llvm.org  Fri Jul  3 03:00:36 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 03 Jul 2026 03:00:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a478844.ac469dc7.e3aa6.1710@mx.google.com>


================

----------------
ioghiban wrote:

If this assert is reached, `getSingleNonUnitDim(srcType)` returned true. Therefore, the source is guaranteed to have one non-unit dimension. This region also assumes dynamic offset, but that's not relevant for this assert, so I will move it before the offset check. Does this comment motivate the assert:
```
  // A source with exactly one non-unit dimension cannot be indexed directly by
  // multiple non-unit result dimensions.
```
?


https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Fri Jul  3 03:13:28 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 03 Jul 2026 03:13:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a478b48.d1496011.3d49cb.a098@mx.google.com>


================

----------------
ioghiban wrote:

These are the tests for the load rewrite. Will split the file in a future PR.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Fri Jul  3 03:16:13 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 03 Jul 2026 03:16:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a478bed.f732ad81.1d8ca.7aca@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/11] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/11] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/11] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/11] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/11] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/11] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/11] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/11] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/11] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/11] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From ce98ffa843c1eb164e4757c25a88a63725030117 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/11] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 155 ++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 232 ++++++++++--------
 2 files changed, 172 insertions(+), 215 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,31 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
-///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,19 +152,24 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,12 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -259,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -278,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -193,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -220,6 +257,74 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +414,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +464,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +513,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Fri Jul  3 03:18:41 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 03 Jul 2026 03:18:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a478c81.65dfdb71.1ca534.a99b@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/11] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/11] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/11] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/11] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/11] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/11] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/11] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/11] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/11] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/11] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/11] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Fri Jul  3 03:20:06 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 03 Jul 2026 03:20:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a478cd6.92ea0e2a.21f97.eeaa@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/12] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/12] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/12] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/12] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/12] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/12] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/12] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/12] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/12] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/12] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/12] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 53a57bba5c89d408c64fec9187c8c81b7593ca1c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/12] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 5048a6507c08e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f3c47aeab7f58..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst


From llvmlistbot at llvm.org  Fri Jul  3 03:54:36 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 03:54:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4794ec.576b3af0.2cb9b7.d0c4@mx.google.com>


https://github.com/IanTaylerLessa-arm edited https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 03:54:37 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 03:54:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4794ed.a87eb5dc.2cf3a1.955e@mx.google.com>


================

----------------
IanTaylerLessa-arm wrote:

Could we use an enum instead here, or does it introduce problems? This would become invalid if two-dimensional block shapes are ever added to the spec.

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 03:54:37 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 03:54:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4794ed.c56fb102.11841f.d441@mx.google.com>


https://github.com/IanTaylerLessa-arm commented:

Two optional comments, looks good to me otherwise!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 03:54:37 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 03:54:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4794ed.f06f4994.168928.f964@mx.google.com>


================

----------------
IanTaylerLessa-arm wrote:

I think bf16 would also work here since the largest mantissa in mxfp is for mxint8 which behaves as if it had 7 mantissa bits, like bf16.

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 04:05:17 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Fri, 03 Jul 2026 04:05:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix TosaNarrowTypes conversion
 order to prevent rollback crash (PR #207372)
Message-ID: <llvm.org/llvm/llvm-project/pull/207372@github.com>


https://github.com/hockyy created https://github.com/llvm/llvm-project/pull/207372

Fix #206952 by doing `convertRegionTypes` before moving and unlinking the blocks when handling `tosa.cond_if` ops. This will cause a crash when doing rollback. 

>From 94a2e30c84c8d835a7595952399bee6003771ced Mon Sep 17 00:00:00 2001
From: Hocky Yudhiono <hocky.yudhiono at gmail.com>
Date: Fri, 3 Jul 2026 18:43:02 +0800
Subject: [PATCH] [mlir][tosa] Fix TosaNarrowTypes conversion order to prevent
 rollback crash

---
 .../Tosa/Transforms/TosaNarrowTypes.cpp       |  4 ++--
 .../Dialect/Tosa/tosa-narrow-i64-to-i32.mlir  | 21 +++++++++++++++++++
 2 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
index 983472762e791..d441f90c2a074 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
@@ -426,10 +426,10 @@ LogicalResult convertGenericOp(Operation *op, ValueRange operands,
   }
 
   for (Region &region : op->getRegions()) {
+    if (failed(rewriter.convertRegionTypes(&region, *typeConverter)))
+      return failure();
     Region *newRegion = state.addRegion();
     rewriter.inlineRegionBefore(region, *newRegion, newRegion->begin());
-    if (failed(rewriter.convertRegionTypes(newRegion, *typeConverter)))
-      return failure();
   }
 
   Operation *newOp = rewriter.create(state);
diff --git a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
index 9349bd1c1cf7b..f51c5b714378e 100644
--- a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
@@ -75,6 +75,27 @@ func.func @test_regions(%arg0: tensor<1x2xi32>, %arg1: tensor<1xi32>, %arg2: ten
 
 // -----
 
+// CHECK-LABEL: test_cond_if_i64_yield_with_call
+module {
+  func.func @m0() -> () {
+    return
+  }
+  func.func @test_cond_if_i64_yield_with_call(%arg0: tensor<1xi1>, %arg1: tensor<4xi64>, %arg2: tensor<4xi64>) -> () {
+    %0 = tosa.cond_if %arg0 (%arg3 = %arg1) : tensor<1xi1> (tensor<4xi64>) -> tensor<4xi64> {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    } else {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    }
+    // expected-error @+1 {{failed to legalize operation 'func.call'}}
+    call @m0() : () -> ()
+    return
+  }
+}
+
+// -----
+
 // CHECK-LABEL: test_concat
 func.func @test_concat(%arg0: tensor<13x21x3xi64>, %arg1: tensor<13x21x3xi64>) -> tensor<26x21x3xi64> {
   // COMMON: tosa.concat %{{.*}}, %{{.*}} {axis = 0 : i32} : (tensor<13x21x3xi32>, tensor<13x21x3xi32>) -> tensor<26x21x3xi32>


From llvmlistbot at llvm.org  Fri Jul  3 04:05:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 04:05:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix TosaNarrowTypes conversion
 order to prevent rollback crash (PR #207372)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207372@github.com>
Message-ID: <6a479796.bb839515.23e6d3.a3d3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-tosa

@llvm/pr-subscribers-mlir

Author: Hocky Yudhiono (hockyy)

<details>
<summary>Changes</summary>

Fix #<!-- -->206952 by doing `convertRegionTypes` before moving and unlinking the blocks when handling `tosa.cond_if` ops. This will cause a crash when doing rollback. 

---
Full diff: https://github.com/llvm/llvm-project/pull/207372.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp (+2-2) 
- (modified) mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir (+21) 


``````````diff
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
index 983472762e791..d441f90c2a074 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
@@ -426,10 +426,10 @@ LogicalResult convertGenericOp(Operation *op, ValueRange operands,
   }
 
   for (Region &region : op->getRegions()) {
+    if (failed(rewriter.convertRegionTypes(&region, *typeConverter)))
+      return failure();
     Region *newRegion = state.addRegion();
     rewriter.inlineRegionBefore(region, *newRegion, newRegion->begin());
-    if (failed(rewriter.convertRegionTypes(newRegion, *typeConverter)))
-      return failure();
   }
 
   Operation *newOp = rewriter.create(state);
diff --git a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
index 9349bd1c1cf7b..f51c5b714378e 100644
--- a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
@@ -75,6 +75,27 @@ func.func @test_regions(%arg0: tensor<1x2xi32>, %arg1: tensor<1xi32>, %arg2: ten
 
 // -----
 
+// CHECK-LABEL: test_cond_if_i64_yield_with_call
+module {
+  func.func @m0() -> () {
+    return
+  }
+  func.func @test_cond_if_i64_yield_with_call(%arg0: tensor<1xi1>, %arg1: tensor<4xi64>, %arg2: tensor<4xi64>) -> () {
+    %0 = tosa.cond_if %arg0 (%arg3 = %arg1) : tensor<1xi1> (tensor<4xi64>) -> tensor<4xi64> {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    } else {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    }
+    // expected-error @+1 {{failed to legalize operation 'func.call'}}
+    call @m0() : () -> ()
+    return
+  }
+}
+
+// -----
+
 // CHECK-LABEL: test_concat
 func.func @test_concat(%arg0: tensor<13x21x3xi64>, %arg1: tensor<13x21x3xi64>) -> tensor<26x21x3xi64> {
   // COMMON: tosa.concat %{{.*}}, %{{.*}} {axis = 0 : i32} : (tensor<13x21x3xi32>, tensor<13x21x3xi32>) -> tensor<26x21x3xi32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207372

From llvmlistbot at llvm.org  Fri Jul  3 04:50:15 2026
From: llvmlistbot at llvm.org (Michael Klemm)
Date: Fri, 03 Jul 2026 04:50:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a47a1f7.4a342cbc.134cb.661c@mx.google.com>


https://github.com/mjklemm approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Fri Jul  3 05:07:16 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 05:07:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix TosaNarrowTypes conversion
 order to prevent rollback crash (PR #207372)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207372@github.com>
Message-ID: <6a47a5f4.cb30c3e4.6f38b.953d@mx.google.com>


https://github.com/lhutton1 approved this pull request.

LGTM, thanks @hockyy!

https://github.com/llvm/llvm-project/pull/207372

From llvmlistbot at llvm.org  Fri Jul  3 05:24:48 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:24:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47aa10.eea6ab6e.1256b6.9a97@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 5f144a05466b4eb12257e8840e0173225de0e8b3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 1/7] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index c30dd3b07d028..9d810f2613a26 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1092,7 +1092,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 6e1ed05c9d0d0..9aa28ca825f73 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -699,6 +699,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From cde91e515bc3eb100a6524cda7b07417f5a6b4c5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 2/7] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From 0f0b91d8f9627c04fdddb1d2ca9280238de28f27 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 3/7] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9aa28ca825f73..777e5d72ac4b1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -741,7 +741,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -758,22 +757,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -808,7 +791,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -825,22 +807,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From e1c107c642b6c519e7439657e970c9ae9e5a408e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 4/7] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 1da6401974970bf094954cd63c40a91a4b038492 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 5/7] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 777e5d72ac4b1..162a92782380b 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,60 +749,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -818,6 +812,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 9f52cbd5aaa9349b22f1da0493dd306b360569f6 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 6/7] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 162a92782380b..9c6039eeefc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,54 +749,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -812,9 +828,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From f646be2233d25f193c2db276734143e5315c1607 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:24:09 +0000
Subject: [PATCH 7/7] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 92 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 ++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++------
 7 files changed, 138 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..7357c4cc3bdee 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,18 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types. Works for any entry type exposing `src`/`dst` fields
+/// (e.g. FPTruncTableEntry, FPExtTableEntry), with the entry type and table
+/// size deduced from the passed array.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1869,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1923,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1951,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1982,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2078,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2193,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2260,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2299,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2338,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2353,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2430,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9c6039eeefc2c..8754149801124 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -805,9 +805,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }


From llvmlistbot at llvm.org  Fri Jul  3 05:28:13 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 05:28:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47aadd.d5797644.146403.e5a5@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/203583

>From e9755287e97fb9bab0030b391da1a9f5846996f0 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Tue, 19 May 2026 10:44:43 +0100
Subject: [PATCH 1/3] [mlir][tosa] Add new block-scaled tensor type and support
 for MXFP CAST

This commit adds a new compound-block scaled tensor type and uses this
type to implement support for MXFP in the CAST operation, as per
the following specification changes:
https://github.com/arm/tosa-specification/pull/50,
https://github.com/arm/tosa-specification/pull/53.

The new block-scaled type is closely modeled after the `quant` dialect
type and supports the following parameters:
- value type - The type of the data values in each block.
- scale type - The type of the scale value associated with each block.
- block shape - The size and axis of each block.
Support for specifying constant scale values has not been added in this
commit, but will be added in a later one.

Example syntax for the new block-scaled type:
```
tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
```

As a pre-requisite for supporting MXFP CAST, the commit also adds new
extensions intended to split the exisiting EXT-MXFP extension into
separate extensions for each block-scaled type. This allows for more
fine-grained control over which block-scaled types are supported by a
given target. See specification change
https://github.com/arm/tosa-specification/pull/26 for details.

Finally, support for casting to/from the new block-scaled type has been
added, aligning with the behavior specified in the CAST operation
specification.

Change-Id: Idfab945402a95704ffeefbcee44172c3dde7552d
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 120 +++++++++++++++++-
 .../mlir/Dialect/Tosa/IR/TosaOpBase.td        |  61 +++++++--
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h   |   2 +
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  69 +++-------
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   |  30 ++++-
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 117 +++++++++++------
 mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp        |  14 ++
 .../Dialect/Tosa/IR/TosaCanonicalizations.cpp |  44 ++++++-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |  65 ++++++++++
 .../Tosa/Transforms/TosaProfileCompliance.cpp |  92 ++++++++++----
 .../Tosa/Transforms/TosaValidation.cpp        |   2 +-
 mlir/test/Dialect/Tosa/availability.mlir      |   6 +-
 mlir/test/Dialect/Tosa/canonicalize.mlir      |  67 ++++++++++
 mlir/test/Dialect/Tosa/invalid.mlir           |  32 +++++
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  28 ++++
 mlir/test/Dialect/Tosa/ops.mlir               |  26 ++++
 .../test/Dialect/Tosa/tosa-attach-target.mlir |  19 ++-
 .../tosa-validation-version-1p0-invalid.mlir  |   8 ++
 .../tosa-validation-version-1p1-valid.mlir    |  30 ++++-
 mlir/test/Dialect/Tosa/verifier.mlir          |  42 ++++++
 20 files changed, 729 insertions(+), 145 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 50bb9f69c6242..26890abd187bd 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1023,7 +1023,125 @@ extensionComplianceMap = {
        {{{fp8e5m2T, fp16T}, SpecificationVersion::V_1_0},
         {{fp8e5m2T, fp32T}, SpecificationVersion::V_1_0},
         {{fp16T, fp8e5m2T}, SpecificationVersion::V_1_0},
-        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}}}},
+        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.cast_from_block_scaled",
      {{{Extension::bf16, Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, bf16T}, SpecificationVersion::V_1_1_DRAFT},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
index 591073e9985ae..308e98ac42435 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
@@ -254,6 +254,13 @@ class Tosa_I32EnumAttr<string name, string description, string mnemonic,
 // MXFP         : Microscaling formats.
 // MXFP_CONV    : Microscaling format convolution.
 // SHAPE        : Shape calcuation operators.
+// MX_COMMON    : Base for MXFP microscaling formats.
+// MX_FP4E2M1   : Microscaling format FP4E2M1.
+// MX_FP6E2M3   : Microscaling format FP6E2M3.
+// MX_FP6E3M2   : Microscaling format FP6E3M2.
+// MX_FP8E4M3   : Microscaling format FP8E4M3.
+// MX_FP8E5M2   : Microscaling format FP8E5M2.
+// MX_INT8      : Microscaling format INT8.
 //===----------------------------------------------------------------------===//
 
 def Tosa_NONE : I32EnumAttrCase<"none", 0>;
@@ -289,23 +296,35 @@ def Tosa_EXT_MXFP         : I32EnumAttrCase<"mxfp", 12>;
 def Tosa_EXT_INT64        : I32EnumAttrCase<"int64", 13>;
 def Tosa_EXT_MXFP_CONV    : I32EnumAttrCase<"mxfp_conv", 14>;
 def Tosa_EXT_SHAPE        : I32EnumAttrCase<"shape", 15>;
+def Tosa_EXT_MX_COMMON : I32EnumAttrCase<"mx_common", 16>;
+def Tosa_EXT_MX_FP4E2M1 : I32EnumAttrCase<"mx_fp4e2m1", 17>;
+def Tosa_EXT_MX_FP6E2M3 : I32EnumAttrCase<"mx_fp6e2m3", 18>;
+def Tosa_EXT_MX_FP6E3M2 : I32EnumAttrCase<"mx_fp6e3m2", 19>;
+def Tosa_EXT_MX_FP8E4M3 : I32EnumAttrCase<"mx_fp8e4m3", 20>;
+def Tosa_EXT_MX_FP8E5M2 : I32EnumAttrCase<"mx_fp8e5m2", 21>;
+def Tosa_EXT_MX_INT8 : I32EnumAttrCase<"mx_int8", 22>;
 
 def Tosa_ExtensionAttr
-    : Tosa_I32EnumAttr<"Extension", "supported TOSA extensions", "ext", [
-      Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
-      Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
-      Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
-      Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
-      Tosa_EXT_SHAPE,
-    ]> {
+    : Tosa_I32EnumAttr<
+          "Extension", "supported TOSA extensions", "ext",
+          [Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
+           Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
+           Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
+           Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
+           Tosa_EXT_SHAPE, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+           Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+           Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]> {
   let extraClassDeclaration = [{
-    static llvm::SmallVector<Extension, 14> getAllValues() {
+    static llvm::SmallVector<Extension, 22> getAllValues() {
       return {
         Extension::int16, Extension::int4, Extension::bf16,
         Extension::fp8e4m3, Extension::fp8e5m2, Extension::fft,
         Extension::variable, Extension::controlflow, Extension::doubleround,
         Extension::inexactround, Extension::dynamic, Extension::mxfp,
-        Extension::int64, Extension::mxfp_conv, Extension::shape
+        Extension::int64, Extension::mxfp_conv, Extension::shape,
+        Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3,
+        Extension::mx_fp6e3m2, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2,
+        Extension::mx_int8
       };
     }
   }];
@@ -484,6 +503,7 @@ def Tosa_RoundingModeAttr
     : Tosa_I32EnumAttr<"RoundingMode", "Supported rounding modes", "rounding_mode",
                     [Tosa_ROUNDING_SINGLE_ROUND, Tosa_ROUNDING_INEXACT_ROUND, Tosa_ROUNDING_DOUBLE_ROUND]>;
 
+// Block_size attr is deprecated and will be removed in the future
 def Tosa_BLOCK_SIZE_1 : I32EnumAttrCase<"BLOCK_SIZE_1", 1>;
 def Tosa_BLOCK_SIZE_32 : I32EnumAttrCase<"BLOCK_SIZE_32", 32>;
 
@@ -497,6 +517,29 @@ def Tosa_BlockSizeAttr
   }];
 }
 
+def Tosa_BLOCK_SHAPE_32 : I32EnumAttrCase<"BLOCK_SHAPE_32", 32>;
+
+def Tosa_BlockShape
+    : Tosa_I32Enum<
+          "BlockShape",
+          "Block shape for the block_scaled formats."
+          "The names follow the convention of BLOCK_SHAPE_M where M "
+          "is the block size of the innermost dimension. Similarly, "
+          "BLOCK_SHAPE_MxN indicates block size of N in the innermost "
+          "dimension and block size of M in the next innermost dimension. "
+          "As of now, only 1 dimension (innermost) is supported for "
+          "block-scaling.",
+          [Tosa_BLOCK_SHAPE_32]>;
+
+def Tosa_BlockShapeAttr
+    : EnumAttr<Tosa_Dialect, Tosa_BlockShape, "block_shape"> {
+  let extraClassDeclaration = [{
+    static uint32_t getBlockShapeValue(BlockShape blockShape) {
+      return static_cast<uint32_t>(blockShape);
+    }
+  }];
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Interfaces.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index e0626368175ee..a5c6037692ff1 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,6 +92,8 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
+LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+
 } // namespace tosa
 } // namespace OpTrait
 
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index a99fb2fcae547..32f8bf08cbc84 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2689,57 +2689,6 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
 
   let description = [{
     Casts a tensor from one data type to another.
-    * This table is showing the supported conversions from the TOSA Specification.
-    * The MLIR dialect here can be used to represent other conversions.
-
-    | Mode                     | Input   | Output  |
-    |--------------------------|---------|---------|
-    | fp16 to fp32             | float16 | float32 |
-    | fp16 to int 16           | float16 | int16   |
-    | fp16 to int 32           | float16 | int32   |
-    | fp16 to int 8            | float16 | int8    |
-    | fp32 to fp16             | float32 | float16 |
-    | fp32 to int 16           | float32 | int16   |
-    | fp32 to int 32           | float32 | int32   |
-    | fp32 to int 8            | float32 | int8    |
-    | int 16 to fp16           | int16   | float16 |
-    | int 16 to fp32           | int16   | float32 |
-    | int 32 to fp16           | int32   | float16 |
-    | int 32 to fp32           | int32   | float32 |
-    | int 8 to fp16            | int8    | float16 |
-    | int 8 to fp32            | int8    | float32 |
-    | bool to int 16           | Boolean | int16   |
-    | bool to int 32           | Boolean | int32   |
-    | bool to int 8            | Boolean | int8    |
-    | int 16 to bool           | int16   | Boolean |
-    | int 16 to int 32         | int16   | int32   |
-    | int 16 to int 8          | int16   | int8    |
-    | int 32 to bool           | int32   | Boolean |
-    | int 32 to int 16         | int32   | int16   |
-    | int 32 to int 8          | int32   | int8    |
-    | int 8 to bool            | int8    | Boolean |
-    | int 8 to int 16          | int8    | int16   |
-    | int 8 to int 32          | int8    | int32   |
-    | bf16 to fp32             | bf16    | float32 |
-    | bf16 to int 16           | bf16    | int16   |
-    | bf16 to int 32           | bf16    | int32   |
-    | bf16 to int 8            | bf16    | int8    |
-    | fp32 to bf16             | float32 | bf16    |
-    | int 16 to bf16           | int16   | bf16    |
-    | int 32 to bf16           | int32   | bf16    |
-    | int 8 to bf16            | int8    | bf16    |
-    | bf16 to fp8e4m3          | bf16    | fp8e4m3 |
-    | fp8e4m3 to bf16          | fp8e4m3 | bf16    |
-    | bf16 to fp8e5m2          | bf16    | fp8e5m2 |
-    | fp8e5m2 to bf16          | fp8e5m2 | bf16    |
-    | fp16 to fp8e4m3          | float16 | fp8e4m3 |
-    | fp32 to fp8e4m3          | float32 | fp8e4m3 |
-    | fp8e4m3 to fp16          | fp8e4m3 | float16 |
-    | fp8e4m3 to fp32          | fp8e4m3 | float32 |
-    | fp16 to fp8e5m2          | float16 | fp8e5m2 |
-    | fp32 to fp8e5m2          | float32 | fp8e5m2 |
-    | fp8e5m2 to fp16          | fp8e5m2 | float16 |
-    | fp8e5m2 to fp32          | fp8e5m2 | float32 |
   }];
 
   let arguments = (ins
@@ -2750,19 +2699,25 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 
   let hasFolder = 1;
   let hasCanonicalizer = 1;
+  let hasVerifier = 1;
 }
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_from_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled", [Pure]> {
   let summary = "Apply scales from a scale tensor to the values in a value tensor";
@@ -2771,6 +2726,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
     Apply the scales from a scale tensor to the values in a value tensor, casting
     the result to the output type. The block dimension must be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
@@ -2794,6 +2751,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_to_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [Pure]> {
   let summary = "Calculate scale tensor values per block, output to separate scale and data tensors.";
@@ -2803,6 +2762,8 @@ def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [P
     scaled data values from an input tensor. The output tensors are cast to the
     specified scale and value types. The block dimension will be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 0135a651be481..4e14b27a421e4 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -23,10 +23,22 @@ using namespace mlir::tosa;
 // Type Compilance Definition
 //===----------------------------------------------------------------------===//
 
-typedef struct {
+struct TypeInfo {
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(mlir::TypeID()),
+        scaleTypeID(mlir::TypeID()), blockSize(0) {}
+
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth, mlir::TypeID valueTypeID,
+           mlir::TypeID scaleTypeID, uint32_t blockSize)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(valueTypeID),
+        scaleTypeID(scaleTypeID), blockSize(blockSize) {}
+
   mlir::TypeID typeID;
   uint32_t bitWidth;
-} TypeInfo;
+  mlir::TypeID valueTypeID;
+  mlir::TypeID scaleTypeID;
+  uint32_t blockSize;
+};
 
 enum CheckCondition {
   invalid,
@@ -70,6 +82,14 @@ class ProfileInfoDepot {
 
 private:
   TypeInfo convertTypeToInfo(Type type) {
+    if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type)) {
+      Type valueTy = blockScaledTy.getValueType();
+      Type scaleTy = blockScaledTy.getScaleType();
+      return {
+          type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
+          scaleTy.getTypeID(),
+          BlockShapeAttr::getBlockShapeValue(blockScaledTy.getBlockShape())};
+    }
     return {type.getTypeID(), tosa::getBitWidth(type)};
   }
 
@@ -128,7 +148,9 @@ class TosaProfileCompliance {
       const SmallVector<ArrayRef<T>> &specDefinedProfileSet);
 
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
-    return a.typeID == b.typeID && a.bitWidth == b.bitWidth;
+    return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
+           a.valueTypeID == b.valueTypeID && a.scaleTypeID == b.scaleTypeID &&
+           a.blockSize == b.blockSize;
   }
 
   // Find the required profiles or extensions from the compliance info according
@@ -145,7 +167,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<32> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 10ddd3438aedd..0a43f14865f56 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -103,15 +103,39 @@ def Tosa_MXInt8
   }];
 }
 
+def Tosa_MXFPValue
+    : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
+                "micro-scaling format number">;
+def Tosa_MXFPScale
+    : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+
+def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+  let summary = "Block scaled tensor element type.";
+
+  let description = [{
+    This does not specify an implementation type. A tensor of this type is a 
+    tensor which has block-scaled quantization applied.
+
+    This compound type is made up of 3 components:
+    `value_type` - The type of the data values in each block.
+    `scale_type` - The type of the scale value associated with each block.
+    `block_shape` - The size and axis of each block. Currently only supports
+      specifying the block size along the innermost dimension.
+  }];
+
+  let parameters = (ins Tosa_MXFPValue:$value_type, Tosa_MXFPScale:$scale_type,
+      EnumParameter<Tosa_BlockShape>:$block_shape);
+
+  let assemblyFormat =
+      "`<` $value_type```:```$scale_type```:```$block_shape `>`";
+}
+
 //===----------------------------------------------------------------------===//
 // Multi-category types.
 //===----------------------------------------------------------------------===//
-def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat, Tosa_MXInt8],
-                                "number">;
-
-def Tosa_MXFPNumber : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
-                                "micro-scaling format number">;
-def Tosa_MXFPScaleNumber : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat,
+                                Tosa_MXInt8, Tosa_BlockScaled],
+                               "number">;
 
 //===----------------------------------------------------------------------===//
 // TOSA Tensor Conformance
@@ -129,16 +153,29 @@ def AtLeastRankOne : And<[
   IsRankedTensorTypePred,
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
-class TosaTensorOf<
-    list<Type> allowedTypes, string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes, [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>], summary>;
-
-class TosaRankedTensorOf<
-    list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<allowedTypes, !listconcat([HasNo0Dimensions], preds), summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes, preds, summary>;
+def IsValidBlockScaledTensorType
+    : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
+            "verifyBlockScaledTensorType($_op, $_self))">;
+
+class TosaTensorOf<list<Type> allowedTypes,
+                   string summary = "tosa-conformant tensor">
+    : TensorOf<allowedTypes,
+               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+                IsValidBlockScaledTensorType],
+               summary>;
+
+class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                         string summary = "tosa-conformant ranked tensor">
+    : RankedTensorOf<
+          allowedTypes,
+          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
+          summary>;
+
+class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                           string summary = "tosa-conformant unranked tensor">
+    : UnrankedTensorOf<allowedTypes,
+                       !listconcat([IsValidBlockScaledTensorType], preds),
+                       summary>;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaRankedTensorOf<allowedTypes,
@@ -217,32 +254,28 @@ def Tosa_IndexTensor2D : AnyTypeOf<[
 def Tosa_TensorAtLeast1D : AnyTypeOf<[
   Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
 
-def Tosa_MXFPDataTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [3]>
-]>;
-def Tosa_MXFPScaleTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [3]>
-]>;
-def Tosa_MXFPDataTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [4]>
-]>;
-def Tosa_MXFPScaleTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [4]>
-]>;
-def Tosa_MXFPDataTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
-def Tosa_MXFPScaleTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPScaleNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
+def Tosa_MXFPDataTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+def Tosa_MXFPScaleTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+def Tosa_MXFPDataTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+def Tosa_MXFPScaleTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+def Tosa_MXFPDataTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
+def Tosa_MXFPScaleTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
index dc18fcaa04c8a..56e4901811dcb 100644
--- a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
@@ -56,6 +56,13 @@ TosaSpecificationVersion getMinVersion(const Extension &extension) {
   case Extension::int64:
   case Extension::mxfp_conv:
   case Extension::shape:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return TosaSpecificationVersion(1, 1, true);
   case Extension::none:
     return TosaSpecificationVersion(0, 0);
@@ -76,6 +83,13 @@ SmallVector<Profile, 2> getCooperativeProfiles(Extension ext) {
   case Extension::fft:
   case Extension::mxfp:
   case Extension::mxfp_conv:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return {Profile::pro_fp};
   case Extension::variable:
   case Extension::controlflow:
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 63d40ed4a95a5..99315a787f12e 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1184,12 +1184,50 @@ struct NonNarrowingCastsOptimization : public OpRewritePattern<tosa::CastOp> {
   }
 };
 
+struct CancellingBlockScaledCastsOptimization
+    : public OpRewritePattern<tosa::CastOp> {
+  using OpRewritePattern<tosa::CastOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(tosa::CastOp castOp,
+                                PatternRewriter &rewriter) const override {
+    const Value outerInput = castOp.getInput();
+    auto innerCastOp = outerInput.getDefiningOp<tosa::CastOp>();
+    if (!innerCastOp)
+      return rewriter.notifyMatchFailure(castOp,
+                                         "input must be a cast operation");
+
+    const Value innerInput = innerCastOp.getInput();
+    const auto innerInputTy = llvm::cast<ShapedType>(innerInput.getType());
+    const auto innerOutputTy = llvm::cast<ShapedType>(innerCastOp.getType());
+    const auto outerOutputTy = llvm::cast<ShapedType>(castOp.getType());
+
+    if (!llvm::isa<tosa::BlockScaledType>(innerInputTy.getElementType()))
+      return rewriter.notifyMatchFailure(
+          castOp, "inner cast input must have block scaled element type");
+
+    if (innerInputTy != outerOutputTy)
+      return rewriter.notifyMatchFailure(
+          castOp, "inner input type must match outer output type");
+
+    const Type innerOutputElemType = innerOutputTy.getElementType();
+    const bool isLosslessCast = isa<Float32Type>(innerOutputElemType);
+    if (!isLosslessCast)
+      return rewriter.notifyMatchFailure(
+          castOp, "avoid cancelling casts that could by lossy");
+
+    rewriter.replaceOp(castOp, innerInput);
+
+    return success();
+  }
+};
+
 void CastOp::getCanonicalizationPatterns(RewritePatternSet &results,
                                          MLIRContext *context) {
-  results.add<NonNarrowingCastsOptimization>(context);
+  results.add<NonNarrowingCastsOptimization,
+              CancellingBlockScaledCastsOptimization>(context);
 }
 
-struct CancellingBlockScaledCastsOptimization
+struct CancellingCastToFromBlockScaledOptimization
     : public OpRewritePattern<tosa::CastToBlockScaledOp> {
   using OpRewritePattern<tosa::CastToBlockScaledOp>::OpRewritePattern;
 
@@ -1235,7 +1273,7 @@ struct CancellingBlockScaledCastsOptimization
 
 void CastToBlockScaledOp::getCanonicalizationPatterns(
     RewritePatternSet &results, MLIRContext *context) {
-  results.add<CancellingBlockScaledCastsOptimization>(context);
+  results.add<CancellingCastToFromBlockScaledOptimization>(context);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index f05399cf6b00b..486431a562714 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -628,6 +628,8 @@ Value mlir::tosa::createPadConstTensor(OpBuilder &builder, Location loc,
 }
 
 unsigned mlir::tosa::getBitWidth(Type type) {
+  if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type))
+    return getBitWidth(blockScaledTy.getValueType());
   if (dyn_cast<tosa::mxint8Type>(type))
     return 8;
   return type.getIntOrFloatBitWidth();
@@ -734,6 +736,41 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
   return cast<IntegerType>(attrType).convertFromAttribute(attr, result);
 }
 
+//===----------------------------------------------------------------------===//
+// TOSA block scaling utilities.
+//===----------------------------------------------------------------------===//
+
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  const auto tensorType = llvm::cast<ShapedType>(type);
+  const BlockScaledType elemType =
+      llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
+  if (!elemType)
+    return success();
+
+  if (!tensorType.hasRank())
+    return success();
+
+  if (tensorType.getRank() == 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " does not support block scaling on scalar tensors";
+
+  const int64_t blockedDimension = tensorType.getShape().back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
+
+  const uint32_t blockSize =
+      BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+  if (blockedDimension % blockSize != 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " blocked dimension must be a multiple of block size, got "
+           << blockedDimension << " and block size " << blockSize;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -5004,6 +5041,34 @@ LogicalResult RescaleOp::inferReturnTypeComponents(
   return success();
 }
 
+LogicalResult CastOp::verify() {
+  const ShapedType inputType = llvm::cast<ShapedType>(getInput().getType());
+  const ShapedType outputType = llvm::cast<ShapedType>(getType());
+  const Type inputElementType = inputType.getElementType();
+  const Type outputElementType = outputType.getElementType();
+
+  const bool inputIsBlockScaled = llvm::isa<BlockScaledType>(inputElementType);
+  const bool outputIsBlockScaled =
+      llvm::isa<BlockScaledType>(outputElementType);
+  if (!inputIsBlockScaled && !outputIsBlockScaled)
+    return success();
+
+  if (inputIsBlockScaled && outputIsBlockScaled)
+    return emitOpError()
+           << "requires exactly one of input or output to have block scaled "
+              "element type";
+
+  const Type scalarElementType =
+      inputIsBlockScaled ? outputElementType : inputElementType;
+  if (!llvm::isa<FloatType>(scalarElementType))
+    return emitOpError()
+           << "requires non-block-scaled element type to be floating-point "
+              "when casting to or from block scaled element type, got "
+           << scalarElementType;
+
+  return success();
+}
+
 LogicalResult CastFromBlockScaledOp::inferReturnTypeComponents(
     MLIRContext *context, ::std::optional<Location> location,
     CastFromBlockScaledOp::Adaptor adaptor,
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 9511d4da89dbd..9fbdfcc1d6690 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Tosa/IR/TosaProfileCompliance.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/raw_ostream.h"
 
 using namespace mlir;
 using namespace mlir::tosa;
@@ -27,12 +28,37 @@ TosaProfileCompliance::TosaProfileCompliance() {
   const TypeInfo fp8e5m2T = {mlir::Float8E5M2Type::getTypeID(), 8};
 
   // micro-scaling formats
+  // Note: these types exist to suppport the deprecated block_scaled operations
+  // and can be removed once those operations are removed.
   const TypeInfo fp6e2m3T = {mlir::Float6E2M3FNType::getTypeID(), 6};
   const TypeInfo fp6e3m2T = {mlir::Float6E3M2FNType::getTypeID(), 6};
   const TypeInfo fp4e2m1T = {mlir::Float4E2M1FNType::getTypeID(), 4};
   const TypeInfo fp8ue8m0T = {mlir::Float8E8M0FNUType::getTypeID(), 8};
   const TypeInfo mxint8T = {mlir::tosa::mxint8Type::getTypeID(), 8};
 
+  // Block scaled formats
+  const TypeID blockScaledID = mlir::tosa::BlockScaledType::getTypeID();
+  const TypeID fp4e2m1ID = mlir::Float4E2M1FNType::getTypeID();
+  const TypeID fp6e2m3ID = mlir::Float6E2M3FNType::getTypeID();
+  const TypeID fp6e3m2ID = mlir::Float6E3M2FNType::getTypeID();
+  const TypeID fp8e4m3ID = mlir::Float8E4M3FNType::getTypeID();
+  const TypeID fp8e5m2ID = mlir::Float8E5M2Type::getTypeID();
+  const TypeID fp8ue8m0ID = mlir::Float8E8M0FNUType::getTypeID();
+  const TypeID mxint8ID = mlir::tosa::mxint8Type::getTypeID();
+
+  const TypeInfo bs32_fp8ue8m0_fp4e2m1T = {blockScaledID, 4, fp4e2m1ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp6e2m3T = {blockScaledID, 6, fp6e2m3ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp6e3m2T = {blockScaledID, 6, fp6e3m2ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp8e4m3T = {blockScaledID, 8, fp8e4m3ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp8e5m2T = {blockScaledID, 8, fp8e5m2ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_mxint8T = {blockScaledID, 8, mxint8ID,
+                                          fp8ue8m0ID, 32};
+
 // The profile-based compliance content below is auto-generated by a script
 // in https://github.com/arm/tosa-specification
 #include "mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc"
@@ -669,31 +695,47 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<7>
+llvm::SmallString<32>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
-  if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
-    return {"i" + llvm::utostr(typeInfo.bitWidth)};
-  }
-  if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
-    return {"f16"};
-  } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
-    return {"f32"};
-  } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
-    return {"bf16"};
-  } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
-    return {"fp8e4m3"};
-  } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
-    return {"fp8e5m2"};
-  } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
-    return {"fp6e2m3"};
-  } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
-    return {"fp6e3m2"};
-  } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
-    return {"fp4e2m1"};
-  } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
-    return {"fp8e8m0"};
-  } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
-    return {"mxint8"};
+  const auto stringifyScalarTypeInfo =
+      [](const TypeInfo &typeInfo) -> llvm::SmallString<32> {
+    if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
+      return {"i" + llvm::utostr(typeInfo.bitWidth)};
+    }
+    if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
+      return {"f16"};
+    } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
+      return {"f32"};
+    } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
+      return {"bf16"};
+    } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
+      return {"fp8e4m3"};
+    } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
+      return {"fp8e5m2"};
+    } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
+      return {"fp6e2m3"};
+    } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
+      return {"fp6e3m2"};
+    } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
+      return {"fp4e2m1"};
+    } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
+      return {"fp8e8m0"};
+    } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
+      return {"mxint8"};
+    }
+    llvm_unreachable("unknown type");
+  };
+
+  if (typeInfo.typeID == tosa::BlockScaledType::getTypeID()) {
+    TypeInfo valueInfo = {typeInfo.valueTypeID, typeInfo.bitWidth};
+    TypeInfo scaleInfo = {typeInfo.scaleTypeID, 8};
+    llvm::SmallString<32> result;
+    llvm::raw_svector_ostream os(result);
+    os << "bs" << typeInfo.blockSize << "_"
+       << stringifyScalarTypeInfo(scaleInfo) << "_"
+       << stringifyScalarTypeInfo(valueInfo);
+    return result;
   }
-  llvm_unreachable("unknown type");
+
+  return stringifyScalarTypeInfo(typeInfo);
 }
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 34ac84d237f45..8fa2ddf856c7d 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -1482,7 +1482,7 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
     }
   } else if (isa<tosa::shapeType>(type))
     return true;
-  else if (isa<tosa::mxint8Type>(type))
+  else if (isa<tosa::mxint8Type, tosa::BlockScaledType>(type))
     return true;
   return false;
 }
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 81294f2c0c308..450b4556c7b16 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -627,10 +627,10 @@ func.func @test_resize(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x64x64x8xf32> {
 }
 
 // -----
-// CHECK-LABEL: cast
-func.func @test_cast1(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
+// CHECK-LABEL: test_cast
+func.func @test_cast(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.cast %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 2cd040f056db8..d454ca831cf0d 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1651,6 +1651,73 @@ func.func @test_canonicalize_non_narrowing_cast_i8_to_f8E4M3FN_unsupported(%arg0
 
 // -----
 
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1
+// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf32>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2
+// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32xf32>
+  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*xf32>
+  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3
+// CHECK: tosa.cast
+// CHECK: tosa.cast
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf8E4M3FN>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1
 // CHECK: return %arg0, %arg1 : tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>
 func.func @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1(%arg0: tensor<15x3x2x256xf4E2M1FN>, %arg1: tensor<15x3x2x8xf8E8M0FNU>) -> (tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>) {
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index 5e8111061cb3a..eca5c9ef1b14e 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2266,3 +2266,35 @@ func.func @test_shape_func_output() -> !tosa.shape<4> {
   %cst = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   return %cst : !tosa.shape<4>
 }
+
+// -----
+
+func.func @test_cast_f32_plain_fp4(%arg0: tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (f32,fp4e2m1)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN>
+  return %0 : tensor<4x32xf4E2M1FN>
+}
+
+// -----
+
+func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp4e2m1,bs32_fp8e8m0_fp4e2m1), did you mean (fp8e4m3,bs32_fp8e8m0_fp4e2m1)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (bs32_fp8e8m0_fp4e2m1,fp6e2m3), did you mean (bs32_fp8e8m0_fp4e2m1,fp8e4m3)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN>
+  return %0 : tensor<4x32xf6E2M3FN>
+}
+
+// -----
+
+func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp6e3m2,bs32_fp8e8m0_mxint8), did you mean (fp8e4m3,bs32_fp8e8m0_mxint8)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 295d7172bc2c4..9d4e59a9a9c6f 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -295,6 +295,34 @@ func.func @test_cast_f32_bf16(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xbf16
   return %0 : tensor<13x21x3xbf16>
 }
 
+// -----
+func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  return %0 : tensor<4x32xf32>
+}
+
+// -----
+func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [fp8e4m3, mx_common, mx_int8] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
 // -----
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index c7f4c67072d06..86212177d34be 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1059,6 +1059,32 @@ func.func @test_cast3(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3x!quant.unifo
   return %0 : tensor<13x21x3x!quant.uniform<i16:f32, 0.078431375324726104:128>>
 }
 
+// -----
+// CHECK-LABEL: test_cast_to_block_scaled
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+// CHECK-LABEL: test_cast_from_block_scaled
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg2: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
+  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+  return %0, %1, %2 : tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>
+}
+
+// -----
+// CHECK-LABEL: test_cast_block_scaled_dynamic
+func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
+  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<?x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x?xf32>
+  return %0, %1 : tensor<?x32xf32>, tensor<4x?xf32>
+}
+
 // -----
 // CHECK-LABEL: rescale
 func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439347028732:127>>) -> tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>> {
diff --git a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
index a0c59c0c4bb3b..558a599fbbc1e 100644
--- a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
@@ -1,11 +1,26 @@
-// RUN: mlir-opt %s -split-input-file -tosa-attach-target="profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,dynamic level=none" | FileCheck %s --check-prefix=CHECK-ALL
+// DEFINE: %{core_extensions} = int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable
+// DEFINE: %{rounding_extensions} = controlflow,doubleround,inexactround,dynamic
+// DEFINE: %{mx_float_extensions} = mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2
+// DEFINE: %{mx_int_extensions} = mx_fp8e4m3,mx_fp8e5m2,mx_int8
+// DEFINE: %{all_extensions} = %{core_extensions},%{rounding_extensions},%{mx_float_extensions},%{mx_int_extensions}
+// DEFINE: %{all_target} = specification_version=1.1.draft level=none \
+// DEFINE:    profiles=pro_int,pro_fp extensions=%{all_extensions}
+
+// RUN: mlir-opt %s -split-input-file -tosa-attach-target="%{all_target}" | FileCheck %s --check-prefix=CHECK-ALL
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="level=8k" | FileCheck %s --check-prefix=CHECK-LVL-8K
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target | FileCheck %s --check-prefix=CHECK-DEFAULT
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="specification_version=1.1.draft" | FileCheck %s --check-prefix=CHECK-VERSION-1P1
 
 // -----
 
-// CHECK-ALL: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = none, profiles = [pro_int, pro_fp], extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft, variable, controlflow, doubleround, inexactround, dynamic]>}
+// CHECK-ALL: module attributes {
+// CHECK-ALL-SAME: tosa.target_env = #tosa.target_env<specification_version = "1.1.draft",
+// CHECK-ALL-SAME: level = none,
+// CHECK-ALL-SAME: profiles = [pro_int, pro_fp],
+// CHECK-ALL-SAME: extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft,
+// CHECK-ALL-SAME: variable, controlflow, doubleround, inexactround, dynamic,
+// CHECK-ALL-SAME: mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2,
+// CHECK-ALL-SAME: mx_fp8e4m3, mx_fp8e5m2, mx_int8]>}
 // CHECK-LVL-8K: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-DEFAULT: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-VERSION-1P1: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.1.draft", level = "8k", profiles = [], extensions = []>}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index 7ff883e8e5431..c7bd051399ae8 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -135,6 +135,14 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
 func.func @test_dyanmic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   // expected-error at +1 {{'tosa.argmax' op failed level check: operand shape dimension cannot be dynamic when targeting TOSA specification version 1.0 or below}}
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 1bac8bacbaf40..f45ba0a5b7ef5 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
+// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape,mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2,mx_fp8e4m3,mx_fp8e5m2,mx_int8" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
 
 // -----
 
@@ -352,6 +352,34 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+// CHECK-LABEL: test_cast_to_block_scaled_types
+func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
+  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: test_cast_from_block_scaled_types
+func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %mxint8: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf16>
+  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
+  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E4M3FN>
+  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+  return %0, %3, %4, %5, %6 : tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>
+}
+
+// -----
+
 // CHECK-LABEL: test_dynamic_dims
 func.func @test_dynamic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 9b5faf575971b..14a48d5e10ea0 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1517,6 +1517,48 @@ func.func @test_cast_to_block_scaled_block_size_mismatch(%arg0: tensor<4x32xf32>
 
 // -----
 
+func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i8'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i32'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32>
+  return %0 : tensor<4x32xi32>
+}
+
+// -----
+
+func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op requires exactly one of input or output to have block scaled element type}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32> {
+  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
+  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32>
+  return %0 : tensor<1x16x31xf32>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32> {
+  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' does not support block scaling on scalar tensors}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
+  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32>
+  return %0 : tensor<f32>
+}
+
+// -----
+
 func.func @test_clamp_quantized(%arg0:tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) {
     // expected-error at +1 {{'tosa.clamp' op min/max attributes types are incompatible with input/output element types.}}
     %0 = tosa.clamp %arg0 {max_val = 127 : i8, min_val = -128 : i8} : (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>

>From 06c28ec21f372f40f210ac05d76fab6ca074c64a Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 25 Jun 2026 17:40:56 +0100
Subject: [PATCH 2/3] Align block_scale type elements with spec order

Aligns block_scale type with the TOSA specification
definition.

Change-Id: I4d4a6a51e5582a93c8b272fd4f6baf30da3d8e87
---
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 16 +++--
 mlir/test/Dialect/Tosa/canonicalize.mlir      | 58 +++++++++----------
 mlir/test/Dialect/Tosa/invalid.mlir           | 16 ++---
 mlir/test/Dialect/Tosa/invalid_extension.mlir | 22 +++----
 mlir/test/Dialect/Tosa/ops.mlir               | 24 ++++----
 .../tosa-validation-version-1p0-invalid.mlir  |  6 +-
 .../tosa-validation-version-1p1-valid.mlir    | 34 +++++------
 mlir/test/Dialect/Tosa/verifier.mlir          | 32 +++++-----
 8 files changed, 106 insertions(+), 102 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 0a43f14865f56..5ab2c456f365b 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -113,21 +113,25 @@ def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
   let summary = "Block scaled tensor element type.";
 
   let description = [{
-    This does not specify an implementation type. A tensor of this type is a 
+    This does not specify an implementation type. A tensor of this type is a
     tensor which has block-scaled quantization applied.
 
     This compound type is made up of 3 components:
-    `value_type` - The type of the data values in each block.
-    `scale_type` - The type of the scale value associated with each block.
     `block_shape` - The size and axis of each block. Currently only supports
       specifying the block size along the innermost dimension.
+    `scale_type` - The type of the scale value associated with each block.
+    `value_type` - The type of the data values in each block.
+
   }];
 
-  let parameters = (ins Tosa_MXFPValue:$value_type, Tosa_MXFPScale:$scale_type,
-      EnumParameter<Tosa_BlockShape>:$block_shape);
+  let parameters = (ins
+    EnumParameter<Tosa_BlockShape>:$block_shape,
+    Tosa_MXFPScale:$scale_type,
+    Tosa_MXFPValue:$value_type
+  );
 
   let assemblyFormat =
-      "`<` $value_type```:```$scale_type```:```$block_shape `>`";
+      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index d454ca831cf0d..d2da93ee95138 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1652,21 +1652,21 @@ func.func @test_canonicalize_non_narrowing_cast_i8_to_f8E4M3FN_unsupported(%arg0
 // -----
 
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1
-// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf32>
-  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256xf32>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2
-// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
-  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
 }
 
 // -----
@@ -1674,11 +1674,11 @@ func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: t
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
-  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
 }
 
 // -----
@@ -1686,11 +1686,11 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_differen
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32xf32>
-  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>> {
+  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32xf32>
+  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  return %1 : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
 }
 
 // -----
@@ -1698,11 +1698,11 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_differen
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*xf32>
-  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*xf32>
+  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
 }
 
 // -----
@@ -1710,10 +1710,10 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3
 // CHECK: tosa.cast
 // CHECK: tosa.cast
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf8E4M3FN>
-  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256xf8E4M3FN>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index eca5c9ef1b14e..d0336da15cee8 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2277,24 +2277,24 @@ func.func @test_cast_f32_plain_fp4(%arg0: tensor<4x32xf32>) -> tensor<4x32xf4E2M
 
 // -----
 
-func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp4e2m1,bs32_fp8e8m0_fp4e2m1), did you mean (fp8e4m3,bs32_fp8e8m0_fp4e2m1)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
-func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN> {
+func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (bs32_fp8e8m0_fp4e2m1,fp6e2m3), did you mean (bs32_fp8e8m0_fp4e2m1,fp8e4m3)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN>
   return %0 : tensor<4x32xf6E2M3FN>
 }
 
 // -----
 
-func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp6e3m2,bs32_fp8e8m0_mxint8), did you mean (fp8e4m3,bs32_fp8e8m0_mxint8)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 9d4e59a9a9c6f..ca09ca95066ef 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -296,31 +296,31 @@ func.func @test_cast_f32_bf16(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xbf16
 }
 
 // -----
-func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
-func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32> {
+func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
   return %0 : tensor<4x32xf32>
 }
 
 // -----
-func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
-func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [fp8e4m3, mx_common, mx_int8] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 86212177d34be..46595c1e4ea73 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1061,27 +1061,27 @@ func.func @test_cast3(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3x!quant.unifo
 
 // -----
 // CHECK-LABEL: test_cast_to_block_scaled
-func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
 // CHECK-LABEL: test_cast_from_block_scaled
-func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg2: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
-  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xbf16>
+  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E5M2>
   return %0, %1, %2 : tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>
 }
 
 // -----
 // CHECK-LABEL: test_cast_block_scaled_dynamic
-func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
-  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<?x32xf32>
-  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x?xf32>
+func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
+  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<?x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x?xf32>
   return %0, %1 : tensor<?x32xf32>, tensor<4x?xf32>
 }
 
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index c7bd051399ae8..67237848d6171 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -135,10 +135,10 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
-func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index f45ba0a5b7ef5..ecfe8e1635944 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -353,28 +353,28 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 // -----
 
 // CHECK-LABEL: test_cast_to_block_scaled_types
-func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
-  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
 
 // CHECK-LABEL: test_cast_from_block_scaled_types
-func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %mxint8: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
-  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf16>
-  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
-  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E4M3FN>
-  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %mxint8: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xf32>
+  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x32xf32>
+  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x32xf16>
+  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xbf16>
+  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E4M3FN>
+  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf8E5M2>
   return %0, %3, %4, %5, %6 : tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>
 }
 
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 14a48d5e10ea0..b2c5527b058aa 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1517,43 +1517,43 @@ func.func @test_cast_to_block_scaled_block_size_mismatch(%arg0: tensor<4x32xf32>
 
 // -----
 
-func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i8'}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
-func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32> {
+func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32> {
   // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i32'}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32>
   return %0 : tensor<4x32xi32>
 }
 
 // -----
 
-func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
   // expected-error at +1 {{'tosa.cast' op requires exactly one of input or output to have block scaled element type}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
 }
 
 // -----
 
-func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
-  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32>
+func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
+  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
 }
 
 // -----
 
-func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' does not support block scaling on scalar tensors}}
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
-  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32>
+func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
+  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>
 }
 

>From 3de5539f9bc2d9a1a05038714e0923e78f78b3a9 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Fri, 3 Jul 2026 13:25:36 +0100
Subject: [PATCH 3/3] TypeInfo blockSize -> blockShape

Change-Id: I5c66f8a8fb5934bcf2a8e2988d5491ef0ce4b90d
---
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   | 16 ++++++-------
 .../Tosa/Transforms/TosaProfileCompliance.cpp | 24 ++++++++++++-------
 2 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 4e14b27a421e4..34ccc675c8243 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -26,18 +26,18 @@ using namespace mlir::tosa;
 struct TypeInfo {
   TypeInfo(mlir::TypeID typeID, uint32_t bitWidth)
       : typeID(typeID), bitWidth(bitWidth), valueTypeID(mlir::TypeID()),
-        scaleTypeID(mlir::TypeID()), blockSize(0) {}
+        scaleTypeID(mlir::TypeID()), blockShape(std::nullopt) {}
 
   TypeInfo(mlir::TypeID typeID, uint32_t bitWidth, mlir::TypeID valueTypeID,
-           mlir::TypeID scaleTypeID, uint32_t blockSize)
+           mlir::TypeID scaleTypeID, tosa::BlockShape blockShape)
       : typeID(typeID), bitWidth(bitWidth), valueTypeID(valueTypeID),
-        scaleTypeID(scaleTypeID), blockSize(blockSize) {}
+        scaleTypeID(scaleTypeID), blockShape(blockShape) {}
 
   mlir::TypeID typeID;
   uint32_t bitWidth;
   mlir::TypeID valueTypeID;
   mlir::TypeID scaleTypeID;
-  uint32_t blockSize;
+  std::optional<tosa::BlockShape> blockShape;
 };
 
 enum CheckCondition {
@@ -85,10 +85,8 @@ class ProfileInfoDepot {
     if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type)) {
       Type valueTy = blockScaledTy.getValueType();
       Type scaleTy = blockScaledTy.getScaleType();
-      return {
-          type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
-          scaleTy.getTypeID(),
-          BlockShapeAttr::getBlockShapeValue(blockScaledTy.getBlockShape())};
+      return {type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
+              scaleTy.getTypeID(), blockScaledTy.getBlockShape()};
     }
     return {type.getTypeID(), tosa::getBitWidth(type)};
   }
@@ -150,7 +148,7 @@ class TosaProfileCompliance {
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
     return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
            a.valueTypeID == b.valueTypeID && a.scaleTypeID == b.scaleTypeID &&
-           a.blockSize == b.blockSize;
+           a.blockShape == b.blockShape;
   }
 
   // Find the required profiles or extensions from the compliance info according
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 9fbdfcc1d6690..4f804d32775f7 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -47,17 +47,22 @@ TosaProfileCompliance::TosaProfileCompliance() {
   const TypeID mxint8ID = mlir::tosa::mxint8Type::getTypeID();
 
   const TypeInfo bs32_fp8ue8m0_fp4e2m1T = {blockScaledID, 4, fp4e2m1ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp6e2m3T = {blockScaledID, 6, fp6e2m3ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp6e3m2T = {blockScaledID, 6, fp6e3m2ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp8e4m3T = {blockScaledID, 8, fp8e4m3ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp8e5m2T = {blockScaledID, 8, fp8e5m2ID,
-                                           fp8ue8m0ID, 32};
-  const TypeInfo bs32_fp8ue8m0_mxint8T = {blockScaledID, 8, mxint8ID,
-                                          fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_mxint8T = {
+      blockScaledID, 8, mxint8ID, fp8ue8m0ID, tosa::BlockShape::BLOCK_SHAPE_32};
 
 // The profile-based compliance content below is auto-generated by a script
 // in https://github.com/arm/tosa-specification
@@ -731,8 +736,9 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
     TypeInfo scaleInfo = {typeInfo.scaleTypeID, 8};
     llvm::SmallString<32> result;
     llvm::raw_svector_ostream os(result);
-    os << "bs" << typeInfo.blockSize << "_"
-       << stringifyScalarTypeInfo(scaleInfo) << "_"
+    os << "bs"
+       << tosa::BlockShapeAttr::getBlockShapeValue(typeInfo.blockShape.value())
+       << "_" << stringifyScalarTypeInfo(scaleInfo) << "_"
        << stringifyScalarTypeInfo(valueInfo);
     return result;
   }


From llvmlistbot at llvm.org  Fri Jul  3 05:28:38 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:28:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47aaf6.14d9b9dc.140829.7700@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 5f144a05466b4eb12257e8840e0173225de0e8b3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 1/7] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index c30dd3b07d028..9d810f2613a26 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1092,7 +1092,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 6e1ed05c9d0d0..9aa28ca825f73 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -699,6 +699,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From cde91e515bc3eb100a6524cda7b07417f5a6b4c5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 2/7] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From 0f0b91d8f9627c04fdddb1d2ca9280238de28f27 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 3/7] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9aa28ca825f73..777e5d72ac4b1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -741,7 +741,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -758,22 +757,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -808,7 +791,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -825,22 +807,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From e1c107c642b6c519e7439657e970c9ae9e5a408e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 4/7] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 1da6401974970bf094954cd63c40a91a4b038492 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 5/7] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 777e5d72ac4b1..162a92782380b 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,60 +749,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -818,6 +812,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 9f52cbd5aaa9349b22f1da0493dd306b360569f6 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 6/7] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 162a92782380b..9c6039eeefc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,54 +749,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -812,9 +828,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From ad9430390f24ba3c8c730f1c78ff4afc96534ffb Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 7/7] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9c6039eeefc2c..8754149801124 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -805,9 +805,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }


From llvmlistbot at llvm.org  Fri Jul  3 05:29:07 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:29:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab13.13be6df8.3d8759.4879@mx.google.com>


================
@@ -101,6 +101,7 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+
----------------
Wolfram70 wrote:

Thanks for the catch!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:29:17 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:29:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab1d.13be6df8.3d8759.4e23@mx.google.com>


================
@@ -671,4 +671,78 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths:
+      f64 -> f32, f16, bf16
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
----------------
Wolfram70 wrote:

Done

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:29:28 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:29:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab28.1a4df1a5.118deb.c720@mx.google.com>


================
@@ -671,4 +671,78 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths:
+      f64 -> f32, f16, bf16
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
+
+    The `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
----------------
Wolfram70 wrote:

Makes sense, fixed!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:29:43 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:29:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab37.92ea0e2a.21f97.ce2a@mx.google.com>


================
@@ -1709,6 +1711,685 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// NVGPUConvertFPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
----------------
Wolfram70 wrote:

Fixed

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:30:04 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 05:30:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47ab4c.2020f6c2.1fdf61.fc1f@mx.google.com>


================

----------------
lhutton1 wrote:

Good catch thanks! I just forgot to update this when blockSize -> blockShape some time ago

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 05:30:04 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 05:30:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47ab4c.faf16b37.1e0c59.ba13@mx.google.com>


================

----------------
lhutton1 wrote:

I'd prefer to do this as a follow-up if that's okay?

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 05:30:18 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:30:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab5a.bf38ae28.358429.d641@mx.google.com>


================
@@ -1709,6 +1711,685 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// NVGPUConvertFPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class FPTruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
+  BF16x2_TO_F4x2,
+};
+
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
+};
+
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
+    // f32 source
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
+    // f16 source
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
+    // bf16 source
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
+};
+
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kFPTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry;
+  }
+  return std::nullopt;
+}
+
+/// Extract a single element from a vector.
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+/// Create a typed NVVM truncation conversion.
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPTruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case FPTruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case FPTruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled FPTruncConvOp");
+}
+
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
+
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
+      }
+
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+    }
+
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+  }
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPUConvertFPExtOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+enum class FPExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
+  F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
+};
+
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
+};
+
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
+};
+
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kFPExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry;
+  }
+  return std::nullopt;
+}
----------------
Wolfram70 wrote:

Combined into one templated function in the latest revision, thanks!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:30:44 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:30:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add convert.fpext and
	convert.fptrunc Ops (PR #199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ab74.ff8885e8.118815.d6e4@mx.google.com>


================
@@ -699,6 +699,138 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult ConvertFPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
----------------
Wolfram70 wrote:

That looks better, fixed!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:32:03 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:32:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47abc3.d5797644.146403.64af@mx.google.com>


https://github.com/Wolfram70 edited https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:32:54 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 05:32:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47abf6.3157459c.24b6e5.bcdf@mx.google.com>


================

----------------
IanTaylerLessa-arm wrote:

Sounds good to me!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 05:33:22 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 03 Jul 2026 05:33:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47ac12.f06f4994.168928.6a76@mx.google.com>


IanTaylerLessa-arm wrote:

LGTM. Thanks!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 05:34:34 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:34:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ac5a.f06f4994.168928.9524@mx.google.com>


https://github.com/Wolfram70 edited https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:39:09 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:39:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ad6d.f375f8af.1713f5.0847@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 5f144a05466b4eb12257e8840e0173225de0e8b3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 1/8] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index c30dd3b07d028..9d810f2613a26 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1092,7 +1092,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 6e1ed05c9d0d0..9aa28ca825f73 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -699,6 +699,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From cde91e515bc3eb100a6524cda7b07417f5a6b4c5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 2/8] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From 0f0b91d8f9627c04fdddb1d2ca9280238de28f27 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 3/8] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9aa28ca825f73..777e5d72ac4b1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -741,7 +741,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -758,22 +757,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -808,7 +791,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -825,22 +807,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From e1c107c642b6c519e7439657e970c9ae9e5a408e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 4/8] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 1da6401974970bf094954cd63c40a91a4b038492 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 5/8] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 777e5d72ac4b1..162a92782380b 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,60 +749,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -818,6 +812,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 9f52cbd5aaa9349b22f1da0493dd306b360569f6 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 6/8] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 162a92782380b..9c6039eeefc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,54 +749,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -812,9 +828,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From ad9430390f24ba3c8c730f1c78ff4afc96534ffb Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 7/8] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9c6039eeefc2c..8754149801124 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -805,9 +805,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 911eacdebc316503b83e25f06615e252e5f812ba Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 8/8] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8754149801124..abc50a807fe5d 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -793,10 +793,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir


From llvmlistbot at llvm.org  Fri Jul  3 05:39:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 05:39:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 22e5273 - [mlir][OpenMP] Change device
 declare target functions to hidden visibility (#207234)
Message-ID: <6a47ad84.65dfdb71.1ca534.ff9b@mx.google.com>


Author: Robert Imschweiler
Date: 2026-07-03T14:39:27+02:00
New Revision: 22e5273cc3334906fc18fce2aebb74986259e31d

URL: https://github.com/llvm/llvm-project/commit/22e5273cc3334906fc18fce2aebb74986259e31d
DIFF: https://github.com/llvm/llvm-project/commit/22e5273cc3334906fc18fce2aebb74986259e31d.diff

LOG: [mlir][OpenMP] Change device declare target functions to hidden visibility (#207234)

During OpenMP lowering, globally visible device functions are emitted.
These functions might not be kernels themselves, but are designed to
only be called in a kernel context. However, if they are unused, and not
inlined, and reference LDS, the AMDGPU ISel emits lots of misleading
warnings related to "local memory global used by non-kernel function".
Fix by changing visibility from external+default to external+hidden,
which allows DCE to just remove the functions.

Claude assisted with this patch.

Added: 
    mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir

Modified: 
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
    mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
    mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
    mlir/test/Target/LLVMIR/openmp-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index eb34cabdf20f8..a287644207724 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8819,6 +8819,21 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
         // a deleted block.
         ompBuilder->Builder.ClearInsertionPoint();
         ompBuilder->Builder.SetCurrentDebugLocation(llvm::DebugLoc());
+      } else if (llvm::Function *llvmFunc =
+                     moduleTranslation.lookupFunction(funcOp.getName())) {
+        // Device-side declare target functions are externally visible by
+        // default so they can be referenced from other device translation
+        // units. That also prevents the offload LTO from internalizing and
+        // deleting them when they end up unused in the final device image.
+        // Such dead functions can still reference internal LDS and trigger
+        // spurious "local memory global used by non-kernel function" backend
+        // warnings. Marking them hidden keeps the symbol usable within the
+        // device image's linkage unit while letting LTO drop it when nothing
+        // references it; symbols that must stay reachable (e.g. via an offload
+        // entry that takes their address) are kept alive by that reference.
+        if (!llvmFunc->isDeclaration() && llvmFunc->hasExternalLinkage() &&
+            llvmFunc->getVisibility() == llvm::GlobalValue::DefaultVisibility)
+          llvmFunc->setVisibility(llvm::GlobalValue::HiddenVisibility);
       }
     }
     return success();

diff  --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
new file mode 100644
index 0000000000000..f5f3c9f419f0b
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-func-visibility.mlir
@@ -0,0 +1,38 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Device-side `declare target` functions are externally visible by default so
+// that they can be referenced from other device translation units. They are
+// emitted with hidden visibility so that the offload LTO can internalize and
+// delete them when they turn out to be unused in the final device image. This
+// avoids leaving dead functions behind that may, for example, reference LDS and
+// trigger spurious backend diagnostics.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: define hidden void @device_any()
+  llvm.func @device_any() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // CHECK: define hidden void @device_nohost()
+  llvm.func @device_nohost() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function with an explicitly requested (non-default) visibility is left
+  // untouched.
+  // CHECK: define protected void @device_protected()
+  llvm.func protected @device_protected() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>} {
+    llvm.return
+  }
+
+  // A function that is not declare target is unaffected.
+  // CHECK: define void @not_declare_target()
+  llvm.func @not_declare_target() {
+    llvm.return
+  }
+
+  // A declaration (no definition) is left untouched: there is nothing to
+  // internalize, and hiding it could over-constrain the symbol's visibility.
+  // CHECK: declare void @device_decl()
+  llvm.func @device_decl() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to)>}
+}

diff  --git a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
index cdebebc3ed233..6fccc0127b347 100644
--- a/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-device-shared-mem.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
 module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memory_space", 5 : ui32>>, llvm.data_layout = "e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-p7:160:256:256:32-p8:128:128:128:48-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-G1-ni:7:8:9", llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
-  // CHECK-LABEL: define void @device_shared_mem(
+  // CHECK-LABEL: define hidden void @device_shared_mem(
   // CHECK-SAME:  i32 %[[N0:.*]], i64 %[[N1:.*]])
   llvm.func @device_shared_mem(%n0: i32, %n1: i64) attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to)>} {
     // CHECK:      %[[CAST_N0:.*]] = zext i32 %[[N0]] to i64

diff  --git a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
index d84641ff9c99b..f120dde8131d6 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop-collapsed.mlir
@@ -22,7 +22,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC_COLLAPSED_WSLOOP:.*]](ptr %[[ARG0:.*]])
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr),
 // CHECK-SAME: ptr @[[COLLAPSED_WSLOOP_BODY_FN:.*]], ptr %[[STRUCT_ARG:.*]], i32 10000,
 // CHECK-SAME: i32 %[[NUM_THREADS:.*]], i8 0)

diff  --git a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
index 7be635f46111b..04458af9654c3 100644
--- a/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-wsloop.mlir
@@ -31,7 +31,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
   }
 }
 
-// CHECK: define void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
+// CHECK: define hidden void @[[FUNC0:.*]](ptr %[[ARG0:.*]])
 // CHECK:   %[[STRUCTARG:.*]] = alloca { ptr }, align 8, addrspace(5)
 // CHECK:   %[[STRUCTARG_ASCAST:.*]] = addrspacecast ptr addrspace(5) %[[STRUCTARG]] to ptr
 // CHECK:   %[[GEP:.*]] = getelementptr { ptr }, ptr addrspace(5) %[[STRUCTARG]], i32 0, i32 0
@@ -45,7 +45,7 @@ module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<"dlti.alloca_memo
 // CHECK:   %[[GEP3:.*]] = getelementptr [10 x i32], ptr %[[LOADGEP]], i32 0, i32 %[[TMP2:.*]]
 // CHECK:   store i32 %[[VAL0:.*]], ptr %[[GEP3]], align 4
 
-// CHECK: define void @[[FUNC_EMPTY_WSLOOP:.*]]()
+// CHECK: define hidden void @[[FUNC_EMPTY_WSLOOP:.*]]()
 // CHECK:   call void @__kmpc_for_static_loop_4u(ptr addrspacecast (ptr addrspace(1) @[[GLOB2:[0-9]+]] to ptr), ptr @[[LOOP_EMPTY_BODY_FN:.*]], ptr null, i32 10, i32 %[[NUM_THREADS:.*]], i32 0, i8 0)
 
 // CHECK: define internal void @[[LOOP_EMPTY_BODY_FN]](i32 %[[LOOP_CNT:.*]])

diff  --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 790c0dfe9b12e..444918fb1a9e3 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3661,7 +3661,7 @@ module attributes {omp.is_target_device = false} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3683,7 +3683,7 @@ module attributes {omp.is_target_device = true} {
 // -----
 
 module attributes {omp.is_target_device = true} {
-  // CHECK: define void @filter_nohost
+  // CHECK: define hidden void @filter_nohost
   llvm.func @filter_nohost() -> ()
       attributes {
         omp.declare_target =
@@ -3993,7 +3993,7 @@ llvm.func @omp_groupprivate_device() attributes {
 // CHECK-DAG: @nohost = internal global i32 undef
 // CHECK-DAG: @[[SHARED_ANY:any.*]] = internal addrspace(3) global i32 poison
 // CHECK-DAG: @[[SHARED_NOHOST:nohost.*]] = internal addrspace(3) global i32 poison
-// CHECK: define void @omp_groupprivate_device()
+// CHECK: define hidden void @omp_groupprivate_device()
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_ANY]], align 4
 // CHECK: store i32 1, ptr @host, align 4
 // CHECK: store i32 1, ptr addrspace(3) @[[SHARED_NOHOST]], align 4


        

From llvmlistbot at llvm.org  Fri Jul  3 05:39:34 2026
From: llvmlistbot at llvm.org (Robert Imschweiler)
Date: Fri, 03 Jul 2026 05:39:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a47ad86.07a3e4bc.1336d3.7002@mx.google.com>


https://github.com/ro-i closed https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Fri Jul  3 05:44:00 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 05:44:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ae90.eb165e5c.113c2a.3b92@mx.google.com>


================

----------------
durga4github wrote:

if this is used only in one place, let us make it a lambda there locally

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:44:07 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 05:44:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47ae97.491d90f5.1f0d58.dede@mx.google.com>


================

----------------
durga4github wrote:

same here

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:46:45 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:46:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47af35.bf38ae28.358429.47e1@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 5f144a05466b4eb12257e8840e0173225de0e8b3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 1/9] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index c30dd3b07d028..9d810f2613a26 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1092,7 +1092,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 6e1ed05c9d0d0..9aa28ca825f73 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -699,6 +699,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From cde91e515bc3eb100a6524cda7b07417f5a6b4c5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 2/9] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From 0f0b91d8f9627c04fdddb1d2ca9280238de28f27 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 3/9] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9aa28ca825f73..777e5d72ac4b1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -741,7 +741,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -758,22 +757,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -808,7 +791,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -825,22 +807,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From e1c107c642b6c519e7439657e970c9ae9e5a408e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 4/9] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 1da6401974970bf094954cd63c40a91a4b038492 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 5/9] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 777e5d72ac4b1..162a92782380b 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,60 +749,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -818,6 +812,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 9f52cbd5aaa9349b22f1da0493dd306b360569f6 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 6/9] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 162a92782380b..9c6039eeefc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,54 +749,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -812,9 +828,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From ad9430390f24ba3c8c730f1c78ff4afc96534ffb Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 7/9] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9c6039eeefc2c..8754149801124 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -805,9 +805,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 911eacdebc316503b83e25f06615e252e5f812ba Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 8/9] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8754149801124..abc50a807fe5d 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -793,10 +793,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir

>From cd3af0d98a6c2ac1ae0f416ef520e8962db88f26 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:46:13 +0000
Subject: [PATCH 9/9] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index bb0bbc2d0a284..d2f7fe768930b 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2121,10 +2121,9 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerExtf(nvgpu::ExtfOp op,
-                               nvgpu::ExtfOp::Adaptor adaptor,
-                                ConversionPatternRewriter &rewriter,
-                                const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
+                               ConversionPatternRewriter &rewriter,
+                               const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i8Ty = b.getI8Type();
@@ -2271,8 +2270,7 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUExtfOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
@@ -2434,6 +2432,6 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUExtfCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUTruncfCanonicalizePattern, NVGPUExtfCanonicalizePattern>(
+      patterns.getContext());
 }


From llvmlistbot at llvm.org  Fri Jul  3 05:47:57 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 05:47:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47af7d.13be6df8.3d8759.981d@mx.google.com>


================

----------------
durga4github wrote:

in general, the indent here looks too verbose and uses more lines than necessary..

is this output of clang-format as is?

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:50:39 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 05:50:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b01f.fdc477d6.118e5e.db3a@mx.google.com>


================

----------------
durga4github wrote:

this indent does not match to the above one .. something feels amiss..



https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:52:50 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 05:52:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b0a2.ac469dc7.e3aa6.1085@mx.google.com>


================

----------------
durga4github wrote:

in general, the indent on this function and the similar funtion above looks very different from our conventions..

Can you just write everything into a single line and let clang-format do the indenting?

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 05:57:37 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:57:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b1c1.d1496011.3d49cb.e40b@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 5f144a05466b4eb12257e8840e0173225de0e8b3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 01/10] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index c30dd3b07d028..9d810f2613a26 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1092,7 +1092,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 6e1ed05c9d0d0..9aa28ca825f73 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -699,6 +699,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From cde91e515bc3eb100a6524cda7b07417f5a6b4c5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 02/10] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From 0f0b91d8f9627c04fdddb1d2ca9280238de28f27 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 03/10] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9aa28ca825f73..777e5d72ac4b1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -741,7 +741,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -758,22 +757,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -808,7 +791,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -825,22 +807,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From e1c107c642b6c519e7439657e970c9ae9e5a408e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 04/10] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 1da6401974970bf094954cd63c40a91a4b038492 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 05/10] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 777e5d72ac4b1..162a92782380b 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,60 +749,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -818,6 +812,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 9f52cbd5aaa9349b22f1da0493dd306b360569f6 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 06/10] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 162a92782380b..9c6039eeefc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -749,54 +749,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -812,9 +828,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From ad9430390f24ba3c8c730f1c78ff4afc96534ffb Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 07/10] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 9c6039eeefc2c..8754149801124 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,7 +700,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -736,7 +736,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -805,9 +805,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 911eacdebc316503b83e25f06615e252e5f812ba Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 08/10] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8754149801124..abc50a807fe5d 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -793,10 +793,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir

>From cd3af0d98a6c2ac1ae0f416ef520e8962db88f26 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:46:13 +0000
Subject: [PATCH 09/10] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index bb0bbc2d0a284..d2f7fe768930b 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2121,10 +2121,9 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerExtf(nvgpu::ExtfOp op,
-                               nvgpu::ExtfOp::Adaptor adaptor,
-                                ConversionPatternRewriter &rewriter,
-                                const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
+                               ConversionPatternRewriter &rewriter,
+                               const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i8Ty = b.getI8Type();
@@ -2271,8 +2270,7 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUExtfOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
@@ -2434,6 +2432,6 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUExtfCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUTruncfCanonicalizePattern, NVGPUExtfCanonicalizePattern>(
+      patterns.getContext());
 }

>From e219a1c9e263d6383aa3f14ddee65205733d15ba Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:56:56 +0000
Subject: [PATCH 10/10] convert single use function to lambda

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index d2f7fe768930b..67d3b2e86293a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1751,11 +1751,6 @@ static std::optional<FPKind> classifyFPType(Type t) {
   return std::nullopt;
 }
 
-/// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConvert(FPKind src) {
-  return src == FPKind::F32 ? 2 : 1;
-}
-
 /// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
@@ -1985,6 +1980,11 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
+
+  // Number of source-side i32 register slots consumed by each NVVM convert Op.
+  auto getNumSrcI32PerConvert = [](FPKind src) {
+    return src == FPKind::F32 ? 2 : 1;
+  };
   int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.


From llvmlistbot at llvm.org  Fri Jul  3 05:59:09 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 05:59:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b21d.2020f6c2.1fdf61.ce23@mx.google.com>


================

----------------
Wolfram70 wrote:

Converted `getNumSrcI32PerConvert` to a lambda but this one seems to be used in a bunch of places, so I guess it makes sense to leave it as is.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 06:05:02 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 06:05:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b37e.2020f6c2.1fdf61.9ae6@mx.google.com>


================

----------------
Wolfram70 wrote:

Yes, this is the output of clang format. I think it just feels weird in some places due to the braces around conditionals for more clarity, and also perhaps because the function calls are longer than usual.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 06:06:22 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 06:06:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b3ce.ac469dc7.e3aa6.3a4b@mx.google.com>


================

----------------
Wolfram70 wrote:

I think clang format just puts the whole call into a new line when it fits entirely there, which I agree kind of looks weird in some places.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 06:08:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 06:08:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a47b446.faf16b37.1e0c59.88b9@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 198fd9586..f5232283d 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,8 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4) ||
-             aType.isF8E4M3FN() || aType.isF8E5M2()) {
+             aType.isInteger(8) || aType.isInteger(4) || aType.isF8E4M3FN() ||
+             aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK

``````````

</details>


https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Fri Jul  3 06:09:02 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Fri, 03 Jul 2026 06:09:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a47b46e.30759430.1908d3.b674@mx.google.com>


================
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
----------------
grypp wrote:

can you add tests for mlir-translate in nvvmir.mlir

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 06:09:03 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Fri, 03 Jul 2026 06:09:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a47b46f.bb839515.23e6d3.3007@mx.google.com>


https://github.com/Wolfram70 edited https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Fri Jul  3 06:09:20 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 03 Jul 2026 06:09:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a47b480.bb839515.23e6d3.3db4@mx.google.com>


================
@@ -80,26 +89,29 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = getTypeConverter()->convertType(op.getType());
-    if (!resultTy) {
+    auto resultTy = convertMemRefType(op.getType(), getTypeConverter());
+    if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
-    }
+
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
+    // Rank-0 path
+    if (op.getType().getRank() == 0) {
+      auto var = emitc::VariableOp::create(
+          rewriter, op.getLoc(), emitc::LValueType::get(resultTy), noInit);
----------------
aniragil wrote:

Here you could use the type converter and then query whether the type you got is an array or a pointer.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Fri Jul  3 06:09:20 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 03 Jul 2026 06:09:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a47b480.70920dee.c0a01.6860@mx.google.com>


================
@@ -142,9 +154,12 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-// If `v` is defined through an unrealized cast and the source of that cast
-// is `emitc.ptr`, return the pointer.
-static Value stripPointerUnrealizedCast(Value v) {
+static Value getPointerOrStripUnrealizedCast(Value v) {
----------------
aniragil wrote:

Not sure I follow: won't it return a pointer in any case a pointer to the memref can be retrieved (except when it fails, then it returns null)?

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Fri Jul  3 06:09:21 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 03 Jul 2026 06:09:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a47b481.abb10101.208f3e.5afc@mx.google.com>


================
@@ -60,6 +59,16 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
+Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
+  Type resultTy;
+  if (opTy.getRank() == 0) {
+    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
----------------
aniragil wrote:

Sorry, I wasn't clear: I meant the return value is somewhat inconsistent: for rank>0 it returns the actual type the memref is converted to (ArrayType), but for rank==0 it returns the element type, not the type the memref is converted to. The name "convertMemRefType" got me expecting the function to consistently return the type the memref is converted to, which for rank==0 is `emitc::ptr`, as done in the TypeConverter (TBH I'm not sure why we need this wrapper around the type converter ...)

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Fri Jul  3 06:17:00 2026
From: llvmlistbot at llvm.org (Rahul Joshi)
Date: Fri, 03 Jul 2026 06:17:00 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47b64c.510f576d.185953.c7b1@mx.google.com>


https://github.com/jurahul ready_for_review https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 06:17:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 06:17:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47b66e.491d90f5.1f0d58.ae7a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-amdgpu

Author: Rahul Joshi (jurahul)

<details>
<summary>Changes</summary>

Define valid range for `llvm.prefetch` immarg values and eliminate C++ verification code.

Additionally, change the range verification failure message to include the argument number and simplify formatting using `formatv`.

---

Patch is 28.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207300.diff


13 Files Affected:

- (modified) llvm/include/llvm/IR/Intrinsics.td (+4-2) 
- (modified) llvm/lib/IR/Verifier.cpp (+3-11) 
- (modified) llvm/test/Assembler/invalid-immarg5.ll (+1-1) 
- (modified) llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll (+3-5) 
- (modified) llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll (+2-2) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll (+8-11) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll (+11-15) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll (+8-10) 
- (modified) llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll (+15-15) 
- (modified) llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll (+4-4) 
- (modified) llvm/test/Verifier/NVPTX/setmaxnreg.ll (+2-2) 
- (modified) llvm/test/Verifier/intrinsic-immarg.ll (+12) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+1-1) 


``````````diff
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),
diff --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
index 1c35fbead389e..0cef9c1d9405c 100644
--- a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
@@ -1,14 +1,12 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) writeonly, ptr addrspace(3), ptr readonly, i32, i16, i64, i1 immarg, i1 immarg, i32 immarg range(i32 0, 3))
 
 define void @test_cp_async_bulk_tensor_g2s_tile_1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch) {
-  ; CHECK: immarg value 3 out of range [0, 3)
+  ; CHECK: immarg value 3 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 3)
 
-  ; CHECK: immarg value -1 out of range [0, 3)
+  ; CHECK: immarg value -1 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 -1)
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
index ab35e4fb396d6..52f494ae590b6 100644
--- a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
@@ -1,7 +1,7 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 130 out of range [128, 129)
+  ; CHECK: immarg value 130 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 130);
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
index c0f6f4c7c46bd..bd3ed2e447c0b 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
@@ -1,23 +1,20 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: not llc < %s -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_101a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_110a -march=nvptx64 -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_block_scale_invalid_flags(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b) {
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
index dff829ecf5321..c80755b6711d8 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
@@ -1,37 +1,33 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_101a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_110a -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3)
-  ; CHECK: immarg value 3 out of range [1, 3)
+  ; CHECK: immarg value 3 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 5)
-  ; CHECK: immarg value 2 out of range [0, 2)
+  ; CHECK: immarg value 2 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.ashift(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 2)
   ret void
 }
 
 define void @tcgen05_mma_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_ws_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
index 7c884a70b7530..5987668b10add 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
@@ -1,25 +1,23 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_scale_d_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 6 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_scale_d_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt....
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 06:17:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 06:17:35 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47b66f.eea6ab6e.1256b6.872f@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Rahul Joshi (jurahul)

<details>
<summary>Changes</summary>

Define valid range for `llvm.prefetch` immarg values and eliminate C++ verification code.

Additionally, change the range verification failure message to include the argument number and simplify formatting using `formatv`.

---

Patch is 28.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207300.diff


13 Files Affected:

- (modified) llvm/include/llvm/IR/Intrinsics.td (+4-2) 
- (modified) llvm/lib/IR/Verifier.cpp (+3-11) 
- (modified) llvm/test/Assembler/invalid-immarg5.ll (+1-1) 
- (modified) llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll (+3-5) 
- (modified) llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll (+2-2) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll (+8-11) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll (+11-15) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll (+8-10) 
- (modified) llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll (+15-15) 
- (modified) llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll (+4-4) 
- (modified) llvm/test/Verifier/NVPTX/setmaxnreg.ll (+2-2) 
- (modified) llvm/test/Verifier/intrinsic-immarg.ll (+12) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+1-1) 


``````````diff
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),
diff --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
index 1c35fbead389e..0cef9c1d9405c 100644
--- a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
@@ -1,14 +1,12 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) writeonly, ptr addrspace(3), ptr readonly, i32, i16, i64, i1 immarg, i1 immarg, i32 immarg range(i32 0, 3))
 
 define void @test_cp_async_bulk_tensor_g2s_tile_1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch) {
-  ; CHECK: immarg value 3 out of range [0, 3)
+  ; CHECK: immarg value 3 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 3)
 
-  ; CHECK: immarg value -1 out of range [0, 3)
+  ; CHECK: immarg value -1 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 -1)
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
index ab35e4fb396d6..52f494ae590b6 100644
--- a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
@@ -1,7 +1,7 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 130 out of range [128, 129)
+  ; CHECK: immarg value 130 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 130);
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
index c0f6f4c7c46bd..bd3ed2e447c0b 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
@@ -1,23 +1,20 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: not llc < %s -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_101a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_110a -march=nvptx64 -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_block_scale_invalid_flags(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b) {
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
index dff829ecf5321..c80755b6711d8 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
@@ -1,37 +1,33 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_101a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_110a -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3)
-  ; CHECK: immarg value 3 out of range [1, 3)
+  ; CHECK: immarg value 3 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 5)
-  ; CHECK: immarg value 2 out of range [0, 2)
+  ; CHECK: immarg value 2 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.ashift(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 2)
   ret void
 }
 
 define void @tcgen05_mma_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_ws_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
index 7c884a70b7530..5987668b10add 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
@@ -1,25 +1,23 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_scale_d_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 6 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_scale_d_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt....
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 06:17:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 06:17:35 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47b66f.af6ba3ed.2c2f30.dc45@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-nvptx

Author: Rahul Joshi (jurahul)

<details>
<summary>Changes</summary>

Define valid range for `llvm.prefetch` immarg values and eliminate C++ verification code.

Additionally, change the range verification failure message to include the argument number and simplify formatting using `formatv`.

---

Patch is 28.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207300.diff


13 Files Affected:

- (modified) llvm/include/llvm/IR/Intrinsics.td (+4-2) 
- (modified) llvm/lib/IR/Verifier.cpp (+3-11) 
- (modified) llvm/test/Assembler/invalid-immarg5.ll (+1-1) 
- (modified) llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll (+3-5) 
- (modified) llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll (+2-2) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll (+8-11) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll (+11-15) 
- (modified) llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll (+8-10) 
- (modified) llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll (+15-15) 
- (modified) llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll (+4-4) 
- (modified) llvm/test/Verifier/NVPTX/setmaxnreg.ll (+2-2) 
- (modified) llvm/test/Verifier/intrinsic-immarg.ll (+12) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+1-1) 


``````````diff
diff --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),
diff --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
index 1c35fbead389e..0cef9c1d9405c 100644
--- a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
@@ -1,14 +1,12 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) writeonly, ptr addrspace(3), ptr readonly, i32, i16, i64, i1 immarg, i1 immarg, i32 immarg range(i32 0, 3))
 
 define void @test_cp_async_bulk_tensor_g2s_tile_1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch) {
-  ; CHECK: immarg value 3 out of range [0, 3)
+  ; CHECK: immarg value 3 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 3)
 
-  ; CHECK: immarg value -1 out of range [0, 3)
+  ; CHECK: immarg value -1 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 -1)
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
index ab35e4fb396d6..52f494ae590b6 100644
--- a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
@@ -1,7 +1,7 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 130 out of range [128, 129)
+  ; CHECK: immarg value 130 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 130);
 
   ret void
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
index c0f6f4c7c46bd..bd3ed2e447c0b 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
@@ -1,23 +1,20 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: not llc < %s -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_101a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_110a -march=nvptx64 -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_block_scale_invalid_flags(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b) {
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
index dff829ecf5321..c80755b6711d8 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
@@ -1,37 +1,33 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_101a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_110a -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3)
-  ; CHECK: immarg value 3 out of range [1, 3)
+  ; CHECK: immarg value 3 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 5)
-  ; CHECK: immarg value 2 out of range [0, 2)
+  ; CHECK: immarg value 2 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.ashift(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 2)
   ret void
 }
 
 define void @tcgen05_mma_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_ws_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
index 7c884a70b7530..5987668b10add 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
@@ -1,25 +1,23 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_scale_d_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 6 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_scale_d_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
diff --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt....
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 06:18:59 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 06:18:59 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47b6c3.6c8668f2.346e76.fda9@mx.google.com>


https://github.com/nikic approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 06:39:10 2026
From: llvmlistbot at llvm.org (Andrey Pavlenko)
Date: Fri, 03 Jul 2026 06:39:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a47bb7e.dd2b4bd3.13e95e.cf66@mx.google.com>


================
@@ -0,0 +1,46 @@
+// RUN: mlir-opt %s -convert-vector-to-xegpu -split-input-file | FileCheck %s
+
+// -----
+gpu.module @xevm_module {
----------------
AndreyPavlenko wrote:

Done.

https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Fri Jul  3 07:12:10 2026
From: llvmlistbot at llvm.org (Mikael Holmen)
Date: Fri, 03 Jul 2026 07:12:10 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47c33a.4a342cbc.134cb.a354@mx.google.com>


mikaelholmen wrote:

When doing a clean build I see several errors like
```
<inline asm>:11:5: error: 32 bit reloc applied to a field with a different size
   11 | jmp __interceptor_strlen at plt
      |     ^
```
when compiling compiler-rt. After it fails, if I just rerun the compilation it succesfully builds.
I speculatively reverted this patch locally, and then it doesn't seem to happen anymore.
I have no idea what is going on, does anyone have a clue?


https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 07:17:31 2026
From: llvmlistbot at llvm.org (Hans Wennborg)
Date: Fri, 03 Jul 2026 07:17:31 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47c47b.d1496011.3d49cb.0e0f@mx.google.com>


zmodem wrote:

> When doing a clean build I see several errors like

I was just about to report the same. Local repro on Linux:

```
$ cmake -GNinja -Bbuild.123 -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS=clang -DLLVM_ENABLE_RUNTIMES=compiler-rt -DLLVM_TARGETS_TO_BUILD=X86 llvm/
$ ninja -C build.123 runtimes
...
(a bunch of)
error: 32 bit reloc applied to a field with a different size
```

I assume the buildbots would hit this also if they weren't down for maintenance.

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 07:19:52 2026
From: llvmlistbot at llvm.org (Hans Wennborg)
Date: Fri, 03 Jul 2026 07:19:52 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
Message-ID: <llvm.org/llvm/llvm-project/pull/207399@github.com>


https://github.com/zmodem created https://github.com/llvm/llvm-project/pull/207399

Reverts llvm/llvm-project#204548

This is causing the runtimes build to fail with e.g.:

```
<inline asm>:11:5: error: 32 bit reloc applied to a field with a different size
   11 | jmp __interceptor_strlen at plt
      |     ^
```

See comments on the PR.

>From df4641e2fcff056cc3e1270bdd3ce7ce625965d8 Mon Sep 17 00:00:00 2001
From: Hans Wennborg <hans at hanshq.net>
Date: Fri, 3 Jul 2026 16:18:00 +0200
Subject: [PATCH] Revert "[IR] Explicitly specify target feature for module asm
 (#204548)"

This reverts commit 0341dd51303fc13cafdd21a33e2985cbf603c66b.
---
 clang/lib/CodeGen/CGObjCMac.cpp               |  6 +-
 clang/lib/CodeGen/CodeGenModule.cpp           |  8 +-
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c  | 17 ++--
 clang/test/CodeGen/asm.c                      |  7 +-
 clang/test/CodeGen/asm_incbin.c               |  3 +-
 clang/test/CodeGen/cfi-salt.c                 | 11 +--
 clang/test/CodeGen/kcfi.c                     |  7 +-
 clang/test/CodeGenCUDA/filter-decl.cu         |  5 +-
 clang/test/CodeGenCXX/gnu-asm-constexpr.cpp   |  3 +-
 clang/test/CodeGenObjC/category-class.m       |  7 +-
 clang/test/CodeGenObjC/objc-runtime-name.m    |  7 +-
 clang/test/Frontend/ast-codegen.c             |  3 +-
 clang/test/Modules/codegen.test               |  3 +-
 llvm/docs/LangRef.rst                         | 27 ++----
 llvm/docs/ReleaseNotes.md                     |  4 -
 llvm/include/llvm/Bitcode/LLVMBitCodes.h      |  2 -
 llvm/include/llvm/CodeGen/AsmPrinter.h        | 13 ---
 llvm/include/llvm/IR/Module.h                 | 96 +++----------------
 llvm/lib/AsmParser/LLParser.cpp               | 39 +-------
 llvm/lib/Bitcode/Reader/BitcodeReader.cpp     | 17 +---
 llvm/lib/Bitcode/Writer/BitcodeWriter.cpp     | 16 +---
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    | 28 ++----
 llvm/lib/IR/AsmWriter.cpp                     | 43 +++------
 llvm/lib/IR/Core.cpp                          | 13 +--
 llvm/lib/IR/Module.cpp                        | 20 ----
 llvm/lib/LTO/LTO.cpp                          |  4 +-
 llvm/lib/Linker/IRMover.cpp                   |  9 +-
 llvm/lib/Object/ModuleSymbolTable.cpp         | 90 +++++++++--------
 llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp     | 23 ++---
 llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp     |  2 +-
 llvm/lib/Transforms/IPO/ExtractGV.cpp         |  2 +-
 .../Transforms/IPO/ThinLTOBitcodeWriter.cpp   |  2 +-
 .../Instrumentation/DataFlowSanitizer.cpp     | 26 +++--
 llvm/lib/Transforms/Utils/SplitModule.cpp     |  2 +-
 llvm/test/Assembler/module-asm-invalid.ll     | 25 -----
 llvm/test/Bitcode/compatibility-3.6.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.7.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.8.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.9.ll        |  3 +-
 llvm/test/Bitcode/compatibility-4.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-5.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-6.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility.ll            |  3 +-
 llvm/test/Bitcode/highLevelStructure.3.2.ll   |  3 +-
 llvm/test/Bitcode/module-asm.ll               | 29 ------
 .../test/CodeGen/RISCV/module-asm-features.ll | 17 ----
 .../DataFlowSanitizer/prefix-rename.ll        |  3 +-
 llvm/test/LTO/RISCV/module-asm.ll             | 18 ----
 llvm/test/LTO/X86/inline-asm-lto-discard.ll   | 24 +++--
 llvm/test/Linker/Inputs/module-asm.ll         |  2 -
 .../link-arm-and-thumb-module-inline-asm.ll   | 17 ++--
 llvm/test/Linker/module-asm.ll                |  9 --
 llvm/test/ThinLTO/X86/import-symver.ll        |  3 +-
 .../LowerTypeTests/export-symver.ll           |  3 +-
 .../cfi-icall-static-inline-asm.ll            |  3 +-
 .../ThinLTOBitcodeWriter/x86/module-asm.ll    |  3 +-
 .../llvm-reduce/deltas/ReduceModuleData.cpp   |  2 +-
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 11 +--
 mlir/test/Target/LLVMIR/module-asm.mlir       |  5 +-
 59 files changed, 193 insertions(+), 573 deletions(-)
 delete mode 100644 llvm/test/Assembler/module-asm-invalid.ll
 delete mode 100644 llvm/test/Bitcode/module-asm.ll
 delete mode 100644 llvm/test/CodeGen/RISCV/module-asm-features.ll
 delete mode 100644 llvm/test/LTO/RISCV/module-asm.ll
 delete mode 100644 llvm/test/Linker/Inputs/module-asm.ll
 delete mode 100644 llvm/test/Linker/module-asm.ll

diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index e518b875f44bb..69c5e88f3c768 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,6 +6610,10 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
+    Asm += CGM.getModule().getModuleInlineAsm();
+    if (!Asm.empty() && Asm.back() != '\n')
+      Asm += '\n';
+
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6620,7 +6624,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().appendModuleInlineAsm(OS.str());
+    CGM.getModule().setModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index e155fdd752d7f..813cd67e4dd51 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,13 +8057,7 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-
-    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
-    llvm::Module::GlobalAsmProperties Props;
-    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
-    Props.TargetCPU = TargetOpts.CPU;
-    getModule().appendModuleInlineAsm(
-        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
+    getModule().appendModuleInlineAsm(AD->getAsmString());
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 8a212a0b61905..877da4fc669e3 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,18 +1,13 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
-// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// RISCV64: module asm(target_features: "+64bit,+i")
-// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
-// DEFAULT: module asm
-// CHECK-NEXT: "foo1"
+// CHECK: module asm "foo1"
 __asm__ ("foo1");
-// CHECK-NEXT: "foo2"
+// CHECK: module asm "foo2"
 __asm__ ("foo2");
-// CHECK-NEXT: "foo3"
+// CHECK: module asm "foo3"
 __asm__ ("foo3");
-// CHECK-NEXT: "foo4"
+// CHECK: module asm "foo4"
 __asm__ ("foo4");
-// CHECK-NEXT: "foo5"
+// CHECK: module asm "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index ea69c7a17d1cb..d7465b22fbbf6 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,10 +2,9 @@
 
 // PR10415:
 //
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo1"
-// CHECK-NEXT: "foo2"
-// CHECK-NEXT: "foo3"
+// CHECK:      module asm "foo1"
+// CHECK-NEXT: module asm "foo2"
+// CHECK-NEXT: module asm "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index bbc5d3af55ea2..6234eea1d6738 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,5 +6,4 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm
-// CHECK-NEXT: ".incbin \22foo.h\22"
+// CHECK: module asm ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 18156dc014280..2a90dc31d939b 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,14 +26,13 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_f6"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index f08cf7635fb5a..da32859dd3d88 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,11 +6,10 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index f87d62a56631e..02dacd0ad8ef4 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,9 +5,8 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm
-// CHECK-HOST-NEXT: "file scope asm is host only"
-// CHECK-DEVICE-NOT: "file scope asm is host only"
+// CHECK-HOST: module asm "file scope asm is host only"
+// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 2b4abee5dc05f..175a3b7bc588c 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,8 +19,7 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm(target_features: "{{.*}}")
-    // CHECK-NEXT: "nop; nop; nop; nop"
+    // CHECK: module asm "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index febe94ce5b9f5..92fd36cbe4066 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,10 +1,9 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm
-// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
-// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
-// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
+// CHECK: module asm "\09.objc_category_name_A_foo=0"
+// CHECK: module asm "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index a7f59caced390..c838b8c2c6f1a 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,10 +3,9 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm
-// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
-// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
-// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm {{.*}}objc_class_name_XYZ=0
+// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
+// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index c26b0a1908379..2e2e3d360dd17 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,8 +2,7 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo"
+// CHECK: module asm "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 5cc5b3c1cda2a..0af630a754805 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,8 +18,7 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm
-BOTH-NEXT: "narf"
+BOTH: module asm "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2c8276d04b668..5ce6019660e0b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,34 +3357,19 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks:
+to the GCC "file scope inline asm" blocks. These blocks are internally
+concatenated by LLVM and treated as a single unit, but may be separated
+in the ``.ll`` file if desired. The syntax is very simple:
 
 .. code-block:: llvm
 
-    module asm
-        "inline asm code goes here"
-        "more can go here"
+    module asm "inline asm code goes here"
+    module asm "more can go here"
 
-The sequence of string literals is internally concatenated using newlines as
-separators. The strings can contain any character by escaping non-printable
+The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
-Additionally, module-level inline assembly can specify an optional list of
-properties:
-
-.. code-block:: llvm
-
-    module asm(target_features: "+foo", target_cpu: "bar")
-        "inline asm code goes here"
-        "more can go here"
-
-Currently, the only supported properties are ``target_features`` and
-``target_cpu``, with the same meaning as the ``"target-features"`` and
-``"target-cpu"`` function attributes.
-
-Consecutive blocks with identical properties will be concatenated into one.
-
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cbec4d5b912d0..e42a959ab3101 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -111,10 +111,6 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
-* Module-level inline assembly now accepts optional `target_features` and
-  `target_cpu` properties. This resolves errors during LTO on some
-  architectures.
-
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 1761e7a542a73..af4cac996a78c 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,8 +120,6 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
-
-  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 335bf9de9a071..78e64440657cc 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,19 +930,6 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
-  /// Emit necessary directives to allow use of instructions that are permitted
-  /// by target features enabled by STI, but are not permitted by target
-  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
-  /// Returns whether anything was emitted.
-  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
-    return false;
-  }
-
-  /// Emit necessary directives to restore target feature state.
-  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
-  /// call.
-  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
-
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 63d1013896b47..4f7e33969f16f 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,45 +169,6 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
-  struct GlobalAsmProperties {
-    std::string TargetFeatures;
-    std::string TargetCPU;
-
-    /// Set a property using a string name.
-    /// Returns whether the property name was valid.
-    LLVM_ABI bool set(StringRef Name, std::string Value);
-
-    /// Get a list of set properties as pairs of key and value.
-    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
-
-    bool operator==(const GlobalAsmProperties &Other) const {
-      return TargetFeatures == Other.TargetFeatures &&
-             TargetCPU == Other.TargetCPU;
-    }
-
-    bool operator!=(const GlobalAsmProperties &Other) const {
-      return !(*this == Other);
-    }
-  };
-
-  struct GlobalAsmFragment {
-    std::string Asm;
-    GlobalAsmProperties Props;
-
-    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
-    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
-        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
-      if (!Asm.empty() && Asm.back() != '\n')
-        Asm += '\n';
-    }
-
-    bool empty() const { return Asm.empty(); }
-
-    bool hasSameProperties(const GlobalAsmFragment &Other) const {
-      return Props == Other.Props;
-    }
-  };
-
 /// @}
 /// @name Member Variables
 /// @{
@@ -219,8 +180,7 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  /// Inline Asm at the global scope.
-  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
+  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -327,17 +287,8 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
-    return GlobalScopeAsm;
-  }
-
-  /// Get any module-scope inline assembly blocks.
-  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
-    return GlobalScopeAsm;
-  }
-
-  /// Return whether there is any module-scope inline assembly.
-  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
+  /// @returns a string containing the module-scope inline assembly blocks.
+  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -374,45 +325,20 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
-  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
-
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
-    GlobalScopeAsm.clear();
-    appendModuleInlineAsm(std::move(Fragment));
-  }
-
-  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
-    GlobalScopeAsm.clear();
-    append_range(GlobalScopeAsm, Fragments);
+  void setModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm = std::string(Asm);
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.back().Asm += Fragment.Asm;
-    } else {
-      GlobalScopeAsm.emplace_back(std::move(Fragment));
-    }
-  }
-
-  /// Prepend to the module-scope inline assembly blocks.
-  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
-    } else {
-      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
-    }
+  void appendModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm += Asm;
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 0a5b4c6c91879..75e6add0ec76f 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,49 +641,16 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
-///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
-///                          'property_name2:' STRINGCONSTANT ')'
-///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
+      parseStringConstant(AsmStr))
     return true;
 
-  Module::GlobalAsmProperties Props;
-  if (EatIfPresent(lltok::lparen)) {
-    while (true) {
-      std::string Key, Value;
-      SMLoc Loc = Lex.getLoc();
-      if (Lex.getKind() != lltok::LabelStr)
-        return error(Loc, "expected property name followed by ':'");
-
-      Key = Lex.getStrVal();
-      Lex.Lex();
-
-      if (parseStringConstant(Value))
-        return true;
-
-      if (!Props.set(Key, Value))
-        return error(Loc, "unknown property name");
-
-      if (EatIfPresent(lltok::rparen))
-        break;
-      if (parseToken(lltok::comma, "expected ',' or ')'"))
-        return true;
-    }
-  }
-
-  do {
-    std::string AsmStrPart;
-    if (parseStringConstant(AsmStrPart))
-      return true;
-    AsmStr += AsmStrPart + "\n";
-  } while (Lex.getKind() == lltok::StringConstant);
-
-  M->appendModuleInlineAsm({AsmStr, Props});
+  M->appendModuleInlineAsm(AsmStr);
   return false;
 }
 
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 06e9aea4fd3ac..2bd251efd05ce 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,9 +4580,6 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
-  // Apply to the following module asm.
-  Module::GlobalAsmProperties Props;
-
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4791,23 +4788,11 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
-    case bitc::MODULE_CODE_ASM_PROPERTY: {
-      std::string Str;
-      if (convertToString(Record, 0, Str))
-        return error("Invalid module asm record");
-      size_t SepPos = Str.find('\0');
-      if (SepPos == std::string::npos)
-        return error("Invalid module asm record");
-      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
-        return error("Unknown module asm property");
-      break;
-    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
-      Props = {};
+      TheModule->setModuleInlineAsm(S);
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 493cbec7506f2..4aa203b0d12d1 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,19 +1551,9 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-
-  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    SmallVector<std::pair<StringRef, StringRef>> Props =
-        Frag.Props.getAsStrings();
-    for (auto [Key, Value] : Props) {
-      SmallVector<unsigned, 64> Record;
-      Record.append(Key.begin(), Key.end());
-      Record.push_back(0);
-      Record.append(Value.begin(), Value.end());
-      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
-    }
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
-  }
+  if (!M.getModuleInlineAsm().empty())
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
+                      0 /*TODO*/);
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 8248a05f1a35f..abb20a7919fa1 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,7 +110,6 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
-#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -128,6 +127,7 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
+#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,29 +610,13 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (M.hasModuleInlineAsm()) {
+  if (!M.getModuleInlineAsm().empty()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
-        std::unique_ptr<MCSubtargetInfo> AsmSTI(
-            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
-                                                 Frag.Props.TargetCPU,
-                                                 Frag.Props.TargetFeatures));
-        bool DidPush = emitTargetFeaturePush(*AsmSTI);
-        emitInlineAsm(
-            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-        emitTargetFeaturePop(*AsmSTI, DidPush);
-      } else {
-        // If the module asm does not explicitly specify target features,
-        // fall back to default subtargetinfo.
-        emitInlineAsm(
-            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
-            nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-      }
-    }
+    emitInlineAsm(
+        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
+        TM.Options.MCOptions, nullptr,
+        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 02d6bcd90add0..665d7122af2c7 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,38 +3126,21 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (M->hasModuleInlineAsm()) {
+  if (!M->getModuleInlineAsm().empty()) {
     Out << '\n';
 
-    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
-      Out << "module asm";
-      SmallVector<std::pair<StringRef, StringRef>> Props =
-          Frag.Props.getAsStrings();
-      if (!Props.empty()) {
-        ListSeparator LS;
-        Out << "(";
-        for (auto [Key, Value] : Props) {
-          Out << LS;
-          Out << Key << ": \"";
-          printEscapedString(Value, Out);
-          Out << "\"";
-        }
-        Out << ")";
-      }
-      Out << "\n";
-      // Split the string into lines, to make it easier to read the .ll file.
-      StringRef Asm = Frag.Asm;
-      do {
-        StringRef Front;
-        std::tie(Front, Asm) = Asm.split('\n');
-
-        // We found a newline, print the portion of the asm string from the
-        // last newline up to this newline.
-        Out << "    \"";
-        printEscapedString(Front, Out);
-        Out << "\"\n";
-      } while (!Asm.empty());
-    }
+    // Split the string into lines, to make it easier to read the .ll file.
+    StringRef Asm = M->getModuleInlineAsm();
+    do {
+      StringRef Front;
+      std::tie(Front, Asm) = Asm.split('\n');
+
+      // We found a newline, print the portion of the asm string from the
+      // last newline up to this newline.
+      Out << "module asm \"";
+      printEscapedString(Front, Out);
+      Out << "\"\n";
+    } while (!Asm.empty());
   }
 
   printTypeIdentities();
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index a7abd3eed31c3..6c6c3b5c84ede 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,18 +510,7 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  Module *Mod = unwrap(M);
-  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
-  if (Frags.empty()) {
-    *Len = 0;
-    return nullptr;
-  }
-
-  if (Frags.size() != 1)
-    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
-                          "more than one module inline assembly fragment");
-
-  auto &Str = Frags.begin()->Asm;
+  auto &Str = unwrap(M)->getModuleInlineAsm();
   *Len = Str.length();
   return Str.c_str();
 }
diff --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 34e44e26d5db5..40ef7e354da72 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,23 +977,3 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
-
-bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
-  if (Name == "target_features")
-    TargetFeatures = std::move(Value);
-  else if (Name == "target_cpu")
-    TargetCPU = std::move(Value);
-  else
-    return false;
-  return true;
-}
-
-SmallVector<std::pair<StringRef, StringRef>>
-Module::GlobalAsmProperties::getAsStrings() const {
-  SmallVector<std::pair<StringRef, StringRef>> Props;
-  if (!TargetFeatures.empty())
-    Props.emplace_back("target_features", TargetFeatures);
-  if (!TargetCPU.empty())
-    Props.emplace_back("target_cpu", TargetCPU);
-  return Props;
-}
diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index b672f4e5fac14..6a754f95092bb 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (M.hasModuleInlineAsm()) {
+  if (!M.getModuleInlineAsm().empty()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.prependModuleInlineAsm(NewIA);
+    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
   }
 
   assert(MsymI == MsymE);
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 3b72b412d0b2e..239201bde1cec 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,11 +1562,8 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
-      Module::GlobalAsmFragment NewFrag(Frag);
-      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
-      DstM.appendModuleInlineAsm(std::move(NewFrag));
-    }
+    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
+                                               SrcTriple));
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1576,7 +1573,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(std::string(S));
+        DstM.appendModuleInlineAsm(S);
       }
     });
   }
diff --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 44f717ec12323..1da5fa9c10a0b 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,7 +72,8 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  if (!M.hasModuleInlineAsm())
+  StringRef InlineAsm = M.getModuleInlineAsm();
+  if (InlineAsm.empty())
     return;
 
   std::string Err;
@@ -89,55 +90,52 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
+  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
+  if (!STI)
+    return;
+
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
-        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
-    if (!STI)
-      return;
-
-    std::unique_ptr<MemoryBuffer> Buffer(
-        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
-    SourceMgr SrcMgr;
-    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-    std::unique_ptr<MCObjectFileInfo> MOFI(
-        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-    MCCtx.setObjectFileInfo(MOFI.get());
-    RecordStreamer Streamer(MCCtx, M);
-    T->createNullTargetStreamer(Streamer);
-
-    std::unique_ptr<MCAsmParser> Parser(
-        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-    std::unique_ptr<MCTargetAsmParser> TAP(
-        T->createMCAsmParser(*STI, *Parser, *MCII));
-    if (!TAP)
-      return;
-
-    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                   const SourceMgr &SrcMgr,
-                                   std::vector<const MDNode *> &LocInfos) {
-      M.getContext().diagnose(
-          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-    });
-
-    // Module-level inline asm is assumed to use At&t syntax (see
-    // AsmPrinter::doInitialization()).
-    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-    Parser->setSymbolScanningMode(true);
-
-    Parser->setTargetParser(*TAP);
-    if (Parser->Run(false))
-      return;
-
-    Init(Streamer);
-  }
+  std::unique_ptr<MemoryBuffer> Buffer(
+      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
+  SourceMgr SrcMgr;
+  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+  std::unique_ptr<MCObjectFileInfo> MOFI(
+      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+  MCCtx.setObjectFileInfo(MOFI.get());
+  RecordStreamer Streamer(MCCtx, M);
+  T->createNullTargetStreamer(Streamer);
+
+  std::unique_ptr<MCAsmParser> Parser(
+      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+  std::unique_ptr<MCTargetAsmParser> TAP(
+      T->createMCAsmParser(*STI, *Parser, *MCII));
+  if (!TAP)
+    return;
+
+  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                 const SourceMgr &SrcMgr,
+                                 std::vector<const MDNode *> &LocInfos) {
+    M.getContext().diagnose(
+        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+  });
+
+  // Module-level inline asm is assumed to use At&t syntax (see
+  // AsmPrinter::doInitialization()).
+  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+  Parser->setSymbolScanningMode(true);
+
+  Parser->setTargetParser(*TAP);
+  if (Parser->Run(false))
+    return;
+
+  Init(Streamer);
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index e37ea27364e6e..dbda25c34cee0 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,8 +113,7 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
-  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
+  bool emitDirectiveOptionArch();
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -524,19 +523,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                               : RISCVOptionArchArgType::Minus;
+    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                                : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -550,16 +549,11 @@ bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   return false;
 }
 
-void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
-                                           bool DidPush) {
-  if (DidPush)
-    getTargetStreamer().emitDirectiveOptionPop();
-}
-
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
+  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
+  bool EmittedOptionArch = emitDirectiveOptionArch();
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -567,7 +561,8 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  emitTargetFeaturePop(*STI, EmittedOptionArch);
+  if (EmittedOptionArch)
+    RTS.emitDirectiveOptionPop();
   return false;
 }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 22d0d97090746..2a5d16a470c4a 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.removeModuleInlineAsm();
+    M.setModuleInlineAsm("");
   }
 
   // Register the NSDI handler before calling the base class so that
diff --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 4b76b02c32fe4..16fdb93e2b21b 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.removeModuleInlineAsm();
+    M.setModuleInlineAsm("");
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually
diff --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index e857c94330bad..a486979a58c2d 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->removeModuleInlineAsm();
+  MergedM->setModuleInlineAsm("");
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.
diff --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index 375ac069272eb..c349fe33dd237 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,20 +1283,18 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  for (Module::GlobalAsmFragment &Frag :
-       GV->getParent()->getModuleInlineAsm()) {
-    std::string SearchStr = ".symver " + GVName + ",";
-    size_t Pos = Frag.Asm.find(SearchStr);
-    if (Pos != std::string::npos) {
-      Frag.Asm.replace(Pos, SearchStr.size(),
-                       ".symver " + GVName + Suffix + ",");
-      Pos = Frag.Asm.find('@');
-
-      if (Pos == std::string::npos)
-        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
-
-      Frag.Asm.replace(Pos, 1, Suffix + "@");
-    }
+  std::string Asm = GV->getParent()->getModuleInlineAsm();
+  std::string SearchStr = ".symver " + GVName + ",";
+  size_t Pos = Asm.find(SearchStr);
+  if (Pos != std::string::npos) {
+    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
+    Pos = Asm.find('@');
+
+    if (Pos == std::string::npos)
+      report_fatal_error(Twine("unsupported .symver: ", Asm));
+
+    Asm.replace(Pos, 1, Suffix + "@");
+    GV->getParent()->setModuleInlineAsm(Asm);
   }
 }
 
diff --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 8e70902c4dac2..64910b4d1ce99 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->removeModuleInlineAsm();
+      MPart->setModuleInlineAsm("");
     ModuleCallback(std::move(MPart));
   }
 }
diff --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
deleted file mode 100644
index 34e5b71e683a9..0000000000000
--- a/llvm/test/Assembler/module-asm-invalid.ll
+++ /dev/null
@@ -1,25 +0,0 @@
-; RUN: split-file %s %t
-; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
-; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
-; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
-; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
-
-;--- invalid_prop.ll
-; CHECK: unknown property name
-module asm(foo: "bar")
-    "asm"
-
-;--- missing_value1.ll
-; CHECK: expected property name followed by ':'
-module asm(target_features)
-    "asm"
-
-;--- missing_value2.ll
-; CHECK: expected string constant
-module asm(target_features:)
-    "asm"
-
-;--- missing_paren.ll
-; CHECK: expected ',' or ')'
-module asm(target_features: "bar"
-    "asm"
diff --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index c79086bad1ba5..288e0fc50a9ba 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,8 +14,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index 00b2bb9b30b62..ef9490310fe5d 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,8 +14,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 5b86d5ffa1fee..538d6101b3994 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index eca4bbdf03766..3bfef73e485dd 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index 8551b98815b1a..aaf9fe5492171 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index 883ac3eaaa293..b820417107e8b 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 624d5bd7703c6..67e96c7277cc3 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index 97889d4687516..d55aa1dc496df 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,8 +16,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index cd9a34c810d57..f2770ea4e4d34 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,8 +9,7 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm
-; CHECK-NEXT: "some assembly"
+; CHECK: module asm "some assembly"
 module asm "some assembly"
 
 ; Named Types Test
diff --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
deleted file mode 100644
index 406f25c5dd28c..0000000000000
--- a/llvm/test/Bitcode/module-asm.ll
+++ /dev/null
@@ -1,29 +0,0 @@
-; RUN: llvm-as < %s | llvm-dis | FileCheck %s
-
-; CHECK: module asm
-; CHECK:     "asm line 0"
-; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
-; CHECK:     "asm line 1"
-; CHECK:     "asm line 2"
-; CHECK: module asm(target_features: "+bar")
-; CHECK:     "asm line 3"
-; CHECK: module asm(target_cpu: "bar_cpu")
-; CHECK:     "asm line 4"
-; CHECK: module asm
-; CHECK:     "asm line 5"
-
-module asm
-    "asm line 0"
-
-module asm(target_features: "+foo", target_cpu: "foo_cpu")
-    "asm line 1"
-    "asm line 2"
-
-module asm(target_features: "+bar")
-    "asm line 3"
-
-module asm(target_cpu: "bar_cpu")
-    "asm line 4"
-
-module asm
-    "asm line 5"
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
deleted file mode 100644
index ab16acab48688..0000000000000
--- a/llvm/test/CodeGen/RISCV/module-asm-features.ll
+++ /dev/null
@@ -1,17 +0,0 @@
-; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
-; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
-
-; This should work fine, because the module asm specifies the necessary
-; target features
-
-; SAME-FEATURES-NOT: .option arch
-; EXTRA-FEATURES: .option push
-; EXTRA-FEATURES: .option arch, +d
-; CHECK: fld ft0, 0(sp)
-; EXTRA-FEATURES: .option pop
-
-module asm(target_features: "+d")
-    ".globl func"
-    "func:"
-    "fld f0, 0(sp)"
-    "ret"
diff --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 829dd829d51f9..454c496d1d4bb 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,8 +2,7 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm
-; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan
diff --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
deleted file mode 100644
index 73320185e778a..0000000000000
--- a/llvm/test/LTO/RISCV/module-asm.ll
+++ /dev/null
@@ -1,18 +0,0 @@
-; RUN: llvm-as %s -o %t.o
-; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
-; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
-; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
-; RUN: FileCheck %s --input-file %t.s.0
-
-; NM: T func
-
-; CHECK: fld ft0, 0(sp)
-
-target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
-target triple = "riscv64-unknown-linux-gnu"
-
-module asm(target_features: "+d")
-    ".globl func"
-    "func:"
-    "fld f0, 0(sp)"
-    "ret"
diff --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 6c5173c3f9553..88492e4366489 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,19 +32,17 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1: module asm
-; ASM1-NEXT: ".lto_discard foo"
-; ASM1-NEXT: ".weak foo"
-; ASM1-NEXT: ".equ foo,bar"
-
-; ASM2: module asm
-; ASM2-NEXT: ".lto_discard foo"
-; ASM2-NEXT: ".weak foo"
-; ASM2-NEXT: ".equ foo,bar"
-; ASM2-NEXT: ".lto_discard"
-; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  ".lto_discard foo"
+; ASM1:      module asm ".lto_discard foo"
+; ASM1-NEXT: module asm ".weak foo"
+; ASM1-NEXT: module asm ".equ foo,bar"
+
+; ASM2:      module asm ".lto_discard foo"
+; ASM2-NEXT: module asm ".weak foo"
+; ASM2-NEXT: module asm ".equ foo,bar"
+; ASM2-NEXT: module asm ".lto_discard"
+; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  module asm ".lto_discard foo"
 
 ; SYM: T foo
 
diff --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
deleted file mode 100644
index b097c7067ccc1..0000000000000
--- a/llvm/test/Linker/Inputs/module-asm.ll
+++ /dev/null
@@ -1,2 +0,0 @@
-module asm(target_features: "+bar")
-    "asm 2"
diff --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 612bce5be18f1..5e12a8c88d90a 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,12 +9,11 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK: module asm
-; CHECK-NEXT: ".text"
-; CHECK-NEXT: ".balign 4"
-; CHECK-NEXT: ".arm"
-; CHECK-NEXT: "add r1, r2, r2"
-; CHECK-NEXT: ".text"
-; CHECK-NEXT: ".balign 2"
-; CHECK-NEXT: ".thumb"
-; CHECK-NEXT: "orn r1, r2, r2"
+; CHECK:      module asm ".text"
+; CHECK-NEXT: module asm ".balign 4"
+; CHECK-NEXT: module asm ".arm"
+; CHECK-NEXT: module asm "add r1, r2, r2"
+; CHECK-NEXT: module asm ".text"
+; CHECK-NEXT: module asm ".balign 2"
+; CHECK-NEXT: module asm ".thumb"
+; CHECK-NEXT: module asm "orn r1, r2, r2"
diff --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
deleted file mode 100644
index 6c600981d72a5..0000000000000
--- a/llvm/test/Linker/module-asm.ll
+++ /dev/null
@@ -1,9 +0,0 @@
-; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
-
-; CHECK: module asm(target_features: "+foo")
-; CHECK-NEXT: "asm 1"
-; CHECK: module asm(target_features: "+bar")
-; CHECK-NEXT: "asm 2"
-
-module asm(target_features: "+foo")
-    "asm 1"
diff --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 9525527b4531c..556c4fd992f0d 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,8 +9,7 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm
-; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.
diff --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index 7c3ed3da3537e..ea4594a359cc0 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,7 +1,6 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm
-; CHECK-NEXT: ".symver external_addrtaken, alias1"
+; CHECK: module asm ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index 61141c71335bf..d8ebae17d4693 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,8 +3,7 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm
-; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 5796537661e60..587ab3fbb5020 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,7 +7,6 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm
-; M0-NEXT: "ret"
+; M0: module asm "ret"
 ; M1-NOT: module asm
 module asm "ret"
diff --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index b5b68404e8690..4aeaef6d8d676 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.removeModuleInlineAsm();
+    Program.setModuleInlineAsm("");
 }
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index bb55678f91005..3038a6a2ed986 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,15 +1150,12 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
+  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (const llvm::Module::GlobalAsmFragment &Frag :
-       llvmModule->getModuleInlineAsm()) {
-    // TODO: Preserve module asm properties.
-    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
-      if (!line.empty())
-        asmArrayAttr.push_back(builder.getStringAttr(line));
-  }
+  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
+    if (!line.empty())
+      asmArrayAttr.push_back(builder.getStringAttr(line));
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));
diff --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 813eeeafdeb44..2afb37cf005bf 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,6 +2,5 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm
-// CHECK-NEXT: "foo"
-// CHECK-NEXT: "bar"
+// CHECK: module asm "foo"
+// CHECK: module asm "bar"


From llvmlistbot at llvm.org  Fri Jul  3 07:20:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:20:24 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c528.13be6df8.3d8759.0462@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-risc-v

Author: Hans Wennborg (zmodem)

<details>
<summary>Changes</summary>

Reverts llvm/llvm-project#<!-- -->204548

This is causing the runtimes build to fail with e.g.:

```
&lt;inline asm&gt;:11:5: error: 32 bit reloc applied to a field with a different size
   11 | jmp __interceptor_strlen@<!-- -->plt
      |     ^
```

See comments on the PR.

---

Patch is 56.65 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207399.diff


59 Files Affected:

- (modified) clang/lib/CodeGen/CGObjCMac.cpp (+5-1) 
- (modified) clang/lib/CodeGen/CodeGenModule.cpp (+1-7) 
- (modified) clang/test/CodeGen/2006-01-23-FileScopeAsm.c (+6-11) 
- (modified) clang/test/CodeGen/asm.c (+3-4) 
- (modified) clang/test/CodeGen/asm_incbin.c (+1-2) 
- (modified) clang/test/CodeGen/cfi-salt.c (+5-6) 
- (modified) clang/test/CodeGen/kcfi.c (+3-4) 
- (modified) clang/test/CodeGenCUDA/filter-decl.cu (+2-3) 
- (modified) clang/test/CodeGenCXX/gnu-asm-constexpr.cpp (+1-2) 
- (modified) clang/test/CodeGenObjC/category-class.m (+3-4) 
- (modified) clang/test/CodeGenObjC/objc-runtime-name.m (+3-4) 
- (modified) clang/test/Frontend/ast-codegen.c (+1-2) 
- (modified) clang/test/Modules/codegen.test (+1-2) 
- (modified) llvm/docs/LangRef.rst (+6-21) 
- (modified) llvm/docs/ReleaseNotes.md (-4) 
- (modified) llvm/include/llvm/Bitcode/LLVMBitCodes.h (-2) 
- (modified) llvm/include/llvm/CodeGen/AsmPrinter.h (-13) 
- (modified) llvm/include/llvm/IR/Module.h (+11-85) 
- (modified) llvm/lib/AsmParser/LLParser.cpp (+3-36) 
- (modified) llvm/lib/Bitcode/Reader/BitcodeReader.cpp (+1-16) 
- (modified) llvm/lib/Bitcode/Writer/BitcodeWriter.cpp (+3-13) 
- (modified) llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp (+6-22) 
- (modified) llvm/lib/IR/AsmWriter.cpp (+13-30) 
- (modified) llvm/lib/IR/Core.cpp (+1-12) 
- (modified) llvm/lib/IR/Module.cpp (-20) 
- (modified) llvm/lib/LTO/LTO.cpp (+2-2) 
- (modified) llvm/lib/Linker/IRMover.cpp (+3-6) 
- (modified) llvm/lib/Object/ModuleSymbolTable.cpp (+44-46) 
- (modified) llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp (+9-14) 
- (modified) llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp (+1-1) 
- (modified) llvm/lib/Transforms/IPO/ExtractGV.cpp (+1-1) 
- (modified) llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp (+1-1) 
- (modified) llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp (+12-14) 
- (modified) llvm/lib/Transforms/Utils/SplitModule.cpp (+1-1) 
- (removed) llvm/test/Assembler/module-asm-invalid.ll (-25) 
- (modified) llvm/test/Bitcode/compatibility-3.6.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.7.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.8.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.9.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-4.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-5.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-6.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility.ll (+1-2) 
- (modified) llvm/test/Bitcode/highLevelStructure.3.2.ll (+1-2) 
- (removed) llvm/test/Bitcode/module-asm.ll (-29) 
- (removed) llvm/test/CodeGen/RISCV/module-asm-features.ll (-17) 
- (modified) llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll (+1-2) 
- (removed) llvm/test/LTO/RISCV/module-asm.ll (-18) 
- (modified) llvm/test/LTO/X86/inline-asm-lto-discard.ll (+11-13) 
- (removed) llvm/test/Linker/Inputs/module-asm.ll (-2) 
- (modified) llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll (+8-9) 
- (removed) llvm/test/Linker/module-asm.ll (-9) 
- (modified) llvm/test/ThinLTO/X86/import-symver.ll (+1-2) 
- (modified) llvm/test/Transforms/LowerTypeTests/export-symver.ll (+1-2) 
- (modified) llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll (+1-2) 
- (modified) llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll (+1-2) 
- (modified) llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp (+1-1) 
- (modified) mlir/lib/Target/LLVMIR/ModuleImport.cpp (+4-7) 
- (modified) mlir/test/Target/LLVMIR/module-asm.mlir (+2-3) 


``````````diff
diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index e518b875f44bb..69c5e88f3c768 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,6 +6610,10 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
+    Asm += CGM.getModule().getModuleInlineAsm();
+    if (!Asm.empty() && Asm.back() != '\n')
+      Asm += '\n';
+
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6620,7 +6624,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().appendModuleInlineAsm(OS.str());
+    CGM.getModule().setModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index e155fdd752d7f..813cd67e4dd51 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,13 +8057,7 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-
-    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
-    llvm::Module::GlobalAsmProperties Props;
-    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
-    Props.TargetCPU = TargetOpts.CPU;
-    getModule().appendModuleInlineAsm(
-        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
+    getModule().appendModuleInlineAsm(AD->getAsmString());
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 8a212a0b61905..877da4fc669e3 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,18 +1,13 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
-// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// RISCV64: module asm(target_features: "+64bit,+i")
-// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
-// DEFAULT: module asm
-// CHECK-NEXT: "foo1"
+// CHECK: module asm "foo1"
 __asm__ ("foo1");
-// CHECK-NEXT: "foo2"
+// CHECK: module asm "foo2"
 __asm__ ("foo2");
-// CHECK-NEXT: "foo3"
+// CHECK: module asm "foo3"
 __asm__ ("foo3");
-// CHECK-NEXT: "foo4"
+// CHECK: module asm "foo4"
 __asm__ ("foo4");
-// CHECK-NEXT: "foo5"
+// CHECK: module asm "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index ea69c7a17d1cb..d7465b22fbbf6 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,10 +2,9 @@
 
 // PR10415:
 //
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo1"
-// CHECK-NEXT: "foo2"
-// CHECK-NEXT: "foo3"
+// CHECK:      module asm "foo1"
+// CHECK-NEXT: module asm "foo2"
+// CHECK-NEXT: module asm "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index bbc5d3af55ea2..6234eea1d6738 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,5 +6,4 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm
-// CHECK-NEXT: ".incbin \22foo.h\22"
+// CHECK: module asm ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 18156dc014280..2a90dc31d939b 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,14 +26,13 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_f6"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index f08cf7635fb5a..da32859dd3d88 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,11 +6,10 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index f87d62a56631e..02dacd0ad8ef4 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,9 +5,8 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm
-// CHECK-HOST-NEXT: "file scope asm is host only"
-// CHECK-DEVICE-NOT: "file scope asm is host only"
+// CHECK-HOST: module asm "file scope asm is host only"
+// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 2b4abee5dc05f..175a3b7bc588c 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,8 +19,7 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm(target_features: "{{.*}}")
-    // CHECK-NEXT: "nop; nop; nop; nop"
+    // CHECK: module asm "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index febe94ce5b9f5..92fd36cbe4066 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,10 +1,9 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm
-// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
-// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
-// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
+// CHECK: module asm "\09.objc_category_name_A_foo=0"
+// CHECK: module asm "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index a7f59caced390..c838b8c2c6f1a 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,10 +3,9 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm
-// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
-// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
-// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm {{.*}}objc_class_name_XYZ=0
+// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
+// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index c26b0a1908379..2e2e3d360dd17 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,8 +2,7 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo"
+// CHECK: module asm "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 5cc5b3c1cda2a..0af630a754805 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,8 +18,7 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm
-BOTH-NEXT: "narf"
+BOTH: module asm "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2c8276d04b668..5ce6019660e0b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,34 +3357,19 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks:
+to the GCC "file scope inline asm" blocks. These blocks are internally
+concatenated by LLVM and treated as a single unit, but may be separated
+in the ``.ll`` file if desired. The syntax is very simple:
 
 .. code-block:: llvm
 
-    module asm
-        "inline asm code goes here"
-        "more can go here"
+    module asm "inline asm code goes here"
+    module asm "more can go here"
 
-The sequence of string literals is internally concatenated using newlines as
-separators. The strings can contain any character by escaping non-printable
+The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
-Additionally, module-level inline assembly can specify an optional list of
-properties:
-
-.. code-block:: llvm
-
-    module asm(target_features: "+foo", target_cpu: "bar")
-        "inline asm code goes here"
-        "more can go here"
-
-Currently, the only supported properties are ``target_features`` and
-``target_cpu``, with the same meaning as the ``"target-features"`` and
-``"target-cpu"`` function attributes.
-
-Consecutive blocks with identical properties will be concatenated into one.
-
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cbec4d5b912d0..e42a959ab3101 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -111,10 +111,6 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
-* Module-level inline assembly now accepts optional `target_features` and
-  `target_cpu` properties. This resolves errors during LTO on some
-  architectures.
-
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 1761e7a542a73..af4cac996a78c 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,8 +120,6 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
-
-  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 335bf9de9a071..78e64440657cc 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,19 +930,6 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
-  /// Emit necessary directives to allow use of instructions that are permitted
-  /// by target features enabled by STI, but are not permitted by target
-  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
-  /// Returns whether anything was emitted.
-  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
-    return false;
-  }
-
-  /// Emit necessary directives to restore target feature state.
-  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
-  /// call.
-  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
-
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 63d1013896b47..4f7e33969f16f 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,45 +169,6 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
-  struct GlobalAsmProperties {
-    std::string TargetFeatures;
-    std::string TargetCPU;
-
-    /// Set a property using a string name.
-    /// Returns whether the property name was valid.
-    LLVM_ABI bool set(StringRef Name, std::string Value);
-
-    /// Get a list of set properties as pairs of key and value.
-    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
-
-    bool operator==(const GlobalAsmProperties &Other) const {
-      return TargetFeatures == Other.TargetFeatures &&
-             TargetCPU == Other.TargetCPU;
-    }
-
-    bool operator!=(const GlobalAsmProperties &Other) const {
-      return !(*this == Other);
-    }
-  };
-
-  struct GlobalAsmFragment {
-    std::string Asm;
-    GlobalAsmProperties Props;
-
-    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
-    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
-        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
-      if (!Asm.empty() && Asm.back() != '\n')
-        Asm += '\n';
-    }
-
-    bool empty() const { return Asm.empty(); }
-
-    bool hasSameProperties(const GlobalAsmFragment &Other) const {
-      return Props == Other.Props;
-    }
-  };
-
 /// @}
 /// @name Member Variables
 /// @{
@@ -219,8 +180,7 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  /// Inline Asm at the global scope.
-  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
+  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -327,17 +287,8 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
-    return GlobalScopeAsm;
-  }
-
-  /// Get any module-scope inline assembly blocks.
-  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
-    return GlobalScopeAsm;
-  }
-
-  /// Return whether there is any module-scope inline assembly.
-  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
+  /// @returns a string containing the module-scope inline assembly blocks.
+  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -374,45 +325,20 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
-  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
-
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
-    GlobalScopeAsm.clear();
-    appendModuleInlineAsm(std::move(Fragment));
-  }
-
-  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
-    GlobalScopeAsm.clear();
-    append_range(GlobalScopeAsm, Fragments);
+  void setModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm = std::string(Asm);
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.back().Asm += Fragment.Asm;
-    } else {
-      GlobalScopeAsm.emplace_back(std::move(Fragment));
-    }
-  }
-
-  /// Prepend to the module-scope inline assembly blocks.
-  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
-    } else {
-      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
-    }
+  void appendModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm += Asm;
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 0a5b4c6c91879..75e6add0ec76f 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,49 +641,16 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
-///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
-///                          'property_name2:' STRINGCONSTANT ')'
-///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
+      parseStringConstant(AsmStr))
     return true;
 
-  Module::GlobalAsmProperties Props;
-  if (EatIfPresent(lltok::lparen)) {
-    while (true) {
-      std::string Key, Value;
-      SMLoc Loc = Lex.getLoc();
-      if (Lex.getKind() != lltok::LabelStr)
-        return error(Loc, "expected property name followed by ':'");
-
-      Key = Lex.getStrVal();
-      Lex.Lex();
-
-      if (parseStringConstant(Value))
-        return true;
-
-      if (!Props.set(Key, Value))
-        return error(Loc, "unknown property name");
-
-      if (EatIfPresent(lltok::rparen))
-        break;
-      if (parseToken(lltok::comma, "expected ',' or ')'"))
-        return true;
-    }
-  }
-
-  do {
-    std::string AsmStrPart;
-    if (parseStringConstant(AsmStrPart))
-      return true;
-    AsmStr += AsmStrPart + "...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:20:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:20:25 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c529.2020f6c2.1fdf61.d6b8@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-spir-v

Author: Hans Wennborg (zmodem)

<details>
<summary>Changes</summary>

Reverts llvm/llvm-project#<!-- -->204548

This is causing the runtimes build to fail with e.g.:

```
&lt;inline asm&gt;:11:5: error: 32 bit reloc applied to a field with a different size
   11 | jmp __interceptor_strlen@<!-- -->plt
      |     ^
```

See comments on the PR.

---

Patch is 56.65 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207399.diff


59 Files Affected:

- (modified) clang/lib/CodeGen/CGObjCMac.cpp (+5-1) 
- (modified) clang/lib/CodeGen/CodeGenModule.cpp (+1-7) 
- (modified) clang/test/CodeGen/2006-01-23-FileScopeAsm.c (+6-11) 
- (modified) clang/test/CodeGen/asm.c (+3-4) 
- (modified) clang/test/CodeGen/asm_incbin.c (+1-2) 
- (modified) clang/test/CodeGen/cfi-salt.c (+5-6) 
- (modified) clang/test/CodeGen/kcfi.c (+3-4) 
- (modified) clang/test/CodeGenCUDA/filter-decl.cu (+2-3) 
- (modified) clang/test/CodeGenCXX/gnu-asm-constexpr.cpp (+1-2) 
- (modified) clang/test/CodeGenObjC/category-class.m (+3-4) 
- (modified) clang/test/CodeGenObjC/objc-runtime-name.m (+3-4) 
- (modified) clang/test/Frontend/ast-codegen.c (+1-2) 
- (modified) clang/test/Modules/codegen.test (+1-2) 
- (modified) llvm/docs/LangRef.rst (+6-21) 
- (modified) llvm/docs/ReleaseNotes.md (-4) 
- (modified) llvm/include/llvm/Bitcode/LLVMBitCodes.h (-2) 
- (modified) llvm/include/llvm/CodeGen/AsmPrinter.h (-13) 
- (modified) llvm/include/llvm/IR/Module.h (+11-85) 
- (modified) llvm/lib/AsmParser/LLParser.cpp (+3-36) 
- (modified) llvm/lib/Bitcode/Reader/BitcodeReader.cpp (+1-16) 
- (modified) llvm/lib/Bitcode/Writer/BitcodeWriter.cpp (+3-13) 
- (modified) llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp (+6-22) 
- (modified) llvm/lib/IR/AsmWriter.cpp (+13-30) 
- (modified) llvm/lib/IR/Core.cpp (+1-12) 
- (modified) llvm/lib/IR/Module.cpp (-20) 
- (modified) llvm/lib/LTO/LTO.cpp (+2-2) 
- (modified) llvm/lib/Linker/IRMover.cpp (+3-6) 
- (modified) llvm/lib/Object/ModuleSymbolTable.cpp (+44-46) 
- (modified) llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp (+9-14) 
- (modified) llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp (+1-1) 
- (modified) llvm/lib/Transforms/IPO/ExtractGV.cpp (+1-1) 
- (modified) llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp (+1-1) 
- (modified) llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp (+12-14) 
- (modified) llvm/lib/Transforms/Utils/SplitModule.cpp (+1-1) 
- (removed) llvm/test/Assembler/module-asm-invalid.ll (-25) 
- (modified) llvm/test/Bitcode/compatibility-3.6.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.7.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.8.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-3.9.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-4.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-5.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility-6.0.ll (+1-2) 
- (modified) llvm/test/Bitcode/compatibility.ll (+1-2) 
- (modified) llvm/test/Bitcode/highLevelStructure.3.2.ll (+1-2) 
- (removed) llvm/test/Bitcode/module-asm.ll (-29) 
- (removed) llvm/test/CodeGen/RISCV/module-asm-features.ll (-17) 
- (modified) llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll (+1-2) 
- (removed) llvm/test/LTO/RISCV/module-asm.ll (-18) 
- (modified) llvm/test/LTO/X86/inline-asm-lto-discard.ll (+11-13) 
- (removed) llvm/test/Linker/Inputs/module-asm.ll (-2) 
- (modified) llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll (+8-9) 
- (removed) llvm/test/Linker/module-asm.ll (-9) 
- (modified) llvm/test/ThinLTO/X86/import-symver.ll (+1-2) 
- (modified) llvm/test/Transforms/LowerTypeTests/export-symver.ll (+1-2) 
- (modified) llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll (+1-2) 
- (modified) llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll (+1-2) 
- (modified) llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp (+1-1) 
- (modified) mlir/lib/Target/LLVMIR/ModuleImport.cpp (+4-7) 
- (modified) mlir/test/Target/LLVMIR/module-asm.mlir (+2-3) 


``````````diff
diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index e518b875f44bb..69c5e88f3c768 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,6 +6610,10 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
+    Asm += CGM.getModule().getModuleInlineAsm();
+    if (!Asm.empty() && Asm.back() != '\n')
+      Asm += '\n';
+
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6620,7 +6624,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().appendModuleInlineAsm(OS.str());
+    CGM.getModule().setModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index e155fdd752d7f..813cd67e4dd51 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,13 +8057,7 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-
-    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
-    llvm::Module::GlobalAsmProperties Props;
-    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
-    Props.TargetCPU = TargetOpts.CPU;
-    getModule().appendModuleInlineAsm(
-        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
+    getModule().appendModuleInlineAsm(AD->getAsmString());
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 8a212a0b61905..877da4fc669e3 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,18 +1,13 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
-// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// RISCV64: module asm(target_features: "+64bit,+i")
-// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
-// DEFAULT: module asm
-// CHECK-NEXT: "foo1"
+// CHECK: module asm "foo1"
 __asm__ ("foo1");
-// CHECK-NEXT: "foo2"
+// CHECK: module asm "foo2"
 __asm__ ("foo2");
-// CHECK-NEXT: "foo3"
+// CHECK: module asm "foo3"
 __asm__ ("foo3");
-// CHECK-NEXT: "foo4"
+// CHECK: module asm "foo4"
 __asm__ ("foo4");
-// CHECK-NEXT: "foo5"
+// CHECK: module asm "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index ea69c7a17d1cb..d7465b22fbbf6 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,10 +2,9 @@
 
 // PR10415:
 //
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo1"
-// CHECK-NEXT: "foo2"
-// CHECK-NEXT: "foo3"
+// CHECK:      module asm "foo1"
+// CHECK-NEXT: module asm "foo2"
+// CHECK-NEXT: module asm "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index bbc5d3af55ea2..6234eea1d6738 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,5 +6,4 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm
-// CHECK-NEXT: ".incbin \22foo.h\22"
+// CHECK: module asm ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 18156dc014280..2a90dc31d939b 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,14 +26,13 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_f6"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index f08cf7635fb5a..da32859dd3d88 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,11 +6,10 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index f87d62a56631e..02dacd0ad8ef4 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,9 +5,8 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm
-// CHECK-HOST-NEXT: "file scope asm is host only"
-// CHECK-DEVICE-NOT: "file scope asm is host only"
+// CHECK-HOST: module asm "file scope asm is host only"
+// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 2b4abee5dc05f..175a3b7bc588c 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,8 +19,7 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm(target_features: "{{.*}}")
-    // CHECK-NEXT: "nop; nop; nop; nop"
+    // CHECK: module asm "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index febe94ce5b9f5..92fd36cbe4066 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,10 +1,9 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm
-// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
-// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
-// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
+// CHECK: module asm "\09.objc_category_name_A_foo=0"
+// CHECK: module asm "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index a7f59caced390..c838b8c2c6f1a 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,10 +3,9 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm
-// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
-// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
-// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm {{.*}}objc_class_name_XYZ=0
+// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
+// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index c26b0a1908379..2e2e3d360dd17 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,8 +2,7 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo"
+// CHECK: module asm "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 5cc5b3c1cda2a..0af630a754805 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,8 +18,7 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm
-BOTH-NEXT: "narf"
+BOTH: module asm "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2c8276d04b668..5ce6019660e0b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,34 +3357,19 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks:
+to the GCC "file scope inline asm" blocks. These blocks are internally
+concatenated by LLVM and treated as a single unit, but may be separated
+in the ``.ll`` file if desired. The syntax is very simple:
 
 .. code-block:: llvm
 
-    module asm
-        "inline asm code goes here"
-        "more can go here"
+    module asm "inline asm code goes here"
+    module asm "more can go here"
 
-The sequence of string literals is internally concatenated using newlines as
-separators. The strings can contain any character by escaping non-printable
+The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
-Additionally, module-level inline assembly can specify an optional list of
-properties:
-
-.. code-block:: llvm
-
-    module asm(target_features: "+foo", target_cpu: "bar")
-        "inline asm code goes here"
-        "more can go here"
-
-Currently, the only supported properties are ``target_features`` and
-``target_cpu``, with the same meaning as the ``"target-features"`` and
-``"target-cpu"`` function attributes.
-
-Consecutive blocks with identical properties will be concatenated into one.
-
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cbec4d5b912d0..e42a959ab3101 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -111,10 +111,6 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
-* Module-level inline assembly now accepts optional `target_features` and
-  `target_cpu` properties. This resolves errors during LTO on some
-  architectures.
-
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 1761e7a542a73..af4cac996a78c 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,8 +120,6 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
-
-  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 335bf9de9a071..78e64440657cc 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,19 +930,6 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
-  /// Emit necessary directives to allow use of instructions that are permitted
-  /// by target features enabled by STI, but are not permitted by target
-  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
-  /// Returns whether anything was emitted.
-  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
-    return false;
-  }
-
-  /// Emit necessary directives to restore target feature state.
-  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
-  /// call.
-  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
-
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 63d1013896b47..4f7e33969f16f 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,45 +169,6 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
-  struct GlobalAsmProperties {
-    std::string TargetFeatures;
-    std::string TargetCPU;
-
-    /// Set a property using a string name.
-    /// Returns whether the property name was valid.
-    LLVM_ABI bool set(StringRef Name, std::string Value);
-
-    /// Get a list of set properties as pairs of key and value.
-    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
-
-    bool operator==(const GlobalAsmProperties &Other) const {
-      return TargetFeatures == Other.TargetFeatures &&
-             TargetCPU == Other.TargetCPU;
-    }
-
-    bool operator!=(const GlobalAsmProperties &Other) const {
-      return !(*this == Other);
-    }
-  };
-
-  struct GlobalAsmFragment {
-    std::string Asm;
-    GlobalAsmProperties Props;
-
-    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
-    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
-        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
-      if (!Asm.empty() && Asm.back() != '\n')
-        Asm += '\n';
-    }
-
-    bool empty() const { return Asm.empty(); }
-
-    bool hasSameProperties(const GlobalAsmFragment &Other) const {
-      return Props == Other.Props;
-    }
-  };
-
 /// @}
 /// @name Member Variables
 /// @{
@@ -219,8 +180,7 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  /// Inline Asm at the global scope.
-  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
+  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -327,17 +287,8 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
-    return GlobalScopeAsm;
-  }
-
-  /// Get any module-scope inline assembly blocks.
-  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
-    return GlobalScopeAsm;
-  }
-
-  /// Return whether there is any module-scope inline assembly.
-  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
+  /// @returns a string containing the module-scope inline assembly blocks.
+  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -374,45 +325,20 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
-  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
-
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
-    GlobalScopeAsm.clear();
-    appendModuleInlineAsm(std::move(Fragment));
-  }
-
-  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
-    GlobalScopeAsm.clear();
-    append_range(GlobalScopeAsm, Fragments);
+  void setModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm = std::string(Asm);
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.back().Asm += Fragment.Asm;
-    } else {
-      GlobalScopeAsm.emplace_back(std::move(Fragment));
-    }
-  }
-
-  /// Prepend to the module-scope inline assembly blocks.
-  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
-    } else {
-      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
-    }
+  void appendModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm += Asm;
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 0a5b4c6c91879..75e6add0ec76f 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,49 +641,16 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
-///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
-///                          'property_name2:' STRINGCONSTANT ')'
-///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
+      parseStringConstant(AsmStr))
     return true;
 
-  Module::GlobalAsmProperties Props;
-  if (EatIfPresent(lltok::lparen)) {
-    while (true) {
-      std::string Key, Value;
-      SMLoc Loc = Lex.getLoc();
-      if (Lex.getKind() != lltok::LabelStr)
-        return error(Loc, "expected property name followed by ':'");
-
-      Key = Lex.getStrVal();
-      Lex.Lex();
-
-      if (parseStringConstant(Value))
-        return true;
-
-      if (!Props.set(Key, Value))
-        return error(Loc, "unknown property name");
-
-      if (EatIfPresent(lltok::rparen))
-        break;
-      if (parseToken(lltok::comma, "expected ',' or ')'"))
-        return true;
-    }
-  }
-
-  do {
-    std::string AsmStrPart;
-    if (parseStringConstant(AsmStrPart))
-      return true;
-    AsmStr += AsmStrPart + "...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:21:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:21:25 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c565.bd93e731.b068b.5dba@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,c,cpp -- clang/lib/CodeGen/CGObjCMac.cpp clang/lib/CodeGen/CodeGenModule.cpp clang/test/CodeGen/2006-01-23-FileScopeAsm.c clang/test/CodeGen/asm.c clang/test/CodeGen/asm_incbin.c clang/test/CodeGen/cfi-salt.c clang/test/CodeGen/kcfi.c clang/test/CodeGenCXX/gnu-asm-constexpr.cpp clang/test/Frontend/ast-codegen.c llvm/include/llvm/Bitcode/LLVMBitCodes.h llvm/include/llvm/CodeGen/AsmPrinter.h llvm/include/llvm/IR/Module.h llvm/lib/AsmParser/LLParser.cpp llvm/lib/Bitcode/Reader/BitcodeReader.cpp llvm/lib/Bitcode/Writer/BitcodeWriter.cpp llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp llvm/lib/IR/AsmWriter.cpp llvm/lib/IR/Core.cpp llvm/lib/IR/Module.cpp llvm/lib/LTO/LTO.cpp llvm/lib/Linker/IRMover.cpp llvm/lib/Object/ModuleSymbolTable.cpp llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp llvm/lib/Transforms/IPO/ExtractGV.cpp llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp llvm/lib/Transforms/Utils/SplitModule.cpp llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp mlir/lib/Target/LLVMIR/ModuleImport.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde..cb3782ecd 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,8 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    DstM.appendModuleInlineAsm(
+        adjustInlineAsm(SrcM->getModuleInlineAsm(), SrcTriple));
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,

``````````

</details>


https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:21:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:21:58 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47c586.14d9b9dc.140829.3080@mx.google.com>


ronlieb wrote:

i am seeing same issue:
 <inline asm>:8:5: error: 32 bit reloc applied to a field with a different size
    8 | jmp __interceptor_fread_unlocked at plt
      |     ^


https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 07:23:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:23:07 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c5cb.492e23e4.e71ca.e5be@mx.google.com>


https://github.com/ronlieb approved this pull request.


https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:23:46 2026
From: llvmlistbot at llvm.org (Mikael Holmen)
Date: Fri, 03 Jul 2026 07:23:46 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c5f2.eb24c6ea.1f2e26.7460@mx.google.com>


https://github.com/mikaelholmen approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:27:21 2026
From: llvmlistbot at llvm.org (Hans Wennborg)
Date: Fri, 03 Jul 2026 07:27:21 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47c6c9.e55dc341.29c9b8.73d2@mx.google.com>


https://github.com/zmodem auto_merge_enabled https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 07:30:26 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Fri, 03 Jul 2026 07:30:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
Message-ID: <llvm.org/llvm/llvm-project/pull/207403@github.com>


https://github.com/javedabsar1 created https://github.com/llvm/llvm-project/pull/207403

Introduces an algorithm selection option to the static memory planner pass and
adds a best-fit algorithm that reuses memory from expired allocations by finding
the smallest suitable gap.

The planning algorithms are factored into a separate StaticMemoryPlanning.{h,cpp}
to keep them independent of MLIR IR and easily testable.

>From 98b6b5222a88a2c93e411cf3fc1cdaf2c25a09bd Mon Sep 17 00:00:00 2001
From: mabsar <mabsar at qti.qualcommm.com>
Date: Fri, 3 Jul 2026 07:20:15 -0700
Subject: [PATCH] [mlir][bufferization] Add best-fit algorithm to static memory
 planner

Introduces an algorithm selection option to the static memory planner pass
and add a best-fit algorithm that reuses memory from expired allocations
by finding the smallest suitable gap. The planning algorithms are factored
into a separate StaticMemoryPlanning.{h,cpp} to keep them independent of
MLIR IR and easily testable.

Signed-off-by: mabsar <mabsar at qti.qualcommm.com>
---
 .../Bufferization/IR/BufferizationEnums.td    |   8 ++
 .../Bufferization/Transforms/Passes.td        |  31 +++--
 .../Transforms/StaticMemoryPlanning.h         |  50 ++++++++
 .../Bufferization/Transforms/CMakeLists.txt   |   1 +
 .../StaticMemoryPlannerAnalysis.cpp           |  97 ++++++--------
 .../Transforms/StaticMemoryPlanning.cpp       | 111 ++++++++++++++++
 .../static-memory-planner-best-fit.mlir       | 118 ++++++++++++++++++
 7 files changed, 344 insertions(+), 72 deletions(-)
 create mode 100644 mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
 create mode 100644 mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c695230870bde 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,38 +68,6 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
-}
-
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
-
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
-  }
-
-  return offsets;
-}
-
 //===----------------------------------------------------------------------===//
 // StaticMemoryPlannerAnalysisPass
 //===----------------------------------------------------------------------===//
@@ -137,7 +92,7 @@ struct StaticMemoryPlannerAnalysisPass
       }
     }
 
-    // Step 1: Collect eligible allocation candidates
+    // Step 1: Collect eligible allocation candidates.
     SmallVector<AllocationCandidate> candidates;
 
     funcOp->walk([&](memref::AllocOp allocOp) {
@@ -174,22 +129,40 @@ struct StaticMemoryPlannerAnalysisPass
     if (candidates.empty())
       return;
 
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
+    // Step 2: Build allocation descriptors with lifetime info.
+    SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
     int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
+    for (auto &candidate : candidates) {
+      bufferization::MemoryPlannerAlloc info;
+      info.sizeInBytes = candidate.sizeInBytes;
+      info.alignment = candidate.alignment;
+
+      Block *block = candidate.alloc->getBlock();
+      int64_t opIdx = 0;
+      for (Operation &op : *block) {
+        if (&op == candidate.alloc.getOperation())
+          info.timeStart = opIdx;
+        if (&op == candidate.dealloc.getOperation())
+          info.timeEnd = opIdx;
+        ++opIdx;
+      }
+
+      allocInfos.push_back(info);
       arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
     }
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
+    // Step 3: Run the planning algorithm.
+    SmallVector<int64_t> offsets;
+    switch (algorithm) {
+    case bufferization::MemoryPlannerAlgorithm::Trivial:
+      offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    case bufferization::MemoryPlannerAlgorithm::BestFit:
+      offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    }
 
-    // Assign computed offsets back to candidates
+    // Step 4: Compute total arena size and assign offsets.
     int64_t totalSize = 0;
     for (size_t i = 0; i < candidates.size(); ++i) {
       candidates[i].offset = offsets[i];
@@ -200,7 +173,7 @@ struct StaticMemoryPlannerAnalysisPass
                  << " alignment=" << candidates[i].alignment << "\n");
     }
 
-    // Step 4: Obtain arena based on arena mode
+    // Step 5: Obtain arena based on arena mode.
     Operation *firstAlloc = candidates.front().alloc;
     OpBuilder builder(firstAlloc);
     Value arenaValue;
@@ -242,7 +215,7 @@ struct StaticMemoryPlannerAnalysisPass
       return signalPassFailure();
     }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
+    // Step 6: Replace each alloc with memref.view into the arena.
     for (auto &candidate : candidates) {
       builder.setInsertionPoint(candidate.alloc);
       Location loc = candidate.alloc.getLoc();
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..e1779cb8e5831
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,111 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::trivialMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::bestFitMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // C at offset 5120
+  // CHECK-NEXT: %{{.*}} = arith.constant 5120 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // D at offset 9216
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // E at offset 9216 (best-fit picks 1024-byte trailing gap over 5120-byte gap)
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<256xf32>
+  %c = memref.alloc() : memref<1024xf32>
+  %d = memref.alloc() : memref<256xf32>
+  memref.dealloc %b : memref<256xf32>
+  memref.dealloc %d : memref<256xf32>
+  memref.dealloc %a : memref<1024xf32>
+  %e = memref.alloc() : memref<128xf32>
+  memref.dealloc %c : memref<1024xf32>
+  memref.dealloc %e : memref<128xf32>
+  return
+}
+
+// -----
+
+// Test 4: Alignment padding can disqualify a gap.
+// A(128, align 128) at 0 (live), B(56, align 1) at 128, C(128, align 128) at 256 (live).
+// B is freed => gap [128, 256) = 128 bytes.
+// D(64, align 128): aligned start = 128, fits in gap.
+// E(64, align 256): next 256-aligned offset in [128,256) is 256 = gap end, doesn't fit.
+//   E must go past the arena high-water mark.
+// CHECK-LABEL: func @best_fit_alignment
+func.func @best_fit_alignment() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 256 : i64} : memref<576xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<56xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 256 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // D fits in gap at offset 128
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  // E cannot fit in gap (alignment 256), placed at offset 512
+  // CHECK-NEXT: %{{.*}} = arith.constant 512 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  %a = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  %b = memref.alloc() : memref<56xi8>
+  %c = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  memref.dealloc %b : memref<56xi8>
+  %d = memref.alloc() {alignment = 128 : i64} : memref<64xi8>
+  %e = memref.alloc() {alignment = 256 : i64} : memref<64xi8>
+  memref.dealloc %a : memref<128xi8>
+  memref.dealloc %c : memref<128xi8>
+  memref.dealloc %d : memref<64xi8>
+  memref.dealloc %e : memref<64xi8>
+  return
+}


From llvmlistbot at llvm.org  Fri Jul  3 07:31:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:31:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a47c7a9.e55dc341.29c9b8.1ccd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-bufferization

Author: Javed Absar (javedabsar1)

<details>
<summary>Changes</summary>

Introduces an algorithm selection option to the static memory planner pass and
adds a best-fit algorithm that reuses memory from expired allocations by finding
the smallest suitable gap.

The planning algorithms are factored into a separate StaticMemoryPlanning.{h,cpp}
to keep them independent of MLIR IR and easily testable.

---

Patch is 22.62 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207403.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td (+8) 
- (modified) mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td (+21-10) 
- (added) mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h (+50) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt (+1) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp (+35-62) 
- (added) mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp (+111) 
- (added) mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir (+118) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c695230870bde 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,38 +68,6 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
-}
-
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
-
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
-  }
-
-  return offsets;
-}
-
 //===----------------------------------------------------------------------===//
 // StaticMemoryPlannerAnalysisPass
 //===----------------------------------------------------------------------===//
@@ -137,7 +92,7 @@ struct StaticMemoryPlannerAnalysisPass
       }
     }
 
-    // Step 1: Collect eligible allocation candidates
+    // Step 1: Collect eligible allocation candidates.
     SmallVector<AllocationCandidate> candidates;
 
     funcOp->walk([&](memref::AllocOp allocOp) {
@@ -174,22 +129,40 @@ struct StaticMemoryPlannerAnalysisPass
     if (candidates.empty())
       return;
 
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
+    // Step 2: Build allocation descriptors with lifetime info.
+    SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
     int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
+    for (auto &candidate : candidates) {
+      bufferization::MemoryPlannerAlloc info;
+      info.sizeInBytes = candidate.sizeInBytes;
+      info.alignment = candidate.alignment;
+
+      Block *block = candidate.alloc->getBlock();
+      int64_t opIdx = 0;
+      for (Operation &op : *block) {
+        if (&op == candidate.alloc.getOperation())
+          info.timeStart = opIdx;
+        if (&op == candidate.dealloc.getOperation())
+          info.timeEnd = opIdx;
+        ++opIdx;
+      }
+
+      allocInfos.push_back(info);
       arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
     }
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
+    // Step 3: Run the planning algorithm.
+    SmallVector<int64_t> offsets;
+    switch (algorithm) {
+    case bufferization::MemoryPlannerAlgorithm::Trivial:
+      offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    case bufferization::MemoryPlannerAlgorithm::BestFit:
+      offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    }
 
-    // Assign computed offsets back to candidates
+    // Step 4: Compute total arena size and assign offsets.
     int64_t totalSize = 0;
     for (size_t i = 0; i < candidates.size(); ++i) {
       candidates[i].offset = offsets[i];
@@ -200,7 +173,7 @@ struct StaticMemoryPlannerAnalysisPass
                  << " alignment=" << candidates[i].alignment << "\n");
     }
 
-    // Step 4: Obtain arena based on arena mode
+    // Step 5: Obtain arena based on arena mode.
     Operation *firstAlloc = candidates.front().alloc;
     OpBuilder builder(firstAlloc);
     Value arenaValue;
@@ -242,7 +215,7 @@ struct StaticMemoryPlannerAnalysisPass
       return signalPassFailure();
     }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
+    // Step 6: Replace each alloc with memref.view into the arena.
     for (auto &candidate : candidates) {
       builder.setInsertionPoint(candidate.alloc);
       Location loc = candidate.alloc.getLoc();
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..e1779cb8e5831
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,111 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::trivialMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::bestFitMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}}...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Fri Jul  3 07:31:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:31:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a47c7ae.c56fb102.11841f.79f9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Javed Absar (javedabsar1)

<details>
<summary>Changes</summary>

Introduces an algorithm selection option to the static memory planner pass and
adds a best-fit algorithm that reuses memory from expired allocations by finding
the smallest suitable gap.

The planning algorithms are factored into a separate StaticMemoryPlanning.{h,cpp}
to keep them independent of MLIR IR and easily testable.

---

Patch is 22.62 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207403.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td (+8) 
- (modified) mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td (+21-10) 
- (added) mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h (+50) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt (+1) 
- (modified) mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp (+35-62) 
- (added) mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp (+111) 
- (added) mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir (+118) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c695230870bde 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,38 +68,6 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
-}
-
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
-
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
-  }
-
-  return offsets;
-}
-
 //===----------------------------------------------------------------------===//
 // StaticMemoryPlannerAnalysisPass
 //===----------------------------------------------------------------------===//
@@ -137,7 +92,7 @@ struct StaticMemoryPlannerAnalysisPass
       }
     }
 
-    // Step 1: Collect eligible allocation candidates
+    // Step 1: Collect eligible allocation candidates.
     SmallVector<AllocationCandidate> candidates;
 
     funcOp->walk([&](memref::AllocOp allocOp) {
@@ -174,22 +129,40 @@ struct StaticMemoryPlannerAnalysisPass
     if (candidates.empty())
       return;
 
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
+    // Step 2: Build allocation descriptors with lifetime info.
+    SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
     int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
+    for (auto &candidate : candidates) {
+      bufferization::MemoryPlannerAlloc info;
+      info.sizeInBytes = candidate.sizeInBytes;
+      info.alignment = candidate.alignment;
+
+      Block *block = candidate.alloc->getBlock();
+      int64_t opIdx = 0;
+      for (Operation &op : *block) {
+        if (&op == candidate.alloc.getOperation())
+          info.timeStart = opIdx;
+        if (&op == candidate.dealloc.getOperation())
+          info.timeEnd = opIdx;
+        ++opIdx;
+      }
+
+      allocInfos.push_back(info);
       arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
     }
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
+    // Step 3: Run the planning algorithm.
+    SmallVector<int64_t> offsets;
+    switch (algorithm) {
+    case bufferization::MemoryPlannerAlgorithm::Trivial:
+      offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    case bufferization::MemoryPlannerAlgorithm::BestFit:
+      offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+      break;
+    }
 
-    // Assign computed offsets back to candidates
+    // Step 4: Compute total arena size and assign offsets.
     int64_t totalSize = 0;
     for (size_t i = 0; i < candidates.size(); ++i) {
       candidates[i].offset = offsets[i];
@@ -200,7 +173,7 @@ struct StaticMemoryPlannerAnalysisPass
                  << " alignment=" << candidates[i].alignment << "\n");
     }
 
-    // Step 4: Obtain arena based on arena mode
+    // Step 5: Obtain arena based on arena mode.
     Operation *firstAlloc = candidates.front().alloc;
     OpBuilder builder(firstAlloc);
     Value arenaValue;
@@ -242,7 +215,7 @@ struct StaticMemoryPlannerAnalysisPass
       return signalPassFailure();
     }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
+    // Step 6: Replace each alloc with memref.view into the arena.
     for (auto &candidate : candidates) {
       builder.setInsertionPoint(candidate.alloc);
       Location loc = candidate.alloc.getLoc();
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..e1779cb8e5831
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,111 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::trivialMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t>
+mlir::bufferization::bestFitMemoryPlanner(int64_t arenaAlignment,
+                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}}...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Fri Jul  3 07:31:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:31:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a47c7dd.a151105c.30c1b9.8c2f@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
index e1779cb8e..eadb1307d 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -17,9 +17,8 @@ static int64_t alignOffset(int64_t offset, int64_t alignment) {
   return llvm::alignTo(offset, alignment);
 }
 
-llvm::SmallVector<int64_t>
-mlir::bufferization::trivialMemoryPlanner(int64_t arenaAlignment,
-                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
   llvm::SmallVector<int64_t> offsets;
   int64_t currentOffset = 0;
 
@@ -34,9 +33,8 @@ mlir::bufferization::trivialMemoryPlanner(int64_t arenaAlignment,
   return offsets;
 }
 
-llvm::SmallVector<int64_t>
-mlir::bufferization::bestFitMemoryPlanner(int64_t arenaAlignment,
-                                          llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
   struct Placement {
     int64_t offset;
     int64_t size;

``````````

</details>


https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Fri Jul  3 07:33:41 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Fri, 03 Jul 2026 07:33:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a47c845.510f576d.185953.7ba7@mx.google.com>


================
@@ -0,0 +1,19 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+module {
----------------
kuhar wrote:

you can drop the explicit module

https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Fri Jul  3 07:33:41 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Fri, 03 Jul 2026 07:33:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a47c845.af6ba3ed.2c2f30.fcc1@mx.google.com>


================
@@ -0,0 +1,19 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+module {
+  // CHECK-LABEL: func.func @m0
+  // CHECK: arith.scaling_truncf
+  func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+    %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+    return %0 : tensor<16xf4E2M1FN>
+  }
+
+  // CHECK-LABEL: func.func @m1
+  // CHECK: arith.constant
+  // CHECK: arith.scaling_truncf
+  func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+    %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+    %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+    return %0 : tensor<4xf4E2M1FN>
+  }
+}
----------------
kuhar wrote:

nit: missing newline

https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Fri Jul  3 07:33:41 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Fri, 03 Jul 2026 07:33:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a47c845.491d90f5.1f0d58.04d6@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Fri Jul  3 07:34:02 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Fri, 03 Jul 2026 07:34:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a47c85a.14d9b9dc.140829.0514@mx.google.com>


https://github.com/javedabsar1 edited https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Fri Jul  3 07:34:04 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Fri, 03 Jul 2026 07:34:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix null deref in
 SpecConstantOperationOp::verifyRegions (PR #207328)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207328@github.com>
Message-ID: <6a47c85c.58bdd12c.21857.f05a@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/207328

From llvmlistbot at llvm.org  Fri Jul  3 07:34:28 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Fri, 03 Jul 2026 07:34:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Guard getSizeInBytes()
 optionals in UnifyAliasedResourcePass rewriters (PR #207325)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207325@github.com>
Message-ID: <6a47c874.01f0a1da.2016e9.1f01@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/207325

From llvmlistbot at llvm.org  Fri Jul  3 07:37:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:37:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2a95108 - [mlir][SPIR-V] Guard
 getSizeInBytes() optionals in UnifyAliasedResourcePass rewriters (#207325)
Message-ID: <6a47c93e.8e75e0cb.2e0812.0279@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-03T16:37:45+02:00
New Revision: 2a951083b1fd3e2d221613f3f94f68b05111fefb

URL: https://github.com/llvm/llvm-project/commit/2a951083b1fd3e2d221613f3f94f68b05111fefb
DIFF: https://github.com/llvm/llvm-project/commit/2a951083b1fd3e2d221613f3f94f68b05111fefb.diff

LOG: [mlir][SPIR-V] Guard getSizeInBytes() optionals in UnifyAliasedResourcePass rewriters (#207325)

Added: 
    

Modified: 
    mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
    mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
index 692f2e7616e5a..ecfce68db036b 100644
--- a/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
+++ b/mlir/lib/Dialect/SPIRV/Transforms/UnifyAliasedResourcePass.cpp
@@ -371,8 +371,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // them into a buffer with vector element types. We need to scale the last
       // index for the vector as a whole, then add one level of index for inside
       // the vector.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(dstNumBytes >= srcNumBytes && dstNumBytes % srcNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -398,8 +402,12 @@ struct ConvertAccessChain : public ConvertAliasResource<spirv::AccessChainOp> {
       // The source indices are for a buffer with larger bitwidth scalar/vector
       // element types. Rewrite them into a buffer with smaller bitwidth element
       // types. We only need to scale the last index.
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(acOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes >= dstNumBytes && srcNumBytes % dstNumBytes == 0);
 
       auto indices = llvm::to_vector<4>(acOp.getIndices());
@@ -455,8 +463,12 @@ struct ConvertLoad : public ConvertAliasResource<spirv::LoadOp> {
       // vector types of 
diff erent component counts. For such cases, we load
       // multiple smaller bitwidth values and construct a larger bitwidth one.
 
-      int srcNumBytes = *srcElemType.getSizeInBytes();
-      int dstNumBytes = *dstElemType.getSizeInBytes();
+      std::optional<int64_t> srcBytes = srcElemType.getSizeInBytes();
+      std::optional<int64_t> dstBytes = dstElemType.getSizeInBytes();
+      if (!srcBytes || !dstBytes)
+        return rewriter.notifyMatchFailure(loadOp, "unknown element byte size");
+      int srcNumBytes = *srcBytes;
+      int dstNumBytes = *dstBytes;
       assert(srcNumBytes > dstNumBytes && srcNumBytes % dstNumBytes == 0);
       int ratio = srcNumBytes / dstNumBytes;
       if (ratio > 4)

diff  --git a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
index f5cd490c164d7..d7cc1d0cc1f84 100644
--- a/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
+++ b/mlir/test/Dialect/SPIRV/Transforms/unify-aliased-resource.mlir
@@ -549,6 +549,24 @@ spirv.module Logical GLSL450 {
 
 // -----
 
+// Make sure we do not crash on element types without a defined byte size (i1).
+
+spirv.module Logical GLSL450 {
+  spirv.GlobalVariable @var01_i1 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i1, stride=1> [0])>, StorageBuffer>
+  spirv.GlobalVariable @var01_i32 bind(0, 1) {aliased} : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+
+  spirv.func @unknown_byte_size(%index: i32) -> i32 "None" {
+    %c0 = spirv.Constant 0 : i32
+    %addr = spirv.mlir.addressof @var01_i32 : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>
+    // expected-error at +1 {{failed to legalize operation 'spirv.AccessChain'}}
+    %ac = spirv.AccessChain %addr[%c0, %index] : !spirv.ptr<!spirv.struct<(!spirv.rtarray<i32, stride=4> [0])>, StorageBuffer>, i32, i32 -> !spirv.ptr<i32, StorageBuffer>
+    %value = spirv.Load "StorageBuffer" %ac : i32
+    spirv.ReturnValue %value : i32
+  }
+}
+
+// -----
+
 // Make sure we do not crash on function arguments.
 
 spirv.module Logical GLSL450 {


        

From llvmlistbot at llvm.org  Fri Jul  3 07:37:52 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 03 Jul 2026 07:37:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Guard getSizeInBytes()
 optionals in UnifyAliasedResourcePass rewriters (PR #207325)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207325@github.com>
Message-ID: <6a47c940.01f0a1da.2016e9.9498@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/207325

From llvmlistbot at llvm.org  Fri Jul  3 07:37:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:37:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] fa0822a - [mlir][SPIR-V] Fix null deref in
 SpecConstantOperationOp::verifyRegions (#207328)
Message-ID: <6a47c945.4a342cbc.134cb.3091@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-03T16:37:53+02:00
New Revision: fa0822a9dd7bc50ad61a0b19455ca1d26aafe3d5

URL: https://github.com/llvm/llvm-project/commit/fa0822a9dd7bc50ad61a0b19455ca1d26aafe3d5
DIFF: https://github.com/llvm/llvm-project/commit/fa0822a9dd7bc50ad61a0b19455ca1d26aafe3d5.diff

LOG: [mlir][SPIR-V] Fix null deref in SpecConstantOperationOp::verifyRegions (#207328)

Added: 
    

Modified: 
    mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
    mlir/test/Dialect/SPIRV/IR/structure-ops.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index c0393888af5c3..99bc3913c56c6 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -2083,8 +2083,9 @@ LogicalResult spirv::SpecConstantOperationOp::verifyRegions() {
     return emitOpError("invalid enclosed op");
 
   for (auto operand : enclosedOp.getOperands())
-    if (!isa<spirv::ConstantOp, spirv::ReferenceOfOp,
-             spirv::SpecConstantOperationOp>(operand.getDefiningOp()))
+    if (!isa_and_present<spirv::ConstantOp, spirv::ReferenceOfOp,
+                         spirv::SpecConstantOperationOp>(
+            operand.getDefiningOp()))
       return emitOpError(
           "invalid operand, must be defined by a constant operation");
 

diff  --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index c0b73b926c764..37a9e97106de9 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -1220,3 +1220,14 @@ spirv.module Logical GLSL450 {
     spirv.Return
   }
 }
+
+// -----
+
+spirv.module Logical GLSL450 {
+  spirv.func @foo(%arg0: i32) -> () "None" {
+    // expected-error @+1 {{invalid operand, must be defined by a constant operation}}
+    %0 = spirv.SpecConstantOperation wraps "spirv.IAdd"(%arg0, %arg0) : (i32, i32) -> i32
+
+    spirv.Return
+  }
+}


        

From llvmlistbot at llvm.org  Fri Jul  3 07:37:59 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 03 Jul 2026 07:37:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix null deref in
 SpecConstantOperationOp::verifyRegions (PR #207328)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207328@github.com>
Message-ID: <6a47c947.e9d484b9.11ad58.6e18@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/207328

From llvmlistbot at llvm.org  Fri Jul  3 07:42:36 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 07:42:36 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47ca5c.27b3b003.1610ae.38fb@mx.google.com>


nikic wrote:

Hm, I saw this error in the pre-merge CI and it went away on a re-run, so I assumed it was unrelated. I just tried to reproduce this using @zmodem's cmake configuration, but that built cleanly.

What is the failing command? If the failure is flaky, maybe running it multiple times will reproduce...

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 07:43:15 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 03 Jul 2026 07:43:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a47ca83.01f0a1da.2016e9.67a5@mx.google.com>


https://github.com/Devjiu updated https://github.com/llvm/llvm-project/pull/207241

>From 65e00b4915e4894a77b7fd083c1d6aa06a964d2d Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Thu, 2 Jul 2026 16:32:59 +0000
Subject: [PATCH 1/2] [mlir][tensor] Preserve encoding in more canonicalizers
 (concat, reshape, pad)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This commit fixes encoding drop in similar with InsertSliceOpConstantArgumentFolder cases.
The same anti-pattern — silently producing a `RankedTensorType` without an
encoding while a source value with an encoding flows through the
refined type — is present in several other canonicalizers/inferResultType
implementations:

* ConcatOp::inferResultType built the result type with no encoding. When all
  inputs share an encoding, propagate it; when they differ, drop it (concat
  semantics are undefined across differing encodings, so we do not pick one).

* InferConcatOperandTypes built per-operand refined types with the result
  element type but no encoding. The cast targets an operand value, so it
  must carry that operand's own encoding.

* CollapseShapeOp::inferCollapsedType dropped the encoding. Since collapse
  changes the tensor rank, `VerifiableTensorEncoding` attrs (e.g. sparse
  tensor encoding) would become invalid on the new shape; those are dropped
  as before. Encodings that do not implement `VerifiableTensorEncoding`
  (opaque dict/string/bounds attrs used by downstream dialects) are
  rank-agnostic and are now preserved.

* ConvertToStaticExpandShape built the inserted `tensor.cast` target and
  the refined `expand_shape` result with no encoding. Both types preserve
  the rank of the value flowing through them, so they now carry the source's
  and result's encodings respectively.

* PadOp::inferResultType dropped the encoding. Pad preserves rank, so the
  source encoding stays valid. FoldSourceTensorCast and FoldStaticPadding
  inherit the fix through inferResultType; FoldStaticPadding additionally
  rebuilt the type directly and is updated to propagate the pad's result
  encoding.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 47 ++++++++---
 mlir/test/Dialect/Tensor/canonicalize.mlir | 93 +++++++++++++++++++++-
 2 files changed, 127 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..2c94a10a8e870 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -24,6 +24,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TensorEncoding.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/DestinationStyleOpInterface.h"
 #include "mlir/Interfaces/InferIntRangeInterface.h"
@@ -577,7 +578,18 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  return RankedTensorType::get(sizes, tensorTypes[0].getElementType());
+  // Preserve the encoding when all inputs share it; otherwise drop it (the
+  // semantics of concatenating tensors with differing encodings are undefined
+  // at this level, so we don't try to pick one).
+  Attribute encoding = tensorTypes[0].getEncoding();
+  for (auto tensorType : llvm::drop_begin(tensorTypes)) {
+    if (tensorType.getEncoding() != encoding) {
+      encoding = Attribute();
+      break;
+    }
+  }
+  return RankedTensorType::get(sizes, tensorTypes[0].getElementType(),
+                               encoding);
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -811,11 +823,14 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
     SmallVector<int64_t> inferredOperandShape(inferredResultType.getShape());
     for (auto [operandIdx, operandType] :
          llvm::enumerate(concatOp->getOperandTypes())) {
-      // Compute inferred type for operand.
-      inferredOperandShape[dim] =
-          cast<RankedTensorType>(operandType).getDimSize(dim);
+      // Compute inferred type for operand. The refined type is applied to the
+      // operand itself, so it must carry the operand's own encoding rather
+      // than the (potentially different or missing) result encoding.
+      auto operandRankedType = cast<RankedTensorType>(operandType);
+      inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, inferredResultType.getElementType());
+          inferredOperandShape, inferredResultType.getElementType(),
+          operandRankedType.getEncoding());
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2023,7 +2038,10 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  return RankedTensorType::get(newShape, type.getElementType());
+  Attribute encoding = type.getEncoding();
+  if (llvm::isa_and_present<VerifiableTensorEncoding>(encoding))
+    encoding = {};
+  return RankedTensorType::get(newShape, type.getElementType(), encoding);
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
@@ -2273,10 +2291,13 @@ struct ConvertToStaticExpandShape : public OpRewritePattern<ExpandShapeOp> {
 
     SmallVector<OpFoldResult> outputOfr =
         getMixedValues(newOutputShape, dynamicOutputShape, rewriter);
+    // The refined types keep the ranks of the src / result respectively
     auto inputType = RankedTensorType::get(
-        newInputShape, expandOp.getSrcType().getElementType());
+        newInputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getSrcType().getEncoding());
     auto outputType = RankedTensorType::get(
-        newOutputShape, expandOp.getSrcType().getElementType());
+        newOutputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getResultType().getEncoding());
     auto inputCast = CastOp::create(rewriter, expandOp.getLoc(), inputType,
                                     expandOp.getSrc());
     auto newExpand = ExpandShapeOp::create(
@@ -2385,7 +2406,8 @@ RankedTensorType ExtractSliceOp::inferCanonicalRankReducedResultType(
       if (!dimsToProject.test(pos))
         projectedShape.push_back(shape[pos]);
     inferredType =
-        RankedTensorType::get(projectedShape, inferredType.getElementType());
+        RankedTensorType::get(projectedShape, inferredType.getElementType(),
+                              inferredType.getEncoding());
   }
   return inferredType;
 }
@@ -3323,7 +3345,8 @@ RankedTensorType PadOp::inferResultType(RankedTensorType sourceType,
     }
   }
 
-  return RankedTensorType::get(inferredShape, sourceType.getElementType());
+  return RankedTensorType::get(inferredShape, sourceType.getElementType(),
+                               sourceType.getEncoding());
 }
 
 void PadOp::build(OpBuilder &b, OperationState &result, Type resultType,
@@ -3730,9 +3753,9 @@ struct FoldStaticPadding : public OpRewritePattern<PadOp> {
                      [&](int64_t x) { return x == ShapedType::kDynamic; }))
       return failure();
 
-    // Rewrite the op using the new static type.
     auto newResultType = RankedTensorType::get(
-        newOutDims, padTensorOp.getType().getElementType());
+        newOutDims, padTensorOp.getType().getElementType(),
+        padTensorOp.getType().getEncoding());
     auto newOp = PadOp::create(
         rewriter, padTensorOp->getLoc(), newResultType, input, staticLow,
         staticHigh, newLows, newHighs, padTensorOp.getNofold(),
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..c5fc71e6c7bd5 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -162,6 +162,22 @@ func.func @infer_concat_return_type(%arg0: tensor<5x12xi32>, %arg1: tensor<?x12x
 
 // -----
 
+// ConcatOp::inferResultType must carry the (uniformly-shared) operand encoding
+// onto both the refined operand cast and the refined ConcatOp result.
+// CHECK-LABEL: concat_preserves_uniform_encoding
+//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?xi32, "abc">
+//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?xi32, "abc">
+//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?xi32, "abc"> to tensor<3x?xi32, "abc">
+//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]] : (tensor<3x?xi32, "abc">, tensor<3x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+func.func @concat_preserves_uniform_encoding(
+    %a: tensor<3x?xi32, "abc">, %b: tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc"> {
+  %r = tensor.concat dim(1) %a, %b
+      : (tensor<3x?xi32, "abc">, tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+  return %r : tensor<3x?xi32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1092,6 +1108,22 @@ func.func @collapse_of_cast(%t: tensor<8x12x32xf32>) -> tensor<?x32xf32> {
 
 // -----
 
+// A user-defined (non-VerifiableTensorEncoding) encoding must be preserved
+// through the collapse_of_cast folder; inferCollapsedType propagates it
+// alongside the refined shape.
+// CHECK-LABEL: func.func @collapse_of_cast_preserves_encoding(
+//  CHECK-SAME:     %[[IN:.*]]: tensor<8x12x32xf32, "abc">
+//       CHECK:   %[[COLLAPSE:.*]] = tensor.collapse_shape %[[IN]] {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, "abc"> into tensor<96x32xf32, "abc">
+//       CHECK:   tensor.cast %[[COLLAPSE]] : tensor<96x32xf32, "abc"> to tensor<?x32xf32>
+func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -> tensor<?x32xf32> {
+  %0 = tensor.cast %t : tensor<8x12x32xf32, "abc"> to tensor<?x?x?xf32, "abc">
+  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, "abc"> into tensor<?x?xf32, "abc">
+  %2 = tensor.cast %1 : tensor<?x?xf32, "abc"> to tensor<?x32xf32>
+  return %2 : tensor<?x32xf32>
+}
+
+// -----
+
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>
@@ -1773,6 +1805,28 @@ func.func @pad_nofold_same_static_shape(%arg0: tensor<5x6xf32>, %a: index)
 
 // -----
 
+// FoldSourceTensorCast (via PadOp::inferResultType) must preserve the source
+// encoding onto the refined pad result and the inserted result cast.
+// CHECK-LABEL:   func @pad_after_cast_preserves_encoding(
+//  CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[PADDED:.*]] = tensor.pad %[[INPUT]]
+//       CHECK:       : tensor<?x64x?x?xf32, "abc"> to tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[CAST:.*]] = tensor.cast %[[PADDED]] : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+//       CHECK:     return %[[CAST]]
+func.func @pad_after_cast_preserves_encoding(
+    %arg0: tensor<?x64x?x?xf32, "abc">) -> tensor<?x?x?x?xf32, "abc"> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %dynamic = tensor.cast %arg0
+      : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  %padded = tensor.pad %dynamic low[0, 0, 1, 1] high[0, 0, 1, 1] {
+    ^bb0(%a: index, %b: index, %c: index, %d: index):
+      tensor.yield %cst: f32
+  } : tensor<?x?x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  return %padded: tensor<?x?x?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL:   func @pad_after_cast_different_shape(
 // CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32>) -> tensor<?x?x?x?xf32> {
 // CHECK:           %[[CST:.*]] = arith.constant 0.000000e+00 : f32
@@ -1921,6 +1975,26 @@ func.func @pad_static_zero_cast(%arg0: tensor<?x?x?xf32>, %pad_value: f32) -> te
 
 // -----
 
+// FoldStaticPadding must preserve the pad's result encoding on the refined
+// (more-static) pad and the inserted result cast.
+// CHECK-LABEL: func @fold_static_padding_preserves_encoding(
+//  CHECK-SAME:     %[[SRC:.*]]: tensor<8x?xf32, "abc">
+//       CHECK:   %[[PADDED:.*]] = tensor.pad %[[SRC]] low[1, 2] high[1, 2]
+//       CHECK:     : tensor<8x?xf32, "abc"> to tensor<10x?xf32, "abc">
+//       CHECK:   tensor.cast %[[PADDED]] : tensor<10x?xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @fold_static_padding_preserves_encoding(
+    %arg0: tensor<8x?xf32, "abc">, %pv: f32) -> tensor<?x?xf32, "abc"> {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %r = tensor.pad %arg0 low[%c1, %c2] high[%c1, %c2] {
+    ^bb0(%a: index, %b: index):
+      tensor.yield %pv: f32
+  } : tensor<8x?xf32, "abc"> to tensor<?x?xf32, "abc">
+  return %r : tensor<?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @pad_nofold_static_zero(
 //  CHECK-SAME:                  %[[ARG0:.*]]: tensor<?x?x?xf32>
 //       CHECK:   %[[PAD:.*]] = tensor.pad
@@ -2641,8 +2715,25 @@ func.func @partial_sink_expand_of_cast(%arg0 : tensor<10x10xf32>, %arg1 : index,
 // CHECK-LABEL:  func.func @partial_sink_expand_of_cast
 //       CHECK:   %[[CAST:.+]] = tensor.cast
 //  CHECK-SAME:     tensor<10x10xf32> to tensor<?x10xf32>
-//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]] 
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]]
 //  CHECK-SAME:     output_shape [%{{.*}}, %{{.*}}, 10]
 //       CHECK:   %[[RES:.+]] = tensor.cast %[[EXPAND]]
 //  CHECK-SAME:     tensor<?x?x10xf32> to tensor<?x?x?xf32>
 //       CHECK:   return %[[RES]]
+
+// -----
+
+// ConvertToStaticExpandShape must carry the source's encoding onto the refined
+// cast-target and the (source-side) encoding of the new ExpandShapeOp result.
+// CHECK-LABEL:  func.func @sink_expand_of_cast_preserves_encoding
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape
+//  CHECK-SAME:     tensor<64xf32, "abc"> into tensor<8x8xf32, "abc">
+//       CHECK:   tensor.cast %[[EXPAND]] : tensor<8x8xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @sink_expand_of_cast_preserves_encoding(%t: tensor<64xf32, "abc">) -> tensor<?x?xf32, "abc"> {
+  %c = tensor.cast %t : tensor<64xf32, "abc"> to tensor<?xf32, "abc">
+  %c8 = arith.constant 8 : index
+  %c8b = arith.constant 8 : index
+  %e = tensor.expand_shape %c [[0, 1]] output_shape [%c8, %c8b]
+      : tensor<?xf32, "abc"> into tensor<?x?xf32, "abc">
+  return %e : tensor<?x?xf32, "abc">
+}

>From 8e19f97a147696c5cd2a6f30a64f93f584359a23 Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Fri, 3 Jul 2026 14:40:16 +0000
Subject: [PATCH 2/2] Add VerifiableTensorEncoding check

This commit checks tensor encoding for `VerifiableTensorEncoding::verifyEncoding`
if possible before assigning it source type, updated by canonicalizer.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 73 +++++++++++++-----
 mlir/test/Dialect/Tensor/canonicalize.mlir | 90 ++++++++++++++++++++++
 2 files changed, 142 insertions(+), 21 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 2c94a10a8e870..903eb5f6c411d 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -45,6 +45,25 @@
 using namespace mlir;
 using namespace mlir::tensor;
 
+/// The intent is to keep canonicalizers from silently erasing downstream
+/// metadata: a `RankedTensorType` encoding is a per-value property, and the
+/// canonicalizer is not a place to guess whether it still applies — the
+/// encoding's own contract is.
+static Attribute propagateEncoding(Attribute encoding, ArrayRef<int64_t> shape,
+                                   Type elementType) {
+  auto verifiable = dyn_cast_or_null<VerifiableTensorEncoding>(encoding);
+  if (!verifiable)
+    return encoding;
+
+  MLIRContext *ctx = encoding.getContext();
+  // to avoid user's error stream
+  ScopedDiagnosticHandler swallow(ctx, [](Diagnostic &) { return success(); });
+  auto emit = [ctx]() { return mlir::emitError(UnknownLoc::get(ctx)); };
+  return succeeded(verifiable.verifyEncoding(shape, elementType, emit))
+             ? encoding
+             : Attribute{};
+}
+
 /// Materialize a single constant operation from a given attribute value with
 /// the desired resultant type.
 Operation *TensorDialect::materializeConstant(OpBuilder &builder,
@@ -578,9 +597,9 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  // Preserve the encoding when all inputs share it; otherwise drop it (the
-  // semantics of concatenating tensors with differing encodings are undefined
-  // at this level, so we don't try to pick one).
+  // Only propagate an encoding when all inputs agree on it (concat semantics
+  // across differing encodings are undefined at this level). Let the encoding
+  // itself decide whether it still holds on the new shape.
   Attribute encoding = tensorTypes[0].getEncoding();
   for (auto tensorType : llvm::drop_begin(tensorTypes)) {
     if (tensorType.getEncoding() != encoding) {
@@ -588,8 +607,9 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
       break;
     }
   }
-  return RankedTensorType::get(sizes, tensorTypes[0].getElementType(),
-                               encoding);
+  Type elementType = tensorTypes[0].getElementType();
+  return RankedTensorType::get(sizes, elementType,
+                               propagateEncoding(encoding, sizes, elementType));
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -825,12 +845,15 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
          llvm::enumerate(concatOp->getOperandTypes())) {
       // Compute inferred type for operand. The refined type is applied to the
       // operand itself, so it must carry the operand's own encoding rather
-      // than the (potentially different or missing) result encoding.
+      // than the (potentially different or missing) result encoding, subject
+      // to the encoding still holding on the refined shape.
       auto operandRankedType = cast<RankedTensorType>(operandType);
       inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
+      Type elementType = inferredResultType.getElementType();
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, inferredResultType.getElementType(),
-          operandRankedType.getEncoding());
+          inferredOperandShape, elementType,
+          propagateEncoding(operandRankedType.getEncoding(),
+                            inferredOperandShape, elementType));
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2038,10 +2061,9 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  Attribute encoding = type.getEncoding();
-  if (llvm::isa_and_present<VerifiableTensorEncoding>(encoding))
-    encoding = {};
-  return RankedTensorType::get(newShape, type.getElementType(), encoding);
+  return RankedTensorType::get(
+      newShape, type.getElementType(),
+      propagateEncoding(type.getEncoding(), newShape, type.getElementType()));
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
@@ -2291,13 +2313,18 @@ struct ConvertToStaticExpandShape : public OpRewritePattern<ExpandShapeOp> {
 
     SmallVector<OpFoldResult> outputOfr =
         getMixedValues(newOutputShape, dynamicOutputShape, rewriter);
-    // The refined types keep the ranks of the src / result respectively
+    // The refined types are still applied to the same src/result values, so
+    // propagate their encodings, letting each encoding self-decide whether it
+    // still holds on the more-static shape.
+    Type elementType = expandOp.getSrcType().getElementType();
     auto inputType = RankedTensorType::get(
-        newInputShape, expandOp.getSrcType().getElementType(),
-        expandOp.getSrcType().getEncoding());
+        newInputShape, elementType,
+        propagateEncoding(expandOp.getSrcType().getEncoding(), newInputShape,
+                          elementType));
     auto outputType = RankedTensorType::get(
-        newOutputShape, expandOp.getSrcType().getElementType(),
-        expandOp.getResultType().getEncoding());
+        newOutputShape, elementType,
+        propagateEncoding(expandOp.getResultType().getEncoding(),
+                          newOutputShape, elementType));
     auto inputCast = CastOp::create(rewriter, expandOp.getLoc(), inputType,
                                     expandOp.getSrc());
     auto newExpand = ExpandShapeOp::create(
@@ -3345,8 +3372,10 @@ RankedTensorType PadOp::inferResultType(RankedTensorType sourceType,
     }
   }
 
-  return RankedTensorType::get(inferredShape, sourceType.getElementType(),
-                               sourceType.getEncoding());
+  Type elementType = sourceType.getElementType();
+  return RankedTensorType::get(
+      inferredShape, elementType,
+      propagateEncoding(sourceType.getEncoding(), inferredShape, elementType));
 }
 
 void PadOp::build(OpBuilder &b, OperationState &result, Type resultType,
@@ -3753,9 +3782,11 @@ struct FoldStaticPadding : public OpRewritePattern<PadOp> {
                      [&](int64_t x) { return x == ShapedType::kDynamic; }))
       return failure();
 
+    Type elementType = padTensorOp.getType().getElementType();
     auto newResultType = RankedTensorType::get(
-        newOutDims, padTensorOp.getType().getElementType(),
-        padTensorOp.getType().getEncoding());
+        newOutDims, elementType,
+        propagateEncoding(padTensorOp.getType().getEncoding(), newOutDims,
+                          elementType));
     auto newOp = PadOp::create(
         rewriter, padTensorOp->getLoc(), newResultType, input, staticLow,
         staticHigh, newLows, newHighs, padTensorOp.getNofold(),
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index c5fc71e6c7bd5..a92c94f4cc652 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -178,6 +178,30 @@ func.func @concat_preserves_uniform_encoding(
 
 // -----
 
+// `VerifiableTensorEncoding` case: concat preserves rank, so the sparse
+// encoding's rank invariant still holds on the refined shape and it is kept.
+// The operand refinement inserts a cast whose target must carry the same
+// sparse encoding (not `null`, which would need re-encoding back later).
+#sparse_dense4 = #sparse_tensor.encoding<{
+    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
+}>
+// CHECK-LABEL: concat_preserves_sparse_encoding
+//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}> to tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]]
+//  CHECK-SAME:     -> tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+func.func @concat_preserves_sparse_encoding(
+    %a: tensor<3x?x8x8xf32, #sparse_dense4>,
+    %b: tensor<?x?x8x8xf32, #sparse_dense4>) -> tensor<3x?x8x8xf32, #sparse_dense4> {
+  %r = tensor.concat dim(1) %a, %b
+      : (tensor<3x?x8x8xf32, #sparse_dense4>, tensor<?x?x8x8xf32, #sparse_dense4>)
+     -> tensor<3x?x8x8xf32, #sparse_dense4>
+  return %r : tensor<3x?x8x8xf32, #sparse_dense4>
+}
+
+// -----
+
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1124,6 +1148,25 @@ func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -
 
 // -----
 
+// `VerifiableTensorEncoding` case: collapse changes the rank, so the sparse
+// encoding's rank invariant no longer holds; propagateEncoding asks the
+// encoding via `verifyEncoding` and drops it from the refined type.
+#sparse_dense3 = #sparse_tensor.encoding<{
+    map = (d0, d1, d2) -> (d0 : dense, d1 : dense, d2 : dense)
+}>
+// CHECK-LABEL: func.func @collapse_of_cast_drops_rank_invalid_encoding(
+//   CHECK-NOT:   collapse_shape {{.*}}#{{[a-z_0-9]+}}
+//       CHECK:   tensor.collapse_shape %{{.*}} {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, #{{[a-z_0-9]+}}> into tensor<96x32xf32>
+func.func @collapse_of_cast_drops_rank_invalid_encoding(
+    %t: tensor<8x12x32xf32, #sparse_dense3>) -> tensor<?x32xf32> {
+  %0 = tensor.cast %t : tensor<8x12x32xf32, #sparse_dense3> to tensor<?x?x?xf32, #sparse_dense3>
+  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, #sparse_dense3> into tensor<?x?xf32>
+  %2 = tensor.cast %1 : tensor<?x?xf32> to tensor<?x32xf32>
+  return %2 : tensor<?x32xf32>
+}
+
+// -----
+
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>
@@ -1827,6 +1870,30 @@ func.func @pad_after_cast_preserves_encoding(
 
 // -----
 
+// `VerifiableTensorEncoding` case: pad preserves rank, so the sparse encoding
+// stays valid on the refined shape and PadOp::inferResultType keeps it.
+#sparse_dense4b = #sparse_tensor.encoding<{
+    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
+}>
+// CHECK-LABEL:   func @pad_after_cast_preserves_sparse_encoding(
+//  CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}>
+//       CHECK:     %[[PADDED:.*]] = tensor.pad %[[INPUT]]
+//       CHECK:       : tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}> to tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}>
+//       CHECK:     tensor.cast %[[PADDED]] : tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}> to tensor<?x?x?x?xf32, #{{[a-z_0-9]+}}>
+func.func @pad_after_cast_preserves_sparse_encoding(
+    %arg0: tensor<?x64x?x?xf32, #sparse_dense4b>) -> tensor<?x?x?x?xf32, #sparse_dense4b> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %dynamic = tensor.cast %arg0
+      : tensor<?x64x?x?xf32, #sparse_dense4b> to tensor<?x?x?x?xf32, #sparse_dense4b>
+  %padded = tensor.pad %dynamic low[0, 0, 1, 1] high[0, 0, 1, 1] {
+    ^bb0(%a: index, %b: index, %c: index, %d: index):
+      tensor.yield %cst: f32
+  } : tensor<?x?x?x?xf32, #sparse_dense4b> to tensor<?x?x?x?xf32, #sparse_dense4b>
+  return %padded: tensor<?x?x?x?xf32, #sparse_dense4b>
+}
+
+// -----
+
 // CHECK-LABEL:   func @pad_after_cast_different_shape(
 // CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32>) -> tensor<?x?x?x?xf32> {
 // CHECK:           %[[CST:.*]] = arith.constant 0.000000e+00 : f32
@@ -2737,3 +2804,26 @@ func.func @sink_expand_of_cast_preserves_encoding(%t: tensor<64xf32, "abc">) ->
       : tensor<?xf32, "abc"> into tensor<?x?xf32, "abc">
   return %e : tensor<?x?xf32, "abc">
 }
+
+// -----
+
+// `VerifiableTensorEncoding` case: expand_shape refined shapes keep the src's
+// and result's ranks respectively, so both sparse encodings pass their
+// (rank-based) `verifyEncoding` and are preserved.
+#sparse_v = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+#sparse_m = #sparse_tensor.encoding<{
+    map = (d0, d1) -> (d0 : compressed, d1 : compressed)
+}>
+// CHECK-LABEL:  func.func @sink_expand_of_cast_preserves_sparse_encoding
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape
+//  CHECK-SAME:     tensor<64xf32, #{{[a-z_0-9]+}}> into tensor<8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   tensor.cast %[[EXPAND]] : tensor<8x8xf32, #{{[a-z_0-9]+}}> to tensor<?x?xf32, #{{[a-z_0-9]+}}>
+func.func @sink_expand_of_cast_preserves_sparse_encoding(
+    %t: tensor<64xf32, #sparse_v>) -> tensor<?x?xf32, #sparse_m> {
+  %c = tensor.cast %t : tensor<64xf32, #sparse_v> to tensor<?xf32, #sparse_v>
+  %c8a = arith.constant 8 : index
+  %c8b = arith.constant 8 : index
+  %e = tensor.expand_shape %c [[0, 1]] output_shape [%c8a, %c8b]
+      : tensor<?xf32, #sparse_v> into tensor<?x?xf32, #sparse_m>
+  return %e : tensor<?x?xf32, #sparse_m>
+}


From llvmlistbot at llvm.org  Fri Jul  3 07:43:17 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 07:43:17 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47ca85.a151105c.30c1b9.81db@mx.google.com>


================
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
----------------
durga4github wrote:

Nice ;-)

https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 07:43:21 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 03 Jul 2026 07:43:21 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47ca89.4a342cbc.134cb.e595@mx.google.com>


https://github.com/durga4github approved this pull request.


https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 07:44:04 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Fri, 03 Jul 2026 07:44:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a47cab4.eb165e5c.113c2a.5103@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/205967

From llvmlistbot at llvm.org  Fri Jul  3 07:45:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 07:45:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] e788a46 - [mlir][SPIR-V] Add SPIRVToLLVM
 lowering for GL Radians and Degrees ops (#205967)
Message-ID: <6a47cafd.2020f6c2.1fdf61.530b@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-03T16:45:12+02:00
New Revision: e788a46f7311e49d8275116c7a634a04a43a9cef

URL: https://github.com/llvm/llvm-project/commit/e788a46f7311e49d8275116c7a634a04a43a9cef
DIFF: https://github.com/llvm/llvm-project/commit/e788a46f7311e49d8275116c7a634a04a43a9cef.diff

LOG: [mlir][SPIR-V] Add SPIRVToLLVM lowering for GL Radians and Degrees ops (#205967)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 2682e1d9e26ce..1cd3d79c13e20 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1584,6 +1584,35 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+// Converts spirv.GL.Radians (scale = pi/180) and spirv.GL.Degrees
+// (scale = 180/pi) by multiplying the operand by a compile-time constant.
+template <typename SPIRVOp>
+class ScalePattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  template <typename... Args>
+  ScalePattern(double scale, Args &&...args)
+      : SPIRVToLLVMConversion<SPIRVOp>(std::forward<Args>(args)...),
+        scale(scale) {}
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type srcType = op.getType();
+    Type dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value factor = createFPConstant(loc, srcType, dstType, rewriter, scale);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getOperand(),
+                                              factor);
+    return success();
+  }
+
+private:
+  double scale;
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -2046,6 +2075,12 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
 
   patterns.add<GlobalVariablePattern>(clientAPI, patterns.getContext(),
                                       typeConverter);
+  // pi / 180
+  patterns.add<ScalePattern<spirv::GLRadiansOp>>(
+      0.017453292519943295, patterns.getContext(), typeConverter);
+  // 180 / pi
+  patterns.add<ScalePattern<spirv::GLDegreesOp>>(
+      57.29577951308232, patterns.getContext(), typeConverter);
 }
 
 void mlir::populateSPIRVToLLVMFunctionConversionPatterns(

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index 042b68bef9dcd..24f16602e4147 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -334,6 +334,36 @@ spirv.func @inverse_sqrt(%arg0: f32) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.Radians
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @radians
+spirv.func @radians(%arg0: f32, %arg1: vector<3xf32>) "None" {
+  // CHECK: %[[FACTOR:.*]] = llvm.mlir.constant(0.0174532924 : f32) : f32
+  // CHECK: llvm.fmul %{{.*}}, %[[FACTOR]] : f32
+  %0 = spirv.GL.Radians %arg0 : f32
+  // CHECK: %[[VFACTOR:.*]] = llvm.mlir.constant(dense<0.0174532924> : vector<3xf32>) : vector<3xf32>
+  // CHECK: llvm.fmul %{{.*}}, %[[VFACTOR]] : vector<3xf32>
+  %1 = spirv.GL.Radians %arg1 : vector<3xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.Degrees
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @degrees
+spirv.func @degrees(%arg0: f32, %arg1: vector<3xf32>) "None" {
+  // CHECK: %[[FACTOR:.*]] = llvm.mlir.constant(57.2957802 : f32) : f32
+  // CHECK: llvm.fmul %{{.*}}, %[[FACTOR]] : f32
+  %0 = spirv.GL.Degrees %arg0 : f32
+  // CHECK: %[[VFACTOR:.*]] = llvm.mlir.constant(dense<57.2957802> : vector<3xf32>) : vector<3xf32>
+  // CHECK: llvm.fmul %{{.*}}, %[[VFACTOR]] : vector<3xf32>
+  %1 = spirv.GL.Degrees %arg1 : vector<3xf32>
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Trunc
 //===----------------------------------------------------------------------===//


        

From llvmlistbot at llvm.org  Fri Jul  3 07:45:19 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 03 Jul 2026 07:45:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM lowering for
 GL Radians and Degrees ops (PR #205967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205967@github.com>
Message-ID: <6a47caff.e9d484b9.11ad58.3c7e@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/205967

From llvmlistbot at llvm.org  Fri Jul  3 07:45:50 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 03 Jul 2026 07:45:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a47cb1e.eea6ab6e.1256b6.b702@mx.google.com>


Devjiu wrote:

Updated PR and added lit-tests with `#sparse_tensor.encoding` that implements `VerifiableTensorEncoding` interface. 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul  3 07:48:34 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 03 Jul 2026 07:48:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206934@github.com>
Message-ID: <6a47cbc2.6c8668f2.346e76.9fbf@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206934

>From 2e1952b741d34c2b0dfc1887493498283a21a93c Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:06:39 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FSign and
 GL.SSign

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 64 +++++++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 54 ++++++++++++++++
 2 files changed, 118 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Fri Jul  3 07:51:10 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 03 Jul 2026 07:51:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a47cc5e.f8b79514.3b441.9f55@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206935

>From 4fa5c4ca4a5a4333743f1fc3afa38979d20a176a Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:54:35 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix and
 CL.mix

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 27 +++++++++++++++++++
 .../SPIRVToLLVM/cl-ops-to-llvm.mlir           | 20 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 20 ++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Fri Jul  3 07:53:52 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 07:53:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47cd00.faf16b37.1e0c59.cab6@mx.google.com>


lhutton1 wrote:

Thanks both - I'll aim to merge this on Monday morning (6th July) incase there are any last minute comments.

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 08:01:09 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 03 Jul 2026 08:01:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve source encoding when
 folding insert_slice canonicalizers (PR #207239)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207239@github.com>
Message-ID: <6a47ceb5.d1496011.3d49cb.62b3@mx.google.com>


https://github.com/Devjiu updated https://github.com/llvm/llvm-project/pull/207239

>From 58b03acba8abd581f08554c4600b03d94031330a Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Thu, 2 Jul 2026 16:28:11 +0000
Subject: [PATCH 1/2] [mlir][tensor] Preserve source encoding when folding
 insert_slice canonicalizers
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

InsertSliceOpConstantArgumentFolder currently re-derives the refined source
type via ExtractSliceOp::inferCanonicalRankReducedResultType, which copies
the encoding of the passed-in "source template" — which in this pattern is
insertSliceOp.getDestType(). For a static, encoding-less destination this
silently drops any encoding the actual source carried. Downstream dialects
that stash mandatory metadata in the encoding attribute (upper bounds,
layout, sparsity descriptors) lose that metadata during --canonicalize.

Preserve the original source's encoding by rebuilding the refined
RankedTensorType with srcType.getEncoding() explicitly. Shape refinement
(the `?` -> static direction the pattern already performs) is unchanged
and still satisfies preservesStaticInformation.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 12 +++++-
 mlir/test/Dialect/Tensor/canonicalize.mlir | 48 ++++++++++++++++++++++
 2 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..8dadc83fdc548 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -3014,10 +3014,18 @@ class InsertSliceOpConstantArgumentFolder final
     if (!sliceResult.isValid)
       return failure();
 
-    // Create the new op in canonical form.
-    auto sourceType = ExtractSliceOp::inferCanonicalRankReducedResultType(
+    // Create the new op in canonical form. The refined shape is inferred from
+    // the destination type, but the encoding is a per-value property of the
+    // source and must be preserved: insert_slice does not convert between
+    // encodings, so the source's encoding is what the produced cast/op must
+    // carry (dropping it would silently discard downstream metadata such as
+    // bounds, layout, or sparsity descriptors).
+    auto sourceTypeBase = ExtractSliceOp::inferCanonicalRankReducedResultType(
         insertSliceOp.getSourceType().getRank(), insertSliceOp.getDestType(),
         mixedSizes);
+    auto sourceType = RankedTensorType::get(
+        sourceTypeBase.getShape(), sourceTypeBase.getElementType(),
+        insertSliceOp.getSourceType().getEncoding());
     Value toInsert = insertSliceOp.getSource();
     if (sourceType != insertSliceOp.getSourceType()) {
       OpBuilder::InsertionGuard g(rewriter);
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..df92d3ccfd82b 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -937,6 +937,54 @@ func.func @insert_slice_cast_no_fold(%arg0 : tensor<1x?xf32>, %arg1 : tensor<?x?
 
 // -----
 
+// Verify that the constant-argument folder for insert_slice preserves the
+// source's encoding on the inserted cast, rather than silently picking up the
+// destination's encoding (which is `none` here) via the shape template used by
+// ExtractSliceOp::inferCanonicalRankReducedResultType.
+// CHECK-LABEL: func @preserve_source_encoding_on_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.insert_slice %[[C]] into %[[DST]]
+func.func @preserve_source_encoding_on_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %ivC = arith.constant 0 : index
+  %ivH = arith.constant 0 : index
+  %r = tensor.insert_slice %src into %dst[0, %ivC, %ivH, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+      : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
+// Same invariant for the parallel_insert_slice variant.
+// CHECK-LABEL: func @preserve_source_encoding_on_parallel_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, "abc">
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//   CHECK-NOT:   tensor.cast %{{.*}} : tensor<{{.*}}, "abc"> to tensor<{{[0-9x?]+}}xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, "abc"> to tensor<1x48x16x32xf16, "abc">
+//       CHECK:   tensor.parallel_insert_slice %[[C]] into
+func.func @preserve_source_encoding_on_parallel_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, "abc">,
+    %dst: tensor<1x1280x32x32xf16>,
+    %num_threads: index) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %r = scf.forall (%tid) in (%num_threads) shared_outs(%o = %dst) -> (tensor<1x1280x32x32xf16>) {
+    scf.forall.in_parallel {
+      tensor.parallel_insert_slice %src into %o[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+          : tensor<1x?x?x32xf16, "abc"> into tensor<1x1280x32x32xf16>
+    }
+  }
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @insert_tensor_cast_on_insert_slice_src(
 // CHECK-SAME:      %[[arg0:.*]]: tensor<?x5x?xf32>, %[[arg1:.*]]: tensor<?x?x?xf32>
 //      CHECK:    %[[cast:.*]] = tensor.cast %[[arg0]] : tensor<?x5x?xf32> to tensor<64x5x64xf32>

>From 4dcd8019b046cc8fc61a9f2e6bbc3a41ac61b102 Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Fri, 3 Jul 2026 14:59:09 +0000
Subject: [PATCH 2/2] Align with `VerifiableTensorEncoding` verification

This commit applies same approach with encoding verification
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 34 ++++++++++++++++++----
 mlir/test/Dialect/Tensor/canonicalize.mlir | 26 +++++++++++++++++
 2 files changed, 55 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 8dadc83fdc548..8e7fa47d0b531 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -24,6 +24,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TensorEncoding.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/DestinationStyleOpInterface.h"
 #include "mlir/Interfaces/InferIntRangeInterface.h"
@@ -44,6 +45,25 @@
 using namespace mlir;
 using namespace mlir::tensor;
 
+/// The intent is to keep canonicalizers from silently erasing downstream
+/// metadata: a `RankedTensorType` encoding is a per-value property, and the
+/// canonicalizer is not a place to guess whether it still applies — the
+/// encoding's own contract is.
+static Attribute propagateEncoding(Attribute encoding, ArrayRef<int64_t> shape,
+                                   Type elementType) {
+  auto verifiable = dyn_cast_or_null<VerifiableTensorEncoding>(encoding);
+  if (!verifiable)
+    return encoding;
+
+  MLIRContext *ctx = encoding.getContext();
+  // to avoid user's error stream
+  ScopedDiagnosticHandler swallow(ctx, [](Diagnostic &) { return success(); });
+  auto emit = [ctx]() { return mlir::emitError(UnknownLoc::get(ctx)); };
+  return succeeded(verifiable.verifyEncoding(shape, elementType, emit))
+             ? encoding
+             : Attribute{};
+}
+
 /// Materialize a single constant operation from a given attribute value with
 /// the desired resultant type.
 Operation *TensorDialect::materializeConstant(OpBuilder &builder,
@@ -3016,16 +3036,20 @@ class InsertSliceOpConstantArgumentFolder final
 
     // Create the new op in canonical form. The refined shape is inferred from
     // the destination type, but the encoding is a per-value property of the
-    // source and must be preserved: insert_slice does not convert between
-    // encodings, so the source's encoding is what the produced cast/op must
-    // carry (dropping it would silently discard downstream metadata such as
-    // bounds, layout, or sparsity descriptors).
+    // source: insert_slice does not convert between encodings, so the
+    // produced cast/op must carry the source's encoding (dropping it would
+    // silently discard downstream metadata such as bounds, layout, or
+    // sparsity descriptors). If the source's encoding no longer holds on the
+    // refined shape (e.g. a `VerifiableTensorEncoding` that self-invalidates),
+    // it is dropped in accordance with the encoding's own contract.
     auto sourceTypeBase = ExtractSliceOp::inferCanonicalRankReducedResultType(
         insertSliceOp.getSourceType().getRank(), insertSliceOp.getDestType(),
         mixedSizes);
     auto sourceType = RankedTensorType::get(
         sourceTypeBase.getShape(), sourceTypeBase.getElementType(),
-        insertSliceOp.getSourceType().getEncoding());
+        propagateEncoding(insertSliceOp.getSourceType().getEncoding(),
+                          sourceTypeBase.getShape(),
+                          sourceTypeBase.getElementType()));
     Value toInsert = insertSliceOp.getSource();
     if (sourceType != insertSliceOp.getSourceType()) {
       OpBuilder::InsertionGuard g(rewriter);
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index df92d3ccfd82b..cac1b16cd87f7 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -985,6 +985,32 @@ func.func @preserve_source_encoding_on_parallel_insert_slice_folding(
 
 // -----
 
+// `VerifiableTensorEncoding` case: insert_slice only refines the source shape
+// (dyn -> static), the source rank is unchanged. The sparse encoding's rank
+// invariant still holds on the refined shape, so `propagateEncoding` accepts
+// it and the inserted cast carries the sparse encoding through the folded op.
+#sparse_dense4_is = #sparse_tensor.encoding<{
+    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
+}>
+// CHECK-LABEL: func @preserve_source_sparse_encoding_on_insert_slice_folding
+//  CHECK-SAME:     %[[SRC:[a-zA-Z0-9_]+]]: tensor<1x?x?x32xf16, #{{[a-z_0-9]+}}>
+//  CHECK-SAME:     %[[DST:[a-zA-Z0-9_]+]]: tensor<1x1280x32x32xf16>
+//       CHECK:   %[[C:.+]] = tensor.cast %[[SRC]] : tensor<1x?x?x32xf16, #{{[a-z_0-9]+}}> to tensor<1x48x16x32xf16, #{{[a-z_0-9]+}}>
+//       CHECK:   tensor.insert_slice %[[C]] into %[[DST]]
+func.func @preserve_source_sparse_encoding_on_insert_slice_folding(
+    %src: tensor<1x?x?x32xf16, #sparse_dense4_is>,
+    %dst: tensor<1x1280x32x32xf16>) -> tensor<1x1280x32x32xf16> {
+  %c16 = arith.constant 16 : index
+  %sz1 = arith.constant 48 : index
+  %ivC = arith.constant 0 : index
+  %ivH = arith.constant 0 : index
+  %r = tensor.insert_slice %src into %dst[0, %ivC, %ivH, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
+      : tensor<1x?x?x32xf16, #sparse_dense4_is> into tensor<1x1280x32x32xf16>
+  return %r : tensor<1x1280x32x32xf16>
+}
+
+// -----
+
 // CHECK-LABEL: func @insert_tensor_cast_on_insert_slice_src(
 // CHECK-SAME:      %[[arg0:.*]]: tensor<?x5x?xf32>, %[[arg1:.*]]: tensor<?x?x?xf32>
 //      CHECK:    %[[cast:.*]] = tensor.cast %[[arg0]] : tensor<?x5x?xf32> to tensor<64x5x64xf32>


From llvmlistbot at llvm.org  Fri Jul  3 08:02:06 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Fri, 03 Jul 2026 08:02:06 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a47ceee.d1496011.3d49cb.79c6@mx.google.com>


nikic wrote:

Ah, the failing command is still in https://github.com/llvm/llvm-project/pull/204548#issuecomment-4741459812.

I can reproduce the error when running the command multiple times.

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 08:03:44 2026
From: llvmlistbot at llvm.org (Hans Wennborg)
Date: Fri, 03 Jul 2026 08:03:44 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Revert "[IR] Explicitly
 specify target feature for module asm" (PR #207399)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207399@github.com>
Message-ID: <6a47cf50.ac469dc7.e3aa6.4c1b@mx.google.com>


https://github.com/zmodem closed https://github.com/llvm/llvm-project/pull/207399

From llvmlistbot at llvm.org  Fri Jul  3 08:03:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 08:03:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 433eef6 - Revert "[IR] Explicitly specify
 target feature for module asm" (#207399)
Message-ID: <6a47cf53.58bdd12c.21857.710c@mx.google.com>


Author: Hans Wennborg
Date: 2026-07-03T15:03:37Z
New Revision: 433eef6dea23c18d52c51218bb9d7f9bfbf47ecf

URL: https://github.com/llvm/llvm-project/commit/433eef6dea23c18d52c51218bb9d7f9bfbf47ecf
DIFF: https://github.com/llvm/llvm-project/commit/433eef6dea23c18d52c51218bb9d7f9bfbf47ecf.diff

LOG: Revert "[IR] Explicitly specify target feature for module asm" (#207399)

Reverts llvm/llvm-project#204548

This is causing the runtimes build to fail with e.g.:

```
<inline asm>:11:5: error: 32 bit reloc applied to a field with a different size
   11 | jmp __interceptor_strlen at plt
      |     ^
```

See comments on the PR.

Added: 
    

Modified: 
    clang/lib/CodeGen/CGObjCMac.cpp
    clang/lib/CodeGen/CodeGenModule.cpp
    clang/test/CodeGen/2006-01-23-FileScopeAsm.c
    clang/test/CodeGen/asm.c
    clang/test/CodeGen/asm_incbin.c
    clang/test/CodeGen/cfi-salt.c
    clang/test/CodeGen/kcfi.c
    clang/test/CodeGenCUDA/filter-decl.cu
    clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
    clang/test/CodeGenObjC/category-class.m
    clang/test/CodeGenObjC/objc-runtime-name.m
    clang/test/Frontend/ast-codegen.c
    clang/test/Modules/codegen.test
    llvm/docs/LangRef.rst
    llvm/docs/ReleaseNotes.md
    llvm/include/llvm/Bitcode/LLVMBitCodes.h
    llvm/include/llvm/CodeGen/AsmPrinter.h
    llvm/include/llvm/IR/Module.h
    llvm/lib/AsmParser/LLParser.cpp
    llvm/lib/Bitcode/Reader/BitcodeReader.cpp
    llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
    llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
    llvm/lib/IR/AsmWriter.cpp
    llvm/lib/IR/Core.cpp
    llvm/lib/IR/Module.cpp
    llvm/lib/LTO/LTO.cpp
    llvm/lib/Linker/IRMover.cpp
    llvm/lib/Object/ModuleSymbolTable.cpp
    llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
    llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
    llvm/lib/Transforms/IPO/ExtractGV.cpp
    llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
    llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
    llvm/lib/Transforms/Utils/SplitModule.cpp
    llvm/test/Bitcode/compatibility-3.6.ll
    llvm/test/Bitcode/compatibility-3.7.ll
    llvm/test/Bitcode/compatibility-3.8.ll
    llvm/test/Bitcode/compatibility-3.9.ll
    llvm/test/Bitcode/compatibility-4.0.ll
    llvm/test/Bitcode/compatibility-5.0.ll
    llvm/test/Bitcode/compatibility-6.0.ll
    llvm/test/Bitcode/compatibility.ll
    llvm/test/Bitcode/highLevelStructure.3.2.ll
    llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
    llvm/test/LTO/X86/inline-asm-lto-discard.ll
    llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
    llvm/test/ThinLTO/X86/import-symver.ll
    llvm/test/Transforms/LowerTypeTests/export-symver.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
    llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
    mlir/lib/Target/LLVMIR/ModuleImport.cpp
    mlir/test/Target/LLVMIR/module-asm.mlir

Removed: 
    llvm/test/Assembler/module-asm-invalid.ll
    llvm/test/Bitcode/module-asm.ll
    llvm/test/CodeGen/RISCV/module-asm-features.ll
    llvm/test/LTO/RISCV/module-asm.ll
    llvm/test/Linker/Inputs/module-asm.ll
    llvm/test/Linker/module-asm.ll


################################################################################
diff  --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index e518b875f44bb..69c5e88f3c768 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,6 +6610,10 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
+    Asm += CGM.getModule().getModuleInlineAsm();
+    if (!Asm.empty() && Asm.back() != '\n')
+      Asm += '\n';
+
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6620,7 +6624,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().appendModuleInlineAsm(OS.str());
+    CGM.getModule().setModuleInlineAsm(OS.str());
   }
 }
 

diff  --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index e155fdd752d7f..813cd67e4dd51 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,13 +8057,7 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-
-    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
-    llvm::Module::GlobalAsmProperties Props;
-    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
-    Props.TargetCPU = TargetOpts.CPU;
-    getModule().appendModuleInlineAsm(
-        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
+    getModule().appendModuleInlineAsm(AD->getAsmString());
     break;
   }
 

diff  --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 8a212a0b61905..877da4fc669e3 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,18 +1,13 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
-// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
 
-// RISCV64: module asm(target_features: "+64bit,+i")
-// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
-// DEFAULT: module asm
-// CHECK-NEXT: "foo1"
+// CHECK: module asm "foo1"
 __asm__ ("foo1");
-// CHECK-NEXT: "foo2"
+// CHECK: module asm "foo2"
 __asm__ ("foo2");
-// CHECK-NEXT: "foo3"
+// CHECK: module asm "foo3"
 __asm__ ("foo3");
-// CHECK-NEXT: "foo4"
+// CHECK: module asm "foo4"
 __asm__ ("foo4");
-// CHECK-NEXT: "foo5"
+// CHECK: module asm "foo5"
 __asm__ ("foo5");

diff  --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index ea69c7a17d1cb..d7465b22fbbf6 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,10 +2,9 @@
 
 // PR10415:
 //
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo1"
-// CHECK-NEXT: "foo2"
-// CHECK-NEXT: "foo3"
+// CHECK:      module asm "foo1"
+// CHECK-NEXT: module asm "foo2"
+// CHECK-NEXT: module asm "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");

diff  --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index bbc5d3af55ea2..6234eea1d6738 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,5 +6,4 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm
-// CHECK-NEXT: ".incbin \22foo.h\22"
+// CHECK: module asm ".incbin \22foo.h\22"

diff  --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 18156dc014280..2a90dc31d939b 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,14 +26,13 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_f6"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;

diff  --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index f08cf7635fb5a..da32859dd3d88 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,11 +6,10 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm
-// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2

diff  --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index f87d62a56631e..02dacd0ad8ef4 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,9 +5,8 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm
-// CHECK-HOST-NEXT: "file scope asm is host only"
-// CHECK-DEVICE-NOT: "file scope asm is host only"
+// CHECK-HOST: module asm "file scope asm is host only"
+// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global

diff  --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 2b4abee5dc05f..175a3b7bc588c 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,8 +19,7 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm(target_features: "{{.*}}")
-    // CHECK-NEXT: "nop; nop; nop; nop"
+    // CHECK: module asm "nop; nop; nop; nop"
 }
 
 int func() {return 0;};

diff  --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index febe94ce5b9f5..92fd36cbe4066 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,10 +1,9 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm
-// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
-// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
-// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
+// CHECK: module asm "\09.objc_category_name_A_foo=0"
+// CHECK: module asm "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end

diff  --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index a7f59caced390..c838b8c2c6f1a 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,10 +3,9 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm
-// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
-// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
-// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm {{.*}}objc_class_name_XYZ=0
+// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
+// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"

diff  --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index c26b0a1908379..2e2e3d360dd17 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,8 +2,7 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm(target_features: "{{.*}}")
-// CHECK-NEXT: "foo"
+// CHECK: module asm "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4

diff  --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 5cc5b3c1cda2a..0af630a754805 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,8 +18,7 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm
-BOTH-NEXT: "narf"
+BOTH: module asm "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any

diff  --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 2c8276d04b668..5ce6019660e0b 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,34 +3357,19 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks:
+to the GCC "file scope inline asm" blocks. These blocks are internally
+concatenated by LLVM and treated as a single unit, but may be separated
+in the ``.ll`` file if desired. The syntax is very simple:
 
 .. code-block:: llvm
 
-    module asm
-        "inline asm code goes here"
-        "more can go here"
+    module asm "inline asm code goes here"
+    module asm "more can go here"
 
-The sequence of string literals is internally concatenated using newlines as
-separators. The strings can contain any character by escaping non-printable
+The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
-Additionally, module-level inline assembly can specify an optional list of
-properties:
-
-.. code-block:: llvm
-
-    module asm(target_features: "+foo", target_cpu: "bar")
-        "inline asm code goes here"
-        "more can go here"
-
-Currently, the only supported properties are ``target_features`` and
-``target_cpu``, with the same meaning as the ``"target-features"`` and
-``"target-cpu"`` function attributes.
-
-Consecutive blocks with identical properties will be concatenated into one.
-
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 

diff  --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index cbec4d5b912d0..e42a959ab3101 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -111,10 +111,6 @@ Makes programs 10x faster by doing Special New Thing.
   corresponding metadata) now apply only at the point of definition, instead of
   for the execution of the function (for arguments) or forever (for returns).
 
-* Module-level inline assembly now accepts optional `target_features` and
-  `target_cpu` properties. This resolves errors during LTO on some
-  architectures.
-
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to

diff  --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index 1761e7a542a73..af4cac996a78c 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,8 +120,6 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
-
-  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.

diff  --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 335bf9de9a071..78e64440657cc 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,19 +930,6 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
-  /// Emit necessary directives to allow use of instructions that are permitted
-  /// by target features enabled by STI, but are not permitted by target
-  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
-  /// Returns whether anything was emitted.
-  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
-    return false;
-  }
-
-  /// Emit necessary directives to restore target feature state.
-  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
-  /// call.
-  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
-
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,

diff  --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 63d1013896b47..4f7e33969f16f 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,45 +169,6 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
-  struct GlobalAsmProperties {
-    std::string TargetFeatures;
-    std::string TargetCPU;
-
-    /// Set a property using a string name.
-    /// Returns whether the property name was valid.
-    LLVM_ABI bool set(StringRef Name, std::string Value);
-
-    /// Get a list of set properties as pairs of key and value.
-    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
-
-    bool operator==(const GlobalAsmProperties &Other) const {
-      return TargetFeatures == Other.TargetFeatures &&
-             TargetCPU == Other.TargetCPU;
-    }
-
-    bool operator!=(const GlobalAsmProperties &Other) const {
-      return !(*this == Other);
-    }
-  };
-
-  struct GlobalAsmFragment {
-    std::string Asm;
-    GlobalAsmProperties Props;
-
-    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
-    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
-        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
-      if (!Asm.empty() && Asm.back() != '\n')
-        Asm += '\n';
-    }
-
-    bool empty() const { return Asm.empty(); }
-
-    bool hasSameProperties(const GlobalAsmFragment &Other) const {
-      return Props == Other.Props;
-    }
-  };
-
 /// @}
 /// @name Member Variables
 /// @{
@@ -219,8 +180,7 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  /// Inline Asm at the global scope.
-  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
+  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -327,17 +287,8 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
-    return GlobalScopeAsm;
-  }
-
-  /// Get any module-scope inline assembly blocks.
-  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
-    return GlobalScopeAsm;
-  }
-
-  /// Return whether there is any module-scope inline assembly.
-  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
+  /// @returns a string containing the module-scope inline assembly blocks.
+  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -374,45 +325,20 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
-  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
-
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
-    GlobalScopeAsm.clear();
-    appendModuleInlineAsm(std::move(Fragment));
-  }
-
-  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
-    GlobalScopeAsm.clear();
-    append_range(GlobalScopeAsm, Fragments);
+  void setModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm = std::string(Asm);
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.back().Asm += Fragment.Asm;
-    } else {
-      GlobalScopeAsm.emplace_back(std::move(Fragment));
-    }
-  }
-
-  /// Prepend to the module-scope inline assembly blocks.
-  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
-    if (Fragment.empty())
-      return;
-
-    if (!GlobalScopeAsm.empty() &&
-        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
-      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
-    } else {
-      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
-    }
+  void appendModuleInlineAsm(StringRef Asm) {
+    GlobalScopeAsm += Asm;
+    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
+      GlobalScopeAsm += '\n';
   }
 
 /// @}

diff  --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 0a5b4c6c91879..75e6add0ec76f 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,49 +641,16 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
-///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
-///                          'property_name2:' STRINGCONSTANT ')'
-///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
+      parseStringConstant(AsmStr))
     return true;
 
-  Module::GlobalAsmProperties Props;
-  if (EatIfPresent(lltok::lparen)) {
-    while (true) {
-      std::string Key, Value;
-      SMLoc Loc = Lex.getLoc();
-      if (Lex.getKind() != lltok::LabelStr)
-        return error(Loc, "expected property name followed by ':'");
-
-      Key = Lex.getStrVal();
-      Lex.Lex();
-
-      if (parseStringConstant(Value))
-        return true;
-
-      if (!Props.set(Key, Value))
-        return error(Loc, "unknown property name");
-
-      if (EatIfPresent(lltok::rparen))
-        break;
-      if (parseToken(lltok::comma, "expected ',' or ')'"))
-        return true;
-    }
-  }
-
-  do {
-    std::string AsmStrPart;
-    if (parseStringConstant(AsmStrPart))
-      return true;
-    AsmStr += AsmStrPart + "\n";
-  } while (Lex.getKind() == lltok::StringConstant);
-
-  M->appendModuleInlineAsm({AsmStr, Props});
+  M->appendModuleInlineAsm(AsmStr);
   return false;
 }
 

diff  --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 06e9aea4fd3ac..2bd251efd05ce 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,9 +4580,6 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
-  // Apply to the following module asm.
-  Module::GlobalAsmProperties Props;
-
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4791,23 +4788,11 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
-    case bitc::MODULE_CODE_ASM_PROPERTY: {
-      std::string Str;
-      if (convertToString(Record, 0, Str))
-        return error("Invalid module asm record");
-      size_t SepPos = Str.find('\0');
-      if (SepPos == std::string::npos)
-        return error("Invalid module asm record");
-      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
-        return error("Unknown module asm property");
-      break;
-    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
-      Props = {};
+      TheModule->setModuleInlineAsm(S);
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]

diff  --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 493cbec7506f2..4aa203b0d12d1 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,19 +1551,9 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-
-  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    SmallVector<std::pair<StringRef, StringRef>> Props =
-        Frag.Props.getAsStrings();
-    for (auto [Key, Value] : Props) {
-      SmallVector<unsigned, 64> Record;
-      Record.append(Key.begin(), Key.end());
-      Record.push_back(0);
-      Record.append(Value.begin(), Value.end());
-      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
-    }
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
-  }
+  if (!M.getModuleInlineAsm().empty())
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
+                      0 /*TODO*/);
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.

diff  --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 8248a05f1a35f..abb20a7919fa1 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,7 +110,6 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
-#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -128,6 +127,7 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
+#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,29 +610,13 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (M.hasModuleInlineAsm()) {
+  if (!M.getModuleInlineAsm().empty()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
-        std::unique_ptr<MCSubtargetInfo> AsmSTI(
-            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
-                                                 Frag.Props.TargetCPU,
-                                                 Frag.Props.TargetFeatures));
-        bool DidPush = emitTargetFeaturePush(*AsmSTI);
-        emitInlineAsm(
-            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-        emitTargetFeaturePop(*AsmSTI, DidPush);
-      } else {
-        // If the module asm does not explicitly specify target features,
-        // fall back to default subtargetinfo.
-        emitInlineAsm(
-            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
-            nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-      }
-    }
+    emitInlineAsm(
+        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
+        TM.Options.MCOptions, nullptr,
+        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }

diff  --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 02d6bcd90add0..665d7122af2c7 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,38 +3126,21 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (M->hasModuleInlineAsm()) {
+  if (!M->getModuleInlineAsm().empty()) {
     Out << '\n';
 
-    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
-      Out << "module asm";
-      SmallVector<std::pair<StringRef, StringRef>> Props =
-          Frag.Props.getAsStrings();
-      if (!Props.empty()) {
-        ListSeparator LS;
-        Out << "(";
-        for (auto [Key, Value] : Props) {
-          Out << LS;
-          Out << Key << ": \"";
-          printEscapedString(Value, Out);
-          Out << "\"";
-        }
-        Out << ")";
-      }
-      Out << "\n";
-      // Split the string into lines, to make it easier to read the .ll file.
-      StringRef Asm = Frag.Asm;
-      do {
-        StringRef Front;
-        std::tie(Front, Asm) = Asm.split('\n');
-
-        // We found a newline, print the portion of the asm string from the
-        // last newline up to this newline.
-        Out << "    \"";
-        printEscapedString(Front, Out);
-        Out << "\"\n";
-      } while (!Asm.empty());
-    }
+    // Split the string into lines, to make it easier to read the .ll file.
+    StringRef Asm = M->getModuleInlineAsm();
+    do {
+      StringRef Front;
+      std::tie(Front, Asm) = Asm.split('\n');
+
+      // We found a newline, print the portion of the asm string from the
+      // last newline up to this newline.
+      Out << "module asm \"";
+      printEscapedString(Front, Out);
+      Out << "\"\n";
+    } while (!Asm.empty());
   }
 
   printTypeIdentities();

diff  --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index a7abd3eed31c3..6c6c3b5c84ede 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,18 +510,7 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  Module *Mod = unwrap(M);
-  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
-  if (Frags.empty()) {
-    *Len = 0;
-    return nullptr;
-  }
-
-  if (Frags.size() != 1)
-    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
-                          "more than one module inline assembly fragment");
-
-  auto &Str = Frags.begin()->Asm;
+  auto &Str = unwrap(M)->getModuleInlineAsm();
   *Len = Str.length();
   return Str.c_str();
 }

diff  --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 34e44e26d5db5..40ef7e354da72 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,23 +977,3 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
-
-bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
-  if (Name == "target_features")
-    TargetFeatures = std::move(Value);
-  else if (Name == "target_cpu")
-    TargetCPU = std::move(Value);
-  else
-    return false;
-  return true;
-}
-
-SmallVector<std::pair<StringRef, StringRef>>
-Module::GlobalAsmProperties::getAsStrings() const {
-  SmallVector<std::pair<StringRef, StringRef>> Props;
-  if (!TargetFeatures.empty())
-    Props.emplace_back("target_features", TargetFeatures);
-  if (!TargetCPU.empty())
-    Props.emplace_back("target_cpu", TargetCPU);
-  return Props;
-}

diff  --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index b672f4e5fac14..6a754f95092bb 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (M.hasModuleInlineAsm()) {
+  if (!M.getModuleInlineAsm().empty()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.prependModuleInlineAsm(NewIA);
+    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
   }
 
   assert(MsymI == MsymE);

diff  --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 3b72b412d0b2e..239201bde1cec 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,11 +1562,8 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
-      Module::GlobalAsmFragment NewFrag(Frag);
-      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
-      DstM.appendModuleInlineAsm(std::move(NewFrag));
-    }
+    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
+                                               SrcTriple));
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1576,7 +1573,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(std::string(S));
+        DstM.appendModuleInlineAsm(S);
       }
     });
   }

diff  --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 44f717ec12323..1da5fa9c10a0b 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,7 +72,8 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  if (!M.hasModuleInlineAsm())
+  StringRef InlineAsm = M.getModuleInlineAsm();
+  if (InlineAsm.empty())
     return;
 
   std::string Err;
@@ -89,55 +90,52 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
+  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
+  if (!STI)
+    return;
+
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
-        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
-    if (!STI)
-      return;
-
-    std::unique_ptr<MemoryBuffer> Buffer(
-        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
-    SourceMgr SrcMgr;
-    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-    std::unique_ptr<MCObjectFileInfo> MOFI(
-        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-    MCCtx.setObjectFileInfo(MOFI.get());
-    RecordStreamer Streamer(MCCtx, M);
-    T->createNullTargetStreamer(Streamer);
-
-    std::unique_ptr<MCAsmParser> Parser(
-        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-    std::unique_ptr<MCTargetAsmParser> TAP(
-        T->createMCAsmParser(*STI, *Parser, *MCII));
-    if (!TAP)
-      return;
-
-    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                   const SourceMgr &SrcMgr,
-                                   std::vector<const MDNode *> &LocInfos) {
-      M.getContext().diagnose(
-          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-    });
-
-    // Module-level inline asm is assumed to use At&t syntax (see
-    // AsmPrinter::doInitialization()).
-    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-    Parser->setSymbolScanningMode(true);
-
-    Parser->setTargetParser(*TAP);
-    if (Parser->Run(false))
-      return;
-
-    Init(Streamer);
-  }
+  std::unique_ptr<MemoryBuffer> Buffer(
+      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
+  SourceMgr SrcMgr;
+  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+  std::unique_ptr<MCObjectFileInfo> MOFI(
+      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+  MCCtx.setObjectFileInfo(MOFI.get());
+  RecordStreamer Streamer(MCCtx, M);
+  T->createNullTargetStreamer(Streamer);
+
+  std::unique_ptr<MCAsmParser> Parser(
+      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+  std::unique_ptr<MCTargetAsmParser> TAP(
+      T->createMCAsmParser(*STI, *Parser, *MCII));
+  if (!TAP)
+    return;
+
+  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                 const SourceMgr &SrcMgr,
+                                 std::vector<const MDNode *> &LocInfos) {
+    M.getContext().diagnose(
+        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+  });
+
+  // Module-level inline asm is assumed to use At&t syntax (see
+  // AsmPrinter::doInitialization()).
+  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+  Parser->setSymbolScanningMode(true);
+
+  Parser->setTargetParser(*TAP);
+  if (Parser->Run(false))
+    return;
+
+  Init(Streamer);
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(

diff  --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index e37ea27364e6e..dbda25c34cee0 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,8 +113,7 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
-  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
+  bool emitDirectiveOptionArch();
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -524,19 +523,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                               : RISCVOptionArchArgType::Minus;
+    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                                : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -550,16 +549,11 @@ bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   return false;
 }
 
-void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
-                                           bool DidPush) {
-  if (DidPush)
-    getTargetStreamer().emitDirectiveOptionPop();
-}
-
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
+  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
+  bool EmittedOptionArch = emitDirectiveOptionArch();
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -567,7 +561,8 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  emitTargetFeaturePop(*STI, EmittedOptionArch);
+  if (EmittedOptionArch)
+    RTS.emitDirectiveOptionPop();
   return false;
 }
 

diff  --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 22d0d97090746..2a5d16a470c4a 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.removeModuleInlineAsm();
+    M.setModuleInlineAsm("");
   }
 
   // Register the NSDI handler before calling the base class so that

diff  --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 4b76b02c32fe4..16fdb93e2b21b 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.removeModuleInlineAsm();
+    M.setModuleInlineAsm("");
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually

diff  --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index e857c94330bad..a486979a58c2d 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->removeModuleInlineAsm();
+  MergedM->setModuleInlineAsm("");
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.

diff  --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index 375ac069272eb..c349fe33dd237 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,20 +1283,18 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  for (Module::GlobalAsmFragment &Frag :
-       GV->getParent()->getModuleInlineAsm()) {
-    std::string SearchStr = ".symver " + GVName + ",";
-    size_t Pos = Frag.Asm.find(SearchStr);
-    if (Pos != std::string::npos) {
-      Frag.Asm.replace(Pos, SearchStr.size(),
-                       ".symver " + GVName + Suffix + ",");
-      Pos = Frag.Asm.find('@');
-
-      if (Pos == std::string::npos)
-        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
-
-      Frag.Asm.replace(Pos, 1, Suffix + "@");
-    }
+  std::string Asm = GV->getParent()->getModuleInlineAsm();
+  std::string SearchStr = ".symver " + GVName + ",";
+  size_t Pos = Asm.find(SearchStr);
+  if (Pos != std::string::npos) {
+    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
+    Pos = Asm.find('@');
+
+    if (Pos == std::string::npos)
+      report_fatal_error(Twine("unsupported .symver: ", Asm));
+
+    Asm.replace(Pos, 1, Suffix + "@");
+    GV->getParent()->setModuleInlineAsm(Asm);
   }
 }
 

diff  --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 8e70902c4dac2..64910b4d1ce99 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->removeModuleInlineAsm();
+      MPart->setModuleInlineAsm("");
     ModuleCallback(std::move(MPart));
   }
 }

diff  --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
deleted file mode 100644
index 34e5b71e683a9..0000000000000
--- a/llvm/test/Assembler/module-asm-invalid.ll
+++ /dev/null
@@ -1,25 +0,0 @@
-; RUN: split-file %s %t
-; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
-; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
-; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
-; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
-
-;--- invalid_prop.ll
-; CHECK: unknown property name
-module asm(foo: "bar")
-    "asm"
-
-;--- missing_value1.ll
-; CHECK: expected property name followed by ':'
-module asm(target_features)
-    "asm"
-
-;--- missing_value2.ll
-; CHECK: expected string constant
-module asm(target_features:)
-    "asm"
-
-;--- missing_paren.ll
-; CHECK: expected ',' or ')'
-module asm(target_features: "bar"
-    "asm"

diff  --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index c79086bad1ba5..288e0fc50a9ba 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,8 +14,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index 00b2bb9b30b62..ef9490310fe5d 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,8 +14,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 5b86d5ffa1fee..538d6101b3994 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index eca4bbdf03766..3bfef73e485dd 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index 8551b98815b1a..aaf9fe5492171 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index 883ac3eaaa293..b820417107e8b 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 624d5bd7703c6..67e96c7277cc3 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,8 +13,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index 97889d4687516..d55aa1dc496df 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,8 +16,7 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm
-; CHECK-NEXT: "beep boop"
+; CHECK: module asm "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index cd9a34c810d57..f2770ea4e4d34 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,8 +9,7 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm
-; CHECK-NEXT: "some assembly"
+; CHECK: module asm "some assembly"
 module asm "some assembly"
 
 ; Named Types Test

diff  --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
deleted file mode 100644
index 406f25c5dd28c..0000000000000
--- a/llvm/test/Bitcode/module-asm.ll
+++ /dev/null
@@ -1,29 +0,0 @@
-; RUN: llvm-as < %s | llvm-dis | FileCheck %s
-
-; CHECK: module asm
-; CHECK:     "asm line 0"
-; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
-; CHECK:     "asm line 1"
-; CHECK:     "asm line 2"
-; CHECK: module asm(target_features: "+bar")
-; CHECK:     "asm line 3"
-; CHECK: module asm(target_cpu: "bar_cpu")
-; CHECK:     "asm line 4"
-; CHECK: module asm
-; CHECK:     "asm line 5"
-
-module asm
-    "asm line 0"
-
-module asm(target_features: "+foo", target_cpu: "foo_cpu")
-    "asm line 1"
-    "asm line 2"
-
-module asm(target_features: "+bar")
-    "asm line 3"
-
-module asm(target_cpu: "bar_cpu")
-    "asm line 4"
-
-module asm
-    "asm line 5"

diff  --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
deleted file mode 100644
index ab16acab48688..0000000000000
--- a/llvm/test/CodeGen/RISCV/module-asm-features.ll
+++ /dev/null
@@ -1,17 +0,0 @@
-; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
-; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
-
-; This should work fine, because the module asm specifies the necessary
-; target features
-
-; SAME-FEATURES-NOT: .option arch
-; EXTRA-FEATURES: .option push
-; EXTRA-FEATURES: .option arch, +d
-; CHECK: fld ft0, 0(sp)
-; EXTRA-FEATURES: .option pop
-
-module asm(target_features: "+d")
-    ".globl func"
-    "func:"
-    "fld f0, 0(sp)"
-    "ret"

diff  --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 829dd829d51f9..454c496d1d4bb 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,8 +2,7 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm
-; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan

diff  --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
deleted file mode 100644
index 73320185e778a..0000000000000
--- a/llvm/test/LTO/RISCV/module-asm.ll
+++ /dev/null
@@ -1,18 +0,0 @@
-; RUN: llvm-as %s -o %t.o
-; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
-; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
-; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
-; RUN: FileCheck %s --input-file %t.s.0
-
-; NM: T func
-
-; CHECK: fld ft0, 0(sp)
-
-target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
-target triple = "riscv64-unknown-linux-gnu"
-
-module asm(target_features: "+d")
-    ".globl func"
-    "func:"
-    "fld f0, 0(sp)"
-    "ret"

diff  --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 6c5173c3f9553..88492e4366489 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,19 +32,17 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1: module asm
-; ASM1-NEXT: ".lto_discard foo"
-; ASM1-NEXT: ".weak foo"
-; ASM1-NEXT: ".equ foo,bar"
-
-; ASM2: module asm
-; ASM2-NEXT: ".lto_discard foo"
-; ASM2-NEXT: ".weak foo"
-; ASM2-NEXT: ".equ foo,bar"
-; ASM2-NEXT: ".lto_discard"
-; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  ".lto_discard foo"
+; ASM1:      module asm ".lto_discard foo"
+; ASM1-NEXT: module asm ".weak foo"
+; ASM1-NEXT: module asm ".equ foo,bar"
+
+; ASM2:      module asm ".lto_discard foo"
+; ASM2-NEXT: module asm ".weak foo"
+; ASM2-NEXT: module asm ".equ foo,bar"
+; ASM2-NEXT: module asm ".lto_discard"
+; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  module asm ".lto_discard foo"
 
 ; SYM: T foo
 

diff  --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
deleted file mode 100644
index b097c7067ccc1..0000000000000
--- a/llvm/test/Linker/Inputs/module-asm.ll
+++ /dev/null
@@ -1,2 +0,0 @@
-module asm(target_features: "+bar")
-    "asm 2"

diff  --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 612bce5be18f1..5e12a8c88d90a 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,12 +9,11 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK: module asm
-; CHECK-NEXT: ".text"
-; CHECK-NEXT: ".balign 4"
-; CHECK-NEXT: ".arm"
-; CHECK-NEXT: "add r1, r2, r2"
-; CHECK-NEXT: ".text"
-; CHECK-NEXT: ".balign 2"
-; CHECK-NEXT: ".thumb"
-; CHECK-NEXT: "orn r1, r2, r2"
+; CHECK:      module asm ".text"
+; CHECK-NEXT: module asm ".balign 4"
+; CHECK-NEXT: module asm ".arm"
+; CHECK-NEXT: module asm "add r1, r2, r2"
+; CHECK-NEXT: module asm ".text"
+; CHECK-NEXT: module asm ".balign 2"
+; CHECK-NEXT: module asm ".thumb"
+; CHECK-NEXT: module asm "orn r1, r2, r2"

diff  --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
deleted file mode 100644
index 6c600981d72a5..0000000000000
--- a/llvm/test/Linker/module-asm.ll
+++ /dev/null
@@ -1,9 +0,0 @@
-; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
-
-; CHECK: module asm(target_features: "+foo")
-; CHECK-NEXT: "asm 1"
-; CHECK: module asm(target_features: "+bar")
-; CHECK-NEXT: "asm 2"
-
-module asm(target_features: "+foo")
-    "asm 1"

diff  --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 9525527b4531c..556c4fd992f0d 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,8 +9,7 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm
-; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.

diff  --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index 7c3ed3da3537e..ea4594a359cc0 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,7 +1,6 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm
-; CHECK-NEXT: ".symver external_addrtaken, alias1"
+; CHECK: module asm ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index 61141c71335bf..d8ebae17d4693 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,8 +3,7 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm
-; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 5796537661e60..587ab3fbb5020 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,7 +7,6 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm
-; M0-NEXT: "ret"
+; M0: module asm "ret"
 ; M1-NOT: module asm
 module asm "ret"

diff  --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index b5b68404e8690..4aeaef6d8d676 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.removeModuleInlineAsm();
+    Program.setModuleInlineAsm("");
 }

diff  --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index bb55678f91005..3038a6a2ed986 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,15 +1150,12 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
+  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (const llvm::Module::GlobalAsmFragment &Frag :
-       llvmModule->getModuleInlineAsm()) {
-    // TODO: Preserve module asm properties.
-    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
-      if (!line.empty())
-        asmArrayAttr.push_back(builder.getStringAttr(line));
-  }
+  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
+    if (!line.empty())
+      asmArrayAttr.push_back(builder.getStringAttr(line));
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));

diff  --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 813eeeafdeb44..2afb37cf005bf 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,6 +2,5 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm
-// CHECK-NEXT: "foo"
-// CHECK-NEXT: "bar"
+// CHECK: module asm "foo"
+// CHECK: module asm "bar"


        

From llvmlistbot at llvm.org  Fri Jul  3 08:24:48 2026
From: llvmlistbot at llvm.org (Paul Osmialowski)
Date: Fri, 03 Jul 2026 08:24:48 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [Flang][OpenMP][MLIR]
 Improve SPMD-No-Loop handling (PR #206953)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206953@github.com>
Message-ID: <6a47d440.92ea0e2a.21f97.60c3@mx.google.com>


pawosm-arm wrote:

Heads up: this commit causes the STREAM project (https://github.com/jeffhammond/STREAM.git) to fail at the `checksums` subroutine (https://github.com/jeffhammond/STREAM/blob/6703f7504a38a8da96b353cadafa64d3c2d7a2d3/stream.f#L398), even when running with just a single OpenMP thread. Before opening a bug report, I'll try to build `main` with this commit reverted and see if the problem is gone.

https://github.com/llvm/llvm-project/pull/206953

From llvmlistbot at llvm.org  Fri Jul  3 08:25:49 2026
From: llvmlistbot at llvm.org (Rahul Joshi)
Date: Fri, 03 Jul 2026 08:25:49 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NFC][LLVM] Define range attribute for
 `llvm.prefetch` args (PR #207300)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207300@github.com>
Message-ID: <6a47d47d.27b3b003.1610ae.cfda@mx.google.com>


https://github.com/jurahul closed https://github.com/llvm/llvm-project/pull/207300

From llvmlistbot at llvm.org  Fri Jul  3 08:25:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 08:25:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 780298c - [NFC][LLVM] Define range attribute
 for `llvm.prefetch` args (#207300)
Message-ID: <6a47d47d.ff037943.3bf0e2.1418@mx.google.com>


Author: Rahul Joshi
Date: 2026-07-03T08:25:42-07:00
New Revision: 780298c4424514f0205df5e82c2a135717e6a723

URL: https://github.com/llvm/llvm-project/commit/780298c4424514f0205df5e82c2a135717e6a723
DIFF: https://github.com/llvm/llvm-project/commit/780298c4424514f0205df5e82c2a135717e6a723.diff

LOG: [NFC][LLVM] Define range attribute for `llvm.prefetch` args (#207300)

Define valid range for `llvm.prefetch` immarg values and eliminate C++
verification code.

Additionally, change the range verification failure message to include
the argument number and simplify formatting using `formatv`.

Added: 
    

Modified: 
    llvm/include/llvm/IR/Intrinsics.td
    llvm/lib/IR/Verifier.cpp
    llvm/test/Assembler/invalid-immarg5.ll
    llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
    llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
    llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
    llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
    llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
    llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
    llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
    llvm/test/Verifier/NVPTX/setmaxnreg.ll
    llvm/test/Verifier/intrinsic-immarg.ll
    mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir

Removed: 
    


################################################################################
diff  --git a/llvm/include/llvm/IR/Intrinsics.td b/llvm/include/llvm/IR/Intrinsics.td
index 0c6316ccf0a5c..f75335cd2c60a 100644
--- a/llvm/include/llvm/IR/Intrinsics.td
+++ b/llvm/include/llvm/IR/Intrinsics.td
@@ -1045,8 +1045,10 @@ def int_prefetch
                 [llvm_anyptr_ty, llvm_i32_ty, llvm_i32_ty, llvm_i32_ty],
                 [IntrInaccessibleMemOrArgMemOnly,
                  ReadOnly<ArgIndex<0>>, NoCapture<ArgIndex<0>>,
-                 ImmArg<ArgIndex<1>>, ImmArg<ArgIndex<2>>,
-                 ImmArg<ArgIndex<3>>]>;
+                 ImmArg<ArgIndex<1>>, Range<ArgIndex<1>, 0, 2>,
+                 ImmArg<ArgIndex<2>>, Range<ArgIndex<2>, 0, 4>,
+                 ImmArg<ArgIndex<3>>, Range<ArgIndex<3>, 0, 2>
+                ]>;
 def int_pcmarker      : DefaultAttrsIntrinsic<[], [llvm_i32_ty]>;
 
 def int_readcyclecounter : DefaultAttrsIntrinsic<[llvm_i64_ty]>;

diff  --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5f8b5e187bf14..f35e9066ab1f2 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -119,6 +119,7 @@
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/ModRef.h"
 #include "llvm/Support/TimeProfiler.h"
@@ -3897,9 +3898,8 @@ void Verifier::visitCallBase(CallBase &Call) {
           const ConstantRange &CR =
               Call.getParamAttr(i, Attribute::Range).getValueAsConstantRange();
           Check(CR.contains(CI->getValue()),
-                "immarg value " + Twine(CI->getValue().getSExtValue()) +
-                    " out of range [" + Twine(CR.getLower().getSExtValue()) +
-                    ", " + Twine(CR.getUpper().getSExtValue()) + ")",
+                formatv("immarg value {} for arg {} out of range {}",
+                        CI->getValue(), i, CR),
                 Call);
         }
       }
@@ -6289,14 +6289,6 @@ void Verifier::visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call) {
           "llvm.init_trampoline parameter #2 must resolve to a function.",
           Call);
     break;
-  case Intrinsic::prefetch:
-    Check(cast<ConstantInt>(Call.getArgOperand(1))->getZExtValue() < 2,
-          "rw argument to llvm.prefetch must be 0-1", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
-          "locality argument to llvm.prefetch must be 0-3", Call);
-    Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
-          "cache type argument to llvm.prefetch must be 0-1", Call);
-    break;
   case Intrinsic::reloc_none: {
     Check(isa<MDString>(
               cast<MetadataAsValue>(Call.getArgOperand(0))->getMetadata()),

diff  --git a/llvm/test/Assembler/invalid-immarg5.ll b/llvm/test/Assembler/invalid-immarg5.ll
index deffbd3cad4cc..7c7bf11660e28 100644
--- a/llvm/test/Assembler/invalid-immarg5.ll
+++ b/llvm/test/Assembler/invalid-immarg5.ll
@@ -3,7 +3,7 @@
 declare void @llvm.test.immarg.range.intrinsic.i32(i32 immarg range(i32 -3, 4))
 
 define void @test_int_immarg_with_range() {
-  ; CHECK: immarg value -4 out of range [-3, 4)
+  ; CHECK: immarg value -4 for arg 0 out of range [-3,4)
   call void @llvm.test.immarg.range.intrinsic.i32(i32 -4)
   ret void
 }

diff  --git a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
index 1c35fbead389e..0cef9c1d9405c 100644
--- a/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/cp-async-bulk-tensor-g2s-invalid.ll
@@ -1,14 +1,12 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -o /dev/null 2>&1 | FileCheck %s
-
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) writeonly, ptr addrspace(3), ptr readonly, i32, i16, i64, i1 immarg, i1 immarg, i32 immarg range(i32 0, 3))
 
 define void @test_cp_async_bulk_tensor_g2s_tile_1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch) {
-  ; CHECK: immarg value 3 out of range [0, 3)
+  ; CHECK: immarg value 3 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 3)
 
-  ; CHECK: immarg value -1 out of range [0, 3)
+  ; CHECK: immarg value -1 for arg 8 out of range [0,3)
   tail call void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i16 %mc, i64 %ch, i1 0, i1 0, i32 -1)
 
   ret void

diff  --git a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
index ab35e4fb396d6..52f494ae590b6 100644
--- a/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/fence-proxy-tensormap-invalid.ll
@@ -1,7 +1,7 @@
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_90 -mattr=+ptx83 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 130 out of range [128, 129)
+  ; CHECK: immarg value 130 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 130);
 
   ret void

diff  --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
index c0f6f4c7c46bd..bd3ed2e447c0b 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-block-scale-invalid.ll
@@ -1,23 +1,20 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 5
-; RUN: not llc < %s -o - -mcpu=sm_100a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_101a -march=nvptx64 -mattr=+ptx88 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -o - -mcpu=sm_110a -march=nvptx64 -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_block_scale_invalid_flags(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b) {
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 1, i32 5)
   ret void
 }

diff  --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
index dff829ecf5321..c80755b6711d8 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-invalid.ll
@@ -1,37 +1,33 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_101a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_110a -mattr=+ptx90 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 1, i32 3)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 0, i32 3)
-  ; CHECK: immarg value 3 out of range [1, 3)
+  ; CHECK: immarg value 3 for arg 6 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 3, i32 3)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 5)
-  ; CHECK: immarg value 2 out of range [0, 2)
+  ; CHECK: immarg value 2 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.ashift(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 3, i32 2, i32 2)
   ret void
 }
 
 define void @tcgen05_mma_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_ws_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 5 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 5, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 6 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 5, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 7 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i32 0, i32 0, i32 5)
   ret void
 }

diff  --git a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
index 7c884a70b7530..5987668b10add 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-mma-scale-d-invalid.ll
@@ -1,25 +1,23 @@
-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
-; RUN: not llc < %s -mtriple=nvptx64 -mcpu=sm_100a -mattr=+ptx86 -o /dev/null 2>&1 | FileCheck %s
-target triple = "nvptx64-nvidia-cuda"
+; RUN: not llvm-as -disable-output < %s 2>&1 | FileCheck %s
 
 define void @tcgen05_mma_scale_d_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, i32 0, i32 1, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 6 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 3, i32 1, i32 0)
-  ; CHECK: immarg value 0 out of range [1, 3)
+  ; CHECK: immarg value 0 for arg 7 out of range [1,3)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 0, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %dtmem, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, i32 0, i32 1, i32 5)
   ret void
 }
 
 define void @tcgen05_mma_scale_d_disable_output_lane_invalid_flag_values(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %ashared, i64 %b, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lanev4) {
-  ; CHECK: immarg value 16 out of range [0, 16)
+  ; CHECK: immarg value 16 for arg 5 out of range [0,16)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 16, <4 x i32> %disable_output_lanev4, i32 0, i32 0)
-  ; CHECK: immarg value 3 out of range [0, 2)
+  ; CHECK: immarg value 3 for arg 7 out of range [0,2)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 3, i32 0)
-  ; CHECK: immarg value 5 out of range [0, 4)
+  ; CHECK: immarg value 5 for arg 8 out of range [0,4)
   call void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1(ptr addrspace(6) %dtmem, ptr addrspace(6) %atensor, i64 %b, i32 %idesc, i1 %enable_inp_d, i64 0, <4 x i32> %disable_output_lanev4, i32 0, i32 5)
   ret void
 }

diff  --git a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
index 4e8022711320d..98c73444ed62f 100644
--- a/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
+++ b/llvm/test/Verifier/AMDGPU/test-cvt-fp4f6f8-immarg-ranges.ll
@@ -4,14 +4,14 @@
 ; llvm.amdgcn.cvt.sr.fp8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 10 out of range [0, 4)
+; CHECK: immarg value 10 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
 define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.fp8.f16(half %src, i32 %seed, i32 %old, i32 10)
@@ -22,7 +22,7 @@ define i32 @test_cvt_sr_fp8_f16_byte_sel_way_out_of_range(half %src, i32 %seed,
 ; llvm.amdgcn.cvt.sr.bf8.f16 - byte_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
 define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32 %old) {
   %result = call i32 @llvm.amdgcn.cvt.sr.bf8.f16(half %src, i32 %seed, i32 %old, i32 4)
@@ -33,14 +33,14 @@ define i32 @test_cvt_sr_bf8_f16_byte_sel_out_of_range(half %src, i32 %seed, i32
 ; llvm.amdgcn.cvt.scale.pk8.f16.fp8 - scale_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 16 out of range [0, 16)
+; CHECK: immarg value 16 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 16)
   ret <8 x half> %result
 }
 
-; CHECK: immarg value 100 out of range [0, 16)
+; CHECK: immarg value 100 for arg 2 out of range [0,16)
 ; CHECK-NEXT: %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
 define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i32> %src) {
   %result = call <8 x half> @llvm.amdgcn.cvt.scale.pk8.f16.fp8(<2 x i32> %src, i32 0, i32 100)
@@ -51,14 +51,14 @@ define <8 x half> @test_cvt_scale_pk8_f16_fp8_scale_sel_way_out_of_range(<2 x i3
 ; llvm.amdgcn.cvt.scalef32.f32.fp8 - src_sel out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
 define float @test_cvt_scalef32_f32_fp8_src_sel_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 4)
   ret float %result
 }
 
-; CHECK: immarg value 7 out of range [0, 4)
+; CHECK: immarg value 7 for arg 2 out of range [0,4)
 ; CHECK-NEXT: %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
 define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float %scale) {
   %result = call float @llvm.amdgcn.cvt.scalef32.f32.fp8(i32 %src, float %scale, i32 7)
@@ -69,14 +69,14 @@ define float @test_cvt_scalef32_f32_fp8_src_sel_way_out_of_range(i32 %src, float
 ; llvm.amdgcn.cvt.scalef32.f16.fp8 - src_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 4, i1 false)
   ret <2 x half> %result
 }
 
-; CHECK: immarg value 15 out of range [0, 4)
+; CHECK: immarg value 15 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
 define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x half> %old, i32 %src, float %scale) {
   %result = call <2 x half> @llvm.amdgcn.cvt.scalef32.f16.fp8(<2 x half> %old, i32 %src, float %scale, i32 15, i1 true)
@@ -87,14 +87,14 @@ define <2 x half> @test_cvt_scalef32_f16_fp8_src_sel_index_way_out_of_range(<2 x
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f32 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 8 out of range [0, 4)
+; CHECK: immarg value 8 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
 define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old, float %src0, float %src1, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f32(i32 %old, float %src0, float %src1, float %scale, i32 8)
@@ -105,14 +105,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f32_dst_sel_index_way_out_of_range(i32 %old
 ; llvm.amdgcn.cvt.scalef32.pk.fp4.f16 - dest_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 12 out of range [0, 4)
+; CHECK: immarg value 12 for arg 3 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
 define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %old, <2 x half> %src, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.pk.fp4.f16(i32 %old, <2 x half> %src, float %scale, i32 12)
@@ -123,14 +123,14 @@ define i32 @test_cvt_scalef32_pk_fp4_f16_dest_sel_index_way_out_of_range(i32 %ol
 ; llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16 - dst_sel_index out of range
 ; --------------------------------------------------------------------
 
-; CHECK: immarg value 4 out of range [0, 4)
+; CHECK: immarg value 4 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 4)
   ret i32 %result
 }
 
-; CHECK: immarg value 9 out of range [0, 4)
+; CHECK: immarg value 9 for arg 4 out of range [0,4)
 ; CHECK-NEXT: %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)
 define i32 @test_cvt_scalef32_sr_pk_fp4_f16_dst_sel_index_way_out_of_range(i32 %old, <2 x half> %src, i32 %seed, float %scale) {
   %result = call i32 @llvm.amdgcn.cvt.scalef32.sr.pk.fp4.f16(i32 %old, <2 x half> %src, i32 %seed, float %scale, i32 9)

diff  --git a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
index 4fa7a7ae71001..652990a700a6e 100644
--- a/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
+++ b/llvm/test/Verifier/NVPTX/fence-proxy.tensormap.ll
@@ -1,16 +1,16 @@
 ; RUN: not llvm-as %s -o /dev/null 2>&1 | FileCheck %s
 
 define void @test_fence_proxy_tensormap_generic_acquire(ptr addrspace(0) %addr) {
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cluster(ptr addrspace(0) %addr, i32 129);
 
-  ; CHECK: immarg value 127 out of range [128, 129)
+  ; CHECK: immarg value 127 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.gpu(ptr addrspace(0) %addr, i32 127);
 
-  ; CHECK: immarg value 129 out of range [128, 129)
+  ; CHECK: immarg value 129 for arg 1 out of range [128,129)
   call void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.sys(ptr addrspace(0) %addr, i32 129);
 
   ret void

diff  --git a/llvm/test/Verifier/NVPTX/setmaxnreg.ll b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
index 1afebeab4742c..79714a0521c36 100644
--- a/llvm/test/Verifier/NVPTX/setmaxnreg.ll
+++ b/llvm/test/Verifier/NVPTX/setmaxnreg.ll
@@ -7,10 +7,10 @@ define void @test_set_maxn_reg() {
   ; CHECK: reg_count argument to nvvm.setmaxnreg must be in multiples of 8
   call void @llvm.nvvm.setmaxnreg.inc.sync.aligned.u32(i32 95)
 
-  ; CHECK: immarg value 16 out of range [24, 257)
+  ; CHECK: immarg value 16 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 16)
 
-  ; CHECK: immarg value 264 out of range [24, 257)
+  ; CHECK: immarg value 264 for arg 0 out of range [24,257)
   call void @llvm.nvvm.setmaxnreg.dec.sync.aligned.u32(i32 264)
   ret void
 }

diff  --git a/llvm/test/Verifier/intrinsic-immarg.ll b/llvm/test/Verifier/intrinsic-immarg.ll
index c95c7214fde71..39ba3cb480b4d 100644
--- a/llvm/test/Verifier/intrinsic-immarg.ll
+++ b/llvm/test/Verifier/intrinsic-immarg.ll
@@ -154,6 +154,18 @@ define void @test_prefetch(ptr %ptr, i32 %arg0, i32 %arg1) {
   ; CHECK-NEXT:  i32 %arg1
   call void @llvm.prefetch(ptr %ptr, i32 %arg0, i32 0, i32 0)
   call void @llvm.prefetch(ptr %ptr, i32 0, i32 %arg1, i32 0)
+
+  ; CHECK: immarg value 2 for arg 1 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 2, i32 0, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 2, i32 0, i32 0)
+
+  ; CHECK: immarg value 8 for arg 2 out of range [0,4)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 8, i32 0)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 8, i32 0)
+
+  ; CHECK: immarg value 4 for arg 3 out of range [0,2)
+  ; CHECK-NEXT: call void @llvm.prefetch.p0(ptr %ptr, i32 0, i32 0, i32 4)
+  call void @llvm.prefetch(ptr %ptr, i32 0, i32 0, i32 4)
   ret void
 }
 

diff  --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index ea393dd445eda..5266470891b4d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1420,7 +1420,7 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
 // CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
-// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg, i32 immarg, i32 immarg)
+// CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0


        

From llvmlistbot at llvm.org  Fri Jul  3 08:33:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 08:33:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a47d640.6c8668f2.346e76.19d2@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207307

>From 6695b3e0d8bddeab555e1ca36802df2decb85e82 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 +++++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 24 +++++++++++++++++++++
 mlir/test/Target/LLVMIR/nvvmir.mlir         | 24 +++++++++++++++++++++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {
diff --git a/mlir/test/Target/LLVMIR/nvvmir.mlir b/mlir/test/Target/LLVMIR/nvvmir.mlir
index f2888025d8a08..f726efe3d39b7 100644
--- a/mlir/test/Target/LLVMIR/nvvmir.mlir
+++ b/mlir/test/Target/LLVMIR/nvvmir.mlir
@@ -293,6 +293,30 @@ llvm.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // f32 return type, f32 accumulate type
 // CHECK-LABEL: @nvvm_mma_m16n8k16_f32_f32
 llvm.func @nvvm_mma_m16n8k16_f32_f32(%a0 : vector<2xf16>, %a1 : vector<2xf16>,


From llvmlistbot at llvm.org  Fri Jul  3 08:37:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 08:37:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a47d737.13be6df8.3d8759.03d3@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207342

>From 6695b3e0d8bddeab555e1ca36802df2decb85e82 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH 1/2] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 +++++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 24 +++++++++++++++++++++
 mlir/test/Target/LLVMIR/nvvmir.mlir         | 24 +++++++++++++++++++++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {
diff --git a/mlir/test/Target/LLVMIR/nvvmir.mlir b/mlir/test/Target/LLVMIR/nvvmir.mlir
index f2888025d8a08..f726efe3d39b7 100644
--- a/mlir/test/Target/LLVMIR/nvvmir.mlir
+++ b/mlir/test/Target/LLVMIR/nvvmir.mlir
@@ -293,6 +293,30 @@ llvm.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // f32 return type, f32 accumulate type
 // CHECK-LABEL: @nvvm_mma_m16n8k16_f32_f32
 llvm.func @nvvm_mma_m16n8k16_f32_f32(%a0 : vector<2xf16>, %a1 : vector<2xf16>,

>From eacc2df90b292589350c3fcdfec6327a54b273ff Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:43:32 +0000
Subject: [PATCH 2/2] [NVGPU] Add FP8 (e4m3/e5m2) support to nvgpu.mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..198fd958693fb 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) ||
+             aType.isF8E4M3FN() || aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Fri Jul  3 09:04:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 09:04:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a47dd85.2020f6c2.1fdf61.5d47@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 09:07:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 09:07:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a47de26.bf38ae28.358429.2d36@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul  3 09:40:47 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Fri, 03 Jul 2026 09:40:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a47e60f.7d41f3a1.d1cf7.542e@mx.google.com>


https://github.com/javedabsar1 updated https://github.com/llvm/llvm-project/pull/207403

>From 2511e24db6d65443df39400795bccd9f41de9535 Mon Sep 17 00:00:00 2001
From: mabsar <mabsar at qti.qualcommm.com>
Date: Fri, 3 Jul 2026 07:20:15 -0700
Subject: [PATCH] [mlir][bufferization] Add best-fit algorithm to static memory
 planner

Introduces an algorithm selection option to the static memory planner pass
and add a best-fit algorithm that reuses memory from expired allocations
by finding the smallest suitable gap. The planning algorithms are factored
into a separate StaticMemoryPlanning.{h,cpp} to keep them independent of
MLIR IR and easily testable.

Signed-off-by: mabsar <mabsar at qti.qualcommm.com>
---
 .../Bufferization/IR/BufferizationEnums.td    |   8 +
 .../Bufferization/Transforms/Passes.td        |  31 +-
 .../Transforms/StaticMemoryPlanning.h         |  50 +++
 .../Bufferization/Transforms/CMakeLists.txt   |   1 +
 .../StaticMemoryPlannerAnalysis.cpp           | 345 +++++++++---------
 .../Transforms/StaticMemoryPlanning.cpp       | 109 ++++++
 .../static-memory-planner-best-fit.mlir       | 118 ++++++
 7 files changed, 483 insertions(+), 179 deletions(-)
 create mode 100644 mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
 create mode 100644 mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..892df0787fb5d 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,36 +68,135 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
+/// Build lifetime-annotated allocation descriptors from candidates.
+/// Returns the arena alignment (LCM of all individual alignments).
+static int64_t
+buildAllocInfos(MutableArrayRef<AllocationCandidate> candidates,
+                SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  int64_t arenaAlignment = 1;
+  for (auto &candidate : candidates) {
+    bufferization::MemoryPlannerAlloc info;
+    info.sizeInBytes = candidate.sizeInBytes;
+    info.alignment = candidate.alignment;
+
+    Block *block = candidate.alloc->getBlock();
+    int64_t opIdx = 0;
+    for (Operation &op : *block) {
+      if (&op == candidate.alloc.getOperation())
+        info.timeStart = opIdx;
+      if (&op == candidate.dealloc.getOperation())
+        info.timeEnd = opIdx;
+      ++opIdx;
+    }
+
+    allocInfos.push_back(info);
+    arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
+  }
+  return arenaAlignment;
 }
 
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
+/// Collect alloc/dealloc pairs eligible for arena placement.
+/// An allocation is eligible if it has a static shape and a unique dealloc
+/// in the same block.
+static SmallVector<AllocationCandidate>
+collectCandidates(FunctionOpInterface funcOp,
+                  llvm::Statistic &numSkipDynamic,
+                  llvm::Statistic &numSkipNoDealloc,
+                  llvm::Statistic &numEligible) {
+  SmallVector<AllocationCandidate> candidates;
+
+  funcOp->walk([&](memref::AllocOp allocOp) {
+    MemRefType memrefType = allocOp.getType();
+
+    // Skip dynamic shapes
+    if (!memrefType.hasStaticShape()) {
+      ++numSkipDynamic;
+      return;
+    }
 
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
+    // Find unique dealloc in the same block
+    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
+    if (!deallocOp) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    if (deallocOp->getBlock() != allocOp->getBlock()) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    // This allocation is eligible
+    ++numEligible;
+    AllocationCandidate candidate;
+    candidate.alloc = allocOp;
+    candidate.dealloc = deallocOp;
+    candidate.sizeInBytes = computeSizeInBytes(memrefType);
+    candidate.alignment = allocOp.getAlignment().value_or(1);
+    candidates.push_back(candidate);
+  });
+
+  return candidates;
+}
+
+/// Create or obtain the arena buffer based on the arena mode.
+/// Returns failure if the mode is invalid or preconditions aren't met.
+static FailureOr<Value> createArena(OpBuilder &builder,
+                                    FunctionOpInterface funcOp,
+                                    StringRef arenaMode, int64_t totalSize,
+                                    int64_t arenaAlignment) {
+  Location loc = funcOp->getLoc();
+
+  if (arenaMode == "allocate") {
+    auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
+    auto arenaAlloc = memref::AllocOp::create(
+        builder, loc, arenaType, ValueRange{},
+        builder.getI64IntegerAttr(arenaAlignment));
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] created arena via AllocOp: size="
+               << totalSize << " bytes, alignment=" << arenaAlignment
+               << " bytes\n");
+    return arenaAlloc.getResult();
+  }
+
+  if (arenaMode == "arg") {
+    if (funcOp.getNumArguments() == 0)
+      return funcOp->emitError(
+          "arena-mode=arg requires at least one function argument");
+
+    Value arenaValue = funcOp.getArgument(0);
+    auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
+    if (!arenaType || !arenaType.getElementType().isInteger(8) ||
+        arenaType.getRank() != 1)
+      return funcOp->emitError(
+          "arena-mode=arg requires first argument to be memref<...xi8>");
+
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] using arena from function arg 0\n");
+    return arenaValue;
   }
 
-  return offsets;
+  return funcOp->emitError("invalid arena-mode: '" + arenaMode +
+                           "' (must be 'allocate' or 'arg')");
+}
+
+/// Replace each alloc/dealloc pair with a memref.view into the arena.
+static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
+                               Value arenaValue) {
+  for (auto &candidate : candidates) {
+    OpBuilder builder(candidate.alloc);
+    Location loc = candidate.alloc.getLoc();
+    MemRefType originalType = candidate.alloc.getType();
+
+    Value offsetIndex =
+        arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+    auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
+                                       offsetIndex, SmallVector<Value>{});
+
+    candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
+    candidate.alloc.erase();
+    candidate.dealloc.erase();
+  }
 }
 
 //===----------------------------------------------------------------------===//
@@ -125,143 +211,64 @@ struct StaticMemoryPlannerAnalysisPass
       StaticMemoryPlannerAnalysisPass>;
   using Base::Base;
 
-  void runOnOperation() override {
-    auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
-
-    // Step 0: Check for memref return types (not supported)
-    for (Type resultType : funcOp.getResultTypes()) {
-      if (isa<BaseMemRefType>(resultType)) {
-        funcOp->emitError("static-memory-planner does not support functions "
-                          "with memref return types");
-        return signalPassFailure();
-      }
-    }
-
-    // Step 1: Collect eligible allocation candidates
-    SmallVector<AllocationCandidate> candidates;
-
-    funcOp->walk([&](memref::AllocOp allocOp) {
-      MemRefType memrefType = allocOp.getType();
-
-      // Skip dynamic shapes
-      if (!memrefType.hasStaticShape()) {
-        ++numSkipDynamic;
-        return;
-      }
-
-      // Find unique dealloc in the same block
-      memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-      if (!deallocOp) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      if (deallocOp->getBlock() != allocOp->getBlock()) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      // This allocation is eligible
-      ++numEligible;
-      AllocationCandidate candidate;
-      candidate.alloc = allocOp;
-      candidate.dealloc = deallocOp;
-      candidate.sizeInBytes = computeSizeInBytes(memrefType);
-      candidate.alignment = allocOp.getAlignment().value_or(1);
-      candidates.push_back(candidate);
-    });
-
-    if (candidates.empty())
-      return;
-
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
-    int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
-      arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
-    }
+  void runOnOperation() override;
+};
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
-
-    // Assign computed offsets back to candidates
-    int64_t totalSize = 0;
-    for (size_t i = 0; i < candidates.size(); ++i) {
-      candidates[i].offset = offsets[i];
-      totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] offset=" << candidates[i].offset
-                 << " size=" << candidates[i].sizeInBytes
-                 << " alignment=" << candidates[i].alignment << "\n");
-    }
+void StaticMemoryPlannerAnalysisPass::runOnOperation() {
+  auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
 
-    // Step 4: Obtain arena based on arena mode
-    Operation *firstAlloc = candidates.front().alloc;
-    OpBuilder builder(firstAlloc);
-    Value arenaValue;
-
-    if (arenaMode == "allocate") {
-      // Arena is i8 byte buffer to support multiple data types (f32, i64, etc.)
-      auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
-      auto arenaAlloc = memref::AllocOp::create(
-          builder, firstAlloc->getLoc(), arenaType, ValueRange{},
-          builder.getI64IntegerAttr(arenaAlignment));
-      arenaValue = arenaAlloc.getResult();
-
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] created arena via AllocOp: size="
-                 << totalSize << " bytes, alignment=" << arenaAlignment
-                 << " bytes\n");
-    } else if (arenaMode == "arg") {
-      if (funcOp.getNumArguments() == 0) {
-        funcOp->emitError(
-            "arena-mode=arg requires at least one function argument");
-        return signalPassFailure();
-      }
-
-      arenaValue = funcOp.getArgument(0);
-      auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
-      if (!arenaType || !arenaType.getElementType().isInteger(8) ||
-          arenaType.getRank() != 1) {
-        funcOp->emitError(
-            "arena-mode=arg requires first argument to be memref<...xi8>");
-        return signalPassFailure();
-      }
-
-      LLVM_DEBUG(
-          llvm::dbgs()
-          << "[static-memory-planner] using arena from function arg 0\n");
-    } else {
-      funcOp->emitError("invalid arena-mode: '" + arenaMode +
-                        "' (must be 'allocate' or 'arg')");
+  // Step 0: Check for memref return types (not supported)
+  for (Type resultType : funcOp.getResultTypes()) {
+    if (isa<BaseMemRefType>(resultType)) {
+      funcOp->emitError("static-memory-planner does not support functions "
+                        "with memref return types");
       return signalPassFailure();
     }
+  }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
-    for (auto &candidate : candidates) {
-      builder.setInsertionPoint(candidate.alloc);
-      Location loc = candidate.alloc.getLoc();
+  // Step 1: Collect eligible allocation candidates.
+  SmallVector<AllocationCandidate> candidates =
+      collectCandidates(funcOp, numSkipDynamic, numSkipNoDealloc, numEligible);
 
-      MemRefType originalType = candidate.alloc.getType();
+  if (candidates.empty())
+    return;
 
-      // Create a constant for the byte offset into the arena
-      Value offsetIndex =
-          arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+  // Step 2: Build allocation descriptors with lifetime info.
+  SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
+  int64_t arenaAlignment = buildAllocInfos(candidates, allocInfos);
 
-      // Use memref.view to create a typed view into the i8 arena
-      auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
-                                         offsetIndex, SmallVector<Value>{});
+  // Step 3: Run the planning algorithm.
+  SmallVector<int64_t> offsets;
+  switch (algorithm) {
+  case bufferization::MemoryPlannerAlgorithm::Trivial:
+    offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  case bufferization::MemoryPlannerAlgorithm::BestFit:
+    offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  }
 
-      candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-      candidate.alloc.erase();
-      candidate.dealloc.erase();
-    }
+  // Step 4: Compute total arena size and assign offsets.
+  int64_t totalSize = 0;
+  for (size_t i = 0; i < candidates.size(); ++i) {
+    candidates[i].offset = offsets[i];
+    totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] offset=" << candidates[i].offset
+               << " size=" << candidates[i].sizeInBytes
+               << " alignment=" << candidates[i].alignment << "\n");
   }
-};
+
+  // Step 5: Obtain arena based on arena mode.
+  Operation *firstAlloc = candidates.front().alloc;
+  OpBuilder builder(firstAlloc);
+  FailureOr<Value> arenaValue =
+      createArena(builder, funcOp, arenaMode, totalSize, arenaAlignment);
+  if (failed(arenaValue))
+    return signalPassFailure();
+
+  // Step 6: Replace each alloc with memref.view into the arena.
+  rewriteAllocations(candidates, *arenaValue);
+}
 
 } // end anonymous namespace
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..eadb1307d1b21
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // C at offset 5120
+  // CHECK-NEXT: %{{.*}} = arith.constant 5120 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // D at offset 9216
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // E at offset 9216 (best-fit picks 1024-byte trailing gap over 5120-byte gap)
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<256xf32>
+  %c = memref.alloc() : memref<1024xf32>
+  %d = memref.alloc() : memref<256xf32>
+  memref.dealloc %b : memref<256xf32>
+  memref.dealloc %d : memref<256xf32>
+  memref.dealloc %a : memref<1024xf32>
+  %e = memref.alloc() : memref<128xf32>
+  memref.dealloc %c : memref<1024xf32>
+  memref.dealloc %e : memref<128xf32>
+  return
+}
+
+// -----
+
+// Test 4: Alignment padding can disqualify a gap.
+// A(128, align 128) at 0 (live), B(56, align 1) at 128, C(128, align 128) at 256 (live).
+// B is freed => gap [128, 256) = 128 bytes.
+// D(64, align 128): aligned start = 128, fits in gap.
+// E(64, align 256): next 256-aligned offset in [128,256) is 256 = gap end, doesn't fit.
+//   E must go past the arena high-water mark.
+// CHECK-LABEL: func @best_fit_alignment
+func.func @best_fit_alignment() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 256 : i64} : memref<576xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<56xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 256 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // D fits in gap at offset 128
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  // E cannot fit in gap (alignment 256), placed at offset 512
+  // CHECK-NEXT: %{{.*}} = arith.constant 512 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  %a = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  %b = memref.alloc() : memref<56xi8>
+  %c = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  memref.dealloc %b : memref<56xi8>
+  %d = memref.alloc() {alignment = 128 : i64} : memref<64xi8>
+  %e = memref.alloc() {alignment = 256 : i64} : memref<64xi8>
+  memref.dealloc %a : memref<128xi8>
+  memref.dealloc %c : memref<128xi8>
+  memref.dealloc %d : memref<64xi8>
+  memref.dealloc %e : memref<64xi8>
+  return
+}


From llvmlistbot at llvm.org  Fri Jul  3 09:46:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Fri, 03 Jul 2026 09:46:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47e76c.eea6ab6e.1256b6.55c5@mx.google.com>


https://github.com/sahas3 approved this pull request.

LGTM!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 09:46:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Fri, 03 Jul 2026 09:46:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47e76c.3157459c.24b6e5.4492@mx.google.com>


https://github.com/sahas3 edited https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 09:46:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Fri, 03 Jul 2026 09:46:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47e76c.ac469dc7.e3aa6.890a@mx.google.com>


================

----------------
sahas3 wrote:

nit
```suggestion
          castOp, "avoid cancelling casts that should be lossy");
```

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 09:46:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Fri, 03 Jul 2026 09:46:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47e76c.f06f4994.168928.7bc9@mx.google.com>


================

----------------
sahas3 wrote:

Is my understanding correct that from the RFC these variants will be deprecated in favor of the new ones added here?

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Fri Jul  3 10:13:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 10:13:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a47edd3.6c8668f2.346e76.9e97@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From bb179a3054918920bf0f77f7d8b141d176e630de Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH 1/2] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload

>From 8ce3d9fdeae633ea27cca1ab59c1625d3ca0e465 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 18:52:14 +0200
Subject: [PATCH 2/2] [mlir][vector] Fix unaligned-offset miscompiles in
 narrow-type emulation

ConvertVectorLoad, ConvertVectorMaskedLoad, and ConvertVectorTransferRead
treated a vector size divisble by the container size as proof that the access
starts on a container boundary, and ConvertVectorMaskedStore never looked
at the offset at all. Any size-divisible access at an unaligned offset
(e.g. vector<8xi4> at index 1) was silently read or written one sub-element early.

Always fold the real intra-container offset: loads reuse the existing static/dynamic
extraction paths, and maskedstore bails out unless the offset provably folds to zero,
the same contract I saw in #189235 for vector.store.
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  54 +--
 ...atten-memref-and-emulate-narrow-types.mlir |  11 +-
 ...e-narrow-type-unaligned-dynamic-store.mlir |  49 +++
 .../vector-emulate-narrow-type-unaligned.mlir |  70 ++++
 .../Vector/vector-emulate-narrow-type.mlir    | 312 ++++++++++++------
 5 files changed, 369 insertions(+), 127 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index e37296e50fb29..54c79ebce7289 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -910,6 +910,18 @@ struct ConvertVectorMaskedStore final
     Value linearizedIndices =
         getValueOrCreateConstantIndexOp(rewriter, loc, linearizedIndicesOfr);
 
+    // The load-select-store sequence below assumes the store starts at a
+    // container element boundary. A non-zero intra-container offset would
+    // require shifting both the mask and the value to store; that is not
+    // implemented yet, so bail out to avoid storing at the wrong offset.
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply
+    // such alignment - the offset depends on the (possibly dynamic) indices.
+    std::optional<int64_t> intraDataOffset =
+        getConstantIntValue(linearizedInfo.intraDataOffset);
+    if (!intraDataOffset || *intraDataOffset != 0)
+      return rewriter.notifyMatchFailure(
+          op, "unaligned or dynamically-offset maskedstore is not supported");
+
     // Load the whole data and use arith.select to handle the corner cases.
     //
     // As an example, for this masked store of i4 values:
@@ -1042,8 +1054,6 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
     // compile time as they must be constants.
 
     auto origElements = op.getVectorType().getNumElements();
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize = origElements % emulatedPerContainerElem == 0;
 
     auto stridedMetadata =
         memref::ExtractStridedMetadataOp::create(rewriter, loc, op.getBase());
@@ -1058,9 +1068,11 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // load starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     // Always load enough elements which can cover the original elements.
     int64_t maxintraDataOffset =
@@ -1077,7 +1089,9 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
       result = dynamicallyExtractSubVector(
           rewriter, loc, dyn_cast<TypedValue<VectorType>>(result), resultVector,
           linearizedInfo.intraDataOffset, origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // This is a no-op if the offset is zero and all loaded elements are
+      // used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
@@ -1167,8 +1181,6 @@ struct ConvertVectorMaskedLoad final
     // subvector at the proper offset after bit-casting.
     auto origType = op.getVectorType();
     auto origElements = origType.getNumElements();
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize = origElements % emulatedPerContainerElem == 0;
 
     auto stridedMetadata =
         memref::ExtractStridedMetadataOp::create(rewriter, loc, op.getBase());
@@ -1182,9 +1194,11 @@ struct ConvertVectorMaskedLoad final
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // load starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     int64_t maxIntraDataOffset =
         foldedIntraVectorOffset.value_or(emulatedPerContainerElem - 1);
@@ -1208,7 +1222,8 @@ struct ConvertVectorMaskedLoad final
       passthru = dynamicallyInsertSubVector(
           rewriter, loc, passthru, emptyVector, linearizedInfo.intraDataOffset,
           origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and the sizes match.
       passthru = staticallyInsertSubvector(rewriter, loc, passthru, emptyVector,
                                            *foldedIntraVectorOffset);
     }
@@ -1237,7 +1252,8 @@ struct ConvertVectorMaskedLoad final
       mask = dynamicallyInsertSubVector(rewriter, loc, mask, emptyMask,
                                         linearizedInfo.intraDataOffset,
                                         origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and the sizes match.
       mask = staticallyInsertSubvector(rewriter, loc, op.getMask(), emptyMask,
                                        *foldedIntraVectorOffset);
     }
@@ -1248,7 +1264,8 @@ struct ConvertVectorMaskedLoad final
       result = dynamicallyExtractSubVector(
           rewriter, loc, result, op.getPassThru(),
           linearizedInfo.intraDataOffset, origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and all loaded elements are used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
@@ -1325,10 +1342,6 @@ struct ConvertVectorTransferRead final
 
     auto origElements = op.getVectorType().getNumElements();
 
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize =
-        fitsInMultiByteContainerTy(op.getVectorType(), containerElemTy);
-
     // Pad the padding value with 0s on the left. These bits are discarded and
     // thus their values don't matter.
     Value padding = adaptor.getPadding();
@@ -1355,9 +1368,11 @@ struct ConvertVectorTransferRead final
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // read starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     int64_t maxIntraDataOffset =
         foldedIntraVectorOffset.value_or(emulatedPerContainerElem - 1);
@@ -1382,7 +1397,8 @@ struct ConvertVectorTransferRead final
       result = dynamicallyExtractSubVector(rewriter, loc, bitCast, zeros,
                                            linearizedInfo.intraDataOffset,
                                            origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and all read elements are used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
diff --git a/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir b/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
index ef253624fe0b4..38f4d8b0a6ff0 100644
--- a/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
+++ b/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
@@ -31,10 +31,14 @@ func.func @vector_maskedload_2d_i4(%arg0: index, %passthru: vector<8xi4>) -> vec
 
 // -----
 
+// Note: the column index must be a constant so that the intra-byte offset of
+// the store provably folds to 0 (a maskedstore at an offset that cannot be
+// proven to be byte-aligned cannot be emulated and would fail to legalize).
 func.func @vector_maskedstore_2d_i4(%arg0: index, %value: vector<8xi4>) {
     %0 = memref.alloc() : memref<4x8xi4>
+    %c0 = arith.constant 0 : index
     %mask = vector.constant_mask [5] : vector<8xi1>
-    vector.maskedstore %0[%arg0, %arg0], %mask, %value :
+    vector.maskedstore %0[%arg0, %c0], %mask, %value :
       memref<4x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
@@ -54,9 +58,12 @@ func.func @vector_store_2d_i4(%arg0: index, %value: vector<8xi4>) {
 
 // -----
 
+// Note: the column index is a constant so that the intra-byte offset of the
+// read provably folds to 0 and no extra byte needs to be read.
 func.func @vector_transfer_read_2d_i4(%arg0: index, %padding: i4) -> vector<8xi4> {
     %0 = memref.alloc() : memref<4x8xi4>
-    %1 = vector.transfer_read %0[%arg0, %arg0], %padding {in_bounds = [true]} : memref<4x8xi4>, vector<8xi4>
+    %c0 = arith.constant 0 : index
+    %1 = vector.transfer_read %0[%arg0, %c0], %padding {in_bounds = [true]} : memref<4x8xi4>, vector<8xi4>
     return %1 : vector<8xi4>
 }
 //  CHECK-LABEL: func @vector_transfer_read_2d_i4(
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
index 518d825234502..ba2293c6321e2 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
@@ -20,3 +20,52 @@ func.func @vector_store_i4_dynamic_memref(%arg0: vector<8xi4>, %dim0: index,
   vector.store %arg0, %src[%idx0, %idx1] : memref<?x?xi4>, vector<8xi4>
   return
 }
+
+// -----
+
+// Dynamic sub-byte vector.maskedstore offsets cannot be treated as
+// byte-aligned either: the store could start in the middle of a byte and the
+// emulation (which read-modify-writes whole bytes) would write to the wrong
+// bits.
+
+func.func @vector_maskedstore_i4_dynamic_col(%idx1: index, %idx2: index,
+                                             %num_elements_to_store: index,
+                                             %value: vector<8xi4>) {
+  %0 = memref.alloc() : memref<3x8xi4>
+  %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    memref<3x8xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
+
+// -----
+
+func.func @vector_maskedstore_i4_constant_mask_dynamic_col(%idx1: index,
+                                                           %idx2: index,
+                                                           %value: vector<8xi4>) {
+  %0 = memref.alloc() : memref<3x8xi4>
+  %mask = vector.constant_mask [4] : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    memref<3x8xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
+
+// -----
+
+// A maskedstore at a static but non-container-aligned offset (an odd i4 index
+// gives intra-byte offset 1) must also be rejected - emulating it with the
+// compressed mask would silently store at the wrong bits (the intra-byte
+// offset used to be dropped; see the discussion in llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i4_unaligned_static_offset(%num_elements_to_store: index,
+                                                         %value: vector<8xi4>) {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%c1], %mask, %value :
+    memref<9xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
index bec7736b90973..60ceecf88ddaf 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
@@ -586,3 +586,73 @@ func.func @vector_store_i2_2d_const_nonzero_col(%arg0: vector<4xi2>) {
 // and one for byte 1 (element at position [0]).
 // CHECK: memref.generic_atomic_rmw %[[ALLOC]][%[[C0]]]
 // CHECK: memref.generic_atomic_rmw %[[ALLOC]][{{.+}}]
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// Size-divisible vectors at non-container-aligned static offsets.
+///
+/// A vector length divisible by the number of emulated elements per container
+/// element does NOT imply the access starts at a container element boundary.
+/// These used to be emulated as if aligned, silently reading shifted data
+/// (see the discussion in llvm/llvm-project#206928).
+///----------------------------------------------------------------------------------------
+
+func.func @vector_load_i4_unaligned_static_offset() -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %1 = vector.load %0[%c1] : memref<9xi4>, vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// CHECK: func @vector_load_i4_unaligned_static_offset(
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[LOAD:.+]] = vector.load %[[ALLOC]][%[[C0]]] : memref<5xi8>, vector<5xi8>
+// CHECK: %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[BITCAST]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
+
+// -----
+
+func.func @vector_maskedload_i4_unaligned_static_offset(%num_elems: index, %passthru: vector<8xi4>) -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %mask = vector.create_mask %num_elems : vector<8xi1>
+  %1 = vector.maskedload %0[%c1], %mask, %passthru : memref<9xi4>, vector<8xi1>, vector<8xi4> into vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// The compressed mask accounts for the front offset of 1: ceil((n + 1) / 2).
+// CHECK: #[[MASK_MAP:.+]] = affine_map<()[s0] -> ((s0 + 1) ceildiv 2)>
+// CHECK: func @vector_maskedload_i4_unaligned_static_offset(
+// CHECK-SAME:    %[[NUM_ELEMS:[a-zA-Z0-9]+]]: index, %[[PASSTHRU:[a-zA-Z0-9]+]]: vector<8xi4>)
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_ELEMS]] : vector<8xi1>
+// CHECK: %[[MASK_IDX:.+]] = affine.apply #[[MASK_MAP]]()[%[[NUM_ELEMS]]]
+// CHECK: %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<5xi1>
+// CHECK: %[[NEW_PASSTHRU:.+]] = vector.insert_strided_slice %[[PASSTHRU]], %{{.+}} {offsets = [1], strides = [1]} : vector<8xi4> into vector<10xi4>
+// CHECK: %[[PASSTHRU_BITCAST:.+]] = vector.bitcast %[[NEW_PASSTHRU]] : vector<10xi4> to vector<5xi8>
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[C0]]], %[[NEW_MASK]], %[[PASSTHRU_BITCAST]] : memref<5xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+// CHECK: %[[LOAD_BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SHIFTED_MASK:.+]] = vector.insert_strided_slice %[[ORIG_MASK]], %{{.+}} {offsets = [1], strides = [1]} : vector<8xi1> into vector<10xi1>
+// CHECK: %[[SELECT:.+]] = arith.select %[[SHIFTED_MASK]], %[[LOAD_BITCAST]], %[[NEW_PASSTHRU]] : vector<10xi1>, vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[SELECT]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
+
+// -----
+
+func.func @vector_transfer_read_i4_unaligned_static_offset(%padding: i4) -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %1 = vector.transfer_read %0[%c1], %padding {in_bounds = [true]} : memref<9xi4>, vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// CHECK: func @vector_transfer_read_i4_unaligned_static_offset(
+// CHECK-SAME:    %[[PADDING:[a-zA-Z0-9]+]]: i4)
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[PAD_I8:.+]] = arith.extui %[[PADDING]] : i4 to i8
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[READ:.+]] = vector.transfer_read %[[ALLOC]][%[[C0]]], %[[PAD_I8]] : memref<5xi8>, vector<5xi8>
+// CHECK: %[[BITCAST:.+]] = vector.bitcast %[[READ]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[BITCAST]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
index 72b355e0fed65..af31ac62f859e 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
@@ -17,14 +17,20 @@ func.func @vector_load_i8(%arg1: index, %arg2: index) -> vector<4xi8> {
 // CHECK-NEXT:   [[L:%.+]] = vector.load %[[ALLOC]][%[[ARG0]], %[[ARG1]]] : memref<3x4xi8>, vector<4xi8>
 // CHECK-NEXT:   return
 
-//      CHECK32: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
+//  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 4)>
 //      CHECK32: func @vector_load_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VECLOAD:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VECLOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   return %[[VEC_I4]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VECLOAD:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I8:.+]] = vector.bitcast %[[VECLOAD]] : vector<2xi32> to vector<8xi8>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I8]][%[[FRONT]]] : i8 from vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -35,21 +41,32 @@ func.func @vector_load_i4(%arg1: index, %arg2: index) -> vector<3x8xi4> {
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_load_i4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_load_i4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -60,21 +77,32 @@ func.func @vector_load_f4(%arg1: index, %arg2: index) -> vector<3x8xf4E2M1FN> {
     %2 = vector.insert %1, %cst [0] : vector<8xf4E2M1FN> into vector<3x8xf4E2M1FN>
     return %2 : vector<3x8xf4E2M1FN>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_load_f4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xf4E2M1FN>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xf4E2M1FN>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<10xf4E2M1FN>
+// CHECK-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_load_f4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xf4E2M1FN>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xf4E2M1FN>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<16xf4E2M1FN>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 // -----
 
@@ -83,8 +111,11 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
   %1 = vector.load %0[%arg2, %arg3] : memref<?x?xi4>, vector<8xi4>
   return %1 : vector<8xi4>
 }
+// The intra-container offset of the load is dynamic, so one extra container
+// element is loaded and the target elements are extracted dynamically.
 //  CHECK-DAG: #[[MAP0:.+]] = affine_map<()[s0, s1] -> ((s0 * s1) floordiv 2, s0 floordiv 2)>
 //  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0, s1, s2] -> ((s2 + s0 * s1) floordiv 2)>
+//  CHECK-DAG: #[[MAP2:.+]] = affine_map<()[s0, s1, s2] -> (s0 * s1 + s2 - ((s2 + s0 * s1) floordiv 2) * 2)>
 //      CHECK: func.func @vector_load_i4_dynamic(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: index
 // CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: index
@@ -93,11 +124,15 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
 //      CHECK:   %[[SIZE:.+]] = affine.max #[[MAP0]]()[%[[ARG1]], %[[ARG0]]]
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc(%[[SIZE]]) : memref<?xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP1]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP2]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP0:.+]] = affine_map<()[s0, s1] -> ((s0 * s1) floordiv 8, s0 floordiv 8)>
 //  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0, s1, s2] -> ((s2 + s0 * s1) floordiv 8)>
+//  CHECK32-DAG: #[[MAP2:.+]] = affine_map<()[s0, s1, s2] -> (s0 * s1 + s2 - ((s2 + s0 * s1) floordiv 8) * 8)>
 //      CHECK32: func.func @vector_load_i4_dynamic(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: index
 // CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: index
@@ -106,8 +141,11 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
 //      CHECK32:   %[[SIZE:.+]] = affine.max #[[MAP0]]()[%[[ARG1]], %[[ARG0]]]
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc(%[[SIZE]]) : memref<?xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP1]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP2]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -122,25 +160,36 @@ func.func @vector_transfer_read_i4(%arg1: index, %arg2: index) -> vector<8xi4> {
       memref<3x8xi4>, vector<8xi4>
     return %1 : vector<8xi4>
 }
+// The intra-container offset of the read is dynamic (it depends on %arg2), so
+// one extra container element is read and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_transfer_read_i4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[CONST:.+]] = arith.constant 0 : i4
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[PAD:.+]] = arith.extui %[[CONST]] : i4 to i8
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_transfer_read_i4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[CONST:.+]] = arith.constant 0 : i4
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[PAD:.+]] = arith.extui %[[CONST]] : i4 to i32
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -151,7 +200,11 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
       memref<3x8xf4E2M1FN>, vector<8xf4E2M1FN>
     return %1 : vector<8xf4E2M1FN>
 }
+// The intra-container offset of the read is dynamic (it depends on %arg2), so
+// one extra container element is read and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_transfer_read_f4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[CONST:.+]] = arith.constant 0.{{0+}}e+00 : f4E2M1FN
@@ -159,10 +212,14 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
 //      CHECK:   %[[BC:.+]] = arith.bitcast %[[CONST]] : f4E2M1FN to i4
 //      CHECK:   %[[PAD:.+]] = arith.extui %[[BC]] : i4 to i8
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xf4E2M1FN>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xf4E2M1FN>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<10xf4E2M1FN>
+// CHECK-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_transfer_read_f4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[CONST:.+]] = arith.constant 0.{{0+}}e+00 : f4E2M1FN
@@ -170,8 +227,11 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
 //      CHECK32:   %[[BC:.+]] = arith.bitcast %[[CONST]] : f4E2M1FN to i4
 //      CHECK32:   %[[PAD:.+]] = arith.extui %[[BC]] : i4 to i32
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xf4E2M1FN>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xf4E2M1FN>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<16xf4E2M1FN>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 // -----
 
@@ -196,22 +256,29 @@ func.func @vector_maskedload_i8(%arg1: index, %arg2: index, %arg3: index, %passt
 // CHECK-SAME:     memref<3x4xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
 // CHECK-NEXT:   return
 
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded; the mask, the passthru and the result
+// are shifted dynamically by the front offset.
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
-//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 4)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 4)>
+//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 3) ceildiv 4)>
 //      CHECK32: func @vector_maskedload_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<4xi8>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<4xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK32:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<4xi8> to vector<1xi32>
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<8xi8>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<8xi8> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<4xi1>, vector<4xi8>
-//      CHECK32:   return %[[SELECT]]
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i1 into vector<8xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<8xi1>, vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -224,37 +291,50 @@ func.func @vector_maskedload_i4(%arg1: index, %arg2: index, %arg3: index, %passt
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded; the mask, the passthru and the result
+// are shifted dynamically by the front offset.
 //  CHECK-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
-//  CHECK-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 2)>
+//  CHECK-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 2)>
+//  CHECK-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 1) ceildiv 2)>
 //      CHECK: func @vector_maskedload_i4(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
 //      CHECK:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<4xi1>
-//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<8xi4> to vector<4xi8>
+//      CHECK:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<5xi1>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<10xi4>
+//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<10xi4> to vector<5xi8>
 //      CHECK:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK-SAME:     memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
-//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<4xi8> to vector<8xi4>
-//      CHECK:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<8xi1>, vector<8xi4>
+// CHECK-SAME:     memref<12xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i1 into vector<10xi1>
+//      CHECK:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<10xi1>, vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
-//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 8)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 8)>
+//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 7) ceildiv 8)>
 //      CHECK32: func @vector_maskedload_i4(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK32:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<8xi4> to vector<1xi32>
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<16xi4>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<16xi4> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<8xi4>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<8xi1>, vector<8xi4>
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i1 into vector<16xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<16xi1>, vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -275,20 +355,28 @@ func.func @vector_maskedload_i8_constant_mask(%arg1: index, %arg2: index, %passt
 // CHECK-SAME:     memref<3x4xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
 // CHECK-NEXT:   return
 
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded (with a pessimistic compressed mask);
+// the mask, the passthru and the result are shifted dynamically by the front
+// offset.
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 4)>
 //      CHECK32: func @vector_maskedload_i8_constant_mask(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG3:[a-zA-Z0-9]+]]: vector<4xi8>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<4xi8> to vector<1xi32>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<2xi1>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<8xi8>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<8xi8> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<4xi1>, vector<4xi8>
-//      CHECK32:   return %[[SELECT]]
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i1 into vector<8xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<8xi1>, vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -301,33 +389,47 @@ func.func @vector_maskedload_i4_constant_mask(%arg1: index, %arg2: index, %passt
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded (with a pessimistic compressed mask);
+// the mask, the passthru and the result are shifted dynamically by the front
+// offset.
 //  CHECK-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_maskedload_i4_constant_mask(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
 //      CHECK:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
-//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG2]] : vector<8xi4> to vector<4xi8>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK:   %[[NEW_MASK:.+]] = vector.constant_mask [3] : vector<5xi1>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<10xi4>
+//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<10xi4> to vector<5xi8>
 //      CHECK:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK-SAME:     memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
-//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<4xi8> to vector<8xi4>
-//      CHECK:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG2]] : vector<8xi1>, vector<8xi4>
+// CHECK-SAME:     memref<12xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i1 into vector<10xi1>
+//      CHECK:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<10xi1>, vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_maskedload_i4_constant_mask(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG2]] : vector<8xi4> to vector<1xi32>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<2xi1>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<16xi4>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<16xi4> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<8xi4>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG2]] : vector<8xi1>, vector<8xi4>
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i1 into vector<16xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<16xi1>, vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -533,34 +635,36 @@ func.func @vector_store_f4(%arg0: vector<8xf4E2M1FN>, %arg1: index, %arg2: index
 /// vector.maskedstore
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedstore_i8(%arg0: index, %arg1: index, %arg2: index, %value: vector<8xi8>) {
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
+func.func @vector_maskedstore_i8(%arg0: index, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
   %0 = memref.alloc() : memref<3x8xi8>
-  %mask = vector.create_mask %arg2 : vector<8xi1>
-  vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
+  %mask = vector.create_mask %arg0 : vector<8xi1>
+  vector.maskedstore %0[%c1, %c0], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 // Expect no conversions, i8 is supported.
 //      CHECK: func @vector_maskedstore_i8(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]
 // CHECK-SAME:     %[[VAL:[a-zA-Z0-9]+]]
-// CHECK-NEXT:   %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
-// CHECK-NEXT:   %[[MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
-// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[ARG0]], %[[ARG1]]], %[[MASK]], %[[VAL]]
+// CHECK-DAG:    %[[C0:.+]] = arith.constant 0 : index
+// CHECK-DAG:    %[[C1:.+]] = arith.constant 1 : index
+// CHECK-DAG:    %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
+// CHECK:        %[[MASK:.+]] = vector.create_mask %[[ARG0]] : vector<8xi1>
+// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[C1]], %[[C0]]], %[[MASK]], %[[VAL]]
 // CHECK-NEXT:   return
 
-// CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 2 + s1 floordiv 4)>
 // CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 4)>
 // CHECK32:     func @vector_maskedstore_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]
 // CHECK32-SAME:     %[[VAL:[a-zA-Z0-9]+]]
 // CHECK32:        %[[ALLOC:.+]] = memref.alloc() : memref<6xi32>
-// CHECK32:        %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
-// CHECK32:        %[[LIDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-// CHECK32:        %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
+// CHECK32:        %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG0]] : vector<8xi1>
+// CHECK32:        %[[LIDX:.+]] = arith.constant 2 : index
+// CHECK32:        %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG0]]]
 // CHECK32:        %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
 // CHECK32:        %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<2xi32>
 // CHECK32:        %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]]
@@ -571,29 +675,29 @@ func.func @vector_maskedstore_i8(%arg0: index, %arg1: index, %arg2: index, %valu
 
 // -----
 
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
 func.func @vector_maskedstore_i4(
-  %idx1: index,
-  %idx2: index,
   %num_elements_to_store: index,
   %value: vector<8xi4>) {
 
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
     %0 = memref.alloc() : memref<3x8xi4>
     %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
-    vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    vector.maskedstore %0[%c1, %c0], %mask, %value :
       memref<3x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
-// CHECK: #[[$ATTR_10:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
 // CHECK: #[[$ATTR_11:.+]] = affine_map<()[s0] -> (s0 ceildiv 2)>
 
 // CHECK:         func.func @vector_maskedstore_i4(
-// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[NUM_EL_TO_STORE:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK:           %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 // CHECK:           %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_EL_TO_STORE]] : vector<8xi1>
-// CHECK:           %[[LIDX:.+]] = affine.apply #[[$ATTR_10]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK:           %[[LIDX:.+]] = arith.constant 4 : index
 // CHECK:           %[[MASK_IDX:.+]] = affine.apply #[[$ATTR_11]]()[%[[NUM_EL_TO_STORE]]]
 // CHECK:           %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<4xi1>
 // CHECK:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<4xi8>
@@ -603,17 +707,14 @@ func.func @vector_maskedstore_i4(
 // CHECK:           %[[NEW_VAL:.+]] = vector.bitcast %[[SELECT]] : vector<8xi4> to vector<4xi8>
 // CHECK:           vector.maskedstore %[[ALLOC]]{{\[}}%[[LIDX]]], %[[NEW_MASK]], %[[NEW_VAL]] : memref<12xi8>, vector<4xi1>, vector<4xi8>
 
-// CHECK32: #[[$ATTR_17:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
 // CHECK32: #[[$ATTR_18:.+]] = affine_map<()[s0] -> (s0 ceildiv 8)>
 
 // CHECK32:         func.func @vector_maskedstore_i4(
-// CHECK32-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK32-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[NUM_EL_TO_STORE:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK32:           %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 // CHECK32:           %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_EL_TO_STORE]] : vector<8xi1>
-// CHECK32:           %[[LIDX:.+]] = affine.apply #[[$ATTR_17]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK32:           %[[LIDX:.+]] = arith.constant 1 : index
 // CHECK32:           %[[MASK_IDX:.+]] = affine.apply #[[$ATTR_18]]()[%[[NUM_EL_TO_STORE]]]
 // CHECK32:           %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
 // CHECK32:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<1xi32>
@@ -625,30 +726,32 @@ func.func @vector_maskedstore_i4(
 
 // -----
 
-func.func @vector_maskedstore_i8_constant_mask(%arg0: index, %arg1: index, %value: vector<8xi8>) {
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
+func.func @vector_maskedstore_i8_constant_mask(%value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
   %0 = memref.alloc() : memref<3x8xi8>
   %mask = vector.constant_mask [4] : vector<8xi1>
-  vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
+  vector.maskedstore %0[%c1, %c0], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 // Expect no conversions, i8 is supported.
 //      CHECK: func @vector_maskedstore_i8_constant_mask(
-// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
 // CHECK-SAME:     %[[VAL:[a-zA-Z0-9]+]]
-// CHECK-NEXT:   %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
-// CHECK-NEXT:   %[[MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[ARG0]], %[[ARG1]]], %[[MASK]], %[[VAL]]
+// CHECK-DAG:    %[[C0:.+]] = arith.constant 0 : index
+// CHECK-DAG:    %[[C1:.+]] = arith.constant 1 : index
+// CHECK-DAG:    %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
+// CHECK:        %[[MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
+// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[C1]], %[[C0]]], %[[MASK]], %[[VAL]]
 // CHECK-NEXT:   return
 
-// CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 2 + s1 floordiv 4)>
 // CHECK32:     func @vector_maskedstore_i8_constant_mask(
-// CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
 // CHECK32-SAME:     %[[VAL:[a-zA-Z0-9]+]]
 // CHECK32:        %[[ALLOC:.+]] = memref.alloc() : memref<6xi32>
 // CHECK32:        %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK32:        %[[LIDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+// CHECK32:        %[[LIDX:.+]] = arith.constant 2 : index
 // CHECK32:        %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<2xi1>
 // CHECK32:        %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<2xi32>
 // CHECK32:        %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]]
@@ -659,26 +762,26 @@ func.func @vector_maskedstore_i8_constant_mask(%arg0: index, %arg1: index, %valu
 
 // -----
 
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
 func.func @vector_maskedstore_i4_constant_mask(
-  %idx_1: index,
-  %idx_2: index,
   %val_to_store: vector<8xi4>) {
 
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
     %0 = memref.alloc() : memref<3x8xi4>
     %mask = vector.constant_mask [4] : vector<8xi1>
-    vector.maskedstore %0[%idx_1, %idx_2], %mask, %val_to_store :
+    vector.maskedstore %0[%c1, %c0], %mask, %val_to_store :
       memref<3x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
 
-// CHECK: #[[$ATTR_12:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
 // CHECK:         func.func @vector_maskedstore_i4_constant_mask(
-// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK:           %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 // CHECK:           %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK:           %[[LIDX:.+]] = affine.apply #[[$ATTR_12]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK:           %[[LIDX:.+]] = arith.constant 4 : index
 // CHECK:           %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
 // CHECK:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<4xi8>
 // CHECK:           %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]] : memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
@@ -687,14 +790,11 @@ func.func @vector_maskedstore_i4_constant_mask(
 // CHECK:           %[[NEW_VAL:.+]] = vector.bitcast %[[SELECT]] : vector<8xi4> to vector<4xi8>
 // CHECK:           vector.maskedstore %[[ALLOC]]{{\[}}%[[LIDX]]], %[[NEW_MASK]], %[[NEW_VAL]] : memref<12xi8>, vector<4xi1>, vector<4xi8>
 
-// CHECK32: #[[$ATTR_20:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
 // CHECK32:         func.func @vector_maskedstore_i4_constant_mask(
-// CHECK32-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK32-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK32:           %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 // CHECK32:           %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK32:           %[[LIDX:.+]] = affine.apply #[[$ATTR_20]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK32:           %[[LIDX:.+]] = arith.constant 1 : index
 // CHECK32:           %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
 // CHECK32:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<1xi32>
 // CHECK32:           %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]] : memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>


From llvmlistbot at llvm.org  Fri Jul  3 10:14:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 10:14:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a47edfe.3157459c.24b6e5.4f4c@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From bb179a3054918920bf0f77f7d8b141d176e630de Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH 1/2] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload

>From 8ce3d9fdeae633ea27cca1ab59c1625d3ca0e465 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 18:52:14 +0200
Subject: [PATCH 2/2] [mlir][vector] Fix unaligned-offset miscompiles in
 narrow-type emulation

ConvertVectorLoad, ConvertVectorMaskedLoad, and ConvertVectorTransferRead
treated a vector size divisble by the container size as proof that the access
starts on a container boundary, and ConvertVectorMaskedStore never looked
at the offset at all. Any size-divisible access at an unaligned offset
(e.g. vector<8xi4> at index 1) was silently read or written one sub-element early.

Always fold the real intra-container offset: loads reuse the existing static/dynamic
extraction paths, and maskedstore bails out unless the offset provably folds to zero,
the same contract I saw in #189235 for vector.store.
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  54 +--
 ...atten-memref-and-emulate-narrow-types.mlir |  11 +-
 ...e-narrow-type-unaligned-dynamic-store.mlir |  49 +++
 .../vector-emulate-narrow-type-unaligned.mlir |  70 ++++
 .../Vector/vector-emulate-narrow-type.mlir    | 312 ++++++++++++------
 5 files changed, 369 insertions(+), 127 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index e37296e50fb29..54c79ebce7289 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -910,6 +910,18 @@ struct ConvertVectorMaskedStore final
     Value linearizedIndices =
         getValueOrCreateConstantIndexOp(rewriter, loc, linearizedIndicesOfr);
 
+    // The load-select-store sequence below assumes the store starts at a
+    // container element boundary. A non-zero intra-container offset would
+    // require shifting both the mask and the value to store; that is not
+    // implemented yet, so bail out to avoid storing at the wrong offset.
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply
+    // such alignment - the offset depends on the (possibly dynamic) indices.
+    std::optional<int64_t> intraDataOffset =
+        getConstantIntValue(linearizedInfo.intraDataOffset);
+    if (!intraDataOffset || *intraDataOffset != 0)
+      return rewriter.notifyMatchFailure(
+          op, "unaligned or dynamically-offset maskedstore is not supported");
+
     // Load the whole data and use arith.select to handle the corner cases.
     //
     // As an example, for this masked store of i4 values:
@@ -1042,8 +1054,6 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
     // compile time as they must be constants.
 
     auto origElements = op.getVectorType().getNumElements();
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize = origElements % emulatedPerContainerElem == 0;
 
     auto stridedMetadata =
         memref::ExtractStridedMetadataOp::create(rewriter, loc, op.getBase());
@@ -1058,9 +1068,11 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // load starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     // Always load enough elements which can cover the original elements.
     int64_t maxintraDataOffset =
@@ -1077,7 +1089,9 @@ struct ConvertVectorLoad final : OpConversionPattern<vector::LoadOp> {
       result = dynamicallyExtractSubVector(
           rewriter, loc, dyn_cast<TypedValue<VectorType>>(result), resultVector,
           linearizedInfo.intraDataOffset, origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // This is a no-op if the offset is zero and all loaded elements are
+      // used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
@@ -1167,8 +1181,6 @@ struct ConvertVectorMaskedLoad final
     // subvector at the proper offset after bit-casting.
     auto origType = op.getVectorType();
     auto origElements = origType.getNumElements();
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize = origElements % emulatedPerContainerElem == 0;
 
     auto stridedMetadata =
         memref::ExtractStridedMetadataOp::create(rewriter, loc, op.getBase());
@@ -1182,9 +1194,11 @@ struct ConvertVectorMaskedLoad final
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // load starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     int64_t maxIntraDataOffset =
         foldedIntraVectorOffset.value_or(emulatedPerContainerElem - 1);
@@ -1208,7 +1222,8 @@ struct ConvertVectorMaskedLoad final
       passthru = dynamicallyInsertSubVector(
           rewriter, loc, passthru, emptyVector, linearizedInfo.intraDataOffset,
           origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and the sizes match.
       passthru = staticallyInsertSubvector(rewriter, loc, passthru, emptyVector,
                                            *foldedIntraVectorOffset);
     }
@@ -1237,7 +1252,8 @@ struct ConvertVectorMaskedLoad final
       mask = dynamicallyInsertSubVector(rewriter, loc, mask, emptyMask,
                                         linearizedInfo.intraDataOffset,
                                         origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and the sizes match.
       mask = staticallyInsertSubvector(rewriter, loc, op.getMask(), emptyMask,
                                        *foldedIntraVectorOffset);
     }
@@ -1248,7 +1264,8 @@ struct ConvertVectorMaskedLoad final
       result = dynamicallyExtractSubVector(
           rewriter, loc, result, op.getPassThru(),
           linearizedInfo.intraDataOffset, origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and all loaded elements are used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
@@ -1325,10 +1342,6 @@ struct ConvertVectorTransferRead final
 
     auto origElements = op.getVectorType().getNumElements();
 
-    // Note, per-element-alignment was already verified above.
-    bool isDivisibleInSize =
-        fitsInMultiByteContainerTy(op.getVectorType(), containerElemTy);
-
     // Pad the padding value with 0s on the left. These bits are discarded and
     // thus their values don't matter.
     Value padding = adaptor.getPadding();
@@ -1355,9 +1368,11 @@ struct ConvertVectorTransferRead final
             stridedMetadata.getConstifiedMixedStrides(),
             getAsOpFoldResult(adaptor.getIndices()));
 
+    // Note: a size divisible by `emulatedPerContainerElem` does NOT imply the
+    // read starts at a container element boundary - the intra-container
+    // offset depends on the (possibly dynamic) indices.
     std::optional<int64_t> foldedIntraVectorOffset =
-        isDivisibleInSize ? 0
-                          : getConstantIntValue(linearizedInfo.intraDataOffset);
+        getConstantIntValue(linearizedInfo.intraDataOffset);
 
     int64_t maxIntraDataOffset =
         foldedIntraVectorOffset.value_or(emulatedPerContainerElem - 1);
@@ -1382,7 +1397,8 @@ struct ConvertVectorTransferRead final
       result = dynamicallyExtractSubVector(rewriter, loc, bitCast, zeros,
                                            linearizedInfo.intraDataOffset,
                                            origElements);
-    } else if (!isDivisibleInSize) {
+    } else {
+      // No-op when the offset is zero and all read elements are used.
       result = staticallyExtractSubvector(
           rewriter, loc, result, *foldedIntraVectorOffset, origElements);
     }
diff --git a/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir b/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
index ef253624fe0b4..38f4d8b0a6ff0 100644
--- a/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
+++ b/mlir/test/Dialect/Vector/flatten-memref-and-emulate-narrow-types.mlir
@@ -31,10 +31,14 @@ func.func @vector_maskedload_2d_i4(%arg0: index, %passthru: vector<8xi4>) -> vec
 
 // -----
 
+// Note: the column index must be a constant so that the intra-byte offset of
+// the store provably folds to 0 (a maskedstore at an offset that cannot be
+// proven to be byte-aligned cannot be emulated and would fail to legalize).
 func.func @vector_maskedstore_2d_i4(%arg0: index, %value: vector<8xi4>) {
     %0 = memref.alloc() : memref<4x8xi4>
+    %c0 = arith.constant 0 : index
     %mask = vector.constant_mask [5] : vector<8xi1>
-    vector.maskedstore %0[%arg0, %arg0], %mask, %value :
+    vector.maskedstore %0[%arg0, %c0], %mask, %value :
       memref<4x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
@@ -54,9 +58,12 @@ func.func @vector_store_2d_i4(%arg0: index, %value: vector<8xi4>) {
 
 // -----
 
+// Note: the column index is a constant so that the intra-byte offset of the
+// read provably folds to 0 and no extra byte needs to be read.
 func.func @vector_transfer_read_2d_i4(%arg0: index, %padding: i4) -> vector<8xi4> {
     %0 = memref.alloc() : memref<4x8xi4>
-    %1 = vector.transfer_read %0[%arg0, %arg0], %padding {in_bounds = [true]} : memref<4x8xi4>, vector<8xi4>
+    %c0 = arith.constant 0 : index
+    %1 = vector.transfer_read %0[%arg0, %c0], %padding {in_bounds = [true]} : memref<4x8xi4>, vector<8xi4>
     return %1 : vector<8xi4>
 }
 //  CHECK-LABEL: func @vector_transfer_read_2d_i4(
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
index 518d825234502..ba2293c6321e2 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned-dynamic-store.mlir
@@ -20,3 +20,52 @@ func.func @vector_store_i4_dynamic_memref(%arg0: vector<8xi4>, %dim0: index,
   vector.store %arg0, %src[%idx0, %idx1] : memref<?x?xi4>, vector<8xi4>
   return
 }
+
+// -----
+
+// Dynamic sub-byte vector.maskedstore offsets cannot be treated as
+// byte-aligned either: the store could start in the middle of a byte and the
+// emulation (which read-modify-writes whole bytes) would write to the wrong
+// bits.
+
+func.func @vector_maskedstore_i4_dynamic_col(%idx1: index, %idx2: index,
+                                             %num_elements_to_store: index,
+                                             %value: vector<8xi4>) {
+  %0 = memref.alloc() : memref<3x8xi4>
+  %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    memref<3x8xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
+
+// -----
+
+func.func @vector_maskedstore_i4_constant_mask_dynamic_col(%idx1: index,
+                                                           %idx2: index,
+                                                           %value: vector<8xi4>) {
+  %0 = memref.alloc() : memref<3x8xi4>
+  %mask = vector.constant_mask [4] : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    memref<3x8xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
+
+// -----
+
+// A maskedstore at a static but non-container-aligned offset (an odd i4 index
+// gives intra-byte offset 1) must also be rejected - emulating it with the
+// compressed mask would silently store at the wrong bits (the intra-byte
+// offset used to be dropped; see the discussion in llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i4_unaligned_static_offset(%num_elements_to_store: index,
+                                                         %value: vector<8xi4>) {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
+  // expected-error @below {{failed to legalize operation 'vector.maskedstore' that was explicitly marked illegal}}
+  vector.maskedstore %0[%c1], %mask, %value :
+    memref<9xi4>, vector<8xi1>, vector<8xi4>
+  return
+}
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
index bec7736b90973..60ceecf88ddaf 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-unaligned.mlir
@@ -586,3 +586,73 @@ func.func @vector_store_i2_2d_const_nonzero_col(%arg0: vector<4xi2>) {
 // and one for byte 1 (element at position [0]).
 // CHECK: memref.generic_atomic_rmw %[[ALLOC]][%[[C0]]]
 // CHECK: memref.generic_atomic_rmw %[[ALLOC]][{{.+}}]
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// Size-divisible vectors at non-container-aligned static offsets.
+///
+/// A vector length divisible by the number of emulated elements per container
+/// element does NOT imply the access starts at a container element boundary.
+/// These used to be emulated as if aligned, silently reading shifted data
+/// (see the discussion in llvm/llvm-project#206928).
+///----------------------------------------------------------------------------------------
+
+func.func @vector_load_i4_unaligned_static_offset() -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %1 = vector.load %0[%c1] : memref<9xi4>, vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// CHECK: func @vector_load_i4_unaligned_static_offset(
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[LOAD:.+]] = vector.load %[[ALLOC]][%[[C0]]] : memref<5xi8>, vector<5xi8>
+// CHECK: %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[BITCAST]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
+
+// -----
+
+func.func @vector_maskedload_i4_unaligned_static_offset(%num_elems: index, %passthru: vector<8xi4>) -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %mask = vector.create_mask %num_elems : vector<8xi1>
+  %1 = vector.maskedload %0[%c1], %mask, %passthru : memref<9xi4>, vector<8xi1>, vector<8xi4> into vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// The compressed mask accounts for the front offset of 1: ceil((n + 1) / 2).
+// CHECK: #[[MASK_MAP:.+]] = affine_map<()[s0] -> ((s0 + 1) ceildiv 2)>
+// CHECK: func @vector_maskedload_i4_unaligned_static_offset(
+// CHECK-SAME:    %[[NUM_ELEMS:[a-zA-Z0-9]+]]: index, %[[PASSTHRU:[a-zA-Z0-9]+]]: vector<8xi4>)
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_ELEMS]] : vector<8xi1>
+// CHECK: %[[MASK_IDX:.+]] = affine.apply #[[MASK_MAP]]()[%[[NUM_ELEMS]]]
+// CHECK: %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<5xi1>
+// CHECK: %[[NEW_PASSTHRU:.+]] = vector.insert_strided_slice %[[PASSTHRU]], %{{.+}} {offsets = [1], strides = [1]} : vector<8xi4> into vector<10xi4>
+// CHECK: %[[PASSTHRU_BITCAST:.+]] = vector.bitcast %[[NEW_PASSTHRU]] : vector<10xi4> to vector<5xi8>
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[C0]]], %[[NEW_MASK]], %[[PASSTHRU_BITCAST]] : memref<5xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+// CHECK: %[[LOAD_BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SHIFTED_MASK:.+]] = vector.insert_strided_slice %[[ORIG_MASK]], %{{.+}} {offsets = [1], strides = [1]} : vector<8xi1> into vector<10xi1>
+// CHECK: %[[SELECT:.+]] = arith.select %[[SHIFTED_MASK]], %[[LOAD_BITCAST]], %[[NEW_PASSTHRU]] : vector<10xi1>, vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[SELECT]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
+
+// -----
+
+func.func @vector_transfer_read_i4_unaligned_static_offset(%padding: i4) -> vector<8xi4> {
+  %c1 = arith.constant 1 : index
+  %0 = memref.alloc() : memref<9xi4>
+  %1 = vector.transfer_read %0[%c1], %padding {in_bounds = [true]} : memref<9xi4>, vector<8xi4>
+  return %1 : vector<8xi4>
+}
+// CHECK: func @vector_transfer_read_i4_unaligned_static_offset(
+// CHECK-SAME:    %[[PADDING:[a-zA-Z0-9]+]]: i4)
+// CHECK: %[[ALLOC:.+]] = memref.alloc() : memref<5xi8>
+// CHECK: %[[PAD_I8:.+]] = arith.extui %[[PADDING]] : i4 to i8
+// CHECK: %[[C0:.+]] = arith.constant 0 : index
+// CHECK: %[[READ:.+]] = vector.transfer_read %[[ALLOC]][%[[C0]]], %[[PAD_I8]] : memref<5xi8>, vector<5xi8>
+// CHECK: %[[BITCAST:.+]] = vector.bitcast %[[READ]] : vector<5xi8> to vector<10xi4>
+// CHECK: %[[SLICE:.+]] = vector.extract_strided_slice %[[BITCAST]] {offsets = [1], sizes = [8], strides = [1]} : vector<10xi4> to vector<8xi4>
+// CHECK: return %[[SLICE]]
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
index 72b355e0fed65..af31ac62f859e 100644
--- a/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type.mlir
@@ -17,14 +17,20 @@ func.func @vector_load_i8(%arg1: index, %arg2: index) -> vector<4xi8> {
 // CHECK-NEXT:   [[L:%.+]] = vector.load %[[ALLOC]][%[[ARG0]], %[[ARG1]]] : memref<3x4xi8>, vector<4xi8>
 // CHECK-NEXT:   return
 
-//      CHECK32: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
+//  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 4)>
 //      CHECK32: func @vector_load_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VECLOAD:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VECLOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   return %[[VEC_I4]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VECLOAD:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I8:.+]] = vector.bitcast %[[VECLOAD]] : vector<2xi32> to vector<8xi8>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I8]][%[[FRONT]]] : i8 from vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -35,21 +41,32 @@ func.func @vector_load_i4(%arg1: index, %arg2: index) -> vector<3x8xi4> {
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_load_i4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_load_i4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -60,21 +77,32 @@ func.func @vector_load_f4(%arg1: index, %arg2: index) -> vector<3x8xf4E2M1FN> {
     %2 = vector.insert %1, %cst [0] : vector<8xf4E2M1FN> into vector<3x8xf4E2M1FN>
     return %2 : vector<3x8xf4E2M1FN>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_load_f4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xf4E2M1FN>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xf4E2M1FN>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<10xf4E2M1FN>
+// CHECK-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_load_f4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xf4E2M1FN>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xf4E2M1FN>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<16xf4E2M1FN>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 // -----
 
@@ -83,8 +111,11 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
   %1 = vector.load %0[%arg2, %arg3] : memref<?x?xi4>, vector<8xi4>
   return %1 : vector<8xi4>
 }
+// The intra-container offset of the load is dynamic, so one extra container
+// element is loaded and the target elements are extracted dynamically.
 //  CHECK-DAG: #[[MAP0:.+]] = affine_map<()[s0, s1] -> ((s0 * s1) floordiv 2, s0 floordiv 2)>
 //  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0, s1, s2] -> ((s2 + s0 * s1) floordiv 2)>
+//  CHECK-DAG: #[[MAP2:.+]] = affine_map<()[s0, s1, s2] -> (s0 * s1 + s2 - ((s2 + s0 * s1) floordiv 2) * 2)>
 //      CHECK: func.func @vector_load_i4_dynamic(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: index
 // CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: index
@@ -93,11 +124,15 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
 //      CHECK:   %[[SIZE:.+]] = affine.max #[[MAP0]]()[%[[ARG1]], %[[ARG0]]]
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc(%[[SIZE]]) : memref<?xi8>
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP1]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
-//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP2]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
+//      CHECK:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP0:.+]] = affine_map<()[s0, s1] -> ((s0 * s1) floordiv 8, s0 floordiv 8)>
 //  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0, s1, s2] -> ((s2 + s0 * s1) floordiv 8)>
+//  CHECK32-DAG: #[[MAP2:.+]] = affine_map<()[s0, s1, s2] -> (s0 * s1 + s2 - ((s2 + s0 * s1) floordiv 8) * 8)>
 //      CHECK32: func.func @vector_load_i4_dynamic(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: index
 // CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: index
@@ -106,8 +141,11 @@ func.func @vector_load_i4_dynamic(%arg0 : index, %arg1 : index, %arg2 : index, %
 //      CHECK32:   %[[SIZE:.+]] = affine.max #[[MAP0]]()[%[[ARG1]], %[[ARG0]]]
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc(%[[SIZE]]) : memref<?xi32>
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP1]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
-//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP2]]()[%[[ARG2]], %[[ARG1]], %[[ARG3]]]
+//      CHECK32:   %[[VEC:.+]] = vector.load %[[ALLOC]][%[[INDEX]]] : memref<?xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -122,25 +160,36 @@ func.func @vector_transfer_read_i4(%arg1: index, %arg2: index) -> vector<8xi4> {
       memref<3x8xi4>, vector<8xi4>
     return %1 : vector<8xi4>
 }
+// The intra-container offset of the read is dynamic (it depends on %arg2), so
+// one extra container element is read and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_transfer_read_i4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[CONST:.+]] = arith.constant 0 : i4
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[PAD:.+]] = arith.extui %[[CONST]] : i4 to i8
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xi4>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xi4>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_transfer_read_i4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[CONST:.+]] = arith.constant 0 : i4
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[PAD:.+]] = arith.extui %[[CONST]] : i4 to i32
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xi4>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_I4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xi4>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_I4]][%[[FRONT]]] : i4 from vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -151,7 +200,11 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
       memref<3x8xf4E2M1FN>, vector<8xf4E2M1FN>
     return %1 : vector<8xf4E2M1FN>
 }
+// The intra-container offset of the read is dynamic (it depends on %arg2), so
+// one extra container element is read and the target elements are extracted
+// dynamically.
 //  CHECK-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_transfer_read_f4
 // CHECK-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK:   %[[CONST:.+]] = arith.constant 0.{{0+}}e+00 : f4E2M1FN
@@ -159,10 +212,14 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
 //      CHECK:   %[[BC:.+]] = arith.bitcast %[[CONST]] : f4E2M1FN to i4
 //      CHECK:   %[[PAD:.+]] = arith.extui %[[BC]] : i4 to i8
 //      CHECK:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<4xi8>
-//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<4xi8> to vector<8xf4E2M1FN>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<12xi8>, vector<5xi8>
+//      CHECK:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<5xi8> to vector<10xf4E2M1FN>
+//      CHECK:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<10xf4E2M1FN>
+// CHECK-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 //  CHECK32-DAG: #[[MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[MAP1:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_transfer_read_f4
 // CHECK32-SAME:     (%[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index)
 //      CHECK32:   %[[CONST:.+]] = arith.constant 0.{{0+}}e+00 : f4E2M1FN
@@ -170,8 +227,11 @@ func.func @vector_transfer_read_f4(%arg1: index, %arg2: index) -> vector<8xf4E2M
 //      CHECK32:   %[[BC:.+]] = arith.bitcast %[[CONST]] : f4E2M1FN to i4
 //      CHECK32:   %[[PAD:.+]] = arith.extui %[[BC]] : i4 to i32
 //      CHECK32:   %[[INDEX:.+]] = affine.apply #[[MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<1xi32>
-//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<1xi32> to vector<8xf4E2M1FN>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[MAP1]]()[%[[ARG1]]]
+//      CHECK32:   %[[VEC:.+]] = vector.transfer_read %[[ALLOC]][%[[INDEX]]], %[[PAD]] : memref<3xi32>, vector<2xi32>
+//      CHECK32:   %[[VEC_F4:.+]] = vector.bitcast %[[VEC]] : vector<2xi32> to vector<16xf4E2M1FN>
+//      CHECK32:   %[[EXTRACT:.+]] = vector.extract %[[VEC_F4]][%[[FRONT]]] : f4E2M1FN from vector<16xf4E2M1FN>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : f4E2M1FN into vector<8xf4E2M1FN>
 
 // -----
 
@@ -196,22 +256,29 @@ func.func @vector_maskedload_i8(%arg1: index, %arg2: index, %arg3: index, %passt
 // CHECK-SAME:     memref<3x4xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
 // CHECK-NEXT:   return
 
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded; the mask, the passthru and the result
+// are shifted dynamically by the front offset.
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
-//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 4)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 4)>
+//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 3) ceildiv 4)>
 //      CHECK32: func @vector_maskedload_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<4xi8>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<4xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK32:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<4xi8> to vector<1xi32>
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<8xi8>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<8xi8> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<4xi1>, vector<4xi8>
-//      CHECK32:   return %[[SELECT]]
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i1 into vector<8xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<8xi1>, vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -224,37 +291,50 @@ func.func @vector_maskedload_i4(%arg1: index, %arg2: index, %arg3: index, %passt
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded; the mask, the passthru and the result
+// are shifted dynamically by the front offset.
 //  CHECK-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
-//  CHECK-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 2)>
+//  CHECK-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 2)>
+//  CHECK-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 1) ceildiv 2)>
 //      CHECK: func @vector_maskedload_i4(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
 //      CHECK:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<4xi1>
-//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<8xi4> to vector<4xi8>
+//      CHECK:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<5xi1>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<10xi4>
+//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<10xi4> to vector<5xi8>
 //      CHECK:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK-SAME:     memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
-//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<4xi8> to vector<8xi4>
-//      CHECK:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<8xi1>, vector<8xi4>
+// CHECK-SAME:     memref<12xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i1 into vector<10xi1>
+//      CHECK:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<10xi1>, vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
-//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 8)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 8)>
+//  CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> ((s0 + 7) ceildiv 8)>
 //      CHECK32: func @vector_maskedload_i4(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: index, %[[ARG3:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
 //      CHECK32:   %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<8xi4> to vector<1xi32>
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<16xi4>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<16xi4> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<8xi4>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<8xi1>, vector<8xi4>
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i1 into vector<16xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<16xi1>, vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -275,20 +355,28 @@ func.func @vector_maskedload_i8_constant_mask(%arg1: index, %arg2: index, %passt
 // CHECK-SAME:     memref<3x4xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
 // CHECK-NEXT:   return
 
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded (with a pessimistic compressed mask);
+// the mask, the passthru and the result are shifted dynamically by the front
+// offset.
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 4)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 4)>
 //      CHECK32: func @vector_maskedload_i8_constant_mask(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG3:[a-zA-Z0-9]+]]: vector<4xi8>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG3]] : vector<4xi8> to vector<1xi32>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<2xi1>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<8xi8>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<8xi8> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<4xi8>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG3]] : vector<4xi1>, vector<4xi8>
-//      CHECK32:   return %[[SELECT]]
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i1 into vector<8xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<8xi1>, vector<8xi8>
+// CHECK32-COUNT-4:   vector.insert {{.+}} : i8 into vector<4xi8>
 
 // -----
 
@@ -301,33 +389,47 @@ func.func @vector_maskedload_i4_constant_mask(%arg1: index, %arg2: index, %passt
     %2 = vector.insert %1, %cst [0] : vector<8xi4> into vector<3x8xi4>
     return %2 : vector<3x8xi4>
 }
+// The intra-container offset of the load is dynamic (it depends on %arg2), so
+// one extra container element is loaded (with a pessimistic compressed mask);
+// the mask, the passthru and the result are shifted dynamically by the front
+// offset.
 //  CHECK-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
+//  CHECK-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 2)>
 //      CHECK: func @vector_maskedload_i4_constant_mask(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK:   %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 //      CHECK:   %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
 //      CHECK:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
-//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG2]] : vector<8xi4> to vector<4xi8>
+//      CHECK:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK:   %[[NEW_MASK:.+]] = vector.constant_mask [3] : vector<5xi1>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<10xi4>
+//      CHECK:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<10xi4> to vector<5xi8>
 //      CHECK:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK-SAME:     memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
-//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<4xi8> to vector<8xi4>
-//      CHECK:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG2]] : vector<8xi1>, vector<8xi4>
+// CHECK-SAME:     memref<12xi8>, vector<5xi1>, vector<5xi8> into vector<5xi8>
+//      CHECK:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<5xi8> to vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i1 into vector<10xi1>
+//      CHECK:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<10xi1>, vector<10xi4>
+// CHECK-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 //  CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
+//  CHECK32-DAG: #[[FRONT_MAP:.+]] = affine_map<()[s0] -> (s0 mod 8)>
 //      CHECK32: func @vector_maskedload_i4_constant_mask(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]: index, %[[ARG1:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]: vector<8xi4>)
 //      CHECK32:   %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 //      CHECK32:   %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
 //      CHECK32:   %[[LD_IDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
-//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %[[ARG2]] : vector<8xi4> to vector<1xi32>
+//      CHECK32:   %[[FRONT:.+]] = affine.apply #[[FRONT_MAP]]()[%[[ARG1]]]
+//      CHECK32:   %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<2xi1>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<16xi4>
+//      CHECK32:   %[[NEW_PASSTHRU:.+]] = vector.bitcast %{{.+}} : vector<16xi4> to vector<2xi32>
 //      CHECK32:   %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LD_IDX]]], %[[NEW_MASK]], %[[NEW_PASSTHRU]] :
-// CHECK32-SAME:     memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>
-//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<1xi32> to vector<8xi4>
-//      CHECK32:   %[[SELECT:.+]] = arith.select %[[ORIG_MASK]], %[[BITCAST]], %[[ARG2]] : vector<8xi1>, vector<8xi4>
+// CHECK32-SAME:     memref<3xi32>, vector<2xi1>, vector<2xi32> into vector<2xi32>
+//      CHECK32:   %[[BITCAST:.+]] = vector.bitcast %[[LOAD]] : vector<2xi32> to vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i1 into vector<16xi1>
+//      CHECK32:   %[[SELECT:.+]] = arith.select %{{.+}}, %[[BITCAST]], %{{.+}} : vector<16xi1>, vector<16xi4>
+// CHECK32-COUNT-8:   vector.insert {{.+}} : i4 into vector<8xi4>
 
 // -----
 
@@ -533,34 +635,36 @@ func.func @vector_store_f4(%arg0: vector<8xf4E2M1FN>, %arg1: index, %arg2: index
 /// vector.maskedstore
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedstore_i8(%arg0: index, %arg1: index, %arg2: index, %value: vector<8xi8>) {
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
+func.func @vector_maskedstore_i8(%arg0: index, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
   %0 = memref.alloc() : memref<3x8xi8>
-  %mask = vector.create_mask %arg2 : vector<8xi1>
-  vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
+  %mask = vector.create_mask %arg0 : vector<8xi1>
+  vector.maskedstore %0[%c1, %c0], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 // Expect no conversions, i8 is supported.
 //      CHECK: func @vector_maskedstore_i8(
 // CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG2:[a-zA-Z0-9]+]]
 // CHECK-SAME:     %[[VAL:[a-zA-Z0-9]+]]
-// CHECK-NEXT:   %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
-// CHECK-NEXT:   %[[MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
-// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[ARG0]], %[[ARG1]]], %[[MASK]], %[[VAL]]
+// CHECK-DAG:    %[[C0:.+]] = arith.constant 0 : index
+// CHECK-DAG:    %[[C1:.+]] = arith.constant 1 : index
+// CHECK-DAG:    %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
+// CHECK:        %[[MASK:.+]] = vector.create_mask %[[ARG0]] : vector<8xi1>
+// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[C1]], %[[C0]]], %[[MASK]], %[[VAL]]
 // CHECK-NEXT:   return
 
-// CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 2 + s1 floordiv 4)>
 // CHECK32-DAG: #[[MASK_IDX_MAP:.+]] = affine_map<()[s0] -> (s0 ceildiv 4)>
 // CHECK32:     func @vector_maskedstore_i8(
 // CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG2:[a-zA-Z0-9]+]]
 // CHECK32-SAME:     %[[VAL:[a-zA-Z0-9]+]]
 // CHECK32:        %[[ALLOC:.+]] = memref.alloc() : memref<6xi32>
-// CHECK32:        %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG2]] : vector<8xi1>
-// CHECK32:        %[[LIDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
-// CHECK32:        %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG2]]]
+// CHECK32:        %[[ORIG_MASK:.+]] = vector.create_mask %[[ARG0]] : vector<8xi1>
+// CHECK32:        %[[LIDX:.+]] = arith.constant 2 : index
+// CHECK32:        %[[MASK_IDX:.+]] = affine.apply #[[MASK_IDX_MAP]]()[%[[ARG0]]]
 // CHECK32:        %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<2xi1>
 // CHECK32:        %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<2xi32>
 // CHECK32:        %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]]
@@ -571,29 +675,29 @@ func.func @vector_maskedstore_i8(%arg0: index, %arg1: index, %arg2: index, %valu
 
 // -----
 
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
 func.func @vector_maskedstore_i4(
-  %idx1: index,
-  %idx2: index,
   %num_elements_to_store: index,
   %value: vector<8xi4>) {
 
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
     %0 = memref.alloc() : memref<3x8xi4>
     %mask = vector.create_mask %num_elements_to_store : vector<8xi1>
-    vector.maskedstore %0[%idx1, %idx2], %mask, %value :
+    vector.maskedstore %0[%c1, %c0], %mask, %value :
       memref<3x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
-// CHECK: #[[$ATTR_10:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
 // CHECK: #[[$ATTR_11:.+]] = affine_map<()[s0] -> (s0 ceildiv 2)>
 
 // CHECK:         func.func @vector_maskedstore_i4(
-// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[NUM_EL_TO_STORE:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK:           %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 // CHECK:           %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_EL_TO_STORE]] : vector<8xi1>
-// CHECK:           %[[LIDX:.+]] = affine.apply #[[$ATTR_10]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK:           %[[LIDX:.+]] = arith.constant 4 : index
 // CHECK:           %[[MASK_IDX:.+]] = affine.apply #[[$ATTR_11]]()[%[[NUM_EL_TO_STORE]]]
 // CHECK:           %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<4xi1>
 // CHECK:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<4xi8>
@@ -603,17 +707,14 @@ func.func @vector_maskedstore_i4(
 // CHECK:           %[[NEW_VAL:.+]] = vector.bitcast %[[SELECT]] : vector<8xi4> to vector<4xi8>
 // CHECK:           vector.maskedstore %[[ALLOC]]{{\[}}%[[LIDX]]], %[[NEW_MASK]], %[[NEW_VAL]] : memref<12xi8>, vector<4xi1>, vector<4xi8>
 
-// CHECK32: #[[$ATTR_17:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
 // CHECK32: #[[$ATTR_18:.+]] = affine_map<()[s0] -> (s0 ceildiv 8)>
 
 // CHECK32:         func.func @vector_maskedstore_i4(
-// CHECK32-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK32-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[NUM_EL_TO_STORE:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK32:           %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 // CHECK32:           %[[ORIG_MASK:.+]] = vector.create_mask %[[NUM_EL_TO_STORE]] : vector<8xi1>
-// CHECK32:           %[[LIDX:.+]] = affine.apply #[[$ATTR_17]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK32:           %[[LIDX:.+]] = arith.constant 1 : index
 // CHECK32:           %[[MASK_IDX:.+]] = affine.apply #[[$ATTR_18]]()[%[[NUM_EL_TO_STORE]]]
 // CHECK32:           %[[NEW_MASK:.+]] = vector.create_mask %[[MASK_IDX]] : vector<1xi1>
 // CHECK32:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<1xi32>
@@ -625,30 +726,32 @@ func.func @vector_maskedstore_i4(
 
 // -----
 
-func.func @vector_maskedstore_i8_constant_mask(%arg0: index, %arg1: index, %value: vector<8xi8>) {
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
+func.func @vector_maskedstore_i8_constant_mask(%value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
   %0 = memref.alloc() : memref<3x8xi8>
   %mask = vector.constant_mask [4] : vector<8xi1>
-  vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
+  vector.maskedstore %0[%c1, %c0], %mask, %value : memref<3x8xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 // Expect no conversions, i8 is supported.
 //      CHECK: func @vector_maskedstore_i8_constant_mask(
-// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
 // CHECK-SAME:     %[[VAL:[a-zA-Z0-9]+]]
-// CHECK-NEXT:   %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
-// CHECK-NEXT:   %[[MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[ARG0]], %[[ARG1]]], %[[MASK]], %[[VAL]]
+// CHECK-DAG:    %[[C0:.+]] = arith.constant 0 : index
+// CHECK-DAG:    %[[C1:.+]] = arith.constant 1 : index
+// CHECK-DAG:    %[[ALLOC:.+]] = memref.alloc() : memref<3x8xi8>
+// CHECK:        %[[MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
+// CHECK-NEXT:   vector.maskedstore %[[ALLOC]][%[[C1]], %[[C0]]], %[[MASK]], %[[VAL]]
 // CHECK-NEXT:   return
 
-// CHECK32-DAG: #[[LOAD_IDX_MAP:.+]] = affine_map<()[s0, s1] -> (s0 * 2 + s1 floordiv 4)>
 // CHECK32:     func @vector_maskedstore_i8_constant_mask(
-// CHECK32-SAME:     %[[ARG0:[a-zA-Z0-9]+]]
-// CHECK32-SAME:     %[[ARG1:[a-zA-Z0-9]+]]
 // CHECK32-SAME:     %[[VAL:[a-zA-Z0-9]+]]
 // CHECK32:        %[[ALLOC:.+]] = memref.alloc() : memref<6xi32>
 // CHECK32:        %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK32:        %[[LIDX:.+]] = affine.apply #[[LOAD_IDX_MAP]]()[%[[ARG0]], %[[ARG1]]]
+// CHECK32:        %[[LIDX:.+]] = arith.constant 2 : index
 // CHECK32:        %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<2xi1>
 // CHECK32:        %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<2xi32>
 // CHECK32:        %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]]
@@ -659,26 +762,26 @@ func.func @vector_maskedstore_i8_constant_mask(%arg0: index, %arg1: index, %valu
 
 // -----
 
+// Note: emulating a maskedstore requires a static, container-aligned offset
+// (a dynamic offset makes it impossible to prove alignment and the pattern
+// bails out), hence the constant indices.
 func.func @vector_maskedstore_i4_constant_mask(
-  %idx_1: index,
-  %idx_2: index,
   %val_to_store: vector<8xi4>) {
 
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
     %0 = memref.alloc() : memref<3x8xi4>
     %mask = vector.constant_mask [4] : vector<8xi1>
-    vector.maskedstore %0[%idx_1, %idx_2], %mask, %val_to_store :
+    vector.maskedstore %0[%c1, %c0], %mask, %val_to_store :
       memref<3x8xi4>, vector<8xi1>, vector<8xi4>
     return
 }
 
-// CHECK: #[[$ATTR_12:.+]] = affine_map<()[s0, s1] -> (s0 * 4 + s1 floordiv 2)>
 // CHECK:         func.func @vector_maskedstore_i4_constant_mask(
-// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK:           %[[ALLOC:.+]] = memref.alloc() : memref<12xi8>
 // CHECK:           %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK:           %[[LIDX:.+]] = affine.apply #[[$ATTR_12]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK:           %[[LIDX:.+]] = arith.constant 4 : index
 // CHECK:           %[[NEW_MASK:.+]] = vector.constant_mask [2] : vector<4xi1>
 // CHECK:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<4xi8>
 // CHECK:           %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]] : memref<12xi8>, vector<4xi1>, vector<4xi8> into vector<4xi8>
@@ -687,14 +790,11 @@ func.func @vector_maskedstore_i4_constant_mask(
 // CHECK:           %[[NEW_VAL:.+]] = vector.bitcast %[[SELECT]] : vector<8xi4> to vector<4xi8>
 // CHECK:           vector.maskedstore %[[ALLOC]]{{\[}}%[[LIDX]]], %[[NEW_MASK]], %[[NEW_VAL]] : memref<12xi8>, vector<4xi1>, vector<4xi8>
 
-// CHECK32: #[[$ATTR_20:.+]] = affine_map<()[s0, s1] -> (s0 + s1 floordiv 8)>
 // CHECK32:         func.func @vector_maskedstore_i4_constant_mask(
-// CHECK32-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
-// CHECK32-SAME:      %[[IDX_2:[a-zA-Z0-9]+]]: index,
 // CHECK32-SAME:      %[[VAL_TO_STORE:[a-zA-Z0-9]+]]: vector<8xi4>) {
 // CHECK32:           %[[ALLOC:.+]] = memref.alloc() : memref<3xi32>
 // CHECK32:           %[[ORIG_MASK:.+]] = vector.constant_mask [4] : vector<8xi1>
-// CHECK32:           %[[LIDX:.+]] = affine.apply #[[$ATTR_20]]()[%[[IDX_1]], %[[IDX_2]]]
+// CHECK32:           %[[LIDX:.+]] = arith.constant 1 : index
 // CHECK32:           %[[NEW_MASK:.+]] = vector.constant_mask [1] : vector<1xi1>
 // CHECK32:           %[[PASS_THRU:.+]] = arith.constant dense<0> : vector<1xi32>
 // CHECK32:           %[[LOAD:.+]] = vector.maskedload %[[ALLOC]][%[[LIDX]]], %[[NEW_MASK]], %[[PASS_THRU]] : memref<3xi32>, vector<1xi1>, vector<1xi32> into vector<1xi32>


From llvmlistbot at llvm.org  Fri Jul  3 10:22:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 10:22:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a47efe3.70920dee.c0a01.6283@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From b1737aa3922d37bd33bbe450da525b46111a5bf4 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload


From llvmlistbot at llvm.org  Fri Jul  3 11:29:01 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 03 Jul 2026 11:29:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a47ff6d.a151105c.30c1b9.5b86@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/203583

>From e9755287e97fb9bab0030b391da1a9f5846996f0 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Tue, 19 May 2026 10:44:43 +0100
Subject: [PATCH 1/4] [mlir][tosa] Add new block-scaled tensor type and support
 for MXFP CAST

This commit adds a new compound-block scaled tensor type and uses this
type to implement support for MXFP in the CAST operation, as per
the following specification changes:
https://github.com/arm/tosa-specification/pull/50,
https://github.com/arm/tosa-specification/pull/53.

The new block-scaled type is closely modeled after the `quant` dialect
type and supports the following parameters:
- value type - The type of the data values in each block.
- scale type - The type of the scale value associated with each block.
- block shape - The size and axis of each block.
Support for specifying constant scale values has not been added in this
commit, but will be added in a later one.

Example syntax for the new block-scaled type:
```
tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
```

As a pre-requisite for supporting MXFP CAST, the commit also adds new
extensions intended to split the exisiting EXT-MXFP extension into
separate extensions for each block-scaled type. This allows for more
fine-grained control over which block-scaled types are supported by a
given target. See specification change
https://github.com/arm/tosa-specification/pull/26 for details.

Finally, support for casting to/from the new block-scaled type has been
added, aligning with the behavior specified in the CAST operation
specification.

Change-Id: Idfab945402a95704ffeefbcee44172c3dde7552d
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 120 +++++++++++++++++-
 .../mlir/Dialect/Tosa/IR/TosaOpBase.td        |  61 +++++++--
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h   |   2 +
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  69 +++-------
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   |  30 ++++-
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 117 +++++++++++------
 mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp        |  14 ++
 .../Dialect/Tosa/IR/TosaCanonicalizations.cpp |  44 ++++++-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |  65 ++++++++++
 .../Tosa/Transforms/TosaProfileCompliance.cpp |  92 ++++++++++----
 .../Tosa/Transforms/TosaValidation.cpp        |   2 +-
 mlir/test/Dialect/Tosa/availability.mlir      |   6 +-
 mlir/test/Dialect/Tosa/canonicalize.mlir      |  67 ++++++++++
 mlir/test/Dialect/Tosa/invalid.mlir           |  32 +++++
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  28 ++++
 mlir/test/Dialect/Tosa/ops.mlir               |  26 ++++
 .../test/Dialect/Tosa/tosa-attach-target.mlir |  19 ++-
 .../tosa-validation-version-1p0-invalid.mlir  |   8 ++
 .../tosa-validation-version-1p1-valid.mlir    |  30 ++++-
 mlir/test/Dialect/Tosa/verifier.mlir          |  42 ++++++
 20 files changed, 729 insertions(+), 145 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 50bb9f69c6242..26890abd187bd 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1023,7 +1023,125 @@ extensionComplianceMap = {
        {{{fp8e5m2T, fp16T}, SpecificationVersion::V_1_0},
         {{fp8e5m2T, fp32T}, SpecificationVersion::V_1_0},
         {{fp16T, fp8e5m2T}, SpecificationVersion::V_1_0},
-        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}}}},
+        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.cast_from_block_scaled",
      {{{Extension::bf16, Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, bf16T}, SpecificationVersion::V_1_1_DRAFT},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
index 591073e9985ae..308e98ac42435 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
@@ -254,6 +254,13 @@ class Tosa_I32EnumAttr<string name, string description, string mnemonic,
 // MXFP         : Microscaling formats.
 // MXFP_CONV    : Microscaling format convolution.
 // SHAPE        : Shape calcuation operators.
+// MX_COMMON    : Base for MXFP microscaling formats.
+// MX_FP4E2M1   : Microscaling format FP4E2M1.
+// MX_FP6E2M3   : Microscaling format FP6E2M3.
+// MX_FP6E3M2   : Microscaling format FP6E3M2.
+// MX_FP8E4M3   : Microscaling format FP8E4M3.
+// MX_FP8E5M2   : Microscaling format FP8E5M2.
+// MX_INT8      : Microscaling format INT8.
 //===----------------------------------------------------------------------===//
 
 def Tosa_NONE : I32EnumAttrCase<"none", 0>;
@@ -289,23 +296,35 @@ def Tosa_EXT_MXFP         : I32EnumAttrCase<"mxfp", 12>;
 def Tosa_EXT_INT64        : I32EnumAttrCase<"int64", 13>;
 def Tosa_EXT_MXFP_CONV    : I32EnumAttrCase<"mxfp_conv", 14>;
 def Tosa_EXT_SHAPE        : I32EnumAttrCase<"shape", 15>;
+def Tosa_EXT_MX_COMMON : I32EnumAttrCase<"mx_common", 16>;
+def Tosa_EXT_MX_FP4E2M1 : I32EnumAttrCase<"mx_fp4e2m1", 17>;
+def Tosa_EXT_MX_FP6E2M3 : I32EnumAttrCase<"mx_fp6e2m3", 18>;
+def Tosa_EXT_MX_FP6E3M2 : I32EnumAttrCase<"mx_fp6e3m2", 19>;
+def Tosa_EXT_MX_FP8E4M3 : I32EnumAttrCase<"mx_fp8e4m3", 20>;
+def Tosa_EXT_MX_FP8E5M2 : I32EnumAttrCase<"mx_fp8e5m2", 21>;
+def Tosa_EXT_MX_INT8 : I32EnumAttrCase<"mx_int8", 22>;
 
 def Tosa_ExtensionAttr
-    : Tosa_I32EnumAttr<"Extension", "supported TOSA extensions", "ext", [
-      Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
-      Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
-      Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
-      Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
-      Tosa_EXT_SHAPE,
-    ]> {
+    : Tosa_I32EnumAttr<
+          "Extension", "supported TOSA extensions", "ext",
+          [Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
+           Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
+           Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
+           Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
+           Tosa_EXT_SHAPE, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+           Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+           Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]> {
   let extraClassDeclaration = [{
-    static llvm::SmallVector<Extension, 14> getAllValues() {
+    static llvm::SmallVector<Extension, 22> getAllValues() {
       return {
         Extension::int16, Extension::int4, Extension::bf16,
         Extension::fp8e4m3, Extension::fp8e5m2, Extension::fft,
         Extension::variable, Extension::controlflow, Extension::doubleround,
         Extension::inexactround, Extension::dynamic, Extension::mxfp,
-        Extension::int64, Extension::mxfp_conv, Extension::shape
+        Extension::int64, Extension::mxfp_conv, Extension::shape,
+        Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3,
+        Extension::mx_fp6e3m2, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2,
+        Extension::mx_int8
       };
     }
   }];
@@ -484,6 +503,7 @@ def Tosa_RoundingModeAttr
     : Tosa_I32EnumAttr<"RoundingMode", "Supported rounding modes", "rounding_mode",
                     [Tosa_ROUNDING_SINGLE_ROUND, Tosa_ROUNDING_INEXACT_ROUND, Tosa_ROUNDING_DOUBLE_ROUND]>;
 
+// Block_size attr is deprecated and will be removed in the future
 def Tosa_BLOCK_SIZE_1 : I32EnumAttrCase<"BLOCK_SIZE_1", 1>;
 def Tosa_BLOCK_SIZE_32 : I32EnumAttrCase<"BLOCK_SIZE_32", 32>;
 
@@ -497,6 +517,29 @@ def Tosa_BlockSizeAttr
   }];
 }
 
+def Tosa_BLOCK_SHAPE_32 : I32EnumAttrCase<"BLOCK_SHAPE_32", 32>;
+
+def Tosa_BlockShape
+    : Tosa_I32Enum<
+          "BlockShape",
+          "Block shape for the block_scaled formats."
+          "The names follow the convention of BLOCK_SHAPE_M where M "
+          "is the block size of the innermost dimension. Similarly, "
+          "BLOCK_SHAPE_MxN indicates block size of N in the innermost "
+          "dimension and block size of M in the next innermost dimension. "
+          "As of now, only 1 dimension (innermost) is supported for "
+          "block-scaling.",
+          [Tosa_BLOCK_SHAPE_32]>;
+
+def Tosa_BlockShapeAttr
+    : EnumAttr<Tosa_Dialect, Tosa_BlockShape, "block_shape"> {
+  let extraClassDeclaration = [{
+    static uint32_t getBlockShapeValue(BlockShape blockShape) {
+      return static_cast<uint32_t>(blockShape);
+    }
+  }];
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Interfaces.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index e0626368175ee..a5c6037692ff1 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,6 +92,8 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
+LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+
 } // namespace tosa
 } // namespace OpTrait
 
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index a99fb2fcae547..32f8bf08cbc84 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2689,57 +2689,6 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
 
   let description = [{
     Casts a tensor from one data type to another.
-    * This table is showing the supported conversions from the TOSA Specification.
-    * The MLIR dialect here can be used to represent other conversions.
-
-    | Mode                     | Input   | Output  |
-    |--------------------------|---------|---------|
-    | fp16 to fp32             | float16 | float32 |
-    | fp16 to int 16           | float16 | int16   |
-    | fp16 to int 32           | float16 | int32   |
-    | fp16 to int 8            | float16 | int8    |
-    | fp32 to fp16             | float32 | float16 |
-    | fp32 to int 16           | float32 | int16   |
-    | fp32 to int 32           | float32 | int32   |
-    | fp32 to int 8            | float32 | int8    |
-    | int 16 to fp16           | int16   | float16 |
-    | int 16 to fp32           | int16   | float32 |
-    | int 32 to fp16           | int32   | float16 |
-    | int 32 to fp32           | int32   | float32 |
-    | int 8 to fp16            | int8    | float16 |
-    | int 8 to fp32            | int8    | float32 |
-    | bool to int 16           | Boolean | int16   |
-    | bool to int 32           | Boolean | int32   |
-    | bool to int 8            | Boolean | int8    |
-    | int 16 to bool           | int16   | Boolean |
-    | int 16 to int 32         | int16   | int32   |
-    | int 16 to int 8          | int16   | int8    |
-    | int 32 to bool           | int32   | Boolean |
-    | int 32 to int 16         | int32   | int16   |
-    | int 32 to int 8          | int32   | int8    |
-    | int 8 to bool            | int8    | Boolean |
-    | int 8 to int 16          | int8    | int16   |
-    | int 8 to int 32          | int8    | int32   |
-    | bf16 to fp32             | bf16    | float32 |
-    | bf16 to int 16           | bf16    | int16   |
-    | bf16 to int 32           | bf16    | int32   |
-    | bf16 to int 8            | bf16    | int8    |
-    | fp32 to bf16             | float32 | bf16    |
-    | int 16 to bf16           | int16   | bf16    |
-    | int 32 to bf16           | int32   | bf16    |
-    | int 8 to bf16            | int8    | bf16    |
-    | bf16 to fp8e4m3          | bf16    | fp8e4m3 |
-    | fp8e4m3 to bf16          | fp8e4m3 | bf16    |
-    | bf16 to fp8e5m2          | bf16    | fp8e5m2 |
-    | fp8e5m2 to bf16          | fp8e5m2 | bf16    |
-    | fp16 to fp8e4m3          | float16 | fp8e4m3 |
-    | fp32 to fp8e4m3          | float32 | fp8e4m3 |
-    | fp8e4m3 to fp16          | fp8e4m3 | float16 |
-    | fp8e4m3 to fp32          | fp8e4m3 | float32 |
-    | fp16 to fp8e5m2          | float16 | fp8e5m2 |
-    | fp32 to fp8e5m2          | float32 | fp8e5m2 |
-    | fp8e5m2 to fp16          | fp8e5m2 | float16 |
-    | fp8e5m2 to fp32          | fp8e5m2 | float32 |
   }];
 
   let arguments = (ins
@@ -2750,19 +2699,25 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 
   let hasFolder = 1;
   let hasCanonicalizer = 1;
+  let hasVerifier = 1;
 }
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_from_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled", [Pure]> {
   let summary = "Apply scales from a scale tensor to the values in a value tensor";
@@ -2771,6 +2726,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
     Apply the scales from a scale tensor to the values in a value tensor, casting
     the result to the output type. The block dimension must be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
@@ -2794,6 +2751,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_to_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [Pure]> {
   let summary = "Calculate scale tensor values per block, output to separate scale and data tensors.";
@@ -2803,6 +2762,8 @@ def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [P
     scaled data values from an input tensor. The output tensors are cast to the
     specified scale and value types. The block dimension will be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 0135a651be481..4e14b27a421e4 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -23,10 +23,22 @@ using namespace mlir::tosa;
 // Type Compilance Definition
 //===----------------------------------------------------------------------===//
 
-typedef struct {
+struct TypeInfo {
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(mlir::TypeID()),
+        scaleTypeID(mlir::TypeID()), blockSize(0) {}
+
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth, mlir::TypeID valueTypeID,
+           mlir::TypeID scaleTypeID, uint32_t blockSize)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(valueTypeID),
+        scaleTypeID(scaleTypeID), blockSize(blockSize) {}
+
   mlir::TypeID typeID;
   uint32_t bitWidth;
-} TypeInfo;
+  mlir::TypeID valueTypeID;
+  mlir::TypeID scaleTypeID;
+  uint32_t blockSize;
+};
 
 enum CheckCondition {
   invalid,
@@ -70,6 +82,14 @@ class ProfileInfoDepot {
 
 private:
   TypeInfo convertTypeToInfo(Type type) {
+    if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type)) {
+      Type valueTy = blockScaledTy.getValueType();
+      Type scaleTy = blockScaledTy.getScaleType();
+      return {
+          type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
+          scaleTy.getTypeID(),
+          BlockShapeAttr::getBlockShapeValue(blockScaledTy.getBlockShape())};
+    }
     return {type.getTypeID(), tosa::getBitWidth(type)};
   }
 
@@ -128,7 +148,9 @@ class TosaProfileCompliance {
       const SmallVector<ArrayRef<T>> &specDefinedProfileSet);
 
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
-    return a.typeID == b.typeID && a.bitWidth == b.bitWidth;
+    return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
+           a.valueTypeID == b.valueTypeID && a.scaleTypeID == b.scaleTypeID &&
+           a.blockSize == b.blockSize;
   }
 
   // Find the required profiles or extensions from the compliance info according
@@ -145,7 +167,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<32> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 10ddd3438aedd..0a43f14865f56 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -103,15 +103,39 @@ def Tosa_MXInt8
   }];
 }
 
+def Tosa_MXFPValue
+    : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
+                "micro-scaling format number">;
+def Tosa_MXFPScale
+    : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+
+def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+  let summary = "Block scaled tensor element type.";
+
+  let description = [{
+    This does not specify an implementation type. A tensor of this type is a 
+    tensor which has block-scaled quantization applied.
+
+    This compound type is made up of 3 components:
+    `value_type` - The type of the data values in each block.
+    `scale_type` - The type of the scale value associated with each block.
+    `block_shape` - The size and axis of each block. Currently only supports
+      specifying the block size along the innermost dimension.
+  }];
+
+  let parameters = (ins Tosa_MXFPValue:$value_type, Tosa_MXFPScale:$scale_type,
+      EnumParameter<Tosa_BlockShape>:$block_shape);
+
+  let assemblyFormat =
+      "`<` $value_type```:```$scale_type```:```$block_shape `>`";
+}
+
 //===----------------------------------------------------------------------===//
 // Multi-category types.
 //===----------------------------------------------------------------------===//
-def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat, Tosa_MXInt8],
-                                "number">;
-
-def Tosa_MXFPNumber : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
-                                "micro-scaling format number">;
-def Tosa_MXFPScaleNumber : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat,
+                                Tosa_MXInt8, Tosa_BlockScaled],
+                               "number">;
 
 //===----------------------------------------------------------------------===//
 // TOSA Tensor Conformance
@@ -129,16 +153,29 @@ def AtLeastRankOne : And<[
   IsRankedTensorTypePred,
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
-class TosaTensorOf<
-    list<Type> allowedTypes, string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes, [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>], summary>;
-
-class TosaRankedTensorOf<
-    list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<allowedTypes, !listconcat([HasNo0Dimensions], preds), summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes, preds, summary>;
+def IsValidBlockScaledTensorType
+    : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
+            "verifyBlockScaledTensorType($_op, $_self))">;
+
+class TosaTensorOf<list<Type> allowedTypes,
+                   string summary = "tosa-conformant tensor">
+    : TensorOf<allowedTypes,
+               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+                IsValidBlockScaledTensorType],
+               summary>;
+
+class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                         string summary = "tosa-conformant ranked tensor">
+    : RankedTensorOf<
+          allowedTypes,
+          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
+          summary>;
+
+class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                           string summary = "tosa-conformant unranked tensor">
+    : UnrankedTensorOf<allowedTypes,
+                       !listconcat([IsValidBlockScaledTensorType], preds),
+                       summary>;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaRankedTensorOf<allowedTypes,
@@ -217,32 +254,28 @@ def Tosa_IndexTensor2D : AnyTypeOf<[
 def Tosa_TensorAtLeast1D : AnyTypeOf<[
   Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
 
-def Tosa_MXFPDataTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [3]>
-]>;
-def Tosa_MXFPScaleTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [3]>
-]>;
-def Tosa_MXFPDataTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [4]>
-]>;
-def Tosa_MXFPScaleTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [4]>
-]>;
-def Tosa_MXFPDataTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
-def Tosa_MXFPScaleTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPScaleNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
+def Tosa_MXFPDataTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+def Tosa_MXFPScaleTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+def Tosa_MXFPDataTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+def Tosa_MXFPScaleTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+def Tosa_MXFPDataTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
+def Tosa_MXFPScaleTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
index dc18fcaa04c8a..56e4901811dcb 100644
--- a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
@@ -56,6 +56,13 @@ TosaSpecificationVersion getMinVersion(const Extension &extension) {
   case Extension::int64:
   case Extension::mxfp_conv:
   case Extension::shape:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return TosaSpecificationVersion(1, 1, true);
   case Extension::none:
     return TosaSpecificationVersion(0, 0);
@@ -76,6 +83,13 @@ SmallVector<Profile, 2> getCooperativeProfiles(Extension ext) {
   case Extension::fft:
   case Extension::mxfp:
   case Extension::mxfp_conv:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return {Profile::pro_fp};
   case Extension::variable:
   case Extension::controlflow:
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 63d40ed4a95a5..99315a787f12e 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1184,12 +1184,50 @@ struct NonNarrowingCastsOptimization : public OpRewritePattern<tosa::CastOp> {
   }
 };
 
+struct CancellingBlockScaledCastsOptimization
+    : public OpRewritePattern<tosa::CastOp> {
+  using OpRewritePattern<tosa::CastOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(tosa::CastOp castOp,
+                                PatternRewriter &rewriter) const override {
+    const Value outerInput = castOp.getInput();
+    auto innerCastOp = outerInput.getDefiningOp<tosa::CastOp>();
+    if (!innerCastOp)
+      return rewriter.notifyMatchFailure(castOp,
+                                         "input must be a cast operation");
+
+    const Value innerInput = innerCastOp.getInput();
+    const auto innerInputTy = llvm::cast<ShapedType>(innerInput.getType());
+    const auto innerOutputTy = llvm::cast<ShapedType>(innerCastOp.getType());
+    const auto outerOutputTy = llvm::cast<ShapedType>(castOp.getType());
+
+    if (!llvm::isa<tosa::BlockScaledType>(innerInputTy.getElementType()))
+      return rewriter.notifyMatchFailure(
+          castOp, "inner cast input must have block scaled element type");
+
+    if (innerInputTy != outerOutputTy)
+      return rewriter.notifyMatchFailure(
+          castOp, "inner input type must match outer output type");
+
+    const Type innerOutputElemType = innerOutputTy.getElementType();
+    const bool isLosslessCast = isa<Float32Type>(innerOutputElemType);
+    if (!isLosslessCast)
+      return rewriter.notifyMatchFailure(
+          castOp, "avoid cancelling casts that could by lossy");
+
+    rewriter.replaceOp(castOp, innerInput);
+
+    return success();
+  }
+};
+
 void CastOp::getCanonicalizationPatterns(RewritePatternSet &results,
                                          MLIRContext *context) {
-  results.add<NonNarrowingCastsOptimization>(context);
+  results.add<NonNarrowingCastsOptimization,
+              CancellingBlockScaledCastsOptimization>(context);
 }
 
-struct CancellingBlockScaledCastsOptimization
+struct CancellingCastToFromBlockScaledOptimization
     : public OpRewritePattern<tosa::CastToBlockScaledOp> {
   using OpRewritePattern<tosa::CastToBlockScaledOp>::OpRewritePattern;
 
@@ -1235,7 +1273,7 @@ struct CancellingBlockScaledCastsOptimization
 
 void CastToBlockScaledOp::getCanonicalizationPatterns(
     RewritePatternSet &results, MLIRContext *context) {
-  results.add<CancellingBlockScaledCastsOptimization>(context);
+  results.add<CancellingCastToFromBlockScaledOptimization>(context);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index f05399cf6b00b..486431a562714 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -628,6 +628,8 @@ Value mlir::tosa::createPadConstTensor(OpBuilder &builder, Location loc,
 }
 
 unsigned mlir::tosa::getBitWidth(Type type) {
+  if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type))
+    return getBitWidth(blockScaledTy.getValueType());
   if (dyn_cast<tosa::mxint8Type>(type))
     return 8;
   return type.getIntOrFloatBitWidth();
@@ -734,6 +736,41 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
   return cast<IntegerType>(attrType).convertFromAttribute(attr, result);
 }
 
+//===----------------------------------------------------------------------===//
+// TOSA block scaling utilities.
+//===----------------------------------------------------------------------===//
+
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  const auto tensorType = llvm::cast<ShapedType>(type);
+  const BlockScaledType elemType =
+      llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
+  if (!elemType)
+    return success();
+
+  if (!tensorType.hasRank())
+    return success();
+
+  if (tensorType.getRank() == 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " does not support block scaling on scalar tensors";
+
+  const int64_t blockedDimension = tensorType.getShape().back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
+
+  const uint32_t blockSize =
+      BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+  if (blockedDimension % blockSize != 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " blocked dimension must be a multiple of block size, got "
+           << blockedDimension << " and block size " << blockSize;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -5004,6 +5041,34 @@ LogicalResult RescaleOp::inferReturnTypeComponents(
   return success();
 }
 
+LogicalResult CastOp::verify() {
+  const ShapedType inputType = llvm::cast<ShapedType>(getInput().getType());
+  const ShapedType outputType = llvm::cast<ShapedType>(getType());
+  const Type inputElementType = inputType.getElementType();
+  const Type outputElementType = outputType.getElementType();
+
+  const bool inputIsBlockScaled = llvm::isa<BlockScaledType>(inputElementType);
+  const bool outputIsBlockScaled =
+      llvm::isa<BlockScaledType>(outputElementType);
+  if (!inputIsBlockScaled && !outputIsBlockScaled)
+    return success();
+
+  if (inputIsBlockScaled && outputIsBlockScaled)
+    return emitOpError()
+           << "requires exactly one of input or output to have block scaled "
+              "element type";
+
+  const Type scalarElementType =
+      inputIsBlockScaled ? outputElementType : inputElementType;
+  if (!llvm::isa<FloatType>(scalarElementType))
+    return emitOpError()
+           << "requires non-block-scaled element type to be floating-point "
+              "when casting to or from block scaled element type, got "
+           << scalarElementType;
+
+  return success();
+}
+
 LogicalResult CastFromBlockScaledOp::inferReturnTypeComponents(
     MLIRContext *context, ::std::optional<Location> location,
     CastFromBlockScaledOp::Adaptor adaptor,
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 9511d4da89dbd..9fbdfcc1d6690 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Tosa/IR/TosaProfileCompliance.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/raw_ostream.h"
 
 using namespace mlir;
 using namespace mlir::tosa;
@@ -27,12 +28,37 @@ TosaProfileCompliance::TosaProfileCompliance() {
   const TypeInfo fp8e5m2T = {mlir::Float8E5M2Type::getTypeID(), 8};
 
   // micro-scaling formats
+  // Note: these types exist to suppport the deprecated block_scaled operations
+  // and can be removed once those operations are removed.
   const TypeInfo fp6e2m3T = {mlir::Float6E2M3FNType::getTypeID(), 6};
   const TypeInfo fp6e3m2T = {mlir::Float6E3M2FNType::getTypeID(), 6};
   const TypeInfo fp4e2m1T = {mlir::Float4E2M1FNType::getTypeID(), 4};
   const TypeInfo fp8ue8m0T = {mlir::Float8E8M0FNUType::getTypeID(), 8};
   const TypeInfo mxint8T = {mlir::tosa::mxint8Type::getTypeID(), 8};
 
+  // Block scaled formats
+  const TypeID blockScaledID = mlir::tosa::BlockScaledType::getTypeID();
+  const TypeID fp4e2m1ID = mlir::Float4E2M1FNType::getTypeID();
+  const TypeID fp6e2m3ID = mlir::Float6E2M3FNType::getTypeID();
+  const TypeID fp6e3m2ID = mlir::Float6E3M2FNType::getTypeID();
+  const TypeID fp8e4m3ID = mlir::Float8E4M3FNType::getTypeID();
+  const TypeID fp8e5m2ID = mlir::Float8E5M2Type::getTypeID();
+  const TypeID fp8ue8m0ID = mlir::Float8E8M0FNUType::getTypeID();
+  const TypeID mxint8ID = mlir::tosa::mxint8Type::getTypeID();
+
+  const TypeInfo bs32_fp8ue8m0_fp4e2m1T = {blockScaledID, 4, fp4e2m1ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp6e2m3T = {blockScaledID, 6, fp6e2m3ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp6e3m2T = {blockScaledID, 6, fp6e3m2ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp8e4m3T = {blockScaledID, 8, fp8e4m3ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_fp8e5m2T = {blockScaledID, 8, fp8e5m2ID,
+                                           fp8ue8m0ID, 32};
+  const TypeInfo bs32_fp8ue8m0_mxint8T = {blockScaledID, 8, mxint8ID,
+                                          fp8ue8m0ID, 32};
+
 // The profile-based compliance content below is auto-generated by a script
 // in https://github.com/arm/tosa-specification
 #include "mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc"
@@ -669,31 +695,47 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<7>
+llvm::SmallString<32>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
-  if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
-    return {"i" + llvm::utostr(typeInfo.bitWidth)};
-  }
-  if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
-    return {"f16"};
-  } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
-    return {"f32"};
-  } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
-    return {"bf16"};
-  } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
-    return {"fp8e4m3"};
-  } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
-    return {"fp8e5m2"};
-  } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
-    return {"fp6e2m3"};
-  } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
-    return {"fp6e3m2"};
-  } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
-    return {"fp4e2m1"};
-  } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
-    return {"fp8e8m0"};
-  } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
-    return {"mxint8"};
+  const auto stringifyScalarTypeInfo =
+      [](const TypeInfo &typeInfo) -> llvm::SmallString<32> {
+    if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
+      return {"i" + llvm::utostr(typeInfo.bitWidth)};
+    }
+    if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
+      return {"f16"};
+    } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
+      return {"f32"};
+    } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
+      return {"bf16"};
+    } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
+      return {"fp8e4m3"};
+    } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
+      return {"fp8e5m2"};
+    } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
+      return {"fp6e2m3"};
+    } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
+      return {"fp6e3m2"};
+    } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
+      return {"fp4e2m1"};
+    } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
+      return {"fp8e8m0"};
+    } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
+      return {"mxint8"};
+    }
+    llvm_unreachable("unknown type");
+  };
+
+  if (typeInfo.typeID == tosa::BlockScaledType::getTypeID()) {
+    TypeInfo valueInfo = {typeInfo.valueTypeID, typeInfo.bitWidth};
+    TypeInfo scaleInfo = {typeInfo.scaleTypeID, 8};
+    llvm::SmallString<32> result;
+    llvm::raw_svector_ostream os(result);
+    os << "bs" << typeInfo.blockSize << "_"
+       << stringifyScalarTypeInfo(scaleInfo) << "_"
+       << stringifyScalarTypeInfo(valueInfo);
+    return result;
   }
-  llvm_unreachable("unknown type");
+
+  return stringifyScalarTypeInfo(typeInfo);
 }
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 34ac84d237f45..8fa2ddf856c7d 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -1482,7 +1482,7 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
     }
   } else if (isa<tosa::shapeType>(type))
     return true;
-  else if (isa<tosa::mxint8Type>(type))
+  else if (isa<tosa::mxint8Type, tosa::BlockScaledType>(type))
     return true;
   return false;
 }
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 81294f2c0c308..450b4556c7b16 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -627,10 +627,10 @@ func.func @test_resize(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x64x64x8xf32> {
 }
 
 // -----
-// CHECK-LABEL: cast
-func.func @test_cast1(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
+// CHECK-LABEL: test_cast
+func.func @test_cast(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.cast %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 2cd040f056db8..d454ca831cf0d 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1651,6 +1651,73 @@ func.func @test_canonicalize_non_narrowing_cast_i8_to_f8E4M3FN_unsupported(%arg0
 
 // -----
 
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1
+// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf32>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2
+// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32xf32>
+  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*xf32>
+  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3
+// CHECK: tosa.cast
+// CHECK: tosa.cast
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf8E4M3FN>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1
 // CHECK: return %arg0, %arg1 : tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>
 func.func @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1(%arg0: tensor<15x3x2x256xf4E2M1FN>, %arg1: tensor<15x3x2x8xf8E8M0FNU>) -> (tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>) {
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index 5e8111061cb3a..eca5c9ef1b14e 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2266,3 +2266,35 @@ func.func @test_shape_func_output() -> !tosa.shape<4> {
   %cst = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   return %cst : !tosa.shape<4>
 }
+
+// -----
+
+func.func @test_cast_f32_plain_fp4(%arg0: tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (f32,fp4e2m1)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN>
+  return %0 : tensor<4x32xf4E2M1FN>
+}
+
+// -----
+
+func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp4e2m1,bs32_fp8e8m0_fp4e2m1), did you mean (fp8e4m3,bs32_fp8e8m0_fp4e2m1)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (bs32_fp8e8m0_fp4e2m1,fp6e2m3), did you mean (bs32_fp8e8m0_fp4e2m1,fp8e4m3)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN>
+  return %0 : tensor<4x32xf6E2M3FN>
+}
+
+// -----
+
+func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp6e3m2,bs32_fp8e8m0_mxint8), did you mean (fp8e4m3,bs32_fp8e8m0_mxint8)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 295d7172bc2c4..9d4e59a9a9c6f 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -295,6 +295,34 @@ func.func @test_cast_f32_bf16(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xbf16
   return %0 : tensor<13x21x3xbf16>
 }
 
+// -----
+func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  return %0 : tensor<4x32xf32>
+}
+
+// -----
+func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [fp8e4m3, mx_common, mx_int8] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
 // -----
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index c7f4c67072d06..86212177d34be 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1059,6 +1059,32 @@ func.func @test_cast3(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3x!quant.unifo
   return %0 : tensor<13x21x3x!quant.uniform<i16:f32, 0.078431375324726104:128>>
 }
 
+// -----
+// CHECK-LABEL: test_cast_to_block_scaled
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+// CHECK-LABEL: test_cast_from_block_scaled
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg2: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
+  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+  return %0, %1, %2 : tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>
+}
+
+// -----
+// CHECK-LABEL: test_cast_block_scaled_dynamic
+func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
+  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<?x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x?xf32>
+  return %0, %1 : tensor<?x32xf32>, tensor<4x?xf32>
+}
+
 // -----
 // CHECK-LABEL: rescale
 func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439347028732:127>>) -> tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>> {
diff --git a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
index a0c59c0c4bb3b..558a599fbbc1e 100644
--- a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
@@ -1,11 +1,26 @@
-// RUN: mlir-opt %s -split-input-file -tosa-attach-target="profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,dynamic level=none" | FileCheck %s --check-prefix=CHECK-ALL
+// DEFINE: %{core_extensions} = int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable
+// DEFINE: %{rounding_extensions} = controlflow,doubleround,inexactround,dynamic
+// DEFINE: %{mx_float_extensions} = mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2
+// DEFINE: %{mx_int_extensions} = mx_fp8e4m3,mx_fp8e5m2,mx_int8
+// DEFINE: %{all_extensions} = %{core_extensions},%{rounding_extensions},%{mx_float_extensions},%{mx_int_extensions}
+// DEFINE: %{all_target} = specification_version=1.1.draft level=none \
+// DEFINE:    profiles=pro_int,pro_fp extensions=%{all_extensions}
+
+// RUN: mlir-opt %s -split-input-file -tosa-attach-target="%{all_target}" | FileCheck %s --check-prefix=CHECK-ALL
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="level=8k" | FileCheck %s --check-prefix=CHECK-LVL-8K
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target | FileCheck %s --check-prefix=CHECK-DEFAULT
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="specification_version=1.1.draft" | FileCheck %s --check-prefix=CHECK-VERSION-1P1
 
 // -----
 
-// CHECK-ALL: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = none, profiles = [pro_int, pro_fp], extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft, variable, controlflow, doubleround, inexactround, dynamic]>}
+// CHECK-ALL: module attributes {
+// CHECK-ALL-SAME: tosa.target_env = #tosa.target_env<specification_version = "1.1.draft",
+// CHECK-ALL-SAME: level = none,
+// CHECK-ALL-SAME: profiles = [pro_int, pro_fp],
+// CHECK-ALL-SAME: extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft,
+// CHECK-ALL-SAME: variable, controlflow, doubleround, inexactround, dynamic,
+// CHECK-ALL-SAME: mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2,
+// CHECK-ALL-SAME: mx_fp8e4m3, mx_fp8e5m2, mx_int8]>}
 // CHECK-LVL-8K: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-DEFAULT: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-VERSION-1P1: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.1.draft", level = "8k", profiles = [], extensions = []>}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index 7ff883e8e5431..c7bd051399ae8 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -135,6 +135,14 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
 func.func @test_dyanmic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   // expected-error at +1 {{'tosa.argmax' op failed level check: operand shape dimension cannot be dynamic when targeting TOSA specification version 1.0 or below}}
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 1bac8bacbaf40..f45ba0a5b7ef5 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
+// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape,mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2,mx_fp8e4m3,mx_fp8e5m2,mx_int8" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
 
 // -----
 
@@ -352,6 +352,34 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+// CHECK-LABEL: test_cast_to_block_scaled_types
+func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
+  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+// CHECK-LABEL: test_cast_from_block_scaled_types
+func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %mxint8: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf16>
+  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
+  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E4M3FN>
+  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+  return %0, %3, %4, %5, %6 : tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>
+}
+
+// -----
+
 // CHECK-LABEL: test_dynamic_dims
 func.func @test_dynamic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 9b5faf575971b..14a48d5e10ea0 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1517,6 +1517,48 @@ func.func @test_cast_to_block_scaled_block_size_mismatch(%arg0: tensor<4x32xf32>
 
 // -----
 
+func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i8'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i32'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32>
+  return %0 : tensor<4x32xi32>
+}
+
+// -----
+
+func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+  // expected-error at +1 {{'tosa.cast' op requires exactly one of input or output to have block scaled element type}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  return %0 : tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32> {
+  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
+  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32>
+  return %0 : tensor<1x16x31xf32>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32> {
+  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' does not support block scaling on scalar tensors}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
+  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32>
+  return %0 : tensor<f32>
+}
+
+// -----
+
 func.func @test_clamp_quantized(%arg0:tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) {
     // expected-error at +1 {{'tosa.clamp' op min/max attributes types are incompatible with input/output element types.}}
     %0 = tosa.clamp %arg0 {max_val = 127 : i8, min_val = -128 : i8} : (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>

>From 06c28ec21f372f40f210ac05d76fab6ca074c64a Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 25 Jun 2026 17:40:56 +0100
Subject: [PATCH 2/4] Align block_scale type elements with spec order

Aligns block_scale type with the TOSA specification
definition.

Change-Id: I4d4a6a51e5582a93c8b272fd4f6baf30da3d8e87
---
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 16 +++--
 mlir/test/Dialect/Tosa/canonicalize.mlir      | 58 +++++++++----------
 mlir/test/Dialect/Tosa/invalid.mlir           | 16 ++---
 mlir/test/Dialect/Tosa/invalid_extension.mlir | 22 +++----
 mlir/test/Dialect/Tosa/ops.mlir               | 24 ++++----
 .../tosa-validation-version-1p0-invalid.mlir  |  6 +-
 .../tosa-validation-version-1p1-valid.mlir    | 34 +++++------
 mlir/test/Dialect/Tosa/verifier.mlir          | 32 +++++-----
 8 files changed, 106 insertions(+), 102 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 0a43f14865f56..5ab2c456f365b 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -113,21 +113,25 @@ def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
   let summary = "Block scaled tensor element type.";
 
   let description = [{
-    This does not specify an implementation type. A tensor of this type is a 
+    This does not specify an implementation type. A tensor of this type is a
     tensor which has block-scaled quantization applied.
 
     This compound type is made up of 3 components:
-    `value_type` - The type of the data values in each block.
-    `scale_type` - The type of the scale value associated with each block.
     `block_shape` - The size and axis of each block. Currently only supports
       specifying the block size along the innermost dimension.
+    `scale_type` - The type of the scale value associated with each block.
+    `value_type` - The type of the data values in each block.
+
   }];
 
-  let parameters = (ins Tosa_MXFPValue:$value_type, Tosa_MXFPScale:$scale_type,
-      EnumParameter<Tosa_BlockShape>:$block_shape);
+  let parameters = (ins
+    EnumParameter<Tosa_BlockShape>:$block_shape,
+    Tosa_MXFPScale:$scale_type,
+    Tosa_MXFPValue:$value_type
+  );
 
   let assemblyFormat =
-      "`<` $value_type```:```$scale_type```:```$block_shape `>`";
+      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index d454ca831cf0d..d2da93ee95138 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1652,21 +1652,21 @@ func.func @test_canonicalize_non_narrowing_cast_i8_to_f8E4M3FN_unsupported(%arg0
 // -----
 
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1
-// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf32>
-  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256xf32>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2
-// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
-  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
 }
 
 // -----
@@ -1674,11 +1674,11 @@ func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: t
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<160xf32>
-  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<160x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
 }
 
 // -----
@@ -1686,11 +1686,11 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_differen
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<32xf32>
-  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_different_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>> {
+  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32xf32>
+  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  return %1 : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
 }
 
 // -----
@@ -1698,11 +1698,11 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_differen
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
 // CHECK: %[[values:.+]] = tosa.cast %arg0
 // CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
-// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<*xf32>
-  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<*x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*xf32>
+  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
 }
 
 // -----
@@ -1710,10 +1710,10 @@ func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
 // CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3
 // CHECK: tosa.cast
 // CHECK: tosa.cast
-func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>> {
-  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<15x3x2x256xf8E4M3FN>
-  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %1 : tensor<15x3x2x256x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256xf8E4M3FN>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index eca5c9ef1b14e..d0336da15cee8 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2277,24 +2277,24 @@ func.func @test_cast_f32_plain_fp4(%arg0: tensor<4x32xf32>) -> tensor<4x32xf4E2M
 
 // -----
 
-func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp4e2m1,bs32_fp8e8m0_fp4e2m1), did you mean (fp8e4m3,bs32_fp8e8m0_fp4e2m1)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
-func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN> {
+func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (bs32_fp8e8m0_fp4e2m1,fp6e2m3), did you mean (bs32_fp8e8m0_fp4e2m1,fp8e4m3)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf6E2M3FN>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN>
   return %0 : tensor<4x32xf6E2M3FN>
 }
 
 // -----
 
-func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
   // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp6e3m2,bs32_fp8e8m0_mxint8), did you mean (fp8e4m3,bs32_fp8e8m0_mxint8)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 9d4e59a9a9c6f..ca09ca95066ef 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -296,31 +296,31 @@ func.func @test_cast_f32_bf16(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xbf16
 }
 
 // -----
-func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
-func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32> {
+func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
   return %0 : tensor<4x32xf32>
 }
 
 // -----
-func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
-func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires all of [fp8e4m3, mx_common, mx_int8] profiles/extensions to be specified in the target environment}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 86212177d34be..46595c1e4ea73 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1061,27 +1061,27 @@ func.func @test_cast3(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3x!quant.unifo
 
 // -----
 // CHECK-LABEL: test_cast_to_block_scaled
-func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
 // CHECK-LABEL: test_cast_from_block_scaled
-func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg2: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
-  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xbf16>
+  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E5M2>
   return %0, %1, %2 : tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>
 }
 
 // -----
 // CHECK-LABEL: test_cast_block_scaled_dynamic
-func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %arg1: tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
-  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<?x32xf32>
-  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x?xf32>
+func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
+  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<?x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x?xf32>
   return %0, %1 : tensor<?x32xf32>, tensor<4x?xf32>
 }
 
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index c7bd051399ae8..67237848d6171 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -135,10 +135,10 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
-func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index f45ba0a5b7ef5..ecfe8e1635944 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -353,28 +353,28 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 // -----
 
 // CHECK-LABEL: test_cast_to_block_scaled_types
-func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) {
-  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
-  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>
+func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
 
 // -----
 
 // CHECK-LABEL: test_cast_from_block_scaled_types
-func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>, %mxint8: tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
-  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<f6E2M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<f6E3M2FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf32>
-  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf16>
-  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<f8E5M2:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xbf16>
-  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<!tosa.mxint8:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E4M3FN>
-  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xf8E5M2>
+func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %mxint8: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xf32>
+  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x32xf32>
+  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x32xf16>
+  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xbf16>
+  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E4M3FN>
+  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf8E5M2>
   return %0, %3, %4, %5, %6 : tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>
 }
 
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 14a48d5e10ea0..b2c5527b058aa 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1517,43 +1517,43 @@ func.func @test_cast_to_block_scaled_block_size_mismatch(%arg0: tensor<4x32xf32>
 
 // -----
 
-func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
   // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i8'}}
-  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
 
 // -----
 
-func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32> {
+func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32> {
   // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i32'}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32xi32>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32>
   return %0 : tensor<4x32xi32>
 }
 
 // -----
 
-func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>> {
+func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
   // expected-error at +1 {{'tosa.cast' op requires exactly one of input or output to have block scaled element type}}
-  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<f4E2M1FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
-  return %0 : tensor<4x32x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
 }
 
 // -----
 
-func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
-  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<1x16x31xf32>
+func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
+  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
 }
 
 // -----
 
-func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>' does not support block scaling on scalar tensors}}
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>'}}
-  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<f8E4M3FN:f8E8M0FNU:BLOCK_SHAPE_32>>) -> tensor<f32>
+func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
+  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>
 }
 

>From 3de5539f9bc2d9a1a05038714e0923e78f78b3a9 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Fri, 3 Jul 2026 13:25:36 +0100
Subject: [PATCH 3/4] TypeInfo blockSize -> blockShape

Change-Id: I5c66f8a8fb5934bcf2a8e2988d5491ef0ce4b90d
---
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   | 16 ++++++-------
 .../Tosa/Transforms/TosaProfileCompliance.cpp | 24 ++++++++++++-------
 2 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 4e14b27a421e4..34ccc675c8243 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -26,18 +26,18 @@ using namespace mlir::tosa;
 struct TypeInfo {
   TypeInfo(mlir::TypeID typeID, uint32_t bitWidth)
       : typeID(typeID), bitWidth(bitWidth), valueTypeID(mlir::TypeID()),
-        scaleTypeID(mlir::TypeID()), blockSize(0) {}
+        scaleTypeID(mlir::TypeID()), blockShape(std::nullopt) {}
 
   TypeInfo(mlir::TypeID typeID, uint32_t bitWidth, mlir::TypeID valueTypeID,
-           mlir::TypeID scaleTypeID, uint32_t blockSize)
+           mlir::TypeID scaleTypeID, tosa::BlockShape blockShape)
       : typeID(typeID), bitWidth(bitWidth), valueTypeID(valueTypeID),
-        scaleTypeID(scaleTypeID), blockSize(blockSize) {}
+        scaleTypeID(scaleTypeID), blockShape(blockShape) {}
 
   mlir::TypeID typeID;
   uint32_t bitWidth;
   mlir::TypeID valueTypeID;
   mlir::TypeID scaleTypeID;
-  uint32_t blockSize;
+  std::optional<tosa::BlockShape> blockShape;
 };
 
 enum CheckCondition {
@@ -85,10 +85,8 @@ class ProfileInfoDepot {
     if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type)) {
       Type valueTy = blockScaledTy.getValueType();
       Type scaleTy = blockScaledTy.getScaleType();
-      return {
-          type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
-          scaleTy.getTypeID(),
-          BlockShapeAttr::getBlockShapeValue(blockScaledTy.getBlockShape())};
+      return {type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
+              scaleTy.getTypeID(), blockScaledTy.getBlockShape()};
     }
     return {type.getTypeID(), tosa::getBitWidth(type)};
   }
@@ -150,7 +148,7 @@ class TosaProfileCompliance {
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
     return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
            a.valueTypeID == b.valueTypeID && a.scaleTypeID == b.scaleTypeID &&
-           a.blockSize == b.blockSize;
+           a.blockShape == b.blockShape;
   }
 
   // Find the required profiles or extensions from the compliance info according
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 9fbdfcc1d6690..4f804d32775f7 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -47,17 +47,22 @@ TosaProfileCompliance::TosaProfileCompliance() {
   const TypeID mxint8ID = mlir::tosa::mxint8Type::getTypeID();
 
   const TypeInfo bs32_fp8ue8m0_fp4e2m1T = {blockScaledID, 4, fp4e2m1ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp6e2m3T = {blockScaledID, 6, fp6e2m3ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp6e3m2T = {blockScaledID, 6, fp6e3m2ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp8e4m3T = {blockScaledID, 8, fp8e4m3ID,
-                                           fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
   const TypeInfo bs32_fp8ue8m0_fp8e5m2T = {blockScaledID, 8, fp8e5m2ID,
-                                           fp8ue8m0ID, 32};
-  const TypeInfo bs32_fp8ue8m0_mxint8T = {blockScaledID, 8, mxint8ID,
-                                          fp8ue8m0ID, 32};
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_mxint8T = {
+      blockScaledID, 8, mxint8ID, fp8ue8m0ID, tosa::BlockShape::BLOCK_SHAPE_32};
 
 // The profile-based compliance content below is auto-generated by a script
 // in https://github.com/arm/tosa-specification
@@ -731,8 +736,9 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
     TypeInfo scaleInfo = {typeInfo.scaleTypeID, 8};
     llvm::SmallString<32> result;
     llvm::raw_svector_ostream os(result);
-    os << "bs" << typeInfo.blockSize << "_"
-       << stringifyScalarTypeInfo(scaleInfo) << "_"
+    os << "bs"
+       << tosa::BlockShapeAttr::getBlockShapeValue(typeInfo.blockShape.value())
+       << "_" << stringifyScalarTypeInfo(scaleInfo) << "_"
        << stringifyScalarTypeInfo(valueInfo);
     return result;
   }

>From 19b01b9d0fbf9dcaea3f2537475e4fffeb25c3c9 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Fri, 3 Jul 2026 19:28:28 +0100
Subject: [PATCH 4/4] Address review comments

Change-Id: Id604fc2588ccec8186ce8d7b688e963c8afcbdce
---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td    | 4 ++--
 mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
index 308e98ac42435..392a3ffadca34 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
@@ -251,8 +251,8 @@ class Tosa_I32EnumAttr<string name, string description, string mnemonic,
 // DOUBLEROUND  : Adds double rounding support to the RESCALE operator.
 // INEXACTROUND : Adds inexact rounding support to the RESCALE operator.
 // DYNAMIC      : Removes all Compile Time Constant state for CTC inputs.
-// MXFP         : Microscaling formats.
-// MXFP_CONV    : Microscaling format convolution.
+// MXFP         : Microscaling formats. (Deprecated, this extension will be removed in the future.)
+// MXFP_CONV    : Microscaling format convolution. (Deprecated, this extension will be removed in the future.)
 // SHAPE        : Shape calcuation operators.
 // MX_COMMON    : Base for MXFP microscaling formats.
 // MX_FP4E2M1   : Microscaling format FP4E2M1.
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 99315a787f12e..2493a9f049bf3 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1213,7 +1213,7 @@ struct CancellingBlockScaledCastsOptimization
     const bool isLosslessCast = isa<Float32Type>(innerOutputElemType);
     if (!isLosslessCast)
       return rewriter.notifyMatchFailure(
-          castOp, "avoid cancelling casts that could by lossy");
+          castOp, "avoid cancelling casts that should be lossy");
 
     rewriter.replaceOp(castOp, innerInput);
 


From llvmlistbot at llvm.org  Fri Jul  3 12:04:08 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Fri, 03 Jul 2026 12:04:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Move arch-specific topology
 under uArch check (PR #207449)
Message-ID: <llvm.org/llvm/llvm-project/pull/207449@github.com>


https://github.com/akroviakov created https://github.com/llvm/llvm-project/pull/207449

This PR preserves `order` in dpas layout propagation for inst_data and moves the uarch-specific assumptions under the uarch check.

>From 3606148988591ffbe501e41a524c7372710205f8 Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Fri, 3 Jul 2026 19:02:09 +0000
Subject: [PATCH] [MLIR][XeGPU] Move arch-specific topology under uArch check

---
 .../XeGPU/Transforms/XeGPULayoutImpl.cpp      | 43 +++++++++++--------
 1 file changed, 26 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 678f585bdcca5..6ea55f4c78dd6 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -2003,31 +2003,40 @@ xegpu::completeDpasLaneLayoutFromInstData(xegpu::DistributeLayoutAttr aLayout,
   if (!uArchInstruction)
     return std::nullopt;
   auto subgroupSize = uArch->getSubgroupSize();
-
-  auto [laneLayoutA, laneDataA] = compute2DBlockIOLaneLayoutAndData(
-      aTy.getShape(), subgroupSize,
-      aTy.getElementType().getIntOrFloatBitWidth(),
-      uArchInstruction->getPackedFormatBitSizeA());
-  auto [laneLayoutB, laneDataB] = compute2DBlockIOLaneLayoutAndData(
-      bTy.getShape(), subgroupSize,
-      bTy.getElementType().getIntOrFloatBitWidth(),
-      uArchInstruction->getPackedFormatBitSizeB(), /*vnni=*/true);
-  auto [laneLayoutCD, laneDataCD] = compute2DBlockIOLaneLayoutAndData(
-      cdTy.getShape(), subgroupSize,
-      cdTy.getElementType().getIntOrFloatBitWidth(),
-      cdTy.getElementType().getIntOrFloatBitWidth());
+  llvm::SmallVector<int64_t> laneLayoutA, laneDataA, laneLayoutB, laneDataB,
+      laneLayoutCD, laneDataCD;
   SmallVector<int64_t> instDataA = aLayout.getEffectiveInstDataAsInt();
   SmallVector<int64_t> instDataB = bLayout.getEffectiveInstDataAsInt();
   SmallVector<int64_t> instDataCD = cdLayout.getEffectiveInstDataAsInt();
+
+  if (isa<xegpu::uArch::Xe2, xegpu::uArch::Xe3>(uArch)) {
+    std::tie(laneLayoutA, laneDataA) = compute2DBlockIOLaneLayoutAndData(
+        aTy.getShape(), subgroupSize,
+        aTy.getElementType().getIntOrFloatBitWidth(),
+        uArchInstruction->getPackedFormatBitSizeA());
+    std::tie(laneLayoutB, laneDataB) = compute2DBlockIOLaneLayoutAndData(
+        bTy.getShape(), subgroupSize,
+        bTy.getElementType().getIntOrFloatBitWidth(),
+        uArchInstruction->getPackedFormatBitSizeB(), /*vnni=*/true);
+    std::tie(laneLayoutCD, laneDataCD) = compute2DBlockIOLaneLayoutAndData(
+        cdTy.getShape(), subgroupSize,
+        cdTy.getElementType().getIntOrFloatBitWidth(),
+        cdTy.getElementType().getIntOrFloatBitWidth());
+  } else {
+    assert(false && "Unsupported uArch for DPAS lane layout completion");
+  }
+
   if (!isValidLaneLayout(instDataA, laneLayoutA, laneDataA) ||
       !isValidLaneLayout(instDataB, laneLayoutB, laneDataB) ||
       !isValidLaneLayout(instDataCD, laneLayoutCD, laneDataCD))
     return std::nullopt;
   return std::make_tuple(
-      buildInstDataLayoutWithLane(context, instDataA, laneLayoutA, laneDataA),
-      buildInstDataLayoutWithLane(context, instDataB, laneLayoutB, laneDataB),
-      buildInstDataLayoutWithLane(context, instDataCD, laneLayoutCD,
-                                  laneDataCD));
+      buildInstDataLayoutWithLane(context, instDataA, laneLayoutA, laneDataA,
+                                  aLayout.getOrder()),
+      buildInstDataLayoutWithLane(context, instDataB, laneLayoutB, laneDataB,
+                                  bLayout.getOrder()),
+      buildInstDataLayoutWithLane(context, instDataCD, laneLayoutCD, laneDataCD,
+                                  cdLayout.getOrder()));
 }
 
 /// Like completeDpasLaneLayoutFromInstData, but for dpas_mx: also re-derives


From llvmlistbot at llvm.org  Fri Jul  3 12:04:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 12:04:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Move arch-specific topology
 under uArch check (PR #207449)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207449@github.com>
Message-ID: <6a4807cb.bf38ae28.358429.8eb8@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Artem Kroviakov (akroviakov)

<details>
<summary>Changes</summary>

This PR preserves `order` in dpas layout propagation for inst_data and moves the uarch-specific assumptions under the uarch check.

---
Full diff: https://github.com/llvm/llvm-project/pull/207449.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp (+26-17) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 678f585bdcca5..6ea55f4c78dd6 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -2003,31 +2003,40 @@ xegpu::completeDpasLaneLayoutFromInstData(xegpu::DistributeLayoutAttr aLayout,
   if (!uArchInstruction)
     return std::nullopt;
   auto subgroupSize = uArch->getSubgroupSize();
-
-  auto [laneLayoutA, laneDataA] = compute2DBlockIOLaneLayoutAndData(
-      aTy.getShape(), subgroupSize,
-      aTy.getElementType().getIntOrFloatBitWidth(),
-      uArchInstruction->getPackedFormatBitSizeA());
-  auto [laneLayoutB, laneDataB] = compute2DBlockIOLaneLayoutAndData(
-      bTy.getShape(), subgroupSize,
-      bTy.getElementType().getIntOrFloatBitWidth(),
-      uArchInstruction->getPackedFormatBitSizeB(), /*vnni=*/true);
-  auto [laneLayoutCD, laneDataCD] = compute2DBlockIOLaneLayoutAndData(
-      cdTy.getShape(), subgroupSize,
-      cdTy.getElementType().getIntOrFloatBitWidth(),
-      cdTy.getElementType().getIntOrFloatBitWidth());
+  llvm::SmallVector<int64_t> laneLayoutA, laneDataA, laneLayoutB, laneDataB,
+      laneLayoutCD, laneDataCD;
   SmallVector<int64_t> instDataA = aLayout.getEffectiveInstDataAsInt();
   SmallVector<int64_t> instDataB = bLayout.getEffectiveInstDataAsInt();
   SmallVector<int64_t> instDataCD = cdLayout.getEffectiveInstDataAsInt();
+
+  if (isa<xegpu::uArch::Xe2, xegpu::uArch::Xe3>(uArch)) {
+    std::tie(laneLayoutA, laneDataA) = compute2DBlockIOLaneLayoutAndData(
+        aTy.getShape(), subgroupSize,
+        aTy.getElementType().getIntOrFloatBitWidth(),
+        uArchInstruction->getPackedFormatBitSizeA());
+    std::tie(laneLayoutB, laneDataB) = compute2DBlockIOLaneLayoutAndData(
+        bTy.getShape(), subgroupSize,
+        bTy.getElementType().getIntOrFloatBitWidth(),
+        uArchInstruction->getPackedFormatBitSizeB(), /*vnni=*/true);
+    std::tie(laneLayoutCD, laneDataCD) = compute2DBlockIOLaneLayoutAndData(
+        cdTy.getShape(), subgroupSize,
+        cdTy.getElementType().getIntOrFloatBitWidth(),
+        cdTy.getElementType().getIntOrFloatBitWidth());
+  } else {
+    assert(false && "Unsupported uArch for DPAS lane layout completion");
+  }
+
   if (!isValidLaneLayout(instDataA, laneLayoutA, laneDataA) ||
       !isValidLaneLayout(instDataB, laneLayoutB, laneDataB) ||
       !isValidLaneLayout(instDataCD, laneLayoutCD, laneDataCD))
     return std::nullopt;
   return std::make_tuple(
-      buildInstDataLayoutWithLane(context, instDataA, laneLayoutA, laneDataA),
-      buildInstDataLayoutWithLane(context, instDataB, laneLayoutB, laneDataB),
-      buildInstDataLayoutWithLane(context, instDataCD, laneLayoutCD,
-                                  laneDataCD));
+      buildInstDataLayoutWithLane(context, instDataA, laneLayoutA, laneDataA,
+                                  aLayout.getOrder()),
+      buildInstDataLayoutWithLane(context, instDataB, laneLayoutB, laneDataB,
+                                  bLayout.getOrder()),
+      buildInstDataLayoutWithLane(context, instDataCD, laneLayoutCD, laneDataCD,
+                                  cdLayout.getOrder()));
 }
 
 /// Like completeDpasLaneLayoutFromInstData, but for dpas_mx: also re-derives

``````````

</details>


https://github.com/llvm/llvm-project/pull/207449

From llvmlistbot at llvm.org  Fri Jul  3 12:29:14 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:29:14 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480d8a.bb839515.23e6d3.2f66@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 1/8] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 2/8] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 3/8] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 4/8] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 5/8] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 6/8] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 7/8] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 8/8] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the


From llvmlistbot at llvm.org  Fri Jul  3 12:29:52 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:29:52 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480db0.01f0a1da.2016e9.7699@mx.google.com>


dzhidzhoev wrote:

> Thanks, Vladislav, for driving this work forward and for all the effort you've put into these activities! It's great to see this patch getting close to the finish line.
> 
> I've reviewed the changes and left a few comments and suggestions that may help improve readability, but overall this looks good to me. Thanks again for taking this on and pushing it through.

Thank you for the feedback! I updated the code to address it.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul  3 12:30:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 12:30:44 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480de4.bb839515.23e6d3.638c@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- clang/test/OpenMP/error_codegen.cpp flang/lib/Optimizer/Transforms/AddDebugInfo.cpp llvm/include/llvm/IR/DIBuilder.h llvm/include/llvm/IR/DebugInfo.h llvm/include/llvm/IR/DebugInfoMetadata.h llvm/include/llvm/IR/Metadata.h llvm/lib/Bitcode/Reader/MetadataLoader.cpp llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp llvm/lib/IR/DIBuilder.cpp llvm/lib/IR/DebugInfo.cpp llvm/lib/IR/DebugInfoMetadata.cpp llvm/lib/IR/Verifier.cpp llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp llvm/lib/Target/BPF/BTFDebug.cpp llvm/lib/Transforms/Coroutines/CoroFrame.cpp llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp llvm/lib/Transforms/Utils/CloneFunction.cpp llvm/lib/Transforms/Utils/Debugify.cpp llvm/unittests/Transforms/Utils/CloningTest.cpp mlir/lib/CAPI/Dialect/LLVM.cpp mlir/lib/Target/LLVMIR/DebugImporter.cpp mlir/lib/Target/LLVMIR/DebugTranslation.cpp mlir/lib/Target/LLVMIR/DebugTranslation.h mlir/lib/Target/LLVMIR/ModuleTranslation.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 62523ffef..846342bee 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2548,9 +2548,7 @@ public:
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) {
-    replaceOperandWith(7, N.get());
-  }
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();

``````````

</details>


https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul  3 12:35:57 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:35:57 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480f1d.01f0a1da.2016e9.1c74@mx.google.com>


================
@@ -2124,6 +2124,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
----------------
dzhidzhoev wrote:

> Given that operand numbers are still hardcoded explicitly in other places, I'd prefer to leave these three as they were.

Hmm, I think it goes back to the time when there were no DINode (and children) classes, and all metadata were stored in plain MDNodes. Not sure if this convention should be followed by new code.

> I'd rather not do this, it looks misleading to me. It's not clear what the purpose of these globals is or which metadata kind they belong to.

These constants are defined inside DICompileUnit class, and are used in getter and setters right there. For me it's rather clear, hm.
Anyway, I don't mind conventional approach.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul  3 12:37:06 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:37:06 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480f62.ff037943.3bf0e2.d4bd@mx.google.com>


https://github.com/dzhidzhoev edited https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul  3 12:37:24 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:37:24 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480f74.7d41f3a1.d1cf7.3125@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 1/9] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 2/9] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 3/9] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 4/9] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 5/9] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 6/9] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 7/9] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 8/9] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 9/9] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.


From llvmlistbot at llvm.org  Fri Jul  3 12:37:33 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:37:33 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a480f7d.331483e7.5bc1e.d3f2@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 01/10] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 02/10] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 03/10] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 04/10] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 05/10] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 06/10] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 07/10] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 08/10] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 09/10] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.

>From 39b303e345d8d744745fad8bef4423ed6ce75c2d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:18 +0200
Subject: [PATCH 10/10] Update mlir/lib/Target/LLVMIR/DebugTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fca5e0a8497eb..10ae5ef0ec646 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,7 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (DIGlobalVariableExpressionAttr GVE =
+        if (auto GVE =
                 dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 


From llvmlistbot at llvm.org  Fri Jul  3 12:44:28 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 03 Jul 2026 12:44:28 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Explicitly specify target
 feature for module asm (PR #204548)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204548@github.com>
Message-ID: <6a48111c.a87eb5dc.2cf3a1.6818@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-01` while building `clang,llvm,mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11630

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../BenchmarkResult.h:18,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../Target.h:19,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/Target.cpp:8:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8317/8682] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/PrettyClassLayoutGraphicalDumper.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8319/8682] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/PrettyCompilandDumper.cpp.o
[8320/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/MachOUtils.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachOUtils.cpp:11:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8321/8682] Building CXX object tools/llc/CMakeFiles/llc.dir/llc.cpp.o
[8322/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/MachODebugMapParser.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachODebugMapParser.cpp:10:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8323/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/dsymutil.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.h:20,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.cpp:13:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8324/8682] Building CXX object tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/clang-import-test.cpp.o
[8325/8682] Building CXX object tools/llvm-dis/CMakeFiles/llvm-dis.dir/llvm-dis.cpp.o
[8326/8682] Building CXX object tools/clang/tools/clang-sycl-linker/CMakeFiles/clang-sycl-linker.dir/ClangSYCLLinker.cpp.o
Step 7 (build cmake config) failure: build cmake config (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../BenchmarkResult.h:18,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../Target.h:19,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/Target.cpp:8:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/X86/../Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8317/8682] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/PrettyClassLayoutGraphicalDumper.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8319/8682] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/PrettyCompilandDumper.cpp.o
[8320/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/MachOUtils.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachOUtils.cpp:11:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8321/8682] Building CXX object tools/llc/CMakeFiles/llc.dir/llc.cpp.o
[8322/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/MachODebugMapParser.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachODebugMapParser.cpp:10:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8323/8682] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/dsymutil.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.h:20,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.cpp:13:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8324/8682] Building CXX object tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/clang-import-test.cpp.o
[8325/8682] Building CXX object tools/llvm-dis/CMakeFiles/llvm-dis.dir/llvm-dis.cpp.o
[8326/8682] Building CXX object tools/clang/tools/clang-sycl-linker/CMakeFiles/clang-sycl-linker.dir/ClangSYCLLinker.cpp.o

```

</details>

https://github.com/llvm/llvm-project/pull/204548

From llvmlistbot at llvm.org  Fri Jul  3 12:45:11 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 03 Jul 2026 12:45:11 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a481147.70920dee.c0a01.0616@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 01/12] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 02/12] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 03/12] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 04/12] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 05/12] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 06/12] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 07/12] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 08/12] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 09/12] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.

>From 39b303e345d8d744745fad8bef4423ed6ce75c2d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:18 +0200
Subject: [PATCH 10/12] Update mlir/lib/Target/LLVMIR/DebugTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fca5e0a8497eb..10ae5ef0ec646 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,7 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (DIGlobalVariableExpressionAttr GVE =
+        if (auto GVE =
                 dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 

>From 292e16924dd039876c60e17afc4fa491e448e79d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:36:41 +0200
Subject: [PATCH 11/12] Clang-format fix

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  4 +-
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 52 +++++++++----------
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     | 32 ++++++------
 3 files changed, 41 insertions(+), 47 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 62523ffef3895..846342bee6071 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2548,9 +2548,7 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) {
-    replaceOperandWith(7, N.get());
-  }
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 29b010ee37f5b..72497a5328f78 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -710,36 +710,34 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
                                       [LLVM_DIRecursiveTypeAttrInterface],
                                       "DILocalScopeAttr"> {
   let parameters = (ins
-    // DIRecursiveTypeAttrInterface specific parameters.
-    OptionalParameter<"DistinctAttr">:$recId,
-    OptionalParameter<"bool">:$isRecSelf,
-    // DISubprogramAttr specific parameters.
-    OptionalParameter<"DistinctAttr">:$id,
-    OptionalParameter<"DICompileUnitAttr">:$compileUnit,
-    OptionalParameter<"DIScopeAttr">:$scope,
-    OptionalParameter<"StringAttr">:$name,
-    OptionalParameter<"StringAttr">:$linkageName,
-    OptionalParameter<"DIFileAttr">:$file,
-    OptionalParameter<"unsigned">:$line,
-    OptionalParameter<"unsigned">:$scopeLine,
-    OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
-    OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
-    OptionalArrayRefParameter<"DINodeAttr">:$annotations
-  );
-  let builders = [
-    AttrBuilder<(ins
-      "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
-      "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
-      "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
-      "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
-    ), [{
+      // DIRecursiveTypeAttrInterface specific parameters.
+      OptionalParameter<"DistinctAttr">:$recId,
+      OptionalParameter<"bool">:$isRecSelf,
+      // DISubprogramAttr specific parameters.
+      OptionalParameter<"DistinctAttr">:$id,
+      OptionalParameter<"DICompileUnitAttr">:$compileUnit,
+      OptionalParameter<"DIScopeAttr">:$scope,
+      OptionalParameter<"StringAttr">:$name,
+      OptionalParameter<"StringAttr">:$linkageName,
+      OptionalParameter<"DIFileAttr">:$file,
+      OptionalParameter<"unsigned">:$line,
+      OptionalParameter<"unsigned">:$scopeLine,
+      OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
+      OptionalParameter<"DISubroutineTypeAttr">:$type,
+      OptionalArrayRefParameter<"Attribute">:$retainedNodes,
+      OptionalArrayRefParameter<"DINodeAttr">:$annotations);
+  let builders = [AttrBuilder<
+      (ins "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
+          "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
+          "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
+          "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
+          "ArrayRef<Attribute>":$retainedNodes,
+          "ArrayRef<DINodeAttr>":$annotations),
+      [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
                    subprogramFlags, type, retainedNodes, annotations);
-    }]>
-  ];
+    }]>];
   let assemblyFormat = "`<` struct(params) `>`";
   let extraClassDeclaration = [{
     /// Requirements of DIRecursiveTypeAttrInterface.
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index 0c78689fe1d6b..b25a8c8a9829e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -150,23 +150,21 @@ def DICompileUnitAttr : DialectAttribute<(attr
 // DISubprogramAttr
 //===----------------------------------------------------------------------===//
 
-def DISubprogramAttr : DialectAttribute<(attr
-  OptionalAttribute<"DistinctAttr">:$recId,
-  Bool:$isRecSelf,
-  OptionalAttribute<"DistinctAttr">:$id,
-  OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
-  OptionalAttribute<"DIScopeAttr">:$scope,
-  OptionalAttribute<"StringAttr">:$name,
-  OptionalAttribute<"StringAttr">:$linkageName,
-  OptionalAttribute<"DIFileAttr">:$file,
-  VarInt:$line,
-  VarInt:$scopeLine,
-  EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
-  LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
-  OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"Attribute">:$retainedNodes,
-  OptionalArrayRef<"DINodeAttr">:$annotations
-)>;
+def DISubprogramAttr
+    : DialectAttribute<(attr OptionalAttribute<"DistinctAttr">:$recId,
+          Bool:$isRecSelf, OptionalAttribute<"DistinctAttr">:$id,
+          OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
+          OptionalAttribute<"DIScopeAttr">:$scope,
+          OptionalAttribute<"StringAttr">:$name,
+          OptionalAttribute<"StringAttr">:$linkageName,
+          OptionalAttribute<"DIFileAttr">:$file, VarInt:$line,
+          VarInt:$scopeLine,
+          EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
+          LocalVar<"DISubprogramFlags",
+                   "(DISubprogramFlags)_rawflags">:$subprogramFlags,
+          OptionalAttribute<"DISubroutineTypeAttr">:$type,
+          OptionalArrayRef<"Attribute">:$retainedNodes,
+          OptionalArrayRef<"DINodeAttr">:$annotations)>;
 
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr

>From 3b92877dfd5c5ffbc0b1c557a78bcbf6cafce502 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:44:51 +0200
Subject: [PATCH 12/12] Clang-format fix

---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 10ae5ef0ec646..442f5b6b137e6 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,8 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (auto GVE =
-                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+        if (auto GVE = dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 
         auto diAttr = dyn_cast<DINodeAttr>(attr);


From llvmlistbot at llvm.org  Fri Jul  3 12:57:07 2026
From: llvmlistbot at llvm.org (Kristina Bessonova)
Date: Fri, 03 Jul 2026 12:57:07 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a481413.faf16b37.1e0c59.7f4e@mx.google.com>


================
@@ -2124,6 +2124,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
----------------
chbessonova wrote:

My bad, I misread the code. I initially thought these were new globals, but I now see they're class members. My previous comment was based on that incorrect interpretation. Please disregard it.

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul  3 13:59:16 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 03 Jul 2026 13:59:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a4822a4.e55dc341.29c9b8.d7ad@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11653

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8651/8682] Creating library symlink lib/libFlangOpenMPTransforms.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/r
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8653/8682] Linking CXX shared library lib/libFIRTransforms.so.23.0git
[8654/8682] Linking CXX executable bin/fir-lsp-server
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8651/8682] Creating library symlink lib/libFlangOpenMPTransforms.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/r
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8653/8682] Linking CXX shared library lib/libFIRTransforms.so.23.0git
[8654/8682] Linking CXX executable bin/fir-lsp-server
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=76.355883

```

</details>

https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Fri Jul  3 14:47:17 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 03 Jul 2026 14:47:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Guard getSizeInBytes()
 optionals in UnifyAliasedResourcePass rewriters (PR #207325)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207325@github.com>
Message-ID: <6a482de5.2020f6c2.1fdf61.e653@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-01` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11667

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
[123/1085] Building ASM object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_vfork.S.o
[124/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_errors.cpp.o
[125/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_suppressions.cpp.o
[126/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_malloc_linux.cpp.o
[127/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_exceptions.cpp.o
[128/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_stats.cpp.o
[129/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan_cxx.x86_64.dir/asan_new_delete.cpp.o
[130/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_dynamic_shadow.cpp.o
[131/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_globals.cpp.o
[132/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o
FAILED: compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o 
/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ --target=x86_64-unknown-linux-gnu -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/compiler-rt/lib/asan/.. -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -Wall -Wno-unused-parameter -O3 -DNDEBUG -std=c++17 -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -ftrivial-auto-var-init=pattern -nostdinc++ -fno-rtti -Wno-format -MD -MT compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o -MF compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o.d -o compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o -c /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp
<inline asm>:8:5: error: 32 bit reloc applied to a field with a different size
    8 | jmp __interceptor_memset at plt
      |     ^
<inline asm>:18:5: error: 32 bit reloc applied to a field with a different size
   18 | jmp __interceptor_memmove at plt
      |     ^
<inline asm>:28:5: error: 32 bit reloc applied to a field with a different size
   28 | jmp __interceptor_memcpy at plt
      |     ^
<inline asm>:38:5: error: 32 bit reloc applied to a field with a different size
   38 | jmp __interceptor___bzero at plt
      |     ^
<inline asm>:48:5: error: 32 bit reloc applied to a field with a different size
   48 | jmp __interceptor_bzero at plt
      |     ^
5 errors generated.
[133/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_poisoning.cpp.o
[134/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_rtl.cpp.o
[135/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan_preinit.x86_64.dir/hwasan_preinit.cpp.o
[136/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_allocation_functions.cpp.o
[137/1085] Building ASM object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_interceptors_vfork.S.o
[138/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_type_test.cpp.o
[139/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_poisoning.cpp.o
[140/1085] Building ASM object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_setjmp_x86_64.S.o
[141/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_globals.cpp.o
[142/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_report.cpp.o
[143/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_memintrinsics.cpp.o
[144/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_thread_list.cpp.o
[145/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_fake_stack.cpp.o
[146/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_thread.cpp.o
[147/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan_cxx.x86_64.dir/hwasan_new_delete.cpp.o
[148/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_linux.cpp.o
[149/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasanAliases.x86_64.dir/hwasan_allocation_functions.cpp.o
[150/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_thread.cpp.o
[151/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan.cpp.o
[152/1085] Building CXX object compiler-rt/lib/lsan/CMakeFiles/RTLSanCommon.x86_64.dir/lsan_common.cpp.o
[153/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_report.cpp.o
Step 7 (build cmake config) failure: build cmake config (failure)
...
[123/1085] Building ASM object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_vfork.S.o
[124/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_errors.cpp.o
[125/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_suppressions.cpp.o
[126/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_malloc_linux.cpp.o
[127/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_exceptions.cpp.o
[128/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_stats.cpp.o
[129/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan_cxx.x86_64.dir/asan_new_delete.cpp.o
[130/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_dynamic_shadow.cpp.o
[131/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_globals.cpp.o
[132/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o
FAILED: compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o 
/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ --target=x86_64-unknown-linux-gnu -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/compiler-rt/lib/asan/.. -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -Wall -Wno-unused-parameter -O3 -DNDEBUG -std=c++17 -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -Wthread-safety -Wthread-safety-reference -Wthread-safety-beta -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -ftrivial-auto-var-init=pattern -nostdinc++ -fno-rtti -Wno-format -MD -MT compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o -MF compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o.d -o compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_interceptors_memintrinsics.cpp.o -c /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp
<inline asm>:8:5: error: 32 bit reloc applied to a field with a different size
    8 | jmp __interceptor_memset at plt
      |     ^
<inline asm>:18:5: error: 32 bit reloc applied to a field with a different size
   18 | jmp __interceptor_memmove at plt
      |     ^
<inline asm>:28:5: error: 32 bit reloc applied to a field with a different size
   28 | jmp __interceptor_memcpy at plt
      |     ^
<inline asm>:38:5: error: 32 bit reloc applied to a field with a different size
   38 | jmp __interceptor___bzero at plt
      |     ^
<inline asm>:48:5: error: 32 bit reloc applied to a field with a different size
   48 | jmp __interceptor_bzero at plt
      |     ^
5 errors generated.
[133/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_poisoning.cpp.o
[134/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_rtl.cpp.o
[135/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan_preinit.x86_64.dir/hwasan_preinit.cpp.o
[136/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_allocation_functions.cpp.o
[137/1085] Building ASM object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_interceptors_vfork.S.o
[138/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_type_test.cpp.o
[139/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_poisoning.cpp.o
[140/1085] Building ASM object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_setjmp_x86_64.S.o
[141/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_globals.cpp.o
[142/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_report.cpp.o
[143/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_memintrinsics.cpp.o
[144/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_thread_list.cpp.o
[145/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_fake_stack.cpp.o
[146/1085] Building CXX object compiler-rt/lib/asan/CMakeFiles/RTAsan.x86_64.dir/asan_thread.cpp.o
[147/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan_cxx.x86_64.dir/hwasan_new_delete.cpp.o
[148/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_linux.cpp.o
[149/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasanAliases.x86_64.dir/hwasan_allocation_functions.cpp.o
[150/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_thread.cpp.o
[151/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan.cpp.o
[152/1085] Building CXX object compiler-rt/lib/lsan/CMakeFiles/RTLSanCommon.x86_64.dir/lsan_common.cpp.o
[153/1085] Building CXX object compiler-rt/lib/hwasan/CMakeFiles/RTHwasan.x86_64.dir/hwasan_report.cpp.o

```

</details>

https://github.com/llvm/llvm-project/pull/207325

From llvmlistbot at llvm.org  Fri Jul  3 21:49:29 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 21:49:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
Message-ID: <llvm.org/llvm/llvm-project/pull/207482@github.com>


https://github.com/jpienaar created https://github.com/llvm/llvm-project/pull/207482

Serialize IntegerSetAttr using VarInt for numDims and numSymbols and arrays for contraints and eqFlags. Add ArrayWithKnownSize` an Array variant that omits the length prefix when the size is known from a previously serialized member. This avoids encoding redundant array lengths when, for example, two arrays are always the same size.

Assisted-By: Gemini

>From 47026d2a85a42d284230e368c61569ae8f5ed0a2 Mon Sep 17 00:00:00 2001
From: Jacques Pienaar <jpienaar at google.com>
Date: Mon, 23 Feb 2026 11:33:39 +0200
Subject: [PATCH] [mlirbc] Add IntegerSetAttr

Serialize IntegerSetAttr using VarInt for numDims and numSymbols and arrays for
contraints and eqFlags. Add ArrayWithKnownSize` an Array variant that omits the
length prefix when the size is known from a previously serialized member. This
avoids encoding redundant array lengths when, for example, two arrays are
always the same size.

Assisted-By: Gemini
---
 .../mlir/Bytecode/BytecodeImplementation.h    | 20 +++++++++++++
 .../include/mlir/IR/BuiltinDialectBytecode.td | 29 +++++++++++++++++++
 mlir/include/mlir/IR/BytecodeBase.td          | 10 ++++++-
 mlir/lib/IR/BuiltinDialectBytecode.cpp        | 19 ++++++------
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir | 16 ++++++++++
 mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp | 14 +++++++--
 6 files changed, 95 insertions(+), 13 deletions(-)

diff --git a/mlir/include/mlir/Bytecode/BytecodeImplementation.h b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
index f1381785a2703..d8d0ca7b3b9f7 100644
--- a/mlir/include/mlir/Bytecode/BytecodeImplementation.h
+++ b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
@@ -76,6 +76,18 @@ class DialectBytecodeReader {
     uint64_t size;
     if (failed(readVarInt(size)))
       return failure();
+    return readListWithKnownSize(result, size,
+                                 std::forward<CallbackFn>(callback));
+  }
+
+  /// Read out a list of elements with a known size, invoking the provided
+  /// callback for each element. Unlike readList, this does not read a length
+  /// prefix. The callback function may be in any of the following forms:
+  ///   * LogicalResult(T &)
+  ///   * FailureOr<T>()
+  template <typename T, typename CallbackFn>
+  LogicalResult readListWithKnownSize(SmallVectorImpl<T> &result, uint64_t size,
+                                      CallbackFn &&callback) {
     result.reserve(size);
 
     for (uint64_t i = 0; i < size; ++i) {
@@ -294,6 +306,14 @@ class DialectBytecodeWriter {
       callback(element);
   }
 
+  /// Write out a list of elements without a length prefix, for cases where the
+  /// size is known from another field.
+  template <typename RangeT, typename CallbackFn>
+  void writeListWithKnownSize(RangeT &&range, CallbackFn &&callback) {
+    for (auto &element : range)
+      callback(element);
+  }
+
   /// Write a reference to the given attribute.
   virtual void writeAttribute(Attribute attr) = 0;
   virtual void writeOptionalAttribute(Attribute attr) = 0;
diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index a3a0511791613..f087e5cb23081 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -214,6 +214,34 @@ def AffineMapAttr : EnableAffineMapPrintingJuly2026<(attr
     WithType<"AffineMap">>>:$value
 )>;
 
+def AffineExpr :
+  WithParser<"succeeded(readAffineExpr($_reader, context, $_var))",
+  WithBuilder<"$_args",
+  WithPrinter<"writeAffineExpr($_writer, $_getter)",
+  WithType<"AffineExpr">>>>;
+def AffineExprList : List<AffineExpr>;
+
+// Similar to AffineMapAttr, IntegerSetAttr has everything going via IntegerSet
+// and getValue, so extra indirections.
+def IntegerSetConstraints : Array<AffineExprList> {
+  let cGetter = "$_attrType.getValue().getConstraints()";
+}
+
+def IntegerSetEqFlags : ArrayWithKnownSize<BoolList, "constraints.size()"> {
+  let cGetter = "$_attrType.getValue().getEqFlags()";
+}
+
+class EnableIntegerSetPrintingJuly2026<dag d> : DialectType<d>;
+
+def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
+  WithGetter<"$_attrType.getValue().getNumDims()", VarInt>:$numDims,
+  WithGetter<"$_attrType.getValue().getNumSymbols()", VarInt>:$numSymbols,
+  IntegerSetConstraints:$constraints,
+  IntegerSetEqFlags:$eqFlags
+)> {
+  let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
+}
+
 // Types
 // -----
 
@@ -388,6 +416,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     DistinctAttr,
     FileLineColRange,
     AffineMapAttr,
+    IntegerSetAttr,
   ];
 }
 
diff --git a/mlir/include/mlir/IR/BytecodeBase.td b/mlir/include/mlir/IR/BytecodeBase.td
index df60800ef639b..3eab62493ffef 100644
--- a/mlir/include/mlir/IR/BytecodeBase.td
+++ b/mlir/include/mlir/IR/BytecodeBase.td
@@ -125,6 +125,15 @@ class Array<Bytecode t> {
   Bytecode elemT = t;
 
   string cBuilder = "$_args";
+
+  // Optional custom getter expression for the array from the parent type.
+  string cGetter = "";
+}
+// - Array variant where the size is known from a previously serialized member.
+//   This avoids encoding a redundant length prefix.
+class ArrayWithKnownSize<Bytecode t, string sizeRef> : Array<t> {
+ // Expression providing the element count (e.g., "otherMember.size()").
+  string knownSizeRef = sizeRef;
 }
 // - Array elements currently needs a different bytecode type to accommodate
 //   for the list print/parsing.
@@ -177,4 +186,3 @@ def none;
 def ReservedOrDead : DialectAttrOrType<(none)>;
 
 #endif // BYTECODE_BASE
-
diff --git a/mlir/lib/IR/BuiltinDialectBytecode.cpp b/mlir/lib/IR/BuiltinDialectBytecode.cpp
index 366b3ceb06410..e5476082cd6a0 100644
--- a/mlir/lib/IR/BuiltinDialectBytecode.cpp
+++ b/mlir/lib/IR/BuiltinDialectBytecode.cpp
@@ -16,6 +16,7 @@
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Diagnostics.h"
 #include "mlir/IR/DialectResourceBlobManager.h"
+#include "mlir/IR/IntegerSet.h"
 #include "mlir/IR/Location.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -157,8 +158,8 @@ static AffineExprKind fromBytecodeKind(uint64_t kind) {
 /// is prefix order (operator and then children), which is self-delimiting.
 /// Instead of C++ recursion the reader uses an explicit work stack, bounding
 /// memory to O(depth) and eliminating stack-overflow risk on malicious input.
-static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
-                                            MLIRContext *context) {
+static LogicalResult readAffineExpr(DialectBytecodeReader &reader,
+                                    MLIRContext *context, AffineExpr &expr) {
   // A work-stack item is either ReadOperand (0) or a combine marker whose
   // payload is an AffineExprKind.
   struct WorkItem {
@@ -175,15 +176,14 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
   while (!work.empty()) {
     // Bound total iterations to catch malformed input.
     if (work.size() > 128)
-      return reader.emitError("AffineExpr work stack overflow"), failure();
+      return reader.emitError("AffineExpr work stack overflow");
 
     WorkItem item = work.pop_back_val();
 
     if (item.isCombine) {
       // Pop two operands and combine.
       if (operands.size() < 2)
-        return reader.emitError("malformed AffineExpr: operand underflow"),
-               failure();
+        return reader.emitError("malformed AffineExpr: operand underflow");
       AffineExpr rhs = operands.pop_back_val();
       AffineExpr lhs = operands.pop_back_val();
       operands.push_back(getAffineBinaryOpExpr(item.combineKind, lhs, rhs));
@@ -240,7 +240,8 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
     return reader.emitError("malformed AffineExpr: expected single result"),
            failure();
 
-  return operands.front();
+  expr = operands.front();
+  return success();
 }
 
 /// Write an AffineExpr in prefix order (operator first, then children).
@@ -337,10 +338,10 @@ static LogicalResult readAffineMap(DialectBytecodeReader &reader,
     SmallVector<AffineExpr> results;
     results.reserve(numResults);
     for (uint64_t i = 0; i < numResults; ++i) {
-      auto expr = readAffineExpr(reader, context);
-      if (failed(expr))
+      AffineExpr expr;
+      if (failed(readAffineExpr(reader, context, expr)))
         return failure();
-      results.push_back(*expr);
+      results.push_back(expr);
     }
     map = AffineMap::get(numDims, numSymbols, results, context);
     return success();
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index aac1c44248995..3364a8e9fce8e 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -255,3 +255,19 @@ module @TestAffineMap attributes {
   bytecode.projperm_single = affine_map<(d0, d1, d2) -> (d2)>
 } {}
 
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {}
+
diff --git a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
index dd178b5e5d232..031fe6cc94eb2 100644
--- a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
+++ b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
@@ -183,7 +183,12 @@ static void printParseConditional(mlir::raw_indented_ostream &ios,
             parser = "succeeded($_reader.readAttributes($_var))";
           else if (!composite && def->isSubClassOf("TypeKind"))
             parser = "succeeded($_reader.readTypes($_var))";
-          else
+          else if (attr->isSubClassOf("ArrayWithKnownSize")) {
+            std::string sizeRef = attr->getValueAsString("knownSizeRef").str();
+            parser = ("succeeded($_reader.readListWithKnownSize($_var, " +
+                      sizeRef + ", " + listHelperName(std::get<1>(it)) + "))")
+                         .str();
+          } else
             parser = ("succeeded($_reader.readList($_var, " +
                       listHelperName(std::get<1>(it)) + "))")
                          .str();
@@ -385,8 +390,11 @@ void Generator::emitPrintHelper(const Record *memberRec, StringRef kind,
     }
     std::string returnType = getCType(def);
     std::string nestedName = kind.str();
-    ios << "writer.writeList(" << getter << ", [&](" << returnType << " "
-        << nestedName << ") ";
+    StringRef writeMethod = memberRec->isSubClassOf("ArrayWithKnownSize")
+                                ? "writer.writeListWithKnownSize("
+                                : "writer.writeList(";
+    ios << writeMethod << getter << ", [&](" << returnType << " " << nestedName
+        << ") ";
     auto lambdaScope = ios.scope("{\n", "});\n");
     return emitPrintHelper(def, kind, nestedName, nestedName, ios);
   }


From llvmlistbot at llvm.org  Fri Jul  3 21:50:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 21:50:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207482@github.com>
Message-ID: <6a4890f8.70920dee.c0a01.525d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Jacques Pienaar (jpienaar)

<details>
<summary>Changes</summary>

Serialize IntegerSetAttr using VarInt for numDims and numSymbols and arrays for contraints and eqFlags. Add ArrayWithKnownSize` an Array variant that omits the length prefix when the size is known from a previously serialized member. This avoids encoding redundant array lengths when, for example, two arrays are always the same size.

Assisted-By: Gemini

---
Full diff: https://github.com/llvm/llvm-project/pull/207482.diff


6 Files Affected:

- (modified) mlir/include/mlir/Bytecode/BytecodeImplementation.h (+20) 
- (modified) mlir/include/mlir/IR/BuiltinDialectBytecode.td (+29) 
- (modified) mlir/include/mlir/IR/BytecodeBase.td (+9-1) 
- (modified) mlir/lib/IR/BuiltinDialectBytecode.cpp (+10-9) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/attrs.mlir (+16) 
- (modified) mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp (+11-3) 


``````````diff
diff --git a/mlir/include/mlir/Bytecode/BytecodeImplementation.h b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
index f1381785a2703..d8d0ca7b3b9f7 100644
--- a/mlir/include/mlir/Bytecode/BytecodeImplementation.h
+++ b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
@@ -76,6 +76,18 @@ class DialectBytecodeReader {
     uint64_t size;
     if (failed(readVarInt(size)))
       return failure();
+    return readListWithKnownSize(result, size,
+                                 std::forward<CallbackFn>(callback));
+  }
+
+  /// Read out a list of elements with a known size, invoking the provided
+  /// callback for each element. Unlike readList, this does not read a length
+  /// prefix. The callback function may be in any of the following forms:
+  ///   * LogicalResult(T &)
+  ///   * FailureOr<T>()
+  template <typename T, typename CallbackFn>
+  LogicalResult readListWithKnownSize(SmallVectorImpl<T> &result, uint64_t size,
+                                      CallbackFn &&callback) {
     result.reserve(size);
 
     for (uint64_t i = 0; i < size; ++i) {
@@ -294,6 +306,14 @@ class DialectBytecodeWriter {
       callback(element);
   }
 
+  /// Write out a list of elements without a length prefix, for cases where the
+  /// size is known from another field.
+  template <typename RangeT, typename CallbackFn>
+  void writeListWithKnownSize(RangeT &&range, CallbackFn &&callback) {
+    for (auto &element : range)
+      callback(element);
+  }
+
   /// Write a reference to the given attribute.
   virtual void writeAttribute(Attribute attr) = 0;
   virtual void writeOptionalAttribute(Attribute attr) = 0;
diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index a3a0511791613..f087e5cb23081 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -214,6 +214,34 @@ def AffineMapAttr : EnableAffineMapPrintingJuly2026<(attr
     WithType<"AffineMap">>>:$value
 )>;
 
+def AffineExpr :
+  WithParser<"succeeded(readAffineExpr($_reader, context, $_var))",
+  WithBuilder<"$_args",
+  WithPrinter<"writeAffineExpr($_writer, $_getter)",
+  WithType<"AffineExpr">>>>;
+def AffineExprList : List<AffineExpr>;
+
+// Similar to AffineMapAttr, IntegerSetAttr has everything going via IntegerSet
+// and getValue, so extra indirections.
+def IntegerSetConstraints : Array<AffineExprList> {
+  let cGetter = "$_attrType.getValue().getConstraints()";
+}
+
+def IntegerSetEqFlags : ArrayWithKnownSize<BoolList, "constraints.size()"> {
+  let cGetter = "$_attrType.getValue().getEqFlags()";
+}
+
+class EnableIntegerSetPrintingJuly2026<dag d> : DialectType<d>;
+
+def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
+  WithGetter<"$_attrType.getValue().getNumDims()", VarInt>:$numDims,
+  WithGetter<"$_attrType.getValue().getNumSymbols()", VarInt>:$numSymbols,
+  IntegerSetConstraints:$constraints,
+  IntegerSetEqFlags:$eqFlags
+)> {
+  let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
+}
+
 // Types
 // -----
 
@@ -388,6 +416,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     DistinctAttr,
     FileLineColRange,
     AffineMapAttr,
+    IntegerSetAttr,
   ];
 }
 
diff --git a/mlir/include/mlir/IR/BytecodeBase.td b/mlir/include/mlir/IR/BytecodeBase.td
index df60800ef639b..3eab62493ffef 100644
--- a/mlir/include/mlir/IR/BytecodeBase.td
+++ b/mlir/include/mlir/IR/BytecodeBase.td
@@ -125,6 +125,15 @@ class Array<Bytecode t> {
   Bytecode elemT = t;
 
   string cBuilder = "$_args";
+
+  // Optional custom getter expression for the array from the parent type.
+  string cGetter = "";
+}
+// - Array variant where the size is known from a previously serialized member.
+//   This avoids encoding a redundant length prefix.
+class ArrayWithKnownSize<Bytecode t, string sizeRef> : Array<t> {
+ // Expression providing the element count (e.g., "otherMember.size()").
+  string knownSizeRef = sizeRef;
 }
 // - Array elements currently needs a different bytecode type to accommodate
 //   for the list print/parsing.
@@ -177,4 +186,3 @@ def none;
 def ReservedOrDead : DialectAttrOrType<(none)>;
 
 #endif // BYTECODE_BASE
-
diff --git a/mlir/lib/IR/BuiltinDialectBytecode.cpp b/mlir/lib/IR/BuiltinDialectBytecode.cpp
index 366b3ceb06410..e5476082cd6a0 100644
--- a/mlir/lib/IR/BuiltinDialectBytecode.cpp
+++ b/mlir/lib/IR/BuiltinDialectBytecode.cpp
@@ -16,6 +16,7 @@
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Diagnostics.h"
 #include "mlir/IR/DialectResourceBlobManager.h"
+#include "mlir/IR/IntegerSet.h"
 #include "mlir/IR/Location.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -157,8 +158,8 @@ static AffineExprKind fromBytecodeKind(uint64_t kind) {
 /// is prefix order (operator and then children), which is self-delimiting.
 /// Instead of C++ recursion the reader uses an explicit work stack, bounding
 /// memory to O(depth) and eliminating stack-overflow risk on malicious input.
-static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
-                                            MLIRContext *context) {
+static LogicalResult readAffineExpr(DialectBytecodeReader &reader,
+                                    MLIRContext *context, AffineExpr &expr) {
   // A work-stack item is either ReadOperand (0) or a combine marker whose
   // payload is an AffineExprKind.
   struct WorkItem {
@@ -175,15 +176,14 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
   while (!work.empty()) {
     // Bound total iterations to catch malformed input.
     if (work.size() > 128)
-      return reader.emitError("AffineExpr work stack overflow"), failure();
+      return reader.emitError("AffineExpr work stack overflow");
 
     WorkItem item = work.pop_back_val();
 
     if (item.isCombine) {
       // Pop two operands and combine.
       if (operands.size() < 2)
-        return reader.emitError("malformed AffineExpr: operand underflow"),
-               failure();
+        return reader.emitError("malformed AffineExpr: operand underflow");
       AffineExpr rhs = operands.pop_back_val();
       AffineExpr lhs = operands.pop_back_val();
       operands.push_back(getAffineBinaryOpExpr(item.combineKind, lhs, rhs));
@@ -240,7 +240,8 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
     return reader.emitError("malformed AffineExpr: expected single result"),
            failure();
 
-  return operands.front();
+  expr = operands.front();
+  return success();
 }
 
 /// Write an AffineExpr in prefix order (operator first, then children).
@@ -337,10 +338,10 @@ static LogicalResult readAffineMap(DialectBytecodeReader &reader,
     SmallVector<AffineExpr> results;
     results.reserve(numResults);
     for (uint64_t i = 0; i < numResults; ++i) {
-      auto expr = readAffineExpr(reader, context);
-      if (failed(expr))
+      AffineExpr expr;
+      if (failed(readAffineExpr(reader, context, expr)))
         return failure();
-      results.push_back(*expr);
+      results.push_back(expr);
     }
     map = AffineMap::get(numDims, numSymbols, results, context);
     return success();
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index aac1c44248995..3364a8e9fce8e 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -255,3 +255,19 @@ module @TestAffineMap attributes {
   bytecode.projperm_single = affine_map<(d0, d1, d2) -> (d2)>
 } {}
 
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {}
+
diff --git a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
index dd178b5e5d232..031fe6cc94eb2 100644
--- a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
+++ b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
@@ -183,7 +183,12 @@ static void printParseConditional(mlir::raw_indented_ostream &ios,
             parser = "succeeded($_reader.readAttributes($_var))";
           else if (!composite && def->isSubClassOf("TypeKind"))
             parser = "succeeded($_reader.readTypes($_var))";
-          else
+          else if (attr->isSubClassOf("ArrayWithKnownSize")) {
+            std::string sizeRef = attr->getValueAsString("knownSizeRef").str();
+            parser = ("succeeded($_reader.readListWithKnownSize($_var, " +
+                      sizeRef + ", " + listHelperName(std::get<1>(it)) + "))")
+                         .str();
+          } else
             parser = ("succeeded($_reader.readList($_var, " +
                       listHelperName(std::get<1>(it)) + "))")
                          .str();
@@ -385,8 +390,11 @@ void Generator::emitPrintHelper(const Record *memberRec, StringRef kind,
     }
     std::string returnType = getCType(def);
     std::string nestedName = kind.str();
-    ios << "writer.writeList(" << getter << ", [&](" << returnType << " "
-        << nestedName << ") ";
+    StringRef writeMethod = memberRec->isSubClassOf("ArrayWithKnownSize")
+                                ? "writer.writeListWithKnownSize("
+                                : "writer.writeList(";
+    ios << writeMethod << getter << ", [&](" << returnType << " " << nestedName
+        << ") ";
     auto lambdaScope = ios.scope("{\n", "});\n");
     return emitPrintHelper(def, kind, nestedName, nestedName, ios);
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207482

From llvmlistbot at llvm.org  Fri Jul  3 21:50:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 21:50:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207482@github.com>
Message-ID: <6a4890f9.13be6df8.3d8759.895b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-ods

Author: Jacques Pienaar (jpienaar)

<details>
<summary>Changes</summary>

Serialize IntegerSetAttr using VarInt for numDims and numSymbols and arrays for contraints and eqFlags. Add ArrayWithKnownSize` an Array variant that omits the length prefix when the size is known from a previously serialized member. This avoids encoding redundant array lengths when, for example, two arrays are always the same size.

Assisted-By: Gemini

---
Full diff: https://github.com/llvm/llvm-project/pull/207482.diff


6 Files Affected:

- (modified) mlir/include/mlir/Bytecode/BytecodeImplementation.h (+20) 
- (modified) mlir/include/mlir/IR/BuiltinDialectBytecode.td (+29) 
- (modified) mlir/include/mlir/IR/BytecodeBase.td (+9-1) 
- (modified) mlir/lib/IR/BuiltinDialectBytecode.cpp (+10-9) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/attrs.mlir (+16) 
- (modified) mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp (+11-3) 


``````````diff
diff --git a/mlir/include/mlir/Bytecode/BytecodeImplementation.h b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
index f1381785a2703..d8d0ca7b3b9f7 100644
--- a/mlir/include/mlir/Bytecode/BytecodeImplementation.h
+++ b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
@@ -76,6 +76,18 @@ class DialectBytecodeReader {
     uint64_t size;
     if (failed(readVarInt(size)))
       return failure();
+    return readListWithKnownSize(result, size,
+                                 std::forward<CallbackFn>(callback));
+  }
+
+  /// Read out a list of elements with a known size, invoking the provided
+  /// callback for each element. Unlike readList, this does not read a length
+  /// prefix. The callback function may be in any of the following forms:
+  ///   * LogicalResult(T &)
+  ///   * FailureOr<T>()
+  template <typename T, typename CallbackFn>
+  LogicalResult readListWithKnownSize(SmallVectorImpl<T> &result, uint64_t size,
+                                      CallbackFn &&callback) {
     result.reserve(size);
 
     for (uint64_t i = 0; i < size; ++i) {
@@ -294,6 +306,14 @@ class DialectBytecodeWriter {
       callback(element);
   }
 
+  /// Write out a list of elements without a length prefix, for cases where the
+  /// size is known from another field.
+  template <typename RangeT, typename CallbackFn>
+  void writeListWithKnownSize(RangeT &&range, CallbackFn &&callback) {
+    for (auto &element : range)
+      callback(element);
+  }
+
   /// Write a reference to the given attribute.
   virtual void writeAttribute(Attribute attr) = 0;
   virtual void writeOptionalAttribute(Attribute attr) = 0;
diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index a3a0511791613..f087e5cb23081 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -214,6 +214,34 @@ def AffineMapAttr : EnableAffineMapPrintingJuly2026<(attr
     WithType<"AffineMap">>>:$value
 )>;
 
+def AffineExpr :
+  WithParser<"succeeded(readAffineExpr($_reader, context, $_var))",
+  WithBuilder<"$_args",
+  WithPrinter<"writeAffineExpr($_writer, $_getter)",
+  WithType<"AffineExpr">>>>;
+def AffineExprList : List<AffineExpr>;
+
+// Similar to AffineMapAttr, IntegerSetAttr has everything going via IntegerSet
+// and getValue, so extra indirections.
+def IntegerSetConstraints : Array<AffineExprList> {
+  let cGetter = "$_attrType.getValue().getConstraints()";
+}
+
+def IntegerSetEqFlags : ArrayWithKnownSize<BoolList, "constraints.size()"> {
+  let cGetter = "$_attrType.getValue().getEqFlags()";
+}
+
+class EnableIntegerSetPrintingJuly2026<dag d> : DialectType<d>;
+
+def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
+  WithGetter<"$_attrType.getValue().getNumDims()", VarInt>:$numDims,
+  WithGetter<"$_attrType.getValue().getNumSymbols()", VarInt>:$numSymbols,
+  IntegerSetConstraints:$constraints,
+  IntegerSetEqFlags:$eqFlags
+)> {
+  let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
+}
+
 // Types
 // -----
 
@@ -388,6 +416,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     DistinctAttr,
     FileLineColRange,
     AffineMapAttr,
+    IntegerSetAttr,
   ];
 }
 
diff --git a/mlir/include/mlir/IR/BytecodeBase.td b/mlir/include/mlir/IR/BytecodeBase.td
index df60800ef639b..3eab62493ffef 100644
--- a/mlir/include/mlir/IR/BytecodeBase.td
+++ b/mlir/include/mlir/IR/BytecodeBase.td
@@ -125,6 +125,15 @@ class Array<Bytecode t> {
   Bytecode elemT = t;
 
   string cBuilder = "$_args";
+
+  // Optional custom getter expression for the array from the parent type.
+  string cGetter = "";
+}
+// - Array variant where the size is known from a previously serialized member.
+//   This avoids encoding a redundant length prefix.
+class ArrayWithKnownSize<Bytecode t, string sizeRef> : Array<t> {
+ // Expression providing the element count (e.g., "otherMember.size()").
+  string knownSizeRef = sizeRef;
 }
 // - Array elements currently needs a different bytecode type to accommodate
 //   for the list print/parsing.
@@ -177,4 +186,3 @@ def none;
 def ReservedOrDead : DialectAttrOrType<(none)>;
 
 #endif // BYTECODE_BASE
-
diff --git a/mlir/lib/IR/BuiltinDialectBytecode.cpp b/mlir/lib/IR/BuiltinDialectBytecode.cpp
index 366b3ceb06410..e5476082cd6a0 100644
--- a/mlir/lib/IR/BuiltinDialectBytecode.cpp
+++ b/mlir/lib/IR/BuiltinDialectBytecode.cpp
@@ -16,6 +16,7 @@
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Diagnostics.h"
 #include "mlir/IR/DialectResourceBlobManager.h"
+#include "mlir/IR/IntegerSet.h"
 #include "mlir/IR/Location.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -157,8 +158,8 @@ static AffineExprKind fromBytecodeKind(uint64_t kind) {
 /// is prefix order (operator and then children), which is self-delimiting.
 /// Instead of C++ recursion the reader uses an explicit work stack, bounding
 /// memory to O(depth) and eliminating stack-overflow risk on malicious input.
-static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
-                                            MLIRContext *context) {
+static LogicalResult readAffineExpr(DialectBytecodeReader &reader,
+                                    MLIRContext *context, AffineExpr &expr) {
   // A work-stack item is either ReadOperand (0) or a combine marker whose
   // payload is an AffineExprKind.
   struct WorkItem {
@@ -175,15 +176,14 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
   while (!work.empty()) {
     // Bound total iterations to catch malformed input.
     if (work.size() > 128)
-      return reader.emitError("AffineExpr work stack overflow"), failure();
+      return reader.emitError("AffineExpr work stack overflow");
 
     WorkItem item = work.pop_back_val();
 
     if (item.isCombine) {
       // Pop two operands and combine.
       if (operands.size() < 2)
-        return reader.emitError("malformed AffineExpr: operand underflow"),
-               failure();
+        return reader.emitError("malformed AffineExpr: operand underflow");
       AffineExpr rhs = operands.pop_back_val();
       AffineExpr lhs = operands.pop_back_val();
       operands.push_back(getAffineBinaryOpExpr(item.combineKind, lhs, rhs));
@@ -240,7 +240,8 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
     return reader.emitError("malformed AffineExpr: expected single result"),
            failure();
 
-  return operands.front();
+  expr = operands.front();
+  return success();
 }
 
 /// Write an AffineExpr in prefix order (operator first, then children).
@@ -337,10 +338,10 @@ static LogicalResult readAffineMap(DialectBytecodeReader &reader,
     SmallVector<AffineExpr> results;
     results.reserve(numResults);
     for (uint64_t i = 0; i < numResults; ++i) {
-      auto expr = readAffineExpr(reader, context);
-      if (failed(expr))
+      AffineExpr expr;
+      if (failed(readAffineExpr(reader, context, expr)))
         return failure();
-      results.push_back(*expr);
+      results.push_back(expr);
     }
     map = AffineMap::get(numDims, numSymbols, results, context);
     return success();
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index aac1c44248995..3364a8e9fce8e 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -255,3 +255,19 @@ module @TestAffineMap attributes {
   bytecode.projperm_single = affine_map<(d0, d1, d2) -> (d2)>
 } {}
 
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {}
+
diff --git a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
index dd178b5e5d232..031fe6cc94eb2 100644
--- a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
+++ b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
@@ -183,7 +183,12 @@ static void printParseConditional(mlir::raw_indented_ostream &ios,
             parser = "succeeded($_reader.readAttributes($_var))";
           else if (!composite && def->isSubClassOf("TypeKind"))
             parser = "succeeded($_reader.readTypes($_var))";
-          else
+          else if (attr->isSubClassOf("ArrayWithKnownSize")) {
+            std::string sizeRef = attr->getValueAsString("knownSizeRef").str();
+            parser = ("succeeded($_reader.readListWithKnownSize($_var, " +
+                      sizeRef + ", " + listHelperName(std::get<1>(it)) + "))")
+                         .str();
+          } else
             parser = ("succeeded($_reader.readList($_var, " +
                       listHelperName(std::get<1>(it)) + "))")
                          .str();
@@ -385,8 +390,11 @@ void Generator::emitPrintHelper(const Record *memberRec, StringRef kind,
     }
     std::string returnType = getCType(def);
     std::string nestedName = kind.str();
-    ios << "writer.writeList(" << getter << ", [&](" << returnType << " "
-        << nestedName << ") ";
+    StringRef writeMethod = memberRec->isSubClassOf("ArrayWithKnownSize")
+                                ? "writer.writeListWithKnownSize("
+                                : "writer.writeList(";
+    ios << writeMethod << getter << ", [&](" << returnType << " " << nestedName
+        << ") ";
     auto lambdaScope = ios.scope("{\n", "});\n");
     return emitPrintHelper(def, kind, nestedName, nestedName, ios);
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207482

From llvmlistbot at llvm.org  Fri Jul  3 21:50:52 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 21:50:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207482@github.com>
Message-ID: <6a48912c.af6ba3ed.2c2f30.acef@mx.google.com>


https://github.com/jpienaar auto_merge_enabled https://github.com/llvm/llvm-project/pull/207482

From llvmlistbot at llvm.org  Fri Jul  3 22:06:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 22:06:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 238dbdc - [mlirbc] Add IntegerSetAttr
	(#207482)
Message-ID: <6a4894e0.f06f4994.168928.edf4@mx.google.com>


Author: Jacques Pienaar
Date: 2026-07-04T05:06:36Z
New Revision: 238dbdce63b41b4da1f808b78e3d143f1635642b

URL: https://github.com/llvm/llvm-project/commit/238dbdce63b41b4da1f808b78e3d143f1635642b
DIFF: https://github.com/llvm/llvm-project/commit/238dbdce63b41b4da1f808b78e3d143f1635642b.diff

LOG: [mlirbc] Add IntegerSetAttr (#207482)

Serialize IntegerSetAttr using VarInt for numDims and numSymbols and
arrays for contraints and eqFlags. Add ArrayWithKnownSize` an Array
variant that omits the length prefix when the size is known from a
previously serialized member. This avoids encoding redundant array
lengths when, for example, two arrays are always the same size.

Assisted-By: Gemini

Added: 
    

Modified: 
    mlir/include/mlir/Bytecode/BytecodeImplementation.h
    mlir/include/mlir/IR/BuiltinDialectBytecode.td
    mlir/include/mlir/IR/BytecodeBase.td
    mlir/lib/IR/BuiltinDialectBytecode.cpp
    mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
    mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Bytecode/BytecodeImplementation.h b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
index f1381785a2703..d8d0ca7b3b9f7 100644
--- a/mlir/include/mlir/Bytecode/BytecodeImplementation.h
+++ b/mlir/include/mlir/Bytecode/BytecodeImplementation.h
@@ -76,6 +76,18 @@ class DialectBytecodeReader {
     uint64_t size;
     if (failed(readVarInt(size)))
       return failure();
+    return readListWithKnownSize(result, size,
+                                 std::forward<CallbackFn>(callback));
+  }
+
+  /// Read out a list of elements with a known size, invoking the provided
+  /// callback for each element. Unlike readList, this does not read a length
+  /// prefix. The callback function may be in any of the following forms:
+  ///   * LogicalResult(T &)
+  ///   * FailureOr<T>()
+  template <typename T, typename CallbackFn>
+  LogicalResult readListWithKnownSize(SmallVectorImpl<T> &result, uint64_t size,
+                                      CallbackFn &&callback) {
     result.reserve(size);
 
     for (uint64_t i = 0; i < size; ++i) {
@@ -294,6 +306,14 @@ class DialectBytecodeWriter {
       callback(element);
   }
 
+  /// Write out a list of elements without a length prefix, for cases where the
+  /// size is known from another field.
+  template <typename RangeT, typename CallbackFn>
+  void writeListWithKnownSize(RangeT &&range, CallbackFn &&callback) {
+    for (auto &element : range)
+      callback(element);
+  }
+
   /// Write a reference to the given attribute.
   virtual void writeAttribute(Attribute attr) = 0;
   virtual void writeOptionalAttribute(Attribute attr) = 0;

diff  --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index a3a0511791613..f087e5cb23081 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -214,6 +214,34 @@ def AffineMapAttr : EnableAffineMapPrintingJuly2026<(attr
     WithType<"AffineMap">>>:$value
 )>;
 
+def AffineExpr :
+  WithParser<"succeeded(readAffineExpr($_reader, context, $_var))",
+  WithBuilder<"$_args",
+  WithPrinter<"writeAffineExpr($_writer, $_getter)",
+  WithType<"AffineExpr">>>>;
+def AffineExprList : List<AffineExpr>;
+
+// Similar to AffineMapAttr, IntegerSetAttr has everything going via IntegerSet
+// and getValue, so extra indirections.
+def IntegerSetConstraints : Array<AffineExprList> {
+  let cGetter = "$_attrType.getValue().getConstraints()";
+}
+
+def IntegerSetEqFlags : ArrayWithKnownSize<BoolList, "constraints.size()"> {
+  let cGetter = "$_attrType.getValue().getEqFlags()";
+}
+
+class EnableIntegerSetPrintingJuly2026<dag d> : DialectType<d>;
+
+def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
+  WithGetter<"$_attrType.getValue().getNumDims()", VarInt>:$numDims,
+  WithGetter<"$_attrType.getValue().getNumSymbols()", VarInt>:$numSymbols,
+  IntegerSetConstraints:$constraints,
+  IntegerSetEqFlags:$eqFlags
+)> {
+  let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
+}
+
 // Types
 // -----
 
@@ -388,6 +416,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     DistinctAttr,
     FileLineColRange,
     AffineMapAttr,
+    IntegerSetAttr,
   ];
 }
 

diff  --git a/mlir/include/mlir/IR/BytecodeBase.td b/mlir/include/mlir/IR/BytecodeBase.td
index df60800ef639b..3eab62493ffef 100644
--- a/mlir/include/mlir/IR/BytecodeBase.td
+++ b/mlir/include/mlir/IR/BytecodeBase.td
@@ -125,6 +125,15 @@ class Array<Bytecode t> {
   Bytecode elemT = t;
 
   string cBuilder = "$_args";
+
+  // Optional custom getter expression for the array from the parent type.
+  string cGetter = "";
+}
+// - Array variant where the size is known from a previously serialized member.
+//   This avoids encoding a redundant length prefix.
+class ArrayWithKnownSize<Bytecode t, string sizeRef> : Array<t> {
+ // Expression providing the element count (e.g., "otherMember.size()").
+  string knownSizeRef = sizeRef;
 }
 // - Array elements currently needs a 
diff erent bytecode type to accommodate
 //   for the list print/parsing.
@@ -177,4 +186,3 @@ def none;
 def ReservedOrDead : DialectAttrOrType<(none)>;
 
 #endif // BYTECODE_BASE
-

diff  --git a/mlir/lib/IR/BuiltinDialectBytecode.cpp b/mlir/lib/IR/BuiltinDialectBytecode.cpp
index 366b3ceb06410..e5476082cd6a0 100644
--- a/mlir/lib/IR/BuiltinDialectBytecode.cpp
+++ b/mlir/lib/IR/BuiltinDialectBytecode.cpp
@@ -16,6 +16,7 @@
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Diagnostics.h"
 #include "mlir/IR/DialectResourceBlobManager.h"
+#include "mlir/IR/IntegerSet.h"
 #include "mlir/IR/Location.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -157,8 +158,8 @@ static AffineExprKind fromBytecodeKind(uint64_t kind) {
 /// is prefix order (operator and then children), which is self-delimiting.
 /// Instead of C++ recursion the reader uses an explicit work stack, bounding
 /// memory to O(depth) and eliminating stack-overflow risk on malicious input.
-static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
-                                            MLIRContext *context) {
+static LogicalResult readAffineExpr(DialectBytecodeReader &reader,
+                                    MLIRContext *context, AffineExpr &expr) {
   // A work-stack item is either ReadOperand (0) or a combine marker whose
   // payload is an AffineExprKind.
   struct WorkItem {
@@ -175,15 +176,14 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
   while (!work.empty()) {
     // Bound total iterations to catch malformed input.
     if (work.size() > 128)
-      return reader.emitError("AffineExpr work stack overflow"), failure();
+      return reader.emitError("AffineExpr work stack overflow");
 
     WorkItem item = work.pop_back_val();
 
     if (item.isCombine) {
       // Pop two operands and combine.
       if (operands.size() < 2)
-        return reader.emitError("malformed AffineExpr: operand underflow"),
-               failure();
+        return reader.emitError("malformed AffineExpr: operand underflow");
       AffineExpr rhs = operands.pop_back_val();
       AffineExpr lhs = operands.pop_back_val();
       operands.push_back(getAffineBinaryOpExpr(item.combineKind, lhs, rhs));
@@ -240,7 +240,8 @@ static FailureOr<AffineExpr> readAffineExpr(DialectBytecodeReader &reader,
     return reader.emitError("malformed AffineExpr: expected single result"),
            failure();
 
-  return operands.front();
+  expr = operands.front();
+  return success();
 }
 
 /// Write an AffineExpr in prefix order (operator first, then children).
@@ -337,10 +338,10 @@ static LogicalResult readAffineMap(DialectBytecodeReader &reader,
     SmallVector<AffineExpr> results;
     results.reserve(numResults);
     for (uint64_t i = 0; i < numResults; ++i) {
-      auto expr = readAffineExpr(reader, context);
-      if (failed(expr))
+      AffineExpr expr;
+      if (failed(readAffineExpr(reader, context, expr)))
         return failure();
-      results.push_back(*expr);
+      results.push_back(expr);
     }
     map = AffineMap::get(numDims, numSymbols, results, context);
     return success();

diff  --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index aac1c44248995..3364a8e9fce8e 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -255,3 +255,19 @@ module @TestAffineMap attributes {
   bytecode.projperm_single = affine_map<(d0, d1, d2) -> (d2)>
 } {}
 
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {}
+

diff  --git a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
index dd178b5e5d232..031fe6cc94eb2 100644
--- a/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
+++ b/mlir/tools/mlir-tblgen/BytecodeDialectGen.cpp
@@ -183,7 +183,12 @@ static void printParseConditional(mlir::raw_indented_ostream &ios,
             parser = "succeeded($_reader.readAttributes($_var))";
           else if (!composite && def->isSubClassOf("TypeKind"))
             parser = "succeeded($_reader.readTypes($_var))";
-          else
+          else if (attr->isSubClassOf("ArrayWithKnownSize")) {
+            std::string sizeRef = attr->getValueAsString("knownSizeRef").str();
+            parser = ("succeeded($_reader.readListWithKnownSize($_var, " +
+                      sizeRef + ", " + listHelperName(std::get<1>(it)) + "))")
+                         .str();
+          } else
             parser = ("succeeded($_reader.readList($_var, " +
                       listHelperName(std::get<1>(it)) + "))")
                          .str();
@@ -385,8 +390,11 @@ void Generator::emitPrintHelper(const Record *memberRec, StringRef kind,
     }
     std::string returnType = getCType(def);
     std::string nestedName = kind.str();
-    ios << "writer.writeList(" << getter << ", [&](" << returnType << " "
-        << nestedName << ") ";
+    StringRef writeMethod = memberRec->isSubClassOf("ArrayWithKnownSize")
+                                ? "writer.writeListWithKnownSize("
+                                : "writer.writeList(";
+    ios << writeMethod << getter << ", [&](" << returnType << " " << nestedName
+        << ") ";
     auto lambdaScope = ios.scope("{\n", "});\n");
     return emitPrintHelper(def, kind, nestedName, nestedName, ios);
   }


        

From llvmlistbot at llvm.org  Fri Jul  3 22:06:43 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 22:06:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207482@github.com>
Message-ID: <6a4894e3.fdc477d6.1a5bda.2e7d@mx.google.com>


https://github.com/jpienaar closed https://github.com/llvm/llvm-project/pull/207482

From llvmlistbot at llvm.org  Fri Jul  3 22:53:36 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 03 Jul 2026 22:53:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Add IntegerSetAttr (PR #207482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207482@github.com>
Message-ID: <6a489fe0.a87eb5dc.2cf3a1.e1df@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `sanitizer-aarch64-linux-bootstrap-hwasan` running on `sanitizer-buildbot11` while building `mlir` at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/55/builds/29933

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/wasm-ld
-- Testing: 100151 tests, 72 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 
FAIL: LLVM :: ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s (60820 of 100151)
******************** TEST 'LLVM :: ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp && mkdir -p /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# executed command: rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# note: command had no output on stdout or stderr
# executed command: mkdir -p /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# note: command had no output on stdout or stderr
# RUN: at line 2
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/yaml2obj /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/Inputs/COFF_weak_nolibrary_serach_def.yaml -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/yaml2obj /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/Inputs/COFF_weak_nolibrary_serach_def.yaml -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o
# note: command had no output on stdout or stderr
# RUN: at line 3
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-mc -filetype=obj -triple=x86_64-windows-msvc /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-mc -filetype=obj -triple=x86_64-windows-msvc /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# note: command had no output on stdout or stderr
# RUN: at line 4
not /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-jitlink -noexec -abs __ImageBase=0xfff00000  -slab-allocate 100Kb -slab-address 0xfff00000 -slab-page-size 4096  /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o 2>&1 | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s
# executed command: not /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-jitlink -noexec -abs __ImageBase=0xfff00000 -slab-allocate 100Kb -slab-address 0xfff00000 -slab-page-size 4096 /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# note: command had no output on stdout or stderr
# error: command failed with exit status: 1
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s
# note: command had no output on stdout or stderr

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
Slowest Tests:
--------------------------------------------------------------------------
47.55s: LLVM :: tools/llvm-exegesis/AArch64/all-opcodes.test
37.51s: LLVM :: CodeGen/AMDGPU/sched-group-barrier-pipeline-solver.mir
29.15s: LLVM :: CodeGen/AMDGPU/amdgcn.bitcast.1024bit.ll
26.95s: Clang :: CodeGen/X86/avx-builtins.c
26.53s: Clang :: CodeGen/X86/sse2-builtins.c
Step 11 (stage2/hwasan check) failure: stage2/hwasan check (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/wasm-ld
-- Testing: 100151 tests, 72 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 
FAIL: LLVM :: ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s (60820 of 100151)
******************** TEST 'LLVM :: ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp && mkdir -p /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# executed command: rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# note: command had no output on stdout or stderr
# executed command: mkdir -p /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp
# note: command had no output on stdout or stderr
# RUN: at line 2
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/yaml2obj /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/Inputs/COFF_weak_nolibrary_serach_def.yaml -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/yaml2obj /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/Inputs/COFF_weak_nolibrary_serach_def.yaml -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o
# note: command had no output on stdout or stderr
# RUN: at line 3
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-mc -filetype=obj -triple=x86_64-windows-msvc /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-mc -filetype=obj -triple=x86_64-windows-msvc /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# note: command had no output on stdout or stderr
# RUN: at line 4
not /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-jitlink -noexec -abs __ImageBase=0xfff00000  -slab-allocate 100Kb -slab-address 0xfff00000 -slab-page-size 4096  /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o 2>&1 | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s
# executed command: not /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/llvm-jitlink -noexec -abs __ImageBase=0xfff00000 -slab-allocate 100Kb -slab-address 0xfff00000 -slab-page-size 4096 /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_weak.o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/test/ExecutionEngine/JITLink/x86-64/Output/COFF_nolibrary_search.s.tmp/COFF_main.o
# note: command had no output on stdout or stderr
# error: command failed with exit status: 1
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/test/ExecutionEngine/JITLink/x86-64/COFF_nolibrary_search.s
# note: command had no output on stdout or stderr

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
Slowest Tests:
--------------------------------------------------------------------------
47.55s: LLVM :: tools/llvm-exegesis/AArch64/all-opcodes.test
37.51s: LLVM :: CodeGen/AMDGPU/sched-group-barrier-pipeline-solver.mir
29.15s: LLVM :: CodeGen/AMDGPU/amdgcn.bitcast.1024bit.ll
26.95s: Clang :: CodeGen/X86/avx-builtins.c
26.53s: Clang :: CodeGen/X86/sse2-builtins.c

```

</details>

https://github.com/llvm/llvm-project/pull/207482

From llvmlistbot at llvm.org  Fri Jul  3 23:18:40 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 23:18:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Address remaining missing encodings
 for builtin dialect. (PR #207488)
Message-ID: <llvm.org/llvm/llvm-project/pull/207488@github.com>


https://github.com/jpienaar created https://github.com/llvm/llvm-project/pull/207488

Simple ones that got missed (graph & strided layout), and add some additional tests. Also update the fixed test.

Assisted-By: Gemini

>From b3f78f941ea8c5f01c2916292c4b9210b55f4b65 Mon Sep 17 00:00:00 2001
From: Jacques Pienaar <jpienaar at google.com>
Date: Mon, 23 Feb 2026 11:33:39 +0200
Subject: [PATCH] [mlirbc] Address remaining missing encodings for builtin
 dialect.

Simple ones that got missed (graph & strided layout), and add some
additional tests. Also update the fixed test.

Assisted-By: Gemini
---
 .../include/mlir/IR/BuiltinDialectBytecode.td |  16 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir | 162 ++++++++++++++++--
 .../Builtin/Bytecode/builtin_fixed.mlir       | 102 ++++++++++-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 4447 -> 5470 bytes
 mlir/test/Dialect/Builtin/Bytecode/types.mlir |  38 +++-
 5 files changed, 298 insertions(+), 20 deletions(-)

diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index f087e5cb23081..d098793989dc4 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -242,6 +242,13 @@ def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
   let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
 }
 
+class EnableStridedPrintingJuly2026<dag d> : DialectAttribute<d>;
+
+def StridedLayoutAttr : EnableStridedPrintingJuly2026<(attr
+  SignedVarInt:$offset,
+  Array<SignedVarIntList>:$strides
+)>;
+
 // Types
 // -----
 
@@ -264,6 +271,11 @@ def FunctionType : DialectType<(type
   Array<Type>:$results
 )>;
 
+def GraphType : DialectType<(type
+  Array<Type>:$inputs,
+  Array<Type>:$results
+)>;
+
 def BFloat16Type : DialectType<(type)>;
 
 def Float16Type : DialectType<(type)>;
@@ -417,6 +429,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     FileLineColRange,
     AffineMapAttr,
     IntegerSetAttr,
+    StridedLayoutAttr,
   ];
 }
 
@@ -455,7 +468,8 @@ def BuiltinDialectTypes : DialectTypes<"Builtin"> {
     Float6E2M3FNType,
     Float6E3M2FNType,
     Float8E8M0FNUType,
-    TokenType
+    TokenType,
+    GraphType
   ];
 }
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 3364a8e9fce8e..7dedfbdff10a7 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -29,13 +29,55 @@ module @TestDenseArray attributes {
 //===----------------------------------------------------------------------===//
 
 // CHECK-LABEL: @TestDenseTypedElements
-// CHECK: bytecode.test1 = dense<true> : tensor<256xi1>
-// CHECK: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
-// CHECK: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.test1 = dense<true> : tensor<256xi1>
+// CHECK-DAG: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
+// CHECK-DAG: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.i16 = dense<[100, -200]> : tensor<2xi16>
+// CHECK-DAG: bytecode.i64 = dense<1000000> : tensor<4xi64>
+// CHECK-DAG: bytecode.f16 = dense<[1.500000e+00, 2.500000e+00]> : tensor<2xf16>
+// CHECK-DAG: bytecode.bf16 = dense<[-5.000000e-01, 5.000000e-01]> : tensor<2xbf16>
+// CHECK-DAG: bytecode.complex_val = dense<(1.000000e+00,2.000000e+00)> : tensor<1xcomplex<f32>>
 module @TestDenseTypedElements attributes {
   bytecode.test1 = dense<true> : tensor<256xi1>,
   bytecode.test2 = dense<[10, 32, 255]> : tensor<3xi8>,
-  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>
+  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>,
+  bytecode.i16 = dense<[100, -200]> : tensor<2xi16>,
+  bytecode.i64 = dense<1000000> : tensor<4xi64>,
+  bytecode.f16 = dense<[1.5, 2.5]> : tensor<2xf16>,
+  bytecode.bf16 = dense<[-0.5, 0.5]> : tensor<2xbf16>,
+  bytecode.complex_val = dense<(1.0, 2.0)> : tensor<1xcomplex<f32>>
+} {}
+
+// DenseI1 (single, 8-element, varied sizes)
+
+// CHECK-LABEL: @TestDenseI1SingleElement
+module @TestDenseI1SingleElement attributes {
+  // CHECK-DAG: bytecode.false_1 = dense<false> : tensor<1xi1>
+  // CHECK-DAG: bytecode.true_1 = dense<true> : tensor<1xi1>
+  bytecode.false_1 = dense<false> : tensor<1xi1>,
+  bytecode.true_1 = dense<true> : tensor<1xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1EightElements
+module @TestDenseI1EightElements attributes {
+  // CHECK-DAG: bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_true_8 = dense<true> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_false_8 = dense<false> : tensor<8xi1>
+  bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>,
+  bytecode.all_true_8 = dense<true> : tensor<8xi1>,
+  bytecode.all_false_8 = dense<false> : tensor<8xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1VariedSizes
+module @TestDenseI1VariedSizes attributes {
+  // CHECK-DAG: bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>
+  // CHECK-DAG: bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>
+  // CHECK-DAG: bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>
+  // CHECK-DAG: bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
+  bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>,
+  bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>,
+  bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>,
+  bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -58,10 +100,16 @@ module @TestFloat attributes {
   // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
   // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
+  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
+  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
+  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
-  bytecode.float3 = -0.5 : bf16
+  bytecode.float3 = -0.5 : bf16,
+  bytecode.inf = 0x7FF0000000000000 : f64,
+  bytecode.nan = 0x7FF8000000000000 : f64,
+  bytecode.ninf = 0xFFF0000000000000 : f64
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -74,10 +122,18 @@ module @TestInt attributes {
   // CHECK: bytecode.int1 = -1 : i8
   // CHECK: bytecode.int2 = 800 : ui64
   // CHECK: bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  // CHECK: bytecode.int4 = true
+  // CHECK: bytecode.int5 = 42 : i32
+  // CHECK: bytecode.int6 = -100 : si32
+  // CHECK: bytecode.int7 = 7 : index
   bytecode.int = false,
   bytecode.int1 = -1 : i8,
   bytecode.int2 = 800 : ui64,
-  bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  bytecode.int3 = 90000000000000000300000000000000000001 : i128,
+  bytecode.int4 = true,
+  bytecode.int5 = 42 : i32,
+  bytecode.int6 = -100 : si32,
+  bytecode.int7 = 7 : index
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -87,7 +143,9 @@ module @TestInt attributes {
 // CHECK-LABEL: @TestSparseElements
 module @TestSparseElements attributes {
   // CHECK-LITERAL: bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
-  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
+  // CHECK-LITERAL: bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
+  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>,
+  bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
 } {}
 
 
@@ -99,8 +157,10 @@ module @TestSparseElements attributes {
 module @TestString attributes {
   // CHECK: bytecode.string = "hello"
   // CHECK: bytecode.string2 = "hello" : i32
+  // CHECK: bytecode.string3 = ""
   bytecode.string = "hello",
-  bytecode.string2 = "hello" : i32
+  bytecode.string2 = "hello" : i32,
+  bytecode.string3 = ""
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -146,7 +206,9 @@ module @TestDistinct attributes {
 // CHECK-LABEL: @TestLocCallSite
 module @TestLocCallSite attributes {
   // CHECK: bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
-  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
+  // CHECK: bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
+  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8)),
+  bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -156,7 +218,15 @@ module @TestLocCallSite attributes {
 // CHECK-LABEL: @TestLocFileLineCol
 module @TestLocFileLineCol attributes {
   // CHECK: bytecode.loc = loc("mysource.cc":10:8)
-  bytecode.loc = loc("mysource.cc":10:8)
+  // CHECK: bytecode.loc2 = loc("source.cc":10:8 to 12:4)
+  // CHECK: bytecode.loc3 = loc("source.cc":10:8 to :12)
+  // CHECK: bytecode.loc4 = loc("file.cc":0:0)
+  // CHECK: bytecode.loc5 = loc("file.cc":42:0)
+  bytecode.loc = loc("mysource.cc":10:8),
+  bytecode.loc2 = loc("source.cc":10:8 to 12:4),
+  bytecode.loc3 = loc("source.cc":10:8 to :12),
+  bytecode.loc4 = loc("file.cc":0:0),
+  bytecode.loc5 = loc("file.cc":42:0)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -221,9 +291,11 @@ module @TestAffineMap attributes {
   // Identity maps.
   // CHECK: bytecode.identity_1d = affine_map<(d0) -> (d0)>
   // CHECK: bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+  // CHECK: bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>
 
   // Multi-result maps.
   // CHECK: bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>
+  // CHECK: bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>
   // CHECK: bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>
 
   // Permutation maps.
@@ -247,7 +319,9 @@ module @TestAffineMap attributes {
   bytecode.empty_map = affine_map<() -> ()>,
   bytecode.identity_1d = affine_map<(d0) -> (d0)>,
   bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>,
+  bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>,
   bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>,
+  bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>,
   bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>,
   bytecode.perm_2d = affine_map<(d0, d1) -> (d1, d0)>,
   bytecode.perm_3d = affine_map<(d0, d1, d2) -> (d2, d0, d1)>,
@@ -265,9 +339,75 @@ module @TestIntegerSetAttr attributes {
   // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
   // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
   // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>
+  // CHECK-DAG: bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>
+  // CHECK-DAG: bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
   bytecode.eq = affine_set<(d0) : (d0 == 0)>,
   bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
   bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
-  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>,
+  bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>,
+  bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>,
+  bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
+} {}
+
+//===----------------------------------------------------------------------===//
+// DictionaryAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestDictionaryAttr
+module @TestDictionaryAttr attributes {
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.nested_dict = {inner = {a = 1 : i32}}
+  bytecode.empty_dict = {},
+  bytecode.nested_dict = {inner = {a = 1 : i32}}
+} {}
+
+//===----------------------------------------------------------------------===//
+// UnitAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestUnitAttr
+module @TestUnitAttr attributes {
+  // CHECK-DAG: bytecode.unit
+  bytecode.unit
+} {}
+
+//===----------------------------------------------------------------------===//
+// EmptyContainers
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestEmptyContainers
+module @TestEmptyContainers attributes {
+  // CHECK-DAG: bytecode.empty_arr = []
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.empty_str = ""
+  // CHECK-DAG: bytecode.empty_tuple = tuple<>
+  // CHECK-DAG: bytecode.empty_func = () -> ()
+  // CHECK-DAG: bytecode.empty_map = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.empty_dense_arr = array<i32>
+  // CHECK-DAG: bytecode.empty_tensor = dense<> : tensor<0xi32>
+  bytecode.empty_arr = [],
+  bytecode.empty_dict = {},
+  bytecode.empty_str = "",
+  bytecode.empty_tuple = tuple<>,
+  bytecode.empty_func = () -> (),
+  bytecode.empty_map = affine_map<() -> ()>,
+  bytecode.empty_dense_arr = array<i32>,
+  bytecode.empty_tensor = dense<> : tensor<0xi32>
+} {}
+
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
 } {}
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index 31061a6fa8653..d427035435277 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -127,12 +127,36 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
   // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
   // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
+  // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = 1.000000e+00 : f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = 1.000000e+00 : f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = 1.000000e+00 : f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = 1.000000e+00 : f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = 1.000000e+00 : f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = 1.000000e+00 : f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = 1.000000e+00 : f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = 1.000000e+00 : f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = 1.000000e+00 : f8E8M0FNU
   bytecode.bf16 = -0.5 : bf16,
   bytecode.f16 = 1.5 : f16,
   bytecode.f32 = 3.14 : f32,
   bytecode.f64 = 10.0 : f64,
   bytecode.f80 = 0.1 : f80,
-  bytecode.f128 = 0.1 : f128
+  bytecode.f128 = 0.1 : f128,
+  bytecode.tf32 = 1.0 : tf32,
+  bytecode.f8E5M2 = 1.0 : f8E5M2,
+  bytecode.f8E4M3 = 1.0 : f8E4M3,
+  bytecode.f8E4M3FN = 1.0 : f8E4M3FN,
+  bytecode.f8E5M2FNUZ = 1.0 : f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = 1.0 : f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = 1.0 : f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = 1.0 : f8E3M4,
+  bytecode.f4E2M1FN = 1.0 : f4E2M1FN,
+  bytecode.f6E2M3FN = 1.0 : f6E2M3FN,
+  bytecode.f6E3M2FN = 1.0 : f6E3M2FN,
+  bytecode.f8E8M0FNU = 1.0 : f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -260,12 +284,36 @@ module @TestAllFloatTypes attributes {
   // CHECK-DAG: bytecode.f64 = f64
   // CHECK-DAG: bytecode.f80 = f80
   // CHECK-DAG: bytecode.f128 = f128
+  // CHECK-DAG: bytecode.tf32 = tf32
+  // CHECK-DAG: bytecode.f8E5M2 = f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = f8E8M0FNU
   bytecode.bf16 = bf16,
   bytecode.f16 = f16,
   bytecode.f32 = f32,
   bytecode.f64 = f64,
   bytecode.f80 = f80,
-  bytecode.f128 = f128
+  bytecode.f128 = f128,
+  bytecode.tf32 = tf32,
+  bytecode.f8E5M2 = f8E5M2,
+  bytecode.f8E4M3 = f8E4M3,
+  bytecode.f8E4M3FN = f8E4M3FN,
+  bytecode.f8E5M2FNUZ = f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = f8E3M4,
+  bytecode.f4E2M1FN = f4E2M1FN,
+  bytecode.f6E2M3FN = f6E2M3FN,
+  bytecode.f6E3M2FN = f6E3M2FN,
+  bytecode.f8E8M0FNU = f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -396,6 +444,42 @@ module @TestTupleType attributes {
   bytecode.mixed_tuple = tuple<i32, f64, index>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// AffineMapAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestAffineMapAttr
+module @TestAffineMapAttr attributes {
+  // CHECK-DAG: bytecode.identity = affine_map<(d0) -> (d0)>
+  // CHECK-DAG: bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>
+  // CHECK-DAG: bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>
+  // CHECK-DAG: bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>
+  // CHECK-DAG: bytecode.empty = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.constant = affine_map<() -> (42)>
+  bytecode.identity = affine_map<(d0) -> (d0)>,
+  bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>,
+  bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>,
+  bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>,
+  bytecode.empty = affine_map<() -> ()>,
+  bytecode.constant = affine_map<() -> (42)>
+} {} loc(unknown)
+
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {} loc(unknown)
+
 //===----------------------------------------------------------------------===//
 // DenseResourceElementsAttr
 //===----------------------------------------------------------------------===//
@@ -416,6 +500,20 @@ module @TestDenseStringElementsAttr attributes {
   bytecode.dense_str = dense<["foo", "bar", "baz"]> : tensor<3x!bytecode.string>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
+} {} loc(unknown)
+
 } loc(unknown)
 
 {-#
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 51d48d35b2d23a8d6c31b26b14abf779aa227462..95be4de8fd1ab202b51658ce69f8037276dd74e8 100644
GIT binary patch
delta 2938
zcmZ{l4{TIL9>;fnzxnO!>vMQM-Yt8}vR${E*G=1Om+jJ)(u1;aEeBj%{v04B(CeQq
zSEO5 at 3s{J(K!6C5JJ^U3a(H6IfG0#a;nct>MvM^g#0V!wh!QbE#8V<fi1ua|d<D(T
z%WvLie)F5}{CP9;E)<?G`HL&u>@1;mtb`Oz*ECI6WcGiOt_8PL)P#1pXSw})MAvKe
zJISrJXnxJqV)6-U<F)Csv}wzx--?TOD{c9AI9+wyp?h^r56a$Z<U9M6=JIw~_G-Ew
zlFww_FTH-fNyh4uLb_Y8nk6&0&erOR9Hz{4Y4;>k3cF3yZMtPLO*0$g^=jH2?;N*$
zrS+gcm<=!y>!5j<pA>`bA(b=|1|wZQhG3#1VvtELa+8O=<Rd=?C`cBCsF+GAOc9Dw
zj4Dcrs;Jt~D59dGPSzS`&1Q^I({x5+s=-u-GYlrEhM`WQky={Y(+R3YMsdXXT{Iri
z5LAas&tkK8>2jJt6KOIvC{2u}B0ZAU*3xu!HO<i0(S}VlOMQ%HtB=zhnv0ZiJ>jMn
zYNd9>9XcgwJ{r$F``q(;UU-rAzjW~B!>_z{gt`pXK at S-!qi&;$9N_rlv?){W`_Ge(
zwp$OXks}K>P!BCOlz!?pgpZaQ#{FOP_t(;L!vqiYq2+zk_m=<dz&pW{*1Mtiir+8&
zApBwEZ1khp#}%JcU8w%F=HlqjGm9XU_JB!_+v#_Yn%+Fqcc1AeM}JFaF8!k)&A at Oo
zIMTEpG(#iJ;<0AwZ_MymGm<u={B=LAMB~a=Uw?D$yBpu%`k^1d3HdMx1_M13R279e
z7^Q>Rg<Ty)G8~RaYh_<!X>E|u at UOcvHT};5bu6t5R+HM{7}-x7f=ux+83WQ#8a)=&
zC@^-MF4^ID{m6g&vxznZ<<`l6pr0N`qW}t_2uh#~%ApcQ!5A0^^)Lw<VH(VYW}rV?
zHG#V|c~*Ce6_ev>x-+1RB|kDfVbv=6zbc?7t+<*UD?Me6mqE7^?Xc>wpZ;o1P)o7`
zPa}200$2tsfSxIyn1m%@P0sAW!gPa??Jc{k^lWj{Dp&*7)Z75FLfJ`WQ?C05;tt2(
z;`AIUn_(+#gWBy-OV7)_xrbh$7tysF_QFv(2FKw9oPyJE2F}5G_zb><>o5oh;}9H*
zMRHT_ckP2i;5z_}UUH{j1x=xY^s*aW1|1Hg>M}eky%I)60=;(IVbBo|U4qNtGno4-
z$l3pk-tf{*0L(|<k5cJPpHoV|MQ_tPzBr?kzIXw;3lWjtrT6H4xyj1WSBd^n7#M at W
zacI?JXc87Trm=JyhG$}=8Kd(s)`k_GShWDFlUTC^M=wL+5^kXj6dpmsD}2H)0wO3Z
z5fa6sRD?xDMCk)G*Wh~Gh?{XMy0+nV+=;tzH}1uKcmNOKt9TTTA$=%RIxD8rNA$5!
z=o2C6f=JJxPlZ7j#Vq<<nDhlIr|>kM!E<;XKSTOb%%&@14t*u&($`2=@msu(H<7*(
zEp$z^(s!sRPM*)*;|FokPB%n?zL&0{ynq+-B3{D9e7YsN=m+r-^^;sK9L2|Qa)Ril
ze7TNhdJ-Ss$VHD>OoITGW<HO%@lL*gCpirU(Mv->ETy3!mQ#U at U%^-LHGDm%LJ)ma
z1Y#wX0E(@A8{f`%a&;Hq%_r>T`}hHVh*KFv-13T&C(6(m%Db%`B8r4P5k{3jqhxiO
zp(2kQ(in*7m}U7H#{sIx_;G%MpW>(a8GeqR=Tr|-l$BC!5=4nnwiR<w;~lqyrb%~J
zKv883%E;W2oGIsgnP27941q@&&6AV3$$#WPPNFt>r=(f-U?hQ-Kxj2C16m=MK^aU#
zXebp>A;lcIL!3Rw14OGJCMQO;24b%4dp$4!O|ZDUjDe%doWSzAV$_VT)vF^_(TZ>^
zUg|{^U8D;1Fvw%8y`-uv<9E(0^SJBKP$qa4yNA^_p_k&T$ydfPPL>i-f_0g{JB!k5
zy=bfrwDqMD?L8fdvfk8^uH<|NZzN(<<Pz`_LbjMqWfM{1N_N$zvbkhqw_ at W;)y6HF
zjZX<1pVn=B!}tl)>`U_}W<$Yca~rsA-UuF>cY)XDGW6NJ3jH>pz<|w{F=#Vz%jO|G
zl;w18MwPtSuBhjwHqYc?o0sv3&6|1D=A%4j^Cezk^L1Wj^B}6e%}!nJri~DHtcx2)
z^yJBtnQ-2)VQWe at g9w*Kt3^mv)zD}w<o5+VR%ytxf|*%)Nl!%hLdAh%D-sBJEz4_p
zy*{7(#ruPq(|M81!>U1v?nSA-%#FM at Wkg+hL$tmz)2RMBT`4(M-{`PEZ)as)tfBlK
z&&c6-RgaX_Bkn1WXo&s1JT5BrXEguZeXnYrY}oXRu!@HAdz#7eFJeYE441xvuvBNP
z+>vbUPIP3_YT)30^@KCs-qXEkVdCL|p!S|*Z>lw!8t`;=B$BDFRNsK7d+EYdS4&4%
z_ke48SL#nKy?xznJqsOwNc5)Sot<6DL__PMcq+AIAS|)?Zf at y{CGYmg7I)R>ddZ1R
zOr{d^6HBHiQdzZ3&;9E&y2wk*ai+T3Te2suBhfn`oEu9|XJ>CBm1{md7trzh*1n#l
zIR!`C!k)IMBky93ytC*qqqB^zF=Y%eWuCNU(&QgZ*@;Xk<4id!Z4k+NX)~Qnxh`$9
o!bT{k6d5S3D32+f($-7cE$x!DA*vLTc3j#eX+!dv(wNWw2R|%qEC2ui

delta 1903
zcmY+E32anF7{}jy-}~?I`j|e4WxH9{?XrDt*V2@=)`HkzSr8B#z}g at nL8P!ns4WVJ
z!D~eWM2u+J2vOsSV$i5j6t4gtQKCeQ7hVyfMv0>01xAUkv#mb$d->1&=KALP-W(Wl
zu;$jt at LJT`J)+i at uIGBv{l9s>+c&-?-V&V|O=er&HaF>p-6r$-?gY2Pl-l|pI7 at W&
z9c^fKNB=!EZgStK<?nKRe;#)|uindeQBQPx(#&HL&np(hiG}2xs2St=N#Biz!(rx$
zR8^IP)M6VPE#Se0QbHxN+OS3#5kwKI2jWN|sa>$lHp-^4tw;$W&^$)K8LfpQxJ3&E
zU$BcbW8&l~ZOE`d9rM)bsApCV8n_^5%=Bi at nTsqM(S&9>T3`%gL0&FKi?ax0-EQ<;
zhE``Oa?Wy0z(mF>eN`BfF$HbRt6faRbk^%`yy at m!`ii)1^X+%ub<cfU(5 at W|9oi9?
zp_8!4y6C)f2Y<NP>O4Nn$?jaT4l^-Z%hE}hqg4WPwLalcsnm*jI!rOlXBFKRdnEp7
z;_>7YsqMuhlqtX(u(PiV$6%G_b at kg*C9_!8&%Lm7_e-z5y65%1Z@%^RyYGL1yzfXU
zEP+LS`BWDB3~gE0wHhDe(CveDa?RoPXRDoY=<!o<7Fws5u+$eaCQKJHVk2DUyGV^2
z?^#yC6kpu=L#b?7?n@~vhL><T>rW4U_QjWnzCQfjksp6P`s;7UaYfos{nv?IcC;!T
z|9AeY(&j>NO*$hhhE}7Hu5-#u!|HUs=`&!sA>F_wtVw5`5#{mghBAmkG=!=MH`X;)
zP$i|CiVN(TLbEQ<Rz4owR5yAURa1IQ{{qUM%DX6A|MbIimUV9qH?tf;Bk5G~N0E<P
zOepoi+#E;7(<HipE~HNCq9xQz%V;HCN!QW_+DH%6WAr5LpqEKtbL1R4j}p at i-X1NS
zPp-tBxGTyLje8o{iRhSv`x;mpz?OjujR#}6p4L)AiRgMV+kXh#;@CuY(^g8{VLUyO
zxZk9Y;&D8YNDFLFq at SngE@H;hcm~gzh}cK5cPPG}5+76Y0Hr=B6;V-D at DwYk7+&D;
z_w*ATqu=RI(!?RI;7T6CRXmKVxrS?b1dn9wG;zOMwcsVZq9k5b3VYO8ysk9%s#d(I
z!g!128GI&>=Sh4HPh-5Ta(Guw!24<<K44tH7jh?eF+Nh0 at rf!-!KW-U%XlS+>#k(g
zhJ$Kq*1>0{ay at V0jl7BPU^N|IsCIm*I&cVIGe69aF%GL4_>SQ{&vm<4%~Z2-gykE&
zkKf__{4pP3{HW&OXEhf`)ja&l_=dmdpZFN#H#Hx})dKv%JdNy-jZ!KvOn at +k*j4s0
zyV^#GGK)kxOokAuD6^P{5voZLxk}{B^FTH`B1EFXmfDN6!x1=&BgJ3j#o0)-p0&J|
z<I?I6c$2ISh~vnuM&fSf9MUo+{rckiLA8bMIP30I=c*O?1xpv^M=oEnY;o@*%idu_
zoGXO1H`*-nqFtD{uqB(HvRzC-fE_Zx`A&dWxB(VafNQ(}AJYF|K|B-whs7t6087yT
zYbh4+nG_HB4N3(34J88}#i at YLXFuRxP6xb}GXp%!>}M<21r0m6KH%55A>gk#8}Ogp
z7;vTCG{7NG2(F==wI-+c8Y`Y$k*eHPQ%h=$$~cab8JjNVD|%Hj>#Ml2L?Ve)@!N`Q
zvBjMg8FiW!TtV0Ar&zu7d-4m5+Km_24~`<=bLomz+cyt>FI3D7y?V(dOFJ8_L3^5E
zUdS5+wg~JNI4p2n$Vw7&lhJ;oE*A2b(Ii{Ql|~yN<UXUWkdRxA_8WzyX*cRJT5I&0
N(J`Z{L8g9?_zTk~Z)yMl

diff --git a/mlir/test/Dialect/Builtin/Bytecode/types.mlir b/mlir/test/Dialect/Builtin/Bytecode/types.mlir
index 91d4512998b99..e1d4a18a33c73 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/types.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/types.mlir
@@ -85,9 +85,11 @@ module @TestIndex attributes {
 // CHECK-LABEL: @TestFunc
 module @TestFunc attributes {
   // CHECK: bytecode.func = () -> (),
-  // CHECK: bytecode.func1 = (i1) -> i32
+  // CHECK: bytecode.func1 = (i1) -> i32,
+  // CHECK: bytecode.func2 = (i32) -> (f32, f64)
   bytecode.func = () -> (),
-  bytecode.func1 = (i1) -> (i32)
+  bytecode.func1 = (i1) -> (i32),
+  bytecode.func2 = (i32) -> (f32, f64)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -97,9 +99,13 @@ module @TestFunc attributes {
 // CHECK-LABEL: @TestMemRef
 module @TestMemRef attributes {
   // CHECK: bytecode.test = memref<2xi8>,
-  // CHECK: bytecode.test1 = memref<2xi8, 1>
+  // CHECK: bytecode.test1 = memref<2xi8, 1>,
+  // CHECK: bytecode.test2 = memref<?x?xf32>,
+  // CHECK: bytecode.test3 = memref<2x3xf32, strided<[3, 1]>>
   bytecode.test = memref<2xi8>,
-  bytecode.test1 = memref<2xi8, 1>
+  bytecode.test1 = memref<2xi8, 1>,
+  bytecode.test2 = memref<?x?xf32>,
+  bytecode.test3 = memref<2x3xf32, strided<[3, 1], offset: 0>>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -165,9 +171,11 @@ module @TestUnrankedTensor attributes {
 // CHECK-LABEL: @TestVector
 module @TestVector attributes {
   // CHECK: bytecode.test = vector<8x8x128xi8>,
-  // CHECK: bytecode.test1 = vector<8x[8]xf32>
+  // CHECK: bytecode.test1 = vector<8x[8]xf32>,
+  // CHECK: bytecode.test2 = vector<[4]x[4]xf16>
   bytecode.test = vector<8x8x128xi8>,
-  bytecode.test1 = vector<8x[8]xf32>
+  bytecode.test1 = vector<8x[8]xf32>,
+  bytecode.test2 = vector<[4]x[4]xf16>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -179,3 +187,21 @@ module @TestToken attributes {
   // CHECK: bytecode.test = token
   bytecode.test = token
 } {}
+
+//===----------------------------------------------------------------------===//
+// IntegerType (packing variants)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerTypePacking
+module @TestIntegerTypePacking attributes {
+  // CHECK-DAG: bytecode.i1 = i1
+  // CHECK-DAG: bytecode.i2 = i2
+  // CHECK-DAG: bytecode.si1 = si1
+  // CHECK-DAG: bytecode.ui1 = ui1
+  // CHECK-DAG: bytecode.i4096 = i4096
+  bytecode.i1 = i1,
+  bytecode.i2 = i2,
+  bytecode.si1 = si1,
+  bytecode.ui1 = ui1,
+  bytecode.i4096 = i4096
+} {}


From llvmlistbot at llvm.org  Fri Jul  3 23:18:50 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 23:18:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Address remaining missing encodings
 for builtin dialect. (PR #207488)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207488@github.com>
Message-ID: <6a48a5ca.eb165e5c.113c2a.4fa0@mx.google.com>


https://github.com/jpienaar auto_merge_enabled https://github.com/llvm/llvm-project/pull/207488

From llvmlistbot at llvm.org  Fri Jul  3 23:19:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 23:19:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Address remaining missing encodings
 for builtin dialect. (PR #207488)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207488@github.com>
Message-ID: <6a48a5e1.f1c1923d.3672e2.1c49@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Jacques Pienaar (jpienaar)

<details>
<summary>Changes</summary>

Simple ones that got missed (graph &amp; strided layout), and add some additional tests. Also update the fixed test.

Assisted-By: Gemini

---

Patch is 22.83 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207488.diff


5 Files Affected:

- (modified) mlir/include/mlir/IR/BuiltinDialectBytecode.td (+15-1) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/attrs.mlir (+151-11) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir (+100-2) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc () 
- (modified) mlir/test/Dialect/Builtin/Bytecode/types.mlir (+32-6) 


``````````diff
diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index f087e5cb23081..d098793989dc4 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -242,6 +242,13 @@ def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
   let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
 }
 
+class EnableStridedPrintingJuly2026<dag d> : DialectAttribute<d>;
+
+def StridedLayoutAttr : EnableStridedPrintingJuly2026<(attr
+  SignedVarInt:$offset,
+  Array<SignedVarIntList>:$strides
+)>;
+
 // Types
 // -----
 
@@ -264,6 +271,11 @@ def FunctionType : DialectType<(type
   Array<Type>:$results
 )>;
 
+def GraphType : DialectType<(type
+  Array<Type>:$inputs,
+  Array<Type>:$results
+)>;
+
 def BFloat16Type : DialectType<(type)>;
 
 def Float16Type : DialectType<(type)>;
@@ -417,6 +429,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     FileLineColRange,
     AffineMapAttr,
     IntegerSetAttr,
+    StridedLayoutAttr,
   ];
 }
 
@@ -455,7 +468,8 @@ def BuiltinDialectTypes : DialectTypes<"Builtin"> {
     Float6E2M3FNType,
     Float6E3M2FNType,
     Float8E8M0FNUType,
-    TokenType
+    TokenType,
+    GraphType
   ];
 }
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 3364a8e9fce8e..7dedfbdff10a7 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -29,13 +29,55 @@ module @TestDenseArray attributes {
 //===----------------------------------------------------------------------===//
 
 // CHECK-LABEL: @TestDenseTypedElements
-// CHECK: bytecode.test1 = dense<true> : tensor<256xi1>
-// CHECK: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
-// CHECK: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.test1 = dense<true> : tensor<256xi1>
+// CHECK-DAG: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
+// CHECK-DAG: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.i16 = dense<[100, -200]> : tensor<2xi16>
+// CHECK-DAG: bytecode.i64 = dense<1000000> : tensor<4xi64>
+// CHECK-DAG: bytecode.f16 = dense<[1.500000e+00, 2.500000e+00]> : tensor<2xf16>
+// CHECK-DAG: bytecode.bf16 = dense<[-5.000000e-01, 5.000000e-01]> : tensor<2xbf16>
+// CHECK-DAG: bytecode.complex_val = dense<(1.000000e+00,2.000000e+00)> : tensor<1xcomplex<f32>>
 module @TestDenseTypedElements attributes {
   bytecode.test1 = dense<true> : tensor<256xi1>,
   bytecode.test2 = dense<[10, 32, 255]> : tensor<3xi8>,
-  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>
+  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>,
+  bytecode.i16 = dense<[100, -200]> : tensor<2xi16>,
+  bytecode.i64 = dense<1000000> : tensor<4xi64>,
+  bytecode.f16 = dense<[1.5, 2.5]> : tensor<2xf16>,
+  bytecode.bf16 = dense<[-0.5, 0.5]> : tensor<2xbf16>,
+  bytecode.complex_val = dense<(1.0, 2.0)> : tensor<1xcomplex<f32>>
+} {}
+
+// DenseI1 (single, 8-element, varied sizes)
+
+// CHECK-LABEL: @TestDenseI1SingleElement
+module @TestDenseI1SingleElement attributes {
+  // CHECK-DAG: bytecode.false_1 = dense<false> : tensor<1xi1>
+  // CHECK-DAG: bytecode.true_1 = dense<true> : tensor<1xi1>
+  bytecode.false_1 = dense<false> : tensor<1xi1>,
+  bytecode.true_1 = dense<true> : tensor<1xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1EightElements
+module @TestDenseI1EightElements attributes {
+  // CHECK-DAG: bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_true_8 = dense<true> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_false_8 = dense<false> : tensor<8xi1>
+  bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>,
+  bytecode.all_true_8 = dense<true> : tensor<8xi1>,
+  bytecode.all_false_8 = dense<false> : tensor<8xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1VariedSizes
+module @TestDenseI1VariedSizes attributes {
+  // CHECK-DAG: bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>
+  // CHECK-DAG: bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>
+  // CHECK-DAG: bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>
+  // CHECK-DAG: bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
+  bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>,
+  bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>,
+  bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>,
+  bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -58,10 +100,16 @@ module @TestFloat attributes {
   // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
   // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
+  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
+  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
+  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
-  bytecode.float3 = -0.5 : bf16
+  bytecode.float3 = -0.5 : bf16,
+  bytecode.inf = 0x7FF0000000000000 : f64,
+  bytecode.nan = 0x7FF8000000000000 : f64,
+  bytecode.ninf = 0xFFF0000000000000 : f64
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -74,10 +122,18 @@ module @TestInt attributes {
   // CHECK: bytecode.int1 = -1 : i8
   // CHECK: bytecode.int2 = 800 : ui64
   // CHECK: bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  // CHECK: bytecode.int4 = true
+  // CHECK: bytecode.int5 = 42 : i32
+  // CHECK: bytecode.int6 = -100 : si32
+  // CHECK: bytecode.int7 = 7 : index
   bytecode.int = false,
   bytecode.int1 = -1 : i8,
   bytecode.int2 = 800 : ui64,
-  bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  bytecode.int3 = 90000000000000000300000000000000000001 : i128,
+  bytecode.int4 = true,
+  bytecode.int5 = 42 : i32,
+  bytecode.int6 = -100 : si32,
+  bytecode.int7 = 7 : index
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -87,7 +143,9 @@ module @TestInt attributes {
 // CHECK-LABEL: @TestSparseElements
 module @TestSparseElements attributes {
   // CHECK-LITERAL: bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
-  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
+  // CHECK-LITERAL: bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
+  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>,
+  bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
 } {}
 
 
@@ -99,8 +157,10 @@ module @TestSparseElements attributes {
 module @TestString attributes {
   // CHECK: bytecode.string = "hello"
   // CHECK: bytecode.string2 = "hello" : i32
+  // CHECK: bytecode.string3 = ""
   bytecode.string = "hello",
-  bytecode.string2 = "hello" : i32
+  bytecode.string2 = "hello" : i32,
+  bytecode.string3 = ""
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -146,7 +206,9 @@ module @TestDistinct attributes {
 // CHECK-LABEL: @TestLocCallSite
 module @TestLocCallSite attributes {
   // CHECK: bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
-  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
+  // CHECK: bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
+  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8)),
+  bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -156,7 +218,15 @@ module @TestLocCallSite attributes {
 // CHECK-LABEL: @TestLocFileLineCol
 module @TestLocFileLineCol attributes {
   // CHECK: bytecode.loc = loc("mysource.cc":10:8)
-  bytecode.loc = loc("mysource.cc":10:8)
+  // CHECK: bytecode.loc2 = loc("source.cc":10:8 to 12:4)
+  // CHECK: bytecode.loc3 = loc("source.cc":10:8 to :12)
+  // CHECK: bytecode.loc4 = loc("file.cc":0:0)
+  // CHECK: bytecode.loc5 = loc("file.cc":42:0)
+  bytecode.loc = loc("mysource.cc":10:8),
+  bytecode.loc2 = loc("source.cc":10:8 to 12:4),
+  bytecode.loc3 = loc("source.cc":10:8 to :12),
+  bytecode.loc4 = loc("file.cc":0:0),
+  bytecode.loc5 = loc("file.cc":42:0)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -221,9 +291,11 @@ module @TestAffineMap attributes {
   // Identity maps.
   // CHECK: bytecode.identity_1d = affine_map<(d0) -> (d0)>
   // CHECK: bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+  // CHECK: bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>
 
   // Multi-result maps.
   // CHECK: bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>
+  // CHECK: bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>
   // CHECK: bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>
 
   // Permutation maps.
@@ -247,7 +319,9 @@ module @TestAffineMap attributes {
   bytecode.empty_map = affine_map<() -> ()>,
   bytecode.identity_1d = affine_map<(d0) -> (d0)>,
   bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>,
+  bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>,
   bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>,
+  bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>,
   bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>,
   bytecode.perm_2d = affine_map<(d0, d1) -> (d1, d0)>,
   bytecode.perm_3d = affine_map<(d0, d1, d2) -> (d2, d0, d1)>,
@@ -265,9 +339,75 @@ module @TestIntegerSetAttr attributes {
   // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
   // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
   // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>
+  // CHECK-DAG: bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>
+  // CHECK-DAG: bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
   bytecode.eq = affine_set<(d0) : (d0 == 0)>,
   bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
   bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
-  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>,
+  bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>,
+  bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>,
+  bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
+} {}
+
+//===----------------------------------------------------------------------===//
+// DictionaryAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestDictionaryAttr
+module @TestDictionaryAttr attributes {
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.nested_dict = {inner = {a = 1 : i32}}
+  bytecode.empty_dict = {},
+  bytecode.nested_dict = {inner = {a = 1 : i32}}
+} {}
+
+//===----------------------------------------------------------------------===//
+// UnitAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestUnitAttr
+module @TestUnitAttr attributes {
+  // CHECK-DAG: bytecode.unit
+  bytecode.unit
+} {}
+
+//===----------------------------------------------------------------------===//
+// EmptyContainers
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestEmptyContainers
+module @TestEmptyContainers attributes {
+  // CHECK-DAG: bytecode.empty_arr = []
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.empty_str = ""
+  // CHECK-DAG: bytecode.empty_tuple = tuple<>
+  // CHECK-DAG: bytecode.empty_func = () -> ()
+  // CHECK-DAG: bytecode.empty_map = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.empty_dense_arr = array<i32>
+  // CHECK-DAG: bytecode.empty_tensor = dense<> : tensor<0xi32>
+  bytecode.empty_arr = [],
+  bytecode.empty_dict = {},
+  bytecode.empty_str = "",
+  bytecode.empty_tuple = tuple<>,
+  bytecode.empty_func = () -> (),
+  bytecode.empty_map = affine_map<() -> ()>,
+  bytecode.empty_dense_arr = array<i32>,
+  bytecode.empty_tensor = dense<> : tensor<0xi32>
+} {}
+
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
 } {}
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index 31061a6fa8653..d427035435277 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -127,12 +127,36 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
   // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
   // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
+  // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = 1.000000e+00 : f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = 1.000000e+00 : f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = 1.000000e+00 : f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = 1.000000e+00 : f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = 1.000000e+00 : f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = 1.000000e+00 : f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = 1.000000e+00 : f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = 1.000000e+00 : f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = 1.000000e+00 : f8E8M0FNU
   bytecode.bf16 = -0.5 : bf16,
   bytecode.f16 = 1.5 : f16,
   bytecode.f32 = 3.14 : f32,
   bytecode.f64 = 10.0 : f64,
   bytecode.f80 = 0.1 : f80,
-  bytecode.f128 = 0.1 : f128
+  bytecode.f128 = 0.1 : f128,
+  bytecode.tf32 = 1.0 : tf32,
+  bytecode.f8E5M2 = 1.0 : f8E5M2,
+  bytecode.f8E4M3 = 1.0 : f8E4M3,
+  bytecode.f8E4M3FN = 1.0 : f8E4M3FN,
+  bytecode.f8E5M2FNUZ = 1.0 : f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = 1.0 : f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = 1.0 : f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = 1.0 : f8E3M4,
+  bytecode.f4E2M1FN = 1.0 : f4E2M1FN,
+  bytecode.f6E2M3FN = 1.0 : f6E2M3FN,
+  bytecode.f6E3M2FN = 1.0 : f6E3M2FN,
+  bytecode.f8E8M0FNU = 1.0 : f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -260,12 +284,36 @@ module @TestAllFloatTypes attributes {
   // CHECK-DAG: bytecode.f64 = f64
   // CHECK-DAG: bytecode.f80 = f80
   // CHECK-DAG: bytecode.f128 = f128
+  // CHECK-DAG: bytecode.tf32 = tf32
+  // CHECK-DAG: bytecode.f8E5M2 = f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = f8E8M0FNU
   bytecode.bf16 = bf16,
   bytecode.f16 = f16,
   bytecode.f32 = f32,
   bytecode.f64 = f64,
   bytecode.f80 = f80,
-  bytecode.f128 = f128
+  bytecode.f128 = f128,
+  bytecode.tf32 = tf32,
+  bytecode.f8E5M2 = f8E5M2,
+  bytecode.f8E4M3 = f8E4M3,
+  bytecode.f8E4M3FN = f8E4M3FN,
+  bytecode.f8E5M2FNUZ = f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = f8E3M4,
+  bytecode.f4E2M1FN = f4E2M1FN,
+  bytecode.f6E2M3FN = f6E2M3FN,
+  bytecode.f6E3M2FN = f6E3M2FN,
+  bytecode.f8E8M0FNU = f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -396,6 +444,42 @@ module @TestTupleType attributes {
   bytecode.mixed_tuple = tuple<i32, f64, index>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// AffineMapAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestAffineMapAttr
+module @TestAffineMapAttr attributes {
+  // CHECK-DAG: bytecode.identity = affine_map<(d0) -> (d0)>
+  // CHECK-DAG: bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>
+  // CHECK-DAG: bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>
+  // CHECK-DAG: bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>
+  // CHECK-DAG: bytecode.empty = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.constant = affine_map<() -> (42)>
+  bytecode.identity = affine_map<(d0) -> (d0)>,
+  bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>,
+  bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>,
+  bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>,
+  bytecode.empty = affine_map<() -> ()>,
+  bytecode.constant = affine_map<() -> (42)>
+} {} loc(unknown)
+
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {} loc(unknown)
+
 //===----------------------------------------------------------------------===//
 // DenseResourceElementsAttr
 //===----------------------------------------------------------------------===//
@@ -416,6 +500,20 @@ module @TestDenseStringElementsAttr attributes {
   bytecode.dense_str = dense<["foo", "bar", "baz"]> : tensor<3x!bytecode.string>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
+} {} loc(unknown)
+
 } loc(unknown)
 
 {-#
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/bu...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207488

From llvmlistbot at llvm.org  Fri Jul  3 23:19:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 23:19:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Address remaining missing encodings
 for builtin dialect. (PR #207488)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207488@github.com>
Message-ID: <6a48a5e2.abb10101.208f3e.b684@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jacques Pienaar (jpienaar)

<details>
<summary>Changes</summary>

Simple ones that got missed (graph &amp; strided layout), and add some additional tests. Also update the fixed test.

Assisted-By: Gemini

---

Patch is 22.83 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207488.diff


5 Files Affected:

- (modified) mlir/include/mlir/IR/BuiltinDialectBytecode.td (+15-1) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/attrs.mlir (+151-11) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir (+100-2) 
- (modified) mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc () 
- (modified) mlir/test/Dialect/Builtin/Bytecode/types.mlir (+32-6) 


``````````diff
diff --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index f087e5cb23081..d098793989dc4 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -242,6 +242,13 @@ def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
   let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
 }
 
+class EnableStridedPrintingJuly2026<dag d> : DialectAttribute<d>;
+
+def StridedLayoutAttr : EnableStridedPrintingJuly2026<(attr
+  SignedVarInt:$offset,
+  Array<SignedVarIntList>:$strides
+)>;
+
 // Types
 // -----
 
@@ -264,6 +271,11 @@ def FunctionType : DialectType<(type
   Array<Type>:$results
 )>;
 
+def GraphType : DialectType<(type
+  Array<Type>:$inputs,
+  Array<Type>:$results
+)>;
+
 def BFloat16Type : DialectType<(type)>;
 
 def Float16Type : DialectType<(type)>;
@@ -417,6 +429,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     FileLineColRange,
     AffineMapAttr,
     IntegerSetAttr,
+    StridedLayoutAttr,
   ];
 }
 
@@ -455,7 +468,8 @@ def BuiltinDialectTypes : DialectTypes<"Builtin"> {
     Float6E2M3FNType,
     Float6E3M2FNType,
     Float8E8M0FNUType,
-    TokenType
+    TokenType,
+    GraphType
   ];
 }
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 3364a8e9fce8e..7dedfbdff10a7 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -29,13 +29,55 @@ module @TestDenseArray attributes {
 //===----------------------------------------------------------------------===//
 
 // CHECK-LABEL: @TestDenseTypedElements
-// CHECK: bytecode.test1 = dense<true> : tensor<256xi1>
-// CHECK: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
-// CHECK: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.test1 = dense<true> : tensor<256xi1>
+// CHECK-DAG: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
+// CHECK-DAG: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.i16 = dense<[100, -200]> : tensor<2xi16>
+// CHECK-DAG: bytecode.i64 = dense<1000000> : tensor<4xi64>
+// CHECK-DAG: bytecode.f16 = dense<[1.500000e+00, 2.500000e+00]> : tensor<2xf16>
+// CHECK-DAG: bytecode.bf16 = dense<[-5.000000e-01, 5.000000e-01]> : tensor<2xbf16>
+// CHECK-DAG: bytecode.complex_val = dense<(1.000000e+00,2.000000e+00)> : tensor<1xcomplex<f32>>
 module @TestDenseTypedElements attributes {
   bytecode.test1 = dense<true> : tensor<256xi1>,
   bytecode.test2 = dense<[10, 32, 255]> : tensor<3xi8>,
-  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>
+  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>,
+  bytecode.i16 = dense<[100, -200]> : tensor<2xi16>,
+  bytecode.i64 = dense<1000000> : tensor<4xi64>,
+  bytecode.f16 = dense<[1.5, 2.5]> : tensor<2xf16>,
+  bytecode.bf16 = dense<[-0.5, 0.5]> : tensor<2xbf16>,
+  bytecode.complex_val = dense<(1.0, 2.0)> : tensor<1xcomplex<f32>>
+} {}
+
+// DenseI1 (single, 8-element, varied sizes)
+
+// CHECK-LABEL: @TestDenseI1SingleElement
+module @TestDenseI1SingleElement attributes {
+  // CHECK-DAG: bytecode.false_1 = dense<false> : tensor<1xi1>
+  // CHECK-DAG: bytecode.true_1 = dense<true> : tensor<1xi1>
+  bytecode.false_1 = dense<false> : tensor<1xi1>,
+  bytecode.true_1 = dense<true> : tensor<1xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1EightElements
+module @TestDenseI1EightElements attributes {
+  // CHECK-DAG: bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_true_8 = dense<true> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_false_8 = dense<false> : tensor<8xi1>
+  bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>,
+  bytecode.all_true_8 = dense<true> : tensor<8xi1>,
+  bytecode.all_false_8 = dense<false> : tensor<8xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1VariedSizes
+module @TestDenseI1VariedSizes attributes {
+  // CHECK-DAG: bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>
+  // CHECK-DAG: bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>
+  // CHECK-DAG: bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>
+  // CHECK-DAG: bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
+  bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>,
+  bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>,
+  bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>,
+  bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -58,10 +100,16 @@ module @TestFloat attributes {
   // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
   // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
+  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
+  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
+  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
-  bytecode.float3 = -0.5 : bf16
+  bytecode.float3 = -0.5 : bf16,
+  bytecode.inf = 0x7FF0000000000000 : f64,
+  bytecode.nan = 0x7FF8000000000000 : f64,
+  bytecode.ninf = 0xFFF0000000000000 : f64
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -74,10 +122,18 @@ module @TestInt attributes {
   // CHECK: bytecode.int1 = -1 : i8
   // CHECK: bytecode.int2 = 800 : ui64
   // CHECK: bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  // CHECK: bytecode.int4 = true
+  // CHECK: bytecode.int5 = 42 : i32
+  // CHECK: bytecode.int6 = -100 : si32
+  // CHECK: bytecode.int7 = 7 : index
   bytecode.int = false,
   bytecode.int1 = -1 : i8,
   bytecode.int2 = 800 : ui64,
-  bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  bytecode.int3 = 90000000000000000300000000000000000001 : i128,
+  bytecode.int4 = true,
+  bytecode.int5 = 42 : i32,
+  bytecode.int6 = -100 : si32,
+  bytecode.int7 = 7 : index
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -87,7 +143,9 @@ module @TestInt attributes {
 // CHECK-LABEL: @TestSparseElements
 module @TestSparseElements attributes {
   // CHECK-LITERAL: bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
-  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
+  // CHECK-LITERAL: bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
+  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>,
+  bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
 } {}
 
 
@@ -99,8 +157,10 @@ module @TestSparseElements attributes {
 module @TestString attributes {
   // CHECK: bytecode.string = "hello"
   // CHECK: bytecode.string2 = "hello" : i32
+  // CHECK: bytecode.string3 = ""
   bytecode.string = "hello",
-  bytecode.string2 = "hello" : i32
+  bytecode.string2 = "hello" : i32,
+  bytecode.string3 = ""
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -146,7 +206,9 @@ module @TestDistinct attributes {
 // CHECK-LABEL: @TestLocCallSite
 module @TestLocCallSite attributes {
   // CHECK: bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
-  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
+  // CHECK: bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
+  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8)),
+  bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -156,7 +218,15 @@ module @TestLocCallSite attributes {
 // CHECK-LABEL: @TestLocFileLineCol
 module @TestLocFileLineCol attributes {
   // CHECK: bytecode.loc = loc("mysource.cc":10:8)
-  bytecode.loc = loc("mysource.cc":10:8)
+  // CHECK: bytecode.loc2 = loc("source.cc":10:8 to 12:4)
+  // CHECK: bytecode.loc3 = loc("source.cc":10:8 to :12)
+  // CHECK: bytecode.loc4 = loc("file.cc":0:0)
+  // CHECK: bytecode.loc5 = loc("file.cc":42:0)
+  bytecode.loc = loc("mysource.cc":10:8),
+  bytecode.loc2 = loc("source.cc":10:8 to 12:4),
+  bytecode.loc3 = loc("source.cc":10:8 to :12),
+  bytecode.loc4 = loc("file.cc":0:0),
+  bytecode.loc5 = loc("file.cc":42:0)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -221,9 +291,11 @@ module @TestAffineMap attributes {
   // Identity maps.
   // CHECK: bytecode.identity_1d = affine_map<(d0) -> (d0)>
   // CHECK: bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+  // CHECK: bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>
 
   // Multi-result maps.
   // CHECK: bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>
+  // CHECK: bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>
   // CHECK: bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>
 
   // Permutation maps.
@@ -247,7 +319,9 @@ module @TestAffineMap attributes {
   bytecode.empty_map = affine_map<() -> ()>,
   bytecode.identity_1d = affine_map<(d0) -> (d0)>,
   bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>,
+  bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>,
   bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>,
+  bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>,
   bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>,
   bytecode.perm_2d = affine_map<(d0, d1) -> (d1, d0)>,
   bytecode.perm_3d = affine_map<(d0, d1, d2) -> (d2, d0, d1)>,
@@ -265,9 +339,75 @@ module @TestIntegerSetAttr attributes {
   // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
   // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
   // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>
+  // CHECK-DAG: bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>
+  // CHECK-DAG: bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
   bytecode.eq = affine_set<(d0) : (d0 == 0)>,
   bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
   bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
-  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>,
+  bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>,
+  bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>,
+  bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
+} {}
+
+//===----------------------------------------------------------------------===//
+// DictionaryAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestDictionaryAttr
+module @TestDictionaryAttr attributes {
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.nested_dict = {inner = {a = 1 : i32}}
+  bytecode.empty_dict = {},
+  bytecode.nested_dict = {inner = {a = 1 : i32}}
+} {}
+
+//===----------------------------------------------------------------------===//
+// UnitAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestUnitAttr
+module @TestUnitAttr attributes {
+  // CHECK-DAG: bytecode.unit
+  bytecode.unit
+} {}
+
+//===----------------------------------------------------------------------===//
+// EmptyContainers
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestEmptyContainers
+module @TestEmptyContainers attributes {
+  // CHECK-DAG: bytecode.empty_arr = []
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.empty_str = ""
+  // CHECK-DAG: bytecode.empty_tuple = tuple<>
+  // CHECK-DAG: bytecode.empty_func = () -> ()
+  // CHECK-DAG: bytecode.empty_map = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.empty_dense_arr = array<i32>
+  // CHECK-DAG: bytecode.empty_tensor = dense<> : tensor<0xi32>
+  bytecode.empty_arr = [],
+  bytecode.empty_dict = {},
+  bytecode.empty_str = "",
+  bytecode.empty_tuple = tuple<>,
+  bytecode.empty_func = () -> (),
+  bytecode.empty_map = affine_map<() -> ()>,
+  bytecode.empty_dense_arr = array<i32>,
+  bytecode.empty_tensor = dense<> : tensor<0xi32>
+} {}
+
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
 } {}
 
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index 31061a6fa8653..d427035435277 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -127,12 +127,36 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
   // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
   // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
+  // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = 1.000000e+00 : f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = 1.000000e+00 : f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = 1.000000e+00 : f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = 1.000000e+00 : f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = 1.000000e+00 : f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = 1.000000e+00 : f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = 1.000000e+00 : f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = 1.000000e+00 : f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = 1.000000e+00 : f8E8M0FNU
   bytecode.bf16 = -0.5 : bf16,
   bytecode.f16 = 1.5 : f16,
   bytecode.f32 = 3.14 : f32,
   bytecode.f64 = 10.0 : f64,
   bytecode.f80 = 0.1 : f80,
-  bytecode.f128 = 0.1 : f128
+  bytecode.f128 = 0.1 : f128,
+  bytecode.tf32 = 1.0 : tf32,
+  bytecode.f8E5M2 = 1.0 : f8E5M2,
+  bytecode.f8E4M3 = 1.0 : f8E4M3,
+  bytecode.f8E4M3FN = 1.0 : f8E4M3FN,
+  bytecode.f8E5M2FNUZ = 1.0 : f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = 1.0 : f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = 1.0 : f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = 1.0 : f8E3M4,
+  bytecode.f4E2M1FN = 1.0 : f4E2M1FN,
+  bytecode.f6E2M3FN = 1.0 : f6E2M3FN,
+  bytecode.f6E3M2FN = 1.0 : f6E3M2FN,
+  bytecode.f8E8M0FNU = 1.0 : f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -260,12 +284,36 @@ module @TestAllFloatTypes attributes {
   // CHECK-DAG: bytecode.f64 = f64
   // CHECK-DAG: bytecode.f80 = f80
   // CHECK-DAG: bytecode.f128 = f128
+  // CHECK-DAG: bytecode.tf32 = tf32
+  // CHECK-DAG: bytecode.f8E5M2 = f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = f8E8M0FNU
   bytecode.bf16 = bf16,
   bytecode.f16 = f16,
   bytecode.f32 = f32,
   bytecode.f64 = f64,
   bytecode.f80 = f80,
-  bytecode.f128 = f128
+  bytecode.f128 = f128,
+  bytecode.tf32 = tf32,
+  bytecode.f8E5M2 = f8E5M2,
+  bytecode.f8E4M3 = f8E4M3,
+  bytecode.f8E4M3FN = f8E4M3FN,
+  bytecode.f8E5M2FNUZ = f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = f8E3M4,
+  bytecode.f4E2M1FN = f4E2M1FN,
+  bytecode.f6E2M3FN = f6E2M3FN,
+  bytecode.f6E3M2FN = f6E3M2FN,
+  bytecode.f8E8M0FNU = f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -396,6 +444,42 @@ module @TestTupleType attributes {
   bytecode.mixed_tuple = tuple<i32, f64, index>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// AffineMapAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestAffineMapAttr
+module @TestAffineMapAttr attributes {
+  // CHECK-DAG: bytecode.identity = affine_map<(d0) -> (d0)>
+  // CHECK-DAG: bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>
+  // CHECK-DAG: bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>
+  // CHECK-DAG: bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>
+  // CHECK-DAG: bytecode.empty = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.constant = affine_map<() -> (42)>
+  bytecode.identity = affine_map<(d0) -> (d0)>,
+  bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>,
+  bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>,
+  bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>,
+  bytecode.empty = affine_map<() -> ()>,
+  bytecode.constant = affine_map<() -> (42)>
+} {} loc(unknown)
+
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {} loc(unknown)
+
 //===----------------------------------------------------------------------===//
 // DenseResourceElementsAttr
 //===----------------------------------------------------------------------===//
@@ -416,6 +500,20 @@ module @TestDenseStringElementsAttr attributes {
   bytecode.dense_str = dense<["foo", "bar", "baz"]> : tensor<3x!bytecode.string>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
+} {} loc(unknown)
+
 } loc(unknown)
 
 {-#
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/bu...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207488

From llvmlistbot at llvm.org  Fri Jul  3 23:27:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 03 Jul 2026 23:27:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] c874b1b - [mlirbc] Address remaining missing
 encodings for builtin dialect. (#207488)
Message-ID: <6a48a7cc.331483e7.5bc1e.1d61@mx.google.com>


Author: Jacques Pienaar
Date: 2026-07-04T06:27:19Z
New Revision: c874b1b4b2bd3cefb3479c9fc82af67883b20d28

URL: https://github.com/llvm/llvm-project/commit/c874b1b4b2bd3cefb3479c9fc82af67883b20d28
DIFF: https://github.com/llvm/llvm-project/commit/c874b1b4b2bd3cefb3479c9fc82af67883b20d28.diff

LOG: [mlirbc] Address remaining missing encodings for builtin dialect. (#207488)

Simple ones that got missed (graph & strided layout), and add some
additional tests. Also update the fixed test.

Assisted-By: Gemini

Added: 
    

Modified: 
    mlir/include/mlir/IR/BuiltinDialectBytecode.td
    mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
    mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
    mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
    mlir/test/Dialect/Builtin/Bytecode/types.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/IR/BuiltinDialectBytecode.td b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
index f087e5cb23081..d098793989dc4 100644
--- a/mlir/include/mlir/IR/BuiltinDialectBytecode.td
+++ b/mlir/include/mlir/IR/BuiltinDialectBytecode.td
@@ -242,6 +242,13 @@ def IntegerSetAttr : EnableIntegerSetPrintingJuly2026<(attr
   let cBuilder = "IntegerSetAttr::get(IntegerSet::get(numDims, numSymbols, constraints, eqFlags))";
 }
 
+class EnableStridedPrintingJuly2026<dag d> : DialectAttribute<d>;
+
+def StridedLayoutAttr : EnableStridedPrintingJuly2026<(attr
+  SignedVarInt:$offset,
+  Array<SignedVarIntList>:$strides
+)>;
+
 // Types
 // -----
 
@@ -264,6 +271,11 @@ def FunctionType : DialectType<(type
   Array<Type>:$results
 )>;
 
+def GraphType : DialectType<(type
+  Array<Type>:$inputs,
+  Array<Type>:$results
+)>;
+
 def BFloat16Type : DialectType<(type)>;
 
 def Float16Type : DialectType<(type)>;
@@ -417,6 +429,7 @@ def BuiltinDialectAttributes : DialectAttributes<"Builtin"> {
     FileLineColRange,
     AffineMapAttr,
     IntegerSetAttr,
+    StridedLayoutAttr,
   ];
 }
 
@@ -455,7 +468,8 @@ def BuiltinDialectTypes : DialectTypes<"Builtin"> {
     Float6E2M3FNType,
     Float6E3M2FNType,
     Float8E8M0FNUType,
-    TokenType
+    TokenType,
+    GraphType
   ];
 }
 

diff  --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 3364a8e9fce8e..7dedfbdff10a7 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -29,13 +29,55 @@ module @TestDenseArray attributes {
 //===----------------------------------------------------------------------===//
 
 // CHECK-LABEL: @TestDenseTypedElements
-// CHECK: bytecode.test1 = dense<true> : tensor<256xi1>
-// CHECK: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
-// CHECK: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.test1 = dense<true> : tensor<256xi1>
+// CHECK-DAG: bytecode.test2 = dense<[10, 32, -1]> : tensor<3xi8>
+// CHECK-DAG: bytecode.test3 = dense<[1.{{.*}}e+01, 3.2{{.*}}e+01, 1.809{{.*}}e+03]> : tensor<3xf64>
+// CHECK-DAG: bytecode.i16 = dense<[100, -200]> : tensor<2xi16>
+// CHECK-DAG: bytecode.i64 = dense<1000000> : tensor<4xi64>
+// CHECK-DAG: bytecode.f16 = dense<[1.500000e+00, 2.500000e+00]> : tensor<2xf16>
+// CHECK-DAG: bytecode.bf16 = dense<[-5.000000e-01, 5.000000e-01]> : tensor<2xbf16>
+// CHECK-DAG: bytecode.complex_val = dense<(1.000000e+00,2.000000e+00)> : tensor<1xcomplex<f32>>
 module @TestDenseTypedElements attributes {
   bytecode.test1 = dense<true> : tensor<256xi1>,
   bytecode.test2 = dense<[10, 32, 255]> : tensor<3xi8>,
-  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>
+  bytecode.test3 = dense<[10.0, 32.0, 1809.0]> : tensor<3xf64>,
+  bytecode.i16 = dense<[100, -200]> : tensor<2xi16>,
+  bytecode.i64 = dense<1000000> : tensor<4xi64>,
+  bytecode.f16 = dense<[1.5, 2.5]> : tensor<2xf16>,
+  bytecode.bf16 = dense<[-0.5, 0.5]> : tensor<2xbf16>,
+  bytecode.complex_val = dense<(1.0, 2.0)> : tensor<1xcomplex<f32>>
+} {}
+
+// DenseI1 (single, 8-element, varied sizes)
+
+// CHECK-LABEL: @TestDenseI1SingleElement
+module @TestDenseI1SingleElement attributes {
+  // CHECK-DAG: bytecode.false_1 = dense<false> : tensor<1xi1>
+  // CHECK-DAG: bytecode.true_1 = dense<true> : tensor<1xi1>
+  bytecode.false_1 = dense<false> : tensor<1xi1>,
+  bytecode.true_1 = dense<true> : tensor<1xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1EightElements
+module @TestDenseI1EightElements attributes {
+  // CHECK-DAG: bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_true_8 = dense<true> : tensor<8xi1>
+  // CHECK-DAG: bytecode.all_false_8 = dense<false> : tensor<8xi1>
+  bytecode.mixed_8 = dense<[true, false, true, false, true, false, true, false]> : tensor<8xi1>,
+  bytecode.all_true_8 = dense<true> : tensor<8xi1>,
+  bytecode.all_false_8 = dense<false> : tensor<8xi1>
+} {}
+
+// CHECK-LABEL: @TestDenseI1VariedSizes
+module @TestDenseI1VariedSizes attributes {
+  // CHECK-DAG: bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>
+  // CHECK-DAG: bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>
+  // CHECK-DAG: bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>
+  // CHECK-DAG: bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
+  bytecode.i1_2 = dense<[true, false]> : tensor<2xi1>,
+  bytecode.i1_7 = dense<[true, false, true, false, true, false, true]> : tensor<7xi1>,
+  bytecode.i1_9 = dense<[true, false, true, false, true, false, true, false, true]> : tensor<9xi1>,
+  bytecode.i1_16 = dense<[true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false]> : tensor<16xi1>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -58,10 +100,16 @@ module @TestFloat attributes {
   // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
   // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
+  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
+  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
+  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
-  bytecode.float3 = -0.5 : bf16
+  bytecode.float3 = -0.5 : bf16,
+  bytecode.inf = 0x7FF0000000000000 : f64,
+  bytecode.nan = 0x7FF8000000000000 : f64,
+  bytecode.ninf = 0xFFF0000000000000 : f64
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -74,10 +122,18 @@ module @TestInt attributes {
   // CHECK: bytecode.int1 = -1 : i8
   // CHECK: bytecode.int2 = 800 : ui64
   // CHECK: bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  // CHECK: bytecode.int4 = true
+  // CHECK: bytecode.int5 = 42 : i32
+  // CHECK: bytecode.int6 = -100 : si32
+  // CHECK: bytecode.int7 = 7 : index
   bytecode.int = false,
   bytecode.int1 = -1 : i8,
   bytecode.int2 = 800 : ui64,
-  bytecode.int3 = 90000000000000000300000000000000000001 : i128
+  bytecode.int3 = 90000000000000000300000000000000000001 : i128,
+  bytecode.int4 = true,
+  bytecode.int5 = 42 : i32,
+  bytecode.int6 = -100 : si32,
+  bytecode.int7 = 7 : index
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -87,7 +143,9 @@ module @TestInt attributes {
 // CHECK-LABEL: @TestSparseElements
 module @TestSparseElements attributes {
   // CHECK-LITERAL: bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
-  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>
+  // CHECK-LITERAL: bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
+  bytecode.sparse = sparse<[[0, 0], [1, 2]], [1, 5]> : tensor<3x4xi32>,
+  bytecode.sparse_1d = sparse<[1, 3], [10.0, 20.0]> : tensor<5xf32>
 } {}
 
 
@@ -99,8 +157,10 @@ module @TestSparseElements attributes {
 module @TestString attributes {
   // CHECK: bytecode.string = "hello"
   // CHECK: bytecode.string2 = "hello" : i32
+  // CHECK: bytecode.string3 = ""
   bytecode.string = "hello",
-  bytecode.string2 = "hello" : i32
+  bytecode.string2 = "hello" : i32,
+  bytecode.string3 = ""
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -146,7 +206,9 @@ module @TestDistinct attributes {
 // CHECK-LABEL: @TestLocCallSite
 module @TestLocCallSite attributes {
   // CHECK: bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
-  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8))
+  // CHECK: bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
+  bytecode.loc = loc(callsite("foo" at "mysource.cc":10:8)),
+  bytecode.loc2 = loc(callsite("a":1:1 at callsite("b":2:2 at "c":3:3)))
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -156,7 +218,15 @@ module @TestLocCallSite attributes {
 // CHECK-LABEL: @TestLocFileLineCol
 module @TestLocFileLineCol attributes {
   // CHECK: bytecode.loc = loc("mysource.cc":10:8)
-  bytecode.loc = loc("mysource.cc":10:8)
+  // CHECK: bytecode.loc2 = loc("source.cc":10:8 to 12:4)
+  // CHECK: bytecode.loc3 = loc("source.cc":10:8 to :12)
+  // CHECK: bytecode.loc4 = loc("file.cc":0:0)
+  // CHECK: bytecode.loc5 = loc("file.cc":42:0)
+  bytecode.loc = loc("mysource.cc":10:8),
+  bytecode.loc2 = loc("source.cc":10:8 to 12:4),
+  bytecode.loc3 = loc("source.cc":10:8 to :12),
+  bytecode.loc4 = loc("file.cc":0:0),
+  bytecode.loc5 = loc("file.cc":42:0)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -221,9 +291,11 @@ module @TestAffineMap attributes {
   // Identity maps.
   // CHECK: bytecode.identity_1d = affine_map<(d0) -> (d0)>
   // CHECK: bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+  // CHECK: bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>
 
   // Multi-result maps.
   // CHECK: bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>
+  // CHECK: bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>
   // CHECK: bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>
 
   // Permutation maps.
@@ -247,7 +319,9 @@ module @TestAffineMap attributes {
   bytecode.empty_map = affine_map<() -> ()>,
   bytecode.identity_1d = affine_map<(d0) -> (d0)>,
   bytecode.identity_3d = affine_map<(d0, d1, d2) -> (d0, d1, d2)>,
+  bytecode.identity_large = affine_map<(d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15) -> (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9, d10, d11, d12, d13, d14, d15)>,
   bytecode.multi_mix = affine_map<(d0, d1) -> (d0 + d1, d0 * 2, d1 mod 5)>,
+  bytecode.multi_mix_deep = affine_map<(d0, d1)[s0] -> (((d0 + d1) floordiv 4 + s0) ceildiv 2 + (d0 mod 3 + d1 * 5) floordiv 7)>,
   bytecode.multi_sym = affine_map<(d0, d1)[s0] -> (d0 floordiv s0, d1 ceildiv s0, d0 + d1 + s0)>,
   bytecode.perm_2d = affine_map<(d0, d1) -> (d1, d0)>,
   bytecode.perm_3d = affine_map<(d0, d1, d2) -> (d2, d0, d1)>,
@@ -265,9 +339,75 @@ module @TestIntegerSetAttr attributes {
   // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
   // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
   // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>
+  // CHECK-DAG: bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>
+  // CHECK-DAG: bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
   bytecode.eq = affine_set<(d0) : (d0 == 0)>,
   bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
   bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
-  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>,
+  bytecode.neg_const = affine_set<(d0) : (d0 + 42 == 0)>,
+  bytecode.multi_sym = affine_set<(d0)[s0, s1] : (d0 + s0 >= 0, d0 + s1 >= 0)>,
+  bytecode.complex = affine_set<(d0, d1)[s0] : (d0 * 2 + d1 == 0, d0 - s0 >= 0, d1 mod 3 >= 0)>
+} {}
+
+//===----------------------------------------------------------------------===//
+// DictionaryAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestDictionaryAttr
+module @TestDictionaryAttr attributes {
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.nested_dict = {inner = {a = 1 : i32}}
+  bytecode.empty_dict = {},
+  bytecode.nested_dict = {inner = {a = 1 : i32}}
+} {}
+
+//===----------------------------------------------------------------------===//
+// UnitAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestUnitAttr
+module @TestUnitAttr attributes {
+  // CHECK-DAG: bytecode.unit
+  bytecode.unit
+} {}
+
+//===----------------------------------------------------------------------===//
+// EmptyContainers
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestEmptyContainers
+module @TestEmptyContainers attributes {
+  // CHECK-DAG: bytecode.empty_arr = []
+  // CHECK-DAG: bytecode.empty_dict = {}
+  // CHECK-DAG: bytecode.empty_str = ""
+  // CHECK-DAG: bytecode.empty_tuple = tuple<>
+  // CHECK-DAG: bytecode.empty_func = () -> ()
+  // CHECK-DAG: bytecode.empty_map = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.empty_dense_arr = array<i32>
+  // CHECK-DAG: bytecode.empty_tensor = dense<> : tensor<0xi32>
+  bytecode.empty_arr = [],
+  bytecode.empty_dict = {},
+  bytecode.empty_str = "",
+  bytecode.empty_tuple = tuple<>,
+  bytecode.empty_func = () -> (),
+  bytecode.empty_map = affine_map<() -> ()>,
+  bytecode.empty_dense_arr = array<i32>,
+  bytecode.empty_tensor = dense<> : tensor<0xi32>
+} {}
+
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
 } {}
 

diff  --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index 31061a6fa8653..d427035435277 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -127,12 +127,36 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
   // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
   // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
+  // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = 1.000000e+00 : f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = 1.000000e+00 : f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = 1.000000e+00 : f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = 1.000000e+00 : f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = 1.000000e+00 : f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = 1.000000e+00 : f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = 1.000000e+00 : f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = 1.000000e+00 : f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = 1.000000e+00 : f8E8M0FNU
   bytecode.bf16 = -0.5 : bf16,
   bytecode.f16 = 1.5 : f16,
   bytecode.f32 = 3.14 : f32,
   bytecode.f64 = 10.0 : f64,
   bytecode.f80 = 0.1 : f80,
-  bytecode.f128 = 0.1 : f128
+  bytecode.f128 = 0.1 : f128,
+  bytecode.tf32 = 1.0 : tf32,
+  bytecode.f8E5M2 = 1.0 : f8E5M2,
+  bytecode.f8E4M3 = 1.0 : f8E4M3,
+  bytecode.f8E4M3FN = 1.0 : f8E4M3FN,
+  bytecode.f8E5M2FNUZ = 1.0 : f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = 1.0 : f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = 1.0 : f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = 1.0 : f8E3M4,
+  bytecode.f4E2M1FN = 1.0 : f4E2M1FN,
+  bytecode.f6E2M3FN = 1.0 : f6E2M3FN,
+  bytecode.f6E3M2FN = 1.0 : f6E3M2FN,
+  bytecode.f8E8M0FNU = 1.0 : f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -260,12 +284,36 @@ module @TestAllFloatTypes attributes {
   // CHECK-DAG: bytecode.f64 = f64
   // CHECK-DAG: bytecode.f80 = f80
   // CHECK-DAG: bytecode.f128 = f128
+  // CHECK-DAG: bytecode.tf32 = tf32
+  // CHECK-DAG: bytecode.f8E5M2 = f8E5M2
+  // CHECK-DAG: bytecode.f8E4M3 = f8E4M3
+  // CHECK-DAG: bytecode.f8E4M3FN = f8E4M3FN
+  // CHECK-DAG: bytecode.f8E5M2FNUZ = f8E5M2FNUZ
+  // CHECK-DAG: bytecode.f8E4M3FNUZ = f8E4M3FNUZ
+  // CHECK-DAG: bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ
+  // CHECK-DAG: bytecode.f8E3M4 = f8E3M4
+  // CHECK-DAG: bytecode.f4E2M1FN = f4E2M1FN
+  // CHECK-DAG: bytecode.f6E2M3FN = f6E2M3FN
+  // CHECK-DAG: bytecode.f6E3M2FN = f6E3M2FN
+  // CHECK-DAG: bytecode.f8E8M0FNU = f8E8M0FNU
   bytecode.bf16 = bf16,
   bytecode.f16 = f16,
   bytecode.f32 = f32,
   bytecode.f64 = f64,
   bytecode.f80 = f80,
-  bytecode.f128 = f128
+  bytecode.f128 = f128,
+  bytecode.tf32 = tf32,
+  bytecode.f8E5M2 = f8E5M2,
+  bytecode.f8E4M3 = f8E4M3,
+  bytecode.f8E4M3FN = f8E4M3FN,
+  bytecode.f8E5M2FNUZ = f8E5M2FNUZ,
+  bytecode.f8E4M3FNUZ = f8E4M3FNUZ,
+  bytecode.f8E4M3B11FNUZ = f8E4M3B11FNUZ,
+  bytecode.f8E3M4 = f8E3M4,
+  bytecode.f4E2M1FN = f4E2M1FN,
+  bytecode.f6E2M3FN = f6E2M3FN,
+  bytecode.f6E3M2FN = f6E3M2FN,
+  bytecode.f8E8M0FNU = f8E8M0FNU
 } {} loc(unknown)
 
 //===----------------------------------------------------------------------===//
@@ -396,6 +444,42 @@ module @TestTupleType attributes {
   bytecode.mixed_tuple = tuple<i32, f64, index>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// AffineMapAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestAffineMapAttr
+module @TestAffineMapAttr attributes {
+  // CHECK-DAG: bytecode.identity = affine_map<(d0) -> (d0)>
+  // CHECK-DAG: bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>
+  // CHECK-DAG: bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>
+  // CHECK-DAG: bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>
+  // CHECK-DAG: bytecode.empty = affine_map<() -> ()>
+  // CHECK-DAG: bytecode.constant = affine_map<() -> (42)>
+  bytecode.identity = affine_map<(d0) -> (d0)>,
+  bytecode.multi_dim = affine_map<(d0, d1) -> (d0 + d1)>,
+  bytecode.with_symbol = affine_map<(d0)[s0] -> (d0 + s0)>,
+  bytecode.complex = affine_map<(d0, d1) -> (d0 floordiv 4, d1 mod 8)>,
+  bytecode.empty = affine_map<() -> ()>,
+  bytecode.constant = affine_map<() -> (42)>
+} {} loc(unknown)
+
+//===----------------------------------------------------------------------===//
+// IntegerSetAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerSetAttr
+module @TestIntegerSetAttr attributes {
+  // CHECK-DAG: bytecode.eq = affine_set<(d0) : (d0 == 0)>
+  // CHECK-DAG: bytecode.ineq = affine_set<(d0) : (d0 >= 0)>
+  // CHECK-DAG: bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>
+  // CHECK-DAG: bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+  bytecode.eq = affine_set<(d0) : (d0 == 0)>,
+  bytecode.ineq = affine_set<(d0) : (d0 >= 0)>,
+  bytecode.multi = affine_set<(d0, d1)[s0] : (d0 >= 0, -d0 + s0 >= 0, d1 >= 0)>,
+  bytecode.eq_ineq = affine_set<(d0, d1) : (d0 == 0, d1 >= 0)>
+} {} loc(unknown)
+
 //===----------------------------------------------------------------------===//
 // DenseResourceElementsAttr
 //===----------------------------------------------------------------------===//
@@ -416,6 +500,20 @@ module @TestDenseStringElementsAttr attributes {
   bytecode.dense_str = dense<["foo", "bar", "baz"]> : tensor<3x!bytecode.string>
 } {} loc(unknown)
 
+//===----------------------------------------------------------------------===//
+// StridedLayoutAttr
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestStridedLayoutAttr
+module @TestStridedLayoutAttr attributes {
+  // CHECK-DAG: bytecode.static_strides = strided<[3, 1]>
+  // CHECK-DAG: bytecode.with_offset = strided<[3, 1], offset: 5>
+  // CHECK-DAG: bytecode.dynamic = strided<[?, 1], offset: ?>
+  bytecode.static_strides = strided<[3, 1]>,
+  bytecode.with_offset = strided<[3, 1], offset: 5>,
+  bytecode.dynamic = strided<[?, 1], offset: ?>
+} {} loc(unknown)
+
 } loc(unknown)
 
 {-#

diff  --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 51d48d35b2d23..95be4de8fd1ab 100644
Binary files a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc and b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc 
diff er

diff  --git a/mlir/test/Dialect/Builtin/Bytecode/types.mlir b/mlir/test/Dialect/Builtin/Bytecode/types.mlir
index 91d4512998b99..e1d4a18a33c73 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/types.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/types.mlir
@@ -85,9 +85,11 @@ module @TestIndex attributes {
 // CHECK-LABEL: @TestFunc
 module @TestFunc attributes {
   // CHECK: bytecode.func = () -> (),
-  // CHECK: bytecode.func1 = (i1) -> i32
+  // CHECK: bytecode.func1 = (i1) -> i32,
+  // CHECK: bytecode.func2 = (i32) -> (f32, f64)
   bytecode.func = () -> (),
-  bytecode.func1 = (i1) -> (i32)
+  bytecode.func1 = (i1) -> (i32),
+  bytecode.func2 = (i32) -> (f32, f64)
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -97,9 +99,13 @@ module @TestFunc attributes {
 // CHECK-LABEL: @TestMemRef
 module @TestMemRef attributes {
   // CHECK: bytecode.test = memref<2xi8>,
-  // CHECK: bytecode.test1 = memref<2xi8, 1>
+  // CHECK: bytecode.test1 = memref<2xi8, 1>,
+  // CHECK: bytecode.test2 = memref<?x?xf32>,
+  // CHECK: bytecode.test3 = memref<2x3xf32, strided<[3, 1]>>
   bytecode.test = memref<2xi8>,
-  bytecode.test1 = memref<2xi8, 1>
+  bytecode.test1 = memref<2xi8, 1>,
+  bytecode.test2 = memref<?x?xf32>,
+  bytecode.test3 = memref<2x3xf32, strided<[3, 1], offset: 0>>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -165,9 +171,11 @@ module @TestUnrankedTensor attributes {
 // CHECK-LABEL: @TestVector
 module @TestVector attributes {
   // CHECK: bytecode.test = vector<8x8x128xi8>,
-  // CHECK: bytecode.test1 = vector<8x[8]xf32>
+  // CHECK: bytecode.test1 = vector<8x[8]xf32>,
+  // CHECK: bytecode.test2 = vector<[4]x[4]xf16>
   bytecode.test = vector<8x8x128xi8>,
-  bytecode.test1 = vector<8x[8]xf32>
+  bytecode.test1 = vector<8x[8]xf32>,
+  bytecode.test2 = vector<[4]x[4]xf16>
 } {}
 
 //===----------------------------------------------------------------------===//
@@ -179,3 +187,21 @@ module @TestToken attributes {
   // CHECK: bytecode.test = token
   bytecode.test = token
 } {}
+
+//===----------------------------------------------------------------------===//
+// IntegerType (packing variants)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @TestIntegerTypePacking
+module @TestIntegerTypePacking attributes {
+  // CHECK-DAG: bytecode.i1 = i1
+  // CHECK-DAG: bytecode.i2 = i2
+  // CHECK-DAG: bytecode.si1 = si1
+  // CHECK-DAG: bytecode.ui1 = ui1
+  // CHECK-DAG: bytecode.i4096 = i4096
+  bytecode.i1 = i1,
+  bytecode.i2 = i2,
+  bytecode.si1 = si1,
+  bytecode.ui1 = ui1,
+  bytecode.i4096 = i4096
+} {}


        

From llvmlistbot at llvm.org  Fri Jul  3 23:27:25 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Fri, 03 Jul 2026 23:27:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlirbc] Address remaining missing encodings
 for builtin dialect. (PR #207488)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207488@github.com>
Message-ID: <6a48a7cd.dd2b4bd3.13e95e.c0ee@mx.google.com>


https://github.com/jpienaar closed https://github.com/llvm/llvm-project/pull/207488

From llvmlistbot at llvm.org  Fri Jul  3 23:30:37 2026
From: llvmlistbot at llvm.org (Robert Imschweiler)
Date: Fri, 03 Jul 2026 23:30:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Change device declare target
 functions to hidden visibility (PR #207234)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207234@github.com>
Message-ID: <6a48a88d.a87eb5dc.2cf3a1.5f6a@mx.google.com>


ro-i wrote:

```
CMake Error: Error required internal CMake variable not set, cmake may not be built correctly.
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
```

That seems unrelated to this PR...

https://github.com/llvm/llvm-project/pull/207234

From llvmlistbot at llvm.org  Sat Jul  4 03:42:44 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Sat, 04 Jul 2026 03:42:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a48e3a4.da8bafd3.1eab2b.3dc6@mx.google.com>


https://github.com/IgWod edited https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Sat Jul  4 03:42:44 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Sat, 04 Jul 2026 03:42:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a48e3a4.c56fb102.11841f.6f34@mx.google.com>


https://github.com/IgWod commented:

Unless I'm missing something the fp semantics we get before and after the conversion may be different due to ordering, right?. Not sure if that's a problem or not; and I'm not sure how the conversion has been dealing with this so far. I just wonder whether we should keep the mix ordering and then let the LLVM IR side to fold it into fma if desired.

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Sat Jul  4 03:42:44 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Sat, 04 Jul 2026 03:42:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a48e3a4.bb839515.23e6d3.04cf@mx.google.com>


================
@@ -1584,6 +1584,31 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
----------------
IgWod wrote:

```suggestion
    Type dstType = this->getTypeConverter()->convertType(op.getType());
```

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Sat Jul  4 05:28:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Sat, 04 Jul 2026 05:28:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append (PR #207510)
Message-ID: <llvm.org/llvm/llvm-project/pull/207510@github.com>


https://github.com/zero9178 created https://github.com/llvm/llvm-project/pull/207510

The append-logic of the compile command is already performed by the `tablegen` function, there is no need to replicate the logic twice.

>From 49585d4ba278b601fe36b748c51dc7ff5bb8b787 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Markus=20B=C3=B6ck?= <markus.boeck02 at gmail.com>
Date: Sat, 4 Jul 2026 14:25:16 +0200
Subject: [PATCH] [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append

The append-logic of the compile command is already performed by the `tablegen` function, there is no need to replicate the logic twice.
---
 mlir/cmake/modules/AddMLIR.cmake | 22 ----------------------
 1 file changed, 22 deletions(-)

diff --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index 60b73876d53fe..d3f534a02f9de 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -6,28 +6,6 @@ function(mlir_tablegen ofn)
   tablegen(MLIR ${ARGV})
   set(TABLEGEN_OUTPUT ${TABLEGEN_OUTPUT} ${CMAKE_CURRENT_BINARY_DIR}/${ofn}
       PARENT_SCOPE)
-
-  # Get the current set of include paths for this td file.
-  cmake_parse_arguments(ARG "" "" "DEPENDS;EXTRA_INCLUDES" ${ARGN})
-  get_directory_property(tblgen_includes INCLUDE_DIRECTORIES)
-  list(APPEND tblgen_includes ${ARG_EXTRA_INCLUDES})
-  # Filter out any empty include items.
-  list(REMOVE_ITEM tblgen_includes "")
-
-  # Build the absolute path for the current input file.
-  if (IS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-  else()
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${CMAKE_CURRENT_SOURCE_DIR}/${LLVM_TARGET_DEFINITIONS})
-  endif()
-
-  # Append the includes used for this file to the tablegen_compile_commands
-  # file.
-  file(APPEND ${CMAKE_BINARY_DIR}/tablegen_compile_commands.yml
-      "--- !FileInfo:\n"
-      "  filepath: \"${LLVM_TARGET_DEFINITIONS_ABSOLUTE}\"\n"
-      "  includes: \"${CMAKE_CURRENT_SOURCE_DIR};${tblgen_includes}\"\n"
-  )
 endfunction()
 
 # Clear out any pre-existing compile_commands file before processing. This


From llvmlistbot at llvm.org  Sat Jul  4 05:29:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Sat, 04 Jul 2026 05:29:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append (PR #207510)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207510@github.com>
Message-ID: <6a48fca4.70920dee.c0a01.baeb@mx.google.com>


https://github.com/zero9178 auto_merge_enabled https://github.com/llvm/llvm-project/pull/207510

From llvmlistbot at llvm.org  Sat Jul  4 05:29:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 05:29:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append (PR #207510)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207510@github.com>
Message-ID: <6a48fcac.92ea0e2a.21f97.ab06@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Markus Böck (zero9178)

<details>
<summary>Changes</summary>

The append-logic of the compile command is already performed by the `tablegen` function, there is no need to replicate the logic twice.

---
Full diff: https://github.com/llvm/llvm-project/pull/207510.diff


1 Files Affected:

- (modified) mlir/cmake/modules/AddMLIR.cmake (-22) 


``````````diff
diff --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index 60b73876d53fe..d3f534a02f9de 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -6,28 +6,6 @@ function(mlir_tablegen ofn)
   tablegen(MLIR ${ARGV})
   set(TABLEGEN_OUTPUT ${TABLEGEN_OUTPUT} ${CMAKE_CURRENT_BINARY_DIR}/${ofn}
       PARENT_SCOPE)
-
-  # Get the current set of include paths for this td file.
-  cmake_parse_arguments(ARG "" "" "DEPENDS;EXTRA_INCLUDES" ${ARGN})
-  get_directory_property(tblgen_includes INCLUDE_DIRECTORIES)
-  list(APPEND tblgen_includes ${ARG_EXTRA_INCLUDES})
-  # Filter out any empty include items.
-  list(REMOVE_ITEM tblgen_includes "")
-
-  # Build the absolute path for the current input file.
-  if (IS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-  else()
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${CMAKE_CURRENT_SOURCE_DIR}/${LLVM_TARGET_DEFINITIONS})
-  endif()
-
-  # Append the includes used for this file to the tablegen_compile_commands
-  # file.
-  file(APPEND ${CMAKE_BINARY_DIR}/tablegen_compile_commands.yml
-      "--- !FileInfo:\n"
-      "  filepath: \"${LLVM_TARGET_DEFINITIONS_ABSOLUTE}\"\n"
-      "  includes: \"${CMAKE_CURRENT_SOURCE_DIR};${tblgen_includes}\"\n"
-  )
 endfunction()
 
 # Clear out any pre-existing compile_commands file before processing. This

``````````

</details>


https://github.com/llvm/llvm-project/pull/207510

From llvmlistbot at llvm.org  Sat Jul  4 05:45:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 05:45:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] bf09114 - [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append (#207510)
Message-ID: <6a49005b.70920dee.c0a01.d253@mx.google.com>


Author: Markus Böck
Date: 2026-07-04T12:45:11Z
New Revision: bf09114d6164f522508ae80921a63c5a080c183d

URL: https://github.com/llvm/llvm-project/commit/bf09114d6164f522508ae80921a63c5a080c183d
DIFF: https://github.com/llvm/llvm-project/commit/bf09114d6164f522508ae80921a63c5a080c183d.diff

LOG: [mlir][cmake][NFC] Remove duplicate `tablegen_compile_commands.yml` append (#207510)

The append-logic of the compile command is already performed by the
`tablegen` function, there is no need to replicate the logic twice.

Added: 
    

Modified: 
    mlir/cmake/modules/AddMLIR.cmake

Removed: 
    


################################################################################
diff  --git a/mlir/cmake/modules/AddMLIR.cmake b/mlir/cmake/modules/AddMLIR.cmake
index 60b73876d53fe..d3f534a02f9de 100644
--- a/mlir/cmake/modules/AddMLIR.cmake
+++ b/mlir/cmake/modules/AddMLIR.cmake
@@ -6,28 +6,6 @@ function(mlir_tablegen ofn)
   tablegen(MLIR ${ARGV})
   set(TABLEGEN_OUTPUT ${TABLEGEN_OUTPUT} ${CMAKE_CURRENT_BINARY_DIR}/${ofn}
       PARENT_SCOPE)
-
-  # Get the current set of include paths for this td file.
-  cmake_parse_arguments(ARG "" "" "DEPENDS;EXTRA_INCLUDES" ${ARGN})
-  get_directory_property(tblgen_includes INCLUDE_DIRECTORIES)
-  list(APPEND tblgen_includes ${ARG_EXTRA_INCLUDES})
-  # Filter out any empty include items.
-  list(REMOVE_ITEM tblgen_includes "")
-
-  # Build the absolute path for the current input file.
-  if (IS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${LLVM_TARGET_DEFINITIONS})
-  else()
-    set(LLVM_TARGET_DEFINITIONS_ABSOLUTE ${CMAKE_CURRENT_SOURCE_DIR}/${LLVM_TARGET_DEFINITIONS})
-  endif()
-
-  # Append the includes used for this file to the tablegen_compile_commands
-  # file.
-  file(APPEND ${CMAKE_BINARY_DIR}/tablegen_compile_commands.yml
-      "--- !FileInfo:\n"
-      "  filepath: \"${LLVM_TARGET_DEFINITIONS_ABSOLUTE}\"\n"
-      "  includes: \"${CMAKE_CURRENT_SOURCE_DIR};${tblgen_includes}\"\n"
-  )
 endfunction()
 
 # Clear out any pre-existing compile_commands file before processing. This


        

From llvmlistbot at llvm.org  Sat Jul  4 05:45:18 2026
From: llvmlistbot at llvm.org (=?UTF-8?B?TWFya3VzIELDtmNr?=)
Date: Sat, 04 Jul 2026 05:45:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake][NFC] Remove duplicate
 `tablegen_compile_commands.yml` append (PR #207510)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207510@github.com>
Message-ID: <6a49005e.f8b79514.3b441.8502@mx.google.com>


https://github.com/zero9178 closed https://github.com/llvm/llvm-project/pull/207510

From llvmlistbot at llvm.org  Sat Jul  4 06:37:42 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca6.cbc3f03a.3be201.8734@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
----------------
matthias-springer wrote:

Can you add a small comment here? How come we don't need `timeBegin` here?

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 06:37:42 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca6.f1c1923d.3672e2.b051@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
----------------
matthias-springer wrote:

nit: Add documentation. We are looking for the smallest gap that's still large enough to fit the allocation.

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 06:37:43 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca7.70920dee.c0a01.e2ce@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
----------------
matthias-springer wrote:

What is this check doing? Why not always place the alloc at `alignedTrailing`?

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 06:37:43 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca7.eb24c6ea.1f2e26.6808@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
----------------
matthias-springer wrote:

nit: Add comment here. This branch is taken if gap is large enough.

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 06:37:43 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca7.ac469dc7.e3aa6.c748@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
----------------
matthias-springer wrote:

Can this be defined outside of the `if` check and then used in the `if` condition?

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 06:37:44 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 04 Jul 2026 06:37:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a490ca8.331483e7.5bc1e.16db@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
----------------
matthias-springer wrote:

Can this variable be gradually updated as the outer `for (unsigned idx : order)` loop is running? (Instead of recomputing it every time.)


https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sat Jul  4 07:29:26 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Sat, 04 Jul 2026 07:29:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add normalize pass to MLIR (PR #186647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/186647@github.com>
Message-ID: <6a4918c6.92ea0e2a.21f97.1a80@mx.google.com>


https://github.com/linuxlonelyeagle converted_to_draft https://github.com/llvm/llvm-project/pull/186647

From llvmlistbot at llvm.org  Sat Jul  4 08:22:41 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 08:22:41 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
Message-ID: <llvm.org/llvm/llvm-project/pull/207516@github.com>


https://github.com/aengelke created https://github.com/llvm/llvm-project/pull/207516

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef<> would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.


>From ef7186efff12c9c51bcf1a14855940e5497e6d8b Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 15:22:04 +0000
Subject: [PATCH] [spr] initial version

Created using spr 1.3.8-wip
---
 .../DataFormatters/FormatterBytecode.cpp      |  16 +-
 .../llvm/DebugInfo/CodeView/Formatters.h      |   2 +-
 .../llvm/DebugInfo/PDB/Native/FormatUtil.h    |   2 +-
 llvm/include/llvm/Support/FormatAdapters.h    |  25 ++-
 llvm/include/llvm/Support/FormatCommon.h      |   8 +-
 llvm/include/llvm/Support/FormatProviders.h   |  22 +--
 llvm/include/llvm/Support/FormatVariadic.h    |  20 +--
 .../llvm/Support/FormatVariadicDetails.h      | 158 +++++-------------
 llvm/lib/Support/FormatVariadic.cpp           |   2 -
 llvm/tools/llvm-xray/xray-stacks.cpp          |   2 +-
 llvm/unittests/ADT/TwineTest.cpp              |   2 +-
 llvm/unittests/Support/FormatVariadicTest.cpp |  32 ++--
 mlir/include/mlir/TableGen/Format.h           |  24 ++-
 mlir/lib/TableGen/Format.cpp                  |  12 +-
 mlir/tools/mlir-tblgen/RewriterGen.cpp        |   8 +-
 15 files changed, 120 insertions(+), 215 deletions(-)

diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\", RecordType: \"{2}\"}",
         Converter->SymbolOrNumber(Item.FuncId), Item.TId,
diff --git a/llvm/unittests/ADT/TwineTest.cpp b/llvm/unittests/ADT/TwineTest.cpp
index d45805f4db7a3..fc1cc3a0e0db0 100644
--- a/llvm/unittests/ADT/TwineTest.cpp
+++ b/llvm/unittests/ADT/TwineTest.cpp
@@ -119,7 +119,7 @@ TEST(TwineTest, LazyEvaluation) {
     explicit formatter(int &Count) : FormatAdapter(0), Count(Count) {}
     int &Count;
 
-    void format(raw_ostream &OS, StringRef Style) override { ++Count; }
+    void format(raw_ostream &OS, StringRef Style) { ++Count; }
   };
 
   int Count = 0;
diff --git a/llvm/unittests/Support/FormatVariadicTest.cpp b/llvm/unittests/Support/FormatVariadicTest.cpp
index ea549e7553b0c..afe7c6f6ee6f7 100644
--- a/llvm/unittests/Support/FormatVariadicTest.cpp
+++ b/llvm/unittests/Support/FormatVariadicTest.cpp
@@ -18,22 +18,28 @@ using namespace llvm;
 namespace {
 struct Format : public FormatAdapter<int> {
   Format(int N) : FormatAdapter<int>(std::move(N)) {}
-  void format(raw_ostream &OS, StringRef Opt) override { OS << "Format"; }
+  void format(raw_ostream &OS, StringRef Opt) { OS << "Format"; }
 };
 
-using support::detail::uses_format_member;
-using support::detail::uses_missing_provider;
+using support::detail::FormatFunctor;
 
-static_assert(uses_format_member<Format>::value, "");
-static_assert(uses_format_member<Format &>::value, "");
-static_assert(uses_format_member<Format &&>::value, "");
-static_assert(uses_format_member<const Format>::value, "");
-static_assert(uses_format_member<const Format &>::value, "");
-static_assert(uses_format_member<const volatile Format>::value, "");
-static_assert(uses_format_member<const volatile Format &>::value, "");
+static_assert(FormatFunctor<Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &&>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format &>::HasMemberProvider, "");
 
-struct NoFormat {};
-static_assert(uses_missing_provider<NoFormat>::value, "");
+struct StreamFormat {};
+// Silence warning about never-called function.
+[[maybe_unused]] raw_ostream &operator<<(raw_ostream &, const StreamFormat &);
+static_assert(!FormatFunctor<StreamFormat>::HasMemberProvider, "");
+static_assert(!FormatFunctor<StreamFormat>::HasFormatProvider, "");
+static_assert(FormatFunctor<StreamFormat>::HasStreamProvider, "");
+static_assert(!FormatFunctor<int>::HasMemberProvider, "");
+static_assert(FormatFunctor<int>::HasFormatProvider, "");
+static_assert(FormatFunctor<int>::HasStreamProvider, "");
 }
 
 // Helper to parse format string with no validation.
@@ -780,7 +786,7 @@ TEST(FormatVariadicTest, Adapter) {
   class Negative : public FormatAdapter<int> {
   public:
     explicit Negative(int N) : FormatAdapter<int>(std::move(N)) {}
-    void format(raw_ostream &S, StringRef Options) override { S << -Item; }
+    void format(raw_ostream &S, StringRef Options) { S << -Item; }
   };
 
   EXPECT_EQ("-7", formatv("{0}", Negative(7)).str());
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index e9f151e248340..a3c3df487ab83 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -127,15 +127,15 @@ class FmtObjectBase {
   // std::vector<Base*>.
   struct CreateAdapters {
     template <typename... Ts>
-    std::vector<llvm::support::detail::format_adapter *>
+    std::vector<llvm::support::detail::FormatFunctorRef>
     operator()(Ts &...items) {
-      return std::vector<llvm::support::detail::format_adapter *>{&items...};
+      return std::vector<llvm::support::detail::FormatFunctorRef>{items...};
     }
   };
 
   StringRef fmt;
   const FmtContext *context;
-  std::vector<llvm::support::detail::format_adapter *> adapters;
+  std::vector<llvm::support::detail::FormatFunctorRef> adapters;
   std::vector<FmtReplacement> replacements;
 
 public:
@@ -200,8 +200,7 @@ class FmtObject : public FmtObjectBase {
 
 class FmtStrVecObject : public FmtObjectBase {
 public:
-  using StrFormatAdapter = decltype(llvm::support::detail::build_format_adapter(
-      std::declval<std::string>()));
+  using StrFormatAdapter = llvm::support::detail::FormatFunctor<std::string>;
 
   FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
                   ArrayRef<std::string> params);
@@ -254,19 +253,18 @@ class FmtStrVecObject : public FmtObjectBase {
 ///    in C++ code generation.
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
-    -> FmtObject<
-        decltype(std::make_tuple(llvm::support::detail::build_format_adapter(
-            std::forward<Ts>(vals))...))> {
+    -> FmtObject<decltype(std::make_tuple(
+        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::build_format_adapter(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
-      std::make_tuple(llvm::support::detail::build_format_adapter(
-          std::forward<Ts>(vals))...));
+      std::make_tuple(
+          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
 }
 
-inline FmtStrVecObject tgfmt(StringRef fmt, const FmtContext *ctx,
-                             ArrayRef<std::string> params) {
+inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
+                              ArrayRef<std::string> params) {
   return FmtStrVecObject(fmt, ctx, params);
 }
 
diff --git a/mlir/lib/TableGen/Format.cpp b/mlir/lib/TableGen/Format.cpp
index 65f4ad56dd158..2153de2af4797 100644
--- a/mlir/lib/TableGen/Format.cpp
+++ b/mlir/lib/TableGen/Format.cpp
@@ -183,8 +183,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       range = range.drop_front(repl.index);
       if (repl.end != FmtReplacement::kUnset)
         range = range.drop_back(adapters.size() - repl.end);
-      llvm::interleaveComma(range, s,
-                            [&](auto &x) { x->format(s, /*Options=*/""); });
+      llvm::interleaveComma(range, s, [&](auto &x) { x(s, /*Options=*/""); });
       continue;
     }
 
@@ -194,7 +193,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       s << repl.spec << kMarkerForNoSubst;
       continue;
     }
-    adapters[repl.index]->format(s, /*Options=*/"");
+    adapters[repl.index](s, /*Options=*/"");
   }
 }
 
@@ -203,17 +202,16 @@ FmtStrVecObject::FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
     : FmtObjectBase(fmt, ctx, params.size()) {
   parameters.reserve(params.size());
   for (std::string p : params)
-    parameters.push_back(
-        llvm::support::detail::build_format_adapter(std::move(p)));
+    parameters.emplace_back(std::move(p));
 
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
 
 FmtStrVecObject::FmtStrVecObject(FmtStrVecObject &&that)
     : FmtObjectBase(std::move(that)), parameters(std::move(that.parameters)) {
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
diff --git a/mlir/tools/mlir-tblgen/RewriterGen.cpp b/mlir/tools/mlir-tblgen/RewriterGen.cpp
index e3043708a46d1..5bc2ac48b4726 100644
--- a/mlir/tools/mlir-tblgen/RewriterGen.cpp
+++ b/mlir/tools/mlir-tblgen/RewriterGen.cpp
@@ -524,8 +524,8 @@ void PatternEmitter::emitNativeCodeMatch(DagNode tree, StringRef opName,
                          "passing the defining Operation");
 
   auto nativeCodeCall = std::string(
-      tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
-            static_cast<ArrayRef<std::string>>(capture)));
+      tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
+             static_cast<ArrayRef<std::string>>(capture)));
 
   emitMatchCheck(opName, formatv("!::mlir::failed({0})", nativeCodeCall),
                  formatv("\"{0} return ::mlir::failure\"", nativeCodeCall));
@@ -1490,8 +1490,8 @@ std::string PatternEmitter::handleReplaceWithNativeCodeCall(DagNode tree,
                             << " replacement: " << attrs[i] << "\n");
   }
 
-  std::string symbol = tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse),
-                             static_cast<ArrayRef<std::string>>(attrs));
+  std::string symbol = tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse),
+                              static_cast<ArrayRef<std::string>>(attrs));
 
   // In general, NativeCodeCall without naming binding don't need this. To
   // ensure void helper function has been correctly labeled, i.e., use


From llvmlistbot at llvm.org  Sat Jul  4 08:23:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 08:23:11 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a49255f.f8b79514.3b441.d3f2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-core

@llvm/pr-subscribers-llvm-support

Author: Alexis Engelke (aengelke)

<details>
<summary>Changes</summary>

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef&lt;&gt; would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.


---

Patch is 28.75 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207516.diff


15 Files Affected:

- (modified) lldb/source/DataFormatters/FormatterBytecode.cpp (+8-8) 
- (modified) llvm/include/llvm/DebugInfo/CodeView/Formatters.h (+1-1) 
- (modified) llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h (+1-1) 
- (modified) llvm/include/llvm/Support/FormatAdapters.h (+11-14) 
- (modified) llvm/include/llvm/Support/FormatCommon.h (+4-4) 
- (modified) llvm/include/llvm/Support/FormatProviders.h (+4-18) 
- (modified) llvm/include/llvm/Support/FormatVariadic.h (+9-11) 
- (modified) llvm/include/llvm/Support/FormatVariadicDetails.h (+41-117) 
- (modified) llvm/lib/Support/FormatVariadic.cpp (-2) 
- (modified) llvm/tools/llvm-xray/xray-stacks.cpp (+1-1) 
- (modified) llvm/unittests/ADT/TwineTest.cpp (+1-1) 
- (modified) llvm/unittests/Support/FormatVariadicTest.cpp (+19-13) 
- (modified) mlir/include/mlir/TableGen/Format.h (+11-13) 
- (modified) mlir/lib/TableGen/Format.cpp (+5-7) 
- (modified) mlir/tools/mlir-tblgen/RewriterGen.cpp (+4-4) 


``````````diff
diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\"...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:23:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 08:23:12 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492560.7270d2fd.23dbb3.5f13@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-llvm-adt

Author: Alexis Engelke (aengelke)

<details>
<summary>Changes</summary>

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef&lt;&gt; would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.


---

Patch is 28.75 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207516.diff


15 Files Affected:

- (modified) lldb/source/DataFormatters/FormatterBytecode.cpp (+8-8) 
- (modified) llvm/include/llvm/DebugInfo/CodeView/Formatters.h (+1-1) 
- (modified) llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h (+1-1) 
- (modified) llvm/include/llvm/Support/FormatAdapters.h (+11-14) 
- (modified) llvm/include/llvm/Support/FormatCommon.h (+4-4) 
- (modified) llvm/include/llvm/Support/FormatProviders.h (+4-18) 
- (modified) llvm/include/llvm/Support/FormatVariadic.h (+9-11) 
- (modified) llvm/include/llvm/Support/FormatVariadicDetails.h (+41-117) 
- (modified) llvm/lib/Support/FormatVariadic.cpp (-2) 
- (modified) llvm/tools/llvm-xray/xray-stacks.cpp (+1-1) 
- (modified) llvm/unittests/ADT/TwineTest.cpp (+1-1) 
- (modified) llvm/unittests/Support/FormatVariadicTest.cpp (+19-13) 
- (modified) mlir/include/mlir/TableGen/Format.h (+11-13) 
- (modified) mlir/lib/TableGen/Format.cpp (+5-7) 
- (modified) mlir/tools/mlir-tblgen/RewriterGen.cpp (+4-4) 


``````````diff
diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\"...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:23:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 08:23:12 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492560.331483e7.5bc1e.41b9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-lldb

Author: Alexis Engelke (aengelke)

<details>
<summary>Changes</summary>

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef&lt;&gt; would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.


---

Patch is 28.75 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207516.diff


15 Files Affected:

- (modified) lldb/source/DataFormatters/FormatterBytecode.cpp (+8-8) 
- (modified) llvm/include/llvm/DebugInfo/CodeView/Formatters.h (+1-1) 
- (modified) llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h (+1-1) 
- (modified) llvm/include/llvm/Support/FormatAdapters.h (+11-14) 
- (modified) llvm/include/llvm/Support/FormatCommon.h (+4-4) 
- (modified) llvm/include/llvm/Support/FormatProviders.h (+4-18) 
- (modified) llvm/include/llvm/Support/FormatVariadic.h (+9-11) 
- (modified) llvm/include/llvm/Support/FormatVariadicDetails.h (+41-117) 
- (modified) llvm/lib/Support/FormatVariadic.cpp (-2) 
- (modified) llvm/tools/llvm-xray/xray-stacks.cpp (+1-1) 
- (modified) llvm/unittests/ADT/TwineTest.cpp (+1-1) 
- (modified) llvm/unittests/Support/FormatVariadicTest.cpp (+19-13) 
- (modified) mlir/include/mlir/TableGen/Format.h (+11-13) 
- (modified) mlir/lib/TableGen/Format.cpp (+5-7) 
- (modified) mlir/tools/mlir-tblgen/RewriterGen.cpp (+4-4) 


``````````diff
diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\"...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:23:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 08:23:12 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492560.2020f6c2.1fdf61.3262@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Alexis Engelke (aengelke)

<details>
<summary>Changes</summary>

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef&lt;&gt; would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.


---

Patch is 28.75 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207516.diff


15 Files Affected:

- (modified) lldb/source/DataFormatters/FormatterBytecode.cpp (+8-8) 
- (modified) llvm/include/llvm/DebugInfo/CodeView/Formatters.h (+1-1) 
- (modified) llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h (+1-1) 
- (modified) llvm/include/llvm/Support/FormatAdapters.h (+11-14) 
- (modified) llvm/include/llvm/Support/FormatCommon.h (+4-4) 
- (modified) llvm/include/llvm/Support/FormatProviders.h (+4-18) 
- (modified) llvm/include/llvm/Support/FormatVariadic.h (+9-11) 
- (modified) llvm/include/llvm/Support/FormatVariadicDetails.h (+41-117) 
- (modified) llvm/lib/Support/FormatVariadic.cpp (-2) 
- (modified) llvm/tools/llvm-xray/xray-stacks.cpp (+1-1) 
- (modified) llvm/unittests/ADT/TwineTest.cpp (+1-1) 
- (modified) llvm/unittests/Support/FormatVariadicTest.cpp (+19-13) 
- (modified) mlir/include/mlir/TableGen/Format.h (+11-13) 
- (modified) mlir/lib/TableGen/Format.cpp (+5-7) 
- (modified) mlir/tools/mlir-tblgen/RewriterGen.cpp (+4-4) 


``````````diff
diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\"...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:30:38 2026
From: llvmlistbot at llvm.org (Sergei Barannikov)
Date: Sat, 04 Jul 2026 08:30:38 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a49271e.faf16b37.1e0c59.d464@mx.google.com>


================
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
----------------
s-barannikov wrote:

Is it possible to get build_format_adapter back so that we don't have to specify template arguments everywhere?

https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:32:01 2026
From: llvmlistbot at llvm.org (Sergei Barannikov)
Date: Sat, 04 Jul 2026 08:32:01 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492771.cbc3f03a.3be201.9a74@mx.google.com>


https://github.com/s-barannikov edited https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 08:48:42 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 08:48:42 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492b5a.eb165e5c.113c2a.0c5c@mx.google.com>


https://github.com/aengelke updated https://github.com/llvm/llvm-project/pull/207516

>From ef7186efff12c9c51bcf1a14855940e5497e6d8b Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 15:22:04 +0000
Subject: [PATCH 1/2] [spr] initial version

Created using spr 1.3.8-wip
---
 .../DataFormatters/FormatterBytecode.cpp      |  16 +-
 .../llvm/DebugInfo/CodeView/Formatters.h      |   2 +-
 .../llvm/DebugInfo/PDB/Native/FormatUtil.h    |   2 +-
 llvm/include/llvm/Support/FormatAdapters.h    |  25 ++-
 llvm/include/llvm/Support/FormatCommon.h      |   8 +-
 llvm/include/llvm/Support/FormatProviders.h   |  22 +--
 llvm/include/llvm/Support/FormatVariadic.h    |  20 +--
 .../llvm/Support/FormatVariadicDetails.h      | 158 +++++-------------
 llvm/lib/Support/FormatVariadic.cpp           |   2 -
 llvm/tools/llvm-xray/xray-stacks.cpp          |   2 +-
 llvm/unittests/ADT/TwineTest.cpp              |   2 +-
 llvm/unittests/Support/FormatVariadicTest.cpp |  32 ++--
 mlir/include/mlir/TableGen/Format.h           |  24 ++-
 mlir/lib/TableGen/Format.cpp                  |  12 +-
 mlir/tools/mlir-tblgen/RewriterGen.cpp        |   8 +-
 15 files changed, 120 insertions(+), 215 deletions(-)

diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\", RecordType: \"{2}\"}",
         Converter->SymbolOrNumber(Item.FuncId), Item.TId,
diff --git a/llvm/unittests/ADT/TwineTest.cpp b/llvm/unittests/ADT/TwineTest.cpp
index d45805f4db7a3..fc1cc3a0e0db0 100644
--- a/llvm/unittests/ADT/TwineTest.cpp
+++ b/llvm/unittests/ADT/TwineTest.cpp
@@ -119,7 +119,7 @@ TEST(TwineTest, LazyEvaluation) {
     explicit formatter(int &Count) : FormatAdapter(0), Count(Count) {}
     int &Count;
 
-    void format(raw_ostream &OS, StringRef Style) override { ++Count; }
+    void format(raw_ostream &OS, StringRef Style) { ++Count; }
   };
 
   int Count = 0;
diff --git a/llvm/unittests/Support/FormatVariadicTest.cpp b/llvm/unittests/Support/FormatVariadicTest.cpp
index ea549e7553b0c..afe7c6f6ee6f7 100644
--- a/llvm/unittests/Support/FormatVariadicTest.cpp
+++ b/llvm/unittests/Support/FormatVariadicTest.cpp
@@ -18,22 +18,28 @@ using namespace llvm;
 namespace {
 struct Format : public FormatAdapter<int> {
   Format(int N) : FormatAdapter<int>(std::move(N)) {}
-  void format(raw_ostream &OS, StringRef Opt) override { OS << "Format"; }
+  void format(raw_ostream &OS, StringRef Opt) { OS << "Format"; }
 };
 
-using support::detail::uses_format_member;
-using support::detail::uses_missing_provider;
+using support::detail::FormatFunctor;
 
-static_assert(uses_format_member<Format>::value, "");
-static_assert(uses_format_member<Format &>::value, "");
-static_assert(uses_format_member<Format &&>::value, "");
-static_assert(uses_format_member<const Format>::value, "");
-static_assert(uses_format_member<const Format &>::value, "");
-static_assert(uses_format_member<const volatile Format>::value, "");
-static_assert(uses_format_member<const volatile Format &>::value, "");
+static_assert(FormatFunctor<Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &&>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format &>::HasMemberProvider, "");
 
-struct NoFormat {};
-static_assert(uses_missing_provider<NoFormat>::value, "");
+struct StreamFormat {};
+// Silence warning about never-called function.
+[[maybe_unused]] raw_ostream &operator<<(raw_ostream &, const StreamFormat &);
+static_assert(!FormatFunctor<StreamFormat>::HasMemberProvider, "");
+static_assert(!FormatFunctor<StreamFormat>::HasFormatProvider, "");
+static_assert(FormatFunctor<StreamFormat>::HasStreamProvider, "");
+static_assert(!FormatFunctor<int>::HasMemberProvider, "");
+static_assert(FormatFunctor<int>::HasFormatProvider, "");
+static_assert(FormatFunctor<int>::HasStreamProvider, "");
 }
 
 // Helper to parse format string with no validation.
@@ -780,7 +786,7 @@ TEST(FormatVariadicTest, Adapter) {
   class Negative : public FormatAdapter<int> {
   public:
     explicit Negative(int N) : FormatAdapter<int>(std::move(N)) {}
-    void format(raw_ostream &S, StringRef Options) override { S << -Item; }
+    void format(raw_ostream &S, StringRef Options) { S << -Item; }
   };
 
   EXPECT_EQ("-7", formatv("{0}", Negative(7)).str());
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index e9f151e248340..a3c3df487ab83 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -127,15 +127,15 @@ class FmtObjectBase {
   // std::vector<Base*>.
   struct CreateAdapters {
     template <typename... Ts>
-    std::vector<llvm::support::detail::format_adapter *>
+    std::vector<llvm::support::detail::FormatFunctorRef>
     operator()(Ts &...items) {
-      return std::vector<llvm::support::detail::format_adapter *>{&items...};
+      return std::vector<llvm::support::detail::FormatFunctorRef>{items...};
     }
   };
 
   StringRef fmt;
   const FmtContext *context;
-  std::vector<llvm::support::detail::format_adapter *> adapters;
+  std::vector<llvm::support::detail::FormatFunctorRef> adapters;
   std::vector<FmtReplacement> replacements;
 
 public:
@@ -200,8 +200,7 @@ class FmtObject : public FmtObjectBase {
 
 class FmtStrVecObject : public FmtObjectBase {
 public:
-  using StrFormatAdapter = decltype(llvm::support::detail::build_format_adapter(
-      std::declval<std::string>()));
+  using StrFormatAdapter = llvm::support::detail::FormatFunctor<std::string>;
 
   FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
                   ArrayRef<std::string> params);
@@ -254,19 +253,18 @@ class FmtStrVecObject : public FmtObjectBase {
 ///    in C++ code generation.
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
-    -> FmtObject<
-        decltype(std::make_tuple(llvm::support::detail::build_format_adapter(
-            std::forward<Ts>(vals))...))> {
+    -> FmtObject<decltype(std::make_tuple(
+        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::build_format_adapter(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
-      std::make_tuple(llvm::support::detail::build_format_adapter(
-          std::forward<Ts>(vals))...));
+      std::make_tuple(
+          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
 }
 
-inline FmtStrVecObject tgfmt(StringRef fmt, const FmtContext *ctx,
-                             ArrayRef<std::string> params) {
+inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
+                              ArrayRef<std::string> params) {
   return FmtStrVecObject(fmt, ctx, params);
 }
 
diff --git a/mlir/lib/TableGen/Format.cpp b/mlir/lib/TableGen/Format.cpp
index 65f4ad56dd158..2153de2af4797 100644
--- a/mlir/lib/TableGen/Format.cpp
+++ b/mlir/lib/TableGen/Format.cpp
@@ -183,8 +183,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       range = range.drop_front(repl.index);
       if (repl.end != FmtReplacement::kUnset)
         range = range.drop_back(adapters.size() - repl.end);
-      llvm::interleaveComma(range, s,
-                            [&](auto &x) { x->format(s, /*Options=*/""); });
+      llvm::interleaveComma(range, s, [&](auto &x) { x(s, /*Options=*/""); });
       continue;
     }
 
@@ -194,7 +193,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       s << repl.spec << kMarkerForNoSubst;
       continue;
     }
-    adapters[repl.index]->format(s, /*Options=*/"");
+    adapters[repl.index](s, /*Options=*/"");
   }
 }
 
@@ -203,17 +202,16 @@ FmtStrVecObject::FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
     : FmtObjectBase(fmt, ctx, params.size()) {
   parameters.reserve(params.size());
   for (std::string p : params)
-    parameters.push_back(
-        llvm::support::detail::build_format_adapter(std::move(p)));
+    parameters.emplace_back(std::move(p));
 
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
 
 FmtStrVecObject::FmtStrVecObject(FmtStrVecObject &&that)
     : FmtObjectBase(std::move(that)), parameters(std::move(that.parameters)) {
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
diff --git a/mlir/tools/mlir-tblgen/RewriterGen.cpp b/mlir/tools/mlir-tblgen/RewriterGen.cpp
index e3043708a46d1..5bc2ac48b4726 100644
--- a/mlir/tools/mlir-tblgen/RewriterGen.cpp
+++ b/mlir/tools/mlir-tblgen/RewriterGen.cpp
@@ -524,8 +524,8 @@ void PatternEmitter::emitNativeCodeMatch(DagNode tree, StringRef opName,
                          "passing the defining Operation");
 
   auto nativeCodeCall = std::string(
-      tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
-            static_cast<ArrayRef<std::string>>(capture)));
+      tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
+             static_cast<ArrayRef<std::string>>(capture)));
 
   emitMatchCheck(opName, formatv("!::mlir::failed({0})", nativeCodeCall),
                  formatv("\"{0} return ::mlir::failure\"", nativeCodeCall));
@@ -1490,8 +1490,8 @@ std::string PatternEmitter::handleReplaceWithNativeCodeCall(DagNode tree,
                             << " replacement: " << attrs[i] << "\n");
   }
 
-  std::string symbol = tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse),
-                             static_cast<ArrayRef<std::string>>(attrs));
+  std::string symbol = tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse),
+                              static_cast<ArrayRef<std::string>>(attrs));
 
   // In general, NativeCodeCall without naming binding don't need this. To
   // ensure void helper function has been correctly labeled, i.e., use

>From 6da8c7d2e2302b4f2e46f3269c35bb5c7e604263 Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 15:48:05 +0000
Subject: [PATCH 2/2] add template deduction guide

Created using spr 1.3.8-wip
---
 lldb/source/DataFormatters/FormatterBytecode.cpp  | 14 +++++++-------
 llvm/include/llvm/Support/FormatAdapters.h        |  6 +++---
 llvm/include/llvm/Support/FormatProviders.h       |  4 ++--
 llvm/include/llvm/Support/FormatVariadic.h        |  2 +-
 llvm/include/llvm/Support/FormatVariadicDetails.h |  2 ++
 mlir/include/mlir/TableGen/Format.h               |  6 +++---
 6 files changed, 18 insertions(+), 16 deletions(-)

diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 786f9198a53d0..2d79641f516f1 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -113,20 +113,20 @@ static llvm::Error FormatImpl(DataStack &data) {
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(FormatFunctor<const char *>(s->c_str()));
+      format(FormatFunctor(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(FormatFunctor<uint64_t *&>(u));
+      format(FormatFunctor(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(FormatFunctor<int64_t *&>(i));
+      format(FormatFunctor(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(FormatFunctor<const char *>("null object"));
+        format(FormatFunctor("null object"));
       else
-        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
+      format(FormatFunctor(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(FormatFunctor<std::string>(toString(*sel)));
+      format(FormatFunctor(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 5a22c0bbde8f4..57a9fc03ea51f 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -36,7 +36,7 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
         Fill(Fill) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,7 +50,7 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     Stream.indent(Left);
     Adapter(Stream, Style);
     Stream.indent(Right);
@@ -65,7 +65,7 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
       Adapter(Stream, Style);
     }
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 38a203482d594..95176333e91e0 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -395,13 +395,13 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
       Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
       Adapter(Stream, ArgStyle);
       ++Begin;
     }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 322e6e9d8105a..4c92d45c2f5b9 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -246,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index 9fab9b84a3757..ba919b8896e56 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -74,6 +74,8 @@ template <typename T> class FormatFunctor {
   }
 };
 
+template <typename T> FormatFunctor(T &&) -> FormatFunctor<T>;
+
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index a3c3df487ab83..d0f8be28e493b 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -254,13 +254,13 @@ class FmtStrVecObject : public FmtObjectBase {
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
     -> FmtObject<decltype(std::make_tuple(
-        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
+        llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
       std::make_tuple(
-          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
+          llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
 }
 
 inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,


From llvmlistbot at llvm.org  Sat Jul  4 08:48:55 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 08:48:55 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a492b67.dd2b4bd3.13e95e.4f67@mx.google.com>


================
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
----------------
aengelke wrote:

Added a template deduction guide for this.

https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 09:56:32 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Sat, 04 Jul 2026 09:56:32 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a493b40.af6ba3ed.2c2f30.25b7@mx.google.com>


================
@@ -254,19 +253,18 @@ class FmtStrVecObject : public FmtObjectBase {
 ///    in C++ code generation.
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
-    -> FmtObject<
-        decltype(std::make_tuple(llvm::support::detail::build_format_adapter(
-            std::forward<Ts>(vals))...))> {
+    -> FmtObject<decltype(std::make_tuple(
+        llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::build_format_adapter(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
-      std::make_tuple(llvm::support::detail::build_format_adapter(
-          std::forward<Ts>(vals))...));
+      std::make_tuple(
+          llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
 }
 
-inline FmtStrVecObject tgfmt(StringRef fmt, const FmtContext *ctx,
-                             ArrayRef<std::string> params) {
+inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
----------------
MaskRay wrote:

Either FmtStrVecObject or `tgfmtv` should have a comment.

https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 09:56:43 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Sat, 04 Jul 2026 09:56:43 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a493b4b.01f0a1da.2016e9.821f@mx.google.com>


https://github.com/MaskRay approved this pull request.


https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 10:57:53 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 10:57:53 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a4949a1.f06f4994.168928.669b@mx.google.com>


https://github.com/aengelke updated https://github.com/llvm/llvm-project/pull/207516

>From ef7186efff12c9c51bcf1a14855940e5497e6d8b Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 15:22:04 +0000
Subject: [PATCH 1/3] [spr] initial version

Created using spr 1.3.8-wip
---
 .../DataFormatters/FormatterBytecode.cpp      |  16 +-
 .../llvm/DebugInfo/CodeView/Formatters.h      |   2 +-
 .../llvm/DebugInfo/PDB/Native/FormatUtil.h    |   2 +-
 llvm/include/llvm/Support/FormatAdapters.h    |  25 ++-
 llvm/include/llvm/Support/FormatCommon.h      |   8 +-
 llvm/include/llvm/Support/FormatProviders.h   |  22 +--
 llvm/include/llvm/Support/FormatVariadic.h    |  20 +--
 .../llvm/Support/FormatVariadicDetails.h      | 158 +++++-------------
 llvm/lib/Support/FormatVariadic.cpp           |   2 -
 llvm/tools/llvm-xray/xray-stacks.cpp          |   2 +-
 llvm/unittests/ADT/TwineTest.cpp              |   2 +-
 llvm/unittests/Support/FormatVariadicTest.cpp |  32 ++--
 mlir/include/mlir/TableGen/Format.h           |  24 ++-
 mlir/lib/TableGen/Format.cpp                  |  12 +-
 mlir/tools/mlir-tblgen/RewriterGen.cpp        |   8 +-
 15 files changed, 120 insertions(+), 215 deletions(-)

diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..786f9198a53d0 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor<const char *>(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor<uint64_t *&>(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor<int64_t *&>(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor<const char *>("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor<std::string>(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail
diff --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..5a22c0bbde8f4 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support
diff --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..38a203482d594 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..322e6e9d8105a 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..9fab9b84a3757 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,57 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then stream_operator_format_adapter
+  // would be responsible for consuming it.
+  // Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}
diff --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\", RecordType: \"{2}\"}",
         Converter->SymbolOrNumber(Item.FuncId), Item.TId,
diff --git a/llvm/unittests/ADT/TwineTest.cpp b/llvm/unittests/ADT/TwineTest.cpp
index d45805f4db7a3..fc1cc3a0e0db0 100644
--- a/llvm/unittests/ADT/TwineTest.cpp
+++ b/llvm/unittests/ADT/TwineTest.cpp
@@ -119,7 +119,7 @@ TEST(TwineTest, LazyEvaluation) {
     explicit formatter(int &Count) : FormatAdapter(0), Count(Count) {}
     int &Count;
 
-    void format(raw_ostream &OS, StringRef Style) override { ++Count; }
+    void format(raw_ostream &OS, StringRef Style) { ++Count; }
   };
 
   int Count = 0;
diff --git a/llvm/unittests/Support/FormatVariadicTest.cpp b/llvm/unittests/Support/FormatVariadicTest.cpp
index ea549e7553b0c..afe7c6f6ee6f7 100644
--- a/llvm/unittests/Support/FormatVariadicTest.cpp
+++ b/llvm/unittests/Support/FormatVariadicTest.cpp
@@ -18,22 +18,28 @@ using namespace llvm;
 namespace {
 struct Format : public FormatAdapter<int> {
   Format(int N) : FormatAdapter<int>(std::move(N)) {}
-  void format(raw_ostream &OS, StringRef Opt) override { OS << "Format"; }
+  void format(raw_ostream &OS, StringRef Opt) { OS << "Format"; }
 };
 
-using support::detail::uses_format_member;
-using support::detail::uses_missing_provider;
+using support::detail::FormatFunctor;
 
-static_assert(uses_format_member<Format>::value, "");
-static_assert(uses_format_member<Format &>::value, "");
-static_assert(uses_format_member<Format &&>::value, "");
-static_assert(uses_format_member<const Format>::value, "");
-static_assert(uses_format_member<const Format &>::value, "");
-static_assert(uses_format_member<const volatile Format>::value, "");
-static_assert(uses_format_member<const volatile Format &>::value, "");
+static_assert(FormatFunctor<Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &&>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format &>::HasMemberProvider, "");
 
-struct NoFormat {};
-static_assert(uses_missing_provider<NoFormat>::value, "");
+struct StreamFormat {};
+// Silence warning about never-called function.
+[[maybe_unused]] raw_ostream &operator<<(raw_ostream &, const StreamFormat &);
+static_assert(!FormatFunctor<StreamFormat>::HasMemberProvider, "");
+static_assert(!FormatFunctor<StreamFormat>::HasFormatProvider, "");
+static_assert(FormatFunctor<StreamFormat>::HasStreamProvider, "");
+static_assert(!FormatFunctor<int>::HasMemberProvider, "");
+static_assert(FormatFunctor<int>::HasFormatProvider, "");
+static_assert(FormatFunctor<int>::HasStreamProvider, "");
 }
 
 // Helper to parse format string with no validation.
@@ -780,7 +786,7 @@ TEST(FormatVariadicTest, Adapter) {
   class Negative : public FormatAdapter<int> {
   public:
     explicit Negative(int N) : FormatAdapter<int>(std::move(N)) {}
-    void format(raw_ostream &S, StringRef Options) override { S << -Item; }
+    void format(raw_ostream &S, StringRef Options) { S << -Item; }
   };
 
   EXPECT_EQ("-7", formatv("{0}", Negative(7)).str());
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index e9f151e248340..a3c3df487ab83 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -127,15 +127,15 @@ class FmtObjectBase {
   // std::vector<Base*>.
   struct CreateAdapters {
     template <typename... Ts>
-    std::vector<llvm::support::detail::format_adapter *>
+    std::vector<llvm::support::detail::FormatFunctorRef>
     operator()(Ts &...items) {
-      return std::vector<llvm::support::detail::format_adapter *>{&items...};
+      return std::vector<llvm::support::detail::FormatFunctorRef>{items...};
     }
   };
 
   StringRef fmt;
   const FmtContext *context;
-  std::vector<llvm::support::detail::format_adapter *> adapters;
+  std::vector<llvm::support::detail::FormatFunctorRef> adapters;
   std::vector<FmtReplacement> replacements;
 
 public:
@@ -200,8 +200,7 @@ class FmtObject : public FmtObjectBase {
 
 class FmtStrVecObject : public FmtObjectBase {
 public:
-  using StrFormatAdapter = decltype(llvm::support::detail::build_format_adapter(
-      std::declval<std::string>()));
+  using StrFormatAdapter = llvm::support::detail::FormatFunctor<std::string>;
 
   FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
                   ArrayRef<std::string> params);
@@ -254,19 +253,18 @@ class FmtStrVecObject : public FmtObjectBase {
 ///    in C++ code generation.
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
-    -> FmtObject<
-        decltype(std::make_tuple(llvm::support::detail::build_format_adapter(
-            std::forward<Ts>(vals))...))> {
+    -> FmtObject<decltype(std::make_tuple(
+        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::build_format_adapter(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
-      std::make_tuple(llvm::support::detail::build_format_adapter(
-          std::forward<Ts>(vals))...));
+      std::make_tuple(
+          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
 }
 
-inline FmtStrVecObject tgfmt(StringRef fmt, const FmtContext *ctx,
-                             ArrayRef<std::string> params) {
+inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
+                              ArrayRef<std::string> params) {
   return FmtStrVecObject(fmt, ctx, params);
 }
 
diff --git a/mlir/lib/TableGen/Format.cpp b/mlir/lib/TableGen/Format.cpp
index 65f4ad56dd158..2153de2af4797 100644
--- a/mlir/lib/TableGen/Format.cpp
+++ b/mlir/lib/TableGen/Format.cpp
@@ -183,8 +183,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       range = range.drop_front(repl.index);
       if (repl.end != FmtReplacement::kUnset)
         range = range.drop_back(adapters.size() - repl.end);
-      llvm::interleaveComma(range, s,
-                            [&](auto &x) { x->format(s, /*Options=*/""); });
+      llvm::interleaveComma(range, s, [&](auto &x) { x(s, /*Options=*/""); });
       continue;
     }
 
@@ -194,7 +193,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       s << repl.spec << kMarkerForNoSubst;
       continue;
     }
-    adapters[repl.index]->format(s, /*Options=*/"");
+    adapters[repl.index](s, /*Options=*/"");
   }
 }
 
@@ -203,17 +202,16 @@ FmtStrVecObject::FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
     : FmtObjectBase(fmt, ctx, params.size()) {
   parameters.reserve(params.size());
   for (std::string p : params)
-    parameters.push_back(
-        llvm::support::detail::build_format_adapter(std::move(p)));
+    parameters.emplace_back(std::move(p));
 
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
 
 FmtStrVecObject::FmtStrVecObject(FmtStrVecObject &&that)
     : FmtObjectBase(std::move(that)), parameters(std::move(that.parameters)) {
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
diff --git a/mlir/tools/mlir-tblgen/RewriterGen.cpp b/mlir/tools/mlir-tblgen/RewriterGen.cpp
index e3043708a46d1..5bc2ac48b4726 100644
--- a/mlir/tools/mlir-tblgen/RewriterGen.cpp
+++ b/mlir/tools/mlir-tblgen/RewriterGen.cpp
@@ -524,8 +524,8 @@ void PatternEmitter::emitNativeCodeMatch(DagNode tree, StringRef opName,
                          "passing the defining Operation");
 
   auto nativeCodeCall = std::string(
-      tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
-            static_cast<ArrayRef<std::string>>(capture)));
+      tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
+             static_cast<ArrayRef<std::string>>(capture)));
 
   emitMatchCheck(opName, formatv("!::mlir::failed({0})", nativeCodeCall),
                  formatv("\"{0} return ::mlir::failure\"", nativeCodeCall));
@@ -1490,8 +1490,8 @@ std::string PatternEmitter::handleReplaceWithNativeCodeCall(DagNode tree,
                             << " replacement: " << attrs[i] << "\n");
   }
 
-  std::string symbol = tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse),
-                             static_cast<ArrayRef<std::string>>(attrs));
+  std::string symbol = tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse),
+                              static_cast<ArrayRef<std::string>>(attrs));
 
   // In general, NativeCodeCall without naming binding don't need this. To
   // ensure void helper function has been correctly labeled, i.e., use

>From 6da8c7d2e2302b4f2e46f3269c35bb5c7e604263 Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 15:48:05 +0000
Subject: [PATCH 2/3] add template deduction guide

Created using spr 1.3.8-wip
---
 lldb/source/DataFormatters/FormatterBytecode.cpp  | 14 +++++++-------
 llvm/include/llvm/Support/FormatAdapters.h        |  6 +++---
 llvm/include/llvm/Support/FormatProviders.h       |  4 ++--
 llvm/include/llvm/Support/FormatVariadic.h        |  2 +-
 llvm/include/llvm/Support/FormatVariadicDetails.h |  2 ++
 mlir/include/mlir/TableGen/Format.h               |  6 +++---
 6 files changed, 18 insertions(+), 16 deletions(-)

diff --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 786f9198a53d0..2d79641f516f1 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -113,20 +113,20 @@ static llvm::Error FormatImpl(DataStack &data) {
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(FormatFunctor<const char *>(s->c_str()));
+      format(FormatFunctor(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(FormatFunctor<uint64_t *&>(u));
+      format(FormatFunctor(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(FormatFunctor<int64_t *&>(i));
+      format(FormatFunctor(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(FormatFunctor<const char *>("null object"));
+        format(FormatFunctor("null object"));
       else
-        format(FormatFunctor<const char *>(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(FormatFunctor<llvm::StringRef>(type->GetDisplayTypeName()));
+      format(FormatFunctor(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(FormatFunctor<std::string>(toString(*sel)));
+      format(FormatFunctor(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();
diff --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 5a22c0bbde8f4..57a9fc03ea51f 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -36,7 +36,7 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
         Fill(Fill) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,7 +50,7 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     Stream.indent(Left);
     Adapter(Stream, Style);
     Stream.indent(Right);
@@ -65,7 +65,7 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
   void format(llvm::raw_ostream &Stream, StringRef Style) {
-    auto Adapter = detail::FormatFunctor<T>(std::forward<T>(this->Item));
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
       Adapter(Stream, Style);
     }
diff --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 38a203482d594..95176333e91e0 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -395,13 +395,13 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
       Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::FormatFunctor<decltype(*Begin)>(*Begin);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
       Adapter(Stream, ArgStyle);
       ++Begin;
     }
diff --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 322e6e9d8105a..4c92d45c2f5b9 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -246,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::FormatFunctor<Ts>(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 
diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index 9fab9b84a3757..ba919b8896e56 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -74,6 +74,8 @@ template <typename T> class FormatFunctor {
   }
 };
 
+template <typename T> FormatFunctor(T &&) -> FormatFunctor<T>;
+
 } // namespace detail
 } // namespace support
 } // namespace llvm
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index a3c3df487ab83..d0f8be28e493b 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -254,13 +254,13 @@ class FmtStrVecObject : public FmtObjectBase {
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
     -> FmtObject<decltype(std::make_tuple(
-        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
+        llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
       std::make_tuple(
-          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
+          llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
 }
 
 inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,

>From 00e30973cb237f7d4179ee81dc321af6f05922de Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Sat, 4 Jul 2026 17:57:15 +0000
Subject: [PATCH 3/3] improve comments

Created using spr 1.3.8-wip
---
 llvm/include/llvm/Support/FormatVariadicDetails.h | 5 ++---
 mlir/include/mlir/TableGen/Format.h               | 1 +
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index ba919b8896e56..62126dc90c0f7 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -27,9 +27,8 @@ namespace detail {
 using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
 template <typename T> class FormatFunctor {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
+  // If the caller passed an Error by value, then we would be responsible for
+  // consuming it. Make the caller opt into this by calling fmt_consume().
   static_assert(
       !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
       "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index d0f8be28e493b..daadf8e636feb 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -263,6 +263,7 @@ inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
           llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
 }
 
+/// Like tgfmt, but take replacement parameters as an array of strings instead.
 inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
                               ArrayRef<std::string> params) {
   return FmtStrVecObject(fmt, ctx, params);


From llvmlistbot at llvm.org  Sat Jul  4 13:58:53 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Sat, 04 Jul 2026 13:58:53 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a49740d.fdc477d6.1a5bda.fbfc@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 01/13] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 02/13] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 03/13] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 04/13] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 05/13] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 06/13] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 07/13] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 08/13] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 09/13] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.

>From 39b303e345d8d744745fad8bef4423ed6ce75c2d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:18 +0200
Subject: [PATCH 10/13] Update mlir/lib/Target/LLVMIR/DebugTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fca5e0a8497eb..10ae5ef0ec646 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,7 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (DIGlobalVariableExpressionAttr GVE =
+        if (auto GVE =
                 dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 

>From 292e16924dd039876c60e17afc4fa491e448e79d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:36:41 +0200
Subject: [PATCH 11/13] Clang-format fix

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  4 +-
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 52 +++++++++----------
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     | 32 ++++++------
 3 files changed, 41 insertions(+), 47 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 62523ffef3895..846342bee6071 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2548,9 +2548,7 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) {
-    replaceOperandWith(7, N.get());
-  }
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 29b010ee37f5b..72497a5328f78 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -710,36 +710,34 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
                                       [LLVM_DIRecursiveTypeAttrInterface],
                                       "DILocalScopeAttr"> {
   let parameters = (ins
-    // DIRecursiveTypeAttrInterface specific parameters.
-    OptionalParameter<"DistinctAttr">:$recId,
-    OptionalParameter<"bool">:$isRecSelf,
-    // DISubprogramAttr specific parameters.
-    OptionalParameter<"DistinctAttr">:$id,
-    OptionalParameter<"DICompileUnitAttr">:$compileUnit,
-    OptionalParameter<"DIScopeAttr">:$scope,
-    OptionalParameter<"StringAttr">:$name,
-    OptionalParameter<"StringAttr">:$linkageName,
-    OptionalParameter<"DIFileAttr">:$file,
-    OptionalParameter<"unsigned">:$line,
-    OptionalParameter<"unsigned">:$scopeLine,
-    OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
-    OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
-    OptionalArrayRefParameter<"DINodeAttr">:$annotations
-  );
-  let builders = [
-    AttrBuilder<(ins
-      "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
-      "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
-      "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
-      "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
-    ), [{
+      // DIRecursiveTypeAttrInterface specific parameters.
+      OptionalParameter<"DistinctAttr">:$recId,
+      OptionalParameter<"bool">:$isRecSelf,
+      // DISubprogramAttr specific parameters.
+      OptionalParameter<"DistinctAttr">:$id,
+      OptionalParameter<"DICompileUnitAttr">:$compileUnit,
+      OptionalParameter<"DIScopeAttr">:$scope,
+      OptionalParameter<"StringAttr">:$name,
+      OptionalParameter<"StringAttr">:$linkageName,
+      OptionalParameter<"DIFileAttr">:$file,
+      OptionalParameter<"unsigned">:$line,
+      OptionalParameter<"unsigned">:$scopeLine,
+      OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
+      OptionalParameter<"DISubroutineTypeAttr">:$type,
+      OptionalArrayRefParameter<"Attribute">:$retainedNodes,
+      OptionalArrayRefParameter<"DINodeAttr">:$annotations);
+  let builders = [AttrBuilder<
+      (ins "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
+          "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
+          "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
+          "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
+          "ArrayRef<Attribute>":$retainedNodes,
+          "ArrayRef<DINodeAttr>":$annotations),
+      [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
                    subprogramFlags, type, retainedNodes, annotations);
-    }]>
-  ];
+    }]>];
   let assemblyFormat = "`<` struct(params) `>`";
   let extraClassDeclaration = [{
     /// Requirements of DIRecursiveTypeAttrInterface.
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index 0c78689fe1d6b..b25a8c8a9829e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -150,23 +150,21 @@ def DICompileUnitAttr : DialectAttribute<(attr
 // DISubprogramAttr
 //===----------------------------------------------------------------------===//
 
-def DISubprogramAttr : DialectAttribute<(attr
-  OptionalAttribute<"DistinctAttr">:$recId,
-  Bool:$isRecSelf,
-  OptionalAttribute<"DistinctAttr">:$id,
-  OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
-  OptionalAttribute<"DIScopeAttr">:$scope,
-  OptionalAttribute<"StringAttr">:$name,
-  OptionalAttribute<"StringAttr">:$linkageName,
-  OptionalAttribute<"DIFileAttr">:$file,
-  VarInt:$line,
-  VarInt:$scopeLine,
-  EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
-  LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
-  OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"Attribute">:$retainedNodes,
-  OptionalArrayRef<"DINodeAttr">:$annotations
-)>;
+def DISubprogramAttr
+    : DialectAttribute<(attr OptionalAttribute<"DistinctAttr">:$recId,
+          Bool:$isRecSelf, OptionalAttribute<"DistinctAttr">:$id,
+          OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
+          OptionalAttribute<"DIScopeAttr">:$scope,
+          OptionalAttribute<"StringAttr">:$name,
+          OptionalAttribute<"StringAttr">:$linkageName,
+          OptionalAttribute<"DIFileAttr">:$file, VarInt:$line,
+          VarInt:$scopeLine,
+          EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
+          LocalVar<"DISubprogramFlags",
+                   "(DISubprogramFlags)_rawflags">:$subprogramFlags,
+          OptionalAttribute<"DISubroutineTypeAttr">:$type,
+          OptionalArrayRef<"Attribute">:$retainedNodes,
+          OptionalArrayRef<"DINodeAttr">:$annotations)>;
 
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr

>From 3b92877dfd5c5ffbc0b1c557a78bcbf6cafce502 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:44:51 +0200
Subject: [PATCH 12/13] Clang-format fix

---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 10ae5ef0ec646..442f5b6b137e6 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,8 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (auto GVE =
-                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+        if (auto GVE = dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 
         auto diAttr = dyn_cast<DINodeAttr>(attr);

>From 23a9b20861b2203633fedb5734e645e14175fcf7 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sat, 4 Jul 2026 22:58:30 +0200
Subject: [PATCH 13/13] Fix LVIRReader.cpp

---
 llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
index eb9fa3c89a44e..5726955a029ef 100644
--- a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
+++ b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
@@ -1447,13 +1447,15 @@ LVScope *LVIRReader::getOrCreateSubprogram(LVScope *Function,
   }
 
   // Check for additional retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *IE = dyn_cast<DIImportedEntity>(DN))
       constructImportedEntity(Function, IE);
     else if (const auto *TTP = dyn_cast<DITemplateTypeParameter>(DN))
       constructTemplateTypeParameter(Function, TTP);
     else if (const auto *TVP = dyn_cast<DITemplateValueParameter>(DN))
       constructTemplateValueParameter(Function, TVP);
+    else if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(DN))
+      getOrCreateVariable(GVE);
   }
 
   applySubprogramAttributes(Function, SP);


From llvmlistbot at llvm.org  Sat Jul  4 17:28:40 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Sat, 04 Jul 2026 17:28:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a49a538.f1c1923d.3672e2.521e@mx.google.com>


https://github.com/lforg37 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {


From llvmlistbot at llvm.org  Sat Jul  4 23:45:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 04 Jul 2026 23:45:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3aec6a4 - [Support] Remove virtual functions
	from formatv (#207516)
Message-ID: <6a49fda2.cb30c3e4.6f38b.39fe@mx.google.com>


Author: Alexis Engelke
Date: 2026-07-05T08:45:49+02:00
New Revision: 3aec6a40bb4e49f9ea181ac5c949b6c7c20a5465

URL: https://github.com/llvm/llvm-project/commit/3aec6a40bb4e49f9ea181ac5c949b6c7c20a5465
DIFF: https://github.com/llvm/llvm-project/commit/3aec6a40bb4e49f9ea181ac5c949b6c7c20a5465.diff

LOG: [Support] Remove virtual functions from formatv (#207516)

Currently, formatv erases types using a base class and calls the virtual
function format() to format the objects. To avoid these vtables,
refactor formatv to instead store function_refs to a functor (struct
with overloaded operator()). This saves ~5kiB in vtables.

Additionally, add a static assertion that a formatter is present instead
of relying on errors due to missing templates. This requires a little
change to MLIR's tgfmt to use a different name -- previously, the
substitution failure on ArrayRef<> would cause the variadic function to
be skipped, but a static assertion failure is not a substitution error.

Also, simplify the code in FormatVariadicDetails to use if constexpr
instead of template overloads with enable_if, making the code shorter
and easier to read.

Added: 
    

Modified: 
    lldb/source/DataFormatters/FormatterBytecode.cpp
    llvm/include/llvm/DebugInfo/CodeView/Formatters.h
    llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
    llvm/include/llvm/Support/FormatAdapters.h
    llvm/include/llvm/Support/FormatCommon.h
    llvm/include/llvm/Support/FormatProviders.h
    llvm/include/llvm/Support/FormatVariadic.h
    llvm/include/llvm/Support/FormatVariadicDetails.h
    llvm/lib/Support/FormatVariadic.cpp
    llvm/tools/llvm-xray/xray-stacks.cpp
    llvm/unittests/ADT/TwineTest.cpp
    llvm/unittests/Support/FormatVariadicTest.cpp
    mlir/include/mlir/TableGen/Format.h
    mlir/lib/TableGen/Format.cpp
    mlir/tools/mlir-tblgen/RewriterGen.cpp

Removed: 
    


################################################################################
diff  --git a/lldb/source/DataFormatters/FormatterBytecode.cpp b/lldb/source/DataFormatters/FormatterBytecode.cpp
index 1936524d37dfc..2d79641f516f1 100644
--- a/lldb/source/DataFormatters/FormatterBytecode.cpp
+++ b/lldb/source/DataFormatters/FormatterBytecode.cpp
@@ -107,26 +107,26 @@ static llvm::Error FormatImpl(DataStack &data) {
     }
     using namespace llvm::support::detail;
     auto arg = data[data.size() - num_args + r.Index];
-    auto format = [&](format_adapter &&adapter) {
+    auto format = [&](FormatFunctorRef &&adapter) {
       llvm::FmtAlign Align(adapter, r.Where, r.Width, r.Pad);
       Align.format(os, r.Options);
     };
 
     if (auto s = std::get_if<std::string>(&arg))
-      format(build_format_adapter(s->c_str()));
+      format(FormatFunctor(s->c_str()));
     else if (auto u = std::get_if<uint64_t>(&arg))
-      format(build_format_adapter(u));
+      format(FormatFunctor(u));
     else if (auto i = std::get_if<int64_t>(&arg))
-      format(build_format_adapter(i));
+      format(FormatFunctor(i));
     else if (auto valobj = std::get_if<ValueObjectSP>(&arg)) {
       if (!valobj->get())
-        format(build_format_adapter("null object"));
+        format(FormatFunctor("null object"));
       else
-        format(build_format_adapter(valobj->get()->GetValueAsCString()));
+        format(FormatFunctor(valobj->get()->GetValueAsCString()));
     } else if (auto type = std::get_if<CompilerType>(&arg))
-      format(build_format_adapter(type->GetDisplayTypeName()));
+      format(FormatFunctor(type->GetDisplayTypeName()));
     else if (auto sel = std::get_if<FormatterBytecode::Selectors>(&arg))
-      format(build_format_adapter(toString(*sel)));
+      format(FormatFunctor(toString(*sel)));
   }
   data.Push(s);
   return llvm::Error::success();

diff  --git a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
index bf0340be901e7..dd802d1b144b0 100644
--- a/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
+++ b/llvm/include/llvm/DebugInfo/CodeView/Formatters.h
@@ -34,7 +34,7 @@ class LLVM_ABI GuidAdapter final : public FormatAdapter<ArrayRef<uint8_t>> {
   explicit GuidAdapter(ArrayRef<uint8_t> Guid);
   explicit GuidAdapter(StringRef Guid);
 
-  void format(raw_ostream &Stream, StringRef Style) override;
+  void format(raw_ostream &Stream, StringRef Style);
 };
 
 } // end namespace detail

diff  --git a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
index 7f70d4157c358..97fb3a12f1501 100644
--- a/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
+++ b/llvm/include/llvm/DebugInfo/PDB/Native/FormatUtil.h
@@ -72,7 +72,7 @@ struct EndianAdapter final
   explicit EndianAdapter(EndianType &&Item)
       : FormatAdapter<EndianType>(std::move(Item)) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
     format_provider<T>::format(static_cast<T>(this->Item), Stream, Style);
   }
 };

diff  --git a/llvm/include/llvm/Support/FormatAdapters.h b/llvm/include/llvm/Support/FormatAdapters.h
index 91e9c41d8a395..57a9fc03ea51f 100644
--- a/llvm/include/llvm/Support/FormatAdapters.h
+++ b/llvm/include/llvm/Support/FormatAdapters.h
@@ -16,8 +16,7 @@
 #include "llvm/Support/raw_ostream.h"
 
 namespace llvm {
-template <typename T>
-class FormatAdapter : public support::detail::format_adapter {
+template <typename T> class FormatAdapter {
 protected:
   explicit FormatAdapter(T &&Item) : Item(std::forward<T>(Item)) {}
 
@@ -36,8 +35,8 @@ template <typename T> class AlignAdapter final : public FormatAdapter<T> {
       : FormatAdapter<T>(std::forward<T>(Item)), Where(Where), Amount(Amount),
         Fill(Fill) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     FmtAlign(Adapter, Where, Amount, Fill).format(Stream, Style);
   }
 };
@@ -50,10 +49,10 @@ template <typename T> class PadAdapter final : public FormatAdapter<T> {
   PadAdapter(T &&Item, size_t Left, size_t Right)
       : FormatAdapter<T>(std::forward<T>(Item)), Left(Left), Right(Right) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     Stream.indent(Left);
-    Adapter.format(Stream, Style);
+    Adapter(Stream, Style);
     Stream.indent(Right);
   }
 };
@@ -65,10 +64,10 @@ template <typename T> class RepeatAdapter final : public FormatAdapter<T> {
   RepeatAdapter(T &&Item, size_t Count)
       : FormatAdapter<T>(std::forward<T>(Item)), Count(Count) {}
 
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    auto Adapter = detail::build_format_adapter(std::forward<T>(this->Item));
+  void format(llvm::raw_ostream &Stream, StringRef Style) {
+    auto Adapter = detail::FormatFunctor(std::forward<T>(this->Item));
     for (size_t I = 0; I < Count; ++I) {
-      Adapter.format(Stream, Style);
+      Adapter(Stream, Style);
     }
   }
 };
@@ -77,10 +76,8 @@ class ErrorAdapter : public FormatAdapter<Error> {
 public:
   ErrorAdapter(Error &&Item) : FormatAdapter(std::move(Item)) {}
   ErrorAdapter(ErrorAdapter &&) = default;
-  ~ErrorAdapter() override { consumeError(std::move(Item)); }
-  void format(llvm::raw_ostream &Stream, StringRef Style) override {
-    Stream << Item;
-  }
+  ~ErrorAdapter() { consumeError(std::move(Item)); }
+  void format(llvm::raw_ostream &Stream, StringRef Style) { Stream << Item; }
 };
 } // namespace detail
 } // namespace support

diff  --git a/llvm/include/llvm/Support/FormatCommon.h b/llvm/include/llvm/Support/FormatCommon.h
index eaf291b864c5e..112cf1fd68478 100644
--- a/llvm/include/llvm/Support/FormatCommon.h
+++ b/llvm/include/llvm/Support/FormatCommon.h
@@ -19,12 +19,12 @@ enum class AlignStyle { Left, Center, Right };
 /// Helper class to format to a \p Width wide field, with alignment \p Where
 /// within that field.
 struct FmtAlign {
-  support::detail::format_adapter &Adapter;
+  support::detail::FormatFunctorRef Adapter;
   AlignStyle Where;
   unsigned Width;
   char Fill;
 
-  FmtAlign(support::detail::format_adapter &Adapter, AlignStyle Where,
+  FmtAlign(support::detail::FormatFunctorRef Adapter, AlignStyle Where,
            unsigned Width, char Fill = ' ')
       : Adapter(Adapter), Where(Where), Width(Width), Fill(Fill) {}
 
@@ -35,13 +35,13 @@ struct FmtAlign {
     // TODO: Make the format method return the number of bytes written, that
     // way we can also skip the intermediate stream for left-aligned output.
     if (Width == 0) {
-      Adapter.format(S, Options);
+      Adapter(S, Options);
       return;
     }
     SmallString<64> Item;
     raw_svector_ostream Stream(Item);
 
-    Adapter.format(Stream, Options);
+    Adapter(Stream, Options);
     if (Width <= Item.size()) {
       S << Item;
       return;

diff  --git a/llvm/include/llvm/Support/FormatProviders.h b/llvm/include/llvm/Support/FormatProviders.h
index 78eeec76cf79b..95176333e91e0 100644
--- a/llvm/include/llvm/Support/FormatProviders.h
+++ b/llvm/include/llvm/Support/FormatProviders.h
@@ -325,18 +325,6 @@ struct format_provider<
   }
 };
 
-namespace support {
-namespace detail {
-template <typename IterT>
-using IterValue = typename std::iterator_traits<IterT>::value_type;
-
-template <typename IterT>
-struct range_item_has_provider
-    : public std::bool_constant<
-          !support::detail::uses_missing_provider<IterValue<IterT>>::value> {};
-} // namespace detail
-} // namespace support
-
 /// Implementation of format_provider<T> for ranges.
 ///
 /// This will print an arbitrary range as a delimited sequence of items.
@@ -399,8 +387,6 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
   }
 
 public:
-  static_assert(support::detail::range_item_has_provider<IterT>::value,
-                "Range value_type does not have a format provider!");
   static void format(const llvm::iterator_range<IterT> &V,
                      llvm::raw_ostream &Stream, StringRef Style) {
     StringRef Sep;
@@ -409,14 +395,14 @@ template <typename IterT> class format_provider<llvm::iterator_range<IterT>> {
     auto Begin = V.begin();
     auto End = V.end();
     if (Begin != End) {
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
     while (Begin != End) {
       Stream << Sep;
-      auto Adapter = support::detail::build_format_adapter(*Begin);
-      Adapter.format(Stream, ArgStyle);
+      auto Adapter = support::detail::FormatFunctor(*Begin);
+      Adapter(Stream, ArgStyle);
       ++Begin;
     }
   }

diff  --git a/llvm/include/llvm/Support/FormatVariadic.h b/llvm/include/llvm/Support/FormatVariadic.h
index 145fbb47c91b4..4c92d45c2f5b9 100644
--- a/llvm/include/llvm/Support/FormatVariadic.h
+++ b/llvm/include/llvm/Support/FormatVariadic.h
@@ -65,11 +65,11 @@ struct ReplacementItem {
 class formatv_object_base {
 protected:
   StringRef Fmt;
-  ArrayRef<support::detail::format_adapter *> Adapters;
+  ArrayRef<support::detail::FormatFunctorRef> Adapters;
   bool Validate;
 
   formatv_object_base(StringRef Fmt,
-                      ArrayRef<support::detail::format_adapter *> Adapters,
+                      ArrayRef<support::detail::FormatFunctorRef> Adapters,
                       bool Validate)
       : Fmt(Fmt), Adapters(Adapters), Validate(Validate) {}
 
@@ -89,9 +89,9 @@ class formatv_object_base {
         continue;
       }
 
-      auto *W = Adapters[R.Index];
+      auto &W = Adapters[R.Index];
 
-      FmtAlign Align(*W, R.Where, R.Width, R.Pad);
+      FmtAlign Align(W, R.Where, R.Width, R.Pad);
       Align.format(S, R.Options);
     }
   }
@@ -122,21 +122,19 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
   // of the parameters, we have to own the storage for the parameters here, and
   // have the base class store type-erased pointers into this tuple.
   Tuple Parameters;
-  std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+  std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
       ParameterPointers;
 
   // The parameters are stored in a std::tuple, which does not provide runtime
   // indexing capabilities.  In order to enable runtime indexing, we use this
   // structure to put the parameters into a std::array.  Since the parameters
   // are not all the same type, we use some type-erasure by wrapping the
-  // parameters in a template class that derives from a non-template superclass.
-  // Essentially, we are converting a std::tuple<Derived<Ts...>> to a
-  // std::array<Base*>.
+  // parameters in a template class and refer to them using function_refs.
   struct create_adapters {
     template <typename... Ts>
-    std::array<support::detail::format_adapter *, std::tuple_size<Tuple>::value>
+    std::array<support::detail::FormatFunctorRef, std::tuple_size<Tuple>::value>
     operator()(Ts &...Items) {
-      return {{&Items...}};
+      return {{Items...}};
     }
   };
 
@@ -248,7 +246,7 @@ template <typename Tuple> class formatv_object : public formatv_object_base {
 template <typename... Ts>
 inline auto formatv(bool Validate, const char *Fmt, Ts &&...Vals) {
   auto Params = std::make_tuple(
-      support::detail::build_format_adapter(std::forward<Ts>(Vals))...);
+      support::detail::FormatFunctor(std::forward<Ts>(Vals))...);
   return formatv_object<decltype(Params)>(Fmt, std::move(Params), Validate);
 }
 

diff  --git a/llvm/include/llvm/Support/FormatVariadicDetails.h b/llvm/include/llvm/Support/FormatVariadicDetails.h
index c0b245e297a58..62126dc90c0f7 100644
--- a/llvm/include/llvm/Support/FormatVariadicDetails.h
+++ b/llvm/include/llvm/Support/FormatVariadicDetails.h
@@ -10,6 +10,7 @@
 #define LLVM_SUPPORT_FORMATVARIADICDETAILS_H
 
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/STLForwardCompat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/raw_ostream.h"
@@ -22,134 +23,58 @@ class Error;
 
 namespace support {
 namespace detail {
-class LLVM_ABI format_adapter {
-  virtual void anchor();
 
-protected:
-  virtual ~format_adapter() = default;
+using FormatFunctorRef = function_ref<void(llvm::raw_ostream &, StringRef)>;
 
-public:
-  virtual void format(raw_ostream &S, StringRef Options) = 0;
-};
-
-template <typename T> class provider_format_adapter : public format_adapter {
-  T Item;
-
-public:
-  explicit provider_format_adapter(T &&Item) : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef Options) override {
-    format_provider<std::decay_t<T>>::format(Item, S, Options);
-  }
-};
+template <typename T> class FormatFunctor {
+  // If the caller passed an Error by value, then we would be responsible for
+  // consuming it. Make the caller opt into this by calling fmt_consume().
+  static_assert(
+      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
+      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
 
-template <typename T>
-class stream_operator_format_adapter : public format_adapter {
   T Item;
 
-public:
-  explicit stream_operator_format_adapter(T &&Item)
-      : Item(std::forward<T>(Item)) {}
-
-  void format(llvm::raw_ostream &S, StringRef) override { S << Item; }
-};
-
-template <typename T> class missing_format_adapter;
-
-// Test if format_provider<T> is defined on T and contains a member function
-// with the signature:
-//   static void format(const T&, raw_stream &, StringRef);
-//
-template <class T> class has_FormatProvider {
-public:
-  using Decayed = std::decay_t<T>;
-  using Signature_format = void (*)(const Decayed &, llvm::raw_ostream &,
+  using DecayedT = std::decay_t<T>;
+  using Signature_format = void (*)(const DecayedT &, llvm::raw_ostream &,
                                     StringRef);
 
-  template <typename U> using check = SameType<Signature_format, &U::format>;
-
-  static constexpr bool value =
-      llvm::is_detected<check, llvm::format_provider<Decayed>>::value;
-};
-
-// Test if raw_ostream& << T -> raw_ostream& is findable via ADL.
-template <class T> class has_StreamOperator {
-public:
-  using ConstRefT = const std::decay_t<T> &;
-
   template <typename U>
-  static auto test(int)
-      -> std::is_same<decltype(std::declval<llvm::raw_ostream &>()
-                               << std::declval<U>()),
-                      llvm::raw_ostream &>;
-
-  template <typename U> static auto test(...) -> std::false_type;
+  using MemberFormatCheck = decltype(std::declval<U>().format(
+      std::declval<llvm::raw_ostream &>(), std::declval<llvm::StringRef>()));
+  template <typename U>
+  using StaticFormatCheck = SameType<Signature_format, &U::format>;
+  template <typename U>
+  using StreamCheck = std::is_same<decltype(std::declval<llvm::raw_ostream &>()
+                                            << std::declval<U>()),
+                                   llvm::raw_ostream &>;
 
-  static constexpr bool value = decltype(test<ConstRefT>(0))::value;
+public:
+  static constexpr bool HasMemberProvider =
+      llvm::is_detected<MemberFormatCheck, DecayedT>::value;
+  static constexpr bool HasFormatProvider =
+      llvm::is_detected<StaticFormatCheck,
+                        llvm::format_provider<DecayedT>>::value;
+  static constexpr bool HasStreamProvider =
+      llvm::is_detected<StreamCheck, DecayedT>::value;
+
+  static_assert(HasMemberProvider || HasFormatProvider || HasStreamProvider,
+                "type has no format provider");
+
+  explicit FormatFunctor(T &&Item) : Item(std::forward<T>(Item)) {}
+
+  void operator()(llvm::raw_ostream &S, StringRef Options) {
+    if constexpr (HasMemberProvider)
+      Item.format(S, Options);
+    else if constexpr (HasFormatProvider)
+      format_provider<DecayedT>::format(Item, S, Options);
+    else if constexpr (HasStreamProvider)
+      S << Item;
+  }
 };
 
-// Simple template that decides whether a type T should use the member-function
-// based format() invocation.
-template <typename T>
-struct uses_format_member
-    : public std::is_base_of<format_adapter, std::remove_reference_t<T>> {};
-
-// Simple template that decides whether a type T should use the format_provider
-// based format() invocation.  The member function takes priority, so this test
-// will only be true if there is not ALSO a format member.
-template <typename T>
-struct uses_format_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                has_FormatProvider<T>::value> {};
-
-// Simple template that decides whether a type T should use the operator<<
-// based format() invocation.  This takes last priority.
-template <typename T>
-struct uses_stream_operator
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                has_StreamOperator<T>::value> {};
-
-// Simple template that decides whether a type T has neither a member-function
-// nor format_provider based implementation that it can use.  Mostly used so
-// that the compiler spits out a nice diagnostic when a type with no format
-// implementation can be located.
-template <typename T>
-struct uses_missing_provider
-    : public std::bool_constant<!uses_format_member<T>::value &&
-                                !uses_format_provider<T>::value &&
-                                !uses_stream_operator<T>::value> {};
-
-template <typename T>
-std::enable_if_t<uses_format_member<T>::value, T>
-build_format_adapter(T &&Item) {
-  return std::forward<T>(Item);
-}
-
-template <typename T>
-std::enable_if_t<uses_format_provider<T>::value, provider_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  return provider_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_stream_operator<T>::value,
-                 stream_operator_format_adapter<T>>
-build_format_adapter(T &&Item) {
-  // If the caller passed an Error by value, then stream_operator_format_adapter
-  // would be responsible for consuming it.
-  // Make the caller opt into this by calling fmt_consume().
-  static_assert(
-      !std::is_same_v<llvm::Error, std::remove_cv_t<T>>,
-      "llvm::Error-by-value must be wrapped in fmt_consume() for formatv");
-  return stream_operator_format_adapter<T>(std::forward<T>(Item));
-}
-
-template <typename T>
-std::enable_if_t<uses_missing_provider<T>::value, missing_format_adapter<T>>
-build_format_adapter(T &&) {
-  return missing_format_adapter<T>();
-}
+template <typename T> FormatFunctor(T &&) -> FormatFunctor<T>;
+
 } // namespace detail
 } // namespace support
 } // namespace llvm

diff  --git a/llvm/lib/Support/FormatVariadic.cpp b/llvm/lib/Support/FormatVariadic.cpp
index f3e8d0a7fe6f3..a853bd357aac3 100644
--- a/llvm/lib/Support/FormatVariadic.cpp
+++ b/llvm/lib/Support/FormatVariadic.cpp
@@ -222,5 +222,3 @@ formatv_object_base::parseFormatString(StringRef Fmt, size_t NumArgs,
 #endif // ENABLE_VALIDATION
   return Replacements;
 }
-
-void support::detail::format_adapter::anchor() {}

diff  --git a/llvm/tools/llvm-xray/xray-stacks.cpp b/llvm/tools/llvm-xray/xray-stacks.cpp
index 8101cad28ca1c..28e567cf81c87 100644
--- a/llvm/tools/llvm-xray/xray-stacks.cpp
+++ b/llvm/tools/llvm-xray/xray-stacks.cpp
@@ -115,7 +115,7 @@ struct format_xray_record : public FormatAdapter<XRayRecord> {
   explicit format_xray_record(XRayRecord record,
                               const FuncIdConversionHelper &conv)
       : FormatAdapter<XRayRecord>(std::move(record)), Converter(&conv) {}
-  void format(raw_ostream &Stream, StringRef Style) override {
+  void format(raw_ostream &Stream, StringRef Style) {
     Stream << formatv(
         "{FuncId: \"{0}\", ThreadId: \"{1}\", RecordType: \"{2}\"}",
         Converter->SymbolOrNumber(Item.FuncId), Item.TId,

diff  --git a/llvm/unittests/ADT/TwineTest.cpp b/llvm/unittests/ADT/TwineTest.cpp
index d45805f4db7a3..fc1cc3a0e0db0 100644
--- a/llvm/unittests/ADT/TwineTest.cpp
+++ b/llvm/unittests/ADT/TwineTest.cpp
@@ -119,7 +119,7 @@ TEST(TwineTest, LazyEvaluation) {
     explicit formatter(int &Count) : FormatAdapter(0), Count(Count) {}
     int &Count;
 
-    void format(raw_ostream &OS, StringRef Style) override { ++Count; }
+    void format(raw_ostream &OS, StringRef Style) { ++Count; }
   };
 
   int Count = 0;

diff  --git a/llvm/unittests/Support/FormatVariadicTest.cpp b/llvm/unittests/Support/FormatVariadicTest.cpp
index ea549e7553b0c..afe7c6f6ee6f7 100644
--- a/llvm/unittests/Support/FormatVariadicTest.cpp
+++ b/llvm/unittests/Support/FormatVariadicTest.cpp
@@ -18,22 +18,28 @@ using namespace llvm;
 namespace {
 struct Format : public FormatAdapter<int> {
   Format(int N) : FormatAdapter<int>(std::move(N)) {}
-  void format(raw_ostream &OS, StringRef Opt) override { OS << "Format"; }
+  void format(raw_ostream &OS, StringRef Opt) { OS << "Format"; }
 };
 
-using support::detail::uses_format_member;
-using support::detail::uses_missing_provider;
+using support::detail::FormatFunctor;
 
-static_assert(uses_format_member<Format>::value, "");
-static_assert(uses_format_member<Format &>::value, "");
-static_assert(uses_format_member<Format &&>::value, "");
-static_assert(uses_format_member<const Format>::value, "");
-static_assert(uses_format_member<const Format &>::value, "");
-static_assert(uses_format_member<const volatile Format>::value, "");
-static_assert(uses_format_member<const volatile Format &>::value, "");
+static_assert(FormatFunctor<Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<Format &&>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const Format &>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format>::HasMemberProvider, "");
+static_assert(FormatFunctor<const volatile Format &>::HasMemberProvider, "");
 
-struct NoFormat {};
-static_assert(uses_missing_provider<NoFormat>::value, "");
+struct StreamFormat {};
+// Silence warning about never-called function.
+[[maybe_unused]] raw_ostream &operator<<(raw_ostream &, const StreamFormat &);
+static_assert(!FormatFunctor<StreamFormat>::HasMemberProvider, "");
+static_assert(!FormatFunctor<StreamFormat>::HasFormatProvider, "");
+static_assert(FormatFunctor<StreamFormat>::HasStreamProvider, "");
+static_assert(!FormatFunctor<int>::HasMemberProvider, "");
+static_assert(FormatFunctor<int>::HasFormatProvider, "");
+static_assert(FormatFunctor<int>::HasStreamProvider, "");
 }
 
 // Helper to parse format string with no validation.
@@ -780,7 +786,7 @@ TEST(FormatVariadicTest, Adapter) {
   class Negative : public FormatAdapter<int> {
   public:
     explicit Negative(int N) : FormatAdapter<int>(std::move(N)) {}
-    void format(raw_ostream &S, StringRef Options) override { S << -Item; }
+    void format(raw_ostream &S, StringRef Options) { S << -Item; }
   };
 
   EXPECT_EQ("-7", formatv("{0}", Negative(7)).str());

diff  --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index e9f151e248340..daadf8e636feb 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -127,15 +127,15 @@ class FmtObjectBase {
   // std::vector<Base*>.
   struct CreateAdapters {
     template <typename... Ts>
-    std::vector<llvm::support::detail::format_adapter *>
+    std::vector<llvm::support::detail::FormatFunctorRef>
     operator()(Ts &...items) {
-      return std::vector<llvm::support::detail::format_adapter *>{&items...};
+      return std::vector<llvm::support::detail::FormatFunctorRef>{items...};
     }
   };
 
   StringRef fmt;
   const FmtContext *context;
-  std::vector<llvm::support::detail::format_adapter *> adapters;
+  std::vector<llvm::support::detail::FormatFunctorRef> adapters;
   std::vector<FmtReplacement> replacements;
 
 public:
@@ -200,8 +200,7 @@ class FmtObject : public FmtObjectBase {
 
 class FmtStrVecObject : public FmtObjectBase {
 public:
-  using StrFormatAdapter = decltype(llvm::support::detail::build_format_adapter(
-      std::declval<std::string>()));
+  using StrFormatAdapter = llvm::support::detail::FormatFunctor<std::string>;
 
   FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
                   ArrayRef<std::string> params);
@@ -254,19 +253,19 @@ class FmtStrVecObject : public FmtObjectBase {
 ///    in C++ code generation.
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
-    -> FmtObject<
-        decltype(std::make_tuple(llvm::support::detail::build_format_adapter(
-            std::forward<Ts>(vals))...))> {
+    -> FmtObject<decltype(std::make_tuple(
+        llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::build_format_adapter(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
-      std::make_tuple(llvm::support::detail::build_format_adapter(
-          std::forward<Ts>(vals))...));
+      std::make_tuple(
+          llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
 }
 
-inline FmtStrVecObject tgfmt(StringRef fmt, const FmtContext *ctx,
-                             ArrayRef<std::string> params) {
+/// Like tgfmt, but take replacement parameters as an array of strings instead.
+inline FmtStrVecObject tgfmtv(StringRef fmt, const FmtContext *ctx,
+                              ArrayRef<std::string> params) {
   return FmtStrVecObject(fmt, ctx, params);
 }
 

diff  --git a/mlir/lib/TableGen/Format.cpp b/mlir/lib/TableGen/Format.cpp
index 65f4ad56dd158..2153de2af4797 100644
--- a/mlir/lib/TableGen/Format.cpp
+++ b/mlir/lib/TableGen/Format.cpp
@@ -183,8 +183,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       range = range.drop_front(repl.index);
       if (repl.end != FmtReplacement::kUnset)
         range = range.drop_back(adapters.size() - repl.end);
-      llvm::interleaveComma(range, s,
-                            [&](auto &x) { x->format(s, /*Options=*/""); });
+      llvm::interleaveComma(range, s, [&](auto &x) { x(s, /*Options=*/""); });
       continue;
     }
 
@@ -194,7 +193,7 @@ void FmtObjectBase::format(raw_ostream &s) const {
       s << repl.spec << kMarkerForNoSubst;
       continue;
     }
-    adapters[repl.index]->format(s, /*Options=*/"");
+    adapters[repl.index](s, /*Options=*/"");
   }
 }
 
@@ -203,17 +202,16 @@ FmtStrVecObject::FmtStrVecObject(StringRef fmt, const FmtContext *ctx,
     : FmtObjectBase(fmt, ctx, params.size()) {
   parameters.reserve(params.size());
   for (std::string p : params)
-    parameters.push_back(
-        llvm::support::detail::build_format_adapter(std::move(p)));
+    parameters.emplace_back(std::move(p));
 
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }
 
 FmtStrVecObject::FmtStrVecObject(FmtStrVecObject &&that)
     : FmtObjectBase(std::move(that)), parameters(std::move(that.parameters)) {
   adapters.reserve(parameters.size());
   for (auto &p : parameters)
-    adapters.push_back(&p);
+    adapters.push_back(p);
 }

diff  --git a/mlir/tools/mlir-tblgen/RewriterGen.cpp b/mlir/tools/mlir-tblgen/RewriterGen.cpp
index e3043708a46d1..5bc2ac48b4726 100644
--- a/mlir/tools/mlir-tblgen/RewriterGen.cpp
+++ b/mlir/tools/mlir-tblgen/RewriterGen.cpp
@@ -524,8 +524,8 @@ void PatternEmitter::emitNativeCodeMatch(DagNode tree, StringRef opName,
                          "passing the defining Operation");
 
   auto nativeCodeCall = std::string(
-      tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
-            static_cast<ArrayRef<std::string>>(capture)));
+      tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse).withSelf(opName.str()),
+             static_cast<ArrayRef<std::string>>(capture)));
 
   emitMatchCheck(opName, formatv("!::mlir::failed({0})", nativeCodeCall),
                  formatv("\"{0} return ::mlir::failure\"", nativeCodeCall));
@@ -1490,8 +1490,8 @@ std::string PatternEmitter::handleReplaceWithNativeCodeCall(DagNode tree,
                             << " replacement: " << attrs[i] << "\n");
   }
 
-  std::string symbol = tgfmt(fmt, &fmtCtx.addSubst("_loc", locToUse),
-                             static_cast<ArrayRef<std::string>>(attrs));
+  std::string symbol = tgfmtv(fmt, &fmtCtx.addSubst("_loc", locToUse),
+                              static_cast<ArrayRef<std::string>>(attrs));
 
   // In general, NativeCodeCall without naming binding don't need this. To
   // ensure void helper function has been correctly labeled, i.e., use


        

From llvmlistbot at llvm.org  Sat Jul  4 23:45:57 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 23:45:57 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a49fda5.27b3b003.1610ae.0de2@mx.google.com>


https://github.com/aengelke closed https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Sat Jul  4 23:58:30 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 23:58:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [MC] Generate FeatureKV with
 compact string table (PR #206331)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206331@github.com>
Message-ID: <6a4a0096.ac469dc7.e3aa6.f4bd@mx.google.com>


https://github.com/aengelke edited https://github.com/llvm/llvm-project/pull/206331

From llvmlistbot at llvm.org  Sat Jul  4 23:59:15 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sat, 04 Jul 2026 23:59:15 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [MC] Generate FeatureKV with
 compact string table (PR #206331)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206331@github.com>
Message-ID: <6a4a00c3.7d41f3a1.d1cf7.ef82@mx.google.com>


https://github.com/aengelke auto_merge_enabled https://github.com/llvm/llvm-project/pull/206331

From llvmlistbot at llvm.org  Sun Jul  5 00:42:30 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Sun, 05 Jul 2026 00:42:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [MC] Generate FeatureKV with
 compact string table (PR #206331)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206331@github.com>
Message-ID: <6a4a0ae6.70920dee.c0a01.2639@mx.google.com>


https://github.com/aengelke closed https://github.com/llvm/llvm-project/pull/206331

From llvmlistbot at llvm.org  Sun Jul  5 02:28:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 02:28:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a4a23bd.30759430.1908d3.e149@mx.google.com>


https://github.com/Qixi-1 updated https://github.com/llvm/llvm-project/pull/207327

>From 35287e592ab6221cc2c1ad0a76870d58ae33f71b Mon Sep 17 00:00:00 2001
From: tongjinxuan <tongjinxuan at longcheer.com>
Date: Fri, 3 Jul 2026 14:08:52 +0800
Subject: [PATCH] [mlir][ArithToAMDGPU] Skip tensor scaling ops

---
 .../Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp  | 10 ++++++++++
 .../ArithToAMDGPU/scaling-truncf-tensor.mlir    | 17 +++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100644 mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir

diff --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index 3372faf4b16cd..cf44c2a0033ac 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -464,6 +464,11 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
@@ -577,6 +582,11 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
diff --git a/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
new file mode 100644
index 0000000000000..d22f35a6d07f1
--- /dev/null
+++ b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
@@ -0,0 +1,17 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+// CHECK-LABEL: func.func @m0
+// CHECK: arith.scaling_truncf
+func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+  %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+  return %0 : tensor<16xf4E2M1FN>
+}
+
+// CHECK-LABEL: func.func @m1
+// CHECK: arith.constant
+// CHECK: arith.scaling_truncf
+func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+  %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+  %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+  return %0 : tensor<4xf4E2M1FN>
+}


From llvmlistbot at llvm.org  Sun Jul  5 06:09:57 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Sun, 05 Jul 2026 06:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4a57a5.cbc3f03a.33a4cc.fdb2@mx.google.com>


joker-eph wrote:

>My logic: If encoding depends on shapes, I think better to fail on verification

I'm not sure I understand the logic... It's never OK to produce invalid IR.



https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Sun Jul  5 06:12:13 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Sun, 05 Jul 2026 06:12:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4a582d.a87eb5dc.1d432d.f7b8@mx.google.com>


joker-eph wrote:

The "propagate if it does not break the verifier" is a bit weird to me: it means that a we would propagate a concat with dynamic shape, even though the metadata would be incorrect, just because a static verifier can't catch it is incorrect.

Also this seems to acknowledge that it is OK to drop the metadata, so It should just always be OK to drop it...

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Sun Jul  5 06:52:41 2026
From: llvmlistbot at llvm.org (Darwin Morris)
Date: Sun, 05 Jul 2026 06:52:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
Message-ID: <llvm.org/llvm/llvm-project/pull/207587@github.com>


https://github.com/darwinmorris created https://github.com/llvm/llvm-project/pull/207587

Fixes #204588

`RemoveDeadValues` assumes that a call operation exposes the same number of results as its callee. This is not true for the `call_and_store` test operation, which consumes callee results internally instead of exposing them as operation results.

This results in an error when the pass computes return value cleanup information, as it does not handle the mismatch. Now this is handled by disabling return-value cleanup when there is a result-count mismatch.

Regression tests were added based on the reproducer. 

>From 4f922bc244704f56c2ca3922c0f21c91a3949319 Mon Sep 17 00:00:00 2001
From: darwinmorris <darwinbmorris98 at gmail.com>
Date: Sun, 5 Jul 2026 15:31:14 +0200
Subject: [PATCH] [mlir] Avoid removing returns for mismatch call results

---
 mlir/lib/Transforms/RemoveDeadValues.cpp     | 14 +++++++++++++-
 mlir/test/Transforms/remove-dead-values.mlir | 19 +++++++++++++++++++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..0786ac2a391a2 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
+      canCleanReturnValues = false;
+      nonLiveRets.reset();
+      break;
+    }
+
     BitVector liveCallRets = markLives(callOp->getResults(), nonLiveSet, la);
     nonLiveRets &= liveCallRets.flip();
   }
 
+  if (!canCleanReturnValues)
+    nonLiveArgs.reset();
+    
   // Note that in the absence of control flow ops forcing the control to go from
   // the entry (first) block to the other blocks, the control never reaches any
   // block other than the entry block, because every block has a terminator.
@@ -358,7 +370,7 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   cl.functions.push_back({funcOp, nonLiveArgs, nonLiveRets});
 
   // Do (5) and (6).
-  if (numReturns == 0)
+  if (!canCleanReturnValues || numReturns == 0)
     return;
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..080743e0539e8 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -868,3 +868,22 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Check that function cleanup is conservative when a call-like op does not
+// expose the same number of results as the callee.
+func.func private @call_and_store_callee(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func private @call_and_store_callee
+  // CHECK-SAME: (%[[ARG0:.*]]: memref<f32>) -> memref<f32>
+  // CHECK: return %[[ARG0]] : memref<f32>
+  return %arg0 : memref<f32>
+}
+
+func.func @call_and_store_before(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func @call_and_store_before
+  // CHECK: test.call_and_store
+  // CHECK: return
+  test.call_and_store @call_and_store_callee(%arg0), %arg0 {store_before_call = true, tag_name = "call"} : (memref<f32>, memref<f32>) -> ()
+  return {tag = "return"} %arg0 : memref<f32>
+}


From llvmlistbot at llvm.org  Sun Jul  5 06:52:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 06:52:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a61b7.491d90f5.1f0d58.0bc0@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @darwinmorris :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 06:53:00 2026
From: llvmlistbot at llvm.org (Darwin Morris)
Date: Sun, 05 Jul 2026 06:53:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a61bc.510f576d.185953.f74a@mx.google.com>


https://github.com/darwinmorris updated https://github.com/llvm/llvm-project/pull/207587

>From 4f922bc244704f56c2ca3922c0f21c91a3949319 Mon Sep 17 00:00:00 2001
From: darwinmorris <darwinbmorris98 at gmail.com>
Date: Sun, 5 Jul 2026 15:31:14 +0200
Subject: [PATCH] [mlir] Avoid removing returns for mismatch call results

---
 mlir/lib/Transforms/RemoveDeadValues.cpp     | 14 +++++++++++++-
 mlir/test/Transforms/remove-dead-values.mlir | 19 +++++++++++++++++++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..0786ac2a391a2 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
+      canCleanReturnValues = false;
+      nonLiveRets.reset();
+      break;
+    }
+
     BitVector liveCallRets = markLives(callOp->getResults(), nonLiveSet, la);
     nonLiveRets &= liveCallRets.flip();
   }
 
+  if (!canCleanReturnValues)
+    nonLiveArgs.reset();
+    
   // Note that in the absence of control flow ops forcing the control to go from
   // the entry (first) block to the other blocks, the control never reaches any
   // block other than the entry block, because every block has a terminator.
@@ -358,7 +370,7 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   cl.functions.push_back({funcOp, nonLiveArgs, nonLiveRets});
 
   // Do (5) and (6).
-  if (numReturns == 0)
+  if (!canCleanReturnValues || numReturns == 0)
     return;
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..080743e0539e8 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -868,3 +868,22 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Check that function cleanup is conservative when a call-like op does not
+// expose the same number of results as the callee.
+func.func private @call_and_store_callee(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func private @call_and_store_callee
+  // CHECK-SAME: (%[[ARG0:.*]]: memref<f32>) -> memref<f32>
+  // CHECK: return %[[ARG0]] : memref<f32>
+  return %arg0 : memref<f32>
+}
+
+func.func @call_and_store_before(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func @call_and_store_before
+  // CHECK: test.call_and_store
+  // CHECK: return
+  test.call_and_store @call_and_store_callee(%arg0), %arg0 {store_before_call = true, tag_name = "call"} : (memref<f32>, memref<f32>) -> ()
+  return {tag = "return"} %arg0 : memref<f32>
+}


From llvmlistbot at llvm.org  Sun Jul  5 06:53:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 06:53:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a61da.58bdd12c.21857.5616@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Darwin Morris (darwinmorris)

<details>
<summary>Changes</summary>

Fixes #<!-- -->204588

`RemoveDeadValues` assumes that a call operation exposes the same number of results as its callee. This is not true for the `call_and_store` test operation, which consumes callee results internally instead of exposing them as operation results.

This results in an error when the pass computes return value cleanup information, as it does not handle the mismatch. Now this is handled by disabling return-value cleanup when there is a result-count mismatch.

Regression tests were added based on the reproducer. 

---
Full diff: https://github.com/llvm/llvm-project/pull/207587.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+13-1) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+19) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..0786ac2a391a2 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
+      canCleanReturnValues = false;
+      nonLiveRets.reset();
+      break;
+    }
+
     BitVector liveCallRets = markLives(callOp->getResults(), nonLiveSet, la);
     nonLiveRets &= liveCallRets.flip();
   }
 
+  if (!canCleanReturnValues)
+    nonLiveArgs.reset();
+    
   // Note that in the absence of control flow ops forcing the control to go from
   // the entry (first) block to the other blocks, the control never reaches any
   // block other than the entry block, because every block has a terminator.
@@ -358,7 +370,7 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   cl.functions.push_back({funcOp, nonLiveArgs, nonLiveRets});
 
   // Do (5) and (6).
-  if (numReturns == 0)
+  if (!canCleanReturnValues || numReturns == 0)
     return;
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..080743e0539e8 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -868,3 +868,22 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Check that function cleanup is conservative when a call-like op does not
+// expose the same number of results as the callee.
+func.func private @call_and_store_callee(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func private @call_and_store_callee
+  // CHECK-SAME: (%[[ARG0:.*]]: memref<f32>) -> memref<f32>
+  // CHECK: return %[[ARG0]] : memref<f32>
+  return %arg0 : memref<f32>
+}
+
+func.func @call_and_store_before(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func @call_and_store_before
+  // CHECK: test.call_and_store
+  // CHECK: return
+  test.call_and_store @call_and_store_callee(%arg0), %arg0 {store_before_call = true, tag_name = "call"} : (memref<f32>, memref<f32>) -> ()
+  return {tag = "return"} %arg0 : memref<f32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 06:53:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 06:53:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a61db.492e23e4.e71ca.142a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Darwin Morris (darwinmorris)

<details>
<summary>Changes</summary>

Fixes #<!-- -->204588

`RemoveDeadValues` assumes that a call operation exposes the same number of results as its callee. This is not true for the `call_and_store` test operation, which consumes callee results internally instead of exposing them as operation results.

This results in an error when the pass computes return value cleanup information, as it does not handle the mismatch. Now this is handled by disabling return-value cleanup when there is a result-count mismatch.

Regression tests were added based on the reproducer. 

---
Full diff: https://github.com/llvm/llvm-project/pull/207587.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+13-1) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+19) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..0786ac2a391a2 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
+      canCleanReturnValues = false;
+      nonLiveRets.reset();
+      break;
+    }
+
     BitVector liveCallRets = markLives(callOp->getResults(), nonLiveSet, la);
     nonLiveRets &= liveCallRets.flip();
   }
 
+  if (!canCleanReturnValues)
+    nonLiveArgs.reset();
+    
   // Note that in the absence of control flow ops forcing the control to go from
   // the entry (first) block to the other blocks, the control never reaches any
   // block other than the entry block, because every block has a terminator.
@@ -358,7 +370,7 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   cl.functions.push_back({funcOp, nonLiveArgs, nonLiveRets});
 
   // Do (5) and (6).
-  if (numReturns == 0)
+  if (!canCleanReturnValues || numReturns == 0)
     return;
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..080743e0539e8 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -868,3 +868,22 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Check that function cleanup is conservative when a call-like op does not
+// expose the same number of results as the callee.
+func.func private @call_and_store_callee(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func private @call_and_store_callee
+  // CHECK-SAME: (%[[ARG0:.*]]: memref<f32>) -> memref<f32>
+  // CHECK: return %[[ARG0]] : memref<f32>
+  return %arg0 : memref<f32>
+}
+
+func.func @call_and_store_before(%arg0: memref<f32>) -> memref<f32> {
+  // CHECK-LABEL: func.func @call_and_store_before
+  // CHECK: test.call_and_store
+  // CHECK: return
+  test.call_and_store @call_and_store_callee(%arg0), %arg0 {store_before_call = true, tag_name = "call"} : (memref<f32>, memref<f32>) -> ()
+  return {tag = "return"} %arg0 : memref<f32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 07:00:53 2026
From: llvmlistbot at llvm.org (Darwin Morris)
Date: Sun, 05 Jul 2026 07:00:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a6395.bd93e731.b068b.252f@mx.google.com>


https://github.com/darwinmorris edited https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 07:02:45 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Sun, 05 Jul 2026 07:02:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a6405.cb30c3e4.6f38b.4bb9@mx.google.com>


================
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
----------------
joker-eph wrote:

This seems like an unsafe heuristic: if the call can consume some value, it can also inject some potentially: the fact that the number matches does not mean you have a 1:1 correspondance with the results. Seems like something that the `CallOpInterface` should provide.

https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 07:02:50 2026
From: llvmlistbot at llvm.org (Darwin Morris)
Date: Sun, 05 Jul 2026 07:02:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a640a.faf16b37.1e0c59.0764@mx.google.com>


darwinmorris wrote:

Read the code-review, developer, and AI policies. Added AI usage to PR description.

https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 08:55:38 2026
From: llvmlistbot at llvm.org (Darwin Morris)
Date: Sun, 05 Jul 2026 08:55:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid removing returns for mismatch
	call results (PR #207587)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207587@github.com>
Message-ID: <6a4a7e7a.eb165e5c.113c2a.4017@mx.google.com>


================
@@ -335,14 +335,26 @@ static void processFuncOp(FunctionOpInterface funcOp, Operation *module,
   // can make our optimization strong by even removing a live return value (%0),
   // since it forwards only to non-live value(s) (%1#1).
   size_t numReturns = funcOp.getNumResults();
+  bool canCleanReturnValues = true;
   BitVector nonLiveRets(numReturns, true);
+
   for (SymbolTable::SymbolUse use : uses) {
     Operation *callOp = use.getUser();
     assert(isa<CallOpInterface>(callOp) && "expected a call-like user");
+
+    if (callOp->getNumResults() != numReturns) {
----------------
darwinmorris wrote:

Good point. I looked through `CallOpInterface` and didn’t find anything that guarantees a 1:1 correspondence between callee results and operation results. I’ll look into adding an explicit interface method for result correspondence (with a conservative default), and then use that in `RemoveDeadValues`, if that seems like the right direction.

https://github.com/llvm/llvm-project/pull/207587

From llvmlistbot at llvm.org  Sun Jul  5 09:23:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 09:23:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
Message-ID: <llvm.org/llvm/llvm-project/pull/207597@github.com>


https://github.com/ES3Q created https://github.com/llvm/llvm-project/pull/207597

The host fallback for `acc.atomic.update` only processed the first operation in the region and used its result for the store, ignoring the remaining operations and the `acc.yield` terminator. This generated invalid IR when the region contained multiple operations.

1.Fix this by cloning all operations in the region and using the operand of `acc.yield` as the final result to store.
2.Add tests for atomic read, write, update, and capture operations to cover the host fallback path.

>From b2e0378372c5dd221c0cfe95c9ed62dba8b292e4 Mon Sep 17 00:00:00 2001
From: ES3Q <ES3Q at QQ.COM>
Date: Mon, 6 Jul 2026 00:08:25 +0800
Subject: [PATCH] [MLIR][OpenACC] Fix host fallback for acc.atomic.update

The host fallback for  incorrectly cloned only the
first operation in the region and used its result for the store. This
generated invalid IR when the region contained multiple operations
leading up to the yield.

Fix this by cloning all operations in the region and using the operand
of  as the final result to store.
---
 .../Transforms/ACCSpecializeForHost.cpp       |  8 ++-
 .../acc-specialize-for-host-fallback.mlir     | 63 +++++++++++++++++++
 2 files changed, 69 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
index d3c8b683b76cb..5f41a1175f21f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
@@ -142,9 +142,13 @@ class ACCOrphanAtomicUpdateOpConversion
       }
       IRMapping mapping;
       mapping.map(atomicUpdateOp.getRegion().front().getArgument(0), loadOp);
-      Operation *expr = rewriter.clone(*atomicUpdateOp.getFirstOp(), mapping);
+      Block &block = atomicUpdateOp.getRegion().front();
+      for (Operation &op : block.without_terminator())
+        rewriter.clone(op, mapping);
+      auto yieldOp = cast<acc::YieldOp>(block.getTerminator());
+      Value result = mapping.lookup(yieldOp.getOperand(0));
       if (!ptrLikeType.genStore(rewriter, atomicUpdateOp.getLoc(),
-                                expr->getResult(0), xTyped)) {
+                                result, xTyped)) {
         accSupport.emitNYI(atomicUpdateOp.getLoc(),
                            "failed to generate store for atomic update");
         return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
index 59269b71bf61c..1a02f9fd710db 100644
--- a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
@@ -155,3 +155,66 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
   acc.declare_exit token(%token) dataOperands(%0 : memref<i32>)
   return
 }
+
+//===----------------------------------------------------------------------===//
+// Atomic operations (host fallback)
+//===----------------------------------------------------------------------===//
+
+acc.routine @acc_routine_atomic_read func(@atomic_read) seq
+// CHECK-LABEL: func.func @atomic_read
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.load
+func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
+  acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_write func(@atomic_write) seq
+// CHECK-LABEL: func.func @atomic_write
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.store
+func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
+  %val = arith.constant 5.0e-01 : f64
+  acc.atomic.write %addr = %val : memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_update func(@atomic_update) seq
+// CHECK-LABEL: func.func @atomic_update
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_update]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.update %x : memref<f64> {
+    ^bb0(%arg0: f64):
+      %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+      %r = arith.select %c, %a, %arg0 : f64
+      acc.yield %r : f64
+    }
+    acc.terminator
+  }
+  return
+}
+
+acc.routine @acc_routine_atomic_capture func(@atomic_capture) seq
+// CHECK-LABEL: func.func @atomic_capture
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_capture(%x : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_capture]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.capture {
+      acc.atomic.update %x : memref<f64> {
+      ^bb0(%arg0: f64):
+        %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+        %r = arith.select %c, %a, %arg0 : f64
+        acc.yield %r : f64
+      }
+      acc.atomic.read %dst = %x : memref<f64>, memref<f64>, f64
+      acc.terminator
+    }
+    acc.terminator
+  }
+  return
+}


From llvmlistbot at llvm.org  Sun Jul  5 09:23:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 09:23:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4a850f.14d9b9dc.140829.5958@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @ES3Q :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Sun Jul  5 09:24:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 09:24:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4a852e.c56fb102.11841f.1afe@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-openacc

@llvm/pr-subscribers-mlir-openacc

Author: ES3Q

<details>
<summary>Changes</summary>

The host fallback for `acc.atomic.update` only processed the first operation in the region and used its result for the store, ignoring the remaining operations and the `acc.yield` terminator. This generated invalid IR when the region contained multiple operations.

1.Fix this by cloning all operations in the region and using the operand of `acc.yield` as the final result to store.
2.Add tests for atomic read, write, update, and capture operations to cover the host fallback path.

---
Full diff: https://github.com/llvm/llvm-project/pull/207597.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp (+6-2) 
- (modified) mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir (+63) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
index d3c8b683b76cb..5f41a1175f21f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
@@ -142,9 +142,13 @@ class ACCOrphanAtomicUpdateOpConversion
       }
       IRMapping mapping;
       mapping.map(atomicUpdateOp.getRegion().front().getArgument(0), loadOp);
-      Operation *expr = rewriter.clone(*atomicUpdateOp.getFirstOp(), mapping);
+      Block &block = atomicUpdateOp.getRegion().front();
+      for (Operation &op : block.without_terminator())
+        rewriter.clone(op, mapping);
+      auto yieldOp = cast<acc::YieldOp>(block.getTerminator());
+      Value result = mapping.lookup(yieldOp.getOperand(0));
       if (!ptrLikeType.genStore(rewriter, atomicUpdateOp.getLoc(),
-                                expr->getResult(0), xTyped)) {
+                                result, xTyped)) {
         accSupport.emitNYI(atomicUpdateOp.getLoc(),
                            "failed to generate store for atomic update");
         return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
index 59269b71bf61c..1a02f9fd710db 100644
--- a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
@@ -155,3 +155,66 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
   acc.declare_exit token(%token) dataOperands(%0 : memref<i32>)
   return
 }
+
+//===----------------------------------------------------------------------===//
+// Atomic operations (host fallback)
+//===----------------------------------------------------------------------===//
+
+acc.routine @acc_routine_atomic_read func(@atomic_read) seq
+// CHECK-LABEL: func.func @atomic_read
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.load
+func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
+  acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_write func(@atomic_write) seq
+// CHECK-LABEL: func.func @atomic_write
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.store
+func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
+  %val = arith.constant 5.0e-01 : f64
+  acc.atomic.write %addr = %val : memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_update func(@atomic_update) seq
+// CHECK-LABEL: func.func @atomic_update
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_update]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.update %x : memref<f64> {
+    ^bb0(%arg0: f64):
+      %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+      %r = arith.select %c, %a, %arg0 : f64
+      acc.yield %r : f64
+    }
+    acc.terminator
+  }
+  return
+}
+
+acc.routine @acc_routine_atomic_capture func(@atomic_capture) seq
+// CHECK-LABEL: func.func @atomic_capture
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_capture(%x : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_capture]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.capture {
+      acc.atomic.update %x : memref<f64> {
+      ^bb0(%arg0: f64):
+        %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+        %r = arith.select %c, %a, %arg0 : f64
+        acc.yield %r : f64
+      }
+      acc.atomic.read %dst = %x : memref<f64>, memref<f64>, f64
+      acc.terminator
+    }
+    acc.terminator
+  }
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Sun Jul  5 12:36:18 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 05 Jul 2026 12:36:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4ab232.7d41f3a1.d1cf7.6866@mx.google.com>


KrxGu wrote:

The comment is a bit hard to follow — it describes sizes in bytes but the allocations are typed (e.g. memref<1024xf32> = 4096 bytes). Would be clearer if the comment just stuck to one unit consistently, or used the actual MLIR types directly.
```
DenseMap<Operation *, int64_t> opIndex;
int64_t idx = 0;
for (Operation &op : *block)
opIndex[&op] = idx++;
// then for each candidate:
info.timeStart = opIndex[candidate.alloc.getOperation()];
info.timeEnd = opIndex[candidate.dealloc.getOperation()];
```

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Sun Jul  5 15:21:25 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Sun, 05 Jul 2026 15:21:25 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
Message-ID: <llvm.org/llvm/llvm-project/pull/207617@github.com>


https://github.com/MaskRay created https://github.com/llvm/llvm-project/pull/207617

LoopInfoBase keeps a DenseMap fallback for block types whose GraphTraits
has no getNumber(). MLIR's CFGLoopInfo is the only such in-tree user
(https://reviews.llvm.org/D147323).

Give mlir::Block a stable number, assigned when it is added to a region,
mirroring llvm::BasicBlock/llvm::Function: Block gains getNumber();
Region gains nextBlockNumber/blockNumberEpoch with getMaxBlockNumber(),
getBlockNumberEpoch() and renumberBlocks(); the block ilist traits
assign the number on add/transfer; and
GraphTraits<mlir::Block*>/<mlir::Region*> expose
getNumber/getMaxNumber/getNumberEpoch. This makes
GraphHasNodeNumbers<mlir::Block*> true (and also moves MLIR's dominator
tree, llvm::DominatorTreeBase<mlir::Block>, to the numbered path, since it
keys on the same trait).

With every in-tree user now numbered, require GraphHasNodeNumbers in
LoopInfoBase (static_assert) and drop the DenseMap fallback.

Aided by Claude Opus 4.8


>From 3aaf489b0f8331aa747d5c3b1533cb1c2e283141 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sun, 5 Jul 2026 14:50:36 -0700
Subject: [PATCH] Number MLIR blocks; require GraphHasNodeNumbers in
 llvm::LoopInfoBase

LoopInfoBase keeps a DenseMap fallback for block types whose GraphTraits
has no getNumber(). MLIR's CFGLoopInfo is the only such in-tree user
(https://reviews.llvm.org/D147323).

Give mlir::Block a stable number, assigned when it is added to a region,
mirroring llvm::BasicBlock/llvm::Function: Block gains getNumber();
Region gains nextBlockNumber/blockNumberEpoch with getMaxBlockNumber(),
getBlockNumberEpoch() and renumberBlocks(); the block ilist traits
assign the number on add/transfer; and
GraphTraits<mlir::Block*>/<mlir::Region*> expose
getNumber/getMaxNumber/getNumberEpoch. This makes
GraphHasNodeNumbers<mlir::Block*> true (and also moves MLIR's dominator
tree, llvm::DominatorTreeBase<mlir::Block>, to the numbered path, since it
keys on the same trait).

With every in-tree user now numbered, require GraphHasNodeNumbers in
LoopInfoBase (static_assert) and drop the DenseMap fallback.

Aided by Claude Opus 4.8
---
 llvm/include/llvm/Support/GenericLoopInfo.h   | 78 +++++++------------
 .../llvm/Support/GenericLoopInfoImpl.h        | 51 +++++-------
 mlir/include/mlir/IR/Block.h                  | 18 +++++
 mlir/include/mlir/IR/Region.h                 | 23 ++++++
 mlir/include/mlir/IR/RegionGraphTraits.h      | 11 +++
 mlir/lib/IR/Region.cpp                        | 18 ++++-
 6 files changed, 114 insertions(+), 85 deletions(-)

diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 9b3bcd09e289d..faebdcb526e0d 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -524,10 +524,13 @@ raw_ostream &operator<<(raw_ostream &OS, const LoopBase<BlockT, LoopT> &Loop) {
 ///
 
 template <class BlockT, class LoopT> class LoopInfoBase {
-  // BBMap - Mapping of basic blocks to the inner most loop they occur in
-  std::conditional_t<GraphHasNodeNumbers<const BlockT *>, SmallVector<LoopT *>,
-                     DenseMap<const BlockT *, LoopT *>>
-      BBMap;
+  static_assert(GraphHasNodeNumbers<const BlockT *>,
+                "LoopInfo requires GraphTraits<BlockT *>::getNumber (see "
+                "GraphHasNodeNumbers)");
+
+  // Mapping of each block, indexed by its number, to the innermost loop it
+  // occurs in (or null).
+  SmallVector<LoopT *> BBMap;
 
   using ParentT = decltype(std::declval<const BlockT *>()->getParent());
   ParentT ParentPtr = nullptr;
@@ -615,25 +618,20 @@ template <class BlockT, class LoopT> class LoopInfoBase {
 private:
   /// Verify that used block numbers are still valid.
   void verifyBlockNumberEpoch(ParentT BBParent) const {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      assert(ParentPtr == BBParent &&
-             "loop info queried with block of other function");
-      assert(BlockNumberEpoch ==
-                 GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
-             "loop info used with outdated block numbers");
-    }
+    assert(ParentPtr == BBParent &&
+           "loop info queried with block of other function");
+    assert(BlockNumberEpoch ==
+               GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
+           "loop info used with outdated block numbers");
   }
 
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
   LoopT *getLoopFor(const BlockT *BB) const {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      return Number < BBMap.size() ? BBMap[Number] : nullptr;
-    } else
-      return BBMap.lookup(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
   /// Same as getLoopFor.
@@ -684,22 +682,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// This should be used by transformations that restructure the loop hierarchy
   /// tree.
   void changeLoopFor(const BlockT *BB, LoopT *L) {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size()) {
-        unsigned Max = GraphTraits<decltype(BB->getParent())>::getMaxNumber(
-            BB->getParent());
-        BBMap.resize(Number >= Max ? Number + 1 : Max);
-      }
-      BBMap[Number] = L;
-    } else {
-      if (!L) {
-        BBMap.erase(BB);
-        return;
-      }
-      BBMap[BB] = L;
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size()) {
+      unsigned Max =
+          GraphTraits<decltype(BB->getParent())>::getMaxNumber(BB->getParent());
+      BBMap.resize(Number >= Max ? Number + 1 : Max);
     }
+    BBMap[Number] = L;
   }
 
   /// Replace the specified loop in the top-level loops list with the indicated
@@ -722,24 +712,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// including all of the Loop objects it is nested in and our mapping from
   /// BasicBlocks to loops.
   void removeBlock(BlockT *BB) {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size())
-        return;
-
-      for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
-        L->removeBlockFromLoop(BB);
-      BBMap[Number] = nullptr;
-    } else {
-      auto I = BBMap.find(BB);
-      if (I != BBMap.end()) {
-        for (LoopT *L = I->second; L; L = L->getParentLoop())
-          L->removeBlockFromLoop(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size())
+      return;
 
-        BBMap.erase(I);
-      }
-    }
+    for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
+      L->removeBlockFromLoop(BB);
+    BBMap[Number] = nullptr;
   }
 
   // Internals
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 4666f4dac9cb6..445dfc05a76f5 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -578,12 +578,9 @@ template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   // Postorder traversal of the dominator tree.
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    ParentPtr = DomRoot->getBlock()->getParent();
-    BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
-    unsigned Max = GraphTraits<ParentT>::getMaxNumber(ParentPtr);
-    BBMap.resize(Max);
-  }
+  ParentPtr = DomRoot->getBlock()->getParent();
+  BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
+  BBMap.resize(GraphTraits<ParentT>::getMaxNumber(ParentPtr));
   for (auto DomNode : post_order(DomRoot)) {
 
     BlockT *Header = DomNode->getBlock();
@@ -755,33 +752,21 @@ void LoopInfoBase<BlockT, LoopT>::verify(
 
 // Verify that blocks are mapped to valid loops.
 #ifndef NDEBUG
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    for (auto It : enumerate(BBMap)) {
-      LoopT *L = It.value();
-      unsigned Number = It.index();
-      if (!L)
-        continue;
-      assert(Loops.count(L) && "orphaned loop");
-      // We have no way to map block numbers back to blocks, so find it.
-      auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
-        return GraphTraits<BlockT *>::getNumber(BB) == Number;
-      });
-      BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
-      assert(BB && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
-  } else {
-    for (auto &Entry : BBMap) {
-      const BlockT *BB = Entry.first;
-      LoopT *L = Entry.second;
-      assert(Loops.count(L) && "orphaned loop");
-      assert(L->contains(BB) && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
+  for (auto It : enumerate(BBMap)) {
+    LoopT *L = It.value();
+    unsigned Number = It.index();
+    if (!L)
+      continue;
+    assert(Loops.count(L) && "orphaned loop");
+    // We have no way to map block numbers back to blocks, so find it.
+    auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
+      return GraphTraits<BlockT *>::getNumber(BB) == Number;
+    });
+    BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
+    assert(BB && "orphaned block");
+    for (LoopT *ChildLoop : *L)
+      assert(!ChildLoop->contains(BB) &&
+             "BBMap should point to the innermost loop containing BB");
   }
 
   // Recompute LoopInfo to verify loops structure.
diff --git a/mlir/include/mlir/IR/Block.h b/mlir/include/mlir/IR/Block.h
index bb3484af2aaf7..5486b6afd5c88 100644
--- a/mlir/include/mlir/IR/Block.h
+++ b/mlir/include/mlir/IR/Block.h
@@ -52,6 +52,19 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// model.
   Region *getParent() const;
 
+  /// Return a number uniquely identifying this block within its parent region.
+  /// The number is assigned when the block is added to a region and is stable
+  /// until the region's blocks are renumbered (see Region::renumberBlocks and
+  /// Region::getBlockNumberEpoch). Only valid for a block that is in a region.
+  ///
+  /// Unlike computeBlockNumber(), this is O(1) and stable; it exists so that
+  /// generic graph algorithms (e.g. LoopInfo, DominatorTree) can index blocks
+  /// by number.
+  unsigned getNumber() const {
+    assert(getParent() && "only blocks in a region have a valid number");
+    return blockNumber;
+  }
+
   /// Returns the closest surrounding operation that contains this block.
   Operation *getParentOp();
 
@@ -422,6 +435,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region. See getNumber().
+  unsigned blockNumber = 0;
+
   /// This is the list of operations in the block.
   OpListType operations;
 
@@ -432,6 +449,7 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   void operator=(Block &) = delete;
 
   friend struct llvm::ilist_traits<Block>;
+  friend class Region;
 };
 
 raw_ostream &operator<<(raw_ostream &, Block &);
diff --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index 13b54991832cb..f379ec1d9f3b6 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -72,6 +72,22 @@ class Region {
     return &Region::blocks;
   }
 
+  //===--------------------------------------------------------------------===//
+  // Block numbering
+  //===--------------------------------------------------------------------===//
+
+  /// One past the largest block number handed out in this region; block numbers
+  /// lie in [0, getMaxBlockNumber()). See Block::getNumber().
+  unsigned getMaxBlockNumber() const { return nextBlockNumber; }
+
+  /// A counter bumped whenever the region's block numbers are reassigned
+  /// (renumberBlocks), so users indexing by block number can detect staleness.
+  unsigned getBlockNumberEpoch() const { return blockNumberEpoch; }
+
+  /// Renumber the blocks in this region 0..N in iteration order, compacting the
+  /// number space and bumping the block-number epoch.
+  void renumberBlocks();
+
   //===--------------------------------------------------------------------===//
   // Argument Handling
   //===--------------------------------------------------------------------===//
@@ -343,6 +359,13 @@ class Region {
 
   /// This is the object we are part of.
   Operation *container = nullptr;
+
+  /// Next block number to hand out, and an epoch bumped when the block numbers
+  /// are reassigned by renumberBlocks(). See Block::getNumber().
+  unsigned nextBlockNumber = 0;
+  unsigned blockNumberEpoch = 0;
+
+  friend struct llvm::ilist_traits<Block>;
 };
 
 /// This class provides an abstraction over the different types of ranges over
diff --git a/mlir/include/mlir/IR/RegionGraphTraits.h b/mlir/include/mlir/IR/RegionGraphTraits.h
index d7d80d9b35806..570f303d370e4 100644
--- a/mlir/include/mlir/IR/RegionGraphTraits.h
+++ b/mlir/include/mlir/IR/RegionGraphTraits.h
@@ -31,6 +31,8 @@ struct GraphTraits<mlir::Block *> {
     return node->succ_begin();
   }
   static ChildIteratorType child_end(NodeRef node) { return node->succ_end(); }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -63,6 +65,8 @@ struct GraphTraits<const mlir::Block *> {
   static ChildIteratorType child_end(NodeRef node) {
     return const_cast<mlir::Block *>(node)->succ_end();
   }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -97,6 +101,13 @@ struct GraphTraits<mlir::Region *> : public GraphTraits<mlir::Block *> {
   static nodes_iterator nodes_end(GraphType fn) {
     return nodes_iterator(fn->end());
   }
+
+  static unsigned getMaxNumber(GraphType region) {
+    return region->getMaxBlockNumber();
+  }
+  static unsigned getNumberEpoch(GraphType region) {
+    return region->getBlockNumberEpoch();
+  }
 };
 
 template <>
diff --git a/mlir/lib/IR/Region.cpp b/mlir/lib/IR/Region.cpp
index 15a941f380225..c8cd0a214204d 100644
--- a/mlir/lib/IR/Region.cpp
+++ b/mlir/lib/IR/Region.cpp
@@ -194,7 +194,9 @@ Region *llvm::ilist_traits<::mlir::Block>::getParentRegion() {
 /// We keep the region pointer up to date.
 void llvm::ilist_traits<::mlir::Block>::addNodeToList(Block *block) {
   assert(!block->getParent() && "already in a region!");
-  block->parentValidOpOrderPair.setPointer(getParentRegion());
+  Region *region = getParentRegion();
+  block->parentValidOpOrderPair.setPointer(region);
+  block->blockNumber = region->nextBlockNumber++;
 }
 
 /// This is a trait method invoked when an operation is removed from a
@@ -214,9 +216,19 @@ void llvm::ilist_traits<::mlir::Block>::transferNodesFromList(
   if (curParent == otherList.getParentRegion())
     return;
 
-  // Update the 'parent' member of each Block.
-  for (; first != last; ++first)
+  // Update the 'parent' member of each Block and give it a number in its new
+  // region.
+  for (; first != last; ++first) {
     first->parentValidOpOrderPair.setPointer(curParent);
+    first->blockNumber = curParent->nextBlockNumber++;
+  }
+}
+
+void Region::renumberBlocks() {
+  nextBlockNumber = 0;
+  for (Block &block : *this)
+    block.blockNumber = nextBlockNumber++;
+  ++blockNumberEpoch;
 }
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Sun Jul  5 15:21:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 15:21:55 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4ad903.a87eb5dc.1d432d.6f7a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-llvm-support

Author: Fangrui Song (MaskRay)

<details>
<summary>Changes</summary>

LoopInfoBase keeps a DenseMap fallback for block types whose GraphTraits
has no getNumber(). MLIR's CFGLoopInfo is the only such in-tree user
(https://reviews.llvm.org/D147323).

Give mlir::Block a stable number, assigned when it is added to a region,
mirroring llvm::BasicBlock/llvm::Function: Block gains getNumber();
Region gains nextBlockNumber/blockNumberEpoch with getMaxBlockNumber(),
getBlockNumberEpoch() and renumberBlocks(); the block ilist traits
assign the number on add/transfer; and
GraphTraits&lt;mlir::Block*&gt;/&lt;mlir::Region*&gt; expose
getNumber/getMaxNumber/getNumberEpoch. This makes
GraphHasNodeNumbers&lt;mlir::Block*&gt; true (and also moves MLIR's dominator
tree, llvm::DominatorTreeBase&lt;mlir::Block&gt;, to the numbered path, since it
keys on the same trait).

With every in-tree user now numbered, require GraphHasNodeNumbers in
LoopInfoBase (static_assert) and drop the DenseMap fallback.

Aided by Claude Opus 4.8


---
Full diff: https://github.com/llvm/llvm-project/pull/207617.diff


6 Files Affected:

- (modified) llvm/include/llvm/Support/GenericLoopInfo.h (+29-49) 
- (modified) llvm/include/llvm/Support/GenericLoopInfoImpl.h (+18-33) 
- (modified) mlir/include/mlir/IR/Block.h (+18) 
- (modified) mlir/include/mlir/IR/Region.h (+23) 
- (modified) mlir/include/mlir/IR/RegionGraphTraits.h (+11) 
- (modified) mlir/lib/IR/Region.cpp (+15-3) 


``````````diff
diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 9b3bcd09e289d..faebdcb526e0d 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -524,10 +524,13 @@ raw_ostream &operator<<(raw_ostream &OS, const LoopBase<BlockT, LoopT> &Loop) {
 ///
 
 template <class BlockT, class LoopT> class LoopInfoBase {
-  // BBMap - Mapping of basic blocks to the inner most loop they occur in
-  std::conditional_t<GraphHasNodeNumbers<const BlockT *>, SmallVector<LoopT *>,
-                     DenseMap<const BlockT *, LoopT *>>
-      BBMap;
+  static_assert(GraphHasNodeNumbers<const BlockT *>,
+                "LoopInfo requires GraphTraits<BlockT *>::getNumber (see "
+                "GraphHasNodeNumbers)");
+
+  // Mapping of each block, indexed by its number, to the innermost loop it
+  // occurs in (or null).
+  SmallVector<LoopT *> BBMap;
 
   using ParentT = decltype(std::declval<const BlockT *>()->getParent());
   ParentT ParentPtr = nullptr;
@@ -615,25 +618,20 @@ template <class BlockT, class LoopT> class LoopInfoBase {
 private:
   /// Verify that used block numbers are still valid.
   void verifyBlockNumberEpoch(ParentT BBParent) const {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      assert(ParentPtr == BBParent &&
-             "loop info queried with block of other function");
-      assert(BlockNumberEpoch ==
-                 GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
-             "loop info used with outdated block numbers");
-    }
+    assert(ParentPtr == BBParent &&
+           "loop info queried with block of other function");
+    assert(BlockNumberEpoch ==
+               GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
+           "loop info used with outdated block numbers");
   }
 
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
   LoopT *getLoopFor(const BlockT *BB) const {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      return Number < BBMap.size() ? BBMap[Number] : nullptr;
-    } else
-      return BBMap.lookup(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
   /// Same as getLoopFor.
@@ -684,22 +682,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// This should be used by transformations that restructure the loop hierarchy
   /// tree.
   void changeLoopFor(const BlockT *BB, LoopT *L) {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size()) {
-        unsigned Max = GraphTraits<decltype(BB->getParent())>::getMaxNumber(
-            BB->getParent());
-        BBMap.resize(Number >= Max ? Number + 1 : Max);
-      }
-      BBMap[Number] = L;
-    } else {
-      if (!L) {
-        BBMap.erase(BB);
-        return;
-      }
-      BBMap[BB] = L;
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size()) {
+      unsigned Max =
+          GraphTraits<decltype(BB->getParent())>::getMaxNumber(BB->getParent());
+      BBMap.resize(Number >= Max ? Number + 1 : Max);
     }
+    BBMap[Number] = L;
   }
 
   /// Replace the specified loop in the top-level loops list with the indicated
@@ -722,24 +712,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// including all of the Loop objects it is nested in and our mapping from
   /// BasicBlocks to loops.
   void removeBlock(BlockT *BB) {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size())
-        return;
-
-      for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
-        L->removeBlockFromLoop(BB);
-      BBMap[Number] = nullptr;
-    } else {
-      auto I = BBMap.find(BB);
-      if (I != BBMap.end()) {
-        for (LoopT *L = I->second; L; L = L->getParentLoop())
-          L->removeBlockFromLoop(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size())
+      return;
 
-        BBMap.erase(I);
-      }
-    }
+    for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
+      L->removeBlockFromLoop(BB);
+    BBMap[Number] = nullptr;
   }
 
   // Internals
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 4666f4dac9cb6..445dfc05a76f5 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -578,12 +578,9 @@ template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   // Postorder traversal of the dominator tree.
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    ParentPtr = DomRoot->getBlock()->getParent();
-    BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
-    unsigned Max = GraphTraits<ParentT>::getMaxNumber(ParentPtr);
-    BBMap.resize(Max);
-  }
+  ParentPtr = DomRoot->getBlock()->getParent();
+  BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
+  BBMap.resize(GraphTraits<ParentT>::getMaxNumber(ParentPtr));
   for (auto DomNode : post_order(DomRoot)) {
 
     BlockT *Header = DomNode->getBlock();
@@ -755,33 +752,21 @@ void LoopInfoBase<BlockT, LoopT>::verify(
 
 // Verify that blocks are mapped to valid loops.
 #ifndef NDEBUG
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    for (auto It : enumerate(BBMap)) {
-      LoopT *L = It.value();
-      unsigned Number = It.index();
-      if (!L)
-        continue;
-      assert(Loops.count(L) && "orphaned loop");
-      // We have no way to map block numbers back to blocks, so find it.
-      auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
-        return GraphTraits<BlockT *>::getNumber(BB) == Number;
-      });
-      BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
-      assert(BB && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
-  } else {
-    for (auto &Entry : BBMap) {
-      const BlockT *BB = Entry.first;
-      LoopT *L = Entry.second;
-      assert(Loops.count(L) && "orphaned loop");
-      assert(L->contains(BB) && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
+  for (auto It : enumerate(BBMap)) {
+    LoopT *L = It.value();
+    unsigned Number = It.index();
+    if (!L)
+      continue;
+    assert(Loops.count(L) && "orphaned loop");
+    // We have no way to map block numbers back to blocks, so find it.
+    auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
+      return GraphTraits<BlockT *>::getNumber(BB) == Number;
+    });
+    BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
+    assert(BB && "orphaned block");
+    for (LoopT *ChildLoop : *L)
+      assert(!ChildLoop->contains(BB) &&
+             "BBMap should point to the innermost loop containing BB");
   }
 
   // Recompute LoopInfo to verify loops structure.
diff --git a/mlir/include/mlir/IR/Block.h b/mlir/include/mlir/IR/Block.h
index bb3484af2aaf7..5486b6afd5c88 100644
--- a/mlir/include/mlir/IR/Block.h
+++ b/mlir/include/mlir/IR/Block.h
@@ -52,6 +52,19 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// model.
   Region *getParent() const;
 
+  /// Return a number uniquely identifying this block within its parent region.
+  /// The number is assigned when the block is added to a region and is stable
+  /// until the region's blocks are renumbered (see Region::renumberBlocks and
+  /// Region::getBlockNumberEpoch). Only valid for a block that is in a region.
+  ///
+  /// Unlike computeBlockNumber(), this is O(1) and stable; it exists so that
+  /// generic graph algorithms (e.g. LoopInfo, DominatorTree) can index blocks
+  /// by number.
+  unsigned getNumber() const {
+    assert(getParent() && "only blocks in a region have a valid number");
+    return blockNumber;
+  }
+
   /// Returns the closest surrounding operation that contains this block.
   Operation *getParentOp();
 
@@ -422,6 +435,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region. See getNumber().
+  unsigned blockNumber = 0;
+
   /// This is the list of operations in the block.
   OpListType operations;
 
@@ -432,6 +449,7 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   void operator=(Block &) = delete;
 
   friend struct llvm::ilist_traits<Block>;
+  friend class Region;
 };
 
 raw_ostream &operator<<(raw_ostream &, Block &);
diff --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index 13b54991832cb..f379ec1d9f3b6 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -72,6 +72,22 @@ class Region {
     return &Region::blocks;
   }
 
+  //===--------------------------------------------------------------------===//
+  // Block numbering
+  //===--------------------------------------------------------------------===//
+
+  /// One past the largest block number handed out in this region; block numbers
+  /// lie in [0, getMaxBlockNumber()). See Block::getNumber().
+  unsigned getMaxBlockNumber() const { return nextBlockNumber; }
+
+  /// A counter bumped whenever the region's block numbers are reassigned
+  /// (renumberBlocks), so users indexing by block number can detect staleness.
+  unsigned getBlockNumberEpoch() const { return blockNumberEpoch; }
+
+  /// Renumber the blocks in this region 0..N in iteration order, compacting the
+  /// number space and bumping the block-number epoch.
+  void renumberBlocks();
+
   //===--------------------------------------------------------------------===//
   // Argument Handling
   //===--------------------------------------------------------------------===//
@@ -343,6 +359,13 @@ class Region {
 
   /// This is the object we are part of.
   Operation *container = nullptr;
+
+  /// Next block number to hand out, and an epoch bumped when the block numbers
+  /// are reassigned by renumberBlocks(). See Block::getNumber().
+  unsigned nextBlockNumber = 0;
+  unsigned blockNumberEpoch = 0;
+
+  friend struct llvm::ilist_traits<Block>;
 };
 
 /// This class provides an abstraction over the different types of ranges over
diff --git a/mlir/include/mlir/IR/RegionGraphTraits.h b/mlir/include/mlir/IR/RegionGraphTraits.h
index d7d80d9b35806..570f303d370e4 100644
--- a/mlir/include/mlir/IR/RegionGraphTraits.h
+++ b/mlir/include/mlir/IR/RegionGraphTraits.h
@@ -31,6 +31,8 @@ struct GraphTraits<mlir::Block *> {
     return node->succ_begin();
   }
   static ChildIteratorType child_end(NodeRef node) { return node->succ_end(); }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -63,6 +65,8 @@ struct GraphTraits<const mlir::Block *> {
   static ChildIteratorType child_end(NodeRef node) {
     return const_cast<mlir::Block *>(node)->succ_end();
   }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -97,6 +101,13 @@ struct GraphTraits<mlir::Region *> : public GraphTraits<mlir::Block *> {
   static nodes_iterator nodes_end(GraphType fn) {
     return nodes_iterator(fn->end());
   }
+
+  static unsigned getMaxNumber(GraphType region) {
+    return region->getMaxBlockNumber();
+  }
+  static unsigned getNumberEpoch(GraphType region) {
+    return region->getBlockNumberEpoch();
+  }
 };
 
 template <>
diff --git a/mlir/lib/IR/Region.cpp b/mlir/lib/IR/Region.cpp
index 15a941f380225..c8cd0a214204d 100644
--- a/mlir/lib/IR/Region.cpp
+++ b/mlir/lib/IR/Region.cpp
@@ -194,7 +194,9 @@ Region *llvm::ilist_traits<::mlir::Block>::getParentRegion() {
 /// We keep the region pointer up to date.
 void llvm::ilist_traits<::mlir::Block>::addNodeToList(Block *block) {
   assert(!block->getParent() && "already in a region!");
-  block->parentValidOpOrderPair.setPointer(getParentRegion());
+  Region *region = getParentRegion();
+  block->parentValidOpOrderPair.setPointer(region);
+  block->blockNumber = region->nextBlockNumber++;
 }
 
 /// This is a trait method invoked when an operation is removed from a
@@ -214,9 +216,19 @@ void llvm::ilist_traits<::mlir::Block>::transferNodesFromList(
   if (curParent == otherList.getParentRegion())
     return;
 
-  // Update the 'parent' member of each Block.
-  for (; first != last; ++first)
+  // Update the 'parent' member of each Block and give it a number in its new
+  // region.
+  for (; first != last; ++first) {
     first->parentValidOpOrderPair.setPointer(curParent);
+    first->blockNumber = curParent->nextBlockNumber++;
+  }
+}
+
+void Region::renumberBlocks() {
+  nextBlockNumber = 0;
+  for (Block &block : *this)
+    block.blockNumber = nextBlockNumber++;
+  ++blockNumberEpoch;
 }
 
 //===----------------------------------------------------------------------===//

``````````

</details>


https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Sun Jul  5 16:33:23 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Sun, 05 Jul 2026 16:33:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4ae9c3.d5797644.146403.7102@mx.google.com>


https://github.com/MaskRay updated https://github.com/llvm/llvm-project/pull/207617

>From 3aaf489b0f8331aa747d5c3b1533cb1c2e283141 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sun, 5 Jul 2026 14:50:36 -0700
Subject: [PATCH 1/2] Number MLIR blocks; require GraphHasNodeNumbers in
 llvm::LoopInfoBase

LoopInfoBase keeps a DenseMap fallback for block types whose GraphTraits
has no getNumber(). MLIR's CFGLoopInfo is the only such in-tree user
(https://reviews.llvm.org/D147323).

Give mlir::Block a stable number, assigned when it is added to a region,
mirroring llvm::BasicBlock/llvm::Function: Block gains getNumber();
Region gains nextBlockNumber/blockNumberEpoch with getMaxBlockNumber(),
getBlockNumberEpoch() and renumberBlocks(); the block ilist traits
assign the number on add/transfer; and
GraphTraits<mlir::Block*>/<mlir::Region*> expose
getNumber/getMaxNumber/getNumberEpoch. This makes
GraphHasNodeNumbers<mlir::Block*> true (and also moves MLIR's dominator
tree, llvm::DominatorTreeBase<mlir::Block>, to the numbered path, since it
keys on the same trait).

With every in-tree user now numbered, require GraphHasNodeNumbers in
LoopInfoBase (static_assert) and drop the DenseMap fallback.

Aided by Claude Opus 4.8
---
 llvm/include/llvm/Support/GenericLoopInfo.h   | 78 +++++++------------
 .../llvm/Support/GenericLoopInfoImpl.h        | 51 +++++-------
 mlir/include/mlir/IR/Block.h                  | 18 +++++
 mlir/include/mlir/IR/Region.h                 | 23 ++++++
 mlir/include/mlir/IR/RegionGraphTraits.h      | 11 +++
 mlir/lib/IR/Region.cpp                        | 18 ++++-
 6 files changed, 114 insertions(+), 85 deletions(-)

diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 9b3bcd09e289d..faebdcb526e0d 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -524,10 +524,13 @@ raw_ostream &operator<<(raw_ostream &OS, const LoopBase<BlockT, LoopT> &Loop) {
 ///
 
 template <class BlockT, class LoopT> class LoopInfoBase {
-  // BBMap - Mapping of basic blocks to the inner most loop they occur in
-  std::conditional_t<GraphHasNodeNumbers<const BlockT *>, SmallVector<LoopT *>,
-                     DenseMap<const BlockT *, LoopT *>>
-      BBMap;
+  static_assert(GraphHasNodeNumbers<const BlockT *>,
+                "LoopInfo requires GraphTraits<BlockT *>::getNumber (see "
+                "GraphHasNodeNumbers)");
+
+  // Mapping of each block, indexed by its number, to the innermost loop it
+  // occurs in (or null).
+  SmallVector<LoopT *> BBMap;
 
   using ParentT = decltype(std::declval<const BlockT *>()->getParent());
   ParentT ParentPtr = nullptr;
@@ -615,25 +618,20 @@ template <class BlockT, class LoopT> class LoopInfoBase {
 private:
   /// Verify that used block numbers are still valid.
   void verifyBlockNumberEpoch(ParentT BBParent) const {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      assert(ParentPtr == BBParent &&
-             "loop info queried with block of other function");
-      assert(BlockNumberEpoch ==
-                 GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
-             "loop info used with outdated block numbers");
-    }
+    assert(ParentPtr == BBParent &&
+           "loop info queried with block of other function");
+    assert(BlockNumberEpoch ==
+               GraphTraits<ParentT>::getNumberEpoch(ParentPtr) &&
+           "loop info used with outdated block numbers");
   }
 
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
   LoopT *getLoopFor(const BlockT *BB) const {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      return Number < BBMap.size() ? BBMap[Number] : nullptr;
-    } else
-      return BBMap.lookup(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
   /// Same as getLoopFor.
@@ -684,22 +682,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// This should be used by transformations that restructure the loop hierarchy
   /// tree.
   void changeLoopFor(const BlockT *BB, LoopT *L) {
-    if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size()) {
-        unsigned Max = GraphTraits<decltype(BB->getParent())>::getMaxNumber(
-            BB->getParent());
-        BBMap.resize(Number >= Max ? Number + 1 : Max);
-      }
-      BBMap[Number] = L;
-    } else {
-      if (!L) {
-        BBMap.erase(BB);
-        return;
-      }
-      BBMap[BB] = L;
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<const BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size()) {
+      unsigned Max =
+          GraphTraits<decltype(BB->getParent())>::getMaxNumber(BB->getParent());
+      BBMap.resize(Number >= Max ? Number + 1 : Max);
     }
+    BBMap[Number] = L;
   }
 
   /// Replace the specified loop in the top-level loops list with the indicated
@@ -722,24 +712,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// including all of the Loop objects it is nested in and our mapping from
   /// BasicBlocks to loops.
   void removeBlock(BlockT *BB) {
-    if constexpr (GraphHasNodeNumbers<BlockT *>) {
-      verifyBlockNumberEpoch(BB->getParent());
-      unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
-      if (Number >= BBMap.size())
-        return;
-
-      for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
-        L->removeBlockFromLoop(BB);
-      BBMap[Number] = nullptr;
-    } else {
-      auto I = BBMap.find(BB);
-      if (I != BBMap.end()) {
-        for (LoopT *L = I->second; L; L = L->getParentLoop())
-          L->removeBlockFromLoop(BB);
+    verifyBlockNumberEpoch(BB->getParent());
+    unsigned Number = GraphTraits<BlockT *>::getNumber(BB);
+    if (Number >= BBMap.size())
+      return;
 
-        BBMap.erase(I);
-      }
-    }
+    for (LoopT *L = BBMap[Number]; L; L = L->getParentLoop())
+      L->removeBlockFromLoop(BB);
+    BBMap[Number] = nullptr;
   }
 
   // Internals
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 4666f4dac9cb6..445dfc05a76f5 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -578,12 +578,9 @@ template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   // Postorder traversal of the dominator tree.
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    ParentPtr = DomRoot->getBlock()->getParent();
-    BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
-    unsigned Max = GraphTraits<ParentT>::getMaxNumber(ParentPtr);
-    BBMap.resize(Max);
-  }
+  ParentPtr = DomRoot->getBlock()->getParent();
+  BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
+  BBMap.resize(GraphTraits<ParentT>::getMaxNumber(ParentPtr));
   for (auto DomNode : post_order(DomRoot)) {
 
     BlockT *Header = DomNode->getBlock();
@@ -755,33 +752,21 @@ void LoopInfoBase<BlockT, LoopT>::verify(
 
 // Verify that blocks are mapped to valid loops.
 #ifndef NDEBUG
-  if constexpr (GraphHasNodeNumbers<const BlockT *>) {
-    for (auto It : enumerate(BBMap)) {
-      LoopT *L = It.value();
-      unsigned Number = It.index();
-      if (!L)
-        continue;
-      assert(Loops.count(L) && "orphaned loop");
-      // We have no way to map block numbers back to blocks, so find it.
-      auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
-        return GraphTraits<BlockT *>::getNumber(BB) == Number;
-      });
-      BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
-      assert(BB && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
-  } else {
-    for (auto &Entry : BBMap) {
-      const BlockT *BB = Entry.first;
-      LoopT *L = Entry.second;
-      assert(Loops.count(L) && "orphaned loop");
-      assert(L->contains(BB) && "orphaned block");
-      for (LoopT *ChildLoop : *L)
-        assert(!ChildLoop->contains(BB) &&
-               "BBMap should point to the innermost loop containing BB");
-    }
+  for (auto It : enumerate(BBMap)) {
+    LoopT *L = It.value();
+    unsigned Number = It.index();
+    if (!L)
+      continue;
+    assert(Loops.count(L) && "orphaned loop");
+    // We have no way to map block numbers back to blocks, so find it.
+    auto BBIt = find_if(L->Blocks, [&Number](BlockT *BB) {
+      return GraphTraits<BlockT *>::getNumber(BB) == Number;
+    });
+    BlockT *BB = BBIt != L->Blocks.end() ? *BBIt : nullptr;
+    assert(BB && "orphaned block");
+    for (LoopT *ChildLoop : *L)
+      assert(!ChildLoop->contains(BB) &&
+             "BBMap should point to the innermost loop containing BB");
   }
 
   // Recompute LoopInfo to verify loops structure.
diff --git a/mlir/include/mlir/IR/Block.h b/mlir/include/mlir/IR/Block.h
index bb3484af2aaf7..5486b6afd5c88 100644
--- a/mlir/include/mlir/IR/Block.h
+++ b/mlir/include/mlir/IR/Block.h
@@ -52,6 +52,19 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// model.
   Region *getParent() const;
 
+  /// Return a number uniquely identifying this block within its parent region.
+  /// The number is assigned when the block is added to a region and is stable
+  /// until the region's blocks are renumbered (see Region::renumberBlocks and
+  /// Region::getBlockNumberEpoch). Only valid for a block that is in a region.
+  ///
+  /// Unlike computeBlockNumber(), this is O(1) and stable; it exists so that
+  /// generic graph algorithms (e.g. LoopInfo, DominatorTree) can index blocks
+  /// by number.
+  unsigned getNumber() const {
+    assert(getParent() && "only blocks in a region have a valid number");
+    return blockNumber;
+  }
+
   /// Returns the closest surrounding operation that contains this block.
   Operation *getParentOp();
 
@@ -422,6 +435,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region. See getNumber().
+  unsigned blockNumber = 0;
+
   /// This is the list of operations in the block.
   OpListType operations;
 
@@ -432,6 +449,7 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   void operator=(Block &) = delete;
 
   friend struct llvm::ilist_traits<Block>;
+  friend class Region;
 };
 
 raw_ostream &operator<<(raw_ostream &, Block &);
diff --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index 13b54991832cb..f379ec1d9f3b6 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -72,6 +72,22 @@ class Region {
     return &Region::blocks;
   }
 
+  //===--------------------------------------------------------------------===//
+  // Block numbering
+  //===--------------------------------------------------------------------===//
+
+  /// One past the largest block number handed out in this region; block numbers
+  /// lie in [0, getMaxBlockNumber()). See Block::getNumber().
+  unsigned getMaxBlockNumber() const { return nextBlockNumber; }
+
+  /// A counter bumped whenever the region's block numbers are reassigned
+  /// (renumberBlocks), so users indexing by block number can detect staleness.
+  unsigned getBlockNumberEpoch() const { return blockNumberEpoch; }
+
+  /// Renumber the blocks in this region 0..N in iteration order, compacting the
+  /// number space and bumping the block-number epoch.
+  void renumberBlocks();
+
   //===--------------------------------------------------------------------===//
   // Argument Handling
   //===--------------------------------------------------------------------===//
@@ -343,6 +359,13 @@ class Region {
 
   /// This is the object we are part of.
   Operation *container = nullptr;
+
+  /// Next block number to hand out, and an epoch bumped when the block numbers
+  /// are reassigned by renumberBlocks(). See Block::getNumber().
+  unsigned nextBlockNumber = 0;
+  unsigned blockNumberEpoch = 0;
+
+  friend struct llvm::ilist_traits<Block>;
 };
 
 /// This class provides an abstraction over the different types of ranges over
diff --git a/mlir/include/mlir/IR/RegionGraphTraits.h b/mlir/include/mlir/IR/RegionGraphTraits.h
index d7d80d9b35806..570f303d370e4 100644
--- a/mlir/include/mlir/IR/RegionGraphTraits.h
+++ b/mlir/include/mlir/IR/RegionGraphTraits.h
@@ -31,6 +31,8 @@ struct GraphTraits<mlir::Block *> {
     return node->succ_begin();
   }
   static ChildIteratorType child_end(NodeRef node) { return node->succ_end(); }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -63,6 +65,8 @@ struct GraphTraits<const mlir::Block *> {
   static ChildIteratorType child_end(NodeRef node) {
     return const_cast<mlir::Block *>(node)->succ_end();
   }
+
+  static unsigned getNumber(NodeRef node) { return node->getNumber(); }
 };
 
 template <>
@@ -97,6 +101,13 @@ struct GraphTraits<mlir::Region *> : public GraphTraits<mlir::Block *> {
   static nodes_iterator nodes_end(GraphType fn) {
     return nodes_iterator(fn->end());
   }
+
+  static unsigned getMaxNumber(GraphType region) {
+    return region->getMaxBlockNumber();
+  }
+  static unsigned getNumberEpoch(GraphType region) {
+    return region->getBlockNumberEpoch();
+  }
 };
 
 template <>
diff --git a/mlir/lib/IR/Region.cpp b/mlir/lib/IR/Region.cpp
index 15a941f380225..c8cd0a214204d 100644
--- a/mlir/lib/IR/Region.cpp
+++ b/mlir/lib/IR/Region.cpp
@@ -194,7 +194,9 @@ Region *llvm::ilist_traits<::mlir::Block>::getParentRegion() {
 /// We keep the region pointer up to date.
 void llvm::ilist_traits<::mlir::Block>::addNodeToList(Block *block) {
   assert(!block->getParent() && "already in a region!");
-  block->parentValidOpOrderPair.setPointer(getParentRegion());
+  Region *region = getParentRegion();
+  block->parentValidOpOrderPair.setPointer(region);
+  block->blockNumber = region->nextBlockNumber++;
 }
 
 /// This is a trait method invoked when an operation is removed from a
@@ -214,9 +216,19 @@ void llvm::ilist_traits<::mlir::Block>::transferNodesFromList(
   if (curParent == otherList.getParentRegion())
     return;
 
-  // Update the 'parent' member of each Block.
-  for (; first != last; ++first)
+  // Update the 'parent' member of each Block and give it a number in its new
+  // region.
+  for (; first != last; ++first) {
     first->parentValidOpOrderPair.setPointer(curParent);
+    first->blockNumber = curParent->nextBlockNumber++;
+  }
+}
+
+void Region::renumberBlocks() {
+  nextBlockNumber = 0;
+  for (Block &block : *this)
+    block.blockNumber = nextBlockNumber++;
+  ++blockNumberEpoch;
 }
 
 //===----------------------------------------------------------------------===//

>From 65e6755a68bececd5b2ecb68b1cd6149cf3747b7 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sun, 5 Jul 2026 16:33:11 -0700
Subject: [PATCH 2/2] remove unused blockNumberEpoch

---
 mlir/include/mlir/IR/Block.h  |  6 +++---
 mlir/include/mlir/IR/Region.h | 15 +++++----------
 mlir/lib/IR/Region.cpp        |  7 -------
 3 files changed, 8 insertions(+), 20 deletions(-)

diff --git a/mlir/include/mlir/IR/Block.h b/mlir/include/mlir/IR/Block.h
index 5486b6afd5c88..34e38a2359f05 100644
--- a/mlir/include/mlir/IR/Block.h
+++ b/mlir/include/mlir/IR/Block.h
@@ -53,9 +53,9 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   Region *getParent() const;
 
   /// Return a number uniquely identifying this block within its parent region.
-  /// The number is assigned when the block is added to a region and is stable
-  /// until the region's blocks are renumbered (see Region::renumberBlocks and
-  /// Region::getBlockNumberEpoch). Only valid for a block that is in a region.
+  /// The number is assigned when the block is added to a region and never
+  /// changes while the block stays in that region; removing a block leaves a
+  /// hole (numbers are not reused). Only valid for a block that is in a region.
   ///
   /// Unlike computeBlockNumber(), this is O(1) and stable; it exists so that
   /// generic graph algorithms (e.g. LoopInfo, DominatorTree) can index blocks
diff --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index f379ec1d9f3b6..e1c44eb55e640 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -80,13 +80,10 @@ class Region {
   /// lie in [0, getMaxBlockNumber()). See Block::getNumber().
   unsigned getMaxBlockNumber() const { return nextBlockNumber; }
 
-  /// A counter bumped whenever the region's block numbers are reassigned
-  /// (renumberBlocks), so users indexing by block number can detect staleness.
-  unsigned getBlockNumberEpoch() const { return blockNumberEpoch; }
-
-  /// Renumber the blocks in this region 0..N in iteration order, compacting the
-  /// number space and bumping the block-number epoch.
-  void renumberBlocks();
+  /// The block-number epoch, part of the generic number-indexed graph contract
+  /// (LoopInfo, DominatorTree) for detecting stale numbering. MLIR never
+  /// renumbers blocks, so this is a fixed 0. See Block::getNumber().
+  unsigned getBlockNumberEpoch() const { return 0; }
 
   //===--------------------------------------------------------------------===//
   // Argument Handling
@@ -360,10 +357,8 @@ class Region {
   /// This is the object we are part of.
   Operation *container = nullptr;
 
-  /// Next block number to hand out, and an epoch bumped when the block numbers
-  /// are reassigned by renumberBlocks(). See Block::getNumber().
+  /// Next block number to hand out. See Block::getNumber().
   unsigned nextBlockNumber = 0;
-  unsigned blockNumberEpoch = 0;
 
   friend struct llvm::ilist_traits<Block>;
 };
diff --git a/mlir/lib/IR/Region.cpp b/mlir/lib/IR/Region.cpp
index c8cd0a214204d..71fb893c13fe4 100644
--- a/mlir/lib/IR/Region.cpp
+++ b/mlir/lib/IR/Region.cpp
@@ -224,13 +224,6 @@ void llvm::ilist_traits<::mlir::Block>::transferNodesFromList(
   }
 }
 
-void Region::renumberBlocks() {
-  nextBlockNumber = 0;
-  for (Block &block : *this)
-    block.blockNumber = nextBlockNumber++;
-  ++blockNumberEpoch;
-}
-
 //===----------------------------------------------------------------------===//
 // Region::OpIterator
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Sun Jul  5 16:43:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 16:43:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] bb639a1 - [MLIR][NVVM]Import llvm intrinsics
 for nvvm.barrier (#202862)
Message-ID: <6a4aec3c.7d41f3a1.d1cf7.370b@mx.google.com>


Author: xys-syx
Date: 2026-07-05T19:43:51-04:00
New Revision: bb639a128559172e8b3067ea267b8f37895ae1f8

URL: https://github.com/llvm/llvm-project/commit/bb639a128559172e8b3067ea267b8f37895ae1f8
DIFF: https://github.com/llvm/llvm-project/commit/bb639a128559172e8b3067ea267b8f37895ae1f8.diff

LOG: [MLIR][NVVM]Import llvm intrinsics for nvvm.barrier (#202862)

This PR adds LLVM IR import support for the `bar.sync` intrinsic family,
lifting the four variants into `nvvm.barrier` with the appropriate
`aligned` attribute and optional thread-count operand.

`LLVM_IntrOpBase` and its auto-generated import is one to one, but four
`bar.sync` variants are many-to-one, where they all lift into
`nvvm.barrier`, differing only in `aligned` and the optional
`numberOfThreads` operand.

An alternative implementation would extend mlir-tblgen with many-to-one
intrinsic-to-op mapping support so the conversion could live in
`NVVMOps.td`. But the change is bigger than manually import `bar.sync`
intrinsic family.

Added: 
    

Modified: 
    mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
    mlir/test/Target/LLVMIR/Import/nvvmir.ll

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
index 55e73e839afcb..f15965e01761a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.cpp
@@ -18,21 +18,68 @@
 using namespace mlir;
 using namespace mlir::NVVM;
 
-/// Returns true if the LLVM IR intrinsic is convertible to an MLIR NVVM dialect
-/// intrinsic. Returns false otherwise.
+/// LLVM intrinsic IDs of the four `bar.sync` variants imported into
+/// `nvvm.barrier`.
+static constexpr llvm::Intrinsic::ID kBarrierSyncIntrinsics[] = {
+    llvm::Intrinsic::nvvm_barrier_cta_sync_all,
+    llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_all,
+    llvm::Intrinsic::nvvm_barrier_cta_sync_count,
+    llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count,
+};
+
+/// Returns true if `id` is one of the `bar.sync` intrinsic IDs imported into
+/// `nvvm.barrier`.
+static bool isBarrierSyncIntrinsic(llvm::Intrinsic::ID id) {
+  return llvm::is_contained(kBarrierSyncIntrinsics, id);
+}
+
+/// Imports one of the four `bar.sync` LLVM intrinsic variants into a single
+/// `nvvm.barrier` op, deriving the `aligned` attribute and the optional
+/// `numberOfThreads` operand from the specific intrinsic ID.
+static LogicalResult
+convertBarrierSyncIntrinsic(OpBuilder &odsBuilder, llvm::CallInst *inst,
+                            LLVM::ModuleImport &moduleImport,
+                            ArrayRef<llvm::Value *> llvmOperands,
+                            ArrayRef<llvm::OperandBundleUse> llvmOpBundles) {
+  llvm::Intrinsic::ID id = inst->getIntrinsicID();
+  bool aligned = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_all ||
+                  id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+  bool hasCount = (id == llvm::Intrinsic::nvvm_barrier_cta_sync_count ||
+                   id == llvm::Intrinsic::nvvm_barrier_cta_sync_aligned_count);
+
+  SmallVector<Value> mlirOperands;
+  SmallVector<NamedAttribute> mlirAttrs;
+  if (failed(moduleImport.convertIntrinsicArguments(
+          llvmOperands, llvmOpBundles, /*requiresOpBundles=*/false, {}, {},
+          mlirOperands, mlirAttrs)))
+    return failure();
+
+  auto op = NVVM::BarrierOp::create(
+      odsBuilder, moduleImport.translateLoc(inst->getDebugLoc()),
+      mlirOperands.front(), hasCount ? mlirOperands.back() : Value{},
+      odsBuilder.getBoolAttr(aligned));
+  moduleImport.mapNoResultOp(inst, op);
+  return success();
+}
+
+/// Returns true if the LLVM intrinsic `id` has a corresponding MLIR NVVM op.
 static bool isConvertibleIntrinsic(llvm::Intrinsic::ID id) {
   static const DenseSet<unsigned> convertibleIntrinsics = {
 #include "mlir/Dialect/LLVMIR/NVVMConvertibleLLVMIRIntrinsics.inc"
   };
-  return convertibleIntrinsics.contains(id);
+  return convertibleIntrinsics.contains(id) || isBarrierSyncIntrinsic(id);
 }
 
-/// Returns the list of LLVM IR intrinsic identifiers that are convertible to
-/// MLIR NVVM dialect intrinsics.
+/// Returns the LLVM intrinsic IDs that have a corresponding MLIR NVVM op.
 static ArrayRef<unsigned> getSupportedIntrinsicsImpl() {
-  static const SmallVector<unsigned> convertibleIntrinsics = {
+  static const SmallVector<unsigned> convertibleIntrinsics = []() {
+    SmallVector<unsigned> ids = {
 #include "mlir/Dialect/LLVMIR/NVVMConvertibleLLVMIRIntrinsics.inc"
-  };
+    };
+    ids.append(std::begin(kBarrierSyncIntrinsics),
+               std::end(kBarrierSyncIntrinsics));
+    return ids;
+  }();
   return convertibleIntrinsics;
 }
 
@@ -54,6 +101,11 @@ static LogicalResult convertIntrinsicImpl(OpBuilder &odsBuilder,
     for (unsigned i = 0; i < inst->getNumOperandBundles(); ++i)
       llvmOpBundles.push_back(inst->getOperandBundleAt(i));
 
+    // Route `bar.sync` intrinsics to `nvvm.barrier`.
+    if (isBarrierSyncIntrinsic(intrinsicID))
+      return convertBarrierSyncIntrinsic(odsBuilder, inst, moduleImport,
+                                         llvmOperands, llvmOpBundles);
+
 #include "mlir/Dialect/LLVMIR/NVVMFromLLVMIRConversions.inc"
   }
 

diff  --git a/mlir/test/Target/LLVMIR/Import/nvvmir.ll b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
index 1430f9a44eba1..b2529cc8fac6c 100644
--- a/mlir/test/Target/LLVMIR/Import/nvvmir.ll
+++ b/mlir/test/Target/LLVMIR/Import/nvvmir.ll
@@ -73,11 +73,45 @@ define float @nvvm_rcp(float %0) {
 
 ; CHECK-LABEL: @llvm_nvvm_barrier0()
 define void @llvm_nvvm_barrier0() {
-  ; CHECK: llvm.nvvm.barrier.cta.sync.aligned.all
+  ; CHECK: %[[c0:.*]] = llvm.mlir.constant(0 : i32) : i32
+  ; CHECK: nvvm.barrier id = %[[c0]]
+  ; CHECK-NOT: aligned
   call void @llvm.nvvm.barrier0()
   ret void
 }
 
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_all
+define void @llvm_nvvm_barrier_sync_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}} {aligned = false}
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_all
+define void @llvm_nvvm_barrier_sync_aligned_all(i32 %bar) {
+  ; CHECK: nvvm.barrier id = %{{.*}}
+  ; CHECK-NOT: aligned
+  ; CHECK-NOT: number_of_threads
+  call void @llvm.nvvm.barrier.cta.sync.aligned.all(i32 %bar)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_count
+define void @llvm_nvvm_barrier_sync_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}} {aligned = false}
+  call void @llvm.nvvm.barrier.cta.sync.count(i32 %bar, i32 %n)
+  ret void
+}
+
+; CHECK-LABEL: @llvm_nvvm_barrier_sync_aligned_count
+define void @llvm_nvvm_barrier_sync_aligned_count(i32 %bar, i32 %n) {
+  ; CHECK: nvvm.barrier id = %{{.*}} number_of_threads = %{{.*}}
+  ; CHECK-NOT: aligned
+  call void @llvm.nvvm.barrier.cta.sync.aligned.count(i32 %bar, i32 %n)
+  ret void
+}
+
 ; CHECK-LABEL: @llvm_nvvm_bar_warp_sync
 define void @llvm_nvvm_bar_warp_sync(i32 %mask) {
   ; CHECK: nvvm.bar.warp.sync %{{.*}} : i32
@@ -276,6 +310,14 @@ declare float @llvm.nvvm.rcp.approx.ftz.f(float)
 
 declare void @llvm.nvvm.barrier0()
 
+declare void @llvm.nvvm.barrier.cta.sync.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.all(i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.count(i32, i32)
+
+declare void @llvm.nvvm.barrier.cta.sync.aligned.count(i32, i32)
+
 declare void @llvm.nvvm.bar.warp.sync(i32)
 
 declare i32 @llvm.nvvm.shfl.sync.bfly.i32(i32, i32, i32, i32)


        

From llvmlistbot at llvm.org  Sun Jul  5 16:43:59 2026
From: llvmlistbot at llvm.org (William Moses)
Date: Sun, 05 Jul 2026 16:43:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM]Import llvm intrinsics for
	nvvm.barrier (PR #202862)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202862@github.com>
Message-ID: <6a4aec3f.27b3b003.1610ae.bd27@mx.google.com>


https://github.com/wsmoses closed https://github.com/llvm/llvm-project/pull/202862

From llvmlistbot at llvm.org  Sun Jul  5 16:46:39 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Sun, 05 Jul 2026 16:46:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM]Import llvm intrinsics for
	nvvm.barrier (PR #202862)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202862@github.com>
Message-ID: <6a4aecdf.bd93e731.b068b.3660@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `clang-aarch64-sve-vla` running on `linaro-g3-04` while building `mlir` at step 6 "build stage 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/17/builds/16233

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 6 (build stage 1) failure: 'ninja' (failure)
...

  Tell CMake where to find the compiler by setting either the environment
  variable "FC" or the CMake cache entry CMAKE_Fortran_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[175/184] Linking CXX executable bin/bbc
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-stamps/runtimes-configure 
cd /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-bins && /usr/local/bin/cmake -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang -DCMAKE_CXX_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang -DCMAKE_Fortran_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/flang -DCMAKE_LINKER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/ld.lld -DCMAKE_AR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-ar -DCMAKE_RANLIB=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-ranlib -DCMAKE_NM=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-objcopy -DCMAKE_STRIP=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-strip -DCMAKE_READELF=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1.install -DLLVM_BINARY_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1 -DLLVM_CONFIG_PATH=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-v -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt -DFORTRAN_SUPPORTS_REAL16=1 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=flang-rt;compiler-rt" -DLLVM_USE_LINKER=lld -DCUDAToolkit_NVCC_EXECUTABLE=CUDAToolkit_NVCC_EXECUTABLE-NOTFOUND -DCUDAToolkit_SENTINEL_FILE=CUDAToolkit_SENTINEL_FILE-NOTFOUND -DFLANG_RUNTIME_F128_MATH_LIB= "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt;-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt" -GNinja -C/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../runtimes -B /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-bins && /usr/local/bin/cmake -E touch /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-stamps/runtimes-configure
[177/184] Linking CXX executable bin/flang-23
[178/184] Building CXX object tools/mlir/lib/Target/LLVMIR/Dialect/NVVM/CMakeFiles/obj.MLIRLLVMIRToNVVMTranslation.dir/LLVMIRToNVVMTranslation.cpp.o
ninja: build stopped: subcommand failed.

```

</details>

https://github.com/llvm/llvm-project/pull/202862

From llvmlistbot at llvm.org  Sun Jul  5 18:20:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 05 Jul 2026 18:20:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a4b02c0.abb10101.208f3e.89b7@mx.google.com>


Qixi-1 wrote:

@kuhar Could you please help land this when you get a chance? Thanks!

https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Sun Jul  5 19:35:33 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Sun, 05 Jul 2026 19:35:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reimplement
 SimplifyPackToExpandShape and SimplifyUnPackToCollapseShape for more cases.
 (PR #204971)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204971@github.com>
Message-ID: <6a4b1475.f8b79514.3b441.8a50@mx.google.com>


JerryShih wrote:

> Yes, something along those lines. Basically, I am struggling to see what is currently supported and what new functionality is added. I've not looked at those transformations for a while and some context/reminder would help 😅
> 
> In general, I find the tests in mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir quite poorly structured - it's hard to tell negative from positive cases. Ideally, we should refactor that file following:
> 
> * https://mlir.llvm.org/getting_started/TestingGuide/#test-formatting-best-practices and
> * https://mlir.llvm.org/getting_started/TestingGuide/#test-documentation-best-practices.
> 
> Specifically, any negative tests added here that would already fail without this change, should be submitted in a separate PR. Would you be open iterating a bit here? At the very least, we should clearly label the negative tests. If not, I can update the tests and then you can submit this as a follow-up. I would be more than happy to review your refactoring PR if you decided to that. Please let me know your preference.

@banach-space 
I will move the new negative tests created in this pr into another pr and to some renaming for the existed test case later.

https://github.com/llvm/llvm-project/pull/204971

From llvmlistbot at llvm.org  Sun Jul  5 21:47:47 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Sun, 05 Jul 2026 21:47:47 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4b3373.331483e7.5bc1e.9e31@mx.google.com>


Saieiei wrote:

ping @skatrak 

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Sun Jul  5 21:56:21 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Sun, 05 Jul 2026 21:56:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b3575.a87eb5dc.1d432d.ccdf@mx.google.com>


aobolensk wrote:

> Unless I'm missing something the fp semantics we get before and after the conversion may be different due to ordering, right?. Not sure if that's a problem or not; and I'm not sure how the conversion has been dealing with this so far. I just wonder whether we should keep the mix ordering and then let the LLVM IR side to fold it into fma if desired.

CL.mix explicitly permits FMA contractions, but GL.FMix does not have the statements. The patterns were splited 

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Sun Jul  5 21:56:22 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Sun, 05 Jul 2026 21:56:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b3576.eb24c6ea.1f2e26.ebe5@mx.google.com>


================
@@ -1584,6 +1584,31 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
----------------
aobolensk wrote:

Done

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Sun Jul  5 21:56:35 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Sun, 05 Jul 2026 21:56:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b3583.f8b79514.3b441.4c9c@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206935

>From 4fa5c4ca4a5a4333743f1fc3afa38979d20a176a Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:54:35 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix
 and CL.mix

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 27 +++++++++++++++++++
 .../SPIRVToLLVM/cl-ops-to-llvm.mlir           | 20 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 20 ++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}

>From 5512c6003aa7212a15226316a7ca9c6aef850eb0 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Mon, 6 Jul 2026 06:56:23 +0200
Subject: [PATCH 2/2] Address comments

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 45 ++++++++++++++-----
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 14 ++++--
 2 files changed, 44 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 24fdccb55ffaa..2fcc14777a5eb 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1584,18 +1584,42 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
-/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
-/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
-/// and CL ops name their third operand differently (`a` vs. `z`).
-template <typename SPIRVOp>
-class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+/// Converts `spirv.GL.FMix` to `x * (1 - a) + y * a` as specified by
+/// GL.std.450.
+class GLFMixPattern : public SPIRVToLLVMConversion<spirv::GLFMixOp> {
 public:
-  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+  using SPIRVToLLVMConversion<spirv::GLFMixOp>::SPIRVToLLVMConversion;
 
   LogicalResult
-  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+  matchAndRewrite(spirv::GLFMixOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getA();
+    Value one = createFPConstant(loc, op.getType(), dstType, rewriter, 1.0);
+    Value oneMinusA = LLVM::FSubOp::create(rewriter, loc, dstType, one, a);
+    Value lhs = LLVM::FMulOp::create(rewriter, loc, dstType, x, oneMinusA);
+    Value rhs = LLVM::FMulOp::create(rewriter, loc, dstType, y, a);
+    rewriter.replaceOpWithNewOp<LLVM::FAddOp>(op, dstType, lhs, rhs);
+    return success();
+  }
+};
+
+/// Converts `spirv.CL.mix` to `fma(a, y - x, x)`. The OpenCL spec defines
+/// mix as `x + (y - x) * a` and explicitly permits FMA contractions.
+class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::CLMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::CLMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
@@ -1991,7 +2015,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
-      MixPattern<spirv::GLFMixOp>,
+      GLFMixPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -2024,8 +2048,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMaxOp, LLVM::SMaxOp>,
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
-      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
-      MixPattern<spirv::CLMixOp>,
+      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>, CLMixPattern,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index beff049c7ebc8..caecf1f9f8d32 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -413,16 +413,22 @@ spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
 
 // CHECK-LABEL: @fmix_scalar
 spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
-  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
-  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : f32
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : f32
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : f32
   %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
   spirv.Return
 }
 
 // CHECK-LABEL: @fmix_vector
 spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
-  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
-  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(dense<1.000000e+00> : vector<4xf32>) : vector<4xf32>
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : vector<4xf32>
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : vector<4xf32>
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : vector<4xf32>
   %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
   spirv.Return
 }


From llvmlistbot at llvm.org  Sun Jul  5 23:55:22 2026
From: llvmlistbot at llvm.org (Reinhard Stahn)
Date: Sun, 05 Jul 2026 23:55:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToAffine] Raise scf.for to
	affine.for (PR #200851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200851@github.com>
Message-ID: <6a4b515a.449bfb44.233a36.703b@mx.google.com>


rainij wrote:

Friendly ping. This has been open for a few weeks with feedback addressed. @joker-eph, given your earlier thumbs-up on this, would you have bandwidth to take a look, or could you suggest another reviewer if not?

https://github.com/llvm/llvm-project/pull/200851

From llvmlistbot at llvm.org  Mon Jul  6 00:17:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:17:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [compiler-rt] [flang]
 [libc] [libcxx] [llvm] [mlir] [polly] [NFC] Fix "is is" typos (PR #206288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206288@github.com>
Message-ID: <6a4b568a.27b3b003.1610ae.578f@mx.google.com>


https://github.com/zackc6 approved this pull request.


https://github.com/llvm/llvm-project/pull/206288

From llvmlistbot at llvm.org  Mon Jul  6 00:25:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:25:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
Message-ID: <llvm.org/llvm/llvm-project/pull/207670@github.com>


https://github.com/LouisLu060211 created https://github.com/llvm/llvm-project/pull/207670

Fix a crash in `-test-alias-analysis-modref` preventing when printing block-argument. This operands the parent op carries `test.ptr` as a unit attribute instead of a `StringAttr`.
The test printer handles missing/non-string `test.ptr` attributes safely. The regression test for the reproducer from #206925.

Assisted by: Codex

>From fb9fd156080e0809b97d174b1a570783b23ed5c0 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/2] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 10afc402a685de09e498bc61449459cf8fe230a9 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/2] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";


From llvmlistbot at llvm.org  Mon Jul  6 00:25:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:25:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a4b587c.8e75e0cb.2e0812.fd2a@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @LouisLu060211 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207670

From llvmlistbot at llvm.org  Mon Jul  6 00:25:59 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Mon, 06 Jul 2026 00:25:59 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4b5887.eb24c6ea.1f2e26.0152@mx.google.com>


================
@@ -422,6 +435,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region. See getNumber().
+  unsigned blockNumber = 0;
----------------
aengelke wrote:

Initialize with -1u to signal "invalid number". Is there no way blocks can be removed without being erased from a region?

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Mon Jul  6 00:26:00 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Mon, 06 Jul 2026 00:26:00 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4b5888.bb839515.23e6d3.4305@mx.google.com>


================
@@ -524,10 +524,13 @@ raw_ostream &operator<<(raw_ostream &OS, const LoopBase<BlockT, LoopT> &Loop) {
 ///
 
 template <class BlockT, class LoopT> class LoopInfoBase {
-  // BBMap - Mapping of basic blocks to the inner most loop they occur in
-  std::conditional_t<GraphHasNodeNumbers<const BlockT *>, SmallVector<LoopT *>,
-                     DenseMap<const BlockT *, LoopT *>>
-      BBMap;
+  static_assert(GraphHasNodeNumbers<const BlockT *>,
+                "LoopInfo requires GraphTraits<BlockT *>::getNumber (see "
+                "GraphHasNodeNumbers)");
----------------
aengelke wrote:

Can we separate this? Bolt also needs changes, and I'd prefer PR1 MLIR, PR2 Bolt, PR3 LoopInfo cleanup.

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Mon Jul  6 00:26:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:26:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a4b5895.27b3b003.1610ae.d158@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/207670

From llvmlistbot at llvm.org  Mon Jul  6 00:26:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:26:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a4b58a1.eb165e5c.113c2a.1c36@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: LouisLu060211

<details>
<summary>Changes</summary>

Fix a crash in `-test-alias-analysis-modref` preventing when printing block-argument. This operands the parent op carries `test.ptr` as a unit attribute instead of a `StringAttr`.
The test printer handles missing/non-string `test.ptr` attributes safely. The regression is being added to test the changes. 

Assisted by: Codex

---
Full diff: https://github.com/llvm/llvm-project/pull/207670.diff


4 Files Affected:

- (modified) mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp (+9-4) 
- (modified) mlir/test/Analysis/test-alias-analysis-modref.mlir (+14) 
- (modified) mlir/test/Dialect/MemRef/mem2reg.mlir (+15) 
- (modified) mlir/test/lib/Analysis/TestAliasAnalysis.cpp (+40-10) 


``````````diff
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

``````````

</details>


https://github.com/llvm/llvm-project/pull/207670

From llvmlistbot at llvm.org  Mon Jul  6 00:32:24 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 00:32:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4b5a08.dd2b4bd3.13e95e.dbbd@mx.google.com>


matthias-springer wrote:

> Encodings that do not implement VerifiableTensorEncoding (opaque dict/string/bounds attrs used by downstream dialects) are rank-agnostic

Is this documented somewhere? This is what the documentation of the `VerifiableTensorEncoding` says:
```
        Verifies the encoding is valid for a tensor type with the
        given shape and element type. Generates a diagnostic using
        the supplied callback on failure.
```

This documentation does not specify what happens when an attribute does not implement `VerifiableTensorEncoding`.

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Mon Jul  6 00:34:51 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 06 Jul 2026 00:34:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToAffine] Raise scf.for to
	affine.for (PR #200851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200851@github.com>
Message-ID: <6a4b5a9b.af6ba3ed.2c2f30.33ed@mx.google.com>


joker-eph wrote:

That look fine overall. @ftynse any lingering concerns?

https://github.com/llvm/llvm-project/pull/200851

From llvmlistbot at llvm.org  Mon Jul  6 00:36:22 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 00:36:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Add ValueBoundsOpInterface
 external models for the memref operations extract_strided_metadata and
 assume_alignment (PR #206466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206466@github.com>
Message-ID: <6a4b5af6.cbc3f03a.33a4cc.e9fe@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/206466

From llvmlistbot at llvm.org  Mon Jul  6 00:36:27 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 06 Jul 2026 00:36:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ODS] Add strict property assembly
	format mode (PR #196269)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196269@github.com>
Message-ID: <6a4b5afb.bf38ae28.358429.f736@mx.google.com>


================
@@ -55,6 +55,12 @@ class Dialect {
   // dialect declaration.
   code extraClassDeclaration = "";
 
+  // If this dialect should require declarative parsers for property-backed
+  // operations to bind every inherent attribute and property directly in the
+  // custom assembly format, or otherwise cover them with `prop-dict`. This
+  // stricter mode is disabled by default for now.
+  bit useStrictPropertiesInAssemblyFormat = 0;
----------------
joker-eph wrote:

Ping @jpienaar ?

https://github.com/llvm/llvm-project/pull/196269

From llvmlistbot at llvm.org  Mon Jul  6 00:37:11 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 00:37:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Add ValueBoundsOpInterface
 external models for the memref operations extract_strided_metadata and
 assume_alignment (PR #206466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206466@github.com>
Message-ID: <6a4b5b27.dd2b4bd3.13e95e.3b7d@mx.google.com>


https://github.com/matthias-springer auto_merge_enabled https://github.com/llvm/llvm-project/pull/206466

From llvmlistbot at llvm.org  Mon Jul  6 00:51:44 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Mon, 06 Jul 2026 00:51:44 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
Message-ID: <llvm.org/llvm/llvm-project/pull/207677@github.com>


https://github.com/nikic created https://github.com/llvm/llvm-project/pull/207677

Reapply with a use-after-free fix. The MC subtarget used for inline
assembly has to save alive until finalization, not only during the
emitInlineAsm() call.

-----

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can now be
specified as follows:

    module asm(target_features: "+foo", target_cpu: "bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO scenarios
we don't know what target features to use for parsing the module-level
inline assembly. Now they can be faithfully preserved, even when merging
inline assembly from different modules with different features.

If target_features and target_cpu are empty, we fall back to the old
behavior (which, in the backend, would be to use the default subtarget
of the TargetMachine).

Fixes https://github.com/llvm/llvm-project/issues/61991.
Fixes https://github.com/llvm/llvm-project/issues/67698.


>From 54b10ceb4b5c64841a1baad873f713b8ca243970 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Fri, 3 Jul 2026 09:59:08 +0200
Subject: [PATCH 1/2] [IR] Explicitly specify target feature for module asm
 (#204548)

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can now be
specified as follows:

    module asm(target_features: "+foo", target_cpu: "bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO scenarios
we don't know what target features to use for parsing the module-level
inline assembly. Now they can be faithfully preserved, even when merging
inline assembly from different modules with different features.

If target_features and target_cpu are empty, we fall back to the old
behavior (which, in the backend, would be to use the default subtarget
of the TargetMachine).

Fixes https://github.com/llvm/llvm-project/issues/61991.
Fixes https://github.com/llvm/llvm-project/issues/67698.
---
 clang/lib/CodeGen/CGObjCMac.cpp               |  6 +-
 clang/lib/CodeGen/CodeGenModule.cpp           |  8 +-
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c  | 17 ++--
 clang/test/CodeGen/asm.c                      |  7 +-
 clang/test/CodeGen/asm_incbin.c               |  3 +-
 clang/test/CodeGen/cfi-salt.c                 | 11 ++-
 clang/test/CodeGen/kcfi.c                     |  7 +-
 clang/test/CodeGenCUDA/filter-decl.cu         |  5 +-
 clang/test/CodeGenCXX/gnu-asm-constexpr.cpp   |  3 +-
 clang/test/CodeGenObjC/category-class.m       |  7 +-
 clang/test/CodeGenObjC/objc-runtime-name.m    |  7 +-
 clang/test/Frontend/ast-codegen.c             |  3 +-
 clang/test/Modules/codegen.test               |  3 +-
 llvm/docs/LangRef.rst                         | 27 ++++--
 llvm/docs/ReleaseNotes.md                     |  4 +
 llvm/include/llvm/Bitcode/LLVMBitCodes.h      |  2 +
 llvm/include/llvm/CodeGen/AsmPrinter.h        | 13 +++
 llvm/include/llvm/IR/Module.h                 | 96 ++++++++++++++++---
 llvm/lib/AsmParser/LLParser.cpp               | 39 +++++++-
 llvm/lib/Bitcode/Reader/BitcodeReader.cpp     | 17 +++-
 llvm/lib/Bitcode/Writer/BitcodeWriter.cpp     | 16 +++-
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    | 28 ++++--
 llvm/lib/IR/AsmWriter.cpp                     | 43 ++++++---
 llvm/lib/IR/Core.cpp                          | 13 ++-
 llvm/lib/IR/Module.cpp                        | 20 ++++
 llvm/lib/LTO/LTO.cpp                          |  4 +-
 llvm/lib/Linker/IRMover.cpp                   |  9 +-
 llvm/lib/Object/ModuleSymbolTable.cpp         | 90 ++++++++---------
 llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp     | 23 +++--
 llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp     |  2 +-
 llvm/lib/Transforms/IPO/ExtractGV.cpp         |  2 +-
 .../Transforms/IPO/ThinLTOBitcodeWriter.cpp   |  2 +-
 .../Instrumentation/DataFlowSanitizer.cpp     | 26 ++---
 llvm/lib/Transforms/Utils/SplitModule.cpp     |  2 +-
 llvm/test/Assembler/module-asm-invalid.ll     | 25 +++++
 llvm/test/Bitcode/compatibility-3.6.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.7.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.8.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.9.ll        |  3 +-
 llvm/test/Bitcode/compatibility-4.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-5.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-6.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility.ll            |  3 +-
 llvm/test/Bitcode/highLevelStructure.3.2.ll   |  3 +-
 llvm/test/Bitcode/module-asm.ll               | 29 ++++++
 .../test/CodeGen/RISCV/module-asm-features.ll | 17 ++++
 .../DataFlowSanitizer/prefix-rename.ll        |  3 +-
 llvm/test/LTO/RISCV/module-asm.ll             | 18 ++++
 llvm/test/LTO/X86/inline-asm-lto-discard.ll   | 24 ++---
 llvm/test/Linker/Inputs/module-asm.ll         |  2 +
 .../link-arm-and-thumb-module-inline-asm.ll   | 17 ++--
 llvm/test/Linker/module-asm.ll                |  9 ++
 llvm/test/ThinLTO/X86/import-symver.ll        |  3 +-
 .../LowerTypeTests/export-symver.ll           |  3 +-
 .../cfi-icall-static-inline-asm.ll            |  3 +-
 .../ThinLTOBitcodeWriter/x86/module-asm.ll    |  3 +-
 .../llvm-reduce/deltas/ReduceModuleData.cpp   |  2 +-
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 11 ++-
 mlir/test/Target/LLVMIR/module-asm.mlir       |  5 +-
 59 files changed, 573 insertions(+), 193 deletions(-)
 create mode 100644 llvm/test/Assembler/module-asm-invalid.ll
 create mode 100644 llvm/test/Bitcode/module-asm.ll
 create mode 100644 llvm/test/CodeGen/RISCV/module-asm-features.ll
 create mode 100644 llvm/test/LTO/RISCV/module-asm.ll
 create mode 100644 llvm/test/Linker/Inputs/module-asm.ll
 create mode 100644 llvm/test/Linker/module-asm.ll

diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index 69c5e88f3c768..e518b875f44bb 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,10 +6610,6 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
-    Asm += CGM.getModule().getModuleInlineAsm();
-    if (!Asm.empty() && Asm.back() != '\n')
-      Asm += '\n';
-
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6624,7 +6620,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().setModuleInlineAsm(OS.str());
+    CGM.getModule().appendModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 813cd67e4dd51..e155fdd752d7f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,7 +8057,13 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-    getModule().appendModuleInlineAsm(AD->getAsmString());
+
+    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
+    llvm::Module::GlobalAsmProperties Props;
+    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
+    Props.TargetCPU = TargetOpts.CPU;
+    getModule().appendModuleInlineAsm(
+        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 877da4fc669e3..8a212a0b61905 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,13 +1,18 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
+// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
 
-// CHECK: module asm "foo1"
+// RISCV64: module asm(target_features: "+64bit,+i")
+// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
+// DEFAULT: module asm
+// CHECK-NEXT: "foo1"
 __asm__ ("foo1");
-// CHECK: module asm "foo2"
+// CHECK-NEXT: "foo2"
 __asm__ ("foo2");
-// CHECK: module asm "foo3"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo3");
-// CHECK: module asm "foo4"
+// CHECK-NEXT: "foo4"
 __asm__ ("foo4");
-// CHECK: module asm "foo5"
+// CHECK-NEXT: "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index d7465b22fbbf6..ea69c7a17d1cb 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,9 +2,10 @@
 
 // PR10415:
 //
-// CHECK:      module asm "foo1"
-// CHECK-NEXT: module asm "foo2"
-// CHECK-NEXT: module asm "foo3"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo1"
+// CHECK-NEXT: "foo2"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index 6234eea1d6738..bbc5d3af55ea2 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,4 +6,5 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm ".incbin \22foo.h\22"
+// CHECK: module asm
+// CHECK-NEXT: ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 2a90dc31d939b..18156dc014280 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,13 +26,14 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
+// CHECK-NOT: ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index da32859dd3d88..f08cf7635fb5a 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,10 +6,11 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index 02dacd0ad8ef4..f87d62a56631e 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,8 +5,9 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm "file scope asm is host only"
-// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
+// CHECK-HOST: module asm
+// CHECK-HOST-NEXT: "file scope asm is host only"
+// CHECK-DEVICE-NOT: "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 175a3b7bc588c..2b4abee5dc05f 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,8 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm "nop; nop; nop; nop"
+    // CHECK: module asm(target_features: "{{.*}}")
+    // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index 92fd36cbe4066..febe94ce5b9f5 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,9 +1,10 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
-// CHECK: module asm "\09.objc_category_name_A_foo=0"
-// CHECK: module asm "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm
+// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
+// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
+// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index c838b8c2c6f1a..a7f59caced390 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,9 +3,10 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm {{.*}}objc_class_name_XYZ=0
-// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
-// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm
+// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
+// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
+// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index 2e2e3d360dd17..c26b0a1908379 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,8 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm "foo"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 0af630a754805..5cc5b3c1cda2a 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,7 +18,8 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm "narf"
+BOTH: module asm
+BOTH-NEXT: "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5ce6019660e0b..2c8276d04b668 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,19 +3357,34 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks. These blocks are internally
-concatenated by LLVM and treated as a single unit, but may be separated
-in the ``.ll`` file if desired. The syntax is very simple:
+to the GCC "file scope inline asm" blocks:
 
 .. code-block:: llvm
 
-    module asm "inline asm code goes here"
-    module asm "more can go here"
+    module asm
+        "inline asm code goes here"
+        "more can go here"
 
-The strings can contain any character by escaping non-printable
+The sequence of string literals is internally concatenated using newlines as
+separators. The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
+Additionally, module-level inline assembly can specify an optional list of
+properties:
+
+.. code-block:: llvm
+
+    module asm(target_features: "+foo", target_cpu: "bar")
+        "inline asm code goes here"
+        "more can go here"
+
+Currently, the only supported properties are ``target_features`` and
+``target_cpu``, with the same meaning as the ``"target-features"`` and
+``"target-cpu"`` function attributes.
+
+Consecutive blocks with identical properties will be concatenated into one.
+
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 11d3ebc80fca6..3ca9f5166ad95 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -114,6 +114,10 @@ Makes programs 10x faster by doing Special New Thing.
 * `alwaysinline` no longer bypasses inlining compatibility checks based on
   target features. Inlining will only be performed if it is safe to do so.
 
+* Module-level inline assembly now accepts optional `target_features` and
+  `target_cpu` properties. This resolves errors during LTO on some
+  architectures.
+
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index af4cac996a78c..1761e7a542a73 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,6 +120,8 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
+
+  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 78e64440657cc..335bf9de9a071 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,6 +930,19 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
+  /// Emit necessary directives to allow use of instructions that are permitted
+  /// by target features enabled by STI, but are not permitted by target
+  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
+  /// Returns whether anything was emitted.
+  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+    return false;
+  }
+
+  /// Emit necessary directives to restore target feature state.
+  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
+  /// call.
+  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
+
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 4f7e33969f16f..63d1013896b47 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,6 +169,45 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
+  struct GlobalAsmProperties {
+    std::string TargetFeatures;
+    std::string TargetCPU;
+
+    /// Set a property using a string name.
+    /// Returns whether the property name was valid.
+    LLVM_ABI bool set(StringRef Name, std::string Value);
+
+    /// Get a list of set properties as pairs of key and value.
+    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+
+    bool operator==(const GlobalAsmProperties &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+
+    bool operator!=(const GlobalAsmProperties &Other) const {
+      return !(*this == Other);
+    }
+  };
+
+  struct GlobalAsmFragment {
+    std::string Asm;
+    GlobalAsmProperties Props;
+
+    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
+    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
+        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
+      if (!Asm.empty() && Asm.back() != '\n')
+        Asm += '\n';
+    }
+
+    bool empty() const { return Asm.empty(); }
+
+    bool hasSameProperties(const GlobalAsmFragment &Other) const {
+      return Props == Other.Props;
+    }
+  };
+
 /// @}
 /// @name Member Variables
 /// @{
@@ -180,7 +219,8 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
+  /// Inline Asm at the global scope.
+  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -287,8 +327,17 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  /// @returns a string containing the module-scope inline assembly blocks.
-  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
+  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
+    return GlobalScopeAsm;
+  }
+
+  /// Get any module-scope inline assembly blocks.
+  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
+    return GlobalScopeAsm;
+  }
+
+  /// Return whether there is any module-scope inline assembly.
+  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -325,20 +374,45 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
+  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
+
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm = std::string(Asm);
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
+    GlobalScopeAsm.clear();
+    appendModuleInlineAsm(std::move(Fragment));
+  }
+
+  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
+    GlobalScopeAsm.clear();
+    append_range(GlobalScopeAsm, Fragments);
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm += Asm;
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.back().Asm += Fragment.Asm;
+    } else {
+      GlobalScopeAsm.emplace_back(std::move(Fragment));
+    }
+  }
+
+  /// Prepend to the module-scope inline assembly blocks.
+  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
+    } else {
+      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
+    }
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 75e6add0ec76f..0a5b4c6c91879 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,16 +641,49 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
+///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
+///                          'property_name2:' STRINGCONSTANT ')'
+///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
-      parseStringConstant(AsmStr))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  M->appendModuleInlineAsm(AsmStr);
+  Module::GlobalAsmProperties Props;
+  if (EatIfPresent(lltok::lparen)) {
+    while (true) {
+      std::string Key, Value;
+      SMLoc Loc = Lex.getLoc();
+      if (Lex.getKind() != lltok::LabelStr)
+        return error(Loc, "expected property name followed by ':'");
+
+      Key = Lex.getStrVal();
+      Lex.Lex();
+
+      if (parseStringConstant(Value))
+        return true;
+
+      if (!Props.set(Key, Value))
+        return error(Loc, "unknown property name");
+
+      if (EatIfPresent(lltok::rparen))
+        break;
+      if (parseToken(lltok::comma, "expected ',' or ')'"))
+        return true;
+    }
+  }
+
+  do {
+    std::string AsmStrPart;
+    if (parseStringConstant(AsmStrPart))
+      return true;
+    AsmStr += AsmStrPart + "\n";
+  } while (Lex.getKind() == lltok::StringConstant);
+
+  M->appendModuleInlineAsm({AsmStr, Props});
   return false;
 }
 
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 2bd251efd05ce..06e9aea4fd3ac 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,6 +4580,9 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
+  // Apply to the following module asm.
+  Module::GlobalAsmProperties Props;
+
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4788,11 +4791,23 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
+    case bitc::MODULE_CODE_ASM_PROPERTY: {
+      std::string Str;
+      if (convertToString(Record, 0, Str))
+        return error("Invalid module asm record");
+      size_t SepPos = Str.find('\0');
+      if (SepPos == std::string::npos)
+        return error("Invalid module asm record");
+      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
+        return error("Unknown module asm property");
+      break;
+    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->setModuleInlineAsm(S);
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
+      Props = {};
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 4aa203b0d12d1..493cbec7506f2 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,9 +1551,19 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-  if (!M.getModuleInlineAsm().empty())
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
-                      0 /*TODO*/);
+
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    SmallVector<std::pair<StringRef, StringRef>> Props =
+        Frag.Props.getAsStrings();
+    for (auto [Key, Value] : Props) {
+      SmallVector<unsigned, 64> Record;
+      Record.append(Key.begin(), Key.end());
+      Record.push_back(0);
+      Record.append(Value.begin(), Value.end());
+      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
+    }
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
+  }
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index abb20a7919fa1..8248a05f1a35f 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,6 +110,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -127,7 +128,6 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,13 +610,29 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    emitInlineAsm(
-        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
-        TM.Options.MCOptions, nullptr,
-        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
+        std::unique_ptr<MCSubtargetInfo> AsmSTI(
+            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
+                                                 Frag.Props.TargetCPU,
+                                                 Frag.Props.TargetFeatures));
+        bool DidPush = emitTargetFeaturePush(*AsmSTI);
+        emitInlineAsm(
+            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+        emitTargetFeaturePop(*AsmSTI, DidPush);
+      } else {
+        // If the module asm does not explicitly specify target features,
+        // fall back to default subtargetinfo.
+        emitInlineAsm(
+            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
+            nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      }
+    }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 665d7122af2c7..02d6bcd90add0 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,21 +3126,38 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (!M->getModuleInlineAsm().empty()) {
+  if (M->hasModuleInlineAsm()) {
     Out << '\n';
 
-    // Split the string into lines, to make it easier to read the .ll file.
-    StringRef Asm = M->getModuleInlineAsm();
-    do {
-      StringRef Front;
-      std::tie(Front, Asm) = Asm.split('\n');
-
-      // We found a newline, print the portion of the asm string from the
-      // last newline up to this newline.
-      Out << "module asm \"";
-      printEscapedString(Front, Out);
-      Out << "\"\n";
-    } while (!Asm.empty());
+    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
+      Out << "module asm";
+      SmallVector<std::pair<StringRef, StringRef>> Props =
+          Frag.Props.getAsStrings();
+      if (!Props.empty()) {
+        ListSeparator LS;
+        Out << "(";
+        for (auto [Key, Value] : Props) {
+          Out << LS;
+          Out << Key << ": \"";
+          printEscapedString(Value, Out);
+          Out << "\"";
+        }
+        Out << ")";
+      }
+      Out << "\n";
+      // Split the string into lines, to make it easier to read the .ll file.
+      StringRef Asm = Frag.Asm;
+      do {
+        StringRef Front;
+        std::tie(Front, Asm) = Asm.split('\n');
+
+        // We found a newline, print the portion of the asm string from the
+        // last newline up to this newline.
+        Out << "    \"";
+        printEscapedString(Front, Out);
+        Out << "\"\n";
+      } while (!Asm.empty());
+    }
   }
 
   printTypeIdentities();
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index 6c6c3b5c84ede..a7abd3eed31c3 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,7 +510,18 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  auto &Str = unwrap(M)->getModuleInlineAsm();
+  Module *Mod = unwrap(M);
+  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
+  if (Frags.empty()) {
+    *Len = 0;
+    return nullptr;
+  }
+
+  if (Frags.size() != 1)
+    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
+                          "more than one module inline assembly fragment");
+
+  auto &Str = Frags.begin()->Asm;
   *Len = Str.length();
   return Str.c_str();
 }
diff --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 40ef7e354da72..34e44e26d5db5 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,3 +977,23 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
+
+bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
+  if (Name == "target_features")
+    TargetFeatures = std::move(Value);
+  else if (Name == "target_cpu")
+    TargetCPU = std::move(Value);
+  else
+    return false;
+  return true;
+}
+
+SmallVector<std::pair<StringRef, StringRef>>
+Module::GlobalAsmProperties::getAsStrings() const {
+  SmallVector<std::pair<StringRef, StringRef>> Props;
+  if (!TargetFeatures.empty())
+    Props.emplace_back("target_features", TargetFeatures);
+  if (!TargetCPU.empty())
+    Props.emplace_back("target_cpu", TargetCPU);
+  return Props;
+}
diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 6a754f95092bb..b672f4e5fac14 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
+    M.prependModuleInlineAsm(NewIA);
   }
 
   assert(MsymI == MsymE);
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde1cec..3b72b412d0b2e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,11 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
+      Module::GlobalAsmFragment NewFrag(Frag);
+      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
+      DstM.appendModuleInlineAsm(std::move(NewFrag));
+    }
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1573,7 +1576,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(S);
+        DstM.appendModuleInlineAsm(std::string(S));
       }
     });
   }
diff --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 1da5fa9c10a0b..44f717ec12323 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,8 +72,7 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  StringRef InlineAsm = M.getModuleInlineAsm();
-  if (InlineAsm.empty())
+  if (!M.hasModuleInlineAsm())
     return;
 
   std::string Err;
@@ -90,52 +89,55 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
-  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
-  if (!STI)
-    return;
-
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  std::unique_ptr<MemoryBuffer> Buffer(
-      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
-  SourceMgr SrcMgr;
-  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-  std::unique_ptr<MCObjectFileInfo> MOFI(
-      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-  MCCtx.setObjectFileInfo(MOFI.get());
-  RecordStreamer Streamer(MCCtx, M);
-  T->createNullTargetStreamer(Streamer);
-
-  std::unique_ptr<MCAsmParser> Parser(
-      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-  std::unique_ptr<MCTargetAsmParser> TAP(
-      T->createMCAsmParser(*STI, *Parser, *MCII));
-  if (!TAP)
-    return;
-
-  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                 const SourceMgr &SrcMgr,
-                                 std::vector<const MDNode *> &LocInfos) {
-    M.getContext().diagnose(
-        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-  });
-
-  // Module-level inline asm is assumed to use At&t syntax (see
-  // AsmPrinter::doInitialization()).
-  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-  Parser->setSymbolScanningMode(true);
-
-  Parser->setTargetParser(*TAP);
-  if (Parser->Run(false))
-    return;
-
-  Init(Streamer);
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
+        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
+    if (!STI)
+      return;
+
+    std::unique_ptr<MemoryBuffer> Buffer(
+        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
+    SourceMgr SrcMgr;
+    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+    std::unique_ptr<MCObjectFileInfo> MOFI(
+        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+    MCCtx.setObjectFileInfo(MOFI.get());
+    RecordStreamer Streamer(MCCtx, M);
+    T->createNullTargetStreamer(Streamer);
+
+    std::unique_ptr<MCAsmParser> Parser(
+        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+    std::unique_ptr<MCTargetAsmParser> TAP(
+        T->createMCAsmParser(*STI, *Parser, *MCII));
+    if (!TAP)
+      return;
+
+    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                   const SourceMgr &SrcMgr,
+                                   std::vector<const MDNode *> &LocInfos) {
+      M.getContext().diagnose(
+          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+    });
+
+    // Module-level inline asm is assumed to use At&t syntax (see
+    // AsmPrinter::doInitialization()).
+    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+    Parser->setSymbolScanningMode(true);
+
+    Parser->setTargetParser(*TAP);
+    if (Parser->Run(false))
+      return;
+
+    Init(Streamer);
+  }
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index dbda25c34cee0..e37ea27364e6e 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,7 +113,8 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitDirectiveOptionArch();
+  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
+  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -523,19 +524,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitDirectiveOptionArch() {
+bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                                : RISCVOptionArchArgType::Minus;
+    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                               : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -549,11 +550,16 @@ bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   return false;
 }
 
+void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
+                                           bool DidPush) {
+  if (DidPush)
+    getTargetStreamer().emitDirectiveOptionPop();
+}
+
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
-  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitDirectiveOptionArch();
+  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -561,8 +567,7 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  if (EmittedOptionArch)
-    RTS.emitDirectiveOptionPop();
+  emitTargetFeaturePop(*STI, EmittedOptionArch);
   return false;
 }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 2a5d16a470c4a..22d0d97090746 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
   }
 
   // Register the NSDI handler before calling the base class so that
diff --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 16fdb93e2b21b..4b76b02c32fe4 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually
diff --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index a486979a58c2d..e857c94330bad 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->setModuleInlineAsm("");
+  MergedM->removeModuleInlineAsm();
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.
diff --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index c349fe33dd237..375ac069272eb 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,18 +1283,20 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  std::string Asm = GV->getParent()->getModuleInlineAsm();
-  std::string SearchStr = ".symver " + GVName + ",";
-  size_t Pos = Asm.find(SearchStr);
-  if (Pos != std::string::npos) {
-    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
-    Pos = Asm.find('@');
-
-    if (Pos == std::string::npos)
-      report_fatal_error(Twine("unsupported .symver: ", Asm));
-
-    Asm.replace(Pos, 1, Suffix + "@");
-    GV->getParent()->setModuleInlineAsm(Asm);
+  for (Module::GlobalAsmFragment &Frag :
+       GV->getParent()->getModuleInlineAsm()) {
+    std::string SearchStr = ".symver " + GVName + ",";
+    size_t Pos = Frag.Asm.find(SearchStr);
+    if (Pos != std::string::npos) {
+      Frag.Asm.replace(Pos, SearchStr.size(),
+                       ".symver " + GVName + Suffix + ",");
+      Pos = Frag.Asm.find('@');
+
+      if (Pos == std::string::npos)
+        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
+
+      Frag.Asm.replace(Pos, 1, Suffix + "@");
+    }
   }
 }
 
diff --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 64910b4d1ce99..8e70902c4dac2 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->setModuleInlineAsm("");
+      MPart->removeModuleInlineAsm();
     ModuleCallback(std::move(MPart));
   }
 }
diff --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
new file mode 100644
index 0000000000000..34e5b71e683a9
--- /dev/null
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -0,0 +1,25 @@
+; RUN: split-file %s %t
+; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
+; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
+; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
+; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
+
+;--- invalid_prop.ll
+; CHECK: unknown property name
+module asm(foo: "bar")
+    "asm"
+
+;--- missing_value1.ll
+; CHECK: expected property name followed by ':'
+module asm(target_features)
+    "asm"
+
+;--- missing_value2.ll
+; CHECK: expected string constant
+module asm(target_features:)
+    "asm"
+
+;--- missing_paren.ll
+; CHECK: expected ',' or ')'
+module asm(target_features: "bar"
+    "asm"
diff --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index 288e0fc50a9ba..c79086bad1ba5 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index ef9490310fe5d..00b2bb9b30b62 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 538d6101b3994..5b86d5ffa1fee 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index 3bfef73e485dd..eca4bbdf03766 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index aaf9fe5492171..8551b98815b1a 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index b820417107e8b..883ac3eaaa293 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 67e96c7277cc3..624d5bd7703c6 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index d55aa1dc496df..97889d4687516 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,7 +16,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index f2770ea4e4d34..cd9a34c810d57 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,7 +9,8 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm "some assembly"
+; CHECK: module asm
+; CHECK-NEXT: "some assembly"
 module asm "some assembly"
 
 ; Named Types Test
diff --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
new file mode 100644
index 0000000000000..406f25c5dd28c
--- /dev/null
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -0,0 +1,29 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; CHECK: module asm
+; CHECK:     "asm line 0"
+; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
+; CHECK:     "asm line 1"
+; CHECK:     "asm line 2"
+; CHECK: module asm(target_features: "+bar")
+; CHECK:     "asm line 3"
+; CHECK: module asm(target_cpu: "bar_cpu")
+; CHECK:     "asm line 4"
+; CHECK: module asm
+; CHECK:     "asm line 5"
+
+module asm
+    "asm line 0"
+
+module asm(target_features: "+foo", target_cpu: "foo_cpu")
+    "asm line 1"
+    "asm line 2"
+
+module asm(target_features: "+bar")
+    "asm line 3"
+
+module asm(target_cpu: "bar_cpu")
+    "asm line 4"
+
+module asm
+    "asm line 5"
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
new file mode 100644
index 0000000000000..ab16acab48688
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -0,0 +1,17 @@
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
+
+; This should work fine, because the module asm specifies the necessary
+; target features
+
+; SAME-FEATURES-NOT: .option arch
+; EXTRA-FEATURES: .option push
+; EXTRA-FEATURES: .option arch, +d
+; CHECK: fld ft0, 0(sp)
+; EXTRA-FEATURES: .option pop
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 454c496d1d4bb..829dd829d51f9 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,7 +2,8 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan
diff --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
new file mode 100644
index 0000000000000..73320185e778a
--- /dev/null
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -0,0 +1,18 @@
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
+; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
+; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
+; RUN: FileCheck %s --input-file %t.s.0
+
+; NM: T func
+
+; CHECK: fld ft0, 0(sp)
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+target triple = "riscv64-unknown-linux-gnu"
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 88492e4366489..6c5173c3f9553 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,17 +32,19 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1:      module asm ".lto_discard foo"
-; ASM1-NEXT: module asm ".weak foo"
-; ASM1-NEXT: module asm ".equ foo,bar"
-
-; ASM2:      module asm ".lto_discard foo"
-; ASM2-NEXT: module asm ".weak foo"
-; ASM2-NEXT: module asm ".equ foo,bar"
-; ASM2-NEXT: module asm ".lto_discard"
-; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  module asm ".lto_discard foo"
+; ASM1: module asm
+; ASM1-NEXT: ".lto_discard foo"
+; ASM1-NEXT: ".weak foo"
+; ASM1-NEXT: ".equ foo,bar"
+
+; ASM2: module asm
+; ASM2-NEXT: ".lto_discard foo"
+; ASM2-NEXT: ".weak foo"
+; ASM2-NEXT: ".equ foo,bar"
+; ASM2-NEXT: ".lto_discard"
+; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  ".lto_discard foo"
 
 ; SYM: T foo
 
diff --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
new file mode 100644
index 0000000000000..b097c7067ccc1
--- /dev/null
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -0,0 +1,2 @@
+module asm(target_features: "+bar")
+    "asm 2"
diff --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 5e12a8c88d90a..612bce5be18f1 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,11 +9,12 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK:      module asm ".text"
-; CHECK-NEXT: module asm ".balign 4"
-; CHECK-NEXT: module asm ".arm"
-; CHECK-NEXT: module asm "add r1, r2, r2"
-; CHECK-NEXT: module asm ".text"
-; CHECK-NEXT: module asm ".balign 2"
-; CHECK-NEXT: module asm ".thumb"
-; CHECK-NEXT: module asm "orn r1, r2, r2"
+; CHECK: module asm
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 4"
+; CHECK-NEXT: ".arm"
+; CHECK-NEXT: "add r1, r2, r2"
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 2"
+; CHECK-NEXT: ".thumb"
+; CHECK-NEXT: "orn r1, r2, r2"
diff --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
new file mode 100644
index 0000000000000..6c600981d72a5
--- /dev/null
+++ b/llvm/test/Linker/module-asm.ll
@@ -0,0 +1,9 @@
+; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
+
+; CHECK: module asm(target_features: "+foo")
+; CHECK-NEXT: "asm 1"
+; CHECK: module asm(target_features: "+bar")
+; CHECK-NEXT: "asm 2"
+
+module asm(target_features: "+foo")
+    "asm 1"
diff --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 556c4fd992f0d..9525527b4531c 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,7 +9,8 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm
+; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.
diff --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index ea4594a359cc0..7c3ed3da3537e 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,6 +1,7 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm ".symver external_addrtaken, alias1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index d8ebae17d4693..61141c71335bf 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,7 +3,8 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm
+; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 587ab3fbb5020..5796537661e60 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,6 +7,7 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm "ret"
+; M0: module asm
+; M0-NEXT: "ret"
 ; M1-NOT: module asm
 module asm "ret"
diff --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index 4aeaef6d8d676..b5b68404e8690 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.setModuleInlineAsm("");
+    Program.removeModuleInlineAsm();
 }
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index 3038a6a2ed986..bb55678f91005 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,12 +1150,15 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
-  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
-    if (!line.empty())
-      asmArrayAttr.push_back(builder.getStringAttr(line));
+  for (const llvm::Module::GlobalAsmFragment &Frag :
+       llvmModule->getModuleInlineAsm()) {
+    // TODO: Preserve module asm properties.
+    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
+      if (!line.empty())
+        asmArrayAttr.push_back(builder.getStringAttr(line));
+  }
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));
diff --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 2afb37cf005bf..813eeeafdeb44 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,5 +2,6 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm "foo"
-// CHECK: module asm "bar"
+// CHECK: module asm
+// CHECK-NEXT: "foo"
+// CHECK-NEXT: "bar"

>From e3f8f675b18f74c87270c0c40eb4cb99169a0c98 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Mon, 6 Jul 2026 09:47:31 +0200
Subject: [PATCH 2/2] Fix use after free

The MCSubtargetInfo has to be retained until finalization, as it
may be referenced during relaxation etc.

Handle this the same way we do for function subtargets, by having
a map keyed by features (and other information), which stays live
for the full duration of the TargetMachine.

We can't resue the normal map because it requires a function (and
is more than we need, as we only need the MC subtarget in particular).
---
 llvm/include/llvm/Target/TargetMachine.h   |  9 ++++++++
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp | 25 ++++++----------------
 llvm/lib/Target/TargetMachine.cpp          | 14 ++++++++++++
 3 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/llvm/include/llvm/Target/TargetMachine.h b/llvm/include/llvm/Target/TargetMachine.h
index 8f5721a031aed..d144cb5d72019 100644
--- a/llvm/include/llvm/Target/TargetMachine.h
+++ b/llvm/include/llvm/Target/TargetMachine.h
@@ -114,6 +114,9 @@ class LLVM_ABI TargetMachine {
   std::unique_ptr<const MCInstrInfo> MII;
   std::unique_ptr<const MCSubtargetInfo> STI;
 
+  /// MC subtarget keyed by target features and target CPU.
+  StringMap<std::unique_ptr<const MCSubtargetInfo>> MCSubtargetMap;
+
   unsigned RequireStructuredCFG : 1;
   unsigned O0WantsFastISel : 1;
 
@@ -238,6 +241,12 @@ class LLVM_ABI TargetMachine {
   const MCInstrInfo *getMCInstrInfo() const { return MII.get(); }
   const MCSubtargetInfo &getMCSubtargetInfo() const { return *STI; }
 
+  /// Get the MCSubtargetInfo for the given target CPU and target features.
+  /// For use in contexts where a feature-specific MC subtarget is needed,
+  /// but no MachineFunctionis available, such as for module-level inline
+  /// assembly.
+  const MCSubtargetInfo &getMCSubtargetInfo(StringRef CPU, StringRef FS);
+
   /// Return the ExceptionHandling to use, considering TargetOptions and the
   /// Triple's default.
   ExceptionHandling getExceptionModel() const {
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 8248a05f1a35f..f3ac25f25fcc1 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -614,24 +614,13 @@ bool AsmPrinter::doInitialization(Module &M) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
     for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
-        std::unique_ptr<MCSubtargetInfo> AsmSTI(
-            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
-                                                 Frag.Props.TargetCPU,
-                                                 Frag.Props.TargetFeatures));
-        bool DidPush = emitTargetFeaturePush(*AsmSTI);
-        emitInlineAsm(
-            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-        emitTargetFeaturePop(*AsmSTI, DidPush);
-      } else {
-        // If the module asm does not explicitly specify target features,
-        // fall back to default subtargetinfo.
-        emitInlineAsm(
-            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
-            nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-      }
+      const MCSubtargetInfo &AsmSTI = TM.getMCSubtargetInfo(
+          Frag.Props.TargetCPU, Frag.Props.TargetFeatures);
+      bool DidPush = emitTargetFeaturePush(AsmSTI);
+      emitInlineAsm(
+          Frag.Asm, AsmSTI, TM.Options.MCOptions, nullptr,
+          InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      emitTargetFeaturePop(AsmSTI, DidPush);
     }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
diff --git a/llvm/lib/Target/TargetMachine.cpp b/llvm/lib/Target/TargetMachine.cpp
index 9ec10b5be0fd4..95263f32236b4 100644
--- a/llvm/lib/Target/TargetMachine.cpp
+++ b/llvm/lib/Target/TargetMachine.cpp
@@ -23,6 +23,7 @@
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 using namespace llvm;
@@ -311,3 +312,16 @@ std::pair<int, int> TargetMachine::parseBinutilsVersion(StringRef Version) {
     Version.consumeInteger(10, Ret.second);
   return Ret;
 }
+
+const MCSubtargetInfo &TargetMachine::getMCSubtargetInfo(StringRef CPU,
+                                                         StringRef FS) {
+  if (CPU.empty() && FS.empty())
+    return *STI;
+  SmallString<128> Key = CPU;
+  Key += '/';
+  Key += FS;
+  auto &Entry = MCSubtargetMap[Key];
+  if (!Entry)
+    Entry.reset(getTarget().createMCSubtargetInfo(getTargetTriple(), CPU, FS));
+  return *Entry;
+}


From llvmlistbot at llvm.org  Mon Jul  6 00:53:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:53:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4341c0c - [mlir][memref] Implement
 ValueBoundsOpInterface for memref.extract_strided_metadata/assume_alignment
 (#206466)
Message-ID: <6a4b5f07.f732ad81.1d8ca.5610@mx.google.com>


Author: Hagai Lev Hacohen
Date: 2026-07-06T07:53:39Z
New Revision: 4341c0c9b8b88aac576ce55211ea9db032b74124

URL: https://github.com/llvm/llvm-project/commit/4341c0c9b8b88aac576ce55211ea9db032b74124
DIFF: https://github.com/llvm/llvm-project/commit/4341c0c9b8b88aac576ce55211ea9db032b74124.diff

LOG: [mlir][memref] Implement ValueBoundsOpInterface for memref.extract_strided_metadata/assume_alignment (#206466)

Add ValueBoundsOpInterface external models for memref.assume_alignment
and memref.extract_strided_metadata, so dimension and metadata bounds
can be propagated through these view-like ops during value-bounds
analysis.

Added: 
    

Modified: 
    mlir/lib/Dialect/MemRef/IR/MemRefDialect.cpp
    mlir/lib/Dialect/MemRef/IR/ValueBoundsOpInterfaceImpl.cpp
    mlir/test/Dialect/MemRef/value-bounds-op-interface-impl.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/MemRef/IR/MemRefDialect.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefDialect.cpp
index a1e3f10a871c1..8cecf5a4897e9 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefDialect.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefDialect.cpp
@@ -55,8 +55,9 @@ void mlir::memref::MemRefDialect::initialize() {
   declarePromisedInterfaces<RuntimeVerifiableOpInterface, AssumeAlignmentOp,
                             AtomicRMWOp, CastOp, CopyOp, DimOp, ExpandShapeOp,
                             GenericAtomicRMWOp, LoadOp, StoreOp, SubViewOp>();
-  declarePromisedInterfaces<ValueBoundsOpInterface, AllocOp, AllocaOp, CastOp,
-                            DimOp, GetGlobalOp, RankOp, SubViewOp>();
+  declarePromisedInterfaces<
+      ValueBoundsOpInterface, AllocOp, AllocaOp, AssumeAlignmentOp, CastOp,
+      DimOp, ExtractStridedMetadataOp, GetGlobalOp, RankOp, SubViewOp>();
   declarePromisedInterface<DestructurableTypeInterface, MemRefType>();
 }
 

diff  --git a/mlir/lib/Dialect/MemRef/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/MemRef/IR/ValueBoundsOpInterfaceImpl.cpp
index 69afbcadb0b07..8408b166a6b2a 100644
--- a/mlir/lib/Dialect/MemRef/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -30,6 +30,19 @@ struct AllocOpInterface
   }
 };
 
+struct AssumeAlignmentOpInterface
+    : public ValueBoundsOpInterface::ExternalModel<AssumeAlignmentOpInterface,
+                                                   memref::AssumeAlignmentOp> {
+  void populateBoundsForShapedValueDim(Operation *op, Value value, int64_t dim,
+                                       ValueBoundsConstraintSet &cstr) const {
+    auto assumeAlignmentOp = cast<memref::AssumeAlignmentOp>(op);
+    assert(value == assumeAlignmentOp.getResult() && "invalid value");
+
+    cstr.bound(value)[dim] ==
+        cstr.getExpr(assumeAlignmentOp.getViewSource(), dim);
+  }
+};
+
 struct CastOpInterface
     : public ValueBoundsOpInterface::ExternalModel<CastOpInterface, CastOp> {
   void populateBoundsForShapedValueDim(Operation *op, Value value, int64_t dim,
@@ -70,6 +83,43 @@ struct ExpandShapeOpInterface
   }
 };
 
+struct ExtractStridedMetadataOpInterface
+    : public ValueBoundsOpInterface::ExternalModel<
+          ExtractStridedMetadataOpInterface, memref::ExtractStridedMetadataOp> {
+  void populateBoundsForIndexValue(Operation *op, Value value,
+                                   ValueBoundsConstraintSet &cstr) const {
+    auto metadataOp = cast<memref::ExtractStridedMetadataOp>(op);
+    auto result = llvm::cast<OpResult>(value);
+    assert(result.getOwner() == op && "invalid value");
+    int64_t resultNumber = result.getResultNumber();
+
+    if (resultNumber == 1) {
+      cstr.bound(value) == metadataOp.getConstifiedMixedOffset();
+      return;
+    }
+
+    int64_t sourceRank = metadataOp.getSource().getType().getRank();
+    int64_t sizeStart = 2;
+    int64_t strideStart = sizeStart + sourceRank;
+    if (resultNumber >= sizeStart && resultNumber < strideStart) {
+      int64_t idx = resultNumber - sizeStart;
+      cstr.bound(value) >= 0;
+      cstr.bound(value) == cstr.getExpr(metadataOp.getSource(), idx);
+      return;
+    }
+
+    int64_t strideEnd = strideStart + sourceRank;
+    if (resultNumber >= strideStart && resultNumber < strideEnd) {
+      int64_t idx = resultNumber - strideStart;
+      SmallVector<OpFoldResult> strides =
+          metadataOp.getConstifiedMixedStrides();
+      cstr.bound(value) == strides[idx];
+      return;
+    }
+    llvm_unreachable("unexpected index value from extract_strided_metadata");
+  }
+};
+
 struct GetGlobalOpInterface
     : public ValueBoundsOpInterface::ExternalModel<GetGlobalOpInterface,
                                                    GetGlobalOp> {
@@ -162,5 +212,9 @@ void mlir::memref::registerValueBoundsOpInterfaceExternalModels(
     memref::GetGlobalOp::attachInterface<memref::GetGlobalOpInterface>(*ctx);
     memref::RankOp::attachInterface<memref::RankOpInterface>(*ctx);
     memref::SubViewOp::attachInterface<memref::SubViewOpInterface>(*ctx);
+    memref::AssumeAlignmentOp::attachInterface<
+        memref::AssumeAlignmentOpInterface>(*ctx);
+    memref::ExtractStridedMetadataOp::attachInterface<
+        memref::ExtractStridedMetadataOpInterface>(*ctx);
   });
 }

diff  --git a/mlir/test/Dialect/MemRef/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/MemRef/value-bounds-op-interface-impl.mlir
index d0aec68d54988..32af830aa1ea7 100644
--- a/mlir/test/Dialect/MemRef/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/MemRef/value-bounds-op-interface-impl.mlir
@@ -27,6 +27,22 @@ func.func @memref_alloca(%sz: index) -> (index, index) {
 
 // -----
 
+// CHECK-LABEL: func @memref_assume_alignment(
+//  CHECK-SAME:     %[[sz:.*]]: index
+//       CHECK:   %[[c6:.*]] = arith.constant 6 : index
+//       CHECK:   %[[c1:.*]] = arith.constant 1 : index
+//       CHECK:   %[[dim:.*]] = memref.dim %{{.*}}, %[[c1]] : memref<6x?xf32>
+//       CHECK:   return %[[c6]], %[[dim]]
+func.func @memref_assume_alignment(%sz: index) -> (index, index) {
+  %0 = memref.alloc(%sz) : memref<6x?xf32>
+  %1 = memref.assume_alignment %0, 16 : memref<6x?xf32>
+  %2 = "test.reify_bound"(%1) {dim = 0} : (memref<6x?xf32>) -> (index)
+  %3 = "test.reify_bound"(%1) {dim = 1} : (memref<6x?xf32>) -> (index)
+  return %2, %3 : index, index
+}
+
+// -----
+
 // CHECK-LABEL: func @memref_cast(
 //       CHECK:   %[[c10:.*]] = arith.constant 10 : index
 //       CHECK:   return %[[c10]]
@@ -77,6 +93,29 @@ func.func @memref_expand(%m: memref<?xf32>, %sz: index) -> (index, index) {
 
 // -----
 
+// CHECK-LABEL: func @memref_extract_strided_metadata_static_metadata(
+//  CHECK-SAME:     %[[m:.*]]: memref<4x?xf32, strided<[11, 7], offset: 5>>
+//       CHECK:   %[[c5:.*]] = arith.constant 5 : index
+//       CHECK:   %[[c4:.*]] = arith.constant 4 : index
+//       CHECK:   %[[c1:.*]] = arith.constant 1 : index
+//       CHECK:   %[[dim:.*]] = memref.dim %[[m]], %[[c1]]
+//       CHECK:   %[[c11:.*]] = arith.constant 11 : index
+//       CHECK:   %[[c7:.*]] = arith.constant 7 : index
+//       CHECK:   return %[[c5]], %[[c4]], %[[dim]], %[[c11]], %[[c7]]
+func.func @memref_extract_strided_metadata_static_metadata(
+    %m: memref<4x?xf32, strided<[11, 7], offset: 5>>) -> (index, index, index, index, index) {
+  %base, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %m
+    : memref<4x?xf32, strided<[11, 7], offset: 5>> -> memref<f32>, index, index, index, index, index
+  %0 = "test.reify_bound"(%offset) : (index) -> (index)
+  %1 = "test.reify_bound"(%sizes#0) : (index) -> (index)
+  %2 = "test.reify_bound"(%sizes#1) : (index) -> (index)
+  %3 = "test.reify_bound"(%strides#0) : (index) -> (index)
+  %4 = "test.reify_bound"(%strides#1) : (index) -> (index)
+  return %0, %1, %2, %3, %4 : index, index, index, index, index
+}
+
+// -----
+
 //       CHECK: #[[$MAP:.+]] = affine_map<()[s0] -> (s0 * 2)>
 // CHECK-LABEL: func @memref_collapse(
 //  CHECK-SAME:     %[[sz0:.*]]: index


        

From llvmlistbot at llvm.org  Mon Jul  6 00:53:48 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 00:53:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Add ValueBoundsOpInterface
 external models for the memref operations extract_strided_metadata and
 assume_alignment (PR #206466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206466@github.com>
Message-ID: <6a4b5f0c.449bfb44.233a36.3c46@mx.google.com>


https://github.com/matthias-springer closed https://github.com/llvm/llvm-project/pull/206466

From llvmlistbot at llvm.org  Mon Jul  6 00:54:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 00:54:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Add ValueBoundsOpInterface
 external models for the memref operations extract_strided_metadata and
 assume_alignment (PR #206466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206466@github.com>
Message-ID: <6a4b5f1c.dd2b4bd3.13e95e.37f7@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@HagaiLevHacohen Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/206466

From llvmlistbot at llvm.org  Mon Jul  6 01:01:55 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Mon, 06 Jul 2026 01:01:55 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207677@github.com>
Message-ID: <6a4b60f3.dd2b4bd3.13e95e.177f@mx.google.com>


https://github.com/nikic updated https://github.com/llvm/llvm-project/pull/207677

>From 54b10ceb4b5c64841a1baad873f713b8ca243970 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Fri, 3 Jul 2026 09:59:08 +0200
Subject: [PATCH 1/2] [IR] Explicitly specify target feature for module asm
 (#204548)

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can now be
specified as follows:

    module asm(target_features: "+foo", target_cpu: "bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO scenarios
we don't know what target features to use for parsing the module-level
inline assembly. Now they can be faithfully preserved, even when merging
inline assembly from different modules with different features.

If target_features and target_cpu are empty, we fall back to the old
behavior (which, in the backend, would be to use the default subtarget
of the TargetMachine).

Fixes https://github.com/llvm/llvm-project/issues/61991.
Fixes https://github.com/llvm/llvm-project/issues/67698.
---
 clang/lib/CodeGen/CGObjCMac.cpp               |  6 +-
 clang/lib/CodeGen/CodeGenModule.cpp           |  8 +-
 clang/test/CodeGen/2006-01-23-FileScopeAsm.c  | 17 ++--
 clang/test/CodeGen/asm.c                      |  7 +-
 clang/test/CodeGen/asm_incbin.c               |  3 +-
 clang/test/CodeGen/cfi-salt.c                 | 11 ++-
 clang/test/CodeGen/kcfi.c                     |  7 +-
 clang/test/CodeGenCUDA/filter-decl.cu         |  5 +-
 clang/test/CodeGenCXX/gnu-asm-constexpr.cpp   |  3 +-
 clang/test/CodeGenObjC/category-class.m       |  7 +-
 clang/test/CodeGenObjC/objc-runtime-name.m    |  7 +-
 clang/test/Frontend/ast-codegen.c             |  3 +-
 clang/test/Modules/codegen.test               |  3 +-
 llvm/docs/LangRef.rst                         | 27 ++++--
 llvm/docs/ReleaseNotes.md                     |  4 +
 llvm/include/llvm/Bitcode/LLVMBitCodes.h      |  2 +
 llvm/include/llvm/CodeGen/AsmPrinter.h        | 13 +++
 llvm/include/llvm/IR/Module.h                 | 96 ++++++++++++++++---
 llvm/lib/AsmParser/LLParser.cpp               | 39 +++++++-
 llvm/lib/Bitcode/Reader/BitcodeReader.cpp     | 17 +++-
 llvm/lib/Bitcode/Writer/BitcodeWriter.cpp     | 16 +++-
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp    | 28 ++++--
 llvm/lib/IR/AsmWriter.cpp                     | 43 ++++++---
 llvm/lib/IR/Core.cpp                          | 13 ++-
 llvm/lib/IR/Module.cpp                        | 20 ++++
 llvm/lib/LTO/LTO.cpp                          |  4 +-
 llvm/lib/Linker/IRMover.cpp                   |  9 +-
 llvm/lib/Object/ModuleSymbolTable.cpp         | 90 ++++++++---------
 llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp     | 23 +++--
 llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp     |  2 +-
 llvm/lib/Transforms/IPO/ExtractGV.cpp         |  2 +-
 .../Transforms/IPO/ThinLTOBitcodeWriter.cpp   |  2 +-
 .../Instrumentation/DataFlowSanitizer.cpp     | 26 ++---
 llvm/lib/Transforms/Utils/SplitModule.cpp     |  2 +-
 llvm/test/Assembler/module-asm-invalid.ll     | 25 +++++
 llvm/test/Bitcode/compatibility-3.6.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.7.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.8.ll        |  3 +-
 llvm/test/Bitcode/compatibility-3.9.ll        |  3 +-
 llvm/test/Bitcode/compatibility-4.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-5.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility-6.0.ll        |  3 +-
 llvm/test/Bitcode/compatibility.ll            |  3 +-
 llvm/test/Bitcode/highLevelStructure.3.2.ll   |  3 +-
 llvm/test/Bitcode/module-asm.ll               | 29 ++++++
 .../test/CodeGen/RISCV/module-asm-features.ll | 17 ++++
 .../DataFlowSanitizer/prefix-rename.ll        |  3 +-
 llvm/test/LTO/RISCV/module-asm.ll             | 18 ++++
 llvm/test/LTO/X86/inline-asm-lto-discard.ll   | 24 ++---
 llvm/test/Linker/Inputs/module-asm.ll         |  2 +
 .../link-arm-and-thumb-module-inline-asm.ll   | 17 ++--
 llvm/test/Linker/module-asm.ll                |  9 ++
 llvm/test/ThinLTO/X86/import-symver.ll        |  3 +-
 .../LowerTypeTests/export-symver.ll           |  3 +-
 .../cfi-icall-static-inline-asm.ll            |  3 +-
 .../ThinLTOBitcodeWriter/x86/module-asm.ll    |  3 +-
 .../llvm-reduce/deltas/ReduceModuleData.cpp   |  2 +-
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 11 ++-
 mlir/test/Target/LLVMIR/module-asm.mlir       |  5 +-
 59 files changed, 573 insertions(+), 193 deletions(-)
 create mode 100644 llvm/test/Assembler/module-asm-invalid.ll
 create mode 100644 llvm/test/Bitcode/module-asm.ll
 create mode 100644 llvm/test/CodeGen/RISCV/module-asm-features.ll
 create mode 100644 llvm/test/LTO/RISCV/module-asm.ll
 create mode 100644 llvm/test/Linker/Inputs/module-asm.ll
 create mode 100644 llvm/test/Linker/module-asm.ll

diff --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index 69c5e88f3c768..e518b875f44bb 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,10 +6610,6 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
-    Asm += CGM.getModule().getModuleInlineAsm();
-    if (!Asm.empty() && Asm.back() != '\n')
-      Asm += '\n';
-
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6624,7 +6620,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().setModuleInlineAsm(OS.str());
+    CGM.getModule().appendModuleInlineAsm(OS.str());
   }
 }
 
diff --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 813cd67e4dd51..e155fdd752d7f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,7 +8057,13 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-    getModule().appendModuleInlineAsm(AD->getAsmString());
+
+    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
+    llvm::Module::GlobalAsmProperties Props;
+    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
+    Props.TargetCPU = TargetOpts.CPU;
+    getModule().appendModuleInlineAsm(
+        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
     break;
   }
 
diff --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 877da4fc669e3..8a212a0b61905 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,13 +1,18 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
+// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
 
-// CHECK: module asm "foo1"
+// RISCV64: module asm(target_features: "+64bit,+i")
+// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
+// DEFAULT: module asm
+// CHECK-NEXT: "foo1"
 __asm__ ("foo1");
-// CHECK: module asm "foo2"
+// CHECK-NEXT: "foo2"
 __asm__ ("foo2");
-// CHECK: module asm "foo3"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo3");
-// CHECK: module asm "foo4"
+// CHECK-NEXT: "foo4"
 __asm__ ("foo4");
-// CHECK: module asm "foo5"
+// CHECK-NEXT: "foo5"
 __asm__ ("foo5");
diff --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index d7465b22fbbf6..ea69c7a17d1cb 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,9 +2,10 @@
 
 // PR10415:
 //
-// CHECK:      module asm "foo1"
-// CHECK-NEXT: module asm "foo2"
-// CHECK-NEXT: module asm "foo3"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo1"
+// CHECK-NEXT: "foo2"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");
diff --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index 6234eea1d6738..bbc5d3af55ea2 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,4 +6,5 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm ".incbin \22foo.h\22"
+// CHECK: module asm
+// CHECK-NEXT: ".incbin \22foo.h\22"
diff --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 2a90dc31d939b..18156dc014280 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,13 +26,14 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
+// CHECK-NOT: ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;
diff --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index da32859dd3d88..f08cf7635fb5a 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,10 +6,11 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2
diff --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index 02dacd0ad8ef4..f87d62a56631e 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,8 +5,9 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm "file scope asm is host only"
-// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
+// CHECK-HOST: module asm
+// CHECK-HOST-NEXT: "file scope asm is host only"
+// CHECK-DEVICE-NOT: "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global
diff --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 175a3b7bc588c..2b4abee5dc05f 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,8 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm "nop; nop; nop; nop"
+    // CHECK: module asm(target_features: "{{.*}}")
+    // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
 int func() {return 0;};
diff --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index 92fd36cbe4066..febe94ce5b9f5 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,9 +1,10 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
-// CHECK: module asm "\09.objc_category_name_A_foo=0"
-// CHECK: module asm "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm
+// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
+// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
+// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end
diff --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index c838b8c2c6f1a..a7f59caced390 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,9 +3,10 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm {{.*}}objc_class_name_XYZ=0
-// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
-// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm
+// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
+// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
+// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
diff --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index 2e2e3d360dd17..c26b0a1908379 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,8 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm "foo"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4
diff --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 0af630a754805..5cc5b3c1cda2a 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,7 +18,8 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm "narf"
+BOTH: module asm
+BOTH-NEXT: "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 5ce6019660e0b..2c8276d04b668 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3357,19 +3357,34 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks. These blocks are internally
-concatenated by LLVM and treated as a single unit, but may be separated
-in the ``.ll`` file if desired. The syntax is very simple:
+to the GCC "file scope inline asm" blocks:
 
 .. code-block:: llvm
 
-    module asm "inline asm code goes here"
-    module asm "more can go here"
+    module asm
+        "inline asm code goes here"
+        "more can go here"
 
-The strings can contain any character by escaping non-printable
+The sequence of string literals is internally concatenated using newlines as
+separators. The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
+Additionally, module-level inline assembly can specify an optional list of
+properties:
+
+.. code-block:: llvm
+
+    module asm(target_features: "+foo", target_cpu: "bar")
+        "inline asm code goes here"
+        "more can go here"
+
+Currently, the only supported properties are ``target_features`` and
+``target_cpu``, with the same meaning as the ``"target-features"`` and
+``"target-cpu"`` function attributes.
+
+Consecutive blocks with identical properties will be concatenated into one.
+
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 11d3ebc80fca6..3ca9f5166ad95 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -114,6 +114,10 @@ Makes programs 10x faster by doing Special New Thing.
 * `alwaysinline` no longer bypasses inlining compatibility checks based on
   target features. Inlining will only be performed if it is safe to do so.
 
+* Module-level inline assembly now accepts optional `target_features` and
+  `target_cpu` properties. This resolves errors during LTO on some
+  architectures.
+
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to
diff --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index af4cac996a78c..1761e7a542a73 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,6 +120,8 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
+
+  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.
diff --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 78e64440657cc..335bf9de9a071 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,6 +930,19 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
+  /// Emit necessary directives to allow use of instructions that are permitted
+  /// by target features enabled by STI, but are not permitted by target
+  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
+  /// Returns whether anything was emitted.
+  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+    return false;
+  }
+
+  /// Emit necessary directives to restore target feature state.
+  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
+  /// call.
+  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
+
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,
diff --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 4f7e33969f16f..63d1013896b47 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,6 +169,45 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
+  struct GlobalAsmProperties {
+    std::string TargetFeatures;
+    std::string TargetCPU;
+
+    /// Set a property using a string name.
+    /// Returns whether the property name was valid.
+    LLVM_ABI bool set(StringRef Name, std::string Value);
+
+    /// Get a list of set properties as pairs of key and value.
+    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+
+    bool operator==(const GlobalAsmProperties &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+
+    bool operator!=(const GlobalAsmProperties &Other) const {
+      return !(*this == Other);
+    }
+  };
+
+  struct GlobalAsmFragment {
+    std::string Asm;
+    GlobalAsmProperties Props;
+
+    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
+    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
+        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
+      if (!Asm.empty() && Asm.back() != '\n')
+        Asm += '\n';
+    }
+
+    bool empty() const { return Asm.empty(); }
+
+    bool hasSameProperties(const GlobalAsmFragment &Other) const {
+      return Props == Other.Props;
+    }
+  };
+
 /// @}
 /// @name Member Variables
 /// @{
@@ -180,7 +219,8 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
+  /// Inline Asm at the global scope.
+  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -287,8 +327,17 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  /// @returns a string containing the module-scope inline assembly blocks.
-  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
+  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
+    return GlobalScopeAsm;
+  }
+
+  /// Get any module-scope inline assembly blocks.
+  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
+    return GlobalScopeAsm;
+  }
+
+  /// Return whether there is any module-scope inline assembly.
+  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -325,20 +374,45 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
+  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
+
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm = std::string(Asm);
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
+    GlobalScopeAsm.clear();
+    appendModuleInlineAsm(std::move(Fragment));
+  }
+
+  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
+    GlobalScopeAsm.clear();
+    append_range(GlobalScopeAsm, Fragments);
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm += Asm;
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.back().Asm += Fragment.Asm;
+    } else {
+      GlobalScopeAsm.emplace_back(std::move(Fragment));
+    }
+  }
+
+  /// Prepend to the module-scope inline assembly blocks.
+  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
+    } else {
+      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
+    }
   }
 
 /// @}
diff --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 75e6add0ec76f..0a5b4c6c91879 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,16 +641,49 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
+///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
+///                          'property_name2:' STRINGCONSTANT ')'
+///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
-      parseStringConstant(AsmStr))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  M->appendModuleInlineAsm(AsmStr);
+  Module::GlobalAsmProperties Props;
+  if (EatIfPresent(lltok::lparen)) {
+    while (true) {
+      std::string Key, Value;
+      SMLoc Loc = Lex.getLoc();
+      if (Lex.getKind() != lltok::LabelStr)
+        return error(Loc, "expected property name followed by ':'");
+
+      Key = Lex.getStrVal();
+      Lex.Lex();
+
+      if (parseStringConstant(Value))
+        return true;
+
+      if (!Props.set(Key, Value))
+        return error(Loc, "unknown property name");
+
+      if (EatIfPresent(lltok::rparen))
+        break;
+      if (parseToken(lltok::comma, "expected ',' or ')'"))
+        return true;
+    }
+  }
+
+  do {
+    std::string AsmStrPart;
+    if (parseStringConstant(AsmStrPart))
+      return true;
+    AsmStr += AsmStrPart + "\n";
+  } while (Lex.getKind() == lltok::StringConstant);
+
+  M->appendModuleInlineAsm({AsmStr, Props});
   return false;
 }
 
diff --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 2bd251efd05ce..06e9aea4fd3ac 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4580,6 +4580,9 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
+  // Apply to the following module asm.
+  Module::GlobalAsmProperties Props;
+
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4788,11 +4791,23 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
+    case bitc::MODULE_CODE_ASM_PROPERTY: {
+      std::string Str;
+      if (convertToString(Record, 0, Str))
+        return error("Invalid module asm record");
+      size_t SepPos = Str.find('\0');
+      if (SepPos == std::string::npos)
+        return error("Invalid module asm record");
+      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
+        return error("Unknown module asm property");
+      break;
+    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->setModuleInlineAsm(S);
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
+      Props = {};
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]
diff --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 4aa203b0d12d1..493cbec7506f2 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,9 +1551,19 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-  if (!M.getModuleInlineAsm().empty())
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
-                      0 /*TODO*/);
+
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    SmallVector<std::pair<StringRef, StringRef>> Props =
+        Frag.Props.getAsStrings();
+    for (auto [Key, Value] : Props) {
+      SmallVector<unsigned, 64> Record;
+      Record.append(Key.begin(), Key.end());
+      Record.push_back(0);
+      Record.append(Value.begin(), Value.end());
+      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
+    }
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
+  }
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index abb20a7919fa1..8248a05f1a35f 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,6 +110,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -127,7 +128,6 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,13 +610,29 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    emitInlineAsm(
-        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
-        TM.Options.MCOptions, nullptr,
-        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
+        std::unique_ptr<MCSubtargetInfo> AsmSTI(
+            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
+                                                 Frag.Props.TargetCPU,
+                                                 Frag.Props.TargetFeatures));
+        bool DidPush = emitTargetFeaturePush(*AsmSTI);
+        emitInlineAsm(
+            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+        emitTargetFeaturePop(*AsmSTI, DidPush);
+      } else {
+        // If the module asm does not explicitly specify target features,
+        // fall back to default subtargetinfo.
+        emitInlineAsm(
+            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
+            nullptr,
+            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      }
+    }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }
diff --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 665d7122af2c7..02d6bcd90add0 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,21 +3126,38 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (!M->getModuleInlineAsm().empty()) {
+  if (M->hasModuleInlineAsm()) {
     Out << '\n';
 
-    // Split the string into lines, to make it easier to read the .ll file.
-    StringRef Asm = M->getModuleInlineAsm();
-    do {
-      StringRef Front;
-      std::tie(Front, Asm) = Asm.split('\n');
-
-      // We found a newline, print the portion of the asm string from the
-      // last newline up to this newline.
-      Out << "module asm \"";
-      printEscapedString(Front, Out);
-      Out << "\"\n";
-    } while (!Asm.empty());
+    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
+      Out << "module asm";
+      SmallVector<std::pair<StringRef, StringRef>> Props =
+          Frag.Props.getAsStrings();
+      if (!Props.empty()) {
+        ListSeparator LS;
+        Out << "(";
+        for (auto [Key, Value] : Props) {
+          Out << LS;
+          Out << Key << ": \"";
+          printEscapedString(Value, Out);
+          Out << "\"";
+        }
+        Out << ")";
+      }
+      Out << "\n";
+      // Split the string into lines, to make it easier to read the .ll file.
+      StringRef Asm = Frag.Asm;
+      do {
+        StringRef Front;
+        std::tie(Front, Asm) = Asm.split('\n');
+
+        // We found a newline, print the portion of the asm string from the
+        // last newline up to this newline.
+        Out << "    \"";
+        printEscapedString(Front, Out);
+        Out << "\"\n";
+      } while (!Asm.empty());
+    }
   }
 
   printTypeIdentities();
diff --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index 6c6c3b5c84ede..a7abd3eed31c3 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,7 +510,18 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  auto &Str = unwrap(M)->getModuleInlineAsm();
+  Module *Mod = unwrap(M);
+  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
+  if (Frags.empty()) {
+    *Len = 0;
+    return nullptr;
+  }
+
+  if (Frags.size() != 1)
+    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
+                          "more than one module inline assembly fragment");
+
+  auto &Str = Frags.begin()->Asm;
   *Len = Str.length();
   return Str.c_str();
 }
diff --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 40ef7e354da72..34e44e26d5db5 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,3 +977,23 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
+
+bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
+  if (Name == "target_features")
+    TargetFeatures = std::move(Value);
+  else if (Name == "target_cpu")
+    TargetCPU = std::move(Value);
+  else
+    return false;
+  return true;
+}
+
+SmallVector<std::pair<StringRef, StringRef>>
+Module::GlobalAsmProperties::getAsStrings() const {
+  SmallVector<std::pair<StringRef, StringRef>> Props;
+  if (!TargetFeatures.empty())
+    Props.emplace_back("target_features", TargetFeatures);
+  if (!TargetCPU.empty())
+    Props.emplace_back("target_cpu", TargetCPU);
+  return Props;
+}
diff --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 6a754f95092bb..b672f4e5fac14 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
+    M.prependModuleInlineAsm(NewIA);
   }
 
   assert(MsymI == MsymE);
diff --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde1cec..3b72b412d0b2e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,11 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
+      Module::GlobalAsmFragment NewFrag(Frag);
+      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
+      DstM.appendModuleInlineAsm(std::move(NewFrag));
+    }
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1573,7 +1576,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(S);
+        DstM.appendModuleInlineAsm(std::string(S));
       }
     });
   }
diff --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 1da5fa9c10a0b..44f717ec12323 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,8 +72,7 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  StringRef InlineAsm = M.getModuleInlineAsm();
-  if (InlineAsm.empty())
+  if (!M.hasModuleInlineAsm())
     return;
 
   std::string Err;
@@ -90,52 +89,55 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
-  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
-  if (!STI)
-    return;
-
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  std::unique_ptr<MemoryBuffer> Buffer(
-      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
-  SourceMgr SrcMgr;
-  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-  std::unique_ptr<MCObjectFileInfo> MOFI(
-      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-  MCCtx.setObjectFileInfo(MOFI.get());
-  RecordStreamer Streamer(MCCtx, M);
-  T->createNullTargetStreamer(Streamer);
-
-  std::unique_ptr<MCAsmParser> Parser(
-      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-  std::unique_ptr<MCTargetAsmParser> TAP(
-      T->createMCAsmParser(*STI, *Parser, *MCII));
-  if (!TAP)
-    return;
-
-  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                 const SourceMgr &SrcMgr,
-                                 std::vector<const MDNode *> &LocInfos) {
-    M.getContext().diagnose(
-        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-  });
-
-  // Module-level inline asm is assumed to use At&t syntax (see
-  // AsmPrinter::doInitialization()).
-  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-  Parser->setSymbolScanningMode(true);
-
-  Parser->setTargetParser(*TAP);
-  if (Parser->Run(false))
-    return;
-
-  Init(Streamer);
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
+        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
+    if (!STI)
+      return;
+
+    std::unique_ptr<MemoryBuffer> Buffer(
+        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
+    SourceMgr SrcMgr;
+    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+    std::unique_ptr<MCObjectFileInfo> MOFI(
+        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+    MCCtx.setObjectFileInfo(MOFI.get());
+    RecordStreamer Streamer(MCCtx, M);
+    T->createNullTargetStreamer(Streamer);
+
+    std::unique_ptr<MCAsmParser> Parser(
+        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+    std::unique_ptr<MCTargetAsmParser> TAP(
+        T->createMCAsmParser(*STI, *Parser, *MCII));
+    if (!TAP)
+      return;
+
+    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                   const SourceMgr &SrcMgr,
+                                   std::vector<const MDNode *> &LocInfos) {
+      M.getContext().diagnose(
+          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+    });
+
+    // Module-level inline asm is assumed to use At&t syntax (see
+    // AsmPrinter::doInitialization()).
+    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+    Parser->setSymbolScanningMode(true);
+
+    Parser->setTargetParser(*TAP);
+    if (Parser->Run(false))
+      return;
+
+    Init(Streamer);
+  }
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(
diff --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index dbda25c34cee0..e37ea27364e6e 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,7 +113,8 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitDirectiveOptionArch();
+  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
+  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -523,19 +524,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitDirectiveOptionArch() {
+bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                                : RISCVOptionArchArgType::Minus;
+    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                               : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -549,11 +550,16 @@ bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   return false;
 }
 
+void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
+                                           bool DidPush) {
+  if (DidPush)
+    getTargetStreamer().emitDirectiveOptionPop();
+}
+
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
-  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitDirectiveOptionArch();
+  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -561,8 +567,7 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  if (EmittedOptionArch)
-    RTS.emitDirectiveOptionPop();
+  emitTargetFeaturePop(*STI, EmittedOptionArch);
   return false;
 }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 2a5d16a470c4a..22d0d97090746 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
   }
 
   // Register the NSDI handler before calling the base class so that
diff --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 16fdb93e2b21b..4b76b02c32fe4 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually
diff --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index a486979a58c2d..e857c94330bad 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->setModuleInlineAsm("");
+  MergedM->removeModuleInlineAsm();
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.
diff --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index c349fe33dd237..375ac069272eb 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,18 +1283,20 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  std::string Asm = GV->getParent()->getModuleInlineAsm();
-  std::string SearchStr = ".symver " + GVName + ",";
-  size_t Pos = Asm.find(SearchStr);
-  if (Pos != std::string::npos) {
-    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
-    Pos = Asm.find('@');
-
-    if (Pos == std::string::npos)
-      report_fatal_error(Twine("unsupported .symver: ", Asm));
-
-    Asm.replace(Pos, 1, Suffix + "@");
-    GV->getParent()->setModuleInlineAsm(Asm);
+  for (Module::GlobalAsmFragment &Frag :
+       GV->getParent()->getModuleInlineAsm()) {
+    std::string SearchStr = ".symver " + GVName + ",";
+    size_t Pos = Frag.Asm.find(SearchStr);
+    if (Pos != std::string::npos) {
+      Frag.Asm.replace(Pos, SearchStr.size(),
+                       ".symver " + GVName + Suffix + ",");
+      Pos = Frag.Asm.find('@');
+
+      if (Pos == std::string::npos)
+        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
+
+      Frag.Asm.replace(Pos, 1, Suffix + "@");
+    }
   }
 }
 
diff --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 64910b4d1ce99..8e70902c4dac2 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->setModuleInlineAsm("");
+      MPart->removeModuleInlineAsm();
     ModuleCallback(std::move(MPart));
   }
 }
diff --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
new file mode 100644
index 0000000000000..34e5b71e683a9
--- /dev/null
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -0,0 +1,25 @@
+; RUN: split-file %s %t
+; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
+; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
+; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
+; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
+
+;--- invalid_prop.ll
+; CHECK: unknown property name
+module asm(foo: "bar")
+    "asm"
+
+;--- missing_value1.ll
+; CHECK: expected property name followed by ':'
+module asm(target_features)
+    "asm"
+
+;--- missing_value2.ll
+; CHECK: expected string constant
+module asm(target_features:)
+    "asm"
+
+;--- missing_paren.ll
+; CHECK: expected ',' or ')'
+module asm(target_features: "bar"
+    "asm"
diff --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index 288e0fc50a9ba..c79086bad1ba5 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index ef9490310fe5d..00b2bb9b30b62 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 538d6101b3994..5b86d5ffa1fee 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index 3bfef73e485dd..eca4bbdf03766 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index aaf9fe5492171..8551b98815b1a 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index b820417107e8b..883ac3eaaa293 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 67e96c7277cc3..624d5bd7703c6 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index d55aa1dc496df..97889d4687516 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,7 +16,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any
diff --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index f2770ea4e4d34..cd9a34c810d57 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,7 +9,8 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm "some assembly"
+; CHECK: module asm
+; CHECK-NEXT: "some assembly"
 module asm "some assembly"
 
 ; Named Types Test
diff --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
new file mode 100644
index 0000000000000..406f25c5dd28c
--- /dev/null
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -0,0 +1,29 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; CHECK: module asm
+; CHECK:     "asm line 0"
+; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
+; CHECK:     "asm line 1"
+; CHECK:     "asm line 2"
+; CHECK: module asm(target_features: "+bar")
+; CHECK:     "asm line 3"
+; CHECK: module asm(target_cpu: "bar_cpu")
+; CHECK:     "asm line 4"
+; CHECK: module asm
+; CHECK:     "asm line 5"
+
+module asm
+    "asm line 0"
+
+module asm(target_features: "+foo", target_cpu: "foo_cpu")
+    "asm line 1"
+    "asm line 2"
+
+module asm(target_features: "+bar")
+    "asm line 3"
+
+module asm(target_cpu: "bar_cpu")
+    "asm line 4"
+
+module asm
+    "asm line 5"
diff --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
new file mode 100644
index 0000000000000..ab16acab48688
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -0,0 +1,17 @@
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
+
+; This should work fine, because the module asm specifies the necessary
+; target features
+
+; SAME-FEATURES-NOT: .option arch
+; EXTRA-FEATURES: .option push
+; EXTRA-FEATURES: .option arch, +d
+; CHECK: fld ft0, 0(sp)
+; EXTRA-FEATURES: .option pop
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 454c496d1d4bb..829dd829d51f9 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,7 +2,8 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan
diff --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
new file mode 100644
index 0000000000000..73320185e778a
--- /dev/null
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -0,0 +1,18 @@
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
+; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
+; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
+; RUN: FileCheck %s --input-file %t.s.0
+
+; NM: T func
+
+; CHECK: fld ft0, 0(sp)
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+target triple = "riscv64-unknown-linux-gnu"
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"
diff --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 88492e4366489..6c5173c3f9553 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,17 +32,19 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1:      module asm ".lto_discard foo"
-; ASM1-NEXT: module asm ".weak foo"
-; ASM1-NEXT: module asm ".equ foo,bar"
-
-; ASM2:      module asm ".lto_discard foo"
-; ASM2-NEXT: module asm ".weak foo"
-; ASM2-NEXT: module asm ".equ foo,bar"
-; ASM2-NEXT: module asm ".lto_discard"
-; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  module asm ".lto_discard foo"
+; ASM1: module asm
+; ASM1-NEXT: ".lto_discard foo"
+; ASM1-NEXT: ".weak foo"
+; ASM1-NEXT: ".equ foo,bar"
+
+; ASM2: module asm
+; ASM2-NEXT: ".lto_discard foo"
+; ASM2-NEXT: ".weak foo"
+; ASM2-NEXT: ".equ foo,bar"
+; ASM2-NEXT: ".lto_discard"
+; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  ".lto_discard foo"
 
 ; SYM: T foo
 
diff --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
new file mode 100644
index 0000000000000..b097c7067ccc1
--- /dev/null
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -0,0 +1,2 @@
+module asm(target_features: "+bar")
+    "asm 2"
diff --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 5e12a8c88d90a..612bce5be18f1 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,11 +9,12 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK:      module asm ".text"
-; CHECK-NEXT: module asm ".balign 4"
-; CHECK-NEXT: module asm ".arm"
-; CHECK-NEXT: module asm "add r1, r2, r2"
-; CHECK-NEXT: module asm ".text"
-; CHECK-NEXT: module asm ".balign 2"
-; CHECK-NEXT: module asm ".thumb"
-; CHECK-NEXT: module asm "orn r1, r2, r2"
+; CHECK: module asm
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 4"
+; CHECK-NEXT: ".arm"
+; CHECK-NEXT: "add r1, r2, r2"
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 2"
+; CHECK-NEXT: ".thumb"
+; CHECK-NEXT: "orn r1, r2, r2"
diff --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
new file mode 100644
index 0000000000000..6c600981d72a5
--- /dev/null
+++ b/llvm/test/Linker/module-asm.ll
@@ -0,0 +1,9 @@
+; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
+
+; CHECK: module asm(target_features: "+foo")
+; CHECK-NEXT: "asm 1"
+; CHECK: module asm(target_features: "+bar")
+; CHECK-NEXT: "asm 2"
+
+module asm(target_features: "+foo")
+    "asm 1"
diff --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 556c4fd992f0d..9525527b4531c 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,7 +9,8 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm
+; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.
diff --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index ea4594a359cc0..7c3ed3da3537e 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,6 +1,7 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm ".symver external_addrtaken, alias1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index d8ebae17d4693..61141c71335bf 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,7 +3,8 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm
+; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8
diff --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 587ab3fbb5020..5796537661e60 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,6 +7,7 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm "ret"
+; M0: module asm
+; M0-NEXT: "ret"
 ; M1-NOT: module asm
 module asm "ret"
diff --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index 4aeaef6d8d676..b5b68404e8690 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.setModuleInlineAsm("");
+    Program.removeModuleInlineAsm();
 }
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index 3038a6a2ed986..bb55678f91005 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,12 +1150,15 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
-  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
-    if (!line.empty())
-      asmArrayAttr.push_back(builder.getStringAttr(line));
+  for (const llvm::Module::GlobalAsmFragment &Frag :
+       llvmModule->getModuleInlineAsm()) {
+    // TODO: Preserve module asm properties.
+    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
+      if (!line.empty())
+        asmArrayAttr.push_back(builder.getStringAttr(line));
+  }
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));
diff --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 2afb37cf005bf..813eeeafdeb44 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,5 +2,6 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm "foo"
-// CHECK: module asm "bar"
+// CHECK: module asm
+// CHECK-NEXT: "foo"
+// CHECK-NEXT: "bar"

>From 7707e5b79529c63fa66f89d4991111c676b18352 Mon Sep 17 00:00:00 2001
From: Nikita Popov <npopov at redhat.com>
Date: Mon, 6 Jul 2026 09:47:31 +0200
Subject: [PATCH 2/2] Fix use after free

The MCSubtargetInfo has to be retained until finalization, as it
may be referenced during relaxation etc.

Handle this the same way we do for function subtargets, by having
a map keyed by features (and other information), which stays live
for the full duration of the TargetMachine.

We can't resue the normal map because it requires a function (and
is more than we need, as we only need the MC subtarget in particular).
---
 llvm/include/llvm/Target/TargetMachine.h   | 10 +++++++++
 llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp | 25 ++++++----------------
 llvm/lib/Target/TargetMachine.cpp          | 14 ++++++++++++
 3 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/llvm/include/llvm/Target/TargetMachine.h b/llvm/include/llvm/Target/TargetMachine.h
index 8f5721a031aed..6fdd603092fc6 100644
--- a/llvm/include/llvm/Target/TargetMachine.h
+++ b/llvm/include/llvm/Target/TargetMachine.h
@@ -13,6 +13,7 @@
 #ifndef LLVM_TARGET_TARGETMACHINE_H
 #define LLVM_TARGET_TARGETMACHINE_H
 
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/PassManager.h"
@@ -114,6 +115,9 @@ class LLVM_ABI TargetMachine {
   std::unique_ptr<const MCInstrInfo> MII;
   std::unique_ptr<const MCSubtargetInfo> STI;
 
+  /// MC subtarget keyed by target features and target CPU.
+  StringMap<std::unique_ptr<const MCSubtargetInfo>> MCSubtargetMap;
+
   unsigned RequireStructuredCFG : 1;
   unsigned O0WantsFastISel : 1;
 
@@ -238,6 +242,12 @@ class LLVM_ABI TargetMachine {
   const MCInstrInfo *getMCInstrInfo() const { return MII.get(); }
   const MCSubtargetInfo &getMCSubtargetInfo() const { return *STI; }
 
+  /// Get the MCSubtargetInfo for the given target CPU and target features.
+  /// For use in contexts where a feature-specific MC subtarget is needed,
+  /// but no MachineFunctionis available, such as for module-level inline
+  /// assembly.
+  const MCSubtargetInfo &getMCSubtargetInfo(StringRef CPU, StringRef FS);
+
   /// Return the ExceptionHandling to use, considering TargetOptions and the
   /// Triple's default.
   ExceptionHandling getExceptionModel() const {
diff --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index 8248a05f1a35f..f3ac25f25fcc1 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -614,24 +614,13 @@ bool AsmPrinter::doInitialization(Module &M) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
     for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
-      if (!Frag.Props.TargetFeatures.empty() || !Frag.Props.TargetCPU.empty()) {
-        std::unique_ptr<MCSubtargetInfo> AsmSTI(
-            TM.getTarget().createMCSubtargetInfo(TM.getTargetTriple(),
-                                                 Frag.Props.TargetCPU,
-                                                 Frag.Props.TargetFeatures));
-        bool DidPush = emitTargetFeaturePush(*AsmSTI);
-        emitInlineAsm(
-            Frag.Asm, *AsmSTI, TM.Options.MCOptions, nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-        emitTargetFeaturePop(*AsmSTI, DidPush);
-      } else {
-        // If the module asm does not explicitly specify target features,
-        // fall back to default subtargetinfo.
-        emitInlineAsm(
-            Frag.Asm + "\n", TM.getMCSubtargetInfo(), TM.Options.MCOptions,
-            nullptr,
-            InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
-      }
+      const MCSubtargetInfo &AsmSTI = TM.getMCSubtargetInfo(
+          Frag.Props.TargetCPU, Frag.Props.TargetFeatures);
+      bool DidPush = emitTargetFeaturePush(AsmSTI);
+      emitInlineAsm(
+          Frag.Asm, AsmSTI, TM.Options.MCOptions, nullptr,
+          InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      emitTargetFeaturePop(AsmSTI, DidPush);
     }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
diff --git a/llvm/lib/Target/TargetMachine.cpp b/llvm/lib/Target/TargetMachine.cpp
index 9ec10b5be0fd4..95263f32236b4 100644
--- a/llvm/lib/Target/TargetMachine.cpp
+++ b/llvm/lib/Target/TargetMachine.cpp
@@ -23,6 +23,7 @@
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 using namespace llvm;
@@ -311,3 +312,16 @@ std::pair<int, int> TargetMachine::parseBinutilsVersion(StringRef Version) {
     Version.consumeInteger(10, Ret.second);
   return Ret;
 }
+
+const MCSubtargetInfo &TargetMachine::getMCSubtargetInfo(StringRef CPU,
+                                                         StringRef FS) {
+  if (CPU.empty() && FS.empty())
+    return *STI;
+  SmallString<128> Key = CPU;
+  Key += '/';
+  Key += FS;
+  auto &Entry = MCSubtargetMap[Key];
+  if (!Entry)
+    Entry.reset(getTarget().createMCSubtargetInfo(getTargetTriple(), CPU, FS));
+  return *Entry;
+}


From llvmlistbot at llvm.org  Mon Jul  6 01:28:45 2026
From: llvmlistbot at llvm.org (Dmitry Chigarev)
Date: Mon, 06 Jul 2026 01:28:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix blocking pass for scf.if
	distribution (PR #207060)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207060@github.com>
Message-ID: <6a4b673d.bb985344.17fb3b.46fc@mx.google.com>


https://github.com/dchigarev approved this pull request.


https://github.com/llvm/llvm-project/pull/207060

From llvmlistbot at llvm.org  Mon Jul  6 01:43:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 01:43:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] QuantileType relax quantileType conditions
 and inheritance issue (PR #204793)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204793@github.com>
Message-ID: <6a4b6ab4.8e75e0cb.2e0812.7b65@mx.google.com>


vsimion26 wrote:

@javedabsar1 gentle ping on this PR. Changes are in place

https://github.com/llvm/llvm-project/pull/204793

From llvmlistbot at llvm.org  Mon Jul  6 01:44:31 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 01:44:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add normalize pass to MLIR (PR #186647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/186647@github.com>
Message-ID: <6a4b6aef.7d41f3a1.d1cf7.8d31@mx.google.com>


https://github.com/linuxlonelyeagle updated https://github.com/llvm/llvm-project/pull/186647

>From d6f66c9c22d70eb3722b8e520331e872234467af Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Fri, 13 Mar 2026 02:26:50 +0000
Subject: [PATCH 1/4] complete basic normalize.

---
 mlir/include/mlir/Transforms/Passes.h  |   1 +
 mlir/include/mlir/Transforms/Passes.td |   8 ++
 mlir/lib/Transforms/CMakeLists.txt     |   1 +
 mlir/lib/Transforms/Normalize.cpp      | 112 +++++++++++++++++++++++++
 mlir/test/Transforms/normalize.mlir    |  84 +++++++++++++++++++
 5 files changed, 206 insertions(+)
 create mode 100644 mlir/lib/Transforms/Normalize.cpp
 create mode 100644 mlir/test/Transforms/normalize.mlir

diff --git a/mlir/include/mlir/Transforms/Passes.h b/mlir/include/mlir/Transforms/Passes.h
index 313755fde3479..be56a548f63b1 100644
--- a/mlir/include/mlir/Transforms/Passes.h
+++ b/mlir/include/mlir/Transforms/Passes.h
@@ -42,6 +42,7 @@ class GreedyRewriteConfig;
 #define GEN_PASS_DECL_LOOPINVARIANTSUBSETHOISTINGPASS
 #define GEN_PASS_DECL_INLINERPASS
 #define GEN_PASS_DECL_MEM2REG
+#define GEN_PASS_DECL_NORMALIZEPASS
 #define GEN_PASS_DECL_PRINTIRPASS
 #define GEN_PASS_DECL_PRINTOPSTATSPASS
 #define GEN_PASS_DECL_REMOVEDEADVALUESPASS
diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index 74ac370ea950b..10193bbcef2bd 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -656,4 +656,12 @@ def BubbleDownMemorySpaceCasts :
   }];
 }
 
+def NormalizePass : InterfacePass<"normalize", "FunctionOpInterface"> {
+  let summary = "Transforms IR into a normal form that's easier to diff.";
+  let description = [{
+    This pass attempts to relocate the defining ops of operands for any
+    side-effecting or terminator operation to their nearest dominating positions.
+  }];
+}
+
 #endif // MLIR_TRANSFORMS_PASSES
diff --git a/mlir/lib/Transforms/CMakeLists.txt b/mlir/lib/Transforms/CMakeLists.txt
index 66b39f53c91df..41347d23b0f49 100644
--- a/mlir/lib/Transforms/CMakeLists.txt
+++ b/mlir/lib/Transforms/CMakeLists.txt
@@ -12,6 +12,7 @@ add_mlir_library(MLIRTransforms
   LocationSnapshot.cpp
   LoopInvariantCodeMotion.cpp
   Mem2Reg.cpp
+  Normalize.cpp
   OpStats.cpp
   PrintIR.cpp
   RemoveDeadValues.cpp
diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
new file mode 100644
index 0000000000000..1904d25d36674
--- /dev/null
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -0,0 +1,112 @@
+//===- Normalize.cpp - Transforms IR into a normal form ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/IR/Dominance.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/Passes.h"
+#include "llvm/Support/DebugLog.h"
+
+using namespace mlir;
+
+namespace mlir {
+#define GEN_PASS_DEF_NORMALIZEPASS
+#include "mlir/Transforms/Passes.h.inc"
+} // namespace mlir
+
+#define DEBUG_TYPE "normalize"
+
+namespace {
+
+bool isOutput(Operation *op) {
+  if (!op)
+    return false;
+  return !isMemoryEffectFree(op) || op->hasTrait<OpTrait::IsTerminator>();
+}
+
+/// Returns a vector of output ops. An output is a op which
+/// has side-effects or is terminator.
+SmallVector<Operation *> collectOutputs(Operation *root) {
+  SmallVector<Operation *> outputs;
+  root->walk([&](Operation *op) {
+    if (isOutput(op))
+      outputs.push_back(op);
+  });
+  return outputs;
+}
+
+/// The function returns the operation that dominates all other operations in
+/// the given list.
+Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
+  if (ops.empty())
+    return {};
+  Operation *curDomOp = ops.front();
+  DominanceInfo domInfo(curDomOp);
+  for (size_t i = 1, e = ops.size(); i < e; ++i) {
+    bool dominateA = domInfo.dominates(ops[i], curDomOp);
+    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
+    if (dominateA) {
+      LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
+             << "\ndominate\n"
+             << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions());
+      curDomOp = ops[i];
+    }
+    if (!dominateA && !dominateB) {
+      LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
+             << "\nand\n"
+             << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions())
+             << "\ndo not dominate each other";
+      return {};
+    }
+  }
+  return curDomOp;
+}
+
+/// Move used to its nearest user and recursively perform the same process on
+/// the defining operations of its operands.
+void reorderOutput(IRRewriter &rewriter, Operation *used) {
+  if (!isPure(used))
+    return;
+  SmallVector<Operation *> users(used->getUsers());
+  if (Operation *domOp = getDominateOp(users)) {
+    rewriter.moveOpBefore(used, domOp);
+    for (Value operand : used->getOperands())
+      if (Operation *defineOp = operand.getDefiningOp())
+        reorderOutput(rewriter, defineOp);
+  }
+}
+
+/// Reorders ops by walking up the tree from each operand of an output op and
+/// reducing the def-use distance. This method assumes that output ops were
+/// collected top-down, otherwise the def-use chain may be broken. This method
+/// is a wrapper for recursive reorderOutput().
+void reorderOutputs(IRRewriter &rewriter,
+                    SmallVectorImpl<Operation *> &outputs) {
+  SmallPtrSet<Operation *, 16> visited;
+  for (Operation *output : outputs) {
+    for (Value operand : output->getOperands()) {
+      if (Operation *defineOp = operand.getDefiningOp();
+          defineOp && !visited.contains(defineOp)) {
+        reorderOutput(rewriter, defineOp);
+      }
+    }
+  }
+}
+
+struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
+  using impl::NormalizePassBase<NormalizePass>::NormalizePassBase;
+  void runOnOperation() override;
+};
+} // namespace
+
+void NormalizePass::runOnOperation() {
+  IRRewriter rewriter(&getContext());
+  SmallVector<Operation *> outputs = collectOutputs(getOperation());
+  reorderOutputs(rewriter, outputs);
+}
diff --git a/mlir/test/Transforms/normalize.mlir b/mlir/test/Transforms/normalize.mlir
new file mode 100644
index 0000000000000..241e28c3c385d
--- /dev/null
+++ b/mlir/test/Transforms/normalize.mlir
@@ -0,0 +1,84 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize))" -split-input-file | FileCheck %s
+
+// CHECK-LABEL: func @multiple_memref_store
+//  CHECK-SAME:   %[[ARG0:.*]]: index,
+//  CHECK-SAME:   %[[ARG1:.*]]: memref<?xf32>
+func.func @multiple_memref_store(%arg0: index, %arg1 : memref<?xf32>) {
+  %f0 = arith.constant 0.0 : f32
+  %f1 = arith.constant 1.0 : f32
+  %add = arith.addi %arg0, %arg0 : index
+  %sub = arith.subi %arg0, %arg0 : index
+  memref.store %f0, %arg1[%add] : memref<?xf32>
+  memref.store %f1, %arg1[%sub] : memref<?xf32>
+  return
+}
+
+// CHECK-NEXT: %[[C0:.*]] = arith.constant 0.000000e+00 : f32
+// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: memref.store %[[C0]], %[[ARG1]]{{\[}}%[[ADD]]] : memref<?xf32>
+// CHECK-NEXT: %[[C1:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: memref.store %[[C1]], %[[ARG1]]{{\[}}%[[SUB]]] : memref<?xf32>
+
+// -----
+
+// CHECK-LABEL: func @normalize_return
+// CHECK-SAME:      %[[ARG0:.*]]: index,
+// CHECK-SAME:      %[[ARG1:.*]]: memref<?xf32>
+func.func @normalize_return(%arg0: index, %arg1 : memref<?xf32>) -> index {
+  %f0 = arith.constant 0.0 : f32
+  %add = arith.addi %arg0, %arg0 : index
+  %sub = arith.subi %arg0, %arg0 : index
+  memref.store %f0, %arg1[%add] : memref<?xf32>
+  return %sub : index
+}
+
+//      CHECK: memref.store
+// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: return %[[SUB]] : index
+
+// -----
+
+// CHECK-LABEL: func @cross_region
+//  CHECK-SAME:   %[[ARG0:.*]]: f32,
+//  CHECK-SAME:   %[[ARG1:.*]]: memref<10xf32>
+func.func @cross_region(%arg0: f32, %arg1 : memref<10xf32>) {
+  %add = arith.addf %arg0, %arg0 : f32
+  affine.for %i = 0 to 5 {
+    memref.store %add, %arg1[%i] : memref<10xf32>
+  }
+  %exp = math.log2 %add : f32
+  affine.for %i = 6 to 10 {
+    memref.store %exp, %arg1[%i] : memref<10xf32>
+  } 
+  return
+}
+
+//      CHECK: affine.for %[[IV:.*]] = 6 to 10 {
+// CHECK-NEXT:   %[[LOG:.*]] = math.log2
+// CHECK-NEXT:   memref.store %[[LOG]], %[[ARG1]]{{\[}}%[[IV]]] : memref<10xf32>
+// CHECK-NEXT: }
+
+// -----
+
+
+// CHECK-LABEL: func @side_effect_for_op
+//  CHECK-SAME:   %[[ARG0:.*]]: memref<?xf32>
+func.func @side_effect_for_op(%arg1 : memref<?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %upper = memref.dim %arg1, %c0 : memref<?xf32>
+  %f1 = arith.constant 1.0 : f32
+  scf.for %i = %c0 to %upper step %c1 {
+    memref.store %f1, %arg1[%i] : memref<?xf32>
+  }
+  return
+}
+
+// CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+// CHECK-NEXT: %[[DIM:.*]] = memref.dim %[[ARG0]], %[[C0]] : memref<?xf32>
+// CHECK-NEXT: %[[C1:.*]] = arith.constant 1 : index
+// CHECK-NEXT: scf.for %[[IV:.*]] = %[[C0]] to %[[DIM]] step %[[C1]] {
+// CHECK-NEXT:   %[[F1:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-NEXT:   memref.store %[[F1]], %[[ARG0]]{{\[}}%[[IV]]] : memref<?xf32>
+// CHECK-NEXT: }

>From b025af24e07486c0142bec3f76c0e9cafb559d3f Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 16 Mar 2026 06:58:47 +0000
Subject: [PATCH 2/4] fix nit and add more comment.

---
 mlir/include/mlir/Transforms/Passes.td | 10 ++++++++--
 mlir/lib/Transforms/Normalize.cpp      | 20 ++++++++++---------
 mlir/test/Transforms/normalize.mlir    | 27 +++++++++++++-------------
 3 files changed, 33 insertions(+), 24 deletions(-)

diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index 10193bbcef2bd..a03c69c12461e 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -659,8 +659,14 @@ def BubbleDownMemorySpaceCasts :
 def NormalizePass : InterfacePass<"normalize", "FunctionOpInterface"> {
   let summary = "Transforms IR into a normal form that's easier to diff.";
   let description = [{
-    This pass attempts to relocate the defining ops of operands for any
-    side-effecting or terminator operation to their nearest dominating positions.
+    This pass aims to transform MLIR Modules into a normal form by reordering
+    operations while preserving the same semantics. It attempts to relocate
+    the defining ops of operands for any side-effecting or terminator operation
+    to their nearest dominating positions.
+
+    Note: The pass trying to increase syntactic equivalence of code to reduce
+    diff size while retaining semantic equivalence. It cannot replace the
+    canonicalization pass (a pass that aims for increasing semantic equivalence instead).
   }];
 }
 
diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 1904d25d36674..7e63c42b6d163 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -43,21 +43,22 @@ SmallVector<Operation *> collectOutputs(Operation *root) {
 
 /// The function returns the operation that dominates all other operations in
 /// the given list.
-Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
+Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
   if (ops.empty())
     return {};
   Operation *curDomOp = ops.front();
   DominanceInfo domInfo(curDomOp);
   for (size_t i = 1, e = ops.size(); i < e; ++i) {
     bool dominateA = domInfo.dominates(ops[i], curDomOp);
-    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
     if (dominateA) {
       LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
              << "\ndominate\n"
              << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions());
       curDomOp = ops[i];
+      continue;
     }
-    if (!dominateA && !dominateB) {
+    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
+    if (!dominateB) {
       LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
              << "\nand\n"
              << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions())
@@ -70,13 +71,13 @@ Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
 
 /// Move used to its nearest user and recursively perform the same process on
 /// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *used) {
-  if (!isPure(used))
+void reorderOutput(IRRewriter &rewriter, Operation *producer) {
+  if (!isPure(producer))
     return;
-  SmallVector<Operation *> users(used->getUsers());
+  SmallVector<Operation *> users(producer->getUsers());
   if (Operation *domOp = getDominateOp(users)) {
-    rewriter.moveOpBefore(used, domOp);
-    for (Value operand : used->getOperands())
+    rewriter.moveOpBefore(producer, domOp);
+    for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
         reorderOutput(rewriter, defineOp);
   }
@@ -87,12 +88,13 @@ void reorderOutput(IRRewriter &rewriter, Operation *used) {
 /// collected top-down, otherwise the def-use chain may be broken. This method
 /// is a wrapper for recursive reorderOutput().
 void reorderOutputs(IRRewriter &rewriter,
-                    SmallVectorImpl<Operation *> &outputs) {
+                    const SmallVectorImpl<Operation *> &outputs) {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
+        visited.insert(defineOp);
         reorderOutput(rewriter, defineOp);
       }
     }
diff --git a/mlir/test/Transforms/normalize.mlir b/mlir/test/Transforms/normalize.mlir
index 241e28c3c385d..28f0c05263278 100644
--- a/mlir/test/Transforms/normalize.mlir
+++ b/mlir/test/Transforms/normalize.mlir
@@ -22,20 +22,17 @@ func.func @multiple_memref_store(%arg0: index, %arg1 : memref<?xf32>) {
 
 // -----
 
-// CHECK-LABEL: func @normalize_return
-// CHECK-SAME:      %[[ARG0:.*]]: index,
-// CHECK-SAME:      %[[ARG1:.*]]: memref<?xf32>
-func.func @normalize_return(%arg0: index, %arg1 : memref<?xf32>) -> index {
-  %f0 = arith.constant 0.0 : f32
-  %add = arith.addi %arg0, %arg0 : index
-  %sub = arith.subi %arg0, %arg0 : index
-  memref.store %f0, %arg1[%add] : memref<?xf32>
-  return %sub : index
+// CHECK-LABEL: func @return_multiple_operands
+//  CHECK-SAME:   %[[ARG0:.*]]: index
+func.func @return_multiple_operands (%arg0: index) -> (index, index) {
+  %0 = arith.addi %arg0, %arg0 : index
+  %1 = arith.subi %arg0, %arg0 : index
+  return %1, %0 : index, index
 }
 
-//      CHECK: memref.store
 // CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
-// CHECK-NEXT: return %[[SUB]] : index
+// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: return %[[SUB]], %[[ADD]] : index, index
 
 // -----
 
@@ -61,10 +58,14 @@ func.func @cross_region(%arg0: f32, %arg1 : memref<10xf32>) {
 
 // -----
 
+// This test verifies the reordering of scf.for ops.
+// The memref.store within the scf.for causes the loop to have side effects.
+// The lower bound of the scf.for remains in its original position
+// because the upper bound depends on it, but the step has been reordered.
 
-// CHECK-LABEL: func @side_effect_for_op
+// CHECK-LABEL: func @side_effect_loop_op
 //  CHECK-SAME:   %[[ARG0:.*]]: memref<?xf32>
-func.func @side_effect_for_op(%arg1 : memref<?xf32>) {
+func.func @side_effect_loop_op(%arg1 : memref<?xf32>) {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %upper = memref.dim %arg1, %c0 : memref<?xf32>

>From c91b6a5f4b2bada8263b6063854c8cb39fc408c7 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Sun, 19 Apr 2026 05:00:23 +0000
Subject: [PATCH 3/4] use getAnalysis to get DominanceInfo and use
 markAnalysesPreserved.

---
 mlir/lib/Transforms/Normalize.cpp | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 7e63c42b6d163..6f42c61ee7d18 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -43,11 +43,11 @@ SmallVector<Operation *> collectOutputs(Operation *root) {
 
 /// The function returns the operation that dominates all other operations in
 /// the given list.
-Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
+Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
+                         const DominanceInfo &domInfo) {
   if (ops.empty())
     return {};
   Operation *curDomOp = ops.front();
-  DominanceInfo domInfo(curDomOp);
   for (size_t i = 1, e = ops.size(); i < e; ++i) {
     bool dominateA = domInfo.dominates(ops[i], curDomOp);
     if (dominateA) {
@@ -71,15 +71,16 @@ Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
 
 /// Move used to its nearest user and recursively perform the same process on
 /// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *producer) {
+void reorderOutput(IRRewriter &rewriter, Operation *producer,
+                   const DominanceInfo &domInfo) {
   if (!isPure(producer))
     return;
   SmallVector<Operation *> users(producer->getUsers());
-  if (Operation *domOp = getDominateOp(users)) {
+  if (Operation *domOp = getDominateOp(users, domInfo)) {
     rewriter.moveOpBefore(producer, domOp);
     for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
-        reorderOutput(rewriter, defineOp);
+        reorderOutput(rewriter, defineOp, domInfo);
   }
 }
 
@@ -88,14 +89,15 @@ void reorderOutput(IRRewriter &rewriter, Operation *producer) {
 /// collected top-down, otherwise the def-use chain may be broken. This method
 /// is a wrapper for recursive reorderOutput().
 void reorderOutputs(IRRewriter &rewriter,
-                    const SmallVectorImpl<Operation *> &outputs) {
+                    const SmallVectorImpl<Operation *> &outputs,
+                    const DominanceInfo &domInfo) {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
         visited.insert(defineOp);
-        reorderOutput(rewriter, defineOp);
+        reorderOutput(rewriter, defineOp, domInfo);
       }
     }
   }
@@ -108,7 +110,11 @@ struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
 } // namespace
 
 void NormalizePass::runOnOperation() {
+  DominanceInfo &domInfo = getAnalysis<DominanceInfo>();
   IRRewriter rewriter(&getContext());
   SmallVector<Operation *> outputs = collectOutputs(getOperation());
-  reorderOutputs(rewriter, outputs);
+  reorderOutputs(rewriter, outputs, domInfo);
+  // Since we only changed the positions of the operations, DominanceInfo and
+  // PostDominanceInfo are marked as preserved.
+  markAnalysesPreserved<DominanceInfo, PostDominanceInfo>();
 }

>From 6ec1af226d1cebf159ccfd1c17406078f5b772fa Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 08:44:03 +0000
Subject: [PATCH 4/4] update code.

---
 mlir/lib/Transforms/Normalize.cpp | 365 +++++++++++++++++++++++++++---
 1 file changed, 338 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 6f42c61ee7d18..5de76bd681dd9 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -6,12 +6,29 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Block.h"
 #include "mlir/IR/Dominance.h"
+#include "mlir/IR/Location.h"
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/Operation.h"
+#include "mlir/IR/OperationSupport.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/Region.h"
+#include "mlir/IR/Value.h"
+#include "mlir/IR/Visitors.h"
+#include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Pass/Pass.h"
-#include "mlir/Transforms/Passes.h"
+#include "mlir/Support/WalkResult.h"
+#include "mlir/Transforms/CommutativityUtils.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/Support/DebugLog.h"
+#include <cstdint>
 
 using namespace mlir;
 
@@ -24,25 +41,104 @@ namespace mlir {
 
 namespace {
 
-bool isOutput(Operation *op) {
+class Normalize {
+public:
+  Normalize(IRRewriter &rewriter, DominanceInfo &domInfo)
+      : rewriter(rewriter), domInfo(domInfo) {}
+  /// Collect a vector of output ops within in \p root.
+  void collectOutputs(Operation *root);
+
+  /// Reorders ops by walking up the tree from each operand of an output op and
+  /// reducing the def-use distance. This method assumes that output ops were
+  /// collected top-down, otherwise the def-use chain may be broken. This method
+  /// is a wrapper for recursive reorderOutput().
+  void reorderOutputs();
+
+  /// Assigns unique, sequential names (e.g., "a0", "a1") to all block arguments
+  /// within \p root.
+  void nameBlockArguments(Operation *root);
+
+  /// Assigns unique, sequential names to all collected output operations.
+  void nameOperations();
+
+  /// Fold the operation name within \p root.
+  void foldOperationsName(Operation *root);
+
+  /// Greedily applies commutativity patterns using \p root to define the
+  /// transformation scope.
+  LogicalResult sortCommutativeOperands(Operation *root);
+
+private:
+  /// Reorders operations along the def-use chain from left to right, bottom to
+  /// top, starting from \p producer.
+  void reorderOutput(Operation *producer);
+
+  /// Returns true if the \p is a terminator or contains memory/side effects.
+  bool isOutput(Operation *op);
+
+  /// Returns true if the \p op is an initial operation (has no operands or only
+  /// constant-like operands).
+  bool isInitialOpeartion(Operation *op);
+  void nameOpeartion(Operation *op, SmallPtrSet<Operation *, 32> &visited);
+  void nameAsInitialOpeartion(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited);
+
+  void nameAsRegularOpeartion(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited);
+  SetVector<int> getOutputFootprint(Operation *op,
+                                    SmallPtrSet<Operation *, 32> &visited);
+  void foldOperationName(Operation *op);
+  IRRewriter &rewriter;
+  DominanceInfo &domInfo;
+  SmallVector<Operation *> outputs;
+  // Random constant for hashing, so the state isn't zero.
+  const uint64_t magicHashConstant = 0x6acaa36bef8325c5ULL;
+};
+
+// Frozen mixer; basic-block names derived from these hashes appear in
+// the normalized IR text and must be deterministic across processes
+// for the normalizer's "compare normalized IR" workflow to work.
+static constexpr uint64_t hash_16_bytes(uint64_t Low, uint64_t High) {
+  const uint64_t kMul = 0x9ddfea08eb382d69ULL;
+  uint64_t A = (Low ^ High) * kMul;
+  A ^= (A >> 47);
+  uint64_t B = (High ^ A) * kMul;
+  B ^= (B >> 47);
+  B *= kMul;
+  return B;
+}
+
+/// Computes the 64-bit FNV-1a hash value of the given string \p data.
+static constexpr uint64_t strHash(std::string_view data) noexcept {
+  const uint64_t fnvOffset = 0xcbf29ce484222325ULL;
+  const uint64_t fnvPrime = 0x100000001b3ULL;
+  uint64_t hash = fnvOffset;
+  for (const auto &c : data) {
+    hash ^= static_cast<uint64_t>(c);
+    hash *= fnvPrime;
+  }
+  return hash;
+}
+
+bool Normalize::isOutput(Operation *op) {
   if (!op)
     return false;
   return !isMemoryEffectFree(op) || op->hasTrait<OpTrait::IsTerminator>();
 }
 
-/// Returns a vector of output ops. An output is a op which
-/// has side-effects or is terminator.
-SmallVector<Operation *> collectOutputs(Operation *root) {
-  SmallVector<Operation *> outputs;
+void Normalize::collectOutputs(Operation *root) {
   root->walk([&](Operation *op) {
-    if (isOutput(op))
+    if (op == root)
+      return WalkResult::advance();
+    if (isOutput(op)) {
+      LDBG() << "insert " << OpWithFlags(op, OpPrintingFlags().skipRegions())
+             << " to outputs";
       outputs.push_back(op);
+    }
+    return WalkResult::advance();
   });
-  return outputs;
 }
 
-/// The function returns the operation that dominates all other operations in
-/// the given list.
 Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
                          const DominanceInfo &domInfo) {
   if (ops.empty())
@@ -69,10 +165,7 @@ Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
   return curDomOp;
 }
 
-/// Move used to its nearest user and recursively perform the same process on
-/// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *producer,
-                   const DominanceInfo &domInfo) {
+void Normalize::reorderOutput(Operation *producer) {
   if (!isPure(producer))
     return;
   SmallVector<Operation *> users(producer->getUsers());
@@ -80,27 +173,238 @@ void reorderOutput(IRRewriter &rewriter, Operation *producer,
     rewriter.moveOpBefore(producer, domOp);
     for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
-        reorderOutput(rewriter, defineOp, domInfo);
+        reorderOutput(defineOp);
   }
 }
 
-/// Reorders ops by walking up the tree from each operand of an output op and
-/// reducing the def-use distance. This method assumes that output ops were
-/// collected top-down, otherwise the def-use chain may be broken. This method
-/// is a wrapper for recursive reorderOutput().
-void reorderOutputs(IRRewriter &rewriter,
-                    const SmallVectorImpl<Operation *> &outputs,
-                    const DominanceInfo &domInfo) {
+void Normalize::reorderOutputs() {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
         visited.insert(defineOp);
-        reorderOutput(rewriter, defineOp, domInfo);
+        reorderOutput(defineOp);
+      }
+    }
+  }
+}
+
+bool Normalize::isInitialOpeartion(Operation *op) {
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp();
+        !define || !define->hasTrait<OpTrait::ConstantLike>())
+      return false;
+  }
+  return true;
+}
+
+SetVector<int>
+Normalize::getOutputFootprint(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited) {
+  SetVector<int> outputs;
+  if (visited.count(op))
+    return outputs;
+  visited.insert(op);
+
+  if (isOutput(op)) {
+    int count = 0;
+    do {
+      Region *parentRegion = op->getParentRegion();
+      int distance = 0;
+      for (Operation &curOp : parentRegion->getOps()) {
+        if (&curOp == op)
+          break;
+        ++distance;
       }
+      count += distance;
+      op = parentRegion->getParentOp();
+      if (isa<FunctionOpInterface>(op))
+        break;
+    } while (!isa<FunctionOpInterface>(op));
+    outputs.insert(count);
+    return outputs;
+  }
+
+  for (Operation *user : op->getUsers()) {
+    SetVector<int> outputsUser = getOutputFootprint(user, visited);
+    outputs.insert(outputsUser.begin(), outputsUser.end());
+  }
+  return outputs;
+}
+
+void appendCallAndOperandNames(Operation *op, SmallString<512> &name,
+                               SmallVectorImpl<StringRef> &operandNames) {
+  // In case of CallInst, consider callee in the instruction name.
+  if (auto callOp = dyn_cast<CallOpInterface>(op))
+    if (auto funcOp = dyn_cast<FunctionOpInterface>(callOp.resolveCallable()))
+      name.append(funcOp.getNameAttr());
+
+  if (operandNames.size() > 0) {
+    name.append("$-");
+    for (size_t i = 0, e = operandNames.size(); i < e; ++i) {
+      name.append(operandNames[i]);
+      if (i < e - 1)
+        name.append(".");
+    }
+    name.append("-$");
+  }
+  NameLoc loc = NameLoc::get(StringAttr::get(op->getContext(), name));
+  LDBG() << "set NameLoc: " << loc
+         << "\nfor: " << OpWithFlags(op, OpPrintingFlags().skipRegions());
+  op->setLoc(loc);
+}
+
+void Normalize::nameAsInitialOpeartion(Operation *op,
+                                       SmallPtrSet<Operation *, 32> &visited) {
+  // Operation operands for further sorting.
+  SmallVector<StringRef, 4> operandNames;
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp())
+      nameOpeartion(define, visited);
+    if (NameLoc loc = dyn_cast<NameLoc>(operand.getLoc()))
+      operandNames.push_back(loc.getName());
+  }
+
+  if (op->use_empty())
+    return;
+
+  // Initialize to a magic constant, so the state isn't zero.
+  uint64_t hash = magicHashConstant;
+
+  // Consider instruction's opcode in the hash.
+  hash = hash_16_bytes(hash, strHash(op->getName().getStringRef().str()));
+
+  // Get output footprint for \p op.
+  SmallPtrSet<Operation *, 32> visitedOutputFoot;
+  SetVector<int> outputFootprint = getOutputFootprint(op, visitedOutputFoot);
+  LDBG() << "getOutputFootprint success";
+
+  // Consider output footprint in the hash.
+  for (const int &output : outputFootprint)
+    hash = hash_16_bytes(hash, output);
+
+  hash = hash_16_bytes(
+      hash, std::distance(op->getBlock()->begin(), op->getIterator()));
+
+  // Base operation name.
+  SmallString<512> name;
+  name.append("vl" + std::to_string(hash).substr(0, 5));
+
+  // Append call and operand name.
+  appendCallAndOperandNames(op, name, operandNames);
+}
+
+void Normalize::nameAsRegularOpeartion(Operation *op,
+                                       SmallPtrSet<Operation *, 32> &visited) {
+  SmallVector<StringRef, 2> operandNames;
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp())
+      nameOpeartion(define, visited);
+    if (NameLoc loc = dyn_cast<NameLoc>(operand.getLoc())) {
+      operandNames.push_back(loc.getName());
+    }
+  }
+
+  if (op->use_empty())
+    return;
+
+  // Initialize to a magic constant, so the state isn't zero.
+  uint64_t hash = magicHashConstant;
+
+  // Consider operation opcode in the hash.
+  uint64_t ophash = strHash(op->getName().getStringRef().str());
+  hash = hash_16_bytes(hash, ophash);
+
+  // Operand opcodes for further sorting (commutative).
+  SmallVector<int, 4> operandsOpcodes;
+
+  for (Value operand : op->getOperands())
+    if (Operation *define = operand.getDefiningOp())
+      hash =
+          hash_16_bytes(hash, strHash(define->getName().getStringRef().str()));
+
+  hash = hash_16_bytes(
+      hash, std::distance(op->getBlock()->begin(), op->getIterator()));
+
+  // Base operation name.
+  SmallString<512> name;
+  name.append("op" + std::to_string(hash).substr(0, 5));
+
+  // Append call and operand name.
+  appendCallAndOperandNames(op, name, operandNames);
+}
+
+void Normalize::nameOpeartion(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited) {
+  if (visited.count(op))
+    return;
+  visited.insert(op);
+  LDBG() << "rename : " << OpWithFlags(op, OpPrintingFlags().skipRegions());
+
+  // Determine the type of instruction to name.
+  if (isInitialOpeartion(op)) {
+    // This is an initial instruction.
+    nameAsInitialOpeartion(op, visited);
+  } else {
+    // This must be a regular instruction.
+    nameAsRegularOpeartion(op, visited);
+  }
+}
+
+void Normalize::nameOperations() {
+  SmallPtrSet<Operation *, 32> visited;
+  for (Operation *op : outputs)
+    nameOpeartion(op, visited);
+}
+
+void Normalize::nameBlockArguments(Operation *root) {
+  size_t argumentCount = 0;
+  MLIRContext *context = root->getContext();
+  root->walk<WalkOrder::PreOrder>([&](Block *b) {
+    for (auto argument : b->getArguments()) {
+      NameLoc loc =
+          NameLoc::get(StringAttr::get(context, "a" + Twine(argumentCount++)));
+      argument.setLoc(loc);
     }
+    return;
+  });
+}
+
+void Normalize::foldOperationName(Operation *op) {
+  NameLoc loc = dyn_cast<NameLoc>(op->getLoc());
+  if (!loc || loc.getName().empty() || loc.getName().str().substr(0, 2) != "op")
+    return;
+
+  SmallString<512> name;
+  name.append(loc.getName().str().substr(0, 7));
+  name.append("$-");
+  for (size_t i = 0, e = op->getNumOperands(); i < e; ++i) {
+    if (NameLoc loc = dyn_cast<NameLoc>(op->getOperand(i).getLoc()))
+      name.append(loc.getName().str().substr(0, 7));
+    if (i < e - 1)
+      name.append(".");
   }
+  name.append("-$");
+
+  loc = NameLoc::get(StringAttr::get(op->getContext(), name));
+  LDBG() << "set NameLoc: " << loc
+         << "\nfor: " << OpWithFlags(op, OpPrintingFlags().skipRegions());
+  op->setLoc(loc);
+}
+
+void Normalize::foldOperationsName(Operation *root) {
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *op) { foldOperationName(op); });
+}
+
+LogicalResult Normalize::sortCommutativeOperands(Operation *root) {
+  MLIRContext *context = root->getContext();
+  RewritePatternSet patterns(context);
+  populateCommutativityUtilsPatterns(patterns);
+  if (failed(applyPatternsGreedily(root, std::move(patterns))))
+    return failure();
+  return success();
 }
 
 struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
@@ -112,9 +416,16 @@ struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
 void NormalizePass::runOnOperation() {
   DominanceInfo &domInfo = getAnalysis<DominanceInfo>();
   IRRewriter rewriter(&getContext());
-  SmallVector<Operation *> outputs = collectOutputs(getOperation());
-  reorderOutputs(rewriter, outputs, domInfo);
-  // Since we only changed the positions of the operations, DominanceInfo and
-  // PostDominanceInfo are marked as preserved.
+  Normalize normalize(rewriter, domInfo);
+  if (failed(normalize.sortCommutativeOperands(getOperation())))
+    signalPassFailure();
+  normalize.collectOutputs(getOperation());
+  normalize.reorderOutputs();
+  normalize.nameBlockArguments(getOperation());
+  normalize.nameOperations();
+  normalize.foldOperationsName(getOperation());
+
+  // Since we only changed the positions of the operations, `DominanceInfo` and
+  // `PostDominanceInfo` are marked as preserved.
   markAnalysesPreserved<DominanceInfo, PostDominanceInfo>();
 }


From llvmlistbot at llvm.org  Mon Jul  6 02:12:37 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 06 Jul 2026 02:12:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b7185.7270d2fd.23dbb3.34b6@mx.google.com>


================
@@ -1584,6 +1584,55 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix` to `x * (1 - a) + y * a` as specified by
+/// GL.std.450.
+class GLFMixPattern : public SPIRVToLLVMConversion<spirv::GLFMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getA();
+    Value one = createFPConstant(loc, op.getType(), dstType, rewriter, 1.0);
+    Value oneMinusA = LLVM::FSubOp::create(rewriter, loc, dstType, one, a);
+    Value lhs = LLVM::FMulOp::create(rewriter, loc, dstType, x, oneMinusA);
+    Value rhs = LLVM::FMulOp::create(rewriter, loc, dstType, y, a);
+    rewriter.replaceOpWithNewOp<LLVM::FAddOp>(op, dstType, lhs, rhs);
+    return success();
+  }
+};
+
+/// Converts `spirv.CL.mix` to `fma(a, y - x, x)`. The OpenCL spec defines
+/// mix as `x + (y - x) * a` and explicitly permits FMA contractions.
+class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::CLMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::CLMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
----------------
IgWod wrote:

Can you now use `getX`, `getY`, etc., since the patterns are separate?

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Mon Jul  6 02:12:37 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 06 Jul 2026 02:12:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b7185.7270d2fd.23dbb3.34b8@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Mon Jul  6 02:15:42 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:15:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b723e.fdc477d6.1a5bda.fbca@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206935

>From 4fa5c4ca4a5a4333743f1fc3afa38979d20a176a Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 11:54:35 +0200
Subject: [PATCH 1/3] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix
 and CL.mix

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 27 +++++++++++++++++++
 .../SPIRVToLLVM/cl-ops-to-llvm.mlir           | 20 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 20 ++++++++++++++
 3 files changed, 67 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..c3fff3b55587c 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,31 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
+/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
+/// and CL ops name their third operand differently (`a` vs. `z`).
+template <typename SPIRVOp>
+class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
+    Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1942,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      MixPattern<spirv::GLFMixOp>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1950,6 +1976,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      MixPattern<spirv::CLMixOp>,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..f405ef48c6fe1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,23 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}

>From 5512c6003aa7212a15226316a7ca9c6aef850eb0 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Mon, 6 Jul 2026 06:56:23 +0200
Subject: [PATCH 2/3] Address comments

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 45 ++++++++++++++-----
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 14 ++++--
 2 files changed, 44 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 24fdccb55ffaa..2fcc14777a5eb 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1584,18 +1584,42 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
-/// Converts `spirv.GL.FMix`/`spirv.CL.mix` to `fma(a, y - x, x)`, the linear
-/// blend `x * (1 - a) + y * a`. Operands are taken positionally because the GL
-/// and CL ops name their third operand differently (`a` vs. `z`).
-template <typename SPIRVOp>
-class MixPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+/// Converts `spirv.GL.FMix` to `x * (1 - a) + y * a` as specified by
+/// GL.std.450.
+class GLFMixPattern : public SPIRVToLLVMConversion<spirv::GLFMixOp> {
 public:
-  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+  using SPIRVToLLVMConversion<spirv::GLFMixOp>::SPIRVToLLVMConversion;
 
   LogicalResult
-  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+  matchAndRewrite(spirv::GLFMixOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getA();
+    Value one = createFPConstant(loc, op.getType(), dstType, rewriter, 1.0);
+    Value oneMinusA = LLVM::FSubOp::create(rewriter, loc, dstType, one, a);
+    Value lhs = LLVM::FMulOp::create(rewriter, loc, dstType, x, oneMinusA);
+    Value rhs = LLVM::FMulOp::create(rewriter, loc, dstType, y, a);
+    rewriter.replaceOpWithNewOp<LLVM::FAddOp>(op, dstType, lhs, rhs);
+    return success();
+  }
+};
+
+/// Converts `spirv.CL.mix` to `fma(a, y - x, x)`. The OpenCL spec defines
+/// mix as `x + (y - x) * a` and explicitly permits FMA contractions.
+class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::CLMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::CLMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
@@ -1991,7 +2015,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
-      MixPattern<spirv::GLFMixOp>,
+      GLFMixPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -2024,8 +2048,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMaxOp, LLVM::SMaxOp>,
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
-      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
-      MixPattern<spirv::CLMixOp>,
+      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>, CLMixPattern,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index beff049c7ebc8..caecf1f9f8d32 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -413,16 +413,22 @@ spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
 
 // CHECK-LABEL: @fmix_scalar
 spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
-  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
-  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : f32
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : f32
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : f32
   %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
   spirv.Return
 }
 
 // CHECK-LABEL: @fmix_vector
 spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
-  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
-  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(dense<1.000000e+00> : vector<4xf32>) : vector<4xf32>
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : vector<4xf32>
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : vector<4xf32>
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : vector<4xf32>
   %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
   spirv.Return
 }

>From 6b8d06e38a7791aced5f7adb9f043ab91ac3f142 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Mon, 6 Jul 2026 11:15:31 +0200
Subject: [PATCH 3/3] Apply comment

---
 mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 2fcc14777a5eb..d1c8822bf0f05 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1624,9 +1624,9 @@ class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
     Location loc = op.getLoc();
-    Value x = adaptor.getOperands()[0];
-    Value y = adaptor.getOperands()[1];
-    Value a = adaptor.getOperands()[2];
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getZ();
     Value diff = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
     rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, diff, x);
     return success();


From llvmlistbot at llvm.org  Mon Jul  6 02:15:43 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:15:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b723f.da8bafd3.2e1688.fc7e@mx.google.com>


================
@@ -1584,6 +1584,55 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix` to `x * (1 - a) + y * a` as specified by
+/// GL.std.450.
+class GLFMixPattern : public SPIRVToLLVMConversion<spirv::GLFMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getA();
+    Value one = createFPConstant(loc, op.getType(), dstType, rewriter, 1.0);
+    Value oneMinusA = LLVM::FSubOp::create(rewriter, loc, dstType, one, a);
+    Value lhs = LLVM::FMulOp::create(rewriter, loc, dstType, x, oneMinusA);
+    Value rhs = LLVM::FMulOp::create(rewriter, loc, dstType, y, a);
+    rewriter.replaceOpWithNewOp<LLVM::FAddOp>(op, dstType, lhs, rhs);
+    return success();
+  }
+};
+
+/// Converts `spirv.CL.mix` to `fma(a, y - x, x)`. The OpenCL spec defines
+/// mix as `x + (y - x) * a` and explicitly permits FMA contractions.
+class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::CLMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::CLMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getOperands()[0];
+    Value y = adaptor.getOperands()[1];
+    Value a = adaptor.getOperands()[2];
----------------
aobolensk wrote:

Done

https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Mon Jul  6 02:17:23 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 06 Jul 2026 02:17:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 GL.FClamp/SClamp/UClamp (PR #203831)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203831@github.com>
Message-ID: <6a4b72a3.fdc477d6.1a5bda.34a3@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/203831

From llvmlistbot at llvm.org  Mon Jul  6 02:17:24 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 06 Jul 2026 02:17:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 GL.FClamp/SClamp/UClamp (PR #203831)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203831@github.com>
Message-ID: <6a4b72a4.2020f6c2.1fdf61.da28@mx.google.com>


================
@@ -921,6 +921,29 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts the GLSL clamp ops (FClamp, SClamp, UClamp) into a nested
+/// min/max sequence, following the op semantics `min(max(x, minVal), maxVal)`.
+template <typename SPIRVOp, typename LLVMMinOp, typename LLVMMaxOp>
+class ClampPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
----------------
IgWod wrote:

```suggestion
    Type dstType = this->getTypeConverter()->convertType(op.getType());
```

https://github.com/llvm/llvm-project/pull/203831

From llvmlistbot at llvm.org  Mon Jul  6 02:18:14 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:18:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 GL.FClamp/SClamp/UClamp (PR #203831)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203831@github.com>
Message-ID: <6a4b72d6.92d2873c.38c951.0637@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/203831

>From 80a6849c7d7da987f19ca9e401d00b1aebbc6d01 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Mon, 15 Jun 2026 08:07:27 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 GL.FClamp/SClamp/UClamp

Lower the GLSL clamp ops as nested min/max following `min(max(x, y), z)`
---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 26 ++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 36 +++++++++++++++++++
 2 files changed, 62 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..bc1d690061a91 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,29 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts the GLSL clamp ops (FClamp, SClamp, UClamp) into a nested
+/// min/max sequence, following the op semantics `min(max(x, minVal), maxVal)`.
+template <typename SPIRVOp, typename LLVMMinOp, typename LLVMMaxOp>
+class ClampPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value max = LLVMMaxOp::create(rewriter, loc, dstType, adaptor.getX(),
+                                  adaptor.getY());
+    rewriter.template replaceOpWithNewOp<LLVMMinOp>(op, dstType, max,
+                                                    adaptor.getZ());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1897,6 +1920,9 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLFAbsOp, LLVM::FAbsOp>,
       DirectConversionPattern<spirv::GLFloorOp, LLVM::FFloorOp>,
       DirectConversionPattern<spirv::GLFmaOp, LLVM::FMAOp>,
+      ClampPattern<spirv::GLFClampOp, LLVM::MinNumOp, LLVM::MaxNumOp>,
+      ClampPattern<spirv::GLSClampOp, LLVM::SMinOp, LLVM::SMaxOp>,
+      ClampPattern<spirv::GLUClampOp, LLVM::UMinOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::GLFMaxOp, LLVM::MaxNumOp>,
       DirectConversionPattern<spirv::GLFMinOp, LLVM::MinNumOp>,
       DirectConversionPattern<spirv::GLLogOp, LLVM::LogOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..c23b33c1625cd 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -91,6 +91,18 @@ spirv.func @fmin(%arg0: f32, %arg1: vector<3xf16>) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.FClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fclamp
+spirv.func @fclamp(%arg0: f32, %arg1: f32, %arg2: f32) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.maxnum(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
+  // CHECK: llvm.intr.minnum(%[[MAX]], %{{.*}}) : (f32, f32) -> f32
+  %0 = spirv.GL.FClamp %arg0, %arg1, %arg2 : f32
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Log
 //===----------------------------------------------------------------------===//
@@ -208,6 +220,30 @@ spirv.func @umin(%arg0: i16, %arg1: vector<3xi32>) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.SClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @sclamp
+spirv.func @sclamp(%arg0: i16, %arg1: i16, %arg2: i16) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.smax(%{{.*}}, %{{.*}}) : (i16, i16) -> i16
+  // CHECK: llvm.intr.smin(%[[MAX]], %{{.*}}) : (i16, i16) -> i16
+  %0 = spirv.GL.SClamp %arg0, %arg1, %arg2 : i16
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.UClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @uclamp
+spirv.func @uclamp(%arg0: i32, %arg1: i32, %arg2: i32) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.umax(%{{.*}}, %{{.*}}) : (i32, i32) -> i32
+  // CHECK: llvm.intr.umin(%[[MAX]], %{{.*}}) : (i32, i32) -> i32
+  %0 = spirv.GL.UClamp %arg0, %arg1, %arg2 : i32
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Sqrt
 //===----------------------------------------------------------------------===//

>From 030c2466aa092e7183e755f53aeaa2f1f045f07d Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <gooddoog at student.su>
Date: Mon, 6 Jul 2026 11:18:06 +0200
Subject: [PATCH 2/2] Update mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp

Co-authored-by: Igor Wodiany <dev at wodiany.com>
---
 mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index bc1d690061a91..a81922e739232 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -931,7 +931,7 @@ class ClampPattern : public SPIRVToLLVMConversion<SPIRVOp> {
   LogicalResult
   matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto dstType = this->getTypeConverter()->convertType(op.getType());
+    Type dstType = this->getTypeConverter()->convertType(op.getType());
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 


From llvmlistbot at llvm.org  Mon Jul  6 02:32:55 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Mon, 06 Jul 2026 02:32:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4b7647.ff037943.3bf0e2.0bf8@mx.google.com>


Devjiu wrote:

> The "propagate if it does not break the verifier" is a bit weird to me

To be precise: we don't ask a generic/weak verifier, we ask the encoding's own `verifyEncoding` — the predicate its author committed to as the contract for that attribute. Its encoding's own contract, not in our canonicalizer's policy. We're not writing a new check here — we're trusting the one the encoding author already exposed. 

> this seems to acknowledge that it is OK to drop the metadata, so it
> should just always be OK to drop it

I don't think "OK to drop when the encoding says it no longer holds" implies "OK to drop unconditionally." Those are different decision procedures with very different odds of throwing away information.
Current implementation drops it every time the pattern fires, regardless of whether it's still valid; the proposed policy drops it only when the encoding's own predicate says it no longer applies (and never for opaque encodings, which don't get asked at all).

Concretely for our downstream: we stash bounds/stride metadata in the encoding. After tiling we get a dynamic-shaped tensor whose bounds are still valid and still required by a later lowering. The current unconditional-drop canonicalizer removes that metadata anyway. 
So for our downstream, the current canonicalizer produces "incorrect IR".
That's the "incorrect IR" I care about: not caught by `verify()`, but wrong all the same, and much harder to debug because the
failure surfaces far from the cause.


https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Mon Jul  6 02:40:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 02:40:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 334eb77 - [mlir][SPIR-V] Add SPIRVToLLVM
 conversions for GL.FMix and CL.mix (#206935)
Message-ID: <6a4b7820.e55dc341.29c9b8.cf93@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-06T11:40:44+02:00
New Revision: 334eb772a25d700feb1f6c64b872cc693731b0b3

URL: https://github.com/llvm/llvm-project/commit/334eb772a25d700feb1f6c64b872cc693731b0b3
DIFF: https://github.com/llvm/llvm-project/commit/334eb772a25d700feb1f6c64b872cc693731b0b3.diff

LOG: [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FMix and CL.mix (#206935)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
    mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 1cd3d79c13e20..d1c8822bf0f05 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1584,6 +1584,55 @@ class FractPattern : public SPIRVToLLVMConversion<spirv::GLFractOp> {
   }
 };
 
+/// Converts `spirv.GL.FMix` to `x * (1 - a) + y * a` as specified by
+/// GL.std.450.
+class GLFMixPattern : public SPIRVToLLVMConversion<spirv::GLFMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::GLFMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::GLFMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getA();
+    Value one = createFPConstant(loc, op.getType(), dstType, rewriter, 1.0);
+    Value oneMinusA = LLVM::FSubOp::create(rewriter, loc, dstType, one, a);
+    Value lhs = LLVM::FMulOp::create(rewriter, loc, dstType, x, oneMinusA);
+    Value rhs = LLVM::FMulOp::create(rewriter, loc, dstType, y, a);
+    rewriter.replaceOpWithNewOp<LLVM::FAddOp>(op, dstType, lhs, rhs);
+    return success();
+  }
+};
+
+/// Converts `spirv.CL.mix` to `fma(a, y - x, x)`. The OpenCL spec defines
+/// mix as `x + (y - x) * a` and explicitly permits FMA contractions.
+class CLMixPattern : public SPIRVToLLVMConversion<spirv::CLMixOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::CLMixOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::CLMixOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value x = adaptor.getX();
+    Value y = adaptor.getY();
+    Value a = adaptor.getZ();
+    Value 
diff  = LLVM::FSubOp::create(rewriter, loc, dstType, y, x);
+    rewriter.replaceOpWithNewOp<LLVM::FMAOp>(op, dstType, a, 
diff , x);
+    return success();
+  }
+};
+
 // Converts spirv.GL.Radians (scale = pi/180) and spirv.GL.Degrees
 // (scale = 180/pi) by multiplying the operand by a compile-time constant.
 template <typename SPIRVOp>
@@ -1966,6 +2015,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
+      GLFMixPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
@@ -1998,7 +2048,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLSMaxOp, LLVM::SMaxOp>,
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,
-      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>,
+      DirectConversionPattern<spirv::CLUMinOp, LLVM::UMinOp>, CLMixPattern,
 
       // Logical ops
       DirectConversionPattern<spirv::LogicalAndOp, LLVM::AndOp>,

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index f0568e1035cd9..a986be42a5437 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -99,3 +99,23 @@ spirv.func @cl_integer(%arg0: i32, %arg1: i32) "None" {
   %3 = spirv.CL.u_min %arg0, %arg1 : i32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.CL.mix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @mix_scalar
+spirv.func @mix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (f32, f32, f32) -> f32
+  %0 = spirv.CL.mix %x, %y, %a : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @mix_vector
+spirv.func @mix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[DIFF:.*]] = llvm.fsub %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.intr.fma(%{{.*}}, %[[DIFF]], %{{.*}}) : (vector<4xf32>, vector<4xf32>, vector<4xf32>) -> vector<4xf32>
+  %0 = spirv.CL.mix %x, %y, %a : vector<4xf32>
+  spirv.Return
+}

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index 24f16602e4147..caecf1f9f8d32 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -406,3 +406,29 @@ spirv.func @fract(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %1 = spirv.GL.Fract %arg1 : vector<3xf16>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FMix
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmix_scalar
+spirv.func @fmix_scalar(%x: f32, %y: f32, %a: f32) "None" {
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : f32
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : f32
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : f32
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : f32
+  %0 = spirv.GL.FMix %x : f32, %y : f32, %a : f32 -> f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmix_vector
+spirv.func @fmix_vector(%x: vector<4xf32>, %y: vector<4xf32>, %a: vector<4xf32>) "None" {
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(dense<1.000000e+00> : vector<4xf32>) : vector<4xf32>
+  // CHECK: %[[ONE_MINUS_A:.*]] = llvm.fsub %[[ONE]], %{{.*}} : vector<4xf32>
+  // CHECK: %[[LHS:.*]] = llvm.fmul %{{.*}}, %[[ONE_MINUS_A]] : vector<4xf32>
+  // CHECK: %[[RHS:.*]] = llvm.fmul %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: llvm.fadd %[[LHS]], %[[RHS]] : vector<4xf32>
+  %0 = spirv.GL.FMix %x : vector<4xf32>, %y : vector<4xf32>, %a : vector<4xf32> -> vector<4xf32>
+  spirv.Return
+}


        

From llvmlistbot at llvm.org  Mon Jul  6 02:40:52 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:40:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FMix and CL.mix (PR #206935)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206935@github.com>
Message-ID: <6a4b7824.f06f4994.168928.4e59@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/206935

From llvmlistbot at llvm.org  Mon Jul  6 02:44:08 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 02:44:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
Message-ID: <llvm.org/llvm/llvm-project/pull/207689@github.com>


https://github.com/linuxlonelyeagle created https://github.com/llvm/llvm-project/pull/207689

This Patch adds more precise diagnostics for when a pass is missing a target/source materialization function, making it easier to locate the issue in the pass.

>From e39e0169e95af201ba7de7646ed75d9eb72f2036 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 09:36:26 +0000
Subject: [PATCH] add miss diag for dialect convertion.

---
 .../Transforms/Utils/DialectConversion.cpp    | 11 ++++++++++
 .../Arith/emulate-wide-int-unsupported.mlir   |  4 ++--
 .../test-legalize-type-conversion.mlir        | 20 +++++++++----------
 3 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..a27fcc888f1eb 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3383,6 +3383,17 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
       rewriter.replaceOp(op, newMaterialization);
       return success();
     }
+    StringRef direction =
+        info.getMaterializationKind() == MaterializationKind::Target ? "target"
+                                                                     : "source";
+    InFlightDiagnostic diag = op.emitError()
+                              << "miss " << direction
+                              << " materialization function from ("
+                              << inputOperands.getTypes() << ") to ("
+                              << op.getResultTypes() << ")";
+    diag.attachNote(op->getUsers().begin()->getLoc())
+        << "require this materialization is here";
+    return failure();
   }
 
   InFlightDiagnostic diag = op->emitError()
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 44f09b920ed4f..6be6f4f04e5d5 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,9 +45,9 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{failed to legalize unresolved materialization from ('vector<2x2xi32>') to ('vector<2xi64>') that remained live after conversion}}
+  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
-  // expected-note at +1 {{see existing live user here}}
+  // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
   return
 }
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 1453045c3bfbc..5dcfb34ceb39d 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,9 +2,9 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{failed to legalize unresolved materialization from () to ('i16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from () to ('i16')}}
   %arg0: i16) {
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
 }
 
@@ -21,18 +21,18 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -50,9 +50,9 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -194,9 +194,9 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
-  // expected-note at below {{see existing live user here}}
+  // expected-note at below {{require this materialization is here}}
   "foo.return"(%a, %b) : (f16, f16) -> ()
 }


From llvmlistbot at llvm.org  Mon Jul  6 02:44:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 02:44:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4b790a.ac469dc7.ed218.5c62@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-arith

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

This Patch adds more precise diagnostics for when a pass is missing a target/source materialization function, making it easier to locate the issue in the pass.

---
Full diff: https://github.com/llvm/llvm-project/pull/207689.diff


3 Files Affected:

- (modified) mlir/lib/Transforms/Utils/DialectConversion.cpp (+11) 
- (modified) mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir (+2-2) 
- (modified) mlir/test/Transforms/test-legalize-type-conversion.mlir (+10-10) 


``````````diff
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..a27fcc888f1eb 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3383,6 +3383,17 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
       rewriter.replaceOp(op, newMaterialization);
       return success();
     }
+    StringRef direction =
+        info.getMaterializationKind() == MaterializationKind::Target ? "target"
+                                                                     : "source";
+    InFlightDiagnostic diag = op.emitError()
+                              << "miss " << direction
+                              << " materialization function from ("
+                              << inputOperands.getTypes() << ") to ("
+                              << op.getResultTypes() << ")";
+    diag.attachNote(op->getUsers().begin()->getLoc())
+        << "require this materialization is here";
+    return failure();
   }
 
   InFlightDiagnostic diag = op->emitError()
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 44f09b920ed4f..6be6f4f04e5d5 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,9 +45,9 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{failed to legalize unresolved materialization from ('vector<2x2xi32>') to ('vector<2xi64>') that remained live after conversion}}
+  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
-  // expected-note at +1 {{see existing live user here}}
+  // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
   return
 }
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 1453045c3bfbc..5dcfb34ceb39d 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,9 +2,9 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{failed to legalize unresolved materialization from () to ('i16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from () to ('i16')}}
   %arg0: i16) {
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
 }
 
@@ -21,18 +21,18 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -50,9 +50,9 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -194,9 +194,9 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
-  // expected-note at below {{see existing live user here}}
+  // expected-note at below {{require this materialization is here}}
   "foo.return"(%a, %b) : (f16, f16) -> ()
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 02:44:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 02:44:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4b790a.481afadf.134314.c335@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

This Patch adds more precise diagnostics for when a pass is missing a target/source materialization function, making it easier to locate the issue in the pass.

---
Full diff: https://github.com/llvm/llvm-project/pull/207689.diff


3 Files Affected:

- (modified) mlir/lib/Transforms/Utils/DialectConversion.cpp (+11) 
- (modified) mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir (+2-2) 
- (modified) mlir/test/Transforms/test-legalize-type-conversion.mlir (+10-10) 


``````````diff
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..a27fcc888f1eb 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3383,6 +3383,17 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
       rewriter.replaceOp(op, newMaterialization);
       return success();
     }
+    StringRef direction =
+        info.getMaterializationKind() == MaterializationKind::Target ? "target"
+                                                                     : "source";
+    InFlightDiagnostic diag = op.emitError()
+                              << "miss " << direction
+                              << " materialization function from ("
+                              << inputOperands.getTypes() << ") to ("
+                              << op.getResultTypes() << ")";
+    diag.attachNote(op->getUsers().begin()->getLoc())
+        << "require this materialization is here";
+    return failure();
   }
 
   InFlightDiagnostic diag = op->emitError()
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 44f09b920ed4f..6be6f4f04e5d5 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,9 +45,9 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{failed to legalize unresolved materialization from ('vector<2x2xi32>') to ('vector<2xi64>') that remained live after conversion}}
+  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
-  // expected-note at +1 {{see existing live user here}}
+  // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
   return
 }
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 1453045c3bfbc..5dcfb34ceb39d 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,9 +2,9 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{failed to legalize unresolved materialization from () to ('i16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from () to ('i16')}}
   %arg0: i16) {
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
 }
 
@@ -21,18 +21,18 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -50,9 +50,9 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -194,9 +194,9 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
-  // expected-note at below {{see existing live user here}}
+  // expected-note at below {{require this materialization is here}}
   "foo.return"(%a, %b) : (f16, f16) -> ()
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 02:45:09 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Mon, 06 Jul 2026 02:45:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] AMX memref source check (PR
	#206785)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206785@github.com>
Message-ID: <6a4b7925.fdc477d6.1a5bda.3e3c@mx.google.com>


adam-smnk wrote:

@arun-thmn ping

https://github.com/llvm/llvm-project/pull/206785

From llvmlistbot at llvm.org  Mon Jul  6 02:46:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 02:46:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 65bbeff - [mlir][SPIR-V] Add SPIRVToLLVM
 conversion for GL.FClamp/SClamp/UClamp (#203831)
Message-ID: <6a4b7971.491d90f5.1f0d58.9f4b@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-06T11:46:21+02:00
New Revision: 65bbeff0bdcfa53e72517ed59ab510ab6ebd32bf

URL: https://github.com/llvm/llvm-project/commit/65bbeff0bdcfa53e72517ed59ab510ab6ebd32bf
DIFF: https://github.com/llvm/llvm-project/commit/65bbeff0bdcfa53e72517ed59ab510ab6ebd32bf.diff

LOG: [mlir][SPIR-V] Add SPIRVToLLVM conversion for GL.FClamp/SClamp/UClamp (#203831)

Lower the GLSL clamp ops as nested min/max following `min(max(x, y), z)`

---------

Co-authored-by: Igor Wodiany <dev at wodiany.com>

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index d1c8822bf0f05..8a9b84d1a63c1 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,29 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts the GLSL clamp ops (FClamp, SClamp, UClamp) into a nested
+/// min/max sequence, following the op semantics `min(max(x, minVal), maxVal)`.
+template <typename SPIRVOp, typename LLVMMinOp, typename LLVMMaxOp>
+class ClampPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type dstType = this->getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value max = LLVMMaxOp::create(rewriter, loc, dstType, adaptor.getX(),
+                                  adaptor.getY());
+    rewriter.template replaceOpWithNewOp<LLVMMinOp>(op, dstType, max,
+                                                    adaptor.getZ());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1995,6 +2018,9 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLFAbsOp, LLVM::FAbsOp>,
       DirectConversionPattern<spirv::GLFloorOp, LLVM::FFloorOp>,
       DirectConversionPattern<spirv::GLFmaOp, LLVM::FMAOp>,
+      ClampPattern<spirv::GLFClampOp, LLVM::MinNumOp, LLVM::MaxNumOp>,
+      ClampPattern<spirv::GLSClampOp, LLVM::SMinOp, LLVM::SMaxOp>,
+      ClampPattern<spirv::GLUClampOp, LLVM::UMinOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::GLFMaxOp, LLVM::MaxNumOp>,
       DirectConversionPattern<spirv::GLFMinOp, LLVM::MinNumOp>,
       DirectConversionPattern<spirv::GLLogOp, LLVM::LogOp>,

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index caecf1f9f8d32..7be0bf201175f 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -91,6 +91,18 @@ spirv.func @fmin(%arg0: f32, %arg1: vector<3xf16>) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.FClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fclamp
+spirv.func @fclamp(%arg0: f32, %arg1: f32, %arg2: f32) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.maxnum(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
+  // CHECK: llvm.intr.minnum(%[[MAX]], %{{.*}}) : (f32, f32) -> f32
+  %0 = spirv.GL.FClamp %arg0, %arg1, %arg2 : f32
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Log
 //===----------------------------------------------------------------------===//
@@ -208,6 +220,30 @@ spirv.func @umin(%arg0: i16, %arg1: vector<3xi32>) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.SClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @sclamp
+spirv.func @sclamp(%arg0: i16, %arg1: i16, %arg2: i16) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.smax(%{{.*}}, %{{.*}}) : (i16, i16) -> i16
+  // CHECK: llvm.intr.smin(%[[MAX]], %{{.*}}) : (i16, i16) -> i16
+  %0 = spirv.GL.SClamp %arg0, %arg1, %arg2 : i16
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.UClamp
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @uclamp
+spirv.func @uclamp(%arg0: i32, %arg1: i32, %arg2: i32) "None" {
+  // CHECK: %[[MAX:.*]] = llvm.intr.umax(%{{.*}}, %{{.*}}) : (i32, i32) -> i32
+  // CHECK: llvm.intr.umin(%[[MAX]], %{{.*}}) : (i32, i32) -> i32
+  %0 = spirv.GL.UClamp %arg0, %arg1, %arg2 : i32
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Sqrt
 //===----------------------------------------------------------------------===//


        

From llvmlistbot at llvm.org  Mon Jul  6 02:46:28 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:46:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 GL.FClamp/SClamp/UClamp (PR #203831)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203831@github.com>
Message-ID: <6a4b7974.27b3b003.1610ae.0dee@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/203831

From llvmlistbot at llvm.org  Mon Jul  6 02:47:21 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:47:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206934@github.com>
Message-ID: <6a4b79a9.bb839515.23e6d3.94e9@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206934

>From 2e1952b741d34c2b0dfc1887493498283a21a93c Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:06:39 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FSign and
 GL.SSign

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 64 +++++++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 54 ++++++++++++++++
 2 files changed, 118 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Mon Jul  6 02:50:27 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Mon, 06 Jul 2026 02:50:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4b7a63.7270d2fd.23dbb3.5e08@mx.google.com>


Devjiu wrote:

> > Encodings that do not implement VerifiableTensorEncoding (opaque dict/string/bounds attrs used by downstream dialects) are rank-agnostic
> 
> Is this documented somewhere? This is what the documentation of the `VerifiableTensorEncoding` says:
> 
> ```
>         Verifies the encoding is valid for a tensor type with the
>         given shape and element type. Generates a diagnostic using
>         the supplied callback on failure.
> ```
> 
> This documentation does not specify what happens when an attribute does not implement `VerifiableTensorEncoding`.
> 
> This discussion reminds me a bit of "inherent vs. discardable attributes".

No - fair catch, that was my interpretation, not an existing contract. `VerifiableTensorEncoding`'s doc doesn't say what happens when an attribute doesn't implement it, and there's no documented expectation for `encoding` handling in canonicalizers generally — some passes preserve it, some drop it silently, which is exactly why this PR exists.

Once we land on a policy, I'll write it up in `TensorEncoding.td` to secure the current decision. 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Mon Jul  6 02:53:32 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:53:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for FMod and SMod (PR #206933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206933@github.com>
Message-ID: <6a4b7b1c.14d9b9dc.140829.842a@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206933

>From 2244a3cbcb0294fc1e9adb865ef974b6ff802a26 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:03:16 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for FMod and SMod

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 80 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 54 +++++++++++++
 2 files changed, 133 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..5be4cc6bc4ebd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,83 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.FMod` to `x - y * floor(x / y)`. The SPIR-V op requires the
+/// result to take the sign of the divisor, whereas `llvm.frem` keeps the sign
+/// of the dividend, so `frem` cannot be used directly.
+class FModPattern : public SPIRVToLLVMConversion<spirv::FModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::FModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::FModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto dstType = getTypeConverter()->convertType(op.getType());
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Value div = LLVM::FDivOp::create(rewriter, loc, dstType, lhs, rhs);
+    Value floored = LLVM::FFloorOp::create(rewriter, loc, dstType, div);
+    Value scaled = LLVM::FMulOp::create(rewriter, loc, dstType, rhs, floored);
+    rewriter.replaceOpWithNewOp<LLVM::FSubOp>(op, dstType, lhs, scaled);
+    return success();
+  }
+};
+
+/// Converts `spirv.SMod` to a signed remainder corrected to take the sign of
+/// the divisor. `llvm.srem` keeps the sign of the dividend, so the result is
+/// adjusted by adding the divisor when the remainder is non-zero and its sign
+/// differs from the divisor's.
+class SModPattern : public SPIRVToLLVMConversion<spirv::SModOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SModOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SModOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value lhs = adaptor.getOperand1();
+    Value rhs = adaptor.getOperand2();
+    Type i1Type = rewriter.getI1Type();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value rem = LLVM::SRemOp::create(rewriter, loc, dstType, lhs, rhs);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+
+    Value remNonZero = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                            LLVM::ICmpPredicate::ne, rem, zero);
+    Value remNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rem, zero);
+    Value rhsNeg = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                        LLVM::ICmpPredicate::slt, rhs, zero);
+    Value signMismatch =
+        LLVM::XOrOp::create(rewriter, loc, cmpType, remNeg, rhsNeg);
+    Value needsAdjust =
+        LLVM::AndOp::create(rewriter, loc, cmpType, remNonZero, signMismatch);
+
+    Value adjusted = LLVM::AddOp::create(rewriter, loc, dstType, rem, rhs);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, needsAdjust,
+                                                adjusted, rem);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1901,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, FModPattern,
+      SModPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..36d4c5facb057 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,57 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.FMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fmod_scalar
+spirv.func @fmod_scalar(%arg0: f32, %arg1: f32) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : f32
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (f32) -> f32
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : f32
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : f32
+  %0 = spirv.FMod %arg0, %arg1 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fmod_vector
+spirv.func @fmod_vector(%arg0: vector<4xf32>, %arg1: vector<4xf32>) "None" {
+  // CHECK: %[[DIV:.*]] = llvm.fdiv %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[FLOOR:.*]] = llvm.intr.floor(%[[DIV]]) : (vector<4xf32>) -> vector<4xf32>
+  // CHECK: %[[MUL:.*]] = llvm.fmul %{{.*}}, %[[FLOOR]] : vector<4xf32>
+  // CHECK: llvm.fsub %{{.*}}, %[[MUL]] : vector<4xf32>
+  %0 = spirv.FMod %arg0, %arg1 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.SMod
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @smod_scalar
+spirv.func @smod_scalar(%arg0: i32, %arg1: i32) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : i32
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[RNEG:.*]] = llvm.icmp "slt" %[[REM]], %[[ZERO]] : i32
+  // CHECK: %[[DNEG:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[XOR:.*]] = llvm.xor %[[RNEG]], %[[DNEG]] : i1
+  // CHECK: %[[ADJ:.*]] = llvm.and %[[NZ]], %[[XOR]] : i1
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : i32
+  // CHECK: llvm.select %[[ADJ]], %[[ADD]], %[[REM]] : i1, i32
+  %0 = spirv.SMod %arg0, %arg1 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @smod_vector
+spirv.func @smod_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
+  // CHECK: %[[REM:.*]] = llvm.srem %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: %[[NZ:.*]] = llvm.icmp "ne" %[[REM]], %[[ZERO]] : vector<4xi32>
+  // CHECK: %[[ADD:.*]] = llvm.add %[[REM]], %{{.*}} : vector<4xi32>
+  // CHECK: llvm.select %{{.*}}, %[[ADD]], %[[REM]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.SMod %arg0, %arg1 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Mon Jul  6 02:53:34 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Mon, 06 Jul 2026 02:53:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] AMX memref source check (PR
	#206785)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206785@github.com>
Message-ID: <6a4b7b1e.14d9b9dc.140829.8513@mx.google.com>


https://github.com/rengolin approved this pull request.


https://github.com/llvm/llvm-project/pull/206785

From llvmlistbot at llvm.org  Mon Jul  6 02:54:47 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:54:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 VectorTimesScalar (PR #206949)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206949@github.com>
Message-ID: <6a4b7b67.30759430.1908d3.dc05@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206949

>From 86c221950fd16910955f66c6be5c59c7f76b99f0 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:10:12 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
 VectorTimesScalar

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 26 +++++++++++++++++++
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 15 +++++++++++
 2 files changed, 41 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..23007a58713bd 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,31 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.VectorTimesScalar` to a broadcast of the scalar followed by
+/// an `llvm.fmul`.
+class VectorTimesScalarPattern
+    : public SPIRVToLLVMConversion<spirv::VectorTimesScalarOp> {
+public:
+  using SPIRVToLLVMConversion<
+      spirv::VectorTimesScalarOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::VectorTimesScalarOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    unsigned numElements = op.getVector().getType().getNumElements();
+    Value broadcasted = broadcast(op.getLoc(), adaptor.getScalar(), numElements,
+                                  *getTypeConverter(), rewriter);
+    rewriter.replaceOpWithNewOp<LLVM::FMulOp>(op, dstType, adaptor.getVector(),
+                                              broadcasted);
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1825,6 +1850,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
       DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      VectorTimesScalarPattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..0739f377c008f 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,18 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.VectorTimesScalar
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @vector_times_scalar
+spirv.func @vector_times_scalar(%vector: vector<4xf32>, %scalar: f32) "None" {
+  // CHECK: %[[UNDEF:.*]] = llvm.mlir.poison : vector<4xf32>
+  // CHECK: llvm.insertelement %{{.*}}, %[[UNDEF]]
+  // CHECK-COUNT-2: llvm.insertelement
+  // CHECK: %[[BCAST:.*]] = llvm.insertelement
+  // CHECK: llvm.fmul %{{.*}}, %[[BCAST]] : vector<4xf32>
+  %0 = spirv.VectorTimesScalar %vector, %scalar : (vector<4xf32>, f32) -> vector<4xf32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Mon Jul  6 02:56:01 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 06 Jul 2026 02:56:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a4b7bb1.01f0a1da.2016e9.a19d@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206950

>From 187cd538f98c32b897f1d8839a38a3b6915917a1 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:12:30 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversion for SNegate

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 31 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 20 ++++++++++++
 2 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..610c052b64ea7 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1853,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Mon Jul  6 02:56:31 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Mon, 06 Jul 2026 02:56:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] AMX memref source check (PR
	#206785)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206785@github.com>
Message-ID: <6a4b7bcf.ac469dc7.ed218.1d6a@mx.google.com>


https://github.com/arun-thmn approved this pull request.

Thanks @adam-smnk. LGTM.

https://github.com/llvm/llvm-project/pull/206785

From llvmlistbot at llvm.org  Mon Jul  6 02:57:45 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Mon, 06 Jul 2026 02:57:45 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207677@github.com>
Message-ID: <6a4b7c19.ac469dc7.ed218.4bce@mx.google.com>


https://github.com/nikic closed https://github.com/llvm/llvm-project/pull/207677

From llvmlistbot at llvm.org  Mon Jul  6 02:57:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 02:57:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] fcfc916 - Reapply [IR] Explicitly specify
 target feature for module asm (#204548) (#207677)
Message-ID: <6a4b7c18.510f576d.185953.409c@mx.google.com>


Author: Nikita Popov
Date: 2026-07-06T11:57:38+02:00
New Revision: fcfc9167b628a2fae8fe1d662989ff7752d905ad

URL: https://github.com/llvm/llvm-project/commit/fcfc9167b628a2fae8fe1d662989ff7752d905ad
DIFF: https://github.com/llvm/llvm-project/commit/fcfc9167b628a2fae8fe1d662989ff7752d905ad.diff

LOG: Reapply [IR] Explicitly specify target feature for module asm (#204548) (#207677)

Reapply with a use-after-free fix. The MC subtarget used for inline
assembly has to stay alive until finalization, not only during the
emitInlineAsm() call.

-----

Support specifying additional properties on module-level inline
assembly. In particular, the target features and target CPU can now be
specified as follows:

    module asm(target_features: "+foo", target_cpu: "bar")
        "asm line 1"
        "asm line 2"

There may be multiple module inline assembly blocks with different
properties.

This is intended to fix the long standing issue where in LTO scenarios
we don't know what target features to use for parsing the module-level
inline assembly. Now they can be faithfully preserved, even when merging
inline assembly from different modules with different features.

If target_features and target_cpu are empty, we fall back to the old
behavior (which, in the backend, would be to use the default subtarget
of the TargetMachine).

Fixes https://github.com/llvm/llvm-project/issues/61991.
Fixes https://github.com/llvm/llvm-project/issues/67698.

Added: 
    llvm/test/Assembler/module-asm-invalid.ll
    llvm/test/Bitcode/module-asm.ll
    llvm/test/CodeGen/RISCV/module-asm-features.ll
    llvm/test/LTO/RISCV/module-asm.ll
    llvm/test/Linker/Inputs/module-asm.ll
    llvm/test/Linker/module-asm.ll

Modified: 
    clang/lib/CodeGen/CGObjCMac.cpp
    clang/lib/CodeGen/CodeGenModule.cpp
    clang/test/CodeGen/2006-01-23-FileScopeAsm.c
    clang/test/CodeGen/asm.c
    clang/test/CodeGen/asm_incbin.c
    clang/test/CodeGen/cfi-salt.c
    clang/test/CodeGen/kcfi.c
    clang/test/CodeGenCUDA/filter-decl.cu
    clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
    clang/test/CodeGenObjC/category-class.m
    clang/test/CodeGenObjC/objc-runtime-name.m
    clang/test/Frontend/ast-codegen.c
    clang/test/Modules/codegen.test
    llvm/docs/LangRef.rst
    llvm/docs/ReleaseNotes.md
    llvm/include/llvm/Bitcode/LLVMBitCodes.h
    llvm/include/llvm/CodeGen/AsmPrinter.h
    llvm/include/llvm/IR/Module.h
    llvm/include/llvm/Target/TargetMachine.h
    llvm/lib/AsmParser/LLParser.cpp
    llvm/lib/Bitcode/Reader/BitcodeReader.cpp
    llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
    llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
    llvm/lib/IR/AsmWriter.cpp
    llvm/lib/IR/Core.cpp
    llvm/lib/IR/Module.cpp
    llvm/lib/LTO/LTO.cpp
    llvm/lib/Linker/IRMover.cpp
    llvm/lib/Object/ModuleSymbolTable.cpp
    llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
    llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
    llvm/lib/Target/TargetMachine.cpp
    llvm/lib/Transforms/IPO/ExtractGV.cpp
    llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
    llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
    llvm/lib/Transforms/Utils/SplitModule.cpp
    llvm/test/Bitcode/compatibility-3.6.ll
    llvm/test/Bitcode/compatibility-3.7.ll
    llvm/test/Bitcode/compatibility-3.8.ll
    llvm/test/Bitcode/compatibility-3.9.ll
    llvm/test/Bitcode/compatibility-4.0.ll
    llvm/test/Bitcode/compatibility-5.0.ll
    llvm/test/Bitcode/compatibility-6.0.ll
    llvm/test/Bitcode/compatibility.ll
    llvm/test/Bitcode/highLevelStructure.3.2.ll
    llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
    llvm/test/LTO/X86/inline-asm-lto-discard.ll
    llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
    llvm/test/ThinLTO/X86/import-symver.ll
    llvm/test/Transforms/LowerTypeTests/export-symver.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
    llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
    llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
    mlir/lib/Target/LLVMIR/ModuleImport.cpp
    mlir/test/Target/LLVMIR/module-asm.mlir

Removed: 
    


################################################################################
diff  --git a/clang/lib/CodeGen/CGObjCMac.cpp b/clang/lib/CodeGen/CGObjCMac.cpp
index 69c5e88f3c768..e518b875f44bb 100644
--- a/clang/lib/CodeGen/CGObjCMac.cpp
+++ b/clang/lib/CodeGen/CGObjCMac.cpp
@@ -6610,10 +6610,6 @@ void CGObjCMac::FinishModule() {
   if ((!LazySymbols.empty() || !DefinedSymbols.empty()) &&
       CGM.getTriple().isOSBinFormatMachO()) {
     SmallString<256> Asm;
-    Asm += CGM.getModule().getModuleInlineAsm();
-    if (!Asm.empty() && Asm.back() != '\n')
-      Asm += '\n';
-
     llvm::raw_svector_ostream OS(Asm);
     for (const auto *Sym : DefinedSymbols)
       OS << "\t.objc_class_name_" << Sym->getName() << "=0\n"
@@ -6624,7 +6620,7 @@ void CGObjCMac::FinishModule() {
       OS << "\t.objc_category_name_" << Category << "=0\n"
          << "\t.globl .objc_category_name_" << Category << "\n";
 
-    CGM.getModule().setModuleInlineAsm(OS.str());
+    CGM.getModule().appendModuleInlineAsm(OS.str());
   }
 }
 

diff  --git a/clang/lib/CodeGen/CodeGenModule.cpp b/clang/lib/CodeGen/CodeGenModule.cpp
index 813cd67e4dd51..e155fdd752d7f 100644
--- a/clang/lib/CodeGen/CodeGenModule.cpp
+++ b/clang/lib/CodeGen/CodeGenModule.cpp
@@ -8057,7 +8057,13 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
     if (LangOpts.SYCLIsDevice)
       break;
     auto *AD = cast<FileScopeAsmDecl>(D);
-    getModule().appendModuleInlineAsm(AD->getAsmString());
+
+    const TargetOptions &TargetOpts = getTarget().getTargetOpts();
+    llvm::Module::GlobalAsmProperties Props;
+    Props.TargetFeatures = llvm::join(TargetOpts.Features, ",");
+    Props.TargetCPU = TargetOpts.CPU;
+    getModule().appendModuleInlineAsm(
+        llvm::Module::GlobalAsmFragment(AD->getAsmString(), Props));
     break;
   }
 

diff  --git a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
index 877da4fc669e3..8a212a0b61905 100644
--- a/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
+++ b/clang/test/CodeGen/2006-01-23-FileScopeAsm.c
@@ -1,13 +1,18 @@
 // UNSUPPORTED: target={{.*}}-zos{{.*}}
-// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s
+// RUN: %clang_cc1 %s -triple=riscv64 -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64
+// RUN: %clang_cc1 %s -triple=riscv64 -target-feature +d -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,RISCV64-D
+// RUN: %clang_cc1 %s -emit-llvm -o - | FileCheck %s --check-prefixes=CHECK,DEFAULT
 
-// CHECK: module asm "foo1"
+// RISCV64: module asm(target_features: "+64bit,+i")
+// RISCV64-D: module asm(target_features: "+64bit,+d,+f,+i,+zicsr")
+// DEFAULT: module asm
+// CHECK-NEXT: "foo1"
 __asm__ ("foo1");
-// CHECK: module asm "foo2"
+// CHECK-NEXT: "foo2"
 __asm__ ("foo2");
-// CHECK: module asm "foo3"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo3");
-// CHECK: module asm "foo4"
+// CHECK-NEXT: "foo4"
 __asm__ ("foo4");
-// CHECK: module asm "foo5"
+// CHECK-NEXT: "foo5"
 __asm__ ("foo5");

diff  --git a/clang/test/CodeGen/asm.c b/clang/test/CodeGen/asm.c
index d7465b22fbbf6..ea69c7a17d1cb 100644
--- a/clang/test/CodeGen/asm.c
+++ b/clang/test/CodeGen/asm.c
@@ -2,9 +2,10 @@
 
 // PR10415:
 //
-// CHECK:      module asm "foo1"
-// CHECK-NEXT: module asm "foo2"
-// CHECK-NEXT: module asm "foo3"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo1"
+// CHECK-NEXT: "foo2"
+// CHECK-NEXT: "foo3"
 __asm__ ("foo1");
 __asm__ ("foo2");
 __asm__ ("foo3");

diff  --git a/clang/test/CodeGen/asm_incbin.c b/clang/test/CodeGen/asm_incbin.c
index 6234eea1d6738..bbc5d3af55ea2 100644
--- a/clang/test/CodeGen/asm_incbin.c
+++ b/clang/test/CodeGen/asm_incbin.c
@@ -6,4 +6,5 @@ asm(".incbin \"foo.h\"");
 // RUN: cd %t
 // RUN: %clang -c -emit-llvm %t/tu.c -o %t/tu.ll
 // RUN: llvm-dis %t/tu.ll -o - | FileCheck %s
-// CHECK: module asm ".incbin \22foo.h\22"
+// CHECK: module asm
+// CHECK-NEXT: ".incbin \22foo.h\22"

diff  --git a/clang/test/CodeGen/cfi-salt.c b/clang/test/CodeGen/cfi-salt.c
index 2a90dc31d939b..18156dc014280 100644
--- a/clang/test/CodeGen/cfi-salt.c
+++ b/clang/test/CodeGen/cfi-salt.c
@@ -26,13 +26,14 @@ typedef unsigned int (*ufn_t)(void);
 typedef unsigned int (* __cfi_salt ufn_salt_t)(void);
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
-// CHECK: module asm ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,LOW_SODIUM_HASH:]] */"
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4_SALT:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4_SALT]], {{[0-9]+}} /* [[#%d,ASM_SALTY_HASH:]] */"
 
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_f6"
+// CHECK-NOT: ".weak __kcfi_typeid_f6"
 
 int f1(void);
 int f1_salt(void) __cfi_salt;

diff  --git a/clang/test/CodeGen/kcfi.c b/clang/test/CodeGen/kcfi.c
index da32859dd3d88..f08cf7635fb5a 100644
--- a/clang/test/CodeGen/kcfi.c
+++ b/clang/test/CodeGen/kcfi.c
@@ -6,10 +6,11 @@
 #endif
 
 /// Must emit __kcfi_typeid symbols for address-taken function declarations
-// CHECK: module asm ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
-// CHECK: module asm ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
+// CHECK: module asm
+// CHECK-NEXT: ".weak __kcfi_typeid_[[F4:[a-zA-Z0-9_]+]]"
+// CHECK-NEXT: ".set __kcfi_typeid_[[F4]], {{[0-9]+}} /* [[#%d,HASH:]] */"
 /// Must not __kcfi_typeid symbols for non-address-taken declarations
-// CHECK-NOT: module asm ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
+// CHECK-NOT: ".weak __kcfi_typeid_{{f6|_Z2f6v}}"
 
 // C: @ifunc1 = ifunc i32 (i32), ptr @resolver1
 // C: @ifunc2 = ifunc i64 (i64), ptr @resolver2

diff  --git a/clang/test/CodeGenCUDA/filter-decl.cu b/clang/test/CodeGenCUDA/filter-decl.cu
index 02dacd0ad8ef4..f87d62a56631e 100644
--- a/clang/test/CodeGenCUDA/filter-decl.cu
+++ b/clang/test/CodeGenCUDA/filter-decl.cu
@@ -5,8 +5,9 @@
 
 // This has to be at the top of the file as that's where file-scope
 // asm ends up.
-// CHECK-HOST: module asm "file scope asm is host only"
-// CHECK-DEVICE-NOT: module asm "file scope asm is host only"
+// CHECK-HOST: module asm
+// CHECK-HOST-NEXT: "file scope asm is host only"
+// CHECK-DEVICE-NOT: "file scope asm is host only"
 __asm__("file scope asm is host only");
 
 // CHECK-HOST: constantdata = internal global

diff  --git a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
index 175a3b7bc588c..2b4abee5dc05f 100644
--- a/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
+++ b/clang/test/CodeGenCXX/gnu-asm-constexpr.cpp
@@ -19,7 +19,8 @@ struct string_view {
 namespace GH143242 {
     constexpr string_view code2 = R"(nop; nop; nop; nop)";
     asm((code2));
-    // CHECK: module asm "nop; nop; nop; nop"
+    // CHECK: module asm(target_features: "{{.*}}")
+    // CHECK-NEXT: "nop; nop; nop; nop"
 }
 
 int func() {return 0;};

diff  --git a/clang/test/CodeGenObjC/category-class.m b/clang/test/CodeGenObjC/category-class.m
index 92fd36cbe4066..febe94ce5b9f5 100644
--- a/clang/test/CodeGenObjC/category-class.m
+++ b/clang/test/CodeGenObjC/category-class.m
@@ -1,9 +1,10 @@
 // RUN: %clang_cc1 -triple i386-apple-darwin9 -fobjc-runtime=macosx-fragile-10.5  -emit-llvm -o - %s | FileCheck %s
 // PR7431
 
-// CHECK: module asm "\09.lazy_reference .objc_class_name_A"
-// CHECK: module asm "\09.objc_category_name_A_foo=0"
-// CHECK: module asm "\09.globl .objc_category_name_A_foo"
+// CHECK: module asm
+// CHECK-NEXT: "\09.lazy_reference .objc_class_name_A"
+// CHECK-NEXT: "\09.objc_category_name_A_foo=0"
+// CHECK-NEXT: "\09.globl .objc_category_name_A_foo"
 
 @interface A
 @end

diff  --git a/clang/test/CodeGenObjC/objc-runtime-name.m b/clang/test/CodeGenObjC/objc-runtime-name.m
index c838b8c2c6f1a..a7f59caced390 100644
--- a/clang/test/CodeGenObjC/objc-runtime-name.m
+++ b/clang/test/CodeGenObjC/objc-runtime-name.m
@@ -3,9 +3,10 @@
 // Check that the runtime name is emitted and used instead of the class
 // identifier.
 
-// CHECK: module asm {{.*}}objc_class_name_XYZ=0
-// CHECK: module asm {{.*}}globl .objc_class_name_XYZ
-// CHECK: module asm {{.*}}lazy_reference .objc_class_name_XYZ
+// CHECK: module asm
+// CHECK-NEXT: {{.*}}objc_class_name_XYZ=0
+// CHECK-NEXT: {{.*}}globl .objc_class_name_XYZ
+// CHECK-NEXT: {{.*}}lazy_reference .objc_class_name_XYZ
 
 // CHECK: @[[OBJC_CLASS_NAME:.*]] = private unnamed_addr constant [4 x i8] c"XYZ{{.*}}, section "__TEXT,__cstring,cstring_literals",
 // CHECK: = private global {{.*}} @[[OBJC_CLASS_NAME]], section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"

diff  --git a/clang/test/Frontend/ast-codegen.c b/clang/test/Frontend/ast-codegen.c
index 2e2e3d360dd17..c26b0a1908379 100644
--- a/clang/test/Frontend/ast-codegen.c
+++ b/clang/test/Frontend/ast-codegen.c
@@ -2,7 +2,8 @@
 // RUN: %clang -target i386-unknown-unknown -emit-ast -o %t.ast %s
 // RUN: %clang -target i386-unknown-unknown -emit-llvm -S -o - %t.ast | FileCheck %s
 
-// CHECK: module asm "foo"
+// CHECK: module asm(target_features: "{{.*}}")
+// CHECK-NEXT: "foo"
 __asm__("foo");
 
 // CHECK: @g0 = dso_local global i32 0, align 4

diff  --git a/clang/test/Modules/codegen.test b/clang/test/Modules/codegen.test
index 0af630a754805..5cc5b3c1cda2a 100644
--- a/clang/test/Modules/codegen.test
+++ b/clang/test/Modules/codegen.test
@@ -18,7 +18,8 @@ was implemented to workaround/support the initialization of iostreams
 (implemented as a namespace scope static in the header - only to be provided
 when that specific header is included in the program).
 
-BOTH: module asm "narf"
+BOTH: module asm
+BOTH-NEXT: "narf"
 
 FOO: $_Z2f1PKcz = comdat any
 FOO: $_ZN13implicit_dtorD1Ev = comdat any

diff  --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index bac98bcbb2fad..85eec0f4bee49 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -3358,19 +3358,34 @@ Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
-to the GCC "file scope inline asm" blocks. These blocks are internally
-concatenated by LLVM and treated as a single unit, but may be separated
-in the ``.ll`` file if desired. The syntax is very simple:
+to the GCC "file scope inline asm" blocks:
 
 .. code-block:: llvm
 
-    module asm "inline asm code goes here"
-    module asm "more can go here"
+    module asm
+        "inline asm code goes here"
+        "more can go here"
 
-The strings can contain any character by escaping non-printable
+The sequence of string literals is internally concatenated using newlines as
+separators. The strings can contain any character by escaping non-printable
 characters. The escape sequence used is simply "\\xx" where "xx" is the
 two digit hex code for the number.
 
+Additionally, module-level inline assembly can specify an optional list of
+properties:
+
+.. code-block:: llvm
+
+    module asm(target_features: "+foo", target_cpu: "bar")
+        "inline asm code goes here"
+        "more can go here"
+
+Currently, the only supported properties are ``target_features`` and
+``target_cpu``, with the same meaning as the ``"target-features"`` and
+``"target-cpu"`` function attributes.
+
+Consecutive blocks with identical properties will be concatenated into one.
+
 Note that the assembly string *must* be parseable by LLVM's integrated assembler
 (unless it is disabled), even when emitting a ``.s`` file.
 

diff  --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 11d3ebc80fca6..3ca9f5166ad95 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -114,6 +114,10 @@ Makes programs 10x faster by doing Special New Thing.
 * `alwaysinline` no longer bypasses inlining compatibility checks based on
   target features. Inlining will only be performed if it is safe to do so.
 
+* Module-level inline assembly now accepts optional `target_features` and
+  `target_cpu` properties. This resolves errors during LTO on some
+  architectures.
+
 ### Changes to LLVM infrastructure
 
 * Removed ``Constant::isZeroValue``. It was functionally identical to

diff  --git a/llvm/include/llvm/Bitcode/LLVMBitCodes.h b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
index af4cac996a78c..1761e7a542a73 100644
--- a/llvm/include/llvm/Bitcode/LLVMBitCodes.h
+++ b/llvm/include/llvm/Bitcode/LLVMBitCodes.h
@@ -120,6 +120,8 @@ enum ModuleCodes {
 
   // IFUNC: [ifunc value type, addrspace, resolver val#, linkage, visibility]
   MODULE_CODE_IFUNC = 18,
+
+  MODULE_CODE_ASM_PROPERTY = 19, // [strchr x N]
 };
 
 /// PARAMATTR blocks have code for defining a parameter attribute set.

diff  --git a/llvm/include/llvm/CodeGen/AsmPrinter.h b/llvm/include/llvm/CodeGen/AsmPrinter.h
index 78e64440657cc..335bf9de9a071 100644
--- a/llvm/include/llvm/CodeGen/AsmPrinter.h
+++ b/llvm/include/llvm/CodeGen/AsmPrinter.h
@@ -930,6 +930,19 @@ class LLVM_ABI AsmPrinter : public MachineFunctionPass {
                                 const MCSubtargetInfo *EndInfo,
                                 const MachineInstr *MI);
 
+  /// Emit necessary directives to allow use of instructions that are permitted
+  /// by target features enabled by STI, but are not permitted by target
+  /// features enabled by the global subtarget (TM.getSubTargetInfo()).
+  /// Returns whether anything was emitted.
+  virtual bool emitTargetFeaturePush(const MCSubtargetInfo &STI) {
+    return false;
+  }
+
+  /// Emit necessary directives to restore target feature state.
+  /// The \p DidPush argument is the result of the prior emitTargetFeaturePush()
+  /// call.
+  virtual void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) {}
+
   /// This emits visibility information about symbol, if this is supported by
   /// the target.
   void emitVisibility(MCSymbol *Sym, unsigned Visibility,

diff  --git a/llvm/include/llvm/IR/Module.h b/llvm/include/llvm/IR/Module.h
index 4f7e33969f16f..63d1013896b47 100644
--- a/llvm/include/llvm/IR/Module.h
+++ b/llvm/include/llvm/IR/Module.h
@@ -169,6 +169,45 @@ class LLVM_ABI Module {
         : Behavior(B), Key(K), Val(V) {}
   };
 
+  struct GlobalAsmProperties {
+    std::string TargetFeatures;
+    std::string TargetCPU;
+
+    /// Set a property using a string name.
+    /// Returns whether the property name was valid.
+    LLVM_ABI bool set(StringRef Name, std::string Value);
+
+    /// Get a list of set properties as pairs of key and value.
+    LLVM_ABI SmallVector<std::pair<StringRef, StringRef>> getAsStrings() const;
+
+    bool operator==(const GlobalAsmProperties &Other) const {
+      return TargetFeatures == Other.TargetFeatures &&
+             TargetCPU == Other.TargetCPU;
+    }
+
+    bool operator!=(const GlobalAsmProperties &Other) const {
+      return !(*this == Other);
+    }
+  };
+
+  struct GlobalAsmFragment {
+    std::string Asm;
+    GlobalAsmProperties Props;
+
+    GlobalAsmFragment(StringRef Asm) : GlobalAsmFragment(Asm.str()) {}
+    GlobalAsmFragment(std::string AsmArg, GlobalAsmProperties Props = {})
+        : Asm(std::move(AsmArg)), Props(std::move(Props)) {
+      if (!Asm.empty() && Asm.back() != '\n')
+        Asm += '\n';
+    }
+
+    bool empty() const { return Asm.empty(); }
+
+    bool hasSameProperties(const GlobalAsmFragment &Other) const {
+      return Props == Other.Props;
+    }
+  };
+
 /// @}
 /// @name Member Variables
 /// @{
@@ -180,7 +219,8 @@ class LLVM_ABI Module {
   AliasListType AliasList;        ///< The Aliases in the module
   IFuncListType IFuncList;        ///< The IFuncs in the module
   NamedMDListType NamedMDList;    ///< The named metadata in the module
-  std::string GlobalScopeAsm;     ///< Inline Asm at global scope.
+  /// Inline Asm at the global scope.
+  SmallVector<GlobalAsmFragment, 0> GlobalScopeAsm;
   std::unique_ptr<ValueSymbolTable> ValSymTab; ///< Symbol table for values
   ComdatSymTabType ComdatSymTab;  ///< Symbol table for COMDATs
   std::unique_ptr<MemoryBuffer>
@@ -287,8 +327,17 @@ class LLVM_ABI Module {
   LLVMContext &getContext() const { return Context; }
 
   /// Get any module-scope inline assembly blocks.
-  /// @returns a string containing the module-scope inline assembly blocks.
-  const std::string &getModuleInlineAsm() const { return GlobalScopeAsm; }
+  ArrayRef<GlobalAsmFragment> getModuleInlineAsm() const {
+    return GlobalScopeAsm;
+  }
+
+  /// Get any module-scope inline assembly blocks.
+  MutableArrayRef<GlobalAsmFragment> getModuleInlineAsm() {
+    return GlobalScopeAsm;
+  }
+
+  /// Return whether there is any module-scope inline assembly.
+  bool hasModuleInlineAsm() const { return !GlobalScopeAsm.empty(); }
 
   /// Get a RandomNumberGenerator salted for use with this module. The
   /// RNG can be seeded via -rng-seed=<uint64> and is salted with the
@@ -325,20 +374,45 @@ class LLVM_ABI Module {
   /// Set the target triple.
   void setTargetTriple(Triple T) { TargetTriple = std::move(T); }
 
+  void removeModuleInlineAsm() { GlobalScopeAsm.clear(); }
+
   /// Set the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void setModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm = std::string(Asm);
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void setModuleInlineAsm(GlobalAsmFragment Fragment) {
+    GlobalScopeAsm.clear();
+    appendModuleInlineAsm(std::move(Fragment));
+  }
+
+  void setModuleInlineAsm(ArrayRef<GlobalAsmFragment> Fragments) {
+    GlobalScopeAsm.clear();
+    append_range(GlobalScopeAsm, Fragments);
   }
 
   /// Append to the module-scope inline assembly blocks.
   /// A trailing newline is added if the input doesn't have one.
-  void appendModuleInlineAsm(StringRef Asm) {
-    GlobalScopeAsm += Asm;
-    if (!GlobalScopeAsm.empty() && GlobalScopeAsm.back() != '\n')
-      GlobalScopeAsm += '\n';
+  void appendModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.back().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.back().Asm += Fragment.Asm;
+    } else {
+      GlobalScopeAsm.emplace_back(std::move(Fragment));
+    }
+  }
+
+  /// Prepend to the module-scope inline assembly blocks.
+  void prependModuleInlineAsm(GlobalAsmFragment Fragment) {
+    if (Fragment.empty())
+      return;
+
+    if (!GlobalScopeAsm.empty() &&
+        GlobalScopeAsm.front().hasSameProperties(Fragment)) {
+      GlobalScopeAsm.front().Asm.insert(0, Fragment.Asm);
+    } else {
+      GlobalScopeAsm.insert(GlobalScopeAsm.begin(), std::move(Fragment));
+    }
   }
 
 /// @}

diff  --git a/llvm/include/llvm/Target/TargetMachine.h b/llvm/include/llvm/Target/TargetMachine.h
index 8f5721a031aed..6fdd603092fc6 100644
--- a/llvm/include/llvm/Target/TargetMachine.h
+++ b/llvm/include/llvm/Target/TargetMachine.h
@@ -13,6 +13,7 @@
 #ifndef LLVM_TARGET_TARGETMACHINE_H
 #define LLVM_TARGET_TARGETMACHINE_H
 
+#include "llvm/ADT/StringMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/PassManager.h"
@@ -114,6 +115,9 @@ class LLVM_ABI TargetMachine {
   std::unique_ptr<const MCInstrInfo> MII;
   std::unique_ptr<const MCSubtargetInfo> STI;
 
+  /// MC subtarget keyed by target features and target CPU.
+  StringMap<std::unique_ptr<const MCSubtargetInfo>> MCSubtargetMap;
+
   unsigned RequireStructuredCFG : 1;
   unsigned O0WantsFastISel : 1;
 
@@ -238,6 +242,12 @@ class LLVM_ABI TargetMachine {
   const MCInstrInfo *getMCInstrInfo() const { return MII.get(); }
   const MCSubtargetInfo &getMCSubtargetInfo() const { return *STI; }
 
+  /// Get the MCSubtargetInfo for the given target CPU and target features.
+  /// For use in contexts where a feature-specific MC subtarget is needed,
+  /// but no MachineFunctionis available, such as for module-level inline
+  /// assembly.
+  const MCSubtargetInfo &getMCSubtargetInfo(StringRef CPU, StringRef FS);
+
   /// Return the ExceptionHandling to use, considering TargetOptions and the
   /// Triple's default.
   ExceptionHandling getExceptionModel() const {

diff  --git a/llvm/lib/AsmParser/LLParser.cpp b/llvm/lib/AsmParser/LLParser.cpp
index 75e6add0ec76f..0a5b4c6c91879 100644
--- a/llvm/lib/AsmParser/LLParser.cpp
+++ b/llvm/lib/AsmParser/LLParser.cpp
@@ -641,16 +641,49 @@ bool LLParser::parseTopLevelEntities() {
 
 /// toplevelentity
 ///   ::= 'module' 'asm' STRINGCONSTANT
+///   ::= 'module' 'asm' '(' 'property_name1:' STRINGCONSTANT ','
+///                          'property_name2:' STRINGCONSTANT ')'
+///                      STRINGCONSTANT
 bool LLParser::parseModuleAsm() {
   assert(Lex.getKind() == lltok::kw_module);
   Lex.Lex();
 
   std::string AsmStr;
-  if (parseToken(lltok::kw_asm, "expected 'module asm'") ||
-      parseStringConstant(AsmStr))
+  if (parseToken(lltok::kw_asm, "expected 'module asm'"))
     return true;
 
-  M->appendModuleInlineAsm(AsmStr);
+  Module::GlobalAsmProperties Props;
+  if (EatIfPresent(lltok::lparen)) {
+    while (true) {
+      std::string Key, Value;
+      SMLoc Loc = Lex.getLoc();
+      if (Lex.getKind() != lltok::LabelStr)
+        return error(Loc, "expected property name followed by ':'");
+
+      Key = Lex.getStrVal();
+      Lex.Lex();
+
+      if (parseStringConstant(Value))
+        return true;
+
+      if (!Props.set(Key, Value))
+        return error(Loc, "unknown property name");
+
+      if (EatIfPresent(lltok::rparen))
+        break;
+      if (parseToken(lltok::comma, "expected ',' or ')'"))
+        return true;
+    }
+  }
+
+  do {
+    std::string AsmStrPart;
+    if (parseStringConstant(AsmStrPart))
+      return true;
+    AsmStr += AsmStrPart + "\n";
+  } while (Lex.getKind() == lltok::StringConstant);
+
+  M->appendModuleInlineAsm({AsmStr, Props});
   return false;
 }
 

diff  --git a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
index 7bd569db09a76..264f68ccaf450 100644
--- a/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
+++ b/llvm/lib/Bitcode/Reader/BitcodeReader.cpp
@@ -4600,6 +4600,9 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
   // Initialize to the current module's layout string in case none is specified.
   std::string TentativeDataLayoutStr = TheModule->getDataLayoutStr();
 
+  // Apply to the following module asm.
+  Module::GlobalAsmProperties Props;
+
   auto ResolveDataLayout = [&]() -> Error {
     if (ResolvedDataLayout)
       return Error::success();
@@ -4808,11 +4811,23 @@ Error BitcodeReader::parseModule(uint64_t ResumeBit,
         return error("Invalid data layout record");
       break;
     }
+    case bitc::MODULE_CODE_ASM_PROPERTY: {
+      std::string Str;
+      if (convertToString(Record, 0, Str))
+        return error("Invalid module asm record");
+      size_t SepPos = Str.find('\0');
+      if (SepPos == std::string::npos)
+        return error("Invalid module asm record");
+      if (!Props.set(StringRef(Str.data(), SepPos), Str.substr(SepPos + 1)))
+        return error("Unknown module asm property");
+      break;
+    }
     case bitc::MODULE_CODE_ASM: {  // ASM: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid asm record");
-      TheModule->setModuleInlineAsm(S);
+      TheModule->appendModuleInlineAsm(Module::GlobalAsmFragment(S, Props));
+      Props = {};
       break;
     }
     case bitc::MODULE_CODE_DEPLIB: {  // DEPLIB: [strchr x N]

diff  --git a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
index 12b508c51cbfc..c1a221bd45eaf 100644
--- a/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ b/llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -1551,9 +1551,19 @@ void ModuleBitcodeWriter::writeModuleInfo() {
   const std::string &DL = M.getDataLayoutStr();
   if (!DL.empty())
     writeStringRecord(Stream, bitc::MODULE_CODE_DATALAYOUT, DL, 0 /*TODO*/);
-  if (!M.getModuleInlineAsm().empty())
-    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, M.getModuleInlineAsm(),
-                      0 /*TODO*/);
+
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    SmallVector<std::pair<StringRef, StringRef>> Props =
+        Frag.Props.getAsStrings();
+    for (auto [Key, Value] : Props) {
+      SmallVector<unsigned, 64> Record;
+      Record.append(Key.begin(), Key.end());
+      Record.push_back(0);
+      Record.append(Value.begin(), Value.end());
+      Stream.EmitRecord(bitc::MODULE_CODE_ASM_PROPERTY, Record);
+    }
+    writeStringRecord(Stream, bitc::MODULE_CODE_ASM, Frag.Asm, 0 /*TODO*/);
+  }
 
   // Emit information about sections and GC, computing how many there are. Also
   // compute the maximum alignment value.

diff  --git a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
index abb20a7919fa1..f3ac25f25fcc1 100644
--- a/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/AsmPrinter.cpp
@@ -110,6 +110,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/MC/MCValue.h"
 #include "llvm/MC/SectionKind.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Object/ELFTypes.h"
 #include "llvm/Pass.h"
 #include "llvm/Remarks/RemarkStreamer.h"
@@ -127,7 +128,6 @@
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"
-#include "llvm/TargetParser/Triple.h"
 #include <algorithm>
 #include <cassert>
 #include <cinttypes>
@@ -610,13 +610,18 @@ bool AsmPrinter::doInitialization(Module &M) {
   BeginGCAssembly(M);
 
   // Emit module-level inline asm if it exists.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     OutStreamer->AddComment("Start of file scope inline assembly");
     OutStreamer->addBlankLine();
-    emitInlineAsm(
-        M.getModuleInlineAsm() + "\n", TM.getMCSubtargetInfo(),
-        TM.Options.MCOptions, nullptr,
-        InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+    for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+      const MCSubtargetInfo &AsmSTI = TM.getMCSubtargetInfo(
+          Frag.Props.TargetCPU, Frag.Props.TargetFeatures);
+      bool DidPush = emitTargetFeaturePush(AsmSTI);
+      emitInlineAsm(
+          Frag.Asm, AsmSTI, TM.Options.MCOptions, nullptr,
+          InlineAsm::AsmDialect(TM.getMCAsmInfo().getAssemblerDialect()));
+      emitTargetFeaturePop(AsmSTI, DidPush);
+    }
     OutStreamer->AddComment("End of file scope inline assembly");
     OutStreamer->addBlankLine();
   }

diff  --git a/llvm/lib/IR/AsmWriter.cpp b/llvm/lib/IR/AsmWriter.cpp
index 665d7122af2c7..02d6bcd90add0 100644
--- a/llvm/lib/IR/AsmWriter.cpp
+++ b/llvm/lib/IR/AsmWriter.cpp
@@ -3126,21 +3126,38 @@ void AssemblyWriter::printModule(const Module *M) {
   if (!M->getTargetTriple().empty())
     Out << "target triple = \"" << M->getTargetTriple().str() << "\"\n";
 
-  if (!M->getModuleInlineAsm().empty()) {
+  if (M->hasModuleInlineAsm()) {
     Out << '\n';
 
-    // Split the string into lines, to make it easier to read the .ll file.
-    StringRef Asm = M->getModuleInlineAsm();
-    do {
-      StringRef Front;
-      std::tie(Front, Asm) = Asm.split('\n');
-
-      // We found a newline, print the portion of the asm string from the
-      // last newline up to this newline.
-      Out << "module asm \"";
-      printEscapedString(Front, Out);
-      Out << "\"\n";
-    } while (!Asm.empty());
+    for (const Module::GlobalAsmFragment &Frag : M->getModuleInlineAsm()) {
+      Out << "module asm";
+      SmallVector<std::pair<StringRef, StringRef>> Props =
+          Frag.Props.getAsStrings();
+      if (!Props.empty()) {
+        ListSeparator LS;
+        Out << "(";
+        for (auto [Key, Value] : Props) {
+          Out << LS;
+          Out << Key << ": \"";
+          printEscapedString(Value, Out);
+          Out << "\"";
+        }
+        Out << ")";
+      }
+      Out << "\n";
+      // Split the string into lines, to make it easier to read the .ll file.
+      StringRef Asm = Frag.Asm;
+      do {
+        StringRef Front;
+        std::tie(Front, Asm) = Asm.split('\n');
+
+        // We found a newline, print the portion of the asm string from the
+        // last newline up to this newline.
+        Out << "    \"";
+        printEscapedString(Front, Out);
+        Out << "\"\n";
+      } while (!Asm.empty());
+    }
   }
 
   printTypeIdentities();

diff  --git a/llvm/lib/IR/Core.cpp b/llvm/lib/IR/Core.cpp
index 6c6c3b5c84ede..a7abd3eed31c3 100644
--- a/llvm/lib/IR/Core.cpp
+++ b/llvm/lib/IR/Core.cpp
@@ -510,7 +510,18 @@ void LLVMAppendModuleInlineAsm(LLVMModuleRef M, const char *Asm, size_t Len) {
 }
 
 const char *LLVMGetModuleInlineAsm(LLVMModuleRef M, size_t *Len) {
-  auto &Str = unwrap(M)->getModuleInlineAsm();
+  Module *Mod = unwrap(M);
+  ArrayRef<Module::GlobalAsmFragment> Frags = Mod->getModuleInlineAsm();
+  if (Frags.empty()) {
+    *Len = 0;
+    return nullptr;
+  }
+
+  if (Frags.size() != 1)
+    reportFatalUsageError("LLVMGetModuleInlineAsm is not supported if there is "
+                          "more than one module inline assembly fragment");
+
+  auto &Str = Frags.begin()->Asm;
   *Len = Str.length();
   return Str.c_str();
 }

diff  --git a/llvm/lib/IR/Module.cpp b/llvm/lib/IR/Module.cpp
index 40ef7e354da72..34e44e26d5db5 100644
--- a/llvm/lib/IR/Module.cpp
+++ b/llvm/lib/IR/Module.cpp
@@ -977,3 +977,23 @@ ControlFlowGuardMode Module::getControlFlowGuardMode() const {
     return static_cast<ControlFlowGuardMode>(CI->getZExtValue());
   return ControlFlowGuardMode::Disabled;
 }
+
+bool Module::GlobalAsmProperties::set(StringRef Name, std::string Value) {
+  if (Name == "target_features")
+    TargetFeatures = std::move(Value);
+  else if (Name == "target_cpu")
+    TargetCPU = std::move(Value);
+  else
+    return false;
+  return true;
+}
+
+SmallVector<std::pair<StringRef, StringRef>>
+Module::GlobalAsmProperties::getAsStrings() const {
+  SmallVector<std::pair<StringRef, StringRef>> Props;
+  if (!TargetFeatures.empty())
+    Props.emplace_back("target_features", TargetFeatures);
+  if (!TargetCPU.empty())
+    Props.emplace_back("target_cpu", TargetCPU);
+  return Props;
+}

diff  --git a/llvm/lib/LTO/LTO.cpp b/llvm/lib/LTO/LTO.cpp
index 6a754f95092bb..b672f4e5fac14 100644
--- a/llvm/lib/LTO/LTO.cpp
+++ b/llvm/lib/LTO/LTO.cpp
@@ -1112,7 +1112,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
 
   // Prepend ".lto_discard <sym>, <sym>*" directive to each module inline asm
   // block.
-  if (!M.getModuleInlineAsm().empty()) {
+  if (M.hasModuleInlineAsm()) {
     std::string NewIA = ".lto_discard";
     if (!NonPrevailingAsmSymbols.empty()) {
       // Don't dicard a symbol if there is a live .symver for it.
@@ -1124,7 +1124,7 @@ LTO::addRegularLTO(InputFile &Input, ArrayRef<SymbolResolution> InputRes,
       NewIA += " " + llvm::join(NonPrevailingAsmSymbols, ", ");
     }
     NewIA += "\n";
-    M.setModuleInlineAsm(NewIA + M.getModuleInlineAsm());
+    M.prependModuleInlineAsm(NewIA);
   }
 
   assert(MsymI == MsymE);

diff  --git a/llvm/lib/Linker/IRMover.cpp b/llvm/lib/Linker/IRMover.cpp
index 239201bde1cec..3b72b412d0b2e 100644
--- a/llvm/lib/Linker/IRMover.cpp
+++ b/llvm/lib/Linker/IRMover.cpp
@@ -1562,8 +1562,11 @@ Error IRLinker::run() {
 
   if (!IsPerformingImport && !SrcM->getModuleInlineAsm().empty()) {
     // Append the module inline asm string.
-    DstM.appendModuleInlineAsm(adjustInlineAsm(SrcM->getModuleInlineAsm(),
-                                               SrcTriple));
+    for (const Module::GlobalAsmFragment &Frag : SrcM->getModuleInlineAsm()) {
+      Module::GlobalAsmFragment NewFrag(Frag);
+      NewFrag.Asm = adjustInlineAsm(NewFrag.Asm, SrcTriple);
+      DstM.appendModuleInlineAsm(std::move(NewFrag));
+    }
   } else if (IsPerformingImport) {
     // Import any symver directives for symbols in DstM.
     ModuleSymbolTable::CollectAsmSymvers(*SrcM,
@@ -1573,7 +1576,7 @@ Error IRLinker::run() {
         S += Name;
         S += ", ";
         S += Alias;
-        DstM.appendModuleInlineAsm(S);
+        DstM.appendModuleInlineAsm(std::string(S));
       }
     });
   }

diff  --git a/llvm/lib/Object/ModuleSymbolTable.cpp b/llvm/lib/Object/ModuleSymbolTable.cpp
index 1da5fa9c10a0b..44f717ec12323 100644
--- a/llvm/lib/Object/ModuleSymbolTable.cpp
+++ b/llvm/lib/Object/ModuleSymbolTable.cpp
@@ -72,8 +72,7 @@ initializeRecordStreamer(const Module &M,
   // caused errors in the first run, suppress the second run.
   if (M.getContext().getDiagHandlerPtr()->HasErrors)
     return;
-  StringRef InlineAsm = M.getModuleInlineAsm();
-  if (InlineAsm.empty())
+  if (!M.hasModuleInlineAsm())
     return;
 
   std::string Err;
@@ -90,52 +89,55 @@ initializeRecordStreamer(const Module &M,
   if (!MAI)
     return;
 
-  std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(TT, "", ""));
-  if (!STI)
-    return;
-
   std::unique_ptr<MCInstrInfo> MCII(T->createMCInstrInfo());
   if (!MCII)
     return;
 
-  std::unique_ptr<MemoryBuffer> Buffer(
-      MemoryBuffer::getMemBuffer(InlineAsm, "<inline asm>"));
-  SourceMgr SrcMgr;
-  SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
-
-  MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
-  std::unique_ptr<MCObjectFileInfo> MOFI(
-      T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
-  MCCtx.setObjectFileInfo(MOFI.get());
-  RecordStreamer Streamer(MCCtx, M);
-  T->createNullTargetStreamer(Streamer);
-
-  std::unique_ptr<MCAsmParser> Parser(
-      createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
-
-  std::unique_ptr<MCTargetAsmParser> TAP(
-      T->createMCAsmParser(*STI, *Parser, *MCII));
-  if (!TAP)
-    return;
-
-  MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
-                                 const SourceMgr &SrcMgr,
-                                 std::vector<const MDNode *> &LocInfos) {
-    M.getContext().diagnose(
-        DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
-  });
-
-  // Module-level inline asm is assumed to use At&t syntax (see
-  // AsmPrinter::doInitialization()).
-  Parser->setAssemblerDialect(InlineAsm::AD_ATT);
-
-  Parser->setSymbolScanningMode(true);
-
-  Parser->setTargetParser(*TAP);
-  if (Parser->Run(false))
-    return;
-
-  Init(Streamer);
+  for (const Module::GlobalAsmFragment &Frag : M.getModuleInlineAsm()) {
+    std::unique_ptr<MCSubtargetInfo> STI(T->createMCSubtargetInfo(
+        TT, Frag.Props.TargetCPU, Frag.Props.TargetFeatures));
+    if (!STI)
+      return;
+
+    std::unique_ptr<MemoryBuffer> Buffer(
+        MemoryBuffer::getMemBuffer(Frag.Asm, "<inline asm>"));
+    SourceMgr SrcMgr;
+    SrcMgr.AddNewSourceBuffer(std::move(Buffer), SMLoc());
+
+    MCContext MCCtx(TT, *MAI, *MRI, *STI, &SrcMgr);
+    std::unique_ptr<MCObjectFileInfo> MOFI(
+        T->createMCObjectFileInfo(MCCtx, /*PIC=*/false));
+    MCCtx.setObjectFileInfo(MOFI.get());
+    RecordStreamer Streamer(MCCtx, M);
+    T->createNullTargetStreamer(Streamer);
+
+    std::unique_ptr<MCAsmParser> Parser(
+        createMCAsmParser(SrcMgr, MCCtx, Streamer, *MAI));
+
+    std::unique_ptr<MCTargetAsmParser> TAP(
+        T->createMCAsmParser(*STI, *Parser, *MCII));
+    if (!TAP)
+      return;
+
+    MCCtx.setDiagnosticHandler([&](const SMDiagnostic &SMD, bool IsInlineAsm,
+                                   const SourceMgr &SrcMgr,
+                                   std::vector<const MDNode *> &LocInfos) {
+      M.getContext().diagnose(
+          DiagnosticInfoSrcMgr(SMD, M.getName(), IsInlineAsm, /*LocCookie=*/0));
+    });
+
+    // Module-level inline asm is assumed to use At&t syntax (see
+    // AsmPrinter::doInitialization()).
+    Parser->setAssemblerDialect(InlineAsm::AD_ATT);
+
+    Parser->setSymbolScanningMode(true);
+
+    Parser->setTargetParser(*TAP);
+    if (Parser->Run(false))
+      return;
+
+    Init(Streamer);
+  }
 }
 
 void ModuleSymbolTable::CollectAsmSymbols(

diff  --git a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
index dbda25c34cee0..e37ea27364e6e 100644
--- a/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
+++ b/llvm/lib/Target/RISCV/RISCVAsmPrinter.cpp
@@ -113,7 +113,8 @@ class RISCVAsmPrinter : public AsmPrinter {
   void emitEndOfAsmFile(Module &M) override;
 
   void emitFunctionEntryLabel() override;
-  bool emitDirectiveOptionArch();
+  bool emitTargetFeaturePush(const MCSubtargetInfo &STI) override;
+  void emitTargetFeaturePop(const MCSubtargetInfo &STI, bool DidPush) override;
 
   void emitNoteGnuProperty(const Module &M);
 
@@ -523,19 +524,19 @@ bool RISCVAsmPrinter::PrintAsmMemoryOperand(const MachineInstr *MI,
   return false;
 }
 
-bool RISCVAsmPrinter::emitDirectiveOptionArch() {
+bool RISCVAsmPrinter::emitTargetFeaturePush(const MCSubtargetInfo &STI) {
   RISCVTargetStreamer &RTS = getTargetStreamer();
   SmallVector<RISCVOptionArchArg> NeedEmitStdOptionArgs;
   const MCSubtargetInfo &MCSTI = TM.getMCSubtargetInfo();
   for (const auto &Feature : MCSTI.getAllProcessorFeatures()) {
-    if (STI->hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
+    if (STI.hasFeature(Feature.Value) == MCSTI.hasFeature(Feature.Value))
       continue;
 
     if (!llvm::RISCVISAInfo::isSupportedExtensionFeature(Feature.key()))
       continue;
 
-    auto Delta = STI->hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
-                                                : RISCVOptionArchArgType::Minus;
+    auto Delta = STI.hasFeature(Feature.Value) ? RISCVOptionArchArgType::Plus
+                                               : RISCVOptionArchArgType::Minus;
     StringRef ExtName = Feature.key();
     ExtName.consume_front("experimental-");
     NeedEmitStdOptionArgs.emplace_back(Delta, ExtName.str());
@@ -549,11 +550,16 @@ bool RISCVAsmPrinter::emitDirectiveOptionArch() {
   return false;
 }
 
+void RISCVAsmPrinter::emitTargetFeaturePop(const MCSubtargetInfo &STI,
+                                           bool DidPush) {
+  if (DidPush)
+    getTargetStreamer().emitDirectiveOptionPop();
+}
+
 bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   STI = &MF.getSubtarget<RISCVSubtarget>();
-  RISCVTargetStreamer &RTS = getTargetStreamer();
 
-  bool EmittedOptionArch = emitDirectiveOptionArch();
+  bool EmittedOptionArch = emitTargetFeaturePush(*STI);
 
   SetupMachineFunction(MF);
   emitFunctionBody();
@@ -561,8 +567,7 @@ bool RISCVAsmPrinter::runOnMachineFunction(MachineFunction &MF) {
   // Emit the XRay table
   emitXRayTable();
 
-  if (EmittedOptionArch)
-    RTS.emitDirectiveOptionPop();
+  emitTargetFeaturePop(*STI, EmittedOptionArch);
   return false;
 }
 

diff  --git a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
index 2a5d16a470c4a..22d0d97090746 100644
--- a/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVAsmPrinter.cpp
@@ -955,7 +955,7 @@ bool SPIRVAsmPrinter::doInitialization(Module &M) {
   if (!M.getModuleInlineAsm().empty()) {
     M.getContext().emitError(
         "SPIR-V does not support module-level inline assembly");
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
   }
 
   // Register the NSDI handler before calling the base class so that

diff  --git a/llvm/lib/Target/TargetMachine.cpp b/llvm/lib/Target/TargetMachine.cpp
index 9ec10b5be0fd4..95263f32236b4 100644
--- a/llvm/lib/Target/TargetMachine.cpp
+++ b/llvm/lib/Target/TargetMachine.cpp
@@ -23,6 +23,7 @@
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MC/TargetRegistry.h"
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 using namespace llvm;
@@ -311,3 +312,16 @@ std::pair<int, int> TargetMachine::parseBinutilsVersion(StringRef Version) {
     Version.consumeInteger(10, Ret.second);
   return Ret;
 }
+
+const MCSubtargetInfo &TargetMachine::getMCSubtargetInfo(StringRef CPU,
+                                                         StringRef FS) {
+  if (CPU.empty() && FS.empty())
+    return *STI;
+  SmallString<128> Key = CPU;
+  Key += '/';
+  Key += FS;
+  auto &Entry = MCSubtargetMap[Key];
+  if (!Entry)
+    Entry.reset(getTarget().createMCSubtargetInfo(getTargetTriple(), CPU, FS));
+  return *Entry;
+}

diff  --git a/llvm/lib/Transforms/IPO/ExtractGV.cpp b/llvm/lib/Transforms/IPO/ExtractGV.cpp
index 16fdb93e2b21b..4b76b02c32fe4 100644
--- a/llvm/lib/Transforms/IPO/ExtractGV.cpp
+++ b/llvm/lib/Transforms/IPO/ExtractGV.cpp
@@ -58,7 +58,7 @@ ExtractGVPass::ExtractGVPass(std::vector<GlobalValue *> &GVs, bool deleteS,
 PreservedAnalyses ExtractGVPass::run(Module &M, ModuleAnalysisManager &) {
   // Visit the global inline asm.
   if (!deleteStuff)
-    M.setModuleInlineAsm("");
+    M.removeModuleInlineAsm();
 
   // For simplicity, just give all GlobalValues ExternalLinkage. A trickier
   // implementation could figure out which GlobalValues are actually

diff  --git a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
index a486979a58c2d..e857c94330bad 100644
--- a/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
+++ b/llvm/lib/Transforms/IPO/ThinLTOBitcodeWriter.cpp
@@ -388,7 +388,7 @@ void splitAndWriteThinLTOBitcode(
         return false;
       }));
   StripDebugInfo(*MergedM);
-  MergedM->setModuleInlineAsm("");
+  MergedM->removeModuleInlineAsm();
 
   // Clone any llvm.*used globals to ensure the included values are
   // not deleted.

diff  --git a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
index c349fe33dd237..375ac069272eb 100644
--- a/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
+++ b/llvm/lib/Transforms/Instrumentation/DataFlowSanitizer.cpp
@@ -1283,18 +1283,20 @@ void DataFlowSanitizer::addGlobalNameSuffix(GlobalValue *GV) {
   // corrupting asm which happens to contain the symbol name as a substring.
   // Note that the substitution for .symver assumes that the versioned symbol
   // also has an instrumented name.
-  std::string Asm = GV->getParent()->getModuleInlineAsm();
-  std::string SearchStr = ".symver " + GVName + ",";
-  size_t Pos = Asm.find(SearchStr);
-  if (Pos != std::string::npos) {
-    Asm.replace(Pos, SearchStr.size(), ".symver " + GVName + Suffix + ",");
-    Pos = Asm.find('@');
-
-    if (Pos == std::string::npos)
-      report_fatal_error(Twine("unsupported .symver: ", Asm));
-
-    Asm.replace(Pos, 1, Suffix + "@");
-    GV->getParent()->setModuleInlineAsm(Asm);
+  for (Module::GlobalAsmFragment &Frag :
+       GV->getParent()->getModuleInlineAsm()) {
+    std::string SearchStr = ".symver " + GVName + ",";
+    size_t Pos = Frag.Asm.find(SearchStr);
+    if (Pos != std::string::npos) {
+      Frag.Asm.replace(Pos, SearchStr.size(),
+                       ".symver " + GVName + Suffix + ",");
+      Pos = Frag.Asm.find('@');
+
+      if (Pos == std::string::npos)
+        report_fatal_error(Twine("unsupported .symver: ", Frag.Asm));
+
+      Frag.Asm.replace(Pos, 1, Suffix + "@");
+    }
   }
 }
 

diff  --git a/llvm/lib/Transforms/Utils/SplitModule.cpp b/llvm/lib/Transforms/Utils/SplitModule.cpp
index 64910b4d1ce99..8e70902c4dac2 100644
--- a/llvm/lib/Transforms/Utils/SplitModule.cpp
+++ b/llvm/lib/Transforms/Utils/SplitModule.cpp
@@ -304,7 +304,7 @@ void llvm::SplitModule(
             return isInPartition(GV, I, N);
         }));
     if (I != 0)
-      MPart->setModuleInlineAsm("");
+      MPart->removeModuleInlineAsm();
     ModuleCallback(std::move(MPart));
   }
 }

diff  --git a/llvm/test/Assembler/module-asm-invalid.ll b/llvm/test/Assembler/module-asm-invalid.ll
new file mode 100644
index 0000000000000..34e5b71e683a9
--- /dev/null
+++ b/llvm/test/Assembler/module-asm-invalid.ll
@@ -0,0 +1,25 @@
+; RUN: split-file %s %t
+; RUN: not llvm-as -disable-output < %t/invalid_prop.ll 2>&1 | FileCheck %t/invalid_prop.ll
+; RUN: not llvm-as -disable-output < %t/missing_value1.ll 2>&1 | FileCheck %t/missing_value1.ll
+; RUN: not llvm-as -disable-output < %t/missing_value2.ll 2>&1 | FileCheck %t/missing_value2.ll
+; RUN: not llvm-as -disable-output < %t/missing_paren.ll 2>&1 | FileCheck %t/missing_paren.ll
+
+;--- invalid_prop.ll
+; CHECK: unknown property name
+module asm(foo: "bar")
+    "asm"
+
+;--- missing_value1.ll
+; CHECK: expected property name followed by ':'
+module asm(target_features)
+    "asm"
+
+;--- missing_value2.ll
+; CHECK: expected string constant
+module asm(target_features:)
+    "asm"
+
+;--- missing_paren.ll
+; CHECK: expected ',' or ')'
+module asm(target_features: "bar"
+    "asm"

diff  --git a/llvm/test/Bitcode/compatibility-3.6.ll b/llvm/test/Bitcode/compatibility-3.6.ll
index 288e0fc50a9ba..c79086bad1ba5 100644
--- a/llvm/test/Bitcode/compatibility-3.6.ll
+++ b/llvm/test/Bitcode/compatibility-3.6.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.7.ll b/llvm/test/Bitcode/compatibility-3.7.ll
index ef9490310fe5d..00b2bb9b30b62 100644
--- a/llvm/test/Bitcode/compatibility-3.7.ll
+++ b/llvm/test/Bitcode/compatibility-3.7.ll
@@ -14,7 +14,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.8.ll b/llvm/test/Bitcode/compatibility-3.8.ll
index 538d6101b3994..5b86d5ffa1fee 100644
--- a/llvm/test/Bitcode/compatibility-3.8.ll
+++ b/llvm/test/Bitcode/compatibility-3.8.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-3.9.ll b/llvm/test/Bitcode/compatibility-3.9.ll
index 3bfef73e485dd..eca4bbdf03766 100644
--- a/llvm/test/Bitcode/compatibility-3.9.ll
+++ b/llvm/test/Bitcode/compatibility-3.9.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-4.0.ll b/llvm/test/Bitcode/compatibility-4.0.ll
index aaf9fe5492171..8551b98815b1a 100644
--- a/llvm/test/Bitcode/compatibility-4.0.ll
+++ b/llvm/test/Bitcode/compatibility-4.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-5.0.ll b/llvm/test/Bitcode/compatibility-5.0.ll
index b820417107e8b..883ac3eaaa293 100644
--- a/llvm/test/Bitcode/compatibility-5.0.ll
+++ b/llvm/test/Bitcode/compatibility-5.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility-6.0.ll b/llvm/test/Bitcode/compatibility-6.0.ll
index 67e96c7277cc3..624d5bd7703c6 100644
--- a/llvm/test/Bitcode/compatibility-6.0.ll
+++ b/llvm/test/Bitcode/compatibility-6.0.ll
@@ -13,7 +13,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/compatibility.ll b/llvm/test/Bitcode/compatibility.ll
index d55aa1dc496df..97889d4687516 100644
--- a/llvm/test/Bitcode/compatibility.ll
+++ b/llvm/test/Bitcode/compatibility.ll
@@ -16,7 +16,8 @@ target triple = "x86_64-apple-macosx10.10.0"
 
 ;; Module-level assembly
 module asm "beep boop"
-; CHECK: module asm "beep boop"
+; CHECK: module asm
+; CHECK-NEXT: "beep boop"
 
 ;; Comdats
 $comdat.any = comdat any

diff  --git a/llvm/test/Bitcode/highLevelStructure.3.2.ll b/llvm/test/Bitcode/highLevelStructure.3.2.ll
index f2770ea4e4d34..cd9a34c810d57 100644
--- a/llvm/test/Bitcode/highLevelStructure.3.2.ll
+++ b/llvm/test/Bitcode/highLevelStructure.3.2.ll
@@ -9,7 +9,8 @@
 target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v24:32:32-v32:32:32-v48:64:64-v64:64:64-v96:128:128-v128:128:128-v192:256:256-v256:256:256-v512:512:512-v1024:1024:1024-a0:0:64-f80:32:32-n8:16:32-S32"
 
 ; Module-Level Inline Assembly Test
-; CHECK: module asm "some assembly"
+; CHECK: module asm
+; CHECK-NEXT: "some assembly"
 module asm "some assembly"
 
 ; Named Types Test

diff  --git a/llvm/test/Bitcode/module-asm.ll b/llvm/test/Bitcode/module-asm.ll
new file mode 100644
index 0000000000000..406f25c5dd28c
--- /dev/null
+++ b/llvm/test/Bitcode/module-asm.ll
@@ -0,0 +1,29 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; CHECK: module asm
+; CHECK:     "asm line 0"
+; CHECK: module asm(target_features: "+foo", target_cpu: "foo_cpu")
+; CHECK:     "asm line 1"
+; CHECK:     "asm line 2"
+; CHECK: module asm(target_features: "+bar")
+; CHECK:     "asm line 3"
+; CHECK: module asm(target_cpu: "bar_cpu")
+; CHECK:     "asm line 4"
+; CHECK: module asm
+; CHECK:     "asm line 5"
+
+module asm
+    "asm line 0"
+
+module asm(target_features: "+foo", target_cpu: "foo_cpu")
+    "asm line 1"
+    "asm line 2"
+
+module asm(target_features: "+bar")
+    "asm line 3"
+
+module asm(target_cpu: "bar_cpu")
+    "asm line 4"
+
+module asm
+    "asm line 5"

diff  --git a/llvm/test/CodeGen/RISCV/module-asm-features.ll b/llvm/test/CodeGen/RISCV/module-asm-features.ll
new file mode 100644
index 0000000000000..ab16acab48688
--- /dev/null
+++ b/llvm/test/CodeGen/RISCV/module-asm-features.ll
@@ -0,0 +1,17 @@
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu < %s | FileCheck %s --check-prefixes=CHECK,EXTRA-FEATURES
+; RUN: llc -mtriple=riscv64-unknown-linux-gnu -mattr=+d < %s | FileCheck %s --check-prefixes=CHECK,SAME-FEATURES
+
+; This should work fine, because the module asm specifies the necessary
+; target features
+
+; SAME-FEATURES-NOT: .option arch
+; EXTRA-FEATURES: .option push
+; EXTRA-FEATURES: .option arch, +d
+; CHECK: fld ft0, 0(sp)
+; EXTRA-FEATURES: .option pop
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"

diff  --git a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
index 454c496d1d4bb..829dd829d51f9 100644
--- a/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
+++ b/llvm/test/Instrumentation/DataFlowSanitizer/prefix-rename.ll
@@ -2,7 +2,8 @@
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".symver f1.dfsan,f.dfsan@@version1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver f1.dfsan,f.dfsan@@version1"
 module asm ".symver f1,f@@version1"
 
 ; CHECK: @f2.dfsan = alias {{.*}} @f1.dfsan

diff  --git a/llvm/test/LTO/RISCV/module-asm.ll b/llvm/test/LTO/RISCV/module-asm.ll
new file mode 100644
index 0000000000000..73320185e778a
--- /dev/null
+++ b/llvm/test/LTO/RISCV/module-asm.ll
@@ -0,0 +1,18 @@
+; RUN: llvm-as %s -o %t.o
+; RUN: llvm-lto2 run -save-temps -filetype=asm -o %t.s %t.o -r=%t.o,func,p
+; RUN: llvm-nm %t.o | FileCheck %s --check-prefix NM
+; RUN: llvm-nm %t.s.0.5.precodegen.bc | FileCheck %s --check-prefix NM
+; RUN: FileCheck %s --input-file %t.s.0
+
+; NM: T func
+
+; CHECK: fld ft0, 0(sp)
+
+target datalayout = "e-m:e-p:64:64-i64:64-i128:128-n32:64-S128"
+target triple = "riscv64-unknown-linux-gnu"
+
+module asm(target_features: "+d")
+    ".globl func"
+    "func:"
+    "fld f0, 0(sp)"
+    "ret"

diff  --git a/llvm/test/LTO/X86/inline-asm-lto-discard.ll b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
index 88492e4366489..6c5173c3f9553 100644
--- a/llvm/test/LTO/X86/inline-asm-lto-discard.ll
+++ b/llvm/test/LTO/X86/inline-asm-lto-discard.ll
@@ -32,17 +32,19 @@
 ; RUN:  -r %t4,foo@@VER1,px
 ; RUN: llvm-dis < %to3.0.0.preopt.bc | FileCheck %s --check-prefix=ASM3
 
-; ASM1:      module asm ".lto_discard foo"
-; ASM1-NEXT: module asm ".weak foo"
-; ASM1-NEXT: module asm ".equ foo,bar"
-
-; ASM2:      module asm ".lto_discard foo"
-; ASM2-NEXT: module asm ".weak foo"
-; ASM2-NEXT: module asm ".equ foo,bar"
-; ASM2-NEXT: module asm ".lto_discard"
-; ASM2-NEXT: module asm " .global foo ; foo: leal    2(%rdi), %eax"
-
-; ASM3-NOT:  module asm ".lto_discard foo"
+; ASM1: module asm
+; ASM1-NEXT: ".lto_discard foo"
+; ASM1-NEXT: ".weak foo"
+; ASM1-NEXT: ".equ foo,bar"
+
+; ASM2: module asm
+; ASM2-NEXT: ".lto_discard foo"
+; ASM2-NEXT: ".weak foo"
+; ASM2-NEXT: ".equ foo,bar"
+; ASM2-NEXT: ".lto_discard"
+; ASM2-NEXT: " .global foo ; foo: leal    2(%rdi), %eax"
+
+; ASM3-NOT:  ".lto_discard foo"
 
 ; SYM: T foo
 

diff  --git a/llvm/test/Linker/Inputs/module-asm.ll b/llvm/test/Linker/Inputs/module-asm.ll
new file mode 100644
index 0000000000000..b097c7067ccc1
--- /dev/null
+++ b/llvm/test/Linker/Inputs/module-asm.ll
@@ -0,0 +1,2 @@
+module asm(target_features: "+bar")
+    "asm 2"

diff  --git a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
index 5e12a8c88d90a..612bce5be18f1 100644
--- a/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
+++ b/llvm/test/Linker/link-arm-and-thumb-module-inline-asm.ll
@@ -9,11 +9,12 @@ target triple = "armv7-linux-gnueabihf"
 
 module asm "add r1, r2, r2"
 
-; CHECK:      module asm ".text"
-; CHECK-NEXT: module asm ".balign 4"
-; CHECK-NEXT: module asm ".arm"
-; CHECK-NEXT: module asm "add r1, r2, r2"
-; CHECK-NEXT: module asm ".text"
-; CHECK-NEXT: module asm ".balign 2"
-; CHECK-NEXT: module asm ".thumb"
-; CHECK-NEXT: module asm "orn r1, r2, r2"
+; CHECK: module asm
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 4"
+; CHECK-NEXT: ".arm"
+; CHECK-NEXT: "add r1, r2, r2"
+; CHECK-NEXT: ".text"
+; CHECK-NEXT: ".balign 2"
+; CHECK-NEXT: ".thumb"
+; CHECK-NEXT: "orn r1, r2, r2"

diff  --git a/llvm/test/Linker/module-asm.ll b/llvm/test/Linker/module-asm.ll
new file mode 100644
index 0000000000000..6c600981d72a5
--- /dev/null
+++ b/llvm/test/Linker/module-asm.ll
@@ -0,0 +1,9 @@
+; RUN: llvm-link %s %p/Inputs/module-asm.ll -S | FileCheck %s
+
+; CHECK: module asm(target_features: "+foo")
+; CHECK-NEXT: "asm 1"
+; CHECK: module asm(target_features: "+bar")
+; CHECK-NEXT: "asm 2"
+
+module asm(target_features: "+foo")
+    "asm 1"

diff  --git a/llvm/test/ThinLTO/X86/import-symver.ll b/llvm/test/ThinLTO/X86/import-symver.ll
index 556c4fd992f0d..9525527b4531c 100644
--- a/llvm/test/ThinLTO/X86/import-symver.ll
+++ b/llvm/test/ThinLTO/X86/import-symver.ll
@@ -9,7 +9,8 @@
 ; RUN: llvm-dis %t1.bc.thinlto.imported.bc -o - | FileCheck --check-prefix=NOIMPORT %s
 
 ; When @bar gets imported, the symver must be imported too.
-; IMPORT: module asm ".symver bar, bar at BAR_1.2.3"
+; IMPORT: module asm
+; IMPORT-NEXT: ".symver bar, bar at BAR_1.2.3"
 ; IMPORT: declare dso_local i32 @bar()
 
 ; When @bar isn't imported, the symver is also not imported.

diff  --git a/llvm/test/Transforms/LowerTypeTests/export-symver.ll b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
index ea4594a359cc0..7c3ed3da3537e 100644
--- a/llvm/test/Transforms/LowerTypeTests/export-symver.ll
+++ b/llvm/test/Transforms/LowerTypeTests/export-symver.ll
@@ -1,6 +1,7 @@
 ; RUN: opt -S %s -passes=lowertypetests -lowertypetests-summary-action=export -lowertypetests-read-summary=%S/Inputs/exported-funcs.yaml | FileCheck %s
 ;
-; CHECK: module asm ".symver external_addrtaken, alias1"
+; CHECK: module asm
+; CHECK-NEXT: ".symver external_addrtaken, alias1"
 ; CHECK-NOT: .symver external_addrtaken2
 ; CHECK-NOT: .symver not_exported
 

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
index d8ebae17d4693..61141c71335bf 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/cfi-icall-static-inline-asm.ll
@@ -3,7 +3,8 @@
 
 target triple = "x86_64-unknown-linux-gnu"
 
-; CHECK: module asm ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
+; CHECK: module asm
+; CHECK-NEXT: ".lto_set_conditional a,a.[[HASH:[0-9a-f]+]]"
 
 define void @b() {
   %f = alloca ptr, align 8

diff  --git a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
index 587ab3fbb5020..5796537661e60 100644
--- a/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
+++ b/llvm/test/Transforms/ThinLTOBitcodeWriter/x86/module-asm.ll
@@ -7,6 +7,7 @@ target triple = "x86_64-unknown-linux-gnu"
 @g = constant i32 0, !type !0
 !0 = !{i32 0, !"typeid"}
 
-; M0: module asm "ret"
+; M0: module asm
+; M0-NEXT: "ret"
 ; M1-NOT: module asm
 module asm "ret"

diff  --git a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
index 4aeaef6d8d676..b5b68404e8690 100644
--- a/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
+++ b/llvm/tools/llvm-reduce/deltas/ReduceModuleData.cpp
@@ -23,5 +23,5 @@ void llvm::reduceModuleDataDeltaPass(Oracle &O, ReducerWorkItem &WorkItem) {
     Program.setSourceFileName("");
   // TODO: clear line by line rather than all at once
   if (!Program.getModuleInlineAsm().empty() && !O.shouldKeep())
-    Program.setModuleInlineAsm("");
+    Program.removeModuleInlineAsm();
 }

diff  --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index 3038a6a2ed986..bb55678f91005 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -1150,12 +1150,15 @@ void ModuleImport::convertTargetTriple() {
 }
 
 void ModuleImport::convertModuleLevelAsm() {
-  llvm::StringRef asmStr = llvmModule->getModuleInlineAsm();
   llvm::SmallVector<mlir::Attribute> asmArrayAttr;
 
-  for (llvm::StringRef line : llvm::split(asmStr, '\n'))
-    if (!line.empty())
-      asmArrayAttr.push_back(builder.getStringAttr(line));
+  for (const llvm::Module::GlobalAsmFragment &Frag :
+       llvmModule->getModuleInlineAsm()) {
+    // TODO: Preserve module asm properties.
+    for (llvm::StringRef line : llvm::split(Frag.Asm, '\n'))
+      if (!line.empty())
+        asmArrayAttr.push_back(builder.getStringAttr(line));
+  }
 
   mlirModule->setAttr(LLVM::LLVMDialect::getModuleLevelAsmAttrName(),
                       builder.getArrayAttr(asmArrayAttr));

diff  --git a/mlir/test/Target/LLVMIR/module-asm.mlir b/mlir/test/Target/LLVMIR/module-asm.mlir
index 2afb37cf005bf..813eeeafdeb44 100644
--- a/mlir/test/Target/LLVMIR/module-asm.mlir
+++ b/mlir/test/Target/LLVMIR/module-asm.mlir
@@ -2,5 +2,6 @@
 
 module attributes {llvm.module_asm = ["foo", "bar"]} {}
 
-// CHECK: module asm "foo"
-// CHECK: module asm "bar"
+// CHECK: module asm
+// CHECK-NEXT: "foo"
+// CHECK-NEXT: "bar"


        

From llvmlistbot at llvm.org  Mon Jul  6 02:57:53 2026
From: llvmlistbot at llvm.org (Nikita Popov)
Date: Mon, 06 Jul 2026 02:57:53 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207677@github.com>
Message-ID: <6a4b7c21.af6ba3ed.2c2f30.ab40@mx.google.com>


https://github.com/nikic edited https://github.com/llvm/llvm-project/pull/207677

From llvmlistbot at llvm.org  Mon Jul  6 03:02:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 03:02:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 58bc8df - [mlir][x86] AMX memref source check
	(#206785)
Message-ID: <6a4b7d52.492e23e4.e71ca.e091@mx.google.com>


Author: Adam Siemieniuk
Date: 2026-07-06T12:02:53+02:00
New Revision: 58bc8dff332e0d4d7609b6dbda62200355bbc75f

URL: https://github.com/llvm/llvm-project/commit/58bc8dff332e0d4d7609b6dbda62200355bbc75f
DIFF: https://github.com/llvm/llvm-project/commit/58bc8dff332e0d4d7609b6dbda62200355bbc75f.diff

LOG: [mlir][x86] AMX memref source check (#206785)

Adds extra checks to AMX lowering to ensure data is read from memrefs as
tensors are not supported by x86 dialect ops.

Added: 
    

Modified: 
    mlir/lib/Dialect/X86/Transforms/VectorContractToAMXDotProduct.cpp
    mlir/test/Dialect/X86/AMX/vector-contract-to-tiled-dp.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/X86/Transforms/VectorContractToAMXDotProduct.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractToAMXDotProduct.cpp
index 86291ae03f29e..238d03ce4f6af 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractToAMXDotProduct.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractToAMXDotProduct.cpp
@@ -68,6 +68,20 @@ static Value collapseInnerDims(OpBuilder &builder, mlir::Location loc,
   return memref::CollapseShapeOp::create(builder, loc, input, reassociation);
 }
 
+// Check if a vector.contract operand has a memref read source.
+static bool isReadSrcMemref(Value operand) {
+  Operation *defOp = operand.getDefiningOp();
+  if (!defOp)
+    return false;
+
+  Value srcBuff;
+  llvm::TypeSwitch<Operation *>(operand.getDefiningOp())
+      .Case<TransferReadOp, LoadOp>(
+          [&](auto readOp) { srcBuff = readOp.getOperand(0); });
+
+  return srcBuff && isa<MemRefType>(srcBuff.getType());
+}
+
 // Get the MemRef source and offset index for the operands of
 // vector.contract.
 static FailureOr<std::pair<Value, SmallVector<Value>>>
@@ -86,7 +100,7 @@ getSrcIndxValue(OpBuilder &rewriter, Location loc, Value operand,
         srcBuff = readOp.getOperand(0);
       });
 
-  if (!srcBuff)
+  if (!srcBuff || !isa<MemRefType>(srcBuff.getType()))
     return failure();
 
   if (isNotAcc)
@@ -816,6 +830,11 @@ struct VectorContractToAMXDotProduct
       return rewriter.notifyMatchFailure(
           contractOp, "The accumulator read is in 
diff erent block.");
 
+    if (!(isReadSrcMemref(contractOp.getLhs()) &&
+          isReadSrcMemref(contractOp.getRhs())))
+      return rewriter.notifyMatchFailure(
+          contractOp, "The LHS or RHS src is not a MemRef type.");
+
     unsigned int dimValue = blockingFactor;
     if (!isVnni)
       dimValue = 16 * blockingFactor;
@@ -825,6 +844,10 @@ struct VectorContractToAMXDotProduct
     if (accReadOp->getBlock() == contractOp->getBlock() &&
         resultWriteOp->getBlock() == contractOp->getBlock()) {
 
+      if (!isReadSrcMemref(contractOp.getAcc()))
+        return rewriter.notifyMatchFailure(contractOp,
+                                           "The ACC src is not a MemRef type.");
+
       bool collapse = false;
       if (isVnni)
         collapse = true;
@@ -844,14 +867,14 @@ struct VectorContractToAMXDotProduct
                                         contractOp.getLhs(), collapse);
       if (failed(srcIndxLhs))
         return rewriter.notifyMatchFailure(contractOp,
-                                           "The LHS src is not a MemRef type.");
+                                           "Failed to get the LHS src.");
       auto [srcBuffLhs, indicesLhs] = *srcIndxLhs;
 
       auto srcIndxRhs = getSrcIndxValue(rewriter, contractOp.getLoc(),
                                         contractOp.getRhs(), collapse);
       if (failed(srcIndxRhs))
         return rewriter.notifyMatchFailure(contractOp,
-                                           "The RHS src is not a MemRef type.");
+                                           "Failed to get the RHS src.");
       auto rhsSrc = *srcIndxRhs;
       auto srcBuffRhs = rhsSrc.first;
       auto indicesRhs = rhsSrc.second;
@@ -860,7 +883,7 @@ struct VectorContractToAMXDotProduct
                                         contractOp.getAcc(), false);
       if (failed(srcIndxAcc))
         return rewriter.notifyMatchFailure(contractOp,
-                                           "The ACC src is not a MemRef type.");
+                                           "Failed to get the ACC src.");
       auto [srcBuffAcc, indicesAcc] = *srcIndxAcc;
 
       Value c0 = arith::ConstantIndexOp::create(rewriter, loc, 0);
@@ -1043,14 +1066,14 @@ struct VectorContractToAMXDotProduct
                                       contractOp.getLhs(), false);
     if (failed(srcIndxLhs))
       return rewriter.notifyMatchFailure(contractOp,
-                                         "The LHS src is not a MemRef type.");
+                                         "Failed to get the LHS src.");
     auto [srcBuffLhs, indicesLhs] = *srcIndxLhs;
 
     auto srcIndxRhs = getSrcIndxValue(rewriter, contractOp.getLoc(),
                                       contractOp.getRhs(), false);
     if (failed(srcIndxRhs))
       return rewriter.notifyMatchFailure(contractOp,
-                                         "The RHS src is not a MemRef type.");
+                                         "Failed to get the RHS src.");
     auto [srcBuffRhs, indicesRhs] = *srcIndxRhs;
     Operation *vectorOpLhs;
     llvm::TypeSwitch<Operation *>(contractOp.getLhs().getDefiningOp())

diff  --git a/mlir/test/Dialect/X86/AMX/vector-contract-to-tiled-dp.mlir b/mlir/test/Dialect/X86/AMX/vector-contract-to-tiled-dp.mlir
index 71bf62c56a6e5..ad983ee5d6ff1 100644
--- a/mlir/test/Dialect/X86/AMX/vector-contract-to-tiled-dp.mlir
+++ b/mlir/test/Dialect/X86/AMX/vector-contract-to-tiled-dp.mlir
@@ -2078,3 +2078,147 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+!vecA = vector<16x16x4xi8>
+!vecB = vector<16x16x4xi8>
+!vecC = vector<16x16xi32>
+!tensorA = tensor<32x16x4xi8>
+!memrefB = memref<16x32x4xi8>
+!memrefC = memref<32x32xi32>
+#map = affine_map<(d4, d1, d2, d3) -> (d1, d3, d4)>
+#map1 = affine_map<(d4, d1, d2, d3) -> (d3, d2, d4)>
+#map2 = affine_map<(d4, d1, d2, d3) -> (d1, d2)>
+func.func @negative_lhs_tensor(
+  %arg0: !tensorA, %arg1: !memrefB, %arg2: !memrefC) -> !memrefC
+{
+  %c0 = arith.constant 0 : index
+  %0 = ub.poison : i8
+  %32 = ub.poison : i32
+
+  %1 = vector.transfer_read %arg0[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !tensorA, !vecA
+  %2 = vector.transfer_read %arg1[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !memrefB, !vecB
+  %3 = vector.transfer_read %arg2[%c0, %c0], %32 {in_bounds = [true, true]} : !memrefC, !vecC
+
+  %4 = vector.contract {
+    indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>}
+    %1, %2, %3 : !vecA, !vecB into !vecC
+
+  vector.transfer_write %4, %arg2[%c0, %c0] {in_bounds = [true, true]} : !vecC, !memrefC
+  return %arg2 : !memrefC
+}
+
+// CHECK-LABEL: @negative_lhs_tensor
+// CHECK-NOT: x86.amx
+// CHECK: vector.contract
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.vector_contract_to_amx_dot_product
+    } : !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+!vecA = vector<16x16x4xi8>
+!vecB = vector<16x16x4xi8>
+!vecC = vector<16x16xi32>
+!memrefA = memref<32x16x4xi8>
+!tensorB = tensor<16x32x4xi8>
+!memrefC = memref<32x32xi32>
+#map = affine_map<(d4, d1, d2, d3) -> (d1, d3, d4)>
+#map1 = affine_map<(d4, d1, d2, d3) -> (d3, d2, d4)>
+#map2 = affine_map<(d4, d1, d2, d3) -> (d1, d2)>
+func.func @negative_rhs_tensor(
+  %arg0: !memrefA, %arg1: !tensorB, %arg2: !memrefC) -> !memrefC
+{
+  %c0 = arith.constant 0 : index
+  %0 = ub.poison : i8
+  %32 = ub.poison : i32
+
+  %1 = vector.transfer_read %arg0[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !memrefA, !vecA
+  %2 = vector.transfer_read %arg1[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !tensorB, !vecB
+  %3 = vector.transfer_read %arg2[%c0, %c0], %32 {in_bounds = [true, true]} : !memrefC, !vecC
+
+  %4 = vector.contract {
+    indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>}
+    %1, %2, %3 : !vecA, !vecB into !vecC
+
+  vector.transfer_write %4, %arg2[%c0, %c0] {in_bounds = [true, true]} : !vecC, !memrefC
+  return %arg2 : !memrefC
+}
+
+// CHECK-LABEL: @negative_rhs_tensor
+// CHECK-NOT: x86.amx
+// CHECK: vector.contract
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.vector_contract_to_amx_dot_product
+    } : !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+!vecA = vector<16x16x4xi8>
+!vecB = vector<16x16x4xi8>
+!vecC = vector<16x16xi32>
+!memrefA = memref<32x16x4xi8>
+!memrefB = memref<16x32x4xi8>
+!tensorC = tensor<32x32xi32>
+#map = affine_map<(d4, d1, d2, d3) -> (d1, d3, d4)>
+#map1 = affine_map<(d4, d1, d2, d3) -> (d3, d2, d4)>
+#map2 = affine_map<(d4, d1, d2, d3) -> (d1, d2)>
+func.func @negative_acc_tensor(
+  %arg0: !memrefA, %arg1: !memrefB, %arg2: !tensorC) -> !tensorC
+{
+  %c0 = arith.constant 0 : index
+  %0 = ub.poison : i8
+  %32 = ub.poison : i32
+
+  %1 = vector.transfer_read %arg0[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !memrefA, !vecA
+  %2 = vector.transfer_read %arg1[%c0, %c0, %c0], %0 {in_bounds = [true, true, true]} :
+        !memrefB, !vecB
+  %3 = vector.transfer_read %arg2[%c0, %c0], %32 {in_bounds = [true, true]} : !tensorC, !vecC
+
+  %4 = vector.contract {
+    indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>}
+    %1, %2, %3 : !vecA, !vecB into !vecC
+
+  %5 = vector.transfer_write %4, %arg2[%c0, %c0] {in_bounds = [true, true]} : !vecC, !tensorC
+  return %5 : !tensorC
+}
+
+// CHECK-LABEL: @negative_acc_tensor
+// CHECK-NOT: x86.amx
+// CHECK: vector.contract
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.vector_contract_to_amx_dot_product
+    } : !transform.any_op
+    transform.yield
+  }
+}


        

From llvmlistbot at llvm.org  Mon Jul  6 03:03:00 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Mon, 06 Jul 2026 03:03:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] AMX memref source check (PR
	#206785)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206785@github.com>
Message-ID: <6a4b7d54.27b3b003.1610ae.705e@mx.google.com>


https://github.com/adam-smnk closed https://github.com/llvm/llvm-project/pull/206785

From llvmlistbot at llvm.org  Mon Jul  6 03:07:52 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Mon, 06 Jul 2026 03:07:52 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207677@github.com>
Message-ID: <6a4b7e78.8b87e04f.199b71.6c0a@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `clang,llvm,mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11809

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachODebugMapParser.cpp:10:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8313/8689] Building CXX object tools/llvm-opt-report/CMakeFiles/llvm-opt-report.dir/OptReport.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTA
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8315/8689] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/BytesOutputStyle.cpp.o
[8316/8689] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVGPU.dir/NVGPU.cpp.o
[8317/8689] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/dsymutil.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.h:20,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.cpp:13:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8318/8689] Building CXX object tools/llc/CMakeFiles/llc.dir/llc.cpp.o
[8319/8689] Building CXX object tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/clang-import-test.cpp.o
[8320/8689] Building CXX object tools/clang/tools/clang-sycl-linker/CMakeFiles/clang-sycl-linker.dir/ClangSYCLLinker.cpp.o
[8321/8689] Building CXX object tools/llc/CMakeFiles/llc.dir/NewPMDriver.cpp.o
[8322/8689] Building CXX object tools/clang/tools/clang-linker-wrapper/CMakeFiles/clang-linker-wrapper.dir/ClangLinkerWrapper.cpp.o
[8323/8689] Building CXX object tools/llvm-dis/CMakeFiles/llvm-dis.dir/llvm-dis.cpp.o
[8324/8689] Building CXX object tools/llvm-modextract/CMakeFiles/llvm-modextract.dir/llvm-modextract.cpp.o
[8325/8689] Building CXX object tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/llvm-isel-fuzzer.cpp.o
[8326/8689] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/DwarfLinkerForBinary.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8327/8689] Building CXX object tools/llvm-offload-wrapper/CMakeFiles/llvm-offload-wrapper.dir/llvm-offload-wrapper.cpp.o
[8328/8689] Building CXX object tools/llvm-lipo/CMakeFiles/llvm-lipo.dir/llvm-lipo.cpp.o
[8329/8689] Building CXX object tools/llvm-dwarfutil/CMakeFiles/llvm-dwarfutil.dir/DebugInfoLinker.cpp.o
[8330/8689] Building CXX object tools/llvm-ir2vec/CMakeFiles/llvm-ir2vec.dir/llvm-ir2vec.cpp.o
Step 7 (build cmake config) failure: build cmake config (failure)
...
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/MachODebugMapParser.cpp:10:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8313/8689] Building CXX object tools/llvm-opt-report/CMakeFiles/llvm-opt-report.dir/OptReport.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTA
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8315/8689] Building CXX object tools/llvm-pdbutil/CMakeFiles/llvm-pdbutil.dir/BytesOutputStyle.cpp.o
[8316/8689] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVGPU.dir/NVGPU.cpp.o
[8317/8689] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/dsymutil.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.h:20,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/dsymutil.cpp:13:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8318/8689] Building CXX object tools/llc/CMakeFiles/llc.dir/llc.cpp.o
[8319/8689] Building CXX object tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/clang-import-test.cpp.o
[8320/8689] Building CXX object tools/clang/tools/clang-sycl-linker/CMakeFiles/clang-sycl-linker.dir/ClangSYCLLinker.cpp.o
[8321/8689] Building CXX object tools/llc/CMakeFiles/llc.dir/NewPMDriver.cpp.o
[8322/8689] Building CXX object tools/clang/tools/clang-linker-wrapper/CMakeFiles/clang-linker-wrapper.dir/ClangLinkerWrapper.cpp.o
[8323/8689] Building CXX object tools/llvm-dis/CMakeFiles/llvm-dis.dir/llvm-dis.cpp.o
[8324/8689] Building CXX object tools/llvm-modextract/CMakeFiles/llvm-modextract.dir/llvm-modextract.cpp.o
[8325/8689] Building CXX object tools/llvm-isel-fuzzer/CMakeFiles/llvm-isel-fuzzer.dir/llvm-isel-fuzzer.cpp.o
[8326/8689] Building CXX object tools/dsymutil/CMakeFiles/dsymutil.dir/DwarfLinkerForBinary.cpp.o
In file included from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.h:13,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DwarfLinkerForBinary.cpp:9:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects()’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:118:48: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  118 |     return reinterpret_cast<ObjectContainer &>(Objects);
      |                                                ^~~~~~~
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h: In member function ‘const llvm::dsymutil::DebugMap::ObjectContainer& llvm::dsymutil::DebugMap::getObjects() const’:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/dsymutil/DebugMap.h:121:54: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
  121 |     return reinterpret_cast<const ObjectContainer &>(Objects);
      |                                                      ^~~~~~~
[8327/8689] Building CXX object tools/llvm-offload-wrapper/CMakeFiles/llvm-offload-wrapper.dir/llvm-offload-wrapper.cpp.o
[8328/8689] Building CXX object tools/llvm-lipo/CMakeFiles/llvm-lipo.dir/llvm-lipo.cpp.o
[8329/8689] Building CXX object tools/llvm-dwarfutil/CMakeFiles/llvm-dwarfutil.dir/DebugInfoLinker.cpp.o
[8330/8689] Building CXX object tools/llvm-ir2vec/CMakeFiles/llvm-ir2vec.dir/llvm-ir2vec.cpp.o

```

</details>

https://github.com/llvm/llvm-project/pull/207677

From llvmlistbot at llvm.org  Mon Jul  6 03:18:20 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Mon, 06 Jul 2026 03:18:20 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Reapply [IR] Explicitly
 specify target feature for module asm (#204548) (PR #207677)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207677@github.com>
Message-ID: <6a4b80ec.510f576d.185953.88dc@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `fuchsia-x86_64-linux` running on `fuchsia-debian-64-us-central1-b-1` while building `clang,llvm,mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/11/builds/43517

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: 'python ../llvm-zorg/zorg/buildbot/builders/annotated/fuchsia-linux.py ...' (failure)
...
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/lld-link
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/wasm-ld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/lld-link
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/wasm-ld
-- Testing: 25304 tests, 60 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50
FAIL: Clang :: Driver/darwin-ld.c (13829 of 25304)
******************** TEST 'Clang :: Driver/darwin-ld.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 3
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -arch i386 -arch x86_64 /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -### -o foo 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -arch i386 -arch x86_64 /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c '-###' -o foo
# RUN: at line 4
grep '".*ld.*" .*"-arch_multiple" "-final_output" "foo"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep '".*ld.*" .*"-arch_multiple" "-final_output" "foo"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld" "-demangle" "-dynamic" "-arch" "i386" "-platform_version" "macos" "10.5.0" "10.5.0" "-mllvm" "-enable-linkonceodr-outlining" "-o" "/tmp/lit-tmp-uej38dwa/darwin-ld-d63f43/darwin-ld-i386.out" "-lcrt1.10.5.o" "/tmp/lit-tmp-uej38dwa/darwin-ld-6214c7/darwin-ld-i386.o" "-arch_multiple" "-final_output" "foo" "-lSystem" "-lgcc_s.1"
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld" "-demangle" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "10.5.0" "10.5.0" "-mllvm" "-enable-linkonceodr-outlining" "-o" "/tmp/lit-tmp-uej38dwa/darwin-ld-14f623/darwin-ld-x86_64.out" "-lcrt1.10.5.o" "/tmp/lit-tmp-uej38dwa/darwin-ld-8aafe0/darwin-ld-x86_64.o" "-arch_multiple" "-final_output" "foo" "-lSystem" "-lgcc_s.1"
# `-----------------------------
# RUN: at line 7
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -### -g /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 '-###' -g /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR
# RUN: at line 8
grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/dsymutil" "-o" "BAR.dSYM" "BAR"
# `-----------------------------
# RUN: at line 9
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -### -g -filelist FOO /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 '-###' -g -filelist FOO /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR
# RUN: at line 10
grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/dsymutil" "-o" "BAR.dSYM" "BAR"
# `-----------------------------
# RUN: at line 13
touch /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o
# executed command: touch /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o
# RUN: at line 14
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -fuse-ld= -mlinker-version=400 -### -arch armv6 -miphoneos-version-min=3.0 /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
Step 7 (check) failure: check (failure)
...
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/lld-link
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/wasm-ld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/lld-link
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld
llvm-lit: /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/wasm-ld
-- Testing: 25304 tests, 60 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50
FAIL: Clang :: Driver/darwin-ld.c (13829 of 25304)
******************** TEST 'Clang :: Driver/darwin-ld.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 3
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -arch i386 -arch x86_64 /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -### -o foo 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -arch i386 -arch x86_64 /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c '-###' -o foo
# RUN: at line 4
grep '".*ld.*" .*"-arch_multiple" "-final_output" "foo"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep '".*ld.*" .*"-arch_multiple" "-final_output" "foo"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld" "-demangle" "-dynamic" "-arch" "i386" "-platform_version" "macos" "10.5.0" "10.5.0" "-mllvm" "-enable-linkonceodr-outlining" "-o" "/tmp/lit-tmp-uej38dwa/darwin-ld-d63f43/darwin-ld-i386.out" "-lcrt1.10.5.o" "/tmp/lit-tmp-uej38dwa/darwin-ld-6214c7/darwin-ld-i386.o" "-arch_multiple" "-final_output" "foo" "-lSystem" "-lgcc_s.1"
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/ld64.lld" "-demangle" "-dynamic" "-arch" "x86_64" "-platform_version" "macos" "10.5.0" "10.5.0" "-mllvm" "-enable-linkonceodr-outlining" "-o" "/tmp/lit-tmp-uej38dwa/darwin-ld-14f623/darwin-ld-x86_64.out" "-lcrt1.10.5.o" "/tmp/lit-tmp-uej38dwa/darwin-ld-8aafe0/darwin-ld-x86_64.o" "-arch_multiple" "-final_output" "foo" "-lSystem" "-lgcc_s.1"
# `-----------------------------
# RUN: at line 7
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -### -g /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 '-###' -g /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR
# RUN: at line 8
grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/dsymutil" "-o" "BAR.dSYM" "BAR"
# `-----------------------------
# RUN: at line 9
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -### -g -filelist FOO /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 '-###' -g -filelist FOO /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/clang/test/Driver/darwin-ld.c -o BAR
# RUN: at line 10
grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# executed command: grep -E '".*dsymutil(\.exe)?" "-o" "BAR.dSYM" "BAR"' /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log
# .---command stdout------------
# |  "/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/dsymutil" "-o" "BAR.dSYM" "BAR"
# `-----------------------------
# RUN: at line 13
touch /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o
# executed command: touch /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o
# RUN: at line 14
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/bin/clang -target i386-apple-darwin9 -fuse-ld= -mlinker-version=400 -### -arch armv6 -miphoneos-version-min=3.0 /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.o 2> /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-7crtmodg/tools/clang/test/Driver/Output/darwin-ld.c.tmp.log

```

</details>

https://github.com/llvm/llvm-project/pull/207677

From llvmlistbot at llvm.org  Mon Jul  6 03:39:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 03:39:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4b85df.ff037943.3bf0e2.df4f@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 8381 tests passed
* 630 tests skipped
* 1 test failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.python/rewrite.py</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/usr/bin/python3 /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py 2>&1 | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py
# executed command: /usr/bin/python3 /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py:319:11: error: CHECK: expected string not found in input
# |  # CHECK: failed to legalize unresolved materialization
# |           ^
# | <stdin>:77:44: note: scanning from here
# | caught exception: partial conversion failed:
# |                                            ^
# | <stdin>:80:2: note: possible intended match here
# |  note: "-":5:26: require this materialization is here
# |  ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/python/rewrite.py
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |              .
# |              .
# |              .
# |             72:  %3 = builtin.unrealized_conversion_cast %2 : !smt.int to i64 
# |             73:  return %3 : i64 
# |             74:  } 
# |             75: } 
# |             76:  
# |             77: caught exception: partial conversion failed: 
# | check:319'0                                               {    search range start (exclusive)
# | check:319'1                                                    error: no match found in search range
# |             78: error: "-":5:26: miss target materialization function from ('i64') to ('!smt.int') 
# |             79:  note: "-":5:26: see current operation: %0 = "builtin.unrealized_conversion_cast"(%arg0) : (i64) -> !smt.int 
# |             80:  note: "-":5:26: require this materialization is here 
# | check:319'2      ?                                                      possible intended match
# |             81: <class 'mlir._mlir_libs._mlirDialectsSMT.IntType'> 
# |             82: !smt.int 
# |             83: None 
# | check:319'3          } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 03:59:37 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 03:59:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
Message-ID: <llvm.org/llvm/llvm-project/pull/207701@github.com>


https://github.com/HagaiLevHacohen created https://github.com/llvm/llvm-project/pull/207701

Previously, the valuebounds implementation for MinUI and MaxUI was too conservative.
We can already make correct assumptions even when we know only one operand is provably not negative.

>From f83e31c0d0c939bbd784507a45591474676bcbc6 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 11:27:19 +0300
Subject: [PATCH] Improving MinUI and MaxUI implementation in valuebounds

---
 .../Arith/IR/ValueBoundsOpInterfaceImpl.cpp   | 34 ++++++++++++-------
 .../Arith/value-bounds-op-interface-impl.mlir | 26 ++++++++++----
 2 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..074f7efffc81b 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,24 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +388,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace
diff --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }


From llvmlistbot at llvm.org  Mon Jul  6 04:00:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 04:00:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b8abb.bb985344.17fb3b.8885@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Hagai Lev Hacohen (HagaiLevHacohen)

<details>
<summary>Changes</summary>

Previously, the valuebounds implementation for MinUI and MaxUI was too conservative.
We can already make correct assumptions even when we know only one operand is provably not negative.

---
Full diff: https://github.com/llvm/llvm-project/pull/207701.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp (+21-13) 
- (modified) mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir (+19-7) 


``````````diff
diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..074f7efffc81b 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,24 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +388,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace
diff --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 04:02:00 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 04:02:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b8b28.6c8668f2.346e76.7973@mx.google.com>


HagaiLevHacohen wrote:

@matthias-springer

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 04:07:11 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 04:07:11 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4b8c5f.492e23e4.e71ca.19a9@mx.google.com>


https://github.com/skatrak commented:

I still have a couple of comments on implementation details.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul  6 04:07:11 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 04:07:11 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4b8c5f.491d90f5.1f0d58.616c@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul  6 04:07:12 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 04:07:12 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4b8c60.eb24c6ea.1f2e26.0f80@mx.google.com>


================
@@ -8222,6 +8226,61 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Validate and resolve in_reduction clauses on omp.target. We currently
+  // only support the non-offload host-fallback path: the per-task private
+  // pointer is obtained by calling __kmpc_task_reduction_get_th_data inside
+  // the to-be-outlined target task body. Threading that pointer through the
+  // device kernel argument list is left as follow-up work.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  if (!targetOp.getInReductionVars().empty()) {
+    if (isTargetDevice || isOffloadEntry)
+      return opInst.emitError(
+          "not yet implemented: in_reduction clause on omp.target with "
+          "offload / target-device compilation");
+    if (auto inRedSyms = targetOp.getInReductionSyms()) {
+      for (auto sym : inRedSyms->getAsRange<SymbolRefAttr>()) {
+        auto decl =
+            SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                targetOp, sym);
+        if (!decl)
+          return targetOp.emitError()
+                 << "failed to resolve in_reduction declare_reduction symbol "
+                 << sym.getRootReference() << " on omp.target";
+        if (decl.getInitializerRegion().front().getNumArguments() != 1)
+          return targetOp.emitError()
+                 << "not yet implemented: in_reduction with two-argument "
+                    "initializer on omp.target";
+        if (!decl.getCleanupRegion().empty())
+          return targetOp.emitError()
+                 << "not yet implemented: in_reduction with cleanup region "
+                    "on omp.target";
+        // The reduction combiner region is intentionally not required here:
+        // the in_reduction lowering on omp.target only locates the per-task
+        // private storage via __kmpc_task_reduction_get_th_data. The combiner
+        // is owned by the enclosing taskgroup's task_reduction registration.
+      }
+    }
+    // Each in_reduction variable must also be captured by the target via a
+    // map_entries entry referring to the same outer SSA value. OMPIRBuilder
+    // outlines the target body and only rewires uses of values that enter
+    // the kernel through the map-derived input set. The runtime call below
+    // uses that same outer SSA value as its `orig` argument, so without a
+    // matching map entry the outlined kernel would reference a value defined
+    // in the host function and fail IR verification.
+    llvm::SmallPtrSet<Value, 4> mappedVarPtrs;
+    for (Value mapV : targetOp.getMapVars())
+      if (auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>())
+        mappedVarPtrs.insert(mapInfo.getVarPtr());
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      if (!mappedVarPtrs.contains(v))
+        return targetOp.emitError()
+               << "not yet implemented: in_reduction variable on omp.target "
+                  "must also be captured by a matching map_entries entry";
----------------
skatrak wrote:

This is still not addressed, I'll copy again my previous suggestion:
> The best we can do at the moment is to check in the verifier that all in_reduction operands are produced by the same operation as the var_ptr of at least one of the map_entries of the same operation. And then, where you're currently making the "not yet implemented: in_reduction variable on omp.target must also be captured by a matching map_entries entry" check put an assert instead. That'd be my suggestion in this case.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul  6 04:07:12 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 04:07:12 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4b8c60.f732ad81.1d8ca.ae19@mx.google.com>


================

----------------
skatrak wrote:

The gtid creation is an OMPIRBuilder task, we shouldn't be directly building LLVM IR here. The block outside of the loop and the loop itself should be part of the `createTargetInReduction` function, and possibly pass a lambda to it to allow mapping the map block arguments to the generated LLVM IR values. Just as we do it for other constructs using `bodyCB`.

I can see you've done this for other `in_reduction` cases, but I'd strongly suggest to revisit that design decision there as well (in a follow-up patch) and refactor that shared logic within the OMPIRBuilder, as it's currently breaking a previously established separation of responsibilities for no clear gain.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul  6 04:09:33 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 04:09:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b8ced.2020f6c2.1fdf61.7183@mx.google.com>


https://github.com/HagaiLevHacohen updated https://github.com/llvm/llvm-project/pull/207701

>From f83e31c0d0c939bbd784507a45591474676bcbc6 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 11:27:19 +0300
Subject: [PATCH 1/2] Improving MinUI and MaxUI implementation in valuebounds

---
 .../Arith/IR/ValueBoundsOpInterfaceImpl.cpp   | 34 ++++++++++++-------
 .../Arith/value-bounds-op-interface-impl.mlir | 26 ++++++++++----
 2 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..074f7efffc81b 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,24 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +388,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace
diff --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }

>From 3637788c5a8dcc58ccbbcd58481fbf2fe2931d2e Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 14:09:10 +0300
Subject: [PATCH 2/2] adding non negative bound for MinUI

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index 074f7efffc81b..d2d55b8b1562f 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -363,7 +363,8 @@ struct MinUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
     if (!lhsNonNegative && !rhsNonNegative)
       return;
-
+    
+    cstr.bound(value) >= 0;
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());
       cstr.bound(value) <= lhs;


From llvmlistbot at llvm.org  Mon Jul  6 04:10:32 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 04:10:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b8d28.13be6df8.3d8759.c12d@mx.google.com>


https://github.com/HagaiLevHacohen updated https://github.com/llvm/llvm-project/pull/207701

>From f83e31c0d0c939bbd784507a45591474676bcbc6 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 11:27:19 +0300
Subject: [PATCH 1/3] Improving MinUI and MaxUI implementation in valuebounds

---
 .../Arith/IR/ValueBoundsOpInterfaceImpl.cpp   | 34 ++++++++++++-------
 .../Arith/value-bounds-op-interface-impl.mlir | 26 ++++++++++----
 2 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..074f7efffc81b 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,24 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +388,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace
diff --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }

>From 3637788c5a8dcc58ccbbcd58481fbf2fe2931d2e Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 14:09:10 +0300
Subject: [PATCH 2/3] adding non negative bound for MinUI

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index 074f7efffc81b..d2d55b8b1562f 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -363,7 +363,8 @@ struct MinUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
     if (!lhsNonNegative && !rhsNonNegative)
       return;
-
+    
+    cstr.bound(value) >= 0;
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());
       cstr.bound(value) <= lhs;

>From 0a18237baeee085e681c8be534d0cc90a71306e4 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 14:10:09 +0300
Subject: [PATCH 3/3] lint

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index d2d55b8b1562f..6c6fc6fa128a1 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -363,7 +363,7 @@ struct MinUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
     if (!lhsNonNegative && !rhsNonNegative)
       return;
-    
+
     cstr.bound(value) >= 0;
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());


From llvmlistbot at llvm.org  Mon Jul  6 04:51:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 04:51:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4b96da.e9d484b9.11ad58.15b1@mx.google.com>


https://github.com/ES3Q edited https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Mon Jul  6 04:56:17 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 04:56:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4b97e1.c56fb102.11841f.ee8d@mx.google.com>


https://github.com/matthias-springer commented:

Based on the changed test cases, this just renames the error message, but does not add any extra information. How does that help?

https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 05:14:45 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 05:14:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b9c35.ed2b82d4.26b4e.5f9a@mx.google.com>


================
@@ -354,20 +354,25 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
     cstr.bound(value) >= 0;
----------------
matthias-springer wrote:

Can you add more details about the `>= 0` bound? Something along the lines of:
```
A negative signed integer bit pattern reinterpreted as an 
unsigned integer is greater than SIGNED_INT_MAX. If
one of the operands is signed non-negative, it is smaller than
or equal to SIGNED_INT_MAX in unsigned interpretation,
and `minui` will choose that operand over a negative signed
integer operand.
```

(Please double-check if what I wrote is correct.)


https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 05:14:45 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 05:14:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b9c35.3c65ca91.91f2d.7d34@mx.google.com>


================
@@ -384,13 +389,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
----------------
matthias-springer wrote:

Can this check be dropped entirely?

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 05:14:45 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 05:14:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b9c35.ebfc8a3f.25513e.5b88@mx.google.com>


================
@@ -354,20 +354,25 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
     cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
----------------
matthias-springer wrote:

This one here is much simpler than the `>= 0` bound: if the signed interpretation and the unsigned interpretation match, then `minsi` and `minui` have the same bound. I would still add a comment about this.

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 05:15:17 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 05:15:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4b9c55.970836ae.3da2af.3452@mx.google.com>


matthias-springer wrote:

Overall, I think this makes sense. But let's try to improve the documentation a bit. I had to think about the `>= 0` quite some time...

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 05:21:17 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 05:21:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4b9dbd.2730d81b.ba467.bdf1@mx.google.com>


linuxlonelyeagle wrote:

> Based on the changed test cases, this just renames the error message, but does not add any extra information. How does that help?

This is the original logic. As you can see, when the statements in both if conditions evaluate to false, an error is thrown via the final diag. However, for the nested if statement, this specific error message is inaccurate and makes debugging quite difficult. I actually ran into this exact issue today, which I also mentioned in the PR description.
```
  // Try to materialize the conversion.
  if (const TypeConverter *converter = info.getConverter()) {
    if (!newMaterialization.empty()) {
       return success();
     }
  }

  InFlightDiagnostic diag = op->emitError()
                            << "failed to legalize unresolved materialization "
                               "from ("
                            << inputOperands.getTypes() << ") to ("
                            << op.getResultTypes()
                            << ") that remained live after conversion";
  diag.attachNote(op->getUsers().begin()->getLoc())
      << "see existing live user here: " << *op->getUsers().begin();
  return failure();
}
```


https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 05:31:15 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 05:31:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4ba013.9e93dfde.1d132f.5a70@mx.google.com>


matthias-springer wrote:

There are two cases:
1. No type converter present
2. Type converter present, but unable to build a materialization

Do you want to distinguish between these two? In either case, the error message looks accurate to me. But we could add "`(no type converter specified)` to the error message for Case 1. `miss source materialization function` is not entirely accurate because there could be a materialization function that was called but did not match (i.e. produced an "empty" materialization).


https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 05:31:43 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 05:31:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4ba02f.e28fa2c5.e3eca.6f27@mx.google.com>


================

----------------
lhutton1 wrote:

Yes that's correct, I'll update the comment to state this so that we don't forget, thanks!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Mon Jul  6 05:31:43 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 05:31:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4ba02f.e0660744.238c05.4272@mx.google.com>


================

----------------
lhutton1 wrote:

Done, thanks!

https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Mon Jul  6 05:36:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 05:36:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
Message-ID: <llvm.org/llvm/llvm-project/pull/207719@github.com>


https://github.com/skatrak created https://github.com/llvm/llvm-project/pull/207719

This attribute is only intended for an OpenMP target device but a recent patch started causing it to appear on the host. This causes the OpenMPOpt pass to treat the LLVM IR module as an OpenMP offloading target, potentially introducing invalid modifications.

Intended to fix #207423.

>From 9dcf1794d48fc95be8eb37141a3b1417cf5938a2 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Mon, 6 Jul 2026 13:31:56 +0100
Subject: [PATCH] [MLIR][OpenMP] Prevent openmp-device attribute being added to
 the host

This attribute is only intended for an OpenMP target device but a recent
patch started causing it to appear on the host. This causes the OpenMPOpt pass
to treat the LLVM IR module as an OpenMP offloading target, potentially
introducing invalid modifications.

Intended to fix #207423.
---
 .../LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp     | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index a287644207724..27493bfedadeb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7796,13 +7796,13 @@ convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
-  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
-                              attribute.getOpenmpDeviceVersion());
-
   // The flags below are only intended to be emitted for GPU offload targets.
   if (!offloadMod.getIsGPU())
     return success();
 
+  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
+                              attribute.getOpenmpDeviceVersion());
+
   if (attribute.getNoGpuLib())
     return success();
 


From llvmlistbot at llvm.org  Mon Jul  6 05:36:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 05:36:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4ba168.a5ccebaf.109fb6.47f4@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Sergio Afonso (skatrak)

<details>
<summary>Changes</summary>

This attribute is only intended for an OpenMP target device but a recent patch started causing it to appear on the host. This causes the OpenMPOpt pass to treat the LLVM IR module as an OpenMP offloading target, potentially introducing invalid modifications.

Intended to fix #<!-- -->207423.

---
Full diff: https://github.com/llvm/llvm-project/pull/207719.diff


1 Files Affected:

- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+3-3) 


``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index a287644207724..27493bfedadeb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7796,13 +7796,13 @@ convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
-  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
-                              attribute.getOpenmpDeviceVersion());
-
   // The flags below are only intended to be emitted for GPU offload targets.
   if (!offloadMod.getIsGPU())
     return success();
 
+  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
+                              attribute.getOpenmpDeviceVersion());
+
   if (attribute.getNoGpuLib())
     return success();
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 05:37:44 2026
From: llvmlistbot at llvm.org (Corentin Jabot)
Date: Mon, 06 Jul 2026 05:37:44 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [compiler-rt] [flang]
 [libc] [libcxx] [llvm] [mlir] [polly] [NFC] Fix "is is" typos (PR #206288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206288@github.com>
Message-ID: <6a4ba198.566b9172.11e4e5.64fe@mx.google.com>


https://github.com/cor3ntin approved this pull request.

Clamg changes LGTM

https://github.com/llvm/llvm-project/pull/206288

From llvmlistbot at llvm.org  Mon Jul  6 05:38:12 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 05:38:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4ba1b4.821cd8d4.14bedc.9659@mx.google.com>


https://github.com/linuxlonelyeagle updated https://github.com/llvm/llvm-project/pull/207689

>From e39e0169e95af201ba7de7646ed75d9eb72f2036 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 09:36:26 +0000
Subject: [PATCH 1/2] add miss diag for dialect convertion.

---
 .../Transforms/Utils/DialectConversion.cpp    | 11 ++++++++++
 .../Arith/emulate-wide-int-unsupported.mlir   |  4 ++--
 .../test-legalize-type-conversion.mlir        | 20 +++++++++----------
 3 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..a27fcc888f1eb 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3383,6 +3383,17 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
       rewriter.replaceOp(op, newMaterialization);
       return success();
     }
+    StringRef direction =
+        info.getMaterializationKind() == MaterializationKind::Target ? "target"
+                                                                     : "source";
+    InFlightDiagnostic diag = op.emitError()
+                              << "miss " << direction
+                              << " materialization function from ("
+                              << inputOperands.getTypes() << ") to ("
+                              << op.getResultTypes() << ")";
+    diag.attachNote(op->getUsers().begin()->getLoc())
+        << "require this materialization is here";
+    return failure();
   }
 
   InFlightDiagnostic diag = op->emitError()
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 44f09b920ed4f..6be6f4f04e5d5 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,9 +45,9 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{failed to legalize unresolved materialization from ('vector<2x2xi32>') to ('vector<2xi64>') that remained live after conversion}}
+  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
-  // expected-note at +1 {{see existing live user here}}
+  // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
   return
 }
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 1453045c3bfbc..5dcfb34ceb39d 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,9 +2,9 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{failed to legalize unresolved materialization from () to ('i16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from () to ('i16')}}
   %arg0: i16) {
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
 }
 
@@ -21,18 +21,18 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -50,9 +50,9 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -194,9 +194,9 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
-  // expected-note at below {{see existing live user here}}
+  // expected-note at below {{require this materialization is here}}
   "foo.return"(%a, %b) : (f16, f16) -> ()
 }

>From 5e992d7711ce1d6290ea94b9c38d3e226c4cf7b1 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 12:37:49 +0000
Subject: [PATCH 2/2] fix python test.

---
 mlir/test/python/rewrite.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/python/rewrite.py b/mlir/test/python/rewrite.py
index 35d88833e69e0..8b667810b9953 100644
--- a/mlir/test/python/rewrite.py
+++ b/mlir/test/python/rewrite.py
@@ -316,7 +316,7 @@ def convert_muli(op, adaptor, type_converter, rewriter):
             apply_partial_conversion(module, target, frozen)
         except MLIRError as e:
             # CHECK: caught exception: partial conversion failed
-            # CHECK: failed to legalize unresolved materialization
+            # CHECK: miss target materialization function
             print("caught exception:", e)
 
         t1 = converter.convert_type(IntegerType.get_signless(64))


From llvmlistbot at llvm.org  Mon Jul  6 05:43:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 05:43:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4ba2ec.821cd8d4.14bedc.c374@mx.google.com>


https://github.com/ronlieb approved this pull request.

seems sensible to me, lets wait for another person to approve as well

https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 05:43:50 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 05:43:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4ba306.9ff26ae1.1a3e89.7c20@mx.google.com>


https://github.com/skatrak updated https://github.com/llvm/llvm-project/pull/207719

>From 546cde12a46f37504e6aba357d2a26cc7a99f634 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Mon, 6 Jul 2026 13:31:56 +0100
Subject: [PATCH] [MLIR][OpenMP] Prevent openmp-device attribute being added to
 the host

This attribute is only intended for an OpenMP target device but a recent
patch started causing it to appear on the host. This causes the OpenMPOpt pass
to treat the LLVM IR module as an OpenMP offloading target, potentially
introducing invalid modifications.

Intended to fix #207423.
---
 .../LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp      | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index a287644207724..4574b20ce27fd 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7796,8 +7796,9 @@ convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
-  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
-                              attribute.getOpenmpDeviceVersion());
+  if (offloadMod.getIsTargetDevice())
+    ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
+                                attribute.getOpenmpDeviceVersion());
 
   // The flags below are only intended to be emitted for GPU offload targets.
   if (!offloadMod.getIsGPU())


From llvmlistbot at llvm.org  Mon Jul  6 05:54:39 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 05:54:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4ba58f.dd2b4bd3.13e95e.2200@mx.google.com>


https://github.com/skatrak updated https://github.com/llvm/llvm-project/pull/207719

>From 546cde12a46f37504e6aba357d2a26cc7a99f634 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Mon, 6 Jul 2026 13:31:56 +0100
Subject: [PATCH 1/2] [MLIR][OpenMP] Prevent openmp-device attribute being
 added to the host

This attribute is only intended for an OpenMP target device but a recent
patch started causing it to appear on the host. This causes the OpenMPOpt pass
to treat the LLVM IR module as an OpenMP offloading target, potentially
introducing invalid modifications.

Intended to fix #207423.
---
 .../LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp      | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index a287644207724..4574b20ce27fd 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7796,8 +7796,9 @@ convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
-  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
-                              attribute.getOpenmpDeviceVersion());
+  if (offloadMod.getIsTargetDevice())
+    ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
+                                attribute.getOpenmpDeviceVersion());
 
   // The flags below are only intended to be emitted for GPU offload targets.
   if (!offloadMod.getIsGPU())

>From ac3f8400022b913aa642d26ea39a6f6527adff82 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Mon, 6 Jul 2026 13:54:25 +0100
Subject: [PATCH 2/2] Update test

---
 mlir/test/Target/LLVMIR/openmp-llvm.mlir | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 444918fb1a9e3..db38c8da30777 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3556,7 +3556,7 @@ module attributes {omp.flags = #omp.flags<debug_kind = 1, assume_teams_oversubsc
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
-module attributes {omp.flags = #omp.flags<>, omp.is_gpu = true} {}
+module attributes {omp.flags = #omp.flags<>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
@@ -3566,7 +3566,7 @@ module attributes {omp.flags = #omp.flags<>, omp.is_gpu = true} {}
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 51}
-module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_gpu = true} {}
+module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
@@ -3577,13 +3577,13 @@ module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_gp
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 50}
-module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>, omp.is_gpu = true} {}
+module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 51}
 // CHECK-NOT: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
-module attributes {omp.version = #omp.version<version = 51>} {}
+module attributes {omp.version = #omp.version<version = 51>, omp.is_target_device = true} {}
 
 // -----
 // CHECK: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0


From llvmlistbot at llvm.org  Mon Jul  6 05:56:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 05:56:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] ed2b1d7 - [mlir][tosa] Add new block-scaled
 tensor type and support for MXFP CAST (#203583)
Message-ID: <6a4ba609.7d41f3a1.d1cf7.b8ac@mx.google.com>


Author: Luke Hutton
Date: 2026-07-06T13:56:36+01:00
New Revision: ed2b1d7920365a59e2f6c01f732157f1144d0a00

URL: https://github.com/llvm/llvm-project/commit/ed2b1d7920365a59e2f6c01f732157f1144d0a00
DIFF: https://github.com/llvm/llvm-project/commit/ed2b1d7920365a59e2f6c01f732157f1144d0a00.diff

LOG: [mlir][tosa] Add new block-scaled tensor type and support for MXFP CAST (#203583)

This commit adds a new compound-block scaled tensor type and uses this
type to implement support for MXFP in the CAST operation, as per the
following specification changes:
https://github.com/arm/tosa-specification/pull/50,
https://github.com/arm/tosa-specification/pull/53.

The new block-scaled type is closely modelled after the `quant` dialect
type and supports the following parameters:
- block shape - The size and axis of each block. Support for specifying
constant scale values has not been added in this commit, but will be
added in a later one.
- scale type - The type of the scale value associated with each block.
- value type - The type of the data values in each block.

Example syntax for the new block-scaled type:
```
tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
```

As a pre-requisite for supporting MXFP CAST, the commit also adds new
extensions intended to split the existing EXT-MXFP extension into
separate extensions for each block-scaled type. This allows for more
fine-grained control over which block-scaled types are supported by a
given target. See specification change
https://github.com/arm/tosa-specification/pull/26 for details.

Finally, support for casting to/from the new block-scaled type has been
added, aligning with the behaviour specified in the CAST operation
specification.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
    mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
    mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
    mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
    mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
    mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
    mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/canonicalize.mlir
    mlir/test/Dialect/Tosa/invalid.mlir
    mlir/test/Dialect/Tosa/invalid_extension.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-attach-target.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
    mlir/test/Dialect/Tosa/verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 45d56416fdba0..ef644845ed937 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1137,7 +1137,125 @@ extensionComplianceMap = {
        {{{fp8e5m2T, fp16T}, SpecificationVersion::V_1_0},
         {{fp8e5m2T, fp32T}, SpecificationVersion::V_1_0},
         {{fp16T, fp8e5m2T}, SpecificationVersion::V_1_0},
-        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}}}},
+        {{fp32T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.cast_from_block_scaled",
      {{{Extension::bf16, Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, bf16T}, SpecificationVersion::V_1_1_DRAFT},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
index 3c7d870a44879..d527bde997a7d 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
@@ -257,9 +257,16 @@ class Tosa_I32EnumAttr<string name, string description, string mnemonic,
 // DOUBLEROUND  : Adds double rounding support to the RESCALE operator.
 // INEXACTROUND : Adds inexact rounding support to the RESCALE operator.
 // DYNAMIC      : Removes all Compile Time Constant state for CTC inputs.
-// MXFP         : Microscaling formats.
-// MXFP_CONV    : Microscaling format convolution.
+// MXFP         : Microscaling formats. (Deprecated, this extension will be removed in the future.)
+// MXFP_CONV    : Microscaling format convolution. (Deprecated, this extension will be removed in the future.)
 // SHAPE        : Shape calcuation operators.
+// MX_COMMON    : Base for MXFP microscaling formats.
+// MX_FP4E2M1   : Microscaling format FP4E2M1.
+// MX_FP6E2M3   : Microscaling format FP6E2M3.
+// MX_FP6E3M2   : Microscaling format FP6E3M2.
+// MX_FP8E4M3   : Microscaling format FP8E4M3.
+// MX_FP8E5M2   : Microscaling format FP8E5M2.
+// MX_INT8      : Microscaling format INT8.
 //===----------------------------------------------------------------------===//
 
 def Tosa_NONE : I32EnumAttrCase<"none", 0>;
@@ -295,23 +302,35 @@ def Tosa_EXT_MXFP         : I32EnumAttrCase<"mxfp", 12>;
 def Tosa_EXT_INT64        : I32EnumAttrCase<"int64", 13>;
 def Tosa_EXT_MXFP_CONV    : I32EnumAttrCase<"mxfp_conv", 14>;
 def Tosa_EXT_SHAPE        : I32EnumAttrCase<"shape", 15>;
+def Tosa_EXT_MX_COMMON : I32EnumAttrCase<"mx_common", 16>;
+def Tosa_EXT_MX_FP4E2M1 : I32EnumAttrCase<"mx_fp4e2m1", 17>;
+def Tosa_EXT_MX_FP6E2M3 : I32EnumAttrCase<"mx_fp6e2m3", 18>;
+def Tosa_EXT_MX_FP6E3M2 : I32EnumAttrCase<"mx_fp6e3m2", 19>;
+def Tosa_EXT_MX_FP8E4M3 : I32EnumAttrCase<"mx_fp8e4m3", 20>;
+def Tosa_EXT_MX_FP8E5M2 : I32EnumAttrCase<"mx_fp8e5m2", 21>;
+def Tosa_EXT_MX_INT8 : I32EnumAttrCase<"mx_int8", 22>;
 
 def Tosa_ExtensionAttr
-    : Tosa_I32EnumAttr<"Extension", "supported TOSA extensions", "ext", [
-      Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
-      Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
-      Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
-      Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
-      Tosa_EXT_SHAPE,
-    ]> {
+    : Tosa_I32EnumAttr<
+          "Extension", "supported TOSA extensions", "ext",
+          [Tosa_EXT_NONE, Tosa_EXT_INT16, Tosa_EXT_INT4, Tosa_EXT_BF16,
+           Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_FFT, Tosa_EXT_VARIABLE,
+           Tosa_EXT_CONTROLFLOW, Tosa_EXT_DOUBLEROUND, Tosa_EXT_INEXACTROUND,
+           Tosa_EXT_DYNAMIC, Tosa_EXT_MXFP, Tosa_EXT_INT64, Tosa_EXT_MXFP_CONV,
+           Tosa_EXT_SHAPE, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+           Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+           Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]> {
   let extraClassDeclaration = [{
-    static llvm::SmallVector<Extension, 14> getAllValues() {
+    static llvm::SmallVector<Extension, 22> getAllValues() {
       return {
         Extension::int16, Extension::int4, Extension::bf16,
         Extension::fp8e4m3, Extension::fp8e5m2, Extension::fft,
         Extension::variable, Extension::controlflow, Extension::doubleround,
         Extension::inexactround, Extension::dynamic, Extension::mxfp,
-        Extension::int64, Extension::mxfp_conv, Extension::shape
+        Extension::int64, Extension::mxfp_conv, Extension::shape,
+        Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3,
+        Extension::mx_fp6e3m2, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2,
+        Extension::mx_int8
       };
     }
   }];
@@ -490,6 +509,7 @@ def Tosa_RoundingModeAttr
     : Tosa_I32EnumAttr<"RoundingMode", "Supported rounding modes", "rounding_mode",
                     [Tosa_ROUNDING_SINGLE_ROUND, Tosa_ROUNDING_INEXACT_ROUND, Tosa_ROUNDING_DOUBLE_ROUND]>;
 
+// Block_size attr is deprecated and will be removed in the future
 def Tosa_BLOCK_SIZE_1 : I32EnumAttrCase<"BLOCK_SIZE_1", 1>;
 def Tosa_BLOCK_SIZE_32 : I32EnumAttrCase<"BLOCK_SIZE_32", 32>;
 
@@ -503,6 +523,29 @@ def Tosa_BlockSizeAttr
   }];
 }
 
+def Tosa_BLOCK_SHAPE_32 : I32EnumAttrCase<"BLOCK_SHAPE_32", 32>;
+
+def Tosa_BlockShape
+    : Tosa_I32Enum<
+          "BlockShape",
+          "Block shape for the block_scaled formats."
+          "The names follow the convention of BLOCK_SHAPE_M where M "
+          "is the block size of the innermost dimension. Similarly, "
+          "BLOCK_SHAPE_MxN indicates block size of N in the innermost "
+          "dimension and block size of M in the next innermost dimension. "
+          "As of now, only 1 dimension (innermost) is supported for "
+          "block-scaling.",
+          [Tosa_BLOCK_SHAPE_32]>;
+
+def Tosa_BlockShapeAttr
+    : EnumAttr<Tosa_Dialect, Tosa_BlockShape, "block_shape"> {
+  let extraClassDeclaration = [{
+    static uint32_t getBlockShapeValue(BlockShape blockShape) {
+      return static_cast<uint32_t>(blockShape);
+    }
+  }];
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Interfaces.
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 2d96fb6891139..66c3753899124 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,6 +92,8 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
+LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+
 } // namespace tosa
 } // namespace OpTrait
 

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 8b065a4794c43..3518a62800963 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2764,57 +2764,6 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
 
   let description = [{
     Casts a tensor from one data type to another.
-    * This table is showing the supported conversions from the TOSA Specification.
-    * The MLIR dialect here can be used to represent other conversions.
-
-    | Mode                     | Input   | Output  |
-    |--------------------------|---------|---------|
-    | fp16 to fp32             | float16 | float32 |
-    | fp16 to int 16           | float16 | int16   |
-    | fp16 to int 32           | float16 | int32   |
-    | fp16 to int 8            | float16 | int8    |
-    | fp32 to fp16             | float32 | float16 |
-    | fp32 to int 16           | float32 | int16   |
-    | fp32 to int 32           | float32 | int32   |
-    | fp32 to int 8            | float32 | int8    |
-    | int 16 to fp16           | int16   | float16 |
-    | int 16 to fp32           | int16   | float32 |
-    | int 32 to fp16           | int32   | float16 |
-    | int 32 to fp32           | int32   | float32 |
-    | int 8 to fp16            | int8    | float16 |
-    | int 8 to fp32            | int8    | float32 |
-    | bool to int 16           | Boolean | int16   |
-    | bool to int 32           | Boolean | int32   |
-    | bool to int 8            | Boolean | int8    |
-    | int 16 to bool           | int16   | Boolean |
-    | int 16 to int 32         | int16   | int32   |
-    | int 16 to int 8          | int16   | int8    |
-    | int 32 to bool           | int32   | Boolean |
-    | int 32 to int 16         | int32   | int16   |
-    | int 32 to int 8          | int32   | int8    |
-    | int 8 to bool            | int8    | Boolean |
-    | int 8 to int 16          | int8    | int16   |
-    | int 8 to int 32          | int8    | int32   |
-    | bf16 to fp32             | bf16    | float32 |
-    | bf16 to int 16           | bf16    | int16   |
-    | bf16 to int 32           | bf16    | int32   |
-    | bf16 to int 8            | bf16    | int8    |
-    | fp32 to bf16             | float32 | bf16    |
-    | int 16 to bf16           | int16   | bf16    |
-    | int 32 to bf16           | int32   | bf16    |
-    | int 8 to bf16            | int8    | bf16    |
-    | bf16 to fp8e4m3          | bf16    | fp8e4m3 |
-    | fp8e4m3 to bf16          | fp8e4m3 | bf16    |
-    | bf16 to fp8e5m2          | bf16    | fp8e5m2 |
-    | fp8e5m2 to bf16          | fp8e5m2 | bf16    |
-    | fp16 to fp8e4m3          | float16 | fp8e4m3 |
-    | fp32 to fp8e4m3          | float32 | fp8e4m3 |
-    | fp8e4m3 to fp16          | fp8e4m3 | float16 |
-    | fp8e4m3 to fp32          | fp8e4m3 | float32 |
-    | fp16 to fp8e5m2          | float16 | fp8e5m2 |
-    | fp32 to fp8e5m2          | float32 | fp8e5m2 |
-    | fp8e5m2 to fp16          | fp8e5m2 | float16 |
-    | fp8e5m2 to fp32          | fp8e5m2 | float32 |
   }];
 
   let arguments = (ins
@@ -2825,19 +2774,25 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 
   let hasFolder = 1;
   let hasCanonicalizer = 1;
+  let hasVerifier = 1;
 }
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_from_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled", [Pure]> {
   let summary = "Apply scales from a scale tensor to the values in a value tensor";
@@ -2846,6 +2801,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
     Apply the scales from a scale tensor to the values in a value tensor, casting
     the result to the output type. The block dimension must be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
@@ -2869,6 +2826,8 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
 
 //===----------------------------------------------------------------------===//
 // Operator: cast_to_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [Pure]> {
   let summary = "Calculate scale tensor values per block, output to separate scale and data tensors.";
@@ -2878,6 +2837,8 @@ def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [P
     scaled data values from an input tensor. The output tensors are cast to the
     specified scale and value types. The block dimension will be the last dimension
     of the tensor.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 0135a651be481..34ccc675c8243 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -23,10 +23,22 @@ using namespace mlir::tosa;
 // Type Compilance Definition
 //===----------------------------------------------------------------------===//
 
-typedef struct {
+struct TypeInfo {
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(mlir::TypeID()),
+        scaleTypeID(mlir::TypeID()), blockShape(std::nullopt) {}
+
+  TypeInfo(mlir::TypeID typeID, uint32_t bitWidth, mlir::TypeID valueTypeID,
+           mlir::TypeID scaleTypeID, tosa::BlockShape blockShape)
+      : typeID(typeID), bitWidth(bitWidth), valueTypeID(valueTypeID),
+        scaleTypeID(scaleTypeID), blockShape(blockShape) {}
+
   mlir::TypeID typeID;
   uint32_t bitWidth;
-} TypeInfo;
+  mlir::TypeID valueTypeID;
+  mlir::TypeID scaleTypeID;
+  std::optional<tosa::BlockShape> blockShape;
+};
 
 enum CheckCondition {
   invalid,
@@ -70,6 +82,12 @@ class ProfileInfoDepot {
 
 private:
   TypeInfo convertTypeToInfo(Type type) {
+    if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type)) {
+      Type valueTy = blockScaledTy.getValueType();
+      Type scaleTy = blockScaledTy.getScaleType();
+      return {type.getTypeID(), tosa::getBitWidth(valueTy), valueTy.getTypeID(),
+              scaleTy.getTypeID(), blockScaledTy.getBlockShape()};
+    }
     return {type.getTypeID(), tosa::getBitWidth(type)};
   }
 
@@ -128,7 +146,9 @@ class TosaProfileCompliance {
       const SmallVector<ArrayRef<T>> &specDefinedProfileSet);
 
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
-    return a.typeID == b.typeID && a.bitWidth == b.bitWidth;
+    return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
+           a.valueTypeID == b.valueTypeID && a.scaleTypeID == b.scaleTypeID &&
+           a.blockShape == b.blockShape;
   }
 
   // Find the required profiles or extensions from the compliance info according
@@ -145,7 +165,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<32> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index b7d9ed45f3a03..69f836cf1b9d3 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -103,15 +103,43 @@ def Tosa_MXInt8
   }];
 }
 
+def Tosa_MXFPValue
+    : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
+                "micro-scaling format number">;
+def Tosa_MXFPScale
+    : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+
+def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+  let summary = "Block scaled tensor element type.";
+
+  let description = [{
+    This does not specify an implementation type. A tensor of this type is a
+    tensor which has block-scaled quantization applied.
+
+    This compound type is made up of 3 components:
+    `block_shape` - The size and axis of each block. Currently only supports
+      specifying the block size along the innermost dimension.
+    `scale_type` - The type of the scale value associated with each block.
+    `value_type` - The type of the data values in each block.
+
+  }];
+
+  let parameters = (ins
+    EnumParameter<Tosa_BlockShape>:$block_shape,
+    Tosa_MXFPScale:$scale_type,
+    Tosa_MXFPValue:$value_type
+  );
+
+  let assemblyFormat =
+      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
+}
+
 //===----------------------------------------------------------------------===//
 // Multi-category types.
 //===----------------------------------------------------------------------===//
-def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat, Tosa_MXInt8],
-                                "number">;
-
-def Tosa_MXFPNumber : AnyTypeOf<[F8E4M3FN, F8E5M2, F4E2M1FN, F6E2M3FN, F6E3M2FN, Tosa_MXInt8],
-                                "micro-scaling format number">;
-def Tosa_MXFPScaleNumber : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
+def Tosa_AnyNumber : AnyTypeOf<[Tosa_Int, Tosa_QuantizedInt, AnyFloat,
+                                Tosa_MXInt8, Tosa_BlockScaled],
+                               "number">;
 
 //===----------------------------------------------------------------------===//
 // TOSA Tensor Conformance
@@ -129,16 +157,29 @@ def AtLeastRankOne : And<[
   IsRankedTensorTypePred,
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
-class TosaTensorOf<
-    list<Type> allowedTypes, string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes, [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>], summary>;
-
-class TosaRankedTensorOf<
-    list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<allowedTypes, !listconcat([HasNo0Dimensions], preds), summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [], string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes, preds, summary>;
+def IsValidBlockScaledTensorType
+    : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
+            "verifyBlockScaledTensorType($_op, $_self))">;
+
+class TosaTensorOf<list<Type> allowedTypes,
+                   string summary = "tosa-conformant tensor">
+    : TensorOf<allowedTypes,
+               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+                IsValidBlockScaledTensorType],
+               summary>;
+
+class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                         string summary = "tosa-conformant ranked tensor">
+    : RankedTensorOf<
+          allowedTypes,
+          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
+          summary>;
+
+class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
+                           string summary = "tosa-conformant unranked tensor">
+    : UnrankedTensorOf<allowedTypes,
+                       !listconcat([IsValidBlockScaledTensorType], preds),
+                       summary>;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaRankedTensorOf<allowedTypes,
@@ -217,32 +258,28 @@ def Tosa_IndexTensor2D : AnyTypeOf<[
 def Tosa_TensorAtLeast1D : AnyTypeOf<[
   Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
 
-def Tosa_MXFPDataTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [3]>
-]>;
-def Tosa_MXFPScaleTensor3D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [3]>
-]>;
-def Tosa_MXFPDataTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPNumber], [4]>
-]>;
-def Tosa_MXFPScaleTensor4D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaTensorRankOf<[Tosa_MXFPScaleNumber], [4]>
-]>;
-def Tosa_MXFPDataTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
-def Tosa_MXFPScaleTensorAtLeast1D : AnyTypeOf<[
-  TosaUnrankedTensorOf<[Tosa_MXFPScaleNumber]>,
-  TosaRankedTensorOf<[Tosa_MXFPScaleNumber], [AtLeastRankOne]>],
-  "tosa-conformant tensor of at least rank 1", "::mlir::TensorType"
->;
+def Tosa_MXFPDataTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+def Tosa_MXFPScaleTensor3D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+def Tosa_MXFPDataTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+def Tosa_MXFPScaleTensor4D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+def Tosa_MXFPDataTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
+                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
+def Tosa_MXFPScaleTensorAtLeast1D
+    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
+                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
+                "tosa-conformant tensor of at least rank 1",
+                "::mlir::TensorType">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.

diff  --git a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
index dc18fcaa04c8a..56e4901811dcb 100644
--- a/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TargetEnv.cpp
@@ -56,6 +56,13 @@ TosaSpecificationVersion getMinVersion(const Extension &extension) {
   case Extension::int64:
   case Extension::mxfp_conv:
   case Extension::shape:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return TosaSpecificationVersion(1, 1, true);
   case Extension::none:
     return TosaSpecificationVersion(0, 0);
@@ -76,6 +83,13 @@ SmallVector<Profile, 2> getCooperativeProfiles(Extension ext) {
   case Extension::fft:
   case Extension::mxfp:
   case Extension::mxfp_conv:
+  case Extension::mx_common:
+  case Extension::mx_fp4e2m1:
+  case Extension::mx_fp6e2m3:
+  case Extension::mx_fp6e3m2:
+  case Extension::mx_fp8e4m3:
+  case Extension::mx_fp8e5m2:
+  case Extension::mx_int8:
     return {Profile::pro_fp};
   case Extension::variable:
   case Extension::controlflow:

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index b2c6540aa74e2..4c98aca0ee8d7 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1235,12 +1235,50 @@ struct NonNarrowingCastsOptimization : public OpRewritePattern<tosa::CastOp> {
   }
 };
 
+struct CancellingBlockScaledCastsOptimization
+    : public OpRewritePattern<tosa::CastOp> {
+  using OpRewritePattern<tosa::CastOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(tosa::CastOp castOp,
+                                PatternRewriter &rewriter) const override {
+    const Value outerInput = castOp.getInput();
+    auto innerCastOp = outerInput.getDefiningOp<tosa::CastOp>();
+    if (!innerCastOp)
+      return rewriter.notifyMatchFailure(castOp,
+                                         "input must be a cast operation");
+
+    const Value innerInput = innerCastOp.getInput();
+    const auto innerInputTy = llvm::cast<ShapedType>(innerInput.getType());
+    const auto innerOutputTy = llvm::cast<ShapedType>(innerCastOp.getType());
+    const auto outerOutputTy = llvm::cast<ShapedType>(castOp.getType());
+
+    if (!llvm::isa<tosa::BlockScaledType>(innerInputTy.getElementType()))
+      return rewriter.notifyMatchFailure(
+          castOp, "inner cast input must have block scaled element type");
+
+    if (innerInputTy != outerOutputTy)
+      return rewriter.notifyMatchFailure(
+          castOp, "inner input type must match outer output type");
+
+    const Type innerOutputElemType = innerOutputTy.getElementType();
+    const bool isLosslessCast = isa<Float32Type>(innerOutputElemType);
+    if (!isLosslessCast)
+      return rewriter.notifyMatchFailure(
+          castOp, "avoid cancelling casts that should be lossy");
+
+    rewriter.replaceOp(castOp, innerInput);
+
+    return success();
+  }
+};
+
 void CastOp::getCanonicalizationPatterns(RewritePatternSet &results,
                                          MLIRContext *context) {
-  results.add<NonNarrowingCastsOptimization>(context);
+  results.add<NonNarrowingCastsOptimization,
+              CancellingBlockScaledCastsOptimization>(context);
 }
 
-struct CancellingBlockScaledCastsOptimization
+struct CancellingCastToFromBlockScaledOptimization
     : public OpRewritePattern<tosa::CastToBlockScaledOp> {
   using OpRewritePattern<tosa::CastToBlockScaledOp>::OpRewritePattern;
 
@@ -1286,7 +1324,7 @@ struct CancellingBlockScaledCastsOptimization
 
 void CastToBlockScaledOp::getCanonicalizationPatterns(
     RewritePatternSet &results, MLIRContext *context) {
-  results.add<CancellingBlockScaledCastsOptimization>(context);
+  results.add<CancellingCastToFromBlockScaledOptimization>(context);
 }
 
 struct RowGatherToGather : public OpRewritePattern<tosa::RowGatherOp> {

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index c4705c8080069..e25774856f59d 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -628,6 +628,8 @@ Value mlir::tosa::createPadConstTensor(OpBuilder &builder, Location loc,
 }
 
 unsigned mlir::tosa::getBitWidth(Type type) {
+  if (auto blockScaledTy = dyn_cast<tosa::BlockScaledType>(type))
+    return getBitWidth(blockScaledTy.getValueType());
   if (dyn_cast<tosa::mxint8Type>(type))
     return 8;
   return type.getIntOrFloatBitWidth();
@@ -734,6 +736,41 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
   return cast<IntegerType>(attrType).convertFromAttribute(attr, result);
 }
 
+//===----------------------------------------------------------------------===//
+// TOSA block scaling utilities.
+//===----------------------------------------------------------------------===//
+
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  const auto tensorType = llvm::cast<ShapedType>(type);
+  const BlockScaledType elemType =
+      llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
+  if (!elemType)
+    return success();
+
+  if (!tensorType.hasRank())
+    return success();
+
+  if (tensorType.getRank() == 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " does not support block scaling on scalar tensors";
+
+  const int64_t blockedDimension = tensorType.getShape().back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
+
+  const uint32_t blockSize =
+      BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+  if (blockedDimension % blockSize != 0)
+    return op.emitError()
+           << "tensor type " << type
+           << " blocked dimension must be a multiple of block size, got "
+           << blockedDimension << " and block size " << blockSize;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -5187,6 +5224,34 @@ LogicalResult RescaleOp::inferReturnTypeComponents(
   return success();
 }
 
+LogicalResult CastOp::verify() {
+  const ShapedType inputType = llvm::cast<ShapedType>(getInput().getType());
+  const ShapedType outputType = llvm::cast<ShapedType>(getType());
+  const Type inputElementType = inputType.getElementType();
+  const Type outputElementType = outputType.getElementType();
+
+  const bool inputIsBlockScaled = llvm::isa<BlockScaledType>(inputElementType);
+  const bool outputIsBlockScaled =
+      llvm::isa<BlockScaledType>(outputElementType);
+  if (!inputIsBlockScaled && !outputIsBlockScaled)
+    return success();
+
+  if (inputIsBlockScaled && outputIsBlockScaled)
+    return emitOpError()
+           << "requires exactly one of input or output to have block scaled "
+              "element type";
+
+  const Type scalarElementType =
+      inputIsBlockScaled ? outputElementType : inputElementType;
+  if (!llvm::isa<FloatType>(scalarElementType))
+    return emitOpError()
+           << "requires non-block-scaled element type to be floating-point "
+              "when casting to or from block scaled element type, got "
+           << scalarElementType;
+
+  return success();
+}
+
 LogicalResult CastFromBlockScaledOp::inferReturnTypeComponents(
     MLIRContext *context, ::std::optional<Location> location,
     CastFromBlockScaledOp::Adaptor adaptor,

diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 0b4983bea4bc6..348e4ec0ad44c 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Tosa/IR/TosaProfileCompliance.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/raw_ostream.h"
 
 using namespace mlir;
 using namespace mlir::tosa;
@@ -27,12 +28,42 @@ TosaProfileCompliance::TosaProfileCompliance() {
   const TypeInfo fp8e5m2T = {mlir::Float8E5M2Type::getTypeID(), 8};
 
   // micro-scaling formats
+  // Note: these types exist to suppport the deprecated block_scaled operations
+  // and can be removed once those operations are removed.
   const TypeInfo fp6e2m3T = {mlir::Float6E2M3FNType::getTypeID(), 6};
   const TypeInfo fp6e3m2T = {mlir::Float6E3M2FNType::getTypeID(), 6};
   const TypeInfo fp4e2m1T = {mlir::Float4E2M1FNType::getTypeID(), 4};
   const TypeInfo fp8ue8m0T = {mlir::Float8E8M0FNUType::getTypeID(), 8};
   const TypeInfo mxint8T = {mlir::tosa::mxint8Type::getTypeID(), 8};
 
+  // Block scaled formats
+  const TypeID blockScaledID = mlir::tosa::BlockScaledType::getTypeID();
+  const TypeID fp4e2m1ID = mlir::Float4E2M1FNType::getTypeID();
+  const TypeID fp6e2m3ID = mlir::Float6E2M3FNType::getTypeID();
+  const TypeID fp6e3m2ID = mlir::Float6E3M2FNType::getTypeID();
+  const TypeID fp8e4m3ID = mlir::Float8E4M3FNType::getTypeID();
+  const TypeID fp8e5m2ID = mlir::Float8E5M2Type::getTypeID();
+  const TypeID fp8ue8m0ID = mlir::Float8E8M0FNUType::getTypeID();
+  const TypeID mxint8ID = mlir::tosa::mxint8Type::getTypeID();
+
+  const TypeInfo bs32_fp8ue8m0_fp4e2m1T = {blockScaledID, 4, fp4e2m1ID,
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_fp6e2m3T = {blockScaledID, 6, fp6e2m3ID,
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_fp6e3m2T = {blockScaledID, 6, fp6e3m2ID,
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_fp8e4m3T = {blockScaledID, 8, fp8e4m3ID,
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_fp8e5m2T = {blockScaledID, 8, fp8e5m2ID,
+                                           fp8ue8m0ID,
+                                           tosa::BlockShape::BLOCK_SHAPE_32};
+  const TypeInfo bs32_fp8ue8m0_mxint8T = {
+      blockScaledID, 8, mxint8ID, fp8ue8m0ID, tosa::BlockShape::BLOCK_SHAPE_32};
+
 // The profile-based compliance content below is auto-generated by a script
 // in https://github.com/arm/tosa-specification
 #include "mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc"
@@ -687,31 +718,48 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<7>
+llvm::SmallString<32>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
-  if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
-    return {"i" + llvm::utostr(typeInfo.bitWidth)};
-  }
-  if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
-    return {"f16"};
-  } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
-    return {"f32"};
-  } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
-    return {"bf16"};
-  } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
-    return {"fp8e4m3"};
-  } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
-    return {"fp8e5m2"};
-  } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
-    return {"fp6e2m3"};
-  } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
-    return {"fp6e3m2"};
-  } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
-    return {"fp4e2m1"};
-  } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
-    return {"fp8e8m0"};
-  } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
-    return {"mxint8"};
+  const auto stringifyScalarTypeInfo =
+      [](const TypeInfo &typeInfo) -> llvm::SmallString<32> {
+    if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
+      return {"i" + llvm::utostr(typeInfo.bitWidth)};
+    }
+    if (typeInfo.typeID == mlir::Float16Type::getTypeID()) {
+      return {"f16"};
+    } else if (typeInfo.typeID == mlir::Float32Type::getTypeID()) {
+      return {"f32"};
+    } else if (typeInfo.typeID == mlir::BFloat16Type::getTypeID()) {
+      return {"bf16"};
+    } else if (typeInfo.typeID == mlir::Float8E4M3FNType::getTypeID()) {
+      return {"fp8e4m3"};
+    } else if (typeInfo.typeID == mlir::Float8E5M2Type::getTypeID()) {
+      return {"fp8e5m2"};
+    } else if (typeInfo.typeID == mlir::Float6E2M3FNType::getTypeID()) {
+      return {"fp6e2m3"};
+    } else if (typeInfo.typeID == mlir::Float6E3M2FNType::getTypeID()) {
+      return {"fp6e3m2"};
+    } else if (typeInfo.typeID == mlir::Float4E2M1FNType::getTypeID()) {
+      return {"fp4e2m1"};
+    } else if (typeInfo.typeID == mlir::Float8E8M0FNUType::getTypeID()) {
+      return {"fp8e8m0"};
+    } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
+      return {"mxint8"};
+    }
+    llvm_unreachable("unknown type");
+  };
+
+  if (typeInfo.typeID == tosa::BlockScaledType::getTypeID()) {
+    TypeInfo valueInfo = {typeInfo.valueTypeID, typeInfo.bitWidth};
+    TypeInfo scaleInfo = {typeInfo.scaleTypeID, 8};
+    llvm::SmallString<32> result;
+    llvm::raw_svector_ostream os(result);
+    os << "bs"
+       << tosa::BlockShapeAttr::getBlockShapeValue(typeInfo.blockShape.value())
+       << "_" << stringifyScalarTypeInfo(scaleInfo) << "_"
+       << stringifyScalarTypeInfo(valueInfo);
+    return result;
   }
-  llvm_unreachable("unknown type");
+
+  return stringifyScalarTypeInfo(typeInfo);
 }

diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 5d4a2affbd7cd..a3d5d21294b0d 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -1495,7 +1495,7 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
     }
   } else if (isa<tosa::shapeType>(type))
     return true;
-  else if (isa<tosa::mxint8Type>(type))
+  else if (isa<tosa::mxint8Type, tosa::BlockScaledType>(type))
     return true;
   return false;
 }

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index f4f31d3f476e8..c358460bbc662 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -637,10 +637,10 @@ func.func @test_resize(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x64x64x8xf32> {
 }
 
 // -----
-// CHECK-LABEL: cast
-func.func @test_cast1(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
+// CHECK-LABEL: test_cast
+func.func @test_cast(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.cast %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }

diff  --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 992059e06e541..6a3d84e8d133b 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1681,6 +1681,73 @@ func.func @test_canonicalize_non_narrowing_cast_i8_to_f8E4M3FN_unsupported(%arg0
 
 // -----
 
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1
+// CHECK: return %arg0 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f4E2M1(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<15x3x2x256xf32>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf32>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2
+// CHECK: return %arg0 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+func.func @test_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_
diff erent_types_f8E5M2_f6E2M3
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_
diff erent_types_f8E5M2_f6E2M3(%arg0: tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<160xf32>
+  %1 = tosa.cast %0 : (tensor<160xf32>) -> tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<160x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_
diff erent_types_f6E2M3_f6E3M2
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_
diff erent_types_f6E2M3_f6E3M2(%arg0: tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>> {
+  %0 = tosa.cast %arg0 : (tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<32xf32>
+  %1 = tosa.cast %0 : (tensor<32xf32>) -> tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  return %1 : tensor<32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked
+// CHECK: %[[values:.+]] = tosa.cast %arg0
+// CHECK: %[[block_scaled:.+]] = tosa.cast %[[values]]
+// CHECK: return %[[block_scaled]] : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_unranked(%arg0: tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>> {
+  %0 = tosa.cast %arg0 : (tensor<3x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<*xf32>
+  %1 = tosa.cast %0 : (tensor<*xf32>) -> tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %1 : tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3
+// CHECK: tosa.cast
+// CHECK: tosa.cast
+func.func @test_do_not_canonicalize_cast_from_cast_to_block_scaled_type_f8E5M2_f8E4M3(%arg0: tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>> {
+  %0 = tosa.cast %arg0 : (tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<15x3x2x256xf8E4M3FN>
+  %1 = tosa.cast %0 : (tensor<15x3x2x256xf8E4M3FN>) -> tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  return %1 : tensor<15x3x2x256x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+}
+
+// -----
+
 // CHECK-LABEL: @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1
 // CHECK: return %arg0, %arg1 : tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>
 func.func @test_canonicalize_cast_from_cast_to_block_scaled_f4E2M1(%arg0: tensor<15x3x2x256xf4E2M1FN>, %arg1: tensor<15x3x2x8xf8E8M0FNU>) -> (tensor<15x3x2x256xf4E2M1FN>, tensor<15x3x2x8xf8E8M0FNU>) {

diff  --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index 5e8111061cb3a..d0336da15cee8 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2266,3 +2266,35 @@ func.func @test_shape_func_output() -> !tosa.shape<4> {
   %cst = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   return %cst : !tosa.shape<4>
 }
+
+// -----
+
+func.func @test_cast_f32_plain_fp4(%arg0: tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (f32,fp4e2m1)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32xf4E2M1FN>
+  return %0 : tensor<4x32xf4E2M1FN>
+}
+
+// -----
+
+func.func @test_cast_fp4_block_scaled(%arg0: tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp4e2m1,bs32_fp8e8m0_fp4e2m1), did you mean (fp8e4m3,bs32_fp8e8m0_fp4e2m1)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf4E2M1FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_fp6e2m3(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (bs32_fp8e8m0_fp4e2m1,fp6e2m3), did you mean (bs32_fp8e8m0_fp4e2m1,fp8e4m3)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf6E2M3FN>
+  return %0 : tensor<4x32xf6E2M3FN>
+}
+
+// -----
+
+func.func @test_cast_fp6e3m2_block_scaled(%arg0: tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: operation operand/result data types did not align with any profile or extension, got (fp6e3m2,bs32_fp8e8m0_mxint8), did you mean (fp8e4m3,bs32_fp8e8m0_mxint8)? Otherwise, please refer to the 'supported data types' for 'tosa.cast' in the specification.}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf6E3M2FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}

diff  --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index c69ea83f5ef36..5e4dd17b24090 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -303,6 +303,34 @@ func.func @test_cast_f32_bf16(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xbf16
   return %0 : tensor<13x21x3xbf16>
 }
 
+// -----
+func.func @test_cast_f32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+func.func @test_cast_block_scaled_f32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
+  return %0 : tensor<4x32xf32>
+}
+
+// -----
+func.func @test_cast_bf16_block_scaled(%arg0: tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires all of [fp8e4m3, mx_common, mx_int8] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
 // -----
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 262b8aa548506..ec3d0c881d857 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1179,6 +1179,32 @@ func.func @test_cast3(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3x!quant.unifo
   return %0 : tensor<13x21x3x!quant.uniform<i16:f32, 0.078431375324726104:128>>
 }
 
+// -----
+// CHECK-LABEL: test_cast_to_block_scaled
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>, %arg1: tensor<4x32xbf16>, %arg2: tensor<4x32xf8E4M3FN>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %arg1 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %2 = tosa.cast %arg2 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+// CHECK-LABEL: test_cast_from_block_scaled
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xbf16>
+  %2 = tosa.cast %arg2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E5M2>
+  return %0, %1, %2 : tensor<4x32xf32>, tensor<4x32xbf16>, tensor<4x32xf8E5M2>
+}
+
+// -----
+// CHECK-LABEL: test_cast_block_scaled_dynamic
+func.func @test_cast_block_scaled_dynamic(%arg0: tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %arg1: tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> (tensor<?x32xf32>, tensor<4x?xf32>) {
+  %0 = tosa.cast %arg0 : (tensor<?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<?x32xf32>
+  %1 = tosa.cast %arg1 : (tensor<4x?x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x?xf32>
+  return %0, %1 : tensor<?x32xf32>, tensor<4x?xf32>
+}
+
 // -----
 // CHECK-LABEL: rescale
 func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439347028732:127>>) -> tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>> {

diff  --git a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
index a0c59c0c4bb3b..558a599fbbc1e 100644
--- a/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-attach-target.mlir
@@ -1,11 +1,26 @@
-// RUN: mlir-opt %s -split-input-file -tosa-attach-target="profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,dynamic level=none" | FileCheck %s --check-prefix=CHECK-ALL
+// DEFINE: %{core_extensions} = int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable
+// DEFINE: %{rounding_extensions} = controlflow,doubleround,inexactround,dynamic
+// DEFINE: %{mx_float_extensions} = mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2
+// DEFINE: %{mx_int_extensions} = mx_fp8e4m3,mx_fp8e5m2,mx_int8
+// DEFINE: %{all_extensions} = %{core_extensions},%{rounding_extensions},%{mx_float_extensions},%{mx_int_extensions}
+// DEFINE: %{all_target} = specification_version=1.1.draft level=none \
+// DEFINE:    profiles=pro_int,pro_fp extensions=%{all_extensions}
+
+// RUN: mlir-opt %s -split-input-file -tosa-attach-target="%{all_target}" | FileCheck %s --check-prefix=CHECK-ALL
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="level=8k" | FileCheck %s --check-prefix=CHECK-LVL-8K
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target | FileCheck %s --check-prefix=CHECK-DEFAULT
 // RUN: mlir-opt %s -split-input-file -tosa-attach-target="specification_version=1.1.draft" | FileCheck %s --check-prefix=CHECK-VERSION-1P1
 
 // -----
 
-// CHECK-ALL: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = none, profiles = [pro_int, pro_fp], extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft, variable, controlflow, doubleround, inexactround, dynamic]>}
+// CHECK-ALL: module attributes {
+// CHECK-ALL-SAME: tosa.target_env = #tosa.target_env<specification_version = "1.1.draft",
+// CHECK-ALL-SAME: level = none,
+// CHECK-ALL-SAME: profiles = [pro_int, pro_fp],
+// CHECK-ALL-SAME: extensions = [int16, int4, bf16, fp8e4m3, fp8e5m2, fft,
+// CHECK-ALL-SAME: variable, controlflow, doubleround, inexactround, dynamic,
+// CHECK-ALL-SAME: mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2,
+// CHECK-ALL-SAME: mx_fp8e4m3, mx_fp8e5m2, mx_int8]>}
 // CHECK-LVL-8K: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-DEFAULT: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.0", level = "8k", profiles = [], extensions = []>}
 // CHECK-VERSION-1P1: module attributes {tosa.target_env = #tosa.target_env<specification_version = "1.1.draft", level = "8k", profiles = [], extensions = []>}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index cbff768ec1ecb..a5e4d4397f290 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -154,6 +154,14 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+func.func @test_cast_fp32_block_scaled(%arg0: tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  // expected-error at +1 {{'tosa.cast' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_dyanmic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   // expected-error at +1 {{'tosa.argmax' op failed level check: operand shape dimension cannot be dynamic when targeting TOSA specification version 1.0 or below}}
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index f4b5ab5dd11fd..e7a4fe6735457 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
+// RUN: mlir-opt %s -split-input-file -verify-diagnostics -tosa-attach-target="specification_version=1.1.draft profiles=pro_int,pro_fp extensions=int16,int4,bf16,fp8e4m3,fp8e5m2,fft,variable,controlflow,doubleround,inexactround,mxfp,int64,mxfp_conv,shape,mx_common,mx_fp4e2m1,mx_fp6e2m3,mx_fp6e3m2,mx_fp8e4m3,mx_fp8e5m2,mx_int8" -tosa-validate="strict-op-spec-alignment" | FileCheck %s
 
 // -----
 
@@ -389,6 +389,34 @@ func.func @test_cast_i64_bool(%arg0: tensor<13x21x3xi64>) -> tensor<13x21x3xi1>
 
 // -----
 
+// CHECK-LABEL: test_cast_to_block_scaled_types
+func.func @test_cast_to_block_scaled_types(%fp16: tensor<4x32xf16>, %fp32: tensor<4x32xf32>, %bf16: tensor<4x32xbf16>, %fp8e4m3: tensor<4x32xf8E4M3FN>, %fp8e5m2: tensor<4x32xf8E5M2>) -> (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  %2 = tosa.cast %fp32 : (tensor<4x32xf32>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %3 = tosa.cast %fp16 : (tensor<4x32xf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %4 = tosa.cast %bf16 : (tensor<4x32xbf16>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  %5 = tosa.cast %fp8e4m3 : (tensor<4x32xf8E4M3FN>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  %6 = tosa.cast %fp8e5m2 : (tensor<4x32xf8E5M2>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  return %0, %1, %2, %3, %4, %5 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+// CHECK-LABEL: test_cast_from_block_scaled_types
+func.func @test_cast_from_block_scaled_types(%fp4: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %fp6e2m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %fp6e3m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, %fp8e4m3: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %fp8e5m2: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, %mxint8: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> (tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>) {
+  %0 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf32>
+  %1 = tosa.cast %fp6e2m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x32xf32>
+  %2 = tosa.cast %fp6e3m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x32xf32>
+  %3 = tosa.cast %fp8e4m3 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x32xf16>
+  %4 = tosa.cast %fp8e5m2 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>) -> tensor<4x32xbf16>
+  %5 = tosa.cast %mxint8 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<4x32xf8E4M3FN>
+  %6 = tosa.cast %fp4 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xf8E5M2>
+  return %0, %3, %4, %5, %6 : tensor<4x32xf32>, tensor<4x32xf16>, tensor<4x32xbf16>, tensor<4x32xf8E4M3FN>, tensor<4x32xf8E5M2>
+}
+
+// -----
+
 // CHECK-LABEL: test_dynamic_dims
 func.func @test_dynamic_dims(%arg0: tensor<?x8x16xi8>) -> tensor<?x16xi32> {
   %0 = tosa.argmax %arg0 { axis = 1 : i32 } : (tensor<?x8x16xi8>) -> tensor<?x16xi32>

diff  --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 0eb679ca7ced3..20c0dfd2a094c 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1650,6 +1650,48 @@ func.func @test_cast_to_block_scaled_block_size_mismatch(%arg0: tensor<4x32xf32>
 
 // -----
 
+func.func @test_cast_i8_block_scaled(%arg0: tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i8'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32xi8>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+func.func @test_cast_block_scaled_i32(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32> {
+  // expected-error at +1 {{'tosa.cast' op requires non-block-scaled element type to be floating-point when casting to or from block scaled element type, got 'i32'}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32xi32>
+  return %0 : tensor<4x32xi32>
+}
+
+// -----
+
+func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{'tosa.cast' op requires exactly one of input or output to have block scaled element type}}
+  %0 = tosa.cast %arg0 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
+  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
+  return %0 : tensor<1x16x31xf32>
+}
+
+// -----
+
+func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
+  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
+  return %0 : tensor<f32>
+}
+
+// -----
+
 func.func @test_clamp_quantized(%arg0:tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) {
     // expected-error at +1 {{'tosa.clamp' op min/max attributes types are incompatible with input/output element types.}}
     %0 = tosa.clamp %arg0 {max_val = 127 : i8, min_val = -128 : i8} : (tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>) -> tensor<?x112x112x32x!quant.uniform<u8:f32, 0.023529412224888802:-128>>


        

From llvmlistbot at llvm.org  Mon Jul  6 05:56:45 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 05:56:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add new block-scaled tensor type
 and support for MXFP CAST (PR #203583)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203583@github.com>
Message-ID: <6a4ba60d.a87eb5dc.1d432d.db68@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/203583

From llvmlistbot at llvm.org  Mon Jul  6 06:03:45 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 06:03:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4ba7b1.bf38ae28.358429.bc89@mx.google.com>


https://github.com/HagaiLevHacohen updated https://github.com/llvm/llvm-project/pull/207701

>From f83e31c0d0c939bbd784507a45591474676bcbc6 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 11:27:19 +0300
Subject: [PATCH 1/5] Improving MinUI and MaxUI implementation in valuebounds

---
 .../Arith/IR/ValueBoundsOpInterfaceImpl.cpp   | 34 ++++++++++++-------
 .../Arith/value-bounds-op-interface-impl.mlir | 26 ++++++++++----
 2 files changed, 40 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..074f7efffc81b 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,24 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +388,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace
diff --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }

>From 3637788c5a8dcc58ccbbcd58481fbf2fe2931d2e Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 14:09:10 +0300
Subject: [PATCH 2/5] adding non negative bound for MinUI

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index 074f7efffc81b..d2d55b8b1562f 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -363,7 +363,8 @@ struct MinUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
     if (!lhsNonNegative && !rhsNonNegative)
       return;
-
+    
+    cstr.bound(value) >= 0;
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());
       cstr.bound(value) <= lhs;

>From 0a18237baeee085e681c8be534d0cc90a71306e4 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 14:10:09 +0300
Subject: [PATCH 3/5] lint

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index d2d55b8b1562f..6c6fc6fa128a1 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -363,7 +363,7 @@ struct MinUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
     if (!lhsNonNegative && !rhsNonNegative)
       return;
-    
+
     cstr.bound(value) >= 0;
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());

>From 0ccd18bcb88be2e27caa65ec38be934e670555aa Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 16:01:19 +0300
Subject: [PATCH 4/5] added documentation

---
 .../Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp    | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index 6c6fc6fa128a1..87fb2bdc619ff 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -364,7 +364,15 @@ struct MinUIOpInterface
     if (!lhsNonNegative && !rhsNonNegative)
       return;
 
+    // A negative signed integer bit pattern reinterpreted as an 
+    // unsigned integer is greater than SIGNED_INT_MAX. If
+    // one of the operands is signed non-negative, it is smaller than
+    // or equal to SIGNED_INT_MAX in unsigned interpretation,
+    // and `minui` will choose that operand over a negative signed
+    // integer operand.
     cstr.bound(value) >= 0;
+    // If an operand is provably non-negative, its signed and unsigned value interpretations
+    // agree, so `minsi` and `minui` impose the same upper bound: `result <= operand`.
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());
       cstr.bound(value) <= lhs;
@@ -389,8 +397,6 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative && !rhsNonNegative)
-      return;
 
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(maxOp.getLhs());

>From 8ce0e0c8b6a82c1d2ab198265a4bd9df58cd6036 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Mon, 6 Jul 2026 16:02:43 +0300
Subject: [PATCH 5/5] lint

---
 mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index 87fb2bdc619ff..61e84e3cdae0f 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -364,15 +364,16 @@ struct MinUIOpInterface
     if (!lhsNonNegative && !rhsNonNegative)
       return;
 
-    // A negative signed integer bit pattern reinterpreted as an 
+    // A negative signed integer bit pattern reinterpreted as an
     // unsigned integer is greater than SIGNED_INT_MAX. If
     // one of the operands is signed non-negative, it is smaller than
     // or equal to SIGNED_INT_MAX in unsigned interpretation,
     // and `minui` will choose that operand over a negative signed
     // integer operand.
     cstr.bound(value) >= 0;
-    // If an operand is provably non-negative, its signed and unsigned value interpretations
-    // agree, so `minsi` and `minui` impose the same upper bound: `result <= operand`.
+    // If an operand is provably non-negative, its signed and unsigned value
+    // interpretations agree, so `minsi` and `minui` impose the same upper
+    // bound: `result <= operand`.
     if (lhsNonNegative) {
       AffineExpr lhs = cstr.getExpr(minOp.getLhs());
       cstr.bound(value) <= lhs;


From llvmlistbot at llvm.org  Mon Jul  6 06:06:16 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 06:06:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4ba848.c56fb102.11841f.8e5a@mx.google.com>


================
@@ -354,20 +354,25 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
     cstr.bound(value) >= 0;
----------------
HagaiLevHacohen wrote:

What you wrote is correct, and it's exactly how I thought about it.
I added this comment.

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:06:48 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 06:06:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4ba868.8e75e0cb.2e0812.2ef0@mx.google.com>


================
@@ -384,13 +389,17 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
----------------
HagaiLevHacohen wrote:

yes, I dropped it.

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:07:19 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Mon, 06 Jul 2026 06:07:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4ba887.331483e7.5bc1e.fde0@mx.google.com>


================
@@ -354,20 +354,25 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
     cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
----------------
HagaiLevHacohen wrote:

I added a comment.

https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:07:56 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Mon, 06 Jul 2026 06:07:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a4ba8ac.e9d484b9.11ad58.d487@mx.google.com>


https://github.com/akroviakov approved this pull request.


https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Mon Jul  6 06:12:05 2026
From: llvmlistbot at llvm.org (Kareem Ergawy)
Date: Mon, 06 Jul 2026 06:12:05 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add opt-in
 query caching; use it in flang LICM (PR #207030)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207030@github.com>
Message-ID: <6a4ba9a5.92ea0e2a.21f97.b4f1@mx.google.com>


https://github.com/ergawy commented:

Thanks Slava, I have a question about the design. Can we make the cache itself RAII-based and not just its scope?

The main advantages are:
1. We decouple the cache from AA's API and do not have to extend MLIR's AA for it.                                   
2. We already had to add a RAII scoping component: QueryCacheScope, so I think moving the resource management to the cache itself away from the analysis makes sense.
3. Maybe in the future we can implement different caches (for different query results) all implementing a shared interface: `AAResultCache` and can augment AA with customized caches based on a particular client's needs.
 
Does that make sense in your opinion?

https://github.com/llvm/llvm-project/pull/207030

From llvmlistbot at llvm.org  Mon Jul  6 06:14:55 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 06:14:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4baa4f.2020f6c2.1fdf61.c854@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:15:13 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 06:15:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4baa61.331483e7.5bc1e.27a8@mx.google.com>


https://github.com/matthias-springer auto_merge_enabled https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:15:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 06:15:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] c827c90 - Improving MinUI and MaxUI
 implementation in valuebounds (#207701)
Message-ID: <6a4baa6a.da8bafd3.2e1688.8802@mx.google.com>


Author: Hagai Lev Hacohen
Date: 2026-07-06T13:15:17Z
New Revision: c827c9011f81ef8b74143dd0ef40db5950aaec81

URL: https://github.com/llvm/llvm-project/commit/c827c9011f81ef8b74143dd0ef40db5950aaec81
DIFF: https://github.com/llvm/llvm-project/commit/c827c9011f81ef8b74143dd0ef40db5950aaec81.diff

LOG: Improving MinUI and MaxUI implementation in valuebounds (#207701)

Previously, the valuebounds implementation for MinUI and MaxUI was too
conservative.
We can already make correct assumptions even when we know only one
operand is provably not negative.

Added: 
    

Modified: 
    mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
    mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
index bc15d3b22f1b3..61e84e3cdae0f 100644
--- a/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -354,20 +354,34 @@ struct MinUIOpInterface
     assert(value == minOp.getResult() && "invalid value");
 
     // ValueBoundsConstraintSet models values as signed integers (e.g. an i8
-    // 0xff is treated as -1, not 255).So, we can only derive bounds for minui
-    // if both operands are provably non-negative.
+    // 0xff is treated as -1, not 255). For an unsigned minimum it is enough
+    // that a single operand is provably non-negative: minui(x, y) is in
+    // [0, y] whenever y >= 0 (and symmetrically for x).
     bool lhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(minOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
+    if (!lhsNonNegative && !rhsNonNegative)
       return;
 
+    // A negative signed integer bit pattern reinterpreted as an
+    // unsigned integer is greater than SIGNED_INT_MAX. If
+    // one of the operands is signed non-negative, it is smaller than
+    // or equal to SIGNED_INT_MAX in unsigned interpretation,
+    // and `minui` will choose that operand over a negative signed
+    // integer operand.
     cstr.bound(value) >= 0;
-    AffineExpr lhs = cstr.getExpr(minOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(minOp.getRhs());
-    cstr.bound(value) <= lhs;
-    cstr.bound(value) <= rhs;
+    // If an operand is provably non-negative, its signed and unsigned value
+    // interpretations agree, so `minsi` and `minui` impose the same upper
+    // bound: `result <= operand`.
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(minOp.getLhs());
+      cstr.bound(value) <= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(minOp.getRhs());
+      cstr.bound(value) <= rhs;
+    }
   }
 };
 
@@ -384,13 +398,15 @@ struct MaxUIOpInterface
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getLhs(), cstr);
     bool rhsNonNegative =
         ValueBoundsConstraintSet::isProvablyNonNegative(maxOp.getRhs(), cstr);
-    if (!lhsNonNegative || !rhsNonNegative)
-      return;
 
-    AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
-    AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
-    cstr.bound(value) >= lhs;
-    cstr.bound(value) >= rhs;
+    if (lhsNonNegative) {
+      AffineExpr lhs = cstr.getExpr(maxOp.getLhs());
+      cstr.bound(value) >= lhs;
+    }
+    if (rhsNonNegative) {
+      AffineExpr rhs = cstr.getExpr(maxOp.getRhs());
+      cstr.bound(value) >= rhs;
+    }
   }
 };
 } // namespace

diff  --git a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
index 41f7f18baefec..ceb8256003b1e 100644
--- a/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
+++ b/mlir/test/Dialect/Arith/value-bounds-op-interface-impl.mlir
@@ -486,32 +486,38 @@ func.func @arith_maxui() -> index {
 
 // -----
 
-func.func @arith_maxui_unknown_sign(%a: index) -> index {
+// CHECK-LABEL: func @arith_maxui_unknown_operand(
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
+func.func @arith_maxui_unknown_operand(%a: index) -> index {
   %c4 = arith.constant 4 : index
   %0 = arith.maxui %a, %c4 : index
-  // expected-error @below{{could not reify bound}}
-  %1 = "test.reify_bound"(%0) {type = "LB"} : (index) -> (index)
+  %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_wraparound(
+//       CHECK:   %[[ub:.*]] = arith.constant 11 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.minui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_wraparound(
+//       CHECK:   %[[lb:.*]] = arith.constant 10 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_wraparound() -> index {
   %c255 = arith.constant 0xFF : i8
   %c10 = arith.constant 10 : i8
   %0 = arith.maxui %c255, %c10 : i8
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", allow_integer_type} : (i8) -> (index)
   return %1 : index
 }
@@ -547,12 +553,15 @@ func.func @arith_minui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_minui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[ub:.*]] = arith.constant 5 : index
+//       CHECK:   return %[[ub]]
 func.func @arith_minui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.minui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "UB", constant} : (index) -> (index)
   return %1 : index
 }
@@ -574,12 +583,15 @@ func.func @arith_maxui_nonneg_symbolic(%a: index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @arith_maxui_negative_symbolic(
+//  CHECK-SAME:     %[[a:.*]]: index
+//       CHECK:   %[[lb:.*]] = arith.constant 4 : index
+//       CHECK:   return %[[lb]]
 func.func @arith_maxui_negative_symbolic(%a: index) -> index {
   %cm1 = arith.constant -1 : index
   %c4 = arith.constant 4 : index
   %neg = arith.minsi %a, %cm1 : index
   %0 = arith.maxui %neg, %c4 : index
-  // expected-error @below{{could not reify bound}}
   %1 = "test.reify_bound"(%0) {type = "LB", constant} : (index) -> (index)
   return %1 : index
 }


        

From llvmlistbot at llvm.org  Mon Jul  6 06:15:27 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 06 Jul 2026 06:15:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Improving MinUI and MaxUI implementation in
	valuebounds (PR #207701)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207701@github.com>
Message-ID: <6a4baa6f.f06f4994.168928.35bc@mx.google.com>


https://github.com/matthias-springer closed https://github.com/llvm/llvm-project/pull/207701

From llvmlistbot at llvm.org  Mon Jul  6 06:20:30 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 06:20:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4bab9e.f06f4994.168928.fab4@mx.google.com>


linuxlonelyeagle wrote:

> There are two cases:
> 
> 1. No type converter present
> 2. Type converter present, but unable to build a materialization
> 
> Do you want to distinguish between these two? In either case, the error message looks accurate to me. But we could add "`(no type converter specified)` to the error message for Case 1. `miss source materialization function` is not entirely accurate because there could be a materialization function that was called but did not match (i.e. produced an "empty" materialization).

Do you want to distinguish between these two? In either case, the error message looks accurate to me

Yes.

But we could add "(no type converter specified) to the error message for Case 1. miss source materialization function is not entirely accurate because there could be a materialization function that was called but did not match (i.e. produced an "empty" materialization).

May be we can say `mismatch source materialization function`, it is accurate and we can add `(no type converter specified)` to case1. This looks good.


https://github.com/llvm/llvm-project/pull/207689

From llvmlistbot at llvm.org  Mon Jul  6 06:30:01 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 06:30:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4badd9.a87eb5dc.1d432d.9e11@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:31:13 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 06:31:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4bae21.fdc477d6.1a5bda.382a@mx.google.com>


skatrak wrote:

Tested locally to address the reported STREAM runtime segfault.

https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:31:37 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 06 Jul 2026 06:31:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4bae39.d1496011.3d49cb.764d@mx.google.com>


https://github.com/skatrak closed https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:35:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 06:35:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3fbe826 - [MLIR][OpenMP] Prevent
 openmp-device attribute being added to the host (#207719)
Message-ID: <6a4baf10.492e23e4.e71ca.7851@mx.google.com>


Author: Sergio Afonso
Date: 2026-07-06T14:31:29+01:00
New Revision: 3fbe826e3707540c9b7f942c573fbcb2d51b6bcc

URL: https://github.com/llvm/llvm-project/commit/3fbe826e3707540c9b7f942c573fbcb2d51b6bcc
DIFF: https://github.com/llvm/llvm-project/commit/3fbe826e3707540c9b7f942c573fbcb2d51b6bcc.diff

LOG: [MLIR][OpenMP] Prevent openmp-device attribute being added to the host (#207719)

This attribute is only intended for an OpenMP target device but a recent
patch started causing it to appear on the host. This causes the
OpenMPOpt pass to treat the LLVM IR module as an OpenMP offloading
target, potentially introducing invalid modifications.

Fixes #207423.

Added: 
    

Modified: 
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/openmp-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index a287644207724..4574b20ce27fd 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -7796,8 +7796,9 @@ convertFlagsAttr(Operation *op, mlir::omp::FlagsAttr attribute,
 
   llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
 
-  ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
-                              attribute.getOpenmpDeviceVersion());
+  if (offloadMod.getIsTargetDevice())
+    ompBuilder->M.addModuleFlag(llvm::Module::Max, "openmp-device",
+                                attribute.getOpenmpDeviceVersion());
 
   // The flags below are only intended to be emitted for GPU offload targets.
   if (!offloadMod.getIsGPU())

diff  --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 444918fb1a9e3..db38c8da30777 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -3556,7 +3556,7 @@ module attributes {omp.flags = #omp.flags<debug_kind = 1, assume_teams_oversubsc
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
-module attributes {omp.flags = #omp.flags<>, omp.is_gpu = true} {}
+module attributes {omp.flags = #omp.flags<>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
@@ -3566,7 +3566,7 @@ module attributes {omp.flags = #omp.flags<>, omp.is_gpu = true} {}
 // CHECK: @__omp_rtl_assume_no_thread_state = weak_odr hidden constant i32 0
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 51}
-module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_gpu = true} {}
+module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
@@ -3577,13 +3577,13 @@ module attributes {omp.flags = #omp.flags<openmp_device_version = 51>, omp.is_gp
 // CHECK: @__omp_rtl_assume_no_nested_parallelism = weak_odr hidden constant i32 0
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 50}
-module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>, omp.is_gpu = true} {}
+module attributes {omp.version = #omp.version<version = 50>, omp.flags = #omp.flags<>, omp.is_target_device = true, omp.is_gpu = true} {}
 
 // -----
 
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 51}
 // CHECK-NOT: [[META0:![0-9]+]] = !{i32 7, !"openmp-device", i32 50}
-module attributes {omp.version = #omp.version<version = 51>} {}
+module attributes {omp.version = #omp.version<version = 51>, omp.is_target_device = true} {}
 
 // -----
 // CHECK: @__omp_rtl_debug_kind = weak_odr hidden constant i32 0


        

From llvmlistbot at llvm.org  Mon Jul  6 06:36:08 2026
From: llvmlistbot at llvm.org (Paul Osmialowski)
Date: Mon, 06 Jul 2026 06:36:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4baf48.a87eb5dc.1d432d.7e1a@mx.google.com>


https://github.com/pawosm-arm approved this pull request.

Works for me. Thanks!

https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:38:09 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Mon, 06 Jul 2026 06:38:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4bafc1.af6ba3ed.2c2f30.5250@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11842

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8641/8689] Creating library symlink lib/libFIROpenMPSupport.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtim
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8643/8689] Linking CXX shared library lib/libFortranUtils.so.23.0git
[8644/8689] Linking CXX shared library lib/libFIROpenACCSupport.so.23.0git
[8645/8689] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8646/8689] Linking CXX executable bin/mlir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8641/8689] Creating library symlink lib/libFIROpenMPSupport.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtim
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8643/8689] Linking CXX shared library lib/libFortranUtils.so.23.0git
[8644/8689] Linking CXX shared library lib/libFIROpenACCSupport.so.23.0git
[8645/8689] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8646/8689] Linking CXX executable bin/mlir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=76.115496

```

</details>

https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:39:59 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 06:39:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4bb02f.bb839515.23e6d3.76e7@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/205506

>From e453094ec307ef208b703b51f2f7f877b903bdfe Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 1 Jun 2026 20:38:58 +0100
Subject: [PATCH] [mlir][tosa] Add constant block scaled support

This commit adds support for block scaled tensors.

In particular, the block scaled type has been extended with the
`DenseElementTypeInterface` to allow block scaled data values to
be specified in dense element attributes.

The block scaled type has also been extended to allow optional scale
values to be specified by the type. For now, scale values are not
expected to be propagated beyond their use in the attribute input of
a constant operation. In the future, we may want to propagate these
values to allow certain optimizations.

The `tosa.const` operation has also been updated in the validation pass.

Change-Id: I879f9b8742b784fda976a879813e68b928f6b77e
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  |  20 ++-
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  10 +-
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     |  20 ++-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          | 149 ++++++++++++++++--
 mlir/test/Dialect/Tosa/availability.mlir      |   2 +-
 mlir/test/Dialect/Tosa/ops.mlir               |  72 +++++++++
 .../tosa-validation-version-1p0-invalid.mlir  |  18 +++
 .../tosa-validation-version-1p1-valid.mlir    |  13 ++
 mlir/test/Dialect/Tosa/verifier.mlir          |  69 ++++++++
 9 files changed, 352 insertions(+), 21 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..291097a7f7514 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1305,7 +1305,25 @@ extensionComplianceMap = {
         {{fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
-        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.identity",
      {{{Extension::int4}, {{{i4T, i4T}, SpecificationVersion::V_1_0}}},
       {{Extension::int16}, {{{i48T, i48T}, SpecificationVersion::V_1_0}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..5e5760e33b5f6 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2970,9 +2970,13 @@ def Tosa_ConstOp : Tosa_Op<"const", [ConstantLike, Pure,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3,
+                  Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasFolder = 1;
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..e6dace5a8d3df 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -109,7 +109,12 @@ def Tosa_MXFPValue
 def Tosa_MXFPScale
     : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
 
-def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+def Tosa_BlockScaled
+    : Tosa_Type<"BlockScaled", "block_scaled",
+                [DeclareTypeInterfaceMethods<
+                    DenseElementTypeInterface, ["getDenseElementBitSize",
+                                                "convertToAttribute",
+                                                "convertFromAttribute"]>]> {
   let summary = "Block scaled tensor element type.";
 
   let description = [{
@@ -121,17 +126,26 @@ def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
       specifying the block size along the innermost dimension.
     `scale_type` - The type of the scale value associated with each block.
     `value_type` - The type of the data values in each block.
+    `scale_values` - Optional array of per-block scale values. If not provided,
+      the type is assumed to be dynamically quantized at runtime.
 
   }];
 
   let parameters = (ins
     EnumParameter<Tosa_BlockShape>:$block_shape,
     Tosa_MXFPScale:$scale_type,
-    Tosa_MXFPValue:$value_type
+    Tosa_MXFPValue:$value_type,
+    OptionalArrayRefParameter<"Attribute">:$scale_values
   );
 
   let assemblyFormat =
-      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
+      "`<` $block_shape```:```$scale_type```:```$value_type "
+      "(`,` `{` custom<ScaleValues>($scale_values, ref($scale_type))^ `}`)? "
+      "`>`";
+
+  let extraClassDeclaration = [{
+    bool hasScaleValues() const { return !getScaleValues().empty(); }
+  }];
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..ece1b0ffa29ba 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/Tosa/Utils/ShapeUtils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/VerificationUtils.h"
+#include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/Matchers.h"
@@ -740,14 +741,55 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+static ParseResult parseScaleValues(AsmParser &parser,
+                                    SmallVector<Attribute> &scaleValues,
+                                    Type scaleType) {
+  const auto parseScaleValue = [&]() -> ParseResult {
+    const SMLoc loc = parser.getCurrentLocation();
+
+    double floatValue;
+    if (parser.parseFloat(floatValue))
+      return failure();
+
+    if (floatValue < 0.0)
+      return parser.emitError(loc, "scale value must be non-negative, got ")
+             << floatValue;
+
+    Type attrType = scaleType;
+    if (succeeded(parser.parseOptionalColon()) && parser.parseType(attrType))
+      return failure();
+
+    if (attrType != scaleType)
+      return parser.emitError(loc, "parsed attribute type ")
+             << attrType << " does not match expected scale type " << scaleType;
+
+    scaleValues.push_back(FloatAttr::get(attrType, floatValue));
+    return success();
+  };
+
+  return parser.parseCommaSeparatedList(parseScaleValue);
+}
+
+static void printScaleValues(AsmPrinter &printer,
+                             ArrayRef<Attribute> scaleValues, Type) {
+  llvm::interleaveComma(scaleValues, printer, [&](Attribute scaleValue) {
+    printer.printAttributeWithoutType(scaleValue);
+  });
+}
+
+static LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type,
+                                                 bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
+  if (!allowScaleValues && elemType.hasScaleValues())
+    return op.emitError()
+           << "tensor type " << type
+           << " does not support scale values for this operation";
+
   if (!tensorType.hasRank())
     return success();
 
@@ -756,12 +798,24 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
            << "tensor type " << type
            << " does not support block scaling on scalar tensors";
 
-  const int64_t blockedDimension = tensorType.getShape().back();
-  if (ShapedType::isDynamic(blockedDimension))
-    return success();
-
+  const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+
+  if (allowScaleValues && elemType.hasScaleValues() &&
+      tensorType.hasStaticShape()) {
+    const size_t numBlocks =
+        llvm::accumulate(tensorShape, int64_t(1), std::multiplies<int64_t>());
+    if (elemType.getScaleValues().size() != numBlocks / blockSize)
+      return op.emitError()
+             << "tensor type " << type << " has " << numBlocks / blockSize
+             << " blocks but got " << elemType.getScaleValues().size()
+             << " scale values";
+  }
+
+  const int64_t blockedDimension = tensorShape.back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
   if (blockedDimension % blockSize != 0)
     return op.emitError()
            << "tensor type " << type
@@ -771,6 +825,43 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   return success();
 }
 
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  return ::verifyBlockScaledTensorType(op, type, /*allowScaleValues=*/false);
+}
+
+size_t mlir::tosa::BlockScaledType::getDenseElementBitSize() const {
+  const Type valueType = getValueType();
+  if (isa<tosa::mxint8Type>(valueType))
+    return 8;
+  return valueType.getIntOrFloatBitWidth();
+}
+
+Attribute
+mlir::tosa::BlockScaledType::convertToAttribute(ArrayRef<char> rawData) const {
+  assert(rawData.size() == 1 && "expected 1 byte for block_scaled element");
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertToAttribute(rawData);
+  if (!isa<FloatType>(valueType))
+    return {};
+  return mlir::detail::convertFloatTypeToAttribute(valueType, rawData);
+}
+
+LogicalResult mlir::tosa::BlockScaledType::convertFromAttribute(
+    Attribute attr, SmallVectorImpl<char> &result) const {
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertFromAttribute(attr, result);
+
+  const auto floatAttr = dyn_cast<FloatAttr>(attr);
+  if (!floatAttr || floatAttr.getType() != valueType)
+    return failure();
+  const APFloat value = floatAttr.getValue();
+  return mlir::detail::convertFloatTypeFromAttribute(
+      valueType, FloatAttr::get(valueType, value), result);
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -857,7 +948,7 @@ static LogicalResult verifyConvOp(T op) {
 }
 
 LogicalResult tosa::ConstOp::verify() {
-
+  Operation &op = *getOperation();
   auto attrType = llvm::dyn_cast<TensorType>(getValuesAttr().getType());
   auto outputType = llvm::dyn_cast<TensorType>(getOutput().getType());
 
@@ -866,17 +957,49 @@ LogicalResult tosa::ConstOp::verify() {
     return failure();
   }
 
-  if (auto result = llvm::dyn_cast<mlir::quant::QuantizedType>(
-          outputType.getElementType())) {
-    if (getStorageElementTypeFromQuantized(result) == attrType.getElementType())
+  const Type attrElemType = attrType.getElementType();
+  const Type resultElemType = outputType.getElementType();
+
+  if (auto result =
+          llvm::dyn_cast<mlir::quant::QuantizedType>(resultElemType)) {
+    if (getStorageElementTypeFromQuantized(result) == attrElemType)
       return success();
   }
 
-  if (attrType.getElementType() != outputType.getElementType()) {
-    emitOpError("expected same attr/result element types");
-    return failure();
+  if (auto attrBlockScaledType =
+          llvm::dyn_cast<mlir::tosa::BlockScaledType>(attrElemType)) {
+    if (failed(verifyBlockScaledTensorType(op, attrType, true)) ||
+        failed(verifyBlockScaledTensorType(op, outputType, false)))
+      return failure();
+
+    if (!attrBlockScaledType.hasScaleValues())
+      return op.emitOpError(
+          "attribute block scaled type must have scale values");
+
+    const BlockScaledType resultBlockScaledType =
+        llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
+    if (!resultBlockScaledType)
+      return op.emitOpError(
+          "result type must be block scaled type if attribute is block "
+          "scaled type");
+
+    if (attrBlockScaledType.getValueType() !=
+            resultBlockScaledType.getValueType() ||
+        attrBlockScaledType.getScaleType() !=
+            resultBlockScaledType.getScaleType() ||
+        attrBlockScaledType.getBlockShape() !=
+            resultBlockScaledType.getBlockShape())
+      return op.emitOpError(
+                 "expected block scaled element type to be compatible "
+                 "between attr and result, got ")
+             << attrBlockScaledType << " vs. " << resultBlockScaledType;
+
+    return success();
   }
 
+  if (attrElemType != resultElemType)
+    return emitOpError("expected same attr/result element types");
+
   return success();
 }
 
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..bc014fb9b928e 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -660,7 +660,7 @@ func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439
 // CHECK-LABEL: test_const
 func.func @test_const(%arg0 : index) -> tensor<4xi32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64] ]
+  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
     %0 = "tosa.const"() {values = dense<[3, 0, 1, 2]> : tensor<4xi32>} : () -> tensor<4xi32>
     return %0 : tensor<4xi32>
 }
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..4e71d512def2d 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1938,3 +1938,75 @@ func.func @test_assert_equal_shape() {
   tosa.assert_equal_shape %0, %1 {allow_broadcast = true} : (!tosa.shape<2>, !tosa.shape<2>) -> ()
   return
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const
+func.func @test_block_scaled_const() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 1.0}>> : [[0.0 : f8E4M3FN, 1.0 : f8E4M3FN, 0.001953125 : f8E4M3FN, 0.0078125 : f8E4M3FN, 
+                                                                                                                          2.0 : f8E4M3FN, 2.25 : f8E4M3FN, 2.5 : f8E4M3FN, 2.75 : f8E4M3FN, 
+                                                                                                                          15.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,  
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN], 
+                                                                                                                         [0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN]]>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat
+func.func @test_block_scaled_const_splat() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0, 1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat_mxint8
+func.func @test_block_scaled_const_splat_mxint8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 1.0}>> : 0 : i8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values
+func.func @test_block_scaled_const_scale_values() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_explicit_type
+func.func @test_block_scaled_const_scale_values_explicit_type() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : f8E8M0FNU, 2.0 : f8E8M0FNU}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_wide_inner_dim
+func.func @test_block_scaled_const_scale_values_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0, 4.0, 8.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_cast_scale_values_no_propagate
+func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x32xf32> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %1 = tosa.cast %0 : (tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<2x32xf32>
+  return %1 : tensor<2x32xf32>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..bea262a11a391 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -193,6 +193,24 @@ func.func @test_const_fp6e3m2(%arg0 : index) -> tensor<4xf6E3M2FN> {
 
 // -----
 
+func.func @test_const_block_scaled_types() -> (tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %0 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %1 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN, {1.0}>> : 0.0 : f6E2M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %2 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN, {1.0}>> : 0.0 : f6E3M2FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %3 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %4 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2, {1.0}>> : 0.0 : f8E5M2>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %5 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0}>> : 0 : i8>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2, %3, %4, %5 : tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 func.func @test_cast_from_block_scaled(%arg0: tensor<4x32xf8E5M2>, %arg1: tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32> {
   // expected-error at +1 {{'tosa.cast_from_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires any of [mxfp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<4x32xf8E5M2>, tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..d290ff71f052d 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -152,6 +152,19 @@ func.func @test_const_mxint8() -> tensor<2x!tosa.mxint8> {
 
 // -----
 
+// CHECK-LABEL: test_const_block_scaled_types
+func.func @test_const_block_scaled_types() -> (tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN, {1.0}>> : 0.0 : f6E2M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  %2 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN, {1.0}>> : 0.0 : f6E3M2FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %3 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %4 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2, {1.0}>> : 0.0 : f8E5M2>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  %5 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0}>> : 0 : i8>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2, %3, %4, %5 : tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_matmul_t_block_scaled_mxint8
 func.func @test_matmul_t_block_scaled_mxint8(%arg0: tensor<4x8x32x!tosa.mxint8>, %arg1: tensor<4x8x1xf8E8M0FNU>, %arg2: tensor<4x16x32x!tosa.mxint8>, %arg3: tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32> {
   %0 = tosa.matmul_t_block_scaled %arg0, %arg1, %arg2, %arg3 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x8x32x!tosa.mxint8>, tensor<4x8x1xf8E8M0FNU>, tensor<4x16x32x!tosa.mxint8>, tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..50e75d9a702dd 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -2270,3 +2270,72 @@ func.func @test_const_mxint8_int64(%arg0 : index) -> tensor<2x!tosa.mxint8> {
   %0 = "tosa.const"() {values = dense<tensor<2x!tosa.mxint8> : [127, 245]>} : () -> tensor<2x!tosa.mxint8>
   return %0 : tensor<2x!tosa.mxint8>
 }
+
+// -----
+
+func.func @test_block_scaled_const_splat_ui8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{incompatible attribute for element type}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> : 0 : ui8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_splat_fp64() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{incompatible attribute for element type}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> : 0.0>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_integer_scale_value() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +2 {{unexpected decimal integer literal for a floating point value}}
+  // expected-note at +1 {{add a trailing dot to make the literal a float}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1 : i32, 2 : i32}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_negative_scale_value() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{scale value must be non-negative, got -1.000000e+00}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : f8E8M0FNU, -1.0 : f8E8M0FNU}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_scale_value_non_float_explicit_type() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{parsed attribute type 'i32' does not match expected scale type 'f8E8M0FNU'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : i32}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+!mxint8 = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>
+!mxint8_scale = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0, 4.0}>
+
+func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8> {
+  // expected-error at +1 {{tensor type 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00, 4.000000e+00}>>' has 2 blocks but got 3 scale values}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!mxint8_scale> : 0 : i8>}> : () -> tensor<2x32x!mxint8>
+  return %0 : tensor<2x32x!mxint8>
+}
+
+// -----
+
+func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  // expected-error at +1 {{tensor type 'tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00}>>' has 4 blocks but got 2 scale values}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0}>> : 0 : i8>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_cast_scale_values_propagate() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> {
+  // expected-error at +2 {{tensor type 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>' does not support scale values for this operation}}
+  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
+}


From llvmlistbot at llvm.org  Mon Jul  6 06:40:17 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 06:40:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4bb041.7d41f3a1.d1cf7.d760@mx.google.com>


https://github.com/lhutton1 ready_for_review https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Mon Jul  6 06:40:41 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 06:40:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4bb059.ac469dc7.ed218.6521@mx.google.com>


lhutton1 wrote:

cc @sahas3 @IanTaylerLessa-arm @Yuvaraj-Venkatesh @Tai78641 @udaya-ranga 

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Mon Jul  6 06:41:27 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Mon, 06 Jul 2026 06:41:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Prevent openmp-device
 attribute being added to the host (PR #207719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207719@github.com>
Message-ID: <6a4bb087.d5797644.146403.9880@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11842

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8641/8689] Creating library symlink lib/libFIROpenMPSupport.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtim
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8643/8689] Linking CXX shared library lib/libFortranUtils.so.23.0git
[8644/8689] Linking CXX shared library lib/libFIROpenACCSupport.so.23.0git
[8645/8689] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8646/8689] Linking CXX executable bin/mlir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:148 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8641/8689] Creating library symlink lib/libFIROpenMPSupport.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtim
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8643/8689] Linking CXX shared library lib/libFortranUtils.so.23.0git
[8644/8689] Linking CXX shared library lib/libFIROpenACCSupport.so.23.0git
[8645/8689] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8646/8689] Linking CXX executable bin/mlir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=76.115496

```

</details>

https://github.com/llvm/llvm-project/pull/207719

From llvmlistbot at llvm.org  Mon Jul  6 06:49:58 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 06:49:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix TosaNarrowTypes conversion
 order to prevent rollback crash (PR #207372)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207372@github.com>
Message-ID: <6a4bb286.30c2cbf3.391c08.5a2b@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/207372

From llvmlistbot at llvm.org  Mon Jul  6 06:50:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 06:50:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 6dd93e3 - [mlir][tosa] Fix TosaNarrowTypes
 conversion order to prevent rollback crash (#207372)
Message-ID: <6a4bb297.f375f8af.1713f5.0203@mx.google.com>


Author: Hocky Yudhiono
Date: 2026-07-06T14:49:51+01:00
New Revision: 6dd93e338442f070f55691e5374aac966287ee3c

URL: https://github.com/llvm/llvm-project/commit/6dd93e338442f070f55691e5374aac966287ee3c
DIFF: https://github.com/llvm/llvm-project/commit/6dd93e338442f070f55691e5374aac966287ee3c.diff

LOG: [mlir][tosa] Fix TosaNarrowTypes conversion order to prevent rollback crash (#207372)

Fix #206952 by doing `convertRegionTypes` before moving and unlinking
the blocks when handling `tosa.cond_if` ops. This will cause a crash
when doing rollback.

Added: 
    

Modified: 
    mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
    mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
index 983472762e791..d441f90c2a074 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaNarrowTypes.cpp
@@ -426,10 +426,10 @@ LogicalResult convertGenericOp(Operation *op, ValueRange operands,
   }
 
   for (Region &region : op->getRegions()) {
+    if (failed(rewriter.convertRegionTypes(&region, *typeConverter)))
+      return failure();
     Region *newRegion = state.addRegion();
     rewriter.inlineRegionBefore(region, *newRegion, newRegion->begin());
-    if (failed(rewriter.convertRegionTypes(newRegion, *typeConverter)))
-      return failure();
   }
 
   Operation *newOp = rewriter.create(state);

diff  --git a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
index 9349bd1c1cf7b..f51c5b714378e 100644
--- a/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-narrow-i64-to-i32.mlir
@@ -75,6 +75,27 @@ func.func @test_regions(%arg0: tensor<1x2xi32>, %arg1: tensor<1xi32>, %arg2: ten
 
 // -----
 
+// CHECK-LABEL: test_cond_if_i64_yield_with_call
+module {
+  func.func @m0() -> () {
+    return
+  }
+  func.func @test_cond_if_i64_yield_with_call(%arg0: tensor<1xi1>, %arg1: tensor<4xi64>, %arg2: tensor<4xi64>) -> () {
+    %0 = tosa.cond_if %arg0 (%arg3 = %arg1) : tensor<1xi1> (tensor<4xi64>) -> tensor<4xi64> {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    } else {
+    ^bb0(%arg3: tensor<4xi64>):
+      tosa.yield %arg3 : tensor<4xi64>
+    }
+    // expected-error @+1 {{failed to legalize operation 'func.call'}}
+    call @m0() : () -> ()
+    return
+  }
+}
+
+// -----
+
 // CHECK-LABEL: test_concat
 func.func @test_concat(%arg0: tensor<13x21x3xi64>, %arg1: tensor<13x21x3xi64>) -> tensor<26x21x3xi64> {
   // COMMON: tosa.concat %{{.*}}, %{{.*}} {axis = 0 : i32} : (tensor<13x21x3xi32>, tensor<13x21x3xi32>) -> tensor<26x21x3xi32>


        

From llvmlistbot at llvm.org  Mon Jul  6 06:58:26 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 06:58:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a4bb482.492e23e4.e71ca.f9d7@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Mon Jul  6 06:58:26 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 06:58:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a4bb482.ff037943.3bf0e2.0db0@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you for the fix! Do you have commit access to land this?

https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Mon Jul  6 06:58:26 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 06:58:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a4bb482.2020f6c2.1fdf61.b39a@mx.google.com>


================
@@ -477,3 +477,52 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+// A contiguous `tensor.extract` from a source whose rank is *smaller* than the
+// iteration space (here: a 1-D source inside a 2-D loop nest). The contiguous
+// `vector.transfer_read` only reads the trailing (contiguous) dim - the leading
+// dim is broadcast via the permutation map - so its mask must be rank-reduced
+// (`vector<4xi1>`) rather than the full iteration-space mask (`vector<1x4xi1>`).
+// Regression test: previously the full iteration mask was attached, producing
+// an invalid `vector.transfer_read` ("inferred mask type ... don't match").
+
+func.func @masked_contiguous_extract_rank_reducing_mask(
+    %src: tensor<16xf32>,
+    %output : tensor<1x3xf32>,
+    %idx: index) -> tensor<1x3xf32> {
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%output : tensor<1x3xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%2, %idx)
+    %extracted = tensor.extract %src[%3] : tensor<16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x3xf32>
+  return %1 : tensor<1x3xf32>
----------------
banach-space wrote:

>  I checked. It looks like having a leading dimension > 1 doesn't actually use this code branch, it winds up being a gather instead. 

That's all I needed, thank you!

https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Mon Jul  6 07:01:29 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Mon, 06 Jul 2026 07:01:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add missing materialization function
 diag for dialect convertion (PR #207689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207689@github.com>
Message-ID: <6a4bb539.f8b79514.3b441.cbbc@mx.google.com>


https://github.com/linuxlonelyeagle updated https://github.com/llvm/llvm-project/pull/207689

>From e39e0169e95af201ba7de7646ed75d9eb72f2036 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 09:36:26 +0000
Subject: [PATCH 1/3] add miss diag for dialect convertion.

---
 .../Transforms/Utils/DialectConversion.cpp    | 11 ++++++++++
 .../Arith/emulate-wide-int-unsupported.mlir   |  4 ++--
 .../test-legalize-type-conversion.mlir        | 20 +++++++++----------
 3 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..a27fcc888f1eb 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3383,6 +3383,17 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
       rewriter.replaceOp(op, newMaterialization);
       return success();
     }
+    StringRef direction =
+        info.getMaterializationKind() == MaterializationKind::Target ? "target"
+                                                                     : "source";
+    InFlightDiagnostic diag = op.emitError()
+                              << "miss " << direction
+                              << " materialization function from ("
+                              << inputOperands.getTypes() << ") to ("
+                              << op.getResultTypes() << ")";
+    diag.attachNote(op->getUsers().begin()->getLoc())
+        << "require this materialization is here";
+    return failure();
   }
 
   InFlightDiagnostic diag = op->emitError()
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 44f09b920ed4f..6be6f4f04e5d5 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,9 +45,9 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{failed to legalize unresolved materialization from ('vector<2x2xi32>') to ('vector<2xi64>') that remained live after conversion}}
+  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
-  // expected-note at +1 {{see existing live user here}}
+  // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
   return
 }
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 1453045c3bfbc..5dcfb34ceb39d 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,9 +2,9 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{failed to legalize unresolved materialization from () to ('i16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from () to ('i16')}}
   %arg0: i16) {
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
 }
 
@@ -21,18 +21,18 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -50,9 +50,9 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
-  // expected-note at below{{see existing live user here}}
+  // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
 }
 
@@ -194,9 +194,9 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f16') that remained live after conversion}}
+  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
-  // expected-note at below {{see existing live user here}}
+  // expected-note at below {{require this materialization is here}}
   "foo.return"(%a, %b) : (f16, f16) -> ()
 }

>From 5e992d7711ce1d6290ea94b9c38d3e226c4cf7b1 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 12:37:49 +0000
Subject: [PATCH 2/3] fix python test.

---
 mlir/test/python/rewrite.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/python/rewrite.py b/mlir/test/python/rewrite.py
index 35d88833e69e0..8b667810b9953 100644
--- a/mlir/test/python/rewrite.py
+++ b/mlir/test/python/rewrite.py
@@ -316,7 +316,7 @@ def convert_muli(op, adaptor, type_converter, rewriter):
             apply_partial_conversion(module, target, frozen)
         except MLIRError as e:
             # CHECK: caught exception: partial conversion failed
-            # CHECK: failed to legalize unresolved materialization
+            # CHECK: miss target materialization function
             print("caught exception:", e)
 
         t1 = converter.convert_type(IntegerType.get_signless(64))

>From 96db30de8eaa78bbf4ac6f348822faeaebf88d20 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 14:01:10 +0000
Subject: [PATCH 3/3] update code.

---
 mlir/lib/Transforms/Utils/DialectConversion.cpp    | 14 +++++++-------
 .../Arith/emulate-wide-int-unsupported.mlir        |  2 +-
 .../Transforms/test-legalize-type-conversion.mlir  | 12 ++++++------
 mlir/test/python/rewrite.py                        |  2 +-
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index a27fcc888f1eb..856b27c29d913 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3387,7 +3387,7 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
         info.getMaterializationKind() == MaterializationKind::Target ? "target"
                                                                      : "source";
     InFlightDiagnostic diag = op.emitError()
-                              << "miss " << direction
+                              << "mismatch " << direction
                               << " materialization function from ("
                               << inputOperands.getTypes() << ") to ("
                               << op.getResultTypes() << ")";
@@ -3396,12 +3396,12 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
     return failure();
   }
 
-  InFlightDiagnostic diag = op->emitError()
-                            << "failed to legalize unresolved materialization "
-                               "from ("
-                            << inputOperands.getTypes() << ") to ("
-                            << op.getResultTypes()
-                            << ") that remained live after conversion";
+  InFlightDiagnostic diag =
+      op->emitError()
+      << "failed to legalize unresolved materialization "
+         "from ("
+      << inputOperands.getTypes() << ") to (" << op.getResultTypes()
+      << ") that remained live after conversion (no type converter specified)";
   diag.attachNote(op->getUsers().begin()->getLoc())
       << "see existing live user here: " << *op->getUsers().begin();
   return failure();
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
index 6be6f4f04e5d5..ad8393ec7de89 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int-unsupported.mlir
@@ -45,7 +45,7 @@ func.func @unsupported_argument_type(%arg0: vector<4xi128>) -> vector<4xi64> {
 
 // Ensure this case not crash
 func.func @unsupported_vector(%arg0: vector<2xi1>) {
-  // expected-error at +1 {{miss source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
+  // expected-error at +1 {{mismatch source materialization function from ('vector<2x2xi32>') to ('vector<2xi64>')}}
   %cst_0 = arith.constant dense<0> : vector<2xi64>
   // expected-note at +1 {{require this materialization is here}}
   %0 = vector.mask %arg0 { vector.multi_reduction <xor>, %cst_0, %cst_0 [] : vector<2xi64> to vector<2xi64> } : vector<2xi1> -> vector<2xi64>
diff --git a/mlir/test/Transforms/test-legalize-type-conversion.mlir b/mlir/test/Transforms/test-legalize-type-conversion.mlir
index 5dcfb34ceb39d..c8fbbbfa7fdd9 100644
--- a/mlir/test/Transforms/test-legalize-type-conversion.mlir
+++ b/mlir/test/Transforms/test-legalize-type-conversion.mlir
@@ -2,7 +2,7 @@
 
 
 func.func @test_invalid_arg_materialization(
-  // expected-error at below {{miss source materialization function from () to ('i16')}}
+  // expected-error at below {{mismatch source materialization function from () to ('i16')}}
   %arg0: i16) {
   // expected-note at below{{require this materialization is here}}
   "foo.return"(%arg0) : (i16) -> ()
@@ -21,7 +21,7 @@ func.func @test_valid_arg_materialization(%arg0: i64) {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
+  // expected-error at below {{mismatch source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
   // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
@@ -30,7 +30,7 @@ func.func @test_invalid_result_materialization() {
 // -----
 
 func.func @test_invalid_result_materialization() {
-  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
+  // expected-error at below {{mismatch source materialization function from ('f64') to ('f16')}}
   %result = "test.type_producer"() : () -> f16
   // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
@@ -50,7 +50,7 @@ func.func @test_transitive_use_materialization() {
 // -----
 
 func.func @test_transitive_use_invalid_materialization() {
-  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
+  // expected-error at below {{mismatch source materialization function from ('f64') to ('f16')}}
   %result = "test.another_type_producer"() : () -> f16
   // expected-note at below{{require this materialization is here}}
   "foo.return"(%result) : (f16) -> ()
@@ -102,7 +102,7 @@ func.func @test_block_argument_not_converted() {
 // Make sure argument type changes aren't implicitly forwarded.
 func.func @test_signature_conversion_no_converter() {
   "test.signature_conversion_no_converter"() ({
-  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f32') that remained live after conversion}}
+  // expected-error at below {{failed to legalize unresolved materialization from ('f64') to ('f32') that remained live after conversion (no type converter specified)}}
   ^bb0(%arg0: f32):
     // expected-note at below{{see existing live user here}}
     "test.type_consumer"(%arg0) : (f32) -> ()
@@ -194,7 +194,7 @@ gpu.module @cuda_events {
 // hashes and could differ with LLVM_ENABLE_REVERSE_ITERATION.
 
 func.func @test_deterministic_materialization_order() {
-  // expected-error at below {{miss source materialization function from ('f64') to ('f16')}}
+  // expected-error at below {{mismatch source materialization function from ('f64') to ('f16')}}
   %a = "test.type_producer"() : () -> f16
   %b = "test.type_producer"() : () -> f16
   // expected-note at below {{require this materialization is here}}
diff --git a/mlir/test/python/rewrite.py b/mlir/test/python/rewrite.py
index 8b667810b9953..d0419ae4aad6e 100644
--- a/mlir/test/python/rewrite.py
+++ b/mlir/test/python/rewrite.py
@@ -316,7 +316,7 @@ def convert_muli(op, adaptor, type_converter, rewriter):
             apply_partial_conversion(module, target, frozen)
         except MLIRError as e:
             # CHECK: caught exception: partial conversion failed
-            # CHECK: miss target materialization function
+            # CHECK: mismatch target materialization function
             print("caught exception:", e)
 
         t1 = converter.convert_type(IntegerType.get_signless(64))


From llvmlistbot at llvm.org  Mon Jul  6 07:13:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 07:13:07 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a4bb7f3.8b87e04f.199b71.7a4a@mx.google.com>


https://github.com/ShashwathiNavada updated https://github.com/llvm/llvm-project/pull/206642

>From d6cdc9ec1ffc9ad1b70b2c8cbfb0430548c3b724 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Thu, 25 Jun 2026 01:40:11 -0500
Subject: [PATCH 1/2] Implement support for partial in unroll construct

---
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    |  9 ++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 24 +++++++--
 flang/test/Lower/OpenMP/unroll-partial01.f90  | 54 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 35 ++++++++++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 50 +++++++++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 27 ++++++++++
 7 files changed, 195 insertions(+), 5 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/unroll-partial01.f90

diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index e2a2c8575ffc8..9031f6ddf0744 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -795,6 +795,15 @@ bool ClauseProcessor::processOrdered(
   return false;
 }
 
+bool ClauseProcessor::processPartial(std::optional<int64_t> &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Partial>()) {
+    if (clause->v.has_value())
+      result = evaluate::ToInt64(*clause->v);
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processPriority(
     lower::StatementContext &stmtCtx,
     mlir::omp::PriorityClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1fc221b721ebf..e92db70ca707b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -111,6 +111,7 @@ class ClauseProcessor {
                          mlir::omp::NumThreadsClauseOps &result) const;
   bool processOrder(mlir::omp::OrderClauseOps &result) const;
   bool processOrdered(mlir::omp::OrderedClauseOps &result) const;
+  bool processPartial(std::optional<int64_t> &result) const;
   bool processPriority(lower::StatementContext &stmtCtx,
                        mlir::omp::PriorityClauseOps &result) const;
   bool processProcBind(mlir::omp::ProcBindClauseOps &result) const;
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 266b06f353675..c81e4a59de197 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2539,10 +2539,17 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
                         ConstructQueue::const_iterator item) {
   fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
 
-  // Clauses for unrolling not yet implemnted
   ClauseProcessor cp(converter, semaCtx, item->clauses);
-  cp.processTODO<clause::Partial, clause::Full>(
-      loc, llvm::omp::Directive::OMPD_unroll);
+
+  // The `full` clause is not yet implemented.
+  cp.processTODO<clause::Full>(loc, llvm::omp::Directive::OMPD_unroll);
+
+  // Process the `partial` clause. If present, it may carry a constant unroll
+  // factor.
+  std::optional<int64_t> partialFactor;
+  bool hasPartial = cp.processPartial(partialFactor);
+  if (hasPartial && !partialFactor.has_value())
+    TODO(loc, "PARTIAL clause on UNROLL without a constant factor");
 
   // Emit the associated loop
   llvm::SmallVector<mlir::omp::CanonicalLoopOp, 1> canonLoops;
@@ -2554,9 +2561,16 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
   for (auto &&canonLoop : canonLoops)
     applyees.push_back(canonLoop.getCli());
 
-  // Apply unrolling to it
   auto cli = llvm::getSingleElement(canonLoops).getCli();
-  mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+
+  if (partialFactor.has_value()) {
+    // Partially unroll the loop by the given constant factor.
+    mlir::omp::UnrollPartialOp::create(firOpBuilder, loc, cli,
+                                       static_cast<uint64_t>(*partialFactor));
+  } else {
+    // Apply heuristic unrolling to it.
+    mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+  }
 }
 
 static mlir::omp::MaskedOp
diff --git a/flang/test/Lower/OpenMP/unroll-partial01.f90 b/flang/test/Lower/OpenMP/unroll-partial01.f90
new file mode 100644
index 0000000000000..fe76d8f077753
--- /dev/null
+++ b/flang/test/Lower/OpenMP/unroll-partial01.f90
@@ -0,0 +1,54 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+
+subroutine omp_unroll_partial01(lb, ub, inc)
+  integer res, i, lb, ub, inc
+
+  !$omp unroll partial(4)
+  do i = lb, ub, inc
+    res = i
+  end do
+  !$omp end unroll
+
+end subroutine omp_unroll_partial01
+
+
+! CHECK-LABEL:   func.func @_QPomp_unroll_partial01(
+! CHECK-SAME:      %[[ARG0:.*]]: !fir.ref<i32> {fir.bindc_name = "lb"},
+! CHECK-SAME:      %[[ARG1:.*]]: !fir.ref<i32> {fir.bindc_name = "ub"},
+! CHECK-SAME:      %[[ARG2:.*]]: !fir.ref<i32> {fir.bindc_name = "inc"}) {
+! CHECK:           %[[VAL_0:.*]] = fir.dummy_scope : !fir.dscope
+! CHECK:           %[[VAL_1:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFomp_unroll_partial01Ei"}
+! CHECK:           %[[VAL_2:.*]]:2 = hlfir.declare %[[VAL_1]] {uniq_name = "_QFomp_unroll_partial01Ei"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_3:.*]]:2 = hlfir.declare %[[ARG2]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Einc"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_4:.*]]:2 = hlfir.declare %[[ARG0]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Elb"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_5:.*]] = fir.alloca i32 {bindc_name = "res", uniq_name = "_QFomp_unroll_partial01Eres"}
+! CHECK:           %[[VAL_6:.*]]:2 = hlfir.declare %[[VAL_5]] {uniq_name = "_QFomp_unroll_partial01Eres"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_7:.*]]:2 = hlfir.declare %[[ARG1]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Eub"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_8:.*]] = fir.load %[[VAL_4]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_9:.*]] = fir.load %[[VAL_7]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_10:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+! CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+! CHECK:           %[[VAL_13:.*]] = arith.cmpi slt, %[[VAL_10]], %[[VAL_11]] : i32
+! CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_15:.*]] = arith.select %[[VAL_13]], %[[VAL_14]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_16:.*]] = arith.select %[[VAL_13]], %[[VAL_9]], %[[VAL_8]] : i32
+! CHECK:           %[[VAL_17:.*]] = arith.select %[[VAL_13]], %[[VAL_8]], %[[VAL_9]] : i32
+! CHECK:           %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_16]] overflow<nuw> : i32
+! CHECK:           %[[VAL_19:.*]] = arith.divui %[[VAL_18]], %[[VAL_15]] : i32
+! CHECK:           %[[VAL_20:.*]] = arith.addi %[[VAL_19]], %[[VAL_12]] overflow<nuw> : i32
+! CHECK:           %[[VAL_21:.*]] = arith.cmpi slt, %[[VAL_17]], %[[VAL_16]] : i32
+! CHECK:           %[[VAL_22:.*]] = arith.select %[[VAL_21]], %[[VAL_11]], %[[VAL_20]] : i32
+! CHECK:           %[[VAL_23:.*]] = omp.new_cli
+! CHECK:           omp.canonical_loop(%[[VAL_23]]) %[[VAL_24:.*]] : i32 in range(%[[VAL_22]]) {
+! CHECK:             %[[VAL_25:.*]] = arith.muli %[[VAL_24]], %[[VAL_10]] : i32
+! CHECK:             %[[VAL_26:.*]] = arith.addi %[[VAL_8]], %[[VAL_25]] : i32
+! CHECK:             hlfir.assign %[[VAL_26]] to %[[VAL_2]]#0 : i32, !fir.ref<i32>
+! CHECK:             %[[VAL_27:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:             hlfir.assign %[[VAL_27]] to %[[VAL_6]]#0 : i32, !fir.ref<i32>
+! CHECK:             omp.terminator
+! CHECK:           }
+! CHECK:           omp.unroll_partial(%[[VAL_23]]) {unroll_factor = 4 : i64}
+! CHECK:           return
+! CHECK:         }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..b5e4cefa1ce72 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -556,6 +556,41 @@ def UnrollHeuristicOp : OpenMPTransform_Op<"unroll_heuristic", []> {
   let hasCustomAssemblyFormat = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMP unroll_partial operation
+//===----------------------------------------------------------------------===//
+
+def UnrollPartialOp : OpenMPTransform_Op<"unroll_partial", []> {
+  let summary = "OpenMP partial unroll operation";
+  let description = [{
+    Represents a `#pragma omp unroll partial` construct introduced in OpenMP
+    5.1.
+
+    The operation has one applyee and no generatees. The applyee is partially
+    unrolled by the factor given in the `unroll_factor` attribute, which must be
+    a positive constant. That is, the operation replaces `unroll_factor`
+    consecutive logical iterations of the applyee by a single iteration that
+    executes the original body `unroll_factor` times.
+
+    The resulting unrolled loop is currently not accessible as a generatee, so
+    no further loop-associated transformations can be chained onto it.
+
+    The lowering is implemented using `OpenMPIRBuilder::unrollLoopPartial`.
+
+    Assembly formats:
+      omp.unroll_partial(%cli) {unroll_factor = 4 : i64}
+  }];
+
+  let arguments = (ins CanonicalLoopInfoType:$applyee,
+                       ConfinedAttr<I64Attr, [IntPositive]>:$unroll_factor);
+
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$cli, "uint64_t":$unrollFactor)>,
+  ];
+
+  let hasCustomAssemblyFormat = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // OpenMP tile operation
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..27296190d6ac3 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4193,6 +4193,56 @@ UnrollHeuristicOp::getGenerateesODSOperandIndexAndLength() {
   return {0, 0};
 }
 
+//===----------------------------------------------------------------------===//
+// UnrollPartialOp
+//===----------------------------------------------------------------------===//
+
+void UnrollPartialOp::build(::mlir::OpBuilder &odsBuilder,
+                            ::mlir::OperationState &odsState, ::mlir::Value cli,
+                            uint64_t unrollFactor) {
+  odsState.addOperands(cli);
+  Properties &props = odsState.getOrAddProperties<Properties>();
+  props.unroll_factor = odsBuilder.getI64IntegerAttr(unrollFactor);
+}
+
+void UnrollPartialOp::print(OpAsmPrinter &p) {
+  p << '(' << getApplyee() << ')';
+
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+mlir::ParseResult UnrollPartialOp::parse(::mlir::OpAsmParser &parser,
+                                         ::mlir::OperationState &result) {
+  auto cliType = CanonicalLoopInfoType::get(parser.getContext());
+
+  if (parser.parseLParen())
+    return failure();
+
+  OpAsmParser::UnresolvedOperand applyee;
+  if (parser.parseOperand(applyee) ||
+      parser.resolveOperand(applyee, cliType, result.operands))
+    return failure();
+
+  if (parser.parseRParen())
+    return failure();
+
+  // The unroll factor is carried by the `unroll_factor` attribute.
+  if (parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return mlir::success();
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getApplyeesODSOperandIndexAndLength() {
+  return getODSOperandIndexAndLength(odsIndex_applyee);
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getGenerateesODSOperandIndexAndLength() {
+  return {0, 0};
+}
+
 //===----------------------------------------------------------------------===//
 // TileOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f0511bb4be7dd..d15c7f01c3674 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4490,6 +4490,30 @@ applyUnrollHeuristic(omp::UnrollHeuristicOp op, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Apply a `#pragma omp unroll partial` / `!$omp unroll partial`
+/// transformation using the OpenMPIRBuilder.
+static LogicalResult
+applyUnrollPartial(omp::UnrollPartialOp op, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  Value applyee = op.getApplyee();
+  assert(applyee && "Loop to apply unrolling on required");
+
+  llvm::CanonicalLoopInfo *consBuilderCLI =
+      moduleTranslation.lookupOMPLoop(applyee);
+  llvm::OpenMPIRBuilder::LocationDescription loc(builder);
+
+  // No generatee is supported yet, so the unrolled loop's CanonicalLoopInfo is
+  // not requested and unrolling is deferred to LLVM's LoopUnroll pass.
+  int32_t factor = static_cast<int32_t>(op.getUnrollFactor());
+  ompBuilder->unrollLoopPartial(loc.DL, consBuilderCLI, factor,
+                                /*UnrolledCLI=*/nullptr);
+
+  moduleTranslation.invalidateOmpLoop(applyee);
+  return success();
+}
+
 /// Apply a `#pragma omp tile` / `!$omp tile` transformation using the
 /// OpenMPIRBuilder.
 static LogicalResult applyTile(omp::TileOp op, llvm::IRBuilderBase &builder,
@@ -8889,6 +8913,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             // the omp.canonical_loop.
             return applyUnrollHeuristic(op, builder, moduleTranslation);
           })
+          .Case([&](omp::UnrollPartialOp op) {
+            return applyUnrollPartial(op, builder, moduleTranslation);
+          })
           .Case([&](omp::TileOp op) {
             return applyTile(op, builder, moduleTranslation);
           })

>From cac9adbd34fa5b2e0dee49d6e9997fb0c7fa0332 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Mon, 6 Jul 2026 09:11:26 -0500
Subject: [PATCH 2/2] testcases

---
 .../Dialect/OpenMP/cli-unroll-partial.mlir    | 59 +++++++++++
 mlir/test/Dialect/OpenMP/invalid.mlir         | 16 +++
 .../LLVMIR/openmp-unroll-partial01.mlir       | 57 +++++++++++
 .../LLVMIR/openmp-unroll-partial02.mlir       | 97 +++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100644 mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir

diff --git a/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
new file mode 100644
index 0000000000000..5ac3a1557ba79
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
@@ -0,0 +1,59 @@
+// RUN: mlir-opt %s            | FileCheck %s --enable-var-scope
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --enable-var-scope
+
+
+// CHECK-LABEL: @omp_unroll_partial_raw(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_raw(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %canonloop = "omp.new_cli" () : () -> (!omp.cli)
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  "omp.canonical_loop" (%tc, %canonloop) ({
+    ^bb0(%iv: i32):
+      omp.terminator
+  }) : (i32, !omp.cli) -> ()
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  "omp.unroll_partial" (%canonloop) {unroll_factor = 4 : i64} : (!omp.cli) -> ()
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %[[CANONLOOP:.+]] = omp.new_cli
+  %canonloop = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%canonloop) %iv : i32 in range(%tc) {
+    omp.terminator
+  }
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_nested_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_nested_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %cli_outer = omp.new_cli
+  // CHECK-NEXT: %canonloop_d1 = omp.new_cli
+  %cli_inner = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%cli_outer) %iv_outer : i32 in range(%tc) {
+    // CHECK-NEXT: omp.canonical_loop(%canonloop_d1) %iv_d1 : i32 in range(%[[tc]]) {
+    omp.canonical_loop(%cli_inner) %iv_inner : i32 in range(%tc) {
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    // CHECK: omp.terminator
+    omp.terminator
+  }
+
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  omp.unroll_partial(%cli_outer) {unroll_factor = 4 : i64}
+  // CHECK-NEXT: omp.unroll_partial(%canonloop_d1) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%cli_inner) {unroll_factor = 2 : i64}
+  return
+}
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 7ff4755174053..bc14d791340cf 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4522,3 +4522,19 @@ func.func @free_shared_mem_invalid_alignment2(%n: i32, %ptr : !llvm.ptr) -> () {
   omp.free_shared_mem [%n x i64 : (i32) align(3)] %ptr : !llvm.ptr
   return
 }
+
+// -----
+// unroll_factor = 0 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_zero(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = 0 : i64}
+  return
+}
+
+// -----
+// unroll_factor = -1 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
+  return
+}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
new file mode 100644
index 0000000000000..b9938afea0c30
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
@@ -0,0 +1,57 @@
+// Test lowering of the omp.unroll_partial (single loop)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_trivial_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_trivial_loop(%ptr: !llvm.ptr, %tc: i32) -> () {
+  %literal_cli = omp.new_cli
+  omp.canonical_loop(%literal_cli) %iv : i32 in range(%tc) {
+    %val = llvm.mlir.constant(42.0 : f32) : f32
+    llvm.store %val, %ptr : f32, !llvm.ptr
+    omp.terminator
+  }
+  omp.unroll_partial(%literal_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
new file mode 100644
index 0000000000000..a62bc499255db
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
@@ -0,0 +1,97 @@
+// Test lowering of the omp.unroll_partial (nested loops)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_nested_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[outer_tc:.+]], i32 %[[inner_tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[outer_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   br label %omp_omp.loop.preheader1
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader1:
+// CHECK-NEXT:   br label %omp_omp.loop.header2
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header2:
+// CHECK-NEXT:   %omp_omp.loop.iv8 = phi i32 [ 0, %omp_omp.loop.preheader1 ], [ %omp_omp.loop.next10, %omp_omp.loop.inc5 ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond3
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond3:
+// CHECK-NEXT:   %omp_omp.loop.cmp9 = icmp ult i32 %omp_omp.loop.iv8, %[[inner_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp9, label %omp_omp.loop.body4, label %omp_omp.loop.exit6
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body4:
+// CHECK-NEXT:   br label %omp.loop.region12
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region12:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont11
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont11:
+// CHECK-NEXT:   br label %omp_omp.loop.inc5
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc5:
+// CHECK-NEXT:   %omp_omp.loop.next10 = add nuw i32 %omp_omp.loop.iv8, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header2, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit6:
+// CHECK-NEXT:   br label %omp_omp.loop.after7
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after7:
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD4:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_nested_loop(%ptr: !llvm.ptr, %outer_tc: i32, %inner_tc: i32) -> () {
+  %outer_cli = omp.new_cli
+  %inner_cli = omp.new_cli
+  omp.canonical_loop(%outer_cli) %outer_iv : i32 in range(%outer_tc) {
+    omp.canonical_loop(%inner_cli) %inner_iv : i32 in range(%inner_tc) {
+      %val = llvm.mlir.constant(42.0 : f32) : f32
+      llvm.store %val, %ptr : f32, !llvm.ptr
+      omp.terminator
+    }
+    omp.terminator
+  }
+  omp.unroll_partial(%outer_cli) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%inner_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// The inner loop back-edge (header2) is encountered first in the IR dump, so
+// its metadata node gets a lower ID than the outer loop's metadata node.
+// Both loops share the same llvm.loop.unroll.enable node (uniqued by content).
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
+// CHECK: ![[$MD4]] = distinct !{![[$MD4]], ![[$MD2]], ![[$MD5:[0-9]+]]}
+// CHECK: ![[$MD5]] = !{!"llvm.loop.unroll.count", i32 2}


From llvmlistbot at llvm.org  Mon Jul  6 07:13:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 07:13:23 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a4bb803.f8b79514.3b441.2c81@mx.google.com>


https://github.com/ShashwathiNavada updated https://github.com/llvm/llvm-project/pull/206642

>From d6cdc9ec1ffc9ad1b70b2c8cbfb0430548c3b724 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Thu, 25 Jun 2026 01:40:11 -0500
Subject: [PATCH 1/2] Implement support for partial in unroll construct

---
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    |  9 ++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 24 +++++++--
 flang/test/Lower/OpenMP/unroll-partial01.f90  | 54 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 35 ++++++++++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 50 +++++++++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 27 ++++++++++
 7 files changed, 195 insertions(+), 5 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/unroll-partial01.f90

diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index e2a2c8575ffc8..9031f6ddf0744 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -795,6 +795,15 @@ bool ClauseProcessor::processOrdered(
   return false;
 }
 
+bool ClauseProcessor::processPartial(std::optional<int64_t> &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Partial>()) {
+    if (clause->v.has_value())
+      result = evaluate::ToInt64(*clause->v);
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processPriority(
     lower::StatementContext &stmtCtx,
     mlir::omp::PriorityClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1fc221b721ebf..e92db70ca707b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -111,6 +111,7 @@ class ClauseProcessor {
                          mlir::omp::NumThreadsClauseOps &result) const;
   bool processOrder(mlir::omp::OrderClauseOps &result) const;
   bool processOrdered(mlir::omp::OrderedClauseOps &result) const;
+  bool processPartial(std::optional<int64_t> &result) const;
   bool processPriority(lower::StatementContext &stmtCtx,
                        mlir::omp::PriorityClauseOps &result) const;
   bool processProcBind(mlir::omp::ProcBindClauseOps &result) const;
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 266b06f353675..c81e4a59de197 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2539,10 +2539,17 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
                         ConstructQueue::const_iterator item) {
   fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
 
-  // Clauses for unrolling not yet implemnted
   ClauseProcessor cp(converter, semaCtx, item->clauses);
-  cp.processTODO<clause::Partial, clause::Full>(
-      loc, llvm::omp::Directive::OMPD_unroll);
+
+  // The `full` clause is not yet implemented.
+  cp.processTODO<clause::Full>(loc, llvm::omp::Directive::OMPD_unroll);
+
+  // Process the `partial` clause. If present, it may carry a constant unroll
+  // factor.
+  std::optional<int64_t> partialFactor;
+  bool hasPartial = cp.processPartial(partialFactor);
+  if (hasPartial && !partialFactor.has_value())
+    TODO(loc, "PARTIAL clause on UNROLL without a constant factor");
 
   // Emit the associated loop
   llvm::SmallVector<mlir::omp::CanonicalLoopOp, 1> canonLoops;
@@ -2554,9 +2561,16 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
   for (auto &&canonLoop : canonLoops)
     applyees.push_back(canonLoop.getCli());
 
-  // Apply unrolling to it
   auto cli = llvm::getSingleElement(canonLoops).getCli();
-  mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+
+  if (partialFactor.has_value()) {
+    // Partially unroll the loop by the given constant factor.
+    mlir::omp::UnrollPartialOp::create(firOpBuilder, loc, cli,
+                                       static_cast<uint64_t>(*partialFactor));
+  } else {
+    // Apply heuristic unrolling to it.
+    mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+  }
 }
 
 static mlir::omp::MaskedOp
diff --git a/flang/test/Lower/OpenMP/unroll-partial01.f90 b/flang/test/Lower/OpenMP/unroll-partial01.f90
new file mode 100644
index 0000000000000..fe76d8f077753
--- /dev/null
+++ b/flang/test/Lower/OpenMP/unroll-partial01.f90
@@ -0,0 +1,54 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+
+subroutine omp_unroll_partial01(lb, ub, inc)
+  integer res, i, lb, ub, inc
+
+  !$omp unroll partial(4)
+  do i = lb, ub, inc
+    res = i
+  end do
+  !$omp end unroll
+
+end subroutine omp_unroll_partial01
+
+
+! CHECK-LABEL:   func.func @_QPomp_unroll_partial01(
+! CHECK-SAME:      %[[ARG0:.*]]: !fir.ref<i32> {fir.bindc_name = "lb"},
+! CHECK-SAME:      %[[ARG1:.*]]: !fir.ref<i32> {fir.bindc_name = "ub"},
+! CHECK-SAME:      %[[ARG2:.*]]: !fir.ref<i32> {fir.bindc_name = "inc"}) {
+! CHECK:           %[[VAL_0:.*]] = fir.dummy_scope : !fir.dscope
+! CHECK:           %[[VAL_1:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFomp_unroll_partial01Ei"}
+! CHECK:           %[[VAL_2:.*]]:2 = hlfir.declare %[[VAL_1]] {uniq_name = "_QFomp_unroll_partial01Ei"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_3:.*]]:2 = hlfir.declare %[[ARG2]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Einc"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_4:.*]]:2 = hlfir.declare %[[ARG0]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Elb"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_5:.*]] = fir.alloca i32 {bindc_name = "res", uniq_name = "_QFomp_unroll_partial01Eres"}
+! CHECK:           %[[VAL_6:.*]]:2 = hlfir.declare %[[VAL_5]] {uniq_name = "_QFomp_unroll_partial01Eres"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_7:.*]]:2 = hlfir.declare %[[ARG1]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Eub"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_8:.*]] = fir.load %[[VAL_4]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_9:.*]] = fir.load %[[VAL_7]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_10:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+! CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+! CHECK:           %[[VAL_13:.*]] = arith.cmpi slt, %[[VAL_10]], %[[VAL_11]] : i32
+! CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_15:.*]] = arith.select %[[VAL_13]], %[[VAL_14]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_16:.*]] = arith.select %[[VAL_13]], %[[VAL_9]], %[[VAL_8]] : i32
+! CHECK:           %[[VAL_17:.*]] = arith.select %[[VAL_13]], %[[VAL_8]], %[[VAL_9]] : i32
+! CHECK:           %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_16]] overflow<nuw> : i32
+! CHECK:           %[[VAL_19:.*]] = arith.divui %[[VAL_18]], %[[VAL_15]] : i32
+! CHECK:           %[[VAL_20:.*]] = arith.addi %[[VAL_19]], %[[VAL_12]] overflow<nuw> : i32
+! CHECK:           %[[VAL_21:.*]] = arith.cmpi slt, %[[VAL_17]], %[[VAL_16]] : i32
+! CHECK:           %[[VAL_22:.*]] = arith.select %[[VAL_21]], %[[VAL_11]], %[[VAL_20]] : i32
+! CHECK:           %[[VAL_23:.*]] = omp.new_cli
+! CHECK:           omp.canonical_loop(%[[VAL_23]]) %[[VAL_24:.*]] : i32 in range(%[[VAL_22]]) {
+! CHECK:             %[[VAL_25:.*]] = arith.muli %[[VAL_24]], %[[VAL_10]] : i32
+! CHECK:             %[[VAL_26:.*]] = arith.addi %[[VAL_8]], %[[VAL_25]] : i32
+! CHECK:             hlfir.assign %[[VAL_26]] to %[[VAL_2]]#0 : i32, !fir.ref<i32>
+! CHECK:             %[[VAL_27:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:             hlfir.assign %[[VAL_27]] to %[[VAL_6]]#0 : i32, !fir.ref<i32>
+! CHECK:             omp.terminator
+! CHECK:           }
+! CHECK:           omp.unroll_partial(%[[VAL_23]]) {unroll_factor = 4 : i64}
+! CHECK:           return
+! CHECK:         }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..b5e4cefa1ce72 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -556,6 +556,41 @@ def UnrollHeuristicOp : OpenMPTransform_Op<"unroll_heuristic", []> {
   let hasCustomAssemblyFormat = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMP unroll_partial operation
+//===----------------------------------------------------------------------===//
+
+def UnrollPartialOp : OpenMPTransform_Op<"unroll_partial", []> {
+  let summary = "OpenMP partial unroll operation";
+  let description = [{
+    Represents a `#pragma omp unroll partial` construct introduced in OpenMP
+    5.1.
+
+    The operation has one applyee and no generatees. The applyee is partially
+    unrolled by the factor given in the `unroll_factor` attribute, which must be
+    a positive constant. That is, the operation replaces `unroll_factor`
+    consecutive logical iterations of the applyee by a single iteration that
+    executes the original body `unroll_factor` times.
+
+    The resulting unrolled loop is currently not accessible as a generatee, so
+    no further loop-associated transformations can be chained onto it.
+
+    The lowering is implemented using `OpenMPIRBuilder::unrollLoopPartial`.
+
+    Assembly formats:
+      omp.unroll_partial(%cli) {unroll_factor = 4 : i64}
+  }];
+
+  let arguments = (ins CanonicalLoopInfoType:$applyee,
+                       ConfinedAttr<I64Attr, [IntPositive]>:$unroll_factor);
+
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$cli, "uint64_t":$unrollFactor)>,
+  ];
+
+  let hasCustomAssemblyFormat = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // OpenMP tile operation
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..27296190d6ac3 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4193,6 +4193,56 @@ UnrollHeuristicOp::getGenerateesODSOperandIndexAndLength() {
   return {0, 0};
 }
 
+//===----------------------------------------------------------------------===//
+// UnrollPartialOp
+//===----------------------------------------------------------------------===//
+
+void UnrollPartialOp::build(::mlir::OpBuilder &odsBuilder,
+                            ::mlir::OperationState &odsState, ::mlir::Value cli,
+                            uint64_t unrollFactor) {
+  odsState.addOperands(cli);
+  Properties &props = odsState.getOrAddProperties<Properties>();
+  props.unroll_factor = odsBuilder.getI64IntegerAttr(unrollFactor);
+}
+
+void UnrollPartialOp::print(OpAsmPrinter &p) {
+  p << '(' << getApplyee() << ')';
+
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+mlir::ParseResult UnrollPartialOp::parse(::mlir::OpAsmParser &parser,
+                                         ::mlir::OperationState &result) {
+  auto cliType = CanonicalLoopInfoType::get(parser.getContext());
+
+  if (parser.parseLParen())
+    return failure();
+
+  OpAsmParser::UnresolvedOperand applyee;
+  if (parser.parseOperand(applyee) ||
+      parser.resolveOperand(applyee, cliType, result.operands))
+    return failure();
+
+  if (parser.parseRParen())
+    return failure();
+
+  // The unroll factor is carried by the `unroll_factor` attribute.
+  if (parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return mlir::success();
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getApplyeesODSOperandIndexAndLength() {
+  return getODSOperandIndexAndLength(odsIndex_applyee);
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getGenerateesODSOperandIndexAndLength() {
+  return {0, 0};
+}
+
 //===----------------------------------------------------------------------===//
 // TileOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f0511bb4be7dd..d15c7f01c3674 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4490,6 +4490,30 @@ applyUnrollHeuristic(omp::UnrollHeuristicOp op, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Apply a `#pragma omp unroll partial` / `!$omp unroll partial`
+/// transformation using the OpenMPIRBuilder.
+static LogicalResult
+applyUnrollPartial(omp::UnrollPartialOp op, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  Value applyee = op.getApplyee();
+  assert(applyee && "Loop to apply unrolling on required");
+
+  llvm::CanonicalLoopInfo *consBuilderCLI =
+      moduleTranslation.lookupOMPLoop(applyee);
+  llvm::OpenMPIRBuilder::LocationDescription loc(builder);
+
+  // No generatee is supported yet, so the unrolled loop's CanonicalLoopInfo is
+  // not requested and unrolling is deferred to LLVM's LoopUnroll pass.
+  int32_t factor = static_cast<int32_t>(op.getUnrollFactor());
+  ompBuilder->unrollLoopPartial(loc.DL, consBuilderCLI, factor,
+                                /*UnrolledCLI=*/nullptr);
+
+  moduleTranslation.invalidateOmpLoop(applyee);
+  return success();
+}
+
 /// Apply a `#pragma omp tile` / `!$omp tile` transformation using the
 /// OpenMPIRBuilder.
 static LogicalResult applyTile(omp::TileOp op, llvm::IRBuilderBase &builder,
@@ -8889,6 +8913,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             // the omp.canonical_loop.
             return applyUnrollHeuristic(op, builder, moduleTranslation);
           })
+          .Case([&](omp::UnrollPartialOp op) {
+            return applyUnrollPartial(op, builder, moduleTranslation);
+          })
           .Case([&](omp::TileOp op) {
             return applyTile(op, builder, moduleTranslation);
           })

>From cac9adbd34fa5b2e0dee49d6e9997fb0c7fa0332 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Mon, 6 Jul 2026 09:11:26 -0500
Subject: [PATCH 2/2] testcases

---
 .../Dialect/OpenMP/cli-unroll-partial.mlir    | 59 +++++++++++
 mlir/test/Dialect/OpenMP/invalid.mlir         | 16 +++
 .../LLVMIR/openmp-unroll-partial01.mlir       | 57 +++++++++++
 .../LLVMIR/openmp-unroll-partial02.mlir       | 97 +++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100644 mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir

diff --git a/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
new file mode 100644
index 0000000000000..5ac3a1557ba79
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
@@ -0,0 +1,59 @@
+// RUN: mlir-opt %s            | FileCheck %s --enable-var-scope
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --enable-var-scope
+
+
+// CHECK-LABEL: @omp_unroll_partial_raw(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_raw(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %canonloop = "omp.new_cli" () : () -> (!omp.cli)
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  "omp.canonical_loop" (%tc, %canonloop) ({
+    ^bb0(%iv: i32):
+      omp.terminator
+  }) : (i32, !omp.cli) -> ()
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  "omp.unroll_partial" (%canonloop) {unroll_factor = 4 : i64} : (!omp.cli) -> ()
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %[[CANONLOOP:.+]] = omp.new_cli
+  %canonloop = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%canonloop) %iv : i32 in range(%tc) {
+    omp.terminator
+  }
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_nested_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_nested_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %cli_outer = omp.new_cli
+  // CHECK-NEXT: %canonloop_d1 = omp.new_cli
+  %cli_inner = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%cli_outer) %iv_outer : i32 in range(%tc) {
+    // CHECK-NEXT: omp.canonical_loop(%canonloop_d1) %iv_d1 : i32 in range(%[[tc]]) {
+    omp.canonical_loop(%cli_inner) %iv_inner : i32 in range(%tc) {
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    // CHECK: omp.terminator
+    omp.terminator
+  }
+
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  omp.unroll_partial(%cli_outer) {unroll_factor = 4 : i64}
+  // CHECK-NEXT: omp.unroll_partial(%canonloop_d1) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%cli_inner) {unroll_factor = 2 : i64}
+  return
+}
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 7ff4755174053..bc14d791340cf 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4522,3 +4522,19 @@ func.func @free_shared_mem_invalid_alignment2(%n: i32, %ptr : !llvm.ptr) -> () {
   omp.free_shared_mem [%n x i64 : (i32) align(3)] %ptr : !llvm.ptr
   return
 }
+
+// -----
+// unroll_factor = 0 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_zero(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = 0 : i64}
+  return
+}
+
+// -----
+// unroll_factor = -1 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
+  return
+}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
new file mode 100644
index 0000000000000..b9938afea0c30
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
@@ -0,0 +1,57 @@
+// Test lowering of the omp.unroll_partial (single loop)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_trivial_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_trivial_loop(%ptr: !llvm.ptr, %tc: i32) -> () {
+  %literal_cli = omp.new_cli
+  omp.canonical_loop(%literal_cli) %iv : i32 in range(%tc) {
+    %val = llvm.mlir.constant(42.0 : f32) : f32
+    llvm.store %val, %ptr : f32, !llvm.ptr
+    omp.terminator
+  }
+  omp.unroll_partial(%literal_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
new file mode 100644
index 0000000000000..a62bc499255db
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
@@ -0,0 +1,97 @@
+// Test lowering of the omp.unroll_partial (nested loops)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_nested_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[outer_tc:.+]], i32 %[[inner_tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[outer_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   br label %omp_omp.loop.preheader1
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader1:
+// CHECK-NEXT:   br label %omp_omp.loop.header2
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header2:
+// CHECK-NEXT:   %omp_omp.loop.iv8 = phi i32 [ 0, %omp_omp.loop.preheader1 ], [ %omp_omp.loop.next10, %omp_omp.loop.inc5 ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond3
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond3:
+// CHECK-NEXT:   %omp_omp.loop.cmp9 = icmp ult i32 %omp_omp.loop.iv8, %[[inner_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp9, label %omp_omp.loop.body4, label %omp_omp.loop.exit6
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body4:
+// CHECK-NEXT:   br label %omp.loop.region12
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region12:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont11
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont11:
+// CHECK-NEXT:   br label %omp_omp.loop.inc5
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc5:
+// CHECK-NEXT:   %omp_omp.loop.next10 = add nuw i32 %omp_omp.loop.iv8, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header2, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit6:
+// CHECK-NEXT:   br label %omp_omp.loop.after7
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after7:
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD4:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_nested_loop(%ptr: !llvm.ptr, %outer_tc: i32, %inner_tc: i32) -> () {
+  %outer_cli = omp.new_cli
+  %inner_cli = omp.new_cli
+  omp.canonical_loop(%outer_cli) %outer_iv : i32 in range(%outer_tc) {
+    omp.canonical_loop(%inner_cli) %inner_iv : i32 in range(%inner_tc) {
+      %val = llvm.mlir.constant(42.0 : f32) : f32
+      llvm.store %val, %ptr : f32, !llvm.ptr
+      omp.terminator
+    }
+    omp.terminator
+  }
+  omp.unroll_partial(%outer_cli) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%inner_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// The inner loop back-edge (header2) is encountered first in the IR dump, so
+// its metadata node gets a lower ID than the outer loop's metadata node.
+// Both loops share the same llvm.loop.unroll.enable node (uniqued by content).
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
+// CHECK: ![[$MD4]] = distinct !{![[$MD4]], ![[$MD2]], ![[$MD5:[0-9]+]]}
+// CHECK: ![[$MD5]] = !{!"llvm.loop.unroll.count", i32 2}


From llvmlistbot at llvm.org  Mon Jul  6 07:33:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 07:33:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4109893 - [mlir][ArithToAMDGPU] Skip tensor
	scaling ops (#207327)
Message-ID: <6a4bbcbf.8e75e0cb.2e0812.2a1e@mx.google.com>


Author: Qixi
Date: 2026-07-06T10:33:31-04:00
New Revision: 4109893350feeea69214d938864e200ca5eec7b5

URL: https://github.com/llvm/llvm-project/commit/4109893350feeea69214d938864e200ca5eec7b5
DIFF: https://github.com/llvm/llvm-project/commit/4109893350feeea69214d938864e200ca5eec7b5.diff

LOG: [mlir][ArithToAMDGPU] Skip tensor scaling ops (#207327)

This fixes a crash in -convert-arith-to-amdgpu=chipset=gfx950 when the
input contains tensor arith.scaling_truncf operations.

The scaling rewrite patterns are written for scalar/vector values.
Tensor values should not be rewritten by these patterns, so this patch
returns failure for tensor operands and leaves those ops unchanged.

A regression test is added for the tensor reproducer from #207311.

Co-authored-by: tongjinxuan <tongjinxuan at longcheer.com>

Added: 
    mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir

Modified: 
    mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
index 3372faf4b16cd..cf44c2a0033ac 100644
--- a/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
+++ b/mlir/lib/Conversion/ArithToAMDGPU/ArithToAMDGPU.cpp
@@ -464,6 +464,11 @@ ScalingExtFRewritePattern::matchAndRewrite(arith::ScalingExtFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)
@@ -577,6 +582,11 @@ ScalingTruncFRewritePattern::matchAndRewrite(arith::ScalingTruncFOp op,
   if (outVecType && outVecType.isScalable())
     return failure();
 
+  if (isa<RankedTensorType>(out.getType()) ||
+      isa<RankedTensorType>(in.getType()) ||
+      isa<RankedTensorType>(scale.getType()))
+    return failure();
+
   Type scaleF32Type =
       scaleVecType ? VectorType::get(scaleVecType.getShape(), f32) : f32;
   if (scaleType.getIntOrFloatBitWidth() < 32)

diff  --git a/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
new file mode 100644
index 0000000000000..d22f35a6d07f1
--- /dev/null
+++ b/mlir/test/Conversion/ArithToAMDGPU/scaling-truncf-tensor.mlir
@@ -0,0 +1,17 @@
+// RUN: mlir-opt %s -convert-arith-to-amdgpu=chipset=gfx950 | FileCheck %s
+
+// CHECK-LABEL: func.func @m0
+// CHECK: arith.scaling_truncf
+func.func @m0(%arg0: tensor<16xf16>, %arg1: tensor<16xf8E8M0FNU>) -> tensor<16xf4E2M1FN> {
+  %0 = arith.scaling_truncf %arg0, %arg1 : tensor<16xf16>, tensor<16xf8E8M0FNU> to tensor<16xf4E2M1FN>
+  return %0 : tensor<16xf4E2M1FN>
+}
+
+// CHECK-LABEL: func.func @m1
+// CHECK: arith.constant
+// CHECK: arith.scaling_truncf
+func.func @m1(%arg0: tensor<4xf32>) -> tensor<4xf4E2M1FN> {
+  %cst = arith.constant dense<1.000000e+00> : tensor<4xf8E8M0FNU>
+  %0 = arith.scaling_truncf %arg0, %cst : tensor<4xf32>, tensor<4xf8E8M0FNU> to tensor<4xf4E2M1FN>
+  return %0 : tensor<4xf4E2M1FN>
+}


        

From llvmlistbot at llvm.org  Mon Jul  6 07:33:38 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Mon, 06 Jul 2026 07:33:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArithToAMDGPU] Skip tensor scaling ops
	(PR #207327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207327@github.com>
Message-ID: <6a4bbcc2.f732ad81.1d8ca.3fa9@mx.google.com>


https://github.com/kuhar closed https://github.com/llvm/llvm-project/pull/207327

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.30759430.1908d3.869a@mx.google.com>


https://github.com/banach-space commented:

Makes sense, thank you for the fix! Some minor comments inline.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.8e75e0cb.2e0812.9f23@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.13be6df8.3d8759.6c4f@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
----------------
banach-space wrote:

I know that you are trying to follow the pre-existing convention in this file, but could you instead move `negative` to the beginning of the test function name? As per our guidelines:
* https://mlir.llvm.org/getting_started/TestingGuide/#test-naming-convention

I would in fact appreciate if you updated all negative tests in this file 🙏🏻 

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.d5797644.146403.320b@mx.google.com>


================
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
----------------
banach-space wrote:

I prefer early and clear bail out. Also, I would update the comments.
```suggestion
// Chain of extract Ops that lead to the original Op that created the mask.
// TODO: add support to `vector.broadcast`.
SmallVector<vector::ExtractOp, 2> extractOps;

// Find the mask creation operation.
while (maskOp &&
!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
maskOp)) {

  auto extractOp = dyn_cast<vector::ExtractOp>(maskOp));
  if (!extractOp) {
        // Unsupported mask-defining op (e.g. a block argument, which has no
        // defining op, or an op we cannot trace through). Bail out rather than
        // looping forever or dereferencing a null op below.
     return failure()
  }
  maskOp = extractOp.getSource().getDefiningOp();
  extractOps.push_back(extractOp);
}
```


https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.01f0a1da.2016e9.d1bc@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload
----------------
banach-space wrote:

Note, this "block" of tests is for `maskedstore`.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:37:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:37:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4bbdb5.e9d484b9.11ad58.66a7@mx.google.com>


================
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
----------------
banach-space wrote:

This change would not be required.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Mon Jul  6 07:45:33 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:45:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a4bbf8d.492e23e4.e71ca.ab09@mx.google.com>


banach-space wrote:

My biggish concern here is that we've been trying to make sure that `vector.shape_cast` is a NOP. The unrolling introduces `tensor.extract_strided_slice`, which is suspect lowers to something other that a NOP?

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Mon Jul  6 07:47:01 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:47:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4bbfe5.30759430.1908d3.cc0f@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Mon Jul  6 07:51:03 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:51:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4bc0d7.f1c1923d.3672e2.0b51@mx.google.com>


================

----------------
banach-space wrote:

We try to prefix negative tests with `@negative_{}`. Also, a more descriptive function name would be welcome. 

I appreciate that you simply followed the pre- existing style, but it leaves since that leaves room for improvement, we could start here :) 

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Mon Jul  6 07:51:31 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:51:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4bc0f3.6c8668f2.346e76.f3d5@mx.google.com>


================

----------------
banach-space wrote:

If the fix is applied here, then anyone calling `collapseOpIterationDims` directly will get the wrong behaviour, right?

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Mon Jul  6 07:54:39 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 06 Jul 2026 07:54:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fold vector.to_elements from
	constants (PR #207098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207098@github.com>
Message-ID: <6a4bc1af.d1496011.3d49cb.8061@mx.google.com>


================

----------------
banach-space wrote:

There's a dedicated test file for canonicalizations for `vector.to_elements`:

```bash
$ ls mlir/test/Dialect/Vector/canonicalize
vector-extract.mlir             vector-from-elements.mlir       vector-step.mlir                vector-to-elements.mlir         vector-to-shape-cast.mlir       vector-transfer-read-write.mlir vector-transpose.mlir
```

You can re-use that.

https://github.com/llvm/llvm-project/pull/207098

From llvmlistbot at llvm.org  Mon Jul  6 07:54:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 07:54:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4bc1b7.ac469dc7.ed218.7801@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-tosa

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

This commit adds support for block scaled tensors.

In particular, the block scaled type has been extended with the `DenseElementTypeInterface` to allow block scaled data values to be specified in dense element attributes.

The block scaled type has also been extended to allow optional scale values to be specified by the type. For now, scale values are not expected to be propagated beyond their use in the attribute input of a constant operation. In the future, we may want to propagate these values to allow certain optimizations.

The `tosa.const` operation has also been updated in the validation pass.

Note: this PR relies on #<!-- -->203583 so also contains the contents of #<!-- -->203583

---

Patch is 32.11 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/205506.diff


9 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+19-1) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+7-3) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td (+17-3) 
- (modified) mlir/lib/Dialect/Tosa/IR/TosaOps.cpp (+136-13) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+72) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+18) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+13) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+69) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..291097a7f7514 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1305,7 +1305,25 @@ extensionComplianceMap = {
         {{fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
-        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.identity",
      {{{Extension::int4}, {{{i4T, i4T}, SpecificationVersion::V_1_0}}},
       {{Extension::int16}, {{{i48T, i48T}, SpecificationVersion::V_1_0}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..5e5760e33b5f6 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2970,9 +2970,13 @@ def Tosa_ConstOp : Tosa_Op<"const", [ConstantLike, Pure,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3,
+                  Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasFolder = 1;
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..e6dace5a8d3df 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -109,7 +109,12 @@ def Tosa_MXFPValue
 def Tosa_MXFPScale
     : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
 
-def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+def Tosa_BlockScaled
+    : Tosa_Type<"BlockScaled", "block_scaled",
+                [DeclareTypeInterfaceMethods<
+                    DenseElementTypeInterface, ["getDenseElementBitSize",
+                                                "convertToAttribute",
+                                                "convertFromAttribute"]>]> {
   let summary = "Block scaled tensor element type.";
 
   let description = [{
@@ -121,17 +126,26 @@ def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
       specifying the block size along the innermost dimension.
     `scale_type` - The type of the scale value associated with each block.
     `value_type` - The type of the data values in each block.
+    `scale_values` - Optional array of per-block scale values. If not provided,
+      the type is assumed to be dynamically quantized at runtime.
 
   }];
 
   let parameters = (ins
     EnumParameter<Tosa_BlockShape>:$block_shape,
     Tosa_MXFPScale:$scale_type,
-    Tosa_MXFPValue:$value_type
+    Tosa_MXFPValue:$value_type,
+    OptionalArrayRefParameter<"Attribute">:$scale_values
   );
 
   let assemblyFormat =
-      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
+      "`<` $block_shape```:```$scale_type```:```$value_type "
+      "(`,` `{` custom<ScaleValues>($scale_values, ref($scale_type))^ `}`)? "
+      "`>`";
+
+  let extraClassDeclaration = [{
+    bool hasScaleValues() const { return !getScaleValues().empty(); }
+  }];
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..ece1b0ffa29ba 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/Tosa/Utils/ShapeUtils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/VerificationUtils.h"
+#include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/Matchers.h"
@@ -740,14 +741,55 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+static ParseResult parseScaleValues(AsmParser &parser,
+                                    SmallVector<Attribute> &scaleValues,
+                                    Type scaleType) {
+  const auto parseScaleValue = [&]() -> ParseResult {
+    const SMLoc loc = parser.getCurrentLocation();
+
+    double floatValue;
+    if (parser.parseFloat(floatValue))
+      return failure();
+
+    if (floatValue < 0.0)
+      return parser.emitError(loc, "scale value must be non-negative, got ")
+             << floatValue;
+
+    Type attrType = scaleType;
+    if (succeeded(parser.parseOptionalColon()) && parser.parseType(attrType))
+      return failure();
+
+    if (attrType != scaleType)
+      return parser.emitError(loc, "parsed attribute type ")
+             << attrType << " does not match expected scale type " << scaleType;
+
+    scaleValues.push_back(FloatAttr::get(attrType, floatValue));
+    return success();
+  };
+
+  return parser.parseCommaSeparatedList(parseScaleValue);
+}
+
+static void printScaleValues(AsmPrinter &printer,
+                             ArrayRef<Attribute> scaleValues, Type) {
+  llvm::interleaveComma(scaleValues, printer, [&](Attribute scaleValue) {
+    printer.printAttributeWithoutType(scaleValue);
+  });
+}
+
+static LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type,
+                                                 bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
+  if (!allowScaleValues && elemType.hasScaleValues())
+    return op.emitError()
+           << "tensor type " << type
+           << " does not support scale values for this operation";
+
   if (!tensorType.hasRank())
     return success();
 
@@ -756,12 +798,24 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
            << "tensor type " << type
            << " does not support block scaling on scalar tensors";
 
-  const int64_t blockedDimension = tensorType.getShape().back();
-  if (ShapedType::isDynamic(blockedDimension))
-    return success();
-
+  const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+
+  if (allowScaleValues && elemType.hasScaleValues() &&
+      tensorType.hasStaticShape()) {
+    const size_t numBlocks =
+        llvm::accumulate(tensorShape, int64_t(1), std::multiplies<int64_t>());
+    if (elemType.getScaleValues().size() != numBlocks / blockSize)
+      return op.emitError()
+             << "tensor type " << type << " has " << numBlocks / blockSize
+             << " blocks but got " << elemType.getScaleValues().size()
+             << " scale values";
+  }
+
+  const int64_t blockedDimension = tensorShape.back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
   if (blockedDimension % blockSize != 0)
     return op.emitError()
            << "tensor type " << type
@@ -771,6 +825,43 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   return success();
 }
 
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  return ::verifyBlockScaledTensorType(op, type, /*allowScaleValues=*/false);
+}
+
+size_t mlir::tosa::BlockScaledType::getDenseElementBitSize() const {
+  const Type valueType = getValueType();
+  if (isa<tosa::mxint8Type>(valueType))
+    return 8;
+  return valueType.getIntOrFloatBitWidth();
+}
+
+Attribute
+mlir::tosa::BlockScaledType::convertToAttribute(ArrayRef<char> rawData) const {
+  assert(rawData.size() == 1 && "expected 1 byte for block_scaled element");
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertToAttribute(rawData);
+  if (!isa<FloatType>(valueType))
+    return {};
+  return mlir::detail::convertFloatTypeToAttribute(valueType, rawData);
+}
+
+LogicalResult mlir::tosa::BlockScaledType::convertFromAttribute(
+    Attribute attr, SmallVectorImpl<char> &result) const {
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertFromAttribute(attr, result);
+
+  const auto floatAttr = dyn_cast<FloatAttr>(attr);
+  if (!floatAttr || floatAttr.getType() != valueType)
+    return failure();
+  const APFloat value = floatAttr.getValue();
+  return mlir::detail::convertFloatTypeFromAttribute(
+      valueType, FloatAttr::get(valueType, value), result);
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -857,7 +948,7 @@ static LogicalResult verifyConvOp(T op) {
 }
 
 LogicalResult tosa::ConstOp::verify() {
-
+  Operation &op = *getOperation();
   auto attrType = llvm::dyn_cast<TensorType>(getValuesAttr().getType());
   auto outputType = llvm::dyn_cast<TensorType>(getOutput().getType());
 
@@ -866,17 +957,49 @@ LogicalResult tosa::ConstOp::verify() {
     return failure();
   }
 
-  if (auto result = llvm::dyn_cast<mlir::quant::QuantizedType>(
-          outputType.getElementType())) {
-    if (getStorageElementTypeFromQuantized(result) == attrType.getElementType())
+  const Type attrElemType = attrType.getElementType();
+  const Type resultElemType = outputType.getElementType();
+
+  if (auto result =
+          llvm::dyn_cast<mlir::quant::QuantizedType>(resultElemType)) {
+    if (getStorageElementTypeFromQuantized(result) == attrElemType)
       return success();
   }
 
-  if (attrType.getElementType() != outputType.getElementType()) {
-    emitOpError("expected same attr/result element types");
-    return failure();
+  if (auto attrBlockScaledType =
+          llvm::dyn_cast<mlir::tosa::BlockScaledType>(attrElemType)) {
+    if (failed(verifyBlockScaledTensorType(op, attrType, true)) ||
+        failed(verifyBlockScaledTensorType(op, outputType, false)))
+      return failure();
+
+    if (!attrBlockScaledType.hasScaleValues())
+      return op.emitOpError(
+          "attribute block scaled type must have scale values");
+
+    const BlockScaledType resultBlockScaledType =
+        llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
+    if (!resultBlockScaledType)
+      return op.emitOpError(
+          "result type must be block scaled type if attribute is block "
+          "scaled type");
+
+    if (attrBlockScaledType.getValueType() !=
+            resultBlockScaledType.getValueType() ||
+        attrBlockScaledType.getScaleType() !=
+            resultBlockScaledType.getScaleType() ||
+        attrBlockScaledType.getBlockShape() !=
+            resultBlockScaledType.getBlockShape())
+      return op.emitOpError(
+                 "expected block scaled element type to be compatible "
+                 "between attr and result, got ")
+             << attrBlockScaledType << " vs. " << resultBlockScaledType;
+
+    return success();
   }
 
+  if (attrElemType != resultElemType)
+    return emitOpError("expected same attr/result element types");
+
   return success();
 }
 
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..bc014fb9b928e 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -660,7 +660,7 @@ func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439
 // CHECK-LABEL: test_const
 func.func @test_const(%arg0 : index) -> tensor<4xi32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64] ]
+  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
     %0 = "tosa.const"() {values = dense<[3, 0, 1, 2]> : tensor<4xi32>} : () -> tensor<4xi32>
     return %0 : tensor<4xi32>
 }
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..4e71d512def2d 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1938,3 +1938,75 @@ func.func @test_assert_equal_shape() {
   tosa.assert_equal_shape %0, %1 {allow_broadcast = true} : (!tosa.shape<2>, !tosa.shape<2>) -> ()
   return
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const
+func.func @test_block_scaled_const() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 1.0}>> : [[0.0 : f8E4M3FN, 1.0 : f8E4M3FN, 0.001953125 : f8E4M3FN, 0.0078125 : f8E4M3FN, 
+                                                                                                                          2.0 : f8E4M3FN, 2.25 : f8E4M3FN, 2.5 : f8E4M3FN, 2.75 : f8E4M3FN, 
+                                                                                                                          15.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,  
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN], 
+                                                                                                                         [0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN]]>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat
+func.func @test_block_scaled_const_splat() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0, 1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat_mxint8
+func.func @test_block_scaled_const_splat_mxint8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 1.0}>> : 0 : i8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values
+func.func @test_block_scaled_const_scale_values() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_explicit_type
+func.func @test_block_scaled_const_scale_values_explicit_type() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : f8E8M0FNU, 2.0 : f8E8M0FNU}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_wide_inner_dim
+func.func @test_block_scaled_const_scale_values_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0, 4.0, 8.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_cast_scale_values_no_propagate
+func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x32xf32> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Mon Jul  6 08:02:45 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Mon, 06 Jul 2026 08:02:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4bc395.481afadf.134314.8621@mx.google.com>


================

----------------
nujaa wrote:

They will only if `collapseOpIterationDims` gets called with a wrong reassociation list.

Glad to hear from you :) 


https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Mon Jul  6 08:51:21 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 08:51:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold tiled input into binary
 elementwise operations (PR #203941)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203941@github.com>
Message-ID: <6a4bcef9.e55dc341.29c9b8.7661@mx.google.com>


lhutton1 wrote:

cc @sahas3 

https://github.com/llvm/llvm-project/pull/203941

From llvmlistbot at llvm.org  Mon Jul  6 08:51:51 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 06 Jul 2026 08:51:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4bcf17.d1496011.3d49cb.7dea@mx.google.com>


https://github.com/razvanlupusoru edited https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Mon Jul  6 08:51:51 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 06 Jul 2026 08:51:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4bcf17.f1c1923d.3672e2.5036@mx.google.com>


https://github.com/razvanlupusoru commented:

Thank you.

https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Mon Jul  6 08:51:51 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 06 Jul 2026 08:51:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4bcf17.bb839515.23e6d3.9ad1@mx.google.com>


================
@@ -155,3 +155,66 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
   acc.declare_exit token(%token) dataOperands(%0 : memref<i32>)
   return
 }
+
+//===----------------------------------------------------------------------===//
+// Atomic operations (host fallback)
+//===----------------------------------------------------------------------===//
+
+acc.routine @acc_routine_atomic_read func(@atomic_read) seq
+// CHECK-LABEL: func.func @atomic_read
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.load
+func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
+  acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_write func(@atomic_write) seq
+// CHECK-LABEL: func.func @atomic_write
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.store
+func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
+  %val = arith.constant 5.0e-01 : f64
+  acc.atomic.write %addr = %val : memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_update func(@atomic_update) seq
+// CHECK-LABEL: func.func @atomic_update
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_update]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.update %x : memref<f64> {
+    ^bb0(%arg0: f64):
+      %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+      %r = arith.select %c, %a, %arg0 : f64
+      acc.yield %r : f64
+    }
+    acc.terminator
+  }
+  return
+}
+
+acc.routine @acc_routine_atomic_capture func(@atomic_capture) seq
+// CHECK-LABEL: func.func @atomic_capture
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
----------------
razvanlupusoru wrote:

Same here.

https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Mon Jul  6 08:51:52 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 06 Jul 2026 08:51:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4bcf18.7d41f3a1.d1cf7.b111@mx.google.com>


================
@@ -155,3 +155,66 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
   acc.declare_exit token(%token) dataOperands(%0 : memref<i32>)
   return
 }
+
+//===----------------------------------------------------------------------===//
+// Atomic operations (host fallback)
+//===----------------------------------------------------------------------===//
+
+acc.routine @acc_routine_atomic_read func(@atomic_read) seq
+// CHECK-LABEL: func.func @atomic_read
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.load
+func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
+  acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_write func(@atomic_write) seq
+// CHECK-LABEL: func.func @atomic_write
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.store
+func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
+  %val = arith.constant 5.0e-01 : f64
+  acc.atomic.write %addr = %val : memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_update func(@atomic_update) seq
+// CHECK-LABEL: func.func @atomic_update
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
----------------
razvanlupusoru wrote:

Can you please update the CHECK lines to include all of the cloned operations and also include ssa value checks to make sure everything is threaded through correctly?

https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Mon Jul  6 09:47:47 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 06 Jul 2026 09:47:47 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4bdc33.481afadf.134314.f03f@mx.google.com>


Saieiei wrote:

Done. I moved the target `in_reduction` runtime-generation path into `OpenMPIRBuilder`, including the gtid setup, per-item lookup loop, runtime call, and address-space normalization.

I also moved the missing matching `map_entries` check into the `omp.target` verifier and updated translation to use the same matching predicate, so verifier and translation now agree on exact-value and same-defining-op cases. Duplicate matching map entries still produce a clean ambiguity diagnostic.


https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul  6 09:55:24 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Mon, 06 Jul 2026 09:55:24 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4bddfc.481afadf.134314.cfbe@mx.google.com>


================
@@ -524,10 +524,13 @@ raw_ostream &operator<<(raw_ostream &OS, const LoopBase<BlockT, LoopT> &Loop) {
 ///
 
 template <class BlockT, class LoopT> class LoopInfoBase {
-  // BBMap - Mapping of basic blocks to the inner most loop they occur in
-  std::conditional_t<GraphHasNodeNumbers<const BlockT *>, SmallVector<LoopT *>,
-                     DenseMap<const BlockT *, LoopT *>>
-      BBMap;
+  static_assert(GraphHasNodeNumbers<const BlockT *>,
+                "LoopInfo requires GraphTraits<BlockT *>::getNumber (see "
+                "GraphHasNodeNumbers)");
----------------
MaskRay wrote:

I was wrong that MLIR was the only in-tree user. 
`bolt/include/bolt/Core/BinaryLoop.h` has a use case as well... Will split.

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Mon Jul  6 10:47:52 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 06 Jul 2026 10:47:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add non-splat constant
 distribution in SgToLane pass (PR #205575)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205575@github.com>
Message-ID: <6a4bea48.f8b79514.3b441.c9f0@mx.google.com>


================
@@ -180,6 +180,27 @@ gpu.func @arith_constant_non_splat() {
   gpu.return
 }
 
+// With lane_data > 1 each lane owns a lane_data-sized block, so multiple
+// elements are extracted and assembled with vector.from_elements.
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[RES:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
----------------
Jianhui-Li wrote:

The form_elements always form a 1d vector.  What if the constant data is nd? I think it should keep original rank by doing extract_slice/insert_slice. 

https://github.com/llvm/llvm-project/pull/205575

From llvmlistbot at llvm.org  Mon Jul  6 10:50:02 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 06 Jul 2026 10:50:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Move arch-specific topology
 under uArch check (PR #207449)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207449@github.com>
Message-ID: <6a4beaca.c56fb102.11841f.c828@mx.google.com>


https://github.com/Jianhui-Li approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207449

From llvmlistbot at llvm.org  Mon Jul  6 11:56:36 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 06 Jul 2026 11:56:36 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
Message-ID: <llvm.org/llvm/llvm-project/pull/207809@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/207809

This commit removes the "QueryProfileInterface" and "QueryExtensionInterface" as they are superfluous in most cases, with TosaComplianceData.h.inc already providing much of the data we need. Removing these also helps prevent mistakes as observed in: https://github.com/llvm/llvm-project/pull/206687

This commit removes these interfaces and updates the profile conformance to extract supported profile/extension information from TosaComplianceData.h.inc instead.

One factor that isn't duplicated is operations that rely on extensions, but do not declare any data type support. This is a small set of operations, such as tosa.custom, tosa.cond_if, tosa.while_loop and tosa.const_shape, and we can add them to the TosaComplianceData.h.inc file as needed.

>From 2024aac743cadafaf1c7205c3126b1c67c0cb502 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 2 Jul 2026 16:41:57 +0100
Subject: [PATCH] [mlir][tosa] Remove availability interfaces from profile
 conformance

This commit removes the "QueryProfileInterface" and
"QueryExtensionInterface" as they are superflous in most cases, with
TosaComplianceData.h.inc already providing much of the data we need.
Removing these also helps prevent mistakes as observed in:
https://github.com/llvm/llvm-project/pull/206687

This commit removes these interfaces and updates the profile
conformance to extract supported profile/extension information from
TosaComplianceData.h.inc instead.

One factor that isn't duplicated is operations that rely on extensions,
but do not declare any data type support. This is a small set of
operations, such as tosa.custom, tosa.cond_if, tosa.while_loop and
tosa.const_shape, and we can add them to the TosaComplianceData.h.inc
file as needed.

Change-Id: I5b63aa9f00757f31302a255291511d27dd153986
---
 .../mlir/Dialect/Tosa/IR/CMakeLists.txt       |   6 -
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  |  18 +
 .../mlir/Dialect/Tosa/IR/TosaOpBase.td        | 142 +---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h   |   6 -
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  | 429 ----------
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   |   5 +-
 .../mlir/Dialect/Tosa/IR/TosaShapeOps.td      |  14 -
 mlir/lib/Dialect/Tosa/CMakeLists.txt          |   1 -
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |   2 -
 .../Tosa/Transforms/TosaProfileCompliance.cpp |  75 +-
 mlir/test/Dialect/Tosa/availability.mlir      | 749 ------------------
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  42 +-
 .../Dialect/Tosa/profile_all_unsupported.mlir |  22 +-
 .../Tosa/profile_pro_fp_unsupported.mlir      |  31 -
 mlir/test/lib/Dialect/Tosa/CMakeLists.txt     |   1 -
 .../lib/Dialect/Tosa/TestAvailability.cpp     |  80 --
 mlir/tools/mlir-opt/mlir-opt.cpp              |   2 -
 mlir/tools/mlir-tblgen/CMakeLists.txt         |   1 -
 mlir/tools/mlir-tblgen/TosaUtilsGen.cpp       | 215 -----
 .../llvm-project-overlay/mlir/BUILD.bazel     |  13 -
 20 files changed, 95 insertions(+), 1759 deletions(-)
 delete mode 100644 mlir/test/Dialect/Tosa/availability.mlir
 delete mode 100644 mlir/test/lib/Dialect/Tosa/TestAvailability.cpp
 delete mode 100644 mlir/tools/mlir-tblgen/TosaUtilsGen.cpp

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/CMakeLists.txt b/mlir/include/mlir/Dialect/Tosa/IR/CMakeLists.txt
index 5d78f76d8810a..c9300bfe42d9b 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/CMakeLists.txt
+++ b/mlir/include/mlir/Dialect/Tosa/IR/CMakeLists.txt
@@ -17,9 +17,3 @@ set(LLVM_TARGET_DEFINITIONS TosaOpBase.td)
 mlir_tablegen(TosaEnums.h.inc -gen-enum-decls)
 mlir_tablegen(TosaEnums.cpp.inc -gen-enum-defs)
 add_mlir_dialect_tablegen_target(MLIRTosaEnumsIncGen)
-
-set(LLVM_TARGET_DEFINITIONS TosaOps.td)
-mlir_tablegen(TosaAvailability.h.inc -gen-avail-interface-decls)
-mlir_tablegen(TosaAvailability.cpp.inc -gen-avail-interface-defs)
-mlir_tablegen(TosaOpAvailabilityImpl.inc -gen-tosa-avail-impls)
-add_mlir_dialect_tablegen_target(MLIRTosaAvailabilityIncGen)
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..9a63d6fb5ea3f 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -550,6 +550,8 @@ profileComplianceMap = {
       {{Profile::pro_fp},
        {{{fp16T}, SpecificationVersion::V_1_0},
         {{fp32T}, SpecificationVersion::V_1_0}}}}},
+    {"tosa.custom", {{{Profile::pro_int, Profile::pro_fp}, {}, anyOf}}},
+    {"tosa.const_shape", {{{Profile::pro_int, Profile::pro_fp}, {}, anyOf}}},
 };
 
 extensionComplianceMap = {
@@ -1330,6 +1332,22 @@ extensionComplianceMap = {
        {{{i8T}, SpecificationVersion::V_1_0},
         {{fp16T}, SpecificationVersion::V_1_0},
         {{fp32T}, SpecificationVersion::V_1_0}}}}},
+    {"tosa.add_shape", {{{Extension::shape}, {}}}},
+    {"tosa.assert_equal_shape", {{{Extension::shape}, {}}}},
+    {"tosa.concat_shape", {{{Extension::shape}, {}}}},
+    {"tosa.div_ceil_shape", {{{Extension::shape}, {}}}},
+    {"tosa.div_floor_shape", {{{Extension::shape}, {}}}},
+    {"tosa.exp2_shape", {{{Extension::shape}, {}}}},
+    {"tosa.log2_ceil_shape", {{{Extension::shape}, {}}}},
+    {"tosa.log2_floor_shape", {{{Extension::shape}, {}}}},
+    {"tosa.max_shape", {{{Extension::shape}, {}}}},
+    {"tosa.min_shape", {{{Extension::shape}, {}}}},
+    {"tosa.mod_shape", {{{Extension::shape}, {}}}},
+    {"tosa.mul_shape", {{{Extension::shape}, {}}}},
+    {"tosa.slice_shape", {{{Extension::shape}, {}}}},
+    {"tosa.sub_shape", {{{Extension::shape}, {}}}},
+    {"tosa.while_loop", {{{Extension::controlflow}, {}}}},
+    {"tosa.cond_if", {{{Extension::controlflow}, {}}}},
     {"tosa.dim",
      {{{Extension::shape},
        {{{boolT}, SpecificationVersion::V_1_1_DRAFT},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
index d527bde997a7d..63acbb09b7a04 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOpBase.td
@@ -339,111 +339,6 @@ def Tosa_ExtensionAttr
 def Tosa_ExtensionArrayAttr
     : TypedArrayAttrBase<Tosa_ExtensionAttr, "TOSA extension array attribute">;
 
-// The base class for defining op availability dimensions.
-class Availability {
-  // The following are fields for controlling the generated C++ OpInterface.
-
-  // The namespace for the generated C++ OpInterface subclass.
-  string cppNamespace = "::mlir::tosa";
-
-  // The name for the generated C++ OpInterface subclass.
-  string interfaceName = ?;
-
-  // The description for the generated C++ OpInterface subclass.
-  string interfaceDescription = "";
-
-  // The query function's return type in the generated C++ OpInterface subclass.
-  string queryFnRetType = ?;
-
-  // The query function's name in the generated C++ OpInterface subclass.
-  string queryFnName = ?;
-
-  // The logic for merging two availability requirements.
-  code mergeAction = ?;
-
-  // The initializer for the final availability requirement.
-  string initializer = ?;
-
-  // An availability instance's type.
-  string instanceType = ?;
-
-  // The following are fields for a concrete availability instance.
-
-  // The code for preparing a concrete instance. This should be C++ statements
-  // and will be generated before the `mergeAction` logic.
-  code instancePreparation = "";
-
-  // The availability requirement carried by a concrete instance.
-  string instance = ?;
-}
-
-
-class Profile<list<I32EnumAttrCase> profiles> : Availability {
-  let interfaceName = "QueryProfileInterface";
-  let interfaceDescription = [{
-    Querying interface for the supported set of Tosa profile.
-
-    This interface provides a `getProfiles()` method to query
-    the supported set of Tosa profile. The returned value is a
-    list of `mlir::Tosa::Profile` enum number.
-  }];
-
-  let queryFnRetType = "::llvm::SmallVector<::llvm::ArrayRef<"
-                          "::mlir::tosa::Profile>, 1>";
-  let queryFnName = "getProfiles";
-
-  let mergeAction = !if(
-      !empty(profiles), "", "$overall.emplace_back($instance)");
-
-  let initializer = "{}";
-
-  let instanceType = "::llvm::ArrayRef<::mlir::tosa::Profile>";
-
-  // Pack all profiles as a static array and get its reference.
-  let instancePreparation = !if(!empty(profiles), "",
-    "static const ::mlir::tosa::Profile profs[] = {" #
-    !interleave(!foreach(prof, profiles,
-                         "::mlir::tosa::Profile::" # prof.symbol), ", ") #
-    "}; " #
-    "ArrayRef<::mlir::tosa::Profile> " #
-      "ref(profs, std::size(profs));");
-
-  let instance = "ref";
-}
-
-class Extension<list<I32EnumAttrCase> extensions> : Availability {
-  let interfaceName = "QueryExtensionInterface";
-  let interfaceDescription = [{
-    Querying interface for the supported set of TOSA extension.
-
-    This interface provides a `getExtensions()` method to query
-    the supported set of Tosa extension. The returned value is a
-    list of `mlir::Tosa::Extension` enum number.
-  }];
-
-  let queryFnRetType = "::llvm::SmallVector<::llvm::ArrayRef<"
-                          "::mlir::tosa::Extension>, 1>";
-  let queryFnName = "getExtensions";
-
-  let mergeAction = !if(
-      !empty(extensions), "", "$overall.emplace_back($instance)");
-
-  let initializer = "{}";
-
-  let instanceType = "::llvm::ArrayRef<::mlir::tosa::Extension>";
-
-  // Pack all extensions as a static array and get its reference.
-  let instancePreparation = !if(!empty(extensions), "",
-    "static const ::mlir::tosa::Extension exts[] = {" #
-    !interleave(!foreach(ext, extensions,
-                         "::mlir::tosa::Extension::" # ext.symbol), ", ") #
-    "}; " #
-    "ArrayRef<::mlir::tosa::Extension> " #
-      "ref(exts, std::size(exts));");
-
-  let instance = "ref";
-}
-
 //===----------------------------------------------------------------------===//
 // TOSA Levels
 //===----------------------------------------------------------------------===//
@@ -546,30 +441,6 @@ def Tosa_BlockShapeAttr
   }];
 }
 
-//===----------------------------------------------------------------------===//
-// TOSA Interfaces.
-//===----------------------------------------------------------------------===//
-
-def QueryProfileInterface : OpInterface<"QueryProfileInterface"> {
-  let cppNamespace = "::mlir::tosa";
-  let methods = [InterfaceMethod<
-    "get supported profiles",
-    "::llvm::SmallVector<::llvm::ArrayRef<::mlir::tosa::Profile>, 1>",
-    "getProfiles">];
-}
-
-def QueryExtensionInterface : OpInterface<"QueryExtensionInterface"> {
-  let cppNamespace = "::mlir::tosa";
-  let methods = [InterfaceMethod<
-    "get supported extensions",
-    "::llvm::SmallVector<::llvm::ArrayRef<::mlir::tosa::Extension>, 1>",
-    "getExtensions">];
-}
-
-//===----------------------------------------------------------------------===//
-// TOSA Operator Trait.
-//===----------------------------------------------------------------------===//
-
 // Permits broadcasting. Elementwise trait is too strict.
 def TosaElementwiseOperator : NativeOpTrait<"TosaElementwiseOperator"> {
   let cppNamespace = "mlir::OpTrait::tosa";
@@ -589,18 +460,7 @@ def TosaResolvableShapeOperands : NativeOpTrait<"TosaResolvableShapeOperands"> {
 
 class Tosa_Op<string mnemonic, list<Trait> traits = []> :
     Op<Tosa_Dialect, mnemonic, !listconcat(traits, [TosaOpInterface,
-    DeclareOpInterfaceMethods<QueryProfileInterface>,
-    DeclareOpInterfaceMethods<QueryExtensionInterface>,
-    TosaResolvableShapeOperands])> {
-
-  // Default availability specification.
-  list<Availability> availability = [
-    Profile<[]>,
-    Extension<[]>];
-
-  // When not set, manual implementation of these methods is required.
-  bit autogenAvailability = 1;
-}
+    TosaResolvableShapeOperands])> {}
 
 class Tosa_ElementwiseOp<string mnemonic, list<Trait> traits = []> :
     Tosa_Op<mnemonic, !listconcat(traits, [
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 66c3753899124..69ccfa85aad1f 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -33,12 +33,6 @@
 #include "mlir/Dialect/Tosa/IR/TosaOpsDialect.h.inc"
 #include "mlir/Transforms/DialectConversion.h"
 
-//===----------------------------------------------------------------------===//
-// TOSA operation validation includes.
-//===----------------------------------------------------------------------===//
-
-#include "mlir/Dialect/Tosa/IR/TosaAvailability.h.inc"
-
 namespace mlir {
 class PatternRewriter;
 
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..9aa636fe54c33 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -54,11 +54,6 @@ def Tosa_ArgMaxOp : Tosa_InferShapedTypeOp<"argmax", [NoMemoryEffect]> {
     Tosa_Tensor: $output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
@@ -101,11 +96,6 @@ def Tosa_AvgPool2dOp : Tosa_InferShapedTypeOp<"avg_pool2d", [NoMemoryEffect]> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let builders = [Tosa_AvgPool2dOpQuantInfoBuilder];
 
   let extraClassDeclaration = [{
@@ -146,12 +136,6 @@ def Tosa_AvgPool2dAdaptiveOp
 
   let results = (outs Tosa_Tensor4D:$output);
 
-  list<Availability> availability =
-      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-       Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
-  ];
-
   let builders = [Tosa_AvgPool2dAdaptiveOpQuantInfoBuilder];
 
   let extraClassDeclaration = [{
@@ -201,11 +185,6 @@ def Tosa_Conv2DOp : Tosa_ConvOp<"conv2d"> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getInputZeroPoint();
     FailureOr<int64_t> getWeightZeroPoint();
@@ -250,11 +229,6 @@ def Tosa_Conv2DBlockScaledOp : Tosa_InferShapedTypeOp<"conv2d_block_scaled", [No
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_MXFP_CONV]>,
-  ];
-
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
 }
@@ -291,11 +265,6 @@ def Tosa_Conv3DOp : Tosa_ConvOp<"conv3d"> {
     Tosa_Tensor5D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getInputZeroPoint();
     FailureOr<int64_t> getWeightZeroPoint();
@@ -340,11 +309,6 @@ def Tosa_DepthwiseConv2DOp : Tosa_ConvOp<"depthwise_conv2d"> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getInputZeroPoint();
     FailureOr<int64_t> getWeightZeroPoint();
@@ -395,11 +359,6 @@ def Tosa_FFT2dOp : Tosa_InferShapedTypeOp<"fft2d", [
     Tosa_Tensor3D:$output_imag
   );
 
-  list<Availability> availability = [
-    Profile<[]>,
-    Extension<[Tosa_EXT_FFT]>,
-  ];
-
   let assemblyFormat = [{
     $input_real `,` $input_imag attr-dict `:` `(` type($input_real) `,`
     type($input_imag) `)` `->` `(` type($output_real) `,` type($output_imag) `)`
@@ -435,11 +394,6 @@ def Tosa_MatMulOp : Tosa_InferShapedTypeOp<"matmul", [NoMemoryEffect]> {
     Tosa_Tensor3D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getAZeroPoint();
     FailureOr<int64_t> getBZeroPoint();
@@ -472,12 +426,6 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
 
   let results = (outs Tosa_Tensor3D:$output);
 
-  list<Availability> availability =
-      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-       Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getAZeroPoint();
     FailureOr<int64_t> getBZeroPoint();
@@ -522,11 +470,6 @@ def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled",
 
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
-
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_MXFP]>
-  ];
 }
 
 //===----------------------------------------------------------------------===//
@@ -555,11 +498,6 @@ def Tosa_MaxPool2dOp : Tosa_InferShapedTypeOp<"max_pool2d", [Pure]> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
@@ -590,12 +528,6 @@ def Tosa_MaxPool2dAdaptiveOp
 
   let results = (outs Tosa_Tensor4D:$output);
 
-  list<Availability> availability =
-      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-       Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
-  ];
-
   let hasVerifier = 1;
   let hasCanonicalizer = 1;
   let hasCustomAssemblyFormat = 1;
@@ -637,11 +569,6 @@ def Tosa_RFFT2dOp : Tosa_InferShapedTypeOp<"rfft2d", [
     Tosa_Tensor3D:$output_imag
   );
 
-  list<Availability> availability = [
-    Profile<[]>,
-    Extension<[Tosa_EXT_FFT]>,
-  ];
-
   let assemblyFormat = [{
     $input_real attr-dict `:` `(` type($input_real) `)` `->` `(` type($output_real) `,` type($output_imag) `)`
   }];
@@ -684,11 +611,6 @@ def Tosa_TransposeConv2DOp : Tosa_ConvOp<"transpose_conv2d"> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getInputZeroPoint();
     FailureOr<int64_t> getWeightZeroPoint();
@@ -729,11 +651,6 @@ def Tosa_ClampOp : Tosa_ElementwiseUnaryOp<"clamp", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_BF16]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
@@ -760,11 +677,6 @@ def Tosa_ErfOp : Tosa_ElementwiseUnaryOp<"erf", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -792,11 +704,6 @@ def Tosa_SigmoidOp : Tosa_ElementwiseUnaryOp<"sigmoid", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -823,11 +730,6 @@ def Tosa_TanhOp : Tosa_ElementwiseUnaryOp<"tanh", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -870,11 +772,6 @@ def Tosa_AddOp : Tosa_ElementwiseOp<"add", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -906,11 +803,6 @@ def Tosa_ArithmeticRightShiftOp : Tosa_ElementwiseOp<"arithmetic_right_shift",
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -937,11 +829,6 @@ def Tosa_BitwiseAndOp : Tosa_ElementwiseOp<"bitwise_and", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -968,11 +855,6 @@ def Tosa_BitwiseOrOp : Tosa_ElementwiseOp<"bitwise_or", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -999,11 +881,6 @@ def Tosa_BitwiseXorOp : Tosa_ElementwiseOp<"bitwise_xor", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1032,11 +909,6 @@ def Tosa_IntDivOp : Tosa_ElementwiseOp<"intdiv", [SameOperandsAndResultElementTy
     Tosa_Int32Or64Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -1065,11 +937,6 @@ def Tosa_LogicalAndOp : Tosa_ElementwiseOp<"logical_and", [
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1098,11 +965,6 @@ def Tosa_LogicalLeftShiftOp : Tosa_ElementwiseOp<"logical_left_shift",
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1131,11 +993,6 @@ def Tosa_LogicalRightShiftOp : Tosa_ElementwiseOp<"logical_right_shift",
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1162,11 +1019,6 @@ def Tosa_LogicalOrOp : Tosa_ElementwiseOp<"logical_or", [
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1193,11 +1045,6 @@ def Tosa_LogicalXorOp : Tosa_ElementwiseOp<"logical_xor", [
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1224,11 +1071,6 @@ def Tosa_MaximumOp : Tosa_ElementwiseOp<"maximum", [
   let results = (outs
     Tosa_Tensor:$output
   );
-
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
   let hasCustomAssemblyFormat = 1;
 }
 
@@ -1256,11 +1098,6 @@ def Tosa_MinimumOp : Tosa_ElementwiseOp<"minimum", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCustomAssemblyFormat = 1;
 }
 
@@ -1293,11 +1130,6 @@ def Tosa_MulOp : Tosa_Op<"mul", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -1329,11 +1161,6 @@ def Tosa_PowOp : Tosa_ElementwiseOp<"pow", [SameOperandsAndResultElementType]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1360,11 +1187,6 @@ def Tosa_SubOp : Tosa_ElementwiseOp<"sub", [SameOperandsAndResultElementType]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -1405,11 +1227,6 @@ def Tosa_TableOp : Tosa_InferShapedTypeOp<"table", [NoMemoryEffect]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT16]>,
-  ];
-
   let assemblyFormat = [{
     $input1 `,` $table attr-dict `:` `(` type($input1) `,` type($table) `)` `->` type($output)
   }];
@@ -1451,11 +1268,6 @@ def Tosa_AbsOp : Tosa_ElementwiseUnaryOp<"abs", [NoMemoryEffect]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -1479,11 +1291,6 @@ def Tosa_BitwiseNotOp : Tosa_ElementwiseUnaryOp<"bitwise_not", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1505,11 +1312,6 @@ def Tosa_CeilOp : Tosa_ElementwiseUnaryOp<"ceil", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1531,11 +1333,6 @@ def Tosa_ClzOp : Tosa_ElementwiseUnaryOp<"clz", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1557,11 +1354,6 @@ def Tosa_CosOp : Tosa_ElementwiseUnaryOp<"cos", [Pure]> {
     Tosa_FloatTensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1583,11 +1375,6 @@ def Tosa_ExpOp : Tosa_ElementwiseUnaryOp<"exp", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1609,11 +1396,6 @@ def Tosa_FloorOp : Tosa_ElementwiseUnaryOp<"floor", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1635,11 +1417,6 @@ def Tosa_LogOp : Tosa_ElementwiseUnaryOp<"log", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1661,11 +1438,6 @@ def Tosa_LogicalNotOp : Tosa_ElementwiseUnaryOp<"logical_not", [Pure]> {
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1694,11 +1466,6 @@ def Tosa_NegateOp : Tosa_InferShapedTypeOp<"negate", [
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let builders = [Tosa_NegateOpQuantInfoBuilder];
 
   let extraClassDeclaration = [{
@@ -1734,11 +1501,6 @@ def Tosa_ReciprocalOp : Tosa_ElementwiseUnaryOp<"reciprocal", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let extraClassDeclaration = [{
     /// Return the reciprocal result on the operand.
     static inline APFloat calcOneElement(const APFloat &operand) {
@@ -1772,11 +1534,6 @@ def Tosa_RsqrtOp : Tosa_ElementwiseUnaryOp<"rsqrt", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1798,11 +1555,6 @@ def Tosa_SinOp : Tosa_ElementwiseUnaryOp<"sin", [Pure]> {
     Tosa_FloatTensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -1830,11 +1582,6 @@ def Tosa_SelectOp : Tosa_ElementwiseOp<"select", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCanonicalizeMethod = 1;
   let hasFolder = 1;
   let hasVerifier = 1;
@@ -1881,11 +1628,6 @@ def Tosa_EqualOp : Tosa_ElementwiseOp<"equal", [
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let extraClassDeclaration = [{
     /// Returns when two result types are compatible for this op; method used by
     /// InferTypeOpInterface.
@@ -1917,11 +1659,6 @@ def Tosa_GreaterOp : Tosa_ElementwiseOp<"greater", [SameOperandsElementType,
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -1947,11 +1684,6 @@ def Tosa_GreaterEqualOp : Tosa_ElementwiseOp<"greater_equal",
     Tosa_I1Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
 
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
@@ -1980,11 +1712,6 @@ def Tosa_ReduceAllOp : Tosa_InferTensorTypeOp<"reduce_all", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2021,11 +1748,6 @@ def Tosa_ReduceAnyOp : Tosa_InferTensorTypeOp<"reduce_any", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2063,11 +1785,6 @@ def Tosa_ReduceMaxOp : Tosa_InferTensorTypeOp<"reduce_max", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2105,11 +1822,6 @@ def Tosa_ReduceMinOp : Tosa_InferTensorTypeOp<"reduce_min", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2149,11 +1861,6 @@ def Tosa_ReduceProductOp : Tosa_InferTensorTypeOp<"reduce_product", [NoMemoryEff
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2193,11 +1900,6 @@ def Tosa_ReduceSumOp : Tosa_InferTensorTypeOp<"reduce_sum", [NoMemoryEffect]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2239,11 +1941,6 @@ def Tosa_ConcatOp : Tosa_InferTensorTypeOp<"concat", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasVerifier = 1;
 
@@ -2294,11 +1991,6 @@ def Tosa_PadOp : Tosa_InferShapedTypeOp<"pad", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let builders = [Tosa_PadOpQuantInfoBuilder];
 
   let hasFolder = 1;
@@ -2332,11 +2024,6 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let extraClassDeclaration = [{
     /// Returns true when two result types are compatible for this op;
     /// Method used by InferTypeOpInterface.
@@ -2369,12 +2056,6 @@ def Tosa_ReshapeBlockScaledOp
 
   let results = (outs Variadic<Tosa_Tensor>:$output);
 
-  list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-                                     // NOTE: Validation of extensions is
-                                     // disabled for this op
-                                     Extension<[]>,
-  ];
-
   let extraClassDeclaration = [{
     /// Returns true when two result types are compatible for this op;
     /// Method used by InferTypeOpInterface.
@@ -2412,11 +2093,6 @@ def Tosa_ReverseOp: Tosa_Op<"reverse", [
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 
@@ -2445,11 +2121,6 @@ def Tosa_SliceOp : Tosa_InferShapedTypeOp<"slice", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasFolder = 1;
   let hasVerifier = 1;
@@ -2476,11 +2147,6 @@ def Tosa_TileOp : Tosa_InferShapedTypeOp<"tile", [Pure]> {
     Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let extraClassDeclaration = [{
     LogicalResult getConstantMultiples(llvm::SmallVector<int64_t> &multiples);
   }];
@@ -2517,11 +2183,6 @@ def Tosa_TransposeOp : Tosa_InferShapedTypeOp<"transpose",
     outs Tosa_TensorAtLeast1D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasFolder = 1;
   let hasVerifier = 1;
@@ -2559,11 +2220,6 @@ def Tosa_GatherOp : Tosa_InferShapedTypeOp<"gather", [NoMemoryEffect]> {
     Tosa_Tensor3D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasVerifier = 1;
 
   let assemblyFormat =
@@ -2593,12 +2249,6 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
   let results = (outs Tosa_Tensor3D:$output);
 
-  list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-                                     Extension<[Tosa_EXT_FP8E4M3,
-                                                Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
-  ];
-
   let hasCanonicalizer = 1;
   let hasVerifier = 1;
 
@@ -2637,12 +2287,6 @@ def Tosa_RowGatherBlockScaledOp
 
   let results = (outs Variadic<Tosa_Tensor3D>:$output);
 
-  list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-                                     Extension<[Tosa_EXT_FP8E4M3,
-                                                Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
-  ];
-
   let hasVerifier = 1;
 
   let assemblyFormat =
@@ -2679,11 +2323,6 @@ def Tosa_ScatterOp : Tosa_InferShapedTypeOp<"scatter", [NoMemoryEffect]> {
     Tosa_Tensor3D:$values_out
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let hasVerifier = 1;
 
   let assemblyFormat =
@@ -2739,11 +2378,6 @@ def Tosa_ResizeOp : Tosa_InferShapedTypeOp<"resize", [Pure]> {
     Tosa_Tensor4D:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT16, Tosa_EXT_BF16]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
@@ -2774,14 +2408,6 @@ def Tosa_CastOp: Tosa_Op<"cast", [Pure, SameOperandsAndResultShape,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability =
-      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-       Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
-                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
-                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 
   let hasFolder = 1;
@@ -2815,11 +2441,6 @@ def Tosa_CastFromBlockScaledOp: Tosa_InferShapedTypeOp<"cast_from_block_scaled",
     Tosa_TensorAtLeast1D: $output_data
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_MXFP]>,
-  ];
-
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
 }
@@ -2851,11 +2472,6 @@ def Tosa_CastToBlockScaledOp : Tosa_InferShapedTypeOp<"cast_to_block_scaled", [P
     Tosa_MXFPScaleTensorAtLeast1D:$output_scale
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_BF16, Tosa_EXT_MXFP]>
-  ];
-
   let hasCanonicalizer = 1;
   let hasVerifier = 1;
   let hasCustomAssemblyFormat = 1;
@@ -2922,11 +2538,6 @@ def Tosa_RescaleOp : Tosa_InferShapedTypeOp<"rescale", [NoMemoryEffect]> {
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT]>,
-    Extension<[Tosa_EXT_INT16]>,
-  ];
-
   let extraClassDeclaration = [{
     FailureOr<int64_t> getInputZeroPoint();
     FailureOr<int64_t> getOutputZeroPoint();
@@ -2970,11 +2581,6 @@ def Tosa_ConstOp : Tosa_Op<"const", [ConstantLike, Pure,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
-  ];
-
   let hasFolder = 1;
   let hasVerifier = 1;
 }
@@ -2998,11 +2604,6 @@ def Tosa_IdentityOp: Tosa_Op<"identity", [Pure,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -3057,11 +2658,6 @@ def Tosa_CustomOp : Tosa_Op<"custom"> {
     Variadic<Tosa_Tensor>:$output_list
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let assemblyFormat = "operands attr-dict `:` functional-type(operands, results)";
 }
 
@@ -3092,11 +2688,6 @@ def Tosa_IfOp : Tosa_Op<"cond_if",
     Variadic<Tosa_Tensor>:$output_list
   );
 
-  list<Availability> availability = [
-    Profile<[]>,
-    Extension<[Tosa_EXT_CONTROLFLOW]>,
-  ];
-
   let regions = (region
     SizedRegion<1>:$then_graph,
     SizedRegion<1>:$else_graph
@@ -3131,11 +2722,6 @@ def Tosa_WhileOp : Tosa_Op<"while_loop", [
     Variadic<Tosa_Tensor>:$output_list
   );
 
-  list<Availability> availability = [
-    Profile<[]>,
-    Extension<[Tosa_EXT_CONTROLFLOW]>,
-  ];
-
   let regions = (region
     SizedRegion<1>:$cond_graph,
     SizedRegion<1>:$body_graph
@@ -3166,11 +2752,6 @@ def Tosa_VariableOp : Tosa_Op<"variable", [Symbol]> {
     OptionalAttr<AnyAttr>:$initial_value
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_VARIABLE]>,
-  ];
-
   let hasCustomAssemblyFormat = 1;
 
   let assemblyFormat = [{
@@ -3203,11 +2784,6 @@ def Tosa_VariableWriteOp : Tosa_Op<"variable_write", []> {
     Tosa_Tensor:$input1
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_VARIABLE]>,
-  ];
-
   let assemblyFormat = [{
     $name attr-dict `,` $input1 `:` type($input1)
   }];
@@ -3233,11 +2809,6 @@ def Tosa_VariableReadOp : Tosa_Op<"variable_read", []> {
     Tosa_Tensor:$output1
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_VARIABLE]>,
-  ];
-
   let assemblyFormat = [{
     $name attr-dict `:` type($output1)
   }];
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 34ccc675c8243..faff61c58a612 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -141,9 +141,8 @@ class TosaProfileCompliance {
   LogicalResult checkInvalid(Operation *op);
 
   template <typename T>
-  LogicalResult checkProfileOrExtension(
-      Operation *op, const tosa::TargetEnv &targetEnv,
-      const SmallVector<ArrayRef<T>> &specDefinedProfileSet);
+  LogicalResult checkProfileOrExtension(Operation *op,
+                                        const tosa::TargetEnv &targetEnv);
 
   bool isSameTypeInfo(TypeInfo a, TypeInfo b) {
     return a.typeID == b.typeID && a.bitWidth == b.bitWidth &&
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaShapeOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaShapeOps.td
index be4d31a4372c8..1bbfdc5776f2e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaShapeOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaShapeOps.td
@@ -30,10 +30,6 @@ def TosaShapeOperator : NativeOpTrait<"TosaShapeOperator"> {
 
 class Tosa_ShapeOp<string mnemonic, list<Trait> traits = []>
     : Tosa_Op<mnemonic, !listconcat(traits, [TosaShapeOperator])> {
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_SHAPE]>,
-  ];
 
   let assemblyFormat =
       "operands attr-dict `:` functional-type(operands, results)";
@@ -114,11 +110,6 @@ def Tosa_ConstShapeOp : Tosa_ShapeOp<"const_shape", [ConstantLike, Pure]> {
 
   let results = (outs Tosa_Shape:$output);
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[]>,
-  ];
-
   let hasVerifier = 1;
   let hasFolder = 1;
 }
@@ -140,11 +131,6 @@ def Tosa_DimOp : Tosa_ShapeOp<"dim", [Pure]> {
 
   let results = (outs Tosa_Shape:$output);
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_SHAPE, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
-  ];
-
   let hasVerifier = 1;
 
   let hasFolder = 1;
diff --git a/mlir/lib/Dialect/Tosa/CMakeLists.txt b/mlir/lib/Dialect/Tosa/CMakeLists.txt
index a95906aa8352e..6404ca8fc5a56 100644
--- a/mlir/lib/Dialect/Tosa/CMakeLists.txt
+++ b/mlir/lib/Dialect/Tosa/CMakeLists.txt
@@ -14,7 +14,6 @@ add_mlir_dialect_library(MLIRTosaDialect
   MLIRTosaOpsIncGen
   MLIRTosaInterfacesIncGen
   MLIRTosaEnumsIncGen
-  MLIRTosaAvailabilityIncGen
   MLIRShardingInterfaceIncGen
 
   LINK_LIBS PUBLIC
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..03545bd54e75a 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -43,10 +43,8 @@ using namespace mlir::tosa;
 // Tosa dialect interface includes.
 //===----------------------------------------------------------------------===//
 
-#include "mlir/Dialect/Tosa/IR/TosaAvailability.cpp.inc"
 #include "mlir/Dialect/Tosa/IR/TosaEnums.cpp.inc"
 #include "mlir/Dialect/Tosa/IR/TosaInterfaces.cpp.inc"
-#include "mlir/Dialect/Tosa/IR/TosaOpAvailabilityImpl.inc"
 
 namespace {
 #include "mlir/Dialect/Tosa/IR/TosaDialectBytecode.cpp.inc"
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 348e4ec0ad44c..820bee1131148 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -460,34 +460,10 @@ TosaProfileCompliance::getOperatorMatchedEntries(Operation *op) {
 
 template <typename T>
 LogicalResult TosaProfileCompliance::checkProfileOrExtension(
-    Operation *op, const tosa::TargetEnv &targetEnv,
-    const SmallVector<ArrayRef<T>> &specRequiredModeSet) {
-
-  // None of profile requirement is set in the specification.
-  if (specRequiredModeSet.size() == 0)
-    return success();
-
+    Operation *op, const tosa::TargetEnv &targetEnv) {
   const auto maybeOpEntries = getOperatorMatchedEntries<T>(op);
-  if (failed(maybeOpEntries)) {
-    // Operators such as control-flow and shape ops do not have an operand type
-    // restriction. When the profile compliance information of operation is not
-    // found, confirm if the target have enabled the profile required from the
-    // specification.
-    int modeCount = 0;
-    for (const auto &cands : specRequiredModeSet) {
-      if (targetEnv.allowsAnyOf(cands))
-        return success();
-      modeCount += cands.size();
-    }
-
-    op->emitOpError() << "illegal: requires"
-                      << (modeCount > 1 ? " any of " : " ") << "["
-                      << llvm::join(stringifyProfile<T>(specRequiredModeSet),
-                                    ", ")
-                      << "] but not enabled in target\n";
-
-    return failure();
-  }
+  if (failed(maybeOpEntries))
+    return success();
 
   const auto opEntries = maybeOpEntries.value();
   if (opEntries.size() == 0) {
@@ -508,22 +484,18 @@ LogicalResult TosaProfileCompliance::checkProfileOrExtension(
   const TosaSpecificationVersion targetVersion{targetEnv.getSpecVersion()};
   const auto isVersionCompatible =
       [&targetVersion](const OpComplianceInfo<T> &info) -> bool {
+    // Assume compatible if there are no versioned type constraints
+    if (info.operandTypeInfoSet.empty())
+      return true;
     const TosaSpecificationVersion complianceVersion{
         info.operandTypeInfoSet.front().second};
     return targetVersion.isBackwardsCompatibleWith(complianceVersion);
   };
 
   for (const auto &info : opEntries) {
-    // Ensure the profile compliance is compatible with the profile knowledge of
-    // the op definition.
-    assert(llvm::all_of(info.mode,
-                        [&specRequiredModeSet](const T &mode) {
-                          return llvm::is_contained(specRequiredModeSet.front(),
-                                                    mode);
-                        }) &&
-           "the profile/extension requirement of the operator should be "
-           "included in the profile compliance information");
-
+    // Type constraints exist, check if they are compatible with the current
+    // target definition, including: profile, extension and specification
+    // version
     if (isModeAllowed(info) && isVersionCompatible(info))
       return success();
   }
@@ -564,21 +536,13 @@ LogicalResult TosaProfileCompliance::checkProfileOrExtension(
 LogicalResult
 TosaProfileCompliance::checkProfile(Operation *op,
                                     const tosa::TargetEnv &targetEnv) {
-  if (auto interface = dyn_cast<tosa::QueryProfileInterface>(op))
-    return checkProfileOrExtension<Profile>(op, targetEnv,
-                                            interface.getProfiles());
-
-  return success();
+  return checkProfileOrExtension<Profile>(op, targetEnv);
 }
 
 LogicalResult
 TosaProfileCompliance::checkExtension(Operation *op,
                                       const tosa::TargetEnv &targetEnv) {
-  if (auto interface = dyn_cast<tosa::QueryExtensionInterface>(op))
-    return checkProfileOrExtension<Extension>(op, targetEnv,
-                                              interface.getExtensions());
-
-  return success();
+  return checkProfileOrExtension<Extension>(op, targetEnv);
 }
 
 LogicalResult TosaProfileCompliance::checkInvalid(Operation *op) {
@@ -651,13 +615,26 @@ SmallVector<OpComplianceInfo<T>> TosaProfileCompliance::findMatchedEntries(
   assert(compInfo.size() != 0 &&
          "profile-based compliance information is empty");
 
-  // Populate the type of profile/extension relevant operands.
+  // If there are no type constraints for the given operation, return the
+  // matched entries directly so required profile/extension information can
+  // be checked
+  SmallVector<OpComplianceInfo<T>> matchedInfos;
+  if (llvm::all_of(compInfo, [&matchedInfos](const OpComplianceInfo<T> &info) {
+        if (info.operandTypeInfoSet.empty()) {
+          matchedInfos.push_back(info);
+          return true;
+        }
+        return false;
+      }))
+    return matchedInfos;
+
+  // Get the type information of the operator we're trying to match
   ProfileInfoDepot depot(op);
   SmallVector<TypeInfo> present = depot.getInfo();
   if (present.size() == 0)
     return {};
 
-  SmallVector<OpComplianceInfo<T>> matchedInfos;
+  // Compare 'present' against known type signatures in the compliance metadata
   for (size_t i = 0; i < compInfo.size(); i++) {
     SmallVector<VersionedTypeInfo> sets = compInfo[i].operandTypeInfoSet;
     for (const auto &set : sets) {
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
deleted file mode 100644
index c358460bbc662..0000000000000
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ /dev/null
@@ -1,749 +0,0 @@
-//--------------------------------------------------------------------------------------------------
-// Test whether the supported profile and extension are attached to the operation properly.
-// The data type of arguments of operation are irrelevant in this test.
-//--------------------------------------------------------------------------------------------------
-
-// RUN: mlir-opt -test-tosa-op-availability %s | FileCheck %s
-
-// -----
-// CHECK-LABEL: argmax
-func.func @test_argmax(%arg0: tensor<14x19xf32>) -> tensor<14xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.argmax %arg0 {axis = 1 : i32} : (tensor<14x19xf32>) -> tensor<14xi32>
-  return %0 : tensor<14xi32>
-}
-
-// -----
-// CHECK-LABEL: avg_pool2d
-func.func @test_avg_pool2d(%arg0: tensor<1x7x7x9xf32>) -> tensor<1x7x7x9xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %output_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.avg_pool2d %arg0, %input_zp, %output_zp {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>} : (tensor<1x7x7x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
-  return %0 : tensor<1x7x7x9xf32>
-}
-
-// -----
-// CHECK-LABEL: avg_pool2d_adaptive
-func.func @test_avg_pool2d_adaptive(%arg0: tensor<1x7x7x9xf32>) -> tensor<1x7x7x9xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %output_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %kernel = tosa.const_shape {values = dense<[2, 2]> : tensor<2xindex>} : () -> !tosa.shape<2>
-  %stride = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
-  %pad = tosa.const_shape {values = dense<[0, 1, 0, 1]> : tensor<4xindex>} : () -> !tosa.shape<4>
-  %0 = tosa.avg_pool2d_adaptive %arg0, %input_zp, %output_zp, %kernel, %stride, %pad {acc_type = f32} : (tensor<1x7x7x9xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x7x7x9xf32>
-  return %0 : tensor<1x7x7x9xf32>
-}
-
-// -----
-// CHECK-LABEL: conv2d
-func.func @test_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<8x1x1x4xf32>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x4xf32>, tensor<8x1x1x4xf32>, tensor<8xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf32>
-  return %0 : tensor<1x4x4x8xf32>
-}
-
-// -----
-// CHECK-LABEL: conv3d
-func.func @test_conv3d(%arg0: tensor<1x4x8x21x17xf32>, %arg1: tensor<34x1x1x1x17xf32>, %arg2: tensor<34xf32>) -> tensor<1x4x8x21x34xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.conv3d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1, 1>, pad = array<i64: 0, 0, 0, 0, 0, 0>, stride = array<i64: 1, 1, 1>} : (tensor<1x4x8x21x17xf32>, tensor<34x1x1x1x17xf32>, tensor<34xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x8x21x34xf32>
-  return %0 : tensor<1x4x8x21x34xf32>
-}
-
-// -----
-// CHECK-LABEL: depthwise_conv2d
-func.func @test_depthwise_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<1x1x4x2xf32>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.depthwise_conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x4x4x4xf32>, tensor<1x1x4x2xf32>, tensor<8xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf32>
-  return %0 : tensor<1x4x4x8xf32>
-}
-
-// -----
-// CHECK-LABEL: fft2d
-func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
-  // CHECK: profiles: [ ]
-  // CHECK: extensions: [ [fft] ]
-  %0, %1 = tosa.fft2d %arg0, %arg1 {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
-  return %0, %1 : tensor<1x4x8xf32>, tensor<1x4x8xf32>
-}
-
-// -----
-// CHECK-LABEL: matmul
-func.func @test_matmul(%arg0: tensor<1x14x19xf32>, %arg1: tensor<1x19x28xf32>, %a_zp: tensor<1xf32>, %b_zp: tensor<1xf32>) -> tensor<1x14x28xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, fp8e4m3, fp8e5m2, bf16] ]
-  %0 = tosa.matmul %arg0, %arg1, %a_zp, %b_zp : (tensor<1x14x19xf32>, tensor<1x19x28xf32>, tensor<1xf32>, tensor<1xf32>)  -> tensor<1x14x28xf32>
-  return %0 : tensor<1x14x28xf32>
-}
-
-// -----
-// CHECK-LABEL: max_pool2d_f32
-func.func @test_max_pool2d_f32(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, fp8e4m3, fp8e5m2, bf16] ]
-  %0 = tosa.max_pool2d %arg0 {kernel = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32>
-  return %0 : tensor<1x32x32x8xf32>
-}
-
-// -----
-// CHECK-LABEL: rfft2d
-func.func @test_rfft2d(%arg0: tensor<13x8x16xf32>) -> (tensor<13x8x9xf32>, tensor<13x8x9xf32>) {
-  // CHECK: profiles: [ ]
-  // CHECK: extensions: [ [fft] ]
-  %0, %1 = tosa.rfft2d %arg0 : (tensor<13x8x16xf32>) -> (tensor<13x8x9xf32>, tensor<13x8x9xf32>)
-  return %0, %1 : tensor<13x8x9xf32>, tensor<13x8x9xf32>
-}
-
-// -----
-// CHECK-LABEL: transpose_conv2d
-func.func @test_transpose_conv2d(%arg0: tensor<1x32x32x8xf32>, %arg1: tensor<16x1x1x8xf32>, %arg2: tensor<16xf32>) -> tensor<1x32x32x16xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.transpose_conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, out_pad = array<i64: 0, 0, 0, 0>, out_shape = array<i64: 1, 32, 32, 16>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xf32>, tensor<16x1x1x8xf32>, tensor<16xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x32x32x16xf32>
-  return %0 : tensor<1x32x32x16xf32>
-}
-
-// -----
-// CHECK-LABEL: clamp
-func.func @test_clamp(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, bf16] ]
-  %0 = tosa.clamp %arg0 {min_val = -3.40282347E+38 : f32, max_val = 3.40282347E+38 : f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: sigmoid
-func.func @test_sigmoid(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.sigmoid %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: tanh
-func.func @test_tanh(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.tanh %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-// -----
-// CHECK-LABEL: erf
-func.func @test_erf(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.erf %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: add
-func.func @test_add(%arg0: tensor<13x21x1xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.add %arg0, %arg1 : (tensor<13x21x1xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: arithmetic_right_shift
-func.func @test_arithmetic_right_shift(%arg0: tensor<13x21x1xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.arithmetic_right_shift %arg0, %arg1 {round = false} : (tensor<13x21x1xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: bitwise_and
-func.func @test_bitwise_and(%arg0: tensor<13x21x3xi32>, %arg1: tensor<13x21x1xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.bitwise_and %arg0, %arg1 : (tensor<13x21x3xi32>, tensor<13x21x1xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: bitwise_or
-func.func @test_bitwise_or(%arg0: tensor<13x21x3xi32>, %arg1: tensor<13x1x3xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.bitwise_or %arg0, %arg1 : (tensor<13x21x3xi32>, tensor<13x1x3xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: bitwise_xor
-func.func @test_bitwise_xor(%arg0: tensor<13x21x1xi32>, %arg1: tensor<13x21x3xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.bitwise_xor %arg0, %arg1 : (tensor<13x21x1xi32>, tensor<13x21x3xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: intdiv
-func.func @test_intdiv(%arg0: tensor<13x21x1xi32>, %arg1: tensor<13x21x3xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.intdiv %arg0, %arg1 : (tensor<13x21x1xi32>, tensor<13x21x3xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: logical_and
-func.func @test_logical_and(%arg0: tensor<13x21x3xi1>, %arg1: tensor<13x21x1xi1>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.logical_and %arg0, %arg1 : (tensor<13x21x3xi1>, tensor<13x21x1xi1>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: logical_left_shift
-func.func @test_logical_left_shift(%arg0: tensor<13x21x3xi32>, %arg1: tensor<13x21x1xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.logical_left_shift %arg0, %arg1 : (tensor<13x21x3xi32>, tensor<13x21x1xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: logical_right_shift
-func.func @test_logical_right_shift(%arg0: tensor<13x21x3xi32>, %arg1: tensor<13x21x1xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.logical_right_shift %arg0, %arg1 : (tensor<13x21x3xi32>, tensor<13x21x1xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: logical_or
-func.func @test_logical_or(%arg0: tensor<13x1x3xi1>, %arg1: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.logical_or %arg0, %arg1 : (tensor<13x1x3xi1>, tensor<13x21x3xi1>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: logical_xor
-func.func @test_logical_xor(%arg0: tensor<13x1x3xi1>, %arg1: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.logical_xor %arg0, %arg1 : (tensor<13x1x3xi1>, tensor<13x21x3xi1>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: maximum
-func.func @test_max(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x1xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.maximum %arg0, %arg1 : (tensor<13x21x3xf32>, tensor<13x21x1xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: minimum
-func.func @test_min(%arg0: tensor<13x21x3xf32>, %arg1: tensor<1x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.minimum %arg0, %arg1 : (tensor<13x21x3xf32>, tensor<1x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: mul
-func.func @test_mul(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
-  %shift = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<1xi8>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: pow
-func.func @test_pow(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x1xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.pow %arg0, %arg1 : (tensor<13x21x3xf32>, tensor<13x21x1xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: sub
-func.func @test_sub(%arg0: tensor<1x21x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.sub %arg0, %arg1 : (tensor<1x21x3xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: table
-func.func @test_table(%arg0: tensor<64xi32>, %arg1: tensor<513x!quant.uniform<i16:f32, 1.0:0>>) -> tensor<64x!quant.uniform<i16:f32, 1.0:0>> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int16] ]
-  %0 = tosa.table %arg0, %arg1 : (tensor<64xi32>, tensor<513x!quant.uniform<i16:f32, 1.000000e+00>>) -> tensor<64x!quant.uniform<i16:f32, 1.000000e+00>>
-  return %0 : tensor<64x!quant.uniform<i16:f32, 1.0:0>>
-}
-
-// -----
-// CHECK-LABEL: abs
-func.func @test_abs(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.abs %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: bitwise_not
-func.func @test_bitwise_not(%arg0: tensor<13x21x1xi32>) -> tensor<13x21x1xi32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.bitwise_not %arg0 : (tensor<13x21x1xi32>) -> tensor<13x21x1xi32>
-  return %0 : tensor<13x21x1xi32>
-}
-
-// -----
-// CHECK-LABEL: ceil
-func.func @test_ceil(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.ceil %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: clz
-func.func @test_clz(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int] ]
-  // CHECK: extensions: [ [int64] ]
-  %0 = tosa.clz %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: cos
-func.func @test_cos(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.cos %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: exp
-func.func @test_exp(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.exp %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: floor
-func.func @test_floor(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.floor %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: log
-func.func @test_log(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.log %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: logical_not
-func.func @test_logical_not(%arg0: tensor<1x21x3xi1>) -> tensor<1x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.logical_not %arg0 : (tensor<1x21x3xi1>) -> tensor<1x21x3xi1>
-  return %0 : tensor<1x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: negate
-func.func @test_negate(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %output_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
-  %0 = tosa.negate %arg0, %input_zp, %output_zp : (tensor<13x21x3xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: reciprocal
-func.func @test_reciprocal(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.reciprocal %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: rsqrt
-func.func @test_rsqrt(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.rsqrt %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: sin
-func.func @test_sin(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.sin %arg0 : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: select
-func.func @test_select(%arg0: tensor<1x1x1xi1>, %arg1: tensor<13x21x3xf32>, %arg2: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.select %arg0, %arg1, %arg2 : (tensor<1x1x1xi1>, tensor<13x21x3xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: equal
-func.func @test_equal(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.equal %arg0, %arg1 : (tensor<13x21x3xf32>, tensor<13x1x3xf32>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: greater
-func.func @test_greater(%arg0: tensor<13x21x1xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.greater %arg0, %arg1 : (tensor<13x21x1xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: greater_equal
-func.func @test_greater_equal(%arg0: tensor<13x1x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<13x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.greater_equal %arg0, %arg1 : (tensor<13x1x3xf32>, tensor<13x21x3xf32>) -> tensor<13x21x3xi1>
-  return %0 : tensor<13x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: reduce_all
-func.func @test_reduce_all(%arg0: tensor<13x21x3xi1>) -> tensor<1x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.reduce_all %arg0 {axis = 0 : i32} : (tensor<13x21x3xi1>) -> tensor<1x21x3xi1>
-  return %0 : tensor<1x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: reduce_any
-func.func @test_reduce_any(%arg0: tensor<13x21x3xi1>) -> tensor<1x21x3xi1> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.reduce_any %arg0 {axis = 0 : i32} : (tensor<13x21x3xi1>) -> tensor<1x21x3xi1>
-  return %0 : tensor<1x21x3xi1>
-}
-
-// -----
-// CHECK-LABEL: reduce_max
-func.func @test_reduce_max(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.reduce_max %arg0 {axis = 0 : i32} : (tensor<13x21x3xf32>) -> tensor<1x21x3xf32>
-  return %0 : tensor<1x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: reduce_min
-func.func @test_reduce_min(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.reduce_min %arg0 {axis = 0 : i32} : (tensor<13x21x3xf32>) -> tensor<1x21x3xf32>
-  return %0 : tensor<1x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: reduce_product
-func.func @test_reduce_product(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16] ]
-  %0 = tosa.reduce_product %arg0 {axis = 0 : i32} : (tensor<13x21x3xf32>) -> tensor<1x21x3xf32>
-  return %0 : tensor<1x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: reduce_sum
-func.func @test_reduce_sum(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [bf16, int64] ]
-  %0 = tosa.reduce_sum %arg0 {axis = 0 : i32} : (tensor<13x21x3xf32>) -> tensor<1x21x3xf32>
-  return %0 : tensor<1x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: concat
-func.func @test_concat(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<26x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
-  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x3xf32>, tensor<13x21x3xf32>) -> tensor<26x21x3xf32>
-  return %0 : tensor<26x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: pad
-func.func @test_pad(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
-  %pad_const = "tosa.const"() {values = dense<3.14> : tensor<1xf32>} : () -> tensor<1xf32>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.pad %arg0, %padding, %pad_const : (tensor<13x21x3xf32>, !tosa.shape<6>, tensor<1xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: reshape
-func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
-  %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xf32>, !tosa.shape<2>) -> tensor<1x819xf32>
-  return %0 : tensor<1x819xf32>
-}
-
-// -----
-// CHECK-LABEL: reverse
-func.func @test_reverse(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.reverse %arg0 {axis = 0 : i32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: slice
-func.func @test_slice(%arg0: tensor<13x21x3xf32>) -> tensor<4x11x1xf32> {
-  %size = tosa.const_shape {values = dense<[4, 11, 1]> : tensor<3xindex>} : () -> !tosa.shape<3>
-  %start = tosa.const_shape {values = dense<[6, 8, 0]> : tensor<3xindex>} : () -> !tosa.shape<3>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %2 = tosa.slice %arg0, %start, %size : (tensor<13x21x3xf32>, !tosa.shape<3>, !tosa.shape<3>) -> tensor<4x11x1xf32>
-  return %2 : tensor<4x11x1xf32>
-}
-
-// -----
-// CHECK-LABEL: tile
-func.func @test_tile(%arg0: tensor<13x21x3xf32>) -> tensor<39x21x6xf32> {
-  %cst = tosa.const_shape { values = dense<[3, 1, 2]> : tensor<3xindex> } : () -> !tosa.shape<3>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.tile %arg0, %cst: (tensor<13x21x3xf32>, !tosa.shape<3>) -> tensor<39x21x6xf32>
-  return %0 : tensor<39x21x6xf32>
-}
-
-// -----
-// CHECK-LABEL: transpose
-func.func @test_transpose(%arg0: tensor<13x21x3xf32>) -> tensor<3x13x21xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %1 = tosa.transpose %arg0 {perms = array<i32: 2, 0, 1>}: (tensor<13x21x3xf32>) -> tensor<3x13x21xf32>
-  return %1 : tensor<3x13x21xf32>
-}
-
-// -----
-// CHECK-LABEL: gather
-func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x26x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.gather %arg0, %arg1 : (tensor<13x21x3xf32>, tensor<13x26xi32>) -> tensor<13x26x3xf32>
-  return %0 : tensor<13x26x3xf32>
-}
-
-// -----
-// CHECK-LABEL: test_row_gather
-func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
-  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
-  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
-  return %0 : tensor<13x52x3xf32>
-}
-
-// -----
-// CHECK-LABEL: row_gather_block_scaled
-func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
-  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, mxfp, int64] ]
-  %0:2 = tosa.row_gather_block_scaled %arg0, %arg1, %arg2, %row_count {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<13x21x32xf4E2M1FN>, tensor<13x21x1xf8E8M0FNU>, tensor<13x26xi32>, tensor<1xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>)
-  return %0#0, %0#1 : tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>
-}
-
-// -----
-// CHECK-LABEL: scatter
-func.func @test_scatter(%arg0: tensor<13x28x3xf32>, %arg1: tensor<13x26xi32>, %arg2: tensor<13x26x3xf32>) -> tensor<13x28x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.scatter %arg0, %arg1, %arg2 : (tensor<13x28x3xf32>, tensor<13x26xi32>, tensor<13x26x3xf32>) -> tensor<13x28x3xf32>
-  return %0 : tensor<13x28x3xf32>
-}
-
-// -----
-// CHECK-LABEL: resize
-func.func @test_resize(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x64x64x8xf32> {
-  %scale = tosa.const_shape { values = dense<[4, 2, 4, 2]> : tensor<4xindex> } : () -> !tosa.shape<4>
-  %offset = tosa.const_shape { values = dense<[-1, -1]> : tensor<2xindex> } : () -> !tosa.shape<2>
-  %border = tosa.const_shape { values = dense<[1, 1]> : tensor<2xindex> } : () -> !tosa.shape<2>
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int16, bf16] ]
-  %1 = tosa.resize %arg0, %scale, %offset, %border {mode = BILINEAR} : (tensor<1x32x32x8xf32>, !tosa.shape<4>, !tosa.shape<2>, !tosa.shape<2>) -> tensor<1x64x64x8xf32>
-  return %1 : tensor<1x64x64x8xf32>
-}
-
-// -----
-// CHECK-LABEL: test_cast
-func.func @test_cast(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
-  %0 = tosa.cast %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xf32>
-  return %0 : tensor<13x21x3xf32>
-}
-
-// -----
-// CHECK-LABEL: test_rescale
-func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439347028732:127>>, %multiplier : tensor<1xi32>, %shift : tensor<1xi8>) -> tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>> {
-  %input_zp = "tosa.const"() {values = dense<127> : tensor<1xi8>} : () -> tensor<1xi8>
-  %output_zp = "tosa.const"() {values = dense<-1> : tensor<1xi8>} : () -> tensor<1xi8>
-  // CHECK: tosa.rescale profiles: [ [pro_int] ]
-  // CHECK: tosa.rescale extensions: [ [int16] ]
-  %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, scale32 = true, per_channel = false, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439347028732:127>>, tensor<1xi32>, tensor<1xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>>
-  return %0 : tensor<13x21x3x!quant.uniform<i8:f32, 0.015655439347028732:-1>>
-}
-
-// -----
-// CHECK-LABEL: test_const
-func.func @test_const(%arg0 : index) -> tensor<4xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64] ]
-    %0 = "tosa.const"() {values = dense<[3, 0, 1, 2]> : tensor<4xi32>} : () -> tensor<4xi32>
-    return %0 : tensor<4xi32>
-}
-
-// -----
-// CHECK-LABEL: identity
-func.func @test_identity(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, int64] ]
-  %0 = tosa.identity %arg0 : (tensor<13x21x3xi32>) -> tensor<13x21x3xi32>
-  return %0 : tensor<13x21x3xi32>
-}
-
-// -----
-// CHECK-LABEL: cond_if
-func.func @test_cond_if(%arg0: tensor<f32>, %arg1: tensor<f32>, %arg2: tensor<i1>) -> tensor<f32> {
-  // CHECK: tosa.cond_if profiles: [ ]
-  // CHECK: tosa.cond_if extensions: [ [controlflow] ]
-  %0 = tosa.cond_if %arg2 : tensor<i1> -> tensor<f32> {
-    %1 = tosa.add %arg0, %arg1 : (tensor<f32>, tensor<f32>) -> tensor<f32>
-    tosa.yield %1 : tensor<f32>
-  } else {
-    %1 = tosa.sub %arg0, %arg1 : (tensor<f32>, tensor<f32>) -> tensor<f32>
-    tosa.yield %1 : tensor<f32>
-  }
-  return %0 : tensor<f32>
-}
-
-// -----
-// CHECK-LABEL: while_loop
-func.func @test_while_loop(%arg0: tensor<10xi32>, %arg1: tensor<i32>) {
-  %0 = "tosa.const"() {values = dense<0> : tensor<i32>} : () -> tensor<i32>
-  // CHECK: profiles: [ ]
-  // CHECK: extensions: [ [controlflow] ]
-  %1:3 = tosa.while_loop (%arg2 = %0, %arg3 = %0, %arg4 = %arg0) : (tensor<i32>, tensor<i32>, tensor<10xi32>) -> (tensor<i32>, tensor<i32>, tensor<10xi32>) {
-    %2 = tosa.greater_equal %arg3, %arg1 : (tensor<i32>, tensor<i32>) -> tensor<i1>
-    %3 = tosa.logical_not %2 : (tensor<i1>) -> tensor<i1>
-    tosa.yield %3 : tensor<i1>
-  } do {
-  ^bb0(%arg2: tensor<i32>, %arg3: tensor<i32>, %arg4: tensor<10xi32>):
-    %2 = "tosa.const"() {values = dense<1> : tensor<i32>} : () -> tensor<i32>
-    %3 = tosa.add %arg3, %2 : (tensor<i32>, tensor<i32>) -> tensor<i32>
-    %7 = tosa.const_shape {values = dense<[1]> : tensor<1xindex>} : () -> !tosa.shape<1>
-    %4 = tosa.reshape %2, %7 : (tensor<i32>, !tosa.shape<1>) -> tensor<1xi32>
-    %5 = tosa.add %arg4, %4 : (tensor<10xi32>, tensor<1xi32>) -> tensor<10xi32>
-    %6 = tosa.add %arg2, %2 : (tensor<i32>, tensor<i32>) -> tensor<i32>
-    tosa.yield %6, %3, %5 : tensor<i32>, tensor<i32>, tensor<10xi32>
-  }
-  return
-}
-
-// -----
-// CHECK-LABEL: custom
-func.func @test_custom(%arg0: tensor<10xi32>) -> tensor<10xi32> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %0 = tosa.custom %arg0 {operator_name="custom_test", domain_name="tosa.mlir_test", implementation_attrs="" } : (tensor<10xi32>) -> (tensor<10xi32>)
-  return %0 : tensor<10xi32>
-}
-
-// -----
-// CHECK-LABEL: const_shape
-func.func @test_const_shape() -> !tosa.shape<4> {
-  // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ ]
-  %cst = tosa.const_shape {values = dense<1> : tensor<4xindex>} : () -> !tosa.shape<4>
-  return %cst : !tosa.shape<4>
-}
-
-// -----
-// CHECK-LABEL: test_cast_from_block_scaled
-func.func @test_cast_from_block_scaled(%arg0: tensor<4x32xf4E2M1FN>, %arg1: tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32> {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16, mxfp] ]
-  %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32>
-  return %0 : tensor<4x32xf32>
-}
-
-// -----
-// CHECK-LABEL: test_cast_to_block_scaled
-func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) {
-  // CHECK: profiles: [ [pro_fp] ]
-  // CHECK: extensions: [ [bf16, mxfp] ]
-  %0:2 = tosa.cast_to_block_scaled %arg0 {block_size = BLOCK_SIZE_32, stochastic_round = false} : (tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>)
-  return %0#0, %0#1 : tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>
-}
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..7a973e9efb3f9 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -393,7 +393,7 @@ func.func @test_cast_bf16_i32(%arg0: tensor<13x21x3xbf16>) -> tensor<13x21x3xi32
 
 // -----
 func.func @test_cond_if(%arg0: tensor<f32>, %arg1: tensor<f32>, %arg2: tensor<i1>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cond_if' op illegal: requires [controlflow] but not enabled in target}}
+  // expected-error at +1 {{'tosa.cond_if' op illegal: requires any of [controlflow] profiles/extensions to be specified in the target environment}}
   %0 = tosa.cond_if %arg2 : tensor<i1> -> tensor<f32> {
     %1 = tosa.add %arg0, %arg1 : (tensor<f32>, tensor<f32>) -> tensor<f32>
     tosa.yield %1 : tensor<f32>
@@ -407,7 +407,7 @@ func.func @test_cond_if(%arg0: tensor<f32>, %arg1: tensor<f32>, %arg2: tensor<i1
 // -----
 func.func @test_while_loop(%arg0: tensor<10xi32>, %arg1: tensor<i32>) {
   %0 = "tosa.const"() {values = dense<0> : tensor<i32>} : () -> tensor<i32>
-  // expected-error at +1 {{'tosa.while_loop' op illegal: requires [controlflow] but not enabled in target}}
+  // expected-error at +1 {{'tosa.while_loop' op illegal: requires any of [controlflow] profiles/extensions to be specified in the target environment}}
   %1:3 = tosa.while_loop (%arg2 = %0, %arg3 = %0, %arg4 = %arg0) : (tensor<i32>, tensor<i32>, tensor<10xi32>) -> (tensor<i32>, tensor<i32>, tensor<10xi32>) {
     %2 = tosa.greater_equal %arg3, %arg1 : (tensor<i32>, tensor<i32>) -> tensor<i1>
     %3 = tosa.logical_not %2 : (tensor<i1>) -> tensor<i1>
@@ -624,7 +624,7 @@ func.func @test_avg_pool2d_adaptive_missing_bf16_extension(%arg0: tensor<1x7x7x9
 func.func @test_mul_shape() {
   %a = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   %b = tosa.const_shape {values = dense<[5, 6, 7, 8]> : tensor<4xindex>} : () -> !tosa.shape<4>
-  // expected-error at +1 {{'tosa.mul_shape' op illegal: requires [shape] but not enabled in target}}
+  // expected-error at +1 {{'tosa.mul_shape' op illegal: requires any of [shape] profiles/extensions to be specified in the target environment}}
   %c = tosa.mul_shape %a, %b : (!tosa.shape<4>, !tosa.shape<4>) -> !tosa.shape<4>
   return
 }
@@ -634,7 +634,7 @@ func.func @test_mul_shape() {
 func.func @test_max_shape() {
   %a = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   %b = tosa.const_shape {values = dense<[5, 6, 7, 8]> : tensor<4xindex>} : () -> !tosa.shape<4>
-  // expected-error at +1 {{'tosa.max_shape' op illegal: requires [shape] but not enabled in target}}
+  // expected-error at +1 {{'tosa.max_shape' op illegal: requires any of [shape] profiles/extensions to be specified in the target environment}}
   %c = tosa.max_shape %a, %b : (!tosa.shape<4>, !tosa.shape<4>) -> !tosa.shape<4>
   return
 }
@@ -644,7 +644,39 @@ func.func @test_max_shape() {
 func.func @test_min_shape() {
   %a = tosa.const_shape {values = dense<[1, 2, 3, 4]> : tensor<4xindex>} : () -> !tosa.shape<4>
   %b = tosa.const_shape {values = dense<[5, 6, 7, 8]> : tensor<4xindex>} : () -> !tosa.shape<4>
-  // expected-error at +1 {{'tosa.min_shape' op illegal: requires [shape] but not enabled in target}}
+  // expected-error at +1 {{'tosa.min_shape' op illegal: requires any of [shape] profiles/extensions to be specified in the target environment}}
   %c = tosa.min_shape %a, %b : (!tosa.shape<4>, !tosa.shape<4>) -> !tosa.shape<4>
   return
 }
+
+// -----
+
+func.func @test_conv2d_block_scaled(%arg0: tensor<*xf4E2M1FN>, %arg1: tensor<*xf8E8M0FNU>, %arg2: tensor<*xf4E2M1FN>, %arg3: tensor<*xf8E8M0FNU>, %arg4: tensor<*xf32>) -> tensor<*xf32> {
+  %0 = tosa.const_shape {values = dense<[0, 0, 0, 0]> : tensor<4xindex>} : () -> !tosa.shape<4>
+  %1 = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %2 = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.conv2d_block_scaled' op illegal: requires any of [mxfp_conv] profiles/extensions to be specified in the target environment}}
+  %3 = tosa.conv2d_block_scaled %arg0, %arg1, %arg2, %arg3, %arg4, %0, %1, %2 {block_size = BLOCK_SIZE_32} : (tensor<*xf4E2M1FN>, tensor<*xf8E8M0FNU>, tensor<*xf4E2M1FN>, tensor<*xf8E8M0FNU>, tensor<*xf32>, !tosa.shape<4>, !tosa.shape<2>, !tosa.shape<2>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// -----
+func.func @test_matmul_t_block_scaled(%arg0: tensor<4x8x32xf6E3M2FN>, %arg1: tensor<4x8x1xf8E8M0FNU>, %arg2: tensor<4x16x32xf6E3M2FN>, %arg3: tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32> {
+  // expected-error at +1 {{'tosa.matmul_t_block_scaled' op illegal: requires any of [mxfp] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.matmul_t_block_scaled %arg0, %arg1, %arg2, %arg3 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x8x32xf6E3M2FN>, tensor<4x8x1xf8E8M0FNU>, tensor<4x16x32xf6E3M2FN>, tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+func.func @test_cast_from_block_scaled(%arg0: tensor<4x32xf4E2M1FN>, %arg1: tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32> {
+  // expected-error at +1 {{'tosa.cast_from_block_scaled' op illegal: requires any of [mxfp] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32>
+  return %0 : tensor<4x32xf32>
+}
+
+// -----
+func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) {
+  // expected-error at +1 {{'tosa.cast_to_block_scaled' op illegal: requires any of [mxfp] profiles/extensions to be specified in the target environment}}
+  %0:2 = tosa.cast_to_block_scaled %arg0 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>)
+  return %0#0, %0#1 : tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>
+}
diff --git a/mlir/test/Dialect/Tosa/profile_all_unsupported.mlir b/mlir/test/Dialect/Tosa/profile_all_unsupported.mlir
index caa547a0830e7..1c2899151cf4e 100644
--- a/mlir/test/Dialect/Tosa/profile_all_unsupported.mlir
+++ b/mlir/test/Dialect/Tosa/profile_all_unsupported.mlir
@@ -194,7 +194,7 @@ func.func @test_concat(%arg0: tensor<13x21x3xi1>, %arg1: tensor<13x21x3xi1>) ->
 
 // -----
 func.func @test_pad(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   // expected-error at +1 {{'tosa.const' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %pad_const = "tosa.const"() {values = dense<1> : tensor<1xi1>} : () -> tensor<1xi1>
@@ -205,7 +205,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
 
 // -----
 func.func @test_reshape(%arg0: tensor<13x21x3xi1>) -> tensor<1x819xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // expected-error at +1 {{'tosa.reshape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xi1>, !tosa.shape<2>) -> tensor<1x819xi1>
@@ -221,9 +221,9 @@ func.func @test_reverse(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
 
 // -----
 func.func @test_slice(%arg0: tensor<13x21x3xi1>) -> tensor<4x11x1xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %size = tosa.const_shape {values = dense<[4, 11, 1]> : tensor<3xindex>} : () -> !tosa.shape<3>
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %start = tosa.const_shape {values = dense<[6, 8, 0]> : tensor<3xindex>} : () -> !tosa.shape<3>
   // expected-error at +1 {{'tosa.slice' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %2 = tosa.slice %arg0, %start, %size : (tensor<13x21x3xi1>, !tosa.shape<3>, !tosa.shape<3>) -> tensor<4x11x1xi1>
@@ -232,7 +232,7 @@ func.func @test_slice(%arg0: tensor<13x21x3xi1>) -> tensor<4x11x1xi1> {
 
 // -----
 func.func @test_tile(%arg0: tensor<13x21x3xi1>) -> tensor<39x21x6xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %cst = tosa.const_shape { values = dense<[3, 1, 2]> : tensor<3xindex> } : () -> !tosa.shape<3>
   // expected-error at +1 {{'tosa.tile' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.tile %arg0, %cst: (tensor<13x21x3xi1>, !tosa.shape<3>) -> tensor<39x21x6xi1>
@@ -262,7 +262,7 @@ func.func @test_concat(%arg0: tensor<13x21x3xi1>, %arg1: tensor<13x21x3xi1>) ->
 
 // -----
 func.func @test_pad(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   // expected-error at +1 {{'tosa.const' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %pad_const = "tosa.const"() {values = dense<1> : tensor<1xi1>} : () -> tensor<1xi1>
@@ -273,7 +273,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
 
 // -----
 func.func @test_reshape(%arg0: tensor<13x21x3xi1>) -> tensor<1x819xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // expected-error at +1 {{'tosa.reshape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xi1>, !tosa.shape<2>) -> tensor<1x819xi1>
@@ -289,9 +289,9 @@ func.func @test_reverse(%arg0: tensor<13x21x3xi1>) -> tensor<13x21x3xi1> {
 
 // -----
 func.func @test_slice(%arg0: tensor<13x21x3xi1>) -> tensor<4x11x1xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %size = tosa.const_shape {values = dense<[4, 11, 1]> : tensor<3xindex>} : () -> !tosa.shape<3>
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %start = tosa.const_shape {values = dense<[6, 8, 0]> : tensor<3xindex>} : () -> !tosa.shape<3>
   // expected-error at +1 {{'tosa.slice' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.slice %arg0, %start, %size : (tensor<13x21x3xi1>, !tosa.shape<3>, !tosa.shape<3>) -> tensor<4x11x1xi1>
@@ -300,7 +300,7 @@ func.func @test_slice(%arg0: tensor<13x21x3xi1>) -> tensor<4x11x1xi1> {
 
 // -----
 func.func @test_tile(%arg0: tensor<13x21x3xi1>) -> tensor<39x21x6xi1> {
-  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.const_shape' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %cst = tosa.const_shape { values = dense<[3, 1, 2]> : tensor<3xindex> } : () -> !tosa.shape<3>
   // expected-error at +1 {{'tosa.tile' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.tile %arg0, %cst: (tensor<13x21x3xi1>, !tosa.shape<3>) -> tensor<39x21x6xi1>
@@ -322,7 +322,7 @@ func.func @test_cast_i32_f32(%arg0: tensor<13x21x3xi32>) -> tensor<13x21x3xf32>
 
 // -----
 func.func @test_custom(%arg0: tensor<10xi32>) -> tensor<10xi32> {
-  // expected-error at +1 {{'tosa.custom' op illegal: requires any of [pro_int, pro_fp] but not enabled in target}}
+  // expected-error at +1 {{'tosa.custom' op illegal: requires any of [pro_int, pro_fp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.custom %arg0 {operator_name="custom_test", domain_name="tosa.mlir_test", implementation_attrs="" } : (tensor<10xi32>) -> (tensor<10xi32>)
   return %0 : tensor<10xi32>
 }
diff --git a/mlir/test/Dialect/Tosa/profile_pro_fp_unsupported.mlir b/mlir/test/Dialect/Tosa/profile_pro_fp_unsupported.mlir
index 052271776770e..3f0ffab5fcd7d 100644
--- a/mlir/test/Dialect/Tosa/profile_pro_fp_unsupported.mlir
+++ b/mlir/test/Dialect/Tosa/profile_pro_fp_unsupported.mlir
@@ -336,34 +336,3 @@ func.func @test_resize(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x64x64x8xf32> {
   %1 = tosa.resize %arg0, %scale, %offset, %border { mode = BILINEAR } : (tensor<1x32x32x8xf32>, !tosa.shape<4>, !tosa.shape<2>, !tosa.shape<2>) -> tensor<1x64x64x8xf32>
   return %1 : tensor<1x64x64x8xf32>
 }
-
-// -----
-func.func @test_matmul_t_block_scaled(%arg0: tensor<4x8x32xf6E3M2FN>, %arg1: tensor<4x8x1xf8E8M0FNU>, %arg2: tensor<4x16x32xf6E3M2FN>, %arg3: tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32> {
-  // expected-error at +1 {{'tosa.matmul_t_block_scaled' op illegal: requires [pro_fp] but not enabled in target}}
-  %0 = tosa.matmul_t_block_scaled %arg0, %arg1, %arg2, %arg3 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x8x32xf6E3M2FN>, tensor<4x8x1xf8E8M0FNU>, tensor<4x16x32xf6E3M2FN>, tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32>
-  return %0 : tensor<4x8x16xf32>
-}
-
-// -----
-func.func @test_cast_from_block_scaled(%arg0: tensor<4x32xf4E2M1FN>, %arg1: tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32> {
-  // expected-error at +1 {{'tosa.cast_from_block_scaled' op illegal: requires [pro_fp] but not enabled in target}}
-  %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32>
-  return %0 : tensor<4x32xf32>
-}
-
-// -----
-func.func @test_cast_to_block_scaled(%arg0: tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>) {
-  // expected-error at +1 {{'tosa.cast_to_block_scaled' op illegal: requires [pro_fp] but not enabled in target}}
-  %0:2 = tosa.cast_to_block_scaled %arg0 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x32xf32>) -> (tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>)
-  return %0#0, %0#1 : tensor<4x32xf4E2M1FN>, tensor<4x1xf8E8M0FNU>
-}
-
-// -----
-func.func @test_conv2d_block_scaled(%arg0: tensor<*xf4E2M1FN>, %arg1: tensor<*xf8E8M0FNU>, %arg2: tensor<*xf4E2M1FN>, %arg3: tensor<*xf8E8M0FNU>, %arg4: tensor<*xf32>) -> tensor<*xf32> {
-  %0 = tosa.const_shape {values = dense<[0, 0, 0, 0]> : tensor<4xindex>} : () -> !tosa.shape<4>
-  %1 = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
-  %2 = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
-  // expected-error at +1 {{'tosa.conv2d_block_scaled' op illegal: requires [pro_fp] but not enabled in target}}
-  %3 = tosa.conv2d_block_scaled %arg0, %arg1, %arg2, %arg3, %arg4, %0, %1, %2 {block_size = BLOCK_SIZE_32} : (tensor<*xf4E2M1FN>, tensor<*xf8E8M0FNU>, tensor<*xf4E2M1FN>, tensor<*xf8E8M0FNU>, tensor<*xf32>, !tosa.shape<4>, !tosa.shape<2>, !tosa.shape<2>) -> tensor<*xf32>
-  return %3 : tensor<*xf32>
-}
diff --git a/mlir/test/lib/Dialect/Tosa/CMakeLists.txt b/mlir/test/lib/Dialect/Tosa/CMakeLists.txt
index 43f0d0d21c1c0..7d40881ee6ee4 100644
--- a/mlir/test/lib/Dialect/Tosa/CMakeLists.txt
+++ b/mlir/test/lib/Dialect/Tosa/CMakeLists.txt
@@ -1,6 +1,5 @@
 add_mlir_dialect_library(MLIRTosaTestPasses
   TosaTestPasses.cpp
-  TestAvailability.cpp
 
   EXCLUDE_FROM_LIBMLIR
 
diff --git a/mlir/test/lib/Dialect/Tosa/TestAvailability.cpp b/mlir/test/lib/Dialect/Tosa/TestAvailability.cpp
deleted file mode 100644
index ce18a936d32d2..0000000000000
--- a/mlir/test/lib/Dialect/Tosa/TestAvailability.cpp
+++ /dev/null
@@ -1,80 +0,0 @@
-//===- TestAvailability.cpp - Pass to test Tosa op availability ---------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "mlir/Dialect/Func/IR/FuncOps.h"
-#include "mlir/Dialect/Tosa/IR/TosaOps.h"
-#include "mlir/Pass/Pass.h"
-
-using namespace mlir;
-
-//===----------------------------------------------------------------------===//
-// Printing op availability pass
-//===----------------------------------------------------------------------===//
-
-namespace {
-/// A pass for testing Tosa op availability.
-struct PrintOpAvailability
-    : public PassWrapper<PrintOpAvailability, OperationPass<ModuleOp>> {
-  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(PrintOpAvailability)
-
-  void runOnOperation() override;
-  StringRef getArgument() const final { return "test-tosa-op-availability"; }
-  StringRef getDescription() const final { return "Test Tosa op availability"; }
-};
-} // namespace
-
-void PrintOpAvailability::runOnOperation() {
-  auto module = getOperation();
-  for (auto f : module.getOps<func::FuncOp>()) {
-    llvm::outs() << f.getName() << "\n";
-
-    Dialect *tosaDialect = getContext().getLoadedDialect("tosa");
-
-    f->walk([&](Operation *op) {
-      if (op->getDialect() != tosaDialect)
-        return WalkResult::advance();
-
-      auto opName = op->getName();
-      auto &os = llvm::outs();
-
-      if (auto profile = dyn_cast<tosa::QueryProfileInterface>(op)) {
-        os << opName << " profiles: [";
-        for (const auto &profs : profile.getProfiles()) {
-          os << " [";
-          llvm::interleaveComma(profs, os, [&](tosa::Profile prof) {
-            os << tosa::stringifyProfile(prof);
-          });
-          os << "]";
-        }
-        os << " ]\n";
-      }
-
-      if (auto extension = dyn_cast<tosa::QueryExtensionInterface>(op)) {
-        os << opName << " extensions: [";
-        for (const auto &exts : extension.getExtensions()) {
-          os << " [";
-          llvm::interleaveComma(exts, os, [&](tosa::Extension ext) {
-            os << tosa::stringifyExtension(ext);
-          });
-          os << "]";
-        }
-        os << " ]\n";
-      }
-
-      os.flush();
-
-      return WalkResult::advance();
-    });
-  }
-}
-
-namespace mlir {
-void registerPrintTosaAvailabilityPass() {
-  PassRegistration<PrintOpAvailability>();
-}
-} // namespace mlir
diff --git a/mlir/tools/mlir-opt/mlir-opt.cpp b/mlir/tools/mlir-opt/mlir-opt.cpp
index 13c0934f34656..47dffb1b2d28b 100644
--- a/mlir/tools/mlir-opt/mlir-opt.cpp
+++ b/mlir/tools/mlir-opt/mlir-opt.cpp
@@ -39,7 +39,6 @@ void registerLoopLikeInterfaceTestPasses();
 void registerPassManagerTestPass();
 void registerPrintSpirvAvailabilityPass();
 void registerRegionTestPasses();
-void registerPrintTosaAvailabilityPass();
 void registerShapeFunctionTestPasses();
 void registerSideEffectTestPasses();
 void registerSliceAnalysisTestPass();
@@ -185,7 +184,6 @@ void registerTestTransformsTransformDialectExtension(DialectRegistry &);
 static void registerTestPasses() {
   registerCloneTestPasses();
   registerConvertToTargetEnvPass();
-  registerPrintTosaAvailabilityPass();
   registerLazyLoadingTestPasses();
   registerLoopLikeInterfaceTestPasses();
   registerPassManagerTestPass();
diff --git a/mlir/tools/mlir-tblgen/CMakeLists.txt b/mlir/tools/mlir-tblgen/CMakeLists.txt
index d7087cba3c874..936c84ef58b2b 100644
--- a/mlir/tools/mlir-tblgen/CMakeLists.txt
+++ b/mlir/tools/mlir-tblgen/CMakeLists.txt
@@ -33,7 +33,6 @@ add_tablegen(mlir-tblgen MLIR
   PassGen.cpp
   RewriterGen.cpp
   SPIRVUtilsGen.cpp
-  TosaUtilsGen.cpp
   CppGenUtilities.cpp
   )
 
diff --git a/mlir/tools/mlir-tblgen/TosaUtilsGen.cpp b/mlir/tools/mlir-tblgen/TosaUtilsGen.cpp
deleted file mode 100644
index dc8cc58498230..0000000000000
--- a/mlir/tools/mlir-tblgen/TosaUtilsGen.cpp
+++ /dev/null
@@ -1,215 +0,0 @@
-//===----------------------------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// TosaUtilsGen generates common utility functions for Tosa validation.
-//
-//===----------------------------------------------------------------------===//
-
-#include "mlir/TableGen/Attribute.h"
-#include "mlir/TableGen/CodeGenHelpers.h"
-#include "mlir/TableGen/EnumInfo.h"
-#include "mlir/TableGen/Format.h"
-#include "mlir/TableGen/GenInfo.h"
-#include "mlir/TableGen/Operator.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/ADT/Sequence.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/ADT/StringExtras.h"
-#include "llvm/ADT/StringMap.h"
-#include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/StringSet.h"
-#include "llvm/Support/FormatVariadic.h"
-#include "llvm/Support/raw_ostream.h"
-#include "llvm/TableGen/Error.h"
-#include "llvm/TableGen/Record.h"
-#include "llvm/TableGen/TableGenBackend.h"
-
-#include <list>
-#include <optional>
-
-using llvm::formatv;
-using llvm::raw_ostream;
-using llvm::Record;
-using llvm::RecordKeeper;
-using llvm::StringMap;
-using llvm::StringRef;
-using mlir::tblgen::Operator;
-
-//===----------------------------------------------------------------------===//
-// Availability Wrapper Class
-//===----------------------------------------------------------------------===//
-
-namespace {
-// Wrapper class with helper methods for accessing availability defined in
-// TableGen.
-class Availability {
-public:
-  explicit Availability(const Record *def);
-
-  // Returns the name of the direct TableGen class for this availability
-  // instance.
-  StringRef getClass() const;
-
-  // Returns the name of the query function insided the generated C++ interface.
-  StringRef getQueryFnName() const;
-
-  // Returns the return type of the query function insided the generated C++
-  // interface.
-  StringRef getQueryFnRetType() const;
-
-  // Returns the code for merging availability requirements.
-  StringRef getMergeActionCode() const;
-
-  // Returns the initializer expression for initializing the final availability
-  // requirements.
-  StringRef getMergeInitializer() const;
-
-  // Returns the C++ statements for preparing availability instance.
-  StringRef getMergeInstancePreparation() const;
-
-  // Returns the concrete availability instance carried in this case.
-  StringRef getMergeInstance() const;
-
-  // Returns the underlying LLVM TableGen Record.
-  const llvm::Record *getDef() const { return def; }
-
-private:
-  // The TableGen definition of this availability.
-  const llvm::Record *def;
-};
-} // namespace
-
-Availability::Availability(const llvm::Record *def) : def(def) {
-  assert(def->isSubClassOf("Availability") &&
-         "must be subclass of TableGen 'Availability' class");
-}
-
-StringRef Availability::getClass() const {
-  if (def->getDirectSuperClasses().size() != 1) {
-    PrintFatalError(def->getLoc(),
-                    "expected to only have one direct superclass");
-  }
-  return def->getDirectSuperClasses().front().first->getName();
-}
-
-StringRef Availability::getQueryFnRetType() const {
-  return def->getValueAsString("queryFnRetType");
-}
-
-StringRef Availability::getQueryFnName() const {
-  return def->getValueAsString("queryFnName");
-}
-
-StringRef Availability::getMergeActionCode() const {
-  return def->getValueAsString("mergeAction");
-}
-
-StringRef Availability::getMergeInitializer() const {
-  return def->getValueAsString("initializer");
-}
-
-StringRef Availability::getMergeInstancePreparation() const {
-  return def->getValueAsString("instancePreparation");
-}
-
-StringRef Availability::getMergeInstance() const {
-  return def->getValueAsString("instance");
-}
-
-// Returns the availability spec of the given `def`.
-static std::vector<Availability> getAvailabilities(const Record &def) {
-  std::vector<Availability> availabilities;
-
-  if (def.getValue("availability")) {
-    std::vector<const Record *> availDefs =
-        def.getValueAsListOfDefs("availability");
-    availabilities.reserve(availDefs.size());
-    for (const Record *avail : availDefs)
-      availabilities.emplace_back(avail);
-  }
-
-  return availabilities;
-}
-
-//===----------------------------------------------------------------------===//
-// Tosa Availability Impl AutoGen
-//===----------------------------------------------------------------------===//
-
-static void emitAvailabilityImpl(const Operator &srcOp, raw_ostream &os) {
-  mlir::tblgen::FmtContext fctx;
-  fctx.addSubst("overall", "tblgen_overall");
-
-  std::vector<Availability> opAvailabilities =
-      getAvailabilities(srcOp.getDef());
-
-  // First collect all availability classes this op should implement.
-  // All availability instances keep information for the generated interface and
-  // the instance's specific requirement. Here we remember a random instance so
-  // we can get the information regarding the generated interface.
-  llvm::StringMap<Availability> availClasses;
-  for (const Availability &avail : opAvailabilities)
-    availClasses.try_emplace(avail.getClass(), avail);
-
-  // Then generate implementation for each availability class.
-  for (const auto &availClass : availClasses) {
-    StringRef availClassName = availClass.getKey();
-    Availability avail = availClass.getValue();
-
-    // Generate the implementation method signature.
-    os << formatv("{0} {1}::{2}() {{\n", avail.getQueryFnRetType(),
-                  srcOp.getCppClassName(), avail.getQueryFnName());
-
-    // Create the variable for the final requirement and initialize it.
-    os << formatv("  {0} tblgen_overall = {1};\n", avail.getQueryFnRetType(),
-                  avail.getMergeInitializer());
-
-    // Update with the op's specific availability spec.
-    for (const Availability &avail : opAvailabilities)
-      if (avail.getClass() == availClassName &&
-          (!avail.getMergeInstancePreparation().empty() ||
-           !avail.getMergeActionCode().empty())) {
-        os << "  {\n    "
-           // Prepare this instance.
-           << avail.getMergeInstancePreparation()
-           << "\n    "
-           // Merge this instance.
-           << std::string(
-                  tgfmt(avail.getMergeActionCode(),
-                        &fctx.addSubst("instance", avail.getMergeInstance())))
-           << ";\n  }\n";
-      }
-
-    os << "  return tblgen_overall;\n";
-    os << "}\n";
-  }
-}
-
-static bool emitAvailabilityImpl(const RecordKeeper &recordKeeper,
-                                 raw_ostream &os) {
-  llvm::emitSourceFileHeader("Tosa Op Availability Implementations", os,
-                             recordKeeper);
-
-  auto defs = recordKeeper.getAllDerivedDefinitions("Tosa_Op");
-  for (const auto *def : defs) {
-    Operator op(def);
-    if (def->getValueAsBit("autogenAvailability"))
-      emitAvailabilityImpl(op, os);
-  }
-  return false;
-}
-
-//===----------------------------------------------------------------------===//
-// Op Availability Implementation Hook Registration
-//===----------------------------------------------------------------------===//
-
-static mlir::GenRegistration
-    genOpAvailabilityImpl("gen-tosa-avail-impls",
-                          "Generate Tosa operation utility definitions",
-                          [](const RecordKeeper &records, raw_ostream &os) {
-                            return emitAvailabilityImpl(records, os);
-                          });
diff --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
index 2d409818e57a1..4e261e0d28b01 100644
--- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
@@ -12161,18 +12161,6 @@ gentbl_cc_library(
     deps = [":TosaDialectTdFiles"],
 )
 
-gentbl_cc_library(
-    name = "TosaAvailabilityIncGen",
-    tbl_outs = {
-        "include/mlir/Dialect/Tosa/IR/TosaAvailability.h.inc": ["-gen-avail-interface-decls"],
-        "include/mlir/Dialect/Tosa/IR/TosaAvailability.cpp.inc": ["-gen-avail-interface-defs"],
-        "include/mlir/Dialect/Tosa/IR/TosaOpAvailabilityImpl.inc": ["-gen-tosa-avail-impls"],
-    },
-    tblgen = ":mlir-tblgen",
-    td_file = "include/mlir/Dialect/Tosa/IR/TosaOps.td",
-    deps = [":TosaDialectTdFiles"],
-)
-
 gentbl_cc_library(
     name = "TosaDialectBytecodeGen",
     strip_include_prefix = "include",
@@ -12256,7 +12244,6 @@ cc_library(
         ":SideEffectInterfaces",
         ":Support",
         ":TensorDialect",
-        ":TosaAvailabilityIncGen",
         ":TosaDialectBytecodeGen",
         ":TosaDialectIncGen",
         ":TosaEnumsIncGen",


From llvmlistbot at llvm.org  Mon Jul  6 12:50:54 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Mon, 06 Jul 2026 12:50:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a4c071e.eb24c6ea.1f2e26.4c30@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/205464

>From ce76fb7aa44a3320fdb45acc6e011758ed3afc77 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 1/6] [mlir][EmitC] Create a pass to add a reflection map to a
 class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |   2 +
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  52 +++++
 .../Dialect/EmitC/Transforms/Transforms.h     |   7 +
 .../EmitC/Transforms/AddReflectionMap.cpp     | 181 ++++++++++++++++++
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |   1 +
 .../Dialect/EmitC/add-reflection-map.mlir     |  58 ++++++
 6 files changed, 301 insertions(+)
 create mode 100644 mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
 create mode 100644 mlir/test/Dialect/EmitC/add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 1af4aa06fa811..56596110a1e90 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,6 +16,8 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
+#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index c34c3303a6ab3..6da57c07a8047 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,4 +60,56 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `field-attr-name` option) that is an array containing a single string
+    attribute.
+
+
+    Example:
+
+    ```mlir
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.func @execute() { ... }
+    }
+
+    Becomes:
+
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
+      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
+          -> !emitc.ptr<!emitc.opaque<"char">> {
+        %0 = get_field @reflectionMap
+          : !emitc.opaque<"const std::map<std::string, char*>">
+        %1 = member_call_opaque %0 "at"(%arg0)
+          : !emitc.opaque<"const std::map<std::string, char*>">,
+            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      }
+      emitc.func @execute() { ... }
+    }
+    ```
+  }];
+  let dependentDialects = ["emitc::EmitCDialect"];
+  let options = [
+    Option<"fieldAttrName", "field-attr-name", "std::string",
+      /*default=*/"",
+      "Attribute key used to extract field names from a field's "
+      "attribute dictionary">,
+    ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
+      "Attribute keys that, if present on a field, exclude it "
+      "from the reflection map">
+  ];
+}
+
 #endif // MLIR_DIALECT_EMITC_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 791e545a8edcf..8eadfe9a2460d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/EmitC/IR/EmitC.h"
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -37,6 +38,12 @@ void populateWrapFuncInClass(
     RewritePatternSet &patterns, StringRef funcName,
     DenseMap<FuncOp, llvm::DenseSet<GlobalOp>> &globalsToMove);
 
+//===----------------------------------------------------------------------===//
+// The AddReflectionMap pass.
+//===----------------------------------------------------------------------===//
+
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+
 } // namespace emitc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
new file mode 100644
index 0000000000000..c1588463ffb7a
--- /dev/null
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -0,0 +1,181 @@
+//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(
+      builder, module.getLoc(), includeAttr,
+      /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class AddReflectionMapPass
+    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
+  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+
+    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+        
+        hasMapHdr = include == mapLibraryHeader;
+        hasStringHdr = include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect all field names
+    std::vector<std::pair<std::string, std::string>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
+        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
+          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
+          fieldNames.emplace_back(stringAttr.getValue().str(),
+                                  fieldOp.getName().str());
+          return;
+        }
+      }
+
+      bool shouldIgnore = false;
+      for (const std::string &ignoreAttr : excludedFieldAttrs) {
+        if (fieldOp->hasAttr(ignoreAttr)) {
+          shouldIgnore = true;
+          break;
+        }
+      }
+
+      if (shouldIgnore)
+        return;
+
+      fieldOp.emitError()
+          << "FieldOp must have a dictionary attribute named '"
+          << fieldAttrName << "'"
+          << "with an array containing a string attribute";
+    });
+
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
+          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+    }
+    reflectionMapContents += " }";
+
+    if (FuncOp executeFunc =
+            classOp.lookupSymbol<FuncOp>("operator()"))
+      rewriter.setInsertionPoint(executeFunc);
+    else {
+      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
+                             "to add reflection map";
+      return failure();
+    }
+
+    FieldOp reflectionMapField = FieldOp::create(
+        rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
+        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+
+    // Create getBufferForName method
+    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    FuncOp getBufferForNameFunc = FuncOp::create(
+        rewriter, reflectionMapField->getLoc(), "getBufferForName",
+        FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
+
+    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    rewriter.setInsertionPointToStart(body);
+    GetFieldOp mapField = GetFieldOp::create(
+        rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
+    Value nameArg = body->getArgument(0);
+    MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
+        rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
+        "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+
+    return success();
+  }
+
+private:
+  /// The name of the attribute on FieldOps that contains the field name
+  /// metadata for the reflection map.
+  StringRef fieldAttrName;
+
+  /// Attributes that, if present on a field, exclude it from the
+  /// reflection map.
+  llvm::SmallVector<std::string> excludedFieldAttrs;
+};
+
+void mlir::emitc::populateAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    llvm::ArrayRef<std::string> excludedFieldAttrs) {
+  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                      excludedFieldAttrs);
+}
\ No newline at end of file
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index baf67afc30072..dd8f014dc4737 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
+  AddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
new file mode 100644
index 0000000000000..a1ec0356e19f4
--- /dev/null
+++ b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+
+
+// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+// Test that a reflection map is created for fields with a certain named attribute
+// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
\ No newline at end of file

>From 60d4d88576ecc713db0a2cf7ae184308c3cd745f Mon Sep 17 00:00:00 2001
From: Bhavesh M <85327930+beamandala at users.noreply.github.com>
Date: Tue, 23 Jun 2026 02:37:42 +0530
Subject: [PATCH 2/6] [mlir][EmitC] Make `GlobalOps` `FieldOps` in
 wrap-emitc-func-in-class pass (#203641)

Update the `WrapFuncInClassPass` pass so that `GlobalOp`s are moved into
the `ClassOp` as `FieldOps`. This respects MLIR's behavior of resolving
references to the closest parent operation that defines a symbol table
which is the `ClassOp` that we are creating in this pass.

Without this change, references to a `GlobalOp` in `GetGlobalOp` are
failing to resolve.

Details:
- Identify `GlobalOp`s
- Create a `FieldOp` within the `ClassOp` for each `GlobalOp`
- Delete the `GlobalOp`s after all functions have been wrapped in a
class. Doing this after every function can cause an error when multiple
functions refer to the same `GlobalOp`(s) which would be deleted after
the first function is wrapped in a class.

Also renamed `fName` parameter in `populateWrapFuncInClass` to
`funcName` to match naming in `WrapFuncInClass`.

Based on PR #153452. Key differences:
- No size is set for the `globalsToMove` `SmallVector` type because I'm
not sure if the number of global variables is consistent across
different models.
- `GlobalOp`s are deleted after all functions have been processed.
- Instead of directly cloning the `GlobalOp`, an equivalent `FieldOp` is
created
- `GetGlobalOp`s are translated to `GetFieldOp`s

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8eadfe9a2460d..4c8d3978af0f1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,6 +13,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace mlir {
 namespace emitc {

>From b6af183f868036d825f4133a2671b67e5a9b643e Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 3/6] [mlir][EmitC] Create a pass to add a reflection map to a
 class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 4c8d3978af0f1..5a4781dc31ea8 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {

>From 8d7429a032c6e0d91d1857ff5ef5c324ac35999f Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 24 Jun 2026 10:31:17 -0700
Subject: [PATCH 4/6] Formatting

---
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +--
 .../EmitC/Transforms/AddReflectionMap.cpp     | 47 +++++++++++--------
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 5a4781dc31ea8..8f1784f61f540 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,8 +13,6 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -44,7 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
+                                      StringRef fieldAttrName,
+                                      ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
index c1588463ffb7a..d444f3abbf4b1 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -31,9 +31,8 @@ constexpr const char *stringLibraryHeader = "string";
 
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
-  return IncludeOp::create(
-      builder, module.getLoc(), includeAttr,
-      /*is_standard_include=*/builder.getUnitAttr());
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
 }
 
 class AddReflectionMapPass
@@ -43,7 +42,8 @@ class AddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+    populateAddReflectionMapPatterns(patterns, fieldAttrName,
+                                     excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -55,7 +55,7 @@ class AddReflectionMapPass
 
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
-        
+
         hasMapHdr = include == mapLibraryHeader;
         hasStringHdr = include == stringLibraryHeader;
       }
@@ -82,7 +82,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   AddReflectionMapClass(MLIRContext *context, StringRef attrName,
                         llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
-        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+        excludedFieldAttrs(excludedFieldAttrs.begin(),
+                           excludedFieldAttrs.end()) {}
 
   LogicalResult matchAndRewrite(ClassOp classOp,
                                 PatternRewriter &rewriter) const override {
@@ -114,10 +115,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError()
-          << "FieldOp must have a dictionary attribute named '"
-          << fieldAttrName << "'"
-          << "with an array containing a string attribute";
+      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
+                          << fieldAttrName << "'"
+                          << "with an array containing a string attribute";
     });
 
     std::string reflectionMapContents;
@@ -129,28 +129,34 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc =
-            classOp.lookupSymbol<FuncOp>("operator()"))
+    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
-                             "to add reflection map";
+      classOp.emitError()
+          << "ClassOp must contain a function named 'operator()' "
+             "to add reflection map";
       return failure();
     }
 
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
-        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+        TypeAttr::get(mapType),
+        emitc::OpaqueAttr::get(context, reflectionMapContents));
 
     // Create getBufferForName method
-    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
-    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
-    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    emitc::OpaqueType nameType =
+        emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType =
+        emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType =
+        emitc::PointerType::get(rewriter.getContext(), charType);
     FuncOp getBufferForNameFunc = FuncOp::create(
         rewriter, reflectionMapField->getLoc(), "getBufferForName",
         FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
 
-    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    Block *body =
+        rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType},
+                             {reflectionMapField->getLoc()});
     rewriter.setInsertionPointToStart(body);
     GetFieldOp mapField = GetFieldOp::create(
         rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
@@ -158,7 +164,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
         rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
         "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
-    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(),
+                     lookupCall.getResult(0));
 
     return success();
   }

>From 266f21b78f0534b8bdf8354587cc883a81464a48 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Fri, 26 Jun 2026 16:04:13 -0700
Subject: [PATCH 5/6] Address review comments

---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |  2 +-
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 10 ++-
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +-
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |  2 +-
 ...ectionMap.cpp => MLGOAddReflectionMap.cpp} | 76 +++++++++--------
 ...-map.mlir => mlgo-add-reflection-map.mlir} | 84 +++++++++++++++++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 57 +++++++++++++
 7 files changed, 188 insertions(+), 49 deletions(-)
 rename mlir/lib/Dialect/EmitC/Transforms/{AddReflectionMap.cpp => MLGOAddReflectionMap.cpp} (71%)
 rename mlir/test/Dialect/EmitC/{add-reflection-map.mlir => mlgo-add-reflection-map.mlir} (52%)
 create mode 100644 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 56596110a1e90..4f46d63774625 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,7 +16,7 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
-#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DECL_MLGOADDREFLECTIONMAPPASS
 
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 6da57c07a8047..3d17f53032a7a 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,7 +60,7 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
-def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
     This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
@@ -68,9 +68,11 @@ def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
     `field-attr-name` option) that is an array containing a single string
-    attribute.
-
-
+    attribute. If a field possesses both the attribute specified by
+    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
+    the `field-attr-name` attribute takes precedence and the field will be
+    included in the reflection map.
+    
     Example:
 
     ```mlir
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8f1784f61f540..6425614e1fbcd 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -42,9 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
-                                      StringRef fieldAttrName,
-                                      ArrayRef<std::string> excludedFieldAttrs);
+void populateMLGOAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index dd8f014dc4737..ede2cb5e208f0 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,7 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
-  AddReflectionMap.cpp
+  MLGOAddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
similarity index 71%
rename from mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
rename to mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index d444f3abbf4b1..9f506b10c722a 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -1,7 +1,7 @@
-//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt license information.
+// See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
@@ -22,7 +22,7 @@ using namespace emitc;
 
 namespace mlir {
 namespace emitc {
-#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 namespace {
@@ -35,15 +35,15 @@ IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
                            /*is_standard_include=*/builder.getUnitAttr());
 }
 
-class AddReflectionMapPass
-    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
-  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
   void runOnOperation() override {
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName,
-                                     excludedFieldAttrs);
+    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+                                         excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -56,8 +56,8 @@ class AddReflectionMapPass
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
 
-        hasMapHdr = include == mapLibraryHeader;
-        hasStringHdr = include == stringLibraryHeader;
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
       }
 
       if (hasMapHdr && hasStringHdr)
@@ -77,10 +77,10 @@ class AddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
-class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
-                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                            llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
@@ -93,33 +93,35 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
         context, "const std::map<std::string, char*>");
 
     // Collect all field names
-    std::vector<std::pair<std::string, std::string>> fieldNames;
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
-        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
-          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
-          fieldNames.emplace_back(stringAttr.getValue().str(),
-                                  fieldOp.getName().str());
-          return;
-        }
-      }
-
-      bool shouldIgnore = false;
-      for (const std::string &ignoreAttr : excludedFieldAttrs) {
-        if (fieldOp->hasAttr(ignoreAttr)) {
-          shouldIgnore = true;
-          break;
+      if (auto arrayAttr =
+              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
+        if (!arrayAttr.empty()) {
+          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
+            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
+            return;
+          }
         }
       }
 
+      bool shouldIgnore =
+          llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
+            return fieldOp->hasAttr(ignoreAttr);
+          });
       if (shouldIgnore)
         return;
 
       fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "'"
+                          << fieldAttrName << "' "
                           << "with an array containing a string attribute";
+      hasError = true;
     });
 
+    if (hasError || fieldNames.empty())
+      return failure();
+
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
@@ -138,6 +140,14 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       return failure();
     }
 
+    // To generate the following C++ code
+    // const std::map<std::string, char*> reflectionMap = {
+    //  { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+    //  { "some_feature", reinterpret_cast<char*>(&fieldName1) },
+    //  ...
+    // };
+    // This can be used to retrieve a pointer to the field's contents given the
+    // attribute string identifying the field
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
         TypeAttr::get(mapType),
@@ -180,9 +190,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   llvm::SmallVector<std::string> excludedFieldAttrs;
 };
 
-void mlir::emitc::populateAddReflectionMapPatterns(
+void mlir::emitc::populateMLGOAddReflectionMapPatterns(
     RewritePatternSet &patterns, StringRef fieldAttrName,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                      excludedFieldAttrs);
-}
\ No newline at end of file
+  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                          excludedFieldAttrs);
+}
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
similarity index 52%
rename from mlir/test/Dialect/EmitC/add-reflection-map.mlir
rename to mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index a1ec0356e19f4..2bea6cfaf2677 100644
--- a/mlir/test/Dialect/EmitC/add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,7 +1,7 @@
-// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
 
-
-// Tests that a reflection map is created for fields with a certain attribute.
+/// Tests that a reflection map is created for fields with a certain attribute.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -29,10 +29,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:  }
 
 // -----
-// Test that a reflection map is created for fields with a certain named attribute
-// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClass {
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -41,7 +42,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @actionClassExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @actionClassNoFields {
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if the ClassOp doesn't have
+/// a FunctionOp named operator()
+
+// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
+emitc.class @actionClassNoOperator {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if a FieldOp has the specified
+/// dictionary attribute with an array containing a type other than string
+
+emitc.class @actionClassNonStringAttr {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
\ No newline at end of file
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
new file mode 100644
index 0000000000000..eb8a8e58c8b77
--- /dev/null
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 044316b2985c962d2a4b29fa700cf6974b5b7d60 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Mon, 6 Jul 2026 12:50:33 -0700
Subject: [PATCH 6/6] update MLGOAddReflectionMap to use notifyMatchFailure,
 update example to be more minimal and clear about the transformation, use
 range based for loop in reflection map construction

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 24 ++++++------------
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 25 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        |  5 +---
 3 files changed, 24 insertions(+), 30 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 3d17f53032a7a..29b053f3469c7 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -77,28 +77,20 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
 
     ```mlir
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.func @execute() { ... }
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      emitc.func @operator()() { ... }
     }
 
     Becomes:
 
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
-      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
-          -> !emitc.ptr<!emitc.opaque<"char">> {
-        %0 = get_field @reflectionMap
-          : !emitc.opaque<"const std::map<std::string, char*>">
-        %1 = member_call_opaque %0 "at"(%arg0)
-          : !emitc.opaque<"const std::map<std::string, char*>">,
-            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
-        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      // reflectionMap maps the string "feature0" to the address of @field0
+      emitc.field @reflectionMap : const std::map<string, char*> = {{"feature0", &field0}}
+      emitc.func @getBufferForName(%name: string) -> char* {
+        return reflectionMap.at(%name)
       }
-      emitc.func @execute() { ... }
+      emitc.func @operator()() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 9f506b10c722a..b93138987719b 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -113,9 +113,11 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "' "
-                          << "with an array containing a string attribute";
+      (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
+        diag << "FieldOp must have a dictionary attribute named '"
+             << fieldAttrName << "' "
+             << "with an array containing a string attribute";
+      });
       hasError = true;
     });
 
@@ -124,20 +126,23 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
-    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+      
+      first = false;
       reflectionMapContents += llvm::formatv(
-          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
-          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
     }
     reflectionMapContents += " }";
 
     if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError()
-          << "ClassOp must contain a function named 'operator()' "
-             "to add reflection map";
-      return failure();
+      return rewriter.notifyMatchFailure(
+          classOp, "ClassOp must contain a function named 'operator()' "
+                   "to add reflection map");
     }
 
     // To generate the following C++ code
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2bea6cfaf2677..1f8647910ada9 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -63,7 +63,6 @@ emitc.class @actionClassExcluded {
 /// Test that the pass leaves IR unchanged if fields don't have any attributes
 
 emitc.class @actionClassNoAttrs {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -98,7 +97,6 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
 emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
@@ -113,7 +111,6 @@ emitc.class @actionClassNoOperator {
 /// dictionary attribute with an array containing a type other than string
 
 emitc.class @actionClassNonStringAttr {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return


From llvmlistbot at llvm.org  Mon Jul  6 13:47:11 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 06 Jul 2026 13:47:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix blocking pass for scf.if
	distribution (PR #207060)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207060@github.com>
Message-ID: <6a4c144f.f06f4994.80921.a5a1@mx.google.com>


================
@@ -924,6 +924,21 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
         recordTypes(init, {arg, res, yieldVal});
       return;
     }
+    if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
+      // scf.if results expand 1:N during blocking. Record each result with the
----------------
Jianhui-Li wrote:

please update the comments before the function to include IfOp. I would avoid too detailed comment if they are really necessary but try to maintain the high level summary comment to be consistent. 

https://github.com/llvm/llvm-project/pull/207060

From llvmlistbot at llvm.org  Mon Jul  6 14:47:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 14:47:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a4c2277.491d90f5.2facd.581d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

Builds on https://github.com/llvm/llvm-project/pull/203894 to enable MXFP block scaled fotmats within matmul_t.

---

Patch is 48.87 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/207851.diff


11 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+520-41) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+7-1) 
- (modified) mlir/lib/Dialect/Tosa/IR/TosaOps.cpp (+16-6) 
- (modified) mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp (+3) 
- (modified) mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp (+2) 
- (modified) mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp (+10-6) 
- (modified) mlir/test/Dialect/Tosa/invalid_extension.mlir (+11) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+27) 
- (modified) mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir (+10) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+30) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+10) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..c1fb71110d4a1 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
      {{{Profile::pro_int}, {{{i8T, i8T}, SpecificationVersion::V_1_0}}},
       {{Profile::pro_fp},
@@ -696,52 +699,528 @@ extensionComplianceMap = {
        {{{bf16T, bf16T, bf16T, bf16T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Extension::int16},
-       {{{i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i16T, i16T, i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
-       {{{fp8e4m3T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3, Extension::fp8e5m2},
-       {{{fp8e4m3T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16, Extension::fp8e4m3},
-       {{{fp8e4m3T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::fp8e5m2},
-       {{{fp8e5m2T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16, Extension::fp8e5m2},
-       {{{fp8e5m2T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16},
-       {{{fp16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Mon Jul  6 15:51:45 2026
From: llvmlistbot at llvm.org (Eddy Shieh)
Date: Mon, 06 Jul 2026 15:51:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [SPIRV] Lower ptr dialect to PBA addresses
	(PR #206159)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206159@github.com>
Message-ID: <6a4c3181.eb24c6ea.27aedc.7e8d@mx.google.com>


edmundshieh wrote:

> The change looks reasonable and it has e2e testing, so LGTM, but let's wait for @kuhar input.

sounds good. Let me know if any changes needed @kuhar 

https://github.com/llvm/llvm-project/pull/206159

From llvmlistbot at llvm.org  Mon Jul  6 17:58:42 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Mon, 06 Jul 2026 17:58:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a4c4f42.30759430.24d50c.41d8@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 01/14] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 02/14] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 03/14] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 04/14] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 05/14] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 06/14] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 07/14] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 08/14] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 09/14] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.

>From 39b303e345d8d744745fad8bef4423ed6ce75c2d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:18 +0200
Subject: [PATCH 10/14] Update mlir/lib/Target/LLVMIR/DebugTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fca5e0a8497eb..10ae5ef0ec646 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,7 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (DIGlobalVariableExpressionAttr GVE =
+        if (auto GVE =
                 dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 

>From 292e16924dd039876c60e17afc4fa491e448e79d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:36:41 +0200
Subject: [PATCH 11/14] Clang-format fix

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  4 +-
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 52 +++++++++----------
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     | 32 ++++++------
 3 files changed, 41 insertions(+), 47 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 62523ffef3895..846342bee6071 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2548,9 +2548,7 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) {
-    replaceOperandWith(7, N.get());
-  }
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 29b010ee37f5b..72497a5328f78 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -710,36 +710,34 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
                                       [LLVM_DIRecursiveTypeAttrInterface],
                                       "DILocalScopeAttr"> {
   let parameters = (ins
-    // DIRecursiveTypeAttrInterface specific parameters.
-    OptionalParameter<"DistinctAttr">:$recId,
-    OptionalParameter<"bool">:$isRecSelf,
-    // DISubprogramAttr specific parameters.
-    OptionalParameter<"DistinctAttr">:$id,
-    OptionalParameter<"DICompileUnitAttr">:$compileUnit,
-    OptionalParameter<"DIScopeAttr">:$scope,
-    OptionalParameter<"StringAttr">:$name,
-    OptionalParameter<"StringAttr">:$linkageName,
-    OptionalParameter<"DIFileAttr">:$file,
-    OptionalParameter<"unsigned">:$line,
-    OptionalParameter<"unsigned">:$scopeLine,
-    OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
-    OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
-    OptionalArrayRefParameter<"DINodeAttr">:$annotations
-  );
-  let builders = [
-    AttrBuilder<(ins
-      "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
-      "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
-      "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
-      "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
-    ), [{
+      // DIRecursiveTypeAttrInterface specific parameters.
+      OptionalParameter<"DistinctAttr">:$recId,
+      OptionalParameter<"bool">:$isRecSelf,
+      // DISubprogramAttr specific parameters.
+      OptionalParameter<"DistinctAttr">:$id,
+      OptionalParameter<"DICompileUnitAttr">:$compileUnit,
+      OptionalParameter<"DIScopeAttr">:$scope,
+      OptionalParameter<"StringAttr">:$name,
+      OptionalParameter<"StringAttr">:$linkageName,
+      OptionalParameter<"DIFileAttr">:$file,
+      OptionalParameter<"unsigned">:$line,
+      OptionalParameter<"unsigned">:$scopeLine,
+      OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
+      OptionalParameter<"DISubroutineTypeAttr">:$type,
+      OptionalArrayRefParameter<"Attribute">:$retainedNodes,
+      OptionalArrayRefParameter<"DINodeAttr">:$annotations);
+  let builders = [AttrBuilder<
+      (ins "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
+          "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
+          "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
+          "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
+          "ArrayRef<Attribute>":$retainedNodes,
+          "ArrayRef<DINodeAttr>":$annotations),
+      [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
                    subprogramFlags, type, retainedNodes, annotations);
-    }]>
-  ];
+    }]>];
   let assemblyFormat = "`<` struct(params) `>`";
   let extraClassDeclaration = [{
     /// Requirements of DIRecursiveTypeAttrInterface.
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index 0c78689fe1d6b..b25a8c8a9829e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -150,23 +150,21 @@ def DICompileUnitAttr : DialectAttribute<(attr
 // DISubprogramAttr
 //===----------------------------------------------------------------------===//
 
-def DISubprogramAttr : DialectAttribute<(attr
-  OptionalAttribute<"DistinctAttr">:$recId,
-  Bool:$isRecSelf,
-  OptionalAttribute<"DistinctAttr">:$id,
-  OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
-  OptionalAttribute<"DIScopeAttr">:$scope,
-  OptionalAttribute<"StringAttr">:$name,
-  OptionalAttribute<"StringAttr">:$linkageName,
-  OptionalAttribute<"DIFileAttr">:$file,
-  VarInt:$line,
-  VarInt:$scopeLine,
-  EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
-  LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
-  OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"Attribute">:$retainedNodes,
-  OptionalArrayRef<"DINodeAttr">:$annotations
-)>;
+def DISubprogramAttr
+    : DialectAttribute<(attr OptionalAttribute<"DistinctAttr">:$recId,
+          Bool:$isRecSelf, OptionalAttribute<"DistinctAttr">:$id,
+          OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
+          OptionalAttribute<"DIScopeAttr">:$scope,
+          OptionalAttribute<"StringAttr">:$name,
+          OptionalAttribute<"StringAttr">:$linkageName,
+          OptionalAttribute<"DIFileAttr">:$file, VarInt:$line,
+          VarInt:$scopeLine,
+          EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
+          LocalVar<"DISubprogramFlags",
+                   "(DISubprogramFlags)_rawflags">:$subprogramFlags,
+          OptionalAttribute<"DISubroutineTypeAttr">:$type,
+          OptionalArrayRef<"Attribute">:$retainedNodes,
+          OptionalArrayRef<"DINodeAttr">:$annotations)>;
 
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr

>From 3b92877dfd5c5ffbc0b1c557a78bcbf6cafce502 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:44:51 +0200
Subject: [PATCH 12/14] Clang-format fix

---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 10ae5ef0ec646..442f5b6b137e6 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,8 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (auto GVE =
-                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+        if (auto GVE = dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 
         auto diAttr = dyn_cast<DINodeAttr>(attr);

>From 23a9b20861b2203633fedb5734e645e14175fcf7 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sat, 4 Jul 2026 22:58:30 +0200
Subject: [PATCH 13/14] Fix LVIRReader.cpp

---
 llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
index eb9fa3c89a44e..5726955a029ef 100644
--- a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
+++ b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
@@ -1447,13 +1447,15 @@ LVScope *LVIRReader::getOrCreateSubprogram(LVScope *Function,
   }
 
   // Check for additional retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *IE = dyn_cast<DIImportedEntity>(DN))
       constructImportedEntity(Function, IE);
     else if (const auto *TTP = dyn_cast<DITemplateTypeParameter>(DN))
       constructTemplateTypeParameter(Function, TTP);
     else if (const auto *TVP = dyn_cast<DITemplateValueParameter>(DN))
       constructTemplateValueParameter(Function, TVP);
+    else if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(DN))
+      getOrCreateVariable(GVE);
   }
 
   applySubprogramAttributes(Function, SP);

>From e34e1d4170095c62d49141f2e97bc3ecc64ea561 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 5 Jul 2026 01:55:38 +0200
Subject: [PATCH 14/14] Fix debug-info-correlate-bitmap.ll

---
 llvm/lib/IR/DIBuilder.cpp                           | 13 ++++++-------
 .../Transforms/Instrumentation/InstrProfiling.cpp   |  1 +
 .../InstrProfiling/debug-info-correlate-bitmap.ll   |  3 ++-
 3 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 6e285a8a155fa..60854c11ff014 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -56,20 +56,19 @@ void DIBuilder::finalizeSubprogram(DISubprogram *SP) {
   if (PN == SubprogramTrackedNodes.end())
     return;
 
-  SmallVector<Metadata *, 16> RetainedNodes;
-  for (MDNode *N : PN->second) {
+  SetVector<Metadata *> RetainedNodes;
+  for (MDNode *N : llvm::concat<MDNode *>(SP->getRetainedNodes(), PN->second)) {
     // If the tracked node N was temporary, and the DIBuilder user replaced it
     // with a node that does not belong to SP or is non-local, do not add N to
     // SP's retainedNodes list.
     DILocalScope *Scope = dyn_cast_or_null<DILocalScope>(
         DISubprogram::getRawRetainedNodeScope(N));
-    if (!Scope || Scope->getSubprogram() != SP)
-      continue;
-
-    RetainedNodes.push_back(N);
+    if (Scope && Scope->getSubprogram() == SP)
+      RetainedNodes.insert(N);
   }
 
-  SP->replaceRetainedNodes(MDTuple::get(VMContext, RetainedNodes));
+  SP->replaceRetainedNodes(
+      MDTuple::get(VMContext, RetainedNodes.getArrayRef()));
 }
 
 void DIBuilder::finalize() {
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index 1edc6d8c159e9..5e58b5c95f776 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1904,6 +1904,7 @@ InstrLowerer::getOrCreateRegionBitmaps(InstrProfMCDCBitmapInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       BitmapPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
index 29bd0037a7e8b..43a1db5b2cf4e 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
@@ -22,7 +22,7 @@ define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
   call void @llvm.instrprof.mcdc.parameters(ptr @__profn_foo, i64 12345678, i32 1)
   call void @llvm.instrprof.mcdc.tvbitmap.update(ptr @__profn_foo, i64 12345678, i32 0, ptr %mcdc.addr)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -50,6 +50,7 @@ declare void @llvm.instrprof.mcdc.tvbitmap.update(ptr, i64, i32, ptr)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, column: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram


From llvmlistbot at llvm.org  Mon Jul  6 19:31:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 06 Jul 2026 19:31:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a4c6506.510f576d.1b3cb9.f32a@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207342

>From 6695b3e0d8bddeab555e1ca36802df2decb85e82 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH 1/2] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 +++++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 24 +++++++++++++++++++++
 mlir/test/Target/LLVMIR/nvvmir.mlir         | 24 +++++++++++++++++++++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..9445770ebb026 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,30 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {
diff --git a/mlir/test/Target/LLVMIR/nvvmir.mlir b/mlir/test/Target/LLVMIR/nvvmir.mlir
index f2888025d8a08..f726efe3d39b7 100644
--- a/mlir/test/Target/LLVMIR/nvvmir.mlir
+++ b/mlir/test/Target/LLVMIR/nvvmir.mlir
@@ -293,6 +293,30 @@ llvm.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
 // f32 return type, f32 accumulate type
 // CHECK-LABEL: @nvvm_mma_m16n8k16_f32_f32
 llvm.func @nvvm_mma_m16n8k16_f32_f32(%a0 : vector<2xf16>, %a1 : vector<2xf16>,

>From 68a9d4634942de01c48f1d2366bc47d4b134e2c7 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:43:32 +0000
Subject: [PATCH 2/2] [NVGPU] Add FP8 (e4m3/e5m2) support to nvgpu.mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..c647fdc1e7a75 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -329,6 +329,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..f5232283dfc2c 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) || aType.isF8E4M3FN() ||
+             aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Mon Jul  6 22:09:25 2026
From: llvmlistbot at llvm.org (Kareem Ergawy)
Date: Mon, 06 Jul 2026 22:09:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add opt-in
 query caching; use it in flang LICM (PR #207030)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207030@github.com>
Message-ID: <6a4c8a05.f06f4994.80921.115f@mx.google.com>


ergawy wrote:

> Can you clarify who should own the cache in this model?

The owner would be the user of AA, so similar to how `QueryCacheScope` is currently being used.

I attached [tmp2.patch](https://github.com/user-attachments/files/29729921/tmp2.patch) to clarify what I meant a bit better.



https://github.com/llvm/llvm-project/pull/207030

From llvmlistbot at llvm.org  Mon Jul  6 22:52:57 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Mon, 06 Jul 2026 22:52:57 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Number MLIR blocks;
 require GraphHasNodeNumbers in llvm::LoopInfoBase (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4c9439.ac469dc7.3a57b1.2c76@mx.google.com>


================
@@ -422,6 +435,11 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region; -1u while the block has no parent. See
+  /// getNumber().
+  unsigned blockNumber = -1u;
----------------
MaskRay wrote:

On int nit: LLVM's BasicBlock uses unsigned, and the API returns unsigned.

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Tue Jul  7 00:54:49 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 07 Jul 2026 00:54:49 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cb0c9.af6ba3ed.36d9db.4fbe@mx.google.com>


https://github.com/madhur13490 updated https://github.com/llvm/llvm-project/pull/201077

>From 3b92820148baf17c603fdced40b84d750321ba72 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Wed, 15 Apr 2026 01:00:40 -0700
Subject: [PATCH 1/2] Enforce single-operand form for llvm.loop.distribute
 metadata

This patch implements Phase 1 of the RFC "Enforce Single-Operand Format for All
.enable Metadata Nodes". The two-operand boolean form
!{!"llvm.loop.distribute.enable", i1 0/1} is replaced by a
single-operand enable/disable pair:

  !{!"llvm.loop.distribute.enable"}    ; force distribution
  !{!"llvm.loop.distribute.disable"}   ; suppress distribution
---
 clang/lib/CodeGen/CGLoopInfo.cpp              | 22 ++++-----
 .../test/CodeGenCXX/pragma-followup_inner.cpp |  2 +-
 .../test/CodeGenCXX/pragma-followup_outer.cpp |  2 +-
 .../CodeGenCXX/pragma-loop-distribute.cpp     |  2 +-
 clang/test/CodeGenCXX/pragma-loop.cpp         |  2 +-
 llvm/docs/LangRef.rst                         | 15 +++---
 llvm/lib/Analysis/LoopAccessAnalysis.cpp      | 11 ++---
 llvm/lib/IR/Verifier.cpp                      |  6 +++
 llvm/lib/Transforms/Scalar/LoopDistribute.cpp | 13 ++---
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp |  3 +-
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |  3 ++
 .../Analysis/LoopAccessAnalysis/pr64637.ll    |  2 +-
 .../Other/optimization-remarks-lazy-bfi.ll    |  2 +-
 .../IRCE/add-metadata-pre-post-loops.ll       |  2 +-
 .../test/Transforms/IRCE/correct-loop-info.ll |  2 +-
 .../Transforms/IRCE/loop-guarded-bounds.ll    |  2 +-
 llvm/test/Transforms/IRCE/pre_post_loops.ll   |  2 +-
 .../IRCE/range_intersect_miscompile.ll        |  2 +-
 .../IRCE/unsigned_comparisons_ugt.ll          |  2 +-
 .../IRCE/unsigned_comparisons_ult.ll          |  2 +-
 .../Inline/dilocation-loop-metadata-update.ll |  4 +-
 .../LoopDistribute/basic-with-memchecks.ll    |  2 +-
 .../diagnostics-with-hotness.ll               |  2 +-
 .../Transforms/LoopDistribute/diagnostics.ll  |  2 +-
 .../disable_nonforced_enable.ll               |  2 +-
 .../Transforms/LoopDistribute/early-exit.ll   |  2 +-
 .../Transforms/LoopDistribute/followup.ll     |  2 +-
 .../Transforms/LoopDistribute/metadata.ll     |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 llvm/test/Verifier/llvm.loop.distribute.ll    | 49 +++++++++++++++++++
 llvm/unittests/Analysis/LoopInfoTest.cpp      |  8 +--
 .../Target/LLVMIR/LoopAnnotationImporter.cpp  |  6 ++-
 .../LLVMIR/LoopAnnotationTranslation.cpp      |  7 ++-
 .../Target/LLVMIR/Import/metadata-loop.ll     |  2 +-
 mlir/test/Target/LLVMIR/loop-metadata.mlir    |  2 +-
 36 files changed, 129 insertions(+), 72 deletions(-)
 create mode 100644 llvm/test/Verifier/llvm.loop.distribute.ll

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 369b052bf7a28..8961db50f9f84 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -334,11 +334,12 @@ LoopInfo::createLoopDistributeMetadata(const LoopAttributes &Attrs,
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
-      NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.append(
+          LoopProperties.begin(), LoopProperties.end());
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx,
+          {MDString::get(
+              Ctx, "llvm.loop.distribute.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createLoopVectorizeMetadata(Attrs, LoopProperties,
@@ -347,16 +348,15 @@ LoopInfo::createLoopDistributeMetadata(const LoopAttributes &Attrs,
 
   bool FollowupHasTransforms = false;
   SmallVector<Metadata *, 4> Followup =
-      createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
+      createLoopVectorizeMetadata(
+          Attrs, LoopProperties, FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
-  Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                      ConstantAsMetadata::get(ConstantInt::get(
-                          llvm::Type::getInt1Ty(Ctx),
-                          (Attrs.DistributeEnable == LoopAttributes::Enable)))};
-  Args.push_back(MDNode::get(Ctx, Vals));
+  Args.push_back(MDNode::get(
+      Ctx,
+      {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
 
   if (FollowupHasTransforms)
     Args.push_back(
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..2cf2dba5b913d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_inner(int n, int *x) {
 
 // CHECK-DAG: ![[INNERLOOP_3]] = distinct !{![[INNERLOOP_3]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..2e907fb814b3d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -19,7 +19,7 @@ extern "C" void followup_outer(int n, int *x) {
 
 // CHECK-DAG: ![[LOOP_3:[0-9]+]] = distinct !{![[LOOP_3:[0-9]+]], [[MP:![0-9]+]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4:[0-9]+]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
index 98aef21da2acd..66f0fb07b51fb 100644
--- a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
@@ -20,5 +20,5 @@ void while_test(int *List, int Length, int *List2, int Length2) {
 }
 
 // CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], [[MP:![0-9]+]], ![[DISTRIBUTE_ENABLE:.*]]}
-// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable"}
 // CHECK: [[LOOP_2]] = distinct !{[[LOOP_2]], [[MP]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..f9106725ad99d 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -248,7 +248,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[UNROLL_32:[0-9]+]] = !{!"llvm.loop.unroll.count", i32 32}
 // CHECK-DAG: ![[UNROLL_FULL:[0-9]+]] = !{!"llvm.loop.unroll.full"}
 
-// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.disable"}
 
 // CHECK-DAG: ![[INTERLEAVE_2:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 2}
 // CHECK-DAG: ![[INTERLEAVE_4:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 4}
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 86325ef731381..bfd1d70e49ad1 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -8128,18 +8128,17 @@ this is only performed if the entire loop cannot be vectorized due to unsafe
 memory dependencies.  The transformation will attempt to isolate the unsafe
 dependencies into their own loop.
 
-This metadata can be used to selectively enable or disable distribution of the
-loop.  The first operand is the string ``llvm.loop.distribute.enable`` and the
-second operand is a bit. If the bit operand value is 1 distribution is
-enabled. A value of 0 disables distribution:
+This metadata can be used to selectively enable or disable
+distribution of the loop.  Each node has a single operand
+containing the name string:
 
 .. code-block:: llvm
 
-   !0 = !{!"llvm.loop.distribute.enable", i1 0}
-   !1 = !{!"llvm.loop.distribute.enable", i1 1}
+   !0 = !{!"llvm.loop.distribute.enable"}
+   !1 = !{!"llvm.loop.distribute.disable"}
 
-This metadata should be used in conjunction with ``llvm.loop`` loop
-identification metadata.
+This metadata should be used in conjunction with ``llvm.loop``
+loop identification metadata.
 
 '``llvm.loop.distribute.followup_coincident``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 2b9efd22131c6..ca5354930b018 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2901,14 +2901,9 @@ void LoopAccessInfo::emitUnsafeDependenceRemark() {
   LLVM_DEBUG(dbgs() << "LAA: unsafe dependent memory operations in loop\n");
 
   // Emit remark for first unsafe dependence
-  bool HasForcedDistribution = false;
-  std::optional<const MDOperand *> Value =
-      findStringMetadataForLoop(TheLoop, "llvm.loop.distribute.enable");
-  if (Value) {
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    HasForcedDistribution = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
-  }
+  bool HasForcedDistribution =
+      getBooleanLoopAttribute(
+          TheLoop, "llvm.loop.distribute.enable");
 
   const std::string Info =
       HasForcedDistribution
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index d4ade9c7ce534..0706737e45cdc 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1145,6 +1145,12 @@ void Verifier::visitMDNode(const MDNode &MD, AreDebugLocsAllowed AllowLocs) {
           &MD);
   }
 
+  if (MD.getNumOperands() > 0 &&
+      (MD.getOperand(0).equalsStr("llvm.loop.distribute.enable") ||
+       MD.getOperand(0).equalsStr("llvm.loop.distribute.disable")))
+    Check(MD.getNumOperands() == 1,
+          "Expected one operand for llvm.loop.distribute metadata", &MD);
+
   // Check these last, so we diagnose problems in operands first.
   Check(!MD.isTemporary(), "Expected no forward declarations!", &MD);
   Check(MD.isResolved(), "All nodes should be resolved!", &MD);
diff --git a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
index 00eac14845c57..006cc43de3ca5 100644
--- a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -938,14 +938,11 @@ class LoopDistributeForLoop {
   /// Check whether the loop metadata is forcing distribution to be
   /// enabled/disabled.
   void setForced() {
-    std::optional<const MDOperand *> Value =
-        findStringMetadataForLoop(L, "llvm.loop.distribute.enable");
-    if (!Value)
-      return;
-
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    IsForced = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
+    if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
+      IsForced = true;
+    else if (getBooleanLoopAttribute(
+                 L, "llvm.loop.distribute.disable"))
+      IsForced = false;
   }
 
   Loop *L;
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 72fae62e56e88..74cd2a729d285 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -453,7 +453,8 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
       Context,
-      {MDString::get(Context, "llvm.loop.distribute.enable"), FalseVal});
+      {MDString::get(
+          Context, "llvm.loop.distribute.disable")});
   MDNode *NewLoopID =
       MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
                             DisableLICMVersioning, DisableDistribution});
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 7573467917c73..290bd73fa2c18 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -429,6 +429,9 @@ TransformationMode llvm::hasVectorizeTransformation(const Loop *L) {
 }
 
 TransformationMode llvm::hasDistributeTransformation(const Loop *L) {
+  if (getBooleanLoopAttribute(L, "llvm.loop.distribute.disable"))
+    return TM_SuppressedByUser;
+
   if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
     return TM_ForcedByUser;
 
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
index d3e589cf99cf3..7b25d1fffcbba 100644
--- a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
+++ b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
@@ -76,4 +76,4 @@ for.body:                                         ; preds = %for.body, %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Other/optimization-remarks-lazy-bfi.ll b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
index c5099bb2179e4..5fc34cc14bdc8 100644
--- a/llvm/test/Other/optimization-remarks-lazy-bfi.ll
+++ b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
@@ -75,7 +75,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 99, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
index eb72907730c6d..3a360814e361e 100644
--- a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
+++ b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
@@ -76,7 +76,7 @@ attributes #0 = { alwaysinline }
 !3 = !{!"llvm.loop.unroll.disable"}
 !4 = !{!"llvm.loop.vectorize.enable", i1 false}
 !5 = !{!"llvm.loop.licm_versioning.disable"}
-!6 = !{!"llvm.loop.distribute.enable", i1 false}
+!6 = !{!"llvm.loop.distribute.disable"}
 !7 = !{}
 !8 = distinct !{!8, !3, !4, !5}
 !9 = distinct !{!9, !3, !4, !5}
diff --git a/llvm/test/Transforms/IRCE/correct-loop-info.ll b/llvm/test/Transforms/IRCE/correct-loop-info.ll
index c75de167681ad..b1ce5383b24b8 100644
--- a/llvm/test/Transforms/IRCE/correct-loop-info.ll
+++ b/llvm/test/Transforms/IRCE/correct-loop-info.ll
@@ -181,6 +181,6 @@ declare void @pluto()
 !1 = !{!"llvm.loop.unroll.disable"}
 !2 = !{!"llvm.loop.vectorize.enable", i1 false}
 !3 = !{!"llvm.loop.licm_versioning.disable"}
-!4 = !{!"llvm.loop.distribute.enable", i1 false}
+!4 = !{!"llvm.loop.distribute.disable"}
 !5 = !{}
 !6 = distinct !{!6, !1, !2, !3, !4}
diff --git a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
index ea169987cf2fc..61cfef6f1492a 100644
--- a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
+++ b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
@@ -130,7 +130,7 @@ exit:                                       ; preds = %guarded, %loop_, %prehead
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/pre_post_loops.ll b/llvm/test/Transforms/IRCE/pre_post_loops.ll
index da2faa34719af..2807adba35940 100644
--- a/llvm/test/Transforms/IRCE/pre_post_loops.ll
+++ b/llvm/test/Transforms/IRCE/pre_post_loops.ll
@@ -163,7 +163,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
index 68613feacba32..2e21bb5ad0bd0 100644
--- a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
+++ b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
@@ -490,7 +490,7 @@ deopt:                                          ; preds = %range_check_block
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ; CHECK: [[RNG7]] = !{i32 0, i32 50}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
index 7236cf4817ade..cdd7235ac45b4 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
@@ -447,7 +447,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
index 546e1b0faa1c3..107a3554282ad 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
@@ -654,7 +654,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
index 1bc132663331b..0d8dcfbcf2a8b 100644
--- a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
+++ b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
@@ -47,7 +47,7 @@ entry:
 !7 = !DILocation(line: 6, column: 3, scope: !3)
 !8 = !DILocation(line: 7, column: 22, scope: !3)
 !9 = !{!"llvm.loop.mustprogress"}
-!10 = !{!"llvm.loop.distribute.enable", i1 true}
+!10 = !{!"llvm.loop.distribute.enable"}
 !11 = !{!"llvm.loop.distribute.followup_all", !7, !8, !9, !12, !13, !14}
 !12 = !{!"llvm.loop.vectorize.width", i32 8}
 !13 = !{!"llvm.loop.vectorize.enable", i1 true}
@@ -66,7 +66,7 @@ entry:
 ; CHECK: [[META7]] = !DILocation(line: 6, column: 3, scope: [[DBG3]])
 ; CHECK: [[META8]] = !DILocation(line: 7, column: 22, scope: [[DBG3]])
 ; CHECK: [[META9]] = !{!"llvm.loop.mustprogress"}
-; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable"}
 ; CHECK: [[META11]] = !{!"llvm.loop.distribute.followup_all", [[META7]], [[META8]], [[META9]], [[META12:![0-9]+]], [[META13:![0-9]+]], [[META14:![0-9]+]]}
 ; CHECK: [[META12]] = !{!"llvm.loop.vectorize.width", i32 8}
 ; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
index 2828882afe779..dcd231a900ee2 100644
--- a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
+++ b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
@@ -288,5 +288,5 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: ![[PROF1]] = !{!"unknown", !"loop-versioning"}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
index b5ef3578eadd9..4f2f76191e615 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
@@ -74,7 +74,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 2000, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics.ll b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
index e6a0d83bd63d2..80b217d0abb2b 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
@@ -194,7 +194,7 @@ attributes #1 = { nounwind convergent }
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = distinct !DISubprogram(name: "not_forced", scope: !1, file: !1, line: 8, type: !8, isLocal: false, isDefinition: true, scopeLine: 8, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !2)
 !23 = !DILocation(line: 9, column: 20, scope: !22)
 !24 = !DILocation(line: 9, column: 3, scope: !22)
diff --git a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
index 45a2d31256a27..eff1621cd7ada 100644
--- a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
+++ b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
@@ -43,4 +43,4 @@ for.end:
   ret void
 }
 
-!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable", i32 1}}
+!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable"}}
diff --git a/llvm/test/Transforms/LoopDistribute/early-exit.ll b/llvm/test/Transforms/LoopDistribute/early-exit.ll
index 9353d842523f4..d2ad9b1bcd6c5 100644
--- a/llvm/test/Transforms/LoopDistribute/early-exit.ll
+++ b/llvm/test/Transforms/LoopDistribute/early-exit.ll
@@ -89,4 +89,4 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/LoopDistribute/followup.ll b/llvm/test/Transforms/LoopDistribute/followup.ll
index ae418ad9e6b42..2753bad5e1c1e 100644
--- a/llvm/test/Transforms/LoopDistribute/followup.ll
+++ b/llvm/test/Transforms/LoopDistribute/followup.ll
@@ -125,7 +125,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1, !2, !3, !4, !5}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = !{!"llvm.loop.distribute.followup_all", !{!"FollowupAll"}}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !{!"FollowupCoincident", i1 false}}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !{!"FollowupSequential", i32 8}}
diff --git a/llvm/test/Transforms/LoopDistribute/metadata.ll b/llvm/test/Transforms/LoopDistribute/metadata.ll
index b0e461fe0ea75..8b74ac4249550 100644
--- a/llvm/test/Transforms/LoopDistribute/metadata.ll
+++ b/llvm/test/Transforms/LoopDistribute/metadata.ll
@@ -131,6 +131,6 @@ for.end:                                          ; preds = %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.distribute.enable", i1 false}
+!3 = !{!"llvm.loop.distribute.disable"}
diff --git a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
index 2c1dff7ee5a5f..049880a97a211 100644
--- a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
@@ -65,6 +65,6 @@ cleanup:                                          ; preds = %cleanup.loopexit, %
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
index a95b6ec15b129..ec87c9635dc32 100644
--- a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
@@ -147,7 +147,7 @@ while.end:                                        ; preds = %while.cond
 
 !0 = !{i32 2, !"Debug Info Version", i32 3}
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = !{!"llvm.loop.distribute.enable"}
 !3 = distinct !{!3, !4}
 !4 = !{!"llvm.loop.mustprogress"}
 !5 = distinct !{!5, !4, !6}
@@ -161,7 +161,7 @@ while.end:                                        ; preds = %while.cond
 !13 = !DILocation(line: 9, column: 23, scope: !12)
 
 ; CHECK: !1 = distinct !{!1, !2}
-; CHECK: !2 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !2 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !3 = distinct !{!3, !4}
 ; CHECK: !4 = !{!"llvm.loop.mustprogress"}
 ; CHECK: !5 = distinct !{!5, !4, !6}
diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
new file mode 100644
index 0000000000000..9e1e8138e93dc
--- /dev/null
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -0,0 +1,49 @@
+; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
+; single-operand validation.
+
+; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
+; DEFINE:   FileCheck %s -allow-empty -check-prefix
+
+define void @test() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+!0 = distinct !{!0, !1}
+
+; GOOD-NOT: {{.}}
+
+;      BAD: Expected one operand for llvm.loop.distribute metadata
+
+; Single-operand enable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Single-operand disable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Two-operand enable with boolean false (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand enable with boolean true (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand disable (rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
diff --git a/llvm/unittests/Analysis/LoopInfoTest.cpp b/llvm/unittests/Analysis/LoopInfoTest.cpp
index 4828573c32c10..5321f72aa1784 100644
--- a/llvm/unittests/Analysis/LoopInfoTest.cpp
+++ b/llvm/unittests/Analysis/LoopInfoTest.cpp
@@ -72,7 +72,7 @@ TEST(LoopInfoTest, LoopWithSingleLatch) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -122,7 +122,7 @@ TEST(LoopInfoTest, LoopWithMultipleLatches) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1397,7 +1397,7 @@ TEST(LoopInfoTest, LoopUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1440,7 +1440,7 @@ TEST(LoopInfoTest, LoopNonLatchUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index e4905423347a2..4022afc433a61 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -358,8 +358,10 @@ FailureOr<LoopLICMAttr> LoopMetadataConversion::convertLICMAttr() {
 }
 
 FailureOr<LoopDistributeAttr> LoopMetadataConversion::convertDistributeAttr() {
-  FailureOr<BoolAttr> disable =
-      lookupBoolNode("llvm.loop.distribute.enable", true);
+  FailureOr<BoolAttr> disable = lookupBooleanUnitNode(
+      "llvm.loop.distribute.enable",
+      "llvm.loop.distribute.disable",
+      /*negated=*/true);
   FailureOr<LoopAnnotationAttr> followupCoincident =
       lookupFollowupNode("llvm.loop.distribute.followup_coincident");
   FailureOr<LoopAnnotationAttr> followupSequential =
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
index f3ec18c9074e4..41ba31106f788 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
@@ -162,7 +162,12 @@ void LoopAnnotationConversion::convertLoopOptions(LoopLICMAttr options) {
 }
 
 void LoopAnnotationConversion::convertLoopOptions(LoopDistributeAttr options) {
-  convertBoolNode("llvm.loop.distribute.enable", options.getDisable(), true);
+  if (auto disable = options.getDisable()) {
+    if (disable.getValue())
+      addUnitNode("llvm.loop.distribute.disable");
+    else
+      addUnitNode("llvm.loop.distribute.enable");
+  }
   convertFollowupNode("llvm.loop.distribute.followup_coincident",
                       options.getFollowupCoincident());
   convertFollowupNode("llvm.loop.distribute.followup_sequential",
diff --git a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
index 20431a7412bd1..62cf5dd3e1605 100644
--- a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
+++ b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
@@ -219,7 +219,7 @@ end:
 }
 
 !1 = distinct !{!1, !2, !3, !4, !5, !6}
-!2 = !{!"llvm.loop.distribute.enable", i1 0}
+!2 = !{!"llvm.loop.distribute.disable"}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !9}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !9}
 !5 = !{!"llvm.loop.distribute.followup_fallback", !9}
diff --git a/mlir/test/Target/LLVMIR/loop-metadata.mlir b/mlir/test/Target/LLVMIR/loop-metadata.mlir
index 2fe4a994aeb66..a0636cfe90297 100644
--- a/mlir/test/Target/LLVMIR/loop-metadata.mlir
+++ b/mlir/test/Target/LLVMIR/loop-metadata.mlir
@@ -185,7 +185,7 @@ llvm.func @distributeOptions() {
 // CHECK-DAG: ![[NON_FORCED:[0-9]+]] = !{!"llvm.loop.disable_nonforced"}
 // CHECK-DAG: ![[FOLLOWUP:[0-9]+]] = distinct !{![[FOLLOWUP]], ![[NON_FORCED]]}
 // CHECK-DAG: ![[LOOP_NODE]] = distinct !{![[LOOP_NODE]], !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}}
-// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.disable"}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_coincident", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_sequential", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_fallback", ![[FOLLOWUP]]}

>From ab71a6c0e499e3304292505870c5e98f8b669d57 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Tue, 7 Jul 2026 00:52:57 -0700
Subject: [PATCH 2/2] fixup! Enforce single-operand form for
 llvm.loop.distribute metadata

auto-upgrade old two-operand llvm.loop.distribute.enable metadata
---
 llvm/include/llvm/IR/AutoUpgrade.h            |   7 ++-
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    |   3 ++
 llvm/lib/IR/AutoUpgrade.cpp                   |  47 ++++++++++++++++--
 .../Bitcode/upgrade-loop-distribute-enable.ll |  37 ++++++++++++++
 .../upgrade-loop-distribute-enable.ll.bc      | Bin 0 -> 2080 bytes
 llvm/test/Verifier/llvm.loop.distribute.ll    |  30 ++++-------
 6 files changed, 101 insertions(+), 23 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc

diff --git a/llvm/include/llvm/IR/AutoUpgrade.h b/llvm/include/llvm/IR/AutoUpgrade.h
index 540d60afc5a01..f5d04d19160ea 100644
--- a/llvm/include/llvm/IR/AutoUpgrade.h
+++ b/llvm/include/llvm/IR/AutoUpgrade.h
@@ -108,7 +108,12 @@ namespace llvm {
 
   /// Check whether a string looks like an old loop attachment tag.
   inline bool mayBeOldLoopAttachmentTag(StringRef Name) {
-    return Name.starts_with("llvm.vectorizer.");
+    // "llvm.loop.distribute.enable" is intentionally included: the current
+    // single-operand form shares the tag with the removed two-operand form
+    // (!{!"llvm.loop.distribute.enable", i1 X}), so we can only decide by
+    // inspecting the operands, which happens in upgradeLoopArgument().
+    return Name.starts_with("llvm.vectorizer.") ||
+           Name == "llvm.loop.distribute.enable";
   }
 
   /// Upgrade the loop attachment metadata node.
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..373f1f9c91563 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -2429,6 +2429,9 @@ Error MetadataLoader::MetadataLoaderImpl::parseOneMetadata(
   }
   case bitc::METADATA_STRINGS: {
     auto CreateNextMDString = [&](StringRef Str) {
+      // Modern bitcode encodes MDStrings via this bulk record, so mirror the
+      // METADATA_STRING check above to arm the loop-attachment upgrader.
+      HasSeenOldLoopTags |= mayBeOldLoopAttachmentTag(Str);
       ++NumMDStringLoaded;
       MetadataList.assignValue(MDString::get(Context, Str), NextMetadataNo);
       NextMetadataNo++;
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 2728897372009..885ffb34cd7e9 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6596,6 +6596,18 @@ void llvm::copyModuleAttrToFunctions(Module &M) {
   }
 }
 
+// Old two-operand form: !{!"llvm.loop.distribute.enable", i1 X}. The new
+// single-operand form uses "llvm.loop.distribute.enable" for X = true and
+// "llvm.loop.distribute.disable" for X = false.
+static bool isOldDistributeEnable(const MDTuple *T) {
+  if (T->getNumOperands() != 2)
+    return false;
+  auto *Tag = dyn_cast_or_null<MDString>(T->getOperand(0));
+  if (!Tag || Tag->getString() != "llvm.loop.distribute.enable")
+    return false;
+  return mdconst::hasa<ConstantInt>(T->getOperand(1));
+}
+
 static bool isOldLoopArgument(Metadata *MD) {
   auto *T = dyn_cast_or_null<MDTuple>(MD);
   if (!T)
@@ -6605,7 +6617,9 @@ static bool isOldLoopArgument(Metadata *MD) {
   auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!S)
     return false;
-  return S->getString().starts_with("llvm.vectorizer.");
+  if (S->getString().starts_with("llvm.vectorizer."))
+    return true;
+  return isOldDistributeEnable(T);
 }
 
 static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
@@ -6629,17 +6643,28 @@ static Metadata *upgradeLoopArgument(Metadata *MD) {
   auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!OldTag)
     return MD;
+
+  LLVMContext &C = T->getContext();
+
+  // Rewrite the old two-operand distribute form to the single-operand pair.
+  if (isOldDistributeEnable(T)) {
+    bool Enable = !mdconst::extract<ConstantInt>(T->getOperand(1))->isZero();
+    return MDTuple::get(
+        C, {MDString::get(C, Enable ? "llvm.loop.distribute.enable"
+                                    : "llvm.loop.distribute.disable")});
+  }
+
   if (!OldTag->getString().starts_with("llvm.vectorizer."))
     return MD;
 
   // This has an old tag.  Upgrade it.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
-  Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
+  Ops.push_back(upgradeLoopTag(C, OldTag->getString()));
   for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
     Ops.push_back(T->getOperand(I));
 
-  return MDTuple::get(T->getContext(), Ops);
+  return MDTuple::get(C, Ops);
 }
 
 MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
@@ -6650,6 +6675,22 @@ MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
   if (none_of(T->operands(), isOldLoopArgument))
     return &N;
 
+  // Fix the old two-operand llvm.loop.distribute.enable nodes in place: the
+  // Verifier rejects any MDNode carrying the distribute tag with more than one
+  // operand, so a leftover reference (from the distinct loop-ID) would still
+  // trigger a diagnostic. In-place mutation is safe on distinct MDNodes.
+  if (T->isDistinct()) {
+    for (unsigned I = 0, E = T->getNumOperands(); I < E; ++I) {
+      auto *OpT = dyn_cast_or_null<MDTuple>(T->getOperand(I));
+      if (OpT && isOldDistributeEnable(OpT))
+        T->replaceOperandWith(I, upgradeLoopArgument(OpT));
+    }
+    if (none_of(T->operands(), isOldLoopArgument))
+      return &N;
+  }
+
+  // Remaining old arguments (e.g. llvm.vectorizer.*) are handled via a wrapper
+  // attachment; the original distinct loop-ID is kept as the first operand.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
   for (Metadata *MD : T->operands())
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
new file mode 100644
index 0000000000000..8f1b5aec2d140
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
@@ -0,0 +1,37 @@
+; Test that older bitcode carrying the two-operand form of
+; "llvm.loop.distribute.enable" is auto-upgraded to the single-operand
+; enable/disable pair on load.
+;
+; RUN: llvm-dis < %s.bc | FileCheck %s
+; RUN: verify-uselistorder < %s.bc
+
+define void @enable_true() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+
+define void @enable_false() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !2
+exit:
+  ret void
+}
+
+; i1 true  -> single-operand enable.
+; i1 false -> disable.
+; CHECK: !{!"llvm.loop.distribute.enable"}
+; CHECK: !{!"llvm.loop.distribute.disable"}
+; The old two-operand nodes must be gone from the module.
+; CHECK-NOT: llvm.loop.distribute.enable", i1
+; CHECK-NOT: llvm.loop.distribute.disable", i1
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = distinct !{!2, !3}
+!3 = !{!"llvm.loop.distribute.enable", i1 false}
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..fdcbda4584522c06b5f5668e53c6c24fede99c65
GIT binary patch
literal 2080
zcmbtVZ%iBK8GntzXA*N~5;pAQ&btdVved*Jh87#!lI=^GD`lIT<^$cbg)_DTnFGcc
z0;J3A*yG4*b#CQlmX=JQPF1UZAuS(Jix$i|%t at _`rt5?n%Ixwd$wUH4*i>zqwC7B>
zec89Y(!F=D?!D)p-=F7so!sp5b2SK6AcQ<Bw*EV>{O;|a{A22S`*Zb2WX1}_s1W)@
zhS2UZ29?2{I=HZC-5ZWv`HKp*(fp<?u6m|IsnwOg=833YD0BY2E?!?7&(U6eP|aSd
z at p{#7blh-`s;VnrH+kc(FO$^=T*o3REpw^Nn^RWTo9%6*>W^hcV_a!G?mkYxHEnMT
zMO9kkC~@goqgSJS5jT3X?V%)7*Oq)nO|^z|W3MNfBcq9OV>3b*X~+$7I^Zu?zwf+o
zLwyXUWZ#_IwX5<VH(%j29#pFK$@C6I+1qGcDM3K{n4@@i@%s5+h#R$k|H5$XbKdx&
zfw;C9IQLP-udaQz*7(os{yTRXMuxxp)5=e`AN~5o`Ok`{hSMi{uKu}q=(Wqg`E;$j
z at 5+yQe)$F8_|b;v1EGEPfE4!M;}BxA-`4<V{}~-Jbg`=Jh4<tLy>H*)+NevbwADwG
ziOPfDo-gN4S0xa-szGQAK59}%$ZV1H*T`&RoGC6QoDpnz2<uPxsn!wkK#gxl at Fta0
zH<zJqO^|-JqO>ixKQFiCI9oz)%MT^OLF}9o8+Hl7AT|<|4-N at ITF~aJ at r@*2WQiV@
zSlPn2+<22ZOD!|xB16sbIzM~SG83~baONqwbtYzw%Psk>n!yh2To=~TC4^v(2*IQf
z?7#-~*x*+c5cX_DIH?r+G(spX45y=q9&X_e3;17W at eNk)$xV=}6J)+LFY)9}hRjB(
zo4n#rNzR<(%<0lREt*oCX-sZOi4E}6rxz at Llv^f6)$hB7V7m|s#=>3LP=^rK2;p%-
zd+TN5AtD~T at zo^0Vke3;z5ys`J++k4%{g?p8FDGATPl#NMd~jyonP~kX_9Nb9JA!z
z^<M+&5Y7%^L+O`}ZYK$e0o`fcO`g2RP|F$Wyi6x0%urjr;`q;Ero3pG<ZM95(>d!Z
zXBmT~n7F<RkP2rZA0;-T$3p4oi4`@zF^jLc at h5Cu=C(t(TC80xQp=ElOKX{o*`{Lq
z<Dx0Sne(D`I%Cd>fYF+FP4y`;$f^$rY!|e5=JAaV;z2>NMH6c*u|^YrbK^Z~e1*kf
z`J|w at U%*%0_*ZP*mm54e7o`@9wHpkz&QMu~0tN3e-}2K}EHj*KT5e9o%=z{iP#?gK
zK%Oq)Y*5gCd>7wjiDCi2Zzt}L!(T+1y#E~xu$)YTRV0SP3m|1Q?l2cbpw0$L<?Y#V
z>gEJhaOjpZq`$tC#fy@@1O!=QIVLus!TMdo$#LP;2z+}rC9 at _(Yf3aNaIIjcJTRtm
zG2d|VI!N_ahB_9l at gF_K;+r(FSx{`ciESwqzE)7I(?oFrz`kXV4R&F}N<n*U-?V4}
zAO4mz<$*)FIT^F&<yIgwC92x0=E1t)s0s3XRQFg0I?i8JfCC{8XwkIid5sW^U?cBf
z!+MY`GvzQ%$}KaZ>1mm5Mr=(%Ze2?>^|?$3?9Bs#nJP(re)MrE2qIQyrDA!)#-g#3
zQW8e+f4?3ZNk7UuTBjkgnB?1Wt`({e+`9 at jK1?ID7wY&;or~tiRN6|fH+O at MLc?ln
zYug)jjwc^wb4s&wG~uzo3j6lM6$lKTF$8*hg9dMRe>l|b83_9fz8>Ba at WESz-i6-K
z0{7j!LVBn<Y@|Jy2Aw5|%nRa$b#i{9>YhVt3Bd7=m%rJ((t+P~Lx~WpQlSuc-T3xo
zBr1_1Rq&7im?ZzU2QQ1JDH%YqrNpuyLo*<kisTYauJY1+FRGj4$%P3rw_`d!BO|s!
zD9|l}?;hI`zv2%jka`gcS_JD^!0FJ3a&pTdJ^1uCMB_6@@HOx<_^yK}!VG<4LGdI>
zJfQK-SzN;Si5ejV+_AhBSiKAlL2et5S)n;3MX8<g`W;^Dl%rcLtbuB at H|}g_R9OWB
z-x`LZ>dAt=be7fH%Wp*}yQBH#_!Z<hkt>J2G7l=9VLuqP?}24${O_2a|0`x6y_1^}
zS09L at _v9L6E&c0<aF#v`PkZDkn}laa-jm_bfbZ)^KOgA#8Ji5Ofj!k7rkgxwqshoN
zHk&-Wzp<tBdA7;#?euw at S&ymN+wvUSWNh^CEyjjGZzmsU=;}S=YdFJuy9PphgJh_~
x!BFpMUuU?qXbAiI!wt^vaA&XA*Dw$~72>_V!;<9=|JQPdcM>rK0;P-*`WG%g;B5c^

literal 0
HcmV?d00001

diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
index 9e1e8138e93dc..65ca27720542b 100644
--- a/llvm/test/Verifier/llvm.loop.distribute.ll
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -1,8 +1,7 @@
 ; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
 ; single-operand validation.
 
-; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
-; DEFINE:   FileCheck %s -allow-empty -check-prefix
+; DEFINE: %{VERIFY} = llvm-as -disable-output %t 2>&1
 
 define void @test() {
 entry:
@@ -14,36 +13,29 @@ exit:
 }
 !0 = distinct !{!0, !1}
 
-; GOOD-NOT: {{.}}
-
 ;      BAD: Expected one operand for llvm.loop.distribute metadata
 
 ; Single-operand enable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Single-operand disable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Two-operand enable with boolean false (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand enable with boolean true (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand disable (rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD


From llvmlistbot at llvm.org  Tue Jul  7 00:54:57 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 07 Jul 2026 00:54:57 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cb0d1.ff037943.2e959a.ad75@mx.google.com>


================

----------------
madhur13490 wrote:

Done.

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 00:55:13 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 07 Jul 2026 00:55:13 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cb0e1.ac469dc7.3a57b1.6abc@mx.google.com>


madhur13490 wrote:

> [critical] With this change, there will be an hard error with metadata with the old format. We will need the AutoUpgrade otherwise older .bc files will be rejected, violating the IR backwards compatibility guarantee
> 
> > The current LLVM version supports loading any bitcode since version 3.0.
> 
> > Non-debug metadata is defined to be safe to drop, so a valid way to upgrade it is to drop it. That is not very user friendly and a bit more effort is expected, but no promises are made.
> 
> https://llvm.org/docs/DeveloperPolicy.html#ir-backwards-compatibility

Done changes in AutoUpgrade.

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 01:16:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 01:16:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Fix crash on 0D vector in
 vector-to-xegpu transfer lowering (PR #205812)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205812@github.com>
Message-ID: <6a4cb5f2.510f576d.1b3cb9.93f2@mx.google.com>


Chennesxu wrote:

 @Jianhui-Li @charithaintc All CI passed! Could someone help merge this? Thanks!

https://github.com/llvm/llvm-project/pull/205812

From llvmlistbot at llvm.org  Tue Jul  7 01:54:38 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Tue, 07 Jul 2026 01:54:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
Message-ID: <llvm.org/llvm/llvm-project/pull/207933@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/207933

Adds profile compliance support for MXFP row_gather.

>From c363b2f0946198528bb58950f6f85ecaa8104c73 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 29 Jun 2026 16:00:18 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP row_gather

Adds profile compliance support for MXFP row_gather.

Change-Id: I3a256bd60e857955c5a849b951143fdb2df48fab
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 114 +++++++++++++-----
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  17 ++-
 mlir/test/Dialect/Tosa/availability.mlir      |   2 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |   8 ++
 mlir/test/Dialect/Tosa/ops.mlir               |   8 ++
 .../tosa-validation-version-1p0-invalid.mlir  |   9 ++
 .../tosa-validation-version-1p1-valid.mlir    |  18 +++
 7 files changed, 140 insertions(+), 36 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..98b0c47748942 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -992,37 +992,89 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..ffa80a24b231e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2596,7 +2596,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2608,6 +2616,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2624,11 +2634,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..151a9a27302ae 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..e42650dc88cc1 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..81137921be19f 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1438,6 +1438,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..87d407c11e386 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9aee861c6a8d3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


From llvmlistbot at llvm.org  Tue Jul  7 01:55:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 01:55:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a4cbef7.13be6df8.145c.c24e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-tosa

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

Adds profile compliance support for MXFP row_gather.

---
Full diff: https://github.com/llvm/llvm-project/pull/207933.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+83-31) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+13-4) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid_extension.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+9) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+18) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..98b0c47748942 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -992,37 +992,89 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..ffa80a24b231e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2596,7 +2596,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2608,6 +2616,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2624,11 +2634,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..151a9a27302ae 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..e42650dc88cc1 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..81137921be19f 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1438,6 +1438,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..87d407c11e386 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9aee861c6a8d3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207933

From llvmlistbot at llvm.org  Tue Jul  7 01:55:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 01:55:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a4cbef7.f8b79514.1d52d0.e403@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

Adds profile compliance support for MXFP row_gather.

---
Full diff: https://github.com/llvm/llvm-project/pull/207933.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+83-31) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+13-4) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid_extension.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+9) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+18) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..98b0c47748942 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -992,37 +992,89 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..ffa80a24b231e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2596,7 +2596,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2608,6 +2616,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2624,11 +2634,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..151a9a27302ae 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..e42650dc88cc1 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..81137921be19f 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1438,6 +1438,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..87d407c11e386 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9aee861c6a8d3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207933

From llvmlistbot at llvm.org  Tue Jul  7 02:16:03 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Tue, 07 Jul 2026 02:16:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a4cc3d3.f8b79514.1d52d0.2aee@mx.google.com>


================
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
----------------
lhutton1 wrote:

Note that these changes simply add a typed zero point, as discussed here: https://github.com/llvm/llvm-project/pull/203894#discussion_r3465756850

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Tue Jul  7 02:19:24 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Tue, 07 Jul 2026 02:19:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
Message-ID: <llvm.org/llvm/llvm-project/pull/207944@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/207944

Adds profile compliance support for MXFP reshape.

>From 26c87cc55501da0243f318523dbc12311aafda79 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 29 Jun 2026 17:07:15 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP reshape

Adds profile compliance support for MXFP reshape.

Change-Id: I91354f8db555fc000835efe18dd846506025c781
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 26 ++++++++++++++++++-
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  | 10 ++++++-
 mlir/test/Dialect/Tosa/availability.mlir      |  2 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  8 ++++++
 mlir/test/Dialect/Tosa/ops.mlir               |  8 ++++++
 .../tosa-validation-version-1p0-invalid.mlir  |  9 +++++++
 .../tosa-validation-version-1p1-valid.mlir    | 18 +++++++++++++
 7 files changed, 78 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..8f840cf01e525 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -938,7 +938,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.reverse",
      {{{Extension::int64}, {{{i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..696b562a6447b 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2334,7 +2334,11 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2,
+               Tosa_EXT_MX_FP8E4M3, Tosa_EXT_MX_FP8E5M2,
+               Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -2348,6 +2352,8 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: reshape_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_ReshapeBlockScaledOp
     : Tosa_InferTensorTypeOp<"reshape_block_scaled", [Pure]> {
@@ -2360,6 +2366,8 @@ def Tosa_ReshapeBlockScaledOp
     single non-block-scaled tensor of any rank. No data conversion happens during
     a reshape operation. Reshape must retain the relationship between values
     and their scale in a block for block-scaled content.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let hasVerifier = 1;
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..21099ed13b3e9 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -542,7 +542,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
 func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xf32>, !tosa.shape<2>) -> tensor<1x819xf32>
   return %0 : tensor<1x819xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..1370a66f273f4 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -211,6 +211,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xbf16>) -> tensor<1x819xbf16> {
   return %0 : tensor<1x819xbf16>
 }
 
+// -----
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_reverse(%arg0: tensor<13x21x3xbf16>) -> tensor<13x21x3xbf16> {
   // expected-error at +1 {{'tosa.reverse' op illegal: requires any of [bf16] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..a4c87f407a9e7 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -904,6 +904,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   return %0 : tensor<1x819xf32>
 }
 
+// -----
+// CHECK-LABEL: reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: reshape_unranked_output
 func.func @test_reshape_unranked_output(%arg0: tensor<13x21x3xf32>) -> tensor<*xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..985c0f00c9027 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..1d0b64dd9bbe1 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_reshape_mxfp_mxint8
+func.func @test_reshape_mxfp_mxint8(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


From llvmlistbot at llvm.org  Tue Jul  7 02:20:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 02:20:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a4cc4c2.f06f4994.80921.75cb@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-tosa

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

Adds profile compliance support for MXFP reshape.

---
Full diff: https://github.com/llvm/llvm-project/pull/207944.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+25-1) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+9-1) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid_extension.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+9) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+18) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..8f840cf01e525 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -938,7 +938,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.reverse",
      {{{Extension::int64}, {{{i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..696b562a6447b 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2334,7 +2334,11 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2,
+               Tosa_EXT_MX_FP8E4M3, Tosa_EXT_MX_FP8E5M2,
+               Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -2348,6 +2352,8 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: reshape_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_ReshapeBlockScaledOp
     : Tosa_InferTensorTypeOp<"reshape_block_scaled", [Pure]> {
@@ -2360,6 +2366,8 @@ def Tosa_ReshapeBlockScaledOp
     single non-block-scaled tensor of any rank. No data conversion happens during
     a reshape operation. Reshape must retain the relationship between values
     and their scale in a block for block-scaled content.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let hasVerifier = 1;
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..21099ed13b3e9 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -542,7 +542,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
 func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xf32>, !tosa.shape<2>) -> tensor<1x819xf32>
   return %0 : tensor<1x819xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..1370a66f273f4 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -211,6 +211,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xbf16>) -> tensor<1x819xbf16> {
   return %0 : tensor<1x819xbf16>
 }
 
+// -----
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_reverse(%arg0: tensor<13x21x3xbf16>) -> tensor<13x21x3xbf16> {
   // expected-error at +1 {{'tosa.reverse' op illegal: requires any of [bf16] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..a4c87f407a9e7 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -904,6 +904,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   return %0 : tensor<1x819xf32>
 }
 
+// -----
+// CHECK-LABEL: reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: reshape_unranked_output
 func.func @test_reshape_unranked_output(%arg0: tensor<13x21x3xf32>) -> tensor<*xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..985c0f00c9027 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..1d0b64dd9bbe1 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_reshape_mxfp_mxint8
+func.func @test_reshape_mxfp_mxint8(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207944

From llvmlistbot at llvm.org  Tue Jul  7 02:26:45 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 02:26:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
Message-ID: <llvm.org/llvm/llvm-project/pull/207947@github.com>


https://github.com/FedericoBruzzone created https://github.com/llvm/llvm-project/pull/207947

This PR simply fix what we identified in iree-org/iree#24689.

@banach-space @egebeysel 


>From 5463401c92343bcf98437f9aca344c09b496358c Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 7 Jul 2026 11:20:48 +0200
Subject: [PATCH] [mlir][ArmSME] f64 scalable matmul crashes in
 VectorLegalizationPass

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../ArmSME/Transforms/VectorLegalization.cpp  |  4 ++-
 .../Dialect/ArmSME/vector-legalization.mlir   | 33 +++++++++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
index 6e2654ab468f9..7a6c510920e22 100644
--- a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
+++ b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
@@ -943,7 +943,9 @@ struct LowerColumnTransferReadToLoops
     auto upperBound = createVscaleMultiple(numRows);
     auto step = arith::ConstantIndexOp::create(rewriter, loc, 1);
     Value init = arith::ConstantOp::create(
-        rewriter, loc, newResType, DenseElementsAttr::get(newResType, 0.0f));
+        rewriter, loc, newResType,
+        DenseElementsAttr::get(newResType,
+                               rewriter.getZeroAttr(resType.getElementType())));
 
     scf::ForOp loadLoop;
     {
diff --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index 6cdf576272ebc..3490f67cf68ff 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -650,6 +650,39 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 
 // -----
 
+/// Same as above but for f64, which exercises a different zero-init constant
+/// (see https://github.com/iree-org/iree/issues/24689).
+
+// CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
+// CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[PAD:.*]]: f64,
+// CHECK-SAME:      %[[SRC:.*]]: memref<?x?xf64>) -> vector<[4]x1xf64> {
+func.func @xfer_read_scalable_column_f64(%a: index, %b: index, %pad: f64, %src: memref<?x?xf64>) -> (vector<[4]x1xf64>) {
+  // CHECK:           %[[INIT:.*]] = arith.constant dense<0.000000e+00> : vector<[4]xf64>
+  // CHECK:           %[[STEP:.*]] = arith.constant 1 : index
+  // CHECK:           %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:           %[[LB:.*]] = arith.constant 0 : index
+  // CHECK:           %[[VSCALE:.*]] = vector.vscale
+  // CHECK:           %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+
+  // <scf.for>
+  // CHECK:           %[[SCF:.*]] = scf.for %[[IND_VAR:.*]] = %[[LB]] to %[[C4_VSCALE]] step %[[STEP]] iter_args(%[[SCF_RES:.*]] = %[[INIT]]) -> (vector<[4]xf64>) {
+  // CHECK:             %[[IDX_0_UPDATED:.*]] = arith.addi %[[IND_VAR]], %[[IDX_0]] : index
+  // CHECK:             %[[VAL_10:.*]] = memref.load %[[SRC]][%[[IDX_0_UPDATED]], %[[IDX_1]]] : memref<?x?xf64>
+  // CHECK:             %[[RES_UPDATED:.*]] = vector.insert %[[VAL_10]], %[[SCF_RES]] [%[[IND_VAR]]] : f64 into vector<[4]xf64>
+  // CHECK:             scf.yield %[[RES_UPDATED]] : vector<[4]xf64>
+  // CHECK:           }
+
+  // <shape-cast>
+  // CHECK:           %[[SC:.*]] = vector.shape_cast %[[SCF]] : vector<[4]xf64> to vector<[4]x1xf64>
+  // CHECK:           return %[[SC]]
+  %read = vector.transfer_read %src[%a, %b], %pad : memref<?x?xf64>, vector<[4]x1xf64>
+  return %read : vector<[4]x1xf64>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @negative_xfer_read_scalable_column_x2
 func.func @negative_xfer_read_scalable_column_x2(%a: index, %b: index, %pad: f32, %src: memref<?x?xf32>) -> (vector<[4]x2xf32>) {
   // CHECK-NOT: scf.for


From llvmlistbot at llvm.org  Tue Jul  7 02:27:03 2026
From: llvmlistbot at llvm.org (Yuvaraj Venkatesh)
Date: Tue, 07 Jul 2026 02:27:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4cc667.bb985344.1e9879.cb3c@mx.google.com>


Yuvaraj-Venkatesh wrote:

LGTM.

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 02:27:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 02:27:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4cc67c.bd93e731.251b21.f6b4@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-sme

@llvm/pr-subscribers-mlir

Author: Federico Bruzzone (FedericoBruzzone)

<details>
<summary>Changes</summary>

This PR simply fix what we identified in iree-org/iree#<!-- -->24689.

@<!-- -->banach-space @<!-- -->egebeysel 


---
Full diff: https://github.com/llvm/llvm-project/pull/207947.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp (+3-1) 
- (modified) mlir/test/Dialect/ArmSME/vector-legalization.mlir (+33) 


``````````diff
diff --git a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
index 6e2654ab468f9..7a6c510920e22 100644
--- a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
+++ b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
@@ -943,7 +943,9 @@ struct LowerColumnTransferReadToLoops
     auto upperBound = createVscaleMultiple(numRows);
     auto step = arith::ConstantIndexOp::create(rewriter, loc, 1);
     Value init = arith::ConstantOp::create(
-        rewriter, loc, newResType, DenseElementsAttr::get(newResType, 0.0f));
+        rewriter, loc, newResType,
+        DenseElementsAttr::get(newResType,
+                               rewriter.getZeroAttr(resType.getElementType())));
 
     scf::ForOp loadLoop;
     {
diff --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index 6cdf576272ebc..3490f67cf68ff 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -650,6 +650,39 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 
 // -----
 
+/// Same as above but for f64, which exercises a different zero-init constant
+/// (see https://github.com/iree-org/iree/issues/24689).
+
+// CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
+// CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[PAD:.*]]: f64,
+// CHECK-SAME:      %[[SRC:.*]]: memref<?x?xf64>) -> vector<[4]x1xf64> {
+func.func @xfer_read_scalable_column_f64(%a: index, %b: index, %pad: f64, %src: memref<?x?xf64>) -> (vector<[4]x1xf64>) {
+  // CHECK:           %[[INIT:.*]] = arith.constant dense<0.000000e+00> : vector<[4]xf64>
+  // CHECK:           %[[STEP:.*]] = arith.constant 1 : index
+  // CHECK:           %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:           %[[LB:.*]] = arith.constant 0 : index
+  // CHECK:           %[[VSCALE:.*]] = vector.vscale
+  // CHECK:           %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+
+  // <scf.for>
+  // CHECK:           %[[SCF:.*]] = scf.for %[[IND_VAR:.*]] = %[[LB]] to %[[C4_VSCALE]] step %[[STEP]] iter_args(%[[SCF_RES:.*]] = %[[INIT]]) -> (vector<[4]xf64>) {
+  // CHECK:             %[[IDX_0_UPDATED:.*]] = arith.addi %[[IND_VAR]], %[[IDX_0]] : index
+  // CHECK:             %[[VAL_10:.*]] = memref.load %[[SRC]][%[[IDX_0_UPDATED]], %[[IDX_1]]] : memref<?x?xf64>
+  // CHECK:             %[[RES_UPDATED:.*]] = vector.insert %[[VAL_10]], %[[SCF_RES]] [%[[IND_VAR]]] : f64 into vector<[4]xf64>
+  // CHECK:             scf.yield %[[RES_UPDATED]] : vector<[4]xf64>
+  // CHECK:           }
+
+  // <shape-cast>
+  // CHECK:           %[[SC:.*]] = vector.shape_cast %[[SCF]] : vector<[4]xf64> to vector<[4]x1xf64>
+  // CHECK:           return %[[SC]]
+  %read = vector.transfer_read %src[%a, %b], %pad : memref<?x?xf64>, vector<[4]x1xf64>
+  return %read : vector<[4]x1xf64>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @negative_xfer_read_scalable_column_x2
 func.func @negative_xfer_read_scalable_column_x2(%a: index, %b: index, %pad: f32, %src: memref<?x?xf32>) -> (vector<[4]x2xf32>) {
   // CHECK-NOT: scf.for

``````````

</details>


https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 02:28:32 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 02:28:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4cc6c0.c56fb102.2b5c33.2634@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 02:29:04 2026
From: llvmlistbot at llvm.org (Jeremy Johnson)
Date: Tue, 07 Jul 2026 02:29:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a4cc6e0.331483e7.2440f8.6d8e@mx.google.com>


jjohnson-arm wrote:

LGTM

https://github.com/llvm/llvm-project/pull/207944

From llvmlistbot at llvm.org  Tue Jul  7 02:29:28 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 02:29:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4cc6f8.f1c1923d.991d1.4b35@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 02:31:30 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Tue, 07 Jul 2026 02:31:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRVToLLVM] Add spirv.Unreachable to
 llvm.unreachable conversion (PR #207762)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207762@github.com>
Message-ID: <6a4cc772.c56fb102.2b5c33.989e@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/207762

From llvmlistbot at llvm.org  Tue Jul  7 02:48:15 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Tue, 07 Jul 2026 02:48:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4ccb5f.58bdd12c.19c81.4409@mx.google.com>


andrey-golubev wrote:

@matthias-springer @nicolasvasilache gentle ping.

https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 02:50:35 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Tue, 07 Jul 2026 02:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4ccbeb.abb10101.3d0bf8.8542@mx.google.com>


https://github.com/matthias-springer approved this pull request.

Can you add a "Note for LLVM integration:" paragraph that describes how to migrate code?

https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 02:50:45 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Tue, 07 Jul 2026 02:50:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4ccbf5.f1c1923d.991d1.06e9@mx.google.com>


https://github.com/matthias-springer edited https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 03:54:45 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Tue, 07 Jul 2026 03:54:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4cdaf5.449bfb44.172ed9.4f2b@mx.google.com>


https://github.com/andrey-golubev edited https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 03:55:35 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Tue, 07 Jul 2026 03:55:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4cdb27.14d9b9dc.f3195.b8da@mx.google.com>


https://github.com/andrey-golubev updated https://github.com/llvm/llvm-project/pull/206966

>From c45f417f0dae6ac6ea6801c820f3b4c2f736c2db Mon Sep 17 00:00:00 2001
From: "Golubev, Andrey" <andrey.golubev at intel.com>
Date: Wed, 1 Jul 2026 12:33:24 +0000
Subject: [PATCH] [NFC][mlir][bufferization] Align alloc/memcpy/cast options
 hooks to the rest

There are three hooks that have a different, more complicated API than
the other ones: allocationFn, memCpyFn, castFn. All three are called via
"helper" functions that check whether the hooks are set to call them and
fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options
have a different "API": they are always set to some default
implementation and can be overwritten by the user. This is a simpler
API overall and seems it can be universal.
---
 .../IR/BufferizableOpInterface.h              | 26 +++----
 .../BufferizableOpInterfaceImpl.cpp           |  6 +-
 .../IR/BufferizableOpInterface.cpp            | 74 ++++++++-----------
 .../Bufferization/IR/BufferizationOps.cpp     | 23 +++---
 .../Bufferization/Transforms/Bufferize.cpp    |  4 +-
 .../BufferizableOpInterfaceImpl.cpp           |  2 +-
 .../BufferizableOpInterfaceImpl.cpp           |  2 +-
 .../BufferizableOpInterfaceImpl.cpp           |  8 +-
 8 files changed, 61 insertions(+), 84 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f49a31c1583b..203680796a289 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -303,23 +303,6 @@ struct BufferizationOptions {
   /// Return `true` if the given op should be bufferized.
   bool isOpAllowed(Operation *op) const;
 
-  /// Helper functions for allocation and memory copying.
-  std::optional<AllocationFn> allocationFn;
-  std::optional<MemCpyFn> memCpyFn;
-  std::optional<CastFn> castFn;
-
-  /// Create a memref allocation with the given type and dynamic extents.
-  FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
-                               ValueRange dynShape) const;
-
-  /// Creates a memcpy between two given buffers.
-  LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
-                             Value to) const;
-
-  /// Creates a cast function from a buffer value to a new type.
-  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
-                              Value value) const;
-
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
@@ -353,6 +336,15 @@ struct BufferizationOptions {
   /// predictable.
   void setFunctionBoundaryTypeConversion(LayoutMapOption layoutMapOption);
 
+  /// Create a memref allocation with the given type and dynamic extents.
+  AllocationFn allocationFn = nullptr;
+
+  /// Creates a memcpy between two given buffers.
+  MemCpyFn memCpyFn = nullptr;
+
+  /// Creates a cast function from a buffer value to a new type.
+  CastFn castFn = nullptr;
+
   /// Type conversion from tensors to buffers. This type conversion is used to
   /// determine bufferized function argument and result types.
   ///
diff --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index f810000dd082f..0fe7be82959a3 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -173,12 +173,10 @@ struct SelectOpInterface
             "incompatible buffer types on true/false operands");
       }
       if (trueBuffer.getType() != *targetType) {
-        trueBuffer =
-            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+        trueBuffer = *options.castFn(rewriter, loc, *targetType, trueBuffer);
       }
       if (falseBuffer.getType() != *targetType) {
-        falseBuffer =
-            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+        falseBuffer = *options.castFn(rewriter, loc, *targetType, falseBuffer);
       }
     }
 
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 7f656164f3a15..e39b20492e5ed 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,6 +358,34 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+/// Create a memref allocation with the given type and dynamic extents.
+FailureOr<Value> defaultCreateAlloc(OpBuilder &b, Location loc, MemRefType type,
+                                    ValueRange dynShape,
+                                    unsigned int bufferAlignment) {
+  // Default buffer allocation via AllocOp.
+  if (bufferAlignment != 0)
+    return memref::AllocOp::create(b, loc, type, dynShape,
+                                   b.getI64IntegerAttr(bufferAlignment))
+        .getResult();
+  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
+}
+
+/// Create a memory copy between two memref buffers.
+LogicalResult defaultCreateMemCpy(OpBuilder &b, Location loc, Value from,
+                                  Value to) {
+  memref::CopyOp::create(b, loc, from, to);
+  return success();
+}
+
+FailureOr<Value> defaultCreateCast(OpBuilder &b, Location loc, Type dest,
+                                   Value value) {
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
@@ -405,7 +433,9 @@ defaultReconcileBufferTypeMismatch(BufferLikeType x, BufferLikeType y,
 
 // Default constructor for BufferizationOptions.
 BufferizationOptions::BufferizationOptions()
-    : functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
+    : allocationFn(defaultCreateAlloc), memCpyFn(defaultCreateMemCpy),
+      castFn(defaultCreateCast),
+      functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
       unknownTypeConverterFn(defaultUnknownTypeConverter),
       reconcileBufferTypeMismatchFn(defaultReconcileBufferTypeMismatch) {}
 
@@ -811,48 +841,6 @@ void bufferization::replaceOpWithBufferizedValues(RewriterBase &rewriter,
   rewriter.replaceOp(op, replacements);
 }
 
-//===----------------------------------------------------------------------===//
-// Bufferization-specific scoped alloc insertion support.
-//===----------------------------------------------------------------------===//
-
-/// Create a memref allocation with the given type and dynamic extents.
-FailureOr<Value> BufferizationOptions::createAlloc(OpBuilder &b, Location loc,
-                                                   MemRefType type,
-                                                   ValueRange dynShape) const {
-  if (allocationFn)
-    return (*allocationFn)(b, loc, type, dynShape, bufferAlignment);
-
-  // Default bufferallocation via AllocOp.
-  if (bufferAlignment != 0)
-    return memref::AllocOp::create(b, loc, type, dynShape,
-                                   b.getI64IntegerAttr(bufferAlignment))
-        .getResult();
-  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
-}
-
-/// Create a memory copy between two memref buffers.
-LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
-                                                 Value from, Value to) const {
-  if (memCpyFn)
-    return (*memCpyFn)(b, loc, from, to);
-
-  memref::CopyOp::create(b, loc, from, to);
-  return success();
-}
-
-FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
-                                                  Type dest,
-                                                  Value value) const {
-  if (castFn)
-    return (*castFn)(b, loc, dest, value);
-
-  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
-  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
-         "cast incompatible");
-  return memref::CastOp::create(b, loc, dest, value).getResult();
-}
-
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 320f426985750..45f5b3eaa5aea 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = *options.createCast(b, value.getLoc(), destType, value);
+    Value casted = *options.castFn(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -73,11 +73,11 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
     dynamicOperands.push_back(size);
   }
 
-  FailureOr<Value> copy =
-      options.createAlloc(b, loc, destType, dynamicOperands);
+  FailureOr<Value> copy = options.allocationFn(
+      b, loc, destType, dynamicOperands, options.bufferAlignment);
   if (failed(copy))
     return failure();
-  if (failed(options.createMemCpy(b, loc, value, *copy)))
+  if (failed(options.memCpyFn(b, loc, value, *copy)))
     return failure();
   return copy;
 }
@@ -103,11 +103,9 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (!llvm::isa<BaseMemRefType>(srcType) ||
       !llvm::isa<BaseMemRefType>(destType)) {
     // Non-builtin case: the best is to try the user-provided cast.
-    assert(options.castFn.has_value() &&
-           "user-provided cast is required for non-builtin types");
     auto replacement =
-        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
-                           destType, bufferToTensor.getBuffer());
+        options.castFn(rewriter, bufferToTensor.getBuffer().getLoc(), destType,
+                       bufferToTensor.getBuffer());
     if (failed(replacement))
       return failure();
     rewriter.replaceOp(toBuffer, *replacement);
@@ -194,14 +192,15 @@ LogicalResult AllocTensorOp::bufferize(RewriterBase &rewriter,
     assert(dynamicDims.empty() && "expected either `copy` or `dynamicDims`");
     populateDynamicDimSizes(rewriter, loc, copyBuffer, dynamicDims);
   }
-  FailureOr<Value> alloc = options.createAlloc(
-      rewriter, loc, llvm::cast<MemRefType>(*allocType), dynamicDims);
+  FailureOr<Value> alloc =
+      options.allocationFn(rewriter, loc, llvm::cast<MemRefType>(*allocType),
+                           dynamicDims, options.bufferAlignment);
   if (failed(alloc))
     return failure();
 
   // Create memory copy (if any).
   if (getCopy()) {
-    if (failed(options.createMemCpy(rewriter, loc, copyBuffer, *alloc)))
+    if (failed(options.memCpyFn(rewriter, loc, copyBuffer, *alloc)))
       return failure();
   }
 
@@ -619,7 +618,7 @@ MaterializeInDestinationOp::bufferize(RewriterBase &rewriter,
   auto srcBuffer = getBuffer(rewriter, getSource(), options, state);
   if (failed(srcBuffer))
     return failure();
-  if (failed(options.createMemCpy(rewriter, getLoc(), *srcBuffer, buffer)))
+  if (failed(options.memCpyFn(rewriter, getLoc(), *srcBuffer, buffer)))
     return failure();
   replaceOpWithBufferizedValues(rewriter, getOperation(),
                                 tensorDest ? ValueRange(buffer) : ValueRange());
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index e0262abf313b8..d6e0b12733976 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -477,8 +477,8 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
       // A cast is needed if the operand and the block argument have different
       // bufferized types.
       if (type != *operandBufferType) {
-        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
-                                                type, bufferizedOperand);
+        bufferizedOperand = *options.castFn(rewriter, operand.getLoc(), type,
+                                            bufferizedOperand);
       }
       newOperands.push_back(bufferizedOperand);
     }
diff --git a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
index c2cbc95e94343..a7c5934132b35 100644
--- a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -146,7 +146,7 @@ struct GlobalStoreOpInterface
     }
 
     auto memcpy =
-        options.createMemCpy(rewriter, loc, sourceMemref.value(), targetMemref);
+        options.memCpyFn(rewriter, loc, sourceMemref.value(), targetMemref);
     if (failed(memcpy)) {
       return failure();
     }
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index 401be7440f5a2..f8302126a4a4e 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -37,7 +37,7 @@ static Value castBuffer(OpBuilder &b, Value buffer, Type type,
   if (buffer.getType() == type)
     return buffer;
 
-  return *options.createCast(b, buffer.getLoc(), type, buffer);
+  return *options.castFn(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value
diff --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 41a3d96b832d1..525d57341cee5 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -724,7 +724,7 @@ struct InsertSliceOpInterface
         getBuffer(rewriter, insertSliceOp.getSource(), options, state);
     if (failed(srcMemref))
       return failure();
-    if (failed(options.createMemCpy(rewriter, loc, *srcMemref, subView)))
+    if (failed(options.memCpyFn(rewriter, loc, *srcMemref, subView)))
       return failure();
 
     replaceOpWithBufferizedValues(rewriter, op, *dstMemref);
@@ -1002,8 +1002,8 @@ struct ParallelInsertSliceOpInterface
         parallelInsertSliceOp.getMixedStrides());
 
     // This memcpy will fold away if everything bufferizes in-place.
-    if (failed(options.createMemCpy(rewriter, parallelInsertSliceOp.getLoc(),
-                                    *srcBuffer, subview)))
+    if (failed(options.memCpyFn(rewriter, parallelInsertSliceOp.getLoc(),
+                                *srcBuffer, subview)))
       return failure();
 
     // In case the source was allocated in the same block, make sure that the
@@ -1175,7 +1175,7 @@ struct ConcatOpInterface
           rewriter, loc, subviewMemRefType, dstBuffer, offsets, sizes, strides);
 
       // Copy the source buffer into the destination subview.
-      if (failed(options.createMemCpy(rewriter, loc, *srcBuffer, subview)))
+      if (failed(options.memCpyFn(rewriter, loc, *srcBuffer, subview)))
         return failure();
 
       concatDimOffset = sum(concatDimOffset, concatDimSize);


From llvmlistbot at llvm.org  Tue Jul  7 03:56:28 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Tue, 07 Jul 2026 03:56:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4cdb5c.6c8668f2.e4f82.dc2f@mx.google.com>


andrey-golubev wrote:

> Can you add a "Note for LLVM integration:" paragraph to the commit message that describes how to migrate code?

Added! Will rerun the CI on fresh main to double-check for issues. I plan to merge once CI is done / green.

https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 04:15:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 04:15:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] de348bc - [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (#206966)
Message-ID: <6a4cdfef.7d41f3a1.237e53.580d@mx.google.com>


Author: Andrei Golubev
Date: 2026-07-07T13:15:54+02:00
New Revision: de348bcd5f7898dc7478fed6bbd95c97a1848d98

URL: https://github.com/llvm/llvm-project/commit/de348bcd5f7898dc7478fed6bbd95c97a1848d98
DIFF: https://github.com/llvm/llvm-project/commit/de348bcd5f7898dc7478fed6bbd95c97a1848d98.diff

LOG: [NFC][mlir][bufferization] Align alloc/memcpy/cast options hooks to the rest (#206966)

There are three hooks that have a different, more complicated API than
the other ones: allocationFn, memCpyFn, castFn. All three are called via
"helper" functions that check whether the hooks are set to call them and
fall back to default implementations otherwise.

Other hooks (e.g. unknownTypeConverterFn) in the bufferization options
have a different "API": they are always set to some default
implementation and can be overwritten by the user. This is a simpler API
overall and seems it can be universal.

Note for LLVM integration: BufferizationOptions::{createAlloc,
createMemCpy, createCast} functions are "replaced" with direct calls to
the underlying std::function objects. At any call-site that relied on
any of the removed APIs, a call to the respective hook should now be
used instead. The exception is the `allocationFn` hook which needs an
explicit "bufferAlignment" parameter. Beforehand, it was implicitly
taken from BufferizationOptions::bufferAlignment field, so a
non-functional migration path is to go from
`options.createAlloc(builder, loc, destType, value)` to
`options.allocationFn(builder, loc, destType, value,
options.bufferAlignment)`.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
    mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
    mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
    mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
    mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
    mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
    mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
    mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f49a31c1583b..203680796a289 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -303,23 +303,6 @@ struct BufferizationOptions {
   /// Return `true` if the given op should be bufferized.
   bool isOpAllowed(Operation *op) const;
 
-  /// Helper functions for allocation and memory copying.
-  std::optional<AllocationFn> allocationFn;
-  std::optional<MemCpyFn> memCpyFn;
-  std::optional<CastFn> castFn;
-
-  /// Create a memref allocation with the given type and dynamic extents.
-  FailureOr<Value> createAlloc(OpBuilder &b, Location loc, MemRefType type,
-                               ValueRange dynShape) const;
-
-  /// Creates a memcpy between two given buffers.
-  LogicalResult createMemCpy(OpBuilder &b, Location loc, Value from,
-                             Value to) const;
-
-  /// Creates a cast function from a buffer value to a new type.
-  FailureOr<Value> createCast(OpBuilder &b, Location loc, Type dest,
-                              Value value) const;
-
   /// Specifies whether not bufferizable ops are allowed in the input. If so,
   /// bufferization.to_buffer and bufferization.to_tensor ops are inserted at
   /// the boundaries.
@@ -353,6 +336,15 @@ struct BufferizationOptions {
   /// predictable.
   void setFunctionBoundaryTypeConversion(LayoutMapOption layoutMapOption);
 
+  /// Create a memref allocation with the given type and dynamic extents.
+  AllocationFn allocationFn = nullptr;
+
+  /// Creates a memcpy between two given buffers.
+  MemCpyFn memCpyFn = nullptr;
+
+  /// Creates a cast function from a buffer value to a new type.
+  CastFn castFn = nullptr;
+
   /// Type conversion from tensors to buffers. This type conversion is used to
   /// determine bufferized function argument and result types.
   ///

diff  --git a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
index f810000dd082f..0fe7be82959a3 100644
--- a/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -173,12 +173,10 @@ struct SelectOpInterface
             "incompatible buffer types on true/false operands");
       }
       if (trueBuffer.getType() != *targetType) {
-        trueBuffer =
-            *options.createCast(rewriter, loc, *targetType, trueBuffer);
+        trueBuffer = *options.castFn(rewriter, loc, *targetType, trueBuffer);
       }
       if (falseBuffer.getType() != *targetType) {
-        falseBuffer =
-            *options.createCast(rewriter, loc, *targetType, falseBuffer);
+        falseBuffer = *options.castFn(rewriter, loc, *targetType, falseBuffer);
       }
     }
 

diff  --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index 7f656164f3a15..e39b20492e5ed 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,6 +358,34 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+/// Create a memref allocation with the given type and dynamic extents.
+FailureOr<Value> defaultCreateAlloc(OpBuilder &b, Location loc, MemRefType type,
+                                    ValueRange dynShape,
+                                    unsigned int bufferAlignment) {
+  // Default buffer allocation via AllocOp.
+  if (bufferAlignment != 0)
+    return memref::AllocOp::create(b, loc, type, dynShape,
+                                   b.getI64IntegerAttr(bufferAlignment))
+        .getResult();
+  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
+}
+
+/// Create a memory copy between two memref buffers.
+LogicalResult defaultCreateMemCpy(OpBuilder &b, Location loc, Value from,
+                                  Value to) {
+  memref::CopyOp::create(b, loc, from, to);
+  return success();
+}
+
+FailureOr<Value> defaultCreateCast(OpBuilder &b, Location loc, Type dest,
+                                   Value value) {
+  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
+  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
+  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
+         "cast incompatible");
+  return memref::CastOp::create(b, loc, dest, value).getResult();
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
@@ -405,7 +433,9 @@ defaultReconcileBufferTypeMismatch(BufferLikeType x, BufferLikeType y,
 
 // Default constructor for BufferizationOptions.
 BufferizationOptions::BufferizationOptions()
-    : functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
+    : allocationFn(defaultCreateAlloc), memCpyFn(defaultCreateMemCpy),
+      castFn(defaultCreateCast),
+      functionArgTypeConverterFn(defaultFunctionArgTypeConverter),
       unknownTypeConverterFn(defaultUnknownTypeConverter),
       reconcileBufferTypeMismatchFn(defaultReconcileBufferTypeMismatch) {}
 
@@ -811,48 +841,6 @@ void bufferization::replaceOpWithBufferizedValues(RewriterBase &rewriter,
   rewriter.replaceOp(op, replacements);
 }
 
-//===----------------------------------------------------------------------===//
-// Bufferization-specific scoped alloc insertion support.
-//===----------------------------------------------------------------------===//
-
-/// Create a memref allocation with the given type and dynamic extents.
-FailureOr<Value> BufferizationOptions::createAlloc(OpBuilder &b, Location loc,
-                                                   MemRefType type,
-                                                   ValueRange dynShape) const {
-  if (allocationFn)
-    return (*allocationFn)(b, loc, type, dynShape, bufferAlignment);
-
-  // Default bufferallocation via AllocOp.
-  if (bufferAlignment != 0)
-    return memref::AllocOp::create(b, loc, type, dynShape,
-                                   b.getI64IntegerAttr(bufferAlignment))
-        .getResult();
-  return memref::AllocOp::create(b, loc, type, dynShape).getResult();
-}
-
-/// Create a memory copy between two memref buffers.
-LogicalResult BufferizationOptions::createMemCpy(OpBuilder &b, Location loc,
-                                                 Value from, Value to) const {
-  if (memCpyFn)
-    return (*memCpyFn)(b, loc, from, to);
-
-  memref::CopyOp::create(b, loc, from, to);
-  return success();
-}
-
-FailureOr<Value> BufferizationOptions::createCast(OpBuilder &b, Location loc,
-                                                  Type dest,
-                                                  Value value) const {
-  if (castFn)
-    return (*castFn)(b, loc, dest, value);
-
-  assert(isa<BaseMemRefType>(dest) && "expected BaseMemRefType");
-  assert(isa<BaseMemRefType>(value.getType()) && "expected BaseMemRefType");
-  assert(memref::CastOp::areCastCompatible(value.getType(), dest) &&
-         "cast incompatible");
-  return memref::CastOp::create(b, loc, dest, value).getResult();
-}
-
 //===----------------------------------------------------------------------===//
 // Bufferization-specific IRMapping support with debugging.
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 320f426985750..45f5b3eaa5aea 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -60,7 +60,7 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
   // a fix extra conditions in `isGuaranteedCastCompatible`.
   if (memref::CastOp::areCastCompatible(srcType, destType) &&
       isGuaranteedCastCompatible(srcType, destType)) {
-    Value casted = *options.createCast(b, value.getLoc(), destType, value);
+    Value casted = *options.castFn(b, value.getLoc(), destType, value);
     return casted;
   }
 
@@ -73,11 +73,11 @@ FailureOr<Value> mlir::bufferization::castOrReallocMemRefValue(
     dynamicOperands.push_back(size);
   }
 
-  FailureOr<Value> copy =
-      options.createAlloc(b, loc, destType, dynamicOperands);
+  FailureOr<Value> copy = options.allocationFn(
+      b, loc, destType, dynamicOperands, options.bufferAlignment);
   if (failed(copy))
     return failure();
-  if (failed(options.createMemCpy(b, loc, value, *copy)))
+  if (failed(options.memCpyFn(b, loc, value, *copy)))
     return failure();
   return copy;
 }
@@ -103,11 +103,9 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (!llvm::isa<BaseMemRefType>(srcType) ||
       !llvm::isa<BaseMemRefType>(destType)) {
     // Non-builtin case: the best is to try the user-provided cast.
-    assert(options.castFn.has_value() &&
-           "user-provided cast is required for non-builtin types");
     auto replacement =
-        options.createCast(rewriter, bufferToTensor.getBuffer().getLoc(),
-                           destType, bufferToTensor.getBuffer());
+        options.castFn(rewriter, bufferToTensor.getBuffer().getLoc(), destType,
+                       bufferToTensor.getBuffer());
     if (failed(replacement))
       return failure();
     rewriter.replaceOp(toBuffer, *replacement);
@@ -194,14 +192,15 @@ LogicalResult AllocTensorOp::bufferize(RewriterBase &rewriter,
     assert(dynamicDims.empty() && "expected either `copy` or `dynamicDims`");
     populateDynamicDimSizes(rewriter, loc, copyBuffer, dynamicDims);
   }
-  FailureOr<Value> alloc = options.createAlloc(
-      rewriter, loc, llvm::cast<MemRefType>(*allocType), dynamicDims);
+  FailureOr<Value> alloc =
+      options.allocationFn(rewriter, loc, llvm::cast<MemRefType>(*allocType),
+                           dynamicDims, options.bufferAlignment);
   if (failed(alloc))
     return failure();
 
   // Create memory copy (if any).
   if (getCopy()) {
-    if (failed(options.createMemCpy(rewriter, loc, copyBuffer, *alloc)))
+    if (failed(options.memCpyFn(rewriter, loc, copyBuffer, *alloc)))
       return failure();
   }
 
@@ -619,7 +618,7 @@ MaterializeInDestinationOp::bufferize(RewriterBase &rewriter,
   auto srcBuffer = getBuffer(rewriter, getSource(), options, state);
   if (failed(srcBuffer))
     return failure();
-  if (failed(options.createMemCpy(rewriter, getLoc(), *srcBuffer, buffer)))
+  if (failed(options.memCpyFn(rewriter, getLoc(), *srcBuffer, buffer)))
     return failure();
   replaceOpWithBufferizedValues(rewriter, getOperation(),
                                 tensorDest ? ValueRange(buffer) : ValueRange());

diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
index e0262abf313b8..d6e0b12733976 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/Bufferize.cpp
@@ -477,8 +477,8 @@ bufferization::bufferizeBlockSignature(Block *block, RewriterBase &rewriter,
       // A cast is needed if the operand and the block argument have 
diff erent
       // bufferized types.
       if (type != *operandBufferType) {
-        bufferizedOperand = *options.createCast(rewriter, operand.getLoc(),
-                                                type, bufferizedOperand);
+        bufferizedOperand = *options.castFn(rewriter, operand.getLoc(), type,
+                                            bufferizedOperand);
       }
       newOperands.push_back(bufferizedOperand);
     }

diff  --git a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
index c2cbc95e94343..a7c5934132b35 100644
--- a/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/MLProgram/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -146,7 +146,7 @@ struct GlobalStoreOpInterface
     }
 
     auto memcpy =
-        options.createMemCpy(rewriter, loc, sourceMemref.value(), targetMemref);
+        options.memCpyFn(rewriter, loc, sourceMemref.value(), targetMemref);
     if (failed(memcpy)) {
       return failure();
     }

diff  --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index 401be7440f5a2..f8302126a4a4e 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -37,7 +37,7 @@ static Value castBuffer(OpBuilder &b, Value buffer, Type type,
   if (buffer.getType() == type)
     return buffer;
 
-  return *options.createCast(b, buffer.getLoc(), type, buffer);
+  return *options.castFn(b, buffer.getLoc(), type, buffer);
 }
 
 /// Helper function for loop bufferization. Return "true" if the given value

diff  --git a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
index 41a3d96b832d1..525d57341cee5 100644
--- a/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Tensor/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -724,7 +724,7 @@ struct InsertSliceOpInterface
         getBuffer(rewriter, insertSliceOp.getSource(), options, state);
     if (failed(srcMemref))
       return failure();
-    if (failed(options.createMemCpy(rewriter, loc, *srcMemref, subView)))
+    if (failed(options.memCpyFn(rewriter, loc, *srcMemref, subView)))
       return failure();
 
     replaceOpWithBufferizedValues(rewriter, op, *dstMemref);
@@ -1002,8 +1002,8 @@ struct ParallelInsertSliceOpInterface
         parallelInsertSliceOp.getMixedStrides());
 
     // This memcpy will fold away if everything bufferizes in-place.
-    if (failed(options.createMemCpy(rewriter, parallelInsertSliceOp.getLoc(),
-                                    *srcBuffer, subview)))
+    if (failed(options.memCpyFn(rewriter, parallelInsertSliceOp.getLoc(),
+                                *srcBuffer, subview)))
       return failure();
 
     // In case the source was allocated in the same block, make sure that the
@@ -1175,7 +1175,7 @@ struct ConcatOpInterface
           rewriter, loc, subviewMemRefType, dstBuffer, offsets, sizes, strides);
 
       // Copy the source buffer into the destination subview.
-      if (failed(options.createMemCpy(rewriter, loc, *srcBuffer, subview)))
+      if (failed(options.memCpyFn(rewriter, loc, *srcBuffer, subview)))
         return failure();
 
       concatDimOffset = sum(concatDimOffset, concatDimSize);


        

From llvmlistbot at llvm.org  Tue Jul  7 04:16:04 2026
From: llvmlistbot at llvm.org (Andrei Golubev)
Date: Tue, 07 Jul 2026 04:16:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4cdff4.492e23e4.157d47.5c64@mx.google.com>


https://github.com/andrey-golubev closed https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 04:23:17 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Tue, 07 Jul 2026 04:23:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][bufferization] Align
 alloc/memcpy/cast options hooks to the rest (PR #206966)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206966@github.com>
Message-ID: <6a4ce1a5.492e23e4.157d47.7912@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-01` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/11977

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8600/8691] Creating library symlink lib/libFIRCodeGen.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8602/8691] Linking CXX shared library lib/libclang-cpp.so.23.0git
[8603/8691] Linking CXX shared library lib/libFlangOpenMPTransforms.so.23.0git
[8604/8691] Linking CXX shared library lib/libFIRTransforms.so.23.0git
[8605/8691] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
[8606/8691] Linking CXX executable bin/fir-lsp-server
[8607/8691] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8608/8691] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8609/8691] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8610/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8611/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8612/8691] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8613/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8600/8691] Creating library symlink lib/libFIRCodeGen.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8602/8691] Linking CXX shared library lib/libclang-cpp.so.23.0git
[8603/8691] Linking CXX shared library lib/libFlangOpenMPTransforms.so.23.0git
[8604/8691] Linking CXX shared library lib/libFIRTransforms.so.23.0git
[8605/8691] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
[8606/8691] Linking CXX executable bin/fir-lsp-server
[8607/8691] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8608/8691] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8609/8691] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8610/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8611/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8612/8691] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8613/8691] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=132.990003

```

</details>

https://github.com/llvm/llvm-project/pull/206966

From llvmlistbot at llvm.org  Tue Jul  7 04:46:15 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 04:46:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4ce707.e55dc341.1f19bd.3f64@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 05:00:19 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 07 Jul 2026 05:00:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVM] Add arithmetic fence intrinsic
	op (PR #207975)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207975@github.com>
Message-ID: <6a4cea53.f1c1923d.991d1.0588@mx.google.com>


https://github.com/gysit approved this pull request.

Thanks!

LGTM

https://github.com/llvm/llvm-project/pull/207975

From llvmlistbot at llvm.org  Tue Jul  7 05:15:24 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:15:24 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [openmp]
 [LoopTiling][Clang][OpenMP] Canonical Intra-tile Loops (PR #191114)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/191114@github.com>
Message-ID: <6a4ceddc.e55dc341.1f19bd.89e9@mx.google.com>


Meinersbur wrote:

> In the min-bound default, it is obvious to see predicate not getting emitted resulting in incorrect semantics. So, either way, we rely on the marker to be handled properly.
  
So we have two versions of the same loop:

```cpp
for (int i_intratile = i_floor; i_intratile < min(n, i_floor + i_tilesize); ++i_intratile)
  Body(i_intratile);
```

and

```cpp
for (int i_intratile = i_floor; i_intratile < i_floor + i_tilesize; ++i_intratile)
  if (i_intratile < n)
    Body(i_intratile);
```

I think we agree that either version is correct. The attribute is supposed to convey that one version can be transformed into the other, NOT that something has to be added in order to make it semantically correct. Most `__attribute__`/`[[attribute]]` are not used for correctness.

For reasons stated before, I would take the first form as starting point, then attribute it:

```cpp
AttributedStmt: __attribute__(("The following loop can be reinterpreted as a canonical loop lb=i_floor, ub=i_floor+i_tilesize with predicate if (i_intratile < n) inserted before the body"))
ForStmt:         for (int i_intratile = i_floor; i_intratile < min(n, i_floor + i_tilesize); ++i_intratile)
CompoundStmt:      Body(i_intratile);
```

If encountering in CGStmt: Ignore the AttributedStmt, generate code from ForStmt

If encountering in checkOpenMPLoop: Use the stated lb/ub[^2] instead of trying to extract it from ForStmt[^1]. Remember the we will have to insert a predicate in the innermost loop's body (like PreInit is what we have to remember to insert outside the outermost loop).


[^1]: OpenMP will use the result from `LoopHelper`/`checkOpenMPLoop` anyway and not even look at the ForStmt after `checkOpenMPLoop`.

[^2]: I am not sure whether it makes sense to store lb/ub/predicate directly in the AttributedStmt. It could also be a hint "I guarantee that the Stmt I am attributing as the form `ForStmt(i = lb; i < min(global_max, ub); ++i)`. Use lb/ub/i < global_max for building LoopHelper"



https://github.com/llvm/llvm-project/pull/191114

From llvmlistbot at llvm.org  Tue Jul  7 05:36:36 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:36:36 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cf2d4.d61da0cb.148ed6.9b47@mx.google.com>


https://github.com/Meinersbur edited https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 05:36:36 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:36:36 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cf2d4.3c65ca91.2d2666.1a1a@mx.google.com>


https://github.com/Meinersbur approved this pull request.

LGTM

There is currently a merge conflict with HEAD. After solving, please give @arsenm a chance to respond as well before landing.

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 05:36:37 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:36:37 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cf2d5.67562a99.3a110d.15c4@mx.google.com>


================

----------------
Meinersbur wrote:

Do you know why this is this does not go through the auto-upgrader?

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 05:36:37 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:36:37 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cf2d5.fa459026.144231.361f@mx.google.com>


================

----------------
Meinersbur wrote:

[not a change request] `CHECK-NOT` tests are not reliable. Here, if the `!{!"llvm.loop.distribute.disable"}` matches, the `-NOT` will only check after the line. That is, the following will not cause an error:

```
!1 = !{!"llvm.loop.distribute.enable"}
...
!2 = !{!"llvm.loop.distribute.enable", i1 true}
...
!3 = !{!"llvm.loop.distribute.disable"}
```

I'd avoid relying on `-NOT` tests, but they will also not hurt here either an may make the intention of this test clear.

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 05:36:37 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Tue, 07 Jul 2026 05:36:37 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4cf2d5.a1c10a39.299b42.e4bd@mx.google.com>


================

----------------
Meinersbur wrote:

[nit] please avoid change noise

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 05:49:27 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Tue, 07 Jul 2026 05:49:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
Message-ID: <llvm.org/llvm/llvm-project/pull/207995@github.com>


https://github.com/mydatascience created https://github.com/llvm/llvm-project/pull/207995

StableHLO tests have regression due to use of additional op parameter in verifyBlockScaledTensorType

This fixes https://github.com/llvm/llvm-project/pull/203583 


>From d0e65044589913a3c3565411102df67507881d23 Mon Sep 17 00:00:00 2001
From: Vladimir Suvorov <suvorovv at google.com>
Date: Tue, 7 Jul 2026 16:48:52 +0400
Subject: [PATCH] Fix verifyBlockScaledTensorType

Signed-off-by: Vladimir Suvorov <suvorovv at google.com>
---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h        |  2 +-
 mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td |  2 +-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp               | 12 +++---------
 mlir/test/Dialect/Tosa/verifier.mlir               |  2 --
 4 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 66c3753899124..dfecda940d8c7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,7 +92,7 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
-LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+LogicalResult verifyBlockScaledTensorType(mlir::Type type);
 
 } // namespace tosa
 } // namespace OpTrait
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..63be7b69b6398 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -159,7 +159,7 @@ def AtLeastRankOne : And<[
 
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
-            "verifyBlockScaledTensorType($_op, $_self))">;
+            "verifyBlockScaledTensorType($_self))">;
 
 class TosaTensorOf<list<Type> allowedTypes,
                    string summary = "tosa-conformant tensor">
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..829c0b41a68cf 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -740,8 +740,7 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(mlir::Type type) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
@@ -752,9 +751,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
     return success();
 
   if (tensorType.getRank() == 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " does not support block scaling on scalar tensors";
+    return failure();
 
   const int64_t blockedDimension = tensorType.getShape().back();
   if (ShapedType::isDynamic(blockedDimension))
@@ -763,10 +760,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
   if (blockedDimension % blockSize != 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " blocked dimension must be a multiple of block size, got "
-           << blockedDimension << " and block size " << blockSize;
+    return failure();
 
   return success();
 }
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..478532e52c438 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1675,7 +1675,6 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
@@ -1684,7 +1683,6 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>


From llvmlistbot at llvm.org  Tue Jul  7 05:49:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 05:49:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4cf5eb.20a258a7.86a3d.22a7@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @mydatascience :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 05:50:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 05:50:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4cf617.79dd0093.26135f.5ca6@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-tosa

@llvm/pr-subscribers-mlir

Author: Vladimir Suvorov (mydatascience)

<details>
<summary>Changes</summary>

StableHLO tests have regression due to use of additional op parameter in verifyBlockScaledTensorType

This fixes https://github.com/llvm/llvm-project/pull/203583 


---
Full diff: https://github.com/llvm/llvm-project/pull/207995.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h (+1-1) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td (+1-1) 
- (modified) mlir/lib/Dialect/Tosa/IR/TosaOps.cpp (+3-9) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (-2) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 66c3753899124..dfecda940d8c7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,7 +92,7 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
-LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+LogicalResult verifyBlockScaledTensorType(mlir::Type type);
 
 } // namespace tosa
 } // namespace OpTrait
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..63be7b69b6398 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -159,7 +159,7 @@ def AtLeastRankOne : And<[
 
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
-            "verifyBlockScaledTensorType($_op, $_self))">;
+            "verifyBlockScaledTensorType($_self))">;
 
 class TosaTensorOf<list<Type> allowedTypes,
                    string summary = "tosa-conformant tensor">
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..829c0b41a68cf 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -740,8 +740,7 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(mlir::Type type) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
@@ -752,9 +751,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
     return success();
 
   if (tensorType.getRank() == 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " does not support block scaling on scalar tensors";
+    return failure();
 
   const int64_t blockedDimension = tensorType.getShape().back();
   if (ShapedType::isDynamic(blockedDimension))
@@ -763,10 +760,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
   if (blockedDimension % blockSize != 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " blocked dimension must be a multiple of block size, got "
-           << blockedDimension << " and block size " << blockSize;
+    return failure();
 
   return success();
 }
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..478532e52c438 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1675,7 +1675,6 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
@@ -1684,7 +1683,6 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 05:55:31 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Tue, 07 Jul 2026 05:55:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4cf743.eb1b2d33.125797.100e@mx.google.com>


mydatascience wrote:

@lhutton1  @sahas3 please 

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 06:17:46 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Tue, 07 Jul 2026 06:17:46 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a4cfc7a.e17f22e8.9bc4b.57c0@mx.google.com>


tblah wrote:

Stepping back, I think this needs a more systematic validation pass before further review.

This PR adds support for atomic compare-capture, but several valid forms close to the added tests still either abort or fail during LLVM translation. For example, postfix `<`/`>` compare-capture and complex compare-capture are both accepted by the front end but fail later. These are not obscure transformations; they are natural variants of the feature being added.

Could you please audit the supported/unsupported matrix for:

- prefix vs postfix vs fail-only capture
- integer, real, logical, and complex operands
- `==`, `<`, and `>` compare forms where accepted
- Flang lowering and MLIR-to-LLVMIR translation coverage

For each unsupported case, please add an explicit diagnostic/NYI test rather than allowing it to reach an assertion or translation failure. Please also check for regressions in the LLVM test-suite Fortran tests and the Fujitsu test-suite, since this changes OpenMP lowering/translation behavior beyond the specific new lit cases.

Once that matrix is covered and the broader Fortran test suites are checked, it will be much easier to review the remaining implementation details.


https://github.com/llvm/llvm-project/pull/202315

From llvmlistbot at llvm.org  Tue Jul  7 06:31:47 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Tue, 07 Jul 2026 06:31:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4cffc3.f732ad81.3a4994.622a@mx.google.com>


https://github.com/mydatascience edited https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 07:41:01 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 07 Jul 2026 07:41:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4d0ffd.e55dc341.1f19bd.3286@mx.google.com>


================

----------------
egebeysel wrote:

I think you can delete the entire comment, sorry

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 07:41:45 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 07 Jul 2026 07:41:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4d1029.481afadf.196d57.659e@mx.google.com>


================

----------------
egebeysel wrote:

Jfyi, we do not put IREE links to llvm code :)

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 07:43:12 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 07:43:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4d1080.af6ba3ed.36d9db.39d2@mx.google.com>


================

----------------
FedericoBruzzone wrote:

This is exactly the feedback I was waiting for! Thanks :'D

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Tue Jul  7 07:43:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 07:43:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4d108b.e55dc341.1f19bd.acd0@mx.google.com>


https://github.com/ES3Q updated https://github.com/llvm/llvm-project/pull/207597

>From b2e0378372c5dd221c0cfe95c9ed62dba8b292e4 Mon Sep 17 00:00:00 2001
From: ES3Q <ES3Q at QQ.COM>
Date: Mon, 6 Jul 2026 00:08:25 +0800
Subject: [PATCH 1/2] [MLIR][OpenACC] Fix host fallback for acc.atomic.update

The host fallback for  incorrectly cloned only the
first operation in the region and used its result for the store. This
generated invalid IR when the region contained multiple operations
leading up to the yield.

Fix this by cloning all operations in the region and using the operand
of  as the final result to store.
---
 .../Transforms/ACCSpecializeForHost.cpp       |  8 ++-
 .../acc-specialize-for-host-fallback.mlir     | 63 +++++++++++++++++++
 2 files changed, 69 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
index d3c8b683b76cb..5f41a1175f21f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
@@ -142,9 +142,13 @@ class ACCOrphanAtomicUpdateOpConversion
       }
       IRMapping mapping;
       mapping.map(atomicUpdateOp.getRegion().front().getArgument(0), loadOp);
-      Operation *expr = rewriter.clone(*atomicUpdateOp.getFirstOp(), mapping);
+      Block &block = atomicUpdateOp.getRegion().front();
+      for (Operation &op : block.without_terminator())
+        rewriter.clone(op, mapping);
+      auto yieldOp = cast<acc::YieldOp>(block.getTerminator());
+      Value result = mapping.lookup(yieldOp.getOperand(0));
       if (!ptrLikeType.genStore(rewriter, atomicUpdateOp.getLoc(),
-                                expr->getResult(0), xTyped)) {
+                                result, xTyped)) {
         accSupport.emitNYI(atomicUpdateOp.getLoc(),
                            "failed to generate store for atomic update");
         return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
index 59269b71bf61c..1a02f9fd710db 100644
--- a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
@@ -155,3 +155,66 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
   acc.declare_exit token(%token) dataOperands(%0 : memref<i32>)
   return
 }
+
+//===----------------------------------------------------------------------===//
+// Atomic operations (host fallback)
+//===----------------------------------------------------------------------===//
+
+acc.routine @acc_routine_atomic_read func(@atomic_read) seq
+// CHECK-LABEL: func.func @atomic_read
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.load
+func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
+  acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_write func(@atomic_write) seq
+// CHECK-LABEL: func.func @atomic_write
+// CHECK-NOT:   acc.atomic
+// CHECK:       memref.store
+func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
+  %val = arith.constant 5.0e-01 : f64
+  acc.atomic.write %addr = %val : memref<f64>, f64
+  return
+}
+
+acc.routine @acc_routine_atomic_update func(@atomic_update) seq
+// CHECK-LABEL: func.func @atomic_update
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_update]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.update %x : memref<f64> {
+    ^bb0(%arg0: f64):
+      %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+      %r = arith.select %c, %a, %arg0 : f64
+      acc.yield %r : f64
+    }
+    acc.terminator
+  }
+  return
+}
+
+acc.routine @acc_routine_atomic_capture func(@atomic_capture) seq
+// CHECK-LABEL: func.func @atomic_capture
+// CHECK-NOT:   acc.atomic
+// CHECK:       arith.select
+func.func @atomic_capture(%x : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_capture]>} {
+  %a = arith.constant 5.0e-01 : f64
+  acc.parallel {
+    acc.atomic.capture {
+      acc.atomic.update %x : memref<f64> {
+      ^bb0(%arg0: f64):
+        %c = arith.cmpf ogt, %a, %arg0 fastmath<contract> : f64
+        %r = arith.select %c, %a, %arg0 : f64
+        acc.yield %r : f64
+      }
+      acc.atomic.read %dst = %x : memref<f64>, memref<f64>, f64
+      acc.terminator
+    }
+    acc.terminator
+  }
+  return
+}

>From f8ac5cd2c0ffe4959075d752ac0381bf07c59837 Mon Sep 17 00:00:00 2001
From: ES3Q <ES3Q at QQ.COM>
Date: Tue, 7 Jul 2026 22:42:38 +0800
Subject: [PATCH 2/2] [MLIR][OpenACC] Improve test coverage for atomic host
 fallback

---
 .../OpenACC/acc-specialize-for-host-fallback.mlir   | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
index 1a02f9fd710db..587244184ab14 100644
--- a/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-specialize-for-host-fallback.mlir
@@ -163,7 +163,6 @@ func.func @declare_enter_exit(%arg0 : memref<i32>) attributes {acc.routine_info
 acc.routine @acc_routine_atomic_read func(@atomic_read) seq
 // CHECK-LABEL: func.func @atomic_read
 // CHECK-NOT:   acc.atomic
-// CHECK:       memref.load
 func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_read]>} {
   acc.atomic.read %dst = %src : memref<f64>, memref<f64>, f64
   return
@@ -172,7 +171,6 @@ func.func @atomic_read(%src : memref<f64>, %dst : memref<f64>) attributes {acc.r
 acc.routine @acc_routine_atomic_write func(@atomic_write) seq
 // CHECK-LABEL: func.func @atomic_write
 // CHECK-NOT:   acc.atomic
-// CHECK:       memref.store
 func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_write]>} {
   %val = arith.constant 5.0e-01 : f64
   acc.atomic.write %addr = %val : memref<f64>, f64
@@ -182,7 +180,10 @@ func.func @atomic_write(%addr : memref<f64>) attributes {acc.routine_info = #acc
 acc.routine @acc_routine_atomic_update func(@atomic_update) seq
 // CHECK-LABEL: func.func @atomic_update
 // CHECK-NOT:   acc.atomic
-// CHECK:       arith.select
+// CHECK: %[[VAL:.*]] = memref.load %arg0[] : memref<f64>
+// CHECK: %[[CMP:.*]] = arith.cmpf ogt, %[[A:.*]], %[[VAL]] fastmath<contract> : f64
+// CHECK: %[[RES:.*]] = arith.select %[[CMP]], %[[A]], %[[VAL]] : f64
+// CHECK: memref.store %[[RES]], %arg0[] : memref<f64>
 func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_update]>} {
   %a = arith.constant 5.0e-01 : f64
   acc.parallel {
@@ -200,7 +201,11 @@ func.func @atomic_update(%x : memref<f64>) attributes {acc.routine_info = #acc.r
 acc.routine @acc_routine_atomic_capture func(@atomic_capture) seq
 // CHECK-LABEL: func.func @atomic_capture
 // CHECK-NOT:   acc.atomic
-// CHECK:       arith.select
+// CHECK: %[[VAL:.*]] = memref.load %arg0[] : memref<f64>
+// CHECK: %[[CMP:.*]] = arith.cmpf ogt, %[[A:.*]], %[[VAL]] fastmath<contract> : f64
+// CHECK: %[[RES:.*]] = arith.select %[[CMP]], %[[A]], %[[VAL]] : f64
+// CHECK: memref.store %[[RES]], %arg0[] : memref<f64>
+// CHECK: memref.copy %arg0, %arg1 : memref<f64> to memref<f64>
 func.func @atomic_capture(%x : memref<f64>, %dst : memref<f64>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_atomic_capture]>} {
   %a = arith.constant 5.0e-01 : f64
   acc.parallel {


From llvmlistbot at llvm.org  Tue Jul  7 07:46:15 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 07 Jul 2026 07:46:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4d1137.92ea0e2a.13fd37.2334@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/207947

>From 5463401c92343bcf98437f9aca344c09b496358c Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 7 Jul 2026 11:20:48 +0200
Subject: [PATCH 1/3] [mlir][ArmSME] f64 scalable matmul crashes in
 VectorLegalizationPass

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../ArmSME/Transforms/VectorLegalization.cpp  |  4 ++-
 .../Dialect/ArmSME/vector-legalization.mlir   | 33 +++++++++++++++++++
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
index 6e2654ab468f9..7a6c510920e22 100644
--- a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
+++ b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
@@ -943,7 +943,9 @@ struct LowerColumnTransferReadToLoops
     auto upperBound = createVscaleMultiple(numRows);
     auto step = arith::ConstantIndexOp::create(rewriter, loc, 1);
     Value init = arith::ConstantOp::create(
-        rewriter, loc, newResType, DenseElementsAttr::get(newResType, 0.0f));
+        rewriter, loc, newResType,
+        DenseElementsAttr::get(newResType,
+                               rewriter.getZeroAttr(resType.getElementType())));
 
     scf::ForOp loadLoop;
     {
diff --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index 6cdf576272ebc..3490f67cf68ff 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -650,6 +650,39 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 
 // -----
 
+/// Same as above but for f64, which exercises a different zero-init constant
+/// (see https://github.com/iree-org/iree/issues/24689).
+
+// CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
+// CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[PAD:.*]]: f64,
+// CHECK-SAME:      %[[SRC:.*]]: memref<?x?xf64>) -> vector<[4]x1xf64> {
+func.func @xfer_read_scalable_column_f64(%a: index, %b: index, %pad: f64, %src: memref<?x?xf64>) -> (vector<[4]x1xf64>) {
+  // CHECK:           %[[INIT:.*]] = arith.constant dense<0.000000e+00> : vector<[4]xf64>
+  // CHECK:           %[[STEP:.*]] = arith.constant 1 : index
+  // CHECK:           %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:           %[[LB:.*]] = arith.constant 0 : index
+  // CHECK:           %[[VSCALE:.*]] = vector.vscale
+  // CHECK:           %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+
+  // <scf.for>
+  // CHECK:           %[[SCF:.*]] = scf.for %[[IND_VAR:.*]] = %[[LB]] to %[[C4_VSCALE]] step %[[STEP]] iter_args(%[[SCF_RES:.*]] = %[[INIT]]) -> (vector<[4]xf64>) {
+  // CHECK:             %[[IDX_0_UPDATED:.*]] = arith.addi %[[IND_VAR]], %[[IDX_0]] : index
+  // CHECK:             %[[VAL_10:.*]] = memref.load %[[SRC]][%[[IDX_0_UPDATED]], %[[IDX_1]]] : memref<?x?xf64>
+  // CHECK:             %[[RES_UPDATED:.*]] = vector.insert %[[VAL_10]], %[[SCF_RES]] [%[[IND_VAR]]] : f64 into vector<[4]xf64>
+  // CHECK:             scf.yield %[[RES_UPDATED]] : vector<[4]xf64>
+  // CHECK:           }
+
+  // <shape-cast>
+  // CHECK:           %[[SC:.*]] = vector.shape_cast %[[SCF]] : vector<[4]xf64> to vector<[4]x1xf64>
+  // CHECK:           return %[[SC]]
+  %read = vector.transfer_read %src[%a, %b], %pad : memref<?x?xf64>, vector<[4]x1xf64>
+  return %read : vector<[4]x1xf64>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @negative_xfer_read_scalable_column_x2
 func.func @negative_xfer_read_scalable_column_x2(%a: index, %b: index, %pad: f32, %src: memref<?x?xf32>) -> (vector<[4]x2xf32>) {
   // CHECK-NOT: scf.for

>From f85aab3c8d460f844675c31eed12015b3979b4b4 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 7 Jul 2026 11:37:02 +0200
Subject: [PATCH 2/3] Update mlir/test/Dialect/ArmSME/vector-legalization.mlir

Co-authored-by: Artem Gindinson <gindinson at roofline.ai>
---
 mlir/test/Dialect/ArmSME/vector-legalization.mlir | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index 3490f67cf68ff..a75590770799e 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -651,7 +651,8 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 // -----
 
 /// Same as above but for f64, which exercises a different zero-init constant
-/// (see https://github.com/iree-org/iree/issues/24689).
+/// Same as xfer_read_scalable_column but for f64, which exercises a different
+/// zero-init constant (see https://github.com/iree-org/iree/issues/24689).
 
 // CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
 // CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,

>From a95e020ac4c1cea33bbef45b1826ad558e68788f Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 7 Jul 2026 16:44:09 +0200
Subject: [PATCH 3/3] Address comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/test/Dialect/ArmSME/vector-legalization.mlir | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index a75590770799e..a31417f106a96 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -650,10 +650,6 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 
 // -----
 
-/// Same as above but for f64, which exercises a different zero-init constant
-/// Same as xfer_read_scalable_column but for f64, which exercises a different
-/// zero-init constant (see https://github.com/iree-org/iree/issues/24689).
-
 // CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
 // CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,
 // CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,


From llvmlistbot at llvm.org  Tue Jul  7 07:46:49 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Tue, 07 Jul 2026 07:46:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4d1159.bb839515.27d21e.6aa4@mx.google.com>


lhutton1 wrote:

Thanks for the fix @mydatascience and apologies for introducing these regressions. I'm curious to understand more about the failures you see in StableHLO? This fix makes the error message very generic "op operand #0 must be tosa-conformant tensor of number values" - it seems unfortunate to lose the ability to inform a user exactly what went wrong when constructing a block_scaled tensor

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 07:47:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 07:47:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3a6d3f7 - [mlir][LLVM] Add arithmetic fence
	intrinsic op (#207975)
Message-ID: <6a4d1167.af6ba3ed.36d9db.a66f@mx.google.com>


Author: Tom Eccles
Date: 2026-07-07T15:46:58+01:00
New Revision: 3a6d3f77960b4dded18d15e28eaa3f953be18faa

URL: https://github.com/llvm/llvm-project/commit/3a6d3f77960b4dded18d15e28eaa3f953be18faa
DIFF: https://github.com/llvm/llvm-project/commit/3a6d3f77960b4dded18d15e28eaa3f953be18faa.diff

LOG: [mlir][LLVM] Add arithmetic fence intrinsic op (#207975)

Add a generated LLVM dialect op for llvm.arithmetic.fence, including
LLVM IR import and export coverage for scalar and vector floating-point
forms.

Assisted-by: Codex

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
    mlir/test/Target/LLVMIR/Import/intrinsic.ll
    mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index 73629b95a06e5..76a3afad9a360 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -100,6 +100,13 @@ def LLVM_AbsOp : LLVM_OneResultIntrOp<"abs", [], [0], [Pure],
                    I1Attr:$is_int_min_poison);
 }
 
+// Do not use LLVM_UnaryIntrOpF here: arithmetic.fence operates on floating
+// types, but the fence itself should not carry fastmath flags.
+def LLVM_ArithmeticFenceOp
+    : LLVM_UnaryIntrOpBase<"arithmetic.fence", LLVM_AnyFloat> {
+  let arguments = commonArgs;
+}
+
 def LLVM_IsFPClass : LLVM_OneResultIntrOp<"is.fpclass", [], [0], [Pure],
   /*requiresFastmath=*/0, /*requiresArgAndResultAttrs=*/0,
   /*immArgPositions=*/[1], /*immArgAttrNames=*/["bit"]> {

diff  --git a/mlir/test/Target/LLVMIR/Import/intrinsic.ll b/mlir/test/Target/LLVMIR/Import/intrinsic.ll
index d3427b4ab9771..d50a5e39bbe3a 100644
--- a/mlir/test/Target/LLVMIR/Import/intrinsic.ll
+++ b/mlir/test/Target/LLVMIR/Import/intrinsic.ll
@@ -104,6 +104,14 @@ define void @fabs_test(float %0, <8 x float> %1) {
   %4 = call <8 x float> @llvm.fabs.v8f32(<8 x float> %1)
   ret void
 }
+; CHECK-LABEL:  llvm.func @arithmetic_fence_test
+define void @arithmetic_fence_test(float %0, <8 x float> %1) {
+  ; CHECK: llvm.intr.arithmetic.fence(%{{.*}}) : (f32) -> f32
+  %3 = call float @llvm.arithmetic.fence.f32(float %0)
+  ; CHECK: llvm.intr.arithmetic.fence(%{{.*}}) : (vector<8xf32>) -> vector<8xf32>
+  %4 = call <8 x float> @llvm.arithmetic.fence.v8f32(<8 x float> %1)
+  ret void
+}
 ; CHECK-LABEL:  llvm.func @sqrt_test
 define void @sqrt_test(float %0, <8 x float> %1) {
   ; CHECK: llvm.intr.sqrt(%{{.*}}) : (f32) -> f32

diff  --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 5266470891b4d..6910a06992b55 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -103,6 +103,15 @@ llvm.func @fabs_test(%arg0: f32, %arg1: vector<8xf32>) {
   llvm.return
 }
 
+// CHECK-LABEL: @arithmetic_fence_test
+llvm.func @arithmetic_fence_test(%arg0: f32, %arg1: vector<8xf32>) {
+  // CHECK: call float @llvm.arithmetic.fence.f32
+  "llvm.intr.arithmetic.fence"(%arg0) : (f32) -> f32
+  // CHECK: call <8 x float> @llvm.arithmetic.fence.v8f32
+  "llvm.intr.arithmetic.fence"(%arg1) : (vector<8xf32>) -> vector<8xf32>
+  llvm.return
+}
+
 // CHECK-LABEL: @sqrt_test
 llvm.func @sqrt_test(%arg0: f32, %arg1: vector<8xf32>) {
   // CHECK: call float @llvm.sqrt.f32


        

From llvmlistbot at llvm.org  Tue Jul  7 07:47:05 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Tue, 07 Jul 2026 07:47:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVM] Add arithmetic fence intrinsic
	op (PR #207975)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207975@github.com>
Message-ID: <6a4d1169.da8bafd3.14cd50.5bb7@mx.google.com>


https://github.com/tblah closed https://github.com/llvm/llvm-project/pull/207975

From llvmlistbot at llvm.org  Tue Jul  7 07:56:01 2026
From: llvmlistbot at llvm.org (Ming Yan)
Date: Tue, 07 Jul 2026 07:56:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToAffine] Raise scf.for to
	affine.for (PR #200851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200851@github.com>
Message-ID: <6a4d1381.cbc3f03a.10d998.87e3@mx.google.com>


NexMing wrote:

If no one else has any objections, we can merge this PR.

https://github.com/llvm/llvm-project/pull/200851

From llvmlistbot at llvm.org  Tue Jul  7 07:56:45 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Tue, 07 Jul 2026 07:56:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4d13ad.fdc477d6.23591.a0af@mx.google.com>


mydatascience wrote:

https://github.com/openxla/stablehlo/blob/9793611e490531cc40a7a4e2a4e92caa7ea89f0f/stablehlo/conversions/tosa/transforms/StablehloLegalizeToTosa.cpp#L52 here I see error: use of undeclared identifier 

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 07:59:14 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Tue, 07 Jul 2026 07:59:14 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a4d1442.58bdd12c.19c81.c6c2@mx.google.com>


https://github.com/tblah approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Tue Jul  7 08:04:32 2026
From: llvmlistbot at llvm.org (Kevin Gleason)
Date: Tue, 07 Jul 2026 08:04:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4d1580.cbc3f03a.10d998.75ce@mx.google.com>


GleasonK wrote:

I could see this as a bug in two places, but I'm not sure which is right since I'm unclear if it should be supported that a Type verifier can have a dependency on an Operation. There are plenty of places a type can exist without an op (I'm curious if using one of these as a `TypedAttr` would cause a crash).

(1) Bug that Type verifiers support passing `$_op` since it permits this dependency.
(2) Support Types with Op dependencies in PDLL.

Our failure was that PDLL lowerings cannot use type verifiers with op dependencies, lead to a "could not resolve $_op" error message.

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Tue Jul  7 08:08:32 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 07 Jul 2026 08:08:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4d1670.70920dee.290dca.ca32@mx.google.com>


https://github.com/nujaa updated https://github.com/llvm/llvm-project/pull/207005

>From 3a79baaa2f3e2c39ee7479a7f716865626b88881 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Wed, 1 Jul 2026 21:08:28 +0800
Subject: [PATCH 1/2] [MLIR][Linalg] Fix FlattenElementwiseOp on broadcasted
 linalgs

---
 .../TransformOps/LinalgTransformOps.cpp       |  8 +++
 .../Dialect/Linalg/flatten-elementwise.mlir   | 53 +++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index f44693096b26b..d39c1cb0e4a20 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4297,6 +4297,14 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return DiagnosedSilenceableFailure::success();
   }
 
+  // Bail out on linalgs with broadcasting semantics
+  if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
+        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+      })) {
+    results.push_back(target);
+    return DiagnosedSilenceableFailure::success();
+  }
+
   // Attempt to flatten all dims to one.
   ReassociationIndices reassociation(target.getNumLoops());
   std::iota(reassociation.begin(), reassociation.end(), 0);
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 9fe50a521d2d8..1c6ba43888d7b 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -118,3 +118,56 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<1x2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:     %[[RES:.*]] = linalg.generic
+// CHECK:          return %[[RES]]
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+// CHECK-LABEL: func.func @generic
+// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<2x1xi32>
+// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
+// CHECK-NEXT:     %[[RES:.*]] = linalg.generic
+// CHECK:          return %[[RES]]
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @generic(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+      ^bb0(%in: i32, %out: i32):
+        linalg.yield %in : i32
+    } -> tensor<32x2x2xi32>
+    return %13 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
\ No newline at end of file

>From c181d014aa9a96cd9e13d9cc9786f13eed40aff3 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Tue, 7 Jul 2026 23:07:32 +0800
Subject: [PATCH 2/2] fixup! [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs

---
 .../TransformOps/LinalgTransformOps.cpp       | 14 ++--
 .../Dialect/Linalg/flatten-elementwise.mlir   | 69 +++----------------
 .../Dialect/Linalg/flatten-unsupported.mlir   | 51 +++++++++++++-
 3 files changed, 63 insertions(+), 71 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index d39c1cb0e4a20..2b7d4e4ff6d8c 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4291,17 +4291,17 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return mlir::emitSilenceableFailure(target->getLoc())
            << "only elementwise flattening is supported";
 
-  // If rank <= 1, do nothing
-  if (target.getNumLoops() <= 1) {
-    results.push_back(target);
-    return DiagnosedSilenceableFailure::success();
-  }
-
-  // Bail out on linalgs with broadcasting semantics
   if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
         return m.isProjectedPermutation(/*allowZeroInResults=*/false);
       })) {
     results.push_back(target);
+    return mlir::emitSilenceableFailure(target->getLoc())
+           << "operators with broadcasting semantics are not supported";
+  }
+
+  // If rank <= 1, do nothing
+  if (target.getNumLoops() <= 1) {
+    results.push_back(target);
     return DiagnosedSilenceableFailure::success();
   }
 
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 1c6ba43888d7b..f9f8f9e2b8b44 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -1,11 +1,11 @@
 // RUN: mlir-opt %s -transform-interpreter -split-input-file | FileCheck %s
 
-// CHECK-LABEL: func.func @fill(
+// CHECK-LABEL: func.func @fill_memref(
 // CHECK-SAME:                  %[[ARG0:.*]]: f32,
 // CHECK-SAME:                  %[[ARG1:.*]]: memref<32x7xf32>
 // CHECK-NEXT:    %[[FLATTENED:.*]] = memref.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
 // CHECK-NEXT:    linalg.fill ins(%[[ARG0]] : f32) outs(%[[FLATTENED]] : memref<224xf32>)
-func.func @fill(%cst: f32, %arg: memref<32x7xf32>) {
+func.func @fill_memref(%cst: f32, %arg: memref<32x7xf32>) {
     linalg.fill ins(%cst: f32) outs(%arg: memref<32x7xf32>)
     return
 }
@@ -43,7 +43,7 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
-// CHECK-LABEL: func.func @map(
+// CHECK-LABEL: func.func @map_memref(
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32x7xf32>
@@ -51,7 +51,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK-NEXT:    %[[FLATTENED_1:.*]] = memref.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
 // CHECK-NEXT:    %[[FLATTENED_2:.*]] = memref.collapse_shape %[[ARG2]] {{\[}}[0, 1]]
 // CHECK-NEXT:    linalg.map { arith.addf } ins(%[[FLATTENED_0]], %[[FLATTENED_1]] : memref<224xf32>, memref<224xf32>) outs(%[[FLATTENED_2]] : memref<224xf32>)
-func.func @map(%arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
+func.func @map_memref(%arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
     linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32>, memref<32x7xf32>) outs(%arg2: memref<32x7xf32>)
     return
 }
@@ -67,12 +67,12 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
-// CHECK-LABEL: func.func @map_already_flat(
+// CHECK-LABEL: func.func @map_already_flat_memref(
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-NEXT:    linalg.map { arith.addf } ins(%[[ARG0]], %[[ARG1]] : memref<32xf32>, memref<32xf32>) outs(%[[ARG2]] : memref<32xf32>)
-func.func @map_already_flat(%arg0: memref<32xf32>, %arg1: memref<32xf32>, %arg2: memref<32xf32>) {
+func.func @map_already_flat_memref(%arg0: memref<32xf32>, %arg1: memref<32xf32>, %arg2: memref<32xf32>) {
     linalg.map {arith.addf} ins(%arg0, %arg1: memref<32xf32>, memref<32xf32>) outs(%arg2: memref<32xf32>)
     return
 }
@@ -89,7 +89,7 @@ module attributes {transform.with_named_sequence} {
 // -----
 
 // CHECK: #[[$MAP0:.*]] = affine_map<(d0) -> (d0)>
-// CHECK-LABEL: func.func @generic
+// CHECK-LABEL: func.func @elementwise_memref
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32x7xf32>
@@ -101,7 +101,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK-NEXT:         %[[SUM:.*]] = arith.addf %[[A]], %[[B]]
 // CHECK-NEXT:         linalg.yield %[[SUM]]
 #map = affine_map<(d0, d1) -> (d0, d1)>
-func.func @generic( %arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
+func.func @elementwise_memref( %arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
     linalg.generic {indexing_maps = [#map, #map, #map], iterator_types = ["parallel", "parallel"]} ins(%arg0, %arg1: memref<32x7xf32>, memref<32x7xf32>) outs(%arg2: memref<32x7xf32>) {
         ^bb0(%a: f32, %b: f32, %c: f32):
             %0 = arith.addf %a, %b : f32
@@ -118,56 +118,3 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
-
-// -----
-// CHECK-LABEL: func.func @generic
-// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<1x2x1xi32>
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
-// CHECK-NEXT:     %[[RES:.*]] = linalg.generic
-// CHECK:          return %[[RES]]
-#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
-#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
-
-func.func @generic(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
-    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
-      ^bb0(%in: i32, %out: i32):
-        linalg.yield %in : i32
-    } -> tensor<32x2x2xi32>
-    return %13 : tensor<32x2x2xi32>
-}
-
-module attributes {transform.with_named_sequence} {
-  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
-    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
-    %flattened = transform.structured.flatten_elementwise %0
-      : (!transform.any_op) -> !transform.any_op
-    transform.yield
-  }
-}
-
-// -----
-
-// CHECK-LABEL: func.func @generic
-// CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]*]]: tensor<2x1xi32>
-// CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]*]]: tensor<32x2x2xi32>
-// CHECK-NEXT:     %[[RES:.*]] = linalg.generic
-// CHECK:          return %[[RES]]
-#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
-#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
-
-func.func @generic(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
-    %13 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
-      ^bb0(%in: i32, %out: i32):
-        linalg.yield %in : i32
-    } -> tensor<32x2x2xi32>
-    return %13 : tensor<32x2x2xi32>
-}
-
-module attributes {transform.with_named_sequence} {
-  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
-    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
-    %flattened = transform.structured.flatten_elementwise %0
-      : (!transform.any_op) -> !transform.any_op
-    transform.yield
-  }
-}
\ No newline at end of file
diff --git a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
index 499db4cfb3294..f0b8b8b153bbb 100644
--- a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -transform-interpreter -split-input-file -verify-diagnostics
 
-func.func @non_elementwise(%arg0: memref<2x3xf32>, %arg1: memref<3x4xf32>, %arg2: memref<2x4xf32>) {
+func.func @unsupported_non_elementwise(%arg0: memref<2x3xf32>, %arg1: memref<3x4xf32>, %arg2: memref<2x4xf32>) {
   // expected-error @below {{only elementwise flattening is supported}}
   linalg.matmul ins(%arg0, %arg1 : memref<2x3xf32>, memref<3x4xf32>) outs(%arg2: memref<2x4xf32>)
   return
@@ -19,8 +19,53 @@ module attributes {transform.with_named_sequence} {
 
 func.func @unsupported_memref(%arg0: memref<32x7xf32, strided<[7, 2]>>, %arg1: memref<32x7xf32, strided<[7, 2]>>, %arg2: memref<32x7xf32, strided<[7, 2]>>) {
   // expected-error @below {{attempted to flatten, but failed}}
-    linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32, strided<[7, 2]>>, memref<32x7xf32, strided<[7, 2]>>) outs(%arg2: memref<32x7xf32, strided<[7, 2]>>)
-    return
+  linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32, strided<[7, 2]>>, memref<32x7xf32, strided<[7, 2]>>) outs(%arg2: memref<32x7xf32, strided<[7, 2]>>)
+  return
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @unsupported_broadcasting_elementwise(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+    ^bb0(%in: i32, %out: i32):
+      linalg.yield %in : i32
+  } -> tensor<32x2x2xi32>
+  return %0 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @unsupported_rank_expanding_broadcasting_elementwise(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+    ^bb0(%in: i32, %out: i32):
+      linalg.yield %in : i32
+  } -> tensor<32x2x2xi32>
+  return %0 : tensor<32x2x2xi32>
 }
 
 module attributes {transform.with_named_sequence} {


From llvmlistbot at llvm.org  Tue Jul  7 08:10:47 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 07 Jul 2026 08:10:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a4d16f7.cb30c3e4.3ce559.9420@mx.google.com>


================

----------------
nujaa wrote:

Preferred to return a silenceable error and moved tests to `flatten-unsupported.mlir`. Seized the opportunity for some test renaming :)

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Tue Jul  7 08:19:22 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Tue, 07 Jul 2026 08:19:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add normalize pass to MLIR (PR #186647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/186647@github.com>
Message-ID: <6a4d18fa.d5797644.338ca1.f7ab@mx.google.com>


https://github.com/linuxlonelyeagle updated https://github.com/llvm/llvm-project/pull/186647

>From d6f66c9c22d70eb3722b8e520331e872234467af Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Fri, 13 Mar 2026 02:26:50 +0000
Subject: [PATCH 1/4] complete basic normalize.

---
 mlir/include/mlir/Transforms/Passes.h  |   1 +
 mlir/include/mlir/Transforms/Passes.td |   8 ++
 mlir/lib/Transforms/CMakeLists.txt     |   1 +
 mlir/lib/Transforms/Normalize.cpp      | 112 +++++++++++++++++++++++++
 mlir/test/Transforms/normalize.mlir    |  84 +++++++++++++++++++
 5 files changed, 206 insertions(+)
 create mode 100644 mlir/lib/Transforms/Normalize.cpp
 create mode 100644 mlir/test/Transforms/normalize.mlir

diff --git a/mlir/include/mlir/Transforms/Passes.h b/mlir/include/mlir/Transforms/Passes.h
index 313755fde3479..be56a548f63b1 100644
--- a/mlir/include/mlir/Transforms/Passes.h
+++ b/mlir/include/mlir/Transforms/Passes.h
@@ -42,6 +42,7 @@ class GreedyRewriteConfig;
 #define GEN_PASS_DECL_LOOPINVARIANTSUBSETHOISTINGPASS
 #define GEN_PASS_DECL_INLINERPASS
 #define GEN_PASS_DECL_MEM2REG
+#define GEN_PASS_DECL_NORMALIZEPASS
 #define GEN_PASS_DECL_PRINTIRPASS
 #define GEN_PASS_DECL_PRINTOPSTATSPASS
 #define GEN_PASS_DECL_REMOVEDEADVALUESPASS
diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index 74ac370ea950b..10193bbcef2bd 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -656,4 +656,12 @@ def BubbleDownMemorySpaceCasts :
   }];
 }
 
+def NormalizePass : InterfacePass<"normalize", "FunctionOpInterface"> {
+  let summary = "Transforms IR into a normal form that's easier to diff.";
+  let description = [{
+    This pass attempts to relocate the defining ops of operands for any
+    side-effecting or terminator operation to their nearest dominating positions.
+  }];
+}
+
 #endif // MLIR_TRANSFORMS_PASSES
diff --git a/mlir/lib/Transforms/CMakeLists.txt b/mlir/lib/Transforms/CMakeLists.txt
index 66b39f53c91df..41347d23b0f49 100644
--- a/mlir/lib/Transforms/CMakeLists.txt
+++ b/mlir/lib/Transforms/CMakeLists.txt
@@ -12,6 +12,7 @@ add_mlir_library(MLIRTransforms
   LocationSnapshot.cpp
   LoopInvariantCodeMotion.cpp
   Mem2Reg.cpp
+  Normalize.cpp
   OpStats.cpp
   PrintIR.cpp
   RemoveDeadValues.cpp
diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
new file mode 100644
index 0000000000000..1904d25d36674
--- /dev/null
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -0,0 +1,112 @@
+//===- Normalize.cpp - Transforms IR into a normal form ---------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/IR/Dominance.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/Passes.h"
+#include "llvm/Support/DebugLog.h"
+
+using namespace mlir;
+
+namespace mlir {
+#define GEN_PASS_DEF_NORMALIZEPASS
+#include "mlir/Transforms/Passes.h.inc"
+} // namespace mlir
+
+#define DEBUG_TYPE "normalize"
+
+namespace {
+
+bool isOutput(Operation *op) {
+  if (!op)
+    return false;
+  return !isMemoryEffectFree(op) || op->hasTrait<OpTrait::IsTerminator>();
+}
+
+/// Returns a vector of output ops. An output is a op which
+/// has side-effects or is terminator.
+SmallVector<Operation *> collectOutputs(Operation *root) {
+  SmallVector<Operation *> outputs;
+  root->walk([&](Operation *op) {
+    if (isOutput(op))
+      outputs.push_back(op);
+  });
+  return outputs;
+}
+
+/// The function returns the operation that dominates all other operations in
+/// the given list.
+Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
+  if (ops.empty())
+    return {};
+  Operation *curDomOp = ops.front();
+  DominanceInfo domInfo(curDomOp);
+  for (size_t i = 1, e = ops.size(); i < e; ++i) {
+    bool dominateA = domInfo.dominates(ops[i], curDomOp);
+    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
+    if (dominateA) {
+      LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
+             << "\ndominate\n"
+             << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions());
+      curDomOp = ops[i];
+    }
+    if (!dominateA && !dominateB) {
+      LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
+             << "\nand\n"
+             << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions())
+             << "\ndo not dominate each other";
+      return {};
+    }
+  }
+  return curDomOp;
+}
+
+/// Move used to its nearest user and recursively perform the same process on
+/// the defining operations of its operands.
+void reorderOutput(IRRewriter &rewriter, Operation *used) {
+  if (!isPure(used))
+    return;
+  SmallVector<Operation *> users(used->getUsers());
+  if (Operation *domOp = getDominateOp(users)) {
+    rewriter.moveOpBefore(used, domOp);
+    for (Value operand : used->getOperands())
+      if (Operation *defineOp = operand.getDefiningOp())
+        reorderOutput(rewriter, defineOp);
+  }
+}
+
+/// Reorders ops by walking up the tree from each operand of an output op and
+/// reducing the def-use distance. This method assumes that output ops were
+/// collected top-down, otherwise the def-use chain may be broken. This method
+/// is a wrapper for recursive reorderOutput().
+void reorderOutputs(IRRewriter &rewriter,
+                    SmallVectorImpl<Operation *> &outputs) {
+  SmallPtrSet<Operation *, 16> visited;
+  for (Operation *output : outputs) {
+    for (Value operand : output->getOperands()) {
+      if (Operation *defineOp = operand.getDefiningOp();
+          defineOp && !visited.contains(defineOp)) {
+        reorderOutput(rewriter, defineOp);
+      }
+    }
+  }
+}
+
+struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
+  using impl::NormalizePassBase<NormalizePass>::NormalizePassBase;
+  void runOnOperation() override;
+};
+} // namespace
+
+void NormalizePass::runOnOperation() {
+  IRRewriter rewriter(&getContext());
+  SmallVector<Operation *> outputs = collectOutputs(getOperation());
+  reorderOutputs(rewriter, outputs);
+}
diff --git a/mlir/test/Transforms/normalize.mlir b/mlir/test/Transforms/normalize.mlir
new file mode 100644
index 0000000000000..241e28c3c385d
--- /dev/null
+++ b/mlir/test/Transforms/normalize.mlir
@@ -0,0 +1,84 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize))" -split-input-file | FileCheck %s
+
+// CHECK-LABEL: func @multiple_memref_store
+//  CHECK-SAME:   %[[ARG0:.*]]: index,
+//  CHECK-SAME:   %[[ARG1:.*]]: memref<?xf32>
+func.func @multiple_memref_store(%arg0: index, %arg1 : memref<?xf32>) {
+  %f0 = arith.constant 0.0 : f32
+  %f1 = arith.constant 1.0 : f32
+  %add = arith.addi %arg0, %arg0 : index
+  %sub = arith.subi %arg0, %arg0 : index
+  memref.store %f0, %arg1[%add] : memref<?xf32>
+  memref.store %f1, %arg1[%sub] : memref<?xf32>
+  return
+}
+
+// CHECK-NEXT: %[[C0:.*]] = arith.constant 0.000000e+00 : f32
+// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: memref.store %[[C0]], %[[ARG1]]{{\[}}%[[ADD]]] : memref<?xf32>
+// CHECK-NEXT: %[[C1:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: memref.store %[[C1]], %[[ARG1]]{{\[}}%[[SUB]]] : memref<?xf32>
+
+// -----
+
+// CHECK-LABEL: func @normalize_return
+// CHECK-SAME:      %[[ARG0:.*]]: index,
+// CHECK-SAME:      %[[ARG1:.*]]: memref<?xf32>
+func.func @normalize_return(%arg0: index, %arg1 : memref<?xf32>) -> index {
+  %f0 = arith.constant 0.0 : f32
+  %add = arith.addi %arg0, %arg0 : index
+  %sub = arith.subi %arg0, %arg0 : index
+  memref.store %f0, %arg1[%add] : memref<?xf32>
+  return %sub : index
+}
+
+//      CHECK: memref.store
+// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: return %[[SUB]] : index
+
+// -----
+
+// CHECK-LABEL: func @cross_region
+//  CHECK-SAME:   %[[ARG0:.*]]: f32,
+//  CHECK-SAME:   %[[ARG1:.*]]: memref<10xf32>
+func.func @cross_region(%arg0: f32, %arg1 : memref<10xf32>) {
+  %add = arith.addf %arg0, %arg0 : f32
+  affine.for %i = 0 to 5 {
+    memref.store %add, %arg1[%i] : memref<10xf32>
+  }
+  %exp = math.log2 %add : f32
+  affine.for %i = 6 to 10 {
+    memref.store %exp, %arg1[%i] : memref<10xf32>
+  } 
+  return
+}
+
+//      CHECK: affine.for %[[IV:.*]] = 6 to 10 {
+// CHECK-NEXT:   %[[LOG:.*]] = math.log2
+// CHECK-NEXT:   memref.store %[[LOG]], %[[ARG1]]{{\[}}%[[IV]]] : memref<10xf32>
+// CHECK-NEXT: }
+
+// -----
+
+
+// CHECK-LABEL: func @side_effect_for_op
+//  CHECK-SAME:   %[[ARG0:.*]]: memref<?xf32>
+func.func @side_effect_for_op(%arg1 : memref<?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %upper = memref.dim %arg1, %c0 : memref<?xf32>
+  %f1 = arith.constant 1.0 : f32
+  scf.for %i = %c0 to %upper step %c1 {
+    memref.store %f1, %arg1[%i] : memref<?xf32>
+  }
+  return
+}
+
+// CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+// CHECK-NEXT: %[[DIM:.*]] = memref.dim %[[ARG0]], %[[C0]] : memref<?xf32>
+// CHECK-NEXT: %[[C1:.*]] = arith.constant 1 : index
+// CHECK-NEXT: scf.for %[[IV:.*]] = %[[C0]] to %[[DIM]] step %[[C1]] {
+// CHECK-NEXT:   %[[F1:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-NEXT:   memref.store %[[F1]], %[[ARG0]]{{\[}}%[[IV]]] : memref<?xf32>
+// CHECK-NEXT: }

>From b025af24e07486c0142bec3f76c0e9cafb559d3f Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 16 Mar 2026 06:58:47 +0000
Subject: [PATCH 2/4] fix nit and add more comment.

---
 mlir/include/mlir/Transforms/Passes.td | 10 ++++++++--
 mlir/lib/Transforms/Normalize.cpp      | 20 ++++++++++---------
 mlir/test/Transforms/normalize.mlir    | 27 +++++++++++++-------------
 3 files changed, 33 insertions(+), 24 deletions(-)

diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index 10193bbcef2bd..a03c69c12461e 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -659,8 +659,14 @@ def BubbleDownMemorySpaceCasts :
 def NormalizePass : InterfacePass<"normalize", "FunctionOpInterface"> {
   let summary = "Transforms IR into a normal form that's easier to diff.";
   let description = [{
-    This pass attempts to relocate the defining ops of operands for any
-    side-effecting or terminator operation to their nearest dominating positions.
+    This pass aims to transform MLIR Modules into a normal form by reordering
+    operations while preserving the same semantics. It attempts to relocate
+    the defining ops of operands for any side-effecting or terminator operation
+    to their nearest dominating positions.
+
+    Note: The pass trying to increase syntactic equivalence of code to reduce
+    diff size while retaining semantic equivalence. It cannot replace the
+    canonicalization pass (a pass that aims for increasing semantic equivalence instead).
   }];
 }
 
diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 1904d25d36674..7e63c42b6d163 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -43,21 +43,22 @@ SmallVector<Operation *> collectOutputs(Operation *root) {
 
 /// The function returns the operation that dominates all other operations in
 /// the given list.
-Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
+Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
   if (ops.empty())
     return {};
   Operation *curDomOp = ops.front();
   DominanceInfo domInfo(curDomOp);
   for (size_t i = 1, e = ops.size(); i < e; ++i) {
     bool dominateA = domInfo.dominates(ops[i], curDomOp);
-    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
     if (dominateA) {
       LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
              << "\ndominate\n"
              << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions());
       curDomOp = ops[i];
+      continue;
     }
-    if (!dominateA && !dominateB) {
+    bool dominateB = domInfo.dominates(curDomOp, ops[i]);
+    if (!dominateB) {
       LDBG() << OpWithFlags(ops[i], OpPrintingFlags().skipRegions())
              << "\nand\n"
              << OpWithFlags(curDomOp, OpPrintingFlags().skipRegions())
@@ -70,13 +71,13 @@ Operation *getDominateOp(SmallVectorImpl<Operation *> &ops) {
 
 /// Move used to its nearest user and recursively perform the same process on
 /// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *used) {
-  if (!isPure(used))
+void reorderOutput(IRRewriter &rewriter, Operation *producer) {
+  if (!isPure(producer))
     return;
-  SmallVector<Operation *> users(used->getUsers());
+  SmallVector<Operation *> users(producer->getUsers());
   if (Operation *domOp = getDominateOp(users)) {
-    rewriter.moveOpBefore(used, domOp);
-    for (Value operand : used->getOperands())
+    rewriter.moveOpBefore(producer, domOp);
+    for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
         reorderOutput(rewriter, defineOp);
   }
@@ -87,12 +88,13 @@ void reorderOutput(IRRewriter &rewriter, Operation *used) {
 /// collected top-down, otherwise the def-use chain may be broken. This method
 /// is a wrapper for recursive reorderOutput().
 void reorderOutputs(IRRewriter &rewriter,
-                    SmallVectorImpl<Operation *> &outputs) {
+                    const SmallVectorImpl<Operation *> &outputs) {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
+        visited.insert(defineOp);
         reorderOutput(rewriter, defineOp);
       }
     }
diff --git a/mlir/test/Transforms/normalize.mlir b/mlir/test/Transforms/normalize.mlir
index 241e28c3c385d..28f0c05263278 100644
--- a/mlir/test/Transforms/normalize.mlir
+++ b/mlir/test/Transforms/normalize.mlir
@@ -22,20 +22,17 @@ func.func @multiple_memref_store(%arg0: index, %arg1 : memref<?xf32>) {
 
 // -----
 
-// CHECK-LABEL: func @normalize_return
-// CHECK-SAME:      %[[ARG0:.*]]: index,
-// CHECK-SAME:      %[[ARG1:.*]]: memref<?xf32>
-func.func @normalize_return(%arg0: index, %arg1 : memref<?xf32>) -> index {
-  %f0 = arith.constant 0.0 : f32
-  %add = arith.addi %arg0, %arg0 : index
-  %sub = arith.subi %arg0, %arg0 : index
-  memref.store %f0, %arg1[%add] : memref<?xf32>
-  return %sub : index
+// CHECK-LABEL: func @return_multiple_operands
+//  CHECK-SAME:   %[[ARG0:.*]]: index
+func.func @return_multiple_operands (%arg0: index) -> (index, index) {
+  %0 = arith.addi %arg0, %arg0 : index
+  %1 = arith.subi %arg0, %arg0 : index
+  return %1, %0 : index, index
 }
 
-//      CHECK: memref.store
 // CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
-// CHECK-NEXT: return %[[SUB]] : index
+// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: return %[[SUB]], %[[ADD]] : index, index
 
 // -----
 
@@ -61,10 +58,14 @@ func.func @cross_region(%arg0: f32, %arg1 : memref<10xf32>) {
 
 // -----
 
+// This test verifies the reordering of scf.for ops.
+// The memref.store within the scf.for causes the loop to have side effects.
+// The lower bound of the scf.for remains in its original position
+// because the upper bound depends on it, but the step has been reordered.
 
-// CHECK-LABEL: func @side_effect_for_op
+// CHECK-LABEL: func @side_effect_loop_op
 //  CHECK-SAME:   %[[ARG0:.*]]: memref<?xf32>
-func.func @side_effect_for_op(%arg1 : memref<?xf32>) {
+func.func @side_effect_loop_op(%arg1 : memref<?xf32>) {
   %c0 = arith.constant 0 : index
   %c1 = arith.constant 1 : index
   %upper = memref.dim %arg1, %c0 : memref<?xf32>

>From c91b6a5f4b2bada8263b6063854c8cb39fc408c7 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Sun, 19 Apr 2026 05:00:23 +0000
Subject: [PATCH 3/4] use getAnalysis to get DominanceInfo and use
 markAnalysesPreserved.

---
 mlir/lib/Transforms/Normalize.cpp | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 7e63c42b6d163..6f42c61ee7d18 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -43,11 +43,11 @@ SmallVector<Operation *> collectOutputs(Operation *root) {
 
 /// The function returns the operation that dominates all other operations in
 /// the given list.
-Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
+Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
+                         const DominanceInfo &domInfo) {
   if (ops.empty())
     return {};
   Operation *curDomOp = ops.front();
-  DominanceInfo domInfo(curDomOp);
   for (size_t i = 1, e = ops.size(); i < e; ++i) {
     bool dominateA = domInfo.dominates(ops[i], curDomOp);
     if (dominateA) {
@@ -71,15 +71,16 @@ Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops) {
 
 /// Move used to its nearest user and recursively perform the same process on
 /// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *producer) {
+void reorderOutput(IRRewriter &rewriter, Operation *producer,
+                   const DominanceInfo &domInfo) {
   if (!isPure(producer))
     return;
   SmallVector<Operation *> users(producer->getUsers());
-  if (Operation *domOp = getDominateOp(users)) {
+  if (Operation *domOp = getDominateOp(users, domInfo)) {
     rewriter.moveOpBefore(producer, domOp);
     for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
-        reorderOutput(rewriter, defineOp);
+        reorderOutput(rewriter, defineOp, domInfo);
   }
 }
 
@@ -88,14 +89,15 @@ void reorderOutput(IRRewriter &rewriter, Operation *producer) {
 /// collected top-down, otherwise the def-use chain may be broken. This method
 /// is a wrapper for recursive reorderOutput().
 void reorderOutputs(IRRewriter &rewriter,
-                    const SmallVectorImpl<Operation *> &outputs) {
+                    const SmallVectorImpl<Operation *> &outputs,
+                    const DominanceInfo &domInfo) {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
         visited.insert(defineOp);
-        reorderOutput(rewriter, defineOp);
+        reorderOutput(rewriter, defineOp, domInfo);
       }
     }
   }
@@ -108,7 +110,11 @@ struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
 } // namespace
 
 void NormalizePass::runOnOperation() {
+  DominanceInfo &domInfo = getAnalysis<DominanceInfo>();
   IRRewriter rewriter(&getContext());
   SmallVector<Operation *> outputs = collectOutputs(getOperation());
-  reorderOutputs(rewriter, outputs);
+  reorderOutputs(rewriter, outputs, domInfo);
+  // Since we only changed the positions of the operations, DominanceInfo and
+  // PostDominanceInfo are marked as preserved.
+  markAnalysesPreserved<DominanceInfo, PostDominanceInfo>();
 }

>From 4046f799eb3296b58af7e8dedf0987946f509112 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Mon, 6 Jul 2026 08:44:03 +0000
Subject: [PATCH 4/4] impl rename logic and add test.

---
 mlir/include/mlir/Transforms/Passes.td |  10 +-
 mlir/lib/Transforms/Normalize.cpp      | 473 +++++++++++++++++++++++--
 mlir/test/Transforms/normalize.mlir    | 117 +++++-
 3 files changed, 557 insertions(+), 43 deletions(-)

diff --git a/mlir/include/mlir/Transforms/Passes.td b/mlir/include/mlir/Transforms/Passes.td
index a03c69c12461e..f195588e4ca67 100644
--- a/mlir/include/mlir/Transforms/Passes.td
+++ b/mlir/include/mlir/Transforms/Passes.td
@@ -662,12 +662,20 @@ def NormalizePass : InterfacePass<"normalize", "FunctionOpInterface"> {
     This pass aims to transform MLIR Modules into a normal form by reordering
     operations while preserving the same semantics. It attempts to relocate
     the defining ops of operands for any side-effecting or terminator operation
-    to their nearest dominating positions.
+    to their nearest dominating positions. In addition, it makes it easier to see
+    which SSA values an op uses just by reading its renamed name.
 
     Note: The pass trying to increase syntactic equivalence of code to reduce
     diff size while retaining semantic equivalence. It cannot replace the
     canonicalization pass (a pass that aims for increasing semantic equivalence instead).
   }];
+  let options = [
+    Option<"foldDepth", "fold-depth", "int64_t",
+          /*default=*/"0",
+          "Control the maximum depth of operands to include in the name."
+          "If less than 0, name folding is entirely disabled."
+          "If set to 0, only folds the op name itself without operands.">,
+  ];
 }
 
 #endif // MLIR_TRANSFORMS_PASSES
diff --git a/mlir/lib/Transforms/Normalize.cpp b/mlir/lib/Transforms/Normalize.cpp
index 6f42c61ee7d18..3309c4cbb2c3b 100644
--- a/mlir/lib/Transforms/Normalize.cpp
+++ b/mlir/lib/Transforms/Normalize.cpp
@@ -6,12 +6,33 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Block.h"
 #include "mlir/IR/Dominance.h"
+#include "mlir/IR/Location.h"
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/Operation.h"
+#include "mlir/IR/OperationSupport.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/Region.h"
+#include "mlir/IR/Value.h"
+#include "mlir/IR/Visitors.h"
+#include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Support/WalkResult.h"
+#include "mlir/Transforms/CommutativityUtils.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/Passes.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/Support/DebugLog.h"
+#include "llvm/Support/raw_ostream.h"
+#include <cstdint>
+#include <string>
 
 using namespace mlir;
 
@@ -24,27 +45,133 @@ namespace mlir {
 
 namespace {
 
-bool isOutput(Operation *op) {
+class Normalize {
+public:
+  Normalize(IRRewriter &rewriter, DominanceInfo &domInfo,
+            NormalizePassOptions &options)
+      : rewriter(rewriter), domInfo(domInfo), options(options) {}
+  /// Collect a vector of output ops within in \p root.
+  void collectOutputs(Operation *root);
+
+  /// Reorders ops by walking up the tree from each operand of an output op and
+  /// reducing the def-use distance. This method assumes that output ops were
+  /// collected top-down, otherwise the def-use chain may be broken. This method
+  /// is a wrapper for recursive reorderOutput().
+  void reorderOutputs();
+
+  /// Assigns unique, sequential names (e.g., "a0", "a1") to all block arguments
+  /// within \p root.
+  void nameBlockArguments(Operation *root);
+
+  /// Assigns unique, sequential names to all collected output operations.
+  void nameOperations();
+
+  /// Fold the operation name within \p root.
+  void foldOperationsName(Operation *root);
+
+  /// Greedily applies commutativity patterns using \p root to define the
+  /// transformation scope.
+  LogicalResult sortCommutativeOperands(Operation *root);
+
+private:
+  /// Reorders operations along the def-use chain from left to right, bottom to
+  /// top, starting from \p producer.
+  void reorderOutput(Operation *producer);
+
+  /// Returns true if the \p is a terminator or contains memory/side effects.
+  bool isOutput(Operation *op);
+
+  /// Returns true if the \p op is an initial operation (has no operands or only
+  /// constant-like operands).
+  bool isInitialOperation(Operation *op);
+
+  /// Assigns a unique name to \p op, using \p visited to track and skip already
+  /// processed operations.
+  void nameOperation(Operation *op, SmallPtrSet<Operation *, 32> &visited);
+
+  /// Generates and assigns a stable, deterministic name to the initial \p op,
+  /// while recursively resolving names for its upstream operands.
+  void nameAsInitialOperation(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited);
+  void nameAsRegularOperation(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited);
+
+  /// Computes the output footprint for the given \p op.
+  SetVector<int> getOutputFootprint(Operation *op,
+                                    SmallPtrSet<Operation *, 32> &visited);
+  /// Simplifies the name of the given \p op.
+  void foldOperationName(Operation *op);
+
+  Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops);
+
+  void appendCallAndOperandNames(Operation *op, SmallString<512> &name,
+                                 SmallVectorImpl<StringRef> &operandNames);
+
+  /// Collapses "$-"..."-$" nesting in \p name beyond \p depth, keeping only
+  /// the outer \p depth levels of markers.
+  std::string trimNameByDepth(StringRef name, int64_t depth);
+
+  IRRewriter &rewriter;
+  DominanceInfo &domInfo;
+  NormalizePassOptions &options;
+
+  /// Outputs collected by collectOutputs.
+  SmallVector<Operation *> outputs;
+
+  /// Caches, for output ops, their accumulated nested distance: the sum of
+  /// each enclosing region's op-position, walked upward until the parent
+  /// operation is a FunctionOpInterface.
+  DenseMap<Operation *, int64_t> footprintCache;
+
+  // Random constant for hashing, so the state isn't zero.
+  const uint64_t magicHashConstant = 0x6acaa36bef8325c5ULL;
+};
+
+// Frozen mixer; basic-block names derived from these hashes appear in
+// the normalized IR text and must be deterministic across processes
+// for the normalizer's "compare normalized IR" workflow to work.
+static constexpr uint64_t hash_16_bytes(uint64_t Low, uint64_t High) {
+  const uint64_t kMul = 0x9ddfea08eb382d69ULL;
+  uint64_t A = (Low ^ High) * kMul;
+  A ^= (A >> 47);
+  uint64_t B = (High ^ A) * kMul;
+  B ^= (B >> 47);
+  B *= kMul;
+  return B;
+}
+
+/// Computes the 64-bit FNV-1a hash value of the given string \p data.
+static constexpr uint64_t strHash(std::string_view data) noexcept {
+  const uint64_t fnvOffset = 0xcbf29ce484222325ULL;
+  const uint64_t fnvPrime = 0x100000001b3ULL;
+  uint64_t hash = fnvOffset;
+  for (const auto &c : data) {
+    hash ^= static_cast<uint64_t>(c);
+    hash *= fnvPrime;
+  }
+  return hash;
+}
+
+bool Normalize::isOutput(Operation *op) {
   if (!op)
     return false;
   return !isMemoryEffectFree(op) || op->hasTrait<OpTrait::IsTerminator>();
 }
 
-/// Returns a vector of output ops. An output is a op which
-/// has side-effects or is terminator.
-SmallVector<Operation *> collectOutputs(Operation *root) {
-  SmallVector<Operation *> outputs;
+void Normalize::collectOutputs(Operation *root) {
   root->walk([&](Operation *op) {
-    if (isOutput(op))
+    if (op == root)
+      return WalkResult::advance();
+    if (isOutput(op)) {
+      LDBG() << "insert " << OpWithFlags(op, OpPrintingFlags().skipRegions())
+             << " to outputs";
       outputs.push_back(op);
+    }
+    return WalkResult::advance();
   });
-  return outputs;
 }
 
-/// The function returns the operation that dominates all other operations in
-/// the given list.
-Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
-                         const DominanceInfo &domInfo) {
+Operation *Normalize::getDominateOp(const SmallVectorImpl<Operation *> &ops) {
   if (ops.empty())
     return {};
   Operation *curDomOp = ops.front();
@@ -69,38 +196,315 @@ Operation *getDominateOp(const SmallVectorImpl<Operation *> &ops,
   return curDomOp;
 }
 
-/// Move used to its nearest user and recursively perform the same process on
-/// the defining operations of its operands.
-void reorderOutput(IRRewriter &rewriter, Operation *producer,
-                   const DominanceInfo &domInfo) {
+void Normalize::reorderOutput(Operation *producer) {
   if (!isPure(producer))
     return;
   SmallVector<Operation *> users(producer->getUsers());
-  if (Operation *domOp = getDominateOp(users, domInfo)) {
+  if (Operation *domOp = getDominateOp(users)) {
     rewriter.moveOpBefore(producer, domOp);
     for (Value operand : producer->getOperands())
       if (Operation *defineOp = operand.getDefiningOp())
-        reorderOutput(rewriter, defineOp, domInfo);
+        reorderOutput(defineOp);
   }
 }
 
-/// Reorders ops by walking up the tree from each operand of an output op and
-/// reducing the def-use distance. This method assumes that output ops were
-/// collected top-down, otherwise the def-use chain may be broken. This method
-/// is a wrapper for recursive reorderOutput().
-void reorderOutputs(IRRewriter &rewriter,
-                    const SmallVectorImpl<Operation *> &outputs,
-                    const DominanceInfo &domInfo) {
+void Normalize::reorderOutputs() {
   SmallPtrSet<Operation *, 16> visited;
   for (Operation *output : outputs) {
     for (Value operand : output->getOperands()) {
       if (Operation *defineOp = operand.getDefiningOp();
           defineOp && !visited.contains(defineOp)) {
         visited.insert(defineOp);
-        reorderOutput(rewriter, defineOp, domInfo);
+        reorderOutput(defineOp);
+      }
+    }
+  }
+}
+
+bool Normalize::isInitialOperation(Operation *op) {
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp();
+        !define || !define->hasTrait<OpTrait::ConstantLike>())
+      return false;
+  }
+  return true;
+}
+
+/// Computes the output footprint for the given \p op.
+///
+/// Traverses downstream users recursively to find all reachable output
+/// operations. For each output operation, it calculates its precise distance
+/// (in terms of operation count) relative to the entry block of its enclosing
+/// `FunctionOpInterface`.
+SetVector<int>
+Normalize::getOutputFootprint(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited) {
+  SetVector<int> outputs;
+  if (visited.count(op))
+    return outputs;
+  visited.insert(op);
+
+  // If the operation is an output, compute its nested absolute distance to the
+  // Function entry.
+  if (isOutput(op)) {
+    if (footprintCache.contains(op)) {
+      outputs.insert(footprintCache[op]);
+      return outputs;
+    }
+
+    // Calculates the total distance of 'op' to its enclosing parent region's
+    // start, accumulating nested offsets upward until the parent operation
+    // matches `FunctionOpInterface`.
+    int count = 0;
+    Operation *curOp = op;
+    do {
+      Region *parentRegion = curOp->getParentRegion();
+      int distance = 0;
+      for (Operation &it : parentRegion->getOps()) {
+        if (&it == curOp)
+          break;
+        ++distance;
       }
+      count += distance;
+      curOp = parentRegion->getParentOp();
+    } while (!isa<FunctionOpInterface>(curOp));
+    outputs.insert(count);
+    footprintCache[op] = count;
+    return outputs;
+  }
+
+  // Otherwise, recursively aggregate footprints from all downstream users.
+  for (Operation *user : op->getUsers()) {
+    SetVector<int> outputsUser = getOutputFootprint(user, visited);
+    outputs.insert(outputsUser.begin(), outputsUser.end());
+  }
+  return outputs;
+}
+
+void Normalize::appendCallAndOperandNames(
+    Operation *op, SmallString<512> &name,
+    SmallVectorImpl<StringRef> &operandNames) {
+  // In case of CallInst, consider callee in the operation name.
+  if (auto callOp = dyn_cast<CallOpInterface>(op))
+    if (auto funcOp = dyn_cast<FunctionOpInterface>(callOp.resolveCallable()))
+      name.append(funcOp.getNameAttr());
+
+  if (operandNames.size() > 0) {
+    name.append("$-");
+    for (size_t i = 0, e = operandNames.size(); i < e; ++i) {
+      name.append(operandNames[i]);
+      if (i < e - 1)
+        name.append(".");
     }
+    name.append("-$");
   }
+  NameLoc loc = NameLoc::get(StringAttr::get(op->getContext(), name));
+  LDBG() << "set NameLoc: " << loc
+         << "\nfor: " << OpWithFlags(op, OpPrintingFlags().skipRegions());
+  op->setLoc(loc);
+}
+
+/// Names operation following the scheme:
+/// vl00000Callee(Operands)
+///
+/// Where 00000 is a hash calculated considering operation's opcode, output
+/// footprint and block position. Callee's name is only included when
+/// operation's type is `CallOpInterface`. The Operands are derived from the
+/// names of the operation's operands.
+void Normalize::nameAsInitialOperation(Operation *op,
+                                       SmallPtrSet<Operation *, 32> &visited) {
+  // Recursively name defining ops of operands and collect their names.
+  SmallVector<StringRef, 4> operandNames;
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp())
+      nameOperation(define, visited);
+    if (NameLoc loc = dyn_cast<NameLoc>(operand.getLoc()))
+      operandNames.push_back(loc.getName());
+  }
+
+  // Early exit if the op don't have results.
+  if (!op->getNumResults())
+    return;
+
+  // Initialize to a magic constant, so the state isn't zero.
+  uint64_t hash = magicHashConstant;
+
+  // Consider operation's opcode in the hash.
+  hash = hash_16_bytes(hash, strHash(op->getName().getStringRef().str()));
+
+  // Get output footprint for \p op.
+  SmallPtrSet<Operation *, 32> visitedOutputFoot;
+  SetVector<int> outputFootprint = getOutputFootprint(op, visitedOutputFoot);
+
+  // Consider output footprint in the hash.
+  for (const int &output : outputFootprint)
+    hash = hash_16_bytes(hash, output);
+
+  // Include the operation's relative position within its basic block.
+  hash = hash_16_bytes(
+      hash, std::distance(op->getBlock()->begin(), op->getIterator()));
+
+  // Base operation name.
+  SmallString<512> name;
+  name.append("vl" + std::to_string(hash).substr(0, 5));
+
+  // Append call and operand name.
+  appendCallAndOperandNames(op, name, operandNames);
+}
+
+/// Names operation following the scheme:
+/// op00000Callee(Operands)
+///
+/// Where 00000 is a hash calculated considering operation's opcode, its
+/// operands' opcodes, and block position. Callee's name is only included
+/// when operation's type is `CallOpInterface`, The Operands are derived from
+/// the names of the operation's operands.
+void Normalize::nameAsRegularOperation(Operation *op,
+                                       SmallPtrSet<Operation *, 32> &visited) {
+  // Recursively name defining ops of operands and collect their names.
+  SmallVector<StringRef, 2> operandNames;
+  for (Value operand : op->getOperands()) {
+    if (Operation *define = operand.getDefiningOp())
+      nameOperation(define, visited);
+    if (NameLoc loc = dyn_cast<NameLoc>(operand.getLoc())) {
+      operandNames.push_back(loc.getName());
+    }
+  }
+
+  // Early exit if the op don't have results.
+  if (!op->getNumResults())
+    return;
+
+  // Initialize to a magic constant, so the state isn't zero.
+  uint64_t hash = magicHashConstant;
+
+  // Consider operation opcode in the hash.
+  uint64_t ophash = strHash(op->getName().getStringRef().str());
+  hash = hash_16_bytes(hash, ophash);
+
+  // Fuses the opcodes of upstream defining ops into the hash.
+  for (Value operand : op->getOperands())
+    if (Operation *define = operand.getDefiningOp())
+      hash =
+          hash_16_bytes(hash, strHash(define->getName().getStringRef().str()));
+
+  // Include the operation's relative position within its basic block.
+  hash = hash_16_bytes(
+      hash, std::distance(op->getBlock()->begin(), op->getIterator()));
+
+  // Base operation name.
+  SmallString<512> name;
+  name.append("op" + std::to_string(hash).substr(0, 5));
+
+  // Append call and operand name.
+  appendCallAndOperandNames(op, name, operandNames);
+}
+
+void Normalize::nameOperation(Operation *op,
+                              SmallPtrSet<Operation *, 32> &visited) {
+  if (visited.count(op))
+    return;
+  visited.insert(op);
+
+  // Determine the type of operation to name.
+  if (isInitialOperation(op)) {
+    // This is an initial operation.
+    nameAsInitialOperation(op, visited);
+  } else {
+    // This must be a regular operation.
+    nameAsRegularOperation(op, visited);
+  }
+}
+
+void Normalize::nameOperations() {
+  SmallPtrSet<Operation *, 32> visited;
+  for (Operation *op : outputs)
+    nameOperation(op, visited);
+}
+
+void Normalize::nameBlockArguments(Operation *root) {
+  size_t argumentCount = 0;
+  MLIRContext *context = root->getContext();
+  root->walk<WalkOrder::PreOrder>([&](Block *b) {
+    for (auto argument : b->getArguments()) {
+      NameLoc loc =
+          NameLoc::get(StringAttr::get(context, "a" + Twine(argumentCount++)));
+      argument.setLoc(loc);
+    }
+    return;
+  });
+}
+
+/// Collapses nested "$-"..."-$" segments in `name` beyond the given
+/// `depth`, keeping only the outer `depth` levels of markers.
+///
+/// "$-" / "-$" mark the start/end of a nested scope, e.g.
+/// op80011$-a0.op11483-$ has one level of nesting.
+/// depth == 0 strips it down to "op80011" (no markers left);
+/// depth < 0, or fewer than depth+1 marker pairs, returns `name` unchanged.
+std::string Normalize::trimNameByDepth(StringRef name, int64_t depth) {
+  std::string str = name.str();
+  if (depth < 0)
+    return str;
+
+  int targetCount = depth + 1;
+
+  size_t startPos = 0;
+  for (int i = 0, e = targetCount; i < e; ++i) {
+    startPos = str.find("$-", startPos);
+    if (startPos == std::string::npos)
+      return str;
+    startPos += 2;
+  }
+
+  size_t endPos = str.size() - 1;
+  for (int i = 0, e = targetCount; i < e; ++i) {
+    endPos = str.rfind("-$", endPos);
+    if (endPos == std::string::npos)
+      return str;
+    endPos -= 2;
+  }
+
+  startPos -= 2;
+  endPos += 2;
+
+  if (startPos >= endPos)
+    return str;
+
+  return str.substr(0, startPos) + str.substr(endPos + 2);
+}
+
+/// Folds the name of \p op into a simplified form containing a truncated
+/// prefix of its own name and its operands' names,
+void Normalize::foldOperationName(Operation *op) {
+  NameLoc loc = dyn_cast<NameLoc>(op->getLoc());
+
+  // Only process operations prefixed with "op" since their names are complex
+  // and need simplification.
+  if (!loc || loc.getName().empty() || loc.getName().str().substr(0, 2) != "op")
+    return;
+
+  std::string name = trimNameByDepth(loc.getName().str(), options.foldDepth);
+  loc = NameLoc::get(StringAttr::get(op->getContext(), name));
+  LDBG() << "fold NameLoc: " << loc
+         << "\nfor: " << OpWithFlags(op, OpPrintingFlags().skipRegions());
+  op->setLoc(loc);
+}
+
+void Normalize::foldOperationsName(Operation *root) {
+  if (options.foldDepth < 0)
+    return;
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *op) { foldOperationName(op); });
+}
+
+LogicalResult Normalize::sortCommutativeOperands(Operation *root) {
+  MLIRContext *context = root->getContext();
+  RewritePatternSet patterns(context);
+  populateCommutativityUtilsPatterns(patterns);
+  if (failed(applyPatternsGreedily(root, std::move(patterns))))
+    return failure();
+  return success();
 }
 
 struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
@@ -112,9 +516,20 @@ struct NormalizePass : public impl::NormalizePassBase<NormalizePass> {
 void NormalizePass::runOnOperation() {
   DominanceInfo &domInfo = getAnalysis<DominanceInfo>();
   IRRewriter rewriter(&getContext());
-  SmallVector<Operation *> outputs = collectOutputs(getOperation());
-  reorderOutputs(rewriter, outputs, domInfo);
-  // Since we only changed the positions of the operations, DominanceInfo and
-  // PostDominanceInfo are marked as preserved.
+  NormalizePassOptions options = {foldDepth};
+  Normalize normalize(rewriter, domInfo, options);
+
+  // Sort commutative operands up front, so operand order doesn't need to be
+  // re-sorted (by name) later when renaming ops.
+  if (failed(normalize.sortCommutativeOperands(getOperation())))
+    signalPassFailure();
+  normalize.collectOutputs(getOperation());
+  normalize.reorderOutputs();
+  normalize.nameBlockArguments(getOperation());
+  normalize.nameOperations();
+  normalize.foldOperationsName(getOperation());
+
+  // Since we only changed the positions of the operations, `DominanceInfo` and
+  // `PostDominanceInfo` are marked as preserved.
   markAnalysesPreserved<DominanceInfo, PostDominanceInfo>();
 }
diff --git a/mlir/test/Transforms/normalize.mlir b/mlir/test/Transforms/normalize.mlir
index 28f0c05263278..4b11c1f7427c5 100644
--- a/mlir/test/Transforms/normalize.mlir
+++ b/mlir/test/Transforms/normalize.mlir
@@ -1,4 +1,9 @@
 // RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize))" -split-input-file | FileCheck %s
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize{fold-depth=1}))" -mlir-use-nameloc-as-prefix -split-input-file | FileCheck %s -check-prefix=CHECK-NAMELOC
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize{fold-depth=0}))" -mlir-use-nameloc-as-prefix -split-input-file | FileCheck %s -check-prefix=CHECK-NAMELOC-DEPTH-0
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(normalize{fold-depth=2}))" -mlir-use-nameloc-as-prefix -split-input-file | FileCheck %s -check-prefix=CHECK-NAMELOC-DEPTH-2
+
+// This test verifies op ordering and the sorting of commutative operands.
 
 // CHECK-LABEL: func @multiple_memref_store
 //  CHECK-SAME:   %[[ARG0:.*]]: index,
@@ -7,35 +12,53 @@ func.func @multiple_memref_store(%arg0: index, %arg1 : memref<?xf32>) {
   %f0 = arith.constant 0.0 : f32
   %f1 = arith.constant 1.0 : f32
   %add = arith.addi %arg0, %arg0 : index
-  %sub = arith.subi %arg0, %arg0 : index
+  %sub = arith.addi %add, %arg0 : index
   memref.store %f0, %arg1[%add] : memref<?xf32>
   memref.store %f1, %arg1[%sub] : memref<?xf32>
   return
 }
 
-// CHECK-NEXT: %[[C0:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
-// CHECK-NEXT: memref.store %[[C0]], %[[ARG1]]{{\[}}%[[ADD]]] : memref<?xf32>
-// CHECK-NEXT: %[[C1:.*]] = arith.constant 1.000000e+00 : f32
-// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
-// CHECK-NEXT: memref.store %[[C1]], %[[ARG1]]{{\[}}%[[SUB]]] : memref<?xf32>
+// CHECK-NEXT: %[[C_0:.*]] = arith.constant 0.000000e+00 : f32
+// CHECK-NEXT: %[[ADD_0:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: memref.store %[[C_0]], %[[ARG1]]{{\[}}%[[ADD_0]]] : memref<?xf32>
+// CHECK-NEXT: %[[CONSTANT_1:.*]] = arith.constant 1.000000e+00 : f32
+// CHECK-NEXT: %[[ADD_1:.*]] = arith.addi %[[ARG0]], %[[ADD_0]] : index
+// CHECK-NEXT: memref.store %[[CONSTANT_1]], %[[ARG1]]{{\[}}%[[ADD_1]]] : memref<?xf32>
+
+// CHECK-NAMELOC-LABEL: func @multiple_memref_store
+//  CHECK-NAMELOC-NEXT:   %vl48293 = arith.constant 0.000000e+00 : f32
+//  CHECK-NAMELOC-NEXT:   %op13600$-a0.a0-$ = arith.addi %a0, %a0 : index
+//  CHECK-NAMELOC-NEXT:   memref.store %vl48293, %a1[%op13600$-a0.a0-$] : memref<?xf32>
+//  CHECK-NAMELOC-NEXT:   %vl15553 = arith.constant 1.000000e+00 : f32
+//  CHECK-NAMELOC-NEXT:   %op69768$-a0.op13600-$ = arith.addi %a0, %op13600$-a0.a0-$ : index
+//  CHECK-NAMELOC-NEXT:   memref.store %vl15553, %a1[%op69768$-a0.op13600-$] : memref<?xf32>
 
 // -----
 
+// This test verifies an 'output' op with multiple operands that are all results of another op.
+
 // CHECK-LABEL: func @return_multiple_operands
-//  CHECK-SAME:   %[[ARG0:.*]]: index
-func.func @return_multiple_operands (%arg0: index) -> (index, index) {
-  %0 = arith.addi %arg0, %arg0 : index
-  %1 = arith.subi %arg0, %arg0 : index
+//  CHECK-SAME:   %[[ARG0:.*]]: index,
+//  CHECK-SAME:   %[[ARG1:.*]]: index
+func.func @return_multiple_operands (%arg0: index, %arg1: index) -> (index, index) {
+  %0 = arith.addi %arg0, %arg1 : index
+  %1 = arith.subi %arg0, %arg1 : index
   return %1, %0 : index, index
 }
 
-// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG0]] : index
-// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG0]] : index
+// CHECK-NEXT: %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ARG1]] : index
+// CHECK-NEXT: %[[ADD:.*]] = arith.addi %[[ARG0]], %[[ARG1]] : index
 // CHECK-NEXT: return %[[SUB]], %[[ADD]] : index, index
 
+// CHECK-NAMELOC-LABEL: func @return_multiple_operands
+//  CHECK-NAMELOC-NEXT:   %op89776$-a0.a1-$ = arith.subi %a0, %a1 : index
+//  CHECK-NAMELOC-NEXT:   %op13600$-a0.a1-$ = arith.addi %a0, %a1 : index
+//  CHECK-NAMELOC-NEXT:   return %op89776$-a0.a1-$, %op13600$-a0.a1-$ : index, index
+
 // -----
 
+// This test checks if '%add' is scheduled down to the second 'memref.store' site.
+
 // CHECK-LABEL: func @cross_region
 //  CHECK-SAME:   %[[ARG0:.*]]: f32,
 //  CHECK-SAME:   %[[ARG1:.*]]: memref<10xf32>
@@ -56,6 +79,12 @@ func.func @cross_region(%arg0: f32, %arg1 : memref<10xf32>) {
 // CHECK-NEXT:   memref.store %[[LOG]], %[[ARG1]]{{\[}}%[[IV]]] : memref<10xf32>
 // CHECK-NEXT: }
 
+// CHECK-NAMELOC-LABEL: func @cross_region
+//       CHECK-NAMELOC:   affine.for %a3 = 6 to 10 {
+//  CHECK-NAMELOC-NEXT:     %op16592$-op10970-$ = math.log2 %op10970$-a0.a0-$ : f32
+//  CHECK-NAMELOC-NEXT:     memref.store %op16592$-op10970-$, %a1[%a3] : memref<10xf32>
+//  CHECK-NAMELOC-NEXT:   }
+
 // -----
 
 // This test verifies the reordering of scf.for ops.
@@ -83,3 +112,65 @@ func.func @side_effect_loop_op(%arg1 : memref<?xf32>) {
 // CHECK-NEXT:   %[[F1:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-NEXT:   memref.store %[[F1]], %[[ARG0]]{{\[}}%[[IV]]] : memref<?xf32>
 // CHECK-NEXT: }
+
+// CHECK-NAMELOC-LABEL: func @side_effect_loop_op
+//  CHECK-NAMELOC-NEXT:   %vl15499 = arith.constant 0 : index
+//  CHECK-NAMELOC-NEXT:   %op18509$-a0.vl15499-$ = memref.dim %a0, %vl15499 : memref<?xf32>
+//  CHECK-NAMELOC-NEXT:   %vl14483 = arith.constant 1 : index
+//  CHECK-NAMELOC-NEXT:   scf.for %a1 = %vl15499 to %op18509$-a0.vl15499-$ step %vl14483 {
+//  CHECK-NAMELOC-NEXT:     %vl71256 = arith.constant 1.000000e+00 : f32
+//  CHECK-NAMELOC-NEXT:     memref.store %vl71256, %a0[%a1] : memref<?xf32>
+//  CHECK-NAMELOC-NEXT:   }
+
+// -----
+
+// This test verifies the naming of func.call operations.
+
+func.func private @callee_0(%arg0: f32) -> f32
+func.func private @callee_1(%arg0: f32) -> f32
+
+func.func @test_call_operation(%arg0: f32) -> f32 {
+  %0 = func.call @callee_0(%arg0) : (f32) -> f32
+  %1 = func.call @callee_1(%0) : (f32) -> f32
+  return %1 : f32
+}
+
+// CHECK-NAMELOC-LABEL: func @test_call_operation
+//  CHECK-NAMELOC-NEXT:   %op71372callee_0$-a0-$ = call @callee_0(%a0) : (f32) -> f32
+//  CHECK-NAMELOC-NEXT:   %op15508callee_1$-op71372callee_0-$ = call @callee_1(%op71372callee_0$-a0-$) : (f32) -> f32
+//  CHECK-NAMELOC-NEXT:   return %op15508callee_1$-op71372callee_0-$ : f32
+
+// -----
+
+func.func @deep_use_chain(%arg0: i32, %arg1: i32) -> i32 {
+  %0 = arith.addi %arg0, %arg1 : i32
+  %1 = arith.muli %arg0, %0 : i32
+  %2 = arith.addi %arg0, %1 : i32
+  %3 = arith.muli %2, %2 : i32
+  %4 = arith.addi %arg1, %3 : i32
+  return %4 : i32
+}
+
+// CHECK-NAMELOC-DEPTH-0-LABEL: func @deep_use_chain
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   %op11483 = arith.addi %a0, %a1 : i32
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   %op80011 = arith.muli %a0, %op11483 : i32
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   %op14133 = arith.addi %a0, %op80011 : i32
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   %op27579 = arith.muli %op14133, %op14133 : i32
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   %op17770 = arith.addi %a1, %op27579 : i32
+//  CHECK-NAMELOC-DEPTH-0-NEXT:   return %op17770 : i32
+
+// CHECK-NAMELOC-LABEL: func @deep_use_chain
+//  CHECK-NAMELOC-NEXT:   %op11483$-a0.a1-$ = arith.addi %a0, %a1 : i32
+//  CHECK-NAMELOC-NEXT:   %op80011$-a0.op11483-$ = arith.muli %a0, %op11483$-a0.a1-$ : i32
+//  CHECK-NAMELOC-NEXT:   %op14133$-a0.op80011-$ = arith.addi %a0, %op80011$-a0.op11483-$ : i32
+//  CHECK-NAMELOC-NEXT:   %op27579$-op14133-$ = arith.muli %op14133$-a0.op80011-$, %op14133$-a0.op80011-$ : i32
+//  CHECK-NAMELOC-NEXT:   %op17770$-a1.op27579-$ = arith.addi %a1, %op27579$-op14133-$ : i32
+//  CHECK-NAMELOC-NEXT:   return %op17770$-a1.op27579-$ : i32
+
+// CHECK-NAMELOC-DEPTH-2-LABEL: func @deep_use_chain
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   %op11483$-a0.a1-$ = arith.addi %a0, %a1 : i32
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   %op80011$-a0.op11483$-a0.a1-$-$ = arith.muli %a0, %op11483$-a0.a1-$ : i32
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   %op14133$-a0.op80011$-a0.op11483-$-$ = arith.addi %a0, %op80011$-a0.op11483$-a0.a1-$-$ : i32
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   %op27579$-op14133$-a0.op80011-$-$ = arith.muli %op14133$-a0.op80011$-a0.op11483-$-$, %op14133$-a0.op80011$-a0.op11483-$-$ : i32
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   %op17770$-a1.op27579$-op14133-$-$ = arith.addi %a1, %op27579$-op14133$-a0.op80011-$-$ : i32
+//  CHECK-NAMELOC-DEPTH-2-NEXT:   return %op17770$-a1.op27579$-op14133-$-$ : i32


From llvmlistbot at llvm.org  Tue Jul  7 08:20:34 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Tue, 07 Jul 2026 08:20:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add normalize pass to MLIR (PR #186647)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/186647@github.com>
Message-ID: <6a4d1942.30759430.24d50c.ecfd@mx.google.com>


https://github.com/linuxlonelyeagle ready_for_review https://github.com/llvm/llvm-project/pull/186647

From llvmlistbot at llvm.org  Tue Jul  7 08:27:14 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 07 Jul 2026 08:27:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4d1ad2.e55dc341.1f19bd.ae0a@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/13] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/13] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/13] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/13] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/13] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/13] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/13] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/13] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/13] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/13] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/13] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 53a57bba5c89d408c64fec9187c8c81b7593ca1c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/13] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 5048a6507c08e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f3c47aeab7f58..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst

>From e89c1d53a2027757778ea25c7e0db3cf1197381d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 7 Jul 2026 17:26:24 +0200
Subject: [PATCH 13/13] Constrain RC to either scalar OR single non-unit dim
 collapsed result

---
 .../Transforms/ElideReinterpretCast.cpp       | 179 +++++++------
 .../MemRef/elide-reinterpret-cast.mlir        | 242 ++++++++++++------
 2 files changed, 268 insertions(+), 153 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 38c1b7d9a8d35..e3a2bb6457b69 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,35 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
+static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
+                                             MemRefType resType) {
+  assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
+         "expected static shapes");
+  assert(srcType.getRank() == resType.getRank() &&
+         "expected rank-preserving reinterpret_cast");
+
+  unsigned collapsedDims = 0;
+
+  for (auto [srcSize, resSize] :
+       llvm::zip_equal(srcType.getShape(), resType.getShape())) {
+    if (srcSize == resSize)
+      continue;
+
+    // Only allow collapsing one non-unit source dim to a unit result dim.
+    if (srcSize != 1 && resSize == 1) {
+      ++collapsedDims;
+      continue;
+    }
+
+    // The sizes differ and both of them are non-unit - ATM not supported.
+    return false;
+  }
+
+  // Make sure there is only one collapsed dimension.
+  return collapsedDims == 1;
+}
+
+/// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
@@ -105,27 +133,27 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// the corresponding source indices.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Non-scalar results must have static strides
-/// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources. Returns nullopt for unsupported
+/// identity-layout sources. Scalar-shaped results may have arbitrary result
+/// strides. Non-scalar results must have static offsets, static result strides
+/// identical to the source identity strides, and exactly one non-unit source
+/// dimension collapsed to unit size. Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[?, M, 1]>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[?, N, 1]>>
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[K, M, L]>>
-///       ( identity-layout != [K, M, L] )
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -136,7 +164,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts
+  // TODO: Support rank-modifying reinterpret_casts.
   if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
@@ -151,8 +179,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  assert(resType.hasStaticShape() && "expected static shape");
-
   // Track result dimensions that produce varying indices; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -160,12 +186,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
       dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
   }
 
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For non-scalar results, verify that strides do not introduce
-  // non-contiguity that would require extra logic.
   if (!isScalarRes) {
+    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
+    // views with static metadata.
+    if (ShapedType::isDynamic(rcOffsets[0]))
+      return std::nullopt;
+
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -177,53 +211,32 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-  }
 
-  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
-
-  // A source with exactly one non-unit dimension cannot be indexed directly by
-  // multiple non-unit result dimensions.
-  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
-    return std::nullopt;
-
-  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
-  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
-  // only a single offset. That should be fixed at the op definition level.
-  assert(rcOffsets.size() == 1 && "Expecting single offset");
+    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+      return std::nullopt;
+  }
 
-  // CASE 1: Static ReinterpretCast offset
-  if (ShapedType::isStatic(rcOffsets[0])) {
-    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every source dimension).
-    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(dimsAndOffs.delinearizedOffsets &&
-           "static reinterpret_cast offset must delinearize to in-bounds "
-           "reinterpret_cast source indices");
+  // CASE 1: Dynamic ReinterpretCast offset.
+  //
+  // Dynamic offsets are supported only for scalar-shaped results, under the
+  // previous effectively-1D source restriction.
+  if (ShapedType::isDynamic(rcOffsets[0])) {
+    // With an effectively-1D source, a dynamic offset can be mapped to its
+    // unique non-unit dim. For other cases, bail out.
+    if (llvm::count_if(srcType.getShape(),
+                       [](int64_t size) { return size != 1; }) != 1)
+      return std::nullopt;
 
-    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
-    // MemRef. Such cases should probably be rejected by Op verifier.
-    // FIXME: Add run-time verification for cases like this.
-    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
-                        [&](auto it) {
-                          unsigned dim = it.index();
-                          int64_t resultSize = it.value();
-                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
-                                     resultSize <=
-                                 srcType.getDimSize(dim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
     return dimsAndOffs;
   }
 
-  // CASE 2: Dynamic ReinterpretCast offset.
-  // TODO: Support dynamic offsets into sources with multiple non-unit
-  // dimensions by delinearizing the offset into source start indices at runtime
-  // before adding loop IVs.
-
-  // With an effectively-1D source, a dynamic offset can be mapped to the unique
-  // non-unit dim. For other cases, bail out.
-  if (!srcNonUnitDim)
-    return std::nullopt;
+  // CASE 2: Static ReinterpretCast offset
+  // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+  // every source dimension).
+  dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+  assert(dimsAndOffs.delinearizedOffsets &&
+         "static reinterpret_cast offset must delinearize to in-bounds "
+         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -231,37 +244,36 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
 /// scalar load/store operations.
 ///
-///   // BEFORE (scalar copy)
-///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %strided
+/// Supported cases:
+///   1. Scalar-shaped reinterpret_cast results. Result strides are ignored;
+///      the store index is derived from the reinterpret_cast offset.
 ///
-///   // AFTER
-///   %v = memref.load %src[0, ..., 0]
-///   memref.store %v, %dst[delinearized(O)]
+///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
+///      dimensions except one collapsed-to-unit dimension. Result strides must
+///      be static and identical to the identity strides of the source, and the
+///      static offset selects the collapsed dimension.
 ///
-///   // BEFORE (effectively-1D copy)
+///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, 1], strides: [...]
+///     to offset: [OFF], sizes: [1, ..., 1], strides: [...]
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
-///   }
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (multidimensional copy with static offset)
+///   // BEFORE (one collapsed non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, K], strides: [...]
+///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     : memref<1xMxNxf32>
+///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     scf.for %j = 0 to K step 1 {
-///       %v = memref.load %src[0, %i, %j]
-///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
-///     }
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   scf.for %i = 0 to M step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[0, %i, OFF]
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
@@ -292,6 +304,19 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType dstType = cast<MemRefType>(dst.getType());
     MemRefType rcResType = cast<MemRefType>(rc.getType());
 
+    // Sanity check that the copy doesn't access strided MemRef out-of-bounds.
+    // Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
+    if (ShapedType::isStatic(rc.getStaticOffsets()[0]) &&
+        llvm::any_of(llvm::enumerate(rcResType.getShape()), [&](auto it) {
+          unsigned dim = it.index();
+          int64_t rcResultSize = it.value();
+          return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
+                 dstType.getDimSize(dim);
+        }))
+      return rewriter.notifyMatchFailure(
+          op, "copy accesses invalid accessible region");
+
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
     std::array<Value, 2> cachedIndexConstants;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index a1181b3d54a29..afa6833598c02 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -179,8 +179,59 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
+/// Offset delinearized to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [10], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[OFF]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// Offset delinearized into more dimensions: [0, 2, 1]
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [23], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C2:.*]] = arith.constant 2 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C2]], %[[C1]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
 // CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
   %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
@@ -202,6 +253,23 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
+func.func private @negative_copy_1D_into_1D_strided(
+  %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -228,12 +296,38 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+  %src : memref<1x1x11xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 11], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 11 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x1x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x1x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Offset delinearized to [0, 0, 10], therefore is only
 /// added to the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -257,36 +351,25 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   return
 }
 
-/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
-/// both the looped source dimension and the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
-    : memref<1x4x11xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
-  // CHECK:      }
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
-  // CHECK-NOT:  memref.copy
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
-/// Reject rc result strides that not equal to rc source identity strides.
+/// Reject rc result strides that are not equal to rc source identity strides.
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
 func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
@@ -325,91 +408,98 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+  return
+}
+
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
+func.func private @negative_copy_2D_into_2D_strided(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
-  // CHECK:        }
-  // CHECK:      }
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_zero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    to offset: [0], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
   // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
   // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-/// rc result dynamic offset:
-///    supported only for effectively-1D rc source
-///    (runtime delinearization not implemented)
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_nonzero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [10], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[OFF]]] : memref<3x1x4x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }


From llvmlistbot at llvm.org  Tue Jul  7 08:37:02 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Tue, 07 Jul 2026 08:37:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4d1d1e.eb165e5c.1817a5.1f42@mx.google.com>


================
@@ -0,0 +1,109 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect intervals that are still live at this allocation's start time.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    // Arena high-water mark from all placements so far.
+    int64_t arenaEnd = 0;
+    for (const auto &p : placements)
+      arenaEnd = std::max(arenaEnd, p.offset + p.size);
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      int64_t gapEnd = occStart;
+      if (alignedStart + alloc.sizeInBytes <= gapEnd) {
+        int64_t gapSize = gapEnd - alignedStart;
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied and arena end).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
----------------
javedabsar1 wrote:

explained.

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Tue Jul  7 08:45:09 2026
From: llvmlistbot at llvm.org (Yuvaraj Venkatesh)
Date: Tue, 07 Jul 2026 08:45:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4d1f05.a5745248.3db4b6.0163@mx.google.com>


================
@@ -1938,3 +1938,75 @@ func.func @test_assert_equal_shape() {
   tosa.assert_equal_shape %0, %1 {allow_broadcast = true} : (!tosa.shape<2>, !tosa.shape<2>) -> ()
   return
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const
+func.func @test_block_scaled_const() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 1.0}>> : [[0.0 : f8E4M3FN, 1.0 : f8E4M3FN, 0.001953125 : f8E4M3FN, 0.0078125 : f8E4M3FN, 
+                                                                                                                          2.0 : f8E4M3FN, 2.25 : f8E4M3FN, 2.5 : f8E4M3FN, 2.75 : f8E4M3FN, 
+                                                                                                                          15.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,  
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN], 
+                                                                                                                         [0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN]]>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat
+func.func @test_block_scaled_const_splat() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0, 1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat_mxint8
+func.func @test_block_scaled_const_splat_mxint8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 1.0}>> : 0 : i8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
----------------
Yuvaraj-Venkatesh wrote:

Shouldn't the type be !tosa.mxint8 here?

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 08:46:25 2026
From: llvmlistbot at llvm.org (Yuvaraj Venkatesh)
Date: Tue, 07 Jul 2026 08:46:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4d1f51.70920dee.290dca.eac1@mx.google.com>


https://github.com/Yuvaraj-Venkatesh commented:

One minor doubt. Other than that LGTM

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 08:46:45 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Tue, 07 Jul 2026 08:46:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4d1f65.01f0a1da.1e9662.032a@mx.google.com>


https://github.com/javedabsar1 updated https://github.com/llvm/llvm-project/pull/207403

>From 63bad1fb8d1de9fd84c3adc391d0eb4b9f56427f Mon Sep 17 00:00:00 2001
From: mabsar <mabsar at qti.qualcommm.com>
Date: Fri, 3 Jul 2026 07:20:15 -0700
Subject: [PATCH] [mlir][bufferization] Add best-fit algorithm to static memory
 planner

Introduces an algorithm selection option to the static memory planner pass
and add a best-fit algorithm that reuses memory from expired allocations
by finding the smallest suitable gap. The planning algorithms are factored
into a separate StaticMemoryPlanning.{h,cpp} to keep them independent of
MLIR IR and easily testable.

Signed-off-by: mabsar <mabsar at qti.qualcommm.com>
---
 .../Bufferization/IR/BufferizationEnums.td    |   8 +
 .../Bufferization/Transforms/Passes.td        |  31 +-
 .../Transforms/StaticMemoryPlanning.h         |  50 +++
 .../Bufferization/Transforms/CMakeLists.txt   |   1 +
 .../StaticMemoryPlannerAnalysis.cpp           | 344 +++++++++---------
 .../Transforms/StaticMemoryPlanning.cpp       | 131 +++++++
 .../static-memory-planner-best-fit.mlir       | 118 ++++++
 7 files changed, 504 insertions(+), 179 deletions(-)
 create mode 100644 mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
 create mode 100644 mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c2ac40a8427e7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,36 +68,134 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
+/// Build lifetime-annotated allocation descriptors from candidates.
+/// Returns the arena alignment (LCM of all individual alignments).
+static int64_t buildAllocInfos(
+    MutableArrayRef<AllocationCandidate> candidates,
+    SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  int64_t arenaAlignment = 1;
+  for (auto &candidate : candidates) {
+    bufferization::MemoryPlannerAlloc info;
+    info.sizeInBytes = candidate.sizeInBytes;
+    info.alignment = candidate.alignment;
+
+    Block *block = candidate.alloc->getBlock();
+    int64_t opIdx = 0;
+    for (Operation &op : *block) {
+      if (&op == candidate.alloc.getOperation())
+        info.timeStart = opIdx;
+      if (&op == candidate.dealloc.getOperation())
+        info.timeEnd = opIdx;
+      ++opIdx;
+    }
+
+    allocInfos.push_back(info);
+    arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
+  }
+  return arenaAlignment;
 }
 
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
+/// Collect alloc/dealloc pairs eligible for arena placement.
+/// An allocation is eligible if it has a static shape and a unique dealloc
+/// in the same block.
+static SmallVector<AllocationCandidate>
+collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
+                  llvm::Statistic &numSkipNoDealloc,
+                  llvm::Statistic &numEligible) {
+  SmallVector<AllocationCandidate> candidates;
+
+  funcOp->walk([&](memref::AllocOp allocOp) {
+    MemRefType memrefType = allocOp.getType();
+
+    // Skip dynamic shapes
+    if (!memrefType.hasStaticShape()) {
+      ++numSkipDynamic;
+      return;
+    }
 
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
+    // Find unique dealloc in the same block
+    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
+    if (!deallocOp) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    if (deallocOp->getBlock() != allocOp->getBlock()) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    // This allocation is eligible
+    ++numEligible;
+    AllocationCandidate candidate;
+    candidate.alloc = allocOp;
+    candidate.dealloc = deallocOp;
+    candidate.sizeInBytes = computeSizeInBytes(memrefType);
+    candidate.alignment = allocOp.getAlignment().value_or(1);
+    candidates.push_back(candidate);
+  });
+
+  return candidates;
+}
+
+/// Create or obtain the arena buffer based on the arena mode.
+/// Returns failure if the mode is invalid or preconditions aren't met.
+static FailureOr<Value> createArena(OpBuilder &builder,
+                                    FunctionOpInterface funcOp,
+                                    StringRef arenaMode, int64_t totalSize,
+                                    int64_t arenaAlignment) {
+  Location loc = funcOp->getLoc();
+
+  if (arenaMode == "allocate") {
+    auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
+    auto arenaAlloc =
+        memref::AllocOp::create(builder, loc, arenaType, ValueRange{},
+                                builder.getI64IntegerAttr(arenaAlignment));
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] created arena via AllocOp: size="
+               << totalSize << " bytes, alignment=" << arenaAlignment
+               << " bytes\n");
+    return arenaAlloc.getResult();
+  }
+
+  if (arenaMode == "arg") {
+    if (funcOp.getNumArguments() == 0)
+      return funcOp->emitError(
+          "arena-mode=arg requires at least one function argument");
+
+    Value arenaValue = funcOp.getArgument(0);
+    auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
+    if (!arenaType || !arenaType.getElementType().isInteger(8) ||
+        arenaType.getRank() != 1)
+      return funcOp->emitError(
+          "arena-mode=arg requires first argument to be memref<...xi8>");
+
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] using arena from function arg 0\n");
+    return arenaValue;
   }
 
-  return offsets;
+  return funcOp->emitError("invalid arena-mode: '" + arenaMode +
+                           "' (must be 'allocate' or 'arg')");
+}
+
+/// Replace each alloc/dealloc pair with a memref.view into the arena.
+static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
+                               Value arenaValue) {
+  for (auto &candidate : candidates) {
+    OpBuilder builder(candidate.alloc);
+    Location loc = candidate.alloc.getLoc();
+    MemRefType originalType = candidate.alloc.getType();
+
+    Value offsetIndex =
+        arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+    auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
+                                       offsetIndex, SmallVector<Value>{});
+
+    candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
+    candidate.alloc.erase();
+    candidate.dealloc.erase();
+  }
 }
 
 //===----------------------------------------------------------------------===//
@@ -125,143 +210,64 @@ struct StaticMemoryPlannerAnalysisPass
       StaticMemoryPlannerAnalysisPass>;
   using Base::Base;
 
-  void runOnOperation() override {
-    auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
-
-    // Step 0: Check for memref return types (not supported)
-    for (Type resultType : funcOp.getResultTypes()) {
-      if (isa<BaseMemRefType>(resultType)) {
-        funcOp->emitError("static-memory-planner does not support functions "
-                          "with memref return types");
-        return signalPassFailure();
-      }
-    }
-
-    // Step 1: Collect eligible allocation candidates
-    SmallVector<AllocationCandidate> candidates;
-
-    funcOp->walk([&](memref::AllocOp allocOp) {
-      MemRefType memrefType = allocOp.getType();
-
-      // Skip dynamic shapes
-      if (!memrefType.hasStaticShape()) {
-        ++numSkipDynamic;
-        return;
-      }
-
-      // Find unique dealloc in the same block
-      memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-      if (!deallocOp) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      if (deallocOp->getBlock() != allocOp->getBlock()) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      // This allocation is eligible
-      ++numEligible;
-      AllocationCandidate candidate;
-      candidate.alloc = allocOp;
-      candidate.dealloc = deallocOp;
-      candidate.sizeInBytes = computeSizeInBytes(memrefType);
-      candidate.alignment = allocOp.getAlignment().value_or(1);
-      candidates.push_back(candidate);
-    });
-
-    if (candidates.empty())
-      return;
-
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
-    int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
-      arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
-    }
+  void runOnOperation() override;
+};
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
-
-    // Assign computed offsets back to candidates
-    int64_t totalSize = 0;
-    for (size_t i = 0; i < candidates.size(); ++i) {
-      candidates[i].offset = offsets[i];
-      totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] offset=" << candidates[i].offset
-                 << " size=" << candidates[i].sizeInBytes
-                 << " alignment=" << candidates[i].alignment << "\n");
-    }
+void StaticMemoryPlannerAnalysisPass::runOnOperation() {
+  auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
 
-    // Step 4: Obtain arena based on arena mode
-    Operation *firstAlloc = candidates.front().alloc;
-    OpBuilder builder(firstAlloc);
-    Value arenaValue;
-
-    if (arenaMode == "allocate") {
-      // Arena is i8 byte buffer to support multiple data types (f32, i64, etc.)
-      auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
-      auto arenaAlloc = memref::AllocOp::create(
-          builder, firstAlloc->getLoc(), arenaType, ValueRange{},
-          builder.getI64IntegerAttr(arenaAlignment));
-      arenaValue = arenaAlloc.getResult();
-
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] created arena via AllocOp: size="
-                 << totalSize << " bytes, alignment=" << arenaAlignment
-                 << " bytes\n");
-    } else if (arenaMode == "arg") {
-      if (funcOp.getNumArguments() == 0) {
-        funcOp->emitError(
-            "arena-mode=arg requires at least one function argument");
-        return signalPassFailure();
-      }
-
-      arenaValue = funcOp.getArgument(0);
-      auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
-      if (!arenaType || !arenaType.getElementType().isInteger(8) ||
-          arenaType.getRank() != 1) {
-        funcOp->emitError(
-            "arena-mode=arg requires first argument to be memref<...xi8>");
-        return signalPassFailure();
-      }
-
-      LLVM_DEBUG(
-          llvm::dbgs()
-          << "[static-memory-planner] using arena from function arg 0\n");
-    } else {
-      funcOp->emitError("invalid arena-mode: '" + arenaMode +
-                        "' (must be 'allocate' or 'arg')");
+  // Step 0: Check for memref return types (not supported)
+  for (Type resultType : funcOp.getResultTypes()) {
+    if (isa<BaseMemRefType>(resultType)) {
+      funcOp->emitError("static-memory-planner does not support functions "
+                        "with memref return types");
       return signalPassFailure();
     }
+  }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
-    for (auto &candidate : candidates) {
-      builder.setInsertionPoint(candidate.alloc);
-      Location loc = candidate.alloc.getLoc();
+  // Step 1: Collect eligible allocation candidates.
+  SmallVector<AllocationCandidate> candidates =
+      collectCandidates(funcOp, numSkipDynamic, numSkipNoDealloc, numEligible);
 
-      MemRefType originalType = candidate.alloc.getType();
+  if (candidates.empty())
+    return;
 
-      // Create a constant for the byte offset into the arena
-      Value offsetIndex =
-          arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+  // Step 2: Build allocation descriptors with lifetime info.
+  SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
+  int64_t arenaAlignment = buildAllocInfos(candidates, allocInfos);
 
-      // Use memref.view to create a typed view into the i8 arena
-      auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
-                                         offsetIndex, SmallVector<Value>{});
+  // Step 3: Run the planning algorithm.
+  SmallVector<int64_t> offsets;
+  switch (algorithm) {
+  case bufferization::MemoryPlannerAlgorithm::Trivial:
+    offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  case bufferization::MemoryPlannerAlgorithm::BestFit:
+    offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  }
 
-      candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-      candidate.alloc.erase();
-      candidate.dealloc.erase();
-    }
+  // Step 4: Compute total arena size and assign offsets.
+  int64_t totalSize = 0;
+  for (size_t i = 0; i < candidates.size(); ++i) {
+    candidates[i].offset = offsets[i];
+    totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] offset=" << candidates[i].offset
+               << " size=" << candidates[i].sizeInBytes
+               << " alignment=" << candidates[i].alignment << "\n");
   }
-};
+
+  // Step 5: Obtain arena based on arena mode.
+  Operation *firstAlloc = candidates.front().alloc;
+  OpBuilder builder(firstAlloc);
+  FailureOr<Value> arenaValue =
+      createArena(builder, funcOp, arenaMode, totalSize, arenaAlignment);
+  if (failed(arenaValue))
+    return signalPassFailure();
+
+  // Step 6: Replace each alloc with memref.view into the arena.
+  rewriteAllocations(candidates, *arenaValue);
+}
 
 } // end anonymous namespace
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..b0c237507d2f3
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,131 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+/// Trivial sequential packing: places each allocation immediately after the
+/// previous one with alignment padding. Does not consider lifetimes, so no
+/// memory is reused. This gives a simple upper bound on arena size.
+/// Complexity: O(n) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+/// Best-fit lifetime-aware packing: processes allocations in start-time order
+/// and places each one in the smallest gap left by allocations whose lifetimes
+/// have ended. If no existing gap is large enough, the arena is extended.
+/// This minimizes peak memory usage when allocations have non-overlapping
+/// lifetimes.
+/// Complexity: O(n^2) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  // Tracks where each allocation was placed. We only need timeEnd because
+  // allocations are processed in timeStart order — by the time we place a new
+  // allocation, all earlier placements already started, so we only need to
+  // check which ones are still live.
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+  int64_t arenaEnd = 0;
+
+  // Loop over all required allocations and fit them into best gaps.
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect allocations that are still live at this alloc's start time.
+    // occupied is pairs of <offset_start, offset_end>.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      if (alignedStart >= occStart) {
+        gapStart = std::max(gapStart, occEnd);
+        continue;
+      }
+      int64_t gapEnd = occStart;
+      int64_t gapSize = gapEnd - alignedStart;
+      // Gap is large enough to fit this allocation (after alignment).
+      if (gapSize >= alloc.sizeInBytes) {
+        // Track the smallest sufficient gap (best-fit strategy).
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied region and arena end).
+    // This is only a reuse candidate if the allocation fits within the current
+    // arena bounds. If it doesn't fit, placing here would extend the arena —
+    // that case is handled by the fallback below (bestOffset < 0).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+    arenaEnd = std::max(arenaEnd, bestOffset + alloc.sizeInBytes);
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // C at offset 5120
+  // CHECK-NEXT: %{{.*}} = arith.constant 5120 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // D at offset 9216
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // E at offset 9216 (best-fit picks 1024-byte trailing gap over 5120-byte gap)
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<256xf32>
+  %c = memref.alloc() : memref<1024xf32>
+  %d = memref.alloc() : memref<256xf32>
+  memref.dealloc %b : memref<256xf32>
+  memref.dealloc %d : memref<256xf32>
+  memref.dealloc %a : memref<1024xf32>
+  %e = memref.alloc() : memref<128xf32>
+  memref.dealloc %c : memref<1024xf32>
+  memref.dealloc %e : memref<128xf32>
+  return
+}
+
+// -----
+
+// Test 4: Alignment padding can disqualify a gap.
+// A(128, align 128) at 0 (live), B(56, align 1) at 128, C(128, align 128) at 256 (live).
+// B is freed => gap [128, 256) = 128 bytes.
+// D(64, align 128): aligned start = 128, fits in gap.
+// E(64, align 256): next 256-aligned offset in [128,256) is 256 = gap end, doesn't fit.
+//   E must go past the arena high-water mark.
+// CHECK-LABEL: func @best_fit_alignment
+func.func @best_fit_alignment() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 256 : i64} : memref<576xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<56xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 256 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // D fits in gap at offset 128
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  // E cannot fit in gap (alignment 256), placed at offset 512
+  // CHECK-NEXT: %{{.*}} = arith.constant 512 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  %a = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  %b = memref.alloc() : memref<56xi8>
+  %c = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  memref.dealloc %b : memref<56xi8>
+  %d = memref.alloc() {alignment = 128 : i64} : memref<64xi8>
+  %e = memref.alloc() {alignment = 256 : i64} : memref<64xi8>
+  memref.dealloc %a : memref<128xi8>
+  memref.dealloc %c : memref<128xi8>
+  memref.dealloc %d : memref<64xi8>
+  memref.dealloc %e : memref<64xi8>
+  return
+}


From llvmlistbot at llvm.org  Tue Jul  7 08:47:23 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Tue, 07 Jul 2026 08:47:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4d1f8b.7270d2fd.2d4879.0f3e@mx.google.com>


https://github.com/javedabsar1 updated https://github.com/llvm/llvm-project/pull/207403

>From 63bad1fb8d1de9fd84c3adc391d0eb4b9f56427f Mon Sep 17 00:00:00 2001
From: mabsar <mabsar at qti.qualcommm.com>
Date: Fri, 3 Jul 2026 07:20:15 -0700
Subject: [PATCH] [mlir][bufferization] Add best-fit algorithm to static memory
 planner

Introduces an algorithm selection option to the static memory planner pass
and add a best-fit algorithm that reuses memory from expired allocations
by finding the smallest suitable gap. The planning algorithms are factored
into a separate StaticMemoryPlanning.{h,cpp} to keep them independent of
MLIR IR and easily testable.

Signed-off-by: mabsar <mabsar at qti.qualcommm.com>
---
 .../Bufferization/IR/BufferizationEnums.td    |   8 +
 .../Bufferization/Transforms/Passes.td        |  31 +-
 .../Transforms/StaticMemoryPlanning.h         |  50 +++
 .../Bufferization/Transforms/CMakeLists.txt   |   1 +
 .../StaticMemoryPlannerAnalysis.cpp           | 344 +++++++++---------
 .../Transforms/StaticMemoryPlanning.cpp       | 131 +++++++
 .../static-memory-planner-best-fit.mlir       | 118 ++++++
 7 files changed, 504 insertions(+), 179 deletions(-)
 create mode 100644 mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
 create mode 100644 mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of different element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of different element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",
diff --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c2ac40a8427e7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,36 +68,134 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
+/// Build lifetime-annotated allocation descriptors from candidates.
+/// Returns the arena alignment (LCM of all individual alignments).
+static int64_t buildAllocInfos(
+    MutableArrayRef<AllocationCandidate> candidates,
+    SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  int64_t arenaAlignment = 1;
+  for (auto &candidate : candidates) {
+    bufferization::MemoryPlannerAlloc info;
+    info.sizeInBytes = candidate.sizeInBytes;
+    info.alignment = candidate.alignment;
+
+    Block *block = candidate.alloc->getBlock();
+    int64_t opIdx = 0;
+    for (Operation &op : *block) {
+      if (&op == candidate.alloc.getOperation())
+        info.timeStart = opIdx;
+      if (&op == candidate.dealloc.getOperation())
+        info.timeEnd = opIdx;
+      ++opIdx;
+    }
+
+    allocInfos.push_back(info);
+    arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
+  }
+  return arenaAlignment;
 }
 
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
+/// Collect alloc/dealloc pairs eligible for arena placement.
+/// An allocation is eligible if it has a static shape and a unique dealloc
+/// in the same block.
+static SmallVector<AllocationCandidate>
+collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
+                  llvm::Statistic &numSkipNoDealloc,
+                  llvm::Statistic &numEligible) {
+  SmallVector<AllocationCandidate> candidates;
+
+  funcOp->walk([&](memref::AllocOp allocOp) {
+    MemRefType memrefType = allocOp.getType();
+
+    // Skip dynamic shapes
+    if (!memrefType.hasStaticShape()) {
+      ++numSkipDynamic;
+      return;
+    }
 
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
+    // Find unique dealloc in the same block
+    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
+    if (!deallocOp) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    if (deallocOp->getBlock() != allocOp->getBlock()) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    // This allocation is eligible
+    ++numEligible;
+    AllocationCandidate candidate;
+    candidate.alloc = allocOp;
+    candidate.dealloc = deallocOp;
+    candidate.sizeInBytes = computeSizeInBytes(memrefType);
+    candidate.alignment = allocOp.getAlignment().value_or(1);
+    candidates.push_back(candidate);
+  });
+
+  return candidates;
+}
+
+/// Create or obtain the arena buffer based on the arena mode.
+/// Returns failure if the mode is invalid or preconditions aren't met.
+static FailureOr<Value> createArena(OpBuilder &builder,
+                                    FunctionOpInterface funcOp,
+                                    StringRef arenaMode, int64_t totalSize,
+                                    int64_t arenaAlignment) {
+  Location loc = funcOp->getLoc();
+
+  if (arenaMode == "allocate") {
+    auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
+    auto arenaAlloc =
+        memref::AllocOp::create(builder, loc, arenaType, ValueRange{},
+                                builder.getI64IntegerAttr(arenaAlignment));
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] created arena via AllocOp: size="
+               << totalSize << " bytes, alignment=" << arenaAlignment
+               << " bytes\n");
+    return arenaAlloc.getResult();
+  }
+
+  if (arenaMode == "arg") {
+    if (funcOp.getNumArguments() == 0)
+      return funcOp->emitError(
+          "arena-mode=arg requires at least one function argument");
+
+    Value arenaValue = funcOp.getArgument(0);
+    auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
+    if (!arenaType || !arenaType.getElementType().isInteger(8) ||
+        arenaType.getRank() != 1)
+      return funcOp->emitError(
+          "arena-mode=arg requires first argument to be memref<...xi8>");
+
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] using arena from function arg 0\n");
+    return arenaValue;
   }
 
-  return offsets;
+  return funcOp->emitError("invalid arena-mode: '" + arenaMode +
+                           "' (must be 'allocate' or 'arg')");
+}
+
+/// Replace each alloc/dealloc pair with a memref.view into the arena.
+static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
+                               Value arenaValue) {
+  for (auto &candidate : candidates) {
+    OpBuilder builder(candidate.alloc);
+    Location loc = candidate.alloc.getLoc();
+    MemRefType originalType = candidate.alloc.getType();
+
+    Value offsetIndex =
+        arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+    auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
+                                       offsetIndex, SmallVector<Value>{});
+
+    candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
+    candidate.alloc.erase();
+    candidate.dealloc.erase();
+  }
 }
 
 //===----------------------------------------------------------------------===//
@@ -125,143 +210,64 @@ struct StaticMemoryPlannerAnalysisPass
       StaticMemoryPlannerAnalysisPass>;
   using Base::Base;
 
-  void runOnOperation() override {
-    auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
-
-    // Step 0: Check for memref return types (not supported)
-    for (Type resultType : funcOp.getResultTypes()) {
-      if (isa<BaseMemRefType>(resultType)) {
-        funcOp->emitError("static-memory-planner does not support functions "
-                          "with memref return types");
-        return signalPassFailure();
-      }
-    }
-
-    // Step 1: Collect eligible allocation candidates
-    SmallVector<AllocationCandidate> candidates;
-
-    funcOp->walk([&](memref::AllocOp allocOp) {
-      MemRefType memrefType = allocOp.getType();
-
-      // Skip dynamic shapes
-      if (!memrefType.hasStaticShape()) {
-        ++numSkipDynamic;
-        return;
-      }
-
-      // Find unique dealloc in the same block
-      memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-      if (!deallocOp) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      if (deallocOp->getBlock() != allocOp->getBlock()) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      // This allocation is eligible
-      ++numEligible;
-      AllocationCandidate candidate;
-      candidate.alloc = allocOp;
-      candidate.dealloc = deallocOp;
-      candidate.sizeInBytes = computeSizeInBytes(memrefType);
-      candidate.alignment = allocOp.getAlignment().value_or(1);
-      candidates.push_back(candidate);
-    });
-
-    if (candidates.empty())
-      return;
-
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
-    int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
-      arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
-    }
+  void runOnOperation() override;
+};
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
-
-    // Assign computed offsets back to candidates
-    int64_t totalSize = 0;
-    for (size_t i = 0; i < candidates.size(); ++i) {
-      candidates[i].offset = offsets[i];
-      totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] offset=" << candidates[i].offset
-                 << " size=" << candidates[i].sizeInBytes
-                 << " alignment=" << candidates[i].alignment << "\n");
-    }
+void StaticMemoryPlannerAnalysisPass::runOnOperation() {
+  auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
 
-    // Step 4: Obtain arena based on arena mode
-    Operation *firstAlloc = candidates.front().alloc;
-    OpBuilder builder(firstAlloc);
-    Value arenaValue;
-
-    if (arenaMode == "allocate") {
-      // Arena is i8 byte buffer to support multiple data types (f32, i64, etc.)
-      auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
-      auto arenaAlloc = memref::AllocOp::create(
-          builder, firstAlloc->getLoc(), arenaType, ValueRange{},
-          builder.getI64IntegerAttr(arenaAlignment));
-      arenaValue = arenaAlloc.getResult();
-
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] created arena via AllocOp: size="
-                 << totalSize << " bytes, alignment=" << arenaAlignment
-                 << " bytes\n");
-    } else if (arenaMode == "arg") {
-      if (funcOp.getNumArguments() == 0) {
-        funcOp->emitError(
-            "arena-mode=arg requires at least one function argument");
-        return signalPassFailure();
-      }
-
-      arenaValue = funcOp.getArgument(0);
-      auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
-      if (!arenaType || !arenaType.getElementType().isInteger(8) ||
-          arenaType.getRank() != 1) {
-        funcOp->emitError(
-            "arena-mode=arg requires first argument to be memref<...xi8>");
-        return signalPassFailure();
-      }
-
-      LLVM_DEBUG(
-          llvm::dbgs()
-          << "[static-memory-planner] using arena from function arg 0\n");
-    } else {
-      funcOp->emitError("invalid arena-mode: '" + arenaMode +
-                        "' (must be 'allocate' or 'arg')");
+  // Step 0: Check for memref return types (not supported)
+  for (Type resultType : funcOp.getResultTypes()) {
+    if (isa<BaseMemRefType>(resultType)) {
+      funcOp->emitError("static-memory-planner does not support functions "
+                        "with memref return types");
       return signalPassFailure();
     }
+  }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
-    for (auto &candidate : candidates) {
-      builder.setInsertionPoint(candidate.alloc);
-      Location loc = candidate.alloc.getLoc();
+  // Step 1: Collect eligible allocation candidates.
+  SmallVector<AllocationCandidate> candidates =
+      collectCandidates(funcOp, numSkipDynamic, numSkipNoDealloc, numEligible);
 
-      MemRefType originalType = candidate.alloc.getType();
+  if (candidates.empty())
+    return;
 
-      // Create a constant for the byte offset into the arena
-      Value offsetIndex =
-          arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+  // Step 2: Build allocation descriptors with lifetime info.
+  SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
+  int64_t arenaAlignment = buildAllocInfos(candidates, allocInfos);
 
-      // Use memref.view to create a typed view into the i8 arena
-      auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
-                                         offsetIndex, SmallVector<Value>{});
+  // Step 3: Run the planning algorithm.
+  SmallVector<int64_t> offsets;
+  switch (algorithm) {
+  case bufferization::MemoryPlannerAlgorithm::Trivial:
+    offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  case bufferization::MemoryPlannerAlgorithm::BestFit:
+    offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  }
 
-      candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-      candidate.alloc.erase();
-      candidate.dealloc.erase();
-    }
+  // Step 4: Compute total arena size and assign offsets.
+  int64_t totalSize = 0;
+  for (size_t i = 0; i < candidates.size(); ++i) {
+    candidates[i].offset = offsets[i];
+    totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] offset=" << candidates[i].offset
+               << " size=" << candidates[i].sizeInBytes
+               << " alignment=" << candidates[i].alignment << "\n");
   }
-};
+
+  // Step 5: Obtain arena based on arena mode.
+  Operation *firstAlloc = candidates.front().alloc;
+  OpBuilder builder(firstAlloc);
+  FailureOr<Value> arenaValue =
+      createArena(builder, funcOp, arenaMode, totalSize, arenaAlignment);
+  if (failed(arenaValue))
+    return signalPassFailure();
+
+  // Step 6: Replace each alloc with memref.view into the arena.
+  rewriteAllocations(candidates, *arenaValue);
+}
 
 } // end anonymous namespace
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..b0c237507d2f3
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,131 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+/// Trivial sequential packing: places each allocation immediately after the
+/// previous one with alignment padding. Does not consider lifetimes, so no
+/// memory is reused. This gives a simple upper bound on arena size.
+/// Complexity: O(n) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+/// Best-fit lifetime-aware packing: processes allocations in start-time order
+/// and places each one in the smallest gap left by allocations whose lifetimes
+/// have ended. If no existing gap is large enough, the arena is extended.
+/// This minimizes peak memory usage when allocations have non-overlapping
+/// lifetimes.
+/// Complexity: O(n^2) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  // Tracks where each allocation was placed. We only need timeEnd because
+  // allocations are processed in timeStart order — by the time we place a new
+  // allocation, all earlier placements already started, so we only need to
+  // check which ones are still live.
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+  int64_t arenaEnd = 0;
+
+  // Loop over all required allocations and fit them into best gaps.
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect allocations that are still live at this alloc's start time.
+    // occupied is pairs of <offset_start, offset_end>.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      if (alignedStart >= occStart) {
+        gapStart = std::max(gapStart, occEnd);
+        continue;
+      }
+      int64_t gapEnd = occStart;
+      int64_t gapSize = gapEnd - alignedStart;
+      // Gap is large enough to fit this allocation (after alignment).
+      if (gapSize >= alloc.sizeInBytes) {
+        // Track the smallest sufficient gap (best-fit strategy).
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied region and arena end).
+    // This is only a reuse candidate if the allocation fits within the current
+    // arena bounds. If it doesn't fit, placing here would extend the arena —
+    // that case is handled by the fallback below (bestOffset < 0).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+    arenaEnd = std::max(arenaEnd, bestOffset + alloc.sizeInBytes);
+  }
+
+  return offsets;
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // C at offset 5120
+  // CHECK-NEXT: %{{.*}} = arith.constant 5120 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // D at offset 9216
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // E at offset 9216 (best-fit picks 1024-byte trailing gap over 5120-byte gap)
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<256xf32>
+  %c = memref.alloc() : memref<1024xf32>
+  %d = memref.alloc() : memref<256xf32>
+  memref.dealloc %b : memref<256xf32>
+  memref.dealloc %d : memref<256xf32>
+  memref.dealloc %a : memref<1024xf32>
+  %e = memref.alloc() : memref<128xf32>
+  memref.dealloc %c : memref<1024xf32>
+  memref.dealloc %e : memref<128xf32>
+  return
+}
+
+// -----
+
+// Test 4: Alignment padding can disqualify a gap.
+// A(128, align 128) at 0 (live), B(56, align 1) at 128, C(128, align 128) at 256 (live).
+// B is freed => gap [128, 256) = 128 bytes.
+// D(64, align 128): aligned start = 128, fits in gap.
+// E(64, align 256): next 256-aligned offset in [128,256) is 256 = gap end, doesn't fit.
+//   E must go past the arena high-water mark.
+// CHECK-LABEL: func @best_fit_alignment
+func.func @best_fit_alignment() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 256 : i64} : memref<576xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<56xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 256 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // D fits in gap at offset 128
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  // E cannot fit in gap (alignment 256), placed at offset 512
+  // CHECK-NEXT: %{{.*}} = arith.constant 512 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  %a = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  %b = memref.alloc() : memref<56xi8>
+  %c = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  memref.dealloc %b : memref<56xi8>
+  %d = memref.alloc() {alignment = 128 : i64} : memref<64xi8>
+  %e = memref.alloc() {alignment = 256 : i64} : memref<64xi8>
+  memref.dealloc %a : memref<128xi8>
+  memref.dealloc %c : memref<128xi8>
+  memref.dealloc %d : memref<64xi8>
+  memref.dealloc %e : memref<64xi8>
+  return
+}


From llvmlistbot at llvm.org  Tue Jul  7 09:49:11 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 07 Jul 2026 09:49:11 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4d2e07.13be6df8.145c.f6d2@mx.google.com>


================

----------------
madhur13490 wrote:

Done

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 09:49:11 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 07 Jul 2026 09:49:11 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4d2e07.cbc3f03a.10d998.4a6d@mx.google.com>


https://github.com/madhur13490 updated https://github.com/llvm/llvm-project/pull/201077

>From f4a850b7455781116f95bcb5421fc0ed81ea725d Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Wed, 15 Apr 2026 01:00:40 -0700
Subject: [PATCH 1/3] Enforce single-operand form for llvm.loop.distribute
 metadata

This patch implements Phase 1 of the RFC "Enforce Single-Operand Format for All
.enable Metadata Nodes". The two-operand boolean form
!{!"llvm.loop.distribute.enable", i1 0/1} is replaced by a
single-operand enable/disable pair:

  !{!"llvm.loop.distribute.enable"}    ; force distribution
  !{!"llvm.loop.distribute.disable"}   ; suppress distribution
---
 clang/lib/CodeGen/CGLoopInfo.cpp              | 22 ++++-----
 .../test/CodeGenCXX/pragma-followup_inner.cpp |  2 +-
 .../test/CodeGenCXX/pragma-followup_outer.cpp |  2 +-
 .../CodeGenCXX/pragma-loop-distribute.cpp     |  2 +-
 clang/test/CodeGenCXX/pragma-loop.cpp         |  2 +-
 llvm/docs/LangRef.rst                         | 15 +++---
 llvm/lib/Analysis/LoopAccessAnalysis.cpp      | 11 ++---
 llvm/lib/IR/Verifier.cpp                      |  8 +++
 llvm/lib/Transforms/Scalar/LoopDistribute.cpp | 13 ++---
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp |  3 +-
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |  3 ++
 .../Analysis/LoopAccessAnalysis/pr64637.ll    |  2 +-
 .../Other/optimization-remarks-lazy-bfi.ll    |  2 +-
 .../IRCE/add-metadata-pre-post-loops.ll       |  2 +-
 .../test/Transforms/IRCE/correct-loop-info.ll |  2 +-
 .../Transforms/IRCE/loop-guarded-bounds.ll    |  2 +-
 llvm/test/Transforms/IRCE/pre_post_loops.ll   |  2 +-
 .../IRCE/range_intersect_miscompile.ll        |  2 +-
 .../IRCE/unsigned_comparisons_ugt.ll          |  2 +-
 .../IRCE/unsigned_comparisons_ult.ll          |  2 +-
 .../Inline/dilocation-loop-metadata-update.ll |  4 +-
 .../LoopDistribute/basic-with-memchecks.ll    |  2 +-
 .../diagnostics-with-hotness.ll               |  2 +-
 .../Transforms/LoopDistribute/diagnostics.ll  |  2 +-
 .../disable_nonforced_enable.ll               |  2 +-
 .../Transforms/LoopDistribute/early-exit.ll   |  2 +-
 .../Transforms/LoopDistribute/followup.ll     |  2 +-
 .../Transforms/LoopDistribute/metadata.ll     |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 llvm/test/Verifier/llvm.loop.distribute.ll    | 49 +++++++++++++++++++
 llvm/unittests/Analysis/LoopInfoTest.cpp      |  8 +--
 .../Target/LLVMIR/LoopAnnotationImporter.cpp  |  6 ++-
 .../LLVMIR/LoopAnnotationTranslation.cpp      |  7 ++-
 .../Target/LLVMIR/Import/metadata-loop.ll     |  2 +-
 mlir/test/Target/LLVMIR/loop-metadata.mlir    |  2 +-
 36 files changed, 131 insertions(+), 72 deletions(-)
 create mode 100644 llvm/test/Verifier/llvm.loop.distribute.ll

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 6b0b1b55213e1..1ae6bbe0ff595 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -331,11 +331,12 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
-      NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.append(
+          LoopProperties.begin(), LoopProperties.end());
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx,
+          {MDString::get(
+              Ctx, "llvm.loop.distribute.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createLoopVectorizeMetadata(Attrs, LoopProperties,
@@ -344,16 +345,15 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
 
   bool FollowupHasTransforms = false;
   SmallVector<Metadata *, 4> Followup =
-      createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
+      createLoopVectorizeMetadata(
+          Attrs, LoopProperties, FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
-  Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                      ConstantAsMetadata::get(ConstantInt::get(
-                          llvm::Type::getInt1Ty(Ctx),
-                          (Attrs.DistributeEnable == LoopAttributes::Enable)))};
-  Args.push_back(MDNode::get(Ctx, Vals));
+  Args.push_back(MDNode::get(
+      Ctx,
+      {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
 
   if (FollowupHasTransforms)
     Args.push_back(
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..2cf2dba5b913d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_inner(int n, int *x) {
 
 // CHECK-DAG: ![[INNERLOOP_3]] = distinct !{![[INNERLOOP_3]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..2e907fb814b3d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -19,7 +19,7 @@ extern "C" void followup_outer(int n, int *x) {
 
 // CHECK-DAG: ![[LOOP_3:[0-9]+]] = distinct !{![[LOOP_3:[0-9]+]], [[MP:![0-9]+]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4:[0-9]+]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
index 98aef21da2acd..66f0fb07b51fb 100644
--- a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
@@ -20,5 +20,5 @@ void while_test(int *List, int Length, int *List2, int Length2) {
 }
 
 // CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], [[MP:![0-9]+]], ![[DISTRIBUTE_ENABLE:.*]]}
-// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable"}
 // CHECK: [[LOOP_2]] = distinct !{[[LOOP_2]], [[MP]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..f9106725ad99d 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -248,7 +248,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[UNROLL_32:[0-9]+]] = !{!"llvm.loop.unroll.count", i32 32}
 // CHECK-DAG: ![[UNROLL_FULL:[0-9]+]] = !{!"llvm.loop.unroll.full"}
 
-// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.disable"}
 
 // CHECK-DAG: ![[INTERLEAVE_2:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 2}
 // CHECK-DAG: ![[INTERLEAVE_4:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 4}
diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 6e7f63ed7fb69..9d37207753686 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -8385,18 +8385,17 @@ this is only performed if the entire loop cannot be vectorized due to unsafe
 memory dependencies.  The transformation will attempt to isolate the unsafe
 dependencies into their own loop.
 
-This metadata can be used to selectively enable or disable distribution of the
-loop.  The first operand is the string ``llvm.loop.distribute.enable`` and the
-second operand is a bit. If the bit operand value is 1 distribution is
-enabled. A value of 0 disables distribution:
+This metadata can be used to selectively enable or disable
+distribution of the loop.  Each node has a single operand
+containing the name string:
 
 .. code-block:: llvm
 
-   !0 = !{!"llvm.loop.distribute.enable", i1 0}
-   !1 = !{!"llvm.loop.distribute.enable", i1 1}
+   !0 = !{!"llvm.loop.distribute.enable"}
+   !1 = !{!"llvm.loop.distribute.disable"}
 
-This metadata should be used in conjunction with ``llvm.loop`` loop
-identification metadata.
+This metadata should be used in conjunction with ``llvm.loop``
+loop identification metadata.
 
 '``llvm.loop.distribute.followup_coincident``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index f28b570d6191b..d63208e2fe5d3 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2929,14 +2929,9 @@ void LoopAccessInfo::emitUnsafeDependenceRemark() {
   LLVM_DEBUG(dbgs() << "LAA: unsafe dependent memory operations in loop\n");
 
   // Emit remark for first unsafe dependence
-  bool HasForcedDistribution = false;
-  std::optional<const MDOperand *> Value =
-      findStringMetadataForLoop(TheLoop, "llvm.loop.distribute.enable");
-  if (Value) {
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    HasForcedDistribution = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
-  }
+  bool HasForcedDistribution =
+      getBooleanLoopAttribute(
+          TheLoop, "llvm.loop.distribute.enable");
 
   const std::string Info =
       HasForcedDistribution
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 198ff055f9721..857fa1200c590 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -995,6 +995,14 @@ void Verifier::visitMDNode(const MDNode &BaseMD,
             CurrentMD);
     }
 
+    // Enforce the single-operand form of llvm.loop.distribute metadata.
+    if (CurrentMD->getNumOperands() > 0 &&
+        (CurrentMD->getOperand(0).equalsStr("llvm.loop.distribute.enable") ||
+         CurrentMD->getOperand(0).equalsStr("llvm.loop.distribute.disable")))
+      Check(CurrentMD->getNumOperands() == 1,
+            "Expected one operand for llvm.loop.distribute metadata",
+            CurrentMD);
+
     // Check these last, so we diagnose problems in operands first.
     Check(!CurrentMD->isTemporary(), "Expected no forward declarations!",
           CurrentMD);
diff --git a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
index 2d0b08fed859b..07cfd8628a4f6 100644
--- a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -941,14 +941,11 @@ class LoopDistributeForLoop {
   /// Check whether the loop metadata is forcing distribution to be
   /// enabled/disabled.
   void setForced() {
-    std::optional<const MDOperand *> Value =
-        findStringMetadataForLoop(L, "llvm.loop.distribute.enable");
-    if (!Value)
-      return;
-
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    IsForced = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
+    if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
+      IsForced = true;
+    else if (getBooleanLoopAttribute(
+                 L, "llvm.loop.distribute.disable"))
+      IsForced = false;
   }
 
   Loop *L;
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 72fae62e56e88..74cd2a729d285 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -453,7 +453,8 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
       Context,
-      {MDString::get(Context, "llvm.loop.distribute.enable"), FalseVal});
+      {MDString::get(
+          Context, "llvm.loop.distribute.disable")});
   MDNode *NewLoopID =
       MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
                             DisableLICMVersioning, DisableDistribution});
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index a2630c481a72e..3133bf2804034 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -441,6 +441,9 @@ TransformationMode llvm::hasVectorizeTransformation(const Loop *L) {
 }
 
 TransformationMode llvm::hasDistributeTransformation(const Loop *L) {
+  if (getBooleanLoopAttribute(L, "llvm.loop.distribute.disable"))
+    return TM_SuppressedByUser;
+
   if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
     return TM_ForcedByUser;
 
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
index d3e589cf99cf3..7b25d1fffcbba 100644
--- a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
+++ b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
@@ -76,4 +76,4 @@ for.body:                                         ; preds = %for.body, %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Other/optimization-remarks-lazy-bfi.ll b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
index c5099bb2179e4..5fc34cc14bdc8 100644
--- a/llvm/test/Other/optimization-remarks-lazy-bfi.ll
+++ b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
@@ -75,7 +75,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 99, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
index eb72907730c6d..3a360814e361e 100644
--- a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
+++ b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
@@ -76,7 +76,7 @@ attributes #0 = { alwaysinline }
 !3 = !{!"llvm.loop.unroll.disable"}
 !4 = !{!"llvm.loop.vectorize.enable", i1 false}
 !5 = !{!"llvm.loop.licm_versioning.disable"}
-!6 = !{!"llvm.loop.distribute.enable", i1 false}
+!6 = !{!"llvm.loop.distribute.disable"}
 !7 = !{}
 !8 = distinct !{!8, !3, !4, !5}
 !9 = distinct !{!9, !3, !4, !5}
diff --git a/llvm/test/Transforms/IRCE/correct-loop-info.ll b/llvm/test/Transforms/IRCE/correct-loop-info.ll
index 7ea445718289a..bc5987ebba2e4 100644
--- a/llvm/test/Transforms/IRCE/correct-loop-info.ll
+++ b/llvm/test/Transforms/IRCE/correct-loop-info.ll
@@ -178,6 +178,6 @@ declare void @pluto()
 !1 = !{!"llvm.loop.unroll.disable"}
 !2 = !{!"llvm.loop.vectorize.enable", i1 false}
 !3 = !{!"llvm.loop.licm_versioning.disable"}
-!4 = !{!"llvm.loop.distribute.enable", i1 false}
+!4 = !{!"llvm.loop.distribute.disable"}
 !5 = !{}
 !6 = distinct !{!6, !1, !2, !3, !4}
diff --git a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
index ea169987cf2fc..61cfef6f1492a 100644
--- a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
+++ b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
@@ -130,7 +130,7 @@ exit:                                       ; preds = %guarded, %loop_, %prehead
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/pre_post_loops.ll b/llvm/test/Transforms/IRCE/pre_post_loops.ll
index da2faa34719af..2807adba35940 100644
--- a/llvm/test/Transforms/IRCE/pre_post_loops.ll
+++ b/llvm/test/Transforms/IRCE/pre_post_loops.ll
@@ -163,7 +163,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
index 68613feacba32..2e21bb5ad0bd0 100644
--- a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
+++ b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
@@ -490,7 +490,7 @@ deopt:                                          ; preds = %range_check_block
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ; CHECK: [[RNG7]] = !{i32 0, i32 50}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
index 7236cf4817ade..cdd7235ac45b4 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
@@ -447,7 +447,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
index 546e1b0faa1c3..107a3554282ad 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
@@ -654,7 +654,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
index 1bc132663331b..0d8dcfbcf2a8b 100644
--- a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
+++ b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
@@ -47,7 +47,7 @@ entry:
 !7 = !DILocation(line: 6, column: 3, scope: !3)
 !8 = !DILocation(line: 7, column: 22, scope: !3)
 !9 = !{!"llvm.loop.mustprogress"}
-!10 = !{!"llvm.loop.distribute.enable", i1 true}
+!10 = !{!"llvm.loop.distribute.enable"}
 !11 = !{!"llvm.loop.distribute.followup_all", !7, !8, !9, !12, !13, !14}
 !12 = !{!"llvm.loop.vectorize.width", i32 8}
 !13 = !{!"llvm.loop.vectorize.enable", i1 true}
@@ -66,7 +66,7 @@ entry:
 ; CHECK: [[META7]] = !DILocation(line: 6, column: 3, scope: [[DBG3]])
 ; CHECK: [[META8]] = !DILocation(line: 7, column: 22, scope: [[DBG3]])
 ; CHECK: [[META9]] = !{!"llvm.loop.mustprogress"}
-; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable"}
 ; CHECK: [[META11]] = !{!"llvm.loop.distribute.followup_all", [[META7]], [[META8]], [[META9]], [[META12:![0-9]+]], [[META13:![0-9]+]], [[META14:![0-9]+]]}
 ; CHECK: [[META12]] = !{!"llvm.loop.vectorize.width", i32 8}
 ; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
index 2828882afe779..dcd231a900ee2 100644
--- a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
+++ b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
@@ -288,5 +288,5 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: ![[PROF1]] = !{!"unknown", !"loop-versioning"}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
index b5ef3578eadd9..4f2f76191e615 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
@@ -74,7 +74,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 2000, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics.ll b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
index e6a0d83bd63d2..80b217d0abb2b 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
@@ -194,7 +194,7 @@ attributes #1 = { nounwind convergent }
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = distinct !DISubprogram(name: "not_forced", scope: !1, file: !1, line: 8, type: !8, isLocal: false, isDefinition: true, scopeLine: 8, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !2)
 !23 = !DILocation(line: 9, column: 20, scope: !22)
 !24 = !DILocation(line: 9, column: 3, scope: !22)
diff --git a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
index 45a2d31256a27..eff1621cd7ada 100644
--- a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
+++ b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
@@ -43,4 +43,4 @@ for.end:
   ret void
 }
 
-!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable", i32 1}}
+!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable"}}
diff --git a/llvm/test/Transforms/LoopDistribute/early-exit.ll b/llvm/test/Transforms/LoopDistribute/early-exit.ll
index 9353d842523f4..d2ad9b1bcd6c5 100644
--- a/llvm/test/Transforms/LoopDistribute/early-exit.ll
+++ b/llvm/test/Transforms/LoopDistribute/early-exit.ll
@@ -89,4 +89,4 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/LoopDistribute/followup.ll b/llvm/test/Transforms/LoopDistribute/followup.ll
index ae418ad9e6b42..2753bad5e1c1e 100644
--- a/llvm/test/Transforms/LoopDistribute/followup.ll
+++ b/llvm/test/Transforms/LoopDistribute/followup.ll
@@ -125,7 +125,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1, !2, !3, !4, !5}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = !{!"llvm.loop.distribute.followup_all", !{!"FollowupAll"}}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !{!"FollowupCoincident", i1 false}}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !{!"FollowupSequential", i32 8}}
diff --git a/llvm/test/Transforms/LoopDistribute/metadata.ll b/llvm/test/Transforms/LoopDistribute/metadata.ll
index b0e461fe0ea75..8b74ac4249550 100644
--- a/llvm/test/Transforms/LoopDistribute/metadata.ll
+++ b/llvm/test/Transforms/LoopDistribute/metadata.ll
@@ -131,6 +131,6 @@ for.end:                                          ; preds = %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.distribute.enable", i1 false}
+!3 = !{!"llvm.loop.distribute.disable"}
diff --git a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
index 2c1dff7ee5a5f..049880a97a211 100644
--- a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
@@ -65,6 +65,6 @@ cleanup:                                          ; preds = %cleanup.loopexit, %
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
index 8b3bdbee2249c..d565b860a9ef6 100644
--- a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
@@ -147,7 +147,7 @@ while.end:                                        ; preds = %while.cond
 
 !0 = !{i32 2, !"Debug Info Version", i32 3}
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = !{!"llvm.loop.distribute.enable"}
 !3 = distinct !{!3, !4}
 !4 = !{!"llvm.loop.mustprogress"}
 !5 = distinct !{!5, !4, !6}
@@ -161,7 +161,7 @@ while.end:                                        ; preds = %while.cond
 !13 = !DILocation(line: 9, column: 23, scope: !12)
 
 ; CHECK: !1 = distinct !{!1, !2}
-; CHECK: !2 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !2 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !3 = distinct !{!3, !4}
 ; CHECK: !4 = !{!"llvm.loop.mustprogress"}
 ; CHECK: !5 = distinct !{!5, !4, !6}
diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
new file mode 100644
index 0000000000000..9e1e8138e93dc
--- /dev/null
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -0,0 +1,49 @@
+; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
+; single-operand validation.
+
+; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
+; DEFINE:   FileCheck %s -allow-empty -check-prefix
+
+define void @test() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+!0 = distinct !{!0, !1}
+
+; GOOD-NOT: {{.}}
+
+;      BAD: Expected one operand for llvm.loop.distribute metadata
+
+; Single-operand enable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Single-operand disable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Two-operand enable with boolean false (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand enable with boolean true (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand disable (rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
diff --git a/llvm/unittests/Analysis/LoopInfoTest.cpp b/llvm/unittests/Analysis/LoopInfoTest.cpp
index 4828573c32c10..5321f72aa1784 100644
--- a/llvm/unittests/Analysis/LoopInfoTest.cpp
+++ b/llvm/unittests/Analysis/LoopInfoTest.cpp
@@ -72,7 +72,7 @@ TEST(LoopInfoTest, LoopWithSingleLatch) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -122,7 +122,7 @@ TEST(LoopInfoTest, LoopWithMultipleLatches) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1397,7 +1397,7 @@ TEST(LoopInfoTest, LoopUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1440,7 +1440,7 @@ TEST(LoopInfoTest, LoopNonLatchUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index e4905423347a2..4022afc433a61 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -358,8 +358,10 @@ FailureOr<LoopLICMAttr> LoopMetadataConversion::convertLICMAttr() {
 }
 
 FailureOr<LoopDistributeAttr> LoopMetadataConversion::convertDistributeAttr() {
-  FailureOr<BoolAttr> disable =
-      lookupBoolNode("llvm.loop.distribute.enable", true);
+  FailureOr<BoolAttr> disable = lookupBooleanUnitNode(
+      "llvm.loop.distribute.enable",
+      "llvm.loop.distribute.disable",
+      /*negated=*/true);
   FailureOr<LoopAnnotationAttr> followupCoincident =
       lookupFollowupNode("llvm.loop.distribute.followup_coincident");
   FailureOr<LoopAnnotationAttr> followupSequential =
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
index f3ec18c9074e4..41ba31106f788 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
@@ -162,7 +162,12 @@ void LoopAnnotationConversion::convertLoopOptions(LoopLICMAttr options) {
 }
 
 void LoopAnnotationConversion::convertLoopOptions(LoopDistributeAttr options) {
-  convertBoolNode("llvm.loop.distribute.enable", options.getDisable(), true);
+  if (auto disable = options.getDisable()) {
+    if (disable.getValue())
+      addUnitNode("llvm.loop.distribute.disable");
+    else
+      addUnitNode("llvm.loop.distribute.enable");
+  }
   convertFollowupNode("llvm.loop.distribute.followup_coincident",
                       options.getFollowupCoincident());
   convertFollowupNode("llvm.loop.distribute.followup_sequential",
diff --git a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
index c09fe92dc9cdc..021818ded3d3c 100644
--- a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
+++ b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
@@ -219,7 +219,7 @@ end:
 }
 
 !1 = distinct !{!1, !2, !3, !4, !5, !6}
-!2 = !{!"llvm.loop.distribute.enable", i1 0}
+!2 = !{!"llvm.loop.distribute.disable"}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !9}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !9}
 !5 = !{!"llvm.loop.distribute.followup_fallback", !9}
diff --git a/mlir/test/Target/LLVMIR/loop-metadata.mlir b/mlir/test/Target/LLVMIR/loop-metadata.mlir
index 72f0cf06ed0fe..299bcb481f67f 100644
--- a/mlir/test/Target/LLVMIR/loop-metadata.mlir
+++ b/mlir/test/Target/LLVMIR/loop-metadata.mlir
@@ -185,7 +185,7 @@ llvm.func @distributeOptions() {
 // CHECK-DAG: ![[NON_FORCED:[0-9]+]] = !{!"llvm.loop.disable_nonforced"}
 // CHECK-DAG: ![[FOLLOWUP:[0-9]+]] = distinct !{![[FOLLOWUP]], ![[NON_FORCED]]}
 // CHECK-DAG: ![[LOOP_NODE]] = distinct !{![[LOOP_NODE]], !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}}
-// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.disable"}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_coincident", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_sequential", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_fallback", ![[FOLLOWUP]]}

>From beb17d017d5a38a3226afd1a8997880d3b374093 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Tue, 7 Jul 2026 00:52:57 -0700
Subject: [PATCH 2/3] fixup! Enforce single-operand form for
 llvm.loop.distribute metadata

auto-upgrade old two-operand llvm.loop.distribute.enable metadata
---
 llvm/include/llvm/IR/AutoUpgrade.h            |   7 ++-
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    |   3 ++
 llvm/lib/IR/AutoUpgrade.cpp                   |  47 ++++++++++++++++--
 .../Bitcode/upgrade-loop-distribute-enable.ll |  37 ++++++++++++++
 .../upgrade-loop-distribute-enable.ll.bc      | Bin 0 -> 2080 bytes
 llvm/test/Verifier/llvm.loop.distribute.ll    |  30 ++++-------
 6 files changed, 101 insertions(+), 23 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc

diff --git a/llvm/include/llvm/IR/AutoUpgrade.h b/llvm/include/llvm/IR/AutoUpgrade.h
index feb19e69389a9..423fea4205a80 100644
--- a/llvm/include/llvm/IR/AutoUpgrade.h
+++ b/llvm/include/llvm/IR/AutoUpgrade.h
@@ -112,7 +112,12 @@ namespace llvm {
 
   /// Check whether a string looks like an old loop attachment tag.
   inline bool mayBeOldLoopAttachmentTag(StringRef Name) {
-    return Name.starts_with("llvm.vectorizer.");
+    // "llvm.loop.distribute.enable" is intentionally included: the current
+    // single-operand form shares the tag with the removed two-operand form
+    // (!{!"llvm.loop.distribute.enable", i1 X}), so we can only decide by
+    // inspecting the operands, which happens in upgradeLoopArgument().
+    return Name.starts_with("llvm.vectorizer.") ||
+           Name == "llvm.loop.distribute.enable";
   }
 
   /// Upgrade the loop attachment metadata node.
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 33062a0256bb8..5c4f8cb9fbeb1 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -2455,6 +2455,9 @@ Error MetadataLoader::MetadataLoaderImpl::parseOneMetadata(
   }
   case bitc::METADATA_STRINGS: {
     auto CreateNextMDString = [&](StringRef Str) {
+      // Modern bitcode encodes MDStrings via this bulk record, so mirror the
+      // METADATA_STRING check above to arm the loop-attachment upgrader.
+      HasSeenOldLoopTags |= mayBeOldLoopAttachmentTag(Str);
       ++NumMDStringLoaded;
       MetadataList.assignValue(MDString::get(Context, Str), NextMetadataNo);
       NextMetadataNo++;
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 229c7757db3dc..2cd42c0822b6c 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6847,6 +6847,18 @@ void llvm::copyModuleAttrToFunctions(Module &M) {
   }
 }
 
+// Old two-operand form: !{!"llvm.loop.distribute.enable", i1 X}. The new
+// single-operand form uses "llvm.loop.distribute.enable" for X = true and
+// "llvm.loop.distribute.disable" for X = false.
+static bool isOldDistributeEnable(const MDTuple *T) {
+  if (T->getNumOperands() != 2)
+    return false;
+  auto *Tag = dyn_cast_or_null<MDString>(T->getOperand(0));
+  if (!Tag || Tag->getString() != "llvm.loop.distribute.enable")
+    return false;
+  return mdconst::hasa<ConstantInt>(T->getOperand(1));
+}
+
 static bool isOldLoopArgument(Metadata *MD) {
   auto *T = dyn_cast_or_null<MDTuple>(MD);
   if (!T)
@@ -6856,7 +6868,9 @@ static bool isOldLoopArgument(Metadata *MD) {
   auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!S)
     return false;
-  return S->getString().starts_with("llvm.vectorizer.");
+  if (S->getString().starts_with("llvm.vectorizer."))
+    return true;
+  return isOldDistributeEnable(T);
 }
 
 static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
@@ -6880,17 +6894,28 @@ static Metadata *upgradeLoopArgument(Metadata *MD) {
   auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!OldTag)
     return MD;
+
+  LLVMContext &C = T->getContext();
+
+  // Rewrite the old two-operand distribute form to the single-operand pair.
+  if (isOldDistributeEnable(T)) {
+    bool Enable = !mdconst::extract<ConstantInt>(T->getOperand(1))->isZero();
+    return MDTuple::get(
+        C, {MDString::get(C, Enable ? "llvm.loop.distribute.enable"
+                                    : "llvm.loop.distribute.disable")});
+  }
+
   if (!OldTag->getString().starts_with("llvm.vectorizer."))
     return MD;
 
   // This has an old tag.  Upgrade it.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
-  Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
+  Ops.push_back(upgradeLoopTag(C, OldTag->getString()));
   for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
     Ops.push_back(T->getOperand(I));
 
-  return MDTuple::get(T->getContext(), Ops);
+  return MDTuple::get(C, Ops);
 }
 
 MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
@@ -6901,6 +6926,22 @@ MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
   if (none_of(T->operands(), isOldLoopArgument))
     return &N;
 
+  // Fix the old two-operand llvm.loop.distribute.enable nodes in place: the
+  // Verifier rejects any MDNode carrying the distribute tag with more than one
+  // operand, so a leftover reference (from the distinct loop-ID) would still
+  // trigger a diagnostic. In-place mutation is safe on distinct MDNodes.
+  if (T->isDistinct()) {
+    for (unsigned I = 0, E = T->getNumOperands(); I < E; ++I) {
+      auto *OpT = dyn_cast_or_null<MDTuple>(T->getOperand(I));
+      if (OpT && isOldDistributeEnable(OpT))
+        T->replaceOperandWith(I, upgradeLoopArgument(OpT));
+    }
+    if (none_of(T->operands(), isOldLoopArgument))
+      return &N;
+  }
+
+  // Remaining old arguments (e.g. llvm.vectorizer.*) are handled via a wrapper
+  // attachment; the original distinct loop-ID is kept as the first operand.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
   for (Metadata *MD : T->operands())
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
new file mode 100644
index 0000000000000..8f1b5aec2d140
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
@@ -0,0 +1,37 @@
+; Test that older bitcode carrying the two-operand form of
+; "llvm.loop.distribute.enable" is auto-upgraded to the single-operand
+; enable/disable pair on load.
+;
+; RUN: llvm-dis < %s.bc | FileCheck %s
+; RUN: verify-uselistorder < %s.bc
+
+define void @enable_true() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+
+define void @enable_false() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !2
+exit:
+  ret void
+}
+
+; i1 true  -> single-operand enable.
+; i1 false -> disable.
+; CHECK: !{!"llvm.loop.distribute.enable"}
+; CHECK: !{!"llvm.loop.distribute.disable"}
+; The old two-operand nodes must be gone from the module.
+; CHECK-NOT: llvm.loop.distribute.enable", i1
+; CHECK-NOT: llvm.loop.distribute.disable", i1
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = distinct !{!2, !3}
+!3 = !{!"llvm.loop.distribute.enable", i1 false}
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..fdcbda4584522c06b5f5668e53c6c24fede99c65
GIT binary patch
literal 2080
zcmbtVZ%iBK8GntzXA*N~5;pAQ&btdVved*Jh87#!lI=^GD`lIT<^$cbg)_DTnFGcc
z0;J3A*yG4*b#CQlmX=JQPF1UZAuS(Jix$i|%t at _`rt5?n%Ixwd$wUH4*i>zqwC7B>
zec89Y(!F=D?!D)p-=F7so!sp5b2SK6AcQ<Bw*EV>{O;|a{A22S`*Zb2WX1}_s1W)@
zhS2UZ29?2{I=HZC-5ZWv`HKp*(fp<?u6m|IsnwOg=833YD0BY2E?!?7&(U6eP|aSd
z at p{#7blh-`s;VnrH+kc(FO$^=T*o3REpw^Nn^RWTo9%6*>W^hcV_a!G?mkYxHEnMT
zMO9kkC~@goqgSJS5jT3X?V%)7*Oq)nO|^z|W3MNfBcq9OV>3b*X~+$7I^Zu?zwf+o
zLwyXUWZ#_IwX5<VH(%j29#pFK$@C6I+1qGcDM3K{n4@@i@%s5+h#R$k|H5$XbKdx&
zfw;C9IQLP-udaQz*7(os{yTRXMuxxp)5=e`AN~5o`Ok`{hSMi{uKu}q=(Wqg`E;$j
z at 5+yQe)$F8_|b;v1EGEPfE4!M;}BxA-`4<V{}~-Jbg`=Jh4<tLy>H*)+NevbwADwG
ziOPfDo-gN4S0xa-szGQAK59}%$ZV1H*T`&RoGC6QoDpnz2<uPxsn!wkK#gxl at Fta0
zH<zJqO^|-JqO>ixKQFiCI9oz)%MT^OLF}9o8+Hl7AT|<|4-N at ITF~aJ at r@*2WQiV@
zSlPn2+<22ZOD!|xB16sbIzM~SG83~baONqwbtYzw%Psk>n!yh2To=~TC4^v(2*IQf
z?7#-~*x*+c5cX_DIH?r+G(spX45y=q9&X_e3;17W at eNk)$xV=}6J)+LFY)9}hRjB(
zo4n#rNzR<(%<0lREt*oCX-sZOi4E}6rxz at Llv^f6)$hB7V7m|s#=>3LP=^rK2;p%-
zd+TN5AtD~T at zo^0Vke3;z5ys`J++k4%{g?p8FDGATPl#NMd~jyonP~kX_9Nb9JA!z
z^<M+&5Y7%^L+O`}ZYK$e0o`fcO`g2RP|F$Wyi6x0%urjr;`q;Ero3pG<ZM95(>d!Z
zXBmT~n7F<RkP2rZA0;-T$3p4oi4`@zF^jLc at h5Cu=C(t(TC80xQp=ElOKX{o*`{Lq
z<Dx0Sne(D`I%Cd>fYF+FP4y`;$f^$rY!|e5=JAaV;z2>NMH6c*u|^YrbK^Z~e1*kf
z`J|w at U%*%0_*ZP*mm54e7o`@9wHpkz&QMu~0tN3e-}2K}EHj*KT5e9o%=z{iP#?gK
zK%Oq)Y*5gCd>7wjiDCi2Zzt}L!(T+1y#E~xu$)YTRV0SP3m|1Q?l2cbpw0$L<?Y#V
z>gEJhaOjpZq`$tC#fy@@1O!=QIVLus!TMdo$#LP;2z+}rC9 at _(Yf3aNaIIjcJTRtm
zG2d|VI!N_ahB_9l at gF_K;+r(FSx{`ciESwqzE)7I(?oFrz`kXV4R&F}N<n*U-?V4}
zAO4mz<$*)FIT^F&<yIgwC92x0=E1t)s0s3XRQFg0I?i8JfCC{8XwkIid5sW^U?cBf
z!+MY`GvzQ%$}KaZ>1mm5Mr=(%Ze2?>^|?$3?9Bs#nJP(re)MrE2qIQyrDA!)#-g#3
zQW8e+f4?3ZNk7UuTBjkgnB?1Wt`({e+`9 at jK1?ID7wY&;or~tiRN6|fH+O at MLc?ln
zYug)jjwc^wb4s&wG~uzo3j6lM6$lKTF$8*hg9dMRe>l|b83_9fz8>Ba at WESz-i6-K
z0{7j!LVBn<Y@|Jy2Aw5|%nRa$b#i{9>YhVt3Bd7=m%rJ((t+P~Lx~WpQlSuc-T3xo
zBr1_1Rq&7im?ZzU2QQ1JDH%YqrNpuyLo*<kisTYauJY1+FRGj4$%P3rw_`d!BO|s!
zD9|l}?;hI`zv2%jka`gcS_JD^!0FJ3a&pTdJ^1uCMB_6@@HOx<_^yK}!VG<4LGdI>
zJfQK-SzN;Si5ejV+_AhBSiKAlL2et5S)n;3MX8<g`W;^Dl%rcLtbuB at H|}g_R9OWB
z-x`LZ>dAt=be7fH%Wp*}yQBH#_!Z<hkt>J2G7l=9VLuqP?}24${O_2a|0`x6y_1^}
zS09L at _v9L6E&c0<aF#v`PkZDkn}laa-jm_bfbZ)^KOgA#8Ji5Ofj!k7rkgxwqshoN
zHk&-Wzp<tBdA7;#?euw at S&ymN+wvUSWNh^CEyjjGZzmsU=;}S=YdFJuy9PphgJh_~
x!BFpMUuU?qXbAiI!wt^vaA&XA*Dw$~72>_V!;<9=|JQPdcM>rK0;P-*`WG%g;B5c^

literal 0
HcmV?d00001

diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
index 9e1e8138e93dc..65ca27720542b 100644
--- a/llvm/test/Verifier/llvm.loop.distribute.ll
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -1,8 +1,7 @@
 ; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
 ; single-operand validation.
 
-; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
-; DEFINE:   FileCheck %s -allow-empty -check-prefix
+; DEFINE: %{VERIFY} = llvm-as -disable-output %t 2>&1
 
 define void @test() {
 entry:
@@ -14,36 +13,29 @@ exit:
 }
 !0 = distinct !{!0, !1}
 
-; GOOD-NOT: {{.}}
-
 ;      BAD: Expected one operand for llvm.loop.distribute metadata
 
 ; Single-operand enable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Single-operand disable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Two-operand enable with boolean false (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand enable with boolean true (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand disable (rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD

>From e4f257b97ca79e0df10d1623f9462e7b390f369d Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Tue, 7 Jul 2026 09:47:40 -0700
Subject: [PATCH 3/3] fixup! LangRef changes

---
 llvm/docs/LangRef.rst | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst
index 9d37207753686..9e31caa1aa8b2 100644
--- a/llvm/docs/LangRef.rst
+++ b/llvm/docs/LangRef.rst
@@ -8385,17 +8385,16 @@ this is only performed if the entire loop cannot be vectorized due to unsafe
 memory dependencies.  The transformation will attempt to isolate the unsafe
 dependencies into their own loop.
 
-This metadata can be used to selectively enable or disable
-distribution of the loop.  Each node has a single operand
-containing the name string:
+This metadata can be used to selectively enable or disable distribution of the
+loop.  Each node has a single operand containing the name string:
 
 .. code-block:: llvm
 
    !0 = !{!"llvm.loop.distribute.enable"}
    !1 = !{!"llvm.loop.distribute.disable"}
 
-This metadata should be used in conjunction with ``llvm.loop``
-loop identification metadata.
+This metadata should be used in conjunction with ``llvm.loop`` loop
+identification metadata.
 
 '``llvm.loop.distribute.followup_coincident``' Metadata
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


From llvmlistbot at llvm.org  Tue Jul  7 09:51:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 09:51:21 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4d2e89.eb24c6ea.27aedc.6297@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp,h -- clang/lib/CodeGen/CGLoopInfo.cpp clang/test/CodeGenCXX/pragma-followup_inner.cpp clang/test/CodeGenCXX/pragma-followup_outer.cpp clang/test/CodeGenCXX/pragma-loop-distribute.cpp clang/test/CodeGenCXX/pragma-loop.cpp llvm/include/llvm/IR/AutoUpgrade.h llvm/lib/Analysis/LoopAccessAnalysis.cpp llvm/lib/Bitcode/Reader/MetadataLoader.cpp llvm/lib/IR/AutoUpgrade.cpp llvm/lib/IR/Verifier.cpp llvm/lib/Transforms/Scalar/LoopDistribute.cpp llvm/lib/Transforms/Utils/LoopConstrainer.cpp llvm/lib/Transforms/Utils/LoopUtils.cpp llvm/unittests/Analysis/LoopInfoTest.cpp mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 1ae6bbe0f..499e2287a 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -331,12 +331,9 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
-      NewLoopProperties.append(
-          LoopProperties.begin(), LoopProperties.end());
+      NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
       NewLoopProperties.push_back(MDNode::get(
-          Ctx,
-          {MDString::get(
-              Ctx, "llvm.loop.distribute.disable")}));
+          Ctx, {MDString::get(Ctx, "llvm.loop.distribute.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createLoopVectorizeMetadata(Attrs, LoopProperties,
@@ -345,15 +342,13 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
 
   bool FollowupHasTransforms = false;
   SmallVector<Metadata *, 4> Followup =
-      createLoopVectorizeMetadata(
-          Attrs, LoopProperties, FollowupHasTransforms);
+      createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
-  Args.push_back(MDNode::get(
-      Ctx,
-      {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
+  Args.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
 
   if (FollowupHasTransforms)
     Args.push_back(
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index d63208e2f..bd7884192 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2930,8 +2930,7 @@ void LoopAccessInfo::emitUnsafeDependenceRemark() {
 
   // Emit remark for first unsafe dependence
   bool HasForcedDistribution =
-      getBooleanLoopAttribute(
-          TheLoop, "llvm.loop.distribute.enable");
+      getBooleanLoopAttribute(TheLoop, "llvm.loop.distribute.enable");
 
   const std::string Info =
       HasForcedDistribution
diff --git a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
index 07cfd8628..19c6d2da6 100644
--- a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -943,8 +943,7 @@ private:
   void setForced() {
     if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
       IsForced = true;
-    else if (getBooleanLoopAttribute(
-                 L, "llvm.loop.distribute.disable"))
+    else if (getBooleanLoopAttribute(L, "llvm.loop.distribute.disable"))
       IsForced = false;
   }
 
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 74cd2a729..58ebd0e5e 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -452,9 +452,7 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
   MDNode *DisableLICMVersioning = MDNode::get(
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
-      Context,
-      {MDString::get(
-          Context, "llvm.loop.distribute.disable")});
+      Context, {MDString::get(Context, "llvm.loop.distribute.disable")});
   MDNode *NewLoopID =
       MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
                             DisableLICMVersioning, DisableDistribution});
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index 4022afc43..fcb11ca46 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -359,8 +359,7 @@ FailureOr<LoopLICMAttr> LoopMetadataConversion::convertLICMAttr() {
 
 FailureOr<LoopDistributeAttr> LoopMetadataConversion::convertDistributeAttr() {
   FailureOr<BoolAttr> disable = lookupBooleanUnitNode(
-      "llvm.loop.distribute.enable",
-      "llvm.loop.distribute.disable",
+      "llvm.loop.distribute.enable", "llvm.loop.distribute.disable",
       /*negated=*/true);
   FailureOr<LoopAnnotationAttr> followupCoincident =
       lookupFollowupNode("llvm.loop.distribute.followup_coincident");

``````````

</details>


https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Tue Jul  7 09:55:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 09:55:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][nvgpu] Add negative tests for invalid
 FP8 warpgroup MMA types (PR #208047)
Message-ID: <llvm.org/llvm/llvm-project/pull/208047@github.com>


https://github.com/Vaisman created https://github.com/llvm/llvm-project/pull/208047

Add negative tests for invalid FP8 type combinations in
nvgpu.warpgroup.mma.

FP8 inputs are supported for f16/f32 accumulators, but the verifier
should still reject unsupported combinations such as FP8 accumulators
and mixed FP8/non-FP8 input element types.

Tested:
  llvm-lit mlir/test/Dialect/NVGPU/invalid.mlir

>From a5371bfed110fea735f7ee892425331543a8fa5a Mon Sep 17 00:00:00 2001
From: Vasili Svirski <vasili.svirski at gmail.com>
Date: Mon, 6 Jul 2026 12:37:28 +0200
Subject: [PATCH] [mlir][nvgpu] Add negative tests for invalid FP8 warpgroup
 MMA types

Add verifier tests for unsupported FP8 type combinations in
nvgpu.warpgroup.mma.

FP8 inputs are supported for f16/f32 accumulators, but the verifier
should still reject unsupported combinations such as FP8 accumulators
and mixed FP8/non-FP8 input element types.

Tested:
  llvm-lit mlir/test/Dialect/NVGPU/invalid.mlir
---
 mlir/test/Dialect/NVGPU/invalid.mlir | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/mlir/test/Dialect/NVGPU/invalid.mlir b/mlir/test/Dialect/NVGPU/invalid.mlir
index 0341a08a2c778..49c471c881350 100644
--- a/mlir/test/Dialect/NVGPU/invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/invalid.mlir
@@ -465,3 +465,25 @@ func.func @warpgroup_mma_store_mismatched_shape(
       to memref<64x64xf32, 3>
   return
 }
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf8E4M3FN>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf8E4M3FN, 3>>
+func.func @warpgroup_mma_unsupported_fp8_accumulator(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f8E4M3FN' += 'f8E4M3FN' * 'f8E4M3FN'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf32>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf16, 3>>
+func.func @warpgroup_mma_unsupported_mixed_fp8_f16_inputs(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f32' += 'f8E4M3FN' * 'f16'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}


From llvmlistbot at llvm.org  Tue Jul  7 09:57:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 09:57:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][nvgpu] Add negative tests for invalid
 FP8 warpgroup MMA types (PR #208047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208047@github.com>
Message-ID: <6a4d2fdf.e9d484b9.15e549.2958@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-nvgpu

@llvm/pr-subscribers-mlir

Author: Vaisman

<details>
<summary>Changes</summary>

Add negative tests for invalid FP8 type combinations in
nvgpu.warpgroup.mma.

FP8 inputs are supported for f16/f32 accumulators, but the verifier
should still reject unsupported combinations such as FP8 accumulators
and mixed FP8/non-FP8 input element types.

Tested:
  llvm-lit mlir/test/Dialect/NVGPU/invalid.mlir

---
Full diff: https://github.com/llvm/llvm-project/pull/208047.diff


1 Files Affected:

- (modified) mlir/test/Dialect/NVGPU/invalid.mlir (+22) 


``````````diff
diff --git a/mlir/test/Dialect/NVGPU/invalid.mlir b/mlir/test/Dialect/NVGPU/invalid.mlir
index 0341a08a2c778..49c471c881350 100644
--- a/mlir/test/Dialect/NVGPU/invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/invalid.mlir
@@ -465,3 +465,25 @@ func.func @warpgroup_mma_store_mismatched_shape(
       to memref<64x64xf32, 3>
   return
 }
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf8E4M3FN>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf8E4M3FN, 3>>
+func.func @warpgroup_mma_unsupported_fp8_accumulator(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f8E4M3FN' += 'f8E4M3FN' * 'f8E4M3FN'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf32>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf16, 3>>
+func.func @warpgroup_mma_unsupported_mixed_fp8_f16_inputs(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f32' += 'f8E4M3FN' * 'f16'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208047

From llvmlistbot at llvm.org  Tue Jul  7 09:57:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 09:57:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][nvgpu] Add negative tests for invalid
 FP8 warpgroup MMA types (PR #208047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208047@github.com>
Message-ID: <6a4d2fe1.d1496011.26913c.503a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Vaisman

<details>
<summary>Changes</summary>

Add negative tests for invalid FP8 type combinations in
nvgpu.warpgroup.mma.

FP8 inputs are supported for f16/f32 accumulators, but the verifier
should still reject unsupported combinations such as FP8 accumulators
and mixed FP8/non-FP8 input element types.

Tested:
  llvm-lit mlir/test/Dialect/NVGPU/invalid.mlir

---
Full diff: https://github.com/llvm/llvm-project/pull/208047.diff


1 Files Affected:

- (modified) mlir/test/Dialect/NVGPU/invalid.mlir (+22) 


``````````diff
diff --git a/mlir/test/Dialect/NVGPU/invalid.mlir b/mlir/test/Dialect/NVGPU/invalid.mlir
index 0341a08a2c778..49c471c881350 100644
--- a/mlir/test/Dialect/NVGPU/invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/invalid.mlir
@@ -465,3 +465,25 @@ func.func @warpgroup_mma_store_mismatched_shape(
       to memref<64x64xf32, 3>
   return
 }
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf8E4M3FN>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf8E4M3FN, 3>>
+func.func @warpgroup_mma_unsupported_fp8_accumulator(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f8E4M3FN' += 'f8E4M3FN' * 'f8E4M3FN'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}
+
+// -----
+
+!tResult = !nvgpu.warpgroup.accumulator<fragmented = vector<128x128xf32>>
+!tDescA  = !nvgpu.warpgroup.descriptor<tensor = memref<128x64xf8E4M3FN, 3>>
+!tDescB  = !nvgpu.warpgroup.descriptor<tensor = memref<64x128xf16, 3>>
+func.func @warpgroup_mma_unsupported_mixed_fp8_f16_inputs(%descA: !tDescA, %descB: !tDescB, %acc: !tResult) {
+  // expected-error @+1 {{'f32' += 'f8E4M3FN' * 'f16'}}
+  %0 = nvgpu.warpgroup.mma %descA, %descB, %acc: !tDescA, !tDescB, !tResult -> !tResult
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208047

From llvmlistbot at llvm.org  Tue Jul  7 11:46:40 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 07 Jul 2026 11:46:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToAffine] Raise scf.for to
	affine.for (PR #200851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200851@github.com>
Message-ID: <6a4d4990.eb24c6ea.27aedc.9a05@mx.google.com>


joker-eph wrote:

I'd leave a few days for @ftynse to have an opportunity to acknowledge or yield.

https://github.com/llvm/llvm-project/pull/200851

From llvmlistbot at llvm.org  Tue Jul  7 12:49:28 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Tue, 07 Jul 2026 12:49:28 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][acc][flang] Add type sizing
	utilities (PR #208074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208074@github.com>
Message-ID: <6a4d5848.faf16b37.2f0e1d.e0ae@mx.google.com>


https://github.com/razvanlupusoru updated https://github.com/llvm/llvm-project/pull/208074

>From 537a535502bab6d9b3dd079a4e93a17927344e06 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Tue, 7 Jul 2026 12:06:23 -0700
Subject: [PATCH 1/3] [mlir][acc][flang] Add type sizing utilities

Add a general acc utility for computing the size and alignment
of a type. It works for simple scalar types as well as nested and
aggregate types like arrays, tuples, and structures.

Because some types come from other dialects, the utility can hand
those off to a specialized helper that understands them. This lets
sizing work seamlessly even for mixed types, such as an aggregate
whose members come from a different dialect.

Add a Fortran-specific helper so Fortran types are sized correctly,
falling back to the general utility for everything else.

Include unit tests covering a range of scenarios, including scalars,
arrays, aggregates, and mixed-dialect types.
---
 .../Analysis/FIROpenACCSupportAnalysis.h      |   5 +
 .../Optimizer/OpenACC/Analysis/CMakeLists.txt |   1 +
 .../Analysis/FIROpenACCSupportAnalysis.cpp    |  40 +++++
 flang/unittests/Optimizer/CMakeLists.txt      |   4 +
 .../OpenACC/FIROpenACCSupportAnalysisTest.cpp | 149 ++++++++++++++++++
 .../Dialect/OpenACC/Analysis/OpenACCSupport.h |  37 +++++
 .../mlir/Dialect/OpenACC/OpenACCUtilsType.h   |  56 +++++++
 .../OpenACC/Analysis/OpenACCSupport.cpp       |   1 +
 mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt |   1 +
 .../OpenACC/Utils/OpenACCUtilsType.cpp        |  94 +++++++++++
 mlir/unittests/Dialect/OpenACC/CMakeLists.txt |   1 +
 .../Dialect/OpenACC/OpenACCUtilsTypeTest.cpp  |  83 ++++++++++
 12 files changed, 472 insertions(+)
 create mode 100644 flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
 create mode 100644 mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
 create mode 100644 mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
 create mode 100644 mlir/unittests/Dialect/OpenACC/OpenACCUtilsTypeTest.cpp

diff --git a/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h b/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
index c2aaaee13419c..8d2a90d3520d1 100644
--- a/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
+++ b/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
@@ -13,6 +13,7 @@
 #ifndef FORTRAN_OPTIMIZER_OPENACC_ANALYSIS_FIROPENACCSUPPORTANALYSIS_H
 #define FORTRAN_OPTIMIZER_OPENACC_ANALYSIS_FIROPENACCSUPPORTANALYSIS_H
 
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/Region.h"
@@ -50,6 +51,10 @@ class FIROpenACCSupportAnalysis {
                         mlir::Operation **definingOpPtr);
 
   bool isValidValueUse(mlir::Value v, mlir::Region &region);
+
+  std::optional<mlir::acc::TypeSizeAndAlignment>
+  getTypeSizeAndAlignment(mlir::Type ty, mlir::ModuleOp module,
+                            mlir::acc::OpenACCSupport &support);
 };
 
 } // namespace acc
diff --git a/flang/lib/Optimizer/OpenACC/Analysis/CMakeLists.txt b/flang/lib/Optimizer/OpenACC/Analysis/CMakeLists.txt
index 22ed8ea246117..0eb387d58d1b9 100644
--- a/flang/lib/Optimizer/OpenACC/Analysis/CMakeLists.txt
+++ b/flang/lib/Optimizer/OpenACC/Analysis/CMakeLists.txt
@@ -13,6 +13,7 @@ add_flang_library(FIROpenACCAnalysis
   CUFAttrs
   CUFDialect
   FIRAnalysis
+  FIRCodeGen
   FIRDialect
   FIROpenACCSupport
   HLFIRDialect
diff --git a/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp b/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
index cf4c38453c8ee..df6202563a461 100644
--- a/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
+++ b/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
@@ -13,10 +13,16 @@
 #include "flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h"
 
 #include "flang/Optimizer/Builder/Todo.h"
+#include "flang/Optimizer/CodeGen/TypeConverter.h"
 #include "flang/Optimizer/Dialect/CUF/Attributes/CUFAttr.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/OpenACC/Support/FIROpenACCUtils.h"
+#include "mlir/Dialect/LLVMIR/LLVMTypes.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/IR/BuiltinOps.h"
 
 using namespace mlir;
 
@@ -86,5 +92,39 @@ bool FIROpenACCSupportAnalysis::isValidValueUse(Value v, Region &region) {
   return false;
 }
 
+std::optional<mlir::acc::TypeSizeAndAlignment>
+FIROpenACCSupportAnalysis::getTypeSizeAndAlignment(
+    Type ty, ModuleOp module, mlir::acc::OpenACCSupport &support) {
+  std::optional<DataLayout> dl = mlir::acc::getDataLayout(module);
+  if (!dl)
+    return std::nullopt;
+
+  if (isa<fir::ReferenceType, fir::HeapType, fir::LLVMPointerType>(ty)) {
+    return mlir::acc::getTypeSizeAndAlignment(
+        LLVM::LLVMPointerType::get(ty.getContext()), module, *dl, &support);
+  }
+
+  if (!fir::isa_fir_type(ty))
+    return mlir::acc::getTypeSizeAndAlignment(ty, module, *dl, &support);
+
+  fir::LLVMTypeConverter typeConverter(module, /*applyTBAA=*/false,
+                                       /*forceUnifiedTBAATree=*/false, *dl);
+  fir::KindMapping kindMap = typeConverter.getKindMap();
+
+  if (auto boxTy = dyn_cast<fir::BaseBoxType>(ty)) {
+    return mlir::acc::getTypeSizeAndAlignment(
+        typeConverter.convertBoxTypeAsStruct(boxTy), module, *dl, &support);
+  }
+
+  auto sizeAndAlignment = fir::getTypeSizeAndAlignment(
+      UnknownLoc::get(ty.getContext()), ty, *dl, kindMap);
+  if (!sizeAndAlignment)
+    return std::nullopt;
+
+  return mlir::acc::TypeSizeAndAlignment{
+      llvm::TypeSize::getFixed(sizeAndAlignment->first),
+      llvm::TypeSize::getFixed(sizeAndAlignment->second)};
+}
+
 } // namespace acc
 } // namespace fir
diff --git a/flang/unittests/Optimizer/CMakeLists.txt b/flang/unittests/Optimizer/CMakeLists.txt
index 3af6f9d2a3724..1697b64f12ed9 100644
--- a/flang/unittests/Optimizer/CMakeLists.txt
+++ b/flang/unittests/Optimizer/CMakeLists.txt
@@ -9,9 +9,11 @@ set(LIBS
   CUFDialect
   FIRBuilder
   FIRAnalysis
+  FIRCodeGen
   FIRCodeGenDialect
   FIRDialect
   FIRDialectSupport
+  FIROpenACCAnalysis
   FIROpenACCSupport
   FIRSupport
   FIRTransforms
@@ -37,6 +39,7 @@ add_flang_unittest(FlangOptimizerTests
   Builder/Runtime/StopTest.cpp
   Builder/Runtime/TransformationalTest.cpp
   OpenACC/FIROpenACCPointerLikeTypeInterfaceTest.cpp
+  OpenACC/FIROpenACCSupportAnalysisTest.cpp
   FIRCallInterfaceTest.cpp
   FIRContextTest.cpp
   FIRTypesTest.cpp
@@ -64,4 +67,5 @@ mlir_target_link_libraries(FlangOptimizerTests
   ${dialect_libs}
   ${extension_libs}
   MLIROpenACCDialect
+  MLIROpenACCUtils
   )
diff --git a/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp b/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
new file mode 100644
index 0000000000000..89454a48ede34
--- /dev/null
+++ b/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
@@ -0,0 +1,149 @@
+//===- FIROpenACCSupportAnalysisTest.cpp --------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h"
+#include "gtest/gtest.h"
+#include "mlir/Dialect/DLTI/DLTI.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/LLVMIR/LLVMTypes.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Interfaces/DataLayoutInterfaces.h"
+#include "flang/Optimizer/CodeGen/TypeConverter.h"
+#include "flang/Optimizer/Dialect/FIRDialect.h"
+#include "flang/Optimizer/Dialect/FIRType.h"
+#include "flang/Optimizer/Dialect/Support/FIRContext.h"
+#include "flang/Optimizer/Dialect/Support/KindMapping.h"
+
+using namespace mlir;
+
+namespace {
+
+struct FIROpenACCSupportAnalysisTest : public testing::Test {
+  void SetUp() override {
+    context.loadDialect<fir::FIROpsDialect, DLTIDialect, LLVM::LLVMDialect>();
+    kindMap = std::make_unique<fir::KindMapping>(&context);
+    module = ModuleOp::create(UnknownLoc::get(&context));
+    fir::setKindMapping(module, *kindMap);
+    support.setImplementation(fir::acc::FIROpenACCSupportAnalysis());
+  }
+
+  std::optional<acc::TypeSizeAndAlignment> getExpectedFIRSize(Type ty) {
+    std::optional<DataLayout> dl = acc::getDataLayout(module);
+    if (!dl) {
+      return std::nullopt;
+    }
+    fir::LLVMTypeConverter typeConverter(module, /*applyTBAA=*/false,
+        /*forceUnifiedTBAATree=*/false, *dl);
+    std::optional<std::pair<uint64_t, unsigned short>> sizeAndAlignment =
+        fir::getTypeSizeAndAlignment(
+            UnknownLoc::get(&context), ty, *dl, typeConverter.getKindMap());
+    if (!sizeAndAlignment) {
+      return std::nullopt;
+    }
+    return acc::TypeSizeAndAlignment{
+        llvm::TypeSize::getFixed(sizeAndAlignment->first),
+        llvm::TypeSize::getFixed(sizeAndAlignment->second)};
+  }
+
+  MLIRContext context;
+  std::unique_ptr<fir::KindMapping> kindMap;
+  ModuleOp module;
+  acc::OpenACCSupport support;
+};
+
+TEST_F(FIROpenACCSupportAnalysisTest, FIRLogicalScalarSizeAndAlignment) {
+  Type logicalTy = fir::LogicalType::get(&context, /*kind=*/4);
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(logicalTy, module);
+  std::optional<acc::TypeSizeAndAlignment> expected =
+      getExpectedFIRSize(logicalTy);
+  ASSERT_TRUE(result.has_value());
+  ASSERT_TRUE(expected.has_value());
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+TEST_F(FIROpenACCSupportAnalysisTest, FIRCharacterScalarSizeAndAlignment) {
+  Type charTy = fir::CharacterType::get(&context, /*kind=*/1, /*len=*/8);
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(charTy, module);
+  std::optional<acc::TypeSizeAndAlignment> expected =
+      getExpectedFIRSize(charTy);
+  ASSERT_TRUE(result.has_value());
+  ASSERT_TRUE(expected.has_value());
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+TEST_F(FIROpenACCSupportAnalysisTest, FIRReferenceTypeUsesPointerSize) {
+  Type logicalTy = fir::LogicalType::get(&context, /*kind=*/4);
+  Type refTy = fir::ReferenceType::get(logicalTy);
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(refTy, module);
+  LLVM::LLVMPointerType ptrTy = LLVM::LLVMPointerType::get(&context);
+  std::optional<acc::TypeSizeAndAlignment> expected =
+      acc::getTypeSizeAndAlignment(ptrTy, module);
+  ASSERT_TRUE(result.has_value());
+  ASSERT_TRUE(expected.has_value());
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+TEST_F(FIROpenACCSupportAnalysisTest, BuiltinTypeDelegatesToAccUtilities) {
+  Type i32 = IntegerType::get(&context, 32);
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(i32, module);
+  std::optional<acc::TypeSizeAndAlignment> expected =
+      acc::getTypeSizeAndAlignment(i32, module);
+  ASSERT_TRUE(result.has_value());
+  ASSERT_TRUE(expected.has_value());
+  DataLayout dl(module);
+  EXPECT_EQ(result->first, dl.getTypeSize(i32));
+  EXPECT_EQ(result->second.getFixedValue(), dl.getTypeABIAlignment(i32));
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+TEST_F(FIROpenACCSupportAnalysisTest, TupleWithFIRArrayMemberSizeAndAlignment) {
+  Type f32 = Float32Type::get(&context);
+  Type seqTy = fir::SequenceType::get({1024}, f32);
+  Type tupleTy = TupleType::get(&context, {seqTy});
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(tupleTy, module);
+  std::optional<acc::TypeSizeAndAlignment> expected = getExpectedFIRSize(seqTy);
+  ASSERT_TRUE(result.has_value());
+  ASSERT_TRUE(expected.has_value());
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+TEST_F(FIROpenACCSupportAnalysisTest, FIRBoxTypeSizeAndAlignment) {
+  Type f32 = Float32Type::get(&context);
+  Type seqTy = fir::SequenceType::get({4, 3}, f32);
+  Type boxTy = fir::BoxType::get(seqTy);
+  std::optional<acc::TypeSizeAndAlignment> result =
+      support.getTypeSizeAndAlignment(boxTy, module);
+  ASSERT_TRUE(result.has_value());
+
+  std::optional<DataLayout> dl = acc::getDataLayout(module);
+  ASSERT_TRUE(dl.has_value());
+  fir::LLVMTypeConverter typeConverter(module, /*applyTBAA=*/false,
+      /*forceUnifiedTBAATree=*/false, *dl);
+  Type structTy =
+      typeConverter.convertBoxTypeAsStruct(cast<fir::BaseBoxType>(boxTy));
+  std::optional<acc::TypeSizeAndAlignment> expected =
+      acc::getTypeSizeAndAlignment(structTy, module, *dl, &support);
+  ASSERT_TRUE(expected.has_value());
+  EXPECT_EQ(result->first, expected->first);
+  EXPECT_EQ(result->second, expected->second);
+}
+
+} // namespace
diff --git a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
index 8649160aa46d7..32fd27f427f32 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
+++ b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
@@ -53,12 +53,15 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Remarks.h"
 #include "mlir/IR/Value.h"
 #include "mlir/Pass/AnalysisManager.h"
 #include "llvm/ADT/StringRef.h"
 #include <functional>
 #include <memory>
+#include <optional>
 #include <string>
 
 namespace mlir {
@@ -101,6 +104,11 @@ struct OpenACCSupportTraits {
     /// Get or optionally create a GPU module in the given module.
     virtual std::optional<gpu::GPUModuleOp>
     getOrCreateGPUModule(ModuleOp mod, bool create, llvm::StringRef name) = 0;
+
+    /// Returns the size and ABI alignment in bytes for \p ty.
+    virtual std::optional<TypeSizeAndAlignment>
+    getTypeSizeAndAlignment(Type ty, ModuleOp module,
+                            OpenACCSupport &support) = 0;
   };
 
   /// SFINAE helpers to detect if implementation has optional methods
@@ -141,6 +149,16 @@ struct OpenACCSupportTraits {
       llvm::is_detected<getOrCreateGPUModule_t, ImplT, ModuleOp, bool,
                         llvm::StringRef>;
 
+  template <typename ImplT, typename... Args>
+  using getTypeSizeAndAlignment_t =
+      decltype(std::declval<ImplT>().getTypeSizeAndAlignment(
+          std::declval<Args>()...));
+
+  template <typename ImplT>
+  using has_getTypeSizeAndAlignment =
+      llvm::is_detected<getTypeSizeAndAlignment_t, ImplT, Type, ModuleOp,
+                        OpenACCSupport &>;
+
   /// This class wraps a concrete OpenACCSupport implementation and forwards
   /// interface calls to it. This provides type erasure, allowing different
   /// implementation types to be used interchangeably without inheritance.
@@ -197,6 +215,15 @@ struct OpenACCSupportTraits {
         return acc::getOrCreateGPUModule(mod, create, name);
     }
 
+    std::optional<TypeSizeAndAlignment>
+    getTypeSizeAndAlignment(Type ty, ModuleOp module,
+                            OpenACCSupport &support) final {
+      if constexpr (has_getTypeSizeAndAlignment<ImplT>::value)
+        return impl.getTypeSizeAndAlignment(ty, module, support);
+      else
+        return acc::getTypeSizeAndAlignment(ty, module, &support);
+    }
+
   private:
     ImplT impl;
   };
@@ -307,6 +334,16 @@ class OpenACCSupport {
   getOrCreateGPUModule(ModuleOp mod, bool create = true,
                        llvm::StringRef name = "");
 
+  /// Returns the size and ABI alignment in bytes for \p ty.
+  std::optional<TypeSizeAndAlignment> getTypeSizeAndAlignment(Type ty,
+                                                              ModuleOp module) {
+    if (impl) {
+      if (auto result = impl->getTypeSizeAndAlignment(ty, module, *this))
+        return result;
+    }
+    return acc::getTypeSizeAndAlignment(ty, module, this);
+  }
+
   /// Signal that this analysis should always be preserved so that
   /// underlying implementation registration is not lost.
   bool isInvalidated(const AnalysisManager::PreservedAnalyses &pa) {
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
new file mode 100644
index 0000000000000..6fac3ad715ae4
--- /dev/null
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
@@ -0,0 +1,56 @@
+//===- OpenACCUtilsType.h - OpenACC Type Utilities -------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines type utilities for OpenACC.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_OPENACC_OPENACCUTILSTYPE_H_
+#define MLIR_DIALECT_OPENACC_OPENACCUTILSTYPE_H_
+
+#include "llvm/Support/TypeSize.h"
+#include <optional>
+#include <utility>
+
+namespace mlir {
+class DataLayout;
+class ModuleOp;
+class Type;
+
+namespace acc {
+
+class OpenACCSupport;
+
+using TypeSizeAndAlignment = std::pair<llvm::TypeSize, llvm::TypeSize>;
+
+/// Returns the size and ABI alignment in bytes.
+///
+/// For aggregate structures and arrays, padding between members or elements is
+/// not taken into account. The result is a close estimate suitable for early
+/// OpenACC layout decisions, but not a complete ABI guarantee. For final size
+/// computations, use LLVM materialized types.
+///
+/// When \p support is provided, aggregate element types are sized by recursing
+/// through \p support so dialect-specific implementations can handle nested
+/// types.
+///
+/// Returns std::nullopt when the size is not statically computable or the type
+/// is not supported.
+std::optional<TypeSizeAndAlignment> getTypeSizeAndAlignment(
+    Type ty, ModuleOp module, const DataLayout &dl,
+    OpenACCSupport *support = nullptr);
+
+/// Same as above, obtaining \p dl from \p module via getDataLayout.
+std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignment(Type ty, ModuleOp module,
+                        OpenACCSupport *support = nullptr);
+
+} // namespace acc
+} // namespace mlir
+
+#endif // MLIR_DIALECT_OPENACC_OPENACCUTILSTYPE_H_
diff --git a/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp b/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
index 3875122152194..0f2f9d02ef2b3 100644
--- a/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
+++ b/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
@@ -13,6 +13,7 @@
 #include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
 
 namespace mlir {
 namespace acc {
diff --git a/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt b/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
index baf248fd9b2ae..67e1fc269ffa4 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
@@ -4,6 +4,7 @@ add_mlir_dialect_library(MLIROpenACCUtils
   OpenACCUtilsGPU.cpp
   OpenACCUtilsLoop.cpp
   OpenACCUtilsTiling.cpp
+  OpenACCUtilsType.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/OpenACC
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
new file mode 100644
index 0000000000000..ae43006c65a26
--- /dev/null
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
@@ -0,0 +1,94 @@
+//===- OpenACCUtilsType.cpp - OpenACC Type Utilities ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/Dialect/LLVMIR/LLVMTypes.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Interfaces/DataLayoutInterfaces.h"
+
+namespace mlir {
+namespace acc {
+
+static std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignmentHelper(Type ty, ModuleOp module, const DataLayout &dl,
+                              OpenACCSupport *support) {
+  if (support)
+    return support->getTypeSizeAndAlignment(ty, module);
+  return getTypeSizeAndAlignment(ty, module, dl);
+}
+
+std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignment(Type ty, ModuleOp module, const DataLayout &dl,
+                        OpenACCSupport *support) {
+  if (ty.isIntOrIndexOrFloat() ||
+      isa<ComplexType, VectorType, DataLayoutTypeInterface>(ty)) {
+    return TypeSizeAndAlignment{
+        dl.getTypeSize(ty),
+        llvm::TypeSize::getFixed(dl.getTypeABIAlignment(ty))};
+  }
+
+  // Product of element size and static dimensions; no inter-element padding or
+  // array alignment rules are applied. This is acceptable as per API
+  // documentation.
+  if (auto memrefTy = dyn_cast<MemRefType>(ty)) {
+    if (!memrefTy.hasStaticShape())
+      return std::nullopt;
+    auto elemSizeAndAlignment = getTypeSizeAndAlignmentHelper(
+        memrefTy.getElementType(), module, dl, support);
+    if (!elemSizeAndAlignment)
+      return std::nullopt;
+    int64_t totalSize = elemSizeAndAlignment->first.getFixedValue();
+    int64_t alignment = elemSizeAndAlignment->second.getFixedValue();
+    for (int64_t dim : memrefTy.getShape())
+      totalSize *= dim;
+    return TypeSizeAndAlignment{llvm::TypeSize::getFixed(totalSize),
+                                llvm::TypeSize::getFixed(alignment)};
+  }
+
+  // Sum of member sizes with no padding between members or tuple alignment
+  // rules applied. This is acceptable as per API documentation.
+  if (auto tupleTy = dyn_cast<TupleType>(ty)) {
+    if (tupleTy.size() == 0)
+      return std::nullopt;
+    auto sizeAndAlignment =
+        getTypeSizeAndAlignmentHelper(tupleTy.getType(0), module, dl, support);
+    if (!sizeAndAlignment)
+      return std::nullopt;
+    llvm::TypeSize size = sizeAndAlignment->first;
+    for (unsigned i = 1, e = tupleTy.size(); i < e; ++i) {
+      auto next = getTypeSizeAndAlignmentHelper(tupleTy.getType(i), module, dl,
+                                                support);
+      if (!next)
+        return std::nullopt;
+      size += next->first;
+    }
+    return TypeSizeAndAlignment{size, sizeAndAlignment->second};
+  }
+
+  if (isa<FunctionType>(ty)) {
+    return getTypeSizeAndAlignmentHelper(
+        LLVM::LLVMPointerType::get(ty.getContext()), module, dl, support);
+  }
+
+  return std::nullopt;
+}
+
+std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignment(Type ty, ModuleOp module, OpenACCSupport *support) {
+  std::optional<DataLayout> dl = getDataLayout(module);
+  if (!dl)
+    return std::nullopt;
+  return getTypeSizeAndAlignment(ty, module, *dl, support);
+}
+
+} // namespace acc
+} // namespace mlir
diff --git a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
index 7bcb652b69185..3a14d3e6796cb 100644
--- a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
+++ b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
@@ -6,6 +6,7 @@ add_mlir_unittest(MLIROpenACCTests
   OpenACCUtilsCGTest.cpp
   OpenACCUtilsGPUTest.cpp
   OpenACCUtilsTest.cpp
+  OpenACCUtilsTypeTest.cpp
   OpenACCUtilsTilingTest.cpp
   OpenACCUtilsLoopTest.cpp
 )
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTypeTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTypeTest.cpp
new file mode 100644
index 0000000000000..9b178688749c4
--- /dev/null
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTypeTest.cpp
@@ -0,0 +1,83 @@
+//===- OpenACCUtilsTypeTest.cpp - Unit tests for OpenACC type utilities ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/Dialect/DLTI/DLTI.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/OwningOpRef.h"
+#include "mlir/Interfaces/DataLayoutInterfaces.h"
+#include "gtest/gtest.h"
+
+using namespace mlir;
+using namespace mlir::acc;
+
+class OpenACCUtilsTypeTest : public ::testing::Test {
+protected:
+  OpenACCUtilsTypeTest() : b(&context), loc(UnknownLoc::get(&context)) {
+    context.loadDialect<DLTIDialect, func::FuncDialect, memref::MemRefDialect,
+                        LLVM::LLVMDialect>();
+  }
+
+  MLIRContext context;
+  OpBuilder b;
+  Location loc;
+};
+
+TEST_F(OpenACCUtilsTypeTest, IntegerTypeSizeAndAlignment) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  auto i32 = b.getI32Type();
+  auto result = getTypeSizeAndAlignment(i32, *module);
+  ASSERT_TRUE(result.has_value());
+  DataLayout dl(*module);
+  EXPECT_EQ(result->first, dl.getTypeSize(i32));
+  EXPECT_EQ(result->second.getFixedValue(), dl.getTypeABIAlignment(i32));
+}
+
+TEST_F(OpenACCUtilsTypeTest, StaticMemRefTypeSizeAndAlignment) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  auto f64 = b.getF64Type();
+  auto memrefTy = MemRefType::get({4, 3}, f64);
+  auto result = getTypeSizeAndAlignment(memrefTy, *module);
+  ASSERT_TRUE(result.has_value());
+  DataLayout dl(*module);
+  EXPECT_EQ(result->first,
+            llvm::TypeSize::getFixed(dl.getTypeSize(f64).getFixedValue() * 12));
+  EXPECT_EQ(result->second.getFixedValue(), dl.getTypeABIAlignment(f64));
+}
+
+TEST_F(OpenACCUtilsTypeTest, VectorTypeUsesDataLayout) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  auto vectorTy = VectorType::get({3}, b.getF32Type());
+  auto result = getTypeSizeAndAlignment(vectorTy, *module);
+  ASSERT_TRUE(result.has_value());
+  DataLayout dl(*module);
+  EXPECT_EQ(result->first, dl.getTypeSize(vectorTy));
+  EXPECT_EQ(result->second.getFixedValue(), dl.getTypeABIAlignment(vectorTy));
+}
+
+TEST_F(OpenACCUtilsTypeTest, FunctionTypeUsesPointerSize) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  auto funcTy = b.getFunctionType({}, {});
+  auto ptrResult = getTypeSizeAndAlignment(funcTy, *module);
+  auto ptrTy = mlir::LLVM::LLVMPointerType::get(&context);
+  auto directPtrResult = getTypeSizeAndAlignment(ptrTy, *module);
+  ASSERT_TRUE(ptrResult.has_value());
+  ASSERT_TRUE(directPtrResult.has_value());
+  EXPECT_EQ(ptrResult->first, directPtrResult->first);
+}
+
+TEST_F(OpenACCUtilsTypeTest, DynamicMemRefReturnsNullopt) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  auto memrefTy = MemRefType::get({ShapedType::kDynamic}, b.getI32Type());
+  EXPECT_FALSE(getTypeSizeAndAlignment(memrefTy, *module).has_value());
+}

>From 71f31c636bb0c637b25ddbc6ed3b987538851639 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Tue, 7 Jul 2026 12:24:43 -0700
Subject: [PATCH 2/3] Fix formatting

---
 .../Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h  | 2 +-
 mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h        | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h b/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
index 8d2a90d3520d1..29706edb9c637 100644
--- a/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
+++ b/flang/include/flang/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.h
@@ -54,7 +54,7 @@ class FIROpenACCSupportAnalysis {
 
   std::optional<mlir::acc::TypeSizeAndAlignment>
   getTypeSizeAndAlignment(mlir::Type ty, mlir::ModuleOp module,
-                            mlir::acc::OpenACCSupport &support);
+                          mlir::acc::OpenACCSupport &support);
 };
 
 } // namespace acc
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
index 6fac3ad715ae4..42915ac954e7d 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsType.h
@@ -41,9 +41,9 @@ using TypeSizeAndAlignment = std::pair<llvm::TypeSize, llvm::TypeSize>;
 ///
 /// Returns std::nullopt when the size is not statically computable or the type
 /// is not supported.
-std::optional<TypeSizeAndAlignment> getTypeSizeAndAlignment(
-    Type ty, ModuleOp module, const DataLayout &dl,
-    OpenACCSupport *support = nullptr);
+std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignment(Type ty, ModuleOp module, const DataLayout &dl,
+                        OpenACCSupport *support = nullptr);
 
 /// Same as above, obtaining \p dl from \p module via getDataLayout.
 std::optional<TypeSizeAndAlignment>

>From e74fb5e02e27f37b5b2045e4b3352f3588336bc8 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Tue, 7 Jul 2026 12:48:43 -0700
Subject: [PATCH 3/3] Fix braces to match style

---
 .../OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp          | 6 ++----
 .../Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp     | 6 ++----
 mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h | 3 +--
 mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp         | 6 ++----
 4 files changed, 7 insertions(+), 14 deletions(-)

diff --git a/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp b/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
index df6202563a461..62f3a7dd44a6f 100644
--- a/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
+++ b/flang/lib/Optimizer/OpenACC/Analysis/FIROpenACCSupportAnalysis.cpp
@@ -99,10 +99,9 @@ FIROpenACCSupportAnalysis::getTypeSizeAndAlignment(
   if (!dl)
     return std::nullopt;
 
-  if (isa<fir::ReferenceType, fir::HeapType, fir::LLVMPointerType>(ty)) {
+  if (isa<fir::ReferenceType, fir::HeapType, fir::LLVMPointerType>(ty))
     return mlir::acc::getTypeSizeAndAlignment(
         LLVM::LLVMPointerType::get(ty.getContext()), module, *dl, &support);
-  }
 
   if (!fir::isa_fir_type(ty))
     return mlir::acc::getTypeSizeAndAlignment(ty, module, *dl, &support);
@@ -111,10 +110,9 @@ FIROpenACCSupportAnalysis::getTypeSizeAndAlignment(
                                        /*forceUnifiedTBAATree=*/false, *dl);
   fir::KindMapping kindMap = typeConverter.getKindMap();
 
-  if (auto boxTy = dyn_cast<fir::BaseBoxType>(ty)) {
+  if (auto boxTy = dyn_cast<fir::BaseBoxType>(ty))
     return mlir::acc::getTypeSizeAndAlignment(
         typeConverter.convertBoxTypeAsStruct(boxTy), module, *dl, &support);
-  }
 
   auto sizeAndAlignment = fir::getTypeSizeAndAlignment(
       UnknownLoc::get(ty.getContext()), ty, *dl, kindMap);
diff --git a/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp b/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
index 89454a48ede34..d00aef28d290a 100644
--- a/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
+++ b/flang/unittests/Optimizer/OpenACC/FIROpenACCSupportAnalysisTest.cpp
@@ -37,17 +37,15 @@ struct FIROpenACCSupportAnalysisTest : public testing::Test {
 
   std::optional<acc::TypeSizeAndAlignment> getExpectedFIRSize(Type ty) {
     std::optional<DataLayout> dl = acc::getDataLayout(module);
-    if (!dl) {
+    if (!dl)
       return std::nullopt;
-    }
     fir::LLVMTypeConverter typeConverter(module, /*applyTBAA=*/false,
         /*forceUnifiedTBAATree=*/false, *dl);
     std::optional<std::pair<uint64_t, unsigned short>> sizeAndAlignment =
         fir::getTypeSizeAndAlignment(
             UnknownLoc::get(&context), ty, *dl, typeConverter.getKindMap());
-    if (!sizeAndAlignment) {
+    if (!sizeAndAlignment)
       return std::nullopt;
-    }
     return acc::TypeSizeAndAlignment{
         llvm::TypeSize::getFixed(sizeAndAlignment->first),
         llvm::TypeSize::getFixed(sizeAndAlignment->second)};
diff --git a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
index 32fd27f427f32..03bb6fb81d1ed 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
+++ b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
@@ -337,10 +337,9 @@ class OpenACCSupport {
   /// Returns the size and ABI alignment in bytes for \p ty.
   std::optional<TypeSizeAndAlignment> getTypeSizeAndAlignment(Type ty,
                                                               ModuleOp module) {
-    if (impl) {
+    if (impl)
       if (auto result = impl->getTypeSizeAndAlignment(ty, module, *this))
         return result;
-    }
     return acc::getTypeSizeAndAlignment(ty, module, this);
   }
 
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
index ae43006c65a26..5761e853c650c 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsType.cpp
@@ -30,11 +30,10 @@ std::optional<TypeSizeAndAlignment>
 getTypeSizeAndAlignment(Type ty, ModuleOp module, const DataLayout &dl,
                         OpenACCSupport *support) {
   if (ty.isIntOrIndexOrFloat() ||
-      isa<ComplexType, VectorType, DataLayoutTypeInterface>(ty)) {
+      isa<ComplexType, VectorType, DataLayoutTypeInterface>(ty))
     return TypeSizeAndAlignment{
         dl.getTypeSize(ty),
         llvm::TypeSize::getFixed(dl.getTypeABIAlignment(ty))};
-  }
 
   // Product of element size and static dimensions; no inter-element padding or
   // array alignment rules are applied. This is acceptable as per API
@@ -74,10 +73,9 @@ getTypeSizeAndAlignment(Type ty, ModuleOp module, const DataLayout &dl,
     return TypeSizeAndAlignment{size, sizeAndAlignment->second};
   }
 
-  if (isa<FunctionType>(ty)) {
+  if (isa<FunctionType>(ty))
     return getTypeSizeAndAlignmentHelper(
         LLVM::LLVMPointerType::get(ty.getContext()), module, dl, support);
-  }
 
   return std::nullopt;
 }


From llvmlistbot at llvm.org  Tue Jul  7 12:50:34 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588a.481afadf.196d57.0363@mx.google.com>


================

----------------
jopperm wrote:

Please add a `matmul` and `batch_matmul` case for completeness.

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:50:35 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588b.58bdd12c.19c81.c6da@mx.google.com>


================

----------------
jopperm wrote:

Keep comments intact (throughout the PR) when moving around code, please.

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:50:35 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588b.f8b79514.1d52d0.7b4b@mx.google.com>


================

----------------
jopperm wrote:

Neat, the scope of the utility makes sense to me (modulo the question below about the return type).

Nit: I don't think you necessarily have to move the implementation to a common place; why not keep it in `Specialize.cpp`, I believe then you wouldn't have to touch the enum and `matchOperandMap` at all.

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:50:35 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588b.da8bafd3.14cd50.0ef3@mx.google.com>


================

----------------
jopperm wrote:

In which situations do we create maps here that are different than the original ones?

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:50:38 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588e.f8b79514.1d52d0.7c52@mx.google.com>


================

----------------
jopperm wrote:

Please add a `matmul` and `batch_matmul` case for completeness.

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:50:38 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:50:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d588e.bb985344.1e9879.c94c@mx.google.com>


================

----------------
jopperm wrote:

Nit: Braces unnecessary (also below). 

https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:51:28 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Tue, 07 Jul 2026 12:51:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add category-to-named specialization to
	linalg-morph-ops (PR #190116)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/190116@github.com>
Message-ID: <6a4d58c0.491d90f5.2facd.53bf@mx.google.com>


https://github.com/jopperm deleted https://github.com/llvm/llvm-project/pull/190116

From llvmlistbot at llvm.org  Tue Jul  7 12:52:06 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Tue, 07 Jul 2026 12:52:06 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][acc][flang] Add type sizing
	utilities (PR #208074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208074@github.com>
Message-ID: <6a4d58e6.8b87e04f.184605.37f4@mx.google.com>


================
@@ -0,0 +1,94 @@
+//===- OpenACCUtilsType.cpp - OpenACC Type Utilities ----------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/Dialect/LLVMIR/LLVMTypes.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Interfaces/DataLayoutInterfaces.h"
+
+namespace mlir {
+namespace acc {
+
+static std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignmentHelper(Type ty, ModuleOp module, const DataLayout &dl,
+                              OpenACCSupport *support) {
+  if (support)
+    return support->getTypeSizeAndAlignment(ty, module);
+  return getTypeSizeAndAlignment(ty, module, dl);
+}
+
+std::optional<TypeSizeAndAlignment>
+getTypeSizeAndAlignment(Type ty, ModuleOp module, const DataLayout &dl,
+                        OpenACCSupport *support) {
+  if (ty.isIntOrIndexOrFloat() ||
+      isa<ComplexType, VectorType, DataLayoutTypeInterface>(ty)) {
+    return TypeSizeAndAlignment{
+        dl.getTypeSize(ty),
+        llvm::TypeSize::getFixed(dl.getTypeABIAlignment(ty))};
+  }
+
+  // Product of element size and static dimensions; no inter-element padding or
+  // array alignment rules are applied. This is acceptable as per API
+  // documentation.
+  if (auto memrefTy = dyn_cast<MemRefType>(ty)) {
+    if (!memrefTy.hasStaticShape())
+      return std::nullopt;
+    auto elemSizeAndAlignment = getTypeSizeAndAlignmentHelper(
+        memrefTy.getElementType(), module, dl, support);
+    if (!elemSizeAndAlignment)
+      return std::nullopt;
+    int64_t totalSize = elemSizeAndAlignment->first.getFixedValue();
+    int64_t alignment = elemSizeAndAlignment->second.getFixedValue();
+    for (int64_t dim : memrefTy.getShape())
+      totalSize *= dim;
+    return TypeSizeAndAlignment{llvm::TypeSize::getFixed(totalSize),
+                                llvm::TypeSize::getFixed(alignment)};
+  }
+
+  // Sum of member sizes with no padding between members or tuple alignment
+  // rules applied. This is acceptable as per API documentation.
+  if (auto tupleTy = dyn_cast<TupleType>(ty)) {
+    if (tupleTy.size() == 0)
+      return std::nullopt;
+    auto sizeAndAlignment =
+        getTypeSizeAndAlignmentHelper(tupleTy.getType(0), module, dl, support);
+    if (!sizeAndAlignment)
+      return std::nullopt;
+    llvm::TypeSize size = sizeAndAlignment->first;
+    for (unsigned i = 1, e = tupleTy.size(); i < e; ++i) {
+      auto next = getTypeSizeAndAlignmentHelper(tupleTy.getType(i), module, dl,
+                                                support);
+      if (!next)
+        return std::nullopt;
+      size += next->first;
+    }
+    return TypeSizeAndAlignment{size, sizeAndAlignment->second};
+  }
+
+  if (isa<FunctionType>(ty)) {
+    return getTypeSizeAndAlignmentHelper(
+        LLVM::LLVMPointerType::get(ty.getContext()), module, dl, support);
+  }
----------------
razvanlupusoru wrote:

Done. Thank you :)

https://github.com/llvm/llvm-project/pull/208074

From llvmlistbot at llvm.org  Tue Jul  7 12:52:33 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Tue, 07 Jul 2026 12:52:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][acc][flang] Add type sizing
	utilities (PR #208074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208074@github.com>
Message-ID: <6a4d5901.2020f6c2.243e78.a01f@mx.google.com>


razvanlupusoru wrote:

> Looks great!

Thank you for the review.

https://github.com/llvm/llvm-project/pull/208074

From llvmlistbot at llvm.org  Tue Jul  7 16:49:15 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Tue, 07 Jul 2026 16:49:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for working with
	acc par dims (PR #208120)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208120@github.com>
Message-ID: <6a4d907b.92ea0e2a.13fd37.3c33@mx.google.com>


https://github.com/vzakhari approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/208120

From llvmlistbot at llvm.org  Tue Jul  7 17:10:32 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Tue, 07 Jul 2026 17:10:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add non-splat constant
 distribution in SgToLane pass (PR #205575)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205575@github.com>
Message-ID: <6a4d9578.70920dee.290dca.dcc5@mx.google.com>


https://github.com/nbpatel updated https://github.com/llvm/llvm-project/pull/205575

>From 0b3f5844be4ec0ca4d1d0fac641c4b299536d1ed Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 22 Jun 2026 21:15:54 +0000
Subject: [PATCH 1/3] Add non-splat constant distribution for SgToLane

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 68 ++++++++++++++++---
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 20 ++++++
 2 files changed, 79 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 75a87f84b3da8..25fcd6054e569 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -355,6 +355,13 @@ struct SgToLaneElementWise : public ConversionPattern {
 
 /// Distributes a subgroup-level arith ConstantOp to lane-level arith
 /// ConstantOp.
+///
+/// Splat constants are distributed by simply rebuilding the splat with the
+/// lane-local vector type. Non-splat dense constants are distributed as :
+/// `computeDistributedCoords` yields the coordinates each lane owns, each
+/// element is extracted from the full (subgroup-level) constant, and the
+/// per-lane elements are assembled into the distributed vector with
+/// `vector.from_elements` (or `vector.broadcast` for a single element).
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -365,11 +372,11 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
     if (!resultType)
       return failure();
 
-    // Only handle dense vector constants
-    auto dense = dyn_cast<SplatElementsAttr>(op.getValue());
-    if (!dense)
+    // Only handle dense vector constants.
+    auto denseAttr = dyn_cast<DenseElementsAttr>(op.getValue());
+    if (!denseAttr)
       return rewriter.notifyMatchFailure(
-          op, "only dense splat vector constants are supported");
+          op, "only dense vector constants are supported");
 
     xegpu::DistributeLayoutAttr layout =
         xegpu::getTemporaryLayout(llvm::cast<OpResult>(op.getResult()));
@@ -385,12 +392,55 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
           op, "unable to compute lane vector type from the layout");
 
     VectorType newResultType = laneShapeOrFailure.value();
-    auto sclarValue = dense.getSplatValue<Attribute>();
-    auto newDenseAttr = DenseElementsAttr::get(newResultType, sclarValue);
+    Location loc = op.getLoc();
 
-    auto newOp = arith::ConstantOp::create(rewriter, op.getLoc(), newResultType,
-                                           newDenseAttr);
-    rewriter.replaceOp(op, newOp.getResult());
+    // Splat constants: every lane gets the same value, so just rebuild the
+    // splat with the distributed type.
+    if (denseAttr.isSplat()) {
+      auto scalarValue = denseAttr.getSplatValue<Attribute>();
+      auto newDenseAttr = DenseElementsAttr::get(newResultType, scalarValue);
+      auto newOp =
+          arith::ConstantOp::create(rewriter, loc, newResultType, newDenseAttr);
+      rewriter.replaceOp(op, newOp.getResult());
+      return success();
+    }
+
+    // Non-splat constants: each lane extracts the elements it owns from the
+    // full constant using the distributed coordinates from the layout.
+    auto fullConst =
+        arith::ConstantOp::create(rewriter, loc, resultType, denseAttr);
+
+    Value laneId = gpu::LaneIdOp::create(rewriter, loc, rewriter.getIndexType(),
+                                         /*upperBound=*/mlir::IntegerAttr());
+    auto maybeCoordsVec = layout.computeDistributedCoords(
+        rewriter, loc, laneId, resultType.getShape());
+    if (failed(maybeCoordsVec))
+      return rewriter.notifyMatchFailure(
+          op, "failed to compute distributed coordinates from layout");
+
+    SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
+    int64_t numElements = newResultType.getNumElements();
+    assert(static_cast<int64_t>(coordsVec.size()) == numElements &&
+           "number of coordinate sets must match number of distributed "
+           "elements");
+
+    SmallVector<Value> elements;
+    for (auto &coords : coordsVec) {
+      SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
+      elements.push_back(vector::ExtractOp::create(
+          rewriter, loc, fullConst.getResult(), mixedPos));
+    }
+
+    // Assemble the distributed vector.
+    Value result;
+    if (numElements == 1) {
+      result = vector::BroadcastOp::create(rewriter, loc, newResultType,
+                                           elements[0]);
+    } else {
+      result = vector::FromElementsOp::create(rewriter, loc, newResultType,
+                                              elements);
+    }
+    rewriter.replaceOp(op, result);
     return success();
   }
 };
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fcc2da3d5005b..61f4cc5ceb675 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -160,6 +160,26 @@ gpu.func @arith_constant() {
   gpu.return
 }
 
+// A non-splat constant is distributed by extracting the element each lane owns
+// from the full constant (using the layout's distributed coordinates).
+// CHECK-LABEL: gpu.func @arith_constant_non_splat
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ELEM:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<16xindex>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[ELEM]] : index to vector<1xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
+    }> : vector<16xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>

>From 61cfd91883db7075b92c45d08c66e3e1fdf28e7b Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 30 Jun 2026 14:47:29 +0000
Subject: [PATCH 2/3] Address feedback

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 42 +++++++++++++++----
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 25 ++++++++++-
 2 files changed, 57 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 25fcd6054e569..4cd242b5ddca5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -10,6 +10,7 @@
 #include "mlir/Dialect/Math/IR/Math.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/XeGPU/IR/XeGPU.h"
 #include "mlir/Dialect/XeGPU/Transforms/Passes.h"
@@ -356,12 +357,10 @@ struct SgToLaneElementWise : public ConversionPattern {
 /// Distributes a subgroup-level arith ConstantOp to lane-level arith
 /// ConstantOp.
 ///
-/// Splat constants are distributed by simply rebuilding the splat with the
-/// lane-local vector type. Non-splat dense constants are distributed as :
-/// `computeDistributedCoords` yields the coordinates each lane owns, each
-/// element is extracted from the full (subgroup-level) constant, and the
-/// per-lane elements are assembled into the distributed vector with
-/// `vector.from_elements` (or `vector.broadcast` for a single element).
+/// Splat constants are rebuilt with the lane-local vector type. Non-splat
+/// constants are distributed by extracting the elements each lane owns from
+/// the full constant and assembling them with vector.from_elements (or
+/// vector.broadcast for a single element).
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -420,12 +419,39 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
 
     SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
     int64_t numElements = newResultType.getNumElements();
-    assert(static_cast<int64_t>(coordsVec.size()) == numElements &&
+
+    // computeDistributedCoords returns the block start each lane owns. With
+    // all-ones lane_data the block is a single element. Otherwise expand each
+    // block start into its lane_data-sized element coordinates (row-major, to
+    // match vector.from_elements below).
+    SmallVector<int64_t> laneData = layout.getEffectiveLaneDataAsInt();
+    bool unitLaneData =
+        llvm::all_of(laneData, [](int64_t d) { return d == 1; });
+
+    SmallVector<SmallVector<Value>> elementCoords;
+    if (unitLaneData) {
+      elementCoords = std::move(coordsVec);
+    } else {
+      SmallVector<int64_t> unitTile(laneData.size(), 1);
+      for (const SmallVector<Value> &start : coordsVec) {
+        for (SmallVector<int64_t> off :
+             StaticTileOffsetRange(laneData, unitTile)) {
+          SmallVector<Value> coord(start.size());
+          for (size_t i = 0; i < start.size(); ++i)
+            coord[i] = arith::AddIOp::create(
+                rewriter, loc, start[i],
+                arith::ConstantIndexOp::create(rewriter, loc, off[i]));
+          elementCoords.push_back(std::move(coord));
+        }
+      }
+    }
+
+    assert(static_cast<int64_t>(elementCoords.size()) == numElements &&
            "number of coordinate sets must match number of distributed "
            "elements");
 
     SmallVector<Value> elements;
-    for (auto &coords : coordsVec) {
+    for (auto &coords : elementCoords) {
       SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
       elements.push_back(vector::ExtractOp::create(
           rewriter, loc, fullConst.getResult(), mixedPos));
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index 61f4cc5ceb675..fd16dc3cb6b4c 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -160,8 +160,8 @@ gpu.func @arith_constant() {
   gpu.return
 }
 
-// A non-splat constant is distributed by extracting the element each lane owns
-// from the full constant (using the layout's distributed coordinates).
+// Non-splat constant: each lane extracts the element it owns from the full
+// constant.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
@@ -180,6 +180,27 @@ gpu.func @arith_constant_non_splat() {
   gpu.return
 }
 
+// With lane_data > 1 each lane owns a lane_data-sized block, so multiple
+// elements are extracted and assembled with vector.from_elements.
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[RES:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_lane_data() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>
+    }> : vector<32xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>

>From 678f39ce0b51a9d996ea15f3bbd02dd459c934ec Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 6 Jul 2026 20:54:53 +0000
Subject: [PATCH 3/3] Address feedback

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 93 ++++++++++---------
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 52 ++++++++++-
 2 files changed, 95 insertions(+), 50 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 4cd242b5ddca5..0f06f16e2f8e7 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -358,9 +358,9 @@ struct SgToLaneElementWise : public ConversionPattern {
 /// ConstantOp.
 ///
 /// Splat constants are rebuilt with the lane-local vector type. Non-splat
-/// constants are distributed by extracting the elements each lane owns from
-/// the full constant and assembling them with vector.from_elements (or
-/// vector.broadcast for a single element).
+/// constants are distributed by extracting each lane_data-sized block from
+/// the full constant and inserting it at the correct position in the
+/// distributed vector using insert_strided_slice.
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -418,54 +418,55 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
           op, "failed to compute distributed coordinates from layout");
 
     SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
-    int64_t numElements = newResultType.getNumElements();
-
-    // computeDistributedCoords returns the block start each lane owns. With
-    // all-ones lane_data the block is a single element. Otherwise expand each
-    // block start into its lane_data-sized element coordinates (row-major, to
-    // match vector.from_elements below).
     SmallVector<int64_t> laneData = layout.getEffectiveLaneDataAsInt();
-    bool unitLaneData =
-        llvm::all_of(laneData, [](int64_t d) { return d == 1; });
-
-    SmallVector<SmallVector<Value>> elementCoords;
-    if (unitLaneData) {
-      elementCoords = std::move(coordsVec);
-    } else {
-      SmallVector<int64_t> unitTile(laneData.size(), 1);
-      for (const SmallVector<Value> &start : coordsVec) {
-        for (SmallVector<int64_t> off :
-             StaticTileOffsetRange(laneData, unitTile)) {
-          SmallVector<Value> coord(start.size());
-          for (size_t i = 0; i < start.size(); ++i)
-            coord[i] = arith::AddIOp::create(
-                rewriter, loc, start[i],
-                arith::ConstantIndexOp::create(rewriter, loc, off[i]));
-          elementCoords.push_back(std::move(coord));
-        }
+    ArrayRef<int64_t> distShape = newResultType.getShape();
+    int64_t rank = newResultType.getRank();
+
+    // Each lane owns one lane_data-sized block per distribution unit.
+    // computeDistributedCoords returns those block starts in row-major order
+    // over the block grid (distShape / laneData).
+    SmallVector<int64_t> blockGridShape(rank);
+    for (int64_t d = 0; d < rank; d++)
+      blockGridShape[d] = distShape[d] / laneData[d];
+    SmallVector<int64_t> blockGridStrides = computeStrides(blockGridShape);
+
+    auto blockType = VectorType::get(laneData, newResultType.getElementType());
+    SmallVector<int64_t> unitTile(rank, 1);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value result = arith::ConstantOp::create(
+        rewriter, loc, newResultType, rewriter.getZeroAttr(newResultType));
+
+    for (auto [blockIdx, blockStart] : llvm::enumerate(coordsVec)) {
+      // Gather the block's elements from the full constant. The block start is
+      // lane-dynamic, so extract element-by-element (row-major over lane_data)
+      // instead.
+      SmallVector<Value> blockElems;
+      for (SmallVector<int64_t> off :
+           StaticTileOffsetRange(laneData, unitTile)) {
+        SmallVector<OpFoldResult> pos(rank);
+        for (int64_t d = 0; d < rank; d++)
+          pos[d] = getAsOpFoldResult(arith::AddIOp::create(
+              rewriter, loc, blockStart[d],
+              arith::ConstantIndexOp::create(rewriter, loc, off[d])));
+        blockElems.push_back(vector::ExtractOp::create(
+            rewriter, loc, fullConst.getResult(), pos));
       }
-    }
-
-    assert(static_cast<int64_t>(elementCoords.size()) == numElements &&
-           "number of coordinate sets must match number of distributed "
-           "elements");
 
-    SmallVector<Value> elements;
-    for (auto &coords : elementCoords) {
-      SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
-      elements.push_back(vector::ExtractOp::create(
-          rewriter, loc, fullConst.getResult(), mixedPos));
+      // Rebuild the block keeping its lane_data shape, then place it with
+      // insert_strided_slice so the block keeps its orientation in the
+      // distributed vector (e.g. a [2, 1] block stays a vertical 2x1 slice).
+      Value block =
+          vector::FromElementsOp::create(rewriter, loc, blockType, blockElems);
+      SmallVector<int64_t> blockGridPos =
+          delinearize(blockIdx, blockGridStrides);
+      SmallVector<int64_t> offsets(rank);
+      for (int64_t d = 0; d < rank; d++)
+        offsets[d] = blockGridPos[d] * laneData[d];
+      result = vector::InsertStridedSliceOp::create(rewriter, loc, block,
+                                                    result, offsets, strides);
     }
 
-    // Assemble the distributed vector.
-    Value result;
-    if (numElements == 1) {
-      result = vector::BroadcastOp::create(rewriter, loc, newResultType,
-                                           elements[0]);
-    } else {
-      result = vector::FromElementsOp::create(rewriter, loc, newResultType,
-                                              elements);
-    }
     rewriter.replaceOp(op, result);
     return success();
   }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fd16dc3cb6b4c..fe356e6af35c1 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -161,12 +161,14 @@ gpu.func @arith_constant() {
 }
 
 // Non-splat constant: each lane extracts the element it owns from the full
-// constant.
+// constant and inserts its lane_data-sized block into the distributed vector.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<1xindex>
 // CHECK: %[[ELEM:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<16xindex>
-// CHECK: %[[BCAST:.*]] = vector.broadcast %[[ELEM]] : index to vector<1xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM]] : vector<1xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<1xindex> into vector<1xindex>
 // CHECK: gpu.return
 gpu.func @arith_constant_non_splat() {
   %0 = arith.constant
@@ -181,13 +183,16 @@ gpu.func @arith_constant_non_splat() {
 }
 
 // With lane_data > 1 each lane owns a lane_data-sized block, so multiple
-// elements are extracted and assembled with vector.from_elements.
+// elements are extracted, reassembled into the block, and placed with
+// insert_strided_slice.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<2xindex>
 // CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
 // CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
-// CHECK: %[[RES:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<2xindex> into vector<2xindex>
 // CHECK: gpu.return
 gpu.func @arith_constant_non_splat_lane_data() {
   %0 = arith.constant
@@ -201,6 +206,45 @@ gpu.func @arith_constant_non_splat_lane_data() {
   gpu.return
 }
 
+// 2D non-splat constant with vertical lane_data [2, 1]. The distributed type
+// is vector<4x2xindex>. Each [2, 1] block is correctly placed vertically in
+// the result (same column, adjacent rows).
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_2d_vertical_lanedata
+// CHECK: %[[CST:.*]] = arith.constant dense<{{.*}}> : vector<4x32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<4x2xindex>
+// CHECK: %[[E0:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E1:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B0:.*]] = vector.from_elements %[[E0]], %[[E1]] : vector<2x1xindex>
+// CHECK: %[[I0:.*]] = vector.insert_strided_slice %[[B0]], %[[ZERO]] {offsets = [0, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E2:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E3:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B1:.*]] = vector.from_elements %[[E2]], %[[E3]] : vector<2x1xindex>
+// CHECK: %[[I1:.*]] = vector.insert_strided_slice %[[B1]], %[[I0]] {offsets = [0, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E4:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E5:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B2:.*]] = vector.from_elements %[[E4]], %[[E5]] : vector<2x1xindex>
+// CHECK: %[[I2:.*]] = vector.insert_strided_slice %[[B2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E6:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E7:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B3:.*]] = vector.from_elements %[[E6]], %[[E7]] : vector<2x1xindex>
+// CHECK: %[[I3:.*]] = vector.insert_strided_slice %[[B3]], %[[I2]] {offsets = [2, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_2d_vertical_lanedata() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
+    dense<[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
+           [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63],
+           [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95],
+           [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127]]> : vector<4x32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>
+    }> : vector<4x32xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>


From llvmlistbot at llvm.org  Tue Jul  7 17:22:20 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Tue, 07 Jul 2026 17:22:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix blocking pass for scf.if
	distribution (PR #207060)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207060@github.com>
Message-ID: <6a4d983c.a87eb5dc.7d817.f2e9@mx.google.com>


https://github.com/nbpatel updated https://github.com/llvm/llvm-project/pull/207060

>From d81f090b2cb072e29317ebac50070010344492bd Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 30 Jun 2026 17:59:49 +0000
Subject: [PATCH 1/3] Fix blocking for scf.if with 1:N expanded results

---
 .../XeGPU/Transforms/XeGPUBlocking.cpp        |  2 +-
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   | 17 +++++++++
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   | 35 +++++++++++++++++++
 3 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
index 4f81085c21e86..57b8ac73f41ac 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
@@ -430,7 +430,7 @@ void XeGPUBlockingPass::runOnOperation() {
     // `layout_result_N` lands on the wrong (renumbered) result, corrupting the
     // count invariant and leaving the loop illegal.
     op->walk([](Operation *loopOp) {
-      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp>(loopOp))
+      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp, scf::IfOp>(loopOp))
         return;
       SmallVector<StringRef> toRemove;
       for (const NamedAttribute &attr : loopOp->getAttrs()) {
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 1aad2aa77741b..9ab0d55f51be4 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -924,6 +924,23 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
         recordTypes(init, {arg, res, yieldVal});
       return;
     }
+    if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
+      // scf.if is not loop-carried, but its results still expand 1:N during
+      // blocking. Record each result together with the then/else yield operands
+      // that feed it so they all convert to the same distributed type by Value
+      // identity. The yield operand's layout is the authoritative source (the
+      // result's own per-position layout attr is stripped before conversion).
+      scf::YieldOp thenYield = ifOp.thenYield();
+      scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
+      for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
+        Value thenVal = thenYield.getOperand(idx);
+        SmallVector<Value> dests{res, thenVal};
+        if (elseYield)
+          dests.push_back(elseYield.getOperand(idx));
+        recordTypes(thenVal, dests);
+      }
+      return;
+    }
   });
   return loopArgTypes;
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index f2c25f45f45bc..b94119fa33121 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -815,3 +815,38 @@ gpu.module @test_kernel {
     gpu.return
   }
 }
+
+// -----
+// Test that an scf.if with results that expand 1:N during blocking is
+// legalized. The two results use different layouts (a regular 2D layout that
+// expands 1:N and a slice layout), and the slice result is consumed by a
+// broadcast/transpose chain. This exercises the scf.if structural type
+// conversion, which previously failed to legalize because the per-position
+// layout attrs were not stripped and the results were not recorded in the
+// loop-arg type map.
+#if_a = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#if_b = #xegpu.slice<#xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>
+gpu.module @test_kernel {
+  // CHECK-LABEL: func @if_one_to_n_results
+  gpu.func @if_one_to_n_results(%c: i1, %p: i64) {
+    %c0 = arith.constant 0 : index
+    // CHECK: scf.if
+    // CHECK-COUNT-8: vector<8x16xf32>
+    %r:2 = scf.if %c -> (vector<16x64xf32>, vector<16xf32>) {
+      %a = arith.constant {layout_result_0 = #if_a} dense<1.0> : vector<16x64xf32>
+      %b = arith.constant {layout_result_0 = #if_b} dense<1.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    } else {
+      %a = arith.constant {layout_result_0 = #if_a} dense<0.0> : vector<16x64xf32>
+      %b = arith.constant {layout_result_0 = #if_b} dense<0.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    } {layout_result_0 = #if_a, layout_result_1 = #if_b}
+    %bc = vector.broadcast %r#1 {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<64x16xf32>
+    %tp = vector.transpose %bc, [1, 0] {layout_result_0 = #if_a} : vector<64x16xf32> to vector<16x64xf32>
+    %add = arith.addf %r#0, %tp {layout_result_0 = #if_a} : vector<16x64xf32>
+    %tr = arith.truncf %add {layout_result_0 = #if_a} : vector<16x64xf32> to vector<16x64xf16>
+    %td = xegpu.create_nd_tdesc %p, shape:[16, 64], strides:[64, 1] : i64 -> !xegpu.tensor_desc<16x64xf16, #if_a>
+    xegpu.store_nd %tr, %td[%c0, %c0] <{layout = #if_a}> : vector<16x64xf16>, !xegpu.tensor_desc<16x64xf16, #if_a>
+    gpu.return
+  }
+}

>From c5ceb9208dd9f0f99b42dd8d2e6d009d43a3753f Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Wed, 1 Jul 2026 17:18:20 +0000
Subject: [PATCH 2/3] Fix comments

---
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp |  8 +++---
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir | 27 ++++++++-------------
 2 files changed, 13 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9ab0d55f51be4..7faada2b8223f 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -925,11 +925,9 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
       return;
     }
     if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
-      // scf.if is not loop-carried, but its results still expand 1:N during
-      // blocking. Record each result together with the then/else yield operands
-      // that feed it so they all convert to the same distributed type by Value
-      // identity. The yield operand's layout is the authoritative source (the
-      // result's own per-position layout attr is stripped before conversion).
+      // scf.if results expand 1:N during blocking. Record each result with the
+      // then/else yield operands that feed it so they convert to the same
+      // distributed type by Value identity, using the yield operand's layout.
       scf::YieldOp thenYield = ifOp.thenYield();
       scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
       for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b94119fa33121..b0b16c2adba6e 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -817,15 +817,8 @@ gpu.module @test_kernel {
 }
 
 // -----
-// Test that an scf.if with results that expand 1:N during blocking is
-// legalized. The two results use different layouts (a regular 2D layout that
-// expands 1:N and a slice layout), and the slice result is consumed by a
-// broadcast/transpose chain. This exercises the scf.if structural type
-// conversion, which previously failed to legalize because the per-position
-// layout attrs were not stripped and the results were not recorded in the
-// loop-arg type map.
+// Test that an scf.if whose results expand 1:N during blocking is legalized.
 #if_a = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
-#if_b = #xegpu.slice<#xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>
 gpu.module @test_kernel {
   // CHECK-LABEL: func @if_one_to_n_results
   gpu.func @if_one_to_n_results(%c: i1, %p: i64) {
@@ -833,18 +826,18 @@ gpu.module @test_kernel {
     // CHECK: scf.if
     // CHECK-COUNT-8: vector<8x16xf32>
     %r:2 = scf.if %c -> (vector<16x64xf32>, vector<16xf32>) {
-      %a = arith.constant {layout_result_0 = #if_a} dense<1.0> : vector<16x64xf32>
-      %b = arith.constant {layout_result_0 = #if_b} dense<1.0> : vector<16xf32>
+      %a = arith.constant dense<1.0> : vector<16x64xf32>
+      %b = arith.constant dense<1.0> : vector<16xf32>
       scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
     } else {
-      %a = arith.constant {layout_result_0 = #if_a} dense<0.0> : vector<16x64xf32>
-      %b = arith.constant {layout_result_0 = #if_b} dense<0.0> : vector<16xf32>
+      %a = arith.constant dense<0.0> : vector<16x64xf32>
+      %b = arith.constant dense<0.0> : vector<16xf32>
       scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
-    } {layout_result_0 = #if_a, layout_result_1 = #if_b}
-    %bc = vector.broadcast %r#1 {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<64x16xf32>
-    %tp = vector.transpose %bc, [1, 0] {layout_result_0 = #if_a} : vector<64x16xf32> to vector<16x64xf32>
-    %add = arith.addf %r#0, %tp {layout_result_0 = #if_a} : vector<16x64xf32>
-    %tr = arith.truncf %add {layout_result_0 = #if_a} : vector<16x64xf32> to vector<16x64xf16>
+    }
+    %bc = vector.broadcast %r#1 : vector<16xf32> to vector<64x16xf32>
+    %tp = vector.transpose %bc, [1, 0] : vector<64x16xf32> to vector<16x64xf32>
+    %add = arith.addf %r#0, %tp : vector<16x64xf32>
+    %tr = arith.truncf %add : vector<16x64xf32> to vector<16x64xf16>
     %td = xegpu.create_nd_tdesc %p, shape:[16, 64], strides:[64, 1] : i64 -> !xegpu.tensor_desc<16x64xf16, #if_a>
     xegpu.store_nd %tr, %td[%c0, %c0] <{layout = #if_a}> : vector<16x64xf16>, !xegpu.tensor_desc<16x64xf16, #if_a>
     gpu.return

>From 5e3ca5318a8e9e62f3c3a6de2214102b54ac06a8 Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 6 Jul 2026 21:34:40 +0000
Subject: [PATCH 3/3] Address feedback

---
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 7faada2b8223f..9620e21f9bfdf 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -862,12 +862,13 @@ bool xegpu::matchSplitDimExpansion(
 //===----------------------------------------------------------------------===//
 
 // Pre-computes distributed VectorType mappings for every value carried through
-// an SCF loop (scf.while, scf.for): block args (iter_args /
-// before-/after-args), loop results, and the terminator operands feeding them.
+// an SCF region-branch op (scf.while, scf.for, scf.if): block args (iter_args /
+// before-/after-args), op results, and the terminator operands feeding them.
 // These positions share one logical value and must convert identically, so each
 // is derived from a single source -- the layout of the feeding value (loop
-// init, or `scf.condition` operand) -- via `getDistributeLayoutAttr(Value)`,
-// and keyed by `Value`. Keying by Value is required because the SCF converters
+// init, `scf.condition` operand, or `scf.if` result) -- via
+// `getDistributeLayoutAttr(Value)`, and keyed by `Value`. Keying by Value is
+// required because the SCF converters
 // detach/replace the loop body mid-conversion (scf.while detaches before/after
 // blocks -> a detached-arg layout query trips an ilist assertion; scf.for
 // rebuilds the op, which loses the temporary `layout_operand_N` attrs -> the
@@ -925,17 +926,15 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
       return;
     }
     if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
-      // scf.if results expand 1:N during blocking. Record each result with the
-      // then/else yield operands that feed it so they convert to the same
-      // distributed type by Value identity, using the yield operand's layout.
+      // Each result and its then/else yield operands share one position and
+      // must convert identically; derive all from the result's layout.
       scf::YieldOp thenYield = ifOp.thenYield();
       scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
       for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
-        Value thenVal = thenYield.getOperand(idx);
-        SmallVector<Value> dests{res, thenVal};
+        SmallVector<Value> dests{res, thenYield.getOperand(idx)};
         if (elseYield)
           dests.push_back(elseYield.getOperand(idx));
-        recordTypes(thenVal, dests);
+        recordTypes(res, dests);
       }
       return;
     }


From llvmlistbot at llvm.org  Tue Jul  7 17:26:23 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Tue, 07 Jul 2026 17:26:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix layout recovery for dead
 loop-carried values (PR #205884)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205884@github.com>
Message-ID: <6a4d992f.8b87e04f.184605.a25b@mx.google.com>


https://github.com/nbpatel closed https://github.com/llvm/llvm-project/pull/205884

From llvmlistbot at llvm.org  Tue Jul  7 18:05:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 18:05:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a4da271.da8bafd3.14cd50.3a13@mx.google.com>


1sgtpepper wrote:

@gysit gentle ping here. Thanks 

https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Tue Jul  7 18:08:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 07 Jul 2026 18:08:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4da314.f732ad81.3a4994.3263@mx.google.com>


================
@@ -771,6 +825,43 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   return success();
 }
 
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  return ::verifyBlockScaledTensorType(op, type, /*allowScaleValues=*/false);
+}
+
+size_t mlir::tosa::BlockScaledType::getDenseElementBitSize() const {
+  const Type valueType = getValueType();
+  if (isa<tosa::mxint8Type>(valueType))
+    return 8;
+  return valueType.getIntOrFloatBitWidth();
+}
+
+Attribute
+mlir::tosa::BlockScaledType::convertToAttribute(ArrayRef<char> rawData) const {
+  assert(rawData.size() == 1 && "expected 1 byte for block_scaled element");
----------------
sahas3 wrote:

is this because 8 bit data is 1 byte? a comment will help readability

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 18:08:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 07 Jul 2026 18:08:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4da314.13be6df8.145c.02bf@mx.google.com>


https://github.com/sahas3 approved this pull request.

LG, a few minor comments.

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 18:08:36 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 07 Jul 2026 18:08:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4da314.331483e7.2440f8.d4d9@mx.google.com>


https://github.com/sahas3 edited https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 18:08:37 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 07 Jul 2026 18:08:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4da315.58bdd12c.19c81.1596@mx.google.com>


================
@@ -771,6 +825,43 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   return success();
 }
 
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
+                                                         mlir::Type type) {
+  return ::verifyBlockScaledTensorType(op, type, /*allowScaleValues=*/false);
+}
+
+size_t mlir::tosa::BlockScaledType::getDenseElementBitSize() const {
+  const Type valueType = getValueType();
+  if (isa<tosa::mxint8Type>(valueType))
+    return 8;
+  return valueType.getIntOrFloatBitWidth();
+}
+
+Attribute
+mlir::tosa::BlockScaledType::convertToAttribute(ArrayRef<char> rawData) const {
+  assert(rawData.size() == 1 && "expected 1 byte for block_scaled element");
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertToAttribute(rawData);
+  if (!isa<FloatType>(valueType))
+    return {};
+  return mlir::detail::convertFloatTypeToAttribute(valueType, rawData);
+}
+
+LogicalResult mlir::tosa::BlockScaledType::convertFromAttribute(
+    Attribute attr, SmallVectorImpl<char> &result) const {
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertFromAttribute(attr, result);
+
+  const auto floatAttr = dyn_cast<FloatAttr>(attr);
+  if (!floatAttr || floatAttr.getType() != valueType)
+    return failure();
+  const APFloat value = floatAttr.getValue();
+  return mlir::detail::convertFloatTypeFromAttribute(
+      valueType, FloatAttr::get(valueType, value), result);
----------------
sahas3 wrote:

```suggestion
      valueType, floatAttr, result);
```

https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Tue Jul  7 19:26:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 19:26:53 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
Message-ID: <llvm.org/llvm/llvm-project/pull/208133@github.com>


https://github.com/agozillon created https://github.com/llvm/llvm-project/pull/208133

Whilst there is no data in theory to map with a zero sized array, the Fortran OpenMP specification considers the case of a zero sized array being mapped to device as the data being present on device. This includes both the data and the descriptor, and it should fundamentlaly work as a zero-sized array would on device with the respective Fortran functions for presence and size checking etc.

We can't actually map a nullptr to device and expect it to be present in current OpenMP (for good reason), however, thankfully, a zero sized array isn't actually a nullptr, it is a descriptor contianing an allocated 1-byte of data. So, we can map this to device, alongside the descriptor and then the zero sized array is correctlly on device for all intents and purposes. This case is notably different from a non-allocated or non-zero sized array so we can do this without shooting ourselves in the foot.

>From 563d1b9f3439389389c0bd6f8dfb2daf200ee765 Mon Sep 17 00:00:00 2001
From: agozillon <Andrew.Gozillon at amd.com>
Date: Tue, 7 Jul 2026 21:09:13 -0500
Subject: [PATCH] [Flang][OpenMP] Support mapping of zero-sized arrays

Whilst there is no data in theory to map with a zero sized array, the Fortran
OpenMP specification considers the case of a zero sized array being mapped
to device as the data being present on device. This includes both the data
and the descriptor, and it should fundamentlaly work as a zero-sized array
would on device with the respective Fortran functions for presence and size
checking etc.

We can't actually map a nullptr to device and expect it to be present in
current OpenMP (for good reason), however, thankfully, a zero sized array
isn't actually a nullptr, it is a descriptor contianing an allocated 1-byte
of data. So, we can map this to device, alongside the descriptor and then
the zero sized array is correctlly on device for all intents and purposes.
This case is notably different from a non-allocated or non-zero sized array
so we can do this without shooting ourselves in the foot.
---
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  | 30 +++---
 .../OpenMP/map-types-and-sizes.f90            | 20 +++-
 flang/test/Lower/OpenMP/defaultmap.f90        |  2 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 91 ++++++++++++++-----
 ...rget-record-type-with-ptr-member-host.mlir |  8 +-
 .../fortran/map-zero-size-array.f90           | 24 +++++
 6 files changed, 130 insertions(+), 45 deletions(-)
 create mode 100644 offload/test/offloading/fortran/map-zero-size-array.f90

diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index d4b343de988f2..d37b0c0a36191 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -27,6 +27,7 @@
 #include "flang/Optimizer/Builder/DirectivesCommon.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Builder/MutableBox.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
@@ -377,21 +378,21 @@ class MapInfoFinalizationPass
       builder.restoreInsertionPoint(insPt);
     }
 
-    // We should only emit a store if the passed in data is present, it is
-    // possible a user passes in no argument to an optional parameter, in which
-    // case we cannot store or we'll segfault on the emitted memcpy.
-    // TODO: We currently emit a present -> load/store every time we use a
-    // mapped value that requires a local allocation, this isn't the most
-    // efficient, although, it is more correct in a lot of situations. One
-    // such situation is emitting a this series of instructions in separate
-    // segments of a branch (e.g. two target regions in separate else/if branch
-    // mapping the same function argument), however, it would be nice to be able
-    // to optimize these situations e.g. raising the load/store out of the
-    // branch if possible. But perhaps this is best left to lower level
-    // optimisation passes.
+    // Make sure that our new allocation is "allocated" to default box state.
+    mlir::Type boxType = fir::unwrapRefType(alloca.getType());
+    mlir::Value nullBox = fir::factory::createUnallocatedBox(
+        builder, loc, boxType, /*nonDeferredLenParams=*/{});
+    fir::StoreOp::create(builder, loc, nullBox, alloca);
+
+    // Only overwrite with actual descriptor data if present.
+    // TODO: We currently emit a present check every time we use a mapped
+    // value that requires a local allocation. This isn't the most efficient,
+    // but it is correct for situations like separate target regions in
+    // different branches mapping the same function argument. Optimization
+    // (e.g., hoisting the load/store) is best left to lower level passes.
     auto isPresent =
         fir::IsPresentOp::create(builder, loc, builder.getI1Type(), descriptor);
-    builder.genIfOp(loc, {}, isPresent, false)
+    builder.genIfOp(loc, {}, isPresent, /*withElseRegion=*/false)
         .genThen([&]() {
           descriptor = builder.loadIfRef(loc, descriptor);
           fir::StoreOp::create(builder, loc, descriptor, alloca);
@@ -762,7 +763,8 @@ class MapInfoFinalizationPass
       return mapTypeFlag;
     }
 
-    flags |= MapFlags::to | (mapTypeFlag & MapFlags::implicit);
+    flags |=
+        MapFlags::to | (mapTypeFlag & (MapFlags::implicit | MapFlags::present));
 
     // Descriptors for objects will always be copied. This is because the
     // descriptor can be rematerialized by the compiler, and so the address
diff --git a/flang/test/Integration/OpenMP/map-types-and-sizes.f90 b/flang/test/Integration/OpenMP/map-types-and-sizes.f90
index 93f28a7525e19..c96e320edb7a1 100644
--- a/flang/test/Integration/OpenMP/map-types-and-sizes.f90
+++ b/flang/test/Integration/OpenMP/map-types-and-sizes.f90
@@ -615,6 +615,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[MEMBER_SIZE_CALC_2:.*]] = add i64 %[[MEMBER_SIZE_CALC_1]], 1
 !CHECK: %[[MEMBER_SIZE_CALC_3:.*]] = mul i64 1, %[[MEMBER_SIZE_CALC_2]]
 !CHECK: %[[MEMBER_SIZE_CALC_4:.*]] = mul i64 %[[MEMBER_SIZE_CALC_3]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[MEMBER_SIZE_CALC_4]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[MEMBER_SIZE_CALC_4]]
 !CHECK: %[[DTYPE_BASE_ADDR_LOAD_3:.*]] = load ptr, ptr %[[DTYPE_BASE_ADDR_ACCESS_3]], align 8
 !CHECK: %[[DTYPE_BASE_ADDR_LOAD_3_1:.*]] = load ptr, ptr %[[DTYPE_BASE_ADDR_ACCESS_3]], align 8
 !CHECK: %[[LOAD_DTYPE_DESC_MEMBER:.*]] = load ptr, ptr %[[DTYPE_ALLOCA_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -626,7 +628,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[DTYPE_BASE_ADDR_LOAD_3_1]], null
 !CHECK: %[[SEL_SZ:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 136
 !CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %{{.*}}, null
-!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[MEMBER_SIZE_CALC_4]]
+!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_ptrs, i32 0, i32 0
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 1
@@ -663,6 +665,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_3:.*]] = add i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_2]], 1
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_4:.*]] = mul i64 1, %[[ALLOCATABLE_MEMBER_SIZE_CALC_3]]
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_5:.*]] = mul i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_4]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[DTYPE_DESC_BASE_ADDR]], align 8
 !CHECK: %[[LOAD_BASE_ADDR2:.*]] = load ptr, ptr %[[DTYPE_DESC_BASE_ADDR]], align 8
 !CHECK: %[[LOAD_DESC_MEMBER_BASE_ADDR:.*]] = load ptr, ptr %[[MAPPED_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -670,7 +674,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[LOAD_BASE_ADDR2]], null
 !CHECK: %[[SEL_SZ:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 240
 !CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %[[ARRAY_OFFSET]], null
-!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
+!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: store ptr %[[DTYPE_DESC_ALLOCA_3]], ptr %[[BASE_PTR_ARR]], align 8
 !CHECK: %[[OFFLOAD_PTR_ARR:.*]] = getelementptr inbounds [9 x ptr], ptr %.offload_ptrs, i32 0, i32 0
@@ -700,6 +704,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_3:.*]] = add i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_2]], 1
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_4:.*]] = mul i64 1, %[[ALLOCATABLE_MEMBER_SIZE_CALC_3]]
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_5:.*]] = mul i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_4]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[NESTED_MEMBER_BASE_ADDR_ACCESS]], align 8
 !CHECK: %[[ARR_OFFS:.*]] = getelementptr inbounds i32, ptr %[[LOAD_BASE_ADDR]], i64 0
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[NESTED_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -709,7 +715,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[DTYPE_SEGMENT_SIZE_CALC_2:.*]] = ptrtoaddr ptr %[[NESTED_MEMBER_ACCESS]] to i64
 !CHECK: %[[DTYPE_SEGMENT_SIZE_CALC_3:.*]] = sub i64 %[[DTYPE_SEGMENT_SIZE_CALC_1]], %[[DTYPE_SEGMENT_SIZE_CALC_2]]
 !CHECK: %[[DATA_CMP:.*]] = icmp eq ptr %[[ARR_OFFS2]], null
-!CHECK: %[[DATA_SEL:.*]] = select i1 %[[DATA_CMP]], i64 0, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
+!CHECK: %[[DATA_SEL:.*]] = select i1 %[[DATA_CMP]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [5 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: store ptr %[[ALLOCA]], ptr %[[BASE_PTR_ARR]], align 8
 !CHECK: %[[OFFLOAD_PTR_ARR:.*]] = getelementptr inbounds [5 x ptr], ptr %.offload_ptrs, i32 0, i32 0
@@ -762,10 +768,14 @@ end subroutine mapType_common_block_members
 !CHECK: %[[OFF_PTR_CALC_2:.*]] = add i64 %[[OFF_PTR_CALC_1]], 1
 !CHECK: %[[OFF_PTR_CALC_3:.*]] = mul i64 1, %[[OFF_PTR_CALC_2]]
 !CHECK: %[[OFF_PTR_3:.*]] = mul i64 %[[OFF_PTR_CALC_3]], 104
+!CHECK: %[[SIZE_ZERO_CMP_1:.*]] = icmp eq i64 %[[OFF_PTR_3]], 0
+!CHECK: %[[SIZE_ADJUSTED_1:.*]] = select i1 %[[SIZE_ZERO_CMP_1]], i64 1, i64 %[[OFF_PTR_3]]
 !CHECK: %[[SZ_CALC_1_2:.*]] = sub i64 %[[BOUNDS_CALC]], 0
 !CHECK: %[[SZ_CALC_2_2:.*]] = add i64 %[[SZ_CALC_1_2]], 1
 !CHECK: %[[SZ_CALC_3_2:.*]] = mul i64 1, %[[SZ_CALC_2_2]]
 !CHECK: %[[SZ_CALC_4_2:.*]] = mul i64 %[[SZ_CALC_3_2]], 4
+!CHECK: %[[SIZE_ZERO_CMP_2:.*]] = icmp eq i64 %[[SZ_CALC_4_2]], 0
+!CHECK: %[[SIZE_ADJUSTED_2:.*]] = select i1 %[[SIZE_ZERO_CMP_2]], i64 1, i64 %[[SZ_CALC_4_2]]
 !CHECK: %[[LOAD_OFF_PTR:.*]] = load ptr, ptr %[[OFF_PTR_2]], align 8
 !CHECK: %[[ARR_OFFS:.*]] = getelementptr inbounds %_QFmaptype_nested_derived_type_member_idxTvertexes, ptr %[[LOAD_OFF_PTR]], i64 0
 !CHECK: %[[LOAD_ARR_OFFS:.*]] = load ptr, ptr %[[OFF_PTR_4]], align 8
@@ -779,9 +789,9 @@ end subroutine mapType_common_block_members
 !CHECK: %[[SZ_CALC_3:.*]] = ptrtoaddr ptr %[[OFF_PTR_1]] to i64
 !CHECK: %[[SZ_CALC_4:.*]] = sub i64 %[[SZ_CALC_2]], %[[SZ_CALC_3]]
 !CHECK: %[[SIZE_CMP:.*]] = icmp eq ptr %[[ARR_OFFS_2]], null
-!CHECK: %[[SIZE_SEL:.*]] = select i1 %[[SIZE_CMP]], i64 0, i64 %[[OFF_PTR_3]]
+!CHECK: %[[SIZE_SEL:.*]] = select i1 %[[SIZE_CMP]], i64 0, i64 %[[SIZE_ADJUSTED_1]]
 !CHECK: %[[SIZE_CMP2:.*]] = icmp eq ptr %[[ARR_OFFS_3]], null
-!CHECK: %[[SIZE_SEL2:.*]] = select i1 %[[SIZE_CMP2]], i64 0, i64 %[[SZ_CALC_4_2]]
+!CHECK: %[[SIZE_SEL2:.*]] = select i1 %[[SIZE_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED_2]]
 !CHECK: %[[SIZE_CMP3:.*]] = icmp eq ptr %[[ARR_OFFS]], null
 !CHECK: %[[SIZE_SEL3:.*]] = select i1 %[[SIZE_CMP3]], i64 0, i64 64
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [8 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
diff --git a/flang/test/Lower/OpenMP/defaultmap.f90 b/flang/test/Lower/OpenMP/defaultmap.f90
index 345c18bda465f..d2c9b48b13834 100644
--- a/flang/test/Lower/OpenMP/defaultmap.f90
+++ b/flang/test/Lower/OpenMP/defaultmap.f90
@@ -6,7 +6,7 @@ subroutine defaultmap_allocatable_present()
     integer, dimension(:), allocatable :: arr
 
 ! CHECK: %[[MAP_1:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(implicit, present) capture(ByRef) var_ptr_ptr({{.*}}) bounds({{.*}}) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>> {name = ""}
-! CHECK: %[[MAP_2:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, implicit, to) capture(ByRef) members({{.*}}) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "arr"}
+! CHECK: %[[MAP_2:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, implicit, present, to) capture(ByRef) members({{.*}}) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "arr"}
 !$omp target defaultmap(present: allocatable)
     arr(1) = 10
 !$omp end target
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..8a809ea9b76d7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -6221,6 +6221,27 @@ static uint64_t getArrayElementSizeInBits(LLVM::LLVMArrayType arrTy,
   return dl.getTypeSizeInBits(arrTy.getElementType());
 }
 
+// The intent is to verify if the mapped data being passed is a
+// pointer -> pointee that requires special handling in certain cases,
+// e.g. applying the OMP_MAP_PTR_AND_OBJ map type.
+//
+// There may be a better way to verify this, but unfortunately with
+// opaque pointers we lose the ability to easily check if something is
+// a pointer whilst maintaining access to the underlying type.
+static bool checkIfPointerMap(omp::MapInfoOp mapOp) {
+  // If we have a varPtrPtr field assigned then the underlying type is a pointer
+  if (mapOp.getVarPtrPtr())
+    return true;
+
+  // If the map data is declare target with a link clause, then it's represented
+  // as a pointer when we lower it to LLVM-IR even if at the MLIR level it has
+  // no relation to pointers.
+  if (isDeclareTargetLink(mapOp.getVarPtr()))
+    return true;
+
+  return false;
+}
+
 // This function calculates the size to be offloaded for a specified type, given
 // its associated map clause (which can contain bounds information which affects
 // the total size), this size is calculated based on the underlying element type
@@ -6273,8 +6294,53 @@ static llvm::Value *getSizeInBytes(DataLayout &dl, const mlir::Type &type,
       // size, so we do some on the fly runtime math to get the size in
       // bytes from the extent (ub - lb) * sizeInBytes. NOTE: This may need
       // some adjustment for members with more complex types.
-      return builder.CreateMul(elementCount,
-                               builder.getInt64(underlyingTypeSzInBits / 8));
+      llvm::Value *sizeCalc = builder.CreateMul(
+          elementCount, builder.getInt64(underlyingTypeSzInBits / 8),
+          "element_count");
+
+      // This is a part of a "complicated" bit of size calculation logic that is
+      // in place to handle a couple of scenarios, one specific to Fortran and
+      // the other a more general OpenMP issue. The other piece of the
+      // calculation can be found as the final size calculation within the
+      // processIndividualMap function. Ideally we would move it here, but due
+      // to the complexity of calculating the final base address of some
+      // constructs (required for a nullary check), it's left as the final step.
+      // So, in the below 2 cases, the nullary check is in processIndividualMap
+      // and the size equality check is here. The cases this modifications help
+      // cover are:
+      //
+      // 1) If an argument has a null base pointer, then the size must be set to
+      // 0
+      //    to avoid the runtime exploding/complaining about an illegal pointer
+      //    map. The size returning non-zero is feasible in certain cases if for
+      //    example someone has specified there own bounds/range.
+      // 2) We wish to support a very specific OpenMP Fortran edge-case where a
+      // size
+      //    zero array can be legally presence checked and found to be on device
+      //    when it has been mapped. In these rare occasions the
+      //    allocatable/pointer will have a size of 1 allocated for the
+      //    underlying data, but this wall not be represented within the size of
+      //    the descriptor, so we get a non-nullary pointer and a size of 0,
+      //    allowing us to specify a size of 1 in these cases registering it on
+      //    the device mapping table as present.
+      //
+      // The default fall through case is just returning the size calculation
+      // above, if we are not nullary and the size we calculate is non-zero,
+      // which is basically any pointer type that is allocated in someway
+      // (providing you are not running on a rare system that allows malloc's of
+      // size 0 with whatever caveats that may come with).
+      //
+      // Later in the nullary check in processIndividualMap it just devolves to
+      // selecting a size of 0 if we are nullary, if we are not, we will return
+      // either 1 or the calculated size, depending on the outcome of this
+      // select.
+      if (checkIfPointerMap(memberClause)) {
+        return builder.CreateSelect(
+            builder.CreateICmpEQ(sizeCalc, builder.getInt64(0)),
+            builder.getInt64(1), sizeCalc);
+      }
+
+      return sizeCalc;
     }
   }
 
@@ -6749,27 +6815,6 @@ getOverlappedMembers(llvm::SmallVectorImpl<size_t> &overlapMapDataIdxs,
       overlapMapDataIdxs.push_back(i);
 }
 
-// The intent is to verify if the mapped data being passed is a
-// pointer -> pointee that requires special handling in certain cases,
-// e.g. applying the OMP_MAP_PTR_AND_OBJ map type.
-//
-// There may be a better way to verify this, but unfortunately with
-// opaque pointers we lose the ability to easily check if something is
-// a pointer whilst maintaining access to the underlying type.
-static bool checkIfPointerMap(omp::MapInfoOp mapOp) {
-  // If we have a varPtrPtr field assigned then the underlying type is a pointer
-  if (mapOp.getVarPtrPtr())
-    return true;
-
-  // If the map data is declare target with a link clause, then it's represented
-  // as a pointer when we lower it to LLVM-IR even if at the MLIR level it has
-  // no relation to pointers.
-  if (isDeclareTargetLink(mapOp.getVarPtr()))
-    return true;
-
-  return false;
-}
-
 /// This function handles the insertion of a single item of map data from
 /// MapInfoData into the OMPIRBuilder's MapInfo list. Utilising this function
 /// means the map being inserted can be treated as a non-parent map entity,
diff --git a/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir b/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
index 8d06ece45d9b9..a8e9d5855f7f9 100644
--- a/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
@@ -73,19 +73,23 @@ module attributes {omp.is_target_device = false, omp.target_triples = ["amdgcn-a
 // CHECK: %[[FULL_ARR_SIZE2:.*]] = add i64 %[[FULL_ARR_SIZE3]], 1
 // CHECK: %[[FULL_ARR_SIZE1:.*]] = mul i64 1, %[[FULL_ARR_SIZE2]]
 // CHECK: %[[FULL_ARR_SIZE:.*]] = mul i64 %[[FULL_ARR_SIZE1]], 4
+// CHECK: %[[FULL_ARR_SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[FULL_ARR_SIZE]], 0
+// CHECK: %[[FULL_ARR_SIZE_ADJUSTED:.*]] = select i1 %[[FULL_ARR_SIZE_ZERO_CMP]], i64 1, i64 %[[FULL_ARR_SIZE]]
 // CHECK: %[[ARR_SECT_SIZE3:.*]] = sub i64 %[[ARR_SECT_SIZE4]], %[[ARR_SECT_OFFSET2]]
 // CHECK: %[[ARR_SECT_SIZE2:.*]] = add i64 %[[ARR_SECT_SIZE3]], 1
 // CHECK: %[[ARR_SECT_SIZE1:.*]] = mul i64 1, %[[ARR_SECT_SIZE2]]
 // CHECK: %[[ARR_SECT_SIZE:.*]] = mul i64 %[[ARR_SECT_SIZE1]], 4
+// CHECK: %[[ARR_SECT_SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ARR_SECT_SIZE]], 0
+// CHECK: %[[ARR_SECT_SIZE_ADJUSTED:.*]] = select i1 %[[ARR_SECT_SIZE_ZERO_CMP]], i64 1, i64 %[[ARR_SECT_SIZE]]
 // CHECK: %[[LFULL_ARR:.*]] = load ptr, ptr @full_arr, align 8
 // CHECK: %[[FULL_ARR_PTR:.*]] = getelementptr inbounds float, ptr %[[LFULL_ARR]], i64 0
 // CHECK: %[[LARR_SECT:.*]] = load ptr, ptr @sect_arr, align 8
 // CHECK: %[[ARR_SECT_PTR:.*]] = getelementptr inbounds i32, ptr %[[LARR_SECT]], i64 %[[ARR_SECT_OFFSET2]]
 // CHECK: %[[SCALAR_PTR_LOAD:.*]] = load ptr, ptr %[[SCALAR_BASE]], align 8
 // CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[FULL_ARR_PTR]], null
-// CHECK: %[[IS_NULL:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 %[[FULL_ARR_SIZE]]
+// CHECK: %[[IS_NULL:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 %[[FULL_ARR_SIZE_ADJUSTED]]
 // CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %[[ARR_SECT_PTR]], null
-// CHECK: %[[IS_NULL2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ARR_SECT_SIZE]]
+// CHECK: %[[IS_NULL2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ARR_SECT_SIZE_ADJUSTED]]
 // CHECK: %[[SCALAR_DESC_SZ4:.*]] = getelementptr { ptr, i64, i32, i8, i8, i8, i8 }, ptr %[[SCALAR_ALLOCA]], i32 1
 // CHECK: %[[SCALAR_DESC_SZ3:.*]] = ptrtoaddr ptr %[[SCALAR_DESC_SZ4]] to i64
 // CHECK: %[[SCALAR_DESC_SZ2:.*]] = ptrtoaddr ptr %[[SCALAR_ALLOCA]] to i64
diff --git a/offload/test/offloading/fortran/map-zero-size-array.f90 b/offload/test/offloading/fortran/map-zero-size-array.f90
new file mode 100644
index 0000000000000..f54f7ef48e312
--- /dev/null
+++ b/offload/test/offloading/fortran/map-zero-size-array.f90
@@ -0,0 +1,24 @@
+! OpenMP offloading test that checks we correctly map zero sized arrays
+! allowing subsequent presence checks to pass. This is legal in Fortran
+! as zero sized arrays are a little bit different to the scenario where
+! we have a not present argument or something that's not been allocated
+! at all.
+! REQUIRES: flang, amdgpu
+
+! RUN: %libomptarget-compile-fortran-run-and-check-generic
+program main
+ integer, allocatable :: array(:)
+ allocate(array(0))
+
+ !$omp target enter data map(to: array) 
+
+ ! Despite there technically being no data, the
+ ! presence check should legally return true here
+ ! and not fail.
+ !$omp target map(present, alloc: array)
+ !$omp end target
+
+ print *, "PASS"
+end program
+
+! CHECK: PASS


From llvmlistbot at llvm.org  Tue Jul  7 19:27:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 19:27:47 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4db5a3.eb24c6ea.27aedc.cb56@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-offload

@llvm/pr-subscribers-mlir-llvm

Author: agozillon

<details>
<summary>Changes</summary>

Whilst there is no data in theory to map with a zero sized array, the Fortran OpenMP specification considers the case of a zero sized array being mapped to device as the data being present on device. This includes both the data and the descriptor, and it should fundamentlaly work as a zero-sized array would on device with the respective Fortran functions for presence and size checking etc.

We can't actually map a nullptr to device and expect it to be present in current OpenMP (for good reason), however, thankfully, a zero sized array isn't actually a nullptr, it is a descriptor contianing an allocated 1-byte of data. So, we can map this to device, alongside the descriptor and then the zero sized array is correctlly on device for all intents and purposes. This case is notably different from a non-allocated or non-zero sized array so we can do this without shooting ourselves in the foot.

---

Patch is 20.28 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/208133.diff


6 Files Affected:

- (modified) flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp (+16-14) 
- (modified) flang/test/Integration/OpenMP/map-types-and-sizes.f90 (+15-5) 
- (modified) flang/test/Lower/OpenMP/defaultmap.f90 (+1-1) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+68-23) 
- (modified) mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir (+6-2) 
- (added) offload/test/offloading/fortran/map-zero-size-array.f90 (+24) 


``````````diff
diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index d4b343de988f2..d37b0c0a36191 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -27,6 +27,7 @@
 #include "flang/Optimizer/Builder/DirectivesCommon.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/HLFIRTools.h"
+#include "flang/Optimizer/Builder/MutableBox.h"
 #include "flang/Optimizer/Dialect/FIRType.h"
 #include "flang/Optimizer/Dialect/Support/KindMapping.h"
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
@@ -377,21 +378,21 @@ class MapInfoFinalizationPass
       builder.restoreInsertionPoint(insPt);
     }
 
-    // We should only emit a store if the passed in data is present, it is
-    // possible a user passes in no argument to an optional parameter, in which
-    // case we cannot store or we'll segfault on the emitted memcpy.
-    // TODO: We currently emit a present -> load/store every time we use a
-    // mapped value that requires a local allocation, this isn't the most
-    // efficient, although, it is more correct in a lot of situations. One
-    // such situation is emitting a this series of instructions in separate
-    // segments of a branch (e.g. two target regions in separate else/if branch
-    // mapping the same function argument), however, it would be nice to be able
-    // to optimize these situations e.g. raising the load/store out of the
-    // branch if possible. But perhaps this is best left to lower level
-    // optimisation passes.
+    // Make sure that our new allocation is "allocated" to default box state.
+    mlir::Type boxType = fir::unwrapRefType(alloca.getType());
+    mlir::Value nullBox = fir::factory::createUnallocatedBox(
+        builder, loc, boxType, /*nonDeferredLenParams=*/{});
+    fir::StoreOp::create(builder, loc, nullBox, alloca);
+
+    // Only overwrite with actual descriptor data if present.
+    // TODO: We currently emit a present check every time we use a mapped
+    // value that requires a local allocation. This isn't the most efficient,
+    // but it is correct for situations like separate target regions in
+    // different branches mapping the same function argument. Optimization
+    // (e.g., hoisting the load/store) is best left to lower level passes.
     auto isPresent =
         fir::IsPresentOp::create(builder, loc, builder.getI1Type(), descriptor);
-    builder.genIfOp(loc, {}, isPresent, false)
+    builder.genIfOp(loc, {}, isPresent, /*withElseRegion=*/false)
         .genThen([&]() {
           descriptor = builder.loadIfRef(loc, descriptor);
           fir::StoreOp::create(builder, loc, descriptor, alloca);
@@ -762,7 +763,8 @@ class MapInfoFinalizationPass
       return mapTypeFlag;
     }
 
-    flags |= MapFlags::to | (mapTypeFlag & MapFlags::implicit);
+    flags |=
+        MapFlags::to | (mapTypeFlag & (MapFlags::implicit | MapFlags::present));
 
     // Descriptors for objects will always be copied. This is because the
     // descriptor can be rematerialized by the compiler, and so the address
diff --git a/flang/test/Integration/OpenMP/map-types-and-sizes.f90 b/flang/test/Integration/OpenMP/map-types-and-sizes.f90
index 93f28a7525e19..c96e320edb7a1 100644
--- a/flang/test/Integration/OpenMP/map-types-and-sizes.f90
+++ b/flang/test/Integration/OpenMP/map-types-and-sizes.f90
@@ -615,6 +615,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[MEMBER_SIZE_CALC_2:.*]] = add i64 %[[MEMBER_SIZE_CALC_1]], 1
 !CHECK: %[[MEMBER_SIZE_CALC_3:.*]] = mul i64 1, %[[MEMBER_SIZE_CALC_2]]
 !CHECK: %[[MEMBER_SIZE_CALC_4:.*]] = mul i64 %[[MEMBER_SIZE_CALC_3]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[MEMBER_SIZE_CALC_4]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[MEMBER_SIZE_CALC_4]]
 !CHECK: %[[DTYPE_BASE_ADDR_LOAD_3:.*]] = load ptr, ptr %[[DTYPE_BASE_ADDR_ACCESS_3]], align 8
 !CHECK: %[[DTYPE_BASE_ADDR_LOAD_3_1:.*]] = load ptr, ptr %[[DTYPE_BASE_ADDR_ACCESS_3]], align 8
 !CHECK: %[[LOAD_DTYPE_DESC_MEMBER:.*]] = load ptr, ptr %[[DTYPE_ALLOCA_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -626,7 +628,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[DTYPE_BASE_ADDR_LOAD_3_1]], null
 !CHECK: %[[SEL_SZ:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 136
 !CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %{{.*}}, null
-!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[MEMBER_SIZE_CALC_4]]
+!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_ptrs, i32 0, i32 0
 !CHECK: getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 1
@@ -663,6 +665,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_3:.*]] = add i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_2]], 1
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_4:.*]] = mul i64 1, %[[ALLOCATABLE_MEMBER_SIZE_CALC_3]]
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_5:.*]] = mul i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_4]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[DTYPE_DESC_BASE_ADDR]], align 8
 !CHECK: %[[LOAD_BASE_ADDR2:.*]] = load ptr, ptr %[[DTYPE_DESC_BASE_ADDR]], align 8
 !CHECK: %[[LOAD_DESC_MEMBER_BASE_ADDR:.*]] = load ptr, ptr %[[MAPPED_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -670,7 +674,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[LOAD_BASE_ADDR2]], null
 !CHECK: %[[SEL_SZ:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 240
 !CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %[[ARRAY_OFFSET]], null
-!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
+!CHECK: %[[SEL_SZ2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [9 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: store ptr %[[DTYPE_DESC_ALLOCA_3]], ptr %[[BASE_PTR_ARR]], align 8
 !CHECK: %[[OFFLOAD_PTR_ARR:.*]] = getelementptr inbounds [9 x ptr], ptr %.offload_ptrs, i32 0, i32 0
@@ -700,6 +704,8 @@ end subroutine mapType_common_block_members
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_3:.*]] = add i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_2]], 1
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_4:.*]] = mul i64 1, %[[ALLOCATABLE_MEMBER_SIZE_CALC_3]]
 !CHECK: %[[ALLOCATABLE_MEMBER_SIZE_CALC_5:.*]] = mul i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_4]], 4
+!CHECK: %[[SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]], 0
+!CHECK: %[[SIZE_ADJUSTED:.*]] = select i1 %[[SIZE_ZERO_CMP]], i64 1, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[NESTED_MEMBER_BASE_ADDR_ACCESS]], align 8
 !CHECK: %[[ARR_OFFS:.*]] = getelementptr inbounds i32, ptr %[[LOAD_BASE_ADDR]], i64 0
 !CHECK: %[[LOAD_BASE_ADDR:.*]] = load ptr, ptr %[[NESTED_MEMBER_BASE_ADDR_ACCESS]], align 8
@@ -709,7 +715,7 @@ end subroutine mapType_common_block_members
 !CHECK: %[[DTYPE_SEGMENT_SIZE_CALC_2:.*]] = ptrtoaddr ptr %[[NESTED_MEMBER_ACCESS]] to i64
 !CHECK: %[[DTYPE_SEGMENT_SIZE_CALC_3:.*]] = sub i64 %[[DTYPE_SEGMENT_SIZE_CALC_1]], %[[DTYPE_SEGMENT_SIZE_CALC_2]]
 !CHECK: %[[DATA_CMP:.*]] = icmp eq ptr %[[ARR_OFFS2]], null
-!CHECK: %[[DATA_SEL:.*]] = select i1 %[[DATA_CMP]], i64 0, i64 %[[ALLOCATABLE_MEMBER_SIZE_CALC_5]]
+!CHECK: %[[DATA_SEL:.*]] = select i1 %[[DATA_CMP]], i64 0, i64 %[[SIZE_ADJUSTED]]
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [5 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
 !CHECK: store ptr %[[ALLOCA]], ptr %[[BASE_PTR_ARR]], align 8
 !CHECK: %[[OFFLOAD_PTR_ARR:.*]] = getelementptr inbounds [5 x ptr], ptr %.offload_ptrs, i32 0, i32 0
@@ -762,10 +768,14 @@ end subroutine mapType_common_block_members
 !CHECK: %[[OFF_PTR_CALC_2:.*]] = add i64 %[[OFF_PTR_CALC_1]], 1
 !CHECK: %[[OFF_PTR_CALC_3:.*]] = mul i64 1, %[[OFF_PTR_CALC_2]]
 !CHECK: %[[OFF_PTR_3:.*]] = mul i64 %[[OFF_PTR_CALC_3]], 104
+!CHECK: %[[SIZE_ZERO_CMP_1:.*]] = icmp eq i64 %[[OFF_PTR_3]], 0
+!CHECK: %[[SIZE_ADJUSTED_1:.*]] = select i1 %[[SIZE_ZERO_CMP_1]], i64 1, i64 %[[OFF_PTR_3]]
 !CHECK: %[[SZ_CALC_1_2:.*]] = sub i64 %[[BOUNDS_CALC]], 0
 !CHECK: %[[SZ_CALC_2_2:.*]] = add i64 %[[SZ_CALC_1_2]], 1
 !CHECK: %[[SZ_CALC_3_2:.*]] = mul i64 1, %[[SZ_CALC_2_2]]
 !CHECK: %[[SZ_CALC_4_2:.*]] = mul i64 %[[SZ_CALC_3_2]], 4
+!CHECK: %[[SIZE_ZERO_CMP_2:.*]] = icmp eq i64 %[[SZ_CALC_4_2]], 0
+!CHECK: %[[SIZE_ADJUSTED_2:.*]] = select i1 %[[SIZE_ZERO_CMP_2]], i64 1, i64 %[[SZ_CALC_4_2]]
 !CHECK: %[[LOAD_OFF_PTR:.*]] = load ptr, ptr %[[OFF_PTR_2]], align 8
 !CHECK: %[[ARR_OFFS:.*]] = getelementptr inbounds %_QFmaptype_nested_derived_type_member_idxTvertexes, ptr %[[LOAD_OFF_PTR]], i64 0
 !CHECK: %[[LOAD_ARR_OFFS:.*]] = load ptr, ptr %[[OFF_PTR_4]], align 8
@@ -779,9 +789,9 @@ end subroutine mapType_common_block_members
 !CHECK: %[[SZ_CALC_3:.*]] = ptrtoaddr ptr %[[OFF_PTR_1]] to i64
 !CHECK: %[[SZ_CALC_4:.*]] = sub i64 %[[SZ_CALC_2]], %[[SZ_CALC_3]]
 !CHECK: %[[SIZE_CMP:.*]] = icmp eq ptr %[[ARR_OFFS_2]], null
-!CHECK: %[[SIZE_SEL:.*]] = select i1 %[[SIZE_CMP]], i64 0, i64 %[[OFF_PTR_3]]
+!CHECK: %[[SIZE_SEL:.*]] = select i1 %[[SIZE_CMP]], i64 0, i64 %[[SIZE_ADJUSTED_1]]
 !CHECK: %[[SIZE_CMP2:.*]] = icmp eq ptr %[[ARR_OFFS_3]], null
-!CHECK: %[[SIZE_SEL2:.*]] = select i1 %[[SIZE_CMP2]], i64 0, i64 %[[SZ_CALC_4_2]]
+!CHECK: %[[SIZE_SEL2:.*]] = select i1 %[[SIZE_CMP2]], i64 0, i64 %[[SIZE_ADJUSTED_2]]
 !CHECK: %[[SIZE_CMP3:.*]] = icmp eq ptr %[[ARR_OFFS]], null
 !CHECK: %[[SIZE_SEL3:.*]] = select i1 %[[SIZE_CMP3]], i64 0, i64 64
 !CHECK: %[[BASE_PTR_ARR:.*]] = getelementptr inbounds [8 x ptr], ptr %.offload_baseptrs, i32 0, i32 0
diff --git a/flang/test/Lower/OpenMP/defaultmap.f90 b/flang/test/Lower/OpenMP/defaultmap.f90
index 345c18bda465f..d2c9b48b13834 100644
--- a/flang/test/Lower/OpenMP/defaultmap.f90
+++ b/flang/test/Lower/OpenMP/defaultmap.f90
@@ -6,7 +6,7 @@ subroutine defaultmap_allocatable_present()
     integer, dimension(:), allocatable :: arr
 
 ! CHECK: %[[MAP_1:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(implicit, present) capture(ByRef) var_ptr_ptr({{.*}}) bounds({{.*}}) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>> {name = ""}
-! CHECK: %[[MAP_2:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, implicit, to) capture(ByRef) members({{.*}}) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "arr"}
+! CHECK: %[[MAP_2:.*]] = omp.map.info var_ptr({{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, implicit, present, to) capture(ByRef) members({{.*}}) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "arr"}
 !$omp target defaultmap(present: allocatable)
     arr(1) = 10
 !$omp end target
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..8a809ea9b76d7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -6221,6 +6221,27 @@ static uint64_t getArrayElementSizeInBits(LLVM::LLVMArrayType arrTy,
   return dl.getTypeSizeInBits(arrTy.getElementType());
 }
 
+// The intent is to verify if the mapped data being passed is a
+// pointer -> pointee that requires special handling in certain cases,
+// e.g. applying the OMP_MAP_PTR_AND_OBJ map type.
+//
+// There may be a better way to verify this, but unfortunately with
+// opaque pointers we lose the ability to easily check if something is
+// a pointer whilst maintaining access to the underlying type.
+static bool checkIfPointerMap(omp::MapInfoOp mapOp) {
+  // If we have a varPtrPtr field assigned then the underlying type is a pointer
+  if (mapOp.getVarPtrPtr())
+    return true;
+
+  // If the map data is declare target with a link clause, then it's represented
+  // as a pointer when we lower it to LLVM-IR even if at the MLIR level it has
+  // no relation to pointers.
+  if (isDeclareTargetLink(mapOp.getVarPtr()))
+    return true;
+
+  return false;
+}
+
 // This function calculates the size to be offloaded for a specified type, given
 // its associated map clause (which can contain bounds information which affects
 // the total size), this size is calculated based on the underlying element type
@@ -6273,8 +6294,53 @@ static llvm::Value *getSizeInBytes(DataLayout &dl, const mlir::Type &type,
       // size, so we do some on the fly runtime math to get the size in
       // bytes from the extent (ub - lb) * sizeInBytes. NOTE: This may need
       // some adjustment for members with more complex types.
-      return builder.CreateMul(elementCount,
-                               builder.getInt64(underlyingTypeSzInBits / 8));
+      llvm::Value *sizeCalc = builder.CreateMul(
+          elementCount, builder.getInt64(underlyingTypeSzInBits / 8),
+          "element_count");
+
+      // This is a part of a "complicated" bit of size calculation logic that is
+      // in place to handle a couple of scenarios, one specific to Fortran and
+      // the other a more general OpenMP issue. The other piece of the
+      // calculation can be found as the final size calculation within the
+      // processIndividualMap function. Ideally we would move it here, but due
+      // to the complexity of calculating the final base address of some
+      // constructs (required for a nullary check), it's left as the final step.
+      // So, in the below 2 cases, the nullary check is in processIndividualMap
+      // and the size equality check is here. The cases this modifications help
+      // cover are:
+      //
+      // 1) If an argument has a null base pointer, then the size must be set to
+      // 0
+      //    to avoid the runtime exploding/complaining about an illegal pointer
+      //    map. The size returning non-zero is feasible in certain cases if for
+      //    example someone has specified there own bounds/range.
+      // 2) We wish to support a very specific OpenMP Fortran edge-case where a
+      // size
+      //    zero array can be legally presence checked and found to be on device
+      //    when it has been mapped. In these rare occasions the
+      //    allocatable/pointer will have a size of 1 allocated for the
+      //    underlying data, but this wall not be represented within the size of
+      //    the descriptor, so we get a non-nullary pointer and a size of 0,
+      //    allowing us to specify a size of 1 in these cases registering it on
+      //    the device mapping table as present.
+      //
+      // The default fall through case is just returning the size calculation
+      // above, if we are not nullary and the size we calculate is non-zero,
+      // which is basically any pointer type that is allocated in someway
+      // (providing you are not running on a rare system that allows malloc's of
+      // size 0 with whatever caveats that may come with).
+      //
+      // Later in the nullary check in processIndividualMap it just devolves to
+      // selecting a size of 0 if we are nullary, if we are not, we will return
+      // either 1 or the calculated size, depending on the outcome of this
+      // select.
+      if (checkIfPointerMap(memberClause)) {
+        return builder.CreateSelect(
+            builder.CreateICmpEQ(sizeCalc, builder.getInt64(0)),
+            builder.getInt64(1), sizeCalc);
+      }
+
+      return sizeCalc;
     }
   }
 
@@ -6749,27 +6815,6 @@ getOverlappedMembers(llvm::SmallVectorImpl<size_t> &overlapMapDataIdxs,
       overlapMapDataIdxs.push_back(i);
 }
 
-// The intent is to verify if the mapped data being passed is a
-// pointer -> pointee that requires special handling in certain cases,
-// e.g. applying the OMP_MAP_PTR_AND_OBJ map type.
-//
-// There may be a better way to verify this, but unfortunately with
-// opaque pointers we lose the ability to easily check if something is
-// a pointer whilst maintaining access to the underlying type.
-static bool checkIfPointerMap(omp::MapInfoOp mapOp) {
-  // If we have a varPtrPtr field assigned then the underlying type is a pointer
-  if (mapOp.getVarPtrPtr())
-    return true;
-
-  // If the map data is declare target with a link clause, then it's represented
-  // as a pointer when we lower it to LLVM-IR even if at the MLIR level it has
-  // no relation to pointers.
-  if (isDeclareTargetLink(mapOp.getVarPtr()))
-    return true;
-
-  return false;
-}
-
 /// This function handles the insertion of a single item of map data from
 /// MapInfoData into the OMPIRBuilder's MapInfo list. Utilising this function
 /// means the map being inserted can be treated as a non-parent map entity,
diff --git a/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir b/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
index 8d06ece45d9b9..a8e9d5855f7f9 100644
--- a/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-record-type-with-ptr-member-host.mlir
@@ -73,19 +73,23 @@ module attributes {omp.is_target_device = false, omp.target_triples = ["amdgcn-a
 // CHECK: %[[FULL_ARR_SIZE2:.*]] = add i64 %[[FULL_ARR_SIZE3]], 1
 // CHECK: %[[FULL_ARR_SIZE1:.*]] = mul i64 1, %[[FULL_ARR_SIZE2]]
 // CHECK: %[[FULL_ARR_SIZE:.*]] = mul i64 %[[FULL_ARR_SIZE1]], 4
+// CHECK: %[[FULL_ARR_SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[FULL_ARR_SIZE]], 0
+// CHECK: %[[FULL_ARR_SIZE_ADJUSTED:.*]] = select i1 %[[FULL_ARR_SIZE_ZERO_CMP]], i64 1, i64 %[[FULL_ARR_SIZE]]
 // CHECK: %[[ARR_SECT_SIZE3:.*]] = sub i64 %[[ARR_SECT_SIZE4]], %[[ARR_SECT_OFFSET2]]
 // CHECK: %[[ARR_SECT_SIZE2:.*]] = add i64 %[[ARR_SECT_SIZE3]], 1
 // CHECK: %[[ARR_SECT_SIZE1:.*]] = mul i64 1, %[[ARR_SECT_SIZE2]]
 // CHECK: %[[ARR_SECT_SIZE:.*]] = mul i64 %[[ARR_SECT_SIZE1]], 4
+// CHECK: %[[ARR_SECT_SIZE_ZERO_CMP:.*]] = icmp eq i64 %[[ARR_SECT_SIZE]], 0
+// CHECK: %[[ARR_SECT_SIZE_ADJUSTED:.*]] = select i1 %[[ARR_SECT_SIZE_ZERO_CMP]], i64 1, i64 %[[ARR_SECT_SIZE]]
 // CHECK: %[[LFULL_ARR:.*]] = load ptr, ptr @full_arr, align 8
 // CHECK: %[[FULL_ARR_PTR:.*]] = getelementptr inbounds float, ptr %[[LFULL_ARR]], i64 0
 // CHECK: %[[LARR_SECT:.*]] = load ptr, ptr @sect_arr, align 8
 // CHECK: %[[ARR_SECT_PTR:.*]] = getelementptr inbounds i32, ptr %[[LARR_SECT]], i64 %[[ARR_SECT_OFFSET2]]
 // CHECK: %[[SCALAR_PTR_LOAD:.*]] = load ptr, ptr %[[SCALAR_BASE]], align 8
 // CHECK: %[[NULL_CMP:.*]] = icmp eq ptr %[[FULL_ARR_PTR]], null
-// CHECK: %[[IS_NULL:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 %[[FULL_ARR_SIZE]]
+// CHECK: %[[IS_NULL:.*]] = select i1 %[[NULL_CMP]], i64 0, i64 %[[FULL_ARR_SIZE_ADJUSTED]]
 // CHECK: %[[NULL_CMP2:.*]] = icmp eq ptr %[[ARR_SECT_PTR]], null
-// CHECK: %[[IS_NULL2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ARR_SECT_SIZE]]
+// CHECK: %[[IS_NULL2:.*]] = select i1 %[[NULL_CMP2]], i64 0, i64 %[[ARR_SECT_SIZE_ADJUSTED]]
 // CHECK: %[[SCALAR_DESC_SZ4:.*]] = getelementptr { ptr, i64, i32, i8, i8, i8, i8 }, ptr %[[SCALAR_ALLOCA]], i32 1
 // CHECK: %[[SCALAR_DESC_SZ3:.*]] = ptrtoaddr ptr %[[SCALAR_DESC_SZ4]] to i64
 // CHECK: %[[SCALAR_DESC_SZ2:.*]] = ptrtoaddr ptr %[[SCALAR_ALLOCA]] to i64
diff --git a/offload/test/offloading/fortran/map-zero-size-array.f90 b/offload/test/offloading/fortran/map-zero-size-array.f90
new file mode 100644
index 0000000000000..f54f7ef48e312
--- /dev/null
+++ b/offload/test/offloading/fortran/map-zero-size-array.f90
@@ -0,0 +1,24 @@
+! OpenMP offloading test that checks we correctly map zero sized arrays
+! allowing subsequent presence checks to pass. This is legal in Fortran
+! as zero sized arrays are a little bit different to the scenario where
+! we have a not present argument or something that's not been allocated
+! at all.
+! REQUIRES: flang, amdgpu
+
+! RUN: %libomptarget-compile-fortran-run-and-check-generic
+program main
+ integer, allocatable :: array(:)
+ allocate...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Tue Jul  7 21:03:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 21:03:19 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] [NFC] Fix
 spelling of definition in comments (PR #208138)
Message-ID: <llvm.org/llvm/llvm-project/pull/208138@github.com>


https://github.com/divedb created https://github.com/llvm/llvm-project/pull/208138

Fix several occurrences of:

- defination   -> definition
- definations -> definitions

This change only updates comments and does not introduce any functional changes.

>From bdbd2ceeb8ce7569e2a1e0205de161d87295b2e3 Mon Sep 17 00:00:00 2001
From: gc <gc1365798 at gmail.com>
Date: Wed, 8 Jul 2026 11:53:54 +0800
Subject: [PATCH] [NFC] Fix spelling of definition in comments

---
 .../test/clang-tidy/checkers/bugprone/argument-comment.cpp    | 2 +-
 clang/test/CodeGenCoroutines/coro-gro3.cpp                    | 2 +-
 llvm/lib/CodeGen/MachineSink.cpp                              | 4 ++--
 llvm/lib/MC/XCOFFObjectWriter.cpp                             | 2 +-
 llvm/lib/Target/PowerPC/PPCCTRLoops.cpp                       | 4 ++--
 mlir/include/mlir/IR/Interfaces.td                            | 4 ++--
 mlir/include/mlir/IR/Properties.td                            | 2 +-
 mlir/include/mlir/IR/Traits.td                                | 4 ++--
 8 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp b/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
index dd8221772a300..aed96a9d7a5ed 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
@@ -150,5 +150,5 @@ void test() {
 
 void testInvalidSlocCxxConstructExpr() {
   __builtin_va_list __args;
-  // __builtin_va_list has no defination in any source file
+  // __builtin_va_list has no definition in any source file
 } 
diff --git a/clang/test/CodeGenCoroutines/coro-gro3.cpp b/clang/test/CodeGenCoroutines/coro-gro3.cpp
index 8c37b1e9838d3..7f5a8e2df1444 100644
--- a/clang/test/CodeGenCoroutines/coro-gro3.cpp
+++ b/clang/test/CodeGenCoroutines/coro-gro3.cpp
@@ -1,4 +1,4 @@
-// Tests defination of get-return-object-invocation [dcl.fct.def.coroutine] (and CWG2563)
+// Tests definition of get-return-object-invocation [dcl.fct.def.coroutine] (and CWG2563)
 // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -std=c++20 -emit-llvm %s -o - -disable-llvm-passes | FileCheck %s
 
 #include "Inputs/coroutine.h"
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index da9eaa68d22e0..6784752fba357 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -1343,8 +1343,8 @@ bool MachineSinking::isProfitableToSinkTo(Register Reg, MachineInstr &MI,
       MachineCycle *Cycle = CI->getCycle(DefMI->getParent());
       // DefMI is defined outside of cycle. There should be no live range
       // impact for this operand. Defination outside of cycle means:
-      // 1: defination is outside of cycle.
-      // 2: defination is in this cycle, but it is a PHI in the cycle header.
+      // 1: definition is outside of cycle.
+      // 2: definition is in this cycle, but it is a PHI in the cycle header.
       if (Cycle != MCycle || (DefMI->isPHI() && Cycle && Cycle->isReducible() &&
                               Cycle->getHeader() == DefMI->getParent()))
         continue;
diff --git a/llvm/lib/MC/XCOFFObjectWriter.cpp b/llvm/lib/MC/XCOFFObjectWriter.cpp
index d9f47d6f4ba25..c4900b47b8956 100644
--- a/llvm/lib/MC/XCOFFObjectWriter.cpp
+++ b/llvm/lib/MC/XCOFFObjectWriter.cpp
@@ -100,7 +100,7 @@ struct XCOFFSection {
 using CsectGroup = std::deque<XCOFFSection>;
 using CsectGroups = std::deque<CsectGroup *>;
 
-// The basic section entry defination. This Section represents a section entry
+// The basic section entry definition. This Section represents a section entry
 // in XCOFF section header table.
 struct SectionEntry {
   char Name[XCOFF::NameSize];
diff --git a/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp b/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
index ef5f93d170408..74520a4d17207 100644
--- a/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
+++ b/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
@@ -115,8 +115,8 @@ bool PPCCTRLoops::isCTRClobber(MachineInstr *MI, bool CheckReads) const {
   if (!CheckReads) {
     // If we are only checking for defs, that is we are going to find
     // definitions before MTCTRloop, for this case:
-    // CTR defination inside the callee of a call instruction will not impact
-    // the defination of MTCTRloop, so we can use definesRegister() for the
+    // CTR definition inside the callee of a call instruction will not impact
+    // the definition of MTCTRloop, so we can use definesRegister() for the
     // check, no need to check the regmask.
     return MI->definesRegister(PPC::CTR, /*TRI=*/nullptr) ||
            MI->definesRegister(PPC::CTR8, /*TRI=*/nullptr);
diff --git a/mlir/include/mlir/IR/Interfaces.td b/mlir/include/mlir/IR/Interfaces.td
index e16de2942a043..4954b2a6a652d 100644
--- a/mlir/include/mlir/IR/Interfaces.td
+++ b/mlir/include/mlir/IR/Interfaces.td
@@ -1,4 +1,4 @@
-//===-- Interfaces.td - Interfaces defination file ------------------*- tablegen -*-===//
+//===-- Interfaces.td - Interfaces definition file ------------------*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This file contains definations for Interfaces.
+// This file contains definitions for Interfaces.
 //
 //===----------------------------------------------------------------------===//
 
diff --git a/mlir/include/mlir/IR/Properties.td b/mlir/include/mlir/IR/Properties.td
index 2830ba96fb787..89e232feee617 100644
--- a/mlir/include/mlir/IR/Properties.td
+++ b/mlir/include/mlir/IR/Properties.td
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This is the base properties defination file.
+// This is the base properties definition file.
 //
 //===----------------------------------------------------------------------===//
 
diff --git a/mlir/include/mlir/IR/Traits.td b/mlir/include/mlir/IR/Traits.td
index 221f8633ad954..41e745674f605 100644
--- a/mlir/include/mlir/IR/Traits.td
+++ b/mlir/include/mlir/IR/Traits.td
@@ -1,4 +1,4 @@
-//===-- Traits.td - Trait definations file ------------------*- tablegen -*-===//
+//===-- Traits.td - Trait definitions file ------------------*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This file contains definations for traits.
+// This file contains definitions for traits.
 //
 //===----------------------------------------------------------------------===//
 


From llvmlistbot at llvm.org  Tue Jul  7 21:04:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 21:04:02 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] [NFC] Fix
 spelling of definition in comments (PR #208138)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208138@github.com>
Message-ID: <6a4dcc32.13be6df8.145c.ecd5@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @divedb :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/208138

From llvmlistbot at llvm.org  Tue Jul  7 21:04:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 21:04:38 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] [NFC] Fix
 spelling of definition in comments (PR #208138)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208138@github.com>
Message-ID: <6a4dcc56.f1c1923d.991d1.e63d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang-tools-extra

Author: GuoC (divedb)

<details>
<summary>Changes</summary>

Fix several occurrences of:

- defination   -&gt; definition
- definations -&gt; definitions

This change only updates comments and does not introduce any functional changes.

---
Full diff: https://github.com/llvm/llvm-project/pull/208138.diff


8 Files Affected:

- (modified) clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp (+1-1) 
- (modified) clang/test/CodeGenCoroutines/coro-gro3.cpp (+1-1) 
- (modified) llvm/lib/CodeGen/MachineSink.cpp (+2-2) 
- (modified) llvm/lib/MC/XCOFFObjectWriter.cpp (+1-1) 
- (modified) llvm/lib/Target/PowerPC/PPCCTRLoops.cpp (+2-2) 
- (modified) mlir/include/mlir/IR/Interfaces.td (+2-2) 
- (modified) mlir/include/mlir/IR/Properties.td (+1-1) 
- (modified) mlir/include/mlir/IR/Traits.td (+2-2) 


``````````diff
diff --git a/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp b/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
index dd8221772a300..aed96a9d7a5ed 100644
--- a/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
+++ b/clang-tools-extra/test/clang-tidy/checkers/bugprone/argument-comment.cpp
@@ -150,5 +150,5 @@ void test() {
 
 void testInvalidSlocCxxConstructExpr() {
   __builtin_va_list __args;
-  // __builtin_va_list has no defination in any source file
+  // __builtin_va_list has no definition in any source file
 } 
diff --git a/clang/test/CodeGenCoroutines/coro-gro3.cpp b/clang/test/CodeGenCoroutines/coro-gro3.cpp
index 8c37b1e9838d3..7f5a8e2df1444 100644
--- a/clang/test/CodeGenCoroutines/coro-gro3.cpp
+++ b/clang/test/CodeGenCoroutines/coro-gro3.cpp
@@ -1,4 +1,4 @@
-// Tests defination of get-return-object-invocation [dcl.fct.def.coroutine] (and CWG2563)
+// Tests definition of get-return-object-invocation [dcl.fct.def.coroutine] (and CWG2563)
 // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -std=c++20 -emit-llvm %s -o - -disable-llvm-passes | FileCheck %s
 
 #include "Inputs/coroutine.h"
diff --git a/llvm/lib/CodeGen/MachineSink.cpp b/llvm/lib/CodeGen/MachineSink.cpp
index da9eaa68d22e0..6784752fba357 100644
--- a/llvm/lib/CodeGen/MachineSink.cpp
+++ b/llvm/lib/CodeGen/MachineSink.cpp
@@ -1343,8 +1343,8 @@ bool MachineSinking::isProfitableToSinkTo(Register Reg, MachineInstr &MI,
       MachineCycle *Cycle = CI->getCycle(DefMI->getParent());
       // DefMI is defined outside of cycle. There should be no live range
       // impact for this operand. Defination outside of cycle means:
-      // 1: defination is outside of cycle.
-      // 2: defination is in this cycle, but it is a PHI in the cycle header.
+      // 1: definition is outside of cycle.
+      // 2: definition is in this cycle, but it is a PHI in the cycle header.
       if (Cycle != MCycle || (DefMI->isPHI() && Cycle && Cycle->isReducible() &&
                               Cycle->getHeader() == DefMI->getParent()))
         continue;
diff --git a/llvm/lib/MC/XCOFFObjectWriter.cpp b/llvm/lib/MC/XCOFFObjectWriter.cpp
index d9f47d6f4ba25..c4900b47b8956 100644
--- a/llvm/lib/MC/XCOFFObjectWriter.cpp
+++ b/llvm/lib/MC/XCOFFObjectWriter.cpp
@@ -100,7 +100,7 @@ struct XCOFFSection {
 using CsectGroup = std::deque<XCOFFSection>;
 using CsectGroups = std::deque<CsectGroup *>;
 
-// The basic section entry defination. This Section represents a section entry
+// The basic section entry definition. This Section represents a section entry
 // in XCOFF section header table.
 struct SectionEntry {
   char Name[XCOFF::NameSize];
diff --git a/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp b/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
index ef5f93d170408..74520a4d17207 100644
--- a/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
+++ b/llvm/lib/Target/PowerPC/PPCCTRLoops.cpp
@@ -115,8 +115,8 @@ bool PPCCTRLoops::isCTRClobber(MachineInstr *MI, bool CheckReads) const {
   if (!CheckReads) {
     // If we are only checking for defs, that is we are going to find
     // definitions before MTCTRloop, for this case:
-    // CTR defination inside the callee of a call instruction will not impact
-    // the defination of MTCTRloop, so we can use definesRegister() for the
+    // CTR definition inside the callee of a call instruction will not impact
+    // the definition of MTCTRloop, so we can use definesRegister() for the
     // check, no need to check the regmask.
     return MI->definesRegister(PPC::CTR, /*TRI=*/nullptr) ||
            MI->definesRegister(PPC::CTR8, /*TRI=*/nullptr);
diff --git a/mlir/include/mlir/IR/Interfaces.td b/mlir/include/mlir/IR/Interfaces.td
index e16de2942a043..4954b2a6a652d 100644
--- a/mlir/include/mlir/IR/Interfaces.td
+++ b/mlir/include/mlir/IR/Interfaces.td
@@ -1,4 +1,4 @@
-//===-- Interfaces.td - Interfaces defination file ------------------*- tablegen -*-===//
+//===-- Interfaces.td - Interfaces definition file ------------------*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This file contains definations for Interfaces.
+// This file contains definitions for Interfaces.
 //
 //===----------------------------------------------------------------------===//
 
diff --git a/mlir/include/mlir/IR/Properties.td b/mlir/include/mlir/IR/Properties.td
index 2830ba96fb787..89e232feee617 100644
--- a/mlir/include/mlir/IR/Properties.td
+++ b/mlir/include/mlir/IR/Properties.td
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This is the base properties defination file.
+// This is the base properties definition file.
 //
 //===----------------------------------------------------------------------===//
 
diff --git a/mlir/include/mlir/IR/Traits.td b/mlir/include/mlir/IR/Traits.td
index 221f8633ad954..41e745674f605 100644
--- a/mlir/include/mlir/IR/Traits.td
+++ b/mlir/include/mlir/IR/Traits.td
@@ -1,4 +1,4 @@
-//===-- Traits.td - Trait definations file ------------------*- tablegen -*-===//
+//===-- Traits.td - Trait definitions file ------------------*- tablegen -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -6,7 +6,7 @@
 //
 //===----------------------------------------------------------------------===//
 //
-// This file contains definations for traits.
+// This file contains definitions for traits.
 //
 //===----------------------------------------------------------------------===//
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/208138

From llvmlistbot at llvm.org  Tue Jul  7 21:34:06 2026
From: llvmlistbot at llvm.org (Zeyi Xu)
Date: Tue, 07 Jul 2026 21:34:06 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] [NFC] Fix
 spelling of definition in comments (PR #208138)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208138@github.com>
Message-ID: <6a4dd33e.30759430.24d50c.8b90@mx.google.com>


https://github.com/zeyi2 approved this pull request.


https://github.com/llvm/llvm-project/pull/208138

From llvmlistbot at llvm.org  Tue Jul  7 21:50:19 2026
From: llvmlistbot at llvm.org (Jacky Li)
Date: Tue, 07 Jul 2026 21:50:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Document and assert Liveness coverage
 preconditions (PR #208142)
Message-ID: <llvm.org/llvm/llvm-project/pull/208142@github.com>


https://github.com/JPL11 created https://github.com/llvm/llvm-project/pull/208142

`Liveness::getLiveIn`, `getLiveOut`, `isDeadAfter` and `resolveLiveness` dereference `getLiveness(block)` without a null check, so querying a block that was created after the analysis was constructed (e.g. by splitting a critical edge, or by signature conversion during dialect conversion) crashes with a plain nullptr dereference. I hit this twice downstream in IREE while replacing its own liveness implementation with `mlir::Liveness` (iree-org/iree#24687); in both cases the symptom was a segfault deep inside the query with no hint that the analysis was stale.

This PR:

- documents the precondition on the class and the affected methods: queries expect blocks that were covered by the analysis at construction time, and `getLiveness` (which returns nullptr for unknown blocks) can be used to check coverage;
- turns the nullptr dereferences into assertions with an actionable message;
- adds a unit test covering `getLiveness`'s nullptr contract for blocks created after construction;
- fixes an `isDeafAfter` typo in the class doc comment.

I deliberately did not make the queries gracefully return empty/conservative answers for unknown blocks: that is a semantic change that can silently hide stale-analysis bugs (IREE's old implementation did exactly that and it masked a real bug for years). If there's appetite for recomputing on demand instead, happy to discuss on the issue.

Fixes #208140

Per the LLVM AI tools policy: I used Claude Code to help draft this patch; I reviewed and tested everything myself (`check-mlir` and the new `MLIRAnalysisTests` locally).


>From f03e9be7ecfaf8800acc79eda3fd2c25842522cf Mon Sep 17 00:00:00 2001
From: Jacky Li <jackydli95 at gmail.com>
Date: Tue, 7 Jul 2026 21:53:07 -0700
Subject: [PATCH] [mlir] Document and assert Liveness coverage preconditions

Liveness::getLiveIn, getLiveOut, isDeadAfter and resolveLiveness
dereference getLiveness(block) without a null check, so querying a
block that was created after the analysis was constructed (e.g. by
splitting a critical edge, or by signature conversion during dialect
conversion) crashes with a plain nullptr dereference.

Document the precondition on the class and the affected methods, turn
the crashes into assertions with an actionable message, and add a unit
test for getLiveness's nullptr contract. Also fix an isDeafAfter typo
in the class doc comment.

Fixes #208140

Assisted-by: Claude Code
---
 mlir/include/mlir/Analysis/Liveness.h    | 17 +++++-
 mlir/lib/Analysis/Liveness.cpp           | 16 +++--
 mlir/unittests/Analysis/CMakeLists.txt   |  9 +++
 mlir/unittests/Analysis/LivenessTest.cpp | 74 ++++++++++++++++++++++++
 4 files changed, 110 insertions(+), 6 deletions(-)
 create mode 100644 mlir/unittests/Analysis/LivenessTest.cpp

diff --git a/mlir/include/mlir/Analysis/Liveness.h b/mlir/include/mlir/Analysis/Liveness.h
index ca4d8dfab3c91..c56762586251a 100644
--- a/mlir/include/mlir/Analysis/Liveness.h
+++ b/mlir/include/mlir/Analysis/Liveness.h
@@ -43,7 +43,13 @@ class Value;
 ///   auto &allInValues = liveness.getLiveIn(block);
 ///   auto &allOutValues = liveness.getLiveOut(block);
 ///   auto allOperationsInWhichValueIsLive = liveness.resolveLiveness(value);
-///   bool isDeafAfter = liveness.isDeadAfter(value, operation);
+///   bool isDeadAfter = liveness.isDeadAfter(value, operation);
+///
+/// The analysis is computed once at construction time. All queries expect
+/// blocks and operations that were part of the analyzed operation when the
+/// analysis was constructed; querying blocks created afterwards (e.g. by a
+/// rewrite) is not supported. `getLiveness` returns nullptr for such blocks
+/// and can be used to check whether a block is covered.
 class Liveness {
 public:
   using OperationListT = std::vector<Operation *>;
@@ -63,18 +69,25 @@ class Liveness {
   /// Note that the operations in this list are not ordered and the current
   /// implementation is computationally expensive (as it iterates over all
   /// blocks in which the given value is live).
+  /// All blocks in which the value is live must be covered by the analysis
+  /// (see `getLiveness`).
   OperationListT resolveLiveness(Value value) const;
 
-  /// Gets liveness info (if any) for the block.
+  /// Gets liveness info (if any) for the block. Returns nullptr if the block
+  /// was not covered by the analysis at construction time.
   const LivenessBlockInfo *getLiveness(Block *block) const;
 
   /// Returns a reference to a set containing live-in values (unordered).
+  /// The block must be covered by the analysis (see `getLiveness`).
   const ValueSetT &getLiveIn(Block *block) const;
 
   /// Returns a reference to a set containing live-out values (unordered).
+  /// The block must be covered by the analysis (see `getLiveness`).
   const ValueSetT &getLiveOut(Block *block) const;
 
   /// Returns true if `value` is not live after `operation`.
+  /// The operation's block must be covered by the analysis (see
+  /// `getLiveness`).
   bool isDeadAfter(Value value, Operation *operation) const;
 
   /// Dumps the liveness information in a human readable format.
diff --git a/mlir/lib/Analysis/Liveness.cpp b/mlir/lib/Analysis/Liveness.cpp
index 3fcc0b5a6ffef..e784b97bcaf84 100644
--- a/mlir/lib/Analysis/Liveness.cpp
+++ b/mlir/lib/Analysis/Liveness.cpp
@@ -200,6 +200,7 @@ Liveness::OperationListT Liveness::resolveLiveness(Value value) const {
     // Get block and block liveness information.
     Block *block = toProcess.pop_back_val();
     const LivenessBlockInfo *blockInfo = getLiveness(block);
+    assert(blockInfo && "expected block to be covered by the analysis");
 
     // Note that start and end will be in the same block.
     Operation *start = blockInfo->getStartOperation(value);
@@ -212,8 +213,10 @@ Liveness::OperationListT Liveness::resolveLiveness(Value value) const {
     }
 
     for (Block *successor : block->getSuccessors()) {
-      if (getLiveness(successor)->isLiveIn(value) &&
-          visited.insert(successor).second)
+      const LivenessBlockInfo *successorInfo = getLiveness(successor);
+      assert(successorInfo &&
+             "expected successor to be covered by the analysis");
+      if (successorInfo->isLiveIn(value) && visited.insert(successor).second)
         toProcess.push_back(successor);
     }
   }
@@ -229,18 +232,23 @@ const LivenessBlockInfo *Liveness::getLiveness(Block *block) const {
 
 /// Returns a reference to a set containing live-in values.
 const Liveness::ValueSetT &Liveness::getLiveIn(Block *block) const {
-  return getLiveness(block)->in();
+  const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
+  return blockInfo->in();
 }
 
 /// Returns a reference to a set containing live-out values.
 const Liveness::ValueSetT &Liveness::getLiveOut(Block *block) const {
-  return getLiveness(block)->out();
+  const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
+  return blockInfo->out();
 }
 
 /// Returns true if `value` is not live after `operation`.
 bool Liveness::isDeadAfter(Value value, Operation *operation) const {
   Block *block = operation->getBlock();
   const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
 
   // The given value escapes the associated block.
   if (blockInfo->isLiveOut(value))
diff --git a/mlir/unittests/Analysis/CMakeLists.txt b/mlir/unittests/Analysis/CMakeLists.txt
index a05d09f89070c..2af936fdc855a 100644
--- a/mlir/unittests/Analysis/CMakeLists.txt
+++ b/mlir/unittests/Analysis/CMakeLists.txt
@@ -1 +1,10 @@
+add_mlir_unittest(MLIRAnalysisTests
+  LivenessTest.cpp
+)
+
+mlir_target_link_libraries(MLIRAnalysisTests
+  PRIVATE MLIRAnalysis
+  MLIRParser
+  )
+
 add_subdirectory(Presburger)
diff --git a/mlir/unittests/Analysis/LivenessTest.cpp b/mlir/unittests/Analysis/LivenessTest.cpp
new file mode 100644
index 0000000000000..075252cb2d961
--- /dev/null
+++ b/mlir/unittests/Analysis/LivenessTest.cpp
@@ -0,0 +1,74 @@
+//===- LivenessTest.cpp - Liveness analysis unit tests --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Analysis/Liveness.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/Parser/Parser.h"
+
+#include "gtest/gtest.h"
+
+using namespace mlir;
+
+namespace {
+
+const StringLiteral moduleStr = R"mlir(
+"test.func"() ({
+^bb0:
+  %0 = "test.def"() : () -> i32
+  "test.br"()[^bb1] : () -> ()
+^bb1:
+  "test.use"(%0) : (i32) -> ()
+  "test.ret"() : () -> ()
+}) : () -> ()
+)mlir";
+
+TEST(LivenessTest, CoveredBlocks) {
+  MLIRContext context;
+  context.allowUnregisteredDialects();
+  OwningOpRef<ModuleOp> module =
+      parseSourceString<ModuleOp>(moduleStr, &context);
+  ASSERT_TRUE(module);
+
+  Region &region = module->getBody()->getOperations().front().getRegion(0);
+  Block &entryBlock = region.front();
+  Block &secondBlock = region.back();
+
+  Liveness liveness(module.get());
+
+  // Blocks that existed when the analysis was constructed are covered.
+  EXPECT_NE(liveness.getLiveness(&entryBlock), nullptr);
+  EXPECT_NE(liveness.getLiveness(&secondBlock), nullptr);
+
+  Value def = entryBlock.front().getResult(0);
+  EXPECT_TRUE(liveness.getLiveOut(&entryBlock).contains(def));
+  EXPECT_TRUE(liveness.getLiveIn(&secondBlock).contains(def));
+  EXPECT_FALSE(liveness.isDeadAfter(def, &entryBlock.front()));
+  EXPECT_TRUE(liveness.isDeadAfter(def, &secondBlock.front()));
+  EXPECT_EQ(liveness.resolveLiveness(def).size(), 3u);
+}
+
+TEST(LivenessTest, BlockCreatedAfterConstructionIsNotCovered) {
+  MLIRContext context;
+  context.allowUnregisteredDialects();
+  OwningOpRef<ModuleOp> module =
+      parseSourceString<ModuleOp>(moduleStr, &context);
+  ASSERT_TRUE(module);
+
+  Region &region = module->getBody()->getOperations().front().getRegion(0);
+  Block &secondBlock = region.back();
+
+  Liveness liveness(module.get());
+
+  // Blocks created after the analysis was constructed (e.g. by a rewrite)
+  // are not covered; `getLiveness` returns nullptr for them.
+  Block *newBlock = secondBlock.splitBlock(&secondBlock.front());
+  EXPECT_EQ(liveness.getLiveness(newBlock), nullptr);
+  EXPECT_NE(liveness.getLiveness(&secondBlock), nullptr);
+}
+
+} // namespace


From llvmlistbot at llvm.org  Tue Jul  7 21:50:27 2026
From: llvmlistbot at llvm.org (Jacky Li)
Date: Tue, 07 Jul 2026 21:50:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Document and assert Liveness coverage
 preconditions (PR #208142)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208142@github.com>
Message-ID: <6a4dd713.8b87e04f.184605.5d11@mx.google.com>


JPL11 wrote:

@joker-eph @Mogball @matthias-springer would appreciate a look when you have a moment — small analysis-robustness patch (docs + asserts + unit test, no functional change for covered blocks). Context in #208140.

https://github.com/llvm/llvm-project/pull/208142

From llvmlistbot at llvm.org  Tue Jul  7 21:50:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 21:50:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Document and assert Liveness coverage
 preconditions (PR #208142)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208142@github.com>
Message-ID: <6a4dd718.e55dc341.1f19bd.975e@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @JPL11 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/208142

From llvmlistbot at llvm.org  Tue Jul  7 21:51:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 07 Jul 2026 21:51:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Document and assert Liveness coverage
 preconditions (PR #208142)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208142@github.com>
Message-ID: <6a4dd75c.8e75e0cb.5e108.ba20@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jacky Li (JPL11)

<details>
<summary>Changes</summary>

`Liveness::getLiveIn`, `getLiveOut`, `isDeadAfter` and `resolveLiveness` dereference `getLiveness(block)` without a null check, so querying a block that was created after the analysis was constructed (e.g. by splitting a critical edge, or by signature conversion during dialect conversion) crashes with a plain nullptr dereference. I hit this twice downstream in IREE while replacing its own liveness implementation with `mlir::Liveness` (iree-org/iree#<!-- -->24687); in both cases the symptom was a segfault deep inside the query with no hint that the analysis was stale.

This PR:

- documents the precondition on the class and the affected methods: queries expect blocks that were covered by the analysis at construction time, and `getLiveness` (which returns nullptr for unknown blocks) can be used to check coverage;
- turns the nullptr dereferences into assertions with an actionable message;
- adds a unit test covering `getLiveness`'s nullptr contract for blocks created after construction;
- fixes an `isDeafAfter` typo in the class doc comment.

I deliberately did not make the queries gracefully return empty/conservative answers for unknown blocks: that is a semantic change that can silently hide stale-analysis bugs (IREE's old implementation did exactly that and it masked a real bug for years). If there's appetite for recomputing on demand instead, happy to discuss on the issue.

Fixes #<!-- -->208140

Per the LLVM AI tools policy: I used Claude Code to help draft this patch; I reviewed and tested everything myself (`check-mlir` and the new `MLIRAnalysisTests` locally).


---
Full diff: https://github.com/llvm/llvm-project/pull/208142.diff


4 Files Affected:

- (modified) mlir/include/mlir/Analysis/Liveness.h (+15-2) 
- (modified) mlir/lib/Analysis/Liveness.cpp (+12-4) 
- (modified) mlir/unittests/Analysis/CMakeLists.txt (+9) 
- (added) mlir/unittests/Analysis/LivenessTest.cpp (+74) 


``````````diff
diff --git a/mlir/include/mlir/Analysis/Liveness.h b/mlir/include/mlir/Analysis/Liveness.h
index ca4d8dfab3c91..c56762586251a 100644
--- a/mlir/include/mlir/Analysis/Liveness.h
+++ b/mlir/include/mlir/Analysis/Liveness.h
@@ -43,7 +43,13 @@ class Value;
 ///   auto &allInValues = liveness.getLiveIn(block);
 ///   auto &allOutValues = liveness.getLiveOut(block);
 ///   auto allOperationsInWhichValueIsLive = liveness.resolveLiveness(value);
-///   bool isDeafAfter = liveness.isDeadAfter(value, operation);
+///   bool isDeadAfter = liveness.isDeadAfter(value, operation);
+///
+/// The analysis is computed once at construction time. All queries expect
+/// blocks and operations that were part of the analyzed operation when the
+/// analysis was constructed; querying blocks created afterwards (e.g. by a
+/// rewrite) is not supported. `getLiveness` returns nullptr for such blocks
+/// and can be used to check whether a block is covered.
 class Liveness {
 public:
   using OperationListT = std::vector<Operation *>;
@@ -63,18 +69,25 @@ class Liveness {
   /// Note that the operations in this list are not ordered and the current
   /// implementation is computationally expensive (as it iterates over all
   /// blocks in which the given value is live).
+  /// All blocks in which the value is live must be covered by the analysis
+  /// (see `getLiveness`).
   OperationListT resolveLiveness(Value value) const;
 
-  /// Gets liveness info (if any) for the block.
+  /// Gets liveness info (if any) for the block. Returns nullptr if the block
+  /// was not covered by the analysis at construction time.
   const LivenessBlockInfo *getLiveness(Block *block) const;
 
   /// Returns a reference to a set containing live-in values (unordered).
+  /// The block must be covered by the analysis (see `getLiveness`).
   const ValueSetT &getLiveIn(Block *block) const;
 
   /// Returns a reference to a set containing live-out values (unordered).
+  /// The block must be covered by the analysis (see `getLiveness`).
   const ValueSetT &getLiveOut(Block *block) const;
 
   /// Returns true if `value` is not live after `operation`.
+  /// The operation's block must be covered by the analysis (see
+  /// `getLiveness`).
   bool isDeadAfter(Value value, Operation *operation) const;
 
   /// Dumps the liveness information in a human readable format.
diff --git a/mlir/lib/Analysis/Liveness.cpp b/mlir/lib/Analysis/Liveness.cpp
index 3fcc0b5a6ffef..e784b97bcaf84 100644
--- a/mlir/lib/Analysis/Liveness.cpp
+++ b/mlir/lib/Analysis/Liveness.cpp
@@ -200,6 +200,7 @@ Liveness::OperationListT Liveness::resolveLiveness(Value value) const {
     // Get block and block liveness information.
     Block *block = toProcess.pop_back_val();
     const LivenessBlockInfo *blockInfo = getLiveness(block);
+    assert(blockInfo && "expected block to be covered by the analysis");
 
     // Note that start and end will be in the same block.
     Operation *start = blockInfo->getStartOperation(value);
@@ -212,8 +213,10 @@ Liveness::OperationListT Liveness::resolveLiveness(Value value) const {
     }
 
     for (Block *successor : block->getSuccessors()) {
-      if (getLiveness(successor)->isLiveIn(value) &&
-          visited.insert(successor).second)
+      const LivenessBlockInfo *successorInfo = getLiveness(successor);
+      assert(successorInfo &&
+             "expected successor to be covered by the analysis");
+      if (successorInfo->isLiveIn(value) && visited.insert(successor).second)
         toProcess.push_back(successor);
     }
   }
@@ -229,18 +232,23 @@ const LivenessBlockInfo *Liveness::getLiveness(Block *block) const {
 
 /// Returns a reference to a set containing live-in values.
 const Liveness::ValueSetT &Liveness::getLiveIn(Block *block) const {
-  return getLiveness(block)->in();
+  const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
+  return blockInfo->in();
 }
 
 /// Returns a reference to a set containing live-out values.
 const Liveness::ValueSetT &Liveness::getLiveOut(Block *block) const {
-  return getLiveness(block)->out();
+  const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
+  return blockInfo->out();
 }
 
 /// Returns true if `value` is not live after `operation`.
 bool Liveness::isDeadAfter(Value value, Operation *operation) const {
   Block *block = operation->getBlock();
   const LivenessBlockInfo *blockInfo = getLiveness(block);
+  assert(blockInfo && "expected block to be covered by the analysis");
 
   // The given value escapes the associated block.
   if (blockInfo->isLiveOut(value))
diff --git a/mlir/unittests/Analysis/CMakeLists.txt b/mlir/unittests/Analysis/CMakeLists.txt
index a05d09f89070c..2af936fdc855a 100644
--- a/mlir/unittests/Analysis/CMakeLists.txt
+++ b/mlir/unittests/Analysis/CMakeLists.txt
@@ -1 +1,10 @@
+add_mlir_unittest(MLIRAnalysisTests
+  LivenessTest.cpp
+)
+
+mlir_target_link_libraries(MLIRAnalysisTests
+  PRIVATE MLIRAnalysis
+  MLIRParser
+  )
+
 add_subdirectory(Presburger)
diff --git a/mlir/unittests/Analysis/LivenessTest.cpp b/mlir/unittests/Analysis/LivenessTest.cpp
new file mode 100644
index 0000000000000..075252cb2d961
--- /dev/null
+++ b/mlir/unittests/Analysis/LivenessTest.cpp
@@ -0,0 +1,74 @@
+//===- LivenessTest.cpp - Liveness analysis unit tests --------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Analysis/Liveness.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/Parser/Parser.h"
+
+#include "gtest/gtest.h"
+
+using namespace mlir;
+
+namespace {
+
+const StringLiteral moduleStr = R"mlir(
+"test.func"() ({
+^bb0:
+  %0 = "test.def"() : () -> i32
+  "test.br"()[^bb1] : () -> ()
+^bb1:
+  "test.use"(%0) : (i32) -> ()
+  "test.ret"() : () -> ()
+}) : () -> ()
+)mlir";
+
+TEST(LivenessTest, CoveredBlocks) {
+  MLIRContext context;
+  context.allowUnregisteredDialects();
+  OwningOpRef<ModuleOp> module =
+      parseSourceString<ModuleOp>(moduleStr, &context);
+  ASSERT_TRUE(module);
+
+  Region &region = module->getBody()->getOperations().front().getRegion(0);
+  Block &entryBlock = region.front();
+  Block &secondBlock = region.back();
+
+  Liveness liveness(module.get());
+
+  // Blocks that existed when the analysis was constructed are covered.
+  EXPECT_NE(liveness.getLiveness(&entryBlock), nullptr);
+  EXPECT_NE(liveness.getLiveness(&secondBlock), nullptr);
+
+  Value def = entryBlock.front().getResult(0);
+  EXPECT_TRUE(liveness.getLiveOut(&entryBlock).contains(def));
+  EXPECT_TRUE(liveness.getLiveIn(&secondBlock).contains(def));
+  EXPECT_FALSE(liveness.isDeadAfter(def, &entryBlock.front()));
+  EXPECT_TRUE(liveness.isDeadAfter(def, &secondBlock.front()));
+  EXPECT_EQ(liveness.resolveLiveness(def).size(), 3u);
+}
+
+TEST(LivenessTest, BlockCreatedAfterConstructionIsNotCovered) {
+  MLIRContext context;
+  context.allowUnregisteredDialects();
+  OwningOpRef<ModuleOp> module =
+      parseSourceString<ModuleOp>(moduleStr, &context);
+  ASSERT_TRUE(module);
+
+  Region &region = module->getBody()->getOperations().front().getRegion(0);
+  Block &secondBlock = region.back();
+
+  Liveness liveness(module.get());
+
+  // Blocks created after the analysis was constructed (e.g. by a rewrite)
+  // are not covered; `getLiveness` returns nullptr for them.
+  Block *newBlock = secondBlock.splitBlock(&secondBlock.front());
+  EXPECT_EQ(liveness.getLiveness(newBlock), nullptr);
+  EXPECT_NE(liveness.getLiveness(&secondBlock), nullptr);
+}
+
+} // namespace

``````````

</details>


https://github.com/llvm/llvm-project/pull/208142

From llvmlistbot at llvm.org  Tue Jul  7 23:00:55 2026
From: llvmlistbot at llvm.org (Pradeep Kumar)
Date: Tue, 07 Jul 2026 23:00:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4de797.abb10101.3d0bf8.e000@mx.google.com>


schwarzschild-radius wrote:

@weimin023 Thanks for your change! I believe you are missing tests for 2 x f16x2 results type? Otherwise, LGTM

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Tue Jul  7 23:15:28 2026
From: llvmlistbot at llvm.org (Pradeep Kumar)
Date: Tue, 07 Jul 2026 23:15:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4deb00.3cce8983.225732.0aa5@mx.google.com>


https://github.com/schwarzschild-radius approved this pull request.


https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Wed Jul  8 00:45:56 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Wed, 08 Jul 2026 00:45:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4e0034.2020f6c2.243e78.2693@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Wed Jul  8 01:40:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 01:40:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] ea6d5c6 - [MLIR][XeGPU] Fix blocking pass for
 scf.if distribution (#207060)
Message-ID: <6a4e0d1b.13be6df8.145c.ac07@mx.google.com>


Author: Nishant Patel
Date: 2026-07-08T10:40:47+02:00
New Revision: ea6d5c6a2e37705688c74167990115e535058065

URL: https://github.com/llvm/llvm-project/commit/ea6d5c6a2e37705688c74167990115e535058065
DIFF: https://github.com/llvm/llvm-project/commit/ea6d5c6a2e37705688c74167990115e535058065.diff

LOG: [MLIR][XeGPU] Fix blocking pass for scf.if distribution (#207060)

Added: 
    

Modified: 
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
    mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
    mlir/test/Dialect/XeGPU/xegpu-blocking.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
index 4f81085c21e86..57b8ac73f41ac 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
@@ -430,7 +430,7 @@ void XeGPUBlockingPass::runOnOperation() {
     // `layout_result_N` lands on the wrong (renumbered) result, corrupting the
     // count invariant and leaving the loop illegal.
     op->walk([](Operation *loopOp) {
-      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp>(loopOp))
+      if (!isa<scf::ForOp, scf::WhileOp, scf::ConditionOp, scf::IfOp>(loopOp))
         return;
       SmallVector<StringRef> toRemove;
       for (const NamedAttribute &attr : loopOp->getAttrs()) {

diff  --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 1aad2aa77741b..9620e21f9bfdf 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -862,12 +862,13 @@ bool xegpu::matchSplitDimExpansion(
 //===----------------------------------------------------------------------===//
 
 // Pre-computes distributed VectorType mappings for every value carried through
-// an SCF loop (scf.while, scf.for): block args (iter_args /
-// before-/after-args), loop results, and the terminator operands feeding them.
+// an SCF region-branch op (scf.while, scf.for, scf.if): block args (iter_args /
+// before-/after-args), op results, and the terminator operands feeding them.
 // These positions share one logical value and must convert identically, so each
 // is derived from a single source -- the layout of the feeding value (loop
-// init, or `scf.condition` operand) -- via `getDistributeLayoutAttr(Value)`,
-// and keyed by `Value`. Keying by Value is required because the SCF converters
+// init, `scf.condition` operand, or `scf.if` result) -- via
+// `getDistributeLayoutAttr(Value)`, and keyed by `Value`. Keying by Value is
+// required because the SCF converters
 // detach/replace the loop body mid-conversion (scf.while detaches before/after
 // blocks -> a detached-arg layout query trips an ilist assertion; scf.for
 // rebuilds the op, which loses the temporary `layout_operand_N` attrs -> the
@@ -924,6 +925,19 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
         recordTypes(init, {arg, res, yieldVal});
       return;
     }
+    if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
+      // Each result and its then/else yield operands share one position and
+      // must convert identically; derive all from the result's layout.
+      scf::YieldOp thenYield = ifOp.thenYield();
+      scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
+      for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
+        SmallVector<Value> dests{res, thenYield.getOperand(idx)};
+        if (elseYield)
+          dests.push_back(elseYield.getOperand(idx));
+        recordTypes(res, dests);
+      }
+      return;
+    }
   });
   return loopArgTypes;
 }

diff  --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index f2c25f45f45bc..b0b16c2adba6e 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -815,3 +815,31 @@ gpu.module @test_kernel {
     gpu.return
   }
 }
+
+// -----
+// Test that an scf.if whose results expand 1:N during blocking is legalized.
+#if_a = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+gpu.module @test_kernel {
+  // CHECK-LABEL: func @if_one_to_n_results
+  gpu.func @if_one_to_n_results(%c: i1, %p: i64) {
+    %c0 = arith.constant 0 : index
+    // CHECK: scf.if
+    // CHECK-COUNT-8: vector<8x16xf32>
+    %r:2 = scf.if %c -> (vector<16x64xf32>, vector<16xf32>) {
+      %a = arith.constant dense<1.0> : vector<16x64xf32>
+      %b = arith.constant dense<1.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    } else {
+      %a = arith.constant dense<0.0> : vector<16x64xf32>
+      %b = arith.constant dense<0.0> : vector<16xf32>
+      scf.yield %a, %b : vector<16x64xf32>, vector<16xf32>
+    }
+    %bc = vector.broadcast %r#1 : vector<16xf32> to vector<64x16xf32>
+    %tp = vector.transpose %bc, [1, 0] : vector<64x16xf32> to vector<16x64xf32>
+    %add = arith.addf %r#0, %tp : vector<16x64xf32>
+    %tr = arith.truncf %add : vector<16x64xf32> to vector<16x64xf16>
+    %td = xegpu.create_nd_tdesc %p, shape:[16, 64], strides:[64, 1] : i64 -> !xegpu.tensor_desc<16x64xf16, #if_a>
+    xegpu.store_nd %tr, %td[%c0, %c0] <{layout = #if_a}> : vector<16x64xf16>, !xegpu.tensor_desc<16x64xf16, #if_a>
+    gpu.return
+  }
+}


        

From llvmlistbot at llvm.org  Wed Jul  8 01:41:08 2026
From: llvmlistbot at llvm.org (Dmitry Chigarev)
Date: Wed, 08 Jul 2026 01:41:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix blocking pass for scf.if
	distribution (PR #207060)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207060@github.com>
Message-ID: <6a4e0d24.a87eb5dc.7d817.f882@mx.google.com>


https://github.com/dchigarev closed https://github.com/llvm/llvm-project/pull/207060

From llvmlistbot at llvm.org  Wed Jul  8 01:43:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 01:43:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] b5ed0ae - [mlir][bufferization] Add best-fit
 algorithm to static memory planner (#207403)
Message-ID: <6a4e0db0.481afadf.196d57.3f5e@mx.google.com>


Author: Javed Absar
Date: 2026-07-08T09:43:15+01:00
New Revision: b5ed0ae6bc70792cb9749444b2803c3fa72276e4

URL: https://github.com/llvm/llvm-project/commit/b5ed0ae6bc70792cb9749444b2803c3fa72276e4
DIFF: https://github.com/llvm/llvm-project/commit/b5ed0ae6bc70792cb9749444b2803c3fa72276e4.diff

LOG: [mlir][bufferization] Add best-fit algorithm to static memory planner (#207403)

Introduces an algorithm selection option to the static memory planner
pass and adds a best-fit algorithm that reuses memory from expired allocations by
finding the smallest suitable gap.

I have tested the correctness of the best-fit algorithm with an
independent randomized verifier.

Signed-off-by: mabsar <mabsar at qti.qualcomm.com>

Added: 
    mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
    mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
    mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir

Modified: 
    mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
    mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
    mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
    mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
index bafa84645e57b..a8cad26e9341b 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizationEnums.td
@@ -24,4 +24,12 @@ def LayoutMapOption : I32EnumAttr<"LayoutMapOption",
   let cppNamespace = "::mlir::bufferization";
 }
 
+def MemoryPlannerAlgorithm : I32EnumAttr<"MemoryPlannerAlgorithm",
+                                          "memory planning algorithm", [
+  I32EnumAttrCase<"Trivial", 0, "trivial">,
+  I32EnumAttrCase<"BestFit", 1, "best-fit">
+]> {
+  let cppNamespace = "::mlir::bufferization";
+}
+
 #endif // BUFFERIZATION_ENUMS

diff  --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
index c7f6cae571e40..8408315dda607 100644
--- a/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/Passes.td
@@ -197,13 +197,11 @@ def StaticMemoryPlannerAnalysisPass
     - Unique same-block `memref.dealloc`.
     - Allocations in nested blocks are ignored for now.
 
-    Eligible allocations are packed into a single arena. Currently, we use a
-    trivial sequential allocation strategy with alignment padding. But the
-    interface will also allow any packing algorithm to be plugged in as long as
-    it respects the interface. The arena is an i8 byte buffer
-    (`memref<Nxi8>`) that can hold allocations of 
diff erent element types.
-    Each original allocation is replaced with a `memref.view` operation that
-    creates a typed view into the arena at the computed offset.
+    Eligible allocations are packed into a single arena using a configurable
+    planning algorithm (see the `algorithm` option). The arena is an i8 byte
+    buffer (`memref<Nxi8>`) that can hold allocations of 
diff erent element
+    types. Each original allocation is replaced with a `memref.view` operation
+    that creates a typed view into the arena at the computed offset.
 
     Ineligible allocations are skipped and retain their original
     alloc/dealloc operations. Skip reasons are reported via op remarks.
@@ -234,11 +232,24 @@ def StaticMemoryPlannerAnalysisPass
     ```
   }];
 
-  let options = [Option<
-      "arenaMode", "arena-mode", "std::string",
+  let options = [
+    Option<"arenaMode", "arena-mode", "std::string",
       /*default=*/"\"allocate\"",
       "Arena allocation mode: 'allocate' creates arena via AllocOp, "
-      "'arg' extracts arena from function arguments">];
+      "'arg' extracts arena from function arguments">,
+    Option<"algorithm", "algorithm",
+      "::mlir::bufferization::MemoryPlannerAlgorithm",
+      /*default=*/"::mlir::bufferization::MemoryPlannerAlgorithm::Trivial",
+      "Memory planning algorithm to use.",
+      [{::llvm::cl::values(
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::Trivial,
+                   "trivial",
+                   "Sequential packing without lifetime overlap"),
+        clEnumValN(::mlir::bufferization::MemoryPlannerAlgorithm::BestFit,
+                   "best-fit",
+                   "Best-fit packing with lifetime-aware gap reuse")
+      )}]>,
+  ];
 
   let statistics =
       [Statistic<"numEligible", "num-eligible",

diff  --git a/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
new file mode 100644
index 0000000000000..626e384ce4642
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h
@@ -0,0 +1,50 @@
+//===- StaticMemoryPlanning.h - Memory planning algorithms ------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Pure memory planning algorithms for static arena allocation. These operate
+// on abstract allocation descriptors (size, alignment, lifetime) and produce
+// byte offsets. They are independent of MLIR IR.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+#define MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallVector.h"
+#include <cstdint>
+
+namespace mlir {
+namespace bufferization {
+
+/// Descriptor for a single allocation to be placed by the memory planner.
+struct MemoryPlannerAlloc {
+  int64_t sizeInBytes = 0;
+  int64_t alignment = 1;
+  int64_t timeStart = 0; // Operation index when allocation becomes live
+  int64_t timeEnd = 0;   // Operation index when allocation is freed
+};
+
+/// Sequential packing without lifetime overlap. Each allocation is placed
+/// immediately after the previous one (with alignment padding). Ignores
+/// lifetimes entirely.
+llvm::SmallVector<int64_t>
+trivialMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+/// Best-fit packing with lifetime-aware gap reuse. Processes allocations in
+/// time order and places each one in the smallest gap left by expired
+/// allocations. Falls back to extending the arena if no gap fits.
+llvm::SmallVector<int64_t>
+bestFitMemoryPlanner(int64_t arenaAlignment,
+                     llvm::ArrayRef<MemoryPlannerAlloc> allocs);
+
+} // namespace bufferization
+} // namespace mlir
+
+#endif // MLIR_DIALECT_BUFFERIZATION_TRANSFORMS_STATICMEMORYPLANNING_H

diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
index 5df9f19a5e30a..006fcd1ce0ec7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Bufferization/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRBufferizationTransforms
   OwnershipBasedBufferDeallocation.cpp
   TensorCopyInsertion.cpp
   OptimizeAllocationLiveness.cpp
+  StaticMemoryPlanning.cpp
   StaticMemoryPlannerAnalysis.cpp
 
   ADDITIONAL_HEADER_DIRS

diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 48b6e97c809ef..c2ac40a8427e7 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -7,18 +7,18 @@
 //===----------------------------------------------------------------------===//
 //
 // Transforms memref.alloc/memref.dealloc pairs into a single arena allocation
-// with memref.view. Uses simple sequential offset assignment where each
-// allocation gets its own space without overlap (baseline algorithm).
+// with memref.view. Delegates offset computation to planning algorithms in
+// StaticMemoryPlanning.h.
 //
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/Support/Debug.h"
-#include "llvm/Support/MathExtras.h"
 #include <numeric>
 
 #define DEBUG_TYPE "static-memory-planner"
@@ -34,19 +34,6 @@ using namespace mlir;
 
 namespace {
 
-//===----------------------------------------------------------------------===//
-// Data structures
-//===----------------------------------------------------------------------===//
-
-/// Allocation info for memory planning (independent of MLIR).
-/// This can be used with pure planning algorithms.
-struct Alloc {
-  int64_t sizeInBytes = 0; // Size in bytes
-  int64_t alignment = 1;   // Required alignment in bytes
-  int64_t timeStart = 0;   // Operation index when allocation starts
-  int64_t timeEnd = 0;     // Operation index when allocation ends (dealloc)
-};
-
 /// A candidate allocation with its matching deallocation and assigned offset.
 struct AllocationCandidate {
   memref::AllocOp alloc;
@@ -81,36 +68,134 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
   return (numElements * elementSizeInBits + 7) / 8; // Round up to bytes
 }
 
-/// Align an offset to the specified alignment.
-/// Returns the smallest value >= offset that is a multiple of alignment.
-static int64_t alignOffset(int64_t offset, int64_t alignment) {
-  return llvm::alignTo(offset, alignment);
+/// Build lifetime-annotated allocation descriptors from candidates.
+/// Returns the arena alignment (LCM of all individual alignments).
+static int64_t buildAllocInfos(
+    MutableArrayRef<AllocationCandidate> candidates,
+    SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  int64_t arenaAlignment = 1;
+  for (auto &candidate : candidates) {
+    bufferization::MemoryPlannerAlloc info;
+    info.sizeInBytes = candidate.sizeInBytes;
+    info.alignment = candidate.alignment;
+
+    Block *block = candidate.alloc->getBlock();
+    int64_t opIdx = 0;
+    for (Operation &op : *block) {
+      if (&op == candidate.alloc.getOperation())
+        info.timeStart = opIdx;
+      if (&op == candidate.dealloc.getOperation())
+        info.timeEnd = opIdx;
+      ++opIdx;
+    }
+
+    allocInfos.push_back(info);
+    arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
+  }
+  return arenaAlignment;
 }
 
-//===----------------------------------------------------------------------===//
-// Memory Planning Algorithms
-//===----------------------------------------------------------------------===//
+/// Collect alloc/dealloc pairs eligible for arena placement.
+/// An allocation is eligible if it has a static shape and a unique dealloc
+/// in the same block.
+static SmallVector<AllocationCandidate>
+collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
+                  llvm::Statistic &numSkipNoDealloc,
+                  llvm::Statistic &numEligible) {
+  SmallVector<AllocationCandidate> candidates;
+
+  funcOp->walk([&](memref::AllocOp allocOp) {
+    MemRefType memrefType = allocOp.getType();
+
+    // Skip dynamic shapes
+    if (!memrefType.hasStaticShape()) {
+      ++numSkipDynamic;
+      return;
+    }
 
-/// Simple sequential memory planner (baseline algorithm).
-/// arenaAlignment must be a multiple (LCM) of all alloc.alignment values.
-/// Allocates each buffer one after another with proper alignment padding.
-/// Returns offsets in bytes for each allocation.
-static SmallVector<int64_t> trivialMemoryPlanner(int64_t arenaAlignment,
-                                                 ArrayRef<Alloc> allocs) {
-  SmallVector<int64_t> offsets;
-  int64_t currentOffset = 0;
-
-  for (const auto &alloc : allocs) {
-    currentOffset = alignOffset(currentOffset, alloc.alignment);
-#ifndef NDEBUG
-    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
-           "invalid alignment");
-#endif
-    offsets.push_back(currentOffset);
-    currentOffset += alloc.sizeInBytes;
+    // Find unique dealloc in the same block
+    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
+    if (!deallocOp) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    if (deallocOp->getBlock() != allocOp->getBlock()) {
+      ++numSkipNoDealloc;
+      return;
+    }
+
+    // This allocation is eligible
+    ++numEligible;
+    AllocationCandidate candidate;
+    candidate.alloc = allocOp;
+    candidate.dealloc = deallocOp;
+    candidate.sizeInBytes = computeSizeInBytes(memrefType);
+    candidate.alignment = allocOp.getAlignment().value_or(1);
+    candidates.push_back(candidate);
+  });
+
+  return candidates;
+}
+
+/// Create or obtain the arena buffer based on the arena mode.
+/// Returns failure if the mode is invalid or preconditions aren't met.
+static FailureOr<Value> createArena(OpBuilder &builder,
+                                    FunctionOpInterface funcOp,
+                                    StringRef arenaMode, int64_t totalSize,
+                                    int64_t arenaAlignment) {
+  Location loc = funcOp->getLoc();
+
+  if (arenaMode == "allocate") {
+    auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
+    auto arenaAlloc =
+        memref::AllocOp::create(builder, loc, arenaType, ValueRange{},
+                                builder.getI64IntegerAttr(arenaAlignment));
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] created arena via AllocOp: size="
+               << totalSize << " bytes, alignment=" << arenaAlignment
+               << " bytes\n");
+    return arenaAlloc.getResult();
+  }
+
+  if (arenaMode == "arg") {
+    if (funcOp.getNumArguments() == 0)
+      return funcOp->emitError(
+          "arena-mode=arg requires at least one function argument");
+
+    Value arenaValue = funcOp.getArgument(0);
+    auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
+    if (!arenaType || !arenaType.getElementType().isInteger(8) ||
+        arenaType.getRank() != 1)
+      return funcOp->emitError(
+          "arena-mode=arg requires first argument to be memref<...xi8>");
+
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] using arena from function arg 0\n");
+    return arenaValue;
   }
 
-  return offsets;
+  return funcOp->emitError("invalid arena-mode: '" + arenaMode +
+                           "' (must be 'allocate' or 'arg')");
+}
+
+/// Replace each alloc/dealloc pair with a memref.view into the arena.
+static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
+                               Value arenaValue) {
+  for (auto &candidate : candidates) {
+    OpBuilder builder(candidate.alloc);
+    Location loc = candidate.alloc.getLoc();
+    MemRefType originalType = candidate.alloc.getType();
+
+    Value offsetIndex =
+        arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+    auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
+                                       offsetIndex, SmallVector<Value>{});
+
+    candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
+    candidate.alloc.erase();
+    candidate.dealloc.erase();
+  }
 }
 
 //===----------------------------------------------------------------------===//
@@ -125,143 +210,64 @@ struct StaticMemoryPlannerAnalysisPass
       StaticMemoryPlannerAnalysisPass>;
   using Base::Base;
 
-  void runOnOperation() override {
-    auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
-
-    // Step 0: Check for memref return types (not supported)
-    for (Type resultType : funcOp.getResultTypes()) {
-      if (isa<BaseMemRefType>(resultType)) {
-        funcOp->emitError("static-memory-planner does not support functions "
-                          "with memref return types");
-        return signalPassFailure();
-      }
-    }
-
-    // Step 1: Collect eligible allocation candidates
-    SmallVector<AllocationCandidate> candidates;
-
-    funcOp->walk([&](memref::AllocOp allocOp) {
-      MemRefType memrefType = allocOp.getType();
-
-      // Skip dynamic shapes
-      if (!memrefType.hasStaticShape()) {
-        ++numSkipDynamic;
-        return;
-      }
-
-      // Find unique dealloc in the same block
-      memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-      if (!deallocOp) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      if (deallocOp->getBlock() != allocOp->getBlock()) {
-        ++numSkipNoDealloc;
-        return;
-      }
-
-      // This allocation is eligible
-      ++numEligible;
-      AllocationCandidate candidate;
-      candidate.alloc = allocOp;
-      candidate.dealloc = deallocOp;
-      candidate.sizeInBytes = computeSizeInBytes(memrefType);
-      candidate.alignment = allocOp.getAlignment().value_or(1);
-      candidates.push_back(candidate);
-    });
-
-    if (candidates.empty())
-      return;
-
-    // Step 2: Prepare allocation info for planner
-    SmallVector<Alloc> allocInfos;
-    int64_t arenaAlignment = 1;
-    for (const auto &candidate : candidates) {
-      Alloc allocInfo;
-      allocInfo.sizeInBytes = candidate.sizeInBytes;
-      allocInfo.alignment = candidate.alignment;
-      allocInfos.push_back(allocInfo);
-      arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
-    }
+  void runOnOperation() override;
+};
 
-    // Step 3: Run the planning algorithm
-    SmallVector<int64_t> offsets =
-        trivialMemoryPlanner(arenaAlignment, allocInfos);
-
-    // Assign computed offsets back to candidates
-    int64_t totalSize = 0;
-    for (size_t i = 0; i < candidates.size(); ++i) {
-      candidates[i].offset = offsets[i];
-      totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] offset=" << candidates[i].offset
-                 << " size=" << candidates[i].sizeInBytes
-                 << " alignment=" << candidates[i].alignment << "\n");
-    }
+void StaticMemoryPlannerAnalysisPass::runOnOperation() {
+  auto funcOp = llvm::cast<FunctionOpInterface>(getOperation());
 
-    // Step 4: Obtain arena based on arena mode
-    Operation *firstAlloc = candidates.front().alloc;
-    OpBuilder builder(firstAlloc);
-    Value arenaValue;
-
-    if (arenaMode == "allocate") {
-      // Arena is i8 byte buffer to support multiple data types (f32, i64, etc.)
-      auto arenaType = MemRefType::get({totalSize}, builder.getI8Type());
-      auto arenaAlloc = memref::AllocOp::create(
-          builder, firstAlloc->getLoc(), arenaType, ValueRange{},
-          builder.getI64IntegerAttr(arenaAlignment));
-      arenaValue = arenaAlloc.getResult();
-
-      LLVM_DEBUG(llvm::dbgs()
-                 << "[static-memory-planner] created arena via AllocOp: size="
-                 << totalSize << " bytes, alignment=" << arenaAlignment
-                 << " bytes\n");
-    } else if (arenaMode == "arg") {
-      if (funcOp.getNumArguments() == 0) {
-        funcOp->emitError(
-            "arena-mode=arg requires at least one function argument");
-        return signalPassFailure();
-      }
-
-      arenaValue = funcOp.getArgument(0);
-      auto arenaType = dyn_cast<MemRefType>(arenaValue.getType());
-      if (!arenaType || !arenaType.getElementType().isInteger(8) ||
-          arenaType.getRank() != 1) {
-        funcOp->emitError(
-            "arena-mode=arg requires first argument to be memref<...xi8>");
-        return signalPassFailure();
-      }
-
-      LLVM_DEBUG(
-          llvm::dbgs()
-          << "[static-memory-planner] using arena from function arg 0\n");
-    } else {
-      funcOp->emitError("invalid arena-mode: '" + arenaMode +
-                        "' (must be 'allocate' or 'arg')");
+  // Step 0: Check for memref return types (not supported)
+  for (Type resultType : funcOp.getResultTypes()) {
+    if (isa<BaseMemRefType>(resultType)) {
+      funcOp->emitError("static-memory-planner does not support functions "
+                        "with memref return types");
       return signalPassFailure();
     }
+  }
 
-    // Step 5: Replace each alloc with memref.view directly on arena
-    for (auto &candidate : candidates) {
-      builder.setInsertionPoint(candidate.alloc);
-      Location loc = candidate.alloc.getLoc();
+  // Step 1: Collect eligible allocation candidates.
+  SmallVector<AllocationCandidate> candidates =
+      collectCandidates(funcOp, numSkipDynamic, numSkipNoDealloc, numEligible);
 
-      MemRefType originalType = candidate.alloc.getType();
+  if (candidates.empty())
+    return;
 
-      // Create a constant for the byte offset into the arena
-      Value offsetIndex =
-          arith::ConstantIndexOp::create(builder, loc, candidate.offset);
+  // Step 2: Build allocation descriptors with lifetime info.
+  SmallVector<bufferization::MemoryPlannerAlloc> allocInfos;
+  int64_t arenaAlignment = buildAllocInfos(candidates, allocInfos);
 
-      // Use memref.view to create a typed view into the i8 arena
-      auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
-                                         offsetIndex, SmallVector<Value>{});
+  // Step 3: Run the planning algorithm.
+  SmallVector<int64_t> offsets;
+  switch (algorithm) {
+  case bufferization::MemoryPlannerAlgorithm::Trivial:
+    offsets = bufferization::trivialMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  case bufferization::MemoryPlannerAlgorithm::BestFit:
+    offsets = bufferization::bestFitMemoryPlanner(arenaAlignment, allocInfos);
+    break;
+  }
 
-      candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-      candidate.alloc.erase();
-      candidate.dealloc.erase();
-    }
+  // Step 4: Compute total arena size and assign offsets.
+  int64_t totalSize = 0;
+  for (size_t i = 0; i < candidates.size(); ++i) {
+    candidates[i].offset = offsets[i];
+    totalSize = std::max(totalSize, offsets[i] + candidates[i].sizeInBytes);
+    LLVM_DEBUG(llvm::dbgs()
+               << "[static-memory-planner] offset=" << candidates[i].offset
+               << " size=" << candidates[i].sizeInBytes
+               << " alignment=" << candidates[i].alignment << "\n");
   }
-};
+
+  // Step 5: Obtain arena based on arena mode.
+  Operation *firstAlloc = candidates.front().alloc;
+  OpBuilder builder(firstAlloc);
+  FailureOr<Value> arenaValue =
+      createArena(builder, funcOp, arenaMode, totalSize, arenaAlignment);
+  if (failed(arenaValue))
+    return signalPassFailure();
+
+  // Step 6: Replace each alloc with memref.view into the arena.
+  rewriteAllocations(candidates, *arenaValue);
+}
 
 } // end anonymous namespace

diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
new file mode 100644
index 0000000000000..b0c237507d2f3
--- /dev/null
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlanning.cpp
@@ -0,0 +1,131 @@
+//===- StaticMemoryPlanning.cpp - Memory planning algorithms --------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
+#include "llvm/Support/MathExtras.h"
+#include <numeric>
+
+using namespace mlir::bufferization;
+
+/// Align an offset to the specified alignment.
+static int64_t alignOffset(int64_t offset, int64_t alignment) {
+  return llvm::alignTo(offset, alignment);
+}
+
+/// Trivial sequential packing: places each allocation immediately after the
+/// previous one with alignment padding. Does not consider lifetimes, so no
+/// memory is reused. This gives a simple upper bound on arena size.
+/// Complexity: O(n) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::trivialMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  llvm::SmallVector<int64_t> offsets;
+  int64_t currentOffset = 0;
+
+  for (const auto &alloc : allocs) {
+    currentOffset = alignOffset(currentOffset, alloc.alignment);
+    assert((arenaAlignment + currentOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets.push_back(currentOffset);
+    currentOffset += alloc.sizeInBytes;
+  }
+
+  return offsets;
+}
+
+/// Best-fit lifetime-aware packing: processes allocations in start-time order
+/// and places each one in the smallest gap left by allocations whose lifetimes
+/// have ended. If no existing gap is large enough, the arena is extended.
+/// This minimizes peak memory usage when allocations have non-overlapping
+/// lifetimes.
+/// Complexity: O(n^2) where n is the number of allocations.
+llvm::SmallVector<int64_t> mlir::bufferization::bestFitMemoryPlanner(
+    int64_t arenaAlignment, llvm::ArrayRef<MemoryPlannerAlloc> allocs) {
+  // Tracks where each allocation was placed. We only need timeEnd because
+  // allocations are processed in timeStart order — by the time we place a new
+  // allocation, all earlier placements already started, so we only need to
+  // check which ones are still live.
+  struct Placement {
+    int64_t offset;
+    int64_t size;
+    int64_t timeEnd;
+  };
+
+  // Process allocations in order of start time.
+  llvm::SmallVector<unsigned> order(allocs.size());
+  std::iota(order.begin(), order.end(), 0);
+  llvm::sort(order, [&](unsigned a, unsigned b) {
+    return allocs[a].timeStart < allocs[b].timeStart;
+  });
+
+  llvm::SmallVector<Placement> placements;
+  llvm::SmallVector<int64_t> offsets(allocs.size(), 0);
+  int64_t arenaEnd = 0;
+
+  // Loop over all required allocations and fit them into best gaps.
+  for (unsigned idx : order) {
+    const MemoryPlannerAlloc &alloc = allocs[idx];
+
+    // Collect allocations that are still live at this alloc's start time.
+    // occupied is pairs of <offset_start, offset_end>.
+    llvm::SmallVector<std::pair<int64_t, int64_t>> occupied;
+    for (const auto &p : placements) {
+      if (p.timeEnd > alloc.timeStart)
+        occupied.push_back({p.offset, p.offset + p.size});
+    }
+    llvm::sort(occupied);
+
+    // Find the best (smallest) gap that fits this allocation.
+    int64_t bestOffset = -1;
+    int64_t bestGapSize = INT64_MAX;
+
+    int64_t gapStart = 0;
+    for (const auto &[occStart, occEnd] : occupied) {
+      int64_t alignedStart = alignOffset(gapStart, alloc.alignment);
+      if (alignedStart >= occStart) {
+        gapStart = std::max(gapStart, occEnd);
+        continue;
+      }
+      int64_t gapEnd = occStart;
+      int64_t gapSize = gapEnd - alignedStart;
+      // Gap is large enough to fit this allocation (after alignment).
+      if (gapSize >= alloc.sizeInBytes) {
+        // Track the smallest sufficient gap (best-fit strategy).
+        if (gapSize < bestGapSize) {
+          bestGapSize = gapSize;
+          bestOffset = alignedStart;
+        }
+      }
+      gapStart = std::max(gapStart, occEnd);
+    }
+
+    // Check the trailing gap (between last occupied region and arena end).
+    // This is only a reuse candidate if the allocation fits within the current
+    // arena bounds. If it doesn't fit, placing here would extend the arena —
+    // that case is handled by the fallback below (bestOffset < 0).
+    int64_t alignedTrailing = alignOffset(gapStart, alloc.alignment);
+    if (alignedTrailing + alloc.sizeInBytes <= arenaEnd) {
+      int64_t trailingSize = arenaEnd - alignedTrailing;
+      if (trailingSize < bestGapSize) {
+        bestGapSize = trailingSize;
+        bestOffset = alignedTrailing;
+      }
+    }
+
+    // If no existing gap worked, append at the end.
+    if (bestOffset < 0)
+      bestOffset = alignedTrailing;
+
+    assert((arenaAlignment + bestOffset) % alloc.alignment == 0 &&
+           "invalid alignment");
+    offsets[idx] = bestOffset;
+    placements.push_back({bestOffset, alloc.sizeInBytes, alloc.timeEnd});
+    arenaEnd = std::max(arenaEnd, bestOffset + alloc.sizeInBytes);
+  }
+
+  return offsets;
+}

diff  --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
new file mode 100644
index 0000000000000..6c261880140a5
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-best-fit.mlir
@@ -0,0 +1,118 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis{algorithm=best-fit}))" \
+// RUN:     -split-input-file | FileCheck %s
+
+// -----
+
+// Test 1: Non-overlapping lifetimes reuse the same memory.
+// With trivial packing this would be 8192 bytes; best-fit reuses the space.
+// CHECK-LABEL: func @reuse_non_overlapping
+func.func @reuse_non_overlapping() {
+  // Arena should be 4096 bytes (1024 * 4), not 8192.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // First allocation at offset 0
+  // CHECK-NEXT: %[[C0_0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_0]]][] : memref<4096xi8> to memref<1024xf32>
+  // Second allocation also at offset 0 (reuses freed space)
+  // CHECK-NEXT: %[[C0_1:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0_1]]][] : memref<4096xi8> to memref<1024xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  %1 = memref.alloc() : memref<1024xf32>
+  memref.dealloc %1 : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Overlapping lifetimes cannot reuse memory.
+// CHECK-LABEL: func @no_reuse_overlapping
+func.func @no_reuse_overlapping() {
+  // Both are live at the same time, so arena = 4096 + 2048 = 6144 bytes.
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<6144xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<6144xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<6144xi8> to memref<512xf32>
+  %0 = memref.alloc() : memref<1024xf32>
+  %1 = memref.alloc() : memref<512xf32>
+  memref.dealloc %0 : memref<1024xf32>
+  memref.dealloc %1 : memref<512xf32>
+  return
+}
+
+// -----
+
+// Test 3: Best-fit picks the smallest suitable gap.
+// Layout: A(4096) at 0, B(1024) at 4096, C(4096) at 5120, D(1024) at 9216.
+// B and D are freed while A and C are still live, creating two gaps:
+//   [4096, 5120) = 1024 bytes (B's slot)
+//   [9216, 10240) = 1024 bytes (D's slot)
+// Then we free A, creating gap [0, 4096) = 4096 bytes.
+// Now allocate E(512 bytes). Gaps: [0,4096)=4096, [4096,5120)=1024, [9216,10240)=1024.
+// Best-fit should pick one of the 1024-byte gaps (smallest fit for 512).
+// CHECK-LABEL: func @best_fit_smallest_gap
+func.func @best_fit_smallest_gap() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<10240xi8>
+  // A at offset 0
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // B at offset 4096
+  // CHECK-NEXT: %{{.*}} = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // C at offset 5120
+  // CHECK-NEXT: %{{.*}} = arith.constant 5120 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // D at offset 9216
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  // E at offset 9216 (best-fit picks 1024-byte trailing gap over 5120-byte gap)
+  // CHECK-NEXT: %{{.*}} = arith.constant 9216 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<256xf32>
+  %c = memref.alloc() : memref<1024xf32>
+  %d = memref.alloc() : memref<256xf32>
+  memref.dealloc %b : memref<256xf32>
+  memref.dealloc %d : memref<256xf32>
+  memref.dealloc %a : memref<1024xf32>
+  %e = memref.alloc() : memref<128xf32>
+  memref.dealloc %c : memref<1024xf32>
+  memref.dealloc %e : memref<128xf32>
+  return
+}
+
+// -----
+
+// Test 4: Alignment padding can disqualify a gap.
+// A(128, align 128) at 0 (live), B(56, align 1) at 128, C(128, align 128) at 256 (live).
+// B is freed => gap [128, 256) = 128 bytes.
+// D(64, align 128): aligned start = 128, fits in gap.
+// E(64, align 256): next 256-aligned offset in [128,256) is 256 = gap end, doesn't fit.
+//   E must go past the arena high-water mark.
+// CHECK-LABEL: func @best_fit_alignment
+func.func @best_fit_alignment() {
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 256 : i64} : memref<576xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<56xi8>
+  // CHECK-NEXT: %{{.*}} = arith.constant 256 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<128xi8>
+  // D fits in gap at offset 128
+  // CHECK-NEXT: %{{.*}} = arith.constant 128 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  // E cannot fit in gap (alignment 256), placed at offset 512
+  // CHECK-NEXT: %{{.*}} = arith.constant 512 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]]{{.*}} to memref<64xi8>
+  %a = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  %b = memref.alloc() : memref<56xi8>
+  %c = memref.alloc() {alignment = 128 : i64} : memref<128xi8>
+  memref.dealloc %b : memref<56xi8>
+  %d = memref.alloc() {alignment = 128 : i64} : memref<64xi8>
+  %e = memref.alloc() {alignment = 256 : i64} : memref<64xi8>
+  memref.dealloc %a : memref<128xi8>
+  memref.dealloc %c : memref<128xi8>
+  memref.dealloc %d : memref<64xi8>
+  memref.dealloc %e : memref<64xi8>
+  return
+}


        

From llvmlistbot at llvm.org  Wed Jul  8 01:43:30 2026
From: llvmlistbot at llvm.org (Javed Absar)
Date: Wed, 08 Jul 2026 01:43:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4e0db2.bd93e731.251b21.2e68@mx.google.com>


https://github.com/javedabsar1 closed https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Wed Jul  8 01:46:42 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 08 Jul 2026 01:46:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower arith.maxnumf/minnumf to
 spirv.GL.NMax/NMin (PR #205975)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205975@github.com>
Message-ID: <6a4e0e72.58bdd12c.19c81.67c4@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/205975

From llvmlistbot at llvm.org  Wed Jul  8 01:52:28 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 08 Jul 2026 01:52:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Add best-fit algorithm
 to static memory planner (PR #207403)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207403@github.com>
Message-ID: <6a4e0fcc.13be6df8.145c.c707@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-01` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12068

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8610/8695] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8612/8695] Linking CXX shared library lib/libclang-cpp.so.23.0git
[8613/8695] Linking CXX shared library lib/libHLFIRTransforms.so.23.0git
[8614/8695] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8615/8695] Linking CXX shared library lib/libFortranLower.so.23.0git
[8616/8695] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8617/8695] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8618/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8619/8695] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8620/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8621/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Clustering.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Analysis.h:17,
                 from /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/llvm-exegesis.cpp:14:
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/include/llvm/Support/Compiler.h:651:65: warning: ‘cfi-icall’ attribute directive ignored [-Wattributes]
  651 | #define LLVM_NO_SANITIZE(KIND) __attribute__((no_sanitize(KIND)))
      |                                                                 ^
/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/tools/llvm-exegesis/lib/Assembler.h:119:7: note: in expansion of macro ‘LLVM_NO_SANITIZE’
  119 |       LLVM_NO_SANITIZE("cfi-icall") /* Incompatible with JIT */ {
      |       ^~~~~~~~~~~~~~~~
[8610/8695] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestInliningCallback.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8612/8695] Linking CXX shared library lib/libclang-cpp.so.23.0git
[8613/8695] Linking CXX shared library lib/libHLFIRTransforms.so.23.0git
[8614/8695] Building CXX object tools/mlir/test/lib/Transforms/CMakeFiles/MLIRTestTransforms.dir/TestMakeIsolatedFromAbove.cpp.o
[8615/8695] Linking CXX shared library lib/libFortranLower.so.23.0git
[8616/8695] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPILinalg.dir/LinalgPasses.cpp.o
[8617/8695] Building CXX object tools/mlir/test/lib/Dialect/Vector/CMakeFiles/MLIRVectorTestPasses.dir/TestVectorTransforms.cpp.o
[8618/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
[8619/8695] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8620/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8621/8695] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestDialect.dir/TestOpsSyntax.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=136.416745

```

</details>

https://github.com/llvm/llvm-project/pull/207403

From llvmlistbot at llvm.org  Wed Jul  8 02:10:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:10:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 0064239 - [mlir][SPIR-V] Lower
 arith.maxnumf/minnumf to spirv.GL.NMax/NMin (#205975)
Message-ID: <6a4e140f.1905e68e.18ad68.e6b8@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-08T11:10:26+02:00
New Revision: 006423944b8efdb115d8fd001cffca888605cd7f

URL: https://github.com/llvm/llvm-project/commit/006423944b8efdb115d8fd001cffca888605cd7f
DIFF: https://github.com/llvm/llvm-project/commit/006423944b8efdb115d8fd001cffca888605cd7f.diff

LOG: [mlir][SPIR-V] Lower arith.maxnumf/minnumf to spirv.GL.NMax/NMin (#205975)

The previous lowering targeted spirv.GL.FMax/FMin + NaN guards

spirv.GL.NMax/NMin natively treat NaN as missing, matching
arith.maxnumf/minnumf exactly (no guards required)

Added: 
    

Modified: 
    mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
    mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
    mlir/test/Conversion/ArithToSPIRV/fast-math.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
index 9a6d330db72fe..614ac2b43b5f2 100644
--- a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
+++ b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
@@ -1403,7 +1403,7 @@ class MinimumMaximumFOpPattern final : public OpConversionPattern<Op> {
 // MinNumFOp, MaxNumFOp
 //===----------------------------------------------------------------------===//
 
-/// Converts arith.maxnumf/minnumf to spirv.GL.FMax/FMin or
+/// Converts arith.maxnumf/minnumf to spirv.GL.NMax/NMin or
 /// spirv.CL.fmax/fmin.
 template <typename Op, typename SPIRVOp>
 class MinNumMaxNumFOpPattern final : public OpConversionPattern<Op> {
@@ -1425,11 +1425,11 @@ class MinNumMaxNumFOpPattern final : public OpConversionPattern<Op> {
     // arith.maxnumf/minnumf:
     //   "If one of the arguments is NaN, then the result is the other
     //   argument."
-    // spirv.GL.FMax/FMin
-    //   "which operand is the result is undefined if one of the operands
-    //   is a NaN."
+    // spirv.GL.NMax/NMin: NaN is treated as missing, matches arith semantics.
     // spirv.CL.fmax/fmin:
     //   "If one argument is a NaN, Fmin returns the other argument."
+    // spirv.GL.FMax/FMin: undefined when either operand is NaN, requires
+    //   select guards to implement arith.maxnumf semantics.
 
     Location loc = op.getLoc();
     Value spirvOp =
@@ -1519,8 +1519,8 @@ void mlir::arith::populateArithToSPIRVPatterns(
 
     MinimumMaximumFOpPattern<arith::MaximumFOp, spirv::GLFMaxOp>,
     MinimumMaximumFOpPattern<arith::MinimumFOp, spirv::GLFMinOp>,
-    MinNumMaxNumFOpPattern<arith::MaxNumFOp, spirv::GLFMaxOp>,
-    MinNumMaxNumFOpPattern<arith::MinNumFOp, spirv::GLFMinOp>,
+    MinNumMaxNumFOpPattern<arith::MaxNumFOp, spirv::GLNMaxOp>,
+    MinNumMaxNumFOpPattern<arith::MinNumFOp, spirv::GLNMinOp>,
     BoolIOpPattern<arith::MaxSIOp, spirv::LogicalAndOp>, // signed i1: 1=-1, so max=0 unless both are 1
     BoolIOpPattern<arith::MaxUIOp, spirv::LogicalOrOp>,  // unsigned max on i1: 1 when either is 1
     BoolIOpPattern<arith::MinSIOp, spirv::LogicalOrOp>,  // signed i1: -1<0, so min=1 when either is 1

diff  --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index b6a488f0dad73..5c9966055b540 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -1494,15 +1494,9 @@ func.func @float32_minimumf_scalar(%arg0 : f32, %arg1 : f32) -> f32 {
 }
 
 // CHECK-LABEL: @float32_minnumf_scalar
-// CHECK-SAME: %[[LHS:.+]]: f32, %[[RHS:.+]]: f32
 func.func @float32_minnumf_scalar(%arg0 : f32, %arg1 : f32) -> f32 {
-  // CHECK: %[[MIN:.+]] = spirv.GL.FMin %arg0, %arg1 : f32
-  // CHECK: %[[LHS_NAN:.+]] = spirv.IsNan %[[LHS]] : f32
-  // CHECK: %[[RHS_NAN:.+]] = spirv.IsNan %[[RHS]] : f32
-  // CHECK: %[[SELECT1:.+]] = spirv.Select %[[LHS_NAN]], %[[RHS]], %[[MIN]]
-  // CHECK: %[[SELECT2:.+]] = spirv.Select %[[RHS_NAN]], %[[LHS]], %[[SELECT1]]
+  // CHECK: spirv.GL.NMin %arg0, %arg1 : f32
   %0 = arith.minnumf %arg0, %arg1 : f32
-  // CHECK: return %[[SELECT2]]
   return %0: f32
 }
 
@@ -1520,15 +1514,9 @@ func.func @float32_maximumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>)
 }
 
 // CHECK-LABEL: @float32_maxnumf_scalar
-// CHECK-SAME: %[[LHS:.+]]: vector<2xf32>, %[[RHS:.+]]: vector<2xf32>
 func.func @float32_maxnumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>) -> vector<2xf32> {
-  // CHECK: %[[MAX:.+]] = spirv.GL.FMax %arg0, %arg1 : vector<2xf32>
-  // CHECK: %[[LHS_NAN:.+]] = spirv.IsNan %[[LHS]] : vector<2xf32>
-  // CHECK: %[[RHS_NAN:.+]] = spirv.IsNan %[[RHS]] : vector<2xf32>
-  // CHECK: %[[SELECT1:.+]] = spirv.Select %[[LHS_NAN]], %[[RHS]], %[[MAX]]
-  // CHECK: %[[SELECT2:.+]] = spirv.Select %[[RHS_NAN]], %[[LHS]], %[[SELECT1]]
+  // CHECK: spirv.GL.NMax %arg0, %arg1 : vector<2xf32>
   %0 = arith.maxnumf %arg0, %arg1 : vector<2xf32>
-  // CHECK: return %[[SELECT2]]
   return %0: vector<2xf32>
 }
 

diff  --git a/mlir/test/Conversion/ArithToSPIRV/fast-math.mlir b/mlir/test/Conversion/ArithToSPIRV/fast-math.mlir
index 9bbe28fb127a7..e13bef977ddb7 100644
--- a/mlir/test/Conversion/ArithToSPIRV/fast-math.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/fast-math.mlir
@@ -49,20 +49,16 @@ func.func @maximumf(%arg0 : vector<4xf32>, %arg1 : vector<4xf32>) -> vector<4xf3
 }
 
 // CHECK-LABEL: @minnumf
-// CHECK-SAME: %[[LHS:.+]]: f32, %[[RHS:.+]]: f32
 func.func @minnumf(%arg0 : f32, %arg1 : f32) -> f32 {
-  // CHECK: %[[F:.+]] = spirv.GL.FMin %[[LHS]], %[[RHS]]
+  // CHECK: spirv.GL.NMin %{{.*}}, %{{.*}} : f32
   %0 = arith.minnumf %arg0, %arg1 fastmath<fast> : f32
-  // CHECK: return %[[F]]
   return %0: f32
 }
 
 // CHECK-LABEL: @maxnumf
-// CHECK-SAME: %[[LHS:.+]]: vector<4xf32>, %[[RHS:.+]]: vector<4xf32>
 func.func @maxnumf(%arg0 : vector<4xf32>, %arg1 : vector<4xf32>) -> vector<4xf32> {
-  // CHECK: %[[F:.+]] = spirv.GL.FMax %[[LHS]], %[[RHS]]
+  // CHECK: spirv.GL.NMax %{{.*}}, %{{.*}} : vector<4xf32>
   %0 = arith.maxnumf %arg0, %arg1 fastmath<fast> : vector<4xf32>
-  // CHECK: return %[[F]]
   return %0: vector<4xf32>
 }
 


        

From llvmlistbot at llvm.org  Wed Jul  8 02:10:40 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 08 Jul 2026 02:10:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower arith.maxnumf/minnumf to
 spirv.GL.NMax/NMin (PR #205975)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205975@github.com>
Message-ID: <6a4e1410.27b3b003.3b871.f16f@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/205975

From llvmlistbot at llvm.org  Wed Jul  8 02:15:34 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Wed, 08 Jul 2026 02:15:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][IR] Add SymbolUserTypeInterface
	(PR #198435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198435@github.com>
Message-ID: <6a4e1536.30759430.24d50c.5653@mx.google.com>


https://github.com/jpienaar approved this pull request.

Apologies, thought I had approved

https://github.com/llvm/llvm-project/pull/198435

From llvmlistbot at llvm.org  Wed Jul  8 02:16:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:16:56 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir] Avoid crash in test alias
	analysis (PR #207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a4e1588.da8bafd3.14cd50.a406@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/207670

>From fb9fd156080e0809b97d174b1a570783b23ed5c0 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/3] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 10afc402a685de09e498bc61449459cf8fe230a9 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/3] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 2c5e065268722fbd1db1c82dbcf2542d43d7df19 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Wed, 8 Jul 2026 17:15:19 +0800
Subject: [PATCH 3/3] [mlir][vector] Fix crash in narrow-type emulation when
 mask has no defining op

---
 crash-min.log                                 | 41 +++++++++++++++++++
 crash.log                                     |  1 +
 .../Transforms/VectorEmulateNarrowType.cpp    |  5 ++-
 .../emulate-narrow-type-unsupported.mlir      | 18 ++++++++
 4 files changed, 64 insertions(+), 1 deletion(-)
 create mode 100644 crash-min.log
 create mode 100644 crash.log

diff --git a/crash-min.log b/crash-min.log
new file mode 100644
index 0000000000000..b4ab9da77a0ea
--- /dev/null
+++ b/crash-min.log
@@ -0,0 +1,41 @@
+PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
+Stack dump:
+0.	Program arguments: ./build/bin/mlir-opt -test-emulate-narrow-int a-min.mlir
+ #0 0x000055fbd082932a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (./build/bin/mlir-opt+0x20be32a)
+ #1 0x000055fbd082617f llvm::sys::RunSignalHandlers() (./build/bin/mlir-opt+0x20bb17f)
+ #2 0x000055fbd082a4b5 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
+ #3 0x00007e4cc2442520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
+ #4 0x000055fbd7f6496e bool llvm::isa<mlir::arith::ConstantOp, mlir::vector::CreateMaskOp, mlir::vector::ConstantMaskOp, mlir::Operation*>(mlir::Operation* const&) (./build/bin/mlir-opt+0x97f996e)
+ #5 0x000055fbd7f634ee getCompressedMaskOp(mlir::OpBuilder&, mlir::Location, mlir::Value, int, int, int) VectorEmulateNarrowType.cpp:0:0
+ #6 0x000055fbd7f62cff (anonymous namespace)::ConvertVectorMaskedLoad::matchAndRewrite(mlir::vector::MaskedLoadOp, mlir::vector::MaskedLoadOpAdaptor, mlir::ConversionPatternRewriter&) const VectorEmulateNarrowType.cpp:0:0
+ #7 0x000055fbd7f64db2 llvm::LogicalResult mlir::ConversionPattern::dispatchTo1To1<mlir::OpConversionPattern<mlir::vector::MaskedLoadOp>, mlir::vector::MaskedLoadOp>(mlir::OpConversionPattern<mlir::vector::MaskedLoadOp> const&, mlir::vector::MaskedLoadOp, mlir::vector::MaskedLoadOp::GenericAdaptor<llvm::ArrayRef<mlir::ValueRange>>, mlir::ConversionPatternRewriter&) (./build/bin/mlir-opt+0x97f9db2)
+ #8 0x000055fbd7f6267a mlir::OpConversionPattern<mlir::vector::MaskedLoadOp>::matchAndRewrite(mlir::Operation*, llvm::ArrayRef<mlir::ValueRange>, mlir::ConversionPatternRewriter&) const (./build/bin/mlir-opt+0x97f767a)
+ #9 0x000055fbd9e54df6 mlir::ConversionPattern::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&) const (./build/bin/mlir-opt+0xb6e9df6)
+#10 0x000055fbd9ea41fe void llvm::function_ref<void ()>::callback_fn<mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>)::$_6>(long) PatternApplicator.cpp:0:0
+#11 0x000055fbd9ea08b3 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>) (./build/bin/mlir-opt+0xb7358b3)
+#12 0x000055fbd9e56c1d (anonymous namespace)::OperationLegalizer::legalize(mlir::Operation*) DialectConversion.cpp:0:0
+#13 0x000055fbd9e56405 mlir::OperationConverter::convert(mlir::Operation*, bool) (./build/bin/mlir-opt+0xb6eb405)
+#14 0x000055fbd9e576b3 mlir::OperationConverter::applyConversion(llvm::ArrayRef<mlir::Operation*>) (./build/bin/mlir-opt+0xb6ec6b3)
+#15 0x000055fbd9e66719 void llvm::function_ref<void ()>::callback_fn<applyConversion(llvm::ArrayRef<mlir::Operation*>, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig, (anonymous namespace)::OpConversionMode)::$_43>(long) DialectConversion.cpp:0:0
+#16 0x000055fbd9e5d045 applyConversion(llvm::ArrayRef<mlir::Operation*>, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig, (anonymous namespace)::OpConversionMode) DialectConversion.cpp:0:0
+#17 0x000055fbd9e5d12b mlir::applyPartialConversion(mlir::Operation*, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig) (./build/bin/mlir-opt+0xb6f212b)
+#18 0x000055fbd091ad6a (anonymous namespace)::TestEmulateNarrowTypePass::runOnOperation() TestEmulateNarrowType.cpp:0:0
+#19 0x000055fbd9eff4fa mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) (./build/bin/mlir-opt+0xb7944fa)
+#20 0x000055fbd9f001b9 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) (./build/bin/mlir-opt+0xb7951b9)
+#21 0x000055fbd9f0b39a auto void mlir::parallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_102>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_102&&)::'lambda'(__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>&&)::operator()<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&>(__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>&&) const Pass.cpp:0:0
+#22 0x000055fbd9f02de8 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) (./build/bin/mlir-opt+0xb797de8)
+#23 0x000055fbd9eff691 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) (./build/bin/mlir-opt+0xb794691)
+#24 0x000055fbd9f001b9 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) (./build/bin/mlir-opt+0xb7951b9)
+#25 0x000055fbd9f070c1 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) (./build/bin/mlir-opt+0xb79c0c1)
+#26 0x000055fbd9f06944 mlir::PassManager::run(mlir::Operation*) (./build/bin/mlir-opt+0xb79b944)
+#27 0x000055fbd08d949a performActions(llvm::raw_ostream&, std::shared_ptr<llvm::SourceMgr> const&, mlir::MLIRContext*, mlir::MlirOptMainConfig const&) MlirOptMain.cpp:0:0
+#28 0x000055fbd08d877e llvm::LogicalResult llvm::function_ref<llvm::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&)::$_3>(long, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&) MlirOptMain.cpp:0:0
+#29 0x000055fbda2579d7 mlir::splitAndProcessBuffer(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<llvm::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&)>, llvm::raw_ostream&, llvm::StringRef, llvm::StringRef) (./build/bin/mlir-opt+0xbaec9d7)
+#30 0x000055fbd08cf7c2 mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&) (./build/bin/mlir-opt+0x21647c2)
+#31 0x000055fbd08cfbc1 mlir::MlirOptMain(int, char**, llvm::StringRef, llvm::StringRef, mlir::DialectRegistry&) (./build/bin/mlir-opt+0x2164bc1)
+#32 0x000055fbd08cfdfd mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&) (./build/bin/mlir-opt+0x2164dfd)
+#33 0x000055fbd080f0e2 main (./build/bin/mlir-opt+0x20a40e2)
+#34 0x00007e4cc2429d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
+#35 0x00007e4cc2429e40 call_init ./csu/../csu/libc-start.c:128:20
+#36 0x00007e4cc2429e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
+#37 0x000055fbd080ec15 _start (./build/bin/mlir-opt+0x20a3c15)
diff --git a/crash.log b/crash.log
new file mode 100644
index 0000000000000..1bc8458b55d1d
--- /dev/null
+++ b/crash.log
@@ -0,0 +1 @@
+cannot open input file 'a.mlir': No such file or directory
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..9aa28ff899bc4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,13 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    }else {
+      return failure();
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..17bd24e2c1959 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,21 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// vector.maskedload
+///----------------------------------------------------------------------------------------
+
+func.func @vector_maskedload_blockarg_mask_negative(%arg0: memref<16xi8>, %arg1: vector<8xi1>, %arg2: vector<8xi8>) -> vector<8xi8> {
+    %c0 = arith.constant 0 : index
+    %0 = vector.maskedload %arg0[%c0], %arg1, %arg2 : memref<16xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+    return %0 : vector<8xi8>
+}
+// The mask is a block argument with no defining op, so it cannot be
+// compressed - expect no conversion (and, crucially, no crash).
+//  CHECK-LABEL: func @vector_maskedload_blockarg_mask_negative
+//   CHECK-SAME:   %{{.*}}: memref<16xi8>, %[[MASK:.+]]: vector<8xi1>
+//        CHECK:   vector.maskedload %{{.*}}, %[[MASK]]
+//    CHECK-NOT: i32


From llvmlistbot at llvm.org  Wed Jul  8 02:20:07 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Wed, 08 Jul 2026 02:20:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e1647.fdc477d6.23591.c84d@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
----------------
durga4github wrote:

I wonder why should we carry the `optional`ity at all..

Can we not make the attr itself a DefaultValuedAttr with `wrapped` as the `default` value?

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 02:22:32 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Wed, 08 Jul 2026 02:22:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e16d8.331483e7.2440f8.9262@mx.google.com>


jpienaar wrote:

Yes previously we had used this as a predicate for legality of declarative rewrites. It was changed to be more a op verification function rather than type predicate. Another option here may be to use ODS string interpolation for the error reporting and then the predicate can remain a plain predicate and useable both to gate rewrites and to verify op: upon failure of predicate, message specified fires and error is emitted on op so no need to inject, the values can be interpolated into error string and its just on type.

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 02:22:58 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Wed, 08 Jul 2026 02:22:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4e16f2.491d90f5.2facd.28c5@mx.google.com>


================

----------------
durga4github wrote:

Can we also move these else-if to just `if` since we only have return statements here?

(This also aligns it cleanly with the style of the `ExtfOp::verify()` below)

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Wed Jul  8 02:25:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:25:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a4e179a.cbc3f03a.10d998.8a0d@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/207670

>From fb9fd156080e0809b97d174b1a570783b23ed5c0 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/3] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 10afc402a685de09e498bc61449459cf8fe230a9 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/3] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 1b2ed96b6c291c781f7d8953c62e847edbd22f89 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Wed, 8 Jul 2026 17:15:19 +0800
Subject: [PATCH 3/3] [mlir][vector] Fix crash in narrow-type emulation when
 mask has no defining op

---
 .../Transforms/VectorEmulateNarrowType.cpp     |  5 ++++-
 .../emulate-narrow-type-unsupported.mlir       | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..9aa28ff899bc4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,13 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    }else {
+      return failure();
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..17bd24e2c1959 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,21 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// vector.maskedload
+///----------------------------------------------------------------------------------------
+
+func.func @vector_maskedload_blockarg_mask_negative(%arg0: memref<16xi8>, %arg1: vector<8xi1>, %arg2: vector<8xi8>) -> vector<8xi8> {
+    %c0 = arith.constant 0 : index
+    %0 = vector.maskedload %arg0[%c0], %arg1, %arg2 : memref<16xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+    return %0 : vector<8xi8>
+}
+// The mask is a block argument with no defining op, so it cannot be
+// compressed - expect no conversion (and, crucially, no crash).
+//  CHECK-LABEL: func @vector_maskedload_blockarg_mask_negative
+//   CHECK-SAME:   %{{.*}}: memref<16xi8>, %[[MASK:.+]]: vector<8xi1>
+//        CHECK:   vector.maskedload %{{.*}}, %[[MASK]]
+//    CHECK-NOT: i32


From llvmlistbot at llvm.org  Wed Jul  8 02:26:44 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Wed, 08 Jul 2026 02:26:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4e17d4.7270d2fd.2d4879.c779@mx.google.com>


https://github.com/durga4github approved this pull request.

The latest version LGTM modulo a nitpick

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Wed Jul  8 02:47:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:47:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix narrow type mask crash (PR #208185)
Message-ID: <llvm.org/llvm/llvm-project/pull/208185@github.com>


https://github.com/LouisLu060211 created https://github.com/llvm/llvm-project/pull/208185

The issue exists in `getCompressedMaskOp`, when it tries to trace the mask of a
`vector.maskedstore` back to the op 

This breaks when the mask is a block argument: `getDefiningOp()` returns
null. The `isa<>` check crashes on the null pointer. In addition
if the mask comes from some other unsupported op (e.g.
`vector.broadcast`), the traversal loop never advances and hangs forever.

Fix both by bailing out with `failure()`, so the conversion just reports
a normal legalization error instead of crashing. Added a regression test
with a block-argument mask.

Assisted by: Fable-5

>From fb9fd156080e0809b97d174b1a570783b23ed5c0 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/3] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 10afc402a685de09e498bc61449459cf8fe230a9 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/3] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 1b2ed96b6c291c781f7d8953c62e847edbd22f89 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Wed, 8 Jul 2026 17:15:19 +0800
Subject: [PATCH 3/3] [mlir][vector] Fix crash in narrow-type emulation when
 mask has no defining op

---
 .../Transforms/VectorEmulateNarrowType.cpp     |  5 ++++-
 .../emulate-narrow-type-unsupported.mlir       | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..9aa28ff899bc4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,13 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    }else {
+      return failure();
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..17bd24e2c1959 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,21 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// vector.maskedload
+///----------------------------------------------------------------------------------------
+
+func.func @vector_maskedload_blockarg_mask_negative(%arg0: memref<16xi8>, %arg1: vector<8xi1>, %arg2: vector<8xi8>) -> vector<8xi8> {
+    %c0 = arith.constant 0 : index
+    %0 = vector.maskedload %arg0[%c0], %arg1, %arg2 : memref<16xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+    return %0 : vector<8xi8>
+}
+// The mask is a block argument with no defining op, so it cannot be
+// compressed - expect no conversion (and, crucially, no crash).
+//  CHECK-LABEL: func @vector_maskedload_blockarg_mask_negative
+//   CHECK-SAME:   %{{.*}}: memref<16xi8>, %[[MASK:.+]]: vector<8xi1>
+//        CHECK:   vector.maskedload %{{.*}}, %[[MASK]]
+//    CHECK-NOT: i32


From llvmlistbot at llvm.org  Wed Jul  8 02:48:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:48:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a4e1cd3.d1496011.26913c.66fc@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @LouisLu060211 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/208185

From llvmlistbot at llvm.org  Wed Jul  8 02:48:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 02:48:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a4e1cfc.ff037943.2e959a.2af3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-vector

Author: LouisLu060211

<details>
<summary>Changes</summary>

The issue exists in `getCompressedMaskOp`, when it tries to trace the mask of a
`vector.maskedstore` back to the op 

This breaks when the mask is a block argument: `getDefiningOp()` returns
null. The `isa&lt;&gt;` check crashes on the null pointer. In addition
if the mask comes from some other unsupported op (e.g.
`vector.broadcast`), the traversal loop never advances and hangs forever.

Fix both by bailing out with `failure()`, so the conversion just reports
a normal legalization error instead of crashing. Added a regression test
with a block-argument mask.

Assisted by: Fable-5

---
Full diff: https://github.com/llvm/llvm-project/pull/208185.diff


6 Files Affected:

- (modified) mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp (+9-4) 
- (modified) mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp (+4-1) 
- (modified) mlir/test/Analysis/test-alias-analysis-modref.mlir (+14) 
- (modified) mlir/test/Dialect/MemRef/mem2reg.mlir (+15) 
- (modified) mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir (+18) 
- (modified) mlir/test/lib/Analysis/TestAliasAnalysis.cpp (+40-10) 


``````````diff
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..9aa28ff899bc4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,13 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    }else {
+      return failure();
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..17bd24e2c1959 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,21 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// vector.maskedload
+///----------------------------------------------------------------------------------------
+
+func.func @vector_maskedload_blockarg_mask_negative(%arg0: memref<16xi8>, %arg1: vector<8xi1>, %arg2: vector<8xi8>) -> vector<8xi8> {
+    %c0 = arith.constant 0 : index
+    %0 = vector.maskedload %arg0[%c0], %arg1, %arg2 : memref<16xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+    return %0 : vector<8xi8>
+}
+// The mask is a block argument with no defining op, so it cannot be
+// compressed - expect no conversion (and, crucially, no crash).
+//  CHECK-LABEL: func @vector_maskedload_blockarg_mask_negative
+//   CHECK-SAME:   %{{.*}}: memref<16xi8>, %[[MASK:.+]]: vector<8xi1>
+//        CHECK:   vector.maskedload %{{.*}}, %[[MASK]]
+//    CHECK-NOT: i32
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

``````````

</details>


https://github.com/llvm/llvm-project/pull/208185

From llvmlistbot at llvm.org  Wed Jul  8 02:50:47 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 08 Jul 2026 02:50:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][nvgpu] Add negative tests for invalid
 FP8 warpgroup MMA types (PR #208047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208047@github.com>
Message-ID: <6a4e1d77.27b3b003.3b871.0b44@mx.google.com>


https://github.com/grypp approved this pull request.


https://github.com/llvm/llvm-project/pull/208047

From llvmlistbot at llvm.org  Wed Jul  8 03:18:24 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Wed, 08 Jul 2026 03:18:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
Message-ID: <llvm.org/llvm/llvm-project/pull/208188@github.com>


https://github.com/skatrak created https://github.com/llvm/llvm-project/pull/208188

This patch introduces various changes to the handling of `declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when compiling for an OpenMP offload target. This prevents potential symbol redefinition issues related to globals that don't actually exist on the device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to prevent their associated "internal" linkage from producing broken device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g. `declare_target link(...)`), the associated and unused full-storage global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to external linkage when compiling for a target device, causing linker errors if accessed. This mirrors Clang's behavior.

Fixes #195188, fixes #195468.

Assisted-by: Claude Opus 4.8.

>From fdfdcb6a697d349581c932a7fcb7b46ab2cee448 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Mon, 22 Jun 2026 13:59:32 +0100
Subject: [PATCH] [Flang][MLIR][OpenMP] Fix declare_target globals visibility

This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
  compiling for an OpenMP offload target. This prevents potential symbol
  redefinition issues related to globals that don't actually exist on the
  device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to
  prevent their associated "internal" linkage from producing broken
  device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
  `declare_target link(...)`), the associated and unused full-storage
  global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
  external linkage when compiling for a target device, causing linker
  errors if accessed. This mirrors Clang's behavior.
---
 .../lib/Optimizer/OpenMP/HostOpFiltering.cpp  |  25 +-
 .../OpenMP/function-filtering-host-ops.mlir   |  21 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  53 +++-
 ...eclare-target-all-device-types-device.mlir | 252 ++++++++++++++++++
 .../omptarget-declare-target-llvm-device.mlir |   1 +
 5 files changed, 334 insertions(+), 18 deletions(-)
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir

diff --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
index e42e10a82e6c2..05ea0bc08c4ab 100644
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
@@ -65,6 +65,15 @@ static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
   rewrites.insert(value);
 }
 
+/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
 namespace {
 class HostOpFilteringPass
     : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
@@ -78,11 +87,8 @@ class HostOpFilteringPass
 
     op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
       omp::DeclareTargetDeviceType declareType =
-          omp::DeclareTargetDeviceType::host;
-      auto declareTargetOp =
-          dyn_cast<omp::DeclareTargetInterface>(funcOp.getOperation());
-      if (declareTargetOp && declareTargetOp.isDeclareTarget())
-        declareType = declareTargetOp.getDeclareTargetDeviceType();
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
 
       // Only process host function definitions.
       if (funcOp.isExternal() ||
@@ -95,6 +101,15 @@ class HostOpFilteringPass
       }
       return WalkResult::advance();
     });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time issues.
+    op->walk([&](fir::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkName("internal");
+    });
   }
 
 private:
diff --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
index f3b15bd028bc4..3acdfda876812 100644
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
@@ -523,10 +523,6 @@ module attributes {omp.is_target_device = true} {
   }
 
   func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  fir.global internal @global_scalar constant : i32 {
-    %0 = arith.constant 10 : i32
-    fir.has_value %0 : i32
-  }
   omp.private {type = firstprivate} @privatizer : i32 copy {
   ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
     %0 = fir.load %arg0 : !fir.ref<i32>
@@ -544,4 +540,21 @@ module attributes {omp.is_target_device = true} {
     %1 = arith.addi %arg0, %arg1 : i32
     omp.yield (%1 : i32)
   }
+
+  // CHECK: fir.global internal @global_scalar constant : i32
+  fir.global @global_scalar constant : i32
+
+  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..b2d2ef6d8f770 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8843,14 +8843,16 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (LLVM::GlobalOp gOp = dyn_cast<LLVM::GlobalOp>(op)) {
     llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
     if (auto *gVal = llvmModule->getNamedValue(gOp.getSymName())) {
+      auto *gVar = cast<llvm::GlobalVariable>(gVal);
       llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
       bool isDeclaration = gOp.isDeclaration();
       bool isExternallyVisible =
           gOp.getVisibility() != mlir::SymbolTable::Visibility::Private;
       auto loc = op->getLoc()->findInstanceOf<FileLineColLoc>();
       llvm::StringRef mangledName = gOp.getSymName();
-      auto captureClause =
-          convertToCaptureClauseKind(attribute.getCaptureClause().getValue());
+      mlir::omp::DeclareTargetCaptureClause captureClause =
+          attribute.getCaptureClause().getValue();
+      auto captureClauseKind = convertToCaptureClauseKind(captureClause);
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
       // unused for MLIR at the moment, required in Clang for book
@@ -8877,31 +8879,64 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                                                      lineNo);
       };
 
-      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
+      bool isToOrEnter =
+          captureClause == omp::DeclareTargetCaptureClause::to ||
+          captureClause == omp::DeclareTargetCaptureClause::enter;
+      bool isHostOnly = attribute.getDeviceType().getValue() ==
+                        omp::DeclareTargetDeviceType::host;
+
+      // A to/enter declare-target variable needs a device-resident,
+      // name-resolvable copy and a host offloading entry. A local-linkage
+      // global provides neither, so we promote it to external.
+      if (isToOrEnter && !isHostOnly && !requiresUSM &&
+          gVar->hasLocalLinkage()) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        isExternallyVisible = true;
+
+        // Clear the stale dso_local flag so it is referenced like a
+        // module-scope declare target global.
+        if (ompBuilder->Config.isTargetDevice())
+          gVar->setDSOLocal(false);
+      }
 
+      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
       ompBuilder->registerTargetGlobalVariable(
-          captureClause, deviceClause, isDeclaration, isExternallyVisible,
+          captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
           mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
           /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
           gVal->getType(), gVal);
 
-      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
       if (ompBuilder->Config.isTargetDevice() &&
-          (attribute.getCaptureClause().getValue() ==
-               mlir::omp::DeclareTargetCaptureClause::link ||
+          (captureClause == omp::DeclareTargetCaptureClause::link ||
            requiresUSM)) {
         llvm::Type *ptrTy = gVal->getType();
         // For USM the global type becomes a pointer handle, as opposed to the
         // globals original type.
         if (requiresUSM)
           ptrTy = llvm::PointerType::get(llvmModule->getContext(), 0);
-        ompBuilder->getAddrOfDeclareTargetVar(
-            captureClause, deviceClause, isDeclaration, isExternallyVisible,
+        bool addrGlobalCreated = ompBuilder->getAddrOfDeclareTargetVar(
+            captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
             ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
             mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
             ptrTy, /*GlobalInitializer*/ nullptr,
             /*VariableLinkage*/ nullptr);
+
+        // For indirectly-accessed global pointers, we rely on "internal"
+        // linkage to optimize out the unneeded full-variable storage later,
+        // since we can't prevent the LLVM dialect from generating globals
+        // without also breaking target lowering.
+        if (addrGlobalCreated)
+          gVar->setLinkage(llvm::GlobalValue::InternalLinkage);
+      }
+
+      // Mark 'device_type(host) enter(...)' variables as external in the device
+      // since they're not supposed to have their own copy. This will cause
+      // linker errors if accesses are attempted from the target device.
+      if (ompBuilder->Config.isTargetDevice() && isHostOnly && isToOrEnter) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        gVar->setInitializer(nullptr);
       }
     }
   }
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
new file mode 100644
index 0000000000000..2cd3fc3c44231
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
@@ -0,0 +1,252 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Tests lowering to LLVM IR of all combinations of OpenMP declare target
+// device_type (any/host/nohost) and capture clause (link/enter) when compiling
+// for device, for both external and internal linkage global variables.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  // --- link any ---
+
+  // CHECK-DAG: @ial = internal global float 0.000000e+00
+  // CHECK-DAG: @ial_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ial() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eal = internal global float 0.000000e+00
+  // CHECK-DAG: @eal_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @eal() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link host ---
+
+  // CHECK-DAG: @ihl = internal global float 0.000000e+00
+  // CHECK-DAG: @ihl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ihl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehl = internal global float 0.000000e+00
+  // CHECK-DAG: @ehl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @ehl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link nohost ---
+
+  // CHECK-DAG: @inl = internal global float 0.000000e+00
+  // CHECK-DAG: @inl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @inl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @enl = internal global float 0.000000e+00
+  // CHECK-DAG: @enl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @enl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter any ---
+
+  // CHECK-DAG: @iae = global float 0.000000e+00
+  llvm.mlir.global internal @iae() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eae = dso_local global float 0.000000e+00
+  llvm.mlir.global external @eae() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter host ---
+
+  // CHECK-DAG: @ihe = external dso_local global float
+  llvm.mlir.global internal @ihe() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehe = external dso_local global float
+  llvm.mlir.global external @ehe() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter nohost ---
+
+  // CHECK-DAG: @ine = global float 0.000000e+00
+  llvm.mlir.global internal @ine() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ene = dso_local global float 0.000000e+00
+  llvm.mlir.global external @ene() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  llvm.func @_QQmain() {
+    %0 = llvm.mlir.addressof @ial : !llvm.ptr
+    %1 = llvm.addrspacecast %0 : !llvm.ptr to !llvm.ptr
+    %2 = llvm.mlir.addressof @ihl : !llvm.ptr
+    %3 = llvm.addrspacecast %2 : !llvm.ptr to !llvm.ptr
+    %4 = llvm.mlir.addressof @inl : !llvm.ptr
+    %5 = llvm.addrspacecast %4 : !llvm.ptr to !llvm.ptr
+    %6 = llvm.mlir.addressof @iae : !llvm.ptr
+    %7 = llvm.addrspacecast %6 : !llvm.ptr to !llvm.ptr
+    %8 = llvm.mlir.addressof @ihe : !llvm.ptr
+    %9 = llvm.addrspacecast %8 : !llvm.ptr to !llvm.ptr
+    %10 = llvm.mlir.addressof @ine : !llvm.ptr
+    %11 = llvm.addrspacecast %10 : !llvm.ptr to !llvm.ptr
+    %12 = llvm.mlir.addressof @eal : !llvm.ptr
+    %13 = llvm.addrspacecast %12 : !llvm.ptr to !llvm.ptr
+    %14 = llvm.mlir.addressof @ehl : !llvm.ptr
+    %15 = llvm.addrspacecast %14 : !llvm.ptr to !llvm.ptr
+    %16 = llvm.mlir.addressof @enl : !llvm.ptr
+    %17 = llvm.addrspacecast %16 : !llvm.ptr to !llvm.ptr
+    %18 = llvm.mlir.addressof @eae : !llvm.ptr
+    %19 = llvm.addrspacecast %18 : !llvm.ptr to !llvm.ptr
+    %20 = llvm.mlir.addressof @ehe : !llvm.ptr
+    %21 = llvm.addrspacecast %20 : !llvm.ptr to !llvm.ptr
+    %22 = llvm.mlir.addressof @ene : !llvm.ptr
+    %23 = llvm.addrspacecast %22 : !llvm.ptr to !llvm.ptr
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ial_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.100000e+00, ptr %[[REF]], align 4
+    %map_ial = omp.map.info var_ptr(%1 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ial"}
+    omp.target kernel_type(generic) map_entries(%map_ial -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @eal_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.200000e+00, ptr %[[REF]], align 4
+    %map_eal = omp.map.info var_ptr(%13 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "eal"}
+    omp.target kernel_type(generic) map_entries(%map_eal -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ihl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.100000e+00, ptr %[[REF]], align 4
+    %map_ihl = omp.map.info var_ptr(%3 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ihl"}
+    omp.target kernel_type(generic) map_entries(%map_ihl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ehl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.200000e+00, ptr %[[REF]], align 4
+    %map_ehl = omp.map.info var_ptr(%15 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ehl"}
+    omp.target kernel_type(generic) map_entries(%map_ehl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @inl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.100000e+00, ptr %[[REF]], align 4
+    %map_inl = omp.map.info var_ptr(%5 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "inl"}
+    omp.target kernel_type(generic) map_entries(%map_inl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @enl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.200000e+00, ptr %[[REF]], align 4
+    %map_enl = omp.map.info var_ptr(%17 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "enl"}
+    omp.target kernel_type(generic) map_entries(%map_enl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 4.100000e+00, ptr @iae, align 4
+    %map_iae = omp.map.info var_ptr(%7 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "iae"}
+    omp.target kernel_type(generic) map_entries(%map_iae -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(4.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 4.200000e+00, ptr @eae, align 4
+    %map_eae = omp.map.info var_ptr(%19 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "eae"}
+    omp.target kernel_type(generic) map_entries(%map_eae -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(4.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 5.100000e+00, ptr @ihe, align 4
+    %map_ihe = omp.map.info var_ptr(%9 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ihe"}
+    omp.target kernel_type(generic) map_entries(%map_ihe -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(5.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 5.200000e+00, ptr @ehe, align 4
+    %map_ehe = omp.map.info var_ptr(%21 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ehe"}
+    omp.target kernel_type(generic) map_entries(%map_ehe -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(5.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 6.100000e+00, ptr @ine, align 4
+    %map_ine = omp.map.info var_ptr(%11 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ine"}
+    omp.target kernel_type(generic) map_entries(%map_ine -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(6.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 6.210000e+00, ptr @ene, align 4
+    %map_ene = omp.map.info var_ptr(%23 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ene"}
+    omp.target kernel_type(generic) map_entries(%map_ene -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(6.210000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    llvm.return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
index bc59fdda2cc25..745ab424fb5f8 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
@@ -8,6 +8,7 @@
 // file created by the host and appended as an attribute to the module.
 
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK-DAG: @_QMtest_0Esp = internal global i32
   // CHECK-DAG: @_QMtest_0Esp_decl_tgt_ref_ptr = weak global ptr null, align 8
   llvm.mlir.global external @_QMtest_0Esp() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : i32 {
     %0 = llvm.mlir.constant(0 : i32) : i32


From llvmlistbot at llvm.org  Wed Jul  8 03:18:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 03:18:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4e23fa.0e9b1f07.395ac1.62e5@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207307

>From efeff5c5f0e40eeef32c158ca5b7138647e10aad Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:16:56 +0000
Subject: [PATCH] [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td |  6 ++-
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp  |  9 ++++
 mlir/test/Dialect/LLVMIR/nvvm.mlir          | 48 +++++++++++++++++++++
 mlir/test/Target/LLVMIR/nvvmir.mlir         | 48 +++++++++++++++++++++
 4 files changed, 110 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..67e0ef41d927f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;
diff --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..ae3d18ec46812 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,54 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3_f16
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3_f16(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2_f16
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2_f16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {
diff --git a/mlir/test/Target/LLVMIR/nvvmir.mlir b/mlir/test/Target/LLVMIR/nvvmir.mlir
index f2888025d8a08..778a5c2229225 100644
--- a/mlir/test/Target/LLVMIR/nvvmir.mlir
+++ b/mlir/test/Target/LLVMIR/nvvmir.mlir
@@ -293,6 +293,54 @@ llvm.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3_f16
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3_f16(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: call { <2 x half>, <2 x half> } @llvm.nvvm.mma.m16n8k16.row.col.f16.e4m3.e4m3.f16
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2_f16
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2_f16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: call { <2 x half>, <2 x half> } @llvm.nvvm.mma.m16n8k32.row.col.f16.e4m3.e5m2.f16
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
 // f32 return type, f32 accumulate type
 // CHECK-LABEL: @nvvm_mma_m16n8k16_f32_f32
 llvm.func @nvvm_mma_m16n8k16_f32_f32(%a0 : vector<2xf16>, %a1 : vector<2xf16>,


From llvmlistbot at llvm.org  Wed Jul  8 03:19:30 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Wed, 08 Jul 2026 03:19:30 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4e2432.f1c1923d.991d1.ab0b@mx.google.com>


skatrak wrote:

PR stack:
- #208188 ◀️
- #208189

https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Wed Jul  8 03:21:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 03:21:16 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4e249c.eb165e5c.1817a5.a7c8@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-openmp

@llvm/pr-subscribers-mlir

Author: Sergio Afonso (skatrak)

<details>
<summary>Changes</summary>

This patch introduces various changes to the handling of `declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when compiling for an OpenMP offload target. This prevents potential symbol redefinition issues related to globals that don't actually exist on the device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to prevent their associated "internal" linkage from producing broken device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g. `declare_target link(...)`), the associated and unused full-storage global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to external linkage when compiling for a target device, causing linker errors if accessed. This mirrors Clang's behavior.

Fixes #<!-- -->195188, fixes #<!-- -->195468.

Assisted-by: Claude Opus 4.8.

---

Patch is 23.43 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/208188.diff


5 Files Affected:

- (modified) flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp (+20-5) 
- (modified) flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir (+17-4) 
- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+44-9) 
- (added) mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir (+252) 
- (modified) mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir (+1) 


``````````diff
diff --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
index e42e10a82e6c2..05ea0bc08c4ab 100644
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
@@ -65,6 +65,15 @@ static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
   rewrites.insert(value);
 }
 
+/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
 namespace {
 class HostOpFilteringPass
     : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
@@ -78,11 +87,8 @@ class HostOpFilteringPass
 
     op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
       omp::DeclareTargetDeviceType declareType =
-          omp::DeclareTargetDeviceType::host;
-      auto declareTargetOp =
-          dyn_cast<omp::DeclareTargetInterface>(funcOp.getOperation());
-      if (declareTargetOp && declareTargetOp.isDeclareTarget())
-        declareType = declareTargetOp.getDeclareTargetDeviceType();
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
 
       // Only process host function definitions.
       if (funcOp.isExternal() ||
@@ -95,6 +101,15 @@ class HostOpFilteringPass
       }
       return WalkResult::advance();
     });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time issues.
+    op->walk([&](fir::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkName("internal");
+    });
   }
 
 private:
diff --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
index f3b15bd028bc4..3acdfda876812 100644
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
@@ -523,10 +523,6 @@ module attributes {omp.is_target_device = true} {
   }
 
   func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  fir.global internal @global_scalar constant : i32 {
-    %0 = arith.constant 10 : i32
-    fir.has_value %0 : i32
-  }
   omp.private {type = firstprivate} @privatizer : i32 copy {
   ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
     %0 = fir.load %arg0 : !fir.ref<i32>
@@ -544,4 +540,21 @@ module attributes {omp.is_target_device = true} {
     %1 = arith.addi %arg0, %arg1 : i32
     omp.yield (%1 : i32)
   }
+
+  // CHECK: fir.global internal @global_scalar constant : i32
+  fir.global @global_scalar constant : i32
+
+  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..b2d2ef6d8f770 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8843,14 +8843,16 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (LLVM::GlobalOp gOp = dyn_cast<LLVM::GlobalOp>(op)) {
     llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
     if (auto *gVal = llvmModule->getNamedValue(gOp.getSymName())) {
+      auto *gVar = cast<llvm::GlobalVariable>(gVal);
       llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
       bool isDeclaration = gOp.isDeclaration();
       bool isExternallyVisible =
           gOp.getVisibility() != mlir::SymbolTable::Visibility::Private;
       auto loc = op->getLoc()->findInstanceOf<FileLineColLoc>();
       llvm::StringRef mangledName = gOp.getSymName();
-      auto captureClause =
-          convertToCaptureClauseKind(attribute.getCaptureClause().getValue());
+      mlir::omp::DeclareTargetCaptureClause captureClause =
+          attribute.getCaptureClause().getValue();
+      auto captureClauseKind = convertToCaptureClauseKind(captureClause);
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
       // unused for MLIR at the moment, required in Clang for book
@@ -8877,31 +8879,64 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                                                      lineNo);
       };
 
-      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
+      bool isToOrEnter =
+          captureClause == omp::DeclareTargetCaptureClause::to ||
+          captureClause == omp::DeclareTargetCaptureClause::enter;
+      bool isHostOnly = attribute.getDeviceType().getValue() ==
+                        omp::DeclareTargetDeviceType::host;
+
+      // A to/enter declare-target variable needs a device-resident,
+      // name-resolvable copy and a host offloading entry. A local-linkage
+      // global provides neither, so we promote it to external.
+      if (isToOrEnter && !isHostOnly && !requiresUSM &&
+          gVar->hasLocalLinkage()) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        isExternallyVisible = true;
+
+        // Clear the stale dso_local flag so it is referenced like a
+        // module-scope declare target global.
+        if (ompBuilder->Config.isTargetDevice())
+          gVar->setDSOLocal(false);
+      }
 
+      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
       ompBuilder->registerTargetGlobalVariable(
-          captureClause, deviceClause, isDeclaration, isExternallyVisible,
+          captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
           mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
           /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
           gVal->getType(), gVal);
 
-      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
       if (ompBuilder->Config.isTargetDevice() &&
-          (attribute.getCaptureClause().getValue() ==
-               mlir::omp::DeclareTargetCaptureClause::link ||
+          (captureClause == omp::DeclareTargetCaptureClause::link ||
            requiresUSM)) {
         llvm::Type *ptrTy = gVal->getType();
         // For USM the global type becomes a pointer handle, as opposed to the
         // globals original type.
         if (requiresUSM)
           ptrTy = llvm::PointerType::get(llvmModule->getContext(), 0);
-        ompBuilder->getAddrOfDeclareTargetVar(
-            captureClause, deviceClause, isDeclaration, isExternallyVisible,
+        bool addrGlobalCreated = ompBuilder->getAddrOfDeclareTargetVar(
+            captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
             ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
             mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
             ptrTy, /*GlobalInitializer*/ nullptr,
             /*VariableLinkage*/ nullptr);
+
+        // For indirectly-accessed global pointers, we rely on "internal"
+        // linkage to optimize out the unneeded full-variable storage later,
+        // since we can't prevent the LLVM dialect from generating globals
+        // without also breaking target lowering.
+        if (addrGlobalCreated)
+          gVar->setLinkage(llvm::GlobalValue::InternalLinkage);
+      }
+
+      // Mark 'device_type(host) enter(...)' variables as external in the device
+      // since they're not supposed to have their own copy. This will cause
+      // linker errors if accesses are attempted from the target device.
+      if (ompBuilder->Config.isTargetDevice() && isHostOnly && isToOrEnter) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        gVar->setInitializer(nullptr);
       }
     }
   }
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
new file mode 100644
index 0000000000000..2cd3fc3c44231
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
@@ -0,0 +1,252 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Tests lowering to LLVM IR of all combinations of OpenMP declare target
+// device_type (any/host/nohost) and capture clause (link/enter) when compiling
+// for device, for both external and internal linkage global variables.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  // --- link any ---
+
+  // CHECK-DAG: @ial = internal global float 0.000000e+00
+  // CHECK-DAG: @ial_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ial() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eal = internal global float 0.000000e+00
+  // CHECK-DAG: @eal_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @eal() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link host ---
+
+  // CHECK-DAG: @ihl = internal global float 0.000000e+00
+  // CHECK-DAG: @ihl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ihl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehl = internal global float 0.000000e+00
+  // CHECK-DAG: @ehl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @ehl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link nohost ---
+
+  // CHECK-DAG: @inl = internal global float 0.000000e+00
+  // CHECK-DAG: @inl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @inl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @enl = internal global float 0.000000e+00
+  // CHECK-DAG: @enl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @enl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter any ---
+
+  // CHECK-DAG: @iae = global float 0.000000e+00
+  llvm.mlir.global internal @iae() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eae = dso_local global float 0.000000e+00
+  llvm.mlir.global external @eae() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter host ---
+
+  // CHECK-DAG: @ihe = external dso_local global float
+  llvm.mlir.global internal @ihe() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehe = external dso_local global float
+  llvm.mlir.global external @ehe() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter nohost ---
+
+  // CHECK-DAG: @ine = global float 0.000000e+00
+  llvm.mlir.global internal @ine() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ene = dso_local global float 0.000000e+00
+  llvm.mlir.global external @ene() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  llvm.func @_QQmain() {
+    %0 = llvm.mlir.addressof @ial : !llvm.ptr
+    %1 = llvm.addrspacecast %0 : !llvm.ptr to !llvm.ptr
+    %2 = llvm.mlir.addressof @ihl : !llvm.ptr
+    %3 = llvm.addrspacecast %2 : !llvm.ptr to !llvm.ptr
+    %4 = llvm.mlir.addressof @inl : !llvm.ptr
+    %5 = llvm.addrspacecast %4 : !llvm.ptr to !llvm.ptr
+    %6 = llvm.mlir.addressof @iae : !llvm.ptr
+    %7 = llvm.addrspacecast %6 : !llvm.ptr to !llvm.ptr
+    %8 = llvm.mlir.addressof @ihe : !llvm.ptr
+    %9 = llvm.addrspacecast %8 : !llvm.ptr to !llvm.ptr
+    %10 = llvm.mlir.addressof @ine : !llvm.ptr
+    %11 = llvm.addrspacecast %10 : !llvm.ptr to !llvm.ptr
+    %12 = llvm.mlir.addressof @eal : !llvm.ptr
+    %13 = llvm.addrspacecast %12 : !llvm.ptr to !llvm.ptr
+    %14 = llvm.mlir.addressof @ehl : !llvm.ptr
+    %15 = llvm.addrspacecast %14 : !llvm.ptr to !llvm.ptr
+    %16 = llvm.mlir.addressof @enl : !llvm.ptr
+    %17 = llvm.addrspacecast %16 : !llvm.ptr to !llvm.ptr
+    %18 = llvm.mlir.addressof @eae : !llvm.ptr
+    %19 = llvm.addrspacecast %18 : !llvm.ptr to !llvm.ptr
+    %20 = llvm.mlir.addressof @ehe : !llvm.ptr
+    %21 = llvm.addrspacecast %20 : !llvm.ptr to !llvm.ptr
+    %22 = llvm.mlir.addressof @ene : !llvm.ptr
+    %23 = llvm.addrspacecast %22 : !llvm.ptr to !llvm.ptr
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ial_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.100000e+00, ptr %[[REF]], align 4
+    %map_ial = omp.map.info var_ptr(%1 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ial"}
+    omp.target kernel_type(generic) map_entries(%map_ial -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @eal_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.200000e+00, ptr %[[REF]], align 4
+    %map_eal = omp.map.info var_ptr(%13 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "eal"}
+    omp.target kernel_type(generic) map_entries(%map_eal -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ihl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.100000e+00, ptr %[[REF]], align 4
+    %map_ihl = omp.map.info var_ptr(%3 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ihl"}
+    omp.target kernel_type(generic) map_entries(%map_ihl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ehl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.200000e+00, ptr %[[REF]], align 4
+    %map_ehl = omp.map.info var_ptr(%15 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ehl"}
+    omp.target kernel_type(generic) map_entries(%map_ehl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @inl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.100000e+00, ptr %[[REF]], align 4
+    %map_inl = omp.map.info var_ptr(%5 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "inl"}
+    omp.target kernel_type(generic) map_entries(%map_inl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @enl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.200000e+00, ptr %[[REF]], align 4
+    %map_enl = omp.map.info var_ptr(%17 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "enl"}
+    omp.target kernel_type(generic) map_entries(%map_enl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 4.100000e+00, ptr @iae, align 4
+    %map_iae = omp.map.info var_ptr(%7 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "iae"}
+    omp.target kernel_type(generic) map_entries(%map_iae -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(4.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Wed Jul  8 03:21:52 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 08 Jul 2026 03:21:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e24c0.13be6df8.145c.a359@mx.google.com>


https://github.com/lhutton1 approved this pull request.

Thanks @GleasonK @jpienaar. If it helps unblock things, I'm happy to remove the op dependency with the changes in this PR as it may take me some time to get my head around the table-gen changes required based on @jpienaar's suggestion.

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 03:22:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 03:22:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a4e24d7.92d2873c.1ab170.658d@mx.google.com>


weimin023 wrote:

Hi @schwarzschild-radius, thanks for reviewing, missing tests for 2 x f16x2 are added.

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Wed Jul  8 03:23:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 08 Jul 2026 03:23:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e2509.a87eb5dc.7d817.f133@mx.google.com>


lhutton1 wrote:

@mydatascience - would it be possible to update the commit message to reflect the changes being made?

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 03:31:23 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fb.7270d2fd.2d4879.9293@mx.google.com>


================

----------------
banach-space wrote:

Missing docs and also:
*  `scrType` - source of what?, and
* `resType` of what?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:23 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fb.510f576d.1b3cb9.5c96@mx.google.com>


================

----------------
banach-space wrote:

"under the previous effectively-1D" - where is this restriction implemented?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:23 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fb.6c8668f2.e4f82.7ce0@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
          op, "copy accesses are OOB");
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.e9d484b9.15e549.8399@mx.google.com>


================

----------------
banach-space wrote:

* Please clarify what "scalar case" means (it's not obvious)
* What does "same-dimension" mean in this context?

Also, if this comment is referring to the whole block, I'd put it above the `if` statement.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.70920dee.290dca.37c2@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
/// Returns reinterpret_cast's result non-unit dimensions and, for static offsets,
/// delinearized offset.
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.1905e68e.18ad68.f854@mx.google.com>


================

----------------
banach-space wrote:

[nit] To me it would make more sense to bail out rather than assert  It's a matter of taste at this point.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.58bdd12c.19c81.c472@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
/// identity-layout sources. In addition:
///  * Non-scalar results must have static offsets, static result strides
///     identical to the source identity strides, and exactly one non-unit source
///     dimension collapsed to unit size 
/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars, strides are effectively irrelevant).  
///
/// Returns nullopt for unsupported
/// reinterpret_casts.
```

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.07d7ca98.9dee3.e992@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
    // Constant Op cache to reuse common index constants across bounds, steps, and static offsets,
```

Could you also document at `0` is stored at index `0` and `1` is stored at index `1`?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.e55dc341.1f19bd.69b7@mx.google.com>


================

----------------
banach-space wrote:

This error method should somehow be linked to what `getResultNonUnitDimsAndOffsetsForRC` does.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.2020f6c2.243e78.eb22@mx.google.com>


================

----------------
banach-space wrote:

I don't understand the term "varying indices". What do you mean? From what I can tell, this is searching for non-unit dimension in the result shape?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fc.f732ad81.3a4994.3112@mx.google.com>


================

----------------
banach-space wrote:

I don't see `delinearizeStaticRCOffset` ever failing, so this `assert` will never execute.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:25 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fd.3cce8983.225732.ddc7@mx.google.com>


================

----------------
banach-space wrote:

The function name (`delinearizeStaticRCOffset`) and the comment ("returns source indices") are inconsistent. Could you use one of the terms consistently (i.e. `delinearize` vs `source indices`)? I prefer `delinearize` as I'm not sure what "source indices" means without documenting (*).

(*) I know what _you_ mean, but based on my experience "delinearized offset" is more standard term.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 03:31:25 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 03:31:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e26fd.af6ba3ed.36d9db.e72a@mx.google.com>


================

----------------
banach-space wrote:

```suggestion
    bool isOffsetDynamic = ShapedType::isDynamic(rcOffsets[0]);
    if (isOffsetDynamic)
```

... and then re-use `isOffsetDynamic` below.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 04:11:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:11:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a4e3044.d1496011.26913c.85c1@mx.google.com>


https://github.com/weimin023 edited https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Wed Jul  8 04:12:50 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 04:12:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a4e30b2.481afadf.196d57.a301@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/205774

>From e9787b54e9992a9ba4ab61be46286d3ff7d27f54 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 14:05:19 +0200
Subject: [PATCH 1/4] [mlir][EmitC] Add rank-0 MemRef conversion

---
 .../Conversion/EmitCCommon/TypeConverter.cpp  |  3 +-
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 72 ++++++++++++++-----
 .../FuncToEmitC/func-to-emitc-failed.mlir     | 15 ----
 .../memref-to-emitc-alloc-dealloc.mlir        | 26 +++++++
 .../memref-to-emitc-alloc-load-store.mlir     | 48 +++++++++++++
 .../MemRefToEmitC/memref-to-emitc-copy.mlir   | 21 +++++-
 6 files changed, 150 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
index a65edcf8306b8..1722fb95524d5 100644
--- a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
+++ b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
@@ -18,7 +18,6 @@ namespace {
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -50,6 +49,8 @@ EmitCTypeConverter::EmitCTypeConverter(MLIRContext *ctx) {
     if (!convertedElementType)
       return {};
 
+    if (memRefType.getRank() == 0)
+      return emitc::PointerType::get(convertedElementType);
     return emitc::ArrayType::get(memRefType.getShape(), convertedElementType);
   });
 
diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 8acb737f0e9b8..3af48fe23d140 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -30,7 +30,6 @@ using namespace mlir;
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -73,6 +72,11 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with dynamic shape");
     }
 
+    if (op.getType().getRank() == 0) {
+      return rewriter.notifyMatchFailure(
+          op.getLoc(), "cannot transform alloca with zero rank");
+    }
+
     if (op.getAlignment().value_or(1) > 1) {
       // TODO: Allow alignment if it is not more than the natural alignment
       // of the C array.
@@ -142,9 +146,12 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-// If `v` is defined through an unrealized cast and the source of that cast
-// is `emitc.ptr`, return the pointer.
-static Value stripPointerUnrealizedCast(Value v) {
+static Value getPointerOrStripUnrealizedCast(Value v) {
+  if (isa<emitc::PointerType>(v.getType()))
+    return v;
+
+  // If `v` is defined through an unrealized cast and the source of that cast
+  // is `emitc.ptr`, return the pointer.
   if (auto cast = v.getDefiningOp<UnrealizedConversionCastOp>())
     if (cast.getNumOperands() == 1 &&
         isa<emitc::PointerType>(cast.getOperand(0).getType()))
@@ -165,6 +172,9 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
+  if (shape.empty())
+    return linearIndex;
+
   for (auto [dim, idx] : llvm::zip(shape.drop_front(), indices.drop_front())) {
     Value dimSize =
         emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(dim));
@@ -238,10 +248,9 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = deallocOp.getLoc();
     // `free` can only be emitted when the dealloc operand is recoverable as an
-    // `emitc.ptr<T>`. In the current conversion, that happens via an
-    // unrealized_conversion_cast from the pointer-backed EmitC form.
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr) {
+    // `emitc.ptr<T>`.
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr) {
       return rewriter.notifyMatchFailure(
           loc, "expected pointer-backed memref for EmitC deallocation");
     }
@@ -252,7 +261,7 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
     Type opaqueVoidPtrType = emitc::PointerType::get(
         emitc::OpaqueType::get(rewriter.getContext(), "void"));
     Value freeArg =
-        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, strippedPtr);
+        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, ptr);
     emitc::CallOpaqueOp freeCall = emitc::CallOpaqueOp::create(
         rewriter, loc, TypeRange{}, rewriter.getStringAttr(freeFunctionName),
         ValueRange{freeArg});
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getPointerOrStripUnrealizedCast(operands.getSource());
+      Value targetPtr = getPointerOrStripUnrealizedCast(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
+                                         srcLValue.getResult());
+
+      auto targetLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(targetPtr),
+          zeroIndex);
+      rewriter.replaceOpWithNewOp<emitc::AssignOp>(
+          copyOp, targetLValue.getResult(), value.getResult());
+      return success();
+    }
+
     auto srcArrayValue =
         cast<TypedValue<emitc::ArrayType>>(operands.getSource());
     emitc::AddressOfOp srcPtr =
@@ -414,8 +452,8 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
 
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
 
@@ -423,14 +461,14 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 
@@ -448,8 +486,8 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
     Location loc = op.getLoc();
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
       rewriter.replaceOpWithNewOp<emitc::AssignOp>(op, subscript,
@@ -457,14 +495,14 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 
diff --git a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
index 46e5319d7d17c..85410ec0134e6 100644
--- a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
@@ -7,21 +7,6 @@ func.func @unsuppoted_emitc_type(%arg0: i4) -> i4 {
 
 // -----
 
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_alloc() -> memref<i32> {
-  %alloc = memref.alloc() : memref<i32>
-  return %alloc : memref<i32>
-}
-
-// -----
-
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_arg(%arg0: memref<i32>) -> memref<i32> {
-  return %arg0 : memref<i32>
-}
-
-// -----
-
 // expected-error at +1 {{failed to legalize operation 'func.func'}}
 func.func private @return_rank1_alloc() -> memref<1xi32> {
   %alloc = memref.alloc() : memref<1xi32>
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
index 9d20667a6490c..3194a40c16eeb 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
@@ -89,3 +89,29 @@ func.func @allocating_and_deallocating_multi() {
 // NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
 // NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
 // NOCPP-NEXT: return
+
+func.func @alloc_and_dealloc_rank0() {
+  %alloc = memref.alloc() : memref<i32>
+  memref.dealloc %alloc : memref<i32>
+  return
+}
+
+// CPP-LABEL: alloc_and_dealloc_rank0
+// CPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// CPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// CPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// CPP-NEXT: return
+
+// NOCPP-LABEL: alloc_and_dealloc_rank0
+// NOCPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// NOCPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// NOCPP-NEXT: return
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
index 07cad3b0c4dc2..5e466630a66a0 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
@@ -72,3 +72,51 @@ func.func @memref_load_store(%buff0: memref<2xf32>,
   memref.store %v, %buff1[%i, %j] : memref<4x8xf32>
   return
 }
+
+/// Rank-0 alloc-backed load/store lower through pointer subscript at index 0.
+// CHECK-LABEL: emitc.func private @memref_rank0_alloc_store(
+// CHECK-SAME:  %[[VAL:.*]]: i32)
+func.func private @memref_rank0_alloc_store(%v : i32) {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[ELEM_LVALUE]] : <i32>
+  memref.store %v, %alloc[] : memref<i32>
+  return
+}
+
+// CHECK-LABEL: emitc.func private @memref_rank0_alloc_load() -> i32
+func.func private @memref_rank0_alloc_load() -> i32 {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[LOADED_VAL:.*]] = load %[[ELEM_LVALUE]] : <i32>
+  %v = memref.load %alloc[] : memref<i32>
+  // CHECK:     return %[[LOADED_VAL]] : i32
+  return %v : i32
+}
+
+// CHECK-LABEL: emitc.func @memref_rank0_load_store(
+// CHECK-SAME:  %[[SRC:.*]]: !emitc.ptr<i32>,
+// CHECK-SAME:  %[[DST:.*]]: !emitc.ptr<i32>)
+func.func @memref_rank0_load_store(%src: memref<i32>, %dst: memref<i32>) {
+  // CHECK:     %[[LOAD_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[SRC_LVALUE:.*]] = subscript %[[SRC]]{{\[}}%[[LOAD_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[VAL:.*]] = load %[[SRC_LVALUE]] : <i32>
+  %v = memref.load %src[] : memref<i32>
+  // CHECK:     %[[STORE_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[DST_LVALUE:.*]] = subscript %[[DST]]{{\[}}%[[STORE_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[DST_LVALUE]] : <i32>
+  memref.store %v, %dst[] : memref<i32>
+  return
+}
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
index 04e6edd5b6981..de3741e83c01c 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
@@ -1,5 +1,5 @@
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s -split-input-file | FileCheck %s --check-prefixes=CPP,CHECK
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s -split-input-file | FileCheck %s --check-prefixes=NOCPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s | FileCheck %s --check-prefixes=CPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s | FileCheck %s --check-prefixes=NOCPP,CHECK
 
 func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
   memref.copy %arg0, %arg1 : memref<9x4x5x7xf32> to memref<9x4x5x7xf32>
@@ -28,3 +28,20 @@ func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
 // CHECK:           return
 // CHECK:         }
 
+func.func @copying_rank0(%arg0 : memref<i32>, %arg1 : memref<i32>) {
+  memref.copy %arg0, %arg1 : memref<i32> to memref<i32>
+  return
+}
+
+// CHECK-LABEL:   func.func @copying_rank0(
+// CHECK-SAME:      %[[ARG0:.*]]: memref<i32>,
+// CHECK-SAME:      %[[ARG1:.*]]: memref<i32>) {
+// CHECK:           %[[TARGET_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG1]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[SOURCE_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG0]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+// CHECK:           %[[SOURCE_LVALUE:.*]] = emitc.subscript %[[SOURCE_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           %[[VALUE:.*]] = emitc.load %[[SOURCE_LVALUE]] : <i32>
+// CHECK:           %[[TARGET_LVALUE:.*]] = emitc.subscript %[[TARGET_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           emitc.assign %[[VALUE]] : i32 to %[[TARGET_LVALUE]] : <i32>
+// CHECK:           return
+// CHECK:         }

>From bd62e993d9d6af4c1f3bae4ca1007e89116b673e Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 17:31:08 +0200
Subject: [PATCH 2/4] Support lowering rank-0 AllocaOp

---
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 44 +++++++++++--------
 .../MemRefToEmitC/memref-to-emitc-failed.mlir |  8 ----
 .../MemRefToEmitC/memref-to-emitc.mlir        |  9 ++++
 3 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 3af48fe23d140..6a76ddddd4eeb 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -59,6 +59,16 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
+Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
+  Type resultTy;
+  if (opTy.getRank() == 0) {
+    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
+  } else {
+    resultTy = typeConverter->convertType(opTy);
+  }
+  return resultTy;
+}
+
 namespace {
 struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
   using OpConversionPattern::OpConversionPattern;
@@ -72,11 +82,6 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with dynamic shape");
     }
 
-    if (op.getType().getRank() == 0) {
-      return rewriter.notifyMatchFailure(
-          op.getLoc(), "cannot transform alloca with zero rank");
-    }
-
     if (op.getAlignment().value_or(1) > 1) {
       // TODO: Allow alignment if it is not more than the natural alignment
       // of the C array.
@@ -84,26 +89,29 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = getTypeConverter()->convertType(op.getType());
-    if (!resultTy) {
+    auto resultTy = convertMemRefType(op.getType(), getTypeConverter());
+    if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
-    }
+
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
+    // Rank-0 path
+    if (op.getType().getRank() == 0) {
+      auto var = emitc::VariableOp::create(
+          rewriter, op.getLoc(), emitc::LValueType::get(resultTy), noInit);
+
+      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(),
+                                            emitc::PointerType::get(resultTy),
+                                            var.getResult());
+
+      rewriter.replaceOp(op, ptr.getResult());
+      return success();
+    }
+    // Rank > 0 path
     rewriter.replaceOpWithNewOp<emitc::VariableOp>(op, resultTy, noInit);
     return success();
   }
 };
 
-Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
-  Type resultTy;
-  if (opTy.getRank() == 0) {
-    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  } else {
-    resultTy = typeConverter->convertType(opTy);
-  }
-  return resultTy;
-}
-
 static Value calculateMemrefTotalSizeBytes(Location loc, MemRefType memrefType,
                                            OpBuilder &builder,
                                            Type convertedElementType) {
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
index 753f749fabbd6..f2d0e9a5eee66 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
@@ -72,14 +72,6 @@ func.func @non_identity_layout() {
 
 // -----
 
-func.func @zero_rank() {
-  // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
-  %0 = memref.alloca() : memref<f32>
-  return
-}
-
-// -----
-
 func.func @zero_dim_rank_1() {
   // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
   %0 = memref.alloca() : memref<0xf32>
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
index 5ae22490ff05e..9749906e0c98a 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
@@ -10,6 +10,15 @@ func.func @alloca() {
 
 // -----
 
+func.func @alloca_zero_rank() {
+  // CHECK: %[[LVALUE:.*]] = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<f32>
+  // CHECK: %[[PTR:.*]] = emitc.address_of %[[LVALUE]] : !emitc.lvalue<f32>
+  %0 = memref.alloca() : memref<f32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: memref_store
 // CHECK-SAME:  %[[buff:.*]]: memref<4x8xf32>, %[[v:.*]]: f32, %[[i:.*]]: index, %[[j:.*]]: index
 func.func @memref_store(%buff : memref<4x8xf32>, %v : f32, %i: index, %j: index) {

>From 780f95dbff109317cbf329c24e93d31c8563d49d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 15:00:58 +0200
Subject: [PATCH 3/4] Address first round of comments

---
 mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp       | 7 +++----
 .../Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir    | 8 ++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 6a76ddddd4eeb..b87bcd5476596 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -61,11 +61,10 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 
 Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
   Type resultTy;
-  if (opTy.getRank() == 0) {
+  if (opTy.getRank() == 0)
     resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  } else {
+  else
     resultTy = typeConverter->convertType(opTy);
-  }
   return resultTy;
 }
 
@@ -180,7 +179,7 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
-  if (shape.empty())
+  if (indices.empty())
     return linearIndex;
 
   for (auto [dim, idx] : llvm::zip(shape.drop_front(), indices.drop_front())) {
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
index de3741e83c01c..6105521d9326d 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
@@ -1,5 +1,5 @@
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s | FileCheck %s --check-prefixes=CPP,CHECK
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s | FileCheck %s --check-prefixes=NOCPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s -split-input-file | FileCheck %s --check-prefixes=CPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s -split-input-file | FileCheck %s --check-prefixes=NOCPP,CHECK
 
 func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
   memref.copy %arg0, %arg1 : memref<9x4x5x7xf32> to memref<9x4x5x7xf32>
@@ -28,11 +28,15 @@ func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
 // CHECK:           return
 // CHECK:         }
 
+// -----
+
 func.func @copying_rank0(%arg0 : memref<i32>, %arg1 : memref<i32>) {
   memref.copy %arg0, %arg1 : memref<i32> to memref<i32>
   return
 }
 
+// CHECK: module {
+
 // CHECK-LABEL:   func.func @copying_rank0(
 // CHECK-SAME:      %[[ARG0:.*]]: memref<i32>,
 // CHECK-SAME:      %[[ARG1:.*]]: memref<i32>) {

>From 77174a0aa7c301d85f20897f1fc24b2d58e3d0b0 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 13:11:58 +0200
Subject: [PATCH 4/4] Remove wrapper for rank-0 conversion

---
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 46 +++++++++----------
 1 file changed, 22 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index b87bcd5476596..8275749144f8d 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -59,15 +59,6 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
-Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
-  Type resultTy;
-  if (opTy.getRank() == 0)
-    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  else
-    resultTy = typeConverter->convertType(opTy);
-  return resultTy;
-}
-
 namespace {
 struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
   using OpConversionPattern::OpConversionPattern;
@@ -88,18 +79,20 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = convertMemRefType(op.getType(), getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(op.getType());
     if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
 
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
     // Rank-0 path
     if (op.getType().getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
       auto var = emitc::VariableOp::create(
-          rewriter, op.getLoc(), emitc::LValueType::get(resultTy), noInit);
+          rewriter, op.getLoc(), emitc::LValueType::get(elemTy), noInit);
 
-      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(),
-                                            emitc::PointerType::get(resultTy),
+      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(), resultTy,
                                             var.getResult());
 
       rewriter.replaceOp(op, ptr.getResult());
@@ -153,7 +146,7 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-static Value getPointerOrStripUnrealizedCast(Value v) {
+static Value getMemRefPointer(Value v) {
   if (isa<emitc::PointerType>(v.getType()))
     return v;
 
@@ -256,7 +249,7 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
     Location loc = deallocOp.getLoc();
     // `free` can only be emitted when the dealloc operand is recoverable as an
     // `emitc.ptr<T>`.
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr) {
       return rewriter.notifyMatchFailure(
           loc, "expected pointer-backed memref for EmitC deallocation");
@@ -304,8 +297,8 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       if (!elementType)
         return rewriter.notifyMatchFailure(loc, "cannot convert element type");
 
-      Value srcPtr = getPointerOrStripUnrealizedCast(operands.getSource());
-      Value targetPtr = getPointerOrStripUnrealizedCast(operands.getTarget());
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
       if (!srcPtr || !targetPtr)
         return rewriter.notifyMatchFailure(loc, "expected pointer operands");
 
@@ -376,7 +369,7 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
                        "currently not supported");
     }
 
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -397,6 +390,9 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
 
     Attribute initialValue = operands.getInitialValueAttr();
     if (opTy.getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      resultTy = pointerTy.getPointee();
       // special case for `variable : memref<i32> = dense<-1>`
       if (std::optional<Attribute> initValueAttr = op.getInitialValue()) {
         if (auto elementsAttr = llvm::dyn_cast<ElementsAttr>(*initValueAttr)) {
@@ -423,7 +419,7 @@ struct ConvertGetGlobal final
                   ConversionPatternRewriter &rewriter) const override {
 
     MemRefType opTy = op.getType();
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -431,11 +427,13 @@ struct ConvertGetGlobal final
     }
 
     if (opTy.getRank() == 0) {
-      emitc::LValueType lvalueType = emitc::LValueType::get(resultTy);
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
+      emitc::LValueType lvalueType = emitc::LValueType::get(elemTy);
       emitc::GetGlobalOp globalLValue = emitc::GetGlobalOp::create(
           rewriter, op.getLoc(), lvalueType, operands.getNameAttr());
-      emitc::PointerType pointerType = emitc::PointerType::get(resultTy);
-      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, pointerType,
+      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, resultTy,
                                                       globalLValue);
       return success();
     }
@@ -459,7 +457,7 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
 
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
@@ -493,7 +491,7 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
     Location loc = op.getLoc();
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());


From llvmlistbot at llvm.org  Wed Jul  8 04:21:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:21:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e32a8.da8bafd3.14cd50.3b53@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From b1737aa3922d37bd33bbe450da525b46111a5bf4 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH 1/2] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload

>From 561f1ab1a0472ae1416b66b56a180113653473f9 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Wed, 8 Jul 2026 13:20:02 +0200
Subject: [PATCH 2/2] [mlir][vector][NFC] Tidy getCompressedMaskOp bail-out and
 negative tests

Replace the break-then-recheck bail-out in the mask trace loop with an
early return when a defining op is not a vector.extract, and use
isa_and_present for the post-loop check so the block-argument (null
maskOp) case is handled without a separate guard. NFC.

Rename the negative tests in emulate-narrow-type-unsupported.mlir to use
the negative_ prefix per the MLIR testing guide, and move the
block-argument maskedload test into the vector.maskedload section.
---
 .../Transforms/VectorEmulateNarrowType.cpp    | 28 ++++-----
 .../emulate-narrow-type-unsupported.mlir      | 60 +++++++++----------
 2 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index e37296e50fb29..e237ae0568ae4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -89,26 +89,26 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
       numSrcElemsPerDest;
 
   Operation *maskOp = mask.getDefiningOp();
-  SmallVector<vector::ExtractOp, 2> extractOps;
+  // Chain of `vector.extract` ops that lead to the op that created the mask.
   // TODO: add support to `vector.broadcast`.
-  // Finding the mask creation operation.
+  SmallVector<vector::ExtractOp, 2> extractOps;
+
+  // Trace the mask back to its creation op, looking through `vector.extract`
+  // ops. Any other defining op is unsupported.
   while (maskOp &&
          !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
              maskOp)) {
-    if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
-      maskOp = extractOp.getSource().getDefiningOp();
-      extractOps.push_back(extractOp);
-    } else {
-      // Unsupported mask-defining op (e.g. a block argument, which has no
-      // defining op, or an op we cannot trace through). Bail out rather than
-      // looping forever or dereferencing a null op below.
-      break;
-    }
+    auto extractOp = dyn_cast<vector::ExtractOp>(maskOp);
+    if (!extractOp)
+      return failure();
+    maskOp = extractOp.getSource().getDefiningOp();
+    extractOps.push_back(extractOp);
   }
 
-  if (!maskOp ||
-      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
-          maskOp))
+  // `maskOp` is null when the mask is a block argument, which has no defining
+  // op.
+  if (!isa_and_present<arith::ConstantOp, vector::CreateMaskOp,
+                       vector::ConstantMaskOp>(maskOp))
     return failure();
 
   // Computing the "compressed" mask. All the emulation logic (i.e. computing
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index 74f72ed5a5d9c..d3abccf641914 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -7,14 +7,14 @@
 /// vector.load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4xi8> {
+func.func @negative_vector_load_2d_i8(%arg1: index, %arg2: index) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %1 = vector.load %0[%arg1, %arg2] : memref<3x4xi8>, vector<2x4xi8>
     return %1 : vector<2x4xi8>
 }
 
 // No support for loading 2D vectors - expect no conversions
-//  CHECK-LABEL: func @vector_load_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_load_2d_i8
 //        CHECK:   memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
@@ -24,14 +24,14 @@ func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4x
 /// vector.transfer_read
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> vector<2x8xi4> {
+func.func @negative_vector_transfer_read_2d_i4(%arg1: index, %arg2: index) -> vector<2x8xi4> {
     %c0 = arith.constant 0 : i4
     %0 = memref.alloc() : memref<3x8xi4>
     %1 = vector.transfer_read %0[%arg1, %arg2], %c0 {in_bounds = [true, true]} :
       memref<3x8xi4>, vector<2x8xi4>
     return %1 : vector<2x8xi4>
 }
-//  CHECK-LABEL: func @vector_transfer_read_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_transfer_read_2d_i4
 //        CHECK:  memref.alloc() : memref<3x8xi4>
 //    CHECK-NOT: i32
 
@@ -41,7 +41,7 @@ func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> ve
 /// vector.maskedload
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
+func.func @negative_vector_maskedload_2d_i8(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %mask = vector.create_mask %arg3, %arg3 : vector<2x4xi1>
     %1 = vector.maskedload %0[%arg1, %arg2], %mask, %passthru :
@@ -49,17 +49,31 @@ func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: i
     return %1 : vector<2x4xi8>
 }
 
-//  CHECK-LABEL: func @vector_maskedload_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedload_2d_i8
 //        CHECK:  memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
 // -----
 
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the load is left unconverted.
+
+func.func @negative_vector_maskedload_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @negative_vector_maskedload_i8_block_arg_mask
+//        CHECK: vector.maskedload
+
+// -----
+
 ///----------------------------------------------------------------------------------------
 /// vector.extract -> vector.masked_load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x16xi4> {
+func.func @negative_vector_extract_maskedload_2d_i4(%arg1: index) -> vector<8x8x16xi4> {
     %0 = memref.alloc() : memref<8x8x16xi4>
     %c0 = arith.constant 0 : index
     %c16 = arith.constant 16 : index
@@ -73,7 +87,7 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
     return %63 : vector<8x8x16xi4>
 }
 
-//  CHECK-LABEL: func @vector_extract_maskedload_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_extract_maskedload_2d_i4
 //        CHECK: memref.alloc() : memref<8x8x16xi4>
 //    CHECK-NOT: i32
 
@@ -83,13 +97,13 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
 /// vector.store
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
+func.func @negative_vector_store_2d_i8(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
     %0 = memref.alloc() : memref<4x8xi8>
     vector.store %arg0, %0[%arg1, %arg2] :memref<4x8xi8>, vector<2x8xi8>
     return
 }
 
-//  CHECK-LABEL: func @vector_store_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_store_2d_i8
 //        CHECK: memref.alloc() : memref<4x8xi8>
 //    CHECK-NOT: i32
 
@@ -99,41 +113,27 @@ func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg
 /// vector.maskedstore
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
+func.func @negative_vector_maskedstore_2d_i8(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
   %0 = memref.alloc() : memref<3x8xi8>
   %mask = vector.create_mask %arg2, %arg2 : vector<2x8xi1>
   vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<2x8xi1>, vector<2x8xi8>
   return
 }
 
-//  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedstore_2d_i8
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
 
 // -----
 
-// The mask is a block argument, so its defining op cannot be traced back to a
-// supported mask-creation op. This must gracefully bail out instead of crashing
-// (see llvm/llvm-project#206928).
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the store is left unconverted.
 
-func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+func.func @negative_vector_maskedstore_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
   %c0 = arith.constant 0 : index
   vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 
-//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//  CHECK-LABEL: func @negative_vector_maskedstore_i8_block_arg_mask
 //        CHECK: vector.maskedstore
-
-// -----
-
-// As above, but for `vector.maskedload`.
-
-func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
-  %c0 = arith.constant 0 : index
-  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
-  return %0 : vector<8xi8>
-}
-
-//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
-//        CHECK: vector.maskedload


From llvmlistbot at llvm.org  Wed Jul  8 04:22:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:22:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e32e6.f732ad81.3a4994.118a@mx.google.com>


================
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
----------------
marquisburg wrote:

Done, restructured to bail out early with `return failure()` when the op isn't a `vector.extract`, and I went ahead and updated the comments

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:25:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:25:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e338c.07d7ca98.9dee3.4b30@mx.google.com>


================
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
----------------
marquisburg wrote:

Still needed for the null case. A block-argument leaves `maskIOo` null and `isa<>` on null segfaults in a release build. I did go ahead and switch to `isa_and_present<...>` to stay null-safe without the explicit `!maskOp ||`

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:25:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:25:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e339a.7270d2fd.2d4879.a233@mx.google.com>


https://github.com/marquisburg edited https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:25:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:25:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e33be.481afadf.196d57.7665@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload
----------------
marquisburg wrote:

Thanks, renamed all the negative tests in the file to use the prefix.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:28:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:28:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e344f.7270d2fd.2d4879.1320@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
----------------
marquisburg wrote:

Done, renamed all the negative tests in the file to use the correct prefix as per the guidelines!

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:29:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:29:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e3481.fdc477d6.23591.4387@mx.google.com>


https://github.com/marquisburg deleted https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:29:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:29:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e349e.331483e7.2440f8.d74e@mx.google.com>


https://github.com/marquisburg deleted https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:30:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:30:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e34e4.3cce8983.225732.6934@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
----------------
marquisburg wrote:

Absolutely, all done. I just wanted to follow the existing convention in the file not to stick out like a sore thumb, but I've gone ahead and renamed all the negative tests in the file to use the correct prefix.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:32:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:32:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e353f.f732ad81.3a4994.afe3@mx.google.com>


================
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload
----------------
marquisburg wrote:

Moved the maskedload block-arg test into the `vector.maskedload` section, left the maskedstore one here.

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Wed Jul  8 04:37:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 04:37:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a4e367f.f06f4994.1fc857.946b@mx.google.com>


https://github.com/marquisburg updated https://github.com/llvm/llvm-project/pull/207299

>From b1737aa3922d37bd33bbe450da525b46111a5bf4 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Fri, 3 Jul 2026 02:40:25 +0200
Subject: [PATCH 1/2] [mlir][vector] Fix crash on untraceable masks in
 getCompressedMaskOp

The loop tracing a mask back to its creation op assumed the chain
always ends at a create_mask/constant_mask/arith.constant, a
block-argument mask made it fall through to isa<> on a null op
(segfault), and any other unhandled defining op spun it forever.

Fixes #206928
---
 .../Transforms/VectorEmulateNarrowType.cpp    |  8 +++++-
 .../emulate-narrow-type-unsupported.mlir      | 28 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e37296e50fb29 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,16 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    } else {
+      // Unsupported mask-defining op (e.g. a block argument, which has no
+      // defining op, or an op we cannot trace through). Bail out rather than
+      // looping forever or dereferencing a null op below.
+      break;
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..74f72ed5a5d9c 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,31 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument, so its defining op cannot be traced back to a
+// supported mask-creation op. This must gracefully bail out instead of crashing
+// (see llvm/llvm-project#206928).
+
+func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//        CHECK: vector.maskedstore
+
+// -----
+
+// As above, but for `vector.maskedload`.
+
+func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
+//        CHECK: vector.maskedload

>From 561f1ab1a0472ae1416b66b56a180113653473f9 Mon Sep 17 00:00:00 2001
From: GhostDragonWasTook <marquis at marquiseco.co.za>
Date: Wed, 8 Jul 2026 13:20:02 +0200
Subject: [PATCH 2/2] [mlir][vector][NFC] Tidy getCompressedMaskOp bail-out and
 negative tests

Replace the break-then-recheck bail-out in the mask trace loop with an
early return when a defining op is not a vector.extract, and use
isa_and_present for the post-loop check so the block-argument (null
maskOp) case is handled without a separate guard. NFC.

Rename the negative tests in emulate-narrow-type-unsupported.mlir to use
the negative_ prefix per the MLIR testing guide, and move the
block-argument maskedload test into the vector.maskedload section.
---
 .../Transforms/VectorEmulateNarrowType.cpp    | 28 ++++-----
 .../emulate-narrow-type-unsupported.mlir      | 60 +++++++++----------
 2 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index e37296e50fb29..e237ae0568ae4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -89,26 +89,26 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
       numSrcElemsPerDest;
 
   Operation *maskOp = mask.getDefiningOp();
-  SmallVector<vector::ExtractOp, 2> extractOps;
+  // Chain of `vector.extract` ops that lead to the op that created the mask.
   // TODO: add support to `vector.broadcast`.
-  // Finding the mask creation operation.
+  SmallVector<vector::ExtractOp, 2> extractOps;
+
+  // Trace the mask back to its creation op, looking through `vector.extract`
+  // ops. Any other defining op is unsupported.
   while (maskOp &&
          !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
              maskOp)) {
-    if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
-      maskOp = extractOp.getSource().getDefiningOp();
-      extractOps.push_back(extractOp);
-    } else {
-      // Unsupported mask-defining op (e.g. a block argument, which has no
-      // defining op, or an op we cannot trace through). Bail out rather than
-      // looping forever or dereferencing a null op below.
-      break;
-    }
+    auto extractOp = dyn_cast<vector::ExtractOp>(maskOp);
+    if (!extractOp)
+      return failure();
+    maskOp = extractOp.getSource().getDefiningOp();
+    extractOps.push_back(extractOp);
   }
 
-  if (!maskOp ||
-      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
-          maskOp))
+  // `maskOp` is null when the mask is a block argument, which has no defining
+  // op.
+  if (!isa_and_present<arith::ConstantOp, vector::CreateMaskOp,
+                       vector::ConstantMaskOp>(maskOp))
     return failure();
 
   // Computing the "compressed" mask. All the emulation logic (i.e. computing
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index 74f72ed5a5d9c..d3abccf641914 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -7,14 +7,14 @@
 /// vector.load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4xi8> {
+func.func @negative_vector_load_2d_i8(%arg1: index, %arg2: index) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %1 = vector.load %0[%arg1, %arg2] : memref<3x4xi8>, vector<2x4xi8>
     return %1 : vector<2x4xi8>
 }
 
 // No support for loading 2D vectors - expect no conversions
-//  CHECK-LABEL: func @vector_load_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_load_2d_i8
 //        CHECK:   memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
@@ -24,14 +24,14 @@ func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4x
 /// vector.transfer_read
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> vector<2x8xi4> {
+func.func @negative_vector_transfer_read_2d_i4(%arg1: index, %arg2: index) -> vector<2x8xi4> {
     %c0 = arith.constant 0 : i4
     %0 = memref.alloc() : memref<3x8xi4>
     %1 = vector.transfer_read %0[%arg1, %arg2], %c0 {in_bounds = [true, true]} :
       memref<3x8xi4>, vector<2x8xi4>
     return %1 : vector<2x8xi4>
 }
-//  CHECK-LABEL: func @vector_transfer_read_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_transfer_read_2d_i4
 //        CHECK:  memref.alloc() : memref<3x8xi4>
 //    CHECK-NOT: i32
 
@@ -41,7 +41,7 @@ func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> ve
 /// vector.maskedload
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
+func.func @negative_vector_maskedload_2d_i8(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %mask = vector.create_mask %arg3, %arg3 : vector<2x4xi1>
     %1 = vector.maskedload %0[%arg1, %arg2], %mask, %passthru :
@@ -49,17 +49,31 @@ func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: i
     return %1 : vector<2x4xi8>
 }
 
-//  CHECK-LABEL: func @vector_maskedload_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedload_2d_i8
 //        CHECK:  memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
 // -----
 
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the load is left unconverted.
+
+func.func @negative_vector_maskedload_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @negative_vector_maskedload_i8_block_arg_mask
+//        CHECK: vector.maskedload
+
+// -----
+
 ///----------------------------------------------------------------------------------------
 /// vector.extract -> vector.masked_load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x16xi4> {
+func.func @negative_vector_extract_maskedload_2d_i4(%arg1: index) -> vector<8x8x16xi4> {
     %0 = memref.alloc() : memref<8x8x16xi4>
     %c0 = arith.constant 0 : index
     %c16 = arith.constant 16 : index
@@ -73,7 +87,7 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
     return %63 : vector<8x8x16xi4>
 }
 
-//  CHECK-LABEL: func @vector_extract_maskedload_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_extract_maskedload_2d_i4
 //        CHECK: memref.alloc() : memref<8x8x16xi4>
 //    CHECK-NOT: i32
 
@@ -83,13 +97,13 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
 /// vector.store
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
+func.func @negative_vector_store_2d_i8(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
     %0 = memref.alloc() : memref<4x8xi8>
     vector.store %arg0, %0[%arg1, %arg2] :memref<4x8xi8>, vector<2x8xi8>
     return
 }
 
-//  CHECK-LABEL: func @vector_store_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_store_2d_i8
 //        CHECK: memref.alloc() : memref<4x8xi8>
 //    CHECK-NOT: i32
 
@@ -99,41 +113,27 @@ func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg
 /// vector.maskedstore
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
+func.func @negative_vector_maskedstore_2d_i8(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
   %0 = memref.alloc() : memref<3x8xi8>
   %mask = vector.create_mask %arg2, %arg2 : vector<2x8xi1>
   vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<2x8xi1>, vector<2x8xi8>
   return
 }
 
-//  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedstore_2d_i8
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
 
 // -----
 
-// The mask is a block argument, so its defining op cannot be traced back to a
-// supported mask-creation op. This must gracefully bail out instead of crashing
-// (see llvm/llvm-project#206928).
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the store is left unconverted.
 
-func.func @vector_maskedstore_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+func.func @negative_vector_maskedstore_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
   %c0 = arith.constant 0 : index
   vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
   return
 }
 
-//  CHECK-LABEL: func @vector_maskedstore_i8_block_arg_mask_negative
+//  CHECK-LABEL: func @negative_vector_maskedstore_i8_block_arg_mask
 //        CHECK: vector.maskedstore
-
-// -----
-
-// As above, but for `vector.maskedload`.
-
-func.func @vector_maskedload_i8_block_arg_mask_negative(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
-  %c0 = arith.constant 0 : index
-  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
-  return %0 : vector<8xi8>
-}
-
-//  CHECK-LABEL: func @vector_maskedload_i8_block_arg_mask_negative
-//        CHECK: vector.maskedload


From llvmlistbot at llvm.org  Wed Jul  8 04:40:14 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 08 Jul 2026 04:40:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e371e.bd93e731.251b21.d336@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
----------------
grypp wrote:

This attribute is integer-only. Making it `DefaultValuedAttr` would leak an integer-overflow flag onto float ops during round-trip, which is odd.

Current solution looks good ot me

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 04:40:15 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 08 Jul 2026 04:40:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e371f.8e75e0cb.5e108.0e8d@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
----------------
grypp wrote:

could we cut some of the comments

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 04:42:20 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 08 Jul 2026 04:42:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e379c.2020f6c2.243e78.3c4e@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
----------------
grypp wrote:

The better design could be add `none` to `MMAIntOverflow` enum and make `DefaultValuedAttr` in the op. But we don't have `none`. Let's not add it, because changing enum will cause a lot of codegen change churn on higher level compiler.s 




https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 04:55:38 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 04:55:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e3aba.27b3b003.3b871.85b9@mx.google.com>


================

----------------
ioghiban wrote:

The matching rank check with bail-out is in `getResultNonUnitDimsAndOffsetsForRC`, which calls `hasExactlyOneCollapsedNonUnitDim`. Therefore I implemented this check as an assumption here.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 04:55:45 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 04:55:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e3ac1.70920dee.290dca.2f6e@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 05:10:34 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Wed, 08 Jul 2026 05:10:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a4e3e3a.eb165e5c.1817a5.ba43@mx.google.com>


https://github.com/flemairen6 updated https://github.com/llvm/llvm-project/pull/205990

>From f294988d80afcf50d52102adf91aa52dd2e9199e Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 12:48:24 +0900
Subject: [PATCH 1/7] [mlir][wasm] RaiseWasmMLIRPass: Add suport for comparison
 operators

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 139 ++++++
 .../wasm-comparisons-to-arith-cmp.mlir        | 421 ++++++++++++++++++
 .../RaiseWasm/wasm-eqz-to-arith-cmp.mlir      |  28 ++
 3 files changed, 588 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 83bfde7032ef8..1fb4564032421 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,104 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+template <typename SourceOp, typename TargetOp, typename AttrType,
+          typename ValType, ValType flag>
+struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto cmpRes =
+        rewriter
+            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                              AttrType::get(rewriter.getContext(), flag),
+                              adaptor.getLhs(), adaptor.getRhs())
+            .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
+template <typename SourceOp, arith::CmpFPredicate compFlag>
+using FPComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpFOp, arith::CmpFPredicateAttr,
+                           arith::CmpFPredicate, compFlag>;
+
+template <typename SourceOp, arith::CmpIPredicate compFlag>
+using IntComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpIOp, arith::CmpIPredicateAttr,
+                           arith::CmpIPredicate, compFlag>;
+
+using WasmLtSIOpConversion =
+    IntComparisonConversion<LtSIOp, arith::CmpIPredicate::slt>;
+using WasmLeSIOpConversion =
+    IntComparisonConversion<LeSIOp, arith::CmpIPredicate::sle>;
+using WasmGtSIOpConversion =
+    IntComparisonConversion<GtSIOp, arith::CmpIPredicate::sgt>;
+using WasmGeSIOpConversion =
+    IntComparisonConversion<GeSIOp, arith::CmpIPredicate::sge>;
+using WasmLtUIOpConversion =
+    IntComparisonConversion<LtUIOp, arith::CmpIPredicate::ult>;
+using WasmLeUIOpConversion =
+    IntComparisonConversion<LeUIOp, arith::CmpIPredicate::ule>;
+using WasmGtUIOpConversion =
+    IntComparisonConversion<GtUIOp, arith::CmpIPredicate::ugt>;
+using WasmGeUIOpConversion =
+    IntComparisonConversion<GeUIOp, arith::CmpIPredicate::uge>;
+using WasmLtOpConversion =
+    FPComparisonConversion<LtOp, arith::CmpFPredicate::OLT>;
+using WasmLeOpConversion =
+    FPComparisonConversion<LeOp, arith::CmpFPredicate::OLE>;
+using WasmGtOpConversion =
+    FPComparisonConversion<GtOp, arith::CmpFPredicate::OGT>;
+using WasmGeOpConversion =
+    FPComparisonConversion<GeOp, arith::CmpFPredicate::OGE>;
+
+template <typename SourceOp, arith::CmpIPredicate IntFlag,
+          arith::CmpFPredicate FloatFlag>
+struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Value comparisonResult;
+    if (srcOp.getLhs().getType().isInteger())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpIOp>(
+                  srcOp.getLoc(), rewriter.getI1Type(),
+                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                  adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else if (srcOp.getLhs().getType().isFloat())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                                     arith::CmpFPredicateAttr::get(
+                                         rewriter.getContext(), FloatFlag),
+                                     adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else
+      return rewriter.notifyMatchFailure(
+          srcOp.getLoc(), "Unsupported datatype for comparison OP.");
+
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                comparisonResult);
+    return success();
+  }
+};
+
+using WasmEqOpConversion =
+    IntFpComparisonOpConversion<EqOp, arith::CmpIPredicate::eq,
+                                arith::CmpFPredicate::OEQ>;
+using WasmNeOpConversion =
+    IntFpComparisonOpConversion<NeOp, arith::CmpIPredicate::ne,
+                                arith::CmpFPredicate::ONE>;
+
 struct WasmCallOpConversion : OpConversionPattern<FuncCallOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -148,6 +246,32 @@ struct WasmConstOpConversion : OpConversionPattern<ConstOp> {
   }
 };
 
+struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = eqzOp->getLoc();
+    auto zero =
+        rewriter
+            .create<arith::ConstantOp>(
+                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+            .getResult();
+    auto cmpRes = rewriter
+                      .create<arith::CmpIOp>(
+                          loc, rewriter.getI1Type(),
+                          arith::CmpIPredicateAttr::get(
+                              rewriter.getContext(), arith::CmpIPredicate::eq),
+                          adaptor.getInput(), zero)
+                      .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -429,21 +553,36 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivFPOpConversion,
            WasmDivSIOpConversion,
            WasmDivUIOpConversion,
+           WasmEqOpConversion,
+           WasmEqzOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
            WasmFuncImportOpConversion,
            WasmFuncOpConversion,
+           WasmGeOpConversion,
+           WasmGeSIOpConversion,
+           WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
+           WasmGtOpConversion,
+           WasmGtSIOpConversion,
+           WasmGtUIOpConversion,
+           WasmLeOpConversion,
+           WasmLeSIOpConversion,
+           WasmLeUIOpConversion,
            WasmLocalConversion,
            WasmLocalGetConversion,
            WasmLocalSetConversion,
            WasmLocalTeeConversion,
+           WasmLtOpConversion,
+           WasmLtSIOpConversion,
+           WasmLtUIOpConversion,
            WasmMaxOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
+           WasmNeOpConversion,
            WasmNegOpConversion,
            WasmOrOpConversion,
            WasmPopCntOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..bc952b47520ca
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -0,0 +1,421 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+
+module {
+  wasmssa.func nested @func_lt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+
+  wasmssa.func nested @func_le_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.lt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.le %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.gt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ge %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.lt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.le %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.gt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.ge %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.eq %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.eq %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.eq %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.eq %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+    wasmssa.func nested @func_ne_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ne %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.ne %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ne %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.ne %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_lt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..083a24c5f36a8
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
@@ -0,0 +1,28 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+module {
+  wasmssa.func @eqz_i32() -> i32 {
+    %0 = wasmssa.const 13 : i32
+    %1 = wasmssa.eqz %0 : i32 -> i32
+    wasmssa.return %1 : i32
+  }
+  wasmssa.func @eqz_i64() -> i32 {
+    %0 = wasmssa.const 13 : i64
+    %1 = wasmssa.eqz %0 : i64 -> i32
+    wasmssa.return %1 : i32
+  }
+}
+
+// CHECK-LABEL:   func.func @eqz_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @eqz_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32

>From 49944e5c6ec9e356338dfaabbd248fb969ce49b8 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:06:50 +0900
Subject: [PATCH 2/7] [mlir][wasm] RaiseWasmMLIRPass: Add rotl and extend
 support

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 87 +++++++++++++++++++
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   | 76 ++++++++++++++++
 .../RaiseWasm/wasm-rotl-to-arith.mlir         | 75 ++++++++++++++++
 .../RaiseWasm/wasm-rotr-to-arith.mlir         | 77 ++++++++++++++++
 4 files changed, 315 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 1fb4564032421..bf958aad5012b 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,72 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+/// Lower a rotate to a series of bitwise operations. Intended for us
+/// in dialects that do not natively support rotate operations.
+///
+/// Result stays in the wasm dialect. It will then subsequently be lowered to
+/// the target dialect.
+///
+/// The rotate will be lowered to a pattern like so:
+///
+/// (val LHSShiftOp (bits & (width-1))) | (val RHSShiftOp (-bits & (width-1)))
+///
+/// Where LHSShiftOp and RHSShiftOp are shift operations. Concretely,
+///
+/// rotr = (val >> (bits & (width - 1))) | (val << (-bits & (width - 1)))
+/// rotl = (val << (bits & (width - 1))) | (val >> (-bits & (width - 1)))
+///
+/// Using this variant ensures that our rotate is defined in the target dialect.
+///
+/// \p SourceOp - Rotate operation to replace.
+/// \p LHSShiftOp - Shift operation to use on the left-hand side of the OR.
+/// \p RHSShiftOp - Shift operation to use on the right-hand side of the OR.
+template <typename SourceOp, typename LHSShiftOp, typename RHSShiftOp>
+struct RotateOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    const Type ty = srcOp->getResultTypes()[0];
+    const Location loc = srcOp->getLoc();
+    const Value val = adaptor.getVal();
+    const Value bits = adaptor.getBits();
+    const unsigned width = ty.getIntOrFloatBitWidth();
+
+    // Materialize (width - 1) for use in both sides of the expression.
+    auto cstWidthMinusOne =
+        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+
+    // Form the left-hand side of the OR:
+    // (val (lhs shift op) (bits & (width - 1)))
+    auto orLHS = rewriter.create<LHSShiftOp>(
+        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+
+    // Form the right-hand side of the OR:
+    // (val (rhs shift op) (-bits & (width - 1)))
+    auto orRHS = rewriter.create<RHSShiftOp>(
+        loc, val,
+        // (-bits & (width - 1))
+        rewriter.create<AndOp>(
+            loc,
+            // 0 - bits == -bits
+            rewriter.create<SubOp>(
+                loc,
+                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
+                bits),
+            cstWidthMinusOne));
+
+    // OR together the two shifts and replace the rotate with the new
+    // expression.
+    rewriter.replaceOpWithNewOp<OrOp>(srcOp, orLHS, orRHS);
+    return success();
+  }
+};
+
+using WasmRotrOpConversion = RotateOpConversion<RotrOp, ShRUOp, ShLOp>;
+using WasmRotlOpConversion = RotateOpConversion<RotlOp, ShLOp, ShRUOp>;
+
 template <typename SourceOp, typename TargetOp, typename AttrType,
           typename ValType, ValType flag>
 struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
@@ -272,6 +338,24 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   }
 };
 
+struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(ExtendLowBitsSOp extendLowBytesSOp,
+                  ExtendLowBitsSOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
+    auto truncation = rewriter.create<arith::TruncIOp>(
+        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
+        adaptor.getInput());
+    rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
+        extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
+        truncation.getResult());
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -555,6 +639,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivUIOpConversion,
            WasmEqOpConversion,
            WasmEqzOpConversion,
+           WasmExtendLowBitsOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
@@ -591,6 +676,8 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmRemSIOpConversion,
            WasmRemUIOpConversion,
            WasmReturnOpConversion,
+           WasmRotlOpConversion,
+           WasmRotrOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
new file mode 100644
index 0000000000000..90c92001a78d6
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @func_0() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_0() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_s %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_1() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_1() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_u %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 8 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 16 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_4() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_4() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 8 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_5() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_5() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 16 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_6() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_6() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 32 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
new file mode 100644
index 0000000000000..91e929bd5ff35
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -0,0 +1,75 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotl %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotl_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val << (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i32
+
+// (val >> (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotl_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val << (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i64
+
+// (val >> (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
new file mode 100644
index 0000000000000..77e301c3221e3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotr %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotr_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val >> (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i32
+
+// (val << (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+// CHECK:         }
+wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotr_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val >> (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i64
+
+// (val << (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+// CHECK:         }
+wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}

>From 6431a7494a5edc89de997962d92b1803d6aed43f Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:24:21 +0900
Subject: [PATCH 3/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for control
 flow operations in functions

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 184 ++++++++-
 .../RaiseWasm/wasm-blocks-to-cf.mlir          | 366 ++++++++++++++++++
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 206 ++++++++++
 3 files changed, 755 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index bf958aad5012b..d9dcb756790f8 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -371,6 +371,187 @@ struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
 
 struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   using OpConversionPattern::OpConversionPattern;
+  ///
+  /// Control flow conversion needs shared state for tracking which block
+  /// corresponds to which operation at which level.
+  ///
+  /// This class handles such tracking and performs the conversion of control
+  /// flow related ops contained in a function.
+  class CFRewriterVisitor {
+  private:
+    using branch_to_dest_t =
+        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+    Value getCompResultAsI1(Value compResult,
+                            ConversionPatternRewriter &rewriter) {
+      auto testValue = rewriter.create<arith::ConstantOp>(
+          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = rewriter
+                      .create<arith::CmpIOp>(
+                          compResult.getLoc(), rewriter.getIntegerType(1),
+                          arith::CmpIPredicate::ne, compResult, testValue)
+                      .getResult();
+      return flag;
+    }
+
+    void replaceNestLevelWithBranch(BlockOp blockOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(blockOp, regionsToEntry[0],
+                                                blockOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(LoopOp loopOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(loopOp, regionsToEntry[0],
+                                                loopOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(IfOp ifOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      Block *falseDest =
+          regionsToEntry.size() == 2 ? regionsToEntry[1] : ifOp.getTarget();
+      auto flag = getCompResultAsI1(ifOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          ifOp, flag, regionsToEntry[0], ifOp.getInputs(), falseDest,
+          ifOp.getInputs());
+    }
+
+    template <typename LevelType>
+    LogicalResult
+    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+                                      llvm::ArrayRef<Block *> regionsToEntry,
+                                      ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
+      if (!cast)
+        return failure();
+      replaceNestLevelWithBranch(cast, regionsToEntry, rewriter);
+      return success();
+    }
+
+    template <typename... LevelTypes>
+    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+                                       ConversionPatternRewriter &rewriter) {
+      auto sip = rewriter.saveInsertionPoint();
+      Block *blockSuccessor = nestingOp->getSuccessor(0);
+      llvm::SmallVector<Block *, 2> regionEntries;
+      LLVM_DEBUG(llvm::dbgs()
+                     << "Starting inlining blocks for " << nestingOp << "\n";);
+      for (auto &region : nestingOp->getRegions()) {
+        if (region.empty())
+          continue;
+        regionEntries.push_back(&region.front());
+        /// Inline blocks of nested ops
+        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
+            region.getOps<WasmSSALabelLevelInterface>()};
+        for (auto nestedOp : nestedOps) {
+          LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
+          if (failed(inlineBlocks(nestedOp, rewriter)))
+            return failure();
+        }
+        rewriter.inlineRegionBefore(region, blockSuccessor);
+      }
+      LLVM_DEBUG(llvm::dbgs() << "End of region inlining\n");
+      LLVM_DEBUG(llvm::dbgs() << "Replacing initial op with branching\n");
+      rewriter.setInsertionPoint(nestingOp);
+      auto res = success(
+          (... || succeeded(replaceNestLevelWithBranchWrapper<LevelTypes>(
+                      nestingOp, regionEntries, rewriter))));
+      rewriter.restoreInsertionPoint(sip);
+      if (failed(res))
+        return emitError(nestingOp->getLoc(),
+                         "Unable to inline the operation regions.");
+      return success();
+    }
+
+    /// Take a nesting level defining op and inline it in the parent region.
+    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+                               ConversionPatternRewriter &rewriter) {
+      return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
+    }
+
+    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+      auto destIter = branchToDest.find(branchOp);
+      if (destIter == branchToDest.end())
+        return branchOp->emitError("No indexed label op for this operation: ")
+               << branchOp;
+      return destIter->second;
+    }
+
+    inline void convertBranch(BranchIfOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      auto flag = getCompResultAsI1(brOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          brOp, flag, dest, brOp.getInputs(), brOp.getElseSuccessor(),
+          ValueRange{});
+    }
+
+    inline void convertBranch(BlockReturnOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(brOp, dest, brOp.getInputs());
+    }
+
+    template <typename LevelInterfaceT>
+    inline LogicalResult
+    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+                         ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
+      if (!cast)
+        return failure();
+      auto sip = rewriter.saveInsertionPoint();
+      rewriter.setInsertionPoint(branchOp);
+      convertBranch(cast, dest, rewriter);
+      rewriter.restoreInsertionPoint(sip);
+      return success();
+    }
+
+    template <typename... BranchInterfaceT>
+    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+                                        ConversionPatternRewriter &rewriter) {
+      auto dest = getBlockFor(branchOp);
+      if (failed(dest))
+        return failure();
+      auto res =
+          success((... || succeeded(convertBranchWrapper<BranchInterfaceT>(
+                              branchOp, *dest, rewriter))));
+      if (failed(res))
+        return emitError(branchOp->getLoc(), "No known converter for op ")
+               << branchOp;
+      return res;
+    }
+
+    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+                                ConversionPatternRewriter &rewriter) {
+      return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
+                                                              rewriter);
+    }
+
+    func::FuncOp func;
+    branch_to_dest_t branchToDest;
+
+  public:
+    CFRewriterVisitor(func::FuncOp func) : func{func} {
+      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+        branchToDest.insert({branchOp, branchOp.getTarget()});
+      });
+    }
+    LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
+      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
+          func.getOps<WasmSSALabelLevelInterface>()};
+      for (auto nestingOp : nestingOps)
+        if (failed(inlineBlocks(nestingOp, rewriter)))
+          return failure();
+
+      auto res =
+          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+            if (failed(convertBranch(branchOp, rewriter)))
+              return WalkResult::interrupt();
+            return WalkResult::advance();
+          });
+      return failure(res.wasInterrupted());
+    }
+  };
 
   LogicalResult
   matchAndRewrite(FuncOp funcOp, FuncOp::Adaptor adaptor,
@@ -393,7 +574,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     rewriter.applySignatureConversion(oldEntryBlock, sC, getTypeConverter());
     rewriter.replaceOp(funcOp, newFunc);
-    return success();
+    CFRewriterVisitor cfRewriter{newFunc};
+    return cfRewriter.rewrite(rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
new file mode 100644
index 0000000000000..3b275573bcbd3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -0,0 +1,366 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK_CANONICALIZED %s
+
+// CHECK-LABEL:   func.func @i_am_a_block(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 17 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 42 : i32
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_3:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @i_am_a_block(
+// CHECK_CANONICALIZED-SAME:      %[[VAL_0:.*]]: i32) -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_1:.*]] = arith.constant 42 : i32
+// CHECK_CANONICALIZED:           %[[VAL_3:.*]] = memref.alloca() : memref<i32>
+// CHECK_CANONICALIZED:           memref.store %[[VAL_1]], %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED-NOT:       memref.store
+// CHECK_CANONICALIZED:           %[[VAL_4:.*]] = memref.load %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED:           return %[[VAL_4]] : i32
+wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.const 17 : i32
+  wasmssa.local_set %arg0 : ref to i32 to %1 : i32
+  wasmssa.block : {
+    %2 = wasmssa.const 42 : i32
+    wasmssa.local_set %arg0 : ref to i32 to %2 : i32
+    wasmssa.block_return
+  }> ^bb1
+  ^bb1:
+  %res = wasmssa.local_get %arg0 : ref to i32
+  wasmssa.return %res : i32
+}
+
+
+// CHECK-LABEL:   func.func @func_0(
+// CHECK-SAME:                      %[[ARG0:.*]]: i32) {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block : {
+      %3 = wasmssa.const 3: i32
+      wasmssa.block : {
+        %4 = wasmssa.const 4: i32
+        wasmssa.block_return
+      }> ^bb1
+    ^bb1:  // pred: ^bb0
+      %5 = wasmssa.const 5: i32
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_1() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_1() {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %3 = wasmssa.const 3: i32
+  wasmssa.block : {
+    %4 = wasmssa.const 4: i32
+    wasmssa.block_return
+  }> ^bb2
+^bb2:  // pred: ^bb1
+  %5 = wasmssa.const 5: i32
+  wasmssa.block : {
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb3
+^bb3:  // pred: ^bb2
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 14 : i32
+// CHECK:           cf.br ^bb1(%[[VAL_0]] : i32)
+// CHECK:         ^bb1(%[[VAL_1:.*]]: i32):
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.addi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
+// CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
+// CHECK:           return %[[VAL_4]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 14 : i32
+  wasmssa.block(%0) : i32 : {
+  ^bb0(%arg0: i32):
+    %2 = wasmssa.const 1 : i32
+    %3 = wasmssa.add %arg0 %2 : i32
+    wasmssa.block_return %3 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
+// CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
+// CHECK:           return %[[VAL_1]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 17 : i32
+    wasmssa.block_return %1 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+//// ============= Branch instructions etc ==========
+
+// CHECK-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+
+wasmssa.func nested @branch_if_taken() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 2 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+// CHECK_CANONICALIZED:         }
+wasmssa.func nested @branch_if_continue() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 0 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @if(
+// CHECK-SAME:                  %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.andi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.cmpi ne, %[[VAL_3]], %[[VAL_4]] : i32
+// CHECK:           cf.cond_br %[[VAL_5]], ^bb1, ^bb2
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 3 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.muli %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_10:.*]] = arith.addi %[[VAL_8]], %[[VAL_9]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_10]] : i32)
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_13:.*]] = arith.shrui %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
+// CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
+// CHECK:           return %[[VAL_14]] : i32
+wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.const 1 : i32
+  %3 = wasmssa.and %1 %2 : i32
+  "wasmssa.if"(%3)[^bb1] ({
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 3 : i32
+    %7 = wasmssa.mul %5 %6 : i32
+    %8 = wasmssa.const 1 : i32
+    %9 = wasmssa.add %7 %8 : i32
+    wasmssa.block_return %9 : i32
+  }, {
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    wasmssa.block_return %7 : i32
+  }) : (i32) -> ()
+^bb1(%4: i32):  // pred: ^bb0
+  wasmssa.return %4 : i32
+}
+
+// CHECK-LABEL:   func.func @if_else(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.andi %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.cmpi ne, %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           cf.cond_br %[[VAL_6]], ^bb1(%[[VAL_1]] : i32), ^bb2(%[[VAL_1]] : i32)
+// CHECK:         ^bb1(%[[VAL_7:.*]]: i32):
+// CHECK:           %[[VAL_8:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
+// CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
+// CHECK:           return %[[VAL_10]] : i32
+wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.local_get %arg0 : ref to i32
+  %3 = wasmssa.const 1 : i32
+  %4 = wasmssa.and %2 %3 : i32
+  "wasmssa.if"(%4, %1)[^bb1] ({
+  ^bb0(%arg1: i32):
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.add %arg1 %6 : i32
+    wasmssa.block_return %7 : i32
+  }, {
+  }) : (i32, i32) -> ()
+^bb1(%5: i32):  // pred: ^bb0
+  wasmssa.return %5 : i32
+}
+
+// CHECK-LABEL:   func.func @if_if(
+// CHECK-SAME:                     %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = math.cttz %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi ne, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           cf.cond_br %[[VAL_4]], ^bb1, ^bb4
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.shrui %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = math.cttz %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_11:.*]] = arith.cmpi ne, %[[VAL_9]], %[[VAL_10]] : i32
+// CHECK:           cf.cond_br %[[VAL_11]], ^bb2(%[[VAL_5]] : i32), ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_12:.*]]: i32):
+// CHECK:           %[[VAL_13:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_14:.*]] = arith.addi %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_14]] : i32)
+// CHECK:         ^bb3(%[[VAL_15:.*]]: i32):
+// CHECK:           cf.br ^bb5(%[[VAL_15]] : i32)
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_16:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
+// CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
+// CHECK:           return %[[VAL_17]] : i32
+wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.ctz %1 : i32
+  "wasmssa.if"(%2)[^bb1] ({
+    %4 = wasmssa.const 2 : i32
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    %8 = wasmssa.ctz %7 : i32
+    "wasmssa.if"(%8, %4)[^bb1] ({
+    ^bb0(%arg1: i32):
+      %10 = wasmssa.const 2 : i32
+      %11 = wasmssa.add %arg1 %10 : i32
+      wasmssa.block_return %11 : i32
+    }, {
+    }) : (i32, i32) -> ()
+  ^bb1(%9: i32):  // pred: ^bb0
+    wasmssa.block_return %9 : i32
+  }, {
+    %4 = wasmssa.const 1 : i32
+    wasmssa.block_return %4 : i32
+  }) : (i32) -> ()
+^bb1(%3: i32):  // pred: ^bb0
+  wasmssa.return %3 : i32
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
new file mode 100644
index 0000000000000..e49909cc9417e
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -0,0 +1,206 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
+
+module {
+  wasmssa.func nested @func_0() {
+    wasmssa.loop : {
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   module {
+// CHECK:           func.func @func_0() {
+// CHECK:             cf.br ^bb1
+// CHECK:           ^bb1:
+// CHECK:             cf.br ^bb2
+// CHECK:           ^bb2:
+// CHECK:             return
+// CHECK:           }
+
+// CHECK-CANONICAL-LABEL:  func.func @func_0() {
+// CHECK-CANONICAL:             return
+// CHECK-CANONICAL:           }
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() -> i32 {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.block_return %3 : i32
+    }> ^bb1
+  ^bb1(%1: i32):  // pred: ^bb0
+    wasmssa.return %1 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_0() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           cf.br ^bb2(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() -> i32 {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK-CANONICAL:           return %[[VAL_5]] : i32
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.branch_if %3 to level 0 else ^bb1
+    ^bb1:  // pred: ^bb0
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_7:.*]] = arith.cmpi ne, %[[VAL_5]], %[[VAL_6]] : i32
+// CHECK:           cf.cond_br %[[VAL_7]], ^bb1, ^bb2
+// CHECK:         ^bb2:
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_4]], ^bb1, ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           return
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    %1 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %2 = wasmssa.local_get %0 : ref to i32
+      %3 = wasmssa.const 1 : i32
+      %4 = wasmssa.add %2 %3 : i32
+      wasmssa.local_set %0 : ref to i32 to %4 : i32
+      wasmssa.loop : {
+        %8 = wasmssa.const 12 : i32
+        %9 = wasmssa.local_get %0 : ref to i32
+        %10 = wasmssa.gt_si %8 %9 : i32 -> i32
+        wasmssa.branch_if %10 to level 0 else ^bb1
+      ^bb1:  // pred: ^bb0
+        wasmssa.block_return %8 : i32
+      }> ^bb1
+    ^bb1(%5: i32):  // pred: ^bb0
+      %6 = wasmssa.const 10 : i32
+      %7 = wasmssa.lt_si %5 %6 : i32 -> i32
+      wasmssa.branch_if %7 to level 0 else ^bb2
+    ^bb2:  // pred: ^bb1
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_3]], %[[VAL_2]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_4:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_5:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.addi %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           memref.store %[[VAL_6]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_7:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_9:.*]] = arith.cmpi sgt, %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.extui %[[VAL_9]] : i1 to i32
+// CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_12:.*]] = arith.cmpi ne, %[[VAL_10]], %[[VAL_11]] : i32
+// CHECK:           cf.cond_br %[[VAL_12]], ^bb2, ^bb3
+// CHECK:         ^bb3:
+// CHECK:           cf.br ^bb4(%[[VAL_7]] : i32)
+// CHECK:         ^bb4(%[[VAL_13:.*]]: i32):
+// CHECK:           %[[VAL_14:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_15:.*]] = arith.cmpi slt, %[[VAL_13]], %[[VAL_14]] : i32
+// CHECK:           %[[VAL_16:.*]] = arith.extui %[[VAL_15]] : i1 to i32
+// CHECK:           %[[VAL_17:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_18:.*]] = arith.cmpi ne, %[[VAL_16]], %[[VAL_17]] : i32
+// CHECK:           cf.cond_br %[[VAL_18]], ^bb1, ^bb5
+// CHECK:         ^bb5:
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
+// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
+// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
+// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
+// CHECK-CANONICAL:         ^bb4:
+// CHECK-CANONICAL:           return

>From 2af7f4e6300c16db66abf0ad83efa9e338ca21bd Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:36:04 +0900
Subject: [PATCH 4/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for memory

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 41 +++++++++++++++++++
 .../RaiseWasm/wasm-memory-to-memref.mlir      | 14 +++++++
 2 files changed, 55 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index d9dcb756790f8..53a8900b965a3 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -677,6 +677,46 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(MemOp memOp, MemOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = memOp.getLoc();
+    auto bufferType =
+        MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
+    auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
+        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
+    auto initializerName = (memPtr.getSymName() + "::initializer").str();
+    auto memInitializer = rewriter.create<func::FuncOp>(
+        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    memInitializer->setAttr(rewriter.getStringAttr("initializer"),
+                            rewriter.getUnitAttr());
+    auto *initializerBody = memInitializer.addEntryBlock();
+    auto sip = rewriter.saveInsertionPoint();
+    rewriter.setInsertionPointToStart(initializerBody);
+    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
+        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = rewriter.create<memref::AllocOp>(
+        loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto castOp =
+        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
+    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
+    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
+                                     memRefPtr.getResult(),
+                                     ValueRange{idx.getResult()});
+    rewriter.create<func::ReturnOp>(loc);
+    rewriter.restoreInsertionPoint(sip);
+    rewriter.create<func::CallOp>(loc, memInitializer);
+    return success();
+  }
+};
+
 inline TypedAttr getInitializerAttr(Type t) {
   assert(t.isIntOrFloat() &&
          "This helper is intended to use with int and float types");
@@ -847,6 +887,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmLtSIOpConversion,
            WasmLtUIOpConversion,
            WasmMaxOpConversion,
+           WasmMemoryOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
            WasmNeOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
new file mode 100644
index 0000000000000..09b07184d97fa
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
@@ -0,0 +1,14 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+"wasmssa.memory"() <{limits = !wasmssa<limit[17:]>, sym_name = "mem_0", sym_visibility = "nested"}> : () -> ()
+
+// CHECK-LABEL:   memref.global "nested" @mem_0 : memref<1xmemref<?xi8>>
+
+// CHECK-LABEL:   func.func @"mem_0::initializer"() attributes {initializer} {
+// CHECK:           %[[VAL_0:.*]] = memref.get_global @mem_0 : memref<1xmemref<?xi8>>
+// CHECK:           %[[VAL_1:.*]] = memref.alloc() : memref<17xi8>
+// CHECK:           %[[VAL_2:.*]] = memref.cast %[[VAL_1]] : memref<17xi8> to memref<?xi8>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]]{{\[}}%[[VAL_3]]] : memref<1xmemref<?xi8>>
+// CHECK:           return
+// CHECK:         func.call @"mem_0::initializer"() : () -> ()

>From ec4763983eff03797ce2a4f1d60e296b250450a1 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 30 Apr 2026 21:00:48 +0900
Subject: [PATCH 5/7] [mlir][wasm] Deprecate rewriter.create in favor of
 <op>::create and cleanup the tests

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 123 ++++++++----------
 .../RaiseWasm/wasm-blocks-to-cf.mlir          |  18 +--
 .../wasm-comparisons-to-arith-cmp.mlir        |  64 ++++-----
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   |  14 +-
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir |   8 +-
 .../RaiseWasm/wasm-rotl-to-arith.mlir         |   4 +-
 .../RaiseWasm/wasm-rotr-to-arith.mlir         |   4 +-
 7 files changed, 112 insertions(+), 123 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 53a8900b965a3..0e6e0c05b79b6 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
+#include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinDialect.h"
 #include "mlir/IR/ValueRange.h"
@@ -159,25 +160,22 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
 
     // Materialize (width - 1) for use in both sides of the expression.
     auto cstWidthMinusOne =
-        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+        ConstOp::create(rewriter, loc, IntegerAttr::get(ty, width - 1));
 
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
-    auto orLHS = rewriter.create<LHSShiftOp>(
-        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+    auto orLHS = LHSShiftOp::create(
+        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
-    auto orRHS = rewriter.create<RHSShiftOp>(
-        loc, val,
+    auto orRHS = RHSShiftOp::create(
+        rewriter, loc, val,
         // (-bits & (width - 1))
-        rewriter.create<AndOp>(
-            loc,
+        AndOp::create(
+            rewriter, loc,
             // 0 - bits == -bits
-            rewriter.create<SubOp>(
-                loc,
-                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
-                bits),
+            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
             cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
@@ -199,8 +197,7 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
   matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
-        rewriter
-            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+        TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
                               AttrType::get(rewriter.getContext(), flag),
                               adaptor.getLhs(), adaptor.getRhs())
             .getResult();
@@ -257,19 +254,15 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          rewriter
-              .create<arith::CmpIOp>(
-                  srcOp.getLoc(), rewriter.getI1Type(),
-                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                  adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          rewriter
-              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
-                                     arith::CmpFPredicateAttr::get(
-                                         rewriter.getContext(), FloatFlag),
-                                     adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -320,13 +313,9 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
     auto zero =
-        rewriter
-            .create<arith::ConstantOp>(
-                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
             .getResult();
-    auto cmpRes = rewriter
-                      .create<arith::CmpIOp>(
-                          loc, rewriter.getI1Type(),
+    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
                           arith::CmpIPredicateAttr::get(
                               rewriter.getContext(), arith::CmpIPredicate::eq),
                           adaptor.getInput(), zero)
@@ -346,9 +335,7 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = rewriter.create<arith::TruncIOp>(
-        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
-        adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -380,15 +367,11 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   class CFRewriterVisitor {
   private:
     using branch_to_dest_t =
-        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = rewriter.create<arith::ConstantOp>(
-          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = rewriter
-                      .create<arith::CmpIOp>(
-                          compResult.getLoc(), rewriter.getIntegerType(1),
-                          arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -420,7 +403,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelType>
     LogicalResult
-    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+    replaceNestLevelWithBranchWrapper(LabelLevelOpInterface nestingOp,
                                       llvm::ArrayRef<Block *> regionsToEntry,
                                       ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
@@ -431,7 +414,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... LevelTypes>
-    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineNestDispatcher(LabelLevelOpInterface nestingOp,
                                        ConversionPatternRewriter &rewriter) {
       auto sip = rewriter.saveInsertionPoint();
       Block *blockSuccessor = nestingOp->getSuccessor(0);
@@ -443,8 +426,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
           continue;
         regionEntries.push_back(&region.front());
         /// Inline blocks of nested ops
-        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
-            region.getOps<WasmSSALabelLevelInterface>()};
+        llvm::SmallVector<LabelLevelOpInterface> nestedOps{
+            region.getOps<LabelLevelOpInterface>()};
         for (auto nestedOp : nestedOps) {
           LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
           if (failed(inlineBlocks(nestedOp, rewriter)))
@@ -466,12 +449,12 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     /// Take a nesting level defining op and inline it in the parent region.
-    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineBlocks(LabelLevelOpInterface nestingOp,
                                ConversionPatternRewriter &rewriter) {
       return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
     }
 
-    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+    llvm::FailureOr<Block *> getBlockFor(LabelBranchingOpInterface branchOp) {
       auto destIter = branchToDest.find(branchOp);
       if (destIter == branchToDest.end())
         return branchOp->emitError("No indexed label op for this operation: ")
@@ -494,7 +477,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelInterfaceT>
     inline LogicalResult
-    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+    convertBranchWrapper(LabelBranchingOpInterface branchOp, Block *dest,
                          ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
       if (!cast)
@@ -507,7 +490,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... BranchInterfaceT>
-    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranchDispatch(LabelBranchingOpInterface branchOp,
                                         ConversionPatternRewriter &rewriter) {
       auto dest = getBlockFor(branchOp);
       if (failed(dest))
@@ -521,7 +504,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
       return res;
     }
 
-    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranch(LabelBranchingOpInterface branchOp,
                                 ConversionPatternRewriter &rewriter) {
       return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
                                                               rewriter);
@@ -532,19 +515,19 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
   public:
     CFRewriterVisitor(func::FuncOp func) : func{func} {
-      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+      func.walk([this](LabelBranchingOpInterface branchOp) {
         branchToDest.insert({branchOp, branchOp.getTarget()});
       });
     }
     LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
-      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
-          func.getOps<WasmSSALabelLevelInterface>()};
+      llvm::SmallVector<LabelLevelOpInterface> nestingOps{
+          func.getOps<LabelLevelOpInterface>()};
       for (auto nestingOp : nestingOps)
         if (failed(inlineBlocks(nestingOp, rewriter)))
           return failure();
 
       auto res =
-          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+          func->walk([this, &rewriter](LabelBranchingOpInterface branchOp) {
             if (failed(convertBranch(branchOp, rewriter)))
               return WalkResult::interrupt();
             return WalkResult::advance();
@@ -687,32 +670,38 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
     auto bufferType =
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memVisibility = memOp.getVisibility();
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    mlir::StringAttr visAttr;
+    if (memVisibility == mlir::SymbolTable::Visibility::Public)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+    else if (memVisibility == mlir::SymbolTable::Visibility::Private)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+    else
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr,
         TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = rewriter.create<func::FuncOp>(
-        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
+                                               FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
-        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = rewriter.create<memref::AllocOp>(
-        loc,
-        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
-        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
-    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
-    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
-                                     memRefPtr.getResult(),
-                                     ValueRange{idx.getResult()});
-    rewriter.create<func::ReturnOp>(loc);
+        memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    memref::StoreOp::create(rewriter, loc, castOp.getResult(),
+                            memRefPtr.getResult(), ValueRange{idx.getResult()});
+    func::ReturnOp::create(rewriter, loc);
     rewriter.restoreInsertionPoint(sip);
-    rewriter.create<func::CallOp>(loc, memInitializer);
+    func::CallOp::create(rewriter, loc, memInitializer);
     return success();
   }
 };
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
index 3b275573bcbd3..fdce71cab096b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -60,7 +60,7 @@ wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+wasmssa.func @func_0(%arg0: !wasmssa<local ref to i32>) {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -104,7 +104,7 @@ wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_1() {
+wasmssa.func @func_1() {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -136,7 +136,7 @@ wasmssa.func nested @func_1() {
 // CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
 // CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
 // CHECK:           return %[[VAL_4]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 14 : i32
   wasmssa.block(%0) : i32 : {
   ^bb0(%arg0: i32):
@@ -155,7 +155,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
 // CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
 // CHECK:           return %[[VAL_1]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 17 : i32
     wasmssa.block_return %1 : i32
@@ -185,7 +185,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 
-wasmssa.func nested @branch_if_taken() -> i32 {
+wasmssa.func @branch_if_taken() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 2 : i32
@@ -218,7 +218,7 @@ wasmssa.func nested @branch_if_taken() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 // CHECK_CANONICALIZED:         }
-wasmssa.func nested @branch_if_continue() -> i32 {
+wasmssa.func @branch_if_continue() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 0 : i32
@@ -256,7 +256,7 @@ wasmssa.func nested @branch_if_continue() -> i32 {
 // CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
 // CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
 // CHECK:           return %[[VAL_14]] : i32
-wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.const 1 : i32
   %3 = wasmssa.and %1 %2 : i32
@@ -294,7 +294,7 @@ wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
 // CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
 // CHECK:           return %[[VAL_10]] : i32
-wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.local_get %arg0 : ref to i32
   %3 = wasmssa.const 1 : i32
@@ -339,7 +339,7 @@ wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
 // CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
 // CHECK:           return %[[VAL_17]] : i32
-wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.ctz %1 : i32
   "wasmssa.if"(%2)[^bb1] ({
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
index bc952b47520ca..6643b0389f405 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -2,194 +2,194 @@
 
 
 module {
-  wasmssa.func nested @func_lt_si32() -> i32 {
+  wasmssa.func @func_lt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
 
-  wasmssa.func nested @func_le_si32() -> i32 {
+  wasmssa.func @func_le_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui32() -> i32 {
+  wasmssa.func @func_lt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui32() -> i32 {
+  wasmssa.func @func_le_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si32() -> i32 {
+  wasmssa.func @func_gt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui32() -> i32 {
+  wasmssa.func @func_gt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si32() -> i32 {
+  wasmssa.func @func_ge_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui32() -> i32 {
+  wasmssa.func @func_ge_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_si64() -> i32 {
+  wasmssa.func @func_lt_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_si64() -> i32 {
+  wasmssa.func @func_le_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+  wasmssa.func @func_lt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui_i64() -> i32 {
+  wasmssa.func @func_le_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si_i64() -> i32 {
+  wasmssa.func @func_gt_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+  wasmssa.func @func_gt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si_i64() -> i32 {
+  wasmssa.func @func_ge_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+  wasmssa.func @func_ge_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f32() -> i32 {
+  wasmssa.func @func_lt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.lt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f32() -> i32 {
+  wasmssa.func @func_le_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.le %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f32() -> i32 {
+  wasmssa.func @func_gt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.gt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f32() -> i32 {
+  wasmssa.func @func_ge_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ge %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f64() -> i32 {
+  wasmssa.func @func_lt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.lt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f64() -> i32 {
+  wasmssa.func @func_le_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.le %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f64() -> i32 {
+  wasmssa.func @func_gt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.gt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f64() -> i32 {
+  wasmssa.func @func_ge_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.ge %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i32() -> i32 {
+  wasmssa.func @func_eq_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.eq %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i64() -> i32 {
+  wasmssa.func @func_eq_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.eq %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f32() -> i32 {
+  wasmssa.func @func_eq_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.eq %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f64() -> i32 {
+  wasmssa.func @func_eq_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.eq %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-    wasmssa.func nested @func_ne_i32() -> i32 {
+    wasmssa.func @func_ne_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ne %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_i64() -> i32 {
+  wasmssa.func @func_ne_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.ne %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f32() -> i32 {
+  wasmssa.func @func_ne_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ne %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f64() -> i32 {
+  wasmssa.func @func_ne_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.ne %0 %1 : f64 -> i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
index 90c92001a78d6..2b8bb651bbe5b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -4,7 +4,7 @@
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_0() -> i64 {
+wasmssa.func @func_0() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_s %0 to i64
   wasmssa.return %1 : i64
@@ -14,7 +14,7 @@ wasmssa.func nested @func_0() -> i64 {
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_1() -> i64 {
+wasmssa.func @func_1() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_u %0 to i64
   wasmssa.return %1 : i64
@@ -25,7 +25,7 @@ wasmssa.func nested @func_1() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 8 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -36,7 +36,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 16 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -47,7 +47,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_4() -> i64 {
+wasmssa.func @func_4() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 8 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -58,7 +58,7 @@ wasmssa.func nested @func_4() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_5() -> i64 {
+wasmssa.func @func_5() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 16 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -69,7 +69,7 @@ wasmssa.func nested @func_5() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_6() -> i64 {
+wasmssa.func @func_6() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 32 low bits from %0: i64
   wasmssa.return %1 : i64
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index e49909cc9417e..6ab812008a2f8 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -2,7 +2,7 @@
 // RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     wasmssa.loop : {
       wasmssa.block_return
     }> ^bb1
@@ -27,7 +27,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() -> i32 {
+  wasmssa.func @func_0() -> i32 {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -67,7 +67,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -117,7 +117,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     %1 = wasmssa.local of type i32
     wasmssa.loop : {
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
index 91e929bd5ff35..b69f88afd4d7c 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -31,7 +31,7 @@
 
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
-wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -66,7 +66,7 @@ wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // Form final result.
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
-wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
index 77e301c3221e3..f0b77271c86dd 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -32,7 +32,7 @@
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
 // CHECK:         }
-wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -68,7 +68,7 @@ wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
 // CHECK:         }
-wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 

>From a249e73f27672c4d5dfb8552cd0057da55e9e3c0 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 16:03:07 +0900
Subject: [PATCH 6/7] [mlir][wasm] Fix format

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 88 +++++++++++--------
 1 file changed, 52 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 0e6e0c05b79b6..b4b5006352c11 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -165,18 +165,21 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
     auto orLHS = LHSShiftOp::create(
-        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
+        rewriter, loc, val,
+        AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
     auto orRHS = RHSShiftOp::create(
         rewriter, loc, val,
         // (-bits & (width - 1))
-        AndOp::create(
-            rewriter, loc,
-            // 0 - bits == -bits
-            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
-            cstWidthMinusOne));
+        AndOp::create(rewriter, loc,
+                      // 0 - bits == -bits
+                      SubOp::create(rewriter, loc,
+                                    ConstOp::create(rewriter, loc,
+                                                    IntegerAttr::get(ty, 0)),
+                                    bits),
+                      cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
     // expression.
@@ -198,8 +201,8 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
         TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                              AttrType::get(rewriter.getContext(), flag),
-                              adaptor.getLhs(), adaptor.getRhs())
+                         AttrType::get(rewriter.getContext(), flag),
+                         adaptor.getLhs(), adaptor.getRhs())
             .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -254,15 +257,17 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -312,13 +317,15 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
-    auto zero =
-        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
-            .getResult();
-    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
-                          arith::CmpIPredicateAttr::get(
-                              rewriter.getContext(), arith::CmpIPredicate::eq),
-                          adaptor.getInput(), zero)
+    auto zero = arith::ConstantOp::create(
+                    rewriter, loc,
+                    rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+                    .getResult();
+    auto cmpRes = arith::CmpIOp::create(
+                      rewriter, loc, rewriter.getI1Type(),
+                      arith::CmpIPredicateAttr::get(rewriter.getContext(),
+                                                    arith::CmpIPredicate::eq),
+                      adaptor.getInput(), zero)
                       .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -335,7 +342,9 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(
+        rewriter, extendLowBytesSOp->getLoc(),
+        rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -366,12 +375,14 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   /// flow related ops contained in a function.
   class CFRewriterVisitor {
   private:
-    using branch_to_dest_t =
-        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
+    using branch_to_dest_t = llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(),
+                                                 rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(
+                      rewriter, compResult.getLoc(), rewriter.getIntegerType(1),
+                      arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -671,29 +682,34 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
     auto memVisibility = memOp.getVisibility();
-    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a
+    // string attribute.
     mlir::StringAttr visAttr;
     if (memVisibility == mlir::SymbolTable::Visibility::Public)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
     else if (memVisibility == mlir::SymbolTable::Visibility::Private)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
     else
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
 
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), visAttr,
-        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr, TypeAttr::get(bufferPtrType),
+        /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
-                                               FunctionType::get(getContext(), {}, {}));
+    auto memInitializer =
+        func::FuncOp::create(rewriter, loc, initializerName,
+                             FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(
+        rewriter, loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
         memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
     auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);

>From 528e3fc46245cf2e62904b6d60508b8aa301b5ca Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 21:26:54 +0900
Subject: [PATCH 7/7] [mlir][wasm] Update tests references

---
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 29 ++++++++-----------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index 6ab812008a2f8..cdaf249d9ccf6 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -183,24 +183,19 @@ module {
 // CHECK:         }
 
 // CHECK-CANONICAL-LABEL:   func.func @func_0() {
-// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
-// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
-// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
-// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
-// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
-// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CONSTANT_0:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_1:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_2:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[ALLOCA_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[CONSTANT_2]], %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[LOAD_0:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[ADDI_0:.*]] = arith.addi %[[LOAD_0]], %[[CONSTANT_1]] : i32
+// CHECK-CANONICAL:           memref.store %[[ADDI_0]], %[[ALLOCA_0]][] : memref<i32>
 // CHECK-CANONICAL:           cf.br ^bb1
 // CHECK-CANONICAL:         ^bb1:
-// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
-// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:           %[[LOAD_1:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CMPI_0:.*]] = arith.cmpi slt, %[[LOAD_1]], %[[CONSTANT_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[CMPI_0]], ^bb1, ^bb2
 // CHECK-CANONICAL:         ^bb2:
-// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
-// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
-// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
-// CHECK-CANONICAL:         ^bb4:
 // CHECK-CANONICAL:           return
+// CHECK-CANONICAL:         }


From llvmlistbot at llvm.org  Wed Jul  8 05:15:06 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 05:15:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e3f4a.ff037943.2e959a.dadf@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/14] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/14] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/14] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/14] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/14] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/14] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/14] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/14] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/14] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/14] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/14] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 53a57bba5c89d408c64fec9187c8c81b7593ca1c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/14] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 5048a6507c08e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f3c47aeab7f58..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst

>From e89c1d53a2027757778ea25c7e0db3cf1197381d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 7 Jul 2026 17:26:24 +0200
Subject: [PATCH 13/14] Constrain RC to either scalar OR single non-unit dim
 collapsed result

---
 .../Transforms/ElideReinterpretCast.cpp       | 179 +++++++------
 .../MemRef/elide-reinterpret-cast.mlir        | 242 ++++++++++++------
 2 files changed, 268 insertions(+), 153 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 38c1b7d9a8d35..e3a2bb6457b69 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,35 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
+static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
+                                             MemRefType resType) {
+  assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
+         "expected static shapes");
+  assert(srcType.getRank() == resType.getRank() &&
+         "expected rank-preserving reinterpret_cast");
+
+  unsigned collapsedDims = 0;
+
+  for (auto [srcSize, resSize] :
+       llvm::zip_equal(srcType.getShape(), resType.getShape())) {
+    if (srcSize == resSize)
+      continue;
+
+    // Only allow collapsing one non-unit source dim to a unit result dim.
+    if (srcSize != 1 && resSize == 1) {
+      ++collapsedDims;
+      continue;
+    }
+
+    // The sizes differ and both of them are non-unit - ATM not supported.
+    return false;
+  }
+
+  // Make sure there is only one collapsed dimension.
+  return collapsedDims == 1;
+}
+
+/// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
@@ -105,27 +133,27 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// the corresponding source indices.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Non-scalar results must have static strides
-/// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources. Returns nullopt for unsupported
+/// identity-layout sources. Scalar-shaped results may have arbitrary result
+/// strides. Non-scalar results must have static offsets, static result strides
+/// identical to the source identity strides, and exactly one non-unit source
+/// dimension collapsed to unit size. Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[?, M, 1]>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[?, N, 1]>>
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[K, M, L]>>
-///       ( identity-layout != [K, M, L] )
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -136,7 +164,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts
+  // TODO: Support rank-modifying reinterpret_casts.
   if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
@@ -151,8 +179,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  assert(resType.hasStaticShape() && "expected static shape");
-
   // Track result dimensions that produce varying indices; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -160,12 +186,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
       dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
   }
 
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For non-scalar results, verify that strides do not introduce
-  // non-contiguity that would require extra logic.
   if (!isScalarRes) {
+    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
+    // views with static metadata.
+    if (ShapedType::isDynamic(rcOffsets[0]))
+      return std::nullopt;
+
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -177,53 +211,32 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-  }
 
-  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
-
-  // A source with exactly one non-unit dimension cannot be indexed directly by
-  // multiple non-unit result dimensions.
-  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
-    return std::nullopt;
-
-  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
-  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
-  // only a single offset. That should be fixed at the op definition level.
-  assert(rcOffsets.size() == 1 && "Expecting single offset");
+    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+      return std::nullopt;
+  }
 
-  // CASE 1: Static ReinterpretCast offset
-  if (ShapedType::isStatic(rcOffsets[0])) {
-    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every source dimension).
-    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(dimsAndOffs.delinearizedOffsets &&
-           "static reinterpret_cast offset must delinearize to in-bounds "
-           "reinterpret_cast source indices");
+  // CASE 1: Dynamic ReinterpretCast offset.
+  //
+  // Dynamic offsets are supported only for scalar-shaped results, under the
+  // previous effectively-1D source restriction.
+  if (ShapedType::isDynamic(rcOffsets[0])) {
+    // With an effectively-1D source, a dynamic offset can be mapped to its
+    // unique non-unit dim. For other cases, bail out.
+    if (llvm::count_if(srcType.getShape(),
+                       [](int64_t size) { return size != 1; }) != 1)
+      return std::nullopt;
 
-    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
-    // MemRef. Such cases should probably be rejected by Op verifier.
-    // FIXME: Add run-time verification for cases like this.
-    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
-                        [&](auto it) {
-                          unsigned dim = it.index();
-                          int64_t resultSize = it.value();
-                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
-                                     resultSize <=
-                                 srcType.getDimSize(dim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
     return dimsAndOffs;
   }
 
-  // CASE 2: Dynamic ReinterpretCast offset.
-  // TODO: Support dynamic offsets into sources with multiple non-unit
-  // dimensions by delinearizing the offset into source start indices at runtime
-  // before adding loop IVs.
-
-  // With an effectively-1D source, a dynamic offset can be mapped to the unique
-  // non-unit dim. For other cases, bail out.
-  if (!srcNonUnitDim)
-    return std::nullopt;
+  // CASE 2: Static ReinterpretCast offset
+  // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+  // every source dimension).
+  dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+  assert(dimsAndOffs.delinearizedOffsets &&
+         "static reinterpret_cast offset must delinearize to in-bounds "
+         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -231,37 +244,36 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
 /// scalar load/store operations.
 ///
-///   // BEFORE (scalar copy)
-///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %strided
+/// Supported cases:
+///   1. Scalar-shaped reinterpret_cast results. Result strides are ignored;
+///      the store index is derived from the reinterpret_cast offset.
 ///
-///   // AFTER
-///   %v = memref.load %src[0, ..., 0]
-///   memref.store %v, %dst[delinearized(O)]
+///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
+///      dimensions except one collapsed-to-unit dimension. Result strides must
+///      be static and identical to the identity strides of the source, and the
+///      static offset selects the collapsed dimension.
 ///
-///   // BEFORE (effectively-1D copy)
+///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, 1], strides: [...]
+///     to offset: [OFF], sizes: [1, ..., 1], strides: [...]
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
-///   }
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (multidimensional copy with static offset)
+///   // BEFORE (one collapsed non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, K], strides: [...]
+///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     : memref<1xMxNxf32>
+///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     scf.for %j = 0 to K step 1 {
-///       %v = memref.load %src[0, %i, %j]
-///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
-///     }
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   scf.for %i = 0 to M step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[0, %i, OFF]
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
@@ -292,6 +304,19 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType dstType = cast<MemRefType>(dst.getType());
     MemRefType rcResType = cast<MemRefType>(rc.getType());
 
+    // Sanity check that the copy doesn't access strided MemRef out-of-bounds.
+    // Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
+    if (ShapedType::isStatic(rc.getStaticOffsets()[0]) &&
+        llvm::any_of(llvm::enumerate(rcResType.getShape()), [&](auto it) {
+          unsigned dim = it.index();
+          int64_t rcResultSize = it.value();
+          return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
+                 dstType.getDimSize(dim);
+        }))
+      return rewriter.notifyMatchFailure(
+          op, "copy accesses invalid accessible region");
+
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
     std::array<Value, 2> cachedIndexConstants;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index a1181b3d54a29..afa6833598c02 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -179,8 +179,59 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
+/// Offset delinearized to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [10], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[OFF]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// Offset delinearized into more dimensions: [0, 2, 1]
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [23], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C2:.*]] = arith.constant 2 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C2]], %[[C1]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
 // CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
   %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
@@ -202,6 +253,23 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
+func.func private @negative_copy_1D_into_1D_strided(
+  %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -228,12 +296,38 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+  %src : memref<1x1x11xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 11], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 11 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x1x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x1x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Offset delinearized to [0, 0, 10], therefore is only
 /// added to the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -257,36 +351,25 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   return
 }
 
-/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
-/// both the looped source dimension and the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
-    : memref<1x4x11xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
-  // CHECK:      }
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
-  // CHECK-NOT:  memref.copy
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
-/// Reject rc result strides that not equal to rc source identity strides.
+/// Reject rc result strides that are not equal to rc source identity strides.
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
 func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
@@ -325,91 +408,98 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+  return
+}
+
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
+func.func private @negative_copy_2D_into_2D_strided(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
-  // CHECK:        }
-  // CHECK:      }
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_zero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    to offset: [0], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
   // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
   // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-/// rc result dynamic offset:
-///    supported only for effectively-1D rc source
-///    (runtime delinearization not implemented)
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_nonzero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [10], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[OFF]]] : memref<3x1x4x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }

>From 439436a5b975771987f7d519d98df388e9246227 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:13:32 +0200
Subject: [PATCH 14/14] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 1 file changed, 30 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e3a2bb6457b69..0285620e22524 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,8 +84,9 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
-                                             MemRefType resType) {
+static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
          "expected static shapes");
   assert(srcType.getRank() == resType.getRank() &&
@@ -129,14 +130,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
-/// the corresponding source indices.
+/// Returns reinterpret_cast's result non-unit dimensions and, for static
+/// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Scalar-shaped results may have arbitrary result
-/// strides. Non-scalar results must have static offsets, static result strides
-/// identical to the source identity strides, and exactly one non-unit source
-/// dimension collapsed to unit size. Returns nullopt for unsupported
+/// identity-layout sources. In addition:
+///     identical to the source identity strides, and exactly one non-unit
+///     source
+///  * Non-scalar results must have static offsets, static result strides
+///     dimension collapsed to unit size
+/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
+/// strides are effectively irrelevant).
+///
+/// Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
@@ -179,7 +185,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  // Track result dimensions that produce varying indices; unit dimensions are
+  // Track non-unit result dimensions; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize != 1)
@@ -194,10 +200,13 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
+  bool isOffsetDynamic = ShapedType::isDynamic(rcOffsets[0]);
+
+  // Cases with at least one non-unit dimension in reinterpret_cast's result are
+  // restricted to preserving all but one dimension from the source, which
+  // collapsed to `1` in the result, and fully static metadata.
   if (!isScalarRes) {
-    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
-    // views with static metadata.
-    if (ShapedType::isDynamic(rcOffsets[0]))
+    if (isOffsetDynamic)
       return std::nullopt;
 
     SmallVector<int64_t> srcIdentityStrides =
@@ -212,15 +221,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+    if (!hasExactlyOneCollapsedNonUnitDim(rc))
       return std::nullopt;
   }
 
   // CASE 1: Dynamic ReinterpretCast offset.
   //
-  // Dynamic offsets are supported only for scalar-shaped results, under the
-  // previous effectively-1D source restriction.
-  if (ShapedType::isDynamic(rcOffsets[0])) {
+  // Dynamic offsets are supported only for effectively-1D to scalar
+  // reinterpret_casts.
+  if (isOffsetDynamic) {
     // With an effectively-1D source, a dynamic offset can be mapped to its
     // unique non-unit dim. For other cases, bail out.
     if (llvm::count_if(srcType.getShape(),
@@ -234,9 +243,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // Delinearize static ReinterpretCast offset as in-bounds indices (one for
   // every source dimension).
   dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-  assert(dimsAndOffs.delinearizedOffsets &&
-         "static reinterpret_cast offset must delinearize to in-bounds "
-         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -297,7 +303,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         getResultNonUnitDimsAndOffsetsForRC(rc);
     if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar or loop copy region");
+          op,
+          "unsupported reinterpret_cast result dimensions, strides, or offset");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
@@ -314,11 +321,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
                  dstType.getDimSize(dim);
         }))
-      return rewriter.notifyMatchFailure(
-          op, "copy accesses invalid accessible region");
+      return rewriter.notifyMatchFailure(op, "copy accesses are OOB");
 
-    // Reuse common index constants across bounds, steps, and static offsets,
-    // but avoid creating them for rank-0 copies.
+    // Constant Op cache to reuse common index constants across bounds, steps,
+    // and static offsets: 0 is stored at index 0 and 1 is stored at index 1.
     std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
       if (value == 0 || value == 1) {


From llvmlistbot at llvm.org  Wed Jul  8 05:45:06 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Wed, 08 Jul 2026 05:45:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e4652.faf16b37.2f0e1d.e0ee@mx.google.com>


mydatascience wrote:

yes sure I'll update the commit

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 05:47:55 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Wed, 08 Jul 2026 05:47:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e46fb.e9d484b9.15e549.d5d6@mx.google.com>


https://github.com/mydatascience updated https://github.com/llvm/llvm-project/pull/207995

>From 501bc772c8b5fe39e4ab1237b2702fad2623eaed Mon Sep 17 00:00:00 2001
From: Vladimir Suvorov <suvorovv at google.com>
Date: Tue, 7 Jul 2026 16:48:52 +0400
Subject: [PATCH] Remove op dependency from verifyBlockScaledTensorType

Signed-off-by: Vladimir Suvorov <suvorovv at google.com>
---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h        |  2 +-
 mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td |  2 +-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp               | 12 +++---------
 mlir/test/Dialect/Tosa/verifier.mlir               |  2 --
 4 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 66c3753899124..dfecda940d8c7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,7 +92,7 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
-LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+LogicalResult verifyBlockScaledTensorType(mlir::Type type);
 
 } // namespace tosa
 } // namespace OpTrait
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..63be7b69b6398 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -159,7 +159,7 @@ def AtLeastRankOne : And<[
 
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
-            "verifyBlockScaledTensorType($_op, $_self))">;
+            "verifyBlockScaledTensorType($_self))">;
 
 class TosaTensorOf<list<Type> allowedTypes,
                    string summary = "tosa-conformant tensor">
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..829c0b41a68cf 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -740,8 +740,7 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(mlir::Type type) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
@@ -752,9 +751,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
     return success();
 
   if (tensorType.getRank() == 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " does not support block scaling on scalar tensors";
+    return failure();
 
   const int64_t blockedDimension = tensorType.getShape().back();
   if (ShapedType::isDynamic(blockedDimension))
@@ -763,10 +760,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
   if (blockedDimension % blockSize != 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " blocked dimension must be a multiple of block size, got "
-           << blockedDimension << " and block size " << blockSize;
+    return failure();
 
   return success();
 }
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..478532e52c438 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1675,7 +1675,6 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
@@ -1684,7 +1683,6 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>


From llvmlistbot at llvm.org  Wed Jul  8 05:48:40 2026
From: llvmlistbot at llvm.org (Vladimir Suvorov)
Date: Wed, 08 Jul 2026 05:48:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e4728.cbc3f03a.10d998.99fb@mx.google.com>


mydatascience wrote:

updated @lhutton1 


https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 05:53:54 2026
From: llvmlistbot at llvm.org (Adrian Kuegel)
Date: Wed, 08 Jul 2026 05:53:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e4862.cbc3f03a.10d998.5afb@mx.google.com>


https://github.com/akuegel auto_merge_enabled https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 05:57:21 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Wed, 08 Jul 2026 05:57:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4e4931.cb30c3e4.3ce559.c381@mx.google.com>


Devjiu wrote:

@matthias-springer @joker-eph gentle ping to share your opinion on this 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Wed Jul  8 06:00:28 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 06:00:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e49ec.fae8f0c5.2eabe.92ee@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/15] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/15] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/15] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/15] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/15] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/15] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/15] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/15] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/15] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/15] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/15] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 53a57bba5c89d408c64fec9187c8c81b7593ca1c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/15] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 5048a6507c08e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f3c47aeab7f58..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst

>From e89c1d53a2027757778ea25c7e0db3cf1197381d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 7 Jul 2026 17:26:24 +0200
Subject: [PATCH 13/15] Constrain RC to either scalar OR single non-unit dim
 collapsed result

---
 .../Transforms/ElideReinterpretCast.cpp       | 179 +++++++------
 .../MemRef/elide-reinterpret-cast.mlir        | 242 ++++++++++++------
 2 files changed, 268 insertions(+), 153 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 38c1b7d9a8d35..e3a2bb6457b69 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,35 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
+static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
+                                             MemRefType resType) {
+  assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
+         "expected static shapes");
+  assert(srcType.getRank() == resType.getRank() &&
+         "expected rank-preserving reinterpret_cast");
+
+  unsigned collapsedDims = 0;
+
+  for (auto [srcSize, resSize] :
+       llvm::zip_equal(srcType.getShape(), resType.getShape())) {
+    if (srcSize == resSize)
+      continue;
+
+    // Only allow collapsing one non-unit source dim to a unit result dim.
+    if (srcSize != 1 && resSize == 1) {
+      ++collapsedDims;
+      continue;
+    }
+
+    // The sizes differ and both of them are non-unit - ATM not supported.
+    return false;
+  }
+
+  // Make sure there is only one collapsed dimension.
+  return collapsedDims == 1;
+}
+
+/// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
@@ -105,27 +133,27 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// the corresponding source indices.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Non-scalar results must have static strides
-/// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources. Returns nullopt for unsupported
+/// identity-layout sources. Scalar-shaped results may have arbitrary result
+/// strides. Non-scalar results must have static offsets, static result strides
+/// identical to the source identity strides, and exactly one non-unit source
+/// dimension collapsed to unit size. Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[?, M, 1]>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[?, N, 1]>>
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[K, M, L]>>
-///       ( identity-layout != [K, M, L] )
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -136,7 +164,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts
+  // TODO: Support rank-modifying reinterpret_casts.
   if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
@@ -151,8 +179,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  assert(resType.hasStaticShape() && "expected static shape");
-
   // Track result dimensions that produce varying indices; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -160,12 +186,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
       dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
   }
 
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For non-scalar results, verify that strides do not introduce
-  // non-contiguity that would require extra logic.
   if (!isScalarRes) {
+    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
+    // views with static metadata.
+    if (ShapedType::isDynamic(rcOffsets[0]))
+      return std::nullopt;
+
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -177,53 +211,32 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-  }
 
-  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
-
-  // A source with exactly one non-unit dimension cannot be indexed directly by
-  // multiple non-unit result dimensions.
-  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
-    return std::nullopt;
-
-  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
-  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
-  // only a single offset. That should be fixed at the op definition level.
-  assert(rcOffsets.size() == 1 && "Expecting single offset");
+    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+      return std::nullopt;
+  }
 
-  // CASE 1: Static ReinterpretCast offset
-  if (ShapedType::isStatic(rcOffsets[0])) {
-    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every source dimension).
-    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(dimsAndOffs.delinearizedOffsets &&
-           "static reinterpret_cast offset must delinearize to in-bounds "
-           "reinterpret_cast source indices");
+  // CASE 1: Dynamic ReinterpretCast offset.
+  //
+  // Dynamic offsets are supported only for scalar-shaped results, under the
+  // previous effectively-1D source restriction.
+  if (ShapedType::isDynamic(rcOffsets[0])) {
+    // With an effectively-1D source, a dynamic offset can be mapped to its
+    // unique non-unit dim. For other cases, bail out.
+    if (llvm::count_if(srcType.getShape(),
+                       [](int64_t size) { return size != 1; }) != 1)
+      return std::nullopt;
 
-    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
-    // MemRef. Such cases should probably be rejected by Op verifier.
-    // FIXME: Add run-time verification for cases like this.
-    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
-                        [&](auto it) {
-                          unsigned dim = it.index();
-                          int64_t resultSize = it.value();
-                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
-                                     resultSize <=
-                                 srcType.getDimSize(dim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
     return dimsAndOffs;
   }
 
-  // CASE 2: Dynamic ReinterpretCast offset.
-  // TODO: Support dynamic offsets into sources with multiple non-unit
-  // dimensions by delinearizing the offset into source start indices at runtime
-  // before adding loop IVs.
-
-  // With an effectively-1D source, a dynamic offset can be mapped to the unique
-  // non-unit dim. For other cases, bail out.
-  if (!srcNonUnitDim)
-    return std::nullopt;
+  // CASE 2: Static ReinterpretCast offset
+  // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+  // every source dimension).
+  dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+  assert(dimsAndOffs.delinearizedOffsets &&
+         "static reinterpret_cast offset must delinearize to in-bounds "
+         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -231,37 +244,36 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
 /// scalar load/store operations.
 ///
-///   // BEFORE (scalar copy)
-///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %strided
+/// Supported cases:
+///   1. Scalar-shaped reinterpret_cast results. Result strides are ignored;
+///      the store index is derived from the reinterpret_cast offset.
 ///
-///   // AFTER
-///   %v = memref.load %src[0, ..., 0]
-///   memref.store %v, %dst[delinearized(O)]
+///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
+///      dimensions except one collapsed-to-unit dimension. Result strides must
+///      be static and identical to the identity strides of the source, and the
+///      static offset selects the collapsed dimension.
 ///
-///   // BEFORE (effectively-1D copy)
+///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, 1], strides: [...]
+///     to offset: [OFF], sizes: [1, ..., 1], strides: [...]
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
-///   }
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (multidimensional copy with static offset)
+///   // BEFORE (one collapsed non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, K], strides: [...]
+///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     : memref<1xMxNxf32>
+///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     scf.for %j = 0 to K step 1 {
-///       %v = memref.load %src[0, %i, %j]
-///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
-///     }
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   scf.for %i = 0 to M step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[0, %i, OFF]
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
@@ -292,6 +304,19 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType dstType = cast<MemRefType>(dst.getType());
     MemRefType rcResType = cast<MemRefType>(rc.getType());
 
+    // Sanity check that the copy doesn't access strided MemRef out-of-bounds.
+    // Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
+    if (ShapedType::isStatic(rc.getStaticOffsets()[0]) &&
+        llvm::any_of(llvm::enumerate(rcResType.getShape()), [&](auto it) {
+          unsigned dim = it.index();
+          int64_t rcResultSize = it.value();
+          return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
+                 dstType.getDimSize(dim);
+        }))
+      return rewriter.notifyMatchFailure(
+          op, "copy accesses invalid accessible region");
+
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
     std::array<Value, 2> cachedIndexConstants;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index a1181b3d54a29..afa6833598c02 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -179,8 +179,59 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
+/// Offset delinearized to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [10], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[OFF]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// Offset delinearized into more dimensions: [0, 2, 1]
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [23], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C2:.*]] = arith.constant 2 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C2]], %[[C1]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
 // CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
   %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
@@ -202,6 +253,23 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
+func.func private @negative_copy_1D_into_1D_strided(
+  %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -228,12 +296,38 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+  %src : memref<1x1x11xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 11], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 11 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x1x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x1x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Offset delinearized to [0, 0, 10], therefore is only
 /// added to the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -257,36 +351,25 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   return
 }
 
-/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
-/// both the looped source dimension and the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
-    : memref<1x4x11xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
-  // CHECK:      }
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
-  // CHECK-NOT:  memref.copy
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
-/// Reject rc result strides that not equal to rc source identity strides.
+/// Reject rc result strides that are not equal to rc source identity strides.
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
 func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
@@ -325,91 +408,98 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+  return
+}
+
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
+func.func private @negative_copy_2D_into_2D_strided(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
-  // CHECK:        }
-  // CHECK:      }
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_zero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    to offset: [0], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
   // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
   // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-/// rc result dynamic offset:
-///    supported only for effectively-1D rc source
-///    (runtime delinearization not implemented)
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_nonzero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [10], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[OFF]]] : memref<3x1x4x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }

>From 439436a5b975771987f7d519d98df388e9246227 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:13:32 +0200
Subject: [PATCH 14/15] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 1 file changed, 30 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e3a2bb6457b69..0285620e22524 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,8 +84,9 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
-                                             MemRefType resType) {
+static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
          "expected static shapes");
   assert(srcType.getRank() == resType.getRank() &&
@@ -129,14 +130,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
-/// the corresponding source indices.
+/// Returns reinterpret_cast's result non-unit dimensions and, for static
+/// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Scalar-shaped results may have arbitrary result
-/// strides. Non-scalar results must have static offsets, static result strides
-/// identical to the source identity strides, and exactly one non-unit source
-/// dimension collapsed to unit size. Returns nullopt for unsupported
+/// identity-layout sources. In addition:
+///     identical to the source identity strides, and exactly one non-unit
+///     source
+///  * Non-scalar results must have static offsets, static result strides
+///     dimension collapsed to unit size
+/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
+/// strides are effectively irrelevant).
+///
+/// Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
@@ -179,7 +185,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  // Track result dimensions that produce varying indices; unit dimensions are
+  // Track non-unit result dimensions; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize != 1)
@@ -194,10 +200,13 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
+  bool isOffsetDynamic = ShapedType::isDynamic(rcOffsets[0]);
+
+  // Cases with at least one non-unit dimension in reinterpret_cast's result are
+  // restricted to preserving all but one dimension from the source, which
+  // collapsed to `1` in the result, and fully static metadata.
   if (!isScalarRes) {
-    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
-    // views with static metadata.
-    if (ShapedType::isDynamic(rcOffsets[0]))
+    if (isOffsetDynamic)
       return std::nullopt;
 
     SmallVector<int64_t> srcIdentityStrides =
@@ -212,15 +221,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+    if (!hasExactlyOneCollapsedNonUnitDim(rc))
       return std::nullopt;
   }
 
   // CASE 1: Dynamic ReinterpretCast offset.
   //
-  // Dynamic offsets are supported only for scalar-shaped results, under the
-  // previous effectively-1D source restriction.
-  if (ShapedType::isDynamic(rcOffsets[0])) {
+  // Dynamic offsets are supported only for effectively-1D to scalar
+  // reinterpret_casts.
+  if (isOffsetDynamic) {
     // With an effectively-1D source, a dynamic offset can be mapped to its
     // unique non-unit dim. For other cases, bail out.
     if (llvm::count_if(srcType.getShape(),
@@ -234,9 +243,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // Delinearize static ReinterpretCast offset as in-bounds indices (one for
   // every source dimension).
   dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-  assert(dimsAndOffs.delinearizedOffsets &&
-         "static reinterpret_cast offset must delinearize to in-bounds "
-         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -297,7 +303,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         getResultNonUnitDimsAndOffsetsForRC(rc);
     if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar or loop copy region");
+          op,
+          "unsupported reinterpret_cast result dimensions, strides, or offset");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
@@ -314,11 +321,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
                  dstType.getDimSize(dim);
         }))
-      return rewriter.notifyMatchFailure(
-          op, "copy accesses invalid accessible region");
+      return rewriter.notifyMatchFailure(op, "copy accesses are OOB");
 
-    // Reuse common index constants across bounds, steps, and static offsets,
-    // but avoid creating them for rank-0 copies.
+    // Constant Op cache to reuse common index constants across bounds, steps,
+    // and static offsets: 0 is stored at index 0 and 1 is stored at index 1.
     std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
       if (value == 0 || value == 1) {

>From 02afce707f3007d12829602ff84b31e1c5db3e95 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:59:34 +0200
Subject: [PATCH 15/15] Fixup

---
 mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 0285620e22524..1be4b5675dc2d 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -44,8 +44,8 @@ struct ResultNonUnitDimsAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Returns source indices for a static reinterpret_cast offset of an
-/// identity-layout source.
+/// Returns delinearized offset indices for a static reinterpret_cast offset of
+/// an identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();


From llvmlistbot at llvm.org  Wed Jul  8 06:03:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 06:03:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 5fd64bb - [mlir][tosa] Fix regression in mlir
 StableHLO tests due to PR #203583 (#207995)
Message-ID: <6a4e4aae.f1c1923d.991d1.44f2@mx.google.com>


Author: Vladimir Suvorov
Date: 2026-07-08T13:03:36Z
New Revision: 5fd64bb8f4878dcc002a5ba1bde3c23f0f2714ac

URL: https://github.com/llvm/llvm-project/commit/5fd64bb8f4878dcc002a5ba1bde3c23f0f2714ac
DIFF: https://github.com/llvm/llvm-project/commit/5fd64bb8f4878dcc002a5ba1bde3c23f0f2714ac.diff

LOG: [mlir][tosa] Fix regression in  mlir StableHLO tests due to PR #203583 (#207995)

StableHLO tests have regression due to use of additional op parameter in
verifyBlockScaledTensorType

This adjusts https://github.com/llvm/llvm-project/pull/203583 and fixes
many of stablehlo/tests

Signed-off-by: Vladimir Suvorov <suvorovv at google.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
    mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
    mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
    mlir/test/Dialect/Tosa/verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index 66c3753899124..dfecda940d8c7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,7 +92,7 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
-LogicalResult verifyBlockScaledTensorType(Operation &op, mlir::Type type);
+LogicalResult verifyBlockScaledTensorType(mlir::Type type);
 
 } // namespace tosa
 } // namespace OpTrait

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 69f836cf1b9d3..63be7b69b6398 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -159,7 +159,7 @@ def AtLeastRankOne : And<[
 
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
-            "verifyBlockScaledTensorType($_op, $_self))">;
+            "verifyBlockScaledTensorType($_self))">;
 
 class TosaTensorOf<list<Type> allowedTypes,
                    string summary = "tosa-conformant tensor">

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..829c0b41a68cf 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -740,8 +740,7 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
-                                                         mlir::Type type) {
+LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(mlir::Type type) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
@@ -752,9 +751,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
     return success();
 
   if (tensorType.getRank() == 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " does not support block scaling on scalar tensors";
+    return failure();
 
   const int64_t blockedDimension = tensorType.getShape().back();
   if (ShapedType::isDynamic(blockedDimension))
@@ -763,10 +760,7 @@ LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(Operation &op,
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
   if (blockedDimension % blockSize != 0)
-    return op.emitError()
-           << "tensor type " << type
-           << " blocked dimension must be a multiple of block size, got "
-           << blockedDimension << " and block size " << blockSize;
+    return failure();
 
   return success();
 }

diff  --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..478532e52c438 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1675,7 +1675,6 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +2 {{tensor type 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' blocked dimension must be a multiple of block size, got 31 and block size 32}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
@@ -1684,7 +1683,6 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +2 {{tensor type 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>' does not support block scaling on scalar tensors}}
   // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>


        

From llvmlistbot at llvm.org  Wed Jul  8 06:03:44 2026
From: llvmlistbot at llvm.org (Adrian Kuegel)
Date: Wed, 08 Jul 2026 06:03:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e4ab0.2020f6c2.243e78.fe93@mx.google.com>


https://github.com/akuegel closed https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 06:04:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 06:04:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a4e4ac2.af6ba3ed.36d9db.e19d@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@mydatascience Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul  8 06:05:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 06:05:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e4b1f.e55dc341.1f19bd.4f8d@mx.google.com>


================
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
----------------
jeanPerier wrote:

Mmh, I think you will need to indicate some kind of memory effect here (maybe by defining a resource for the acc runtime that is not addressable in the runtime).

Technically this operation will read memory, it will read the present table inside the acc runtime.
I do not think non speculatibility is enough here because it only tells that the operation may crash if evaluated, not that its effect may differ depending on where the operation is.

A concrete example is that if you have:

```
%b1 = acc.on_device %addr     // evaluate to false
%token = acc.declare_enter dataOperands(%addr)
%b2 = acc.on_device %addr    // evaluate to true
```

I think that MLIR CSE will merged %b1 and %b2 while they evaluate to different runtime values.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 06:31:58 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:31:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e514e.27b3b003.3b871.c203@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:31:58 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:31:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e514e.510f576d.1b3cb9.6d66@mx.google.com>


https://github.com/banach-space approved this pull request.

All in all LG, thank you!

I've left some further comments. Negative tests would benefit from more comments - it's not always obvious what the reason for failing is.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:31:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:31:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e514f.cbc3f03a.10d998.5537@mx.google.com>


================

----------------
banach-space wrote:

Non-trivial example - add a comment explaining what happens to offset = 23.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:31:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:31:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e514f.ac469dc7.3a57b1.5e48@mx.google.com>


================

----------------
banach-space wrote:

Non-trivial example - add a comment explaining what happens to offset = 10.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:31:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:31:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e514f.6c8668f2.e4f82.4ca2@mx.google.com>


================

----------------
banach-space wrote:

Add a comment - all indices are initialised to zero.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.e55dc341.1f19bd.dd5e@mx.google.com>


================

----------------
banach-space wrote:

There is only one dynamic offset.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.cb30c3e4.3ce559.a3fe@mx.google.com>


================

----------------
banach-space wrote:

How does this test differ from ` @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout`? They have almost identical names (which implies similar case), yet one is negative and the other is positive.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.c56fb102.7c9cd.4799@mx.google.com>


================

----------------
banach-space wrote:

What does "loop trailing dim" mean here?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.491d90f5.2facd.132d@mx.google.com>


================

----------------
banach-space wrote:

This is exercising the same code-path as  @negative_copy_1D_into_1D_strided - to me it's a duplication, I'd remove.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.92ea0e2a.13fd37.32fc@mx.google.com>


================

----------------
banach-space wrote:

It's always the non-unit dim that is collapsed. It's the result of collapsing that is not unit.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.8e75e0cb.5e108.08f5@mx.google.com>


================

----------------
banach-space wrote:

[nit] I would add a comment repeating that only the index corresponding to the only non-unit dim is updated

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:00 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 06:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e5150.58bdd12c.19c81.0b4b@mx.google.com>


================

----------------
banach-space wrote:

It's not obvious what goes wrong here - could you add a comment that `4` is collapsed to a non-unit dim? 

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 06:32:10 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:32:10 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4e515a.abb10101.3d0bf8.e7cd@mx.google.com>


https://github.com/Meinersbur approved this pull request.

LGTM, thnk you

https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Wed Jul  8 06:40:44 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:40:44 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4e535c.5ecc370d.5ddaa.fa74@mx.google.com>


https://github.com/Meinersbur approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Wed Jul  8 06:40:44 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:40:44 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4e535c.93108353.e0ff9.ac53@mx.google.com>


https://github.com/Meinersbur edited https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Wed Jul  8 06:40:45 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:40:45 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4e535d.88aec2a3.293ed9.d5e4@mx.google.com>


================
@@ -6273,8 +6294,53 @@ static llvm::Value *getSizeInBytes(DataLayout &dl, const mlir::Type &type,
       // size, so we do some on the fly runtime math to get the size in
       // bytes from the extent (ub - lb) * sizeInBytes. NOTE: This may need
       // some adjustment for members with more complex types.
-      return builder.CreateMul(elementCount,
-                               builder.getInt64(underlyingTypeSzInBits / 8));
+      llvm::Value *sizeCalc = builder.CreateMul(
+          elementCount, builder.getInt64(underlyingTypeSzInBits / 8),
+          "element_count");
+
+      // This is a part of a "complicated" bit of size calculation logic that is
+      // in place to handle a couple of scenarios, one specific to Fortran and
+      // the other a more general OpenMP issue. The other piece of the
+      // calculation can be found as the final size calculation within the
+      // processIndividualMap function. Ideally we would move it here, but due
+      // to the complexity of calculating the final base address of some
+      // constructs (required for a nullary check), it's left as the final step.
+      // So, in the below 2 cases, the nullary check is in processIndividualMap
+      // and the size equality check is here. The cases this modifications help
+      // cover are:
+      //
+      // 1) If an argument has a null base pointer, then the size must be set to
+      // 0
+      //    to avoid the runtime exploding/complaining about an illegal pointer
+      //    map. The size returning non-zero is feasible in certain cases if for
+      //    example someone has specified there own bounds/range.
+      // 2) We wish to support a very specific OpenMP Fortran edge-case where a
+      // size
----------------
Meinersbur wrote:

[nit] awkward line break; also maybe add an empty line betwee 1) and 2)

https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Wed Jul  8 06:40:45 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:40:45 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4e535d.eb7060da.38024.160a@mx.google.com>


================
@@ -6273,8 +6294,53 @@ static llvm::Value *getSizeInBytes(DataLayout &dl, const mlir::Type &type,
       // size, so we do some on the fly runtime math to get the size in
       // bytes from the extent (ub - lb) * sizeInBytes. NOTE: This may need
       // some adjustment for members with more complex types.
-      return builder.CreateMul(elementCount,
-                               builder.getInt64(underlyingTypeSzInBits / 8));
+      llvm::Value *sizeCalc = builder.CreateMul(
+          elementCount, builder.getInt64(underlyingTypeSzInBits / 8),
+          "element_count");
+
+      // This is a part of a "complicated" bit of size calculation logic that is
+      // in place to handle a couple of scenarios, one specific to Fortran and
+      // the other a more general OpenMP issue. The other piece of the
+      // calculation can be found as the final size calculation within the
+      // processIndividualMap function. Ideally we would move it here, but due
+      // to the complexity of calculating the final base address of some
+      // constructs (required for a nullary check), it's left as the final step.
+      // So, in the below 2 cases, the nullary check is in processIndividualMap
+      // and the size equality check is here. The cases this modifications help
+      // cover are:
+      //
+      // 1) If an argument has a null base pointer, then the size must be set to
+      // 0
----------------
Meinersbur wrote:

[nit] awkward line break

https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Wed Jul  8 06:40:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 06:40:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e535f.d0de91a4.59ea8.832c@mx.google.com>


https://github.com/jeanPerier edited https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 06:46:20 2026
From: llvmlistbot at llvm.org (Vinit Deodhar)
Date: Wed, 08 Jul 2026 06:46:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Add constant folder for
 linalg.elementwise ops (PR #203608)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203608@github.com>
Message-ID: <6a4e54ac.95816bbb.48780.7475@mx.google.com>


vinitdeodhar wrote:

@javedabsar1  can you please review the changes ?

https://github.com/llvm/llvm-project/pull/203608

From llvmlistbot at llvm.org  Wed Jul  8 06:46:30 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Wed, 08 Jul 2026 06:46:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a4e54b6.7b044958.eb235.bef0@mx.google.com>


gysit wrote:

@1sgtpepper  Looks like the build bots don't terminate for some reason. Can you try to rebase maybe this unblocks them.

https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Wed Jul  8 06:47:50 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:47:50 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4e5506.92178f4f.de0b5.6a0f@mx.google.com>


https://github.com/Meinersbur approved this pull request.


https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Wed Jul  8 06:48:26 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:48:26 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4e552a.c813de82.3a21a.897b@mx.google.com>


https://github.com/Meinersbur edited https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Wed Jul  8 06:48:56 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Wed, 08 Jul 2026 06:48:56 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a4e5548.566b9172.11e4e5.fe2b@mx.google.com>


https://github.com/Meinersbur edited https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Wed Jul  8 07:03:56 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Wed, 08 Jul 2026 07:03:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
Message-ID: <llvm.org/llvm/llvm-project/pull/208226@github.com>


https://github.com/steplong created https://github.com/llvm/llvm-project/pull/208226

Lowers mmt4d to arm sme with hoisting the load/store of the accumulator out of the K-loop. Similar to ArmSVE/pack-unpack-mmt4d.mlir with some additional changes.

Appreciate @banach-space for helping me out with this (#201562)

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/4] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/4] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/4] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/4] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 


From llvmlistbot at llvm.org  Wed Jul  8 07:04:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 07:04:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4e58fa.f06f4994.1fc857.5e9a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Stephen Long (steplong)

<details>
<summary>Changes</summary>

Lowers mmt4d to arm sme with hoisting the load/store of the accumulator out of the K-loop. Similar to ArmSVE/pack-unpack-mmt4d.mlir with some additional changes.

Appreciate @<!-- -->banach-space for helping me out with this (#<!-- -->201562)

---
Full diff: https://github.com/llvm/llvm-project/pull/208226.diff


1 Files Affected:

- (added) mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir (+354) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..745c076c265f8
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
+
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

``````````

</details>


https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Wed Jul  8 07:04:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 07:04:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4e58fb.f1c1923d.991d1.5a41@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-linalg

Author: Stephen Long (steplong)

<details>
<summary>Changes</summary>

Lowers mmt4d to arm sme with hoisting the load/store of the accumulator out of the K-loop. Similar to ArmSVE/pack-unpack-mmt4d.mlir with some additional changes.

Appreciate @<!-- -->banach-space for helping me out with this (#<!-- -->201562)

---
Full diff: https://github.com/llvm/llvm-project/pull/208226.diff


1 Files Affected:

- (added) mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir (+354) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..745c076c265f8
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
+
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

``````````

</details>


https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Wed Jul  8 07:34:24 2026
From: llvmlistbot at llvm.org (Michael Klemm)
Date: Wed, 08 Jul 2026 07:34:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a4e5ff0.4a342cbc.ca844.66ca@mx.google.com>


https://github.com/mjklemm approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Wed Jul  8 07:37:36 2026
From: llvmlistbot at llvm.org (Pradeep Kumar)
Date: Wed, 08 Jul 2026 07:37:36 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [SDAG][NVPTX] Enable custom
 legalization of v1 type for Intrinsic results and operands (PR #203237)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203237@github.com>
Message-ID: <6a4e60b0.0c05e68e.1e82f1.16b6@mx.google.com>


https://github.com/schwarzschild-radius closed https://github.com/llvm/llvm-project/pull/203237

From llvmlistbot at llvm.org  Wed Jul  8 07:37:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 07:37:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 81b2e47 - [SDAG][NVPTX] Enable custom
 legalization of v1 type for Intrinsic results and operands (#203237)
Message-ID: <6a4e60af.bb839515.27d21e.3943@mx.google.com>


Author: Pradeep Kumar
Date: 2026-07-08T20:07:28+05:30
New Revision: 81b2e47b0d29526a7bb8a7674af61acff019a5a4

URL: https://github.com/llvm/llvm-project/commit/81b2e47b0d29526a7bb8a7674af61acff019a5a4
DIFF: https://github.com/llvm/llvm-project/commit/81b2e47b0d29526a7bb8a7674af61acff019a5a4.diff

LOG: [SDAG][NVPTX] Enable custom legalization of v1 type for Intrinsic results and operands (#203237)

This commit enables custom legalization for v1 types for intrinsic results and operand by calling CustomLowerNode which gives the target a chance to custom handle the node before scalarizing

To demonstrate, I have used tcgen05.ld/st intrinsics which previously used a scalar i32 type while all wider variants used vector types

Assisted by: Claude Code

Added: 
    

Modified: 
    llvm/include/llvm/IR/IntrinsicsNVVM.td
    llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
    llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
    llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
    llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
    llvm/test/CodeGen/NVPTX/tcgen05-st.ll
    mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
    mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
    mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
    mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir

Removed: 
    


################################################################################
diff  --git a/llvm/include/llvm/IR/IntrinsicsNVVM.td b/llvm/include/llvm/IR/IntrinsicsNVVM.td
index 9d3d2ab15f9a0..40f3c40a76bc6 100644
--- a/llvm/include/llvm/IR/IntrinsicsNVVM.td
+++ b/llvm/include/llvm/IR/IntrinsicsNVVM.td
@@ -1145,7 +1145,8 @@ class SHFL_INFO<bit sync, string mode, string type, bit return_pred> {
     [OpType, llvm_i32_ty, llvm_i32_ty]);
 }
 
-class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num, string ElemType = "i32"> {
+class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num,
+                                    string ElemType = "i32"> {
   int shift = !cond(!eq(Shape, "16x128b"): 1,
                     !eq(Shape, "16x256b"): 2,
                     true : 0);
@@ -1153,15 +1154,7 @@ class NVVM_TCGEN05_LDST_ACCESS_SIZE<string Shape, int Num, string ElemType = "i3
   int veclen = !shl(1, !add(Num, shift));
 
   int valid = !le(veclen, 128);
-  LLVMType type = !cond(!eq(veclen,   1): LLVMType<!cast<ValueType>(ElemType)>,
-                        !eq(veclen,   2): LLVMType<!cast<ValueType>("v"#2#ElemType)>,
-                        !eq(veclen,   4): LLVMType<!cast<ValueType>("v"#4#ElemType)>,
-                        !eq(veclen,   8): LLVMType<!cast<ValueType>("v"#8#ElemType)>,
-                        !eq(veclen,  16): LLVMType<!cast<ValueType>("v"#16#ElemType)>,
-                        !eq(veclen,  32): LLVMType<!cast<ValueType>("v"#32#ElemType)>,
-                        !eq(veclen,  64): LLVMType<!cast<ValueType>("v"#64#ElemType)>,
-                        !eq(veclen, 128): LLVMType<!cast<ValueType>("v"#128#ElemType)>,
-                        true : llvm_void_ty);
+  LLVMType type = LLVMType<!cast<ValueType>("v"#veclen#ElemType)>;
 }
 
 class NVVM_TCGEN05_MMA_BASE<string Space, bit IsSparse, string Kind = ""> {

diff  --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index 6902ec30ab13c..3ac1ca367abc8 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -43,6 +43,10 @@ void DAGTypeLegalizer::ScalarizeVectorResult(SDNode *N, unsigned ResNo) {
              N->dump(&DAG));
   SDValue R = SDValue();
 
+  // See if the target wants to custom expand this node.
+  if (CustomLowerNode(N, N->getValueType(ResNo), true))
+    return;
+
   switch (N->getOpcode()) {
   default:
 #ifndef NDEBUG
@@ -860,6 +864,10 @@ bool DAGTypeLegalizer::ScalarizeVectorOperand(SDNode *N, unsigned OpNo) {
              N->dump(&DAG));
   SDValue Res = SDValue();
 
+  // See if the target wants to custom scalarize this node.
+  if (CustomLowerNode(N, N->getOperand(OpNo).getValueType(), false))
+    return false;
+
   switch (N->getOpcode()) {
   default:
 #ifndef NDEBUG

diff  --git a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
index fa0b9fe5f7051..bacd857b19e8b 100644
--- a/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
+++ b/llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp
@@ -1124,19 +1124,19 @@ NVPTXTargetLowering::NVPTXTargetLowering(const NVPTXTargetMachine &TM,
 
   // Custom lowering for tcgen05.ld vector operands
   setOperationAction(ISD::INTRINSIC_W_CHAIN,
-                     {MVT::v2i32, MVT::v4i32, MVT::v8i32, MVT::v16i32,
-                      MVT::v32i32, MVT::v64i32, MVT::v128i32, MVT::v2f32,
-                      MVT::v4f32, MVT::v8f32, MVT::v16f32, MVT::v32f32,
-                      MVT::v64f32, MVT::v128f32},
+                     {MVT::v1i32, MVT::v2i32, MVT::v4i32, MVT::v8i32,
+                      MVT::v16i32, MVT::v32i32, MVT::v64i32, MVT::v128i32,
+                      MVT::v2f32, MVT::v4f32, MVT::v8f32, MVT::v16f32,
+                      MVT::v32f32, MVT::v64f32, MVT::v128f32},
                      Custom);
 
   // Custom lowering for tcgen05.st vector operands and the st.async
   // i128 (.b128) operand. MVT::i8 is needed for the st.async.{sys,gpu} b8
   // variant.
   setOperationAction(ISD::INTRINSIC_VOID,
-                     {MVT::i8, MVT::v2i32, MVT::v4i32, MVT::v8i32, MVT::v16i32,
-                      MVT::v32i32, MVT::v64i32, MVT::v128i32, MVT::i128,
-                      MVT::Other},
+                     {MVT::i8, MVT::v1i32, MVT::v2i32, MVT::v4i32, MVT::v8i32,
+                      MVT::v16i32, MVT::v32i32, MVT::v64i32, MVT::v128i32,
+                      MVT::i128, MVT::Other},
                      Custom);
 
   // Enable custom lowering for the following:
@@ -2796,6 +2796,8 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_st_async_gpu:
   case Intrinsic::nvvm_st_async_mmio_sys:
     return lowerStAsyncRelease(Op, DAG);
+
+  case Intrinsic::nvvm_tcgen05_st_16x64b_x1:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x2:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x4:
   case Intrinsic::nvvm_tcgen05_st_16x64b_x8:
@@ -2815,6 +2817,7 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_tcgen05_st_16x256b_x8:
   case Intrinsic::nvvm_tcgen05_st_16x256b_x16:
   case Intrinsic::nvvm_tcgen05_st_16x256b_x32:
+  case Intrinsic::nvvm_tcgen05_st_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x2:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x4:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x8:
@@ -2824,6 +2827,7 @@ static SDValue lowerIntrinsicVoid(SDValue Op, SelectionDAG &DAG) {
   case Intrinsic::nvvm_tcgen05_st_32x32b_x64:
   case Intrinsic::nvvm_tcgen05_st_32x32b_x128:
     return lowerTcgen05St(Op, DAG);
+  case Intrinsic::nvvm_tcgen05_st_16x32bx2_x1:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x2:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x4:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x8:
@@ -5357,7 +5361,7 @@ void NVPTXTargetLowering::getTgtMemIntrinsic(
   case Intrinsic::nvvm_tcgen05_st_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_st_16x32bx2_x1: {
     Info.opc = ISD::INTRINSIC_VOID;
-    Info.memVT = MVT::i32;
+    Info.memVT = MVT::v1i32;
     Info.ptrVal = I.getArgOperand(0);
     Info.offset = 0;
     Info.flags = MachineMemOperand::MOStore;
@@ -7249,12 +7253,14 @@ static void ReplaceINTRINSIC_W_CHAIN(SDNode *N, SelectionDAG &DAG,
     return;
   }
 
+  case Intrinsic::nvvm_tcgen05_ld_16x64b_x1:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x4:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x8:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x16:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x32:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x64:
   case Intrinsic::nvvm_tcgen05_ld_16x64b_x128:
+  case Intrinsic::nvvm_tcgen05_ld_32x32b_x1:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x4:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x8:
   case Intrinsic::nvvm_tcgen05_ld_32x32b_x16:
@@ -7279,6 +7285,7 @@ static void ReplaceINTRINSIC_W_CHAIN(SDNode *N, SelectionDAG &DAG,
     }
     return;
 
+  case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x1:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x4:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x8:
   case Intrinsic::nvvm_tcgen05_ld_16x32bx2_x16:

diff  --git a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
index 3819b3143f447..5af5ae2fe9e52 100644
--- a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
+++ b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
@@ -5769,7 +5769,10 @@ class TCGEN05_LDST_REGINFO<int Veclen> {
   list<NVPTXRegClass> regs = !listsplat(B32, Veclen);
   // generate list of regnames for load/store operands
   list<string> reg_names = !foreach(x, !range(0, Veclen), "r" # x);
-  string regstring = "{{" # !interleave(!foreach(n, !range(0, Veclen), "$r" # n), ", ") # "}}";
+  string regstring = "{{" #
+                      !interleave(
+                        !foreach(n, !range(0, Veclen), "$r" # n), ", ") #
+                     "}}";
   dag Ins = !dag(ins, regs, reg_names);
   dag Outs = !dag(outs, regs, reg_names);
 }
@@ -5785,7 +5788,8 @@ class TCGEN05_LD_INST<string Shape, int Num, bit Pack> :
                                 NVVM_TCGEN05_LDST_ACCESS_SIZE<Shape, Num>.veclen>;
 
   let InOperandList = !con((ins B32:$taddr),
-                           !if(!eq(Shape, "16x32bx2"), (ins i64imm:$offset), (ins)));
+                           !if(!eq(Shape, "16x32bx2"),
+                                  (ins i64imm:$offset), (ins)));
   let OutOperandList = Info.Outs;
   let AsmString = "tcgen05.ld.sync.aligned"
                   # "." # Shape
@@ -5809,7 +5813,8 @@ class TCGEN05_ST_INST<string Shape, int Num, bit Unpack> :
                                 NVVM_TCGEN05_LDST_ACCESS_SIZE<Shape, Num>.veclen>;
 
   let InOperandList = !con((ins B32:$taddr),
-                           !if(!eq(Shape, "16x32bx2"), (ins i64imm:$offset), (ins)),
+                           !if(!eq(Shape, "16x32bx2"),
+                                  (ins i64imm:$offset), (ins)),
                            Info.Ins);
   let OutOperandList = (outs);
   let AsmString = "tcgen05.st.sync.aligned"

diff  --git a/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll b/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
index 22eb7298133bb..33eb39ac7701e 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-ld.ll
@@ -27,7 +27,7 @@ define void @nvvm_tcgen05_ld_16x64b(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) %taddr, i1 0)
 
@@ -62,7 +62,7 @@ define void @nvvm_tcgen05_ld_16x64b_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x64b.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) %taddr, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) %taddr, i1 1)
 
@@ -219,7 +219,7 @@ define void @nvvm_tcgen05_ld_32x32b(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) %taddr, i1 0)
 
@@ -253,7 +253,7 @@ define void @nvvm_tcgen05_ld_32x32b_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1];
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.32x32b.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1];
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) %taddr, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) %taddr, i1 1)
 
@@ -288,7 +288,7 @@ define void @nvvm_tcgen05_ld_16x32bx2(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x64.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1], 2;
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x128.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1], 2;
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 0)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 0)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, i1 0)
 
@@ -322,7 +322,7 @@ define void @nvvm_tcgen05_ld_16x32bx2_pack(ptr addrspace(6) %taddr) {
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x64.pack::16b.b32 {%r65, %r66, %r67, %r68, %r69, %r70, %r71, %r72, %r73, %r74, %r75, %r76, %r77, %r78, %r79, %r80, %r81, %r82, %r83, %r84, %r85, %r86, %r87, %r88, %r89, %r90, %r91, %r92, %r93, %r94, %r95, %r96, %r97, %r98, %r99, %r100, %r101, %r102, %r103, %r104, %r105, %r106, %r107, %r108, %r109, %r110, %r111, %r112, %r113, %r114, %r115, %r116, %r117, %r118, %r119, %r120, %r121, %r122, %r123, %r124, %r125, %r126, %r127, %r128}, [%r1], 2;
 ; CHECK-NEXT:    tcgen05.ld.sync.aligned.16x32bx2.x128.pack::16b.b32 {%r129, %r130, %r131, %r132, %r133, %r134, %r135, %r136, %r137, %r138, %r139, %r140, %r141, %r142, %r143, %r144, %r145, %r146, %r147, %r148, %r149, %r150, %r151, %r152, %r153, %r154, %r155, %r156, %r157, %r158, %r159, %r160, %r161, %r162, %r163, %r164, %r165, %r166, %r167, %r168, %r169, %r170, %r171, %r172, %r173, %r174, %r175, %r176, %r177, %r178, %r179, %r180, %r181, %r182, %r183, %r184, %r185, %r186, %r187, %r188, %r189, %r190, %r191, %r192, %r193, %r194, %r195, %r196, %r197, %r198, %r199, %r200, %r201, %r202, %r203, %r204, %r205, %r206, %r207, %r208, %r209, %r210, %r211, %r212, %r213, %r214, %r215, %r216, %r217, %r218, %r219, %r220, %r221, %r222, %r223, %r224, %r225, %r226, %r227, %r228, %r229, %r230, %r231, %r232, %r233, %r234, %r235, %r236, %r237, %r238, %r239, %r240, %r241, %r242, %r243, %r244, %r245, %r246, %r247, %r248, %r249, %r250, %r251, %r252, %r253, %r254, %r255, %r256}, [%r1], 2;
 ; CHECK-NEXT:    ret;
-  tail call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 1)
+  tail call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i1 1)
 
   tail call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, i1 1)
 

diff  --git a/llvm/test/CodeGen/NVPTX/tcgen05-st.ll b/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
index ccf6541d01973..952e1cf39b6fc 100644
--- a/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-st.ll
@@ -11,7 +11,7 @@
 ; RUN: %if ptxas-sm_110f && ptxas-isa-9.0 %{ llc < %s -march=nvptx64 -mcpu=sm_110f -mattr=+ptx90 | %ptxas-verify -arch=sm_110f %}
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b
-define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -92,7 +92,7 @@ define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x64b_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x64b.x128.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 0)
 
@@ -111,7 +111,7 @@ define void @nvvm_tcgen05_st_16x64b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b_unpack
-define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x64b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -192,7 +192,7 @@ define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x64b_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x64b.x128.unpack::16b.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 1)
 
@@ -211,7 +211,7 @@ define void @nvvm_tcgen05_st_16x64b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b
-define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<256>;
@@ -307,7 +307,7 @@ define void @nvvm_tcgen05_st_16x128b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i3
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b_unpack
-define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x128b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<256>;
@@ -403,7 +403,7 @@ define void @nvvm_tcgen05_st_16x128b_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b
-define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<254>;
@@ -495,7 +495,7 @@ define void @nvvm_tcgen05_st_16x256b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i3
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b_unpack
-define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x256b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<254>;
@@ -587,7 +587,7 @@ define void @nvvm_tcgen05_st_16x256b_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b
-define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -668,7 +668,7 @@ define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_32x32b_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.32x32b.x128.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 0)
 
@@ -687,7 +687,7 @@ define void @nvvm_tcgen05_st_32x32b(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b_unpack
-define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_32x32b_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -768,7 +768,7 @@ define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_32x32b_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.32x32b.x128.unpack::16b.b32 [%r1], {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) %taddr, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) %taddr, <2 x i32> %stv2, i1 1)
 
@@ -787,7 +787,7 @@ define void @nvvm_tcgen05_st_32x32b_unpack(ptr addrspace(6) %taddr, i32 %stv1, <
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2
-define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -868,7 +868,7 @@ define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x32bx2_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x32bx2.x128.b32 [%r1], 2, {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i32 %stv1, i1 0)
+  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, <1 x i32> %stv1, i1 0)
 
   tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, <2 x i32> %stv2, i1 0)
 
@@ -887,7 +887,7 @@ define void @nvvm_tcgen05_st_16x32bx2(ptr addrspace(6) %taddr, i32 %stv1, <2 x i
 }
 
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2_unpack
-define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, i32 %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
+define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, <1 x i32> %stv1, <2 x i32> %stv2, <4 x i32> %stv4, <8 x i32> %stv8, <16 x i32> %stv16, <32 x i32> %stv32, <64 x i32> %stv64, <128 x i32> %stv128) {
 ; CHECK-LABEL: nvvm_tcgen05_st_16x32bx2_unpack(
 ; CHECK:       {
 ; CHECK-NEXT:    .reg .b32 %r<257>;
@@ -968,7 +968,7 @@ define void @nvvm_tcgen05_st_16x32bx2_unpack(ptr addrspace(6) %taddr, i32 %stv1,
 ; CHECK-NEXT:    ld.param.v4.b32 {%r253, %r254, %r255, %r256}, [nvvm_tcgen05_st_16x32bx2_unpack_param_8];
 ; CHECK-NEXT:    tcgen05.st.sync.aligned.16x32bx2.x128.unpack::16b.b32 [%r1], 2, {%r253, %r254, %r255, %r256, %r249, %r250, %r251, %r252, %r245, %r246, %r247, %r248, %r241, %r242, %r243, %r244, %r237, %r238, %r239, %r240, %r233, %r234, %r235, %r236, %r229, %r230, %r231, %r232, %r225, %r226, %r227, %r228, %r221, %r222, %r223, %r224, %r217, %r218, %r219, %r220, %r213, %r214, %r215, %r216, %r209, %r210, %r211, %r212, %r205, %r206, %r207, %r208, %r201, %r202, %r203, %r204, %r197, %r198, %r199, %r200, %r193, %r194, %r195, %r196, %r189, %r190, %r191, %r192, %r185, %r186, %r187, %r188, %r181, %r182, %r183, %r184, %r177, %r178, %r179, %r180, %r173, %r174, %r175, %r176, %r169, %r170, %r171, %r172, %r165, %r166, %r167, %r168, %r161, %r162, %r163, %r164, %r157, %r158, %r159, %r160, %r153, %r154, %r155, %r156, %r149, %r150, %r151, %r152, %r145, %r146, %r147, %r148, %r141, %r142, %r143, %r144, %r137, %r138, %r139, %r140, %r133, %r134, %r135, %r136, %r129, %r130, %r131, %r132};
 ; CHECK-NEXT:    ret;
-  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, i32 %stv1, i1 1)
+  tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) %taddr, i64 2, <1 x i32> %stv1, i1 1)
 
   tail call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) %taddr, i64 2, <2 x i32> %stv2, i1 1)
 

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..89950d95e647f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5538,8 +5538,8 @@ def NVVM_Tcgen05LdOp : NVVM_Op<"tcgen05.ld", [NVVMRequiresSMf<[100, 101, 110]>]>
     Optional<I64>:$offset
   );
 
-  let results = (outs AnyTypeOf<[I32, VectorOfLengthAndType<
-                                  [2, 4, 8, 16, 32, 64, 128], [I32]>]>:$res);
+  let results = (outs VectorOfLengthAndType<
+                        [1, 2, 4, 8, 16, 32, 64, 128], [I32]>:$res);
 
   let assemblyFormat = [{
     $tmemAddr (`,` $offset^)? (`pack` $pack^)? attr-dict `:` type($res)
@@ -5598,11 +5598,9 @@ def NVVM_Tcgen05LdOp : NVVM_Op<"tcgen05.ld", [NVVMRequiresSMf<[100, 101, 110]>]>
     llvm::LLVMContext &Context = moduleTranslation.getLLVMContext();
     auto Pack = llvm::ConstantInt::get(Context, llvm::APInt(1, $pack));
 
-    unsigned num = $_resultType->isVectorTy()
-                       ? llvm::cast<llvm::VectorType>($_resultType)
+    unsigned num = llvm::cast<llvm::VectorType>($_resultType)
                              ->getElementCount()
-                             .getFixedValue()
-                       : 1;
+                             .getFixedValue();
 
     auto ID = getTcgen05LdIntrinsicID($shape, num);
     if (ID == llvm::Intrinsic::not_intrinsic)
@@ -5717,8 +5715,7 @@ def NVVM_Tcgen05StOp : NVVM_Op<"tcgen05.st", [NVVMRequiresSMf<[100, 101, 110]>]>
     Tcgen05LdStShapeAttr:$shape,
     // Arguments
     LLVM_PointerTensor:$tmemAddr,
-    AnyTypeOf<[I32, VectorOfLengthAndType<
-                      [2, 4, 8, 16, 32, 64, 128], [I32]>]>:$val,
+    VectorOfLengthAndType<[1, 2, 4, 8, 16, 32, 64, 128], [I32]>:$val,
     Optional<I64>:$offset
   );
 
@@ -5777,10 +5774,9 @@ def NVVM_Tcgen05StOp : NVVM_Op<"tcgen05.st", [NVVMRequiresSMf<[100, 101, 110]>]>
     auto Unpack = llvm::ConstantInt::get(Context, llvm::APInt(1, $unpack));
 
     auto valTy = $val->getType();
-    uint32_t num = valTy->isVectorTy() ? llvm::cast<llvm::VectorType>(valTy)
+    uint32_t num = llvm::cast<llvm::VectorType>(valTy)
                                              ->getElementCount()
-                                             .getFixedValue()
-                                       : 1;
+                                             .getFixedValue();
 
     auto ID = getTcgen05StIntrinsicID($shape, num);
     if (ID == llvm::Intrinsic::not_intrinsic)

diff  --git a/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir b/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
index ff90ad47ba410..34f7f5b87e72b 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm_check_target_sm.mlir
@@ -83,7 +83,7 @@ gpu.module @tcgen05_cp_sm90a [#nvvm.target<chip = "sm_90a">] {
 gpu.module @tcgen05_ld_sm90 [#nvvm.target<chip = "sm_90">] {
   func.func @tcgen05_ld_sm90(%taddr: !llvm.ptr<6>) {
     // expected-error @below {{'nvvm.tcgen05.ld' op is not supported on sm_90}}
-    %0 = nvvm.tcgen05.ld %taddr {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+    %0 = nvvm.tcgen05.ld %taddr {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
     return
   }
 }
@@ -91,9 +91,9 @@ gpu.module @tcgen05_ld_sm90 [#nvvm.target<chip = "sm_90">] {
 // -----
 
 gpu.module @tcgen05_st_sm120f [#nvvm.target<chip = "sm_120f">] {
-  func.func @tcgen05_st_sm120f(%taddr: !llvm.ptr<6>, %val: i32) {
+  func.func @tcgen05_st_sm120f(%taddr: !llvm.ptr<6>, %val: vector<1 x i32>) {
     // expected-error @below {{'nvvm.tcgen05.st' op is not supported on sm_120f}}
-    nvvm.tcgen05.st %taddr, %val {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+    nvvm.tcgen05.st %taddr, %val {shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
     return
   }
 }

diff  --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
index b1266b0e8151d..bcd342e2e6e94 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-ld.mlir
@@ -3,8 +3,8 @@
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b
 llvm.func @nvvm_tcgen05_ld_16x64b(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<2 x i32>
@@ -33,8 +33,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b_pack
 llvm.func @nvvm_tcgen05_ld_16x64b_pack(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>} : vector<2 x i32>
@@ -165,8 +165,8 @@ llvm.func @nvvm_tcgen05_ld_16x256b_pack(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b
 llvm.func @nvvm_tcgen05_ld_32x32b(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<2 x i32>
@@ -195,8 +195,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b(%tmemAddr : !llvm.ptr<6>) {
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b_pack
 llvm.func @nvvm_tcgen05_ld_32x32b_pack(%tmemAddr : !llvm.ptr<6>) {
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr pack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>} : vector<2 x i32>
@@ -227,8 +227,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(%tmemAddr : !llvm.ptr<6>) {
 
   %halfSplitOffset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 false)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<2 x i32>
@@ -259,8 +259,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(%tmemAddr : !llvm.ptr<6>) {
 
   %halfSplitOffset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK:  call i32 @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
-  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : i32
+// CHECK:  call <1 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
+  %ldv1 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<1 x i32>
 
 // CHECK:  call <2 x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, i1 true)
   %ldv2 = nvvm.tcgen05.ld %tmemAddr, %halfSplitOffset pack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>} : vector<2 x i32>

diff  --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
index 119746133625d..123c6308794a6 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-st.mlir
@@ -3,7 +3,7 @@
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b
 llvm.func @nvvm_tcgen05_ld_16x64b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -12,8 +12,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2 { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=2:i32 } : vector<2xi32>
@@ -42,7 +42,7 @@ llvm.func @nvvm_tcgen05_ld_16x64b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x64b_pack
 llvm.func @nvvm_tcgen05_ld_16x64b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -51,8 +51,8 @@ llvm.func @nvvm_tcgen05_ld_16x64b_pack(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x64b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x64b>, num=2:i32 } : vector<2xi32>
@@ -81,7 +81,7 @@ llvm.func @nvvm_tcgen05_ld_16x64b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x128b
 llvm.func @nvvm_tcgen05_ld_16x128b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -117,7 +117,7 @@ llvm.func @nvvm_tcgen05_ld_16x128b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x128b_pack
 llvm.func @nvvm_tcgen05_ld_16x128b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -153,7 +153,7 @@ llvm.func @nvvm_tcgen05_ld_16x128b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x256b
 llvm.func @nvvm_tcgen05_ld_16x256b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -186,7 +186,7 @@ llvm.func @nvvm_tcgen05_ld_16x256b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x256b_pack
 llvm.func @nvvm_tcgen05_ld_16x256b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -219,7 +219,7 @@ llvm.func @nvvm_tcgen05_ld_16x256b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b
 llvm.func @nvvm_tcgen05_ld_32x32b(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -228,8 +228,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2 { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=2:i32 } : vector<2xi32>
@@ -258,7 +258,7 @@ llvm.func @nvvm_tcgen05_ld_32x32b(
 // CHECK-LABEL: @nvvm_tcgen05_ld_32x32b_pack
 llvm.func @nvvm_tcgen05_ld_32x32b_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -267,8 +267,8 @@ llvm.func @nvvm_tcgen05_ld_32x32b_pack(
   %stv64    : vector<64xi32>,
   %stv128   : vector<128xi32>) {
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x1(ptr addrspace(6) {{%[0-9]+}}, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.32x32b.x2(ptr addrspace(6) {{%[0-9]+}}, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2 unpack { shape = #nvvm.tcgen05_ldst_shape<shape_32x32b>, num=2:i32 } : vector<2xi32>
@@ -297,7 +297,7 @@ llvm.func @nvvm_tcgen05_ld_32x32b_pack(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x32bx2
 llvm.func @nvvm_tcgen05_ld_16x32bx2(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -308,8 +308,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(
 
   %offset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i32 {{%[0-9]+}}, i1 false)
-  nvvm.tcgen05.st %tmemAddr, %stv1, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, <1 x i32> {{%[0-9]+}}, i1 false)
+  nvvm.tcgen05.st %tmemAddr, %stv1, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, <2 x i32> {{%[0-9]+}}, i1 false)
   nvvm.tcgen05.st %tmemAddr, %stv2, %offset { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=2:i32 } : vector<2xi32>
@@ -338,7 +338,7 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2(
 // CHECK-LABEL: @nvvm_tcgen05_ld_16x32bx2_pack
 llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(
   %tmemAddr : !llvm.ptr<6>,
-  %stv1     : i32,
+  %stv1     : vector<1 x i32>,
   %stv2     : vector<2xi32>,
   %stv4     : vector<4xi32>,
   %stv8     : vector<8xi32>,
@@ -349,8 +349,8 @@ llvm.func @nvvm_tcgen05_ld_16x32bx2_pack(
 
   %offset = llvm.mlir.constant(2:i64) : i64
 
-// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, i32 {{%[0-9]+}}, i1 true)
-  nvvm.tcgen05.st %tmemAddr, %stv1, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : i32
+// CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x1(ptr addrspace(6) {{%[0-9]+}}, i64 2, <1 x i32> {{%[0-9]+}}, i1 true)
+  nvvm.tcgen05.st %tmemAddr, %stv1, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=1:i32 } : vector<1 x i32>
 
 // CHECK: call void @llvm.nvvm.tcgen05.st.16x32bx2.x2(ptr addrspace(6) {{%[0-9]+}}, i64 2, <2 x i32> {{%[0-9]+}}, i1 true)
   nvvm.tcgen05.st %tmemAddr, %stv2, %offset unpack { shape = #nvvm.tcgen05_ldst_shape<shape_16x32bx2>, num=2:i32 } : vector<2xi32>


        

From llvmlistbot at llvm.org  Wed Jul  8 07:40:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 07:40:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4e6177.449bfb44.172ed9.0446@mx.google.com>


https://github.com/agozillon approved this pull request.

LGTM, thank you for the fixes! :-)

https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Wed Jul  8 07:41:12 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 08 Jul 2026 07:41:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add non-splat constant
 distribution in SgToLane pass (PR #205575)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205575@github.com>
Message-ID: <6a4e6188.c56fb102.7c9cd.e597@mx.google.com>


https://github.com/nbpatel updated https://github.com/llvm/llvm-project/pull/205575

>From 0b3f5844be4ec0ca4d1d0fac641c4b299536d1ed Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 22 Jun 2026 21:15:54 +0000
Subject: [PATCH 1/3] Add non-splat constant distribution for SgToLane

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 68 ++++++++++++++++---
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 20 ++++++
 2 files changed, 79 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 75a87f84b3da8..25fcd6054e569 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -355,6 +355,13 @@ struct SgToLaneElementWise : public ConversionPattern {
 
 /// Distributes a subgroup-level arith ConstantOp to lane-level arith
 /// ConstantOp.
+///
+/// Splat constants are distributed by simply rebuilding the splat with the
+/// lane-local vector type. Non-splat dense constants are distributed as :
+/// `computeDistributedCoords` yields the coordinates each lane owns, each
+/// element is extracted from the full (subgroup-level) constant, and the
+/// per-lane elements are assembled into the distributed vector with
+/// `vector.from_elements` (or `vector.broadcast` for a single element).
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -365,11 +372,11 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
     if (!resultType)
       return failure();
 
-    // Only handle dense vector constants
-    auto dense = dyn_cast<SplatElementsAttr>(op.getValue());
-    if (!dense)
+    // Only handle dense vector constants.
+    auto denseAttr = dyn_cast<DenseElementsAttr>(op.getValue());
+    if (!denseAttr)
       return rewriter.notifyMatchFailure(
-          op, "only dense splat vector constants are supported");
+          op, "only dense vector constants are supported");
 
     xegpu::DistributeLayoutAttr layout =
         xegpu::getTemporaryLayout(llvm::cast<OpResult>(op.getResult()));
@@ -385,12 +392,55 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
           op, "unable to compute lane vector type from the layout");
 
     VectorType newResultType = laneShapeOrFailure.value();
-    auto sclarValue = dense.getSplatValue<Attribute>();
-    auto newDenseAttr = DenseElementsAttr::get(newResultType, sclarValue);
+    Location loc = op.getLoc();
 
-    auto newOp = arith::ConstantOp::create(rewriter, op.getLoc(), newResultType,
-                                           newDenseAttr);
-    rewriter.replaceOp(op, newOp.getResult());
+    // Splat constants: every lane gets the same value, so just rebuild the
+    // splat with the distributed type.
+    if (denseAttr.isSplat()) {
+      auto scalarValue = denseAttr.getSplatValue<Attribute>();
+      auto newDenseAttr = DenseElementsAttr::get(newResultType, scalarValue);
+      auto newOp =
+          arith::ConstantOp::create(rewriter, loc, newResultType, newDenseAttr);
+      rewriter.replaceOp(op, newOp.getResult());
+      return success();
+    }
+
+    // Non-splat constants: each lane extracts the elements it owns from the
+    // full constant using the distributed coordinates from the layout.
+    auto fullConst =
+        arith::ConstantOp::create(rewriter, loc, resultType, denseAttr);
+
+    Value laneId = gpu::LaneIdOp::create(rewriter, loc, rewriter.getIndexType(),
+                                         /*upperBound=*/mlir::IntegerAttr());
+    auto maybeCoordsVec = layout.computeDistributedCoords(
+        rewriter, loc, laneId, resultType.getShape());
+    if (failed(maybeCoordsVec))
+      return rewriter.notifyMatchFailure(
+          op, "failed to compute distributed coordinates from layout");
+
+    SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
+    int64_t numElements = newResultType.getNumElements();
+    assert(static_cast<int64_t>(coordsVec.size()) == numElements &&
+           "number of coordinate sets must match number of distributed "
+           "elements");
+
+    SmallVector<Value> elements;
+    for (auto &coords : coordsVec) {
+      SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
+      elements.push_back(vector::ExtractOp::create(
+          rewriter, loc, fullConst.getResult(), mixedPos));
+    }
+
+    // Assemble the distributed vector.
+    Value result;
+    if (numElements == 1) {
+      result = vector::BroadcastOp::create(rewriter, loc, newResultType,
+                                           elements[0]);
+    } else {
+      result = vector::FromElementsOp::create(rewriter, loc, newResultType,
+                                              elements);
+    }
+    rewriter.replaceOp(op, result);
     return success();
   }
 };
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fcc2da3d5005b..61f4cc5ceb675 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -160,6 +160,26 @@ gpu.func @arith_constant() {
   gpu.return
 }
 
+// A non-splat constant is distributed by extracting the element each lane owns
+// from the full constant (using the layout's distributed coordinates).
+// CHECK-LABEL: gpu.func @arith_constant_non_splat
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ELEM:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<16xindex>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[ELEM]] : index to vector<1xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
+    }> : vector<16xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>

>From 61cfd91883db7075b92c45d08c66e3e1fdf28e7b Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 30 Jun 2026 14:47:29 +0000
Subject: [PATCH 2/3] Address feedback

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 42 +++++++++++++++----
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 25 ++++++++++-
 2 files changed, 57 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 25fcd6054e569..4cd242b5ddca5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -10,6 +10,7 @@
 #include "mlir/Dialect/Math/IR/Math.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/XeGPU/IR/XeGPU.h"
 #include "mlir/Dialect/XeGPU/Transforms/Passes.h"
@@ -356,12 +357,10 @@ struct SgToLaneElementWise : public ConversionPattern {
 /// Distributes a subgroup-level arith ConstantOp to lane-level arith
 /// ConstantOp.
 ///
-/// Splat constants are distributed by simply rebuilding the splat with the
-/// lane-local vector type. Non-splat dense constants are distributed as :
-/// `computeDistributedCoords` yields the coordinates each lane owns, each
-/// element is extracted from the full (subgroup-level) constant, and the
-/// per-lane elements are assembled into the distributed vector with
-/// `vector.from_elements` (or `vector.broadcast` for a single element).
+/// Splat constants are rebuilt with the lane-local vector type. Non-splat
+/// constants are distributed by extracting the elements each lane owns from
+/// the full constant and assembling them with vector.from_elements (or
+/// vector.broadcast for a single element).
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -420,12 +419,39 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
 
     SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
     int64_t numElements = newResultType.getNumElements();
-    assert(static_cast<int64_t>(coordsVec.size()) == numElements &&
+
+    // computeDistributedCoords returns the block start each lane owns. With
+    // all-ones lane_data the block is a single element. Otherwise expand each
+    // block start into its lane_data-sized element coordinates (row-major, to
+    // match vector.from_elements below).
+    SmallVector<int64_t> laneData = layout.getEffectiveLaneDataAsInt();
+    bool unitLaneData =
+        llvm::all_of(laneData, [](int64_t d) { return d == 1; });
+
+    SmallVector<SmallVector<Value>> elementCoords;
+    if (unitLaneData) {
+      elementCoords = std::move(coordsVec);
+    } else {
+      SmallVector<int64_t> unitTile(laneData.size(), 1);
+      for (const SmallVector<Value> &start : coordsVec) {
+        for (SmallVector<int64_t> off :
+             StaticTileOffsetRange(laneData, unitTile)) {
+          SmallVector<Value> coord(start.size());
+          for (size_t i = 0; i < start.size(); ++i)
+            coord[i] = arith::AddIOp::create(
+                rewriter, loc, start[i],
+                arith::ConstantIndexOp::create(rewriter, loc, off[i]));
+          elementCoords.push_back(std::move(coord));
+        }
+      }
+    }
+
+    assert(static_cast<int64_t>(elementCoords.size()) == numElements &&
            "number of coordinate sets must match number of distributed "
            "elements");
 
     SmallVector<Value> elements;
-    for (auto &coords : coordsVec) {
+    for (auto &coords : elementCoords) {
       SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
       elements.push_back(vector::ExtractOp::create(
           rewriter, loc, fullConst.getResult(), mixedPos));
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index 61f4cc5ceb675..fd16dc3cb6b4c 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -160,8 +160,8 @@ gpu.func @arith_constant() {
   gpu.return
 }
 
-// A non-splat constant is distributed by extracting the element each lane owns
-// from the full constant (using the layout's distributed coordinates).
+// Non-splat constant: each lane extracts the element it owns from the full
+// constant.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
@@ -180,6 +180,27 @@ gpu.func @arith_constant_non_splat() {
   gpu.return
 }
 
+// With lane_data > 1 each lane owns a lane_data-sized block, so multiple
+// elements are extracted and assembled with vector.from_elements.
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[RES:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_lane_data() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>
+    }> : vector<32xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>

>From 678f39ce0b51a9d996ea15f3bbd02dd459c934ec Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 6 Jul 2026 20:54:53 +0000
Subject: [PATCH 3/3] Address feedback

---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 93 ++++++++++---------
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 52 ++++++++++-
 2 files changed, 95 insertions(+), 50 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 4cd242b5ddca5..0f06f16e2f8e7 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -358,9 +358,9 @@ struct SgToLaneElementWise : public ConversionPattern {
 /// ConstantOp.
 ///
 /// Splat constants are rebuilt with the lane-local vector type. Non-splat
-/// constants are distributed by extracting the elements each lane owns from
-/// the full constant and assembling them with vector.from_elements (or
-/// vector.broadcast for a single element).
+/// constants are distributed by extracting each lane_data-sized block from
+/// the full constant and inserting it at the correct position in the
+/// distributed vector using insert_strided_slice.
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -418,54 +418,55 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
           op, "failed to compute distributed coordinates from layout");
 
     SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
-    int64_t numElements = newResultType.getNumElements();
-
-    // computeDistributedCoords returns the block start each lane owns. With
-    // all-ones lane_data the block is a single element. Otherwise expand each
-    // block start into its lane_data-sized element coordinates (row-major, to
-    // match vector.from_elements below).
     SmallVector<int64_t> laneData = layout.getEffectiveLaneDataAsInt();
-    bool unitLaneData =
-        llvm::all_of(laneData, [](int64_t d) { return d == 1; });
-
-    SmallVector<SmallVector<Value>> elementCoords;
-    if (unitLaneData) {
-      elementCoords = std::move(coordsVec);
-    } else {
-      SmallVector<int64_t> unitTile(laneData.size(), 1);
-      for (const SmallVector<Value> &start : coordsVec) {
-        for (SmallVector<int64_t> off :
-             StaticTileOffsetRange(laneData, unitTile)) {
-          SmallVector<Value> coord(start.size());
-          for (size_t i = 0; i < start.size(); ++i)
-            coord[i] = arith::AddIOp::create(
-                rewriter, loc, start[i],
-                arith::ConstantIndexOp::create(rewriter, loc, off[i]));
-          elementCoords.push_back(std::move(coord));
-        }
+    ArrayRef<int64_t> distShape = newResultType.getShape();
+    int64_t rank = newResultType.getRank();
+
+    // Each lane owns one lane_data-sized block per distribution unit.
+    // computeDistributedCoords returns those block starts in row-major order
+    // over the block grid (distShape / laneData).
+    SmallVector<int64_t> blockGridShape(rank);
+    for (int64_t d = 0; d < rank; d++)
+      blockGridShape[d] = distShape[d] / laneData[d];
+    SmallVector<int64_t> blockGridStrides = computeStrides(blockGridShape);
+
+    auto blockType = VectorType::get(laneData, newResultType.getElementType());
+    SmallVector<int64_t> unitTile(rank, 1);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value result = arith::ConstantOp::create(
+        rewriter, loc, newResultType, rewriter.getZeroAttr(newResultType));
+
+    for (auto [blockIdx, blockStart] : llvm::enumerate(coordsVec)) {
+      // Gather the block's elements from the full constant. The block start is
+      // lane-dynamic, so extract element-by-element (row-major over lane_data)
+      // instead.
+      SmallVector<Value> blockElems;
+      for (SmallVector<int64_t> off :
+           StaticTileOffsetRange(laneData, unitTile)) {
+        SmallVector<OpFoldResult> pos(rank);
+        for (int64_t d = 0; d < rank; d++)
+          pos[d] = getAsOpFoldResult(arith::AddIOp::create(
+              rewriter, loc, blockStart[d],
+              arith::ConstantIndexOp::create(rewriter, loc, off[d])));
+        blockElems.push_back(vector::ExtractOp::create(
+            rewriter, loc, fullConst.getResult(), pos));
       }
-    }
-
-    assert(static_cast<int64_t>(elementCoords.size()) == numElements &&
-           "number of coordinate sets must match number of distributed "
-           "elements");
 
-    SmallVector<Value> elements;
-    for (auto &coords : elementCoords) {
-      SmallVector<OpFoldResult> mixedPos = getAsOpFoldResult(coords);
-      elements.push_back(vector::ExtractOp::create(
-          rewriter, loc, fullConst.getResult(), mixedPos));
+      // Rebuild the block keeping its lane_data shape, then place it with
+      // insert_strided_slice so the block keeps its orientation in the
+      // distributed vector (e.g. a [2, 1] block stays a vertical 2x1 slice).
+      Value block =
+          vector::FromElementsOp::create(rewriter, loc, blockType, blockElems);
+      SmallVector<int64_t> blockGridPos =
+          delinearize(blockIdx, blockGridStrides);
+      SmallVector<int64_t> offsets(rank);
+      for (int64_t d = 0; d < rank; d++)
+        offsets[d] = blockGridPos[d] * laneData[d];
+      result = vector::InsertStridedSliceOp::create(rewriter, loc, block,
+                                                    result, offsets, strides);
     }
 
-    // Assemble the distributed vector.
-    Value result;
-    if (numElements == 1) {
-      result = vector::BroadcastOp::create(rewriter, loc, newResultType,
-                                           elements[0]);
-    } else {
-      result = vector::FromElementsOp::create(rewriter, loc, newResultType,
-                                              elements);
-    }
     rewriter.replaceOp(op, result);
     return success();
   }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fd16dc3cb6b4c..fe356e6af35c1 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -161,12 +161,14 @@ gpu.func @arith_constant() {
 }
 
 // Non-splat constant: each lane extracts the element it owns from the full
-// constant.
+// constant and inserts its lane_data-sized block into the distributed vector.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<1xindex>
 // CHECK: %[[ELEM:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<16xindex>
-// CHECK: %[[BCAST:.*]] = vector.broadcast %[[ELEM]] : index to vector<1xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM]] : vector<1xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<1xindex> into vector<1xindex>
 // CHECK: gpu.return
 gpu.func @arith_constant_non_splat() {
   %0 = arith.constant
@@ -181,13 +183,16 @@ gpu.func @arith_constant_non_splat() {
 }
 
 // With lane_data > 1 each lane owns a lane_data-sized block, so multiple
-// elements are extracted and assembled with vector.from_elements.
+// elements are extracted, reassembled into the block, and placed with
+// insert_strided_slice.
 // CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
 // CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
 // CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<2xindex>
 // CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
 // CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
-// CHECK: %[[RES:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<2xindex> into vector<2xindex>
 // CHECK: gpu.return
 gpu.func @arith_constant_non_splat_lane_data() {
   %0 = arith.constant
@@ -201,6 +206,45 @@ gpu.func @arith_constant_non_splat_lane_data() {
   gpu.return
 }
 
+// 2D non-splat constant with vertical lane_data [2, 1]. The distributed type
+// is vector<4x2xindex>. Each [2, 1] block is correctly placed vertically in
+// the result (same column, adjacent rows).
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_2d_vertical_lanedata
+// CHECK: %[[CST:.*]] = arith.constant dense<{{.*}}> : vector<4x32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<4x2xindex>
+// CHECK: %[[E0:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E1:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B0:.*]] = vector.from_elements %[[E0]], %[[E1]] : vector<2x1xindex>
+// CHECK: %[[I0:.*]] = vector.insert_strided_slice %[[B0]], %[[ZERO]] {offsets = [0, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E2:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E3:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B1:.*]] = vector.from_elements %[[E2]], %[[E3]] : vector<2x1xindex>
+// CHECK: %[[I1:.*]] = vector.insert_strided_slice %[[B1]], %[[I0]] {offsets = [0, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E4:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E5:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B2:.*]] = vector.from_elements %[[E4]], %[[E5]] : vector<2x1xindex>
+// CHECK: %[[I2:.*]] = vector.insert_strided_slice %[[B2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E6:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E7:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B3:.*]] = vector.from_elements %[[E6]], %[[E7]] : vector<2x1xindex>
+// CHECK: %[[I3:.*]] = vector.insert_strided_slice %[[B3]], %[[I2]] {offsets = [2, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_2d_vertical_lanedata() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
+    dense<[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
+           [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63],
+           [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95],
+           [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127]]> : vector<4x32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>
+    }> : vector<4x32xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>


From llvmlistbot at llvm.org  Wed Jul  8 07:47:14 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 07:47:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e62f2.e55dc341.1f19bd.ec81@mx.google.com>


================

----------------
ioghiban wrote:

Added a comment.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 07:50:45 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 07:50:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e63c5.7d41f3a1.237e53.0e0a@mx.google.com>


================
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
----------------
ivanradanov wrote:

Yes this is a _very_ good point, thank you. I was only thinking about when we are already in an IsolatedFromAbove region, but that's not the case at the start of the pipeline - cse will gladly merge these. I think I will just remove the memory interface.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 07:51:14 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 07:51:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e63e2.0c05e68e.1e82f1.d550@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 5f862cac4e31c6a1481525fbf56f1abedfda06fe Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/16] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 455 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 577 insertions(+), 140 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1c5e07f89b338..c2ead38d644da 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -38,6 +38,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefUtils
   MLIRNVGPUDialect
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 41dad1384da75..47aac6b64d092 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
 
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
 
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
 
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
 
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
 
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+  }
+
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
+
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
 
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -466,7 +680,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -474,7 +688,8 @@ struct ElideReinterpretCastPass
         return true;
       return !isPureRankExpansionOrCollapsingRC(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -485,6 +700,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 61b6d480ce7a0..90431fb507c1e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From d355a9b2dcfedb56f62a12eadf20de1d9c8b79a9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/16] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47aac6b64d092..e510ef9057b3f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -680,7 +706,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 90431fb507c1e..044e12cec42af 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 7399855abf4eaf455ae06282947c2345d41d5a41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/16] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e510ef9057b3f..95d7b1f1dc286 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From ec9d6848bd0b349da2436011bf7c9c6b1e15478f Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/16] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From bb787ee565ccd81cdd5e29a086e2b26f80665431 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/16] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 247 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 95d7b1f1dc286..6d806b087bdb4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
@@ -548,7 +539,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
   if (!inputNonUnitDim || !outputNonUnitDim)
     return false;
 
-  // The source and result must either both have a single non-unit dimension
+  // The base and result must either both have a single non-unit dimension
   // or both be all-ones.
   if (inputNonUnitDim->allOnes != outputNonUnitDim->allOnes)
     return false;
@@ -592,7 +583,7 @@ static bool isPureRankExpansionOrCollapsingRC(memref::ReinterpretCastOp rc) {
 }
 
 /// Rewrites `memref.load` through a pure rank-only `reinterpret_cast` by
-/// mapping the load indices directly onto the source MemRef.
+/// mapping the load indices directly onto the base MemRef.
 
 /// Shape restriction gated by isPureRankExpansionOrCollapsingRC().
 ///
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 044e12cec42af..34678871e8cfe 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From c11bb46b611739656cf1229dfc5191b416a53a96 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/16] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d806b087bdb4..435600f58f7fa 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
 
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
-
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -709,7 +705,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 34678871e8cfe..4a879a916c625 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From 8c90ec715552457fc2676af41000a7b008719197 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/16] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 4a879a916c625..eaa823e4afd3d 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 96a6bb69f529416e1bda4e1bd4461933b6e311ea Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/16] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 435600f58f7fa..c79e522661350 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index eaa823e4afd3d..f0b449d59a1ae 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From d0c18498d7a5748cf17d8ac19694b92b77290c88 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/16] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index c79e522661350..57a10b0f24ce9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f0b449d59a1ae..908cfdcfbcc7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c4437cd9d1dde7e1bb8447e075ca7b53edcfa6d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/16] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 57a10b0f24ce9..483d5c94ea97e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
+///
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -717,7 +737,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 908cfdcfbcc7f..be0fca979f764 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From 4c67fa2f9ad493d9da385e1fc48577f0e97a50d1 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/16] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 483d5c94ea97e..5048a6507c08e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index be0fca979f764..f3c47aeab7f58 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 53a57bba5c89d408c64fec9187c8c81b7593ca1c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/16] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 5048a6507c08e..38c1b7d9a8d35 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index f3c47aeab7f58..a1181b3d54a29 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst

>From e89c1d53a2027757778ea25c7e0db3cf1197381d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 7 Jul 2026 17:26:24 +0200
Subject: [PATCH 13/16] Constrain RC to either scalar OR single non-unit dim
 collapsed result

---
 .../Transforms/ElideReinterpretCast.cpp       | 179 +++++++------
 .../MemRef/elide-reinterpret-cast.mlir        | 242 ++++++++++++------
 2 files changed, 268 insertions(+), 153 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 38c1b7d9a8d35..e3a2bb6457b69 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,35 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
+static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
+                                             MemRefType resType) {
+  assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
+         "expected static shapes");
+  assert(srcType.getRank() == resType.getRank() &&
+         "expected rank-preserving reinterpret_cast");
+
+  unsigned collapsedDims = 0;
+
+  for (auto [srcSize, resSize] :
+       llvm::zip_equal(srcType.getShape(), resType.getShape())) {
+    if (srcSize == resSize)
+      continue;
+
+    // Only allow collapsing one non-unit source dim to a unit result dim.
+    if (srcSize != 1 && resSize == 1) {
+      ++collapsedDims;
+      continue;
+    }
+
+    // The sizes differ and both of them are non-unit - ATM not supported.
+    return false;
+  }
+
+  // Make sure there is only one collapsed dimension.
+  return collapsedDims == 1;
+}
+
+/// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
@@ -105,27 +133,27 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// the corresponding source indices.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Non-scalar results must have static strides
-/// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources. Returns nullopt for unsupported
+/// identity-layout sources. Scalar-shaped results may have arbitrary result
+/// strides. Non-scalar results must have static offsets, static result strides
+/// identical to the source identity strides, and exactly one non-unit source
+/// dimension collapsed to unit size. Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[?, M, 1]>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[?, N, 1]>>
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[K, M, L]>>
-///       ( identity-layout != [K, M, L] )
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -136,7 +164,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts
+  // TODO: Support rank-modifying reinterpret_casts.
   if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
@@ -151,8 +179,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  assert(resType.hasStaticShape() && "expected static shape");
-
   // Track result dimensions that produce varying indices; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -160,12 +186,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
       dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
   }
 
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For non-scalar results, verify that strides do not introduce
-  // non-contiguity that would require extra logic.
   if (!isScalarRes) {
+    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
+    // views with static metadata.
+    if (ShapedType::isDynamic(rcOffsets[0]))
+      return std::nullopt;
+
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -177,53 +211,32 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-  }
 
-  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
-
-  // A source with exactly one non-unit dimension cannot be indexed directly by
-  // multiple non-unit result dimensions.
-  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
-    return std::nullopt;
-
-  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
-  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
-  // only a single offset. That should be fixed at the op definition level.
-  assert(rcOffsets.size() == 1 && "Expecting single offset");
+    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+      return std::nullopt;
+  }
 
-  // CASE 1: Static ReinterpretCast offset
-  if (ShapedType::isStatic(rcOffsets[0])) {
-    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every source dimension).
-    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(dimsAndOffs.delinearizedOffsets &&
-           "static reinterpret_cast offset must delinearize to in-bounds "
-           "reinterpret_cast source indices");
+  // CASE 1: Dynamic ReinterpretCast offset.
+  //
+  // Dynamic offsets are supported only for scalar-shaped results, under the
+  // previous effectively-1D source restriction.
+  if (ShapedType::isDynamic(rcOffsets[0])) {
+    // With an effectively-1D source, a dynamic offset can be mapped to its
+    // unique non-unit dim. For other cases, bail out.
+    if (llvm::count_if(srcType.getShape(),
+                       [](int64_t size) { return size != 1; }) != 1)
+      return std::nullopt;
 
-    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
-    // MemRef. Such cases should probably be rejected by Op verifier.
-    // FIXME: Add run-time verification for cases like this.
-    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
-                        [&](auto it) {
-                          unsigned dim = it.index();
-                          int64_t resultSize = it.value();
-                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
-                                     resultSize <=
-                                 srcType.getDimSize(dim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
     return dimsAndOffs;
   }
 
-  // CASE 2: Dynamic ReinterpretCast offset.
-  // TODO: Support dynamic offsets into sources with multiple non-unit
-  // dimensions by delinearizing the offset into source start indices at runtime
-  // before adding loop IVs.
-
-  // With an effectively-1D source, a dynamic offset can be mapped to the unique
-  // non-unit dim. For other cases, bail out.
-  if (!srcNonUnitDim)
-    return std::nullopt;
+  // CASE 2: Static ReinterpretCast offset
+  // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+  // every source dimension).
+  dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+  assert(dimsAndOffs.delinearizedOffsets &&
+         "static reinterpret_cast offset must delinearize to in-bounds "
+         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -231,37 +244,36 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
 /// scalar load/store operations.
 ///
-///   // BEFORE (scalar copy)
-///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %strided
+/// Supported cases:
+///   1. Scalar-shaped reinterpret_cast results. Result strides are ignored;
+///      the store index is derived from the reinterpret_cast offset.
 ///
-///   // AFTER
-///   %v = memref.load %src[0, ..., 0]
-///   memref.store %v, %dst[delinearized(O)]
+///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
+///      dimensions except one collapsed-to-unit dimension. Result strides must
+///      be static and identical to the identity strides of the source, and the
+///      static offset selects the collapsed dimension.
 ///
-///   // BEFORE (effectively-1D copy)
+///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, 1], strides: [...]
+///     to offset: [OFF], sizes: [1, ..., 1], strides: [...]
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
-///   }
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (multidimensional copy with static offset)
+///   // BEFORE (one collapsed non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, K], strides: [...]
+///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     : memref<1xMxNxf32>
+///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     scf.for %j = 0 to K step 1 {
-///       %v = memref.load %src[0, %i, %j]
-///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
-///     }
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   scf.for %i = 0 to M step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[0, %i, OFF]
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
@@ -292,6 +304,19 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType dstType = cast<MemRefType>(dst.getType());
     MemRefType rcResType = cast<MemRefType>(rc.getType());
 
+    // Sanity check that the copy doesn't access strided MemRef out-of-bounds.
+    // Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
+    if (ShapedType::isStatic(rc.getStaticOffsets()[0]) &&
+        llvm::any_of(llvm::enumerate(rcResType.getShape()), [&](auto it) {
+          unsigned dim = it.index();
+          int64_t rcResultSize = it.value();
+          return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
+                 dstType.getDimSize(dim);
+        }))
+      return rewriter.notifyMatchFailure(
+          op, "copy accesses invalid accessible region");
+
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
     std::array<Value, 2> cachedIndexConstants;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index a1181b3d54a29..afa6833598c02 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -179,8 +179,59 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
+/// Offset delinearized to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [10], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[OFF]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// Offset delinearized into more dimensions: [0, 2, 1]
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [23], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C2:.*]] = arith.constant 2 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C2]], %[[C1]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
 // CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
   %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
@@ -202,6 +253,23 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
+func.func private @negative_copy_1D_into_1D_strided(
+  %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -228,12 +296,38 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+  %src : memref<1x1x11xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 11], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 11 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x1x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x1x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Offset delinearized to [0, 0, 10], therefore is only
 /// added to the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -257,36 +351,25 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   return
 }
 
-/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
-/// both the looped source dimension and the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
-    : memref<1x4x11xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
-  // CHECK:      }
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
-  // CHECK-NOT:  memref.copy
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
-/// Reject rc result strides that not equal to rc source identity strides.
+/// Reject rc result strides that are not equal to rc source identity strides.
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
 func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
@@ -325,91 +408,98 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+  return
+}
+
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
+func.func private @negative_copy_2D_into_2D_strided(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
-  // CHECK:        }
-  // CHECK:      }
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_zero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    to offset: [0], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
   // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
   // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-/// rc result dynamic offset:
-///    supported only for effectively-1D rc source
-///    (runtime delinearization not implemented)
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_nonzero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [10], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[OFF]]] : memref<3x1x4x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }

>From 439436a5b975771987f7d519d98df388e9246227 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:13:32 +0200
Subject: [PATCH 14/16] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 1 file changed, 30 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index e3a2bb6457b69..0285620e22524 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,8 +84,9 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
-                                             MemRefType resType) {
+static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
          "expected static shapes");
   assert(srcType.getRank() == resType.getRank() &&
@@ -129,14 +130,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
-/// the corresponding source indices.
+/// Returns reinterpret_cast's result non-unit dimensions and, for static
+/// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Scalar-shaped results may have arbitrary result
-/// strides. Non-scalar results must have static offsets, static result strides
-/// identical to the source identity strides, and exactly one non-unit source
-/// dimension collapsed to unit size. Returns nullopt for unsupported
+/// identity-layout sources. In addition:
+///     identical to the source identity strides, and exactly one non-unit
+///     source
+///  * Non-scalar results must have static offsets, static result strides
+///     dimension collapsed to unit size
+/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
+/// strides are effectively irrelevant).
+///
+/// Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
@@ -179,7 +185,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  // Track result dimensions that produce varying indices; unit dimensions are
+  // Track non-unit result dimensions; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize != 1)
@@ -194,10 +200,13 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
+  bool isOffsetDynamic = ShapedType::isDynamic(rcOffsets[0]);
+
+  // Cases with at least one non-unit dimension in reinterpret_cast's result are
+  // restricted to preserving all but one dimension from the source, which
+  // collapsed to `1` in the result, and fully static metadata.
   if (!isScalarRes) {
-    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
-    // views with static metadata.
-    if (ShapedType::isDynamic(rcOffsets[0]))
+    if (isOffsetDynamic)
       return std::nullopt;
 
     SmallVector<int64_t> srcIdentityStrides =
@@ -212,15 +221,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+    if (!hasExactlyOneCollapsedNonUnitDim(rc))
       return std::nullopt;
   }
 
   // CASE 1: Dynamic ReinterpretCast offset.
   //
-  // Dynamic offsets are supported only for scalar-shaped results, under the
-  // previous effectively-1D source restriction.
-  if (ShapedType::isDynamic(rcOffsets[0])) {
+  // Dynamic offsets are supported only for effectively-1D to scalar
+  // reinterpret_casts.
+  if (isOffsetDynamic) {
     // With an effectively-1D source, a dynamic offset can be mapped to its
     // unique non-unit dim. For other cases, bail out.
     if (llvm::count_if(srcType.getShape(),
@@ -234,9 +243,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // Delinearize static ReinterpretCast offset as in-bounds indices (one for
   // every source dimension).
   dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-  assert(dimsAndOffs.delinearizedOffsets &&
-         "static reinterpret_cast offset must delinearize to in-bounds "
-         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -297,7 +303,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         getResultNonUnitDimsAndOffsetsForRC(rc);
     if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar or loop copy region");
+          op,
+          "unsupported reinterpret_cast result dimensions, strides, or offset");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
@@ -314,11 +321,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
                  dstType.getDimSize(dim);
         }))
-      return rewriter.notifyMatchFailure(
-          op, "copy accesses invalid accessible region");
+      return rewriter.notifyMatchFailure(op, "copy accesses are OOB");
 
-    // Reuse common index constants across bounds, steps, and static offsets,
-    // but avoid creating them for rank-0 copies.
+    // Constant Op cache to reuse common index constants across bounds, steps,
+    // and static offsets: 0 is stored at index 0 and 1 is stored at index 1.
     std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
       if (value == 0 || value == 1) {

>From 02afce707f3007d12829602ff84b31e1c5db3e95 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:59:34 +0200
Subject: [PATCH 15/16] Fixup

---
 mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 0285620e22524..1be4b5675dc2d 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -44,8 +44,8 @@ struct ResultNonUnitDimsAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Returns source indices for a static reinterpret_cast offset of an
-/// identity-layout source.
+/// Returns delinearized offset indices for a static reinterpret_cast offset of
+/// an identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();

>From 6fa9981a9f0ad9c61dcf2be116954ba53cf3acba Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 16:50:35 +0200
Subject: [PATCH 16/16] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       |  4 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 40 ++++++++-----------
 2 files changed, 19 insertions(+), 25 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 1be4b5675dc2d..d269cc3ca0c41 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -353,6 +353,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getOrCreateIndexConstant(rcResType.getDimSize(dim)));
     }
 
+    // All indices are initialised to zero.
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
     if (dimsAndOffs->delinearizedOffsets) {
@@ -365,7 +366,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Dynamic offsets are used directly only for effectively-1D sources.
+      // Dynamic offset is used directly only for effectively-1D sources.
       assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
              "Expecting at most one non-unit result dimension.");
 
@@ -378,6 +379,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
+      // Only the index corresponding to the single non-unit dim is updated.
       rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index afa6833598c02..d303cbc449d12 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -49,8 +49,8 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
 }
 
 /// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
-func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_base_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_base_non_identity_layout(
   %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -179,8 +179,12 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Offset delinearized to [0, 0, 10], therefore is only
-/// added to the trailing source dimension.
+/// %dst has identity strides [33, 11, 1].
+/// Offset 10 delinearizes as:
+///   dim 0: 10 / 33 = 0, remainder 10
+///   dim 1: 10 / 11 = 0, remainder 10
+///   dim 2: 10 /  1 = 10, remainder 0
+/// Therefore the scalar is stored at %dst[0, 0, 10].
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -204,7 +208,12 @@ func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearize
   return
 }
 
-// Offset delinearized into more dimensions: [0, 2, 1]
+/// %dst has identity strides [33, 11, 1].
+/// Offset 23 delinearizes as:
+///   dim 0: 23 / 33 = 0, remainder 23
+///   dim 1: 23 / 11 = 2, remainder 1
+///   dim 2:  1 /  1 = 1, remainder 0
+/// Therefore the scalar is stored at %dst[0, 2, 1].
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -296,6 +305,7 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
+/// Copied non-unit dimension is the right-most dimension, therefore the loop indices are placed there.
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -408,6 +418,7 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
+/// 4 is collapsed to a non-unit dimension.
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
 func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
@@ -426,25 +437,6 @@ func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   return
 }
 
-/// No non-unit dimension collapsed
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
-func.func private @negative_copy_2D_into_2D_strided(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>


From llvmlistbot at llvm.org  Wed Jul  8 07:57:07 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 07:57:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e6543.13be6df8.145c.3f71@mx.google.com>


https://github.com/ivanradanov updated https://github.com/llvm/llvm-project/pull/208096

>From fd41d92a901dabf805d1117b7263a6f74f3727dd Mon Sep 17 00:00:00 2001
From: Ivan Radanov Ivanov <iivanov at nvidia.com>
Date: Tue, 7 Jul 2026 12:25:07 -0700
Subject: [PATCH 1/2] [mlir][acc] Add acc.on_device op

Add an operation to represent the runtime call to acc_on_device.

This runtime call is important to fold early in the compilation pipeline
and having an operation allows us to easily recognize it when emitted by
frontends.
---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        | 27 +++++++++++++++++++
 mlir/test/Dialect/OpenACC/ops.mlir            |  8 ++++++
 2 files changed, 35 insertions(+)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 26a9344fa3a15..8af352c08c156 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
+  let summary = "acc_on_device intrinsic operation";
+
+  let description = [{
+    Represents a call to the OpenACC `acc_on_device` runtime function.
+    Returns whether the current thread is executing on the given device type.
+
+    Example:
+
+    ```mlir
+    %host = arith.constant 1 : i32
+    %on_host = acc.on_device %host : i32 -> i1
+    ```
+  }];
+
+  let arguments = (ins I32:$deviceType);
+  let results = (outs I1:$result);
+
+  let assemblyFormat = [{
+    $deviceType attr-dict `:` type($deviceType) `->` type($result)
+  }];
+}
+
 include "mlir/Dialect/OpenACC/OpenACCCGAttributes.td"
 include "mlir/Dialect/OpenACC/OpenACCCGOps.td"
 
diff --git a/mlir/test/Dialect/OpenACC/ops.mlir b/mlir/test/Dialect/OpenACC/ops.mlir
index 2fb73e400001f..022b71071950e 100644
--- a/mlir/test/Dialect/OpenACC/ops.mlir
+++ b/mlir/test/Dialect/OpenACC/ops.mlir
@@ -1187,6 +1187,14 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 
 // -----
 
+%hostDevType = arith.constant 2 : i32
+%onHost = acc.on_device %hostDevType : i32 -> i1
+
+// CHECK: [[HOSTDEVTYPE:%.*]] = arith.constant 2 : i32
+// CHECK: %[[ONHOST:.*]] = acc.on_device [[HOSTDEVTYPE]] : i32 -> i1
+
+// -----
+
 func.func @testexitdataop(%a: !llvm.ptr) -> () {
   %ifCond = arith.constant true
   %i64Value = arith.constant 1 : i64

>From 7927ec2fc15f204cc4f14cd3cf1f825f9cd39fa3 Mon Sep 17 00:00:00 2001
From: Ivan Radanov Ivanov <iivanov at nvidia.com>
Date: Wed, 8 Jul 2026 07:54:40 -0700
Subject: [PATCH 2/2] op fix

---
 mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td |  4 ++--
 mlir/test/Dialect/OpenACC/ops.mlir              | 14 +++++++++-----
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 8af352c08c156..084536dd79960 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -3717,7 +3717,7 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
 // acc_on_device runtime call
 //===----------------------------------------------------------------------===//
 
-def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device"> {
   let summary = "acc_on_device intrinsic operation";
 
   let description = [{
@@ -3732,7 +3732,7 @@ def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
     ```
   }];
 
-  let arguments = (ins I32:$deviceType);
+  let arguments = (ins IntOrIndex:$deviceType);
   let results = (outs I1:$result);
 
   let assemblyFormat = [{
diff --git a/mlir/test/Dialect/OpenACC/ops.mlir b/mlir/test/Dialect/OpenACC/ops.mlir
index 022b71071950e..c696cefb94600 100644
--- a/mlir/test/Dialect/OpenACC/ops.mlir
+++ b/mlir/test/Dialect/OpenACC/ops.mlir
@@ -1187,11 +1187,15 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 
 // -----
 
-%hostDevType = arith.constant 2 : i32
-%onHost = acc.on_device %hostDevType : i32 -> i1
-
-// CHECK: [[HOSTDEVTYPE:%.*]] = arith.constant 2 : i32
-// CHECK: %[[ONHOST:.*]] = acc.on_device [[HOSTDEVTYPE]] : i32 -> i1
+%devTypeInt = arith.constant 2 : i32
+%onHostInt = acc.on_device %devType : i32 -> i1
+%devTypeIndex = arith.constant 2 : index
+%onHostIndex = acc.on_device %devType : index -> i1
+
+// CHECK: [[DEVTYPEINT:%.*]] = arith.constant 2 : i32
+// CHECK: %[[ONHOSTINT:.*]] = acc.on_device [[DEVTYPEINT]] : i32 -> i1
+// CHECK: [[DEVTYPEINDEX:%.*]] = arith.constant 2 : index
+// CHECK: %[[ONHOSTINDEX:.*]] = acc.on_device [[DEVTYPEINDEX]] : index -> i1
 
 // -----
 


From llvmlistbot at llvm.org  Wed Jul  8 07:58:11 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 07:58:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e6583.cb30c3e4.3ce559.51b2@mx.google.com>


================
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
+  let summary = "acc_on_device intrinsic operation";
+
+  let description = [{
+    Represents a call to the OpenACC `acc_on_device` runtime function.
+    Returns whether the current thread is executing on the given device type.
+
+    Example:
+
+    ```mlir
+    %host = arith.constant 1 : i32
+    %on_host = acc.on_device %host : i32 -> i1
+    ```
+  }];
+
+  let arguments = (ins I32:$deviceType);
----------------
ivanradanov wrote:

That sounds good, I changed it to `IntOrIndex`. Thanks.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 07:58:17 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 07:58:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e6589.6c8668f2.e4f82.0122@mx.google.com>


================
@@ -1187,6 +1187,14 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 
 // -----
 
+%hostDevType = arith.constant 2 : i32
----------------
ivanradanov wrote:

Done, thanks.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 07:58:26 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 07:58:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e6592.eb165e5c.1817a5.03cf@mx.google.com>


================
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
----------------
ivanradanov wrote:

Fixed.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 07:59:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 07:59:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1538aeb - [MLIR][XeGPU] Add non-splat
 constant distribution in SgToLane pass (#205575)
Message-ID: <6a4e65dc.70920dee.290dca.3a2e@mx.google.com>


Author: Nishant Patel
Date: 2026-07-08T07:59:36-07:00
New Revision: 1538aebce77546372d39ebe25bfe75cee6ed39f8

URL: https://github.com/llvm/llvm-project/commit/1538aebce77546372d39ebe25bfe75cee6ed39f8
DIFF: https://github.com/llvm/llvm-project/commit/1538aebce77546372d39ebe25bfe75cee6ed39f8.diff

LOG: [MLIR][XeGPU] Add non-splat constant distribution in SgToLane pass (#205575)

Added: 
    

Modified: 
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
    mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index f9a82bf2b1684..874487da10b30 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -10,6 +10,7 @@
 #include "mlir/Dialect/Math/IR/Math.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/XeGPU/IR/XeGPU.h"
 #include "mlir/Dialect/XeGPU/Transforms/Passes.h"
@@ -357,6 +358,11 @@ struct SgToLaneElementWise : public ConversionPattern {
 
 /// Distributes a subgroup-level arith ConstantOp to lane-level arith
 /// ConstantOp.
+///
+/// Splat constants are rebuilt with the lane-local vector type. Non-splat
+/// constants are distributed by extracting each lane_data-sized block from
+/// the full constant and inserting it at the correct position in the
+/// distributed vector using insert_strided_slice.
 struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
   using OpConversionPattern<arith::ConstantOp>::OpConversionPattern;
 
@@ -367,11 +373,11 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
     if (!resultType)
       return failure();
 
-    // Only handle dense vector constants
-    auto dense = dyn_cast<SplatElementsAttr>(op.getValue());
-    if (!dense)
+    // Only handle dense vector constants.
+    auto denseAttr = dyn_cast<DenseElementsAttr>(op.getValue());
+    if (!denseAttr)
       return rewriter.notifyMatchFailure(
-          op, "only dense splat vector constants are supported");
+          op, "only dense vector constants are supported");
 
     xegpu::DistributeLayoutAttr layout =
         xegpu::getTemporaryLayout(llvm::cast<OpResult>(op.getResult()));
@@ -387,12 +393,83 @@ struct SgToLaneArithConstant : public OpConversionPattern<arith::ConstantOp> {
           op, "unable to compute lane vector type from the layout");
 
     VectorType newResultType = laneShapeOrFailure.value();
-    auto sclarValue = dense.getSplatValue<Attribute>();
-    auto newDenseAttr = DenseElementsAttr::get(newResultType, sclarValue);
+    Location loc = op.getLoc();
 
-    auto newOp = arith::ConstantOp::create(rewriter, op.getLoc(), newResultType,
-                                           newDenseAttr);
-    rewriter.replaceOp(op, newOp.getResult());
+    // Splat constants: every lane gets the same value, so just rebuild the
+    // splat with the distributed type.
+    if (denseAttr.isSplat()) {
+      auto scalarValue = denseAttr.getSplatValue<Attribute>();
+      auto newDenseAttr = DenseElementsAttr::get(newResultType, scalarValue);
+      auto newOp =
+          arith::ConstantOp::create(rewriter, loc, newResultType, newDenseAttr);
+      rewriter.replaceOp(op, newOp.getResult());
+      return success();
+    }
+
+    // Non-splat constants: each lane extracts the elements it owns from the
+    // full constant using the distributed coordinates from the layout.
+    auto fullConst =
+        arith::ConstantOp::create(rewriter, loc, resultType, denseAttr);
+
+    Value laneId = gpu::LaneIdOp::create(rewriter, loc, rewriter.getIndexType(),
+                                         /*upperBound=*/mlir::IntegerAttr());
+    auto maybeCoordsVec = layout.computeDistributedCoords(
+        rewriter, loc, laneId, resultType.getShape());
+    if (failed(maybeCoordsVec))
+      return rewriter.notifyMatchFailure(
+          op, "failed to compute distributed coordinates from layout");
+
+    SmallVector<SmallVector<Value>> coordsVec = maybeCoordsVec.value();
+    SmallVector<int64_t> laneData = layout.getEffectiveLaneDataAsInt();
+    ArrayRef<int64_t> distShape = newResultType.getShape();
+    int64_t rank = newResultType.getRank();
+
+    // Each lane owns one lane_data-sized block per distribution unit.
+    // computeDistributedCoords returns those block starts in row-major order
+    // over the block grid (distShape / laneData).
+    SmallVector<int64_t> blockGridShape(rank);
+    for (int64_t d = 0; d < rank; d++)
+      blockGridShape[d] = distShape[d] / laneData[d];
+    SmallVector<int64_t> blockGridStrides = computeStrides(blockGridShape);
+
+    auto blockType = VectorType::get(laneData, newResultType.getElementType());
+    SmallVector<int64_t> unitTile(rank, 1);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value result = arith::ConstantOp::create(
+        rewriter, loc, newResultType, rewriter.getZeroAttr(newResultType));
+
+    for (auto [blockIdx, blockStart] : llvm::enumerate(coordsVec)) {
+      // Gather the block's elements from the full constant. The block start is
+      // lane-dynamic, so extract element-by-element (row-major over lane_data)
+      // instead.
+      SmallVector<Value> blockElems;
+      for (SmallVector<int64_t> off :
+           StaticTileOffsetRange(laneData, unitTile)) {
+        SmallVector<OpFoldResult> pos(rank);
+        for (int64_t d = 0; d < rank; d++)
+          pos[d] = getAsOpFoldResult(arith::AddIOp::create(
+              rewriter, loc, blockStart[d],
+              arith::ConstantIndexOp::create(rewriter, loc, off[d])));
+        blockElems.push_back(vector::ExtractOp::create(
+            rewriter, loc, fullConst.getResult(), pos));
+      }
+
+      // Rebuild the block keeping its lane_data shape, then place it with
+      // insert_strided_slice so the block keeps its orientation in the
+      // distributed vector (e.g. a [2, 1] block stays a vertical 2x1 slice).
+      Value block =
+          vector::FromElementsOp::create(rewriter, loc, blockType, blockElems);
+      SmallVector<int64_t> blockGridPos =
+          delinearize(blockIdx, blockGridStrides);
+      SmallVector<int64_t> offsets(rank);
+      for (int64_t d = 0; d < rank; d++)
+        offsets[d] = blockGridPos[d] * laneData[d];
+      result = vector::InsertStridedSliceOp::create(rewriter, loc, block,
+                                                    result, offsets, strides);
+    }
+
+    rewriter.replaceOp(op, result);
     return success();
   }
 };

diff  --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fcc2da3d5005b..fe356e6af35c1 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -160,6 +160,91 @@ gpu.func @arith_constant() {
   gpu.return
 }
 
+// Non-splat constant: each lane extracts the element it owns from the full
+// constant and inserts its lane_data-sized block into the distributed vector.
+// CHECK-LABEL: gpu.func @arith_constant_non_splat
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<1xindex>
+// CHECK: %[[ELEM:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<16xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM]] : vector<1xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<1xindex> into vector<1xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
+    }> : vector<16xindex>
+  gpu.return
+}
+
+// With lane_data > 1 each lane owns a lane_data-sized block, so multiple
+// elements are extracted, reassembled into the block, and placed with
+// insert_strided_slice.
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_lane_data
+// CHECK: %[[CST:.*]] = arith.constant dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<2xindex>
+// CHECK: %[[ELEM0:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[ELEM1:.*]] = vector.extract %[[CST]][%{{.*}}] : index from vector<32xindex>
+// CHECK: %[[BLK:.*]] = vector.from_elements %[[ELEM0]], %[[ELEM1]] : vector<2xindex>
+// CHECK: %[[RES:.*]] = vector.insert_strided_slice %[[BLK]], %[[ZERO]] {offsets = [0], strides = [1]} : vector<2xindex> into vector<2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_lane_data() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>}
+    dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>,
+      target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>
+    }> : vector<32xindex>
+  gpu.return
+}
+
+// 2D non-splat constant with vertical lane_data [2, 1]. The distributed type
+// is vector<4x2xindex>. Each [2, 1] block is correctly placed vertically in
+// the result (same column, adjacent rows).
+// CHECK-LABEL: gpu.func @arith_constant_non_splat_2d_vertical_lanedata
+// CHECK: %[[CST:.*]] = arith.constant dense<{{.*}}> : vector<4x32xindex>
+// CHECK: %[[LANE:.*]] = gpu.lane_id
+// CHECK: %[[ZERO:.*]] = arith.constant dense<0> : vector<4x2xindex>
+// CHECK: %[[E0:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E1:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B0:.*]] = vector.from_elements %[[E0]], %[[E1]] : vector<2x1xindex>
+// CHECK: %[[I0:.*]] = vector.insert_strided_slice %[[B0]], %[[ZERO]] {offsets = [0, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E2:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E3:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B1:.*]] = vector.from_elements %[[E2]], %[[E3]] : vector<2x1xindex>
+// CHECK: %[[I1:.*]] = vector.insert_strided_slice %[[B1]], %[[I0]] {offsets = [0, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E4:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E5:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B2:.*]] = vector.from_elements %[[E4]], %[[E5]] : vector<2x1xindex>
+// CHECK: %[[I2:.*]] = vector.insert_strided_slice %[[B2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: %[[E6:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[E7:.*]] = vector.extract %[[CST]][%{{.*}}, %{{.*}}] : index from vector<4x32xindex>
+// CHECK: %[[B3:.*]] = vector.from_elements %[[E6]], %[[E7]] : vector<2x1xindex>
+// CHECK: %[[I3:.*]] = vector.insert_strided_slice %[[B3]], %[[I2]] {offsets = [2, 1], strides = [1, 1]} : vector<2x1xindex> into vector<4x2xindex>
+// CHECK: gpu.return
+gpu.func @arith_constant_non_splat_2d_vertical_lanedata() {
+  %0 = arith.constant
+    {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
+    dense<[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
+           [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63],
+           [64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95],
+           [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127]]> : vector<4x32xindex>
+  %cl0 = xegpu.convert_layout %0
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>
+    }> : vector<4x32xindex>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @prefetch_nd
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: xegpu.prefetch_nd %{{.*}}[%[[C0]], %[[C0]]] : !xegpu.tensor_desc<16x16xf16>


        

From llvmlistbot at llvm.org  Wed Jul  8 07:59:44 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 08 Jul 2026 07:59:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add non-splat constant
 distribution in SgToLane pass (PR #205575)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205575@github.com>
Message-ID: <6a4e65e0.f06f4994.1fc857.220c@mx.google.com>


https://github.com/nbpatel closed https://github.com/llvm/llvm-project/pull/205575

From llvmlistbot at llvm.org  Wed Jul  8 08:01:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 08:01:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 7468b56 - [mlir][acc] Add utilities for
 working with acc par dims (#208120)
Message-ID: <6a4e6653.510f576d.1b3cb9.eb5a@mx.google.com>


Author: Razvan Lupusoru
Date: 2026-07-08T08:01:34-07:00
New Revision: 7468b5634b038ba915fc326a8fb2f3fe6c3ed098

URL: https://github.com/llvm/llvm-project/commit/7468b5634b038ba915fc326a8fb2f3fe6c3ed098
DIFF: https://github.com/llvm/llvm-project/commit/7468b5634b038ba915fc326a8fb2f3fe6c3ed098.diff

LOG: [mlir][acc] Add utilities for working with acc par dims (#208120)

Add shared helpers for reading, setting, and manipulating parallel
dimensions on operations, covering both discardable and inherent
attributes.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
    mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
    mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
    mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index bb132f5d02e8c..64de4eda7366a 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/IR/IRMapping.h"
 #include "mlir/Interfaces/DataLayoutInterfaces.h"
+#include "llvm/ADT/SmallVector.h"
 #include <optional>
 
 namespace mlir {
@@ -62,6 +63,33 @@ ComputeRegionOp buildComputeRegion(Location loc, ValueRange launchArgs,
                                    Value stream = {},
                                    ValueRange inputArgsToMap = {});
 
+/// Insert \p parDim into \p parDims while preserving dimension ordering. If the
+/// dimension is already present, this is a no-op.
+void insertParDim(llvm::SmallVector<GPUParallelDimAttr> &parDims,
+                  GPUParallelDimAttr parDim);
+
+/// Remove \p parDim from \p parDims if present.
+void removeParDim(llvm::SmallVector<GPUParallelDimAttr> &parDims,
+                  GPUParallelDimAttr parDim);
+
+/// Obtain the parallel dimensions carried by \p op, if any.
+GPUParallelDimsAttr getParDimsAttr(Operation *op);
+
+/// Return whether \p op carries parallel dimensions.
+bool hasParDimsAttr(Operation *op);
+
+/// Return whether \p op carries sequential parallel dimensions.
+bool hasSeqParDims(Operation *op);
+
+/// Set parallel dimensions on \p op.
+void setParDimsAttr(Operation *op, GPUParallelDimsAttr attr);
+
+/// Update parallel dimensions on \p op.
+void updateParDimsAttr(Operation *op, GPUParallelDimsAttr attr);
+
+/// Copy parallel dimensions from \p from to \p to.
+void copyParDimsAttr(Operation *from, Operation *to);
+
 } // namespace acc
 } // namespace mlir
 

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
index 1c5f302f23192..b3d54d2e05697 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
@@ -138,10 +138,6 @@ static bool isOpInSerialRegion(Operation *op) {
   return false;
 }
 
-static void setParDimsAttr(Operation *op, GPUParallelDimsAttr attr) {
-  op->setAttr(GPUParallelDimsAttr::name, attr);
-}
-
 /// Clone defining ops of constant live-in values into `region`, rewrite uses
 /// inside the region to the clones, and remove those values from
 /// `liveInValues` so they are not threaded through `acc.compute_region` ins.
@@ -172,19 +168,6 @@ static void materializeConstantLiveInsIntoRegion(Region &region,
   }
 }
 
-/// Insert a parallel dimension into the list, maintaining order by
-/// GPUParallelDimAttr::getOrder (descending).
-static void insertParDim(SmallVectorImpl<GPUParallelDimAttr> &parDims,
-                         GPUParallelDimAttr parDim) {
-  GPUParallelDimAttr *lb = llvm::lower_bound(
-      parDims, parDim,
-      [](const GPUParallelDimAttr &a, const GPUParallelDimAttr &b) {
-        return a.getOrder() > b.getOrder();
-      });
-  if (lb == parDims.end() || *lb != parDim)
-    parDims.insert(lb, parDim);
-}
-
 /// Return the device type from which gang/worker/vector clauses should be read.
 /// If the requested device type has any such clauses, use that exclusively;
 /// otherwise fall back to the default (DeviceType::None).

diff  --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index d18522c3c440f..06605371bdf42 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -16,6 +16,8 @@
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/IRMapping.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/TypeSwitch.h"
 
 namespace mlir {
 namespace acc {
@@ -128,5 +130,81 @@ ComputeRegionOp buildComputeRegion(Location loc, ValueRange launchArgs,
   return computeRegion;
 }
 
+static SmallVector<GPUParallelDimAttr>::iterator
+findParDim(SmallVector<GPUParallelDimAttr> &parDims,
+           GPUParallelDimAttr parDim) {
+  return llvm::lower_bound(
+      parDims, parDim,
+      [](const GPUParallelDimAttr &lhs, const GPUParallelDimAttr &rhs) {
+        return lhs.getOrder() > rhs.getOrder();
+      });
+}
+
+void insertParDim(SmallVector<GPUParallelDimAttr> &parDims,
+                  GPUParallelDimAttr parDim) {
+  SmallVector<GPUParallelDimAttr>::iterator lb = findParDim(parDims, parDim);
+  if (lb == parDims.end() || *lb != parDim)
+    parDims.insert(lb, parDim);
+}
+
+void removeParDim(SmallVector<GPUParallelDimAttr> &parDims,
+                  GPUParallelDimAttr parDim) {
+  SmallVector<GPUParallelDimAttr>::iterator lb = findParDim(parDims, parDim);
+  if (lb != parDims.end() && *lb == parDim)
+    parDims.erase(lb);
+}
+
+#define ACC_OP_WITH_PAR_DIMS_LIST                                              \
+  PrivatizeOp, ReductionAccumulateOp, ReductionAccumulateArrayOp
+
+GPUParallelDimsAttr getParDimsAttr(Operation *op) {
+  return llvm::TypeSwitch<Operation *, GPUParallelDimsAttr>(op)
+      .Case<ACC_OP_WITH_PAR_DIMS_LIST>(
+          [](auto parOp) { return parOp.getParDimsAttr(); })
+      .Default([](Operation *op) -> GPUParallelDimsAttr {
+        if (Attribute attr = op->getAttr(GPUParallelDimsAttr::name)) {
+          GPUParallelDimsAttr parDimsAttr = dyn_cast<GPUParallelDimsAttr>(attr);
+          assert(parDimsAttr && "acc.par_dims must be a GPUParallelDimsAttr");
+          return parDimsAttr;
+        }
+        return nullptr;
+      });
+}
+
+bool hasParDimsAttr(Operation *op) { return getParDimsAttr(op) != nullptr; }
+
+bool hasSeqParDims(Operation *op) {
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(op))
+    return parDimsAttr.isSeq();
+  return false;
+}
+
+void setParDimsAttr(Operation *op, GPUParallelDimsAttr attr) {
+  assert(!hasParDimsAttr(op) && "parallel dimensions attribute is already set");
+  llvm::TypeSwitch<Operation *>(op)
+      .Case<ACC_OP_WITH_PAR_DIMS_LIST>(
+          [&](auto parOp) { parOp.setParDimsAttr(attr); })
+      .Default(
+          [&](Operation *op) { op->setAttr(GPUParallelDimsAttr::name, attr); });
+}
+
+void updateParDimsAttr(Operation *op, GPUParallelDimsAttr attr) {
+  assert(hasParDimsAttr(op) &&
+         "expected parallel dimensions attribute to already be set");
+  llvm::TypeSwitch<Operation *>(op)
+      .Case<ACC_OP_WITH_PAR_DIMS_LIST>(
+          [&](auto parOp) { parOp.setParDimsAttr(attr); })
+      .Default(
+          [&](Operation *op) { op->setAttr(GPUParallelDimsAttr::name, attr); });
+}
+
+#undef ACC_OP_WITH_PAR_DIMS_LIST
+
+void copyParDimsAttr(Operation *from, Operation *to) {
+  assert(hasParDimsAttr(from) &&
+         "expected parallel dimensions attribute to already be set");
+  setParDimsAttr(to, getParDimsAttr(from));
+}
+
 } // namespace acc
 } // namespace mlir

diff  --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 6fe0ffb2d54fe..75cf385af096f 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -83,6 +83,95 @@ TEST_F(OpenACCUtilsCGTest, getDataLayoutWithSpec) {
   EXPECT_TRUE(dl2.has_value());
 }
 
+//===----------------------------------------------------------------------===//
+// ParDim utilities Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, insertParDimOrdersAndDeduplicates) {
+  SmallVector<GPUParallelDimAttr> parDims;
+  GPUParallelDimAttr threadX = GPUParallelDimAttr::threadXDim(&context);
+  GPUParallelDimAttr blockX = GPUParallelDimAttr::blockXDim(&context);
+  GPUParallelDimAttr threadY = GPUParallelDimAttr::threadYDim(&context);
+
+  insertParDim(parDims, threadX);
+  insertParDim(parDims, blockX);
+  insertParDim(parDims, threadY);
+  insertParDim(parDims, threadX);
+
+  ASSERT_EQ(parDims.size(), 3u);
+  EXPECT_EQ(parDims[0], blockX);
+  EXPECT_EQ(parDims[1], threadY);
+  EXPECT_EQ(parDims[2], threadX);
+}
+
+TEST_F(OpenACCUtilsCGTest, removeParDimRemovesOnlyMatchingDim) {
+  GPUParallelDimAttr threadX = GPUParallelDimAttr::threadXDim(&context);
+  GPUParallelDimAttr blockX = GPUParallelDimAttr::blockXDim(&context);
+  GPUParallelDimAttr threadY = GPUParallelDimAttr::threadYDim(&context);
+  SmallVector<GPUParallelDimAttr> parDims{blockX, threadY, threadX};
+
+  removeParDim(parDims, threadX);
+  removeParDim(parDims, GPUParallelDimAttr::blockYDim(&context));
+
+  ASSERT_EQ(parDims.size(), 2u);
+  EXPECT_EQ(parDims[0], blockX);
+  EXPECT_EQ(parDims[1], threadY);
+}
+
+TEST_F(OpenACCUtilsCGTest, parDimsOperationAttributes) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  OwningOpRef<ModuleOp> otherModule = ModuleOp::create(b, loc);
+  Operation *op = module->getOperation();
+  Operation *otherOp = otherModule->getOperation();
+  GPUParallelDimsAttr seqAttr = GPUParallelDimsAttr::seq(&context);
+  GPUParallelDimsAttr blockAttr = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+
+  EXPECT_FALSE(hasParDimsAttr(op));
+  setParDimsAttr(op, seqAttr);
+  EXPECT_TRUE(hasParDimsAttr(op));
+  EXPECT_TRUE(hasSeqParDims(op));
+  EXPECT_EQ(getParDimsAttr(op), seqAttr);
+
+  updateParDimsAttr(op, blockAttr);
+  EXPECT_FALSE(hasSeqParDims(op));
+  EXPECT_EQ(getParDimsAttr(op), blockAttr);
+
+  copyParDimsAttr(op, otherOp);
+  EXPECT_EQ(getParDimsAttr(otherOp), blockAttr);
+}
+
+TEST_F(OpenACCUtilsCGTest, getParDimsAttrReadsInherentAttribute) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  GPUParallelDimsAttr blockAttr = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+  Type privateTy = PrivateType::get(&context, b.getI32Type());
+  auto privatize =
+      PrivatizeOp::create(b, loc, privateTy,
+                          /*dynamicSizes=*/ValueRange{}, blockAttr);
+
+  EXPECT_TRUE(hasParDimsAttr(privatize));
+  EXPECT_EQ(getParDimsAttr(privatize), blockAttr);
+  EXPECT_EQ(privatize.getParDimsAttr(), blockAttr);
+}
+
+TEST_F(OpenACCUtilsCGTest, setParDimsAttrSetsInherentAttribute) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  Type privateTy = PrivateType::get(&context, b.getI32Type());
+  auto privatize =
+      PrivatizeOp::create(b, loc, privateTy, /*dynamicSizes=*/ValueRange{});
+  GPUParallelDimsAttr blockAttr = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+
+  setParDimsAttr(privatize, blockAttr);
+  EXPECT_EQ(getParDimsAttr(privatize), blockAttr);
+  EXPECT_EQ(privatize.getParDimsAttr(), blockAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // buildComputeRegion Tests
 //===----------------------------------------------------------------------===//


        

From llvmlistbot at llvm.org  Wed Jul  8 08:01:42 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 08:01:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for working with
	acc par dims (PR #208120)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208120@github.com>
Message-ID: <6a4e6656.8e75e0cb.5e108.fb4d@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/208120

From llvmlistbot at llvm.org  Wed Jul  8 08:09:11 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 08 Jul 2026 08:09:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a4e6817.30759430.24d50c.1aeb@mx.google.com>


ioghiban wrote:

I believe all reviewer comments have now been addressed (*).

This is a significant blocker for MLGO integration: https://github.com/llvm/llvm-project/issues/199007. It would be great if we could land it soon.

If there are no new comments by Friday EOD (10/7/26), I suggest we proceed with merging.
Thank you everyone for your feedback!

(*) Please leave a comment if I missed anything.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Wed Jul  8 08:18:05 2026
From: llvmlistbot at llvm.org (Ivan R. Ivanov)
Date: Wed, 08 Jul 2026 08:18:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e6a2d.8e75e0cb.5e108.9a6b@mx.google.com>


https://github.com/ivanradanov updated https://github.com/llvm/llvm-project/pull/208096

>From fd41d92a901dabf805d1117b7263a6f74f3727dd Mon Sep 17 00:00:00 2001
From: Ivan Radanov Ivanov <iivanov at nvidia.com>
Date: Tue, 7 Jul 2026 12:25:07 -0700
Subject: [PATCH 1/3] [mlir][acc] Add acc.on_device op

Add an operation to represent the runtime call to acc_on_device.

This runtime call is important to fold early in the compilation pipeline
and having an operation allows us to easily recognize it when emitted by
frontends.
---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        | 27 +++++++++++++++++++
 mlir/test/Dialect/OpenACC/ops.mlir            |  8 ++++++
 2 files changed, 35 insertions(+)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 26a9344fa3a15..8af352c08c156 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -3713,6 +3713,33 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// acc_on_device runtime call
+//===----------------------------------------------------------------------===//
+
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
+  let summary = "acc_on_device intrinsic operation";
+
+  let description = [{
+    Represents a call to the OpenACC `acc_on_device` runtime function.
+    Returns whether the current thread is executing on the given device type.
+
+    Example:
+
+    ```mlir
+    %host = arith.constant 1 : i32
+    %on_host = acc.on_device %host : i32 -> i1
+    ```
+  }];
+
+  let arguments = (ins I32:$deviceType);
+  let results = (outs I1:$result);
+
+  let assemblyFormat = [{
+    $deviceType attr-dict `:` type($deviceType) `->` type($result)
+  }];
+}
+
 include "mlir/Dialect/OpenACC/OpenACCCGAttributes.td"
 include "mlir/Dialect/OpenACC/OpenACCCGOps.td"
 
diff --git a/mlir/test/Dialect/OpenACC/ops.mlir b/mlir/test/Dialect/OpenACC/ops.mlir
index 2fb73e400001f..022b71071950e 100644
--- a/mlir/test/Dialect/OpenACC/ops.mlir
+++ b/mlir/test/Dialect/OpenACC/ops.mlir
@@ -1187,6 +1187,14 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 
 // -----
 
+%hostDevType = arith.constant 2 : i32
+%onHost = acc.on_device %hostDevType : i32 -> i1
+
+// CHECK: [[HOSTDEVTYPE:%.*]] = arith.constant 2 : i32
+// CHECK: %[[ONHOST:.*]] = acc.on_device [[HOSTDEVTYPE]] : i32 -> i1
+
+// -----
+
 func.func @testexitdataop(%a: !llvm.ptr) -> () {
   %ifCond = arith.constant true
   %i64Value = arith.constant 1 : i64

>From 7927ec2fc15f204cc4f14cd3cf1f825f9cd39fa3 Mon Sep 17 00:00:00 2001
From: Ivan Radanov Ivanov <iivanov at nvidia.com>
Date: Wed, 8 Jul 2026 07:54:40 -0700
Subject: [PATCH 2/3] op fix

---
 mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td |  4 ++--
 mlir/test/Dialect/OpenACC/ops.mlir              | 14 +++++++++-----
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 8af352c08c156..084536dd79960 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -3717,7 +3717,7 @@ def OpenACC_WaitOp : OpenACC_Op<"wait", [AttrSizedOperandSegments]> {
 // acc_on_device runtime call
 //===----------------------------------------------------------------------===//
 
-def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
+def OpenACC_OnDeviceOp : OpenACC_Op<"on_device"> {
   let summary = "acc_on_device intrinsic operation";
 
   let description = [{
@@ -3732,7 +3732,7 @@ def OpenACC_OnDeviceOp : OpenACC_Op<"on_device", [NoMemoryEffect]> {
     ```
   }];
 
-  let arguments = (ins I32:$deviceType);
+  let arguments = (ins IntOrIndex:$deviceType);
   let results = (outs I1:$result);
 
   let assemblyFormat = [{
diff --git a/mlir/test/Dialect/OpenACC/ops.mlir b/mlir/test/Dialect/OpenACC/ops.mlir
index 022b71071950e..c696cefb94600 100644
--- a/mlir/test/Dialect/OpenACC/ops.mlir
+++ b/mlir/test/Dialect/OpenACC/ops.mlir
@@ -1187,11 +1187,15 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 
 // -----
 
-%hostDevType = arith.constant 2 : i32
-%onHost = acc.on_device %hostDevType : i32 -> i1
-
-// CHECK: [[HOSTDEVTYPE:%.*]] = arith.constant 2 : i32
-// CHECK: %[[ONHOST:.*]] = acc.on_device [[HOSTDEVTYPE]] : i32 -> i1
+%devTypeInt = arith.constant 2 : i32
+%onHostInt = acc.on_device %devType : i32 -> i1
+%devTypeIndex = arith.constant 2 : index
+%onHostIndex = acc.on_device %devType : index -> i1
+
+// CHECK: [[DEVTYPEINT:%.*]] = arith.constant 2 : i32
+// CHECK: %[[ONHOSTINT:.*]] = acc.on_device [[DEVTYPEINT]] : i32 -> i1
+// CHECK: [[DEVTYPEINDEX:%.*]] = arith.constant 2 : index
+// CHECK: %[[ONHOSTINDEX:.*]] = acc.on_device [[DEVTYPEINDEX]] : index -> i1
 
 // -----
 

>From 74459791fdc38bbc5f51e9be358e5c2c6b2275c1 Mon Sep 17 00:00:00 2001
From: Ivan Radanov Ivanov <iivanov at nvidia.com>
Date: Wed, 8 Jul 2026 08:17:23 -0700
Subject: [PATCH 3/3] test

---
 mlir/test/Dialect/OpenACC/ops.mlir | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/ops.mlir b/mlir/test/Dialect/OpenACC/ops.mlir
index c696cefb94600..a2775390bf12b 100644
--- a/mlir/test/Dialect/OpenACC/ops.mlir
+++ b/mlir/test/Dialect/OpenACC/ops.mlir
@@ -1188,9 +1188,9 @@ acc.shutdown device_num(%idxValue : index) if(%ifCond)
 // -----
 
 %devTypeInt = arith.constant 2 : i32
-%onHostInt = acc.on_device %devType : i32 -> i1
+%onHostInt = acc.on_device %devTypeInt : i32 -> i1
 %devTypeIndex = arith.constant 2 : index
-%onHostIndex = acc.on_device %devType : index -> i1
+%onHostIndex = acc.on_device %devTypeIndex : index -> i1
 
 // CHECK: [[DEVTYPEINT:%.*]] = arith.constant 2 : i32
 // CHECK: %[[ONHOSTINT:.*]] = acc.on_device [[DEVTYPEINT]] : i32 -> i1


From llvmlistbot at llvm.org  Wed Jul  8 08:18:26 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Wed, 08 Jul 2026 08:18:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e6a42.af6ba3ed.36d9db.cc45@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
----------------
smada3 wrote:

As mentioned, the issue with setting this as a default is that the printer will show this for float ops, and this would leak an integer-overflow flag onto float ops during round-trip

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 08:18:49 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Wed, 08 Jul 2026 08:18:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e6a59.ac469dc7.3a57b1.5900@mx.google.com>


https://github.com/smada3 deleted https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 08:19:24 2026
From: llvmlistbot at llvm.org (Moazin K.)
Date: Wed, 08 Jul 2026 08:19:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
Message-ID: <llvm.org/llvm/llvm-project/pull/208238@github.com>


https://github.com/moazin created https://github.com/llvm/llvm-project/pull/208238

Introduce an attribute `OpenACCLoopLocAttr` that can be attached to an `acc.loop` operation to record loop locations and directive location separately.

Attach it to `acc.loop` when building the op in flang.

>From 719804362ed6aa0712d586845176e31347273dfc Mon Sep 17 00:00:00 2001
From: Moazin Khatti <mkhatti at nvidia.com>
Date: Thu, 2 Jul 2026 06:25:53 -0700
Subject: [PATCH] [flang][OpenACC] Attach `OpenACCLoopLocAttr` to `acc.loop` op

Introduce an attribute `OpenACCLoopLocAttr` that can be attached to
an `acc.loop` operation to record loop locations and directive
location separately.

Attach it to `acc.loop` when building the op in flang.

Assisted-by: Claude Code
---
 flang/lib/Lower/OpenACC.cpp                   | 30 ++++++++++++++-----
 flang/test/Lower/OpenACC/locations.f90        | 23 +++++++++++---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        |  9 ++++++
 3 files changed, 51 insertions(+), 11 deletions(-)

diff --git a/flang/lib/Lower/OpenACC.cpp b/flang/lib/Lower/OpenACC.cpp
index e28c31f16b9d9..8c1f071a8d31d 100644
--- a/flang/lib/Lower/OpenACC.cpp
+++ b/flang/lib/Lower/OpenACC.cpp
@@ -1828,7 +1828,6 @@ static void processDoLoopBounds(
         std::pair<Fortran::semantics::SymbolRef, Fortran::semantics::SymbolRef>>
         &localSymPairs) {
   assert(loopsToProcess > 0 && "expect at least one loop");
-  locs.push_back(currentLocation); // Location of the directive
   bool isDoConcurrent = outerDoConstruct.IsDoConcurrent();
 
   if (isDoConcurrent) {
@@ -2105,7 +2104,7 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
                llvm::SmallVector<mlir::Value> &cacheOperands,
                llvm::SmallVector<mlir::Value> &reductionOperands,
                llvm::SmallVector<mlir::Type> &retTy, mlir::Value yieldValue,
-               uint64_t loopsToProcess) {
+               uint64_t loopsToProcess, bool hasDirective) {
   fir::FirOpBuilder &builder = converter.getFirOpBuilder();
 
   llvm::SmallVector<std::pair<mlir::Value, Fortran::semantics::SymbolRef>>
@@ -2151,10 +2150,26 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
   addOperands(operands, operandSegments, firstprivateOperands);
   addOperands(operands, operandSegments, reductionOperands);
 
+  // list of locations to build FusedLoc to be attached to acc loop
+  llvm::SmallVector<mlir::Location> fusedLocs;
+  if (hasDirective)
+    fusedLocs.push_back(currentLocation);
+  fusedLocs.append(locs.begin(), locs.end());
+
+  // list of only loop locations used to build an `mlir::acc::LoopLocAttr` to be
+  // attached to the `FusedLoc` as metadata so that consumers of location
+  // information can separate out the directive's location from the loops'
+  // locations
+  llvm::SmallVector<mlir::LocationAttr> loopLocAttrs(locs.begin(), locs.end());
+  auto loopLocMeta = mlir::acc::LoopLocAttr::get(
+      builder.getContext(),
+      hasDirective ? mlir::LocationAttr(currentLocation) : mlir::LocationAttr{},
+      loopLocAttrs);
+
   auto loopOp = createRegionOp<mlir::acc::LoopOp, mlir::acc::YieldOp>(
-      builder, builder.getFusedLoc(locs), currentLocation, eval, operands,
-      operandSegments, /*outerCombined=*/false, retTy, yieldValue, ivTypes,
-      ivLocs);
+      builder, builder.getFusedLoc(fusedLocs, loopLocMeta), currentLocation,
+      eval, operands, operandSegments, /*outerCombined=*/false, retTy,
+      yieldValue, ivTypes, ivLocs);
   // Ensure the iv symbol is mapped to private iv SSA value for the scope of
   // the loop even if it did not appear explicitly in a PRIVATE clause (if it
   // appeared explicitly in such clause, that is also fine because duplicates
@@ -2419,7 +2434,7 @@ static mlir::acc::LoopOp createLoopOp(
       converter, currentLocation, semanticsContext, stmtCtx, outerDoConstruct,
       eval, privateOperands, dataMap, gangOperands, workerNumOperands,
       vectorOperands, tileOperands, cacheOperands, reductionOperands, retTy,
-      yieldValue, loopsToProcess);
+      yieldValue, loopsToProcess, /*hasDirective=*/true);
 
   if (!gangDeviceTypes.empty())
     loopOp.setGangAttr(builder.getArrayAttr(gangDeviceTypes));
@@ -5355,7 +5370,8 @@ mlir::Operation *Fortran::lower::genOpenACCLoopFromDoConstruct(
       converter, converter.getCurrentLocation(), semanticsContext, stmtCtx,
       doConstruct, eval, privateOperands, dataMap, gangOperands,
       workerNumOperands, vectorOperands, tileOperands, cacheOperands,
-      reductionOperands, retTy, yieldValue, loopsToProcess);
+      reductionOperands, retTy, yieldValue, loopsToProcess,
+      /*hasDirective=*/false);
 
   // Normal do loops which are not annotated with `acc loop` should be
   // left for analysis by marking with `auto`. This is the case even in the case
diff --git a/flang/test/Lower/OpenACC/locations.f90 b/flang/test/Lower/OpenACC/locations.f90
index 46e8df0e9dac4..fe1dc5c918e2d 100644
--- a/flang/test/Lower/OpenACC/locations.f90
+++ b/flang/test/Lower/OpenACC/locations.f90
@@ -53,7 +53,7 @@ subroutine nested_acc_locations(arr1d)
     !CHECK: acc.loop
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":44:11)
-    !CHECK-NEXT: } attributes {{.*}} loc(fused["{{.*}}locations.f90":44:11, "{{.*}}locations.f90":45:5])
+    !CHECK-NEXT: } attributes {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":44:11), loops = loc("{{[^"]*}}locations.f90":45:5)>>["{{[^"]*}}locations.f90":44:11, "{{[^"]*}}locations.f90":45:5])
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":43:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":43:11)
@@ -87,7 +87,7 @@ subroutine combined_directive_locations(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":82:11, "{{.*}}locations.f90":83:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":82:11), loops = loc("{{[^"]*}}locations.f90":83:5)>>["{{[^"]*}}locations.f90":82:11, "{{[^"]*}}locations.f90":83:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":82:11)
   end subroutine
@@ -106,7 +106,7 @@ subroutine if_clause_expr_location(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":99:11, "{{.*}}locations.f90":100:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":99:11), loops = loc("{{[^"]*}}locations.f90":100:5)>>["{{[^"]*}}locations.f90":99:11, "{{[^"]*}}locations.f90":100:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":99:11)
   end subroutine
@@ -169,7 +169,7 @@ subroutine acc_loop_fused_locations(arr)
 
 ! CHECK-LABEL: func.func @_QMacc_locationsPacc_loop_fused_locations
 ! CHECK: acc.loop
-! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused["{{.*}}locations.f90":160:11, "{{.*}}locations.f90":161:5, "{{.*}}locations.f90":162:7, "{{.*}}locations.f90":163:9])
+! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":160:11), loops = loc("{{[^"]*}}locations.f90":161:5), loc("{{[^"]*}}locations.f90":162:7), loc("{{[^"]*}}locations.f90":163:9)>>["{{[^"]*}}locations.f90":160:11, "{{[^"]*}}locations.f90":161:5, "{{[^"]*}}locations.f90":162:7, "{{[^"]*}}locations.f90":163:9])
 
   subroutine data_end_locations(arr)
     real, dimension(10) :: arr
@@ -182,4 +182,19 @@ subroutine data_end_locations(arr)
     !CHECK-LABEL: acc.copyout
     !CHECK-SAME:  loc("{{.*}}locations.f90":181:11)
   end subroutine
+
+  subroutine acc_kernel_with_loop_locations(arr)
+    real, dimension(10) :: arr
+    integer :: i
+
+    !$acc kernels
+    do i = 1, 10
+      arr(i) = arr(i) + 1
+    end do
+    !$acc end kernels
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMacc_locationsPacc_kernel_with_loop_locations
+! CHECK: acc.loop
+! CHECK: } attributes {{.*}} loc(fused<#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>>["{{[^"]*}}locations.f90":191:5])
 end module
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 26a9344fa3a15..070e6688684e6 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -384,6 +384,15 @@ def DeclareActionAttr : OpenACC_Attr<"DeclareAction", "declare_action"> {
   let assemblyFormat = "`<` struct(params) `>`";
 }
 
+// Attribute to store locations for an optional OpenACC loop directive and one
+// or more loops
+def OpenACC_LoopLocAttr : OpenACC_Attr<"LoopLoc", "loop_loc"> {
+  let summary = "Stores the OpenACC loop directive and per-loop source locations";
+  let parameters = (ins OptionalParameter<"::mlir::LocationAttr">:$directive,
+                        ArrayRefParameter<"::mlir::LocationAttr">:$loops);
+  let assemblyFormat = "`<` struct(params) `>`";
+}
+
 // Device type enumeration.
 def OpenACC_DeviceTypeNone      : I32EnumAttrCase<"None", 0, "none">;
 def OpenACC_DeviceTypeStar      : I32EnumAttrCase<"Star", 1, "star">;


From llvmlistbot at llvm.org  Wed Jul  8 08:20:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 08:20:00 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4e6aa0.34fdf6f1.18ac85.21fd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Moazin K. (moazin)

<details>
<summary>Changes</summary>

Introduce an attribute `OpenACCLoopLocAttr` that can be attached to an `acc.loop` operation to record loop locations and directive location separately.

Attach it to `acc.loop` when building the op in flang.

---
Full diff: https://github.com/llvm/llvm-project/pull/208238.diff


3 Files Affected:

- (modified) flang/lib/Lower/OpenACC.cpp (+23-7) 
- (modified) flang/test/Lower/OpenACC/locations.f90 (+19-4) 
- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td (+9) 


``````````diff
diff --git a/flang/lib/Lower/OpenACC.cpp b/flang/lib/Lower/OpenACC.cpp
index e28c31f16b9d9..8c1f071a8d31d 100644
--- a/flang/lib/Lower/OpenACC.cpp
+++ b/flang/lib/Lower/OpenACC.cpp
@@ -1828,7 +1828,6 @@ static void processDoLoopBounds(
         std::pair<Fortran::semantics::SymbolRef, Fortran::semantics::SymbolRef>>
         &localSymPairs) {
   assert(loopsToProcess > 0 && "expect at least one loop");
-  locs.push_back(currentLocation); // Location of the directive
   bool isDoConcurrent = outerDoConstruct.IsDoConcurrent();
 
   if (isDoConcurrent) {
@@ -2105,7 +2104,7 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
                llvm::SmallVector<mlir::Value> &cacheOperands,
                llvm::SmallVector<mlir::Value> &reductionOperands,
                llvm::SmallVector<mlir::Type> &retTy, mlir::Value yieldValue,
-               uint64_t loopsToProcess) {
+               uint64_t loopsToProcess, bool hasDirective) {
   fir::FirOpBuilder &builder = converter.getFirOpBuilder();
 
   llvm::SmallVector<std::pair<mlir::Value, Fortran::semantics::SymbolRef>>
@@ -2151,10 +2150,26 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
   addOperands(operands, operandSegments, firstprivateOperands);
   addOperands(operands, operandSegments, reductionOperands);
 
+  // list of locations to build FusedLoc to be attached to acc loop
+  llvm::SmallVector<mlir::Location> fusedLocs;
+  if (hasDirective)
+    fusedLocs.push_back(currentLocation);
+  fusedLocs.append(locs.begin(), locs.end());
+
+  // list of only loop locations used to build an `mlir::acc::LoopLocAttr` to be
+  // attached to the `FusedLoc` as metadata so that consumers of location
+  // information can separate out the directive's location from the loops'
+  // locations
+  llvm::SmallVector<mlir::LocationAttr> loopLocAttrs(locs.begin(), locs.end());
+  auto loopLocMeta = mlir::acc::LoopLocAttr::get(
+      builder.getContext(),
+      hasDirective ? mlir::LocationAttr(currentLocation) : mlir::LocationAttr{},
+      loopLocAttrs);
+
   auto loopOp = createRegionOp<mlir::acc::LoopOp, mlir::acc::YieldOp>(
-      builder, builder.getFusedLoc(locs), currentLocation, eval, operands,
-      operandSegments, /*outerCombined=*/false, retTy, yieldValue, ivTypes,
-      ivLocs);
+      builder, builder.getFusedLoc(fusedLocs, loopLocMeta), currentLocation,
+      eval, operands, operandSegments, /*outerCombined=*/false, retTy,
+      yieldValue, ivTypes, ivLocs);
   // Ensure the iv symbol is mapped to private iv SSA value for the scope of
   // the loop even if it did not appear explicitly in a PRIVATE clause (if it
   // appeared explicitly in such clause, that is also fine because duplicates
@@ -2419,7 +2434,7 @@ static mlir::acc::LoopOp createLoopOp(
       converter, currentLocation, semanticsContext, stmtCtx, outerDoConstruct,
       eval, privateOperands, dataMap, gangOperands, workerNumOperands,
       vectorOperands, tileOperands, cacheOperands, reductionOperands, retTy,
-      yieldValue, loopsToProcess);
+      yieldValue, loopsToProcess, /*hasDirective=*/true);
 
   if (!gangDeviceTypes.empty())
     loopOp.setGangAttr(builder.getArrayAttr(gangDeviceTypes));
@@ -5355,7 +5370,8 @@ mlir::Operation *Fortran::lower::genOpenACCLoopFromDoConstruct(
       converter, converter.getCurrentLocation(), semanticsContext, stmtCtx,
       doConstruct, eval, privateOperands, dataMap, gangOperands,
       workerNumOperands, vectorOperands, tileOperands, cacheOperands,
-      reductionOperands, retTy, yieldValue, loopsToProcess);
+      reductionOperands, retTy, yieldValue, loopsToProcess,
+      /*hasDirective=*/false);
 
   // Normal do loops which are not annotated with `acc loop` should be
   // left for analysis by marking with `auto`. This is the case even in the case
diff --git a/flang/test/Lower/OpenACC/locations.f90 b/flang/test/Lower/OpenACC/locations.f90
index 46e8df0e9dac4..fe1dc5c918e2d 100644
--- a/flang/test/Lower/OpenACC/locations.f90
+++ b/flang/test/Lower/OpenACC/locations.f90
@@ -53,7 +53,7 @@ subroutine nested_acc_locations(arr1d)
     !CHECK: acc.loop
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":44:11)
-    !CHECK-NEXT: } attributes {{.*}} loc(fused["{{.*}}locations.f90":44:11, "{{.*}}locations.f90":45:5])
+    !CHECK-NEXT: } attributes {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":44:11), loops = loc("{{[^"]*}}locations.f90":45:5)>>["{{[^"]*}}locations.f90":44:11, "{{[^"]*}}locations.f90":45:5])
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":43:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":43:11)
@@ -87,7 +87,7 @@ subroutine combined_directive_locations(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":82:11, "{{.*}}locations.f90":83:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":82:11), loops = loc("{{[^"]*}}locations.f90":83:5)>>["{{[^"]*}}locations.f90":82:11, "{{[^"]*}}locations.f90":83:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":82:11)
   end subroutine
@@ -106,7 +106,7 @@ subroutine if_clause_expr_location(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":99:11, "{{.*}}locations.f90":100:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":99:11), loops = loc("{{[^"]*}}locations.f90":100:5)>>["{{[^"]*}}locations.f90":99:11, "{{[^"]*}}locations.f90":100:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":99:11)
   end subroutine
@@ -169,7 +169,7 @@ subroutine acc_loop_fused_locations(arr)
 
 ! CHECK-LABEL: func.func @_QMacc_locationsPacc_loop_fused_locations
 ! CHECK: acc.loop
-! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused["{{.*}}locations.f90":160:11, "{{.*}}locations.f90":161:5, "{{.*}}locations.f90":162:7, "{{.*}}locations.f90":163:9])
+! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":160:11), loops = loc("{{[^"]*}}locations.f90":161:5), loc("{{[^"]*}}locations.f90":162:7), loc("{{[^"]*}}locations.f90":163:9)>>["{{[^"]*}}locations.f90":160:11, "{{[^"]*}}locations.f90":161:5, "{{[^"]*}}locations.f90":162:7, "{{[^"]*}}locations.f90":163:9])
 
   subroutine data_end_locations(arr)
     real, dimension(10) :: arr
@@ -182,4 +182,19 @@ subroutine data_end_locations(arr)
     !CHECK-LABEL: acc.copyout
     !CHECK-SAME:  loc("{{.*}}locations.f90":181:11)
   end subroutine
+
+  subroutine acc_kernel_with_loop_locations(arr)
+    real, dimension(10) :: arr
+    integer :: i
+
+    !$acc kernels
+    do i = 1, 10
+      arr(i) = arr(i) + 1
+    end do
+    !$acc end kernels
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMacc_locationsPacc_kernel_with_loop_locations
+! CHECK: acc.loop
+! CHECK: } attributes {{.*}} loc(fused<#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>>["{{[^"]*}}locations.f90":191:5])
 end module
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 26a9344fa3a15..070e6688684e6 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -384,6 +384,15 @@ def DeclareActionAttr : OpenACC_Attr<"DeclareAction", "declare_action"> {
   let assemblyFormat = "`<` struct(params) `>`";
 }
 
+// Attribute to store locations for an optional OpenACC loop directive and one
+// or more loops
+def OpenACC_LoopLocAttr : OpenACC_Attr<"LoopLoc", "loop_loc"> {
+  let summary = "Stores the OpenACC loop directive and per-loop source locations";
+  let parameters = (ins OptionalParameter<"::mlir::LocationAttr">:$directive,
+                        ArrayRefParameter<"::mlir::LocationAttr">:$loops);
+  let assemblyFormat = "`<` struct(params) `>`";
+}
+
 // Device type enumeration.
 def OpenACC_DeviceTypeNone      : I32EnumAttrCase<"None", 0, "none">;
 def OpenACC_DeviceTypeStar      : I32EnumAttrCase<"Star", 1, "star">;

``````````

</details>


https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 08:25:36 2026
From: llvmlistbot at llvm.org (Sunil Shrestha)
Date: Wed, 08 Jul 2026 08:25:36 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Support conditional
 lastprivate on host (PR #200086)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200086@github.com>
Message-ID: <6a4e6bf0.449bfb44.172ed9.e88b@mx.google.com>


sshrestha-aa wrote:

@tblah , any more feedback on this?

https://github.com/llvm/llvm-project/pull/200086

From llvmlistbot at llvm.org  Wed Jul  8 08:26:07 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 08 Jul 2026 08:26:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for working with
	acc par dims (PR #208120)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208120@github.com>
Message-ID: <6a4e6c0f.bf38ae28.39239a.e467@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12145

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)


-- Configuring incomplete, errors occurred!
[8675/8695] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/..
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8677/8695] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8678/8695] Linking CXX executable bin/tco
[8679/8695] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)
-- Configuring incomplete, errors occurred!
[8675/8695] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/..
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8677/8695] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8678/8695] Linking CXX executable bin/tco
[8679/8695] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=76.951997

```

</details>

https://github.com/llvm/llvm-project/pull/208120

From llvmlistbot at llvm.org  Wed Jul  8 08:28:06 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 08 Jul 2026 08:28:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for working with
	acc par dims (PR #208120)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208120@github.com>
Message-ID: <6a4e6c86.bd93e731.358318.2835@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12145

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)


-- Configuring incomplete, errors occurred!
[8675/8695] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/..
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8677/8695] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8678/8695] Linking CXX executable bin/tco
[8679/8695] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)
-- Configuring incomplete, errors occurred!
[8675/8695] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/..
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8677/8695] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8678/8695] Linking CXX executable bin/tco
[8679/8695] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=76.951997

```

</details>

https://github.com/llvm/llvm-project/pull/208120

From llvmlistbot at llvm.org  Wed Jul  8 08:29:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 08:29:44 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4e6ce8.a87eb5dc.7d817.1b66@mx.google.com>


https://github.com/jeanPerier approved this pull request.

Looks great, thanks @moazin 

https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 08:31:11 2026
From: llvmlistbot at llvm.org (Philipp Schilk)
Date: Wed, 08 Jul 2026 08:31:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
Message-ID: <llvm.org/llvm/llvm-project/pull/208240@github.com>


https://github.com/schilkp created https://github.com/llvm/llvm-project/pull/208240

I was trying to print some rather large functions (5000+ ops) to a graphivz/dot file, and noticed that the `ViewOpGraph` pass rather slow.

`--view-op-graph` previously always called `Operation::printAsOperand` with the default `OpPrintingFlags` causing a new `AsmState` to be constructed every time, which is extremely inefficient.

This patch instead creates a top-level `AsmState` once for each pass run, and re-uses it for all printing.

This makes printing graphviz/dot files for large graphs *significantly* faster: A single `func.func` with 5000 operations previously took 33 seconds to print, and can now be printed (essentially) instantly :)

---

quick benchmark:

gen.bash:
```bash
#!/bin/bash
# Usage: ./gen.bash <num_ops> <output_file>
num_ops=$1
output_file=$2
{
    echo "func.func @core(%arg0: i32) -> i32 {"
    echo "  %c1 = arith.constant 1 : i32"
    echo "  %0 = arith.addi %arg0, %c1 : i32"
    for ((i=1; i<num_ops; i++)); do
        echo "  %$i = arith.addi %$((i-1)), %c1 : i32"
    done
    echo "  return %$((num_ops-1)) : i32"
    echo "}"
} > "$output_file"
```


```bash
> gen.bash 5000 test_5000.mlir
> # Before:
> time ./build/bin/mlir-opt --view-op-graph ./test_5000.mlir 2> /dev/null -o /dev/null
./build/bin/mlir-opt --view-op-graph ./test_5000.mlir -o out.mlir 2> out.dot  35.15s user 0.06s system 99% cpu 35.279 total
> # After:
> time ./build/bin/mlir-opt --view-op-graph ./test_5000.mlir 2> out.dot -o out.mlir
./build/bin/mlir-opt --view-op-graph ./test_5000.mlir -o out.mlir 2> out.dot  0.03s user 0.04s system 99% cpu 0.073 total
```


>From c33d0d6b10dbc8082d646e9858219207188099c6 Mon Sep 17 00:00:00 2001
From: schilkp <schilk.philipp at gmail.com>
Date: Wed, 8 Jul 2026 16:44:47 +0200
Subject: [PATCH] [ViewOpGraph] Re-use `AsmState` to significantly speed-up
 printing.

`--view-op-graph` previously always called `Operation::printAsOperand`
with the default `OpPrintingFlags` causing a new `AsmState` to be
constructed every time, which is extremly slow.

This patch instead creates a top-level `AsmState` once for each pass
run, and re-uses it for all printing.

This makes printing graphviz/dot files for large graphs *significantly*
faster: A single `func.func` with 5000 operations previously took 33 seconds
to print, and can now be printed (essentially) instantly :)
---
 mlir/lib/Transforms/ViewOpGraph.cpp | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Transforms/ViewOpGraph.cpp b/mlir/lib/Transforms/ViewOpGraph.cpp
index 2d7e40d18efca..26923984ccdf3 100644
--- a/mlir/lib/Transforms/ViewOpGraph.cpp
+++ b/mlir/lib/Transforms/ViewOpGraph.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Transforms/ViewOpGraph.h"
 
+#include "mlir/IR/AsmState.h"
 #include "mlir/IR/Block.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Operation.h"
@@ -108,10 +109,12 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   void runOnOperation() override {
     initColorMapping(*getOperation());
+    asmState.emplace(getOperation(), OpPrintingFlags());
     emitGraph([&]() {
       processOperation(getOperation());
       emitAllEdgeStmts();
     });
+    asmState.reset();
     markAllAnalysesPreserved();
   }
 
@@ -120,7 +123,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
     printControlFlowEdges = true;
     printDataFlowEdges = false;
     initColorMapping(region);
+    asmState.emplace(region.getParentOp(), OpPrintingFlags());
     emitGraph([&]() { processRegion(region); });
+    asmState.reset();
   }
 
 private:
@@ -234,7 +239,7 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   // Print a truncated and escaped MLIR operand to `os`.
   void emitMlirOperand(raw_ostream &os, Value operand) {
-    operand.printAsOperand(os, OpPrintingFlags());
+    operand.printAsOperand(os, *asmState);
   }
 
   /// Append an edge to the list of edges.
@@ -292,9 +297,8 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   std::string getValuePortName(Value operand) {
     // Print value as an operand and omit the leading '%' character.
-    auto str = strFromOs([&](raw_ostream &os) {
-      operand.printAsOperand(os, OpPrintingFlags());
-    });
+    auto str = strFromOs(
+        [&](raw_ostream &os) { operand.printAsOperand(os, *asmState); });
     // Replace % and # with _
     llvm::replace(str, '%', '_');
     llvm::replace(str, '#', '_');
@@ -376,7 +380,7 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
   std::string getLabel(BlockArgument arg) {
     return strFromOs([&](raw_ostream &os) {
       os << "<res" << getValuePortName(arg) << "> ";
-      arg.printAsOperand(os, OpPrintingFlags());
+      arg.printAsOperand(os, *asmState);
       if (printResultTypes) {
         os << " ";
         emitMlirType(os, arg.getType());
@@ -448,6 +452,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   /// Output stream to write DOT file to.
   raw_indented_ostream os;
+  /// Re-usable assembly printer state for efficient printing. Initialized
+  /// on each pass run.
+  std::optional<AsmState> asmState;
   /// A list of edges. For simplicity, should be emitted after all nodes were
   /// emitted.
   std::vector<std::string> edges;


From llvmlistbot at llvm.org  Wed Jul  8 08:31:40 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Wed, 08 Jul 2026 08:31:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e6d5c.0c05e68e.1e82f1.575a@mx.google.com>


https://github.com/smada3 updated https://github.com/llvm/llvm-project/pull/208063

>From 5458e4a6a9d12bab08793042addb0b87cf91117e Mon Sep 17 00:00:00 2001
From: Stefan Mada <smada at nvidia.com>
Date: Tue, 7 Jul 2026 18:20:31 +0000
Subject: [PATCH 1/2] [NVVM][MLIR] Fixed valgrind leak in MMAOp

---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td | 22 +++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 40f7f15b694cb..1f5738589a6a0 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3384,9 +3384,9 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     `intOverflowBehavior` is only relevant when the `multiplicandType` attribute
     is one of `u8, s8, u4, s4`, this attribute describes how overflow is handled
     in the accumulator. When the attribute is `satfinite`, the accumulator values
-    are clamped in the int32 range on overflow. This is the default behavior.
-    Alternatively, accumulator behavior `wrapped` can also be specified, in
-    which case overflow wraps from one end of the range to the other.
+    are clamped in the int32 range on overflow. Alternatively, accumulator
+    behavior `wrapped` can be specified (this is the default), in which case
+    overflow wraps from one end of the range to the other.
 
     `layoutA` and `layoutB` are required and should generally be set to
     `#nvvm.mma_layout<row>` and `#nvvm.mma_layout<col>` respectively, but other
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
+    if (mlir::NVVM::MMAIntOverflowAttr satAttr = op.getIntOverflowBehaviorAttr())
+      intOverflow = satAttr.getValue();
     auto intId = mlir::NVVM::MmaOp::getIntrinsicID(
         $shape.getM(), $shape.getN(), $shape.getK(),
-        $b1Op, $intOverflowBehavior,
+        $b1Op, intOverflow,
         $layoutA, $layoutB,
         *$multiplicandAPtxType,
         *$multiplicandBPtxType,

>From 964544ea817f23d0a055d14fcfc72c5493755475 Mon Sep 17 00:00:00 2001
From: Stefan Mada <smada at nvidia.com>
Date: Wed, 8 Jul 2026 15:31:25 +0000
Subject: [PATCH 2/2] Shortened comment on mmaop valgrind fix

---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f5738589a6a0..627d9a861d780 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3477,17 +3477,10 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
-    // `intOverflowBehavior` is optional; an omitted attribute selects the
-    // default `wrapped` behavior. Resolve it to a fully-initialized value here
-    // so the intrinsic selector always inspects an engaged optional and never
-    // reads an empty optional's uninitialized payload (a benign read the
-    // optimizer can hoist out of the guarding `has_value()` check under -O2).
-    // An empty std::optional still reserves storage for its payload but leaves
-    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
-    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
-    // before the guard discards it. This is not UB -- the read touches validly
-    // allocated storage and its result is never used -- but Valgrind still
-    // reports the load of uninitialized bytes.
+    // `intOverflowBehavior` is optional and defaults to `wrapped` when omitted.
+    // We initialize it to a concrete value so that the intrinsic selector never
+    // speculatively reads an empty optional's uninitialized payload (benign,
+    // but flagged by Valgrind).
     std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
     if (mlir::NVVM::MMAIntOverflowAttr satAttr = op.getIntOverflowBehaviorAttr())
       intOverflow = satAttr.getValue();


From llvmlistbot at llvm.org  Wed Jul  8 08:32:16 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Wed, 08 Jul 2026 08:32:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4e6d80.eb165e5c.1817a5.010a@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
----------------
smada3 wrote:

Yep, shortened it.

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Wed Jul  8 08:32:33 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 08:32:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4e6d91.58bdd12c.19c81.5b80@mx.google.com>


banach-space wrote:

CC @FedericoBruzzone 

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Wed Jul  8 08:37:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 08:37:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208240@github.com>
Message-ID: <6a4e6ec6.ff037943.2e959a.c8c6@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Philipp Schilk (schilkp)

<details>
<summary>Changes</summary>

I was trying to print some rather large functions (5000+ ops) to a graphivz/dot file, and noticed that the `ViewOpGraph` pass rather slow.

`--view-op-graph` previously always called `Operation::printAsOperand` with the default `OpPrintingFlags` causing a new `AsmState` to be constructed every time, which is extremely inefficient.

This patch instead creates a top-level `AsmState` once for each pass run, and re-uses it for all printing.

This makes printing graphviz/dot files for large graphs *significantly* faster: A single `func.func` with 5000 operations previously took 33 seconds to print, and can now be printed (essentially) instantly :)

---

quick benchmark:

gen.bash:
```bash
#!/bin/bash
# Usage: ./gen.bash &lt;num_ops&gt; &lt;output_file&gt;
num_ops=$1
output_file=$2
{
    echo "func.func @<!-- -->core(%arg0: i32) -&gt; i32 {"
    echo "  %c1 = arith.constant 1 : i32"
    echo "  %0 = arith.addi %arg0, %c1 : i32"
    for ((i=1; i&lt;num_ops; i++)); do
        echo "  %$i = arith.addi %$((i-1)), %c1 : i32"
    done
    echo "  return %$((num_ops-1)) : i32"
    echo "}"
} &gt; "$output_file"
```


```bash
&gt; gen.bash 5000 test_5000.mlir
&gt; # Before:
&gt; time ./build/bin/mlir-opt --view-op-graph ./test_5000.mlir 2&gt; /dev/null -o /dev/null
./build/bin/mlir-opt --view-op-graph ./test_5000.mlir -o out.mlir 2&gt; out.dot  35.15s user 0.06s system 99% cpu 35.279 total
&gt; # After:
&gt; time ./build/bin/mlir-opt --view-op-graph ./test_5000.mlir 2&gt; out.dot -o out.mlir
./build/bin/mlir-opt --view-op-graph ./test_5000.mlir -o out.mlir 2&gt; out.dot  0.03s user 0.04s system 99% cpu 0.073 total
```


---
Full diff: https://github.com/llvm/llvm-project/pull/208240.diff


1 Files Affected:

- (modified) mlir/lib/Transforms/ViewOpGraph.cpp (+12-5) 


``````````diff
diff --git a/mlir/lib/Transforms/ViewOpGraph.cpp b/mlir/lib/Transforms/ViewOpGraph.cpp
index 2d7e40d18efca..26923984ccdf3 100644
--- a/mlir/lib/Transforms/ViewOpGraph.cpp
+++ b/mlir/lib/Transforms/ViewOpGraph.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Transforms/ViewOpGraph.h"
 
+#include "mlir/IR/AsmState.h"
 #include "mlir/IR/Block.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Operation.h"
@@ -108,10 +109,12 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   void runOnOperation() override {
     initColorMapping(*getOperation());
+    asmState.emplace(getOperation(), OpPrintingFlags());
     emitGraph([&]() {
       processOperation(getOperation());
       emitAllEdgeStmts();
     });
+    asmState.reset();
     markAllAnalysesPreserved();
   }
 
@@ -120,7 +123,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
     printControlFlowEdges = true;
     printDataFlowEdges = false;
     initColorMapping(region);
+    asmState.emplace(region.getParentOp(), OpPrintingFlags());
     emitGraph([&]() { processRegion(region); });
+    asmState.reset();
   }
 
 private:
@@ -234,7 +239,7 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   // Print a truncated and escaped MLIR operand to `os`.
   void emitMlirOperand(raw_ostream &os, Value operand) {
-    operand.printAsOperand(os, OpPrintingFlags());
+    operand.printAsOperand(os, *asmState);
   }
 
   /// Append an edge to the list of edges.
@@ -292,9 +297,8 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   std::string getValuePortName(Value operand) {
     // Print value as an operand and omit the leading '%' character.
-    auto str = strFromOs([&](raw_ostream &os) {
-      operand.printAsOperand(os, OpPrintingFlags());
-    });
+    auto str = strFromOs(
+        [&](raw_ostream &os) { operand.printAsOperand(os, *asmState); });
     // Replace % and # with _
     llvm::replace(str, '%', '_');
     llvm::replace(str, '#', '_');
@@ -376,7 +380,7 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
   std::string getLabel(BlockArgument arg) {
     return strFromOs([&](raw_ostream &os) {
       os << "<res" << getValuePortName(arg) << "> ";
-      arg.printAsOperand(os, OpPrintingFlags());
+      arg.printAsOperand(os, *asmState);
       if (printResultTypes) {
         os << " ";
         emitMlirType(os, arg.getType());
@@ -448,6 +452,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   /// Output stream to write DOT file to.
   raw_indented_ostream os;
+  /// Re-usable assembly printer state for efficient printing. Initialized
+  /// on each pass run.
+  std::optional<AsmState> asmState;
   /// A list of edges. For simplicity, should be emitted after all nodes were
   /// emitted.
   std::vector<std::string> edges;

``````````

</details>


https://github.com/llvm/llvm-project/pull/208240

From llvmlistbot at llvm.org  Wed Jul  8 08:45:41 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 08:45:41 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4e70a5.e55dc341.1f19bd.4f17@mx.google.com>


================
@@ -182,4 +182,19 @@ subroutine data_end_locations(arr)
     !CHECK-LABEL: acc.copyout
     !CHECK-SAME:  loc("{{.*}}locations.f90":181:11)
   end subroutine
+
+  subroutine acc_kernel_with_loop_locations(arr)
+    real, dimension(10) :: arr
+    integer :: i
+
+    !$acc kernels
+    do i = 1, 10
+      arr(i) = arr(i) + 1
+    end do
+    !$acc end kernels
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMacc_locationsPacc_kernel_with_loop_locations
+! CHECK: acc.loop
+! CHECK: } attributes {{.*}} loc(fused<#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>>["{{[^"]*}}locations.f90":191:5])
----------------
razvanlupusoru wrote:

Worth a CHECK-NOT check that it does not have the "directive" location?

https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 08:45:48 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 08:45:48 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4e70ac.da8bafd3.14cd50.0d10@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Thank you.

https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 08:49:18 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Wed, 08 Jul 2026 08:49:18 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a4e717e.cb30c3e4.3ce559.82e6@mx.google.com>


dzhidzhoev wrote:

I had to make a small change to fix debug-info-correlate-bitmap.ll and, more generally, the InstrProfiling pass.

On the current main branch, when finalizeSubprogram() is called, it overwrites the list of retained nodes for the subprogram. This is fine when the subprogram is newly created. However, if the subprogram already exists in the module and DIBuilder is used to modify it (for example, to add a local entity that should be included in the retained nodes), finalizeSubprogram() discards the previously retained nodes.

The InstrProfiling pass adds two global variables to the same pre-existing subprogram using two different DIBuilder instances. After the second global variable is added, the first one is removed from the retained nodes list.

To fix this, I modified finalizeSubprogram() so that it updates the existing retained nodes list instead of overwriting it.

@dwblaikie, @OCHyams , @jmorse could you please take a look at this change?

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Wed Jul  8 09:00:03 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 08 Jul 2026 09:00:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] add consistent stride
 verification to masked load/store and gather/scatter ops (PR #205869)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205869@github.com>
Message-ID: <6a4e7403.a87eb5dc.7d817.94b5@mx.google.com>


FedericoBruzzone wrote:

@banach-space I think we can merge this PR :D

https://github.com/llvm/llvm-project/pull/205869

From llvmlistbot at llvm.org  Wed Jul  8 09:01:11 2026
From: llvmlistbot at llvm.org (Moazin K.)
Date: Wed, 08 Jul 2026 09:01:11 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4e7447.faf16b37.2f0e1d.e442@mx.google.com>


================
@@ -182,4 +182,19 @@ subroutine data_end_locations(arr)
     !CHECK-LABEL: acc.copyout
     !CHECK-SAME:  loc("{{.*}}locations.f90":181:11)
   end subroutine
+
+  subroutine acc_kernel_with_loop_locations(arr)
+    real, dimension(10) :: arr
+    integer :: i
+
+    !$acc kernels
+    do i = 1, 10
+      arr(i) = arr(i) + 1
+    end do
+    !$acc end kernels
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMacc_locationsPacc_kernel_with_loop_locations
+! CHECK: acc.loop
+! CHECK: } attributes {{.*}} loc(fused<#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>>["{{[^"]*}}locations.f90":191:5])
----------------
moazin wrote:

I could add a separate `CHECK-NOT` but I think this `CHECK` covers that too:
```mlir
#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>
```
if there is a `directive = ` along with the `loops = `, it'll fail this check. 

https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 09:01:36 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Wed, 08 Jul 2026 09:01:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4e7460.92d2873c.1ab170.ed8a@mx.google.com>


FedericoBruzzone wrote:

@egebeysel @AGindinson I think we can merge this PR. 
But that's up to you :D

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Wed Jul  8 09:06:28 2026
From: llvmlistbot at llvm.org (Philipp Schilk)
Date: Wed, 08 Jul 2026 09:06:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208240@github.com>
Message-ID: <6a4e7584.92d2873c.1ab170.6c41@mx.google.com>


schilkp wrote:

@jpienaar 
@joker-eph 
@weiweichen 

If one of you could have a look, that would be great :)

https://github.com/llvm/llvm-project/pull/208240

From llvmlistbot at llvm.org  Wed Jul  8 09:13:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:13:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 33ef532 - [mlir][linalg] Fix mask rank for
 masked contiguous `tensor.extract` (#206207)
Message-ID: <6a4e7729.58bdd12c.19c81.338a@mx.google.com>


Author: pstarkcdpr
Date: 2026-07-08T17:13:24+01:00
New Revision: 33ef532c79b321cf22232ba0054b36d259c62079

URL: https://github.com/llvm/llvm-project/commit/33ef532c79b321cf22232ba0054b36d259c62079
DIFF: https://github.com/llvm/llvm-project/commit/33ef532c79b321cf22232ba0054b36d259c62079.diff

LOG: [mlir][linalg] Fix mask rank for masked contiguous `tensor.extract` (#206207)

### Summary

Note: This fix was made mostly by Claude based on a failure case in
IREE. It addresses issue
https://github.com/llvm/llvm-project/issues/206209

When `vectorizeTensorExtract` lowers a `tensor.extract` recognized as a
*contiguous load*, it builds a `vector.transfer_read` whose permutation
map broadcasts the leading iteration dims and only reads the trailing
`min(dstRank, srcRank)` dims of the source. Until now this read was
returned unmasked and masked later by the generic path, which applies a
**full iteration-space identity mask**. When the source rank is smaller
than the loop nest, that mask is over-ranked relative to the
(rank-reduced) read — e.g. a `vector<1x4xi1>` mask on a read whose
inferred mask type is `vector<4xi1>` — and the op fails verification:

```
'vector.mask' op expects a 'vector<4xi1>' mask for the maskable operation
```

(or, after the masked read is rank-reduced by canonicalization in a full
pipeline, `'vector.transfer_read' op inferred mask type ... don't
match`).

### Root cause

The three `tensor.extract` lowering paths in `vectorizeTensorExtract`
handle masking inconsistently:

- **gather** masks itself — correct, its result is full-rank;
- **scalar broadcast** masks itself with a rank-1 mask, explicitly
noting that the generic identity-map masking "wouldn't be valid here";
- **contiguous load** does neither and relies on the generic path, which
masks over all loop dims. That is only correct when the read is
full-rank (`srcRank >= numLoops`); when `srcRank < numLoops` the read
broadcasts its leading dims and needs a rank-reduced mask.

Existing tests only covered the full-rank case (e.g. a
`tensor<80x16xf32>` source in a 2-D nest), so the rank-reducing case was
never exercised.

### Fix

Mask the contiguous read in place, mirroring the scalar-broadcast path,
using a masking map that projects the iteration space onto exactly the
trailing `min(dstRank, srcRank)` dims that are read:

```cpp
int64_t numReadDims = std::min(dstRank, srcRank);
auto maskingMap = AffineMap::getMinorIdentityMap(
    linalgOp.getNumLoops(), numReadDims, rewriter.getContext());
Operation *maskedReadOp =
    state.maskOperation(rewriter, transferReadOp, linalgOp, maskingMap);
```

This is **behavior-preserving for the existing full-rank case**: when
`min(dstRank, srcRank) == numLoops`, `getMinorIdentityMap` collapses to
the full identity map, producing the same mask (same `activeMaskCache`
key) and identical
IR. Only the previously-broken rank-reducing case changes.

### Testing

- New regression test `@masked_contiguous_extract_rank_reducing_mask` in
`mlir/test/Dialect/Linalg/vectorization/extract.mlir` (1-D source inside
a 2-D loop nest). It fails verification before this change and checks
the rank-reduced mask after it.
- `mlir/test/Dialect/Linalg` (164 tests) and `mlir/test/Dialect/Vector`
+ `mlir/test/Dialect/Linalg/vectorization` (113 tests) all pass.

### Notes for reviewers

- The change is localized to the contiguous-load branch of
`vectorizeTensorExtract`; the gather and scalar-broadcast branches are
untouched.
- No new mask is created in the full-rank case — the existing cached
iteration-space mask is reused, so there is no codegen/IR churn for
current callers.

---------

Signed-off-by: Paul Stark <paul.stark at cdprojektred.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply at anthropic.com>

Added: 
    

Modified: 
    mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
    mlir/test/Dialect/Linalg/vectorization/extract.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
index b57e66a1c3580..21ca3108efcd6 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
@@ -1299,9 +1299,21 @@ vectorizeTensorExtract(RewriterBase &rewriter, VectorizationState &state,
       rewriter, loc, resultType, extractOp.getTensor(), transferReadIdxs,
       /*padding=*/std::nullopt, permutationMap, inBounds);
 
+  // Mask this contiguous xfer_read here rather than relying on the generic
+  // path (the generic path assumes an identity masking map over all the loop
+  // dims, which wouldn't be valid here). A contiguous load only reads the
+  // trailing `min(dstRank, srcRank)` dims of the iteration space - the leading
+  // dims are broadcast via `permutationMap` above - so its inferred mask is
+  // rank-reduced. Build a masking map that projects the iteration space onto
+  // exactly those trailing dims so the created mask matches the xfer_read.
+  int64_t numReadDims = std::min(dstRank, srcRank);
+  auto maskingMap = AffineMap::getMinorIdentityMap(
+      linalgOp.getNumLoops(), numReadDims, rewriter.getContext());
+  Operation *maskedReadOp =
+      state.maskOperation(rewriter, transferReadOp, linalgOp, maskingMap);
+
   LDBG() << "Vectorised as contiguous load: " << extractOp;
-  return VectorizationHookResult{VectorizationHookStatus::NewOp,
-                                 transferReadOp};
+  return VectorizationHookResult{VectorizationHookStatus::NewOp, maskedReadOp};
 }
 
 /// Emit reduction operations if the shapes of the value to reduce is 
diff erent

diff  --git a/mlir/test/Dialect/Linalg/vectorization/extract.mlir b/mlir/test/Dialect/Linalg/vectorization/extract.mlir
index 76ac4b8398069..fc0a11ef9810e 100644
--- a/mlir/test/Dialect/Linalg/vectorization/extract.mlir
+++ b/mlir/test/Dialect/Linalg/vectorization/extract.mlir
@@ -477,3 +477,52 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+// A contiguous `tensor.extract` from a source whose rank is *smaller* than the
+// iteration space (here: a 1-D source inside a 2-D loop nest). The contiguous
+// `vector.transfer_read` only reads the trailing (contiguous) dim - the leading
+// dim is broadcast via the permutation map - so its mask must be rank-reduced
+// (`vector<4xi1>`) rather than the full iteration-space mask (`vector<1x4xi1>`).
+// Regression test: previously the full iteration mask was attached, producing
+// an invalid `vector.transfer_read` ("inferred mask type ... don't match").
+
+func.func @masked_contiguous_extract_rank_reducing_mask(
+    %src: tensor<16xf32>,
+    %output : tensor<1x3xf32>,
+    %idx: index) -> tensor<1x3xf32> {
+  %1 = linalg.generic {
+    indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
+    iterator_types = ["parallel", "parallel"]
+  } outs(%output : tensor<1x3xf32>) {
+  ^bb0(%out: f32):
+    %2 = linalg.index 1 : index
+    %3 = affine.apply affine_map<(d0, d1) -> (d0 + d1)>(%2, %idx)
+    %extracted = tensor.extract %src[%3] : tensor<16xf32>
+    linalg.yield %extracted : f32
+  } -> tensor<1x3xf32>
+  return %1 : tensor<1x3xf32>
+}
+
+// CHECK-LABEL:   func.func @masked_contiguous_extract_rank_reducing_mask
+// CHECK-SAME:      %[[SRC:.*]]: tensor<16xf32>,
+// CHECK-SAME:      %[[OUTPUT:.*]]: tensor<1x3xf32>,
+// CHECK-SAME:      %[[IDX_IN:.*]]: index) -> tensor<1x3xf32> {
+
+/// Full iteration-space mask (used for the output read/write).
+// CHECK:           %[[MASK_2D:.*]] = vector.create_mask {{.*}} : vector<1x4xi1>
+
+/// The contiguous read of the 1-D source uses a rank-reduced mask.
+// CHECK:           %[[MASK_1D:.*]] = vector.create_mask {{.*}} : vector<4xi1>
+// CHECK:           vector.mask %[[MASK_1D]] {
+// CHECK-SAME:          vector.transfer_read %[[SRC]]{{.*}} {in_bounds = [true, true], permutation_map = {{.*}}} : tensor<16xf32>, vector<1x4xf32>
+// CHECK-SAME:      } : vector<4xi1> -> vector<1x4xf32>
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+     %0 = transform.structured.match ops{["linalg.generic"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+     transform.structured.vectorize %0 vector_sizes [1, 4] {vectorize_nd_extract} : !transform.any_op
+     transform.yield
+   }
+}


        

From llvmlistbot at llvm.org  Wed Jul  8 09:13:32 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 09:13:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a4e772c.bd93e731.358318.e0b9@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Wed Jul  8 09:13:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:13:43 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] [NFC] Fix
 spelling of definition in comments (PR #208138)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208138@github.com>
Message-ID: <6a4e7737.30759430.24d50c.d1b1@mx.google.com>


divedb wrote:

I have read the LLVM contribution policies, including the LLVM AI Tool Use Policy. No AI tools were used for this change.

https://github.com/llvm/llvm-project/pull/208138

From llvmlistbot at llvm.org  Wed Jul  8 09:13:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:13:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Fix mask rank for masked
 contiguous `tensor.extract` (PR #206207)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206207@github.com>
Message-ID: <6a4e773f.abb10101.3d0bf8.2df6@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@pstarkcdpr Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/206207

From llvmlistbot at llvm.org  Wed Jul  8 09:13:56 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 09:13:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add acc.on_device op (PR #208096)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208096@github.com>
Message-ID: <6a4e7744.e9d484b9.15e549.30ce@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Thank you.

https://github.com/llvm/llvm-project/pull/208096

From llvmlistbot at llvm.org  Wed Jul  8 09:18:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:22 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e784e.2020f6c2.243e78.da8f@mx.google.com>


https://github.com/agozillon commented:

A few more questions about some code segments and a suggestion to verify some recent work from Sergio doesn't clash and then we're good I think! Thank you very much for your hard work looking into this, it's greatly appreciated :-)

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:18:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:22 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e784e.449bfb44.172ed9.57b3@mx.google.com>


https://github.com/agozillon edited https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:18:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:22 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e784e.eb165e5c.1817a5.f5ef@mx.google.com>


================
@@ -6565,7 +6571,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy)
+        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
----------------
agozillon wrote:

If my assumptions in the comment above are correct (which they might not be, so I would love to see the edge-case we are working with here to understand a bit better) then this might be a little redundant, as an attach map will always come tied with ref_ptr or ref_ptee I think (or both), and in the cases where attach map is not true, we would pass through in any case, so we basically pass through in all cases if I'm thinking about this correctly. I think we might end up with the same interaction as if we just left it isPtrTy, but I would nudge towards not special casing the load here unless absolutely necessary as it could cause the attach maps to become out of synch with the regular base pointer/descriptor maps or have weird bound offsets applied, but it's unlikely I think, regardless might be worth checking if you absolutely need this casing still  :-) 

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:18:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:22 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e784e.92d2873c.1ab170.acf9@mx.google.com>


================
@@ -7678,6 +7688,35 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
           convertToCaptureClauseKind(attribute.getCaptureClause().getValue());
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
+      llvm::StringRef entryMangledName = mangledName;
+      llvm::Constant *entryAddr = llvm::cast<llvm::Constant>(gVal);
----------------
agozillon wrote:

Changes are all good with me then! But it might be worth checking this PR: https://github.com/llvm/llvm-project/pull/208188 and following up with Sergio, he's also doing some linkage related stuff in this function right now, so his or your changes might help each other or more likely than not they're different edge cases that will need to live in harmony (just a rebase on which ever lands first :-) )! 

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:18:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e7850.f8b79514.1d52d0.c5c5@mx.google.com>


================
@@ -1612,7 +1612,9 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (!bitEnumContainsAll(op.getMapType(),
----------------
agozillon wrote:

Probably a silly question, but do we still need this check for the attach map flag? I don't think we apply this outside of this pass, if I remember correctly at least... which I might not after being away for a few weeks :-) Is there another case we'd need this for, do we perhaps trigger this by generating an attach map inside of here and then re-processing it? Happy to keep it if it's necessary, just would love to know the reason so I can watch out for it in the future!

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:18:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:18:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a4e7850.492e23e4.157d47.6914@mx.google.com>


================
@@ -5719,16 +5719,18 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
+    bool isAttachMap =
+        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
+    Value pointeePtr = isRefPtrOrPteeMapWithAttach
----------------
agozillon wrote:

I'm not entirely sure we want to do this, would it be possible to print the map entry list from the trace you're aiming for here and the one you're working from without these changes (or pseudo code it, whatever is easiest)?

I think, in this case, the "isAttachMap ?" true segment is perhaps redundant as I think whenever isRefPtrOrPteeMapWithAttach is true is also the only case where isAttachMap will be true, I don't believe (but I could be wrong) we ever emit attach without ref_ptr/ptee/ref_ptr_ptee and if someone has explicitly specified attach_never we just drop the attach map entirely. I think this means even if we fall into the else case for isRefPtrOrrPteeMapWithAttach, we have no attach map so we default back to the offloadPtr which is what the old code that we're removing at line 5728 used to do. So, perhaps the original variation might work for your case now, unsure though!

The original logic (annoyingly complex because of myself, so my apologies), basically just selects between the descriptor address and the base pointer address (or defaults to the old logic) as the attach map is a special case where we basically add the descriptor address as the varPtr and the data's base pointer address as the varPtrPtr so we have access to them both, as an attach map entry expects both so that they can be glued together by the runtime. The caveat of this is that if we offset the base pointer via bounds specification we have to replicate it.

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Wed Jul  8 09:20:09 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 09:20:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a4e78b9.af6ba3ed.36d9db.395c@mx.google.com>


================
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
----------------
banach-space wrote:

Please add a negative test.

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Wed Jul  8 09:20:09 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 09:20:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a4e78b9.faf16b37.2f0e1d.e0da@mx.google.com>


================
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
+
+      ArrayRef<int64_t> srcSub =
+          sourceShape.slice(g.srcBegin, g.srcEnd - g.srcBegin);
+      int64_t groupTargetElements = ShapedType::getNumElements(tgtSub);
+      std::optional<SmallVector<int64_t>> groupExtract =
+          calculateSourceExtractShape(srcSub, groupTargetElements);
+      if (!groupExtract)
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "cannot extract the target number of elements "
+                         "contiguously from a source reassociation group");
----------------
banach-space wrote:

Please add a negative test.

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Wed Jul  8 09:23:24 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 09:23:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a4e797c.cb30c3e4.3ce559.bcd4@mx.google.com>


banach-space wrote:

> @banach-space does my explanation make sense to you? For HW target which has the issue (extract_strided_slice non-contiguous high-d vector lowering to complex ops like shuffling), it should choose proper tile size avoiding it. But vector unrolling should support the case which can be properly use vector.extract_strided_slice and lower to NOP, and this PR exactly does so.

Yes, sorry for the delay replying!

Your reply addresses my concerns. It would be good to add comments in code to re-enforce these assumptions/guidelines, i.e. this unrolling should not be used by targets for which `tensor.extract_strided_slice` is a concern and `vector.shape_cast` should be treated as NOP.

Otherwise LGTM

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Wed Jul  8 09:49:16 2026
From: llvmlistbot at llvm.org (Sunil Shrestha)
Date: Wed, 08 Jul 2026 09:49:16 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a4e7f8c.fdc477d6.23591.1d42@mx.google.com>


https://github.com/sshrestha-aa updated https://github.com/llvm/llvm-project/pull/203959

>From d355f314c8f325671d739c84b892dc8153b79639 Mon Sep 17 00:00:00 2001
From: Sunil Shrestha <sunil.shrestha at hpe.com>
Date: Mon, 1 Jun 2026 17:48:49 -0500
Subject: [PATCH 1/2] [Flang][OpenMP] Add support for interop construct

This adds support for the OpenMP interop construct, which enables
interoperability with foreign runtime environments (e.g., CUDA, HIP).
The init, use, and destroy action clauses are lowered from the Fortran
parse tree to MLIR interop ops and translated to the appropriate runtime
calls. The omp_lib module is extended with interop query API interfaces,
property constants, and return code constants. A semantics fix relaxes
the depend/targetsync constraint for use/destroy directives where the
interop type was established at a separate init site. Unsupported depend
clauses and prefer_type selector-list forms emit explicit diagnostics
instead of being silently dropped.

Assisted-by: Claude Opus 4.6
---
 flang/lib/Lower/OpenMP/Clauses.cpp            |  50 +++++-
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 142 +++++++++++++++++-
 flang/lib/Semantics/check-omp-structure.cpp   |   7 +-
 .../Lower/OpenMP/Todo/interop-construct.f90   |   9 --
 flang/test/Lower/OpenMP/interop.f90           | 114 ++++++++++++++
 .../mlir/Dialect/OpenMP/OpenMPAttrDefs.td     |  10 ++
 .../mlir/Dialect/OpenMP/OpenMPEnums.td        |  18 +++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 112 ++++++++++++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |  40 +++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 101 +++++++++++++
 .../Target/LLVMIR/openmp-interop-depend.mlir  |   7 +
 mlir/test/Target/LLVMIR/openmp-interop.mlir   |  71 +++++++++
 openmp/module/omp_lib.F90.var                 |  94 ++++++++++++
 13 files changed, 761 insertions(+), 14 deletions(-)
 delete mode 100644 flang/test/Lower/OpenMP/Todo/interop-construct.f90
 create mode 100644 flang/test/Lower/OpenMP/interop.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-interop.mlir

diff --git a/flang/lib/Lower/OpenMP/Clauses.cpp b/flang/lib/Lower/OpenMP/Clauses.cpp
index 03852dc9e7d74..49cf6e5556445 100644
--- a/flang/lib/Lower/OpenMP/Clauses.cpp
+++ b/flang/lib/Lower/OpenMP/Clauses.cpp
@@ -1030,8 +1030,54 @@ Indirect make(const parser::OmpClause::Indirect &inp,
 
 Init make(const parser::OmpClause::Init &inp,
           semantics::SemanticsContext &semaCtx) {
-  // inp -> empty
-  llvm_unreachable("Empty: init");
+  // OmpInitClause has modifiers: OmpPreferType, OmpInteropType,
+  //   OmpDepinfoModifier, and an OmpObject (the interop variable).
+  auto &mods = semantics::OmpGetModifiers(inp.v);
+
+  // Extract interop types (repeatable modifier).
+  Init::InteropTypes interopTypes;
+  for (auto *interopType :
+       semantics::OmpGetRepeatableModifier<parser::OmpInteropType>(mods)) {
+    switch (interopType->v) {
+    case parser::OmpInteropType::Value::Target:
+      interopTypes.push_back(Init::InteropType::Target);
+      break;
+    case parser::OmpInteropType::Value::Targetsync:
+      interopTypes.push_back(Init::InteropType::Targetsync);
+      break;
+    }
+  }
+
+  // Extract prefer_type (optional modifier).
+  std::optional<Init::InteropPreference> interopPreference;
+  if (auto *preferType =
+          semantics::OmpGetUniqueModifier<parser::OmpPreferType>(mods)) {
+    Init::InteropPreference prefs;
+    for (auto &prefSpec : preferType->v) {
+      common::visit(
+          common::visitors{
+              [&](const parser::OmpPreferenceSpecification::
+                      ForeignRuntimeIdentifier &fri) {
+                prefs.push_back(makeExpr(fri.value(), semaCtx));
+              },
+              [&](const std::list<parser::OmpPreferenceSelector> &selectors) {
+                // TODO: Handle preference-selector lists.
+                if (!selectors.empty())
+                  llvm::report_fatal_error(
+                      "not yet implemented: prefer_type with "
+                      "preference-selector lists in interop init clause");
+              },
+          },
+          prefSpec.u);
+    }
+    if (!prefs.empty())
+      interopPreference = std::move(prefs);
+  }
+
+  auto &interopVar = std::get<parser::OmpObject>(inp.v.t);
+  return Init{{/*InteropPreference=*/std::move(interopPreference),
+               /*InteropTypes=*/std::move(interopTypes),
+               /*InteropVar=*/makeObject(interopVar, semaCtx)}};
 }
 
 Initializer make(const parser::OmpClause::Initializer &inp,
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 1ed79b077d57a..47a70efeb293a 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -19,6 +19,7 @@
 #include "Utils.h"
 #include "flang/Common/idioms.h"
 #include "flang/Evaluate/expression.h"
+#include "flang/Evaluate/fold.h"
 #include "flang/Evaluate/tools.h"
 #include "flang/Evaluate/type.h"
 #include "flang/Lower/Bridge.h"
@@ -53,6 +54,7 @@
 #include "mlir/Support/StateStack.h"
 #include "mlir/Transforms/RegionUtils.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringSwitch.h"
 
 using namespace Fortran::lower::omp;
 using namespace Fortran::common::openmp;
@@ -5107,8 +5109,144 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
                    const parser::OpenMPInteropConstruct &interopConstruct) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPInteropConstruct");
+  if (semaCtx.langOptions().OpenMPSimd)
+    return;
+
+  fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
+  mlir::Location loc = converter.genLocation(interopConstruct.source);
+  mlir::MLIRContext *context = firOpBuilder.getContext();
+
+  List<Clause> clauses = makeClauses(interopConstruct.v.Clauses(), semaCtx);
+
+  // Process shared clauses (depend, nowait, device) that apply to all
+  // action clauses.
+  lower::StatementContext stmtCtx;
+  mlir::omp::DependClauseOps dependOps;
+  mlir::omp::NowaitClauseOps nowaitOps;
+  mlir::Value deviceValue;
+
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processDepend(symTable, stmtCtx, dependOps);
+  cp.processNowait(nowaitOps);
+
+  // Process device clause manually.
+  for (auto &clause : clauses) {
+    if (auto *deviceClause = std::get_if<clause::Device>(&clause.u)) {
+      const auto &deviceExpr = std::get<SomeExpr>(deviceClause->t);
+      deviceValue = fir::getBase(converter.genExprValue(deviceExpr, stmtCtx));
+    }
+  }
+
+  // Helper to get the address of an interop variable from an Object.
+  auto getInteropVarAddr = [&](const Object &object) -> mlir::Value {
+    const semantics::Symbol *sym = object.sym();
+    assert(sym && "interop variable must have a symbol");
+    mlir::Value addr = converter.getSymbolAddress(*sym);
+    assert(addr && "interop variable must have a valid address");
+    return addr;
+  };
+
+  // Process action clauses: init, destroy, use.
+  for (auto &clause : clauses) {
+    if (auto *initClause = std::get_if<clause::Init>(&clause.u)) {
+      auto &interopVar = std::get<clause::Init::InteropVar>(initClause->t);
+      auto &interopTypes = std::get<clause::Init::InteropTypes>(initClause->t);
+      auto &interopPref =
+          std::get<std::optional<clause::Init::InteropPreference>>(
+              initClause->t);
+
+      mlir::Value interopVarAddr = getInteropVarAddr(interopVar);
+
+      // Convert interop types to MLIR attributes.
+      llvm::SmallVector<mlir::Attribute> interopTypeAttrs;
+      for (auto itype : interopTypes) {
+        mlir::omp::InteropType mlirType;
+        switch (itype) {
+        case clause::Init::InteropType::Target:
+          mlirType = mlir::omp::InteropType::target;
+          break;
+        case clause::Init::InteropType::Targetsync:
+          mlirType = mlir::omp::InteropType::targetsync;
+          break;
+        }
+        interopTypeAttrs.push_back(
+            mlir::omp::InteropTypeAttr::get(context, mlirType));
+      }
+      auto interopTypesAttr = mlir::ArrayAttr::get(context, interopTypeAttrs);
+
+      // Convert prefer_type to I64ArrayAttr if present.
+      mlir::ArrayAttr preferTypeAttr;
+      if (interopPref) {
+        llvm::SmallVector<int64_t> prefValues;
+        for (auto &pref : *interopPref) {
+          // Try to evaluate as a constant integer (e.g., omp_ifr_cuda).
+          if (auto constVal = evaluate::ToInt64(pref)) {
+            prefValues.push_back(*constVal);
+          } else if (auto str =
+                         evaluate::GetScalarConstantValue<evaluate::Ascii>(
+                             pref)) {
+            // Map standard OpenMP foreign-runtime identifier strings to
+            // their well-known integer values (OpenMP 5.1, Table 22.2).
+            auto frId = llvm::StringSwitch<std::optional<int64_t>>(
+                            llvm::StringRef(*str).lower())
+                            .Case("cuda", 1)
+                            .Case("cuda_driver", 2)
+                            .Case("opencl", 3)
+                            .Case("sycl", 4)
+                            .Case("hip", 5)
+                            .Case("level_zero", 6)
+                            .Case("hsa", 7)
+                            .Default(std::nullopt);
+            if (frId)
+              prefValues.push_back(*frId);
+            else
+              TODO(loc, "unknown foreign-runtime identifier in "
+                        "prefer_type");
+          } else {
+            TODO(loc, "non-constant prefer_type value in interop init");
+          }
+        }
+        if (!prefValues.empty())
+          preferTypeAttr = firOpBuilder.getI64ArrayAttr(prefValues);
+      }
+
+      mlir::omp::InteropInitOp::create(
+          firOpBuilder, loc, interopVarAddr, interopTypesAttr, preferTypeAttr,
+          deviceValue,
+          dependOps.dependKinds.empty()
+              ? nullptr
+              : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+          dependOps.dependVars,
+          /*depend_iterated_kinds=*/nullptr,
+          /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+
+    } else if (auto *destroyClause = std::get_if<clause::Destroy>(&clause.u)) {
+      assert(destroyClause->v &&
+             "destroy clause must have an interop variable");
+      if (destroyClause->v) {
+        mlir::Value interopVarAddr = getInteropVarAddr(*destroyClause->v);
+        mlir::omp::InteropDestroyOp::create(
+            firOpBuilder, loc, interopVarAddr, deviceValue,
+            dependOps.dependKinds.empty()
+                ? nullptr
+                : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+            dependOps.dependVars,
+            /*depend_iterated_kinds=*/nullptr,
+            /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+      }
+
+    } else if (auto *useClause = std::get_if<clause::Use>(&clause.u)) {
+      mlir::Value interopVarAddr = getInteropVarAddr(useClause->v);
+      mlir::omp::InteropUseOp::create(
+          firOpBuilder, loc, interopVarAddr, deviceValue,
+          dependOps.dependKinds.empty()
+              ? nullptr
+              : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+          dependOps.dependVars,
+          /*depend_iterated_kinds=*/nullptr,
+          /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+    }
+  }
 }
 
 static void
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 7c531ae0046ae..333a71705b5b7 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -5872,7 +5872,12 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     context_.Say(GetContext().directiveSource,
         "Each interop-type may be specified at most once."_err_en_US);
   }
-  if (isDependClauseOccurred && !targetSyncCount) {
+  // Only enforce the depend/targetsync constraint when an init clause is
+  // present on this directive. For use/destroy-only directives, the interop
+  // object's type was established at its init site, which may be in a
+  // different statement — we cannot verify it statically here.
+  bool hasInitClause = (targetCount > 0 || targetSyncCount > 0);
+  if (isDependClauseOccurred && hasInitClause && !targetSyncCount) {
     context_.Say(GetContext().directiveSource,
         "A DEPEND clause can only appear on the directive if the interop-type includes TARGETSYNC"_err_en_US);
   }
diff --git a/flang/test/Lower/OpenMP/Todo/interop-construct.f90 b/flang/test/Lower/OpenMP/Todo/interop-construct.f90
deleted file mode 100644
index 4f69667863990..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/interop-construct.f90
+++ /dev/null
@@ -1,9 +0,0 @@
-! REQUIRES: openmp_runtime
-! RUN: %not_todo_cmd %flang_fc1 -emit-llvm %openmp_flags -fopenmp-version=52 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPInteropConstruct
-program interop_test
-  use omp_lib
-  integer(omp_interop_kind) :: obj
-  !$omp interop init(targetsync,target: obj)
-end program interop_test
diff --git a/flang/test/Lower/OpenMP/interop.f90 b/flang/test/Lower/OpenMP/interop.f90
new file mode 100644
index 0000000000000..85ccfdbbef19d
--- /dev/null
+++ b/flang/test/Lower/OpenMP/interop.f90
@@ -0,0 +1,114 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+
+!===============================================================================
+! Interop Init — target
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init(
+!CHECK-SAME:    %[[ARG:.*]]: !fir.ref<i64> {fir.bindc_name = "obj"})
+!CHECK:         %[[DECL:.*]]:2 = hlfir.declare %[[ARG]]
+!CHECK:         omp.interop.init %[[DECL]]#0 : !fir.ref<i64> interop_types([#omp<interop_type(target)>])
+subroutine test_interop_init(obj)
+  integer(8) :: obj
+  !$omp interop init(target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — targetsync
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_targetsync(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>])
+subroutine test_interop_init_targetsync(obj)
+  integer(8) :: obj
+  !$omp interop init(targetsync: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — targetsync, target (both)
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_both(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>])
+subroutine test_interop_init_both(obj)
+  integer(8) :: obj
+  !$omp interop init(targetsync, target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Use
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_use(
+!CHECK:         omp.interop.use %{{.*}} : !fir.ref<i64>
+subroutine test_interop_use(obj)
+  integer(8) :: obj
+  !$omp interop use(obj)
+end subroutine
+
+!===============================================================================
+! Interop Destroy
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_destroy(
+!CHECK:         omp.interop.destroy %{{.*}} : !fir.ref<i64>
+subroutine test_interop_destroy(obj)
+  integer(8) :: obj
+  !$omp interop destroy(obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — nowait
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_nowait(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) nowait
+subroutine test_interop_init_nowait(obj)
+  integer(8) :: obj
+  !$omp interop init(target: obj) nowait
+end subroutine
+
+!===============================================================================
+! Interop Destroy — nowait
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_destroy_nowait(
+!CHECK:         omp.interop.destroy %{{.*}} : !fir.ref<i64> nowait
+subroutine test_interop_destroy_nowait(obj)
+  integer(8) :: obj
+  !$omp interop destroy(obj) nowait
+end subroutine
+
+!===============================================================================
+! Interop Init — device clause
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_device(
+!CHECK:         %[[DEV:.*]] = fir.load %{{.*}} : !fir.ref<i32>
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) device(%[[DEV]] : i32)
+subroutine test_interop_device(obj, dev)
+  integer(8) :: obj
+  integer :: dev
+  !$omp interop device(dev) init(target: obj)
+end subroutine
+!===============================================================================
+! Interop Init — prefer_type with string FR identifiers
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_prefer_str(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>]) prefer_type([1, 6])
+subroutine test_interop_prefer_str(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type("cuda", "level_zero"), targetsync: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — prefer_type with string FR identifier (hip)
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_prefer_hip(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) prefer_type([5])
+subroutine test_interop_prefer_hip(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type("hip"), target: obj)
+end subroutine
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
index c9e6764e7d634..f6bc8f11ccccb 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
@@ -79,6 +79,16 @@ def TaskDependArrayAttr
   let constBuilderCall = ?;
 }
 
+//===----------------------------------------------------------------------===//
+// InteropTypeArrayAttr
+//===----------------------------------------------------------------------===//
+
+def InteropTypeArrayAttr
+    : TypedArrayAttrBase<InteropTypeAttr,
+                         InteropTypeAttr.summary # " array"> {
+  let constBuilderCall = ?;
+}
+
 //===----------------------------------------------------------------------===//
 // VersionAttr
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
index 55ffcce956171..c961d3e853680 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
@@ -373,4 +373,22 @@ def FallbackModifierAttr : OpenMP_EnumAttr<FallbackModifier,
   let assemblyFormat = "`(` $value `)`";
 }
 
+//===----------------------------------------------------------------------===//
+// interop_type for interop construct
+//===----------------------------------------------------------------------===//
+
+def InteropTypeTarget : I32EnumAttrCase<"target", 0>;
+def InteropTypeTargetSync : I32EnumAttrCase<"targetsync", 1>;
+
+def InteropType : OpenMP_I32EnumAttr<
+    "InteropType",
+    "interop type", [
+      InteropTypeTarget,
+      InteropTypeTargetSync
+    ]>;
+
+def InteropTypeAttr : OpenMP_EnumAttr<InteropType, "interop_type"> {
+  let assemblyFormat = "`(` $value `)`";
+}
+
 #endif // OPENMP_ENUMS
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 51e7080db5b29..e5aae8c332a99 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2614,4 +2614,116 @@ def GroupprivateOp : OpenMP_Op<"groupprivate",
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// 14.1 interop construct
+//===----------------------------------------------------------------------===//
+
+def InteropInitOp
+    : OpenMP_Op<"interop.init", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop init operation";
+  let description = [{
+    The interop init operation initializes an interop object for
+    foreign runtime interoperability. The `interop_type` attribute specifies
+    whether the interop object provides access to the target execution
+    environment (`target`), a synchronization object (`targetsync`), or both.
+
+    The optional `prefer_type` attribute provides a list of preferred foreign
+    runtime types as integer identifiers.
+
+    The optional `device` operand specifies the device number.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      InteropTypeArrayAttr:$interop_types,
+      OptionalAttr<I64ArrayAttr>:$prefer_type,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    `interop_types` `(` $interop_types `)`
+    (`prefer_type` `(` $prefer_type^ `)`)?
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+
+  let builders = [
+    OpBuilder<(ins CArg<"const InteropInitOperands &">:$clauses)>
+  ];
+}
+
+def InteropUseOp
+    : OpenMP_Op<"interop.use", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop use operation";
+  let description = [{
+    The interop use operation indicates use of an interop object. This is
+    used in combination with depend clauses to synchronize tasks with
+    foreign runtime operations.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+
+  let builders = [
+    OpBuilder<(ins CArg<"const InteropUseOperands &">:$clauses)>
+  ];
+}
+
+def InteropDestroyOp
+    : OpenMP_Op<"interop.destroy", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop destroy operation";
+  let description = [{
+    The interop destroy operation releases an interop object and frees
+    any resources associated with it.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+
+  let builders = [
+    OpBuilder<(ins CArg<"const InteropDestroyOperands &">:$clauses)>
+  ];
+}
+
 #endif // OPENMP_OPS
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 7c75791611218..8270d5fa85a1c 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4527,6 +4527,46 @@ void OrderedRegionOp::build(OpBuilder &builder, OperationState &state,
 
 LogicalResult OrderedRegionOp::verify() { return verifyOrderedParent(**this); }
 
+//===----------------------------------------------------------------------===//
+// InteropInitOp
+//===----------------------------------------------------------------------===//
+
+void InteropInitOp::build(OpBuilder &builder, OperationState &state,
+                          const InteropInitOperands &clauses) {
+  // TODO Store clauses in op: dependKinds, dependVars, nowait.
+  InteropInitOp::build(builder, state, /*interop_var=*/nullptr,
+                       /*interop_types=*/nullptr, /*prefer_type=*/nullptr,
+                       /*device=*/nullptr, /*depend_kinds=*/nullptr,
+                       /*depend_vars=*/{}, /*depend_iterated_kinds=*/nullptr,
+                       /*depend_iterated=*/{}, /*nowait=*/nullptr);
+}
+
+//===----------------------------------------------------------------------===//
+// InteropUseOp
+//===----------------------------------------------------------------------===//
+
+void InteropUseOp::build(OpBuilder &builder, OperationState &state,
+                         const InteropUseOperands &clauses) {
+  // TODO Store clauses in op: dependKinds, dependVars, nowait.
+  InteropUseOp::build(builder, state, /*interop_var=*/nullptr,
+                      /*device=*/nullptr, /*depend_kinds=*/nullptr,
+                      /*depend_vars=*/{}, /*depend_iterated_kinds=*/nullptr,
+                      /*depend_iterated=*/{}, /*nowait=*/nullptr);
+}
+
+//===----------------------------------------------------------------------===//
+// InteropDestroyOp
+//===----------------------------------------------------------------------===//
+
+void InteropDestroyOp::build(OpBuilder &builder, OperationState &state,
+                             const InteropDestroyOperands &clauses) {
+  // TODO Store clauses in op: dependKinds, dependVars, nowait.
+  InteropDestroyOp::build(builder, state, /*interop_var=*/nullptr,
+                          /*device=*/nullptr, /*depend_kinds=*/nullptr,
+                          /*depend_vars=*/{}, /*depend_iterated_kinds=*/nullptr,
+                          /*depend_iterated=*/{}, /*nowait=*/nullptr);
+}
+
 //===----------------------------------------------------------------------===//
 // TaskwaitOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..9b42e93a55c5a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -3711,6 +3711,98 @@ convertOmpTaskgroupOp(omp::TaskgroupOp tgOp, llvm::IRBuilderBase &builder,
   return success();
 }
 
+static LogicalResult
+convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
+                        LLVM::ModuleTranslation &moduleTranslation) {
+  if (!initOp.getDependVars().empty() || initOp.getDependKinds() ||
+      !initOp.getDependIterated().empty() || initOp.getDependIteratedKinds())
+    return initOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropInitOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(initOp.getInteropVar());
+  llvm::Value *device = initOp.getDevice()
+                            ? moduleTranslation.lookupValue(initOp.getDevice())
+                            : nullptr;
+
+  // TODO: Handle depend clauses when supported.
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = initOp.getNowait();
+
+  // Emit one call per interop type. The runtime expects a single type per call.
+  for (auto typeAttr : initOp.getInteropTypes()) {
+    auto interopTypeEnum = cast<omp::InteropTypeAttr>(typeAttr).getValue();
+    llvm::omp::OMPInteropType interopType;
+    switch (interopTypeEnum) {
+    case omp::InteropType::target:
+      interopType = llvm::omp::OMPInteropType::Target;
+      break;
+    case omp::InteropType::targetsync:
+      interopType = llvm::omp::OMPInteropType::TargetSync;
+      break;
+    }
+    ompBuilder->createOMPInteropInit(builder, interopVar, interopType, device,
+                                     numDeps, depArray, hasNowait);
+  }
+  return success();
+}
+
+static LogicalResult
+convertOmpInteropDestroyOp(omp::InteropDestroyOp destroyOp,
+                           llvm::IRBuilderBase &builder,
+                           LLVM::ModuleTranslation &moduleTranslation) {
+  if (!destroyOp.getDependVars().empty() || destroyOp.getDependKinds() ||
+      !destroyOp.getDependIterated().empty() ||
+      destroyOp.getDependIteratedKinds())
+    return destroyOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropDestroyOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(destroyOp.getInteropVar());
+  llvm::Value *device =
+      destroyOp.getDevice()
+          ? moduleTranslation.lookupValue(destroyOp.getDevice())
+          : nullptr;
+
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = destroyOp.getNowait();
+
+  ompBuilder->createOMPInteropDestroy(builder, interopVar, device, numDeps,
+                                      depArray, hasNowait);
+  return success();
+}
+
+static LogicalResult
+convertOmpInteropUseOp(omp::InteropUseOp useOp, llvm::IRBuilderBase &builder,
+                       LLVM::ModuleTranslation &moduleTranslation) {
+  if (!useOp.getDependVars().empty() || useOp.getDependKinds() ||
+      !useOp.getDependIterated().empty() || useOp.getDependIteratedKinds())
+    return useOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropUseOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(useOp.getInteropVar());
+  llvm::Value *device = useOp.getDevice()
+                            ? moduleTranslation.lookupValue(useOp.getDevice())
+                            : nullptr;
+
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = useOp.getNowait();
+
+  ompBuilder->createOMPInteropUse(builder, interopVar, device, numDeps,
+                                  depArray, hasNowait);
+  return success();
+}
+
 static LogicalResult
 convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
                      LLVM::ModuleTranslation &moduleTranslation) {
@@ -8884,6 +8976,15 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::TaskwaitOp op) {
             return convertOmpTaskwaitOp(op, builder, moduleTranslation);
           })
+          .Case([&](omp::InteropInitOp op) {
+            return convertOmpInteropInitOp(op, builder, moduleTranslation);
+          })
+          .Case([&](omp::InteropDestroyOp op) {
+            return convertOmpInteropDestroyOp(op, builder, moduleTranslation);
+          })
+          .Case([&](omp::InteropUseOp op) {
+            return convertOmpInteropUseOp(op, builder, moduleTranslation);
+          })
           .Case<omp::YieldOp, omp::TerminatorOp, omp::DeclareMapperOp,
                 omp::DeclareMapperInfoOp, omp::DeclareReductionOp,
                 omp::CriticalDeclareOp>([](auto op) {
diff --git a/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir b/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
new file mode 100644
index 0000000000000..ebd601558f392
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
@@ -0,0 +1,7 @@
+// RUN: not mlir-translate --mlir-to-llvmir %s 2>&1 | FileCheck %s
+
+// CHECK: error: not yet implemented: Unhandled clause depend in omp.interop.init operation
+llvm.func @test_interop_init_depend(%interop: !llvm.ptr, %dep: !llvm.ptr) {
+  omp.interop.init %interop : !llvm.ptr interop_types([#omp<interop_type(targetsync)>]) depend(taskdependinout -> %dep : !llvm.ptr)
+  llvm.return
+}
diff --git a/mlir/test/Target/LLVMIR/openmp-interop.mlir b/mlir/test/Target/LLVMIR/openmp-interop.mlir
new file mode 100644
index 0000000000000..9fd1fd583aba8
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-interop.mlir
@@ -0,0 +1,71 @@
+// RUN: mlir-translate --mlir-to-llvmir %s | FileCheck %s
+
+// CHECK-LABEL: define void @test_interop_init(
+// CHECK-SAME:    ptr %[[OBJ:.*]])
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %[[GTID]], ptr %[[OBJ]], i32 1, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_init(%arg0: !llvm.ptr) {
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_targetsync(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 2, i32 -1, i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_targetsync(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(targetsync)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_both(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 2, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 -1, i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_both(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_use(
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_use(ptr @{{.*}}, i32 %[[GTID]], ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_use(%arg0: !llvm.ptr) {
+  omp.interop.use %arg0 : !llvm.ptr
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy(
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %[[GTID]], ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_destroy(%arg0: !llvm.ptr) {
+  omp.interop.destroy %arg0 : !llvm.ptr
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_nowait(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 -1, i32 0, ptr null, i32 1)
+llvm.func @test_interop_init_nowait(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) nowait
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy_nowait(
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 1)
+llvm.func @test_interop_destroy_nowait(%arg0: !llvm.ptr) {
+  omp.interop.destroy %arg0 : !llvm.ptr nowait
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_device(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 %{{.*}}, i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_device(%arg0: !llvm.ptr, %arg1: i32) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) device(%arg1 : i32)
+  llvm.return
+}
+
+// CHECK: declare i32 @__kmpc_global_thread_num(ptr)
+// CHECK: declare void @__tgt_interop_init(ptr, i32, ptr, i32, i32, i32, ptr, i32)
+// CHECK: declare void @__tgt_interop_use(ptr, i32, ptr, i32, i32, ptr, i32)
+// CHECK: declare void @__tgt_interop_destroy(ptr, i32, ptr, i32, i32, ptr, i32)
diff --git a/openmp/module/omp_lib.F90.var b/openmp/module/omp_lib.F90.var
index 464056847ab92..9a7db73f97542 100644
--- a/openmp/module/omp_lib.F90.var
+++ b/openmp/module/omp_lib.F90.var
@@ -40,6 +40,8 @@
         integer, parameter, public :: omp_alloctrait_val_kind = c_intptr_t
         integer, parameter, public :: omp_interop_kind = c_intptr_t
         integer, parameter, public :: omp_interop_fr_kind = omp_integer_kind
+        integer, parameter, public :: omp_interop_property_kind = omp_integer_kind
+        integer, parameter, public :: omp_interop_rc_kind = omp_integer_kind
 
         type omp_alloctrait
           integer(kind=omp_alloctrait_key_kind) key
@@ -84,6 +86,8 @@
         public :: omp_alloctrait_val_kind
         public :: omp_interop_kind
         public :: omp_interop_fr_kind
+        public :: omp_interop_property_kind
+        public :: omp_interop_rc_kind
         public :: omp_alloctrait
         public :: omp_pause_resource_kind
         public :: omp_depend_kind
@@ -218,6 +222,28 @@
 
         integer (kind=omp_interop_kind), parameter, public :: omp_interop_none = 0
 
+!       interop property constants
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_id = -1
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_name = -2
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_vendor = -3
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_vendor_name = -4
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device_num = -5
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_platform = -6
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device = -7
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device_context = -8
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_targetsync = -9
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_first = -9
+
+!       interop return code constants
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_no_value = 1
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_success = 0
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_empty = -1
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_out_of_range = -2
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_int = -3
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_ptr = -4
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_str = -5
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_other = -6
+
         interface
 
 !         ***
@@ -1061,6 +1087,67 @@
             logical (kind=omp_logical_kind) kmp_get_cancellation_status
           end function kmp_get_cancellation_status
 
+!         ***
+!         *** interop query routines
+!         ***
+
+          function omp_get_num_interop_properties(interop) bind(c)
+            use omp_lib_kinds
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_integer_kind) omp_get_num_interop_properties
+          end function omp_get_num_interop_properties
+
+          function omp_get_interop_int(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_intptr_t, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            integer (kind=c_intptr_t) omp_get_interop_int
+          end function omp_get_interop_int
+
+          function omp_get_interop_ptr(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            type(c_ptr) omp_get_interop_ptr
+          end function omp_get_interop_ptr
+
+          function omp_get_interop_str(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            type(c_ptr) omp_get_interop_str
+          end function omp_get_interop_str
+
+          function omp_get_interop_name(interop, property_id) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            type(c_ptr) omp_get_interop_name
+          end function omp_get_interop_name
+
+          function omp_get_interop_type_desc(interop, property_id) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            type(c_ptr) omp_get_interop_type_desc
+          end function omp_get_interop_type_desc
+
+          function omp_get_interop_rc_desc(interop, ret_code) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_rc_kind), value :: ret_code
+            type(c_ptr) omp_get_interop_rc_desc
+          end function omp_get_interop_rc_desc
+
         end interface
 
         ! make the above routine definitions public
@@ -1182,5 +1269,12 @@
         public :: kmp_set_warnings_on
         public :: kmp_set_warnings_off
         public :: kmp_get_cancellation_status
+        public :: omp_get_num_interop_properties
+        public :: omp_get_interop_int
+        public :: omp_get_interop_ptr
+        public :: omp_get_interop_str
+        public :: omp_get_interop_name
+        public :: omp_get_interop_type_desc
+        public :: omp_get_interop_rc_desc
 
       end module omp_lib

>From 336e7bb81cc71d2192599cf170c6575e216265ca Mon Sep 17 00:00:00 2001
From: Sunil Shrestha <sunil.shrestha at hpe.com>
Date: Mon, 6 Jul 2026 16:44:38 -0500
Subject: [PATCH 2/2] Address feedback

  - track presence of the init clause instead of counting target/targetsync
  - collapse interop types into a single interop_init call
  - cast device id to match the runtime's expected type
  - ensure parity between the omp_lib module and header
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             |  17 ++-
 flang/lib/Semantics/check-omp-structure.cpp   |   3 +-
 .../OpenMP/interop-construct-v60.f90          |  24 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |   6 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  24 ++--
 mlir/test/Target/LLVMIR/openmp-interop.mlir   |  26 ++++-
 openmp/module/omp_lib.h.var                   | 105 ++++++++++++++++++
 7 files changed, 188 insertions(+), 17 deletions(-)
 create mode 100644 flang/test/Semantics/OpenMP/interop-construct-v60.f90

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 47a70efeb293a..fc6847b2104ce 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -5157,20 +5157,22 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
       mlir::Value interopVarAddr = getInteropVarAddr(interopVar);
 
-      // Convert interop types to MLIR attributes.
+      // Convert interop types to MLIR attributes. A single init clause may
+      // list both `target` and `targetsync`; keep them on one op and let the
+      // LLVM-IR translation collapse them into a single runtime init call,
+      // matching Clang.
       llvm::SmallVector<mlir::Attribute> interopTypeAttrs;
       for (auto itype : interopTypes) {
-        mlir::omp::InteropType mlirType;
         switch (itype) {
         case clause::Init::InteropType::Target:
-          mlirType = mlir::omp::InteropType::target;
+          interopTypeAttrs.push_back(mlir::omp::InteropTypeAttr::get(
+              context, mlir::omp::InteropType::target));
           break;
         case clause::Init::InteropType::Targetsync:
-          mlirType = mlir::omp::InteropType::targetsync;
+          interopTypeAttrs.push_back(mlir::omp::InteropTypeAttr::get(
+              context, mlir::omp::InteropType::targetsync));
           break;
         }
-        interopTypeAttrs.push_back(
-            mlir::omp::InteropTypeAttr::get(context, mlirType));
       }
       auto interopTypesAttr = mlir::ArrayAttr::get(context, interopTypeAttrs);
 
@@ -5210,6 +5212,9 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
           preferTypeAttr = firOpBuilder.getI64ArrayAttr(prefValues);
       }
 
+      // Emit a single InteropInitOp carrying all interop-types from this init
+      // clause. The translation to LLVM IR collapses them into one runtime
+      // call, matching Clang (one init clause -> one __tgt_interop_init).
       mlir::omp::InteropInitOp::create(
           firOpBuilder, loc, interopVarAddr, interopTypesAttr, preferTypeAttr,
           deviceValue,
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 333a71705b5b7..0ba3d8659d516 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -5787,6 +5787,7 @@ void OmpStructureChecker::Enter(const parser::OmpClause::ThreadLimit &x) {
 
 void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
   bool isDependClauseOccurred{false};
+  bool hasInitClause{false};
   int targetCount{0}, targetSyncCount{0};
   const auto &dir{std::get<parser::OmpDirectiveName>(x.v.t)};
   std::set<const Symbol *> objectSymbolList;
@@ -5796,6 +5797,7 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     common::visit(
         common::visitors{
             [&](const parser::OmpClause::Init &initClause) {
+              hasInitClause = true;
               if (OmpVerifyModifiers(initClause.v, llvm::omp::OMPC_init,
                       GetContext().directiveSource, context_)) {
 
@@ -5876,7 +5878,6 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
   // present on this directive. For use/destroy-only directives, the interop
   // object's type was established at its init site, which may be in a
   // different statement — we cannot verify it statically here.
-  bool hasInitClause = (targetCount > 0 || targetSyncCount > 0);
   if (isDependClauseOccurred && hasInitClause && !targetSyncCount) {
     context_.Say(GetContext().directiveSource,
         "A DEPEND clause can only appear on the directive if the interop-type includes TARGETSYNC"_err_en_US);
diff --git a/flang/test/Semantics/OpenMP/interop-construct-v60.f90 b/flang/test/Semantics/OpenMP/interop-construct-v60.f90
new file mode 100644
index 0000000000000..78a226f8a6b1f
--- /dev/null
+++ b/flang/test/Semantics/OpenMP/interop-construct-v60.f90
@@ -0,0 +1,24 @@
+! REQUIRES: openmp_runtime
+
+! RUN: %python %S/../test_errors.py %s %flang %openmp_flags -fopenmp-version=60
+! OpenMP Version 6.0
+! 14.1 Interop construct
+! In OpenMP 6.0 the interop-type modifier on an INIT clause is optional, so
+! INIT(obj) alone is accepted. A DEPEND clause, however, requires the
+! interop-type to include TARGETSYNC, which an untyped INIT does not provide,
+! so INIT(obj) combined with DEPEND is diagnosed.
+
+SUBROUTINE test_interop_untyped()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !$OMP INTEROP INIT(obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_interop_untyped
+
+SUBROUTINE test_interop_untyped_depend()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !ERROR: A DEPEND clause can only appear on the directive if the interop-type includes TARGETSYNC
+  !$OMP INTEROP INIT(obj) DEPEND(INOUT: obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_interop_untyped_depend
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 92eb7de0d882f..cabfcdf68eacb 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -8048,6 +8048,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropInit(
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   Constant *InteropTypeVal = ConstantInt::get(Int32, (int)InteropType);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
@@ -8076,6 +8078,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropDestroy(
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
     PointerType *PointerTypeVar = PointerType::getUnqual(M.getContext());
@@ -8104,6 +8108,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropUse(const LocationDescription &Loc,
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
     PointerType *PointerTypeVar = PointerType::getUnqual(M.getContext());
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 9b42e93a55c5a..14b0b912d7c1b 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -3732,21 +3732,27 @@ convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
   llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
   bool hasNowait = initOp.getNowait();
 
-  // Emit one call per interop type. The runtime expects a single type per call.
-  for (auto typeAttr : initOp.getInteropTypes()) {
-    auto interopTypeEnum = cast<omp::InteropTypeAttr>(typeAttr).getValue();
-    llvm::omp::OMPInteropType interopType;
-    switch (interopTypeEnum) {
+  // A single `init` clause may list both `target` and `targetsync`, but the
+  // runtime init call takes a single interop-type. Collapse the set to one
+  // value, matching Clang: if `target` is present use Target, otherwise
+  // TargetSync. The offload runtime object model supports only one type per
+  // object; representing both would require a runtime change.
+  bool hasTarget = false, hasTargetSync = false;
+  for (mlir::Attribute typeAttr : initOp.getInteropTypes()) {
+    switch (cast<omp::InteropTypeAttr>(typeAttr).getValue()) {
     case omp::InteropType::target:
-      interopType = llvm::omp::OMPInteropType::Target;
+      hasTarget = true;
       break;
     case omp::InteropType::targetsync:
-      interopType = llvm::omp::OMPInteropType::TargetSync;
+      hasTargetSync = true;
       break;
     }
-    ompBuilder->createOMPInteropInit(builder, interopVar, interopType, device,
-                                     numDeps, depArray, hasNowait);
   }
+  llvm::omp::OMPInteropType interopType =
+      (!hasTarget && hasTargetSync) ? llvm::omp::OMPInteropType::TargetSync
+                                    : llvm::omp::OMPInteropType::Target;
+  ompBuilder->createOMPInteropInit(builder, interopVar, interopType, device,
+                                   numDeps, depArray, hasNowait);
   return success();
 }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-interop.mlir b/mlir/test/Target/LLVMIR/openmp-interop.mlir
index 9fd1fd583aba8..85cadd241e23d 100644
--- a/mlir/test/Target/LLVMIR/openmp-interop.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-interop.mlir
@@ -19,8 +19,8 @@ llvm.func @test_interop_init_targetsync(%arg0: !llvm.ptr) {
 }
 
 // CHECK-LABEL: define void @test_interop_init_both(
-// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 2, i32 -1, i32 0, ptr null, i32 0)
 // CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 -1, i32 0, ptr null, i32 0)
+// CHECK-NOT:     call void @__tgt_interop_init
 llvm.func @test_interop_init_both(%arg0: !llvm.ptr) {
   omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>])
   llvm.return
@@ -65,6 +65,30 @@ llvm.func @test_interop_init_device(%arg0: !llvm.ptr, %arg1: i32) {
   llvm.return
 }
 
+// CHECK-LABEL: define void @test_interop_init_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) device(%arg1 : i64)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_destroy_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.destroy %arg0 : !llvm.ptr device(%arg1 : i64)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_use_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_use(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_use_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.use %arg0 : !llvm.ptr device(%arg1 : i64)
+  llvm.return
+}
+
 // CHECK: declare i32 @__kmpc_global_thread_num(ptr)
 // CHECK: declare void @__tgt_interop_init(ptr, i32, ptr, i32, i32, i32, ptr, i32)
 // CHECK: declare void @__tgt_interop_use(ptr, i32, ptr, i32, i32, ptr, i32)
diff --git a/openmp/module/omp_lib.h.var b/openmp/module/omp_lib.h.var
index e515c9434f120..5e1996e31fe0a 100644
--- a/openmp/module/omp_lib.h.var
+++ b/openmp/module/omp_lib.h.var
@@ -56,6 +56,10 @@
       parameter(omp_interop_kind=int_ptr_kind())
       integer omp_interop_fr_kind
       parameter(omp_interop_fr_kind=omp_integer_kind)
+      integer omp_interop_property_kind
+      parameter(omp_interop_property_kind=omp_integer_kind)
+      integer omp_interop_rc_kind
+      parameter(omp_interop_rc_kind=omp_integer_kind)
 
       integer(kind=omp_integer_kind)openmp_version
       parameter(openmp_version=@LIBOMP_OMP_YEAR_MONTH@)
@@ -296,6 +300,46 @@
       integer(kind=omp_interop_kind)omp_interop_none
       parameter(omp_interop_none=0)
 
+!     interop property constants
+      integer(kind=omp_interop_property_kind)omp_ipr_fr_id
+      parameter(omp_ipr_fr_id=-1)
+      integer(kind=omp_interop_property_kind)omp_ipr_fr_name
+      parameter(omp_ipr_fr_name=-2)
+      integer(kind=omp_interop_property_kind)omp_ipr_vendor
+      parameter(omp_ipr_vendor=-3)
+      integer(kind=omp_interop_property_kind)omp_ipr_vendor_name
+      parameter(omp_ipr_vendor_name=-4)
+      integer(kind=omp_interop_property_kind)omp_ipr_device_num
+      parameter(omp_ipr_device_num=-5)
+      integer(kind=omp_interop_property_kind)omp_ipr_platform
+      parameter(omp_ipr_platform=-6)
+      integer(kind=omp_interop_property_kind)omp_ipr_device
+      parameter(omp_ipr_device=-7)
+      integer(kind=omp_interop_property_kind)omp_ipr_device_context
+      parameter(omp_ipr_device_context=-8)
+      integer(kind=omp_interop_property_kind)omp_ipr_targetsync
+      parameter(omp_ipr_targetsync=-9)
+      integer(kind=omp_interop_property_kind)omp_ipr_first
+      parameter(omp_ipr_first=-9)
+
+!     interop return code constants
+      integer(kind=omp_interop_rc_kind)omp_irc_no_value
+      parameter(omp_irc_no_value=1)
+      integer(kind=omp_interop_rc_kind)omp_irc_success
+      parameter(omp_irc_success=0)
+      integer(kind=omp_interop_rc_kind)omp_irc_empty
+      parameter(omp_irc_empty=-1)
+      integer(kind=omp_interop_rc_kind)omp_irc_out_of_range
+      parameter(omp_irc_out_of_range=-2)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_int
+      parameter(omp_irc_type_int=-3)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_ptr
+      parameter(omp_irc_type_ptr=-4)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_str
+      parameter(omp_irc_type_str=-5)
+      integer(kind=omp_interop_rc_kind)omp_irc_other
+      parameter(omp_irc_other=-6)
+
       interface
 
 !       ***
@@ -1129,6 +1173,67 @@
 
         subroutine kmp_set_warnings_off() bind(c)
         end subroutine kmp_set_warnings_off
+
+!       ***
+!       *** interop query routines
+!       ***
+
+        function omp_get_num_interop_properties(interop) bind(c)
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_integer_kind) omp_get_num_interop_properties
+        end function omp_get_num_interop_properties
+
+        function omp_get_interop_int(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_intptr_t, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          integer (kind=c_intptr_t) omp_get_interop_int
+        end function omp_get_interop_int
+
+        function omp_get_interop_ptr(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          type(c_ptr) omp_get_interop_ptr
+        end function omp_get_interop_ptr
+
+        function omp_get_interop_str(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          type(c_ptr) omp_get_interop_str
+        end function omp_get_interop_str
+
+        function omp_get_interop_name(interop, property_id) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          type(c_ptr) omp_get_interop_name
+        end function omp_get_interop_name
+
+        function omp_get_interop_type_desc(interop, property_id) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          type(c_ptr) omp_get_interop_type_desc
+        end function omp_get_interop_type_desc
+
+        function omp_get_interop_rc_desc(interop, ret_code) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_rc_kind), value :: ret_code
+          type(c_ptr) omp_get_interop_rc_desc
+        end function omp_get_interop_rc_desc
       end interface
 
 #ifdef __INTEL_COMPILER


From llvmlistbot at llvm.org  Wed Jul  8 09:51:52 2026
From: llvmlistbot at llvm.org (Sunil Shrestha)
Date: Wed, 08 Jul 2026 09:51:52 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a4e8028.ac469dc7.3a57b1.e0f0@mx.google.com>


================
@@ -5872,7 +5872,12 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     context_.Say(GetContext().directiveSource,
         "Each interop-type may be specified at most once."_err_en_US);
   }
-  if (isDependClauseOccurred && !targetSyncCount) {
+  // Only enforce the depend/targetsync constraint when an init clause is
+  // present on this directive. For use/destroy-only directives, the interop
+  // object's type was established at its init site, which may be in a
+  // different statement — we cannot verify it statically here.
+  bool hasInitClause = (targetCount > 0 || targetSyncCount > 0);
----------------
sshrestha-aa wrote:

Thank you for pointing out this change in OpenMP 6.0

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul  8 09:53:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 09:53:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][nvgpu] Add negative tests for invalid
 FP8 warpgroup MMA types (PR #208047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208047@github.com>
Message-ID: <6a4e8072.bf38ae28.39239a.dd5f@mx.google.com>


Vaisman wrote:

@grypp Thank you for the review! 
Could you please merge when you have a chance?

https://github.com/llvm/llvm-project/pull/208047

From llvmlistbot at llvm.org  Wed Jul  8 09:57:32 2026
From: llvmlistbot at llvm.org (Sunil Shrestha)
Date: Wed, 08 Jul 2026 09:57:32 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a4e817c.3cce8983.225732.9a36@mx.google.com>


================
@@ -3711,6 +3711,98 @@ convertOmpTaskgroupOp(omp::TaskgroupOp tgOp, llvm::IRBuilderBase &builder,
   return success();
 }
 
+static LogicalResult
+convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
+                        LLVM::ModuleTranslation &moduleTranslation) {
+  if (!initOp.getDependVars().empty() || initOp.getDependKinds() ||
+      !initOp.getDependIterated().empty() || initOp.getDependIteratedKinds())
+    return initOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropInitOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(initOp.getInteropVar());
+  llvm::Value *device = initOp.getDevice()
+                            ? moduleTranslation.lookupValue(initOp.getDevice())
+                            : nullptr;
+
+  // TODO: Handle depend clauses when supported.
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = initOp.getNowait();
+
+  // Emit one call per interop type. The runtime expects a single type per call.
+  for (auto typeAttr : initOp.getInteropTypes()) {
----------------
sshrestha-aa wrote:

I updated the lowering to match Clang: one init clause now produces a single `__tgt_interop_init call`. When a clause lists both `target` and `targetsync`, it collapses to `target`, because the runtime's interop type is a single value and can't represent both. I can follow up with a separate PR that extends the runtime to support both interop types.

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul  8 10:48:43 2026
From: llvmlistbot at llvm.org (Susan Tan =?utf-8?b?44K5LeOCtuODs+OAgOOCv+ODsw==?=)
Date: Wed, 08 Jul 2026 10:48:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][acc] Remove dangling operands in
 OpenACCUtilsTiling (PR #208261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208261@github.com>
Message-ID: <6a4e8d7b.6c8668f2.e4f82.ab4c@mx.google.com>


https://github.com/SusanTan updated https://github.com/llvm/llvm-project/pull/208261

>From 7428e41773155ef0216861f8f1166e15a3cfa167 Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 09:39:14 -0700
Subject: [PATCH 1/3] add fix

---
 mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index dddaebdee5ebc..6452d7e0ed8c8 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -97,6 +97,13 @@ createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
     elementLoop.removeGangOperandsArgTypeAttr();
     elementLoop.removeGangOperandsSegmentsAttr();
     elementLoop.removeGangOperandsDeviceTypeAttr();
+    // Also drop the operand values themselves so that elementLoop does not
+    // end up with a non-empty gang operand list but no corresponding
+    // device-type/segment/arg-type attributes. Leaving stale operands behind
+    // makes elementLoop look like it still has gang operands to later
+    // queries (e.g. LoopOp::getGangValue), which then dereference the
+    // now-missing device-type attribute.
+    elementLoop.getGangOperandsMutable().clear();
   }
   if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
     elementLoop.removeWorkerAttr();

>From 02340ce1bce6f42f8c41d295909f3c48fe8a9caf Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 10:05:40 -0700
Subject: [PATCH 2/3] add test

---
 .../test/Dialect/OpenACC/acc-loop-tiling.mlir | 43 +++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index cd13b3d8427f0..3b032959b1cc6 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -74,6 +74,49 @@ func.func @nested_loop_tile(%arg0: memref<100x50xf32>) {
   return
 }
 
+// Regression test: a loop with GANG(STATIC: N) combined with a multi-dim
+// TILE clause used to crash with an assertion failure inside
+// LoopOp::getGangValue() because uncollapseLoops() (needed here since the
+// tile count exceeds the implicit collapse count of 1) left one of the
+// generated inner loops with a leftover gang operand but no corresponding
+// gang device-type attribute. Check that the pass completes and that gang
+// is only preserved on the outermost tile loop.
+
+// CHECK-LABEL: func.func @gang_static_with_multi_dim_tile
+// CHECK:         acc.loop gang({static=%{{.*}} : i32}) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       gang
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         gang
+// CHECK:             acc.loop control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           gang
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             gang
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cs = arith.constant 1 : i32
+  acc.loop gang({static=%cs : i32}) tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
 // Test unknown tile size (*) represented as -1
 // Should use default tile size (32)
 

>From 7a79628faffedbc0f8e3a47bd1d9940a7cad4190 Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 10:48:30 -0700
Subject: [PATCH 3/3] remove worker value operands as well

---
 .../OpenACC/Utils/OpenACCUtilsTiling.cpp      |  3 ++
 .../test/Dialect/OpenACC/acc-loop-tiling.mlir | 49 +++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index 6452d7e0ed8c8..09e83af2bc686 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -108,6 +108,9 @@ createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
   if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
     elementLoop.removeWorkerAttr();
     elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    // As above for gang, also drop the worker operand values so elementLoop
+    // does not keep a dangling worker operand with no device-type attribute.
+    elementLoop.getWorkerNumOperandsMutable().clear();
   }
   rewriter.finalizeOpModification(elementLoop);
 
diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index 3b032959b1cc6..a1f80ad9defdf 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -117,6 +117,55 @@ func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
   return
 }
 
+// Regression test: a loop with WORKER(N) combined with VECTOR and a
+// multi-dim TILE clause used to produce invalid IR because createInnerLoop()
+// (via uncollapseLoops()) and removeWorkerVectorFromLoop() removed the
+// worker attributes from generated inner/tile loops without also clearing
+// the worker operand value copied onto them. This left loops with a
+// non-empty worker operand list but no worker device-type attribute,
+// which the verifier rejects ('worker operands count must match worker
+// device_type count'). Check that the pass produces valid IR, with worker
+// only preserved on the outermost tile loop and vector only on the
+// outermost element loop.
+
+// CHECK-LABEL: func.func @worker_num_with_multi_dim_tile
+// CHECK:         acc.loop worker(%{{.*}} : i32) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       worker
+// CHECK-NOT:       vector
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         worker
+// CHECK-NOT:         vector
+// CHECK:             acc.loop vector control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           worker
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             worker
+// CHECK-NOT:             vector
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @worker_num_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cw = arith.constant 4 : i32
+  acc.loop worker(%cw : i32) vector tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
 // Test unknown tile size (*) represented as -1
 // Should use default tile size (32)
 


From llvmlistbot at llvm.org  Wed Jul  8 10:49:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 10:49:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] af53003 - [mlir][ArmSME] fix f64 scalable
 matmul crashes in `VectorLegalizationPass` (#207947)
Message-ID: <6a4e8db2.8e75e0cb.5e108.d8c3@mx.google.com>


Author: Federico Bruzzone
Date: 2026-07-08T19:49:34+02:00
New Revision: af53003dfee4afe3f003996dc09f55d59d63003a

URL: https://github.com/llvm/llvm-project/commit/af53003dfee4afe3f003996dc09f55d59d63003a
DIFF: https://github.com/llvm/llvm-project/commit/af53003dfee4afe3f003996dc09f55d59d63003a.diff

LOG: [mlir][ArmSME] fix f64 scalable matmul crashes in `VectorLegalizationPass` (#207947)

This PR simply fix what we identified in iree-org/iree#24689.

**Note**: the new test did not pass the checks prior to this PR 🫶

@banach-space @egebeysel @AGindinson

---------

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Co-authored-by: Artem Gindinson <gindinson at roofline.ai>

Added: 
    

Modified: 
    mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
    mlir/test/Dialect/ArmSME/vector-legalization.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
index 6e2654ab468f9..7a6c510920e22 100644
--- a/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
+++ b/mlir/lib/Dialect/ArmSME/Transforms/VectorLegalization.cpp
@@ -943,7 +943,9 @@ struct LowerColumnTransferReadToLoops
     auto upperBound = createVscaleMultiple(numRows);
     auto step = arith::ConstantIndexOp::create(rewriter, loc, 1);
     Value init = arith::ConstantOp::create(
-        rewriter, loc, newResType, DenseElementsAttr::get(newResType, 0.0f));
+        rewriter, loc, newResType,
+        DenseElementsAttr::get(newResType,
+                               rewriter.getZeroAttr(resType.getElementType())));
 
     scf::ForOp loadLoop;
     {

diff  --git a/mlir/test/Dialect/ArmSME/vector-legalization.mlir b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
index 6cdf576272ebc..a31417f106a96 100644
--- a/mlir/test/Dialect/ArmSME/vector-legalization.mlir
+++ b/mlir/test/Dialect/ArmSME/vector-legalization.mlir
@@ -650,6 +650,36 @@ func.func @xfer_read_scalable_column(%a: index, %b: index, %pad: f32, %src: memr
 
 // -----
 
+// CHECK-LABEL:   func.func @xfer_read_scalable_column_f64(
+// CHECK-SAME:      %[[IDX_0:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[IDX_1:[a-zA-Z0-9]+]]: index,
+// CHECK-SAME:      %[[PAD:.*]]: f64,
+// CHECK-SAME:      %[[SRC:.*]]: memref<?x?xf64>) -> vector<[4]x1xf64> {
+func.func @xfer_read_scalable_column_f64(%a: index, %b: index, %pad: f64, %src: memref<?x?xf64>) -> (vector<[4]x1xf64>) {
+  // CHECK:           %[[INIT:.*]] = arith.constant dense<0.000000e+00> : vector<[4]xf64>
+  // CHECK:           %[[STEP:.*]] = arith.constant 1 : index
+  // CHECK:           %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:           %[[LB:.*]] = arith.constant 0 : index
+  // CHECK:           %[[VSCALE:.*]] = vector.vscale
+  // CHECK:           %[[C4_VSCALE:.*]] = arith.muli %[[VSCALE]], %[[C4]] : index
+
+  // <scf.for>
+  // CHECK:           %[[SCF:.*]] = scf.for %[[IND_VAR:.*]] = %[[LB]] to %[[C4_VSCALE]] step %[[STEP]] iter_args(%[[SCF_RES:.*]] = %[[INIT]]) -> (vector<[4]xf64>) {
+  // CHECK:             %[[IDX_0_UPDATED:.*]] = arith.addi %[[IND_VAR]], %[[IDX_0]] : index
+  // CHECK:             %[[VAL_10:.*]] = memref.load %[[SRC]][%[[IDX_0_UPDATED]], %[[IDX_1]]] : memref<?x?xf64>
+  // CHECK:             %[[RES_UPDATED:.*]] = vector.insert %[[VAL_10]], %[[SCF_RES]] [%[[IND_VAR]]] : f64 into vector<[4]xf64>
+  // CHECK:             scf.yield %[[RES_UPDATED]] : vector<[4]xf64>
+  // CHECK:           }
+
+  // <shape-cast>
+  // CHECK:           %[[SC:.*]] = vector.shape_cast %[[SCF]] : vector<[4]xf64> to vector<[4]x1xf64>
+  // CHECK:           return %[[SC]]
+  %read = vector.transfer_read %src[%a, %b], %pad : memref<?x?xf64>, vector<[4]x1xf64>
+  return %read : vector<[4]x1xf64>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @negative_xfer_read_scalable_column_x2
 func.func @negative_xfer_read_scalable_column_x2(%a: index, %b: index, %pad: f32, %src: memref<?x?xf32>) -> (vector<[4]x2xf32>) {
   // CHECK-NOT: scf.for


        

From llvmlistbot at llvm.org  Wed Jul  8 10:49:42 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Wed, 08 Jul 2026 10:49:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4e8db6.30759430.24d50c.3e14@mx.google.com>


https://github.com/AGindinson closed https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Wed Jul  8 10:49:53 2026
From: llvmlistbot at llvm.org (Charitha Saumya)
Date: Wed, 08 Jul 2026 10:49:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
Message-ID: <llvm.org/llvm/llvm-project/pull/208276@github.com>


https://github.com/charithaintc created https://github.com/llvm/llvm-project/pull/208276

Currently vector-to-xegpu only support 2D SLM buffers. However load/store_matrix op definition allows 1D access.

Assisted by: Claude

>From ea777cd2577eb4be81ea0358db31474bce64afa6 Mon Sep 17 00:00:00 2001
From: Charitha Saumya <charitha.saumya.gusthinna.waduge at intel.com>
Date: Wed, 8 Jul 2026 17:48:00 +0000
Subject: [PATCH] add 1D SLM support in vector to xegpu

---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 12 ++---
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 44 +++++++++++--------
 .../transfer-write-to-xegpu.mlir              | 20 ++++++---
 3 files changed, 46 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 8dcc9771d8db4..ee9327caab002 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -560,10 +560,10 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isSharedMemory = xegpu::XeGPUDialect::isSharedMemory(readMemTy);
     // Handle the SLM case.
     if (isSharedMemory) {
-      // If the memref is SLM only support 2D case for now.
-      if (loadedVecTy.getRank() != 2)
+      // load_matrix supports 1D and 2D loads from SLM.
+      if (loadedVecTy.getRank() != 1 && loadedVecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            readOp, "Only 2D vector loads are supported for SLM");
+            readOp, "Only 1D and 2D vector loads are supported for SLM");
       AffineMap readMap = readOp.getPermutationMap();
       if (!readMap.isMinorIdentity())
         return rewriter.notifyMatchFailure(
@@ -700,10 +700,10 @@ struct TransferWriteLowering
     // For shared local memory (address space 3), use create_mem_desc +
     // store_matrix
     if (isSharedMemory) {
-      // Only support 2D case for now.
-      if (vecTy.getRank() != 2)
+      // store_matrix supports 1D and 2D stores to SLM.
+      if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            writeOp, "Only 2D vector stores are supported for SLM");
+            writeOp, "Only 1D and 2D vector stores are supported for SLM");
       // Create mem_desc for SLM
       auto memDescType =
           xegpu::MemDescType::get(rewriter.getContext(), writeMemTy.getShape(),
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index f64bcc27940ea..c6793a2e5ca26 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -598,7 +598,7 @@ gpu.func @load_2D_vector_addrspace3(%source: memref<16x32xf32, 3>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
+gpu.func @load_1D_vector_addrspace3(%source: memref<32xf32, 3>,
     %offset: index) -> vector<8xf32> {
   %c0 = arith.constant 0.0 : f32
   %0 = vector.transfer_read %source[%offset], %c0
@@ -606,11 +606,19 @@ gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-ND: vector.transfer_read
+// LOAD-ND-LABEL: @load_1D_vector_addrspace3
+// LOAD-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-ND-SAME: %[[OFFSET:.+]]: index
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-ND: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-ND: gpu.return %[[DATA]] : vector<8xf32>
 
-// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-GATHER: vector.transfer_read
+// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3
+// LOAD-GATHER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-GATHER-SAME: %[[OFFSET:.+]]: index
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-GATHER: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-GATHER: gpu.return %[[DATA]] : vector<8xf32>
 
 }
 
@@ -642,10 +650,10 @@ gpu.func @load_2D_vector_alloca_promoted_to_slm(%offset: index)
 
 // -----
 // memref.alloca is unconditionally promoted to SLM (address space 3). A 1D
-// transfer_read on it cannot be lowered to xegpu.load_matrix (which requires
-// rank 2), so the transfer_read is left as-is on the SLM memref.
+// transfer_read on it is lowered to xegpu.load_matrix, which supports the
+// 1D SLM case.
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
+gpu.func @load_1D_vector_alloca_promoted_to_slm(%offset: index)
     -> vector<8xf32> {
   %buf = memref.alloca() : memref<16xf32>
   %c0 = arith.constant 0.0 : f32
@@ -654,17 +662,15 @@ gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-ND: memref.alloca() : memref<16xf32, 3>
-// LOAD-ND: vector.transfer_read
-// LOAD-ND-NOT: xegpu.create_mem_desc
-// LOAD-ND-NOT: xegpu.load_matrix
+// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-ND: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-ND: xegpu.load_matrix %[[MEM_DESC]]
 
-// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-GATHER: memref.alloca() : memref<16xf32, 3>
-// LOAD-GATHER: vector.transfer_read
-// LOAD-GATHER-NOT: xegpu.create_mem_desc
-// LOAD-GATHER-NOT: xegpu.load_matrix
+// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-GATHER: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-GATHER: xegpu.load_matrix %[[MEM_DESC]]
 
 }
 
@@ -679,4 +685,4 @@ gpu.func @load_0D_memref_unsupported(%source: memref<f16>) -> vector<f16> {
 // CHECK-LABEL: @load_0D_memref_unsupported
 // CHECK: vector.transfer_read
 
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index a7cd3d7652d85..27a83922fab37 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -397,7 +397,7 @@ gpu.func @store_2D_vector_addrspace3(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
+gpu.func @store_1D_vector_addrspace3(%vec: vector<8xf32>,
     %source: memref<32xf32, 3>, %offset: index) {
   vector.transfer_write %vec, %source[%offset]
     {in_bounds = [true]}
@@ -405,10 +405,20 @@ gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-ND: vector.transfer_write
+// STORE-ND-LABEL: @store_1D_vector_addrspace3
+// STORE-ND-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-ND-SAME: %[[OFFSET:.+]]: index
+// STORE-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-ND: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-ND: gpu.return
 
-// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-SCATTER: vector.transfer_write
+// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3
+// STORE-SCATTER-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-SCATTER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-SCATTER-SAME: %[[OFFSET:.+]]: index
+// STORE-SCATTER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-SCATTER: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-SCATTER: gpu.return
 
 }


From llvmlistbot at llvm.org  Wed Jul  8 10:50:10 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Wed, 08 Jul 2026 10:50:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4e8dd2.af6ba3ed.36d9db.690d@mx.google.com>


AGindinson wrote:

Fair point :)

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Wed Jul  8 10:50:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 10:50:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a4e8de9.92d2873c.1ab170.a7a2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Charitha Saumya (charithaintc)

<details>
<summary>Changes</summary>

Currently vector-to-xegpu only support 2D SLM buffers. However load/store_matrix op definition allows 1D access.

Assisted by: Claude

---
Full diff: https://github.com/llvm/llvm-project/pull/208276.diff


3 Files Affected:

- (modified) mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp (+6-6) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir (+25-19) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir (+15-5) 


``````````diff
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 8dcc9771d8db4..ee9327caab002 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -560,10 +560,10 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isSharedMemory = xegpu::XeGPUDialect::isSharedMemory(readMemTy);
     // Handle the SLM case.
     if (isSharedMemory) {
-      // If the memref is SLM only support 2D case for now.
-      if (loadedVecTy.getRank() != 2)
+      // load_matrix supports 1D and 2D loads from SLM.
+      if (loadedVecTy.getRank() != 1 && loadedVecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            readOp, "Only 2D vector loads are supported for SLM");
+            readOp, "Only 1D and 2D vector loads are supported for SLM");
       AffineMap readMap = readOp.getPermutationMap();
       if (!readMap.isMinorIdentity())
         return rewriter.notifyMatchFailure(
@@ -700,10 +700,10 @@ struct TransferWriteLowering
     // For shared local memory (address space 3), use create_mem_desc +
     // store_matrix
     if (isSharedMemory) {
-      // Only support 2D case for now.
-      if (vecTy.getRank() != 2)
+      // store_matrix supports 1D and 2D stores to SLM.
+      if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            writeOp, "Only 2D vector stores are supported for SLM");
+            writeOp, "Only 1D and 2D vector stores are supported for SLM");
       // Create mem_desc for SLM
       auto memDescType =
           xegpu::MemDescType::get(rewriter.getContext(), writeMemTy.getShape(),
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index f64bcc27940ea..c6793a2e5ca26 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -598,7 +598,7 @@ gpu.func @load_2D_vector_addrspace3(%source: memref<16x32xf32, 3>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
+gpu.func @load_1D_vector_addrspace3(%source: memref<32xf32, 3>,
     %offset: index) -> vector<8xf32> {
   %c0 = arith.constant 0.0 : f32
   %0 = vector.transfer_read %source[%offset], %c0
@@ -606,11 +606,19 @@ gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-ND: vector.transfer_read
+// LOAD-ND-LABEL: @load_1D_vector_addrspace3
+// LOAD-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-ND-SAME: %[[OFFSET:.+]]: index
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-ND: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-ND: gpu.return %[[DATA]] : vector<8xf32>
 
-// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-GATHER: vector.transfer_read
+// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3
+// LOAD-GATHER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-GATHER-SAME: %[[OFFSET:.+]]: index
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-GATHER: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-GATHER: gpu.return %[[DATA]] : vector<8xf32>
 
 }
 
@@ -642,10 +650,10 @@ gpu.func @load_2D_vector_alloca_promoted_to_slm(%offset: index)
 
 // -----
 // memref.alloca is unconditionally promoted to SLM (address space 3). A 1D
-// transfer_read on it cannot be lowered to xegpu.load_matrix (which requires
-// rank 2), so the transfer_read is left as-is on the SLM memref.
+// transfer_read on it is lowered to xegpu.load_matrix, which supports the
+// 1D SLM case.
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
+gpu.func @load_1D_vector_alloca_promoted_to_slm(%offset: index)
     -> vector<8xf32> {
   %buf = memref.alloca() : memref<16xf32>
   %c0 = arith.constant 0.0 : f32
@@ -654,17 +662,15 @@ gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-ND: memref.alloca() : memref<16xf32, 3>
-// LOAD-ND: vector.transfer_read
-// LOAD-ND-NOT: xegpu.create_mem_desc
-// LOAD-ND-NOT: xegpu.load_matrix
+// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-ND: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-ND: xegpu.load_matrix %[[MEM_DESC]]
 
-// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-GATHER: memref.alloca() : memref<16xf32, 3>
-// LOAD-GATHER: vector.transfer_read
-// LOAD-GATHER-NOT: xegpu.create_mem_desc
-// LOAD-GATHER-NOT: xegpu.load_matrix
+// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-GATHER: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-GATHER: xegpu.load_matrix %[[MEM_DESC]]
 
 }
 
@@ -679,4 +685,4 @@ gpu.func @load_0D_memref_unsupported(%source: memref<f16>) -> vector<f16> {
 // CHECK-LABEL: @load_0D_memref_unsupported
 // CHECK: vector.transfer_read
 
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index a7cd3d7652d85..27a83922fab37 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -397,7 +397,7 @@ gpu.func @store_2D_vector_addrspace3(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
+gpu.func @store_1D_vector_addrspace3(%vec: vector<8xf32>,
     %source: memref<32xf32, 3>, %offset: index) {
   vector.transfer_write %vec, %source[%offset]
     {in_bounds = [true]}
@@ -405,10 +405,20 @@ gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-ND: vector.transfer_write
+// STORE-ND-LABEL: @store_1D_vector_addrspace3
+// STORE-ND-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-ND-SAME: %[[OFFSET:.+]]: index
+// STORE-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-ND: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-ND: gpu.return
 
-// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-SCATTER: vector.transfer_write
+// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3
+// STORE-SCATTER-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-SCATTER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-SCATTER-SAME: %[[OFFSET:.+]]: index
+// STORE-SCATTER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-SCATTER: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-SCATTER: gpu.return
 
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/208276

From llvmlistbot at llvm.org  Wed Jul  8 10:50:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 10:50:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a4e8dea.da8bafd3.14cd50.940e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Charitha Saumya (charithaintc)

<details>
<summary>Changes</summary>

Currently vector-to-xegpu only support 2D SLM buffers. However load/store_matrix op definition allows 1D access.

Assisted by: Claude

---
Full diff: https://github.com/llvm/llvm-project/pull/208276.diff


3 Files Affected:

- (modified) mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp (+6-6) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir (+25-19) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir (+15-5) 


``````````diff
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 8dcc9771d8db4..ee9327caab002 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -560,10 +560,10 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isSharedMemory = xegpu::XeGPUDialect::isSharedMemory(readMemTy);
     // Handle the SLM case.
     if (isSharedMemory) {
-      // If the memref is SLM only support 2D case for now.
-      if (loadedVecTy.getRank() != 2)
+      // load_matrix supports 1D and 2D loads from SLM.
+      if (loadedVecTy.getRank() != 1 && loadedVecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            readOp, "Only 2D vector loads are supported for SLM");
+            readOp, "Only 1D and 2D vector loads are supported for SLM");
       AffineMap readMap = readOp.getPermutationMap();
       if (!readMap.isMinorIdentity())
         return rewriter.notifyMatchFailure(
@@ -700,10 +700,10 @@ struct TransferWriteLowering
     // For shared local memory (address space 3), use create_mem_desc +
     // store_matrix
     if (isSharedMemory) {
-      // Only support 2D case for now.
-      if (vecTy.getRank() != 2)
+      // store_matrix supports 1D and 2D stores to SLM.
+      if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            writeOp, "Only 2D vector stores are supported for SLM");
+            writeOp, "Only 1D and 2D vector stores are supported for SLM");
       // Create mem_desc for SLM
       auto memDescType =
           xegpu::MemDescType::get(rewriter.getContext(), writeMemTy.getShape(),
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index f64bcc27940ea..c6793a2e5ca26 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -598,7 +598,7 @@ gpu.func @load_2D_vector_addrspace3(%source: memref<16x32xf32, 3>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
+gpu.func @load_1D_vector_addrspace3(%source: memref<32xf32, 3>,
     %offset: index) -> vector<8xf32> {
   %c0 = arith.constant 0.0 : f32
   %0 = vector.transfer_read %source[%offset], %c0
@@ -606,11 +606,19 @@ gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-ND: vector.transfer_read
+// LOAD-ND-LABEL: @load_1D_vector_addrspace3
+// LOAD-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-ND-SAME: %[[OFFSET:.+]]: index
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-ND: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-ND: gpu.return %[[DATA]] : vector<8xf32>
 
-// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-GATHER: vector.transfer_read
+// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3
+// LOAD-GATHER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-GATHER-SAME: %[[OFFSET:.+]]: index
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-GATHER: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-GATHER: gpu.return %[[DATA]] : vector<8xf32>
 
 }
 
@@ -642,10 +650,10 @@ gpu.func @load_2D_vector_alloca_promoted_to_slm(%offset: index)
 
 // -----
 // memref.alloca is unconditionally promoted to SLM (address space 3). A 1D
-// transfer_read on it cannot be lowered to xegpu.load_matrix (which requires
-// rank 2), so the transfer_read is left as-is on the SLM memref.
+// transfer_read on it is lowered to xegpu.load_matrix, which supports the
+// 1D SLM case.
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
+gpu.func @load_1D_vector_alloca_promoted_to_slm(%offset: index)
     -> vector<8xf32> {
   %buf = memref.alloca() : memref<16xf32>
   %c0 = arith.constant 0.0 : f32
@@ -654,17 +662,15 @@ gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-ND: memref.alloca() : memref<16xf32, 3>
-// LOAD-ND: vector.transfer_read
-// LOAD-ND-NOT: xegpu.create_mem_desc
-// LOAD-ND-NOT: xegpu.load_matrix
+// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-ND: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-ND: xegpu.load_matrix %[[MEM_DESC]]
 
-// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-GATHER: memref.alloca() : memref<16xf32, 3>
-// LOAD-GATHER: vector.transfer_read
-// LOAD-GATHER-NOT: xegpu.create_mem_desc
-// LOAD-GATHER-NOT: xegpu.load_matrix
+// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-GATHER: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-GATHER: xegpu.load_matrix %[[MEM_DESC]]
 
 }
 
@@ -679,4 +685,4 @@ gpu.func @load_0D_memref_unsupported(%source: memref<f16>) -> vector<f16> {
 // CHECK-LABEL: @load_0D_memref_unsupported
 // CHECK: vector.transfer_read
 
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index a7cd3d7652d85..27a83922fab37 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -397,7 +397,7 @@ gpu.func @store_2D_vector_addrspace3(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
+gpu.func @store_1D_vector_addrspace3(%vec: vector<8xf32>,
     %source: memref<32xf32, 3>, %offset: index) {
   vector.transfer_write %vec, %source[%offset]
     {in_bounds = [true]}
@@ -405,10 +405,20 @@ gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-ND: vector.transfer_write
+// STORE-ND-LABEL: @store_1D_vector_addrspace3
+// STORE-ND-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-ND-SAME: %[[OFFSET:.+]]: index
+// STORE-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-ND: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-ND: gpu.return
 
-// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-SCATTER: vector.transfer_write
+// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3
+// STORE-SCATTER-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-SCATTER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-SCATTER-SAME: %[[OFFSET:.+]]: index
+// STORE-SCATTER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-SCATTER: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-SCATTER: gpu.return
 
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/208276

From llvmlistbot at llvm.org  Wed Jul  8 11:49:03 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Wed, 08 Jul 2026 11:49:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NFC] Add missing MLIRPass dep when
 DMLIR_ENABLE_PDL_IN_PATTERNMATCH=OFF used (PR #208289)
Message-ID: <llvm.org/llvm/llvm-project/pull/208289@github.com>


https://github.com/MaxGraey created https://github.com/llvm/llvm-project/pull/208289

When  PDL deialed disbled during build stage (`DMLIR_ENABLE_PDL_IN_PATTERNMATCH=OFF`) I got a compiler errors due to some of pdl deps transitively includes `MLIRPass` but without PDL compilation is not possible so this patch fix this

>From 305536d74fdea7e0f0dd7d65ccb8fb27360f1d51 Mon Sep 17 00:00:00 2001
From: MaxGraey <maxgraey at gmail.com>
Date: Wed, 8 Jul 2026 21:45:26 +0300
Subject: [PATCH] add missing MLIRPass dep

---
 mlir/lib/Transforms/Utils/CMakeLists.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/lib/Transforms/Utils/CMakeLists.txt b/mlir/lib/Transforms/Utils/CMakeLists.txt
index 335c2cacd2a4a..3ecf3859a7f4c 100644
--- a/mlir/lib/Transforms/Utils/CMakeLists.txt
+++ b/mlir/lib/Transforms/Utils/CMakeLists.txt
@@ -24,4 +24,5 @@ add_mlir_library(MLIRTransformUtils
   MLIRSideEffectInterfaces
   MLIRSubsetOpInterface
   MLIRRewrite
+  MLIRPass
   )


From llvmlistbot at llvm.org  Wed Jul  8 11:51:57 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Wed, 08 Jul 2026 11:51:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Add missing MLIRPass dep when
 DMLIR_ENABLE_PDL_IN_PATTERNMATCH=OFF used (NFC) (PR #208289)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208289@github.com>
Message-ID: <6a4e9c4d.8e75e0cb.5e108.9241@mx.google.com>


https://github.com/MaxGraey edited https://github.com/llvm/llvm-project/pull/208289

From llvmlistbot at llvm.org  Wed Jul  8 11:53:21 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 08 Jul 2026 11:53:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a4e9ca1.492e23e4.157d47.df5e@mx.google.com>


================
@@ -700,10 +700,10 @@ struct TransferWriteLowering
     // For shared local memory (address space 3), use create_mem_desc +
     // store_matrix
     if (isSharedMemory) {
-      // Only support 2D case for now.
-      if (vecTy.getRank() != 2)
+      // store_matrix supports 1D and 2D stores to SLM.
+      if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
----------------
nbpatel wrote:

same as above

https://github.com/llvm/llvm-project/pull/208276

From llvmlistbot at llvm.org  Wed Jul  8 11:53:21 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 08 Jul 2026 11:53:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a4e9ca1.eb165e5c.1817a5.d53f@mx.google.com>


================
@@ -560,10 +560,10 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isSharedMemory = xegpu::XeGPUDialect::isSharedMemory(readMemTy);
     // Handle the SLM case.
     if (isSharedMemory) {
-      // If the memref is SLM only support 2D case for now.
-      if (loadedVecTy.getRank() != 2)
+      // load_matrix supports 1D and 2D loads from SLM.
+      if (loadedVecTy.getRank() != 1 && loadedVecTy.getRank() != 2)
----------------
nbpatel wrote:

why not just check rank > 2? can scalars sneak in here?

https://github.com/llvm/llvm-project/pull/208276

From llvmlistbot at llvm.org  Wed Jul  8 11:54:35 2026
From: llvmlistbot at llvm.org (Igor Zamyatin)
Date: Wed, 08 Jul 2026 11:54:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support subgroup layout
 assignment for scatter stores (PR #205955)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205955@github.com>
Message-ID: <6a4e9ceb.ff037943.2e959a.dadf@mx.google.com>


Garra1980 wrote:

cc @akroviakov 

https://github.com/llvm/llvm-project/pull/205955

From llvmlistbot at llvm.org  Wed Jul  8 12:47:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 08 Jul 2026 12:47:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4ea95d.bd93e731.358318.7ccc@mx.google.com>


banach-space wrote:

> This PR simply fix what we identified in [iree-org/iree#24689](https://github.com/iree-org/iree/issues/24689).
> 
> **Note**: the new test did not pass the checks prior to this PR 🫶
> 
> @banach-space @egebeysel @AGindinson

Kind requests for future PRs:

* “This PR simply fix what we identified in [iree-org/iree#24689](https://github.com/iree-org/iree/issues/24689).” - Since LLVM is a separate project, could you make sure that every change is self-contained, including the commit/PR summary, and does not require following links to issues in other projects? In this case, I do not know what the original issue was or what exactly this PR fixes.
* “Note: the new test did not pass the checks prior to this PR 🫶” — Could you be more specific about what happened? For example, did it crash, fail verification, or produce incorrect output? These are different failure modes and it would be helpful to capture that in the summary.

I do not want to come across as gatekeeping `Vector` patches. Quite the opposite: once a patch has enough approval, I do not think every Vector maintainer needs to sign off. That said, please make sure that each change is self-contained and that the summary provides the relevant context and rationale.

Thank you for fixing this!

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Wed Jul  8 12:51:56 2026
From: llvmlistbot at llvm.org (Christopher Bate)
Date: Wed, 08 Jul 2026 12:51:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][AliasAnalysis] Add precise
 listener-driven cache invalidation (PR #207032)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207032@github.com>
Message-ID: <6a4eaa5c.ac469dc7.3a57b1.306d@mx.google.com>


https://github.com/christopherbate commented:

I'm probably not the best person to review this. The API changes on the MLIR side look OK to me. 

https://github.com/llvm/llvm-project/pull/207032

From llvmlistbot at llvm.org  Wed Jul  8 12:59:30 2026
From: llvmlistbot at llvm.org (Susan Tan =?utf-8?b?44K5LeOCtuODs+OAgOOCv+ODsw==?=)
Date: Wed, 08 Jul 2026 12:59:30 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Remove dangling operands
 in OpenACCUtilsTiling (PR #208261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208261@github.com>
Message-ID: <6a4eac22.e55dc341.1f19bd.bf64@mx.google.com>


https://github.com/SusanTan updated https://github.com/llvm/llvm-project/pull/208261

>From 7428e41773155ef0216861f8f1166e15a3cfa167 Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 09:39:14 -0700
Subject: [PATCH 1/4] add fix

---
 mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index dddaebdee5ebc..6452d7e0ed8c8 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -97,6 +97,13 @@ createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
     elementLoop.removeGangOperandsArgTypeAttr();
     elementLoop.removeGangOperandsSegmentsAttr();
     elementLoop.removeGangOperandsDeviceTypeAttr();
+    // Also drop the operand values themselves so that elementLoop does not
+    // end up with a non-empty gang operand list but no corresponding
+    // device-type/segment/arg-type attributes. Leaving stale operands behind
+    // makes elementLoop look like it still has gang operands to later
+    // queries (e.g. LoopOp::getGangValue), which then dereference the
+    // now-missing device-type attribute.
+    elementLoop.getGangOperandsMutable().clear();
   }
   if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
     elementLoop.removeWorkerAttr();

>From 02340ce1bce6f42f8c41d295909f3c48fe8a9caf Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 10:05:40 -0700
Subject: [PATCH 2/4] add test

---
 .../test/Dialect/OpenACC/acc-loop-tiling.mlir | 43 +++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index cd13b3d8427f0..3b032959b1cc6 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -74,6 +74,49 @@ func.func @nested_loop_tile(%arg0: memref<100x50xf32>) {
   return
 }
 
+// Regression test: a loop with GANG(STATIC: N) combined with a multi-dim
+// TILE clause used to crash with an assertion failure inside
+// LoopOp::getGangValue() because uncollapseLoops() (needed here since the
+// tile count exceeds the implicit collapse count of 1) left one of the
+// generated inner loops with a leftover gang operand but no corresponding
+// gang device-type attribute. Check that the pass completes and that gang
+// is only preserved on the outermost tile loop.
+
+// CHECK-LABEL: func.func @gang_static_with_multi_dim_tile
+// CHECK:         acc.loop gang({static=%{{.*}} : i32}) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       gang
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         gang
+// CHECK:             acc.loop control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           gang
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             gang
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cs = arith.constant 1 : i32
+  acc.loop gang({static=%cs : i32}) tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
 // Test unknown tile size (*) represented as -1
 // Should use default tile size (32)
 

>From 7a79628faffedbc0f8e3a47bd1d9940a7cad4190 Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 10:48:30 -0700
Subject: [PATCH 3/4] remove worker value operands as well

---
 .../OpenACC/Utils/OpenACCUtilsTiling.cpp      |  3 ++
 .../test/Dialect/OpenACC/acc-loop-tiling.mlir | 49 +++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index 6452d7e0ed8c8..09e83af2bc686 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -108,6 +108,9 @@ createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
   if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
     elementLoop.removeWorkerAttr();
     elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    // As above for gang, also drop the worker operand values so elementLoop
+    // does not keep a dangling worker operand with no device-type attribute.
+    elementLoop.getWorkerNumOperandsMutable().clear();
   }
   rewriter.finalizeOpModification(elementLoop);
 
diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index 3b032959b1cc6..a1f80ad9defdf 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -117,6 +117,55 @@ func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
   return
 }
 
+// Regression test: a loop with WORKER(N) combined with VECTOR and a
+// multi-dim TILE clause used to produce invalid IR because createInnerLoop()
+// (via uncollapseLoops()) and removeWorkerVectorFromLoop() removed the
+// worker attributes from generated inner/tile loops without also clearing
+// the worker operand value copied onto them. This left loops with a
+// non-empty worker operand list but no worker device-type attribute,
+// which the verifier rejects ('worker operands count must match worker
+// device_type count'). Check that the pass produces valid IR, with worker
+// only preserved on the outermost tile loop and vector only on the
+// outermost element loop.
+
+// CHECK-LABEL: func.func @worker_num_with_multi_dim_tile
+// CHECK:         acc.loop worker(%{{.*}} : i32) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       worker
+// CHECK-NOT:       vector
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         worker
+// CHECK-NOT:         vector
+// CHECK:             acc.loop vector control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           worker
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             worker
+// CHECK-NOT:             vector
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @worker_num_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cw = arith.constant 4 : i32
+  acc.loop worker(%cw : i32) vector tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
 // Test unknown tile size (*) represented as -1
 // Should use default tile size (32)
 

>From 5d65b3deddac5758cb2e687b8b721beac7080478 Mon Sep 17 00:00:00 2001
From: Susan Tan <zujunt at nvidia.com>
Date: Wed, 8 Jul 2026 12:59:17 -0700
Subject: [PATCH 4/4] change to drop dummy of original

---
 .../OpenACC/Support/FIROpenACCOpsInterfaces.h | 14 ++++++++++++
 .../Support/FIROpenACCOpsInterfaces.cpp       | 20 +++++++++++++++++
 .../Support/RegisterOpenACCExtensions.cpp     | 10 +++++++++
 .../Dialect/OpenACC/OpenACCOpsInterfaces.td   | 22 +++++++++++++++++++
 .../OffloadLiveInValueCanonicalization.cpp    | 15 +++++++++++++
 5 files changed, 81 insertions(+)

diff --git a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
index a39d517412c07..31e1593e11325 100644
--- a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
+++ b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
@@ -99,6 +99,20 @@ struct OutlineRematerializationModel<fir::ConvertOp>
   bool isRematerializationCandidate(mlir::Operation *op) const;
 };
 
+/// External model for OutlineIdentityOperandOpInterface.
+/// Drops the dummy_scope operand (an identity token that must not be
+/// duplicated) from a [hl]fir.declare instance that has been sunk or
+/// rematerialized into an offload region. Alias analysis falls back to a
+/// dominance-based lookup of the enclosing dummy scope when this operand is
+/// absent, so correctness is preserved without needing to keep or clone the
+/// original operand.
+template <typename Op>
+struct OutlineIdentityOperandDeclareModel
+    : public mlir::acc::OutlineIdentityOperandOpInterface::ExternalModel<
+          OutlineIdentityOperandDeclareModel<Op>, Op> {
+  void dropOutlinedIdentityOperands(mlir::Operation *op) const;
+};
+
 /// External model for OffloadRegionOpInterface.
 /// This interface marks operations whose regions are targets for offloading
 /// and outlining.
diff --git a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
index 590619a31a33f..8e4b55fe5f4a8 100644
--- a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
@@ -143,6 +143,26 @@ bool OutlineRematerializationModel<
          fir::ConvertOp::isIntegerCompatible(outTy);
 }
 
+template <>
+void OutlineIdentityOperandDeclareModel<
+    fir::DeclareOp>::dropOutlinedIdentityOperands(mlir::Operation *op) const {
+  auto declareOp = mlir::cast<fir::DeclareOp>(op);
+  if (declareOp.getDummyScope()) {
+    declareOp.getDummyScopeMutable().clear();
+    declareOp->removeAttr(declareOp.getDummyArgNoAttrName());
+  }
+}
+
+template <>
+void OutlineIdentityOperandDeclareModel<
+    hlfir::DeclareOp>::dropOutlinedIdentityOperands(mlir::Operation *op) const {
+  auto declareOp = mlir::cast<hlfir::DeclareOp>(op);
+  if (declareOp.getDummyScope()) {
+    declareOp.getDummyScopeMutable().clear();
+    declareOp->removeAttr(declareOp.getDummyArgNoAttrName());
+  }
+}
+
 // Helper to recursively process address-of operations in derived type
 // descriptors and collect all needed fir.globals.
 static void processAddrOfOpInDerivedTypeDescriptor(
diff --git a/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp b/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
index 070208d22ba05..28fbbc7b5444c 100644
--- a/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
@@ -53,6 +53,12 @@ void registerOpenACCExtensions(mlir::DialectRegistry &registry) {
         PartialEntityAccessModel<fir::CoordinateOp>>(*ctx);
     fir::DeclareOp::attachInterface<PartialEntityAccessModel<fir::DeclareOp>>(
         *ctx);
+    // fir.declare's dummy_scope operand is an identity token (see
+    // fir.dummy_scope) that must not be duplicated when the declare is
+    // sunk/rematerialized into an offload region; it must instead be
+    // dropped from the sunk/cloned instance.
+    fir::DeclareOp::attachInterface<
+        OutlineIdentityOperandDeclareModel<fir::DeclareOp>>(*ctx);
 
     fir::AddrOfOp::attachInterface<AddressOfGlobalModel>(*ctx);
     fir::GlobalOp::attachInterface<GlobalVariableModel>(*ctx);
@@ -94,6 +100,10 @@ void registerOpenACCExtensions(mlir::DialectRegistry &registry) {
             PartialEntityAccessModel<hlfir::DesignateOp>>(*ctx);
         hlfir::DeclareOp::attachInterface<
             PartialEntityAccessModel<hlfir::DeclareOp>>(*ctx);
+        // See the fir::DeclareOp registration above for why dummy_scope must
+        // be dropped rather than duplicated.
+        hlfir::DeclareOp::attachInterface<
+            OutlineIdentityOperandDeclareModel<hlfir::DeclareOp>>(*ctx);
       });
 
   // Register CUF operation interfaces
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index a42bb7a66aae7..be7b5950eddfc 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -45,6 +45,28 @@ def PartialEntityAccessOpInterface : OpInterface<"PartialEntityAccessOpInterface
   ];
 }
 
+def OutlineIdentityOperandOpInterface
+    : OpInterface<"OutlineIdentityOperandOpInterface"> {
+  let cppNamespace = "::mlir::acc";
+
+  let description = [{
+    An interface for operations with an "identity" operand (e.g. FIR's
+    dummy scope handle) that must never be duplicated, unlike operands
+    handled by `OutlineRematerializationOpInterface`. When such an
+    operation is sunk/cloned into an offload region and the operand would
+    otherwise reference a value outside it, drop the operand instead of
+    cloning it; implementations must preserve correctness some other way
+    (e.g. FIR's dominance-based scope lookup fallback).
+  }];
+
+  let methods = [
+    InterfaceMethod<"Drop any identity operands from this operation "
+                      "instance that reference values defined outside its "
+                      "containing region.",
+                      "void", "dropOutlinedIdentityOperands", (ins), [{}]>,
+  ];
+}
+
 def AddressOfGlobalOpInterface : OpInterface<"AddressOfGlobalOpInterface"> {
   let cppNamespace = "::mlir::acc";
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
index 60285c8a02bb9..4b57354b8b5d8 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
@@ -246,6 +246,13 @@ class OffloadLiveInValueCanonicalization
       Operation *sinkOp = sinkCandidate.getDefiningOp();
       assert(sinkOp && "must have op to be considered");
       sinkOp->moveBefore(&region.front().front());
+      // Some operations (e.g. fir.declare) may carry identity operands that
+      // must not be left referencing a value defined outside the region.
+      // Unlike other operands, such identity operands cannot simply be
+      // treated as new live-ins to legalize; they must be dropped instead.
+      if (auto identityOp =
+              dyn_cast<acc::OutlineIdentityOperandOpInterface>(sinkOp))
+        identityOp.dropOutlinedIdentityOperands();
       LLVM_DEBUG(llvm::dbgs() << "\t\tSunk: " << *sinkOp << "\n");
     }
 
@@ -262,6 +269,14 @@ class OffloadLiveInValueCanonicalization
     computeTopologicalSorting(opsToRematerialize);
     for (Operation *rematerializationOp : opsToRematerialize) {
       Operation *clonedOp = builder.clone(*rematerializationOp);
+      // See the comment above: identity operands must be dropped from the
+      // clone rather than duplicated, since duplicating them would make the
+      // clone appear to belong to a different instantiation than the
+      // original operation (and any of its siblings left outside the
+      // region).
+      if (auto identityOp =
+              dyn_cast<acc::OutlineIdentityOperandOpInterface>(clonedOp))
+        identityOp.dropOutlinedIdentityOperands();
       for (auto [oldResult, newResult] : llvm::zip(
                rematerializationOp->getResults(), clonedOp->getResults())) {
         replaceAllUsesInRegionWith(oldResult, newResult, region);


From llvmlistbot at llvm.org  Wed Jul  8 13:47:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 13:47:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f550160 - [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (#208238)
Message-ID: <6a4eb74b.faf16b37.2f0e1d.a495@mx.google.com>


Author: Moazin K.
Date: 2026-07-08T13:47:00-07:00
New Revision: f550160b2d59a780ed86ae2a72cd2e3b8b3a70fa

URL: https://github.com/llvm/llvm-project/commit/f550160b2d59a780ed86ae2a72cd2e3b8b3a70fa
DIFF: https://github.com/llvm/llvm-project/commit/f550160b2d59a780ed86ae2a72cd2e3b8b3a70fa.diff

LOG: [flang][OpenACC] Attach `OpenACCLoopLocAttr` to `acc.loop` op (#208238)

Introduce an attribute `OpenACCLoopLocAttr` that can be attached to an
`acc.loop` operation to record loop locations and directive location
separately.

Attach it to `acc.loop` when building the op in flang.

Added: 
    

Modified: 
    flang/lib/Lower/OpenACC.cpp
    flang/test/Lower/OpenACC/locations.f90
    mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td

Removed: 
    


################################################################################
diff  --git a/flang/lib/Lower/OpenACC.cpp b/flang/lib/Lower/OpenACC.cpp
index e28c31f16b9d9..8c1f071a8d31d 100644
--- a/flang/lib/Lower/OpenACC.cpp
+++ b/flang/lib/Lower/OpenACC.cpp
@@ -1828,7 +1828,6 @@ static void processDoLoopBounds(
         std::pair<Fortran::semantics::SymbolRef, Fortran::semantics::SymbolRef>>
         &localSymPairs) {
   assert(loopsToProcess > 0 && "expect at least one loop");
-  locs.push_back(currentLocation); // Location of the directive
   bool isDoConcurrent = outerDoConstruct.IsDoConcurrent();
 
   if (isDoConcurrent) {
@@ -2105,7 +2104,7 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
                llvm::SmallVector<mlir::Value> &cacheOperands,
                llvm::SmallVector<mlir::Value> &reductionOperands,
                llvm::SmallVector<mlir::Type> &retTy, mlir::Value yieldValue,
-               uint64_t loopsToProcess) {
+               uint64_t loopsToProcess, bool hasDirective) {
   fir::FirOpBuilder &builder = converter.getFirOpBuilder();
 
   llvm::SmallVector<std::pair<mlir::Value, Fortran::semantics::SymbolRef>>
@@ -2151,10 +2150,26 @@ buildACCLoopOp(Fortran::lower::AbstractConverter &converter,
   addOperands(operands, operandSegments, firstprivateOperands);
   addOperands(operands, operandSegments, reductionOperands);
 
+  // list of locations to build FusedLoc to be attached to acc loop
+  llvm::SmallVector<mlir::Location> fusedLocs;
+  if (hasDirective)
+    fusedLocs.push_back(currentLocation);
+  fusedLocs.append(locs.begin(), locs.end());
+
+  // list of only loop locations used to build an `mlir::acc::LoopLocAttr` to be
+  // attached to the `FusedLoc` as metadata so that consumers of location
+  // information can separate out the directive's location from the loops'
+  // locations
+  llvm::SmallVector<mlir::LocationAttr> loopLocAttrs(locs.begin(), locs.end());
+  auto loopLocMeta = mlir::acc::LoopLocAttr::get(
+      builder.getContext(),
+      hasDirective ? mlir::LocationAttr(currentLocation) : mlir::LocationAttr{},
+      loopLocAttrs);
+
   auto loopOp = createRegionOp<mlir::acc::LoopOp, mlir::acc::YieldOp>(
-      builder, builder.getFusedLoc(locs), currentLocation, eval, operands,
-      operandSegments, /*outerCombined=*/false, retTy, yieldValue, ivTypes,
-      ivLocs);
+      builder, builder.getFusedLoc(fusedLocs, loopLocMeta), currentLocation,
+      eval, operands, operandSegments, /*outerCombined=*/false, retTy,
+      yieldValue, ivTypes, ivLocs);
   // Ensure the iv symbol is mapped to private iv SSA value for the scope of
   // the loop even if it did not appear explicitly in a PRIVATE clause (if it
   // appeared explicitly in such clause, that is also fine because duplicates
@@ -2419,7 +2434,7 @@ static mlir::acc::LoopOp createLoopOp(
       converter, currentLocation, semanticsContext, stmtCtx, outerDoConstruct,
       eval, privateOperands, dataMap, gangOperands, workerNumOperands,
       vectorOperands, tileOperands, cacheOperands, reductionOperands, retTy,
-      yieldValue, loopsToProcess);
+      yieldValue, loopsToProcess, /*hasDirective=*/true);
 
   if (!gangDeviceTypes.empty())
     loopOp.setGangAttr(builder.getArrayAttr(gangDeviceTypes));
@@ -5355,7 +5370,8 @@ mlir::Operation *Fortran::lower::genOpenACCLoopFromDoConstruct(
       converter, converter.getCurrentLocation(), semanticsContext, stmtCtx,
       doConstruct, eval, privateOperands, dataMap, gangOperands,
       workerNumOperands, vectorOperands, tileOperands, cacheOperands,
-      reductionOperands, retTy, yieldValue, loopsToProcess);
+      reductionOperands, retTy, yieldValue, loopsToProcess,
+      /*hasDirective=*/false);
 
   // Normal do loops which are not annotated with `acc loop` should be
   // left for analysis by marking with `auto`. This is the case even in the case

diff  --git a/flang/test/Lower/OpenACC/locations.f90 b/flang/test/Lower/OpenACC/locations.f90
index 46e8df0e9dac4..fe1dc5c918e2d 100644
--- a/flang/test/Lower/OpenACC/locations.f90
+++ b/flang/test/Lower/OpenACC/locations.f90
@@ -53,7 +53,7 @@ subroutine nested_acc_locations(arr1d)
     !CHECK: acc.loop
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":44:11)
-    !CHECK-NEXT: } attributes {{.*}} loc(fused["{{.*}}locations.f90":44:11, "{{.*}}locations.f90":45:5])
+    !CHECK-NEXT: } attributes {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":44:11), loops = loc("{{[^"]*}}locations.f90":45:5)>>["{{[^"]*}}locations.f90":44:11, "{{[^"]*}}locations.f90":45:5])
 
     !CHECK:        acc.yield loc("{{.*}}locations.f90":43:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":43:11)
@@ -87,7 +87,7 @@ subroutine combined_directive_locations(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":82:11, "{{.*}}locations.f90":83:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":82:11), loops = loc("{{[^"]*}}locations.f90":83:5)>>["{{[^"]*}}locations.f90":82:11, "{{[^"]*}}locations.f90":83:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":82:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":82:11)
   end subroutine
@@ -106,7 +106,7 @@ subroutine if_clause_expr_location(arr)
     !CHECK: acc.parallel
     !CHECK: acc.loop
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
-    !CHECK-NEXT: } {{.*}} loc(fused["{{.*}}locations.f90":99:11, "{{.*}}locations.f90":100:5])
+    !CHECK-NEXT: } {{.*}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":99:11), loops = loc("{{[^"]*}}locations.f90":100:5)>>["{{[^"]*}}locations.f90":99:11, "{{[^"]*}}locations.f90":100:5])
     !CHECK:      acc.yield loc("{{.*}}locations.f90":99:11)
     !CHECK-NEXT: } loc("{{.*}}locations.f90":99:11)
   end subroutine
@@ -169,7 +169,7 @@ subroutine acc_loop_fused_locations(arr)
 
 ! CHECK-LABEL: func.func @_QMacc_locationsPacc_loop_fused_locations
 ! CHECK: acc.loop
-! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused["{{.*}}locations.f90":160:11, "{{.*}}locations.f90":161:5, "{{.*}}locations.f90":162:7, "{{.*}}locations.f90":163:9])
+! CHECK: } attributes {collapse = [3]{{.*}}} loc(fused<#acc.loop_loc<directive = loc("{{[^"]*}}locations.f90":160:11), loops = loc("{{[^"]*}}locations.f90":161:5), loc("{{[^"]*}}locations.f90":162:7), loc("{{[^"]*}}locations.f90":163:9)>>["{{[^"]*}}locations.f90":160:11, "{{[^"]*}}locations.f90":161:5, "{{[^"]*}}locations.f90":162:7, "{{[^"]*}}locations.f90":163:9])
 
   subroutine data_end_locations(arr)
     real, dimension(10) :: arr
@@ -182,4 +182,19 @@ subroutine data_end_locations(arr)
     !CHECK-LABEL: acc.copyout
     !CHECK-SAME:  loc("{{.*}}locations.f90":181:11)
   end subroutine
+
+  subroutine acc_kernel_with_loop_locations(arr)
+    real, dimension(10) :: arr
+    integer :: i
+
+    !$acc kernels
+    do i = 1, 10
+      arr(i) = arr(i) + 1
+    end do
+    !$acc end kernels
+  end subroutine
+
+! CHECK-LABEL: func.func @_QMacc_locationsPacc_kernel_with_loop_locations
+! CHECK: acc.loop
+! CHECK: } attributes {{.*}} loc(fused<#acc.loop_loc<loops = loc("{{[^"]*}}locations.f90":191:5)>>["{{[^"]*}}locations.f90":191:5])
 end module

diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 26a9344fa3a15..070e6688684e6 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -384,6 +384,15 @@ def DeclareActionAttr : OpenACC_Attr<"DeclareAction", "declare_action"> {
   let assemblyFormat = "`<` struct(params) `>`";
 }
 
+// Attribute to store locations for an optional OpenACC loop directive and one
+// or more loops
+def OpenACC_LoopLocAttr : OpenACC_Attr<"LoopLoc", "loop_loc"> {
+  let summary = "Stores the OpenACC loop directive and per-loop source locations";
+  let parameters = (ins OptionalParameter<"::mlir::LocationAttr">:$directive,
+                        ArrayRefParameter<"::mlir::LocationAttr">:$loops);
+  let assemblyFormat = "`<` struct(params) `>`";
+}
+
 // Device type enumeration.
 def OpenACC_DeviceTypeNone      : I32EnumAttrCase<"None", 0, "none">;
 def OpenACC_DeviceTypeStar      : I32EnumAttrCase<"Star", 1, "star">;


        

From llvmlistbot at llvm.org  Wed Jul  8 13:47:08 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 13:47:08 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenACC] Attach
 `OpenACCLoopLocAttr` to `acc.loop` op (PR #208238)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208238@github.com>
Message-ID: <6a4eb74c.7d41f3a1.237e53.9a64@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/208238

From llvmlistbot at llvm.org  Wed Jul  8 13:48:28 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 08 Jul 2026 13:48:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] apply par dims to reductions in
 parallel regions (PR #208258)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208258@github.com>
Message-ID: <6a4eb79c.0c05e68e.1e82f1.17de@mx.google.com>


https://github.com/rscottmanley updated https://github.com/llvm/llvm-project/pull/208258

>From aad6dde1b961cf5cb2d3c57660e597ea95986f45 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Wed, 8 Jul 2026 09:16:06 -0700
Subject: [PATCH 1/2] [OpenACC] apply par dims to reductions in parallel
 regions

For reductions that come from parallel constructs, explicitly set the
GPU parallel dimensions attribute to blockXDim since they will always be
gang private.
---
 .../OpenACC/Transforms/ACCRecipeMaterialization.cpp  | 12 ++++++++++++
 .../acc-recipe-materialization-reduction.mlir        |  4 ++--
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index 398f4d4bbfdf6..d2f7d9e1cb80e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -44,6 +44,7 @@
 #include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsLoop.h"
 #include "mlir/Dialect/OpenACC/Transforms/Passes.h"
 #include "mlir/IR/Block.h"
@@ -380,6 +381,17 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
     cloneRegionIntoAccRegion(&combinerRegion, &combineRegionOp.getRegion(),
                              /*hasResult=*/false);
 
+    // For reductions that come from parallel constructs, explicitly set the
+    // GPU parallel dimensions attribute to blockXDim since they will always be
+    // gang private. GPU parallel dimensions cannot be determined for acc.loop
+    // at this point.
+    if constexpr (std::is_same_v<AccOpTy, acc::ParallelOp>) {
+      auto gpuParDimsAttr = acc::GPUParallelDimsAttr::get(b.getContext(),
+                              {acc::GPUParallelDimAttr::blockXDim(b.getContext())});
+      acc::setParDimsAttr(reductionOp, gpuParDimsAttr);
+      acc::setParDimsAttr(combineRegionOp, gpuParDimsAttr);
+    }
+
     auto setSeqParDimsForRecipeLoops = [](Region *r) {
       r->walk([](LoopLikeOpInterface loopLike) {
         loopLike->setAttr(
diff --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
index 083a6839b7d0a..6f2bafb2b6165 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
@@ -28,7 +28,7 @@ acc.reduction.recipe @reduction_add_memref_f64 : memref<f64> reduction_operator
 // CHECK-NEXT:  [[ALLOCA:%.*]] = memref.alloca() : memref<f64>
 // CHECK-NEXT:  memref.store [[ZERO]], [[ALLOCA]][]
 // CHECK-NEXT:  acc.yield {{.*}}
-// CHECK:       } {{.*}}acc.var_name = #acc.var_name<"tmp">
+// CHECK:       } {{.*}}acc.par_dims = #acc<par_dims[block_x]>, acc.var_name = #acc.var_name<"tmp">
 // CHECK:       memref.load [[PRIVATE]][]
 // CHECK:       memref.store {{.*}}, [[PRIVATE]][]
 // CHECK:       acc.reduction_combine_region [[PRIVATE]] into [[REDUCVAR:%.*]] : memref<f64> {
@@ -36,7 +36,7 @@ acc.reduction.recipe @reduction_add_memref_f64 : memref<f64> reduction_operator
 // CHECK-NEXT:  [[LOADPRIV:%.*]] = memref.load [[PRIVATE]][]
 // CHECK-NEXT:  [[COMBINE:%.*]] = arith.addf [[LOADVAR]], [[LOADPRIV]]
 // CHECK-NEXT:  memref.store [[COMBINE]], [[REDUCVAR]][]
-// CHECK-NEXT:  }
+// CHECK-NEXT:  } {acc.par_dims = #acc<par_dims[block_x]>}
 // CHECK-NEXT:  memref.dealloc [[PRIVATE]] : memref<f64>
 // CHECK:       acc.yield
 

>From d3aac4c4088dde504e61bc2e2b17ec9e378c45e7 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Wed, 8 Jul 2026 09:41:18 -0700
Subject: [PATCH 2/2] format

---
 .../Transforms/ACCRecipeMaterialization.cpp   | 49 +++++++++++--------
 1 file changed, 29 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index d2f7d9e1cb80e..d2e8d77c168d1 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -43,6 +43,7 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsLoop.h"
@@ -182,10 +183,11 @@ class ACCRecipeMaterialization
   void removeRecipe(OpTy op, ModuleOp moduleOp) const;
   template <typename OpTy, typename RecipeOpTy, typename AccOpTy>
   LogicalResult materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
-                            acc::OpenACCSupport &accSupport) const;
+                            acc::OpenACCSupport &accSupport,
+                            acc::ACCToGPUMappingPolicy &policy) const;
   template <typename OpTy>
-  LogicalResult materializeForACCOp(OpTy accOp,
-                                    acc::OpenACCSupport &accSupport) const;
+  LogicalResult materializeForACCOp(OpTy accOp, acc::OpenACCSupport &accSupport,
+                                    acc::ACCToGPUMappingPolicy &policy) const;
 };
 
 void ACCRecipeMaterialization::handleFirstprivateMapping(
@@ -221,9 +223,9 @@ void ACCRecipeMaterialization::removeRecipe(OpTy op, ModuleOp moduleOp) const {
 }
 
 template <typename OpTy, typename RecipeOpTy, typename AccOpTy>
-LogicalResult
-ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
-                                      acc::OpenACCSupport &accSupport) const {
+LogicalResult ACCRecipeMaterialization::materialize(
+    OpTy op, RecipeOpTy recipe, AccOpTy accOp, acc::OpenACCSupport &accSupport,
+    acc::ACCToGPUMappingPolicy &policy) const {
   Region &region = accOp.getRegion();
   Value origPtr = op.getVar();
   Value accPtr = op.getAccVar();
@@ -381,22 +383,25 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
     cloneRegionIntoAccRegion(&combinerRegion, &combineRegionOp.getRegion(),
                              /*hasResult=*/false);
 
+    auto ctx = b.getContext();
+
     // For reductions that come from parallel constructs, explicitly set the
     // GPU parallel dimensions attribute to blockXDim since they will always be
     // gang private. GPU parallel dimensions cannot be determined for acc.loop
     // at this point.
     if constexpr (std::is_same_v<AccOpTy, acc::ParallelOp>) {
-      auto gpuParDimsAttr = acc::GPUParallelDimsAttr::get(b.getContext(),
-                              {acc::GPUParallelDimAttr::blockXDim(b.getContext())});
-      acc::setParDimsAttr(reductionOp, gpuParDimsAttr);
-      acc::setParDimsAttr(combineRegionOp, gpuParDimsAttr);
+      auto parDimsAttr = acc::GPUParallelDimsAttr::get(
+          ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+      acc::setParDimsAttr(reductionOp, parDimsAttr);
+      acc::setParDimsAttr(combineRegionOp, parDimsAttr);
     }
 
-    auto setSeqParDimsForRecipeLoops = [](Region *r) {
-      r->walk([](LoopLikeOpInterface loopLike) {
-        loopLike->setAttr(
-            acc::GPUParallelDimsAttr::name,
-            acc::GPUParallelDimsAttr::seq(loopLike->getContext()));
+    // Set sequential parallel dimensions attribute for loops in the recipe.
+    auto seqParDimsAttr =
+        acc::GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
+    auto setSeqParDimsForRecipeLoops = [&](Region *r) {
+      r->walk([&](LoopLikeOpInterface loopLike) {
+        acc::setParDimsAttr(loopLike, seqParDimsAttr);
       });
     };
     setSeqParDimsForRecipeLoops(&reductionOp.getRegion());
@@ -418,7 +423,8 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
 
 template <typename OpTy>
 LogicalResult ACCRecipeMaterialization::materializeForACCOp(
-    OpTy accOp, acc::OpenACCSupport &accSupport) const {
+    OpTy accOp, acc::OpenACCSupport &accSupport,
+    acc::ACCToGPUMappingPolicy &policy) const {
   assert(isa<ACC_COMPUTE_CONSTRUCT_AND_LOOP_OPS>(accOp));
 
   if (!accOp.getFirstprivateOperands().empty()) {
@@ -434,7 +440,8 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << firstprivateOp << "\n"
                               << symbolRef << "\n");
       handleFirstprivateMapping(firstprivateOp);
-      if (failed(materialize(firstprivateOp, recipeOp, accOp, accSupport)))
+      if (failed(
+              materialize(firstprivateOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -451,7 +458,7 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       auto recipeOp = cast<acc::PrivateRecipeOp>(decl);
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << privateOp << "\n"
                               << symbolRef << "\n");
-      if (failed(materialize(privateOp, recipeOp, accOp, accSupport)))
+      if (failed(materialize(privateOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -468,7 +475,7 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       auto recipeOp = cast<acc::ReductionRecipeOp>(decl);
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << reductionOp << "\n"
                               << symbolRef << "\n");
-      if (failed(materialize(reductionOp, recipeOp, accOp, accSupport)))
+      if (failed(materialize(reductionOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -479,6 +486,8 @@ void ACCRecipeMaterialization::runOnOperation() {
   ModuleOp moduleOp = getOperation();
   acc::OpenACCSupport &accSupport = getAnalysis<acc::OpenACCSupport>();
 
+  acc::DefaultACCToGPUMappingPolicy policy;
+
   // Materialize all recipes for all compute constructs and loop constructs.
   bool anyFailed = false;
   moduleOp.walk([&](Operation *op) {
@@ -486,7 +495,7 @@ void ACCRecipeMaterialization::runOnOperation() {
       return;
     TypeSwitch<Operation *>(op).Case<ACC_COMPUTE_CONSTRUCT_AND_LOOP_OPS>(
         [&](auto constructOp) {
-          if (failed(materializeForACCOp(constructOp, accSupport)))
+          if (failed(materializeForACCOp(constructOp, accSupport, policy)))
             anyFailed = true;
         });
   });


From llvmlistbot at llvm.org  Wed Jul  8 13:51:52 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 08 Jul 2026 13:51:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4eb868.fdc477d6.23591.2f87@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.


https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Wed Jul  8 13:53:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 13:53:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix host fallback for
	acc.atomic.update (PR #207597)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207597@github.com>
Message-ID: <6a4eb8cf.ac469dc7.3a57b1.186d@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
index 5f41a1175..aaa364bf8 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCSpecializeForHost.cpp
@@ -147,8 +147,8 @@ public:
         rewriter.clone(op, mapping);
       auto yieldOp = cast<acc::YieldOp>(block.getTerminator());
       Value result = mapping.lookup(yieldOp.getOperand(0));
-      if (!ptrLikeType.genStore(rewriter, atomicUpdateOp.getLoc(),
-                                result, xTyped)) {
+      if (!ptrLikeType.genStore(rewriter, atomicUpdateOp.getLoc(), result,
+                                xTyped)) {
         accSupport.emitNYI(atomicUpdateOp.getLoc(),
                            "failed to generate store for atomic update");
         return failure();

``````````

</details>


https://github.com/llvm/llvm-project/pull/207597

From llvmlistbot at llvm.org  Wed Jul  8 17:07:23 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 08 Jul 2026 17:07:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Give each mlir::Block a stable ID
 within its parent region (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4ee63b.fdc477d6.23591.18c1@mx.google.com>


https://github.com/joker-eph approved this pull request.


https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Wed Jul  8 19:06:10 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 08 Jul 2026 19:06:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
Message-ID: <llvm.org/llvm/llvm-project/pull/208365@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/208365

 **Summary**

  During backward layout propagation a value can be demanded by multiple consumers with conflicting
  layouts. Previously LayoutInfo::meet kept whichever demand was assigned first, so the winner depended on
  the dataflow worklist's visitation order — nondeterministic and often not what we want.

  This changes meet to prefer the layout demanded by the consumer nearer to the producer in program order.
  This tends to preserve a consumer's layout as far up the def chain as possible, reducing layout
  conversions. It is a heuristic hint, not a proven optimum.

  - Each op gets a program-order index from a pre-order walk (matching printed-IR order), so a use inside
  an scf.for body is nearer than a use after the loop.
  - LayoutInfo carries the demanding op's index in a new programOrder field. visitOperation sets a
  file-scoped currentProgramOrder, and the single-arg LayoutInfo constructor stamps it automatically.
  - meet keeps the smaller programOrder; ties keep lhs.
  - Removes a dead LayoutInfo::transpose declaration.
  - Adds truncf_prefers_nearer_user in propagate-layout-inst-data.mlir.
  - Updates the multiple-use tests in propagate-layout.mlir (scatter_ops_preserve_load_perm_layout,
  binary_op_multiple_uses, if_multiple_uses) to reflect that the nearer consumer's layout now wins.
  - a minor layout fix for simple_mxfp_gemm.mlir

>From c589918d9f4111e049d6165a56fb7fc2bfa6d344 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 8 Jul 2026 23:30:49 +0000
Subject: [PATCH 1/4] [MLIR][XeGPU] Prefer slice layout in LayoutInfo::meet

During backward layout propagation a value can receive different
candidate layouts from its consumers. Previously LayoutInfo::meet kept
whichever layout was assigned first, so a value feeding both a
slice-producing consumer (e.g. a broadcast whose inferred source layout
is a slice) and a plain-layout consumer (e.g. a dpas_mx scale operand)
could end up with the plain layout depending on visitation order.

meet() now prefers the slice layout when both sides are assigned
(falling back to lhs when both/neither are slices). For that preference
to take effect, operator== compares the stored layout (via isEqualTo)
rather than only the assigned bit; otherwise the dataflow framework
treats the refined result as a no-op and discards it. The refinement
only moves a plain layout toward a slice layout, so it stays monotonic.

Adds a propagate-layout-inst-data test and updates the a-operand
lane_data in the simple_mxfp_gemm integration test to match the HW
constraint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 25 +++++++++++-----
 .../XeGPU/propagate-layout-inst-data.mlir     | 29 +++++++++++++++++++
 .../Dialect/XeGPU/WG/simple_mxfp_gemm.mlir    |  2 +-
 3 files changed, 47 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 64d0d8063b7ff..ff9239b637a10 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -71,12 +71,13 @@ namespace {
 /// Given this, LayoutInfo  satisifies the following properties:
 ///  1) A LayoutInfo value can be in one of two states - `assigned` or `not
 ///  assigned`.
-///  2) Two LayoutInfo values are equal if they are both assigned or
-///  both not assigned. The concrete value of assigned state does not matter.
+///  2) Two LayoutInfo values are equal if they are both not assigned, or both
+///  assigned with the same layout.
 ///  3) The meet operator works as follows:
-///     - If current state is assigned, return the current state. (already
-///     a unique layout is assigned. don't change it)
-///     - Otherwise, return the other state.
+///     - If only one side is assigned, return that side.
+///     - If both sides are assigned, prefer the side carrying a slice layout.
+///       If both (or neither) are slice layouts, prefer the lhs (current
+///       state) so an already assigned unique layout is not changed.
 
 struct LayoutInfo {
 private:
@@ -86,10 +87,14 @@ struct LayoutInfo {
   LayoutInfo() = default;
   LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
 
-  // Two lattice values are equal if they have `some` layout. The actual
-  // content of the layout does not matter.
+  // Two lattice values are equal if they are both unassigned, or both assigned
+  // with the same layout.
   bool operator==(const LayoutInfo &other) const {
-    return this->isAssigned() == other.isAssigned();
+    if (isAssigned() != other.isAssigned())
+      return false;
+    if (!isAssigned())
+      return true;
+    return storage.isEqualTo(other.storage);
   }
 
   static LayoutInfo meet(const LayoutInfo &lhs, const LayoutInfo &rhs);
@@ -141,6 +146,10 @@ void LayoutInfo::print(raw_ostream &os) const {
 LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
   if (!lhs.isAssigned())
     return rhs;
+  if (!rhs.isAssigned())
+    return lhs;
+  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+    return rhs;
   return lhs;
 }
 
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 515c59db72819..6bf7fff810006 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -653,3 +653,32 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
   return
 }
 }
+
+// -----
+// A value with two consumers: one back-propagates a plain layout (the store_nd
+// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
+// chain feeding the reduction result). `meet` must prefer the slice layout, and
+// the inst_data / lane_layout / lane_data fields must be preserved on it.
+gpu.module @test {
+  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK: %[[TRUNC:.*]] = arith.truncf
+  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
+  // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+      %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
+    %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
+    %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>
+    %load = xegpu.load_nd %tdesc[0, 0] : !xegpu.tensor_desc<32x128xbf16> -> vector<32x128xbf16>
+    %sc1 = vector.shape_cast %load : vector<32x128xbf16> to vector<32x4x32xbf16>
+    %red = vector.multi_reduction <maximumf>, %sc1, %cst [2] : vector<32x4x32xbf16> to vector<32x4xbf16>
+    %trunc = arith.truncf %red : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+    %bcast = vector.broadcast %trunc : vector<32x4xf8E8M0FNU> to vector<32x32x4xf8E8M0FNU>
+    %bcast2 = vector.transpose %bcast, [1, 2, 0] : vector<32x32x4xf8E8M0FNU> to vector<32x4x32xf8E8M0FNU>
+    %sc2 = vector.shape_cast %bcast2 : vector<32x4x32xf8E8M0FNU> to vector<32x128xf8E8M0FNU>
+    %tdesc_red = xegpu.create_nd_tdesc %dst_red : memref<32x128xf8E8M0FNU> -> !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    xegpu.store_nd %sc2, %tdesc_red[0, 0] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<32x128xf8E8M0FNU>, !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    %tdesc_plain = xegpu.create_nd_tdesc %dst_plain : memref<32x4xf8E8M0FNU> -> !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    xegpu.store_nd %trunc, %tdesc_plain[0, 0] <{layout = #xegpu.layout<inst_data = [8, 2], lane_layout = [8, 1], lane_data = [1, 1]>}> : vector<32x4xf8E8M0FNU>, !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    gpu.return
+  }
+}
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..9bfb783299055 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -8,7 +8,7 @@
 
 // XFAIL: *
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>

>From c31ad5f6b4f3d97c50c91e62737b39cd07a23ecf Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:55:14 +0000
Subject: [PATCH 2/4] [MLIR][XeGPU] Prefer nearer consumer's layout in
 LayoutInfo::meet

When a value is demanded by multiple consumers during backward layout
propagation, prefer the layout of the consumer that is nearer to the
producer in program order. This tends to preserve a consumer's layout as
far up the def chain as possible, reducing layout conversions. It is a
hint, not an optimum.

Each op is assigned a program-order index via a pre-order walk (matching
printed-IR order), so a use inside an scf.for body is nearer than a use
after the loop. LayoutInfo carries the demanding op's index in a new
`programOrder` field, stamped by the single-argument constructor from a
file-scoped `currentProgramOrder` that visitOperation sets, so the ~30
meet() call sites are unchanged. `programOrder` is not propagated (each
visited op stamps its own index) and is excluded from operator==.

This replaces the earlier slice-layout preference in meet(), which is
now redundant: distinct users always have distinct indices, so program
order decides every real conflict. Also removes the dead
LayoutInfo::transpose declaration.

Updates the multiple-uses propagate-layout tests to reflect that the
nearer consumer's layout now wins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 75 +++++++++++++++++--
 mlir/test/Dialect/XeGPU/propagate-layout.mlir | 20 ++---
 2 files changed, 77 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index ff9239b637a10..72b10e1d3265f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -30,6 +30,7 @@
 #include "mlir/Interfaces/LoopLikeInterface.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
@@ -38,6 +39,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/LogicalResult.h"
 #include "llvm/Support/raw_ostream.h"
+#include <limits>
 
 namespace mlir {
 namespace xegpu {
@@ -58,6 +60,14 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
+/// Program-order index of the op currently being visited by the backward
+/// analysis. `visitOperation` sets this before dispatching, and the
+/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
+/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
+/// change. A larger index means farther from the producer; the sentinel max
+/// marks demands with no associated op (e.g. exit state).
+static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
+
 /// Helper class for tracking the analysis state of an mlir value. For layout
 /// propagation, the analysis state is simply the distribution layout of
 /// each value. The distribution layout information is encapsulated using
@@ -75,20 +85,34 @@ namespace {
 ///  assigned with the same layout.
 ///  3) The meet operator works as follows:
 ///     - If only one side is assigned, return that side.
-///     - If both sides are assigned, prefer the side carrying a slice layout.
-///       If both (or neither) are slice layouts, prefer the lhs (current
-///       state) so an already assigned unique layout is not changed.
+///     - If both sides are assigned, prefer the layout demanded by the user
+///       that is nearer to the producer in program order (smaller
+///       `programOrder`); on a tie keep lhs.
+///
+/// The `programOrder` field records the program-order index of the consumer op
+/// that demanded the layout (see `currentProgramOrder`). During this backward
+/// analysis a value can be demanded by several users; keeping the nearest one
+/// tends to preserve a consumer's layout as far up the def chain as possible,
+/// minimizing layout conversions. This is a hint, not an optimum.
+/// `programOrder` is never propagated up the chain - each visited op stamps its
+/// own index - so it is deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
   xegpu::DistributeLayoutAttr storage = nullptr;
+  // Program-order index of the consumer op that demanded this layout. Smaller
+  // means nearer to the producer. Unassigned/unknown demands sort last.
+  int64_t programOrder = std::numeric_limits<int64_t>::max();
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
+      : storage(layout), programOrder(programOrder) {}
 
   // Two lattice values are equal if they are both unassigned, or both assigned
-  // with the same layout.
+  // with the same layout. `programOrder` is intentionally excluded: it is not
+  // propagated, so a pure order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
     if (isAssigned() != other.isAssigned())
       return false;
@@ -105,8 +129,6 @@ struct LayoutInfo {
 
   bool isAssigned() const { return storage != nullptr; }
 
-  LayoutInfo transpose(ArrayRef<int64_t> permutation) const;
-
   SmallVector<int> getLaneLayout() const;
 
   SmallVector<int> getLaneData() const;
@@ -135,6 +157,11 @@ struct LayoutInfo {
   void set(const xegpu::DistributeLayoutAttr &layout) { storage = layout; }
 };
 
+// Stamp every demand pushed by the current op with that op's program-order
+// index so `meet` can prefer the nearest consumer.
+LayoutInfo::LayoutInfo(const xegpu::DistributeLayoutAttr &layout)
+    : storage(layout), programOrder(currentProgramOrder) {}
+
 void LayoutInfo::print(raw_ostream &os) const {
   if (isAssigned()) {
     os << storage;
@@ -148,7 +175,10 @@ LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
     return rhs;
   if (!rhs.isAssigned())
     return lhs;
-  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+  // Prefer the demand from the user nearer to the producer in program order.
+  // Distinct users always have distinct indices, so this decides every
+  // real conflict; on a tie (same op, or both unknown) keep lhs.
+  if (rhs.programOrder < lhs.programOrder)
     return rhs;
   return lhs;
 }
@@ -185,6 +215,15 @@ class LayoutInfoPropagation
 private:
   xegpu::LayoutKind layoutKind;
   unsigned indexBitWidth;
+
+  // Program-order index of every op, built lazily on first use via a pre-order
+  // walk of the top-level module/function (matching printed-IR order). Used to
+  // tell which consumer of a value is nearer to its producer.
+  DenseMap<Operation *, int64_t> programOrder;
+  // Returns the program-order index of `op`, populating `programOrder` from
+  // `op`'s top-level ancestor on first call.
+  int64_t getProgramOrder(Operation *op);
+
   void visitDpasOp(xegpu::DpasOp dpas, ArrayRef<LayoutInfoLattice *> operands,
                    ArrayRef<const LayoutInfoLattice *> results);
 
@@ -300,9 +339,29 @@ class LayoutInfoPropagation
 };
 } // namespace
 
+int64_t LayoutInfoPropagation::getProgramOrder(Operation *op) {
+  auto it = programOrder.find(op);
+  if (it != programOrder.end())
+    return it->second;
+  // First time we see this op's tree: number every op under its top-level
+  // ancestor in pre-order (i.e. printed-IR order). Nested ops (e.g. inside an
+  // scf.for body) get an index between their parent and the parent's next
+  // sibling, so a use inside a loop is "nearer" than a use after it.
+  Operation *root = op;
+  while (root->getParentOp())
+    root = root->getParentOp();
+  int64_t counter = 0;
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *o) { programOrder[o] = counter++; });
+  return programOrder.lookup(op);
+}
+
 LogicalResult LayoutInfoPropagation::visitOperation(
     Operation *op, ArrayRef<LayoutInfoLattice *> operands,
     ArrayRef<const LayoutInfoLattice *> results) {
+  // Stamp demands pushed by this op with its program-order index so `meet` can
+  // prefer the nearest consumer.
+  currentProgramOrder = getProgramOrder(op);
   TypeSwitch<Operation *>(op)
       .Case(
           [&](xegpu::DpasOp dpasOp) { visitDpasOp(dpasOp, operands, results); })
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 1a741ba21e128..49006e1ef9507 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -219,8 +219,8 @@ func.func @scatter_ops_custom_perm_layout(%src: memref<256xf16>) {
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_preserve_load_perm_layout(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<256xf16>) {
-// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<true> : vector<16xi1>
-// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<12> : vector<16xindex>
+// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<true> : vector<16xi1>
+// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<12> : vector<16xindex>
 // CHECK: %[[LOAD_VEC:.*]] = xegpu.load %[[ARG0]][%[[OFFSETS]]], %[[MASK]]
 // CHECK-SAME: memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16xf16>
 // CHECK: %[[ADD_RES:.*]] = arith.addf %[[LOAD_VEC]], %[[LOAD_VEC]] {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} : vector<16xf16>
@@ -341,10 +341,10 @@ func.func @binary_op_one_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.
 gpu.module @test {
 // CHECK-LABEL: func.func @binary_op_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
-// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf16>
+// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>} : vector<16x16xf16>
 // CHECK: %[[T3:.*]] = xegpu.dpas %{{.*}}, %[[T2]] {layout_a = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, layout_b = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>, layout_cd = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
 // CHECK-NEXT: xegpu.store_nd %[[T3]], %[[ARG2]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x16xf32>, !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
 // CHECK-NEXT: xegpu.store_nd %[[T2]], %[[ARG3]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>, !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
@@ -429,18 +429,18 @@ func.func @if_single_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tens
 gpu.module @test {
 // CHECK-LABEL: func.func @if_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: i1, %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG4:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
 // CHECK: %[[T1:.*]] = scf.if %[[ARG2]] -> (vector<16x16xf16>) {
-// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T3]] : vector<16x16xf16>
 // CHECK-NEXT:     } else {
-// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T4]] : vector<16x16xf16>
-// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}
+// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
 func.func @if_multiple_uses(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor_desc<16x16xf16>, %arg2: i1, %arg3: !xegpu.tensor_desc<8x16xf32>, %arg4: !xegpu.tensor_desc<16x16xf16>) {
   %0 = xegpu.load_nd %arg0[0, 0]  : !xegpu.tensor_desc<8x16xf16> -> vector<8x16xf16>
   %1 = scf.if %arg2 -> (vector<16x16xf16>) {

>From 3c4ada8b1f50d0d9c48f3c8543aa0662f1f2164c Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:58:24 +0000
Subject: [PATCH 3/4] [MLIR][XeGPU] Reframe propagate-layout test as
 nearer-user preference

Rename truncf_prefers_slice to truncf_prefers_nearer_user and update its
comment: on this branch %trunc keeps the broadcast chain's slice layout
because that consumer is nearer in program order, not because meet()
prefers slice layouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../Dialect/XeGPU/propagate-layout-inst-data.mlir    | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 6bf7fff810006..47471b890d552 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -655,16 +655,16 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
 }
 
 // -----
-// A value with two consumers: one back-propagates a plain layout (the store_nd
-// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
-// chain feeding the reduction result). `meet` must prefer the slice layout, and
-// the inst_data / lane_layout / lane_data fields must be preserved on it.
+// %trunc has two consumers: the broadcast/transpose chain (nearer in program
+// order, back-propagates a slice layout) and the store_nd of %trunc (farther,
+// back-propagates a plain layout). `meet` must keep the nearer consumer's
+// layout, preserving its inst_data / lane_layout / lane_data fields.
 gpu.module @test {
-  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK-LABEL: truncf_prefers_nearer_user
   // CHECK: %[[TRUNC:.*]] = arith.truncf
   // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
   // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
-  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+  gpu.func @truncf_prefers_nearer_user(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
       %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
     %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
     %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>

>From 9c4de99b6094f502c82dcd411b8eca2c1c18141d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 02:00:09 +0000
Subject: [PATCH 4/4] git format

---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp      | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 72b10e1d3265f..75e8db468f361 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override {};
+  void visitBranchOperand(OpOperand &operand) override{};
 
-  void visitCallOperand(OpOperand &operand) override {};
+  void visitCallOperand(OpOperand &operand) override{};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override {};
+                               ArrayRef<BlockArgument> arguments) override{};
 
-  void visitExternalCall(CallOpInterface call,
-                         ArrayRef<LayoutInfoLattice *> operands,
-                         ArrayRef<const LayoutInfoLattice *> results) override {
-  };
+  void
+  visitExternalCall(CallOpInterface call,
+                    ArrayRef<LayoutInfoLattice *> operands,
+                    ArrayRef<const LayoutInfoLattice *> results) override{};
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());


From llvmlistbot at llvm.org  Wed Jul  8 19:06:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:06:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f0239.e9d484b9.15e549.c0c3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

 **Summary**

  During backward layout propagation a value can be demanded by multiple consumers with conflicting
  layouts. Previously LayoutInfo::meet kept whichever demand was assigned first, so the winner depended on
  the dataflow worklist's visitation order — nondeterministic and often not what we want.

  This changes meet to prefer the layout demanded by the consumer nearer to the producer in program order.
  This tends to preserve a consumer's layout as far up the def chain as possible, reducing layout
  conversions. It is a heuristic hint, not a proven optimum.

  - Each op gets a program-order index from a pre-order walk (matching printed-IR order), so a use inside
  an scf.for body is nearer than a use after the loop.
  - LayoutInfo carries the demanding op's index in a new programOrder field. visitOperation sets a
  file-scoped currentProgramOrder, and the single-arg LayoutInfo constructor stamps it automatically.
  - meet keeps the smaller programOrder; ties keep lhs.
  - Removes a dead LayoutInfo::transpose declaration.
  - Adds truncf_prefers_nearer_user in propagate-layout-inst-data.mlir.
  - Updates the multiple-use tests in propagate-layout.mlir (scatter_ops_preserve_load_perm_layout,
  binary_op_multiple_uses, if_multiple_uses) to reflect that the nearer consumer's layout now wins.
  - a minor layout fix for simple_mxfp_gemm.mlir

---
Full diff: https://github.com/llvm/llvm-project/pull/208365.diff


4 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp (+86-18) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir (+29) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout.mlir (+10-10) 
- (modified) mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir (+1-1) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 64d0d8063b7ff..75e8db468f361 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -30,6 +30,7 @@
 #include "mlir/Interfaces/LoopLikeInterface.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
@@ -38,6 +39,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/LogicalResult.h"
 #include "llvm/Support/raw_ostream.h"
+#include <limits>
 
 namespace mlir {
 namespace xegpu {
@@ -58,6 +60,14 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
+/// Program-order index of the op currently being visited by the backward
+/// analysis. `visitOperation` sets this before dispatching, and the
+/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
+/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
+/// change. A larger index means farther from the producer; the sentinel max
+/// marks demands with no associated op (e.g. exit state).
+static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
+
 /// Helper class for tracking the analysis state of an mlir value. For layout
 /// propagation, the analysis state is simply the distribution layout of
 /// each value. The distribution layout information is encapsulated using
@@ -71,25 +81,44 @@ namespace {
 /// Given this, LayoutInfo  satisifies the following properties:
 ///  1) A LayoutInfo value can be in one of two states - `assigned` or `not
 ///  assigned`.
-///  2) Two LayoutInfo values are equal if they are both assigned or
-///  both not assigned. The concrete value of assigned state does not matter.
+///  2) Two LayoutInfo values are equal if they are both not assigned, or both
+///  assigned with the same layout.
 ///  3) The meet operator works as follows:
-///     - If current state is assigned, return the current state. (already
-///     a unique layout is assigned. don't change it)
-///     - Otherwise, return the other state.
+///     - If only one side is assigned, return that side.
+///     - If both sides are assigned, prefer the layout demanded by the user
+///       that is nearer to the producer in program order (smaller
+///       `programOrder`); on a tie keep lhs.
+///
+/// The `programOrder` field records the program-order index of the consumer op
+/// that demanded the layout (see `currentProgramOrder`). During this backward
+/// analysis a value can be demanded by several users; keeping the nearest one
+/// tends to preserve a consumer's layout as far up the def chain as possible,
+/// minimizing layout conversions. This is a hint, not an optimum.
+/// `programOrder` is never propagated up the chain - each visited op stamps its
+/// own index - so it is deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
   xegpu::DistributeLayoutAttr storage = nullptr;
+  // Program-order index of the consumer op that demanded this layout. Smaller
+  // means nearer to the producer. Unassigned/unknown demands sort last.
+  int64_t programOrder = std::numeric_limits<int64_t>::max();
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
+      : storage(layout), programOrder(programOrder) {}
 
-  // Two lattice values are equal if they have `some` layout. The actual
-  // content of the layout does not matter.
+  // Two lattice values are equal if they are both unassigned, or both assigned
+  // with the same layout. `programOrder` is intentionally excluded: it is not
+  // propagated, so a pure order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
-    return this->isAssigned() == other.isAssigned();
+    if (isAssigned() != other.isAssigned())
+      return false;
+    if (!isAssigned())
+      return true;
+    return storage.isEqualTo(other.storage);
   }
 
   static LayoutInfo meet(const LayoutInfo &lhs, const LayoutInfo &rhs);
@@ -100,8 +129,6 @@ struct LayoutInfo {
 
   bool isAssigned() const { return storage != nullptr; }
 
-  LayoutInfo transpose(ArrayRef<int64_t> permutation) const;
-
   SmallVector<int> getLaneLayout() const;
 
   SmallVector<int> getLaneData() const;
@@ -130,6 +157,11 @@ struct LayoutInfo {
   void set(const xegpu::DistributeLayoutAttr &layout) { storage = layout; }
 };
 
+// Stamp every demand pushed by the current op with that op's program-order
+// index so `meet` can prefer the nearest consumer.
+LayoutInfo::LayoutInfo(const xegpu::DistributeLayoutAttr &layout)
+    : storage(layout), programOrder(currentProgramOrder) {}
+
 void LayoutInfo::print(raw_ostream &os) const {
   if (isAssigned()) {
     os << storage;
@@ -141,6 +173,13 @@ void LayoutInfo::print(raw_ostream &os) const {
 LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
   if (!lhs.isAssigned())
     return rhs;
+  if (!rhs.isAssigned())
+    return lhs;
+  // Prefer the demand from the user nearer to the producer in program order.
+  // Distinct users always have distinct indices, so this decides every
+  // real conflict; on a tie (same op, or both unknown) keep lhs.
+  if (rhs.programOrder < lhs.programOrder)
+    return rhs;
   return lhs;
 }
 
@@ -176,6 +215,15 @@ class LayoutInfoPropagation
 private:
   xegpu::LayoutKind layoutKind;
   unsigned indexBitWidth;
+
+  // Program-order index of every op, built lazily on first use via a pre-order
+  // walk of the top-level module/function (matching printed-IR order). Used to
+  // tell which consumer of a value is nearer to its producer.
+  DenseMap<Operation *, int64_t> programOrder;
+  // Returns the program-order index of `op`, populating `programOrder` from
+  // `op`'s top-level ancestor on first call.
+  int64_t getProgramOrder(Operation *op);
+
   void visitDpasOp(xegpu::DpasOp dpas, ArrayRef<LayoutInfoLattice *> operands,
                    ArrayRef<const LayoutInfoLattice *> results);
 
@@ -272,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override {};
+  void visitBranchOperand(OpOperand &operand) override{};
 
-  void visitCallOperand(OpOperand &operand) override {};
+  void visitCallOperand(OpOperand &operand) override{};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override {};
+                               ArrayRef<BlockArgument> arguments) override{};
 
-  void visitExternalCall(CallOpInterface call,
-                         ArrayRef<LayoutInfoLattice *> operands,
-                         ArrayRef<const LayoutInfoLattice *> results) override {
-  };
+  void
+  visitExternalCall(CallOpInterface call,
+                    ArrayRef<LayoutInfoLattice *> operands,
+                    ArrayRef<const LayoutInfoLattice *> results) override{};
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());
@@ -291,9 +339,29 @@ class LayoutInfoPropagation
 };
 } // namespace
 
+int64_t LayoutInfoPropagation::getProgramOrder(Operation *op) {
+  auto it = programOrder.find(op);
+  if (it != programOrder.end())
+    return it->second;
+  // First time we see this op's tree: number every op under its top-level
+  // ancestor in pre-order (i.e. printed-IR order). Nested ops (e.g. inside an
+  // scf.for body) get an index between their parent and the parent's next
+  // sibling, so a use inside a loop is "nearer" than a use after it.
+  Operation *root = op;
+  while (root->getParentOp())
+    root = root->getParentOp();
+  int64_t counter = 0;
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *o) { programOrder[o] = counter++; });
+  return programOrder.lookup(op);
+}
+
 LogicalResult LayoutInfoPropagation::visitOperation(
     Operation *op, ArrayRef<LayoutInfoLattice *> operands,
     ArrayRef<const LayoutInfoLattice *> results) {
+  // Stamp demands pushed by this op with its program-order index so `meet` can
+  // prefer the nearest consumer.
+  currentProgramOrder = getProgramOrder(op);
   TypeSwitch<Operation *>(op)
       .Case(
           [&](xegpu::DpasOp dpasOp) { visitDpasOp(dpasOp, operands, results); })
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 515c59db72819..47471b890d552 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -653,3 +653,32 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
   return
 }
 }
+
+// -----
+// %trunc has two consumers: the broadcast/transpose chain (nearer in program
+// order, back-propagates a slice layout) and the store_nd of %trunc (farther,
+// back-propagates a plain layout). `meet` must keep the nearer consumer's
+// layout, preserving its inst_data / lane_layout / lane_data fields.
+gpu.module @test {
+  // CHECK-LABEL: truncf_prefers_nearer_user
+  // CHECK: %[[TRUNC:.*]] = arith.truncf
+  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
+  // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+  gpu.func @truncf_prefers_nearer_user(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+      %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
+    %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
+    %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>
+    %load = xegpu.load_nd %tdesc[0, 0] : !xegpu.tensor_desc<32x128xbf16> -> vector<32x128xbf16>
+    %sc1 = vector.shape_cast %load : vector<32x128xbf16> to vector<32x4x32xbf16>
+    %red = vector.multi_reduction <maximumf>, %sc1, %cst [2] : vector<32x4x32xbf16> to vector<32x4xbf16>
+    %trunc = arith.truncf %red : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+    %bcast = vector.broadcast %trunc : vector<32x4xf8E8M0FNU> to vector<32x32x4xf8E8M0FNU>
+    %bcast2 = vector.transpose %bcast, [1, 2, 0] : vector<32x32x4xf8E8M0FNU> to vector<32x4x32xf8E8M0FNU>
+    %sc2 = vector.shape_cast %bcast2 : vector<32x4x32xf8E8M0FNU> to vector<32x128xf8E8M0FNU>
+    %tdesc_red = xegpu.create_nd_tdesc %dst_red : memref<32x128xf8E8M0FNU> -> !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    xegpu.store_nd %sc2, %tdesc_red[0, 0] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<32x128xf8E8M0FNU>, !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    %tdesc_plain = xegpu.create_nd_tdesc %dst_plain : memref<32x4xf8E8M0FNU> -> !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    xegpu.store_nd %trunc, %tdesc_plain[0, 0] <{layout = #xegpu.layout<inst_data = [8, 2], lane_layout = [8, 1], lane_data = [1, 1]>}> : vector<32x4xf8E8M0FNU>, !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    gpu.return
+  }
+}
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 1a741ba21e128..49006e1ef9507 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -219,8 +219,8 @@ func.func @scatter_ops_custom_perm_layout(%src: memref<256xf16>) {
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_preserve_load_perm_layout(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<256xf16>) {
-// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<true> : vector<16xi1>
-// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<12> : vector<16xindex>
+// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<true> : vector<16xi1>
+// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<12> : vector<16xindex>
 // CHECK: %[[LOAD_VEC:.*]] = xegpu.load %[[ARG0]][%[[OFFSETS]]], %[[MASK]]
 // CHECK-SAME: memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16xf16>
 // CHECK: %[[ADD_RES:.*]] = arith.addf %[[LOAD_VEC]], %[[LOAD_VEC]] {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} : vector<16xf16>
@@ -341,10 +341,10 @@ func.func @binary_op_one_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.
 gpu.module @test {
 // CHECK-LABEL: func.func @binary_op_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
-// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf16>
+// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>} : vector<16x16xf16>
 // CHECK: %[[T3:.*]] = xegpu.dpas %{{.*}}, %[[T2]] {layout_a = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, layout_b = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>, layout_cd = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
 // CHECK-NEXT: xegpu.store_nd %[[T3]], %[[ARG2]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x16xf32>, !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
 // CHECK-NEXT: xegpu.store_nd %[[T2]], %[[ARG3]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>, !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
@@ -429,18 +429,18 @@ func.func @if_single_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tens
 gpu.module @test {
 // CHECK-LABEL: func.func @if_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: i1, %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG4:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
 // CHECK: %[[T1:.*]] = scf.if %[[ARG2]] -> (vector<16x16xf16>) {
-// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T3]] : vector<16x16xf16>
 // CHECK-NEXT:     } else {
-// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T4]] : vector<16x16xf16>
-// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}
+// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
 func.func @if_multiple_uses(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor_desc<16x16xf16>, %arg2: i1, %arg3: !xegpu.tensor_desc<8x16xf32>, %arg4: !xegpu.tensor_desc<16x16xf16>) {
   %0 = xegpu.load_nd %arg0[0, 0]  : !xegpu.tensor_desc<8x16xf16> -> vector<8x16xf16>
   %1 = scf.if %arg2 -> (vector<16x16xf16>) {
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..9bfb783299055 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -8,7 +8,7 @@
 
 // XFAIL: *
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>

``````````

</details>


https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Wed Jul  8 19:07:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:07:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f0279.824e11a6.f5f27.5774@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 75e8db468..72b10e1d3 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ public:
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override{};
+  void visitBranchOperand(OpOperand &operand) override {};
 
-  void visitCallOperand(OpOperand &operand) override{};
+  void visitCallOperand(OpOperand &operand) override {};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override{};
+                               ArrayRef<BlockArgument> arguments) override {};
 
-  void
-  visitExternalCall(CallOpInterface call,
-                    ArrayRef<LayoutInfoLattice *> operands,
-                    ArrayRef<const LayoutInfoLattice *> results) override{};
+  void visitExternalCall(CallOpInterface call,
+                         ArrayRef<LayoutInfoLattice *> operands,
+                         ArrayRef<const LayoutInfoLattice *> results) override {
+  };
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());

``````````

</details>


https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Wed Jul  8 19:08:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:08:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a4f0291.9edb5c9d.208698.abac@mx.google.com>


https://github.com/LouisLu060211 edited https://github.com/llvm/llvm-project/pull/208185

From llvmlistbot at llvm.org  Wed Jul  8 19:11:52 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Wed, 08 Jul 2026 19:11:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f0368.27b3b003.3b871.236c@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/208365

>From c589918d9f4111e049d6165a56fb7fc2bfa6d344 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 8 Jul 2026 23:30:49 +0000
Subject: [PATCH 1/5] [MLIR][XeGPU] Prefer slice layout in LayoutInfo::meet

During backward layout propagation a value can receive different
candidate layouts from its consumers. Previously LayoutInfo::meet kept
whichever layout was assigned first, so a value feeding both a
slice-producing consumer (e.g. a broadcast whose inferred source layout
is a slice) and a plain-layout consumer (e.g. a dpas_mx scale operand)
could end up with the plain layout depending on visitation order.

meet() now prefers the slice layout when both sides are assigned
(falling back to lhs when both/neither are slices). For that preference
to take effect, operator== compares the stored layout (via isEqualTo)
rather than only the assigned bit; otherwise the dataflow framework
treats the refined result as a no-op and discards it. The refinement
only moves a plain layout toward a slice layout, so it stays monotonic.

Adds a propagate-layout-inst-data test and updates the a-operand
lane_data in the simple_mxfp_gemm integration test to match the HW
constraint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 25 +++++++++++-----
 .../XeGPU/propagate-layout-inst-data.mlir     | 29 +++++++++++++++++++
 .../Dialect/XeGPU/WG/simple_mxfp_gemm.mlir    |  2 +-
 3 files changed, 47 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 64d0d8063b7ff..ff9239b637a10 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -71,12 +71,13 @@ namespace {
 /// Given this, LayoutInfo  satisifies the following properties:
 ///  1) A LayoutInfo value can be in one of two states - `assigned` or `not
 ///  assigned`.
-///  2) Two LayoutInfo values are equal if they are both assigned or
-///  both not assigned. The concrete value of assigned state does not matter.
+///  2) Two LayoutInfo values are equal if they are both not assigned, or both
+///  assigned with the same layout.
 ///  3) The meet operator works as follows:
-///     - If current state is assigned, return the current state. (already
-///     a unique layout is assigned. don't change it)
-///     - Otherwise, return the other state.
+///     - If only one side is assigned, return that side.
+///     - If both sides are assigned, prefer the side carrying a slice layout.
+///       If both (or neither) are slice layouts, prefer the lhs (current
+///       state) so an already assigned unique layout is not changed.
 
 struct LayoutInfo {
 private:
@@ -86,10 +87,14 @@ struct LayoutInfo {
   LayoutInfo() = default;
   LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
 
-  // Two lattice values are equal if they have `some` layout. The actual
-  // content of the layout does not matter.
+  // Two lattice values are equal if they are both unassigned, or both assigned
+  // with the same layout.
   bool operator==(const LayoutInfo &other) const {
-    return this->isAssigned() == other.isAssigned();
+    if (isAssigned() != other.isAssigned())
+      return false;
+    if (!isAssigned())
+      return true;
+    return storage.isEqualTo(other.storage);
   }
 
   static LayoutInfo meet(const LayoutInfo &lhs, const LayoutInfo &rhs);
@@ -141,6 +146,10 @@ void LayoutInfo::print(raw_ostream &os) const {
 LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
   if (!lhs.isAssigned())
     return rhs;
+  if (!rhs.isAssigned())
+    return lhs;
+  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+    return rhs;
   return lhs;
 }
 
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 515c59db72819..6bf7fff810006 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -653,3 +653,32 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
   return
 }
 }
+
+// -----
+// A value with two consumers: one back-propagates a plain layout (the store_nd
+// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
+// chain feeding the reduction result). `meet` must prefer the slice layout, and
+// the inst_data / lane_layout / lane_data fields must be preserved on it.
+gpu.module @test {
+  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK: %[[TRUNC:.*]] = arith.truncf
+  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
+  // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+      %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
+    %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
+    %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>
+    %load = xegpu.load_nd %tdesc[0, 0] : !xegpu.tensor_desc<32x128xbf16> -> vector<32x128xbf16>
+    %sc1 = vector.shape_cast %load : vector<32x128xbf16> to vector<32x4x32xbf16>
+    %red = vector.multi_reduction <maximumf>, %sc1, %cst [2] : vector<32x4x32xbf16> to vector<32x4xbf16>
+    %trunc = arith.truncf %red : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+    %bcast = vector.broadcast %trunc : vector<32x4xf8E8M0FNU> to vector<32x32x4xf8E8M0FNU>
+    %bcast2 = vector.transpose %bcast, [1, 2, 0] : vector<32x32x4xf8E8M0FNU> to vector<32x4x32xf8E8M0FNU>
+    %sc2 = vector.shape_cast %bcast2 : vector<32x4x32xf8E8M0FNU> to vector<32x128xf8E8M0FNU>
+    %tdesc_red = xegpu.create_nd_tdesc %dst_red : memref<32x128xf8E8M0FNU> -> !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    xegpu.store_nd %sc2, %tdesc_red[0, 0] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<32x128xf8E8M0FNU>, !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    %tdesc_plain = xegpu.create_nd_tdesc %dst_plain : memref<32x4xf8E8M0FNU> -> !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    xegpu.store_nd %trunc, %tdesc_plain[0, 0] <{layout = #xegpu.layout<inst_data = [8, 2], lane_layout = [8, 1], lane_data = [1, 1]>}> : vector<32x4xf8E8M0FNU>, !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    gpu.return
+  }
+}
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..9bfb783299055 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -8,7 +8,7 @@
 
 // XFAIL: *
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>

>From c31ad5f6b4f3d97c50c91e62737b39cd07a23ecf Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:55:14 +0000
Subject: [PATCH 2/5] [MLIR][XeGPU] Prefer nearer consumer's layout in
 LayoutInfo::meet

When a value is demanded by multiple consumers during backward layout
propagation, prefer the layout of the consumer that is nearer to the
producer in program order. This tends to preserve a consumer's layout as
far up the def chain as possible, reducing layout conversions. It is a
hint, not an optimum.

Each op is assigned a program-order index via a pre-order walk (matching
printed-IR order), so a use inside an scf.for body is nearer than a use
after the loop. LayoutInfo carries the demanding op's index in a new
`programOrder` field, stamped by the single-argument constructor from a
file-scoped `currentProgramOrder` that visitOperation sets, so the ~30
meet() call sites are unchanged. `programOrder` is not propagated (each
visited op stamps its own index) and is excluded from operator==.

This replaces the earlier slice-layout preference in meet(), which is
now redundant: distinct users always have distinct indices, so program
order decides every real conflict. Also removes the dead
LayoutInfo::transpose declaration.

Updates the multiple-uses propagate-layout tests to reflect that the
nearer consumer's layout now wins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 75 +++++++++++++++++--
 mlir/test/Dialect/XeGPU/propagate-layout.mlir | 20 ++---
 2 files changed, 77 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index ff9239b637a10..72b10e1d3265f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -30,6 +30,7 @@
 #include "mlir/Interfaces/LoopLikeInterface.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
@@ -38,6 +39,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/LogicalResult.h"
 #include "llvm/Support/raw_ostream.h"
+#include <limits>
 
 namespace mlir {
 namespace xegpu {
@@ -58,6 +60,14 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
+/// Program-order index of the op currently being visited by the backward
+/// analysis. `visitOperation` sets this before dispatching, and the
+/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
+/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
+/// change. A larger index means farther from the producer; the sentinel max
+/// marks demands with no associated op (e.g. exit state).
+static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
+
 /// Helper class for tracking the analysis state of an mlir value. For layout
 /// propagation, the analysis state is simply the distribution layout of
 /// each value. The distribution layout information is encapsulated using
@@ -75,20 +85,34 @@ namespace {
 ///  assigned with the same layout.
 ///  3) The meet operator works as follows:
 ///     - If only one side is assigned, return that side.
-///     - If both sides are assigned, prefer the side carrying a slice layout.
-///       If both (or neither) are slice layouts, prefer the lhs (current
-///       state) so an already assigned unique layout is not changed.
+///     - If both sides are assigned, prefer the layout demanded by the user
+///       that is nearer to the producer in program order (smaller
+///       `programOrder`); on a tie keep lhs.
+///
+/// The `programOrder` field records the program-order index of the consumer op
+/// that demanded the layout (see `currentProgramOrder`). During this backward
+/// analysis a value can be demanded by several users; keeping the nearest one
+/// tends to preserve a consumer's layout as far up the def chain as possible,
+/// minimizing layout conversions. This is a hint, not an optimum.
+/// `programOrder` is never propagated up the chain - each visited op stamps its
+/// own index - so it is deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
   xegpu::DistributeLayoutAttr storage = nullptr;
+  // Program-order index of the consumer op that demanded this layout. Smaller
+  // means nearer to the producer. Unassigned/unknown demands sort last.
+  int64_t programOrder = std::numeric_limits<int64_t>::max();
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
+      : storage(layout), programOrder(programOrder) {}
 
   // Two lattice values are equal if they are both unassigned, or both assigned
-  // with the same layout.
+  // with the same layout. `programOrder` is intentionally excluded: it is not
+  // propagated, so a pure order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
     if (isAssigned() != other.isAssigned())
       return false;
@@ -105,8 +129,6 @@ struct LayoutInfo {
 
   bool isAssigned() const { return storage != nullptr; }
 
-  LayoutInfo transpose(ArrayRef<int64_t> permutation) const;
-
   SmallVector<int> getLaneLayout() const;
 
   SmallVector<int> getLaneData() const;
@@ -135,6 +157,11 @@ struct LayoutInfo {
   void set(const xegpu::DistributeLayoutAttr &layout) { storage = layout; }
 };
 
+// Stamp every demand pushed by the current op with that op's program-order
+// index so `meet` can prefer the nearest consumer.
+LayoutInfo::LayoutInfo(const xegpu::DistributeLayoutAttr &layout)
+    : storage(layout), programOrder(currentProgramOrder) {}
+
 void LayoutInfo::print(raw_ostream &os) const {
   if (isAssigned()) {
     os << storage;
@@ -148,7 +175,10 @@ LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
     return rhs;
   if (!rhs.isAssigned())
     return lhs;
-  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+  // Prefer the demand from the user nearer to the producer in program order.
+  // Distinct users always have distinct indices, so this decides every
+  // real conflict; on a tie (same op, or both unknown) keep lhs.
+  if (rhs.programOrder < lhs.programOrder)
     return rhs;
   return lhs;
 }
@@ -185,6 +215,15 @@ class LayoutInfoPropagation
 private:
   xegpu::LayoutKind layoutKind;
   unsigned indexBitWidth;
+
+  // Program-order index of every op, built lazily on first use via a pre-order
+  // walk of the top-level module/function (matching printed-IR order). Used to
+  // tell which consumer of a value is nearer to its producer.
+  DenseMap<Operation *, int64_t> programOrder;
+  // Returns the program-order index of `op`, populating `programOrder` from
+  // `op`'s top-level ancestor on first call.
+  int64_t getProgramOrder(Operation *op);
+
   void visitDpasOp(xegpu::DpasOp dpas, ArrayRef<LayoutInfoLattice *> operands,
                    ArrayRef<const LayoutInfoLattice *> results);
 
@@ -300,9 +339,29 @@ class LayoutInfoPropagation
 };
 } // namespace
 
+int64_t LayoutInfoPropagation::getProgramOrder(Operation *op) {
+  auto it = programOrder.find(op);
+  if (it != programOrder.end())
+    return it->second;
+  // First time we see this op's tree: number every op under its top-level
+  // ancestor in pre-order (i.e. printed-IR order). Nested ops (e.g. inside an
+  // scf.for body) get an index between their parent and the parent's next
+  // sibling, so a use inside a loop is "nearer" than a use after it.
+  Operation *root = op;
+  while (root->getParentOp())
+    root = root->getParentOp();
+  int64_t counter = 0;
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *o) { programOrder[o] = counter++; });
+  return programOrder.lookup(op);
+}
+
 LogicalResult LayoutInfoPropagation::visitOperation(
     Operation *op, ArrayRef<LayoutInfoLattice *> operands,
     ArrayRef<const LayoutInfoLattice *> results) {
+  // Stamp demands pushed by this op with its program-order index so `meet` can
+  // prefer the nearest consumer.
+  currentProgramOrder = getProgramOrder(op);
   TypeSwitch<Operation *>(op)
       .Case(
           [&](xegpu::DpasOp dpasOp) { visitDpasOp(dpasOp, operands, results); })
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 1a741ba21e128..49006e1ef9507 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -219,8 +219,8 @@ func.func @scatter_ops_custom_perm_layout(%src: memref<256xf16>) {
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_preserve_load_perm_layout(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<256xf16>) {
-// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<true> : vector<16xi1>
-// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<12> : vector<16xindex>
+// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<true> : vector<16xi1>
+// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<12> : vector<16xindex>
 // CHECK: %[[LOAD_VEC:.*]] = xegpu.load %[[ARG0]][%[[OFFSETS]]], %[[MASK]]
 // CHECK-SAME: memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16xf16>
 // CHECK: %[[ADD_RES:.*]] = arith.addf %[[LOAD_VEC]], %[[LOAD_VEC]] {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} : vector<16xf16>
@@ -341,10 +341,10 @@ func.func @binary_op_one_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.
 gpu.module @test {
 // CHECK-LABEL: func.func @binary_op_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
-// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf16>
+// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>} : vector<16x16xf16>
 // CHECK: %[[T3:.*]] = xegpu.dpas %{{.*}}, %[[T2]] {layout_a = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, layout_b = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>, layout_cd = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
 // CHECK-NEXT: xegpu.store_nd %[[T3]], %[[ARG2]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x16xf32>, !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
 // CHECK-NEXT: xegpu.store_nd %[[T2]], %[[ARG3]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>, !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
@@ -429,18 +429,18 @@ func.func @if_single_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tens
 gpu.module @test {
 // CHECK-LABEL: func.func @if_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: i1, %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG4:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
 // CHECK: %[[T1:.*]] = scf.if %[[ARG2]] -> (vector<16x16xf16>) {
-// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T3]] : vector<16x16xf16>
 // CHECK-NEXT:     } else {
-// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T4]] : vector<16x16xf16>
-// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}
+// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
 func.func @if_multiple_uses(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor_desc<16x16xf16>, %arg2: i1, %arg3: !xegpu.tensor_desc<8x16xf32>, %arg4: !xegpu.tensor_desc<16x16xf16>) {
   %0 = xegpu.load_nd %arg0[0, 0]  : !xegpu.tensor_desc<8x16xf16> -> vector<8x16xf16>
   %1 = scf.if %arg2 -> (vector<16x16xf16>) {

>From 3c4ada8b1f50d0d9c48f3c8543aa0662f1f2164c Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:58:24 +0000
Subject: [PATCH 3/5] [MLIR][XeGPU] Reframe propagate-layout test as
 nearer-user preference

Rename truncf_prefers_slice to truncf_prefers_nearer_user and update its
comment: on this branch %trunc keeps the broadcast chain's slice layout
because that consumer is nearer in program order, not because meet()
prefers slice layouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../Dialect/XeGPU/propagate-layout-inst-data.mlir    | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 6bf7fff810006..47471b890d552 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -655,16 +655,16 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
 }
 
 // -----
-// A value with two consumers: one back-propagates a plain layout (the store_nd
-// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
-// chain feeding the reduction result). `meet` must prefer the slice layout, and
-// the inst_data / lane_layout / lane_data fields must be preserved on it.
+// %trunc has two consumers: the broadcast/transpose chain (nearer in program
+// order, back-propagates a slice layout) and the store_nd of %trunc (farther,
+// back-propagates a plain layout). `meet` must keep the nearer consumer's
+// layout, preserving its inst_data / lane_layout / lane_data fields.
 gpu.module @test {
-  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK-LABEL: truncf_prefers_nearer_user
   // CHECK: %[[TRUNC:.*]] = arith.truncf
   // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
   // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
-  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+  gpu.func @truncf_prefers_nearer_user(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
       %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
     %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
     %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>

>From 9c4de99b6094f502c82dcd411b8eca2c1c18141d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 02:00:09 +0000
Subject: [PATCH 4/5] git format

---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp      | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 72b10e1d3265f..75e8db468f361 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override {};
+  void visitBranchOperand(OpOperand &operand) override{};
 
-  void visitCallOperand(OpOperand &operand) override {};
+  void visitCallOperand(OpOperand &operand) override{};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override {};
+                               ArrayRef<BlockArgument> arguments) override{};
 
-  void visitExternalCall(CallOpInterface call,
-                         ArrayRef<LayoutInfoLattice *> operands,
-                         ArrayRef<const LayoutInfoLattice *> results) override {
-  };
+  void
+  visitExternalCall(CallOpInterface call,
+                    ArrayRef<LayoutInfoLattice *> operands,
+                    ArrayRef<const LayoutInfoLattice *> results) override{};
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());

>From 1db9aa8f8f3ccd82194b591fc3f4754a056c416d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 02:11:28 +0000
Subject: [PATCH 5/5] [MLIR][XeGPU] clang-format XeGPUPropagateLayout.cpp

Apply clang-format fixes flagged by CI (override{} -> override {}) on the
empty visitor overrides touched by this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp      | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 75e8db468f361..72b10e1d3265f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override{};
+  void visitBranchOperand(OpOperand &operand) override {};
 
-  void visitCallOperand(OpOperand &operand) override{};
+  void visitCallOperand(OpOperand &operand) override {};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override{};
+                               ArrayRef<BlockArgument> arguments) override {};
 
-  void
-  visitExternalCall(CallOpInterface call,
-                    ArrayRef<LayoutInfoLattice *> operands,
-                    ArrayRef<const LayoutInfoLattice *> results) override{};
+  void visitExternalCall(CallOpInterface call,
+                         ArrayRef<LayoutInfoLattice *> operands,
+                         ArrayRef<const LayoutInfoLattice *> results) override {
+  };
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());


From llvmlistbot at llvm.org  Wed Jul  8 19:14:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:14:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f03f9.0c05e68e.1e82f1.47fb@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/207336

>From 7602c85f86225c3abd602a5df6d71af0c92ec92a Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 3 Jul 2026 00:20:18 -0700
Subject: [PATCH] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake

---
 mlir/cmake/modules/CMakeLists.txt      | 3 +++
 mlir/cmake/modules/MLIRConfig.cmake.in | 1 +
 2 files changed, 4 insertions(+)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..b82049e150e93 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -59,6 +59,9 @@ set(MLIR_CONFIG_INCLUDE_DIRS
 set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+set(MLIR_CONFIG_LINK_MLIR_DYLIB "${MLIR_LINK_MLIR_DYLIB}")
+llvm_canonicalize_cmake_booleans(
+  MLIR_CONFIG_LINK_MLIR_DYLIB)
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..fd04a64ce625b 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -16,6 +16,7 @@ set(MLIR_IRDL_TO_CPP_EXE "@MLIR_CONFIG_IRDL_TO_CPP_EXE@")
 set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
+set(MLIR_LINK_MLIR_DYLIB @MLIR_CONFIG_LINK_MLIR_DYLIB@)
 
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")


From llvmlistbot at llvm.org  Wed Jul  8 19:18:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:18:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f04e4.e9d484b9.15e549.a4c6@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 19:21:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:21:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f059e.d1496011.26913c.cb42@mx.google.com>


================
@@ -60,6 +60,11 @@ set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
 
+if (MLIR_LINK_MLIR_DYLIB)
+  set(MLIR_CONFIG_LINK_MLIR_DYLIB
+      "set(MLIR_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})")
----------------
qyingwu wrote:

Thanks, that makes sense. I updated the patch to canonicalize the value instead of following the LLVM pattern.
also tested a small out-of-tree consumer using `find_package(MLIR CONFIG)` and `include(AddMLIR)` against both cases.

https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 19:25:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:25:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f06b3.449bfb44.172ed9.4c72@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/207336

>From 28db4abcc3fb1977927e6b596568e4fe0633da96 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 3 Jul 2026 00:20:18 -0700
Subject: [PATCH] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake

---
 mlir/cmake/modules/CMakeLists.txt      | 4 ++++
 mlir/cmake/modules/MLIRConfig.cmake.in | 1 +
 2 files changed, 5 insertions(+)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..ef0ba104c2525 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -59,6 +59,10 @@ set(MLIR_CONFIG_INCLUDE_DIRS
 set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+set(MLIR_CONFIG_LINK_MLIR_DYLIB "${MLIR_LINK_MLIR_DYLIB}")
+llvm_canonicalize_cmake_booleans(
+  MLIR_CONFIG_LINK_MLIR_DYLIB
+)
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..fd04a64ce625b 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -16,6 +16,7 @@ set(MLIR_IRDL_TO_CPP_EXE "@MLIR_CONFIG_IRDL_TO_CPP_EXE@")
 set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
+set(MLIR_LINK_MLIR_DYLIB @MLIR_CONFIG_LINK_MLIR_DYLIB@)
 
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")


From llvmlistbot at llvm.org  Wed Jul  8 19:26:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:26:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f06cc.70920dee.290dca.17c3@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/207336

>From 28db4abcc3fb1977927e6b596568e4fe0633da96 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 3 Jul 2026 00:20:18 -0700
Subject: [PATCH] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake

---
 mlir/cmake/modules/CMakeLists.txt      | 4 ++++
 mlir/cmake/modules/MLIRConfig.cmake.in | 1 +
 2 files changed, 5 insertions(+)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..ef0ba104c2525 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -59,6 +59,10 @@ set(MLIR_CONFIG_INCLUDE_DIRS
 set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+set(MLIR_CONFIG_LINK_MLIR_DYLIB "${MLIR_LINK_MLIR_DYLIB}")
+llvm_canonicalize_cmake_booleans(
+  MLIR_CONFIG_LINK_MLIR_DYLIB
+)
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in
diff --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..fd04a64ce625b 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -16,6 +16,7 @@ set(MLIR_IRDL_TO_CPP_EXE "@MLIR_CONFIG_IRDL_TO_CPP_EXE@")
 set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
+set(MLIR_LINK_MLIR_DYLIB @MLIR_CONFIG_LINK_MLIR_DYLIB@)
 
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")


From llvmlistbot at llvm.org  Wed Jul  8 19:34:45 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Wed, 08 Jul 2026 19:34:45 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Give each mlir::Block a stable ID
 within its parent region (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4f08c5.e55dc341.1f19bd.ab18@mx.google.com>


================
@@ -422,6 +435,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique number of this block within its parent region, assigned when the
+  /// block is added to a region. See getNumber().
+  unsigned blockNumber = 0;
----------------
MaskRay wrote:

Done

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Wed Jul  8 19:34:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:34:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 697bd97 - Give each mlir::Block a stable ID
 within its parent region (#207617)
Message-ID: <6a4f08cd.f8b79514.1d52d0.9b0b@mx.google.com>


Author: Fangrui Song
Date: 2026-07-08T19:34:48-07:00
New Revision: 697bd9704894691d6b0f40b3150d70047a6f181f

URL: https://github.com/llvm/llvm-project/commit/697bd9704894691d6b0f40b3150d70047a6f181f
DIFF: https://github.com/llvm/llvm-project/commit/697bd9704894691d6b0f40b3150d70047a6f181f.diff

LOG: Give each mlir::Block a stable ID within its parent region (#207617)

Assign each mlir::Block a stable ID within its parent region, mirroring
llvm::BasicBlock/llvm::Function (called ID, not number, since a block/op
number denotes position in MLIR): Block gains getBlockID() and reads -1u
while it has no parent region; Region gains nextBlockID with
getMaxBlockID() and getBlockIDEpoch(); the block ilist traits assign the
ID on add/transfer and invalidate it on removal; and
GraphTraits<mlir::Block*>/<mlir::Region*> expose
getNumber/getMaxNumber/getNumberEpoch. This makes
GraphHasNodeNumbers<mlir::Block*> true, moving MLIR's CFGLoopInfo and
dominator tree onto the number-indexed path. MLIR never renumbers
blocks,
so the epoch is a fixed 0.

Prerequisite for requiring GraphHasNodeNumbers in llvm::LoopInfoBase and
dropping its DenseMap fallback.

Aided by Claude Opus 4.8

Added: 
    

Modified: 
    mlir/include/mlir/IR/Block.h
    mlir/include/mlir/IR/Region.h
    mlir/include/mlir/IR/RegionGraphTraits.h
    mlir/lib/IR/Region.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/IR/Block.h b/mlir/include/mlir/IR/Block.h
index bb3484af2aaf7..c84921db220af 100644
--- a/mlir/include/mlir/IR/Block.h
+++ b/mlir/include/mlir/IR/Block.h
@@ -52,6 +52,20 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// model.
   Region *getParent() const;
 
+  /// Return an ID uniquely identifying this block within its parent region.
+  /// The ID is assigned when the block joins a region and reassigned when the
+  /// block is moved to a 
diff erent region; it is stable while the block stays
+  /// in a region, and removing a block leaves a hole (IDs are not reused). Only
+  /// valid for a block that is in a region.
+  ///
+  /// Unlike computeBlockNumber(), this is O(1) and stable; it exists so that
+  /// generic graph algorithms (e.g. LoopInfo, DominatorTree) can index blocks
+  /// by ID.
+  unsigned getBlockID() const {
+    assert(getParent() && "only blocks in a region have a valid ID");
+    return blockID;
+  }
+
   /// Returns the closest surrounding operation that contains this block.
   Operation *getParentOp();
 
@@ -422,6 +436,10 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   /// the operations within this block have a valid ordering.
   llvm::PointerIntPair<Region *, /*IntBits=*/1, bool> parentValidOpOrderPair;
 
+  /// Unique ID of this block within its parent region, (re)assigned when the
+  /// block joins a region; -1u while the block has no parent. See getBlockID().
+  unsigned blockID = -1u;
+
   /// This is the list of operations in the block.
   OpListType operations;
 
@@ -432,6 +450,7 @@ class alignas(8) Block : public IRObjectWithUseList<BlockOperand>,
   void operator=(Block &) = delete;
 
   friend struct llvm::ilist_traits<Block>;
+  friend class Region;
 };
 
 raw_ostream &operator<<(raw_ostream &, Block &);

diff  --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index 13b54991832cb..fa9d8f08d84bb 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -72,6 +72,19 @@ class Region {
     return &Region::blocks;
   }
 
+  //===--------------------------------------------------------------------===//
+  // Block numbering
+  //===--------------------------------------------------------------------===//
+
+  /// One past the largest block ID handed out in this region; block IDs lie in
+  /// [0, getMaxBlockID()). See Block::getBlockID().
+  unsigned getMaxBlockID() const { return nextBlockID; }
+
+  /// The block-ID epoch, part of the generic number-indexed graph contract
+  /// (LoopInfo, DominatorTree) for detecting stale IDs. MLIR never renumbers a
+  /// region's blocks, so this is a fixed 0. See Block::getBlockID().
+  unsigned getBlockIDEpoch() const { return 0; }
+
   //===--------------------------------------------------------------------===//
   // Argument Handling
   //===--------------------------------------------------------------------===//
@@ -343,6 +356,11 @@ class Region {
 
   /// This is the object we are part of.
   Operation *container = nullptr;
+
+  /// Next block ID to hand out. See Block::getBlockID().
+  unsigned nextBlockID = 0;
+
+  friend struct llvm::ilist_traits<Block>;
 };
 
 /// This class provides an abstraction over the 
diff erent types of ranges over

diff  --git a/mlir/include/mlir/IR/RegionGraphTraits.h b/mlir/include/mlir/IR/RegionGraphTraits.h
index d7d80d9b35806..bf21d9720169c 100644
--- a/mlir/include/mlir/IR/RegionGraphTraits.h
+++ b/mlir/include/mlir/IR/RegionGraphTraits.h
@@ -31,6 +31,8 @@ struct GraphTraits<mlir::Block *> {
     return node->succ_begin();
   }
   static ChildIteratorType child_end(NodeRef node) { return node->succ_end(); }
+
+  static unsigned getNumber(NodeRef node) { return node->getBlockID(); }
 };
 
 template <>
@@ -63,6 +65,8 @@ struct GraphTraits<const mlir::Block *> {
   static ChildIteratorType child_end(NodeRef node) {
     return const_cast<mlir::Block *>(node)->succ_end();
   }
+
+  static unsigned getNumber(NodeRef node) { return node->getBlockID(); }
 };
 
 template <>
@@ -97,6 +101,13 @@ struct GraphTraits<mlir::Region *> : public GraphTraits<mlir::Block *> {
   static nodes_iterator nodes_end(GraphType fn) {
     return nodes_iterator(fn->end());
   }
+
+  static unsigned getMaxNumber(GraphType region) {
+    return region->getMaxBlockID();
+  }
+  static unsigned getNumberEpoch(GraphType region) {
+    return region->getBlockIDEpoch();
+  }
 };
 
 template <>

diff  --git a/mlir/lib/IR/Region.cpp b/mlir/lib/IR/Region.cpp
index 15a941f380225..775485a3755e0 100644
--- a/mlir/lib/IR/Region.cpp
+++ b/mlir/lib/IR/Region.cpp
@@ -194,7 +194,9 @@ Region *llvm::ilist_traits<::mlir::Block>::getParentRegion() {
 /// We keep the region pointer up to date.
 void llvm::ilist_traits<::mlir::Block>::addNodeToList(Block *block) {
   assert(!block->getParent() && "already in a region!");
-  block->parentValidOpOrderPair.setPointer(getParentRegion());
+  Region *region = getParentRegion();
+  block->parentValidOpOrderPair.setPointer(region);
+  block->blockID = region->nextBlockID++;
 }
 
 /// This is a trait method invoked when an operation is removed from a
@@ -202,6 +204,8 @@ void llvm::ilist_traits<::mlir::Block>::addNodeToList(Block *block) {
 void llvm::ilist_traits<::mlir::Block>::removeNodeFromList(Block *block) {
   assert(block->getParent() && "not already in a region!");
   block->parentValidOpOrderPair.setPointer(nullptr);
+  // The ID is invalid until the block is added to a region again.
+  block->blockID = -1u;
 }
 
 /// This is a trait method invoked when an operation is moved from one block
@@ -214,9 +218,12 @@ void llvm::ilist_traits<::mlir::Block>::transferNodesFromList(
   if (curParent == otherList.getParentRegion())
     return;
 
-  // Update the 'parent' member of each Block.
-  for (; first != last; ++first)
+  // Update the 'parent' member of each Block and give it an ID in its new
+  // region.
+  for (; first != last; ++first) {
     first->parentValidOpOrderPair.setPointer(curParent);
+    first->blockID = curParent->nextBlockID++;
+  }
 }
 
 //===----------------------------------------------------------------------===//


        

From llvmlistbot at llvm.org  Wed Jul  8 19:34:56 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Wed, 08 Jul 2026 19:34:56 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Give each mlir::Block a stable ID
 within its parent region (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4f08d0.d1496011.26913c.0074@mx.google.com>


https://github.com/MaskRay closed https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Wed Jul  8 19:55:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 19:55:46 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a4f0db2.faf16b37.2f0e1d.c6ad@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/208185

>From 7f35e2e4d91e7bfa678f7806332f3087b8c7309e Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/2] [mlir][memref] Avoid mem2reg crash on element count
 overflow

Avoid crash in test alias analysis operand printing

[mlir][vector] Fix crash in narrow-type emulation when mask has no defining op
---
 .../Dialect/MemRef/IR/MemRefMemorySlot.cpp    | 13 +++--
 .../Transforms/VectorEmulateNarrowType.cpp    |  5 +-
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/Dialect/MemRef/mem2reg.mlir         | 15 ++++++
 .../emulate-narrow-type-unsupported.mlir      | 18 +++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 6 files changed, 100 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..9aa28ff899bc4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -98,10 +98,13 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
     if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
       maskOp = extractOp.getSource().getDefiningOp();
       extractOps.push_back(extractOp);
+    }else {
+      return failure();
     }
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
+  if (!maskOp ||
+      !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
           maskOp))
     return failure();
 
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}
diff --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..17bd24e2c1959 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -109,3 +109,21 @@ func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2:
 //  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+///----------------------------------------------------------------------------------------
+/// vector.maskedload
+///----------------------------------------------------------------------------------------
+
+func.func @vector_maskedload_blockarg_mask_negative(%arg0: memref<16xi8>, %arg1: vector<8xi1>, %arg2: vector<8xi8>) -> vector<8xi8> {
+    %c0 = arith.constant 0 : index
+    %0 = vector.maskedload %arg0[%c0], %arg1, %arg2 : memref<16xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+    return %0 : vector<8xi8>
+}
+// The mask is a block argument with no defining op, so it cannot be
+// compressed - expect no conversion (and, crucially, no crash).
+//  CHECK-LABEL: func @vector_maskedload_blockarg_mask_negative
+//   CHECK-SAME:   %{{.*}}: memref<16xi8>, %[[MASK:.+]]: vector<8xi1>
+//        CHECK:   vector.maskedload %{{.*}}, %[[MASK]]
+//    CHECK-NOT: i32
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 7bbb6400d202e55eed986e87fd50f9ac6c8a19c1 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Thu, 9 Jul 2026 10:55:05 +0800
Subject: [PATCH 2/2] Fix narrow type mask crash

---
 crash-min.log | 41 +++++++++++++++++++++++++++++++++++++++++
 crash.log     |  1 +
 2 files changed, 42 insertions(+)
 create mode 100644 crash-min.log
 create mode 100644 crash.log

diff --git a/crash-min.log b/crash-min.log
new file mode 100644
index 0000000000000..b4ab9da77a0ea
--- /dev/null
+++ b/crash-min.log
@@ -0,0 +1,41 @@
+PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace and instructions to reproduce the bug.
+Stack dump:
+0.	Program arguments: ./build/bin/mlir-opt -test-emulate-narrow-int a-min.mlir
+ #0 0x000055fbd082932a llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (./build/bin/mlir-opt+0x20be32a)
+ #1 0x000055fbd082617f llvm::sys::RunSignalHandlers() (./build/bin/mlir-opt+0x20bb17f)
+ #2 0x000055fbd082a4b5 SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
+ #3 0x00007e4cc2442520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
+ #4 0x000055fbd7f6496e bool llvm::isa<mlir::arith::ConstantOp, mlir::vector::CreateMaskOp, mlir::vector::ConstantMaskOp, mlir::Operation*>(mlir::Operation* const&) (./build/bin/mlir-opt+0x97f996e)
+ #5 0x000055fbd7f634ee getCompressedMaskOp(mlir::OpBuilder&, mlir::Location, mlir::Value, int, int, int) VectorEmulateNarrowType.cpp:0:0
+ #6 0x000055fbd7f62cff (anonymous namespace)::ConvertVectorMaskedLoad::matchAndRewrite(mlir::vector::MaskedLoadOp, mlir::vector::MaskedLoadOpAdaptor, mlir::ConversionPatternRewriter&) const VectorEmulateNarrowType.cpp:0:0
+ #7 0x000055fbd7f64db2 llvm::LogicalResult mlir::ConversionPattern::dispatchTo1To1<mlir::OpConversionPattern<mlir::vector::MaskedLoadOp>, mlir::vector::MaskedLoadOp>(mlir::OpConversionPattern<mlir::vector::MaskedLoadOp> const&, mlir::vector::MaskedLoadOp, mlir::vector::MaskedLoadOp::GenericAdaptor<llvm::ArrayRef<mlir::ValueRange>>, mlir::ConversionPatternRewriter&) (./build/bin/mlir-opt+0x97f9db2)
+ #8 0x000055fbd7f6267a mlir::OpConversionPattern<mlir::vector::MaskedLoadOp>::matchAndRewrite(mlir::Operation*, llvm::ArrayRef<mlir::ValueRange>, mlir::ConversionPatternRewriter&) const (./build/bin/mlir-opt+0x97f767a)
+ #9 0x000055fbd9e54df6 mlir::ConversionPattern::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&) const (./build/bin/mlir-opt+0xb6e9df6)
+#10 0x000055fbd9ea41fe void llvm::function_ref<void ()>::callback_fn<mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>)::$_6>(long) PatternApplicator.cpp:0:0
+#11 0x000055fbd9ea08b3 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>) (./build/bin/mlir-opt+0xb7358b3)
+#12 0x000055fbd9e56c1d (anonymous namespace)::OperationLegalizer::legalize(mlir::Operation*) DialectConversion.cpp:0:0
+#13 0x000055fbd9e56405 mlir::OperationConverter::convert(mlir::Operation*, bool) (./build/bin/mlir-opt+0xb6eb405)
+#14 0x000055fbd9e576b3 mlir::OperationConverter::applyConversion(llvm::ArrayRef<mlir::Operation*>) (./build/bin/mlir-opt+0xb6ec6b3)
+#15 0x000055fbd9e66719 void llvm::function_ref<void ()>::callback_fn<applyConversion(llvm::ArrayRef<mlir::Operation*>, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig, (anonymous namespace)::OpConversionMode)::$_43>(long) DialectConversion.cpp:0:0
+#16 0x000055fbd9e5d045 applyConversion(llvm::ArrayRef<mlir::Operation*>, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig, (anonymous namespace)::OpConversionMode) DialectConversion.cpp:0:0
+#17 0x000055fbd9e5d12b mlir::applyPartialConversion(mlir::Operation*, mlir::ConversionTarget const&, mlir::FrozenRewritePatternSet const&, mlir::ConversionConfig) (./build/bin/mlir-opt+0xb6f212b)
+#18 0x000055fbd091ad6a (anonymous namespace)::TestEmulateNarrowTypePass::runOnOperation() TestEmulateNarrowType.cpp:0:0
+#19 0x000055fbd9eff4fa mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) (./build/bin/mlir-opt+0xb7944fa)
+#20 0x000055fbd9f001b9 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) (./build/bin/mlir-opt+0xb7951b9)
+#21 0x000055fbd9f0b39a auto void mlir::parallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_102>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_102&&)::'lambda'(__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>&&)::operator()<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&>(__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo>>>&&) const Pass.cpp:0:0
+#22 0x000055fbd9f02de8 mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) (./build/bin/mlir-opt+0xb797de8)
+#23 0x000055fbd9eff691 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) (./build/bin/mlir-opt+0xb794691)
+#24 0x000055fbd9f001b9 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) (./build/bin/mlir-opt+0xb7951b9)
+#25 0x000055fbd9f070c1 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) (./build/bin/mlir-opt+0xb79c0c1)
+#26 0x000055fbd9f06944 mlir::PassManager::run(mlir::Operation*) (./build/bin/mlir-opt+0xb79b944)
+#27 0x000055fbd08d949a performActions(llvm::raw_ostream&, std::shared_ptr<llvm::SourceMgr> const&, mlir::MLIRContext*, mlir::MlirOptMainConfig const&) MlirOptMain.cpp:0:0
+#28 0x000055fbd08d877e llvm::LogicalResult llvm::function_ref<llvm::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&)::$_3>(long, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&) MlirOptMain.cpp:0:0
+#29 0x000055fbda2579d7 mlir::splitAndProcessBuffer(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<llvm::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::MemoryBufferRef const&, llvm::raw_ostream&)>, llvm::raw_ostream&, llvm::StringRef, llvm::StringRef) (./build/bin/mlir-opt+0xbaec9d7)
+#30 0x000055fbd08cf7c2 mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&) (./build/bin/mlir-opt+0x21647c2)
+#31 0x000055fbd08cfbc1 mlir::MlirOptMain(int, char**, llvm::StringRef, llvm::StringRef, mlir::DialectRegistry&) (./build/bin/mlir-opt+0x2164bc1)
+#32 0x000055fbd08cfdfd mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&) (./build/bin/mlir-opt+0x2164dfd)
+#33 0x000055fbd080f0e2 main (./build/bin/mlir-opt+0x20a40e2)
+#34 0x00007e4cc2429d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
+#35 0x00007e4cc2429e40 call_init ./csu/../csu/libc-start.c:128:20
+#36 0x00007e4cc2429e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
+#37 0x000055fbd080ec15 _start (./build/bin/mlir-opt+0x20a3c15)
diff --git a/crash.log b/crash.log
new file mode 100644
index 0000000000000..1bc8458b55d1d
--- /dev/null
+++ b/crash.log
@@ -0,0 +1 @@
+cannot open input file 'a.mlir': No such file or directory


From llvmlistbot at llvm.org  Wed Jul  8 20:10:37 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 08 Jul 2026 20:10:37 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Give each mlir::Block a stable ID
 within its parent region (PR #207617)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207617@github.com>
Message-ID: <6a4f112d.7d41f3a1.237e53.7158@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12219

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)


-- Configuring incomplete, errors occurred!
[8137/8698] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
ld-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8139/8698] Building CXX object tools/clang/lib/Analysis/plugins/SampleAnalyzer/CMakeFiles/SampleAnalyzerPlugin.dir/MainCallChecker.cpp.o
[8140/8698] Linking CXX shared library lib/libMLIRTestFromLLVMIRTranslation.so.23.0git
[8141/8698] Building CXX object tools/clang/lib/Analysis/plugins/CheckerDependencyHandling/CMakeFiles/CheckerDependencyHandlingAnalyzerPlugin.dir/CheckerDependencyHandling.cpp.o
[8142/8698] Building CXX object tools/clang/lib/Analysis/plugins/CheckerOptionHandling/CMakeFiles/CheckerOptionHandlingAnalyzerPlugin.dir/CheckerOptionHandling.cpp.o
[8143/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIOpenMP.dir/OpenMP.cpp.o
[8144/8698] Linking CXX shared library lib/libMLIRCAPIRegisterEverything.so.23.0git
[8145/8698] Building CXX object tools/mlir/tools/mlir-reduce/CMakeFiles/mlir-reduce.dir/mlir-reduce.cpp.o
[8146/8698] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8147/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIXeGPU.dir/XeGPU.cpp.o
[8148/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt.dir/mlir-minimal-opt.cpp.o
[8149/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPITosa.dir/TosaPasses.cpp.o
[8150/8698] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8151/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIVector.dir/VectorPasses.cpp.o
[8152/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-cat.dir/mlir-cat.cpp.o
[8153/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIVCIX.dir/VCIX.cpp.o
[8154/8698] Building CXX object tools/mlir/tools/mlir-opt/CMakeFiles/mlir-opt.dir/mlir-opt.cpp.o
[8155/8698] Building CXX object tools/mlir/tools/mlir-parser-fuzzer/bytecode/CMakeFiles/mlir-bytecode-parser-fuzzer.dir/mlir-bytecode-parser-fuzzer.cpp.o
[8156/8698] Building CXX object tools/mlir/tools/mlir-parser-fuzzer/text/CMakeFiles/mlir-text-parser-fuzzer.dir/mlir-text-parser-fuzzer.cpp.o
[8157/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt-canonicalize.dir/mlir-minimal-opt-canonicalize.cpp.o
[8158/8698] Building CXX object tools/mlir/tools/mlir-query/CMakeFiles/mlir-query.dir/mlir-query.cpp.o
[8159/8698] Building CXX object tools/mlir/tools/mlir-rewrite/CMakeFiles/mlir-rewrite.dir/mlir-rewrite.cpp.o
[8160/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIX86.dir/X86.cpp.o
[8161/8698] Building CXX object tools/mlir/lib/CAPI/ExecutionEngine/CMakeFiles/obj.MLIRCAPIExecutionEngine.dir/ExecutionEngine.cpp.o
[8162/8698] Building CXX object tools/mlir/tools/tblgen-to-irdl/CMakeFiles/tblgen-to-irdl.dir/OpDefinitionsGen.cpp.o
[8163/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIROCDL.dir/ROCDL.cpp.o
[8164/8698] Building CXX object tools/mlir/examples/transform-opt/CMakeFiles/mlir-transform-opt.dir/mlir-transform-opt.cpp.o
[8165/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVGPU.dir/NVGPU.cpp.o
[8166/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVVM.dir/NVVM.cpp.o
[8167/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIXeVM.dir/XeVM.cpp.o
[8168/8698] Building CXX object tools/mlir/lib/CAPI/Target/CMakeFiles/obj.MLIRCAPITarget.dir/LLVMIR.cpp.o
[8169/8698] Building CXX object tools/mlir/tools/mlir-runner/CMakeFiles/mlir-runner.dir/mlir-runner.cpp.o
[8170/8698] Building CXX object tools/mlir/lib/CAPI/Conversion/CMakeFiles/obj.MLIRCAPIConversion.dir/Passes.cpp.o
[8171/8698] Building CXX object tools/flang/lib/Lower/CMakeFiles/FortranLower.dir/OpenMP/OpenMP.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)
-- Configuring incomplete, errors occurred!
[8137/8698] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestFromLLVMIRTranslation.dir/TestFromLLVMIRTranslation.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp;-DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload;-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-bui
ld-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8139/8698] Building CXX object tools/clang/lib/Analysis/plugins/SampleAnalyzer/CMakeFiles/SampleAnalyzerPlugin.dir/MainCallChecker.cpp.o
[8140/8698] Linking CXX shared library lib/libMLIRTestFromLLVMIRTranslation.so.23.0git
[8141/8698] Building CXX object tools/clang/lib/Analysis/plugins/CheckerDependencyHandling/CMakeFiles/CheckerDependencyHandlingAnalyzerPlugin.dir/CheckerDependencyHandling.cpp.o
[8142/8698] Building CXX object tools/clang/lib/Analysis/plugins/CheckerOptionHandling/CMakeFiles/CheckerOptionHandlingAnalyzerPlugin.dir/CheckerOptionHandling.cpp.o
[8143/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIOpenMP.dir/OpenMP.cpp.o
[8144/8698] Linking CXX shared library lib/libMLIRCAPIRegisterEverything.so.23.0git
[8145/8698] Building CXX object tools/mlir/tools/mlir-reduce/CMakeFiles/mlir-reduce.dir/mlir-reduce.cpp.o
[8146/8698] Building CXX object tools/mlir/test/lib/Dialect/Test/CMakeFiles/MLIRTestToLLVMIRTranslation.dir/TestToLLVMIRTranslation.cpp.o
[8147/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIXeGPU.dir/XeGPU.cpp.o
[8148/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt.dir/mlir-minimal-opt.cpp.o
[8149/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPITosa.dir/TosaPasses.cpp.o
[8150/8698] Linking CXX shared library lib/libMLIRMlirOptMain.so.23.0git
[8151/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIVector.dir/VectorPasses.cpp.o
[8152/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-cat.dir/mlir-cat.cpp.o
[8153/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIVCIX.dir/VCIX.cpp.o
[8154/8698] Building CXX object tools/mlir/tools/mlir-opt/CMakeFiles/mlir-opt.dir/mlir-opt.cpp.o
[8155/8698] Building CXX object tools/mlir/tools/mlir-parser-fuzzer/bytecode/CMakeFiles/mlir-bytecode-parser-fuzzer.dir/mlir-bytecode-parser-fuzzer.cpp.o
[8156/8698] Building CXX object tools/mlir/tools/mlir-parser-fuzzer/text/CMakeFiles/mlir-text-parser-fuzzer.dir/mlir-text-parser-fuzzer.cpp.o
[8157/8698] Building CXX object tools/mlir/examples/minimal-opt/CMakeFiles/mlir-minimal-opt-canonicalize.dir/mlir-minimal-opt-canonicalize.cpp.o
[8158/8698] Building CXX object tools/mlir/tools/mlir-query/CMakeFiles/mlir-query.dir/mlir-query.cpp.o
[8159/8698] Building CXX object tools/mlir/tools/mlir-rewrite/CMakeFiles/mlir-rewrite.dir/mlir-rewrite.cpp.o
[8160/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIX86.dir/X86.cpp.o
[8161/8698] Building CXX object tools/mlir/lib/CAPI/ExecutionEngine/CMakeFiles/obj.MLIRCAPIExecutionEngine.dir/ExecutionEngine.cpp.o
[8162/8698] Building CXX object tools/mlir/tools/tblgen-to-irdl/CMakeFiles/tblgen-to-irdl.dir/OpDefinitionsGen.cpp.o
[8163/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIROCDL.dir/ROCDL.cpp.o
[8164/8698] Building CXX object tools/mlir/examples/transform-opt/CMakeFiles/mlir-transform-opt.dir/mlir-transform-opt.cpp.o
[8165/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVGPU.dir/NVGPU.cpp.o
[8166/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPINVVM.dir/NVVM.cpp.o
[8167/8698] Building CXX object tools/mlir/lib/CAPI/Dialect/CMakeFiles/obj.MLIRCAPIXeVM.dir/XeVM.cpp.o
[8168/8698] Building CXX object tools/mlir/lib/CAPI/Target/CMakeFiles/obj.MLIRCAPITarget.dir/LLVMIR.cpp.o
[8169/8698] Building CXX object tools/mlir/tools/mlir-runner/CMakeFiles/mlir-runner.dir/mlir-runner.cpp.o
[8170/8698] Building CXX object tools/mlir/lib/CAPI/Conversion/CMakeFiles/obj.MLIRCAPIConversion.dir/Passes.cpp.o
[8171/8698] Building CXX object tools/flang/lib/Lower/CMakeFiles/FortranLower.dir/OpenMP/OpenMP.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...

```

</details>

https://github.com/llvm/llvm-project/pull/207617

From llvmlistbot at llvm.org  Wed Jul  8 20:15:44 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 08 Jul 2026 20:15:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f1260.824e11a6.f5f27.4803@mx.google.com>


================
@@ -59,6 +59,10 @@ set(MLIR_CONFIG_INCLUDE_DIRS
 set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+set(MLIR_CONFIG_LINK_MLIR_DYLIB "${MLIR_LINK_MLIR_DYLIB}")
----------------
makslevental wrote:

You want
```suggestion
set(MLIR_CONFIG_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})
```
otherwise you might get something like `MLIR_CONFIG_LINK_MLIR_DYLIB="OFF"` which is truthy in CMake. With no quotes this will be either `ON/`OFF`/`""` (where the last is falsey).


https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 20:16:05 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 08 Jul 2026 20:16:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f1275.481afadf.196d57.ba53@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 20:49:03 2026
From: llvmlistbot at llvm.org (Nikhil Kalra)
Date: Wed, 08 Jul 2026 20:49:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Make the Python binding type casters
 well-formed under C++23 (PR #208093)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208093@github.com>
Message-ID: <6a4f1a2f.92d2873c.1ab170.a334@mx.google.com>


https://github.com/nikalra edited https://github.com/llvm/llvm-project/pull/208093

From llvmlistbot at llvm.org  Wed Jul  8 20:50:12 2026
From: llvmlistbot at llvm.org (Nikhil Kalra)
Date: Wed, 08 Jul 2026 20:50:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Make the Python binding type casters
 well-formed under C++23 (PR #208093)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208093@github.com>
Message-ID: <6a4f1a74.da8bafd3.14cd50.3ba9@mx.google.com>


nikalra wrote:

> Seems fine (explicit construction over implicit is better anyway) but I guess this pessimizes the move also? So I'm wondering if there's a way to get both (23 compliance and move)? Then again maybe it doesn't even really matter because of RVO 🤷‍♂️.

Updated the description to point out that the behavior is equivalent to pre-C++23 builds, we just call the lvalue constructor manually for the xvalue (whereas the rvalue decayed to an lvalue automatically before).

https://github.com/llvm/llvm-project/pull/208093

From llvmlistbot at llvm.org  Wed Jul  8 20:50:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 20:50:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 06488f6 - [mlir] Make the Python binding type
 casters well-formed under C++23 (#208093)
Message-ID: <6a4f1a9e.1905e68e.18ad68.dd90@mx.google.com>


Author: Nikhil Kalra
Date: 2026-07-08T20:50:49-07:00
New Revision: 06488f6c8d8941cda4973df32d420c71877be3e6

URL: https://github.com/llvm/llvm-project/commit/06488f6c8d8941cda4973df32d420c71877be3e6
DIFF: https://github.com/llvm/llvm-project/commit/06488f6c8d8941cda4973df32d420c71877be3e6.diff

LOG: [mlir] Make the Python binding type casters well-formed under C++23 (#208093)

The nanobind type/attr/loc/value casters in IRCore.h return a by-value
PyType/PyAttribute/PyLocation/PyValue as their DerivedTy result.

Under pre-C++23 rules, this was a rvalue that would decay to a lvalue
when passed to the DerivedTy constructor.

Under C++23's P2266 (implicit move on return), that return operand is an
xvalue, which cannot bind the PyConcrete* constructors that take a
non-const lvalue reference.

This patch constructs the derived type explicitly (return
DerivedTy(arg);) using the lvalue constructor; this preserves the
pre-C++23 behavior on C++23 builds.

Added: 
    

Modified: 
    mlir/include/mlir/Bindings/Python/IRCore.h

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Bindings/Python/IRCore.h b/mlir/include/mlir/Bindings/Python/IRCore.h
index 1b8c232b009c0..43a751bc9cd9f 100644
--- a/mlir/include/mlir/Bindings/Python/IRCore.h
+++ b/mlir/include/mlir/Bindings/Python/IRCore.h
@@ -996,7 +996,7 @@ class MLIR_PYTHON_API_EXPORTED PyConcreteType : public BaseTy {
       PyGlobals::get().registerTypeCaster(
           DerivedTy::getTypeIdFunction(),
           nanobind::cast<nanobind::callable>(nanobind::cpp_function(
-              [](PyType pyType) -> DerivedTy { return pyType; })),
+              [](PyType pyType) -> DerivedTy { return DerivedTy(pyType); })),
           /*replace*/ true);
     }
 
@@ -1137,7 +1137,7 @@ class MLIR_PYTHON_API_EXPORTED PyConcreteAttribute : public BaseTy {
           DerivedTy::getTypeIdFunction(),
           nanobind::cast<nanobind::callable>(
               nanobind::cpp_function([](PyAttribute pyAttribute) -> DerivedTy {
-                return pyAttribute;
+                return DerivedTy(pyAttribute);
               })),
           /*replace*/ true);
     }
@@ -1223,7 +1223,7 @@ class MLIR_PYTHON_API_EXPORTED PyConcreteLocation : public BaseTy {
       PyGlobals::get().registerTypeCaster(
           DerivedTy::getTypeIdFunction(),
           nanobind::cast<nanobind::callable>(nanobind::cpp_function(
-              [](PyLocation pyLoc) -> DerivedTy { return pyLoc; })),
+              [](PyLocation pyLoc) -> DerivedTy { return DerivedTy(pyLoc); })),
           /*replace*/ true);
     }
     DerivedTy::bindDerived(cls);
@@ -1700,7 +1700,7 @@ class MLIR_PYTHON_API_EXPORTED PyConcreteValue : public PyValue {
       PyGlobals::get().registerValueCaster(
           DerivedTy::getTypeIdFunction(),
           nanobind::cast<nanobind::callable>(nanobind::cpp_function(
-              [](PyValue pyValue) -> DerivedTy { return pyValue; })),
+              [](PyValue pyValue) -> DerivedTy { return DerivedTy(pyValue); })),
           /*replace*/ true);
     }
 


        

From llvmlistbot at llvm.org  Wed Jul  8 20:50:55 2026
From: llvmlistbot at llvm.org (Nikhil Kalra)
Date: Wed, 08 Jul 2026 20:50:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Make the Python binding type casters
 well-formed under C++23 (PR #208093)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208093@github.com>
Message-ID: <6a4f1a9f.faf16b37.2f0e1d.23d8@mx.google.com>


https://github.com/nikalra closed https://github.com/llvm/llvm-project/pull/208093

From llvmlistbot at llvm.org  Wed Jul  8 22:06:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 22:06:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 91746ca - [mlir][cmake] Export
 MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake (#207336)
Message-ID: <6a4f2c71.bb839515.27d21e.6f21@mx.google.com>


Author: qyingwu
Date: 2026-07-08T22:06:52-07:00
New Revision: 91746ca146d4047e9d0ec98fd7d4a3d91f88a759

URL: https://github.com/llvm/llvm-project/commit/91746ca146d4047e9d0ec98fd7d4a3d91f88a759
DIFF: https://github.com/llvm/llvm-project/commit/91746ca146d4047e9d0ec98fd7d4a3d91f88a759.diff

LOG: [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in MLIRConfig.cmake (#207336)

Fixes #197175.

`MLIRConfig.cmake` did not propagate `MLIR_LINK_MLIR_DYLIB` to out-of-tree MLIR users. Standalone projects using `find_package(MLIR CONFIG)` and `include(AddMLIR)` therefore could not observe the value used by `mlir_target_link_libraries(...)` when MLIR was configured with `-DMLIR_LINK_MLIR_DYLIB=ON`.

Export `MLIR_LINK_MLIR_DYLIB` through `MLIRConfig.cmake`.

AI tool usage:
I used ChatGPT/Codex to help understand the issue, identify the analogous LLVM CMake pattern, and draft the PR description. I manually reviewed the final patch and verification.

Added: 
    

Modified: 
    mlir/cmake/modules/CMakeLists.txt
    mlir/cmake/modules/MLIRConfig.cmake.in

Removed: 
    


################################################################################
diff  --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..8a00a3d4de98c 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -59,6 +59,10 @@ set(MLIR_CONFIG_INCLUDE_DIRS
 set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
 set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
 set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+set(MLIR_CONFIG_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})
+llvm_canonicalize_cmake_booleans(
+  MLIR_CONFIG_LINK_MLIR_DYLIB
+)
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in

diff  --git a/mlir/cmake/modules/MLIRConfig.cmake.in b/mlir/cmake/modules/MLIRConfig.cmake.in
index 71f3e028b1e88..fd04a64ce625b 100644
--- a/mlir/cmake/modules/MLIRConfig.cmake.in
+++ b/mlir/cmake/modules/MLIRConfig.cmake.in
@@ -16,6 +16,7 @@ set(MLIR_IRDL_TO_CPP_EXE "@MLIR_CONFIG_IRDL_TO_CPP_EXE@")
 set(MLIR_INSTALL_AGGREGATE_OBJECTS "@MLIR_INSTALL_AGGREGATE_OBJECTS@")
 set(MLIR_ENABLE_BINDINGS_PYTHON "@MLIR_ENABLE_BINDINGS_PYTHON@")
 set(MLIR_ENABLE_EXECUTION_ENGINE "@MLIR_ENABLE_EXECUTION_ENGINE@")
+set(MLIR_LINK_MLIR_DYLIB @MLIR_CONFIG_LINK_MLIR_DYLIB@)
 
 set_property(GLOBAL PROPERTY MLIR_ALL_LIBS "@MLIR_ALL_LIBS@")
 set_property(GLOBAL PROPERTY MLIR_DIALECT_LIBS "@MLIR_DIALECT_LIBS@")


        

From llvmlistbot at llvm.org  Wed Jul  8 22:07:00 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 08 Jul 2026 22:07:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f2c74.3cce8983.225732.3bdd@mx.google.com>


https://github.com/makslevental closed https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 22:07:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 22:07:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][cmake] Export MLIR_LINK_MLIR_DYLIB in
 MLIRConfig.cmake (PR #207336)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207336@github.com>
Message-ID: <6a4f2c83.c56fb102.7c9cd.c96a@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@qyingwu Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/207336

From llvmlistbot at llvm.org  Wed Jul  8 23:32:49 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Wed, 08 Jul 2026 23:32:49 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a4f4091.f0e68a03.3a70dd.0fae@mx.google.com>


================
@@ -4527,6 +4527,46 @@ void OrderedRegionOp::build(OpBuilder &builder, OperationState &state,
 
 LogicalResult OrderedRegionOp::verify() { return verifyOrderedParent(**this); }
 
+//===----------------------------------------------------------------------===//
+// InteropInitOp
+//===----------------------------------------------------------------------===//
+
+void InteropInitOp::build(OpBuilder &builder, OperationState &state,
----------------
Saieiei wrote:

Small cleanup question: do we need these three `build(..., const Interop*Operands &clauses)` overloads?

The current Flang lowering seems to use the full default builders directly, and these overloads appear to build invalid ops if called: `interop_var` is required for all three ops, and `interop_types` is required for `InteropInitOp`, but the clause structs only carry the shared `depend`/`nowait` operands.

Since the default builders remain available, could we drop these custom builder declarations/stubs unless a generic clause-based path needs them?

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul  8 23:45:59 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Wed, 08 Jul 2026 23:45:59 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a4f43a7.92ea0e2a.13fd37.264f@mx.google.com>


================
@@ -218,6 +222,28 @@
 
         integer (kind=omp_interop_kind), parameter, public :: omp_interop_none = 0
 
+!       interop property constants
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_id = -1
----------------
Saieiei wrote:

Thanks, this looks addressed to me.

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul  8 23:55:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 08 Jul 2026 23:55:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a4f45e2.fdc477d6.23591.b911@mx.google.com>


1sgtpepper wrote:

Rebased onto current main as suggested.

https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Thu Jul  9 01:14:42 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Thu, 09 Jul 2026 01:14:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4f5872.ac469dc7.3a57b1.51a7@mx.google.com>


hockyy wrote:

Have you tried newer clang's version? 

https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 01:19:44 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Thu, 09 Jul 2026 01:19:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4f59a0.fdc477d6.23591.7d73@mx.google.com>


hockyy wrote:

#189172 fyi

https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 01:42:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 01:42:01 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Lower DECLARE TARGET
 INDIRECT clause (PR #208387)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208387@github.com>
Message-ID: <6a4f5ed9.8e75e0cb.5e108.3dc3@mx.google.com>


https://github.com/blazie2004 updated https://github.com/llvm/llvm-project/pull/208387

>From 8bb9eb130a42d37d39a2cfba3e4eb544324ba65b Mon Sep 17 00:00:00 2001
From: Jay Satish Kumar Patel <kumarpat at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 8 Jul 2026 04:53:19 -0500
Subject: [PATCH] [Flang][OpenMP] Lower DECLARE TARGET INDIRECT clause

---
 flang/include/flang/Lower/OpenMP.h            |  1 +
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    | 21 +++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 40 ++++++---
 flang/lib/Lower/OpenMP/Utils.h                |  1 +
 .../Optimizer/OpenMP/FunctionFiltering.cpp    |  3 +-
 flang/lib/Semantics/check-omp-structure.cpp   | 13 +++
 .../Lower/OpenMP/Todo/omp-clause-indirect.f90 | 34 --------
 flang/test/Lower/OpenMP/common-block-map.f90  |  2 +-
 .../test/Lower/OpenMP/declare-target-data.f90 | 40 ++++-----
 ...are-target-deferred-marking-reductions.f90 |  4 +-
 .../declare-target-deferred-marking.f90       |  8 +-
 .../OpenMP/declare-target-func-and-subr.f90   | 46 +++++-----
 ...arget-implicit-func-and-subr-cap-enter.f90 | 38 ++++-----
 ...lare-target-implicit-func-and-subr-cap.f90 | 42 +++++-----
 .../declare-target-implicit-tarop-cap.f90     | 14 ++--
 .../OpenMP/declare-target-indirect-clause.f90 | 30 +++++++
 .../OpenMP/declare-target-indirect-merge.f90  | 47 +++++++++++
 .../Lower/OpenMP/declare-target-indirect.f90  | 39 +++++++++
 .../OpenMP/declare-target-unnamed-main.f90    |  2 +-
 .../Lower/OpenMP/function-filtering-2.f90     |  8 +-
 .../OpenMP/omp-declare-target-program-var.f90 |  2 +-
 flang/test/Semantics/indirect02.f90           |  7 +-
 flang/test/Semantics/indirect03.f90           | 31 +++++++
 .../mlir/Dialect/OpenMP/OpenMPAttrDefs.td     |  3 +-
 .../Dialect/OpenMP/OpenMPClauseOperands.h     | 10 ++-
 .../Dialect/OpenMP/OpenMPOpsInterfaces.td     | 20 ++++-
 .../OpenMP/Transforms/MarkDeclareTarget.cpp   |  9 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 83 ++++++++++++++++++-
 .../OpenMP/mark-declare-target-indirect.mlir  | 39 +++++++++
 ...target-declare-target-indirect-device.mlir | 19 +++++
 ...mptarget-declare-target-indirect-host.mlir | 20 +++++
 32 files changed, 515 insertions(+), 162 deletions(-)
 delete mode 100644 flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect.f90
 create mode 100644 flang/test/Semantics/indirect03.f90
 create mode 100644 mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir

diff --git a/flang/include/flang/Lower/OpenMP.h b/flang/include/flang/Lower/OpenMP.h
index 852a120c3782c..06bc3bd929b41 100644
--- a/flang/include/flang/Lower/OpenMP.h
+++ b/flang/include/flang/Lower/OpenMP.h
@@ -58,6 +58,7 @@ struct OMPDeferredDeclareTargetInfo {
   mlir::omp::DeclareTargetCaptureClause declareTargetCaptureClause;
   mlir::omp::DeclareTargetDeviceType declareTargetDeviceType;
   bool automap = false;
+  bool indirect = false;
   const Fortran::semantics::Symbol &sym;
 };
 
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index ddca47f4ba771..1b5cd7194d95b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -13,6 +13,8 @@
 #include "ClauseProcessor.h"
 #include "Utils.h"
 
+#include "flang/Evaluate/fold.h"
+#include "flang/Evaluate/tools.h"
 #include "flang/Lower/ConvertCall.h"
 #include "flang/Lower/ConvertExprToHLFIR.h"
 #include "flang/Lower/OpenMP/Clauses.h"
@@ -444,6 +446,25 @@ bool ClauseProcessor::processDeviceType(
   return false;
 }
 
+bool ClauseProcessor::processIndirect(
+    mlir::omp::IndirectClauseOps &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Indirect>()) {
+    // Case: declare target ... indirect[(scalar-logical-constant)]
+    // An `indirect` clause with no argument defaults to `.true.`.
+    bool isIndirect = true;
+    if (clause->v) {
+      auto foldedExpr = Fortran::evaluate::Fold(
+          semaCtx.foldingContext(), Fortran::common::Clone(*clause->v));
+      if (auto logicalVal = Fortran::evaluate::GetScalarConstantValue<
+              Fortran::evaluate::LogicalResult>(foldedExpr))
+        isIndirect = logicalVal->IsTrue();
+    }
+    result.indirect = isIndirect;
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processDistSchedule(
     lower::StatementContext &stmtCtx,
     mlir::omp::DistScheduleClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 319dbe25bf651..5c2f85ab2dc63 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -143,6 +143,7 @@ class ClauseProcessor {
   processEnter(llvm::SmallVectorImpl<DeclareTargetCaptureInfo> &result) const;
   bool processIf(omp::clause::If::DirectiveNameModifier directiveName,
                  mlir::omp::IfClauseOps &result) const;
+  bool processIndirect(mlir::omp::IndirectClauseOps &result) const;
   bool
   processInReduction(mlir::Location currentLocation,
                      mlir::omp::InReductionClauseOps &result,
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 395746a60af7d..9315f3a25e210 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -888,9 +888,13 @@ static void getDeclareTargetInfo(
     cp.processEnter(symbolAndClause);
     cp.processLink(symbolAndClause);
     cp.processTo(symbolAndClause);
+    cp.processIndirect(clauseOps);
 
-    cp.processTODO<clause::Indirect>(converter.getCurrentLocation(),
-                                     llvm::omp::Directive::OMPD_declare_target);
+    // The `indirect` clause applies to the functions named by the directive
+    // (it requires an `enter` clause). Propagate the directive-level value to
+    // each captured symbol so it reaches the declare target attribute.
+    for (DeclareTargetCaptureInfo &sym : symbolAndClause)
+      sym.indirect = clauseOps.indirect;
   }
 }
 
@@ -914,7 +918,8 @@ static void collectDeferredDeclareTargets(
 
     if (!op) {
       deferredDeclareTarget.push_back({symClause.clause, clauseOps.deviceType,
-                                       symClause.automap, symClause.symbol});
+                                       symClause.automap, symClause.indirect,
+                                       symClause.symbol});
     }
   }
 }
@@ -1162,7 +1167,8 @@ getImplicitMapTypeAndKind(fir::FirOpBuilder &firOpBuilder,
 static void
 markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
                   mlir::omp::DeclareTargetCaptureClause captureClause,
-                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap) {
+                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap,
+                  bool indirect) {
   // TODO: Add support for program local variables with declare target applied
   auto declareTargetOp = llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(op);
   if (!declareTargetOp)
@@ -1172,16 +1178,30 @@ markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
 
   // The function or global already has a declare target applied to it, very
   // likely through implicit capture (usage in another declare target
-  // function/subroutine). It should be marked as any if it has been assigned
-  // both host and nohost, else we skip, as there is no change
+  // function/subroutine), or because it is named by more than one declare
+  // target directive. It should be marked as any if it has been assigned both
+  // host and nohost. The `indirect` modifier is a capability: once any
+  // declaration requests it, it must stay set, so it is merged with logical OR
+  // rather than overwritten (which could drop a previous `indirect = true`).
   if (declareTargetOp.isDeclareTarget()) {
+    bool mergedIndirect =
+        declareTargetOp.getDeclareTargetIndirect() || indirect;
+
     if (declareTargetOp.getDeclareTargetDeviceType() != deviceType)
       declareTargetOp.setDeclareTarget(mlir::omp::DeclareTargetDeviceType::any,
-                                       captureClause, automap);
+                                       captureClause, automap, mergedIndirect);
+    else if (mergedIndirect != declareTargetOp.getDeclareTargetIndirect())
+      // Same device type, but a later declaration added `indirect`; update it
+      // while preserving the already-established capture clause and automap.
+      declareTargetOp.setDeclareTarget(
+          declareTargetOp.getDeclareTargetDeviceType(),
+          declareTargetOp.getDeclareTargetCaptureClause(),
+          declareTargetOp.getDeclareTargetAutomap(), mergedIndirect);
     return;
   }
 
-  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap);
+  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap,
+                                   indirect);
 }
 
 //===----------------------------------------------------------------------===//
@@ -4616,7 +4636,7 @@ genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
       continue;
 
     markDeclareTarget(op, converter, symClause.clause, clauseOps.deviceType,
-                      symClause.automap);
+                      symClause.automap, symClause.indirect);
   }
 }
 
@@ -5592,7 +5612,7 @@ bool Fortran::lower::markOpenMPDeferredDeclareTargetFunctions(
       deviceCodeFound = true;
 
     markDeclareTarget(op, converter, declTar.declareTargetCaptureClause,
-                      devType, declTar.automap);
+                      devType, declTar.automap, declTar.indirect);
   }
 
   return deviceCodeFound;
diff --git a/flang/lib/Lower/OpenMP/Utils.h b/flang/lib/Lower/OpenMP/Utils.h
index 36fde727535d8..218dcdc65059d 100644
--- a/flang/lib/Lower/OpenMP/Utils.h
+++ b/flang/lib/Lower/OpenMP/Utils.h
@@ -49,6 +49,7 @@ namespace omp {
 struct DeclareTargetCaptureInfo {
   mlir::omp::DeclareTargetCaptureClause clause;
   bool automap = false;
+  bool indirect = false;
   const semantics::Symbol &symbol;
 
   DeclareTargetCaptureInfo(mlir::omp::DeclareTargetCaptureClause c,
diff --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index b78b33f31b142..721760bfa72b8 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -180,7 +180,8 @@ class FunctionFilteringPass
         if (declareTargetOp)
           declareTargetOp.setDeclareTarget(
               declareType, omp::DeclareTargetCaptureClause::to,
-              declareTargetOp.getDeclareTargetAutomap());
+              declareTargetOp.getDeclareTargetAutomap(),
+              declareTargetOp.getDeclareTargetIndirect());
       }
       return WalkResult::advance();
     });
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 2de83dccda16f..b29a9365441a3 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2363,6 +2363,19 @@ void OmpStructureChecker::Leave(const parser::OmpDeclareTargetDirective &x) {
     }
     if (indirectClause) {
       CheckAllowedClause(llvm::omp::Clause::OMPC_indirect);
+      // The INDIRECT clause is only allowed together with DEVICE_TYPE(ANY) (an
+      // absent DEVICE_TYPE clause also implies ANY). A host- or device-only
+      // procedure cannot be the target of an indirect device invocation.
+      if (const parser::OmpClause *
+          deviceTypeClause{FindClause(llvm::omp::Clause::OMPC_device_type)}) {
+        const auto &deviceType{
+            std::get<parser::OmpClause::DeviceType>(deviceTypeClause->u)};
+        if (deviceType.v.v !=
+            parser::OmpDeviceTypeClause::DeviceTypeDescription::Any) {
+          context_.Say(x.source,
+              "Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive"_err_en_US);
+        }
+      }
     }
   }
 
diff --git a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90 b/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
deleted file mode 100644
index 82efa8818a83c..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
+++ /dev/null
@@ -1,34 +0,0 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
-
-! RUN: not %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
-
-module functions
-  implicit none
-
-  interface
-    function func() result(i)
-      character(1) :: i
-    end function
-  end interface
-
-contains
-  function func1() result(i)
-    !CHECK: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
-    !$omp declare target enter(func1) indirect(.true.)
-    character(1) :: i
-    i = 'a'
-    return
-  end function
-end module
-
-program main
-  use functions
-  implicit none
-  procedure (func), pointer :: ptr1=>func1
-  character(1) :: val1
-
-  !$omp target map(from: val1)
-  val1 = ptr1()
-  !$omp end target
-
-end program
diff --git a/flang/test/Lower/OpenMP/common-block-map.f90 b/flang/test/Lower/OpenMP/common-block-map.f90
index 7c690c96ddb0b..11b6cc5dee56d 100644
--- a/flang/test/Lower/OpenMP/common-block-map.f90
+++ b/flang/test/Lower/OpenMP/common-block-map.f90
@@ -1,7 +1,7 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s
 
 !CHECK: fir.global common @var_common_(dense<0> : vector<8xi8>) {{.*}} : !fir.array<8xi8>
-!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<8xi8>
+!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<8xi8>
 
 !CHECK-LABEL: func.func @_QPmap_full_block
 !CHECK: %[[CB_ADDR:.*]] = fir.address_of(@var_common_) : !fir.ref<!fir.array<8xi8>>
diff --git a/flang/test/Lower/OpenMP/declare-target-data.f90 b/flang/test/Lower/OpenMP/declare-target-data.f90
index 2e3790303984c..ffffb76eee37f 100644
--- a/flang/test/Lower/OpenMP/declare-target-data.f90
+++ b/flang/test/Lower/OpenMP/declare-target-data.f90
@@ -4,61 +4,61 @@
 module test_0
     implicit none
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : i32
 INTEGER :: data_int = 10
 !$omp declare target link(data_int)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<3xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<3xi32>
 INTEGER :: array_1d(3) = (/1,2,3/)
 !$omp declare target link(array_1d)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<2x2xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<2x2xi32>
 INTEGER :: array_2d(2,2) = reshape((/1,2,3,4/), (/2,2/))
 !$omp declare target link(array_2d)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt1
 !$omp declare target link(pt1)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} target : i32
+!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} target : i32
 INTEGER, TARGET :: pt2_tar = 5
 !$omp declare target link(pt2_tar)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt2 => pt2_tar
 !$omp declare target link(pt2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_to = 5
 !$omp declare target to(data_int_to)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : i32
 INTEGER :: data_int_enter = 5
 !$omp declare target enter(data_int_enter)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_clauseless = 1
 !$omp declare target(data_int_clauseless)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
 REAL :: data_extended_to_1 = 2
 REAL :: data_extended_to_2 = 3
 !$omp declare target to(data_extended_to_1, data_extended_to_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
 REAL :: data_extended_enter_1 = 2
 REAL :: data_extended_enter_2 = 3
 !$omp declare target enter(data_extended_enter_1, data_extended_enter_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
 REAL :: data_extended_link_1 = 2
 REAL :: data_extended_link_2 = 3
 !$omp declare target link(data_extended_link_1, data_extended_link_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true>} target : !fir.box<!fir.heap<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true, indirect = false>} target : !fir.box<!fir.heap<i32>>
 INTEGER, ALLOCATABLE, TARGET :: automap_data
 !$omp declare target enter(automap : automap_data)
 
@@ -66,25 +66,25 @@ module test_0
 end module test_0
 
 PROGRAM commons
-    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one = 1
     REAL :: two = 2
     COMMON /numbers/ one, two
     !$omp declare target(/numbers/)
 
-    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_link = 1
     REAL :: two_link = 2
     COMMON /numbers_link/ one_link, two_link
     !$omp declare target link(/numbers_link/)
 
-    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_to = 1
     REAL :: two_to = 2
     COMMON /numbers_to/ one_to, two_to
     !$omp declare target to(/numbers_to/)
 
-    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_enter = 1
     REAL :: two_enter = 2
     COMMON /numbers_enter/ one_enter, two_enter
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
index 66697ef6bbe70..2bbdd2d071129 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
@@ -31,7 +31,7 @@ end function mycombine
  end program main
 
 !CHECK: func.func {{.*}} @myinit(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !CHECK-LABEL: func.func {{.*}} @mycombine(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
index 528563aba6962..974e0f8a344ab 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
@@ -51,10 +51,10 @@ end subroutine unused_unemitted_interface
  end program main
 
 !HOST-LABEL: func.func {{.*}} @host_interface()
-!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @called_from_target_interface(!fir.ref<i64>, !fir.ref<i64>)
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @any_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @device_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
diff --git a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90 b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
index 46465eece5050..fa181521aa293 100644
--- a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
+++ b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
@@ -6,7 +6,7 @@
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPfunc_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_DEVICE() RESULT(I)
 !$omp declare target to(FUNC_T_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -14,7 +14,7 @@ FUNCTION FUNC_T_DEVICE() RESULT(I)
 END FUNCTION FUNC_T_DEVICE
 
 ! DEVICE-LABEL: func.func @_QPfunc_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -22,7 +22,7 @@ FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 END FUNCTION FUNC_ENTER_DEVICE
 
 ! HOST-LABEL: func.func @_QPfunc_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_HOST() RESULT(I)
 !$omp declare target to(FUNC_T_HOST) device_type(host)
     INTEGER :: I
@@ -30,7 +30,7 @@ FUNCTION FUNC_T_HOST() RESULT(I)
 END FUNCTION FUNC_T_HOST
 
 ! HOST-LABEL: func.func @_QPfunc_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_HOST() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_HOST) device_type(host)
     INTEGER :: I
@@ -38,7 +38,7 @@ FUNCTION FUNC_ENTER_HOST() RESULT(I)
 END FUNCTION FUNC_ENTER_HOST
 
 ! ALL-LABEL: func.func @_QPfunc_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_T_ANY) device_type(any)
     INTEGER :: I
@@ -46,7 +46,7 @@ FUNCTION FUNC_T_ANY() RESULT(I)
 END FUNCTION FUNC_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_ANY) device_type(any)
     INTEGER :: I
@@ -54,7 +54,7 @@ FUNCTION FUNC_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_DEFAULT_T_ANY)
     INTEGER :: I
@@ -62,7 +62,7 @@ FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_DEFAULT_ENTER_ANY)
     INTEGER :: I
@@ -70,7 +70,7 @@ FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 !$omp declare target
     INTEGER :: I
@@ -78,7 +78,7 @@ FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 !$omp declare target(FUNC_DEFAULT_EXTENDEDLIST)
     INTEGER :: I
@@ -86,7 +86,7 @@ FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 END FUNCTION FUNC_DEFAULT_EXTENDEDLIST
 
 ! ALL-LABEL: func.func @_QPfunc_name_as_result()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_NAME_AS_RESULT()
 !$omp declare target(FUNC_NAME_AS_RESULT)
   FUNC_NAME_AS_RESULT = 1.0
@@ -99,61 +99,61 @@ END FUNCTION FUNC_NAME_AS_RESULT
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPsubr_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_DEVICE()
 !$omp declare target to(SUBR_T_DEVICE) device_type(nohost)
 END
 
 ! DEVICE-LABEL: func.func @_QPsubr_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_DEVICE()
 !$omp declare target enter(SUBR_ENTER_DEVICE) device_type(nohost)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_HOST()
 !$omp declare target to(SUBR_T_HOST) device_type(host)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_HOST()
 !$omp declare target enter(SUBR_ENTER_HOST) device_type(host)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_ANY()
 !$omp declare target to(SUBR_T_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_ANY()
 !$omp declare target enter(SUBR_ENTER_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_T_ANY()
 !$omp declare target to(SUBR_DEFAULT_T_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ENTER_ANY()
 !$omp declare target enter(SUBR_DEFAULT_ENTER_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ANY()
 !$omp declare target
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !$omp declare target(SUBR_DEFAULT_EXTENDEDLIST)
 END
@@ -161,7 +161,7 @@ SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 !$omp declare target to(RECURSIVE_DECLARE_TARGET) device_type(nohost)
     INTEGER :: INCREMENT, K
@@ -173,7 +173,7 @@ RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 END FUNCTION RECURSIVE_DECLARE_TARGET
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target_enter
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET_ENTER(INCREMENT) RESULT(K)
 !$omp declare target enter(RECURSIVE_DECLARE_TARGET_ENTER) device_type(nohost)
     INTEGER :: INCREMENT, K
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
index 2caa6ba0b5b1e..b914b8b9814b9 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -12,7 +12,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target enter(target_function_twice_host) device_type(host)
    integer :: i
@@ -20,7 +20,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target enter(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -30,7 +30,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -44,7 +44,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -52,7 +52,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target enter(target_function_test) device_type(nohost)
    integer :: i, j
@@ -63,7 +63,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -71,13 +71,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -85,7 +85,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target enter(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -94,7 +94,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target enter(target_function_test_host) device_type(host)
    integer :: i, j
@@ -105,7 +105,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target enter(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -117,7 +117,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target enter(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -129,28 +129,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target enter(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target enter(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -174,7 +174,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -185,7 +185,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target enter(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
index 32a5058029174..b4ccce4e4b114 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured(toggle) result(k)
    integer :: i, j, k
    logical :: toggle
@@ -19,7 +19,7 @@ end function implicitly_captured
 
 
 ! CHECK-LABEL: func.func @_QPtarget_function
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function(toggle) result(i)
 !$omp declare target
    integer :: i
@@ -30,7 +30,7 @@ end function target_function
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -38,7 +38,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target to(target_function_twice_host) device_type(host)
    integer :: i
@@ -46,7 +46,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target to(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -56,7 +56,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -70,7 +70,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -78,7 +78,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target to(target_function_test) device_type(nohost)
    integer :: i, j
@@ -89,7 +89,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -97,13 +97,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -111,7 +111,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target to(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -120,7 +120,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target to(target_function_test_host) device_type(host)
    integer :: i, j
@@ -131,7 +131,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target to(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -143,7 +143,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target to(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -155,28 +155,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target to(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target to(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -200,7 +200,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -211,7 +211,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target to(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
index c31bfd1e61e8e..109c28f7d0fe5 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! DEVICE-LABEL: func.func @_QPimplicit_capture
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicit_capture() result(i)
    implicit none
    integer :: i
@@ -21,35 +21,35 @@ subroutine subr_target()
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
 !$omp declare target to(implicitly_captured_one_twice) device_type(host)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice_enter() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice_enter
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice_enter() result(k)
 !$omp declare target enter(implicitly_captured_one_twice_enter) device_type(host)
    k = implicitly_captured_nest_twice_enter()
 end function implicitly_captured_one_twice_enter
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(y)
    integer :: y
    y = 5
@@ -67,7 +67,7 @@ end function target_function_test_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_recursive(increment) result(k)
    integer :: increment, k
    if (increment == 10) then
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
new file mode 100644
index 0000000000000..ced2535925888
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
@@ -0,0 +1,30 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+! Check that the INDIRECT clause on a DECLARE TARGET directive is lowered to the
+! `indirect` field of the omp.declare_target attribute.
+
+module functions
+  implicit none
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc_true({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_true() result(i)
+    !$omp declare target enter(func_true) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_implicit({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_implicit() result(i)
+    !$omp declare target enter(func_implicit) indirect
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_false({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  function func_false() result(i)
+    !$omp declare target enter(func_false) indirect(.false.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
new file mode 100644
index 0000000000000..cee898cb84bee
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
@@ -0,0 +1,47 @@
+! Check that the INDIRECT modifier is preserved (merged with logical OR) when a
+! procedure is named by more than one DECLARE TARGET directive. A prior
+! `indirect = true` must not be dropped, either by the device_type merge to
+! `any` or by an early return when the device_type already matches.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+
+module m1
+  implicit none
+contains
+  ! A later directive adds `indirect` with the same (default) device_type.
+  ! CHECK: func.func @_QMm1Pfoo1() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo1() result(i)
+    !$omp declare target enter(foo1)
+    !$omp declare target enter(foo1) indirect(.true.)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m2
+  implicit none
+contains
+  ! `indirect` first, plain second: it must stay set.
+  ! CHECK: func.func @_QMm2Pfoo2() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo2() result(i)
+    !$omp declare target enter(foo2) indirect(.true.)
+    !$omp declare target enter(foo2)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m3
+  implicit none
+contains
+  ! `indirect` (device_type any) followed by a device_type(nohost) declaration:
+  ! the device type merges to `any` and the `indirect = true` must be carried
+  ! over rather than overwritten.
+  ! CHECK: func.func @_QMm3Pfoo3() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo3() result(i)
+    !$omp declare target enter(foo3) indirect(.true.)
+    !$omp declare target enter(foo3) device_type(nohost)
+    integer :: i
+    i = 1
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect.f90 b/flang/test/Lower/OpenMP/declare-target-indirect.f90
new file mode 100644
index 0000000000000..3cd87d9b82575
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect.f90
@@ -0,0 +1,39 @@
+! This test checks the lowering of the OpenMP INDIRECT clause when used with the
+! DECLARE TARGET directive, together with an indirect call (through a procedure
+! pointer) from within a target region.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+module functions
+  implicit none
+
+  interface
+    function func() result(i)
+      character(1) :: i
+    end function
+  end interface
+
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc1({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func1() result(i)
+    !$omp declare target enter(func1) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+    return
+  end function
+end module
+
+program main
+  use functions
+  implicit none
+  procedure (func), pointer :: ptr1=>func1
+  character(1) :: val1
+
+  ! CHECK-LABEL: func.func @_QQmain()
+  ! CHECK: omp.target
+  !$omp target map(from: val1)
+  val1 = ptr1()
+  !$omp end target
+
+end program
diff --git a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90 b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
index f54f7ed14d1d5..9435eae996a98 100644
--- a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
+++ b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
@@ -7,7 +7,7 @@
 ! appropriately mark the function as declare target, even when
 ! unused within the target region.
 
-!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 
 interface
 real function foo (x)
diff --git a/flang/test/Lower/OpenMP/function-filtering-2.f90 b/flang/test/Lower/OpenMP/function-filtering-2.f90
index 6ced7851dd4ea..5cfbe4ad6e031 100644
--- a/flang/test/Lower/OpenMP/function-filtering-2.f90
+++ b/flang/test/Lower/OpenMP/function-filtering-2.f90
@@ -5,13 +5,13 @@
 ! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
 ! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
 
-! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}implicit_invocation{{.*}}(
 subroutine implicit_invocation()
 end subroutine implicit_invocation
 
-! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget{{.*}}(
 subroutine declaretarget()
@@ -19,7 +19,7 @@ subroutine declaretarget()
     call implicit_invocation()
 end subroutine declaretarget
 
-! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget_enter{{.*}}(
 subroutine declaretarget_enter()
@@ -27,7 +27,7 @@ subroutine declaretarget_enter()
     call implicit_invocation()
 end subroutine declaretarget_enter
 
-! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}no_declaretarget{{.*}}(
 subroutine no_declaretarget()
diff --git a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90 b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
index dc23a8190cb8d..df96448eda129 100644
--- a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
+++ b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
@@ -5,7 +5,7 @@ PROGRAM main
     ! HOST-DAG: %[[I_REF:.*]] = fir.alloca f32 {bindc_name = "i", uniq_name = "_QFEi"}
     ! HOST-DAG: %[[I_DECL:.*]]:2 = hlfir.declare %[[I_REF]] {uniq_name = "_QFEi"} : (!fir.ref<f32>) -> (!fir.ref<f32>, !fir.ref<f32>)
     REAL :: I
-    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32 {
+    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32 {
     ! ALL-DAG: %[[UNDEF:.*]] = fir.zero_bits f32
     ! ALL-DAG: fir.has_value %[[UNDEF]] : f32
     ! ALL-DAG: }
diff --git a/flang/test/Semantics/indirect02.f90 b/flang/test/Semantics/indirect02.f90
index cfae406486c67..165599ab072c6 100644
--- a/flang/test/Semantics/indirect02.f90
+++ b/flang/test/Semantics/indirect02.f90
@@ -1,7 +1,8 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
+! This test checks that the OpenMP INDIRECT clause on the DECLARE TARGET
+! directive is rejected before OpenMP 5.1 and accepted from 5.1 onwards.
 
 ! RUN: not %flang -fopenmp -fopenmp-version=50 %s 2>&1 | FileCheck %s --check-prefix="CHECK-50"
-! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -fdebug-unparse %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
 
 module functions
   implicit none
@@ -15,7 +16,7 @@ function func() result(i)
 contains
   function func1() result(i)
     !CHECK-50: INDIRECT clause is not allowed on directive DECLARE TARGET in OpenMP v5.0, try -fopenmp-version=51
-    !CHECK-52: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
+    !CHECK-52: !$OMP DECLARE TARGET ENTER(func1) INDIRECT(.true._4)
     !$omp declare target enter(func1) indirect(.true.)
     character(1) :: i
     i = 'a'
diff --git a/flang/test/Semantics/indirect03.f90 b/flang/test/Semantics/indirect03.f90
new file mode 100644
index 0000000000000..0bab188a9ff51
--- /dev/null
+++ b/flang/test/Semantics/indirect03.f90
@@ -0,0 +1,31 @@
+! This test checks the OpenMP restriction that the INDIRECT clause on a DECLARE
+! TARGET directive is only allowed with DEVICE_TYPE(ANY) (an absent DEVICE_TYPE
+! clause also implies ANY). A host- or device-only procedure cannot be the
+! target of an indirect device invocation.
+
+! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
+
+module functions
+  implicit none
+contains
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_host() result(i)
+    !$omp declare target enter(func_host) device_type(host) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_nohost() result(i)
+    !$omp declare target enter(func_nohost) device_type(nohost) indirect(.true.)
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! DEVICE_TYPE(ANY) with INDIRECT is allowed, so no error is expected here.
+  function func_any() result(i)
+    !$omp declare target enter(func_any) device_type(any) indirect(.true.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
index c9e6764e7d634..bccdc8f6fdd19 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
@@ -45,7 +45,8 @@ def DeclareTargetAttr : OpenMP_Attr<"DeclareTarget", "declaretarget"> {
   let parameters =
       (ins OptionalParameter<"DeclareTargetDeviceTypeAttr">:$device_type,
           OptionalParameter<"DeclareTargetCaptureClauseAttr">:$capture_clause,
-          OptionalParameter<"BoolAttr">:$automap);
+          OptionalParameter<"BoolAttr">:$automap,
+          OptionalParameter<"BoolAttr">:$indirect);
 
   let assemblyFormat = "`<` struct(params) `>`";
 }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
index 6a92b136ef51c..ceb6f0e73e69d 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
@@ -33,6 +33,12 @@ struct DeviceTypeClauseOps {
   DeclareTargetDeviceType deviceType = DeclareTargetDeviceType::any;
 };
 
+struct IndirectClauseOps {
+  /// Whether the declare target entities may be invoked indirectly (through a
+  /// function pointer) from within a target region.
+  bool indirect = false;
+};
+
 //===----------------------------------------------------------------------===//
 // Extra operation operand structures.
 //===----------------------------------------------------------------------===//
@@ -43,8 +49,8 @@ using HostEvaluatedOperands =
     detail::Clauses<CollapseClauseOps, LoopRelatedClauseOps, NumTeamsClauseOps,
                     NumThreadsClauseOps, ThreadLimitClauseOps>;
 
-// TODO: Add `indirect` clause.
-using DeclareTargetOperands = detail::Clauses<DeviceTypeClauseOps>;
+using DeclareTargetOperands =
+    detail::Clauses<DeviceTypeClauseOps, IndirectClauseOps>;
 
 /// omp.target_enter_data, omp.target_exit_data and omp.target_update take the
 /// same clauses, so we give the structure to be shared by all of them a
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index fd500134e10f9..80c46e2ae07c3 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -335,7 +335,7 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
       /*methodName=*/"setDeclareTarget",
       (ins "mlir::omp::DeclareTargetDeviceType":$deviceType,
            "mlir::omp::DeclareTargetCaptureClause":$captureClause,
-           "bool":$automap), [{}], [{
+           "bool":$automap, "bool":$indirect), [{}], [{
         $_op->setAttr("omp.declare_target",
                   mlir::omp::DeclareTargetAttr::get(
                       $_op->getContext(),
@@ -343,7 +343,8 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
                           $_op->getContext(), deviceType),
                       mlir::omp::DeclareTargetCaptureClauseAttr::get(
                           $_op->getContext(), captureClause),
-                      mlir::BoolAttr::get($_op->getContext(), automap)));
+                      mlir::BoolAttr::get($_op->getContext(), automap),
+                      mlir::BoolAttr::get($_op->getContext(), indirect)));
       }]>,
       InterfaceMethod<
       /*description=*/[{
@@ -394,6 +395,21 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
             if (auto autoVal = dAttr.getAutomap())
               return autoVal.getValue();
          return false;
+      }]>,
+      InterfaceMethod<
+      /*description=*/[{
+        Return true if the DeclareTarget attribute has the INDIRECT modifier,
+        indicating that the entity may be invoked through a function pointer
+        from within a target region.
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getDeclareTargetIndirect",
+      (ins), [{}], [{
+        if (mlir::Attribute dTar = $_op->getAttr("omp.declare_target"))
+          if (auto dAttr = llvm::dyn_cast_or_null<mlir::omp::DeclareTargetAttr>(dTar))
+            if (auto indVal = dAttr.getIndirect())
+              return indVal.getValue();
+         return false;
       }]>
   ];
 }
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
index 84b0d40949dba..1d066facdecd3 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
+++ b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
@@ -58,11 +58,16 @@ class MarkDeclareTargetPass
           currentDt != omp::DeclareTargetDeviceType::any) {
         current.setDeclareTarget(omp::DeclareTargetDeviceType::any,
                                  current.getDeclareTargetCaptureClause(),
-                                 current.getDeclareTargetAutomap());
+                                 current.getDeclareTargetAutomap(),
+                                 current.getDeclareTargetIndirect());
       }
     } else {
+      // The `indirect` modifier is a property of the specific declare target
+      // declaration; it is not inherited by functions that are reached only
+      // through (direct) calls. Such implicitly captured functions are marked
+      // declare target, but must not be registered as indirect call targets.
       current.setDeclareTarget(parentInfo.devTy, parentInfo.capClause,
-                               parentInfo.automap);
+                               parentInfo.automap, /*indirect=*/false);
     }
 
     markNestedFuncs(parentInfo, symOp, visited);
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 07fa92bdabe50..53bde833248d3 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8116,6 +8116,69 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Register a `declare target ... indirect` function so that the OpenMP runtime
+/// can resolve indirect calls (through a function pointer) to it from within a
+/// target region. This mirrors clang's
+/// `CGOpenMPRuntime::emitDeclareTargetFunction`.
+///
+/// On the host the function itself is registered as an indirect offload entry,
+/// which causes an offloading entry (with the indirect flag) to be emitted at
+/// module finalization. On the target device a new, externally visible global
+/// holding the address of the function is generated (so the runtime can read
+/// the device address while leaving the function's own linkage and visibility
+/// unchanged) and that global is registered instead.
+static void registerIndirectDeclareTargetFunction(
+    FunctionOpInterface funcOp, llvm::OpenMPIRBuilder *ompBuilder,
+    LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::Function *llvmFunc = moduleTranslation.lookupFunction(funcOp.getName());
+  if (!llvmFunc)
+    return;
+
+  // Build the unique offload entry name using the function as the parent name,
+  // e.g. `__omp_offloading_<device>_<file>_<func>_l<line>`.
+  auto loc = funcOp->getLoc()->findInstanceOf<FileLineColLoc>();
+  auto fileInfoCallBack = [&loc]() {
+    std::string filename = "";
+    std::uint64_t lineNo = 0;
+    if (loc) {
+      filename = loc.getFilename().str();
+      lineNo = loc.getLine();
+    }
+    return std::pair<std::string, std::uint64_t>(llvm::StringRef(filename),
+                                                 lineNo);
+  };
+
+  llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+  llvm::TargetRegionEntryInfo entryInfo = ompBuilder->getTargetEntryUniqueInfo(
+      fileInfoCallBack, vfs, funcOp.getName());
+  llvm::SmallString<128> name;
+  ompBuilder->OffloadInfoManager.getTargetRegionEntryFnName(name, entryInfo);
+
+  llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
+  const llvm::DataLayout &dl = llvmModule->getDataLayout();
+  // The entry tracks a pointer to the function, so its size is the store size
+  // of a pointer.
+  int64_t varSize = dl.getTypeStoreSize(
+      llvm::PointerType::getUnqual(llvmModule->getContext()));
+
+  llvm::Constant *addr = llvmFunc;
+  if (ompBuilder->Config.isTargetDevice()) {
+    llvm::PointerType *fnPtrTy = llvm::PointerType::get(
+        llvmModule->getContext(), dl.getProgramAddressSpace());
+    auto *addrGlobal = new llvm::GlobalVariable(
+        *llvmModule, fnPtrTy, /*isConstant=*/true,
+        llvm::GlobalValue::ExternalLinkage, llvmFunc, name, nullptr,
+        llvm::GlobalValue::NotThreadLocal, dl.getDefaultGlobalsAddressSpace());
+    addrGlobal->setVisibility(llvm::GlobalValue::ProtectedVisibility);
+    addr = addrGlobal;
+  }
+
+  ompBuilder->OffloadInfoManager.registerDeviceGlobalVarEntryInfo(
+      name, addr, varSize,
+      llvm::OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect,
+      llvm::GlobalValue::WeakODRLinkage);
+}
+
 static LogicalResult
 convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                          llvm::OpenMPIRBuilder *ompBuilder,
@@ -8130,13 +8193,25 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (FunctionOpInterface funcOp = dyn_cast<FunctionOpInterface>(op)) {
     if (auto offloadMod = dyn_cast<omp::OffloadModuleInterface>(
             op->getParentOfType<ModuleOp>().getOperation())) {
-      if (!offloadMod.getIsTargetDevice())
-        return success();
-
+      bool isTargetDevice = offloadMod.getIsTargetDevice();
       omp::DeclareTargetDeviceType declareType =
           attribute.getDeviceType().getValue();
+      bool isHostFunc = declareType == omp::DeclareTargetDeviceType::host;
+
+      // A `declare target ... indirect(.true.)` function must be registered so
+      // that indirect calls to it from within a target region can be resolved
+      // by the runtime. This applies to both host and device compilation, but
+      // not to host-only functions that are about to be deleted on the device.
+      mlir::BoolAttr indirectAttr = attribute.getIndirect();
+      if (indirectAttr && indirectAttr.getValue() &&
+          !(isTargetDevice && isHostFunc))
+        registerIndirectDeclareTargetFunction(funcOp, ompBuilder,
+                                              moduleTranslation);
+
+      if (!isTargetDevice)
+        return success();
 
-      if (declareType == omp::DeclareTargetDeviceType::host) {
+      if (isHostFunc) {
         llvm::Function *llvmFunc =
             moduleTranslation.lookupFunction(funcOp.getName());
         llvmFunc->dropAllReferences();
diff --git a/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
new file mode 100644
index 0000000000000..3ebdcb66cd017
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
@@ -0,0 +1,39 @@
+// RUN: mlir-opt -omp-mark-declare-target -split-input-file %s | FileCheck %s
+
+// The `omp-mark-declare-target` pass marks functions that are reachable from
+// explicit target code as implicitly declare target. The `indirect` modifier
+// however is a property of the specific declare target declaration and must NOT
+// be propagated to functions that are only reached through (direct) calls.
+
+// A function explicitly declared `indirect` that directly calls another
+// function: the callee is implicitly captured and must be marked declare target
+// with `indirect = false`, not inherit the parent's `indirect = true`.
+module {
+  // CHECK: func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @direct_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @direct_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  func.func @direct_callee() {
+    return
+  }
+}
+
+// -----
+
+// A callee that is itself explicitly declared `indirect` keeps its own value
+// (the pass must not clobber it).
+module {
+  // CHECK: func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @explicitly_indirect_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
new file mode 100644
index 0000000000000..ad86bed2770fa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the device-side lowering of `declare target ... indirect` functions. A
+// new global holding the address of the function is generated with protected
+// visibility so the runtime can access it. A function marked `indirect = false`
+// must not generate such a global.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: @[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = protected constant ptr @indirect_fn
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an indirect global.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
new file mode 100644
index 0000000000000..1d0c40e3e3914
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
@@ -0,0 +1,20 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the host-side lowering of `declare target ... indirect` functions. The
+// runtime needs an offload entry (flag 8 == OMPTargetGlobalVarEntryIndirect)
+// so that indirect calls within a target region can be resolved. A function
+// marked `indirect = false` must not register an offload entry.
+
+// CHECK-DAG: %struct.__tgt_offload_entry = type { i64, i16, i16, i32, ptr, ptr, i64, i64, ptr }
+module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_target_device = false} {
+  // CHECK: @.offloading.entry.[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = weak_odr constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 8, ptr @indirect_fn, ptr @{{.*}}, i64 8, i64 0, ptr null }
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an offload entry.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}


From llvmlistbot at llvm.org  Thu Jul  9 01:52:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 01:52:20 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Lower DECLARE TARGET
 INDIRECT clause (PR #208387)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208387@github.com>
Message-ID: <6a4f6144.58bdd12c.19c81.d663@mx.google.com>


https://github.com/blazie2004 converted_to_draft https://github.com/llvm/llvm-project/pull/208387

From llvmlistbot at llvm.org  Thu Jul  9 01:58:04 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Thu, 09 Jul 2026 01:58:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4f629c.af6ba3ed.36d9db.154f@mx.google.com>


https://github.com/hockyy approved this pull request.


https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 02:48:21 2026
From: llvmlistbot at llvm.org (Momchil Velikov)
Date: Thu, 09 Jul 2026 02:48:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Install missing standalone helper
	tools (PR #205066)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205066@github.com>
Message-ID: <6a4f6e65.331483e7.2440f8.a431@mx.google.com>


momchil-velikov wrote:

Ping?

https://github.com/llvm/llvm-project/pull/205066

From llvmlistbot at llvm.org  Thu Jul  9 03:00:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 03:00:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Lower DECLARE TARGET
 INDIRECT clause (PR #208387)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208387@github.com>
Message-ID: <6a4f7141.e55dc341.1f19bd.d22c@mx.google.com>


https://github.com/blazie2004 updated https://github.com/llvm/llvm-project/pull/208387

>From efc99a67c223770ac3503fbe52738e073630a363 Mon Sep 17 00:00:00 2001
From: Jay Satish Kumar Patel <kumarpat at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 8 Jul 2026 04:53:19 -0500
Subject: [PATCH] [Flang][OpenMP] Lower DECLARE TARGET INDIRECT clause

---
 flang/include/flang/Lower/OpenMP.h            |  1 +
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    | 21 +++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 40 ++++++---
 flang/lib/Lower/OpenMP/Utils.h                |  1 +
 .../Optimizer/OpenMP/FunctionFiltering.cpp    |  3 +-
 flang/lib/Semantics/check-omp-structure.cpp   | 15 ++++
 .../Lower/OpenMP/Todo/omp-clause-indirect.f90 | 34 --------
 flang/test/Lower/OpenMP/common-block-map.f90  |  2 +-
 .../test/Lower/OpenMP/declare-target-data.f90 | 40 ++++-----
 ...are-target-deferred-marking-reductions.f90 |  4 +-
 .../declare-target-deferred-marking.f90       |  8 +-
 .../OpenMP/declare-target-func-and-subr.f90   | 46 +++++-----
 ...arget-implicit-func-and-subr-cap-enter.f90 | 38 ++++-----
 ...lare-target-implicit-func-and-subr-cap.f90 | 42 +++++-----
 .../declare-target-implicit-tarop-cap.f90     | 14 ++--
 .../OpenMP/declare-target-indirect-clause.f90 | 30 +++++++
 .../OpenMP/declare-target-indirect-merge.f90  | 47 +++++++++++
 .../Lower/OpenMP/declare-target-indirect.f90  | 39 +++++++++
 .../OpenMP/declare-target-unnamed-main.f90    |  2 +-
 .../Lower/OpenMP/function-filtering-2.f90     |  8 +-
 .../OpenMP/omp-declare-target-program-var.f90 |  2 +-
 flang/test/Semantics/indirect02.f90           |  7 +-
 flang/test/Semantics/indirect03.f90           | 31 +++++++
 .../mlir/Dialect/OpenMP/OpenMPAttrDefs.td     |  3 +-
 .../Dialect/OpenMP/OpenMPClauseOperands.h     | 10 ++-
 .../Dialect/OpenMP/OpenMPOpsInterfaces.td     | 20 ++++-
 .../OpenMP/Transforms/MarkDeclareTarget.cpp   |  9 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 83 ++++++++++++++++++-
 .../OpenMP/mark-declare-target-indirect.mlir  | 39 +++++++++
 ...target-declare-target-indirect-device.mlir | 19 +++++
 ...mptarget-declare-target-indirect-host.mlir | 20 +++++
 32 files changed, 517 insertions(+), 162 deletions(-)
 delete mode 100644 flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect.f90
 create mode 100644 flang/test/Semantics/indirect03.f90
 create mode 100644 mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir

diff --git a/flang/include/flang/Lower/OpenMP.h b/flang/include/flang/Lower/OpenMP.h
index c70393c78bbec..fec8661e7c64f 100644
--- a/flang/include/flang/Lower/OpenMP.h
+++ b/flang/include/flang/Lower/OpenMP.h
@@ -57,6 +57,7 @@ struct OMPDeferredDeclareTargetInfo {
   mlir::omp::DeclareTargetCaptureClause declareTargetCaptureClause;
   mlir::omp::DeclareTargetDeviceType declareTargetDeviceType;
   bool automap = false;
+  bool indirect = false;
   const Fortran::semantics::Symbol &sym;
 };
 
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index 05e7edc8bdc6a..7297e974e356b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -13,6 +13,8 @@
 #include "ClauseProcessor.h"
 #include "Utils.h"
 
+#include "flang/Evaluate/fold.h"
+#include "flang/Evaluate/tools.h"
 #include "flang/Lower/ConvertCall.h"
 #include "flang/Lower/ConvertExprToHLFIR.h"
 #include "flang/Lower/OpenMP/Clauses.h"
@@ -457,6 +459,25 @@ bool ClauseProcessor::processDeviceType(
   return false;
 }
 
+bool ClauseProcessor::processIndirect(
+    mlir::omp::IndirectClauseOps &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Indirect>()) {
+    // Case: declare target ... indirect[(scalar-logical-constant)]
+    // An `indirect` clause with no argument defaults to `.true.`.
+    bool isIndirect = true;
+    if (clause->v) {
+      auto foldedExpr = Fortran::evaluate::Fold(
+          semaCtx.foldingContext(), Fortran::common::Clone(*clause->v));
+      if (auto logicalVal = Fortran::evaluate::GetScalarConstantValue<
+              Fortran::evaluate::LogicalResult>(foldedExpr))
+        isIndirect = logicalVal->IsTrue();
+    }
+    result.indirect = isIndirect;
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processDistSchedule(
     lower::StatementContext &stmtCtx,
     mlir::omp::DistScheduleClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1435af3e844e9..b00236c81b40d 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -142,6 +142,7 @@ class ClauseProcessor {
   processEnter(llvm::SmallVectorImpl<DeclareTargetCaptureInfo> &result) const;
   bool processIf(omp::clause::If::DirectiveNameModifier directiveName,
                  mlir::omp::IfClauseOps &result) const;
+  bool processIndirect(mlir::omp::IndirectClauseOps &result) const;
   bool
   processInReduction(mlir::Location currentLocation,
                      mlir::omp::InReductionClauseOps &result,
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 7939e0db655e4..4554f8c5db88f 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1409,9 +1409,13 @@ static void getDeclareTargetInfo(
     cp.processEnter(symbolAndClause);
     cp.processLink(symbolAndClause);
     cp.processTo(symbolAndClause);
+    cp.processIndirect(clauseOps);
 
-    cp.processTODO<clause::Indirect>(converter.getCurrentLocation(),
-                                     llvm::omp::Directive::OMPD_declare_target);
+    // The `indirect` clause applies to the functions named by the directive
+    // (it requires an `enter` clause). Propagate the directive-level value to
+    // each captured symbol so it reaches the declare target attribute.
+    for (DeclareTargetCaptureInfo &sym : symbolAndClause)
+      sym.indirect = clauseOps.indirect;
   }
 }
 
@@ -1435,7 +1439,8 @@ static void collectDeferredDeclareTargets(
 
     if (!op) {
       deferredDeclareTarget.push_back({symClause.clause, clauseOps.deviceType,
-                                       symClause.automap, symClause.symbol});
+                                       symClause.automap, symClause.indirect,
+                                       symClause.symbol});
     }
   }
 }
@@ -1714,7 +1719,8 @@ getImplicitMapTypeAndKind(fir::FirOpBuilder &firOpBuilder,
 static void
 markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
                   mlir::omp::DeclareTargetCaptureClause captureClause,
-                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap) {
+                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap,
+                  bool indirect) {
   // TODO: Add support for program local variables with declare target applied
   auto declareTargetOp = llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(op);
   if (!declareTargetOp)
@@ -1724,16 +1730,30 @@ markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
 
   // The function or global already has a declare target applied to it, very
   // likely through implicit capture (usage in another declare target
-  // function/subroutine). It should be marked as any if it has been assigned
-  // both host and nohost, else we skip, as there is no change
+  // function/subroutine), or because it is named by more than one declare
+  // target directive. It should be marked as any if it has been assigned both
+  // host and nohost. The `indirect` modifier is a capability: once any
+  // declaration requests it, it must stay set, so it is merged with logical OR
+  // rather than overwritten (which could drop a previous `indirect = true`).
   if (declareTargetOp.isDeclareTarget()) {
+    bool mergedIndirect =
+        declareTargetOp.getDeclareTargetIndirect() || indirect;
+
     if (declareTargetOp.getDeclareTargetDeviceType() != deviceType)
       declareTargetOp.setDeclareTarget(mlir::omp::DeclareTargetDeviceType::any,
-                                       captureClause, automap);
+                                       captureClause, automap, mergedIndirect);
+    else if (mergedIndirect != declareTargetOp.getDeclareTargetIndirect())
+      // Same device type, but a later declaration added `indirect`; update it
+      // while preserving the already-established capture clause and automap.
+      declareTargetOp.setDeclareTarget(
+          declareTargetOp.getDeclareTargetDeviceType(),
+          declareTargetOp.getDeclareTargetCaptureClause(),
+          declareTargetOp.getDeclareTargetAutomap(), mergedIndirect);
     return;
   }
 
-  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap);
+  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap,
+                                   indirect);
 }
 
 //===----------------------------------------------------------------------===//
@@ -5498,7 +5518,7 @@ genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
       continue;
 
     markDeclareTarget(op, converter, symClause.clause, clauseOps.deviceType,
-                      symClause.automap);
+                      symClause.automap, symClause.indirect);
   }
 }
 
@@ -6465,7 +6485,7 @@ bool Fortran::lower::markOpenMPDeferredDeclareTargetFunctions(
       deviceCodeFound = true;
 
     markDeclareTarget(op, converter, declTar.declareTargetCaptureClause,
-                      devType, declTar.automap);
+                      devType, declTar.automap, declTar.indirect);
   }
 
   return deviceCodeFound;
diff --git a/flang/lib/Lower/OpenMP/Utils.h b/flang/lib/Lower/OpenMP/Utils.h
index 94f85c43f7033..db38c2b755be7 100644
--- a/flang/lib/Lower/OpenMP/Utils.h
+++ b/flang/lib/Lower/OpenMP/Utils.h
@@ -53,6 +53,7 @@ namespace omp {
 struct DeclareTargetCaptureInfo {
   mlir::omp::DeclareTargetCaptureClause clause;
   bool automap = false;
+  bool indirect = false;
   const semantics::Symbol &symbol;
 
   DeclareTargetCaptureInfo(mlir::omp::DeclareTargetCaptureClause c,
diff --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index e8bcc0c7fa062..22dc3ec8fbf69 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -136,7 +136,8 @@ class FunctionFilteringPass
         if (declareTargetOp)
           declareTargetOp.setDeclareTarget(
               declareType, omp::DeclareTargetCaptureClause::to,
-              declareTargetOp.getDeclareTargetAutomap());
+              declareTargetOp.getDeclareTargetAutomap(),
+              declareTargetOp.getDeclareTargetIndirect());
       }
       return WalkResult::advance();
     });
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 02928401a382a..d8ee8f4574229 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2557,6 +2557,21 @@ void OmpStructureChecker::Leave(const parser::OmpDeclareTargetDirective &x) {
       context_.Warn(common::UsageWarning::OpenMPUsage, toClause->source,
           "The usage of TO clause on DECLARE TARGET directive has been deprecated. Use ENTER clause instead."_warn_en_US);
     }
+    if (indirectClause) {
+      // The INDIRECT clause is only allowed together with DEVICE_TYPE(ANY) (an
+      // absent DEVICE_TYPE clause also implies ANY). A host- or device-only
+      // procedure cannot be the target of an indirect device invocation.
+      if (const parser::OmpClause *deviceTypeClause{
+              FindClause(llvm::omp::Clause::OMPC_device_type)}) {
+        const auto &deviceType{
+            std::get<parser::OmpClause::DeviceType>(deviceTypeClause->u)};
+        if (deviceType.v.v !=
+            parser::OmpDeviceTypeClause::DeviceTypeDescription::Any) {
+          context_.Say(x.source,
+              "Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive"_err_en_US);
+        }
+      }
+    }
   }
 
   bool toClauseFound{false};
diff --git a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90 b/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
deleted file mode 100644
index 82efa8818a83c..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
+++ /dev/null
@@ -1,34 +0,0 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
-
-! RUN: not %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
-
-module functions
-  implicit none
-
-  interface
-    function func() result(i)
-      character(1) :: i
-    end function
-  end interface
-
-contains
-  function func1() result(i)
-    !CHECK: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
-    !$omp declare target enter(func1) indirect(.true.)
-    character(1) :: i
-    i = 'a'
-    return
-  end function
-end module
-
-program main
-  use functions
-  implicit none
-  procedure (func), pointer :: ptr1=>func1
-  character(1) :: val1
-
-  !$omp target map(from: val1)
-  val1 = ptr1()
-  !$omp end target
-
-end program
diff --git a/flang/test/Lower/OpenMP/common-block-map.f90 b/flang/test/Lower/OpenMP/common-block-map.f90
index 98836306ef041..3daf19e97e6fd 100644
--- a/flang/test/Lower/OpenMP/common-block-map.f90
+++ b/flang/test/Lower/OpenMP/common-block-map.f90
@@ -1,7 +1,7 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s
 
 !CHECK: fir.global common @var_common_(dense<0> : vector<8xi8>) {{.*}} : !fir.array<8xi8>
-!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<8xi8>
+!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<8xi8>
 
 !CHECK-LABEL: func.func @_QPmap_full_block
 !CHECK: %[[CB_ADDR:.*]] = fir.address_of(@var_common_) : !fir.ref<!fir.array<8xi8>>
diff --git a/flang/test/Lower/OpenMP/declare-target-data.f90 b/flang/test/Lower/OpenMP/declare-target-data.f90
index 2e3790303984c..ffffb76eee37f 100644
--- a/flang/test/Lower/OpenMP/declare-target-data.f90
+++ b/flang/test/Lower/OpenMP/declare-target-data.f90
@@ -4,61 +4,61 @@
 module test_0
     implicit none
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : i32
 INTEGER :: data_int = 10
 !$omp declare target link(data_int)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<3xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<3xi32>
 INTEGER :: array_1d(3) = (/1,2,3/)
 !$omp declare target link(array_1d)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<2x2xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<2x2xi32>
 INTEGER :: array_2d(2,2) = reshape((/1,2,3,4/), (/2,2/))
 !$omp declare target link(array_2d)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt1
 !$omp declare target link(pt1)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} target : i32
+!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} target : i32
 INTEGER, TARGET :: pt2_tar = 5
 !$omp declare target link(pt2_tar)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt2 => pt2_tar
 !$omp declare target link(pt2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_to = 5
 !$omp declare target to(data_int_to)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : i32
 INTEGER :: data_int_enter = 5
 !$omp declare target enter(data_int_enter)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_clauseless = 1
 !$omp declare target(data_int_clauseless)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
 REAL :: data_extended_to_1 = 2
 REAL :: data_extended_to_2 = 3
 !$omp declare target to(data_extended_to_1, data_extended_to_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
 REAL :: data_extended_enter_1 = 2
 REAL :: data_extended_enter_2 = 3
 !$omp declare target enter(data_extended_enter_1, data_extended_enter_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
 REAL :: data_extended_link_1 = 2
 REAL :: data_extended_link_2 = 3
 !$omp declare target link(data_extended_link_1, data_extended_link_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true>} target : !fir.box<!fir.heap<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true, indirect = false>} target : !fir.box<!fir.heap<i32>>
 INTEGER, ALLOCATABLE, TARGET :: automap_data
 !$omp declare target enter(automap : automap_data)
 
@@ -66,25 +66,25 @@ module test_0
 end module test_0
 
 PROGRAM commons
-    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one = 1
     REAL :: two = 2
     COMMON /numbers/ one, two
     !$omp declare target(/numbers/)
 
-    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_link = 1
     REAL :: two_link = 2
     COMMON /numbers_link/ one_link, two_link
     !$omp declare target link(/numbers_link/)
 
-    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_to = 1
     REAL :: two_to = 2
     COMMON /numbers_to/ one_to, two_to
     !$omp declare target to(/numbers_to/)
 
-    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_enter = 1
     REAL :: two_enter = 2
     COMMON /numbers_enter/ one_enter, two_enter
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
index 66697ef6bbe70..2bbdd2d071129 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
@@ -31,7 +31,7 @@ end function mycombine
  end program main
 
 !CHECK: func.func {{.*}} @myinit(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !CHECK-LABEL: func.func {{.*}} @mycombine(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
index 528563aba6962..974e0f8a344ab 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
@@ -51,10 +51,10 @@ end subroutine unused_unemitted_interface
  end program main
 
 !HOST-LABEL: func.func {{.*}} @host_interface()
-!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @called_from_target_interface(!fir.ref<i64>, !fir.ref<i64>)
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @any_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @device_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
diff --git a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90 b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
index 46465eece5050..fa181521aa293 100644
--- a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
+++ b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
@@ -6,7 +6,7 @@
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPfunc_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_DEVICE() RESULT(I)
 !$omp declare target to(FUNC_T_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -14,7 +14,7 @@ FUNCTION FUNC_T_DEVICE() RESULT(I)
 END FUNCTION FUNC_T_DEVICE
 
 ! DEVICE-LABEL: func.func @_QPfunc_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -22,7 +22,7 @@ FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 END FUNCTION FUNC_ENTER_DEVICE
 
 ! HOST-LABEL: func.func @_QPfunc_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_HOST() RESULT(I)
 !$omp declare target to(FUNC_T_HOST) device_type(host)
     INTEGER :: I
@@ -30,7 +30,7 @@ FUNCTION FUNC_T_HOST() RESULT(I)
 END FUNCTION FUNC_T_HOST
 
 ! HOST-LABEL: func.func @_QPfunc_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_HOST() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_HOST) device_type(host)
     INTEGER :: I
@@ -38,7 +38,7 @@ FUNCTION FUNC_ENTER_HOST() RESULT(I)
 END FUNCTION FUNC_ENTER_HOST
 
 ! ALL-LABEL: func.func @_QPfunc_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_T_ANY) device_type(any)
     INTEGER :: I
@@ -46,7 +46,7 @@ FUNCTION FUNC_T_ANY() RESULT(I)
 END FUNCTION FUNC_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_ANY) device_type(any)
     INTEGER :: I
@@ -54,7 +54,7 @@ FUNCTION FUNC_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_DEFAULT_T_ANY)
     INTEGER :: I
@@ -62,7 +62,7 @@ FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_DEFAULT_ENTER_ANY)
     INTEGER :: I
@@ -70,7 +70,7 @@ FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 !$omp declare target
     INTEGER :: I
@@ -78,7 +78,7 @@ FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 !$omp declare target(FUNC_DEFAULT_EXTENDEDLIST)
     INTEGER :: I
@@ -86,7 +86,7 @@ FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 END FUNCTION FUNC_DEFAULT_EXTENDEDLIST
 
 ! ALL-LABEL: func.func @_QPfunc_name_as_result()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_NAME_AS_RESULT()
 !$omp declare target(FUNC_NAME_AS_RESULT)
   FUNC_NAME_AS_RESULT = 1.0
@@ -99,61 +99,61 @@ END FUNCTION FUNC_NAME_AS_RESULT
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPsubr_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_DEVICE()
 !$omp declare target to(SUBR_T_DEVICE) device_type(nohost)
 END
 
 ! DEVICE-LABEL: func.func @_QPsubr_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_DEVICE()
 !$omp declare target enter(SUBR_ENTER_DEVICE) device_type(nohost)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_HOST()
 !$omp declare target to(SUBR_T_HOST) device_type(host)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_HOST()
 !$omp declare target enter(SUBR_ENTER_HOST) device_type(host)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_ANY()
 !$omp declare target to(SUBR_T_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_ANY()
 !$omp declare target enter(SUBR_ENTER_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_T_ANY()
 !$omp declare target to(SUBR_DEFAULT_T_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ENTER_ANY()
 !$omp declare target enter(SUBR_DEFAULT_ENTER_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ANY()
 !$omp declare target
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !$omp declare target(SUBR_DEFAULT_EXTENDEDLIST)
 END
@@ -161,7 +161,7 @@ SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 !$omp declare target to(RECURSIVE_DECLARE_TARGET) device_type(nohost)
     INTEGER :: INCREMENT, K
@@ -173,7 +173,7 @@ RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 END FUNCTION RECURSIVE_DECLARE_TARGET
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target_enter
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET_ENTER(INCREMENT) RESULT(K)
 !$omp declare target enter(RECURSIVE_DECLARE_TARGET_ENTER) device_type(nohost)
     INTEGER :: INCREMENT, K
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
index 2caa6ba0b5b1e..b914b8b9814b9 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -12,7 +12,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target enter(target_function_twice_host) device_type(host)
    integer :: i
@@ -20,7 +20,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target enter(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -30,7 +30,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -44,7 +44,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -52,7 +52,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target enter(target_function_test) device_type(nohost)
    integer :: i, j
@@ -63,7 +63,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -71,13 +71,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -85,7 +85,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target enter(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -94,7 +94,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target enter(target_function_test_host) device_type(host)
    integer :: i, j
@@ -105,7 +105,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target enter(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -117,7 +117,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target enter(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -129,28 +129,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target enter(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target enter(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -174,7 +174,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -185,7 +185,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target enter(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
index 32a5058029174..b4ccce4e4b114 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured(toggle) result(k)
    integer :: i, j, k
    logical :: toggle
@@ -19,7 +19,7 @@ end function implicitly_captured
 
 
 ! CHECK-LABEL: func.func @_QPtarget_function
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function(toggle) result(i)
 !$omp declare target
    integer :: i
@@ -30,7 +30,7 @@ end function target_function
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -38,7 +38,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target to(target_function_twice_host) device_type(host)
    integer :: i
@@ -46,7 +46,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target to(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -56,7 +56,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -70,7 +70,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -78,7 +78,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target to(target_function_test) device_type(nohost)
    integer :: i, j
@@ -89,7 +89,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -97,13 +97,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -111,7 +111,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target to(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -120,7 +120,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target to(target_function_test_host) device_type(host)
    integer :: i, j
@@ -131,7 +131,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target to(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -143,7 +143,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target to(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -155,28 +155,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target to(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target to(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -200,7 +200,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -211,7 +211,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target to(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
index c31bfd1e61e8e..109c28f7d0fe5 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! DEVICE-LABEL: func.func @_QPimplicit_capture
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicit_capture() result(i)
    implicit none
    integer :: i
@@ -21,35 +21,35 @@ subroutine subr_target()
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
 !$omp declare target to(implicitly_captured_one_twice) device_type(host)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice_enter() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice_enter
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice_enter() result(k)
 !$omp declare target enter(implicitly_captured_one_twice_enter) device_type(host)
    k = implicitly_captured_nest_twice_enter()
 end function implicitly_captured_one_twice_enter
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(y)
    integer :: y
    y = 5
@@ -67,7 +67,7 @@ end function target_function_test_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_recursive(increment) result(k)
    integer :: increment, k
    if (increment == 10) then
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
new file mode 100644
index 0000000000000..ced2535925888
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
@@ -0,0 +1,30 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+! Check that the INDIRECT clause on a DECLARE TARGET directive is lowered to the
+! `indirect` field of the omp.declare_target attribute.
+
+module functions
+  implicit none
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc_true({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_true() result(i)
+    !$omp declare target enter(func_true) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_implicit({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_implicit() result(i)
+    !$omp declare target enter(func_implicit) indirect
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_false({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  function func_false() result(i)
+    !$omp declare target enter(func_false) indirect(.false.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
new file mode 100644
index 0000000000000..cee898cb84bee
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
@@ -0,0 +1,47 @@
+! Check that the INDIRECT modifier is preserved (merged with logical OR) when a
+! procedure is named by more than one DECLARE TARGET directive. A prior
+! `indirect = true` must not be dropped, either by the device_type merge to
+! `any` or by an early return when the device_type already matches.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+
+module m1
+  implicit none
+contains
+  ! A later directive adds `indirect` with the same (default) device_type.
+  ! CHECK: func.func @_QMm1Pfoo1() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo1() result(i)
+    !$omp declare target enter(foo1)
+    !$omp declare target enter(foo1) indirect(.true.)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m2
+  implicit none
+contains
+  ! `indirect` first, plain second: it must stay set.
+  ! CHECK: func.func @_QMm2Pfoo2() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo2() result(i)
+    !$omp declare target enter(foo2) indirect(.true.)
+    !$omp declare target enter(foo2)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m3
+  implicit none
+contains
+  ! `indirect` (device_type any) followed by a device_type(nohost) declaration:
+  ! the device type merges to `any` and the `indirect = true` must be carried
+  ! over rather than overwritten.
+  ! CHECK: func.func @_QMm3Pfoo3() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo3() result(i)
+    !$omp declare target enter(foo3) indirect(.true.)
+    !$omp declare target enter(foo3) device_type(nohost)
+    integer :: i
+    i = 1
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect.f90 b/flang/test/Lower/OpenMP/declare-target-indirect.f90
new file mode 100644
index 0000000000000..3cd87d9b82575
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect.f90
@@ -0,0 +1,39 @@
+! This test checks the lowering of the OpenMP INDIRECT clause when used with the
+! DECLARE TARGET directive, together with an indirect call (through a procedure
+! pointer) from within a target region.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+module functions
+  implicit none
+
+  interface
+    function func() result(i)
+      character(1) :: i
+    end function
+  end interface
+
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc1({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func1() result(i)
+    !$omp declare target enter(func1) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+    return
+  end function
+end module
+
+program main
+  use functions
+  implicit none
+  procedure (func), pointer :: ptr1=>func1
+  character(1) :: val1
+
+  ! CHECK-LABEL: func.func @_QQmain()
+  ! CHECK: omp.target
+  !$omp target map(from: val1)
+  val1 = ptr1()
+  !$omp end target
+
+end program
diff --git a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90 b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
index f54f7ed14d1d5..9435eae996a98 100644
--- a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
+++ b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
@@ -7,7 +7,7 @@
 ! appropriately mark the function as declare target, even when
 ! unused within the target region.
 
-!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 
 interface
 real function foo (x)
diff --git a/flang/test/Lower/OpenMP/function-filtering-2.f90 b/flang/test/Lower/OpenMP/function-filtering-2.f90
index 6ced7851dd4ea..5cfbe4ad6e031 100644
--- a/flang/test/Lower/OpenMP/function-filtering-2.f90
+++ b/flang/test/Lower/OpenMP/function-filtering-2.f90
@@ -5,13 +5,13 @@
 ! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
 ! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
 
-! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}implicit_invocation{{.*}}(
 subroutine implicit_invocation()
 end subroutine implicit_invocation
 
-! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget{{.*}}(
 subroutine declaretarget()
@@ -19,7 +19,7 @@ subroutine declaretarget()
     call implicit_invocation()
 end subroutine declaretarget
 
-! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget_enter{{.*}}(
 subroutine declaretarget_enter()
@@ -27,7 +27,7 @@ subroutine declaretarget_enter()
     call implicit_invocation()
 end subroutine declaretarget_enter
 
-! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}no_declaretarget{{.*}}(
 subroutine no_declaretarget()
diff --git a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90 b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
index dc23a8190cb8d..df96448eda129 100644
--- a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
+++ b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
@@ -5,7 +5,7 @@ PROGRAM main
     ! HOST-DAG: %[[I_REF:.*]] = fir.alloca f32 {bindc_name = "i", uniq_name = "_QFEi"}
     ! HOST-DAG: %[[I_DECL:.*]]:2 = hlfir.declare %[[I_REF]] {uniq_name = "_QFEi"} : (!fir.ref<f32>) -> (!fir.ref<f32>, !fir.ref<f32>)
     REAL :: I
-    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32 {
+    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32 {
     ! ALL-DAG: %[[UNDEF:.*]] = fir.zero_bits f32
     ! ALL-DAG: fir.has_value %[[UNDEF]] : f32
     ! ALL-DAG: }
diff --git a/flang/test/Semantics/indirect02.f90 b/flang/test/Semantics/indirect02.f90
index 3fae39f1c2281..8835919341a9b 100644
--- a/flang/test/Semantics/indirect02.f90
+++ b/flang/test/Semantics/indirect02.f90
@@ -1,7 +1,8 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
+! This test checks that the OpenMP INDIRECT clause on the DECLARE TARGET
+! directive is rejected before OpenMP 5.1 and accepted from 5.1 onwards.
 
 ! RUN: not %flang -fopenmp -fopenmp-version=50 %s 2>&1 | FileCheck %s --check-prefix="CHECK-50"
-! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -fdebug-unparse %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
 
 module functions
   implicit none
@@ -15,7 +16,7 @@ function func() result(i)
 contains
   function func1() result(i)
     !CHECK-50: INDIRECT clause is not allowed on DECLARE TARGET directive in OpenMP v5.0, try -fopenmp-version=51
-    !CHECK-52: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
+    !CHECK-52: !$OMP DECLARE TARGET ENTER(func1) INDIRECT(.true._4)
     !$omp declare target enter(func1) indirect(.true.)
     character(1) :: i
     i = 'a'
diff --git a/flang/test/Semantics/indirect03.f90 b/flang/test/Semantics/indirect03.f90
new file mode 100644
index 0000000000000..0bab188a9ff51
--- /dev/null
+++ b/flang/test/Semantics/indirect03.f90
@@ -0,0 +1,31 @@
+! This test checks the OpenMP restriction that the INDIRECT clause on a DECLARE
+! TARGET directive is only allowed with DEVICE_TYPE(ANY) (an absent DEVICE_TYPE
+! clause also implies ANY). A host- or device-only procedure cannot be the
+! target of an indirect device invocation.
+
+! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
+
+module functions
+  implicit none
+contains
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_host() result(i)
+    !$omp declare target enter(func_host) device_type(host) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_nohost() result(i)
+    !$omp declare target enter(func_nohost) device_type(nohost) indirect(.true.)
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! DEVICE_TYPE(ANY) with INDIRECT is allowed, so no error is expected here.
+  function func_any() result(i)
+    !$omp declare target enter(func_any) device_type(any) indirect(.true.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
index c9e6764e7d634..bccdc8f6fdd19 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
@@ -45,7 +45,8 @@ def DeclareTargetAttr : OpenMP_Attr<"DeclareTarget", "declaretarget"> {
   let parameters =
       (ins OptionalParameter<"DeclareTargetDeviceTypeAttr">:$device_type,
           OptionalParameter<"DeclareTargetCaptureClauseAttr">:$capture_clause,
-          OptionalParameter<"BoolAttr">:$automap);
+          OptionalParameter<"BoolAttr">:$automap,
+          OptionalParameter<"BoolAttr">:$indirect);
 
   let assemblyFormat = "`<` struct(params) `>`";
 }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
index 35b58375698f6..fbe5ee576228b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
@@ -33,6 +33,12 @@ struct DeviceTypeClauseOps {
   DeclareTargetDeviceType deviceType = DeclareTargetDeviceType::any;
 };
 
+struct IndirectClauseOps {
+  /// Whether the declare target entities may be invoked indirectly (through a
+  /// function pointer) from within a target region.
+  bool indirect = false;
+};
+
 //===----------------------------------------------------------------------===//
 // Extra operation operand structures.
 //===----------------------------------------------------------------------===//
@@ -43,8 +49,8 @@ using HostEvaluatedOperands =
     detail::Clauses<CollapseClauseOps, LoopRelatedClauseOps, NumTeamsClauseOps,
                     NumThreadsClauseOps, ThreadLimitClauseOps>;
 
-// TODO: Add `indirect` clause.
-using DeclareTargetOperands = detail::Clauses<DeviceTypeClauseOps>;
+using DeclareTargetOperands =
+    detail::Clauses<DeviceTypeClauseOps, IndirectClauseOps>;
 
 /// omp.target_enter_data, omp.target_exit_data and omp.target_update take the
 /// same clauses, so we give the structure to be shared by all of them a
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index 51f925b17f47e..10dd4b054623b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -376,7 +376,7 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
       /*methodName=*/"setDeclareTarget",
       (ins "mlir::omp::DeclareTargetDeviceType":$deviceType,
            "mlir::omp::DeclareTargetCaptureClause":$captureClause,
-           "bool":$automap), [{}], [{
+           "bool":$automap, "bool":$indirect), [{}], [{
         $_op->setAttr("omp.declare_target",
                   mlir::omp::DeclareTargetAttr::get(
                       $_op->getContext(),
@@ -384,7 +384,8 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
                           $_op->getContext(), deviceType),
                       mlir::omp::DeclareTargetCaptureClauseAttr::get(
                           $_op->getContext(), captureClause),
-                      mlir::BoolAttr::get($_op->getContext(), automap)));
+                      mlir::BoolAttr::get($_op->getContext(), automap),
+                      mlir::BoolAttr::get($_op->getContext(), indirect)));
       }]>,
       InterfaceMethod<
       /*description=*/[{
@@ -435,6 +436,21 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
             if (auto autoVal = dAttr.getAutomap())
               return autoVal.getValue();
          return false;
+      }]>,
+      InterfaceMethod<
+      /*description=*/[{
+        Return true if the DeclareTarget attribute has the INDIRECT modifier,
+        indicating that the entity may be invoked through a function pointer
+        from within a target region.
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getDeclareTargetIndirect",
+      (ins), [{}], [{
+        if (mlir::Attribute dTar = $_op->getAttr("omp.declare_target"))
+          if (auto dAttr = llvm::dyn_cast_or_null<mlir::omp::DeclareTargetAttr>(dTar))
+            if (auto indVal = dAttr.getIndirect())
+              return indVal.getValue();
+         return false;
       }]>
   ];
 }
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
index 84b0d40949dba..1d066facdecd3 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
+++ b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
@@ -58,11 +58,16 @@ class MarkDeclareTargetPass
           currentDt != omp::DeclareTargetDeviceType::any) {
         current.setDeclareTarget(omp::DeclareTargetDeviceType::any,
                                  current.getDeclareTargetCaptureClause(),
-                                 current.getDeclareTargetAutomap());
+                                 current.getDeclareTargetAutomap(),
+                                 current.getDeclareTargetIndirect());
       }
     } else {
+      // The `indirect` modifier is a property of the specific declare target
+      // declaration; it is not inherited by functions that are reached only
+      // through (direct) calls. Such implicitly captured functions are marked
+      // declare target, but must not be registered as indirect call targets.
       current.setDeclareTarget(parentInfo.devTy, parentInfo.capClause,
-                               parentInfo.automap);
+                               parentInfo.automap, /*indirect=*/false);
     }
 
     markNestedFuncs(parentInfo, symOp, visited);
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..f447d68ef441f 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8790,6 +8790,69 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Register a `declare target ... indirect` function so that the OpenMP runtime
+/// can resolve indirect calls (through a function pointer) to it from within a
+/// target region. This mirrors clang's
+/// `CGOpenMPRuntime::emitDeclareTargetFunction`.
+///
+/// On the host the function itself is registered as an indirect offload entry,
+/// which causes an offloading entry (with the indirect flag) to be emitted at
+/// module finalization. On the target device a new, externally visible global
+/// holding the address of the function is generated (so the runtime can read
+/// the device address while leaving the function's own linkage and visibility
+/// unchanged) and that global is registered instead.
+static void registerIndirectDeclareTargetFunction(
+    FunctionOpInterface funcOp, llvm::OpenMPIRBuilder *ompBuilder,
+    LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::Function *llvmFunc = moduleTranslation.lookupFunction(funcOp.getName());
+  if (!llvmFunc)
+    return;
+
+  // Build the unique offload entry name using the function as the parent name,
+  // e.g. `__omp_offloading_<device>_<file>_<func>_l<line>`.
+  auto loc = funcOp->getLoc()->findInstanceOf<FileLineColLoc>();
+  auto fileInfoCallBack = [&loc]() {
+    std::string filename = "";
+    std::uint64_t lineNo = 0;
+    if (loc) {
+      filename = loc.getFilename().str();
+      lineNo = loc.getLine();
+    }
+    return std::pair<std::string, std::uint64_t>(llvm::StringRef(filename),
+                                                 lineNo);
+  };
+
+  llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+  llvm::TargetRegionEntryInfo entryInfo = ompBuilder->getTargetEntryUniqueInfo(
+      fileInfoCallBack, vfs, funcOp.getName());
+  llvm::SmallString<128> name;
+  ompBuilder->OffloadInfoManager.getTargetRegionEntryFnName(name, entryInfo);
+
+  llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
+  const llvm::DataLayout &dl = llvmModule->getDataLayout();
+  // The entry tracks a pointer to the function, so its size is the store size
+  // of a pointer.
+  int64_t varSize = dl.getTypeStoreSize(
+      llvm::PointerType::getUnqual(llvmModule->getContext()));
+
+  llvm::Constant *addr = llvmFunc;
+  if (ompBuilder->Config.isTargetDevice()) {
+    llvm::PointerType *fnPtrTy = llvm::PointerType::get(
+        llvmModule->getContext(), dl.getProgramAddressSpace());
+    auto *addrGlobal = new llvm::GlobalVariable(
+        *llvmModule, fnPtrTy, /*isConstant=*/true,
+        llvm::GlobalValue::ExternalLinkage, llvmFunc, name, nullptr,
+        llvm::GlobalValue::NotThreadLocal, dl.getDefaultGlobalsAddressSpace());
+    addrGlobal->setVisibility(llvm::GlobalValue::ProtectedVisibility);
+    addr = addrGlobal;
+  }
+
+  ompBuilder->OffloadInfoManager.registerDeviceGlobalVarEntryInfo(
+      name, addr, varSize,
+      llvm::OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect,
+      llvm::GlobalValue::WeakODRLinkage);
+}
+
 static LogicalResult
 convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                          llvm::OpenMPIRBuilder *ompBuilder,
@@ -8804,13 +8867,25 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (FunctionOpInterface funcOp = dyn_cast<FunctionOpInterface>(op)) {
     if (auto offloadMod = dyn_cast<omp::OffloadModuleInterface>(
             op->getParentOfType<ModuleOp>().getOperation())) {
-      if (!offloadMod.getIsTargetDevice())
-        return success();
-
+      bool isTargetDevice = offloadMod.getIsTargetDevice();
       omp::DeclareTargetDeviceType declareType =
           attribute.getDeviceType().getValue();
+      bool isHostFunc = declareType == omp::DeclareTargetDeviceType::host;
+
+      // A `declare target ... indirect(.true.)` function must be registered so
+      // that indirect calls to it from within a target region can be resolved
+      // by the runtime. This applies to both host and device compilation, but
+      // not to host-only functions that are about to be deleted on the device.
+      mlir::BoolAttr indirectAttr = attribute.getIndirect();
+      if (indirectAttr && indirectAttr.getValue() &&
+          !(isTargetDevice && isHostFunc))
+        registerIndirectDeclareTargetFunction(funcOp, ompBuilder,
+                                              moduleTranslation);
+
+      if (!isTargetDevice)
+        return success();
 
-      if (declareType == omp::DeclareTargetDeviceType::host) {
+      if (isHostFunc) {
         llvm::Function *llvmFunc =
             moduleTranslation.lookupFunction(funcOp.getName());
         llvmFunc->dropAllReferences();
diff --git a/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
new file mode 100644
index 0000000000000..3ebdcb66cd017
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
@@ -0,0 +1,39 @@
+// RUN: mlir-opt -omp-mark-declare-target -split-input-file %s | FileCheck %s
+
+// The `omp-mark-declare-target` pass marks functions that are reachable from
+// explicit target code as implicitly declare target. The `indirect` modifier
+// however is a property of the specific declare target declaration and must NOT
+// be propagated to functions that are only reached through (direct) calls.
+
+// A function explicitly declared `indirect` that directly calls another
+// function: the callee is implicitly captured and must be marked declare target
+// with `indirect = false`, not inherit the parent's `indirect = true`.
+module {
+  // CHECK: func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @direct_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @direct_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  func.func @direct_callee() {
+    return
+  }
+}
+
+// -----
+
+// A callee that is itself explicitly declared `indirect` keeps its own value
+// (the pass must not clobber it).
+module {
+  // CHECK: func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @explicitly_indirect_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
new file mode 100644
index 0000000000000..ad86bed2770fa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the device-side lowering of `declare target ... indirect` functions. A
+// new global holding the address of the function is generated with protected
+// visibility so the runtime can access it. A function marked `indirect = false`
+// must not generate such a global.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: @[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = protected constant ptr @indirect_fn
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an indirect global.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
new file mode 100644
index 0000000000000..f3b098018d196
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
@@ -0,0 +1,20 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the host-side lowering of `declare target ... indirect` functions. The
+// runtime needs an offload entry (flag 8 == OMPTargetGlobalVarEntryIndirect)
+// so that indirect calls within a target region can be resolved. A function
+// marked `indirect = false` must not register an offload entry.
+
+// CHECK-DAG: %struct.__tgt_offload_entry = type { i64, i16, i16, i32, ptr, ptr, i64, i64, ptr }
+module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_target_device = false} {
+  // CHECK: @.offloading.entry.[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 8, ptr @indirect_fn, ptr @{{.*}}, i64 8, i64 0, ptr null }
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an offload entry.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}


From llvmlistbot at llvm.org  Thu Jul  9 03:01:51 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 03:01:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f718f.b7ecdeec.2a6187.0445@mx.google.com>


https://github.com/akroviakov commented:

I do not think making `currentProgramOrder` global worth the implicit constructor dependencies.

https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Thu Jul  9 03:01:51 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 03:01:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f718f.d7b890e2.322648.dfb8@mx.google.com>


================
@@ -71,25 +81,44 @@ namespace {
 /// Given this, LayoutInfo  satisifies the following properties:
 ///  1) A LayoutInfo value can be in one of two states - `assigned` or `not
 ///  assigned`.
-///  2) Two LayoutInfo values are equal if they are both assigned or
-///  both not assigned. The concrete value of assigned state does not matter.
+///  2) Two LayoutInfo values are equal if they are both not assigned, or both
+///  assigned with the same layout.
 ///  3) The meet operator works as follows:
-///     - If current state is assigned, return the current state. (already
-///     a unique layout is assigned. don't change it)
-///     - Otherwise, return the other state.
+///     - If only one side is assigned, return that side.
+///     - If both sides are assigned, prefer the layout demanded by the user
+///       that is nearer to the producer in program order (smaller
+///       `programOrder`); on a tie keep lhs.
+///
+/// The `programOrder` field records the program-order index of the consumer op
+/// that demanded the layout (see `currentProgramOrder`). During this backward
+/// analysis a value can be demanded by several users; keeping the nearest one
+/// tends to preserve a consumer's layout as far up the def chain as possible,
+/// minimizing layout conversions. This is a hint, not an optimum.
+/// `programOrder` is never propagated up the chain - each visited op stamps its
+/// own index - so it is deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
   xegpu::DistributeLayoutAttr storage = nullptr;
+  // Program-order index of the consumer op that demanded this layout. Smaller
+  // means nearer to the producer. Unassigned/unknown demands sort last.
+  int64_t programOrder = std::numeric_limits<int64_t>::max();
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
+      : storage(layout), programOrder(programOrder) {}
 
-  // Two lattice values are equal if they have `some` layout. The actual
-  // content of the layout does not matter.
+  // Two lattice values are equal if they are both unassigned, or both assigned
+  // with the same layout. `programOrder` is intentionally excluded: it is not
+  // propagated, so a pure order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
-    return this->isAssigned() == other.isAssigned();
+    if (isAssigned() != other.isAssigned())
+      return false;
+    if (!isAssigned())
----------------
akroviakov wrote:

The comment above does not describe this condition

https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Thu Jul  9 03:01:52 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 03:01:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f7190.f06f4994.1fc857.e6f6@mx.google.com>


https://github.com/akroviakov edited https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Thu Jul  9 03:01:52 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 03:01:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a4f7190.af6ba3ed.36d9db.511f@mx.google.com>


================
@@ -58,6 +60,14 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
+/// Program-order index of the op currently being visited by the backward
+/// analysis. `visitOperation` sets this before dispatching, and the
+/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
+/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
+/// change. A larger index means farther from the producer; the sentinel max
+/// marks demands with no associated op (e.g. exit state).
+static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
----------------
akroviakov wrote:

A 30-line change is less of a concern compared to introducing an implicit dependency of the LayoutInfo constructor on this global mutable state (with an explicit comment about saving a change of 30 lines).

The `currentProgramOrder` is set, tracked, and used by `LayoutInfoPropagation`, so it being a member is expected.

If the goal is to prevent explicit order passing in each visitor's layout creation, a private `LayoutInfoPropagation` helper would allow avoiding a call signature change and just rename call sites:
```
LayoutInfo makeLayoutInfo(DistributeLayoutAttr layout) {
  return LayoutInfo(layout, this->currentProgramOrder);
}
```

https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Thu Jul  9 03:05:13 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Thu, 09 Jul 2026 03:05:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4f7259.510f576d.1b3cb9.b0b7@mx.google.com>


================
@@ -3477,9 +3477,23 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional; an omitted attribute selects the
+    // default `wrapped` behavior. Resolve it to a fully-initialized value here
+    // so the intrinsic selector always inspects an engaged optional and never
+    // reads an empty optional's uninitialized payload (a benign read the
+    // optimizer can hoist out of the guarding `has_value()` check under -O2).
+    // An empty std::optional still reserves storage for its payload but leaves
+    // it uninitialized, and at -O2 the optimizer may if-convert the selector's
+    // `has_value() ? ... *sat ... : true` guard and speculatively load `*sat`
+    // before the guard discards it. This is not UB -- the read touches validly
+    // allocated storage and its result is never used -- but Valgrind still
+    // reports the load of uninitialized bytes.
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
----------------
durga4github wrote:

ok, I see what you mentioned about "leaking this attr on float Ops" too. So, `DefaultValuedAttr` is not the right choice here. I think `DefaultValuedOptionalAttr` will help us prevent the round-trip leak but I see that this needs a different set of Verifier checks now (w/o buying us much).

So, this change LGTM as is. Thanks, Guray for clarifying!

https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Thu Jul  9 03:05:31 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Thu, 09 Jul 2026 03:05:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4f726b.510f576d.1b3cb9.bb9f@mx.google.com>


https://github.com/durga4github approved this pull request.


https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Thu Jul  9 03:07:34 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Thu, 09 Jul 2026 03:07:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a4f72e6.13be6df8.145c.df97@mx.google.com>


Devjiu wrote:

@joker-eph 

> I'm not sure what point you're trying to make actually?

Let me make it concrete with the actual downstream case, since that's where "verify() passes" and "semantically correct" diverge.

Our downstream stashes tensor bounds in the encoding - used later by bufferization to compute allocation sizes for dynamic dims. Before canonicalization:
```mlir
  %r = tensor.insert_slice %src into %dst[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
      : tensor<1x?x?x32xf16, {bounds=[1, 8, 16, 32]}> into tensor<1x1280x32x32xf16>
```
after canonicalization today
```mlir
  %cast = tensor.cast %src : tensor<1x?x?x32xf16, {bounds=[1, 8, 16, 32]}> to tensor<1x?x16x32xf16>
  %r = tensor.insert_slice %cast into %dst[0, 0, 0, 0] [1, %sz1, %c16, 32] [1, 1, 1, 1]
      : tensor<1x?x16x32xf16> into tensor<1x1280x32x32xf16>
```
`%cast`'s result type has no encoding — the bounds are gone. `RankedTensorType::verify` has nothing to say about this: it's a well-formed type either way, an absent attribute isn't invalid IR by any check we have. But downstream, bufferization has no more bounds to compute the allocation size for the dynamic dim, and either falls back to an unbounded/incorrect allocation or fails deep inside a pass that has no idea a canonicalizer upstream silently dropped the information it needed.


>>I don't think "OK to drop when the encoding says it no longer holds" implies "OK to drop unconditionally." Those are different decision procedures with very different odds of throwing away information.

>I don't quite see why actually. I mentioned an example why this seems completely arbitrary to me and not robust.

This example is exactly why: dropping was never conditioned on anything - `{bounds=...}` doesn't implement `VerifiableTensorEncoding`, it never gets asked whether it still holds, it's just discarded because the pattern
re-derives the type from a template that doesn't carry it. The shape change here is dyn -> static, the bounds are still perfectly valid for that shape, and nothing about them became stale, yet they're dropped anyway. 

`insert_slice`/`extract_slice` are also not some rarely-used op pair — they're core to the tiling mechanism/interface that a lot of downstream lowering pipelines are built on. Silently erasing arbitrary encoding metadata in such a central, generic op feels like the wrong default, especially since different downstreams attach very different (and mutually opaque) meaning
to that attribute.

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Thu Jul  9 03:10:20 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 03:10:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support subgroup layout
 assignment for scatter stores (PR #205955)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205955@github.com>
Message-ID: <6a4f738c.7270d2fd.2d4879.fb61@mx.google.com>


https://github.com/akroviakov approved this pull request.

The usage is fine. Note that `getSgLayoutCandidates` is hardcoded to support 2D layouts.

https://github.com/llvm/llvm-project/pull/205955

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.af6ba3ed.36d9db.1e39@mx.google.com>


https://github.com/egebeysel commented:

Thanks for the patch! 

I have some Initial comments/concerns regarding the memory layout chosen, I'll take another look afterwards.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.4a342cbc.ca844.50ae@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
----------------
egebeysel wrote:

do we actually need the +sve flag here?

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.f8b79514.1d52d0.7d1a@mx.google.com>


https://github.com/egebeysel edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.eb24c6ea.27aedc.62d3@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
----------------
egebeysel wrote:

```suggestion
///   * The matrix-multiplication dimensions that are scalable: M and N.
```

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.f06f4994.1fc857.5baa@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
----------------
egebeysel wrote:

```suggestion
/// pack/unpack/mmt4d ops into one function:
```

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fa.510f576d.1b3cb9.be53@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
----------------
egebeysel wrote:

we also have unpack after this I guess :) 

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:12 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fc.481afadf.196d57.665b@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
----------------
egebeysel wrote:

ditto

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:11 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fb.3cce8983.225732.2ffc@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
----------------
egebeysel wrote:

ditto

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:11 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fb.6c8668f2.e4f82.1490@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
----------------
egebeysel wrote:

I think this should also be 8*vscale

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:11 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fb.f732ad81.3a4994.6c94@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
----------------
egebeysel wrote:

ditto

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:11 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fb.492e23e4.157d47.f095@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
----------------
egebeysel wrote:

actually, we recently merged #204007 which allows fixing this issue, but I don't have a problem with taking this as a next step.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:12 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fc.bd93e731.358318.60c2@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
----------------
egebeysel wrote:

While I was suggesting this, I assumed that we would be making both M and N dims scalable, but that's not the case. I think we should actually do this, i.e. use the inner tiles `8*vscale` for both M and N dims, that would be the actual use-case.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:12 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fc.3cce8983.225732.3004@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
----------------
egebeysel wrote:

I think we could implement this here already and get rid of the TODO

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:12 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fc.6c8668f2.e4f82.1495@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
----------------
egebeysel wrote:

you also wouldn't need this if you directly create named contractions

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:12:12 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 03:12:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4f73fc.ac469dc7.3a57b1.daf1@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
----------------
egebeysel wrote:

I think we should also pass `assume_dynamic_dims_match_vec_sizes` here and get rid of the subsequent masks...

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 03:55:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 03:55:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 015162d - [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (#208188)
Message-ID: <6a4f7e0e.f06f4994.1fc857.abd4@mx.google.com>


Author: Sergio Afonso
Date: 2026-07-09T11:55:03+01:00
New Revision: 015162dacdd5f0752cea50fc400d454a15b6afbe

URL: https://github.com/llvm/llvm-project/commit/015162dacdd5f0752cea50fc400d454a15b6afbe
DIFF: https://github.com/llvm/llvm-project/commit/015162dacdd5f0752cea50fc400d454a15b6afbe.diff

LOG: [Flang][MLIR][OpenMP] Fix declare_target globals visibility (#208188)

This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
compiling for an OpenMP offload target. This prevents potential symbol
redefinition issues related to globals that don't actually exist on the
device.
- Local SAVE variables handling for OpenMP offloading programs is fixed
to prevent their associated "internal" linkage from producing broken
device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
`declare_target link(...)`), the associated and unused full-storage
global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
external linkage when compiling for a target device, causing linker
errors if accessed. This mirrors Clang's behavior.

Fixes #195188, fixes #195468.

Assisted-by: Claude Opus 4.8.

Added: 
    mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir

Modified: 
    flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
    flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir

Removed: 
    


################################################################################
diff  --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
index e42e10a82e6c2..05ea0bc08c4ab 100644
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
@@ -65,6 +65,15 @@ static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
   rewrites.insert(value);
 }
 
+/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
 namespace {
 class HostOpFilteringPass
     : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
@@ -78,11 +87,8 @@ class HostOpFilteringPass
 
     op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
       omp::DeclareTargetDeviceType declareType =
-          omp::DeclareTargetDeviceType::host;
-      auto declareTargetOp =
-          dyn_cast<omp::DeclareTargetInterface>(funcOp.getOperation());
-      if (declareTargetOp && declareTargetOp.isDeclareTarget())
-        declareType = declareTargetOp.getDeclareTargetDeviceType();
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
 
       // Only process host function definitions.
       if (funcOp.isExternal() ||
@@ -95,6 +101,15 @@ class HostOpFilteringPass
       }
       return WalkResult::advance();
     });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time issues.
+    op->walk([&](fir::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkName("internal");
+    });
   }
 
 private:

diff  --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
index f3b15bd028bc4..3acdfda876812 100644
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
@@ -523,10 +523,6 @@ module attributes {omp.is_target_device = true} {
   }
 
   func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  fir.global internal @global_scalar constant : i32 {
-    %0 = arith.constant 10 : i32
-    fir.has_value %0 : i32
-  }
   omp.private {type = firstprivate} @privatizer : i32 copy {
   ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
     %0 = fir.load %arg0 : !fir.ref<i32>
@@ -544,4 +540,21 @@ module attributes {omp.is_target_device = true} {
     %1 = arith.addi %arg0, %arg1 : i32
     omp.yield (%1 : i32)
   }
+
+  // CHECK: fir.global internal @global_scalar constant : i32
+  fir.global @global_scalar constant : i32
+
+  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
 }

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..b2d2ef6d8f770 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8843,14 +8843,16 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (LLVM::GlobalOp gOp = dyn_cast<LLVM::GlobalOp>(op)) {
     llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
     if (auto *gVal = llvmModule->getNamedValue(gOp.getSymName())) {
+      auto *gVar = cast<llvm::GlobalVariable>(gVal);
       llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
       bool isDeclaration = gOp.isDeclaration();
       bool isExternallyVisible =
           gOp.getVisibility() != mlir::SymbolTable::Visibility::Private;
       auto loc = op->getLoc()->findInstanceOf<FileLineColLoc>();
       llvm::StringRef mangledName = gOp.getSymName();
-      auto captureClause =
-          convertToCaptureClauseKind(attribute.getCaptureClause().getValue());
+      mlir::omp::DeclareTargetCaptureClause captureClause =
+          attribute.getCaptureClause().getValue();
+      auto captureClauseKind = convertToCaptureClauseKind(captureClause);
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
       // unused for MLIR at the moment, required in Clang for book
@@ -8877,31 +8879,64 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                                                      lineNo);
       };
 
-      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
+      bool isToOrEnter =
+          captureClause == omp::DeclareTargetCaptureClause::to ||
+          captureClause == omp::DeclareTargetCaptureClause::enter;
+      bool isHostOnly = attribute.getDeviceType().getValue() ==
+                        omp::DeclareTargetDeviceType::host;
+
+      // A to/enter declare-target variable needs a device-resident,
+      // name-resolvable copy and a host offloading entry. A local-linkage
+      // global provides neither, so we promote it to external.
+      if (isToOrEnter && !isHostOnly && !requiresUSM &&
+          gVar->hasLocalLinkage()) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        isExternallyVisible = true;
+
+        // Clear the stale dso_local flag so it is referenced like a
+        // module-scope declare target global.
+        if (ompBuilder->Config.isTargetDevice())
+          gVar->setDSOLocal(false);
+      }
 
+      llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
       ompBuilder->registerTargetGlobalVariable(
-          captureClause, deviceClause, isDeclaration, isExternallyVisible,
+          captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
           mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
           /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
           gVal->getType(), gVal);
 
-      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
       if (ompBuilder->Config.isTargetDevice() &&
-          (attribute.getCaptureClause().getValue() ==
-               mlir::omp::DeclareTargetCaptureClause::link ||
+          (captureClause == omp::DeclareTargetCaptureClause::link ||
            requiresUSM)) {
         llvm::Type *ptrTy = gVal->getType();
         // For USM the global type becomes a pointer handle, as opposed to the
         // globals original type.
         if (requiresUSM)
           ptrTy = llvm::PointerType::get(llvmModule->getContext(), 0);
-        ompBuilder->getAddrOfDeclareTargetVar(
-            captureClause, deviceClause, isDeclaration, isExternallyVisible,
+        bool addrGlobalCreated = ompBuilder->getAddrOfDeclareTargetVar(
+            captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
             ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
             mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
             ptrTy, /*GlobalInitializer*/ nullptr,
             /*VariableLinkage*/ nullptr);
+
+        // For indirectly-accessed global pointers, we rely on "internal"
+        // linkage to optimize out the unneeded full-variable storage later,
+        // since we can't prevent the LLVM dialect from generating globals
+        // without also breaking target lowering.
+        if (addrGlobalCreated)
+          gVar->setLinkage(llvm::GlobalValue::InternalLinkage);
+      }
+
+      // Mark 'device_type(host) enter(...)' variables as external in the device
+      // since they're not supposed to have their own copy. This will cause
+      // linker errors if accesses are attempted from the target device.
+      if (ompBuilder->Config.isTargetDevice() && isHostOnly && isToOrEnter) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        gVar->setInitializer(nullptr);
       }
     }
   }

diff  --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
new file mode 100644
index 0000000000000..2cd3fc3c44231
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
@@ -0,0 +1,252 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Tests lowering to LLVM IR of all combinations of OpenMP declare target
+// device_type (any/host/nohost) and capture clause (link/enter) when compiling
+// for device, for both external and internal linkage global variables.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  // --- link any ---
+
+  // CHECK-DAG: @ial = internal global float 0.000000e+00
+  // CHECK-DAG: @ial_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ial() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eal = internal global float 0.000000e+00
+  // CHECK-DAG: @eal_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @eal() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link host ---
+
+  // CHECK-DAG: @ihl = internal global float 0.000000e+00
+  // CHECK-DAG: @ihl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @ihl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehl = internal global float 0.000000e+00
+  // CHECK-DAG: @ehl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @ehl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- link nohost ---
+
+  // CHECK-DAG: @inl = internal global float 0.000000e+00
+  // CHECK-DAG: @inl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global internal @inl() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @enl = internal global float 0.000000e+00
+  // CHECK-DAG: @enl_decl_tgt_ref_ptr = weak global ptr null, align 8
+  llvm.mlir.global external @enl() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter any ---
+
+  // CHECK-DAG: @iae = global float 0.000000e+00
+  llvm.mlir.global internal @iae() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @eae = dso_local global float 0.000000e+00
+  llvm.mlir.global external @eae() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter host ---
+
+  // CHECK-DAG: @ihe = external dso_local global float
+  llvm.mlir.global internal @ihe() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ehe = external dso_local global float
+  llvm.mlir.global external @ehe() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // --- enter nohost ---
+
+  // CHECK-DAG: @ine = global float 0.000000e+00
+  llvm.mlir.global internal @ine() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  // CHECK-DAG: @ene = dso_local global float 0.000000e+00
+  llvm.mlir.global external @ene() {addr_space = 0 : i32, dso_local, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter)>} : f32 {
+    %0 = llvm.mlir.zero : f32
+    llvm.return %0 : f32
+  }
+
+  llvm.func @_QQmain() {
+    %0 = llvm.mlir.addressof @ial : !llvm.ptr
+    %1 = llvm.addrspacecast %0 : !llvm.ptr to !llvm.ptr
+    %2 = llvm.mlir.addressof @ihl : !llvm.ptr
+    %3 = llvm.addrspacecast %2 : !llvm.ptr to !llvm.ptr
+    %4 = llvm.mlir.addressof @inl : !llvm.ptr
+    %5 = llvm.addrspacecast %4 : !llvm.ptr to !llvm.ptr
+    %6 = llvm.mlir.addressof @iae : !llvm.ptr
+    %7 = llvm.addrspacecast %6 : !llvm.ptr to !llvm.ptr
+    %8 = llvm.mlir.addressof @ihe : !llvm.ptr
+    %9 = llvm.addrspacecast %8 : !llvm.ptr to !llvm.ptr
+    %10 = llvm.mlir.addressof @ine : !llvm.ptr
+    %11 = llvm.addrspacecast %10 : !llvm.ptr to !llvm.ptr
+    %12 = llvm.mlir.addressof @eal : !llvm.ptr
+    %13 = llvm.addrspacecast %12 : !llvm.ptr to !llvm.ptr
+    %14 = llvm.mlir.addressof @ehl : !llvm.ptr
+    %15 = llvm.addrspacecast %14 : !llvm.ptr to !llvm.ptr
+    %16 = llvm.mlir.addressof @enl : !llvm.ptr
+    %17 = llvm.addrspacecast %16 : !llvm.ptr to !llvm.ptr
+    %18 = llvm.mlir.addressof @eae : !llvm.ptr
+    %19 = llvm.addrspacecast %18 : !llvm.ptr to !llvm.ptr
+    %20 = llvm.mlir.addressof @ehe : !llvm.ptr
+    %21 = llvm.addrspacecast %20 : !llvm.ptr to !llvm.ptr
+    %22 = llvm.mlir.addressof @ene : !llvm.ptr
+    %23 = llvm.addrspacecast %22 : !llvm.ptr to !llvm.ptr
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ial_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.100000e+00, ptr %[[REF]], align 4
+    %map_ial = omp.map.info var_ptr(%1 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ial"}
+    omp.target kernel_type(generic) map_entries(%map_ial -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @eal_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 1.200000e+00, ptr %[[REF]], align 4
+    %map_eal = omp.map.info var_ptr(%13 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "eal"}
+    omp.target kernel_type(generic) map_entries(%map_eal -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(1.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ihl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.100000e+00, ptr %[[REF]], align 4
+    %map_ihl = omp.map.info var_ptr(%3 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ihl"}
+    omp.target kernel_type(generic) map_entries(%map_ihl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @ehl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 2.200000e+00, ptr %[[REF]], align 4
+    %map_ehl = omp.map.info var_ptr(%15 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ehl"}
+    omp.target kernel_type(generic) map_entries(%map_ehl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(2.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @inl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.100000e+00, ptr %[[REF]], align 4
+    %map_inl = omp.map.info var_ptr(%5 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "inl"}
+    omp.target kernel_type(generic) map_entries(%map_inl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           %[[REF:.*]] = load ptr, ptr @enl_decl_tgt_ref_ptr, align 8
+    // CHECK:           store float 3.200000e+00, ptr %[[REF]], align 4
+    %map_enl = omp.map.info var_ptr(%17 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "enl"}
+    omp.target kernel_type(generic) map_entries(%map_enl -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(3.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 4.100000e+00, ptr @iae, align 4
+    %map_iae = omp.map.info var_ptr(%7 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "iae"}
+    omp.target kernel_type(generic) map_entries(%map_iae -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(4.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 4.200000e+00, ptr @eae, align 4
+    %map_eae = omp.map.info var_ptr(%19 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "eae"}
+    omp.target kernel_type(generic) map_entries(%map_eae -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(4.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 5.100000e+00, ptr @ihe, align 4
+    %map_ihe = omp.map.info var_ptr(%9 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ihe"}
+    omp.target kernel_type(generic) map_entries(%map_ihe -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(5.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 5.200000e+00, ptr @ehe, align 4
+    %map_ehe = omp.map.info var_ptr(%21 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ehe"}
+    omp.target kernel_type(generic) map_entries(%map_ehe -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(5.200000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 6.100000e+00, ptr @ine, align 4
+    %map_ine = omp.map.info var_ptr(%11 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ine"}
+    omp.target kernel_type(generic) map_entries(%map_ine -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(6.100000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    // CHECK-LABEL: define {{.*}} @__omp_offloading_{{.*}}_l{{[0-9]+}}(
+    // CHECK:         omp.target:
+    // CHECK:           store float 6.210000e+00, ptr @ene, align 4
+    %map_ene = omp.map.info var_ptr(%23 : !llvm.ptr, f32) map_clauses(always, from) capture(ByRef) -> !llvm.ptr {name = "ene"}
+    omp.target kernel_type(generic) map_entries(%map_ene -> %arg0 : !llvm.ptr) {
+      %v = llvm.mlir.constant(6.210000e+00 : f32) : f32
+      llvm.store %v, %arg0 : f32, !llvm.ptr
+      omp.terminator
+    }
+
+    llvm.return
+  }
+}

diff  --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
index bc59fdda2cc25..745ab424fb5f8 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-device.mlir
@@ -8,6 +8,7 @@
 // file created by the host and appended as an attribute to the module.
 
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK-DAG: @_QMtest_0Esp = internal global i32
   // CHECK-DAG: @_QMtest_0Esp_decl_tgt_ref_ptr = weak global ptr null, align 8
   llvm.mlir.global external @_QMtest_0Esp() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link)>} : i32 {
     %0 = llvm.mlir.constant(0 : i32) : i32


        

From llvmlistbot at llvm.org  Thu Jul  9 03:55:11 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 09 Jul 2026 03:55:11 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4f7e0f.eb165e5c.1817a5.4495@mx.google.com>


https://github.com/skatrak closed https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Thu Jul  9 03:55:13 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 09 Jul 2026 03:55:13 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (PR #208189)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208189@github.com>
Message-ID: <6a4f7e11.c56fb102.7c9cd.1615@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/208189

From llvmlistbot at llvm.org  Thu Jul  9 03:55:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 03:55:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 50d1864 - [OpenACC] apply par dims to
 reductions in parallel regions (#208258)
Message-ID: <6a4f7e33.bd93e731.358318.c38b@mx.google.com>


Author: Scott Manley
Date: 2026-07-09T05:55:43-05:00
New Revision: 50d186480ee9b0fa3f2500c4b963a47ee78db902

URL: https://github.com/llvm/llvm-project/commit/50d186480ee9b0fa3f2500c4b963a47ee78db902
DIFF: https://github.com/llvm/llvm-project/commit/50d186480ee9b0fa3f2500c4b963a47ee78db902.diff

LOG: [OpenACC] apply par dims to reductions in parallel regions (#208258)

For reductions that come from parallel constructs, explicitly set the
GPU parallel dimensions attribute to blockXDim on the acc.reduction_init
and acc.reduction_combine* ops since they will always be gang private

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
    mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index 398f4d4bbfdf6..d2e8d77c168d1 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -43,7 +43,9 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtils.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsLoop.h"
 #include "mlir/Dialect/OpenACC/Transforms/Passes.h"
 #include "mlir/IR/Block.h"
@@ -181,10 +183,11 @@ class ACCRecipeMaterialization
   void removeRecipe(OpTy op, ModuleOp moduleOp) const;
   template <typename OpTy, typename RecipeOpTy, typename AccOpTy>
   LogicalResult materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
-                            acc::OpenACCSupport &accSupport) const;
+                            acc::OpenACCSupport &accSupport,
+                            acc::ACCToGPUMappingPolicy &policy) const;
   template <typename OpTy>
-  LogicalResult materializeForACCOp(OpTy accOp,
-                                    acc::OpenACCSupport &accSupport) const;
+  LogicalResult materializeForACCOp(OpTy accOp, acc::OpenACCSupport &accSupport,
+                                    acc::ACCToGPUMappingPolicy &policy) const;
 };
 
 void ACCRecipeMaterialization::handleFirstprivateMapping(
@@ -220,9 +223,9 @@ void ACCRecipeMaterialization::removeRecipe(OpTy op, ModuleOp moduleOp) const {
 }
 
 template <typename OpTy, typename RecipeOpTy, typename AccOpTy>
-LogicalResult
-ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
-                                      acc::OpenACCSupport &accSupport) const {
+LogicalResult ACCRecipeMaterialization::materialize(
+    OpTy op, RecipeOpTy recipe, AccOpTy accOp, acc::OpenACCSupport &accSupport,
+    acc::ACCToGPUMappingPolicy &policy) const {
   Region &region = accOp.getRegion();
   Value origPtr = op.getVar();
   Value accPtr = op.getAccVar();
@@ -380,11 +383,25 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
     cloneRegionIntoAccRegion(&combinerRegion, &combineRegionOp.getRegion(),
                              /*hasResult=*/false);
 
-    auto setSeqParDimsForRecipeLoops = [](Region *r) {
-      r->walk([](LoopLikeOpInterface loopLike) {
-        loopLike->setAttr(
-            acc::GPUParallelDimsAttr::name,
-            acc::GPUParallelDimsAttr::seq(loopLike->getContext()));
+    auto ctx = b.getContext();
+
+    // For reductions that come from parallel constructs, explicitly set the
+    // GPU parallel dimensions attribute to blockXDim since they will always be
+    // gang private. GPU parallel dimensions cannot be determined for acc.loop
+    // at this point.
+    if constexpr (std::is_same_v<AccOpTy, acc::ParallelOp>) {
+      auto parDimsAttr = acc::GPUParallelDimsAttr::get(
+          ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+      acc::setParDimsAttr(reductionOp, parDimsAttr);
+      acc::setParDimsAttr(combineRegionOp, parDimsAttr);
+    }
+
+    // Set sequential parallel dimensions attribute for loops in the recipe.
+    auto seqParDimsAttr =
+        acc::GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
+    auto setSeqParDimsForRecipeLoops = [&](Region *r) {
+      r->walk([&](LoopLikeOpInterface loopLike) {
+        acc::setParDimsAttr(loopLike, seqParDimsAttr);
       });
     };
     setSeqParDimsForRecipeLoops(&reductionOp.getRegion());
@@ -406,7 +423,8 @@ ACCRecipeMaterialization::materialize(OpTy op, RecipeOpTy recipe, AccOpTy accOp,
 
 template <typename OpTy>
 LogicalResult ACCRecipeMaterialization::materializeForACCOp(
-    OpTy accOp, acc::OpenACCSupport &accSupport) const {
+    OpTy accOp, acc::OpenACCSupport &accSupport,
+    acc::ACCToGPUMappingPolicy &policy) const {
   assert(isa<ACC_COMPUTE_CONSTRUCT_AND_LOOP_OPS>(accOp));
 
   if (!accOp.getFirstprivateOperands().empty()) {
@@ -422,7 +440,8 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << firstprivateOp << "\n"
                               << symbolRef << "\n");
       handleFirstprivateMapping(firstprivateOp);
-      if (failed(materialize(firstprivateOp, recipeOp, accOp, accSupport)))
+      if (failed(
+              materialize(firstprivateOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -439,7 +458,7 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       auto recipeOp = cast<acc::PrivateRecipeOp>(decl);
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << privateOp << "\n"
                               << symbolRef << "\n");
-      if (failed(materialize(privateOp, recipeOp, accOp, accSupport)))
+      if (failed(materialize(privateOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -456,7 +475,7 @@ LogicalResult ACCRecipeMaterialization::materializeForACCOp(
       auto recipeOp = cast<acc::ReductionRecipeOp>(decl);
       LLVM_DEBUG(llvm::dbgs() << "materializing: " << reductionOp << "\n"
                               << symbolRef << "\n");
-      if (failed(materialize(reductionOp, recipeOp, accOp, accSupport)))
+      if (failed(materialize(reductionOp, recipeOp, accOp, accSupport, policy)))
         return failure();
     }
   }
@@ -467,6 +486,8 @@ void ACCRecipeMaterialization::runOnOperation() {
   ModuleOp moduleOp = getOperation();
   acc::OpenACCSupport &accSupport = getAnalysis<acc::OpenACCSupport>();
 
+  acc::DefaultACCToGPUMappingPolicy policy;
+
   // Materialize all recipes for all compute constructs and loop constructs.
   bool anyFailed = false;
   moduleOp.walk([&](Operation *op) {
@@ -474,7 +495,7 @@ void ACCRecipeMaterialization::runOnOperation() {
       return;
     TypeSwitch<Operation *>(op).Case<ACC_COMPUTE_CONSTRUCT_AND_LOOP_OPS>(
         [&](auto constructOp) {
-          if (failed(materializeForACCOp(constructOp, accSupport)))
+          if (failed(materializeForACCOp(constructOp, accSupport, policy)))
             anyFailed = true;
         });
   });

diff  --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
index 083a6839b7d0a..6f2bafb2b6165 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
@@ -28,7 +28,7 @@ acc.reduction.recipe @reduction_add_memref_f64 : memref<f64> reduction_operator
 // CHECK-NEXT:  [[ALLOCA:%.*]] = memref.alloca() : memref<f64>
 // CHECK-NEXT:  memref.store [[ZERO]], [[ALLOCA]][]
 // CHECK-NEXT:  acc.yield {{.*}}
-// CHECK:       } {{.*}}acc.var_name = #acc.var_name<"tmp">
+// CHECK:       } {{.*}}acc.par_dims = #acc<par_dims[block_x]>, acc.var_name = #acc.var_name<"tmp">
 // CHECK:       memref.load [[PRIVATE]][]
 // CHECK:       memref.store {{.*}}, [[PRIVATE]][]
 // CHECK:       acc.reduction_combine_region [[PRIVATE]] into [[REDUCVAR:%.*]] : memref<f64> {
@@ -36,7 +36,7 @@ acc.reduction.recipe @reduction_add_memref_f64 : memref<f64> reduction_operator
 // CHECK-NEXT:  [[LOADPRIV:%.*]] = memref.load [[PRIVATE]][]
 // CHECK-NEXT:  [[COMBINE:%.*]] = arith.addf [[LOADVAR]], [[LOADPRIV]]
 // CHECK-NEXT:  memref.store [[COMBINE]], [[REDUCVAR]][]
-// CHECK-NEXT:  }
+// CHECK-NEXT:  } {acc.par_dims = #acc<par_dims[block_x]>}
 // CHECK-NEXT:  memref.dealloc [[PRIVATE]] : memref<f64>
 // CHECK:       acc.yield
 


        

From llvmlistbot at llvm.org  Thu Jul  9 03:55:51 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Thu, 09 Jul 2026 03:55:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] apply par dims to reductions in
 parallel regions (PR #208258)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208258@github.com>
Message-ID: <6a4f7e37.f8b79514.1d52d0.1def@mx.google.com>


https://github.com/rscottmanley closed https://github.com/llvm/llvm-project/pull/208258

From llvmlistbot at llvm.org  Thu Jul  9 03:55:57 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 09 Jul 2026 03:55:57 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (PR #208189)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208189@github.com>
Message-ID: <6a4f7e3d.58bdd12c.19c81.7d21@mx.google.com>


https://github.com/skatrak updated https://github.com/llvm/llvm-project/pull/208189

>From 47357e3a683326d76a7abb8c00ede0c8ac1a6aa5 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Thu, 2 Jul 2026 15:23:20 +0100
Subject: [PATCH] [Flang][MLIR][OpenMP] Move host op filtering to the omp
 dialect

The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.

By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.

As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
operations pointing to deleted host functions, while `fir.dispatch` ops
pointing to the same functions would remain. The `omp-function-filtering`
pass is updated to prevent `fir.dt_entry` from being deleted and let
them point to an undefined symbol. This lets FIR lowering to LLVM work
well enough to get to the host op filtering stage, where any resulting
operations for this host dynamic dispatch are cleaned up.
---
 .../include/flang/Optimizer/OpenMP/Passes.td  |  22 -
 flang/lib/Optimizer/OpenMP/CMakeLists.txt     |   1 -
 .../Optimizer/OpenMP/FunctionFiltering.cpp    |  14 +-
 .../lib/Optimizer/OpenMP/HostOpFiltering.cpp  | 422 -------------
 flang/lib/Optimizer/Passes/Pipelines.cpp      |  15 +-
 flang/test/Fir/basic-program.fir              |   2 +-
 .../Lower/OpenMP/function-filtering-4.f90     |  63 ++
 flang/test/Lower/OpenMP/host-eval.f90         |  50 +-
 .../OpenMP/function-filtering-host-ops.mlir   | 560 ------------------
 .../mlir/Dialect/OpenMP/Transforms/Passes.td  |  24 +
 .../Dialect/OpenMP/Transforms/CMakeLists.txt  |   1 +
 .../OpenMP/Transforms/HostOpFiltering.cpp     | 292 +++++++++
 .../Dialect/OpenMP/host-op-filtering.mlir     | 500 ++++++++++++++++
 13 files changed, 920 insertions(+), 1046 deletions(-)
 delete mode 100644 flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
 create mode 100644 flang/test/Lower/OpenMP/function-filtering-4.f90
 delete mode 100644 flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
 create mode 100644 mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
 create mode 100644 mlir/test/Dialect/OpenMP/host-op-filtering.mlir

diff --git a/flang/include/flang/Optimizer/OpenMP/Passes.td b/flang/include/flang/Optimizer/OpenMP/Passes.td
index b5d9372e356f1..9ec159e1ba1e0 100644
--- a/flang/include/flang/Optimizer/OpenMP/Passes.td
+++ b/flang/include/flang/Optimizer/OpenMP/Passes.td
@@ -57,28 +57,6 @@ def FunctionFilteringPass : Pass<"omp-function-filtering"> {
   ];
 }
 
-def HostOpFilteringPass : Pass<"omp-host-op-filtering"> {
-  let summary =
-      "Removes host-only operations from host functions retained by the "
-      "function filtering pass when compiling for a target device.";
-  let description = [{
-    Removes operations from host functions that are not placed inside of an
-    `omp.target` region when compiling for the target device.
-    
-    It must run after any passes that might introduce new host operations,
-    especially if they belong to the 'omp' dialect, like `LowerWorkdistribute`.
-    This ensures all potentially problematic host operations are accounted for
-    prior to LLVM IR translation.
-
-    This pass works at the FIR level of abstraction.
-  }];
-  let dependentDialects = [
-    "mlir::arith::ArithDialect",
-    "mlir::func::FuncDialect",
-    "fir::FIROpsDialect"
-  ];
-}
-
 def DoConcurrentConversionPass : Pass<"omp-do-concurrent-conversion", "mlir::ModuleOp"> {
   let summary = "Map `DO CONCURRENT` loops to OpenMP worksharing loops.";
 
diff --git a/flang/lib/Optimizer/OpenMP/CMakeLists.txt b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
index f25e95f041d20..db29e93b71dad 100644
--- a/flang/lib/Optimizer/OpenMP/CMakeLists.txt
+++ b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
@@ -5,7 +5,6 @@ add_flang_library(FlangOpenMPTransforms
   DoConcurrentConversion.cpp
   FunctionFiltering.cpp
   GenericLoopConversion.cpp
-  HostOpFiltering.cpp
   MapsForPrivatizedSymbols.cpp
   MapInfoFinalization.cpp
   DeleteUnreachableTargets.cpp
diff --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index e8bcc0c7fa062..ee61dd8881fd0 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -48,8 +48,7 @@ checkDeviceImplementationStatus(omp::OffloadModuleInterface offloadModule) {
     if (!redOp.getByrefElementType())
       return WalkResult::advance();
 
-    auto seqTy =
-        mlir::dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
+    auto seqTy = dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
 
     bool isByRefReductionSupported =
         !seqTy || !fir::sequenceWithNonConstantShape(seqTy);
@@ -104,7 +103,7 @@ class FunctionFilteringPass
         SymbolTable::UseRange funcUses = *funcOp.getSymbolUses(op);
         for (SymbolTable::SymbolUse use : funcUses) {
           Operation *callOp = use.getUser();
-          if (auto internalFunc = mlir::dyn_cast<func::FuncOp>(callOp)) {
+          if (auto internalFunc = dyn_cast<func::FuncOp>(callOp)) {
             // Do not delete internal procedures holding the symbol of their
             // Fortran host procedure as attribute.
             internalFunc->removeAttr(fir::getHostSymbolAttrName());
@@ -114,6 +113,15 @@ class FunctionFilteringPass
             internalFunc.setVisibility(mlir::SymbolTable::Visibility::Public);
             continue;
           }
+          // Prevent dispatch table entries pointing to deleted functions
+          // from being removed. This prevents the lowering of any
+          // corresponding fir.dispatch ops from triggering errors. These
+          // fir.dt_entry ops will point to an undefined symbol as a result,
+          // which currently doesn't cause an issue, as fir.dispatch-related ops
+          // are later removed by the host op filtering pass.
+          if (isa<fir::DTEntryOp>(callOp))
+            continue;
+
           // If the callOp has users then replace them with Undef values.
           if (!callOp->use_empty()) {
             SmallVector<Value> undefResults;
diff --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
deleted file mode 100644
index 05ea0bc08c4ab..0000000000000
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ /dev/null
@@ -1,422 +0,0 @@
-//===- HostOpFiltering.cpp -----------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements transforms to filter out host-only operations from
-// any remaining host functions when compiling for a target device.
-//
-//===----------------------------------------------------------------------===//
-
-#include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Dialect/FIROps.h"
-#include "flang/Optimizer/OpenMP/Passes.h"
-
-#include "mlir/Dialect/Arith/IR/Arith.h"
-#include "mlir/Dialect/Func/IR/FuncOps.h"
-#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
-#include "llvm/ADT/SetVector.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/ADT/TypeSwitch.h"
-
-namespace flangomp {
-#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
-#include "flang/Optimizer/OpenMP/Passes.h.inc"
-} // namespace flangomp
-
-using namespace mlir;
-
-/// Add an operation to one of the output sets to be later rewritten.
-template <typename OpTy>
-static void collectRewrite(OpTy op, llvm::SetVector<OpTy> &rewrites) {
-  rewrites.insert(op);
-}
-
-/// Add an \c omp.map.info operation and all its members recursively to the
-/// output set to be later rewritten.
-///
-/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
-/// are added to the output sets before operations based on them.
-template <>
-void collectRewrite(omp::MapInfoOp mapOp,
-                    llvm::SetVector<omp::MapInfoOp> &rewrites) {
-  for (Value member : mapOp.getMembers())
-    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
-
-  rewrites.insert(mapOp);
-}
-
-/// Add the given value to a sorted set if it should be replaced by a
-/// placeholder when used as an operand that must remain for the device.
-///
-/// Values that are block arguments of \c func.func operations are skipped,
-/// since they will still be available after all rewrites are completed.
-static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
-  if ((isa<BlockArgument>(value) &&
-       isa<func::FuncOp>(
-           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
-      rewrites.contains(value))
-    return;
-
-  rewrites.insert(value);
-}
-
-/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
-static std::optional<omp::DeclareTargetDeviceType>
-getDeclareTargetDevice(Operation &op) {
-  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
-  if (declareTargetOp && declareTargetOp.isDeclareTarget())
-    return declareTargetOp.getDeclareTargetDeviceType();
-  return std::nullopt;
-}
-
-namespace {
-class HostOpFilteringPass
-    : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
-public:
-  HostOpFilteringPass() = default;
-
-  void runOnOperation() override {
-    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
-    if (!op || !op.getIsTargetDevice())
-      return;
-
-    op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
-      omp::DeclareTargetDeviceType declareType =
-          getDeclareTargetDevice(*funcOp.getOperation())
-              .value_or(omp::DeclareTargetDeviceType::host);
-
-      // Only process host function definitions.
-      if (funcOp.isExternal() ||
-          declareType != omp::DeclareTargetDeviceType::host)
-        return WalkResult::advance();
-
-      if (failed(rewriteHostFunction(funcOp))) {
-        funcOp.emitOpError() << "could not filter host-only operations";
-        return WalkResult::interrupt();
-      }
-      return WalkResult::advance();
-    });
-
-    // Make non-declare target globals internal for the device. They cannot be
-    // deleted, because they are needed in order to properly lower map clauses.
-    // However, no uses will remain in the device module, so we make them
-    // internal to prevent link time issues.
-    op->walk([&](fir::GlobalOp globalOp) {
-      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
-        globalOp.setLinkName("internal");
-    });
-  }
-
-private:
-  /// Rewrite the given host device function containing \c omp.target
-  /// operations, to remove host-only operations that are not used by device
-  /// codegen.
-  ///
-  /// It is based on the expected form of the MLIR module as produced by Flang
-  /// lowering, after HLFIR to FIR lowering, and it performs the following
-  /// mutations:
-  ///   - Replace all values returned by the function with \c fir.undefined.
-  ///   - \c omp.target operations are moved to the end of the function. If they
-  ///     are nested inside of any other operations, they are hoisted out of
-  ///     them.
-  ///   - \c depend, \c device and \c if clauses are removed from these target
-  ///     functions. Values used to initialize other clauses are replaced by
-  ///     placeholders as follows:
-  ///     - Values defined by block arguments are replaced by placeholders only
-  ///       if they are not attached to the parent \c func.func operation. In
-  ///       that case, they are passed unmodified.
-  ///     - \c arith.constant and \c fir.address_of ops are maintained.
-  ///     - Values of type \c fir.boxchar are replaced with a combination of
-  ///       \c fir.alloca for a single bit and a \c fir.emboxchar.
-  ///     - Other values are replaced by a combination of an \c fir.alloca for a
-  ///       single bit and an \c fir.convert to the original type of the value.
-  ///       This can be done because the code eventually generated for these
-  ///       operations will be discarded, as they aren't runnable by the target
-  ///       device.
-  ///   - \c omp.map.info operations associated to these target regions are
-  ///     preserved. These are moved above all \c omp.target and sorted to
-  ///     satisfy dependencies among them.
-  ///   - \c bounds arguments are removed from \c omp.map.info operations.
-  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
-  ///     handled as follows:
-  ///     - \c var_ptr_ptr is expected to be defined by a \c fir.box_offset
-  ///       operation which is preserved. Otherwise, the pass will fail.
-  ///     - \c var_ptr can be defined by a \c fir.declare which is also
-  ///       preserved. Its \c memref argument is replaced by a placeholder or
-  ///       maintained, similarly to non-map clauses of target operations
-  ///       described above. If it has \c shape or \c typeparams arguments, they
-  ///       are replaced by applicable constants. \c dummy_scope arguments
-  ///       are discarded.
-  ///   - Every other operation not located inside of an \c omp.target is
-  ///     removed.
-  LogicalResult rewriteHostFunction(func::FuncOp funcOp) {
-    Region &region = funcOp.getRegion();
-
-    // Collect target operations inside of the function.
-    llvm::SmallVector<omp::TargetOp> targetOps;
-    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
-      // Skip the inside of omp.target regions, since these contain device code.
-      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
-        targetOps.push_back(targetOp);
-        return WalkResult::skip();
-      }
-
-      // Replace omp.target_data entry block argument uses with the value used
-      // to initialize the associated omp.map.info operation. This way,
-      // references are still valid once the omp.target operation has been
-      // extracted out of the omp.target_data region.
-      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
-        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
-        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
-            .getBlockArgsPairs(argPairs);
-        for (auto [operand, blockArg] : argPairs) {
-          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
-          Value varPtr = mapInfo.getVarPtr();
-
-          // If the var_ptr operand of the omp.map.info op defining this entry
-          // block argument is a fir.declare, the uses of all users of that
-          // entry block argument that are themselves fir.declare are replaced
-          // by the value produced by the outer one.
-          //
-          // This prevents this pass from producing chains of fir.declare of the
-          // type:
-          // %0 = ...
-          // %1 = fir.declare %0
-          // %2 = fir.declare %1...
-          // %3 = omp.map.info var_ptr(%2 ...
-          if (auto outerDeclare = varPtr.getDefiningOp<fir::DeclareOp>())
-            for (Operation *user : blockArg.getUsers())
-              if (isa<fir::DeclareOp>(user))
-                user->replaceAllUsesWith(outerDeclare);
-
-          // All remaining uses of the entry block argument are replaced with
-          // the var_ptr initialization value.
-          blockArg.replaceAllUsesWith(varPtr);
-        }
-      }
-      return WalkResult::advance();
-    });
-
-    // Make a temporary clone of the parent operation with an empty region,
-    // and update all references to entry block arguments to those of the new
-    // region. Users will later either be moved to the new region or deleted
-    // when the original region is replaced by the new.
-    OpBuilder builder(&getContext());
-    builder.setInsertionPointAfter(funcOp);
-    Operation *newOp = builder.cloneWithoutRegions(funcOp);
-    Block &block = newOp->getRegion(0).emplaceBlock();
-
-    llvm::SmallVector<Location> locs;
-    locs.reserve(region.getNumArguments());
-    llvm::transform(region.getArguments(), std::back_inserter(locs),
-                    [](const BlockArgument &arg) { return arg.getLoc(); });
-    block.addArguments(region.getArgumentTypes(), locs);
-
-    for (auto [oldArg, newArg] :
-         llvm::zip_equal(region.getArguments(), block.getArguments()))
-      oldArg.replaceAllUsesWith(newArg);
-
-    // Collect omp.map.info ops while satisfying interdependencies and remove
-    // operands that aren't used by target device codegen.
-    //
-    // This logic must be updated whenever operands to omp.target change.
-    llvm::SetVector<Value> rewriteValues;
-    llvm::SetVector<omp::MapInfoOp> mapInfos;
-    for (omp::TargetOp targetOp : targetOps) {
-      assert(targetOp.getHostEvalVars().empty() &&
-             "unexpected host_eval in target device module");
-
-      // Variables unused by the device.
-      targetOp.getDependVarsMutable().clear();
-      targetOp.setDependKindsAttr(nullptr);
-      targetOp.getDependIteratedMutable().clear();
-      targetOp.setDependIteratedKindsAttr(nullptr);
-      targetOp.getDeviceMutable().clear();
-      targetOp.getDynGroupprivateSizeMutable().clear();
-      targetOp.getIfExprMutable().clear();
-
-      // TODO: Clear some of these operands rather than rewriting them,
-      // depending on whether they are needed by device codegen once support for
-      // them is fully implemented.
-      for (Value allocVar : targetOp.getAllocateVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value allocVar : targetOp.getAllocatorVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value inReduction : targetOp.getInReductionVars())
-        collectRewrite(inReduction, rewriteValues);
-      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
-        collectRewrite(isDevPtr, rewriteValues);
-      for (Value mapVar : targetOp.getHasDeviceAddrVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value mapVar : targetOp.getMapVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value privateVar : targetOp.getPrivateVars())
-        collectRewrite(privateVar, rewriteValues);
-      for (Value threadLimit : targetOp.getThreadLimitVars())
-        collectRewrite(threadLimit, rewriteValues);
-    }
-
-    // Move omp.map.info ops to the new block and collect dependencies.
-    llvm::SetVector<fir::DeclareOp> declareOps;
-    llvm::SetVector<fir::BoxOffsetOp> boxOffsets;
-    for (omp::MapInfoOp mapOp : mapInfos) {
-      if (auto declareOp = dyn_cast_if_present<fir::DeclareOp>(
-              mapOp.getVarPtr().getDefiningOp()))
-        collectRewrite(declareOp, declareOps);
-      else
-        collectRewrite(mapOp.getVarPtr(), rewriteValues);
-
-      if (Value varPtrPtr = mapOp.getVarPtrPtr()) {
-        if (auto boxOffset = llvm::dyn_cast_if_present<fir::BoxOffsetOp>(
-                varPtrPtr.getDefiningOp()))
-          collectRewrite(boxOffset, boxOffsets);
-        else
-          return mapOp->emitOpError() << "var_ptr_ptr rewrite only supported "
-                                         "if defined by fir.box_offset";
-      }
-
-      // Bounds are not used during target device codegen.
-      mapOp.getBoundsMutable().clear();
-      mapOp->moveBefore(&block, block.end());
-    }
-
-    // Create a temporary marker to simplify the op moving process below.
-    builder.setInsertionPointToStart(&block);
-    auto marker = fir::UndefOp::create(builder, builder.getUnknownLoc(),
-                                       builder.getNoneType());
-    builder.setInsertionPoint(marker);
-
-    // Handle dependencies of fir.declare ops.
-    for (fir::DeclareOp declareOp : declareOps) {
-      collectRewrite(declareOp.getMemref(), rewriteValues);
-
-      if (declareOp.getStorage())
-        collectRewrite(declareOp.getStorage(), rewriteValues);
-
-      // Shape and typeparams aren't needed for target device codegen, but
-      // removing them would break verifiers.
-      Value zero;
-      if (declareOp.getShape() || !declareOp.getTypeparams().empty())
-        zero = arith::ConstantOp::create(builder, declareOp.getLoc(),
-                                         builder.getI64IntegerAttr(0));
-
-      if (auto shape = declareOp.getShape()) {
-        // The pre-cg rewrite pass requires the shape to be defined by one of
-        // fir.shape, fir.shapeshift or fir.shift, so we need to make sure it's
-        // still defined by one of these after this pass.
-        Operation *shapeOp = shape.getDefiningOp();
-        llvm::SmallVector<Value> extents(shapeOp->getNumOperands(), zero);
-        Value newShape =
-            llvm::TypeSwitch<Operation *, Value>(shapeOp)
-                .Case([&](fir::ShapeOp op) {
-                  return fir::ShapeOp::create(builder, op.getLoc(), extents);
-                })
-                .Case([&](fir::ShapeShiftOp op) {
-                  auto type = fir::ShapeShiftType::get(op.getContext(),
-                                                       extents.size() / 2);
-                  return fir::ShapeShiftOp::create(builder, op.getLoc(), type,
-                                                   extents);
-                })
-                .Case([&](fir::ShiftOp op) {
-                  auto type =
-                      fir::ShiftType::get(op.getContext(), extents.size());
-                  return fir::ShiftOp::create(builder, op.getLoc(), type,
-                                              extents);
-                })
-                .Default([](Operation *op) {
-                  op->emitOpError()
-                      << "fir.declare shape expected to be one of: "
-                         "fir.shape, fir.shapeshift or fir.shift";
-                  return nullptr;
-                });
-
-        if (!newShape)
-          return failure();
-
-        declareOp.getShapeMutable().assign(newShape);
-      }
-
-      for (OpOperand &typeParam : declareOp.getTypeparamsMutable())
-        typeParam.assign(zero);
-
-      declareOp.getDummyScopeMutable().clear();
-    }
-
-    // We don't actually need the proper initialization, but rather just
-    // maintain the basic form of these operands. Generally, we create 1-bit
-    // placeholder allocas that we "typecast" to the expected type and replace
-    // all uses. Using fir.undefined here instead is not possible because these
-    // variables cannot be constants, as that would trigger different codegen
-    // for target regions.
-    for (Value value : rewriteValues) {
-      Location loc = value.getLoc();
-      Value rewriteValue;
-      if (isa_and_present<arith::ConstantOp, fir::AddrOfOp>(
-              value.getDefiningOp())) {
-        // If it's defined by fir.address_of, then we need to keep that op as
-        // well because it might be pointing to a 'declare target' global.
-        // Constants can also trigger different codegen paths, so we keep them
-        // as well.
-        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
-      } else if (auto boxCharType =
-                     dyn_cast<fir::BoxCharType>(value.getType())) {
-        // !fir.boxchar types cannot be directly obtained by converting a
-        // !fir.ref<i1>, as they aren't reference types. Since they can appear
-        // representing some `target firstprivate` clauses, we need to create
-        // a special case here based on creating a placeholder fir.emboxchar op.
-        MLIRContext *ctx = &getContext();
-        fir::KindTy kind = boxCharType.getKind();
-        auto placeholder = fir::AllocaOp::create(
-            builder, loc, fir::CharacterType::getSingleton(ctx, kind));
-        auto one = arith::ConstantOp::create(builder, loc, builder.getI32Type(),
-                                             builder.getI32IntegerAttr(1));
-        rewriteValue = fir::EmboxCharOp::create(builder, loc, boxCharType,
-                                                placeholder, one);
-      } else {
-        Value placeholder =
-            fir::AllocaOp::create(builder, loc, builder.getI1Type());
-        rewriteValue =
-            fir::ConvertOp::create(builder, loc, value.getType(), placeholder);
-      }
-      value.replaceAllUsesWith(rewriteValue);
-    }
-
-    // Move omp.map.info dependencies.
-    for (fir::DeclareOp declareOp : declareOps)
-      declareOp->moveBefore(marker);
-
-    // The box_ref argument of fir.box_offset is expected to be the same value
-    // that was passed as var_ptr to the corresponding omp.map.info, so we don't
-    // need to handle its defining op here.
-    for (fir::BoxOffsetOp boxOffset : boxOffsets)
-      boxOffset->moveBefore(marker);
-
-    marker->erase();
-
-    // Move target operations to the end of the new block.
-    for (omp::TargetOp targetOp : targetOps)
-      targetOp->moveBefore(&block, block.end());
-
-    // Add terminator to the new block.
-    builder.setInsertionPointToEnd(&block);
-    llvm::SmallVector<Value> returnValues;
-    returnValues.reserve(funcOp.getNumResults());
-    for (auto type : funcOp.getResultTypes())
-      returnValues.push_back(
-          fir::UndefOp::create(builder, funcOp.getLoc(), type));
-
-    func::ReturnOp::create(builder, funcOp.getLoc(), returnValues);
-
-    // Replace old region (now missing ops) with the new one and remove the
-    // temporary operation clone.
-    region.takeBody(newOp->getRegion(0));
-    newOp->erase();
-    return success();
-  }
-};
-} // namespace
diff --git a/flang/lib/Optimizer/Passes/Pipelines.cpp b/flang/lib/Optimizer/Passes/Pipelines.cpp
index a8e598ab95ac8..5fa70096e0e41 100644
--- a/flang/lib/Optimizer/Passes/Pipelines.cpp
+++ b/flang/lib/Optimizer/Passes/Pipelines.cpp
@@ -352,7 +352,6 @@ void createHLFIRToFIRPassPipeline(mlir::PassManager &pm,
   if (enableOpenMP != EnableOpenMP::None) {
     pm.addPass(flangomp::createLowerWorkshare());
     pm.addPass(flangomp::createLowerWorkdistribute());
-    pm.addPass(flangomp::createHostOpFilteringPass());
   }
   if (enableOpenMP == EnableOpenMP::Simd)
     pm.addPass(flangomp::createSimdOnlyPass());
@@ -462,11 +461,17 @@ void createDefaultFIRCodeGenPassPipeline(mlir::PassManager &pm,
 
   fir::addFIRToLLVMPass(pm, config);
 
-  // Convert applicable OpenMP stack allocations to shared memory allocations
-  // for GPU targets. This pass must run after any alloca-generating passes to
-  // ensure all are adequately accounted for.
-  if (config.EnableOpenMP && !config.EnableOpenMPSimd)
+  if (config.EnableOpenMP && !config.EnableOpenMPSimd) {
+    // Remove all non target-related operations from host functions still
+    // remaining at this point, if compiling for an OpenMP target device. This
+    // is required before translating 'omp' dialect operations to LLVM IR.
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
+
+    // Convert applicable OpenMP stack allocations to shared memory allocations
+    // for GPU targets. This pass must run after any alloca-generating passes to
+    // ensure all are adequately accounted for.
     pm.addPass(mlir::omp::createStackToSharedPass());
+  }
 }
 
 /// Create a pass pipeline for lowering from MLIR to LLVM IR
diff --git a/flang/test/Fir/basic-program.fir b/flang/test/Fir/basic-program.fir
index cbf45b216a911..471e5a89046eb 100644
--- a/flang/test/Fir/basic-program.fir
+++ b/flang/test/Fir/basic-program.fir
@@ -85,7 +85,6 @@ func.func @_QQmain() {
 // PASSES-NEXT:   ConvertHLFIRtoFIR
 // PASSES-NEXT:   LowerWorkshare
 // PASSES-NEXT:   LowerWorkdistribute
-// PASSES-NEXT:   HostOpFilteringPass
 // PASSES-NEXT:   CSE
 // PASSES-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 // PASSES-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
@@ -185,6 +184,7 @@ func.func @_QQmain() {
 // PASSES-NEXT:  LowerNontemporalPass
 // PASSES-NEXT: FIRToLLVMLowering
 // PASSES-NEXT: ReconcileUnrealizedCasts
+// PASSES-NEXT:  HostOpFilteringPass
 // PASSES-NEXT: 'llvm.func' Pipeline
 // PASSES-NEXT:  StackToSharedPass
 // PASSES-NEXT: PrepareForOMPOffloadPrivatizationPass
diff --git a/flang/test/Lower/OpenMP/function-filtering-4.f90 b/flang/test/Lower/OpenMP/function-filtering-4.f90
new file mode 100644
index 0000000000000..de8a26c789dc5
--- /dev/null
+++ b/flang/test/Lower/OpenMP/function-filtering-4.f90
@@ -0,0 +1,63 @@
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-HOST,LLVM-ALL %s
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-DEVICE,LLVM-ALL %s %}
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+
+! Check that the correct LLVM IR operations are kept for the host and device
+! after running the whole set of translation and transformation passes from
+! Fortran for a type-bound procedure.
+
+module mymodule
+  implicit none
+
+  type :: myclass
+  contains
+    procedure :: myfunc => myfunc
+  end type myclass
+
+contains
+  ! MLIR-HOST: func.func @_QMmymodulePmyfunc
+  ! MLIR-DEVICE-NOT: func.func @_QMmymodulePmyfunc
+
+  ! LLVM-HOST: define void @_QMmymodulePmyfunc
+  ! LLVM-DEVICE-NOT: define void @_QMmymodulePmyfunc
+  subroutine myfunc(self)
+    class(myclass) :: self
+    call foo()
+  end subroutine
+end module
+
+! MLIR-ALL: func.func @_QPmain
+
+! LLVM-HOST: define void @main_
+! LLVM-DEVICE-NOT: define void @main_
+subroutine main(x)
+  use mymodule
+  implicit none
+
+  integer, intent(inout) :: x
+  class(myclass), allocatable :: myobj
+  allocate(myobj)
+
+  ! MLIR-ALL: fir.dispatch "myfunc"
+
+  ! LLVM-HOST: %[[MYFUNC_PTR:.*]] = inttoptr i64 %{{.*}} to ptr
+  ! LLVM-HOST: call void %[[MYFUNC_PTR]](ptr %{{.*}})
+  ! LLVM-DEVICE-NOT: call void %{{.*}}(ptr %{{.*}})
+  call myobj%myfunc()
+
+  !$omp target map(tofrom: x)
+  x = x + 1
+  !$omp end target
+
+  deallocate(myobj)
+
+  ! LLVM-HOST: ret void
+end subroutine main
+
+! LLVM-ALL: define {{.*}}void @__omp_offloading{{.*}}main_{{.*}}
+
+! MLIR-ALL: fir.type_info @_QMmymoduleTmyclass {{.*}} dispatch_table
+! MLIR-ALL-NEXT: fir.dt_entry "myfunc", @_QMmymodulePmyfunc
diff --git a/flang/test/Lower/OpenMP/host-eval.f90 b/flang/test/Lower/OpenMP/host-eval.f90
index 3bd873009530c..35749a6ac5050 100644
--- a/flang/test/Lower/OpenMP/host-eval.f90
+++ b/flang/test/Lower/OpenMP/host-eval.f90
@@ -22,10 +22,8 @@ subroutine teams()
 
   !$omp end target
 
-  ! HOST: omp.teams
-  ! HOST-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
-
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
+  ! BOTH-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
   !$omp teams num_teams(1) thread_limit(2)
   call foo()
   !$omp end teams
@@ -78,18 +76,13 @@ subroutine distribute_parallel_do()
   !$omp end distribute parallel do
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-
-  ! DEVICE-NOT: omp.parallel
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.wsloop
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
   !$omp distribute parallel do num_threads(1)
   do i=1,10
     call foo()
@@ -147,15 +140,14 @@ subroutine distribute_parallel_do_simd()
   !$omp end distribute parallel do simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-  ! HOST-NEXT: omp.simd
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
+  ! BOTH-NEXT: omp.simd
 
   ! DEVICE-NOT: omp.parallel
   ! DEVICE-NOT: omp.distribute
@@ -207,12 +199,10 @@ subroutine distribute()
   !$omp end distribute
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! DEVICE-NOT: omp.distribute
+  ! BOTH: omp.distribute
   !$omp distribute
   do i=1,10
     call foo()
@@ -261,15 +251,11 @@ subroutine distribute_simd()
   !$omp end distribute simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.simd
-
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.simd
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.simd
   !$omp distribute simd
   do i=1,10
     call foo()
diff --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
deleted file mode 100644
index 3acdfda876812..0000000000000
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ /dev/null
@@ -1,560 +0,0 @@
-// RUN: fir-opt --omp-host-op-filtering %s | FileCheck %s
-
-module attributes {omp.is_target_device = true} {
-  // CHECK-LABEL: func.func private @external
-  // CHECK-NOT: return
-  func.func private @external()
-
-  // CHECK-LABEL: func.func @basic_checks
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.ref<i32>) -> (i32, f32)
-  func.func @basic_checks(%arg: !fir.ref<i32>) -> (i32, f32) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOC:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %r0 = arith.constant 10 : i32
-    %r1 = arith.constant 2.5 : f32
-
-    func.call @foo() : () -> ()
-
-    %0 = fir.declare %arg {uniq_name = "arg"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %alloc = fir.alloca i32
-    %2 = fir.declare %alloc {uniq_name = "alloc"} : (!fir.ref<i32>) -> !fir.ref<i32>
-
-    // CHECK-NEXT: %[[ALLOC_DECL:.*]] = fir.declare %[[ALLOC]] {uniq_name = "alloc"}
-    // CHECK-NEXT: %[[ARG_DECL:.*]] = fir.declare %[[ARG]] {uniq_name = "arg"}
-    // CHECK-NEXT: %[[GLOBAL_DECL:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOC_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP3:.*]] = omp.map.info var_ptr(%[[ALLOC]]{{.*}})
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[MAP2]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP3]] -> {{.*}} : {{.*}})
-    omp.target kernel_type(generic) has_device_addr(%m2 -> %arg0 : !fir.ref<i32>) map_entries(%m0 -> %arg1, %m1 -> %arg2, %m3 -> %arg3 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NEXT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: func.call
-    // CHECK-NOT: omp.map.info
-    omp.parallel {
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.map.info
-    %m4 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m5 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m6 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m7 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m4, %m5, %m6, %m7 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.target_enter_data
-    // CHECK-NOT: omp.target_exit_data
-    // CHECK-NOT: omp.target_update
-    %m8 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_enter_data map_entries(%m8 : !fir.ref<i32>)
-
-    %m9 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(from) capture(ByRef) -> !fir.ref<i32>
-    omp.target_exit_data map_entries(%m9 : !fir.ref<i32>)
-
-    %m10 = omp.map.info var_ptr(%2 : !fir.ref<i32>, !fir.ref<i32>) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_update map_entries(%m10 : !fir.ref<i32>)
-
-    // CHECK-NOT: func.call
-    func.call @foo() : () -> ()
-
-    // CHECK:      %[[RETURN0:.*]] = fir.undefined i32
-    // CHECK-NEXT: %[[RETURN1:.*]] = fir.undefined f32
-    // CHECK-NEXT: return %[[RETURN0]], %[[RETURN1]]
-    return %r0, %r1 : i32, f32
-  }
-
-  // CHECK-LABEL: func.func @allocatable_array
-  // CHECK-SAME: (%[[ALLOCATABLE:.*]]: [[ALLOCATABLE_TYPE:.*]], %[[ARRAY:.*]]: [[ARRAY_TYPE:[^)]*]])
-  func.func @allocatable_array(%allocatable: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, %array: !fir.ref<!fir.array<9xi32>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape %[[ZERO]] : (i64) -> !fir.shape<1>
-    // CHECK-NEXT: %[[ALLOCATABLE_DECL:.*]] = fir.declare %[[ALLOCATABLE]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : ([[ALLOCATABLE_TYPE]]) -> [[ALLOCATABLE_TYPE]]
-    // CHECK-NEXT: %[[ARRAY_DECL:.*]] = fir.declare %[[ARRAY]](%[[SHAPE]]) {uniq_name = "array"} : ([[ARRAY_TYPE]], !fir.shape<1>) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCATABLE_DECL]] base_addr : ([[ALLOCATABLE_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP_ALLOCATABLE:.*]] = omp.map.info var_ptr(%[[ALLOCATABLE_DECL]] : [[ALLOCATABLE_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP_ARRAY:.*]] = omp.map.info var_ptr(%[[ARRAY_DECL]] : [[ARRAY_TYPE]], !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP_ALLOCATABLE]] -> %{{.*}}, %[[MAP_ARRAY]] -> %{{.*}} : [[VAR_PTR_PTR_TYPE]], [[ARRAY_TYPE]])
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c8 = arith.constant 8 : index
-    %c9 = arith.constant 9 : index
-
-    %0 = fir.declare %allocatable {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %1 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %2 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%2 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%1) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-
-    %3 = fir.shape %c9 : (index) -> !fir.shape<1>
-    %4 = fir.declare %array(%3) {uniq_name = "array"} : (!fir.ref<!fir.array<9xi32>>, !fir.shape<1>) -> !fir.ref<!fir.array<9xi32>>
-    %5 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %6 = omp.map.info var_ptr(%4 : !fir.ref<!fir.array<9xi32>>, !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) bounds(%5) -> !fir.ref<!fir.array<9xi32>>
-
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg0, %6 -> %arg1 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, !fir.ref<!fir.array<9xi32>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @character
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @character(%x: !fir.ref<!fir.char<1>>) {
-    // CHECK-NEXT: %[[ZERO]] = arith.constant 0 : i64
-    %0 = fir.dummy_scope : !fir.dscope
-    %c1 = arith.constant 1 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] typeparams %[[ZERO]] {uniq_name = "x"} : ([[X_TYPE]], i64) -> [[X_TYPE]]
-    %3 = fir.declare %x typeparams %c1 dummy_scope %0 {uniq_name = "x"} : (!fir.ref<!fir.char<1>>, index, !fir.dscope) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> [[X_TYPE]]
-    %map = omp.map.info var_ptr(%3 : !fir.ref<!fir.char<1>>, !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}})
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<!fir.char<1>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_rank
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @assumed_rank(%x: !fir.box<!fir.array<*:f32>>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<[[X_TYPE]]>
-    %0 = fir.alloca !fir.box<!fir.array<*:f32>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {uniq_name = "x"} : (!fir.box<!fir.array<*:f32>>, !fir.dscope) -> !fir.box<!fir.array<*:f32>>
-    %3 = fir.rebox_assumed_rank %2 lbs ones : (!fir.box<!fir.array<*:f32>>) -> !fir.box<!fir.array<*:f32>>
-    %4 = fir.box_addr %3 : (!fir.box<!fir.array<*:f32>>) -> !fir.ref<!fir.array<*:f32>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA]] base_addr : (!fir.ref<[[X_TYPE]]>) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %5 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.array<*:f32>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], !fir.array<*:f32>) -> [[VAR_PTR_PTR_TYPE]]
-    %6 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%5 : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>, !fir.array<*:f32>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> !fir.ref<!fir.array<*:f32>>
-    %7 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(to) capture(ByRef) members(%6 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) -> !fir.ref<!fir.array<*:f32>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> {{.*}})
-    omp.target kernel_type(generic) map_entries(%7 -> %arg1, %6 -> %arg2 : !fir.ref<!fir.array<*:f32>>, !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @box_ptr
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @box_ptr(%x: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape_shift %[[ZERO]], %[[ZERO]] : (i64, i64) -> !fir.shapeshift<1>
-    // CHECK-NEXT: %[[PLACEHOLDER_X:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_X:.*]] = fir.convert %[[PLACEHOLDER_X]] : (!fir.ref<i1>) -> [[X_TYPE]]
-    %0 = fir.alloca !fir.box<!fir.ptr<!fir.array<?xi32>>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {fortran_attrs = #fir.var_attrs<contiguous, pointer>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.dscope) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    %3 = fir.load %2 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-
-    // CHECK-NEXT: %[[PLACEHOLDER_Y:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_Y:.*]] = fir.convert %[[PLACEHOLDER_Y]] : (!fir.ref<i1>) -> [[Y_TYPE:.*]]
-    %c0 = arith.constant 0 : index
-    %4:3 = fir.box_dims %3, %c0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c1 = arith.constant 1 : index
-    %c0_0 = arith.constant 0 : index
-    %5:3 = fir.box_dims %3, %c0_0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c0_1 = arith.constant 0 : index
-    %6 = arith.subi %5#1, %c1 : index
-    %7 = omp.map.bounds lower_bound(%c0_1 : index) upper_bound(%6 : index) extent(%5#1 : index) stride(%5#2 : index) start_idx(%4#0 : index) {stride_in_bytes = true}
-    %8 = fir.box_addr %3 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>) -> !fir.ptr<!fir.array<?xi32>>
-    %c0_2 = arith.constant 0 : index
-    %9:3 = fir.box_dims %3, %c0_2 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %10 = fir.shape_shift %9#0, %9#1 : (index, index) -> !fir.shapeshift<1>
-    
-    // CHECK-NEXT: %[[Y_DECL:.*]] = fir.declare %[[ALLOCA_Y]](%[[SHAPE]]) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : ([[Y_TYPE]], !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %11 = fir.declare %8(%10) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %12 = fir.embox %11(%10) : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.box<!fir.array<?xi32>>
-    %c1_3 = arith.constant 1 : index
-    %c0_4 = arith.constant 0 : index
-    %13:3 = fir.box_dims %12, %c0_4 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
-    %c0_5 = arith.constant 0 : index
-    %14 = arith.subi %13#1, %c1_3 : index
-    %15 = omp.map.bounds lower_bound(%c0_5 : index) upper_bound(%14 : index) extent(%13#1 : index) stride(%13#2 : index) start_idx(%9#0 : index) {stride_in_bytes = true}
-    
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA_X]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[Y_DECL]] : [[Y_TYPE]], i32) {{.*}} -> [[Y_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], i32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} members(%[[MAP1]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %16 = omp.map.info var_ptr(%11 : !fir.ptr<!fir.array<?xi32>>, i32) map_clauses(tofrom) capture(ByRef) bounds(%15) -> !fir.ptr<!fir.array<?xi32>>
-    %17 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %18 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) var_ptr_ptr(%17 : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) bounds(%7) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %19 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) members(%18 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP2]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : [[Y_TYPE]], [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%16 -> %arg1, %19 -> %arg2, %18 -> %arg3 : !fir.ptr<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @target_data
-  // CHECK-SAME: (%[[MAPPED:.*]]: [[MAPPED_TYPE:[^)]*]], %[[USEDEVADDR:.*]]: [[USEDEVADDR_TYPE:[^)]*]], %[[USEDEVPTR:.*]]: [[USEDEVPTR_TYPE:[^)]*]])
-  func.func @target_data(%mapped: !fir.ref<i32>, %usedevaddr: !fir.ref<i32>, %usedevptr: !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-    // CHECK-NEXT: %[[MAPPED_DECL:.*]] = fir.declare %[[MAPPED]] {uniq_name = "mapped"} : ([[MAPPED_TYPE]]) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_DECL:.*]] = fir.declare %[[USEDEVADDR]] {uniq_name = "usedevaddr"} : ([[USEDEVADDR_TYPE]]) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_DECL:.*]] = fir.declare %[[USEDEVPTR]] {uniq_name = "usedevptr"} : ([[USEDEVPTR_TYPE]]) -> [[USEDEVPTR_TYPE]]
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %1 = fir.declare %usedevaddr {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %2 = fir.declare %usedevptr {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-    // CHECK-NEXT: %[[MAPPED_MAP:.*]] = omp.map.info var_ptr(%[[MAPPED_DECL]] : [[MAPPED_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVADDR_DECL]] : [[USEDEVADDR_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVPTR_DECL]] : [[USEDEVPTR_TYPE]], !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> [[USEDEVPTR_TYPE]]
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(return_param) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(return_param) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-    
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) use_device_addr(%m1 -> %arg0 : !fir.ref<i32>) use_device_ptr(%m2 -> %arg1 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-      %3 = fir.declare %arg0 {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %4 = fir.declare %arg1 {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-      %m3 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m4 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m5 = omp.map.info var_ptr(%4 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAPPED_MAP]] -> %{{.*}}, %[[USEDEVADDR_MAP]] -> %{{.*}}, %[[USEDEVPTR_MAP]] -> %{{.*}} : {{.*}})
-      omp.target kernel_type(generic) map_entries(%m3 -> %arg2, %m4 -> %arg3, %m5 -> %arg4 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-        omp.terminator
-      }
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      omp.terminator
-    }
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @no_target
-  // CHECK-SAME: (%[[MAPPED:.*]]: !fir.ref<i32>)
-  func.func @no_target(%mapped: !fir.ref<i32>) {
-    // CHECK-NEXT: return
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        func.call @foo() : () -> ()
-        omp.terminator
-      }
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @map_info_members
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @map_info_members(%x: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) {
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c9 = arith.constant 9 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    %23 = fir.declare %x {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %63 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %64:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %65:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %66 = arith.subi %c1, %64#0 : index
-    %67 = arith.subi %c9, %64#0 : index
-    %68 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %69:3 = fir.box_dims %68, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %70 = omp.map.bounds lower_bound(%66 : index) upper_bound(%67 : index) extent(%69#1 : index) stride(%65#2 : index) start_idx(%64#0 : index) {stride_in_bytes = true}
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[X_DECL]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %71 = fir.box_offset %23 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    %72 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%71 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%70) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %73 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(to) capture(ByRef) members(%72 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> %{{.*}} : [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%73 -> %arg0, %72 -> %arg1 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @control_flow
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^,]*]], %[[COND:.*]]: [[COND_TYPE:[^)]*]])
-  func.func @control_flow(%x: !fir.ref<i32>, %cond: !fir.ref<!fir.logical<4>>) {
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %cond_decl = fir.declare %cond {uniq_name = "cond"} : (!fir.ref<!fir.logical<4>>) -> !fir.ref<!fir.logical<4>>
-    %0 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-    %1 = fir.convert %0 : (!fir.logical<4>) -> i1
-    cf.cond_br %1, ^bb1, ^bb2
-  ^bb1:  // pred: ^bb0
-    fir.call @foo() : () -> ()
-    %m0 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-    cf.br ^bb2
-  ^bb2:  // 2 preds: ^bb0, ^bb1
-    // CHECK-NOT: fir.call
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    fir.call @foo() : () -> ()
-    %m1 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-      fir.call @foo() : () -> ()
-      %8 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-      %9 = fir.convert %8 : (!fir.logical<4>) -> i1
-      cf.cond_br %9, ^bb1, ^bb2
-    ^bb1:  // pred: ^bb0
-      fir.call @foo() : () -> ()
-      %m2 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      // CHECK-NOT: fir.call
-      // CHECK-NOT: cf.br
-      fir.call @foo() : () -> ()
-      cf.br ^bb2
-    ^bb2:  // 2 preds: ^bb0, ^bb1
-      fir.call @foo() : () -> ()
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @block_args
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @block_args(%x: !fir.ref<i32>) {
-    // CHECK-NEXT: %[[PLACEHOLDER0:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA0:.*]] = fir.convert %[[PLACEHOLDER0]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[PLACEHOLDER1:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA1:.*]] = fir.convert %[[PLACEHOLDER1]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[X_DECL0:.*]] = fir.declare %[[ALLOCA0]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[X_DECL1:.*]] = fir.declare %[[ALLOCA1]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL0]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL1]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %0 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %1 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        omp.parallel private(@privatizer %1 -> %arg1 : !fir.ref<i32>) {
-          %2 = fir.declare %arg1 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-          %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-          omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-            omp.terminator
-          }
-          omp.terminator
-        }
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    return
-  }
-
-  // CHECK-LABEL: func.func @reuse_tests()
-  func.func @reuse_tests() {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[THREAD_LIMIT:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> i32
-    // CHECK-NEXT: %[[CONST:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL_DECL0:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL1:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL2:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL0]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL1]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL2]] : !fir.ref<i32>, i32)
-    %0 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m2 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : !fir.ref<i32>, !fir.ref<i32>)
-      omp.target kernel_type(generic) map_entries(%m1 -> %arg0, %m2 -> %arg1 : !fir.ref<i32>, !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-    // CHECK-NOT: fir.load
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    %2 = fir.load %global : !fir.ref<i32>
-    %3 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) thread_limit(%2 : i32) map_entries(%m3 -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    %c1 = arith.constant 1 : i32
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @all_non_map_clauses
-  // CHECK-SAME: (%[[REF:.*]]: !fir.ref<i32>, %[[INT:.*]]: i32, %[[BOOL:.*]]: i1)
-  func.func @all_non_map_clauses(%ref: !fir.ref<i32>, %int: i32, %bool: i1) {
-    %m0 = omp.map.info var_ptr(%ref : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data device(%int : i32) if(%bool) map_entries(%m0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate({{[^)]*}}) thread_limit({{[^)]*}}) in_reduction({{[^)]*}}) private({{[^)]*}}) {
-    omp.target kernel_type(generic)
-               allocate(%ref : !fir.ref<i32> -> %ref : !fir.ref<i32>)
-               depend(taskdependin -> %ref : !fir.ref<i32>)
-               device(%int : i32) if(%bool) thread_limit(%int : i32)
-               in_reduction(@reduction %ref -> %arg0 : !fir.ref<i32>)
-               private(@privatizer %ref -> %arg1 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NOT: omp.target_enter_data
-    omp.target_enter_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                          device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_exit_data
-    omp.target_exit_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                         device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_update
-    omp.target_update depend(taskdependin -> %ref : !fir.ref<i32>)
-                      device(%int : i32) if(%bool)
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_length
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.boxchar<1>)
-  func.func @assumed_length(%arg: !fir.boxchar<1>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca !fir.char<1>
-    // CHECK-NEXT: %[[ONE:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[EMBOXCHAR:.*]] = fir.emboxchar %[[PLACEHOLDER]], %[[ONE]] : (!fir.ref<!fir.char<1>>, i32) -> !fir.boxchar<1>
-    // CHECK-NEXT: omp.target kernel_type(generic) private(@boxchar_firstprivatizer %[[EMBOXCHAR]] -> %{{.*}} [map_idx=0] : !fir.boxchar<1>)
-    %0 = fir.alloca !fir.boxchar<1>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2:2 = fir.unboxchar %arg : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
-    %3 = fir.declare %2#0 typeparams %2#1 dummy_scope %1 {uniq_name = "arg"} : (!fir.ref<!fir.char<1,?>>, index, !fir.dscope) -> !fir.ref<!fir.char<1,?>>
-    %4 = fir.emboxchar %3, %2#1 : (!fir.ref<!fir.char<1,?>>, index) -> !fir.boxchar<1>
-    omp.target kernel_type(generic) private(@boxchar_firstprivatizer %4 -> %arg3 [map_idx=0] : !fir.boxchar<1>) {
-      omp.terminator
-    }
-    return
-  }
-
-  omp.private {type = firstprivate} @boxchar_firstprivatizer : !fir.boxchar<1> copy {
-  ^bb0(%arg0: !fir.boxchar<1>, %arg1: !fir.boxchar<1>):
-    omp.yield(%arg0 : !fir.boxchar<1>)
-  }
-
-  // CHECK-LABEL: func.func @declare_storage
-  func.func @declare_storage(%i : index) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[VAR:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %0 = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %1 = fir.convert %0 : (!fir.ref<!fir.array<8xi8>>) -> !fir.ref<!fir.array<?xi8>>
-    %2 = fir.coordinate_of %1, %i : (!fir.ref<!fir.array<?xi8>>, index) -> !fir.ref<i8>
-    %3 = fir.convert %2 : (!fir.ref<i8>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[DECL:.*]] = fir.declare %[[VAR]] storage(%[[GLOBAL]][0])
-    %4 = fir.declare %3 storage (%0[0]) {uniq_name = "a"} : (!fir.ref<i32>, !fir.ref<!fir.array<8xi8>>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[DECL]] : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %map = omp.map.info var_ptr(%4 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    return
-  }
-
-  func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  omp.private {type = firstprivate} @privatizer : i32 copy {
-  ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
-    %0 = fir.load %arg0 : !fir.ref<i32>
-    fir.store %0 to %arg1 : !fir.ref<i32>
-    omp.yield(%arg1 : !fir.ref<i32>)
-  }
-  omp.declare_reduction @reduction : i32
-  init {
-  ^bb0(%arg: i32):
-    %0 = arith.constant 0 : i32
-    omp.yield (%0 : i32)
-  }
-  combiner {
-  ^bb1(%arg0: i32, %arg1: i32):
-    %1 = arith.addi %arg0, %arg1 : i32
-    omp.yield (%1 : i32)
-  }
-
-  // CHECK: fir.global internal @global_scalar constant : i32
-  fir.global @global_scalar constant : i32
-
-  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-
-  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-}
diff --git a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
index e6321ef58b45f..da90ef508ce64 100644
--- a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
@@ -11,6 +11,30 @@
 
 include "mlir/Pass/PassBase.td"
 
+def HostOpFilteringPass : Pass<"omp-host-op-filter"> {
+  let summary = "Removes all non target-related ops from host functions.";
+  let description = [{
+    When compiling for an OpenMP offloading target, host functions holding
+    target regions must be maintained in order for the `omp.target` ops to be
+    translated to LLVM IR as functions.
+    
+    However, these host functions might contain more host code than just what's
+    necessary for generating code for their contained target regions on the
+    target device, potentially causing problematic artifacts to be generated
+    into the target device's module.
+
+    This pass strips out all non-essential operations from host functions when
+    compiling for an OpenMP target device, leaving just `omp.target` operations,
+    related `omp.map.info` and placeholders in the form of function arguments to
+    keep the MLIR representation legal.
+
+    It must run on a module at a low level of abstraction (i.e. directly
+    translatable to LLVM IR), after any passes that might produce new OpenMP
+    operations have already completed.
+  }];
+  let dependentDialects = ["LLVM::LLVMDialect"];
+}
+
 def MarkDeclareTargetPass : Pass<"omp-mark-declare-target", "ModuleOp"> {
   let summary = "Marks all functions called by an OpenMP declare target "
                 "function as declare target";
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
index 569786fe95cf3..bbdbc556fc52f 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
@@ -1,4 +1,5 @@
 add_mlir_dialect_library(MLIROpenMPTransforms
+  HostOpFiltering.cpp
   MarkDeclareTarget.cpp
   OpenMPOffloadPrivatizationPrepare.cpp
   StackToShared.cpp
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
new file mode 100644
index 0000000000000..9c61b2717af57
--- /dev/null
+++ b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
@@ -0,0 +1,292 @@
+//===- HostOpFiltering.cpp ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements transforms to swap stack allocations on the target
+// device with device shared memory where applicable.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+
+namespace mlir {
+namespace omp {
+#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h.inc"
+} // namespace omp
+} // namespace mlir
+
+using namespace mlir;
+
+/// Some host operations, like \c llvm.mlir.addressof and constants, must remain
+/// in the device module because they impact how device code is generated when
+/// attached to an \c omp.target operation.
+///
+/// This function identifies the operations that need this special handling.
+/// This includes cast-style operations to avoid losing information about the
+/// original source of an operand.
+static bool keepHostOpInDevice(Operation &op) {
+  return isPure(&op) &&
+         op.getDialect() ==
+             op.getContext()->getLoadedDialect<LLVM::LLVMDialect>();
+}
+
+/// Add an \c omp.map.info operation and all its members recursively to the
+/// output set to be later rewritten.
+///
+/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
+/// are added to the output sets before operations based on them.
+static void collectRewrite(omp::MapInfoOp mapOp,
+                           llvm::SetVector<omp::MapInfoOp> &rewrites) {
+  for (Value member : mapOp.getMembers())
+    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
+
+  rewrites.insert(mapOp);
+}
+
+/// Add the given value to a sorted set if it should be replaced by a
+/// placeholder when used as an operand that must remain for the device.
+///
+/// Values that are block arguments of function operations are skipped, since
+/// they will still be available after all rewrites are completed, and operands
+/// of operations that need to remain on the host are recursively collected.
+static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
+  if ((isa<BlockArgument>(value) &&
+       isa<FunctionOpInterface>(
+           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
+      rewrites.contains(value))
+    return;
+
+  Operation *op = value.getDefiningOp();
+  if (op && keepHostOpInDevice(*op))
+    for (Value operand : op->getOperands())
+      collectRewrite(operand, rewrites);
+
+  rewrites.insert(value);
+}
+
+/// Provide the \c device_type of an \c omp.declare_target attribute, if
+/// defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
+namespace {
+class HostOpFilteringPass
+    : public omp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
+public:
+  HostOpFilteringPass() = default;
+
+  void runOnOperation() override {
+    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
+    if (!op || !op.getIsTargetDevice())
+      return;
+
+    op->walk<WalkOrder::PreOrder>([&](LLVM::LLVMFuncOp funcOp) {
+      omp::DeclareTargetDeviceType declareType =
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
+
+      // Only process host function definitions.
+      if (funcOp.isExternal() ||
+          declareType != omp::DeclareTargetDeviceType::host)
+        return WalkResult::advance();
+
+      if (failed(rewriteHostFunction(funcOp))) {
+        funcOp.emitOpError() << "could not filter host-only operations";
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time redefinitions.
+    op->walk([&](LLVM::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkage(LLVM::Linkage::Internal);
+    });
+  }
+
+private:
+  /// Rewrite the given host device function containing \c omp.target
+  /// operations, to remove host-only operations that are not used by device
+  /// codegen.
+  ///
+  /// It is based on the expected form of an MLIR module lowered to where it can
+  /// be directly translated to LLVM IR and it performs the following mutations:
+  ///   - Removes all returned values from the function.
+  ///   - \c omp.target operations are moved to the end of the function. If they
+  ///     are nested inside of any other operations, they are hoisted out of
+  ///     them.
+  ///   - \c depend, \c device, \c dyn_groupprivate, \c if and \c in_reduction
+  ///     clauses are removed from these target functions. Values used to
+  ///     initialize other clauses are replaced by placeholders as follows:
+  ///     - Values defined by block arguments are replaced by placeholders only
+  ///       if they are not attached to the parent function. In that case, they
+  ///       are passed unmodified.
+  ///     - Pure operations of the LLVM dialect are maintained, and any value
+  ///       operands they might have are also replaced by placeholders following
+  ///       the same rules.
+  ///     - Other values are replaced by new function arguments.
+  ///   - \c omp.map.info operations associated to these target regions are
+  ///     preserved. These are moved above all \c omp.target and sorted to
+  ///     satisfy dependencies among them.
+  ///   - \c bounds arguments are removed from \c omp.map.info operations.
+  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
+  ///     replaced by placeholders as described above.
+  ///   - Every other operation not located inside of an \c omp.target is
+  ///     removed.
+  LogicalResult rewriteHostFunction(LLVM::LLVMFuncOp funcOp) {
+    Region &region = funcOp.getFunctionBody();
+    LLVM::LLVMFunctionType functionType = funcOp.getFunctionType();
+
+    // Collect target operations inside of the function.
+    llvm::SmallVector<omp::TargetOp> targetOps;
+    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
+      // Skip the inside of omp.target regions, since these contain device code.
+      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
+        targetOps.push_back(targetOp);
+        return WalkResult::skip();
+      }
+
+      // Replace omp.target_data entry block argument uses with the value used
+      // to initialize the associated omp.map.info operation. This way,
+      // references are still valid once the omp.target operation has been
+      // extracted out of the omp.target_data region.
+      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
+        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
+        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
+            .getBlockArgsPairs(argPairs);
+        for (auto [operand, blockArg] : argPairs) {
+          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
+          blockArg.replaceAllUsesWith(mapInfo.getVarPtr());
+        }
+      }
+      return WalkResult::advance();
+    });
+
+    // Make a temporary clone of the parent function with an empty region,
+    // and update all references to entry block arguments to those of the new
+    // region. Users of these arguments will later either be moved to the new
+    // region or deleted when the original region is replaced by the new.
+    OpBuilder builder(&getContext());
+    builder.setInsertionPointAfter(funcOp);
+    Operation *newFuncOp = builder.cloneWithoutRegions(funcOp);
+    Block &block = newFuncOp->getRegion(0).emplaceBlock();
+
+    llvm::SmallVector<Location> locs;
+    locs.reserve(region.getNumArguments());
+    llvm::transform(region.getArguments(), std::back_inserter(locs),
+                    [](const BlockArgument &arg) { return arg.getLoc(); });
+    block.addArguments(region.getArgumentTypes(), locs);
+
+    for (auto [oldArg, newArg] :
+         llvm::zip_equal(region.getArguments(), block.getArguments()))
+      oldArg.replaceAllUsesWith(newArg);
+
+    // Collect omp.map.info ops while satisfying interdependencies and remove
+    // operands that aren't used by target device codegen.
+    //
+    // This logic must be updated whenever operands to omp.target change.
+    llvm::SetVector<Value> rewriteValues;
+    llvm::SetVector<omp::MapInfoOp> mapInfos;
+    for (omp::TargetOp targetOp : targetOps) {
+      assert(targetOp.getHostEvalVars().empty() &&
+             "unexpected host_eval in target device module");
+
+      // Variables unused by the device.
+      targetOp.getDependVarsMutable().clear();
+      targetOp.setDependKindsAttr(nullptr);
+      targetOp.getDependIteratedMutable().clear();
+      targetOp.setDependIteratedKindsAttr(nullptr);
+      targetOp.getDeviceMutable().clear();
+      targetOp.getDynGroupprivateSizeMutable().clear();
+      targetOp.getIfExprMutable().clear();
+      targetOp.getInReductionVarsMutable().clear();
+      targetOp.setInReductionByrefAttr(nullptr);
+      targetOp.setInReductionSymsAttr(nullptr);
+
+      // TODO: Clear some of these operands rather than rewriting them,
+      // depending on whether they are needed by device codegen once support for
+      // them is fully implemented.
+      for (Value allocVar : targetOp.getAllocateVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value allocVar : targetOp.getAllocatorVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
+        collectRewrite(isDevPtr, rewriteValues);
+      for (Value mapVar : targetOp.getHasDeviceAddrVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value mapVar : targetOp.getMapVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value privateVar : targetOp.getPrivateVars())
+        collectRewrite(privateVar, rewriteValues);
+      for (Value threadLimit : targetOp.getThreadLimitVars())
+        collectRewrite(threadLimit, rewriteValues);
+    }
+
+    // Move omp.map.info ops to the new block and collect dependencies.
+    for (omp::MapInfoOp mapOp : mapInfos) {
+      collectRewrite(mapOp.getVarPtr(), rewriteValues);
+
+      if (Value varPtrPtr = mapOp.getVarPtrPtr())
+        collectRewrite(varPtrPtr, rewriteValues);
+
+      // Bounds are not used during target device codegen.
+      mapOp.getBoundsMutable().clear();
+      mapOp->moveBefore(&block, block.end());
+    }
+
+    builder.setInsertionPointToStart(&block);
+
+    // We don't actually need the proper initialization for all operands, but
+    // rather just to maintain the basic form of omp.target operations. We
+    // create new function arguments as placeholders for rewritten values.
+    llvm::SmallVector<Type> newFnArgTypes(functionType.getParams());
+    for (Value value : rewriteValues) {
+      Value rewriteValue;
+      Operation *definingOp = value.getDefiningOp();
+      if (definingOp && keepHostOpInDevice(*definingOp)) {
+        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
+      } else {
+        rewriteValue = block.addArgument(value.getType(), value.getLoc());
+        newFnArgTypes.push_back(rewriteValue.getType());
+      }
+      value.replaceAllUsesWith(rewriteValue);
+    }
+
+    // Move target operations to the end of the new block.
+    for (omp::TargetOp targetOp : targetOps)
+      targetOp->moveBefore(&block, block.end());
+
+    // Add terminator to the new block.
+    builder.setInsertionPointToEnd(&block);
+    LLVM::ReturnOp::create(builder, funcOp.getLoc(), ValueRange());
+
+    // Replace old region with the new one, now only containing the required
+    // operations, and remove the temporary operation clone.
+    region.takeBody(newFuncOp->getRegion(0));
+    newFuncOp->erase();
+
+    // Update function type after modifying the terminator and argument list.
+    funcOp.setType(LLVM::LLVMFunctionType::get(
+        LLVM::LLVMVoidType::get(&getContext()), newFnArgTypes));
+
+    return success();
+  }
+};
+} // namespace
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
new file mode 100644
index 0000000000000..015e901126a76
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -0,0 +1,500 @@
+// RUN: mlir-opt --omp-host-op-filter %s | FileCheck %s
+module attributes {omp.is_target_device = true} {
+  // CHECK-LABEL: llvm.func @external() attributes {sym_visibility = "private"}
+  // CHECK-NOT: llvm.return
+  llvm.func @external() attributes {sym_visibility = "private"}
+
+  // CHECK-LABEL: llvm.func @basic_checks
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @basic_checks(%arg0: !llvm.ptr) -> !llvm.struct<(i32, f32)> {
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = llvm.mlir.constant(10 : i32) : i32
+    %3 = llvm.mlir.constant(2.500000e+00 : f32) : f32
+
+    llvm.call @foo() : () -> ()
+
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %4 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %5 = llvm.mlir.constant(1 : i64) : i64
+
+    // CHECK-NEXT: %[[HDA:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]]{{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]]{{.*}})
+    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    %6 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %7 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %8 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %9 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[HDA]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP2]] -> {{.*}} : {{.*}})
+    omp.target kernel_type(generic) has_device_addr(%8 -> %arg1 : !llvm.ptr) map_entries(%6 -> %arg2, %7 -> %arg3, %9 -> %arg4 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NEXT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: llvm.call
+    omp.parallel {
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.map.info
+    %10 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %11 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %12 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %13 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%10, %11, %12, %13 : !llvm.ptr, !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    %14 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_enter_data map_entries(%14 : !llvm.ptr)
+    %15 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(from) capture(ByRef) -> !llvm.ptr
+    omp.target_exit_data map_entries(%15 : !llvm.ptr)
+    %16 = omp.map.info var_ptr(%1 : !llvm.ptr, !llvm.ptr) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_update map_entries(%16 : !llvm.ptr)
+
+    // CHECK-NOT: llvm.call
+    llvm.call @foo() : () -> ()
+
+    // CHECK: {{llvm\.return$}}
+    %17 = llvm.mlir.poison : !llvm.struct<(i32, f32)>
+    %18 = llvm.insertvalue %2, %17[0] : !llvm.struct<(i32, f32)>
+    %19 = llvm.insertvalue %3, %18[1] : !llvm.struct<(i32, f32)>
+    llvm.return %19 : !llvm.struct<(i32, f32)>
+  }
+
+  // CHECK-LABEL: llvm.func @member_map_complex_init
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @member_map_complex_init(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[PLACEHOLDER]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<{{.*}}>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(1 : i64) : i64
+    %3 = llvm.alloca %2 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %4 = llvm.mlir.constant(1 : i64) : i64
+    %5 = llvm.alloca %4 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %6 = llvm.mlir.constant(1 : i32) : i32
+    %7 = llvm.mlir.constant(0 : i8) : i8
+    %8 = llvm.mlir.constant(1 : i64) : i64
+    %9 = llvm.mlir.constant(1 : i64) : i64
+    %10 = llvm.mlir.zero : !llvm.ptr
+    llvm.call @LangRTPlaceholderFunc(%5, %arg0, %10, %7, %6) : (!llvm.ptr, !llvm.ptr, !llvm.ptr, i8, i32) -> ()
+    %11 = llvm.mlir.constant(384 : i32) : i32
+    "llvm.intr.memcpy"(%1, %5, %11) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %12 = llvm.mlir.constant(24 : i32) : i32
+    %13 = llvm.getelementptr %1[0, 3] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+    %14 = llvm.load %13 : !llvm.ptr -> i8
+    %15 = llvm.sext %14 : i8 to i32
+    %16 = llvm.mlir.constant(24 : i32) : i32
+    %17 = llvm.mul %16, %15 : i32
+    %18 = llvm.add %12, %17 : i32
+    "llvm.intr.memcpy"(%3, %1, %18) <{isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %19 = llvm.getelementptr %3[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[GEP]] : {{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr)
+    %20 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%19 : !llvm.ptr, f32) -> !llvm.ptr
+    %21 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%20 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> %{{.*}} : {{.*}})
+    omp.target kernel_type(generic) map_entries(%21 -> %arg1, %20 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @target_data
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr, %[[ARG2:.*]]: !llvm.ptr)
+  llvm.func @target_data(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: !llvm.ptr) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%arg2 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%0 : !llvm.ptr) use_device_addr(%1 -> %arg3 : !llvm.ptr) use_device_ptr(%2 -> %arg4 : !llvm.ptr) {
+      // CHECK:      %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ARG2]] : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef)
+      %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %4 = omp.map.info var_ptr(%arg3 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%arg4 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}}, %[[MAP2]] -> %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%3 -> %arg5, %4 -> %arg6, %5 -> %arg7 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @no_target
+  // CHECK-SAME: (%{{.*}}: !llvm.ptr)
+  llvm.func @no_target(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: llvm.return
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%0 : !llvm.ptr) {
+      %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%1 : !llvm.ptr) {
+        llvm.call @foo() : () -> ()
+        omp.terminator
+      }
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @map_info_members
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr)
+  llvm.func @map_info_members(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = llvm.getelementptr %[[ARG]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(0 : index) : i64
+    %3 = llvm.mlir.constant(1 : index) : i64
+    %4 = llvm.mlir.constant(9 : index) : i64
+    %5 = llvm.mlir.constant(48 : i32) : i32
+    "llvm.intr.memcpy"(%1, %arg0, %5) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %6 = llvm.getelementptr %1[0, 7, %2, 0] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %7 = llvm.load %6 : !llvm.ptr -> i64
+    %8 = llvm.getelementptr %1[0, 7, %2, 1] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %9 = llvm.load %8 : !llvm.ptr -> i64
+    %10 = llvm.getelementptr %1[0, 7, %2, 2] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %11 = llvm.load %10 : !llvm.ptr -> i64
+    %12 = llvm.sub %3, %7 : i64
+    %13 = llvm.sub %4, %7 : i64
+    %14 = omp.map.bounds lower_bound(%12 : i64) upper_bound(%13 : i64) extent(%9 : i64) stride(%11 : i64) start_idx(%7 : i64) {stride_in_bytes = true}
+    %15 = llvm.getelementptr %arg0[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : !llvm.ptr, f32) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr) -> !llvm.ptr
+    %16 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%15 : !llvm.ptr, f32) bounds(%14) -> !llvm.ptr
+    %17 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%16 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> {{.*}} : !llvm.ptr, !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%17 -> %arg1, %16 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @control_flow
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr)
+  llvm.func @control_flow(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %0 = llvm.load %arg1 : !llvm.ptr -> i32
+    %1 = llvm.mlir.constant(0 : i32) : i32
+    %2 = llvm.icmp "ne" %0, %1 : i32
+    llvm.cond_br %2, ^bb1, ^bb2
+  ^bb1:  // pred: ^bb0
+    llvm.call @foo() : () -> ()
+    %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%3 -> %arg2 : !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+    llvm.br ^bb2
+  ^bb2:  // 2 preds: ^bb0, ^bb1
+    // CHECK-NOT: llvm.call
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    llvm.call @foo() : () -> ()
+
+    %4 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%4 : !llvm.ptr) {
+      llvm.call @foo() : () -> ()
+      %5 = llvm.load %arg1 : !llvm.ptr -> i32
+      %6 = llvm.mlir.constant(0 : i32) : i32
+      %7 = llvm.icmp "ne" %5, %6 : i32
+      llvm.cond_br %7, ^bb1, ^bb2
+    ^bb1:  // pred: ^bb0
+      llvm.call @foo() : () -> ()
+      %8 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%8 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      // CHECK-NOT: llvm.br
+      llvm.call @foo() : () -> ()
+      llvm.br ^bb2
+    ^bb2:  // 2 preds: ^bb0, ^bb1
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @block_args
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[PLACEHOLDER0:.*]]: !llvm.ptr, %[[PLACEHOLDER1:.*]]: !llvm.ptr)
+  llvm.func @block_args(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%0 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%0 : !llvm.ptr) {
+        omp.parallel private(@privatizer %arg1 -> %arg2 : !llvm.ptr) {
+          %1 = omp.map.info var_ptr(%arg2 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+          omp.target kernel_type(generic) map_entries(%1 -> %arg3 : !llvm.ptr) {
+            omp.terminator
+          }
+          omp.terminator
+        }
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reuse_tests
+  // CHECK-SAME: (%[[THREAD_LIMIT:.*]]: i32)
+  llvm.func @reuse_tests() {
+    // CHECK-NEXT: %[[CONST_THREAD_LIMIT:.*]] = llvm.mlir.constant(1 : i32) : i32
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%2 : !llvm.ptr) {
+      // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %6 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}} : !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%5 -> %arg0, %6 -> %arg1 : !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: llvm.load
+    // CHECK-NOT: omp.map.info
+    %3 = llvm.load %1 : !llvm.ptr -> i32
+    %4 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) thread_limit(%3 : i32) map_entries(%4 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @all_non_map_clauses
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: i1)
+  llvm.func @all_non_map_clauses(%arg0: !llvm.ptr, %arg1: i32, %arg2: i1) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data device(%arg1 : i32) if(%arg2) map_entries(%0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    omp.target_enter_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_exit_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_update depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @private_with_map_idx
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i64)
+  llvm.func @private_with_map_idx(%arg0: !llvm.ptr, %arg1: i64) {
+    // CHECK-NEXT: %[[VAL0:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL1:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL2:.*]] = llvm.insertvalue %[[ARG0]], %[[VAL1]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL3:.*]] = llvm.insertvalue %[[ARG1]], %[[VAL2]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL4:.*]] = llvm.extractvalue %[[VAL3]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL5:.*]] = llvm.insertvalue %[[VAL4]], %[[VAL0]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL6:.*]] = llvm.extractvalue %[[VAL3]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL7:.*]] = llvm.insertvalue %[[VAL6]], %[[VAL5]][1] : !llvm.struct<(ptr, i64)>
+    %0 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, i64)>
+    %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, i64)>
+    %3 = llvm.extractvalue %2[0] : !llvm.struct<(ptr, i64)>
+    %4 = llvm.extractvalue %2[1] : !llvm.struct<(ptr, i64)>
+    %5 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %6 = llvm.insertvalue %3, %5[0] : !llvm.struct<(ptr, i64)>
+    %7 = llvm.insertvalue %4, %6[1] : !llvm.struct<(ptr, i64)>
+
+    // CHECK: omp.target kernel_type(generic) private(@struct_firstprivatizer %[[VAL7]] -> %{{.*}} [map_idx=0] : !llvm.struct<(ptr, i64)>)
+    omp.target kernel_type(generic) private(@struct_firstprivatizer %7 -> %arg2 [map_idx=0] : !llvm.struct<(ptr, i64)>) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  omp.private {type = firstprivate} @struct_firstprivatizer : !llvm.struct<(ptr, i64)> copy {
+  ^bb0(%arg0: !llvm.struct<(ptr, i64)>, %arg1: !llvm.struct<(ptr, i64)>):
+    omp.yield(%arg0 : !llvm.struct<(ptr, i64)>)
+  }
+
+  // CHECK-LABEL: llvm.func @map_from_global_gep
+  // CHECK-SAME: (%[[ARG0:.*]]: i64)
+  llvm.func @map_from_global_gep(%arg0: i64) {
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_array : !llvm.ptr
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[GLOBAL]][%[[ARG0]]] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+    %0 = llvm.mlir.addressof @global_array : !llvm.ptr
+    %1 = llvm.getelementptr %0[%arg0] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[GEP]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_a
+  // CHECK-SAME: (%[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_a() {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_b
+    llvm.call @reciprocal_b(%0) : (i64) -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_b
+  // CHECK-SAME: (%[[ARG0:.*]]: i64, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_b(%arg0 : i64) {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_a
+    llvm.call @reciprocal_a() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  llvm.func @foo() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>, sym_visibility = "private"}
+  omp.private {type = firstprivate} @privatizer : i32 copy {
+  ^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+    %0 = llvm.load %arg0 : !llvm.ptr -> i32
+    llvm.store %0, %arg1 : i32, !llvm.ptr
+    omp.yield(%arg1 : !llvm.ptr)
+  }
+  omp.declare_reduction @reduction : i32 init {
+
+  ^bb0(%arg0: i32):
+    %0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%0 : i32)
+  } combiner {
+
+  ^bb0(%arg0: i32, %arg1: i32):
+    %0 = llvm.add %arg0, %arg1 : i32
+    omp.yield(%0 : i32)
+  }
+
+  // CHECK: llvm.mlir.global internal constant @global_scalar
+  // CHECK: llvm.mlir.global internal @global_array
+  llvm.mlir.global external constant @global_scalar() {addr_space = 0 : i32} : i32
+  llvm.mlir.global external @global_array() {addr_space = 0 : i32} : !llvm.array<8 x i8>
+
+  // CHECK: llvm.mlir.global external @declare_target_enter_any
+  // CHECK: llvm.mlir.global external @declare_target_enter_host
+  // CHECK: llvm.mlir.global external @declare_target_enter_nohost
+  llvm.mlir.global external @declare_target_enter_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: llvm.mlir.global external @declare_target_link_any
+  // CHECK: llvm.mlir.global external @declare_target_link_host
+  // CHECK: llvm.mlir.global external @declare_target_link_nohost
+  llvm.mlir.global external @declare_target_link_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+
+  llvm.func @LangRTPlaceholderFunc(!llvm.ptr {llvm.nocapture}, !llvm.ptr {llvm.nocapture}, !llvm.ptr, i8 {llvm.signext}, i32) attributes {sym_visibility = "private"}
+}


From llvmlistbot at llvm.org  Thu Jul  9 03:56:41 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 09 Jul 2026 03:56:41 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (PR #208189)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208189@github.com>
Message-ID: <6a4f7e69.491d90f5.2facd.b322@mx.google.com>


https://github.com/skatrak updated https://github.com/llvm/llvm-project/pull/208189

>From e5ded32ae18ec33959eb2964bb764e549fa569b5 Mon Sep 17 00:00:00 2001
From: Sergio Afonso <Sergio.AfonsoFumero at amd.com>
Date: Thu, 2 Jul 2026 15:23:20 +0100
Subject: [PATCH] [Flang][MLIR][OpenMP] Move host op filtering to the omp
 dialect

The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.

By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.

As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
operations pointing to deleted host functions, while `fir.dispatch` ops
pointing to the same functions would remain. The `omp-function-filtering`
pass is updated to prevent `fir.dt_entry` from being deleted and let
them point to an undefined symbol. This lets FIR lowering to LLVM work
well enough to get to the host op filtering stage, where any resulting
operations for this host dynamic dispatch are cleaned up.
---
 .../include/flang/Optimizer/OpenMP/Passes.td  |  22 -
 flang/lib/Optimizer/OpenMP/CMakeLists.txt     |   1 -
 .../Optimizer/OpenMP/FunctionFiltering.cpp    |  14 +-
 .../lib/Optimizer/OpenMP/HostOpFiltering.cpp  | 422 -------------
 flang/lib/Optimizer/Passes/Pipelines.cpp      |  15 +-
 flang/test/Fir/basic-program.fir              |   2 +-
 .../Lower/OpenMP/function-filtering-4.f90     |  63 ++
 flang/test/Lower/OpenMP/host-eval.f90         |  50 +-
 .../OpenMP/function-filtering-host-ops.mlir   | 560 ------------------
 .../mlir/Dialect/OpenMP/Transforms/Passes.td  |  24 +
 .../Dialect/OpenMP/Transforms/CMakeLists.txt  |   1 +
 .../OpenMP/Transforms/HostOpFiltering.cpp     | 292 +++++++++
 .../Dialect/OpenMP/host-op-filtering.mlir     | 500 ++++++++++++++++
 13 files changed, 920 insertions(+), 1046 deletions(-)
 delete mode 100644 flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
 create mode 100644 flang/test/Lower/OpenMP/function-filtering-4.f90
 delete mode 100644 flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
 create mode 100644 mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
 create mode 100644 mlir/test/Dialect/OpenMP/host-op-filtering.mlir

diff --git a/flang/include/flang/Optimizer/OpenMP/Passes.td b/flang/include/flang/Optimizer/OpenMP/Passes.td
index b5d9372e356f1..9ec159e1ba1e0 100644
--- a/flang/include/flang/Optimizer/OpenMP/Passes.td
+++ b/flang/include/flang/Optimizer/OpenMP/Passes.td
@@ -57,28 +57,6 @@ def FunctionFilteringPass : Pass<"omp-function-filtering"> {
   ];
 }
 
-def HostOpFilteringPass : Pass<"omp-host-op-filtering"> {
-  let summary =
-      "Removes host-only operations from host functions retained by the "
-      "function filtering pass when compiling for a target device.";
-  let description = [{
-    Removes operations from host functions that are not placed inside of an
-    `omp.target` region when compiling for the target device.
-    
-    It must run after any passes that might introduce new host operations,
-    especially if they belong to the 'omp' dialect, like `LowerWorkdistribute`.
-    This ensures all potentially problematic host operations are accounted for
-    prior to LLVM IR translation.
-
-    This pass works at the FIR level of abstraction.
-  }];
-  let dependentDialects = [
-    "mlir::arith::ArithDialect",
-    "mlir::func::FuncDialect",
-    "fir::FIROpsDialect"
-  ];
-}
-
 def DoConcurrentConversionPass : Pass<"omp-do-concurrent-conversion", "mlir::ModuleOp"> {
   let summary = "Map `DO CONCURRENT` loops to OpenMP worksharing loops.";
 
diff --git a/flang/lib/Optimizer/OpenMP/CMakeLists.txt b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
index f25e95f041d20..db29e93b71dad 100644
--- a/flang/lib/Optimizer/OpenMP/CMakeLists.txt
+++ b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
@@ -5,7 +5,6 @@ add_flang_library(FlangOpenMPTransforms
   DoConcurrentConversion.cpp
   FunctionFiltering.cpp
   GenericLoopConversion.cpp
-  HostOpFiltering.cpp
   MapsForPrivatizedSymbols.cpp
   MapInfoFinalization.cpp
   DeleteUnreachableTargets.cpp
diff --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index e8bcc0c7fa062..ee61dd8881fd0 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -48,8 +48,7 @@ checkDeviceImplementationStatus(omp::OffloadModuleInterface offloadModule) {
     if (!redOp.getByrefElementType())
       return WalkResult::advance();
 
-    auto seqTy =
-        mlir::dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
+    auto seqTy = dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
 
     bool isByRefReductionSupported =
         !seqTy || !fir::sequenceWithNonConstantShape(seqTy);
@@ -104,7 +103,7 @@ class FunctionFilteringPass
         SymbolTable::UseRange funcUses = *funcOp.getSymbolUses(op);
         for (SymbolTable::SymbolUse use : funcUses) {
           Operation *callOp = use.getUser();
-          if (auto internalFunc = mlir::dyn_cast<func::FuncOp>(callOp)) {
+          if (auto internalFunc = dyn_cast<func::FuncOp>(callOp)) {
             // Do not delete internal procedures holding the symbol of their
             // Fortran host procedure as attribute.
             internalFunc->removeAttr(fir::getHostSymbolAttrName());
@@ -114,6 +113,15 @@ class FunctionFilteringPass
             internalFunc.setVisibility(mlir::SymbolTable::Visibility::Public);
             continue;
           }
+          // Prevent dispatch table entries pointing to deleted functions
+          // from being removed. This prevents the lowering of any
+          // corresponding fir.dispatch ops from triggering errors. These
+          // fir.dt_entry ops will point to an undefined symbol as a result,
+          // which currently doesn't cause an issue, as fir.dispatch-related ops
+          // are later removed by the host op filtering pass.
+          if (isa<fir::DTEntryOp>(callOp))
+            continue;
+
           // If the callOp has users then replace them with Undef values.
           if (!callOp->use_empty()) {
             SmallVector<Value> undefResults;
diff --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
deleted file mode 100644
index 05ea0bc08c4ab..0000000000000
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ /dev/null
@@ -1,422 +0,0 @@
-//===- HostOpFiltering.cpp -----------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements transforms to filter out host-only operations from
-// any remaining host functions when compiling for a target device.
-//
-//===----------------------------------------------------------------------===//
-
-#include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Dialect/FIROps.h"
-#include "flang/Optimizer/OpenMP/Passes.h"
-
-#include "mlir/Dialect/Arith/IR/Arith.h"
-#include "mlir/Dialect/Func/IR/FuncOps.h"
-#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
-#include "llvm/ADT/SetVector.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/ADT/TypeSwitch.h"
-
-namespace flangomp {
-#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
-#include "flang/Optimizer/OpenMP/Passes.h.inc"
-} // namespace flangomp
-
-using namespace mlir;
-
-/// Add an operation to one of the output sets to be later rewritten.
-template <typename OpTy>
-static void collectRewrite(OpTy op, llvm::SetVector<OpTy> &rewrites) {
-  rewrites.insert(op);
-}
-
-/// Add an \c omp.map.info operation and all its members recursively to the
-/// output set to be later rewritten.
-///
-/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
-/// are added to the output sets before operations based on them.
-template <>
-void collectRewrite(omp::MapInfoOp mapOp,
-                    llvm::SetVector<omp::MapInfoOp> &rewrites) {
-  for (Value member : mapOp.getMembers())
-    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
-
-  rewrites.insert(mapOp);
-}
-
-/// Add the given value to a sorted set if it should be replaced by a
-/// placeholder when used as an operand that must remain for the device.
-///
-/// Values that are block arguments of \c func.func operations are skipped,
-/// since they will still be available after all rewrites are completed.
-static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
-  if ((isa<BlockArgument>(value) &&
-       isa<func::FuncOp>(
-           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
-      rewrites.contains(value))
-    return;
-
-  rewrites.insert(value);
-}
-
-/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
-static std::optional<omp::DeclareTargetDeviceType>
-getDeclareTargetDevice(Operation &op) {
-  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
-  if (declareTargetOp && declareTargetOp.isDeclareTarget())
-    return declareTargetOp.getDeclareTargetDeviceType();
-  return std::nullopt;
-}
-
-namespace {
-class HostOpFilteringPass
-    : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
-public:
-  HostOpFilteringPass() = default;
-
-  void runOnOperation() override {
-    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
-    if (!op || !op.getIsTargetDevice())
-      return;
-
-    op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
-      omp::DeclareTargetDeviceType declareType =
-          getDeclareTargetDevice(*funcOp.getOperation())
-              .value_or(omp::DeclareTargetDeviceType::host);
-
-      // Only process host function definitions.
-      if (funcOp.isExternal() ||
-          declareType != omp::DeclareTargetDeviceType::host)
-        return WalkResult::advance();
-
-      if (failed(rewriteHostFunction(funcOp))) {
-        funcOp.emitOpError() << "could not filter host-only operations";
-        return WalkResult::interrupt();
-      }
-      return WalkResult::advance();
-    });
-
-    // Make non-declare target globals internal for the device. They cannot be
-    // deleted, because they are needed in order to properly lower map clauses.
-    // However, no uses will remain in the device module, so we make them
-    // internal to prevent link time issues.
-    op->walk([&](fir::GlobalOp globalOp) {
-      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
-        globalOp.setLinkName("internal");
-    });
-  }
-
-private:
-  /// Rewrite the given host device function containing \c omp.target
-  /// operations, to remove host-only operations that are not used by device
-  /// codegen.
-  ///
-  /// It is based on the expected form of the MLIR module as produced by Flang
-  /// lowering, after HLFIR to FIR lowering, and it performs the following
-  /// mutations:
-  ///   - Replace all values returned by the function with \c fir.undefined.
-  ///   - \c omp.target operations are moved to the end of the function. If they
-  ///     are nested inside of any other operations, they are hoisted out of
-  ///     them.
-  ///   - \c depend, \c device and \c if clauses are removed from these target
-  ///     functions. Values used to initialize other clauses are replaced by
-  ///     placeholders as follows:
-  ///     - Values defined by block arguments are replaced by placeholders only
-  ///       if they are not attached to the parent \c func.func operation. In
-  ///       that case, they are passed unmodified.
-  ///     - \c arith.constant and \c fir.address_of ops are maintained.
-  ///     - Values of type \c fir.boxchar are replaced with a combination of
-  ///       \c fir.alloca for a single bit and a \c fir.emboxchar.
-  ///     - Other values are replaced by a combination of an \c fir.alloca for a
-  ///       single bit and an \c fir.convert to the original type of the value.
-  ///       This can be done because the code eventually generated for these
-  ///       operations will be discarded, as they aren't runnable by the target
-  ///       device.
-  ///   - \c omp.map.info operations associated to these target regions are
-  ///     preserved. These are moved above all \c omp.target and sorted to
-  ///     satisfy dependencies among them.
-  ///   - \c bounds arguments are removed from \c omp.map.info operations.
-  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
-  ///     handled as follows:
-  ///     - \c var_ptr_ptr is expected to be defined by a \c fir.box_offset
-  ///       operation which is preserved. Otherwise, the pass will fail.
-  ///     - \c var_ptr can be defined by a \c fir.declare which is also
-  ///       preserved. Its \c memref argument is replaced by a placeholder or
-  ///       maintained, similarly to non-map clauses of target operations
-  ///       described above. If it has \c shape or \c typeparams arguments, they
-  ///       are replaced by applicable constants. \c dummy_scope arguments
-  ///       are discarded.
-  ///   - Every other operation not located inside of an \c omp.target is
-  ///     removed.
-  LogicalResult rewriteHostFunction(func::FuncOp funcOp) {
-    Region &region = funcOp.getRegion();
-
-    // Collect target operations inside of the function.
-    llvm::SmallVector<omp::TargetOp> targetOps;
-    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
-      // Skip the inside of omp.target regions, since these contain device code.
-      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
-        targetOps.push_back(targetOp);
-        return WalkResult::skip();
-      }
-
-      // Replace omp.target_data entry block argument uses with the value used
-      // to initialize the associated omp.map.info operation. This way,
-      // references are still valid once the omp.target operation has been
-      // extracted out of the omp.target_data region.
-      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
-        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
-        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
-            .getBlockArgsPairs(argPairs);
-        for (auto [operand, blockArg] : argPairs) {
-          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
-          Value varPtr = mapInfo.getVarPtr();
-
-          // If the var_ptr operand of the omp.map.info op defining this entry
-          // block argument is a fir.declare, the uses of all users of that
-          // entry block argument that are themselves fir.declare are replaced
-          // by the value produced by the outer one.
-          //
-          // This prevents this pass from producing chains of fir.declare of the
-          // type:
-          // %0 = ...
-          // %1 = fir.declare %0
-          // %2 = fir.declare %1...
-          // %3 = omp.map.info var_ptr(%2 ...
-          if (auto outerDeclare = varPtr.getDefiningOp<fir::DeclareOp>())
-            for (Operation *user : blockArg.getUsers())
-              if (isa<fir::DeclareOp>(user))
-                user->replaceAllUsesWith(outerDeclare);
-
-          // All remaining uses of the entry block argument are replaced with
-          // the var_ptr initialization value.
-          blockArg.replaceAllUsesWith(varPtr);
-        }
-      }
-      return WalkResult::advance();
-    });
-
-    // Make a temporary clone of the parent operation with an empty region,
-    // and update all references to entry block arguments to those of the new
-    // region. Users will later either be moved to the new region or deleted
-    // when the original region is replaced by the new.
-    OpBuilder builder(&getContext());
-    builder.setInsertionPointAfter(funcOp);
-    Operation *newOp = builder.cloneWithoutRegions(funcOp);
-    Block &block = newOp->getRegion(0).emplaceBlock();
-
-    llvm::SmallVector<Location> locs;
-    locs.reserve(region.getNumArguments());
-    llvm::transform(region.getArguments(), std::back_inserter(locs),
-                    [](const BlockArgument &arg) { return arg.getLoc(); });
-    block.addArguments(region.getArgumentTypes(), locs);
-
-    for (auto [oldArg, newArg] :
-         llvm::zip_equal(region.getArguments(), block.getArguments()))
-      oldArg.replaceAllUsesWith(newArg);
-
-    // Collect omp.map.info ops while satisfying interdependencies and remove
-    // operands that aren't used by target device codegen.
-    //
-    // This logic must be updated whenever operands to omp.target change.
-    llvm::SetVector<Value> rewriteValues;
-    llvm::SetVector<omp::MapInfoOp> mapInfos;
-    for (omp::TargetOp targetOp : targetOps) {
-      assert(targetOp.getHostEvalVars().empty() &&
-             "unexpected host_eval in target device module");
-
-      // Variables unused by the device.
-      targetOp.getDependVarsMutable().clear();
-      targetOp.setDependKindsAttr(nullptr);
-      targetOp.getDependIteratedMutable().clear();
-      targetOp.setDependIteratedKindsAttr(nullptr);
-      targetOp.getDeviceMutable().clear();
-      targetOp.getDynGroupprivateSizeMutable().clear();
-      targetOp.getIfExprMutable().clear();
-
-      // TODO: Clear some of these operands rather than rewriting them,
-      // depending on whether they are needed by device codegen once support for
-      // them is fully implemented.
-      for (Value allocVar : targetOp.getAllocateVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value allocVar : targetOp.getAllocatorVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value inReduction : targetOp.getInReductionVars())
-        collectRewrite(inReduction, rewriteValues);
-      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
-        collectRewrite(isDevPtr, rewriteValues);
-      for (Value mapVar : targetOp.getHasDeviceAddrVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value mapVar : targetOp.getMapVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value privateVar : targetOp.getPrivateVars())
-        collectRewrite(privateVar, rewriteValues);
-      for (Value threadLimit : targetOp.getThreadLimitVars())
-        collectRewrite(threadLimit, rewriteValues);
-    }
-
-    // Move omp.map.info ops to the new block and collect dependencies.
-    llvm::SetVector<fir::DeclareOp> declareOps;
-    llvm::SetVector<fir::BoxOffsetOp> boxOffsets;
-    for (omp::MapInfoOp mapOp : mapInfos) {
-      if (auto declareOp = dyn_cast_if_present<fir::DeclareOp>(
-              mapOp.getVarPtr().getDefiningOp()))
-        collectRewrite(declareOp, declareOps);
-      else
-        collectRewrite(mapOp.getVarPtr(), rewriteValues);
-
-      if (Value varPtrPtr = mapOp.getVarPtrPtr()) {
-        if (auto boxOffset = llvm::dyn_cast_if_present<fir::BoxOffsetOp>(
-                varPtrPtr.getDefiningOp()))
-          collectRewrite(boxOffset, boxOffsets);
-        else
-          return mapOp->emitOpError() << "var_ptr_ptr rewrite only supported "
-                                         "if defined by fir.box_offset";
-      }
-
-      // Bounds are not used during target device codegen.
-      mapOp.getBoundsMutable().clear();
-      mapOp->moveBefore(&block, block.end());
-    }
-
-    // Create a temporary marker to simplify the op moving process below.
-    builder.setInsertionPointToStart(&block);
-    auto marker = fir::UndefOp::create(builder, builder.getUnknownLoc(),
-                                       builder.getNoneType());
-    builder.setInsertionPoint(marker);
-
-    // Handle dependencies of fir.declare ops.
-    for (fir::DeclareOp declareOp : declareOps) {
-      collectRewrite(declareOp.getMemref(), rewriteValues);
-
-      if (declareOp.getStorage())
-        collectRewrite(declareOp.getStorage(), rewriteValues);
-
-      // Shape and typeparams aren't needed for target device codegen, but
-      // removing them would break verifiers.
-      Value zero;
-      if (declareOp.getShape() || !declareOp.getTypeparams().empty())
-        zero = arith::ConstantOp::create(builder, declareOp.getLoc(),
-                                         builder.getI64IntegerAttr(0));
-
-      if (auto shape = declareOp.getShape()) {
-        // The pre-cg rewrite pass requires the shape to be defined by one of
-        // fir.shape, fir.shapeshift or fir.shift, so we need to make sure it's
-        // still defined by one of these after this pass.
-        Operation *shapeOp = shape.getDefiningOp();
-        llvm::SmallVector<Value> extents(shapeOp->getNumOperands(), zero);
-        Value newShape =
-            llvm::TypeSwitch<Operation *, Value>(shapeOp)
-                .Case([&](fir::ShapeOp op) {
-                  return fir::ShapeOp::create(builder, op.getLoc(), extents);
-                })
-                .Case([&](fir::ShapeShiftOp op) {
-                  auto type = fir::ShapeShiftType::get(op.getContext(),
-                                                       extents.size() / 2);
-                  return fir::ShapeShiftOp::create(builder, op.getLoc(), type,
-                                                   extents);
-                })
-                .Case([&](fir::ShiftOp op) {
-                  auto type =
-                      fir::ShiftType::get(op.getContext(), extents.size());
-                  return fir::ShiftOp::create(builder, op.getLoc(), type,
-                                              extents);
-                })
-                .Default([](Operation *op) {
-                  op->emitOpError()
-                      << "fir.declare shape expected to be one of: "
-                         "fir.shape, fir.shapeshift or fir.shift";
-                  return nullptr;
-                });
-
-        if (!newShape)
-          return failure();
-
-        declareOp.getShapeMutable().assign(newShape);
-      }
-
-      for (OpOperand &typeParam : declareOp.getTypeparamsMutable())
-        typeParam.assign(zero);
-
-      declareOp.getDummyScopeMutable().clear();
-    }
-
-    // We don't actually need the proper initialization, but rather just
-    // maintain the basic form of these operands. Generally, we create 1-bit
-    // placeholder allocas that we "typecast" to the expected type and replace
-    // all uses. Using fir.undefined here instead is not possible because these
-    // variables cannot be constants, as that would trigger different codegen
-    // for target regions.
-    for (Value value : rewriteValues) {
-      Location loc = value.getLoc();
-      Value rewriteValue;
-      if (isa_and_present<arith::ConstantOp, fir::AddrOfOp>(
-              value.getDefiningOp())) {
-        // If it's defined by fir.address_of, then we need to keep that op as
-        // well because it might be pointing to a 'declare target' global.
-        // Constants can also trigger different codegen paths, so we keep them
-        // as well.
-        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
-      } else if (auto boxCharType =
-                     dyn_cast<fir::BoxCharType>(value.getType())) {
-        // !fir.boxchar types cannot be directly obtained by converting a
-        // !fir.ref<i1>, as they aren't reference types. Since they can appear
-        // representing some `target firstprivate` clauses, we need to create
-        // a special case here based on creating a placeholder fir.emboxchar op.
-        MLIRContext *ctx = &getContext();
-        fir::KindTy kind = boxCharType.getKind();
-        auto placeholder = fir::AllocaOp::create(
-            builder, loc, fir::CharacterType::getSingleton(ctx, kind));
-        auto one = arith::ConstantOp::create(builder, loc, builder.getI32Type(),
-                                             builder.getI32IntegerAttr(1));
-        rewriteValue = fir::EmboxCharOp::create(builder, loc, boxCharType,
-                                                placeholder, one);
-      } else {
-        Value placeholder =
-            fir::AllocaOp::create(builder, loc, builder.getI1Type());
-        rewriteValue =
-            fir::ConvertOp::create(builder, loc, value.getType(), placeholder);
-      }
-      value.replaceAllUsesWith(rewriteValue);
-    }
-
-    // Move omp.map.info dependencies.
-    for (fir::DeclareOp declareOp : declareOps)
-      declareOp->moveBefore(marker);
-
-    // The box_ref argument of fir.box_offset is expected to be the same value
-    // that was passed as var_ptr to the corresponding omp.map.info, so we don't
-    // need to handle its defining op here.
-    for (fir::BoxOffsetOp boxOffset : boxOffsets)
-      boxOffset->moveBefore(marker);
-
-    marker->erase();
-
-    // Move target operations to the end of the new block.
-    for (omp::TargetOp targetOp : targetOps)
-      targetOp->moveBefore(&block, block.end());
-
-    // Add terminator to the new block.
-    builder.setInsertionPointToEnd(&block);
-    llvm::SmallVector<Value> returnValues;
-    returnValues.reserve(funcOp.getNumResults());
-    for (auto type : funcOp.getResultTypes())
-      returnValues.push_back(
-          fir::UndefOp::create(builder, funcOp.getLoc(), type));
-
-    func::ReturnOp::create(builder, funcOp.getLoc(), returnValues);
-
-    // Replace old region (now missing ops) with the new one and remove the
-    // temporary operation clone.
-    region.takeBody(newOp->getRegion(0));
-    newOp->erase();
-    return success();
-  }
-};
-} // namespace
diff --git a/flang/lib/Optimizer/Passes/Pipelines.cpp b/flang/lib/Optimizer/Passes/Pipelines.cpp
index a8e598ab95ac8..5fa70096e0e41 100644
--- a/flang/lib/Optimizer/Passes/Pipelines.cpp
+++ b/flang/lib/Optimizer/Passes/Pipelines.cpp
@@ -352,7 +352,6 @@ void createHLFIRToFIRPassPipeline(mlir::PassManager &pm,
   if (enableOpenMP != EnableOpenMP::None) {
     pm.addPass(flangomp::createLowerWorkshare());
     pm.addPass(flangomp::createLowerWorkdistribute());
-    pm.addPass(flangomp::createHostOpFilteringPass());
   }
   if (enableOpenMP == EnableOpenMP::Simd)
     pm.addPass(flangomp::createSimdOnlyPass());
@@ -462,11 +461,17 @@ void createDefaultFIRCodeGenPassPipeline(mlir::PassManager &pm,
 
   fir::addFIRToLLVMPass(pm, config);
 
-  // Convert applicable OpenMP stack allocations to shared memory allocations
-  // for GPU targets. This pass must run after any alloca-generating passes to
-  // ensure all are adequately accounted for.
-  if (config.EnableOpenMP && !config.EnableOpenMPSimd)
+  if (config.EnableOpenMP && !config.EnableOpenMPSimd) {
+    // Remove all non target-related operations from host functions still
+    // remaining at this point, if compiling for an OpenMP target device. This
+    // is required before translating 'omp' dialect operations to LLVM IR.
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
+
+    // Convert applicable OpenMP stack allocations to shared memory allocations
+    // for GPU targets. This pass must run after any alloca-generating passes to
+    // ensure all are adequately accounted for.
     pm.addPass(mlir::omp::createStackToSharedPass());
+  }
 }
 
 /// Create a pass pipeline for lowering from MLIR to LLVM IR
diff --git a/flang/test/Fir/basic-program.fir b/flang/test/Fir/basic-program.fir
index cbf45b216a911..471e5a89046eb 100644
--- a/flang/test/Fir/basic-program.fir
+++ b/flang/test/Fir/basic-program.fir
@@ -85,7 +85,6 @@ func.func @_QQmain() {
 // PASSES-NEXT:   ConvertHLFIRtoFIR
 // PASSES-NEXT:   LowerWorkshare
 // PASSES-NEXT:   LowerWorkdistribute
-// PASSES-NEXT:   HostOpFilteringPass
 // PASSES-NEXT:   CSE
 // PASSES-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 // PASSES-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
@@ -185,6 +184,7 @@ func.func @_QQmain() {
 // PASSES-NEXT:  LowerNontemporalPass
 // PASSES-NEXT: FIRToLLVMLowering
 // PASSES-NEXT: ReconcileUnrealizedCasts
+// PASSES-NEXT:  HostOpFilteringPass
 // PASSES-NEXT: 'llvm.func' Pipeline
 // PASSES-NEXT:  StackToSharedPass
 // PASSES-NEXT: PrepareForOMPOffloadPrivatizationPass
diff --git a/flang/test/Lower/OpenMP/function-filtering-4.f90 b/flang/test/Lower/OpenMP/function-filtering-4.f90
new file mode 100644
index 0000000000000..de8a26c789dc5
--- /dev/null
+++ b/flang/test/Lower/OpenMP/function-filtering-4.f90
@@ -0,0 +1,63 @@
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-HOST,LLVM-ALL %s
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-DEVICE,LLVM-ALL %s %}
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+
+! Check that the correct LLVM IR operations are kept for the host and device
+! after running the whole set of translation and transformation passes from
+! Fortran for a type-bound procedure.
+
+module mymodule
+  implicit none
+
+  type :: myclass
+  contains
+    procedure :: myfunc => myfunc
+  end type myclass
+
+contains
+  ! MLIR-HOST: func.func @_QMmymodulePmyfunc
+  ! MLIR-DEVICE-NOT: func.func @_QMmymodulePmyfunc
+
+  ! LLVM-HOST: define void @_QMmymodulePmyfunc
+  ! LLVM-DEVICE-NOT: define void @_QMmymodulePmyfunc
+  subroutine myfunc(self)
+    class(myclass) :: self
+    call foo()
+  end subroutine
+end module
+
+! MLIR-ALL: func.func @_QPmain
+
+! LLVM-HOST: define void @main_
+! LLVM-DEVICE-NOT: define void @main_
+subroutine main(x)
+  use mymodule
+  implicit none
+
+  integer, intent(inout) :: x
+  class(myclass), allocatable :: myobj
+  allocate(myobj)
+
+  ! MLIR-ALL: fir.dispatch "myfunc"
+
+  ! LLVM-HOST: %[[MYFUNC_PTR:.*]] = inttoptr i64 %{{.*}} to ptr
+  ! LLVM-HOST: call void %[[MYFUNC_PTR]](ptr %{{.*}})
+  ! LLVM-DEVICE-NOT: call void %{{.*}}(ptr %{{.*}})
+  call myobj%myfunc()
+
+  !$omp target map(tofrom: x)
+  x = x + 1
+  !$omp end target
+
+  deallocate(myobj)
+
+  ! LLVM-HOST: ret void
+end subroutine main
+
+! LLVM-ALL: define {{.*}}void @__omp_offloading{{.*}}main_{{.*}}
+
+! MLIR-ALL: fir.type_info @_QMmymoduleTmyclass {{.*}} dispatch_table
+! MLIR-ALL-NEXT: fir.dt_entry "myfunc", @_QMmymodulePmyfunc
diff --git a/flang/test/Lower/OpenMP/host-eval.f90 b/flang/test/Lower/OpenMP/host-eval.f90
index 3bd873009530c..35749a6ac5050 100644
--- a/flang/test/Lower/OpenMP/host-eval.f90
+++ b/flang/test/Lower/OpenMP/host-eval.f90
@@ -22,10 +22,8 @@ subroutine teams()
 
   !$omp end target
 
-  ! HOST: omp.teams
-  ! HOST-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
-
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
+  ! BOTH-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
   !$omp teams num_teams(1) thread_limit(2)
   call foo()
   !$omp end teams
@@ -78,18 +76,13 @@ subroutine distribute_parallel_do()
   !$omp end distribute parallel do
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-
-  ! DEVICE-NOT: omp.parallel
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.wsloop
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
   !$omp distribute parallel do num_threads(1)
   do i=1,10
     call foo()
@@ -147,15 +140,14 @@ subroutine distribute_parallel_do_simd()
   !$omp end distribute parallel do simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-  ! HOST-NEXT: omp.simd
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
+  ! BOTH-NEXT: omp.simd
 
   ! DEVICE-NOT: omp.parallel
   ! DEVICE-NOT: omp.distribute
@@ -207,12 +199,10 @@ subroutine distribute()
   !$omp end distribute
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! DEVICE-NOT: omp.distribute
+  ! BOTH: omp.distribute
   !$omp distribute
   do i=1,10
     call foo()
@@ -261,15 +251,11 @@ subroutine distribute_simd()
   !$omp end distribute simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.simd
-
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.simd
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.simd
   !$omp distribute simd
   do i=1,10
     call foo()
diff --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
deleted file mode 100644
index 3acdfda876812..0000000000000
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ /dev/null
@@ -1,560 +0,0 @@
-// RUN: fir-opt --omp-host-op-filtering %s | FileCheck %s
-
-module attributes {omp.is_target_device = true} {
-  // CHECK-LABEL: func.func private @external
-  // CHECK-NOT: return
-  func.func private @external()
-
-  // CHECK-LABEL: func.func @basic_checks
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.ref<i32>) -> (i32, f32)
-  func.func @basic_checks(%arg: !fir.ref<i32>) -> (i32, f32) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOC:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %r0 = arith.constant 10 : i32
-    %r1 = arith.constant 2.5 : f32
-
-    func.call @foo() : () -> ()
-
-    %0 = fir.declare %arg {uniq_name = "arg"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %alloc = fir.alloca i32
-    %2 = fir.declare %alloc {uniq_name = "alloc"} : (!fir.ref<i32>) -> !fir.ref<i32>
-
-    // CHECK-NEXT: %[[ALLOC_DECL:.*]] = fir.declare %[[ALLOC]] {uniq_name = "alloc"}
-    // CHECK-NEXT: %[[ARG_DECL:.*]] = fir.declare %[[ARG]] {uniq_name = "arg"}
-    // CHECK-NEXT: %[[GLOBAL_DECL:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOC_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP3:.*]] = omp.map.info var_ptr(%[[ALLOC]]{{.*}})
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[MAP2]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP3]] -> {{.*}} : {{.*}})
-    omp.target kernel_type(generic) has_device_addr(%m2 -> %arg0 : !fir.ref<i32>) map_entries(%m0 -> %arg1, %m1 -> %arg2, %m3 -> %arg3 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NEXT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: func.call
-    // CHECK-NOT: omp.map.info
-    omp.parallel {
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.map.info
-    %m4 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m5 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m6 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m7 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m4, %m5, %m6, %m7 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.target_enter_data
-    // CHECK-NOT: omp.target_exit_data
-    // CHECK-NOT: omp.target_update
-    %m8 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_enter_data map_entries(%m8 : !fir.ref<i32>)
-
-    %m9 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(from) capture(ByRef) -> !fir.ref<i32>
-    omp.target_exit_data map_entries(%m9 : !fir.ref<i32>)
-
-    %m10 = omp.map.info var_ptr(%2 : !fir.ref<i32>, !fir.ref<i32>) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_update map_entries(%m10 : !fir.ref<i32>)
-
-    // CHECK-NOT: func.call
-    func.call @foo() : () -> ()
-
-    // CHECK:      %[[RETURN0:.*]] = fir.undefined i32
-    // CHECK-NEXT: %[[RETURN1:.*]] = fir.undefined f32
-    // CHECK-NEXT: return %[[RETURN0]], %[[RETURN1]]
-    return %r0, %r1 : i32, f32
-  }
-
-  // CHECK-LABEL: func.func @allocatable_array
-  // CHECK-SAME: (%[[ALLOCATABLE:.*]]: [[ALLOCATABLE_TYPE:.*]], %[[ARRAY:.*]]: [[ARRAY_TYPE:[^)]*]])
-  func.func @allocatable_array(%allocatable: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, %array: !fir.ref<!fir.array<9xi32>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape %[[ZERO]] : (i64) -> !fir.shape<1>
-    // CHECK-NEXT: %[[ALLOCATABLE_DECL:.*]] = fir.declare %[[ALLOCATABLE]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : ([[ALLOCATABLE_TYPE]]) -> [[ALLOCATABLE_TYPE]]
-    // CHECK-NEXT: %[[ARRAY_DECL:.*]] = fir.declare %[[ARRAY]](%[[SHAPE]]) {uniq_name = "array"} : ([[ARRAY_TYPE]], !fir.shape<1>) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCATABLE_DECL]] base_addr : ([[ALLOCATABLE_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP_ALLOCATABLE:.*]] = omp.map.info var_ptr(%[[ALLOCATABLE_DECL]] : [[ALLOCATABLE_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP_ARRAY:.*]] = omp.map.info var_ptr(%[[ARRAY_DECL]] : [[ARRAY_TYPE]], !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP_ALLOCATABLE]] -> %{{.*}}, %[[MAP_ARRAY]] -> %{{.*}} : [[VAR_PTR_PTR_TYPE]], [[ARRAY_TYPE]])
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c8 = arith.constant 8 : index
-    %c9 = arith.constant 9 : index
-
-    %0 = fir.declare %allocatable {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %1 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %2 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%2 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%1) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-
-    %3 = fir.shape %c9 : (index) -> !fir.shape<1>
-    %4 = fir.declare %array(%3) {uniq_name = "array"} : (!fir.ref<!fir.array<9xi32>>, !fir.shape<1>) -> !fir.ref<!fir.array<9xi32>>
-    %5 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %6 = omp.map.info var_ptr(%4 : !fir.ref<!fir.array<9xi32>>, !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) bounds(%5) -> !fir.ref<!fir.array<9xi32>>
-
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg0, %6 -> %arg1 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, !fir.ref<!fir.array<9xi32>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @character
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @character(%x: !fir.ref<!fir.char<1>>) {
-    // CHECK-NEXT: %[[ZERO]] = arith.constant 0 : i64
-    %0 = fir.dummy_scope : !fir.dscope
-    %c1 = arith.constant 1 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] typeparams %[[ZERO]] {uniq_name = "x"} : ([[X_TYPE]], i64) -> [[X_TYPE]]
-    %3 = fir.declare %x typeparams %c1 dummy_scope %0 {uniq_name = "x"} : (!fir.ref<!fir.char<1>>, index, !fir.dscope) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> [[X_TYPE]]
-    %map = omp.map.info var_ptr(%3 : !fir.ref<!fir.char<1>>, !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}})
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<!fir.char<1>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_rank
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @assumed_rank(%x: !fir.box<!fir.array<*:f32>>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<[[X_TYPE]]>
-    %0 = fir.alloca !fir.box<!fir.array<*:f32>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {uniq_name = "x"} : (!fir.box<!fir.array<*:f32>>, !fir.dscope) -> !fir.box<!fir.array<*:f32>>
-    %3 = fir.rebox_assumed_rank %2 lbs ones : (!fir.box<!fir.array<*:f32>>) -> !fir.box<!fir.array<*:f32>>
-    %4 = fir.box_addr %3 : (!fir.box<!fir.array<*:f32>>) -> !fir.ref<!fir.array<*:f32>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA]] base_addr : (!fir.ref<[[X_TYPE]]>) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %5 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.array<*:f32>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], !fir.array<*:f32>) -> [[VAR_PTR_PTR_TYPE]]
-    %6 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%5 : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>, !fir.array<*:f32>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> !fir.ref<!fir.array<*:f32>>
-    %7 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(to) capture(ByRef) members(%6 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) -> !fir.ref<!fir.array<*:f32>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> {{.*}})
-    omp.target kernel_type(generic) map_entries(%7 -> %arg1, %6 -> %arg2 : !fir.ref<!fir.array<*:f32>>, !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @box_ptr
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @box_ptr(%x: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape_shift %[[ZERO]], %[[ZERO]] : (i64, i64) -> !fir.shapeshift<1>
-    // CHECK-NEXT: %[[PLACEHOLDER_X:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_X:.*]] = fir.convert %[[PLACEHOLDER_X]] : (!fir.ref<i1>) -> [[X_TYPE]]
-    %0 = fir.alloca !fir.box<!fir.ptr<!fir.array<?xi32>>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {fortran_attrs = #fir.var_attrs<contiguous, pointer>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.dscope) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    %3 = fir.load %2 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-
-    // CHECK-NEXT: %[[PLACEHOLDER_Y:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_Y:.*]] = fir.convert %[[PLACEHOLDER_Y]] : (!fir.ref<i1>) -> [[Y_TYPE:.*]]
-    %c0 = arith.constant 0 : index
-    %4:3 = fir.box_dims %3, %c0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c1 = arith.constant 1 : index
-    %c0_0 = arith.constant 0 : index
-    %5:3 = fir.box_dims %3, %c0_0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c0_1 = arith.constant 0 : index
-    %6 = arith.subi %5#1, %c1 : index
-    %7 = omp.map.bounds lower_bound(%c0_1 : index) upper_bound(%6 : index) extent(%5#1 : index) stride(%5#2 : index) start_idx(%4#0 : index) {stride_in_bytes = true}
-    %8 = fir.box_addr %3 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>) -> !fir.ptr<!fir.array<?xi32>>
-    %c0_2 = arith.constant 0 : index
-    %9:3 = fir.box_dims %3, %c0_2 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %10 = fir.shape_shift %9#0, %9#1 : (index, index) -> !fir.shapeshift<1>
-    
-    // CHECK-NEXT: %[[Y_DECL:.*]] = fir.declare %[[ALLOCA_Y]](%[[SHAPE]]) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : ([[Y_TYPE]], !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %11 = fir.declare %8(%10) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %12 = fir.embox %11(%10) : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.box<!fir.array<?xi32>>
-    %c1_3 = arith.constant 1 : index
-    %c0_4 = arith.constant 0 : index
-    %13:3 = fir.box_dims %12, %c0_4 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
-    %c0_5 = arith.constant 0 : index
-    %14 = arith.subi %13#1, %c1_3 : index
-    %15 = omp.map.bounds lower_bound(%c0_5 : index) upper_bound(%14 : index) extent(%13#1 : index) stride(%13#2 : index) start_idx(%9#0 : index) {stride_in_bytes = true}
-    
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA_X]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[Y_DECL]] : [[Y_TYPE]], i32) {{.*}} -> [[Y_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], i32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} members(%[[MAP1]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %16 = omp.map.info var_ptr(%11 : !fir.ptr<!fir.array<?xi32>>, i32) map_clauses(tofrom) capture(ByRef) bounds(%15) -> !fir.ptr<!fir.array<?xi32>>
-    %17 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %18 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) var_ptr_ptr(%17 : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) bounds(%7) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %19 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) members(%18 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP2]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : [[Y_TYPE]], [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%16 -> %arg1, %19 -> %arg2, %18 -> %arg3 : !fir.ptr<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @target_data
-  // CHECK-SAME: (%[[MAPPED:.*]]: [[MAPPED_TYPE:[^)]*]], %[[USEDEVADDR:.*]]: [[USEDEVADDR_TYPE:[^)]*]], %[[USEDEVPTR:.*]]: [[USEDEVPTR_TYPE:[^)]*]])
-  func.func @target_data(%mapped: !fir.ref<i32>, %usedevaddr: !fir.ref<i32>, %usedevptr: !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-    // CHECK-NEXT: %[[MAPPED_DECL:.*]] = fir.declare %[[MAPPED]] {uniq_name = "mapped"} : ([[MAPPED_TYPE]]) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_DECL:.*]] = fir.declare %[[USEDEVADDR]] {uniq_name = "usedevaddr"} : ([[USEDEVADDR_TYPE]]) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_DECL:.*]] = fir.declare %[[USEDEVPTR]] {uniq_name = "usedevptr"} : ([[USEDEVPTR_TYPE]]) -> [[USEDEVPTR_TYPE]]
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %1 = fir.declare %usedevaddr {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %2 = fir.declare %usedevptr {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-    // CHECK-NEXT: %[[MAPPED_MAP:.*]] = omp.map.info var_ptr(%[[MAPPED_DECL]] : [[MAPPED_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVADDR_DECL]] : [[USEDEVADDR_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVPTR_DECL]] : [[USEDEVPTR_TYPE]], !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> [[USEDEVPTR_TYPE]]
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(return_param) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(return_param) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-    
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) use_device_addr(%m1 -> %arg0 : !fir.ref<i32>) use_device_ptr(%m2 -> %arg1 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-      %3 = fir.declare %arg0 {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %4 = fir.declare %arg1 {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-      %m3 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m4 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m5 = omp.map.info var_ptr(%4 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAPPED_MAP]] -> %{{.*}}, %[[USEDEVADDR_MAP]] -> %{{.*}}, %[[USEDEVPTR_MAP]] -> %{{.*}} : {{.*}})
-      omp.target kernel_type(generic) map_entries(%m3 -> %arg2, %m4 -> %arg3, %m5 -> %arg4 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-        omp.terminator
-      }
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      omp.terminator
-    }
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @no_target
-  // CHECK-SAME: (%[[MAPPED:.*]]: !fir.ref<i32>)
-  func.func @no_target(%mapped: !fir.ref<i32>) {
-    // CHECK-NEXT: return
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        func.call @foo() : () -> ()
-        omp.terminator
-      }
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @map_info_members
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @map_info_members(%x: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) {
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c9 = arith.constant 9 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    %23 = fir.declare %x {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %63 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %64:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %65:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %66 = arith.subi %c1, %64#0 : index
-    %67 = arith.subi %c9, %64#0 : index
-    %68 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %69:3 = fir.box_dims %68, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %70 = omp.map.bounds lower_bound(%66 : index) upper_bound(%67 : index) extent(%69#1 : index) stride(%65#2 : index) start_idx(%64#0 : index) {stride_in_bytes = true}
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[X_DECL]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %71 = fir.box_offset %23 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    %72 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%71 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%70) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %73 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(to) capture(ByRef) members(%72 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> %{{.*}} : [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%73 -> %arg0, %72 -> %arg1 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @control_flow
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^,]*]], %[[COND:.*]]: [[COND_TYPE:[^)]*]])
-  func.func @control_flow(%x: !fir.ref<i32>, %cond: !fir.ref<!fir.logical<4>>) {
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %cond_decl = fir.declare %cond {uniq_name = "cond"} : (!fir.ref<!fir.logical<4>>) -> !fir.ref<!fir.logical<4>>
-    %0 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-    %1 = fir.convert %0 : (!fir.logical<4>) -> i1
-    cf.cond_br %1, ^bb1, ^bb2
-  ^bb1:  // pred: ^bb0
-    fir.call @foo() : () -> ()
-    %m0 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-    cf.br ^bb2
-  ^bb2:  // 2 preds: ^bb0, ^bb1
-    // CHECK-NOT: fir.call
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    fir.call @foo() : () -> ()
-    %m1 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-      fir.call @foo() : () -> ()
-      %8 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-      %9 = fir.convert %8 : (!fir.logical<4>) -> i1
-      cf.cond_br %9, ^bb1, ^bb2
-    ^bb1:  // pred: ^bb0
-      fir.call @foo() : () -> ()
-      %m2 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      // CHECK-NOT: fir.call
-      // CHECK-NOT: cf.br
-      fir.call @foo() : () -> ()
-      cf.br ^bb2
-    ^bb2:  // 2 preds: ^bb0, ^bb1
-      fir.call @foo() : () -> ()
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @block_args
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @block_args(%x: !fir.ref<i32>) {
-    // CHECK-NEXT: %[[PLACEHOLDER0:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA0:.*]] = fir.convert %[[PLACEHOLDER0]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[PLACEHOLDER1:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA1:.*]] = fir.convert %[[PLACEHOLDER1]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[X_DECL0:.*]] = fir.declare %[[ALLOCA0]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[X_DECL1:.*]] = fir.declare %[[ALLOCA1]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL0]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL1]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %0 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %1 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        omp.parallel private(@privatizer %1 -> %arg1 : !fir.ref<i32>) {
-          %2 = fir.declare %arg1 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-          %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-          omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-            omp.terminator
-          }
-          omp.terminator
-        }
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    return
-  }
-
-  // CHECK-LABEL: func.func @reuse_tests()
-  func.func @reuse_tests() {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[THREAD_LIMIT:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> i32
-    // CHECK-NEXT: %[[CONST:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL_DECL0:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL1:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL2:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL0]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL1]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL2]] : !fir.ref<i32>, i32)
-    %0 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m2 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : !fir.ref<i32>, !fir.ref<i32>)
-      omp.target kernel_type(generic) map_entries(%m1 -> %arg0, %m2 -> %arg1 : !fir.ref<i32>, !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-    // CHECK-NOT: fir.load
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    %2 = fir.load %global : !fir.ref<i32>
-    %3 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) thread_limit(%2 : i32) map_entries(%m3 -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    %c1 = arith.constant 1 : i32
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @all_non_map_clauses
-  // CHECK-SAME: (%[[REF:.*]]: !fir.ref<i32>, %[[INT:.*]]: i32, %[[BOOL:.*]]: i1)
-  func.func @all_non_map_clauses(%ref: !fir.ref<i32>, %int: i32, %bool: i1) {
-    %m0 = omp.map.info var_ptr(%ref : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data device(%int : i32) if(%bool) map_entries(%m0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate({{[^)]*}}) thread_limit({{[^)]*}}) in_reduction({{[^)]*}}) private({{[^)]*}}) {
-    omp.target kernel_type(generic)
-               allocate(%ref : !fir.ref<i32> -> %ref : !fir.ref<i32>)
-               depend(taskdependin -> %ref : !fir.ref<i32>)
-               device(%int : i32) if(%bool) thread_limit(%int : i32)
-               in_reduction(@reduction %ref -> %arg0 : !fir.ref<i32>)
-               private(@privatizer %ref -> %arg1 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NOT: omp.target_enter_data
-    omp.target_enter_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                          device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_exit_data
-    omp.target_exit_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                         device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_update
-    omp.target_update depend(taskdependin -> %ref : !fir.ref<i32>)
-                      device(%int : i32) if(%bool)
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_length
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.boxchar<1>)
-  func.func @assumed_length(%arg: !fir.boxchar<1>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca !fir.char<1>
-    // CHECK-NEXT: %[[ONE:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[EMBOXCHAR:.*]] = fir.emboxchar %[[PLACEHOLDER]], %[[ONE]] : (!fir.ref<!fir.char<1>>, i32) -> !fir.boxchar<1>
-    // CHECK-NEXT: omp.target kernel_type(generic) private(@boxchar_firstprivatizer %[[EMBOXCHAR]] -> %{{.*}} [map_idx=0] : !fir.boxchar<1>)
-    %0 = fir.alloca !fir.boxchar<1>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2:2 = fir.unboxchar %arg : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
-    %3 = fir.declare %2#0 typeparams %2#1 dummy_scope %1 {uniq_name = "arg"} : (!fir.ref<!fir.char<1,?>>, index, !fir.dscope) -> !fir.ref<!fir.char<1,?>>
-    %4 = fir.emboxchar %3, %2#1 : (!fir.ref<!fir.char<1,?>>, index) -> !fir.boxchar<1>
-    omp.target kernel_type(generic) private(@boxchar_firstprivatizer %4 -> %arg3 [map_idx=0] : !fir.boxchar<1>) {
-      omp.terminator
-    }
-    return
-  }
-
-  omp.private {type = firstprivate} @boxchar_firstprivatizer : !fir.boxchar<1> copy {
-  ^bb0(%arg0: !fir.boxchar<1>, %arg1: !fir.boxchar<1>):
-    omp.yield(%arg0 : !fir.boxchar<1>)
-  }
-
-  // CHECK-LABEL: func.func @declare_storage
-  func.func @declare_storage(%i : index) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[VAR:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %0 = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %1 = fir.convert %0 : (!fir.ref<!fir.array<8xi8>>) -> !fir.ref<!fir.array<?xi8>>
-    %2 = fir.coordinate_of %1, %i : (!fir.ref<!fir.array<?xi8>>, index) -> !fir.ref<i8>
-    %3 = fir.convert %2 : (!fir.ref<i8>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[DECL:.*]] = fir.declare %[[VAR]] storage(%[[GLOBAL]][0])
-    %4 = fir.declare %3 storage (%0[0]) {uniq_name = "a"} : (!fir.ref<i32>, !fir.ref<!fir.array<8xi8>>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[DECL]] : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %map = omp.map.info var_ptr(%4 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    return
-  }
-
-  func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  omp.private {type = firstprivate} @privatizer : i32 copy {
-  ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
-    %0 = fir.load %arg0 : !fir.ref<i32>
-    fir.store %0 to %arg1 : !fir.ref<i32>
-    omp.yield(%arg1 : !fir.ref<i32>)
-  }
-  omp.declare_reduction @reduction : i32
-  init {
-  ^bb0(%arg: i32):
-    %0 = arith.constant 0 : i32
-    omp.yield (%0 : i32)
-  }
-  combiner {
-  ^bb1(%arg0: i32, %arg1: i32):
-    %1 = arith.addi %arg0, %arg1 : i32
-    omp.yield (%1 : i32)
-  }
-
-  // CHECK: fir.global internal @global_scalar constant : i32
-  fir.global @global_scalar constant : i32
-
-  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-
-  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-}
diff --git a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
index e6321ef58b45f..da90ef508ce64 100644
--- a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
@@ -11,6 +11,30 @@
 
 include "mlir/Pass/PassBase.td"
 
+def HostOpFilteringPass : Pass<"omp-host-op-filter"> {
+  let summary = "Removes all non target-related ops from host functions.";
+  let description = [{
+    When compiling for an OpenMP offloading target, host functions holding
+    target regions must be maintained in order for the `omp.target` ops to be
+    translated to LLVM IR as functions.
+    
+    However, these host functions might contain more host code than just what's
+    necessary for generating code for their contained target regions on the
+    target device, potentially causing problematic artifacts to be generated
+    into the target device's module.
+
+    This pass strips out all non-essential operations from host functions when
+    compiling for an OpenMP target device, leaving just `omp.target` operations,
+    related `omp.map.info` and placeholders in the form of function arguments to
+    keep the MLIR representation legal.
+
+    It must run on a module at a low level of abstraction (i.e. directly
+    translatable to LLVM IR), after any passes that might produce new OpenMP
+    operations have already completed.
+  }];
+  let dependentDialects = ["LLVM::LLVMDialect"];
+}
+
 def MarkDeclareTargetPass : Pass<"omp-mark-declare-target", "ModuleOp"> {
   let summary = "Marks all functions called by an OpenMP declare target "
                 "function as declare target";
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
index 569786fe95cf3..bbdbc556fc52f 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
@@ -1,4 +1,5 @@
 add_mlir_dialect_library(MLIROpenMPTransforms
+  HostOpFiltering.cpp
   MarkDeclareTarget.cpp
   OpenMPOffloadPrivatizationPrepare.cpp
   StackToShared.cpp
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
new file mode 100644
index 0000000000000..9c61b2717af57
--- /dev/null
+++ b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
@@ -0,0 +1,292 @@
+//===- HostOpFiltering.cpp ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements transforms to swap stack allocations on the target
+// device with device shared memory where applicable.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+
+namespace mlir {
+namespace omp {
+#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h.inc"
+} // namespace omp
+} // namespace mlir
+
+using namespace mlir;
+
+/// Some host operations, like \c llvm.mlir.addressof and constants, must remain
+/// in the device module because they impact how device code is generated when
+/// attached to an \c omp.target operation.
+///
+/// This function identifies the operations that need this special handling.
+/// This includes cast-style operations to avoid losing information about the
+/// original source of an operand.
+static bool keepHostOpInDevice(Operation &op) {
+  return isPure(&op) &&
+         op.getDialect() ==
+             op.getContext()->getLoadedDialect<LLVM::LLVMDialect>();
+}
+
+/// Add an \c omp.map.info operation and all its members recursively to the
+/// output set to be later rewritten.
+///
+/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
+/// are added to the output sets before operations based on them.
+static void collectRewrite(omp::MapInfoOp mapOp,
+                           llvm::SetVector<omp::MapInfoOp> &rewrites) {
+  for (Value member : mapOp.getMembers())
+    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
+
+  rewrites.insert(mapOp);
+}
+
+/// Add the given value to a sorted set if it should be replaced by a
+/// placeholder when used as an operand that must remain for the device.
+///
+/// Values that are block arguments of function operations are skipped, since
+/// they will still be available after all rewrites are completed, and operands
+/// of operations that need to remain on the host are recursively collected.
+static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
+  if ((isa<BlockArgument>(value) &&
+       isa<FunctionOpInterface>(
+           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
+      rewrites.contains(value))
+    return;
+
+  Operation *op = value.getDefiningOp();
+  if (op && keepHostOpInDevice(*op))
+    for (Value operand : op->getOperands())
+      collectRewrite(operand, rewrites);
+
+  rewrites.insert(value);
+}
+
+/// Provide the \c device_type of an \c omp.declare_target attribute, if
+/// defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
+namespace {
+class HostOpFilteringPass
+    : public omp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
+public:
+  HostOpFilteringPass() = default;
+
+  void runOnOperation() override {
+    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
+    if (!op || !op.getIsTargetDevice())
+      return;
+
+    op->walk<WalkOrder::PreOrder>([&](LLVM::LLVMFuncOp funcOp) {
+      omp::DeclareTargetDeviceType declareType =
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
+
+      // Only process host function definitions.
+      if (funcOp.isExternal() ||
+          declareType != omp::DeclareTargetDeviceType::host)
+        return WalkResult::advance();
+
+      if (failed(rewriteHostFunction(funcOp))) {
+        funcOp.emitOpError() << "could not filter host-only operations";
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time redefinitions.
+    op->walk([&](LLVM::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkage(LLVM::Linkage::Internal);
+    });
+  }
+
+private:
+  /// Rewrite the given host device function containing \c omp.target
+  /// operations, to remove host-only operations that are not used by device
+  /// codegen.
+  ///
+  /// It is based on the expected form of an MLIR module lowered to where it can
+  /// be directly translated to LLVM IR and it performs the following mutations:
+  ///   - Removes all returned values from the function.
+  ///   - \c omp.target operations are moved to the end of the function. If they
+  ///     are nested inside of any other operations, they are hoisted out of
+  ///     them.
+  ///   - \c depend, \c device, \c dyn_groupprivate, \c if and \c in_reduction
+  ///     clauses are removed from these target functions. Values used to
+  ///     initialize other clauses are replaced by placeholders as follows:
+  ///     - Values defined by block arguments are replaced by placeholders only
+  ///       if they are not attached to the parent function. In that case, they
+  ///       are passed unmodified.
+  ///     - Pure operations of the LLVM dialect are maintained, and any value
+  ///       operands they might have are also replaced by placeholders following
+  ///       the same rules.
+  ///     - Other values are replaced by new function arguments.
+  ///   - \c omp.map.info operations associated to these target regions are
+  ///     preserved. These are moved above all \c omp.target and sorted to
+  ///     satisfy dependencies among them.
+  ///   - \c bounds arguments are removed from \c omp.map.info operations.
+  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
+  ///     replaced by placeholders as described above.
+  ///   - Every other operation not located inside of an \c omp.target is
+  ///     removed.
+  LogicalResult rewriteHostFunction(LLVM::LLVMFuncOp funcOp) {
+    Region &region = funcOp.getFunctionBody();
+    LLVM::LLVMFunctionType functionType = funcOp.getFunctionType();
+
+    // Collect target operations inside of the function.
+    llvm::SmallVector<omp::TargetOp> targetOps;
+    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
+      // Skip the inside of omp.target regions, since these contain device code.
+      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
+        targetOps.push_back(targetOp);
+        return WalkResult::skip();
+      }
+
+      // Replace omp.target_data entry block argument uses with the value used
+      // to initialize the associated omp.map.info operation. This way,
+      // references are still valid once the omp.target operation has been
+      // extracted out of the omp.target_data region.
+      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
+        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
+        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
+            .getBlockArgsPairs(argPairs);
+        for (auto [operand, blockArg] : argPairs) {
+          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
+          blockArg.replaceAllUsesWith(mapInfo.getVarPtr());
+        }
+      }
+      return WalkResult::advance();
+    });
+
+    // Make a temporary clone of the parent function with an empty region,
+    // and update all references to entry block arguments to those of the new
+    // region. Users of these arguments will later either be moved to the new
+    // region or deleted when the original region is replaced by the new.
+    OpBuilder builder(&getContext());
+    builder.setInsertionPointAfter(funcOp);
+    Operation *newFuncOp = builder.cloneWithoutRegions(funcOp);
+    Block &block = newFuncOp->getRegion(0).emplaceBlock();
+
+    llvm::SmallVector<Location> locs;
+    locs.reserve(region.getNumArguments());
+    llvm::transform(region.getArguments(), std::back_inserter(locs),
+                    [](const BlockArgument &arg) { return arg.getLoc(); });
+    block.addArguments(region.getArgumentTypes(), locs);
+
+    for (auto [oldArg, newArg] :
+         llvm::zip_equal(region.getArguments(), block.getArguments()))
+      oldArg.replaceAllUsesWith(newArg);
+
+    // Collect omp.map.info ops while satisfying interdependencies and remove
+    // operands that aren't used by target device codegen.
+    //
+    // This logic must be updated whenever operands to omp.target change.
+    llvm::SetVector<Value> rewriteValues;
+    llvm::SetVector<omp::MapInfoOp> mapInfos;
+    for (omp::TargetOp targetOp : targetOps) {
+      assert(targetOp.getHostEvalVars().empty() &&
+             "unexpected host_eval in target device module");
+
+      // Variables unused by the device.
+      targetOp.getDependVarsMutable().clear();
+      targetOp.setDependKindsAttr(nullptr);
+      targetOp.getDependIteratedMutable().clear();
+      targetOp.setDependIteratedKindsAttr(nullptr);
+      targetOp.getDeviceMutable().clear();
+      targetOp.getDynGroupprivateSizeMutable().clear();
+      targetOp.getIfExprMutable().clear();
+      targetOp.getInReductionVarsMutable().clear();
+      targetOp.setInReductionByrefAttr(nullptr);
+      targetOp.setInReductionSymsAttr(nullptr);
+
+      // TODO: Clear some of these operands rather than rewriting them,
+      // depending on whether they are needed by device codegen once support for
+      // them is fully implemented.
+      for (Value allocVar : targetOp.getAllocateVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value allocVar : targetOp.getAllocatorVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
+        collectRewrite(isDevPtr, rewriteValues);
+      for (Value mapVar : targetOp.getHasDeviceAddrVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value mapVar : targetOp.getMapVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value privateVar : targetOp.getPrivateVars())
+        collectRewrite(privateVar, rewriteValues);
+      for (Value threadLimit : targetOp.getThreadLimitVars())
+        collectRewrite(threadLimit, rewriteValues);
+    }
+
+    // Move omp.map.info ops to the new block and collect dependencies.
+    for (omp::MapInfoOp mapOp : mapInfos) {
+      collectRewrite(mapOp.getVarPtr(), rewriteValues);
+
+      if (Value varPtrPtr = mapOp.getVarPtrPtr())
+        collectRewrite(varPtrPtr, rewriteValues);
+
+      // Bounds are not used during target device codegen.
+      mapOp.getBoundsMutable().clear();
+      mapOp->moveBefore(&block, block.end());
+    }
+
+    builder.setInsertionPointToStart(&block);
+
+    // We don't actually need the proper initialization for all operands, but
+    // rather just to maintain the basic form of omp.target operations. We
+    // create new function arguments as placeholders for rewritten values.
+    llvm::SmallVector<Type> newFnArgTypes(functionType.getParams());
+    for (Value value : rewriteValues) {
+      Value rewriteValue;
+      Operation *definingOp = value.getDefiningOp();
+      if (definingOp && keepHostOpInDevice(*definingOp)) {
+        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
+      } else {
+        rewriteValue = block.addArgument(value.getType(), value.getLoc());
+        newFnArgTypes.push_back(rewriteValue.getType());
+      }
+      value.replaceAllUsesWith(rewriteValue);
+    }
+
+    // Move target operations to the end of the new block.
+    for (omp::TargetOp targetOp : targetOps)
+      targetOp->moveBefore(&block, block.end());
+
+    // Add terminator to the new block.
+    builder.setInsertionPointToEnd(&block);
+    LLVM::ReturnOp::create(builder, funcOp.getLoc(), ValueRange());
+
+    // Replace old region with the new one, now only containing the required
+    // operations, and remove the temporary operation clone.
+    region.takeBody(newFuncOp->getRegion(0));
+    newFuncOp->erase();
+
+    // Update function type after modifying the terminator and argument list.
+    funcOp.setType(LLVM::LLVMFunctionType::get(
+        LLVM::LLVMVoidType::get(&getContext()), newFnArgTypes));
+
+    return success();
+  }
+};
+} // namespace
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
new file mode 100644
index 0000000000000..015e901126a76
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -0,0 +1,500 @@
+// RUN: mlir-opt --omp-host-op-filter %s | FileCheck %s
+module attributes {omp.is_target_device = true} {
+  // CHECK-LABEL: llvm.func @external() attributes {sym_visibility = "private"}
+  // CHECK-NOT: llvm.return
+  llvm.func @external() attributes {sym_visibility = "private"}
+
+  // CHECK-LABEL: llvm.func @basic_checks
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @basic_checks(%arg0: !llvm.ptr) -> !llvm.struct<(i32, f32)> {
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = llvm.mlir.constant(10 : i32) : i32
+    %3 = llvm.mlir.constant(2.500000e+00 : f32) : f32
+
+    llvm.call @foo() : () -> ()
+
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %4 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %5 = llvm.mlir.constant(1 : i64) : i64
+
+    // CHECK-NEXT: %[[HDA:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]]{{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]]{{.*}})
+    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    %6 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %7 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %8 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %9 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[HDA]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP2]] -> {{.*}} : {{.*}})
+    omp.target kernel_type(generic) has_device_addr(%8 -> %arg1 : !llvm.ptr) map_entries(%6 -> %arg2, %7 -> %arg3, %9 -> %arg4 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NEXT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: llvm.call
+    omp.parallel {
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.map.info
+    %10 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %11 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %12 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %13 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%10, %11, %12, %13 : !llvm.ptr, !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    %14 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_enter_data map_entries(%14 : !llvm.ptr)
+    %15 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(from) capture(ByRef) -> !llvm.ptr
+    omp.target_exit_data map_entries(%15 : !llvm.ptr)
+    %16 = omp.map.info var_ptr(%1 : !llvm.ptr, !llvm.ptr) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_update map_entries(%16 : !llvm.ptr)
+
+    // CHECK-NOT: llvm.call
+    llvm.call @foo() : () -> ()
+
+    // CHECK: {{llvm\.return$}}
+    %17 = llvm.mlir.poison : !llvm.struct<(i32, f32)>
+    %18 = llvm.insertvalue %2, %17[0] : !llvm.struct<(i32, f32)>
+    %19 = llvm.insertvalue %3, %18[1] : !llvm.struct<(i32, f32)>
+    llvm.return %19 : !llvm.struct<(i32, f32)>
+  }
+
+  // CHECK-LABEL: llvm.func @member_map_complex_init
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @member_map_complex_init(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[PLACEHOLDER]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<{{.*}}>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(1 : i64) : i64
+    %3 = llvm.alloca %2 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %4 = llvm.mlir.constant(1 : i64) : i64
+    %5 = llvm.alloca %4 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %6 = llvm.mlir.constant(1 : i32) : i32
+    %7 = llvm.mlir.constant(0 : i8) : i8
+    %8 = llvm.mlir.constant(1 : i64) : i64
+    %9 = llvm.mlir.constant(1 : i64) : i64
+    %10 = llvm.mlir.zero : !llvm.ptr
+    llvm.call @LangRTPlaceholderFunc(%5, %arg0, %10, %7, %6) : (!llvm.ptr, !llvm.ptr, !llvm.ptr, i8, i32) -> ()
+    %11 = llvm.mlir.constant(384 : i32) : i32
+    "llvm.intr.memcpy"(%1, %5, %11) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %12 = llvm.mlir.constant(24 : i32) : i32
+    %13 = llvm.getelementptr %1[0, 3] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+    %14 = llvm.load %13 : !llvm.ptr -> i8
+    %15 = llvm.sext %14 : i8 to i32
+    %16 = llvm.mlir.constant(24 : i32) : i32
+    %17 = llvm.mul %16, %15 : i32
+    %18 = llvm.add %12, %17 : i32
+    "llvm.intr.memcpy"(%3, %1, %18) <{isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %19 = llvm.getelementptr %3[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[GEP]] : {{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr)
+    %20 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%19 : !llvm.ptr, f32) -> !llvm.ptr
+    %21 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%20 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> %{{.*}} : {{.*}})
+    omp.target kernel_type(generic) map_entries(%21 -> %arg1, %20 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @target_data
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr, %[[ARG2:.*]]: !llvm.ptr)
+  llvm.func @target_data(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: !llvm.ptr) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%arg2 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%0 : !llvm.ptr) use_device_addr(%1 -> %arg3 : !llvm.ptr) use_device_ptr(%2 -> %arg4 : !llvm.ptr) {
+      // CHECK:      %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ARG2]] : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef)
+      %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %4 = omp.map.info var_ptr(%arg3 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%arg4 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}}, %[[MAP2]] -> %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%3 -> %arg5, %4 -> %arg6, %5 -> %arg7 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @no_target
+  // CHECK-SAME: (%{{.*}}: !llvm.ptr)
+  llvm.func @no_target(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: llvm.return
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%0 : !llvm.ptr) {
+      %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%1 : !llvm.ptr) {
+        llvm.call @foo() : () -> ()
+        omp.terminator
+      }
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @map_info_members
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr)
+  llvm.func @map_info_members(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = llvm.getelementptr %[[ARG]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(0 : index) : i64
+    %3 = llvm.mlir.constant(1 : index) : i64
+    %4 = llvm.mlir.constant(9 : index) : i64
+    %5 = llvm.mlir.constant(48 : i32) : i32
+    "llvm.intr.memcpy"(%1, %arg0, %5) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %6 = llvm.getelementptr %1[0, 7, %2, 0] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %7 = llvm.load %6 : !llvm.ptr -> i64
+    %8 = llvm.getelementptr %1[0, 7, %2, 1] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %9 = llvm.load %8 : !llvm.ptr -> i64
+    %10 = llvm.getelementptr %1[0, 7, %2, 2] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %11 = llvm.load %10 : !llvm.ptr -> i64
+    %12 = llvm.sub %3, %7 : i64
+    %13 = llvm.sub %4, %7 : i64
+    %14 = omp.map.bounds lower_bound(%12 : i64) upper_bound(%13 : i64) extent(%9 : i64) stride(%11 : i64) start_idx(%7 : i64) {stride_in_bytes = true}
+    %15 = llvm.getelementptr %arg0[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : !llvm.ptr, f32) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr) -> !llvm.ptr
+    %16 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%15 : !llvm.ptr, f32) bounds(%14) -> !llvm.ptr
+    %17 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%16 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> {{.*}} : !llvm.ptr, !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%17 -> %arg1, %16 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @control_flow
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr)
+  llvm.func @control_flow(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %0 = llvm.load %arg1 : !llvm.ptr -> i32
+    %1 = llvm.mlir.constant(0 : i32) : i32
+    %2 = llvm.icmp "ne" %0, %1 : i32
+    llvm.cond_br %2, ^bb1, ^bb2
+  ^bb1:  // pred: ^bb0
+    llvm.call @foo() : () -> ()
+    %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%3 -> %arg2 : !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+    llvm.br ^bb2
+  ^bb2:  // 2 preds: ^bb0, ^bb1
+    // CHECK-NOT: llvm.call
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    llvm.call @foo() : () -> ()
+
+    %4 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%4 : !llvm.ptr) {
+      llvm.call @foo() : () -> ()
+      %5 = llvm.load %arg1 : !llvm.ptr -> i32
+      %6 = llvm.mlir.constant(0 : i32) : i32
+      %7 = llvm.icmp "ne" %5, %6 : i32
+      llvm.cond_br %7, ^bb1, ^bb2
+    ^bb1:  // pred: ^bb0
+      llvm.call @foo() : () -> ()
+      %8 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%8 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      // CHECK-NOT: llvm.br
+      llvm.call @foo() : () -> ()
+      llvm.br ^bb2
+    ^bb2:  // 2 preds: ^bb0, ^bb1
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @block_args
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[PLACEHOLDER0:.*]]: !llvm.ptr, %[[PLACEHOLDER1:.*]]: !llvm.ptr)
+  llvm.func @block_args(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%0 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%0 : !llvm.ptr) {
+        omp.parallel private(@privatizer %arg1 -> %arg2 : !llvm.ptr) {
+          %1 = omp.map.info var_ptr(%arg2 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+          omp.target kernel_type(generic) map_entries(%1 -> %arg3 : !llvm.ptr) {
+            omp.terminator
+          }
+          omp.terminator
+        }
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reuse_tests
+  // CHECK-SAME: (%[[THREAD_LIMIT:.*]]: i32)
+  llvm.func @reuse_tests() {
+    // CHECK-NEXT: %[[CONST_THREAD_LIMIT:.*]] = llvm.mlir.constant(1 : i32) : i32
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%2 : !llvm.ptr) {
+      // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %6 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}} : !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%5 -> %arg0, %6 -> %arg1 : !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: llvm.load
+    // CHECK-NOT: omp.map.info
+    %3 = llvm.load %1 : !llvm.ptr -> i32
+    %4 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) thread_limit(%3 : i32) map_entries(%4 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @all_non_map_clauses
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: i1)
+  llvm.func @all_non_map_clauses(%arg0: !llvm.ptr, %arg1: i32, %arg2: i1) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data device(%arg1 : i32) if(%arg2) map_entries(%0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    omp.target_enter_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_exit_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_update depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @private_with_map_idx
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i64)
+  llvm.func @private_with_map_idx(%arg0: !llvm.ptr, %arg1: i64) {
+    // CHECK-NEXT: %[[VAL0:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL1:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL2:.*]] = llvm.insertvalue %[[ARG0]], %[[VAL1]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL3:.*]] = llvm.insertvalue %[[ARG1]], %[[VAL2]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL4:.*]] = llvm.extractvalue %[[VAL3]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL5:.*]] = llvm.insertvalue %[[VAL4]], %[[VAL0]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL6:.*]] = llvm.extractvalue %[[VAL3]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL7:.*]] = llvm.insertvalue %[[VAL6]], %[[VAL5]][1] : !llvm.struct<(ptr, i64)>
+    %0 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, i64)>
+    %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, i64)>
+    %3 = llvm.extractvalue %2[0] : !llvm.struct<(ptr, i64)>
+    %4 = llvm.extractvalue %2[1] : !llvm.struct<(ptr, i64)>
+    %5 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %6 = llvm.insertvalue %3, %5[0] : !llvm.struct<(ptr, i64)>
+    %7 = llvm.insertvalue %4, %6[1] : !llvm.struct<(ptr, i64)>
+
+    // CHECK: omp.target kernel_type(generic) private(@struct_firstprivatizer %[[VAL7]] -> %{{.*}} [map_idx=0] : !llvm.struct<(ptr, i64)>)
+    omp.target kernel_type(generic) private(@struct_firstprivatizer %7 -> %arg2 [map_idx=0] : !llvm.struct<(ptr, i64)>) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  omp.private {type = firstprivate} @struct_firstprivatizer : !llvm.struct<(ptr, i64)> copy {
+  ^bb0(%arg0: !llvm.struct<(ptr, i64)>, %arg1: !llvm.struct<(ptr, i64)>):
+    omp.yield(%arg0 : !llvm.struct<(ptr, i64)>)
+  }
+
+  // CHECK-LABEL: llvm.func @map_from_global_gep
+  // CHECK-SAME: (%[[ARG0:.*]]: i64)
+  llvm.func @map_from_global_gep(%arg0: i64) {
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_array : !llvm.ptr
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[GLOBAL]][%[[ARG0]]] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+    %0 = llvm.mlir.addressof @global_array : !llvm.ptr
+    %1 = llvm.getelementptr %0[%arg0] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[GEP]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_a
+  // CHECK-SAME: (%[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_a() {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_b
+    llvm.call @reciprocal_b(%0) : (i64) -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_b
+  // CHECK-SAME: (%[[ARG0:.*]]: i64, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_b(%arg0 : i64) {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_a
+    llvm.call @reciprocal_a() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  llvm.func @foo() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>, sym_visibility = "private"}
+  omp.private {type = firstprivate} @privatizer : i32 copy {
+  ^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+    %0 = llvm.load %arg0 : !llvm.ptr -> i32
+    llvm.store %0, %arg1 : i32, !llvm.ptr
+    omp.yield(%arg1 : !llvm.ptr)
+  }
+  omp.declare_reduction @reduction : i32 init {
+
+  ^bb0(%arg0: i32):
+    %0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%0 : i32)
+  } combiner {
+
+  ^bb0(%arg0: i32, %arg1: i32):
+    %0 = llvm.add %arg0, %arg1 : i32
+    omp.yield(%0 : i32)
+  }
+
+  // CHECK: llvm.mlir.global internal constant @global_scalar
+  // CHECK: llvm.mlir.global internal @global_array
+  llvm.mlir.global external constant @global_scalar() {addr_space = 0 : i32} : i32
+  llvm.mlir.global external @global_array() {addr_space = 0 : i32} : !llvm.array<8 x i8>
+
+  // CHECK: llvm.mlir.global external @declare_target_enter_any
+  // CHECK: llvm.mlir.global external @declare_target_enter_host
+  // CHECK: llvm.mlir.global external @declare_target_enter_nohost
+  llvm.mlir.global external @declare_target_enter_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: llvm.mlir.global external @declare_target_link_any
+  // CHECK: llvm.mlir.global external @declare_target_link_host
+  // CHECK: llvm.mlir.global external @declare_target_link_nohost
+  llvm.mlir.global external @declare_target_link_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+
+  llvm.func @LangRTPlaceholderFunc(!llvm.ptr {llvm.nocapture}, !llvm.ptr {llvm.nocapture}, !llvm.ptr, i8 {llvm.signext}, i32) attributes {sym_visibility = "private"}
+}


From llvmlistbot at llvm.org  Thu Jul  9 03:56:56 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 09 Jul 2026 03:56:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (PR #208189)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208189@github.com>
Message-ID: <6a4f7e78.8e75e0cb.5e108.3e0c@mx.google.com>


https://github.com/skatrak closed https://github.com/llvm/llvm-project/pull/208189

From llvmlistbot at llvm.org  Thu Jul  9 03:56:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 03:56:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 5b9fa24 - [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (#208189)
Message-ID: <6a4f7e78.824e11a6.f5f27.60b4@mx.google.com>


Author: Sergio Afonso
Date: 2026-07-09T11:56:50+01:00
New Revision: 5b9fa24dbd7b504b99143ebd2f5f12f8560c6331

URL: https://github.com/llvm/llvm-project/commit/5b9fa24dbd7b504b99143ebd2f5f12f8560c6331
DIFF: https://github.com/llvm/llvm-project/commit/5b9fa24dbd7b504b99143ebd2f5f12f8560c6331.diff

LOG: [Flang][MLIR][OpenMP] Move host op filtering to the omp dialect (#208189)

The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.

By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.

As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
operations pointing to deleted host functions, while `fir.dispatch` ops
pointing to the same functions would remain. The
`omp-function-filtering` pass is updated to prevent `fir.dt_entry` from
being deleted and let them point to an undefined symbol. This lets FIR
lowering to LLVM work well enough to get to the host op filtering stage,
where any resulting operations for this host dynamic dispatch are
cleaned up.

Added: 
    flang/test/Lower/OpenMP/function-filtering-4.f90
    mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
    mlir/test/Dialect/OpenMP/host-op-filtering.mlir

Modified: 
    flang/include/flang/Optimizer/OpenMP/Passes.td
    flang/lib/Optimizer/OpenMP/CMakeLists.txt
    flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
    flang/lib/Optimizer/Passes/Pipelines.cpp
    flang/test/Fir/basic-program.fir
    flang/test/Lower/OpenMP/host-eval.f90
    mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
    mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt

Removed: 
    flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
    flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir


################################################################################
diff  --git a/flang/include/flang/Optimizer/OpenMP/Passes.td b/flang/include/flang/Optimizer/OpenMP/Passes.td
index b5d9372e356f1..9ec159e1ba1e0 100644
--- a/flang/include/flang/Optimizer/OpenMP/Passes.td
+++ b/flang/include/flang/Optimizer/OpenMP/Passes.td
@@ -57,28 +57,6 @@ def FunctionFilteringPass : Pass<"omp-function-filtering"> {
   ];
 }
 
-def HostOpFilteringPass : Pass<"omp-host-op-filtering"> {
-  let summary =
-      "Removes host-only operations from host functions retained by the "
-      "function filtering pass when compiling for a target device.";
-  let description = [{
-    Removes operations from host functions that are not placed inside of an
-    `omp.target` region when compiling for the target device.
-    
-    It must run after any passes that might introduce new host operations,
-    especially if they belong to the 'omp' dialect, like `LowerWorkdistribute`.
-    This ensures all potentially problematic host operations are accounted for
-    prior to LLVM IR translation.
-
-    This pass works at the FIR level of abstraction.
-  }];
-  let dependentDialects = [
-    "mlir::arith::ArithDialect",
-    "mlir::func::FuncDialect",
-    "fir::FIROpsDialect"
-  ];
-}
-
 def DoConcurrentConversionPass : Pass<"omp-do-concurrent-conversion", "mlir::ModuleOp"> {
   let summary = "Map `DO CONCURRENT` loops to OpenMP worksharing loops.";
 

diff  --git a/flang/lib/Optimizer/OpenMP/CMakeLists.txt b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
index f25e95f041d20..db29e93b71dad 100644
--- a/flang/lib/Optimizer/OpenMP/CMakeLists.txt
+++ b/flang/lib/Optimizer/OpenMP/CMakeLists.txt
@@ -5,7 +5,6 @@ add_flang_library(FlangOpenMPTransforms
   DoConcurrentConversion.cpp
   FunctionFiltering.cpp
   GenericLoopConversion.cpp
-  HostOpFiltering.cpp
   MapsForPrivatizedSymbols.cpp
   MapInfoFinalization.cpp
   DeleteUnreachableTargets.cpp

diff  --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index e8bcc0c7fa062..ee61dd8881fd0 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -48,8 +48,7 @@ checkDeviceImplementationStatus(omp::OffloadModuleInterface offloadModule) {
     if (!redOp.getByrefElementType())
       return WalkResult::advance();
 
-    auto seqTy =
-        mlir::dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
+    auto seqTy = dyn_cast<fir::SequenceType>(*redOp.getByrefElementType());
 
     bool isByRefReductionSupported =
         !seqTy || !fir::sequenceWithNonConstantShape(seqTy);
@@ -104,7 +103,7 @@ class FunctionFilteringPass
         SymbolTable::UseRange funcUses = *funcOp.getSymbolUses(op);
         for (SymbolTable::SymbolUse use : funcUses) {
           Operation *callOp = use.getUser();
-          if (auto internalFunc = mlir::dyn_cast<func::FuncOp>(callOp)) {
+          if (auto internalFunc = dyn_cast<func::FuncOp>(callOp)) {
             // Do not delete internal procedures holding the symbol of their
             // Fortran host procedure as attribute.
             internalFunc->removeAttr(fir::getHostSymbolAttrName());
@@ -114,6 +113,15 @@ class FunctionFilteringPass
             internalFunc.setVisibility(mlir::SymbolTable::Visibility::Public);
             continue;
           }
+          // Prevent dispatch table entries pointing to deleted functions
+          // from being removed. This prevents the lowering of any
+          // corresponding fir.dispatch ops from triggering errors. These
+          // fir.dt_entry ops will point to an undefined symbol as a result,
+          // which currently doesn't cause an issue, as fir.dispatch-related ops
+          // are later removed by the host op filtering pass.
+          if (isa<fir::DTEntryOp>(callOp))
+            continue;
+
           // If the callOp has users then replace them with Undef values.
           if (!callOp->use_empty()) {
             SmallVector<Value> undefResults;

diff  --git a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp b/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
deleted file mode 100644
index 05ea0bc08c4ab..0000000000000
--- a/flang/lib/Optimizer/OpenMP/HostOpFiltering.cpp
+++ /dev/null
@@ -1,422 +0,0 @@
-//===- HostOpFiltering.cpp -----------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements transforms to filter out host-only operations from
-// any remaining host functions when compiling for a target device.
-//
-//===----------------------------------------------------------------------===//
-
-#include "flang/Optimizer/Dialect/FIRDialect.h"
-#include "flang/Optimizer/Dialect/FIROps.h"
-#include "flang/Optimizer/OpenMP/Passes.h"
-
-#include "mlir/Dialect/Arith/IR/Arith.h"
-#include "mlir/Dialect/Func/IR/FuncOps.h"
-#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
-#include "llvm/ADT/SetVector.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/ADT/TypeSwitch.h"
-
-namespace flangomp {
-#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
-#include "flang/Optimizer/OpenMP/Passes.h.inc"
-} // namespace flangomp
-
-using namespace mlir;
-
-/// Add an operation to one of the output sets to be later rewritten.
-template <typename OpTy>
-static void collectRewrite(OpTy op, llvm::SetVector<OpTy> &rewrites) {
-  rewrites.insert(op);
-}
-
-/// Add an \c omp.map.info operation and all its members recursively to the
-/// output set to be later rewritten.
-///
-/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
-/// are added to the output sets before operations based on them.
-template <>
-void collectRewrite(omp::MapInfoOp mapOp,
-                    llvm::SetVector<omp::MapInfoOp> &rewrites) {
-  for (Value member : mapOp.getMembers())
-    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
-
-  rewrites.insert(mapOp);
-}
-
-/// Add the given value to a sorted set if it should be replaced by a
-/// placeholder when used as an operand that must remain for the device.
-///
-/// Values that are block arguments of \c func.func operations are skipped,
-/// since they will still be available after all rewrites are completed.
-static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
-  if ((isa<BlockArgument>(value) &&
-       isa<func::FuncOp>(
-           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
-      rewrites.contains(value))
-    return;
-
-  rewrites.insert(value);
-}
-
-/// Provide the `device_type` of an `omp.declare_target` attribute, if defined.
-static std::optional<omp::DeclareTargetDeviceType>
-getDeclareTargetDevice(Operation &op) {
-  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
-  if (declareTargetOp && declareTargetOp.isDeclareTarget())
-    return declareTargetOp.getDeclareTargetDeviceType();
-  return std::nullopt;
-}
-
-namespace {
-class HostOpFilteringPass
-    : public flangomp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
-public:
-  HostOpFilteringPass() = default;
-
-  void runOnOperation() override {
-    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
-    if (!op || !op.getIsTargetDevice())
-      return;
-
-    op->walk<WalkOrder::PreOrder>([&](func::FuncOp funcOp) {
-      omp::DeclareTargetDeviceType declareType =
-          getDeclareTargetDevice(*funcOp.getOperation())
-              .value_or(omp::DeclareTargetDeviceType::host);
-
-      // Only process host function definitions.
-      if (funcOp.isExternal() ||
-          declareType != omp::DeclareTargetDeviceType::host)
-        return WalkResult::advance();
-
-      if (failed(rewriteHostFunction(funcOp))) {
-        funcOp.emitOpError() << "could not filter host-only operations";
-        return WalkResult::interrupt();
-      }
-      return WalkResult::advance();
-    });
-
-    // Make non-declare target globals internal for the device. They cannot be
-    // deleted, because they are needed in order to properly lower map clauses.
-    // However, no uses will remain in the device module, so we make them
-    // internal to prevent link time issues.
-    op->walk([&](fir::GlobalOp globalOp) {
-      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
-        globalOp.setLinkName("internal");
-    });
-  }
-
-private:
-  /// Rewrite the given host device function containing \c omp.target
-  /// operations, to remove host-only operations that are not used by device
-  /// codegen.
-  ///
-  /// It is based on the expected form of the MLIR module as produced by Flang
-  /// lowering, after HLFIR to FIR lowering, and it performs the following
-  /// mutations:
-  ///   - Replace all values returned by the function with \c fir.undefined.
-  ///   - \c omp.target operations are moved to the end of the function. If they
-  ///     are nested inside of any other operations, they are hoisted out of
-  ///     them.
-  ///   - \c depend, \c device and \c if clauses are removed from these target
-  ///     functions. Values used to initialize other clauses are replaced by
-  ///     placeholders as follows:
-  ///     - Values defined by block arguments are replaced by placeholders only
-  ///       if they are not attached to the parent \c func.func operation. In
-  ///       that case, they are passed unmodified.
-  ///     - \c arith.constant and \c fir.address_of ops are maintained.
-  ///     - Values of type \c fir.boxchar are replaced with a combination of
-  ///       \c fir.alloca for a single bit and a \c fir.emboxchar.
-  ///     - Other values are replaced by a combination of an \c fir.alloca for a
-  ///       single bit and an \c fir.convert to the original type of the value.
-  ///       This can be done because the code eventually generated for these
-  ///       operations will be discarded, as they aren't runnable by the target
-  ///       device.
-  ///   - \c omp.map.info operations associated to these target regions are
-  ///     preserved. These are moved above all \c omp.target and sorted to
-  ///     satisfy dependencies among them.
-  ///   - \c bounds arguments are removed from \c omp.map.info operations.
-  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
-  ///     handled as follows:
-  ///     - \c var_ptr_ptr is expected to be defined by a \c fir.box_offset
-  ///       operation which is preserved. Otherwise, the pass will fail.
-  ///     - \c var_ptr can be defined by a \c fir.declare which is also
-  ///       preserved. Its \c memref argument is replaced by a placeholder or
-  ///       maintained, similarly to non-map clauses of target operations
-  ///       described above. If it has \c shape or \c typeparams arguments, they
-  ///       are replaced by applicable constants. \c dummy_scope arguments
-  ///       are discarded.
-  ///   - Every other operation not located inside of an \c omp.target is
-  ///     removed.
-  LogicalResult rewriteHostFunction(func::FuncOp funcOp) {
-    Region &region = funcOp.getRegion();
-
-    // Collect target operations inside of the function.
-    llvm::SmallVector<omp::TargetOp> targetOps;
-    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
-      // Skip the inside of omp.target regions, since these contain device code.
-      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
-        targetOps.push_back(targetOp);
-        return WalkResult::skip();
-      }
-
-      // Replace omp.target_data entry block argument uses with the value used
-      // to initialize the associated omp.map.info operation. This way,
-      // references are still valid once the omp.target operation has been
-      // extracted out of the omp.target_data region.
-      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
-        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
-        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
-            .getBlockArgsPairs(argPairs);
-        for (auto [operand, blockArg] : argPairs) {
-          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
-          Value varPtr = mapInfo.getVarPtr();
-
-          // If the var_ptr operand of the omp.map.info op defining this entry
-          // block argument is a fir.declare, the uses of all users of that
-          // entry block argument that are themselves fir.declare are replaced
-          // by the value produced by the outer one.
-          //
-          // This prevents this pass from producing chains of fir.declare of the
-          // type:
-          // %0 = ...
-          // %1 = fir.declare %0
-          // %2 = fir.declare %1...
-          // %3 = omp.map.info var_ptr(%2 ...
-          if (auto outerDeclare = varPtr.getDefiningOp<fir::DeclareOp>())
-            for (Operation *user : blockArg.getUsers())
-              if (isa<fir::DeclareOp>(user))
-                user->replaceAllUsesWith(outerDeclare);
-
-          // All remaining uses of the entry block argument are replaced with
-          // the var_ptr initialization value.
-          blockArg.replaceAllUsesWith(varPtr);
-        }
-      }
-      return WalkResult::advance();
-    });
-
-    // Make a temporary clone of the parent operation with an empty region,
-    // and update all references to entry block arguments to those of the new
-    // region. Users will later either be moved to the new region or deleted
-    // when the original region is replaced by the new.
-    OpBuilder builder(&getContext());
-    builder.setInsertionPointAfter(funcOp);
-    Operation *newOp = builder.cloneWithoutRegions(funcOp);
-    Block &block = newOp->getRegion(0).emplaceBlock();
-
-    llvm::SmallVector<Location> locs;
-    locs.reserve(region.getNumArguments());
-    llvm::transform(region.getArguments(), std::back_inserter(locs),
-                    [](const BlockArgument &arg) { return arg.getLoc(); });
-    block.addArguments(region.getArgumentTypes(), locs);
-
-    for (auto [oldArg, newArg] :
-         llvm::zip_equal(region.getArguments(), block.getArguments()))
-      oldArg.replaceAllUsesWith(newArg);
-
-    // Collect omp.map.info ops while satisfying interdependencies and remove
-    // operands that aren't used by target device codegen.
-    //
-    // This logic must be updated whenever operands to omp.target change.
-    llvm::SetVector<Value> rewriteValues;
-    llvm::SetVector<omp::MapInfoOp> mapInfos;
-    for (omp::TargetOp targetOp : targetOps) {
-      assert(targetOp.getHostEvalVars().empty() &&
-             "unexpected host_eval in target device module");
-
-      // Variables unused by the device.
-      targetOp.getDependVarsMutable().clear();
-      targetOp.setDependKindsAttr(nullptr);
-      targetOp.getDependIteratedMutable().clear();
-      targetOp.setDependIteratedKindsAttr(nullptr);
-      targetOp.getDeviceMutable().clear();
-      targetOp.getDynGroupprivateSizeMutable().clear();
-      targetOp.getIfExprMutable().clear();
-
-      // TODO: Clear some of these operands rather than rewriting them,
-      // depending on whether they are needed by device codegen once support for
-      // them is fully implemented.
-      for (Value allocVar : targetOp.getAllocateVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value allocVar : targetOp.getAllocatorVars())
-        collectRewrite(allocVar, rewriteValues);
-      for (Value inReduction : targetOp.getInReductionVars())
-        collectRewrite(inReduction, rewriteValues);
-      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
-        collectRewrite(isDevPtr, rewriteValues);
-      for (Value mapVar : targetOp.getHasDeviceAddrVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value mapVar : targetOp.getMapVars())
-        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
-      for (Value privateVar : targetOp.getPrivateVars())
-        collectRewrite(privateVar, rewriteValues);
-      for (Value threadLimit : targetOp.getThreadLimitVars())
-        collectRewrite(threadLimit, rewriteValues);
-    }
-
-    // Move omp.map.info ops to the new block and collect dependencies.
-    llvm::SetVector<fir::DeclareOp> declareOps;
-    llvm::SetVector<fir::BoxOffsetOp> boxOffsets;
-    for (omp::MapInfoOp mapOp : mapInfos) {
-      if (auto declareOp = dyn_cast_if_present<fir::DeclareOp>(
-              mapOp.getVarPtr().getDefiningOp()))
-        collectRewrite(declareOp, declareOps);
-      else
-        collectRewrite(mapOp.getVarPtr(), rewriteValues);
-
-      if (Value varPtrPtr = mapOp.getVarPtrPtr()) {
-        if (auto boxOffset = llvm::dyn_cast_if_present<fir::BoxOffsetOp>(
-                varPtrPtr.getDefiningOp()))
-          collectRewrite(boxOffset, boxOffsets);
-        else
-          return mapOp->emitOpError() << "var_ptr_ptr rewrite only supported "
-                                         "if defined by fir.box_offset";
-      }
-
-      // Bounds are not used during target device codegen.
-      mapOp.getBoundsMutable().clear();
-      mapOp->moveBefore(&block, block.end());
-    }
-
-    // Create a temporary marker to simplify the op moving process below.
-    builder.setInsertionPointToStart(&block);
-    auto marker = fir::UndefOp::create(builder, builder.getUnknownLoc(),
-                                       builder.getNoneType());
-    builder.setInsertionPoint(marker);
-
-    // Handle dependencies of fir.declare ops.
-    for (fir::DeclareOp declareOp : declareOps) {
-      collectRewrite(declareOp.getMemref(), rewriteValues);
-
-      if (declareOp.getStorage())
-        collectRewrite(declareOp.getStorage(), rewriteValues);
-
-      // Shape and typeparams aren't needed for target device codegen, but
-      // removing them would break verifiers.
-      Value zero;
-      if (declareOp.getShape() || !declareOp.getTypeparams().empty())
-        zero = arith::ConstantOp::create(builder, declareOp.getLoc(),
-                                         builder.getI64IntegerAttr(0));
-
-      if (auto shape = declareOp.getShape()) {
-        // The pre-cg rewrite pass requires the shape to be defined by one of
-        // fir.shape, fir.shapeshift or fir.shift, so we need to make sure it's
-        // still defined by one of these after this pass.
-        Operation *shapeOp = shape.getDefiningOp();
-        llvm::SmallVector<Value> extents(shapeOp->getNumOperands(), zero);
-        Value newShape =
-            llvm::TypeSwitch<Operation *, Value>(shapeOp)
-                .Case([&](fir::ShapeOp op) {
-                  return fir::ShapeOp::create(builder, op.getLoc(), extents);
-                })
-                .Case([&](fir::ShapeShiftOp op) {
-                  auto type = fir::ShapeShiftType::get(op.getContext(),
-                                                       extents.size() / 2);
-                  return fir::ShapeShiftOp::create(builder, op.getLoc(), type,
-                                                   extents);
-                })
-                .Case([&](fir::ShiftOp op) {
-                  auto type =
-                      fir::ShiftType::get(op.getContext(), extents.size());
-                  return fir::ShiftOp::create(builder, op.getLoc(), type,
-                                              extents);
-                })
-                .Default([](Operation *op) {
-                  op->emitOpError()
-                      << "fir.declare shape expected to be one of: "
-                         "fir.shape, fir.shapeshift or fir.shift";
-                  return nullptr;
-                });
-
-        if (!newShape)
-          return failure();
-
-        declareOp.getShapeMutable().assign(newShape);
-      }
-
-      for (OpOperand &typeParam : declareOp.getTypeparamsMutable())
-        typeParam.assign(zero);
-
-      declareOp.getDummyScopeMutable().clear();
-    }
-
-    // We don't actually need the proper initialization, but rather just
-    // maintain the basic form of these operands. Generally, we create 1-bit
-    // placeholder allocas that we "typecast" to the expected type and replace
-    // all uses. Using fir.undefined here instead is not possible because these
-    // variables cannot be constants, as that would trigger 
diff erent codegen
-    // for target regions.
-    for (Value value : rewriteValues) {
-      Location loc = value.getLoc();
-      Value rewriteValue;
-      if (isa_and_present<arith::ConstantOp, fir::AddrOfOp>(
-              value.getDefiningOp())) {
-        // If it's defined by fir.address_of, then we need to keep that op as
-        // well because it might be pointing to a 'declare target' global.
-        // Constants can also trigger 
diff erent codegen paths, so we keep them
-        // as well.
-        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
-      } else if (auto boxCharType =
-                     dyn_cast<fir::BoxCharType>(value.getType())) {
-        // !fir.boxchar types cannot be directly obtained by converting a
-        // !fir.ref<i1>, as they aren't reference types. Since they can appear
-        // representing some `target firstprivate` clauses, we need to create
-        // a special case here based on creating a placeholder fir.emboxchar op.
-        MLIRContext *ctx = &getContext();
-        fir::KindTy kind = boxCharType.getKind();
-        auto placeholder = fir::AllocaOp::create(
-            builder, loc, fir::CharacterType::getSingleton(ctx, kind));
-        auto one = arith::ConstantOp::create(builder, loc, builder.getI32Type(),
-                                             builder.getI32IntegerAttr(1));
-        rewriteValue = fir::EmboxCharOp::create(builder, loc, boxCharType,
-                                                placeholder, one);
-      } else {
-        Value placeholder =
-            fir::AllocaOp::create(builder, loc, builder.getI1Type());
-        rewriteValue =
-            fir::ConvertOp::create(builder, loc, value.getType(), placeholder);
-      }
-      value.replaceAllUsesWith(rewriteValue);
-    }
-
-    // Move omp.map.info dependencies.
-    for (fir::DeclareOp declareOp : declareOps)
-      declareOp->moveBefore(marker);
-
-    // The box_ref argument of fir.box_offset is expected to be the same value
-    // that was passed as var_ptr to the corresponding omp.map.info, so we don't
-    // need to handle its defining op here.
-    for (fir::BoxOffsetOp boxOffset : boxOffsets)
-      boxOffset->moveBefore(marker);
-
-    marker->erase();
-
-    // Move target operations to the end of the new block.
-    for (omp::TargetOp targetOp : targetOps)
-      targetOp->moveBefore(&block, block.end());
-
-    // Add terminator to the new block.
-    builder.setInsertionPointToEnd(&block);
-    llvm::SmallVector<Value> returnValues;
-    returnValues.reserve(funcOp.getNumResults());
-    for (auto type : funcOp.getResultTypes())
-      returnValues.push_back(
-          fir::UndefOp::create(builder, funcOp.getLoc(), type));
-
-    func::ReturnOp::create(builder, funcOp.getLoc(), returnValues);
-
-    // Replace old region (now missing ops) with the new one and remove the
-    // temporary operation clone.
-    region.takeBody(newOp->getRegion(0));
-    newOp->erase();
-    return success();
-  }
-};
-} // namespace

diff  --git a/flang/lib/Optimizer/Passes/Pipelines.cpp b/flang/lib/Optimizer/Passes/Pipelines.cpp
index a8e598ab95ac8..5fa70096e0e41 100644
--- a/flang/lib/Optimizer/Passes/Pipelines.cpp
+++ b/flang/lib/Optimizer/Passes/Pipelines.cpp
@@ -352,7 +352,6 @@ void createHLFIRToFIRPassPipeline(mlir::PassManager &pm,
   if (enableOpenMP != EnableOpenMP::None) {
     pm.addPass(flangomp::createLowerWorkshare());
     pm.addPass(flangomp::createLowerWorkdistribute());
-    pm.addPass(flangomp::createHostOpFilteringPass());
   }
   if (enableOpenMP == EnableOpenMP::Simd)
     pm.addPass(flangomp::createSimdOnlyPass());
@@ -462,11 +461,17 @@ void createDefaultFIRCodeGenPassPipeline(mlir::PassManager &pm,
 
   fir::addFIRToLLVMPass(pm, config);
 
-  // Convert applicable OpenMP stack allocations to shared memory allocations
-  // for GPU targets. This pass must run after any alloca-generating passes to
-  // ensure all are adequately accounted for.
-  if (config.EnableOpenMP && !config.EnableOpenMPSimd)
+  if (config.EnableOpenMP && !config.EnableOpenMPSimd) {
+    // Remove all non target-related operations from host functions still
+    // remaining at this point, if compiling for an OpenMP target device. This
+    // is required before translating 'omp' dialect operations to LLVM IR.
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
+
+    // Convert applicable OpenMP stack allocations to shared memory allocations
+    // for GPU targets. This pass must run after any alloca-generating passes to
+    // ensure all are adequately accounted for.
     pm.addPass(mlir::omp::createStackToSharedPass());
+  }
 }
 
 /// Create a pass pipeline for lowering from MLIR to LLVM IR

diff  --git a/flang/test/Fir/basic-program.fir b/flang/test/Fir/basic-program.fir
index cbf45b216a911..471e5a89046eb 100644
--- a/flang/test/Fir/basic-program.fir
+++ b/flang/test/Fir/basic-program.fir
@@ -85,7 +85,6 @@ func.func @_QQmain() {
 // PASSES-NEXT:   ConvertHLFIRtoFIR
 // PASSES-NEXT:   LowerWorkshare
 // PASSES-NEXT:   LowerWorkdistribute
-// PASSES-NEXT:   HostOpFilteringPass
 // PASSES-NEXT:   CSE
 // PASSES-NEXT:   (S) 0 num-cse'd - Number of operations CSE'd
 // PASSES-NEXT:   (S) 0 num-dce'd - Number of operations DCE'd
@@ -185,6 +184,7 @@ func.func @_QQmain() {
 // PASSES-NEXT:  LowerNontemporalPass
 // PASSES-NEXT: FIRToLLVMLowering
 // PASSES-NEXT: ReconcileUnrealizedCasts
+// PASSES-NEXT:  HostOpFilteringPass
 // PASSES-NEXT: 'llvm.func' Pipeline
 // PASSES-NEXT:  StackToSharedPass
 // PASSES-NEXT: PrepareForOMPOffloadPrivatizationPass

diff  --git a/flang/test/Lower/OpenMP/function-filtering-4.f90 b/flang/test/Lower/OpenMP/function-filtering-4.f90
new file mode 100644
index 0000000000000..de8a26c789dc5
--- /dev/null
+++ b/flang/test/Lower/OpenMP/function-filtering-4.f90
@@ -0,0 +1,63 @@
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-HOST,LLVM-ALL %s
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-llvm %s -o - | FileCheck --check-prefixes=LLVM-DEVICE,LLVM-ALL %s %}
+! RUN: %if amdgpu-registered-target %{ %flang_fc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
+! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
+
+! Check that the correct LLVM IR operations are kept for the host and device
+! after running the whole set of translation and transformation passes from
+! Fortran for a type-bound procedure.
+
+module mymodule
+  implicit none
+
+  type :: myclass
+  contains
+    procedure :: myfunc => myfunc
+  end type myclass
+
+contains
+  ! MLIR-HOST: func.func @_QMmymodulePmyfunc
+  ! MLIR-DEVICE-NOT: func.func @_QMmymodulePmyfunc
+
+  ! LLVM-HOST: define void @_QMmymodulePmyfunc
+  ! LLVM-DEVICE-NOT: define void @_QMmymodulePmyfunc
+  subroutine myfunc(self)
+    class(myclass) :: self
+    call foo()
+  end subroutine
+end module
+
+! MLIR-ALL: func.func @_QPmain
+
+! LLVM-HOST: define void @main_
+! LLVM-DEVICE-NOT: define void @main_
+subroutine main(x)
+  use mymodule
+  implicit none
+
+  integer, intent(inout) :: x
+  class(myclass), allocatable :: myobj
+  allocate(myobj)
+
+  ! MLIR-ALL: fir.dispatch "myfunc"
+
+  ! LLVM-HOST: %[[MYFUNC_PTR:.*]] = inttoptr i64 %{{.*}} to ptr
+  ! LLVM-HOST: call void %[[MYFUNC_PTR]](ptr %{{.*}})
+  ! LLVM-DEVICE-NOT: call void %{{.*}}(ptr %{{.*}})
+  call myobj%myfunc()
+
+  !$omp target map(tofrom: x)
+  x = x + 1
+  !$omp end target
+
+  deallocate(myobj)
+
+  ! LLVM-HOST: ret void
+end subroutine main
+
+! LLVM-ALL: define {{.*}}void @__omp_offloading{{.*}}main_{{.*}}
+
+! MLIR-ALL: fir.type_info @_QMmymoduleTmyclass {{.*}} dispatch_table
+! MLIR-ALL-NEXT: fir.dt_entry "myfunc", @_QMmymodulePmyfunc

diff  --git a/flang/test/Lower/OpenMP/host-eval.f90 b/flang/test/Lower/OpenMP/host-eval.f90
index 3bd873009530c..35749a6ac5050 100644
--- a/flang/test/Lower/OpenMP/host-eval.f90
+++ b/flang/test/Lower/OpenMP/host-eval.f90
@@ -22,10 +22,8 @@ subroutine teams()
 
   !$omp end target
 
-  ! HOST: omp.teams
-  ! HOST-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
-
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
+  ! BOTH-SAME: num_teams({{.*}}) thread_limit({{.*}}) {
   !$omp teams num_teams(1) thread_limit(2)
   call foo()
   !$omp end teams
@@ -78,18 +76,13 @@ subroutine distribute_parallel_do()
   !$omp end distribute parallel do
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-
-  ! DEVICE-NOT: omp.parallel
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.wsloop
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
   !$omp distribute parallel do num_threads(1)
   do i=1,10
     call foo()
@@ -147,15 +140,14 @@ subroutine distribute_parallel_do_simd()
   !$omp end distribute parallel do simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.parallel
-  ! HOST-SAME: num_threads({{.*}})
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.wsloop
-  ! HOST-NEXT: omp.simd
+  ! BOTH: omp.parallel
+  ! BOTH-SAME: num_threads({{.*}})
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.wsloop
+  ! BOTH-NEXT: omp.simd
 
   ! DEVICE-NOT: omp.parallel
   ! DEVICE-NOT: omp.distribute
@@ -207,12 +199,10 @@ subroutine distribute()
   !$omp end distribute
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! DEVICE-NOT: omp.distribute
+  ! BOTH: omp.distribute
   !$omp distribute
   do i=1,10
     call foo()
@@ -261,15 +251,11 @@ subroutine distribute_simd()
   !$omp end distribute simd
   !$omp end target teams
 
-  ! HOST: omp.teams
-  ! DEVICE-NOT: omp.teams
+  ! BOTH: omp.teams
   !$omp teams
 
-  ! HOST: omp.distribute
-  ! HOST-NEXT: omp.simd
-
-  ! DEVICE-NOT: omp.distribute
-  ! DEVICE-NOT: omp.simd
+  ! BOTH: omp.distribute
+  ! BOTH-NEXT: omp.simd
   !$omp distribute simd
   do i=1,10
     call foo()

diff  --git a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir b/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
deleted file mode 100644
index 3acdfda876812..0000000000000
--- a/flang/test/Transforms/OpenMP/function-filtering-host-ops.mlir
+++ /dev/null
@@ -1,560 +0,0 @@
-// RUN: fir-opt --omp-host-op-filtering %s | FileCheck %s
-
-module attributes {omp.is_target_device = true} {
-  // CHECK-LABEL: func.func private @external
-  // CHECK-NOT: return
-  func.func private @external()
-
-  // CHECK-LABEL: func.func @basic_checks
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.ref<i32>) -> (i32, f32)
-  func.func @basic_checks(%arg: !fir.ref<i32>) -> (i32, f32) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOC:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %r0 = arith.constant 10 : i32
-    %r1 = arith.constant 2.5 : f32
-
-    func.call @foo() : () -> ()
-
-    %0 = fir.declare %arg {uniq_name = "arg"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %alloc = fir.alloca i32
-    %2 = fir.declare %alloc {uniq_name = "alloc"} : (!fir.ref<i32>) -> !fir.ref<i32>
-
-    // CHECK-NEXT: %[[ALLOC_DECL:.*]] = fir.declare %[[ALLOC]] {uniq_name = "alloc"}
-    // CHECK-NEXT: %[[ARG_DECL:.*]] = fir.declare %[[ARG]] {uniq_name = "arg"}
-    // CHECK-NEXT: %[[GLOBAL_DECL:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOC_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL]]{{.*}})
-    // CHECK-NEXT: %[[MAP3:.*]] = omp.map.info var_ptr(%[[ALLOC]]{{.*}})
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[MAP2]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP3]] -> {{.*}} : {{.*}})
-    omp.target kernel_type(generic) has_device_addr(%m2 -> %arg0 : !fir.ref<i32>) map_entries(%m0 -> %arg1, %m1 -> %arg2, %m3 -> %arg3 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NEXT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: func.call
-    // CHECK-NOT: omp.map.info
-    omp.parallel {
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.map.info
-    %m4 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m5 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m6 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m7 = omp.map.info var_ptr(%alloc : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m4, %m5, %m6, %m7 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>, !fir.ref<i32>) {
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.target_enter_data
-    // CHECK-NOT: omp.target_exit_data
-    // CHECK-NOT: omp.target_update
-    %m8 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_enter_data map_entries(%m8 : !fir.ref<i32>)
-
-    %m9 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(from) capture(ByRef) -> !fir.ref<i32>
-    omp.target_exit_data map_entries(%m9 : !fir.ref<i32>)
-
-    %m10 = omp.map.info var_ptr(%2 : !fir.ref<i32>, !fir.ref<i32>) map_clauses(to) capture(ByRef) -> !fir.ref<i32>
-    omp.target_update map_entries(%m10 : !fir.ref<i32>)
-
-    // CHECK-NOT: func.call
-    func.call @foo() : () -> ()
-
-    // CHECK:      %[[RETURN0:.*]] = fir.undefined i32
-    // CHECK-NEXT: %[[RETURN1:.*]] = fir.undefined f32
-    // CHECK-NEXT: return %[[RETURN0]], %[[RETURN1]]
-    return %r0, %r1 : i32, f32
-  }
-
-  // CHECK-LABEL: func.func @allocatable_array
-  // CHECK-SAME: (%[[ALLOCATABLE:.*]]: [[ALLOCATABLE_TYPE:.*]], %[[ARRAY:.*]]: [[ARRAY_TYPE:[^)]*]])
-  func.func @allocatable_array(%allocatable: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, %array: !fir.ref<!fir.array<9xi32>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape %[[ZERO]] : (i64) -> !fir.shape<1>
-    // CHECK-NEXT: %[[ALLOCATABLE_DECL:.*]] = fir.declare %[[ALLOCATABLE]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : ([[ALLOCATABLE_TYPE]]) -> [[ALLOCATABLE_TYPE]]
-    // CHECK-NEXT: %[[ARRAY_DECL:.*]] = fir.declare %[[ARRAY]](%[[SHAPE]]) {uniq_name = "array"} : ([[ARRAY_TYPE]], !fir.shape<1>) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCATABLE_DECL]] base_addr : ([[ALLOCATABLE_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP_ALLOCATABLE:.*]] = omp.map.info var_ptr(%[[ALLOCATABLE_DECL]] : [[ALLOCATABLE_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP_ARRAY:.*]] = omp.map.info var_ptr(%[[ARRAY_DECL]] : [[ARRAY_TYPE]], !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) -> [[ARRAY_TYPE]]
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP_ALLOCATABLE]] -> %{{.*}}, %[[MAP_ARRAY]] -> %{{.*}} : [[VAR_PTR_PTR_TYPE]], [[ARRAY_TYPE]])
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c8 = arith.constant 8 : index
-    %c9 = arith.constant 9 : index
-
-    %0 = fir.declare %allocatable {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "allocatable"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %1 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %2 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%2 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%1) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-
-    %3 = fir.shape %c9 : (index) -> !fir.shape<1>
-    %4 = fir.declare %array(%3) {uniq_name = "array"} : (!fir.ref<!fir.array<9xi32>>, !fir.shape<1>) -> !fir.ref<!fir.array<9xi32>>
-    %5 = omp.map.bounds lower_bound(%c0 : index) upper_bound(%c8 : index) extent(%c9 : index) stride(%c1 : index) start_idx(%c1 : index)
-    %6 = omp.map.info var_ptr(%4 : !fir.ref<!fir.array<9xi32>>, !fir.array<9xi32>) map_clauses(tofrom) capture(ByRef) bounds(%5) -> !fir.ref<!fir.array<9xi32>>
-
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg0, %6 -> %arg1 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, !fir.ref<!fir.array<9xi32>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @character
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @character(%x: !fir.ref<!fir.char<1>>) {
-    // CHECK-NEXT: %[[ZERO]] = arith.constant 0 : i64
-    %0 = fir.dummy_scope : !fir.dscope
-    %c1 = arith.constant 1 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] typeparams %[[ZERO]] {uniq_name = "x"} : ([[X_TYPE]], i64) -> [[X_TYPE]]
-    %3 = fir.declare %x typeparams %c1 dummy_scope %0 {uniq_name = "x"} : (!fir.ref<!fir.char<1>>, index, !fir.dscope) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> [[X_TYPE]]
-    %map = omp.map.info var_ptr(%3 : !fir.ref<!fir.char<1>>, !fir.char<1>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.char<1>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}})
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<!fir.char<1>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_rank
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @assumed_rank(%x: !fir.box<!fir.array<*:f32>>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<[[X_TYPE]]>
-    %0 = fir.alloca !fir.box<!fir.array<*:f32>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {uniq_name = "x"} : (!fir.box<!fir.array<*:f32>>, !fir.dscope) -> !fir.box<!fir.array<*:f32>>
-    %3 = fir.rebox_assumed_rank %2 lbs ones : (!fir.box<!fir.array<*:f32>>) -> !fir.box<!fir.array<*:f32>>
-    %4 = fir.box_addr %3 : (!fir.box<!fir.array<*:f32>>) -> !fir.ref<!fir.array<*:f32>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA]] base_addr : (!fir.ref<[[X_TYPE]]>) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %5 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.array<*:f32>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], !fir.array<*:f32>) -> [[VAR_PTR_PTR_TYPE]]
-    %6 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%5 : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>, !fir.array<*:f32>) -> !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA]] : !fir.ref<[[X_TYPE]]>, !fir.box<!fir.array<*:f32>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> !fir.ref<!fir.array<*:f32>>
-    %7 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.array<*:f32>>>, !fir.box<!fir.array<*:f32>>) map_clauses(to) capture(ByRef) members(%6 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) -> !fir.ref<!fir.array<*:f32>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> {{.*}})
-    omp.target kernel_type(generic) map_entries(%7 -> %arg1, %6 -> %arg2 : !fir.ref<!fir.array<*:f32>>, !fir.llvm_ptr<!fir.ref<!fir.array<*:f32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @box_ptr
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @box_ptr(%x: !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) {
-    // CHECK-NEXT: %[[ZERO:.*]] = arith.constant 0 : i64
-    // CHECK-NEXT: %[[SHAPE:.*]] = fir.shape_shift %[[ZERO]], %[[ZERO]] : (i64, i64) -> !fir.shapeshift<1>
-    // CHECK-NEXT: %[[PLACEHOLDER_X:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_X:.*]] = fir.convert %[[PLACEHOLDER_X]] : (!fir.ref<i1>) -> [[X_TYPE]]
-    %0 = fir.alloca !fir.box<!fir.ptr<!fir.array<?xi32>>>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2 = fir.declare %x dummy_scope %1 {fortran_attrs = #fir.var_attrs<contiguous, pointer>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.dscope) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    %3 = fir.load %2 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    fir.store %3 to %0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-
-    // CHECK-NEXT: %[[PLACEHOLDER_Y:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA_Y:.*]] = fir.convert %[[PLACEHOLDER_Y]] : (!fir.ref<i1>) -> [[Y_TYPE:.*]]
-    %c0 = arith.constant 0 : index
-    %4:3 = fir.box_dims %3, %c0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c1 = arith.constant 1 : index
-    %c0_0 = arith.constant 0 : index
-    %5:3 = fir.box_dims %3, %c0_0 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %c0_1 = arith.constant 0 : index
-    %6 = arith.subi %5#1, %c1 : index
-    %7 = omp.map.bounds lower_bound(%c0_1 : index) upper_bound(%6 : index) extent(%5#1 : index) stride(%5#2 : index) start_idx(%4#0 : index) {stride_in_bytes = true}
-    %8 = fir.box_addr %3 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>) -> !fir.ptr<!fir.array<?xi32>>
-    %c0_2 = arith.constant 0 : index
-    %9:3 = fir.box_dims %3, %c0_2 : (!fir.box<!fir.ptr<!fir.array<?xi32>>>, index) -> (index, index, index)
-    %10 = fir.shape_shift %9#0, %9#1 : (index, index) -> !fir.shapeshift<1>
-    
-    // CHECK-NEXT: %[[Y_DECL:.*]] = fir.declare %[[ALLOCA_Y]](%[[SHAPE]]) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : ([[Y_TYPE]], !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %11 = fir.declare %8(%10) {fortran_attrs = #fir.var_attrs<target>, uniq_name = "y"} : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.ptr<!fir.array<?xi32>>
-    %12 = fir.embox %11(%10) : (!fir.ptr<!fir.array<?xi32>>, !fir.shapeshift<1>) -> !fir.box<!fir.array<?xi32>>
-    %c1_3 = arith.constant 1 : index
-    %c0_4 = arith.constant 0 : index
-    %13:3 = fir.box_dims %12, %c0_4 : (!fir.box<!fir.array<?xi32>>, index) -> (index, index, index)
-    %c0_5 = arith.constant 0 : index
-    %14 = arith.subi %13#1, %c1_3 : index
-    %15 = omp.map.bounds lower_bound(%c0_5 : index) upper_bound(%14 : index) extent(%13#1 : index) stride(%13#2 : index) start_idx(%9#0 : index) {stride_in_bytes = true}
-    
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[ALLOCA_X]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[Y_DECL]] : [[Y_TYPE]], i32) {{.*}} -> [[Y_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], i32) -> [[VAR_PTR_PTR_TYPE]]
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ALLOCA_X]] : [[X_TYPE]], !fir.box<!fir.ptr<!fir.array<?xi32>>>) {{.*}} members(%[[MAP1]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %16 = omp.map.info var_ptr(%11 : !fir.ptr<!fir.array<?xi32>>, i32) map_clauses(tofrom) capture(ByRef) bounds(%15) -> !fir.ptr<!fir.array<?xi32>>
-    %17 = fir.box_offset %0 base_addr : (!fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %18 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) var_ptr_ptr(%17 : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) bounds(%7) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
-    %19 = omp.map.info var_ptr(%0 : !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.box<!fir.ptr<!fir.array<?xi32>>>) map_clauses(implicit, to) capture(ByRef) members(%18 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) -> !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>
-    
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP2]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : [[Y_TYPE]], [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%16 -> %arg1, %19 -> %arg2, %18 -> %arg3 : !fir.ptr<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.ptr<!fir.array<?xi32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @target_data
-  // CHECK-SAME: (%[[MAPPED:.*]]: [[MAPPED_TYPE:[^)]*]], %[[USEDEVADDR:.*]]: [[USEDEVADDR_TYPE:[^)]*]], %[[USEDEVPTR:.*]]: [[USEDEVPTR_TYPE:[^)]*]])
-  func.func @target_data(%mapped: !fir.ref<i32>, %usedevaddr: !fir.ref<i32>, %usedevptr: !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-    // CHECK-NEXT: %[[MAPPED_DECL:.*]] = fir.declare %[[MAPPED]] {uniq_name = "mapped"} : ([[MAPPED_TYPE]]) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_DECL:.*]] = fir.declare %[[USEDEVADDR]] {uniq_name = "usedevaddr"} : ([[USEDEVADDR_TYPE]]) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_DECL:.*]] = fir.declare %[[USEDEVPTR]] {uniq_name = "usedevptr"} : ([[USEDEVPTR_TYPE]]) -> [[USEDEVPTR_TYPE]]
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %1 = fir.declare %usedevaddr {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %2 = fir.declare %usedevptr {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-    // CHECK-NEXT: %[[MAPPED_MAP:.*]] = omp.map.info var_ptr(%[[MAPPED_DECL]] : [[MAPPED_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[MAPPED_TYPE]]
-    // CHECK-NEXT: %[[USEDEVADDR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVADDR_DECL]] : [[USEDEVADDR_TYPE]], i32) map_clauses(tofrom) capture(ByRef) -> [[USEDEVADDR_TYPE]]
-    // CHECK-NEXT: %[[USEDEVPTR_MAP:.*]] = omp.map.info var_ptr(%[[USEDEVPTR_DECL]] : [[USEDEVPTR_TYPE]], !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> [[USEDEVPTR_TYPE]]
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(return_param) capture(ByRef) -> !fir.ref<i32>
-    %m2 = omp.map.info var_ptr(%2 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(return_param) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-    
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) use_device_addr(%m1 -> %arg0 : !fir.ref<i32>) use_device_ptr(%m2 -> %arg1 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-      %3 = fir.declare %arg0 {uniq_name = "usedevaddr"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %4 = fir.declare %arg1 {uniq_name = "usedevptr"} : (!fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-      %m3 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m4 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m5 = omp.map.info var_ptr(%4 : !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>, !fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>) map_clauses(tofrom) capture(ByRef) -> !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAPPED_MAP]] -> %{{.*}}, %[[USEDEVADDR_MAP]] -> %{{.*}}, %[[USEDEVPTR_MAP]] -> %{{.*}} : {{.*}})
-      omp.target kernel_type(generic) map_entries(%m3 -> %arg2, %m4 -> %arg3, %m5 -> %arg4 : !fir.ref<i32>, !fir.ref<i32>, !fir.ref<!fir.type<_QM__fortran_builtinsT__builtin_c_ptr{__address:i64}>>) {
-        omp.terminator
-      }
-
-      // CHECK-NOT: func.call
-      func.call @foo() : () -> ()
-
-      omp.terminator
-    }
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @no_target
-  // CHECK-SAME: (%[[MAPPED:.*]]: !fir.ref<i32>)
-  func.func @no_target(%mapped: !fir.ref<i32>) {
-    // CHECK-NEXT: return
-    %0 = fir.declare %mapped {uniq_name = "mapped"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        func.call @foo() : () -> ()
-        omp.terminator
-      }
-      func.call @foo() : () -> ()
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @map_info_members
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @map_info_members(%x: !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) {
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c9 = arith.constant 9 : index
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    %23 = fir.declare %x {fortran_attrs = #fir.var_attrs<allocatable>, uniq_name = "x"} : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %63 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %64:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %65:3 = fir.box_dims %63, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %66 = arith.subi %c1, %64#0 : index
-    %67 = arith.subi %c9, %64#0 : index
-    %68 = fir.load %23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    %69:3 = fir.box_dims %68, %c0 : (!fir.box<!fir.heap<!fir.array<?xf32>>>, index) -> (index, index, index)
-    %70 = omp.map.bounds lower_bound(%66 : index) upper_bound(%67 : index) extent(%69#1 : index) stride(%65#2 : index) start_idx(%64#0 : index) {stride_in_bytes = true}
-    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = fir.box_offset %[[X_DECL]] base_addr : ([[X_TYPE]]) -> [[VAR_PTR_PTR_TYPE:.*]]
-    %71 = fir.box_offset %23 base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} var_ptr_ptr(%[[VAR_PTR_PTR]] : [[VAR_PTR_PTR_TYPE]], f32) -> [[VAR_PTR_PTR_TYPE]]
-    %72 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%71 : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>, f32) bounds(%70) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], !fir.box<!fir.heap<!fir.array<?xf32>>>) {{.*}} members(%[[MAP0]] : [0] : [[VAR_PTR_PTR_TYPE]]) -> [[X_TYPE]]
-    %73 = omp.map.info var_ptr(%23 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.box<!fir.heap<!fir.array<?xf32>>>) map_clauses(to) capture(ByRef) members(%72 : [0] : !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> %{{.*}} : [[X_TYPE]], [[VAR_PTR_PTR_TYPE]])
-    omp.target kernel_type(generic) map_entries(%73 -> %arg0, %72 -> %arg1 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xf32>>>>, !fir.llvm_ptr<!fir.ref<!fir.array<?xf32>>>) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @control_flow
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^,]*]], %[[COND:.*]]: [[COND_TYPE:[^)]*]])
-  func.func @control_flow(%x: !fir.ref<i32>, %cond: !fir.ref<!fir.logical<4>>) {
-    // CHECK-NEXT: %[[X_DECL:.*]] = fir.declare %[[X]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %cond_decl = fir.declare %cond {uniq_name = "cond"} : (!fir.ref<!fir.logical<4>>) -> !fir.ref<!fir.logical<4>>
-    %0 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-    %1 = fir.convert %0 : (!fir.logical<4>) -> i1
-    cf.cond_br %1, ^bb1, ^bb2
-  ^bb1:  // pred: ^bb0
-    fir.call @foo() : () -> ()
-    %m0 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-    omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-    cf.br ^bb2
-  ^bb2:  // 2 preds: ^bb0, ^bb1
-    // CHECK-NOT: fir.call
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    fir.call @foo() : () -> ()
-    %m1 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-      fir.call @foo() : () -> ()
-      %8 = fir.load %cond_decl : !fir.ref<!fir.logical<4>>
-      %9 = fir.convert %8 : (!fir.logical<4>) -> i1
-      cf.cond_br %9, ^bb1, ^bb2
-    ^bb1:  // pred: ^bb0
-      fir.call @foo() : () -> ()
-      %m2 = omp.map.info var_ptr(%x_decl : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      // CHECK-NOT: fir.call
-      // CHECK-NOT: cf.br
-      fir.call @foo() : () -> ()
-      cf.br ^bb2
-    ^bb2:  // 2 preds: ^bb0, ^bb1
-      fir.call @foo() : () -> ()
-      omp.terminator
-    }
-    fir.call @foo() : () -> ()
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @block_args
-  // CHECK-SAME: (%[[X:.*]]: [[X_TYPE:[^)]*]])
-  func.func @block_args(%x: !fir.ref<i32>) {
-    // CHECK-NEXT: %[[PLACEHOLDER0:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA0:.*]] = fir.convert %[[PLACEHOLDER0]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[PLACEHOLDER1:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[ALLOCA1:.*]] = fir.convert %[[PLACEHOLDER1]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[X_DECL0:.*]] = fir.declare %[[ALLOCA0]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[X_DECL1:.*]] = fir.declare %[[ALLOCA1]] {uniq_name = "x"} : ([[X_TYPE]]) -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[X_DECL0]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[X_DECL1]] : [[X_TYPE]], i32) {{.*}} -> [[X_TYPE]]
-    %x_decl = fir.declare %x {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %0 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> {{.*}} : [[X_TYPE]])
-      omp.target kernel_type(generic) map_entries(%m0 -> %arg2 : !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    // CHECK-NOT: omp.parallel
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    // CHECK-NOT: omp.target_data
-    omp.parallel private(@privatizer %x_decl -> %arg0 : !fir.ref<i32>) {
-      %1 = fir.declare %arg0 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      omp.target_data map_entries(%m1 : !fir.ref<i32>) {
-        omp.parallel private(@privatizer %1 -> %arg1 : !fir.ref<i32>) {
-          %2 = fir.declare %arg1 {uniq_name = "x"} : (!fir.ref<i32>) -> !fir.ref<i32>
-          %m2 = omp.map.info var_ptr(%2 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}} : [[X_TYPE]])
-          omp.target kernel_type(generic) map_entries(%m2 -> %arg2 : !fir.ref<i32>) {
-            omp.terminator
-          }
-          omp.terminator
-        }
-        omp.terminator
-      }
-      omp.terminator
-    }
-
-    return
-  }
-
-  // CHECK-LABEL: func.func @reuse_tests()
-  func.func @reuse_tests() {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[THREAD_LIMIT:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> i32
-    // CHECK-NEXT: %[[CONST:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@global_scalar) : !fir.ref<i32>
-    %global = fir.address_of(@global_scalar) : !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL_DECL0:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL1:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[GLOBAL_DECL2:.*]] = fir.declare %[[GLOBAL]] {uniq_name = "global_scalar"}
-    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL0]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL1]] : !fir.ref<i32>, i32)
-    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL_DECL2]] : !fir.ref<i32>, i32)
-    %0 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m0 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data map_entries(%m0 : !fir.ref<i32>) {
-      %1 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-      %m1 = omp.map.info var_ptr(%0 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      %m2 = omp.map.info var_ptr(%1 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> {{.*}} : !fir.ref<i32>, !fir.ref<i32>)
-      omp.target kernel_type(generic) map_entries(%m1 -> %arg0, %m2 -> %arg1 : !fir.ref<i32>, !fir.ref<i32>) {
-        omp.terminator
-      }
-      omp.terminator
-    }
-    // CHECK-NOT: fir.load
-    // CHECK-NOT: fir.declare
-    // CHECK-NOT: omp.map.info
-    %2 = fir.load %global : !fir.ref<i32>
-    %3 = fir.declare %global {uniq_name = "global_scalar"} : (!fir.ref<i32>) -> !fir.ref<i32>
-    %m3 = omp.map.info var_ptr(%3 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) thread_limit(%2 : i32) map_entries(%m3 -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    %c1 = arith.constant 1 : i32
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST]] : i32)
-    omp.target kernel_type(generic) thread_limit(%c1 : i32) {
-      omp.terminator
-    }
-    return
-  }
-
-  // CHECK-LABEL: func.func @all_non_map_clauses
-  // CHECK-SAME: (%[[REF:.*]]: !fir.ref<i32>, %[[INT:.*]]: i32, %[[BOOL:.*]]: i1)
-  func.func @all_non_map_clauses(%ref: !fir.ref<i32>, %int: i32, %bool: i1) {
-    %m0 = omp.map.info var_ptr(%ref : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NOT: omp.target_data
-    omp.target_data device(%int : i32) if(%bool) map_entries(%m0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate({{[^)]*}}) thread_limit({{[^)]*}}) in_reduction({{[^)]*}}) private({{[^)]*}}) {
-    omp.target kernel_type(generic)
-               allocate(%ref : !fir.ref<i32> -> %ref : !fir.ref<i32>)
-               depend(taskdependin -> %ref : !fir.ref<i32>)
-               device(%int : i32) if(%bool) thread_limit(%int : i32)
-               in_reduction(@reduction %ref -> %arg0 : !fir.ref<i32>)
-               private(@privatizer %ref -> %arg1 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    // CHECK-NOT: omp.target_enter_data
-    omp.target_enter_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                          device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_exit_data
-    omp.target_exit_data depend(taskdependin -> %ref : !fir.ref<i32>)
-                         device(%int : i32) if(%bool)
-    // CHECK-NOT: omp.target_update
-    omp.target_update depend(taskdependin -> %ref : !fir.ref<i32>)
-                      device(%int : i32) if(%bool)
-
-    // CHECK: return
-    return
-  }
-
-  // CHECK-LABEL: func.func @assumed_length
-  // CHECK-SAME: (%[[ARG:.*]]: !fir.boxchar<1>)
-  func.func @assumed_length(%arg: !fir.boxchar<1>) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca !fir.char<1>
-    // CHECK-NEXT: %[[ONE:.*]] = arith.constant 1 : i32
-    // CHECK-NEXT: %[[EMBOXCHAR:.*]] = fir.emboxchar %[[PLACEHOLDER]], %[[ONE]] : (!fir.ref<!fir.char<1>>, i32) -> !fir.boxchar<1>
-    // CHECK-NEXT: omp.target kernel_type(generic) private(@boxchar_firstprivatizer %[[EMBOXCHAR]] -> %{{.*}} [map_idx=0] : !fir.boxchar<1>)
-    %0 = fir.alloca !fir.boxchar<1>
-    %1 = fir.dummy_scope : !fir.dscope
-    %2:2 = fir.unboxchar %arg : (!fir.boxchar<1>) -> (!fir.ref<!fir.char<1,?>>, index)
-    %3 = fir.declare %2#0 typeparams %2#1 dummy_scope %1 {uniq_name = "arg"} : (!fir.ref<!fir.char<1,?>>, index, !fir.dscope) -> !fir.ref<!fir.char<1,?>>
-    %4 = fir.emboxchar %3, %2#1 : (!fir.ref<!fir.char<1,?>>, index) -> !fir.boxchar<1>
-    omp.target kernel_type(generic) private(@boxchar_firstprivatizer %4 -> %arg3 [map_idx=0] : !fir.boxchar<1>) {
-      omp.terminator
-    }
-    return
-  }
-
-  omp.private {type = firstprivate} @boxchar_firstprivatizer : !fir.boxchar<1> copy {
-  ^bb0(%arg0: !fir.boxchar<1>, %arg1: !fir.boxchar<1>):
-    omp.yield(%arg0 : !fir.boxchar<1>)
-  }
-
-  // CHECK-LABEL: func.func @declare_storage
-  func.func @declare_storage(%i : index) {
-    // CHECK-NEXT: %[[PLACEHOLDER:.*]] = fir.alloca i1
-    // CHECK-NEXT: %[[VAR:.*]] = fir.convert %[[PLACEHOLDER]] : (!fir.ref<i1>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[GLOBAL:.*]] = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %0 = fir.address_of(@block_) : !fir.ref<!fir.array<8xi8>>
-    %1 = fir.convert %0 : (!fir.ref<!fir.array<8xi8>>) -> !fir.ref<!fir.array<?xi8>>
-    %2 = fir.coordinate_of %1, %i : (!fir.ref<!fir.array<?xi8>>, index) -> !fir.ref<i8>
-    %3 = fir.convert %2 : (!fir.ref<i8>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[DECL:.*]] = fir.declare %[[VAR]] storage(%[[GLOBAL]][0])
-    %4 = fir.declare %3 storage (%0[0]) {uniq_name = "a"} : (!fir.ref<i32>, !fir.ref<!fir.array<8xi8>>) -> !fir.ref<i32>
-    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[DECL]] : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    %map = omp.map.info var_ptr(%4 : !fir.ref<i32>, i32) map_clauses(tofrom) capture(ByRef) -> !fir.ref<i32>
-    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !fir.ref<i32>)
-    omp.target kernel_type(generic) map_entries(%map -> %arg0 : !fir.ref<i32>) {
-      omp.terminator
-    }
-    return
-  }
-
-  func.func private @foo() -> () attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>}
-  omp.private {type = firstprivate} @privatizer : i32 copy {
-  ^bb0(%arg0: !fir.ref<i32>, %arg1: !fir.ref<i32>):
-    %0 = fir.load %arg0 : !fir.ref<i32>
-    fir.store %0 to %arg1 : !fir.ref<i32>
-    omp.yield(%arg1 : !fir.ref<i32>)
-  }
-  omp.declare_reduction @reduction : i32
-  init {
-  ^bb0(%arg: i32):
-    %0 = arith.constant 0 : i32
-    omp.yield (%0 : i32)
-  }
-  combiner {
-  ^bb1(%arg0: i32, %arg1: i32):
-    %1 = arith.addi %arg0, %arg1 : i32
-    omp.yield (%1 : i32)
-  }
-
-  // CHECK: fir.global internal @global_scalar constant : i32
-  fir.global @global_scalar constant : i32
-
-  // CHECK: fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  // CHECK: fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
-  fir.global @declare_target_enter_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
-
-  // CHECK: fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  // CHECK: fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_any {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_host {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
-  fir.global @declare_target_link_nohost {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
-}

diff  --git a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
index e6321ef58b45f..da90ef508ce64 100644
--- a/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenMP/Transforms/Passes.td
@@ -11,6 +11,30 @@
 
 include "mlir/Pass/PassBase.td"
 
+def HostOpFilteringPass : Pass<"omp-host-op-filter"> {
+  let summary = "Removes all non target-related ops from host functions.";
+  let description = [{
+    When compiling for an OpenMP offloading target, host functions holding
+    target regions must be maintained in order for the `omp.target` ops to be
+    translated to LLVM IR as functions.
+    
+    However, these host functions might contain more host code than just what's
+    necessary for generating code for their contained target regions on the
+    target device, potentially causing problematic artifacts to be generated
+    into the target device's module.
+
+    This pass strips out all non-essential operations from host functions when
+    compiling for an OpenMP target device, leaving just `omp.target` operations,
+    related `omp.map.info` and placeholders in the form of function arguments to
+    keep the MLIR representation legal.
+
+    It must run on a module at a low level of abstraction (i.e. directly
+    translatable to LLVM IR), after any passes that might produce new OpenMP
+    operations have already completed.
+  }];
+  let dependentDialects = ["LLVM::LLVMDialect"];
+}
+
 def MarkDeclareTargetPass : Pass<"omp-mark-declare-target", "ModuleOp"> {
   let summary = "Marks all functions called by an OpenMP declare target "
                 "function as declare target";

diff  --git a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
index 569786fe95cf3..bbdbc556fc52f 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenMP/Transforms/CMakeLists.txt
@@ -1,4 +1,5 @@
 add_mlir_dialect_library(MLIROpenMPTransforms
+  HostOpFiltering.cpp
   MarkDeclareTarget.cpp
   OpenMPOffloadPrivatizationPrepare.cpp
   StackToShared.cpp

diff  --git a/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
new file mode 100644
index 0000000000000..9c61b2717af57
--- /dev/null
+++ b/mlir/lib/Dialect/OpenMP/Transforms/HostOpFiltering.cpp
@@ -0,0 +1,292 @@
+//===- HostOpFiltering.cpp ------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements transforms to swap stack allocations on the target
+// device with device shared memory where applicable.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h"
+
+#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/OpenMP/OpenMPDialect.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
+
+namespace mlir {
+namespace omp {
+#define GEN_PASS_DEF_HOSTOPFILTERINGPASS
+#include "mlir/Dialect/OpenMP/Transforms/Passes.h.inc"
+} // namespace omp
+} // namespace mlir
+
+using namespace mlir;
+
+/// Some host operations, like \c llvm.mlir.addressof and constants, must remain
+/// in the device module because they impact how device code is generated when
+/// attached to an \c omp.target operation.
+///
+/// This function identifies the operations that need this special handling.
+/// This includes cast-style operations to avoid losing information about the
+/// original source of an operand.
+static bool keepHostOpInDevice(Operation &op) {
+  return isPure(&op) &&
+         op.getDialect() ==
+             op.getContext()->getLoadedDialect<LLVM::LLVMDialect>();
+}
+
+/// Add an \c omp.map.info operation and all its members recursively to the
+/// output set to be later rewritten.
+///
+/// Dependencies across \c omp.map.info are maintained by ensuring dependencies
+/// are added to the output sets before operations based on them.
+static void collectRewrite(omp::MapInfoOp mapOp,
+                           llvm::SetVector<omp::MapInfoOp> &rewrites) {
+  for (Value member : mapOp.getMembers())
+    collectRewrite(cast<omp::MapInfoOp>(member.getDefiningOp()), rewrites);
+
+  rewrites.insert(mapOp);
+}
+
+/// Add the given value to a sorted set if it should be replaced by a
+/// placeholder when used as an operand that must remain for the device.
+///
+/// Values that are block arguments of function operations are skipped, since
+/// they will still be available after all rewrites are completed, and operands
+/// of operations that need to remain on the host are recursively collected.
+static void collectRewrite(Value value, llvm::SetVector<Value> &rewrites) {
+  if ((isa<BlockArgument>(value) &&
+       isa<FunctionOpInterface>(
+           cast<BlockArgument>(value).getOwner()->getParentOp())) ||
+      rewrites.contains(value))
+    return;
+
+  Operation *op = value.getDefiningOp();
+  if (op && keepHostOpInDevice(*op))
+    for (Value operand : op->getOperands())
+      collectRewrite(operand, rewrites);
+
+  rewrites.insert(value);
+}
+
+/// Provide the \c device_type of an \c omp.declare_target attribute, if
+/// defined.
+static std::optional<omp::DeclareTargetDeviceType>
+getDeclareTargetDevice(Operation &op) {
+  auto declareTargetOp = dyn_cast<omp::DeclareTargetInterface>(op);
+  if (declareTargetOp && declareTargetOp.isDeclareTarget())
+    return declareTargetOp.getDeclareTargetDeviceType();
+  return std::nullopt;
+}
+
+namespace {
+class HostOpFilteringPass
+    : public omp::impl::HostOpFilteringPassBase<HostOpFilteringPass> {
+public:
+  HostOpFilteringPass() = default;
+
+  void runOnOperation() override {
+    auto op = dyn_cast<omp::OffloadModuleInterface>(getOperation());
+    if (!op || !op.getIsTargetDevice())
+      return;
+
+    op->walk<WalkOrder::PreOrder>([&](LLVM::LLVMFuncOp funcOp) {
+      omp::DeclareTargetDeviceType declareType =
+          getDeclareTargetDevice(*funcOp.getOperation())
+              .value_or(omp::DeclareTargetDeviceType::host);
+
+      // Only process host function definitions.
+      if (funcOp.isExternal() ||
+          declareType != omp::DeclareTargetDeviceType::host)
+        return WalkResult::advance();
+
+      if (failed(rewriteHostFunction(funcOp))) {
+        funcOp.emitOpError() << "could not filter host-only operations";
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+
+    // Make non-declare target globals internal for the device. They cannot be
+    // deleted, because they are needed in order to properly lower map clauses.
+    // However, no uses will remain in the device module, so we make them
+    // internal to prevent link time redefinitions.
+    op->walk([&](LLVM::GlobalOp globalOp) {
+      if (!getDeclareTargetDevice(*globalOp.getOperation()).has_value())
+        globalOp.setLinkage(LLVM::Linkage::Internal);
+    });
+  }
+
+private:
+  /// Rewrite the given host device function containing \c omp.target
+  /// operations, to remove host-only operations that are not used by device
+  /// codegen.
+  ///
+  /// It is based on the expected form of an MLIR module lowered to where it can
+  /// be directly translated to LLVM IR and it performs the following mutations:
+  ///   - Removes all returned values from the function.
+  ///   - \c omp.target operations are moved to the end of the function. If they
+  ///     are nested inside of any other operations, they are hoisted out of
+  ///     them.
+  ///   - \c depend, \c device, \c dyn_groupprivate, \c if and \c in_reduction
+  ///     clauses are removed from these target functions. Values used to
+  ///     initialize other clauses are replaced by placeholders as follows:
+  ///     - Values defined by block arguments are replaced by placeholders only
+  ///       if they are not attached to the parent function. In that case, they
+  ///       are passed unmodified.
+  ///     - Pure operations of the LLVM dialect are maintained, and any value
+  ///       operands they might have are also replaced by placeholders following
+  ///       the same rules.
+  ///     - Other values are replaced by new function arguments.
+  ///   - \c omp.map.info operations associated to these target regions are
+  ///     preserved. These are moved above all \c omp.target and sorted to
+  ///     satisfy dependencies among them.
+  ///   - \c bounds arguments are removed from \c omp.map.info operations.
+  ///   - \c var_ptr and \c var_ptr_ptr arguments of \c omp.map.info are
+  ///     replaced by placeholders as described above.
+  ///   - Every other operation not located inside of an \c omp.target is
+  ///     removed.
+  LogicalResult rewriteHostFunction(LLVM::LLVMFuncOp funcOp) {
+    Region &region = funcOp.getFunctionBody();
+    LLVM::LLVMFunctionType functionType = funcOp.getFunctionType();
+
+    // Collect target operations inside of the function.
+    llvm::SmallVector<omp::TargetOp> targetOps;
+    region.walk<WalkOrder::PreOrder>([&](Operation *op) {
+      // Skip the inside of omp.target regions, since these contain device code.
+      if (auto targetOp = dyn_cast<omp::TargetOp>(op)) {
+        targetOps.push_back(targetOp);
+        return WalkResult::skip();
+      }
+
+      // Replace omp.target_data entry block argument uses with the value used
+      // to initialize the associated omp.map.info operation. This way,
+      // references are still valid once the omp.target operation has been
+      // extracted out of the omp.target_data region.
+      if (auto targetDataOp = dyn_cast<omp::TargetDataOp>(op)) {
+        llvm::SmallVector<std::pair<Value, BlockArgument>> argPairs;
+        cast<omp::BlockArgOpenMPOpInterface>(*targetDataOp)
+            .getBlockArgsPairs(argPairs);
+        for (auto [operand, blockArg] : argPairs) {
+          auto mapInfo = cast<omp::MapInfoOp>(operand.getDefiningOp());
+          blockArg.replaceAllUsesWith(mapInfo.getVarPtr());
+        }
+      }
+      return WalkResult::advance();
+    });
+
+    // Make a temporary clone of the parent function with an empty region,
+    // and update all references to entry block arguments to those of the new
+    // region. Users of these arguments will later either be moved to the new
+    // region or deleted when the original region is replaced by the new.
+    OpBuilder builder(&getContext());
+    builder.setInsertionPointAfter(funcOp);
+    Operation *newFuncOp = builder.cloneWithoutRegions(funcOp);
+    Block &block = newFuncOp->getRegion(0).emplaceBlock();
+
+    llvm::SmallVector<Location> locs;
+    locs.reserve(region.getNumArguments());
+    llvm::transform(region.getArguments(), std::back_inserter(locs),
+                    [](const BlockArgument &arg) { return arg.getLoc(); });
+    block.addArguments(region.getArgumentTypes(), locs);
+
+    for (auto [oldArg, newArg] :
+         llvm::zip_equal(region.getArguments(), block.getArguments()))
+      oldArg.replaceAllUsesWith(newArg);
+
+    // Collect omp.map.info ops while satisfying interdependencies and remove
+    // operands that aren't used by target device codegen.
+    //
+    // This logic must be updated whenever operands to omp.target change.
+    llvm::SetVector<Value> rewriteValues;
+    llvm::SetVector<omp::MapInfoOp> mapInfos;
+    for (omp::TargetOp targetOp : targetOps) {
+      assert(targetOp.getHostEvalVars().empty() &&
+             "unexpected host_eval in target device module");
+
+      // Variables unused by the device.
+      targetOp.getDependVarsMutable().clear();
+      targetOp.setDependKindsAttr(nullptr);
+      targetOp.getDependIteratedMutable().clear();
+      targetOp.setDependIteratedKindsAttr(nullptr);
+      targetOp.getDeviceMutable().clear();
+      targetOp.getDynGroupprivateSizeMutable().clear();
+      targetOp.getIfExprMutable().clear();
+      targetOp.getInReductionVarsMutable().clear();
+      targetOp.setInReductionByrefAttr(nullptr);
+      targetOp.setInReductionSymsAttr(nullptr);
+
+      // TODO: Clear some of these operands rather than rewriting them,
+      // depending on whether they are needed by device codegen once support for
+      // them is fully implemented.
+      for (Value allocVar : targetOp.getAllocateVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value allocVar : targetOp.getAllocatorVars())
+        collectRewrite(allocVar, rewriteValues);
+      for (Value isDevPtr : targetOp.getIsDevicePtrVars())
+        collectRewrite(isDevPtr, rewriteValues);
+      for (Value mapVar : targetOp.getHasDeviceAddrVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value mapVar : targetOp.getMapVars())
+        collectRewrite(cast<omp::MapInfoOp>(mapVar.getDefiningOp()), mapInfos);
+      for (Value privateVar : targetOp.getPrivateVars())
+        collectRewrite(privateVar, rewriteValues);
+      for (Value threadLimit : targetOp.getThreadLimitVars())
+        collectRewrite(threadLimit, rewriteValues);
+    }
+
+    // Move omp.map.info ops to the new block and collect dependencies.
+    for (omp::MapInfoOp mapOp : mapInfos) {
+      collectRewrite(mapOp.getVarPtr(), rewriteValues);
+
+      if (Value varPtrPtr = mapOp.getVarPtrPtr())
+        collectRewrite(varPtrPtr, rewriteValues);
+
+      // Bounds are not used during target device codegen.
+      mapOp.getBoundsMutable().clear();
+      mapOp->moveBefore(&block, block.end());
+    }
+
+    builder.setInsertionPointToStart(&block);
+
+    // We don't actually need the proper initialization for all operands, but
+    // rather just to maintain the basic form of omp.target operations. We
+    // create new function arguments as placeholders for rewritten values.
+    llvm::SmallVector<Type> newFnArgTypes(functionType.getParams());
+    for (Value value : rewriteValues) {
+      Value rewriteValue;
+      Operation *definingOp = value.getDefiningOp();
+      if (definingOp && keepHostOpInDevice(*definingOp)) {
+        rewriteValue = builder.clone(*value.getDefiningOp())->getResult(0);
+      } else {
+        rewriteValue = block.addArgument(value.getType(), value.getLoc());
+        newFnArgTypes.push_back(rewriteValue.getType());
+      }
+      value.replaceAllUsesWith(rewriteValue);
+    }
+
+    // Move target operations to the end of the new block.
+    for (omp::TargetOp targetOp : targetOps)
+      targetOp->moveBefore(&block, block.end());
+
+    // Add terminator to the new block.
+    builder.setInsertionPointToEnd(&block);
+    LLVM::ReturnOp::create(builder, funcOp.getLoc(), ValueRange());
+
+    // Replace old region with the new one, now only containing the required
+    // operations, and remove the temporary operation clone.
+    region.takeBody(newFuncOp->getRegion(0));
+    newFuncOp->erase();
+
+    // Update function type after modifying the terminator and argument list.
+    funcOp.setType(LLVM::LLVMFunctionType::get(
+        LLVM::LLVMVoidType::get(&getContext()), newFnArgTypes));
+
+    return success();
+  }
+};
+} // namespace

diff  --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
new file mode 100644
index 0000000000000..015e901126a76
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -0,0 +1,500 @@
+// RUN: mlir-opt --omp-host-op-filter %s | FileCheck %s
+module attributes {omp.is_target_device = true} {
+  // CHECK-LABEL: llvm.func @external() attributes {sym_visibility = "private"}
+  // CHECK-NOT: llvm.return
+  llvm.func @external() attributes {sym_visibility = "private"}
+
+  // CHECK-LABEL: llvm.func @basic_checks
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @basic_checks(%arg0: !llvm.ptr) -> !llvm.struct<(i32, f32)> {
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = llvm.mlir.constant(10 : i32) : i32
+    %3 = llvm.mlir.constant(2.500000e+00 : f32) : f32
+
+    llvm.call @foo() : () -> ()
+
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %4 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %5 = llvm.mlir.constant(1 : i64) : i64
+
+    // CHECK-NEXT: %[[HDA:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]]{{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]]{{.*}})
+    // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]]{{.*}})
+    %6 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %7 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %8 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %9 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) has_device_addr(%[[HDA]] -> {{.*}} : {{.*}}) map_entries(%[[MAP0]] -> {{.*}}, %[[MAP1]] -> {{.*}}, %[[MAP2]] -> {{.*}} : {{.*}})
+    omp.target kernel_type(generic) has_device_addr(%8 -> %arg1 : !llvm.ptr) map_entries(%6 -> %arg2, %7 -> %arg3, %9 -> %arg4 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NEXT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: llvm.call
+    omp.parallel {
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.map.info
+    %10 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %11 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %12 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %13 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%10, %11, %12, %13 : !llvm.ptr, !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    %14 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_enter_data map_entries(%14 : !llvm.ptr)
+    %15 = omp.map.info var_ptr(%4 : !llvm.ptr, i32) map_clauses(from) capture(ByRef) -> !llvm.ptr
+    omp.target_exit_data map_entries(%15 : !llvm.ptr)
+    %16 = omp.map.info var_ptr(%1 : !llvm.ptr, !llvm.ptr) map_clauses(to) capture(ByRef) -> !llvm.ptr
+    omp.target_update map_entries(%16 : !llvm.ptr)
+
+    // CHECK-NOT: llvm.call
+    llvm.call @foo() : () -> ()
+
+    // CHECK: {{llvm\.return$}}
+    %17 = llvm.mlir.poison : !llvm.struct<(i32, f32)>
+    %18 = llvm.insertvalue %2, %17[0] : !llvm.struct<(i32, f32)>
+    %19 = llvm.insertvalue %3, %18[1] : !llvm.struct<(i32, f32)>
+    llvm.return %19 : !llvm.struct<(i32, f32)>
+  }
+
+  // CHECK-LABEL: llvm.func @member_map_complex_init
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @member_map_complex_init(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[PLACEHOLDER]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<{{.*}}>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(1 : i64) : i64
+    %3 = llvm.alloca %2 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %4 = llvm.mlir.constant(1 : i64) : i64
+    %5 = llvm.alloca %4 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)> : (i64) -> !llvm.ptr
+    %6 = llvm.mlir.constant(1 : i32) : i32
+    %7 = llvm.mlir.constant(0 : i8) : i8
+    %8 = llvm.mlir.constant(1 : i64) : i64
+    %9 = llvm.mlir.constant(1 : i64) : i64
+    %10 = llvm.mlir.zero : !llvm.ptr
+    llvm.call @LangRTPlaceholderFunc(%5, %arg0, %10, %7, %6) : (!llvm.ptr, !llvm.ptr, !llvm.ptr, i8, i32) -> ()
+    %11 = llvm.mlir.constant(384 : i32) : i32
+    "llvm.intr.memcpy"(%1, %5, %11) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %12 = llvm.mlir.constant(24 : i32) : i32
+    %13 = llvm.getelementptr %1[0, 3] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+    %14 = llvm.load %13 : !llvm.ptr -> i8
+    %15 = llvm.sext %14 : i8 to i32
+    %16 = llvm.mlir.constant(24 : i32) : i32
+    %17 = llvm.mul %16, %15 : i32
+    %18 = llvm.add %12, %17 : i32
+    "llvm.intr.memcpy"(%3, %1, %18) <{isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %19 = llvm.getelementptr %3[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[GEP]] : {{.*}})
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr)
+    %20 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%19 : !llvm.ptr, f32) -> !llvm.ptr
+    %21 = omp.map.info var_ptr(%3 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<15 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%20 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}}, %[[MAP0]] -> %{{.*}} : {{.*}})
+    omp.target kernel_type(generic) map_entries(%21 -> %arg1, %20 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @target_data
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr, %[[ARG2:.*]]: !llvm.ptr)
+  llvm.func @target_data(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: !llvm.ptr) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%arg2 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(return_param) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NOT: omp.target_data
+    omp.target_data map_entries(%0 : !llvm.ptr) use_device_addr(%1 -> %arg3 : !llvm.ptr) use_device_ptr(%2 -> %arg4 : !llvm.ptr) {
+      // CHECK:      %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef)
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[ARG2]] : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef)
+      %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %4 = omp.map.info var_ptr(%arg3 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%arg4 : !llvm.ptr, !llvm.struct<"MyStruct", (i64)>) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}}, %[[MAP2]] -> %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%3 -> %arg5, %4 -> %arg6, %5 -> %arg7 : !llvm.ptr, !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @no_target
+  // CHECK-SAME: (%{{.*}}: !llvm.ptr)
+  llvm.func @no_target(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: llvm.return
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%0 : !llvm.ptr) {
+      %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%1 : !llvm.ptr) {
+        llvm.call @foo() : () -> ()
+        omp.terminator
+      }
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @map_info_members
+  // CHECK-SAME: (%[[ARG:.*]]: !llvm.ptr)
+  llvm.func @map_info_members(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[VAR_PTR_PTR:.*]] = llvm.getelementptr %[[ARG]][0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.alloca %0 x !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)> {alignment = 8 : i64} : (i32) -> !llvm.ptr
+    %2 = llvm.mlir.constant(0 : index) : i64
+    %3 = llvm.mlir.constant(1 : index) : i64
+    %4 = llvm.mlir.constant(9 : index) : i64
+    %5 = llvm.mlir.constant(48 : i32) : i32
+    "llvm.intr.memcpy"(%1, %arg0, %5) <{arg_attrs = [{llvm.align = 8 : i64}, {llvm.align = 8 : i64}, {}], isVolatile = false}> : (!llvm.ptr, !llvm.ptr, i32) -> ()
+    %6 = llvm.getelementptr %1[0, 7, %2, 0] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %7 = llvm.load %6 : !llvm.ptr -> i64
+    %8 = llvm.getelementptr %1[0, 7, %2, 1] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %9 = llvm.load %8 : !llvm.ptr -> i64
+    %10 = llvm.getelementptr %1[0, 7, %2, 2] : (!llvm.ptr, i64) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+    %11 = llvm.load %10 : !llvm.ptr -> i64
+    %12 = llvm.sub %3, %7 : i64
+    %13 = llvm.sub %4, %7 : i64
+    %14 = omp.map.bounds lower_bound(%12 : i64) upper_bound(%13 : i64) extent(%9 : i64) stride(%11 : i64) start_idx(%7 : i64) {stride_in_bytes = true}
+    %15 = llvm.getelementptr %arg0[0, 0] : (!llvm.ptr) -> !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>
+
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%[[VAR_PTR_PTR]] : !llvm.ptr, f32) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG]] {{.*}} map_clauses(to) capture(ByRef) members(%[[MAP0]] : [0] : !llvm.ptr) -> !llvm.ptr
+    %16 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(tofrom) capture(ByRef) var_ptr_ptr(%15 : !llvm.ptr, f32) bounds(%14) -> !llvm.ptr
+    %17 = omp.map.info var_ptr(%arg0 : !llvm.ptr, !llvm.struct<(ptr, i64, i32, i8, i8, i8, i8, array<1 x array<3 x i64>>)>) map_clauses(to) capture(ByRef) members(%16 : [0] : !llvm.ptr) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> {{.*}}, %[[MAP0]] -> {{.*}} : !llvm.ptr, !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%17 -> %arg1, %16 -> %arg2 : !llvm.ptr, !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @control_flow
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: !llvm.ptr)
+  llvm.func @control_flow(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %0 = llvm.load %arg1 : !llvm.ptr -> i32
+    %1 = llvm.mlir.constant(0 : i32) : i32
+    %2 = llvm.icmp "ne" %0, %1 : i32
+    llvm.cond_br %2, ^bb1, ^bb2
+  ^bb1:  // pred: ^bb0
+    llvm.call @foo() : () -> ()
+    %3 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%3 -> %arg2 : !llvm.ptr) {
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+    llvm.br ^bb2
+  ^bb2:  // 2 preds: ^bb0, ^bb1
+    // CHECK-NOT: llvm.call
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    llvm.call @foo() : () -> ()
+
+    %4 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%4 : !llvm.ptr) {
+      llvm.call @foo() : () -> ()
+      %5 = llvm.load %arg1 : !llvm.ptr -> i32
+      %6 = llvm.mlir.constant(0 : i32) : i32
+      %7 = llvm.icmp "ne" %5, %6 : i32
+      llvm.cond_br %7, ^bb1, ^bb2
+    ^bb1:  // pred: ^bb0
+      llvm.call @foo() : () -> ()
+      %8 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%8 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+
+      // CHECK-NOT: llvm.call
+      // CHECK-NOT: llvm.br
+      llvm.call @foo() : () -> ()
+      llvm.br ^bb2
+    ^bb2:  // 2 preds: ^bb0, ^bb1
+      llvm.call @foo() : () -> ()
+      omp.terminator
+    }
+    llvm.call @foo() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @block_args
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[PLACEHOLDER0:.*]]: !llvm.ptr, %[[PLACEHOLDER1:.*]]: !llvm.ptr)
+  llvm.func @block_args(%arg0: !llvm.ptr) {
+    // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[PLACEHOLDER1]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}} : !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%0 -> %arg2 : !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.parallel
+    // CHECK-NOT: omp.map.info
+    // CHECK-NOT: omp.target_data
+    omp.parallel private(@privatizer %arg0 -> %arg1 : !llvm.ptr) {
+      %0 = omp.map.info var_ptr(%arg1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      omp.target_data map_entries(%0 : !llvm.ptr) {
+        omp.parallel private(@privatizer %arg1 -> %arg2 : !llvm.ptr) {
+          %1 = omp.map.info var_ptr(%arg2 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+          // CHECK: omp.target kernel_type(generic) map_entries(%[[MAP1]] -> %{{.*}} : !llvm.ptr)
+          omp.target kernel_type(generic) map_entries(%1 -> %arg3 : !llvm.ptr) {
+            omp.terminator
+          }
+          omp.terminator
+        }
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reuse_tests
+  // CHECK-SAME: (%[[THREAD_LIMIT:.*]]: i32)
+  llvm.func @reuse_tests() {
+    // CHECK-NEXT: %[[CONST_THREAD_LIMIT:.*]] = llvm.mlir.constant(1 : i32) : i32
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %0 = llvm.mlir.constant(1 : i32) : i32
+    %1 = llvm.mlir.addressof @global_scalar : !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data map_entries(%2 : !llvm.ptr) {
+      // CHECK-NEXT: %[[MAP0:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP1:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      // CHECK-NEXT: %[[MAP2:.*]] = omp.map.info var_ptr(%[[GLOBAL]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %5 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+      %6 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+      // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP0]] -> %{{.*}}, %[[MAP1]] -> %{{.*}} : !llvm.ptr, !llvm.ptr)
+      omp.target kernel_type(generic) map_entries(%5 -> %arg0, %6 -> %arg1 : !llvm.ptr, !llvm.ptr) {
+        omp.terminator
+      }
+      omp.terminator
+    }
+
+    // CHECK-NOT: llvm.load
+    // CHECK-NOT: omp.map.info
+    %3 = llvm.load %1 : !llvm.ptr -> i32
+    %4 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[THREAD_LIMIT]] : i32) map_entries(%[[MAP2]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) thread_limit(%3 : i32) map_entries(%4 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: omp.target kernel_type(generic) thread_limit(%[[CONST_THREAD_LIMIT]] : i32)
+    omp.target kernel_type(generic) thread_limit(%0 : i32) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @all_non_map_clauses
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: i1)
+  llvm.func @all_non_map_clauses(%arg0: !llvm.ptr, %arg1: i32, %arg2: i1) {
+    %0 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target_data device(%arg1 : i32) if(%arg2) map_entries(%0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: omp.target_enter_data
+    // CHECK-NOT: omp.target_exit_data
+    // CHECK-NOT: omp.target_update
+    omp.target_enter_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_exit_data depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+    omp.target_update depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2)
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @private_with_map_idx
+  // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr, %[[ARG1:.*]]: i64)
+  llvm.func @private_with_map_idx(%arg0: !llvm.ptr, %arg1: i64) {
+    // CHECK-NEXT: %[[VAL0:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL1:.*]] = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL2:.*]] = llvm.insertvalue %[[ARG0]], %[[VAL1]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL3:.*]] = llvm.insertvalue %[[ARG1]], %[[VAL2]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL4:.*]] = llvm.extractvalue %[[VAL3]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL5:.*]] = llvm.insertvalue %[[VAL4]], %[[VAL0]][0] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL6:.*]] = llvm.extractvalue %[[VAL3]][1] : !llvm.struct<(ptr, i64)>
+    // CHECK-NEXT: %[[VAL7:.*]] = llvm.insertvalue %[[VAL6]], %[[VAL5]][1] : !llvm.struct<(ptr, i64)>
+    %0 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, i64)>
+    %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, i64)>
+    %3 = llvm.extractvalue %2[0] : !llvm.struct<(ptr, i64)>
+    %4 = llvm.extractvalue %2[1] : !llvm.struct<(ptr, i64)>
+    %5 = llvm.mlir.undef : !llvm.struct<(ptr, i64)>
+    %6 = llvm.insertvalue %3, %5[0] : !llvm.struct<(ptr, i64)>
+    %7 = llvm.insertvalue %4, %6[1] : !llvm.struct<(ptr, i64)>
+
+    // CHECK: omp.target kernel_type(generic) private(@struct_firstprivatizer %[[VAL7]] -> %{{.*}} [map_idx=0] : !llvm.struct<(ptr, i64)>)
+    omp.target kernel_type(generic) private(@struct_firstprivatizer %7 -> %arg2 [map_idx=0] : !llvm.struct<(ptr, i64)>) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  omp.private {type = firstprivate} @struct_firstprivatizer : !llvm.struct<(ptr, i64)> copy {
+  ^bb0(%arg0: !llvm.struct<(ptr, i64)>, %arg1: !llvm.struct<(ptr, i64)>):
+    omp.yield(%arg0 : !llvm.struct<(ptr, i64)>)
+  }
+
+  // CHECK-LABEL: llvm.func @map_from_global_gep
+  // CHECK-SAME: (%[[ARG0:.*]]: i64)
+  llvm.func @map_from_global_gep(%arg0: i64) {
+    // CHECK-NEXT: %[[GLOBAL:.*]] = llvm.mlir.addressof @global_array : !llvm.ptr
+    // CHECK-NEXT: %[[GEP:.*]] = llvm.getelementptr %[[GLOBAL]][%[[ARG0]]] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+    %0 = llvm.mlir.addressof @global_array : !llvm.ptr
+    %1 = llvm.getelementptr %0[%arg0] : (!llvm.ptr, i64) -> !llvm.ptr, i8
+
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[GEP]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target kernel_type(generic) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_a
+  // CHECK-SAME: (%[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_a() {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg0 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_b
+    llvm.call @reciprocal_b(%0) : (i64) -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  // CHECK-LABEL: llvm.func @reciprocal_b
+  // CHECK-SAME: (%[[ARG0:.*]]: i64, %[[PLACEHOLDER:.*]]: !llvm.ptr)
+  llvm.func @reciprocal_b(%arg0 : i64) {
+    // CHECK-NEXT: omp.map.info
+    %0 = llvm.mlir.constant(1 : i64) : i64
+    %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+    %2 = omp.map.info var_ptr(%1 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+
+    // CHECK-NEXT: omp.target
+    omp.target kernel_type(generic) map_entries(%2 -> %arg1 : !llvm.ptr) {
+      omp.terminator
+    }
+
+    // CHECK-NOT: @reciprocal_a
+    llvm.call @reciprocal_a() : () -> ()
+
+    // CHECK: llvm.return
+    llvm.return
+  }
+
+  llvm.func @foo() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>, sym_visibility = "private"}
+  omp.private {type = firstprivate} @privatizer : i32 copy {
+  ^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+    %0 = llvm.load %arg0 : !llvm.ptr -> i32
+    llvm.store %0, %arg1 : i32, !llvm.ptr
+    omp.yield(%arg1 : !llvm.ptr)
+  }
+  omp.declare_reduction @reduction : i32 init {
+
+  ^bb0(%arg0: i32):
+    %0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%0 : i32)
+  } combiner {
+
+  ^bb0(%arg0: i32, %arg1: i32):
+    %0 = llvm.add %arg0, %arg1 : i32
+    omp.yield(%0 : i32)
+  }
+
+  // CHECK: llvm.mlir.global internal constant @global_scalar
+  // CHECK: llvm.mlir.global internal @global_array
+  llvm.mlir.global external constant @global_scalar() {addr_space = 0 : i32} : i32
+  llvm.mlir.global external @global_array() {addr_space = 0 : i32} : !llvm.array<8 x i8>
+
+  // CHECK: llvm.mlir.global external @declare_target_enter_any
+  // CHECK: llvm.mlir.global external @declare_target_enter_host
+  // CHECK: llvm.mlir.global external @declare_target_enter_nohost
+  llvm.mlir.global external @declare_target_enter_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>} : i32
+  llvm.mlir.global external @declare_target_enter_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>} : i32
+
+  // CHECK: llvm.mlir.global external @declare_target_link_any
+  // CHECK: llvm.mlir.global external @declare_target_link_host
+  // CHECK: llvm.mlir.global external @declare_target_link_nohost
+  llvm.mlir.global external @declare_target_link_any() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_host() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (link), automap = false>} : i32
+  llvm.mlir.global external @declare_target_link_nohost() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (link), automap = false>} : i32
+
+  llvm.func @LangRTPlaceholderFunc(!llvm.ptr {llvm.nocapture}, !llvm.ptr {llvm.nocapture}, !llvm.ptr, i8 {llvm.signext}, i32) attributes {sym_visibility = "private"}
+}


        

From llvmlistbot at llvm.org  Thu Jul  9 03:58:33 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Thu, 09 Jul 2026 03:58:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Fix
 declare_target globals visibility (PR #208188)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208188@github.com>
Message-ID: <6a4f7ed9.da8bafd3.14cd50.ce49@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `flang,mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12257

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)


-- Configuring incomplete, errors occurred!
[8674/8698] Creating library symlink lib/libHLFIRTransforms.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtime
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8676/8698] Linking CXX shared library lib/libflangPasses.so.23.0git
[8677/8698] Linking CXX shared library lib/libFortranLower.so.23.0git
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:283 (include)
-- Configuring incomplete, errors occurred!
[8674/8698] Creating library symlink lib/libHLFIRTransforms.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtime
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8676/8698] Linking CXX shared library lib/libflangPasses.so.23.0git
[8677/8698] Linking CXX shared library lib/libFortranLower.so.23.0git
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=77.048515

```

</details>

https://github.com/llvm/llvm-project/pull/208188

From llvmlistbot at llvm.org  Thu Jul  9 04:04:23 2026
From: llvmlistbot at llvm.org (Zmicier Prybysh)
Date: Thu, 09 Jul 2026 04:04:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Support non-unit dilations in
 im2col decomposition (PR #208424)
Message-ID: <llvm.org/llvm/llvm-project/pull/208424@github.com>


https://github.com/dimp-pl created https://github.com/llvm/llvm-project/pull/208424

When learning linang dialect, i noticed this TODO and decided to tackle it.

The im2col patterns for conv_2d_nhwc_hwcf, conv_2d_nchw_fchw and conv_2d_nhwc_fhwc avoided non-unit dilations, but the restriction doesn't seem to be fundamental: dilation only affects the gather step, which can fold it into its indexing map. Generalize the convolved index expression from `oh * stride + fh` to `oh * stride + fh * dilation` and drop the match failures.

Verified by comparing the results of a dilated convolution lowered directly to loops against the im2col decomposition with mlir-runner.

Assisted-By: Claude Code (for generating tests).

>From 508b60348e5e2b46bfc08be4e779ea8af92ed50d Mon Sep 17 00:00:00 2001
From: Zmicier Prybysh <zprybysh at baylibre.com>
Date: Thu, 9 Jul 2026 12:55:35 +0200
Subject: [PATCH] [mlir][linalg] Support non-unit dilations in im2col
 decomposition

The im2col patterns for conv_2d_nhwc_hwcf, conv_2d_nchw_fchw and
conv_2d_nhwc_fhwc bailed out on non-unit dilations, but the restriction
is not fundamental: dilation only affects the gather step, which can
fold it into its indexing map. Generalize the convolved index
expression from `oh * stride + fh` to `oh * stride + fh * dilation` and
drop the match failures.

Verified by comparing the results of a dilated convolution lowered
directly to loops against the im2col decomposition with mlir-runner.
---
 .../Transforms/ConvertConv2DToImg2Col.cpp     |  41 +++----
 .../Linalg/convert-conv2d-to-img2col.mlir     | 112 ++++++++++++++++++
 2 files changed, 126 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp b/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
index 4cd6fdf4f7341..e2dd0a6d79a73 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
@@ -54,16 +54,16 @@ static Value createMul(Location loc, Value x, Value y, Type accType,
 }
 
 // Generate the affine expression to compute the convolved index
-// for the input as `oIndex * stride + fIndex`,
+// for the input as `oIndex * stride + fIndex * dilation`,
 // where oIndex: output iterator; fIndex: filter iterator.
 static AffineExpr getConvolvedExpr(OpBuilder &b, int64_t stride,
-                                   bool useSymbols = true) {
+                                   int64_t dilation, bool useSymbols = true) {
   AffineExpr oExpr, fExpr;
   if (useSymbols)
     bindSymbols(b.getContext(), oExpr, fExpr);
   else
     bindDims(b.getContext(), oExpr, fExpr);
-  return AffineExpr(stride * oExpr + fExpr);
+  return AffineExpr(stride * oExpr + dilation * fExpr);
 }
 
 // Stores the affine expressions to map the iteration space of the im2col matrix
@@ -91,12 +91,14 @@ struct Im2ColToInputDimsExprs {
 ///
 /// @param exprs      Affine expressions for output and filter indices.
 /// @param strides    [height, width] stride values for the convolution.
+/// @param dilations  [height, width] dilation values for the convolution.
 /// @param rewriter   Pattern rewriter.
 /// @return           Affine expressions mapping im2col matrix indices to input
 /// offsets.
 static Im2ColToInputDimsExprs
 getIm2ColInputExpressions(Im2ColToOperandsExprs exprs,
-                          ArrayRef<int64_t> strides, RewriterBase &rewriter) {
+                          ArrayRef<int64_t> strides,
+                          ArrayRef<int64_t> dilations, RewriterBase &rewriter) {
   // maps the iteration space of the im2col matrix to (output_y, filter_y)
   auto hIndicesMap = AffineMap::inferFromExprList(
       {ArrayRef{exprs.ohIndex, exprs.fhIndex}}, rewriter.getContext())[0];
@@ -110,10 +112,10 @@ getIm2ColInputExpressions(Im2ColToOperandsExprs exprs,
   // then we compose them with the maps that map the im2col matrix elements to
   // the (out_element, filter_element) pairs.
   auto bIndexExpr = rewriter.getAffineDimExpr(0U);
-  auto hIndexExpr = getConvolvedExpr(rewriter, strides[0],
+  auto hIndexExpr = getConvolvedExpr(rewriter, strides[0], dilations[0],
                                      /*useSymbols*/ false);
   hIndexExpr = hIndexExpr.compose(hIndicesMap);
-  auto wIndexExpr = getConvolvedExpr(rewriter, strides[1],
+  auto wIndexExpr = getConvolvedExpr(rewriter, strides[1], dilations[1],
                                      /*useSymbols*/ false);
   wIndexExpr = wIndexExpr.compose(wIndicesMap);
   auto cIndexExpr = exprs.icIndex;
@@ -138,11 +140,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcHwcfOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   MLIRContext *context = rewriter.getContext();
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
@@ -203,10 +200,10 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcHwcfOp convOp) {
   i2cToOperExprs.ohIndex = mIndicesExprs[0];
   i2cToOperExprs.owIndex = mIndicesExprs[1];
 
-  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + fh, sw*ow + fw, ic]
+  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + dh*fh, sw*ow + dw*fw, ic]
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.hIndex,
                                              inExprs.wIndex, inExprs.cIndex}},
@@ -431,11 +428,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNchwFchwOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
   Value output = convOp.getOutputs()[0];
@@ -497,12 +489,12 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNchwFchwOp convOp) {
   i2cToOperExprs.owIndex = mIndicesExprs[1];
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.cIndex,
                                              inExprs.hIndex, inExprs.wIndex}},
                                    rewriter.getContext())[0];
-  // im2col[n, ic*fh*fw, oh*ow] = input[n, ic, sh*oh + fh, sw*ow + fw]
+  // im2col[n, ic*fh*fw, oh*ow] = input[n, ic, sh*oh + dh*fh, sw*ow + dw*fw]
   SmallVector<AffineMap> img2colIndexingMaps = {
       inMap, AffineMap::getMultiDimIdentityMap(nloops, context)};
 
@@ -565,11 +557,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcFhwcOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   MLIRContext *context = rewriter.getContext();
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
@@ -632,10 +619,10 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcFhwcOp convOp) {
   i2cToOperExprs.ohIndex = mIndicesExprs[0];
   i2cToOperExprs.owIndex = mIndicesExprs[1];
 
-  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + fh, sw*ow + fw, ic]
+  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + dh*fh, sw*ow + dw*fw, ic]
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.hIndex,
                                              inExprs.wIndex, inExprs.cIndex}},
diff --git a/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir b/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
index 152a392afe247..387f9762d518d 100644
--- a/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
+++ b/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
@@ -556,3 +556,115 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+// Dilated NHWC-HWCF: the im2col gather reads input at
+// oh*stride + fh*dilation, i.e. d1 floordiv 12 + (d2 floordiv 12) * 2.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 12 + (d2 floordiv 12) * 2, d1 mod 12 + ((d2 mod 12) floordiv 4) * 2, d2 mod 4)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nhwc_hwcf_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<1x16x16x4xf32>, %[[FILTER:.+]]: tensor<3x3x4x16xf32>, %[[INIT:.+]]: tensor<1x12x12x16xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0, 1, 2], [3]] : tensor<3x3x4x16xf32> into tensor<36x16xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1, 2], [3]] : tensor<1x12x12x16xf32> into tensor<1x144x16xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<1x144x36xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<1x16x16x4xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<1x144x36xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[IMG2COL]], %[[CS_FILTER]] : tensor<1x144x36xf32>, tensor<36x16xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<1x144x16xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1, 2], [3]] output_shape [1, 12, 12, 16] : tensor<1x144x16xf32> into tensor<1x12x12x16xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nhwc_hwcf_dilated(%arg0: tensor<1x16x16x4xf32>, %arg1: tensor<3x3x4x16xf32>, %arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32> {
+    %0 = linalg.conv_2d_nhwc_hwcf
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<1x16x16x4xf32>, tensor<3x3x4x16xf32>)
+      outs(%arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32>
+    return %0 : tensor<1x12x12x16xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nhwc_hwcf"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Dilated NCHW-FCHW: h index is d2 floordiv 12 + ((d1 mod 9) floordiv 3) * 2.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 9, d2 floordiv 12 + ((d1 mod 9) floordiv 3) * 2, d2 mod 12 + (d1 mod 3) * 2)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nchw_fchw_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<8x4x16x16xf32>, %[[FILTER:.+]]: tensor<16x4x3x3xf32>, %[[INIT:.+]]: tensor<8x16x12x12xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0], [1, 2, 3]] : tensor<16x4x3x3xf32> into tensor<16x36xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1], [2, 3]] : tensor<8x16x12x12xf32> into tensor<8x16x144xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<8x36x144xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<8x4x16x16xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<8x36x144xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[CS_FILTER]], %[[IMG2COL]] : tensor<16x36xf32>, tensor<8x36x144xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<8x16x144xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1], [2, 3]] output_shape [8, 16, 12, 12] : tensor<8x16x144xf32> into tensor<8x16x12x12xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nchw_fchw_dilated(%arg0: tensor<8x4x16x16xf32>, %arg1: tensor<16x4x3x3xf32>, %arg2: tensor<8x16x12x12xf32>) -> tensor<8x16x12x12xf32> {
+    %0 = linalg.conv_2d_nchw_fchw
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<8x4x16x16xf32>, tensor<16x4x3x3xf32>)
+      outs(%arg2: tensor<8x16x12x12xf32>) -> tensor<8x16x12x12xf32>
+    return %0 : tensor<8x16x12x12xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nchw_fchw"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Dilated NHWC-FHWC: same dilated input gather map as NHWC-HWCF.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 12 + (d2 floordiv 12) * 2, d1 mod 12 + ((d2 mod 12) floordiv 4) * 2, d2 mod 4)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nhwc_fhwc_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<1x16x16x4xf32>, %[[FILTER:.+]]: tensor<16x3x3x4xf32>, %[[INIT:.+]]: tensor<1x12x12x16xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0], [1, 2, 3]] : tensor<16x3x3x4xf32> into tensor<16x36xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1, 2], [3]] : tensor<1x12x12x16xf32> into tensor<1x144x16xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<1x144x36xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<1x16x16x4xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<1x144x36xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[IMG2COL]], %[[CS_FILTER]] : tensor<1x144x36xf32>, tensor<16x36xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<1x144x16xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1, 2], [3]] output_shape [1, 12, 12, 16] : tensor<1x144x16xf32> into tensor<1x12x12x16xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nhwc_fhwc_dilated(%arg0: tensor<1x16x16x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32> {
+    %0 = linalg.conv_2d_nhwc_fhwc
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<1x16x16x4xf32>, tensor<16x3x3x4xf32>)
+      outs(%arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32>
+    return %0 : tensor<1x12x12x16xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nhwc_fhwc"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}


From llvmlistbot at llvm.org  Thu Jul  9 04:05:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 04:05:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Support non-unit dilations in
 im2col decomposition (PR #208424)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208424@github.com>
Message-ID: <6a4f8060.8e75e0cb.5e108.1f34@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-linalg

Author: Zmicier Prybysh (dimp-pl)

<details>
<summary>Changes</summary>

When learning linang dialect, i noticed this TODO and decided to tackle it.

The im2col patterns for conv_2d_nhwc_hwcf, conv_2d_nchw_fchw and conv_2d_nhwc_fhwc avoided non-unit dilations, but the restriction doesn't seem to be fundamental: dilation only affects the gather step, which can fold it into its indexing map. Generalize the convolved index expression from `oh * stride + fh` to `oh * stride + fh * dilation` and drop the match failures.

Verified by comparing the results of a dilated convolution lowered directly to loops against the im2col decomposition with mlir-runner.

Assisted-By: Claude Code (for generating tests).

---
Full diff: https://github.com/llvm/llvm-project/pull/208424.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp (+14-27) 
- (modified) mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir (+112) 


``````````diff
diff --git a/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp b/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
index 4cd6fdf4f7341..e2dd0a6d79a73 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/ConvertConv2DToImg2Col.cpp
@@ -54,16 +54,16 @@ static Value createMul(Location loc, Value x, Value y, Type accType,
 }
 
 // Generate the affine expression to compute the convolved index
-// for the input as `oIndex * stride + fIndex`,
+// for the input as `oIndex * stride + fIndex * dilation`,
 // where oIndex: output iterator; fIndex: filter iterator.
 static AffineExpr getConvolvedExpr(OpBuilder &b, int64_t stride,
-                                   bool useSymbols = true) {
+                                   int64_t dilation, bool useSymbols = true) {
   AffineExpr oExpr, fExpr;
   if (useSymbols)
     bindSymbols(b.getContext(), oExpr, fExpr);
   else
     bindDims(b.getContext(), oExpr, fExpr);
-  return AffineExpr(stride * oExpr + fExpr);
+  return AffineExpr(stride * oExpr + dilation * fExpr);
 }
 
 // Stores the affine expressions to map the iteration space of the im2col matrix
@@ -91,12 +91,14 @@ struct Im2ColToInputDimsExprs {
 ///
 /// @param exprs      Affine expressions for output and filter indices.
 /// @param strides    [height, width] stride values for the convolution.
+/// @param dilations  [height, width] dilation values for the convolution.
 /// @param rewriter   Pattern rewriter.
 /// @return           Affine expressions mapping im2col matrix indices to input
 /// offsets.
 static Im2ColToInputDimsExprs
 getIm2ColInputExpressions(Im2ColToOperandsExprs exprs,
-                          ArrayRef<int64_t> strides, RewriterBase &rewriter) {
+                          ArrayRef<int64_t> strides,
+                          ArrayRef<int64_t> dilations, RewriterBase &rewriter) {
   // maps the iteration space of the im2col matrix to (output_y, filter_y)
   auto hIndicesMap = AffineMap::inferFromExprList(
       {ArrayRef{exprs.ohIndex, exprs.fhIndex}}, rewriter.getContext())[0];
@@ -110,10 +112,10 @@ getIm2ColInputExpressions(Im2ColToOperandsExprs exprs,
   // then we compose them with the maps that map the im2col matrix elements to
   // the (out_element, filter_element) pairs.
   auto bIndexExpr = rewriter.getAffineDimExpr(0U);
-  auto hIndexExpr = getConvolvedExpr(rewriter, strides[0],
+  auto hIndexExpr = getConvolvedExpr(rewriter, strides[0], dilations[0],
                                      /*useSymbols*/ false);
   hIndexExpr = hIndexExpr.compose(hIndicesMap);
-  auto wIndexExpr = getConvolvedExpr(rewriter, strides[1],
+  auto wIndexExpr = getConvolvedExpr(rewriter, strides[1], dilations[1],
                                      /*useSymbols*/ false);
   wIndexExpr = wIndexExpr.compose(wIndicesMap);
   auto cIndexExpr = exprs.icIndex;
@@ -138,11 +140,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcHwcfOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   MLIRContext *context = rewriter.getContext();
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
@@ -203,10 +200,10 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcHwcfOp convOp) {
   i2cToOperExprs.ohIndex = mIndicesExprs[0];
   i2cToOperExprs.owIndex = mIndicesExprs[1];
 
-  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + fh, sw*ow + fw, ic]
+  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + dh*fh, sw*ow + dw*fw, ic]
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.hIndex,
                                              inExprs.wIndex, inExprs.cIndex}},
@@ -431,11 +428,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNchwFchwOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
   Value output = convOp.getOutputs()[0];
@@ -497,12 +489,12 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNchwFchwOp convOp) {
   i2cToOperExprs.owIndex = mIndicesExprs[1];
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.cIndex,
                                              inExprs.hIndex, inExprs.wIndex}},
                                    rewriter.getContext())[0];
-  // im2col[n, ic*fh*fw, oh*ow] = input[n, ic, sh*oh + fh, sw*ow + fw]
+  // im2col[n, ic*fh*fw, oh*ow] = input[n, ic, sh*oh + dh*fh, sw*ow + dw*fw]
   SmallVector<AffineMap> img2colIndexingMaps = {
       inMap, AffineMap::getMultiDimIdentityMap(nloops, context)};
 
@@ -565,11 +557,6 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcFhwcOp convOp) {
     return rewriter.notifyMatchFailure(convOp,
                                        "expected a static shape for the input");
 
-  // TODO: Support dilation.
-  if (!hasAllOneValues(convOp.getDilations()))
-    return rewriter.notifyMatchFailure(convOp,
-                                       "expected all ones for dilations");
-
   MLIRContext *context = rewriter.getContext();
   Value input = convOp.getInputs()[0];
   Value filter = convOp.getInputs()[1];
@@ -632,10 +619,10 @@ rewriteInIm2Col(RewriterBase &rewriter, linalg::Conv2DNhwcFhwcOp convOp) {
   i2cToOperExprs.ohIndex = mIndicesExprs[0];
   i2cToOperExprs.owIndex = mIndicesExprs[1];
 
-  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + fh, sw*ow + fw, ic]
+  // im2col[n, oh*ow, fh*fw*ic] = input[n, sh*oh + dh*fh, sw*ow + dw*fw, ic]
   Im2ColToInputDimsExprs inExprs = getIm2ColInputExpressions(
       i2cToOperExprs, llvm::to_vector(convOp.getStrides().getValues<int64_t>()),
-      rewriter);
+      llvm::to_vector(convOp.getDilations().getValues<int64_t>()), rewriter);
   auto inMap =
       AffineMap::inferFromExprList({ArrayRef{inExprs.bIndex, inExprs.hIndex,
                                              inExprs.wIndex, inExprs.cIndex}},
diff --git a/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir b/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
index 152a392afe247..387f9762d518d 100644
--- a/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
+++ b/mlir/test/Dialect/Linalg/convert-conv2d-to-img2col.mlir
@@ -556,3 +556,115 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// -----
+
+// Dilated NHWC-HWCF: the im2col gather reads input at
+// oh*stride + fh*dilation, i.e. d1 floordiv 12 + (d2 floordiv 12) * 2.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 12 + (d2 floordiv 12) * 2, d1 mod 12 + ((d2 mod 12) floordiv 4) * 2, d2 mod 4)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nhwc_hwcf_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<1x16x16x4xf32>, %[[FILTER:.+]]: tensor<3x3x4x16xf32>, %[[INIT:.+]]: tensor<1x12x12x16xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0, 1, 2], [3]] : tensor<3x3x4x16xf32> into tensor<36x16xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1, 2], [3]] : tensor<1x12x12x16xf32> into tensor<1x144x16xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<1x144x36xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<1x16x16x4xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<1x144x36xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[IMG2COL]], %[[CS_FILTER]] : tensor<1x144x36xf32>, tensor<36x16xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<1x144x16xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1, 2], [3]] output_shape [1, 12, 12, 16] : tensor<1x144x16xf32> into tensor<1x12x12x16xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nhwc_hwcf_dilated(%arg0: tensor<1x16x16x4xf32>, %arg1: tensor<3x3x4x16xf32>, %arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32> {
+    %0 = linalg.conv_2d_nhwc_hwcf
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<1x16x16x4xf32>, tensor<3x3x4x16xf32>)
+      outs(%arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32>
+    return %0 : tensor<1x12x12x16xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nhwc_hwcf"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Dilated NCHW-FCHW: h index is d2 floordiv 12 + ((d1 mod 9) floordiv 3) * 2.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 9, d2 floordiv 12 + ((d1 mod 9) floordiv 3) * 2, d2 mod 12 + (d1 mod 3) * 2)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nchw_fchw_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<8x4x16x16xf32>, %[[FILTER:.+]]: tensor<16x4x3x3xf32>, %[[INIT:.+]]: tensor<8x16x12x12xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0], [1, 2, 3]] : tensor<16x4x3x3xf32> into tensor<16x36xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1], [2, 3]] : tensor<8x16x12x12xf32> into tensor<8x16x144xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<8x36x144xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<8x4x16x16xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<8x36x144xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[CS_FILTER]], %[[IMG2COL]] : tensor<16x36xf32>, tensor<8x36x144xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<8x16x144xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1], [2, 3]] output_shape [8, 16, 12, 12] : tensor<8x16x144xf32> into tensor<8x16x12x12xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nchw_fchw_dilated(%arg0: tensor<8x4x16x16xf32>, %arg1: tensor<16x4x3x3xf32>, %arg2: tensor<8x16x12x12xf32>) -> tensor<8x16x12x12xf32> {
+    %0 = linalg.conv_2d_nchw_fchw
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<8x4x16x16xf32>, tensor<16x4x3x3xf32>)
+      outs(%arg2: tensor<8x16x12x12xf32>) -> tensor<8x16x12x12xf32>
+    return %0 : tensor<8x16x12x12xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nchw_fchw"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Dilated NHWC-FHWC: same dilated input gather map as NHWC-HWCF.
+
+//  CHECK-DAG: #[[MAP:.+]] = affine_map<(d0, d1, d2) -> (d0, d1 floordiv 12 + (d2 floordiv 12) * 2, d1 mod 12 + ((d2 mod 12) floordiv 4) * 2, d2 mod 4)>
+//  CHECK-DAG: #[[MAPI2C:.+]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+//      CHECK: func.func @conv_nhwc_fhwc_dilated
+// CHECK-SAME: (%[[INPUT:.+]]: tensor<1x16x16x4xf32>, %[[FILTER:.+]]: tensor<16x3x3x4xf32>, %[[INIT:.+]]: tensor<1x12x12x16xf32>)
+//  CHECK-DAG:   %[[CS_FILTER:.+]] = tensor.collapse_shape %[[FILTER]] {{\[}}[0], [1, 2, 3]] : tensor<16x3x3x4xf32> into tensor<16x36xf32>
+//  CHECK-DAG:   %[[CS_RESULT:.+]] = tensor.collapse_shape %[[INIT]] {{\[}}[0], [1, 2], [3]] : tensor<1x12x12x16xf32> into tensor<1x144x16xf32>
+//      CHECK:   %[[IT:.+]] = tensor.empty() : tensor<1x144x36xf32>
+//      CHECK:   %[[IMG2COL:.+]] = linalg.generic
+// CHECK-SAME:      indexing_maps = [#[[MAP]], #[[MAPI2C]]]
+// CHECK-SAME:   ins(%[[INPUT]] : tensor<1x16x16x4xf32>)
+// CHECK-SAME:   outs(%[[IT]] : tensor<1x144x36xf32>)
+//      CHECK:   %[[MATMUL:.+]] = linalg.generic
+// CHECK-SAME:   ins(%[[IMG2COL]], %[[CS_FILTER]] : tensor<1x144x36xf32>, tensor<16x36xf32>)
+// CHECK-SAME:   outs(%[[CS_RESULT]] : tensor<1x144x16xf32>)
+//      CHECK:   %[[CS_FINAL:.+]] = tensor.expand_shape %[[MATMUL]] {{\[}}[0], [1, 2], [3]] output_shape [1, 12, 12, 16] : tensor<1x144x16xf32> into tensor<1x12x12x16xf32>
+//      CHECK:   return %[[CS_FINAL]]
+func.func @conv_nhwc_fhwc_dilated(%arg0: tensor<1x16x16x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32> {
+    %0 = linalg.conv_2d_nhwc_fhwc
+      {dilations = dense<2> : tensor<2xi64>, strides = dense<1> : tensor<2xi64> }
+       ins(%arg0, %arg1: tensor<1x16x16x4xf32>, tensor<16x3x3x4xf32>)
+      outs(%arg2: tensor<1x12x12x16xf32>) -> tensor<1x12x12x16xf32>
+    return %0 : tensor<1x12x12x16xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match ops{["linalg.conv_2d_nhwc_fhwc"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %1:2 = transform.structured.convert_conv2d_to_img2col %0 : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208424

From llvmlistbot at llvm.org  Thu Jul  9 04:05:18 2026
From: llvmlistbot at llvm.org (Zmicier Prybysh)
Date: Thu, 09 Jul 2026 04:05:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Support non-unit dilations in
 im2col decomposition (PR #208424)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208424@github.com>
Message-ID: <6a4f806e.eb165e5c.1817a5.e403@mx.google.com>


dimp-pl wrote:

@banach-space @GeorgeARM @fabrizio-indirli can i ask one of you to review?

https://github.com/llvm/llvm-project/pull/208424

From llvmlistbot at llvm.org  Thu Jul  9 04:27:58 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Thu, 09 Jul 2026 04:27:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4f85be.492e23e4.157d47.4c13@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 1246175402141857ea45fec146e1a9f35770d2e5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 01/11] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index b92bc1cd31723..a675ce431f567 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1106,7 +1106,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..5e3b7c8df515e 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,6 +700,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From 1cff6f6c7644ebb0467cffb4a64b11928874c8a4 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 02/11] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From cae401fd4789b033cb9e96c2d5ba3514f476dc02 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 03/11] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 5e3b7c8df515e..7fb68681da4c0 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -742,7 +742,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -759,22 +758,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -809,7 +792,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -826,22 +808,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From 16d21ca1d5b31bed58998ff867d95ee20d20913e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 04/11] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 8345ebf05e15824210ea60c40901795f2c5dd69f Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 05/11] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 7fb68681da4c0..8c96eb7843de4 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,60 +750,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -819,6 +813,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From d68e1aa32f62c683726d2827460e805bc4772968 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 06/11] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8c96eb7843de4..98973a733f063 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,54 +750,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -813,9 +829,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 467746b1a968238fc1771e00feb745f7a24580a2 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 07/11] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 98973a733f063..607888b678df1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -806,9 +806,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From bd1e5df96af41323971ac778226419735aee779b Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 08/11] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 607888b678df1..8dd67411292dd 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -794,10 +794,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir

>From 9da2835665afefdd93f3adc06cb50edec612e24c Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:46:13 +0000
Subject: [PATCH 09/11] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index bb0bbc2d0a284..d2f7fe768930b 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2121,10 +2121,9 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerExtf(nvgpu::ExtfOp op,
-                               nvgpu::ExtfOp::Adaptor adaptor,
-                                ConversionPatternRewriter &rewriter,
-                                const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
+                               ConversionPatternRewriter &rewriter,
+                               const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i8Ty = b.getI8Type();
@@ -2271,8 +2270,7 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUExtfOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
@@ -2434,6 +2432,6 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUExtfCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUTruncfCanonicalizePattern, NVGPUExtfCanonicalizePattern>(
+      patterns.getContext());
 }

>From 901c37448621ec25ebc7d5204a0bfa23388b38f3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:56:56 +0000
Subject: [PATCH 10/11] convert single use function to lambda

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index d2f7fe768930b..67d3b2e86293a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1751,11 +1751,6 @@ static std::optional<FPKind> classifyFPType(Type t) {
   return std::nullopt;
 }
 
-/// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConvert(FPKind src) {
-  return src == FPKind::F32 ? 2 : 1;
-}
-
 /// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
@@ -1985,6 +1980,11 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
+
+  // Number of source-side i32 register slots consumed by each NVVM convert Op.
+  auto getNumSrcI32PerConvert = [](FPKind src) {
+    return src == FPKind::F32 ? 2 : 1;
+  };
   int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.

>From 547ebf0ea052d560f2d1d2d3a4f37855be2c79da Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 9 Jul 2026 11:26:53 +0000
Subject: [PATCH 11/11] address comment

---
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8dd67411292dd..817cb8595f9ca 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -764,7 +764,9 @@ LogicalResult TruncfOp::verify() {
       return emitOpError("expects RZ or RP rounding mode when result type is "
                          "e8m0, but got ")
              << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+  }
+  
+  if (rnd == mlir::NVVM::FPRoundingMode::RS) {
     // TODO: Currently, we only support conversions which fit into a single i32
     // register. Support f32->f8/f6/f4 conversions with RS rounding.
     if (!(srcBitWidth == 32 && dstBitWidth == 16))
@@ -773,19 +775,24 @@ LogicalResult TruncfOp::verify() {
              << srcType << " -> " << dstType;
     if (!getRandomBits())
       return emitOpError("random_bits operand is required with RS rounding");
-  } else if (srcType.isF64() && dstBitWidth >= 16) {
+  }
+  
+  if (srcType.isF64() && dstBitWidth >= 16) {
     if (rnd != mlir::NVVM::FPRoundingMode::RN)
       return emitOpError("expects RN rounding mode for f64 input, but got ")
              << getRndAttr();
-  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+  }
+  
+  if (srcBitWidth == 32 && dstBitWidth == 16) {
     if (rnd != mlir::NVVM::FPRoundingMode::RN &&
         rnd != mlir::NVVM::FPRoundingMode::RZ)
       return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
                          "but got ")
              << getRndAttr();
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
   }
+  
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
 
   if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
     return emitOpError("random_bits can only be used with RS rounding mode");


From llvmlistbot at llvm.org  Thu Jul  9 04:28:25 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Thu, 09 Jul 2026 04:28:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4f85d9.eb24c6ea.27aedc.13a8@mx.google.com>


================

----------------
Wolfram70 wrote:

Makes sense, fixed! Thanks!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Thu Jul  9 04:47:39 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Thu, 09 Jul 2026 04:47:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4f8a5b.743400b0.2067c9.d38e@mx.google.com>


================
@@ -699,6 +699,138 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
----------------
grypp wrote:

tensor dialect has different semantic than vector or memref. it's SSA value and not in memory. It needs to be materialized aka bufferized. 

Current pipelines are following. If you support tensor on gpu, this requires some pipeline change since currently nvgpu is lowered down in the stack.   
- tensor
- bufferize
- ... bunch oh others...
- nvgpu-to-nvvm
- convert-llvm

If we don't have direct user for tensor dialect, let's exclude it from the PR.  


https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Thu Jul  9 04:48:46 2026
From: llvmlistbot at llvm.org (Philipp Schilk)
Date: Thu, 09 Jul 2026 04:48:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208240@github.com>
Message-ID: <6a4f8a9e.92ea0e2a.13fd37.db6a@mx.google.com>


https://github.com/schilkp edited https://github.com/llvm/llvm-project/pull/208240

From llvmlistbot at llvm.org  Thu Jul  9 05:59:05 2026
From: llvmlistbot at llvm.org (Umang Yadav)
Date: Thu, 09 Jul 2026 05:59:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ROCm] Build mlir_rocm_runtime
	STANDALONE (PR #205651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205651@github.com>
Message-ID: <6a4f9b19.0c05e68e.1e82f1.bc9a@mx.google.com>


https://github.com/umangyadav updated https://github.com/llvm/llvm-project/pull/205651

>From 20c637dcf10214aea7a424c728ca2c17e69680b1 Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Wed, 24 Jun 2026 19:15:48 +0000
Subject: [PATCH 1/4] [mlir] Build mlir_rocm_runtime STANDALONE

Mark mlir_rocm_runtime STANDALONE so it no longer links LLVMSupport,
mirroring mlir_cuda_runtime, and drop the SmallVector/ArrayRef use in
mgpuMemHostRegisterMemRef in favor of std equivalents. This removes the
DISABLE_PCH_REUSE workaround, which a STANDALONE library no longer needs.

Linking LLVMSupport exported ~1300 default-visibility llvm::* symbols.
When the runtime is dlopen-ed into a process that also reaches ROCm's
libLLVM.so (via the libamdhip64 -> libamd_comgr -> libLLVM.so DT_NEEDED
chain), the dynamic linker interposed these data symbols, causing the two
LLVM instances to share cl::opt / pass-registry / ManagedStatic singletons
and abort during dynamic init (SmallPtrSet.h: Assertion `Bucket < End').

Co-authored-by: Cursor <cursoragent at cursor.com>
---
 mlir/lib/ExecutionEngine/CMakeLists.txt       | 13 +++---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 41 +++++++++++--------
 2 files changed, 31 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/CMakeLists.txt b/mlir/lib/ExecutionEngine/CMakeLists.txt
index 87af4724f159a..3199c88313087 100644
--- a/mlir/lib/ExecutionEngine/CMakeLists.txt
+++ b/mlir/lib/ExecutionEngine/CMakeLists.txt
@@ -403,12 +403,13 @@ if(LLVM_ENABLE_PIC)
 
       EXCLUDE_FROM_LIBMLIR
 
-      # TODO: this is merely a workaround. If this library depends on LLVMSupport,
-      # it should suppress symbols, or if it doesn't, it shouldn't link against
-      # it. This workaround prevents the library from defining the symbol
-      # llvm::EnableABIBreakingChecks, which would cause ODR-violations when
-      # dlopen-ed.
-      DISABLE_PCH_REUSE
+      # This library is dlopen-ed (e.g. by mlir-runner via --shared-libs),
+      # frequently into a process that already hosts another LLVM (ROCm's own
+      # libLLVM.so is reachable through the libamdhip64 -> libamd_comgr
+      # DT_NEEDED chain). Linking LLVMSupport here would duplicate LLVM's
+      # cl::opt / pass-registry / ManagedStatic singletons and abort during
+      # dynamic init. The wrappers don't need LLVMSupport, so build STANDALONE.
+      STANDALONE
     )
 
     # Supress compiler warnings from HIP headers
diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 251245106c56e..86090f410c897 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -13,13 +13,17 @@
 //===----------------------------------------------------------------------===//
 
 #include <cassert>
-#include <numeric>
+#include <cstdint>
+#include <cstdio>
 
 #include "mlir/ExecutionEngine/CRunnerUtils.h"
-#include "llvm/ADT/ArrayRef.h"
 
 #include "hip/hip_runtime.h"
 
+#ifdef _WIN32
+#include <malloc.h>
+#endif // _WIN32
+
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -167,22 +171,25 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-
-  llvm::SmallVector<int64_t, 4> denseStrides(rank);
-  llvm::ArrayRef<int64_t> sizes(descriptor->sizes, rank);
-  llvm::ArrayRef<int64_t> strides(sizes.end(), rank);
-
-  std::partial_sum(sizes.rbegin(), sizes.rend(), denseStrides.rbegin(),
-                   std::multiplies<int64_t>());
-  auto sizeBytes = denseStrides.front() * elementSizeBytes;
-
   // Only densely packed tensors are currently supported.
-  std::rotate(denseStrides.begin(), denseStrides.begin() + 1,
-              denseStrides.end());
-  denseStrides.back() = 1;
-  assert(strides == llvm::ArrayRef(denseStrides));
-
-  auto ptr = descriptor->data + descriptor->offset * elementSizeBytes;
+#ifdef _WIN32
+  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
+#else
+  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
+#endif // _WIN32
+  int64_t *sizes = descriptor->sizes;
+  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
+    denseStrides[i] = runningStride;
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
+  int64_t *strides = &sizes[rank];
+  (void)strides;
+  for (unsigned i = 0; i < rank; ++i)
+    assert(strides[i] == denseStrides[i] &&
+           "Mismatch in computed dense strides");
+
+  auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);
 }
 

>From 722956fc7dd48daeb57b86d768950330555f0b0d Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:07:28 +0000
Subject: [PATCH 2/4] make implementation withotu alloca, we only need to check
 for dense memref, we can keep runningStrides

---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 25 ++++++-------------
 1 file changed, 7 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 86090f410c897..47977a246f0bf 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -20,10 +20,6 @@
 
 #include "hip/hip_runtime.h"
 
-#ifdef _WIN32
-#include <malloc.h>
-#endif // _WIN32
-
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -171,23 +167,16 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-  // Only densely packed tensors are currently supported.
-#ifdef _WIN32
-  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
-#else
-  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
-#endif // _WIN32
   int64_t *sizes = descriptor->sizes;
-  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
-    denseStrides[i] = runningStride;
-    runningStride *= sizes[i];
-  }
-  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
   int64_t *strides = &sizes[rank];
-  (void)strides;
-  for (unsigned i = 0; i < rank; ++i)
-    assert(strides[i] == denseStrides[i] &&
+  int64_t runningStride = 1;
+  // Only densely packed tensors are currently supported.
+  for (int64_t i = rank - 1; i >= 0; --i) {
+    assert(strides[i] == runningStride &&
            "Mismatch in computed dense strides");
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = runningStride * elementSizeBytes;
 
   auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);

>From 6cfe8b9772766a10bb465c7c03ab9c942be7ea4c Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:13:27 +0000
Subject: [PATCH 3/4] Formatting

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 47977a246f0bf..445518111a244 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -172,8 +172,7 @@ mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {
-    assert(strides[i] == runningStride &&
-           "Mismatch in computed dense strides");
+    assert(strides[i] == runningStride && "Mismatch in computed dense strides");
     runningStride *= sizes[i];
   }
   uint64_t sizeBytes = runningStride * elementSizeBytes;

>From 86ee1a0d656e81ec458bdfa335fa06b7b4fe250b Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 9 Jul 2026 12:58:41 +0000
Subject: [PATCH 4/4] add `maybe_unused` for the release builds

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 445518111a244..42018b3722a8a 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -168,7 +168,7 @@ extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
   int64_t *sizes = descriptor->sizes;
-  int64_t *strides = &sizes[rank];
+  [[maybe_unused]] int64_t *strides = &sizes[rank];
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {


From llvmlistbot at llvm.org  Thu Jul  9 06:02:54 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Thu, 09 Jul 2026 06:02:54 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][MLIR][OpenMP] Move host op
 filtering to the omp dialect (PR #208189)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208189@github.com>
Message-ID: <6a4f9bfe.8e75e0cb.5e108.ffa2@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `ppc64le-flang-rhel-clang` running on `ppc64le-flang-rhel-test` while building `flang,mlir` at step 6 "test-build-unified-tree-check-flang".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/157/builds/48168

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 6 (test-build-unified-tree-check-flang) failure: 1200 seconds without output running [b'ninja', b'check-flang'], attempting to kill
...
PASS: Flang :: Transforms/OpenACC/acc-implicit-data-fortran.F90 (4683 of 4694)
PASS: Flang :: Transforms/dlti-dependency.fir (4684 of 4694)
PASS: Flang :: Lower/OpenACC/Todo/acc-unstructured-loop-construct.f90 (4685 of 4694)
PASS: flang-Unit :: Optimizer/./FlangOptimizerTests/171/174 (4686 of 4694)
PASS: Flang :: Driver/linker-flags.f90 (4687 of 4694)
PASS: Flang :: Driver/use-module-error.f90 (4688 of 4694)
PASS: Flang :: Driver/mrecip.f90 (4689 of 4694)
PASS: Flang :: Driver/nostdlib.f90 (4690 of 4694)
PASS: Flang :: Driver/options-loongarch.f90 (4691 of 4694)
PASS: Flang :: Driver/use-module.f90 (4692 of 4694)
command timed out: 1200 seconds without output running [b'ninja', b'check-flang'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=4246.350127

```

</details>

https://github.com/llvm/llvm-project/pull/208189

From llvmlistbot at llvm.org  Thu Jul  9 06:16:36 2026
From: llvmlistbot at llvm.org (Umang Yadav)
Date: Thu, 09 Jul 2026 06:16:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ROCm] Build mlir_rocm_runtime
	STANDALONE (PR #205651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205651@github.com>
Message-ID: <6a4f9f34.eb24c6ea.27aedc.ad7a@mx.google.com>


https://github.com/umangyadav updated https://github.com/llvm/llvm-project/pull/205651

>From 20c637dcf10214aea7a424c728ca2c17e69680b1 Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Wed, 24 Jun 2026 19:15:48 +0000
Subject: [PATCH 1/4] [mlir] Build mlir_rocm_runtime STANDALONE

Mark mlir_rocm_runtime STANDALONE so it no longer links LLVMSupport,
mirroring mlir_cuda_runtime, and drop the SmallVector/ArrayRef use in
mgpuMemHostRegisterMemRef in favor of std equivalents. This removes the
DISABLE_PCH_REUSE workaround, which a STANDALONE library no longer needs.

Linking LLVMSupport exported ~1300 default-visibility llvm::* symbols.
When the runtime is dlopen-ed into a process that also reaches ROCm's
libLLVM.so (via the libamdhip64 -> libamd_comgr -> libLLVM.so DT_NEEDED
chain), the dynamic linker interposed these data symbols, causing the two
LLVM instances to share cl::opt / pass-registry / ManagedStatic singletons
and abort during dynamic init (SmallPtrSet.h: Assertion `Bucket < End').

Co-authored-by: Cursor <cursoragent at cursor.com>
---
 mlir/lib/ExecutionEngine/CMakeLists.txt       | 13 +++---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 41 +++++++++++--------
 2 files changed, 31 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/CMakeLists.txt b/mlir/lib/ExecutionEngine/CMakeLists.txt
index 87af4724f159a..3199c88313087 100644
--- a/mlir/lib/ExecutionEngine/CMakeLists.txt
+++ b/mlir/lib/ExecutionEngine/CMakeLists.txt
@@ -403,12 +403,13 @@ if(LLVM_ENABLE_PIC)
 
       EXCLUDE_FROM_LIBMLIR
 
-      # TODO: this is merely a workaround. If this library depends on LLVMSupport,
-      # it should suppress symbols, or if it doesn't, it shouldn't link against
-      # it. This workaround prevents the library from defining the symbol
-      # llvm::EnableABIBreakingChecks, which would cause ODR-violations when
-      # dlopen-ed.
-      DISABLE_PCH_REUSE
+      # This library is dlopen-ed (e.g. by mlir-runner via --shared-libs),
+      # frequently into a process that already hosts another LLVM (ROCm's own
+      # libLLVM.so is reachable through the libamdhip64 -> libamd_comgr
+      # DT_NEEDED chain). Linking LLVMSupport here would duplicate LLVM's
+      # cl::opt / pass-registry / ManagedStatic singletons and abort during
+      # dynamic init. The wrappers don't need LLVMSupport, so build STANDALONE.
+      STANDALONE
     )
 
     # Supress compiler warnings from HIP headers
diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 251245106c56e..86090f410c897 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -13,13 +13,17 @@
 //===----------------------------------------------------------------------===//
 
 #include <cassert>
-#include <numeric>
+#include <cstdint>
+#include <cstdio>
 
 #include "mlir/ExecutionEngine/CRunnerUtils.h"
-#include "llvm/ADT/ArrayRef.h"
 
 #include "hip/hip_runtime.h"
 
+#ifdef _WIN32
+#include <malloc.h>
+#endif // _WIN32
+
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -167,22 +171,25 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-
-  llvm::SmallVector<int64_t, 4> denseStrides(rank);
-  llvm::ArrayRef<int64_t> sizes(descriptor->sizes, rank);
-  llvm::ArrayRef<int64_t> strides(sizes.end(), rank);
-
-  std::partial_sum(sizes.rbegin(), sizes.rend(), denseStrides.rbegin(),
-                   std::multiplies<int64_t>());
-  auto sizeBytes = denseStrides.front() * elementSizeBytes;
-
   // Only densely packed tensors are currently supported.
-  std::rotate(denseStrides.begin(), denseStrides.begin() + 1,
-              denseStrides.end());
-  denseStrides.back() = 1;
-  assert(strides == llvm::ArrayRef(denseStrides));
-
-  auto ptr = descriptor->data + descriptor->offset * elementSizeBytes;
+#ifdef _WIN32
+  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
+#else
+  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
+#endif // _WIN32
+  int64_t *sizes = descriptor->sizes;
+  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
+    denseStrides[i] = runningStride;
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
+  int64_t *strides = &sizes[rank];
+  (void)strides;
+  for (unsigned i = 0; i < rank; ++i)
+    assert(strides[i] == denseStrides[i] &&
+           "Mismatch in computed dense strides");
+
+  auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);
 }
 

>From 722956fc7dd48daeb57b86d768950330555f0b0d Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:07:28 +0000
Subject: [PATCH 2/4] make implementation withotu alloca, we only need to check
 for dense memref, we can keep runningStrides

---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 25 ++++++-------------
 1 file changed, 7 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 86090f410c897..47977a246f0bf 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -20,10 +20,6 @@
 
 #include "hip/hip_runtime.h"
 
-#ifdef _WIN32
-#include <malloc.h>
-#endif // _WIN32
-
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -171,23 +167,16 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-  // Only densely packed tensors are currently supported.
-#ifdef _WIN32
-  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
-#else
-  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
-#endif // _WIN32
   int64_t *sizes = descriptor->sizes;
-  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
-    denseStrides[i] = runningStride;
-    runningStride *= sizes[i];
-  }
-  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
   int64_t *strides = &sizes[rank];
-  (void)strides;
-  for (unsigned i = 0; i < rank; ++i)
-    assert(strides[i] == denseStrides[i] &&
+  int64_t runningStride = 1;
+  // Only densely packed tensors are currently supported.
+  for (int64_t i = rank - 1; i >= 0; --i) {
+    assert(strides[i] == runningStride &&
            "Mismatch in computed dense strides");
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = runningStride * elementSizeBytes;
 
   auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);

>From 6cfe8b9772766a10bb465c7c03ab9c942be7ea4c Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:13:27 +0000
Subject: [PATCH 3/4] Formatting

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 47977a246f0bf..445518111a244 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -172,8 +172,7 @@ mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {
-    assert(strides[i] == runningStride &&
-           "Mismatch in computed dense strides");
+    assert(strides[i] == runningStride && "Mismatch in computed dense strides");
     runningStride *= sizes[i];
   }
   uint64_t sizeBytes = runningStride * elementSizeBytes;

>From 86ee1a0d656e81ec458bdfa335fa06b7b4fe250b Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 9 Jul 2026 12:58:41 +0000
Subject: [PATCH 4/4] add `maybe_unused` for the release builds

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 445518111a244..42018b3722a8a 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -168,7 +168,7 @@ extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
   int64_t *sizes = descriptor->sizes;
-  int64_t *strides = &sizes[rank];
+  [[maybe_unused]] int64_t *strides = &sizes[rank];
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {


From llvmlistbot at llvm.org  Thu Jul  9 06:19:37 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Thu, 09 Jul 2026 06:19:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4f9fe9.743400b0.2067c9.4bb8@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
----------------
MaxGraey wrote:

Does this work on GCC <= 14.x?

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 06:22:45 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Thu, 09 Jul 2026 06:22:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fa0a5.4a342cbc.ca844.88a6@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
----------------
MaxGraey wrote:

How about to simplify and ged rid of optional?
```cpp
Location getCurrentOpLoc() const {
  assert(currentOpLoc && "expected current opcode location to be set");
  return *currentOpLoc;
}
```

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 06:30:55 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Thu, 09 Jul 2026 06:30:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Support conditional
 lastprivate on host (PR #200086)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200086@github.com>
Message-ID: <6a4fa28f.2020f6c2.243e78.e8cb@mx.google.com>


tblah wrote:

Thanks for following up. I am still finding serious correctness issues in this
patch after several review rounds. Previous rounds caught unsupported constructs
being silently accepted, missing type restrictions, handling problems for some
types, copyback correctness issues, and other semantic problems.

My latest findings are:

- interaction with firstprivate: the loop body reads the variable from the
  zero-initialized struct created for conditional lastprivate, not from the
  firstprivate copy.
- `distribute parallel do simd lastprivate(conditional:)` does not hit the SIMD
  TODO and is silently lowered.
- missing test coverage under `mlir/test/Target/...` for the
  `OpenMPToLLVMIRTranslation.cpp` change.

This is taking a disproportionate amount of reviewer time to construct tests,
verify the behavior, and communicate the results in a useful way.

For the next revision, please add a substantial test matrix covering the
supported and unsupported OpenMP constructs, the relevant supported and
unsupported list item categories, orphaned and non-orphaned forms, diagnostics
or TODOs for unsupported behavior, interactions with other clauses that
influence symbol mapping, and multiple conditional lastprivate variables.

These tests should each be minimal lit tests local to the layer being tested.
For example, cases that should be caught by Flang semantics should be tested
alongside the other Flang semantic checks. Changes to MLIR-to-LLVMIR translation
should have minimal lit coverage alongside the other tests for that translation
unit.

You do not have to support every type or construct combination in this PR. But
at each interface level, both Fortran and MLIR, every accepted case should
either lower correctly or fail gracefully with a diagnostic/TODO. There should
be no silent miscompilations, assertion failures, or compiler crashes.

Once the PR is more strongly tested, I will take another look.


https://github.com/llvm/llvm-project/pull/200086

From llvmlistbot at llvm.org  Thu Jul  9 06:35:05 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Thu, 09 Jul 2026 06:35:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix reduce-transpose sub-byte
	handling (PR #208446)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208446@github.com>
Message-ID: <6a4fa389.2020f6c2.243e78.83d0@mx.google.com>


https://github.com/iliyan-georgiev-arm updated https://github.com/llvm/llvm-project/pull/208446

>From 8da7e598baf10207aa150110c11590eb0bfefe63 Mon Sep 17 00:00:00 2001
From: Iliyan Georgiev <Iliyan.Georgiev at arm.com>
Date: Wed, 8 Jul 2026 14:30:22 +0100
Subject: [PATCH] [mlir][tosa] Fix reduce-transpose sub-byte handling

- Fix element size calculation when processing sub-byte types
- Add tests for i1 and i4

Signed-off-by: Iliyan Georgiev <Iliyan.Georgiev at arm.com>
Change-Id: I06988256384550738becf5691d5880366293d58d
---
 .../Tosa/Transforms/TosaReduceTransposes.cpp   |  3 +--
 .../Dialect/Tosa/tosa-reduce-transposes.mlir   | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
index 0bd415040f22f..ffd48e8ab16ae 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaReduceTransposes.cpp
@@ -233,8 +233,7 @@ TosaReduceTransposes::transposeDenseAttribute(DenseElementsAttr input,
   // perms: dim 0→2, dim 1→0, dim 2→1, giving source position
   // calculated as 1*inputStrides[2] + 1*inputStrides[0] + 2*inputStrides[1]
   // = 1*1 + 1*12 + 2*4 = 21
-
-  size_t elementSize = oldType.getElementTypeBitWidth() / 8;
+  size_t elementSize = llvm::divideCeil(oldType.getElementTypeBitWidth(), 8);
   int64_t numElements = oldType.getNumElements();
 
   SmallVector<char> outputBuffer(numElements * elementSize);
diff --git a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
index 0ad3068011f7f..46e4f504af0ec 100644
--- a/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-reduce-transposes.mlir
@@ -597,3 +597,21 @@ func.func @test_unimplemented_static_diverges_to_one_nullifying_one_non_nullifyi
   %result = tosa.transpose %add {perms = array<i32: 0, 3, 1, 2>}: (tensor<1x3x4x2xi32>) -> tensor<1x2x3x4xi32>
   return %result : tensor<1x2x3x4xi32>
 }
+
+// CHECK-LABEL: @test_transpose_bool
+// CHECK: %{{.*}} = "tosa.const"()
+// CHECK-SAME{LITERAL}: dense<[[true, false], [false, false], [false, true]]>
+func.func @test_transpose_bool() -> tensor<3x2xi1> {
+  %0 = "tosa.const"() <{values = dense<[[true, false, false], [false, false, true]]> : tensor<2x3xi1>}> : () -> tensor<2x3xi1>
+  %1 = tosa.transpose %0 {perms = array<i32: 1, 0>} : (tensor<2x3xi1>) -> tensor<3x2xi1>
+  return %1 : tensor<3x2xi1>
+}
+
+// CHECK-LABEL: @test_transpose_i4
+// CHECK: %{{.*}} = "tosa.const"()
+// CHECK-SAME{LITERAL}: dense<[[1, 4], [2, 5], [3, 6]]>
+func.func @test_transpose_i4() -> tensor<3x2xi4> {
+  %0 = "tosa.const"() <{values = dense<[[1, 2, 3], [4, 5, 6]]> : tensor<2x3xi4>}> : () -> tensor<2x3xi4>
+  %1 = tosa.transpose %0 {perms = array<i32: 1, 0>} : (tensor<2x3xi4>) -> tensor<3x2xi4>
+  return %1 : tensor<3x2xi4>
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul  9 06:44:24 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Thu, 09 Jul 2026 06:44:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4fa5b8.7270d2fd.2d4879.dd97@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 1246175402141857ea45fec146e1a9f35770d2e5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 01/11] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index b92bc1cd31723..a675ce431f567 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1106,7 +1106,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..5e3b7c8df515e 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,6 +700,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From 1cff6f6c7644ebb0467cffb4a64b11928874c8a4 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 02/11] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From cae401fd4789b033cb9e96c2d5ba3514f476dc02 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 03/11] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 5e3b7c8df515e..7fb68681da4c0 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -742,7 +742,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -759,22 +758,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -809,7 +792,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -826,22 +808,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From 16d21ca1d5b31bed58998ff867d95ee20d20913e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 04/11] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 8345ebf05e15824210ea60c40901795f2c5dd69f Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 05/11] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 7fb68681da4c0..8c96eb7843de4 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,60 +750,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -819,6 +813,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From d68e1aa32f62c683726d2827460e805bc4772968 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 06/11] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8c96eb7843de4..98973a733f063 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,54 +750,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -813,9 +829,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 467746b1a968238fc1771e00feb745f7a24580a2 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 07/11] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 98973a733f063..607888b678df1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -806,9 +806,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From bd1e5df96af41323971ac778226419735aee779b Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 08/11] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 607888b678df1..8dd67411292dd 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -794,10 +794,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir

>From 9da2835665afefdd93f3adc06cb50edec612e24c Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:46:13 +0000
Subject: [PATCH 09/11] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index bb0bbc2d0a284..d2f7fe768930b 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2121,10 +2121,9 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerExtf(nvgpu::ExtfOp op,
-                               nvgpu::ExtfOp::Adaptor adaptor,
-                                ConversionPatternRewriter &rewriter,
-                                const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
+                               ConversionPatternRewriter &rewriter,
+                               const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i8Ty = b.getI8Type();
@@ -2271,8 +2270,7 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUExtfOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
@@ -2434,6 +2432,6 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUExtfCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUTruncfCanonicalizePattern, NVGPUExtfCanonicalizePattern>(
+      patterns.getContext());
 }

>From 901c37448621ec25ebc7d5204a0bfa23388b38f3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:56:56 +0000
Subject: [PATCH 10/11] convert single use function to lambda

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index d2f7fe768930b..67d3b2e86293a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1751,11 +1751,6 @@ static std::optional<FPKind> classifyFPType(Type t) {
   return std::nullopt;
 }
 
-/// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConvert(FPKind src) {
-  return src == FPKind::F32 ? 2 : 1;
-}
-
 /// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
@@ -1985,6 +1980,11 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
+
+  // Number of source-side i32 register slots consumed by each NVVM convert Op.
+  auto getNumSrcI32PerConvert = [](FPKind src) {
+    return src == FPKind::F32 ? 2 : 1;
+  };
   int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.

>From e647f5b131237e4e17acfcb2e9966b0ddf38dcd8 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 9 Jul 2026 13:43:34 +0000
Subject: [PATCH 11/11] remove tensor type support

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  8 ++---
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  9 ------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 31 +++++--------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 24 +++-----------
 .../Dialect/NVGPU/nvgpu-truncf-invalid.mlir   | 24 +++-----------
 5 files changed, 20 insertions(+), 76 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 88ffc69c31cb2..d0b07de6f01bf 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -703,7 +703,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
     %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
-  let arguments = (ins FloatLike:$in,
+  let arguments = (ins AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
                        DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
                          [EnumAttrIsOneOf<SaturationModeAttr, 
@@ -712,7 +712,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs FloatLike:$out);
+  let results = (outs AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -738,10 +738,10 @@ def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
     %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins FloatLike:$in,
+  let arguments = (ins AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
+  let results = (outs AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 67d3b2e86293a..0642ad95d0f61 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2263,9 +2263,6 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   LogicalResult
   matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
     return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2276,9 +2273,6 @@ struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   LogicalResult
   matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
     return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2307,9 +2301,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    if (isa<RankedTensorType>(inType))
-      return failure();
-
     Type srcElemTy = getElementTypeOrSelf(inType);
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8dd67411292dd..c77ecb8431ad8 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -704,35 +704,20 @@ LogicalResult RcpOp::verify() {
 // NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
-static bool isShapedContainerType(Type t) {
-  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
-}
-
 static LogicalResult verifyConversionShapes(Operation *op, Type inType,
                                             Type outType) {
-  if (llvm::isa<UnrankedTensorType>(inType) ||
-      llvm::isa<UnrankedTensorType>(outType))
-    return op->emitOpError("unranked tensor types are not supported, got ")
-           << inType << " and " << outType;
-  bool srcIsShaped = isShapedContainerType(inType);
-  bool dstIsShaped = isShapedContainerType(outType);
-  if (srcIsShaped != dstIsShaped)
+  bool srcIsVector = llvm::isa<VectorType>(inType);
+  bool dstIsVector = llvm::isa<VectorType>(outType);
+  if (srcIsVector != dstIsVector)
     return op->emitOpError("input and output must both be scalars or both be "
-                           "vectors/tensors, got ")
+                           "vectors, got ")
            << inType << " and " << outType;
-  if (srcIsShaped) {
-    auto srcShaped = llvm::cast<ShapedType>(inType);
-    auto dstShaped = llvm::cast<ShapedType>(outType);
-    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
-      return op->emitOpError("rank-0 shaped types are not supported, use "
-                             "scalar type instead");
-    if (srcShaped.getShape() != dstShaped.getShape())
+  if (srcIsVector) {
+    auto srcVector = llvm::cast<VectorType>(inType);
+    auto dstVector = llvm::cast<VectorType>(outType);
+    if (srcVector.getShape() != dstVector.getShape())
       return op->emitOpError("input and output shapes must match, got ")
              << inType << " and " << outType;
-    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
-      return op->emitOpError("input and output must be the same container "
-                             "type (both vector or both tensor), got ")
-             << inType << " and " << outType;
   }
   return success();
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
index 874ea7ab0e6fc..ef659efb96f8e 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -52,31 +52,15 @@ func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors}}
   %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
-  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+func.func @fpext_rank0_vector(%in : vector<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op operand #0 must be floating-point or vector of floating-point values, but got 'vector<f8E4M3FN>'}}
+  %out = nvgpu.extf %in : vector<f8E4M3FN> to vector<f16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
index 9cee9720bddda..1b2fb3db5ceca 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
@@ -99,31 +99,15 @@ func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors}}
   %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
-func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
-  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
+func.func @fptrunc_rank0_vector(%in : vector<f32>) {
+  // expected-error @+1 {{'nvgpu.truncf' op operand #0 must be floating-point or vector of floating-point values, but got 'vector<f32>'}}
+  %out = nvgpu.truncf %in : vector<f32> to vector<f16>
   return
 }


From llvmlistbot at llvm.org  Thu Jul  9 06:46:28 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Thu, 09 Jul 2026 06:46:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4fa634.07d7ca98.9dee3.504e@mx.google.com>


================

----------------
Wolfram70 wrote:

Looks like this causes failures because the checks are interrelated and not independent. I couldn't find a way to split them without making the checks too long and hard to read. So, reverting this split for now.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Thu Jul  9 06:48:05 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Thu, 09 Jul 2026 06:48:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a4fa695.bb839515.27d21e.143f@mx.google.com>


================
@@ -699,6 +699,138 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
----------------
Wolfram70 wrote:

That makes sense. We currently cannot lower the tensor types either, so it looks fairly redundant for now. Perhaps we can add them back if it is required in the future.
Removed tensor type support from these Ops for now in the latest revision. Thanks!

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Thu Jul  9 06:50:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 06:50:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a4fa731.6c8668f2.e4f82.5955@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp mlir/lib/Target/LLVMIR/ModuleTranslation.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
index 835941d99..b6dee4e89 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
@@ -150,9 +150,9 @@ static LogicalResult setProfilingAttr(OpBuilder &builder, llvm::MDNode *node,
     }
 
     if (auto funcOp = dyn_cast<LLVMFuncOp>(op)) {
-      funcOp.setFunctionEntryCountAttr(FunctionEntryCountAttr::get(
-          builder.getContext(), *entryCountValue, profileCountType,
-          importGUIDValues));
+      funcOp.setFunctionEntryCountAttr(
+          FunctionEntryCountAttr::get(builder.getContext(), *entryCountValue,
+                                      profileCountType, importGUIDValues));
       return success();
     }
     return op->emitWarning()

``````````

</details>


https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Thu Jul  9 06:54:58 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 09 Jul 2026 06:54:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
Message-ID: <llvm.org/llvm/llvm-project/pull/208466@github.com>


https://github.com/nujaa created https://github.com/llvm/llvm-project/pull/208466

The test case added would crash because the Ranked Tensor-ness of operands was assumed but not checked. This crash only happens when there are multiple inputs as it must bailout somewhere else.

>From df521a8ba9b7907c65f42787e38a620c71a68406 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 21:41:33 +0800
Subject: [PATCH] [MLIR][Linalg] Fix LinalgSpecializeGenericOpsPass for scalar
 inputs

---
 .../DecomposeGenericByUnfoldingPermutation.cpp |  6 +++---
 .../Linalg/specialize-generic-ops-fail.mlir    | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 9015cbb096f88..450cfb2641cb5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -164,8 +164,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
   // out which operand can supply that runtime-value (tensor.dim).
   // Leaving it as a future TODO.
   if (llvm::any_of(op->getOpOperands(), [](OpOperand &oper) {
-        auto opType = cast<RankedTensorType>(oper.get().getType());
-        return ShapedType::isDynamicShape(opType.getShape());
+        auto opType = dyn_cast<RankedTensorType>(oper.get().getType());
+        return !opType || ShapedType::isDynamicShape(opType.getShape());
       }))
     return failure();
 
@@ -210,7 +210,7 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
       Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  inputRTType.getElementType());
+                                                  elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);
diff --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index 5d66837fca510..bb2c2edfa20f2 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -46,3 +46,21 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
   } -> tensor<8xi32>
   return %res : tensor<8xi32>
 }
+
+
+// -----
+
+// CHECK-LABEL: @scalar_input
+// CHECK: linalg.generic
+#map = affine_map<(d0) -> (d0)>
+#map1 = affine_map<(d0) -> ()>
+func.func @scalar_input(%arg0: tensor<128xi64>, %arg1: f32) -> tensor<128xf32> {
+  %0 = tensor.empty() : tensor<128xf32>
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xi64>, f32) outs(%0 : tensor<128xf32>) {
+  ^bb0(%in: i64, %in_1: f32, %out: f32):
+    %2 = arith.sitofp %in : i64 to f32
+    %3 = arith.divf %2, %in_1 : f32
+    linalg.yield %3 : f32
+  } -> tensor<128xf32>
+  return %1 : tensor<128xf32>
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul  9 06:55:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 06:55:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a4fa858.4a342cbc.ca844.824c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-linalg

Author: Hugo Trachino (nujaa)

<details>
<summary>Changes</summary>

The test case added would crash because the Ranked Tensor-ness of operands was assumed but not checked. This crash only happens when there are multiple inputs as it must bailout somewhere else.

---
Full diff: https://github.com/llvm/llvm-project/pull/208466.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp (+3-3) 
- (modified) mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir (+18) 


``````````diff
diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 9015cbb096f88..450cfb2641cb5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -164,8 +164,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
   // out which operand can supply that runtime-value (tensor.dim).
   // Leaving it as a future TODO.
   if (llvm::any_of(op->getOpOperands(), [](OpOperand &oper) {
-        auto opType = cast<RankedTensorType>(oper.get().getType());
-        return ShapedType::isDynamicShape(opType.getShape());
+        auto opType = dyn_cast<RankedTensorType>(oper.get().getType());
+        return !opType || ShapedType::isDynamicShape(opType.getShape());
       }))
     return failure();
 
@@ -210,7 +210,7 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
       Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  inputRTType.getElementType());
+                                                  elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);
diff --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index 5d66837fca510..bb2c2edfa20f2 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -46,3 +46,21 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
   } -> tensor<8xi32>
   return %res : tensor<8xi32>
 }
+
+
+// -----
+
+// CHECK-LABEL: @scalar_input
+// CHECK: linalg.generic
+#map = affine_map<(d0) -> (d0)>
+#map1 = affine_map<(d0) -> ()>
+func.func @scalar_input(%arg0: tensor<128xi64>, %arg1: f32) -> tensor<128xf32> {
+  %0 = tensor.empty() : tensor<128xf32>
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xi64>, f32) outs(%0 : tensor<128xf32>) {
+  ^bb0(%in: i64, %in_1: f32, %out: f32):
+    %2 = arith.sitofp %in : i64 to f32
+    %3 = arith.divf %2, %in_1 : f32
+    linalg.yield %3 : f32
+  } -> tensor<128xf32>
+  return %1 : tensor<128xf32>
+}
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Thu Jul  9 06:56:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 06:56:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a4fa893.6c8668f2.e4f82.54cb@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 450cfb264..8ab0f67d9 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -209,8 +209,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     // Does it require broadcast?
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
-      Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  elType);
+      Value emptyTensor =
+          tensor::EmptyOp::create(rewriter, loc, outputShape, elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);

``````````

</details>


https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Thu Jul  9 07:13:43 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:13:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fac97.91d695f4.18551.e6f5@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/5] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/5] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/5] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/5] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 5/5] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data


From llvmlistbot at llvm.org  Thu Jul  9 07:13:59 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:13:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4faca7.26e5e084.806c0.52d1@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/6] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/6] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/6] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/6] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 5/6] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 6/6] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751


From llvmlistbot at llvm.org  Thu Jul  9 07:14:40 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 09 Jul 2026 07:14:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4facd0.72a2efdc.189c2f.291c@mx.google.com>


https://github.com/lforg37 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {


From llvmlistbot at llvm.org  Thu Jul  9 07:17:11 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:17:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fad67.acd51bca.2dcbea.ef8a@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
----------------
steplong wrote:

Nope, looks like it runs fine. I'll remove it

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 07:18:24 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 09 Jul 2026 07:18:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fadb0.9352724e.ac20f.bd6a@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
----------------
lforg37 wrote:

I don't see why it wouldn't, is there some known bug on GCC?

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 07:19:08 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:19:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4faddc.00a9ba89.231f74.66ef@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/7] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/7] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/7] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/7] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 5/7] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 6/7] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 2f506c83f24eda7f22f58019c7dee22751464f58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 7/7] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s


From llvmlistbot at llvm.org  Thu Jul  9 07:21:23 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 09 Jul 2026 07:21:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fae63.88aec2a3.293ed9.fb09@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
----------------
lforg37 wrote:

I'm not even sure we need the assert as the optional dereferencement operator will error out anyway.

But given how it is used it makes sense indeed, will do. Thanks. 


https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 07:22:18 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 09 Jul 2026 07:22:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fae9a.14a6e4b5.8bf9d.287a@mx.google.com>


https://github.com/lforg37 edited https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 07:22:52 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:22:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4faebc.e0660744.28df04.b7b9@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/8] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/8] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/8] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/8] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 5/8] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 6/8] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 2f506c83f24eda7f22f58019c7dee22751464f58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 7/8] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From 0752206c6433d322eba660ae2abd83491ecf067f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 8/8] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///


From llvmlistbot at llvm.org  Thu Jul  9 07:24:18 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:24:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4faf12.12ed5c4a.1d7608.0703@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
----------------
steplong wrote:

Added comment

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 07:27:05 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:27:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fafb9.2020f6c2.243e78.a8fe@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
----------------
steplong wrote:

Yeah I just didn't pass `assume_dynamic_dims_match_vec_sizes` to demonstrate how they would be lowered

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 07:31:24 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 09 Jul 2026 07:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix reduce-transpose sub-byte
	handling (PR #208446)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208446@github.com>
Message-ID: <6a4fb0bc.f06f4994.1fc857.1b77@mx.google.com>


lhutton1 wrote:

Fixes: https://github.com/llvm/llvm-project/issues/203810

https://github.com/llvm/llvm-project/pull/208446

From llvmlistbot at llvm.org  Thu Jul  9 07:34:29 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Thu, 09 Jul 2026 07:34:29 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Implement
 DEPEND clause for TASKWAIT directive (PR #193568)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/193568@github.com>
Message-ID: <6a4fb175.07d7ca98.9dee3.f8f1@mx.google.com>


https://github.com/tblah approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/193568

From llvmlistbot at llvm.org  Thu Jul  9 07:35:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 07:35:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 06c0d1b - [flang][acc] Remove dangling
 operands in OpenACCUtilsTiling (#208261)
Message-ID: <6a4fb1bc.7270d2fd.2d4879.d06d@mx.google.com>


Author: Susan Tan (ス-ザン　タン)
Date: 2026-07-09T10:35:35-04:00
New Revision: 06c0d1b70402c1c55cdb1975b2a43148dd4fe646

URL: https://github.com/llvm/llvm-project/commit/06c0d1b70402c1c55cdb1975b2a43148dd4fe646
DIFF: https://github.com/llvm/llvm-project/commit/06c0d1b70402c1c55cdb1975b2a43148dd4fe646.diff

LOG: [flang][acc] Remove dangling operands in OpenACCUtilsTiling  (#208261)

When TILE has more dimensions than the loop's collapse count,
uncollapseLoops() synthesizes an inner loop via `createInnerLoop()`,
which
strips gang attributes from it but forgot to clear the gang operand
values. This left the loop with a non-empty gang operand list but no
device-type attribute, causing LoopOp::getGangValue() to dereference a
null attribute and crash with an assertion failure when the loop also
had
`GANG(STATIC:N) (or NUM/DIM)`.

Fix: also clear the gang operands when stripping gang attributes, so the
loop ends up fully gang-free.

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
    mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index dddaebdee5ebc..09e83af2bc686 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -97,10 +97,20 @@ createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
     elementLoop.removeGangOperandsArgTypeAttr();
     elementLoop.removeGangOperandsSegmentsAttr();
     elementLoop.removeGangOperandsDeviceTypeAttr();
+    // Also drop the operand values themselves so that elementLoop does not
+    // end up with a non-empty gang operand list but no corresponding
+    // device-type/segment/arg-type attributes. Leaving stale operands behind
+    // makes elementLoop look like it still has gang operands to later
+    // queries (e.g. LoopOp::getGangValue), which then dereference the
+    // now-missing device-type attribute.
+    elementLoop.getGangOperandsMutable().clear();
   }
   if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
     elementLoop.removeWorkerAttr();
     elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    // As above for gang, also drop the worker operand values so elementLoop
+    // does not keep a dangling worker operand with no device-type attribute.
+    elementLoop.getWorkerNumOperandsMutable().clear();
   }
   rewriter.finalizeOpModification(elementLoop);
 

diff  --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index cd13b3d8427f0..a1f80ad9defdf 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -74,6 +74,98 @@ func.func @nested_loop_tile(%arg0: memref<100x50xf32>) {
   return
 }
 
+// Regression test: a loop with GANG(STATIC: N) combined with a multi-dim
+// TILE clause used to crash with an assertion failure inside
+// LoopOp::getGangValue() because uncollapseLoops() (needed here since the
+// tile count exceeds the implicit collapse count of 1) left one of the
+// generated inner loops with a leftover gang operand but no corresponding
+// gang device-type attribute. Check that the pass completes and that gang
+// is only preserved on the outermost tile loop.
+
+// CHECK-LABEL: func.func @gang_static_with_multi_dim_tile
+// CHECK:         acc.loop gang({static=%{{.*}} : i32}) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       gang
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         gang
+// CHECK:             acc.loop control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           gang
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             gang
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cs = arith.constant 1 : i32
+  acc.loop gang({static=%cs : i32}) tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
+// Regression test: a loop with WORKER(N) combined with VECTOR and a
+// multi-dim TILE clause used to produce invalid IR because createInnerLoop()
+// (via uncollapseLoops()) and removeWorkerVectorFromLoop() removed the
+// worker attributes from generated inner/tile loops without also clearing
+// the worker operand value copied onto them. This left loops with a
+// non-empty worker operand list but no worker device-type attribute,
+// which the verifier rejects ('worker operands count must match worker
+// device_type count'). Check that the pass produces valid IR, with worker
+// only preserved on the outermost tile loop and vector only on the
+// outermost element loop.
+
+// CHECK-LABEL: func.func @worker_num_with_multi_dim_tile
+// CHECK:         acc.loop worker(%{{.*}} : i32) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:       worker
+// CHECK-NOT:       vector
+// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:         worker
+// CHECK-NOT:         vector
+// CHECK:             acc.loop vector control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:           worker
+// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
+// CHECK-NOT:             worker
+// CHECK-NOT:             vector
+// CHECK:                 acc.yield
+// CHECK:               }
+// CHECK:               acc.yield
+// CHECK:             }
+// CHECK:             acc.yield
+// CHECK:           }
+// CHECK:           acc.yield
+// CHECK:         }
+func.func @worker_num_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  %cw = arith.constant 4 : i32
+  acc.loop worker(%cw : i32) vector tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %sum = arith.addi %i, %j : index
+    %val = arith.index_castui %sum : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {independent = [#acc.device_type<none>]}
+  return
+}
+
 // Test unknown tile size (*) represented as -1
 // Should use default tile size (32)
 


        

From llvmlistbot at llvm.org  Thu Jul  9 07:35:41 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 09 Jul 2026 07:35:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix reduce-transpose sub-byte
	handling (PR #208446)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208446@github.com>
Message-ID: <6a4fb1bd.27b3b003.3b871.8633@mx.google.com>


https://github.com/lhutton1 approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/208446

From llvmlistbot at llvm.org  Thu Jul  9 07:35:45 2026
From: llvmlistbot at llvm.org (Susan Tan =?utf-8?b?44K5LeOCtuODs+OAgOOCv+ODsw==?=)
Date: Thu, 09 Jul 2026 07:35:45 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Remove dangling operands
 in OpenACCUtilsTiling (PR #208261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208261@github.com>
Message-ID: <6a4fb1c1.743400b0.2067c9.9095@mx.google.com>


https://github.com/SusanTan closed https://github.com/llvm/llvm-project/pull/208261

From llvmlistbot at llvm.org  Thu Jul  9 07:35:51 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:35:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fb1c7.af6ba3ed.36d9db.9cdd@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 1/9] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 2/9] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 3/9] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 4/9] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 5/9] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 6/9] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 2f506c83f24eda7f22f58019c7dee22751464f58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 7/9] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From 0752206c6433d322eba660ae2abd83491ecf067f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 8/9] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///

>From c6e524f3a855560ea31b282f0804da8906db6f6f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:35:29 -0700
Subject: [PATCH 9/9] Added create_named_contractions. Removing 2nd
 reduction_to_contract causes test to fail

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir        | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index d6a6bedc75a10..32984a04f6a99 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -252,9 +252,8 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
-    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
-    transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+      : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
     transform.apply_patterns to %loop_k {
@@ -270,7 +269,6 @@ module @transforms attributes { transform.with_named_sequence } {
     %func_pre = transform.structured.match ops{["func.func"]} in %module
       : (!transform.any_op) -> !transform.any_op
     transform.apply_patterns to %func_pre {
-      transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.transfer_permutation_patterns
       transform.apply_patterns.canonicalization
     } : !transform.any_op


From llvmlistbot at llvm.org  Thu Jul  9 07:37:16 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:37:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fb21c.449bfb44.172ed9.cac5@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
----------------
steplong wrote:

Removing this one causes the test to fail. Will look into this after resolving some of the other points

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 07:48:12 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 07:48:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fb4ac.f8b79514.1d52d0.31b4@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 01/10] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 02/10] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 03/10] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 04/10] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 05/10] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 06/10] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 2f506c83f24eda7f22f58019c7dee22751464f58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 07/10] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From 0752206c6433d322eba660ae2abd83491ecf067f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 08/10] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///

>From c6e524f3a855560ea31b282f0804da8906db6f6f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:35:29 -0700
Subject: [PATCH 09/10] Added create_named_contractions. Removing 2nd
 reduction_to_contract causes test to fail

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir        | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index d6a6bedc75a10..32984a04f6a99 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -252,9 +252,8 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
-    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
-    transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+      : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
     transform.apply_patterns to %loop_k {
@@ -270,7 +269,6 @@ module @transforms attributes { transform.with_named_sequence } {
     %func_pre = transform.structured.match ops{["func.func"]} in %module
       : (!transform.any_op) -> !transform.any_op
     transform.apply_patterns to %func_pre {
-      transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.transfer_permutation_patterns
       transform.apply_patterns.canonicalization
     } : !transform.any_op

>From 4f07f9de67b3d57d17dd64083b8f982453a60e1f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:47:50 -0700
Subject: [PATCH 10/10] Fix some comments

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir          | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 32984a04f6a99..06dae3ee89042 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -244,14 +244,14 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the M, N dim is scalable!)
+    // Tile parallel dims (note, the M, N dims are scalable!)
     %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
     %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
     transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
       : !transform.any_op
 


From llvmlistbot at llvm.org  Thu Jul  9 07:53:17 2026
From: llvmlistbot at llvm.org (Kazu Hirata)
Date: Thu, 09 Jul 2026 07:53:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4fb5dd.70920dee.290dca.1b36@mx.google.com>


https://github.com/kazutakahirata updated https://github.com/llvm/llvm-project/pull/208359

>From 448aa8f6c740903fb14c5c1df2e93bdd386e0594 Mon Sep 17 00:00:00 2001
From: Kazu Hirata <kazu at google.com>
Date: Wed, 8 Jul 2026 14:23:33 -0700
Subject: [PATCH 1/2] [mlir][IR] Avoid Clang 21 crash during SFINAE overload
 resolution

Clang 21.1.8 segfaults during SFINAE overload resolution for T::getChecked
when matching non-pointer arguments against an unconstrained MLIRContext*
parameter.

This patch avoids the crash by adding an explicit SFINAE constraint to
the second overload of StorageUserBase::getChecked, ensuring that it is
only considered when the second argument is convertible to MLIRContext*.

Assisted-by: Antigravity
---
 mlir/include/mlir/IR/StorageUniquerSupport.h | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/IR/StorageUniquerSupport.h b/mlir/include/mlir/IR/StorageUniquerSupport.h
index 8959dab047103..b8fd6051afa81 100644
--- a/mlir/include/mlir/IR/StorageUniquerSupport.h
+++ b/mlir/include/mlir/IR/StorageUniquerSupport.h
@@ -194,9 +194,16 @@ class StorageUserBase : public BaseT, public Traits<ConcreteT>... {
   /// Get or create a new ConcreteT instance within the ctx. If the arguments
   /// provided are invalid, errors are emitted using the provided `emitError`
   /// and a null object is returned.
-  template <typename... Args>
+  ///
+  /// Workaround: We use Arg1 && instead of MLIRContext * in the parameter list
+  /// to work around a bug in Clang 21.1.8 where Clang segfaults during SFINAE
+  /// overload resolution when attempting to match non-pointer arguments
+  /// against an unconstrained MLIRContext * parameter.
+  template <typename Arg1, typename... Args,
+            typename = std::enable_if_t<std::is_convertible<
+                llvm::remove_cvref_t<Arg1>, MLIRContext *>::value>>
   static ConcreteT getChecked(function_ref<InFlightDiagnostic()> emitErrorFn,
-                              MLIRContext *ctx, Args... args) {
+                              Arg1 &&ctx, Args... args) {
     // If the construction invariants fail then we return a null attribute.
     if (failed(ConcreteT::verifyInvariants(emitErrorFn, args...)))
       return ConcreteT();

>From 949f63fe41ea2594e56f0b35ab15cb6b9b161bed Mon Sep 17 00:00:00 2001
From: Kazu Hirata <kazu at google.com>
Date: Thu, 9 Jul 2026 07:52:57 -0700
Subject: [PATCH 2/2] Address comments.

---
 mlir/include/mlir/IR/StorageUniquerSupport.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/IR/StorageUniquerSupport.h b/mlir/include/mlir/IR/StorageUniquerSupport.h
index b8fd6051afa81..85189b5793fc7 100644
--- a/mlir/include/mlir/IR/StorageUniquerSupport.h
+++ b/mlir/include/mlir/IR/StorageUniquerSupport.h
@@ -200,8 +200,8 @@ class StorageUserBase : public BaseT, public Traits<ConcreteT>... {
   /// overload resolution when attempting to match non-pointer arguments
   /// against an unconstrained MLIRContext * parameter.
   template <typename Arg1, typename... Args,
-            typename = std::enable_if_t<std::is_convertible<
-                llvm::remove_cvref_t<Arg1>, MLIRContext *>::value>>
+            typename = std::enable_if_t<std::is_convertible_v<
+                llvm::remove_cvref_t<Arg1>, MLIRContext *>>>
   static ConcreteT getChecked(function_ref<InFlightDiagnostic()> emitErrorFn,
                               Arg1 &&ctx, Args... args) {
     // If the construction invariants fail then we return a null attribute.


From llvmlistbot at llvm.org  Thu Jul  9 07:53:46 2026
From: llvmlistbot at llvm.org (Kazu Hirata)
Date: Thu, 09 Jul 2026 07:53:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4fb5fa.7270d2fd.2d4879.348e@mx.google.com>


================
@@ -194,9 +194,16 @@ class StorageUserBase : public BaseT, public Traits<ConcreteT>... {
   /// Get or create a new ConcreteT instance within the ctx. If the arguments
   /// provided are invalid, errors are emitted using the provided `emitError`
   /// and a null object is returned.
-  template <typename... Args>
+  ///
+  /// Workaround: We use Arg1 && instead of MLIRContext * in the parameter list
+  /// to work around a bug in Clang 21.1.8 where Clang segfaults during SFINAE
+  /// overload resolution when attempting to match non-pointer arguments
+  /// against an unconstrained MLIRContext * parameter.
+  template <typename Arg1, typename... Args,
+            typename = std::enable_if_t<std::is_convertible<
----------------
kazutakahirata wrote:

Fixed in the latest iteration.  Thanks!

https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 07:57:45 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 09 Jul 2026 07:57:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a4fb6e9.faf16b37.2f0e1d.e47e@mx.google.com>


https://github.com/nujaa updated https://github.com/llvm/llvm-project/pull/208466

>From df521a8ba9b7907c65f42787e38a620c71a68406 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 21:41:33 +0800
Subject: [PATCH 1/2] [MLIR][Linalg] Fix LinalgSpecializeGenericOpsPass for
 scalar inputs

---
 .../DecomposeGenericByUnfoldingPermutation.cpp |  6 +++---
 .../Linalg/specialize-generic-ops-fail.mlir    | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 9015cbb096f88..450cfb2641cb5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -164,8 +164,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
   // out which operand can supply that runtime-value (tensor.dim).
   // Leaving it as a future TODO.
   if (llvm::any_of(op->getOpOperands(), [](OpOperand &oper) {
-        auto opType = cast<RankedTensorType>(oper.get().getType());
-        return ShapedType::isDynamicShape(opType.getShape());
+        auto opType = dyn_cast<RankedTensorType>(oper.get().getType());
+        return !opType || ShapedType::isDynamicShape(opType.getShape());
       }))
     return failure();
 
@@ -210,7 +210,7 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
       Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  inputRTType.getElementType());
+                                                  elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);
diff --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index 5d66837fca510..bb2c2edfa20f2 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -46,3 +46,21 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
   } -> tensor<8xi32>
   return %res : tensor<8xi32>
 }
+
+
+// -----
+
+// CHECK-LABEL: @scalar_input
+// CHECK: linalg.generic
+#map = affine_map<(d0) -> (d0)>
+#map1 = affine_map<(d0) -> ()>
+func.func @scalar_input(%arg0: tensor<128xi64>, %arg1: f32) -> tensor<128xf32> {
+  %0 = tensor.empty() : tensor<128xf32>
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xi64>, f32) outs(%0 : tensor<128xf32>) {
+  ^bb0(%in: i64, %in_1: f32, %out: f32):
+    %2 = arith.sitofp %in : i64 to f32
+    %3 = arith.divf %2, %in_1 : f32
+    linalg.yield %3 : f32
+  } -> tensor<128xf32>
+  return %1 : tensor<128xf32>
+}
\ No newline at end of file

>From d9d23dbb6aac024374ed5fb3cf44712e04eca049 Mon Sep 17 00:00:00 2001
From: Hugo Trachino <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 15:57:34 +0100
Subject: [PATCH 2/2] formatting

---
 .../Transforms/DecomposeGenericByUnfoldingPermutation.cpp    | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 450cfb2641cb5..7ee89c0f06f56 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -209,8 +209,9 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     // Does it require broadcast?
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
-      Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  elType);
+      Value emptyTensor =
+          tensor::EmptyOp::create(rewriter, loc, outputShape, elType);
+
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);


From llvmlistbot at llvm.org  Thu Jul  9 07:59:57 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Thu, 09 Jul 2026 07:59:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fb76d.cb4e7546.990e0.cfc6@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
----------------
MaxGraey wrote:

I mean father specialization of this in different namespaces due to CWG 727

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 08:00:07 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Thu, 09 Jul 2026 08:00:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a4fb777.fae8f0c5.1d16c3.1e57@mx.google.com>


https://github.com/MaxGraey edited https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul  9 08:05:54 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Thu, 09 Jul 2026 08:05:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4fb8d2.bfd8d272.22fb6a.00d6@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 08:11:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 08:11:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2e81bab - [NVVM][MLIR] Fixed valgrind leak in
	MMAOp (#208063)
Message-ID: <6a4fba05.e4991e13.1a79ed.769e@mx.google.com>


Author: Stefan Mada
Date: 2026-07-09T08:10:56-07:00
New Revision: 2e81babdf17d29d2b7e833137f92a598f76a8cbb

URL: https://github.com/llvm/llvm-project/commit/2e81babdf17d29d2b7e833137f92a598f76a8cbb
DIFF: https://github.com/llvm/llvm-project/commit/2e81babdf17d29d2b7e833137f92a598f76a8cbb.diff

LOG: [NVVM][MLIR] Fixed valgrind leak in MMAOp (#208063)

MmaOp::getIntrinsicID triggers an uninitialized-read warning under
Valgrind. `intOverflowBehavior` is optional; in the generated selector
its `has_value()` guarded access can be speculatively loaded before the
guard, reading the empty std::optional's uninitialized payload.

The read is benign: the payload is correctly stack-allocated but
uninitialized while empty, and the loaded value is discarded by the
guard, so this is not incorrect / undefined behavior.

This MR keeps the attribute as optional, but the lowering now resolves
it to a concrete value (wrapped when omitted) before calling
getIntrinsicID, so the selector always reads an initialized value.
`wrapped` matches the PTX spec and the existing lowering, where an
omitted attribute already selected the non-satfinite intrinsic;
intrinsic selection and printed IR are unchanged. Documentation updated
to correct the stale default.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 89950d95e647ff..6dd32615451a7d 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3384,9 +3384,9 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     `intOverflowBehavior` is only relevant when the `multiplicandType` attribute
     is one of `u8, s8, u4, s4`, this attribute describes how overflow is handled
     in the accumulator. When the attribute is `satfinite`, the accumulator values
-    are clamped in the int32 range on overflow. This is the default behavior.
-    Alternatively, accumulator behavior `wrapped` can also be specified, in
-    which case overflow wraps from one end of the range to the other.
+    are clamped in the int32 range on overflow. Alternatively, accumulator
+    behavior `wrapped` can be specified (this is the default), in which case
+    overflow wraps from one end of the range to the other.
 
     `layoutA` and `layoutB` are required and should generally be set to
     `#nvvm.mma_layout<row>` and `#nvvm.mma_layout<col>` respectively, but other
@@ -3477,9 +3477,16 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
 
   string llvmBuilder = [{
     auto operands = moduleTranslation.lookupValues(opInst.getOperands());
+    // `intOverflowBehavior` is optional and defaults to `wrapped` when omitted.
+    // We initialize it to a concrete value so that the intrinsic selector never
+    // speculatively reads an empty optional's uninitialized payload (benign,
+    // but flagged by Valgrind).
+    std::optional<mlir::NVVM::MMAIntOverflow> intOverflow = mlir::NVVM::MMAIntOverflow::wrapped;
+    if (mlir::NVVM::MMAIntOverflowAttr satAttr = op.getIntOverflowBehaviorAttr())
+      intOverflow = satAttr.getValue();
     auto intId = mlir::NVVM::MmaOp::getIntrinsicID(
         $shape.getM(), $shape.getN(), $shape.getK(),
-        $b1Op, $intOverflowBehavior,
+        $b1Op, intOverflow,
         $layoutA, $layoutB,
         *$multiplicandAPtxType,
         *$multiplicandBPtxType,


        

From llvmlistbot at llvm.org  Thu Jul  9 08:11:05 2026
From: llvmlistbot at llvm.org (Stefan Mada)
Date: Thu, 09 Jul 2026 08:11:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM][MLIR] Fixed valgrind leak in MMAOp (PR
	#208063)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208063@github.com>
Message-ID: <6a4fba09.39f23b82.27a08.a1bf@mx.google.com>


https://github.com/smada3 closed https://github.com/llvm/llvm-project/pull/208063

From llvmlistbot at llvm.org  Thu Jul  9 08:14:49 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Thu, 09 Jul 2026 08:14:49 -0700 (PDT)
Subject: [Mlir-commits]  [mlir] [MLIR][XeGPU] Disable test execution,
	allow compilation (PR #208482)
Message-ID: <llvm.org/llvm/llvm-project/pull/208482@github.com>


https://github.com/akroviakov created https://github.com/llvm/llvm-project/pull/208482

This PR follows the https://github.com/llvm/llvm-project/pull/207158 for a different test. 
So far, we want to verify at least the test compilation, not execution.

>From 04d4a4a7eb7ac5b71f6b571fc121a91c1eab6f6c Mon Sep 17 00:00:00 2001
From: Artem Kroviakov <artem.kroviakov at intel.com>
Date: Thu, 9 Jul 2026 15:11:59 +0000
Subject: [PATCH] [MLIR][XeGPU] Disable test execution, allow compilation

---
 .../Dialect/XeGPU/WG/simple_mxfp_gemm.mlir      | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..3393b723d6721 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -1,14 +1,13 @@
 // RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri" \
-// RUN: | mlir-runner \
-// RUN:   --shared-libs=%mlir_levelzero_runtime \
-// RUN:   --shared-libs=%mlir_runner_utils \
-// RUN:   --shared-libs=%mlir_c_runner_utils \
-// RUN:   --entry-point-result=void \
-// RUN: | FileCheck %s
-
-// XFAIL: *
+// RUN-DISABLED: | mlir-runner \
+// RUN-DISABLED:   --shared-libs=%mlir_levelzero_runtime \
+// RUN-DISABLED:   --shared-libs=%mlir_runner_utils \
+// RUN-DISABLED:   --shared-libs=%mlir_c_runner_utils \
+// RUN-DISABLED:   --entry-point-result=void \
+// RUN-DISABLED: | FileCheck %s
+
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>


From llvmlistbot at llvm.org  Thu Jul  9 08:26:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 08:26:54 -0700 (PDT)
Subject: [Mlir-commits]  [mlir] [MLIR][XeGPU] Disable test execution,
	allow compilation (PR #208482)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208482@github.com>
Message-ID: <6a4fbdbe.492e23e4.157d47.2078@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-gpu

Author: Artem Kroviakov (akroviakov)

<details>
<summary>Changes</summary>

This PR follows the https://github.com/llvm/llvm-project/pull/207158 for a different test. 
So far, we want to verify at least the test compilation, not execution.

---
Full diff: https://github.com/llvm/llvm-project/pull/208482.diff


1 Files Affected:

- (modified) mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir (+8-9) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..3393b723d6721 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -1,14 +1,13 @@
 // RUN: mlir-opt %s --gpu-lower-to-xevm-pipeline="xegpu-op-level=workgroup zebin-chip=cri" \
-// RUN: | mlir-runner \
-// RUN:   --shared-libs=%mlir_levelzero_runtime \
-// RUN:   --shared-libs=%mlir_runner_utils \
-// RUN:   --shared-libs=%mlir_c_runner_utils \
-// RUN:   --entry-point-result=void \
-// RUN: | FileCheck %s
-
-// XFAIL: *
+// RUN-DISABLED: | mlir-runner \
+// RUN-DISABLED:   --shared-libs=%mlir_levelzero_runtime \
+// RUN-DISABLED:   --shared-libs=%mlir_runner_utils \
+// RUN-DISABLED:   --shared-libs=%mlir_c_runner_utils \
+// RUN-DISABLED:   --entry-point-result=void \
+// RUN-DISABLED: | FileCheck %s
+
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>

``````````

</details>


https://github.com/llvm/llvm-project/pull/208482

From llvmlistbot at llvm.org  Thu Jul  9 08:26:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 08:26:56 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Lower DECLARE TARGET
 INDIRECT clause (PR #208387)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208387@github.com>
Message-ID: <6a4fbdc0.ff037943.2e959a.718a@mx.google.com>


https://github.com/blazie2004 ready_for_review https://github.com/llvm/llvm-project/pull/208387

From llvmlistbot at llvm.org  Thu Jul  9 08:27:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 08:27:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9722a2c - [mlir][IR] Avoid Clang 21 crash
 during SFINAE overload resolution (#208359)
Message-ID: <6a4fbdc4.07d7ca98.9dee3.c5e3@mx.google.com>


Author: Kazu Hirata
Date: 2026-07-09T08:26:56-07:00
New Revision: 9722a2ceddb6e4fc098f89c9c55c5ad7afd1a64c

URL: https://github.com/llvm/llvm-project/commit/9722a2ceddb6e4fc098f89c9c55c5ad7afd1a64c
DIFF: https://github.com/llvm/llvm-project/commit/9722a2ceddb6e4fc098f89c9c55c5ad7afd1a64c.diff

LOG: [mlir][IR] Avoid Clang 21 crash during SFINAE overload resolution (#208359)

Clang 21.1.8 segfaults during SFINAE overload resolution for
T::getChecked
when matching non-pointer arguments against an unconstrained
MLIRContext*
parameter.

This patch avoids the crash by adding an explicit SFINAE constraint to
the second overload of StorageUserBase::getChecked, ensuring that it is
only considered when the second argument is convertible to MLIRContext*.

Assisted-by: Antigravity

Added: 
    

Modified: 
    mlir/include/mlir/IR/StorageUniquerSupport.h

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/IR/StorageUniquerSupport.h b/mlir/include/mlir/IR/StorageUniquerSupport.h
index 8959dab047103..85189b5793fc7 100644
--- a/mlir/include/mlir/IR/StorageUniquerSupport.h
+++ b/mlir/include/mlir/IR/StorageUniquerSupport.h
@@ -194,9 +194,16 @@ class StorageUserBase : public BaseT, public Traits<ConcreteT>... {
   /// Get or create a new ConcreteT instance within the ctx. If the arguments
   /// provided are invalid, errors are emitted using the provided `emitError`
   /// and a null object is returned.
-  template <typename... Args>
+  ///
+  /// Workaround: We use Arg1 && instead of MLIRContext * in the parameter list
+  /// to work around a bug in Clang 21.1.8 where Clang segfaults during SFINAE
+  /// overload resolution when attempting to match non-pointer arguments
+  /// against an unconstrained MLIRContext * parameter.
+  template <typename Arg1, typename... Args,
+            typename = std::enable_if_t<std::is_convertible_v<
+                llvm::remove_cvref_t<Arg1>, MLIRContext *>>>
   static ConcreteT getChecked(function_ref<InFlightDiagnostic()> emitErrorFn,
-                              MLIRContext *ctx, Args... args) {
+                              Arg1 &&ctx, Args... args) {
     // If the construction invariants fail then we return a null attribute.
     if (failed(ConcreteT::verifyInvariants(emitErrorFn, args...)))
       return ConcreteT();


        

From llvmlistbot at llvm.org  Thu Jul  9 08:27:03 2026
From: llvmlistbot at llvm.org (Kazu Hirata)
Date: Thu, 09 Jul 2026 08:27:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][IR] Avoid Clang 21 crash during SFINAE
 overload resolution (PR #208359)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208359@github.com>
Message-ID: <6a4fbdc7.e55dc341.1f19bd.ef04@mx.google.com>


https://github.com/kazutakahirata closed https://github.com/llvm/llvm-project/pull/208359

From llvmlistbot at llvm.org  Thu Jul  9 08:27:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 08:27:14 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][OpenMP] Lower DECLARE TARGET
 INDIRECT clause (PR #208387)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208387@github.com>
Message-ID: <6a4fbdd2.8e75e0cb.5e108.47e7@mx.google.com>


https://github.com/blazie2004 updated https://github.com/llvm/llvm-project/pull/208387

>From efc99a67c223770ac3503fbe52738e073630a363 Mon Sep 17 00:00:00 2001
From: Jay Satish Kumar Patel <kumarpat at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 8 Jul 2026 04:53:19 -0500
Subject: [PATCH] [Flang][OpenMP] Lower DECLARE TARGET INDIRECT clause

---
 flang/include/flang/Lower/OpenMP.h            |  1 +
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    | 21 +++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 40 ++++++---
 flang/lib/Lower/OpenMP/Utils.h                |  1 +
 .../Optimizer/OpenMP/FunctionFiltering.cpp    |  3 +-
 flang/lib/Semantics/check-omp-structure.cpp   | 15 ++++
 .../Lower/OpenMP/Todo/omp-clause-indirect.f90 | 34 --------
 flang/test/Lower/OpenMP/common-block-map.f90  |  2 +-
 .../test/Lower/OpenMP/declare-target-data.f90 | 40 ++++-----
 ...are-target-deferred-marking-reductions.f90 |  4 +-
 .../declare-target-deferred-marking.f90       |  8 +-
 .../OpenMP/declare-target-func-and-subr.f90   | 46 +++++-----
 ...arget-implicit-func-and-subr-cap-enter.f90 | 38 ++++-----
 ...lare-target-implicit-func-and-subr-cap.f90 | 42 +++++-----
 .../declare-target-implicit-tarop-cap.f90     | 14 ++--
 .../OpenMP/declare-target-indirect-clause.f90 | 30 +++++++
 .../OpenMP/declare-target-indirect-merge.f90  | 47 +++++++++++
 .../Lower/OpenMP/declare-target-indirect.f90  | 39 +++++++++
 .../OpenMP/declare-target-unnamed-main.f90    |  2 +-
 .../Lower/OpenMP/function-filtering-2.f90     |  8 +-
 .../OpenMP/omp-declare-target-program-var.f90 |  2 +-
 flang/test/Semantics/indirect02.f90           |  7 +-
 flang/test/Semantics/indirect03.f90           | 31 +++++++
 .../mlir/Dialect/OpenMP/OpenMPAttrDefs.td     |  3 +-
 .../Dialect/OpenMP/OpenMPClauseOperands.h     | 10 ++-
 .../Dialect/OpenMP/OpenMPOpsInterfaces.td     | 20 ++++-
 .../OpenMP/Transforms/MarkDeclareTarget.cpp   |  9 +-
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 83 ++++++++++++++++++-
 .../OpenMP/mark-declare-target-indirect.mlir  | 39 +++++++++
 ...target-declare-target-indirect-device.mlir | 19 +++++
 ...mptarget-declare-target-indirect-host.mlir | 20 +++++
 32 files changed, 517 insertions(+), 162 deletions(-)
 delete mode 100644 flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
 create mode 100644 flang/test/Lower/OpenMP/declare-target-indirect.f90
 create mode 100644 flang/test/Semantics/indirect03.f90
 create mode 100644 mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir

diff --git a/flang/include/flang/Lower/OpenMP.h b/flang/include/flang/Lower/OpenMP.h
index c70393c78bbec..fec8661e7c64f 100644
--- a/flang/include/flang/Lower/OpenMP.h
+++ b/flang/include/flang/Lower/OpenMP.h
@@ -57,6 +57,7 @@ struct OMPDeferredDeclareTargetInfo {
   mlir::omp::DeclareTargetCaptureClause declareTargetCaptureClause;
   mlir::omp::DeclareTargetDeviceType declareTargetDeviceType;
   bool automap = false;
+  bool indirect = false;
   const Fortran::semantics::Symbol &sym;
 };
 
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index 05e7edc8bdc6a..7297e974e356b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -13,6 +13,8 @@
 #include "ClauseProcessor.h"
 #include "Utils.h"
 
+#include "flang/Evaluate/fold.h"
+#include "flang/Evaluate/tools.h"
 #include "flang/Lower/ConvertCall.h"
 #include "flang/Lower/ConvertExprToHLFIR.h"
 #include "flang/Lower/OpenMP/Clauses.h"
@@ -457,6 +459,25 @@ bool ClauseProcessor::processDeviceType(
   return false;
 }
 
+bool ClauseProcessor::processIndirect(
+    mlir::omp::IndirectClauseOps &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Indirect>()) {
+    // Case: declare target ... indirect[(scalar-logical-constant)]
+    // An `indirect` clause with no argument defaults to `.true.`.
+    bool isIndirect = true;
+    if (clause->v) {
+      auto foldedExpr = Fortran::evaluate::Fold(
+          semaCtx.foldingContext(), Fortran::common::Clone(*clause->v));
+      if (auto logicalVal = Fortran::evaluate::GetScalarConstantValue<
+              Fortran::evaluate::LogicalResult>(foldedExpr))
+        isIndirect = logicalVal->IsTrue();
+    }
+    result.indirect = isIndirect;
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processDistSchedule(
     lower::StatementContext &stmtCtx,
     mlir::omp::DistScheduleClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1435af3e844e9..b00236c81b40d 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -142,6 +142,7 @@ class ClauseProcessor {
   processEnter(llvm::SmallVectorImpl<DeclareTargetCaptureInfo> &result) const;
   bool processIf(omp::clause::If::DirectiveNameModifier directiveName,
                  mlir::omp::IfClauseOps &result) const;
+  bool processIndirect(mlir::omp::IndirectClauseOps &result) const;
   bool
   processInReduction(mlir::Location currentLocation,
                      mlir::omp::InReductionClauseOps &result,
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 7939e0db655e4..4554f8c5db88f 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1409,9 +1409,13 @@ static void getDeclareTargetInfo(
     cp.processEnter(symbolAndClause);
     cp.processLink(symbolAndClause);
     cp.processTo(symbolAndClause);
+    cp.processIndirect(clauseOps);
 
-    cp.processTODO<clause::Indirect>(converter.getCurrentLocation(),
-                                     llvm::omp::Directive::OMPD_declare_target);
+    // The `indirect` clause applies to the functions named by the directive
+    // (it requires an `enter` clause). Propagate the directive-level value to
+    // each captured symbol so it reaches the declare target attribute.
+    for (DeclareTargetCaptureInfo &sym : symbolAndClause)
+      sym.indirect = clauseOps.indirect;
   }
 }
 
@@ -1435,7 +1439,8 @@ static void collectDeferredDeclareTargets(
 
     if (!op) {
       deferredDeclareTarget.push_back({symClause.clause, clauseOps.deviceType,
-                                       symClause.automap, symClause.symbol});
+                                       symClause.automap, symClause.indirect,
+                                       symClause.symbol});
     }
   }
 }
@@ -1714,7 +1719,8 @@ getImplicitMapTypeAndKind(fir::FirOpBuilder &firOpBuilder,
 static void
 markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
                   mlir::omp::DeclareTargetCaptureClause captureClause,
-                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap) {
+                  mlir::omp::DeclareTargetDeviceType deviceType, bool automap,
+                  bool indirect) {
   // TODO: Add support for program local variables with declare target applied
   auto declareTargetOp = llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(op);
   if (!declareTargetOp)
@@ -1724,16 +1730,30 @@ markDeclareTarget(mlir::Operation *op, lower::AbstractConverter &converter,
 
   // The function or global already has a declare target applied to it, very
   // likely through implicit capture (usage in another declare target
-  // function/subroutine). It should be marked as any if it has been assigned
-  // both host and nohost, else we skip, as there is no change
+  // function/subroutine), or because it is named by more than one declare
+  // target directive. It should be marked as any if it has been assigned both
+  // host and nohost. The `indirect` modifier is a capability: once any
+  // declaration requests it, it must stay set, so it is merged with logical OR
+  // rather than overwritten (which could drop a previous `indirect = true`).
   if (declareTargetOp.isDeclareTarget()) {
+    bool mergedIndirect =
+        declareTargetOp.getDeclareTargetIndirect() || indirect;
+
     if (declareTargetOp.getDeclareTargetDeviceType() != deviceType)
       declareTargetOp.setDeclareTarget(mlir::omp::DeclareTargetDeviceType::any,
-                                       captureClause, automap);
+                                       captureClause, automap, mergedIndirect);
+    else if (mergedIndirect != declareTargetOp.getDeclareTargetIndirect())
+      // Same device type, but a later declaration added `indirect`; update it
+      // while preserving the already-established capture clause and automap.
+      declareTargetOp.setDeclareTarget(
+          declareTargetOp.getDeclareTargetDeviceType(),
+          declareTargetOp.getDeclareTargetCaptureClause(),
+          declareTargetOp.getDeclareTargetAutomap(), mergedIndirect);
     return;
   }
 
-  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap);
+  declareTargetOp.setDeclareTarget(deviceType, captureClause, automap,
+                                   indirect);
 }
 
 //===----------------------------------------------------------------------===//
@@ -5498,7 +5518,7 @@ genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
       continue;
 
     markDeclareTarget(op, converter, symClause.clause, clauseOps.deviceType,
-                      symClause.automap);
+                      symClause.automap, symClause.indirect);
   }
 }
 
@@ -6465,7 +6485,7 @@ bool Fortran::lower::markOpenMPDeferredDeclareTargetFunctions(
       deviceCodeFound = true;
 
     markDeclareTarget(op, converter, declTar.declareTargetCaptureClause,
-                      devType, declTar.automap);
+                      devType, declTar.automap, declTar.indirect);
   }
 
   return deviceCodeFound;
diff --git a/flang/lib/Lower/OpenMP/Utils.h b/flang/lib/Lower/OpenMP/Utils.h
index 94f85c43f7033..db38c2b755be7 100644
--- a/flang/lib/Lower/OpenMP/Utils.h
+++ b/flang/lib/Lower/OpenMP/Utils.h
@@ -53,6 +53,7 @@ namespace omp {
 struct DeclareTargetCaptureInfo {
   mlir::omp::DeclareTargetCaptureClause clause;
   bool automap = false;
+  bool indirect = false;
   const semantics::Symbol &symbol;
 
   DeclareTargetCaptureInfo(mlir::omp::DeclareTargetCaptureClause c,
diff --git a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
index e8bcc0c7fa062..22dc3ec8fbf69 100644
--- a/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
+++ b/flang/lib/Optimizer/OpenMP/FunctionFiltering.cpp
@@ -136,7 +136,8 @@ class FunctionFilteringPass
         if (declareTargetOp)
           declareTargetOp.setDeclareTarget(
               declareType, omp::DeclareTargetCaptureClause::to,
-              declareTargetOp.getDeclareTargetAutomap());
+              declareTargetOp.getDeclareTargetAutomap(),
+              declareTargetOp.getDeclareTargetIndirect());
       }
       return WalkResult::advance();
     });
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 02928401a382a..d8ee8f4574229 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2557,6 +2557,21 @@ void OmpStructureChecker::Leave(const parser::OmpDeclareTargetDirective &x) {
       context_.Warn(common::UsageWarning::OpenMPUsage, toClause->source,
           "The usage of TO clause on DECLARE TARGET directive has been deprecated. Use ENTER clause instead."_warn_en_US);
     }
+    if (indirectClause) {
+      // The INDIRECT clause is only allowed together with DEVICE_TYPE(ANY) (an
+      // absent DEVICE_TYPE clause also implies ANY). A host- or device-only
+      // procedure cannot be the target of an indirect device invocation.
+      if (const parser::OmpClause *deviceTypeClause{
+              FindClause(llvm::omp::Clause::OMPC_device_type)}) {
+        const auto &deviceType{
+            std::get<parser::OmpClause::DeviceType>(deviceTypeClause->u)};
+        if (deviceType.v.v !=
+            parser::OmpDeviceTypeClause::DeviceTypeDescription::Any) {
+          context_.Say(x.source,
+              "Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive"_err_en_US);
+        }
+      }
+    }
   }
 
   bool toClauseFound{false};
diff --git a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90 b/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
deleted file mode 100644
index 82efa8818a83c..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/omp-clause-indirect.f90
+++ /dev/null
@@ -1,34 +0,0 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
-
-! RUN: not %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
-
-module functions
-  implicit none
-
-  interface
-    function func() result(i)
-      character(1) :: i
-    end function
-  end interface
-
-contains
-  function func1() result(i)
-    !CHECK: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
-    !$omp declare target enter(func1) indirect(.true.)
-    character(1) :: i
-    i = 'a'
-    return
-  end function
-end module
-
-program main
-  use functions
-  implicit none
-  procedure (func), pointer :: ptr1=>func1
-  character(1) :: val1
-
-  !$omp target map(from: val1)
-  val1 = ptr1()
-  !$omp end target
-
-end program
diff --git a/flang/test/Lower/OpenMP/common-block-map.f90 b/flang/test/Lower/OpenMP/common-block-map.f90
index 98836306ef041..3daf19e97e6fd 100644
--- a/flang/test/Lower/OpenMP/common-block-map.f90
+++ b/flang/test/Lower/OpenMP/common-block-map.f90
@@ -1,7 +1,7 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s
 
 !CHECK: fir.global common @var_common_(dense<0> : vector<8xi8>) {{.*}} : !fir.array<8xi8>
-!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<8xi8>
+!CHECK: fir.global common @var_common_link_(dense<0> : vector<8xi8>) {{{.*}} omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<8xi8>
 
 !CHECK-LABEL: func.func @_QPmap_full_block
 !CHECK: %[[CB_ADDR:.*]] = fir.address_of(@var_common_) : !fir.ref<!fir.array<8xi8>>
diff --git a/flang/test/Lower/OpenMP/declare-target-data.f90 b/flang/test/Lower/OpenMP/declare-target-data.f90
index 2e3790303984c..ffffb76eee37f 100644
--- a/flang/test/Lower/OpenMP/declare-target-data.f90
+++ b/flang/test/Lower/OpenMP/declare-target-data.f90
@@ -4,61 +4,61 @@
 module test_0
     implicit none
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : i32
 INTEGER :: data_int = 10
 !$omp declare target link(data_int)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<3xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_1d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<3xi32>
 INTEGER :: array_1d(3) = (/1,2,3/)
 !$omp declare target link(array_1d)
 
-!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.array<2x2xi32>
+!CHECK-DAG: fir.global @_QMtest_0Earray_2d({{.*}}) {alignment = 64 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.array<2x2xi32>
 INTEGER :: array_2d(2,2) = reshape((/1,2,3,4/), (/2,2/))
 !$omp declare target link(array_2d)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt1
 !$omp declare target link(pt1)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} target : i32
+!CHECK-DAG: fir.global @_QMtest_0Ept2_tar {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} target : i32
 INTEGER, TARGET :: pt2_tar = 5
 !$omp declare target link(pt2_tar)
 
-!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : !fir.box<!fir.ptr<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Ept2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : !fir.box<!fir.ptr<i32>>
 INTEGER, POINTER :: pt2 => pt2_tar
 !$omp declare target link(pt2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_to {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_to = 5
 !$omp declare target to(data_int_to)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_enter {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : i32
 INTEGER :: data_int_enter = 5
 !$omp declare target enter(data_int_enter)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : i32
+!CHECK-DAG: fir.global @_QMtest_0Edata_int_clauseless {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : i32
 INTEGER :: data_int_clauseless = 1
 !$omp declare target(data_int_clauseless)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_to_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32
 REAL :: data_extended_to_1 = 2
 REAL :: data_extended_to_2 = 3
 !$omp declare target to(data_extended_to_1, data_extended_to_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_enter_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : f32
 REAL :: data_extended_enter_1 = 2
 REAL :: data_extended_enter_2 = 3
 !$omp declare target enter(data_extended_enter_1, data_extended_enter_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
-!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_1 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
+!CHECK-DAG: fir.global @_QMtest_0Edata_extended_link_2 {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : f32
 REAL :: data_extended_link_1 = 2
 REAL :: data_extended_link_2 = 3
 !$omp declare target link(data_extended_link_1, data_extended_link_2)
 
-!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true>} target : !fir.box<!fir.heap<i32>>
+!CHECK-DAG: fir.global @_QMtest_0Eautomap_data {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = true, indirect = false>} target : !fir.box<!fir.heap<i32>>
 INTEGER, ALLOCATABLE, TARGET :: automap_data
 !$omp declare target enter(automap : automap_data)
 
@@ -66,25 +66,25 @@ module test_0
 end module test_0
 
 PROGRAM commons
-    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one = 1
     REAL :: two = 2
     COMMON /numbers/ one, two
     !$omp declare target(/numbers/)
 
-    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_link_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (link), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_link = 1
     REAL :: two_link = 2
     COMMON /numbers_link/ one_link, two_link
     !$omp declare target link(/numbers_link/)
 
-    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_to_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_to = 1
     REAL :: two_to = 2
     COMMON /numbers_to/ one_to, two_to
     !$omp declare target to(/numbers_to/)
 
-    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>} : tuple<f32, f32> {
+    !CHECK-DAG: fir.global @numbers_enter_ {alignment = 4 : i64, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>} : tuple<f32, f32> {
     REAL :: one_enter = 1
     REAL :: two_enter = 2
     COMMON /numbers_enter/ one_enter, two_enter
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
index 66697ef6bbe70..2bbdd2d071129 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking-reductions.f90
@@ -31,7 +31,7 @@ end function mycombine
  end program main
 
 !CHECK: func.func {{.*}} @myinit(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !CHECK-LABEL: func.func {{.*}} @mycombine(!fir.ref<i32>, !fir.ref<i32>)
-!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!CHECK-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 
diff --git a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90 b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
index 528563aba6962..974e0f8a344ab 100644
--- a/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
+++ b/flang/test/Lower/OpenMP/declare-target-deferred-marking.f90
@@ -51,10 +51,10 @@ end subroutine unused_unemitted_interface
  end program main
 
 !HOST-LABEL: func.func {{.*}} @host_interface()
-!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+!HOST-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @called_from_target_interface(!fir.ref<i64>, !fir.ref<i64>)
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @any_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 !ALL-LABEL: func.func {{.*}} @device_interface()
-!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+!ALL-SAME: {{.*}}, omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
diff --git a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90 b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
index 46465eece5050..fa181521aa293 100644
--- a/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
+++ b/flang/test/Lower/OpenMP/declare-target-func-and-subr.f90
@@ -6,7 +6,7 @@
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPfunc_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_DEVICE() RESULT(I)
 !$omp declare target to(FUNC_T_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -14,7 +14,7 @@ FUNCTION FUNC_T_DEVICE() RESULT(I)
 END FUNCTION FUNC_T_DEVICE
 
 ! DEVICE-LABEL: func.func @_QPfunc_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_DEVICE) device_type(nohost)
     INTEGER :: I
@@ -22,7 +22,7 @@ FUNCTION FUNC_ENTER_DEVICE() RESULT(I)
 END FUNCTION FUNC_ENTER_DEVICE
 
 ! HOST-LABEL: func.func @_QPfunc_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_HOST() RESULT(I)
 !$omp declare target to(FUNC_T_HOST) device_type(host)
     INTEGER :: I
@@ -30,7 +30,7 @@ FUNCTION FUNC_T_HOST() RESULT(I)
 END FUNCTION FUNC_T_HOST
 
 ! HOST-LABEL: func.func @_QPfunc_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_HOST() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_HOST) device_type(host)
     INTEGER :: I
@@ -38,7 +38,7 @@ FUNCTION FUNC_ENTER_HOST() RESULT(I)
 END FUNCTION FUNC_ENTER_HOST
 
 ! ALL-LABEL: func.func @_QPfunc_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_T_ANY) device_type(any)
     INTEGER :: I
@@ -46,7 +46,7 @@ FUNCTION FUNC_T_ANY() RESULT(I)
 END FUNCTION FUNC_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_ENTER_ANY) device_type(any)
     INTEGER :: I
@@ -54,7 +54,7 @@ FUNCTION FUNC_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 !$omp declare target to(FUNC_DEFAULT_T_ANY)
     INTEGER :: I
@@ -62,7 +62,7 @@ FUNCTION FUNC_DEFAULT_T_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_T_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 !$omp declare target enter(FUNC_DEFAULT_ENTER_ANY)
     INTEGER :: I
@@ -70,7 +70,7 @@ FUNCTION FUNC_DEFAULT_ENTER_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ENTER_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 !$omp declare target
     INTEGER :: I
@@ -78,7 +78,7 @@ FUNCTION FUNC_DEFAULT_ANY() RESULT(I)
 END FUNCTION FUNC_DEFAULT_ANY
 
 ! ALL-LABEL: func.func @_QPfunc_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 !$omp declare target(FUNC_DEFAULT_EXTENDEDLIST)
     INTEGER :: I
@@ -86,7 +86,7 @@ FUNCTION FUNC_DEFAULT_EXTENDEDLIST() RESULT(I)
 END FUNCTION FUNC_DEFAULT_EXTENDEDLIST
 
 ! ALL-LABEL: func.func @_QPfunc_name_as_result()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 FUNCTION FUNC_NAME_AS_RESULT()
 !$omp declare target(FUNC_NAME_AS_RESULT)
   FUNC_NAME_AS_RESULT = 1.0
@@ -99,61 +99,61 @@ END FUNCTION FUNC_NAME_AS_RESULT
 ! zero clause declare target
 
 ! DEVICE-LABEL: func.func @_QPsubr_t_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_DEVICE()
 !$omp declare target to(SUBR_T_DEVICE) device_type(nohost)
 END
 
 ! DEVICE-LABEL: func.func @_QPsubr_enter_device()
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_DEVICE()
 !$omp declare target enter(SUBR_ENTER_DEVICE) device_type(nohost)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_t_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_HOST()
 !$omp declare target to(SUBR_T_HOST) device_type(host)
 END
 
 ! HOST-LABEL: func.func @_QPsubr_enter_host()
-! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}
+! HOST-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_HOST()
 !$omp declare target enter(SUBR_ENTER_HOST) device_type(host)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_T_ANY()
 !$omp declare target to(SUBR_T_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_ENTER_ANY()
 !$omp declare target enter(SUBR_ENTER_ANY) device_type(any)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_t_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_T_ANY()
 !$omp declare target to(SUBR_DEFAULT_T_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_enter_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ENTER_ANY()
 !$omp declare target enter(SUBR_DEFAULT_ENTER_ANY)
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_any()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_ANY()
 !$omp declare target
 END
 
 ! ALL-LABEL: func.func @_QPsubr_default_extendedlist()
-! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}
+! ALL-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}
 SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !$omp declare target(SUBR_DEFAULT_EXTENDEDLIST)
 END
@@ -161,7 +161,7 @@ SUBROUTINE SUBR_DEFAULT_EXTENDEDLIST()
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 !$omp declare target to(RECURSIVE_DECLARE_TARGET) device_type(nohost)
     INTEGER :: INCREMENT, K
@@ -173,7 +173,7 @@ RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET(INCREMENT) RESULT(K)
 END FUNCTION RECURSIVE_DECLARE_TARGET
 
 ! DEVICE-LABEL: func.func @_QPrecursive_declare_target_enter
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}
 RECURSIVE FUNCTION RECURSIVE_DECLARE_TARGET_ENTER(INCREMENT) RESULT(K)
 !$omp declare target enter(RECURSIVE_DECLARE_TARGET_ENTER) device_type(nohost)
     INTEGER :: INCREMENT, K
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
index 2caa6ba0b5b1e..b914b8b9814b9 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap-enter.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -12,7 +12,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target enter(target_function_twice_host) device_type(host)
    integer :: i
@@ -20,7 +20,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target enter(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -30,7 +30,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -44,7 +44,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -52,7 +52,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target enter(target_function_test) device_type(nohost)
    integer :: i, j
@@ -63,7 +63,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -71,13 +71,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -85,7 +85,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target enter(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -94,7 +94,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target enter(target_function_test_host) device_type(host)
    integer :: i, j
@@ -105,7 +105,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target enter(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -117,7 +117,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target enter(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -129,28 +129,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target enter(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target enter(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -174,7 +174,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -185,7 +185,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target enter(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
index 32a5058029174..b4ccce4e4b114 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-func-and-subr-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured(toggle) result(k)
    integer :: i, j, k
    logical :: toggle
@@ -19,7 +19,7 @@ end function implicitly_captured
 
 
 ! CHECK-LABEL: func.func @_QPtarget_function
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function(toggle) result(i)
 !$omp declare target
    integer :: i
@@ -30,7 +30,7 @@ end function target_function
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_twice() result(k)
    integer :: i
    i = 10
@@ -38,7 +38,7 @@ function implicitly_captured_twice() result(k)
 end function implicitly_captured_twice
 
 ! CHECK-LABEL: func.func @_QPtarget_function_twice_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_host() result(i)
 !$omp declare target to(target_function_twice_host) device_type(host)
    integer :: i
@@ -46,7 +46,7 @@ function target_function_twice_host() result(i)
 end function target_function_twice_host
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_twice_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_twice_device() result(i)
 !$omp declare target to(target_function_twice_device) device_type(nohost)
    integer :: i
@@ -56,7 +56,7 @@ end function target_function_twice_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_nest
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest() result(k)
    integer :: i
    i = 10
@@ -70,7 +70,7 @@ function implicitly_captured_one() result(k)
 end function implicitly_captured_one
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two() result(k)
    integer :: i
    i = 10
@@ -78,7 +78,7 @@ function implicitly_captured_two() result(k)
 end function implicitly_captured_two
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test() result(j)
 !$omp declare target to(target_function_test) device_type(nohost)
    integer :: i, j
@@ -89,7 +89,7 @@ end function target_function_test
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(k)
    integer :: i
    i = 10
@@ -97,13 +97,13 @@ function implicitly_captured_nest_twice() result(k)
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_two_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(k)
    integer :: i
    i = 10
@@ -111,7 +111,7 @@ function implicitly_captured_two_twice() result(k)
 end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_device() result(j)
    !$omp declare target to(target_function_test_device) device_type(nohost)
    integer :: i, j
@@ -120,7 +120,7 @@ function target_function_test_device() result(j)
 end function target_function_test_device
 
 ! CHECK-LABEL: func.func @_QPtarget_function_test_host
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_test_host() result(j)
    !$omp declare target to(target_function_test_host) device_type(host)
    integer :: i, j
@@ -131,7 +131,7 @@ end function target_function_test_host
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_with_dev_type_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_with_dev_type_recursive(increment) result(k)
 !$omp declare target to(implicitly_captured_with_dev_type_recursive) device_type(host)
    integer :: increment, k
@@ -143,7 +143,7 @@ recursive function implicitly_captured_with_dev_type_recursive(increment) result
 end function implicitly_captured_with_dev_type_recursive
 
 ! DEVICE-LABEL: func.func @_QPtarget_function_with_dev_type_recurse
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function target_function_with_dev_type_recurse() result(i)
 !$omp declare target to(target_function_with_dev_type_recurse) device_type(nohost)
    integer :: i
@@ -155,28 +155,28 @@ end function target_function_with_dev_type_recurse
 module test_module
 contains
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_nest_twice() result(i)
       integer :: i
       i = 10
    end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QMtest_modulePimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_one_twice() result(k)
       !$omp declare target to(implicitly_captured_one_twice) device_type(host)
       k = implicitly_captured_nest_twice()
    end function implicitly_captured_one_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function implicitly_captured_two_twice() result(y)
       integer :: y
       y = 5
    end function implicitly_captured_two_twice
 
 ! DEVICE-LABEL: func.func @_QMtest_modulePtarget_function_test_device
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
    function target_function_test_device() result(j)
       !$omp declare target to(target_function_test_device) device_type(nohost)
       integer :: i, j
@@ -200,7 +200,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end program
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive subroutine implicitly_captured_recursive(increment)
    integer :: increment
    if (increment == 10) then
@@ -211,7 +211,7 @@ recursive subroutine implicitly_captured_recursive(increment)
 end subroutine
 
 ! DEVICE-LABEL: func.func @_QPcaller_recursive
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 subroutine caller_recursive
 !$omp declare target to(caller_recursive) device_type(nohost)
    call implicitly_captured_recursive(0)
diff --git a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90 b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
index c31bfd1e61e8e..109c28f7d0fe5 100644
--- a/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-implicit-tarop-cap.f90
@@ -4,7 +4,7 @@
 !RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s --check-prefix=DEVICE
 
 ! DEVICE-LABEL: func.func @_QPimplicit_capture
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicit_capture() result(i)
    implicit none
    integer :: i
@@ -21,35 +21,35 @@ subroutine subr_target()
 !! -----
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice() result(k)
 !$omp declare target to(implicitly_captured_one_twice) device_type(host)
    k = implicitly_captured_nest_twice()
 end function implicitly_captured_one_twice
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_nest_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_nest_twice_enter() result(i)
    integer :: i
    i = 10
 end function implicitly_captured_nest_twice_enter
 
 ! CHECK-LABEL: func.func @_QPimplicitly_captured_one_twice_enter
-! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false>{{.*}}}
+! CHECK-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (host), capture_clause = (enter), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_one_twice_enter() result(k)
 !$omp declare target enter(implicitly_captured_one_twice_enter) device_type(host)
    k = implicitly_captured_nest_twice_enter()
 end function implicitly_captured_one_twice_enter
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_two_twice
-! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 function implicitly_captured_two_twice() result(y)
    integer :: y
    y = 5
@@ -67,7 +67,7 @@ end function target_function_test_device
 !! -----
 
 ! DEVICE-LABEL: func.func @_QPimplicitly_captured_recursive
-! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>{{.*}}}
+! DEVICE-SAME: {{.*}}attributes {{.*}}omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 recursive function implicitly_captured_recursive(increment) result(k)
    integer :: increment, k
    if (increment == 10) then
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
new file mode 100644
index 0000000000000..ced2535925888
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-clause.f90
@@ -0,0 +1,30 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+! Check that the INDIRECT clause on a DECLARE TARGET directive is lowered to the
+! `indirect` field of the omp.declare_target attribute.
+
+module functions
+  implicit none
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc_true({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_true() result(i)
+    !$omp declare target enter(func_true) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_implicit({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func_implicit() result(i)
+    !$omp declare target enter(func_implicit) indirect
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! CHECK: func.func @_QMfunctionsPfunc_false({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  function func_false() result(i)
+    !$omp declare target enter(func_false) indirect(.false.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90 b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
new file mode 100644
index 0000000000000..cee898cb84bee
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect-merge.f90
@@ -0,0 +1,47 @@
+! Check that the INDIRECT modifier is preserved (merged with logical OR) when a
+! procedure is named by more than one DECLARE TARGET directive. A prior
+! `indirect = true` must not be dropped, either by the device_type merge to
+! `any` or by an early return when the device_type already matches.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+
+module m1
+  implicit none
+contains
+  ! A later directive adds `indirect` with the same (default) device_type.
+  ! CHECK: func.func @_QMm1Pfoo1() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo1() result(i)
+    !$omp declare target enter(foo1)
+    !$omp declare target enter(foo1) indirect(.true.)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m2
+  implicit none
+contains
+  ! `indirect` first, plain second: it must stay set.
+  ! CHECK: func.func @_QMm2Pfoo2() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo2() result(i)
+    !$omp declare target enter(foo2) indirect(.true.)
+    !$omp declare target enter(foo2)
+    integer :: i
+    i = 1
+  end function
+end module
+
+module m3
+  implicit none
+contains
+  ! `indirect` (device_type any) followed by a device_type(nohost) declaration:
+  ! the device type merges to `any` and the `indirect = true` must be carried
+  ! over rather than overwritten.
+  ! CHECK: func.func @_QMm3Pfoo3() -> i32 attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function foo3() result(i)
+    !$omp declare target enter(foo3) indirect(.true.)
+    !$omp declare target enter(foo3) device_type(nohost)
+    integer :: i
+    i = 1
+  end function
+end module
diff --git a/flang/test/Lower/OpenMP/declare-target-indirect.f90 b/flang/test/Lower/OpenMP/declare-target-indirect.f90
new file mode 100644
index 0000000000000..3cd87d9b82575
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-indirect.f90
@@ -0,0 +1,39 @@
+! This test checks the lowering of the OpenMP INDIRECT clause when used with the
+! DECLARE TARGET directive, together with an indirect call (through a procedure
+! pointer) from within a target region.
+
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=52 -fopenmp-is-target-device %s -o - | FileCheck %s
+
+module functions
+  implicit none
+
+  interface
+    function func() result(i)
+      character(1) :: i
+    end function
+  end interface
+
+contains
+  ! CHECK: func.func @_QMfunctionsPfunc1({{.*}}attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  function func1() result(i)
+    !$omp declare target enter(func1) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+    return
+  end function
+end module
+
+program main
+  use functions
+  implicit none
+  procedure (func), pointer :: ptr1=>func1
+  character(1) :: val1
+
+  ! CHECK-LABEL: func.func @_QQmain()
+  ! CHECK: omp.target
+  !$omp target map(from: val1)
+  val1 = ptr1()
+  !$omp end target
+
+end program
diff --git a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90 b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
index f54f7ed14d1d5..9435eae996a98 100644
--- a/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
+++ b/flang/test/Lower/OpenMP/declare-target-unnamed-main.f90
@@ -7,7 +7,7 @@
 ! appropriately mark the function as declare target, even when
 ! unused within the target region.
 
-!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>{{.*}}}
+!CHECK: func.func @_QPfoo(%{{.*}}: !fir.ref<f32>{{.*}}) -> f32 attributes {{{.*}}omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>{{.*}}}
 
 interface
 real function foo (x)
diff --git a/flang/test/Lower/OpenMP/function-filtering-2.f90 b/flang/test/Lower/OpenMP/function-filtering-2.f90
index 6ced7851dd4ea..5cfbe4ad6e031 100644
--- a/flang/test/Lower/OpenMP/function-filtering-2.f90
+++ b/flang/test/Lower/OpenMP/function-filtering-2.f90
@@ -5,13 +5,13 @@
 ! RUN: bbc -fopenmp -fopenmp-version=52 -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-HOST,MLIR-ALL %s
 ! RUN: %if amdgpu-registered-target %{ bbc -target amdgcn-amd-amdhsa -fopenmp -fopenmp-version=52 -fopenmp-is-target-device -emit-hlfir %s -o - | FileCheck --check-prefixes=MLIR-DEVICE,MLIR-ALL %s %}
 
-! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}implicit_invocation() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}implicit_invocation{{.*}}(
 subroutine implicit_invocation()
 end subroutine implicit_invocation
 
-! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget{{.*}}(
 subroutine declaretarget()
@@ -19,7 +19,7 @@ subroutine declaretarget()
     call implicit_invocation()
 end subroutine declaretarget
 
-! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false>}
+! MLIR: func.func @{{.*}}declaretarget_enter() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}declaretarget_enter{{.*}}(
 subroutine declaretarget_enter()
@@ -27,7 +27,7 @@ subroutine declaretarget_enter()
     call implicit_invocation()
 end subroutine declaretarget_enter
 
-! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false>}
+! MLIR: func.func @{{.*}}no_declaretarget() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (to), automap = false, indirect = false>}
 ! MLIR: return
 ! LLVM: define {{.*}} @{{.*}}no_declaretarget{{.*}}(
 subroutine no_declaretarget()
diff --git a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90 b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
index dc23a8190cb8d..df96448eda129 100644
--- a/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
+++ b/flang/test/Lower/OpenMP/omp-declare-target-program-var.f90
@@ -5,7 +5,7 @@ PROGRAM main
     ! HOST-DAG: %[[I_REF:.*]] = fir.alloca f32 {bindc_name = "i", uniq_name = "_QFEi"}
     ! HOST-DAG: %[[I_DECL:.*]]:2 = hlfir.declare %[[I_REF]] {uniq_name = "_QFEi"} : (!fir.ref<f32>) -> (!fir.ref<f32>, !fir.ref<f32>)
     REAL :: I
-    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false>} : f32 {
+    ! ALL-DAG: fir.global internal @_QFEi {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (to), automap = false, indirect = false>} : f32 {
     ! ALL-DAG: %[[UNDEF:.*]] = fir.zero_bits f32
     ! ALL-DAG: fir.has_value %[[UNDEF]] : f32
     ! ALL-DAG: }
diff --git a/flang/test/Semantics/indirect02.f90 b/flang/test/Semantics/indirect02.f90
index 3fae39f1c2281..8835919341a9b 100644
--- a/flang/test/Semantics/indirect02.f90
+++ b/flang/test/Semantics/indirect02.f90
@@ -1,7 +1,8 @@
-! This test checks the lowering of OpenMP Indirect Clause when used with the Declare Target directive
+! This test checks that the OpenMP INDIRECT clause on the DECLARE TARGET
+! directive is rejected before OpenMP 5.1 and accepted from 5.1 onwards.
 
 ! RUN: not %flang -fopenmp -fopenmp-version=50 %s 2>&1 | FileCheck %s --check-prefix="CHECK-50"
-! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
+! RUN: %flang_fc1 -fopenmp -fopenmp-version=52 -fdebug-unparse %s 2>&1 | FileCheck %s --check-prefix="CHECK-52"
 
 module functions
   implicit none
@@ -15,7 +16,7 @@ function func() result(i)
 contains
   function func1() result(i)
     !CHECK-50: INDIRECT clause is not allowed on DECLARE TARGET directive in OpenMP v5.0, try -fopenmp-version=51
-    !CHECK-52: not yet implemented: Unhandled clause INDIRECT in DECLARE TARGET construct
+    !CHECK-52: !$OMP DECLARE TARGET ENTER(func1) INDIRECT(.true._4)
     !$omp declare target enter(func1) indirect(.true.)
     character(1) :: i
     i = 'a'
diff --git a/flang/test/Semantics/indirect03.f90 b/flang/test/Semantics/indirect03.f90
new file mode 100644
index 0000000000000..0bab188a9ff51
--- /dev/null
+++ b/flang/test/Semantics/indirect03.f90
@@ -0,0 +1,31 @@
+! This test checks the OpenMP restriction that the INDIRECT clause on a DECLARE
+! TARGET directive is only allowed with DEVICE_TYPE(ANY) (an absent DEVICE_TYPE
+! clause also implies ANY). A host- or device-only procedure cannot be the
+! target of an indirect device invocation.
+
+! RUN: not %flang -fopenmp -fopenmp-version=52 %s 2>&1 | FileCheck %s
+
+module functions
+  implicit none
+contains
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_host() result(i)
+    !$omp declare target enter(func_host) device_type(host) indirect(.true.)
+    character(1) :: i
+    i = 'a'
+  end function
+
+  !CHECK: Only the DEVICE_TYPE(ANY) clause is allowed with the INDIRECT clause on the DECLARE TARGET directive
+  function func_nohost() result(i)
+    !$omp declare target enter(func_nohost) device_type(nohost) indirect(.true.)
+    character(1) :: i
+    i = 'b'
+  end function
+
+  ! DEVICE_TYPE(ANY) with INDIRECT is allowed, so no error is expected here.
+  function func_any() result(i)
+    !$omp declare target enter(func_any) device_type(any) indirect(.true.)
+    character(1) :: i
+    i = 'c'
+  end function
+end module
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
index c9e6764e7d634..bccdc8f6fdd19 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
@@ -45,7 +45,8 @@ def DeclareTargetAttr : OpenMP_Attr<"DeclareTarget", "declaretarget"> {
   let parameters =
       (ins OptionalParameter<"DeclareTargetDeviceTypeAttr">:$device_type,
           OptionalParameter<"DeclareTargetCaptureClauseAttr">:$capture_clause,
-          OptionalParameter<"BoolAttr">:$automap);
+          OptionalParameter<"BoolAttr">:$automap,
+          OptionalParameter<"BoolAttr">:$indirect);
 
   let assemblyFormat = "`<` struct(params) `>`";
 }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
index 35b58375698f6..fbe5ee576228b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauseOperands.h
@@ -33,6 +33,12 @@ struct DeviceTypeClauseOps {
   DeclareTargetDeviceType deviceType = DeclareTargetDeviceType::any;
 };
 
+struct IndirectClauseOps {
+  /// Whether the declare target entities may be invoked indirectly (through a
+  /// function pointer) from within a target region.
+  bool indirect = false;
+};
+
 //===----------------------------------------------------------------------===//
 // Extra operation operand structures.
 //===----------------------------------------------------------------------===//
@@ -43,8 +49,8 @@ using HostEvaluatedOperands =
     detail::Clauses<CollapseClauseOps, LoopRelatedClauseOps, NumTeamsClauseOps,
                     NumThreadsClauseOps, ThreadLimitClauseOps>;
 
-// TODO: Add `indirect` clause.
-using DeclareTargetOperands = detail::Clauses<DeviceTypeClauseOps>;
+using DeclareTargetOperands =
+    detail::Clauses<DeviceTypeClauseOps, IndirectClauseOps>;
 
 /// omp.target_enter_data, omp.target_exit_data and omp.target_update take the
 /// same clauses, so we give the structure to be shared by all of them a
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index 51f925b17f47e..10dd4b054623b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -376,7 +376,7 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
       /*methodName=*/"setDeclareTarget",
       (ins "mlir::omp::DeclareTargetDeviceType":$deviceType,
            "mlir::omp::DeclareTargetCaptureClause":$captureClause,
-           "bool":$automap), [{}], [{
+           "bool":$automap, "bool":$indirect), [{}], [{
         $_op->setAttr("omp.declare_target",
                   mlir::omp::DeclareTargetAttr::get(
                       $_op->getContext(),
@@ -384,7 +384,8 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
                           $_op->getContext(), deviceType),
                       mlir::omp::DeclareTargetCaptureClauseAttr::get(
                           $_op->getContext(), captureClause),
-                      mlir::BoolAttr::get($_op->getContext(), automap)));
+                      mlir::BoolAttr::get($_op->getContext(), automap),
+                      mlir::BoolAttr::get($_op->getContext(), indirect)));
       }]>,
       InterfaceMethod<
       /*description=*/[{
@@ -435,6 +436,21 @@ def DeclareTargetInterface : OpInterface<"DeclareTargetInterface"> {
             if (auto autoVal = dAttr.getAutomap())
               return autoVal.getValue();
          return false;
+      }]>,
+      InterfaceMethod<
+      /*description=*/[{
+        Return true if the DeclareTarget attribute has the INDIRECT modifier,
+        indicating that the entity may be invoked through a function pointer
+        from within a target region.
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getDeclareTargetIndirect",
+      (ins), [{}], [{
+        if (mlir::Attribute dTar = $_op->getAttr("omp.declare_target"))
+          if (auto dAttr = llvm::dyn_cast_or_null<mlir::omp::DeclareTargetAttr>(dTar))
+            if (auto indVal = dAttr.getIndirect())
+              return indVal.getValue();
+         return false;
       }]>
   ];
 }
diff --git a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
index 84b0d40949dba..1d066facdecd3 100644
--- a/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
+++ b/mlir/lib/Dialect/OpenMP/Transforms/MarkDeclareTarget.cpp
@@ -58,11 +58,16 @@ class MarkDeclareTargetPass
           currentDt != omp::DeclareTargetDeviceType::any) {
         current.setDeclareTarget(omp::DeclareTargetDeviceType::any,
                                  current.getDeclareTargetCaptureClause(),
-                                 current.getDeclareTargetAutomap());
+                                 current.getDeclareTargetAutomap(),
+                                 current.getDeclareTargetIndirect());
       }
     } else {
+      // The `indirect` modifier is a property of the specific declare target
+      // declaration; it is not inherited by functions that are reached only
+      // through (direct) calls. Such implicitly captured functions are marked
+      // declare target, but must not be registered as indirect call targets.
       current.setDeclareTarget(parentInfo.devTy, parentInfo.capClause,
-                               parentInfo.automap);
+                               parentInfo.automap, /*indirect=*/false);
     }
 
     markNestedFuncs(parentInfo, symOp, visited);
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 4574b20ce27fd..f447d68ef441f 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8790,6 +8790,69 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Register a `declare target ... indirect` function so that the OpenMP runtime
+/// can resolve indirect calls (through a function pointer) to it from within a
+/// target region. This mirrors clang's
+/// `CGOpenMPRuntime::emitDeclareTargetFunction`.
+///
+/// On the host the function itself is registered as an indirect offload entry,
+/// which causes an offloading entry (with the indirect flag) to be emitted at
+/// module finalization. On the target device a new, externally visible global
+/// holding the address of the function is generated (so the runtime can read
+/// the device address while leaving the function's own linkage and visibility
+/// unchanged) and that global is registered instead.
+static void registerIndirectDeclareTargetFunction(
+    FunctionOpInterface funcOp, llvm::OpenMPIRBuilder *ompBuilder,
+    LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::Function *llvmFunc = moduleTranslation.lookupFunction(funcOp.getName());
+  if (!llvmFunc)
+    return;
+
+  // Build the unique offload entry name using the function as the parent name,
+  // e.g. `__omp_offloading_<device>_<file>_<func>_l<line>`.
+  auto loc = funcOp->getLoc()->findInstanceOf<FileLineColLoc>();
+  auto fileInfoCallBack = [&loc]() {
+    std::string filename = "";
+    std::uint64_t lineNo = 0;
+    if (loc) {
+      filename = loc.getFilename().str();
+      lineNo = loc.getLine();
+    }
+    return std::pair<std::string, std::uint64_t>(llvm::StringRef(filename),
+                                                 lineNo);
+  };
+
+  llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
+  llvm::TargetRegionEntryInfo entryInfo = ompBuilder->getTargetEntryUniqueInfo(
+      fileInfoCallBack, vfs, funcOp.getName());
+  llvm::SmallString<128> name;
+  ompBuilder->OffloadInfoManager.getTargetRegionEntryFnName(name, entryInfo);
+
+  llvm::Module *llvmModule = moduleTranslation.getLLVMModule();
+  const llvm::DataLayout &dl = llvmModule->getDataLayout();
+  // The entry tracks a pointer to the function, so its size is the store size
+  // of a pointer.
+  int64_t varSize = dl.getTypeStoreSize(
+      llvm::PointerType::getUnqual(llvmModule->getContext()));
+
+  llvm::Constant *addr = llvmFunc;
+  if (ompBuilder->Config.isTargetDevice()) {
+    llvm::PointerType *fnPtrTy = llvm::PointerType::get(
+        llvmModule->getContext(), dl.getProgramAddressSpace());
+    auto *addrGlobal = new llvm::GlobalVariable(
+        *llvmModule, fnPtrTy, /*isConstant=*/true,
+        llvm::GlobalValue::ExternalLinkage, llvmFunc, name, nullptr,
+        llvm::GlobalValue::NotThreadLocal, dl.getDefaultGlobalsAddressSpace());
+    addrGlobal->setVisibility(llvm::GlobalValue::ProtectedVisibility);
+    addr = addrGlobal;
+  }
+
+  ompBuilder->OffloadInfoManager.registerDeviceGlobalVarEntryInfo(
+      name, addr, varSize,
+      llvm::OffloadEntriesInfoManager::OMPTargetGlobalVarEntryIndirect,
+      llvm::GlobalValue::WeakODRLinkage);
+}
+
 static LogicalResult
 convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                          llvm::OpenMPIRBuilder *ompBuilder,
@@ -8804,13 +8867,25 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
   if (FunctionOpInterface funcOp = dyn_cast<FunctionOpInterface>(op)) {
     if (auto offloadMod = dyn_cast<omp::OffloadModuleInterface>(
             op->getParentOfType<ModuleOp>().getOperation())) {
-      if (!offloadMod.getIsTargetDevice())
-        return success();
-
+      bool isTargetDevice = offloadMod.getIsTargetDevice();
       omp::DeclareTargetDeviceType declareType =
           attribute.getDeviceType().getValue();
+      bool isHostFunc = declareType == omp::DeclareTargetDeviceType::host;
+
+      // A `declare target ... indirect(.true.)` function must be registered so
+      // that indirect calls to it from within a target region can be resolved
+      // by the runtime. This applies to both host and device compilation, but
+      // not to host-only functions that are about to be deleted on the device.
+      mlir::BoolAttr indirectAttr = attribute.getIndirect();
+      if (indirectAttr && indirectAttr.getValue() &&
+          !(isTargetDevice && isHostFunc))
+        registerIndirectDeclareTargetFunction(funcOp, ompBuilder,
+                                              moduleTranslation);
+
+      if (!isTargetDevice)
+        return success();
 
-      if (declareType == omp::DeclareTargetDeviceType::host) {
+      if (isHostFunc) {
         llvm::Function *llvmFunc =
             moduleTranslation.lookupFunction(funcOp.getName());
         llvmFunc->dropAllReferences();
diff --git a/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
new file mode 100644
index 0000000000000..3ebdcb66cd017
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/mark-declare-target-indirect.mlir
@@ -0,0 +1,39 @@
+// RUN: mlir-opt -omp-mark-declare-target -split-input-file %s | FileCheck %s
+
+// The `omp-mark-declare-target` pass marks functions that are reachable from
+// explicit target code as implicitly declare target. The `indirect` modifier
+// however is a property of the specific declare target declaration and must NOT
+// be propagated to functions that are only reached through (direct) calls.
+
+// A function explicitly declared `indirect` that directly calls another
+// function: the callee is implicitly captured and must be marked declare target
+// with `indirect = false`, not inherit the parent's `indirect = true`.
+module {
+  // CHECK: func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @direct_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @direct_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = false>}
+  func.func @direct_callee() {
+    return
+  }
+}
+
+// -----
+
+// A callee that is itself explicitly declared `indirect` keeps its own value
+// (the pass must not clobber it).
+module {
+  // CHECK: func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @indirect_parent2() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    func.call @explicitly_indirect_callee() : () -> ()
+    return
+  }
+
+  // CHECK: func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>}
+  func.func @explicitly_indirect_callee() attributes {omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter), automap = false, indirect = true>} {
+    return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
new file mode 100644
index 0000000000000..ad86bed2770fa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-device.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the device-side lowering of `declare target ... indirect` functions. A
+// new global holding the address of the function is generated with protected
+// visibility so the runtime can access it. A function marked `indirect = false`
+// must not generate such a global.
+
+module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
+  // CHECK: @[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = protected constant ptr @indirect_fn
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an indirect global.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
new file mode 100644
index 0000000000000..f3b098018d196
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-indirect-host.mlir
@@ -0,0 +1,20 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Test the host-side lowering of `declare target ... indirect` functions. The
+// runtime needs an offload entry (flag 8 == OMPTargetGlobalVarEntryIndirect)
+// so that indirect calls within a target region can be resolved. A function
+// marked `indirect = false` must not register an offload entry.
+
+// CHECK-DAG: %struct.__tgt_offload_entry = type { i64, i16, i16, i32, ptr, ptr, i64, i64, ptr }
+module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_target_device = false} {
+  // CHECK: @.offloading.entry.[[ENTRY:__omp_offloading_[0-9a-z]+_[0-9a-z]+_indirect_fn_l[0-9]+]] = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 8, ptr @indirect_fn, ptr @{{.*}}, i64 8, i64 0, ptr null }
+  llvm.func @indirect_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = true>} {
+    llvm.return
+  }
+
+  // A function marked `indirect = false` must not produce an offload entry.
+  // CHECK-NOT: plain_fn_l
+  llvm.func @plain_fn() attributes {omp.declare_target = #omp.declaretarget<device_type = (nohost), capture_clause = (enter), automap = false, indirect = false>} {
+    llvm.return
+  }
+}


From llvmlistbot at llvm.org  Thu Jul  9 09:04:15 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 09 Jul 2026 09:04:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a4fc67f.da8bafd3.14cd50.32e3@mx.google.com>


https://github.com/nujaa edited https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Thu Jul  9 09:09:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 09:09:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] d4dd90a - [mlir][tosa] Add constant block
	scaled support (#205506)
Message-ID: <6a4fc7cc.f06f4994.1fc857.e39b@mx.google.com>


Author: Luke Hutton
Date: 2026-07-09T17:09:44+01:00
New Revision: d4dd90ae4d7a6b559ebc2ef5b75d2eacafadb48f

URL: https://github.com/llvm/llvm-project/commit/d4dd90ae4d7a6b559ebc2ef5b75d2eacafadb48f
DIFF: https://github.com/llvm/llvm-project/commit/d4dd90ae4d7a6b559ebc2ef5b75d2eacafadb48f.diff

LOG: [mlir][tosa] Add constant block scaled support (#205506)

This commit adds support for block scaled tensors.

In particular, the block scaled type has been extended with the
`DenseElementTypeInterface` to allow block scaled data values to be
specified in dense element attributes.

The block scaled type has also been extended to allow optional scale
values to be specified by the type. For now, scale values are not
expected to be propagated beyond their use in the attribute input of a
constant operation. In the future, we may want to propagate these values
to allow certain optimizations.

The `tosa.const` operation has also been updated in the validation pass.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
    mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
    mlir/test/Dialect/Tosa/verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..291097a7f7514 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1305,7 +1305,25 @@ extensionComplianceMap = {
         {{fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT},
         {{fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT},
-        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+        {{mxint8T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.identity",
      {{{Extension::int4}, {{{i4T, i4T}, SpecificationVersion::V_1_0}}},
       {{Extension::int16}, {{{i48T, i48T}, SpecificationVersion::V_1_0}}},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index dfecda940d8c7..b1404d9c700ae 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -92,8 +92,6 @@ class TosaShapeOperatorWithSameRanks
   }
 };
 
-LogicalResult verifyBlockScaledTensorType(mlir::Type type);
-
 } // namespace tosa
 } // namespace OpTrait
 
@@ -143,6 +141,10 @@ Type getStorageElementTypeOrSelf(Type type);
 // Returns the storage element type for a given value
 Type getStorageElementTypeOrSelf(Value value);
 
+// Verify a block scaled tensor type is valid
+LogicalResult verifyBlockScaledTensorType(mlir::Type type,
+                                          bool allowScaleValues);
+
 } // namespace tosa
 } // namespace mlir
 

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..5e5760e33b5f6 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2970,9 +2970,13 @@ def Tosa_ConstOp : Tosa_Op<"const", [ConstantLike, Pure,
     Tosa_Tensor:$output
   );
 
-  list<Availability> availability = [
-    Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP, Tosa_EXT_INT64]>,
+  list<Availability> availability =
+      [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
+       Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3,
+                  Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MXFP,
+                  Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasFolder = 1;

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 63be7b69b6398..1518024ecaa19 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -109,7 +109,12 @@ def Tosa_MXFPValue
 def Tosa_MXFPScale
     : AnyTypeOf<[F8E8M0FNU], "micro-scaling format scale number">;
 
-def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
+def Tosa_BlockScaled
+    : Tosa_Type<"BlockScaled", "block_scaled",
+                [DeclareTypeInterfaceMethods<
+                    DenseElementTypeInterface, ["getDenseElementBitSize",
+                                                "convertToAttribute",
+                                                "convertFromAttribute"]>]> {
   let summary = "Block scaled tensor element type.";
 
   let description = [{
@@ -121,17 +126,26 @@ def Tosa_BlockScaled : Tosa_Type<"BlockScaled", "block_scaled"> {
       specifying the block size along the innermost dimension.
     `scale_type` - The type of the scale value associated with each block.
     `value_type` - The type of the data values in each block.
+    `scale_values` - Optional array of per-block scale values. If not provided,
+      the type is assumed to be dynamically quantized at runtime.
 
   }];
 
   let parameters = (ins
     EnumParameter<Tosa_BlockShape>:$block_shape,
     Tosa_MXFPScale:$scale_type,
-    Tosa_MXFPValue:$value_type
+    Tosa_MXFPValue:$value_type,
+    OptionalArrayRefParameter<"Attribute">:$scale_values
   );
 
   let assemblyFormat =
-      "`<` $block_shape```:```$scale_type```:```$value_type `>`";
+      "`<` $block_shape```:```$scale_type```:```$value_type "
+      "(`,` `{` custom<ScaleValues>($scale_values, ref($scale_type))^ `}`)? "
+      "`>`";
+
+  let extraClassDeclaration = [{
+    bool hasScaleValues() const { return !getScaleValues().empty(); }
+  }];
 }
 
 //===----------------------------------------------------------------------===//
@@ -158,8 +172,7 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::OpTrait::tosa::"
-            "verifyBlockScaledTensorType($_self))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
 class TosaTensorOf<list<Type> allowedTypes,
                    string summary = "tosa-conformant tensor">

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index 829c0b41a68cf..bf9d98ffe35a0 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/Tosa/Utils/ShapeUtils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/VerificationUtils.h"
+#include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/DialectImplementation.h"
 #include "mlir/IR/Matchers.h"
@@ -740,31 +741,114 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult OpTrait::tosa::verifyBlockScaledTensorType(mlir::Type type) {
+LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
+                                                      bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
+  if (!allowScaleValues && elemType.hasScaleValues())
+    return failure();
+
   if (!tensorType.hasRank())
     return success();
 
   if (tensorType.getRank() == 0)
     return failure();
 
-  const int64_t blockedDimension = tensorType.getShape().back();
-  if (ShapedType::isDynamic(blockedDimension))
-    return success();
-
+  const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
       BlockShapeAttr::getBlockShapeValue(elemType.getBlockShape());
+
+  if (allowScaleValues && elemType.hasScaleValues() &&
+      tensorType.hasStaticShape()) {
+    const size_t numBlocks = tensorType.getNumElements() / blockSize;
+    if (elemType.getScaleValues().size() != numBlocks)
+      return failure();
+  }
+
+  const int64_t blockedDimension = tensorShape.back();
+  if (ShapedType::isDynamic(blockedDimension))
+    return success();
   if (blockedDimension % blockSize != 0)
     return failure();
 
   return success();
 }
 
+static ParseResult parseScaleValues(AsmParser &parser,
+                                    SmallVector<Attribute> &scaleValues,
+                                    Type scaleType) {
+  const auto parseScaleValue = [&]() -> ParseResult {
+    const SMLoc loc = parser.getCurrentLocation();
+
+    double floatValue;
+    if (parser.parseFloat(floatValue))
+      return failure();
+
+    if (floatValue < 0.0)
+      return parser.emitError(loc, "scale value must be non-negative, got ")
+             << floatValue;
+
+    Type attrType = scaleType;
+    if (succeeded(parser.parseOptionalColon()) && parser.parseType(attrType))
+      return failure();
+
+    if (attrType != scaleType)
+      return parser.emitError(loc, "parsed attribute type ")
+             << attrType << " does not match expected scale type " << scaleType;
+
+    scaleValues.push_back(FloatAttr::get(attrType, floatValue));
+    return success();
+  };
+
+  return parser.parseCommaSeparatedList(parseScaleValue);
+}
+
+static void printScaleValues(AsmPrinter &printer,
+                             ArrayRef<Attribute> scaleValues, Type) {
+  llvm::interleaveComma(scaleValues, printer, [&](Attribute scaleValue) {
+    printer.printAttributeWithoutType(scaleValue);
+  });
+}
+
+size_t mlir::tosa::BlockScaledType::getDenseElementBitSize() const {
+  const Type valueType = getValueType();
+  if (isa<tosa::mxint8Type>(valueType))
+    return 8;
+  return valueType.getIntOrFloatBitWidth();
+}
+
+Attribute
+mlir::tosa::BlockScaledType::convertToAttribute(ArrayRef<char> rawData) const {
+  // Block scaled values are stored as a single byte. This is because possible
+  // value data types are either 8-bit or sub-byte. Sub-byte types are aligned
+  // to 8-bits.
+  assert(rawData.size() == 1 && "expected 1 byte for block_scaled element");
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertToAttribute(rawData);
+  if (!isa<FloatType>(valueType))
+    return {};
+  return mlir::detail::convertFloatTypeToAttribute(valueType, rawData);
+}
+
+LogicalResult mlir::tosa::BlockScaledType::convertFromAttribute(
+    Attribute attr, SmallVectorImpl<char> &result) const {
+  const Type valueType = getValueType();
+  if (const auto mxint8Value = dyn_cast<tosa::mxint8Type>(valueType))
+    return mxint8Value.convertFromAttribute(attr, result);
+
+  const auto floatAttr = dyn_cast<FloatAttr>(attr);
+  if (!floatAttr || floatAttr.getType() != valueType)
+    return failure();
+  // const APFloat value = floatAttr.getValue();
+  return mlir::detail::convertFloatTypeFromAttribute(valueType, floatAttr,
+                                                     result);
+}
+
 //===----------------------------------------------------------------------===//
 // TOSA Operator Verifiers.
 //===----------------------------------------------------------------------===//
@@ -851,7 +935,7 @@ static LogicalResult verifyConvOp(T op) {
 }
 
 LogicalResult tosa::ConstOp::verify() {
-
+  Operation &op = *getOperation();
   auto attrType = llvm::dyn_cast<TensorType>(getValuesAttr().getType());
   auto outputType = llvm::dyn_cast<TensorType>(getOutput().getType());
 
@@ -860,17 +944,49 @@ LogicalResult tosa::ConstOp::verify() {
     return failure();
   }
 
-  if (auto result = llvm::dyn_cast<mlir::quant::QuantizedType>(
-          outputType.getElementType())) {
-    if (getStorageElementTypeFromQuantized(result) == attrType.getElementType())
+  const Type attrElemType = attrType.getElementType();
+  const Type resultElemType = outputType.getElementType();
+
+  if (auto result =
+          llvm::dyn_cast<mlir::quant::QuantizedType>(resultElemType)) {
+    if (getStorageElementTypeFromQuantized(result) == attrElemType)
       return success();
   }
 
-  if (attrType.getElementType() != outputType.getElementType()) {
-    emitOpError("expected same attr/result element types");
-    return failure();
+  if (auto attrBlockScaledType =
+          llvm::dyn_cast<mlir::tosa::BlockScaledType>(attrElemType)) {
+    if (!attrBlockScaledType.hasScaleValues())
+      return op.emitOpError(
+          "attribute block scaled type must have scale values");
+
+    if (failed(verifyBlockScaledTensorType(attrType, true)))
+      return op.emitOpError("block scaled attribute type is not valid, got ")
+             << attrType;
+
+    const BlockScaledType resultBlockScaledType =
+        llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
+    if (!resultBlockScaledType)
+      return op.emitOpError(
+          "result type must be block scaled type if attribute is block "
+          "scaled type");
+
+    if (attrBlockScaledType.getValueType() !=
+            resultBlockScaledType.getValueType() ||
+        attrBlockScaledType.getScaleType() !=
+            resultBlockScaledType.getScaleType() ||
+        attrBlockScaledType.getBlockShape() !=
+            resultBlockScaledType.getBlockShape())
+      return op.emitOpError(
+                 "expected block scaled element type to be compatible "
+                 "between attr and result, got ")
+             << attrBlockScaledType << " vs. " << resultBlockScaledType;
+
+    return success();
   }
 
+  if (attrElemType != resultElemType)
+    return emitOpError("expected same attr/result element types");
+
   return success();
 }
 

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..bc014fb9b928e 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -660,7 +660,7 @@ func.func @test_rescale(%arg0: tensor<13x21x3x!quant.uniform<u8:f32, 0.015655439
 // CHECK-LABEL: test_const
 func.func @test_const(%arg0 : index) -> tensor<4xi32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64] ]
+  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mxfp, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
     %0 = "tosa.const"() {values = dense<[3, 0, 1, 2]> : tensor<4xi32>} : () -> tensor<4xi32>
     return %0 : tensor<4xi32>
 }

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..4e71d512def2d 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1938,3 +1938,75 @@ func.func @test_assert_equal_shape() {
   tosa.assert_equal_shape %0, %1 {allow_broadcast = true} : (!tosa.shape<2>, !tosa.shape<2>) -> ()
   return
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const
+func.func @test_block_scaled_const() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 1.0}>> : [[0.0 : f8E4M3FN, 1.0 : f8E4M3FN, 0.001953125 : f8E4M3FN, 0.0078125 : f8E4M3FN, 
+                                                                                                                          2.0 : f8E4M3FN, 2.25 : f8E4M3FN, 2.5 : f8E4M3FN, 2.75 : f8E4M3FN, 
+                                                                                                                          15.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN,  
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN], 
+                                                                                                                         [0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 
+                                                                                                                          0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN, 0.0 : f8E4M3FN]]>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat
+func.func @test_block_scaled_const_splat() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0, 1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_splat_mxint8
+func.func @test_block_scaled_const_splat_mxint8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 1.0}>> : 0 : i8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values
+func.func @test_block_scaled_const_scale_values() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_explicit_type
+func.func @test_block_scaled_const_scale_values_explicit_type() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : f8E8M0FNU, 2.0 : f8E8M0FNU}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_scale_values_wide_inner_dim
+func.func @test_block_scaled_const_scale_values_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0, 2.0, 4.0, 8.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_const_cast_scale_values_no_propagate
+func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x32xf32> {
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %1 = tosa.cast %0 : (tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<2x32xf32>
+  return %1 : tensor<2x32xf32>
+}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..bea262a11a391 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -193,6 +193,24 @@ func.func @test_const_fp6e3m2(%arg0 : index) -> tensor<4xf6E3M2FN> {
 
 // -----
 
+func.func @test_const_block_scaled_types() -> (tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %0 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %1 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN, {1.0}>> : 0.0 : f6E2M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %2 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN, {1.0}>> : 0.0 : f6E3M2FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %3 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %4 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2, {1.0}>> : 0.0 : f8E5M2>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  // expected-error at +1 {{'tosa.const' op illegal: requires specification version compatible with 1.1.draft (got 1.0)}}
+  %5 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0}>> : 0 : i8>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2, %3, %4, %5 : tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 func.func @test_cast_from_block_scaled(%arg0: tensor<4x32xf8E5M2>, %arg1: tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32> {
   // expected-error at +1 {{'tosa.cast_from_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires any of [mxfp] profiles/extensions to be specified in the target environment}}
   %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<4x32xf8E5M2>, tensor<4x1xf8E8M0FNU>) -> tensor<4x32xf32>

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..d290ff71f052d 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -152,6 +152,19 @@ func.func @test_const_mxint8() -> tensor<2x!tosa.mxint8> {
 
 // -----
 
+// CHECK-LABEL: test_const_block_scaled_types
+func.func @test_const_block_scaled_types() -> (tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) {
+  %0 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN, {1.0}>> : 0.0 : f4E2M1FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  %1 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN, {1.0}>> : 0.0 : f6E2M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>
+  %2 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN, {1.0}>> : 0.0 : f6E3M2FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>
+  %3 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  %4 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2, {1.0}>> : 0.0 : f8E5M2>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>
+  %5 = "tosa.const"() <{values = dense<tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0}>> : 0 : i8>}> : () -> tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0, %1, %2, %3, %4, %5 : tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E5M2>>, tensor<1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_matmul_t_block_scaled_mxint8
 func.func @test_matmul_t_block_scaled_mxint8(%arg0: tensor<4x8x32x!tosa.mxint8>, %arg1: tensor<4x8x1xf8E8M0FNU>, %arg2: tensor<4x16x32x!tosa.mxint8>, %arg3: tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32> {
   %0 = tosa.matmul_t_block_scaled %arg0, %arg1, %arg2, %arg3 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<4x8x32x!tosa.mxint8>, tensor<4x8x1xf8E8M0FNU>, tensor<4x16x32x!tosa.mxint8>, tensor<4x16x1xf8E8M0FNU>) -> tensor<4x8x16xf32>

diff  --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 478532e52c438..c31effeb6a1d2 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -2268,3 +2268,71 @@ func.func @test_const_mxint8_int64(%arg0 : index) -> tensor<2x!tosa.mxint8> {
   %0 = "tosa.const"() {values = dense<tensor<2x!tosa.mxint8> : [127, 245]>} : () -> tensor<2x!tosa.mxint8>
   return %0 : tensor<2x!tosa.mxint8>
 }
+
+// -----
+
+func.func @test_block_scaled_const_splat_ui8() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{incompatible attribute for element type}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> : 0 : ui8>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_splat_fp64() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{incompatible attribute for element type}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> : 0.0>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_integer_scale_value() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +2 {{unexpected decimal integer literal for a floating point value}}
+  // expected-note at +1 {{add a trailing dot to make the literal a float}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1 : i32, 2 : i32}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_negative_scale_value() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{scale value must be non-negative, got -1.000000e+00}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : f8E8M0FNU, -1.0 : f8E8M0FNU}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_scale_value_non_float_explicit_type() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{parsed attribute type 'i32' does not match expected scale type 'f8E8M0FNU'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {1.0 : i32}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
+
+// -----
+
+!mxint8 = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>
+!mxint8_scale = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0, 4.0}>
+
+func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8> {
+  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00, 4.000000e+00}>>'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!mxint8_scale> : 0 : i8>}> : () -> tensor<2x32x!mxint8>
+  return %0 : tensor<2x32x!mxint8>
+}
+
+// -----
+
+func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00}>>'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0}>> : 0 : i8>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
+func.func @test_block_scaled_const_cast_scale_values_propagate() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> {
+  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
+  %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
+  return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
+}


        

From llvmlistbot at llvm.org  Thu Jul  9 09:09:57 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 09 Jul 2026 09:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add constant block scaled
	support (PR #205506)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205506@github.com>
Message-ID: <6a4fc7d5.481afadf.196d57.3b66@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/205506

From llvmlistbot at llvm.org  Thu Jul  9 09:17:49 2026
From: llvmlistbot at llvm.org (Krzysztof Drewniak)
Date: Thu, 09 Jul 2026 09:17:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ROCm] Build mlir_rocm_runtime
	STANDALONE (PR #205651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205651@github.com>
Message-ID: <6a4fc9ad.92d2873c.258297.af10@mx.google.com>


https://github.com/krzysz00 approved this pull request.

... I thought I'd approved this, thanks for the ping!

https://github.com/llvm/llvm-project/pull/205651

From llvmlistbot at llvm.org  Thu Jul  9 09:27:12 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 09 Jul 2026 09:27:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4fcbe0.da8bafd3.14cd50.894e@mx.google.com>


banach-space wrote:

> Also, since it was just a very minor fix, I thought two approvals would be enough.

Indeed, two is more than enough, from https://llvm.org/docs/CodeReview.html#lgtm-how-a-patch-is-accepted:
> Only approval from a single reviewer is required

My request was also to the reviewers ;-)

> Edit: I updated the body < 3

Thank you, but please note that commit messages cannot be updated after merging and most people rely on commit summaries for identifying context rather than PRs.

Anyway, onwards and upwards :)

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Thu Jul  9 09:37:10 2026
From: llvmlistbot at llvm.org (Umang Yadav)
Date: Thu, 09 Jul 2026 09:37:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ROCm] Build mlir_rocm_runtime
	STANDALONE (PR #205651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205651@github.com>
Message-ID: <6a4fce36.a87eb5dc.7d817.f788@mx.google.com>


https://github.com/umangyadav updated https://github.com/llvm/llvm-project/pull/205651

>From 20c637dcf10214aea7a424c728ca2c17e69680b1 Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Wed, 24 Jun 2026 19:15:48 +0000
Subject: [PATCH 1/4] [mlir] Build mlir_rocm_runtime STANDALONE

Mark mlir_rocm_runtime STANDALONE so it no longer links LLVMSupport,
mirroring mlir_cuda_runtime, and drop the SmallVector/ArrayRef use in
mgpuMemHostRegisterMemRef in favor of std equivalents. This removes the
DISABLE_PCH_REUSE workaround, which a STANDALONE library no longer needs.

Linking LLVMSupport exported ~1300 default-visibility llvm::* symbols.
When the runtime is dlopen-ed into a process that also reaches ROCm's
libLLVM.so (via the libamdhip64 -> libamd_comgr -> libLLVM.so DT_NEEDED
chain), the dynamic linker interposed these data symbols, causing the two
LLVM instances to share cl::opt / pass-registry / ManagedStatic singletons
and abort during dynamic init (SmallPtrSet.h: Assertion `Bucket < End').

Co-authored-by: Cursor <cursoragent at cursor.com>
---
 mlir/lib/ExecutionEngine/CMakeLists.txt       | 13 +++---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 41 +++++++++++--------
 2 files changed, 31 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/CMakeLists.txt b/mlir/lib/ExecutionEngine/CMakeLists.txt
index 87af4724f159a..3199c88313087 100644
--- a/mlir/lib/ExecutionEngine/CMakeLists.txt
+++ b/mlir/lib/ExecutionEngine/CMakeLists.txt
@@ -403,12 +403,13 @@ if(LLVM_ENABLE_PIC)
 
       EXCLUDE_FROM_LIBMLIR
 
-      # TODO: this is merely a workaround. If this library depends on LLVMSupport,
-      # it should suppress symbols, or if it doesn't, it shouldn't link against
-      # it. This workaround prevents the library from defining the symbol
-      # llvm::EnableABIBreakingChecks, which would cause ODR-violations when
-      # dlopen-ed.
-      DISABLE_PCH_REUSE
+      # This library is dlopen-ed (e.g. by mlir-runner via --shared-libs),
+      # frequently into a process that already hosts another LLVM (ROCm's own
+      # libLLVM.so is reachable through the libamdhip64 -> libamd_comgr
+      # DT_NEEDED chain). Linking LLVMSupport here would duplicate LLVM's
+      # cl::opt / pass-registry / ManagedStatic singletons and abort during
+      # dynamic init. The wrappers don't need LLVMSupport, so build STANDALONE.
+      STANDALONE
     )
 
     # Supress compiler warnings from HIP headers
diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 251245106c56e..86090f410c897 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -13,13 +13,17 @@
 //===----------------------------------------------------------------------===//
 
 #include <cassert>
-#include <numeric>
+#include <cstdint>
+#include <cstdio>
 
 #include "mlir/ExecutionEngine/CRunnerUtils.h"
-#include "llvm/ADT/ArrayRef.h"
 
 #include "hip/hip_runtime.h"
 
+#ifdef _WIN32
+#include <malloc.h>
+#endif // _WIN32
+
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -167,22 +171,25 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-
-  llvm::SmallVector<int64_t, 4> denseStrides(rank);
-  llvm::ArrayRef<int64_t> sizes(descriptor->sizes, rank);
-  llvm::ArrayRef<int64_t> strides(sizes.end(), rank);
-
-  std::partial_sum(sizes.rbegin(), sizes.rend(), denseStrides.rbegin(),
-                   std::multiplies<int64_t>());
-  auto sizeBytes = denseStrides.front() * elementSizeBytes;
-
   // Only densely packed tensors are currently supported.
-  std::rotate(denseStrides.begin(), denseStrides.begin() + 1,
-              denseStrides.end());
-  denseStrides.back() = 1;
-  assert(strides == llvm::ArrayRef(denseStrides));
-
-  auto ptr = descriptor->data + descriptor->offset * elementSizeBytes;
+#ifdef _WIN32
+  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
+#else
+  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
+#endif // _WIN32
+  int64_t *sizes = descriptor->sizes;
+  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
+    denseStrides[i] = runningStride;
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
+  int64_t *strides = &sizes[rank];
+  (void)strides;
+  for (unsigned i = 0; i < rank; ++i)
+    assert(strides[i] == denseStrides[i] &&
+           "Mismatch in computed dense strides");
+
+  auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);
 }
 

>From 722956fc7dd48daeb57b86d768950330555f0b0d Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:07:28 +0000
Subject: [PATCH 2/4] make implementation withotu alloca, we only need to check
 for dense memref, we can keep runningStrides

---
 .../ExecutionEngine/RocmRuntimeWrappers.cpp   | 25 ++++++-------------
 1 file changed, 7 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 86090f410c897..47977a246f0bf 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -20,10 +20,6 @@
 
 #include "hip/hip_runtime.h"
 
-#ifdef _WIN32
-#include <malloc.h>
-#endif // _WIN32
-
 #define HIP_REPORT_IF_ERROR(expr)                                              \
   [](hipError_t result) {                                                      \
     if (!result)                                                               \
@@ -171,23 +167,16 @@ extern "C" void mgpuMemHostRegister(void *ptr, uint64_t sizeBytes) {
 extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
-  // Only densely packed tensors are currently supported.
-#ifdef _WIN32
-  int64_t *denseStrides = (int64_t *)_alloca(rank * sizeof(int64_t));
-#else
-  int64_t *denseStrides = (int64_t *)alloca(rank * sizeof(int64_t));
-#endif // _WIN32
   int64_t *sizes = descriptor->sizes;
-  for (int64_t i = rank - 1, runningStride = 1; i >= 0; i--) {
-    denseStrides[i] = runningStride;
-    runningStride *= sizes[i];
-  }
-  uint64_t sizeBytes = sizes[0] * denseStrides[0] * elementSizeBytes;
   int64_t *strides = &sizes[rank];
-  (void)strides;
-  for (unsigned i = 0; i < rank; ++i)
-    assert(strides[i] == denseStrides[i] &&
+  int64_t runningStride = 1;
+  // Only densely packed tensors are currently supported.
+  for (int64_t i = rank - 1; i >= 0; --i) {
+    assert(strides[i] == runningStride &&
            "Mismatch in computed dense strides");
+    runningStride *= sizes[i];
+  }
+  uint64_t sizeBytes = runningStride * elementSizeBytes;
 
   auto *ptr = descriptor->data + descriptor->offset * elementSizeBytes;
   mgpuMemHostRegister(ptr, sizeBytes);

>From 6cfe8b9772766a10bb465c7c03ab9c942be7ea4c Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 25 Jun 2026 13:13:27 +0000
Subject: [PATCH 3/4] Formatting

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 47977a246f0bf..445518111a244 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -172,8 +172,7 @@ mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {
-    assert(strides[i] == runningStride &&
-           "Mismatch in computed dense strides");
+    assert(strides[i] == runningStride && "Mismatch in computed dense strides");
     runningStride *= sizes[i];
   }
   uint64_t sizeBytes = runningStride * elementSizeBytes;

>From 86ee1a0d656e81ec458bdfa335fa06b7b4fe250b Mon Sep 17 00:00:00 2001
From: Umang Yadav <umayadav at amd.com>
Date: Thu, 9 Jul 2026 12:58:41 +0000
Subject: [PATCH 4/4] add `maybe_unused` for the release builds

---
 mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
index 445518111a244..42018b3722a8a 100644
--- a/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
+++ b/mlir/lib/ExecutionEngine/RocmRuntimeWrappers.cpp
@@ -168,7 +168,7 @@ extern "C" void
 mgpuMemHostRegisterMemRef(int64_t rank, StridedMemRefType<char, 1> *descriptor,
                           int64_t elementSizeBytes) {
   int64_t *sizes = descriptor->sizes;
-  int64_t *strides = &sizes[rank];
+  [[maybe_unused]] int64_t *strides = &sizes[rank];
   int64_t runningStride = 1;
   // Only densely packed tensors are currently supported.
   for (int64_t i = rank - 1; i >= 0; --i) {


From llvmlistbot at llvm.org  Thu Jul  9 09:46:09 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 09 Jul 2026 09:46:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Install missing standalone helper
	tools (PR #205066)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205066@github.com>
Message-ID: <6a4fd051.f8b79514.1d52d0.b84a@mx.google.com>


https://github.com/banach-space approved this pull request.

This is not my area of expertise, so I looked for some reference examples to make sure this is following an established pattern.

For `mlir-irdl-to-cpp`, I found this (also defined via `add_llvm_executable`): https://github.com/llvm/llvm-project/blob/adcd0b736e3b949c71b67c658f4c22242a4f21d1/clang/tools/c-index-test/CMakeLists.txt?plain=1#L53-L71

For `mlir-src-shader`, I have found this (also defined via `add_tablegen`): https://github.com/llvm/llvm-project/blob/adcd0b736e3b949c71b67c658f4c22242a4f21d1/mlir/tools/mlir-tblgen/CMakeLists.txt?plain=1#L8-L10

Based on that, LGTM, thank you and sorry for the delay!

https://github.com/llvm/llvm-project/pull/205066

From llvmlistbot at llvm.org  Thu Jul  9 09:51:39 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Thu, 09 Jul 2026 09:51:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSME] fix f64 scalable matmul
 crashes in `VectorLegalizationPass` (PR #207947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207947@github.com>
Message-ID: <6a4fd19b.481afadf.196d57.b224@mx.google.com>


egebeysel wrote:

@banach-space @FedericoBruzzone apologies, I should've flagged the description as well (like the comments) slipped from me since I did have context of the problem. 

https://github.com/llvm/llvm-project/pull/207947

From llvmlistbot at llvm.org  Thu Jul  9 10:49:36 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 09 Jul 2026 10:49:36 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a4fdf30.7d41f3a1.237e53.5485@mx.google.com>


https://github.com/Saieiei updated https://github.com/llvm/llvm-project/pull/199967

>From 05c79db3253c2d251ef89d9ce9ea4640da2347fd Mon Sep 17 00:00:00 2001
From: Sairudra More <sairudra60 at gmail.com>
Date: Fri, 12 Jun 2026 04:57:07 -0500
Subject: [PATCH] [flang][OpenMP] Lower target in_reduction for host fallback

Enable host-fallback lowering for target in_reduction in Flang and MLIR OpenMP translation.

Model target in_reduction through the matching map entry, force address-preserving implicit mapping for Flang in_reduction list items, and emit the host-side task-reduction lookup with __kmpc_task_reduction_get_th_data. The runtime entry point takes and returns a generic, default-address-space pointer, so normalize a non-default-address-space captured pointer to the generic address space before the call and cast the returned private pointer back to the map block argument's address space, mirroring the in_reduction handling on omp.taskloop. Unsupported device/offload-entry and richer reduction forms remain diagnosed.

Add Flang lowering, MLIR verifier/translation, and LLVM IR tests for the supported host-fallback path, including a non-default-address-space case, and the remaining unsupported cases.
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 103 +++++++++--
 .../OpenMP/target-inreduction-llvmir.f90      |  50 ++++++
 .../OpenMP/Todo/target-inreduction-common.f90 |  20 +++
 .../Todo/target-inreduction-equivalence.f90   |  19 +++
 .../Todo/target-inreduction-firstprivate.f90  |  19 +++
 .../Lower/OpenMP/Todo/target-inreduction.f90  |  15 --
 .../OpenMP/target-inreduction-unused.f90      |  27 +++
 .../test/Lower/OpenMP/target-inreduction.f90  |  30 ++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  33 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  51 ++++++
 .../mlir/Dialect/OpenMP/OpenMPClauses.td      |  14 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  22 ++-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 155 ++++++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 161 ++++++++++++++++--
 .../Dialect/OpenMP/host-op-filtering.mlir     |   8 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  85 +++++++++
 .../openmp-target-in-reduction-device.mlir    |  51 ++++++
 .../openmp-target-in-reduction-multi.mlir     |  77 +++++++++
 .../LLVMIR/openmp-target-in-reduction.mlir    | 107 ++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  91 +++++++++-
 20 files changed, 1059 insertions(+), 79 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction-unused.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 9c6352b73a585..00297e4a48e69 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2474,6 +2474,7 @@ genTargetClauses(lower::AbstractConverter &converter,
                  mlir::omp::TargetExtOperands &clauseOps,
                  DefaultMapsTy &defaultMaps,
                  llvm::SmallVectorImpl<Object> &hasDeviceAddrObjects,
+                 llvm::SmallVectorImpl<Object> &inReductionObjects,
                  llvm::SmallVectorImpl<Object> &isDevicePtrObjects,
                  llvm::SmallVectorImpl<Object> &mapObjects) {
   ClauseProcessor cp(converter, semaCtx, clauses);
@@ -2489,12 +2490,33 @@ genTargetClauses(lower::AbstractConverter &converter,
     hostEvalInfo->collectValues(clauseOps.hostEvalVars);
   }
   cp.processIf(llvm::omp::Directive::OMPD_target, clauseOps);
+  cp.processInReduction(loc, clauseOps, inReductionObjects);
+  // The host fallback for `target in_reduction` redirects each list item's
+  // dedicated implicit map entry to the per-task reduction-private storage
+  // returned by the runtime. A COMMON block member is accessed in the target
+  // body through the COMMON block storage map, not through its own member map
+  // entry, so the redirect would rewrite an unused map argument and the body
+  // would accumulate into the mapped original rather than the private copy.
+  // An EQUIVALENCE-associated variable has the same problem: it shares storage
+  // with another variable that may be mapped separately (e.g. `map(tofrom: j)`
+  // with `equivalence(i, j)`), so the body updates the aliased storage while
+  // the redirected runtime-private copy stays dead.
+  // Reject both cases until the redirect can target the storage map.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      if (semantics::FindCommonBlockContaining(sym->GetUltimate()))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a COMMON block member");
+      if (semantics::FindEquivalenceSet(sym->GetUltimate()))
+        TODO(loc, "TARGET construct with IN_REDUCTION of an EQUIVALENCE "
+                  "storage-associated variable");
+    }
   cp.processIsDevicePtr(stmtCtx, clauseOps, isDevicePtrObjects);
   cp.processMap(loc, stmtCtx, clauseOps, llvm::omp::Directive::OMPD_unknown,
                 &mapObjects);
   cp.processNowait(clauseOps);
   cp.processThreadLimit(stmtCtx, clauseOps);
-  cp.processTODO<clause::Allocate, clause::InReduction, clause::UsesAllocators>(
+  cp.processTODO<clause::Allocate, clause::UsesAllocators>(
       loc, llvm::omp::Directive::OMPD_target);
 
   // `target private(..)` is only supported in delayed privatization mode.
@@ -3556,10 +3578,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   mlir::omp::TargetExtOperands clauseOps;
   DefaultMapsTy defaultMaps;
   llvm::SmallVector<Object> mapObjects, hasDeviceAddrObjects,
-      isDevicePtrObjects;
+      inReductionObjects, isDevicePtrObjects;
   genTargetClauses(converter, semaCtx, symTable, stmtCtx, eval, item->clauses,
                    loc, clauseOps, defaultMaps, hasDeviceAddrObjects,
-                   isDevicePtrObjects, mapObjects);
+                   inReductionObjects, isDevicePtrObjects, mapObjects);
 
   KernelTypeVisitor visitor(semaCtx, converter.getModuleOp());
   clauseOps.kernelType = mlir::omp::TargetExecModeAttr::get(
@@ -3607,6 +3629,26 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
                            /*isTargetPrivitization=*/true);
   dsp.processStep1(&clauseOps);
 
+  // The host fallback for `target in_reduction` requires each list item to be
+  // implicitly mapped as an address-preserving entry whose map argument the
+  // redirect rewrites to the runtime reduction-private storage. When the same
+  // variable is also privatized on the target (implicitly, as with a
+  // host-associated scalar that defaults to firstprivate, or explicitly with
+  // `firstprivate(i) in_reduction(+:i)`), the force-map is suppressed because
+  // the symbol is already in the privatization set, and the firstprivate copy
+  // region then has no mold value for it, aborting translation. Reject until
+  // in_reduction can be reconciled with target privatization.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      const semantics::Symbol &ultimate = sym->GetUltimate();
+      if (llvm::any_of(dsp.getAllSymbolsToPrivatize(),
+                       [&](const semantics::Symbol *priv) {
+                         return priv->GetUltimate() == ultimate;
+                       }))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a privatized variable");
+    }
+
   // Collect symbols that have dynamic substring accesses
   llvm::SmallPtrSet<const semantics::Symbol *, 8> symbolsWithDynamicSubstring;
   collectSymbolsWithDynamicSubstring(semaCtx, eval,
@@ -3617,7 +3659,16 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   // symbols used inside the region that do not have explicit data-environment
   // attribute clauses (neither data-sharing; e.g. `private`, nor `map`
   // clauses).
-  auto captureImplicitMap = [&](const semantics::Symbol &sym) {
+  //
+  // When `forceAddressPreserving` is set, the symbol is force-mapped as an
+  // address-preserving `capture(ByRef)` with implicit `tofrom` flags,
+  // bypassing the scalar default capture rules. This is used for `target
+  // in_reduction` list items, whose mapped pointer is passed as the `orig`
+  // argument of `__kmpc_task_reduction_get_th_data`; a ByCopy scalar capture
+  // would break the runtime lookup against the enclosing taskgroup's
+  // task_reduction descriptor.
+  auto captureImplicitMap = [&](const semantics::Symbol &sym,
+                                bool forceAddressPreserving = false) {
     // Structure component symbols don't have bindings, and can only be
     // explicitly mapped individually. If a member is captured implicitly
     // we map the entirety of the derived type when we find its symbol.
@@ -3626,12 +3677,14 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
     // if the symbol is part of an already mapped common block, do not make a
     // map for it.
-    if (const Fortran::semantics::Symbol *common =
-            Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
-      if (llvm::any_of(mapObjects, [=](const Object &object) {
-            return object.sym() == common;
-          }))
-        return;
+    if (!forceAddressPreserving) {
+      if (const Fortran::semantics::Symbol *common =
+              Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
+        if (llvm::any_of(mapObjects, [=](const Object &object) {
+              return object.sym() == common;
+            }))
+          return;
+    }
 
     // If we come across a symbol without a symbol address, we
     // return as we cannot process it, this is intended as a
@@ -3681,13 +3734,21 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
         eleType = refType.getElementType();
 
       std::pair<mlir::omp::ClauseMapFlags, mlir::omp::VariableCaptureKind>
-          mapFlagAndKind = getImplicitMapTypeAndKind(
-              firOpBuilder, converter, defaultMaps, eleType, loc, sym);
+          mapFlagAndKind;
+      if (forceAddressPreserving)
+        mapFlagAndKind = {mlir::omp::ClauseMapFlags::implicit |
+                              mlir::omp::ClauseMapFlags::to |
+                              mlir::omp::ClauseMapFlags::from,
+                          mlir::omp::VariableCaptureKind::ByRef};
+      else
+        mapFlagAndKind = getImplicitMapTypeAndKind(
+            firOpBuilder, converter, defaultMaps, eleType, loc, sym);
 
       mlir::FlatSymbolRefAttr mapperId;
       auto defaultmapBehaviour = getDefaultmapIfPresent(defaultMaps, eleType);
-      if (defaultmapBehaviour ==
-          clause::Defaultmap::ImplicitBehavior::Default) {
+      if (!forceAddressPreserving &&
+          defaultmapBehaviour ==
+              clause::Defaultmap::ImplicitBehavior::Default) {
         const semantics::DerivedTypeSpec *typeSpec =
             sym.GetType() ? sym.GetType()->AsDerived() : nullptr;
         if (typeSpec) {
@@ -3741,6 +3802,15 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
           Object{const_cast<semantics::Symbol *>(&sym), std::nullopt});
     }
   };
+  // OpenMP requires `in_reduction` list items on `target` to be implicitly
+  // data-mapped. Force-map them as address-preserving captures before the
+  // generic implicit-map walk so that walk treats the symbols as already
+  // mapped via `isDuplicateMappedSymbol` and does not downgrade them to
+  // ByCopy.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym())
+      captureImplicitMap(*sym, /*forceAddressPreserving=*/true);
+
   lower::pft::visitAllSymbols(eval, captureImplicitMap);
 
   auto targetOp = mlir::omp::TargetOp::create(firOpBuilder, loc, clauseOps);
@@ -3753,7 +3823,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   args.hasDeviceAddr.objects = hasDeviceAddrObjects;
   args.hasDeviceAddr.vars = hasDeviceAddrBaseValues;
   args.hostEvalVars = clauseOps.hostEvalVars;
-  // TODO: Add in_reduction syms and vars.
+  // `in_reduction` list items do not get their own entry block argument on
+  // `omp.target`; they are implicitly mapped (see the force-map above) and the
+  // target body accesses them through their `map_entries` block argument. The
+  // `in_reduction` operands remain on the op as host-side metadata.
   args.map.objects = mapObjects;
   args.map.vars = mapBaseValues;
   args.priv.objects = makeObjects(dsp.getDelayedPrivSymbols());
diff --git a/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90 b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
new file mode 100644
index 0000000000000..4b5d951a6dd38
--- /dev/null
+++ b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
@@ -0,0 +1,50 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-version=50 -o - %s | FileCheck %s
+
+! End-to-end coverage: Flang lowers a taskgroup task_reduction enclosing a
+! target in_reduction all the way to LLVM IR. Verifies that the full host-
+! fallback path emits __kmpc_taskred_init for the enclosing taskgroup and
+! __kmpc_task_reduction_get_th_data for the target's in_reduction lookup,
+! with load/store through the returned private pointer and no direct update
+! through the original shared pointer.
+
+subroutine target_in_reduction_e2e()
+  integer :: i
+  i = 0
+  !$omp taskgroup task_reduction(+:i)
+    !$omp target in_reduction(+:i)
+    i = i + 1
+    !$omp end target
+  !$omp end taskgroup
+end subroutine target_in_reduction_e2e
+
+! CHECK-LABEL: define void @target_in_reduction_e2e_()
+! The enclosing taskgroup emits __kmpc_taskred_init to register the
+! task_reduction descriptor.
+! CHECK:         call ptr @__kmpc_taskred_init(i32 %{{.+}}, i32 1, ptr %{{.+}})
+
+! The host stub calls the outlined target body passing the captured pointer.
+! CHECK:         call void @__omp_offloading_{{.*}}_target_in_reduction_e2e_{{.*}}(ptr %{{.+}}, ptr null)
+
+! Inside the outlined target body, the in_reduction private pointer is
+! obtained from the runtime using the captured original pointer with a NULL
+! descriptor (the runtime walks enclosing taskgroups). All loads and stores
+! go through the returned private pointer.
+! CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_in_reduction_e2e_
+! CHECK-SAME:    (ptr %[[ORIG:.+]], ptr %{{.+}})
+! CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+! CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[ORIG]])
+! CHECK:         %[[VAL:.+]] = load i32, ptr %[[PRIV]]
+! CHECK:         %[[SUM:.+]] = add i32 %[[VAL]], 1
+! CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+! The outlined body must not store directly through the captured original
+! pointer; all updates go through the runtime-returned private copy.
+! CHECK-NOT:     store i32 %{{.+}}, ptr %[[ORIG]]
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
new file mode 100644
index 0000000000000..25408105ceabc
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is a COMMON block member is accessed in
+! the target body through the COMMON block storage map, not through its own
+! dedicated member map entry. The host-fallback redirect only rewrites the
+! member map argument, so the runtime-private pointer would be dead and the body
+! would accumulate into the mapped original instead of the reduction-private
+! copy. Until the redirect can target the storage map this case is rejected.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a COMMON block member
+
+subroutine omp_target_in_reduction_common()
+  integer :: i, k
+  common /cb/ i, k
+  i = 0
+  !$omp target map(tofrom: /cb/) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_common
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
new file mode 100644
index 0000000000000..816e76c8eeda9
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is EQUIVALENCE storage-associated
+! shares storage with another variable that may be mapped separately (here `j`).
+! The body updates the aliased storage through `j`'s map while the host-fallback
+! redirect rewrites only `i`'s map argument, leaving the runtime-private copy
+! dead. Reject until the redirect can account for storage association.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of an EQUIVALENCE storage-associated variable
+
+subroutine omp_target_in_reduction_equivalence()
+  integer :: i, j
+  equivalence(i, j)
+  i = 0
+  !$omp target map(tofrom: j) in_reduction(+:i)
+  j = j + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_equivalence
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
new file mode 100644
index 0000000000000..6dda967b82938
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! An explicit `firstprivate(i)` on the same `target` as `in_reduction(+:i)`
+! privatizes the reduction variable. The item is then privatized instead of
+! address-preserving-mapped, so the host-fallback redirect has no dedicated map
+! argument to rewrite and the firstprivate copy region has no mold value, which
+! previously aborted translation. Reject until in_reduction can be reconciled
+! with target privatization.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a privatized variable
+
+subroutine omp_target_in_reduction_firstprivate()
+  integer :: i
+  i = 0
+  !$omp target firstprivate(i) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_firstprivate
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
deleted file mode 100644
index e5a9cffac5a11..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-
-!===============================================================================
-! `mergeable` clause
-!===============================================================================
-
-! CHECK: not yet implemented: Unhandled clause IN_REDUCTION in TARGET construct
-subroutine omp_target_inreduction()
-  integer i
-  i = 0
-  !$omp target in_reduction(+:i)
-  i = i + 1
-  !$omp end target
-end subroutine omp_target_inreduction
diff --git a/flang/test/Lower/OpenMP/target-inreduction-unused.f90 b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
new file mode 100644
index 0000000000000..f9dfa4973610c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
@@ -0,0 +1,27 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Per the OpenMP spec, an in_reduction list item on a target construct is
+! implicitly data-mapped. The lowering must not rely on the variable being
+! referenced inside the target body to discover that map: here `i` only
+! appears in the in_reduction clause and is never read or written inside
+! the region. Verify that an omp.map.info for `i` is still emitted and
+! flows into the omp.target's map_entries.
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction_unused()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reduction_unusedEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@{{[^ ]+}} %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %{{[^ ]+}} : !fir.ref<i32>)
+
+subroutine omp_target_in_reduction_unused()
+  interface
+    subroutine sub()
+    end subroutine
+  end interface
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  call sub()
+  !$omp end target
+end subroutine omp_target_in_reduction_unused
diff --git a/flang/test/Lower/OpenMP/target-inreduction.f90 b/flang/test/Lower/OpenMP/target-inreduction.f90
new file mode 100644
index 0000000000000..74f0f003aab4f
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction.f90
@@ -0,0 +1,30 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Verify that in_reduction on a target construct is lowered to an
+! omp.target with both an in_reduction clause and an implicit map_entries
+! entry for the same variable. The in_reduction clause does not define an
+! entry block argument: inside the target body the variable is accessed
+! through its map_entries block argument. The implicit map also captures the
+! original pointer into the target region so the MLIR -> LLVM IR translation
+! can pass it to __kmpc_task_reduction_get_th_data.
+
+!CHECK-LABEL: omp.declare_reduction
+!CHECK-SAME:  @[[RED_I32_NAME:.*]] : i32 init {
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reductionEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@[[RED_I32_NAME]] %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %[[MAPARG:[^ ]+]] : !fir.ref<i32>)
+!CHECK:         hlfir.declare %[[MAPARG]]
+!CHECK:         omp.terminator
+!CHECK:       }
+
+subroutine omp_target_in_reduction()
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 965ac358c259e..6a81ff8c6a0bb 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivate.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivate   Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivate);
+
   /// Return the insertion point used by the underlying IRBuilder.
   InsertPointTy getInsertionPoint() { return Builder.saveIP(); }
 
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..4dc26bc0530f6 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item. Owning the gtid setup and the lookup loop here keeps the runtime
+  // generation inside OpenMPIRBuilder rather than the MLIR translation layer.
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Gtid = getOrCreateThreadID(Ident);
+
+  // The runtime entry point takes (and returns) a generic, default-address-
+  // space `ptr`. A NULL descriptor makes the runtime walk the enclosing
+  // taskgroups to find the matching task_reduction registration for the item.
+  Type *PtrTy = PointerType::getUnqual(M.getContext());
+  Value *NullDesc = ConstantPointerNull::get(PtrTy);
+  FunctionCallee GetThData =
+      getOrCreateRuntimeFunction(M, OMPRTL___kmpc_task_reduction_get_th_data);
+
+  for (unsigned Idx = 0; Idx < OrigPtrs.size(); ++Idx) {
+    // Normalize a non-default-address-space original pointer to the generic
+    // address space before the call.
+    Value *OrigPtr = OrigPtrs[Idx];
+    if (auto *OrigPtrTy = dyn_cast<PointerType>(OrigPtr->getType());
+        OrigPtrTy && OrigPtrTy->getAddressSpace() != 0)
+      OrigPtr = Builder.CreateAddrSpaceCast(OrigPtr, PtrTy);
+
+    Value *Priv = Builder.CreateCall(GetThData, {Gtid, NullDesc, OrigPtr},
+                                     "omp.inred.priv");
+
+    // Cast the returned private pointer back to the requested address space
+    // when it differs.
+    if (auto *ResPtrTy = dyn_cast<PointerType>(ResultPtrTys[Idx]);
+        ResPtrTy && ResPtrTy->getAddressSpace() != 0)
+      Priv = Builder.CreateAddrSpaceCast(Priv, ResultPtrTys[Idx]);
+
+    MapPrivate(Idx, Priv);
+  }
+  return Builder.saveIP();
+}
+
 OpenMPIRBuilder::InsertPointOrErrorTy
 OpenMPIRBuilder::createBarrier(const LocationDescription &Loc, Directive Kind,
                                bool ForceSimpleCall, bool CheckCancelFlag) {
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
index 54e46b74c5b64..cd1223dc1702c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
@@ -764,9 +764,17 @@ class OpenMP_InReductionClauseSkip<
     OptionalAttr<SymbolRefArrayAttr>:$in_reduction_syms
   );
 
-  // Description varies depending on the operation. Assembly format not defined
-  // because this clause must be processed together with the first region of the
-  // operation, as it defines entry block arguments.
+  // This assembly format should only be used by operations where `in_reduction`
+  // does not define entry block arguments (e.g. `omp.target`). Otherwise, it
+  // must be printed and parsed together with the corresponding region, because
+  // it defines entry block arguments.
+  let optAssemblyFormat = [{
+    `in_reduction` `(`
+      custom<InReductionClause>($in_reduction_vars, type($in_reduction_vars),
+                                $in_reduction_byref, $in_reduction_syms) `)`
+  }];
+
+  // Description varies depending on the operation.
 }
 
 def OpenMP_InReductionClause : OpenMP_InReductionClauseSkip<>;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 5b4ce7cc214bf..fd700e081fa01 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -966,7 +966,8 @@ def TaskOp : OpenMP_Op<"task", traits = [
     OutlineableOpenMPOpInterface
   ], clauses = [
     OpenMP_AffinityClause, OpenMP_AllocateClause, OpenMP_DependClause,
-    OpenMP_FinalClause, OpenMP_IfClause, OpenMP_InReductionClause,
+    OpenMP_FinalClause, OpenMP_IfClause,
+    OpenMP_InReductionClauseSkip<assemblyFormat = true>,
     OpenMP_MergeableClause, OpenMP_PriorityClause, OpenMP_PrivateClause,
     OpenMP_UntiedClause, OpenMP_DetachClause
   ], singleRegion = true> {
@@ -1012,9 +1013,10 @@ def TaskloopContextOp : OpenMP_Op<"taskloop.context", traits = [
     DeclareOpInterfaceMethods<OutlineableOpenMPOpInterface>
   ], clauses = [
     OpenMP_AllocateClause, OpenMP_FinalClause, OpenMP_GrainsizeClause,
-    OpenMP_IfClause, OpenMP_InReductionClause, OpenMP_MergeableClause,
-    OpenMP_NogroupClause, OpenMP_NumTasksClause, OpenMP_PriorityClause,
-    OpenMP_PrivateClause, OpenMP_ReductionClause, OpenMP_UntiedClause
+    OpenMP_IfClause, OpenMP_InReductionClauseSkip<assemblyFormat = true>,
+    OpenMP_MergeableClause, OpenMP_NogroupClause, OpenMP_NumTasksClause,
+    OpenMP_PriorityClause, OpenMP_PrivateClause, OpenMP_ReductionClause,
+    OpenMP_UntiedClause
   ], singleRegion = true> {
   let summary = "OutlinableOpenMPOpInterface wrapper for taskloop construct";
   let description = [{
@@ -1627,6 +1629,15 @@ def TargetOp : OpenMP_Op<"target", traits = [
   ];
 
   let extraClassDeclaration = [{
+    // Override BlockArgOpenMPOpInterface method because `in_reduction` list
+    // items on `omp.target` do not define entry block arguments. The reduction
+    // variable is accessed inside the target body through its matching
+    // `map_entries` block argument; the `in_reduction` operands are kept only
+    // as host-side metadata used to look up the per-task private storage.
+    unsigned numInReductionBlockArgs() {
+      return 0;
+    }
+
     mlir::Value getMappedValueForPrivateVar(unsigned privVarIdx) {
       std::optional<DenseI64ArrayAttr> privateMapIdices = getPrivateMapsAttr();
 
@@ -1651,8 +1662,7 @@ def TargetOp : OpenMP_Op<"target", traits = [
     " | `map_iterated` `(` $map_iterated `:` type($map_iterated) `)`)" # [{
       custom<TargetOpRegion>(
           $region, $has_device_addr_vars, type($has_device_addr_vars),
-          $host_eval_vars, type($host_eval_vars), $in_reduction_vars,
-          type($in_reduction_vars), $in_reduction_byref, $in_reduction_syms,
+          $host_eval_vars, type($host_eval_vars),
           $map_vars, type($map_vars), $private_vars, type($private_vars),
           $private_syms, $private_needs_barrier, $private_maps) attr-dict
     }];
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 41d3c547ca618..8b07f08ebe5eb 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1038,6 +1038,82 @@ static void printDynGroupprivateClause(OpAsmPrinter &printer, Operation *op,
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Parser and printer for in_reduction Clause
+//===----------------------------------------------------------------------===//
+
+/// Parses an `in_reduction` clause for an operation that does not give its
+/// list items entry block arguments (e.g. `omp.target`). The expected format is
+/// a comma-separated list of `[byref] @sym %var` followed by `: types`.
+static ParseResult parseInReductionClause(
+    OpAsmParser &parser,
+    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
+    SmallVectorImpl<Type> &inReductionTypes,
+    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms) {
+  SmallVector<SymbolRefAttr> symbolVec;
+  SmallVector<bool> isByRefVec;
+
+  if (parser.parseCommaSeparatedList([&]() {
+        isByRefVec.push_back(parser.parseOptionalKeyword("byref").succeeded());
+        if (parser.parseAttribute(symbolVec.emplace_back()) ||
+            parser.parseOperand(inReductionVars.emplace_back()))
+          return failure();
+        return success();
+      }))
+    return failure();
+
+  if (parser.parseColon())
+    return failure();
+
+  if (parser.parseCommaSeparatedList(
+          [&]() { return parser.parseType(inReductionTypes.emplace_back()); }))
+    return failure();
+
+  if (inReductionVars.size() != inReductionTypes.size())
+    return failure();
+
+  inReductionByref = makeDenseBoolArrayAttr(parser.getContext(), isByRefVec);
+  SmallVector<Attribute> symbolAttrs(symbolVec.begin(), symbolVec.end());
+  inReductionSyms = ArrayAttr::get(parser.getContext(), symbolAttrs);
+  return success();
+}
+
+/// Prints an `in_reduction` clause for an operation that does not give its list
+/// items entry block arguments (e.g. `omp.target`). Mirrors
+/// `parseInReductionClause`.
+static void printInReductionClause(OpAsmPrinter &p, Operation *op,
+                                   ValueRange inReductionVars,
+                                   TypeRange inReductionTypes,
+                                   DenseBoolArrayAttr inReductionByref,
+                                   ArrayAttr inReductionSyms) {
+  MLIRContext *ctx = op->getContext();
+
+  ArrayAttr syms = inReductionSyms;
+  if (!syms) {
+    SmallVector<Attribute> values(inReductionVars.size(), nullptr);
+    syms = ArrayAttr::get(ctx, values);
+  }
+
+  DenseBoolArrayAttr byref = inReductionByref;
+  if (!byref) {
+    SmallVector<bool> values(inReductionVars.size(), false);
+    byref = DenseBoolArrayAttr::get(ctx, values);
+  }
+
+  llvm::interleaveComma(
+      llvm::zip_equal(inReductionVars, syms.getValue(), byref.asArrayRef()), p,
+      [&p](auto t) {
+        auto [var, sym, isByRef] = t;
+        if (isByRef)
+          p << "byref ";
+        if (sym)
+          p << sym << " ";
+        p << var;
+      });
+  p << " : ";
+  llvm::interleaveComma(inReductionTypes, p);
+}
+
 //===----------------------------------------------------------------------===//
 // Parsers for operations including clauses that define entry block arguments.
 //===----------------------------------------------------------------------===//
@@ -1305,9 +1381,6 @@ static ParseResult parseTargetOpRegion(
     SmallVectorImpl<Type> &hasDeviceAddrTypes,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &hostEvalVars,
     SmallVectorImpl<Type> &hostEvalTypes,
-    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
-    SmallVectorImpl<Type> &inReductionTypes,
-    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &mapVars,
     SmallVectorImpl<Type> &mapTypes,
     llvm::SmallVectorImpl<OpAsmParser::UnresolvedOperand> &privateVars,
@@ -1316,8 +1389,6 @@ static ParseResult parseTargetOpRegion(
   AllRegionParseArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, &privateMaps);
@@ -1568,20 +1639,18 @@ static void printBlockArgRegion(OpAsmPrinter &p, Operation *op, Region &region,
 
 // These parseXyz functions correspond to the custom<Xyz> definitions
 // in the .td file(s).
-static void printTargetOpRegion(
-    OpAsmPrinter &p, Operation *op, Region &region,
-    ValueRange hasDeviceAddrVars, TypeRange hasDeviceAddrTypes,
-    ValueRange hostEvalVars, TypeRange hostEvalTypes,
-    ValueRange inReductionVars, TypeRange inReductionTypes,
-    DenseBoolArrayAttr inReductionByref, ArrayAttr inReductionSyms,
-    ValueRange mapVars, TypeRange mapTypes, ValueRange privateVars,
-    TypeRange privateTypes, ArrayAttr privateSyms, UnitAttr privateNeedsBarrier,
-    DenseI64ArrayAttr privateMaps) {
+static void printTargetOpRegion(OpAsmPrinter &p, Operation *op, Region &region,
+                                ValueRange hasDeviceAddrVars,
+                                TypeRange hasDeviceAddrTypes,
+                                ValueRange hostEvalVars,
+                                TypeRange hostEvalTypes, ValueRange mapVars,
+                                TypeRange mapTypes, ValueRange privateVars,
+                                TypeRange privateTypes, ArrayAttr privateSyms,
+                                UnitAttr privateNeedsBarrier,
+                                DenseI64ArrayAttr privateMaps) {
   AllRegionPrintArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, privateMaps);
@@ -2579,8 +2648,7 @@ LogicalResult TargetUpdateOp::verify() {
 void TargetOp::build(OpBuilder &builder, OperationState &state,
                      const TargetExtOperands &clauses) {
   MLIRContext *ctx = builder.getContext();
-  // TODO Store clauses in op: allocateVars, allocatorVars, inReductionVars,
-  // inReductionByref, inReductionSyms.
+  // TODO Store clauses in op: allocateVars, allocatorVars.
   TargetOp::build(
       builder, state, /*allocate_vars=*/{}, /*allocator_vars=*/{},
       makeArrayAttr(ctx, clauses.dependKinds), clauses.dependVars,
@@ -2588,9 +2656,10 @@ void TargetOp::build(OpBuilder &builder, OperationState &state,
       clauses.device, clauses.dynGroupprivateAccessGroup,
       clauses.dynGroupprivateFallback, clauses.dynGroupprivateSize,
       clauses.hasDeviceAddrVars, clauses.hostEvalVars, clauses.ifExpr,
-      /*in_reduction_vars=*/{}, /*in_reduction_byref=*/nullptr,
-      /*in_reduction_syms=*/nullptr, clauses.isDevicePtrVars, clauses.mapVars,
-      clauses.mapIterated, clauses.nowait, clauses.privateVars,
+      clauses.inReductionVars,
+      makeDenseBoolArrayAttr(ctx, clauses.inReductionByref),
+      makeArrayAttr(ctx, clauses.inReductionSyms), clauses.isDevicePtrVars,
+      clauses.mapVars, clauses.mapIterated, clauses.nowait, clauses.privateVars,
       makeArrayAttr(ctx, clauses.privateSyms), clauses.privateNeedsBarrier,
       clauses.threadLimitVars, /*private_maps=*/nullptr, clauses.kernelType);
 }
@@ -2631,6 +2700,21 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `targetInReductionCapturedBy` in
+/// OpenMPToLLVMIRTranslation.cpp so translation matches what the verifier
+/// accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 LogicalResult TargetOp::verify() {
   if (getKernelType() == TargetExecMode::bare && !isCombined())
     return emitOpError() << "bare kernel requires 'omp.combined'";
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (mapInfo &&
+          targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
+    }
+    if (!captured)
+      return emitOpError() << "in_reduction variable must be captured by a "
+                              "matching map_entries entry";
+  }
+
   return verifyPrivateVarsMapping(*this);
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b2d2ef6d8f770..367be0692054c 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -364,9 +364,40 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       op.emitWarning("hint clause discarded");
   };
   auto checkInReduction = [&todo](auto op, LogicalResult &result) {
-    if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
-        op.getInReductionSyms())
+    if (isa<omp::TargetOp, omp::TaskOp, omp::TaskloopContextOp>(
+            op.getOperation())) {
+      if (auto byrefAttr = op.getInReductionByref()) {
+        for (bool isByRef : *byrefAttr) {
+          if (isByRef) {
+            result = todo("in_reduction with byref modifier");
+            return;
+          }
+        }
+      }
+      if (isa<omp::TargetOp>(op.getOperation())) {
+        if (auto inReductionSyms = op.getInReductionSyms()) {
+          for (auto sym :
+               (*inReductionSyms).template getAsRange<SymbolRefAttr>()) {
+            auto decl =
+                SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                    op, sym);
+            assert(decl &&
+                   "symbol resolution should be guaranteed by the op verifier");
+            if (decl.getInitializerRegion().front().getNumArguments() != 1) {
+              result = todo("in_reduction with two-argument initializer");
+              return;
+            }
+            if (!decl.getCleanupRegion().empty()) {
+              result = todo("in_reduction with cleanup region");
+              return;
+            }
+          }
+        }
+      }
+    } else if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
+               op.getInReductionSyms()) {
       result = todo("in_reduction");
+    }
   };
   auto checkNowait = [&todo](auto op, LogicalResult &result) {
     if (op.getNowait())
@@ -423,14 +454,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
           return;
         }
   };
-  auto checkInReductionByref = [&todo](auto op, LogicalResult &result) {
-    if (auto byrefAttr = op.getInReductionByref())
-      for (bool isByRef : *byrefAttr)
-        if (isByRef) {
-          result = todo("in_reduction with byref modifier");
-          return;
-        }
-  };
   auto checkNumTeams = [&todo](auto op, LogicalResult &result) {
     if (op.hasNumTeamsMultiDim())
       result = todo("num_teams with multi-dimensional values");
@@ -482,7 +505,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
       })
       .Case([&](omp::TaskgroupOp op) {
         checkAllocate(op, result);
@@ -494,7 +517,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
         checkReduction(op, result);
         checkReductionByref(op, result);
       })
@@ -8435,6 +8458,20 @@ getDynGroupprivateFallbackType(omp::FallbackModifierAttr fallbackAttr) {
   llvm_unreachable("unexpected dyn_groupprivate fallback type");
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `omp::TargetOp::verify()` in OpenMPDialect.cpp so
+/// translation matches exactly what the verifier accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 static LogicalResult
 convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
                  LLVM::ModuleTranslation &moduleTranslation) {
@@ -8493,6 +8530,59 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand,
+      // using the same predicate as omp::TargetOp::verify(): exact same value,
+      // or another result of the same defining op. The verifier guarantees at
+      // least one match exists; more than one matching entry is a lowering
+      // ambiguity (the redirect cannot pick which map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (!mapInfo || !targetInReductionCapturedBy(v, mapInfo.getVarPtr()))
+          continue;
+        if (matchIdx)
+          return targetOp.emitError()
+                 << "in_reduction variable on omp.target has multiple matching "
+                    "map_entries entries; the redirect target is ambiguous";
+        matchIdx = idx;
+      }
+      // The verifier guarantees a match, so this must hold. The verifier and
+      // this predicate agree, so a verifier-accepted op never reaches an empty
+      // match here.
+      assert(matchIdx &&
+             "TargetOp verifier guarantees a matching map_entries entry for "
+             "each in_reduction variable");
+      inRedMapArgIdx.push_back(*matchIdx);
+      // The runtime `orig` pointer is the in_reduction operand itself (the
+      // reduction variable the enclosing taskgroup registered), not the map
+      // var_ptr. For the same-defining-op case both point to the same storage,
+      // but the in_reduction operand is the canonical reduction variable.
+      inRedOrigPtrs.push_back(moduleTranslation.lookupValue(v));
+    }
+  }
+
   // For some private variables, the MapsForPrivatizedVariablesPass
   // creates MapInfoOp instances. Go through the private variables and
   // the mapped variables so that during codegeneration we are able
@@ -8568,8 +8658,15 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
         attr.isStringAttribute())
       llvmOutlinedFn->addFnAttr(attr);
 
-    for (auto [arg, mapOp] : llvm::zip_equal(mapBlockArgs, mapVars)) {
-      auto mapInfoOp = cast<omp::MapInfoOp>(mapOp.getDefiningOp());
+    for (auto [idx, arg] : llvm::enumerate(mapBlockArgs)) {
+      // in_reduction list items on omp.target are accessed through their
+      // map_entries block argument, which is redirected below to the per-task
+      // reduction-private storage returned by the runtime. Skip the default
+      // host-value mapping for those block arguments so the write-once
+      // mapValue mapping is free to be set to the private pointer.
+      if (llvm::is_contained(inRedMapArgIdx, idx))
+        continue;
+      auto mapInfoOp = cast<omp::MapInfoOp>(mapVars[idx].getDefiningOp());
       llvm::Value *mapOpValue =
           moduleTranslation.lookupValue(mapInfoOp.getVarPtr());
       moduleTranslation.mapValue(arg, mapOpValue);
@@ -8605,6 +8702,42 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // OpenMPIRBuilder owns the runtime-generation path: it computes the
+      // target body gtid once (so several in_reduction items do not each emit a
+      // __kmpc_global_thread_num) and emits the per-item
+      // __kmpc_task_reduction_get_th_data lookup with the required
+      // address-space casts. The MLIR-specific work stays here: collect, per
+      // item, the type the private pointer must have (the map block argument's
+      // type), and, through the callback, rebind the map block argument that
+      // stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
+      SmallVector<llvm::Type *> inRedResultPtrTys;
+      inRedResultPtrTys.reserve(inRedMapArgIdx.size());
+      for (unsigned mapArgIdx : inRedMapArgIdx)
+        inRedResultPtrTys.push_back(
+            moduleTranslation.convertType(mapBlockArgs[mapArgIdx].getType()));
+
+      llvm::OpenMPIRBuilder::LocationDescription bodyLoc(builder);
+      llvm::OpenMPIRBuilder::InsertPointTy redIP =
+          ompBuilder->createTargetInReduction(
+              bodyLoc, inRedOrigPtrs, inRedResultPtrTys,
+              [&](unsigned idx, llvm::Value *priv) {
+                moduleTranslation.mapValue(mapBlockArgs[inRedMapArgIdx[idx]],
+                                           priv);
+              });
+      builder.restoreIP(redIP);
+    }
+
     LLVM::ModuleTranslation::SaveStack<OpenMPAllocStackFrame> frame(
         moduleTranslation, allocaIP, deallocBlocks);
     llvm::Expected<llvm::BasicBlock *> exitBlock = convertOmpOpRegions(
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
index 015e901126a76..8108a1af31e21 100644
--- a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -348,8 +348,12 @@ module attributes {omp.is_target_device = true} {
       omp.terminator
     }
 
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
-    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+    // The `in_reduction` list item on `omp.target` has no dedicated block
+    // argument; it is captured by a matching `map_entries` entry.
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 : !llvm.ptr) map_entries(%1 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
       omp.terminator
     }
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 2db7f953fa3bb..57270ec0694c3 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3255,6 +3255,91 @@ func.func @omp_target_depend(%data_var: memref<i32>) {
 
 // -----
 
+func.func @omp_target_in_reduction_unresolved(%ptr: !llvm.ptr) {
+  // expected-error @below {{op expected symbol reference @add_f32 to point to a reduction declaration}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_duplicate(%ptr: !llvm.ptr) {
+  // expected-error @below {{op accumulator variable used more than once}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr, @add_f32 %ptr : !llvm.ptr, !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg: i32):
+  %0 = arith.constant 0 : i32
+  omp.yield (%0 : i32)
+}
+combiner {
+^bb1(%arg0: i32, %arg1: i32):
+  %1 = arith.addi %arg0, %arg1 : i32
+  omp.yield (%1 : i32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> i32
+  llvm.atomicrmw add %arg2, %2 monotonic : !llvm.ptr, i32
+  omp.yield
+}
+
+func.func @omp_target_in_reduction_type_mismatch(%mem: memref<1xf32>) {
+  // expected-error @below {{op expected accumulator ('memref<1xf32>') to be the same type as reduction declaration ('!llvm.ptr')}}
+  omp.target kernel_type(generic) in_reduction(@add_i32 %mem : memref<1xf32>) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_missing_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_missing_map(%ptr: !llvm.ptr) {
+  // An in_reduction operand on omp.target has no dedicated block argument; it
+  // is accessed inside the body through a matching map_entries block argument.
+  // With no map_entries entry capturing it, the verifier must reject the op.
+  // expected-error @below {{in_reduction variable must be captured by a matching map_entries entry}}
+  omp.target kernel_type(generic) in_reduction(@add_missing_map_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
 func.func @omp_distribute_schedule(%chunk_size : i32, %lb : i32, %ub : i32, %step : i32) -> () {
   // expected-error @below {{op chunk size set without dist_schedule_static being present}}
   "omp.distribute"(%chunk_size) <{operandSegmentSizes = array<i32: 0, 0, 1, 0>}> ({
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
new file mode 100644
index 0000000000000..8cb719c43e34e
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
@@ -0,0 +1,51 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// in_reduction on omp.target compiled for the target *device*. From the
+// device's perspective an in_reduction list item behaves as a regular mapped
+// variable: the host-only redirect to the per-task reduction-private storage
+// (the __kmpc_global_thread_num + __kmpc_task_reduction_get_th_data lookup) is
+// guarded by !isTargetDevice and is therefore not emitted. Verify the device
+// body accesses the mapped variable directly and that none of the host-fallback
+// runtime lookup is present.
+
+module attributes {dlti.dl_spec = #dlti.dl_spec<"dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  omp.declare_reduction @add_i32 : i32
+  init {
+  ^bb0(%arg0: i32):
+    %c0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%c0 : i32)
+  }
+  combiner {
+  ^bb0(%arg0: i32, %arg1: i32):
+    %s = llvm.add %arg0, %arg1 : i32
+    omp.yield(%s : i32)
+  }
+
+  llvm.func @target_inreduction_device(%x : !llvm.ptr) {
+    %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+      %v = llvm.load %marg : !llvm.ptr -> i32
+      %c1 = llvm.mlir.constant(1 : i32) : i32
+      %s = llvm.add %v, %c1 : i32
+      llvm.store %s, %marg : i32, !llvm.ptr
+      omp.terminator
+    }
+    llvm.return
+  }
+}
+
+// The device kernel receives the mapped pointer as a captured argument and
+// accesses the in_reduction variable directly through it.
+// CHECK-LABEL: define {{.*}}amdgpu_kernel void @__omp_offloading_{{.*}}_target_inreduction_device_
+// CHECK-SAME:    (ptr %[[ARG:.+]], ptr %{{.+}})
+// CHECK:         store ptr %[[ARG]], ptr %[[ALLOCA:.+]]
+// CHECK:         %[[MAPPED:.+]] = load ptr, ptr %[[ALLOCA]]
+// The host-only in_reduction redirect must not run on the device: there is no
+// gtid lookup, no __kmpc_task_reduction_get_th_data call and no private copy.
+// CHECK-NOT:     @__kmpc_global_thread_num
+// CHECK-NOT:     @__kmpc_task_reduction_get_th_data
+// CHECK-NOT:     omp.inred.priv
+// The body reads and writes the mapped variable in place.
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[MAPPED]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[MAPPED]]
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
new file mode 100644
index 0000000000000..b438888b6c815
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Multiple in_reduction items on omp.target. Each item is captured into the
+// target region through its own map_entries entry and accessed inside the
+// body via the corresponding map_entries block argument. For the host
+// fallback path every item performs an independent
+// __kmpc_task_reduction_get_th_data lookup using its own captured original
+// pointer, and the returned per-task private pointer is bound to that item's
+// map block argument. This test pins down the pairing so it cannot pass if the
+// two items were swapped or collapsed onto a single pointer.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_multi(%x : !llvm.ptr, %y : !llvm.ptr) {
+  %mx = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %my = omp.map.info var_ptr(%y : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x, @add_i32 %y : !llvm.ptr, !llvm.ptr)
+      map_entries(%mx -> %mxarg, %my -> %myarg : !llvm.ptr, !llvm.ptr) {
+    // First item (x): load, += 1, store back.
+    %vx = llvm.load %mxarg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %sx = llvm.add %vx, %c1 : i32
+    llvm.store %sx, %mxarg : i32, !llvm.ptr
+    // Second item (y): load, += 2, store back.
+    %vy = llvm.load %myarg : !llvm.ptr -> i32
+    %c2 = llvm.mlir.constant(2 : i32) : i32
+    %sy = llvm.add %vy, %c2 : i32
+    llvm.store %sy, %myarg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards both captured pointers into the outlined target
+// kernel (the trailing argument is the unused descriptor slot).
+// CHECK-LABEL: define void @target_inreduction_multi(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_multi_{{.*}}(ptr %{{.+}}, ptr %{{.+}}, ptr null)
+
+// The two captured original pointers arrive as distinct kernel arguments.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_multi_
+// CHECK-SAME:    (ptr %[[CAPTX:.+]], ptr %[[CAPTY:.+]], ptr %{{.+}})
+
+// A single gtid is computed once for the whole target body and shared by both
+// lookups; each item then performs its own __kmpc_task_reduction_get_th_data
+// call against its own captured pointer, and the returned per-task private
+// pointer is bound to that item's map block argument.
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIVX:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTX]])
+// CHECK:         %[[PRIVY:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTY]])
+
+// The first item's private storage is the base of the +1 load/store; the
+// CHECK-NOT below ensures the second item's pointer is not touched until the
+// first item's accumulation has completed (i.e. the items are not swapped or
+// merged onto a single private pointer).
+// CHECK:         %[[LX:.+]] = load i32, ptr %[[PRIVX]]
+// CHECK:         %[[SX:.+]] = add i32 %[[LX]], 1
+// CHECK-NOT:     %[[PRIVY]]
+// CHECK:         store i32 %[[SX]], ptr %[[PRIVX]]
+
+// The second item's private storage is the base of the +2 load/store.
+// CHECK:         %[[LY:.+]] = load i32, ptr %[[PRIVY]]
+// CHECK:         %[[SY:.+]] = add i32 %[[LY]], 2
+// CHECK:         store i32 %[[SY]], ptr %[[PRIVY]]
+
+// Exactly two reduction lookups are emitted; no third call sneaks in. The
+// `call` form is used so this does not match the runtime declaration.
+// CHECK-NOT:     call ptr @__kmpc_task_reduction_get_th_data
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
new file mode 100644
index 0000000000000..124c04dddedda
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+// in_reduction on omp.target: the in_reduction variable is also captured
+// into the target region as a map entry (a frontend is expected to emit this
+// implicit map). The in_reduction clause does not define an entry block
+// argument; inside the target body the variable is accessed through its
+// map_entries block argument. The captured pointer is passed to
+// __kmpc_task_reduction_get_th_data with a NULL descriptor; the runtime
+// walks enclosing taskgroups to locate the matching task_reduction
+// registration. The returned per-task private pointer is bound to the
+// map_entries block argument so subsequent loads/stores inside the region
+// use the private copy.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+    %v = llvm.load %marg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards the captured pointer into the outlined target
+// kernel.
+// CHECK-LABEL: define void @target_inreduction(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_{{.*}}(ptr %{{.+}}, ptr null)
+
+// In the outlined target body the in_reduction private pointer is
+// obtained from the runtime using the captured original pointer; that
+// pointer is then the base of the load and store inside the region.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_
+// CHECK-SAME:    (ptr %[[CAPT:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPT]])
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[PRIV]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+// -----
+
+// Same as the first case but the in_reduction variable lives in a non-default
+// address space (addrspace(1)). __kmpc_task_reduction_get_th_data is declared
+// to take and return a generic (default-AS) `ptr`, so the host-fallback
+// lowering must (1) addrspacecast the captured addrspace(1) original pointer to
+// generic `ptr` before the runtime lookup, and (2) addrspacecast the returned
+// generic private pointer back to addrspace(1) so the body's load/store use the
+// private copy in the right address space. This pins down the address-space
+// handling so a regression that passed the addrspace(1) pointer straight into
+// the runtime call (a bad-signature crash) cannot pass.
+
+omp.declare_reduction @add_i32_as1 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_as1(%x : !llvm.ptr<1>) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr<1>, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr<1>
+  omp.target kernel_type(generic) in_reduction(@add_i32_as1 %x : !llvm.ptr<1>) map_entries(%m -> %marg : !llvm.ptr<1>) {
+    %v = llvm.load %marg : !llvm.ptr<1> -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr<1>
+    omp.terminator
+  }
+  llvm.return
+}
+
+// In the outlined target body the addrspace(1) original pointer is normalized
+// to generic `ptr` before the NULL-descriptor in_reduction lookup, and the
+// returned generic private pointer is cast back to addrspace(1) for the body's
+// load and store. The original captured pointer arrives as the addrspace(1)
+// kernel argument and is only used to derive the generic lookup pointer.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_as1_
+// CHECK-SAME:    (ptr addrspace(1) %[[CAPT_AS1:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID_AS1:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[ORIG_GEN:.+]] = addrspacecast ptr addrspace(1) %[[CAPT_AS1]] to ptr
+// CHECK:         %[[PRIV_AS1_GEN:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID_AS1]], ptr null, ptr %[[ORIG_GEN]])
+// CHECK:         %[[PRIV_AS1:.+]] = addrspacecast ptr %[[PRIV_AS1_GEN]] to ptr addrspace(1)
+// CHECK:         %[[LOADED_AS1:.+]] = load i32, ptr addrspace(1) %[[PRIV_AS1]]
+// CHECK:         %[[SUM_AS1:.+]] = add i32 %[[LOADED_AS1]], 1
+// CHECK:         store i32 %[[SUM_AS1]], ptr addrspace(1) %[[PRIV_AS1]]
+
+// The body must not load or store through the original captured addrspace(1)
+// pointer; all accesses go through the runtime-returned private copy.
+// CHECK-NOT:     store i32 %{{.+}}, ptr addrspace(1) %[[CAPT_AS1]]
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..943b5bad6970f 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -218,10 +218,95 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @target_in_reduction(%x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause in_reduction in omp.target operation}}
+llvm.func @target_in_reduction_byref(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with byref modifier in omp.target operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.target}}
-  omp.target kernel_type(generic) in_reduction(@add_f32 %x -> %prv : !llvm.ptr) {
+  omp.target kernel_type(generic) in_reduction(byref @add_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_cleanup_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+cleanup {
+^bb2(%arg2: f32):
+  omp.yield
+}
+llvm.func @target_in_reduction_cleanup(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with cleanup region in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_cleanup_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_two_arg_init_i32 : !llvm.ptr alloc {
+^bb0(%arg: !llvm.ptr):
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+  omp.yield(%1 : !llvm.ptr)
+} init {
+^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.store %0, %arg1 : i32, !llvm.ptr
+  omp.yield(%arg1 : !llvm.ptr)
+} combiner {
+^bb1(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.load %arg0 : !llvm.ptr -> i32
+  %1 = llvm.load %arg1 : !llvm.ptr -> i32
+  %2 = llvm.add %0, %1 : i32
+  llvm.store %2, %arg0 : i32, !llvm.ptr
+  omp.yield(%arg0 : !llvm.ptr)
+}
+llvm.func @target_in_reduction_two_arg_init(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with two-argument initializer in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_two_arg_init_i32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_dup_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb0(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+llvm.func @target_in_reduction_duplicate_map(%x : !llvm.ptr) {
+  // The in_reduction variable %x has two matching map_entries entries for the
+  // same var_ptr. The translation cannot disambiguate which map block argument
+  // to redirect, so it must reject this as ambiguous.
+  %m1 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %m2 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{in_reduction variable on omp.target has multiple matching map_entries entries; the redirect target is ambiguous}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_dup_map_f32 %x : !llvm.ptr) map_entries(%m1 -> %arg1, %m2 -> %arg2 : !llvm.ptr, !llvm.ptr) {
     omp.terminator
   }
   llvm.return


From llvmlistbot at llvm.org  Thu Jul  9 10:53:56 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 09 Jul 2026 10:53:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a4fe034.bb839515.27d21e.9b17@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
----------------
steplong wrote:

This is what claude is telling me:
> The vector.contract inside vector.mask has a 6D iterator type (1x1x1x[8]x[8]x1), and without reduction_to_contract running first, the cast_away_vector_leading_one_dim pattern doesn't strip the leading 1s off the vector.contract (because that stripping is tied to how reduction_to_contract rewrites the op's iterator
  types).
CombineContractBroadcastMask absorbs vector.broadcast ops that feed into vector.contract operands and compresses the unused iterator dims — which is what reduces the
  contract from 6D (1x1x1x[8]x[8]x1) down to 2D ([8]x[8]), shrinking the mask shape accordingly.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Thu Jul  9 11:47:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 11:47:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1bd6392 - [flang][acc] Add
 `OutlineIdentityOperandOpInterface` to hlfir/fir declare (#208091)
Message-ID: <6a4feca5.58bdd12c.19c81.7ffb@mx.google.com>


Author: Susan Tan (ス-ザン　タン)
Date: 2026-07-09T14:46:53-04:00
New Revision: 1bd63928e4fcf575d93aaef220c19ca41965e015

URL: https://github.com/llvm/llvm-project/commit/1bd63928e4fcf575d93aaef220c19ca41965e015
DIFF: https://github.com/llvm/llvm-project/commit/1bd63928e4fcf575d93aaef220c19ca41965e015.diff

LOG: [flang][acc] Add `OutlineIdentityOperandOpInterface` to hlfir/fir declare (#208091)

Inlining can leave a `fir.declare`'s dummy_scope operand (`!fir.dscope`)
defined outside an OpenACC compute region while the declare itself gets
rematerialized inside, since `fir.dummy_scope` wasn't registered as an
OutlineRematerializationOpInterface candidate.

This dangling, unmappable live-in is illegal for parallelization. 

Fix: Added `OutlineIdentityOperandOpInterface`, implemented by
`fir.declare/hlfir.declare`, so `OffloadLiveInValueCanonicalization`
drops their dummy_scope operand (an identity token that must never be
duplicated) instead of cloning it when sinking/rematerializing them into
offload regions.

Added: 
    

Modified: 
    flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
    flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
    flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
    flang/test/Fir/OpenACC/offload-livein-value-canonicalization.fir
    mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
    mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp

Removed: 
    


################################################################################
diff  --git a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
index a39d517412c07..31e1593e11325 100644
--- a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
+++ b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
@@ -99,6 +99,20 @@ struct OutlineRematerializationModel<fir::ConvertOp>
   bool isRematerializationCandidate(mlir::Operation *op) const;
 };
 
+/// External model for OutlineIdentityOperandOpInterface.
+/// Drops the dummy_scope operand (an identity token that must not be
+/// duplicated) from a [hl]fir.declare instance that has been sunk or
+/// rematerialized into an offload region. Alias analysis falls back to a
+/// dominance-based lookup of the enclosing dummy scope when this operand is
+/// absent, so correctness is preserved without needing to keep or clone the
+/// original operand.
+template <typename Op>
+struct OutlineIdentityOperandDeclareModel
+    : public mlir::acc::OutlineIdentityOperandOpInterface::ExternalModel<
+          OutlineIdentityOperandDeclareModel<Op>, Op> {
+  void dropOutlinedIdentityOperands(mlir::Operation *op) const;
+};
+
 /// External model for OffloadRegionOpInterface.
 /// This interface marks operations whose regions are targets for offloading
 /// and outlining.

diff  --git a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
index 590619a31a33f..8e4b55fe5f4a8 100644
--- a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
@@ -143,6 +143,26 @@ bool OutlineRematerializationModel<
          fir::ConvertOp::isIntegerCompatible(outTy);
 }
 
+template <>
+void OutlineIdentityOperandDeclareModel<
+    fir::DeclareOp>::dropOutlinedIdentityOperands(mlir::Operation *op) const {
+  auto declareOp = mlir::cast<fir::DeclareOp>(op);
+  if (declareOp.getDummyScope()) {
+    declareOp.getDummyScopeMutable().clear();
+    declareOp->removeAttr(declareOp.getDummyArgNoAttrName());
+  }
+}
+
+template <>
+void OutlineIdentityOperandDeclareModel<
+    hlfir::DeclareOp>::dropOutlinedIdentityOperands(mlir::Operation *op) const {
+  auto declareOp = mlir::cast<hlfir::DeclareOp>(op);
+  if (declareOp.getDummyScope()) {
+    declareOp.getDummyScopeMutable().clear();
+    declareOp->removeAttr(declareOp.getDummyArgNoAttrName());
+  }
+}
+
 // Helper to recursively process address-of operations in derived type
 // descriptors and collect all needed fir.globals.
 static void processAddrOfOpInDerivedTypeDescriptor(

diff  --git a/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp b/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
index 070208d22ba05..7e5eeda1f5de9 100644
--- a/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/RegisterOpenACCExtensions.cpp
@@ -53,6 +53,8 @@ void registerOpenACCExtensions(mlir::DialectRegistry &registry) {
         PartialEntityAccessModel<fir::CoordinateOp>>(*ctx);
     fir::DeclareOp::attachInterface<PartialEntityAccessModel<fir::DeclareOp>>(
         *ctx);
+    fir::DeclareOp::attachInterface<
+        OutlineIdentityOperandDeclareModel<fir::DeclareOp>>(*ctx);
 
     fir::AddrOfOp::attachInterface<AddressOfGlobalModel>(*ctx);
     fir::GlobalOp::attachInterface<GlobalVariableModel>(*ctx);
@@ -94,6 +96,8 @@ void registerOpenACCExtensions(mlir::DialectRegistry &registry) {
             PartialEntityAccessModel<hlfir::DesignateOp>>(*ctx);
         hlfir::DeclareOp::attachInterface<
             PartialEntityAccessModel<hlfir::DeclareOp>>(*ctx);
+        hlfir::DeclareOp::attachInterface<
+            OutlineIdentityOperandDeclareModel<hlfir::DeclareOp>>(*ctx);
       });
 
   // Register CUF operation interfaces

diff  --git a/flang/test/Fir/OpenACC/offload-livein-value-canonicalization.fir b/flang/test/Fir/OpenACC/offload-livein-value-canonicalization.fir
index e692e5fb6dc4f..8b1d4cd0b1e16 100644
--- a/flang/test/Fir/OpenACC/offload-livein-value-canonicalization.fir
+++ b/flang/test/Fir/OpenACC/offload-livein-value-canonicalization.fir
@@ -467,3 +467,75 @@ func.func @test_undef_slice_rematerialize(%arg0: !fir.box<!fir.array<?x?xf32>>)
 // CHECK:   %[[UNDEF_INNER:.*]] = fir.undefined index
 // CHECK:   %[[SLICE_INNER:.*]] = fir.slice {{.*}}, %[[UNDEF_INNER]], %[[UNDEF_INNER]]
 // CHECK:   fir.rebox %arg0 [%[[SLICE_INNER]]]
+
+// -----
+
+// Test that fir.declare's dummy_scope operand is dropped, not duplicated,
+// when the declare is sunk into an offload region. Regression test: after
+// inlining, a fir.declare for a dummy argument aliasing a device-valid
+// global (e.g. one with !$ACC DECLARE COPYIN) is sunk/rematerialized into
+// the offload region. Its dummy_scope operand is an identity token (see
+// fir.dummy_scope) that must never be duplicated, so it must be dropped
+// from the sunk/cloned declare instead of being left as an illegal live-in
+// or cloned alongside it.
+
+fir.global @global_for_dummy_scope {acc.declare = #acc.declare<dataClause = acc_copyin>} : i32 {
+  %0 = arith.constant 0 : i32
+  fir.has_value %0 : i32
+}
+
+func.func private @use_ref(!fir.ref<i32>) -> ()
+
+func.func @test_dummy_scope_sink() {
+  %addr = fir.address_of(@global_for_dummy_scope) : !fir.ref<i32>
+  %scope = fir.dummy_scope : !fir.dscope
+  %decl = fir.declare %addr dummy_scope %scope {uniq_name = "global"} : (!fir.ref<i32>, !fir.dscope) -> !fir.ref<i32>
+  acc.serial {
+    fir.call @use_ref(%decl) : (!fir.ref<i32>) -> ()
+    acc.yield
+  }
+  return
+}
+
+// CHECK-LABEL: @test_dummy_scope_sink
+// The dummy scope stays outside, untouched and unduplicated: nothing
+// references it anymore once the declare's operand is dropped.
+// CHECK: fir.dummy_scope
+// CHECK: acc.serial {
+// CHECK-DAG:   fir.address_of(@global_for_dummy_scope)
+// CHECK-DAG:   %[[DECL:.*]] = fir.declare %{{.*}} {uniq_name = "global"} : (!fir.ref<i32>) -> !fir.ref<i32>
+// CHECK:   fir.call @use_ref(%[[DECL]])
+
+// -----
+
+// Test fir.declare rematerialization (used both inside and outside region):
+// the outer declare keeps its dummy_scope untouched, while the clone made
+// for the region drops it rather than duplicating the scope.
+
+fir.global @global_for_dummy_scope_remat {acc.declare = #acc.declare<dataClause = acc_copyin>} : i32 {
+  %0 = arith.constant 0 : i32
+  fir.has_value %0 : i32
+}
+
+func.func private @use_ref(!fir.ref<i32>) -> ()
+
+func.func @test_dummy_scope_rematerialize() {
+  %addr = fir.address_of(@global_for_dummy_scope_remat) : !fir.ref<i32>
+  %scope = fir.dummy_scope : !fir.dscope
+  %decl = fir.declare %addr dummy_scope %scope {uniq_name = "global"} : (!fir.ref<i32>, !fir.dscope) -> !fir.ref<i32>
+  fir.call @use_ref(%decl) : (!fir.ref<i32>) -> ()
+  acc.serial {
+    fir.call @use_ref(%decl) : (!fir.ref<i32>) -> ()
+    acc.yield
+  }
+  return
+}
+
+// CHECK-LABEL: @test_dummy_scope_rematerialize
+// CHECK: %[[SCOPE:.*]] = fir.dummy_scope
+// CHECK: %[[DECL_OUTER:.*]] = fir.declare %{{.*}} dummy_scope %[[SCOPE]] {uniq_name = "global"}
+// CHECK: fir.call @use_ref(%[[DECL_OUTER]])
+// CHECK: acc.serial {
+// CHECK-DAG:   fir.address_of(@global_for_dummy_scope_remat)
+// CHECK-DAG:   %[[DECL_INNER:.*]] = fir.declare %{{.*}} {uniq_name = "global"} : (!fir.ref<i32>) -> !fir.ref<i32>
+// CHECK:   fir.call @use_ref(%[[DECL_INNER]])

diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index a42bb7a66aae7..fa44fe7a49158 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -141,6 +141,28 @@ def OutlineRematerializationOpInterface : OpInterface<"OutlineRematerializationO
   }];
 }
 
+def OutlineIdentityOperandOpInterface
+    : OpInterface<"OutlineIdentityOperandOpInterface"> {
+  let cppNamespace = "::mlir::acc";
+
+  let description = [{
+    An interface for operations with an "identity" operand (e.g. FIR's
+    dummy scope handle) that must never be duplicated, unlike operands
+    handled by `OutlineRematerializationOpInterface`. When such an
+    operation is sunk/cloned into an offload region and the operand would
+    otherwise reference a value outside it, drop the operand instead of
+    cloning it; implementations must preserve correctness some other way
+    (e.g. FIR's dominance-based scope lookup fallback).
+  }];
+
+  let methods = [
+    InterfaceMethod<"Drop any identity operands from this operation "
+                      "instance that reference values defined outside its "
+                      "containing region.",
+                      "void", "dropOutlinedIdentityOperands", (ins), [{}]>,
+  ];
+}
+
 def OffloadRegionOpInterface : OpInterface<"OffloadRegionOpInterface"> {
   let cppNamespace = "::mlir::acc";
 

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
index 60285c8a02bb9..4b57354b8b5d8 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/OffloadLiveInValueCanonicalization.cpp
@@ -246,6 +246,13 @@ class OffloadLiveInValueCanonicalization
       Operation *sinkOp = sinkCandidate.getDefiningOp();
       assert(sinkOp && "must have op to be considered");
       sinkOp->moveBefore(&region.front().front());
+      // Some operations (e.g. fir.declare) may carry identity operands that
+      // must not be left referencing a value defined outside the region.
+      // Unlike other operands, such identity operands cannot simply be
+      // treated as new live-ins to legalize; they must be dropped instead.
+      if (auto identityOp =
+              dyn_cast<acc::OutlineIdentityOperandOpInterface>(sinkOp))
+        identityOp.dropOutlinedIdentityOperands();
       LLVM_DEBUG(llvm::dbgs() << "\t\tSunk: " << *sinkOp << "\n");
     }
 
@@ -262,6 +269,14 @@ class OffloadLiveInValueCanonicalization
     computeTopologicalSorting(opsToRematerialize);
     for (Operation *rematerializationOp : opsToRematerialize) {
       Operation *clonedOp = builder.clone(*rematerializationOp);
+      // See the comment above: identity operands must be dropped from the
+      // clone rather than duplicated, since duplicating them would make the
+      // clone appear to belong to a 
diff erent instantiation than the
+      // original operation (and any of its siblings left outside the
+      // region).
+      if (auto identityOp =
+              dyn_cast<acc::OutlineIdentityOperandOpInterface>(clonedOp))
+        identityOp.dropOutlinedIdentityOperands();
       for (auto [oldResult, newResult] : llvm::zip(
                rematerializationOp->getResults(), clonedOp->getResults())) {
         replaceAllUsesInRegionWith(oldResult, newResult, region);


        

From llvmlistbot at llvm.org  Thu Jul  9 11:47:01 2026
From: llvmlistbot at llvm.org (Susan Tan =?utf-8?b?44K5LeOCtuODs+OAgOOCv+ODsw==?=)
Date: Thu, 09 Jul 2026 11:47:01 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Add
 `OutlineIdentityOperandOpInterface` to hlfir/fir declare (PR #208091)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208091@github.com>
Message-ID: <6a4feca5.743400b0.2067c9.062f@mx.google.com>


https://github.com/SusanTan closed https://github.com/llvm/llvm-project/pull/208091

From llvmlistbot at llvm.org  Thu Jul  9 11:49:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 11:49:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4aba9d2 - [mlir][tosa] Fold tiled input into
 binary elementwise operations (#203941)
Message-ID: <6a4fed4c.70920dee.290dca.77d0@mx.google.com>


Author: Luke Hutton
Date: 2026-07-09T19:49:44+01:00
New Revision: 4aba9d237a1ad07751fff354a56f4f2cf9dfecfe

URL: https://github.com/llvm/llvm-project/commit/4aba9d237a1ad07751fff354a56f4f2cf9dfecfe
DIFF: https://github.com/llvm/llvm-project/commit/4aba9d237a1ad07751fff354a56f4f2cf9dfecfe.diff

LOG: [mlir][tosa] Fold tiled input into binary elementwise operations (#203941)

Canonicalizes explicit broadcasting into implicit broadcasting by
folding the tile input into the binary element-wise operation.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
    mlir/test/Dialect/Tosa/canonicalize.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 5e5760e33b5f6..e3b74f5a07d3f 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2485,6 +2485,7 @@ def Tosa_TileOp : Tosa_InferShapedTypeOp<"tile", [Pure]> {
     LogicalResult getConstantMultiples(llvm::SmallVector<int64_t> &multiples);
   }];
 
+  let hasCanonicalizer = 1;
   let hasFolder = 1;
   let hasVerifier = 1;
 

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 4c98aca0ee8d7..19f35757bdd83 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -20,8 +20,10 @@
 #include "mlir/Dialect/Traits.h"
 #include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/IRMapping.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/FoldUtils.h"
 #include "mlir/Transforms/InliningUtils.h"
 #include "llvm/ADT/APFloat.h"
@@ -2326,6 +2328,82 @@ OpFoldResult tosa::SelectOp::fold(FoldAdaptor adaptor) {
   return {};
 }
 
+static LogicalResult verifyTileIsBroadcast(tosa::TileOp tileOp) {
+  const auto inputType =
+      dyn_cast<RankedTensorType>(tileOp.getInput1().getType());
+  const auto outputType = dyn_cast<RankedTensorType>(tileOp.getType());
+  if (!inputType || !outputType)
+    return failure();
+
+  SmallVector<int64_t> multiples;
+  if (failed(tileOp.getConstantMultiples(multiples)))
+    return failure();
+
+  for (const auto [index, multiple] : llvm::enumerate(multiples)) {
+    if (multiple == 1)
+      continue;
+    if (outputType.isDynamicDim(index))
+      return failure();
+    if (inputType.getDimSize(index) != 1)
+      return failure();
+  }
+
+  return success();
+}
+
+struct RemoveBroadcastTileFromBinaryElementwise
+    : public OpRewritePattern<tosa::TileOp> {
+  using OpRewritePattern<tosa::TileOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(tosa::TileOp tileOp,
+                                PatternRewriter &rewriter) const override {
+    Value tileOutput = tileOp.getOutput();
+    if (!tileOutput.hasOneUse())
+      return rewriter.notifyMatchFailure(tileOp,
+                                         "tile output must have one use");
+
+    Operation *user = *tileOutput.user_begin();
+    const bool isBinaryElementwise =
+        user->getNumOperands() == 2 &&
+        user->hasTrait<OpTrait::tosa::TosaElementwiseOperator>();
+    if (!isBinaryElementwise && !isa<tosa::MulOp>(user))
+      return rewriter.notifyMatchFailure(
+          tileOp, "consumer must be binary broadcastable");
+
+    if (failed(verifyTileIsBroadcast(tileOp)))
+      return rewriter.notifyMatchFailure(
+          tileOp, "tile must only expand statically-known singleton dims");
+
+    Value lhsOperand = user->getOperand(0);
+    Value rhsOperand = user->getOperand(1);
+    Value otherOperand = lhsOperand == tileOutput ? rhsOperand : lhsOperand;
+    Value tileInput = tileOp.getInput1();
+
+    const ShapedType newOtherType = cast<ShapedType>(otherOperand.getType());
+    const ShapedType newTileType = cast<ShapedType>(tileInput.getType());
+    SmallVector<int64_t> broadcastedShape;
+    OpTrait::util::getBroadcastedShape(
+        newOtherType.getShape(), newTileType.getShape(), broadcastedShape);
+
+    const ShapedType outputType = cast<ShapedType>(user->getResultTypes()[0]);
+    if (!llvm::equal(broadcastedShape, outputType.getShape()))
+      return rewriter.notifyMatchFailure(
+          tileOp, "tile output must be broadcastable to consumer operands");
+
+    rewriter.setInsertionPoint(user);
+    IRMapping mapper;
+    mapper.map(tileOutput, tileOp.getInput1());
+    Operation *newUser = rewriter.clone(*user, mapper);
+    rewriter.replaceOp(user, newUser->getResults());
+    return success();
+  }
+};
+
+void TileOp::getCanonicalizationPatterns(RewritePatternSet &results,
+                                         MLIRContext *context) {
+  results.add<RemoveBroadcastTileFromBinaryElementwise>(context);
+}
+
 OpFoldResult TileOp::fold(FoldAdaptor adaptor) {
   if (getInput1().getType() == getType()) {
     if (auto multiples = llvm::dyn_cast_if_present<DenseElementsAttr>(

diff  --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 6a3d84e8d133b..585589ac1405a 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1996,6 +1996,166 @@ func.func @dont_canonicalize_non_const_avg_pool2d_adaptive(%arg0: tensor<1x?x?x8
 
 // -----
 
+// CHECK-LABEL: @canonicalize_tile_broadcast_sub
+// CHECK-SAME: %[[ARG0:[^:]+]]: tensor<96x56x56x96xf32>, %[[ARG1:[^:]+]]: tensor<1x56x56x1xf32>
+// CHECK-NOT: tosa.tile
+// CHECK: %[[SUB:.+]] = tosa.sub %[[ARG0]], %[[ARG1]] : (tensor<96x56x56x96xf32>, tensor<1x56x56x1xf32>) -> tensor<96x56x56x96xf32>
+// CHECK: return %[[SUB]]
+func.func @canonicalize_tile_broadcast_sub(%arg0: tensor<96x56x56x96xf32>, %arg1: tensor<1x56x56x1xf32>) -> tensor<96x56x56x96xf32> {
+  %shape = tosa.const_shape {values = dense<[96, 1, 1, 96]> : tensor<4xindex>} : () -> !tosa.shape<4>
+  %tile = tosa.tile %arg1, %shape : (tensor<1x56x56x1xf32>, !tosa.shape<4>) -> tensor<96x56x56x96xf32>
+  %sub = tosa.sub %arg0, %tile : (tensor<96x56x56x96xf32>, tensor<96x56x56x96xf32>) -> tensor<96x56x56x96xf32>
+  return %sub : tensor<96x56x56x96xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @canonicalize_tile_broadcast_greater
+// CHECK-SAME: %[[ARG0:[^:]+]]: tensor<1x197x768xf32>, %[[ARG1:[^:]+]]: tensor<1x197x1xf32>
+// CHECK-NOT: tosa.tile
+// CHECK: %[[GT:.+]] = tosa.greater %[[ARG0]], %[[ARG1]] : (tensor<1x197x768xf32>, tensor<1x197x1xf32>) -> tensor<1x197x768xi1>
+// CHECK: return %[[GT]]
+func.func @canonicalize_tile_broadcast_greater(%arg0: tensor<1x197x768xf32>, %arg1: tensor<1x197x1xf32>) -> tensor<1x197x768xi1> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 768]> : tensor<3xindex>} : () -> !tosa.shape<3>
+  %tile = tosa.tile %arg1, %shape : (tensor<1x197x1xf32>, !tosa.shape<3>) -> tensor<1x197x768xf32>
+  %gt = tosa.greater %arg0, %tile : (tensor<1x197x768xf32>, tensor<1x197x768xf32>) -> tensor<1x197x768xi1>
+  return %gt : tensor<1x197x768xi1>
+}
+
+// -----
+
+// CHECK-LABEL: @canonicalize_tile_broadcast_mul
+// CHECK-SAME: %[[ARG0:[^:]+]]: tensor<1x197x768xf32>, %[[ARG1:[^:]+]]: tensor<1x197x1xf32>, %[[SHIFT:[^:]+]]: tensor<1xi8>
+// CHECK-NOT: tosa.tile
+// CHECK: %[[MUL:.+]] = tosa.mul %[[ARG0]], %[[ARG1]], %[[SHIFT]] : (tensor<1x197x768xf32>, tensor<1x197x1xf32>, tensor<1xi8>) -> tensor<1x197x768xf32>
+// CHECK: return %[[MUL]]
+func.func @canonicalize_tile_broadcast_mul(%arg0: tensor<1x197x768xf32>, %arg1: tensor<1x197x1xf32>, %shift: tensor<1xi8>) -> tensor<1x197x768xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 768]> : tensor<3xindex>} : () -> !tosa.shape<3>
+  %tile = tosa.tile %arg1, %shape : (tensor<1x197x1xf32>, !tosa.shape<3>) -> tensor<1x197x768xf32>
+  %mul = tosa.mul %arg0, %tile, %shift : (tensor<1x197x768xf32>, tensor<1x197x768xf32>, tensor<1xi8>) -> tensor<1x197x768xf32>
+  return %mul : tensor<1x197x768xf32>
+}
+
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_when_result_no_longer_broadcastable
+// CHECK: tosa.tile
+// CHECK: tosa.sub
+func.func @dont_canonicalize_tile_when_result_no_longer_broadcastable(%arg0: tensor<2x1xf32>) -> tensor<2x4xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile = tosa.tile %arg0, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %sub = tosa.sub %tile, %arg0 : (tensor<2x4xf32>, tensor<2x1xf32>) -> tensor<2x4xf32>
+  return %sub : tensor<2x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_second_tile_when_result_no_longer_broadcastable
+// CHECK-SAME: %[[ARG0:[^:]+]]: tensor<2x1xf32>
+// CHECK: %[[SHAPE:.+]] = tosa.const_shape
+// CHECK: %[[TILE:.+]] = tosa.tile %[[ARG0]], %[[SHAPE]]
+// CHECK: %[[ADD:.+]] = tosa.add %[[ARG0]], %[[TILE]] : (tensor<2x1xf32>, tensor<2x4xf32>) -> tensor<2x4xf32>
+// CHECK: return %[[ADD]]
+func.func @dont_canonicalize_second_tile_when_result_no_longer_broadcastable(%arg0: tensor<2x1xf32>) -> tensor<2x4xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile0 = tosa.tile %arg0, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %tile1 = tosa.tile %arg0, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %add = tosa.add %tile0, %tile1 : (tensor<2x4xf32>, tensor<2x4xf32>) -> tensor<2x4xf32>
+  return %add : tensor<2x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_with_unranked_other_operand
+// CHECK-SAME: %[[ARG0:[^:]+]]: tensor<2x1xf32>, %[[ARG1:[^:]+]]: tensor<*xf32>
+// CHECK: %[[SHAPE:.+]] = tosa.const_shape
+// CHECK: %[[TILE:.+]] = tosa.tile %[[ARG0]], %[[SHAPE]]
+// CHECK: %[[SUB:.+]] = tosa.sub %[[ARG1]], %[[TILE]] : (tensor<*xf32>, tensor<2x4xf32>) -> tensor<2x4xf32>
+// CHECK: return %[[SUB]]
+func.func @dont_canonicalize_tile_with_unranked_other_operand(%arg0: tensor<2x1xf32>, %arg1: tensor<*xf32>) -> tensor<2x4xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile = tosa.tile %arg0, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %sub = tosa.sub %arg1, %tile : (tensor<*xf32>, tensor<2x4xf32>) -> tensor<2x4xf32>
+  return %sub : tensor<2x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_non_singleton_expansion
+// CHECK: tosa.tile
+// CHECK: tosa.sub
+func.func @dont_canonicalize_tile_non_singleton_expansion(%arg0: tensor<1x56x56x96xf32>, %arg1: tensor<1x56x56x2xf32>) -> tensor<1x56x56x96xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 1, 1, 48]> : tensor<4xindex>} : () -> !tosa.shape<4>
+  %tile = tosa.tile %arg1, %shape : (tensor<1x56x56x2xf32>, !tosa.shape<4>) -> tensor<1x56x56x96xf32>
+  %sub = tosa.sub %arg0, %tile : (tensor<1x56x56x96xf32>, tensor<1x56x56x96xf32>) -> tensor<1x56x56x96xf32>
+  return %sub : tensor<1x56x56x96xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_dynamic_expanded_dim
+// CHECK: tosa.tile
+// CHECK: tosa.sub
+func.func @dont_canonicalize_tile_dynamic_expanded_dim(%arg0: tensor<2x?xf32>, %arg1: tensor<2x4xf32>) -> tensor<2x4xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile = tosa.tile %arg0, %shape : (tensor<2x?xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %sub = tosa.sub %arg1, %tile : (tensor<2x4xf32>, tensor<2x4xf32>) -> tensor<2x4xf32>
+  return %sub : tensor<2x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_dynamic_output
+// CHECK: tosa.tile
+// CHECK: tosa.sub
+func.func @dont_canonicalize_tile_dynamic_output(%arg0: tensor<2x?xf32>, %arg1: tensor<2x1xf32>) -> tensor<2x?xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile = tosa.tile %arg1, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x?xf32>
+  %sub = tosa.sub %arg0, %tile : (tensor<2x?xf32>, tensor<2x?xf32>) -> tensor<2x?xf32>
+  return %sub : tensor<2x?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_multi_use
+// CHECK: tosa.tile
+// CHECK: tosa.sub
+// CHECK: tosa.add
+func.func @dont_canonicalize_tile_multi_use(%arg0: tensor<1x56x56x96xf32>, %arg1: tensor<1x56x56x1xf32>) -> (tensor<1x56x56x96xf32>, tensor<1x56x56x96xf32>) {
+  %shape = tosa.const_shape {values = dense<[1, 1, 1, 96]> : tensor<4xindex>} : () -> !tosa.shape<4>
+  %tile = tosa.tile %arg1, %shape : (tensor<1x56x56x1xf32>, !tosa.shape<4>) -> tensor<1x56x56x96xf32>
+  %sub = tosa.sub %arg0, %tile : (tensor<1x56x56x96xf32>, tensor<1x56x56x96xf32>) -> tensor<1x56x56x96xf32>
+  %add = tosa.add %arg0, %tile : (tensor<1x56x56x96xf32>, tensor<1x56x56x96xf32>) -> tensor<1x56x56x96xf32>
+  return %sub, %add : tensor<1x56x56x96xf32>, tensor<1x56x56x96xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_unary_elementwise
+// CHECK: tosa.tile
+// CHECK: tosa.abs
+func.func @dont_canonicalize_tile_unary_elementwise(%arg0: tensor<2x1xf32>) -> tensor<2x4xf32> {
+  %shape = tosa.const_shape {values = dense<[1, 4]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %tile = tosa.tile %arg0, %shape : (tensor<2x1xf32>, !tosa.shape<2>) -> tensor<2x4xf32>
+  %abs = tosa.abs %tile : (tensor<2x4xf32>) -> tensor<2x4xf32>
+  return %abs : tensor<2x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @dont_canonicalize_tile_used_as_mul_shift
+// CHECK: tosa.tile
+// CHECK: tosa.mul
+func.func @dont_canonicalize_tile_used_as_mul_shift(%lhs: tensor<1xf32>, %rhs: tensor<1xf32>, %shift: tensor<?xi8>) -> tensor<1xf32> {
+  %multiples = tosa.const_shape {values = dense<[1]> : tensor<1xindex>} : () -> !tosa.shape<1>
+  %shift_static = tosa.tile %shift, %multiples : (tensor<?xi8>, !tosa.shape<1>) -> tensor<1xi8>
+  %mul = tosa.mul %lhs, %rhs, %shift_static : (tensor<1xf32>, tensor<1xf32>, tensor<1xi8>) -> tensor<1xf32>
+  return %mul : tensor<1xf32>
+}
+
+// -----
+
 // CHECK-LABEL: test_single_concat
 // CHECK: %[[VAL_1:.*]] = tosa.concat %arg0, %arg0 {axis = 1 : i32} : (tensor<1x1x7x7xf32>, tensor<1x1x7x7xf32>) -> tensor<1x2x7x7xf32>
 // CHECK: return %[[VAL_1]] : tensor<1x2x7x7xf32>


        

From llvmlistbot at llvm.org  Thu Jul  9 11:49:53 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 09 Jul 2026 11:49:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fold tiled input into binary
 elementwise operations (PR #203941)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203941@github.com>
Message-ID: <6a4fed51.e9d484b9.15e549.54ee@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/203941

From llvmlistbot at llvm.org  Thu Jul  9 17:13:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 17:13:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a50392d.eb24c6ea.27aedc.433a@mx.google.com>


MattPD wrote:

Just a tiny nit: the commit message says "Unsupported device/offload-entry ... remain diagnosed", but this revision removes the offload-entry rejection (on the target device an in_reduction item is now handled as a plain map(tofrom)). Might be worth updating that line to name what's still diagnosed: the byref modifier, a two-argument initializer, a cleanup region, and the Flang COMMON/EQUIVALENCE/privatized cases.


https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Thu Jul  9 18:53:10 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Thu, 09 Jul 2026 18:53:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a505086.f1c1923d.991d1.b5f7@mx.google.com>


https://github.com/nbpatel approved this pull request.


https://github.com/llvm/llvm-project/pull/208276

From llvmlistbot at llvm.org  Thu Jul  9 19:56:19 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Thu, 09 Jul 2026 19:56:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Fix hasSideEffects of
	emitc.cast (PR #208607)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208607@github.com>
Message-ID: <6a505f53.1905e68e.18ad68.614f@mx.google.com>


https://github.com/jacquesguan approved this pull request.

LGTM, thanks to fix it.

https://github.com/llvm/llvm-project/pull/208607

From llvmlistbot at llvm.org  Thu Jul  9 20:04:39 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Thu, 09 Jul 2026 20:04:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Fix hasSideEffects of
	emitc.cast (PR #208607)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208607@github.com>
Message-ID: <6a506147.e55dc341.1f19bd.0a3c@mx.google.com>


https://github.com/CoTinker updated https://github.com/llvm/llvm-project/pull/208607

>From 668951e33b3a96beabaaef4db1359805d0afe858 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 10:38:38 +0800
Subject: [PATCH 1/3] [mlir][emitc] Fix `hasSideEffects` of emitc.cast

When `pure` attribute is set, the cast op is side-effect-free, so we should return false in `hasSideEffects` method.
---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index 1b776519b1428..d42c74e963769 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -404,7 +404,7 @@ def EmitC_CastOp : EmitC_Op<"cast",
     bool hasSideEffects() {
       // Use the "pure" attribute to see whether this CastOp has side effects.
       // Note that by default, `pure` is not set.
-      return getPure();
+      return !getPure();
     }
   }];
 }

>From 771aadd206ba4dd18e752b7242e41f3bec43d4bd Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 10:43:19 +0800
Subject: [PATCH 2/3] fix test

---
 mlir/test/Target/Cpp/expressions.mlir | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Target/Cpp/expressions.mlir b/mlir/test/Target/Cpp/expressions.mlir
index d447f9a38f3d3..b1c024f9ec98c 100644
--- a/mlir/test/Target/Cpp/expressions.mlir
+++ b/mlir/test/Target/Cpp/expressions.mlir
@@ -89,11 +89,14 @@ func.func @do_not_inline(%arg0: i32, %arg1: i32, %arg2 : i32) -> i32 {
 }
 
 // CPP-DEFAULT:      float parentheses_for_low_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
-// CPP-DEFAULT-NEXT:   return (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DEFAULT-NEXT:   float [[VAL_4:v[0-9]+]] = (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DEFAULT-NEXT:   return [[VAL_4]];
 // CPP-DEFAULT-NEXT: }
 
 // CPP-DECLTOP:      float parentheses_for_low_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
-// CPP-DECLTOP-NEXT:   return (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DECLTOP-NEXT:   float [[VAL_4:v[0-9]+]];
+// CPP-DECLTOP-NEXT:   [[VAL_4]] = (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DECLTOP-NEXT:   return [[VAL_4]];
 // CPP-DECLTOP-NEXT: }
 
 func.func @parentheses_for_low_precedence(%arg0: i32, %arg1: i32, %arg2: i32) -> f32 {

>From 5b1638114d0df1afa28a659366dab851a7f1d08c Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 11:04:29 +0800
Subject: [PATCH 3/3] add cast tests

---
 mlir/test/Target/Cpp/expressions.mlir | 35 +++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/mlir/test/Target/Cpp/expressions.mlir b/mlir/test/Target/Cpp/expressions.mlir
index b1c024f9ec98c..5e76f61933d4d 100644
--- a/mlir/test/Target/Cpp/expressions.mlir
+++ b/mlir/test/Target/Cpp/expressions.mlir
@@ -109,6 +109,41 @@ func.func @parentheses_for_low_precedence(%arg0: i32, %arg1: i32, %arg2: i32) ->
   return %e : f32
 }
 
+// CPP-DEFAULT:      float inline_cast_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DEFAULT-NEXT:   return (float) [[VAL_1]];
+// CPP-DEFAULT-NEXT: }
+
+// CPP-DECLTOP:      float inline_cast_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DECLTOP-NEXT:   return (float) [[VAL_1]];
+// CPP-DECLTOP-NEXT: }
+
+func.func @inline_cast_pure(%arg0: i32) -> f32 {
+  %0 = emitc.expression : f32 {
+    %1 = cast %arg0 {pure} : i32 to f32
+    yield %1 : f32
+  }
+  return %0 : f32
+}
+
+// CPP-DEFAULT:      float do_not_inline_cast_without_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DEFAULT-NEXT:   float [[VAL_2:v[0-9]+]] = (float) [[VAL_1]]
+// CPP-DEFAULT-NEXT:   return [[VAL_2]];
+// CPP-DEFAULT-NEXT: }
+
+// CPP-DECLTOP:      float do_not_inline_cast_without_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DECLTOP-NEXT:   float [[VAL_2:v[0-9]+]];
+// CPP-DECLTOP-NEXT:   [[VAL_2]] = (float) [[VAL_1]];
+// CPP-DECLTOP-NEXT:   return [[VAL_2]];
+// CPP-DECLTOP-NEXT: }
+
+func.func @do_not_inline_cast_without_pure(%arg0: i32) -> f32 {
+  %0 = emitc.expression : f32 {
+    %1 = cast %arg0 : i32 to f32
+    yield %1 : f32
+  }
+  return %0 : f32
+}
+
 // CPP-DEFAULT:      int32_t parentheses_for_same_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
 // CPP-DEFAULT-NEXT:   return [[VAL_3]] / ([[VAL_1]] * [[VAL_2]]);
 // CPP-DEFAULT-NEXT: }


From llvmlistbot at llvm.org  Thu Jul  9 20:17:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 20:17:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Fix hasSideEffects of
	emitc.cast (PR #208607)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208607@github.com>
Message-ID: <6a50644c.160324b0.195192.1c1a@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

* 3906 tests passed
* 440 tests skipped
* 1 test failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Target/Cpp/expressions.mlir</summary>

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
c:\_work\llvm-project\llvm-project\build\bin\mlir-translate.exe -mlir-to-cpp C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir -check-prefix=CPP-DEFAULT
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\mlir-translate.exe' -mlir-to-cpp 'C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir:121:31: error: custom op 'emitc.expression' expected function type
# |   %0 = emitc.expression : f32 {
# |                               ^
# `-----------------------------
# error: command failed with exit status: 1
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir' -check-prefix=CPP-DEFAULT
# .---command stderr------------
# | FileCheck error: '<stdin>' is empty.
# | FileCheck command line:  c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\mlir\test\Target\Cpp\expressions.mlir -check-prefix=CPP-DEFAULT
# `-----------------------------
# error: command failed with exit status: 2

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208607

From llvmlistbot at llvm.org  Thu Jul  9 20:17:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 20:17:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Fix hasSideEffects of
	emitc.cast (PR #208607)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208607@github.com>
Message-ID: <6a50644c.bf38ae28.39239a.100f@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 8420 tests passed
* 630 tests skipped
* 1 test failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Target/Cpp/expressions.mlir</summary>

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-cpp /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir -check-prefix=CPP-DEFAULT
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-cpp /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir:121:31: error: custom op 'emitc.expression' expected function type
# |   %0 = emitc.expression : f32 {
# |                               ^
# `-----------------------------
# error: command failed with exit status: 1
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir -check-prefix=CPP-DEFAULT
# .---command stderr------------
# | FileCheck error: '<stdin>' is empty.
# | FileCheck command line:  /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/Cpp/expressions.mlir -check-prefix=CPP-DEFAULT
# `-----------------------------
# error: command failed with exit status: 2

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208607

From llvmlistbot at llvm.org  Thu Jul  9 20:19:16 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Thu, 09 Jul 2026 20:19:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Fix hasSideEffects of
	emitc.cast (PR #208607)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208607@github.com>
Message-ID: <6a5064b4.70920dee.290dca.7dc6@mx.google.com>


https://github.com/CoTinker updated https://github.com/llvm/llvm-project/pull/208607

>From 668951e33b3a96beabaaef4db1359805d0afe858 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 10:38:38 +0800
Subject: [PATCH 1/4] [mlir][emitc] Fix `hasSideEffects` of emitc.cast

When `pure` attribute is set, the cast op is side-effect-free, so we should return false in `hasSideEffects` method.
---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index 1b776519b1428..d42c74e963769 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -404,7 +404,7 @@ def EmitC_CastOp : EmitC_Op<"cast",
     bool hasSideEffects() {
       // Use the "pure" attribute to see whether this CastOp has side effects.
       // Note that by default, `pure` is not set.
-      return getPure();
+      return !getPure();
     }
   }];
 }

>From 771aadd206ba4dd18e752b7242e41f3bec43d4bd Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 10:43:19 +0800
Subject: [PATCH 2/4] fix test

---
 mlir/test/Target/Cpp/expressions.mlir | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Target/Cpp/expressions.mlir b/mlir/test/Target/Cpp/expressions.mlir
index d447f9a38f3d3..b1c024f9ec98c 100644
--- a/mlir/test/Target/Cpp/expressions.mlir
+++ b/mlir/test/Target/Cpp/expressions.mlir
@@ -89,11 +89,14 @@ func.func @do_not_inline(%arg0: i32, %arg1: i32, %arg2 : i32) -> i32 {
 }
 
 // CPP-DEFAULT:      float parentheses_for_low_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
-// CPP-DEFAULT-NEXT:   return (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DEFAULT-NEXT:   float [[VAL_4:v[0-9]+]] = (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DEFAULT-NEXT:   return [[VAL_4]];
 // CPP-DEFAULT-NEXT: }
 
 // CPP-DECLTOP:      float parentheses_for_low_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
-// CPP-DECLTOP-NEXT:   return (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DECLTOP-NEXT:   float [[VAL_4:v[0-9]+]];
+// CPP-DECLTOP-NEXT:   [[VAL_4]] = (float) (([[VAL_1]] + [[VAL_2]]) * [[VAL_3]]);
+// CPP-DECLTOP-NEXT:   return [[VAL_4]];
 // CPP-DECLTOP-NEXT: }
 
 func.func @parentheses_for_low_precedence(%arg0: i32, %arg1: i32, %arg2: i32) -> f32 {

>From 5b1638114d0df1afa28a659366dab851a7f1d08c Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 11:04:29 +0800
Subject: [PATCH 3/4] add cast tests

---
 mlir/test/Target/Cpp/expressions.mlir | 35 +++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/mlir/test/Target/Cpp/expressions.mlir b/mlir/test/Target/Cpp/expressions.mlir
index b1c024f9ec98c..5e76f61933d4d 100644
--- a/mlir/test/Target/Cpp/expressions.mlir
+++ b/mlir/test/Target/Cpp/expressions.mlir
@@ -109,6 +109,41 @@ func.func @parentheses_for_low_precedence(%arg0: i32, %arg1: i32, %arg2: i32) ->
   return %e : f32
 }
 
+// CPP-DEFAULT:      float inline_cast_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DEFAULT-NEXT:   return (float) [[VAL_1]];
+// CPP-DEFAULT-NEXT: }
+
+// CPP-DECLTOP:      float inline_cast_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DECLTOP-NEXT:   return (float) [[VAL_1]];
+// CPP-DECLTOP-NEXT: }
+
+func.func @inline_cast_pure(%arg0: i32) -> f32 {
+  %0 = emitc.expression : f32 {
+    %1 = cast %arg0 {pure} : i32 to f32
+    yield %1 : f32
+  }
+  return %0 : f32
+}
+
+// CPP-DEFAULT:      float do_not_inline_cast_without_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DEFAULT-NEXT:   float [[VAL_2:v[0-9]+]] = (float) [[VAL_1]]
+// CPP-DEFAULT-NEXT:   return [[VAL_2]];
+// CPP-DEFAULT-NEXT: }
+
+// CPP-DECLTOP:      float do_not_inline_cast_without_pure(int32_t [[VAL_1:v[0-9]+]]) {
+// CPP-DECLTOP-NEXT:   float [[VAL_2:v[0-9]+]];
+// CPP-DECLTOP-NEXT:   [[VAL_2]] = (float) [[VAL_1]];
+// CPP-DECLTOP-NEXT:   return [[VAL_2]];
+// CPP-DECLTOP-NEXT: }
+
+func.func @do_not_inline_cast_without_pure(%arg0: i32) -> f32 {
+  %0 = emitc.expression : f32 {
+    %1 = cast %arg0 : i32 to f32
+    yield %1 : f32
+  }
+  return %0 : f32
+}
+
 // CPP-DEFAULT:      int32_t parentheses_for_same_precedence(int32_t [[VAL_1:v[0-9]+]], int32_t [[VAL_2:v[0-9]+]], int32_t [[VAL_3:v[0-9]+]]) {
 // CPP-DEFAULT-NEXT:   return [[VAL_3]] / ([[VAL_1]] * [[VAL_2]]);
 // CPP-DEFAULT-NEXT: }

>From 28498c1f19cd4a55dc16fe178b33989eb02c2b76 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Fri, 10 Jul 2026 11:19:04 +0800
Subject: [PATCH 4/4] fix tests

---
 mlir/test/Target/Cpp/expressions.mlir | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/test/Target/Cpp/expressions.mlir b/mlir/test/Target/Cpp/expressions.mlir
index 5e76f61933d4d..43e43cda49477 100644
--- a/mlir/test/Target/Cpp/expressions.mlir
+++ b/mlir/test/Target/Cpp/expressions.mlir
@@ -118,7 +118,7 @@ func.func @parentheses_for_low_precedence(%arg0: i32, %arg1: i32, %arg2: i32) ->
 // CPP-DECLTOP-NEXT: }
 
 func.func @inline_cast_pure(%arg0: i32) -> f32 {
-  %0 = emitc.expression : f32 {
+  %0 = emitc.expression %arg0 : (i32) -> f32 {
     %1 = cast %arg0 {pure} : i32 to f32
     yield %1 : f32
   }
@@ -126,7 +126,7 @@ func.func @inline_cast_pure(%arg0: i32) -> f32 {
 }
 
 // CPP-DEFAULT:      float do_not_inline_cast_without_pure(int32_t [[VAL_1:v[0-9]+]]) {
-// CPP-DEFAULT-NEXT:   float [[VAL_2:v[0-9]+]] = (float) [[VAL_1]]
+// CPP-DEFAULT-NEXT:   float [[VAL_2:v[0-9]+]] = (float) [[VAL_1]];
 // CPP-DEFAULT-NEXT:   return [[VAL_2]];
 // CPP-DEFAULT-NEXT: }
 
@@ -137,7 +137,7 @@ func.func @inline_cast_pure(%arg0: i32) -> f32 {
 // CPP-DECLTOP-NEXT: }
 
 func.func @do_not_inline_cast_without_pure(%arg0: i32) -> f32 {
-  %0 = emitc.expression : f32 {
+  %0 = emitc.expression %arg0 : (i32) -> f32 {
     %1 = cast %arg0 : i32 to f32
     yield %1 : f32
   }


From llvmlistbot at llvm.org  Thu Jul  9 20:52:41 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 09 Jul 2026 20:52:41 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a506c89.6c8668f2.e4f82.c463@mx.google.com>


https://github.com/Saieiei updated https://github.com/llvm/llvm-project/pull/199967

>From b34c5df5b8785da1cad68a4e7f54bdd2d895f04b Mon Sep 17 00:00:00 2001
From: Sairudra More <sairudra60 at gmail.com>
Date: Fri, 12 Jun 2026 04:57:07 -0500
Subject: [PATCH] [flang][OpenMP] Lower target in_reduction for host fallback

Enable host-fallback lowering for target in_reduction in Flang and MLIR OpenMP translation.

Model target in_reduction through the matching map entry, force address-preserving implicit mapping for Flang in_reduction list items, and emit the host-side task-reduction lookup with __kmpc_task_reduction_get_th_data. The runtime entry point takes and returns a generic, default-address-space pointer, so normalize a non-default-address-space captured pointer to the generic address space before the call and cast the returned private pointer back to the map block argument's address space, mirroring the in_reduction handling on omp.taskloop. On the target device, in_reduction is handled as a regular map(tofrom) variable. The byref modifier, two-argument initializers, cleanup regions, and the remaining Flang COMMON/EQUIVALENCE/privatized-variable cases continue to be diagnosed.

Add Flang lowering, MLIR verifier/translation, and LLVM IR tests for the supported host-fallback path, including a non-default-address-space case, and the remaining unsupported cases.
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 103 +++++++++--
 .../OpenMP/target-inreduction-llvmir.f90      |  50 ++++++
 .../OpenMP/Todo/target-inreduction-common.f90 |  20 +++
 .../Todo/target-inreduction-equivalence.f90   |  19 +++
 .../Todo/target-inreduction-firstprivate.f90  |  19 +++
 .../Lower/OpenMP/Todo/target-inreduction.f90  |  15 --
 .../OpenMP/target-inreduction-unused.f90      |  27 +++
 .../test/Lower/OpenMP/target-inreduction.f90  |  30 ++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  33 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  51 ++++++
 .../mlir/Dialect/OpenMP/OpenMPClauses.td      |  14 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  22 ++-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 155 ++++++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 161 ++++++++++++++++--
 .../Dialect/OpenMP/host-op-filtering.mlir     |   8 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  85 +++++++++
 .../openmp-target-in-reduction-device.mlir    |  51 ++++++
 .../openmp-target-in-reduction-multi.mlir     |  77 +++++++++
 .../LLVMIR/openmp-target-in-reduction.mlir    | 107 ++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  91 +++++++++-
 20 files changed, 1059 insertions(+), 79 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction-unused.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 9c6352b73a585..00297e4a48e69 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2474,6 +2474,7 @@ genTargetClauses(lower::AbstractConverter &converter,
                  mlir::omp::TargetExtOperands &clauseOps,
                  DefaultMapsTy &defaultMaps,
                  llvm::SmallVectorImpl<Object> &hasDeviceAddrObjects,
+                 llvm::SmallVectorImpl<Object> &inReductionObjects,
                  llvm::SmallVectorImpl<Object> &isDevicePtrObjects,
                  llvm::SmallVectorImpl<Object> &mapObjects) {
   ClauseProcessor cp(converter, semaCtx, clauses);
@@ -2489,12 +2490,33 @@ genTargetClauses(lower::AbstractConverter &converter,
     hostEvalInfo->collectValues(clauseOps.hostEvalVars);
   }
   cp.processIf(llvm::omp::Directive::OMPD_target, clauseOps);
+  cp.processInReduction(loc, clauseOps, inReductionObjects);
+  // The host fallback for `target in_reduction` redirects each list item's
+  // dedicated implicit map entry to the per-task reduction-private storage
+  // returned by the runtime. A COMMON block member is accessed in the target
+  // body through the COMMON block storage map, not through its own member map
+  // entry, so the redirect would rewrite an unused map argument and the body
+  // would accumulate into the mapped original rather than the private copy.
+  // An EQUIVALENCE-associated variable has the same problem: it shares storage
+  // with another variable that may be mapped separately (e.g. `map(tofrom: j)`
+  // with `equivalence(i, j)`), so the body updates the aliased storage while
+  // the redirected runtime-private copy stays dead.
+  // Reject both cases until the redirect can target the storage map.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      if (semantics::FindCommonBlockContaining(sym->GetUltimate()))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a COMMON block member");
+      if (semantics::FindEquivalenceSet(sym->GetUltimate()))
+        TODO(loc, "TARGET construct with IN_REDUCTION of an EQUIVALENCE "
+                  "storage-associated variable");
+    }
   cp.processIsDevicePtr(stmtCtx, clauseOps, isDevicePtrObjects);
   cp.processMap(loc, stmtCtx, clauseOps, llvm::omp::Directive::OMPD_unknown,
                 &mapObjects);
   cp.processNowait(clauseOps);
   cp.processThreadLimit(stmtCtx, clauseOps);
-  cp.processTODO<clause::Allocate, clause::InReduction, clause::UsesAllocators>(
+  cp.processTODO<clause::Allocate, clause::UsesAllocators>(
       loc, llvm::omp::Directive::OMPD_target);
 
   // `target private(..)` is only supported in delayed privatization mode.
@@ -3556,10 +3578,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   mlir::omp::TargetExtOperands clauseOps;
   DefaultMapsTy defaultMaps;
   llvm::SmallVector<Object> mapObjects, hasDeviceAddrObjects,
-      isDevicePtrObjects;
+      inReductionObjects, isDevicePtrObjects;
   genTargetClauses(converter, semaCtx, symTable, stmtCtx, eval, item->clauses,
                    loc, clauseOps, defaultMaps, hasDeviceAddrObjects,
-                   isDevicePtrObjects, mapObjects);
+                   inReductionObjects, isDevicePtrObjects, mapObjects);
 
   KernelTypeVisitor visitor(semaCtx, converter.getModuleOp());
   clauseOps.kernelType = mlir::omp::TargetExecModeAttr::get(
@@ -3607,6 +3629,26 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
                            /*isTargetPrivitization=*/true);
   dsp.processStep1(&clauseOps);
 
+  // The host fallback for `target in_reduction` requires each list item to be
+  // implicitly mapped as an address-preserving entry whose map argument the
+  // redirect rewrites to the runtime reduction-private storage. When the same
+  // variable is also privatized on the target (implicitly, as with a
+  // host-associated scalar that defaults to firstprivate, or explicitly with
+  // `firstprivate(i) in_reduction(+:i)`), the force-map is suppressed because
+  // the symbol is already in the privatization set, and the firstprivate copy
+  // region then has no mold value for it, aborting translation. Reject until
+  // in_reduction can be reconciled with target privatization.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      const semantics::Symbol &ultimate = sym->GetUltimate();
+      if (llvm::any_of(dsp.getAllSymbolsToPrivatize(),
+                       [&](const semantics::Symbol *priv) {
+                         return priv->GetUltimate() == ultimate;
+                       }))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a privatized variable");
+    }
+
   // Collect symbols that have dynamic substring accesses
   llvm::SmallPtrSet<const semantics::Symbol *, 8> symbolsWithDynamicSubstring;
   collectSymbolsWithDynamicSubstring(semaCtx, eval,
@@ -3617,7 +3659,16 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   // symbols used inside the region that do not have explicit data-environment
   // attribute clauses (neither data-sharing; e.g. `private`, nor `map`
   // clauses).
-  auto captureImplicitMap = [&](const semantics::Symbol &sym) {
+  //
+  // When `forceAddressPreserving` is set, the symbol is force-mapped as an
+  // address-preserving `capture(ByRef)` with implicit `tofrom` flags,
+  // bypassing the scalar default capture rules. This is used for `target
+  // in_reduction` list items, whose mapped pointer is passed as the `orig`
+  // argument of `__kmpc_task_reduction_get_th_data`; a ByCopy scalar capture
+  // would break the runtime lookup against the enclosing taskgroup's
+  // task_reduction descriptor.
+  auto captureImplicitMap = [&](const semantics::Symbol &sym,
+                                bool forceAddressPreserving = false) {
     // Structure component symbols don't have bindings, and can only be
     // explicitly mapped individually. If a member is captured implicitly
     // we map the entirety of the derived type when we find its symbol.
@@ -3626,12 +3677,14 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
     // if the symbol is part of an already mapped common block, do not make a
     // map for it.
-    if (const Fortran::semantics::Symbol *common =
-            Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
-      if (llvm::any_of(mapObjects, [=](const Object &object) {
-            return object.sym() == common;
-          }))
-        return;
+    if (!forceAddressPreserving) {
+      if (const Fortran::semantics::Symbol *common =
+              Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
+        if (llvm::any_of(mapObjects, [=](const Object &object) {
+              return object.sym() == common;
+            }))
+          return;
+    }
 
     // If we come across a symbol without a symbol address, we
     // return as we cannot process it, this is intended as a
@@ -3681,13 +3734,21 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
         eleType = refType.getElementType();
 
       std::pair<mlir::omp::ClauseMapFlags, mlir::omp::VariableCaptureKind>
-          mapFlagAndKind = getImplicitMapTypeAndKind(
-              firOpBuilder, converter, defaultMaps, eleType, loc, sym);
+          mapFlagAndKind;
+      if (forceAddressPreserving)
+        mapFlagAndKind = {mlir::omp::ClauseMapFlags::implicit |
+                              mlir::omp::ClauseMapFlags::to |
+                              mlir::omp::ClauseMapFlags::from,
+                          mlir::omp::VariableCaptureKind::ByRef};
+      else
+        mapFlagAndKind = getImplicitMapTypeAndKind(
+            firOpBuilder, converter, defaultMaps, eleType, loc, sym);
 
       mlir::FlatSymbolRefAttr mapperId;
       auto defaultmapBehaviour = getDefaultmapIfPresent(defaultMaps, eleType);
-      if (defaultmapBehaviour ==
-          clause::Defaultmap::ImplicitBehavior::Default) {
+      if (!forceAddressPreserving &&
+          defaultmapBehaviour ==
+              clause::Defaultmap::ImplicitBehavior::Default) {
         const semantics::DerivedTypeSpec *typeSpec =
             sym.GetType() ? sym.GetType()->AsDerived() : nullptr;
         if (typeSpec) {
@@ -3741,6 +3802,15 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
           Object{const_cast<semantics::Symbol *>(&sym), std::nullopt});
     }
   };
+  // OpenMP requires `in_reduction` list items on `target` to be implicitly
+  // data-mapped. Force-map them as address-preserving captures before the
+  // generic implicit-map walk so that walk treats the symbols as already
+  // mapped via `isDuplicateMappedSymbol` and does not downgrade them to
+  // ByCopy.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym())
+      captureImplicitMap(*sym, /*forceAddressPreserving=*/true);
+
   lower::pft::visitAllSymbols(eval, captureImplicitMap);
 
   auto targetOp = mlir::omp::TargetOp::create(firOpBuilder, loc, clauseOps);
@@ -3753,7 +3823,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   args.hasDeviceAddr.objects = hasDeviceAddrObjects;
   args.hasDeviceAddr.vars = hasDeviceAddrBaseValues;
   args.hostEvalVars = clauseOps.hostEvalVars;
-  // TODO: Add in_reduction syms and vars.
+  // `in_reduction` list items do not get their own entry block argument on
+  // `omp.target`; they are implicitly mapped (see the force-map above) and the
+  // target body accesses them through their `map_entries` block argument. The
+  // `in_reduction` operands remain on the op as host-side metadata.
   args.map.objects = mapObjects;
   args.map.vars = mapBaseValues;
   args.priv.objects = makeObjects(dsp.getDelayedPrivSymbols());
diff --git a/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90 b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
new file mode 100644
index 0000000000000..4b5d951a6dd38
--- /dev/null
+++ b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
@@ -0,0 +1,50 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-version=50 -o - %s | FileCheck %s
+
+! End-to-end coverage: Flang lowers a taskgroup task_reduction enclosing a
+! target in_reduction all the way to LLVM IR. Verifies that the full host-
+! fallback path emits __kmpc_taskred_init for the enclosing taskgroup and
+! __kmpc_task_reduction_get_th_data for the target's in_reduction lookup,
+! with load/store through the returned private pointer and no direct update
+! through the original shared pointer.
+
+subroutine target_in_reduction_e2e()
+  integer :: i
+  i = 0
+  !$omp taskgroup task_reduction(+:i)
+    !$omp target in_reduction(+:i)
+    i = i + 1
+    !$omp end target
+  !$omp end taskgroup
+end subroutine target_in_reduction_e2e
+
+! CHECK-LABEL: define void @target_in_reduction_e2e_()
+! The enclosing taskgroup emits __kmpc_taskred_init to register the
+! task_reduction descriptor.
+! CHECK:         call ptr @__kmpc_taskred_init(i32 %{{.+}}, i32 1, ptr %{{.+}})
+
+! The host stub calls the outlined target body passing the captured pointer.
+! CHECK:         call void @__omp_offloading_{{.*}}_target_in_reduction_e2e_{{.*}}(ptr %{{.+}}, ptr null)
+
+! Inside the outlined target body, the in_reduction private pointer is
+! obtained from the runtime using the captured original pointer with a NULL
+! descriptor (the runtime walks enclosing taskgroups). All loads and stores
+! go through the returned private pointer.
+! CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_in_reduction_e2e_
+! CHECK-SAME:    (ptr %[[ORIG:.+]], ptr %{{.+}})
+! CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+! CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[ORIG]])
+! CHECK:         %[[VAL:.+]] = load i32, ptr %[[PRIV]]
+! CHECK:         %[[SUM:.+]] = add i32 %[[VAL]], 1
+! CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+! The outlined body must not store directly through the captured original
+! pointer; all updates go through the runtime-returned private copy.
+! CHECK-NOT:     store i32 %{{.+}}, ptr %[[ORIG]]
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
new file mode 100644
index 0000000000000..25408105ceabc
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is a COMMON block member is accessed in
+! the target body through the COMMON block storage map, not through its own
+! dedicated member map entry. The host-fallback redirect only rewrites the
+! member map argument, so the runtime-private pointer would be dead and the body
+! would accumulate into the mapped original instead of the reduction-private
+! copy. Until the redirect can target the storage map this case is rejected.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a COMMON block member
+
+subroutine omp_target_in_reduction_common()
+  integer :: i, k
+  common /cb/ i, k
+  i = 0
+  !$omp target map(tofrom: /cb/) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_common
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
new file mode 100644
index 0000000000000..816e76c8eeda9
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is EQUIVALENCE storage-associated
+! shares storage with another variable that may be mapped separately (here `j`).
+! The body updates the aliased storage through `j`'s map while the host-fallback
+! redirect rewrites only `i`'s map argument, leaving the runtime-private copy
+! dead. Reject until the redirect can account for storage association.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of an EQUIVALENCE storage-associated variable
+
+subroutine omp_target_in_reduction_equivalence()
+  integer :: i, j
+  equivalence(i, j)
+  i = 0
+  !$omp target map(tofrom: j) in_reduction(+:i)
+  j = j + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_equivalence
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
new file mode 100644
index 0000000000000..6dda967b82938
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! An explicit `firstprivate(i)` on the same `target` as `in_reduction(+:i)`
+! privatizes the reduction variable. The item is then privatized instead of
+! address-preserving-mapped, so the host-fallback redirect has no dedicated map
+! argument to rewrite and the firstprivate copy region has no mold value, which
+! previously aborted translation. Reject until in_reduction can be reconciled
+! with target privatization.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a privatized variable
+
+subroutine omp_target_in_reduction_firstprivate()
+  integer :: i
+  i = 0
+  !$omp target firstprivate(i) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_firstprivate
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
deleted file mode 100644
index e5a9cffac5a11..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-
-!===============================================================================
-! `mergeable` clause
-!===============================================================================
-
-! CHECK: not yet implemented: Unhandled clause IN_REDUCTION in TARGET construct
-subroutine omp_target_inreduction()
-  integer i
-  i = 0
-  !$omp target in_reduction(+:i)
-  i = i + 1
-  !$omp end target
-end subroutine omp_target_inreduction
diff --git a/flang/test/Lower/OpenMP/target-inreduction-unused.f90 b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
new file mode 100644
index 0000000000000..f9dfa4973610c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
@@ -0,0 +1,27 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Per the OpenMP spec, an in_reduction list item on a target construct is
+! implicitly data-mapped. The lowering must not rely on the variable being
+! referenced inside the target body to discover that map: here `i` only
+! appears in the in_reduction clause and is never read or written inside
+! the region. Verify that an omp.map.info for `i` is still emitted and
+! flows into the omp.target's map_entries.
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction_unused()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reduction_unusedEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@{{[^ ]+}} %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %{{[^ ]+}} : !fir.ref<i32>)
+
+subroutine omp_target_in_reduction_unused()
+  interface
+    subroutine sub()
+    end subroutine
+  end interface
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  call sub()
+  !$omp end target
+end subroutine omp_target_in_reduction_unused
diff --git a/flang/test/Lower/OpenMP/target-inreduction.f90 b/flang/test/Lower/OpenMP/target-inreduction.f90
new file mode 100644
index 0000000000000..74f0f003aab4f
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction.f90
@@ -0,0 +1,30 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Verify that in_reduction on a target construct is lowered to an
+! omp.target with both an in_reduction clause and an implicit map_entries
+! entry for the same variable. The in_reduction clause does not define an
+! entry block argument: inside the target body the variable is accessed
+! through its map_entries block argument. The implicit map also captures the
+! original pointer into the target region so the MLIR -> LLVM IR translation
+! can pass it to __kmpc_task_reduction_get_th_data.
+
+!CHECK-LABEL: omp.declare_reduction
+!CHECK-SAME:  @[[RED_I32_NAME:.*]] : i32 init {
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reductionEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@[[RED_I32_NAME]] %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %[[MAPARG:[^ ]+]] : !fir.ref<i32>)
+!CHECK:         hlfir.declare %[[MAPARG]]
+!CHECK:         omp.terminator
+!CHECK:       }
+
+subroutine omp_target_in_reduction()
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 965ac358c259e..6a81ff8c6a0bb 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivate.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivate   Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivate);
+
   /// Return the insertion point used by the underlying IRBuilder.
   InsertPointTy getInsertionPoint() { return Builder.saveIP(); }
 
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..4dc26bc0530f6 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item. Owning the gtid setup and the lookup loop here keeps the runtime
+  // generation inside OpenMPIRBuilder rather than the MLIR translation layer.
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Gtid = getOrCreateThreadID(Ident);
+
+  // The runtime entry point takes (and returns) a generic, default-address-
+  // space `ptr`. A NULL descriptor makes the runtime walk the enclosing
+  // taskgroups to find the matching task_reduction registration for the item.
+  Type *PtrTy = PointerType::getUnqual(M.getContext());
+  Value *NullDesc = ConstantPointerNull::get(PtrTy);
+  FunctionCallee GetThData =
+      getOrCreateRuntimeFunction(M, OMPRTL___kmpc_task_reduction_get_th_data);
+
+  for (unsigned Idx = 0; Idx < OrigPtrs.size(); ++Idx) {
+    // Normalize a non-default-address-space original pointer to the generic
+    // address space before the call.
+    Value *OrigPtr = OrigPtrs[Idx];
+    if (auto *OrigPtrTy = dyn_cast<PointerType>(OrigPtr->getType());
+        OrigPtrTy && OrigPtrTy->getAddressSpace() != 0)
+      OrigPtr = Builder.CreateAddrSpaceCast(OrigPtr, PtrTy);
+
+    Value *Priv = Builder.CreateCall(GetThData, {Gtid, NullDesc, OrigPtr},
+                                     "omp.inred.priv");
+
+    // Cast the returned private pointer back to the requested address space
+    // when it differs.
+    if (auto *ResPtrTy = dyn_cast<PointerType>(ResultPtrTys[Idx]);
+        ResPtrTy && ResPtrTy->getAddressSpace() != 0)
+      Priv = Builder.CreateAddrSpaceCast(Priv, ResultPtrTys[Idx]);
+
+    MapPrivate(Idx, Priv);
+  }
+  return Builder.saveIP();
+}
+
 OpenMPIRBuilder::InsertPointOrErrorTy
 OpenMPIRBuilder::createBarrier(const LocationDescription &Loc, Directive Kind,
                                bool ForceSimpleCall, bool CheckCancelFlag) {
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
index 54e46b74c5b64..cd1223dc1702c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
@@ -764,9 +764,17 @@ class OpenMP_InReductionClauseSkip<
     OptionalAttr<SymbolRefArrayAttr>:$in_reduction_syms
   );
 
-  // Description varies depending on the operation. Assembly format not defined
-  // because this clause must be processed together with the first region of the
-  // operation, as it defines entry block arguments.
+  // This assembly format should only be used by operations where `in_reduction`
+  // does not define entry block arguments (e.g. `omp.target`). Otherwise, it
+  // must be printed and parsed together with the corresponding region, because
+  // it defines entry block arguments.
+  let optAssemblyFormat = [{
+    `in_reduction` `(`
+      custom<InReductionClause>($in_reduction_vars, type($in_reduction_vars),
+                                $in_reduction_byref, $in_reduction_syms) `)`
+  }];
+
+  // Description varies depending on the operation.
 }
 
 def OpenMP_InReductionClause : OpenMP_InReductionClauseSkip<>;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 5b4ce7cc214bf..fd700e081fa01 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -966,7 +966,8 @@ def TaskOp : OpenMP_Op<"task", traits = [
     OutlineableOpenMPOpInterface
   ], clauses = [
     OpenMP_AffinityClause, OpenMP_AllocateClause, OpenMP_DependClause,
-    OpenMP_FinalClause, OpenMP_IfClause, OpenMP_InReductionClause,
+    OpenMP_FinalClause, OpenMP_IfClause,
+    OpenMP_InReductionClauseSkip<assemblyFormat = true>,
     OpenMP_MergeableClause, OpenMP_PriorityClause, OpenMP_PrivateClause,
     OpenMP_UntiedClause, OpenMP_DetachClause
   ], singleRegion = true> {
@@ -1012,9 +1013,10 @@ def TaskloopContextOp : OpenMP_Op<"taskloop.context", traits = [
     DeclareOpInterfaceMethods<OutlineableOpenMPOpInterface>
   ], clauses = [
     OpenMP_AllocateClause, OpenMP_FinalClause, OpenMP_GrainsizeClause,
-    OpenMP_IfClause, OpenMP_InReductionClause, OpenMP_MergeableClause,
-    OpenMP_NogroupClause, OpenMP_NumTasksClause, OpenMP_PriorityClause,
-    OpenMP_PrivateClause, OpenMP_ReductionClause, OpenMP_UntiedClause
+    OpenMP_IfClause, OpenMP_InReductionClauseSkip<assemblyFormat = true>,
+    OpenMP_MergeableClause, OpenMP_NogroupClause, OpenMP_NumTasksClause,
+    OpenMP_PriorityClause, OpenMP_PrivateClause, OpenMP_ReductionClause,
+    OpenMP_UntiedClause
   ], singleRegion = true> {
   let summary = "OutlinableOpenMPOpInterface wrapper for taskloop construct";
   let description = [{
@@ -1627,6 +1629,15 @@ def TargetOp : OpenMP_Op<"target", traits = [
   ];
 
   let extraClassDeclaration = [{
+    // Override BlockArgOpenMPOpInterface method because `in_reduction` list
+    // items on `omp.target` do not define entry block arguments. The reduction
+    // variable is accessed inside the target body through its matching
+    // `map_entries` block argument; the `in_reduction` operands are kept only
+    // as host-side metadata used to look up the per-task private storage.
+    unsigned numInReductionBlockArgs() {
+      return 0;
+    }
+
     mlir::Value getMappedValueForPrivateVar(unsigned privVarIdx) {
       std::optional<DenseI64ArrayAttr> privateMapIdices = getPrivateMapsAttr();
 
@@ -1651,8 +1662,7 @@ def TargetOp : OpenMP_Op<"target", traits = [
     " | `map_iterated` `(` $map_iterated `:` type($map_iterated) `)`)" # [{
       custom<TargetOpRegion>(
           $region, $has_device_addr_vars, type($has_device_addr_vars),
-          $host_eval_vars, type($host_eval_vars), $in_reduction_vars,
-          type($in_reduction_vars), $in_reduction_byref, $in_reduction_syms,
+          $host_eval_vars, type($host_eval_vars),
           $map_vars, type($map_vars), $private_vars, type($private_vars),
           $private_syms, $private_needs_barrier, $private_maps) attr-dict
     }];
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 41d3c547ca618..8b07f08ebe5eb 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1038,6 +1038,82 @@ static void printDynGroupprivateClause(OpAsmPrinter &printer, Operation *op,
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Parser and printer for in_reduction Clause
+//===----------------------------------------------------------------------===//
+
+/// Parses an `in_reduction` clause for an operation that does not give its
+/// list items entry block arguments (e.g. `omp.target`). The expected format is
+/// a comma-separated list of `[byref] @sym %var` followed by `: types`.
+static ParseResult parseInReductionClause(
+    OpAsmParser &parser,
+    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
+    SmallVectorImpl<Type> &inReductionTypes,
+    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms) {
+  SmallVector<SymbolRefAttr> symbolVec;
+  SmallVector<bool> isByRefVec;
+
+  if (parser.parseCommaSeparatedList([&]() {
+        isByRefVec.push_back(parser.parseOptionalKeyword("byref").succeeded());
+        if (parser.parseAttribute(symbolVec.emplace_back()) ||
+            parser.parseOperand(inReductionVars.emplace_back()))
+          return failure();
+        return success();
+      }))
+    return failure();
+
+  if (parser.parseColon())
+    return failure();
+
+  if (parser.parseCommaSeparatedList(
+          [&]() { return parser.parseType(inReductionTypes.emplace_back()); }))
+    return failure();
+
+  if (inReductionVars.size() != inReductionTypes.size())
+    return failure();
+
+  inReductionByref = makeDenseBoolArrayAttr(parser.getContext(), isByRefVec);
+  SmallVector<Attribute> symbolAttrs(symbolVec.begin(), symbolVec.end());
+  inReductionSyms = ArrayAttr::get(parser.getContext(), symbolAttrs);
+  return success();
+}
+
+/// Prints an `in_reduction` clause for an operation that does not give its list
+/// items entry block arguments (e.g. `omp.target`). Mirrors
+/// `parseInReductionClause`.
+static void printInReductionClause(OpAsmPrinter &p, Operation *op,
+                                   ValueRange inReductionVars,
+                                   TypeRange inReductionTypes,
+                                   DenseBoolArrayAttr inReductionByref,
+                                   ArrayAttr inReductionSyms) {
+  MLIRContext *ctx = op->getContext();
+
+  ArrayAttr syms = inReductionSyms;
+  if (!syms) {
+    SmallVector<Attribute> values(inReductionVars.size(), nullptr);
+    syms = ArrayAttr::get(ctx, values);
+  }
+
+  DenseBoolArrayAttr byref = inReductionByref;
+  if (!byref) {
+    SmallVector<bool> values(inReductionVars.size(), false);
+    byref = DenseBoolArrayAttr::get(ctx, values);
+  }
+
+  llvm::interleaveComma(
+      llvm::zip_equal(inReductionVars, syms.getValue(), byref.asArrayRef()), p,
+      [&p](auto t) {
+        auto [var, sym, isByRef] = t;
+        if (isByRef)
+          p << "byref ";
+        if (sym)
+          p << sym << " ";
+        p << var;
+      });
+  p << " : ";
+  llvm::interleaveComma(inReductionTypes, p);
+}
+
 //===----------------------------------------------------------------------===//
 // Parsers for operations including clauses that define entry block arguments.
 //===----------------------------------------------------------------------===//
@@ -1305,9 +1381,6 @@ static ParseResult parseTargetOpRegion(
     SmallVectorImpl<Type> &hasDeviceAddrTypes,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &hostEvalVars,
     SmallVectorImpl<Type> &hostEvalTypes,
-    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
-    SmallVectorImpl<Type> &inReductionTypes,
-    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &mapVars,
     SmallVectorImpl<Type> &mapTypes,
     llvm::SmallVectorImpl<OpAsmParser::UnresolvedOperand> &privateVars,
@@ -1316,8 +1389,6 @@ static ParseResult parseTargetOpRegion(
   AllRegionParseArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, &privateMaps);
@@ -1568,20 +1639,18 @@ static void printBlockArgRegion(OpAsmPrinter &p, Operation *op, Region &region,
 
 // These parseXyz functions correspond to the custom<Xyz> definitions
 // in the .td file(s).
-static void printTargetOpRegion(
-    OpAsmPrinter &p, Operation *op, Region &region,
-    ValueRange hasDeviceAddrVars, TypeRange hasDeviceAddrTypes,
-    ValueRange hostEvalVars, TypeRange hostEvalTypes,
-    ValueRange inReductionVars, TypeRange inReductionTypes,
-    DenseBoolArrayAttr inReductionByref, ArrayAttr inReductionSyms,
-    ValueRange mapVars, TypeRange mapTypes, ValueRange privateVars,
-    TypeRange privateTypes, ArrayAttr privateSyms, UnitAttr privateNeedsBarrier,
-    DenseI64ArrayAttr privateMaps) {
+static void printTargetOpRegion(OpAsmPrinter &p, Operation *op, Region &region,
+                                ValueRange hasDeviceAddrVars,
+                                TypeRange hasDeviceAddrTypes,
+                                ValueRange hostEvalVars,
+                                TypeRange hostEvalTypes, ValueRange mapVars,
+                                TypeRange mapTypes, ValueRange privateVars,
+                                TypeRange privateTypes, ArrayAttr privateSyms,
+                                UnitAttr privateNeedsBarrier,
+                                DenseI64ArrayAttr privateMaps) {
   AllRegionPrintArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, privateMaps);
@@ -2579,8 +2648,7 @@ LogicalResult TargetUpdateOp::verify() {
 void TargetOp::build(OpBuilder &builder, OperationState &state,
                      const TargetExtOperands &clauses) {
   MLIRContext *ctx = builder.getContext();
-  // TODO Store clauses in op: allocateVars, allocatorVars, inReductionVars,
-  // inReductionByref, inReductionSyms.
+  // TODO Store clauses in op: allocateVars, allocatorVars.
   TargetOp::build(
       builder, state, /*allocate_vars=*/{}, /*allocator_vars=*/{},
       makeArrayAttr(ctx, clauses.dependKinds), clauses.dependVars,
@@ -2588,9 +2656,10 @@ void TargetOp::build(OpBuilder &builder, OperationState &state,
       clauses.device, clauses.dynGroupprivateAccessGroup,
       clauses.dynGroupprivateFallback, clauses.dynGroupprivateSize,
       clauses.hasDeviceAddrVars, clauses.hostEvalVars, clauses.ifExpr,
-      /*in_reduction_vars=*/{}, /*in_reduction_byref=*/nullptr,
-      /*in_reduction_syms=*/nullptr, clauses.isDevicePtrVars, clauses.mapVars,
-      clauses.mapIterated, clauses.nowait, clauses.privateVars,
+      clauses.inReductionVars,
+      makeDenseBoolArrayAttr(ctx, clauses.inReductionByref),
+      makeArrayAttr(ctx, clauses.inReductionSyms), clauses.isDevicePtrVars,
+      clauses.mapVars, clauses.mapIterated, clauses.nowait, clauses.privateVars,
       makeArrayAttr(ctx, clauses.privateSyms), clauses.privateNeedsBarrier,
       clauses.threadLimitVars, /*private_maps=*/nullptr, clauses.kernelType);
 }
@@ -2631,6 +2700,21 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `targetInReductionCapturedBy` in
+/// OpenMPToLLVMIRTranslation.cpp so translation matches what the verifier
+/// accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 LogicalResult TargetOp::verify() {
   if (getKernelType() == TargetExecMode::bare && !isCombined())
     return emitOpError() << "bare kernel requires 'omp.combined'";
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (mapInfo &&
+          targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
+    }
+    if (!captured)
+      return emitOpError() << "in_reduction variable must be captured by a "
+                              "matching map_entries entry";
+  }
+
   return verifyPrivateVarsMapping(*this);
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b2d2ef6d8f770..367be0692054c 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -364,9 +364,40 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       op.emitWarning("hint clause discarded");
   };
   auto checkInReduction = [&todo](auto op, LogicalResult &result) {
-    if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
-        op.getInReductionSyms())
+    if (isa<omp::TargetOp, omp::TaskOp, omp::TaskloopContextOp>(
+            op.getOperation())) {
+      if (auto byrefAttr = op.getInReductionByref()) {
+        for (bool isByRef : *byrefAttr) {
+          if (isByRef) {
+            result = todo("in_reduction with byref modifier");
+            return;
+          }
+        }
+      }
+      if (isa<omp::TargetOp>(op.getOperation())) {
+        if (auto inReductionSyms = op.getInReductionSyms()) {
+          for (auto sym :
+               (*inReductionSyms).template getAsRange<SymbolRefAttr>()) {
+            auto decl =
+                SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                    op, sym);
+            assert(decl &&
+                   "symbol resolution should be guaranteed by the op verifier");
+            if (decl.getInitializerRegion().front().getNumArguments() != 1) {
+              result = todo("in_reduction with two-argument initializer");
+              return;
+            }
+            if (!decl.getCleanupRegion().empty()) {
+              result = todo("in_reduction with cleanup region");
+              return;
+            }
+          }
+        }
+      }
+    } else if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
+               op.getInReductionSyms()) {
       result = todo("in_reduction");
+    }
   };
   auto checkNowait = [&todo](auto op, LogicalResult &result) {
     if (op.getNowait())
@@ -423,14 +454,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
           return;
         }
   };
-  auto checkInReductionByref = [&todo](auto op, LogicalResult &result) {
-    if (auto byrefAttr = op.getInReductionByref())
-      for (bool isByRef : *byrefAttr)
-        if (isByRef) {
-          result = todo("in_reduction with byref modifier");
-          return;
-        }
-  };
   auto checkNumTeams = [&todo](auto op, LogicalResult &result) {
     if (op.hasNumTeamsMultiDim())
       result = todo("num_teams with multi-dimensional values");
@@ -482,7 +505,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
       })
       .Case([&](omp::TaskgroupOp op) {
         checkAllocate(op, result);
@@ -494,7 +517,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
         checkReduction(op, result);
         checkReductionByref(op, result);
       })
@@ -8435,6 +8458,20 @@ getDynGroupprivateFallbackType(omp::FallbackModifierAttr fallbackAttr) {
   llvm_unreachable("unexpected dyn_groupprivate fallback type");
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `omp::TargetOp::verify()` in OpenMPDialect.cpp so
+/// translation matches exactly what the verifier accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 static LogicalResult
 convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
                  LLVM::ModuleTranslation &moduleTranslation) {
@@ -8493,6 +8530,59 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand,
+      // using the same predicate as omp::TargetOp::verify(): exact same value,
+      // or another result of the same defining op. The verifier guarantees at
+      // least one match exists; more than one matching entry is a lowering
+      // ambiguity (the redirect cannot pick which map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (!mapInfo || !targetInReductionCapturedBy(v, mapInfo.getVarPtr()))
+          continue;
+        if (matchIdx)
+          return targetOp.emitError()
+                 << "in_reduction variable on omp.target has multiple matching "
+                    "map_entries entries; the redirect target is ambiguous";
+        matchIdx = idx;
+      }
+      // The verifier guarantees a match, so this must hold. The verifier and
+      // this predicate agree, so a verifier-accepted op never reaches an empty
+      // match here.
+      assert(matchIdx &&
+             "TargetOp verifier guarantees a matching map_entries entry for "
+             "each in_reduction variable");
+      inRedMapArgIdx.push_back(*matchIdx);
+      // The runtime `orig` pointer is the in_reduction operand itself (the
+      // reduction variable the enclosing taskgroup registered), not the map
+      // var_ptr. For the same-defining-op case both point to the same storage,
+      // but the in_reduction operand is the canonical reduction variable.
+      inRedOrigPtrs.push_back(moduleTranslation.lookupValue(v));
+    }
+  }
+
   // For some private variables, the MapsForPrivatizedVariablesPass
   // creates MapInfoOp instances. Go through the private variables and
   // the mapped variables so that during codegeneration we are able
@@ -8568,8 +8658,15 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
         attr.isStringAttribute())
       llvmOutlinedFn->addFnAttr(attr);
 
-    for (auto [arg, mapOp] : llvm::zip_equal(mapBlockArgs, mapVars)) {
-      auto mapInfoOp = cast<omp::MapInfoOp>(mapOp.getDefiningOp());
+    for (auto [idx, arg] : llvm::enumerate(mapBlockArgs)) {
+      // in_reduction list items on omp.target are accessed through their
+      // map_entries block argument, which is redirected below to the per-task
+      // reduction-private storage returned by the runtime. Skip the default
+      // host-value mapping for those block arguments so the write-once
+      // mapValue mapping is free to be set to the private pointer.
+      if (llvm::is_contained(inRedMapArgIdx, idx))
+        continue;
+      auto mapInfoOp = cast<omp::MapInfoOp>(mapVars[idx].getDefiningOp());
       llvm::Value *mapOpValue =
           moduleTranslation.lookupValue(mapInfoOp.getVarPtr());
       moduleTranslation.mapValue(arg, mapOpValue);
@@ -8605,6 +8702,42 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // OpenMPIRBuilder owns the runtime-generation path: it computes the
+      // target body gtid once (so several in_reduction items do not each emit a
+      // __kmpc_global_thread_num) and emits the per-item
+      // __kmpc_task_reduction_get_th_data lookup with the required
+      // address-space casts. The MLIR-specific work stays here: collect, per
+      // item, the type the private pointer must have (the map block argument's
+      // type), and, through the callback, rebind the map block argument that
+      // stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
+      SmallVector<llvm::Type *> inRedResultPtrTys;
+      inRedResultPtrTys.reserve(inRedMapArgIdx.size());
+      for (unsigned mapArgIdx : inRedMapArgIdx)
+        inRedResultPtrTys.push_back(
+            moduleTranslation.convertType(mapBlockArgs[mapArgIdx].getType()));
+
+      llvm::OpenMPIRBuilder::LocationDescription bodyLoc(builder);
+      llvm::OpenMPIRBuilder::InsertPointTy redIP =
+          ompBuilder->createTargetInReduction(
+              bodyLoc, inRedOrigPtrs, inRedResultPtrTys,
+              [&](unsigned idx, llvm::Value *priv) {
+                moduleTranslation.mapValue(mapBlockArgs[inRedMapArgIdx[idx]],
+                                           priv);
+              });
+      builder.restoreIP(redIP);
+    }
+
     LLVM::ModuleTranslation::SaveStack<OpenMPAllocStackFrame> frame(
         moduleTranslation, allocaIP, deallocBlocks);
     llvm::Expected<llvm::BasicBlock *> exitBlock = convertOmpOpRegions(
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
index 015e901126a76..8108a1af31e21 100644
--- a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -348,8 +348,12 @@ module attributes {omp.is_target_device = true} {
       omp.terminator
     }
 
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
-    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+    // The `in_reduction` list item on `omp.target` has no dedicated block
+    // argument; it is captured by a matching `map_entries` entry.
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 : !llvm.ptr) map_entries(%1 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
       omp.terminator
     }
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 2db7f953fa3bb..57270ec0694c3 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3255,6 +3255,91 @@ func.func @omp_target_depend(%data_var: memref<i32>) {
 
 // -----
 
+func.func @omp_target_in_reduction_unresolved(%ptr: !llvm.ptr) {
+  // expected-error @below {{op expected symbol reference @add_f32 to point to a reduction declaration}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_duplicate(%ptr: !llvm.ptr) {
+  // expected-error @below {{op accumulator variable used more than once}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr, @add_f32 %ptr : !llvm.ptr, !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg: i32):
+  %0 = arith.constant 0 : i32
+  omp.yield (%0 : i32)
+}
+combiner {
+^bb1(%arg0: i32, %arg1: i32):
+  %1 = arith.addi %arg0, %arg1 : i32
+  omp.yield (%1 : i32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> i32
+  llvm.atomicrmw add %arg2, %2 monotonic : !llvm.ptr, i32
+  omp.yield
+}
+
+func.func @omp_target_in_reduction_type_mismatch(%mem: memref<1xf32>) {
+  // expected-error @below {{op expected accumulator ('memref<1xf32>') to be the same type as reduction declaration ('!llvm.ptr')}}
+  omp.target kernel_type(generic) in_reduction(@add_i32 %mem : memref<1xf32>) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_missing_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_missing_map(%ptr: !llvm.ptr) {
+  // An in_reduction operand on omp.target has no dedicated block argument; it
+  // is accessed inside the body through a matching map_entries block argument.
+  // With no map_entries entry capturing it, the verifier must reject the op.
+  // expected-error @below {{in_reduction variable must be captured by a matching map_entries entry}}
+  omp.target kernel_type(generic) in_reduction(@add_missing_map_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
 func.func @omp_distribute_schedule(%chunk_size : i32, %lb : i32, %ub : i32, %step : i32) -> () {
   // expected-error @below {{op chunk size set without dist_schedule_static being present}}
   "omp.distribute"(%chunk_size) <{operandSegmentSizes = array<i32: 0, 0, 1, 0>}> ({
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
new file mode 100644
index 0000000000000..8cb719c43e34e
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
@@ -0,0 +1,51 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// in_reduction on omp.target compiled for the target *device*. From the
+// device's perspective an in_reduction list item behaves as a regular mapped
+// variable: the host-only redirect to the per-task reduction-private storage
+// (the __kmpc_global_thread_num + __kmpc_task_reduction_get_th_data lookup) is
+// guarded by !isTargetDevice and is therefore not emitted. Verify the device
+// body accesses the mapped variable directly and that none of the host-fallback
+// runtime lookup is present.
+
+module attributes {dlti.dl_spec = #dlti.dl_spec<"dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  omp.declare_reduction @add_i32 : i32
+  init {
+  ^bb0(%arg0: i32):
+    %c0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%c0 : i32)
+  }
+  combiner {
+  ^bb0(%arg0: i32, %arg1: i32):
+    %s = llvm.add %arg0, %arg1 : i32
+    omp.yield(%s : i32)
+  }
+
+  llvm.func @target_inreduction_device(%x : !llvm.ptr) {
+    %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+      %v = llvm.load %marg : !llvm.ptr -> i32
+      %c1 = llvm.mlir.constant(1 : i32) : i32
+      %s = llvm.add %v, %c1 : i32
+      llvm.store %s, %marg : i32, !llvm.ptr
+      omp.terminator
+    }
+    llvm.return
+  }
+}
+
+// The device kernel receives the mapped pointer as a captured argument and
+// accesses the in_reduction variable directly through it.
+// CHECK-LABEL: define {{.*}}amdgpu_kernel void @__omp_offloading_{{.*}}_target_inreduction_device_
+// CHECK-SAME:    (ptr %[[ARG:.+]], ptr %{{.+}})
+// CHECK:         store ptr %[[ARG]], ptr %[[ALLOCA:.+]]
+// CHECK:         %[[MAPPED:.+]] = load ptr, ptr %[[ALLOCA]]
+// The host-only in_reduction redirect must not run on the device: there is no
+// gtid lookup, no __kmpc_task_reduction_get_th_data call and no private copy.
+// CHECK-NOT:     @__kmpc_global_thread_num
+// CHECK-NOT:     @__kmpc_task_reduction_get_th_data
+// CHECK-NOT:     omp.inred.priv
+// The body reads and writes the mapped variable in place.
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[MAPPED]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[MAPPED]]
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
new file mode 100644
index 0000000000000..b438888b6c815
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Multiple in_reduction items on omp.target. Each item is captured into the
+// target region through its own map_entries entry and accessed inside the
+// body via the corresponding map_entries block argument. For the host
+// fallback path every item performs an independent
+// __kmpc_task_reduction_get_th_data lookup using its own captured original
+// pointer, and the returned per-task private pointer is bound to that item's
+// map block argument. This test pins down the pairing so it cannot pass if the
+// two items were swapped or collapsed onto a single pointer.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_multi(%x : !llvm.ptr, %y : !llvm.ptr) {
+  %mx = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %my = omp.map.info var_ptr(%y : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x, @add_i32 %y : !llvm.ptr, !llvm.ptr)
+      map_entries(%mx -> %mxarg, %my -> %myarg : !llvm.ptr, !llvm.ptr) {
+    // First item (x): load, += 1, store back.
+    %vx = llvm.load %mxarg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %sx = llvm.add %vx, %c1 : i32
+    llvm.store %sx, %mxarg : i32, !llvm.ptr
+    // Second item (y): load, += 2, store back.
+    %vy = llvm.load %myarg : !llvm.ptr -> i32
+    %c2 = llvm.mlir.constant(2 : i32) : i32
+    %sy = llvm.add %vy, %c2 : i32
+    llvm.store %sy, %myarg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards both captured pointers into the outlined target
+// kernel (the trailing argument is the unused descriptor slot).
+// CHECK-LABEL: define void @target_inreduction_multi(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_multi_{{.*}}(ptr %{{.+}}, ptr %{{.+}}, ptr null)
+
+// The two captured original pointers arrive as distinct kernel arguments.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_multi_
+// CHECK-SAME:    (ptr %[[CAPTX:.+]], ptr %[[CAPTY:.+]], ptr %{{.+}})
+
+// A single gtid is computed once for the whole target body and shared by both
+// lookups; each item then performs its own __kmpc_task_reduction_get_th_data
+// call against its own captured pointer, and the returned per-task private
+// pointer is bound to that item's map block argument.
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIVX:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTX]])
+// CHECK:         %[[PRIVY:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTY]])
+
+// The first item's private storage is the base of the +1 load/store; the
+// CHECK-NOT below ensures the second item's pointer is not touched until the
+// first item's accumulation has completed (i.e. the items are not swapped or
+// merged onto a single private pointer).
+// CHECK:         %[[LX:.+]] = load i32, ptr %[[PRIVX]]
+// CHECK:         %[[SX:.+]] = add i32 %[[LX]], 1
+// CHECK-NOT:     %[[PRIVY]]
+// CHECK:         store i32 %[[SX]], ptr %[[PRIVX]]
+
+// The second item's private storage is the base of the +2 load/store.
+// CHECK:         %[[LY:.+]] = load i32, ptr %[[PRIVY]]
+// CHECK:         %[[SY:.+]] = add i32 %[[LY]], 2
+// CHECK:         store i32 %[[SY]], ptr %[[PRIVY]]
+
+// Exactly two reduction lookups are emitted; no third call sneaks in. The
+// `call` form is used so this does not match the runtime declaration.
+// CHECK-NOT:     call ptr @__kmpc_task_reduction_get_th_data
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
new file mode 100644
index 0000000000000..124c04dddedda
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+// in_reduction on omp.target: the in_reduction variable is also captured
+// into the target region as a map entry (a frontend is expected to emit this
+// implicit map). The in_reduction clause does not define an entry block
+// argument; inside the target body the variable is accessed through its
+// map_entries block argument. The captured pointer is passed to
+// __kmpc_task_reduction_get_th_data with a NULL descriptor; the runtime
+// walks enclosing taskgroups to locate the matching task_reduction
+// registration. The returned per-task private pointer is bound to the
+// map_entries block argument so subsequent loads/stores inside the region
+// use the private copy.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+    %v = llvm.load %marg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards the captured pointer into the outlined target
+// kernel.
+// CHECK-LABEL: define void @target_inreduction(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_{{.*}}(ptr %{{.+}}, ptr null)
+
+// In the outlined target body the in_reduction private pointer is
+// obtained from the runtime using the captured original pointer; that
+// pointer is then the base of the load and store inside the region.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_
+// CHECK-SAME:    (ptr %[[CAPT:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPT]])
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[PRIV]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+// -----
+
+// Same as the first case but the in_reduction variable lives in a non-default
+// address space (addrspace(1)). __kmpc_task_reduction_get_th_data is declared
+// to take and return a generic (default-AS) `ptr`, so the host-fallback
+// lowering must (1) addrspacecast the captured addrspace(1) original pointer to
+// generic `ptr` before the runtime lookup, and (2) addrspacecast the returned
+// generic private pointer back to addrspace(1) so the body's load/store use the
+// private copy in the right address space. This pins down the address-space
+// handling so a regression that passed the addrspace(1) pointer straight into
+// the runtime call (a bad-signature crash) cannot pass.
+
+omp.declare_reduction @add_i32_as1 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_as1(%x : !llvm.ptr<1>) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr<1>, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr<1>
+  omp.target kernel_type(generic) in_reduction(@add_i32_as1 %x : !llvm.ptr<1>) map_entries(%m -> %marg : !llvm.ptr<1>) {
+    %v = llvm.load %marg : !llvm.ptr<1> -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr<1>
+    omp.terminator
+  }
+  llvm.return
+}
+
+// In the outlined target body the addrspace(1) original pointer is normalized
+// to generic `ptr` before the NULL-descriptor in_reduction lookup, and the
+// returned generic private pointer is cast back to addrspace(1) for the body's
+// load and store. The original captured pointer arrives as the addrspace(1)
+// kernel argument and is only used to derive the generic lookup pointer.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_as1_
+// CHECK-SAME:    (ptr addrspace(1) %[[CAPT_AS1:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID_AS1:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[ORIG_GEN:.+]] = addrspacecast ptr addrspace(1) %[[CAPT_AS1]] to ptr
+// CHECK:         %[[PRIV_AS1_GEN:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID_AS1]], ptr null, ptr %[[ORIG_GEN]])
+// CHECK:         %[[PRIV_AS1:.+]] = addrspacecast ptr %[[PRIV_AS1_GEN]] to ptr addrspace(1)
+// CHECK:         %[[LOADED_AS1:.+]] = load i32, ptr addrspace(1) %[[PRIV_AS1]]
+// CHECK:         %[[SUM_AS1:.+]] = add i32 %[[LOADED_AS1]], 1
+// CHECK:         store i32 %[[SUM_AS1]], ptr addrspace(1) %[[PRIV_AS1]]
+
+// The body must not load or store through the original captured addrspace(1)
+// pointer; all accesses go through the runtime-returned private copy.
+// CHECK-NOT:     store i32 %{{.+}}, ptr addrspace(1) %[[CAPT_AS1]]
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..943b5bad6970f 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -218,10 +218,95 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @target_in_reduction(%x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause in_reduction in omp.target operation}}
+llvm.func @target_in_reduction_byref(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with byref modifier in omp.target operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.target}}
-  omp.target kernel_type(generic) in_reduction(@add_f32 %x -> %prv : !llvm.ptr) {
+  omp.target kernel_type(generic) in_reduction(byref @add_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_cleanup_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+cleanup {
+^bb2(%arg2: f32):
+  omp.yield
+}
+llvm.func @target_in_reduction_cleanup(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with cleanup region in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_cleanup_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_two_arg_init_i32 : !llvm.ptr alloc {
+^bb0(%arg: !llvm.ptr):
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+  omp.yield(%1 : !llvm.ptr)
+} init {
+^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.store %0, %arg1 : i32, !llvm.ptr
+  omp.yield(%arg1 : !llvm.ptr)
+} combiner {
+^bb1(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.load %arg0 : !llvm.ptr -> i32
+  %1 = llvm.load %arg1 : !llvm.ptr -> i32
+  %2 = llvm.add %0, %1 : i32
+  llvm.store %2, %arg0 : i32, !llvm.ptr
+  omp.yield(%arg0 : !llvm.ptr)
+}
+llvm.func @target_in_reduction_two_arg_init(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with two-argument initializer in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_two_arg_init_i32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_dup_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb0(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+llvm.func @target_in_reduction_duplicate_map(%x : !llvm.ptr) {
+  // The in_reduction variable %x has two matching map_entries entries for the
+  // same var_ptr. The translation cannot disambiguate which map block argument
+  // to redirect, so it must reject this as ambiguous.
+  %m1 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %m2 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{in_reduction variable on omp.target has multiple matching map_entries entries; the redirect target is ambiguous}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_dup_map_f32 %x : !llvm.ptr) map_entries(%m1 -> %arg1, %m2 -> %arg2 : !llvm.ptr, !llvm.ptr) {
     omp.terminator
   }
   llvm.return


From llvmlistbot at llvm.org  Thu Jul  9 20:54:00 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 09 Jul 2026 20:54:00 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a506cd8.eb24c6ea.27aedc.f041@mx.google.com>


Saieiei wrote:

Done!


https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Thu Jul  9 20:54:10 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 09 Jul 2026 20:54:10 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a506ce2.331483e7.2440f8.ed19@mx.google.com>


Saieiei wrote:

Updated the commit message to reflect the current behavior and explicitly list the remaining diagnosed cases. Thanks!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Thu Jul  9 21:55:59 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 09 Jul 2026 21:55:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a507b5f.a1accbb4.244acd.ac1f@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/205684

>From e7ea262a41af79d3f18f497a20fff69ba40bf6a1 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 24 Jun 2026 21:22:36 +0000
Subject: [PATCH 1/2] relaxed UnrollShapeCastPattern in VectorUnroll.cpp so it
 unrolls shape_casts whose target tile is contiguous per reassociation group
 rather than requiring it to be contiguous across the entire result vector

---
 .../Vector/Transforms/VectorUnroll.cpp        | 119 +++++++++++++++---
 .../Dialect/Vector/vector-unroll-options.mlir |  44 +++++++
 .../Dialect/Vector/TestVectorTransforms.cpp   |  11 ++
 3 files changed, 154 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index b33828d5d5867..c3dc132195ef5 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -1285,20 +1285,63 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
   return delinearize(linearIndex, computeStrides(sourceShape));
 }
 
+/// A maximal aligned range of source dims [srcBegin, srcEnd) and result dims
+/// [resBegin, resEnd) of a `vector.shape_cast` that hold equal element counts.
+struct ShapeCastReassociationGroup {
+  int64_t srcBegin, srcEnd;
+  int64_t resBegin, resEnd;
+};
+
+/// Splits a shape_cast from `sourceShape` to `resultShape` into reassociation
+/// groups (trailing unit dims absorbed). Returns nullopt if shapes misalign.
+/// E.g. [8, 32, 32] -> [256, 32] ==> {[0,2)->[0,1)}, {[2,3)->[1,2)}
+static std::optional<SmallVector<ShapeCastReassociationGroup>>
+computeShapeCastGroups(ArrayRef<int64_t> sourceShape,
+                       ArrayRef<int64_t> resultShape) {
+  SmallVector<ShapeCastReassociationGroup> groups;
+  int64_t si = 0, ri = 0;
+  int64_t srcRank = sourceShape.size(), resRank = resultShape.size();
+  while (si < srcRank && ri < resRank) {
+    int64_t srcBegin = si, resBegin = ri;
+    int64_t srcProd = sourceShape[si++];
+    int64_t resProd = resultShape[ri++];
+    // Grow the smaller side until both groups span the same element count.
+    while (srcProd != resProd) {
+      if (srcProd < resProd) {
+        if (si >= srcRank)
+          return std::nullopt;
+        srcProd *= sourceShape[si++];
+      } else {
+        if (ri >= resRank)
+          return std::nullopt;
+        resProd *= resultShape[ri++];
+      }
+    }
+    // Absorb trailing unit dimensions into the current group.
+    while (si < srcRank && sourceShape[si] == 1)
+      ++si;
+    while (ri < resRank && resultShape[ri] == 1)
+      ++ri;
+    groups.push_back({srcBegin, si, resBegin, ri});
+  }
+  if (si != srcRank || ri != resRank)
+    return std::nullopt;
+  return groups;
+}
+
 /// This pattern unrolls `vector.shape_cast` operations according to the
 /// provided target unroll shape. It unrolls a large shape cast into smaller
 /// shape casts by extracting contiguous slices from the source vector, casting
 /// each slice to the target shape, and assembling the result by inserting each
 /// computed segment into the appropriate offset of the result vector.
 ///
-/// This pattern only applies when contiguous slices can be extracted from the
-/// source vector and inserted into the result vector such that each slice
-/// remains a valid vector (and not decompose to scalars). In these cases, the
-/// unrolling proceeds as:
+/// The target tile need only be contiguous within each reassociation group of
+/// the cast (not in the whole result vector), so that each extracted slice
+/// remains a valid vector. The unrolling proceeds as:
 /// vector.extract_strided_slice -> vector.shape_cast (on the slice) ->
 /// vector.insert_strided_slice.
 ///
-/// Example:
+/// Example (single group):
 ///   Given a shape cast operation:
 ///     %0 = vector.shape_cast %src : vector<8x2xf32> to vector<4x4xf32>
 ///
@@ -1316,6 +1359,18 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
 ///     %i1 = vector.insert_strided_slice %sc1, %i0 [2, 0], [1, 1]
 ///       : vector<2x4xf32> into vector<4x4xf32>
 ///
+/// Example (multiple groups): with target tile <8x1x4>, the tile is strided in
+/// the result <8x1x32> but contiguous per group (8|32 -> 8x1|32), so the
+/// matching strided box <8x4> is extracted from the source:
+///     %0 = vector.shape_cast %src : vector<8x32xf32> to vector<8x1x32xf32>
+///
+///     %s0 = vector.extract_strided_slice %src [0, 0], [8, 4], [1, 1]
+///       : vector<8x32xf32> to vector<8x4xf32>
+///     %sc0 = vector.shape_cast %s0 : vector<8x4xf32> to vector<8x1x4xf32>
+///     %i0 = vector.insert_strided_slice %sc0, %zero [0, 0, 0], [1, 1, 1]
+///       : vector<8x1x4xf32> into vector<8x1x32xf32>
+///     // ... repeat for the remaining slices.
+///
 struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
   UnrollShapeCastPattern(MLIRContext *context,
                          const vector::UnrollVectorOptions &options,
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
+
+      ArrayRef<int64_t> srcSub =
+          sourceShape.slice(g.srcBegin, g.srcEnd - g.srcBegin);
+      int64_t groupTargetElements = ShapedType::getNumElements(tgtSub);
+      std::optional<SmallVector<int64_t>> groupExtract =
+          calculateSourceExtractShape(srcSub, groupTargetElements);
+      if (!groupExtract)
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "cannot extract the target number of elements "
+                         "contiguously from a source reassociation group");
+      extractShapeStorage.append(groupExtract->begin(), groupExtract->end());
+    }
+    ArrayRef<int64_t> extractShape = extractShapeStorage;
 
     Location loc = shapeCastOp.getLoc();
 
@@ -1359,7 +1438,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     VectorType targetType =
         VectorType::get(*targetShape, sourceType.getElementType());
 
-    SmallVector<int64_t> extractStrides(extractShape->size(), 1);
+    SmallVector<int64_t> extractStrides(extractShape.size(), 1);
     SmallVector<int64_t> insertStrides(targetShape->size(), 1);
 
     for (SmallVector<int64_t> resultOffsets :
@@ -1367,7 +1446,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
       SmallVector<int64_t> sourceOffsets =
           calculateSourceOffsets(resultOffsets, sourceShape, resultShape);
       Value sourceChunk = rewriter.createOrFold<vector::ExtractStridedSliceOp>(
-          loc, shapeCastOp.getSource(), sourceOffsets, *extractShape,
+          loc, shapeCastOp.getSource(), sourceOffsets, extractShape,
           extractStrides);
       Value targetChunk = rewriter.createOrFold<vector::ShapeCastOp>(
           loc, targetType, sourceChunk);
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..58ca27b50e2ee 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -668,6 +668,50 @@ func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0], strides = [1, 1]} : vector<1x1xf32> into vector<2x1xf32>
 // CHECK:   return %[[I1]] : vector<2x1xf32>
 
+
+// Target tile [8, 1, 4] is strided in result <8x1x32> but contiguous per
+// reassociation group (8|32 -> 8x1|32). TargetShape is [8, 1, 4].
+func.func @shape_cast_multi_group_rank_increasing(%v: vector<8x32xf32>) -> vector<8x1x32xf32> {
+  %0 = vector.shape_cast %v : vector<8x32xf32> to vector<8x1x32xf32>
+  return %0 : vector<8x1x32xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_increasing
+// CHECK-SAME: (%[[V:.*]]: vector<8x32xf32>) -> vector<8x1x32xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<8x1x32xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 4], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 0, 4], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   return
+
+
+// Target tile [2, 2] is strided in result <4x4> but contiguous per
+// reassociation group (2x2|4 -> 4|4). TargetShape is [2, 2].
+func.func @shape_cast_multi_group_rank_decreasing(%v: vector<2x2x4xf32>) -> vector<4x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x2x4xf32> to vector<4x4xf32>
+  return %0 : vector<4x4xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_decreasing
+// CHECK-SAME: (%[[V:.*]]: vector<2x2x4xf32>) -> vector<4x4xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<4x4xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S2:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC2:.*]] = vector.shape_cast %[[S2]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[SC2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S3:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC3:.*]] = vector.shape_cast %[[S3]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I3:.*]] = vector.insert_strided_slice %[[SC3]], %[[I2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   return %[[I3]] : vector<4x4xf32>
+
 // -----
 
 // Test BitCastOp unrolling - target shape [4, 4]
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..af0e7e59eab63 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -206,6 +206,17 @@ struct TestVectorUnrollingPatterns
                       resultShape[1] == 1) {
                     return SmallVector<int64_t>{1, 1};
                   }
+                  // Multi-group cases: tile contiguous per reassociation group
+                  // but strided in the whole result.
+                  auto sourceShape = shapeCast.getSourceVectorType().getShape();
+                  if (resultShape.size() == 3 && resultShape[0] == 8 &&
+                      resultShape[1] == 1 && resultShape[2] == 32) {
+                    return SmallVector<int64_t>{8, 1, 4};
+                  }
+                  if (sourceShape.size() == 3 && resultShape.size() == 2 &&
+                      resultShape[0] == 4 && resultShape[1] == 4) {
+                    return SmallVector<int64_t>{2, 2};
+                  }
                   // Default case: [2,4] for all tests.
                   return SmallVector<int64_t>{2, 4};
                 })

>From f14d24ad2e43bc60719a7329c7e12b80848a7ecb Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 04:55:22 +0000
Subject: [PATCH 2/2] [MLIR][Vector] Add negative shape_cast unroll tests and
 NOP note

Add two negative tests for the new per-reassociation-group checks in
UnrollShapeCastPattern: one where the target tile is not contiguous
within a result group, and one where the tile cannot be extracted
contiguously from a source group. Also document that this unrolling
assumes the introduced strided slices lower to a NOP, per review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../Vector/Transforms/VectorUnroll.cpp        |  5 ++++
 .../Dialect/Vector/vector-unroll-options.mlir | 26 +++++++++++++++++++
 .../Dialect/Vector/TestVectorTransforms.cpp   |  4 +++
 3 files changed, 35 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index c3dc132195ef5..62869111496d1 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -1341,6 +1341,11 @@ computeShapeCastGroups(ArrayRef<int64_t> sourceShape,
 /// vector.extract_strided_slice -> vector.shape_cast (on the slice) ->
 /// vector.insert_strided_slice.
 ///
+/// NOTE: This replaces a NOP `vector.shape_cast` with strided slices. Per-group
+/// contiguity keeps those slices contiguous, so they are expected to lower to a
+/// NOP too. Targets where strided slices do not lower to a NOP should not use
+/// this pattern, or should pick a tile that avoids introducing such slices.
+///
 /// Example (single group):
 ///   Given a shape cast operation:
 ///     %0 = vector.shape_cast %src : vector<8x2xf32> to vector<4x4xf32>
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index 139794aeaf59b..14edc15d9e666 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -747,6 +747,32 @@ func.func @shape_cast_multi_group_rank_decreasing(%v: vector<2x2x4xf32>) -> vect
 // CHECK:   %[[I3:.*]] = vector.insert_strided_slice %[[SC3]], %[[I2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 // CHECK:   return %[[I3]] : vector<4x4xf32>
 
+// Negative multi-group case: target tile [2, 2, 2] is not contiguous within
+// the result reassociation group [8, 4], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_target_not_contiguous(%v: vector<2x32xf32>) -> vector<2x8x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x32xf32> to vector<2x8x4xf32>
+  return %0 : vector<2x8x4xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_target_not_contiguous
+// CHECK-SAME: (%[[V:.*]]: vector<2x32xf32>) -> vector<2x8x4xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x32xf32> to vector<2x8x4xf32>
+// CHECK:   return %[[SC]] : vector<2x8x4xf32>
+
+
+// Negative multi-group case: the target tile is contiguous within the result
+// group [24], but its elements cannot be extracted contiguously from the
+// source group [8, 3], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_source_not_determinable(%v: vector<2x8x3xf32>) -> vector<2x24xf32> {
+  %0 = vector.shape_cast %v : vector<2x8x3xf32> to vector<2x24xf32>
+  return %0 : vector<2x24xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_source_not_determinable
+// CHECK-SAME: (%[[V:.*]]: vector<2x8x3xf32>) -> vector<2x24xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x8x3xf32> to vector<2x24xf32>
+// CHECK:   return %[[SC]] : vector<2x24xf32>
+
 // -----
 
 // Test BitCastOp unrolling - target shape [4, 4]
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 52e3444fe6c3b..4523a4cd3c486 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -222,6 +222,10 @@ struct TestVectorUnrollingPatterns
                       resultShape[0] == 4 && resultShape[1] == 4) {
                     return SmallVector<int64_t>{2, 2};
                   }
+                  if (resultShape.size() == 3 && resultShape[0] == 2 &&
+                      resultShape[1] == 8 && resultShape[2] == 4) {
+                    return SmallVector<int64_t>{2, 2, 2};
+                  }
                   // Default case: [2,4] for all tests.
                   return SmallVector<int64_t>{2, 4};
                 })


From llvmlistbot at llvm.org  Thu Jul  9 22:02:11 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 09 Jul 2026 22:02:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a507cd3.af6ba3ed.36d9db.d8d6@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/205684

>From e7ea262a41af79d3f18f497a20fff69ba40bf6a1 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 24 Jun 2026 21:22:36 +0000
Subject: [PATCH 1/3] relaxed UnrollShapeCastPattern in VectorUnroll.cpp so it
 unrolls shape_casts whose target tile is contiguous per reassociation group
 rather than requiring it to be contiguous across the entire result vector

---
 .../Vector/Transforms/VectorUnroll.cpp        | 119 +++++++++++++++---
 .../Dialect/Vector/vector-unroll-options.mlir |  44 +++++++
 .../Dialect/Vector/TestVectorTransforms.cpp   |  11 ++
 3 files changed, 154 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index b33828d5d5867..c3dc132195ef5 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -1285,20 +1285,63 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
   return delinearize(linearIndex, computeStrides(sourceShape));
 }
 
+/// A maximal aligned range of source dims [srcBegin, srcEnd) and result dims
+/// [resBegin, resEnd) of a `vector.shape_cast` that hold equal element counts.
+struct ShapeCastReassociationGroup {
+  int64_t srcBegin, srcEnd;
+  int64_t resBegin, resEnd;
+};
+
+/// Splits a shape_cast from `sourceShape` to `resultShape` into reassociation
+/// groups (trailing unit dims absorbed). Returns nullopt if shapes misalign.
+/// E.g. [8, 32, 32] -> [256, 32] ==> {[0,2)->[0,1)}, {[2,3)->[1,2)}
+static std::optional<SmallVector<ShapeCastReassociationGroup>>
+computeShapeCastGroups(ArrayRef<int64_t> sourceShape,
+                       ArrayRef<int64_t> resultShape) {
+  SmallVector<ShapeCastReassociationGroup> groups;
+  int64_t si = 0, ri = 0;
+  int64_t srcRank = sourceShape.size(), resRank = resultShape.size();
+  while (si < srcRank && ri < resRank) {
+    int64_t srcBegin = si, resBegin = ri;
+    int64_t srcProd = sourceShape[si++];
+    int64_t resProd = resultShape[ri++];
+    // Grow the smaller side until both groups span the same element count.
+    while (srcProd != resProd) {
+      if (srcProd < resProd) {
+        if (si >= srcRank)
+          return std::nullopt;
+        srcProd *= sourceShape[si++];
+      } else {
+        if (ri >= resRank)
+          return std::nullopt;
+        resProd *= resultShape[ri++];
+      }
+    }
+    // Absorb trailing unit dimensions into the current group.
+    while (si < srcRank && sourceShape[si] == 1)
+      ++si;
+    while (ri < resRank && resultShape[ri] == 1)
+      ++ri;
+    groups.push_back({srcBegin, si, resBegin, ri});
+  }
+  if (si != srcRank || ri != resRank)
+    return std::nullopt;
+  return groups;
+}
+
 /// This pattern unrolls `vector.shape_cast` operations according to the
 /// provided target unroll shape. It unrolls a large shape cast into smaller
 /// shape casts by extracting contiguous slices from the source vector, casting
 /// each slice to the target shape, and assembling the result by inserting each
 /// computed segment into the appropriate offset of the result vector.
 ///
-/// This pattern only applies when contiguous slices can be extracted from the
-/// source vector and inserted into the result vector such that each slice
-/// remains a valid vector (and not decompose to scalars). In these cases, the
-/// unrolling proceeds as:
+/// The target tile need only be contiguous within each reassociation group of
+/// the cast (not in the whole result vector), so that each extracted slice
+/// remains a valid vector. The unrolling proceeds as:
 /// vector.extract_strided_slice -> vector.shape_cast (on the slice) ->
 /// vector.insert_strided_slice.
 ///
-/// Example:
+/// Example (single group):
 ///   Given a shape cast operation:
 ///     %0 = vector.shape_cast %src : vector<8x2xf32> to vector<4x4xf32>
 ///
@@ -1316,6 +1359,18 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
 ///     %i1 = vector.insert_strided_slice %sc1, %i0 [2, 0], [1, 1]
 ///       : vector<2x4xf32> into vector<4x4xf32>
 ///
+/// Example (multiple groups): with target tile <8x1x4>, the tile is strided in
+/// the result <8x1x32> but contiguous per group (8|32 -> 8x1|32), so the
+/// matching strided box <8x4> is extracted from the source:
+///     %0 = vector.shape_cast %src : vector<8x32xf32> to vector<8x1x32xf32>
+///
+///     %s0 = vector.extract_strided_slice %src [0, 0], [8, 4], [1, 1]
+///       : vector<8x32xf32> to vector<8x4xf32>
+///     %sc0 = vector.shape_cast %s0 : vector<8x4xf32> to vector<8x1x4xf32>
+///     %i0 = vector.insert_strided_slice %sc0, %zero [0, 0, 0], [1, 1, 1]
+///       : vector<8x1x4xf32> into vector<8x1x32xf32>
+///     // ... repeat for the remaining slices.
+///
 struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
   UnrollShapeCastPattern(MLIRContext *context,
                          const vector::UnrollVectorOptions &options,
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
+
+      ArrayRef<int64_t> srcSub =
+          sourceShape.slice(g.srcBegin, g.srcEnd - g.srcBegin);
+      int64_t groupTargetElements = ShapedType::getNumElements(tgtSub);
+      std::optional<SmallVector<int64_t>> groupExtract =
+          calculateSourceExtractShape(srcSub, groupTargetElements);
+      if (!groupExtract)
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "cannot extract the target number of elements "
+                         "contiguously from a source reassociation group");
+      extractShapeStorage.append(groupExtract->begin(), groupExtract->end());
+    }
+    ArrayRef<int64_t> extractShape = extractShapeStorage;
 
     Location loc = shapeCastOp.getLoc();
 
@@ -1359,7 +1438,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     VectorType targetType =
         VectorType::get(*targetShape, sourceType.getElementType());
 
-    SmallVector<int64_t> extractStrides(extractShape->size(), 1);
+    SmallVector<int64_t> extractStrides(extractShape.size(), 1);
     SmallVector<int64_t> insertStrides(targetShape->size(), 1);
 
     for (SmallVector<int64_t> resultOffsets :
@@ -1367,7 +1446,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
       SmallVector<int64_t> sourceOffsets =
           calculateSourceOffsets(resultOffsets, sourceShape, resultShape);
       Value sourceChunk = rewriter.createOrFold<vector::ExtractStridedSliceOp>(
-          loc, shapeCastOp.getSource(), sourceOffsets, *extractShape,
+          loc, shapeCastOp.getSource(), sourceOffsets, extractShape,
           extractStrides);
       Value targetChunk = rewriter.createOrFold<vector::ShapeCastOp>(
           loc, targetType, sourceChunk);
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index bb6fc4e38813d..58ca27b50e2ee 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -668,6 +668,50 @@ func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0], strides = [1, 1]} : vector<1x1xf32> into vector<2x1xf32>
 // CHECK:   return %[[I1]] : vector<2x1xf32>
 
+
+// Target tile [8, 1, 4] is strided in result <8x1x32> but contiguous per
+// reassociation group (8|32 -> 8x1|32). TargetShape is [8, 1, 4].
+func.func @shape_cast_multi_group_rank_increasing(%v: vector<8x32xf32>) -> vector<8x1x32xf32> {
+  %0 = vector.shape_cast %v : vector<8x32xf32> to vector<8x1x32xf32>
+  return %0 : vector<8x1x32xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_increasing
+// CHECK-SAME: (%[[V:.*]]: vector<8x32xf32>) -> vector<8x1x32xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<8x1x32xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 4], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 0, 4], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   return
+
+
+// Target tile [2, 2] is strided in result <4x4> but contiguous per
+// reassociation group (2x2|4 -> 4|4). TargetShape is [2, 2].
+func.func @shape_cast_multi_group_rank_decreasing(%v: vector<2x2x4xf32>) -> vector<4x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x2x4xf32> to vector<4x4xf32>
+  return %0 : vector<4x4xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_decreasing
+// CHECK-SAME: (%[[V:.*]]: vector<2x2x4xf32>) -> vector<4x4xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<4x4xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S2:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC2:.*]] = vector.shape_cast %[[S2]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[SC2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S3:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC3:.*]] = vector.shape_cast %[[S3]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I3:.*]] = vector.insert_strided_slice %[[SC3]], %[[I2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   return %[[I3]] : vector<4x4xf32>
+
 // -----
 
 // Test BitCastOp unrolling - target shape [4, 4]
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 043181c16c759..af0e7e59eab63 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -206,6 +206,17 @@ struct TestVectorUnrollingPatterns
                       resultShape[1] == 1) {
                     return SmallVector<int64_t>{1, 1};
                   }
+                  // Multi-group cases: tile contiguous per reassociation group
+                  // but strided in the whole result.
+                  auto sourceShape = shapeCast.getSourceVectorType().getShape();
+                  if (resultShape.size() == 3 && resultShape[0] == 8 &&
+                      resultShape[1] == 1 && resultShape[2] == 32) {
+                    return SmallVector<int64_t>{8, 1, 4};
+                  }
+                  if (sourceShape.size() == 3 && resultShape.size() == 2 &&
+                      resultShape[0] == 4 && resultShape[1] == 4) {
+                    return SmallVector<int64_t>{2, 2};
+                  }
                   // Default case: [2,4] for all tests.
                   return SmallVector<int64_t>{2, 4};
                 })

>From f14d24ad2e43bc60719a7329c7e12b80848a7ecb Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 04:55:22 +0000
Subject: [PATCH 2/3] [MLIR][Vector] Add negative shape_cast unroll tests and
 NOP note

Add two negative tests for the new per-reassociation-group checks in
UnrollShapeCastPattern: one where the target tile is not contiguous
within a result group, and one where the tile cannot be extracted
contiguously from a source group. Also document that this unrolling
assumes the introduced strided slices lower to a NOP, per review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../Vector/Transforms/VectorUnroll.cpp        |  5 ++++
 .../Dialect/Vector/vector-unroll-options.mlir | 26 +++++++++++++++++++
 .../Dialect/Vector/TestVectorTransforms.cpp   |  4 +++
 3 files changed, 35 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index c3dc132195ef5..62869111496d1 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -1341,6 +1341,11 @@ computeShapeCastGroups(ArrayRef<int64_t> sourceShape,
 /// vector.extract_strided_slice -> vector.shape_cast (on the slice) ->
 /// vector.insert_strided_slice.
 ///
+/// NOTE: This replaces a NOP `vector.shape_cast` with strided slices. Per-group
+/// contiguity keeps those slices contiguous, so they are expected to lower to a
+/// NOP too. Targets where strided slices do not lower to a NOP should not use
+/// this pattern, or should pick a tile that avoids introducing such slices.
+///
 /// Example (single group):
 ///   Given a shape cast operation:
 ///     %0 = vector.shape_cast %src : vector<8x2xf32> to vector<4x4xf32>
diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index 139794aeaf59b..14edc15d9e666 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -747,6 +747,32 @@ func.func @shape_cast_multi_group_rank_decreasing(%v: vector<2x2x4xf32>) -> vect
 // CHECK:   %[[I3:.*]] = vector.insert_strided_slice %[[SC3]], %[[I2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
 // CHECK:   return %[[I3]] : vector<4x4xf32>
 
+// Negative multi-group case: target tile [2, 2, 2] is not contiguous within
+// the result reassociation group [8, 4], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_target_not_contiguous(%v: vector<2x32xf32>) -> vector<2x8x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x32xf32> to vector<2x8x4xf32>
+  return %0 : vector<2x8x4xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_target_not_contiguous
+// CHECK-SAME: (%[[V:.*]]: vector<2x32xf32>) -> vector<2x8x4xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x32xf32> to vector<2x8x4xf32>
+// CHECK:   return %[[SC]] : vector<2x8x4xf32>
+
+
+// Negative multi-group case: the target tile is contiguous within the result
+// group [24], but its elements cannot be extracted contiguously from the
+// source group [8, 3], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_source_not_determinable(%v: vector<2x8x3xf32>) -> vector<2x24xf32> {
+  %0 = vector.shape_cast %v : vector<2x8x3xf32> to vector<2x24xf32>
+  return %0 : vector<2x24xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_source_not_determinable
+// CHECK-SAME: (%[[V:.*]]: vector<2x8x3xf32>) -> vector<2x24xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x8x3xf32> to vector<2x24xf32>
+// CHECK:   return %[[SC]] : vector<2x24xf32>
+
 // -----
 
 // Test BitCastOp unrolling - target shape [4, 4]
diff --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 52e3444fe6c3b..4523a4cd3c486 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -222,6 +222,10 @@ struct TestVectorUnrollingPatterns
                       resultShape[0] == 4 && resultShape[1] == 4) {
                     return SmallVector<int64_t>{2, 2};
                   }
+                  if (resultShape.size() == 3 && resultShape[0] == 2 &&
+                      resultShape[1] == 8 && resultShape[2] == 4) {
+                    return SmallVector<int64_t>{2, 2, 2};
+                  }
                   // Default case: [2,4] for all tests.
                   return SmallVector<int64_t>{2, 4};
                 })

>From ddce50d6943b3bed54d6e3456abb578cf6ffea38 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 05:01:55 +0000
Subject: [PATCH 3/3] polish comment in test

---
 mlir/test/Dialect/Vector/vector-unroll-options.mlir | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index 14edc15d9e666..da058d5d2410d 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -760,7 +760,7 @@ func.func @negative_shape_cast_multi_group_target_not_contiguous(%v: vector<2x32
 // CHECK:   return %[[SC]] : vector<2x8x4xf32>
 
 
-// Negative multi-group case: the target tile is contiguous within the result
+// Negative multi-group case: the target tile [2, 4] is contiguous within the result
 // group [24], but its elements cannot be extracted contiguously from the
 // source group [8, 3], so the cast is left un-unrolled.
 func.func @negative_shape_cast_multi_group_source_not_determinable(%v: vector<2x8x3xf32>) -> vector<2x24xf32> {


From llvmlistbot at llvm.org  Thu Jul  9 22:04:20 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 09 Jul 2026 22:04:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a507d54.f06f4994.1fc857.66a6@mx.google.com>


Jianhui-Li wrote:

> Your reply addresses my concerns. It would be good to add comments in code to re-enforce these assumptions/guidelines, i.e. this unrolling should not be used by targets for which `tensor.extract_strided_slice` is a concern and `vector.shape_cast` should be treated as NOP.
> 
> Otherwise LGTM

Added comments and two negative tests. Please take a look. Thanks! @banach-space

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Thu Jul  9 22:04:32 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 09 Jul 2026 22:04:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a507d60.481afadf.196d57.0b5e@mx.google.com>


================
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
----------------
Jianhui-Li wrote:

added

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Thu Jul  9 22:04:33 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 09 Jul 2026 22:04:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a507d61.bd93e731.358318.3e47@mx.google.com>


================
@@ -1335,20 +1390,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
+
+      ArrayRef<int64_t> srcSub =
+          sourceShape.slice(g.srcBegin, g.srcEnd - g.srcBegin);
+      int64_t groupTargetElements = ShapedType::getNumElements(tgtSub);
+      std::optional<SmallVector<int64_t>> groupExtract =
+          calculateSourceExtractShape(srcSub, groupTargetElements);
+      if (!groupExtract)
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "cannot extract the target number of elements "
+                         "contiguously from a source reassociation group");
----------------
Jianhui-Li wrote:

added

https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Thu Jul  9 22:14:47 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Thu, 09 Jul 2026 22:14:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a507fc7.d76d81af.39f9be.ebd6@mx.google.com>


https://github.com/aniragil edited https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Thu Jul  9 22:14:47 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Thu, 09 Jul 2026 22:14:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a507fc7.af6ba3ed.36d9db.3c3d@mx.google.com>


https://github.com/aniragil approved this pull request.

LGTM with some nits, thanks @ioghiban !

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Thu Jul  9 22:14:47 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Thu, 09 Jul 2026 22:14:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a507fc7.70920dee.290dca.b60c@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
----------------
aniragil wrote:

```suggestion
      Value srcLValue = emitc::SubscriptOp::create(
```

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Thu Jul  9 22:14:47 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Thu, 09 Jul 2026 22:14:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a507fc7.bb839515.27d21e.1f5e@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
+                                         srcLValue.getResult());
+
+      auto targetLValue = emitc::SubscriptOp::create(
----------------
aniragil wrote:

```suggestion
      Value targetLValue = emitc::SubscriptOp::create(
```

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Thu Jul  9 22:14:48 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Thu, 09 Jul 2026 22:14:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a507fc8.4987c784.63cbc.2f51@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
----------------
aniragil wrote:

```suggestion
      Value value = emitc::LoadOp::create(rewriter, loc, elementType,
```

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Thu Jul  9 23:01:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 09 Jul 2026 23:01:04 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a508aa0.8e75e0cb.5e108.063c@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/202315

>From 216cbd2e710ce3c725bb51c5606bea4ba84a33a3 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Sat, 6 Jun 2026 14:38:54 -0500
Subject: [PATCH 1/9] Support for "atomic compare capture".

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  88 ++++---
 .../Interfaces/AtomicInterfaces.td            |   9 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  10 +
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |   6 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 216 ++++++++++++++----
 5 files changed, 250 insertions(+), 79 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 9d711b92bb520..5e02d3fb10dbb 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -558,16 +558,45 @@ void Fortran::lower::omp::lowerAtomic(
   int action1 = analysis.op1.what & analysis.Action;
   memOrder = makeValidForAction(memOrder, action0, action1, version);
 
+  // --- Shared capture scaffolding ---
+  mlir::Operation *captureOp = nullptr;
+  fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
+  fir::FirOpBuilder::InsertPoint atomicAt, postAt;
+
+  if (construct.IsCapture()) {
+    assert(action0 != analysis.None && action1 != analysis.None &&
+           "Expexcing two actions");
+    (void)action0;
+    (void)action1;
+    captureOp = mlir::omp::AtomicCaptureOp::create(
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+    // Set the non-atomic insertion point to before the atomic.capture.
+    preAt = getInsertionPointBefore(captureOp);
+
+    mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
+    builder.setInsertionPointToEnd(block);
+    // Set the atomic insertion point to before the terminator inside
+    // atomic.capture.
+    mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
+    atomicAt = getInsertionPointBefore(term);
+    postAt = getInsertionPointAfter(captureOp);
+    hint = nullptr;
+    memOrder = std::nullopt;
+  }
+
   if (auto *cond = get(analysis.cond)) {
     // atomic compare: if (x == e) x = d
     // e : expecteVal
     // d : desiredVal
 
-    // Check for compound clauses (fail, capture, weak) that are not yet
+    // Restore insertion point so pre-processing code (e.g. computing
+    // expectedVal) is emitted before the capture op, not after the terminator.
+    builder.restoreInsertionPoint(preAt);
+
+    // Check for compound clauses (fail, weak) that are not yet
     // supported with atomic compare.
     if (llvm::any_of(clauses, [](const omp::Clause &clause) {
           return clause.id == llvm::omp::Clause::OMPC_fail ||
-                 clause.id == llvm::omp::Clause::OMPC_capture ||
                  clause.id == llvm::omp::Clause::OMPC_weak;
         })) {
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
@@ -616,6 +645,17 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // If this is a compare+capture, generate the read op first.
+    if (construct.IsCapture()) {
+      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
+             "Expected a read operation for compare capture");
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
+
     mlir::UnitAttr weakAttr = nullptr;
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
@@ -679,34 +719,9 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
-
     // END omp atomic compare
   } else {
-    mlir::Operation *captureOp = nullptr;
-    fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
-    fir::FirOpBuilder::InsertPoint atomicAt, postAt;
-
-    if (construct.IsCapture()) {
-      // Capturing operation.
-      assert(action0 != analysis.None && action1 != analysis.None &&
-             "Expexcing two actions");
-      (void)action0;
-      (void)action1;
-      captureOp = mlir::omp::AtomicCaptureOp::create(
-          builder, loc, hint, makeMemOrderAttr(converter, memOrder));
-      // Set the non-atomic insertion point to before the atomic.capture.
-      preAt = getInsertionPointBefore(captureOp);
-
-      mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
-      builder.setInsertionPointToEnd(block);
-      // Set the atomic insertion point to before the terminator inside
-      // atomic.capture.
-      mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
-      atomicAt = getInsertionPointBefore(term);
-      postAt = getInsertionPointAfter(captureOp);
-      hint = nullptr;
-      memOrder = std::nullopt;
-    } else {
+    if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
              "Expexcing single action");
@@ -729,16 +744,13 @@ void Fortran::lower::omp::lowerAtomic(
           *get(analysis.op1.assign), hint, memOrder, preAt, atomicAt, postAt);
     }
 
-    if (construct.IsCapture()) {
-      // If this is a capture operation, the first/second ops will be inside
-      // of it. Set the insertion point to past the capture op itself.
-      builder.restoreInsertionPoint(postAt);
-    } else {
-      if (secondOp) {
-        builder.setInsertionPointAfter(secondOp);
-      } else {
-        builder.setInsertionPointAfter(firstOp);
-      }
+    if (!construct.IsCapture()) {
+      builder.setInsertionPointAfter(secondOp ? secondOp : firstOp);
     }
   }
+
+  // Shared capture cleanup.
+  if (construct.IsCapture()) {
+    builder.restoreInsertionPoint(postAt);
+  }
 }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index abb21705b3c1c..8c9015f05bb72 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -289,10 +289,12 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
+        auto secondCompareStmt = dyn_cast<AtomicCompareOpInterface>(secondOp);
 
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
-              (firstReadStmt && secondWriteStmt)))
+              (firstReadStmt && secondWriteStmt) ||
+              (firstReadStmt && secondCompareStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -310,6 +312,11 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
           return firstReadStmt.emitError()
                 << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstReadStmt && secondCompareStmt &&
+            firstReadStmt.getX() != secondCompareStmt.getX())
+          return firstReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "second operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..1241abc10298f 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1928,6 +1928,12 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
         omp.atomic.write ...
         omp.terminator
       }
+
+      omp.atomic.capture {
+        omp.atomic.read ...
+        omp.atomic.compare ...
+        omp.terminator
+      }
     ```
   }] # clausesDescription;
 
@@ -1946,6 +1952,10 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     /// Returns the `atomic.update` operation inside the region, if any.
     /// Otherwise, it returns nullptr.
     AtomicUpdateOp getAtomicUpdateOp();
+
+    /// Returns the `atomic.compare` operation inside the region, if any.
+    /// Otherwise, it returns nullptr.
+    AtomicCompareOp getAtomicCompareOp();
   }] # clausesExtraClassDeclaration;
 
   let hasRegionVerifier = 1;
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..0eafd0a267b97 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4615,6 +4615,12 @@ AtomicUpdateOp AtomicCaptureOp::getAtomicUpdateOp() {
   return dyn_cast<AtomicUpdateOp>(getSecondOp());
 }
 
+AtomicCompareOp AtomicCaptureOp::getAtomicCompareOp() {
+  if (auto op = dyn_cast<AtomicCompareOp>(getFirstOp()))
+    return op;
+  return dyn_cast<AtomicCompareOp>(getSecondOp());
+}
+
 LogicalResult AtomicCaptureOp::verify() {
   return verifySynchronizationHint(*this, getHint());
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 07fa92bdabe50..4928b288adb65 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4795,6 +4795,43 @@ convertOmpAtomicUpdate(omp::AtomicUpdateOp &opInst,
   return success();
 }
 
+/// Helper to extract the OMPAtomicCompareOp from an integer comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::ICmpPredicate::eq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::ICmpPredicate::slt:
+  case LLVM::ICmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::ICmpPredicate::sgt:
+  case LLVM::ICmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
+/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::FCmpPredicate::oeq:
+  case LLVM::FCmpPredicate::ueq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::FCmpPredicate::olt:
+  case LLVM::FCmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::FCmpPredicate::ogt:
+  case LLVM::FCmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -4803,13 +4840,149 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   if (failed(checkImplementationStatus(*atomicCaptureOp)))
     return failure();
 
+  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
+  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
+  omp::AtomicCompareOp atomicCompareOp = atomicCaptureOp.getAtomicCompareOp();
+
+  // If the capture contains an atomic.compare, delegate to
+  // createAtomicCompare with the capture variable (V) set.
+  if (atomicCompareOp) {
+    omp::AtomicReadOp atomicReadOp = atomicCaptureOp.getAtomicReadOp();
+    assert(atomicReadOp && "expected atomic.read in capture+compare");
+
+    Region &region = atomicCompareOp.getRegion();
+    Block &block = region.front();
+
+    llvm::Type *llvmXElementType =
+        moduleTranslation.convertType(block.getArgument(0).getType());
+    llvm::Value *llvmX = moduleTranslation.lookupValue(atomicCompareOp.getX());
+    llvm::Value *llvmV = moduleTranslation.lookupValue(atomicReadOp.getV());
+
+    bool isSigned = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicX = {
+        llvmX, llvmXElementType, isSigned, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicV = {
+        llvmV, llvmXElementType, /*isSigned=*/false, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicR = {nullptr, nullptr, false,
+                                                        false};
+
+    llvm::AtomicOrdering atomicOrdering =
+        convertAtomicOrdering(atomicCaptureOp.getMemoryOrder());
+
+    // Pre-translate non-pattern operations inside the compare region.
+    auto isAtomicComparePatternOp = [](Operation &op) {
+      return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
+                       LLVM::OrOp>(op);
+    };
+    for (Operation &op : block.without_terminator()) {
+      if (isAtomicComparePatternOp(op))
+        continue;
+      bool allOperandsMapped =
+          llvm::all_of(op.getOperands(), [&](mlir::Value v) {
+            return moduleTranslation.lookupValue(v) != nullptr;
+          });
+      if (!allOperandsMapped)
+        continue;
+      if (failed(moduleTranslation.convertOperation(op, builder)))
+        return atomicCompareOp.emitError(
+            "failed to translate operation inside atomic compare region");
+    }
+
+    auto materializeValue = [&](mlir::Value val) -> llvm::Value * {
+      if (llvm::Value *existing = moduleTranslation.lookupValue(val))
+        return existing;
+      if (auto loadOp = val.getDefiningOp<LLVM::LoadOp>()) {
+        if (loadOp->getParentRegion() == &region) {
+          llvm::Value *loadAddr =
+              moduleTranslation.lookupValue(loadOp.getAddr());
+          if (!loadAddr)
+            return nullptr;
+          llvm::Type *loadType =
+              moduleTranslation.convertType(loadOp.getResult().getType());
+          return builder.CreateLoad(loadType, loadAddr);
+        }
+      }
+      return nullptr;
+    };
+
+    // Extract comparison predicate, eVal, and dVal from the region.
+    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    llvm::Value *eVal = nullptr;
+    llvm::Value *dVal = nullptr;
+    bool isXBinopExpr = false;
+
+    for (Operation &op : block.getOperations()) {
+      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
+        auto maybeOp =
+            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+        isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                    pred == LLVM::ICmpPredicate::sgt ||
+                    pred == LLVM::ICmpPredicate::sle ||
+                    pred == LLVM::ICmpPredicate::sge);
+
+        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
+        auto maybeOp =
+            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        if (!dVal)
+          dVal = materializeValue(selectOp.getTrueValue());
+      }
+    }
+
+    if (!eVal)
+      return atomicCompareOp.emitError(
+          "failed to extract expected value (e) from atomic compare region");
+    if (!dVal) {
+      auto yieldOp = cast<omp::YieldOp>(block.getTerminator());
+      if (yieldOp.getResults().empty())
+        return atomicCompareOp.emitError(
+            "failed to extract desired value (d) from atomic compare region");
+      dVal = materializeValue(yieldOp.getResults()[0]);
+    }
+
+    llvmAtomicX.IsSigned = isSigned;
+
+    llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    bool isPostfixUpdate = true;
+
+    bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, /*IsFailOnly=*/false);
+    ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
+
+    if (failed(handleError(afterIP, *atomicCaptureOp)))
+      return failure();
+
+    builder.restoreIP(*afterIP);
+    return success();
+  }
+
   mlir::Value mlirExpr;
   bool isXBinopExpr = false, isPostfixUpdate = false;
   llvm::AtomicRMWInst::BinOp binop = llvm::AtomicRMWInst::BinOp::BAD_BINOP;
 
-  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
-  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
-
   assert((atomicUpdateOp || atomicWriteOp) &&
          "internal op must be an atomic.update or atomic.write op");
 
@@ -4896,43 +5069,6 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   return success();
 }
 
-/// Helper to extract the OMPAtomicCompareOp from an integer comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::ICmpPredicate::eq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::ICmpPredicate::slt:
-  case LLVM::ICmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::ICmpPredicate::sgt:
-  case LLVM::ICmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
-/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::FCmpPredicate::oeq:
-  case LLVM::FCmpPredicate::ueq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::FCmpPredicate::olt:
-  case LLVM::FCmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::FCmpPredicate::ogt:
-  case LLVM::FCmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
 /// Converts an omp.atomic.compare operation to LLVM IR.
 ///
 ///      if (x == e) x = d

>From 7d3293944598e8090a64cdec2ac3c791a64b73f1 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Mon, 8 Jun 2026 05:12:22 -0500
Subject: [PATCH 2/9] Adding tests for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     | 32 +++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 56 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 29 ++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  9 +--
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 41 ++++++++++++++
 6 files changed, 211 insertions(+), 4 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 249fb0dd8fa64..650f64b80af12 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,3 +260,35 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
+! Integer equality compare+capture: cmpxchg + store old value
+!CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
+! Compare+capture with clause order reversed: capture compare
+!CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_capture_compare_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic capture compare
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ac70edbed4e60..752a221aa538d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -161,3 +161,59 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
   !$omp atomic compare weak
   if (x .eq. e) x = d
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_eq(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_gt(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           } {{.*}}weak{{.*}}
+! CHECK:         }
+subroutine atomic_compare_capture_int_gt(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+    v = x
+    if (x > e) x = d
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 4f3cf0eac0338..dc0cc1a62f6c2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -192,6 +192,26 @@ program main
       i = j
    end if
 
+!COMPARE CAPTURE
+!$omp atomic compare capture
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare weak
+   k = i
+   if (i < j) then
+      i = k
+   end if
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -296,6 +316,15 @@ end program main
 !CHECK: !$OMP ATOMIC WEAK COMPARE
 !CHECK: !$OMP ATOMIC COMPARE SEQ_CST WEAK
 
+!COMPARE CAPTURE
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 17f99a8c86d52..c5a07a7dc6cb2 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4963,13 +4963,14 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
     bool isPostfixUpdate = true;
+    bool isWeak = atomicCompareOp.getWeak();
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, /*IsFailOnly=*/false);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            /*IsFailOnly=*/false, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index acf6879906e74..289ec60551cb7 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2047,6 +2047,54 @@ func.func @omp_atomic_compare(%x: memref<i32>, %e: i32, %d: i32) {
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_weak
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: } {weak}
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi sgt, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 1890eaa6a3f0b..6ce57b304bfd9 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2795,6 +2795,47 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From f84987dea68a1dedc6b61bf0c9f5a2e24a0a1b40 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 9 Jun 2026 04:42:48 -0500
Subject: [PATCH 3/9] Minor changes incorporating feedback.

---
 flang/lib/Lower/OpenMP/Atomic.cpp                             | 4 ++--
 .../Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index c0f97aada637d..aa497b838e5b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -567,7 +567,7 @@ void Fortran::lower::omp::lowerAtomic(
 
   if (construct.IsCapture()) {
     assert(action0 != analysis.None && action1 != analysis.None &&
-           "Expexcing two actions");
+           "Expecting two actions");
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
@@ -730,7 +730,7 @@ void Fortran::lower::omp::lowerAtomic(
     if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
-             "Expexcing single action");
+             "Expecting single action");
       assert(!(analysis.op0.what & analysis.Condition));
       postAt = atomicAt = preAt;
     }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 8c9015f05bb72..dc5b04126b757 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -315,7 +315,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be updated in "
+                << "captured variable in atomic.read must be compared/updated in "
                     "second operation";
 
         return mlir::success();

>From ca24a147920d24c188127dbbe37ca62bcaeab792 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 14:18:17 -0500
Subject: [PATCH 4/9] Handling test cases mentionend in OpenMP spec.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 61 ++++++++++++++++---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 ++-
 .../Interfaces/AtomicInterfaces.td            | 11 +++-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  2 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 12 ++--
 5 files changed, 74 insertions(+), 19 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index aa497b838e5b6..209871abd52b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -571,7 +571,8 @@ void Fortran::lower::omp::lowerAtomic(
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
-        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder),
+        /*fail_only=*/nullptr);
     // Set the non-atomic insertion point to before the atomic.capture.
     preAt = getInsertionPointBefore(captureOp);
 
@@ -646,15 +647,45 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
-    // If this is a compare+capture, generate the read op first.
+    // If this is a compare+capture, determine the ordering of ops.
+    // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
+    // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
+    // Pattern 3 (fail-only): if (x == e) x = d; else v = x → read on failure
+    bool isPostfixCapture = false;
+    bool isFailOnly = false;
+    const evaluate::Assignment *readAssign = nullptr;
     if (construct.IsCapture()) {
-      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
-             "Expected a read operation for compare capture");
-      mlir::Operation *readOp = genAtomicRead(
-          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
-          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
-      assert(readOp && "Should have created an atomic read operation");
-      builder.setInsertionPointAfter(readOp);
+      // Determine which op is the read and check for fail-only (IfFalse).
+      int readWhat = 0;
+      if (analysis.op0.what & analysis.Read) {
+        readAssign = get(analysis.op0.assign);
+        readWhat = analysis.op0.what;
+      } else if (analysis.op1.what & analysis.Read) {
+        readAssign = get(analysis.op1.assign);
+        readWhat = analysis.op1.what;
+        isPostfixCapture = true;
+      }
+      assert(readAssign && "Expected a read assignment for compare capture");
+
+      // Check if the read is conditioned on comparison failure (else branch).
+      if (readWhat & analysis.IfFalse)
+        isFailOnly = true;
+
+      if (!isPostfixCapture && !isFailOnly) {
+        // Pattern 1 (prefix): read is first, generate it before compare.
+        mlir::Operation *readOp =
+            genAtomicRead(converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+                          *readAssign, hint, memOrder, preAt, atomicAt, postAt);
+        assert(readOp && "Should have created an atomic read operation");
+        builder.setInsertionPointAfter(readOp);
+      } else {
+        // Pattern 2 (postfix) or 3 (fail-only): compare first, read after.
+        builder.restoreInsertionPoint(atomicAt);
+      }
+
+      // Set the fail_only attribute on the capture op.
+      if (isFailOnly && captureOp)
+        mlir::cast<mlir::omp::AtomicCaptureOp>(captureOp).setFailOnly(true);
     }
 
     mlir::UnitAttr weakAttr = nullptr;
@@ -725,6 +756,18 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
+
+    // Pattern 2 (postfix) or 3 (fail-only): compare first, read second.
+    // Generate read after compare for postfix or fail-only patterns.
+    if (construct.IsCapture() && (isPostfixCapture || isFailOnly)) {
+      fir::FirOpBuilder::InsertPoint afterCompareAt =
+          builder.saveInsertionPoint();
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom, *readAssign, hint,
+          memOrder, preAt, afterCompareAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
     // END omp atomic compare
   } else {
     if (!construct.IsCapture()) {
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 92eb7de0d882f..b2ccc05ad6770 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11321,7 +11321,9 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11377,7 +11379,8 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index dc5b04126b757..5b1527bcb84ec 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -286,6 +286,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto &secondOp = *ops.getNextNode(firstOp);
         auto firstReadStmt = dyn_cast<AtomicReadOpInterface>(firstOp);
         auto firstUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(firstOp);
+        auto firstCompareStmt = dyn_cast<AtomicCompareOpInterface>(firstOp);
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
@@ -294,7 +295,8 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
               (firstReadStmt && secondWriteStmt) ||
-              (firstReadStmt && secondCompareStmt)))
+              (firstReadStmt && secondCompareStmt) ||
+              (firstCompareStmt && secondReadStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -315,8 +317,13 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be compared/updated in "
+                << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstCompareStmt && secondReadStmt &&
+            secondReadStmt.getX() != firstCompareStmt.getX())
+          return secondReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "first operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index ebee887f2afd1..69c22641df6a2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1938,6 +1938,8 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     ```
   }] # clausesDescription;
 
+  let arguments = !con(clausesArgs, (ins UnitAttr:$fail_only));
+
   // Override region definition.
   let regions = (region SizedRegion<1>:$region);
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index dff1ae30e20aa..76616c7fba27a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,15 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    bool isPostfixUpdate = true;
-    bool isWeak = atomicCompareOp.getWeak();
+    bool isFailOnly = atomicCaptureOp.getFailOnly();
+    bool isPostfixUpdate = !isFailOnly;
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(
-            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
-            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
-            /*IsFailOnly=*/false, isWeak);
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, isFailOnly);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))

>From afaf2025463e9e8cf15974747fa9e2d8b8e5fa95 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 18:02:52 -0500
Subject: [PATCH 5/9] Adding tests into flang and mlir for the grammar
 mentionend in the spec.

---
 .../Integration/OpenMP/atomic-compare.f90     | 56 ++++++++++++++++--
 flang/test/Lower/OpenMP/atomic-compare.f90    | 59 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 17 ++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 +++++-
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 58 ++++++++++++++++--
 6 files changed, 245 insertions(+), 10 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 650f64b80af12..da86f3ff6ef16 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,14 +260,17 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
-! Integer equality compare+capture: cmpxchg + store old value
+! Integer equality compare+capture (prefix): v=x; if(x==e) x=d
+! v captures old value of x
 !CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -276,14 +279,16 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
-! Compare+capture with clause order reversed: capture compare
+! Compare+capture with clause order reversed: capture compare (still prefix read)
 !CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
@@ -292,3 +297,46 @@ subroutine atomic_capture_compare_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
+! Postfix compare+capture: if (x == e) x = d; v = x
+! v captures new value (d if swapped, old x if not)
+!CHECK-LABEL: define void @atomic_compare_capture_postfix_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Fail-only compare+capture: if (x == e) x = d; else v = x
+! v is only written when the comparison fails
+!CHECK-LABEL: define void @atomic_compare_capture_fail_only_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+!CHECK: {{.*}}:
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: br label %[[EXIT2:.*]]
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 752a221aa538d..c770605954539 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -217,3 +217,62 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
     if (x > e) x = d
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_fail_only(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index dc0cc1a62f6c2..521c1bb2811e2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -211,6 +211,19 @@ program main
       i = k
    end if
 !$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   end if
+   k = i
+!$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   else
+      k = i
+   end if
+!$omp end atomic
 
 !ATOMIC
 !$omp atomic
@@ -324,6 +337,10 @@ end program main
 !CHECK: !$OMP END ATOMIC
 !CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
 !CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
 
 !ATOMIC
 !CHECK: !$OMP ATOMIC
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 76616c7fba27a..b6fc6d6dc3038 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,28 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    // In OMPIRBuilder::createAtomicCompare:
+    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
+    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
+    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
+    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
+    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
+    bool isPostfixUpdate =
+        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
     bool isFailOnly = atomicCaptureOp.getFailOnly();
-    bool isPostfixUpdate = !isFailOnly;
 
+    // For fail-only, the OMPIRBuilder's conditional store logic is under
+    // the !IsPostfixUpdate path, so force it to false.
+    if (isFailOnly)
+      isPostfixUpdate = false;
+
+    bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
         ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly);
+                                        isPostfixUpdate, isFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 87f9a6f8e4119..6b13be481e0e0 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2113,6 +2113,54 @@ func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e:
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_postfix
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_postfix(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: } {fail_only}
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  } {fail_only}
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 6ce57b304bfd9..978450d598b9b 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,10 +2798,12 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2819,10 +2821,12 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2836,6 +2840,52 @@ llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_postfix
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Postfix compare+capture: v captures new value (d if swapped, old x if not)
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[D]], i32 %[[OLD]]
+  // CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Fail-only compare+capture: v is only written when comparison fails
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+  // CHECK: [[CONT]]:
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: br label %[[EXIT2:.*]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From e8462a267b6d010ebc1e3ed881c1f38a9faca6be Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 24 Jun 2026 14:44:43 -0500
Subject: [PATCH 6/9] Simplifying the IR generator for one of the cases. This
 also fixes the failure in clang for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     |  8 +---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 43 +++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 12 ++----
 4 files changed, 38 insertions(+), 32 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index da86f3ff6ef16..4ae1b9a1f1e52 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -268,9 +268,7 @@ subroutine atomic_compare_weak(x, e, d)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -286,9 +284,7 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 51f210e189e22..f8c1999fe1b89 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11400,9 +11400,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11458,8 +11456,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 85389caa9ac42..e1fb08d780753 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5499,25 +5499,23 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    // In OMPIRBuilder::createAtomicCompare:
-    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
-    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
-    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
-    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
-    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
-    bool isPostfixUpdate =
-        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isReadFirst = isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
-    // For fail-only, the OMPIRBuilder's conditional store logic is under
-    // the !IsPostfixUpdate path, so force it to false.
-    if (isFailOnly)
-      isPostfixUpdate = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
+    // Postfix: bypass V in OMPIRBuilder, handled manually below.
+    if (isPostfixCapture && !isFailOnly)
+      llvmAtomicVForCall = {nullptr, nullptr, false, false};
+
+    // Prefix: IsPostfixUpdate=true → direct store of old value.
+    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
+    bool isPostfixUpdate = !isFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
                                         isPostfixUpdate, isFailOnly, isWeak);
@@ -5527,6 +5525,25 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
       return failure();
 
     builder.restoreIP(*afterIP);
+
+    // Postfix: v = select(success, D, old) — captures new value of x.
+    if (isPostfixCapture && !isFailOnly) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::Instruction *cmpxchgInst = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
+          cmpxchgInst = &inst;
+          break;
+        }
+      }
+      assert(cmpxchgInst && "expected cmpxchg instruction");
+      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
+      llvm::Value *successVal =
+          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+      llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
+      builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+    }
+
     return success();
   }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 978450d598b9b..377bd88894854 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,12 +2798,10 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2821,12 +2819,10 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality weak prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {

>From 101b9fa02882686a21d346ef12e541c717ca605b Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 14:00:46 -0500
Subject: [PATCH 7/9] Handling real and logical types.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 117 ++++++--
 .../Integration/OpenMP/atomic-compare.f90     |  58 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    |  83 +++++-
 flang/test/Parser/OpenMP/atomic-unparse.f90   |  12 +
 .../Interfaces/AtomicInterfaces.td            |  13 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 252 +++++++++++-------
 mlir/test/Dialect/OpenMP/ops.mlir             |  24 ++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++
 8 files changed, 470 insertions(+), 137 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 209871abd52b6..3811ea3d40bbd 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -375,6 +375,8 @@ genAtomicRead(lower::AbstractConverter &converter,
         fir::getBase(converter.genExprValue(assign.rhs, stmtCtx, &loc));
     converter.resetExprOverrides();
 
+    if (value.getType() != storeType)
+      value = builder.createConvert(loc, storeType, value);
     fir::StoreOp::create(builder, loc, value, storeAddr);
   }
   return op;
@@ -634,6 +636,45 @@ void Fortran::lower::omp::lowerAtomic(
           },
           rel->u);
     }
+
+    // Fortran uses .eqv./.neqv. for logical equality/inequality, which are
+    // LogicalOperation expressions rather than Relational expressions.
+    if (!expectedExprStorage) {
+      if (const auto *someLogical =
+              evaluate::UnwrapExpr<evaluate::Expr<evaluate::SomeLogical>>(
+                  *cond)) {
+        common::visit(
+            [&](const auto &kindLogical) {
+              using LogicalExpr = std::decay_t<decltype(kindLogical)>;
+              constexpr int K = LogicalExpr::Result::kind;
+              if (const auto *logOp =
+                      std::get_if<evaluate::LogicalOperation<K>>(
+                          &kindLogical.u)) {
+                if (logOp->logicalOperator == common::LogicalOperator::Eqv ||
+                    logOp->logicalOperator == common::LogicalOperator::Neqv) {
+                  relOpr =
+                      (logOp->logicalOperator == common::LogicalOperator::Eqv)
+                          ? common::RelationalOperator::EQ
+                          : common::RelationalOperator::NE;
+                  using Operand =
+                      typename evaluate::LogicalOperation<K>::Operand;
+                  auto leftExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->left()});
+                  auto rightExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->right()});
+                  if (evaluate::IsSameOrConvertOf(rightExpr, atom)) {
+                    expectedExprStorage = std::move(leftExpr);
+                    relOpr = reverseRelOp(relOpr);
+                  } else {
+                    expectedExprStorage = std::move(rightExpr);
+                  }
+                }
+              }
+            },
+            someLogical->u);
+      }
+    }
+
     if (!expectedExprStorage) {
       mlir::emitError(loc, "internal error: atomic compare condition is not a "
                            "recognized relational expression");
@@ -647,6 +688,20 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // For logical types, convert address and expected value to integer
+    // type here (above the atomic compare region) so that the region
+    // only contains arith.cmpi eq on integers. If done inside the region,
+    // fir.convert logical<4> -> i32 would lower to `icmp ne %val, 0`
+    // which violates the atomic compare verifier's expectations.
+    if (mlir::isa<fir::LogicalType>(elemTypeOfX)) {
+      unsigned kind = mlir::cast<fir::LogicalType>(elemTypeOfX).getFKind();
+      mlir::Type intTy = builder.getIntegerType(kind * 8);
+      mlir::Type intRefTy = builder.getRefType(intTy);
+      atomAddr = builder.createConvert(loc, intRefTy, atomAddr);
+      expectedVal = builder.createConvert(loc, intTy, expectedVal);
+      elemTypeOfX = intTy;
+    }
+
     // If this is a compare+capture, determine the ordering of ops.
     // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
     // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
@@ -694,6 +749,39 @@ void Fortran::lower::omp::lowerAtomic(
         })) {
       weakAttr = builder.getUnitAttr();
     }
+
+    // Extract write assignment (x = d) and generate desired value (d)
+    // before creating the compare region, so that d is defined outside
+    // the region and any intermediate conversions (e.g., logical-to-integer
+    // truthiness normalization) don't appear inside the atomic compare block.
+    [[maybe_unused]] int writeActionCond = 0;
+    const evaluate::Assignment *writeAssign = nullptr;
+    if (analysis.op0.what & analysis.Write) {
+      writeAssign = get(analysis.op0.assign);
+      writeActionCond = analysis.op0.what;
+    }
+    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
+      writeAssign = get(analysis.op1.assign);
+      writeActionCond = analysis.op1.what;
+    }
+    if (!writeAssign) {
+      mlir::emitError(loc,
+                      "internal error: atomic compare has no write assignment");
+      return;
+    }
+    assert((writeActionCond & analysis.IfTrue) &&
+           "atomic compare write should be conditioned on IfTrue");
+
+    // Generate desiredVal before the capture/compare region so any
+    // intermediate ops (loads, conversions) don't pollute the atomic blocks.
+    fir::FirOpBuilder::InsertPoint savedIP = builder.saveInsertionPoint();
+    builder.restoreInsertionPoint(preAt);
+    mlir::Value desiredVal =
+        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
+    if (desiredVal.getType() != elemTypeOfX)
+      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
+    builder.restoreInsertionPoint(savedIP);
+
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
         makeMemOrderAttr(converter, memOrder));
@@ -721,35 +809,6 @@ void Fortran::lower::omp::lowerAtomic(
       return;
     }
 
-    // Check for presence of Assignment (x = d) and wether it is being invoked
-    // only for IfTrue condition.
-
-    // writeActionCond is a bitmask combining the following flags:
-    //  1) the action type (Read/Write/Update)
-    //  2) condition (IfTrue/IfFalse)
-    [[maybe_unused]] int writeActionCond = 0;
-    const evaluate::Assignment *writeAssign = nullptr;
-    if (analysis.op0.what & analysis.Write) {
-      writeAssign = get(analysis.op0.assign);
-      writeActionCond = analysis.op0.what;
-    }
-    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
-      writeAssign = get(analysis.op1.assign);
-      writeActionCond = analysis.op1.what;
-    }
-    if (!writeAssign) {
-      mlir::emitError(loc,
-                      "internal error: atomic compare has no write assignment");
-      return;
-    }
-    assert((writeActionCond & analysis.IfTrue) &&
-           "atomic compare write should be conditioned on IfTrue");
-
-    // Generate new/desired value of x e.g. x = d
-    mlir::Value desiredVal =
-        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
-    if (desiredVal.getType() != elemTypeOfX)
-      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
     mlir::Value newVal = mlir::arith::SelectOp::create(builder, loc, cmpResult,
                                                        desiredVal, blockArg);
 
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 4ae1b9a1f1e52..f30349c42e33e 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -336,3 +336,61 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
   !$omp end atomic
 end
 
+! Real equality compare+capture (postfix): if(x==e) x=d; v=x
+! v captures new value of x (d if success, old if fail)
+!CHECK-LABEL: define void @atomic_compare_capture_real_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load float, ptr %[[E]]{{.*}}
+!CHECK: %[[DVAL:.*]] = load float, ptr %[[D]]{{.*}}
+!CHECK: %[[EBITS:.*]] = bitcast float %[[EVAL]] to i32
+!CHECK: %[[DBITS:.*]] = bitcast float %[[DVAL]] to i32
+!CHECK: %[[XLOAD:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+!CHECK: %[[XFP:.*]] = bitcast i32 %[[XLOAD]] to float
+! Part 1: NaN check - if either x or e is NaN, comparison fails
+!CHECK: %[[E_ISNAN:.*]] = fcmp uno float %[[EVAL]], %[[EVAL]]
+!CHECK: %[[X_ISNAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+!CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_ISNAN]], %[[X_ISNAN]]
+!CHECK: br i1 %[[EITHER_NAN]], label %[[NAN_BB:.*]], label %[[NOTNAN_BB:.*]]
+!CHECK: [[NAN_BB]]:
+!CHECK: br label %[[EXIT_BB:.*]]
+! Part 2: Both-zero check - handles +0.0 vs -0.0 (same value, different bits)
+!CHECK: [[NOTNAN_BB]]:
+!CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+!CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[EVAL]], 0.000000e+00
+!CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+!CHECK: br i1 %[[BOTHZERO]], label %[[ZERO_BB:.*]], label %[[NORMAL_BB:.*]]
+!CHECK: [[ZERO_BB]]:
+!CHECK: %[[ZERORES:.*]] = cmpxchg ptr %[[X]], i32 %[[XLOAD]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Part 3: Normal compare - standard cmpxchg with bitcasted expected value
+!CHECK: [[NORMAL_BB]]:
+!CHECK: %[[NORMRES:.*]] = cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Exit: select v = (success ? d : old_x)
+!CHECK: [[EXIT_BB]]:
+!CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+!CHECK: %[[OK:.*]] = phi i1 {{.*}}
+!CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+!CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[DVAL]], float %[[OLDFP]]
+!CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Logical .eqv. compare+capture (postfix): if(x.eqv.e) x=d; v=x
+! Logicals are compared as integers after truthiness normalization
+!CHECK-LABEL: define void @atomic_compare_capture_logical_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: cmpxchg ptr %[[X]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index c770605954539..1219d05296e4d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -10,10 +10,10 @@
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -31,10 +31,10 @@ subroutine atomic_compare_int_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -52,10 +52,10 @@ subroutine atomic_compare_float_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<complex<f32>>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<complex<f32>> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: complex<f32>):
 ! CHECK:           %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %[[EVAL]] {fastmath = #arith.fastmath<contract>} : complex<f32>
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : complex<f32>
 ! CHECK:           omp.yield(%[[SEL]] : complex<f32>)
 ! CHECK:         }
@@ -71,10 +71,10 @@ subroutine atomic_compare_complex_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi slt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -90,10 +90,10 @@ subroutine atomic_compare_int_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -109,10 +109,10 @@ subroutine atomic_compare_int_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -128,10 +128,10 @@ subroutine atomic_compare_float_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -149,10 +149,10 @@ subroutine atomic_compare_float_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         } {{.*}}weak{{.*}}
@@ -172,12 +172,12 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -200,12 +200,12 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           } {{.*}}weak{{.*}}
@@ -228,11 +228,11 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -256,11 +256,11 @@ subroutine atomic_compare_capture_postfix(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -276,3 +276,62 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
     end if
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<f32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<f32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<f32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<f32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:         %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         %[[DLOAD:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[DCONV:.*]] = fir.convert %[[DLOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DCONV]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 521c1bb2811e2..0d30cd48035ea 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -4,6 +4,7 @@ program main
    implicit none
    integer :: i, j = 10
    integer :: k
+   logical :: l1, l2, l3, l4
 !READ
 !$omp atomic read
    i = j
@@ -225,6 +226,12 @@ program main
    end if
 !$omp end atomic
 
+!COMPARE CAPTURE LOGICAL
+!$omp atomic compare capture
+   if (l1 .eqv. l2) l1 = l3
+   l4 = l1
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -342,6 +349,11 @@ end program main
 !CHECK: !$OMP ATOMIC COMPARE CAPTURE
 !CHECK: !$OMP END ATOMIC
 
+!COMPARE CAPTURE LOGICAL
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 5b1527bcb84ec..a61c43e2fd9cd 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -488,6 +488,19 @@ def AtomicCompareOpInterface : OpInterface<"AtomicCompareOpInterface"> {
               }
             }
             break;
+          } else if (llvm::is_contained(
+                         {"arith.maxsi", "arith.minsi", "arith.maxui",
+                          "arith.minui", "arith.maximumf", "arith.minimumf",
+                          "llvm.intr.smax", "llvm.intr.smin",
+                          "llvm.intr.umax", "llvm.intr.umin",
+                          "llvm.intr.maxnum", "llvm.intr.minnum"},
+                         opName)) {
+            // The canonicalizer may fold select(cmpi slt, e, x) into
+            // maxsi/minsi (or the float equivalents). After fir-to-llvm-ir
+            // these become llvm.intr.smax/smin etc. All are valid
+            // representations of atomic compare min/max operations.
+            foundComparison = true;
+            break;
           }
         }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e1fb08d780753..8e49571cd2461 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 #include "mlir/Analysis/TopologicalSortUtils.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMTypes.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
@@ -5369,6 +5370,102 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Holds the extracted comparison pattern information from an atomic compare
+/// region.
+struct AtomicComparePatternInfo {
+  llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+  llvm::Value *eVal = nullptr;
+  llvm::Value *dVal = nullptr;
+  bool isXBinopExpr = false;
+  bool isSigned = false;
+};
+
+/// Extract comparison predicate, expected value (e), desired value (d), and
+/// related flags from an atomic compare region block by scanning for
+/// icmp/fcmp/select/min/max operations.
+static LogicalResult extractAtomicComparePattern(
+    Block &block,
+    llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
+    omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  for (Operation &op : block.getOperations()) {
+    // Pre-filter: skip icmps that don't involve the block argument
+    // (e.g., truthiness extractions from logical-to-integer conversion).
+    if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op);
+        icmpOp && icmpOp.getOperand(0) != block.getArgument(0) &&
+        icmpOp.getOperand(1) != block.getArgument(0))
+      continue;
+
+    LogicalResult result =
+        llvm::TypeSwitch<Operation *, LogicalResult>(&op)
+            .Case<LLVM::ICmpOp>([&](LLVM::ICmpOp icmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+              info.isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                               pred == LLVM::ICmpPredicate::sgt ||
+                               pred == LLVM::ICmpPredicate::sle ||
+                               pred == LLVM::ICmpPredicate::sge);
+              info.isXBinopExpr =
+                  (icmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? icmpOp.getOperand(1)
+                                                       : icmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::FCmpOp>([&](LLVM::FCmpOp fcmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              info.isXBinopExpr =
+                  (fcmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? fcmpOp.getOperand(1)
+                                                       : fcmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::SelectOp>([&](LLVM::SelectOp selectOp) {
+              if (!info.dVal)
+                info.dVal = materializeValue(selectOp.getTrueValue());
+              return success();
+            })
+            .Case<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                  mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                  mlir::arith::MaximumFOp, mlir::arith::MinimumFOp,
+                  LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp, LLVM::UMinOp,
+                  LLVM::MaxNumOp, LLVM::MinNumOp>([&](Operation *) {
+              // Canonicalized min/max ops (arith or LLVM intrinsic form).
+              // max(x,e) came from slt/ult/olt -> OMPAtomicCompareOp::MIN
+              // min(x,e) came from sgt/ugt/ogt -> OMPAtomicCompareOp::MAX
+              // (OMPIRBuilder inverts: MIN->atomicrmw max, MAX->atomicrmw min)
+              bool isMax = isa<mlir::arith::MaxSIOp, mlir::arith::MaxUIOp,
+                               mlir::arith::MaximumFOp, LLVM::SMaxOp,
+                               LLVM::UMaxOp, LLVM::MaxNumOp>(op);
+              info.compareOp = isMax ? llvm::omp::OMPAtomicCompareOp::MIN
+                                     : llvm::omp::OMPAtomicCompareOp::MAX;
+              info.isSigned = isa<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                                  LLVM::SMaxOp, LLVM::SMinOp>(op);
+              info.isXBinopExpr = (op.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand =
+                  info.isXBinopExpr ? op.getOperand(1) : op.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              info.dVal = info.eVal;
+              return success();
+            })
+            .Default([](Operation *) { return success(); });
+
+    if (failed(result))
+      return result;
+  }
+  return success();
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -5409,7 +5506,11 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Pre-translate non-pattern operations inside the compare region.
     auto isAtomicComparePatternOp = [](Operation &op) {
       return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                       LLVM::OrOp>(op);
+                       LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                       LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                       mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                       mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                       mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
     };
     for (Operation &op : block.without_terminator()) {
       if (isAtomicComparePatternOp(op))
@@ -5443,47 +5544,16 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     };
 
     // Extract comparison predicate, eVal, and dVal from the region.
-    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    llvm::Value *eVal = nullptr;
-    llvm::Value *dVal = nullptr;
-    bool isXBinopExpr = false;
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
 
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
+    llvm::omp::OMPAtomicCompareOp compareOp = patternInfo.compareOp;
+    llvm::Value *eVal = patternInfo.eVal;
+    llvm::Value *dVal = patternInfo.dVal;
+    bool isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
 
     if (!eVal)
       return atomicCompareOp.emitError(
@@ -5529,17 +5599,42 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Postfix: v = select(success, D, old) — captures new value of x.
     if (isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
-      llvm::Instruction *cmpxchgInst = nullptr;
+      llvm::Value *oldVal = nullptr;
+      llvm::Value *successVal = nullptr;
+
+      // Integer path (and non-HandleFPNegZero FP path): a single cmpxchg
+      // lives in the current block.
       for (auto &inst : llvm::reverse(*curBB)) {
         if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
-          cmpxchgInst = &inst;
+          oldVal = builder.CreateExtractValue(&inst, /*Idxs=*/0);
+          successVal = builder.CreateExtractValue(&inst, /*Idxs=*/1);
           break;
         }
       }
-      assert(cmpxchgInst && "expected cmpxchg instruction");
-      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
-      llvm::Value *successVal =
-          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+
+      // FP HandleFPNegZero path: the OMPIRBuilder emits a multi-block
+      // structure (NaN / ±0.0 handling) with cmpxchg in predecessor
+      // blocks. Results are merged via PHI nodes in the current (exit)
+      // block: an i1 PHI for success and a bitcast of an integer PHI
+      // for the old FP value.
+      if (!oldVal) {
+        for (auto &inst : *curBB) {
+          auto *phi = dyn_cast<llvm::PHINode>(&inst);
+          if (!phi)
+            break;
+          if (phi->getType()->isIntegerTy(1))
+            successVal = phi;
+        }
+        for (auto &inst : *curBB) {
+          if (auto *bc = dyn_cast<llvm::BitCastInst>(&inst)) {
+            oldVal = bc;
+            break;
+          }
+        }
+      }
+
+      assert(oldVal && "expected cmpxchg or PHI+bitcast for compare capture");
+      assert(successVal && "expected success flag for compare capture");
       llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
       builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
     }
@@ -5686,7 +5781,11 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
 
   auto isAtomicComparePatternOp = [](Operation &op) {
     return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                     LLVM::OrOp>(op);
+                     LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                     LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                     mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                     mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                     mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
   };
 
   // Pre-translate operations inside the region that compute e and d (e.g.,
@@ -5859,54 +5958,15 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
     }
     return success();
   } else {
-
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        // Identify which operand is the block argument (x) and which is e.
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
-  }
-
-  // For non-complex patterns, also extract dVal from SelectOp.
-  if (!dVal) {
-    for (Operation &op : block.getOperations()) {
-      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        dVal = materializeValue(selectOp.getTrueValue());
-        break;
-      }
-    }
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
+    compareOp = patternInfo.compareOp;
+    eVal = patternInfo.eVal;
+    dVal = patternInfo.dVal;
+    isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
   }
 
   if (!eVal)
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 6b13be481e0e0..cc40506b909b1 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2161,6 +2161,30 @@ func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_real
+// CHECK-SAME: (%[[V:.*]]: memref<f32>, %[[X:.*]]: memref<f32>, %[[E:.*]]: f32, %[[D:.*]]: f32)
+func.func @omp_atomic_compare_capture_real(%v: memref<f32>, %x: memref<f32>, %e: f32, %d: f32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<f32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: f32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[E]] : f32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : f32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : f32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<f32>, memref<f32>, f32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<f32> {
+    ^bb0(%xval: f32):
+      %cmp = arith.cmpf oeq, %xval, %e : f32
+      %sel = arith.select %cmp, %d, %xval : f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : memref<f32>, memref<f32>, f32
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 377bd88894854..b779d20f42d0c 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2882,6 +2882,54 @@ llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr,
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_real
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], float %[[E:.*]], float %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_real(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32, %d : f32) {
+  // Real compare+capture: uses HandleFPNegZero multi-block path
+  // CHECK: %[[EBITS:.*]] = bitcast float %[[E]] to i32
+  // CHECK: %[[DBITS:.*]] = bitcast float %[[D]] to i32
+  // CHECK: %[[XI:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+  // CHECK: %[[XFP:.*]] = bitcast i32 %[[XI]] to float
+  // Part 1: NaN check
+  // CHECK: %[[E_NAN:.*]] = fcmp uno float %[[E]], %[[E]]
+  // CHECK: %[[X_NAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+  // CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_NAN]], %[[X_NAN]]
+  // CHECK: br i1 %[[EITHER_NAN]], label %[[NAN:.*]], label %[[NOTNAN:.*]]
+  // CHECK: [[NAN]]:
+  // CHECK: br label %[[EXIT:.*]]
+  // Part 2: Both-zero check
+  // CHECK: [[NOTNAN]]:
+  // CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+  // CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[E]], 0.000000e+00
+  // CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+  // CHECK: br i1 %[[BOTHZERO]], label %[[ZERO:.*]], label %[[NORMAL:.*]]
+  // CHECK: [[ZERO]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[XI]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Part 3: Normal cmpxchg
+  // CHECK: [[NORMAL]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Exit: select v = (success ? d : old_x)
+  // CHECK: [[EXIT]]:
+  // CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+  // CHECK: %[[OK:.*]] = phi i1 {{.*}}
+  // CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+  // CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[D]], float %[[OLDFP]]
+  // CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "oeq" %xval, %e : f32
+      %sel = llvm.select %cmp, %d, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From d42a6412d158c581c76cadf7d4626b4235e60725 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 07:25:53 -0500
Subject: [PATCH 8/9] Handling a crash in logical "omp atomic compare-capture"
 inside "omp parallel". Adding some extra tests.

---
 flang/lib/Semantics/check-omp-atomic.cpp      |  19 ++-
 .../Integration/OpenMP/atomic-compare.f90     |  27 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 117 ++++++++++++++++++
 .../test/Semantics/OpenMP/atomic-compare.f90  |  12 ++
 4 files changed, 174 insertions(+), 1 deletion(-)

diff --git a/flang/lib/Semantics/check-omp-atomic.cpp b/flang/lib/Semantics/check-omp-atomic.cpp
index ec307be469982..88997df26e4d0 100644
--- a/flang/lib/Semantics/check-omp-atomic.cpp
+++ b/flang/lib/Semantics/check-omp-atomic.cpp
@@ -419,6 +419,22 @@ static bool IsMaybeAtomicWrite(const evaluate::Assignment &assign) {
   return HasStorageOverlap(assign.lhs, assign.rhs) == nullptr;
 }
 
+// Determine whether two designators refer to the same atomic variable.
+// The comparison tolerates OpenMP host-association: a shared variable
+// referenced inside a parallel region may be designated through a
+// host-associated symbol in one statement and through its ultimate symbol
+// in another, even though both denote the same storage. Comparing the
+// (ultimate) symbol vectors together with the source form distinguishes
+// different variables and different constant subscripts (e.g. a(1) vs a(2))
+// while treating host-associated and ultimate references as equal.
+static bool IsSameAtomicVariable(const SomeExpr &x, const SomeExpr &y) {
+  if (x == y) {
+    return true;
+  }
+  return evaluate::GetSymbolVector(x) == evaluate::GetSymbolVector(y) &&
+      x.AsFortran() == y.AsFortran();
+}
+
 static void SetExpr(parser::TypedExpr &expr, MaybeExpr value) {
   if (value) {
     expr.Reset(new evaluate::GenericExprWrapper(std::move(value)),
@@ -822,7 +838,8 @@ void OmpStructureChecker::CheckAtomicCaptureAssignment(
     CheckAtomicVariable(
         atom, rsrc, /*checkTypeOnPointer=*/!IsPointerAssignment(capture));
     // This part should have been checked prior to calling this function.
-    assert(*GetConvertInput(capture.rhs) == atom &&
+    assert(GetConvertInput(capture.rhs) &&
+        IsSameAtomicVariable(*GetConvertInput(capture.rhs), atom) &&
         "This cannot be a capture assignment");
     CheckStorageOverlap(atom, {cap}, source);
   }
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index f30349c42e33e..e98b25e4b4539 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -394,3 +394,30 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
   !$omp end atomic
 end
 
+! Logical compare+capture inside a parallel region.
+!CHECK-LABEL: define void @atomic_compare_capture_logical_parallel_(
+!CHECK-SAME: ptr noalias %[[X:.*]])
+!CHECK: %[[CGEP:.*]] = getelementptr { ptr }, ptr %structArg, i32 0, i32 0
+!CHECK: store ptr %[[X]], ptr %[[CGEP]]
+!CHECK: call void {{.*}}@__kmpc_fork_call{{.*}}@atomic_compare_capture_logical_parallel_..omp_par
+!CHECK-LABEL: define internal void @atomic_compare_capture_logical_parallel_..omp_par(
+!CHECK-SAME: ptr noalias %{{.*}}, ptr noalias %{{.*}}, ptr %[[STRUCTARG:.*]])
+!CHECK: %[[GEP:.*]] = getelementptr { ptr }, ptr %[[STRUCTARG]], i32 0, i32 0
+!CHECK: %[[SHARED:.*]] = load ptr, ptr %[[GEP]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[SHARED]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %{{.*}}
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 1219d05296e4d..ccc44a370be79 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -305,6 +305,83 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Compare-capture with an explicit seq_cst memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
+! CHECK:         omp.atomic.capture memory_order(seq_cst) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_seq_cst(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture seq_cst
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acquire memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acquire(
+! CHECK:         omp.atomic.capture memory_order(acquire) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acquire(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acquire
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit release memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_release(
+! CHECK:         omp.atomic.capture memory_order(release) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_release(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture release
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acq_rel memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acq_rel(
+! CHECK:         omp.atomic.capture memory_order(acq_rel) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acq_rel(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acq_rel
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
@@ -335,3 +412,43 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
     v = x
   !$omp end atomic
 end
+
+! Logical compare-capture inside a parallel region. The shared atom 'x' is
+! host-associated, and the capture reads it while the conditional update writes
+! it. Verify the atom is converted to i32 once (above the atomic region) and
+! that the capture reads then compares the same converted reference.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical_parallel(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"})
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}Ex"
+! CHECK:         omp.parallel private(
+! CHECK:           %[[OLD_DECL:.*]]:2 = hlfir.declare %{{.*}}Eold_value"
+! CHECK:           %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Eexpected"
+! CHECK:           %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:           %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:           omp.atomic.capture memory_order(relaxed) {
+! CHECK:             omp.atomic.read %[[ATOMVAL:.*]] = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:             omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:             ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:               %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:               %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:               omp.yield(%[[SEL]] : i32)
+! CHECK:             }
+! CHECK:           }
+! CHECK:           %[[RES:.*]] = fir.load %[[ATOMVAL]] : !fir.ref<i32>
+! CHECK:           %[[RESCONV:.*]] = fir.convert %[[RES]] : (i32) -> !fir.logical<4>
+! CHECK:           fir.store %[[RESCONV]] to %[[OLD_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           omp.terminator
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
diff --git a/flang/test/Semantics/OpenMP/atomic-compare.f90 b/flang/test/Semantics/OpenMP/atomic-compare.f90
index 0e53729c2f02a..5cee2359f52be 100644
--- a/flang/test/Semantics/OpenMP/atomic-compare.f90
+++ b/flang/test/Semantics/OpenMP/atomic-compare.f90
@@ -9,6 +9,7 @@
 
   real a, b, c
   logical :: r, s
+  logical :: lx, le, ld, lo
   a = 1.0
   b = 2.0
   c = 3.0
@@ -58,6 +59,17 @@
   if (r) b = c
   !$omp end atomic
 
+  ! Compare-capture with a LOGICAL atom using .eqv. inside a parallel region.
+  ! Regression: the host-associated atom (from the update) must be recognized
+  ! as the same variable as the captured one, even though OpenMP resolves them
+  ! through different (but ultimately identical) symbols.
+  !$omp atomic compare capture
+  lo = lx
+  if (lx .eqv. le) then
+    lx = ld
+  end if
+  !$omp end atomic
+
   ! Check for error conditions:
   !ERROR: At most one SEQ_CST clause can appear on the ATOMIC directive
   !$omp atomic seq_cst seq_cst compare

>From b6f101ded4b5847a466a27c6a37185a3203b6ebc Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 10 Jul 2026 00:50:42 -0500
Subject: [PATCH 9/9] Added exhaustive tests covering all atomic compare
 capture combinations Handling changes for atomic compare capture were not
 covered earlier. Completed runs of llvm-test-suite and Fujitsu flang
 test-suite.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  14 +
 flang/test/Fir/atomic-compare-capture.fir     | 343 ++++++++++++++
 .../OpenMP/atomic-compare-capture.f90         | 443 ++++++++++++++++++
 .../Todo/atomic-compare-capture-result.f90    |  20 +
 .../Lower/OpenMP/Todo/atomic-compare-fail.f90 |   8 -
 .../Lower/OpenMP/atomic-compare-capture.f90   | 356 ++++++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 116 +++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 398 ++++++++++++----
 .../LLVMIR/openmp-atomic-compare-capture.mlir | 407 ++++++++++++++++
 9 files changed, 2007 insertions(+), 98 deletions(-)
 create mode 100644 flang/test/Fir/atomic-compare-capture.fir
 create mode 100644 flang/test/Integration/OpenMP/atomic-compare-capture.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
 create mode 100644 flang/test/Lower/OpenMP/atomic-compare-capture.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 3811ea3d40bbd..ca37eb4a73375 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -606,6 +606,20 @@ void Fortran::lower::omp::lowerAtomic(
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
     }
 
+    // The comparison-result forms of atomic compare
+    //     (e.g. `r = x == e;
+    //           if (r) x = d`)
+    // are accepted by semantics, but the assignment to `r` is
+    // not captured by the atomic analysis and would be silently dropped.
+    // Detect the extra assignment here and emit an explicit TODO instead of
+    // generating incorrect code.
+    if (!construct.IsCapture()) {
+      const parser::Block &body = std::get<parser::Block>(construct.t);
+      if (body.size() > 1)
+        TODO(loc, "atomic compare capturing the comparison result "
+                  "(e.g. 'r = x == e')");
+    }
+
     common::RelationalOperator relOpr = common::RelationalOperator::EQ;
     std::optional<semantics::SomeExpr> expectedExprStorage;
     bool isUnsigned = false;
diff --git a/flang/test/Fir/atomic-compare-capture.fir b/flang/test/Fir/atomic-compare-capture.fir
new file mode 100644
index 0000000000000..65264fbc688a8
--- /dev/null
+++ b/flang/test/Fir/atomic-compare-capture.fir
@@ -0,0 +1,343 @@
+// RUN: fir-opt --split-input-file --cfg-conversion --fir-to-llvm-ir="target=aarch64-unknown-linux-gnu" %s | FileCheck %s
+
+// Test FIR-to-LLVM-dialect conversion of OpenMP atomic compare-capture: the
+// !fir.ref operands become !llvm.ptr and the compare region's arith ops become
+// their llvm-dialect equivalents, while the omp.atomic.capture / .compare /
+// .read structure is preserved for the later MLIR-to-LLVM-IR translation.
+
+//
+// Integer postfix compare-capture: {compare, read}
+func.func @cc_int_postfix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> i32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> i32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: i32):
+// CHECK:        %[[CMP:.*]] = llvm.icmp "eq" %[[XVAL]], %[[EVAL]] : i32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, i32
+// CHECK:        omp.yield(%[[SEL]] : i32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Integer prefix compare-capture: {read, compare}
+func.func @cc_int_prefix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:        llvm.select
+// CHECK:        omp.yield
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Integer fail-only compare-capture: {compare, read} with fail_only attribute.
+func.func @cc_int_failonly(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Real postfix compare-capture: !fir.ref<f32> -> !llvm.ptr, arith.cmpf -> llvm.fcmp.
+func.func @cc_real_postfix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> f32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> f32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: f32):
+// CHECK:        %[[CMP:.*]] = llvm.fcmp "oeq" %[[XVAL]], %[[EVAL]] : f32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, f32
+// CHECK:        omp.yield(%[[SEL]] : f32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Real prefix compare-capture.
+func.func @cc_real_prefix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Real fail-only compare-capture.
+func.func @cc_real_failonly(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Complex postfix compare-capture: !fir.ref<complex<f32>> -> !llvm.ptr and the
+// fir.cmpc is decomposed into per-field llvm.fcmp + llvm.and over the struct.
+func.func @cc_complex_postfix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: !llvm.struct<(f32, f32)>):
+// CHECK:        llvm.extractvalue %[[XVAL]][0]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.extractvalue %[[XVAL]][1]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        %[[AND:.*]] = llvm.and
+// CHECK:        %[[SEL:.*]] = llvm.select %[[AND]], %[[DVAL]], %[[XVAL]] : i1, !llvm.struct<(f32, f32)>
+// CHECK:        omp.yield(%[[SEL]] : !llvm.struct<(f32, f32)>)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    }
+
+// -----
+
+// Complex prefix compare-capture.
+func.func @cc_complex_prefix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Complex fail-only compare-capture.
+func.func @cc_complex_failonly(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    } {fail_only}
+
+// -----
+
+// Integer min compare-capture (x > e): the compare region uses arith.cmpi sgt.
+func.func @cc_int_min(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi sgt, %xval, %eval : i32
+      %s = arith.select %c, %eval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "sgt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Real min compare-capture (x > e): the compare region uses arith.cmpf ogt.
+func.func @cc_real_min(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf ogt, %xval, %eval : f32
+      %s = arith.select %c, %eval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "ogt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Weak clause: preserved through FIR-to-LLVM conversion.
+func.func @cc_weak(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    } {weak}
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_weak(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      } {weak}
+// CHECK:    }
diff --git a/flang/test/Integration/OpenMP/atomic-compare-capture.f90 b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..fe97e7b12558b
--- /dev/null
+++ b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,443 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=FIR
+!RUN: %if x86-registered-target %{ %flang_fc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+!RUN: %if aarch64-registered-target %{ %flang_fc1 -triple aarch64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+
+! OpenMP atomic compare-capture tracked through every stage of the compiler
+! (HLFIR, FIR/omp dialect, LLVM IR) for every operand type: integer, real,
+! logical and complex under ==, plus integer and real under min/max (<, >).
+! Equality lowers to a cmpxchg; min/max lowers to an atomicrmw.
+
+! ===========================================================================
+! integer ==  (postfix): cmpxchg + select(new value) + store into v
+! HLFIR-LABEL: func.func @_QPcc_integer(
+! HLFIR:         %[[X:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           }
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real ==  (postfix): equality on floats uses the HandleFPNegZero cmpxchg path.
+! HLFIR-LABEL: func.func @_QPcc_real(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf oeq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf oeq
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! logical ==  (postfix): the logical atom is converted to i32, so the omp op
+! operates on !fir.ref<i32> and lowers to an i32 cmpxchg.
+! HLFIR-LABEL: func.func @_QPcc_logical(
+! HLFIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! HLFIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_logical(
+! FIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_(
+! LLVM:         cmpxchg ptr %{{.*}}, i32
+! LLVM:         store i32 %{{.*}}, ptr
+subroutine cc_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! complex ==  (postfix): fir.cmpc region; lowers to an i64 (bitcast) cmpxchg
+! and the captured new value is a { float, float } struct.
+! HLFIR-LABEL: func.func @_QPcc_complex(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:           fir.cmpc "oeq"
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_complex(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<complex<f32>> {
+! FIR:           fir.cmpc "oeq"
+! FIR:         }
+! LLVM-LABEL: define void @cc_complex_(
+! LLVM:         cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer min (x > e, postfix): atomicrmw min, v gets the new value smin(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi sgt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi sgt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer max (x < e, postfix): atomicrmw max, v gets the new value smax(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi slt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi slt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real min (x > e, postfix): atomicrmw fmin, v gets minnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf ogt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf ogt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real max (x < e, postfix): atomicrmw fmax, v gets maxnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf olt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf olt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! The remaining cases exercise the different *capture-statement* forms allowed
+! for atomic compare-capture (all with an integer == atom).
+
+! ===========================================================================
+! Prefix form  {v = x; cond-update-stmt}:  the read precedes the update, so v
+! captures the OLD value of x - the read is emitted first and v is a direct
+! store of the extracted old value (no select).
+! HLFIR-LABEL: func.func @_QPcc_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_prefix(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:         }
+! LLVM-LABEL: define void @cc_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Fail-only form  {if(x == e){x = d;} else {v = x;}}:  v is captured only when
+! the compare fails; the capture carries the fail_only attribute and lowers to
+! a conditional store of the old value.
+! HLFIR-LABEL: func.func @_QPcc_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_failonly(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %[[EXIT:.*]], label %[[CONT:.*]]
+! LLVM:         [[CONT]]:
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Reversed comparison operand order  {if(e == x){x = d;} ; v = x}:  the front
+! end normalises 'e == x' to the same compare-capture as 'x == e', so this
+! accepts the alternate spelling and lowers identically (cmpxchg + select).
+! HLFIR-LABEL: func.func @_QPcc_eq_reversed(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_eq_reversed(
+! LLVM-LABEL: define void @cc_eq_reversed_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! The prefix and fail-only forms for the remaining operand types (real,
+! logical, complex), to confirm each type honours every capture form.
+
+! real, prefix: v gets the old value of x (float).
+! HLFIR-LABEL: func.func @_QPcc_real_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! LLVM-LABEL: define void @cc_real_prefix_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! real, fail-only: conditional store of the old value (float).
+! HLFIR-LABEL: func.func @_QPcc_real_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_real_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_real_failonly_(
+! LLVM:         cmpxchg ptr
+! LLVM:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! logical, prefix: v gets the old value (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR-LABEL: func.func @_QPcc_logical_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end
+
+! logical, fail-only: conditional store (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_failonly(
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_logical_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_logical_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %{{.*}}, label %{{.*}}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! complex, prefix: v gets the old complex value (struct), no select.
+! HLFIR-LABEL: func.func @_QPcc_complex_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! FIR-LABEL: func.func @_QPcc_complex_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! LLVM-LABEL: define void @cc_complex_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! complex, fail-only: conditional store of the old complex value.
+! HLFIR-LABEL: func.func @_QPcc_complex_failonly(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_complex_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_complex_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         %[[OK:.*]] = extractvalue { i64, i1 } %[[RES]], 1
+! LLVM:         %[[FAILED:.*]] = xor i1 %[[OK]], true
+! LLVM:         br i1 %[[FAILED]], label %{{.*}}, label %{{.*}}
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! weak clause (postfix): cmpxchg gets the 'weak' modifier.
+! HLFIR-LABEL: func.func @_QPcc_weak(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           } {weak}
+! HLFIR:           omp.atomic.read %{{.*}} = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_weak(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           } {weak}
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_weak_(
+! LLVM:         %[[RES:.*]] = cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_weak(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
new file mode 100644
index 0000000000000..ce6f0a3d6454b
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+! The comparison-result form of atomic compare captures the boolean outcome of
+! the comparison into `r`:
+!   r = x == e
+!   if (r) x = d
+! This is accepted by semantics, but the atomic analysis does not record the
+! assignment to `r`, so lowering would silently drop it and never write `r`.
+! Ensure it is diagnosed as "not yet implemented" instead of producing code that
+! leaves `r` unwritten.
+
+! CHECK: not yet implemented: atomic compare capturing the comparison result
+subroutine f(x, e, d, r)
+  integer :: x, e, d
+  logical :: r
+  !$omp atomic compare
+  r = x == e
+  if (r) x = d
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
index 27c7b1fb1a432..7165bdb2fc122 100644
--- a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
@@ -5,20 +5,12 @@ program p
   integer :: x
   integer :: r
   integer :: d
-  integer :: v
   !$omp atomic compare fail(relaxed)
   if (x .eq. 0) then
      x = 2
   end if
   !$omp end atomic
 
-  !$omp atomic compare capture
-  v = x
-  if (x > r) then
-     x = d
-  end if
-  !$omp end atomic
-
   !$omp atomic compare fail(relaxed)
   if (x > r) then
      x = d
diff --git a/flang/test/Lower/OpenMP/atomic-compare-capture.f90 b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..93fbf2d4bd034
--- /dev/null
+++ b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,356 @@
+! This test checks lowering of atomic compare capture constructs to HLFIR.
+! RUN: bbc %openmp_flags -fopenmp-version=51 -emit-hlfir %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir %openmp_flags -fopenmp-version=51 %s -o - | FileCheck %s
+
+! ---------------------------------------------------------------------------
+! integer == postfix: { if (x==e) x=d; v=x }
+! CHECK-LABEL: func.func @_QPcc_int_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == prefix: { v=x; if (x==e) x=d }
+! CHECK-LABEL: func.func @_QPcc_int_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:         }
+subroutine cc_int_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == fail-only: { if (x==e) x=d; else v=x }
+! CHECK-LABEL: func.func @_QPcc_int_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_int_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == reversed (e==x): same lowering as x==e
+! CHECK-LABEL: func.func @_QPcc_int_eq_reversed(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == postfix
+! CHECK-LABEL: func.func @_QPcc_real_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_postfix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == prefix
+! CHECK-LABEL: func.func @_QPcc_real_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == fail-only
+! CHECK-LABEL: func.func @_QPcc_real_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         } {fail_only}
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. postfix
+! CHECK-LABEL: func.func @_QPcc_logical_postfix(
+! CHECK:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_logical_postfix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. prefix
+! CHECK-LABEL: func.func @_QPcc_logical_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. fail-only
+! CHECK-LABEL: func.func @_QPcc_logical_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == postfix
+! CHECK-LABEL: func.func @_QPcc_complex_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine cc_complex_postfix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == prefix
+! CHECK-LABEL: func.func @_QPcc_complex_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:         }
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == fail-only
+! CHECK-LABEL: func.func @_QPcc_complex_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         } {fail_only}
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi sgt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi slt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf ogt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf olt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (prefix form)
+! CHECK-LABEL: func.func @_QPcc_weak_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:         }
+subroutine cc_weak_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (postfix form)
+! CHECK-LABEL: func.func @_QPcc_weak_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_weak_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ccc44a370be79..83e6f81ca00e3 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -218,6 +218,101 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
   !$omp end atomic
 end
 
+! Min/max (postfix) compare-capture: update first, capture the new value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_min_postfix(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! Min/max fail-only compare-capture: v is captured in the else branch (when the
+! comparison, and hence the update, does not happen).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_min_failonly(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) then
+      x = e
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
+! REAL min (postfix): if (x > e) x = e ; v = x -> compare region uses arith.cmpf.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_min(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %{{.*}}Ev"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! REAL max (postfix): if (x < e) x = e ; v = x.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_max(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %{{.*}} fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x < e) x = e
+    v = x
+  !$omp end atomic
+end
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
@@ -305,6 +400,27 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Complex compare-capture (postfix form): the compare region decomposes into
+! per-field comparisons combined with fir.cmpc/arith and the desired value is
+! selected as a whole complex value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_complex(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<complex<f32>> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: complex<f32>):
+! CHECK:             %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %{{.*}} : complex<f32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : complex<f32>
+! CHECK:             omp.yield(%[[SEL]] : complex<f32>)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine atomic_compare_capture_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
 ! Compare-capture with an explicit seq_cst memory order (prefix read form).
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
 ! CHECK:         omp.atomic.capture memory_order(seq_cst) {
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 8e49571cd2461..060e563ecefa8 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5370,6 +5370,99 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Result of matching the decomposed complex equality pattern inside an atomic
+/// compare region.
+struct ComplexComparePattern {
+  bool isComplex = false;
+  bool isNE = false;         // `or` of the field compares => NE (unsupported).
+  mlir::Value eAggregate;    // The complex expected value (`e`).
+  bool isXBinopExpr = false; // True if x is the first fcmp operand.
+};
+
+/// Detect a decomposed complex equality comparison in an atomic compare region:
+///   %re_x = llvm.extractvalue %xval[0]
+///   %re_e = llvm.extractvalue %eStruct[0]
+///   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
+///   %im_x = llvm.extractvalue %xval[1]
+///   %im_e = llvm.extractvalue %eStruct[1]
+///   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
+///   %cmp = llvm.and %cmp_re, %cmp_im   (llvm.or would be NE)
+/// It is recognised by an and/or whose operands are both fcmps operating on
+/// extractvalues, one chain rooted at the block argument (x) and the other at
+/// the expected complex value (e).
+static ComplexComparePattern detectComplexCompareEq(Block &block) {
+  ComplexComparePattern result;
+  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
+    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
+      return extractOp.getContainer();
+    return nullptr;
+  };
+  for (Operation &op : block.getOperations()) {
+    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
+      continue;
+    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
+    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
+    if (!lhsFcmp || !rhsFcmp)
+      continue;
+    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
+    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
+    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
+    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
+    if (!lhsXIsOp0 && !lhsXIsOp1)
+      continue;
+    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
+    if (!eAggregate)
+      continue;
+    result.isComplex = true;
+    result.isNE = isa<LLVM::OrOp>(op);
+    result.eAggregate = eAggregate;
+    result.isXBinopExpr = lhsXIsOp0;
+    break;
+  }
+  return result;
+}
+
+/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
+/// compare. The expected/desired complex values are spilled to allocas and
+/// reloaded as an integer of the same width, then compared and swapped. Returns
+/// the cmpxchg (result type `{iN, i1}`); `maxAlign` receives the alignment used
+/// so callers can reinterpret the captured old value through memory.
+static llvm::AtomicCmpXchgInst *
+emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
+                         llvm::Type *complexTy, llvm::Value *eVal,
+                         llvm::Value *dVal, llvm::AtomicOrdering atomicOrdering,
+                         bool isWeak, llvm::Align &maxAlign) {
+  const llvm::DataLayout &DL =
+      builder.GetInsertBlock()->getModule()->getDataLayout();
+  unsigned totalBits = DL.getTypeStoreSizeInBits(complexTy).getFixedValue();
+  llvm::IntegerType *intTy =
+      llvm::IntegerType::get(builder.getContext(), totalBits);
+  llvm::Align complexAlign = DL.getABITypeAlign(complexTy);
+  llvm::Align intAlign = DL.getABITypeAlign(intTy);
+  maxAlign = std::max(complexAlign, intAlign);
+
+  llvm::AllocaInst *eAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.e");
+  eAlloca->setAlignment(maxAlign);
+  llvm::AllocaInst *dAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.d");
+  dAlloca->setAlignment(maxAlign);
+
+  builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
+  llvm::Value *eInt =
+      builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
+  builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
+  llvm::Value *dInt =
+      builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
+
+  llvm::AtomicOrdering failOrdering =
+      llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
+  auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
+                                              atomicOrdering, failOrdering);
+  cmpXchg->setWeak(isWeak);
+  return cmpXchg;
+}
+
 /// Holds the extracted comparison pattern information from an atomic compare
 /// region.
 struct AtomicComparePatternInfo {
@@ -5379,7 +5472,6 @@ struct AtomicComparePatternInfo {
   bool isXBinopExpr = false;
   bool isSigned = false;
 };
-
 /// Extract comparison predicate, expected value (e), desired value (d), and
 /// related flags from an atomic compare region block by scanning for
 /// icmp/fcmp/select/min/max operations.
@@ -5387,6 +5479,27 @@ static LogicalResult extractAtomicComparePattern(
     Block &block,
     llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
     omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  // Complex equality is a decomposed per-field pattern (extractvalue + fcmp +
+  // and) rather than a single scalar compare. Detect it first so the scalar
+  // icmp/fcmp handling below does not mistake a real/imaginary field for the
+  // whole expected value.
+  if (ComplexComparePattern cplx = detectComplexCompareEq(block);
+      cplx.isComplex) {
+    if (cplx.isNE)
+      return atomicCompareOp.emitError(
+          "unsupported comparison predicate (NE) for complex atomic compare");
+    info.compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    info.isXBinopExpr = cplx.isXBinopExpr;
+    info.eVal = materializeValue(cplx.eAggregate);
+    for (Operation &op : block.getOperations()) {
+      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        info.dVal = materializeValue(selectOp.getTrueValue());
+        break;
+      }
+    }
+    return success();
+  }
+
   for (Operation &op : block.getOperations()) {
     // Pre-filter: skip icmps that don't involve the block argument
     // (e.g., truthiness extractions from logical-to-integer conversion).
@@ -5573,22 +5686,101 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
+    // Complex equality capture: x is struct-typed, which the OMPIRBuilder
+    // cannot handle, so emit a bitcast-to-integer cmpxchg (as in the
+    // non-capture complex path) and reconstruct the captured value from its
+    // result through memory. Complex only supports the == comparison.
+    if (llvmXElementType->isStructTy()) {
+      llvm::Align maxAlign;
+      llvm::AtomicCmpXchgInst *cmpXchg = emitComplexAtomicCmpXchg(
+          builder, llvmX, llvmXElementType, eVal, dVal, atomicOrdering,
+          atomicCompareOp.getWeak(), maxAlign);
+
+      llvm::Value *oldInt = builder.CreateExtractValue(cmpXchg, 0);
+      llvm::Value *cmpOk = builder.CreateExtractValue(cmpXchg, 1);
+
+      // Reinterpret the old integer value as the complex struct via memory.
+      llvm::AllocaInst *oldAlloca =
+          builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.old");
+      oldAlloca->setAlignment(maxAlign);
+      builder.CreateAlignedStore(oldInt, oldAlloca, maxAlign);
+      llvm::Value *oldComplex = builder.CreateAlignedLoad(
+          llvmXElementType, oldAlloca, maxAlign, "cmplx.old.val");
+
+      if (isFailOnly) {
+        // v is written only when the compare fails (cmpOk == false).
+        llvm::Value *cmpFailed = builder.CreateNot(cmpOk);
+        llvm::BasicBlock *curBB = builder.GetInsertBlock();
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(cmpFailed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else if (isPostfixCapture) {
+        // v gets the new value of x: d on success, old x otherwise.
+        llvm::Value *newComplex = builder.CreateSelect(cmpOk, dVal, oldComplex);
+        builder.CreateStore(newComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      } else {
+        // Prefix: v gets the old value of x.
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      }
+
+      // Emit flush after atomic compare if needed (release/acq_rel/seq_cst).
+      if (atomicOrdering == llvm::AtomicOrdering::Release ||
+          atomicOrdering == llvm::AtomicOrdering::AcquireRelease ||
+          atomicOrdering == llvm::AtomicOrdering::SequentiallyConsistent) {
+        llvm::OpenMPIRBuilder::LocationDescription flushLoc(builder);
+        ompBuilder->createFlush(flushLoc);
+      }
+      return success();
+    }
+
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw, so the fail-only capture
+    // form (v written only when the compare fails) has no valid mapping and is
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw (it asserts on IsFailOnly),
+    // so for min/max the fail-only capture is reconstructed manually below.
+    bool isMinMax = compareOp != llvm::omp::OMPAtomicCompareOp::EQ;
+
     llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
-    // Postfix: bypass V in OMPIRBuilder, handled manually below.
-    if (isPostfixCapture && !isFailOnly)
+    // Capture into V is reconstructed manually below for:
+    //   * postfix capture (v gets the new value of x): equality from the
+    //     cmpxchg result, min/max from the atomicrmw result;
+    //   * min/max fail-only capture (the OMPIRBuilder cannot express it).
+    // Bypass V in the OMPIRBuilder for those cases so it does not also emit its
+    // own (for min/max, incorrect or unsupported) capture store.
+    bool minMaxManualCapture = isMinMax && (isPostfixCapture || isFailOnly);
+    bool eqPostfixManualCapture = !isMinMax && isPostfixCapture && !isFailOnly;
+    if (minMaxManualCapture || eqPostfixManualCapture)
       llvmAtomicVForCall = {nullptr, nullptr, false, false};
 
-    // Prefix: IsPostfixUpdate=true → direct store of old value.
-    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
-    bool isPostfixUpdate = !isFailOnly;
+    // The OMPIRBuilder only understands IsFailOnly for the equality (cmpxchg)
+    // path; for min/max it would assert. Min/max fail-only is handled here.
+    bool builderFailOnly = isFailOnly && !isMinMax;
+
+    // IsPostfixUpdate selects which value the OMPIRBuilder captures into V:
+    //   * min/max prefix and equality prefix: a direct store of the old value
+    //     (IsPostfixUpdate=true).
+    //   * equality fail-only: a conditional store (IsPostfixUpdate=false).
+    // Manually-reconstructed captures bypass V above.
+    bool isPostfixUpdate = !builderFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly, isWeak);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicVForCall, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            builderFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
@@ -5596,8 +5788,99 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     builder.restoreIP(*afterIP);
 
-    // Postfix: v = select(success, D, old) — captures new value of x.
-    if (isPostfixCapture && !isFailOnly) {
+    // Min/max capture is reconstructed from the atomicrmw the OMPIRBuilder
+    // emits (its result is the old value of x). V was bypassed above.
+    //   * postfix: v gets the new value min/max(old, e);
+    //   * fail-only: v gets the old value, but only when the compare failed
+    //     (i.e. the atomicrmw did not change x).
+    // (Prefix min/max captures the old value directly through V, so nothing
+    // extra is needed there.)
+    if (isMinMax && (isPostfixCapture || isFailOnly)) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::AtomicRMWInst *rmw = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (auto *r = dyn_cast<llvm::AtomicRMWInst>(&inst)) {
+          rmw = r;
+          break;
+        }
+      }
+      assert(rmw && "expected atomicrmw for min/max compare capture");
+      llvm::Value *oldVal = rmw;
+      llvm::Value *rhs = rmw->getValOperand();
+
+      if (isFailOnly) {
+        // The compare "failed" (the else branch runs) exactly when the
+        // atomicrmw did not change x. Recompute the original update condition
+        // on the old value and negate it. v is stored only in that case.
+        llvm::CmpInst::Predicate updatePred;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          updatePred = llvm::CmpInst::ICMP_SGT;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          updatePred = llvm::CmpInst::ICMP_SLT;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          updatePred = llvm::CmpInst::ICMP_UGT;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          updatePred = llvm::CmpInst::ICMP_ULT;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          updatePred = llvm::CmpInst::FCMP_OGT;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          updatePred = llvm::CmpInst::FCMP_OLT;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *updated = builder.CreateCmp(updatePred, oldVal, rhs);
+        llvm::Value *failed = builder.CreateNot(updated);
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(failed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else {
+        llvm::Intrinsic::ID id;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          id = llvm::Intrinsic::smin;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          id = llvm::Intrinsic::smax;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          id = llvm::Intrinsic::umin;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          id = llvm::Intrinsic::umax;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          id = llvm::Intrinsic::minnum;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          id = llvm::Intrinsic::maxnum;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *newVal = builder.CreateBinaryIntrinsic(id, oldVal, rhs);
+        builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+      }
+    }
+
+    // Equality postfix: v = select(success, D, old) — reconstructs the new
+    // value of x from the cmpxchg result.
+    if (!isMinMax && isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
       llvm::Value *oldVal = nullptr;
       llvm::Value *successVal = nullptr;
@@ -5848,56 +6131,18 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
   llvm::Value *dVal = nullptr;
   bool isXBinopExpr = false;
 
-  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
-    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
-      return extractOp.getContainer();
-    return nullptr;
-  };
-
-  // Check for a decomposed complex comparison pattern:
-  //   %re_x = llvm.extractvalue %xval[0]
-  //   %re_e = llvm.extractvalue %eStruct[0]
-  //   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-  //   %im_x = llvm.extractvalue %xval[1]
-  //   %im_e = llvm.extractvalue %eStruct[1]
-  //   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
-  //   %cmp = llvm.and %cmp_re, %cmp_im   (for EQ)
-  // Detect this by looking for AndOp/OrOp whose operands are both FCmpOps
-  // operating on ExtractValueOps from the block argument.
-  bool isComplexPattern = false;
-  for (Operation &op : block.getOperations()) {
-    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
-      continue;
-
-    // Using : %cmp = llvm.and %cmp_re, %cmp_im
-    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
-    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
-    if (!lhsFcmp || !rhsFcmp)
-      continue;
-
-    // Using : %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-    // Check presence of x (block argument) and get e.
-    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
-    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
-    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
-    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
-    if (!lhsXIsOp0 && !lhsXIsOp1)
-      continue;
-    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
-    if (!eAggregate)
-      continue;
-
-    if (isa<LLVM::AndOp>(op))
-      compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    else
+  // Check for a decomposed complex comparison pattern (extractvalue + fcmp +
+  // and/or of the real/imaginary fields).
+  ComplexComparePattern cplx = detectComplexCompareEq(block);
+  bool isComplexPattern = cplx.isComplex;
+  if (isComplexPattern) {
+    if (cplx.isNE)
       // OrOp corresponds to NE, which is not a valid atomic compare op.
       return atomicCompareOp.emitError(
           "unsupported comparison predicate (NE) for complex atomic compare");
-
-    isXBinopExpr = lhsXIsOp0;
-    eVal = materializeValue(eAggregate);
-    isComplexPattern = true;
-    break;
+    compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    isXBinopExpr = cplx.isXBinopExpr;
+    eVal = materializeValue(cplx.eAggregate);
   }
 
   if (isComplexPattern) {
@@ -5916,37 +6161,10 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
       dVal = materializeValue(yieldOp.getResults()[0]);
     }
 
-    const llvm::DataLayout &DL =
-        builder.GetInsertBlock()->getModule()->getDataLayout();
-    unsigned totalBits =
-        DL.getTypeStoreSizeInBits(llvmXElementType).getFixedValue();
-
-    llvm::IntegerType *intTy =
-        llvm::IntegerType::get(builder.getContext(), totalBits);
-
-    llvm::Align complexAlign = DL.getABITypeAlign(llvmXElementType);
-    llvm::Align intAlign = DL.getABITypeAlign(intTy);
-    llvm::Align maxAlign = std::max(complexAlign, intAlign);
-
-    llvm::AllocaInst *eAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.e");
-    eAlloca->setAlignment(maxAlign);
-    llvm::AllocaInst *dAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.d");
-    dAlloca->setAlignment(maxAlign);
-
-    builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
-    llvm::Value *eInt =
-        builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
-    builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
-    llvm::Value *dInt =
-        builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
-
-    llvm::AtomicOrdering failOrdering =
-        llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
-    auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
-                                                atomicOrdering, failOrdering);
-    cmpXchg->setWeak(atomicCompareOp.getWeak());
+    llvm::Align maxAlign;
+    emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
+                             atomicOrdering, atomicCompareOp.getWeak(),
+                             maxAlign);
 
     // Emit flush after atomic compare if needed (for release, acq_rel,
     // seq_cst orderings).
diff --git a/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
new file mode 100644
index 0000000000000..2a3cb6dc99133
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
@@ -0,0 +1,407 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Prefix compare+capture: {read, compare}
+// CHECK-LABEL: define void @compare_capture_prefix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Postfix compare+capture: {compare, read}
+// CHECK-LABEL: define void @compare_capture_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+// CHECK:         store i32 %[[NEWVAL]], ptr %[[V]], align 4
+llvm.func @compare_capture_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Fail-only compare+capture: {compare, read} with fail_only
+// CHECK-LABEL: define void @compare_capture_failonly(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         br i1 %[[SUCCESS]], label %{{.*}}.atomic.exit, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store i32 %[[OLD]], ptr %[[V]]
+// CHECK:         br label %{{.*}}.atomic.exit
+// CHECK:       {{.*}}.atomic.exit:
+llvm.func @compare_capture_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Weak compare+capture: {read, compare} with weak
+// CHECK-LABEL: define void @compare_capture_weak(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg weak ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Float (real) tests =====
+// Float uses the HandleFPNegZero path with NaN/zero checks and bitcast.
+
+// Float prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_float_prefix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         store float %{{.*}}, ptr %[[V:.*]], align 4
+llvm.func @compare_capture_float_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+  }
+  llvm.return
+}
+
+// Float postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_float_postfix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         %[[NEW:.*]] = select i1 %[[OK]], float %{{.*}}, float %[[OLD_FP]]
+// CHECK:         store float %[[NEW]], ptr %{{.*}}, align 4
+llvm.func @compare_capture_float_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_float_failonly(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         br i1 %[[OK]], label %{{.*}}.atomic.exit{{.*}}, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store float %[[OLD_FP]], ptr %{{.*}}
+llvm.func @compare_capture_float_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Complex (struct<(f32, f32)>) tests =====
+// Complex uses bitcast to i64 for cmpxchg, with FP neg-zero handling.
+
+// Complex prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_complex_prefix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+  }
+  llvm.return
+}
+
+// Complex postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_complex_postfix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         select i1 %{{.*}}, { float, float } %{{.*}}, { float, float } %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  }
+  llvm.return
+}
+
+// Complex fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_complex_failonly(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Weak clause with float =====
+
+// Float weak prefix: cmpxchg weak with FP neg-zero handling
+// CHECK-LABEL: define void @compare_capture_float_weak(
+// CHECK:         cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+// CHECK:         store float %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_float_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Min/Max compare+capture =====
+
+// Integer min, postfix: atomicrmw min, v gets min(old, e)
+// CHECK-LABEL: define void @compare_capture_min_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]])
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %[[X]], i32 %[[E]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %[[V]]
+llvm.func @compare_capture_min_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer max, postfix: atomicrmw max, v gets max(old, e)
+// CHECK-LABEL: define void @compare_capture_max_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_max_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer min, prefix: v gets old value
+// CHECK-LABEL: define void @compare_capture_min_prefix(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %{{.*}} monotonic
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_prefix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Float min, postfix: atomicrmw fmin, v gets minnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmin_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmin_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "ogt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float max, postfix: atomicrmw fmax, v gets maxnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmax_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmax_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "olt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// ===== Min/Max fail-only =====
+
+// Integer min, fail-only: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_min_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp sgt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Integer max, fail-only
+// CHECK-LABEL: define void @compare_capture_max_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp slt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_max_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}


From llvmlistbot at llvm.org  Fri Jul 10 01:22:03 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Fri, 10 Jul 2026 01:22:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
Message-ID: <llvm.org/llvm/llvm-project/pull/208648@github.com>


https://github.com/jacquesguan created https://github.com/llvm/llvm-project/pull/208648

None

>From 8211ec080fa7d8436e533a33caecdc365c6e9d63 Mon Sep 17 00:00:00 2001
From: Jianjian GUAN <jacquesguan at me.com>
Date: Fri, 10 Jul 2026 14:59:57 +0800
Subject: [PATCH] [mlir][emitc] Add increment and decrement ops

---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td | 45 +++++++++++++++++++
 mlir/lib/Target/Cpp/TranslateToCpp.cpp      | 48 ++++++++++++++++++++-
 mlir/test/Dialect/EmitC/invalid_ops.mlir    | 16 +++++++
 mlir/test/Dialect/EmitC/ops.mlir            |  9 ++++
 mlir/test/Target/Cpp/common-cpp.mlir        | 35 +++++++++++++++
 5 files changed, 152 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..c52f5b5f909c4 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -65,6 +65,39 @@ class EmitC_BinaryOp<string mnemonic, list<Trait> traits = []> :
   }];
 }
 
+// Base class for increment and decrement operations.
+class EmitC_IncDecOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, [
+        CExpressionInterface,
+        TypesMatchWith<"input and result reference the same type",
+                       "operand", "result",
+                       "::llvm::cast<emitc::LValueType>($_self).getValueType()">
+      ]> {
+  let summary = summaryStr;
+  let description = [{
+    This operation models a C/C++ increment or decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+
+  let arguments = (ins
+    Res<EmitC_LValueType, "",
+        [MemRead<DefaultResource, 0, FullEffect>,
+         MemWrite<DefaultResource, 0, FullEffect>]>:$operand);
+  let results = (outs EmitCType:$result);
+  let assemblyFormat = "$operand attr-dict `:` type($operand)";
+
+  let extraClassDeclaration = [{
+    bool hasSideEffects() {
+      return true;
+    }
+  }];
+}
+
 // Types only used in binary arithmetic operations.
 def IntegerIndexOrOpaqueType : Type<CPred<"emitc::isIntegerIndexOrOpaqueType($_self)">,
 "integer, index or opaque type supported by EmitC">;
@@ -1259,6 +1292,18 @@ def EmitC_ConditionalOp : EmitC_Op<"conditional",
   }];
 }
 
+def EmitC_PreIncrementOp
+    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation">;
+
+def EmitC_PostIncrementOp
+    : EmitC_IncDecOp<"post_increment", "Post-increment operation">;
+
+def EmitC_PreDecrementOp
+    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation">;
+
+def EmitC_PostDecrementOp
+    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation">;
+
 def EmitC_UnaryMinusOp : EmitC_UnaryOp<"unary_minus", []> {
   let summary = "Unary minus operation";
   let description = [{
diff --git a/mlir/lib/Target/Cpp/TranslateToCpp.cpp b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
index 1bf87e4994d67..5ae51bd580d31 100644
--- a/mlir/lib/Target/Cpp/TranslateToCpp.cpp
+++ b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
@@ -112,6 +112,10 @@ static FailureOr<int> getOperatorPrecedence(Operation *operation) {
       .Case([&](emitc::MemberOfPtrOp op) { return 17; })
       .Case([&](emitc::MemberOp op) { return 17; })
       .Case([&](emitc::MulOp op) { return 13; })
+      .Case([&](emitc::PostDecrementOp op) { return 16; })
+      .Case([&](emitc::PostIncrementOp op) { return 16; })
+      .Case([&](emitc::PreDecrementOp op) { return 15; })
+      .Case([&](emitc::PreIncrementOp op) { return 15; })
       .Case([&](emitc::RemOp op) { return 13; })
       .Case([&](emitc::SubOp op) { return 12; })
       .Case([&](emitc::SubscriptOp op) { return 17; })
@@ -646,6 +650,22 @@ static LogicalResult printUnaryOperation(CppEmitter &emitter,
   return success();
 }
 
+static LogicalResult printPostfixUnaryOperation(CppEmitter &emitter,
+                                                Operation *operation,
+                                                StringRef unaryOperator) {
+  raw_ostream &os = emitter.ostream();
+
+  if (failed(emitter.emitAssignPrefix(*operation)))
+    return failure();
+
+  if (failed(emitter.emitOperand(operation->getOperand(0))))
+    return failure();
+
+  os << unaryOperator;
+
+  return success();
+}
+
 static LogicalResult printOperation(CppEmitter &emitter, emitc::AddOp addOp) {
   Operation *operation = addOp.getOperation();
 
@@ -1038,6 +1058,30 @@ static LogicalResult printOperation(CppEmitter &emitter,
   return printBinaryOperation(emitter, operation, "^");
 }
 
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreIncrementOp preIncrementOp) {
+  Operation *operation = preIncrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostIncrementOp postIncrementOp) {
+  Operation *operation = postIncrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreDecrementOp preDecrementOp) {
+  Operation *operation = preDecrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "--");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostDecrementOp postDecrementOp) {
+  Operation *operation = postDecrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "--");
+}
+
 static LogicalResult printOperation(CppEmitter &emitter,
                                     emitc::UnaryPlusOp unaryPlusOp) {
   Operation *operation = unaryPlusOp.getOperation();
@@ -1896,7 +1940,9 @@ LogicalResult CppEmitter::emitOperation(Operation &op, bool trailingSemicolon) {
                 emitc::IncludeOp, emitc::LiteralOp, emitc::LoadOp,
                 emitc::LogicalAndOp, emitc::LogicalNotOp, emitc::LogicalOrOp,
                 emitc::MemberCallOpaqueOp, emitc::MemberOfPtrOp,
-                emitc::MemberOp, emitc::MulOp, emitc::RemOp, emitc::ReturnOp,
+                emitc::MemberOp, emitc::MulOp, emitc::PostDecrementOp,
+                emitc::PostIncrementOp, emitc::PreDecrementOp,
+                emitc::PreIncrementOp, emitc::RemOp, emitc::ReturnOp,
                 emitc::SubscriptOp, emitc::SubOp, emitc::SwitchOp,
                 emitc::UnaryMinusOp, emitc::UnaryPlusOp, emitc::VariableOp,
                 emitc::VerbatimOp>(
diff --git a/mlir/test/Dialect/EmitC/invalid_ops.mlir b/mlir/test/Dialect/EmitC/invalid_ops.mlir
index eacf4fdedcbd3..3048160d1590c 100644
--- a/mlir/test/Dialect/EmitC/invalid_ops.mlir
+++ b/mlir/test/Dialect/EmitC/invalid_ops.mlir
@@ -981,3 +981,19 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   %1 = "emitc.dereference"(%arg0) : (!emitc.ptr<i32>) -> !emitc.lvalue<i8>
   return
 }
+
+// -----
+
+func.func @pre_increment_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.pre_increment"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
+
+// -----
+
+func.func @post_decrement_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.post_decrement"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
diff --git a/mlir/test/Dialect/EmitC/ops.mlir b/mlir/test/Dialect/EmitC/ops.mlir
index e456db4634892..e5c4af67c4896 100644
--- a/mlir/test/Dialect/EmitC/ops.mlir
+++ b/mlir/test/Dialect/EmitC/ops.mlir
@@ -148,6 +148,15 @@ func.func @unary(%arg0: i32) {
   return
 }
 
+func.func @inc_dec(%arg0: !emitc.ptr<i32>) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
 func.func @test_if(%arg0: i1, %arg1: f32) {
   emitc.if %arg0 {
      %0 = emitc.call_opaque "func_const"(%arg1) : (f32) -> i32
diff --git a/mlir/test/Target/Cpp/common-cpp.mlir b/mlir/test/Target/Cpp/common-cpp.mlir
index 8e84433eaa2f3..1bffd59d4d44c 100644
--- a/mlir/test/Target/Cpp/common-cpp.mlir
+++ b/mlir/test/Target/Cpp/common-cpp.mlir
@@ -107,6 +107,41 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   return
 }
 
+// CHECK-LABEL: void inc_dec() {
+func.func @inc_dec() {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = ++[[V1]];
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]++;
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = --[[V1]];
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V5:[^ ]*]] = [[V1]]--;
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
+// CHECK-LABEL: int32_t inc_dec_expression() {
+func.func @inc_dec_expression() -> i32 {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %inc = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_increment %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = [[V1]]++;
+  %dec = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_decrement %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]--;
+  %r = emitc.add %inc, %dec : (i32, i32) -> i32
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = [[V2]] + [[V3]];
+  // CHECK-NEXT: return [[V4]];
+  return %r : i32
+}
+
 // CHECK: void array_type(int32_t v1[3], float v2[10][20])
 func.func @array_type(%arg0: !emitc.array<3xi32>, %arg1: !emitc.array<10x20xf32>) {
   return


From llvmlistbot at llvm.org  Fri Jul 10 01:22:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 01:22:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a50abcc.c56fb102.7c9cd.1ce7@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-emitc

Author: Jianjian Guan (jacquesguan)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/208648.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/EmitC/IR/EmitC.td (+45) 
- (modified) mlir/lib/Target/Cpp/TranslateToCpp.cpp (+47-1) 
- (modified) mlir/test/Dialect/EmitC/invalid_ops.mlir (+16) 
- (modified) mlir/test/Dialect/EmitC/ops.mlir (+9) 
- (modified) mlir/test/Target/Cpp/common-cpp.mlir (+35) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..c52f5b5f909c4 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -65,6 +65,39 @@ class EmitC_BinaryOp<string mnemonic, list<Trait> traits = []> :
   }];
 }
 
+// Base class for increment and decrement operations.
+class EmitC_IncDecOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, [
+        CExpressionInterface,
+        TypesMatchWith<"input and result reference the same type",
+                       "operand", "result",
+                       "::llvm::cast<emitc::LValueType>($_self).getValueType()">
+      ]> {
+  let summary = summaryStr;
+  let description = [{
+    This operation models a C/C++ increment or decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+
+  let arguments = (ins
+    Res<EmitC_LValueType, "",
+        [MemRead<DefaultResource, 0, FullEffect>,
+         MemWrite<DefaultResource, 0, FullEffect>]>:$operand);
+  let results = (outs EmitCType:$result);
+  let assemblyFormat = "$operand attr-dict `:` type($operand)";
+
+  let extraClassDeclaration = [{
+    bool hasSideEffects() {
+      return true;
+    }
+  }];
+}
+
 // Types only used in binary arithmetic operations.
 def IntegerIndexOrOpaqueType : Type<CPred<"emitc::isIntegerIndexOrOpaqueType($_self)">,
 "integer, index or opaque type supported by EmitC">;
@@ -1259,6 +1292,18 @@ def EmitC_ConditionalOp : EmitC_Op<"conditional",
   }];
 }
 
+def EmitC_PreIncrementOp
+    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation">;
+
+def EmitC_PostIncrementOp
+    : EmitC_IncDecOp<"post_increment", "Post-increment operation">;
+
+def EmitC_PreDecrementOp
+    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation">;
+
+def EmitC_PostDecrementOp
+    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation">;
+
 def EmitC_UnaryMinusOp : EmitC_UnaryOp<"unary_minus", []> {
   let summary = "Unary minus operation";
   let description = [{
diff --git a/mlir/lib/Target/Cpp/TranslateToCpp.cpp b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
index 1bf87e4994d67..5ae51bd580d31 100644
--- a/mlir/lib/Target/Cpp/TranslateToCpp.cpp
+++ b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
@@ -112,6 +112,10 @@ static FailureOr<int> getOperatorPrecedence(Operation *operation) {
       .Case([&](emitc::MemberOfPtrOp op) { return 17; })
       .Case([&](emitc::MemberOp op) { return 17; })
       .Case([&](emitc::MulOp op) { return 13; })
+      .Case([&](emitc::PostDecrementOp op) { return 16; })
+      .Case([&](emitc::PostIncrementOp op) { return 16; })
+      .Case([&](emitc::PreDecrementOp op) { return 15; })
+      .Case([&](emitc::PreIncrementOp op) { return 15; })
       .Case([&](emitc::RemOp op) { return 13; })
       .Case([&](emitc::SubOp op) { return 12; })
       .Case([&](emitc::SubscriptOp op) { return 17; })
@@ -646,6 +650,22 @@ static LogicalResult printUnaryOperation(CppEmitter &emitter,
   return success();
 }
 
+static LogicalResult printPostfixUnaryOperation(CppEmitter &emitter,
+                                                Operation *operation,
+                                                StringRef unaryOperator) {
+  raw_ostream &os = emitter.ostream();
+
+  if (failed(emitter.emitAssignPrefix(*operation)))
+    return failure();
+
+  if (failed(emitter.emitOperand(operation->getOperand(0))))
+    return failure();
+
+  os << unaryOperator;
+
+  return success();
+}
+
 static LogicalResult printOperation(CppEmitter &emitter, emitc::AddOp addOp) {
   Operation *operation = addOp.getOperation();
 
@@ -1038,6 +1058,30 @@ static LogicalResult printOperation(CppEmitter &emitter,
   return printBinaryOperation(emitter, operation, "^");
 }
 
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreIncrementOp preIncrementOp) {
+  Operation *operation = preIncrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostIncrementOp postIncrementOp) {
+  Operation *operation = postIncrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreDecrementOp preDecrementOp) {
+  Operation *operation = preDecrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "--");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostDecrementOp postDecrementOp) {
+  Operation *operation = postDecrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "--");
+}
+
 static LogicalResult printOperation(CppEmitter &emitter,
                                     emitc::UnaryPlusOp unaryPlusOp) {
   Operation *operation = unaryPlusOp.getOperation();
@@ -1896,7 +1940,9 @@ LogicalResult CppEmitter::emitOperation(Operation &op, bool trailingSemicolon) {
                 emitc::IncludeOp, emitc::LiteralOp, emitc::LoadOp,
                 emitc::LogicalAndOp, emitc::LogicalNotOp, emitc::LogicalOrOp,
                 emitc::MemberCallOpaqueOp, emitc::MemberOfPtrOp,
-                emitc::MemberOp, emitc::MulOp, emitc::RemOp, emitc::ReturnOp,
+                emitc::MemberOp, emitc::MulOp, emitc::PostDecrementOp,
+                emitc::PostIncrementOp, emitc::PreDecrementOp,
+                emitc::PreIncrementOp, emitc::RemOp, emitc::ReturnOp,
                 emitc::SubscriptOp, emitc::SubOp, emitc::SwitchOp,
                 emitc::UnaryMinusOp, emitc::UnaryPlusOp, emitc::VariableOp,
                 emitc::VerbatimOp>(
diff --git a/mlir/test/Dialect/EmitC/invalid_ops.mlir b/mlir/test/Dialect/EmitC/invalid_ops.mlir
index eacf4fdedcbd3..3048160d1590c 100644
--- a/mlir/test/Dialect/EmitC/invalid_ops.mlir
+++ b/mlir/test/Dialect/EmitC/invalid_ops.mlir
@@ -981,3 +981,19 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   %1 = "emitc.dereference"(%arg0) : (!emitc.ptr<i32>) -> !emitc.lvalue<i8>
   return
 }
+
+// -----
+
+func.func @pre_increment_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.pre_increment"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
+
+// -----
+
+func.func @post_decrement_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.post_decrement"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
diff --git a/mlir/test/Dialect/EmitC/ops.mlir b/mlir/test/Dialect/EmitC/ops.mlir
index e456db4634892..e5c4af67c4896 100644
--- a/mlir/test/Dialect/EmitC/ops.mlir
+++ b/mlir/test/Dialect/EmitC/ops.mlir
@@ -148,6 +148,15 @@ func.func @unary(%arg0: i32) {
   return
 }
 
+func.func @inc_dec(%arg0: !emitc.ptr<i32>) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
 func.func @test_if(%arg0: i1, %arg1: f32) {
   emitc.if %arg0 {
      %0 = emitc.call_opaque "func_const"(%arg1) : (f32) -> i32
diff --git a/mlir/test/Target/Cpp/common-cpp.mlir b/mlir/test/Target/Cpp/common-cpp.mlir
index 8e84433eaa2f3..1bffd59d4d44c 100644
--- a/mlir/test/Target/Cpp/common-cpp.mlir
+++ b/mlir/test/Target/Cpp/common-cpp.mlir
@@ -107,6 +107,41 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   return
 }
 
+// CHECK-LABEL: void inc_dec() {
+func.func @inc_dec() {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = ++[[V1]];
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]++;
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = --[[V1]];
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V5:[^ ]*]] = [[V1]]--;
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
+// CHECK-LABEL: int32_t inc_dec_expression() {
+func.func @inc_dec_expression() -> i32 {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %inc = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_increment %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = [[V1]]++;
+  %dec = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_decrement %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]--;
+  %r = emitc.add %inc, %dec : (i32, i32) -> i32
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = [[V2]] + [[V3]];
+  // CHECK-NEXT: return [[V4]];
+  return %r : i32
+}
+
 // CHECK: void array_type(int32_t v1[3], float v2[10][20])
 func.func @array_type(%arg0: !emitc.array<3xi32>, %arg1: !emitc.array<10x20xf32>) {
   return

``````````

</details>


https://github.com/llvm/llvm-project/pull/208648

From llvmlistbot at llvm.org  Fri Jul 10 01:29:09 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Fri, 10 Jul 2026 01:29:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Improve memref
 test-memref-dependence-check print format (PR #208656)
Message-ID: <llvm.org/llvm/llvm-project/pull/208656@github.com>


https://github.com/linuxlonelyeagle created https://github.com/llvm/llvm-project/pull/208656

The patch improve `memref test-memref-dependence-check` print format, add print dependence type and shorten distance vectors from.

>From 753532e9ec172cb449087e27d45771f22e2ee7bb Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Fri, 10 Jul 2026 08:26:44 +0000
Subject: [PATCH] improve memref test-memref-dependence-check print format.

---
 .../Affine/memref-dependence-check.mlir       | 50 +++++++++----------
 .../Analysis/TestMemRefDependenceCheck.cpp    | 43 +++++++++++++---
 2 files changed, 61 insertions(+), 32 deletions(-)

diff --git a/mlir/test/Dialect/Affine/memref-dependence-check.mlir b/mlir/test/Dialect/Affine/memref-dependence-check.mlir
index 0a96d1ac65e0a..f4b2b16249f0d 100644
--- a/mlir/test/Dialect/Affine/memref-dependence-check.mlir
+++ b/mlir/test/Dialect/Affine/memref-dependence-check.mlir
@@ -257,13 +257,13 @@ func.func @store_load_func_symbol(%arg0: index, %arg1: index) {
   affine.for %i0 = 0 to %arg1 {
     %a0 = affine.apply affine_map<(d0) -> (d0)> (%arg0)
     affine.store %c7, %m[%a0] : memref<100xf32>
-    // expected-remark at above {{dependence from 0 to 0 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 0 to 0 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = true}}
     %a1 = affine.apply affine_map<(d0) -> (d0)> (%arg0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -291,7 +291,7 @@ func.func @store_range_load_last_in_range() {
     // For dependence from 1 to 0, we have write-after-read (WAR) dependences
     // for all loads in the loop to the store on the last iteration.
     %v0 = affine.load %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -336,7 +336,7 @@ func.func @store_range_load_first_in_range() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = true}}
     %a1 = affine.apply affine_map<(d0) -> (d0 + 1)> (%c0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -358,7 +358,7 @@ func.func @store_plus_3() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [3, 3]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (3)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> (d0)> (%i0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -380,7 +380,7 @@ func.func @load_minus_2() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> (d0 - 2)> (%i0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -437,7 +437,7 @@ func.func @perfectly_nested_loops_loop_carried_at_depth1() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2][0, 0]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2, 0)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a10 = affine.apply affine_map<(d0, d1) -> (d0 - 2)> (%i0, %i1)
@@ -469,7 +469,7 @@ func.func @perfectly_nested_loops_loop_carried_at_depth2() {
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
       // expected-remark at above {{dependence from 0 to 1 at depth 1 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 2 = [0, 0][3, 3]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 2 = (0, 3)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a10 = affine.apply affine_map<(d0, d1) -> (d0)> (%i0, %i1)
       %a11 = affine.apply affine_map<(d0, d1) -> (d1 - 3)> (%i0, %i1)
@@ -558,7 +558,7 @@ func.func @dependence_cycle() {
     // expected-remark at above {{dependence from 2 to 3 at depth 2 = false}}
     %a3 = affine.apply affine_map<(d0) -> (d0 + 1)> (%i0)
     affine.store %v1, %m.a[%a3] : memref<100xf32>
-    // expected-remark at above {{dependence from 3 to 0 at depth 1 = [1, 1]}}
+    // expected-remark at above {{dependence from 3 to 0 at depth 1 = (1)}}
     // expected-remark at above {{dependence from 3 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 3 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 3 to 1 at depth 2 = false}}
@@ -589,7 +589,7 @@ func.func @negative_and_positive_direction_vectors(%arg0: index, %arg1: index) {
       %a10 = affine.apply affine_map<(d0, d1) -> (d0)> (%i0, %i1)
       %a11 = affine.apply affine_map<(d0, d1) -> (d1)> (%i0, %i1)
       affine.store %c7, %m[%a10, %a11] : memref<10x10xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 1][-1, -1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = (1, -1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 0 at depth 3 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
@@ -612,15 +612,15 @@ func.func @war_raw_waw_deps() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9][1, 1]}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 2 = [0, 0][1, 1]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9], 1)}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 2 = (0, 1)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a1 = affine.apply affine_map<(d0) -> (d0)> (%i1)
       affine.store %c7, %m[%a1] : memref<100xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9][-1, -1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9], -1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 1 to 1 at depth 1 = [1, 9][0, 0]}}
+      // expected-remark at above {{dependence from 1 to 1 at depth 1 = ([1, 9], 0)}}
       // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 3 = false}}
     }
@@ -640,13 +640,13 @@ func.func @mod_deps() {
     %v0 = affine.load %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> ( (d0 + 1) mod 2)> (%i0)
     affine.store %c7, %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 1 to 1 at depth 1 = [2, 9]}}
+    // expected-remark at above {{dependence from 1 to 1 at depth 1 = ([2, 9])}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
   }
   return
@@ -700,9 +700,9 @@ func.func @mod_div_3d() {
         %idx1 = affine.apply affine_map<(d0, d1, d2) -> (d1 mod 2)> (%i0, %i1, %i2)
         %idx2 = affine.apply affine_map<(d0, d1, d2) -> (d2 floordiv 4)> (%i0, %i1, %i2)
         affine.store %c0, %M[%idx0, %idx1, %idx2] : memref<2 x 2 x 2 x i32>
-        // expected-remark at above {{dependence from 0 to 0 at depth 1 = [1, 3][-7, 7][-3, 3]}}
-        // expected-remark at above {{dependence from 0 to 0 at depth 2 = [0, 0][2, 7][-3, 3]}}
-        // expected-remark at above {{dependence from 0 to 0 at depth 3 = [0, 0][0, 0][1, 3]}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 1 = ([1, 3], [-7, 7], [-3, 3])}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 2 = (0, [2, 7], [-3, 3])}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 3 = (0, 0, [1, 3])}}
         // expected-remark at above {{dependence from 0 to 0 at depth 4 = false}}
       }
     }
@@ -840,7 +840,7 @@ func.func @strided_loop_with_loop_carried_dependence_at_depth1() {
     affine.store %cf0, %0[%a0] : memref<10xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [4, 4]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (4)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %v0 = affine.load %0[%i0] : memref<10xf32>
     // expected-remark at above {{dependence from 1 to 0 at depth 1 = false}}
@@ -868,7 +868,7 @@ func.func @test_dep_store_depth1_load_depth2() {
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     affine.for %i1 = affine_map<(d0) -> (d0)>(%i0) to affine_map<(d0) -> (d0 + 1)>(%i0) {
       %1 = affine.load %0[%i1] : memref<100xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = (1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -892,7 +892,7 @@ func.func @test_dep_store_depth2_load_depth1() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     }
     %a0 = affine.apply affine_map<(d0) -> (d0 - 2)>(%i0)
@@ -1160,7 +1160,7 @@ func.func @affine_parallel_min_max_bounds(%arg0: memref<4090x2040xf32>, %arg1: f
       affine.store %arg1, %arg0[%arg2 + 3968, %arg3] : memref<4090x2040xf32>
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-      // expected-remark at above {{dependence from 0 to 0 at depth 3 = [0, 0][0, 0][1, 99]}}
+      // expected-remark at above {{dependence from 0 to 0 at depth 3 = (0, 0, [1, 99])}}
       // expected-remark at above {{dependence from 0 to 0 at depth 4 = false}}
     }
   }
diff --git a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
index b3b9a590773b0..91b7f8a4c1d53 100644
--- a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
+++ b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
@@ -51,7 +51,9 @@ getDirectionVectorStr(bool ret, unsigned numCommonLoops, unsigned loopNestDepth,
   if (dependenceComponents.empty() || loopNestDepth > numCommonLoops)
     return "true";
   std::string result;
-  for (const auto &dependenceComponent : dependenceComponents) {
+  result += "(";
+  for (size_t i = 0, e = dependenceComponents.size(); i < e; ++i) {
+    const auto &dependenceComponent = dependenceComponents[i];
     std::string lbStr = "-inf";
     if (dependenceComponent.lb.has_value() &&
         *dependenceComponent.lb != std::numeric_limits<int64_t>::min())
@@ -62,11 +64,40 @@ getDirectionVectorStr(bool ret, unsigned numCommonLoops, unsigned loopNestDepth,
         *dependenceComponent.ub != std::numeric_limits<int64_t>::max())
       ubStr = std::to_string(*dependenceComponent.ub);
 
-    result += "[" + lbStr + ", " + ubStr + "]";
+    if (lbStr == ubStr)
+      result += lbStr;
+    else
+      result += "[" + lbStr + ", " + ubStr + "]";
+
+    if (i < e - 1)
+      result += ", ";
   }
+  result += ")";
   return result;
 }
 
+static std::string getDependenceType(Operation *srcOp, Operation *dstOp) {
+  std::string depandenceRelation;
+
+  auto getAccessTypeLabel = [](Operation *op) {
+    std::string str;
+    if (isa<AffineLoadOp>(op))
+      str += "R";
+    if (isa<AffineStoreOp>(op))
+      str += "W";
+    return str;
+  };
+
+  // A dependence is a pair of statement instances that expresses that the
+  // second statement instance should be executed after the first instance.
+  depandenceRelation += getAccessTypeLabel(dstOp);
+  depandenceRelation += "A";
+  depandenceRelation += getAccessTypeLabel(srcOp);
+  assert(depandenceRelation.size() == 3 &&
+         "srcOp/desOp must be AffineLoadOp/AffineStoreOp");
+  return depandenceRelation;
+}
+
 // For each access in 'loadsAndStores', runs a dependence check between this
 // "source" access and all subsequent "destination" accesses in
 // 'loadsAndStores'. Emits the result of the dependence check as a note with
@@ -90,11 +121,9 @@ static void checkDependences(ArrayRef<Operation *> loadsAndStores) {
           srcOpInst->emitError("dependence check failed");
         } else {
           bool ret = hasDependence(result);
-          // TODO: Print dependence type (i.e. RAW, etc) and print
-          // distance vectors as: ([2, 3], [0, 10]). Also, shorten distance
-          // vectors from ([1, 1], [3, 3]) to (1, 3).
-          srcOpInst->emitRemark("dependence from ")
-              << i << " to " << j << " at depth " << d << " = "
+          srcOpInst->emitRemark(getDependenceType(srcOpInst, dstOpInst))
+              << " dependence from " << i << " to " << j << " at depth " << d
+              << " = "
               << getDirectionVectorStr(ret, numCommonLoops, d,
                                        dependenceComponents);
         }


From llvmlistbot at llvm.org  Fri Jul 10 01:29:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 01:29:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Improve memref
 test-memref-dependence-check print format (PR #208656)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208656@github.com>
Message-ID: <6a50ad7b.4987c784.63cbc.1a9a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-affine

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

The patch improve `memref test-memref-dependence-check` print format, add print dependence type and shorten distance vectors from.

---
Full diff: https://github.com/llvm/llvm-project/pull/208656.diff


2 Files Affected:

- (modified) mlir/test/Dialect/Affine/memref-dependence-check.mlir (+25-25) 
- (modified) mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp (+36-7) 


``````````diff
diff --git a/mlir/test/Dialect/Affine/memref-dependence-check.mlir b/mlir/test/Dialect/Affine/memref-dependence-check.mlir
index 0a96d1ac65e0a..f4b2b16249f0d 100644
--- a/mlir/test/Dialect/Affine/memref-dependence-check.mlir
+++ b/mlir/test/Dialect/Affine/memref-dependence-check.mlir
@@ -257,13 +257,13 @@ func.func @store_load_func_symbol(%arg0: index, %arg1: index) {
   affine.for %i0 = 0 to %arg1 {
     %a0 = affine.apply affine_map<(d0) -> (d0)> (%arg0)
     affine.store %c7, %m[%a0] : memref<100xf32>
-    // expected-remark at above {{dependence from 0 to 0 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 0 to 0 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = true}}
     %a1 = affine.apply affine_map<(d0) -> (d0)> (%arg0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, +inf]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, +inf])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -291,7 +291,7 @@ func.func @store_range_load_last_in_range() {
     // For dependence from 1 to 0, we have write-after-read (WAR) dependences
     // for all loads in the loop to the store on the last iteration.
     %v0 = affine.load %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -336,7 +336,7 @@ func.func @store_range_load_first_in_range() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = true}}
     %a1 = affine.apply affine_map<(d0) -> (d0 + 1)> (%c0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -358,7 +358,7 @@ func.func @store_plus_3() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [3, 3]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (3)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> (d0)> (%i0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -380,7 +380,7 @@ func.func @load_minus_2() {
     affine.store %c7, %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> (d0 - 2)> (%i0)
     %v0 = affine.load %m[%a1] : memref<100xf32>
@@ -437,7 +437,7 @@ func.func @perfectly_nested_loops_loop_carried_at_depth1() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2][0, 0]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2, 0)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a10 = affine.apply affine_map<(d0, d1) -> (d0 - 2)> (%i0, %i1)
@@ -469,7 +469,7 @@ func.func @perfectly_nested_loops_loop_carried_at_depth2() {
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
       // expected-remark at above {{dependence from 0 to 1 at depth 1 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 2 = [0, 0][3, 3]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 2 = (0, 3)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a10 = affine.apply affine_map<(d0, d1) -> (d0)> (%i0, %i1)
       %a11 = affine.apply affine_map<(d0, d1) -> (d1 - 3)> (%i0, %i1)
@@ -558,7 +558,7 @@ func.func @dependence_cycle() {
     // expected-remark at above {{dependence from 2 to 3 at depth 2 = false}}
     %a3 = affine.apply affine_map<(d0) -> (d0 + 1)> (%i0)
     affine.store %v1, %m.a[%a3] : memref<100xf32>
-    // expected-remark at above {{dependence from 3 to 0 at depth 1 = [1, 1]}}
+    // expected-remark at above {{dependence from 3 to 0 at depth 1 = (1)}}
     // expected-remark at above {{dependence from 3 to 0 at depth 2 = false}}
     // expected-remark at above {{dependence from 3 to 1 at depth 1 = false}}
     // expected-remark at above {{dependence from 3 to 1 at depth 2 = false}}
@@ -589,7 +589,7 @@ func.func @negative_and_positive_direction_vectors(%arg0: index, %arg1: index) {
       %a10 = affine.apply affine_map<(d0, d1) -> (d0)> (%i0, %i1)
       %a11 = affine.apply affine_map<(d0, d1) -> (d1)> (%i0, %i1)
       affine.store %c7, %m[%a10, %a11] : memref<10x10xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 1][-1, -1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = (1, -1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 0 at depth 3 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
@@ -612,15 +612,15 @@ func.func @war_raw_waw_deps() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9][1, 1]}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 2 = [0, 0][1, 1]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9], 1)}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 2 = (0, 1)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 3 = false}}
       %a1 = affine.apply affine_map<(d0) -> (d0)> (%i1)
       affine.store %c7, %m[%a1] : memref<100xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9][-1, -1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9], -1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 1 to 1 at depth 1 = [1, 9][0, 0]}}
+      // expected-remark at above {{dependence from 1 to 1 at depth 1 = ([1, 9], 0)}}
       // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 3 = false}}
     }
@@ -640,13 +640,13 @@ func.func @mod_deps() {
     %v0 = affine.load %m[%a0] : memref<100xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %a1 = affine.apply affine_map<(d0) -> ( (d0 + 1) mod 2)> (%i0)
     affine.store %c7, %m[%a1] : memref<100xf32>
-    // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 9]}}
+    // expected-remark at above {{dependence from 1 to 0 at depth 1 = ([1, 9])}}
     // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 1 to 1 at depth 1 = [2, 9]}}
+    // expected-remark at above {{dependence from 1 to 1 at depth 1 = ([2, 9])}}
     // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
   }
   return
@@ -700,9 +700,9 @@ func.func @mod_div_3d() {
         %idx1 = affine.apply affine_map<(d0, d1, d2) -> (d1 mod 2)> (%i0, %i1, %i2)
         %idx2 = affine.apply affine_map<(d0, d1, d2) -> (d2 floordiv 4)> (%i0, %i1, %i2)
         affine.store %c0, %M[%idx0, %idx1, %idx2] : memref<2 x 2 x 2 x i32>
-        // expected-remark at above {{dependence from 0 to 0 at depth 1 = [1, 3][-7, 7][-3, 3]}}
-        // expected-remark at above {{dependence from 0 to 0 at depth 2 = [0, 0][2, 7][-3, 3]}}
-        // expected-remark at above {{dependence from 0 to 0 at depth 3 = [0, 0][0, 0][1, 3]}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 1 = ([1, 3], [-7, 7], [-3, 3])}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 2 = (0, [2, 7], [-3, 3])}}
+        // expected-remark at above {{dependence from 0 to 0 at depth 3 = (0, 0, [1, 3])}}
         // expected-remark at above {{dependence from 0 to 0 at depth 4 = false}}
       }
     }
@@ -840,7 +840,7 @@ func.func @strided_loop_with_loop_carried_dependence_at_depth1() {
     affine.store %cf0, %0[%a0] : memref<10xf32>
     // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
     // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-    // expected-remark at above {{dependence from 0 to 1 at depth 1 = [4, 4]}}
+    // expected-remark at above {{dependence from 0 to 1 at depth 1 = (4)}}
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     %v0 = affine.load %0[%i0] : memref<10xf32>
     // expected-remark at above {{dependence from 1 to 0 at depth 1 = false}}
@@ -868,7 +868,7 @@ func.func @test_dep_store_depth1_load_depth2() {
     // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     affine.for %i1 = affine_map<(d0) -> (d0)>(%i0) to affine_map<(d0) -> (d0 + 1)>(%i0) {
       %1 = affine.load %0[%i1] : memref<100xf32>
-      // expected-remark at above {{dependence from 1 to 0 at depth 1 = [1, 1]}}
+      // expected-remark at above {{dependence from 1 to 0 at depth 1 = (1)}}
       // expected-remark at above {{dependence from 1 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 1 = false}}
       // expected-remark at above {{dependence from 1 to 1 at depth 2 = false}}
@@ -892,7 +892,7 @@ func.func @test_dep_store_depth2_load_depth1() {
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 3 = false}}
-      // expected-remark at above {{dependence from 0 to 1 at depth 1 = [2, 2]}}
+      // expected-remark at above {{dependence from 0 to 1 at depth 1 = (2)}}
       // expected-remark at above {{dependence from 0 to 1 at depth 2 = false}}
     }
     %a0 = affine.apply affine_map<(d0) -> (d0 - 2)>(%i0)
@@ -1160,7 +1160,7 @@ func.func @affine_parallel_min_max_bounds(%arg0: memref<4090x2040xf32>, %arg1: f
       affine.store %arg1, %arg0[%arg2 + 3968, %arg3] : memref<4090x2040xf32>
       // expected-remark at above {{dependence from 0 to 0 at depth 1 = false}}
       // expected-remark at above {{dependence from 0 to 0 at depth 2 = false}}
-      // expected-remark at above {{dependence from 0 to 0 at depth 3 = [0, 0][0, 0][1, 99]}}
+      // expected-remark at above {{dependence from 0 to 0 at depth 3 = (0, 0, [1, 99])}}
       // expected-remark at above {{dependence from 0 to 0 at depth 4 = false}}
     }
   }
diff --git a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
index b3b9a590773b0..91b7f8a4c1d53 100644
--- a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
+++ b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
@@ -51,7 +51,9 @@ getDirectionVectorStr(bool ret, unsigned numCommonLoops, unsigned loopNestDepth,
   if (dependenceComponents.empty() || loopNestDepth > numCommonLoops)
     return "true";
   std::string result;
-  for (const auto &dependenceComponent : dependenceComponents) {
+  result += "(";
+  for (size_t i = 0, e = dependenceComponents.size(); i < e; ++i) {
+    const auto &dependenceComponent = dependenceComponents[i];
     std::string lbStr = "-inf";
     if (dependenceComponent.lb.has_value() &&
         *dependenceComponent.lb != std::numeric_limits<int64_t>::min())
@@ -62,11 +64,40 @@ getDirectionVectorStr(bool ret, unsigned numCommonLoops, unsigned loopNestDepth,
         *dependenceComponent.ub != std::numeric_limits<int64_t>::max())
       ubStr = std::to_string(*dependenceComponent.ub);
 
-    result += "[" + lbStr + ", " + ubStr + "]";
+    if (lbStr == ubStr)
+      result += lbStr;
+    else
+      result += "[" + lbStr + ", " + ubStr + "]";
+
+    if (i < e - 1)
+      result += ", ";
   }
+  result += ")";
   return result;
 }
 
+static std::string getDependenceType(Operation *srcOp, Operation *dstOp) {
+  std::string depandenceRelation;
+
+  auto getAccessTypeLabel = [](Operation *op) {
+    std::string str;
+    if (isa<AffineLoadOp>(op))
+      str += "R";
+    if (isa<AffineStoreOp>(op))
+      str += "W";
+    return str;
+  };
+
+  // A dependence is a pair of statement instances that expresses that the
+  // second statement instance should be executed after the first instance.
+  depandenceRelation += getAccessTypeLabel(dstOp);
+  depandenceRelation += "A";
+  depandenceRelation += getAccessTypeLabel(srcOp);
+  assert(depandenceRelation.size() == 3 &&
+         "srcOp/desOp must be AffineLoadOp/AffineStoreOp");
+  return depandenceRelation;
+}
+
 // For each access in 'loadsAndStores', runs a dependence check between this
 // "source" access and all subsequent "destination" accesses in
 // 'loadsAndStores'. Emits the result of the dependence check as a note with
@@ -90,11 +121,9 @@ static void checkDependences(ArrayRef<Operation *> loadsAndStores) {
           srcOpInst->emitError("dependence check failed");
         } else {
           bool ret = hasDependence(result);
-          // TODO: Print dependence type (i.e. RAW, etc) and print
-          // distance vectors as: ([2, 3], [0, 10]). Also, shorten distance
-          // vectors from ([1, 1], [3, 3]) to (1, 3).
-          srcOpInst->emitRemark("dependence from ")
-              << i << " to " << j << " at depth " << d << " = "
+          srcOpInst->emitRemark(getDependenceType(srcOpInst, dstOpInst))
+              << " dependence from " << i << " to " << j << " at depth " << d
+              << " = "
               << getDirectionVectorStr(ret, numCommonLoops, d,
                                        dependenceComponents);
         }

``````````

</details>


https://github.com/llvm/llvm-project/pull/208656

From llvmlistbot at llvm.org  Fri Jul 10 01:59:28 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 01:59:28 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b470.481afadf.196d57.d9c4@mx.google.com>


https://github.com/iliyan-georgiev-arm edited https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 01:59:28 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 01:59:28 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b470.8e75e0cb.5e108.ad01@mx.google.com>


================
@@ -1348,6 +1350,22 @@ extensionComplianceMap = {
        {{{i8T}, SpecificationVersion::V_1_0},
         {{fp16T}, SpecificationVersion::V_1_0},
         {{fp32T}, SpecificationVersion::V_1_0}}}}},
+    {"tosa.add_shape", {{{Extension::shape}, {}}}},
----------------
iliyan-georgiev-arm wrote:

I find it odd that we don't have spec version against this. Especially since shape ops are not part of the 1.0.x specification.

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 01:59:29 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 01:59:29 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b471.30759430.24d50c.435c@mx.google.com>


================
@@ -508,22 +484,18 @@ LogicalResult TosaProfileCompliance::checkProfileOrExtension(
   const TosaSpecificationVersion targetVersion{targetEnv.getSpecVersion()};
   const auto isVersionCompatible =
       [&targetVersion](const OpComplianceInfo<T> &info) -> bool {
+    // Assume compatible if there are no versioned type constraints
+    if (info.operandTypeInfoSet.empty())
----------------
iliyan-georgiev-arm wrote:

As a continuation from the comment on `TosaComplianceData.h.inc`.

Unless I'm misunderstanding this implies that if we were to add an op in the future (similar to shape ops) which does not have type constraints it would be impoliticly compatible with older versions of the specification?

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 01:59:29 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 01:59:29 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b471.f8b79514.1d52d0.2d43@mx.google.com>


https://github.com/iliyan-georgiev-arm commented:

Overall looks good, thanks! 
Just a couple of clarifying comments.

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:01:41 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Fri, 10 Jul 2026 02:01:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support subgroup layout
 assignment for scatter stores (PR #205955)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205955@github.com>
Message-ID: <6a50b4f5.bb839515.27d21e.78f1@mx.google.com>


akroviakov wrote:

I see the confusion. `getSgLayoutCandidates` exists in `XeGPULayoutImpl.cpp` and `XeGPUPropagateLayout.cpp`. The propagate version is 2D and appears to be dead code.

https://github.com/llvm/llvm-project/pull/205955

From llvmlistbot at llvm.org  Fri Jul 10 02:04:39 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 02:04:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b5a7.4987c784.63cbc.a9a6@mx.google.com>


================
@@ -1348,6 +1350,22 @@ extensionComplianceMap = {
        {{{i8T}, SpecificationVersion::V_1_0},
         {{fp16T}, SpecificationVersion::V_1_0},
         {{fp32T}, SpecificationVersion::V_1_0}}}}},
+    {"tosa.add_shape", {{{Extension::shape}, {}}}},
----------------
lhutton1 wrote:

Versions specified here are per data type combination. Use of shape ops in v1.0 is disallowed since there are version checks on when an extension is introduced which must be loaded in the target_env to allow these shape ops to be used.

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:04:39 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 02:04:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b5a7.1905e68e.18ad68.97b5@mx.google.com>


https://github.com/lhutton1 edited https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:04:40 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 02:04:40 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b5a8.331483e7.2440f8.84a8@mx.google.com>


https://github.com/lhutton1 commented:

Thanks for the review, these are good questions!

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:04:40 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 02:04:40 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b5a8.13be6df8.145c.3724@mx.google.com>


================
@@ -508,22 +484,18 @@ LogicalResult TosaProfileCompliance::checkProfileOrExtension(
   const TosaSpecificationVersion targetVersion{targetEnv.getSpecVersion()};
   const auto isVersionCompatible =
       [&targetVersion](const OpComplianceInfo<T> &info) -> bool {
+    // Assume compatible if there are no versioned type constraints
+    if (info.operandTypeInfoSet.empty())
----------------
lhutton1 wrote:

Typically when an op is added, it either has data type constraints, or it is added along with a new extension, both of which are checked against the specification version. 

You're correct though, there is a gap here if a new operator is added and it doesn't have type constraints and the profile/extension it is added to isn't new. We've not encountered this scenario yet, but we should be able to do so by extending the profile conformance table with versioning per operator when we encounter it.

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:12:10 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 02:12:10 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b76a.e55dc341.1f19bd.f150@mx.google.com>


================
@@ -1348,6 +1350,22 @@ extensionComplianceMap = {
        {{{i8T}, SpecificationVersion::V_1_0},
         {{fp16T}, SpecificationVersion::V_1_0},
         {{fp32T}, SpecificationVersion::V_1_0}}}}},
+    {"tosa.add_shape", {{{Extension::shape}, {}}}},
----------------
iliyan-georgiev-arm wrote:

Makes sense, thanks! 

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:12:46 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 10 Jul 2026 02:12:46 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][tosa] Remove availability
 interfaces from profile conformance (PR #207809)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207809@github.com>
Message-ID: <6a50b78e.e55dc341.1f19bd.2efc@mx.google.com>


================
@@ -508,22 +484,18 @@ LogicalResult TosaProfileCompliance::checkProfileOrExtension(
   const TosaSpecificationVersion targetVersion{targetEnv.getSpecVersion()};
   const auto isVersionCompatible =
       [&targetVersion](const OpComplianceInfo<T> &info) -> bool {
+    // Assume compatible if there are no versioned type constraints
+    if (info.operandTypeInfoSet.empty())
----------------
iliyan-georgiev-arm wrote:

Sounds good, I guess we'll keep an eye out for such cases. I do agree they should be rare.

https://github.com/llvm/llvm-project/pull/207809

From llvmlistbot at llvm.org  Fri Jul 10 02:18:50 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 10 Jul 2026 02:18:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a50b8fa.bd93e731.358318.cb61@mx.google.com>


https://github.com/IanTaylerLessa-arm edited https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Fri Jul 10 02:18:50 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 10 Jul 2026 02:18:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a50b8fa.331483e7.2440f8.c351@mx.google.com>


https://github.com/IanTaylerLessa-arm commented:

Looks good to me otherwise. Naming suggestion is entirely optional.

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Fri Jul 10 02:18:50 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 10 Jul 2026 02:18:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a50b8fa.9edb5c9d.208698.97a2@mx.google.com>


================
@@ -286,6 +286,15 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
   return %0 : tensor<2x14x28xf16>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_block_scaled
+func.func @test_matmul_t_block_scaled(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
----------------
IanTaylerLessa-arm wrote:

Maybe we should call it `test_matmul_t_with_block_scaled_inputs` to differentiate from `matmul_t_block_scaled`.

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Fri Jul 10 02:18:51 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Fri, 10 Jul 2026 02:18:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a50b8fb.4987c784.63cbc.6226@mx.google.com>


================
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
----------------
IanTaylerLessa-arm wrote:

We're missing `fp16 x fp32 -> fp32` and `fp32 x fp16 -> fp32`, is that intended to go on a different MR?

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Fri Jul 10 02:46:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 02:46:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4f3000e - [mlir][SPIR-V] Add
 spirv.GL.NMax/NMin to SPIRVtoLLVM conversion (#205976)
Message-ID: <6a50bf7e.f06f4994.1fc857.014b@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-10T11:46:33+02:00
New Revision: 4f3000e8d6d07a1fa9d3cdfde34a270bd0a2decf

URL: https://github.com/llvm/llvm-project/commit/4f3000e8d6d07a1fa9d3cdfde34a270bd0a2decf
DIFF: https://github.com/llvm/llvm-project/commit/4f3000e8d6d07a1fa9d3cdfde34a270bd0a2decf.diff

LOG: [mlir][SPIR-V] Add spirv.GL.NMax/NMin to SPIRVtoLLVM conversion (#205976)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 8b113478be71d..fdde3352c040b 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -2035,6 +2035,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       ClampPattern<spirv::GLUClampOp, LLVM::UMinOp, LLVM::UMaxOp>,
       DirectConversionPattern<spirv::GLFMaxOp, LLVM::MaxNumOp>,
       DirectConversionPattern<spirv::GLFMinOp, LLVM::MinNumOp>,
+      DirectConversionPattern<spirv::GLNMaxOp, LLVM::MaxNumOp>,
+      DirectConversionPattern<spirv::GLNMinOp, LLVM::MinNumOp>,
       DirectConversionPattern<spirv::GLLogOp, LLVM::LogOp>,
       DirectConversionPattern<spirv::GLLog2Op, LLVM::Log2Op>,
       DirectConversionPattern<spirv::GLPowOp, LLVM::PowOp>,

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index 7be0bf201175f..08c7947861cb4 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -103,6 +103,32 @@ spirv.func @fclamp(%arg0: f32, %arg1: f32, %arg2: f32) "None" {
   spirv.Return
 }
 
+//===----------------------------------------------------------------------===//
+// spirv.GL.NMax
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @nmax
+spirv.func @nmax(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: llvm.intr.maxnum(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
+  %0 = spirv.GL.NMax %arg0, %arg0 : f32
+  // CHECK: llvm.intr.maxnum(%{{.*}}, %{{.*}}) : (vector<3xf16>, vector<3xf16>) -> vector<3xf16>
+  %1 = spirv.GL.NMax %arg1, %arg1 : vector<3xf16>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.NMin
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @nmin
+spirv.func @nmin(%arg0: f32, %arg1: vector<3xf16>) "None" {
+  // CHECK: llvm.intr.minnum(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
+  %0 = spirv.GL.NMin %arg0, %arg0 : f32
+  // CHECK: llvm.intr.minnum(%{{.*}}, %{{.*}}) : (vector<3xf16>, vector<3xf16>) -> vector<3xf16>
+  %1 = spirv.GL.NMin %arg1, %arg1 : vector<3xf16>
+  spirv.Return
+}
+
 //===----------------------------------------------------------------------===//
 // spirv.GL.Log
 //===----------------------------------------------------------------------===//


        

From llvmlistbot at llvm.org  Fri Jul 10 02:46:41 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 10 Jul 2026 02:46:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add spirv.GL.NMax/NMin to
 SPIRVtoLLVM conversion (PR #205976)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205976@github.com>
Message-ID: <6a50bf81.f732ad81.3a4994.27cd@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/205976

From llvmlistbot at llvm.org  Fri Jul 10 02:51:44 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Fri, 10 Jul 2026 02:51:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for CL copysign and fmod ops (PR #205993)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205993@github.com>
Message-ID: <6a50c0b0.6c5720b3.128ed.bbcb@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/205993

From llvmlistbot at llvm.org  Fri Jul 10 02:52:52 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 02:52:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 50143b8 - [mlir][SPIR-V] Add SPIRVToLLVM
 conversions for CL copysign and fmod ops (#205993)
Message-ID: <6a50c0f4.a87eb5dc.7d817.794b@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-10T11:52:48+02:00
New Revision: 50143b877dffdb9016e50fb541806eb59c261b10

URL: https://github.com/llvm/llvm-project/commit/50143b877dffdb9016e50fb541806eb59c261b10
DIFF: https://github.com/llvm/llvm-project/commit/50143b877dffdb9016e50fb541806eb59c261b10.diff

LOG: [mlir][SPIR-V] Add SPIRVToLLVM conversions for CL copysign and fmod ops (#205993)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index fdde3352c040b..0822c44c10f1d 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -2085,6 +2085,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::CLAtan2Op, LLVM::ATan2Op>,
       DirectConversionPattern<spirv::CLSqrtOp, LLVM::SqrtOp>,
       DirectConversionPattern<spirv::CLTruncOp, LLVM::FTruncOp>,
+      DirectConversionPattern<spirv::CLCopysignOp, LLVM::CopySignOp>,
+      DirectConversionPattern<spirv::CLFmodOp, LLVM::FRemOp>,
       DirectConversionPattern<spirv::CLSMaxOp, LLVM::SMaxOp>,
       DirectConversionPattern<spirv::CLSMinOp, LLVM::SMinOp>,
       DirectConversionPattern<spirv::CLUMaxOp, LLVM::UMaxOp>,

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
index a986be42a5437..e6fed930268d4 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/cl-ops-to-llvm.mlir
@@ -69,6 +69,10 @@ spirv.func @cl_binary_float(%arg0: f32, %arg1: f32) "None" {
   %2 = spirv.CL.fmax %arg0, %arg1 : f32
   // CHECK: llvm.intr.minnum(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
   %3 = spirv.CL.fmin %arg0, %arg1 : f32
+  // CHECK: llvm.intr.copysign(%{{.*}}, %{{.*}}) : (f32, f32) -> f32
+  %4 = spirv.CL.copysign %arg0, %arg1 : f32
+  // CHECK: llvm.frem %{{.*}}, %{{.*}} : f32
+  %5 = spirv.CL.fmod %arg0, %arg1 : f32
   spirv.Return
 }
 


        

From llvmlistbot at llvm.org  Fri Jul 10 02:52:55 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 10 Jul 2026 02:52:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for CL copysign and fmod ops (PR #205993)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205993@github.com>
Message-ID: <6a50c0f7.824e11a6.f5f27.faf1@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/205993

From llvmlistbot at llvm.org  Fri Jul 10 03:05:59 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Fri, 10 Jul 2026 03:05:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a50c407.9edb5c9d.208698.0668@mx.google.com>


================
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
----------------
IgWod wrote:

```suggestion
    Type srcType = op.getType();
    Type dstType = getTypeConverter()->convertType(srcType);
```

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Fri Jul 10 03:05:59 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Fri, 10 Jul 2026 03:05:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a50c407.1905e68e.18ad68.91ad@mx.google.com>


================
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
----------------
IgWod wrote:

Would it make sense to generalise `createConstantAllBitsSet` and use it here?

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Fri Jul 10 04:29:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 04:29:31 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [mlir] Diagnose missing call result
	arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a50d79b.492e23e4.157d47.df2b@mx.google.com>


================
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
----------------
mygitljf wrote:

I added this to keep the existing fallback diagnostic when parsing fails before consuming any of the signature. In the missing `->` case, the parser has already consumed `()` and emitted a more specific error, so without this check we'd also emit the generic `expected direct call...` error.
The nearby `call_missing_ptr_type` test covers the no-progress case. I can add a comment here if that would help.

https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Fri Jul 10 04:45:42 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 10 Jul 2026 04:45:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a50db66.481afadf.196d57.17ec@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206950

>From 187cd538f98c32b897f1d8839a38a3b6915917a1 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:12:30 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Add SPIRVToLLVM conversion for SNegate

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 31 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 20 ++++++++++++
 2 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..610c052b64ea7 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1853,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}

>From 204b7a856af639ca4bb46501848cabdcef02e393 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 10 Jul 2026 13:45:23 +0200
Subject: [PATCH 2/2] Address comments

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 34 +++++++++----------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 96acea0f4f127..a8bee01b08d26 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -90,17 +90,22 @@ static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
   return builder.getIntegerAttr(integerType, -1);
 }
 
+/// Creates `llvm.mlir.constant` with a scalar or vector integer value,
+/// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
+static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
+                                   PatternRewriter &rewriter,
+                                   IntegerAttr scalarAttr) {
+  if (auto vecType = dyn_cast<VectorType>(srcType))
+    return LLVM::ConstantOp::create(
+        rewriter, loc, dstType, SplatElementsAttr::get(vecType, scalarAttr));
+  return LLVM::ConstantOp::create(rewriter, loc, dstType, scalarAttr);
+}
+
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
-  if (isa<VectorType>(srcType)) {
-    return LLVM::ConstantOp::create(
-        rewriter, loc, dstType,
-        SplatElementsAttr::get(cast<ShapedType>(srcType),
-                               minusOneIntegerAttribute(srcType, rewriter)));
-  }
-  return LLVM::ConstantOp::create(rewriter, loc, dstType,
-                                  minusOneIntegerAttribute(srcType, rewriter));
+  return createIntegerConstant(loc, srcType, dstType, rewriter,
+                               minusOneIntegerAttribute(srcType, rewriter));
 }
 
 /// Creates `llvm.mlir.constant` with a floating-point scalar or vector value.
@@ -929,21 +934,16 @@ class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
   LogicalResult
   matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto srcType = op.getType();
-    auto dstType = getTypeConverter()->convertType(srcType);
+    Type srcType = op.getType();
+    Type dstType = getTypeConverter()->convertType(srcType);
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
     Location loc = op.getLoc();
-    auto vecSrcType = dyn_cast<VectorType>(srcType);
     IntegerAttr zeroAttr = rewriter.getIntegerAttr(
         cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
-    Value zero;
-    if (vecSrcType)
-      zero = LLVM::ConstantOp::create(
-          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
-    else
-      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    Value zero =
+        createIntegerConstant(loc, srcType, dstType, rewriter, zeroAttr);
     rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
                                              adaptor.getOperand());
     return success();


From llvmlistbot at llvm.org  Fri Jul 10 04:45:51 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 10 Jul 2026 04:45:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a50db6f.da8bafd3.14cd50.fd78@mx.google.com>


================
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
----------------
aobolensk wrote:

Done

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Fri Jul 10 04:45:56 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Fri, 10 Jul 2026 04:45:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a50db74.d1496011.26913c.5bd8@mx.google.com>


================
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
----------------
aobolensk wrote:

Yes, done

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Fri Jul 10 04:55:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 04:55:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a50ddaa.331483e7.2440f8.9e49@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/205873

>From 62993eafa5d487b3a786867622e5a52a37da217e Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 26 Jun 2026 01:54:59 +0000
Subject: [PATCH 1/2] [mlir][tosa] fallback unknown type

---
 .../mlir/Dialect/Tosa/IR/TosaProfileCompliance.h   |  2 +-
 .../Tosa/Transforms/TosaProfileCompliance.cpp      |  4 ++--
 .../Tosa/tosa-validation-version-1p1-invalid.mlir  | 14 ++++++++++++++
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 0135a651be481..7c185b3a9a1f2 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -145,7 +145,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<16> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 0b4983bea4bc6..4b61782a468ec 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -687,7 +687,7 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<7>
+llvm::SmallString<16>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
     return {"i" + llvm::utostr(typeInfo.bitWidth)};
@@ -713,5 +713,5 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
     return {"mxint8"};
   }
-  llvm_unreachable("unknown type");
+  return {"unknown<" + llvm::utostr(typeInfo.bitWidth) + ">"};
 }
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 5ce54980c8d61..7e341254e8ece 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -25,3 +25,17 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
        (tensor<1x32x32x8xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  tosa.variable @dynamic_var : tensor<2x?xf64>
+}
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  tosa.variable @static_var : tensor<2x4xf64>
+}

>From 8362bde4d8143ba6062e45f818420aa6981be9ea Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 19:57:52 +0000
Subject: [PATCH 2/2] [mlir][tosa] Validate variable element types

---
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   |  2 +-
 .../Tosa/Transforms/TosaProfileCompliance.cpp |  4 +-
 .../Tosa/Transforms/TosaValidation.cpp        | 54 +++++++++++++------
 .../tosa-validation-version-1p1-invalid.mlir  |  4 +-
 4 files changed, 43 insertions(+), 21 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 7c185b3a9a1f2..0135a651be481 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -145,7 +145,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<16> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 4b61782a468ec..0b4983bea4bc6 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -687,7 +687,7 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<16>
+llvm::SmallString<7>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
     return {"i" + llvm::utostr(typeInfo.bitWidth)};
@@ -713,5 +713,5 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
     return {"mxint8"};
   }
-  return {"unknown<" + llvm::utostr(typeInfo.bitWidth) + ">"};
+  llvm_unreachable("unknown type");
 }
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 5d4a2affbd7cd..d3abb7c2955c7 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -670,6 +670,10 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
   LogicalResult CheckVariable(Operation *op);
   LogicalResult CheckVariableReadOrWrite(Operation *op);
   bool isValidElementType(Type type, const bool allowUnsigned = false);
+  LogicalResult validateElementType(Operation *op, Type elementTy,
+                                    bool allowUnsigned = false);
+  LogicalResult validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned = false);
 
   SmallVector<
       std::function<LogicalResult(Operation *, const tosa::TargetEnv &)>>
@@ -1500,6 +1504,38 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
   return false;
 }
 
+LogicalResult TosaValidation::validateElementType(Operation *op, Type elementTy,
+                                                  bool allowUnsigned) {
+  if (isValidElementType(elementTy, allowUnsigned))
+    return success();
+
+  return op->emitOpError() << "is not profile-aligned: element type "
+                           << elementTy << " is not legal";
+}
+
+LogicalResult
+TosaValidation::validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned) {
+  for (Value operand : op->getOperands()) {
+    Type elementTy = getElementTypeOrSelf(operand);
+    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  for (Type resultTy : op->getResultTypes()) {
+    Type elementTy = getElementTypeOrSelf(resultTy);
+    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  if (auto variableOp = dyn_cast<tosa::VariableOp>(op)) {
+    if (failed(validateElementType(op, variableOp.getType(), allowUnsigned)))
+      return failure();
+  }
+
+  return success();
+}
+
 void TosaValidation::runOnOperation() {
   ModuleOp modOp = getOperation();
   TosaDialect *tosaDialect = getContext().getLoadedDialect<TosaDialect>();
@@ -1530,22 +1566,8 @@ void TosaValidation::runOnOperation() {
     //   protect rest of code against quantized element types
     const bool allowUnsigned =
         !strictOpSpecAlignment && isa<tosa::RescaleOp>(op);
-    for (Value operand : op->getOperands()) {
-      auto elementTy = getElementTypeOrSelf(operand);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
-    for (Type resultTy : op->getResultTypes()) {
-      auto elementTy = getElementTypeOrSelf(resultTy);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
+    if (failed(validateOperationElementTypes(op, allowUnsigned)))
+      return signalPassFailure();
 
     if (strictOpSpecAlignment &&
         failed(profileComp.checkProfile(op, targetEnv)))
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 7e341254e8ece..19ccef0f52a41 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -29,13 +29,13 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
 // -----
 
 module {
-  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
   tosa.variable @dynamic_var : tensor<2x?xf64>
 }
 
 // -----
 
 module {
-  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
   tosa.variable @static_var : tensor<2x4xf64>
 }


From llvmlistbot at llvm.org  Fri Jul 10 04:57:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 04:57:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a50de11.8e75e0cb.5e108.d190@mx.google.com>


mygitljf wrote:

Thanks. I updated the validation path to use a shared element-type check for operands, results, and the declared type on `tosa.variable`, so the invalid `f64` is rejected before profile/extension compliance runs. I also changed the regression test to expect the earlier `element type 'f64' is not legal` diagnostic instead of the `unknown<64>` profile-compliance diagnostic.

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Fri Jul 10 05:49:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 05:49:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50ea5b.bf38ae28.39239a.bdeb@mx.google.com>


mygitljf wrote:

@IgWod , @joker-eph Thanks for your review and time.

I updated this to keep the tensor-specific materialization contained in `arith-expand`.

The previous version was too broad: it made ArithToLLVM and the SPIR-V test pass load/depend on the tensor dialect, and the added conversion tests were mostly showing that those pipelines could preserve dynamic tensor arith ops rather than actually lowering them. That made the boundary confusing.

In the new version, `populateCeilFloorDivExpandOpsPatterns` keeps the old default behavior. Only `populateArithExpandOpsPatterns` opts into dynamic tensor constant materialization, where dynamic ranked tensor constants are created with `tensor.splat` using the source tensor dimensions. 

I also removed the ArithToLLVM tensor dependency, the SPIR-V tensor dependency, and the off-contract tests.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Fri Jul 10 05:54:28 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Fri, 10 Jul 2026 05:54:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50eb84.07d7ca98.9dee3.e425@mx.google.com>


================

----------------
IgWod wrote:

Unrelated change, please revert it.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Fri Jul 10 06:09:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 06:09:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50ef14.510f576d.1b3cb9.a1b7@mx.google.com>


================

----------------
mygitljf wrote:

ok, I'll revert the SPIR-V test change.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Fri Jul 10 06:09:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 06:09:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50ef14.492e23e4.157d47.d1fc@mx.google.com>


================

----------------
mygitljf wrote:

yeah, this should not leak into the LLVM conversion path. I changed that.
The only remaining Tensor dialect use is inside the standalone `arith-expand` pass, where it is needed to materialize dynamic-shaped tensor constants. A dynamic ranked tensor `0`/`1` cannot be represented as a dense `arith.constant`, so this expansion uses `tensor.splat` from a scalar constant and the source tensor dimensions.
So this is not meant to make dynamic tensor arith directly LLVM-convertible by itself; it is an arith expansion that still expects tensor lowering later. The LLVM conversion helper keeps the old default and does not enable this path.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Fri Jul 10 06:24:40 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 10 Jul 2026 06:24:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a50f298.8e75e0cb.5e108.781c@mx.google.com>


Devjiu wrote:

@joker-eph

Before I push further on proposing options -  when you say this is "more inconsistent," what's the inconsistency you have in mind? I want to make sure I'm arguing against the actual concern and not a guess at it. Is it:

- inconsistent across encodings (some get asked, some don't, depending on whether they implement the interface), or
- inconsistent across transforms (some tensor.* ops would fold through an  encoding, others wouldn't), or
- something about relying on `verifyEncoding` for a purpose it wasn't designed for?

If you already have a preferred design in mind, please share - I would be glad to adopt the current design to your suggestion.

My position: I don't think upstream canonicalizers should be guessing what a given encoding means for a given transform - that's the encoding author's call, not ours. `VerifiableTensorEncoding` already exists as the place to make that call explicit (sparse uses it for rank). If a downstream cares about a different invariant, they can express it there too.

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul 10 06:27:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 06:27:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50f355.d681cab9.333b2b.ca8e@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206249

>From 3663396b2155970b6a4982ec42a0108c749df095 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 20:57:43 +0000
Subject: [PATCH 1/3] [mlir] Fix arith tensor constants

---
 mlir/include/mlir/Conversion/Passes.td        |  2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.td   |  2 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |  3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |  1 +
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 29 +++++++--
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    | 10 +++
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  8 +++
 mlir/test/Dialect/Arith/expand-ops.mlir       | 63 +++++++++++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |  2 +
 9 files changed, 111 insertions(+), 9 deletions(-)
 create mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 550e6b853057d..23463342314b0 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
index 27e9146ec3606..2cee05914d6b5 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
@@ -13,7 +13,7 @@ include "mlir/Pass/PassBase.td"
 
 def ArithExpandOpsPass : Pass<"arith-expand"> {
   let summary = "Legalize Arith ops to be convertible to LLVM.";
-  let dependentDialects = ["vector::VectorDialect"];
+  let dependentDialects = ["tensor::TensorDialect", "vector::VectorDialect"];
   let options =
       [Option<"includeBf16", "include-bf16", "bool", /*default=*/"false",
               "Enable the BF16 expansion patterns">,
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..77e36b45a84af 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -736,7 +737,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect>();
+    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 0a0e25e18b47a..955fcc397dd37 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,4 +16,5 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
+  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..d9cfa6bc78dc6 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Arith/Transforms/Passes.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/Location.h"
@@ -34,6 +35,21 @@ static Value createConst(Location loc, Type type, int value,
   return arith::ConstantOp::create(rewriter, loc, attr);
 }
 
+/// Create an integer or index constant, using source as the dynamic shape
+/// source for ranked tensors whose splat cannot be represented as a dense
+/// attribute.
+static Value createConst(Location loc, Type type, int value,
+                         PatternRewriter &rewriter, Value source) {
+  auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
+    return createConst(loc, type, value, rewriter);
+
+  Value scalar =
+      createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
+  return tensor::SplatOp::create(rewriter, loc, scalar,
+                                 tensor::getMixedSizes(rewriter, loc, source));
+}
+
 /// Create an integer constant from an APInt.
 static Value createAPIntConst(Location loc, Type type, const APInt &value,
                               PatternRewriter &rewriter) {
@@ -76,10 +92,10 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter);
+    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
     Value compare =
         arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter);
+    Value one = createConst(loc, a.getType(), 1, rewriter, a);
     Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
     Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
@@ -104,8 +120,8 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter);
-    Value one = createConst(loc, type, 1, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
+    Value one = createConst(loc, type, 1, rewriter, a);
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -150,7 +166,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
                                        a, zero);
@@ -162,7 +178,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter);
+    Value minusOne = createConst(loc, type, -1, rewriter, a);
     Value quotientMinusOne =
         arith::AddIOp::create(rewriter, loc, quotient, minusOne);
 
@@ -826,6 +842,7 @@ struct ArithExpandOpsPass
     arith::populateArithExpandOpsPatterns(patterns);
 
     target.addLegalDialect<arith::ArithDialect>();
+    target.addLegalDialect<tensor::TensorDialect>();
     target.addLegalDialect<vector::VectorDialect>();
 
     // clang-format off
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
new file mode 100644
index 0000000000000..9d169bc60cade
--- /dev/null
+++ b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
+// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
+
+// CHECK-LABEL: @ceildivui_dynamic_tensor
+// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
+func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
+  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
+  return %0: tensor<8x4x?xi64>
+}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index fa204c4df8ace..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,14 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
 // -----
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 75c4de2168761..80b6592417cc2 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -144,6 +144,69 @@ func.func @ceildivui_index(%arg0: index, %arg1: index) -> (index) {
 
 // -----
 
+// CHECK-LABEL:   func.func @ceildivui_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x4x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+func.func @ceildivui_dynamic_tensor(%arg0: tensor<8x4x?xi64>, %arg1: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+  %res = arith.ceildivui %arg0, %arg1 : tensor<8x4x?xi64>
+  return %res : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C2_IDX:.*]] = arith.constant 2 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX]] : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[ISZERO:.*]] = arith.cmpi eq, %[[ARG0]], %[[ZERO]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C2_IDX_1:.*]] = arith.constant 2 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX_1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divui %[[SUB]], %[[ARG1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[RES:.*]] = arith.select %[[ISZERO]], %[[ZERO]], %[[ADD]] : tensor<8x4x?xi1>, tensor<8x4x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @ceildivsi_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x?xi64>) -> tensor<8x?xi64> {
+func.func @ceildivsi_dynamic_tensor(%arg0: tensor<8x?xi64>, %arg1: tensor<8x?xi64>) -> tensor<8x?xi64> {
+  %res = arith.ceildivsi %arg0, %arg1 : tensor<8x?xi64>
+  return %res : tensor<8x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C1_IDX:.*]] = arith.constant 1 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX]] : tensor<8x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C1_IDX_1:.*]] = arith.constant 1 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX_1]] : tensor<8x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<8x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @floordivsi_dynamic_tensor(
+// CHECK-SAME:                                       %[[ARG0:.*]]: tensor<?x4xi64>,
+// CHECK-SAME:                                       %[[ARG1:.*]]: tensor<?x4xi64>) -> tensor<?x4xi64> {
+func.func @floordivsi_dynamic_tensor(%arg0: tensor<?x4xi64>, %arg1: tensor<?x4xi64>) -> tensor<?x4xi64> {
+  %res = arith.floordivsi %arg0, %arg1 : tensor<?x4xi64>
+  return %res : tensor<?x4xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C0_IDX:.*]] = arith.constant 0 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE_SCALAR:.*]] = arith.constant -1 : i64
+// CHECK:       %[[C0_IDX_1:.*]] = arith.constant 0 : index
+// CHECK:       %[[NEG_ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX_1]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE:.*]] = tensor.splat %[[NEG_ONE_SCALAR]]{{\[}}%[[NEG_ONE_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[SUB_ONE:.*]] = arith.addi %[[DIV]], %[[NEG_ONE]] : tensor<?x4xi64>
+}
+
+// -----
+
 // CHECK-LABEL: func @maximumf
 func.func @maximumf(%a: f32, %b: f32) -> f32 {
   %result = arith.maximumf %a, %b : f32
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 77863de1c79a7c5ae3d6f1b4cc8f62f9caf98498 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 20:47:24 +0000
Subject: [PATCH 2/3] [mlir] Refine tensor splat expansion

---
 mlir/include/mlir/Conversion/Passes.td        |   2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.h    |   3 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |   3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |   1 -
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 117 ++++++++++++++----
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    |  10 --
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  10 --
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |   2 -
 8 files changed, 96 insertions(+), 52 deletions(-)
 delete mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 23463342314b0..550e6b853057d 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
index 5a07a01d0928a..11081342c2244 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
@@ -54,7 +54,8 @@ void populateEmulateUnsupportedFloatsPatterns(RewritePatternSet &patterns,
 void populateEmulateUnsupportedFloatsLegality(ConversionTarget &target,
                                               const TypeConverter &converter);
 /// Add patterns to expand Arith ceil/floor division ops.
-void populateCeilFloorDivExpandOpsPatterns(RewritePatternSet &patterns);
+void populateCeilFloorDivExpandOpsPatterns(
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat = false);
 
 /// Add patterns to expand Arith bf16 patterns to lower level bitcasts/shifts.
 void populateExpandBFloat16Patterns(RewritePatternSet &patterns);
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 77e36b45a84af..6bf0fe85bb62f 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,7 +17,6 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -737,7 +736,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
+    context->loadDialect<LLVM::LLVMDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 955fcc397dd37..0a0e25e18b47a 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,5 +16,4 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
-  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index d9cfa6bc78dc6..b8c0e39dfddda 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -24,6 +24,10 @@ namespace arith {
 
 using namespace mlir;
 
+namespace {
+
+enum class DynamicTensorConstantMaterialization { Disabled, TensorSplat };
+
 /// Create an integer or index constant.
 static Value createConst(Location loc, Type type, int value,
                          PatternRewriter &rewriter) {
@@ -38,16 +42,27 @@ static Value createConst(Location loc, Type type, int value,
 /// Create an integer or index constant, using source as the dynamic shape
 /// source for ranked tensors whose splat cannot be represented as a dense
 /// attribute.
-static Value createConst(Location loc, Type type, int value,
-                         PatternRewriter &rewriter, Value source) {
+static FailureOr<Value>
+createConst(Location loc, Type type, int value, PatternRewriter &rewriter,
+            Value source,
+            DynamicTensorConstantMaterialization dynamicTensorMaterialization) {
   auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  auto sourceTensorTy = dyn_cast<RankedTensorType>(source.getType());
   if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
     return createConst(loc, type, value, rewriter);
 
+  if (!sourceTensorTy ||
+      rankedTensorTy.getShape() != sourceTensorTy.getShape() ||
+      dynamicTensorMaterialization !=
+          DynamicTensorConstantMaterialization::TensorSplat)
+    return failure();
+
+  // Dynamic shaped splats need runtime dimension operands from the source.
   Value scalar =
       createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
   return tensor::SplatOp::create(rewriter, loc, scalar,
-                                 tensor::getMixedSizes(rewriter, loc, source));
+                                 tensor::getMixedSizes(rewriter, loc, source))
+      .getResult();
 }
 
 /// Create an integer constant from an APInt.
@@ -81,27 +96,42 @@ static Type cloneToShapedType(Type cloneFrom, Type cloneTo) {
   return cloneTo;
 }
 
-namespace {
-
 /// Expands CeilDivUIOp (n, m) into
 ///  n == 0 ? 0 : ((n-1) / m) + 1
 struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
+  CeilDivUIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivUIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivUIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
-    Value compare =
-        arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter, a);
-    Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
+    FailureOr<Value> zero = createConst(loc, a.getType(), 0, rewriter, a,
+                                        dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value compare = arith::CmpIOp::create(rewriter, loc,
+                                          arith::CmpIPredicate::eq, a, *zero);
+    FailureOr<Value> one = createConst(loc, a.getType(), 1, rewriter, a,
+                                       dynamicTensorMaterialization);
+    if (failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value minusOne = arith::SubIOp::create(rewriter, loc, a, *one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
-    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, zero, plusOne);
+    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, *one);
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, *zero, plusOne);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands CeilDivSIOp (a, b) into
@@ -112,6 +142,12 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
 ///   return z;
 /// }
 struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
+  CeilDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -120,8 +156,13 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter, a);
-    Value one = createConst(loc, type, 1, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    FailureOr<Value> one =
+        createConst(loc, type, 1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero) || failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -129,21 +170,25 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signEqual = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::eq, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signEqual);
 
-    Value quotientPlusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
+    Value quotientPlusOne =
+        arith::AddIOp::create(rewriter, loc, quotient, *one);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientPlusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands FloorDivSIOp (x, y) into
@@ -154,6 +199,12 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
 ///   return z;
 /// }
 struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
+  FloorDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::FloorDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::FloorDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -166,26 +217,37 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signOpposite = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter, a);
+    FailureOr<Value> minusOne =
+        createConst(loc, type, -1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(minusOne))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
     Value quotientMinusOne =
-        arith::AddIOp::create(rewriter, loc, quotient, minusOne);
+        arith::AddIOp::create(rewriter, loc, quotient, *minusOne);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientMinusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 template <typename OpTy, arith::CmpIPredicate pred>
@@ -921,10 +983,14 @@ struct ArithExpandOpsPass
 } // namespace
 
 void mlir::arith::populateCeilFloorDivExpandOpsPatterns(
-    RewritePatternSet &patterns) {
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat) {
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization =
+      enableDynamicTensorSplat
+          ? DynamicTensorConstantMaterialization::TensorSplat
+          : DynamicTensorConstantMaterialization::Disabled;
   patterns
       .add<CeilDivSIOpConverter, CeilDivUIOpConverter, FloorDivSIOpConverter>(
-          patterns.getContext());
+          patterns.getContext(), dynamicTensorMaterialization);
 }
 
 void mlir::arith::populateExpandBFloat16Patterns(RewritePatternSet &patterns) {
@@ -954,7 +1020,8 @@ void mlir::arith::populateExpandFlushDenormalsPatterns(
 }
 
 void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
-  populateCeilFloorDivExpandOpsPatterns(patterns);
+  populateCeilFloorDivExpandOpsPatterns(patterns,
+                                        /*enableDynamicTensorSplat=*/true);
   populateExpandScalingExtTruncPatterns(patterns);
   // clang-format off
   patterns.add<
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
deleted file mode 100644
index 9d169bc60cade..0000000000000
--- a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
+++ /dev/null
@@ -1,10 +0,0 @@
-// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
-// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
-
-// CHECK-LABEL: @ceildivui_dynamic_tensor
-// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
-func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
-// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
-  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
-  return %0: tensor<8x4x?xi64>
-}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 343c34c9f0ba8..65a8180e480b7 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,16 +89,6 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
-// CHECK-LABEL: @dynamic_tensor_ceildivui
-// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
-func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
-  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
-  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
-  return %0 : tensor<8x4x?xi32>
-}
-
-// -----
-
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 58c66ab6f74c7..3c99d3c5b60ce 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,7 +18,6 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -94,7 +93,6 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
-    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From ae1131363388ffca29bb2d1c70e2d379a716701c Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 21:30:10 +0000
Subject: [PATCH 3/3] [mlir] Restore SPIR-V test

---
 mlir/test/Conversion/ConvertToSPIRV/arith.mlir | 10 ++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp  |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 65a8180e480b7..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,16 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 


From llvmlistbot at llvm.org  Fri Jul 10 06:52:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 06:52:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a50f907.d1496011.26913c.6780@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206249

>From 3663396b2155970b6a4982ec42a0108c749df095 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 20:57:43 +0000
Subject: [PATCH 1/4] [mlir] Fix arith tensor constants

---
 mlir/include/mlir/Conversion/Passes.td        |  2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.td   |  2 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |  3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |  1 +
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 29 +++++++--
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    | 10 +++
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  8 +++
 mlir/test/Dialect/Arith/expand-ops.mlir       | 63 +++++++++++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |  2 +
 9 files changed, 111 insertions(+), 9 deletions(-)
 create mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 550e6b853057d..23463342314b0 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
index 27e9146ec3606..2cee05914d6b5 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
@@ -13,7 +13,7 @@ include "mlir/Pass/PassBase.td"
 
 def ArithExpandOpsPass : Pass<"arith-expand"> {
   let summary = "Legalize Arith ops to be convertible to LLVM.";
-  let dependentDialects = ["vector::VectorDialect"];
+  let dependentDialects = ["tensor::TensorDialect", "vector::VectorDialect"];
   let options =
       [Option<"includeBf16", "include-bf16", "bool", /*default=*/"false",
               "Enable the BF16 expansion patterns">,
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..77e36b45a84af 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -736,7 +737,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect>();
+    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 0a0e25e18b47a..955fcc397dd37 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,4 +16,5 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
+  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..d9cfa6bc78dc6 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Arith/Transforms/Passes.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/Location.h"
@@ -34,6 +35,21 @@ static Value createConst(Location loc, Type type, int value,
   return arith::ConstantOp::create(rewriter, loc, attr);
 }
 
+/// Create an integer or index constant, using source as the dynamic shape
+/// source for ranked tensors whose splat cannot be represented as a dense
+/// attribute.
+static Value createConst(Location loc, Type type, int value,
+                         PatternRewriter &rewriter, Value source) {
+  auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
+    return createConst(loc, type, value, rewriter);
+
+  Value scalar =
+      createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
+  return tensor::SplatOp::create(rewriter, loc, scalar,
+                                 tensor::getMixedSizes(rewriter, loc, source));
+}
+
 /// Create an integer constant from an APInt.
 static Value createAPIntConst(Location loc, Type type, const APInt &value,
                               PatternRewriter &rewriter) {
@@ -76,10 +92,10 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter);
+    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
     Value compare =
         arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter);
+    Value one = createConst(loc, a.getType(), 1, rewriter, a);
     Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
     Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
@@ -104,8 +120,8 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter);
-    Value one = createConst(loc, type, 1, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
+    Value one = createConst(loc, type, 1, rewriter, a);
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -150,7 +166,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
                                        a, zero);
@@ -162,7 +178,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter);
+    Value minusOne = createConst(loc, type, -1, rewriter, a);
     Value quotientMinusOne =
         arith::AddIOp::create(rewriter, loc, quotient, minusOne);
 
@@ -826,6 +842,7 @@ struct ArithExpandOpsPass
     arith::populateArithExpandOpsPatterns(patterns);
 
     target.addLegalDialect<arith::ArithDialect>();
+    target.addLegalDialect<tensor::TensorDialect>();
     target.addLegalDialect<vector::VectorDialect>();
 
     // clang-format off
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
new file mode 100644
index 0000000000000..9d169bc60cade
--- /dev/null
+++ b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
+// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
+
+// CHECK-LABEL: @ceildivui_dynamic_tensor
+// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
+func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
+  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
+  return %0: tensor<8x4x?xi64>
+}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index fa204c4df8ace..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,14 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
 // -----
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 75c4de2168761..80b6592417cc2 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -144,6 +144,69 @@ func.func @ceildivui_index(%arg0: index, %arg1: index) -> (index) {
 
 // -----
 
+// CHECK-LABEL:   func.func @ceildivui_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x4x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+func.func @ceildivui_dynamic_tensor(%arg0: tensor<8x4x?xi64>, %arg1: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+  %res = arith.ceildivui %arg0, %arg1 : tensor<8x4x?xi64>
+  return %res : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C2_IDX:.*]] = arith.constant 2 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX]] : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[ISZERO:.*]] = arith.cmpi eq, %[[ARG0]], %[[ZERO]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C2_IDX_1:.*]] = arith.constant 2 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX_1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divui %[[SUB]], %[[ARG1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[RES:.*]] = arith.select %[[ISZERO]], %[[ZERO]], %[[ADD]] : tensor<8x4x?xi1>, tensor<8x4x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @ceildivsi_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x?xi64>) -> tensor<8x?xi64> {
+func.func @ceildivsi_dynamic_tensor(%arg0: tensor<8x?xi64>, %arg1: tensor<8x?xi64>) -> tensor<8x?xi64> {
+  %res = arith.ceildivsi %arg0, %arg1 : tensor<8x?xi64>
+  return %res : tensor<8x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C1_IDX:.*]] = arith.constant 1 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX]] : tensor<8x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C1_IDX_1:.*]] = arith.constant 1 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX_1]] : tensor<8x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<8x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @floordivsi_dynamic_tensor(
+// CHECK-SAME:                                       %[[ARG0:.*]]: tensor<?x4xi64>,
+// CHECK-SAME:                                       %[[ARG1:.*]]: tensor<?x4xi64>) -> tensor<?x4xi64> {
+func.func @floordivsi_dynamic_tensor(%arg0: tensor<?x4xi64>, %arg1: tensor<?x4xi64>) -> tensor<?x4xi64> {
+  %res = arith.floordivsi %arg0, %arg1 : tensor<?x4xi64>
+  return %res : tensor<?x4xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C0_IDX:.*]] = arith.constant 0 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE_SCALAR:.*]] = arith.constant -1 : i64
+// CHECK:       %[[C0_IDX_1:.*]] = arith.constant 0 : index
+// CHECK:       %[[NEG_ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX_1]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE:.*]] = tensor.splat %[[NEG_ONE_SCALAR]]{{\[}}%[[NEG_ONE_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[SUB_ONE:.*]] = arith.addi %[[DIV]], %[[NEG_ONE]] : tensor<?x4xi64>
+}
+
+// -----
+
 // CHECK-LABEL: func @maximumf
 func.func @maximumf(%a: f32, %b: f32) -> f32 {
   %result = arith.maximumf %a, %b : f32
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 77863de1c79a7c5ae3d6f1b4cc8f62f9caf98498 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 20:47:24 +0000
Subject: [PATCH 2/4] [mlir] Refine tensor splat expansion

---
 mlir/include/mlir/Conversion/Passes.td        |   2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.h    |   3 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |   3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |   1 -
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 117 ++++++++++++++----
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    |  10 --
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  10 --
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |   2 -
 8 files changed, 96 insertions(+), 52 deletions(-)
 delete mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 23463342314b0..550e6b853057d 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
index 5a07a01d0928a..11081342c2244 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
@@ -54,7 +54,8 @@ void populateEmulateUnsupportedFloatsPatterns(RewritePatternSet &patterns,
 void populateEmulateUnsupportedFloatsLegality(ConversionTarget &target,
                                               const TypeConverter &converter);
 /// Add patterns to expand Arith ceil/floor division ops.
-void populateCeilFloorDivExpandOpsPatterns(RewritePatternSet &patterns);
+void populateCeilFloorDivExpandOpsPatterns(
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat = false);
 
 /// Add patterns to expand Arith bf16 patterns to lower level bitcasts/shifts.
 void populateExpandBFloat16Patterns(RewritePatternSet &patterns);
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 77e36b45a84af..6bf0fe85bb62f 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,7 +17,6 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -737,7 +736,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
+    context->loadDialect<LLVM::LLVMDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 955fcc397dd37..0a0e25e18b47a 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,5 +16,4 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
-  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index d9cfa6bc78dc6..b8c0e39dfddda 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -24,6 +24,10 @@ namespace arith {
 
 using namespace mlir;
 
+namespace {
+
+enum class DynamicTensorConstantMaterialization { Disabled, TensorSplat };
+
 /// Create an integer or index constant.
 static Value createConst(Location loc, Type type, int value,
                          PatternRewriter &rewriter) {
@@ -38,16 +42,27 @@ static Value createConst(Location loc, Type type, int value,
 /// Create an integer or index constant, using source as the dynamic shape
 /// source for ranked tensors whose splat cannot be represented as a dense
 /// attribute.
-static Value createConst(Location loc, Type type, int value,
-                         PatternRewriter &rewriter, Value source) {
+static FailureOr<Value>
+createConst(Location loc, Type type, int value, PatternRewriter &rewriter,
+            Value source,
+            DynamicTensorConstantMaterialization dynamicTensorMaterialization) {
   auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  auto sourceTensorTy = dyn_cast<RankedTensorType>(source.getType());
   if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
     return createConst(loc, type, value, rewriter);
 
+  if (!sourceTensorTy ||
+      rankedTensorTy.getShape() != sourceTensorTy.getShape() ||
+      dynamicTensorMaterialization !=
+          DynamicTensorConstantMaterialization::TensorSplat)
+    return failure();
+
+  // Dynamic shaped splats need runtime dimension operands from the source.
   Value scalar =
       createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
   return tensor::SplatOp::create(rewriter, loc, scalar,
-                                 tensor::getMixedSizes(rewriter, loc, source));
+                                 tensor::getMixedSizes(rewriter, loc, source))
+      .getResult();
 }
 
 /// Create an integer constant from an APInt.
@@ -81,27 +96,42 @@ static Type cloneToShapedType(Type cloneFrom, Type cloneTo) {
   return cloneTo;
 }
 
-namespace {
-
 /// Expands CeilDivUIOp (n, m) into
 ///  n == 0 ? 0 : ((n-1) / m) + 1
 struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
+  CeilDivUIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivUIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivUIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
-    Value compare =
-        arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter, a);
-    Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
+    FailureOr<Value> zero = createConst(loc, a.getType(), 0, rewriter, a,
+                                        dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value compare = arith::CmpIOp::create(rewriter, loc,
+                                          arith::CmpIPredicate::eq, a, *zero);
+    FailureOr<Value> one = createConst(loc, a.getType(), 1, rewriter, a,
+                                       dynamicTensorMaterialization);
+    if (failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value minusOne = arith::SubIOp::create(rewriter, loc, a, *one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
-    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, zero, plusOne);
+    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, *one);
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, *zero, plusOne);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands CeilDivSIOp (a, b) into
@@ -112,6 +142,12 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
 ///   return z;
 /// }
 struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
+  CeilDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -120,8 +156,13 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter, a);
-    Value one = createConst(loc, type, 1, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    FailureOr<Value> one =
+        createConst(loc, type, 1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero) || failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -129,21 +170,25 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signEqual = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::eq, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signEqual);
 
-    Value quotientPlusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
+    Value quotientPlusOne =
+        arith::AddIOp::create(rewriter, loc, quotient, *one);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientPlusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands FloorDivSIOp (x, y) into
@@ -154,6 +199,12 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
 ///   return z;
 /// }
 struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
+  FloorDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::FloorDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::FloorDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -166,26 +217,37 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signOpposite = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter, a);
+    FailureOr<Value> minusOne =
+        createConst(loc, type, -1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(minusOne))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
     Value quotientMinusOne =
-        arith::AddIOp::create(rewriter, loc, quotient, minusOne);
+        arith::AddIOp::create(rewriter, loc, quotient, *minusOne);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientMinusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 template <typename OpTy, arith::CmpIPredicate pred>
@@ -921,10 +983,14 @@ struct ArithExpandOpsPass
 } // namespace
 
 void mlir::arith::populateCeilFloorDivExpandOpsPatterns(
-    RewritePatternSet &patterns) {
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat) {
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization =
+      enableDynamicTensorSplat
+          ? DynamicTensorConstantMaterialization::TensorSplat
+          : DynamicTensorConstantMaterialization::Disabled;
   patterns
       .add<CeilDivSIOpConverter, CeilDivUIOpConverter, FloorDivSIOpConverter>(
-          patterns.getContext());
+          patterns.getContext(), dynamicTensorMaterialization);
 }
 
 void mlir::arith::populateExpandBFloat16Patterns(RewritePatternSet &patterns) {
@@ -954,7 +1020,8 @@ void mlir::arith::populateExpandFlushDenormalsPatterns(
 }
 
 void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
-  populateCeilFloorDivExpandOpsPatterns(patterns);
+  populateCeilFloorDivExpandOpsPatterns(patterns,
+                                        /*enableDynamicTensorSplat=*/true);
   populateExpandScalingExtTruncPatterns(patterns);
   // clang-format off
   patterns.add<
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
deleted file mode 100644
index 9d169bc60cade..0000000000000
--- a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
+++ /dev/null
@@ -1,10 +0,0 @@
-// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
-// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
-
-// CHECK-LABEL: @ceildivui_dynamic_tensor
-// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
-func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
-// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
-  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
-  return %0: tensor<8x4x?xi64>
-}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 343c34c9f0ba8..65a8180e480b7 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,16 +89,6 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
-// CHECK-LABEL: @dynamic_tensor_ceildivui
-// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
-func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
-  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
-  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
-  return %0 : tensor<8x4x?xi32>
-}
-
-// -----
-
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 58c66ab6f74c7..3c99d3c5b60ce 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,7 +18,6 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -94,7 +93,6 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
-    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From ae1131363388ffca29bb2d1c70e2d379a716701c Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 21:30:10 +0000
Subject: [PATCH 3/4] [mlir] Restore SPIR-V test

---
 mlir/test/Conversion/ConvertToSPIRV/arith.mlir | 10 ++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp  |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 65a8180e480b7..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,16 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 4f6a92bef6a65f92a103adb2c4a6c8211ea31352 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 21:54:16 +0000
Subject: [PATCH 4/4] [mlir] Drop SPIR-V tensor test

---
 mlir/test/Conversion/ConvertToSPIRV/arith.mlir | 8 --------
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp  | 2 --
 2 files changed, 10 deletions(-)

diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 343c34c9f0ba8..fa204c4df8ace 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,14 +89,6 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
-// CHECK-LABEL: @dynamic_tensor_ceildivui
-// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
-func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
-  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
-  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
-  return %0 : tensor<8x4x?xi32>
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 58c66ab6f74c7..3c99d3c5b60ce 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,7 +18,6 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -94,7 +93,6 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
-    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 


From llvmlistbot at llvm.org  Fri Jul 10 06:52:19 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Fri, 10 Jul 2026 06:52:19 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a50f913.30759430.24d50c.7a70@mx.google.com>


https://github.com/TIFitis updated https://github.com/llvm/llvm-project/pull/162501

>From ba1c14f9f56a8b97e603eaa15ea8ab5cf139beb0 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Wed, 8 Oct 2025 16:51:51 +0100
Subject: [PATCH 1/7] [Flang][OpenMP] Fix Fortran automap handling

	- Move automapped Fortran descriptor mappings from map() to has_device_addr in target regions, updating block args and uses accordingly.
	- In libomptarget, detect CFI descriptors during pointer attachment and compute the correct descriptor size to transfer full metadata (including bounds).
	- Resolves lost bounds for automapped Fortran arrays on device; no change for C/C++.

This fixes test offload/test/offloading/fortran/declare-target-automap.f90 reported broken in #161265.
---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 102 ++++++++++++++++++
 offload/libomptarget/omptarget.cpp            |  32 ++++++
 .../fortran/declare-target-automap.f90        |   3 -
 3 files changed, 134 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index eeb08ebf51191..5159158f53339 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -18,8 +18,12 @@
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Operation.h"
+#include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
+#include "llvm/Frontend/OpenMP/OMPConstants.h"
+#include <algorithm>
+
 namespace flangomp {
 #define GEN_PASS_DEF_AUTOMAPTOTARGETDATAPASS
 #include "flang/Optimizer/OpenMP/Passes.h.inc"
@@ -150,6 +154,104 @@ class AutomapToTargetDataPass
           addMapInfo(globalOp, loadOp);
       }
     }
+
+    // Move automapped descriptors from map() to has_device_addr in target ops.
+    auto originatesFromAutomapGlobal = [&](mlir::Value varPtr) -> bool {
+      if (auto decl = mlir::dyn_cast_or_null<hlfir::DeclareOp>(
+              varPtr.getDefiningOp())) {
+        if (auto addrOp = mlir::dyn_cast_or_null<fir::AddrOfOp>(
+                decl.getMemref().getDefiningOp())) {
+          if (auto g =
+                  mlir::SymbolTable::lookupNearestSymbolFrom<fir::GlobalOp>(
+                      decl, addrOp.getSymbol()))
+            return automapGlobals.contains(g);
+        }
+      }
+      return false;
+    };
+
+    module.walk([&](mlir::omp::TargetOp target) {
+      // Collect candidates to move: descriptor maps of automapped globals.
+      llvm::SmallVector<mlir::Value> newMapOps;
+      llvm::SmallVector<unsigned> removedIndices;
+      llvm::SmallVector<mlir::Value> movedToHDA;
+      llvm::SmallVector<mlir::BlockArgument> oldMapArgsForMoved;
+
+      auto mapRange = target.getMapVars();
+      newMapOps.reserve(mapRange.size());
+
+      auto argIface = llvm::dyn_cast<mlir::omp::BlockArgOpenMPOpInterface>(
+          target.getOperation());
+      llvm::ArrayRef<mlir::BlockArgument> mapBlockArgs =
+          argIface.getMapBlockArgs();
+
+      for (auto [idx, mapVal] : llvm::enumerate(mapRange)) {
+        auto mapOp =
+            mlir::dyn_cast<mlir::omp::MapInfoOp>(mapVal.getDefiningOp());
+        if (!mapOp) {
+          newMapOps.push_back(mapVal);
+          continue;
+        }
+
+        mlir::Type varTy = fir::unwrapRefType(mapOp.getVarType());
+        bool isDescriptor = mlir::isa<fir::BaseBoxType>(varTy);
+        if (isDescriptor && originatesFromAutomapGlobal(mapOp.getVarPtr())) {
+          movedToHDA.push_back(mapVal);
+          removedIndices.push_back(idx);
+          oldMapArgsForMoved.push_back(mapBlockArgs[idx]);
+        } else {
+          newMapOps.push_back(mapVal);
+        }
+      }
+
+      if (movedToHDA.empty())
+        return;
+
+      // Update map vars to exclude moved entries.
+      mlir::MutableOperandRange mapMutable = target.getMapVarsMutable();
+      mapMutable.assign(newMapOps);
+
+      // Append moved entries to has_device_addr and insert corresponding block
+      // arguments.
+      mlir::MutableOperandRange hdaMutable =
+          target.getHasDeviceAddrVarsMutable();
+      llvm::SmallVector<mlir::Value> newHDA;
+      newHDA.reserve(hdaMutable.size() + movedToHDA.size());
+      llvm::for_each(hdaMutable.getAsOperandRange(),
+                     [&](mlir::Value v) { newHDA.push_back(v); });
+
+      unsigned hdaStart = argIface.getHasDeviceAddrBlockArgsStart();
+      unsigned oldHdaCount = argIface.numHasDeviceAddrBlockArgs();
+      llvm::SmallVector<mlir::BlockArgument> newHDAArgsForMoved;
+      unsigned insertIndex = hdaStart + oldHdaCount;
+      for (mlir::Value v : movedToHDA) {
+        newHDA.push_back(v);
+        target->getRegion(0).insertArgument(insertIndex, v.getType(),
+                                            v.getLoc());
+        // Capture the newly inserted block argument.
+        newHDAArgsForMoved.push_back(
+            target->getRegion(0).getArgument(insertIndex));
+        insertIndex++;
+      }
+      hdaMutable.assign(newHDA);
+
+      // Redirect uses in the region: replace old map block args with the
+      // corresponding new has_device_addr block args.
+      for (auto [oldArg, newArg] :
+           llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
+        oldArg.replaceAllUsesWith(newArg);
+
+      // Finally, erase corresponding map block arguments (descending order).
+      unsigned mapStart = argIface.getMapBlockArgsStart();
+      // Convert indices to absolute argument numbers before erasing.
+      llvm::SmallVector<unsigned> absArgNos;
+      absArgNos.reserve(removedIndices.size());
+      for (unsigned idx : removedIndices)
+        absArgNos.push_back(mapStart + idx);
+      std::sort(absArgNos.begin(), absArgNos.end(), std::greater<>());
+      for (unsigned absNo : absArgNos)
+        target->getRegion(0).eraseArgument(absNo);
+    });
   }
 };
 } // namespace
diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 17b215732d51b..2479fc489d051 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -33,6 +33,9 @@
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
 #include "llvm/Object/ObjectFile.h"
 
+#include "flang/ISO_Fortran_binding.h"
+
+#include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <vector>
@@ -381,6 +384,34 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
+// Fortran pointer attachments treated descriptors as plain pointers, so
+// automapped arrays lose their declared bounds on the device. Recognize
+// CFI descriptors to compute their actual size before copying, ensuring the
+// full descriptor (including bounds) is transferred during attachment.
+static int64_t getFortranDescriptorSize(void **HstPtrAddr,
+                                        int64_t ReportedSize) {
+  constexpr int64_t VoidPtrSize = sizeof(void *);
+
+  if (!HstPtrAddr || ReportedSize > VoidPtrSize)
+    return ReportedSize;
+
+  const CFI_cdesc_t *Desc = reinterpret_cast<const CFI_cdesc_t *>(HstPtrAddr);
+
+  if (Desc->version != CFI_VERSION)
+    return ReportedSize;
+
+  if (Desc->rank > CFI_MAX_RANK)
+    return ReportedSize;
+
+  const char *RawDesc = reinterpret_cast<const char *>(Desc);
+  const char *DimsAddr = reinterpret_cast<const char *>(&Desc->dim);
+  size_t HeaderBytes = static_cast<size_t>(DimsAddr - RawDesc);
+  size_t DimsBytes = static_cast<size_t>(Desc->rank) * sizeof(CFI_dim_t);
+  size_t TotalBytes = HeaderBytes + DimsBytes;
+
+  return std::max<int64_t>(ReportedSize, static_cast<int64_t>(TotalBytes));
+}
+
 /// Utility function to perform a pointer attachment operation.
 ///
 /// For something like:
@@ -448,6 +479,7 @@ static int performPointerAttachment(DeviceTy &Device, AsyncInfoTy &AsyncInfo,
          "Need a valid pointer entry to perform pointer-attachment");
 
   constexpr int64_t VoidPtrSize = sizeof(void *);
+  HstPtrSize = getFortranDescriptorSize(HstPtrAddr, HstPtrSize);
   assert(HstPtrSize >= VoidPtrSize && "PointerSize is too small");
 
   void *TgtPteeBase =
diff --git a/offload/test/offloading/fortran/declare-target-automap.f90 b/offload/test/offloading/fortran/declare-target-automap.f90
index b44c0b2815274..b9c2d34c834fa 100644
--- a/offload/test/offloading/fortran/declare-target-automap.f90
+++ b/offload/test/offloading/fortran/declare-target-automap.f90
@@ -1,9 +1,6 @@
 !Offloading test for AUTOMAP modifier in declare target enter
 ! REQUIRES: flang, amdgpu
 
-! FIXME: https://github.com/llvm/llvm-project/issues/161265
-! XFAIL: amdgpu
-
 ! RUN: %libomptarget-compile-fortran-run-and-check-generic
 program automap_program
    use iso_c_binding, only: c_loc

>From ab202f9e708037b4c33c5ae8ac630c4595525927 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 9 Oct 2025 16:30:31 +0100
Subject: [PATCH 2/7] Address copilot suggestions.

---
 flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp | 5 ++++-
 offload/libomptarget/omptarget.cpp                 | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index 5159158f53339..f0e0eefa7e220 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -241,7 +241,10 @@ class AutomapToTargetDataPass
            llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
         oldArg.replaceAllUsesWith(newArg);
 
-      // Finally, erase corresponding map block arguments (descending order).
+      // Finally, erase corresponding map block arguments in descending order.
+      // Descending order is necessary to avoid index invalidation: erasing
+      // arguments from highest to lowest index ensures that earlier erases do
+      // not shift the indices of arguments yet to be erased.
       unsigned mapStart = argIface.getMapBlockArgsStart();
       // Convert indices to absolute argument numbers before erasing.
       llvm::SmallVector<unsigned> absArgNos;
diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 2479fc489d051..6b8ba9e5390c1 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -384,7 +384,7 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
-// Fortran pointer attachments treated descriptors as plain pointers, so
+// Fortran pointer attachments treat descriptors as plain pointers, so
 // automapped arrays lose their declared bounds on the device. Recognize
 // CFI descriptors to compute their actual size before copying, ensuring the
 // full descriptor (including bounds) is transferred during attachment.

>From edf7bec4603aef064d3a10b8c26b57ec16de1602 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 30 Oct 2025 14:46:11 +0000
Subject: [PATCH 3/7] Remove omptarget changes.

---
 offload/libomptarget/omptarget.cpp | 32 ------------------------------
 1 file changed, 32 deletions(-)

diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 6b8ba9e5390c1..17b215732d51b 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -33,9 +33,6 @@
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
 #include "llvm/Object/ObjectFile.h"
 
-#include "flang/ISO_Fortran_binding.h"
-
-#include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <vector>
@@ -384,34 +381,6 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
-// Fortran pointer attachments treat descriptors as plain pointers, so
-// automapped arrays lose their declared bounds on the device. Recognize
-// CFI descriptors to compute their actual size before copying, ensuring the
-// full descriptor (including bounds) is transferred during attachment.
-static int64_t getFortranDescriptorSize(void **HstPtrAddr,
-                                        int64_t ReportedSize) {
-  constexpr int64_t VoidPtrSize = sizeof(void *);
-
-  if (!HstPtrAddr || ReportedSize > VoidPtrSize)
-    return ReportedSize;
-
-  const CFI_cdesc_t *Desc = reinterpret_cast<const CFI_cdesc_t *>(HstPtrAddr);
-
-  if (Desc->version != CFI_VERSION)
-    return ReportedSize;
-
-  if (Desc->rank > CFI_MAX_RANK)
-    return ReportedSize;
-
-  const char *RawDesc = reinterpret_cast<const char *>(Desc);
-  const char *DimsAddr = reinterpret_cast<const char *>(&Desc->dim);
-  size_t HeaderBytes = static_cast<size_t>(DimsAddr - RawDesc);
-  size_t DimsBytes = static_cast<size_t>(Desc->rank) * sizeof(CFI_dim_t);
-  size_t TotalBytes = HeaderBytes + DimsBytes;
-
-  return std::max<int64_t>(ReportedSize, static_cast<int64_t>(TotalBytes));
-}
-
 /// Utility function to perform a pointer attachment operation.
 ///
 /// For something like:
@@ -479,7 +448,6 @@ static int performPointerAttachment(DeviceTy &Device, AsyncInfoTy &AsyncInfo,
          "Need a valid pointer entry to perform pointer-attachment");
 
   constexpr int64_t VoidPtrSize = sizeof(void *);
-  HstPtrSize = getFortranDescriptorSize(HstPtrAddr, HstPtrSize);
   assert(HstPtrSize >= VoidPtrSize && "PointerSize is too small");
 
   void *TgtPteeBase =

>From 18af0fac40306e1fe1f3c2dca64090e7b73390bd Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Wed, 6 May 2026 15:30:12 +0100
Subject: [PATCH 4/7] Reworked PR to avoid kernel changes.

---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 197 +++++++-----------
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  |  31 ++-
 .../Lower/OpenMP/declare-target-automap.f90   |  27 +++
 .../Transforms/omp-automap-to-target-data.fir |  35 ++--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  51 ++++-
 .../omptarget-declare-target-llvm-host.mlir   |  10 +
 mlir/test/Target/LLVMIR/omptarget-llvm.mlir   |  31 +++
 7 files changed, 239 insertions(+), 143 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/declare-target-automap.f90

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index f0e0eefa7e220..dd006e046fa74 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -15,14 +15,11 @@
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Operation.h"
-#include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
-#include <algorithm>
 
 namespace flangomp {
 #define GEN_PASS_DEF_AUTOMAPTOTARGETDATAPASS
@@ -90,11 +87,7 @@ class AutomapToTargetDataPass
   }
 
   void runOnOperation() override {
-    ModuleOp module = getOperation()->getParentOfType<ModuleOp>();
-    if (!module)
-      module = dyn_cast<ModuleOp>(getOperation());
-    if (!module)
-      return;
+    ModuleOp module = getOperation();
 
     // Build FIR builder for helper utilities.
     fir::KindMapping kindMap = fir::getKindMapping(module);
@@ -117,16 +110,61 @@ class AutomapToTargetDataPass
       if (needsBoundsOps(memOp.getMemref()))
         genBoundsOps(builder, memOp.getMemref(), bounds);
 
+      mlir::Value boxValue;
+      if (auto storeOp = mlir::dyn_cast<fir::StoreOp>(memOp.getOperation()))
+        boxValue = storeOp.getValue();
+      else
+        boxValue = mlir::cast<fir::LoadOp>(memOp.getOperation()).getResult();
+
+      mlir::Value baseAddr =
+          fir::BoxAddrOp::create(builder, memOp.getLoc(), boxValue);
+      mlir::Value dataAddr = builder.createConvert(
+          memOp.getLoc(),
+          builder.getRefType(fir::unwrapRefType(baseAddr.getType())), baseAddr);
+      mlir::Type baseTy = fir::unwrapRefType(dataAddr.getType());
+      if (mlir::Type eleTy = fir::dyn_cast_ptrOrBoxEleTy(baseTy))
+        baseTy = eleTy;
+      if (auto seqTy = mlir::dyn_cast<fir::SequenceType>(baseTy))
+        if (seqTy.hasDynamicExtents())
+          baseTy = seqTy.getEleTy();
+
       omp::TargetEnterExitUpdateDataOperands clauses;
+      bool isAlloc = isa<fir::StoreOp>(memOp);
+
+      auto createDescriptorMap =
+          [&](mlir::omp::ClauseMapFlags mapType) -> mlir::omp::MapInfoOp {
+        mlir::Type descriptorTy =
+            fir::unwrapRefType(memOp.getMemref().getType());
+        // The attach entry expects the descriptor object to already have a
+        // device mapping, but this raw object map must not be expanded as a
+        // Fortran descriptor member map.
+        return mlir::omp::MapInfoOp::create(
+            builder, memOp.getLoc(), memOp.getMemref().getType(),
+            memOp.getMemref(), TypeAttr::get(descriptorTy),
+            builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
+            builder.getAttr<omp::VariableCaptureKindAttr>(
+                omp::VariableCaptureKind::ByRef),
+            /*var_ptr_ptr=*/mlir::Value{},
+            /*var_ptr_ptr_type=*/mlir::TypeAttr{},
+            /*members=*/SmallVector<Value>{},
+            /*members_index=*/ArrayAttr{},
+            /*bounds=*/SmallVector<Value>{},
+            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
+            builder.getBoolAttr(true));
+      };
+
+      if (isAlloc)
+        clauses.mapVars.push_back(createDescriptorMap(
+            omp::ClauseMapFlags::to | omp::ClauseMapFlags::always));
+
+      mlir::omp::ClauseMapFlags mapType =
+          isAlloc ? omp::ClauseMapFlags::storage : omp::ClauseMapFlags::del;
       mlir::omp::MapInfoOp mapInfo = mlir::omp::MapInfoOp::create(
-          builder, memOp.getLoc(), memOp.getMemref().getType(),
-          memOp.getMemref(),
-          TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-          builder.getAttr<omp::ClauseMapFlagsAttr>(
-              isa<fir::StoreOp>(memOp) ? omp::ClauseMapFlags::to
-                                       : omp::ClauseMapFlags::del),
+          builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
+          TypeAttr::get(baseTy),
+          builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
           builder.getAttr<omp::VariableCaptureKindAttr>(
-              omp::VariableCaptureKind::ByCopy),
+              omp::VariableCaptureKind::ByRef),
           /*var_ptr_ptr=*/mlir::Value{},
           /*var_ptr_ptr_type=*/mlir::TypeAttr{},
           /*members=*/SmallVector<Value>{},
@@ -134,9 +172,31 @@ class AutomapToTargetDataPass
           /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
           builder.getBoolAttr(false));
       clauses.mapVars.push_back(mapInfo);
-      isa<fir::StoreOp>(memOp)
-          ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
-          : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
+
+      if (isAlloc) {
+        mlir::omp::MapInfoOp attachInfo = mlir::omp::MapInfoOp::create(
+            builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
+            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
+            builder.getAttr<omp::ClauseMapFlagsAttr>(
+                omp::ClauseMapFlags::attach),
+            builder.getAttr<omp::VariableCaptureKindAttr>(
+                omp::VariableCaptureKind::ByRef),
+            /*var_ptr_ptr=*/memOp.getMemref(),
+            /*var_ptr_ptr_type=*/TypeAttr::get(
+                fir::unwrapRefType(memOp.getMemref().getType())),
+            /*members=*/SmallVector<Value>{},
+            /*members_index=*/ArrayAttr{},
+            /*bounds=*/SmallVector<Value>{},
+            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
+            builder.getBoolAttr(false));
+        clauses.mapVars.push_back(attachInfo);
+      } else {
+        clauses.mapVars.push_back(
+            createDescriptorMap(omp::ClauseMapFlags::del));
+      }
+
+      isAlloc ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
+              : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
     };
 
     for (fir::GlobalOp globalOp : automapGlobals) {
@@ -154,107 +214,6 @@ class AutomapToTargetDataPass
           addMapInfo(globalOp, loadOp);
       }
     }
-
-    // Move automapped descriptors from map() to has_device_addr in target ops.
-    auto originatesFromAutomapGlobal = [&](mlir::Value varPtr) -> bool {
-      if (auto decl = mlir::dyn_cast_or_null<hlfir::DeclareOp>(
-              varPtr.getDefiningOp())) {
-        if (auto addrOp = mlir::dyn_cast_or_null<fir::AddrOfOp>(
-                decl.getMemref().getDefiningOp())) {
-          if (auto g =
-                  mlir::SymbolTable::lookupNearestSymbolFrom<fir::GlobalOp>(
-                      decl, addrOp.getSymbol()))
-            return automapGlobals.contains(g);
-        }
-      }
-      return false;
-    };
-
-    module.walk([&](mlir::omp::TargetOp target) {
-      // Collect candidates to move: descriptor maps of automapped globals.
-      llvm::SmallVector<mlir::Value> newMapOps;
-      llvm::SmallVector<unsigned> removedIndices;
-      llvm::SmallVector<mlir::Value> movedToHDA;
-      llvm::SmallVector<mlir::BlockArgument> oldMapArgsForMoved;
-
-      auto mapRange = target.getMapVars();
-      newMapOps.reserve(mapRange.size());
-
-      auto argIface = llvm::dyn_cast<mlir::omp::BlockArgOpenMPOpInterface>(
-          target.getOperation());
-      llvm::ArrayRef<mlir::BlockArgument> mapBlockArgs =
-          argIface.getMapBlockArgs();
-
-      for (auto [idx, mapVal] : llvm::enumerate(mapRange)) {
-        auto mapOp =
-            mlir::dyn_cast<mlir::omp::MapInfoOp>(mapVal.getDefiningOp());
-        if (!mapOp) {
-          newMapOps.push_back(mapVal);
-          continue;
-        }
-
-        mlir::Type varTy = fir::unwrapRefType(mapOp.getVarType());
-        bool isDescriptor = mlir::isa<fir::BaseBoxType>(varTy);
-        if (isDescriptor && originatesFromAutomapGlobal(mapOp.getVarPtr())) {
-          movedToHDA.push_back(mapVal);
-          removedIndices.push_back(idx);
-          oldMapArgsForMoved.push_back(mapBlockArgs[idx]);
-        } else {
-          newMapOps.push_back(mapVal);
-        }
-      }
-
-      if (movedToHDA.empty())
-        return;
-
-      // Update map vars to exclude moved entries.
-      mlir::MutableOperandRange mapMutable = target.getMapVarsMutable();
-      mapMutable.assign(newMapOps);
-
-      // Append moved entries to has_device_addr and insert corresponding block
-      // arguments.
-      mlir::MutableOperandRange hdaMutable =
-          target.getHasDeviceAddrVarsMutable();
-      llvm::SmallVector<mlir::Value> newHDA;
-      newHDA.reserve(hdaMutable.size() + movedToHDA.size());
-      llvm::for_each(hdaMutable.getAsOperandRange(),
-                     [&](mlir::Value v) { newHDA.push_back(v); });
-
-      unsigned hdaStart = argIface.getHasDeviceAddrBlockArgsStart();
-      unsigned oldHdaCount = argIface.numHasDeviceAddrBlockArgs();
-      llvm::SmallVector<mlir::BlockArgument> newHDAArgsForMoved;
-      unsigned insertIndex = hdaStart + oldHdaCount;
-      for (mlir::Value v : movedToHDA) {
-        newHDA.push_back(v);
-        target->getRegion(0).insertArgument(insertIndex, v.getType(),
-                                            v.getLoc());
-        // Capture the newly inserted block argument.
-        newHDAArgsForMoved.push_back(
-            target->getRegion(0).getArgument(insertIndex));
-        insertIndex++;
-      }
-      hdaMutable.assign(newHDA);
-
-      // Redirect uses in the region: replace old map block args with the
-      // corresponding new has_device_addr block args.
-      for (auto [oldArg, newArg] :
-           llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
-        oldArg.replaceAllUsesWith(newArg);
-
-      // Finally, erase corresponding map block arguments in descending order.
-      // Descending order is necessary to avoid index invalidation: erasing
-      // arguments from highest to lowest index ensures that earlier erases do
-      // not shift the indices of arguments yet to be erased.
-      unsigned mapStart = argIface.getMapBlockArgsStart();
-      // Convert indices to absolute argument numbers before erasing.
-      llvm::SmallVector<unsigned> absArgNos;
-      absArgNos.reserve(removedIndices.size());
-      for (unsigned idx : removedIndices)
-        absArgNos.push_back(mapStart + idx);
-      std::sort(absArgNos.begin(), absArgNos.end(), std::greater<>());
-      for (unsigned absNo : absArgNos)
-        target->getRegion(0).eraseArgument(absNo);
-    });
   }
 };
 } // namespace
diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index d4b343de988f2..66afbed93df7a 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1351,6 +1351,34 @@ class MapInfoFinalizationPass
     return false;
   }
 
+  static bool isAttachMap(mlir::omp::MapInfoOp op) {
+    return (op.getMapType() & mlir::omp::ClauseMapFlags::attach) ==
+           mlir::omp::ClauseMapFlags::attach;
+  }
+
+  static bool isDescriptorOnlyMap(mlir::omp::MapInfoOp op) {
+    // A descriptor-only map keeps the descriptor object present so that a
+    // subsequent attach map can update its base address. It intentionally does
+    // not map the descriptor's pointee data, so descriptor member expansion
+    // must leave it alone.
+    if (!op.getPartialMap() || !op.getMembers().empty() ||
+        !op.getBounds().empty() || op.getVarPtrPtr() ||
+        op.getMapCaptureType() != mlir::omp::VariableCaptureKind::ByRef)
+      return false;
+
+    mlir::omp::ClauseMapFlags mapType = op.getMapType();
+    if ((mapType & mlir::omp::ClauseMapFlags::from) ==
+            mlir::omp::ClauseMapFlags::from ||
+        (mapType & mlir::omp::ClauseMapFlags::storage) ==
+            mlir::omp::ClauseMapFlags::storage)
+      return false;
+
+    return (mapType & mlir::omp::ClauseMapFlags::del) ==
+               mlir::omp::ClauseMapFlags::del ||
+           (mapType & mlir::omp::ClauseMapFlags::to) ==
+               mlir::omp::ClauseMapFlags::to;
+  }
+
   // This pass executes on omp::MapInfoOp's containing descriptor based types
   // (allocatables, pointers, assumed shape etc.) and expanding them into
   // multiple omp::MapInfoOp's for each pointer member contained within the
@@ -1612,7 +1640,8 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (!isAttachMap(op) && !isDescriptorOnlyMap(op) &&
+            hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
           mlir::Operation *targetUser = getFirstTargetUser(op);
diff --git a/flang/test/Lower/OpenMP/declare-target-automap.f90 b/flang/test/Lower/OpenMP/declare-target-automap.f90
new file mode 100644
index 0000000000000..fe46a3ab75c59
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-automap.f90
@@ -0,0 +1,27 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=60 %s -o - | FileCheck %s
+
+program automap_program
+   integer, allocatable, target :: automap_array(:)
+   !$omp declare target enter(automap:automap_array)
+
+   allocate (automap_array(10))
+
+   !$omp target
+      automap_array(1) = 1
+   !$omp end target
+
+   deallocate (automap_array)
+end program
+
+! CHECK-LABEL: func.func @_QQmain()
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+! CHECK-NOT: has_device_addr
+! CHECK: omp.target {{.*}}map_entries(
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DELETE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: omp.target_exit_data map_entries(%[[DELETE_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
diff --git a/flang/test/Transforms/omp-automap-to-target-data.fir b/flang/test/Transforms/omp-automap-to-target-data.fir
index 7a19705a248b4..7ed3cfa129e51 100644
--- a/flang/test/Transforms/omp-automap-to-target-data.fir
+++ b/flang/test/Transforms/omp-automap-to-target-data.fir
@@ -38,21 +38,26 @@ module {
 // CHECK-NEXT:      fir.shape
 // CHECK-NEXT:      %[[ARR_BOXED:.*]] = fir.embox %[[ALLOC_MEM]]
 // CHECK-NEXT:      fir.store %[[ARR_BOXED]]
-// CHECK-NEXT:      %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK-NEXT:      %[[ARR_HEAP_PTR:.*]] = fir.box_addr %[[ARR_BOXED_LOADED]]
-// CHECK-NEXT:      %[[DIM0:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
-// CHECK-NEXT:      %[[ONE:.*]] = arith.constant 1 : index
-// CHECK-NEXT:      %[[ZERO:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
-// CHECK-NEXT:      %[[LOWER_BOUND:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
-// CHECK-NEXT:      omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
-// CHECK-NEXT:      arith.muli %[[BOX_DIMS2]]#2, %[[BOX_DIMS2]]#1 : index
-// CHECK-NEXT:      %[[MAP_INFO:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(to) capture(ByCopy)
-// CHECK-NEXT:      omp.target_enter_data map_entries(%[[MAP_INFO]]
+// CHECK:           %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
+// CHECK:           %[[DIM0:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
+// CHECK:           %[[ONE:.*]] = arith.constant 1 : index
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
+// CHECK:           %[[LOWER_BOUND:.*]] = arith.constant 0 : index
+// CHECK:           %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
+// CHECK:           %[[BOUNDS:.*]] = omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
+// CHECK:           %[[DATA_ADDR:.*]] = fir.box_addr %[[ARR_BOXED]]
+// CHECK-NEXT:      %[[DATA_REF:.*]] = fir.convert %[[DATA_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%[[BOUNDS]]) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
 // CHECK:           %[[LOAD:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK:           %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(delete) capture(ByCopy)
-// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]]
+// CHECK:           %[[EXIT_ADDR:.*]] = fir.box_addr %[[LOAD]]
+// CHECK-NEXT:      %[[EXIT_REF:.*]] = fir.convert %[[EXIT_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[EXIT_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef)
+// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 // CHECK-NEXT:      %[[BOXADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      fir.freemem %[[BOXADDR]]
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..2dde7ab1614c0 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5719,16 +5719,18 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
+    bool isAttachMap =
+        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
+    Value pointeePtr = isRefPtrOrPteeMapWithAttach
+                           ? mapOp.getVarPtrPtr()
+                           : (isAttachMap ? mapOp.getVarPtr() : offloadPtr);
     mapData.OriginalValue.push_back(moduleTranslation.lookupValue(offloadPtr));
-    mapData.Pointers.push_back(
-        isRefPtrOrPteeMapWithAttach
-            ? moduleTranslation.lookupValue(mapOp.getVarPtrPtr())
-            : mapData.OriginalValue.back());
+    mapData.Pointers.push_back(moduleTranslation.lookupValue(pointeePtr));
 
     if (llvm::Value *refPtr =
             getRefPtrIfDeclareTarget(offloadPtr, moduleTranslation)) {
@@ -6543,6 +6545,10 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         ((convertClauseMapFlags(mapOp.getMapType()) &
           llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH) ==
          llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH);
+    bool isRefPtrOrPteeMapWithAttach =
+        isAttachMap &&
+        (bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptr) ||
+         bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptee));
 
     // If it's declare target, skip it, it's handled separately. However, if
     // it's declare target, and an attach map, we want to calculate the exact
@@ -6565,7 +6571,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy)
+        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
           newV = builder.CreateLoad(builder.getPtrTy(), newV);
 
         if (!offsetIdx.empty())
@@ -8182,6 +8188,35 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
           convertToCaptureClauseKind(attribute.getCaptureClause().getValue());
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
+      llvm::StringRef entryMangledName = mangledName;
+      llvm::Constant *entryAddr = llvm::cast<llvm::Constant>(gVal);
+      std::function<llvm::GlobalValue::LinkageTypes()> variableLinkage;
+      llvm::SmallString<128> entryNameStorage;
+      if ((attribute.getCaptureClause().getValue() ==
+               mlir::omp::DeclareTargetCaptureClause::to ||
+           attribute.getCaptureClause().getValue() ==
+               mlir::omp::DeclareTargetCaptureClause::enter) &&
+          !isDeclaration &&
+          (gVal->hasLocalLinkage() || gVal->hasHiddenVisibility())) {
+        // Keep the original symbol local so target code can address it using
+        // target-specific local relocations, but create a visible alias for
+        // the offload entry so libomptarget can associate the host global with
+        // the actual device global.
+        entryNameStorage = (mangledName + llvm::Twine("_decl_tgt_entry")).str();
+        entryMangledName = entryNameStorage;
+        if (llvm::GlobalValue *existing =
+                llvmModule->getNamedValue(entryMangledName)) {
+          entryAddr = llvm::cast<llvm::Constant>(existing);
+        } else {
+          entryAddr = llvm::GlobalAlias::create(
+              gVal->getValueType(), gVal->getAddressSpace(),
+              llvm::GlobalValue::WeakAnyLinkage, entryMangledName, entryAddr,
+              llvmModule);
+          llvm::cast<llvm::GlobalAlias>(entryAddr)->setVisibility(
+              llvm::GlobalValue::DefaultVisibility);
+        }
+        variableLinkage = [] { return llvm::GlobalValue::WeakAnyLinkage; };
+      }
       // unused for MLIR at the moment, required in Clang for book
       // keeping
       std::vector<llvm::GlobalVariable *> generatedRefs;
@@ -8211,9 +8246,9 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
       ompBuilder->registerTargetGlobalVariable(
           captureClause, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
-          mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
-          /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
-          gVal->getType(), gVal);
+          entryMangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
+          /*GlobalInitializer*/ nullptr, variableLinkage, gVal->getType(),
+          entryAddr);
 
       bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
       if (ompBuilder->Config.isTargetDevice() &&
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
index 92c85738dbc72..3343c96962f6c 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
@@ -133,6 +133,16 @@ module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_targe
     llvm.return %0 : i32
   }
 
+  // CHECK-DAG: @_QFEinternal_enter = internal global i32 7
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [34 x i8] c"_QFEinternal_enter_decl_tgt_entry\00"
+  // CHECK-DAG: @.offloading.entry._QFEinternal_enter_decl_tgt_entry = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEinternal_enter_decl_tgt_entry, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: @_QFEinternal_enter_decl_tgt_entry = weak alias i32, ptr @_QFEinternal_enter
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEinternal_enter_decl_tgt_entry", i32 {{.*}}, i32 {{.*}}}
+  llvm.mlir.global internal @_QFEinternal_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
+    %0 = llvm.mlir.constant(7 : i32) : i32
+    llvm.return %0 : i32
+  }
+
   // CHECK-DAG: @_QMtest_0Ept1 = global { ptr, i64, i32, i8, i8, i8, i8 } { ptr null, i64 4, i32 20180515, i8 0, i8 9, i8 1, i8 0 }
   // CHECK-DAG: @_QMtest_0Ept1_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Ept1
   // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [31 x i8] c"_QMtest_0Ept1_decl_tgt_ref_ptr\00"
diff --git a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
index b80e6220e6646..bc08fb9414e27 100644
--- a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
@@ -634,3 +634,34 @@ module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
 // CHECK: @.offload_sizes = private unnamed_addr constant [2 x i64] [i64 8, i64 0]
 // CHECK: @.offload_maptypes = private unnamed_addr constant [2 x i64] [i64 288, i64 288]
 // CHECK-LABEL: define void @_QPomp_target_is_device_ptr
+
+// -----
+
+module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
+  llvm.func @_QPomp_target_enter_attach(%desc : !llvm.ptr,
+                                        %pointee : !llvm.ptr) {
+    %map = omp.map.info
+        var_ptr(%pointee : !llvm.ptr, !llvm.struct<(ptr, i64)>)
+        map_clauses(attach) capture(ByRef)
+        var_ptr_ptr(%desc : !llvm.ptr, !llvm.struct<(ptr, i64)>)
+        -> !llvm.ptr {name = ""}
+    omp.target_enter_data map_entries(%map : !llvm.ptr)
+    llvm.return
+  }
+}
+
+// CHECK: @.offload_maptypes = private unnamed_addr constant [1 x i64] [i64 16384]
+// CHECK-LABEL: define void @_QPomp_target_enter_attach
+// CHECK-SAME: (ptr %[[DESC:.*]], ptr %[[POINTEE:.*]]) {
+// CHECK:         %[[BASEPTRS:.*]] = alloca [1 x ptr], align 8
+// CHECK:         %[[PTRS:.*]] = alloca [1 x ptr], align 8
+// CHECK:         %[[SIZES:.*]] = alloca [1 x i64], align 8
+// CHECK:         %[[IS_NULL:.*]] = icmp eq ptr %[[POINTEE]], null
+// CHECK-NEXT:    %[[SIZE:.*]] = select i1 %[[IS_NULL]], i64 0, i64 16
+// CHECK:         %[[BASEPTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr %[[DESC]], ptr %[[BASEPTR]], align 8
+// CHECK-NEXT:    %[[PTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[PTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr %[[POINTEE]], ptr %[[PTR]], align 8
+// CHECK-NEXT:    %[[SIZEPTR:.*]] = getelementptr inbounds [1 x i64], ptr %[[SIZES]], i32 0, i32 0
+// CHECK-NEXT:    store i64 %[[SIZE]], ptr %[[SIZEPTR]], align 8
+// CHECK:         call void @__tgt_target_data_begin_mapper(ptr @2, i64 -1, i32 1, ptr %{{.*}}, ptr %{{.*}}, ptr %{{.*}}, ptr @.offload_maptypes, ptr @.offload_mapnames, ptr null)

>From 7e6cb127cd07f8c00fc24b18ded98fada77f4e58 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Tue, 16 Jun 2026 15:58:03 +0100
Subject: [PATCH 5/7] Fix clang-format.

---
 flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index dd006e046fa74..251bf3aacd39c 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -182,8 +182,8 @@ class AutomapToTargetDataPass
             builder.getAttr<omp::VariableCaptureKindAttr>(
                 omp::VariableCaptureKind::ByRef),
             /*var_ptr_ptr=*/memOp.getMemref(),
-            /*var_ptr_ptr_type=*/TypeAttr::get(
-                fir::unwrapRefType(memOp.getMemref().getType())),
+            /*var_ptr_ptr_type=*/
+            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
             /*members=*/SmallVector<Value>{},
             /*members_index=*/ArrayAttr{},
             /*bounds=*/SmallVector<Value>{},

>From d0b4e30e6788d4b96217c42325fc1082fa450985 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 18 Jun 2026 15:45:08 +0100
Subject: [PATCH 6/7] Addressed reviewer comments.

---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 30 +++++-------------
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  | 31 ++-----------------
 .../Lower/OpenMP/declare-target-automap.f90   |  9 +++---
 .../Transforms/omp-automap-to-target-data.fir |  7 ++---
 4 files changed, 17 insertions(+), 60 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index 251bf3aacd39c..bf269b5dd3ff2 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -135,9 +135,11 @@ class AutomapToTargetDataPass
           [&](mlir::omp::ClauseMapFlags mapType) -> mlir::omp::MapInfoOp {
         mlir::Type descriptorTy =
             fir::unwrapRefType(memOp.getMemref().getType());
-        // The attach entry expects the descriptor object to already have a
-        // device mapping, but this raw object map must not be expanded as a
-        // Fortran descriptor member map.
+        // Keep the descriptor itself present. MapInfoFinalization expands this
+        // ref_ptr map and emits the attach map when the descriptor is created.
+        mapType |= omp::ClauseMapFlags::ref_ptr;
+        if (!isAlloc)
+          mapType |= omp::ClauseMapFlags::attach_never;
         return mlir::omp::MapInfoOp::create(
             builder, memOp.getLoc(), memOp.getMemref().getType(),
             memOp.getMemref(), TypeAttr::get(descriptorTy),
@@ -150,7 +152,7 @@ class AutomapToTargetDataPass
             /*members_index=*/ArrayAttr{},
             /*bounds=*/SmallVector<Value>{},
             /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
-            builder.getBoolAttr(true));
+            builder.getBoolAttr(false));
       };
 
       if (isAlloc)
@@ -173,27 +175,9 @@ class AutomapToTargetDataPass
           builder.getBoolAttr(false));
       clauses.mapVars.push_back(mapInfo);
 
-      if (isAlloc) {
-        mlir::omp::MapInfoOp attachInfo = mlir::omp::MapInfoOp::create(
-            builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
-            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-            builder.getAttr<omp::ClauseMapFlagsAttr>(
-                omp::ClauseMapFlags::attach),
-            builder.getAttr<omp::VariableCaptureKindAttr>(
-                omp::VariableCaptureKind::ByRef),
-            /*var_ptr_ptr=*/memOp.getMemref(),
-            /*var_ptr_ptr_type=*/
-            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-            /*members=*/SmallVector<Value>{},
-            /*members_index=*/ArrayAttr{},
-            /*bounds=*/SmallVector<Value>{},
-            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
-            builder.getBoolAttr(false));
-        clauses.mapVars.push_back(attachInfo);
-      } else {
+      if (!isAlloc)
         clauses.mapVars.push_back(
             createDescriptorMap(omp::ClauseMapFlags::del));
-      }
 
       isAlloc ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
               : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index 66afbed93df7a..b744dcaf3482b 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1351,34 +1351,6 @@ class MapInfoFinalizationPass
     return false;
   }
 
-  static bool isAttachMap(mlir::omp::MapInfoOp op) {
-    return (op.getMapType() & mlir::omp::ClauseMapFlags::attach) ==
-           mlir::omp::ClauseMapFlags::attach;
-  }
-
-  static bool isDescriptorOnlyMap(mlir::omp::MapInfoOp op) {
-    // A descriptor-only map keeps the descriptor object present so that a
-    // subsequent attach map can update its base address. It intentionally does
-    // not map the descriptor's pointee data, so descriptor member expansion
-    // must leave it alone.
-    if (!op.getPartialMap() || !op.getMembers().empty() ||
-        !op.getBounds().empty() || op.getVarPtrPtr() ||
-        op.getMapCaptureType() != mlir::omp::VariableCaptureKind::ByRef)
-      return false;
-
-    mlir::omp::ClauseMapFlags mapType = op.getMapType();
-    if ((mapType & mlir::omp::ClauseMapFlags::from) ==
-            mlir::omp::ClauseMapFlags::from ||
-        (mapType & mlir::omp::ClauseMapFlags::storage) ==
-            mlir::omp::ClauseMapFlags::storage)
-      return false;
-
-    return (mapType & mlir::omp::ClauseMapFlags::del) ==
-               mlir::omp::ClauseMapFlags::del ||
-           (mapType & mlir::omp::ClauseMapFlags::to) ==
-               mlir::omp::ClauseMapFlags::to;
-  }
-
   // This pass executes on omp::MapInfoOp's containing descriptor based types
   // (allocatables, pointers, assumed shape etc.) and expanding them into
   // multiple omp::MapInfoOp's for each pointer member contained within the
@@ -1640,7 +1612,8 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (!isAttachMap(op) && !isDescriptorOnlyMap(op) &&
+        if (!bitEnumContainsAll(op.getMapType(),
+                                mlir::omp::ClauseMapFlags::attach) &&
             hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
diff --git a/flang/test/Lower/OpenMP/declare-target-automap.f90 b/flang/test/Lower/OpenMP/declare-target-automap.f90
index fe46a3ab75c59..63014cbf36725 100644
--- a/flang/test/Lower/OpenMP/declare-target-automap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-automap.f90
@@ -15,13 +15,14 @@ program automap_program
 
 ! CHECK-LABEL: func.func @_QQmain()
 ! CHECK-NOT: has_device_addr
-! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[DESC:.*]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_BASE:.*]] = fir.box_offset %[[DESC]] base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
+! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DESC]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach, ref_ptr) capture(ByRef) var_ptr_ptr(%[[DESC_BASE]] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
 ! CHECK: %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 ! CHECK-NOT: has_device_addr
 ! CHECK: omp.target {{.*}}map_entries(
 ! CHECK-NOT: has_device_addr
 ! CHECK: %[[DELETE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
 ! CHECK: omp.target_exit_data map_entries(%[[DELETE_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
diff --git a/flang/test/Transforms/omp-automap-to-target-data.fir b/flang/test/Transforms/omp-automap-to-target-data.fir
index 7ed3cfa129e51..37187203369a4 100644
--- a/flang/test/Transforms/omp-automap-to-target-data.fir
+++ b/flang/test/Transforms/omp-automap-to-target-data.fir
@@ -49,15 +49,14 @@ module {
 // CHECK:           %[[BOUNDS:.*]] = omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
 // CHECK:           %[[DATA_ADDR:.*]] = fir.box_addr %[[ARR_BOXED]]
 // CHECK-NEXT:      %[[DATA_REF:.*]] = fir.convert %[[DATA_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
-// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
 // CHECK-NEXT:      %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%[[BOUNDS]]) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
-// CHECK-NEXT:      %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
-// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>)
 // CHECK:           %[[LOAD:.*]] = fir.load %[[AUTOMAP_DECL]]#0
 // CHECK:           %[[EXIT_ADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      %[[EXIT_REF:.*]] = fir.convert %[[EXIT_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
 // CHECK-NEXT:      %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[EXIT_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef)
-// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
 // CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 // CHECK-NEXT:      %[[BOXADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      fir.freemem %[[BOXADDR]]

>From dcd2effc40b450dd90b018a6ed8c85db2ff2542e Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Fri, 10 Jul 2026 14:51:46 +0100
Subject: [PATCH 7/7] Addressed reviewer comments.

---
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  |  4 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 16 +++-------
 mlir/test/Target/LLVMIR/omptarget-llvm.mlir   | 31 -------------------
 3 files changed, 6 insertions(+), 45 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index b744dcaf3482b..d4b343de988f2 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1612,9 +1612,7 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (!bitEnumContainsAll(op.getMapType(),
-                                mlir::omp::ClauseMapFlags::attach) &&
-            hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
           mlir::Operation *targetUser = getFirstTargetUser(op);
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 2dde7ab1614c0..7e355334f7be7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5719,18 +5719,16 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
-    bool isAttachMap =
-        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
-    Value pointeePtr = isRefPtrOrPteeMapWithAttach
-                           ? mapOp.getVarPtrPtr()
-                           : (isAttachMap ? mapOp.getVarPtr() : offloadPtr);
     mapData.OriginalValue.push_back(moduleTranslation.lookupValue(offloadPtr));
-    mapData.Pointers.push_back(moduleTranslation.lookupValue(pointeePtr));
+    mapData.Pointers.push_back(
+        isRefPtrOrPteeMapWithAttach
+            ? moduleTranslation.lookupValue(mapOp.getVarPtrPtr())
+            : mapData.OriginalValue.back());
 
     if (llvm::Value *refPtr =
             getRefPtrIfDeclareTarget(offloadPtr, moduleTranslation)) {
@@ -6545,10 +6543,6 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         ((convertClauseMapFlags(mapOp.getMapType()) &
           llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH) ==
          llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH);
-    bool isRefPtrOrPteeMapWithAttach =
-        isAttachMap &&
-        (bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptr) ||
-         bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptee));
 
     // If it's declare target, skip it, it's handled separately. However, if
     // it's declare target, and an attach map, we want to calculate the exact
@@ -6571,7 +6565,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
+        if (isPtrTy)
           newV = builder.CreateLoad(builder.getPtrTy(), newV);
 
         if (!offsetIdx.empty())
diff --git a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
index bc08fb9414e27..b80e6220e6646 100644
--- a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
@@ -634,34 +634,3 @@ module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
 // CHECK: @.offload_sizes = private unnamed_addr constant [2 x i64] [i64 8, i64 0]
 // CHECK: @.offload_maptypes = private unnamed_addr constant [2 x i64] [i64 288, i64 288]
 // CHECK-LABEL: define void @_QPomp_target_is_device_ptr
-
-// -----
-
-module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
-  llvm.func @_QPomp_target_enter_attach(%desc : !llvm.ptr,
-                                        %pointee : !llvm.ptr) {
-    %map = omp.map.info
-        var_ptr(%pointee : !llvm.ptr, !llvm.struct<(ptr, i64)>)
-        map_clauses(attach) capture(ByRef)
-        var_ptr_ptr(%desc : !llvm.ptr, !llvm.struct<(ptr, i64)>)
-        -> !llvm.ptr {name = ""}
-    omp.target_enter_data map_entries(%map : !llvm.ptr)
-    llvm.return
-  }
-}
-
-// CHECK: @.offload_maptypes = private unnamed_addr constant [1 x i64] [i64 16384]
-// CHECK-LABEL: define void @_QPomp_target_enter_attach
-// CHECK-SAME: (ptr %[[DESC:.*]], ptr %[[POINTEE:.*]]) {
-// CHECK:         %[[BASEPTRS:.*]] = alloca [1 x ptr], align 8
-// CHECK:         %[[PTRS:.*]] = alloca [1 x ptr], align 8
-// CHECK:         %[[SIZES:.*]] = alloca [1 x i64], align 8
-// CHECK:         %[[IS_NULL:.*]] = icmp eq ptr %[[POINTEE]], null
-// CHECK-NEXT:    %[[SIZE:.*]] = select i1 %[[IS_NULL]], i64 0, i64 16
-// CHECK:         %[[BASEPTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[BASEPTRS]], i32 0, i32 0
-// CHECK-NEXT:    store ptr %[[DESC]], ptr %[[BASEPTR]], align 8
-// CHECK-NEXT:    %[[PTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[PTRS]], i32 0, i32 0
-// CHECK-NEXT:    store ptr %[[POINTEE]], ptr %[[PTR]], align 8
-// CHECK-NEXT:    %[[SIZEPTR:.*]] = getelementptr inbounds [1 x i64], ptr %[[SIZES]], i32 0, i32 0
-// CHECK-NEXT:    store i64 %[[SIZE]], ptr %[[SIZEPTR]], align 8
-// CHECK:         call void @__tgt_target_data_begin_mapper(ptr @2, i64 -1, i32 1, ptr %{{.*}}, ptr %{{.*}}, ptr %{{.*}}, ptr @.offload_maptypes, ptr @.offload_mapnames, ptr null)


From llvmlistbot at llvm.org  Fri Jul 10 06:56:08 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Fri, 10 Jul 2026 06:56:08 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a50f9f8.4987c784.63cbc.e931@mx.google.com>


================
@@ -1612,7 +1612,9 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (!bitEnumContainsAll(op.getMapType(),
----------------
TIFitis wrote:

Thanks for pointing out, the test works without this so I've removed it.

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Fri Jul 10 06:57:43 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Fri, 10 Jul 2026 06:57:43 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a50fa57.2020f6c2.243e78.8594@mx.google.com>


================
@@ -6565,7 +6571,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy)
+        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
----------------
TIFitis wrote:

I think this is now redundant and I've removed this.

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Fri Jul 10 06:58:30 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Fri, 10 Jul 2026 06:58:30 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a50fa86.4987c784.63cbc.210a@mx.google.com>


================
@@ -5719,16 +5719,18 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
+    bool isAttachMap =
+        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
+    Value pointeePtr = isRefPtrOrPteeMapWithAttach
----------------
TIFitis wrote:

Same as above, removed change.

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Fri Jul 10 07:14:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:14:31 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a50fe47.4a342cbc.ca844.655b@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 8491 tests passed
* 559 tests skipped
* 2 tests failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir</summary>

```
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-llvmir /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-llvmir /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
# .---command stderr------------
# | Alias must point to a definition
# | ptr @ihe_decl_tgt_entry
# `-----------------------------
# error: command failed with exit status: 1
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
# .---command stderr------------
# | FileCheck error: '<stdin>' is empty.
# | FileCheck command line:  /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-all-device-types-device.mlir
# `-----------------------------
# error: command failed with exit status: 2

--

```
</details>
<details>
<summary>MLIR.Target/LLVMIR/omptarget-declare-target-llvm-host.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-llvmir /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-translate -mlir-to-llvmir /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir:136:16: error: CHECK-DAG: expected string not found in input
# |  // CHECK-DAG: @_QFEinternal_enter = internal global i32 7
# |                ^
# | <stdin>:1:1: note: scanning from here
# | ; ModuleID = 'LLVMDialectModule'
# | ^
# | <stdin>:20:1: note: possible intended match here
# | @_QFEinternal_enter = dso_local global i32 7
# | ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: ; ModuleID = 'LLVMDialectModule' 
# | dag:136'0    {                                   search range start (exclusive)
# | dag:136'1                                        error: no match found in search range
# |            2: source_filename = "LLVMDialectModule" 
# |            3: target triple = "x86_64-unknown-linux-gnu" 
# |            4:  
# |            5: %struct.__tgt_offload_entry = type { i64, i16, i16, i32, ptr, ptr, i64, i64, ptr } 
# |            6:  
# |            .
# |            .
# |            .
# |           15: @_QMtest_0Edata_int = global i32 10 
# |           16: @_QMtest_0Edata_int_clauseless_to = global i32 1 
# |           17: @_QMtest_0Edata_int_clauseless_enter = global i32 1 
# |           18: @_QMtest_0Edata_int_to = global i32 5 
# |           19: @_QMtest_0Edata_int_enter = global i32 5 
# |           20: @_QFEinternal_enter = dso_local global i32 7 
# | dag:136'2     ?                                              possible intended match
# |           21: @_QMtest_0Ept1 = global { ptr, i64, i32, i8, i8, i8, i8 } { ptr null, i64 4, i32 20180515, i8 0, i8 9, i8 1, i8 0 } 
# |           22: @_QMtest_0Ept2_tar = global i32 5 
# |           23: @_QMtest_0Earray_1d_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Earray_1d, align 8 
# |           24: @_QMtest_0Earray_2d_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Earray_2d, align 8 
# |           25: @_QMtest_0Edata_extended_link_1_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Edata_extended_link_1, align 8 
# |            .
# |            .
# |            .
# |           96: !27 = !{ptr @.offloading.entry_name.10} 
# |           97: !28 = !{ptr @.offloading.entry_name.11} 
# |           98: !29 = !{ptr @.offloading.entry_name.12} 
# |           99: !30 = !{ptr @.offloading.entry_name.13} 
# |          100: !31 = !{ptr @.offloading.entry_name.14} 
# |          101: !32 = !{ptr @.offloading.entry_name.15} 
# | dag:136'3                                             } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Fri Jul 10 07:19:29 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Fri, 10 Jul 2026 07:19:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Improve
 test-memref-dependence-check print format (PR #208656)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208656@github.com>
Message-ID: <6a50ff71.0c05e68e.1e82f1.6c4b@mx.google.com>


https://github.com/linuxlonelyeagle edited https://github.com/llvm/llvm-project/pull/208656

From llvmlistbot at llvm.org  Fri Jul 10 07:34:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:34:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Harden affine loop folding (PR #208736)
Message-ID: <llvm.org/llvm/llvm-project/pull/208736@github.com>


https://github.com/mygitljf created https://github.com/llvm/llvm-project/pull/208736

I made affine loop folding fail closed when conversion is looking at temporary IR whose value ownership cannot be proven, and I made the loop interface default treat unknown parentage conservatively. This keeps normal stable-IR folding behavior intact while avoiding unsafe conversion-time folding.
Fixes #207352 

>From 0582d9740cb10dc99afe852978e42d04138cabbe Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 22:33:17 +0000
Subject: [PATCH] [mlir] Harden affine loop folding

---
 mlir/include/mlir/Interfaces/LoopLikeInterface.td |  4 +++-
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp          |  5 ++---
 mlir/test/Transforms/test-convert-func-op.mlir    | 15 +++++++++++++++
 3 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/mlir/include/mlir/Interfaces/LoopLikeInterface.td b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
index 5fb897339ffde..d64287b6c1f0b 100644
--- a/mlir/include/mlir/Interfaces/LoopLikeInterface.td
+++ b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
@@ -62,7 +62,9 @@ def LoopLikeOpInterface : OpInterface<"LoopLikeOpInterface"> {
       /*args=*/(ins "::mlir::Value ":$value),
       /*methodBody=*/"",
       /*defaultImplementation=*/[{
-        return !$_op->isAncestor(value.getParentRegion()->getParentOp());
+        ::mlir::Region *region = value.getParentRegion();
+        ::mlir::Operation *parentOp = region ? region->getParentOp() : nullptr;
+        return parentOp && !$_op->isAncestor(parentOp);
       }]
     >,
     InterfaceMethod<[{
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..9caa5841b0e4d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2595,9 +2595,8 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
     if (val == forOp.getInductionVar())
       return {};
     if (iterArgIt == iterArgs.end()) {
-      // `val` is defined outside of the loop.
-      assert(forOp.isDefinedOutsideOfLoop(val) &&
-             "must be defined outside of the loop");
+      if (!forOp.isDefinedOutsideOfLoop(val))
+        return {};
       hasValDefinedOutsideLoop = true;
       replacements.push_back(val);
     } else {
diff --git a/mlir/test/Transforms/test-convert-func-op.mlir b/mlir/test/Transforms/test-convert-func-op.mlir
index 180f16a32991b..8587677e7bc30 100644
--- a/mlir/test/Transforms/test-convert-func-op.mlir
+++ b/mlir/test/Transforms/test-convert-func-op.mlir
@@ -38,3 +38,18 @@ func.func @byref(%arg0: !test.smpla {llvm.byref = !test.smpla}) -> !test.smpla {
 // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr {llvm.byref = !llvm.struct<(i8, i8)>}) -> !llvm.struct<(i8, i8)>
 //      CHECK: %[[LD:.*]] = llvm.load %[[ARG0]] : !llvm.ptr -> !llvm.struct<(i8, i8)>
 //      CHECK: llvm.return %[[LD]] : !llvm.struct<(i8, i8)>
+
+// -----
+
+// CHECK-LABEL: llvm.func @affine_for_iter_args
+func.func @affine_for_iter_args(%arg0: memref<10xf32>) {
+  %0 = affine.for %arg1 = 0 to 10 iter_args(%arg2 = %arg0) -> (memref<10xf32>) {
+    affine.yield %arg0 : memref<10xf32>
+  }
+  return
+}
+
+// CHECK: builtin.unrealized_conversion_cast
+// CHECK: affine.for
+// CHECK: affine.yield
+// CHECK: llvm.return


From llvmlistbot at llvm.org  Fri Jul 10 07:35:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:35:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Harden affine loop folding (PR #208736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208736@github.com>
Message-ID: <6a51032a.eb24c6ea.27aedc.88ba@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-affine

@llvm/pr-subscribers-mlir

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I made affine loop folding fail closed when conversion is looking at temporary IR whose value ownership cannot be proven, and I made the loop interface default treat unknown parentage conservatively. This keeps normal stable-IR folding behavior intact while avoiding unsafe conversion-time folding.
Fixes #<!-- -->207352 

---
Full diff: https://github.com/llvm/llvm-project/pull/208736.diff


3 Files Affected:

- (modified) mlir/include/mlir/Interfaces/LoopLikeInterface.td (+3-1) 
- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+2-3) 
- (modified) mlir/test/Transforms/test-convert-func-op.mlir (+15) 


``````````diff
diff --git a/mlir/include/mlir/Interfaces/LoopLikeInterface.td b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
index 5fb897339ffde..d64287b6c1f0b 100644
--- a/mlir/include/mlir/Interfaces/LoopLikeInterface.td
+++ b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
@@ -62,7 +62,9 @@ def LoopLikeOpInterface : OpInterface<"LoopLikeOpInterface"> {
       /*args=*/(ins "::mlir::Value ":$value),
       /*methodBody=*/"",
       /*defaultImplementation=*/[{
-        return !$_op->isAncestor(value.getParentRegion()->getParentOp());
+        ::mlir::Region *region = value.getParentRegion();
+        ::mlir::Operation *parentOp = region ? region->getParentOp() : nullptr;
+        return parentOp && !$_op->isAncestor(parentOp);
       }]
     >,
     InterfaceMethod<[{
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..9caa5841b0e4d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2595,9 +2595,8 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
     if (val == forOp.getInductionVar())
       return {};
     if (iterArgIt == iterArgs.end()) {
-      // `val` is defined outside of the loop.
-      assert(forOp.isDefinedOutsideOfLoop(val) &&
-             "must be defined outside of the loop");
+      if (!forOp.isDefinedOutsideOfLoop(val))
+        return {};
       hasValDefinedOutsideLoop = true;
       replacements.push_back(val);
     } else {
diff --git a/mlir/test/Transforms/test-convert-func-op.mlir b/mlir/test/Transforms/test-convert-func-op.mlir
index 180f16a32991b..8587677e7bc30 100644
--- a/mlir/test/Transforms/test-convert-func-op.mlir
+++ b/mlir/test/Transforms/test-convert-func-op.mlir
@@ -38,3 +38,18 @@ func.func @byref(%arg0: !test.smpla {llvm.byref = !test.smpla}) -> !test.smpla {
 // CHECK-SAME: (%[[ARG0:.*]]: !llvm.ptr {llvm.byref = !llvm.struct<(i8, i8)>}) -> !llvm.struct<(i8, i8)>
 //      CHECK: %[[LD:.*]] = llvm.load %[[ARG0]] : !llvm.ptr -> !llvm.struct<(i8, i8)>
 //      CHECK: llvm.return %[[LD]] : !llvm.struct<(i8, i8)>
+
+// -----
+
+// CHECK-LABEL: llvm.func @affine_for_iter_args
+func.func @affine_for_iter_args(%arg0: memref<10xf32>) {
+  %0 = affine.for %arg1 = 0 to 10 iter_args(%arg2 = %arg0) -> (memref<10xf32>) {
+    affine.yield %arg0 : memref<10xf32>
+  }
+  return
+}
+
+// CHECK: builtin.unrealized_conversion_cast
+// CHECK: affine.for
+// CHECK: affine.yield
+// CHECK: llvm.return

``````````

</details>


https://github.com/llvm/llvm-project/pull/208736

From llvmlistbot at llvm.org  Fri Jul 10 07:35:27 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 10 Jul 2026 07:35:27 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a51032f.3cce8983.225732.8e80@mx.google.com>


https://github.com/dzhidzhoev updated https://github.com/llvm/llvm-project/pull/187927

>From c06a34bd49cf8185c5803e44cc20245486762c17 Mon Sep 17 00:00:00 2001
From: Kristina Bessonova <kbessonova at accesssoftek.com>
Date: Mon, 13 Mar 2023 13:38:42 +0100
Subject: [PATCH 01/14] [DebugMetadata][DwarfDebug][CodeView] Support
 function-local static variables in lexical block scopes (6/7)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

RFC https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks function-local
static variables (globals in terms of LLVM IR) in DISubprogram's 'retainedNodes'.
DwarfDebug is adjusted in accordance with the aforementioned metadata change and
provided a support of static locals scoped by a lexical block.
CodeViewDebug is modified to collect global variables from DISubprogram's
'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static locals
and DwarfDebug would assert if any locally-scoped variables get placed there.

This change has already been reviewed in https://reviews.llvm.org/D144008, but I’m opening a new PR because it has been a while since it was approved.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
---
 clang/test/OpenMP/error_codegen.cpp           |  91 +++++------
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |   8 +-
 llvm/include/llvm/IR/DIBuilder.h              |   6 +-
 llvm/include/llvm/IR/DebugInfo.h              |   1 +
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  75 +++++++--
 llvm/include/llvm/IR/Metadata.h               |  22 +++
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 115 ++++++--------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp | 102 ++++++------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h   |   4 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  42 +++--
 llvm/lib/IR/DIBuilder.cpp                     |  11 +-
 llvm/lib/IR/DebugInfo.cpp                     |  21 ++-
 llvm/lib/IR/DebugInfoMetadata.cpp             |  30 +---
 llvm/lib/IR/Verifier.cpp                      |  14 +-
 .../Target/BPF/BPFAbstractMemberAccess.cpp    |   2 +-
 llvm/lib/Target/BPF/BTFDebug.cpp              |   4 +-
 llvm/lib/Transforms/Coroutines/CoroFrame.cpp  |   6 +-
 .../Instrumentation/InstrProfiling.cpp        |   1 +
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  15 ++
 llvm/lib/Transforms/Utils/Debugify.cpp        |   4 +-
 .../DIModule-fortran-external-module.ll       |   5 +-
 llvm/test/Bitcode/upgrade-cu-static-locals.ll | 108 +++++++++++++
 .../Bitcode/upgrade-cu-static-locals.ll.bc    | Bin 0 -> 2732 bytes
 .../arm64-2011-03-17-AsmPrinterCrash.ll       |   4 +-
 .../CodeGen/BPF/BTF/static-var-inited-sec.ll  |   5 +-
 .../test/CodeGen/BPF/BTF/static-var-inited.ll |   5 +-
 .../BPF/BTF/static-var-readonly-sec.ll        |   5 +-
 .../CodeGen/BPF/BTF/static-var-readonly.ll    |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var-sec.ll   |   5 +-
 llvm/test/CodeGen/BPF/BTF/static-var.ll       |   5 +-
 llvm/test/CodeGen/BPF/dwarfdump.ll            |   5 +-
 llvm/test/CodeGen/PowerPC/pr24546.ll          |   5 +-
 .../test/CodeGen/X86/dbg-distringtype-uint.ll |   5 +-
 llvm/test/DebugInfo/COFF/fortran-basic.ll     |   4 +-
 .../DebugInfo/COFF/fortran-contained-proc.ll  |   4 +-
 llvm/test/DebugInfo/COFF/global_visibility.ll |   8 +-
 llvm/test/DebugInfo/COFF/type-quals.ll        |   4 +-
 .../Generic/2009-11-05-DeadGlobalVariable.ll  |   4 +-
 .../DebugInfo/Generic/inlined-static-var.ll   |   9 +-
 .../Generic/verifier-invalid-dicompileunit.ll |  62 ++++++++
 .../Generic/verifier-invalid-disubprogram.ll  |   2 +-
 .../Symbolize/XCOFF/xcoff-symbolize-data.ll   |   5 +-
 .../test/DebugInfo/X86/DW_AT_specification.ll |   4 +-
 .../DebugInfo/X86/dbg-value-transfer-order.ll |   4 +-
 .../X86/dimodule-external-fortran.ll          |   5 +-
 llvm/test/DebugInfo/X86/distringtype.ll       |   4 +-
 .../X86/global-sra-struct-fit-segment.ll      |   4 +-
 .../global-sra-struct-part-overlap-segment.ll |   5 +-
 llvm/test/DebugInfo/X86/gnu-public-names.ll   |   5 +-
 .../DebugInfo/X86/lexical-block-static-var.ll | 145 ++++++++++++++++++
 llvm/test/DebugInfo/X86/namelist1.ll          |   4 +-
 .../DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll  | 107 +++++++------
 llvm/test/DebugInfo/duplicate_dbgvalue.ll     |   5 +-
 .../InstrProfiling/debug-info-correlate.ll    |   3 +-
 .../Transforms/Util/clone-dicompileunit.ll    |  11 +-
 .../DWARF/Inputs/ThreadLocalStorage.ll        |   4 +-
 .../Transforms/Utils/CloningTest.cpp          |  40 +++--
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       |   4 +-
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |   2 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |   2 +-
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      |   9 +-
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   |  20 ++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     |   4 +
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp  |  29 +++-
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     |   6 +-
 65 files changed, 856 insertions(+), 408 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll
 create mode 100644 llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
 create mode 100644 llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
 create mode 100644 llvm/test/DebugInfo/X86/lexical-block-static-var.ll

diff --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.
diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 389581c376a6d..11de7802ec300 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -604,7 +604,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -656,8 +656,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -695,7 +695,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =
diff --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index 9753c363166d9..04c360b57322e 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to
diff --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index cb8f6fa1b132b..dc3ff31449a2f 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processImportedEntity(const DIImportedEntity *Import);
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index e28b47d87d4e5..ef156c5e5aacf 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2080,6 +2080,10 @@ class DICompileUnit : public DIScope {
   }
 
 public:
+  constexpr static unsigned GLOBALS_IDX = 6;
+  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
+  constexpr static unsigned ENUMS_IDX = 4;
+
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2163,10 +2167,12 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(4); }
+  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(6); }
-  Metadata *getRawImportedEntities() const { return getOperand(7); }
+  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
+  Metadata *getRawImportedEntities() const {
+    return getOperand(IMPORTED_ENTITIES_IDX);
+  }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2178,14 +2184,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(4, N.get());
+    replaceOperandWith(ENUMS_IDX, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(6, N.get());
+    replaceOperandWith(GLOBALS_IDX, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(7, N.get());
+    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2292,7 +2298,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2333,7 +2339,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2462,7 +2468,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2501,19 +2507,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2525,6 +2536,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2538,12 +2551,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2575,6 +2589,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4780,6 +4805,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index f6c393c1d8c67..b2f088c2eaa93 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1564,6 +1564,28 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&RemovePred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Find first node to be removed from the tuple.
+    auto FirstToRemove = find_if(Ops, RemovePred);
+
+    // Exit if no nodes should be removed.
+    if (FirstToRemove == Ops.end())
+      return this;
+
+    // Copy all nodes before the first removed.
+    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
+    MDs.reserve(Ops.size());
+
+    // Filter the rest of the nodes
+    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
+                 [&](Metadata *N) { return !RemovePred(N); });
+
+    return get(getContext(), MDs);
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 4bc55232a6df6..c6351b77ddfd9 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -545,85 +545,62 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+      if (!LS)
+        return false;
+
+      if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(N);
+
+      return true;
+    }));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(
+          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(
+          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
+                                              DICompileUnit::ENUMS_IDX);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 4b6b76c48dcbf..b674a5b7be3dd 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1466,6 +1466,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3242,9 +3246,57 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
+    CVGlobalVariable CVGV = {DIGV, DIE};
+    GlobalVariables.emplace_back(std::move(CVGV));
+  }
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  CVGlobalVariable CVGV = {DIGV, GV};
+  VariableList->emplace_back(std::move(CVGV));
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3256,51 +3308,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 7fd2cec8c74f2..ba3cd077a4884 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -237,6 +237,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -389,6 +391,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index 72022fcddf316..44fed8d966df3 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1518,6 +1518,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1530,14 +1532,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2084,16 +2092,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2893,12 +2902,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2906,7 +2916,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&
diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 9cb5b63d885d4..d2fa4a3e1e92a 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -103,8 +103,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -932,7 +932,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 
diff --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index d27c3a9a1548a..e5ae3d034c11f 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -359,7 +363,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 5db2c86aafc3d..68b7aec5139c3 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1447,11 +1447,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1473,7 +1472,7 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 void DISubprogram::cleanupRetainedNodes() {
   // Checks if a metadata node from retainedTypes is a type not belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsAlienType = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
       return false;
@@ -1486,26 +1485,7 @@ void DISubprogram::cleanupRetainedNodes() {
     return this != TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsAlienType);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index abda69304b7e4..4ac4d2efb4d80 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1594,8 +1594,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1645,13 +1649,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,
diff --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index 81303faf77b58..b99fb53c8761a 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -208,7 +208,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }
diff --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 46a1df28b8f1d..043f6f67c3467 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -673,7 +673,7 @@ uint32_t BTFDebug::processDISubprogram(const DISubprogram *SP,
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg)
@@ -1324,7 +1324,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   std::unordered_map<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();
diff --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 83b40ee927b51..1ccb7a877b2b9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index dabd495cddd49..3ad9cad9c5e30 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1724,6 +1724,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 031ce978d21ce..21ae2d59429df 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -104,6 +104,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -350,6 +356,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return GVE &&
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.
diff --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 2923633f29d7a..bd98cefa4c490 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -393,7 +393,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -638,7 +638,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }
diff --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")
diff --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..1c57ea7cea417e1cf852ee735bd1fae97bc3be0d
GIT binary patch
literal 2732
zcma)84{Q_X761GvcP4S}E|3+Q+}+LuSXKc!JCNA1E!lBkW&_m}I)P|~9mmEgZsOR^
zU)W^Fp_i&*GRzug1IgKt%CxQvEn-xoTWqJ0rUs>`+Zc-?yGbBSDa#6*&_=brGn95(
zHPzC6-+j;Tz3=zE at 4er<CfnG|=1PQ05JD|Ve$9@(zxBTJuk+7TO{{5_=t_`AiBMFE
zP?=PNU_wv9gGZ~6StsP%FiK}QE*Vs=Em5dh`Kv7v<rZnv at v^2?N>`~@sJ>IG(+$}J
z<C^N`<-#y!JuYpXP at JpeY+Hx8_ar*qpz7(Gr{mhgqt74XvJ>F<6%L_Nm{)t{)SFEM
z?@`af5bw-5?4`@r*)Eqf>DDPqe_V}X%Ue089HE^mghHTSWm)^+-|&+}WR}&J3(TCA
zopjL)QRb78^0 at -@mozgmLQjm8PiUA$jre}k$$n&I7e>l+682__$wb*H2`>bkLSGFY
z;e>Ef2vB%;jZmKw0(QLH7wc5u-S98UY3Lg+c2dIStZZh4nH(w47ntj5W?_V$lZZNa
z#3=-mLZ4688yA8q(N)kV3q^!(1s?p|e7eBg<e2F+ee26H1cV-+tm~}M9Vuccg89Pk
zqWKt;Ni+Y)Jl1CyLIEMj3H4J#FfQ~Yg?b+E^~C^5zXczoGf^<7QHGfrDPMr&NZ7oF
znJ&=(6Obg{uMj%-2=x)XuNUvH5js;sPh6;{gnk|m)!=<T*bDz&4US0kQ+C?LX3{i7
z=F4Jx79uRvCxsp!?}xBBg&qL&BrI^1XdB$A1m+b32(8j=CuZ}{ojx&t at 5E_l-S37x
zhfaR_@`?FhH9lNV{_y=@cD}RuHQ~;k&R0KWg+GN>{=|Gnib+retS#^$sS~}tJIf(N
z!9oF#1u4hklPbs at DXhd@6cI`M2IojP5aP<@7)W$%QmL-k9v at z|?pv4Twr^KdHmMUJ
zKL at zphPh4=p~nlfw^Cvrvgrr2!%fAggp!@gTZsIaBJM|sI%N~YfVnn8d-;;0TC6H3
zGf&ve!!mQOcQ_ot4=HfCpFjZb56HTEg#ah0a}~s5oG9>QJ5SCo5larDjv8lXAmcPl
z#>IO1SB#fp#wnZOyv%edW*U?kb4!(waff_(gAcL{Z- at wiIAk$oF5dm11l;ySg#B=J
zs$xneqwDW35qI;%-^YkWUe*G2FgHTyisQ74p48CeQD)MG{UL5MOxO&`;&?HuPuTQB
zGGiiJ3!6S1Fn%aAj%Jlt+JwL!AsC2-eR!`^2t%f)1ogFD<XuF5;vnYY#H at uZaKs{<
zf~$c{XERpzx`v*PveS85EJ_LMRqfP|+8R&Aj5)`e#|MFo=*4 at JJGbADlVSwe7mKB=
zVP?|I_a&?dSi>y2u-!*u`drpHYBNJX9%f8)HscU1v%~L}3Vo1YfCn<G1`j5q&(BiC
z;utaSAnx(i>FZW at uCVe(ftdjScC~RdW<DRQ8qDg4ZH8RdbTMt1$if*-Is5ry(c_)P
z+*kkkGO_3+Kh9%I968UE^BnmP2hmOuvpfOIdwJ|mo|tnG5BTbj7F}X(-6*UCvbm*U
z#x)Gw;DYAsUhcH<lFfWkW=O;gxjmQQ_TjMofYUDY1O)Z_w~4><WFb%7v5<FCuuE1V
z3!TN`Ec=0w0%C;2P|C^Zpw*DiLg>tJQ#s3c3V4XgTiKa3?Oo&MiGp~)BsiJpZJJn@
z3J=+Z{VCya1Z>YGB&Ol4DUsDr*%~38au6}4UGudj7Yk6YrI}}=m8g$H=ncsAAZzDr
zLrV3spw7-!M~9Cc`=B{R)wG(lx~3n*H9yl^4U(<DE*a<L?`q{%UViipLSauRywP3j
z>v+ji+vf|__H+af){4SVt>54MQf*7u0k_}Pet_%t1Ve2d?VNrizfs3)^;(`ca4xsc
z74&v=w6%v at LcWe*ShUn_boci4x4D~KEpBUb4=8ZoePrD10e9TCc5laLk{;2){!m-L
z2k>vb1Oxo|0ZLh|kZ1>}%4$VayKk3YZo$&4<iG%9yFavIm<1cCQXmXfp at b;&?>MGG
zYGr@|ra-p65*ua%@^D}%4h$iW6s@#a8HGQxawF6Siu}G^fpQ9$(M7-h9a at Z_@F%-y
zG4c!gB0A7NtoSc|>WRL8$KNU}^ydgx;Exdm at W%)&C6og3GX4eUcC|9%#dcw`$jP)a
zqfVtwxCJYbs>YfL2WX=g;Q`2^6jz}V-ZqBSRY<3>qGXx8 at -w}TM|zf$6gc+qsKqHd
zr7*9!a26N&lj%eng}snhc*ufxIq~c_rB}wI2#2uM$$R;%Wy|lCvbB5blz-fPl~TTj
zp(~xI8Y4#<lSlTRm6?)98pmSrUS;`F)kVpwh}4vn-o{dhY`5TH3*Mf`UXPTWQVm5_
z6L#l7(B*#XU`VsV7*hYDYRl9(S6yHKoB9(M8s2QYpj%mYVf3xHcfM?zika<4j$VD^
z>e{2P=Z>}~A|(1baP>FfG-aj18sI>@e@`F at btjgLVJbUti@yA*<3axv3sdBlr#8(x
ziQ5jKEOJgHF7md6xIY?+;?S=|M(q`V4MaZK6F8OCpO+ZV#LS7T^dOKk4Lx(3qvu at W
zxDaJCE_w<+Hy)9ID=i`Kb7a9mLSHBwDi))7Dd+=eN at BmkbEY#f<0&9mnK>0R0olg0
zVrO!3k9yV$xplqRzeHK00={+R<Hh%VawG%MSI`^#ih9nXUMB|$27Yw}wR#ZUlB>`&
z5$Qig2|_Qxw;&7ia#><gTtlHkNkVOE8=#2!18reW>)EVp&}j|s22WjGok6>KleWR@
z at wyx844XDLxHj=ETCKLlP+$982uirN+wW?vZT56~{2c*Lu(m63An0oKY;bpN2!&nY
SHuna9huh^3ZS?!a{r&^d3xs0;

literal 0
HcmV?d00001

diff --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}
diff --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}
diff --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)
diff --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)
diff --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 1a44235553820..bf58b062a9755 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -149,7 +149,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -167,7 +167,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)
diff --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)
diff --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}
diff --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)
diff --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..fd4a366cb64c8
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,62 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 54ce1c56c6b30..52b087e540108 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -38,7 +38,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(retainedNodes: !{!0})
 
diff --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}
diff --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)
diff --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}
diff --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))
diff --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}
diff --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}
diff --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}
diff --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)
diff --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)
diff --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram
diff --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
diff --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)
diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 8c138be840ed7..cdea7740ff460 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -833,7 +833,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !31}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -849,6 +849,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !28 = !DILocalVariable(name: "ptr", scope: !3, type: !27)
     !29 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !27)
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
+    !31 = !DIGlobalVariableExpression(var: !32, expr: !DIExpression())
+    !32 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -876,8 +878,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -892,17 +896,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1187,6 +1194,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1244,14 +1260,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index d53f522e646fb..2c9a9f4225b02 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -702,7 +702,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
     OptionalParameter<"unsigned">:$scopeLine,
     OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
     OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
+    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
     OptionalArrayRefParameter<"DINodeAttr">:$annotations
   );
   let builders = [
@@ -711,7 +711,7 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
       "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
       "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
       "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
+      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
     ), [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index d4e1450d2f516..409c61566e767 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -162,7 +162,7 @@ def DISubprogramAttr : DialectAttribute<(attr
   EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
   LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
   OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
+  OptionalArrayRef<"Attribute">:$retainedNodes,
   OptionalArrayRef<"DINodeAttr">:$annotations
 )>;
 
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index dbd4b1213491b..64addcdcc5e46 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -458,7 +458,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 1a6703fd4056a..072ea2c386112 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -245,9 +245,14 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
+    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index dce5806d48034..5c6bd2d72c126 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -112,6 +112,24 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (DIGlobalVariableExpressionAttr GVE =
+                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -350,7 +368,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index b690d4820d7b0..3809d75fa4af0 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -113,6 +113,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);
diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index f4a6e5f6fc8f6..dd95c41cfc523 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1167,7 +1167,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1255,7 +1257,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1263,15 +1265,22 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+          scope = lbb->getSubprogram();
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1441,10 +1450,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 95895e3e56a05..e2a39516aae1d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -354,10 +354,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">

>From 85f5e0c0e22001d95d99a4ed175f0808a36a157a Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 15:43:17 +0200
Subject: [PATCH 02/14] Fix instrprof-debug-info-correlate-warnings.ll

---
 ...instrprof-debug-info-correlate-warnings.ll | 211 ++++++++++--------
 1 file changed, 124 insertions(+), 87 deletions(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..6e5542f164bb2 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

>From 62fcd330d64c4415a70889699b8b0ce56335d489 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 31 Mar 2026 16:00:53 +0200
Subject: [PATCH 03/14] instrprof-debug-info-correlate-warnings.ll: remove
 unnecessary change

---
 .../profile/Linux/instrprof-debug-info-correlate-warnings.ll    | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 6e5542f164bb2..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -1,7 +1,7 @@
 ; RUN: split-file %s %t
 ; RUN: %clang %t/a.ll -o %t/a.out
-; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 ; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=2 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,LIMIT
+; RUN: llvm-profdata merge --debug-info=%t/a.out %t/a.proftext --max-debug-info-correlation-warnings=0 -o %t/a.profdata 2>&1 | FileCheck %s --implicit-check-not=warning --check-prefixes=CHECK,NOLIMIT
 
 ; CHECK: warning: Incomplete DIE for function None:
 ; CHECK: warning: Incomplete DIE for function no_cfg: CFGHash=None

>From 6bc129720ed518bc73929b864c2415f3deecf0f0 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 19:21:02 +0200
Subject: [PATCH 04/14] Fix verifier-invalid-dicompileunit.ll

---
 .../test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
index fd4a366cb64c8..4535309a2f635 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -44,7 +44,7 @@
 !14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
 
 define void @foo() !dbg !15 { ret void }
-!15 = distinct !DISubprogram(file: !1)
+!15 = distinct !DISubprogram(file: !1, type: !23)
 
 ; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
 !16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
@@ -60,3 +60,6 @@ define void @foo() !dbg !15 { ret void }
 !22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
 
 ; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

>From 87f25d9d8d331dbb958ff9d73fef36240186cf5b Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 21 Jun 2026 18:22:32 +0200
Subject: [PATCH 05/14] Fix global-var-in-abstract-lexical-block.ll

---
 .../DebugInfo/Generic/global-var-in-abstract-lexical-block.ll | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

>From 11ea5e0c2b04862685f70eea39007320ec906a55 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Mon, 29 Jun 2026 22:57:29 +0200
Subject: [PATCH 06/14] Fix CloningTest.cpp after merge

---
 llvm/unittests/Transforms/Utils/CloningTest.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 0441e62bb2245..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -853,7 +853,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
-    !33 = !DIGlobalVariableExpression(var: !33, expr: !DIExpression())
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
     !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 

>From 113435c0ad404ad5aff56980398844053e107620 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Tue, 30 Jun 2026 12:35:59 +0200
Subject: [PATCH 07/14] MLIR: Do not improt GVE from retainedNodes

---
 mlir/lib/Target/LLVMIR/DebugImporter.cpp     |  5 +--
 mlir/test/Target/LLVMIR/Import/debug-info.ll | 32 ++++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 0502e004e8dbc..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -263,8 +263,9 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
   };
-  auto addGVE = [this, &retainedNodes](llvm::DIGlobalVariableExpression *GVE) {
-    retainedNodes.push_back(translateGlobalVariableExpression(GVE));
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
   };
   node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

>From ccbf8d82c18371fe2705474602c1589fbda2055f Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:28:55 +0200
Subject: [PATCH 08/14] Address review comments

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      | 26 +++++++---------
 llvm/include/llvm/IR/Metadata.h               | 21 ++++---------
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    | 30 ++++++++++---------
 llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp |  9 ++----
 llvm/lib/IR/DebugInfoMetadata.cpp             | 10 +++----
 llvm/lib/Transforms/Utils/CloneFunction.cpp   |  4 +--
 6 files changed, 42 insertions(+), 58 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index d5f5836a4be98..62523ffef3895 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2124,10 +2124,6 @@ class DICompileUnit : public DIScope {
   }
 
 public:
-  constexpr static unsigned GLOBALS_IDX = 6;
-  constexpr static unsigned IMPORTED_ENTITIES_IDX = 7;
-  constexpr static unsigned ENUMS_IDX = 4;
-
   static void get() = delete;
   static void getIfExists() = delete;
 
@@ -2215,12 +2211,10 @@ class DICompileUnit : public DIScope {
   MDString *getRawSplitDebugFilename() const {
     return getOperandAs<MDString>(3);
   }
-  Metadata *getRawEnumTypes() const { return getOperand(ENUMS_IDX); }
+  Metadata *getRawEnumTypes() const { return getOperand(4); }
   Metadata *getRawRetainedTypes() const { return getOperand(5); }
-  Metadata *getRawGlobalVariables() const { return getOperand(GLOBALS_IDX); }
-  Metadata *getRawImportedEntities() const {
-    return getOperand(IMPORTED_ENTITIES_IDX);
-  }
+  Metadata *getRawGlobalVariables() const { return getOperand(6); }
+  Metadata *getRawImportedEntities() const { return getOperand(7); }
   Metadata *getRawMacros() const { return getOperand(8); }
   MDString *getRawSysRoot() const { return getOperandAs<MDString>(9); }
   MDString *getRawSDK() const { return getOperandAs<MDString>(10); }
@@ -2231,14 +2225,14 @@ class DICompileUnit : public DIScope {
   /// DICompileUnit should be fairly rare.
   /// @{
   void replaceEnumTypes(DICompositeTypeArray N) {
-    replaceOperandWith(ENUMS_IDX, N.get());
+    replaceOperandWith(4, N.get());
   }
   void replaceRetainedTypes(DITypeArray N) { replaceOperandWith(5, N.get()); }
   void replaceGlobalVariables(DIGlobalVariableExpressionArray N) {
-    replaceOperandWith(GLOBALS_IDX, N.get());
+    replaceOperandWith(6, N.get());
   }
   void replaceImportedEntities(DIImportedEntityArray N) {
-    replaceOperandWith(IMPORTED_ENTITIES_IDX, N.get());
+    replaceOperandWith(7, N.get());
   }
   void replaceMacros(DIMacroNodeArray N) { replaceOperandWith(8, N.get()); }
   /// @}
@@ -2554,7 +2548,9 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+  void replaceRetainedNodes(MDNodeArray N) {
+    replaceOperandWith(7, N.get());
+  }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
@@ -2636,7 +2632,7 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
-  template <typename T> void cleanupRetainedNodesIf(T &&RemovePred) {
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
     MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
     // As this is expected to be called during module loading, before
     // stripping old or incorrect debug info, perform minimal sanity check.
@@ -2644,7 +2640,7 @@ class DISubprogram : public DILocalScope {
       return;
     // replaceRetainedNodes() should not re-unique DISubprogram if new list is
     // the same pointer.
-    replaceRetainedNodes(RetainedNodes->filter(RemovePred));
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
   }
 
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index a2a41b9a0e6a3..20a6030a36164 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1553,26 +1553,15 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
-  /// Filter out tuple elements that satisfy predicate.
+  /// Filter out tuple elements that do not satisfy predicate.
   /// Return this if no elements should be filtered out (without re-uniquing).
-  template <typename T> MDTuple *filter(T &&RemovePred) {
+  template <typename T> MDTuple *filter(T &&Pred) {
     ArrayRef<MDOperand> Ops = operands();
-    // Find first node to be removed from the tuple.
-    auto FirstToRemove = find_if(Ops, RemovePred);
-
     // Exit if no nodes should be removed.
-    if (FirstToRemove == Ops.end())
+    if (llvm::all_of(Ops, Pred))
       return this;
-
-    // Copy all nodes before the first removed.
-    SmallVector<Metadata *> MDs(Ops.begin(), FirstToRemove);
-    MDs.reserve(Ops.size());
-
-    // Filter the rest of the nodes
-    std::copy_if(FirstToRemove + 1, Ops.end(), std::back_inserter(MDs),
-                 [&](Metadata *N) { return !RemovePred(N); });
-
-    return get(getContext(), MDs);
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
   }
 
   static bool classof(const Metadata *MD) {
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 41c73f6dd5b78..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -561,17 +561,22 @@ class MetadataLoader::MetadataLoaderImpl {
     if (!List)
       return;
 
-    CU->replaceOperandWith(ListIndex, List->filter([&](Metadata *N) {
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
       DILocalScope *LS =
-          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(N)));
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
       if (!LS)
-        return false;
-
-      if (auto *SP = findEnclosingSubprogram(LS))
-        SPToEntities[SP].push_back(N);
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
 
-      return true;
-    }));
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
   }
 
   /// Move function-local entities from DICompileUnit's 'imports',
@@ -588,14 +593,11 @@ class MetadataLoader::MetadataLoaderImpl {
         continue;
 
       // Remove all static local variables from CU's globals list.
-      upgradeOneCULocalsList<DIGlobalVariableExpression>(
-          SPToEntities, CU, DICompileUnit::GLOBALS_IDX);
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
       // Remove all local imports from CU's imports list.
-      upgradeOneCULocalsList<DIImportedEntity>(
-          SPToEntities, CU, DICompileUnit::IMPORTED_ENTITIES_IDX);
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
       // Remove all local types from CU's enums list.
-      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU,
-                                              DICompileUnit::ENUMS_IDX);
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
 
       // Retain local entities removed from the CU in their corresponding
       // subprograms.
diff --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index ab2417aefd51d..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -3260,10 +3260,8 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
 
   // Emit constant global variables in a global symbol section.
-  if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-    CVGlobalVariable CVGV = {DIGV, DIE};
-    GlobalVariables.emplace_back(std::move(CVGV));
-  }
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
 
   const auto *GV = GlobalMap.lookup(GVE);
   if (!GV || GV->isDeclarationForLinker())
@@ -3286,8 +3284,7 @@ void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
     // Emit this global variable in a single global symbol section.
     VariableList = &GlobalVariables;
   }
-  CVGlobalVariable CVGV = {DIGV, GV};
-  VariableList->emplace_back(std::move(CVGV));
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
 }
 
 void CodeViewDebug::collectGlobalVariableInfo() {
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 02bf848b8053b..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1474,22 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](Metadata *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  cleanupRetainedNodesIf(IsAlienType);
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,
diff --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 0986bd399ffaf..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -362,8 +362,8 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
   if (DISubprogram *NewSP = NewFunc->getSubprogram())
     NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
       auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
-      return GVE &&
-             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() != NewSP;
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
     });
 
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the

>From 4237475172ed9f0ca64528c1a291eaea4aa1c386 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:10 +0200
Subject: [PATCH 09/14] Update mlir/lib/Target/LLVMIR/ModuleTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/ModuleTranslation.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a4e331c325b46..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1302,8 +1302,9 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
         } else if (auto *lbb =
-                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope))
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
           scope = lbb->getSubprogram();
+        }
 
         // Get the compile unit (scope) of the the global variable, or the
         // subprogram of the static local variable.

>From 39b303e345d8d744745fad8bef4423ed6ce75c2d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <dzhidzhoev at gmail.com>
Date: Fri, 3 Jul 2026 21:37:18 +0200
Subject: [PATCH 10/14] Update mlir/lib/Target/LLVMIR/DebugTranslation.cpp

Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>
---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fca5e0a8497eb..10ae5ef0ec646 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,7 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (DIGlobalVariableExpressionAttr GVE =
+        if (auto GVE =
                 dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 

>From 292e16924dd039876c60e17afc4fa491e448e79d Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:36:41 +0200
Subject: [PATCH 11/14] Clang-format fix

---
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  4 +-
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 52 +++++++++----------
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     | 32 ++++++------
 3 files changed, 41 insertions(+), 47 deletions(-)

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 62523ffef3895..846342bee6071 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2548,9 +2548,7 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(MDNodeArray N) {
-    replaceOperandWith(7, N.get());
-  }
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
 
   template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
     auto RetainedNodes = getRetainedNodes();
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 29b010ee37f5b..72497a5328f78 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -710,36 +710,34 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
                                       [LLVM_DIRecursiveTypeAttrInterface],
                                       "DILocalScopeAttr"> {
   let parameters = (ins
-    // DIRecursiveTypeAttrInterface specific parameters.
-    OptionalParameter<"DistinctAttr">:$recId,
-    OptionalParameter<"bool">:$isRecSelf,
-    // DISubprogramAttr specific parameters.
-    OptionalParameter<"DistinctAttr">:$id,
-    OptionalParameter<"DICompileUnitAttr">:$compileUnit,
-    OptionalParameter<"DIScopeAttr">:$scope,
-    OptionalParameter<"StringAttr">:$name,
-    OptionalParameter<"StringAttr">:$linkageName,
-    OptionalParameter<"DIFileAttr">:$file,
-    OptionalParameter<"unsigned">:$line,
-    OptionalParameter<"unsigned">:$scopeLine,
-    OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
-    OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"Attribute">:$retainedNodes,
-    OptionalArrayRefParameter<"DINodeAttr">:$annotations
-  );
-  let builders = [
-    AttrBuilder<(ins
-      "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
-      "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
-      "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
-      "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<Attribute>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
-    ), [{
+      // DIRecursiveTypeAttrInterface specific parameters.
+      OptionalParameter<"DistinctAttr">:$recId,
+      OptionalParameter<"bool">:$isRecSelf,
+      // DISubprogramAttr specific parameters.
+      OptionalParameter<"DistinctAttr">:$id,
+      OptionalParameter<"DICompileUnitAttr">:$compileUnit,
+      OptionalParameter<"DIScopeAttr">:$scope,
+      OptionalParameter<"StringAttr">:$name,
+      OptionalParameter<"StringAttr">:$linkageName,
+      OptionalParameter<"DIFileAttr">:$file,
+      OptionalParameter<"unsigned">:$line,
+      OptionalParameter<"unsigned">:$scopeLine,
+      OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
+      OptionalParameter<"DISubroutineTypeAttr">:$type,
+      OptionalArrayRefParameter<"Attribute">:$retainedNodes,
+      OptionalArrayRefParameter<"DINodeAttr">:$annotations);
+  let builders = [AttrBuilder<
+      (ins "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
+          "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
+          "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
+          "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
+          "ArrayRef<Attribute>":$retainedNodes,
+          "ArrayRef<DINodeAttr>":$annotations),
+      [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
                    subprogramFlags, type, retainedNodes, annotations);
-    }]>
-  ];
+    }]>];
   let assemblyFormat = "`<` struct(params) `>`";
   let extraClassDeclaration = [{
     /// Requirements of DIRecursiveTypeAttrInterface.
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index 0c78689fe1d6b..b25a8c8a9829e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -150,23 +150,21 @@ def DICompileUnitAttr : DialectAttribute<(attr
 // DISubprogramAttr
 //===----------------------------------------------------------------------===//
 
-def DISubprogramAttr : DialectAttribute<(attr
-  OptionalAttribute<"DistinctAttr">:$recId,
-  Bool:$isRecSelf,
-  OptionalAttribute<"DistinctAttr">:$id,
-  OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
-  OptionalAttribute<"DIScopeAttr">:$scope,
-  OptionalAttribute<"StringAttr">:$name,
-  OptionalAttribute<"StringAttr">:$linkageName,
-  OptionalAttribute<"DIFileAttr">:$file,
-  VarInt:$line,
-  VarInt:$scopeLine,
-  EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
-  LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
-  OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"Attribute">:$retainedNodes,
-  OptionalArrayRef<"DINodeAttr">:$annotations
-)>;
+def DISubprogramAttr
+    : DialectAttribute<(attr OptionalAttribute<"DistinctAttr">:$recId,
+          Bool:$isRecSelf, OptionalAttribute<"DistinctAttr">:$id,
+          OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
+          OptionalAttribute<"DIScopeAttr">:$scope,
+          OptionalAttribute<"StringAttr">:$name,
+          OptionalAttribute<"StringAttr">:$linkageName,
+          OptionalAttribute<"DIFileAttr">:$file, VarInt:$line,
+          VarInt:$scopeLine,
+          EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
+          LocalVar<"DISubprogramFlags",
+                   "(DISubprogramFlags)_rawflags">:$subprogramFlags,
+          OptionalAttribute<"DISubroutineTypeAttr">:$type,
+          OptionalArrayRef<"Attribute">:$retainedNodes,
+          OptionalArrayRef<"DINodeAttr">:$annotations)>;
 
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr

>From 3b92877dfd5c5ffbc0b1c557a78bcbf6cafce502 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Fri, 3 Jul 2026 21:44:51 +0200
Subject: [PATCH 12/14] Clang-format fix

---
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 10ae5ef0ec646..442f5b6b137e6 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -120,8 +120,7 @@ DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
     return nullptr;
   SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
       retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
-        if (auto GVE =
-                dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+        if (auto GVE = dyn_cast<DIGlobalVariableExpressionAttr>(attr))
           return translateGlobalVariableExpression(GVE);
 
         auto diAttr = dyn_cast<DINodeAttr>(attr);

>From 23a9b20861b2203633fedb5734e645e14175fcf7 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sat, 4 Jul 2026 22:58:30 +0200
Subject: [PATCH 13/14] Fix LVIRReader.cpp

---
 llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
index eb9fa3c89a44e..5726955a029ef 100644
--- a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
+++ b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
@@ -1447,13 +1447,15 @@ LVScope *LVIRReader::getOrCreateSubprogram(LVScope *Function,
   }
 
   // Check for additional retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *IE = dyn_cast<DIImportedEntity>(DN))
       constructImportedEntity(Function, IE);
     else if (const auto *TTP = dyn_cast<DITemplateTypeParameter>(DN))
       constructTemplateTypeParameter(Function, TTP);
     else if (const auto *TVP = dyn_cast<DITemplateValueParameter>(DN))
       constructTemplateValueParameter(Function, TVP);
+    else if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(DN))
+      getOrCreateVariable(GVE);
   }
 
   applySubprogramAttributes(Function, SP);

>From e34e1d4170095c62d49141f2e97bc3ecc64ea561 Mon Sep 17 00:00:00 2001
From: Vladislav Dzhidzhoev <vdzhidzhoev at accesssoftek.com>
Date: Sun, 5 Jul 2026 01:55:38 +0200
Subject: [PATCH 14/14] Fix debug-info-correlate-bitmap.ll

---
 llvm/lib/IR/DIBuilder.cpp                           | 13 ++++++-------
 .../Transforms/Instrumentation/InstrProfiling.cpp   |  1 +
 .../InstrProfiling/debug-info-correlate-bitmap.ll   |  3 ++-
 3 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 6e285a8a155fa..60854c11ff014 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -56,20 +56,19 @@ void DIBuilder::finalizeSubprogram(DISubprogram *SP) {
   if (PN == SubprogramTrackedNodes.end())
     return;
 
-  SmallVector<Metadata *, 16> RetainedNodes;
-  for (MDNode *N : PN->second) {
+  SetVector<Metadata *> RetainedNodes;
+  for (MDNode *N : llvm::concat<MDNode *>(SP->getRetainedNodes(), PN->second)) {
     // If the tracked node N was temporary, and the DIBuilder user replaced it
     // with a node that does not belong to SP or is non-local, do not add N to
     // SP's retainedNodes list.
     DILocalScope *Scope = dyn_cast_or_null<DILocalScope>(
         DISubprogram::getRawRetainedNodeScope(N));
-    if (!Scope || Scope->getSubprogram() != SP)
-      continue;
-
-    RetainedNodes.push_back(N);
+    if (Scope && Scope->getSubprogram() == SP)
+      RetainedNodes.insert(N);
   }
 
-  SP->replaceRetainedNodes(MDTuple::get(VMContext, RetainedNodes));
+  SP->replaceRetainedNodes(
+      MDTuple::get(VMContext, RetainedNodes.getArrayRef()));
 }
 
 void DIBuilder::finalize() {
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index 1edc6d8c159e9..5e58b5c95f776 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1904,6 +1904,7 @@ InstrLowerer::getOrCreateRegionBitmaps(InstrProfMCDCBitmapInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       BitmapPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
diff --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
index 29bd0037a7e8b..43a1db5b2cf4e 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
@@ -22,7 +22,7 @@ define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
   call void @llvm.instrprof.mcdc.parameters(ptr @__profn_foo, i64 12345678, i32 1)
   call void @llvm.instrprof.mcdc.tvbitmap.update(ptr @__profn_foo, i64 12345678, i32 0, ptr %mcdc.addr)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -50,6 +50,7 @@ declare void @llvm.instrprof.mcdc.tvbitmap.update(ptr, i64, i32, ptr)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, column: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram


From llvmlistbot at llvm.org  Fri Jul 10 07:36:23 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 10 Jul 2026 07:36:23 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a510367.6c5720b3.128ed.343c@mx.google.com>


https://github.com/dzhidzhoev auto_merge_enabled https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul 10 07:47:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:47:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 72b8db7 - [mlir][vector] add consistent
 stride verification to masked load/store and gather/scatter ops (#205869)
Message-ID: <6a5105f6.afd65eb9.1fb2bd.9703@mx.google.com>


Author: Federico Bruzzone
Date: 2026-07-10T15:47:12+01:00
New Revision: 72b8db717ac2a5dfb03755b592e796a61c7db60b

URL: https://github.com/llvm/llvm-project/commit/72b8db717ac2a5dfb03755b592e796a61c7db60b
DIFF: https://github.com/llvm/llvm-project/commit/72b8db717ac2a5dfb03755b592e796a61c7db60b.diff

LOG: [mlir][vector] add consistent stride verification to masked load/store and gather/scatter ops (#205869)

Extend negative stride checks to MaskedLoadOp, MaskedStoreOp, GatherOp,
and ScatterOp to match LoadOp and StoreOp behavior.

Depends on: https://github.com/llvm/llvm-project/pull/204611.

AI Disclaimer: I used AI for the tests.

@banach-space @dcaballe If you look at the latest commit, you can see
why the integration tests were failing in #204842. Those two calls were
added in this PR, I sincerely apologize to you!

---------

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
    mlir/lib/Dialect/Vector/IR/VectorOps.cpp
    mlir/test/Dialect/Vector/invalid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index a519e6d89a472..6e0134fc0cdc6 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -1946,6 +1946,9 @@ def Vector_MaskedLoadOp :
        : memref<?x?xf32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
     ```
 
+    The memref must have non-negative strides. Negative strides are not supported
+    and will trigger a verification error.
+
     An optional `alignment` attribute allows to specify the byte alignment of the
     load operation. It must be a positive power of 2. The operation must access
     memory at an address aligned to this boundary. Violating this requirement
@@ -2041,6 +2044,9 @@ def Vector_MaskedStoreOp :
       : memref<?x?xf32>, vector<16xi1>, vector<16xf32>
     ```
 
+    The memref must have non-negative strides. Negative strides are not supported
+    and will trigger a verification error.
+
     An optional `alignment` attribute allows to specify the byte alignment of the
     store operation. It must be a positive power of 2. The operation must access
     memory at an address aligned to this boundary. Violating this requirement
@@ -2135,6 +2141,9 @@ def Vector_GatherOp :
     during progressively lowering to bring other memory operations closer to
     hardware ISA support for a gather.
 
+    The memref must have non-negative strides. Negative strides are not supported
+    and will trigger a verification error.
+
     An optional `alignment` attribute allows to specify the byte alignment of the
     gather operation. It must be a positive power of 2. The operation must access
     memory at an address aligned to this boundary. Violating this requirement
@@ -2228,6 +2237,9 @@ def Vector_ScatterOp
     correspond to those of the `llvm.masked.scatter`
     [intrinsic](https://llvm.org/docs/LangRef.html#llvm-masked-scatter-intrinsics).
 
+    The memref must have non-negative strides. Negative strides are not supported
+    and will trigger a verification error.
+
     An optional `alignment` attribute allows to specify the byte alignment of the
     scatter operation. It must be a positive power of 2. The operation must access
     memory at an address aligned to this boundary. Violating this requirement

diff  --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index e0cb9923b1dd4..94f627d12752e 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6195,7 +6195,9 @@ LogicalResult vector::LoadOp::verify() {
   if (failed(verifyLoadStoreMemRefLayout(*this, resVecTy, memRefTy)))
     return failure();
 
-  // Negative strides are not supported on vector.load.
+  // Negative strides are not supported on vector.load. The lowering to LLVM
+  // emits arithmetic operations (e.g., GEP, mul) with nuw flags that assume
+  // non-negative strides to avoid undefined behavior.
   if (memref::hasNegativeStaticStride(memRefTy))
     return emitOpError("memref strides must be non-negative");
 
@@ -6245,7 +6247,9 @@ LogicalResult vector::StoreOp::verify() {
   if (failed(verifyLoadStoreMemRefLayout(*this, valueVecTy, memRefTy)))
     return failure();
 
-  // Negative strides are not supported on vector.store.
+  // Negative strides are not supported on vector.store. The lowering to LLVM
+  // emits arithmetic operations (e.g., GEP, mul) with nuw flags that assume
+  // non-negative strides to avoid undefined behavior.
   if (memref::hasNegativeStaticStride(memRefTy))
     return emitOpError("memref strides must be non-negative");
 
@@ -6293,6 +6297,12 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  // Negative strides are not supported on vector.maskedload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6353,6 +6363,12 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  // Negative strides are not supported on vector.maskedstore. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
@@ -6414,6 +6430,13 @@ LogicalResult GatherOp::verify() {
   if (!llvm::isa<MemRefType, RankedTensorType>(baseType))
     return emitOpError("requires base to be a memref or ranked tensor type");
 
+  // Negative strides are not supported on vector.gather.
+  // The lowering to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw
+  // flags that assume non-negative strides to avoid undefined behavior.
+  if (auto memRefType = dyn_cast<MemRefType>(baseType))
+    if (memref::hasNegativeStaticStride(memRefType))
+      return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, baseType, resVType, "base", "result")))
     return failure();
@@ -6528,6 +6551,13 @@ LogicalResult ScatterOp::verify() {
   if (!llvm::isa<MemRefType, RankedTensorType>(baseType))
     return emitOpError("requires base to be a memref or ranked tensor type");
 
+  // Negative strides are not supported on vector.scatter.
+  // The lowering to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw
+  // flags that assume non-negative strides to avoid undefined behavior.
+  if (auto memRefType = dyn_cast<MemRefType>(baseType))
+    if (memref::hasNegativeStaticStride(memRefType))
+      return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, baseType, valueVType, "base",
                                      "valueToStore")))
     return failure();

diff  --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index 90e8a2f2217b8..aaa55cface958 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1413,6 +1413,15 @@ func.func @maskedload_memref_mismatch(%base: memref<?xf32>, %mask: vector<16xi1>
 
 // -----
 
+func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op memref strides must be non-negative}}
+  %0 = vector.maskedload %src[%c0, %c0], %mask, %pass : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1457,6 +1466,15 @@ func.func @maskedstore_memref_mismatch(%base: memref<?xf32>, %mask: vector<16xi1
 
 // -----
 
+func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op memref strides must be non-negative}}
+  vector.maskedstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1554,6 +1572,16 @@ func.func @gather_tensor_alignment(%base: tensor<16xf32>, %indices: vector<16xi3
 
 // -----
 
+func.func @gather_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %indices: vector<16xi32>,
+                                  %mask: vector<16xi1>, %pass_thru: vector<16xf32>, %idx: index) -> vector<16xf32> {
+  // expected-error @+1 {{'vector.gather' op memref strides must be non-negative}}
+  %0 = vector.gather %src[%idx, %idx][%indices], %mask, %pass_thru
+    : memref<100x100xf32, strided<[-100, 1]>>, vector<16xi32>, vector<16xi1>, vector<16xf32> into vector<16xf32>
+  return %0 : vector<16xf32>
+}
+
+// -----
+
 func.func @scatter_to_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                              %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1642,6 +1670,16 @@ func.func @scatter_tensor_alignment(%base: tensor<?xf32>, %indices: vector<16xi3
 
 // -----
 
+func.func @scatter_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %indices: vector<16xi32>,
+                                   %mask: vector<16xi1>, %value: vector<16xf32>, %idx: index) {
+  // expected-error @+1 {{'vector.scatter' op memref strides must be non-negative}}
+  vector.scatter %src[%idx, %idx][%indices], %mask, %value
+    : memref<100x100xf32, strided<[-100, 1]>>, vector<16xi32>, vector<16xi1>, vector<16xf32>
+  return
+}
+
+// -----
+
 func.func @expand_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.expandload' op base element type ('f64') does not match result element type ('f32')}}


        

From llvmlistbot at llvm.org  Fri Jul 10 07:47:23 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 07:47:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] add consistent stride
 verification to masked load/store and gather/scatter ops (PR #205869)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205869@github.com>
Message-ID: <6a5105fb.28b163cd.361fb3.72ea@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/205869

From llvmlistbot at llvm.org  Fri Jul 10 07:49:04 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 07:49:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a510660.00da7a66.9945d.5a41@mx.google.com>


https://github.com/lhutton1 edited https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Fri Jul 10 07:49:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 07:49:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a510661.4e8829fc.45973.691e@mx.google.com>


https://github.com/lhutton1 commented:

Thanks for reworking this, overall it looks great, I just bad a couple of nitpicks!

Also it would be good if you can update the current PR title and description to reflect the new approach before merging.

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Fri Jul 10 07:49:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 07:49:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a510661.e73849ad.382316.0669@mx.google.com>


================
@@ -1500,6 +1504,38 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
   return false;
 }
 
+LogicalResult TosaValidation::validateElementType(Operation *op, Type elementTy,
----------------
lhutton1 wrote:

nit: I think it would be better to rewrite `isValidElementType` rather than introduce this additional wrapper function. That is, `isValidElementType` should return a LogicalResult. Though once we do that, it would be good to rename `isValidElementType` -> `validateValidElementType`

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Fri Jul 10 07:49:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Fri, 10 Jul 2026 07:49:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] fallback unknown type (PR
	#205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a510661.011f7afd.98bbe.3b05@mx.google.com>


================
@@ -25,3 +25,17 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
        (tensor<1x32x32x8xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
+  tosa.variable @dynamic_var : tensor<2x?xf64>
+}
+
+// -----
+
+module {
----------------
lhutton1 wrote:

This is testing the same functionality as above, I think we can remove it

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Fri Jul 10 07:51:17 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 07:51:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a5106e5.fbcacea0.126c76.2a2e@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you for the fix!

https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Fri Jul 10 07:51:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:51:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] b12c64e - [mlir][vector] Fix crash on
 untraceable masks in getCompressedMaskOp (#207299)
Message-ID: <6a5106ec.12f56480.c7f3a.53ae@mx.google.com>


Author: Marquis
Date: 2026-07-10T15:51:19+01:00
New Revision: b12c64e0a26e9d0a45bc013bb1740e16100a700c

URL: https://github.com/llvm/llvm-project/commit/b12c64e0a26e9d0a45bc013bb1740e16100a700c
DIFF: https://github.com/llvm/llvm-project/commit/b12c64e0a26e9d0a45bc013bb1740e16100a700c.diff

LOG: [mlir][vector] Fix crash on untraceable masks in getCompressedMaskOp (#207299)

The loop tracing a mask back to its creation op assumed the chain always
ends at a create_mask/constant_mask/arith.constant, a block-argument
mask made it fall through to isa<> on a null op (segfault), and any
other unhandled defining op spun it forever.

Fixes #206928

Claude Fable 5 did assist with this.

Added: 
    

Modified: 
    mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
    mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..e237ae0568ae4 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -89,20 +89,26 @@ static FailureOr<Operation *> getCompressedMaskOp(OpBuilder &rewriter,
       numSrcElemsPerDest;
 
   Operation *maskOp = mask.getDefiningOp();
-  SmallVector<vector::ExtractOp, 2> extractOps;
+  // Chain of `vector.extract` ops that lead to the op that created the mask.
   // TODO: add support to `vector.broadcast`.
-  // Finding the mask creation operation.
+  SmallVector<vector::ExtractOp, 2> extractOps;
+
+  // Trace the mask back to its creation op, looking through `vector.extract`
+  // ops. Any other defining op is unsupported.
   while (maskOp &&
          !isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
              maskOp)) {
-    if (auto extractOp = dyn_cast<vector::ExtractOp>(maskOp)) {
-      maskOp = extractOp.getSource().getDefiningOp();
-      extractOps.push_back(extractOp);
-    }
+    auto extractOp = dyn_cast<vector::ExtractOp>(maskOp);
+    if (!extractOp)
+      return failure();
+    maskOp = extractOp.getSource().getDefiningOp();
+    extractOps.push_back(extractOp);
   }
 
-  if (!isa<arith::ConstantOp, vector::CreateMaskOp, vector::ConstantMaskOp>(
-          maskOp))
+  // `maskOp` is null when the mask is a block argument, which has no defining
+  // op.
+  if (!isa_and_present<arith::ConstantOp, vector::CreateMaskOp,
+                       vector::ConstantMaskOp>(maskOp))
     return failure();
 
   // Computing the "compressed" mask. All the emulation logic (i.e. computing

diff  --git a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
index a5a6fc4acfe10..d3abccf641914 100644
--- a/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
+++ b/mlir/test/Dialect/Vector/emulate-narrow-type-unsupported.mlir
@@ -7,14 +7,14 @@
 /// vector.load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4xi8> {
+func.func @negative_vector_load_2d_i8(%arg1: index, %arg2: index) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %1 = vector.load %0[%arg1, %arg2] : memref<3x4xi8>, vector<2x4xi8>
     return %1 : vector<2x4xi8>
 }
 
 // No support for loading 2D vectors - expect no conversions
-//  CHECK-LABEL: func @vector_load_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_load_2d_i8
 //        CHECK:   memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
@@ -24,14 +24,14 @@ func.func @vector_load_2d_i8_negative(%arg1: index, %arg2: index) -> vector<2x4x
 /// vector.transfer_read
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> vector<2x8xi4> {
+func.func @negative_vector_transfer_read_2d_i4(%arg1: index, %arg2: index) -> vector<2x8xi4> {
     %c0 = arith.constant 0 : i4
     %0 = memref.alloc() : memref<3x8xi4>
     %1 = vector.transfer_read %0[%arg1, %arg2], %c0 {in_bounds = [true, true]} :
       memref<3x8xi4>, vector<2x8xi4>
     return %1 : vector<2x8xi4>
 }
-//  CHECK-LABEL: func @vector_transfer_read_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_transfer_read_2d_i4
 //        CHECK:  memref.alloc() : memref<3x8xi4>
 //    CHECK-NOT: i32
 
@@ -41,7 +41,7 @@ func.func @vector_transfer_read_2d_i4_negative(%arg1: index, %arg2: index) -> ve
 /// vector.maskedload
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
+func.func @negative_vector_maskedload_2d_i8(%arg1: index, %arg2: index, %arg3: index, %passthru: vector<2x4xi8>) -> vector<2x4xi8> {
     %0 = memref.alloc() : memref<3x4xi8>
     %mask = vector.create_mask %arg3, %arg3 : vector<2x4xi1>
     %1 = vector.maskedload %0[%arg1, %arg2], %mask, %passthru :
@@ -49,17 +49,31 @@ func.func @vector_maskedload_2d_i8_negative(%arg1: index, %arg2: index, %arg3: i
     return %1 : vector<2x4xi8>
 }
 
-//  CHECK-LABEL: func @vector_maskedload_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedload_2d_i8
 //        CHECK:  memref.alloc() : memref<3x4xi8>
 //    CHECK-NOT: i32
 
 // -----
 
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the load is left unconverted.
+
+func.func @negative_vector_maskedload_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %passthru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  %0 = vector.maskedload %arg0[%c0], %mask, %passthru : memref<?xi8>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+//  CHECK-LABEL: func @negative_vector_maskedload_i8_block_arg_mask
+//        CHECK: vector.maskedload
+
+// -----
+
 ///----------------------------------------------------------------------------------------
 /// vector.extract -> vector.masked_load
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x16xi4> {
+func.func @negative_vector_extract_maskedload_2d_i4(%arg1: index) -> vector<8x8x16xi4> {
     %0 = memref.alloc() : memref<8x8x16xi4>
     %c0 = arith.constant 0 : index
     %c16 = arith.constant 16 : index
@@ -73,7 +87,7 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
     return %63 : vector<8x8x16xi4>
 }
 
-//  CHECK-LABEL: func @vector_extract_maskedload_2d_i4_negative
+//  CHECK-LABEL: func @negative_vector_extract_maskedload_2d_i4
 //        CHECK: memref.alloc() : memref<8x8x16xi4>
 //    CHECK-NOT: i32
 
@@ -83,13 +97,13 @@ func.func @vector_extract_maskedload_2d_i4_negative(%arg1: index) -> vector<8x8x
 /// vector.store
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
+func.func @negative_vector_store_2d_i8(%arg0: vector<2x8xi8>, %arg1: index, %arg2: index) {
     %0 = memref.alloc() : memref<4x8xi8>
     vector.store %arg0, %0[%arg1, %arg2] :memref<4x8xi8>, vector<2x8xi8>
     return
 }
 
-//  CHECK-LABEL: func @vector_store_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_store_2d_i8
 //        CHECK: memref.alloc() : memref<4x8xi8>
 //    CHECK-NOT: i32
 
@@ -99,13 +113,27 @@ func.func @vector_store_2d_i8_negative(%arg0: vector<2x8xi8>, %arg1: index, %arg
 /// vector.maskedstore
 ///----------------------------------------------------------------------------------------
 
-func.func @vector_maskedstore_2d_i8_negative(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
+func.func @negative_vector_maskedstore_2d_i8(%arg0: index, %arg1: index, %arg2: index, %value: vector<2x8xi8>) {
   %0 = memref.alloc() : memref<3x8xi8>
   %mask = vector.create_mask %arg2, %arg2 : vector<2x8xi1>
   vector.maskedstore %0[%arg0, %arg1], %mask, %value : memref<3x8xi8>, vector<2x8xi1>, vector<2x8xi8>
   return
 }
 
-//  CHECK-LABEL: func @vector_maskedstore_2d_i8_negative
+//  CHECK-LABEL: func @negative_vector_maskedstore_2d_i8
 //        CHECK: memref.alloc() : memref<3x8xi8>
 //    CHECK-NOT: i32
+
+// -----
+
+// The mask is a block argument and cannot be traced back to a supported
+// mask-creation op, so the store is left unconverted.
+
+func.func @negative_vector_maskedstore_i8_block_arg_mask(%arg0: memref<?xi8>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  vector.maskedstore %arg0[%c0], %mask, %value : memref<?xi8>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+//  CHECK-LABEL: func @negative_vector_maskedstore_i8_block_arg_mask
+//        CHECK: vector.maskedstore


        

From llvmlistbot at llvm.org  Fri Jul 10 07:51:27 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 07:51:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a5106ef.afd65eb9.1fb2bd.d3d9@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Fri Jul 10 07:51:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:51:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix crash on untraceable masks
 in getCompressedMaskOp (PR #207299)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207299@github.com>
Message-ID: <6a510702.39be5633.14e933.84a2@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@marquisburg Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/207299

From llvmlistbot at llvm.org  Fri Jul 10 07:52:11 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 07:52:11 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a51071b.d0742bf5.11efd3.3e83@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/208185

From llvmlistbot at llvm.org  Fri Jul 10 07:52:12 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 07:52:12 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix narrow type mask crash (PR #208185)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208185@github.com>
Message-ID: <6a51071c.91d695f4.18551.da39@mx.google.com>


banach-space wrote:

Looks like a duplicate of https://github.com/llvm/llvm-project/pull/207299, hence closing. Please re-open if I missed something.

https://github.com/llvm/llvm-project/pull/208185

From llvmlistbot at llvm.org  Fri Jul 10 07:55:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:55:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve live branch arguments (PR
	#208742)
Message-ID: <llvm.org/llvm/llvm-project/pull/208742@github.com>


https://github.com/mygitljf created https://github.com/llvm/llvm-project/pull/208742

I found a case where deadness from one incoming edge was being treated as if the shared successor argument itself was dead. This change keeps that edge-local information from overriding the liveness result, while still allowing dead branch inputs to be cleaned up.
Fixes #206916 

>From 0be1d90d51a20bb0d195a117d731feea625e150a Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 22:54:54 +0000
Subject: [PATCH] [mlir] Preserve live branch arguments

---
 mlir/lib/Transforms/RemoveDeadValues.cpp     |  5 +++
 mlir/test/Transforms/remove-dead-values.mlir | 34 ++++++++++++++++++++
 2 files changed, 39 insertions(+)

diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..9521db2d1aa57 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -511,6 +511,11 @@ static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
     // Do (2)
     BitVector successorNonLive =
         markLives(operandValues, nonLiveSet, la).flip();
+    // A dead incoming edge does not make the successor argument globally dead.
+    for (auto [argIndex, arg] : llvm::enumerate(successorBlock->getArguments()))
+      if (successorNonLive[argIndex] && hasLive(arg, nonLiveSet, la))
+        successorNonLive.reset(argIndex);
+
     collectNonLiveValues(nonLiveSet, successorBlock->getArguments(),
                          successorNonLive);
 
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..437509875df36 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -850,6 +850,40 @@ func.func @replace_dead_operation_results_with_poison(%0: vector<1xindex>) -> ve
 
 // -----
 
+// CHECK-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+// CHECK-CANONICALIZE-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred(%arg0: memref<?x8x16xf32>, %arg1: memref<4x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %cst = arith.constant 1.000000e+00 : f32
+  cf.br ^bb1(%c0 : index)
+^bb1(%0: index):
+  %1 = arith.cmpi slt, %0, %c4 : index
+  cf.cond_br %1, ^bb2, ^bb7
+^bb2:
+  cf.br ^bb3(%c0 : index)
+^bb3(%2: index):
+  %3 = arith.cmpi slt, %2, %c4 : index
+  cf.cond_br %3, ^bb4, ^bb6
+^bb4:
+  %4 = memref.load %arg0[%0, %2, %c1] : memref<?x8x16xf32>
+  %5 = arith.addf %4, %cst : f32
+  memref.store %5, %arg1[%0, %2] : memref<4x?xf32>
+  %6 = arith.addi %2, %c1 : index
+  cf.br ^bb3(%6 : index)
+^bb5:
+  %7 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%7 : index)
+^bb6:
+  %8 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%8 : index)
+^bb7:
+  return
+}
+
+// -----
+
 // Verify that a referenced by a non-call op (spirv.EntryPoint),
 // while still having another usual call site is preserved as-is
 // since the pass cannot analyse non-call users.


From llvmlistbot at llvm.org  Fri Jul 10 07:56:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:56:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve live branch arguments (PR
	#208742)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208742@github.com>
Message-ID: <6a510815.d84a4299.3853e5.d5c5@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I found a case where deadness from one incoming edge was being treated as if the shared successor argument itself was dead. This change keeps that edge-local information from overriding the liveness result, while still allowing dead branch inputs to be cleaned up.
Fixes #<!-- -->206916 

---
Full diff: https://github.com/llvm/llvm-project/pull/208742.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+5) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+34) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..9521db2d1aa57 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -511,6 +511,11 @@ static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
     // Do (2)
     BitVector successorNonLive =
         markLives(operandValues, nonLiveSet, la).flip();
+    // A dead incoming edge does not make the successor argument globally dead.
+    for (auto [argIndex, arg] : llvm::enumerate(successorBlock->getArguments()))
+      if (successorNonLive[argIndex] && hasLive(arg, nonLiveSet, la))
+        successorNonLive.reset(argIndex);
+
     collectNonLiveValues(nonLiveSet, successorBlock->getArguments(),
                          successorNonLive);
 
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..437509875df36 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -850,6 +850,40 @@ func.func @replace_dead_operation_results_with_poison(%0: vector<1xindex>) -> ve
 
 // -----
 
+// CHECK-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+// CHECK-CANONICALIZE-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred(%arg0: memref<?x8x16xf32>, %arg1: memref<4x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %cst = arith.constant 1.000000e+00 : f32
+  cf.br ^bb1(%c0 : index)
+^bb1(%0: index):
+  %1 = arith.cmpi slt, %0, %c4 : index
+  cf.cond_br %1, ^bb2, ^bb7
+^bb2:
+  cf.br ^bb3(%c0 : index)
+^bb3(%2: index):
+  %3 = arith.cmpi slt, %2, %c4 : index
+  cf.cond_br %3, ^bb4, ^bb6
+^bb4:
+  %4 = memref.load %arg0[%0, %2, %c1] : memref<?x8x16xf32>
+  %5 = arith.addf %4, %cst : f32
+  memref.store %5, %arg1[%0, %2] : memref<4x?xf32>
+  %6 = arith.addi %2, %c1 : index
+  cf.br ^bb3(%6 : index)
+^bb5:
+  %7 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%7 : index)
+^bb6:
+  %8 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%8 : index)
+^bb7:
+  return
+}
+
+// -----
+
 // Verify that a referenced by a non-call op (spirv.EntryPoint),
 // while still having another usual call site is preserved as-is
 // since the pass cannot analyse non-call users.

``````````

</details>


https://github.com/llvm/llvm-project/pull/208742

From llvmlistbot at llvm.org  Fri Jul 10 07:56:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 07:56:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve live branch arguments (PR
	#208742)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208742@github.com>
Message-ID: <6a510816.e0660744.28df04.3261@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I found a case where deadness from one incoming edge was being treated as if the shared successor argument itself was dead. This change keeps that edge-local information from overriding the liveness result, while still allowing dead branch inputs to be cleaned up.
Fixes #<!-- -->206916 

---
Full diff: https://github.com/llvm/llvm-project/pull/208742.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+5) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+34) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index f0a210a2ededb..9521db2d1aa57 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -511,6 +511,11 @@ static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
     // Do (2)
     BitVector successorNonLive =
         markLives(operandValues, nonLiveSet, la).flip();
+    // A dead incoming edge does not make the successor argument globally dead.
+    for (auto [argIndex, arg] : llvm::enumerate(successorBlock->getArguments()))
+      if (successorNonLive[argIndex] && hasLive(arg, nonLiveSet, la))
+        successorNonLive.reset(argIndex);
+
     collectNonLiveValues(nonLiveSet, successorBlock->getArguments(),
                          successorNonLive);
 
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 64088ce15cd48..437509875df36 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -850,6 +850,40 @@ func.func @replace_dead_operation_results_with_poison(%0: vector<1xindex>) -> ve
 
 // -----
 
+// CHECK-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+// CHECK-CANONICALIZE-LABEL: func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred
+func.func @dont_mark_live_successor_arg_dead_from_unreachable_pred(%arg0: memref<?x8x16xf32>, %arg1: memref<4x?xf32>) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %cst = arith.constant 1.000000e+00 : f32
+  cf.br ^bb1(%c0 : index)
+^bb1(%0: index):
+  %1 = arith.cmpi slt, %0, %c4 : index
+  cf.cond_br %1, ^bb2, ^bb7
+^bb2:
+  cf.br ^bb3(%c0 : index)
+^bb3(%2: index):
+  %3 = arith.cmpi slt, %2, %c4 : index
+  cf.cond_br %3, ^bb4, ^bb6
+^bb4:
+  %4 = memref.load %arg0[%0, %2, %c1] : memref<?x8x16xf32>
+  %5 = arith.addf %4, %cst : f32
+  memref.store %5, %arg1[%0, %2] : memref<4x?xf32>
+  %6 = arith.addi %2, %c1 : index
+  cf.br ^bb3(%6 : index)
+^bb5:
+  %7 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%7 : index)
+^bb6:
+  %8 = arith.addi %0, %c1 : index
+  cf.br ^bb1(%8 : index)
+^bb7:
+  return
+}
+
+// -----
+
 // Verify that a referenced by a non-call op (spirv.EntryPoint),
 // while still having another usual call site is preserved as-is
 // since the pass cannot analyse non-call users.

``````````

</details>


https://github.com/llvm/llvm-project/pull/208742

From llvmlistbot at llvm.org  Fri Jul 10 08:14:18 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Fri, 10 Jul 2026 08:14:18 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a510c4a.160324b0.195192.4a1b@mx.google.com>


https://github.com/TIFitis updated https://github.com/llvm/llvm-project/pull/162501

>From 841555fbb53897be40c06527858561f2949f4e9b Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Wed, 8 Oct 2025 16:51:51 +0100
Subject: [PATCH 1/8] [Flang][OpenMP] Fix Fortran automap handling

	- Move automapped Fortran descriptor mappings from map() to has_device_addr in target regions, updating block args and uses accordingly.
	- In libomptarget, detect CFI descriptors during pointer attachment and compute the correct descriptor size to transfer full metadata (including bounds).
	- Resolves lost bounds for automapped Fortran arrays on device; no change for C/C++.

This fixes test offload/test/offloading/fortran/declare-target-automap.f90 reported broken in #161265.
---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 102 ++++++++++++++++++
 offload/libomptarget/omptarget.cpp            |  32 ++++++
 .../fortran/declare-target-automap.f90        |   3 -
 3 files changed, 134 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index eeb08ebf51191..5159158f53339 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -18,8 +18,12 @@
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Operation.h"
+#include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
+#include "llvm/Frontend/OpenMP/OMPConstants.h"
+#include <algorithm>
+
 namespace flangomp {
 #define GEN_PASS_DEF_AUTOMAPTOTARGETDATAPASS
 #include "flang/Optimizer/OpenMP/Passes.h.inc"
@@ -150,6 +154,104 @@ class AutomapToTargetDataPass
           addMapInfo(globalOp, loadOp);
       }
     }
+
+    // Move automapped descriptors from map() to has_device_addr in target ops.
+    auto originatesFromAutomapGlobal = [&](mlir::Value varPtr) -> bool {
+      if (auto decl = mlir::dyn_cast_or_null<hlfir::DeclareOp>(
+              varPtr.getDefiningOp())) {
+        if (auto addrOp = mlir::dyn_cast_or_null<fir::AddrOfOp>(
+                decl.getMemref().getDefiningOp())) {
+          if (auto g =
+                  mlir::SymbolTable::lookupNearestSymbolFrom<fir::GlobalOp>(
+                      decl, addrOp.getSymbol()))
+            return automapGlobals.contains(g);
+        }
+      }
+      return false;
+    };
+
+    module.walk([&](mlir::omp::TargetOp target) {
+      // Collect candidates to move: descriptor maps of automapped globals.
+      llvm::SmallVector<mlir::Value> newMapOps;
+      llvm::SmallVector<unsigned> removedIndices;
+      llvm::SmallVector<mlir::Value> movedToHDA;
+      llvm::SmallVector<mlir::BlockArgument> oldMapArgsForMoved;
+
+      auto mapRange = target.getMapVars();
+      newMapOps.reserve(mapRange.size());
+
+      auto argIface = llvm::dyn_cast<mlir::omp::BlockArgOpenMPOpInterface>(
+          target.getOperation());
+      llvm::ArrayRef<mlir::BlockArgument> mapBlockArgs =
+          argIface.getMapBlockArgs();
+
+      for (auto [idx, mapVal] : llvm::enumerate(mapRange)) {
+        auto mapOp =
+            mlir::dyn_cast<mlir::omp::MapInfoOp>(mapVal.getDefiningOp());
+        if (!mapOp) {
+          newMapOps.push_back(mapVal);
+          continue;
+        }
+
+        mlir::Type varTy = fir::unwrapRefType(mapOp.getVarType());
+        bool isDescriptor = mlir::isa<fir::BaseBoxType>(varTy);
+        if (isDescriptor && originatesFromAutomapGlobal(mapOp.getVarPtr())) {
+          movedToHDA.push_back(mapVal);
+          removedIndices.push_back(idx);
+          oldMapArgsForMoved.push_back(mapBlockArgs[idx]);
+        } else {
+          newMapOps.push_back(mapVal);
+        }
+      }
+
+      if (movedToHDA.empty())
+        return;
+
+      // Update map vars to exclude moved entries.
+      mlir::MutableOperandRange mapMutable = target.getMapVarsMutable();
+      mapMutable.assign(newMapOps);
+
+      // Append moved entries to has_device_addr and insert corresponding block
+      // arguments.
+      mlir::MutableOperandRange hdaMutable =
+          target.getHasDeviceAddrVarsMutable();
+      llvm::SmallVector<mlir::Value> newHDA;
+      newHDA.reserve(hdaMutable.size() + movedToHDA.size());
+      llvm::for_each(hdaMutable.getAsOperandRange(),
+                     [&](mlir::Value v) { newHDA.push_back(v); });
+
+      unsigned hdaStart = argIface.getHasDeviceAddrBlockArgsStart();
+      unsigned oldHdaCount = argIface.numHasDeviceAddrBlockArgs();
+      llvm::SmallVector<mlir::BlockArgument> newHDAArgsForMoved;
+      unsigned insertIndex = hdaStart + oldHdaCount;
+      for (mlir::Value v : movedToHDA) {
+        newHDA.push_back(v);
+        target->getRegion(0).insertArgument(insertIndex, v.getType(),
+                                            v.getLoc());
+        // Capture the newly inserted block argument.
+        newHDAArgsForMoved.push_back(
+            target->getRegion(0).getArgument(insertIndex));
+        insertIndex++;
+      }
+      hdaMutable.assign(newHDA);
+
+      // Redirect uses in the region: replace old map block args with the
+      // corresponding new has_device_addr block args.
+      for (auto [oldArg, newArg] :
+           llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
+        oldArg.replaceAllUsesWith(newArg);
+
+      // Finally, erase corresponding map block arguments (descending order).
+      unsigned mapStart = argIface.getMapBlockArgsStart();
+      // Convert indices to absolute argument numbers before erasing.
+      llvm::SmallVector<unsigned> absArgNos;
+      absArgNos.reserve(removedIndices.size());
+      for (unsigned idx : removedIndices)
+        absArgNos.push_back(mapStart + idx);
+      std::sort(absArgNos.begin(), absArgNos.end(), std::greater<>());
+      for (unsigned absNo : absArgNos)
+        target->getRegion(0).eraseArgument(absNo);
+    });
   }
 };
 } // namespace
diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 27ee173be06f9..750ef047aa006 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -33,6 +33,9 @@
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
 #include "llvm/Object/ObjectFile.h"
 
+#include "flang/ISO_Fortran_binding.h"
+
+#include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <vector>
@@ -381,6 +384,34 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
+// Fortran pointer attachments treated descriptors as plain pointers, so
+// automapped arrays lose their declared bounds on the device. Recognize
+// CFI descriptors to compute their actual size before copying, ensuring the
+// full descriptor (including bounds) is transferred during attachment.
+static int64_t getFortranDescriptorSize(void **HstPtrAddr,
+                                        int64_t ReportedSize) {
+  constexpr int64_t VoidPtrSize = sizeof(void *);
+
+  if (!HstPtrAddr || ReportedSize > VoidPtrSize)
+    return ReportedSize;
+
+  const CFI_cdesc_t *Desc = reinterpret_cast<const CFI_cdesc_t *>(HstPtrAddr);
+
+  if (Desc->version != CFI_VERSION)
+    return ReportedSize;
+
+  if (Desc->rank > CFI_MAX_RANK)
+    return ReportedSize;
+
+  const char *RawDesc = reinterpret_cast<const char *>(Desc);
+  const char *DimsAddr = reinterpret_cast<const char *>(&Desc->dim);
+  size_t HeaderBytes = static_cast<size_t>(DimsAddr - RawDesc);
+  size_t DimsBytes = static_cast<size_t>(Desc->rank) * sizeof(CFI_dim_t);
+  size_t TotalBytes = HeaderBytes + DimsBytes;
+
+  return std::max<int64_t>(ReportedSize, static_cast<int64_t>(TotalBytes));
+}
+
 /// Utility function to perform a pointer attachment operation.
 ///
 /// For something like:
@@ -448,6 +479,7 @@ static int performPointerAttachment(DeviceTy &Device, AsyncInfoTy &AsyncInfo,
          "Need a valid pointer entry to perform pointer-attachment");
 
   constexpr int64_t VoidPtrSize = sizeof(void *);
+  HstPtrSize = getFortranDescriptorSize(HstPtrAddr, HstPtrSize);
   assert(HstPtrSize >= VoidPtrSize && "PointerSize is too small");
 
   void *TgtPteeBase =
diff --git a/offload/test/offloading/fortran/declare-target-automap.f90 b/offload/test/offloading/fortran/declare-target-automap.f90
index b44c0b2815274..b9c2d34c834fa 100644
--- a/offload/test/offloading/fortran/declare-target-automap.f90
+++ b/offload/test/offloading/fortran/declare-target-automap.f90
@@ -1,9 +1,6 @@
 !Offloading test for AUTOMAP modifier in declare target enter
 ! REQUIRES: flang, amdgpu
 
-! FIXME: https://github.com/llvm/llvm-project/issues/161265
-! XFAIL: amdgpu
-
 ! RUN: %libomptarget-compile-fortran-run-and-check-generic
 program automap_program
    use iso_c_binding, only: c_loc

>From 5d0d9a77cb2dc1cb64ba42420648ec5e68c79689 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 9 Oct 2025 16:30:31 +0100
Subject: [PATCH 2/8] Address copilot suggestions.

---
 flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp | 5 ++++-
 offload/libomptarget/omptarget.cpp                 | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index 5159158f53339..f0e0eefa7e220 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -241,7 +241,10 @@ class AutomapToTargetDataPass
            llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
         oldArg.replaceAllUsesWith(newArg);
 
-      // Finally, erase corresponding map block arguments (descending order).
+      // Finally, erase corresponding map block arguments in descending order.
+      // Descending order is necessary to avoid index invalidation: erasing
+      // arguments from highest to lowest index ensures that earlier erases do
+      // not shift the indices of arguments yet to be erased.
       unsigned mapStart = argIface.getMapBlockArgsStart();
       // Convert indices to absolute argument numbers before erasing.
       llvm::SmallVector<unsigned> absArgNos;
diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 750ef047aa006..85e1641f3e58e 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -384,7 +384,7 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
-// Fortran pointer attachments treated descriptors as plain pointers, so
+// Fortran pointer attachments treat descriptors as plain pointers, so
 // automapped arrays lose their declared bounds on the device. Recognize
 // CFI descriptors to compute their actual size before copying, ensuring the
 // full descriptor (including bounds) is transferred during attachment.

>From a8f64757d19604cdac79d0e4b983e812316e0cc9 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 30 Oct 2025 14:46:11 +0000
Subject: [PATCH 3/8] Remove omptarget changes.

---
 offload/libomptarget/omptarget.cpp | 32 ------------------------------
 1 file changed, 32 deletions(-)

diff --git a/offload/libomptarget/omptarget.cpp b/offload/libomptarget/omptarget.cpp
index 85e1641f3e58e..27ee173be06f9 100644
--- a/offload/libomptarget/omptarget.cpp
+++ b/offload/libomptarget/omptarget.cpp
@@ -33,9 +33,6 @@
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
 #include "llvm/Object/ObjectFile.h"
 
-#include "flang/ISO_Fortran_binding.h"
-
-#include <algorithm>
 #include <cassert>
 #include <cstdint>
 #include <vector>
@@ -384,34 +381,6 @@ static void *calculateTargetPointeeBase(void *HstPteeBase, void *HstPteeBegin,
   return TgtPteeBase;
 }
 
-// Fortran pointer attachments treat descriptors as plain pointers, so
-// automapped arrays lose their declared bounds on the device. Recognize
-// CFI descriptors to compute their actual size before copying, ensuring the
-// full descriptor (including bounds) is transferred during attachment.
-static int64_t getFortranDescriptorSize(void **HstPtrAddr,
-                                        int64_t ReportedSize) {
-  constexpr int64_t VoidPtrSize = sizeof(void *);
-
-  if (!HstPtrAddr || ReportedSize > VoidPtrSize)
-    return ReportedSize;
-
-  const CFI_cdesc_t *Desc = reinterpret_cast<const CFI_cdesc_t *>(HstPtrAddr);
-
-  if (Desc->version != CFI_VERSION)
-    return ReportedSize;
-
-  if (Desc->rank > CFI_MAX_RANK)
-    return ReportedSize;
-
-  const char *RawDesc = reinterpret_cast<const char *>(Desc);
-  const char *DimsAddr = reinterpret_cast<const char *>(&Desc->dim);
-  size_t HeaderBytes = static_cast<size_t>(DimsAddr - RawDesc);
-  size_t DimsBytes = static_cast<size_t>(Desc->rank) * sizeof(CFI_dim_t);
-  size_t TotalBytes = HeaderBytes + DimsBytes;
-
-  return std::max<int64_t>(ReportedSize, static_cast<int64_t>(TotalBytes));
-}
-
 /// Utility function to perform a pointer attachment operation.
 ///
 /// For something like:
@@ -479,7 +448,6 @@ static int performPointerAttachment(DeviceTy &Device, AsyncInfoTy &AsyncInfo,
          "Need a valid pointer entry to perform pointer-attachment");
 
   constexpr int64_t VoidPtrSize = sizeof(void *);
-  HstPtrSize = getFortranDescriptorSize(HstPtrAddr, HstPtrSize);
   assert(HstPtrSize >= VoidPtrSize && "PointerSize is too small");
 
   void *TgtPteeBase =

>From 1509dcd44a616d8f793b6d9b29b3fcec189c3ca4 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Wed, 6 May 2026 15:30:12 +0100
Subject: [PATCH 4/8] Reworked PR to avoid kernel changes.

---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 197 +++++++-----------
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  |  31 ++-
 .../Lower/OpenMP/declare-target-automap.f90   |  27 +++
 .../Transforms/omp-automap-to-target-data.fir |  35 ++--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  51 ++++-
 .../omptarget-declare-target-llvm-host.mlir   |  10 +
 mlir/test/Target/LLVMIR/omptarget-llvm.mlir   |  31 +++
 7 files changed, 239 insertions(+), 143 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/declare-target-automap.f90

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index f0e0eefa7e220..dd006e046fa74 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -15,14 +15,11 @@
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Operation.h"
-#include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
 #include "llvm/Frontend/OpenMP/OMPConstants.h"
-#include <algorithm>
 
 namespace flangomp {
 #define GEN_PASS_DEF_AUTOMAPTOTARGETDATAPASS
@@ -90,11 +87,7 @@ class AutomapToTargetDataPass
   }
 
   void runOnOperation() override {
-    ModuleOp module = getOperation()->getParentOfType<ModuleOp>();
-    if (!module)
-      module = dyn_cast<ModuleOp>(getOperation());
-    if (!module)
-      return;
+    ModuleOp module = getOperation();
 
     // Build FIR builder for helper utilities.
     fir::KindMapping kindMap = fir::getKindMapping(module);
@@ -117,16 +110,61 @@ class AutomapToTargetDataPass
       if (needsBoundsOps(memOp.getMemref()))
         genBoundsOps(builder, memOp.getMemref(), bounds);
 
+      mlir::Value boxValue;
+      if (auto storeOp = mlir::dyn_cast<fir::StoreOp>(memOp.getOperation()))
+        boxValue = storeOp.getValue();
+      else
+        boxValue = mlir::cast<fir::LoadOp>(memOp.getOperation()).getResult();
+
+      mlir::Value baseAddr =
+          fir::BoxAddrOp::create(builder, memOp.getLoc(), boxValue);
+      mlir::Value dataAddr = builder.createConvert(
+          memOp.getLoc(),
+          builder.getRefType(fir::unwrapRefType(baseAddr.getType())), baseAddr);
+      mlir::Type baseTy = fir::unwrapRefType(dataAddr.getType());
+      if (mlir::Type eleTy = fir::dyn_cast_ptrOrBoxEleTy(baseTy))
+        baseTy = eleTy;
+      if (auto seqTy = mlir::dyn_cast<fir::SequenceType>(baseTy))
+        if (seqTy.hasDynamicExtents())
+          baseTy = seqTy.getEleTy();
+
       omp::TargetEnterExitUpdateDataOperands clauses;
+      bool isAlloc = isa<fir::StoreOp>(memOp);
+
+      auto createDescriptorMap =
+          [&](mlir::omp::ClauseMapFlags mapType) -> mlir::omp::MapInfoOp {
+        mlir::Type descriptorTy =
+            fir::unwrapRefType(memOp.getMemref().getType());
+        // The attach entry expects the descriptor object to already have a
+        // device mapping, but this raw object map must not be expanded as a
+        // Fortran descriptor member map.
+        return mlir::omp::MapInfoOp::create(
+            builder, memOp.getLoc(), memOp.getMemref().getType(),
+            memOp.getMemref(), TypeAttr::get(descriptorTy),
+            builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
+            builder.getAttr<omp::VariableCaptureKindAttr>(
+                omp::VariableCaptureKind::ByRef),
+            /*var_ptr_ptr=*/mlir::Value{},
+            /*var_ptr_ptr_type=*/mlir::TypeAttr{},
+            /*members=*/SmallVector<Value>{},
+            /*members_index=*/ArrayAttr{},
+            /*bounds=*/SmallVector<Value>{},
+            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
+            builder.getBoolAttr(true));
+      };
+
+      if (isAlloc)
+        clauses.mapVars.push_back(createDescriptorMap(
+            omp::ClauseMapFlags::to | omp::ClauseMapFlags::always));
+
+      mlir::omp::ClauseMapFlags mapType =
+          isAlloc ? omp::ClauseMapFlags::storage : omp::ClauseMapFlags::del;
       mlir::omp::MapInfoOp mapInfo = mlir::omp::MapInfoOp::create(
-          builder, memOp.getLoc(), memOp.getMemref().getType(),
-          memOp.getMemref(),
-          TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-          builder.getAttr<omp::ClauseMapFlagsAttr>(
-              isa<fir::StoreOp>(memOp) ? omp::ClauseMapFlags::to
-                                       : omp::ClauseMapFlags::del),
+          builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
+          TypeAttr::get(baseTy),
+          builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
           builder.getAttr<omp::VariableCaptureKindAttr>(
-              omp::VariableCaptureKind::ByCopy),
+              omp::VariableCaptureKind::ByRef),
           /*var_ptr_ptr=*/mlir::Value{},
           /*var_ptr_ptr_type=*/mlir::TypeAttr{},
           /*members=*/SmallVector<Value>{},
@@ -134,9 +172,31 @@ class AutomapToTargetDataPass
           /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
           builder.getBoolAttr(false));
       clauses.mapVars.push_back(mapInfo);
-      isa<fir::StoreOp>(memOp)
-          ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
-          : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
+
+      if (isAlloc) {
+        mlir::omp::MapInfoOp attachInfo = mlir::omp::MapInfoOp::create(
+            builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
+            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
+            builder.getAttr<omp::ClauseMapFlagsAttr>(
+                omp::ClauseMapFlags::attach),
+            builder.getAttr<omp::VariableCaptureKindAttr>(
+                omp::VariableCaptureKind::ByRef),
+            /*var_ptr_ptr=*/memOp.getMemref(),
+            /*var_ptr_ptr_type=*/TypeAttr::get(
+                fir::unwrapRefType(memOp.getMemref().getType())),
+            /*members=*/SmallVector<Value>{},
+            /*members_index=*/ArrayAttr{},
+            /*bounds=*/SmallVector<Value>{},
+            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
+            builder.getBoolAttr(false));
+        clauses.mapVars.push_back(attachInfo);
+      } else {
+        clauses.mapVars.push_back(
+            createDescriptorMap(omp::ClauseMapFlags::del));
+      }
+
+      isAlloc ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
+              : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
     };
 
     for (fir::GlobalOp globalOp : automapGlobals) {
@@ -154,107 +214,6 @@ class AutomapToTargetDataPass
           addMapInfo(globalOp, loadOp);
       }
     }
-
-    // Move automapped descriptors from map() to has_device_addr in target ops.
-    auto originatesFromAutomapGlobal = [&](mlir::Value varPtr) -> bool {
-      if (auto decl = mlir::dyn_cast_or_null<hlfir::DeclareOp>(
-              varPtr.getDefiningOp())) {
-        if (auto addrOp = mlir::dyn_cast_or_null<fir::AddrOfOp>(
-                decl.getMemref().getDefiningOp())) {
-          if (auto g =
-                  mlir::SymbolTable::lookupNearestSymbolFrom<fir::GlobalOp>(
-                      decl, addrOp.getSymbol()))
-            return automapGlobals.contains(g);
-        }
-      }
-      return false;
-    };
-
-    module.walk([&](mlir::omp::TargetOp target) {
-      // Collect candidates to move: descriptor maps of automapped globals.
-      llvm::SmallVector<mlir::Value> newMapOps;
-      llvm::SmallVector<unsigned> removedIndices;
-      llvm::SmallVector<mlir::Value> movedToHDA;
-      llvm::SmallVector<mlir::BlockArgument> oldMapArgsForMoved;
-
-      auto mapRange = target.getMapVars();
-      newMapOps.reserve(mapRange.size());
-
-      auto argIface = llvm::dyn_cast<mlir::omp::BlockArgOpenMPOpInterface>(
-          target.getOperation());
-      llvm::ArrayRef<mlir::BlockArgument> mapBlockArgs =
-          argIface.getMapBlockArgs();
-
-      for (auto [idx, mapVal] : llvm::enumerate(mapRange)) {
-        auto mapOp =
-            mlir::dyn_cast<mlir::omp::MapInfoOp>(mapVal.getDefiningOp());
-        if (!mapOp) {
-          newMapOps.push_back(mapVal);
-          continue;
-        }
-
-        mlir::Type varTy = fir::unwrapRefType(mapOp.getVarType());
-        bool isDescriptor = mlir::isa<fir::BaseBoxType>(varTy);
-        if (isDescriptor && originatesFromAutomapGlobal(mapOp.getVarPtr())) {
-          movedToHDA.push_back(mapVal);
-          removedIndices.push_back(idx);
-          oldMapArgsForMoved.push_back(mapBlockArgs[idx]);
-        } else {
-          newMapOps.push_back(mapVal);
-        }
-      }
-
-      if (movedToHDA.empty())
-        return;
-
-      // Update map vars to exclude moved entries.
-      mlir::MutableOperandRange mapMutable = target.getMapVarsMutable();
-      mapMutable.assign(newMapOps);
-
-      // Append moved entries to has_device_addr and insert corresponding block
-      // arguments.
-      mlir::MutableOperandRange hdaMutable =
-          target.getHasDeviceAddrVarsMutable();
-      llvm::SmallVector<mlir::Value> newHDA;
-      newHDA.reserve(hdaMutable.size() + movedToHDA.size());
-      llvm::for_each(hdaMutable.getAsOperandRange(),
-                     [&](mlir::Value v) { newHDA.push_back(v); });
-
-      unsigned hdaStart = argIface.getHasDeviceAddrBlockArgsStart();
-      unsigned oldHdaCount = argIface.numHasDeviceAddrBlockArgs();
-      llvm::SmallVector<mlir::BlockArgument> newHDAArgsForMoved;
-      unsigned insertIndex = hdaStart + oldHdaCount;
-      for (mlir::Value v : movedToHDA) {
-        newHDA.push_back(v);
-        target->getRegion(0).insertArgument(insertIndex, v.getType(),
-                                            v.getLoc());
-        // Capture the newly inserted block argument.
-        newHDAArgsForMoved.push_back(
-            target->getRegion(0).getArgument(insertIndex));
-        insertIndex++;
-      }
-      hdaMutable.assign(newHDA);
-
-      // Redirect uses in the region: replace old map block args with the
-      // corresponding new has_device_addr block args.
-      for (auto [oldArg, newArg] :
-           llvm::zip_equal(oldMapArgsForMoved, newHDAArgsForMoved))
-        oldArg.replaceAllUsesWith(newArg);
-
-      // Finally, erase corresponding map block arguments in descending order.
-      // Descending order is necessary to avoid index invalidation: erasing
-      // arguments from highest to lowest index ensures that earlier erases do
-      // not shift the indices of arguments yet to be erased.
-      unsigned mapStart = argIface.getMapBlockArgsStart();
-      // Convert indices to absolute argument numbers before erasing.
-      llvm::SmallVector<unsigned> absArgNos;
-      absArgNos.reserve(removedIndices.size());
-      for (unsigned idx : removedIndices)
-        absArgNos.push_back(mapStart + idx);
-      std::sort(absArgNos.begin(), absArgNos.end(), std::greater<>());
-      for (unsigned absNo : absArgNos)
-        target->getRegion(0).eraseArgument(absNo);
-    });
   }
 };
 } // namespace
diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index dfbc1bd6149cc..5dcf37d911147 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1349,6 +1349,34 @@ class MapInfoFinalizationPass
     return false;
   }
 
+  static bool isAttachMap(mlir::omp::MapInfoOp op) {
+    return (op.getMapType() & mlir::omp::ClauseMapFlags::attach) ==
+           mlir::omp::ClauseMapFlags::attach;
+  }
+
+  static bool isDescriptorOnlyMap(mlir::omp::MapInfoOp op) {
+    // A descriptor-only map keeps the descriptor object present so that a
+    // subsequent attach map can update its base address. It intentionally does
+    // not map the descriptor's pointee data, so descriptor member expansion
+    // must leave it alone.
+    if (!op.getPartialMap() || !op.getMembers().empty() ||
+        !op.getBounds().empty() || op.getVarPtrPtr() ||
+        op.getMapCaptureType() != mlir::omp::VariableCaptureKind::ByRef)
+      return false;
+
+    mlir::omp::ClauseMapFlags mapType = op.getMapType();
+    if ((mapType & mlir::omp::ClauseMapFlags::from) ==
+            mlir::omp::ClauseMapFlags::from ||
+        (mapType & mlir::omp::ClauseMapFlags::storage) ==
+            mlir::omp::ClauseMapFlags::storage)
+      return false;
+
+    return (mapType & mlir::omp::ClauseMapFlags::del) ==
+               mlir::omp::ClauseMapFlags::del ||
+           (mapType & mlir::omp::ClauseMapFlags::to) ==
+               mlir::omp::ClauseMapFlags::to;
+  }
+
   // This pass executes on omp::MapInfoOp's containing descriptor based types
   // (allocatables, pointers, assumed shape etc.) and expanding them into
   // multiple omp::MapInfoOp's for each pointer member contained within the
@@ -1610,7 +1638,8 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (!isAttachMap(op) && !isDescriptorOnlyMap(op) &&
+            hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
           mlir::Operation *targetUser = getFirstTargetUser(op);
diff --git a/flang/test/Lower/OpenMP/declare-target-automap.f90 b/flang/test/Lower/OpenMP/declare-target-automap.f90
new file mode 100644
index 0000000000000..fe46a3ab75c59
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-automap.f90
@@ -0,0 +1,27 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=60 %s -o - | FileCheck %s
+
+program automap_program
+   integer, allocatable, target :: automap_array(:)
+   !$omp declare target enter(automap:automap_array)
+
+   allocate (automap_array(10))
+
+   !$omp target
+      automap_array(1) = 1
+   !$omp end target
+
+   deallocate (automap_array)
+end program
+
+! CHECK-LABEL: func.func @_QQmain()
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+! CHECK-NOT: has_device_addr
+! CHECK: omp.target {{.*}}map_entries(
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DELETE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: omp.target_exit_data map_entries(%[[DELETE_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
diff --git a/flang/test/Transforms/omp-automap-to-target-data.fir b/flang/test/Transforms/omp-automap-to-target-data.fir
index 7a19705a248b4..7ed3cfa129e51 100644
--- a/flang/test/Transforms/omp-automap-to-target-data.fir
+++ b/flang/test/Transforms/omp-automap-to-target-data.fir
@@ -38,21 +38,26 @@ module {
 // CHECK-NEXT:      fir.shape
 // CHECK-NEXT:      %[[ARR_BOXED:.*]] = fir.embox %[[ALLOC_MEM]]
 // CHECK-NEXT:      fir.store %[[ARR_BOXED]]
-// CHECK-NEXT:      %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK-NEXT:      %[[ARR_HEAP_PTR:.*]] = fir.box_addr %[[ARR_BOXED_LOADED]]
-// CHECK-NEXT:      %[[DIM0:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
-// CHECK-NEXT:      %[[ONE:.*]] = arith.constant 1 : index
-// CHECK-NEXT:      %[[ZERO:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
-// CHECK-NEXT:      %[[LOWER_BOUND:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
-// CHECK-NEXT:      omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
-// CHECK-NEXT:      arith.muli %[[BOX_DIMS2]]#2, %[[BOX_DIMS2]]#1 : index
-// CHECK-NEXT:      %[[MAP_INFO:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(to) capture(ByCopy)
-// CHECK-NEXT:      omp.target_enter_data map_entries(%[[MAP_INFO]]
+// CHECK:           %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
+// CHECK:           %[[DIM0:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
+// CHECK:           %[[ONE:.*]] = arith.constant 1 : index
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
+// CHECK:           %[[LOWER_BOUND:.*]] = arith.constant 0 : index
+// CHECK:           %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
+// CHECK:           %[[BOUNDS:.*]] = omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
+// CHECK:           %[[DATA_ADDR:.*]] = fir.box_addr %[[ARR_BOXED]]
+// CHECK-NEXT:      %[[DATA_REF:.*]] = fir.convert %[[DATA_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%[[BOUNDS]]) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
 // CHECK:           %[[LOAD:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK:           %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(delete) capture(ByCopy)
-// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]]
+// CHECK:           %[[EXIT_ADDR:.*]] = fir.box_addr %[[LOAD]]
+// CHECK-NEXT:      %[[EXIT_REF:.*]] = fir.convert %[[EXIT_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[EXIT_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef)
+// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 // CHECK-NEXT:      %[[BOXADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      fir.freemem %[[BOXADDR]]
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b2d2ef6d8f770..cfb7f9ecfc120 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -6374,16 +6374,18 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
+    bool isAttachMap =
+        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
+    Value pointeePtr = isRefPtrOrPteeMapWithAttach
+                           ? mapOp.getVarPtrPtr()
+                           : (isAttachMap ? mapOp.getVarPtr() : offloadPtr);
     mapData.OriginalValue.push_back(moduleTranslation.lookupValue(offloadPtr));
-    mapData.Pointers.push_back(
-        isRefPtrOrPteeMapWithAttach
-            ? moduleTranslation.lookupValue(mapOp.getVarPtrPtr())
-            : mapData.OriginalValue.back());
+    mapData.Pointers.push_back(moduleTranslation.lookupValue(pointeePtr));
 
     if (llvm::Value *refPtr =
             getRefPtrIfDeclareTarget(offloadPtr, moduleTranslation)) {
@@ -7197,6 +7199,10 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         ((convertClauseMapFlags(mapOp.getMapType()) &
           llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH) ==
          llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH);
+    bool isRefPtrOrPteeMapWithAttach =
+        isAttachMap &&
+        (bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptr) ||
+         bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptee));
 
     // If it's declare target, skip it, it's handled separately. However, if
     // it's declare target, and an attach map, we want to calculate the exact
@@ -7219,7 +7225,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy)
+        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
           newV = builder.CreateLoad(builder.getPtrTy(), newV);
 
         if (!offsetIdx.empty())
@@ -8855,6 +8861,35 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
       auto captureClauseKind = convertToCaptureClauseKind(captureClause);
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
+      llvm::StringRef entryMangledName = mangledName;
+      llvm::Constant *entryAddr = llvm::cast<llvm::Constant>(gVal);
+      std::function<llvm::GlobalValue::LinkageTypes()> variableLinkage;
+      llvm::SmallString<128> entryNameStorage;
+      if ((attribute.getCaptureClause().getValue() ==
+               mlir::omp::DeclareTargetCaptureClause::to ||
+           attribute.getCaptureClause().getValue() ==
+               mlir::omp::DeclareTargetCaptureClause::enter) &&
+          !isDeclaration &&
+          (gVal->hasLocalLinkage() || gVal->hasHiddenVisibility())) {
+        // Keep the original symbol local so target code can address it using
+        // target-specific local relocations, but create a visible alias for
+        // the offload entry so libomptarget can associate the host global with
+        // the actual device global.
+        entryNameStorage = (mangledName + llvm::Twine("_decl_tgt_entry")).str();
+        entryMangledName = entryNameStorage;
+        if (llvm::GlobalValue *existing =
+                llvmModule->getNamedValue(entryMangledName)) {
+          entryAddr = llvm::cast<llvm::Constant>(existing);
+        } else {
+          entryAddr = llvm::GlobalAlias::create(
+              gVal->getValueType(), gVal->getAddressSpace(),
+              llvm::GlobalValue::WeakAnyLinkage, entryMangledName, entryAddr,
+              llvmModule);
+          llvm::cast<llvm::GlobalAlias>(entryAddr)->setVisibility(
+              llvm::GlobalValue::DefaultVisibility);
+        }
+        variableLinkage = [] { return llvm::GlobalValue::WeakAnyLinkage; };
+      }
       // unused for MLIR at the moment, required in Clang for book
       // keeping
       std::vector<llvm::GlobalVariable *> generatedRefs;
@@ -8904,9 +8939,9 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
       ompBuilder->registerTargetGlobalVariable(
           captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
-          mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
-          /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
-          gVal->getType(), gVal);
+          entryMangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
+          /*GlobalInitializer*/ nullptr, variableLinkage, gVal->getType(),
+          entryAddr);
 
       if (ompBuilder->Config.isTargetDevice() &&
           (captureClause == omp::DeclareTargetCaptureClause::link ||
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
index 92c85738dbc72..3343c96962f6c 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
@@ -133,6 +133,16 @@ module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_targe
     llvm.return %0 : i32
   }
 
+  // CHECK-DAG: @_QFEinternal_enter = internal global i32 7
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [34 x i8] c"_QFEinternal_enter_decl_tgt_entry\00"
+  // CHECK-DAG: @.offloading.entry._QFEinternal_enter_decl_tgt_entry = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEinternal_enter_decl_tgt_entry, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: @_QFEinternal_enter_decl_tgt_entry = weak alias i32, ptr @_QFEinternal_enter
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEinternal_enter_decl_tgt_entry", i32 {{.*}}, i32 {{.*}}}
+  llvm.mlir.global internal @_QFEinternal_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
+    %0 = llvm.mlir.constant(7 : i32) : i32
+    llvm.return %0 : i32
+  }
+
   // CHECK-DAG: @_QMtest_0Ept1 = global { ptr, i64, i32, i8, i8, i8, i8 } { ptr null, i64 4, i32 20180515, i8 0, i8 9, i8 1, i8 0 }
   // CHECK-DAG: @_QMtest_0Ept1_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Ept1
   // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [31 x i8] c"_QMtest_0Ept1_decl_tgt_ref_ptr\00"
diff --git a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
index bb6085840cf07..df9e45ccc3a63 100644
--- a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
@@ -634,3 +634,34 @@ module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
 // CHECK: @.offload_sizes = private unnamed_addr constant [2 x i64] [i64 8, i64 0]
 // CHECK: @.offload_maptypes = private unnamed_addr constant [2 x i64] [i64 288, i64 288]
 // CHECK-LABEL: define void @_QPomp_target_is_device_ptr
+
+// -----
+
+module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
+  llvm.func @_QPomp_target_enter_attach(%desc : !llvm.ptr,
+                                        %pointee : !llvm.ptr) {
+    %map = omp.map.info
+        var_ptr(%pointee : !llvm.ptr, !llvm.struct<(ptr, i64)>)
+        map_clauses(attach) capture(ByRef)
+        var_ptr_ptr(%desc : !llvm.ptr, !llvm.struct<(ptr, i64)>)
+        -> !llvm.ptr {name = ""}
+    omp.target_enter_data map_entries(%map : !llvm.ptr)
+    llvm.return
+  }
+}
+
+// CHECK: @.offload_maptypes = private unnamed_addr constant [1 x i64] [i64 16384]
+// CHECK-LABEL: define void @_QPomp_target_enter_attach
+// CHECK-SAME: (ptr %[[DESC:.*]], ptr %[[POINTEE:.*]]) {
+// CHECK:         %[[BASEPTRS:.*]] = alloca [1 x ptr], align 8
+// CHECK:         %[[PTRS:.*]] = alloca [1 x ptr], align 8
+// CHECK:         %[[SIZES:.*]] = alloca [1 x i64], align 8
+// CHECK:         %[[IS_NULL:.*]] = icmp eq ptr %[[POINTEE]], null
+// CHECK-NEXT:    %[[SIZE:.*]] = select i1 %[[IS_NULL]], i64 0, i64 16
+// CHECK:         %[[BASEPTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[BASEPTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr %[[DESC]], ptr %[[BASEPTR]], align 8
+// CHECK-NEXT:    %[[PTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[PTRS]], i32 0, i32 0
+// CHECK-NEXT:    store ptr %[[POINTEE]], ptr %[[PTR]], align 8
+// CHECK-NEXT:    %[[SIZEPTR:.*]] = getelementptr inbounds [1 x i64], ptr %[[SIZES]], i32 0, i32 0
+// CHECK-NEXT:    store i64 %[[SIZE]], ptr %[[SIZEPTR]], align 8
+// CHECK:         call void @__tgt_target_data_begin_mapper(ptr @2, i64 -1, i32 1, ptr %{{.*}}, ptr %{{.*}}, ptr %{{.*}}, ptr @.offload_maptypes, ptr @.offload_mapnames, ptr null)

>From 5c4b27d6570c7d8d361e4352a2da1c29d52a758b Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Tue, 16 Jun 2026 15:58:03 +0100
Subject: [PATCH 5/8] Fix clang-format.

---
 flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index dd006e046fa74..251bf3aacd39c 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -182,8 +182,8 @@ class AutomapToTargetDataPass
             builder.getAttr<omp::VariableCaptureKindAttr>(
                 omp::VariableCaptureKind::ByRef),
             /*var_ptr_ptr=*/memOp.getMemref(),
-            /*var_ptr_ptr_type=*/TypeAttr::get(
-                fir::unwrapRefType(memOp.getMemref().getType())),
+            /*var_ptr_ptr_type=*/
+            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
             /*members=*/SmallVector<Value>{},
             /*members_index=*/ArrayAttr{},
             /*bounds=*/SmallVector<Value>{},

>From 3e301cd56d4e4c0a8d80f5b46943add25c8e4a3d Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Thu, 18 Jun 2026 15:45:08 +0100
Subject: [PATCH 6/8] Addressed reviewer comments.

---
 .../Optimizer/OpenMP/AutomapToTargetData.cpp  | 30 +++++-------------
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  | 31 ++-----------------
 .../Lower/OpenMP/declare-target-automap.f90   |  9 +++---
 .../Transforms/omp-automap-to-target-data.fir |  7 ++---
 4 files changed, 17 insertions(+), 60 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index 251bf3aacd39c..bf269b5dd3ff2 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -135,9 +135,11 @@ class AutomapToTargetDataPass
           [&](mlir::omp::ClauseMapFlags mapType) -> mlir::omp::MapInfoOp {
         mlir::Type descriptorTy =
             fir::unwrapRefType(memOp.getMemref().getType());
-        // The attach entry expects the descriptor object to already have a
-        // device mapping, but this raw object map must not be expanded as a
-        // Fortran descriptor member map.
+        // Keep the descriptor itself present. MapInfoFinalization expands this
+        // ref_ptr map and emits the attach map when the descriptor is created.
+        mapType |= omp::ClauseMapFlags::ref_ptr;
+        if (!isAlloc)
+          mapType |= omp::ClauseMapFlags::attach_never;
         return mlir::omp::MapInfoOp::create(
             builder, memOp.getLoc(), memOp.getMemref().getType(),
             memOp.getMemref(), TypeAttr::get(descriptorTy),
@@ -150,7 +152,7 @@ class AutomapToTargetDataPass
             /*members_index=*/ArrayAttr{},
             /*bounds=*/SmallVector<Value>{},
             /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
-            builder.getBoolAttr(true));
+            builder.getBoolAttr(false));
       };
 
       if (isAlloc)
@@ -173,27 +175,9 @@ class AutomapToTargetDataPass
           builder.getBoolAttr(false));
       clauses.mapVars.push_back(mapInfo);
 
-      if (isAlloc) {
-        mlir::omp::MapInfoOp attachInfo = mlir::omp::MapInfoOp::create(
-            builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
-            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-            builder.getAttr<omp::ClauseMapFlagsAttr>(
-                omp::ClauseMapFlags::attach),
-            builder.getAttr<omp::VariableCaptureKindAttr>(
-                omp::VariableCaptureKind::ByRef),
-            /*var_ptr_ptr=*/memOp.getMemref(),
-            /*var_ptr_ptr_type=*/
-            TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-            /*members=*/SmallVector<Value>{},
-            /*members_index=*/ArrayAttr{},
-            /*bounds=*/SmallVector<Value>{},
-            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
-            builder.getBoolAttr(false));
-        clauses.mapVars.push_back(attachInfo);
-      } else {
+      if (!isAlloc)
         clauses.mapVars.push_back(
             createDescriptorMap(omp::ClauseMapFlags::del));
-      }
 
       isAlloc ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
               : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index 5dcf37d911147..160d0a0f8e4e6 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1349,34 +1349,6 @@ class MapInfoFinalizationPass
     return false;
   }
 
-  static bool isAttachMap(mlir::omp::MapInfoOp op) {
-    return (op.getMapType() & mlir::omp::ClauseMapFlags::attach) ==
-           mlir::omp::ClauseMapFlags::attach;
-  }
-
-  static bool isDescriptorOnlyMap(mlir::omp::MapInfoOp op) {
-    // A descriptor-only map keeps the descriptor object present so that a
-    // subsequent attach map can update its base address. It intentionally does
-    // not map the descriptor's pointee data, so descriptor member expansion
-    // must leave it alone.
-    if (!op.getPartialMap() || !op.getMembers().empty() ||
-        !op.getBounds().empty() || op.getVarPtrPtr() ||
-        op.getMapCaptureType() != mlir::omp::VariableCaptureKind::ByRef)
-      return false;
-
-    mlir::omp::ClauseMapFlags mapType = op.getMapType();
-    if ((mapType & mlir::omp::ClauseMapFlags::from) ==
-            mlir::omp::ClauseMapFlags::from ||
-        (mapType & mlir::omp::ClauseMapFlags::storage) ==
-            mlir::omp::ClauseMapFlags::storage)
-      return false;
-
-    return (mapType & mlir::omp::ClauseMapFlags::del) ==
-               mlir::omp::ClauseMapFlags::del ||
-           (mapType & mlir::omp::ClauseMapFlags::to) ==
-               mlir::omp::ClauseMapFlags::to;
-  }
-
   // This pass executes on omp::MapInfoOp's containing descriptor based types
   // (allocatables, pointers, assumed shape etc.) and expanding them into
   // multiple omp::MapInfoOp's for each pointer member contained within the
@@ -1638,7 +1610,8 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (!isAttachMap(op) && !isDescriptorOnlyMap(op) &&
+        if (!bitEnumContainsAll(op.getMapType(),
+                                mlir::omp::ClauseMapFlags::attach) &&
             hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
diff --git a/flang/test/Lower/OpenMP/declare-target-automap.f90 b/flang/test/Lower/OpenMP/declare-target-automap.f90
index fe46a3ab75c59..63014cbf36725 100644
--- a/flang/test/Lower/OpenMP/declare-target-automap.f90
+++ b/flang/test/Lower/OpenMP/declare-target-automap.f90
@@ -15,13 +15,14 @@ program automap_program
 
 ! CHECK-LABEL: func.func @_QQmain()
 ! CHECK-NOT: has_device_addr
-! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[DESC:.*]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_BASE:.*]] = fir.box_offset %[[DESC]] base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
+! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DESC]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach, ref_ptr) capture(ByRef) var_ptr_ptr(%[[DESC_BASE]] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
 ! CHECK: %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 ! CHECK-NOT: has_device_addr
 ! CHECK: omp.target {{.*}}map_entries(
 ! CHECK-NOT: has_device_addr
 ! CHECK: %[[DELETE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
-! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array", partial_map = true}
+! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
 ! CHECK: omp.target_exit_data map_entries(%[[DELETE_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
diff --git a/flang/test/Transforms/omp-automap-to-target-data.fir b/flang/test/Transforms/omp-automap-to-target-data.fir
index 7ed3cfa129e51..37187203369a4 100644
--- a/flang/test/Transforms/omp-automap-to-target-data.fir
+++ b/flang/test/Transforms/omp-automap-to-target-data.fir
@@ -49,15 +49,14 @@ module {
 // CHECK:           %[[BOUNDS:.*]] = omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
 // CHECK:           %[[DATA_ADDR:.*]] = fir.box_addr %[[ARR_BOXED]]
 // CHECK-NEXT:      %[[DATA_REF:.*]] = fir.convert %[[DATA_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
-// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
 // CHECK-NEXT:      %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%[[BOUNDS]]) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
-// CHECK-NEXT:      %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach) capture(ByRef) var_ptr_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
-// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.array<?xi32>>)
+// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>)
 // CHECK:           %[[LOAD:.*]] = fir.load %[[AUTOMAP_DECL]]#0
 // CHECK:           %[[EXIT_ADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      %[[EXIT_REF:.*]] = fir.convert %[[EXIT_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
 // CHECK-NEXT:      %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[EXIT_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef)
-// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr", partial_map = true}
+// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
 // CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 // CHECK-NEXT:      %[[BOXADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      fir.freemem %[[BOXADDR]]

>From 105d9900f76bea1f709b4433165e5abd5426348f Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Fri, 10 Jul 2026 14:51:46 +0100
Subject: [PATCH 7/8] Addressed reviewer comments.

---
 .../Optimizer/OpenMP/MapInfoFinalization.cpp  |  4 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 16 +++-------
 mlir/test/Target/LLVMIR/omptarget-llvm.mlir   | 31 -------------------
 3 files changed, 6 insertions(+), 45 deletions(-)

diff --git a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
index 160d0a0f8e4e6..dfbc1bd6149cc 100644
--- a/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
+++ b/flang/lib/Optimizer/OpenMP/MapInfoFinalization.cpp
@@ -1610,9 +1610,7 @@ class MapInfoFinalizationPass
                "single users or up to two users when those users"
                "are a MapInfoOp and Target mapping directive");
 
-        if (!bitEnumContainsAll(op.getMapType(),
-                                mlir::omp::ClauseMapFlags::attach) &&
-            hasADescriptor(op.getVarPtr().getDefiningOp(),
+        if (hasADescriptor(op.getVarPtr().getDefiningOp(),
                            fir::unwrapRefType(op.getVarPtrType()))) {
           builder.setInsertionPoint(op);
           mlir::Operation *targetUser = getFirstTargetUser(op);
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index cfb7f9ecfc120..9c0b3f4ef699a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -6374,18 +6374,16 @@ static void collectMapDataFromMapOperands(
   // Process MapOperands
   for (Value mapValue : mapVars) {
     auto mapOp = cast<omp::MapInfoOp>(mapValue.getDefiningOp());
-    bool isAttachMap =
-        bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::attach);
     bool isRefPtrOrPteeMapWithAttach =
         checkRefPtrOrPteeMapWithAttach(mapOp.getMapType());
     Value offloadPtr = (mapOp.getVarPtrPtr() && !isRefPtrOrPteeMapWithAttach)
                            ? mapOp.getVarPtrPtr()
                            : mapOp.getVarPtr();
-    Value pointeePtr = isRefPtrOrPteeMapWithAttach
-                           ? mapOp.getVarPtrPtr()
-                           : (isAttachMap ? mapOp.getVarPtr() : offloadPtr);
     mapData.OriginalValue.push_back(moduleTranslation.lookupValue(offloadPtr));
-    mapData.Pointers.push_back(moduleTranslation.lookupValue(pointeePtr));
+    mapData.Pointers.push_back(
+        isRefPtrOrPteeMapWithAttach
+            ? moduleTranslation.lookupValue(mapOp.getVarPtrPtr())
+            : mapData.OriginalValue.back());
 
     if (llvm::Value *refPtr =
             getRefPtrIfDeclareTarget(offloadPtr, moduleTranslation)) {
@@ -7199,10 +7197,6 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         ((convertClauseMapFlags(mapOp.getMapType()) &
           llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH) ==
          llvm::omp::OpenMPOffloadMappingFlags::OMP_MAP_ATTACH);
-    bool isRefPtrOrPteeMapWithAttach =
-        isAttachMap &&
-        (bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptr) ||
-         bitEnumContainsAll(mapOp.getMapType(), omp::ClauseMapFlags::ref_ptee));
 
     // If it's declare target, skip it, it's handled separately. However, if
     // it's declare target, and an attach map, we want to calculate the exact
@@ -7225,7 +7219,7 @@ createAlteredByCaptureMap(MapInfoData &mapData,
         std::vector<llvm::Value *> offsetIdx = calculateBoundsOffset(
             moduleTranslation, builder, mapData.BaseType[i]->isArrayTy(),
             mapOp.getBounds());
-        if (isPtrTy && (!isAttachMap || isRefPtrOrPteeMapWithAttach))
+        if (isPtrTy)
           newV = builder.CreateLoad(builder.getPtrTy(), newV);
 
         if (!offsetIdx.empty())
diff --git a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
index df9e45ccc3a63..bb6085840cf07 100644
--- a/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-llvm.mlir
@@ -634,34 +634,3 @@ module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
 // CHECK: @.offload_sizes = private unnamed_addr constant [2 x i64] [i64 8, i64 0]
 // CHECK: @.offload_maptypes = private unnamed_addr constant [2 x i64] [i64 288, i64 288]
 // CHECK-LABEL: define void @_QPomp_target_is_device_ptr
-
-// -----
-
-module attributes {omp.target_triples = ["amdgcn-amd-amdhsa"]} {
-  llvm.func @_QPomp_target_enter_attach(%desc : !llvm.ptr,
-                                        %pointee : !llvm.ptr) {
-    %map = omp.map.info
-        var_ptr(%pointee : !llvm.ptr, !llvm.struct<(ptr, i64)>)
-        map_clauses(attach) capture(ByRef)
-        var_ptr_ptr(%desc : !llvm.ptr, !llvm.struct<(ptr, i64)>)
-        -> !llvm.ptr {name = ""}
-    omp.target_enter_data map_entries(%map : !llvm.ptr)
-    llvm.return
-  }
-}
-
-// CHECK: @.offload_maptypes = private unnamed_addr constant [1 x i64] [i64 16384]
-// CHECK-LABEL: define void @_QPomp_target_enter_attach
-// CHECK-SAME: (ptr %[[DESC:.*]], ptr %[[POINTEE:.*]]) {
-// CHECK:         %[[BASEPTRS:.*]] = alloca [1 x ptr], align 8
-// CHECK:         %[[PTRS:.*]] = alloca [1 x ptr], align 8
-// CHECK:         %[[SIZES:.*]] = alloca [1 x i64], align 8
-// CHECK:         %[[IS_NULL:.*]] = icmp eq ptr %[[POINTEE]], null
-// CHECK-NEXT:    %[[SIZE:.*]] = select i1 %[[IS_NULL]], i64 0, i64 16
-// CHECK:         %[[BASEPTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[BASEPTRS]], i32 0, i32 0
-// CHECK-NEXT:    store ptr %[[DESC]], ptr %[[BASEPTR]], align 8
-// CHECK-NEXT:    %[[PTR:.*]] = getelementptr inbounds [1 x ptr], ptr %[[PTRS]], i32 0, i32 0
-// CHECK-NEXT:    store ptr %[[POINTEE]], ptr %[[PTR]], align 8
-// CHECK-NEXT:    %[[SIZEPTR:.*]] = getelementptr inbounds [1 x i64], ptr %[[SIZES]], i32 0, i32 0
-// CHECK-NEXT:    store i64 %[[SIZE]], ptr %[[SIZEPTR]], align 8
-// CHECK:         call void @__tgt_target_data_begin_mapper(ptr @2, i64 -1, i32 1, ptr %{{.*}}, ptr %{{.*}}, ptr %{{.*}}, ptr @.offload_maptypes, ptr @.offload_mapnames, ptr null)

>From 052ad9f98ac2f352187c12ee1506eaa592db6194 Mon Sep 17 00:00:00 2001
From: Akash Banerjee <Akash.Banerjee at amd.com>
Date: Fri, 10 Jul 2026 16:13:16 +0100
Subject: [PATCH 8/8] rebase

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 58 +++++++++----------
 .../omptarget-declare-target-llvm-host.mlir   | 19 ++++--
 2 files changed, 42 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 9c0b3f4ef699a..f47971774f1bd 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8859,16 +8859,35 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
       llvm::Constant *entryAddr = llvm::cast<llvm::Constant>(gVal);
       std::function<llvm::GlobalValue::LinkageTypes()> variableLinkage;
       llvm::SmallString<128> entryNameStorage;
-      if ((attribute.getCaptureClause().getValue() ==
-               mlir::omp::DeclareTargetCaptureClause::to ||
-           attribute.getCaptureClause().getValue() ==
-               mlir::omp::DeclareTargetCaptureClause::enter) &&
-          !isDeclaration &&
+      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
+      bool isToOrEnter =
+          captureClause == omp::DeclareTargetCaptureClause::to ||
+          captureClause == omp::DeclareTargetCaptureClause::enter;
+      bool isHostOnly = attribute.getDeviceType().getValue() ==
+                        omp::DeclareTargetDeviceType::host;
+
+      // A to/enter declare-target variable needs a device-resident,
+      // name-resolvable copy and a host offloading entry. A local-linkage
+      // global provides neither, so we promote it to external.
+      if (isToOrEnter && !isHostOnly && !requiresUSM &&
+          gVar->hasLocalLinkage()) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        isExternallyVisible = true;
+
+        // Clear the stale dso_local flag so it is referenced like a
+        // module-scope declare target global.
+        if (ompBuilder->Config.isTargetDevice())
+          gVar->setDSOLocal(false);
+      }
+
+      if (isToOrEnter &&
+          deviceClause ==
+              llvm::OffloadEntriesInfoManager::OMPTargetDeviceClauseAny &&
+          !requiresUSM && !isDeclaration &&
           (gVal->hasLocalLinkage() || gVal->hasHiddenVisibility())) {
-        // Keep the original symbol local so target code can address it using
-        // target-specific local relocations, but create a visible alias for
-        // the offload entry so libomptarget can associate the host global with
-        // the actual device global.
+        // Keep the original symbol as-is for target code, but create a visible
+        // alias for the offload entry so libomptarget can associate the host
+        // global with the actual device global.
         entryNameStorage = (mangledName + llvm::Twine("_decl_tgt_entry")).str();
         entryMangledName = entryNameStorage;
         if (llvm::GlobalValue *existing =
@@ -8908,27 +8927,6 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                                                      lineNo);
       };
 
-      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
-      bool isToOrEnter =
-          captureClause == omp::DeclareTargetCaptureClause::to ||
-          captureClause == omp::DeclareTargetCaptureClause::enter;
-      bool isHostOnly = attribute.getDeviceType().getValue() ==
-                        omp::DeclareTargetDeviceType::host;
-
-      // A to/enter declare-target variable needs a device-resident,
-      // name-resolvable copy and a host offloading entry. A local-linkage
-      // global provides neither, so we promote it to external.
-      if (isToOrEnter && !isHostOnly && !requiresUSM &&
-          gVar->hasLocalLinkage()) {
-        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
-        isExternallyVisible = true;
-
-        // Clear the stale dso_local flag so it is referenced like a
-        // module-scope declare target global.
-        if (ompBuilder->Config.isTargetDevice())
-          gVar->setDSOLocal(false);
-      }
-
       llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
       ompBuilder->registerTargetGlobalVariable(
           captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
diff --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
index 3343c96962f6c..0d795eefa1f9d 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
@@ -133,16 +133,25 @@ module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_targe
     llvm.return %0 : i32
   }
 
-  // CHECK-DAG: @_QFEinternal_enter = internal global i32 7
-  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [34 x i8] c"_QFEinternal_enter_decl_tgt_entry\00"
-  // CHECK-DAG: @.offloading.entry._QFEinternal_enter_decl_tgt_entry = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEinternal_enter_decl_tgt_entry, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
-  // CHECK-DAG: @_QFEinternal_enter_decl_tgt_entry = weak alias i32, ptr @_QFEinternal_enter
-  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEinternal_enter_decl_tgt_entry", i32 {{.*}}, i32 {{.*}}}
+  // CHECK-DAG: @_QFEinternal_enter = dso_local global i32 7
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [19 x i8] c"_QFEinternal_enter\00"
+  // CHECK-DAG: @.offloading.entry._QFEinternal_enter = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEinternal_enter, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEinternal_enter", i32 {{.*}}, i32 {{.*}}}
   llvm.mlir.global internal @_QFEinternal_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
     %0 = llvm.mlir.constant(7 : i32) : i32
     llvm.return %0 : i32
   }
 
+  // CHECK-DAG: @_QFEhidden_enter = hidden global i32 8
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [32 x i8] c"_QFEhidden_enter_decl_tgt_entry\00"
+  // CHECK-DAG: @.offloading.entry._QFEhidden_enter_decl_tgt_entry = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEhidden_enter_decl_tgt_entry, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: @_QFEhidden_enter_decl_tgt_entry = weak alias i32, ptr @_QFEhidden_enter
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEhidden_enter_decl_tgt_entry", i32 {{.*}}, i32 {{.*}}}
+  llvm.mlir.global external hidden @_QFEhidden_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
+    %0 = llvm.mlir.constant(8 : i32) : i32
+    llvm.return %0 : i32
+  }
+
   // CHECK-DAG: @_QMtest_0Ept1 = global { ptr, i64, i32, i8, i8, i8, i8 } { ptr null, i64 4, i32 20180515, i8 0, i8 9, i8 1, i8 0 }
   // CHECK-DAG: @_QMtest_0Ept1_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Ept1
   // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [31 x i8] c"_QMtest_0Ept1_decl_tgt_ref_ptr\00"


From llvmlistbot at llvm.org  Fri Jul 10 08:15:25 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 10 Jul 2026 08:15:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][dataflow] Handle call operands to
 callees with unknown callers (PR #206584)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206584@github.com>
Message-ID: <6a510c8d.0c05e68e.1e82f1.a4e2@mx.google.com>


https://github.com/victor-eds closed https://github.com/llvm/llvm-project/pull/206584

From llvmlistbot at llvm.org  Fri Jul 10 08:19:54 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:19:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
Message-ID: <llvm.org/llvm/llvm-project/pull/208752@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/208752

`combineContractAndBroadcast` is a pattern that matches `vector.contract`
operations whose inputs are defined by `vector.broadcast`, e.g.:

```mlir
%0 = vector.broadcast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.broadcast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

The pattern folds away the reduction over a unit dimension introduced by
the defining operations, producing a simplified `vector.contract`:

```mlir
vector.contract {
  indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
  iterator_types = ["parallel", "parallel", "reduction"]
} %0, %1, %acc : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
```

This change extends the pattern to also accept broadcast-like
`vector.shape_cast` operations:

```mlir
%0 = vector.shape_cast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.shape_cast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

Note that `mlir::vector::isBroadcastableTo` was only moved to make it
available to the newly introduced `ShapeCastOp` helper,
`ShapeCastOp::isBroadcastLike()`.


>From 643775665538881e17a0ddb2337ae7f170a30bcb Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 10 Jul 2026 14:56:23 +0000
Subject: [PATCH] [mlir][vector] Extend `combineContractAndBroadcast` to accept
 `vector.shape_cast`

`combineContractAndBroadcast` is a pattern that matches `vector.contract`
operations whose inputs are defined by `vector.broadcast`, e.g.:

```mlir
%0 = vector.broadcast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.broadcast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

The pattern folds away the reduction over a unit dimension introduced by
the defining operations, producing a simplified `vector.contract`:

```mlir
vector.contract {
  indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
  iterator_types = ["parallel", "parallel", "reduction"]
} %0, %1, %acc : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
```

This change extends the pattern to also accept broadcast-like
`vector.shape_cast` operations:

```mlir
%0 = vector.shape_cast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.shape_cast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

Note that `mlir::vector::isBroadcastableTo` was only moved to make it
available to the newly introduced `ShapeCastOp` helper,
`ShapeCastOp::isBroadcastLike()`.
---
 .../mlir/Dialect/Vector/IR/VectorOps.h        |   2 +-
 .../mlir/Dialect/Vector/IR/VectorOps.td       |   3 +
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 121 ++++++++++--------
 .../Vector/Transforms/VectorTransforms.cpp    |  36 ++++--
 .../Vector/vector-reduce-to-contract.mlir     | 103 ++++++++++++++-
 5 files changed, 194 insertions(+), 71 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
index b3a0653b90765..1a7d37d989f5a 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
@@ -67,7 +67,7 @@ enum class ConstantMaskKind { AllFalse = 0, AllTrue };
 /// arguments.
 void buildTerminatedBody(OpBuilder &builder, Location loc);
 
-/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// Models whether `srcType` can be broadcast to `dstVectorType` under the
 /// semantics of the `vector.broadcast` op.
 enum class BroadcastableToResult {
   Success = 0,
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index a519e6d89a472..f2a8ce2951106 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2484,6 +2484,9 @@ def Vector_ShapeCastOp :
     VectorType getResultVectorType() {
       return ::llvm::cast<VectorType>(getResult().getType());
     }
+    // Return true if this Op is effectively a vector.broadcast (i.e. the input
+    // and output shapes satisfy the vector.broadcast constraints).
+    bool isBroadcastLike();
   }];
   let assemblyFormat = "$source attr-dict `:` type($source) `to` type($result)";
   let hasFolder = 1;
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index e0cb9923b1dd4..fb535cdae461f 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -429,6 +429,63 @@ static Attribute convertNumericAttr(Attribute attr, Type expectedType) {
   return attr;
 }
 
+/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// semantics of the `vector.broadcast` op.
+BroadcastableToResult mlir::vector::isBroadcastableTo(
+    Type srcType, VectorType dstVectorType,
+    std::pair<VectorDim, VectorDim> *mismatchingDims) {
+  // Broadcast scalar to vector of the same element type.
+  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
+      srcType == getElementTypeOrSelf(dstVectorType))
+    return BroadcastableToResult::Success;
+  // From now on, only vectors broadcast.
+  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
+  if (!srcVectorType)
+    return BroadcastableToResult::SourceTypeNotAVector;
+
+  int64_t srcRank = srcVectorType.getRank();
+  int64_t dstRank = dstVectorType.getRank();
+  if (srcRank > dstRank)
+    return BroadcastableToResult::SourceRankHigher;
+  // Source has an exact match or singleton value for all trailing dimensions
+  // (all leading dimensions are simply duplicated).
+  int64_t lead = dstRank - srcRank;
+  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
+    // Have mismatching dims (in the sense of vector.broadcast semantics) been
+    // encountered?
+    bool foundMismatchingDims = false;
+
+    // Check fixed-width dims.
+    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
+    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
+    if (srcDim != 1 && srcDim != dstDim)
+      foundMismatchingDims = true;
+
+    // Check scalable flags.
+    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
+    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
+    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
+        // 1 -> [N] is fine, everything else should be rejected when mixing
+        // fixed-width and scalable dims
+        (srcDimScalableFlag != dstDimScalableFlag &&
+         (srcDim != 1 || srcDimScalableFlag)))
+      foundMismatchingDims = true;
+
+    if (foundMismatchingDims) {
+      if (mismatchingDims != nullptr) {
+        mismatchingDims->first.dim = srcDim;
+        mismatchingDims->first.isScalable = srcDimScalableFlag;
+
+        mismatchingDims->second.dim = dstDim;
+        mismatchingDims->second.isScalable = dstDimScalableFlag;
+      }
+      return BroadcastableToResult::DimensionMismatch;
+    }
+  }
+
+  return BroadcastableToResult::Success;
+}
+
 //===----------------------------------------------------------------------===//
 // CombiningKindAttr
 //===----------------------------------------------------------------------===//
@@ -3101,61 +3158,6 @@ Value BroadcastOp::createOrFoldBroadcastOp(
   return res;
 }
 
-BroadcastableToResult mlir::vector::isBroadcastableTo(
-    Type srcType, VectorType dstVectorType,
-    std::pair<VectorDim, VectorDim> *mismatchingDims) {
-  // Broadcast scalar to vector of the same element type.
-  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
-      srcType == getElementTypeOrSelf(dstVectorType))
-    return BroadcastableToResult::Success;
-  // From now on, only vectors broadcast.
-  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
-  if (!srcVectorType)
-    return BroadcastableToResult::SourceTypeNotAVector;
-
-  int64_t srcRank = srcVectorType.getRank();
-  int64_t dstRank = dstVectorType.getRank();
-  if (srcRank > dstRank)
-    return BroadcastableToResult::SourceRankHigher;
-  // Source has an exact match or singleton value for all trailing dimensions
-  // (all leading dimensions are simply duplicated).
-  int64_t lead = dstRank - srcRank;
-  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
-    // Have mismatching dims (in the sense of vector.broadcast semantics) been
-    // encountered?
-    bool foundMismatchingDims = false;
-
-    // Check fixed-width dims.
-    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
-    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
-    if (srcDim != 1 && srcDim != dstDim)
-      foundMismatchingDims = true;
-
-    // Check scalable flags.
-    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
-    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
-    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
-        // 1 -> [N] is fine, everything else should be rejected when mixing
-        // fixed-width and scalable dims
-        (srcDimScalableFlag != dstDimScalableFlag &&
-         (srcDim != 1 || srcDimScalableFlag)))
-      foundMismatchingDims = true;
-
-    if (foundMismatchingDims) {
-      if (mismatchingDims != nullptr) {
-        mismatchingDims->first.dim = srcDim;
-        mismatchingDims->first.isScalable = srcDimScalableFlag;
-
-        mismatchingDims->second.dim = dstDim;
-        mismatchingDims->second.isScalable = dstDimScalableFlag;
-      }
-      return BroadcastableToResult::DimensionMismatch;
-    }
-  }
-
-  return BroadcastableToResult::Success;
-}
-
 LogicalResult BroadcastOp::verify() {
   std::pair<VectorDim, VectorDim> mismatchingDims;
   BroadcastableToResult res = isBroadcastableTo(
@@ -6757,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
+         BroadcastableToResult::Success;
+}
+
 /// Return true if `transpose` does not permute a pair of non-unit dims.
 /// By `order preserving` we mean that the flattened versions of the input and
 /// output vectors are (numerically) identical. In other words `transpose` is
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 752610efc6992..22e7742128ff2 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -282,21 +282,32 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   bool changed = false;
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
+
+    // Make sure that the source
+    auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
-    if (!broadcast)
+    if (!broadcast && !sc)
       continue;
+
+    if (sc && !sc.isBroadcastLike())
+      return rewriter.notifyMatchFailure(
+          contractOp, "Operand defined via vector.shape_cast that has "
+                      "non-broadcast semantics");
+
+    VectorType srcType = (sc) ? sc.getSourceVectorType()
+                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    VectorType resType =
+        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+
     // contractionOp can only take vector as operands.
-    auto srcType = dyn_cast<VectorType>(broadcast.getSourceType());
-    if (!srcType ||
-        srcType.getRank() == broadcast.getResultVectorType().getRank())
+    // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());
+    if (!srcType || srcType.getRank() >= resType.getRank())
       continue;
-    int64_t rankDiff =
-        broadcast.getResultVectorType().getRank() - srcType.getRank();
+    int64_t rankDiff = resType.getRank() - srcType.getRank();
     bool innerDimBroadcast = false;
     SmallVector<AffineExpr> originalDims;
     for (const auto &dim : llvm::enumerate(srcType.getShape())) {
-      if (dim.value() !=
-          broadcast.getResultVectorType().getDimSize(rankDiff + dim.index())) {
+      if (dim.value() != resType.getDimSize(rankDiff + dim.index())) {
         innerDimBroadcast = true;
         break;
       }
@@ -311,7 +322,7 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     // of non-unit size.
     bool nonUnitDimReductionBroadcast = false;
     for (int64_t i = 0; i < rankDiff; ++i) {
-      if (broadcast.getResultVectorType().getDimSize(i) != 1 &&
+      if (resType.getDimSize(i) != 1 &&
           isReductionIterator(contractOp.getIteratorTypes()
                                   .getValue()[map.getDimPosition(i)])) {
         nonUnitDimReductionBroadcast = true;
@@ -321,11 +332,10 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     if (nonUnitDimReductionBroadcast)
       continue;
 
-    AffineMap broadcastMap =
-        AffineMap::get(broadcast.getResultVectorType().getRank(), 0,
-                       originalDims, contractOp.getContext());
+    AffineMap broadcastMap = AffineMap::get(resType.getRank(), 0, originalDims,
+                                            contractOp.getContext());
     map = broadcastMap.compose(map);
-    *operand = broadcast.getSource();
+    *operand = broadcast ? broadcast.getSource() : sc.getSource();
     changed = true;
   }
 
diff --git a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
index 3b51e6b1e1b6f..8e53a815f70f8 100644
--- a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
+++ b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
@@ -202,7 +202,37 @@ func.func @contract_broadcast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1
 
 // -----
 
-// Same as above, but with a mask.
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but
+// `vector.broadcast` is replaced with `vector.shape_cast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>)
+//  CHECK: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but with a mask.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -235,7 +265,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>,
 
 // -----
 
-// Same as above, but with a scalable dim.
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with
+// `vector.shape_cast` instead of `vector.broadcast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>, %[[MASK:.+]]: vector<1x8x8x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x8x4xi1> to vector<8x8x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>, %mask: vector<1x8x8x4xi1>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    } : vector<1x8x8x4xi1> -> vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with a scalable dim.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -267,6 +331,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked_scalable(%arg0 : vector<
 }
 
 // -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked_scalable`), but with
+// vector.shape_cast instead of vector.broadcast.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked_scalable
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<[8]x4xi32>, %[[ARG2:.+]]: vector<8x[8]xi32>, %[[MASK:.+]]: vector<1x8x[8]x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x[8]x4xi1> to vector<8x[8]x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<[8]x4xi32> into vector<8x[8]xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked_scalable(%arg0 : vector<8x4xi32>, %arg1 : vector<[8]x4xi32>, %arg2 : vector<8x[8]xi32>, %mask: vector<1x8x[8]x4xi1>) -> vector<8x[8]xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<[8]x4xi32> to vector<1x[8]x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x[8]x4xi32> into vector<8x[8]xi32>
+    } : vector<1x8x[8]x4xi1> -> vector<8x[8]xi32>
+    return %result : vector<8x[8]xi32>
+}
+
+// -----
+
 // Test that CombineContractBroadcast will not combine a broadcast that creates
 // a non-unit dim that is consumed by a reduction iterator.
 // Moreover, the affine_map's are permuting the position of that reduction


From llvmlistbot at llvm.org  Fri Jul 10 08:20:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 08:20:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a510dbf.0c05e68e.1e82f1.22d5@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

`combineContractAndBroadcast` is a pattern that matches `vector.contract`
operations whose inputs are defined by `vector.broadcast`, e.g.:

```mlir
%0 = vector.broadcast %lhs : vector&lt;8x4xi32&gt; to vector&lt;1x8x4xi32&gt;
%1 = vector.broadcast %rhs : vector&lt;8x4xi32&gt; to vector&lt;1x8x4xi32&gt;
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind&lt;add&gt;
  } %0, %1, %acc : vector&lt;1x8x4xi32&gt;, vector&lt;1x8x4xi32&gt; into vector&lt;8x8xi32&gt;
```

The pattern folds away the reduction over a unit dimension introduced by
the defining operations, producing a simplified `vector.contract`:

```mlir
vector.contract {
  indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
  iterator_types = ["parallel", "parallel", "reduction"]
} %0, %1, %acc : vector&lt;8x4xi32&gt;, vector&lt;8x4xi32&gt; into vector&lt;8x8xi32&gt;
```

This change extends the pattern to also accept broadcast-like
`vector.shape_cast` operations:

```mlir
%0 = vector.shape_cast %lhs : vector&lt;8x4xi32&gt; to vector&lt;1x8x4xi32&gt;
%1 = vector.shape_cast %rhs : vector&lt;8x4xi32&gt; to vector&lt;1x8x4xi32&gt;
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind&lt;add&gt;
  } %0, %1, %acc : vector&lt;1x8x4xi32&gt;, vector&lt;1x8x4xi32&gt; into vector&lt;8x8xi32&gt;
```

Note that `mlir::vector::isBroadcastableTo` was only moved to make it
available to the newly introduced `ShapeCastOp` helper,
`ShapeCastOp::isBroadcastLike()`.


---
Full diff: https://github.com/llvm/llvm-project/pull/208752.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/Vector/IR/VectorOps.h (+1-1) 
- (modified) mlir/include/mlir/Dialect/Vector/IR/VectorOps.td (+3) 
- (modified) mlir/lib/Dialect/Vector/IR/VectorOps.cpp (+66-55) 
- (modified) mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp (+23-13) 
- (modified) mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir (+101-2) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
index b3a0653b90765..1a7d37d989f5a 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
@@ -67,7 +67,7 @@ enum class ConstantMaskKind { AllFalse = 0, AllTrue };
 /// arguments.
 void buildTerminatedBody(OpBuilder &builder, Location loc);
 
-/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// Models whether `srcType` can be broadcast to `dstVectorType` under the
 /// semantics of the `vector.broadcast` op.
 enum class BroadcastableToResult {
   Success = 0,
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index a519e6d89a472..f2a8ce2951106 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2484,6 +2484,9 @@ def Vector_ShapeCastOp :
     VectorType getResultVectorType() {
       return ::llvm::cast<VectorType>(getResult().getType());
     }
+    // Return true if this Op is effectively a vector.broadcast (i.e. the input
+    // and output shapes satisfy the vector.broadcast constraints).
+    bool isBroadcastLike();
   }];
   let assemblyFormat = "$source attr-dict `:` type($source) `to` type($result)";
   let hasFolder = 1;
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index e0cb9923b1dd4..fb535cdae461f 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -429,6 +429,63 @@ static Attribute convertNumericAttr(Attribute attr, Type expectedType) {
   return attr;
 }
 
+/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// semantics of the `vector.broadcast` op.
+BroadcastableToResult mlir::vector::isBroadcastableTo(
+    Type srcType, VectorType dstVectorType,
+    std::pair<VectorDim, VectorDim> *mismatchingDims) {
+  // Broadcast scalar to vector of the same element type.
+  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
+      srcType == getElementTypeOrSelf(dstVectorType))
+    return BroadcastableToResult::Success;
+  // From now on, only vectors broadcast.
+  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
+  if (!srcVectorType)
+    return BroadcastableToResult::SourceTypeNotAVector;
+
+  int64_t srcRank = srcVectorType.getRank();
+  int64_t dstRank = dstVectorType.getRank();
+  if (srcRank > dstRank)
+    return BroadcastableToResult::SourceRankHigher;
+  // Source has an exact match or singleton value for all trailing dimensions
+  // (all leading dimensions are simply duplicated).
+  int64_t lead = dstRank - srcRank;
+  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
+    // Have mismatching dims (in the sense of vector.broadcast semantics) been
+    // encountered?
+    bool foundMismatchingDims = false;
+
+    // Check fixed-width dims.
+    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
+    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
+    if (srcDim != 1 && srcDim != dstDim)
+      foundMismatchingDims = true;
+
+    // Check scalable flags.
+    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
+    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
+    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
+        // 1 -> [N] is fine, everything else should be rejected when mixing
+        // fixed-width and scalable dims
+        (srcDimScalableFlag != dstDimScalableFlag &&
+         (srcDim != 1 || srcDimScalableFlag)))
+      foundMismatchingDims = true;
+
+    if (foundMismatchingDims) {
+      if (mismatchingDims != nullptr) {
+        mismatchingDims->first.dim = srcDim;
+        mismatchingDims->first.isScalable = srcDimScalableFlag;
+
+        mismatchingDims->second.dim = dstDim;
+        mismatchingDims->second.isScalable = dstDimScalableFlag;
+      }
+      return BroadcastableToResult::DimensionMismatch;
+    }
+  }
+
+  return BroadcastableToResult::Success;
+}
+
 //===----------------------------------------------------------------------===//
 // CombiningKindAttr
 //===----------------------------------------------------------------------===//
@@ -3101,61 +3158,6 @@ Value BroadcastOp::createOrFoldBroadcastOp(
   return res;
 }
 
-BroadcastableToResult mlir::vector::isBroadcastableTo(
-    Type srcType, VectorType dstVectorType,
-    std::pair<VectorDim, VectorDim> *mismatchingDims) {
-  // Broadcast scalar to vector of the same element type.
-  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
-      srcType == getElementTypeOrSelf(dstVectorType))
-    return BroadcastableToResult::Success;
-  // From now on, only vectors broadcast.
-  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
-  if (!srcVectorType)
-    return BroadcastableToResult::SourceTypeNotAVector;
-
-  int64_t srcRank = srcVectorType.getRank();
-  int64_t dstRank = dstVectorType.getRank();
-  if (srcRank > dstRank)
-    return BroadcastableToResult::SourceRankHigher;
-  // Source has an exact match or singleton value for all trailing dimensions
-  // (all leading dimensions are simply duplicated).
-  int64_t lead = dstRank - srcRank;
-  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
-    // Have mismatching dims (in the sense of vector.broadcast semantics) been
-    // encountered?
-    bool foundMismatchingDims = false;
-
-    // Check fixed-width dims.
-    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
-    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
-    if (srcDim != 1 && srcDim != dstDim)
-      foundMismatchingDims = true;
-
-    // Check scalable flags.
-    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
-    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
-    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
-        // 1 -> [N] is fine, everything else should be rejected when mixing
-        // fixed-width and scalable dims
-        (srcDimScalableFlag != dstDimScalableFlag &&
-         (srcDim != 1 || srcDimScalableFlag)))
-      foundMismatchingDims = true;
-
-    if (foundMismatchingDims) {
-      if (mismatchingDims != nullptr) {
-        mismatchingDims->first.dim = srcDim;
-        mismatchingDims->first.isScalable = srcDimScalableFlag;
-
-        mismatchingDims->second.dim = dstDim;
-        mismatchingDims->second.isScalable = dstDimScalableFlag;
-      }
-      return BroadcastableToResult::DimensionMismatch;
-    }
-  }
-
-  return BroadcastableToResult::Success;
-}
-
 LogicalResult BroadcastOp::verify() {
   std::pair<VectorDim, VectorDim> mismatchingDims;
   BroadcastableToResult res = isBroadcastableTo(
@@ -6757,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
+         BroadcastableToResult::Success;
+}
+
 /// Return true if `transpose` does not permute a pair of non-unit dims.
 /// By `order preserving` we mean that the flattened versions of the input and
 /// output vectors are (numerically) identical. In other words `transpose` is
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 752610efc6992..22e7742128ff2 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -282,21 +282,32 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   bool changed = false;
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
+
+    // Make sure that the source
+    auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
-    if (!broadcast)
+    if (!broadcast && !sc)
       continue;
+
+    if (sc && !sc.isBroadcastLike())
+      return rewriter.notifyMatchFailure(
+          contractOp, "Operand defined via vector.shape_cast that has "
+                      "non-broadcast semantics");
+
+    VectorType srcType = (sc) ? sc.getSourceVectorType()
+                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    VectorType resType =
+        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+
     // contractionOp can only take vector as operands.
-    auto srcType = dyn_cast<VectorType>(broadcast.getSourceType());
-    if (!srcType ||
-        srcType.getRank() == broadcast.getResultVectorType().getRank())
+    // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());
+    if (!srcType || srcType.getRank() >= resType.getRank())
       continue;
-    int64_t rankDiff =
-        broadcast.getResultVectorType().getRank() - srcType.getRank();
+    int64_t rankDiff = resType.getRank() - srcType.getRank();
     bool innerDimBroadcast = false;
     SmallVector<AffineExpr> originalDims;
     for (const auto &dim : llvm::enumerate(srcType.getShape())) {
-      if (dim.value() !=
-          broadcast.getResultVectorType().getDimSize(rankDiff + dim.index())) {
+      if (dim.value() != resType.getDimSize(rankDiff + dim.index())) {
         innerDimBroadcast = true;
         break;
       }
@@ -311,7 +322,7 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     // of non-unit size.
     bool nonUnitDimReductionBroadcast = false;
     for (int64_t i = 0; i < rankDiff; ++i) {
-      if (broadcast.getResultVectorType().getDimSize(i) != 1 &&
+      if (resType.getDimSize(i) != 1 &&
           isReductionIterator(contractOp.getIteratorTypes()
                                   .getValue()[map.getDimPosition(i)])) {
         nonUnitDimReductionBroadcast = true;
@@ -321,11 +332,10 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     if (nonUnitDimReductionBroadcast)
       continue;
 
-    AffineMap broadcastMap =
-        AffineMap::get(broadcast.getResultVectorType().getRank(), 0,
-                       originalDims, contractOp.getContext());
+    AffineMap broadcastMap = AffineMap::get(resType.getRank(), 0, originalDims,
+                                            contractOp.getContext());
     map = broadcastMap.compose(map);
-    *operand = broadcast.getSource();
+    *operand = broadcast ? broadcast.getSource() : sc.getSource();
     changed = true;
   }
 
diff --git a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
index 3b51e6b1e1b6f..8e53a815f70f8 100644
--- a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
+++ b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
@@ -202,7 +202,37 @@ func.func @contract_broadcast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1
 
 // -----
 
-// Same as above, but with a mask.
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but
+// `vector.broadcast` is replaced with `vector.shape_cast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>)
+//  CHECK: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but with a mask.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -235,7 +265,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>,
 
 // -----
 
-// Same as above, but with a scalable dim.
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with
+// `vector.shape_cast` instead of `vector.broadcast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>, %[[MASK:.+]]: vector<1x8x8x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x8x4xi1> to vector<8x8x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>, %mask: vector<1x8x8x4xi1>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    } : vector<1x8x8x4xi1> -> vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with a scalable dim.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -267,6 +331,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked_scalable(%arg0 : vector<
 }
 
 // -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked_scalable`), but with
+// vector.shape_cast instead of vector.broadcast.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked_scalable
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<[8]x4xi32>, %[[ARG2:.+]]: vector<8x[8]xi32>, %[[MASK:.+]]: vector<1x8x[8]x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x[8]x4xi1> to vector<8x[8]x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<[8]x4xi32> into vector<8x[8]xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked_scalable(%arg0 : vector<8x4xi32>, %arg1 : vector<[8]x4xi32>, %arg2 : vector<8x[8]xi32>, %mask: vector<1x8x[8]x4xi1>) -> vector<8x[8]xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<[8]x4xi32> to vector<1x[8]x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x[8]x4xi32> into vector<8x[8]xi32>
+    } : vector<1x8x[8]x4xi1> -> vector<8x[8]xi32>
+    return %result : vector<8x[8]xi32>
+}
+
+// -----
+
 // Test that CombineContractBroadcast will not combine a broadcast that creates
 // a non-unit dim that is consumed by a reduction iterator.
 // Moreover, the affine_map's are permuting the position of that reduction

``````````

</details>


https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Fri Jul 10 08:32:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 08:32:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] d58b056 -
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (#187927)
Message-ID: <6a51108a.160324b0.195192.f030@mx.google.com>


Author: Vladislav Dzhidzhoev
Date: 2026-07-10T15:32:16Z
New Revision: d58b0562a340b043ed83b0fb4375e89ed74fc621

URL: https://github.com/llvm/llvm-project/commit/d58b0562a340b043ed83b0fb4375e89ed74fc621
DIFF: https://github.com/llvm/llvm-project/commit/d58b0562a340b043ed83b0fb4375e89ed74fc621.diff

LOG: [DebugMetadata][DwarfDebug][CodeView] Support function-local static variables in lexical block scopes (6/7) (#187927)

RFC
https://discourse.llvm.org/t/rfc-dwarfdebug-fix-and-improve-handling-imported-entities-types-and-static-local-in-subprogram-and-lexical-block-scopes/68544

Similar to imported declarations and types, the patch tracks
function-local static variables (globals in terms of LLVM IR) in
DISubprogram's 'retainedNodes'. DwarfDebug is adjusted in accordance
with the aforementioned metadata change and provided a support of static
locals scoped by a lexical block. CodeViewDebug is modified to collect
global variables from DISubprogram's 'retainedNodes' too.

The patch assumes that DICompileUnit's 'globals' no longer tracks static
locals and DwarfDebug would assert if any locally-scoped variables get
placed there.

This change has already been reviewed in
https://reviews.llvm.org/D144008, but I’m opening a new PR because it
has been a while since it was approved. Additional logic was added to
CloneFunction to deal with DISubprograms that have static locals: if
such a subprogram is cloned, DIGlobalVariableExpressions are removed
from retainedNodes of the clone. This is consistent with the current
behavior, in which DIGlobalVariableExpressions are not cloned.

Authored-by: Kristina Bessonova <kbessonova at accesssoftek.com>
Co-authored-by: Tobias Gysi <tobias.gysi at nextsilicon.com>

Added: 
    llvm/test/Bitcode/upgrade-cu-static-locals.ll
    llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
    llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
    llvm/test/DebugInfo/X86/lexical-block-static-var.ll

Modified: 
    clang/test/OpenMP/error_codegen.cpp
    compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
    flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
    llvm/include/llvm/IR/DIBuilder.h
    llvm/include/llvm/IR/DebugInfo.h
    llvm/include/llvm/IR/DebugInfoMetadata.h
    llvm/include/llvm/IR/Metadata.h
    llvm/lib/Bitcode/Reader/MetadataLoader.cpp
    llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
    llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
    llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
    llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
    llvm/lib/IR/DIBuilder.cpp
    llvm/lib/IR/DebugInfo.cpp
    llvm/lib/IR/DebugInfoMetadata.cpp
    llvm/lib/IR/Verifier.cpp
    llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
    llvm/lib/Target/BPF/BTFDebug.cpp
    llvm/lib/Transforms/Coroutines/CoroFrame.cpp
    llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
    llvm/lib/Transforms/Utils/CloneFunction.cpp
    llvm/lib/Transforms/Utils/Debugify.cpp
    llvm/test/Bitcode/DIModule-fortran-external-module.ll
    llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
    llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
    llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
    llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
    llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
    llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
    llvm/test/CodeGen/BPF/BTF/static-var.ll
    llvm/test/CodeGen/BPF/dwarfdump.ll
    llvm/test/CodeGen/PowerPC/pr24546.ll
    llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
    llvm/test/DebugInfo/COFF/fortran-basic.ll
    llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
    llvm/test/DebugInfo/COFF/global_visibility.ll
    llvm/test/DebugInfo/COFF/type-quals.ll
    llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
    llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
    llvm/test/DebugInfo/Generic/inlined-static-var.ll
    llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
    llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
    llvm/test/DebugInfo/X86/DW_AT_specification.ll
    llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
    llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
    llvm/test/DebugInfo/X86/distringtype.ll
    llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
    llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
    llvm/test/DebugInfo/X86/gnu-public-names.ll
    llvm/test/DebugInfo/X86/namelist1.ll
    llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
    llvm/test/DebugInfo/duplicate_dbgvalue.ll
    llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
    llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
    llvm/test/Transforms/Util/clone-dicompileunit.ll
    llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
    llvm/unittests/Transforms/Utils/CloningTest.cpp
    mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
    mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
    mlir/lib/CAPI/Dialect/LLVM.cpp
    mlir/lib/Target/LLVMIR/DebugImporter.cpp
    mlir/lib/Target/LLVMIR/DebugTranslation.cpp
    mlir/lib/Target/LLVMIR/DebugTranslation.h
    mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
    mlir/test/Target/LLVMIR/Import/debug-info.ll
    mlir/test/Target/LLVMIR/llvmir-debug.mlir

Removed: 
    


################################################################################
diff  --git a/clang/test/OpenMP/error_codegen.cpp b/clang/test/OpenMP/error_codegen.cpp
index 0efb0ab098cf3..def68ea020c63 100644
--- a/clang/test/OpenMP/error_codegen.cpp
+++ b/clang/test/OpenMP/error_codegen.cpp
@@ -565,7 +565,7 @@ int main (int argc, char **argv) {
 //
 //
 // SIMD-LABEL: define linkonce_odr noundef i32 @_Z5tmainIiLi10EEiT_PPc(
-// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG21:![0-9]+]] {
+// SIMD-SAME: i32 noundef [[ARGC:%.*]], ptr noundef [[ARGV:%.*]]) #[[ATTR0]] comdat !dbg [[DBG22:![0-9]+]] {
 // SIMD-NEXT:  [[ENTRY:.*:]]
 // SIMD-NEXT:    [[ARGC_ADDR:%.*]] = alloca i32, align 4
 // SIMD-NEXT:    [[ARGV_ADDR:%.*]] = alloca ptr, align 8
@@ -630,7 +630,7 @@ int main (int argc, char **argv) {
 //.
 // SIMD: [[META0:![0-9]+]] = !DIGlobalVariableExpression(var: [[META1:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META1]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG2]], file: [[META3:![0-9]+]], line: 61, type: [[META6:![0-9]+]], isLocal: true, isDefinition: true)
-// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META22:![0-9]+]])
+// SIMD: [[DBG2]] = distinct !DISubprogram(name: "main", scope: [[META3]], file: [[META3]], line: 59, type: [[META4:![0-9]+]], scopeLine: 59, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10:![0-9]+]], retainedNodes: [[META19:![0-9]+]])
 // SIMD: [[META3]] = !DIFile(filename: "{{.*}}error_codegen.cpp", directory: {{.*}})
 // SIMD: [[META4]] = !DISubroutineType(types: [[META5:![0-9]+]])
 // SIMD: [[META5]] = !{[[META6]], [[META6]], [[META7:![0-9]+]]}
@@ -640,20 +640,21 @@ int main (int argc, char **argv) {
 // SIMD: [[META9]] = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
 // SIMD: [[META10]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_11, file: [[META11:![0-9]+]], producer: "{{.*}}clang version {{.*}}", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: [[META12:![0-9]+]], splitDebugInlining: false, nameTableKind: None)
 // SIMD: [[META11]] = !DIFile(filename: "{{.*}}<stdin>", directory: {{.*}})
-// SIMD: [[META12]] = !{[[META0]], [[META13:![0-9]+]], [[META19:![0-9]+]]}
+// SIMD: [[META12]] = !{[[META13:![0-9]+]]}
 // SIMD: [[META13]] = !DIGlobalVariableExpression(var: [[META14:![0-9]+]], expr: !DIExpression())
 // SIMD: [[META14]] = distinct !DIGlobalVariable(scope: null, file: [[META3]], line: 63, type: [[META15:![0-9]+]], isLocal: true, isDefinition: true)
 // SIMD: [[META15]] = !DICompositeType(tag: DW_TAG_array_type, baseType: [[META16:![0-9]+]], size: 32, elements: [[META17:![0-9]+]])
 // SIMD: [[META16]] = !DIDerivedType(tag: DW_TAG_const_type, baseType: [[META9]])
 // SIMD: [[META17]] = !{[[META18:![0-9]+]]}
 // SIMD: [[META18]] = !DISubrange(count: 4)
-// SIMD: [[META19]] = !DIGlobalVariableExpression(var: [[META20:![0-9]+]], expr: !DIExpression())
-// SIMD: [[META20]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG21]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
-// SIMD: [[DBG21]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META23:![0-9]+]], retainedNodes: [[META22]])
-// SIMD: [[META22]] = !{}
-// SIMD: [[META23]] = !{[[META24:![0-9]+]], [[META25:![0-9]+]]}
-// SIMD: [[META24]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
-// SIMD: [[META25]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
+// SIMD: [[META19]] = !{[[META0]]}
+// SIMD: [[META20:![0-9]+]] = !DIGlobalVariableExpression(var: [[META21:![0-9]+]], expr: !DIExpression())
+// SIMD: [[META21]] = distinct !DIGlobalVariable(name: "a", scope: [[DBG22]], file: [[META3]], line: 22, type: [[META6]], isLocal: false, isDefinition: true)
+// SIMD: [[DBG22]] = distinct !DISubprogram(name: "tmain<int, 10>", linkageName: "_Z5tmainIiLi10EEiT_PPc", scope: [[META3]], file: [[META3]], line: 20, type: [[META4]], scopeLine: 20, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]], templateParams: [[META24:![0-9]+]], retainedNodes: [[META23:![0-9]+]])
+// SIMD: [[META23]] = !{[[META20]]}
+// SIMD: [[META24]] = !{[[META25:![0-9]+]], [[META26:![0-9]+]]}
+// SIMD: [[META25]] = !DITemplateTypeParameter(name: "T", type: [[META6]])
+// SIMD: [[META26]] = !DITemplateValueParameter(name: "N", type: [[META6]], value: i32 10)
 // SIMD: [[DBG29]] = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: [[META3]], file: [[META3]], line: 17, type: [[META30:![0-9]+]], scopeLine: 17, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: [[META10]])
 // SIMD: [[META30]] = !DISubroutineType(types: [[META31:![0-9]+]])
 // SIMD: [[META31]] = !{null}
@@ -696,40 +697,40 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG68]] = !DILocation(line: 81, column: 24, scope: [[DBG2]])
 // SIMD: [[DBG69]] = !DILocation(line: 81, column: 3, scope: [[DBG2]])
 // SIMD: [[DBG70]] = !DILocation(line: 82, column: 1, scope: [[DBG2]])
-// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META6]])
-// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG21]])
-// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG21]], file: [[META3]], line: 20, type: [[META7]])
-// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG21]])
-// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG21]])
-// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG21]])
-// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG21]])
-// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG21]])
-// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG21]])
-// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG21]], file: [[META3]], line: 21, type: [[META6]])
-// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG21]])
-// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG21]], file: [[META3]], line: 23, type: [[META15]])
-// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG21]])
-// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG21]], file: [[META3]], line: 24, type: [[META53]])
-// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG21]])
-// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG21]], file: [[META3]], line: 25, type: [[META56]])
-// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG21]])
-// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG21]], file: [[META3]], line: 26, type: [[META8]])
-// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG21]])
-// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG21]])
-// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG21]], file: [[META3]], line: 27, type: [[META62]])
-// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG21]])
-// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG21]])
-// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG21]])
-// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG21]])
-// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG21]])
+// SIMD: [[META71]] = !DILocalVariable(name: "argc", arg: 1, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META6]])
+// SIMD: [[META72]] = !DILocation(line: 20, column: 13, scope: [[DBG22]])
+// SIMD: [[META73]] = !DILocalVariable(name: "argv", arg: 2, scope: [[DBG22]], file: [[META3]], line: 20, type: [[META7]])
+// SIMD: [[META74]] = !DILocation(line: 20, column: 26, scope: [[DBG22]])
+// SIMD: [[META75]] = !DILocalVariable(name: "b", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META76]] = !DILocation(line: 21, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG77]] = !DILocation(line: 21, column: 9, scope: [[DBG22]])
+// SIMD: [[META78]] = !DILocalVariable(name: "c", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META79]] = !DILocation(line: 21, column: 15, scope: [[DBG22]])
+// SIMD: [[META80]] = !DILocalVariable(name: "d", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META81]] = !DILocation(line: 21, column: 18, scope: [[DBG22]])
+// SIMD: [[META82]] = !DILocalVariable(name: "e", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META83]] = !DILocation(line: 21, column: 21, scope: [[DBG22]])
+// SIMD: [[META84]] = !DILocalVariable(name: "f", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META85]] = !DILocation(line: 21, column: 24, scope: [[DBG22]])
+// SIMD: [[META86]] = !DILocalVariable(name: "g", scope: [[DBG22]], file: [[META3]], line: 21, type: [[META6]])
+// SIMD: [[META87]] = !DILocation(line: 21, column: 27, scope: [[DBG22]])
+// SIMD: [[META88]] = !DILocalVariable(name: "str1", scope: [[DBG22]], file: [[META3]], line: 23, type: [[META15]])
+// SIMD: [[META89]] = !DILocation(line: 23, column: 14, scope: [[DBG22]])
+// SIMD: [[META90]] = !DILocalVariable(name: "str2", scope: [[DBG22]], file: [[META3]], line: 24, type: [[META53]])
+// SIMD: [[META91]] = !DILocation(line: 24, column: 15, scope: [[DBG22]])
+// SIMD: [[META92]] = !DILocalVariable(name: "str3", scope: [[DBG22]], file: [[META3]], line: 25, type: [[META56]])
+// SIMD: [[META93]] = !DILocation(line: 25, column: 8, scope: [[DBG22]])
+// SIMD: [[META94]] = !DILocalVariable(name: "str4", scope: [[DBG22]], file: [[META3]], line: 26, type: [[META8]])
+// SIMD: [[META95]] = !DILocation(line: 26, column: 9, scope: [[DBG22]])
+// SIMD: [[DBG96]] = !DILocation(line: 26, column: 16, scope: [[DBG22]])
+// SIMD: [[META97]] = !DILocalVariable(name: "str5", scope: [[DBG22]], file: [[META3]], line: 27, type: [[META62]])
+// SIMD: [[META98]] = !DILocation(line: 27, column: 16, scope: [[DBG22]])
+// SIMD: [[DBG99]] = !DILocation(line: 27, column: 23, scope: [[DBG22]])
+// SIMD: [[DBG100]] = !DILocation(line: 34, column: 7, scope: [[DBG22]])
+// SIMD: [[DBG101]] = !DILocation(line: 34, column: 5, scope: [[DBG22]])
+// SIMD: [[DBG102]] = !DILocation(line: 35, column: 3, scope: [[DBG22]])
 // SIMD: [[META103]] = !DILocalVariable(name: "b", scope: [[META104:![0-9]+]], file: [[META3]], line: 43, type: [[META6]])
-// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG21]], file: [[META3]], line: 42, column: 3)
+// SIMD: [[META104]] = distinct !DILexicalBlock(scope: [[DBG22]], file: [[META3]], line: 42, column: 3)
 // SIMD: [[META105]] = !DILocation(line: 43, column: 9, scope: [[META104]])
 // SIMD: [[META106]] = !DILocalVariable(name: "c", scope: [[META104]], file: [[META3]], line: 44, type: [[META6]])
 // SIMD: [[META107]] = !DILocation(line: 44, column: 7, scope: [[META104]])
@@ -737,6 +738,6 @@ int main (int argc, char **argv) {
 // SIMD: [[DBG109]] = !DILocation(line: 45, column: 13, scope: [[META104]])
 // SIMD: [[DBG110]] = !DILocation(line: 45, column: 11, scope: [[META104]])
 // SIMD: [[DBG111]] = !DILocation(line: 45, column: 7, scope: [[META104]])
-// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG21]])
-// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG21]])
+// SIMD: [[DBG112]] = !DILocation(line: 53, column: 3, scope: [[DBG22]])
+// SIMD: [[DBG113]] = !DILocation(line: 54, column: 1, scope: [[DBG22]])
 //.

diff  --git a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
index 7a8569975c82b..ecd076c811dcc 100644
--- a/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
+++ b/compiler-rt/test/profile/Linux/instrprof-debug-info-correlate-warnings.ll
@@ -21,7 +21,8 @@ void no_name() {}
 void no_cfg() {}
 void no_counter() {}
 void no_profc() {}
-void no_func() {}
+__attribute__((always_inline)) void no_func() {}
+void use_func() { no_func(); }
 
 // NOTE: After generating the IR below, manually remove the follwing pieces
 // 1. Remove "@removed" function and "@__profc_removed" global
@@ -55,6 +56,8 @@ $__profc_no_profc = comdat nodeduplicate
 
 $__profc_no_func = comdat nodeduplicate
 
+$__profc_use_func = comdat nodeduplicate
+
 $__llvm_profile_filename = comdat any
 
 @__llvm_profile_raw_version = hidden constant i64 648518346341351434, comdat
@@ -65,119 +68,153 @@ $__llvm_profile_filename = comdat any
 @__profn_no_counter = private constant [10 x i8] c"no_counter"
 @__profn_no_profc = private constant [8 x i8] c"no_profc"
 @__profn_no_func = private constant [7 x i8] c"no_func"
+ at __profn_use_func = private constant [8 x i8] c"use_func"
 @__profc_main = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !0
- at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !19
- at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !24
- at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !29
- at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !39
+ at __profc_no_name = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !22
+ at __profc_no_cfg = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !28
+ at __profc_no_counter = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !34
+ at __profc_no_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !46
+ at __profc_use_func = private global [1 x i64] zeroinitializer, section "__llvm_prf_cnts", comdat, align 8, !dbg !52
 @llvm.compiler.used = appending global [5 x ptr] [ptr @__profc_main, ptr @__profc_no_name, ptr @__profc_no_cfg, ptr @__profc_no_counter, ptr @__profc_no_func], section "llvm.metadata"
 @__llvm_profile_filename = hidden constant [20 x i8] c"default_%m.proflite\00", comdat
 
 ; Function Attrs: noinline nounwind optnone uwtable
 define dso_local i32 @main() #0 !dbg !2 {
-  %1 = alloca i32, align 4
-  %2 = load i64, ptr @__profc_main, align 8
-  %3 = add i64 %2, 1
-  store i64 %3, ptr @__profc_main, align 8
-  store i32 0, ptr %1, align 4
-  ret i32 0, !dbg !53
+entry:
+  %retval = alloca i32, align 4
+  %pgocount = load i64, ptr @__profc_main, align 8
+  %0 = add i64 %pgocount, 1
+  store i64 %0, ptr @__profc_main, align 8
+  store i32 0, ptr %retval, align 4
+  ret i32 0, !dbg !64
+}
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local void @no_name() #0 !dbg !24 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_name, align 8, !dbg !66
+  %0 = add i64 %pgocount, 1, !dbg !66
+  store i64 %0, ptr @__profc_no_name, align 8, !dbg !66
+  ret void, !dbg !66
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_name() #0 !dbg !21 {
-  %1 = load i64, ptr @__profc_no_name, align 8, !dbg !55
-  %2 = add i64 %1, 1, !dbg !55
-  store i64 %2, ptr @__profc_no_name, align 8, !dbg !55
-  ret void, !dbg !55
+define dso_local void @no_cfg() #0 !dbg !30 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_cfg, align 8, !dbg !67
+  %0 = add i64 %pgocount, 1, !dbg !67
+  store i64 %0, ptr @__profc_no_cfg, align 8, !dbg !67
+  ret void, !dbg !67
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_cfg() #0 !dbg !26 {
-  %1 = load i64, ptr @__profc_no_cfg, align 8, !dbg !56
-  %2 = add i64 %1, 1, !dbg !56
-  store i64 %2, ptr @__profc_no_cfg, align 8, !dbg !56
-  ret void, !dbg !56
+define dso_local void @no_counter() #0 !dbg !36 {
+entry:
+  %pgocount = load i64, ptr @__profc_no_counter, align 8, !dbg !68
+  %0 = add i64 %pgocount, 1, !dbg !68
+  store i64 %0, ptr @__profc_no_counter, align 8, !dbg !68
+  ret void, !dbg !68
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_counter() #0 !dbg !31 {
-  %1 = load i64, ptr @__profc_no_counter, align 8, !dbg !57
-  %2 = add i64 %1, 1, !dbg !57
-  store i64 %2, ptr @__profc_no_counter, align 8, !dbg !57
-  ret void, !dbg !57
+define dso_local void @no_profc() #0 !dbg !42 {
+entry:
+  ret void, !dbg !69
 }
 
 ; Function Attrs: noinline nounwind optnone uwtable
-define dso_local void @no_profc() #0 !dbg !36 {
-  ret void, !dbg !58
+define dso_local void @use_func() #0 !dbg !54 {
+entry:
+  %pgocount = load i64, ptr @__profc_use_func, align 8, !dbg !71
+  %0 = add i64 %pgocount, 1, !dbg !71
+  store i64 %0, ptr @__profc_use_func, align 8, !dbg !71
+  %pgocount.i = load i64, ptr @__profc_no_func, align 8, !dbg !72
+  %1 = add i64 %pgocount.i, 1, !dbg !72
+  store i64 %1, ptr @__profc_no_func, align 8, !dbg !72
+  ret void, !dbg !74
 }
 
 ; Function Attrs: nounwind
-declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #1
+declare void @llvm.instrprof.increment(ptr, i64, i32, i32) #2
 
 attributes #0 = { noinline nounwind optnone uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
-attributes #1 = { nounwind }
+attributes #1 = { alwaysinline nounwind uwtable "frame-pointer"="all" "min-legal-vector-width"="0" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "target-cpu"="x86-64" "target-features"="+cmov,+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "tune-cpu"="generic" }
+attributes #2 = { nounwind }
 
 !llvm.dbg.cu = !{!7}
-!llvm.module.flags = !{!46, !47, !48, !49, !50, !51, !52}
+!llvm.module.flags = !{!58, !59, !60, !61, !62, !63}
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
-!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !44)
-!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7)
-!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "22eee0eada6e6964fca794aa5a0966d0")
+!1 = distinct !DIGlobalVariable(name: "__profc_main", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !10)
+!2 = distinct !DISubprogram(name: "main", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "a.c", directory: "/proc/self/cwd", checksumkind: CSK_MD5, checksum: "7bab6089d746f793d8c0c8d39f3a691c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !19, !24, !29, !34, !39}
-!9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression())
-!10 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !11, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !15)
-!11 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !12, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7)
-!12 = !DISubroutineType(types: !13)
-!13 = !{null}
-!14 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
-!15 = !{!16, !17, !18}
-!16 = !{!"Function Name", !"removed"}
-!17 = !{!"CFG Hash", i64 742261418966908927}
-!18 = !{!"Num Counters", i32 1}
-!19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
-!20 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !21, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !22)
-!21 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !12, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7)
-!22 = !{!17, !18}
-!23 = !{!"Function Name", !"no_name"}
-!24 = !DIGlobalVariableExpression(var: !25, expr: !DIExpression())
-!25 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !26, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !27)
-!26 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !12, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7)
-!27 = !{!28, !18}
-!28 = !{!"Function Name", !"no_cfg"}
-!29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression())
-!30 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !31, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !32)
-!31 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !12, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7)
-!32 = !{!33, !17}
-!33 = !{!"Function Name", !"no_counter"}
+!7 = distinct !DICompileUnit(language: DW_LANG_C11, file: !3, isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!8 = !{!0}
+!9 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "Profile Data Type")
+!10 = !{!11, !12, !13}
+!11 = !{!"Function Name", !"main"}
+!12 = !{!"CFG Hash", i64 742261418966908927}
+!13 = !{!"Num Counters", i32 1}
+!14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
+!15 = distinct !DIGlobalVariable(name: "__profc_removed", scope: !16, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !20)
+!16 = distinct !DISubprogram(name: "removed", scope: !3, file: !3, line: 3, type: !17, scopeLine: 3, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !19)
+!17 = !DISubroutineType(types: !18)
+!18 = !{null}
+!19 = !{!14}
+!20 = !{!21, !12, !13}
+!21 = !{!"Function Name", !"removed"}
+!22 = !DIGlobalVariableExpression(var: !23, expr: !DIExpression())
+!23 = distinct !DIGlobalVariable(name: "__profc_no_name", scope: !24, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !26)
+!24 = distinct !DISubprogram(name: "no_name", scope: !3, file: !3, line: 4, type: !17, scopeLine: 4, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !25)
+!25 = !{!22}
+!26 = !{!12, !13}
+!28 = !DIGlobalVariableExpression(var: !29, expr: !DIExpression())
+!29 = distinct !DIGlobalVariable(name: "__profc_no_cfg", scope: !30, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !32)
+!30 = distinct !DISubprogram(name: "no_cfg", scope: !3, file: !3, line: 5, type: !17, scopeLine: 5, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !31)
+!31 = !{!28}
+!32 = !{!33, !13}
+!33 = !{!"Function Name", !"no_cfg"}
 !34 = !DIGlobalVariableExpression(var: !35, expr: !DIExpression())
-!35 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !36, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !37)
-!36 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !12, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7)
-!37 = !{!38, !17, !18}
-!38 = !{!"Function Name", !"no_profc"}
-!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
-!40 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !41, file: !3, type: !14, isLocal: true, isDefinition: true, annotations: !42)
-!41 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !12, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7)
-!42 = !{!43, !17, !18}
-!43 = !{!"Function Name", !"no_func"}
-!44 = !{!45, !17, !18}
-!45 = !{!"Function Name", !"main"}
-!46 = !{i32 7, !"Dwarf Version", i32 5}
-!47 = !{i32 2, !"Debug Info Version", i32 3}
-!48 = !{i32 1, !"wchar_size", i32 4}
-!49 = !{i32 8, !"PIC Level", i32 2}
-!50 = !{i32 7, !"PIE Level", i32 2}
-!51 = !{i32 7, !"uwtable", i32 2}
-!52 = !{i32 7, !"frame-pointer", i32 2}
-!53 = !DILocation(line: 1, column: 14, scope: !2)
-!54 = !DILocation(line: 3, column: 17, scope: !11)
-!55 = !DILocation(line: 4, column: 17, scope: !21)
-!56 = !DILocation(line: 5, column: 16, scope: !26)
-!57 = !DILocation(line: 6, column: 20, scope: !31)
-!58 = !DILocation(line: 7, column: 18, scope: !36)
-!59 = !DILocation(line: 8, column: 17, scope: !41)
+!35 = distinct !DIGlobalVariable(name: "__profc_no_counter", scope: !36, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !38)
+!36 = distinct !DISubprogram(name: "no_counter", scope: !3, file: !3, line: 6, type: !17, scopeLine: 6, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !37)
+!37 = !{!34}
+!38 = !{!39, !12}
+!39 = !{!"Function Name", !"no_counter"}
+!40 = !DIGlobalVariableExpression(var: !41, expr: !DIExpression())
+!41 = distinct !DIGlobalVariable(name: "__profc_no_profc", scope: !42, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !44)
+!42 = distinct !DISubprogram(name: "no_profc", scope: !3, file: !3, line: 7, type: !17, scopeLine: 7, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !43)
+!43 = !{!40}
+!44 = !{!45, !12, !13}
+!45 = !{!"Function Name", !"no_profc"}
+!46 = !DIGlobalVariableExpression(var: !47, expr: !DIExpression())
+!47 = distinct !DIGlobalVariable(name: "__profc_no_func", scope: !48, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !50)
+!48 = distinct !DISubprogram(name: "no_func", scope: !3, file: !3, line: 8, type: !17, scopeLine: 8, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !49)
+!49 = !{!46}
+!50 = !{!51, !12, !13}
+!51 = !{!"Function Name", !"no_func"}
+!52 = !DIGlobalVariableExpression(var: !53, expr: !DIExpression())
+!53 = distinct !DIGlobalVariable(name: "__profc_use_func", scope: !54, file: !3, type: !9, isLocal: true, isDefinition: true, annotations: !56)
+!54 = distinct !DISubprogram(name: "use_func", scope: !3, file: !3, line: 9, type: !17, scopeLine: 9, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !55)
+!55 = !{!52}
+!56 = !{!57, !12, !13}
+!57 = !{!"Function Name", !"use_func"}
+!58 = !{i32 7, !"Dwarf Version", i32 5}
+!59 = !{i32 2, !"Debug Info Version", i32 3}
+!60 = !{i32 8, !"PIC Level", i32 2}
+!61 = !{i32 7, !"PIE Level", i32 2}
+!62 = !{i32 7, !"uwtable", i32 2}
+!63 = !{i32 7, !"frame-pointer", i32 2}
+!64 = !DILocation(line: 1, column: 14, scope: !2)
+!65 = !DILocation(line: 3, column: 17, scope: !16)
+!66 = !DILocation(line: 4, column: 17, scope: !24)
+!67 = !DILocation(line: 5, column: 16, scope: !30)
+!68 = !DILocation(line: 6, column: 20, scope: !36)
+!69 = !DILocation(line: 7, column: 18, scope: !42)
+!70 = !DILocation(line: 8, column: 48, scope: !48)
+!71 = !DILocation(line: 9, column: 19, scope: !54)
+!72 = !DILocation(line: 8, column: 48, scope: !48, inlinedAt: !73)
+!73 = distinct !DILocation(line: 9, column: 19, scope: !54)
+!74 = !DILocation(line: 9, column: 30, scope: !54)

diff  --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 484d5df4c162e..7c0b32e48832e 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -665,7 +665,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   }
 
   auto addTargetOpDISP = [&](bool lineTableOnly,
-                             llvm::ArrayRef<mlir::LLVM::DINodeAttr> entities) {
+                             llvm::ArrayRef<mlir::Attribute> entities) {
     // When we process the DeclareOp inside the OpenMP target region, all the
     // variables get the DISubprogram of the parent function of the target op as
     // the scope. In the codegen (to llvm ir), OpenMP target op results in the
@@ -717,8 +717,8 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
 
       // Make sure that information about the imported modules is copied in the
       // new function.
-      llvm::SmallVector<mlir::LLVM::DINodeAttr> opEntities;
-      for (mlir::LLVM::DINodeAttr N : entities) {
+      llvm::SmallVector<mlir::Attribute> opEntities;
+      for (mlir::Attribute N : entities) {
         if (auto entity = mlir::dyn_cast<mlir::LLVM::DIImportedEntityAttr>(N)) {
           auto importedEntity = mlir::LLVM::DIImportedEntityAttr::get(
               context, entity.getTag(), spAttr, entity.getEntity(),
@@ -756,7 +756,7 @@ void AddDebugInfoPass::handleFuncOp(mlir::func::FuncOp funcOp,
   });
 
   mlir::LLVM::DISubprogramAttr spAttr;
-  llvm::SmallVector<mlir::LLVM::DINodeAttr> retainedNodes;
+  llvm::SmallVector<mlir::Attribute> retainedNodes;
 
   if (hasUseStmts) {
     mlir::DistinctAttr recId =

diff  --git a/llvm/include/llvm/IR/DIBuilder.h b/llvm/include/llvm/IR/DIBuilder.h
index e0573fb4081d4..20d75adfdb9f8 100644
--- a/llvm/include/llvm/IR/DIBuilder.h
+++ b/llvm/include/llvm/IR/DIBuilder.h
@@ -53,7 +53,7 @@ namespace llvm {
     /// Track the RetainTypes, since they can be updated later on.
     SmallVector<TrackingMDNodeRef, 4> AllRetainTypes;
     SmallVector<DISubprogram *, 4> AllSubprograms;
-    SmallVector<Metadata *, 4> AllGVs;
+    SmallVector<Metadata *, 4> Globals;
     SmallVector<TrackingMDNodeRef, 4> ImportedModules;
     /// Map Macro parent (which can be DIMacroFile or nullptr) to a list of
     /// Metadata all of type DIMacroNode.
@@ -64,8 +64,8 @@ namespace llvm {
     SmallVector<TrackingMDNodeRef, 4> UnresolvedNodes;
     bool AllowUnresolvedNodes;
 
-    /// Each subprogram's preserved local variables, labels, imported entities,
-    /// and types.
+    /// Each subprogram's preserved local and static local variables, labels,
+    /// imported entities, and types.
     ///
     /// Do not use a std::vector.  Some versions of libc++ apparently copy
     /// instead of move on grow operations, and TrackingMDRef is expensive to

diff  --git a/llvm/include/llvm/IR/DebugInfo.h b/llvm/include/llvm/IR/DebugInfo.h
index 01f8e97d24bce..18b94f8f66e48 100644
--- a/llvm/include/llvm/IR/DebugInfo.h
+++ b/llvm/include/llvm/IR/DebugInfo.h
@@ -124,6 +124,7 @@ class DebugInfoFinder {
 
 private:
   void processCompileUnit(DICompileUnit *CU);
+  void processGlobalVariableExpression(DIGlobalVariableExpression *GVE);
   void processScope(DIScope *Scope);
   void processType(DIType *DT);
   void processVariable(DIVariable *DV);

diff  --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 6dfbb2c7e1bc5..846342bee6071 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -2339,7 +2339,7 @@ class DISubprogram : public DILocalScope {
           unsigned VirtualIndex, int ThisAdjustment, DIFlags Flags,
           DISPFlags SPFlags, DICompileUnit *Unit,
           DITemplateParameterArray TemplateParams, DISubprogram *Declaration,
-          DINodeArray RetainedNodes, DITypeArray ThrownTypes,
+          MDNodeArray RetainedNodes, DITypeArray ThrownTypes,
           DINodeArray Annotations, StringRef TargetFuncName,
           bool UsesKeyInstructions, StorageType Storage,
           bool ShouldCreate = true) {
@@ -2380,7 +2380,7 @@ class DISubprogram : public DILocalScope {
        DIType *ContainingType, unsigned VirtualIndex, int ThisAdjustment,
        DIFlags Flags, DISPFlags SPFlags, DICompileUnit *Unit,
        DITemplateParameterArray TemplateParams = nullptr,
-       DISubprogram *Declaration = nullptr, DINodeArray RetainedNodes = nullptr,
+       DISubprogram *Declaration = nullptr, MDNodeArray RetainedNodes = nullptr,
        DITypeArray ThrownTypes = nullptr, DINodeArray Annotations = nullptr,
        StringRef TargetFuncName = "", bool UsesKeyInstructions = false),
       (Scope, Name, LinkageName, File, Line, Type, ScopeLine, ContainingType,
@@ -2509,7 +2509,7 @@ class DISubprogram : public DILocalScope {
     return cast_or_null<DISubprogram>(getRawDeclaration());
   }
   void replaceDeclaration(DISubprogram *Decl) { replaceOperandWith(6, Decl); }
-  DINodeArray getRetainedNodes() const {
+  MDNodeArray getRetainedNodes() const {
     return cast_or_null<MDTuple>(getRawRetainedNodes());
   }
   DITypeArray getThrownTypes() const {
@@ -2548,19 +2548,24 @@ class DISubprogram : public DILocalScope {
   void replaceRawLinkageName(MDString *LinkageName) {
     replaceOperandWith(3, LinkageName);
   }
-  void replaceRetainedNodes(DINodeArray N) {
-    replaceOperandWith(7, N.get());
+  void replaceRetainedNodes(MDNodeArray N) { replaceOperandWith(7, N.get()); }
+
+  template <typename IterT> void retainNodes(IterT NodesBegin, IterT NodesEnd) {
+    auto RetainedNodes = getRetainedNodes();
+    SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
+    MDs.append(NodesBegin, NodesEnd);
+    replaceRetainedNodes(MDNode::get(getContext(), MDs));
   }
 
   /// For the given retained node of DISubprogram, applies one of the
   /// given functions depending on the type of the node.
   template <typename T, typename MetadataT, typename FuncLVT,
             typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT, typename FuncUnknownT>
+            typename FuncTypeT, typename FuncGVET, typename FuncUnknownT>
   static T visitRetainedNode(MetadataT *N, FuncLVT &&FuncLV,
                              FuncLabelT &&FuncLabel,
                              FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
-                             FuncUnknownT &&FuncUnknown) {
+                             FuncGVET &&FuncGVE, FuncUnknownT &&FuncUnknown) {
     static_assert(std::is_base_of_v<Metadata, MetadataT>,
                   "N must point to Metadata or const Metadata");
 
@@ -2572,6 +2577,8 @@ class DISubprogram : public DILocalScope {
       return FuncIE(IE);
     if (auto *Ty = dyn_cast<DIType>(N))
       return FuncType(Ty);
+    if (auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      return FuncGVE(GVE);
     return FuncUnknown(N);
   }
 
@@ -2585,12 +2592,13 @@ class DISubprogram : public DILocalScope {
   /// For each retained node, applies one of the given functions depending
   /// on the type of a node.
   template <typename FuncLVT, typename FuncLabelT, typename FuncImportedEntityT,
-            typename FuncTypeT>
+            typename FuncTypeT, typename FuncGVET>
   void forEachRetainedNode(FuncLVT &&FuncLV, FuncLabelT &&FuncLabel,
-                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType) {
+                           FuncImportedEntityT &&FuncIE, FuncTypeT &&FuncType,
+                           FuncGVET &&FuncGVE) {
     for (MDNode *N : getRetainedNodes())
       visitRetainedNode<void>(
-          N, FuncLV, FuncLabel, FuncIE, FuncType,
+          N, FuncLV, FuncLabel, FuncIE, FuncType, FuncGVE,
           [](auto *N) { llvm_unreachable("Unexpected retained node!"); });
   }
 
@@ -2622,6 +2630,17 @@ class DISubprogram : public DILocalScope {
   /// of a current scope for expression evaluation.
   LLVM_ABI void cleanupRetainedNodes();
 
+  template <typename T> void cleanupRetainedNodesIf(T &&Pred) {
+    MDTuple *RetainedNodes = dyn_cast_or_null<MDTuple>(getRawRetainedNodes());
+    // As this is expected to be called during module loading, before
+    // stripping old or incorrect debug info, perform minimal sanity check.
+    if (!RetainedNodes)
+      return;
+    // replaceRetainedNodes() should not re-unique DISubprogram if new list is
+    // the same pointer.
+    replaceRetainedNodes(RetainedNodes->filter(Pred));
+  }
+
   /// Calls SP->cleanupRetainedNodes() for a range of DISubprograms.
   template <typename RangeT>
   static void cleanupRetainedNodes(const RangeT &NewDistinctSPs) {
@@ -4817,6 +4836,24 @@ class DebugVariableAggregate : public DebugVariable {
 template <>
 struct DenseMapInfo<DebugVariableAggregate>
     : public DenseMapInfo<DebugVariable> {};
+
+template <typename NodeT> static const DIScope *getScope(const NodeT *N) {
+  return N->getScope();
+}
+
+template <typename NodeT> static DIScope *getScope(NodeT *N) {
+  return N->getScope();
+}
+
+template <>
+[[maybe_unused]] const DIScope *
+getScope<>(const DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
+template <>
+[[maybe_unused]] DIScope *getScope<>(DIGlobalVariableExpression *N) {
+  return N->getVariable()->getScope();
+}
 } // end namespace llvm
 
 #undef DEFINE_MDNODE_GET_UNPACK_IMPL

diff  --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index 73adeb3416ab1..c0d0703135f68 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -1546,6 +1546,17 @@ class MDTuple : public MDNode {
   /// Shrink the operands by 1.
   void pop_back() { resize(getNumOperands() - 1); }
 
+  /// Filter out tuple elements that do not satisfy predicate.
+  /// Return this if no elements should be filtered out (without re-uniquing).
+  template <typename T> MDTuple *filter(T &&Pred) {
+    ArrayRef<MDOperand> Ops = operands();
+    // Exit if no nodes should be removed.
+    if (llvm::all_of(Ops, Pred))
+      return this;
+    return get(getContext(),
+               to_vector_of<Metadata *>(llvm::make_filter_range(Ops, Pred)));
+  }
+
   static bool classof(const Metadata *MD) {
     return MD->getMetadataID() == MDTupleKind;
   }

diff  --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 33062a0256bb8..40bc930d124f0 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -546,85 +546,64 @@ class MetadataLoader::MetadataLoaderImpl {
                llvm::dyn_cast_or_null<DISubprogram>(S);
   }
 
-  /// Move local imports from DICompileUnit's 'imports' field to
-  /// DISubprogram's retainedNodes.
-  /// Move function-local enums from DICompileUnit's enums
-  /// to DISubprogram's retainedNodes.
+  /// Map SP -> {Metadata} to store CU locals that should be attached to
+  /// subprogram retainedNodes list during CU upgrade.
+  using SPToEntitiesMap =
+      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>>;
+
+  /// Retrieve the CU operand at position ListIndex, treat it as an MDTuple, and
+  /// remove all local debug info nodes from it. Fill SPToEntities map with
+  /// removed local nodes.
+  template <typename NodeT>
+  void upgradeOneCULocalsList(SPToEntitiesMap &SPToEntities, DICompileUnit *CU,
+                              unsigned ListIndex) {
+    MDTuple *List = cast_if_present<MDTuple>(CU->getOperand(ListIndex));
+    if (!List)
+      return;
+
+    if (llvm::all_of(List->operands(), [](Metadata *MD) {
+          return !isa_and_nonnull<DILocalScope>(getScope(cast<NodeT>(MD)));
+        }))
+      return;
+
+    SmallVector<Metadata *> MDs;
+    for (Metadata *MD : List->operands()) {
+      DILocalScope *LS =
+          dyn_cast_or_null<DILocalScope>(getScope(cast<NodeT>(MD)));
+      if (!LS)
+        MDs.push_back(MD);
+      else if (auto *SP = findEnclosingSubprogram(LS))
+        SPToEntities[SP].push_back(MD);
+    }
+
+    CU->replaceOperandWith(ListIndex, MDNode::get(CU->getContext(), MDs));
+  }
+
+  /// Move function-local entities from DICompileUnit's 'imports',
+  /// 'enums', and 'globals' fields to DISubprogram's retainedNodes.
   void upgradeCULocals() {
     NamedMDNode *CUNodes = TheModule.getNamedMetadata("llvm.dbg.cu");
     if (!CUNodes)
       return;
 
-    // Filter out elements of ToRemove from tuple T.
-    auto FilterTuple = [this](MDNode *T,
-                              const SetVector<Metadata *> &ToRemove) {
-      SmallVector<Metadata *> Result;
-      for (Metadata *Op : T->operands())
-        if (!ToRemove.contains(Op))
-          Result.push_back(Op);
-      return MDTuple::get(Context, Result);
-    };
-
-    // For each CU:
-    // - Collect local metadata nodes from CU's imports: and enums: lists in
-    //   MetadataToRemove set.
-    // - Remove metadata nodes of MetadataToRemove set from CU's imports: and
-    //   enums: lists.
-    // - Group MetadataToRemove items by their parent subprograms (in
-    //   SPToEntities map).
-    // - For each subprogram SP in SPToEntities:
-    //   - Append collected local metadata nodes to SP's retainedNodes: list.
+    SPToEntitiesMap SPToEntities;
     for (MDNode *N : CUNodes->operands()) {
       auto *CU = dyn_cast<DICompileUnit>(N);
       if (!CU)
         continue;
 
-      SetVector<Metadata *> MetadataToRemove;
-      // Collect imported entities to be moved.
-      if (CU->getRawImportedEntities())
-        for (Metadata *Op : CU->getImportedEntities()->operands()) {
-          auto *IE = cast<DIImportedEntity>(Op);
-          if (isa_and_nonnull<DILocalScope>(IE->getScope()))
-            MetadataToRemove.insert(IE);
-        }
-      // Collect enums to be moved.
-      if (CU->getRawEnumTypes())
-        for (Metadata *Op : CU->getEnumTypes()->operands()) {
-          auto *Enum = cast<DICompositeType>(Op);
-          if (isa_and_nonnull<DILocalScope>(Enum->getScope()))
-            MetadataToRemove.insert(Enum);
-        }
-
-      if (MetadataToRemove.empty())
-        continue;
-
-      // Remove entities with local scope from CU.
-      if (CU->getRawImportedEntities())
-        CU->replaceImportedEntities(
-            FilterTuple(CU->getImportedEntities().get(), MetadataToRemove));
-
-      // Remove enums with local scope from CU.
-      if (CU->getRawEnumTypes())
-        CU->replaceEnumTypes(
-            FilterTuple(CU->getEnumTypes().get(), MetadataToRemove));
-
-      // Find DISubprogram corresponding to each entity.
-      SmallDenseMap<DISubprogram *, SmallVector<Metadata *>> SPToEntities;
-      for (auto *I : MetadataToRemove) {
-        DILocalScope *Scope =
-            DISubprogram::getRetainedNodeScope(cast<DINode>(I));
-        if (auto *SP = findEnclosingSubprogram(Scope))
-          SPToEntities[SP].push_back(I);
-      }
-
-      // Update DISubprograms' retainedNodes.
-      for (auto I = SPToEntities.begin(); I != SPToEntities.end(); ++I) {
-        auto *SP = I->first;
-        auto RetainedNodes = SP->getRetainedNodes();
-        SmallVector<Metadata *> MDs(RetainedNodes.begin(), RetainedNodes.end());
-        MDs.append(I->second);
-        SP->replaceRetainedNodes(MDNode::get(Context, MDs));
-      }
+      // Remove all static local variables from CU's globals list.
+      upgradeOneCULocalsList<DIGlobalVariableExpression>(SPToEntities, CU, 6);
+      // Remove all local imports from CU's imports list.
+      upgradeOneCULocalsList<DIImportedEntity>(SPToEntities, CU, 7);
+      // Remove all local types from CU's enums list.
+      upgradeOneCULocalsList<DICompositeType>(SPToEntities, CU, 4);
+
+      // Retain local entities removed from the CU in their corresponding
+      // subprograms.
+      for (auto &[SP, Nodes] : SPToEntities)
+        SP->retainNodes(Nodes.begin(), Nodes.end());
+      SPToEntities.clear();
     }
 
     ParentSubprogram.clear();

diff  --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
index 06353a0e039db..193379d7170d6 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp
@@ -1451,6 +1451,10 @@ void CodeViewDebug::collectVariableInfo(const DISubprogram *SP) {
   // Grab the variable info that was squirreled away in the MMI side-table.
   collectVariableInfoFromMFTable(Processed);
 
+  for (const MDNode *N : SP->getRetainedNodes())
+    if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(N))
+      collectGlobalOrStaticLocalVariableInfo(GVE);
+
   for (const auto &I : DbgValues) {
     InlinedEntity IV = I.first;
     if (Processed.count(IV))
@@ -3236,9 +3240,54 @@ void CodeViewDebug::emitDebugInfoForUDTs(
   }
 }
 
+void CodeViewDebug::collectGlobalOrStaticLocalVariableInfo(
+    const DIGlobalVariableExpression *GVE) {
+  const DIGlobalVariable *DIGV = GVE->getVariable();
+  const DIExpression *DIE = GVE->getExpression();
+  // Don't emit string literals in CodeView, as the only useful parts are
+  // generally the filename and line number, which isn't possible to output
+  // in CodeView. String literals should be the only unnamed GlobalVariable
+  // with debug info.
+  if (DIGV->getName().empty())
+    return;
+
+  if ((DIE->getNumElements() == 2) &&
+      (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
+    // Record the constant offset for the variable.
+    //
+    // A Fortran common block uses this idiom to encode the offset
+    // of a variable from the common block's starting address.
+    CVGlobalVariableOffsets.insert(std::make_pair(DIGV, DIE->getElement(1)));
+
+  // Emit constant global variables in a global symbol section.
+  if (!GlobalMap.count(GVE) && DIE->isConstant())
+    GlobalVariables.emplace_back(CVGlobalVariable{DIGV, DIE});
+
+  const auto *GV = GlobalMap.lookup(GVE);
+  if (!GV || GV->isDeclarationForLinker())
+    return;
+
+  DIScope *Scope = DIGV->getScope();
+  SmallVector<CVGlobalVariable, 1> *VariableList;
+  if (Scope && isa<DILocalScope>(Scope)) {
+    // Locate a global variable list for this scope, creating one if
+    // necessary.
+    auto Insertion =
+        ScopeGlobals.insert({Scope, std::unique_ptr<GlobalVariableList>()});
+    if (Insertion.second)
+      Insertion.first->second = std::make_unique<GlobalVariableList>();
+    VariableList = Insertion.first->second.get();
+  } else if (GV->hasComdat()) {
+    // Emit this global variable into a COMDAT section.
+    VariableList = &ComdatVariables;
+  } else {
+    // Emit this global variable in a single global symbol section.
+    VariableList = &GlobalVariables;
+  }
+  VariableList->emplace_back(CVGlobalVariable{DIGV, GV});
+}
+
 void CodeViewDebug::collectGlobalVariableInfo() {
-  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
-      GlobalMap;
   for (const GlobalVariable &GV : MMI->getModule()->globals()) {
     SmallVector<DIGlobalVariableExpression *, 1> GVEs;
     GV.getDebugInfo(GVEs);
@@ -3250,51 +3299,7 @@ void CodeViewDebug::collectGlobalVariableInfo() {
   for (const MDNode *Node : CUs->operands()) {
     const auto *CU = cast<DICompileUnit>(Node);
     for (const auto *GVE : CU->getGlobalVariables()) {
-      const DIGlobalVariable *DIGV = GVE->getVariable();
-      const DIExpression *DIE = GVE->getExpression();
-      // Don't emit string literals in CodeView, as the only useful parts are
-      // generally the filename and line number, which isn't possible to output
-      // in CodeView. String literals should be the only unnamed GlobalVariable
-      // with debug info.
-      if (DIGV->getName().empty()) continue;
-
-      if ((DIE->getNumElements() == 2) &&
-          (DIE->getElement(0) == dwarf::DW_OP_plus_uconst))
-        // Record the constant offset for the variable.
-        //
-        // A Fortran common block uses this idiom to encode the offset
-        // of a variable from the common block's starting address.
-        CVGlobalVariableOffsets.insert(
-            std::make_pair(DIGV, DIE->getElement(1)));
-
-      // Emit constant global variables in a global symbol section.
-      if (GlobalMap.count(GVE) == 0 && DIE->isConstant()) {
-        CVGlobalVariable CVGV = {DIGV, DIE};
-        GlobalVariables.emplace_back(std::move(CVGV));
-      }
-
-      const auto *GV = GlobalMap.lookup(GVE);
-      if (!GV || GV->isDeclarationForLinker())
-        continue;
-
-      DIScope *Scope = DIGV->getScope();
-      SmallVector<CVGlobalVariable, 1> *VariableList;
-      if (Scope && isa<DILocalScope>(Scope)) {
-        // Locate a global variable list for this scope, creating one if
-        // necessary.
-        auto Insertion = ScopeGlobals.insert(
-            {Scope, std::unique_ptr<GlobalVariableList>()});
-        if (Insertion.second)
-          Insertion.first->second = std::make_unique<GlobalVariableList>();
-        VariableList = Insertion.first->second.get();
-      } else if (GV->hasComdat())
-        // Emit this global variable into a COMDAT section.
-        VariableList = &ComdatVariables;
-      else
-        // Emit this global variable in a single global symbol section.
-        VariableList = &GlobalVariables;
-      CVGlobalVariable CVGV = {DIGV, GV};
-      VariableList->emplace_back(std::move(CVGV));
+      collectGlobalOrStaticLocalVariableInfo(GVE);
     }
   }
 }

diff  --git a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
index 04a75a21d7679..e28e0ab6da36b 100644
--- a/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
+++ b/llvm/lib/CodeGen/AsmPrinter/CodeViewDebug.h
@@ -252,6 +252,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   // belong in. A null local scope represents the global scope.
   typedef SmallVector<CVGlobalVariable, 1> GlobalVariableList;
   DenseMap<const DIScope*, std::unique_ptr<GlobalVariableList> > ScopeGlobals;
+  DenseMap<const DIGlobalVariableExpression *, const GlobalVariable *>
+      GlobalMap;
 
   // Array of global variables which  need to be emitted into a COMDAT section.
   SmallVector<CVGlobalVariable, 1> ComdatVariables;
@@ -404,6 +406,8 @@ class LLVM_LIBRARY_VISIBILITY CodeViewDebug : public DebugHandlerBase {
   using InlinedEntity = DbgValueHistoryMap::InlinedEntity;
 
   void collectGlobalVariableInfo();
+  void
+  collectGlobalOrStaticLocalVariableInfo(const DIGlobalVariableExpression *GVE);
   void collectVariableInfo(const DISubprogram *SP);
 
   void collectVariableInfoFromMFTable(DenseSet<InlinedEntity> &Processed);

diff  --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfd5dfa746c70..cfbb35d516534 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -1522,6 +1522,8 @@ void DwarfDebug::endModule() {
     DenseSet<DIGlobalVariable *> Processed;
     for (auto *GVE : CUNode->getGlobalVariables()) {
       DIGlobalVariable *GV = GVE->getVariable();
+      assert(!isa_and_nonnull<DILocalScope>(GV->getScope()) &&
+             "Unexpected function-local entity in 'globals' CU field.");
       if (Processed.insert(GV).second)
         CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
     }
@@ -1534,14 +1536,20 @@ void DwarfDebug::endModule() {
     }
 
     // Emit function-local entities.
-    for (const auto *D : CU->getDeferredLocalDecls()) {
-      if (auto *IE = dyn_cast<DIImportedEntity>(D))
-        CU->getOrCreateImportedEntityDIE(IE);
-      else if (auto *Ty = dyn_cast<DIType>(D))
-        CU->getOrCreateTypeDIE(Ty);
-      else
-        llvm_unreachable("Unexpected local retained node!");
-    }
+    const auto Unexpected = [](const Metadata *N) {
+      llvm_unreachable("Unexpected local retained node!");
+    };
+    for (const auto *D : CU->getDeferredLocalDecls())
+      DISubprogram::visitRetainedNode<void>(
+          D, Unexpected, Unexpected,
+          [CU](const auto *IE) { CU->getOrCreateImportedEntityDIE(IE); },
+          [CU](const auto *Ty) { CU->getOrCreateTypeDIE(Ty); },
+          [&](const auto *GVE) {
+            DIGlobalVariable *GV = GVE->getVariable();
+            if (Processed.insert(GV).second)
+              CU->getOrCreateGlobalVariableDIE(GV, sortGlobalExprs(GVMap[GV]));
+          },
+          Unexpected);
 
     // Emit base types.
     CU->createBaseTypeDIEs();
@@ -2088,16 +2096,17 @@ void DwarfDebug::collectEntityInfo(DwarfCompileUnit &TheCU,
   }
 
   // Collect info for retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
-    const auto *LS = getRetainedNodeScope(DN);
-    if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+  for (const MDNode *N : SP->getRetainedNodes()) {
+    const auto *LS = getRetainedNodeScope(N);
+    if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+      auto *DN = cast<DINode>(N);
       if (!Processed.insert(InlinedEntity(DN, nullptr)).second)
         continue;
       LexicalScope *LexS = LScopes.findLexicalScope(LS);
       if (LexS)
         createConcreteEntity(TheCU, *LexS, DN, nullptr);
     } else {
-      LocalDeclsPerLS[LS].insert(DN);
+      LocalDeclsPerLS[LS].insert(N);
     }
   }
 }
@@ -2897,12 +2906,13 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
 #endif
   for (LexicalScope *AScope : LScopes.getAbstractScopesList()) {
     const auto *SP = cast<DISubprogram>(AScope->getScopeNode());
-    for (const DINode *DN : SP->getRetainedNodes()) {
-      const auto *LS = getRetainedNodeScope(DN);
+    for (const MDNode *N : SP->getRetainedNodes()) {
+      const auto *LS = getRetainedNodeScope(N);
       // Ensure LexicalScope is created for the scope of this node.
       auto *LexS = LScopes.getOrCreateAbstractScope(LS);
       assert(LexS && "Expected the LexicalScope to be created.");
-      if (isa<DILocalVariable>(DN) || isa<DILabel>(DN)) {
+      if (isa<DILocalVariable>(N) || isa<DILabel>(N)) {
+        auto *DN = cast<DINode>(N);
         // Collect info for variables/labels that were optimized out.
         if (!Processed.insert(InlinedEntity(DN, nullptr)).second ||
             TheCU.getExistingAbstractEntity(DN))
@@ -2910,7 +2920,7 @@ void DwarfDebug::endFunctionImpl(const MachineFunction *MF) {
         TheCU.createAbstractEntity(DN, LexS);
       } else {
         // Remember the node if this is a local declarations.
-        LocalDeclsPerLS[LS].insert(DN);
+        LocalDeclsPerLS[LS].insert(N);
       }
       assert(
           LScopes.getAbstractScopesList().size() == NumAbstractSubprograms &&

diff  --git a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
index eb9fa3c89a44e..5726955a029ef 100644
--- a/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
+++ b/llvm/lib/DebugInfo/LogicalView/Readers/LVIRReader.cpp
@@ -1447,13 +1447,15 @@ LVScope *LVIRReader::getOrCreateSubprogram(LVScope *Function,
   }
 
   // Check for additional retained nodes.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *IE = dyn_cast<DIImportedEntity>(DN))
       constructImportedEntity(Function, IE);
     else if (const auto *TTP = dyn_cast<DITemplateTypeParameter>(DN))
       constructTemplateTypeParameter(Function, TTP);
     else if (const auto *TVP = dyn_cast<DITemplateValueParameter>(DN))
       constructTemplateValueParameter(Function, TVP);
+    else if (const auto *GVE = dyn_cast<DIGlobalVariableExpression>(DN))
+      getOrCreateVariable(GVE);
   }
 
   applySubprogramAttributes(Function, SP);

diff  --git a/llvm/lib/IR/DIBuilder.cpp b/llvm/lib/IR/DIBuilder.cpp
index 5e9c3f627e10c..60854c11ff014 100644
--- a/llvm/lib/IR/DIBuilder.cpp
+++ b/llvm/lib/IR/DIBuilder.cpp
@@ -33,7 +33,7 @@ DIBuilder::DIBuilder(Module &m, bool AllowUnresolvedNodes, DICompileUnit *CU)
     if (const auto &RTs = CUNode->getRetainedTypes())
       AllRetainTypes.assign(RTs.begin(), RTs.end());
     if (const auto &GVs = CUNode->getGlobalVariables())
-      AllGVs.assign(GVs.begin(), GVs.end());
+      Globals.assign(GVs.begin(), GVs.end());
     if (const auto &IMs = CUNode->getImportedEntities())
       ImportedModules.assign(IMs.begin(), IMs.end());
     if (const auto &MNs = CUNode->getMacros())
@@ -56,20 +56,19 @@ void DIBuilder::finalizeSubprogram(DISubprogram *SP) {
   if (PN == SubprogramTrackedNodes.end())
     return;
 
-  SmallVector<Metadata *, 16> RetainedNodes;
-  for (MDNode *N : PN->second) {
+  SetVector<Metadata *> RetainedNodes;
+  for (MDNode *N : llvm::concat<MDNode *>(SP->getRetainedNodes(), PN->second)) {
     // If the tracked node N was temporary, and the DIBuilder user replaced it
     // with a node that does not belong to SP or is non-local, do not add N to
     // SP's retainedNodes list.
     DILocalScope *Scope = dyn_cast_or_null<DILocalScope>(
         DISubprogram::getRawRetainedNodeScope(N));
-    if (!Scope || Scope->getSubprogram() != SP)
-      continue;
-
-    RetainedNodes.push_back(N);
+    if (Scope && Scope->getSubprogram() == SP)
+      RetainedNodes.insert(N);
   }
 
-  SP->replaceRetainedNodes(MDTuple::get(VMContext, RetainedNodes));
+  SP->replaceRetainedNodes(
+      MDTuple::get(VMContext, RetainedNodes.getArrayRef()));
 }
 
 void DIBuilder::finalize() {
@@ -103,8 +102,8 @@ void DIBuilder::finalize() {
     if (auto *SP = dyn_cast<DISubprogram>(N))
       finalizeSubprogram(SP);
 
-  if (!AllGVs.empty())
-    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, AllGVs));
+  if (!Globals.empty())
+    CUNode->replaceGlobalVariables(MDTuple::get(VMContext, Globals));
 
   if (!ImportedModules.empty())
     CUNode->replaceImportedEntities(MDTuple::get(
@@ -961,7 +960,10 @@ DIGlobalVariableExpression *DIBuilder::createGlobalVariableExpression(
   if (!Expr)
     Expr = createExpression();
   auto *N = DIGlobalVariableExpression::get(VMContext, GV, Expr);
-  AllGVs.push_back(N);
+  if (isa_and_nonnull<DILocalScope>(Context))
+    getSubprogramNodesTrackingVector(Context).emplace_back(N);
+  else
+    Globals.push_back(N);
   return N;
 }
 

diff  --git a/llvm/lib/IR/DebugInfo.cpp b/llvm/lib/IR/DebugInfo.cpp
index 0cd8351c52469..3adcab2c62888 100644
--- a/llvm/lib/IR/DebugInfo.cpp
+++ b/llvm/lib/IR/DebugInfo.cpp
@@ -197,13 +197,8 @@ void DebugInfoFinder::processModule(const Module &M) {
 void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
   if (!addCompileUnit(CU))
     return;
-  for (auto *DIG : CU->getGlobalVariables()) {
-    if (!addGlobalVariable(DIG))
-      continue;
-    auto *GV = DIG->getVariable();
-    processScope(GV->getScope());
-    processType(GV->getType());
-  }
+  for (auto *GVE : CU->getGlobalVariables())
+    processGlobalVariableExpression(GVE);
   for (auto *ET : CU->getEnumTypes())
     processType(ET);
   for (auto *RT : CU->getRetainedTypes())
@@ -217,6 +212,15 @@ void DebugInfoFinder::processCompileUnit(DICompileUnit *CU) {
     processMacroNode(Macro, nullptr);
 }
 
+void DebugInfoFinder::processGlobalVariableExpression(
+    DIGlobalVariableExpression *GVE) {
+  if (!addGlobalVariable(GVE))
+    return;
+  auto *GV = GVE->getVariable();
+  processScope(GV->getScope());
+  processType(GV->getType());
+}
+
 void DebugInfoFinder::processInstruction(const Module &M,
                                          const Instruction &I) {
   if (auto *DVI = dyn_cast<DbgVariableIntrinsic>(&I))
@@ -407,7 +411,8 @@ void DebugInfoFinder::processSubprogram(DISubprogram *SP) {
   SP->forEachRetainedNode(
       [this](DILocalVariable *LV) { processVariable(LV); }, [](DILabel *L) {},
       [this](DIImportedEntity *IE) { processImportedEntity(IE); },
-      [this](DIType *T) { processType(T); });
+      [this](DIType *T) { processType(T); },
+      [this](auto *GVE) { return processGlobalVariableExpression(GVE); });
 }
 
 void DebugInfoFinder::processVariable(const DILocalVariable *DV) {

diff  --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 79b7ce040b552..88f7f2f6240e0 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1451,11 +1451,10 @@ bool DISubprogram::describes(const Function *F) const {
 
 template <typename ScopeT, typename NodeT>
 static ScopeT getRawRetainedNodeScopeInternal(NodeT *N) {
-  auto getScope = [](auto *N) { return N->getScope(); };
-
+  auto getScopeLambda = [](auto *N) { return getScope(N); };
   return DISubprogram::visitRetainedNode<ScopeT>(
-      N, getScope, getScope, getScope, getScope,
-      [](auto *N) { return nullptr; });
+      N, getScopeLambda, getScopeLambda, getScopeLambda, getScopeLambda,
+      getScopeLambda, [](auto *N) { return nullptr; });
 }
 
 const DIScope *DISubprogram::getRawRetainedNodeScope(const MDNode *N) {
@@ -1475,41 +1474,22 @@ DILocalScope *DISubprogram::getRetainedNodeScope(MDNode *N) {
 }
 
 void DISubprogram::cleanupRetainedNodes() {
-  // Checks if a metadata node from retainedTypes is a type not belonging to
+  // Checks if a metadata node from retainedTypes is a type belonging to
   // this subprogram.
-  auto IsAlienType = [this](DINode *N) {
+  auto IsTypeInSP = [this](Metadata *N) {
     auto *T = dyn_cast_or_null<DIType>(N);
     if (!T)
-      return false;
+      return true;
 
     DISubprogram *TypeSP = nullptr;
     // The type might have been global in the previously loaded IR modules.
     if (auto *LS = dyn_cast_or_null<DILocalScope>(T->getScope()))
       TypeSP = LS->getSubprogram();
 
-    return this != TypeSP;
+    return this == TypeSP;
   };
 
-  // As this is expected to be called during module loading, before
-  // stripping old or incorrect debug info, perform minimal sanity check.
-  if (!isa_and_present<MDTuple>(getRawRetainedNodes()))
-    return;
-
-  MDTuple *RetainedNodes = cast<MDTuple>(getRawRetainedNodes());
-  SmallVector<Metadata *> MDs;
-  MDs.reserve(RetainedNodes->getNumOperands());
-  for (const MDOperand &Node : RetainedNodes->operands()) {
-    // Ignore malformed retainedNodes.
-    if (Node && !isa<DINode>(Node))
-      return;
-
-    auto *N = cast_or_null<DINode>(Node);
-    if (!IsAlienType(N))
-      MDs.push_back(N);
-  }
-
-  if (MDs.size() != RetainedNodes->getNumOperands())
-    replaceRetainedNodes(MDNode::get(getContext(), MDs));
+  cleanupRetainedNodesIf(IsTypeInSP);
 }
 
 DILexicalBlockBase::DILexicalBlockBase(LLVMContext &C, unsigned ID,

diff  --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 0744a91710b2e..0661a8316d740 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1464,8 +1464,12 @@ void Verifier::visitDICompileUnit(const DICompileUnit &N) {
   if (auto *Array = N.getRawGlobalVariables()) {
     CheckDI(isa<MDTuple>(Array), "invalid global variable list", &N, Array);
     for (Metadata *Op : N.getGlobalVariables()->operands()) {
-      CheckDI(Op && (isa<DIGlobalVariableExpression>(Op)),
-              "invalid global variable ref", &N, Op);
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(Op);
+      CheckDI(GVE, "invalid global variable ref", &N, Op);
+      CheckDI(!isa_and_nonnull<DILocalScope>(GVE->getVariable()->getScope()),
+              "function-local variables are not allowed in a DICompileUnit's "
+              "global variables list",
+              &N, Op);
     }
   }
   if (auto *Array = N.getRawImportedEntities()) {
@@ -1516,13 +1520,13 @@ void Verifier::visitDISubprogram(const DISubprogram &N) {
       auto True = [](const Metadata *) { return true; };
       auto False = [](const Metadata *) { return false; };
       bool IsTypeCorrect = DISubprogram::visitRetainedNode<bool>(
-          Op, True, True, True, True, False);
+          Op, True, True, True, True, True, False);
       CheckDI(IsTypeCorrect,
               "invalid retained nodes, expected DILocalVariable, DILabel, "
-              "DIImportedEntity or DIType",
+              "DIImportedEntity, DIType or DIGlobalVariableExpression",
               &N, Node, Op);
 
-      auto *RetainedNode = cast<DINode>(Op);
+      auto *RetainedNode = cast<MDNode>(Op);
       auto *RetainedNodeScope = dyn_cast_or_null<DILocalScope>(
           DISubprogram::getRawRetainedNodeScope(RetainedNode));
       CheckDI(RetainedNodeScope,

diff  --git a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
index d65726bc59d68..a38d94b0da7ee 100644
--- a/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
+++ b/llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
@@ -209,7 +209,7 @@ bool BPFAbstractMemberAccess::run(Function &F) {
   if (SP && SP->isDefinition()) {
     for (DIType *Ty: SP->getType()->getTypeArray())
       CheckAnonRecordType(nullptr, Ty);
-    for (const DINode *DN : SP->getRetainedNodes()) {
+    for (const MDNode *DN : SP->getRetainedNodes()) {
       if (const auto *DV = dyn_cast<DILocalVariable>(DN))
         CheckAnonRecordType(nullptr, DV->getType());
     }

diff  --git a/llvm/lib/Target/BPF/BTFDebug.cpp b/llvm/lib/Target/BPF/BTFDebug.cpp
index 53cd46f614b78..6de1d1eee7cc4 100644
--- a/llvm/lib/Target/BPF/BTFDebug.cpp
+++ b/llvm/lib/Target/BPF/BTFDebug.cpp
@@ -913,7 +913,7 @@ uint32_t BTFDebug::processDISubprogram(
   uint32_t FuncId = addType(std::move(FuncTypeEntry));
 
   // Process argument annotations.
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       uint32_t Arg = DV->getArg();
       if (Arg) {
@@ -1584,7 +1584,7 @@ void BTFDebug::beginFunctionImpl(const MachineFunction *MF) {
   // Use RetainedNodes so we can collect all argument names
   // even if the argument is not used.
   SmallDenseMap<uint32_t, StringRef> FuncArgNames;
-  for (const DINode *DN : SP->getRetainedNodes()) {
+  for (const MDNode *DN : SP->getRetainedNodes()) {
     if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
       // Collect function arguments for subprogram func type.
       uint32_t Arg = DV->getArg();

diff  --git a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
index 3e3960ea24e88..4a9727f100e28 100644
--- a/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroFrame.cpp
@@ -743,11 +743,7 @@ static void buildFrameDebugInfo(Function &F, coro::Shape &Shape,
   // If we don't add __coro_frame to the RetainedNodes, user may get
   // `no symbol __coro_frame in context` rather than `__coro_frame`
   // is optimized out, which is more precise.
-  auto RetainedNodes = DIS->getRetainedNodes();
-  SmallVector<Metadata *, 32> RetainedNodesVec(RetainedNodes.begin(),
-                                               RetainedNodes.end());
-  RetainedNodesVec.push_back(FrameDIVar);
-  DIS->replaceOperandWith(7, (MDTuple::get(F.getContext(), RetainedNodesVec)));
+  DIS->retainNodes(&FrameDIVar, &FrameDIVar + 1);
 
   // Construct the location for the frame debug variable. The column number
   // is fake but it should be fine.

diff  --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index 73a8182949910..5e58b5c95f776 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -1904,6 +1904,7 @@ InstrLowerer::getOrCreateRegionBitmaps(InstrProfMCDCBitmapInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       BitmapPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 
@@ -1980,6 +1981,7 @@ InstrLowerer::getOrCreateRegionCounters(InstrProfCntrInstBase *Inc) {
           /*Decl=*/nullptr, /*TemplateParams=*/nullptr, /*AlignInBits=*/0,
           Annotations);
       CounterPtr->addDebugInfo(DICounter);
+      DB.finalizeSubprogram(SP);
       DB.finalize();
     }
 

diff  --git a/llvm/lib/Transforms/Utils/CloneFunction.cpp b/llvm/lib/Transforms/Utils/CloneFunction.cpp
index 75342a5d1894f..35de43f14bb23 100644
--- a/llvm/lib/Transforms/Utils/CloneFunction.cpp
+++ b/llvm/lib/Transforms/Utils/CloneFunction.cpp
@@ -105,6 +105,12 @@ createIdentityMDPredicate(const Function &F, CloneFunctionChangeType Changes) {
       if (auto *S = dyn_cast_or_null<DILocalScope>(DV->getScope()))
         return ShouldKeep(S->getSubprogram());
 
+    // DIGlobalVariableExpression representing static local variable may be
+    // encountered in DISubprogram's retainedNodes list. Do not remap it, and
+    // remove it from retainedNodes after mapping.
+    if (isa<DIGlobalVariableExpression>(MD))
+      return true;
+
     // Clone types that are local to subprograms being cloned.
     // Avoid cloning other types.
     auto *Type = dyn_cast<DIType>(MD);
@@ -351,6 +357,15 @@ void llvm::CloneFunctionInto(Function *NewFunc, const Function *OldFunc,
                         NameSuffix, CodeInfo, TypeMapper, Materializer,
                         &IdentityMD);
 
+  // DIGlobalVariableExpressions representing static locals stay in the scope of
+  // OldSP after function cloning. Remove them from retainedNodes of NewSP.
+  if (DISubprogram *NewSP = NewFunc->getSubprogram())
+    NewSP->cleanupRetainedNodesIf([NewSP](Metadata *N) {
+      auto *GVE = dyn_cast_or_null<DIGlobalVariableExpression>(N);
+      return !GVE ||
+             DISubprogram::getRetainedNodeScope(GVE)->getSubprogram() == NewSP;
+    });
+
   // Only update !llvm.dbg.cu for DifferentModule (not CloneModule). In the
   // same module, the compile unit will already be listed (or not). When
   // cloning a module, CloneModule() will handle creating the named metadata.

diff  --git a/llvm/lib/Transforms/Utils/Debugify.cpp b/llvm/lib/Transforms/Utils/Debugify.cpp
index 8656a7c973451..a428ff4f451a4 100644
--- a/llvm/lib/Transforms/Utils/Debugify.cpp
+++ b/llvm/lib/Transforms/Utils/Debugify.cpp
@@ -440,7 +440,7 @@ bool llvm::collectDebugInfoMetadata(Module &M,
     DebugInfoBeforePass.DIFunctions.insert({&F, SP});
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoBeforePass.DIVariables[DV] = 0;
         }
@@ -658,7 +658,7 @@ bool llvm::checkDebugInfoMetadata(Module &M,
 
     if (SP) {
       LLVM_DEBUG(dbgs() << "  Collecting subprogram: " << *SP << '\n');
-      for (const DINode *DN : SP->getRetainedNodes()) {
+      for (const MDNode *DN : SP->getRetainedNodes()) {
         if (const auto *DV = dyn_cast<DILocalVariable>(DN)) {
           DebugInfoAfterPass.DIVariables[DV] = 0;
         }

diff  --git a/llvm/test/Bitcode/DIModule-fortran-external-module.ll b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
index 5ca6031d6a1e3..c41e38d9bc136 100644
--- a/llvm/test/Bitcode/DIModule-fortran-external-module.ll
+++ b/llvm/test/Bitcode/DIModule-fortran-external-module.ll
@@ -41,12 +41,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, imports: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !7, imports: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}

diff  --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
new file mode 100644
index 0000000000000..54757b15642a0
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-cu-static-locals.ll
@@ -0,0 +1,108 @@
+; Test moving of local static variables from DICompileUnit 'globals' to DISubprogram's 'retainedNodes'
+;
+; RUN: llvm-dis -o - %s.bc | FileCheck %s
+
+; Function Attrs: norecurse nounwind
+define dso_local i32 @foo() local_unnamed_addr #0 !dbg !2 {
+  ret i32 0
+}
+
+define dso_local i32 @abc() local_unnamed_addr #0 !dbg !31 {
+  ret i32 0
+}
+
+define dso_local i32 @xyz() local_unnamed_addr #0 !dbg !32 {
+  ret i32 0
+}
+
+attributes #0 = { noinline optnone }
+
+!llvm.dbg.cu = !{!7, !30}
+!llvm.module.flags = !{!16, !17, !18}
+!llvm.ident = !{!19}
+
+; CHECK: ![[CU:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU_GLOBALS:[0-9]+]]
+; CHECK: ![[CU_GLOBALS]] = !{![[AEXPR:[0-9]+]]}
+; CHECK: ![[AEXPR]] = !DIGlobalVariableExpression(var: ![[A:[0-9]+]]
+; CHECK: ![[A]] = distinct !DIGlobalVariable(name: "a", scope: ![[CU]]
+
+; CHECK: ![[CU30:[0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C99, file: ![[F_TEST2:[0-9]+]], producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !{{.*}}, globals: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CU30_GLOBALS]] = !{![[CAP_A_EXPR:[0-9]+]], ![[CAP_B_EXPR:[0-9]+]]}
+; CHECK: ![[CAP_A_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_A:[0-9]+]]
+; CHECK: ![[CAP_A]] = distinct !DIGlobalVariable(name: "A", scope: ![[CU30_GLOBALS:[0-9]+]]
+; CHECK: ![[CAP_B_EXPR]] = !DIGlobalVariableExpression(var: ![[CAP_B:[0-9]+]]
+; CHECK: ![[CAP_B]] = distinct !DIGlobalVariable(name: "B", scope: ![[CU30_GLOBALS:[0-9]+]]
+
+; CHECK: ![[FOO:[0-9]+]] = distinct !DISubprogram(name: "foo", scope: ![[F_TEST]], file: ![[F_TEST]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU]], retainedNodes: ![[FOO_RETAINED:[0-9]+]])
+; CHECK: ![[FOO_RETAINED]] = !{![[B_EXPR:[0-9]+]]}
+; CHECK: ![[B_EXPR]] = !DIGlobalVariableExpression(var: ![[B:[0-9]+]]
+; CHECK: ![[B]] = distinct !DIGlobalVariable(name: "b", scope: ![[FOO:[0-9]+]]
+
+; CHECK: ![[ABC:[0-9]+]] = distinct !DISubprogram(name: "abc", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[ABC_RETAINED:[0-9]+]])
+; CHECK: ![[ABC_RETAINED]] = !{![[ABC_A_EXPR:[0-9]+]], ![[ABC_B_EXPR:[0-9]+]]}
+; CHECK: ![[ABC_A_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_A:[0-9]+]]
+; CHECK: ![[ABC_A]] = distinct !DIGlobalVariable(name: "a", scope: ![[ABC]]
+; CHECK: ![[ABC_B_EXPR]] = !DIGlobalVariableExpression(var: ![[ABC_B:[0-9]+]]
+; CHECK: ![[ABC_B]] = distinct !DIGlobalVariable(name: "b", scope: ![[ABC]]
+
+; CHECK: ![[XYZ:[0-9]+]] = distinct !DISubprogram(name: "xyz", scope: ![[CU30]], file: ![[F_TEST2]], line: 2, type: !{{.*}}, scopeLine: 2, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: ![[CU30]], retainedNodes: ![[XYZ_RETAINED:[0-9]+]])
+; CHECK: ![[XYZ_RETAINED]] = !{![[XYZ_I:[0-9]+]], ![[XYZ_C_EXPR:[0-9]+]]}
+; CHECK: ![[XYZ_I]] = !DILocalVariable(name: "i", arg: 1, scope: ![[XYZ]], file: ![[F_TEST2]]
+; CHECK: ![[XYZ_C_EXPR]] = !DIGlobalVariableExpression(var: ![[XYZ_C:[0-9]+]]
+; CHECK: ![[XYZ_C]] = distinct !DIGlobalVariable(name: "c", scope: ![[XYZ]]
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
+!4 = !DISubroutineType(types: !5)
+!5 = !{!6}
+!6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
+!8 = !{}
+!9 = !{!0, !10}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
+!13 = !DIBasicType(name: "char", size: 8, encoding: DW_ATE_signed_char)
+!14 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !15)
+!15 = !DIBasicType(name: "short", size: 16, encoding: DW_ATE_signed)
+!16 = !{i32 2, !"Dwarf Version", i32 4}
+!17 = !{i32 2, !"Debug Info Version", i32 3}
+!18 = !{i32 1, !"wchar_size", i32 4}
+!19 = !{!"clang version 8.0.20181009 "}
+!20 = !DILocation(line: 4, column: 10, scope: !2)
+!21 = !{!22, !22, i64 0}
+!22 = !{!"omnipotent char", !23, i64 0}
+!23 = !{!"Simple C/C++ TBAA"}
+!24 = !DILocation(line: 4, column: 14, scope: !2)
+!25 = !{!26, !26, i64 0}
+!26 = !{!"short", !22, i64 0}
+!27 = !DILocation(line: 4, column: 12, scope: !2)
+
+; Check CU with several global and several local static variables.
+!30 = distinct !DICompileUnit(language: DW_LANG_C99, file: !50, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !45, nameTableKind: None)
+
+!31 = distinct !DISubprogram(name: "abc", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !8)
+!32 = distinct !DISubprogram(name: "xyz", scope: !30, file: !50, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !30, retainedNodes: !46)
+
+; Globals
+!33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+!34 = distinct !DIGlobalVariable(name: "A", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+!35 = !DIGlobalVariableExpression(var: !36, expr: !DIExpression())
+!36 = distinct !DIGlobalVariable(name: "B", scope: !30, file: !50, line: 1, type: !12, isLocal: true, isDefinition: true)
+
+; Static locals of abc
+!37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
+!38 = distinct !DIGlobalVariable(name: "a", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!39 = !DIGlobalVariableExpression(var: !40, expr: !DIExpression())
+!40 = distinct !DIGlobalVariable(name: "b", scope: !31, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+
+; Locals of xyz
+!41 = !DIGlobalVariableExpression(var: !42, expr: !DIExpression())
+!42 = distinct !DIGlobalVariable(name: "c", scope: !32, file: !50, line: 3, type: !14, isLocal: true, isDefinition: true)
+!43 = !DILocalVariable(name: "i", line: 2, arg: 1, scope: !32, file: !50, type: !14)
+
+!45 = !{!37, !33, !39, !35, !41}
+!46 = !{!43}
+!50 = !DIFile(filename: "test2.c", directory: "/home/yhs/work/tests/llvm/bug")

diff  --git a/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc
new file mode 100644
index 0000000000000..1c57ea7cea417
Binary files /dev/null and b/llvm/test/Bitcode/upgrade-cu-static-locals.ll.bc 
diff er

diff  --git a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
index 86658680fdf5a..b5398b1d49520 100644
--- a/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
+++ b/llvm/test/CodeGen/AArch64/arm64-2011-03-17-AsmPrinterCrash.ll
@@ -28,12 +28,12 @@ attributes #1 = { nounwind readnone }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!9, !10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !2)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug)
 !1 = !DIFile(filename: "print.i", directory: "/Volumes/Ebi/echeng/radars/r9146594")
 !2 = !{!3}
 !3 = !DIGlobalVariableExpression(var: !4, expr: !DIExpression())
 !4 = !DIGlobalVariable(name: "vsplive", scope: !5, file: !1, line: 617, type: !8, isLocal: true, isDefinition: true)
-!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0)
+!5 = distinct !DISubprogram(name: "drt_vsprintf", scope: !1, file: !1, line: 616, type: !6, isLocal: false, isDefinition: true, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !0, retainedNodes: !2)
 !6 = !DISubroutineType(types: !7)
 !7 = !{!8}
 !8 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
index deef48adee9ae..25e62b402dac4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
index 8f29a83fedb13..9c7e492e170b4 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-inited.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
index e16b4672f53d3..603673eee966e 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly-sec.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
index 1ddd49904135b..e5df7c1dc8c11 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-readonly.ll
@@ -105,14 +105,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !15, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !31)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
@@ -134,3 +134,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !28 = !{!"short", !24, i64 0}
 !29 = !DILocation(line: 4, column: 12, scope: !2)
 !30 = !DILocation(line: 4, column: 3, scope: !2)
+!31 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
index 0ff8f2eb7db84..0168d30fffc2a 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var-sec.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/BTF/static-var.ll b/llvm/test/CodeGen/BPF/BTF/static-var.ll
index f7710e3f3ce45..43781e4d91183 100644
--- a/llvm/test/CodeGen/BPF/BTF/static-var.ll
+++ b/llvm/test/CodeGen/BPF/BTF/static-var.ll
@@ -99,14 +99,14 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "b", scope: !2, file: !3, line: 3, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "foo", scope: !3, file: !3, line: 2, type: !4, isLocal: false, isDefinition: true, scopeLine: 2, isOptimized: true, unit: !7, retainedNodes: !29)
 !3 = !DIFile(filename: "test.c", directory: "/home/yhs/work/tests/llvm/bug")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
 !7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 8.0.20181009 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9, nameTableKind: None)
 !8 = !{}
-!9 = !{!0, !10}
+!9 = !{!10}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "a", scope: !7, file: !3, line: 1, type: !12, isLocal: true, isDefinition: true)
 !12 = !DIDerivedType(tag: DW_TAG_volatile_type, baseType: !13)
@@ -126,3 +126,4 @@ define dso_local i32 @foo() local_unnamed_addr !dbg !2 {
 !26 = !{!"short", !22, i64 0}
 !27 = !DILocation(line: 4, column: 12, scope: !2)
 !28 = !DILocation(line: 4, column: 3, scope: !2)
+!29 = !{!0}

diff  --git a/llvm/test/CodeGen/BPF/dwarfdump.ll b/llvm/test/CodeGen/BPF/dwarfdump.ll
index d50c025c4e6f8..68b0779450eac 100644
--- a/llvm/test/CodeGen/BPF/dwarfdump.ll
+++ b/llvm/test/CodeGen/BPF/dwarfdump.ll
@@ -34,10 +34,9 @@ declare void @llvm.dbg.value(metadata, i64, metadata, metadata)
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6, !6, !6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8, globals: !9)
+!7 = distinct !DICompileUnit(language: DW_LANG_C99, file: !3, producer: "clang version 4.0.0 (trunk 287518) (llvm/trunk 287520)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !8)
 !8 = !{}
-!9 = !{!0}
-!10 = !{!11, !12}
+!10 = !{!0, !11, !12}
 !11 = !DILocalVariable(name: "myvar_a", arg: 1, scope: !2, file: !3, line: 1, type: !6)
 !12 = !DILocalVariable(name: "myvar_b", arg: 2, scope: !2, file: !3, line: 1, type: !6)
 !13 = !{i32 2, !"Dwarf Version", i32 4}

diff  --git a/llvm/test/CodeGen/PowerPC/pr24546.ll b/llvm/test/CodeGen/PowerPC/pr24546.ll
index a84b2e894f111..e4ff2ff9fe8a8 100644
--- a/llvm/test/CodeGen/PowerPC/pr24546.ll
+++ b/llvm/test/CodeGen/PowerPC/pr24546.ll
@@ -56,7 +56,7 @@ attributes #3 = { nounwind }
 !llvm.module.flags = !{!29, !30}
 !llvm.ident = !{!31}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3, globals: !23)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.8.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !3)
 !1 = !DIFile(filename: "testcase.i", directory: "/tmp/glibc.build")
 !2 = !{}
 !3 = !{!4}
@@ -76,9 +76,8 @@ attributes #3 = { nounwind }
 !18 = distinct !DISubprogram(name: "php_intpow10", scope: !1, file: !1, line: 1, type: !19, isLocal: true, isDefinition: true, scopeLine: 2, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !21)
 !19 = !DISubroutineType(types: !20)
 !20 = !{!4, !9}
-!21 = !{!22}
+!21 = !{!22, !24}
 !22 = !DILocalVariable(name: "power", arg: 1, scope: !18, file: !1, line: 1, type: !9)
-!23 = !{!24}
 !24 = !DIGlobalVariableExpression(var: !DIGlobalVariable(name: "powers", scope: !18, file: !1, line: 3, type: !25, isLocal: true, isDefinition: true), expr: !DIExpression())
 !25 = !DICompositeType(tag: DW_TAG_array_type, baseType: !26, size: 1472, align: 64, elements: !27)
 !26 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !4)

diff  --git a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
index 47b32967e68f7..f229ab1a6e34f 100644
--- a/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
+++ b/llvm/test/CodeGen/X86/dbg-distringtype-uint.ll
@@ -84,10 +84,9 @@ attributes #3 = { nofree nosync nounwind readnone speculatable willreturn }
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7}
 !7 = !DIStringType(name: "CHARACTER_0", size: 64)
-!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, globals: !10, splitDebugInlining: false, nameTableKind: None)
+!8 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2745", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, splitDebugInlining: false, nameTableKind: None)
 !9 = !{}
-!10 = !{!0}
-!11 = !{!12, !14}
+!11 = !{!0, !12, !14}
 !12 = !DILocalVariable(name: "esym", arg: 1, scope: !2, file: !3, line: 15, type: !13, flags: DIFlagArtificial)
 !13 = !DIStringType(name: "CHARACTER_1", size: 64)
 !14 = !DILocalVariable(name: "i", arg: 2, scope: !2, file: !3, line: 15, type: !15)

diff  --git a/llvm/test/DebugInfo/COFF/fortran-basic.ll b/llvm/test/DebugInfo/COFF/fortran-basic.ll
index 8fa7a12e66919..854ba151039de 100644
--- a/llvm/test/DebugInfo/COFF/fortran-basic.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-basic.ll
@@ -145,7 +145,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !5 = !DISubroutineType(types: !6)
 !6 = !{null}
 !7 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !4, producer: "Fortran", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
-!8 = !{!0, !9, !12, !15, !21}
+!8 = !{!0, !9, !12}
 !9 = !DIGlobalVariableExpression(var: !10, expr: !DIExpression(DW_OP_plus_uconst, 8))
 !10 = distinct !DIGlobalVariable(name: "L", linkageName: "COM", scope: !2, file: !4, line: 6, type: !11, isLocal: false, isDefinition: true)
 !11 = !DIBasicType(name: "LOGICAL*4", size: 32, encoding: DW_ATE_boolean)
@@ -163,7 +163,7 @@ attributes #3 = { nocallback nofree nosync nounwind readnone speculatable willre
 !23 = !DICompositeType(tag: DW_TAG_array_type, baseType: !18, elements: !24)
 !24 = !{!25}
 !25 = !DISubrange(count: 10, lowerBound: 1)
-!26 = !{!27, !29, !31}
+!26 = !{!15, !21, !27, !29, !31}
 !27 = !DILocalVariable(name: "CMP32", scope: !3, file: !4, line: 10, type: !28)
 !28 = !DIBasicType(name: "COMPLEX*32", size: 256, encoding: DW_ATE_complex_float)
 !29 = !DILocalVariable(name: "CMP16", scope: !3, file: !4, line: 9, type: !30)

diff  --git a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
index 9d79d913ba57b..a9371cd7b0886 100644
--- a/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
+++ b/llvm/test/DebugInfo/COFF/fortran-contained-proc.ll
@@ -86,11 +86,11 @@ attributes #4 = { nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "A", linkageName: "IF_TEST$A", scope: !2, file: !3, line: 11, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !8)
+!2 = distinct !DISubprogram(name: "IF_TEST", linkageName: "MAIN__", scope: !3, file: !3, line: 9, type: !4, scopeLine: 9, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !6, retainedNodes: !7)
 !3 = !DIFile(filename: "tr1.f90", directory: "d:\\iusers\\cchen15\\examples\\tests\\jr14335")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !7, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 22.0-1087", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !8, splitDebugInlining: false, nameTableKind: None)
 !7 = !{!0}
 !8 = !{}
 !9 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !10, size: 64)

diff  --git a/llvm/test/DebugInfo/COFF/global_visibility.ll b/llvm/test/DebugInfo/COFF/global_visibility.ll
index 499690a854eeb..12ff172fb5a02 100644
--- a/llvm/test/DebugInfo/COFF/global_visibility.ll
+++ b/llvm/test/DebugInfo/COFF/global_visibility.ll
@@ -208,10 +208,10 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 11.0.0 (https://github.com/llvm/llvm-project.git 202f144bffd0be254a829924195e1b8ebabcbb79)", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !4, globals: !5, nameTableKind: None)
 !3 = !DIFile(filename: "a.cpp", directory: "F:\\llvm-project\\__test", checksumkind: CSK_MD5, checksum: "66a5399777dc9d37656fb00438bca542")
 !4 = !{}
-!5 = !{!0, !6, !12, !14, !17, !19, !29}
+!5 = !{!0, !19, !29}
 !6 = !DIGlobalVariableExpression(var: !7, expr: !DIExpression())
 !7 = distinct !DIGlobalVariable(name: "local_int", scope: !8, file: !3, line: 15, type: !11, isLocal: true, isDefinition: true)
-!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!8 = distinct !DISubprogram(name: "foo", linkageName: "?foo@@YAXXZ", scope: !3, file: !3, line: 14, type: !9, scopeLine: 14, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !67)
 !9 = !DISubroutineType(types: !10)
 !10 = !{null}
 !11 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -219,7 +219,7 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !13 = distinct !DIGlobalVariable(name: "nested_int", scope: !8, file: !3, line: 17, type: !11, isLocal: true, isDefinition: true)
 !14 = !DIGlobalVariableExpression(var: !15, expr: !DIExpression())
 !15 = distinct !DIGlobalVariable(name: "local_int", scope: !16, file: !3, line: 24, type: !11, isLocal: true, isDefinition: true)
-!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !4)
+!16 = distinct !DISubprogram(name: "bar", linkageName: "?bar@@YAXXZ", scope: !3, file: !3, line: 23, type: !9, scopeLine: 23, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !68)
 !17 = !DIGlobalVariableExpression(var: !18, expr: !DIExpression())
 !18 = distinct !DIGlobalVariable(name: "nested_int", scope: !16, file: !3, line: 26, type: !11, isLocal: true, isDefinition: true)
 !19 = !DIGlobalVariableExpression(var: !20, expr: !DIExpression())
@@ -270,3 +270,5 @@ attributes #2 = { nounwind readnone speculatable willreturn }
 !64 = !DILocation(line: 6, scope: !60)
 !65 = !DILocation(line: 7, scope: !60)
 !66 = !DILocation(line: 8, scope: !60)
+!67 = !{!6, !12}
+!68 = !{!14, !17}

diff  --git a/llvm/test/DebugInfo/COFF/type-quals.ll b/llvm/test/DebugInfo/COFF/type-quals.ll
index bca2590ea7969..fd71796ed567c 100644
--- a/llvm/test/DebugInfo/COFF/type-quals.ll
+++ b/llvm/test/DebugInfo/COFF/type-quals.ll
@@ -487,14 +487,14 @@ attributes #2 = { argmemonly nounwind }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "str", scope: !2, file: !3, line: 18, type: !12, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !10)
+!2 = distinct !DISubprogram(name: "g", linkageName: "\01?g@@YAXAEIAH at Z", scope: !3, file: !3, line: 16, type: !4, isLocal: false, isDefinition: true, scopeLine: 16, flags: DIFlagPrototyped, isOptimized: false, unit: !9, retainedNodes: !11)
 !3 = !DIFile(filename: "m.cpp", directory: "C:\5CUsers\5CHui\5Ctmp\5Chui", checksumkind: CSK_MD5, checksum: "a8da0f4dca948db1ef1129c8728a881c")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_restrict_type, baseType: !7)
 !7 = !DIDerivedType(tag: DW_TAG_reference_type, baseType: !8, size: 64)
 !8 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !11)
+!9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 7.0.0 ", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !10, globals: !10)
 !10 = !{}
 !11 = !{!0}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 32, elements: !15)

diff  --git a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
index c26cba6be3b30..dd107d1adc627 100644
--- a/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
+++ b/llvm/test/DebugInfo/Generic/2009-11-05-DeadGlobalVariable.ll
@@ -14,13 +14,13 @@ attributes #0 = { nounwind readnone ssp uwtable }
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!10}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 3.0 (trunk 139632)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, retainedTypes: !2)
 !1 = !DIFile(filename: "fb.c", directory: "/private/tmp")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = !DIGlobalVariable(name: "bar", scope: !6, file: !1, line: 2, type: !9, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0)
+!6 = distinct !DISubprogram(name: "foo", scope: !1, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, virtualIndex: 6, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, align: 32, encoding: DW_ATE_signed)

diff  --git a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
index e31fd295a4c94..4c4225264ab42 100644
--- a/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
+++ b/llvm/test/DebugInfo/Generic/global-var-in-abstract-lexical-block.ll
@@ -47,13 +47,13 @@ entry:
 !1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "_ZZ3fooPiE12static_local", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
 !2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 2, column: 1)
 !3 = !DIFile(filename: "test.cu", directory: "/")
-!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !12)
+!4 = distinct !DISubprogram(name: "foo", linkageName: "_Z3fooPi", scope: !3, file: !3, line: 1, type: !5, scopeLine: 1, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition, unit: !10, retainedNodes: !11)
 !5 = !DISubroutineType(types: !6)
 !6 = !{!7, !8}
 !7 = !DIBasicType(tag: DW_TAG_unspecified_type, name: "void")
 !8 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !9, size: 64, align: 64, dwarfAddressSpace: 12)
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !11)
+!10 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "lgenfe: EDG 6.8", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !12)
 !11 = !{!0}
 !12 = !{}
 !13 = !{i32 1, !"Debug Info Version", i32 3}

diff  --git a/llvm/test/DebugInfo/Generic/inlined-static-var.ll b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
index 60c986fc2b648..99bca52c975dc 100644
--- a/llvm/test/DebugInfo/Generic/inlined-static-var.ll
+++ b/llvm/test/DebugInfo/Generic/inlined-static-var.ll
@@ -59,17 +59,17 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 5, type: !6, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!2 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 5, type: !4, scopeLine: 5, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !9)
 !3 = !DIFile(filename: "example.cpp", directory: "")
 !4 = !DISubroutineType(types: !5)
 !5 = !{!6}
 !6 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
-!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, globals: !9, splitDebugInlining: false, nameTableKind: None)
+!7 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !8, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 !8 = !DIFile(filename: "example.cpp", directory: "")
-!9 = !{!0, !10}
+!9 = !{!0}
 !10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
 !11 = distinct !DIGlobalVariable(name: "A", scope: !12, file: !3, line: 2, type: !6, isLocal: false, isDefinition: true)
-!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !13)
+!12 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 2, type: !4, scopeLine: 2, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !33)
 !13 = !{}
 !14 = !{i32 7, !"Dwarf Version", i32 4}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
@@ -90,3 +90,4 @@ define dso_local i32 @_Z3foov() !dbg !26 {
 !30 = distinct !DILocation(line: 7, column: 32, scope: !26)
 !31 = !DILocation(line: 7, column: 30, scope: !26)
 !32 = !DILocation(line: 7, column: 13, scope: !26)
+!33 = !{!10}

diff  --git a/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
new file mode 100644
index 0000000000000..4535309a2f635
--- /dev/null
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-dicompileunit.ll
@@ -0,0 +1,65 @@
+; RUN: llvm-as -disable-output %s 2>&1 | FileCheck --match-full-lines %s
+
+!llvm.module.flags = !{!0}
+!llvm.dbg.cu = !{!2, !4, !5, !6, !7, !8, !9, !10, !11, !12, !13, !14, !17, !20, !22}
+
+!0 = !{i32 2, !"Debug Info Version", i32 3}
+!1 = !DIFile(filename: "-", directory: "")
+
+; CHECK: invalid file
+!2 = distinct !DICompileUnit(language: 0, file: !{})
+
+; CHECK: invalid filename
+!3 = !DIFile(filename: "", directory: "")
+!4 = distinct !DICompileUnit(language: 0, file: !3)
+
+; CHECK: invalid enum list
+!5 = distinct !DICompileUnit(language: 0, file: !1, enums: !1)
+
+; CHECK: invalid enum type
+!6 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!1})
+
+; CHECK: invalid retained type list
+!7 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !1)
+
+; CHECK: invalid retained type
+!8 = distinct !DICompileUnit(language: 0, file: !1, retainedTypes: !{!1})
+
+; CHECK: invalid global variable list
+!9 = distinct !DICompileUnit(language: 0, file: !1, globals: !1)
+
+; CHECK: invalid global variable ref
+!10 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!1})
+
+; CHECK: invalid imported entity list
+!11 = distinct !DICompileUnit(language: 0, file: !1, imports: !1)
+
+; CHECK: invalid imported entity ref
+!12 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!1})
+
+; CHECK: invalid macro list
+!13 = distinct !DICompileUnit(language: 0, file: !1, macros: !1)
+
+; CHECK: invalid macro ref
+!14 = distinct !DICompileUnit(language: 0, file: !1, macros: !{!1})
+
+define void @foo() !dbg !15 { ret void }
+!15 = distinct !DISubprogram(file: !1, type: !23)
+
+; CHECK: function-local imports are not allowed in a DICompileUnit's imported entities list
+!16 = distinct !DIImportedEntity(tag: DW_TAG_imported_declaration, scope: !15, entity: !15)
+!17 = distinct !DICompileUnit(language: 0, file: !1, imports: !{!16})
+
+; CHECK: function-local variables are not allowed in a DICompileUnit's global variables list
+!18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
+!19 = distinct !DIGlobalVariable(scope: !15)
+!20 = distinct !DICompileUnit(language: 0, file: !1, globals: !{!18})
+
+; CHECK: function-local enum in a DICompileUnit's enum list
+!21 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !15)
+!22 = distinct !DICompileUnit(language: 0, file: !1, enums: !{!21})
+
+; CHECK: warning: ignoring invalid debug info{{.*}}
+
+!23 = !DISubroutineType(types: !24)
+!24 = !{null}

diff  --git a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
index 0a15453951749..4e5f0af1f051d 100644
--- a/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
+++ b/llvm/test/DebugInfo/Generic/verifier-invalid-disubprogram.ll
@@ -40,7 +40,7 @@ define void @invalid_subprogram_declaration() !dbg !9 { ret void }
 define void @invalid_retained_nodes_list() !dbg !10 { ret void }
 !10 = distinct !DISubprogram(type: !21, retainedNodes: !0)
 
-; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity or DIType
+; CHECK: invalid retained nodes, expected DILocalVariable, DILabel, DIImportedEntity, DIType or DIGlobalVariableExpression
 define void @invalid_retained_nodes_expected() !dbg !11 { ret void }
 !11 = distinct !DISubprogram(type: !21, retainedNodes: !{!0})
 

diff  --git a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
index 1a467ec72a75d..3a8ee66e9b797 100644
--- a/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
+++ b/llvm/test/DebugInfo/Symbolize/XCOFF/xcoff-symbolize-data.ll
@@ -116,7 +116,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !1 = distinct !DIGlobalVariable(name: "bss_global", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 18.0.0", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, globals: !4, splitDebugInlining: false, nameTableKind: None)
 !3 = !DIFile(filename: "t.cpp", directory: "/")
-!4 = !{!0, !5, !8, !15, !18, !24, !26}
+!4 = !{!0, !5, !8, !15, !24, !26}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "data_global", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -132,7 +132,7 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !17 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !11, size: 32)
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "function_global", scope: !20, file: !3, line: 8, type: !7, isLocal: true, isDefinition: true)
-!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2)
+!20 = distinct !DISubprogram(name: "f", linkageName: "_Z1fv", scope: !3, file: !3, line: 7, type: !21, scopeLine: 7, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !47)
 !21 = !DISubroutineType(types: !22)
 !22 = !{!23}
 !23 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 32)
@@ -159,3 +159,4 @@ attributes #2 = { nocallback nofree nosync nounwind speculatable willreturn memo
 !44 = !{!"int", !45, i64 0}
 !45 = !{!"omnipotent char", !46, i64 0}
 !46 = !{!"Simple C++ TBAA"}
+!47 = !{!18}

diff  --git a/llvm/test/DebugInfo/X86/DW_AT_specification.ll b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
index 826ec11782032..a8d01bb715c6d 100644
--- a/llvm/test/DebugInfo/X86/DW_AT_specification.ll
+++ b/llvm/test/DebugInfo/X86/DW_AT_specification.ll
@@ -23,13 +23,13 @@ entry:
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = !DIGlobalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11)
+!2 = distinct !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: null, file: !3, line: 4, type: !4, isLocal: false, isDefinition: true, scopeLine: 4, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false, unit: !8, declaration: !11, retainedNodes: !10)
 !3 = !DIFile(filename: "nsNativeAppSupportBase.ii", directory: "/Users/espindola/mozilla-central/obj-x86_64-apple-darwin11.2.0/toolkit/library")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null, !6}
 !6 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64, align: 64, flags: DIFlagArtificial)
 !7 = !DICompositeType(tag: DW_TAG_structure_type, name: "foo", file: !3, line: 1, flags: DIFlagFwdDecl)
-!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, globals: !10, imports: !9)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !3, producer: "clang version 3.0 ()", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !9, retainedTypes: !9, imports: !9)
 !9 = !{}
 !10 = !{!0}
 !11 = !DISubprogram(name: "bar", linkageName: "_ZN3foo3barEv", scope: !12, file: !3, line: 2, type: !4, isLocal: false, isDefinition: false, scopeLine: 2, virtualIndex: 6, flags: DIFlagPrototyped, isOptimized: false)

diff  --git a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
index 1261bc3a1cf6f..56e6be7489e48 100644
--- a/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
+++ b/llvm/test/DebugInfo/X86/dbg-value-transfer-order.ll
@@ -97,13 +97,13 @@ attributes #1 = { nounwind readnone speculatable }
 !llvm.module.flags = !{!14, !15, !16, !17}
 !llvm.ident = !{!18}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 6.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "t.c", directory: "C:\5Csrc\5Cllvm-project\5Cbuild", checksumkind: CSK_MD5, checksum: "f80d0003faf76554dfdec6c95da285cc")
 !2 = !{}
 !3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "pages", scope: !6, file: !1, line: 18, type: !10, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !2)
+!6 = distinct !DISubprogram(name: "main", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 17, isOptimized: true, unit: !0, retainedNodes: !3)
 !7 = !DISubroutineType(types: !8)
 !8 = !{!9}
 !9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)

diff  --git a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
index 60010c50a8f12..c8e1cb29e5df9 100644
--- a/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
+++ b/llvm/test/DebugInfo/X86/dimodule-external-fortran.ll
@@ -73,13 +73,12 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 !3 = !DIFile(filename: "em.f90", directory: "tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2165", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
-!8 = !{!0}
 !9 = !{}
 !10 = !DIImportedEntity(tag: DW_TAG_imported_module, scope: !2, entity: !11, file: !3, line: 2)
 !11 = !DIModule(scope: !2, name: "external_module", isDecl: true)
-!12 = !{!10, !13}
+!12 = !{!0, !10, !13}
 !13 = !DILocalVariable(name: "x", scope: !2, file: !3, line: 5, type: !14)
 !14 = !DIBasicType(name: "REAL*4", size: 32, encoding: DW_ATE_float)
 !15 = !{i32 2, !"Debug Info Version", i32 3}

diff  --git a/llvm/test/DebugInfo/X86/distringtype.ll b/llvm/test/DebugInfo/X86/distringtype.ll
index 80ec7d628a0ee..2930c3aed0485 100644
--- a/llvm/test/DebugInfo/X86/distringtype.ll
+++ b/llvm/test/DebugInfo/X86/distringtype.ll
@@ -145,11 +145,11 @@ attributes #1 = { nofree nosync nounwind readnone speculatable willreturn }
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "deferred", linkageName: "assumedlength_$DEFERRED", scope: !2, file: !3, line: 2, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !7)
+!2 = distinct !DISubprogram(name: "assumedlength", linkageName: "MAIN__", scope: !3, file: !3, line: 1, type: !4, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !6, retainedNodes: !8)
 !3 = !DIFile(filename: "distring.f90", directory: "/iusers/cchen15/examples/tests")
 !4 = !DISubroutineType(types: !5)
 !5 = !{null}
-!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, globals: !8, splitDebugInlining: false, nameTableKind: None)
+!6 = distinct !DICompileUnit(language: DW_LANG_Fortran95, file: !3, producer: "Intel(R) Fortran 21.0-2142", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !7, splitDebugInlining: false, nameTableKind: None)
 !7 = !{}
 !8 = !{!0}
 !9 = !DIStringType(name: ".str.DEFERRED", stringLengthExpression: !DIExpression(DW_OP_push_object_address, DW_OP_plus_uconst, 8), stringLocationExpression: !DIExpression(DW_OP_push_object_address, DW_OP_deref))

diff  --git a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
index ab48ef2411f35..f1d619e7cb187 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-fit-segment.ll
@@ -90,9 +90,9 @@ declare void @llvm.dbg.declare(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "nodea", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4)
+!2 = distinct !DISubprogram(name: "sub", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "global-sra-struct-fit-segment.f90", directory: "/tmp")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g -O0 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0, !7, !10, !29}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression(DW_OP_plus_uconst, 4))

diff  --git a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
index 9dd865edbbdad..3dd4d9c6cca7f 100644
--- a/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
+++ b/llvm/test/DebugInfo/X86/global-sra-struct-part-overlap-segment.ll
@@ -171,10 +171,10 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !3 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !4, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: true, flags: "'+flang -g -O3 -S -emit-llvm -o -O1'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
 !4 = !DIFile(filename: "global-sra-struct-part-overlap-segment_0.f90", directory: "/tmp")
 !5 = !{}
-!6 = !{!0, !7, !29}
+!6 = !{!0}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "rvar", scope: !9, file: !4, line: 13, type: !19, isLocal: true, isDefinition: true)
-!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
+!9 = distinct !DISubprogram(name: "bar", scope: !2, file: !4, line: 10, type: !10, scopeLine: 10, retainedNodes: !32, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !3)
 !10 = !DISubroutineType(types: !11)
 !11 = !{null, !12, !16}
 !12 = !DICompositeType(tag: DW_TAG_array_type, baseType: !13, size: 96, align: 32, elements: !14)
@@ -197,3 +197,4 @@ declare void @llvm.experimental.noalias.scope.decl(metadata)
 !29 = !DIGlobalVariableExpression(var: !30, expr: !DIExpression(DW_OP_plus_uconst, 4))
 !30 = distinct !DIGlobalVariable(name: "ivar", scope: !9, file: !4, line: 13, type: !31, isLocal: true, isDefinition: true)
 !31 = !DIBasicType(name: "integer*8", size: 64, align: 32, encoding: DW_ATE_signed)
+!32 = !{!7, !29}

diff  --git a/llvm/test/DebugInfo/X86/gnu-public-names.ll b/llvm/test/DebugInfo/X86/gnu-public-names.ll
index 5d81468f55379..1c3ac763c6dc6 100644
--- a/llvm/test/DebugInfo/X86/gnu-public-names.ll
+++ b/llvm/test/DebugInfo/X86/gnu-public-names.ll
@@ -332,7 +332,7 @@ attributes #1 = { nounwind readnone speculatable }
 !14 = !{!15}
 !15 = !DIEnumerator(name: "named_enum_class_enumerator", value: 0)
 !16 = !{!13}
-!17 = !{!0, !18, !29, !32, !37, !44, !47, !50}
+!17 = !{!0, !18, !29, !32, !44, !47, !50}
 !18 = !DIGlobalVariableExpression(var: !19, expr: !DIExpression())
 !19 = distinct !DIGlobalVariable(name: "global_variable", scope: !2, file: !3, line: 13, type: !20, isLocal: false, isDefinition: true)
 !20 = distinct !DICompositeType(tag: DW_TAG_structure_type, name: "C", file: !3, line: 1, size: 8, flags: DIFlagTypePassByValue, elements: !21, identifier: "_ZTS1C")
@@ -354,7 +354,7 @@ attributes #1 = { nounwind readnone speculatable }
 !36 = !DIDerivedType(tag: DW_TAG_member, name: "A", scope: !34, file: !3, line: 22, baseType: !13, size: 32)
 !37 = !DIGlobalVariableExpression(var: !38, expr: !DIExpression())
 !38 = distinct !DIGlobalVariable(name: "z", scope: !39, file: !3, line: 33, type: !13, isLocal: true, isDefinition: true)
-!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !43)
+!39 = distinct !DISubprogram(name: "f3", linkageName: "_Z2f3v", scope: !3, file: !3, line: 32, type: !40, scopeLine: 32, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !91)
 !40 = !DISubroutineType(types: !41)
 !41 = !{!42}
 !42 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64)
@@ -406,3 +406,4 @@ attributes #1 = { nounwind readnone speculatable }
 !88 = !DILocation(line: 59, column: 68, scope: !78)
 !89 = !DILocation(line: 60, column: 32, scope: !78)
 !90 = !DILocation(line: 59, column: 3, scope: !78)
+!91 = !{!37}

diff  --git a/llvm/test/DebugInfo/X86/lexical-block-static-var.ll b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
new file mode 100644
index 0000000000000..2c90ac02d4306
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/lexical-block-static-var.ll
@@ -0,0 +1,145 @@
+; RUN: %llc_dwarf -O0 -filetype=obj < %s | llvm-dwarfdump -debug-info - | FileCheck --implicit-check-not "{{DW_TAG|NULL}}" %s
+
+; inline __attribute__((always_inline))
+; int removed() {
+;   {
+;     static int A;
+;     return A++;
+;   }
+; }
+;
+; __attribute__((always_inline))
+; int not_removed() {
+;   {
+;     static int B;
+;     return B++;
+;   }
+; }
+;
+; int foo() {
+;   {
+;     static int C;
+;     return ++C + removed() + not_removed();
+;   }
+; }
+
+; CHECK: DW_TAG_compile_unit
+
+; Out-of-line definition of `not_removed()`.
+; The empty lexical block is created to match abstract origin.
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_abstract_origin {{.*}} "_Z11not_removedv"
+; CHECK:     DW_TAG_lexical_block
+; CHECK:     NULL
+
+; Abstract definition of `removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("A")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   DW_TAG_base_type
+
+; Abstract definition of `not_removed()`
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("not_removed")
+; CHECK:     DW_AT_inline	(DW_INL_inlined)
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("B")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+
+; Definition of foo().
+; CHECK:   DW_TAG_subprogram
+; CHECK:     DW_AT_name	("foo")
+; CHECK:     DW_TAG_lexical_block
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_inlined_subroutine
+; CHECK:         DW_TAG_lexical_block
+; CHECK:         NULL
+; CHECK:       DW_TAG_variable
+; CHECK:         DW_AT_name	("C")
+; CHECK:         DW_AT_location	(DW_OP_addr 0x0)
+; CHECK:       NULL
+; CHECK:     NULL
+; CHECK:   NULL
+
+target triple = "x86_64-unknown-linux-gnu"
+
+ at _ZZ11not_removedvE1B = internal global i32 0, align 4, !dbg !0
+ at _ZZ3foovE1C = internal global i32 0, align 4, !dbg !10
+ at _ZZ7removedvE1A = linkonce_odr dso_local global i32 0, align 4, !dbg !15
+
+define dso_local i32 @_Z11not_removedv() !dbg !4 {
+entry:
+  %0 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  %inc = add nsw i32 %0, 1, !dbg !25
+  store i32 %inc, i32* @_ZZ11not_removedvE1B, align 4, !dbg !25
+  ret i32 %0, !dbg !26
+}
+
+define dso_local i32 @_Z3foov() !dbg !13 {
+entry:
+  %0 = load i32, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %inc = add nsw i32 %0, 1, !dbg !27
+  store i32 %inc, i32* @_ZZ3foovE1C, align 4, !dbg !27
+  %1 = load i32, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %inc.i3 = add nsw i32 %1, 1, !dbg !28
+  store i32 %inc.i3, i32* @_ZZ7removedvE1A, align 4, !dbg !28
+  %add = add nsw i32 %inc, %1, !dbg !30
+  %2 = load i32, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %inc.i = add nsw i32 %2, 1, !dbg !31
+  store i32 %inc.i, i32* @_ZZ11not_removedvE1B, align 4, !dbg !31
+  %add2 = add nsw i32 %add, %2, !dbg !33
+  ret i32 %add2, !dbg !34
+}
+
+!llvm.dbg.cu = !{!8}
+!llvm.module.flags = !{!19, !20, !21, !22, !23}
+!llvm.ident = !{!24}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "B", scope: !2, file: !3, line: 13, type: !7, isLocal: true, isDefinition: true)
+!2 = distinct !DILexicalBlock(scope: !4, file: !3, line: 12, column: 3)
+!3 = !DIFile(filename: "test_static.cpp", directory: "/")
+!4 = distinct !DISubprogram(name: "not_removed", linkageName: "_Z11not_removedv", scope: !3, file: !3, line: 11, type: !5, scopeLine: 11, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !35)
+!5 = !DISubroutineType(types: !6)
+!6 = !{!7}
+!7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!8 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, producer: "clang version 14.0.0", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
+!9 = !{!15}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "C", scope: !12, file: !3, line: 20, type: !7, isLocal: true, isDefinition: true)
+!12 = distinct !DILexicalBlock(scope: !13, file: !3, line: 19, column: 3)
+!13 = distinct !DISubprogram(name: "foo", linkageName: "_Z3foov", scope: !3, file: !3, line: 18, type: !5, scopeLine: 18, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !36)
+!14 = !{}
+!15 = !DIGlobalVariableExpression(var: !16, expr: !DIExpression())
+!16 = distinct !DIGlobalVariable(name: "A", scope: !17, file: !3, line: 5, type: !7, isLocal: false, isDefinition: true)
+!17 = distinct !DILexicalBlock(scope: !18, file: !3, line: 4, column: 3)
+!18 = distinct !DISubprogram(name: "removed", linkageName: "_Z7removedv", scope: !3, file: !3, line: 3, type: !5, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !8, retainedNodes: !9)
+!19 = !{i32 7, !"Dwarf Version", i32 4}
+!20 = !{i32 2, !"Debug Info Version", i32 3}
+!21 = !{i32 1, !"wchar_size", i32 4}
+!22 = !{i32 7, !"uwtable", i32 1}
+!23 = !{i32 7, !"frame-pointer", i32 2}
+!24 = !{!"clang version 14.0.0"}
+!25 = !DILocation(line: 14, column: 13, scope: !2)
+!26 = !DILocation(line: 14, column: 5, scope: !2)
+!27 = !DILocation(line: 21, column: 12, scope: !12)
+!28 = !DILocation(line: 6, column: 13, scope: !17, inlinedAt: !29)
+!29 = distinct !DILocation(line: 21, column: 18, scope: !12)
+!30 = !DILocation(line: 21, column: 16, scope: !12)
+!31 = !DILocation(line: 14, column: 13, scope: !2, inlinedAt: !32)
+!32 = distinct !DILocation(line: 21, column: 30, scope: !12)
+!33 = !DILocation(line: 21, column: 28, scope: !12)
+!34 = !DILocation(line: 21, column: 5, scope: !12)
+!35 = !{!0}
+!36 = !{!10}

diff  --git a/llvm/test/DebugInfo/X86/namelist1.ll b/llvm/test/DebugInfo/X86/namelist1.ll
index 5132c894e8138..8e70809f67e9a 100644
--- a/llvm/test/DebugInfo/X86/namelist1.ll
+++ b/llvm/test/DebugInfo/X86/namelist1.ll
@@ -47,9 +47,9 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression(DW_OP_plus_uconst, 120))
 !1 = distinct !DIGlobalVariable(name: "a", scope: !2, file: !3, line: 3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !7, scopeLine: 1, flags: DIFlagAllCallsDescribed, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !6)
 !3 = !DIFile(filename: "namelist.f90", directory: "/dir")
-!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5, nameTableKind: None)
+!4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5 2017-05-01", isOptimized: false, flags: "'+flang -g namelist.f90 -S -emit-llvm'", runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, imports: !5, nameTableKind: None)
 !5 = !{}
 !6 = !{!0}
 !7 = !DISubroutineType(cc: DW_CC_program, types: !8)

diff  --git a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
index 786c78d33337b..99c24b08c0d75 100644
--- a/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
+++ b/llvm/test/DebugInfo/X86/sdag-dbgvalue-phi-use-4.ll
@@ -19,39 +19,39 @@
 ; INSTRREF-SAME:    debug-instr-number 2
 ; CHECK-NEXT:    [[REG3:%[0-9]+]]:gr32 = PHI
 ; INSTRREF-SAME:    debug-instr-number 3
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !13, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
-; INSTRREF-NEXT: DBG_INSTR_REF !12, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !13, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
-; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
-; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !12, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 0, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 32, 32), dbg-instr-ref(2, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !16, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 64, 16), dbg-instr-ref(3, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 10, 32), dbg-instr-ref(1, 0)
+; INSTRREF-NEXT: DBG_INSTR_REF !15, !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_fragment, 42, 13), dbg-instr-ref(2, 0)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 0, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 32, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG3]], $noreg,  !16, !DIExpression(DW_OP_LLVM_fragment, 64, 16)
+; DBGVALUE-NEXT: DBG_VALUE [[REG1]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 10, 32)
+; DBGVALUE-NEXT: DBG_VALUE [[REG2]], $noreg,  !15, !DIExpression(DW_OP_LLVM_fragment, 42, 13)
 ; CHECK-NOT:  DBG_
 
 target datalayout = "e-m:x-p:32:32-i64:64-f80:32-n8:16:32-a:0:32-S32"
 target triple = "i686-w64-windows-gnu"
 
 ; Function Attrs: nounwind readnone
-define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !6 {
+define dso_local i64 @nextafterl(i80 %a, i1 %arg) local_unnamed_addr #0 !dbg !5 {
 entry:
-  br i1 %arg, label %if.else, label %if.then13, !dbg !28
+  br i1 %arg, label %if.else, label %if.then13, !dbg !27
 
 if.then13:                                        ; preds = %entry
-  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !29
-  br label %if.end36, !dbg !33
+  %u.sroa.0.8.insert.insert = or i80 %a, 2222, !dbg !28
+  br label %if.end36, !dbg !32
 
 if.else:                                          ; preds = %entry
   br label %if.end36
 
 if.end36:                                         ; preds = %if.else, %if.then13
   %u.sroa.0.1.in = phi i80 [ %u.sroa.0.8.insert.insert, %if.then13 ], [ 1234567, %if.else ]
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !13, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !34
-  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !12, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !34
-  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !35
-  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !35
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !16, metadata !DIExpression(DW_OP_LLVM_fragment, 0, 80)), !dbg !33
+  call void @llvm.dbg.value(metadata i80 %u.sroa.0.1.in, metadata !15, metadata !DIExpression(DW_OP_LLVM_fragment, 10, 45)), !dbg !33
+  %u.sroa.0.0.extract.ashr = ashr i80 %u.sroa.0.1.in, 8, !dbg !34
+  %u.sroa.0.0.extract.trunc = trunc i80 %u.sroa.0.0.extract.ashr to i64, !dbg !34
   ret i64 %u.sroa.0.0.extract.trunc
 }
 
@@ -62,42 +62,41 @@ attributes #0 = { nounwind readnone }
 attributes #1 = { nounwind readnone speculatable }
 
 !llvm.dbg.cu = !{!0}
-!llvm.module.flags = !{!26, !27}
+!llvm.module.flags = !{!4, !3}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C99, file: !1, producer: "clang version 7.0.0 (trunk 330808) (llvm/trunk 330813)", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "pr37321.c", directory: "")
 !2 = !{}
-!3 = !{!4}
-!4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
-!5 = distinct !DIGlobalVariable(name: "normal_bit", scope: !6, file: !1, line: 31, type: !25, isLocal: true, isDefinition: true)
-!6 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !7, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !10)
-!7 = !DISubroutineType(types: !8)
-!8 = !{!9, !9, !9}
-!9 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
-!10 = !{!11, !12, !13}
-!11 = !DILocalVariable(name: "x", arg: 1, scope: !6, file: !1, line: 17, type: !9)
-!12 = !DILocalVariable(name: "y", arg: 2, scope: !6, file: !1, line: 17, type: !9)
-!13 = !DILocalVariable(name: "u", scope: !6, file: !1, line: 27, type: !14)
-!14 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !6, file: !1, line: 19, size: 128, elements: !15)
-!15 = !{!16, !17}
-!16 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !14, file: !1, line: 20, baseType: !9, size: 96)
-!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !14, file: !1, line: 26, baseType: !18, size: 128)
-!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !14, file: !1, line: 21, size: 128, elements: !19)
-!19 = !{!20, !22, !24}
-!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !21, size: 64)
-!21 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
-!22 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !23, size: 16, offset: 64)
-!23 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
-!24 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !23, size: 16, offset: 80)
-!25 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !21)
-!26 = !{i32 2, !"Debug Info Version", i32 3}
-!27 = !{i32 1, !"wchar_size", i32 2}
-!28 = !DILocation(line: 47, column: 7, scope: !6)
-!29 = !DILocation(line: 51, column: 14, scope: !30)
-!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 50, column: 11)
-!31 = distinct !DILexicalBlock(scope: !32, file: !1, line: 48, column: 5)
-!32 = distinct !DILexicalBlock(scope: !6, file: !1, line: 47, column: 7)
-!33 = !DILocation(line: 51, column: 2, scope: !30)
-!34 = !DILocation(line: 27, column: 5, scope: !6)
-!35 = !DILocation(line: 63, column: 22, scope: !36)
-!36 = distinct !DILexicalBlock(scope: !6, file: !1, line: 62, column: 7)
+!3 = !{i32 1, !"wchar_size", i32 2}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "nextafterl", scope: !1, file: !1, line: 17, type: !6, isLocal: false, isDefinition: true, scopeLine: 18, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !9)
+!6 = !DISubroutineType(types: !7)
+!7 = !{!8, !8, !8}
+!8 = !DIBasicType(name: "long double", size: 96, encoding: DW_ATE_float)
+!9 = !{!10, !14, !15, !16}
+!10 = !DIGlobalVariableExpression(var: !11, expr: !DIExpression())
+!11 = distinct !DIGlobalVariable(name: "normal_bit", scope: !5, file: !1, line: 31, type: !12, isLocal: true, isDefinition: true)
+!12 = !DIDerivedType(tag: DW_TAG_const_type, baseType: !13)
+!13 = !DIBasicType(name: "long long unsigned int", size: 64, encoding: DW_ATE_unsigned)
+!14 = !DILocalVariable(name: "x", arg: 1, scope: !5, file: !1, line: 17, type: !8)
+!15 = !DILocalVariable(name: "y", arg: 2, scope: !5, file: !1, line: 17, type: !8)
+!16 = !DILocalVariable(name: "u", scope: !5, file: !1, line: 27, type: !24)
+!17 = !DIDerivedType(tag: DW_TAG_member, name: "parts", scope: !24, file: !1, line: 26, baseType: !18, size: 128)
+!18 = distinct !DICompositeType(tag: DW_TAG_structure_type, scope: !24, file: !1, line: 21, size: 128, elements: !19)
+!19 = !{!20, !21, !23}
+!20 = !DIDerivedType(tag: DW_TAG_member, name: "mantissa", scope: !18, file: !1, line: 23, baseType: !13, size: 64)
+!21 = !DIDerivedType(tag: DW_TAG_member, name: "expn", scope: !18, file: !1, line: 24, baseType: !22, size: 16, offset: 64)
+!22 = !DIBasicType(name: "unsigned short", size: 16, encoding: DW_ATE_unsigned)
+!23 = !DIDerivedType(tag: DW_TAG_member, name: "pad", scope: !18, file: !1, line: 25, baseType: !22, size: 16, offset: 80)
+!24 = distinct !DICompositeType(tag: DW_TAG_union_type, scope: !5, file: !1, line: 19, size: 128, elements: !25)
+!25 = !{!26, !17}
+!26 = !DIDerivedType(tag: DW_TAG_member, name: "ld", scope: !24, file: !1, line: 20, baseType: !8, size: 96)
+!27 = !DILocation(line: 47, column: 7, scope: !5)
+!28 = !DILocation(line: 51, column: 14, scope: !29)
+!29 = distinct !DILexicalBlock(scope: !30, file: !1, line: 50, column: 11)
+!30 = distinct !DILexicalBlock(scope: !31, file: !1, line: 48, column: 5)
+!31 = distinct !DILexicalBlock(scope: !5, file: !1, line: 47, column: 7)
+!32 = !DILocation(line: 51, column: 2, scope: !29)
+!33 = !DILocation(line: 27, column: 5, scope: !5)
+!34 = !DILocation(line: 63, column: 22, scope: !35)
+!35 = distinct !DILexicalBlock(scope: !5, file: !1, line: 62, column: 7)

diff  --git a/llvm/test/DebugInfo/duplicate_dbgvalue.ll b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
index 8fb3eeb74e012..21d97aef2c810 100644
--- a/llvm/test/DebugInfo/duplicate_dbgvalue.ll
+++ b/llvm/test/DebugInfo/duplicate_dbgvalue.ll
@@ -107,11 +107,11 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 
 !0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
 !1 = distinct !DIGlobalVariable(name: "res", scope: !2, file: !3, type: !9, isLocal: true, isDefinition: true)
-!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4)
+!2 = distinct !DISubprogram(name: "main", scope: !4, file: !3, line: 1, type: !12, scopeLine: 1, spFlags: DISPFlagDefinition | DISPFlagMainSubprogram, unit: !4, retainedNodes: !60)
 !3 = !DIFile(filename: "duplicate-dbgvalue.f90", directory: "/dir")
 !4 = distinct !DICompileUnit(language: DW_LANG_Fortran90, file: !3, producer: " F90 Flang - 1.5", isOptimized: false, runtimeVersion: 0, emissionKind: FullDebug, enums: !5, retainedTypes: !5, globals: !6, imports: !5)
 !5 = !{}
-!6 = !{!0, !7, !10}
+!6 = !{!7, !10}
 !7 = !DIGlobalVariableExpression(var: !8, expr: !DIExpression())
 !8 = distinct !DIGlobalVariable(name: "res", scope: !4, file: !3, type: !9, isLocal: true, isDefinition: true)
 !9 = !DIBasicType(name: "integer", size: 32, align: 32, encoding: DW_ATE_signed)
@@ -165,3 +165,4 @@ declare void @llvm.dbg.value(metadata, metadata, metadata)
 !57 = !{!"unlimited ptr", !58, i64 0}
 !58 = !{!"Flang FAA 3"}
 !59 = !DILocation(line: 15, column: 1, scope: !52)
+!60 = !{!0}

diff  --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
index 29bd0037a7e8b..43a1db5b2cf4e 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate-bitmap.ll
@@ -22,7 +22,7 @@ define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
   call void @llvm.instrprof.mcdc.parameters(ptr @__profn_foo, i64 12345678, i32 1)
   call void @llvm.instrprof.mcdc.tvbitmap.update(ptr @__profn_foo, i64 12345678, i32 0, ptr %mcdc.addr)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -50,6 +50,7 @@ declare void @llvm.instrprof.mcdc.tvbitmap.update(ptr, i64, i32, ptr)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, column: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram

diff  --git a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
index 84eaab33701a4..c1a610791be2b 100644
--- a/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
+++ b/llvm/test/Instrumentation/InstrProfiling/debug-info-correlate.ll
@@ -20,7 +20,7 @@
 
 define void @_Z3foov() !dbg !12 {
   call void @llvm.instrprof.increment(ptr @__profn_foo, i64 12345678, i32 2, i32 0)
-  ret void
+  ret void, !dbg !17
 }
 
 declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
@@ -46,6 +46,7 @@ declare void @llvm.instrprof.increment(ptr, i64, i32, i32)
 !14 = !DISubroutineType(types: !15)
 !15 = !{null}
 !16 = !{}
+!17 = !DILocation(line: 1, scope: !12)
 
 ; CHECK-DWARF: DW_TAG_compile_unit
 ; CHECK-DWARF:   DW_TAG_subprogram

diff  --git a/llvm/test/Transforms/Util/clone-dicompileunit.ll b/llvm/test/Transforms/Util/clone-dicompileunit.ll
index c417aafdaced3..ba40aa32c1428 100644
--- a/llvm/test/Transforms/Util/clone-dicompileunit.ll
+++ b/llvm/test/Transforms/Util/clone-dicompileunit.ll
@@ -25,18 +25,23 @@
 
 ; CHECK: DICompileUnit
 ; CHECK-NOT: DICompileUnit
+; CHECK: DISubprogram
+; CHECK-NOT: DICompileUnit
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux"
 
+define void @foo() !dbg !6 {
+  ret void
+}
+
 !llvm.dbg.cu = !{!0}
 !llvm.module.flags = !{!28, !29}
 !llvm.ident = !{!30}
 
-!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !3)
+!0 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !1, producer: "clang version 5.0.0 ", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2, globals: !2)
 !1 = !DIFile(filename: "p.cpp", directory: "/usr/local/google/home/pcc/b682773-2-repro/small2")
 !2 = !{}
-!3 = !{!4}
 !4 = !DIGlobalVariableExpression(var: !5, expr: !DIExpression())
 !5 = distinct !DIGlobalVariable(name: "a", scope: !6, file: !1, line: 5, type: !27, isLocal: true, isDefinition: true)
 !6 = distinct !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: true, isDefinition: true, scopeLine: 5, flags: DIFlagPrototyped, isOptimized: true, unit: !0, templateParams: !11, declaration: !23, retainedNodes: !24)
@@ -57,7 +62,7 @@ target triple = "x86_64-unknown-linux"
 !21 = !{null, !22}
 !22 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !13, size: 64, flags: DIFlagArtificial | DIFlagObjectPointer)
 !23 = !DISubprogram(name: "m_fn2<B>", linkageName: "_ZN1A5m_fn2IZ3foovE1BEEvv", scope: !7, file: !1, line: 5, type: !8, isLocal: false, isDefinition: false, scopeLine: 5, flags: DIFlagPublic | DIFlagPrototyped, isOptimized: true, templateParams: !11)
-!24 = !{!25}
+!24 = !{!25, !4}
 !25 = !DILocalVariable(name: "this", arg: 1, scope: !6, type: !26, flags: DIFlagArtificial | DIFlagObjectPointer)
 !26 = !DIDerivedType(tag: DW_TAG_pointer_type, baseType: !7, size: 64)
 !27 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)

diff  --git a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
index 45b7574f1843e..aeee0b3d95764 100644
--- a/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
+++ b/llvm/test/tools/llvm-debuginfo-analyzer/DWARF/Inputs/ThreadLocalStorage.ll
@@ -25,7 +25,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !1 = distinct !DIGlobalVariable(name: "TGlobal", scope: !2, file: !3, line: 1, type: !7, isLocal: false, isDefinition: true)
 !2 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: !3, emissionKind: FullDebug, globals: !4)
 !3 = !DIFile(filename: "ThreadLocalStorage.cpp", directory: "")
-!4 = !{!0, !5, !8}
+!4 = !{!0, !5}
 !5 = !DIGlobalVariableExpression(var: !6, expr: !DIExpression())
 !6 = distinct !DIGlobalVariable(name: "NGlobal", scope: !2, file: !3, line: 2, type: !7, isLocal: false, isDefinition: true)
 !7 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
@@ -34,7 +34,7 @@ declare nonnull ptr @llvm.threadlocal.address.p0(ptr nonnull)
 !10 = distinct !DISubprogram(name: "test", scope: !3, file: !3, line: 3, type: !11, scopeLine: 3, flags: DIFlagPrototyped, spFlags: DISPFlagDefinition, unit: !2, retainedNodes: !13)
 !11 = !DISubroutineType(types: !12)
 !12 = !{null}
-!13 = !{}
+!13 = !{!8}
 !14 = !{i32 7, !"Dwarf Version", i32 5}
 !15 = !{i32 2, !"Debug Info Version", i32 3}
 !22 = !DILocation(line: 5, scope: !10)

diff  --git a/llvm/unittests/Transforms/Utils/CloningTest.cpp b/llvm/unittests/Transforms/Utils/CloningTest.cpp
index 38888cd9e3666..667bbff2a41d1 100644
--- a/llvm/unittests/Transforms/Utils/CloningTest.cpp
+++ b/llvm/unittests/Transforms/Utils/CloningTest.cpp
@@ -835,7 +835,7 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !6 = distinct !DILexicalBlock(scope: !4, file: !1, line: 1)
     !7 = !DILocation(line: 1, scope: !6, inlinedAt: !8)
     !8 = !DILocation(line: 10, scope: !3)
-    !9 = !{!15, !17, !18, !23, !26, !28, !30}
+    !9 = !{!15, !17, !18, !23, !26, !28, !30, !33}
     !14 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !0, file: !1, line: 13, size: 200, elements: !{})
     !15 = !DILocalVariable(name: "a", scope: !3)
     !16 = distinct !DICompositeType(tag: DW_TAG_enumeration_type, scope: !3, file: !1, line: 13, size: 208, elements: !{})
@@ -853,6 +853,8 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
     !30 = !DILocalVariable(name: "const_ptr", scope: !3, type: !29)
     !31 = !DISubroutineType(types: !32)
     !32 = !{null}
+    !33 = !DIGlobalVariableExpression(var: !34, expr: !DIExpression())
+    !34 = distinct !DIGlobalVariable(name: "global_var", scope: !3, file: !1, line: 5, type: !22, isLocal: true, isDefinition: true)
   )";
 
   LLVMContext Context;
@@ -880,8 +882,10 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
   DISubprogram *ClonedSP = ClonedFunc->getSubprogram();
   EXPECT_NE(FuncSP, nullptr);
   EXPECT_NE(ClonedSP, nullptr);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 7u);
-  EXPECT_EQ(FuncSP->getRetainedNodes().size(),
+  EXPECT_EQ(FuncSP->getRetainedNodes().size(), 8u);
+  // DIGlobalVariableExpressions should be excluded from retainedNodes of the
+  // cloned DISubprogram, as we don't clone them.
+  EXPECT_EQ(FuncSP->getRetainedNodes().size() - 1,
             ClonedSP->getRetainedNodes().size());
 
   // Ensure that Orig node is a clone of Copy by checking that they are
@@ -896,17 +900,20 @@ TEST(CloneFunction, CloneFunctionWithRetainedNodes) {
 
   // Check that retained nodes are cloned.
   unsigned I = 0;
-  auto CheckRetainedNode = [&](auto *Node) {
+  auto CheckRetainedNode = [&](auto *Copy) {
     // The order of retained nodes should be preserved.
-    auto *Copy =
-        cast<std::decay_t<decltype(*Node)>>(ClonedSP->getRetainedNodes()[I]);
+    auto *Node =
+        cast<std::decay_t<decltype(*Copy)>>(FuncSP->getRetainedNodes()[I]);
 
     CheckNodeIsCloned(Node, Copy);
 
     ++I;
   };
-  FuncSP->forEachRetainedNode(CheckRetainedNode, CheckRetainedNode,
-                              CheckRetainedNode, CheckRetainedNode);
+  ClonedSP->forEachRetainedNode(
+      CheckRetainedNode, CheckRetainedNode, CheckRetainedNode,
+      CheckRetainedNode, [](MDNode *_) {
+        FAIL() << "DIGlobalVariableExpression should not be cloned.";
+      });
 
   auto ToDerived = [](const DIType *Ty) { return cast<DIDerivedType>(Ty); };
 
@@ -1195,6 +1202,15 @@ TEST_F(CloneModule, Subprogram) {
   EXPECT_EQ(SP->getName(), "f");
   EXPECT_EQ(SP->getFile()->getFilename(), "filename.c");
   EXPECT_EQ(SP->getLine(), (unsigned)4);
+
+  // Check static locals have the correct scope.
+  MDNodeArray LocalDeclsArray = SP->getRetainedNodes();
+  EXPECT_EQ(LocalDeclsArray.size(), 2U);
+  for (auto *Node : LocalDeclsArray) {
+    auto *GVExpr = cast<DIGlobalVariableExpression>(Node);
+    DIGlobalVariable *GV = GVExpr->getVariable();
+    EXPECT_EQ(GV->getScope(), SP);
+  }
 }
 
 TEST_F(CloneModule, FunctionDeclarationMetadata) {
@@ -1252,14 +1268,6 @@ TEST_F(CloneModule, CompileUnit) {
   DISubprogram *SP = NewM->getFunction("f")->getSubprogram();
   EXPECT_TRUE(SP != nullptr);
   EXPECT_EQ(SP->getUnit(), CU);
-
-  // Check globals listed in CU have the correct scope
-  DIGlobalVariableExpressionArray GlobalArray = CU->getGlobalVariables();
-  EXPECT_EQ(GlobalArray.size(), 2U);
-  for (DIGlobalVariableExpression *GVExpr : GlobalArray) {
-    DIGlobalVariable *GV = GVExpr->getVariable();
-    EXPECT_EQ(GV->getScope(), SP);
-  }
 }
 
 TEST_F(CloneModule, Comdat) {

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 9474afb897119..72497a5328f78 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -710,36 +710,34 @@ def LLVM_DISubprogramAttr : LLVM_Attr<"DISubprogram", "di_subprogram",
                                       [LLVM_DIRecursiveTypeAttrInterface],
                                       "DILocalScopeAttr"> {
   let parameters = (ins
-    // DIRecursiveTypeAttrInterface specific parameters.
-    OptionalParameter<"DistinctAttr">:$recId,
-    OptionalParameter<"bool">:$isRecSelf,
-    // DISubprogramAttr specific parameters.
-    OptionalParameter<"DistinctAttr">:$id,
-    OptionalParameter<"DICompileUnitAttr">:$compileUnit,
-    OptionalParameter<"DIScopeAttr">:$scope,
-    OptionalParameter<"StringAttr">:$name,
-    OptionalParameter<"StringAttr">:$linkageName,
-    OptionalParameter<"DIFileAttr">:$file,
-    OptionalParameter<"unsigned">:$line,
-    OptionalParameter<"unsigned">:$scopeLine,
-    OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
-    OptionalParameter<"DISubroutineTypeAttr">:$type,
-    OptionalArrayRefParameter<"DINodeAttr">:$retainedNodes,
-    OptionalArrayRefParameter<"DINodeAttr">:$annotations
-  );
-  let builders = [
-    AttrBuilder<(ins
-      "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
-      "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
-      "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
-      "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
-      "ArrayRef<DINodeAttr>":$retainedNodes, "ArrayRef<DINodeAttr>":$annotations
-    ), [{
+      // DIRecursiveTypeAttrInterface specific parameters.
+      OptionalParameter<"DistinctAttr">:$recId,
+      OptionalParameter<"bool">:$isRecSelf,
+      // DISubprogramAttr specific parameters.
+      OptionalParameter<"DistinctAttr">:$id,
+      OptionalParameter<"DICompileUnitAttr">:$compileUnit,
+      OptionalParameter<"DIScopeAttr">:$scope,
+      OptionalParameter<"StringAttr">:$name,
+      OptionalParameter<"StringAttr">:$linkageName,
+      OptionalParameter<"DIFileAttr">:$file,
+      OptionalParameter<"unsigned">:$line,
+      OptionalParameter<"unsigned">:$scopeLine,
+      OptionalParameter<"DISubprogramFlags">:$subprogramFlags,
+      OptionalParameter<"DISubroutineTypeAttr">:$type,
+      OptionalArrayRefParameter<"Attribute">:$retainedNodes,
+      OptionalArrayRefParameter<"DINodeAttr">:$annotations);
+  let builders = [AttrBuilder<
+      (ins "DistinctAttr":$id, "DICompileUnitAttr":$compileUnit,
+          "DIScopeAttr":$scope, "StringAttr":$name, "StringAttr":$linkageName,
+          "DIFileAttr":$file, "unsigned":$line, "unsigned":$scopeLine,
+          "DISubprogramFlags":$subprogramFlags, "DISubroutineTypeAttr":$type,
+          "ArrayRef<Attribute>":$retainedNodes,
+          "ArrayRef<DINodeAttr>":$annotations),
+      [{
       return $_get($_ctxt, /*recId=*/nullptr, /*isRecSelf=*/false, id, compileUnit,
                    scope, name, linkageName, file, line, scopeLine,
                    subprogramFlags, type, retainedNodes, annotations);
-    }]>
-  ];
+    }]>];
   let assemblyFormat = "`<` struct(params) `>`";
   let extraClassDeclaration = [{
     /// Requirements of DIRecursiveTypeAttrInterface.

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index a5040d5104ab9..b25a8c8a9829e 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -150,23 +150,21 @@ def DICompileUnitAttr : DialectAttribute<(attr
 // DISubprogramAttr
 //===----------------------------------------------------------------------===//
 
-def DISubprogramAttr : DialectAttribute<(attr
-  OptionalAttribute<"DistinctAttr">:$recId,
-  Bool:$isRecSelf,
-  OptionalAttribute<"DistinctAttr">:$id,
-  OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
-  OptionalAttribute<"DIScopeAttr">:$scope,
-  OptionalAttribute<"StringAttr">:$name,
-  OptionalAttribute<"StringAttr">:$linkageName,
-  OptionalAttribute<"DIFileAttr">:$file,
-  VarInt:$line,
-  VarInt:$scopeLine,
-  EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
-  LocalVar<"DISubprogramFlags", "(DISubprogramFlags)_rawflags">:$subprogramFlags,
-  OptionalAttribute<"DISubroutineTypeAttr">:$type,
-  OptionalArrayRef<"DINodeAttr">:$retainedNodes,
-  OptionalArrayRef<"DINodeAttr">:$annotations
-)>;
+def DISubprogramAttr
+    : DialectAttribute<(attr OptionalAttribute<"DistinctAttr">:$recId,
+          Bool:$isRecSelf, OptionalAttribute<"DistinctAttr">:$id,
+          OptionalAttribute<"DICompileUnitAttr">:$compileUnit,
+          OptionalAttribute<"DIScopeAttr">:$scope,
+          OptionalAttribute<"StringAttr">:$name,
+          OptionalAttribute<"StringAttr">:$linkageName,
+          OptionalAttribute<"DIFileAttr">:$file, VarInt:$line,
+          VarInt:$scopeLine,
+          EnumClassFlag<"DISubprogramFlags", "getSubprogramFlags()">:$_rawflags,
+          LocalVar<"DISubprogramFlags",
+                   "(DISubprogramFlags)_rawflags">:$subprogramFlags,
+          OptionalAttribute<"DISubroutineTypeAttr">:$type,
+          OptionalArrayRef<"Attribute">:$retainedNodes,
+          OptionalArrayRef<"DINodeAttr">:$annotations)>;
 
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr

diff  --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index 6b8cac201d8bf..8082de122e74e 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -467,7 +467,7 @@ MlirAttribute mlirLLVMDISubprogramAttrGet(
       cast<DISubroutineTypeAttr>(unwrap(type)),
       llvm::map_to_vector(
           unwrapList(nRetainedNodes, retainedNodes, nodesStorage),
-          llvm::CastTo<DINodeAttr>),
+          llvm::CastTo<Attribute>),
       llvm::map_to_vector(
           unwrapList(nAnnotations, annotations, annotationsStorage),
           llvm::CastTo<DINodeAttr>)));

diff  --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 82e100d254218..85df880d2712a 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -259,9 +259,15 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
     return nullptr;
 
   // Convert the retained nodes but drop all of them if one of them is invalid.
-  SmallVector<DINodeAttr> retainedNodes;
-  for (llvm::DINode *retainedNode : node->getRetainedNodes())
+  SmallVector<Attribute> retainedNodes;
+  auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
     retainedNodes.push_back(translate(retainedNode));
+  };
+  auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
+    // FIXME Import DIGlobalVariableExpressions from retainedNodes without
+    // duplicating them.
+  };
+  node->forEachRetainedNode(add, add, add, add, addGVE);
   if (llvm::is_contained(retainedNodes, nullptr))
     retainedNodes.clear();
 

diff  --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 38d8b96e82d82..442f5b6b137e6 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -114,6 +114,23 @@ DebugTranslation::getMDTupleOrNull(ArrayRef<DINodeAttr> elements) {
   return llvm::MDNode::get(llvmCtx, llvmElements);
 }
 
+llvm::MDTuple *
+DebugTranslation::getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes) {
+  if (retainedNodes.empty())
+    return nullptr;
+  SmallVector<llvm::Metadata *> llvmElements = llvm::map_to_vector(
+      retainedNodes, [&](Attribute attr) -> llvm::Metadata * {
+        if (auto GVE = dyn_cast<DIGlobalVariableExpressionAttr>(attr))
+          return translateGlobalVariableExpression(GVE);
+
+        auto diAttr = dyn_cast<DINodeAttr>(attr);
+        if (!diAttr)
+          llvm_unreachable("unknown retained node kind");
+        return translate(diAttr);
+      });
+  return llvm::MDNode::get(llvmCtx, llvmElements);
+}
+
 llvm::DIBasicType *DebugTranslation::translateImpl(DIBasicTypeAttr attr) {
   return llvm::DIBasicType::get(
       llvmCtx, attr.getTag(), getMDStringOrNull(attr.getName()),
@@ -400,7 +417,7 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getMDTupleOrNull(attr.getRetainedNodes()), nullptr,
+      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);

diff  --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index c12ddb42fe238..c1fdbd89087da 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -114,6 +114,10 @@ class DebugTranslation {
   /// `elements` is empty.
   llvm::MDTuple *getMDTupleOrNull(ArrayRef<DINodeAttr> elements);
 
+  /// Consturct a DISubprogram's retainedNodes metadata node from the
+  /// `retainedNodes`. Returns nullptr if `retainedNodes` is empty.
+  llvm::MDTuple *getRetainedNodesOrNull(ArrayRef<Attribute> retainedNodes);
+
   /// Constructs a DIExpression metadata node from the DIExpressionAttr. Returns
   /// nullptr if `DIExpressionAttr` is null.
   llvm::DIExpression *getExpressionAttrOrNull(DIExpressionAttr attr);

diff  --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index 442e6e16e955e..a9ca4a57e8e8a 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -1198,7 +1198,9 @@ convertMLIRAttributesToLLVM(Location loc, llvm::LLVMContext &ctx,
 
 LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
   // Mapping from compile unit to its respective set of global variables.
-  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> allGVars;
+  DenseMap<llvm::DICompileUnit *, SmallVector<llvm::Metadata *>> globalGVars;
+  // Mapping from subprogram to its respective set of static local variables.
+  DenseMap<llvm::DISubprogram *, SmallVector<llvm::Metadata *>> staticLocals;
 
   // First, create all global variables and global aliases in LLVM IR. A global
   // or alias body may refer to another global/alias or itself, so all the
@@ -1291,7 +1293,7 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
         //
         // 3. For entities like static local variables in C or variable with
         // SAVE attribute in Fortran, the scope hierarchy can be
-        // variable -> DISubprogram -> DICompileUnit
+        // variable (-> DILocalScope)* -> DISubprogram
         llvm::DIScope *scope = diGlobalVar->getScope();
         if (auto *mod = dyn_cast_if_present<llvm::DIModule>(scope))
           scope = mod->getScope();
@@ -1299,15 +1301,23 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
           if (auto *sp =
                   dyn_cast_if_present<llvm::DISubprogram>(cb->getScope()))
             scope = sp->getUnit();
-        } else if (auto *sp = dyn_cast_if_present<llvm::DISubprogram>(scope))
-          scope = sp->getUnit();
+        } else if (auto *lbb =
+                       dyn_cast_if_present<llvm::DILexicalBlockBase>(scope)) {
+          scope = lbb->getSubprogram();
+        }
 
-        // Get the compile unit (scope) of the the global variable.
+        // Get the compile unit (scope) of the the global variable, or the
+        // subprogram of the static local variable.
         if (llvm::DICompileUnit *compileUnit =
                 dyn_cast_if_present<llvm::DICompileUnit>(scope)) {
           // Update the compile unit with this incoming global variable
           // expression during the finalizing step later.
-          allGVars[compileUnit].push_back(diGlobalExpr);
+          globalGVars[compileUnit].push_back(diGlobalExpr);
+        } else if (llvm::DISubprogram *sp =
+                       dyn_cast_if_present<llvm::DISubprogram>(scope)) {
+          // Update the subprogram with this incoming static local variable
+          // expression during the finalizing step later.
+          staticLocals[sp].push_back(diGlobalExpr);
         }
       }
     }
@@ -1499,10 +1509,14 @@ LogicalResult ModuleTranslation::convertGlobalsAndAliases() {
 
   // Finally, update the compile units their respective sets of global variables
   // created earlier.
-  for (const auto &[compileUnit, globals] : allGVars) {
+  for (const auto &[compileUnit, globals] : globalGVars)
     compileUnit->replaceGlobalVariables(
         llvm::MDTuple::get(getLLVMContext(), globals));
-  }
+
+  // And update the subprograms with their respective sets of static local
+  // variables.
+  for (const auto &[sp, globals] : staticLocals)
+    sp->retainNodes(globals.begin(), globals.end());
 
   // Convert global alias bodies.
   for (auto op : getModuleBody(mlirModule).getOps<LLVM::AliasOp>()) {

diff  --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 643d504bcb2ac..28c25809002cd 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1014,3 +1014,35 @@ define void @variant_part_import() !dbg !3 {
 !11 = !DIBasicType(name: "float64", size: 64, encoding: DW_ATE_float)
 !12 = !DIDerivedType(tag: DW_TAG_member, name: "_int64", baseType: !10, size: 64, offset: 64, extraData: i8 1)
 !13 = !DIDerivedType(tag: DW_TAG_member, name: "_float64", baseType: !11, size: 64, offset: 64, extraData: i8 2)
+
+; // -----
+
+; Test that static local DIGlobalVariable listed in DISubprogram's retainedNodes does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<{{.*}}name = "unused_local"{{.*}}>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_static_local"{{.*}}retainedNodes = [#[[LVAR]]]>
+; CHECK-DAG: #[[GVAR:.+]] = #llvm.di_global_variable<scope = #[[SP]], name = "static_local"{{.*}}>
+; CHECK-DAG: #[[GEXPR:.+]] = #llvm.di_global_variable_expression<var = #[[GVAR]], expr = <>>
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable_expression<}}
+; CHECK-DAG-COUNT-1: {{#llvm\.di_global_variable<}}
+
+ at static_local = internal global i64 0, align 8, !dbg !0
+
+define void @fn_with_static_local() !dbg !3 {
+  ret void
+}
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!0 = !DIGlobalVariableExpression(var: !1, expr: !DIExpression())
+!1 = distinct !DIGlobalVariable(name: "static_local", linkageName: "static_local", scope: !3, file: !4, line: 121, type: !9, isLocal: true, isDefinition: true)
+!3 = distinct !DISubprogram(name: "fn_with_static_local", scope: !4, file: !4, spFlags: DISPFlagDefinition, unit: !7, type: !13, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 64, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!0, !12}
+!12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null}

diff  --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 27dcf6979d7bb..72dadb2602c52 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -365,10 +365,12 @@ llvm.mlir.global external @module_global() {dbg_exprs = [#llvm.di_global_variabl
 // -----
 
 // CHECK: @func_global = external global i64, !dbg {{.*}}
-// CHECK-DAG: ![[CU:.*]] = distinct !DICompileUnit({{.*}}globals: ![[GVALS:[0-9]+]], debugInfoForProfiling: true)
-// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU]])
+// CHECK-DAG: ![[SP:.*]] = distinct !DISubprogram(name: "fn_with_gl"{{.*}}unit: ![[CU:.*]], {{.*}}retainedNodes: ![[GVALS:.*]])
 // CHECK-DAG: ![[GVAR:.*]] = distinct !DIGlobalVariable(name: "func_global"{{.*}}, scope: ![[SP]]{{.*}})
 // CHECK-DAG: ![[GEXPR:.*]] = !DIGlobalVariableExpression(var: ![[GVAR]], expr: !DIExpression())
+// CHECK-DAG: ![[CU]] = distinct !DICompileUnit(
+// CHECK-NOT:     globals:
+// CHECK-SAME:    emissionKind: FullDebug, debugInfoForProfiling: true)
 // CHECK-DAG: ![[GVALS]] = !{![[GEXPR]]}
 
 #file = #llvm.di_file<"test.f90" in "existence">


        

From llvmlistbot at llvm.org  Fri Jul 10 08:32:32 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 10 Jul 2026 08:32:32 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a511090.da8bafd3.14cd50.b18a@mx.google.com>


https://github.com/dzhidzhoev closed https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul 10 08:40:58 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:40:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a51128a.1905e68e.18ad68.2cd5@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Fri Jul 10 08:40:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:40:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a51128b.4987c784.63cbc.6c35@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM % one question, thank you for the fix!

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Fri Jul 10 08:40:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:40:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a51128b.da8bafd3.14cd50.5853@mx.google.com>


================

----------------
banach-space wrote:

Do we need to two tests? It seems that both exercise similar code path, no? If yes, I'd only keep one.

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Fri Jul 10 08:40:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:40:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a51128b.3cce8983.225732.22d8@mx.google.com>


================

----------------
banach-space wrote:

> Glad to hear from you :)

Likewise! :)

https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Fri Jul 10 08:49:26 2026
From: llvmlistbot at llvm.org (Vladislav Dzhidzhoev)
Date: Fri, 10 Jul 2026 08:49:26 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a511486.331483e7.2440f8.e00c@mx.google.com>


dzhidzhoev wrote:

> Sure, worth a shot.

Thank you!

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul 10 08:51:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 08:51:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] b4cf9c7 - [MLIR][Vector] Relax shape_cast
 unrolling to per-reassociation-group contiguity (#205684)
Message-ID: <6a511517.0c05e68e.1e82f1.e714@mx.google.com>


Author: Jianhui Li
Date: 2026-07-10T08:51:46-07:00
New Revision: b4cf9c7c89ec541495c38c4e34fa8967713bf843

URL: https://github.com/llvm/llvm-project/commit/b4cf9c7c89ec541495c38c4e34fa8967713bf843
DIFF: https://github.com/llvm/llvm-project/commit/b4cf9c7c89ec541495c38c4e34fa8967713bf843.diff

LOG: [MLIR][Vector] Relax shape_cast unrolling to per-reassociation-group contiguity (#205684)

UnrollShapeCastPattern previously only unrolled a vector.shape_cast when
the target unroll tile was contiguous in the whole result vector
(isContiguous(targetShape, resultShape)). This rejected valid cases such
as:
```mlir
  %0 = vector.shape_cast %src : vector<8x32xf8> to vector<8x1x32xf8>  // target tile [8, 1, 4]
  %1 = vector.shape_cast %src : vector<8x32x32xf8> to vector<256x32xf8>  // target tile [16, 4]
```
This PR factors the source and result shapes of a shape_cast into
independent reassociation groups — maximal aligned source/result dim
ranges that hold equal element counts:
```
  - 8x32 → 8x1x32 ⇒ groups {8 ↔ 8x1}, {32 ↔ 32}
  - 8x32x32 → 256x32 ⇒ groups {8x32 ↔ 256}, {32 ↔ 32}
```
Instead of requiring the target tile to be contiguous in the whole
result vector, it only checks that the tile is contiguous within each
group.
This criterion is correct because a shape_cast preserves row-major
linear order. The groups partition the dimensions into blocks whose
element ranges don't overlap, so the global linear index breaks down
into one independent sub-index per group. A tile that is contiguous
within each group therefore corresponds to a slice that is contiguous
within each group of both the source and result.
  
     Assist-by-Claude

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply at anthropic.com>

Added: 
    

Modified: 
    mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
    mlir/test/Dialect/Vector/vector-unroll-options.mlir
    mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
index b33828d5d5867f..62869111496d19 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorUnroll.cpp
@@ -1285,20 +1285,68 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
   return delinearize(linearIndex, computeStrides(sourceShape));
 }
 
+/// A maximal aligned range of source dims [srcBegin, srcEnd) and result dims
+/// [resBegin, resEnd) of a `vector.shape_cast` that hold equal element counts.
+struct ShapeCastReassociationGroup {
+  int64_t srcBegin, srcEnd;
+  int64_t resBegin, resEnd;
+};
+
+/// Splits a shape_cast from `sourceShape` to `resultShape` into reassociation
+/// groups (trailing unit dims absorbed). Returns nullopt if shapes misalign.
+/// E.g. [8, 32, 32] -> [256, 32] ==> {[0,2)->[0,1)}, {[2,3)->[1,2)}
+static std::optional<SmallVector<ShapeCastReassociationGroup>>
+computeShapeCastGroups(ArrayRef<int64_t> sourceShape,
+                       ArrayRef<int64_t> resultShape) {
+  SmallVector<ShapeCastReassociationGroup> groups;
+  int64_t si = 0, ri = 0;
+  int64_t srcRank = sourceShape.size(), resRank = resultShape.size();
+  while (si < srcRank && ri < resRank) {
+    int64_t srcBegin = si, resBegin = ri;
+    int64_t srcProd = sourceShape[si++];
+    int64_t resProd = resultShape[ri++];
+    // Grow the smaller side until both groups span the same element count.
+    while (srcProd != resProd) {
+      if (srcProd < resProd) {
+        if (si >= srcRank)
+          return std::nullopt;
+        srcProd *= sourceShape[si++];
+      } else {
+        if (ri >= resRank)
+          return std::nullopt;
+        resProd *= resultShape[ri++];
+      }
+    }
+    // Absorb trailing unit dimensions into the current group.
+    while (si < srcRank && sourceShape[si] == 1)
+      ++si;
+    while (ri < resRank && resultShape[ri] == 1)
+      ++ri;
+    groups.push_back({srcBegin, si, resBegin, ri});
+  }
+  if (si != srcRank || ri != resRank)
+    return std::nullopt;
+  return groups;
+}
+
 /// This pattern unrolls `vector.shape_cast` operations according to the
 /// provided target unroll shape. It unrolls a large shape cast into smaller
 /// shape casts by extracting contiguous slices from the source vector, casting
 /// each slice to the target shape, and assembling the result by inserting each
 /// computed segment into the appropriate offset of the result vector.
 ///
-/// This pattern only applies when contiguous slices can be extracted from the
-/// source vector and inserted into the result vector such that each slice
-/// remains a valid vector (and not decompose to scalars). In these cases, the
-/// unrolling proceeds as:
+/// The target tile need only be contiguous within each reassociation group of
+/// the cast (not in the whole result vector), so that each extracted slice
+/// remains a valid vector. The unrolling proceeds as:
 /// vector.extract_strided_slice -> vector.shape_cast (on the slice) ->
 /// vector.insert_strided_slice.
 ///
-/// Example:
+/// NOTE: This replaces a NOP `vector.shape_cast` with strided slices. Per-group
+/// contiguity keeps those slices contiguous, so they are expected to lower to a
+/// NOP too. Targets where strided slices do not lower to a NOP should not use
+/// this pattern, or should pick a tile that avoids introducing such slices.
+///
+/// Example (single group):
 ///   Given a shape cast operation:
 ///     %0 = vector.shape_cast %src : vector<8x2xf32> to vector<4x4xf32>
 ///
@@ -1316,6 +1364,18 @@ calculateSourceOffsets(ArrayRef<int64_t> resultOffsets,
 ///     %i1 = vector.insert_strided_slice %sc1, %i0 [2, 0], [1, 1]
 ///       : vector<2x4xf32> into vector<4x4xf32>
 ///
+/// Example (multiple groups): with target tile <8x1x4>, the tile is strided in
+/// the result <8x1x32> but contiguous per group (8|32 -> 8x1|32), so the
+/// matching strided box <8x4> is extracted from the source:
+///     %0 = vector.shape_cast %src : vector<8x32xf32> to vector<8x1x32xf32>
+///
+///     %s0 = vector.extract_strided_slice %src [0, 0], [8, 4], [1, 1]
+///       : vector<8x32xf32> to vector<8x4xf32>
+///     %sc0 = vector.shape_cast %s0 : vector<8x4xf32> to vector<8x1x4xf32>
+///     %i0 = vector.insert_strided_slice %sc0, %zero [0, 0, 0], [1, 1, 1]
+///       : vector<8x1x4xf32> into vector<8x1x32xf32>
+///     // ... repeat for the remaining slices.
+///
 struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
   UnrollShapeCastPattern(MLIRContext *context,
                          const vector::UnrollVectorOptions &options,
@@ -1335,20 +1395,44 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     ArrayRef<int64_t> sourceShape = sourceType.getShape();
     ArrayRef<int64_t> resultShape = resultType.getShape();
 
-    if (!isContiguous(*targetShape, resultShape))
+    // The cast factors into reassociation groups; the target tile only needs to
+    // be contiguous within each group, not in the whole result vector.
+    std::optional<SmallVector<ShapeCastReassociationGroup>> groups =
+        computeShapeCastGroups(sourceShape, resultShape);
+    if (!groups)
       return rewriter.notifyMatchFailure(
-          shapeCastOp, "Only supports cases where target shape is "
-                       "contiguous in result vector shape");
-
-    int64_t targetElements = ShapedType::getNumElements(*targetShape);
-
-    // Calculate the shape to extract from source.
-    std::optional<SmallVector<int64_t>> extractShape =
-        calculateSourceExtractShape(sourceShape, targetElements);
-    if (!extractShape)
-      return rewriter.notifyMatchFailure(
-          shapeCastOp,
-          "cannot extract target number of elements contiguously from source");
+          shapeCastOp, "cannot align source and result reassociation groups");
+
+    // The tile is right-aligned against the result; left-pad with 1s so it can
+    // be indexed per group.
+    SmallVector<int64_t> paddedTarget(resultShape.size(), 1);
+    llvm::copy(*targetShape,
+               paddedTarget.end() - static_cast<int64_t>(targetShape->size()));
+
+    // Validate per-group contiguity and build the source extract shape.
+    SmallVector<int64_t> extractShapeStorage;
+    for (const ShapeCastReassociationGroup &g : *groups) {
+      ArrayRef<int64_t> resSub =
+          resultShape.slice(g.resBegin, g.resEnd - g.resBegin);
+      ArrayRef<int64_t> tgtSub = ArrayRef<int64_t>(paddedTarget)
+                                     .slice(g.resBegin, g.resEnd - g.resBegin);
+      if (!isContiguous(tgtSub, resSub))
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "target shape is not contiguous within a "
+                         "reassociation group of the result vector shape");
+
+      ArrayRef<int64_t> srcSub =
+          sourceShape.slice(g.srcBegin, g.srcEnd - g.srcBegin);
+      int64_t groupTargetElements = ShapedType::getNumElements(tgtSub);
+      std::optional<SmallVector<int64_t>> groupExtract =
+          calculateSourceExtractShape(srcSub, groupTargetElements);
+      if (!groupExtract)
+        return rewriter.notifyMatchFailure(
+            shapeCastOp, "cannot extract the target number of elements "
+                         "contiguously from a source reassociation group");
+      extractShapeStorage.append(groupExtract->begin(), groupExtract->end());
+    }
+    ArrayRef<int64_t> extractShape = extractShapeStorage;
 
     Location loc = shapeCastOp.getLoc();
 
@@ -1359,7 +1443,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
     VectorType targetType =
         VectorType::get(*targetShape, sourceType.getElementType());
 
-    SmallVector<int64_t> extractStrides(extractShape->size(), 1);
+    SmallVector<int64_t> extractStrides(extractShape.size(), 1);
     SmallVector<int64_t> insertStrides(targetShape->size(), 1);
 
     for (SmallVector<int64_t> resultOffsets :
@@ -1367,7 +1451,7 @@ struct UnrollShapeCastPattern : public OpRewritePattern<vector::ShapeCastOp> {
       SmallVector<int64_t> sourceOffsets =
           calculateSourceOffsets(resultOffsets, sourceShape, resultShape);
       Value sourceChunk = rewriter.createOrFold<vector::ExtractStridedSliceOp>(
-          loc, shapeCastOp.getSource(), sourceOffsets, *extractShape,
+          loc, shapeCastOp.getSource(), sourceOffsets, extractShape,
           extractStrides);
       Value targetChunk = rewriter.createOrFold<vector::ShapeCastOp>(
           loc, targetType, sourceChunk);

diff  --git a/mlir/test/Dialect/Vector/vector-unroll-options.mlir b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
index 0ec71620c5324d..da058d5d2410d5 100644
--- a/mlir/test/Dialect/Vector/vector-unroll-options.mlir
+++ b/mlir/test/Dialect/Vector/vector-unroll-options.mlir
@@ -703,6 +703,76 @@ func.func @shape_cast_with_all_unit_target_shape(%v: vector<2xf32>) -> vector<2x
 // CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [1, 0], strides = [1, 1]} : vector<1x1xf32> into vector<2x1xf32>
 // CHECK:   return %[[I1]] : vector<2x1xf32>
 
+
+// Target tile [8, 1, 4] is strided in result <8x1x32> but contiguous per
+// reassociation group (8|32 -> 8x1|32). TargetShape is [8, 1, 4].
+func.func @shape_cast_multi_group_rank_increasing(%v: vector<8x32xf32>) -> vector<8x1x32xf32> {
+  %0 = vector.shape_cast %v : vector<8x32xf32> to vector<8x1x32xf32>
+  return %0 : vector<8x1x32xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_increasing
+// CHECK-SAME: (%[[V:.*]]: vector<8x32xf32>) -> vector<8x1x32xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<8x1x32xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 4], sizes = [8, 4], strides = [1, 1]} : vector<8x32xf32> to vector<8x4xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<8x4xf32> to vector<8x1x4xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 0, 4], strides = [1, 1, 1]} : vector<8x1x4xf32> into vector<8x1x32xf32>
+// CHECK:   return
+
+
+// Target tile [2, 2] is strided in result <4x4> but contiguous per
+// reassociation group (2x2|4 -> 4|4). TargetShape is [2, 2].
+func.func @shape_cast_multi_group_rank_decreasing(%v: vector<2x2x4xf32>) -> vector<4x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x2x4xf32> to vector<4x4xf32>
+  return %0 : vector<4x4xf32>
+}
+
+// CHECK-LABEL: func @shape_cast_multi_group_rank_decreasing
+// CHECK-SAME: (%[[V:.*]]: vector<2x2x4xf32>) -> vector<4x4xf32> {
+// CHECK:   %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<4x4xf32>
+// CHECK:   %[[S0:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC0:.*]] = vector.shape_cast %[[S0]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I0:.*]] = vector.insert_strided_slice %[[SC0]], %[[CST]] {offsets = [0, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S1:.*]] = vector.extract_strided_slice %[[V]] {offsets = [0, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC1:.*]] = vector.shape_cast %[[S1]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I1:.*]] = vector.insert_strided_slice %[[SC1]], %[[I0]] {offsets = [0, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S2:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 0], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC2:.*]] = vector.shape_cast %[[S2]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I2:.*]] = vector.insert_strided_slice %[[SC2]], %[[I1]] {offsets = [2, 0], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   %[[S3:.*]] = vector.extract_strided_slice %[[V]] {offsets = [1, 0, 2], sizes = [1, 2, 2], strides = [1, 1, 1]} : vector<2x2x4xf32> to vector<1x2x2xf32>
+// CHECK:   %[[SC3:.*]] = vector.shape_cast %[[S3]] : vector<1x2x2xf32> to vector<2x2xf32>
+// CHECK:   %[[I3:.*]] = vector.insert_strided_slice %[[SC3]], %[[I2]] {offsets = [2, 2], strides = [1, 1]} : vector<2x2xf32> into vector<4x4xf32>
+// CHECK:   return %[[I3]] : vector<4x4xf32>
+
+// Negative multi-group case: target tile [2, 2, 2] is not contiguous within
+// the result reassociation group [8, 4], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_target_not_contiguous(%v: vector<2x32xf32>) -> vector<2x8x4xf32> {
+  %0 = vector.shape_cast %v : vector<2x32xf32> to vector<2x8x4xf32>
+  return %0 : vector<2x8x4xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_target_not_contiguous
+// CHECK-SAME: (%[[V:.*]]: vector<2x32xf32>) -> vector<2x8x4xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x32xf32> to vector<2x8x4xf32>
+// CHECK:   return %[[SC]] : vector<2x8x4xf32>
+
+
+// Negative multi-group case: the target tile [2, 4] is contiguous within the result
+// group [24], but its elements cannot be extracted contiguously from the
+// source group [8, 3], so the cast is left un-unrolled.
+func.func @negative_shape_cast_multi_group_source_not_determinable(%v: vector<2x8x3xf32>) -> vector<2x24xf32> {
+  %0 = vector.shape_cast %v : vector<2x8x3xf32> to vector<2x24xf32>
+  return %0 : vector<2x24xf32>
+}
+
+// CHECK-LABEL: func @negative_shape_cast_multi_group_source_not_determinable
+// CHECK-SAME: (%[[V:.*]]: vector<2x8x3xf32>) -> vector<2x24xf32> {
+// CHECK:   %[[SC:.*]] = vector.shape_cast %[[V]] : vector<2x8x3xf32> to vector<2x24xf32>
+// CHECK:   return %[[SC]] : vector<2x24xf32>
+
 // -----
 
 // Test BitCastOp unrolling - target shape [4, 4]

diff  --git a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
index 1a4bef664fbe1f..4523a4cd3c4867 100644
--- a/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
+++ b/mlir/test/lib/Dialect/Vector/TestVectorTransforms.cpp
@@ -211,6 +211,21 @@ struct TestVectorUnrollingPatterns
                       resultShape[1] == 1) {
                     return SmallVector<int64_t>{1, 1};
                   }
+                  // Multi-group cases: tile contiguous per reassociation group
+                  // but strided in the whole result.
+                  auto sourceShape = shapeCast.getSourceVectorType().getShape();
+                  if (resultShape.size() == 3 && resultShape[0] == 8 &&
+                      resultShape[1] == 1 && resultShape[2] == 32) {
+                    return SmallVector<int64_t>{8, 1, 4};
+                  }
+                  if (sourceShape.size() == 3 && resultShape.size() == 2 &&
+                      resultShape[0] == 4 && resultShape[1] == 4) {
+                    return SmallVector<int64_t>{2, 2};
+                  }
+                  if (resultShape.size() == 3 && resultShape[0] == 2 &&
+                      resultShape[1] == 8 && resultShape[2] == 4) {
+                    return SmallVector<int64_t>{2, 2, 2};
+                  }
                   // Default case: [2,4] for all tests.
                   return SmallVector<int64_t>{2, 4};
                 })


        

From llvmlistbot at llvm.org  Fri Jul 10 08:51:54 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 10 Jul 2026 08:51:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Vector] Relax shape_cast unrolling to
 per-reassociation-group contiguity (PR #205684)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205684@github.com>
Message-ID: <6a51151a.481afadf.196d57.118a@mx.google.com>


https://github.com/Jianhui-Li closed https://github.com/llvm/llvm-project/pull/205684

From llvmlistbot at llvm.org  Fri Jul 10 08:52:21 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:52:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a511535.bd93e731.358318.2eb2@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Fri Jul 10 08:52:21 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:52:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a511535.bc28b987.90005.f340@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM % nits, thank you for the fix!

https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Fri Jul 10 08:52:21 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:52:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a511535.6c5720b3.128ed.e6e9@mx.google.com>


================

----------------
banach-space wrote:

Could you try simplifying this test? For example:
* Make it require at most 2 indexing maps.
* Instead of `tensor.empty` (i.e. extra Op), pass `%0` as an input argument.
* Perhaps reduce to `linalg.generic` on scalars?

https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Fri Jul 10 08:52:22 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 10 Jul 2026 08:52:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a511536.58bdd12c.19c81.cef6@mx.google.com>


================

----------------
banach-space wrote:

Missing EOF

https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Fri Jul 10 09:30:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 09:30:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve shared data operations
 during if-clause lowering (PR #208771)
Message-ID: <llvm.org/llvm/llvm-project/pull/208771@github.com>


https://github.com/khaki3 created https://github.com/llvm/llvm-project/pull/208771

Example:
```fortran
!$acc kernels present(grid, c) if(offload_on)
do nc = 1, grid%rc%ncol
  c(1,nc) = 0
end do
c(1,1) = 0
!$acc end kernels
```

In this code, kernels restructuring can create compute constructs that share data-entry operations with an enclosing data region. If-clause lowering could rewrite or erase these shared operations, producing invalid IR or duplicate erasure.

Fix: Preserve externally owned data operations, deduplicate cleanup, and retain duplicate operand ordering while cloning local operations once.

>From 388f51b2e028d5947395f660c126542d9612de92 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Thu, 9 Jul 2026 12:19:46 -0700
Subject: [PATCH 1/2] [mlir][acc] Fix ACCIfClauseLowering for data ops shared
 with an enclosing construct

When a compute construct with an `if` clause shares its data-entry op
(e.g. acc.present) with an enclosing acc.data - as produced when an
earlier pass wraps split kernels in a shared acc.data region - the pass
corrupted the IR. It globally replaced the acc var with the host var
(rewriting the enclosing acc.data operand to a non data-entry value,
failing verification) and unconditionally erased the shared entry/exit
ops (double free when multiple constructs shared them).

Redirect only host (else) region uses to the host value, erase data
entry/private/reduction ops only once they have no remaining users, and
treat data ops owned by an enclosing construct as external: reference
them directly rather than cloning, and leave their exit ops intact.
---
 .../Transforms/ACCIfClauseLowering.cpp        | 80 ++++++++++++++-----
 .../OpenACC/acc-if-clause-lowering.mlir       | 36 +++++++++
 2 files changed, 96 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
index 71df75958a134..8fda4aa3313b9 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
@@ -66,6 +66,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/Support/Debug.h"
 
 namespace mlir {
@@ -92,8 +93,9 @@ class ACCIfClauseLowering
   void convertHostRegion(Operation *computeOp, Region &region);
 
   template <typename OpTy>
-  void lowerIfClauseForComputeConstruct(OpTy computeConstructOp,
-                                        SmallVector<Operation *> &eraseOps);
+  void lowerIfClauseForComputeConstruct(
+      OpTy computeConstructOp, SmallVector<Operation *> &eraseOps,
+      llvm::SetVector<Operation *> &condEraseOps);
 
 public:
   void runOnOperation() override;
@@ -122,7 +124,8 @@ void ACCIfClauseLowering::convertHostRegion(Operation *computeOp,
 // constructs
 template <typename OpTy>
 void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
-    OpTy computeConstructOp, SmallVector<Operation *> &eraseOps) {
+    OpTy computeConstructOp, SmallVector<Operation *> &eraseOps,
+    llvm::SetVector<Operation *> &condEraseOps) {
   Value ifCond = computeConstructOp.getIfCond();
   if (!ifCond)
     return;
@@ -141,18 +144,33 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
   SmallVector<Operation *> privateOps;
   SmallVector<Operation *> reductionOps;
 
+  // A data entry op is "externally owned" when it is also used by a construct
+  // that encloses this one (e.g. an acc.data wrapping this compute construct
+  // that shares the same data entry op). Such ops - and their data exit ops -
+  // belong to the enclosing construct and must not be cloned or erased here.
+  auto isExternallyOwned = [&](Operation *dataOp) {
+    for (Operation *user : dataOp->getUsers())
+      for (Operation *anc = computeConstructOp->getParentOp(); anc;
+           anc = anc->getParentOp())
+        if (anc == user)
+          return true;
+    return false;
+  };
+
   // Collect data entry operations
   for (Value operand : computeConstructOp.getDataClauseOperands())
     if (Operation *defOp = operand.getDefiningOp())
       if (isa<ACC_DATA_ENTRY_OPS>(defOp))
         dataEntryOps.push_back(defOp);
 
-  // Find corresponding exit operations for each entry operation.
+  // Find corresponding exit operations for each local entry operation. Exit ops
+  // of externally owned entry ops belong to the enclosing construct.
   // Iterate backwards through entry ops since exit ops appear in reverse order.
   for (Operation *dataEntryOp : llvm::reverse(dataEntryOps))
-    for (Operation *user : dataEntryOp->getUsers())
-      if (isa<ACC_DATA_EXIT_OPS>(user))
-        dataExitOps.push_back(user);
+    if (!isExternallyOwned(dataEntryOp))
+      for (Operation *user : dataEntryOp->getUsers())
+        if (isa<ACC_DATA_EXIT_OPS>(user))
+          dataExitOps.push_back(user);
 
   // Collect firstprivate, private, and reduction operations
   auto collectOps = [&](SmallVector<Operation *> &ops, OperandRange operands) {
@@ -191,7 +209,17 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
       deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
     }
   };
-  cloneAndMapOps(dataEntryOps, deviceDataOperands);
+  // Local data entry ops are cloned into the device path; externally owned ones
+  // (mapped by an enclosing construct such as acc.data) are referenced directly.
+  for (Operation *op : dataEntryOps) {
+    if (isExternallyOwned(op)) {
+      deviceDataOperands.push_back(op->getResult(0));
+      continue;
+    }
+    Operation *clonedOp = rewriter.clone(*op, deviceMapping);
+    deviceDataOperands.push_back(clonedOp->getResult(0));
+    deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
+  }
   cloneAndMapOps(firstprivateOps, firstprivateOperands);
   cloneAndMapOps(privateOps, privateOperands);
   cloneAndMapOps(reductionOps, reductionOperands);
@@ -247,18 +275,23 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
   for (Operation *dataOp : dataExitOps)
     eraseOps.push_back(dataOp);
 
-  // The new host code may contain uses of the acc variables. Replace them by
-  // the host values.
-  auto replaceAndEraseOps = [&](SmallVector<Operation *> &ops) {
+  // The host (else) region may contain uses of the acc variables. Redirect
+  // only those host-side uses to the host values. Other uses (e.g. an enclosing
+  // acc.data that shares the same data entry op) must be preserved, so these ops
+  // are only erased once they have no remaining users.
+  Region &elseRegion = ifOp.getElseRegion();
+  auto replaceHostUsesAndScheduleErase = [&](SmallVector<Operation *> &ops) {
     for (Operation *op : ops) {
-      getAccVar(op).replaceAllUsesWith(getVar(op));
-      eraseOps.push_back(op);
+      getAccVar(op).replaceUsesWithIf(getVar(op), [&](OpOperand &use) {
+        return elseRegion.isAncestor(use.getOwner()->getParentRegion());
+      });
+      condEraseOps.insert(op);
     }
   };
-  replaceAndEraseOps(dataEntryOps);
-  replaceAndEraseOps(firstprivateOps);
-  replaceAndEraseOps(privateOps);
-  replaceAndEraseOps(reductionOps);
+  replaceHostUsesAndScheduleErase(dataEntryOps);
+  replaceHostUsesAndScheduleErase(firstprivateOps);
+  replaceHostUsesAndScheduleErase(privateOps);
+  replaceHostUsesAndScheduleErase(reductionOps);
 }
 
 void ACCIfClauseLowering::runOnOperation() {
@@ -266,17 +299,24 @@ void ACCIfClauseLowering::runOnOperation() {
   accSupport = &getAnalysis<OpenACCSupport>();
 
   SmallVector<Operation *> eraseOps;
+  llvm::SetVector<Operation *> condEraseOps;
   funcOp.walk([&](Operation *op) {
     if (auto parallelOp = dyn_cast<acc::ParallelOp>(op))
-      lowerIfClauseForComputeConstruct(parallelOp, eraseOps);
+      lowerIfClauseForComputeConstruct(parallelOp, eraseOps, condEraseOps);
     else if (auto kernelsOp = dyn_cast<acc::KernelsOp>(op))
-      lowerIfClauseForComputeConstruct(kernelsOp, eraseOps);
+      lowerIfClauseForComputeConstruct(kernelsOp, eraseOps, condEraseOps);
     else if (auto serialOp = dyn_cast<acc::SerialOp>(op))
-      lowerIfClauseForComputeConstruct(serialOp, eraseOps);
+      lowerIfClauseForComputeConstruct(serialOp, eraseOps, condEraseOps);
   });
 
   for (Operation *op : eraseOps)
     op->erase();
+  // Data entry/private/reduction ops may be shared with other constructs (e.g.
+  // an enclosing acc.data). Erase them only after their users are gone, in
+  // reverse insertion order so consumers are removed before producers.
+  for (Operation *op : llvm::reverse(condEraseOps))
+    if (op->use_empty())
+      op->erase();
 }
 
 } // namespace
diff --git a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
index af4cc72a42645..933b209052f40 100644
--- a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
@@ -406,3 +406,39 @@ func.func @test_parallel_if_atomic_capture(%x: memref<i32>, %v: memref<i32>, %co
   }
   return
 }
+
+// -----
+
+// A data entry op (acc.present) shared by an enclosing acc.data and a nested
+// acc.kernels that has an if clause. Lowering the kernels' if clause must not
+// erase or rewrite the present op that acc.data still uses (otherwise acc.data
+// ends up with a non data-entry op as its data operand and fails to verify).
+// CHECK-LABEL: func.func @test_kernels_if_shared_present
+func.func @test_kernels_if_shared_present(%arg0: memref<10xi32>, %cond: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c10 = arith.constant 10 : index
+  // CHECK: %[[PRESENT:.*]] = acc.present varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  %present = acc.present varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // The enclosing acc.data must keep referencing the original present op.
+  // CHECK: acc.data dataOperands(%[[PRESENT]] : memref<10xi32>) {
+  acc.data dataOperands(%present : memref<10xi32>) {
+    // CHECK: scf.if %{{.*}} {
+    // The externally owned present op is referenced directly, not cloned/erased.
+    // CHECK-NOT: acc.present
+    // CHECK:   acc.kernels dataOperands(%[[PRESENT]] : memref<10xi32>)
+    // CHECK: } else {
+    // Host path uses the original variable, not the present result.
+    // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+    // CHECK: }
+    acc.kernels dataOperands(%present : memref<10xi32>) if(%cond) {
+      scf.for %i = %c0 to %c10 step %c1 {
+        memref.store %c0_i32, %present[%i] : memref<10xi32>
+      }
+      acc.terminator
+    }
+    acc.terminator
+  }
+  return
+}

>From c76753b01b7573f5a83bc4ab389083e1e5e4b0d9 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Fri, 10 Jul 2026 09:25:34 -0700
Subject: [PATCH 2/2] [mlir][acc] Handle data ops shared across compute
 constructs

Treat entry operations with sibling or later users as owned by the surrounding scope, and deduplicate entry, exit, and deferred erasure lists. Preserve duplicate operand multiplicity while cloning each local operation once.
---
 .../Transforms/ACCIfClauseLowering.cpp        | 79 ++++++++++---------
 .../OpenACC/acc-if-clause-lowering.mlir       | 63 +++++++++++++++
 2 files changed, 106 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
index 8fda4aa3313b9..8adf441f7bd44 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
@@ -93,9 +93,10 @@ class ACCIfClauseLowering
   void convertHostRegion(Operation *computeOp, Region &region);
 
   template <typename OpTy>
-  void lowerIfClauseForComputeConstruct(
-      OpTy computeConstructOp, SmallVector<Operation *> &eraseOps,
-      llvm::SetVector<Operation *> &condEraseOps);
+  void
+  lowerIfClauseForComputeConstruct(OpTy computeConstructOp,
+                                   llvm::SetVector<Operation *> &eraseOps,
+                                   llvm::SetVector<Operation *> &condEraseOps);
 
 public:
   void runOnOperation() override;
@@ -124,7 +125,7 @@ void ACCIfClauseLowering::convertHostRegion(Operation *computeOp,
 // constructs
 template <typename OpTy>
 void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
-    OpTy computeConstructOp, SmallVector<Operation *> &eraseOps,
+    OpTy computeConstructOp, llvm::SetVector<Operation *> &eraseOps,
     llvm::SetVector<Operation *> &condEraseOps) {
   Value ifCond = computeConstructOp.getIfCond();
   if (!ifCond)
@@ -138,22 +139,19 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
 
   // Collect data clause operations that need to be recreated in the if
   // condition
-  SmallVector<Operation *> dataEntryOps;
-  SmallVector<Operation *> dataExitOps;
+  llvm::SetVector<Operation *> dataEntryOps;
+  llvm::SetVector<Operation *> dataExitOps;
   SmallVector<Operation *> firstprivateOps;
   SmallVector<Operation *> privateOps;
   SmallVector<Operation *> reductionOps;
 
-  // A data entry op is "externally owned" when it is also used by a construct
-  // that encloses this one (e.g. an acc.data wrapping this compute construct
-  // that shares the same data entry op). Such ops - and their data exit ops -
-  // belong to the enclosing construct and must not be cloned or erased here.
+  // Entries used outside this construct belong to the surrounding scope.
+  // Their entry and exit operations must remain outside the conditional.
   auto isExternallyOwned = [&](Operation *dataOp) {
     for (Operation *user : dataOp->getUsers())
-      for (Operation *anc = computeConstructOp->getParentOp(); anc;
-           anc = anc->getParentOp())
-        if (anc == user)
-          return true;
+      if (!isa<ACC_DATA_EXIT_OPS>(user) && user != computeConstructOp &&
+          !computeConstructOp->isAncestor(user))
+        return true;
     return false;
   };
 
@@ -161,7 +159,7 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
   for (Value operand : computeConstructOp.getDataClauseOperands())
     if (Operation *defOp = operand.getDefiningOp())
       if (isa<ACC_DATA_ENTRY_OPS>(defOp))
-        dataEntryOps.push_back(defOp);
+        dataEntryOps.insert(defOp);
 
   // Find corresponding exit operations for each local entry operation. Exit ops
   // of externally owned entry ops belong to the enclosing construct.
@@ -170,7 +168,7 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
     if (!isExternallyOwned(dataEntryOp))
       for (Operation *user : dataEntryOp->getUsers())
         if (isa<ACC_DATA_EXIT_OPS>(user))
-          dataExitOps.push_back(user);
+          dataExitOps.insert(user);
 
   // Collect firstprivate, private, and reduction operations
   auto collectOps = [&](SmallVector<Operation *> &ops, OperandRange operands) {
@@ -209,17 +207,18 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
       deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
     }
   };
-  // Local data entry ops are cloned into the device path; externally owned ones
-  // (mapped by an enclosing construct such as acc.data) are referenced directly.
+  // Clone each local data entry op once. Externally owned ops (mapped by a
+  // surrounding scope) are referenced directly.
   for (Operation *op : dataEntryOps) {
-    if (isExternallyOwned(op)) {
-      deviceDataOperands.push_back(op->getResult(0));
+    if (isExternallyOwned(op))
       continue;
-    }
     Operation *clonedOp = rewriter.clone(*op, deviceMapping);
-    deviceDataOperands.push_back(clonedOp->getResult(0));
     deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
   }
+  // Preserve the original operand order and multiplicity while using the
+  // cloned value for local entries and the original value for external ones.
+  for (Value operand : computeConstructOp.getDataClauseOperands())
+    deviceDataOperands.push_back(deviceMapping.lookupOrDefault(operand));
   cloneAndMapOps(firstprivateOps, firstprivateOperands);
   cloneAndMapOps(privateOps, privateOperands);
   cloneAndMapOps(reductionOps, reductionOperands);
@@ -268,19 +267,17 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
   }
 
   // The original op is now empty and can be erased
-  eraseOps.push_back(computeConstructOp);
+  eraseOps.insert(computeConstructOp);
 
   // TODO: Can probably 'move' the data ops instead of cloning them
   // which would eliminate need to explicitly erase
   for (Operation *dataOp : dataExitOps)
-    eraseOps.push_back(dataOp);
+    eraseOps.insert(dataOp);
 
-  // The host (else) region may contain uses of the acc variables. Redirect
-  // only those host-side uses to the host values. Other uses (e.g. an enclosing
-  // acc.data that shares the same data entry op) must be preserved, so these ops
-  // are only erased once they have no remaining users.
+  // Redirect host-side uses while preserving uses outside this construct.
+  // Shared operations are erased only after their users are gone.
   Region &elseRegion = ifOp.getElseRegion();
-  auto replaceHostUsesAndScheduleErase = [&](SmallVector<Operation *> &ops) {
+  auto replaceHostUsesAndScheduleErase = [&](auto &ops) {
     for (Operation *op : ops) {
       getAccVar(op).replaceUsesWithIf(getVar(op), [&](OpOperand &use) {
         return elseRegion.isAncestor(use.getOwner()->getParentRegion());
@@ -298,7 +295,7 @@ void ACCIfClauseLowering::runOnOperation() {
   func::FuncOp funcOp = getOperation();
   accSupport = &getAnalysis<OpenACCSupport>();
 
-  SmallVector<Operation *> eraseOps;
+  llvm::SetVector<Operation *> eraseOps;
   llvm::SetVector<Operation *> condEraseOps;
   funcOp.walk([&](Operation *op) {
     if (auto parallelOp = dyn_cast<acc::ParallelOp>(op))
@@ -309,14 +306,24 @@ void ACCIfClauseLowering::runOnOperation() {
       lowerIfClauseForComputeConstruct(serialOp, eraseOps, condEraseOps);
   });
 
-  for (Operation *op : eraseOps)
+  for (Operation *op : llvm::reverse(eraseOps))
     op->erase();
-  // Data entry/private/reduction ops may be shared with other constructs (e.g.
-  // an enclosing acc.data). Erase them only after their users are gone, in
-  // reverse insertion order so consumers are removed before producers.
-  for (Operation *op : llvm::reverse(condEraseOps))
-    if (op->use_empty())
+  // Shared entry/private/reduction ops can become dead in stages.
+  // Revisit deferred producers after their consumers are erased.
+  SmallVector<Operation *> pendingEraseOps(condEraseOps.begin(),
+                                           condEraseOps.end());
+  bool erased;
+  do {
+    erased = false;
+    for (size_t i = pendingEraseOps.size(); i > 0; --i) {
+      Operation *op = pendingEraseOps[i - 1];
+      if (!op->use_empty())
+        continue;
+      pendingEraseOps.erase(pendingEraseOps.begin() + i - 1);
       op->erase();
+      erased = true;
+    }
+  } while (erased);
 }
 
 } // namespace
diff --git a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
index 933b209052f40..9cc0df1dfb107 100644
--- a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
@@ -442,3 +442,66 @@ func.func @test_kernels_if_shared_present(%arg0: memref<10xi32>, %cond: i1) {
   }
   return
 }
+
+// -----
+
+// A data entry and exit shared by sibling compute constructs belong to their
+// surrounding scope. Neither conditional lowering may clone or erase them.
+// CHECK-LABEL: func.func @test_sibling_compute_shared_data
+func.func @test_sibling_compute_shared_data(%arg0: memref<10xi32>, %cond0: i1, %cond1: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[COPYIN:.*]] = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  %copyin = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   acc.parallel dataOperands(%[[COPYIN]] : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  acc.parallel dataOperands(%copyin : memref<10xi32>) if(%cond0) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   acc.serial dataOperands(%[[COPYIN]] : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  acc.serial dataOperands(%copyin : memref<10xi32>) if(%cond1) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  // CHECK: acc.copyout accPtr(%[[COPYIN]] : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  acc.copyout accPtr(%copyin : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  return
+}
+
+// -----
+
+// Duplicate data operands must preserve their multiplicity without cloning or
+// erasing the same entry/exit operation more than once.
+// CHECK-LABEL: func.func @test_duplicate_data_operand
+func.func @test_duplicate_data_operand(%arg0: memref<10xi32>, %cond: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  %copyin = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK-NOT: acc.copyin
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   %[[COPYIN:.*]] = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK:   acc.parallel dataOperands(%[[COPYIN]], %[[COPYIN]] : memref<10xi32>, memref<10xi32>)
+  // CHECK:   acc.copyout accPtr(%[[COPYIN]] : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  // CHECK-NOT: acc.copyout
+  // CHECK: return
+  acc.parallel dataOperands(%copyin, %copyin : memref<10xi32>, memref<10xi32>) if(%cond) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  acc.copyout accPtr(%copyin : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  return
+}


From llvmlistbot at llvm.org  Fri Jul 10 09:30:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 09:30:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve shared data operations
 during if-clause lowering (PR #208771)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208771@github.com>
Message-ID: <6a511e3a.30759430.24d50c.9945@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-openacc

Author: Matsu (khaki3)

<details>
<summary>Changes</summary>

Example:
```fortran
!$acc kernels present(grid, c) if(offload_on)
do nc = 1, grid%rc%ncol
  c(1,nc) = 0
end do
c(1,1) = 0
!$acc end kernels
```

In this code, kernels restructuring can create compute constructs that share data-entry operations with an enclosing data region. If-clause lowering could rewrite or erase these shared operations, producing invalid IR or duplicate erasure.

Fix: Preserve externally owned data operations, deduplicate cleanup, and retain duplicate operand ordering while cloning local operations once.

---
Full diff: https://github.com/llvm/llvm-project/pull/208771.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp (+74-27) 
- (modified) mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir (+99) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
index 71df75958a134..8adf441f7bd44 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCIfClauseLowering.cpp
@@ -66,6 +66,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SetVector.h"
 #include "llvm/Support/Debug.h"
 
 namespace mlir {
@@ -92,8 +93,10 @@ class ACCIfClauseLowering
   void convertHostRegion(Operation *computeOp, Region &region);
 
   template <typename OpTy>
-  void lowerIfClauseForComputeConstruct(OpTy computeConstructOp,
-                                        SmallVector<Operation *> &eraseOps);
+  void
+  lowerIfClauseForComputeConstruct(OpTy computeConstructOp,
+                                   llvm::SetVector<Operation *> &eraseOps,
+                                   llvm::SetVector<Operation *> &condEraseOps);
 
 public:
   void runOnOperation() override;
@@ -122,7 +125,8 @@ void ACCIfClauseLowering::convertHostRegion(Operation *computeOp,
 // constructs
 template <typename OpTy>
 void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
-    OpTy computeConstructOp, SmallVector<Operation *> &eraseOps) {
+    OpTy computeConstructOp, llvm::SetVector<Operation *> &eraseOps,
+    llvm::SetVector<Operation *> &condEraseOps) {
   Value ifCond = computeConstructOp.getIfCond();
   if (!ifCond)
     return;
@@ -135,24 +139,36 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
 
   // Collect data clause operations that need to be recreated in the if
   // condition
-  SmallVector<Operation *> dataEntryOps;
-  SmallVector<Operation *> dataExitOps;
+  llvm::SetVector<Operation *> dataEntryOps;
+  llvm::SetVector<Operation *> dataExitOps;
   SmallVector<Operation *> firstprivateOps;
   SmallVector<Operation *> privateOps;
   SmallVector<Operation *> reductionOps;
 
+  // Entries used outside this construct belong to the surrounding scope.
+  // Their entry and exit operations must remain outside the conditional.
+  auto isExternallyOwned = [&](Operation *dataOp) {
+    for (Operation *user : dataOp->getUsers())
+      if (!isa<ACC_DATA_EXIT_OPS>(user) && user != computeConstructOp &&
+          !computeConstructOp->isAncestor(user))
+        return true;
+    return false;
+  };
+
   // Collect data entry operations
   for (Value operand : computeConstructOp.getDataClauseOperands())
     if (Operation *defOp = operand.getDefiningOp())
       if (isa<ACC_DATA_ENTRY_OPS>(defOp))
-        dataEntryOps.push_back(defOp);
+        dataEntryOps.insert(defOp);
 
-  // Find corresponding exit operations for each entry operation.
+  // Find corresponding exit operations for each local entry operation. Exit ops
+  // of externally owned entry ops belong to the enclosing construct.
   // Iterate backwards through entry ops since exit ops appear in reverse order.
   for (Operation *dataEntryOp : llvm::reverse(dataEntryOps))
-    for (Operation *user : dataEntryOp->getUsers())
-      if (isa<ACC_DATA_EXIT_OPS>(user))
-        dataExitOps.push_back(user);
+    if (!isExternallyOwned(dataEntryOp))
+      for (Operation *user : dataEntryOp->getUsers())
+        if (isa<ACC_DATA_EXIT_OPS>(user))
+          dataExitOps.insert(user);
 
   // Collect firstprivate, private, and reduction operations
   auto collectOps = [&](SmallVector<Operation *> &ops, OperandRange operands) {
@@ -191,7 +207,18 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
       deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
     }
   };
-  cloneAndMapOps(dataEntryOps, deviceDataOperands);
+  // Clone each local data entry op once. Externally owned ops (mapped by a
+  // surrounding scope) are referenced directly.
+  for (Operation *op : dataEntryOps) {
+    if (isExternallyOwned(op))
+      continue;
+    Operation *clonedOp = rewriter.clone(*op, deviceMapping);
+    deviceMapping.map(op->getResult(0), clonedOp->getResult(0));
+  }
+  // Preserve the original operand order and multiplicity while using the
+  // cloned value for local entries and the original value for external ones.
+  for (Value operand : computeConstructOp.getDataClauseOperands())
+    deviceDataOperands.push_back(deviceMapping.lookupOrDefault(operand));
   cloneAndMapOps(firstprivateOps, firstprivateOperands);
   cloneAndMapOps(privateOps, privateOperands);
   cloneAndMapOps(reductionOps, reductionOperands);
@@ -240,43 +267,63 @@ void ACCIfClauseLowering::lowerIfClauseForComputeConstruct(
   }
 
   // The original op is now empty and can be erased
-  eraseOps.push_back(computeConstructOp);
+  eraseOps.insert(computeConstructOp);
 
   // TODO: Can probably 'move' the data ops instead of cloning them
   // which would eliminate need to explicitly erase
   for (Operation *dataOp : dataExitOps)
-    eraseOps.push_back(dataOp);
+    eraseOps.insert(dataOp);
 
-  // The new host code may contain uses of the acc variables. Replace them by
-  // the host values.
-  auto replaceAndEraseOps = [&](SmallVector<Operation *> &ops) {
+  // Redirect host-side uses while preserving uses outside this construct.
+  // Shared operations are erased only after their users are gone.
+  Region &elseRegion = ifOp.getElseRegion();
+  auto replaceHostUsesAndScheduleErase = [&](auto &ops) {
     for (Operation *op : ops) {
-      getAccVar(op).replaceAllUsesWith(getVar(op));
-      eraseOps.push_back(op);
+      getAccVar(op).replaceUsesWithIf(getVar(op), [&](OpOperand &use) {
+        return elseRegion.isAncestor(use.getOwner()->getParentRegion());
+      });
+      condEraseOps.insert(op);
     }
   };
-  replaceAndEraseOps(dataEntryOps);
-  replaceAndEraseOps(firstprivateOps);
-  replaceAndEraseOps(privateOps);
-  replaceAndEraseOps(reductionOps);
+  replaceHostUsesAndScheduleErase(dataEntryOps);
+  replaceHostUsesAndScheduleErase(firstprivateOps);
+  replaceHostUsesAndScheduleErase(privateOps);
+  replaceHostUsesAndScheduleErase(reductionOps);
 }
 
 void ACCIfClauseLowering::runOnOperation() {
   func::FuncOp funcOp = getOperation();
   accSupport = &getAnalysis<OpenACCSupport>();
 
-  SmallVector<Operation *> eraseOps;
+  llvm::SetVector<Operation *> eraseOps;
+  llvm::SetVector<Operation *> condEraseOps;
   funcOp.walk([&](Operation *op) {
     if (auto parallelOp = dyn_cast<acc::ParallelOp>(op))
-      lowerIfClauseForComputeConstruct(parallelOp, eraseOps);
+      lowerIfClauseForComputeConstruct(parallelOp, eraseOps, condEraseOps);
     else if (auto kernelsOp = dyn_cast<acc::KernelsOp>(op))
-      lowerIfClauseForComputeConstruct(kernelsOp, eraseOps);
+      lowerIfClauseForComputeConstruct(kernelsOp, eraseOps, condEraseOps);
     else if (auto serialOp = dyn_cast<acc::SerialOp>(op))
-      lowerIfClauseForComputeConstruct(serialOp, eraseOps);
+      lowerIfClauseForComputeConstruct(serialOp, eraseOps, condEraseOps);
   });
 
-  for (Operation *op : eraseOps)
+  for (Operation *op : llvm::reverse(eraseOps))
     op->erase();
+  // Shared entry/private/reduction ops can become dead in stages.
+  // Revisit deferred producers after their consumers are erased.
+  SmallVector<Operation *> pendingEraseOps(condEraseOps.begin(),
+                                           condEraseOps.end());
+  bool erased;
+  do {
+    erased = false;
+    for (size_t i = pendingEraseOps.size(); i > 0; --i) {
+      Operation *op = pendingEraseOps[i - 1];
+      if (!op->use_empty())
+        continue;
+      pendingEraseOps.erase(pendingEraseOps.begin() + i - 1);
+      op->erase();
+      erased = true;
+    }
+  } while (erased);
 }
 
 } // namespace
diff --git a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
index af4cc72a42645..9cc0df1dfb107 100644
--- a/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-if-clause-lowering.mlir
@@ -406,3 +406,102 @@ func.func @test_parallel_if_atomic_capture(%x: memref<i32>, %v: memref<i32>, %co
   }
   return
 }
+
+// -----
+
+// A data entry op (acc.present) shared by an enclosing acc.data and a nested
+// acc.kernels that has an if clause. Lowering the kernels' if clause must not
+// erase or rewrite the present op that acc.data still uses (otherwise acc.data
+// ends up with a non data-entry op as its data operand and fails to verify).
+// CHECK-LABEL: func.func @test_kernels_if_shared_present
+func.func @test_kernels_if_shared_present(%arg0: memref<10xi32>, %cond: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %c10 = arith.constant 10 : index
+  // CHECK: %[[PRESENT:.*]] = acc.present varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  %present = acc.present varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // The enclosing acc.data must keep referencing the original present op.
+  // CHECK: acc.data dataOperands(%[[PRESENT]] : memref<10xi32>) {
+  acc.data dataOperands(%present : memref<10xi32>) {
+    // CHECK: scf.if %{{.*}} {
+    // The externally owned present op is referenced directly, not cloned/erased.
+    // CHECK-NOT: acc.present
+    // CHECK:   acc.kernels dataOperands(%[[PRESENT]] : memref<10xi32>)
+    // CHECK: } else {
+    // Host path uses the original variable, not the present result.
+    // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+    // CHECK: }
+    acc.kernels dataOperands(%present : memref<10xi32>) if(%cond) {
+      scf.for %i = %c0 to %c10 step %c1 {
+        memref.store %c0_i32, %present[%i] : memref<10xi32>
+      }
+      acc.terminator
+    }
+    acc.terminator
+  }
+  return
+}
+
+// -----
+
+// A data entry and exit shared by sibling compute constructs belong to their
+// surrounding scope. Neither conditional lowering may clone or erase them.
+// CHECK-LABEL: func.func @test_sibling_compute_shared_data
+func.func @test_sibling_compute_shared_data(%arg0: memref<10xi32>, %cond0: i1, %cond1: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[COPYIN:.*]] = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  %copyin = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   acc.parallel dataOperands(%[[COPYIN]] : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  acc.parallel dataOperands(%copyin : memref<10xi32>) if(%cond0) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   acc.serial dataOperands(%[[COPYIN]] : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  acc.serial dataOperands(%copyin : memref<10xi32>) if(%cond1) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  // CHECK: acc.copyout accPtr(%[[COPYIN]] : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  acc.copyout accPtr(%copyin : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  return
+}
+
+// -----
+
+// Duplicate data operands must preserve their multiplicity without cloning or
+// erasing the same entry/exit operation more than once.
+// CHECK-LABEL: func.func @test_duplicate_data_operand
+func.func @test_duplicate_data_operand(%arg0: memref<10xi32>, %cond: i1) {
+  %c0_i32 = arith.constant 0 : i32
+  %c0 = arith.constant 0 : index
+  %copyin = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK-NOT: acc.copyin
+  // CHECK: scf.if %{{.*}} {
+  // CHECK:   %[[COPYIN:.*]] = acc.copyin varPtr(%arg0 : memref<10xi32>) -> memref<10xi32>
+  // CHECK:   acc.parallel dataOperands(%[[COPYIN]], %[[COPYIN]] : memref<10xi32>, memref<10xi32>)
+  // CHECK:   acc.copyout accPtr(%[[COPYIN]] : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  // CHECK-NOT: acc.copyout
+  // CHECK: } else {
+  // CHECK:   memref.store %{{.*}}, %arg0[%{{.*}}] : memref<10xi32>
+  // CHECK: }
+  // CHECK-NOT: acc.copyout
+  // CHECK: return
+  acc.parallel dataOperands(%copyin, %copyin : memref<10xi32>, memref<10xi32>) if(%cond) {
+    memref.store %c0_i32, %copyin[%c0] : memref<10xi32>
+    acc.yield
+  }
+  acc.copyout accPtr(%copyin : memref<10xi32>) to varPtr(%arg0 : memref<10xi32>)
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208771

From llvmlistbot at llvm.org  Fri Jul 10 09:43:18 2026
From: llvmlistbot at llvm.org (Andy Kaylor)
Date: Fri, 10 Jul 2026 09:43:18 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [mlir] Diagnose missing call result
	arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a512126.92d2873c.258297.097b@mx.google.com>


================
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
----------------
andykaylor wrote:

Are you saying that in this case parseFunctionSignature reported an error but returned success()? This just looks like a workaround for a problem that should be addressed some other way.

https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Fri Jul 10 09:51:15 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 10 Jul 2026 09:51:15 -0700 (PDT)
Subject: [Mlir-commits] [clang] [compiler-rt] [flang] [llvm] [mlir]
 [DebugMetadata][DwarfDebug][CodeView] Support function-local static
 variables in lexical block scopes (6/7) (PR #187927)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/187927@github.com>
Message-ID: <6a512303.4a342cbc.ca844.bbc8@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `lldb-aarch64-windows` running on `linaro-armv8-windows-msvc-05` while building `clang,compiler-rt,flang,llvm,mlir` at step 6 "test".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/141/builds/20468

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 6 (test) failure: build (failure)
...
PASS: lldb-api :: lang/cpp/class-template-parameter-pack/TestClassTemplateParameterPack.py (870 of 2625)
PASS: lldb-api :: lang/cpp/class-template-parameter-pack/TestTemplatePackArgs.py (871 of 2625)
PASS: lldb-api :: lang/cpp/class-template-type-parameter-pack/TestClassTemplateTypeParameterPack.py (872 of 2625)
PASS: lldb-api :: lang/cpp/class_static/TestStaticVariables.py (873 of 2625)
PASS: lldb-api :: lang/cpp/class_types/TestClassTypes.py (874 of 2625)
PASS: lldb-api :: lang/cpp/class_types/TestClassTypesDisassembly.py (875 of 2625)
PASS: lldb-api :: lang/cpp/complete-type-check/TestCppIsTypeComplete.py (876 of 2625)
PASS: lldb-api :: lang/cpp/const_static_integral_member/TestConstStaticIntegralMember.py (877 of 2625)
PASS: lldb-api :: lang/cpp/const_static_integral_member_int128/TestConstStaticIntegralMemberInt128.py (878 of 2625)
TIMEOUT: lldb-api :: functionalities/breakpoint/two_hits_one_actual/TestTwoHitsOneActual.py (879 of 2625)
******************** TEST 'lldb-api :: functionalities/breakpoint/two_hits_one_actual/TestTwoHitsOneActual.py' FAILED ********************
Script:
--
C:/Users/tcwg/scoop/apps/python/current/python.exe C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/llvm-project/lldb\test\API\dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./lib --env LLVM_INCLUDE_DIR=C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/include --env LLVM_TOOLS_DIR=C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./bin --triple aarch64-pc-windows-msvc --build-dir C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/lldb-test-build --lldb-module-cache-dir C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/lldb-test-build/module-cache-lldb\lldb-api --clang-module-cache-dir C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/lldb-test-build/module-cache-clang\lldb-api --executable C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./bin/lldb.exe --lldb-python-dir C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build\Lib\site-packages --compiler C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./bin/clang.exe --dsymutil C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./bin/dsymutil.exe --make C:/Users/tcwg/scoop/shims/make.exe --llvm-tools-dir C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./bin --lldb-obj-root C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/tools/lldb --lldb-libs-dir C:/Users/tcwg/llvm-worker/lldb-aarch64-windows/build/./lib --cmake-build-type Release --skip-category=watchpoint --env LLDB_LAUNCH_FLAG_USE_PIPES=1 C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\llvm-project\lldb\test\API\functionalities\breakpoint\two_hits_one_actual -p TestTwoHitsOneActual.py
--
Exit Code: 15
Timeout: Reached timeout of 600 seconds


********************
PASS: lldb-api :: lang/cpp/constructors/TestCppConstructors.py (880 of 2625)
PASS: lldb-api :: lang/cpp/covariant-return-types/TestCovariantReturnTypes.py (881 of 2625)
PASS: lldb-api :: lang/cpp/const_this/TestConstThis.py (882 of 2625)
UNSUPPORTED: lldb-api :: lang/cpp/decl-from-submodule/TestDeclFromSubmodule.py (883 of 2625)
PASS: lldb-api :: lang/cpp/default-template-args/TestDefaultTemplateArgs.py (884 of 2625)
PASS: lldb-api :: lang/cpp/dereferencing_references/TestCPPDereferencingReferences.py (885 of 2625)
PASS: lldb-api :: lang/cpp/crtp/TestCppCRTP.py (886 of 2625)
PASS: lldb-api :: lang/cpp/dynamic-value-same-basename/TestDynamicValueSameBase.py (887 of 2625)
PASS: lldb-api :: lang/cpp/diamond/TestCppDiamond.py (888 of 2625)
PASS: lldb-api :: lang/cpp/dynamic-value/TestCppValueCast.py (889 of 2625)
PASS: lldb-api :: lang/cpp/elaborated-types/TestElaboratedTypes.py (890 of 2625)
XFAIL: lldb-api :: lang/cpp/dynamic-value/TestDynamicValue.py (891 of 2625)
PASS: lldb-api :: lang/cpp/enum_promotion/TestCPPEnumPromotion.py (892 of 2625)
UNSUPPORTED: lldb-api :: lang/cpp/enum_types/TestCPP11EnumTypes.py (893 of 2625)
UNSUPPORTED: lldb-api :: lang/cpp/expr-definition-in-dylib/TestExprDefinitionInDylib.py (894 of 2625)
XFAIL: lldb-api :: lang/cpp/exceptions/TestCPPExceptionBreakpoints.py (895 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/const_method/TestExprInConstMethod.py (896 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/const_volatile_method/TestExprInConstVolatileMethod.py (897 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/cv_qualified_objects/TestExprOnCVQualifiedObjects.py (898 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/fixit/TestExprInConstMethodWithFixit.py (899 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/non_const_method/TestExprInNonConstMethod.py (900 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/template_const_method/TestExprInTemplateConstMethod.py (901 of 2625)
PASS: lldb-api :: lang/cpp/expression-context-qualifiers/template_non_const_method/TestExprInTemplateNonConstMethod.py (902 of 2625)
PASS: lldb-api :: lang/cpp/extern_c/TestExternCSymbols.py (903 of 2625)
PASS: lldb-api :: lang/cpp/fixits/TestCppFixIts.py (904 of 2625)
PASS: lldb-api :: lang/cpp/floating-types-specialization/TestCppFloatingTypesSpecialization.py (905 of 2625)
PASS: lldb-api :: lang/cpp/forward-declared-template-specialization/TestCppForwardDeclaredTemplateSpecialization.py (906 of 2625)
PASS: lldb-api :: lang/cpp/fpnan/TestFPNaN.py (907 of 2625)
PASS: lldb-api :: lang/cpp/frame-var-anon-unions/TestFrameVariableAnonymousUnions.py (908 of 2625)

```

</details>

https://github.com/llvm/llvm-project/pull/187927

From llvmlistbot at llvm.org  Fri Jul 10 09:54:05 2026
From: llvmlistbot at llvm.org (Charitha Saumya)
Date: Fri, 10 Jul 2026 09:54:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Add support for 1D SLM case in
 vector-to-xepgu (PR #208276)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208276@github.com>
Message-ID: <6a5123ad.da8bafd3.14cd50.88dd@mx.google.com>


https://github.com/charithaintc updated https://github.com/llvm/llvm-project/pull/208276

>From ea777cd2577eb4be81ea0358db31474bce64afa6 Mon Sep 17 00:00:00 2001
From: Charitha Saumya <charitha.saumya.gusthinna.waduge at intel.com>
Date: Wed, 8 Jul 2026 17:48:00 +0000
Subject: [PATCH] add 1D SLM support in vector to xegpu

---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 12 ++---
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 44 +++++++++++--------
 .../transfer-write-to-xegpu.mlir              | 20 ++++++---
 3 files changed, 46 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 8dcc9771d8db4..ee9327caab002 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -560,10 +560,10 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isSharedMemory = xegpu::XeGPUDialect::isSharedMemory(readMemTy);
     // Handle the SLM case.
     if (isSharedMemory) {
-      // If the memref is SLM only support 2D case for now.
-      if (loadedVecTy.getRank() != 2)
+      // load_matrix supports 1D and 2D loads from SLM.
+      if (loadedVecTy.getRank() != 1 && loadedVecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            readOp, "Only 2D vector loads are supported for SLM");
+            readOp, "Only 1D and 2D vector loads are supported for SLM");
       AffineMap readMap = readOp.getPermutationMap();
       if (!readMap.isMinorIdentity())
         return rewriter.notifyMatchFailure(
@@ -700,10 +700,10 @@ struct TransferWriteLowering
     // For shared local memory (address space 3), use create_mem_desc +
     // store_matrix
     if (isSharedMemory) {
-      // Only support 2D case for now.
-      if (vecTy.getRank() != 2)
+      // store_matrix supports 1D and 2D stores to SLM.
+      if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
         return rewriter.notifyMatchFailure(
-            writeOp, "Only 2D vector stores are supported for SLM");
+            writeOp, "Only 1D and 2D vector stores are supported for SLM");
       // Create mem_desc for SLM
       auto memDescType =
           xegpu::MemDescType::get(rewriter.getContext(), writeMemTy.getShape(),
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index f64bcc27940ea..c6793a2e5ca26 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -598,7 +598,7 @@ gpu.func @load_2D_vector_addrspace3(%source: memref<16x32xf32, 3>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
+gpu.func @load_1D_vector_addrspace3(%source: memref<32xf32, 3>,
     %offset: index) -> vector<8xf32> {
   %c0 = arith.constant 0.0 : f32
   %0 = vector.transfer_read %source[%offset], %c0
@@ -606,11 +606,19 @@ gpu.func @load_1D_vector_addrspace3_unsupported(%source: memref<32xf32, 3>,
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-ND: vector.transfer_read
+// LOAD-ND-LABEL: @load_1D_vector_addrspace3
+// LOAD-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-ND-SAME: %[[OFFSET:.+]]: index
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-ND: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-ND: gpu.return %[[DATA]] : vector<8xf32>
 
-// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3_unsupported
-// LOAD-GATHER: vector.transfer_read
+// LOAD-GATHER-LABEL: @load_1D_vector_addrspace3
+// LOAD-GATHER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// LOAD-GATHER-SAME: %[[OFFSET:.+]]: index
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// LOAD-GATHER: %[[DATA:.+]] = xegpu.load_matrix %[[MEM_DESC]][%[[OFFSET]]] : !xegpu.mem_desc<32xf32>, index -> vector<8xf32>
+// LOAD-GATHER: gpu.return %[[DATA]] : vector<8xf32>
 
 }
 
@@ -642,10 +650,10 @@ gpu.func @load_2D_vector_alloca_promoted_to_slm(%offset: index)
 
 // -----
 // memref.alloca is unconditionally promoted to SLM (address space 3). A 1D
-// transfer_read on it cannot be lowered to xegpu.load_matrix (which requires
-// rank 2), so the transfer_read is left as-is on the SLM memref.
+// transfer_read on it is lowered to xegpu.load_matrix, which supports the
+// 1D SLM case.
 gpu.module @xevm_module {
-gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
+gpu.func @load_1D_vector_alloca_promoted_to_slm(%offset: index)
     -> vector<8xf32> {
   %buf = memref.alloca() : memref<16xf32>
   %c0 = arith.constant 0.0 : f32
@@ -654,17 +662,15 @@ gpu.func @load_1D_vector_alloca_promoted_unsupported(%offset: index)
   gpu.return %0 : vector<8xf32>
 }
 
-// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-ND: memref.alloca() : memref<16xf32, 3>
-// LOAD-ND: vector.transfer_read
-// LOAD-ND-NOT: xegpu.create_mem_desc
-// LOAD-ND-NOT: xegpu.load_matrix
+// LOAD-ND-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-ND: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-ND: xegpu.load_matrix %[[MEM_DESC]]
 
-// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_unsupported
-// LOAD-GATHER: memref.alloca() : memref<16xf32, 3>
-// LOAD-GATHER: vector.transfer_read
-// LOAD-GATHER-NOT: xegpu.create_mem_desc
-// LOAD-GATHER-NOT: xegpu.load_matrix
+// LOAD-GATHER-LABEL: @load_1D_vector_alloca_promoted_to_slm
+// LOAD-GATHER: %[[BUF:.+]] = memref.alloca() : memref<16xf32, 3>
+// LOAD-GATHER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[BUF]] : memref<16xf32, 3> -> !xegpu.mem_desc<16xf32>
+// LOAD-GATHER: xegpu.load_matrix %[[MEM_DESC]]
 
 }
 
@@ -679,4 +685,4 @@ gpu.func @load_0D_memref_unsupported(%source: memref<f16>) -> vector<f16> {
 // CHECK-LABEL: @load_0D_memref_unsupported
 // CHECK: vector.transfer_read
 
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index a7cd3d7652d85..27a83922fab37 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -397,7 +397,7 @@ gpu.func @store_2D_vector_addrspace3(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
+gpu.func @store_1D_vector_addrspace3(%vec: vector<8xf32>,
     %source: memref<32xf32, 3>, %offset: index) {
   vector.transfer_write %vec, %source[%offset]
     {in_bounds = [true]}
@@ -405,10 +405,20 @@ gpu.func @store_1D_vector_addrspace3_unsupported(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-ND: vector.transfer_write
+// STORE-ND-LABEL: @store_1D_vector_addrspace3
+// STORE-ND-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-ND-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-ND-SAME: %[[OFFSET:.+]]: index
+// STORE-ND: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-ND: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-ND: gpu.return
 
-// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3_unsupported
-// STORE-SCATTER: vector.transfer_write
+// STORE-SCATTER-LABEL: @store_1D_vector_addrspace3
+// STORE-SCATTER-SAME: %[[VEC:.+]]: vector<8xf32>
+// STORE-SCATTER-SAME: %[[SOURCE:.+]]: memref<32xf32, 3>
+// STORE-SCATTER-SAME: %[[OFFSET:.+]]: index
+// STORE-SCATTER: %[[MEM_DESC:.+]] = xegpu.create_mem_desc %[[SOURCE]] : memref<32xf32, 3> -> !xegpu.mem_desc<32xf32>
+// STORE-SCATTER: xegpu.store_matrix %[[VEC]], %[[MEM_DESC]][%[[OFFSET]]] : vector<8xf32>, !xegpu.mem_desc<32xf32>, index
+// STORE-SCATTER: gpu.return
 
 }


From llvmlistbot at llvm.org  Fri Jul 10 11:55:14 2026
From: llvmlistbot at llvm.org (Christopher Bate)
Date: Fri, 10 Jul 2026 11:55:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Normalize path separator in
	MLIRConfig.cmake (PR #208659)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208659@github.com>
Message-ID: <6a514012.a87eb5dc.7d817.e78f@mx.google.com>


https://github.com/christopherbate approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/208659

From llvmlistbot at llvm.org  Fri Jul 10 11:56:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 11:56:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix acc.present in acc.data
	translation (PR #208806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208806@github.com>
Message-ID: <6a51406e.af6ba3ed.36d9db.b5d3@mx.google.com>


https://github.com/StarryCSF updated https://github.com/llvm/llvm-project/pull/208806

>From c7440ed854e41f87c622af83950f474a18d4b56d Mon Sep 17 00:00:00 2001
From: "ZhiQiang.Fan" <571253675 at QQ.com>
Date: Sat, 11 Jul 2026 02:01:34 +0800
Subject: [PATCH 1/2] [flang][OpenACC] Fix acc.present in acc.data translation

---
 .../OpenACC/OpenACCToLLVMIRTranslation.cpp       |  4 ++--
 mlir/test/Target/LLVMIR/openacc-llvm.mlir        | 16 ++++++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
index 9bb6939043621..f198f114464a1 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
@@ -317,7 +317,7 @@ static LogicalResult convertDataOp(acc::DataOp &op,
       create.push_back(createOp.getVarPtr());
     } else if (auto presentOp = mlir::dyn_cast_or_null<acc::PresentOp>(
                    dataOp.getDefiningOp())) {
-      present.push_back(createOp.getVarPtr());
+      present.push_back(presentOp.getVarPtr());
     }
   }
 
@@ -517,7 +517,7 @@ LogicalResult OpenACCDialectLLVMIRTranslationInterface::convertOperation(
         return success();
       })
       .Case<acc::CreateOp, acc::CopyinOp, acc::CopyoutOp, acc::DeleteOp,
-            acc::UpdateDeviceOp, acc::GetDevicePtrOp>([](auto op) {
+            acc::UpdateDeviceOp, acc::GetDevicePtrOp, acc::PresentOp>([](auto op) {
         // NOP
         return success();
       })
diff --git a/mlir/test/Target/LLVMIR/openacc-llvm.mlir b/mlir/test/Target/LLVMIR/openacc-llvm.mlir
index c4b0b6a424417..f9234c1208c2f 100644
--- a/mlir/test/Target/LLVMIR/openacc-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openacc-llvm.mlir
@@ -189,3 +189,19 @@ llvm.func @testdataop(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: !llvm.ptr) {
 
 // CHECK: declare void @__tgt_target_data_begin_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr)
 // CHECK: declare void @__tgt_target_data_end_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr)
+
+// -----
+
+// Test acc.present in acc.data
+llvm.func @test_present_in_data(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
+  %0 = acc.copyin varPtr(%arg0 : !llvm.ptr) varType(f32) -> !llvm.ptr
+  %1 = acc.present varPtr(%arg1 : !llvm.ptr) varType(f32) -> !llvm.ptr
+  acc.data dataOperands(%0, %1 : !llvm.ptr, !llvm.ptr) {
+    acc.terminator
+  }
+  llvm.return
+}
+
+// Verify present flag: 12288 = 0x3000 = kHoldFlag + kPresentFlag
+// CHECK: constant [{{[0-9]*}} x i64] [i64 8193, i64 12288]
+// CHECK: call void @__tgt_target_data_begin_mapper

>From 20c8f066df04749730a3c84029cd86f62e050ec9 Mon Sep 17 00:00:00 2001
From: "ZhiQiang.Fan" <571253675 at QQ.com>
Date: Sat, 11 Jul 2026 02:56:29 +0800
Subject: [PATCH 2/2] fix format

---
 .../Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp       | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
index f198f114464a1..e4fd2480b1132 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
@@ -517,10 +517,11 @@ LogicalResult OpenACCDialectLLVMIRTranslationInterface::convertOperation(
         return success();
       })
       .Case<acc::CreateOp, acc::CopyinOp, acc::CopyoutOp, acc::DeleteOp,
-            acc::UpdateDeviceOp, acc::GetDevicePtrOp, acc::PresentOp>([](auto op) {
-        // NOP
-        return success();
-      })
+            acc::UpdateDeviceOp, acc::GetDevicePtrOp, acc::PresentOp>(
+          [](auto op) {
+            // NOP
+            return success();
+          })
       .Default([&](Operation *op) {
         return op->emitError("unsupported OpenACC operation: ")
                << op->getName();


From llvmlistbot at llvm.org  Fri Jul 10 12:00:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 12:00:22 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [openmp] [polly] Add
 LLVM_OMIT_SHARED_TARGETS to omit building any shared targets (PR #208813)
Message-ID: <llvm.org/llvm/llvm-project/pull/208813@github.com>


https://github.com/zeroomega created https://github.com/llvm/llvm-project/pull/208813

When LLVM is configured with  BUILD_SHARED_LIBS=OFF , several subprojects still unconditionally define and build standalone shared/module library targets (when PIC is enabled):
* LLVM: LTO(libLTO.so), Remarks (libRemarks.so), and LLVMgold.
* Clang: libclang (libclang.so) and clang-cpp (libclang-cpp.so).
* Polly: LLVMPolly (loadable pass plugin module).
* MLIR:  mlir-shlib (libMLIR.so) and shared execution engine/runtime runner libraries ( mlir_c_runner_utils , etc.).
* OpenMP: omp (libomp.so, enabled by default via  LIBOMP_ENABLE_SHARED).

When the stage-2 host build generates export files (LLVMExports.cmake, ClangTargets.cmake ,  PollyConfig.cmake ), these shared targets are exported with  add_library(<target> SHARED IMPORTED). During the child CMake configuration step for baremetal runtimes (where CMAKE_SYSTEM_NAME=Generic ),  find_package(LLVM)  loads LLVMExports.cmake . Under CMake 4+, calling
add_library(... SHARED IMPORTED)  on a target platform that lacks dynamic linking support triggers a fatal error.

We introduced LLVM_OMIT_SHARED_TARGETS flag, if set to ON, all shared targets above will be disabled to make the build static only.

>From 11b2accb76b2c7a4829fe87772482ceabd54f7fe Mon Sep 17 00:00:00 2001
From: Haowei Wu <haowei at google.com>
Date: Fri, 10 Jul 2026 11:50:33 -0700
Subject: [PATCH] Add LLVM_OMIT_SHARED_TARGETS to omit building any shared
 targets

When LLVM is configured with  BUILD_SHARED_LIBS=OFF , several
subprojects still unconditionally define and build standalone
shared/module library targets (when PIC is enabled):
* LLVM: LTO(libLTO.so), Remarks (libRemarks.so), and LLVMgold.
* Clang: libclang (libclang.so) and clang-cpp (libclang-cpp.so).
* Polly: LLVMPolly (loadable pass plugin module).
* MLIR:  mlir-shlib (libMLIR.so) and shared execution
    engine/runtime runner libraries ( mlir_c_runner_utils , etc.).
* OpenMP: omp (libomp.so, enabled by default via  LIBOMP_ENABLE_SHARED).

When the stage-2 host build generates export files (LLVMExports.cmake,
ClangTargets.cmake ,  PollyConfig.cmake ), these shared targets
are exported with  add_library(<target> SHARED IMPORTED). During the
child CMake configuration step for baremetal runtimes (where
CMAKE_SYSTEM_NAME=Generic ),  find_package(LLVM)  loads
LLVMExports.cmake . Under CMake 4+, calling
add_library(... SHARED IMPORTED)  on a target platform that lacks
dynamic linking support triggers a fatal error.

We introduced LLVM_OMIT_SHARED_TARGETS flag, if set to ON, all
shared targets above will be disabled to make the build static
only.
---
 clang/cmake/caches/Fuchsia-stage2.cmake | 1 +
 clang/tools/CMakeLists.txt              | 4 ++--
 clang/tools/libclang/CMakeLists.txt     | 4 ++--
 llvm/CMakeLists.txt                     | 7 +++++++
 llvm/tools/CMakeLists.txt               | 9 +++++++--
 llvm/tools/lto/CMakeLists.txt           | 2 +-
 llvm/tools/remarks-shlib/CMakeLists.txt | 2 +-
 mlir/lib/ExecutionEngine/CMakeLists.txt | 4 ++--
 mlir/tools/mlir-shlib/CMakeLists.txt    | 2 +-
 openmp/runtime/CMakeLists.txt           | 2 +-
 polly/cmake/CMakeLists.txt              | 2 +-
 polly/lib/CMakeLists.txt                | 2 +-
 12 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/clang/cmake/caches/Fuchsia-stage2.cmake b/clang/cmake/caches/Fuchsia-stage2.cmake
index 401258e9ae70e..3f46def846a4a 100644
--- a/clang/cmake/caches/Fuchsia-stage2.cmake
+++ b/clang/cmake/caches/Fuchsia-stage2.cmake
@@ -17,6 +17,7 @@ set(LLVM_ENABLE_LIBCXX ON CACHE BOOL "")
 set(LLVM_ENABLE_LIBEDIT OFF CACHE BOOL "")
 set(LLVM_ENABLE_LLD ON CACHE BOOL "")
 set(LLVM_ENABLE_LTO ON CACHE BOOL "")
+set(LLVM_OMIT_SHARED_TARGETS ON CACHE BOOL "")
 set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ON CACHE BOOL "")
 set(LLVM_ENABLE_PLUGINS OFF CACHE BOOL "")
 set(LLVM_ENABLE_UNWIND_TABLES OFF CACHE BOOL "")
diff --git a/clang/tools/CMakeLists.txt b/clang/tools/CMakeLists.txt
index a16089be041bf..a7cd5ae7a7ca3 100644
--- a/clang/tools/CMakeLists.txt
+++ b/clang/tools/CMakeLists.txt
@@ -32,8 +32,8 @@ endif()
 add_clang_subdirectory(clang-refactor)
 # For MinGW/Cygwin we only enable shared library if LLVM_LINK_LLVM_DYLIB=ON.
 # Without that option resulting library is too close to 2^16 DLL exports limit.
-if((UNIX AND NOT CYGWIN) OR (MSVC AND LLVM_BUILD_LLVM_DYLIB_VIS) OR
-  ((MINGW OR CYGWIN) AND LLVM_LINK_LLVM_DYLIB))
+if(((UNIX AND NOT CYGWIN) OR (MSVC AND LLVM_BUILD_LLVM_DYLIB_VIS) OR
+  ((MINGW OR CYGWIN) AND LLVM_LINK_LLVM_DYLIB)) AND NOT LLVM_OMIT_SHARED_TARGETS)
   add_clang_subdirectory(clang-shlib)
 endif()
 
diff --git a/clang/tools/libclang/CMakeLists.txt b/clang/tools/libclang/CMakeLists.txt
index 38f577499dd22..47903f96974a1 100644
--- a/clang/tools/libclang/CMakeLists.txt
+++ b/clang/tools/libclang/CMakeLists.txt
@@ -109,11 +109,11 @@ if (LLVM_EXPORTED_SYMBOL_FILE)
                      DEPENDS ${LIBCLANG_VERSION_SCRIPT_FILE})
 endif()
 
-if(LLVM_ENABLE_PIC OR ((WIN32 OR CYGWIN) AND NOT LIBCLANG_BUILD_STATIC))
+if((LLVM_ENABLE_PIC OR ((WIN32 OR CYGWIN) AND NOT LIBCLANG_BUILD_STATIC)) AND NOT LLVM_OMIT_SHARED_TARGETS)
   set(ENABLE_SHARED SHARED)
 endif()
 
-if(NOT LLVM_ENABLE_PIC OR LIBCLANG_BUILD_STATIC)
+if(NOT LLVM_ENABLE_PIC OR LIBCLANG_BUILD_STATIC OR LLVM_OMIT_SHARED_TARGETS)
   set(ENABLE_STATIC STATIC)
 endif()
 
diff --git a/llvm/CMakeLists.txt b/llvm/CMakeLists.txt
index 7b8f1b3b3ced9..77a54ba9d3089 100644
--- a/llvm/CMakeLists.txt
+++ b/llvm/CMakeLists.txt
@@ -1017,6 +1017,13 @@ if (CMAKE_BUILD_TYPE AND uppercase_CMAKE_BUILD_TYPE MATCHES "^(RELWITHDEBINFO|DE
   set(LLVM_OMIT_DAGISEL_COMMENTS OFF)
 endif()
 
+option(LLVM_OMIT_SHARED_TARGETS "Do not build targets that can only be built as shared libraries" OFF)
+if(LLVM_OMIT_SHARED_TARGETS)
+  set(BUILD_SHARED_LIBS OFF)
+  set(LLVM_BUILD_LLVM_DYLIB OFF)
+  set(LLVM_BUILD_LLVM_C_DYLIB OFF)
+endif()
+
 if (MSVC_IDE)
   option(LLVM_ADD_NATIVE_VISUALIZERS_TO_SOLUTION "Configure project to use Visual Studio native visualizers" TRUE)
 endif()
diff --git a/llvm/tools/CMakeLists.txt b/llvm/tools/CMakeLists.txt
index a579bdf0f6fb6..e685bf880ed95 100644
--- a/llvm/tools/CMakeLists.txt
+++ b/llvm/tools/CMakeLists.txt
@@ -9,7 +9,7 @@
 # traversing each directory.
 create_llvm_tool_options()
 
-if(NOT LLVM_BUILD_LLVM_DYLIB AND NOT LLVM_BUILD_LLVM_C_DYLIB)
+if(NOT LLVM_BUILD_LLVM_DYLIB AND NOT LLVM_BUILD_LLVM_C_DYLIB OR LLVM_OMIT_SHARED_TARGETS)
   set(LLVM_TOOL_LLVM_SHLIB_BUILD Off)
 endif()
 
@@ -17,8 +17,13 @@ if(NOT LLVM_USE_INTEL_JITEVENTS )
   set(LLVM_TOOL_LLVM_JITLISTENER_BUILD Off)
 endif()
 
-if(CYGWIN OR NOT LLVM_ENABLE_PIC)
+if(CYGWIN OR NOT LLVM_ENABLE_PIC OR LLVM_OMIT_SHARED_TARGETS)
   set(LLVM_TOOL_LTO_BUILD Off)
+  set(LLVM_TOOL_GOLD_BUILD Off)
+endif()
+
+if(LLVM_OMIT_SHARED_TARGETS)
+  set(LLVM_TOOL_REMARKS_SHLIB_BUILD Off)
 endif()
 
 if (LLVM_TOOL_LLVM_DRIVER_BUILD)
diff --git a/llvm/tools/lto/CMakeLists.txt b/llvm/tools/lto/CMakeLists.txt
index 5d13bf5e1e80d..b0a91e37a90a8 100644
--- a/llvm/tools/lto/CMakeLists.txt
+++ b/llvm/tools/lto/CMakeLists.txt
@@ -1,5 +1,5 @@
 # Building shared libraries requires PIC objects.
-if(LLVM_ENABLE_PIC)
+if(LLVM_ENABLE_PIC AND NOT LLVM_OMIT_SHARED_TARGETS)
 
   set(LLVM_LINK_COMPONENTS
     AllTargetsAsmParsers
diff --git a/llvm/tools/remarks-shlib/CMakeLists.txt b/llvm/tools/remarks-shlib/CMakeLists.txt
index 1b14827e68cd3..492dbdae5d6fd 100644
--- a/llvm/tools/remarks-shlib/CMakeLists.txt
+++ b/llvm/tools/remarks-shlib/CMakeLists.txt
@@ -1,5 +1,5 @@
 # Building shared libraries requires PIC objects.
-if(LLVM_ENABLE_PIC)
+if(LLVM_ENABLE_PIC AND NOT LLVM_OMIT_SHARED_TARGETS)
 
   set(LLVM_LINK_COMPONENTS
     Remarks
diff --git a/mlir/lib/ExecutionEngine/CMakeLists.txt b/mlir/lib/ExecutionEngine/CMakeLists.txt
index 87af4724f159a..26cf898165475 100644
--- a/mlir/lib/ExecutionEngine/CMakeLists.txt
+++ b/mlir/lib/ExecutionEngine/CMakeLists.txt
@@ -104,7 +104,7 @@ mlir_target_link_libraries(MLIRExecutionEngine PUBLIC
   MLIRTargetLLVMIRExport
   )
 
-if(LLVM_BUILD_LLVM_DYLIB AND NOT (WIN32 OR MINGW OR CYGWIN)) # Does not build on windows currently, see #106859
+if(LLVM_BUILD_LLVM_DYLIB AND NOT (WIN32 OR MINGW OR CYGWIN) AND NOT LLVM_OMIT_SHARED_TARGETS) # Does not build on windows currently, see #106859
   # Build a shared library for the execution engine. Some downstream projects
   # use this library to build their own CPU runners while preserving dynamic
   # linkage.
@@ -158,7 +158,7 @@ mlir_target_link_libraries(MLIRJitRunner PUBLIC
 
 # When -fPIC is not provided, shared libraries cannot be built if it links against
 # non-PIC code.
-if(LLVM_ENABLE_PIC)
+if(LLVM_ENABLE_PIC AND NOT LLVM_OMIT_SHARED_TARGETS)
   add_mlir_library(mlir_float16_utils
     SHARED
     Float16bits.cpp
diff --git a/mlir/tools/mlir-shlib/CMakeLists.txt b/mlir/tools/mlir-shlib/CMakeLists.txt
index a33c70c5807be..42ecc80069872 100644
--- a/mlir/tools/mlir-shlib/CMakeLists.txt
+++ b/mlir/tools/mlir-shlib/CMakeLists.txt
@@ -1,5 +1,5 @@
 # Building libmlir-cpp.so fails if LLVM_ENABLE_PIC=Off
-if (NOT LLVM_ENABLE_PIC)
+if (NOT LLVM_ENABLE_PIC OR LLVM_OMIT_SHARED_TARGETS)
   return()
 endif()
 
diff --git a/openmp/runtime/CMakeLists.txt b/openmp/runtime/CMakeLists.txt
index c0f5b5a4b1596..9ce480eb8297b 100644
--- a/openmp/runtime/CMakeLists.txt
+++ b/openmp/runtime/CMakeLists.txt
@@ -336,7 +336,7 @@ set(LIBOMP_USE_CANCEL_THREADS "${LIBOMP_HAVE_PTHREAD_CANCEL}" CACHE BOOL
 set(LIBOMP_ENABLE_SHARED TRUE CACHE BOOL
   "Shared library instead of static library?")
 
-if(WASM)
+if(WASM OR LLVM_OMIT_SHARED_TARGETS)
   libomp_warning_say("The WebAssembly build currently only supports static libraries; forcing LIBOMP_ENABLE_SHARED to false")
   set(LIBOMP_ENABLE_SHARED FALSE)
 endif()
diff --git a/polly/cmake/CMakeLists.txt b/polly/cmake/CMakeLists.txt
index cd1f5db842d93..161c894c0f9df 100644
--- a/polly/cmake/CMakeLists.txt
+++ b/polly/cmake/CMakeLists.txt
@@ -23,7 +23,7 @@ else()
 endif()
 
 set(POLLY_CONFIG_EXPORTED_TARGETS Polly ${ISL_TARGET})
-if (NOT (WIN32 OR CYGWIN) AND LLVM_ENABLE_PIC)
+if (NOT (WIN32 OR CYGWIN) AND LLVM_ENABLE_PIC AND NOT LLVM_OMIT_SHARED_TARGETS)
   # LLVMPolly is a dummy target on Win or if PIC code is disabled.
   list(APPEND POLLY_CONFIG_EXPORTED_TARGETS LLVMPolly)
 endif()
diff --git a/polly/lib/CMakeLists.txt b/polly/lib/CMakeLists.txt
index 6f6e4d5782dfb..9d33af774a784 100644
--- a/polly/lib/CMakeLists.txt
+++ b/polly/lib/CMakeLists.txt
@@ -120,7 +120,7 @@ target_link_libraries(Polly PUBLIC
 
 # Create a loadable module Polly.so that can be loaded using
 # LLVM's/clang's "-load" option.
-if (WIN32 OR CYGWIN OR NOT LLVM_ENABLE_PIC)
+if (WIN32 OR CYGWIN OR NOT LLVM_ENABLE_PIC OR LLVM_OMIT_SHARED_TARGETS)
   # Add dummy target, either because loadable modules are not supported
   # as on Windows or because PIC code has been disabled
   add_custom_target(LLVMPolly)


From llvmlistbot at llvm.org  Fri Jul 10 17:46:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 17:46:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix acc.present in acc.data
	translation (PR #208806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208806@github.com>
Message-ID: <6a519282.eb165e5c.1817a5.ac84@mx.google.com>


StarryCSF wrote:

> This is a deprecated path in the OpenACC codegen. We are currently upstreaming the new one that will enable an end to end flow.

Thanks for the review. I wasn't aware this path is deprecated. 

https://github.com/llvm/llvm-project/pull/208806

From llvmlistbot at llvm.org  Fri Jul 10 17:47:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 17:47:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix acc.present in acc.data
	translation (PR #208806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208806@github.com>
Message-ID: <6a519299.e5907290.281f16.f3bd@mx.google.com>


https://github.com/StarryCSF converted_to_draft https://github.com/llvm/llvm-project/pull/208806

From llvmlistbot at llvm.org  Fri Jul 10 17:49:16 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Fri, 10 Jul 2026 17:49:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix acc.present in acc.data
	translation (PR #208806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208806@github.com>
Message-ID: <6a51930c.0c05e68e.1e82f1.b34a@mx.google.com>


clementval wrote:

> > This is a deprecated path in the OpenACC codegen. We are currently upstreaming the new one that will enable an end to end flow.
> 
> Thanks for the review. I wasn't aware this path is deprecated.

Yeah sorry about that, we should remove it 

https://github.com/llvm/llvm-project/pull/208806

From llvmlistbot at llvm.org  Fri Jul 10 17:50:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 17:50:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] New Class Lowering (PR #208864)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208864@github.com>
Message-ID: <6a51936b.d76d81af.39f9be.01fc@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

* 3910 tests passed
* 440 tests skipped
* 2 tests failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Dialect/EmitC/lower-nested-module-to-class.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir -lower-nested-module-to-class -split-input-file | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir' -lower-nested-module-to-class -split-input-file
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir:14:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: emitc.class @tag_class {
# |                ^
# | <stdin>:1:17: note: scanning from here
# | module @outer1 {
# |                 ^
# | <stdin>:2:32: note: possible intended match here
# |  module @tag_class attributes {emitc.class} {
# |                                ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\mlir\test\Dialect\EmitC\lower-nested-module-to-class.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: module @outer1 { 
# | next:14'0                    {   search range start (exclusive)
# | next:14'1                        error: no match found in search range
# |            2:  module @tag_class attributes {emitc.class} { 
# | next:14'2                                    ?                possible intended match
# |            3:  emitc.func @foo() { 
# |            4:  return 
# |            5:  } 
# |            6:  } 
# |            7: } 
# |            .
# |            .
# |            .
# |           23:  emitc.func @foo() { 
# |           24:  return 
# |           25:  } 
# |           26:  } 
# |           27: } 
# |           28:  
# | next:14'3      } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>MLIR.Dialect/Func/wrap-func-in-module.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir --wrap-func-in-module | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir' --wrap-func-in-module
# note: command had no output on stdout or stderr
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir:12:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: module {
# |                ^
# | <stdin>:1:9: note: scanning from here
# | module {
# |         ^
# | <stdin>:2:2: note: possible intended match here
# |  module @foo {
# |  ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\mlir\test\Dialect\Func\wrap-func-in-module.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: module { 
# | next:12'0            {   search range start (exclusive)
# | next:12'1                error: no match found in search range
# |            2:  module @foo { 
# | next:12'2      ?               possible intended match
# |            3:  func.func @foo() { 
# |            4:  return 
# |            5:  } 
# |            6:  } 
# |            7:  module @bar { 
# |            8:  func.func @bar() { 
# |            9:  return 
# |           10:  } 
# |           11:  } 
# |           12: } 
# |           13:  
# | next:12'3      } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208864

From llvmlistbot at llvm.org  Fri Jul 10 17:50:52 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 17:50:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] New Class Lowering (PR #208864)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208864@github.com>
Message-ID: <6a51936c.ff037943.2e959a.1ac1@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 8428 tests passed
* 630 tests skipped
* 2 tests failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Dialect/EmitC/lower-nested-module-to-class.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir -lower-nested-module-to-class -split-input-file | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir -lower-nested-module-to-class -split-input-file
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir:14:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: emitc.class @tag_class {
# |                ^
# | <stdin>:1:17: note: scanning from here
# | module @outer1 {
# |                 ^
# | <stdin>:2:32: note: possible intended match here
# |  module @tag_class attributes {emitc.class} {
# |                                ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/EmitC/lower-nested-module-to-class.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: module @outer1 { 
# | next:14'0                    {   search range start (exclusive)
# | next:14'1                        error: no match found in search range
# |            2:  module @tag_class attributes {emitc.class} { 
# | next:14'2                                    ?                possible intended match
# |            3:  emitc.func @foo() { 
# |            4:  return 
# |            5:  } 
# |            6:  } 
# |            7: } 
# |            .
# |            .
# |            .
# |           23:  emitc.func @foo() { 
# |           24:  return 
# |           25:  } 
# |           26:  } 
# |           27: } 
# |           28:  
# | next:14'3      } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>
<details>
<summary>MLIR.Dialect/Func/wrap-func-in-module.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir --wrap-func-in-module | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir --wrap-func-in-module
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir:12:16: error: CHECK-NEXT: expected string not found in input
# | // CHECK-NEXT: module {
# |                ^
# | <stdin>:1:9: note: scanning from here
# | module {
# |         ^
# | <stdin>:2:2: note: possible intended match here
# |  module @foo {
# |  ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Dialect/Func/wrap-func-in-module.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: module { 
# | next:12'0            {   search range start (exclusive)
# | next:12'1                error: no match found in search range
# |            2:  module @foo { 
# | next:12'2      ?               possible intended match
# |            3:  func.func @foo() { 
# |            4:  return 
# |            5:  } 
# |            6:  } 
# |            7:  module @bar { 
# |            8:  func.func @bar() { 
# |            9:  return 
# |           10:  } 
# |           11:  } 
# |           12: } 
# |           13:  
# | next:12'3      } search range end (exclusive)
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208864

From llvmlistbot at llvm.org  Fri Jul 10 18:03:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 18:03:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [flang][OpenACC] Fix acc.present in acc.data
	translation (PR #208806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208806@github.com>
Message-ID: <6a51964e.30759430.24d50c.0c12@mx.google.com>


StarryCSF wrote:

> > > This is a deprecated path in the OpenACC codegen. We are currently upstreaming the new one that will enable an end to end flow.
> > 
> > 
> > Thanks for the review. I wasn't aware this path is deprecated.
> 
> Yeah sorry about that, we should remove it

Sounds good, thanks for the heads-up. What's the plan for the translation codegen? Will it be completely rewritten, or will some parts carry over? Looking forward to the new PR and RFC

https://github.com/llvm/llvm-project/pull/208806

From llvmlistbot at llvm.org  Fri Jul 10 19:00:33 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 10 Jul 2026 19:00:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a51a3c1.1905e68e.18ad68.f98a@mx.google.com>


https://github.com/Jianhui-Li edited https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Fri Jul 10 20:37:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 20:37:50 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Support
 mapping of zero-sized arrays (PR #208133)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208133@github.com>
Message-ID: <6a51ba8e.27b3b003.3b871.a5f4@mx.google.com>


agozillon wrote:

Looks like a sporadic but rare failure that's not quite related to this PR from what I can tell, that is linked to the fortran library (hence the PR ping probably), the buildbot has went back to green and there doesn't appear to be an error message in the log indicating this PR as a contributor to the failure. However, if someone else believes otherwise please don't hesitate to ping me and I'll fix it.  

https://github.com/llvm/llvm-project/pull/208133

From llvmlistbot at llvm.org  Fri Jul 10 20:44:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 20:44:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Guard transfer read narrow
	type emulation (PR #208873)
Message-ID: <llvm.org/llvm/llvm-project/pull/208873@github.com>


https://github.com/qyingwu created https://github.com/llvm/llvm-project/pull/208873

Fixes #208198.

`ConvertVectorTransferRead` in the vector narrow type emulation pass could
call
`fitsInMultiByteContainerTy` with a non-sub-byte element type, such as `f16`.
That helper asserts that the vector element type is sub-byte, so the test pass
could crash on unsupported input instead of failing conversion cleanly.

Add an explicit supported-width check before calling the helper. The check
uses a shared width predicate because existing patterns support both integer and
float element types with 2-bit and 4-bit widths.

Also add a regression test for the reported `vector.transfer_read` case to
verify that the pass emits a normal legalization failure instead of asserting.

Verification:
- `cmake --build /tmp/mlir-208198 --target mlir-opt`
- `/tmp/mlir-208198/bin/mlir-opt --test-emulate-narrow-int="memref-load-
bitwidth=32" --cse --verify-diagnostics --split-input-file mlir/test/Dialect/
Vector/vector-emulate-narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type.mlir`

AI tool usage:
I used ChatGPT/Codex to help analyze the assertion path, draft the patch, and
run focused verification. I manually reviewed the final diff and test results.

>From 7bc50a3c052d3253af97bafec16793036a02d305 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 10 Jul 2026 20:42:06 -0700
Subject: [PATCH] [mlir][vector] Guard transfer read narrow type emulation

---
 .../Vector/Transforms/VectorEmulateNarrowType.cpp  | 14 +++++++++++++-
 .../Vector/vector-emulate-narrow-type-invalid.mlir | 10 ++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..2286dd1cb995d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -51,6 +51,13 @@ using MemRefValue = TypedValue<MemRefType>;
 // Utils
 //===----------------------------------------------------------------------===//
 
+/// Returns true for element widths supported by this sub-byte emulation. This
+/// is intentionally width-only: both integer and float element types with these
+/// widths are handled by existing patterns.
+static bool isSupportedSubByteElementWidth(unsigned bitWidth) {
+  return bitWidth == 2 || bitWidth == 4;
+}
+
 /// Returns a compressed mask for the emulated vector. For example, when
 /// emulating an eight-element `i8` vector with `i32` (i.e. when the source
 /// elements span two dest elements), this method compresses `vector<8xi1>`
@@ -1307,6 +1314,11 @@ struct ConvertVectorTransferRead final
         cast<MemRefType>(adaptor.getBase().getType()).getElementType();
     Type emulatedElemTy = op.getType().getElementType();
     int emulatedBits = emulatedElemTy.getIntOrFloatBitWidth();
+    if (!isSupportedSubByteElementWidth(emulatedBits))
+      return rewriter.notifyMatchFailure(
+          op, "only 2-bit and 4-bit sub-byte type is supported at this "
+              "moment");
+
     int containerBits = containerElemTy.getIntOrFloatBitWidth();
 
     // Check per-element alignment.
@@ -1683,7 +1695,7 @@ static LogicalResult alignedConversionPrecondition(PatternRewriter &rewriter,
   assert(containerBits % 8 == 0 && "Not a multi-byte scalar type!");
 
   // TODO: Add support other widths (when/if needed)
-  if (subByteBits != 2 && subByteBits != 4)
+  if (!isSupportedSubByteElementWidth(subByteBits))
     return rewriter.notifyMatchFailure(
         op, "only 2-bit and 4-bit sub-byte type is supported at this moment");
 
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
new file mode 100644
index 0000000000000..4b092331ce3df
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt --test-emulate-narrow-int="memref-load-bitwidth=32" --cse --verify-diagnostics --split-input-file %s
+
+func.func @transfer_read_non_subbyte_element(
+    %arg0: memref<4x?x16xf16>, %arg1: index, %arg2: index, %arg3: index) {
+  %cst = arith.constant 3.000000e+00 : f16
+  // expected-error @below {{failed to legalize operation 'vector.transfer_read' that was explicitly marked illegal}}
+  vector.transfer_read %arg0[%arg1, %arg2, %arg3], %cst :
+    memref<4x?x16xf16>, vector<8xf16>
+  return
+}


From llvmlistbot at llvm.org  Fri Jul 10 20:45:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 20:45:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Guard transfer read narrow
	type emulation (PR #208873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208873@github.com>
Message-ID: <6a51bc46.492e23e4.157d47.a922@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: qyingwu

<details>
<summary>Changes</summary>

Fixes #<!-- -->208198.

`ConvertVectorTransferRead` in the vector narrow type emulation pass could
call
`fitsInMultiByteContainerTy` with a non-sub-byte element type, such as `f16`.
That helper asserts that the vector element type is sub-byte, so the test pass
could crash on unsupported input instead of failing conversion cleanly.

Add an explicit supported-width check before calling the helper. The check
uses a shared width predicate because existing patterns support both integer and
float element types with 2-bit and 4-bit widths.

Also add a regression test for the reported `vector.transfer_read` case to
verify that the pass emits a normal legalization failure instead of asserting.

Verification:
- `cmake --build /tmp/mlir-208198 --target mlir-opt`
- `/tmp/mlir-208198/bin/mlir-opt --test-emulate-narrow-int="memref-load-
bitwidth=32" --cse --verify-diagnostics --split-input-file mlir/test/Dialect/
Vector/vector-emulate-narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type.mlir`

AI tool usage:
I used ChatGPT/Codex to help analyze the assertion path, draft the patch, and
run focused verification. I manually reviewed the final diff and test results.

---
Full diff: https://github.com/llvm/llvm-project/pull/208873.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp (+13-1) 
- (added) mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir (+10) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..2286dd1cb995d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -51,6 +51,13 @@ using MemRefValue = TypedValue<MemRefType>;
 // Utils
 //===----------------------------------------------------------------------===//
 
+/// Returns true for element widths supported by this sub-byte emulation. This
+/// is intentionally width-only: both integer and float element types with these
+/// widths are handled by existing patterns.
+static bool isSupportedSubByteElementWidth(unsigned bitWidth) {
+  return bitWidth == 2 || bitWidth == 4;
+}
+
 /// Returns a compressed mask for the emulated vector. For example, when
 /// emulating an eight-element `i8` vector with `i32` (i.e. when the source
 /// elements span two dest elements), this method compresses `vector<8xi1>`
@@ -1307,6 +1314,11 @@ struct ConvertVectorTransferRead final
         cast<MemRefType>(adaptor.getBase().getType()).getElementType();
     Type emulatedElemTy = op.getType().getElementType();
     int emulatedBits = emulatedElemTy.getIntOrFloatBitWidth();
+    if (!isSupportedSubByteElementWidth(emulatedBits))
+      return rewriter.notifyMatchFailure(
+          op, "only 2-bit and 4-bit sub-byte type is supported at this "
+              "moment");
+
     int containerBits = containerElemTy.getIntOrFloatBitWidth();
 
     // Check per-element alignment.
@@ -1683,7 +1695,7 @@ static LogicalResult alignedConversionPrecondition(PatternRewriter &rewriter,
   assert(containerBits % 8 == 0 && "Not a multi-byte scalar type!");
 
   // TODO: Add support other widths (when/if needed)
-  if (subByteBits != 2 && subByteBits != 4)
+  if (!isSupportedSubByteElementWidth(subByteBits))
     return rewriter.notifyMatchFailure(
         op, "only 2-bit and 4-bit sub-byte type is supported at this moment");
 
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
new file mode 100644
index 0000000000000..4b092331ce3df
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt --test-emulate-narrow-int="memref-load-bitwidth=32" --cse --verify-diagnostics --split-input-file %s
+
+func.func @transfer_read_non_subbyte_element(
+    %arg0: memref<4x?x16xf16>, %arg1: index, %arg2: index, %arg3: index) {
+  %cst = arith.constant 3.000000e+00 : f16
+  // expected-error @below {{failed to legalize operation 'vector.transfer_read' that was explicitly marked illegal}}
+  vector.transfer_read %arg0[%arg1, %arg2, %arg3], %cst :
+    memref<4x?x16xf16>, vector<8xf16>
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208873

From llvmlistbot at llvm.org  Fri Jul 10 20:45:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 20:45:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Guard transfer read narrow
	type emulation (PR #208873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208873@github.com>
Message-ID: <6a51bc47.92ea0e2a.13fd37.2942@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-vector

Author: qyingwu

<details>
<summary>Changes</summary>

Fixes #<!-- -->208198.

`ConvertVectorTransferRead` in the vector narrow type emulation pass could
call
`fitsInMultiByteContainerTy` with a non-sub-byte element type, such as `f16`.
That helper asserts that the vector element type is sub-byte, so the test pass
could crash on unsupported input instead of failing conversion cleanly.

Add an explicit supported-width check before calling the helper. The check
uses a shared width predicate because existing patterns support both integer and
float element types with 2-bit and 4-bit widths.

Also add a regression test for the reported `vector.transfer_read` case to
verify that the pass emits a normal legalization failure instead of asserting.

Verification:
- `cmake --build /tmp/mlir-208198 --target mlir-opt`
- `/tmp/mlir-208198/bin/mlir-opt --test-emulate-narrow-int="memref-load-
bitwidth=32" --cse --verify-diagnostics --split-input-file mlir/test/Dialect/
Vector/vector-emulate-narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type-invalid.mlir`
- `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/Vector/vector-emulate-
narrow-type.mlir`

AI tool usage:
I used ChatGPT/Codex to help analyze the assertion path, draft the patch, and
run focused verification. I manually reviewed the final diff and test results.

---
Full diff: https://github.com/llvm/llvm-project/pull/208873.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp (+13-1) 
- (added) mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir (+10) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..2286dd1cb995d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -51,6 +51,13 @@ using MemRefValue = TypedValue<MemRefType>;
 // Utils
 //===----------------------------------------------------------------------===//
 
+/// Returns true for element widths supported by this sub-byte emulation. This
+/// is intentionally width-only: both integer and float element types with these
+/// widths are handled by existing patterns.
+static bool isSupportedSubByteElementWidth(unsigned bitWidth) {
+  return bitWidth == 2 || bitWidth == 4;
+}
+
 /// Returns a compressed mask for the emulated vector. For example, when
 /// emulating an eight-element `i8` vector with `i32` (i.e. when the source
 /// elements span two dest elements), this method compresses `vector<8xi1>`
@@ -1307,6 +1314,11 @@ struct ConvertVectorTransferRead final
         cast<MemRefType>(adaptor.getBase().getType()).getElementType();
     Type emulatedElemTy = op.getType().getElementType();
     int emulatedBits = emulatedElemTy.getIntOrFloatBitWidth();
+    if (!isSupportedSubByteElementWidth(emulatedBits))
+      return rewriter.notifyMatchFailure(
+          op, "only 2-bit and 4-bit sub-byte type is supported at this "
+              "moment");
+
     int containerBits = containerElemTy.getIntOrFloatBitWidth();
 
     // Check per-element alignment.
@@ -1683,7 +1695,7 @@ static LogicalResult alignedConversionPrecondition(PatternRewriter &rewriter,
   assert(containerBits % 8 == 0 && "Not a multi-byte scalar type!");
 
   // TODO: Add support other widths (when/if needed)
-  if (subByteBits != 2 && subByteBits != 4)
+  if (!isSupportedSubByteElementWidth(subByteBits))
     return rewriter.notifyMatchFailure(
         op, "only 2-bit and 4-bit sub-byte type is supported at this moment");
 
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
new file mode 100644
index 0000000000000..4b092331ce3df
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt --test-emulate-narrow-int="memref-load-bitwidth=32" --cse --verify-diagnostics --split-input-file %s
+
+func.func @transfer_read_non_subbyte_element(
+    %arg0: memref<4x?x16xf16>, %arg1: index, %arg2: index, %arg3: index) {
+  %cst = arith.constant 3.000000e+00 : f16
+  // expected-error @below {{failed to legalize operation 'vector.transfer_read' that was explicitly marked illegal}}
+  vector.transfer_read %arg0[%arg1, %arg2, %arg3], %cst :
+    memref<4x?x16xf16>, vector<8xf16>
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208873

From llvmlistbot at llvm.org  Fri Jul 10 20:56:08 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 20:56:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Guard transfer read narrow
	type emulation (PR #208873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208873@github.com>
Message-ID: <6a51bed8.e624eb8d.39cd4e.5e91@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/208873

>From 7bc50a3c052d3253af97bafec16793036a02d305 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Fri, 10 Jul 2026 20:42:06 -0700
Subject: [PATCH] [mlir][vector] Guard transfer read narrow type emulation

---
 .../Vector/Transforms/VectorEmulateNarrowType.cpp  | 14 +++++++++++++-
 .../Vector/vector-emulate-narrow-type-invalid.mlir | 10 ++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
index 9faaebdcf8f35..2286dd1cb995d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp
@@ -51,6 +51,13 @@ using MemRefValue = TypedValue<MemRefType>;
 // Utils
 //===----------------------------------------------------------------------===//
 
+/// Returns true for element widths supported by this sub-byte emulation. This
+/// is intentionally width-only: both integer and float element types with these
+/// widths are handled by existing patterns.
+static bool isSupportedSubByteElementWidth(unsigned bitWidth) {
+  return bitWidth == 2 || bitWidth == 4;
+}
+
 /// Returns a compressed mask for the emulated vector. For example, when
 /// emulating an eight-element `i8` vector with `i32` (i.e. when the source
 /// elements span two dest elements), this method compresses `vector<8xi1>`
@@ -1307,6 +1314,11 @@ struct ConvertVectorTransferRead final
         cast<MemRefType>(adaptor.getBase().getType()).getElementType();
     Type emulatedElemTy = op.getType().getElementType();
     int emulatedBits = emulatedElemTy.getIntOrFloatBitWidth();
+    if (!isSupportedSubByteElementWidth(emulatedBits))
+      return rewriter.notifyMatchFailure(
+          op, "only 2-bit and 4-bit sub-byte type is supported at this "
+              "moment");
+
     int containerBits = containerElemTy.getIntOrFloatBitWidth();
 
     // Check per-element alignment.
@@ -1683,7 +1695,7 @@ static LogicalResult alignedConversionPrecondition(PatternRewriter &rewriter,
   assert(containerBits % 8 == 0 && "Not a multi-byte scalar type!");
 
   // TODO: Add support other widths (when/if needed)
-  if (subByteBits != 2 && subByteBits != 4)
+  if (!isSupportedSubByteElementWidth(subByteBits))
     return rewriter.notifyMatchFailure(
         op, "only 2-bit and 4-bit sub-byte type is supported at this moment");
 
diff --git a/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
new file mode 100644
index 0000000000000..4b092331ce3df
--- /dev/null
+++ b/mlir/test/Dialect/Vector/vector-emulate-narrow-type-invalid.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt --test-emulate-narrow-int="memref-load-bitwidth=32" --cse --verify-diagnostics --split-input-file %s
+
+func.func @transfer_read_non_subbyte_element(
+    %arg0: memref<4x?x16xf16>, %arg1: index, %arg2: index, %arg3: index) {
+  %cst = arith.constant 3.000000e+00 : f16
+  // expected-error @below {{failed to legalize operation 'vector.transfer_read' that was explicitly marked illegal}}
+  vector.transfer_read %arg0[%arg1, %arg2, %arg3], %cst :
+    memref<4x?x16xf16>, vector<8xf16>
+  return
+}


From llvmlistbot at llvm.org  Fri Jul 10 22:56:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 22:56:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transforms] Replace dead function
 argument uses with poison (PR #208881)
Message-ID: <llvm.org/llvm/llvm-project/pull/208881@github.com>


https://github.com/jjppp created https://github.com/llvm/llvm-project/pull/208881

Dropping all uses of a dead function argument can leave null operands in surviving side-effecting operations in unreachable code.
Subsequent canonicalization may then crash while folding the invalid operation.

This PR replaces the remaining uses with `ub.poison` before erasing the argument and add a regression test.

Fixes #206920, #203226

>From 9899ab7105763e09099c40e041c7210121cf0ca7 Mon Sep 17 00:00:00 2001
From: jjppp <jpwang at smail.nju.edu.cn>
Date: Sat, 11 Jul 2026 13:51:22 +0800
Subject: [PATCH] [mlir][Transforms] Replace dead function argument uses with
 poison

---
 mlir/lib/Transforms/RemoveDeadValues.cpp     | 17 +++++--
 mlir/test/Transforms/remove-dead-values.mlir | 49 ++++++++++++++++++++
 2 files changed, 63 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index 6e55bc390be23..247c66aacef3a 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -640,9 +640,20 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
       llvm::interleaveComma(f.nonLiveRets.set_bits(), os);
       os << "]";
     });
-    // Drop all uses of the dead arguments.
-    for (auto deadIdx : f.nonLiveArgs.set_bits())
-      f.funcOp.getArgument(deadIdx).dropAllUses();
+    // Replace remaining uses of the dead arguments with poison values. Simply
+    // dropping the uses would leave null operands behind in ops that survive
+    // the pass (e.g. a side-effecting op in an unreachable function, whose
+    // values are initialized as dead by the liveness analysis).
+    for (auto deadIdx : f.nonLiveArgs.set_bits()) {
+      BlockArgument arg = f.funcOp.getArgument(deadIdx);
+      // Avoid creating an unused poison value if there are no uses to replace.
+      if (arg.use_empty())
+        continue;
+      rewriter.setInsertionPointToStart(arg.getOwner());
+      Value poison =
+          ub::PoisonOp::create(rewriter, arg.getLoc(), arg.getType());
+      rewriter.replaceAllUsesWith(arg, poison);
+    }
     // Some functions may not allow erasing arguments or results. These calls
     // return failure in such cases without modifying the function, so it's okay
     // to proceed.
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 390a448060b7f..207ce48cba4f5 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -895,3 +895,52 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Verify that the upper bound of a for loop inside an unreachable
+// function is correctly replaced with a poison value.
+// CHECK-LABEL: module @unreachable_func_with_for_loops
+// CHECK-NEXT:    func.func private @main_func() {
+// CHECK-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      return
+
+// CHECK-CANONICALIZE-LABEL: module @unreachable_func_with_for_loops
+// CHECK-CANONICALIZE-NEXT:    func.func private @main_func() {
+// CHECK-CANONICALIZE-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      return
+module @unreachable_func_with_for_loops {
+  func.func private @main_func(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}


From llvmlistbot at llvm.org  Fri Jul 10 22:56:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 22:56:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transforms] Replace dead function
 argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a51db0f.d76d81af.39f9be.bd0d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: jpwang (jjppp)

<details>
<summary>Changes</summary>

Dropping all uses of a dead function argument can leave null operands in surviving side-effecting operations in unreachable code.
Subsequent canonicalization may then crash while folding the invalid operation.

This PR replaces the remaining uses with `ub.poison` before erasing the argument and add a regression test.

Fixes #<!-- -->206920, #<!-- -->203226

---
Full diff: https://github.com/llvm/llvm-project/pull/208881.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+14-3) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+49) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index 6e55bc390be23..247c66aacef3a 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -640,9 +640,20 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
       llvm::interleaveComma(f.nonLiveRets.set_bits(), os);
       os << "]";
     });
-    // Drop all uses of the dead arguments.
-    for (auto deadIdx : f.nonLiveArgs.set_bits())
-      f.funcOp.getArgument(deadIdx).dropAllUses();
+    // Replace remaining uses of the dead arguments with poison values. Simply
+    // dropping the uses would leave null operands behind in ops that survive
+    // the pass (e.g. a side-effecting op in an unreachable function, whose
+    // values are initialized as dead by the liveness analysis).
+    for (auto deadIdx : f.nonLiveArgs.set_bits()) {
+      BlockArgument arg = f.funcOp.getArgument(deadIdx);
+      // Avoid creating an unused poison value if there are no uses to replace.
+      if (arg.use_empty())
+        continue;
+      rewriter.setInsertionPointToStart(arg.getOwner());
+      Value poison =
+          ub::PoisonOp::create(rewriter, arg.getLoc(), arg.getType());
+      rewriter.replaceAllUsesWith(arg, poison);
+    }
     // Some functions may not allow erasing arguments or results. These calls
     // return failure in such cases without modifying the function, so it's okay
     // to proceed.
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 390a448060b7f..207ce48cba4f5 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -895,3 +895,52 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Verify that the upper bound of a for loop inside an unreachable
+// function is correctly replaced with a poison value.
+// CHECK-LABEL: module @unreachable_func_with_for_loops
+// CHECK-NEXT:    func.func private @main_func() {
+// CHECK-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      return
+
+// CHECK-CANONICALIZE-LABEL: module @unreachable_func_with_for_loops
+// CHECK-CANONICALIZE-NEXT:    func.func private @main_func() {
+// CHECK-CANONICALIZE-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      return
+module @unreachable_func_with_for_loops {
+  func.func private @main_func(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Fri Jul 10 22:56:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 10 Jul 2026 22:56:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transforms] Replace dead function
 argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a51db10.bd93e731.358318.5b2f@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: jpwang (jjppp)

<details>
<summary>Changes</summary>

Dropping all uses of a dead function argument can leave null operands in surviving side-effecting operations in unreachable code.
Subsequent canonicalization may then crash while folding the invalid operation.

This PR replaces the remaining uses with `ub.poison` before erasing the argument and add a regression test.

Fixes #<!-- -->206920, #<!-- -->203226

---
Full diff: https://github.com/llvm/llvm-project/pull/208881.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+14-3) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+49) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index 6e55bc390be23..247c66aacef3a 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -640,9 +640,20 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
       llvm::interleaveComma(f.nonLiveRets.set_bits(), os);
       os << "]";
     });
-    // Drop all uses of the dead arguments.
-    for (auto deadIdx : f.nonLiveArgs.set_bits())
-      f.funcOp.getArgument(deadIdx).dropAllUses();
+    // Replace remaining uses of the dead arguments with poison values. Simply
+    // dropping the uses would leave null operands behind in ops that survive
+    // the pass (e.g. a side-effecting op in an unreachable function, whose
+    // values are initialized as dead by the liveness analysis).
+    for (auto deadIdx : f.nonLiveArgs.set_bits()) {
+      BlockArgument arg = f.funcOp.getArgument(deadIdx);
+      // Avoid creating an unused poison value if there are no uses to replace.
+      if (arg.use_empty())
+        continue;
+      rewriter.setInsertionPointToStart(arg.getOwner());
+      Value poison =
+          ub::PoisonOp::create(rewriter, arg.getLoc(), arg.getType());
+      rewriter.replaceAllUsesWith(arg, poison);
+    }
     // Some functions may not allow erasing arguments or results. These calls
     // return failure in such cases without modifying the function, so it's okay
     // to proceed.
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 390a448060b7f..207ce48cba4f5 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -895,3 +895,52 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Verify that the upper bound of a for loop inside an unreachable
+// function is correctly replaced with a poison value.
+// CHECK-LABEL: module @unreachable_func_with_for_loops
+// CHECK-NEXT:    func.func private @main_func() {
+// CHECK-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-NEXT:        gpu.barrier
+// CHECK-NEXT:      }
+// CHECK-NEXT:      return
+
+// CHECK-CANONICALIZE-LABEL: module @unreachable_func_with_for_loops
+// CHECK-CANONICALIZE-NEXT:    func.func private @main_func() {
+// CHECK-CANONICALIZE-NEXT:      %[[ARG3:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C0:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C1:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      %[[C10:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[C10]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      scf.for %{{.*}} = %[[C0]] to %[[ARG3]] step %[[C1]] {
+// CHECK-CANONICALIZE-NEXT:        gpu.barrier
+// CHECK-CANONICALIZE-NEXT:      }
+// CHECK-CANONICALIZE-NEXT:      return
+module @unreachable_func_with_for_loops {
+  func.func private @main_func(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 11 02:11:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 02:11:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transforms] Replace dead function
 argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a5208dc.266b0cc5.29d78.342c@mx.google.com>


https://github.com/jjppp edited https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 11 06:34:38 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sat, 11 Jul 2026 06:34:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
Message-ID: <llvm.org/llvm/llvm-project/pull/208913@github.com>


https://github.com/secona created https://github.com/llvm/llvm-project/pull/208913

Fixes #202787

The crash comes from a call to `primaryTypeEncoding` where it reaches an `llvm_unreachable`. Previous change in #183898 fixes a the same crash issue of `primaryTypeEncoding` by adding `isValidPrimaryType()` check in SparseTensorNewConverter. The same fix is applied here by adding a call to `isValidPrimaryType()` in SparseTensorEmptyConverter. Now the pass errors instead of crashing.

>From 06b101ed45b533b8017949c18a07081dfccca6f1 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sat, 11 Jul 2026 20:16:57 +0700
Subject: [PATCH] [mlir][SparseTensor] Add guards for unknown primary types

---
 .../Transforms/SparseTensorConversion.cpp     |  2 ++
 .../SparseTensor/conversion_invalid.mlir      | 23 +++++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
index c9bcefe92de79..fb336c2f02f52 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
@@ -449,6 +449,8 @@ class SparseTensorEmptyConverter : public OpConversionPattern<tensor::EmptyOp> {
     const auto stt = getSparseTensorType(op);
     if (!stt.hasEncoding())
       return failure();
+    if (!isValidPrimaryType(stt.getElementType()))
+      return rewriter.notifyMatchFailure(op, "unsupported element type");
     // Gather all dimension sizes as SSA values.
     const Dimension dimRank = stt.getDimRank();
     SmallVector<Value> dimSizesValues;
diff --git a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
index 456c1898a9311..9166e4c3a0512 100644
--- a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
+++ b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
@@ -12,3 +12,26 @@ func.func @new_index_elem_type(%arg0: index) {
   %0 = sparse_tensor.new %arg0 : index to tensor<?xindex, #sparse>
   return
 }
+
+// -----
+
+#map = affine_map<(d0) -> (0, d0)>
+#map1 = affine_map<(d0) -> (d0)>
+#sparse = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+module {
+  func.func @main(%arg0: tensor<1x77xi1>, %arg1: tensor<1x77xi1>) -> tensor<77xi1, #sparse> {
+    %0 = tensor.empty() : tensor<77xi1, #sparse>
+    %1 = linalg.generic {indexing_maps = [#map, #map, #map1], iterator_types = ["parallel"]}
+      ins(%arg0, %arg1 : tensor<1x77xi1>, tensor<1x77xi1>)
+      outs(%0 : tensor<77xi1, #sparse>)
+    {
+    ^bb0(%in: i1, %in_0: i1, %out: i1):
+      %2 = arith.addi %in, %in_0 : i1
+      linalg.yield %2 : i1
+    } -> tensor<77xi1, #sparse>
+
+    // expected-error at +2 {{failed to legalize unresolved materialization}}
+    // expected-note at +1 {{see existing live user here}}
+    return %1 : tensor<77xi1, #sparse>
+  }
+}


From llvmlistbot at llvm.org  Sat Jul 11 06:35:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 06:35:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a52468f.e55dc341.1f19bd.ccba@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Vito Secona (secona)

<details>
<summary>Changes</summary>

Fixes #<!-- -->202787

The crash comes from a call to `primaryTypeEncoding` where it reaches an `llvm_unreachable`. Previous change in #<!-- -->183898 fixes a the same crash issue of `primaryTypeEncoding` by adding `isValidPrimaryType()` check in SparseTensorNewConverter. The same fix is applied here by adding a call to `isValidPrimaryType()` in SparseTensorEmptyConverter. Now the pass errors instead of crashing.

---
Full diff: https://github.com/llvm/llvm-project/pull/208913.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp (+2) 
- (modified) mlir/test/Dialect/SparseTensor/conversion_invalid.mlir (+23) 


``````````diff
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
index c9bcefe92de79..fb336c2f02f52 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
@@ -449,6 +449,8 @@ class SparseTensorEmptyConverter : public OpConversionPattern<tensor::EmptyOp> {
     const auto stt = getSparseTensorType(op);
     if (!stt.hasEncoding())
       return failure();
+    if (!isValidPrimaryType(stt.getElementType()))
+      return rewriter.notifyMatchFailure(op, "unsupported element type");
     // Gather all dimension sizes as SSA values.
     const Dimension dimRank = stt.getDimRank();
     SmallVector<Value> dimSizesValues;
diff --git a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
index 456c1898a9311..9166e4c3a0512 100644
--- a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
+++ b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
@@ -12,3 +12,26 @@ func.func @new_index_elem_type(%arg0: index) {
   %0 = sparse_tensor.new %arg0 : index to tensor<?xindex, #sparse>
   return
 }
+
+// -----
+
+#map = affine_map<(d0) -> (0, d0)>
+#map1 = affine_map<(d0) -> (d0)>
+#sparse = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+module {
+  func.func @main(%arg0: tensor<1x77xi1>, %arg1: tensor<1x77xi1>) -> tensor<77xi1, #sparse> {
+    %0 = tensor.empty() : tensor<77xi1, #sparse>
+    %1 = linalg.generic {indexing_maps = [#map, #map, #map1], iterator_types = ["parallel"]}
+      ins(%arg0, %arg1 : tensor<1x77xi1>, tensor<1x77xi1>)
+      outs(%0 : tensor<77xi1, #sparse>)
+    {
+    ^bb0(%in: i1, %in_0: i1, %out: i1):
+      %2 = arith.addi %in, %in_0 : i1
+      linalg.yield %2 : i1
+    } -> tensor<77xi1, #sparse>
+
+    // expected-error at +2 {{failed to legalize unresolved materialization}}
+    // expected-note at +1 {{see existing live user here}}
+    return %1 : tensor<77xi1, #sparse>
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Sat Jul 11 06:45:11 2026
From: llvmlistbot at llvm.org (Arjun Pitchanathan)
Date: Sat, 11 Jul 2026 06:45:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (PR #199911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199911@github.com>
Message-ID: <6a5248e7.f732ad81.3a4994.ff15@mx.google.com>


https://github.com/superty approved this pull request.


https://github.com/llvm/llvm-project/pull/199911

From llvmlistbot at llvm.org  Sat Jul 11 08:31:35 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 11 Jul 2026 08:31:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] SLM Privatization for WG-to-SG
	Distribution (PR #208921)
Message-ID: <llvm.org/llvm/llvm-project/pull/208921@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/208921

Adds a pre-phase to xegpu-wg-to-sg-distribute that demotes shared local memory (space 3) scratch buffers to subgroup-private memory (space 4, register-backed) when they're accessed identically by every subgroup.

When a buffer qualifies: all load_matrix/store_matrix on it share the same sg_layout, sg_data, data shape, and offsets; the
workgroup tile distributes evenly across subgroups (no broadcast/overlap); and the source alloca has exactly one create_mem_desc view (doesn't escape).  It flips the memref to space 4, shrinks the buffer +  mem_desc to the per-subgroup size, and re-indexes matrix ops with local, subgroup-id-free offsets.

>From c589918d9f4111e049d6165a56fb7fc2bfa6d344 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 8 Jul 2026 23:30:49 +0000
Subject: [PATCH 1/9] [MLIR][XeGPU] Prefer slice layout in LayoutInfo::meet

During backward layout propagation a value can receive different
candidate layouts from its consumers. Previously LayoutInfo::meet kept
whichever layout was assigned first, so a value feeding both a
slice-producing consumer (e.g. a broadcast whose inferred source layout
is a slice) and a plain-layout consumer (e.g. a dpas_mx scale operand)
could end up with the plain layout depending on visitation order.

meet() now prefers the slice layout when both sides are assigned
(falling back to lhs when both/neither are slices). For that preference
to take effect, operator== compares the stored layout (via isEqualTo)
rather than only the assigned bit; otherwise the dataflow framework
treats the refined result as a no-op and discards it. The refinement
only moves a plain layout toward a slice layout, so it stays monotonic.

Adds a propagate-layout-inst-data test and updates the a-operand
lane_data in the simple_mxfp_gemm integration test to match the HW
constraint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 25 +++++++++++-----
 .../XeGPU/propagate-layout-inst-data.mlir     | 29 +++++++++++++++++++
 .../Dialect/XeGPU/WG/simple_mxfp_gemm.mlir    |  2 +-
 3 files changed, 47 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 64d0d8063b7ff..ff9239b637a10 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -71,12 +71,13 @@ namespace {
 /// Given this, LayoutInfo  satisifies the following properties:
 ///  1) A LayoutInfo value can be in one of two states - `assigned` or `not
 ///  assigned`.
-///  2) Two LayoutInfo values are equal if they are both assigned or
-///  both not assigned. The concrete value of assigned state does not matter.
+///  2) Two LayoutInfo values are equal if they are both not assigned, or both
+///  assigned with the same layout.
 ///  3) The meet operator works as follows:
-///     - If current state is assigned, return the current state. (already
-///     a unique layout is assigned. don't change it)
-///     - Otherwise, return the other state.
+///     - If only one side is assigned, return that side.
+///     - If both sides are assigned, prefer the side carrying a slice layout.
+///       If both (or neither) are slice layouts, prefer the lhs (current
+///       state) so an already assigned unique layout is not changed.
 
 struct LayoutInfo {
 private:
@@ -86,10 +87,14 @@ struct LayoutInfo {
   LayoutInfo() = default;
   LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
 
-  // Two lattice values are equal if they have `some` layout. The actual
-  // content of the layout does not matter.
+  // Two lattice values are equal if they are both unassigned, or both assigned
+  // with the same layout.
   bool operator==(const LayoutInfo &other) const {
-    return this->isAssigned() == other.isAssigned();
+    if (isAssigned() != other.isAssigned())
+      return false;
+    if (!isAssigned())
+      return true;
+    return storage.isEqualTo(other.storage);
   }
 
   static LayoutInfo meet(const LayoutInfo &lhs, const LayoutInfo &rhs);
@@ -141,6 +146,10 @@ void LayoutInfo::print(raw_ostream &os) const {
 LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
   if (!lhs.isAssigned())
     return rhs;
+  if (!rhs.isAssigned())
+    return lhs;
+  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+    return rhs;
   return lhs;
 }
 
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 515c59db72819..6bf7fff810006 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -653,3 +653,32 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
   return
 }
 }
+
+// -----
+// A value with two consumers: one back-propagates a plain layout (the store_nd
+// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
+// chain feeding the reduction result). `meet` must prefer the slice layout, and
+// the inst_data / lane_layout / lane_data fields must be preserved on it.
+gpu.module @test {
+  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK: %[[TRUNC:.*]] = arith.truncf
+  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
+  // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+      %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
+    %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
+    %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>
+    %load = xegpu.load_nd %tdesc[0, 0] : !xegpu.tensor_desc<32x128xbf16> -> vector<32x128xbf16>
+    %sc1 = vector.shape_cast %load : vector<32x128xbf16> to vector<32x4x32xbf16>
+    %red = vector.multi_reduction <maximumf>, %sc1, %cst [2] : vector<32x4x32xbf16> to vector<32x4xbf16>
+    %trunc = arith.truncf %red : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
+    %bcast = vector.broadcast %trunc : vector<32x4xf8E8M0FNU> to vector<32x32x4xf8E8M0FNU>
+    %bcast2 = vector.transpose %bcast, [1, 2, 0] : vector<32x32x4xf8E8M0FNU> to vector<32x4x32xf8E8M0FNU>
+    %sc2 = vector.shape_cast %bcast2 : vector<32x4x32xf8E8M0FNU> to vector<32x128xf8E8M0FNU>
+    %tdesc_red = xegpu.create_nd_tdesc %dst_red : memref<32x128xf8E8M0FNU> -> !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    xegpu.store_nd %sc2, %tdesc_red[0, 0] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<32x128xf8E8M0FNU>, !xegpu.tensor_desc<32x128xf8E8M0FNU>
+    %tdesc_plain = xegpu.create_nd_tdesc %dst_plain : memref<32x4xf8E8M0FNU> -> !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    xegpu.store_nd %trunc, %tdesc_plain[0, 0] <{layout = #xegpu.layout<inst_data = [8, 2], lane_layout = [8, 1], lane_data = [1, 1]>}> : vector<32x4xf8E8M0FNU>, !xegpu.tensor_desc<32x4xf8E8M0FNU>
+    gpu.return
+  }
+}
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
index f584ede003bd4..9bfb783299055 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm.mlir
@@ -8,7 +8,7 @@
 
 // XFAIL: *
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>
 #c = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 16], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>

>From c31ad5f6b4f3d97c50c91e62737b39cd07a23ecf Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:55:14 +0000
Subject: [PATCH 2/9] [MLIR][XeGPU] Prefer nearer consumer's layout in
 LayoutInfo::meet

When a value is demanded by multiple consumers during backward layout
propagation, prefer the layout of the consumer that is nearer to the
producer in program order. This tends to preserve a consumer's layout as
far up the def chain as possible, reducing layout conversions. It is a
hint, not an optimum.

Each op is assigned a program-order index via a pre-order walk (matching
printed-IR order), so a use inside an scf.for body is nearer than a use
after the loop. LayoutInfo carries the demanding op's index in a new
`programOrder` field, stamped by the single-argument constructor from a
file-scoped `currentProgramOrder` that visitOperation sets, so the ~30
meet() call sites are unchanged. `programOrder` is not propagated (each
visited op stamps its own index) and is excluded from operator==.

This replaces the earlier slice-layout preference in meet(), which is
now redundant: distinct users always have distinct indices, so program
order decides every real conflict. Also removes the dead
LayoutInfo::transpose declaration.

Updates the multiple-uses propagate-layout tests to reflect that the
nearer consumer's layout now wins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 75 +++++++++++++++++--
 mlir/test/Dialect/XeGPU/propagate-layout.mlir | 20 ++---
 2 files changed, 77 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index ff9239b637a10..72b10e1d3265f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -30,6 +30,7 @@
 #include "mlir/Interfaces/LoopLikeInterface.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/SmallVector.h"
@@ -38,6 +39,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/LogicalResult.h"
 #include "llvm/Support/raw_ostream.h"
+#include <limits>
 
 namespace mlir {
 namespace xegpu {
@@ -58,6 +60,14 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
+/// Program-order index of the op currently being visited by the backward
+/// analysis. `visitOperation` sets this before dispatching, and the
+/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
+/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
+/// change. A larger index means farther from the producer; the sentinel max
+/// marks demands with no associated op (e.g. exit state).
+static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
+
 /// Helper class for tracking the analysis state of an mlir value. For layout
 /// propagation, the analysis state is simply the distribution layout of
 /// each value. The distribution layout information is encapsulated using
@@ -75,20 +85,34 @@ namespace {
 ///  assigned with the same layout.
 ///  3) The meet operator works as follows:
 ///     - If only one side is assigned, return that side.
-///     - If both sides are assigned, prefer the side carrying a slice layout.
-///       If both (or neither) are slice layouts, prefer the lhs (current
-///       state) so an already assigned unique layout is not changed.
+///     - If both sides are assigned, prefer the layout demanded by the user
+///       that is nearer to the producer in program order (smaller
+///       `programOrder`); on a tie keep lhs.
+///
+/// The `programOrder` field records the program-order index of the consumer op
+/// that demanded the layout (see `currentProgramOrder`). During this backward
+/// analysis a value can be demanded by several users; keeping the nearest one
+/// tends to preserve a consumer's layout as far up the def chain as possible,
+/// minimizing layout conversions. This is a hint, not an optimum.
+/// `programOrder` is never propagated up the chain - each visited op stamps its
+/// own index - so it is deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
   xegpu::DistributeLayoutAttr storage = nullptr;
+  // Program-order index of the consumer op that demanded this layout. Smaller
+  // means nearer to the producer. Unassigned/unknown demands sort last.
+  int64_t programOrder = std::numeric_limits<int64_t>::max();
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout) : storage(layout) {}
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
+  LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
+      : storage(layout), programOrder(programOrder) {}
 
   // Two lattice values are equal if they are both unassigned, or both assigned
-  // with the same layout.
+  // with the same layout. `programOrder` is intentionally excluded: it is not
+  // propagated, so a pure order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
     if (isAssigned() != other.isAssigned())
       return false;
@@ -105,8 +129,6 @@ struct LayoutInfo {
 
   bool isAssigned() const { return storage != nullptr; }
 
-  LayoutInfo transpose(ArrayRef<int64_t> permutation) const;
-
   SmallVector<int> getLaneLayout() const;
 
   SmallVector<int> getLaneData() const;
@@ -135,6 +157,11 @@ struct LayoutInfo {
   void set(const xegpu::DistributeLayoutAttr &layout) { storage = layout; }
 };
 
+// Stamp every demand pushed by the current op with that op's program-order
+// index so `meet` can prefer the nearest consumer.
+LayoutInfo::LayoutInfo(const xegpu::DistributeLayoutAttr &layout)
+    : storage(layout), programOrder(currentProgramOrder) {}
+
 void LayoutInfo::print(raw_ostream &os) const {
   if (isAssigned()) {
     os << storage;
@@ -148,7 +175,10 @@ LayoutInfo LayoutInfo::meet(const LayoutInfo &lhs, const LayoutInfo &rhs) {
     return rhs;
   if (!rhs.isAssigned())
     return lhs;
-  if (!lhs.isSliceLayout() && rhs.isSliceLayout())
+  // Prefer the demand from the user nearer to the producer in program order.
+  // Distinct users always have distinct indices, so this decides every
+  // real conflict; on a tie (same op, or both unknown) keep lhs.
+  if (rhs.programOrder < lhs.programOrder)
     return rhs;
   return lhs;
 }
@@ -185,6 +215,15 @@ class LayoutInfoPropagation
 private:
   xegpu::LayoutKind layoutKind;
   unsigned indexBitWidth;
+
+  // Program-order index of every op, built lazily on first use via a pre-order
+  // walk of the top-level module/function (matching printed-IR order). Used to
+  // tell which consumer of a value is nearer to its producer.
+  DenseMap<Operation *, int64_t> programOrder;
+  // Returns the program-order index of `op`, populating `programOrder` from
+  // `op`'s top-level ancestor on first call.
+  int64_t getProgramOrder(Operation *op);
+
   void visitDpasOp(xegpu::DpasOp dpas, ArrayRef<LayoutInfoLattice *> operands,
                    ArrayRef<const LayoutInfoLattice *> results);
 
@@ -300,9 +339,29 @@ class LayoutInfoPropagation
 };
 } // namespace
 
+int64_t LayoutInfoPropagation::getProgramOrder(Operation *op) {
+  auto it = programOrder.find(op);
+  if (it != programOrder.end())
+    return it->second;
+  // First time we see this op's tree: number every op under its top-level
+  // ancestor in pre-order (i.e. printed-IR order). Nested ops (e.g. inside an
+  // scf.for body) get an index between their parent and the parent's next
+  // sibling, so a use inside a loop is "nearer" than a use after it.
+  Operation *root = op;
+  while (root->getParentOp())
+    root = root->getParentOp();
+  int64_t counter = 0;
+  root->walk<WalkOrder::PreOrder>(
+      [&](Operation *o) { programOrder[o] = counter++; });
+  return programOrder.lookup(op);
+}
+
 LogicalResult LayoutInfoPropagation::visitOperation(
     Operation *op, ArrayRef<LayoutInfoLattice *> operands,
     ArrayRef<const LayoutInfoLattice *> results) {
+  // Stamp demands pushed by this op with its program-order index so `meet` can
+  // prefer the nearest consumer.
+  currentProgramOrder = getProgramOrder(op);
   TypeSwitch<Operation *>(op)
       .Case(
           [&](xegpu::DpasOp dpasOp) { visitDpasOp(dpasOp, operands, results); })
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 1a741ba21e128..49006e1ef9507 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -219,8 +219,8 @@ func.func @scatter_ops_custom_perm_layout(%src: memref<256xf16>) {
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_preserve_load_perm_layout(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<256xf16>) {
-// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<true> : vector<16xi1>
-// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} dense<12> : vector<16xindex>
+// CHECK: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<true> : vector<16xi1>
+// CHECK: %[[OFFSETS:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [1]>} dense<12> : vector<16xindex>
 // CHECK: %[[LOAD_VEC:.*]] = xegpu.load %[[ARG0]][%[[OFFSETS]]], %[[MASK]]
 // CHECK-SAME: memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16xf16>
 // CHECK: %[[ADD_RES:.*]] = arith.addf %[[LOAD_VEC]], %[[LOAD_VEC]] {layout_result_0 = #xegpu.layout<lane_layout = [8], lane_data = [1]>} : vector<16xf16>
@@ -341,10 +341,10 @@ func.func @binary_op_one_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.
 gpu.module @test {
 // CHECK-LABEL: func.func @binary_op_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
-// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf16>
+// CHECK: %[[T2:.*]] = arith.addf %{{.*}}, %{{.*}} {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>} : vector<16x16xf16>
 // CHECK: %[[T3:.*]] = xegpu.dpas %{{.*}}, %[[T2]] {layout_a = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, layout_b = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>, layout_cd = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
 // CHECK-NEXT: xegpu.store_nd %[[T3]], %[[ARG2]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x16xf32>, !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
 // CHECK-NEXT: xegpu.store_nd %[[T2]], %[[ARG3]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>, !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>
@@ -429,18 +429,18 @@ func.func @if_single_use(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tens
 gpu.module @test {
 // CHECK-LABEL: func.func @if_multiple_uses(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
-// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
+// CHECK-SAME: %[[ARG1:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>>,
 // CHECK-SAME: %[[ARG2:[0-9a-zA-Z]+]]: i1, %[[ARG3:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<8x16xf32, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>,
 // CHECK-SAME: %[[ARG4:[0-9a-zA-Z]+]]: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>>) {
 // CHECK: %[[T1:.*]] = scf.if %[[ARG2]] -> (vector<16x16xf16>) {
-// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T3:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T3]] : vector<16x16xf16>
 // CHECK-NEXT:     } else {
-// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> :
-// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<16x16xf16>
+// CHECK-NEXT:       %[[T4:.*]] = xegpu.load_nd %[[ARG1]][0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}> :
+// CHECK-SAME: !xegpu.tensor_desc<16x16xf16, #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>> -> vector<16x16xf16>
 // CHECK-NEXT:       scf.yield %[[T4]] : vector<16x16xf16>
-// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}
+// CHECK-NEXT:     } {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>}
 func.func @if_multiple_uses(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor_desc<16x16xf16>, %arg2: i1, %arg3: !xegpu.tensor_desc<8x16xf32>, %arg4: !xegpu.tensor_desc<16x16xf16>) {
   %0 = xegpu.load_nd %arg0[0, 0]  : !xegpu.tensor_desc<8x16xf16> -> vector<8x16xf16>
   %1 = scf.if %arg2 -> (vector<16x16xf16>) {

>From 3c4ada8b1f50d0d9c48f3c8543aa0662f1f2164c Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 01:58:24 +0000
Subject: [PATCH 3/9] [MLIR][XeGPU] Reframe propagate-layout test as
 nearer-user preference

Rename truncf_prefers_slice to truncf_prefers_nearer_user and update its
comment: on this branch %trunc keeps the broadcast chain's slice layout
because that consumer is nearer in program order, not because meet()
prefers slice layouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../Dialect/XeGPU/propagate-layout-inst-data.mlir    | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 6bf7fff810006..47471b890d552 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -655,16 +655,16 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
 }
 
 // -----
-// A value with two consumers: one back-propagates a plain layout (the store_nd
-// of %trunc), the other back-propagates a slice layout (the broadcast/transpose
-// chain feeding the reduction result). `meet` must prefer the slice layout, and
-// the inst_data / lane_layout / lane_data fields must be preserved on it.
+// %trunc has two consumers: the broadcast/transpose chain (nearer in program
+// order, back-propagates a slice layout) and the store_nd of %trunc (farther,
+// back-propagates a plain layout). `meet` must keep the nearer consumer's
+// layout, preserving its inst_data / lane_layout / lane_data fields.
 gpu.module @test {
-  // CHECK-LABEL: truncf_prefers_slice
+  // CHECK-LABEL: truncf_prefers_nearer_user
   // CHECK: %[[TRUNC:.*]] = arith.truncf
   // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
   // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
-  gpu.func @truncf_prefers_slice(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
+  gpu.func @truncf_prefers_nearer_user(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
       %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
     %cst = arith.constant dense<0xFF80> : vector<32x4xbf16>
     %tdesc = xegpu.create_nd_tdesc %src : memref<32x128xbf16> -> !xegpu.tensor_desc<32x128xbf16>

>From 9c4de99b6094f502c82dcd411b8eca2c1c18141d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 02:00:09 +0000
Subject: [PATCH 4/9] git format

---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp      | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 72b10e1d3265f..75e8db468f361 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override {};
+  void visitBranchOperand(OpOperand &operand) override{};
 
-  void visitCallOperand(OpOperand &operand) override {};
+  void visitCallOperand(OpOperand &operand) override{};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override {};
+                               ArrayRef<BlockArgument> arguments) override{};
 
-  void visitExternalCall(CallOpInterface call,
-                         ArrayRef<LayoutInfoLattice *> operands,
-                         ArrayRef<const LayoutInfoLattice *> results) override {
-  };
+  void
+  visitExternalCall(CallOpInterface call,
+                    ArrayRef<LayoutInfoLattice *> operands,
+                    ArrayRef<const LayoutInfoLattice *> results) override{};
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());

>From 1db9aa8f8f3ccd82194b591fc3f4754a056c416d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Thu, 9 Jul 2026 02:11:28 +0000
Subject: [PATCH 5/9] [MLIR][XeGPU] clang-format XeGPUPropagateLayout.cpp

Apply clang-format fixes flagged by CI (override{} -> override {}) on the
empty visitor overrides touched by this change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp      | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 75e8db468f361..72b10e1d3265f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -320,18 +320,18 @@ class LayoutInfoPropagation
   visitOperation(Operation *op, ArrayRef<LayoutInfoLattice *> operands,
                  ArrayRef<const LayoutInfoLattice *> results) override;
 
-  void visitBranchOperand(OpOperand &operand) override{};
+  void visitBranchOperand(OpOperand &operand) override {};
 
-  void visitCallOperand(OpOperand &operand) override{};
+  void visitCallOperand(OpOperand &operand) override {};
 
   void
   visitNonControlFlowArguments(RegionSuccessor &successor,
-                               ArrayRef<BlockArgument> arguments) override{};
+                               ArrayRef<BlockArgument> arguments) override {};
 
-  void
-  visitExternalCall(CallOpInterface call,
-                    ArrayRef<LayoutInfoLattice *> operands,
-                    ArrayRef<const LayoutInfoLattice *> results) override{};
+  void visitExternalCall(CallOpInterface call,
+                         ArrayRef<LayoutInfoLattice *> operands,
+                         ArrayRef<const LayoutInfoLattice *> results) override {
+  };
 
   void setToExitState(LayoutInfoLattice *lattice) override {
     (void)lattice->meet(LayoutInfo());

>From a6ee4af226d56c74c108d08ab300d5aa318308e9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 19:24:55 +0000
Subject: [PATCH 6/9] [MLIR][XeGPU] Address review: replace global
 currentProgramOrder with member helper

Move currentProgramOrder from a file-scoped global into LayoutInfoPropagation
and add a private makeLayoutInfo helper that stamps it, so the LayoutInfo
constructor no longer depends on implicit global mutable state. Rename the
single-arg LayoutInfo(...) call sites to makeLayoutInfo(...) and drop the now
unused single-arg constructor. Also fix the stale operator== comment to
describe each branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp | 194 ++++++++----------
 1 file changed, 82 insertions(+), 112 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index a5618b5f84ed3..c73749128906b 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -60,14 +60,6 @@ namespace {
 // LayoutInfo
 //===----------------------------------------------------------------------===//
 
-/// Program-order index of the op currently being visited by the backward
-/// analysis. `visitOperation` sets this before dispatching, and the
-/// single-argument LayoutInfo constructor stamps it onto every demand pushed to
-/// an operand, so the ~30 `operand->meet(LayoutInfo(...))` call sites need no
-/// change. A larger index means farther from the producer; the sentinel max
-/// marks demands with no associated op (e.g. exit state).
-static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
-
 /// Helper class for tracking the analysis state of an mlir value. For layout
 /// propagation, the analysis state is simply the distribution layout of
 /// each value. The distribution layout information is encapsulated using
@@ -90,12 +82,14 @@ static int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
 ///       `programOrder`); on a tie keep lhs.
 ///
 /// The `programOrder` field records the program-order index of the consumer op
-/// that demanded the layout (see `currentProgramOrder`). During this backward
-/// analysis a value can be demanded by several users; keeping the nearest one
-/// tends to preserve a consumer's layout as far up the def chain as possible,
-/// minimizing layout conversions. This is a hint, not an optimum.
-/// `programOrder` is never propagated up the chain - each visited op stamps its
-/// own index - so it is deliberately excluded from `operator==`.
+/// that demanded the layout (stamped via
+/// `LayoutInfoPropagation::makeLayoutInfo` from
+/// `LayoutInfoPropagation::currentProgramOrder`). During this backward analysis
+/// a value can be demanded by several users; keeping the nearest one tends to
+/// preserve a consumer's layout as far up the def chain as possible, minimizing
+/// layout conversions. This is a hint, not an optimum. `programOrder` is never
+/// propagated up the chain - each visited op stamps its own index - so it is
+/// deliberately excluded from `operator==`.
 
 struct LayoutInfo {
 private:
@@ -106,13 +100,15 @@ struct LayoutInfo {
 
 public:
   LayoutInfo() = default;
-  LayoutInfo(const xegpu::DistributeLayoutAttr &layout);
   LayoutInfo(const xegpu::DistributeLayoutAttr &layout, int64_t programOrder)
       : storage(layout), programOrder(programOrder) {}
 
-  // Two lattice values are equal if they are both unassigned, or both assigned
-  // with the same layout. `programOrder` is intentionally excluded: it is not
-  // propagated, so a pure order refinement must not be reported as a change.
+  // Equality by assignment state and, when both assigned, by the layout:
+  //  - one assigned, the other not -> not equal;
+  //  - both unassigned             -> equal;
+  //  - both assigned               -> equal iff the layouts match.
+  // `programOrder` is intentionally excluded: it is not propagated, so a pure
+  // order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
     if (isAssigned() != other.isAssigned())
       return false;
@@ -157,11 +153,6 @@ struct LayoutInfo {
   void set(const xegpu::DistributeLayoutAttr &layout) { storage = layout; }
 };
 
-// Stamp every demand pushed by the current op with that op's program-order
-// index so `meet` can prefer the nearest consumer.
-LayoutInfo::LayoutInfo(const xegpu::DistributeLayoutAttr &layout)
-    : storage(layout), programOrder(currentProgramOrder) {}
-
 void LayoutInfo::print(raw_ostream &os) const {
   if (isAssigned()) {
     os << storage;
@@ -224,6 +215,11 @@ class LayoutInfoPropagation
   // `op`'s top-level ancestor on first call.
   int64_t getProgramOrder(Operation *op);
 
+  int64_t currentProgramOrder = std::numeric_limits<int64_t>::max();
+  LayoutInfo makeLayoutInfo(const xegpu::DistributeLayoutAttr &layout) {
+    return LayoutInfo(layout, currentProgramOrder);
+  }
+
   void visitDpasOp(xegpu::DpasOp dpas, ArrayRef<LayoutInfoLattice *> operands,
                    ArrayRef<const LayoutInfoLattice *> results);
 
@@ -459,42 +455,6 @@ bool LayoutInfoPropagation::hasParamsOfLayoutKind(
   return false;
 }
 
-// This function returns all layouts for the given sgCount, whose sgData:
-// 1. Evenly divides the wgShape.
-// 2. Is a multiple of instData.
-// Example:
-//   wgShape = [128, 64], instData = [8, 16], sgCount = 32
-// Returns layouts:
-//   [(8,4), (16,2)], which correspond to sgData [16,16] and [8,32].
-SmallVector<std::pair<int, int>>
-getSgLayoutCandidates(ArrayRef<int64_t> wgShape, ArrayRef<int> instData,
-                      int64_t sgCount) {
-  SmallVector<std::pair<int, int>> candidates;
-  for (int sgLayout0 = 1; sgLayout0 <= sgCount; ++sgLayout0) {
-    if (sgCount % sgLayout0)
-      continue;
-    int sgLayout1 = sgCount / sgLayout0;
-    int sgData0 = wgShape[0] / sgLayout0;
-    int sgData1 = wgShape[1] / sgLayout1;
-    if ((wgShape[0] % sgLayout0 || wgShape[1] % sgLayout1) ||
-        (sgData0 % instData[0] || sgData1 % instData[1]))
-      continue;
-    candidates.emplace_back(sgLayout0, sgLayout1);
-  }
-  // Sort primarily by how balanced they are
-  // (i.e., minimize the absolute difference between the two dimensions), and
-  // secondarily by the first dimension in ascending order.
-  llvm::sort(candidates, [](const std::pair<int, int> &lhs,
-                            const std::pair<int, int> &rhs) {
-    int diffLhs = std::abs(lhs.first - lhs.second);
-    int diffRhs = std::abs(rhs.first - rhs.second);
-    if (diffLhs != diffRhs)
-      return diffLhs < diffRhs;
-    return lhs.first < rhs.first;
-  });
-  return candidates;
-}
-
 FailureOr<int64_t>
 getNumSg(Operation *op, const int sgSize,
          xegpu::DistributeLayoutAttr consumerLayout = nullptr) {
@@ -525,7 +485,7 @@ void LayoutInfoPropagation::visitPrefetchNdOp(
     return;
   xegpu::DistributeLayoutAttr anchorLayout = prefetch.getLayoutAttr();
   if (hasParamsOfLayoutKind(anchorLayout)) {
-    prefetchLayout = LayoutInfo(anchorLayout);
+    prefetchLayout = makeLayoutInfo(anchorLayout);
     if (layoutKind == xegpu::LayoutKind::InstData) {
       const auto *uArchInstruction =
           dyn_cast<xegpu::uArch::Subgroup2DBlockPrefetchInstruction>(
@@ -542,7 +502,7 @@ void LayoutInfoPropagation::visitPrefetchNdOp(
         return;
       }
       prefetch.setLayoutAttr(*completed);
-      prefetchLayout = LayoutInfo(*completed);
+      prefetchLayout = makeLayoutInfo(*completed);
     }
   } else {
     auto tdescTy = prefetch.getTensorDescType();
@@ -560,7 +520,7 @@ void LayoutInfoPropagation::visitPrefetchNdOp(
           "Failed to determine required layout for prefetch_nd.");
       return;
     }
-    prefetchLayout = LayoutInfo(layoutAttr);
+    prefetchLayout = makeLayoutInfo(layoutAttr);
     prefetch.setLayoutAttr(layoutAttr);
   }
   // Propagate the layout to the source tensor descriptor.
@@ -614,10 +574,11 @@ void LayoutInfoPropagation::visitVectorMultiReductionOp(
   auto srcLayoutAttr = xegpu::inferMultiReductionSourceLayout(
       requiredResLayoutAttr, reductionDims);
 
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
   // Accumulator should have the same layout as the result.
   propagateIfChanged(operands[1],
-                     operands[1]->meet(LayoutInfo(requiredResLayoutAttr)));
+                     operands[1]->meet(makeLayoutInfo(requiredResLayoutAttr)));
 }
 
 void LayoutInfoPropagation::visitVectorReductionOp(
@@ -635,10 +596,11 @@ void LayoutInfoPropagation::visitVectorReductionOp(
   xegpu::setTemporaryLayout(reduction->getResult(0), requiredResLayoutAttr);
 
   auto srcLayoutAttr = xegpu::inferReductionSourceLayout(requiredResLayoutAttr);
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
   if (reduction.getAcc())
-    propagateIfChanged(operands[1],
-                       operands[1]->meet(LayoutInfo(requiredResLayoutAttr)));
+    propagateIfChanged(
+        operands[1], operands[1]->meet(makeLayoutInfo(requiredResLayoutAttr)));
 }
 
 void LayoutInfoPropagation::visitVectorBroadCastOp(
@@ -665,7 +627,8 @@ void LayoutInfoPropagation::visitVectorBroadCastOp(
   xegpu::DistributeLayoutAttr srcLayoutAttr =
       xegpu::inferBroadcastSourceLayout(resultLayoutAttr, resShape, srcShape);
 
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 void LayoutInfoPropagation::visitShapeCastOp(
@@ -690,7 +653,8 @@ void LayoutInfoPropagation::visitShapeCastOp(
     return;
   }
 
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 /// Set the layouts for DPAS A, B, and C operands.
@@ -716,9 +680,9 @@ void LayoutInfoPropagation::visitDpasOp(
            "Expected anchor layout for DPAS A operand.");
     assert(hasParamsOfLayoutKind(anchorLayoutB) &&
            "Expected anchor layout for DPAS B operand.");
-    dpasALayout = LayoutInfo(anchorLayoutA);
-    dpasBLayout = LayoutInfo(anchorLayoutB);
-    dpasCDLayout = LayoutInfo(anchorLayoutCD);
+    dpasALayout = makeLayoutInfo(anchorLayoutA);
+    dpasBLayout = makeLayoutInfo(anchorLayoutB);
+    dpasCDLayout = makeLayoutInfo(anchorLayoutCD);
     if (layoutKind == xegpu::LayoutKind::InstData) {
       auto completed = xegpu::completeDpasLaneLayoutFromInstData(
           anchorLayoutA, anchorLayoutB, anchorLayoutCD, aTy, bTy, cdTy, uArch);
@@ -731,9 +695,9 @@ void LayoutInfoPropagation::visitDpasOp(
       dpas.setLayoutAAttr(completedA);
       dpas.setLayoutBAttr(completedB);
       dpas.setLayoutCdAttr(completedCD);
-      dpasALayout = LayoutInfo(completedA);
-      dpasBLayout = LayoutInfo(completedB);
-      dpasCDLayout = LayoutInfo(completedCD);
+      dpasALayout = makeLayoutInfo(completedA);
+      dpasBLayout = makeLayoutInfo(completedB);
+      dpasCDLayout = makeLayoutInfo(completedCD);
     }
   } else {
 
@@ -769,9 +733,9 @@ void LayoutInfoPropagation::visitDpasOp(
     dpas.setLayoutAAttr(requiredALayout);
     dpas.setLayoutBAttr(requiredBLayout);
     dpas.setLayoutCdAttr(requiredCDLayoutAttr);
-    dpasALayout = LayoutInfo(requiredALayout);
-    dpasBLayout = LayoutInfo(requiredBLayout);
-    dpasCDLayout = LayoutInfo(requiredCDLayoutAttr);
+    dpasALayout = makeLayoutInfo(requiredALayout);
+    dpasBLayout = makeLayoutInfo(requiredBLayout);
+    dpasCDLayout = makeLayoutInfo(requiredCDLayoutAttr);
   }
   propagateIfChanged(operands[0], operands[0]->meet(dpasALayout));
   propagateIfChanged(operands[1], operands[1]->meet(dpasBLayout));
@@ -818,9 +782,9 @@ void LayoutInfoPropagation::visitDpasMxOp(
       hasParamsOfLayoutKind(anchorLayoutA) &&
       hasParamsOfLayoutKind(anchorLayoutB) &&
       hasParamsOfLayoutKind(anchorLayoutCD)) {
-    dpasMxALayout = LayoutInfo(anchorLayoutA);
-    dpasMxBLayout = LayoutInfo(anchorLayoutB);
-    dpasMxCDLayout = LayoutInfo(anchorLayoutCD);
+    dpasMxALayout = makeLayoutInfo(anchorLayoutA);
+    dpasMxBLayout = makeLayoutInfo(anchorLayoutB);
+    dpasMxCDLayout = makeLayoutInfo(anchorLayoutCD);
 
     // Get scale layouts if available
     xegpu::DistributeLayoutAttr anchorLayoutAScale =
@@ -828,9 +792,9 @@ void LayoutInfoPropagation::visitDpasMxOp(
     xegpu::DistributeLayoutAttr anchorLayoutBScale =
         dpasMx.getLayoutBScaleAttr();
     if (anchorLayoutAScale)
-      dpasMxAScaleLayout = LayoutInfo(anchorLayoutAScale);
+      dpasMxAScaleLayout = makeLayoutInfo(anchorLayoutAScale);
     if (anchorLayoutBScale)
-      dpasMxBScaleLayout = LayoutInfo(anchorLayoutBScale);
+      dpasMxBScaleLayout = makeLayoutInfo(anchorLayoutBScale);
 
     if (layoutKind == xegpu::LayoutKind::InstData) {
       auto completed = xegpu::completeDpasMxLaneLayoutFromInstData(
@@ -846,16 +810,16 @@ void LayoutInfoPropagation::visitDpasMxOp(
       dpasMx.setLayoutAAttr(completedA);
       dpasMx.setLayoutBAttr(completedB);
       dpasMx.setLayoutCdAttr(completedCD);
-      dpasMxALayout = LayoutInfo(completedA);
-      dpasMxBLayout = LayoutInfo(completedB);
-      dpasMxCDLayout = LayoutInfo(completedCD);
+      dpasMxALayout = makeLayoutInfo(completedA);
+      dpasMxBLayout = makeLayoutInfo(completedB);
+      dpasMxCDLayout = makeLayoutInfo(completedCD);
       if (completedAScale) {
         dpasMx.setLayoutAScaleAttr(completedAScale);
-        dpasMxAScaleLayout = LayoutInfo(completedAScale);
+        dpasMxAScaleLayout = makeLayoutInfo(completedAScale);
       }
       if (completedBScale) {
         dpasMx.setLayoutBScaleAttr(completedBScale);
-        dpasMxBScaleLayout = LayoutInfo(completedBScale);
+        dpasMxBScaleLayout = makeLayoutInfo(completedBScale);
       }
     }
   } else {
@@ -897,13 +861,13 @@ void LayoutInfoPropagation::visitDpasMxOp(
     if (requiredBScaleLayout)
       dpasMx.setLayoutBScaleAttr(requiredBScaleLayout);
 
-    dpasMxALayout = LayoutInfo(requiredALayout);
-    dpasMxBLayout = LayoutInfo(requiredBLayout);
-    dpasMxCDLayout = LayoutInfo(requiredCDLayoutAttr);
+    dpasMxALayout = makeLayoutInfo(requiredALayout);
+    dpasMxBLayout = makeLayoutInfo(requiredBLayout);
+    dpasMxCDLayout = makeLayoutInfo(requiredCDLayoutAttr);
     if (requiredAScaleLayout)
-      dpasMxAScaleLayout = LayoutInfo(requiredAScaleLayout);
+      dpasMxAScaleLayout = makeLayoutInfo(requiredAScaleLayout);
     if (requiredBScaleLayout)
-      dpasMxBScaleLayout = LayoutInfo(requiredBScaleLayout);
+      dpasMxBScaleLayout = makeLayoutInfo(requiredBScaleLayout);
   }
 
   // Propagate layouts to operands. Because acc, scale_a, scale_b are all
@@ -941,7 +905,7 @@ void LayoutInfoPropagation::visitStoreNdOp(
     return;
   xegpu::DistributeLayoutAttr anchorLayout = store.getLayoutAttr();
   if (hasParamsOfLayoutKind(anchorLayout)) {
-    storeLayout = LayoutInfo(anchorLayout);
+    storeLayout = makeLayoutInfo(anchorLayout);
     if (layoutKind == xegpu::LayoutKind::InstData) {
 
       const auto *uArchInstruction =
@@ -959,7 +923,7 @@ void LayoutInfoPropagation::visitStoreNdOp(
         return;
       }
       store.setLayoutAttr(*completed);
-      storeLayout = LayoutInfo(*completed);
+      storeLayout = makeLayoutInfo(*completed);
     }
   } else {
     auto numSgOrErr = getNumSg(store, uArch->getSubgroupSize());
@@ -975,7 +939,7 @@ void LayoutInfoPropagation::visitStoreNdOp(
       store.emitWarning("Failed to determine required layout for store_nd.");
       return;
     }
-    storeLayout = LayoutInfo(layoutAttr);
+    storeLayout = makeLayoutInfo(layoutAttr);
     store.setLayoutAttr(layoutAttr);
   }
   // Propagate the layout to the value operand.
@@ -1001,7 +965,7 @@ void LayoutInfoPropagation::visitLoadNdOp(
       dyn_cast<xegpu::DistributeLayoutAttr>(valueLayout.get());
   xegpu::DistributeLayoutAttr anchorLayout = load.getLayoutAttr();
   if (hasParamsOfLayoutKind(anchorLayout)) {
-    loadLayout = LayoutInfo(anchorLayout);
+    loadLayout = makeLayoutInfo(anchorLayout);
     if (layoutKind == xegpu::LayoutKind::InstData &&
         !consumerLayoutAttr.getEffectiveLaneLayoutAsInt().empty()) {
       const auto *uArchInstruction =
@@ -1019,7 +983,7 @@ void LayoutInfoPropagation::visitLoadNdOp(
         return;
       }
       load.setLayoutAttr(*completed);
-      loadLayout = LayoutInfo(*completed);
+      loadLayout = makeLayoutInfo(*completed);
     }
   } else {
     auto numSgOrErr =
@@ -1036,7 +1000,7 @@ void LayoutInfoPropagation::visitLoadNdOp(
       load.emitWarning("Failed to determine required layout for load_nd.");
       return;
     }
-    loadLayout = LayoutInfo(layoutAttr);
+    loadLayout = makeLayoutInfo(layoutAttr);
     load.setLayoutAttr(layoutAttr);
   }
   // Propagate the new layout to the tensor descriptor operand.
@@ -1092,7 +1056,7 @@ void LayoutInfoPropagation::visitConvertLayoutOp(
   }
 
   xegpu::DistributeLayoutAttr anchorLayout = convert.getInputLayoutAttr();
-  LayoutInfo convertLayout(anchorLayout);
+  LayoutInfo convertLayout = makeLayoutInfo(anchorLayout);
   // Propagate the new layout to the tensor descriptor operand.
   propagateIfChanged(operands[0], operands[0]->meet(convertLayout));
 }
@@ -1113,7 +1077,8 @@ void LayoutInfoPropagation::visitTransposeOp(
       consumerLayoutAttr, transpose.getPermutation());
 
   // Propagate the new layout to the vector operand.
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 /// For vector::BitCastOp, the lane_data of the source layout is changed based
@@ -1147,7 +1112,8 @@ void LayoutInfoPropagation::visitVectorBitcastOp(
   auto srcLayoutAttr = xegpu::inferBitCastSourceLayout(
       requiredResLayoutAttr, outElemTyBitWidth, inElemTyBitWidth);
 
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 /// For vector::InterleaveOp, the result has double the innermost dimension
@@ -1182,8 +1148,10 @@ void LayoutInfoPropagation::visitVectorInterleaveOp(
       xegpu::inferInterleaveSourceLayout(requiredResLayoutAttr);
 
   // Both operands (lhs and rhs) get the same source layout
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
-  propagateIfChanged(operands[1], operands[1]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[1],
+                     operands[1]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 /// For vector::DeinterleaveOp, the source has double the innermost dimension
@@ -1205,7 +1173,8 @@ void LayoutInfoPropagation::visitVectorDeinterleaveOp(
   // dim) No setup function needed - just infer directly
   auto srcLayoutAttr = xegpu::inferDeinterleaveSourceLayout(consumerLayoutAttr);
 
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
 }
 
 void LayoutInfoPropagation::visitInsertStridedSliceOp(
@@ -1234,9 +1203,10 @@ void LayoutInfoPropagation::visitInsertStridedSliceOp(
 
   auto srcLayoutAttr = xegpu::inferInsertStridedSliceSourceLayout(
       requiredResLayoutAttr, resVecType.getShape(), srcVecType.getShape());
-  propagateIfChanged(operands[0], operands[0]->meet(LayoutInfo(srcLayoutAttr)));
+  propagateIfChanged(operands[0],
+                     operands[0]->meet(makeLayoutInfo(srcLayoutAttr)));
   propagateIfChanged(operands[1],
-                     operands[1]->meet(LayoutInfo(requiredResLayoutAttr)));
+                     operands[1]->meet(makeLayoutInfo(requiredResLayoutAttr)));
 }
 
 /// Propagate the layout of the result to the tensor descriptor, mask and
@@ -1291,8 +1261,8 @@ void LayoutInfoPropagation::visitLoadGatherOp(
   assert((chunkSize <= 1) || (layoutKind != xegpu::LayoutKind::Subgroup));
   auto maskLayoutAttr = xegpu::inferMaskOffsetLayoutForScatterIO(
       requiredAnchorLayoutAttr, chunkSize);
-  LayoutInfo maskLayoutInfo = LayoutInfo(maskLayoutAttr);
-  auto loadLayoutInfo = LayoutInfo(requiredAnchorLayoutAttr);
+  LayoutInfo maskLayoutInfo = makeLayoutInfo(maskLayoutAttr);
+  auto loadLayoutInfo = makeLayoutInfo(requiredAnchorLayoutAttr);
 
   // Propagate the new layout to the tensor descriptor operand.
   if (isa<xegpu::TensorDescType>(load.getSourceType()))
@@ -1357,11 +1327,11 @@ void LayoutInfoPropagation::visitStoreScatterOp(
     storeScatter.setLayoutAttr(requiredAnchorLayoutAttr);
   }
 
-  LayoutInfo srcLayoutInfo = LayoutInfo(requiredAnchorLayoutAttr);
+  LayoutInfo srcLayoutInfo = makeLayoutInfo(requiredAnchorLayoutAttr);
   assert((chunkSize <= 1) || (layoutKind != xegpu::LayoutKind::Subgroup));
   auto maskLayoutAttr = xegpu::inferMaskOffsetLayoutForScatterIO(
       requiredAnchorLayoutAttr, chunkSize);
-  LayoutInfo maskLayoutInfo = LayoutInfo(maskLayoutAttr);
+  LayoutInfo maskLayoutInfo = makeLayoutInfo(maskLayoutAttr);
 
   // Propagate the payload operand layout
   propagateIfChanged(operands[0], operands[0]->meet(srcLayoutInfo));
@@ -1451,7 +1421,7 @@ void LayoutInfoPropagation::visitStoreMatrixOp(
     }
     storeMatrix.setLayoutAttr(requiredAnchorLayoutAttr);
   }
-  layout = LayoutInfo(requiredAnchorLayoutAttr);
+  layout = makeLayoutInfo(requiredAnchorLayoutAttr);
   propagateIfChanged(operands[0], operands[0]->meet(layout));
 }
 

>From e19b889e3d33f34618d7d2b65eb8939956b0f080 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 19:26:12 +0000
Subject: [PATCH 7/9] [MLIR][XeGPU] Update test expectations for nearer-user
 layout preference

Update the truncf_prefers_nearer_user CHECK line and the simple_mxfp_gemm
quantizeA layout to match the layouts produced by preferring the nearer
consumer's demand in LayoutInfo::meet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir         | 2 +-
 .../Dialect/XeGPU/WG/simple_mxfp_gemm_quantizeA_F4.mlir         | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index 1f0acc68b131e..e8cddb8fa1de3 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -653,7 +653,7 @@ func.func @complete_dpas_mx_inst_data(%arg0: vector<16x1024xf8E5M2>, %arg1: vect
 gpu.module @test {
   // CHECK-LABEL: truncf_prefers_nearer_user
   // CHECK: %[[TRUNC:.*]] = arith.truncf
-  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [4, 8, 4], lane_layout = [4, 1, 4], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
+  // CHECK-SAME: {layout_result_0 = #xegpu.slice<#xegpu.layout<inst_data = [16, 8, 1], lane_layout = [16, 1, 1], lane_data = [1, 1, 1], order = [0, 2, 1]>, dims = [0]>}
   // CHECK-SAME: : vector<32x4xbf16> to vector<32x4xf8E8M0FNU>
   gpu.func @truncf_prefers_nearer_user(%src: memref<32x128xbf16>, %dst_red: memref<32x128xf8E8M0FNU>,
       %dst_plain: memref<32x4xf8E8M0FNU>) kernel {
diff --git a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_quantizeA_F4.mlir b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_quantizeA_F4.mlir
index cfaa8077c742c..87603594db5bf 100644
--- a/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_quantizeA_F4.mlir
+++ b/mlir/test/Integration/Dialect/XeGPU/WG/simple_mxfp_gemm_quantizeA_F4.mlir
@@ -8,7 +8,7 @@
 
 // XFAIL: *
 // Note: layouts used by dpas_mx need to match HW constaint. Otherwise dpas_mx is not unrolled.
-#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 1]>
+#a = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
 #a_ld = #xegpu.layout<sg_layout = [2, 2], sg_data = [16, 1024], inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
 #b_packed = #xegpu.layout<sg_layout = [2, 2], sg_data = [512, 16], inst_data = [32, 16], lane_layout = [1, 16], lane_data = [4, 1]>
 #b = #xegpu.layout<sg_layout = [2, 2], sg_data = [1024, 16], inst_data = [64, 16], lane_layout = [1, 16], lane_data = [8, 1]>

>From 557c0167940ef4dacfc64c010e4122e5a20cb7c9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 10 Jul 2026 23:11:09 +0000
Subject: [PATCH 8/9] [MLIR][XeGPU] Fix overstated programOrder comments

The comments claimed excluding programOrder from operator== was required
for convergence. It is not: order decreases monotonically under meet, so
including it would only cause redundant re-propagation, not incorrect
results. Drop the misleading wording.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply at anthropic.com>
---
 mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index c73749128906b..1681d295ae0ff 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -89,7 +89,7 @@ namespace {
 /// preserve a consumer's layout as far up the def chain as possible, minimizing
 /// layout conversions. This is a hint, not an optimum. `programOrder` is never
 /// propagated up the chain - each visited op stamps its own index - so it is
-/// deliberately excluded from `operator==`.
+/// excluded from `operator==`.
 
 struct LayoutInfo {
 private:
@@ -107,8 +107,6 @@ struct LayoutInfo {
   //  - one assigned, the other not -> not equal;
   //  - both unassigned             -> equal;
   //  - both assigned               -> equal iff the layouts match.
-  // `programOrder` is intentionally excluded: it is not propagated, so a pure
-  // order refinement must not be reported as a change.
   bool operator==(const LayoutInfo &other) const {
     if (isAssigned() != other.isAssigned())
       return false;

>From f45cf2a99627e236189da51344a6f763cbfbb46b Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 11 Jul 2026 15:26:24 +0000
Subject: [PATCH 9/9] [MLIR][XeGPU] Add SLM privatization to WG-to-SG
 distribution

Demote a shared local memory (space 3) buffer to subgroup-private memory
(space 4) when every load_matrix/store_matrix on it uses the same sg_layout,
sg_data, data shape and offsets, and the workgroup tile is evenly distributed
across subgroups. In that case each subgroup accesses an identical,
non-overlapping region, so the buffer and mem_desc are shrunk to the
per-subgroup size and matrix ops are re-indexed with local, subgroup-id-free
offsets.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../mlir/Dialect/XeGPU/IR/XeGPUDialect.td     |   5 +
 .../mlir/Dialect/XeGPU/IR/XeGPUTypes.td       |  11 +-
 mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp    |  11 +
 .../Transforms/XeGPUWgToSgDistribute.cpp      | 308 +++++++++++++++++-
 mlir/test/Dialect/XeGPU/invalid.mlir          |   2 +-
 .../XeGPU/xegpu-wg-to-sg-privatize.mlir       | 207 ++++++++++++
 6 files changed, 529 insertions(+), 15 deletions(-)
 create mode 100644 mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-privatize.mlir

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUDialect.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUDialect.td
index b1490c7742a26..34d38844051d1 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUDialect.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUDialect.td
@@ -43,6 +43,11 @@ def XeGPU_Dialect : Dialect {
       /// xevm::AddrSpace::SHARED, or a GPU workgroup memory address space.
       static bool isSharedMemory(const MemRefType &memrefTy);
 
+      /// Checks if the given memref type represents subgroup-private memory.
+      /// Returns true if the memory space is address space 4, or a GPU private
+      /// memory address space.
+      static bool isPrivateMemory(const MemRefType &memrefTy);
+
       /// drops/slices the shape in the specified dims, and return the rest. e.g.,
       /// for shape = [32, 64, 8], dims = [0, 2], it will return [64]
       template<typename T, typename U>
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
index 0423303c23493..0c2acc550d93a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
@@ -40,14 +40,17 @@ class XeGPUTypeDef<string name, string typeMnemonic, list<Trait> traits = [],
 }
 
 def isSharedPred : CPred<"XeGPUDialect::isSharedMemory(llvm::cast<mlir::MemRefType>($_self))">;
+def isSharedOrPrivatePred : CPred<[{
+    XeGPUDialect::isSharedMemory(llvm::cast<mlir::MemRefType>($_self)) ||
+    XeGPUDialect::isPrivateMemory(llvm::cast<mlir::MemRefType>($_self))}]>;
 class StaticShared1DMemRefOf<list<Type> allowedTypes> :
-  ConfinedType<MemRefRankOf<allowedTypes, [1]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 1d shaped " # MemRefOf<allowedTypes>.summary # " ",
+  ConfinedType<MemRefRankOf<allowedTypes, [1]>, [HasStaticShapePred, isSharedOrPrivatePred],
+     "reside in shared or private memory and statically 1d shaped " # MemRefOf<allowedTypes>.summary # " ",
      "mlir::MemRefType">;
 
 class StaticShared2DMemRefOf<list<Type> allowedTypes>:
-  ConfinedType<MemRefRankOf<allowedTypes, [2]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 2d shaped " # MemRefOf<allowedTypes>.summary # " ",
+  ConfinedType<MemRefRankOf<allowedTypes, [2]>, [HasStaticShapePred, isSharedOrPrivatePred],
+     "reside in shared or private memory and statically 2d shaped " # MemRefOf<allowedTypes>.summary # " ",
      "mlir::MemRefType">;
 
 def XeGPU_TensorDesc: XeGPUTypeDef<"TensorDesc", "tensor_desc",
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp
index 0951350e2fe82..1b1deb1edcf4f 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp
@@ -194,6 +194,17 @@ bool XeGPUDialect::isSharedMemory(const MemRefType &memrefTy) {
   return gpu::GPUDialect::isWorkgroupMemoryAddressSpace(attr);
 }
 
+bool XeGPUDialect::isPrivateMemory(const MemRefType &memrefTy) {
+  Attribute attr = memrefTy.getMemorySpace();
+  if (!attr)
+    return false; // Default memory space is not private memory
+  if (auto intAttr = llvm::dyn_cast_if_present<IntegerAttr>(attr))
+    return intAttr.getInt() == 4;
+  if (auto gpuAttr = llvm::dyn_cast_if_present<gpu::AddressSpaceAttr>(attr))
+    return gpuAttr.getValue() == gpu::GPUDialect::getPrivateAddressSpace();
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 // XeGPU_BlockTensorDescAttr
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..bc540708264fc 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -36,6 +36,12 @@ using namespace mlir;
 
 namespace {
 
+// Marker attribute carrying the common workgroup layout onto create_mem_desc so
+// the conversion pattern can shrink the private buffer to the per-subgroup size.
+// Set by the SLM privatization pre-phase and consumed by WgToSgCreateMemDescOp.
+static constexpr StringLiteral kPrivatizeLayoutAttrName =
+    "__xegpu_privatize_layout__";
+
 // Retrieve the RangeAttr if it is specified.
 static xegpu::RangeAttr getRangeSpecAttr(Operation *op) {
   Operation *parent = op->getParentOfType<scf::IfOp>();
@@ -887,28 +893,76 @@ struct WgToSgStoreScatterOp
   }
 };
 
+// Returns true if `memDesc` was produced by a create_mem_desc whose source
+// memref lives in private memory (i.e. it was privatized by the pre-phase).
+static bool isPrivateMemDesc(Value memDesc) {
+  auto createOp = memDesc.getDefiningOp<xegpu::CreateMemDescOp>();
+  if (!createOp)
+    return false;
+  auto srcTy = dyn_cast<MemRefType>(createOp.getSource().getType());
+  return srcTy && xegpu::XeGPUDialect::isPrivateMemory(srcTy);
+}
+
+// Computes the per-subgroup local offset lists into a privatized (shrunk)
+// mem_desc. Because every subgroup owns a private copy sized to the distributed
+// shape, offsets are subgroup-id independent: round `r` maps to `r * sg_data`.
+static SmallVector<SmallVector<OpFoldResult>>
+genPrivateOffsetsList(ConversionPatternRewriter &rewriter, Location loc,
+                      ArrayRef<int64_t> wgShape,
+                      xegpu::DistributeLayoutAttr layout) {
+  SmallVector<int64_t> sgData = layout.getEffectiveSgDataAsInt();
+  SmallVector<int64_t> sgLayout = layout.getEffectiveSgLayoutAsInt();
+
+  // The private buffer holds the per-subgroup distributed shape, which spans all
+  // distribution rounds stacked along each dim (roundShape[d] = wgShape[d] /
+  // sg_layout[d] = sg_data[d] * rounds). This matches the shape allocated by
+  // WgToSgCreateMemDescOp, so stepping by sg_data yields in-bounds local offsets.
+  SmallVector<int64_t> roundShape(wgShape.size());
+  for (auto [i, dim] : llvm::enumerate(wgShape))
+    roundShape[i] = dim / sgLayout[i];
+
+  SmallVector<SmallVector<OpFoldResult>> offsetsList;
+  for (SmallVector<int64_t> off : StaticTileOffsetRange(roundShape, sgData)) {
+    SmallVector<OpFoldResult> offsets;
+    for (int64_t o : off)
+      offsets.push_back(rewriter.getIndexAttr(o));
+    offsetsList.push_back(std::move(offsets));
+  }
+  return offsetsList;
+}
+
 struct WgToSgLoadMatrixOp : public OpConversionPattern<xegpu::LoadMatrixOp> {
   using OpConversionPattern<xegpu::LoadMatrixOp>::OpConversionPattern;
   LogicalResult
   matchAndRewrite(xegpu::LoadMatrixOp op, OneToNOpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
 
-    SmallVector<SmallVector<OpFoldResult>> offsetsList;
-    if (failed(genOffsetsList(rewriter, op, offsetsList)))
-      return failure();
-
     ArrayRef<int64_t> wgShape = op.getDataShape();
     VectorType valueTy = llvm::dyn_cast<VectorType>(op.getRes().getType());
     assert(valueTy && "the value type must be vector type!");
     Type elemTy = valueTy.getElementType();
 
     xegpu::DistributeLayoutAttr layout = op.getLayoutAttr();
+
+    // Privatized buffers are shrunk to the per-subgroup size and indexed with
+    // local, subgroup-id-free offsets; SLM buffers keep the sg-relative offsets.
+    bool isPrivate = isPrivateMemDesc(op.getMemDesc());
+    auto memDesc = cast<TypedValue<xegpu::MemDescType>>(
+        isPrivate ? adaptor.getMemDesc()[0] : op.getMemDesc());
+    SmallVector<SmallVector<OpFoldResult>> offsetsList;
+    if (isPrivate) {
+      offsetsList = genPrivateOffsetsList(rewriter, op.getLoc(), wgShape,
+                                          layout);
+    } else if (failed(genOffsetsList(rewriter, op, offsetsList))) {
+      return failure();
+    }
+
     SmallVector<int64_t> sgShape = getSgShapeAndCount(wgShape, layout).first;
     VectorType newResTy = VectorType::get(sgShape, elemTy);
     SmallVector<Value> newOps;
     for (auto offsets : offsetsList) {
       auto newOp = xegpu::LoadMatrixOp::create(rewriter, op.getLoc(), newResTy,
-                                               op.getMemDesc(), offsets,
+                                               memDesc, offsets,
                                                layout.dropSgLayoutAndData());
       newOps.push_back(newOp);
     }
@@ -924,19 +978,69 @@ struct WgToSgStoreMatrixOp : public OpConversionPattern<xegpu::StoreMatrixOp> {
   matchAndRewrite(xegpu::StoreMatrixOp op, OneToNOpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
 
+    xegpu::DistributeLayoutAttr layout = op.getLayoutAttr();
+
+    bool isPrivate = isPrivateMemDesc(op.getMemDesc());
+    auto memDesc = cast<TypedValue<xegpu::MemDescType>>(
+        isPrivate ? adaptor.getMemDesc()[0] : op.getMemDesc());
     SmallVector<SmallVector<OpFoldResult>> offsetsList;
-    if (failed(genOffsetsList(rewriter, op, offsetsList)))
+    if (isPrivate) {
+      offsetsList = genPrivateOffsetsList(rewriter, op.getLoc(),
+                                          op.getDataShape(), layout);
+    } else if (failed(genOffsetsList(rewriter, op, offsetsList))) {
       return failure();
+    }
 
-    xegpu::DistributeLayoutAttr layout = op.getLayoutAttr();
     for (auto [v, offsets] : llvm::zip(adaptor.getData(), offsetsList))
-      xegpu::StoreMatrixOp::create(rewriter, op.getLoc(), v, op.getMemDesc(),
-                                   offsets, layout.dropSgLayoutAndData());
+      xegpu::StoreMatrixOp::create(rewriter, op.getLoc(), v, memDesc, offsets,
+                                   layout.dropSgLayoutAndData());
     rewriter.eraseOp(op);
     return success();
   }
 };
 
+// Shrinks a privatized create_mem_desc (marked by the pre-phase) to the
+// per-subgroup size: it allocates a smaller private buffer and builds a
+// mem_desc of the distributed shape, so each subgroup owns a private copy.
+struct WgToSgCreateMemDescOp
+    : public OpConversionPattern<xegpu::CreateMemDescOp> {
+  using OpConversionPattern<xegpu::CreateMemDescOp>::OpConversionPattern;
+  LogicalResult
+  matchAndRewrite(xegpu::CreateMemDescOp op, OneToNOpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto layout = op->getAttrOfType<xegpu::DistributeLayoutAttr>(
+        kPrivatizeLayoutAttrName);
+    if (!layout)
+      return failure();
+
+    Location loc = op.getLoc();
+    xegpu::MemDescType mdescTy = op.getMemDesc().getType();
+    Type elemTy = mdescTy.getElementType();
+
+    // Per-subgroup distributed shape: the workgroup shape divided by sg_layout,
+    // holding every distribution round this subgroup owns (sg_data * rounds).
+    auto distShape =
+        layout.computeDistributedShape(llvm::to_vector(mdescTy.getShape()));
+    if (failed(distShape))
+      return failure();
+    SmallVector<int64_t> sgShape = *distShape;
+
+    // Allocate a private buffer just large enough for one subgroup's copy.
+    auto bytesPerElement = elemTy.getIntOrFloatBitWidth() / 8;
+    auto slmSize = computeProduct(sgShape) * bytesPerElement;
+    auto memrefTy =
+        MemRefType::get({slmSize}, rewriter.getI8Type(), {}, /*space=*/4);
+    auto buffer = memref::AllocaOp::create(rewriter, loc, memrefTy);
+
+    auto newMdescTy = xegpu::MemDescType::get(rewriter.getContext(), sgShape,
+                                              elemTy, mdescTy.getMemLayout());
+    auto newOp =
+        xegpu::CreateMemDescOp::create(rewriter, loc, newMdescTy, buffer);
+    rewriter.replaceOp(op, newOp.getResult());
+    return success();
+  }
+};
+
 // This pattern distributes the vector.step ops to work at subgroup level
 struct WgToSgVectorStepOp : public OpConversionPattern<vector::StepOp> {
   using OpConversionPattern<vector::StepOp>::OpConversionPattern;
@@ -1529,7 +1633,8 @@ void populateXeGPUWgToSgDistributePatterns(RewritePatternSet &patterns) {
                WgToSgDpasMxOp, WgToSgPrefetchNdOp, WgToSgElementwiseOp,
                WgToSgVectorBroadcastOp, WgToSgConvertLayoutOp,
                WgToSgArithConstantOp, WgToSgLoadGatherOp, WgToSgStoreScatterOp,
-               WgToSgLoadMatrixOp, WgToSgStoreMatrixOp, WgToSgVectorStepOp,
+               WgToSgLoadMatrixOp, WgToSgStoreMatrixOp, WgToSgCreateMemDescOp,
+               WgToSgVectorStepOp,
                WgToSgVectorShapeCastOp, WgToSgMultiDimReductionOp,
                WgToSgVectorTransposeOp, WgToSgVectorConstantMaskOp,
                WgToSgVectorCreateMaskOp, WgToSgVectorBitCastOp,
@@ -1540,6 +1645,166 @@ void populateXeGPUWgToSgDistributePatterns(RewritePatternSet &patterns) {
 } // namespace mlir
 
 namespace {
+
+//===----------------------------------------------------------------------===//
+// SLM privatization
+//===----------------------------------------------------------------------===//
+// An SLM buffer whose every matrix access (load_matrix/store_matrix) uses the
+// same subgroup layout, the same data (vector) shape, and the same offsets is
+// accessed identically by every subgroup. Because the same mem_desc implies the
+// same physical layout, an identical (sg_layout, sg_data, offset) means every
+// subgroup reads and writes the exact same SLM region, so the region is
+// effectively private to each subgroup and can be demoted to private memory
+// (space 4, backed by registers).
+//
+// This runs as a pre-phase of WG-to-SG distribution. For a qualifying buffer it
+// (a) flips the source memref's memory space from 3 (SLM) to 4 (private), and
+// (b) stashes the common workgroup layout on the create_mem_desc op as a
+// discardable marker attribute so the WgToSgCreateMemDescOp pattern can later
+// shrink the memref/mem_desc to the per-subgroup size. The matrix ops are left
+// intact for the regular distribution patterns that follow.
+
+// Compares two offset lists for structural equality. Static offsets compare by
+// value; dynamic offsets compare by SSA value identity.
+static bool sameOffsets(ArrayRef<OpFoldResult> a, ArrayRef<OpFoldResult> b) {
+  if (a.size() != b.size())
+    return false;
+  for (auto [lhs, rhs] : llvm::zip(a, b)) {
+    auto lhsAttr = dyn_cast<Attribute>(lhs);
+    auto rhsAttr = dyn_cast<Attribute>(rhs);
+    if (static_cast<bool>(lhsAttr) != static_cast<bool>(rhsAttr))
+      return false;
+    if (lhsAttr) {
+      if (lhsAttr != rhsAttr)
+        return false;
+    } else if (cast<Value>(lhs) != cast<Value>(rhs)) {
+      return false;
+    }
+  }
+  return true;
+}
+
+// Returns the single create_mem_desc op that views `source` if `source` is a
+// privatizable local buffer. The buffer must be a local alloca/alloc in SLM
+// whose only user is exactly one create_mem_desc op; any other user (or a
+// second view) means the buffer might escape or be aliased, so it is rejected.
+static xegpu::CreateMemDescOp getPrivatizableView(Value source) {
+  Operation *defOp = source.getDefiningOp();
+  if (!defOp || !isa<memref::AllocaOp, memref::AllocOp>(defOp))
+    return nullptr;
+
+  auto memrefTy = dyn_cast<MemRefType>(source.getType());
+  if (!memrefTy || !xegpu::XeGPUDialect::isSharedMemory(memrefTy))
+    return nullptr;
+
+  // The buffer must be viewed by exactly one create_mem_desc and nothing else,
+  // so it neither escapes nor aliases another mem_desc.
+  if (!source.hasOneUse())
+    return nullptr;
+  return dyn_cast<xegpu::CreateMemDescOp>(*source.getUsers().begin());
+}
+
+// If every load_matrix/store_matrix on `createOp`'s mem_desc is a workgroup
+// access sharing the same sg_layout, sg_data, data shape, and offsets, and the
+// workgroup tile is evenly distributed to the subgroups (no broadcast/overlap),
+// returns that common layout. Otherwise returns nullptr.
+static xegpu::DistributeLayoutAttr
+getSubgroupPrivateLayout(xegpu::CreateMemDescOp createOp) {
+  xegpu::DistributeLayoutAttr commonLayout;
+  SmallVector<OpFoldResult> commonOffsets;
+  ArrayRef<int64_t> commonDataShape;
+  bool seen = false;
+
+  auto checkAccess = [&](xegpu::DistributeLayoutAttr layout,
+                         SmallVector<OpFoldResult> offsets,
+                         ArrayRef<int64_t> dataShape) -> bool {
+    // Every access must be a distributed workgroup-level access.
+    if (!layout || !layout.isForWorkgroup())
+      return false;
+
+    // The workgroup tile must be evenly distributed across the subgroups with
+    // no broadcast: sg_layout[d] * sg_data[d] must not exceed the tile in any
+    // dim. A larger product means a dimension wraps around and is broadcast to
+    // multiple subgroups, so their regions overlap and are not private.
+    SmallVector<int64_t> sgLayout = layout.getEffectiveSgLayoutAsInt();
+    SmallVector<int64_t> sgData = layout.getEffectiveSgDataAsInt();
+    if (sgLayout.size() != dataShape.size() ||
+        sgData.size() != dataShape.size())
+      return false;
+    for (auto [l, d, tile] : llvm::zip_equal(sgLayout, sgData, dataShape))
+      if (l * d > tile)
+        return false;
+
+    if (!seen) {
+      commonLayout = layout;
+      commonOffsets = std::move(offsets);
+      commonDataShape = dataShape;
+      seen = true;
+      return true;
+    }
+    return commonLayout.getEffectiveSgLayoutAsInt() == sgLayout &&
+           commonLayout.getEffectiveSgDataAsInt() == sgData &&
+           commonDataShape == dataShape && sameOffsets(commonOffsets, offsets);
+  };
+
+  for (Operation *memUser : createOp.getMemDesc().getUsers()) {
+    if (auto loadOp = dyn_cast<xegpu::LoadMatrixOp>(memUser)) {
+      if (!checkAccess(loadOp.getLayoutAttr(), loadOp.getMixedOffsets(),
+                       loadOp.getDataShape()))
+        return nullptr;
+    } else if (auto storeOp = dyn_cast<xegpu::StoreMatrixOp>(memUser)) {
+      if (!checkAccess(storeOp.getLayoutAttr(), storeOp.getMixedOffsets(),
+                       storeOp.getDataShape()))
+        return nullptr;
+    } else {
+      // Any other use of the mem_desc is unexpected; be conservative.
+      return nullptr;
+    }
+  }
+
+  // Require at least one access to have been observed.
+  return seen ? commonLayout : nullptr;
+}
+
+// Rewrites the buffer defined by `defOp` (an alloca/alloc) so that its memory
+// space is private (4) instead of shared (3), updates the source type of the
+// create_mem_desc that consumes it, and marks that op with the common workgroup
+// layout so the conversion pattern can shrink it to the per-subgroup size.
+static void privatizeSlmBuffer(Operation *defOp, xegpu::CreateMemDescOp view,
+                               xegpu::DistributeLayoutAttr layout) {
+  OpBuilder builder(defOp);
+  Value oldBuffer = defOp->getResult(0);
+  auto oldTy = cast<MemRefType>(oldBuffer.getType());
+  auto privateSpace = builder.getI64IntegerAttr(4);
+  auto newTy = MemRefType::get(oldTy.getShape(), oldTy.getElementType(),
+                               oldTy.getLayout(), privateSpace);
+
+  Operation *newDefOp = builder.clone(*defOp);
+  newDefOp->getResult(0).setType(newTy);
+  oldBuffer.replaceAllUsesWith(newDefOp->getResult(0));
+  defOp->erase();
+
+  view->setAttr(kPrivatizeLayoutAttrName, layout);
+}
+
+// Entry point for the SLM privatization pre-phase.
+static void privatizeSharedLocalMemory(Operation *root) {
+  SmallVector<std::tuple<Operation *, xegpu::CreateMemDescOp,
+                         xegpu::DistributeLayoutAttr>>
+      toPrivatize;
+
+  root->walk([&](xegpu::CreateMemDescOp createOp) {
+    Value source = createOp.getSource();
+    if (getPrivatizableView(source) != createOp)
+      return;
+    if (auto layout = getSubgroupPrivateLayout(createOp))
+      toPrivatize.push_back({source.getDefiningOp(), createOp, layout});
+  });
+
+  for (auto [defOp, view, layout] : toPrivatize)
+    privatizeSlmBuffer(defOp, view, layout);
+}
+
 struct XeGPUWgToSgDistributePass
     : public xegpu::impl::XeGPUWgToSgDistributeBase<XeGPUWgToSgDistributePass> {
   void runOnOperation() override;
@@ -1549,11 +1814,19 @@ struct XeGPUWgToSgDistributePass
 void XeGPUWgToSgDistributePass::runOnOperation() {
 
   Operation *op = getOperation();
+
   if (!xegpu::recoverTemporaryLayouts(op)) {
     signalPassFailure();
     return;
   }
 
+  // Pre-phase: demote SLM buffers that are accessed identically by every
+  // subgroup (same sg_layout, sg_data, data shape and offsets) to subgroup-
+  // private memory. Run after recoverTemporaryLayouts, which strips discardable
+  // DistributeLayoutAttr-valued attrs (and would otherwise remove the marker
+  // attribute this phase sets on create_mem_desc).
+  privatizeSharedLocalMemory(op);
+
   // Collect existing UnrealizedConversionCastOps. These must be preserved.
   llvm::SmallSetVector<UnrealizedConversionCastOp, 8> existingCasts;
   getOperation()->walk(
@@ -1625,6 +1898,13 @@ void XeGPUWgToSgDistributePass::runOnOperation() {
         return isLegal(op.getLayoutAttr());
       });
 
+  // A create_mem_desc marked by the privatization pre-phase must be shrunk to
+  // the per-subgroup size; all others are already legal.
+  target.addDynamicallyLegalOp<xegpu::CreateMemDescOp>(
+      [=](xegpu::CreateMemDescOp op) -> bool {
+        return !op->hasAttr(kPrivatizeLayoutAttrName);
+      });
+
   target.addDynamicallyLegalOp<arith::ConstantOp>(
       [=](arith::ConstantOp op) -> bool {
         auto vecType = dyn_cast<VectorType>(op.getType());
@@ -1703,4 +1983,12 @@ void XeGPUWgToSgDistributePass::runOnOperation() {
   // Fold cancelling cast chains and erase dead casts.
   xegpu::cleanupUnrealizedConversionCasts(getOperation(), existingCasts);
   xegpu::removeTemporaryLayoutAttrs(getOperation());
+
+  // Erase the original full-size buffers left dead by privatization: the
+  // WgToSgCreateMemDescOp pattern allocates fresh per-subgroup buffers, so the
+  // pre-phase's space-4 allocas no longer have any users.
+  getOperation()->walk([](Operation *op) {
+    if (isa<memref::AllocaOp, memref::AllocOp>(op) && op->use_empty())
+      op->erase();
+  });
 }
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..157a3bcda7b1d 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -607,7 +607,7 @@ func.func @slice_attr_repeat_dim() {
 // -----
 func.func @create_mem_desc_non_slm() {
   %m = memref.alloca() {alignment = 1024} : memref<2048xi8, 1>
-  // expected-error at +1 {{operand #0 must be reside in share memory and statically 1d shaped memref }}
+  // expected-error at +1 {{operand #0 must be reside in shared or private memory and statically 1d shaped memref }}
   %mem_desc = xegpu.create_mem_desc %m : memref<2048xi8, 1> -> !xegpu.mem_desc<16x64xf16>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-privatize.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-privatize.mlir
new file mode 100644
index 0000000000000..57f2a01189cee
--- /dev/null
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-privatize.mlir
@@ -0,0 +1,207 @@
+// RUN: mlir-opt --xegpu-wg-to-sg-distribute -split-input-file %s | FileCheck %s
+
+// The SLM privatization pre-phase of WG-to-SG distribution demotes a shared
+// local memory (space 3) buffer to subgroup-private memory (space 4) when every
+// matrix access to it uses the same sg_layout, sg_data, data shape and offsets,
+// and the workgroup tile is evenly distributed to the subgroups (no broadcast).
+// In that case each subgroup reads/writes an identical, non-overlapping region,
+// so the buffer and mem_desc are shrunk to the per-subgroup size and the matrix
+// ops are re-indexed with local, subgroup-id-free offsets.
+
+gpu.module @test {
+  // A buffer whose load_matrix and store_matrix share the same sg_layout,
+  // sg_data and offsets is private to each subgroup: it is moved to memory space
+  // 4 and shrunk from 64x128 to the per-subgroup 32x32 tile.
+  // CHECK-LABEL: gpu.func @privatize_same_offset_layout
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<4096xi8, 4>
+  // CHECK: %[[MD:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<4096xi8, 4> -> !xegpu.mem_desc<32x32xf32>
+  // CHECK: xegpu.load_matrix %[[MD]][0, 0] : !xegpu.mem_desc<32x32xf32> -> vector<32x32xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD]][0, 0] : vector<32x32xf32>, !xegpu.mem_desc<32x32xf32>
+  gpu.func @privatize_same_offset_layout() {
+    %cst = arith.constant dense<1.0> : vector<64x128xf32>
+    %a = memref.alloca() : memref<32768xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+    %ld = xegpu.load_matrix %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : !xegpu.mem_desc<64x128xf32> -> vector<64x128xf32>
+    xegpu.store_matrix %cst, %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : vector<64x128xf32>, !xegpu.mem_desc<64x128xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // When a dimension spans multiple distribution rounds (128 / sg_layout 8 = 16
+  // = 2 rounds of sg_data 8), the shrunk buffer holds both rounds and each round
+  // is indexed with a local offset (0 and 8).
+  // CHECK-LABEL: gpu.func @privatize_multi_round
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<64xi8, 4>
+  // CHECK: %[[MD:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<64xi8, 4> -> !xegpu.mem_desc<16xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD]][0] : vector<8xf32>, !xegpu.mem_desc<16xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD]][8] : vector<8xf32>, !xegpu.mem_desc<16xf32>
+  // CHECK: xegpu.load_matrix %[[MD]][0] : !xegpu.mem_desc<16xf32> -> vector<8xf32>
+  // CHECK: xegpu.load_matrix %[[MD]][8] : !xegpu.mem_desc<16xf32> -> vector<8xf32>
+  gpu.func @privatize_multi_round() {
+    %cst = arith.constant dense<1.0> : vector<128xf32>
+    %a = memref.alloca() : memref<512xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<512xi8, 3> -> !xegpu.mem_desc<128xf32>
+    xegpu.store_matrix %cst, %md[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+      : vector<128xf32>, !xegpu.mem_desc<128xf32>
+    %l = xegpu.load_matrix %md[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+      : !xegpu.mem_desc<128xf32> -> vector<128xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // Accesses use different offsets, so the region seen by each subgroup is not
+  // identical: the buffer stays in shared local memory (space 3), full size.
+  // CHECK-LABEL: gpu.func @no_privatize_diff_offsets
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<65536xi8, 3>
+  // CHECK: xegpu.create_mem_desc %[[ALLOCA]] : memref<65536xi8, 3> -> !xegpu.mem_desc<128x128xf32>
+  gpu.func @no_privatize_diff_offsets() {
+    %cst = arith.constant dense<1.0> : vector<64x128xf32>
+    %a = memref.alloca() : memref<65536xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<65536xi8, 3> -> !xegpu.mem_desc<128x128xf32>
+    %ld = xegpu.load_matrix %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : !xegpu.mem_desc<128x128xf32> -> vector<64x128xf32>
+    xegpu.store_matrix %cst, %md[32, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : vector<64x128xf32>, !xegpu.mem_desc<128x128xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // Accesses use different sg_layouts, so subgroups partition the region
+  // differently: the buffer stays in shared local memory (space 3).
+  // CHECK-LABEL: gpu.func @no_privatize_diff_layout
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<32768xi8, 3>
+  // CHECK: xegpu.create_mem_desc %[[ALLOCA]] : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+  gpu.func @no_privatize_diff_layout() {
+    %cst = arith.constant dense<1.0> : vector<64x128xf32>
+    %a = memref.alloca() : memref<32768xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+    %ld = xegpu.load_matrix %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : !xegpu.mem_desc<64x128xf32> -> vector<64x128xf32>
+    xegpu.store_matrix %cst, %md[0, 0] <{layout = #xegpu.layout<sg_layout = [4, 2], sg_data = [16, 64]>}>
+      : vector<64x128xf32>, !xegpu.mem_desc<64x128xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // Accesses use different data (vector) shapes, so the per-subgroup regions
+  // differ: the buffer stays in shared local memory (space 3).
+  // CHECK-LABEL: gpu.func @no_privatize_diff_data_shape
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<32768xi8, 3>
+  // CHECK: xegpu.create_mem_desc %[[ALLOCA]] : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+  gpu.func @no_privatize_diff_data_shape() {
+    %c0 = arith.constant dense<1.0> : vector<64x128xf32>
+    %c1 = arith.constant dense<1.0> : vector<32x128xf32>
+    %a = memref.alloca() : memref<32768xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+    xegpu.store_matrix %c0, %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : vector<64x128xf32>, !xegpu.mem_desc<64x128xf32>
+    %ld = xegpu.load_matrix %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [16, 32]>}>
+      : !xegpu.mem_desc<64x128xf32> -> vector<32x128xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // The workgroup tile wraps around (sg_layout 4 * sg_data 8 = 32 > tile 8), so
+  // the single tile is broadcast to all four subgroups and their regions
+  // overlap: the buffer is not private and stays in shared local memory.
+  // CHECK-LABEL: gpu.func @no_privatize_broadcast
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<32xi8, 3>
+  // CHECK: xegpu.create_mem_desc %[[ALLOCA]] : memref<32xi8, 3> -> !xegpu.mem_desc<8xf32>
+  gpu.func @no_privatize_broadcast() {
+    %cst = arith.constant dense<1.0> : vector<8xf32>
+    %a = memref.alloca() : memref<32xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<32xi8, 3> -> !xegpu.mem_desc<8xf32>
+    xegpu.store_matrix %cst, %md[0] <{layout = #xegpu.layout<sg_layout = [4], sg_data = [8]>}>
+      : vector<8xf32>, !xegpu.mem_desc<8xf32>
+    %l = xegpu.load_matrix %md[0] <{layout = #xegpu.layout<sg_layout = [4], sg_data = [8]>}>
+      : !xegpu.mem_desc<8xf32> -> vector<8xf32>
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // The buffer escapes the function (passed to a call), so it may be observed
+  // by other subgroups and must not be privatized: it stays in space 3.
+  func.func private @use(memref<32768xi8, 3>)
+  // CHECK-LABEL: gpu.func @no_privatize_escaping_buffer
+  // CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<32768xi8, 3>
+  // CHECK: xegpu.create_mem_desc %[[ALLOCA]] : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+  gpu.func @no_privatize_escaping_buffer() {
+    %cst = arith.constant dense<1.0> : vector<64x128xf32>
+    %a = memref.alloca() : memref<32768xi8, 3>
+    %md = xegpu.create_mem_desc %a : memref<32768xi8, 3> -> !xegpu.mem_desc<64x128xf32>
+    xegpu.store_matrix %cst, %md[0, 0] <{layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [32, 32]>}>
+      : vector<64x128xf32>, !xegpu.mem_desc<64x128xf32>
+    func.call @use(%a) : (memref<32768xi8, 3>) -> ()
+    gpu.return
+  }
+}
+
+// -----
+
+gpu.module @test {
+  // End-to-end softmax-like kernel (after vector-to-xegpu, rewritten to use
+  // load_matrix/store_matrix). Both SLM scratch buffers (running max and running
+  // sum) are read and written by every subgroup with the same sg_layout, sg_data
+  // and offset, so both are demoted to subgroup-private memory and shrunk from
+  // 64 elements to the per-subgroup 8 elements.
+  // CHECK-LABEL: gpu.func @payload_kernel
+  // CHECK-DAG: %[[MAX:.*]] = memref.alloca() : memref<32xi8, 4>
+  // CHECK-DAG: %[[MD_MAX:.*]] = xegpu.create_mem_desc %[[MAX]] : memref<32xi8, 4> -> !xegpu.mem_desc<8xf32>
+  // CHECK-DAG: %[[SUM:.*]] = memref.alloca() : memref<32xi8, 4>
+  // CHECK-DAG: %[[MD_SUM:.*]] = xegpu.create_mem_desc %[[SUM]] : memref<32xi8, 4> -> !xegpu.mem_desc<8xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD_MAX]][0] : vector<8xf32>, !xegpu.mem_desc<8xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD_SUM]][0] : vector<8xf32>, !xegpu.mem_desc<8xf32>
+  // CHECK: scf.for
+  // CHECK: xegpu.load_matrix %[[MD_MAX]][0] : !xegpu.mem_desc<8xf32> -> vector<8xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD_MAX]][0] : vector<8xf32>, !xegpu.mem_desc<8xf32>
+  // CHECK: xegpu.load_matrix %[[MD_SUM]][0] : !xegpu.mem_desc<8xf32> -> vector<8xf32>
+  // CHECK: xegpu.store_matrix %{{.*}}, %[[MD_SUM]][0] : vector<8xf32>, !xegpu.mem_desc<8xf32>
+  gpu.func @payload_kernel() kernel {
+    %c16 = arith.constant 16 : index
+    %c512 = arith.constant 512 : index
+    %cst = arith.constant dense<0.000000e+00> : vector<64xf32>
+    %c0 = arith.constant 0 : index
+    %cst_0 = arith.constant dense<0xFFC00000> : vector<64xf32>
+    %alloca = memref.alloca() : memref<256xi8, 3>
+    %alloca_1 = memref.alloca() : memref<256xi8, 3>
+    %md_max = xegpu.create_mem_desc %alloca_1 : memref<256xi8, 3> -> !xegpu.mem_desc<64xf32>
+    %md_sum = xegpu.create_mem_desc %alloca : memref<256xi8, 3> -> !xegpu.mem_desc<64xf32>
+    xegpu.store_matrix %cst_0, %md_max[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+      : vector<64xf32>, !xegpu.mem_desc<64xf32>
+    xegpu.store_matrix %cst, %md_sum[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+      : vector<64xf32>, !xegpu.mem_desc<64xf32>
+    scf.for %arg2 = %c0 to %c512 step %c16 {
+      %4 = xegpu.load_matrix %md_max[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+        : !xegpu.mem_desc<64xf32> -> vector<64xf32>
+      %5 = arith.subf %cst, %4 : vector<64xf32>
+      xegpu.store_matrix %5, %md_max[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+        : vector<64xf32>, !xegpu.mem_desc<64xf32>
+      %11 = xegpu.load_matrix %md_sum[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+        : !xegpu.mem_desc<64xf32> -> vector<64xf32>
+      %12 = arith.addf %11, %5 : vector<64xf32>
+      xegpu.store_matrix %12, %md_sum[0] <{layout = #xegpu.layout<sg_layout = [8], sg_data = [8]>}>
+        : vector<64xf32>, !xegpu.mem_desc<64xf32>
+    }
+    gpu.return
+  }
+}


From llvmlistbot at llvm.org  Sat Jul 11 08:33:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 08:33:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] SLM Privatization for WG-to-SG
	Distribution (PR #208921)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208921@github.com>
Message-ID: <6a526230.481afadf.196d57.609c@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/XeGPU/IR/XeGPUDialect.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index bc5407082..0c43c925b 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -37,8 +37,9 @@ using namespace mlir;
 namespace {
 
 // Marker attribute carrying the common workgroup layout onto create_mem_desc so
-// the conversion pattern can shrink the private buffer to the per-subgroup size.
-// Set by the SLM privatization pre-phase and consumed by WgToSgCreateMemDescOp.
+// the conversion pattern can shrink the private buffer to the per-subgroup
+// size. Set by the SLM privatization pre-phase and consumed by
+// WgToSgCreateMemDescOp.
 static constexpr StringLiteral kPrivatizeLayoutAttrName =
     "__xegpu_privatize_layout__";
 
@@ -913,10 +914,11 @@ genPrivateOffsetsList(ConversionPatternRewriter &rewriter, Location loc,
   SmallVector<int64_t> sgData = layout.getEffectiveSgDataAsInt();
   SmallVector<int64_t> sgLayout = layout.getEffectiveSgLayoutAsInt();
 
-  // The private buffer holds the per-subgroup distributed shape, which spans all
-  // distribution rounds stacked along each dim (roundShape[d] = wgShape[d] /
-  // sg_layout[d] = sg_data[d] * rounds). This matches the shape allocated by
-  // WgToSgCreateMemDescOp, so stepping by sg_data yields in-bounds local offsets.
+  // The private buffer holds the per-subgroup distributed shape, which spans
+  // all distribution rounds stacked along each dim (roundShape[d] = wgShape[d]
+  // / sg_layout[d] = sg_data[d] * rounds). This matches the shape allocated by
+  // WgToSgCreateMemDescOp, so stepping by sg_data yields in-bounds local
+  // offsets.
   SmallVector<int64_t> roundShape(wgShape.size());
   for (auto [i, dim] : llvm::enumerate(wgShape))
     roundShape[i] = dim / sgLayout[i];
@@ -945,14 +947,15 @@ struct WgToSgLoadMatrixOp : public OpConversionPattern<xegpu::LoadMatrixOp> {
     xegpu::DistributeLayoutAttr layout = op.getLayoutAttr();
 
     // Privatized buffers are shrunk to the per-subgroup size and indexed with
-    // local, subgroup-id-free offsets; SLM buffers keep the sg-relative offsets.
+    // local, subgroup-id-free offsets; SLM buffers keep the sg-relative
+    // offsets.
     bool isPrivate = isPrivateMemDesc(op.getMemDesc());
     auto memDesc = cast<TypedValue<xegpu::MemDescType>>(
         isPrivate ? adaptor.getMemDesc()[0] : op.getMemDesc());
     SmallVector<SmallVector<OpFoldResult>> offsetsList;
     if (isPrivate) {
-      offsetsList = genPrivateOffsetsList(rewriter, op.getLoc(), wgShape,
-                                          layout);
+      offsetsList =
+          genPrivateOffsetsList(rewriter, op.getLoc(), wgShape, layout);
     } else if (failed(genOffsetsList(rewriter, op, offsetsList))) {
       return failure();
     }
@@ -961,9 +964,9 @@ struct WgToSgLoadMatrixOp : public OpConversionPattern<xegpu::LoadMatrixOp> {
     VectorType newResTy = VectorType::get(sgShape, elemTy);
     SmallVector<Value> newOps;
     for (auto offsets : offsetsList) {
-      auto newOp = xegpu::LoadMatrixOp::create(rewriter, op.getLoc(), newResTy,
-                                               memDesc, offsets,
-                                               layout.dropSgLayoutAndData());
+      auto newOp =
+          xegpu::LoadMatrixOp::create(rewriter, op.getLoc(), newResTy, memDesc,
+                                      offsets, layout.dropSgLayoutAndData());
       newOps.push_back(newOp);
     }
     rewriter.replaceOpWithMultiple(op, {newOps});
@@ -1634,12 +1637,11 @@ void populateXeGPUWgToSgDistributePatterns(RewritePatternSet &patterns) {
                WgToSgVectorBroadcastOp, WgToSgConvertLayoutOp,
                WgToSgArithConstantOp, WgToSgLoadGatherOp, WgToSgStoreScatterOp,
                WgToSgLoadMatrixOp, WgToSgStoreMatrixOp, WgToSgCreateMemDescOp,
-               WgToSgVectorStepOp,
-               WgToSgVectorShapeCastOp, WgToSgMultiDimReductionOp,
-               WgToSgVectorTransposeOp, WgToSgVectorConstantMaskOp,
-               WgToSgVectorCreateMaskOp, WgToSgVectorBitCastOp,
-               WgToSgVectorInterleaveOp, WgToSgVectorDeinterleaveOp>(
-      patterns.getContext());
+               WgToSgVectorStepOp, WgToSgVectorShapeCastOp,
+               WgToSgMultiDimReductionOp, WgToSgVectorTransposeOp,
+               WgToSgVectorConstantMaskOp, WgToSgVectorCreateMaskOp,
+               WgToSgVectorBitCastOp, WgToSgVectorInterleaveOp,
+               WgToSgVectorDeinterleaveOp>(patterns.getContext());
 }
 } // namespace xegpu
 } // namespace mlir

``````````

</details>


https://github.com/llvm/llvm-project/pull/208921

From llvmlistbot at llvm.org  Sat Jul 11 08:50:34 2026
From: llvmlistbot at llvm.org (Artem Kroviakov)
Date: Sat, 11 Jul 2026 08:50:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Prefer the nearer consumer's
 layout in LayoutInfo::meet (PR #208365)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208365@github.com>
Message-ID: <6a52664a.a87eb5dc.7d817.f9df@mx.google.com>


https://github.com/akroviakov approved this pull request.

Thanks for the `getSgLayoutCandidates` cleanup

https://github.com/llvm/llvm-project/pull/208365

From llvmlistbot at llvm.org  Sat Jul 11 08:58:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 08:58:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 31ddc7e - [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (#199911)
Message-ID: <6a526813.f1c1923d.991d1.edcb@mx.google.com>


Author: lonely eagle
Date: 2026-07-11T23:58:06+08:00
New Revision: 31ddc7e564814a42af39f0e71371469d0b1fca24

URL: https://github.com/llvm/llvm-project/commit/31ddc7e564814a42af39f0e71371469d0b1fca24
DIFF: https://github.com/llvm/llvm-project/commit/31ddc7e564814a42af39f0e71371469d0b1fca24.diff

LOG: [mlir][presburger] Avoid redundant zero-initialization in insertColumns (#199911)

When insertColumns does not trigger a physical reallocation, the inner
loop needlessly loops up to nReservedColumns - 1. This causes massive
redundant zero-writes on trailing columns that are already zero. This
patch truncates the inner loop start boundary to nColumns - 1, when the
reserved capacity is unchanged, optimizing the non-realloc path from
O(nRows * nReservedColumns) to O(nRows * nColumns).

Added: 
    

Modified: 
    mlir/lib/Analysis/Presburger/Matrix.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Analysis/Presburger/Matrix.cpp b/mlir/lib/Analysis/Presburger/Matrix.cpp
index 89d8a15422d95..7c55caa767c45 100644
--- a/mlir/lib/Analysis/Presburger/Matrix.cpp
+++ b/mlir/lib/Analysis/Presburger/Matrix.cpp
@@ -160,17 +160,17 @@ void Matrix<T>::insertColumns(unsigned pos, unsigned count) {
   }
   nColumns += count;
 
+  int colStart = nColumns - 1;
+  if (oldNReservedColumns != nReservedColumns)
+    colStart = nReservedColumns - 1;
   for (int ri = nRows - 1; ri >= 0; --ri) {
-    for (int ci = nReservedColumns - 1; ci >= 0; --ci) {
+    for (int ci = colStart; ci >= 0; --ci) {
       unsigned r = ri;
       unsigned c = ci;
       T &dest = data[r * nReservedColumns + c];
       if (c >= nColumns) { // NOLINT
         // Out of bounds columns are zero-initialized. NOLINT because clang-tidy
         // complains about this branch being the same as the c >= pos one.
-        //
-        // TODO: this case can be skipped if the number of reserved columns
-        // didn't change.
         dest = 0;
       } else if (c >= pos + count) {
         // Shift the data occuring after the inserted columns.


        

From llvmlistbot at llvm.org  Sat Jul 11 08:58:14 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Sat, 11 Jul 2026 08:58:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (PR #199911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199911@github.com>
Message-ID: <6a526816.e624eb8d.39cd4e.b23f@mx.google.com>


https://github.com/linuxlonelyeagle closed https://github.com/llvm/llvm-project/pull/199911

From llvmlistbot at llvm.org  Sat Jul 11 09:20:33 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Sat, 11 Jul 2026 09:20:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (PR #199911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199911@github.com>
Message-ID: <6a526d51.449bfb44.172ed9.eac1@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `clang-aarch64-sve-vla` running on `linaro-g3-02` while building `mlir` at step 6 "build stage 1".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/17/builds/16335

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 6 (build stage 1) failure: 'ninja' (failure)
...

  Tell CMake where to find the compiler by setting either the environment
  variable "FC" or the CMake cache entry CMAKE_Fortran_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[180/186] Linking CXX executable bin/bbc
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-stamps/runtimes-configure 
cd /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-bins && /usr/local/bin/cmake -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang -DCMAKE_CXX_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/clang -DCMAKE_Fortran_COMPILER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/flang -DCMAKE_LINKER=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/ld.lld -DCMAKE_AR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-ar -DCMAKE_RANLIB=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-ranlib -DCMAKE_NM=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-objcopy -DCMAKE_STRIP=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-strip -DCMAKE_READELF=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1.install -DLLVM_BINARY_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1 -DLLVM_CONFIG_PATH=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-v -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt -DFORTRAN_SUPPORTS_REAL16=1 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=flang-rt;compiler-rt" -DLLVM_USE_LINKER=lld -DCUDAToolkit_NVCC_EXECUTABLE=CUDAToolkit_NVCC_EXECUTABLE-NOTFOUND -DCUDAToolkit_SENTINEL_FILE=CUDAToolkit_SENTINEL_FILE-NOTFOUND -DFLANG_RUNTIME_F128_MATH_LIB= "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../flang-rt;-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../compiler-rt" -GNinja -C/home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/llvm/llvm/runtimes/../../runtimes -B /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-bins && /usr/local/bin/cmake -E touch /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1/runtimes/runtimes-stamps/runtimes-configure
[182/186] Linking CXX executable bin/flang-23
ninja: build stopped: subcommand failed.

```

</details>

https://github.com/llvm/llvm-project/pull/199911

From llvmlistbot at llvm.org  Sat Jul 11 10:07:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 10:07:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a527850.266b0cc5.29d78.0fec@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Sat Jul 11 10:09:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 10:09:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a5278ba.8e75e0cb.3dbce4.5156@mx.google.com>


mygitljf wrote:

Thanks! I’ve addressed both nits. I folded the diagnostic into the existing element-type check, changed it to return `LogicalResult`, and renamed it to `validateValidElementType`. The shared validation path still covers operands, results, and the declared type on `tosa.variable`.

I also removed the duplicate static-shape test and kept the dynamic-shape reproducer from the issue. 

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Sat Jul 11 10:09:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 10:09:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a5278e5.af6ba3ed.36d9db.5e9e@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/205873

>From 62993eafa5d487b3a786867622e5a52a37da217e Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 26 Jun 2026 01:54:59 +0000
Subject: [PATCH 1/3] [mlir][tosa] fallback unknown type

---
 .../mlir/Dialect/Tosa/IR/TosaProfileCompliance.h   |  2 +-
 .../Tosa/Transforms/TosaProfileCompliance.cpp      |  4 ++--
 .../Tosa/tosa-validation-version-1p1-invalid.mlir  | 14 ++++++++++++++
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 0135a651be481..7c185b3a9a1f2 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -145,7 +145,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<16> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 0b4983bea4bc6..4b61782a468ec 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -687,7 +687,7 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<7>
+llvm::SmallString<16>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
     return {"i" + llvm::utostr(typeInfo.bitWidth)};
@@ -713,5 +713,5 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
     return {"mxint8"};
   }
-  llvm_unreachable("unknown type");
+  return {"unknown<" + llvm::utostr(typeInfo.bitWidth) + ">"};
 }
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 5ce54980c8d61..7e341254e8ece 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -25,3 +25,17 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
        (tensor<1x32x32x8xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  tosa.variable @dynamic_var : tensor<2x?xf64>
+}
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  tosa.variable @static_var : tensor<2x4xf64>
+}

>From 8362bde4d8143ba6062e45f818420aa6981be9ea Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 19:57:52 +0000
Subject: [PATCH 2/3] [mlir][tosa] Validate variable element types

---
 .../Dialect/Tosa/IR/TosaProfileCompliance.h   |  2 +-
 .../Tosa/Transforms/TosaProfileCompliance.cpp |  4 +-
 .../Tosa/Transforms/TosaValidation.cpp        | 54 +++++++++++++------
 .../tosa-validation-version-1p1-invalid.mlir  |  4 +-
 4 files changed, 43 insertions(+), 21 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
index 7c185b3a9a1f2..0135a651be481 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaProfileCompliance.h
@@ -145,7 +145,7 @@ class TosaProfileCompliance {
   SmallVector<StringRef>
   stringifyProfile(const SmallVector<ArrayRef<T>> &profileSet);
 
-  static llvm::SmallString<16> stringifyTypeInfo(const TypeInfo &typeInfo);
+  static llvm::SmallString<7> stringifyTypeInfo(const TypeInfo &typeInfo);
 
 private:
   template <typename T>
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 4b61782a468ec..0b4983bea4bc6 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -687,7 +687,7 @@ SmallVector<StringRef> TosaProfileCompliance::stringifyProfile(
   return debugStrings;
 }
 
-llvm::SmallString<16>
+llvm::SmallString<7>
 TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   if (typeInfo.typeID == mlir::IntegerType::getTypeID()) {
     return {"i" + llvm::utostr(typeInfo.bitWidth)};
@@ -713,5 +713,5 @@ TosaProfileCompliance::stringifyTypeInfo(const TypeInfo &typeInfo) {
   } else if (typeInfo.typeID == tosa::mxint8Type::getTypeID()) {
     return {"mxint8"};
   }
-  return {"unknown<" + llvm::utostr(typeInfo.bitWidth) + ">"};
+  llvm_unreachable("unknown type");
 }
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index 5d4a2affbd7cd..d3abb7c2955c7 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -670,6 +670,10 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
   LogicalResult CheckVariable(Operation *op);
   LogicalResult CheckVariableReadOrWrite(Operation *op);
   bool isValidElementType(Type type, const bool allowUnsigned = false);
+  LogicalResult validateElementType(Operation *op, Type elementTy,
+                                    bool allowUnsigned = false);
+  LogicalResult validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned = false);
 
   SmallVector<
       std::function<LogicalResult(Operation *, const tosa::TargetEnv &)>>
@@ -1500,6 +1504,38 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
   return false;
 }
 
+LogicalResult TosaValidation::validateElementType(Operation *op, Type elementTy,
+                                                  bool allowUnsigned) {
+  if (isValidElementType(elementTy, allowUnsigned))
+    return success();
+
+  return op->emitOpError() << "is not profile-aligned: element type "
+                           << elementTy << " is not legal";
+}
+
+LogicalResult
+TosaValidation::validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned) {
+  for (Value operand : op->getOperands()) {
+    Type elementTy = getElementTypeOrSelf(operand);
+    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  for (Type resultTy : op->getResultTypes()) {
+    Type elementTy = getElementTypeOrSelf(resultTy);
+    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  if (auto variableOp = dyn_cast<tosa::VariableOp>(op)) {
+    if (failed(validateElementType(op, variableOp.getType(), allowUnsigned)))
+      return failure();
+  }
+
+  return success();
+}
+
 void TosaValidation::runOnOperation() {
   ModuleOp modOp = getOperation();
   TosaDialect *tosaDialect = getContext().getLoadedDialect<TosaDialect>();
@@ -1530,22 +1566,8 @@ void TosaValidation::runOnOperation() {
     //   protect rest of code against quantized element types
     const bool allowUnsigned =
         !strictOpSpecAlignment && isa<tosa::RescaleOp>(op);
-    for (Value operand : op->getOperands()) {
-      auto elementTy = getElementTypeOrSelf(operand);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
-    for (Type resultTy : op->getResultTypes()) {
-      auto elementTy = getElementTypeOrSelf(resultTy);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
+    if (failed(validateOperationElementTypes(op, allowUnsigned)))
+      return signalPassFailure();
 
     if (strictOpSpecAlignment &&
         failed(profileComp.checkProfile(op, targetEnv)))
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 7e341254e8ece..19ccef0f52a41 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -29,13 +29,13 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
 // -----
 
 module {
-  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
   tosa.variable @dynamic_var : tensor<2x?xf64>
 }
 
 // -----
 
 module {
-  // expected-error at +1 {{'tosa.variable' op illegal: operation operand/result data types did not align with any profile or extension, got (unknown<64>), did you mean (i8)? Otherwise, please refer to the 'supported data types' for 'tosa.variable' in the specification.}}
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
   tosa.variable @static_var : tensor<2x4xf64>
 }

>From 6e7fcfabbb89c5f3592bb722dc4164ad91808b5f Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 01:12:22 +0000
Subject: [PATCH 3/3] [mlir][tosa] Refine variable element type validation

---
 .../Tosa/Transforms/TosaValidation.cpp        | 39 ++++++++-----------
 .../tosa-validation-version-1p1-invalid.mlir  |  7 ----
 2 files changed, 17 insertions(+), 29 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index d3abb7c2955c7..d211842c72de2 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -669,9 +669,8 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
 
   LogicalResult CheckVariable(Operation *op);
   LogicalResult CheckVariableReadOrWrite(Operation *op);
-  bool isValidElementType(Type type, const bool allowUnsigned = false);
-  LogicalResult validateElementType(Operation *op, Type elementTy,
-                                    bool allowUnsigned = false);
+  LogicalResult validateValidElementType(Operation *op, Type type,
+                                         bool allowUnsigned = false);
   LogicalResult validateOperationElementTypes(Operation *op,
                                               bool allowUnsigned = false);
 
@@ -1472,11 +1471,13 @@ LogicalResult TosaValidation::applyFunctionSignatureCheck(func::FuncOp op) {
   return success();
 }
 
-bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
+LogicalResult TosaValidation::validateValidElementType(Operation *op, Type type,
+                                                       bool allowUnsigned) {
   if (isa<FloatType>(type)) {
-    return isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
-               Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
-               Float6E3M2FNType, Float8E8M0FNUType>(type);
+    if (isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
+            Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
+            Float6E3M2FNType, Float8E8M0FNUType>(type))
+      return success();
   } else if (auto intTy = dyn_cast<IntegerType>(type)) {
     if (intTy.isSignless()) {
       switch (intTy.getWidth()) {
@@ -1487,30 +1488,23 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
       case 32:
       case 48:
       case 64:
-        return true;
+        return success();
       }
     } else if (allowUnsigned && intTy.isUnsigned()) {
       switch (intTy.getWidth()) {
       case 8:
       case 16:
       case 32:
-        return true;
+        return success();
       }
     }
   } else if (isa<tosa::shapeType>(type))
-    return true;
+    return success();
   else if (isa<tosa::mxint8Type>(type))
-    return true;
-  return false;
-}
-
-LogicalResult TosaValidation::validateElementType(Operation *op, Type elementTy,
-                                                  bool allowUnsigned) {
-  if (isValidElementType(elementTy, allowUnsigned))
     return success();
 
-  return op->emitOpError() << "is not profile-aligned: element type "
-                           << elementTy << " is not legal";
+  return op->emitOpError() << "is not profile-aligned: element type " << type
+                           << " is not legal";
 }
 
 LogicalResult
@@ -1518,18 +1512,19 @@ TosaValidation::validateOperationElementTypes(Operation *op,
                                               bool allowUnsigned) {
   for (Value operand : op->getOperands()) {
     Type elementTy = getElementTypeOrSelf(operand);
-    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
       return failure();
   }
 
   for (Type resultTy : op->getResultTypes()) {
     Type elementTy = getElementTypeOrSelf(resultTy);
-    if (failed(validateElementType(op, elementTy, allowUnsigned)))
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
       return failure();
   }
 
   if (auto variableOp = dyn_cast<tosa::VariableOp>(op)) {
-    if (failed(validateElementType(op, variableOp.getType(), allowUnsigned)))
+    if (failed(
+            validateValidElementType(op, variableOp.getType(), allowUnsigned)))
       return failure();
   }
 
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 19ccef0f52a41..20e004077d291 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -32,10 +32,3 @@ module {
   // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
   tosa.variable @dynamic_var : tensor<2x?xf64>
 }
-
-// -----
-
-module {
-  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
-  tosa.variable @static_var : tensor<2x4xf64>
-}


From llvmlistbot at llvm.org  Sat Jul 11 10:38:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 10:38:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a527f8d.4a342cbc.ca844.c322@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/205873

>From 9fc37d63a3c693a4209722e3a9981fb6b40dbc09 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 01:40:27 +0000
Subject: [PATCH] [mlir][tosa] Validate variable element types

---
 .../Tosa/Transforms/TosaValidation.cpp        | 69 ++++++++++++-------
 .../tosa-validation-version-1p1-invalid.mlir  |  7 ++
 2 files changed, 50 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index a3d5d21294b0d..83367fdf4e5f3 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -669,7 +669,10 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
 
   LogicalResult CheckVariable(Operation *op);
   LogicalResult CheckVariableReadOrWrite(Operation *op);
-  bool isValidElementType(Type type, const bool allowUnsigned = false);
+  LogicalResult validateValidElementType(Operation *op, Type type,
+                                         bool allowUnsigned = false);
+  LogicalResult validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned = false);
 
   SmallVector<
       std::function<LogicalResult(Operation *, const tosa::TargetEnv &)>>
@@ -1468,11 +1471,13 @@ LogicalResult TosaValidation::applyFunctionSignatureCheck(func::FuncOp op) {
   return success();
 }
 
-bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
+LogicalResult TosaValidation::validateValidElementType(Operation *op, Type type,
+                                                       bool allowUnsigned) {
   if (isa<FloatType>(type)) {
-    return isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
-               Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
-               Float6E3M2FNType, Float8E8M0FNUType>(type);
+    if (isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
+            Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
+            Float6E3M2FNType, Float8E8M0FNUType>(type))
+      return success();
   } else if (auto intTy = dyn_cast<IntegerType>(type)) {
     if (intTy.isSignless()) {
       switch (intTy.getWidth()) {
@@ -1483,21 +1488,47 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
       case 32:
       case 48:
       case 64:
-        return true;
+        return success();
       }
     } else if (allowUnsigned && intTy.isUnsigned()) {
       switch (intTy.getWidth()) {
       case 8:
       case 16:
       case 32:
-        return true;
+        return success();
       }
     }
   } else if (isa<tosa::shapeType>(type))
-    return true;
+    return success();
   else if (isa<tosa::mxint8Type, tosa::BlockScaledType>(type))
-    return true;
-  return false;
+    return success();
+
+  return op->emitOpError() << "is not profile-aligned: element type " << type
+                           << " is not legal";
+}
+
+LogicalResult
+TosaValidation::validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned) {
+  for (Value operand : op->getOperands()) {
+    Type elementTy = getElementTypeOrSelf(operand);
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  for (Type resultTy : op->getResultTypes()) {
+    Type elementTy = getElementTypeOrSelf(resultTy);
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  if (auto variableOp = dyn_cast<tosa::VariableOp>(op)) {
+    if (failed(
+            validateValidElementType(op, variableOp.getType(), allowUnsigned)))
+      return failure();
+  }
+
+  return success();
 }
 
 void TosaValidation::runOnOperation() {
@@ -1530,22 +1561,8 @@ void TosaValidation::runOnOperation() {
     //   protect rest of code against quantized element types
     const bool allowUnsigned =
         !strictOpSpecAlignment && isa<tosa::RescaleOp>(op);
-    for (Value operand : op->getOperands()) {
-      auto elementTy = getElementTypeOrSelf(operand);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
-    for (Type resultTy : op->getResultTypes()) {
-      auto elementTy = getElementTypeOrSelf(resultTy);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
+    if (failed(validateOperationElementTypes(op, allowUnsigned)))
+      return signalPassFailure();
 
     if (strictOpSpecAlignment &&
         failed(profileComp.checkProfile(op, targetEnv)))
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 5ce54980c8d61..20e004077d291 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -25,3 +25,10 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
        (tensor<1x32x32x8xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
+  tosa.variable @dynamic_var : tensor<2x?xf64>
+}


From llvmlistbot at llvm.org  Sat Jul 11 11:18:08 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 11 Jul 2026 11:18:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add local forward layout
	propagation (PR #208932)
Message-ID: <llvm.org/llvm/llvm-project/pull/208932@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/208932

Backward layout propagation only assigns layouts to values that are (transitively) consumed by an anchor op. A value whose only consumer is, e.g., the next iteration of a loop is left without a layout.

Add a local forward-fill step, run after the backward materialization walk in propagateLayouts(): it visits ops in producer-first order and, for any un-laid-out vector result, infers the layout from the op's already-known operand layouts via a new inferResultLayoutFromSourceForNonAnchorOp dispatcher (covering elementwise, transpose, and shape_cast; other ops are left as TODO), then stamps it with setDistributeLayoutAttr.

>From d83861877d11a1f6e2d57b933be7cbc1a769269b Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 11 Jul 2026 18:16:28 +0000
Subject: [PATCH] [MLIR][XeGPU] Add local forward layout propagation

Backward layout propagation only assigns layouts to values that are
(transitively) consumed by an anchor op. A value whose only consumer is,
e.g., the next iteration of a loop is left without a layout.

Add a local forward-fill step, run after the backward materialization walk
in propagateLayouts(): it visits ops in producer-first order and, for any
un-laid-out vector result, infers the layout from the op's already-known
operand layouts via a new inferResultLayoutFromSourceForNonAnchorOp
dispatcher (covering elementwise, transpose, and shape_cast; other ops are
left as TODO), then stamps it with setDistributeLayoutAttr.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPULayoutImpl.h        |  43 ++++++
 .../XeGPU/Transforms/XeGPULayoutImpl.cpp      | 123 ++++++++++++++++++
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  56 ++++++++
 .../XeGPU/propagate-layout-subgroup.mlir      |  35 +++++
 4 files changed, 257 insertions(+)

diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
index 75fcb156a9d0d..040227996f0ae 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
@@ -83,6 +83,15 @@ dropSgLayoutAndDataOnAttrs(ArrayRef<NamedAttribute> attrs);
 /// any DistributeLayoutAttr found.
 SmallVector<NamedAttribute> dropInstDataOnAttrs(ArrayRef<NamedAttribute> attrs);
 
+//===----------------------------------------------------------------------===//
+// Backward layout inference (result layout -> source layout)
+//===----------------------------------------------------------------------===//
+//
+// The infer*SourceLayout helpers below derive the layout of an operation's
+// source operand from the layout of its result. They implement the per-op
+// transfer functions used by the backward layout propagation analysis, which
+// flows layouts from anchor ops (dpas, store_nd, ...) back to their producers.
+
 /// Infers the source layout attribute for a broadcast operation given the
 /// result layout attribute, result shape, and source shape.
 DistributeLayoutAttr inferBroadcastSourceLayout(DistributeLayoutAttr resLayout,
@@ -159,6 +168,40 @@ DistributeLayoutAttr
 inferSourceLayoutFromResultForNonAnchorOp(OpOperand &operand,
                                           DistributeLayoutAttr resLayout);
 
+//===----------------------------------------------------------------------===//
+// Forward layout inference (source layout -> result layout)
+//===----------------------------------------------------------------------===//
+//
+// The infer*ResultLayout helpers below are the forward counterparts of the
+// infer*SourceLayout helpers above: given the layout of an operation's source
+// operand they derive the layout of its result. They are used by the local
+// forward-fill step in XeGPUPropagateLayout that assigns layouts to values not
+// reached by the backward propagation analysis (e.g. loop-carried values whose
+// only consumer is the next iteration).
+
+/// Infers the result layout attribute for a transpose operation given the
+/// source layout attribute and permutation. Inverse of
+/// inferTransposeSourceLayout.
+DistributeLayoutAttr inferTransposeResultLayout(DistributeLayoutAttr srcLayout,
+                                                ArrayRef<int64_t> permutation);
+
+/// Infers the result layout attribute for a shape cast operation given the
+/// source layout attribute, source shape, and result shape. Inverse of
+/// inferShapeCastSourceLayout. Returns nullptr for shape-cast patterns whose
+/// forward direction is ambiguous (e.g. unit-dim expansion).
+DistributeLayoutAttr inferShapeCastResultLayout(DistributeLayoutAttr srcLayout,
+                                                ArrayRef<int64_t> srcShape,
+                                                ArrayRef<int64_t> resShape);
+
+/// Infers the result layout attribute for a non-anchor operation from the
+/// layouts of its source operands (the forward counterpart of
+/// inferSourceLayoutFromResultForNonAnchorOp). `operandLayouts` is indexed by
+/// operand number; entries may be null for operands without a known layout.
+/// Returns nullptr when no forward rule applies (the result is then left
+/// un-laid-out).
+DistributeLayoutAttr inferResultLayoutFromSourceForNonAnchorOp(
+    Operation *op, ArrayRef<DistributeLayoutAttr> operandLayouts);
+
 /// Note on the `consumerLayout` argument used by the consumer-driven setup* /
 /// complete* helpers below:
 ///
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 8da151a333fb3..d68f81c7fd3cd 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -835,6 +835,129 @@ xegpu::inferShapeCastSourceLayout(xegpu::DistributeLayoutAttr resLayout,
   return nullptr;
 }
 
+//===----------------------------------------------------------------------===//
+// Forward layout inference (source layout -> result layout)
+//===----------------------------------------------------------------------===//
+
+/// Infers the result layout attribute for a transpose operation given the
+/// source layout attribute and permutation.
+///
+/// vector.transpose semantics is `result[i] = source[permutation[i]]`, so
+/// `result_layout[i] = source_layout[permutation[i]]`, which is exactly
+/// `srcLayout.transposeDims(permutation)`. This is the inverse of
+/// inferTransposeSourceLayout (which applies the inverse permutation).
+xegpu::DistributeLayoutAttr
+xegpu::inferTransposeResultLayout(xegpu::DistributeLayoutAttr srcLayout,
+                                  ArrayRef<int64_t> permutation) {
+  return srcLayout.transposeDims(permutation);
+}
+
+/// Infers the result layout attribute for a shape cast operation given the
+/// source layout attribute, source shape, and result shape. This is the
+/// inverse of inferShapeCastSourceLayout: a dim-split (src -> res) is undone by
+/// collapsing the split groups, and a dim-collapse (src -> res) is undone by
+/// expanding the collapsed groups. The unit-dim-expansion case is not inverted
+/// here because recovering which result dims are the expanded unit dims would
+/// require the SliceAttr the backward direction produces; such patterns return
+/// nullptr (leaving the result un-laid-out).
+xegpu::DistributeLayoutAttr
+xegpu::inferShapeCastResultLayout(xegpu::DistributeLayoutAttr srcLayout,
+                                  ArrayRef<int64_t> srcShape,
+                                  ArrayRef<int64_t> resShape) {
+  // Case: source dims were split into result dims (forward of use case 2 in
+  // inferShapeCastSourceLayout). Undo by expanding each source dim into its
+  // group of result dims.
+  SmallVector<SmallVector<int64_t>> splitDimGroups;
+  if (xegpu::matchSplitDimExpansion(srcShape, resShape, splitDimGroups)) {
+    auto resLayout = srcLayout;
+    // Process source dims from innermost to outermost so that expanding a dim
+    // does not shift the indices of dims not yet processed.
+    for (int64_t srcIdx = static_cast<int64_t>(splitDimGroups.size()) - 1;
+         srcIdx >= 0; --srcIdx) {
+      ArrayRef<int64_t> resDims = splitDimGroups[srcIdx];
+      if (resDims.size() <= 1)
+        continue;
+      SmallVector<int64_t> targetShape;
+      targetShape.reserve(resDims.size());
+      for (int64_t d : resDims)
+        targetShape.push_back(resShape[d]);
+      resLayout = resLayout.expandDim(srcIdx, targetShape);
+    }
+    return resLayout;
+  }
+
+  // Case: source dims were collapsed into result dims (forward of use case 3).
+  // Undo by collapsing each group of source dims into its single result dim.
+  SmallVector<SmallVector<int64_t>> collapseDims;
+  if (xegpu::matchDimCollapse(srcShape, resShape, collapseDims)) {
+    auto resLayout = srcLayout;
+    // Process result dims from innermost to outermost so that collapsing a
+    // group does not shift the indices of groups not yet processed.
+    for (int64_t dstIdx = static_cast<int64_t>(collapseDims.size()) - 1;
+         dstIdx >= 0; --dstIdx) {
+      ArrayRef<int64_t> srcDims = collapseDims[dstIdx];
+      // A result dim with no backing source dims is a trailing/leading unit
+      // dim; its forward inference is ambiguous, so bail out.
+      if (srcDims.empty())
+        return nullptr;
+      if (srcDims.size() == 1)
+        continue;
+      resLayout = resLayout.collapseDims(llvm::to_vector(srcDims));
+    }
+    return resLayout;
+  }
+
+  return nullptr;
+}
+
+/// Infers the result layout attribute for a non-anchor operation from the
+/// layouts of its source operands. Forward counterpart of
+/// inferSourceLayoutFromResultForNonAnchorOp.
+xegpu::DistributeLayoutAttr xegpu::inferResultLayoutFromSourceForNonAnchorOp(
+    Operation *op, ArrayRef<xegpu::DistributeLayoutAttr> operandLayouts) {
+  if (op->getNumResults() != 1)
+    return nullptr;
+
+  // For vector::TransposeOp, infer the result layout from the source layout.
+  if (auto transpose = dyn_cast<vector::TransposeOp>(op)) {
+    if (!operandLayouts[0])
+      return nullptr;
+    return xegpu::inferTransposeResultLayout(operandLayouts[0],
+                                             transpose.getPermutation());
+  }
+
+  // For vector::ShapeCastOp, infer the result layout from the source layout.
+  if (auto shapeCast = dyn_cast<vector::ShapeCastOp>(op)) {
+    if (!operandLayouts[0])
+      return nullptr;
+    return xegpu::inferShapeCastResultLayout(
+        operandLayouts[0], shapeCast.getSourceVectorType().getShape(),
+        shapeCast.getResultVectorType().getShape());
+  }
+
+  // For elementwise operations, all operands and the result share the same
+  // layout. Use the first operand that carries a layout.
+  if (OpTrait::hasElementwiseMappableTraits(op)) {
+    for (xegpu::DistributeLayoutAttr layout : operandLayouts)
+      if (layout)
+        return layout;
+    return nullptr;
+  }
+
+  // TODO: add forward inference rules for the remaining ops; their result is
+  // left un-laid-out until then.
+  //  - vector::BroadcastOp: the forward direction is under-determined. The
+  //    backward rule (inferBroadcastSourceLayout) either sets broadcast dims to
+  //    unit data (losing the original data on those dims) or wraps the result
+  //    in a SliceAttr; neither is generally invertible from the source layout
+  //    alone, so a forward rule must decide how to distribute the new/stretched
+  //    dims.
+  //  - vector::BitCastOp, vector::MultiDimReductionOp / vector::ReductionOp,
+  //    vector::InterleaveOp / vector::DeinterleaveOp, and the insert / extract
+  //    / strided-slice family.
+  return nullptr;
+}
+
 /// Infers the layout attribute for mask and offset operand for Chunked load
 /// and store, given the anchor layout attribute for the value being load/store.
 xegpu::DistributeLayoutAttr xegpu::inferMaskOffsetLayoutForScatterIO(
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1b1abcc4b0e4f..68f8abd1ed92d 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1695,6 +1695,57 @@ ResolveLayoutConflicts::resolveTensorDescConsumer(OpOperand &operand) {
 }
 
 using GetLayoutFnTy = function_ref<xegpu::DistributeLayoutAttr(Value)>;
+
+/// Local forward layout propagation, run after the backward propagation
+/// analysis has materialized its layouts. Backward propagation only assigns
+/// layouts to values that are (transitively) consumed by an anchor op; a value
+/// whose only consumer is, e.g., the next iteration of a loop (its producing op
+/// is not on any anchor's backward slice) is left without a layout. This walk
+/// visits ops in producer-first (forward) order and, for any vector result that
+/// still lacks a layout, infers it from the already-known layouts of the op's
+/// operands via inferResultLayoutFromSourceForNonAnchorOp, then stamps it with
+/// setDistributeLayoutAttr.
+///
+/// A single forward pass suffices: forward-only values form producer ->
+/// consumer chains that the pre-order walk visits in dependency order, so an
+/// operand's layout is already assigned by the time its consumer is visited.
+static void forwardFillLayouts(Operation *root) {
+  root->walk([&](Operation *op) {
+    // Anchor ops carry authoritative layouts; region ops and their terminators
+    // are handled by the control-flow propagation; function ops carry no result
+    // layouts here.
+    if (isa<xegpu::AnchorLayoutInterface, RegionBranchOpInterface,
+            RegionBranchTerminatorOpInterface, FunctionOpInterface>(op))
+      return;
+    if (op->getNumResults() != 1)
+      return;
+    OpResult result = op->getResult(0);
+    if (!isa<VectorType>(result.getType()))
+      return;
+    // Skip results that already have a layout (from backward propagation or an
+    // earlier forward-fill step).
+    if (xegpu::getDistributeLayoutAttr(result))
+      return;
+
+    // Gather operand layouts, indexed by operand number.
+    SmallVector<xegpu::DistributeLayoutAttr> operandLayouts;
+    operandLayouts.reserve(op->getNumOperands());
+    bool anyAssigned = false;
+    for (Value operand : op->getOperands()) {
+      auto layout = xegpu::getDistributeLayoutAttr(operand);
+      operandLayouts.push_back(layout);
+      anyAssigned |= (layout != nullptr);
+    }
+    if (!anyAssigned)
+      return;
+
+    xegpu::DistributeLayoutAttr layout =
+        xegpu::inferResultLayoutFromSourceForNonAnchorOp(op, operandLayouts);
+    if (layout)
+      xegpu::setDistributeLayoutAttr(result, layout);
+  });
+}
+
 /// Update an operation with the layout of its results. If the result type is
 /// a vector type, a temporary layout attribute is added to the operation. If
 /// the result type is a tensor descriptor type, the type is updated with the
@@ -1918,6 +1969,11 @@ LogicalResult xegpu::propagateLayouts(OpBuilder &builder, Operation *target,
   if (walkResult.wasInterrupted())
     return failure();
 
+  // Backward propagation only reaches values consumed by anchor ops. Run a
+  // local forward pass to fill in layouts for values it could not reach (e.g.
+  // loop-carried values whose only consumer is the next iteration).
+  forwardFillLayouts(op);
+
   return success();
 }
 
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..5c29695bafa31 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -672,3 +672,38 @@ gpu.module @test {
     gpu.return
   }
 }
+
+// -----
+// Forward layout fill: %sum is a loop-carried value whose only consumer is the
+// next iteration (via iter_arg %acc), so backward propagation from anchor ops
+// never reaches it. The local forward pass derives its layout from the operand
+// %acc (elementwise rule), and the scf.for's second result / yield inherit it.
+gpu.module @test {
+// CHECK-LABEL: gpu.func @forward_fill_loop_carried(
+gpu.func @forward_fill_loop_carried(%arg0: memref<128x64xf16>, %arg1: memref<64x64xf16>, %arg2: memref<128x64xf32>) kernel attributes {known_block_size = array<i32: 128, 1, 1>} {
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c128 = arith.constant 128 : index
+  %cst = arith.constant dense<0.000000e+00> : vector<128x64xf32>
+  %cst_0 = arith.constant dense<0.000000e+00> : vector<128xf32>
+  %0 = xegpu.create_nd_tdesc %arg0 : memref<128x64xf16> -> !xegpu.tensor_desc<128x64xf16>
+  %1 = xegpu.load_nd %0[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>}> : !xegpu.tensor_desc<128x64xf16> -> vector<128x64xf16>
+  %2 = xegpu.create_nd_tdesc %arg1 : memref<64x64xf16> -> !xegpu.tensor_desc<64x64xf16>
+  %3 = xegpu.load_nd %2[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [1, 1], sg_data = [64, 64]>}> : !xegpu.tensor_desc<64x64xf16> -> vector<64x64xf16>
+  // CHECK: scf.for
+  %4:2 = scf.for %arg4 = %c0 to %c128 step %c64 iter_args(%acc0 = %cst, %acc = %cst_0) -> (vector<128x64xf32>, vector<128xf32>) {
+    %8 = xegpu.dpas %1, %3, %acc0 {layout_a = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, layout_b = #xegpu.layout<sg_layout = [1, 1], sg_data = [64, 64]>, layout_cd = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>} : vector<128x64xf16>, vector<64x64xf16>, vector<128x64xf32> -> vector<128x64xf32>
+    %9 = vector.broadcast %acc : vector<128xf32> to vector<64x128xf32>
+    %10 = vector.transpose %9, [1, 0] : vector<64x128xf32> to vector<128x64xf32>
+    %11 = arith.mulf %8, %10 : vector<128x64xf32>
+    // The forward pass assigns %sum a layout inherited from %acc.
+    // CHECK: arith.addf
+    // CHECK-SAME: layout_result_0 = #xegpu.slice<#xegpu.layout<sg_layout = [1, 8], sg_data = [64, 16], order = [0, 1]>, dims = [0]>
+    %sum = arith.addf %acc, %acc : vector<128xf32>
+    scf.yield %11, %sum : vector<128x64xf32>, vector<128xf32>
+  }
+  %5 = xegpu.create_nd_tdesc %arg2 : memref<128x64xf32> -> !xegpu.tensor_desc<128x64xf32>
+  xegpu.store_nd %4#0, %5[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>}> : vector<128x64xf32>, !xegpu.tensor_desc<128x64xf32>
+  gpu.return
+}
+}


From llvmlistbot at llvm.org  Sat Jul 11 11:37:52 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:37:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
Message-ID: <llvm.org/llvm/llvm-project/pull/208934@github.com>


https://github.com/makslevental created https://github.com/llvm/llvm-project/pull/208934

Exposes `TypeConverter::addSourceMaterialization` and `addTargetMaterialization` through the MLIR C API, continuing the buildout of the dialect-conversion C bindings (follows #206146 and #206161).

## Changes

`mlir/include/mlir-c/Rewrite.h` / `mlir/lib/CAPI/Transforms/Rewrite.cpp`:
- `MlirTypeConverterSourceMaterializationCallback` — `MlirValue(MlirRewriterBase, MlirType outputType, intptr_t nInputs, MlirValue *inputs, MlirLocation, void *userData)`. The builder is passed as an `MlirRewriterBase` (the conversion driver always invokes materializations with a `RewriterBase`). Returning a null `MlirValue` signals the materialization declined, so another registered one may be attempted.
- `MlirTypeConverterTargetMaterializationCallback` — like the source callback but additionally receives `originalType`, the original type of the SSA value being materialized. This selects the 5-argument C++ target-materialization overload, which is what makes the original type observable from C (it cannot in general be recovered from `outputType`/`inputs`).
- `mlirTypeConverterAddSourceMaterialization` / `mlirTypeConverterAddTargetMaterialization`.

Adds end-to-end tests in `mlir/test/CAPI/rewrite.c` covering source and target materialization, `originalType` observation, and the null-return decline path.


>From 496a4a2d6b35cb97e7493d1d1612e6cb9e7b6781 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:57:46 -0700
Subject: [PATCH 1/2] [mlir-c] Add TypeConverter source materialization

---
 mlir/include/mlir-c/Rewrite.h        |  16 ++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  32 +++++++
 mlir/test/CAPI/rewrite.c             | 125 +++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..2d7ed3e4b7111 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,22 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..2fb8eff5ad79d 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,38 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..8e5b98abdb883 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,130 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +942,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 12609605057480049675bea96fc6de538c16141d Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:59:12 -0700
Subject: [PATCH 2/2] [mlir-c] Add TypeConverter target materialization

---
 mlir/include/mlir-c/Rewrite.h        |  7 +++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  9 +++
 mlir/test/CAPI/rewrite.c             | 90 ++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 2d7ed3e4b7111..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -621,6 +621,13 @@ MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
     MlirTypeConverter typeConverter,
     MlirTypeConverterMaterializationCallback callback, void *userData);
 
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 2fb8eff5ad79d..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -701,6 +701,15 @@ void mlirTypeConverterAddSourceMaterialization(
           wrapMaterializationCallback(callback, userData));
 }
 
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 8e5b98abdb883..c1624c54eb461 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -926,6 +926,95 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
   fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
 }
 
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -943,6 +1032,7 @@ int main(void) {
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
   testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;


From llvmlistbot at llvm.org  Sat Jul 11 11:38:17 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:38:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter materialization
 and 1:N dialect-conversion bindings (PR #206528)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206528@github.com>
Message-ID: <6a528d99.d681cab9.333b2b.0875@mx.google.com>


https://github.com/makslevental closed https://github.com/llvm/llvm-project/pull/206528

From llvmlistbot at llvm.org  Sat Jul 11 11:38:18 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:38:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter materialization
 and 1:N dialect-conversion bindings (PR #206528)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206528@github.com>
Message-ID: <6a528d9a.92d2873c.258297.480c@mx.google.com>


makslevental wrote:

Split into two stacked PRs for easier review:
- #208934 — TypeConverter source and target (1:1) materialization
- #208935 — 1:N conversion and materialization bindings (stacked on #208934)

Closing this one in favor of those.

https://github.com/llvm/llvm-project/pull/206528

From llvmlistbot at llvm.org  Sat Jul 11 11:46:48 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:46:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a528f98.9edb5c9d.208698.7cb9@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 11:50:09 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:50:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a529061.f1c1923d.991d1.dbfc@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 11:55:31 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 11:55:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5291a3.492e23e4.157d47.63de@mx.google.com>


https://github.com/makslevental updated https://github.com/llvm/llvm-project/pull/208934

>From 496a4a2d6b35cb97e7493d1d1612e6cb9e7b6781 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:57:46 -0700
Subject: [PATCH 1/3] [mlir-c] Add TypeConverter source materialization

---
 mlir/include/mlir-c/Rewrite.h        |  16 ++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  32 +++++++
 mlir/test/CAPI/rewrite.c             | 125 +++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..2d7ed3e4b7111 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,22 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..2fb8eff5ad79d 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,38 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..8e5b98abdb883 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,130 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +942,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 12609605057480049675bea96fc6de538c16141d Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:59:12 -0700
Subject: [PATCH 2/3] [mlir-c] Add TypeConverter target materialization

---
 mlir/include/mlir-c/Rewrite.h        |  7 +++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  9 +++
 mlir/test/CAPI/rewrite.c             | 90 ++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 2d7ed3e4b7111..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -621,6 +621,13 @@ MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
     MlirTypeConverter typeConverter,
     MlirTypeConverterMaterializationCallback callback, void *userData);
 
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 2fb8eff5ad79d..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -701,6 +701,15 @@ void mlirTypeConverterAddSourceMaterialization(
           wrapMaterializationCallback(callback, userData));
 }
 
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 8e5b98abdb883..c1624c54eb461 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -926,6 +926,95 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
   fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
 }
 
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -943,6 +1032,7 @@ int main(void) {
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
   testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 7fb9b326d964f82016e7553d911d4c479798e1f4 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 11 Jul 2026 11:55:10 -0700
Subject: [PATCH 3/3] [mlir-c] Tighten materialization test CHECKs to pin full
 module body

Use CHECK-NEXT to match the entire module body (module { ... }) rather
than loose CHECK lines, so the tests also assert the absence of any
stray ops -- e.g. a leftover builtin.unrealized_conversion_cast -- that
a loose CHECK would silently allow.
---
 mlir/test/CAPI/rewrite.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index c1624c54eb461..6d87d0236061b 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -911,9 +911,11 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
-  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK-NEXT:   "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);
@@ -1000,9 +1002,11 @@ void testTypeConverterTargetMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
-  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK-NEXT:   "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);


From llvmlistbot at llvm.org  Sat Jul 11 12:05:13 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 12:05:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5293e9.9edb5c9d.208698.abe2@mx.google.com>


https://github.com/makslevental ready_for_review https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 12:05:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:05:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a529408.f1c1923d.991d1.e727@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Maksim Levental (makslevental)

<details>
<summary>Changes</summary>

Exposes `TypeConverter::addSourceMaterialization` and `addTargetMaterialization` through the MLIR C API, continuing the buildout of the dialect-conversion C bindings (follows #<!-- -->206146 and #<!-- -->206161).

Assisted by: Claude

---
Full diff: https://github.com/llvm/llvm-project/pull/208934.diff


3 Files Affected:

- (modified) mlir/include/mlir-c/Rewrite.h (+23) 
- (modified) mlir/lib/CAPI/Transforms/Rewrite.cpp (+41) 
- (modified) mlir/test/CAPI/rewrite.c (+219) 


``````````diff
diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,29 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,47 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..6d87d0236061b 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,223 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK-NEXT:   "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK-NEXT: }
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK-NEXT:   "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK-NEXT: }
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +1035,8 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

``````````

</details>


https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 12:07:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:07:43 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a52947f.e5907290.281f16.6ede@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206215

>From 2fc18628d1c3420e7399ad967866ce401fdf6fbe Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 12:12:37 +0000
Subject: [PATCH 1/2] [MLIR] Diagnose missing call result arrow

---
 clang/test/CIR/IR/invalid-call.cir         |  9 +++++++++
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp |  3 +++
 mlir/lib/Interfaces/CallInterfaces.cpp     |  3 ++-
 mlir/test/Dialect/LLVMIR/invalid.mlir      | 10 ++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/clang/test/CIR/IR/invalid-call.cir b/clang/test/CIR/IR/invalid-call.cir
index a9c7e38f73af6..11b5c4425549e 100644
--- a/clang/test/CIR/IR/invalid-call.cir
+++ b/clang/test/CIR/IR/invalid-call.cir
@@ -10,6 +10,15 @@ cir.func @f1() {
 
 // -----
 
+cir.func private @f_missing_result_arrow()
+cir.func @call_missing_empty_result() {
+  // expected-error @+2 {{expected '->' followed by function result list}}
+  cir.call @f_missing_result_arrow() : ()
+  cir.return
+}
+
+// -----
+
 !u32i = !cir.int<u, 32>
 
 cir.func private @f2()
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 58f569abff8ea..0cc007433ac0e 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
+      return failure();
     if (isDirect)
       return parser.emitError(trailingTypesLoc,
                               "expected direct call to have 1 trailing types");
diff --git a/mlir/lib/Interfaces/CallInterfaces.cpp b/mlir/lib/Interfaces/CallInterfaces.cpp
index e8ed4b339a0cb..af670cb62c1c6 100644
--- a/mlir/lib/Interfaces/CallInterfaces.cpp
+++ b/mlir/lib/Interfaces/CallInterfaces.cpp
@@ -72,7 +72,8 @@ ParseResult call_interface_impl::parseFunctionSignature(
     return call_interface_impl::parseFunctionResultList(parser, resultTypes,
                                                         resultAttrs);
   if (mustParseEmptyResult)
-    return failure();
+    return parser.emitError(parser.getCurrentLocation(),
+                            "expected '->' followed by function result list");
   return success();
 }
 
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index d5ea5c8de862e..1d0bb0b6ef999 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -241,6 +241,16 @@ func.func @call_missing_ptr_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 
 // -----
 
+func.func private @callee()
+
+func.func @call_missing_empty_result() {
+  // expected-error at +2 {{expected '->' followed by function result list}}
+  llvm.call @callee() : ()
+  llvm.return
+}
+
+// -----
+
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {

>From 7c84efca9a10c9f8cbbf974bea754e088388e5fd Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 02:49:36 +0000
Subject: [PATCH 2/2] [MLIR] Unify call signature diagnostics

---
 flang/lib/Optimizer/Dialect/FIROps.cpp        |  2 +-
 flang/test/Fir/invalid.fir                    |  8 ++++++++
 mlir/include/mlir/Interfaces/CallInterfaces.h |  1 +
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp    | 12 ++----------
 mlir/test/Dialect/LLVMIR/invalid.mlir         |  6 ++----
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Optimizer/Dialect/FIROps.cpp b/flang/lib/Optimizer/Dialect/FIROps.cpp
index ba047e71d6aa3..6f0c8eecfd462 100644
--- a/flang/lib/Optimizer/Dialect/FIROps.cpp
+++ b/flang/lib/Optimizer/Dialect/FIROps.cpp
@@ -1714,7 +1714,7 @@ mlir::ParseResult fir::CallOp::parse(mlir::OpAsmParser &parser,
   llvm::SmallVector<mlir::DictionaryAttr> resultAttrs;
   if (mlir::call_interface_impl::parseFunctionSignature(
           parser, argTypes, argAttrs, resTypes, resultAttrs))
-    return parser.emitError(parser.getNameLoc(), "expected function type");
+    return mlir::failure();
   mlir::FunctionType funcType =
       mlir::FunctionType::get(parser.getContext(), argTypes, resTypes);
   if (isDirect) {
diff --git a/flang/test/Fir/invalid.fir b/flang/test/Fir/invalid.fir
index 5471bcd8acc29..937ce891bfef0 100644
--- a/flang/test/Fir/invalid.fir
+++ b/flang/test/Fir/invalid.fir
@@ -1,5 +1,13 @@
 // RUN: fir-opt -split-input-file -verify-diagnostics --strict-fir-volatile-verifier %s
 
+func.func @call_missing_empty_result() {
+  // expected-error at +2{{expected '->' followed by function result list}}
+  fir.call @callee() : ()
+  return
+}
+
+// -----
+
 // expected-error at +1{{custom op 'fir.string_lit' must have character type}}
 %0 = fir.string_lit "Hello, World!"(13) : !fir.int<32>
 
diff --git a/mlir/include/mlir/Interfaces/CallInterfaces.h b/mlir/include/mlir/Interfaces/CallInterfaces.h
index 2bf3a3ca5f8a8..02c04c0f8dd1e 100644
--- a/mlir/include/mlir/Interfaces/CallInterfaces.h
+++ b/mlir/include/mlir/Interfaces/CallInterfaces.h
@@ -53,6 +53,7 @@ parseFunctionResultList(OpAsmParser &parser, SmallVectorImpl<Type> &resultTypes,
 /// signatures containing SSA region arguments (to parse these signatures, use
 /// function_interface_impl::parseFunctionSignature). When
 /// `mustParseEmptyResult`, `-> ()` is expected when there is no result type.
+/// Returns failure after emitting a diagnostic if the signature is malformed.
 ///
 ///   no-ssa-function-signature ::= `(` no-ssa-function-arg-list `)`
 ///                               -> function-result-list
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 0cc007433ac0e..223852d3828c4 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,17 +1388,9 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
-  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
-                                                  resTypes, resultAttrs)) {
-    if (parser.getCurrentLocation() != signatureLoc)
-      return failure();
-    if (isDirect)
-      return parser.emitError(trailingTypesLoc,
-                              "expected direct call to have 1 trailing types");
-    return parser.emitError(trailingTypesLoc,
-                            "expected trailing function type");
-  }
+                                                  resTypes, resultAttrs))
+    return failure();
 
   if (resTypes.size() > 1)
     return parser.emitError(trailingTypesLoc,
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index 1d0bb0b6ef999..d44fa7ebfc4b1 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -254,8 +254,7 @@ func.func @call_missing_empty_result() {
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected direct call to have 1 trailing type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call @standard_func_callee(%arg) : !llvm.ptr, (i8) -> (i8)
   llvm.return
 }
@@ -271,8 +270,7 @@ func.func @call_non_pointer_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 // -----
 
 func.func @call_non_function_type(%callee : !llvm.ptr, %arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected trailing function type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call %callee(%arg) : !llvm.ptr, !llvm.func<i8 (i8)>
   llvm.return
 }


From llvmlistbot at llvm.org  Sat Jul 11 12:08:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:08:17 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a5294a1.481afadf.196d57.8264@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Sat Jul 11 12:23:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:23:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a52984f.e624eb8d.39cd4e.ef9b@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Sat Jul 11 12:24:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:24:44 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a52987c.4987c784.63cbc.3a90@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206215

>From 2fc18628d1c3420e7399ad967866ce401fdf6fbe Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 12:12:37 +0000
Subject: [PATCH 1/2] [MLIR] Diagnose missing call result arrow

---
 clang/test/CIR/IR/invalid-call.cir         |  9 +++++++++
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp |  3 +++
 mlir/lib/Interfaces/CallInterfaces.cpp     |  3 ++-
 mlir/test/Dialect/LLVMIR/invalid.mlir      | 10 ++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/clang/test/CIR/IR/invalid-call.cir b/clang/test/CIR/IR/invalid-call.cir
index a9c7e38f73af6..11b5c4425549e 100644
--- a/clang/test/CIR/IR/invalid-call.cir
+++ b/clang/test/CIR/IR/invalid-call.cir
@@ -10,6 +10,15 @@ cir.func @f1() {
 
 // -----
 
+cir.func private @f_missing_result_arrow()
+cir.func @call_missing_empty_result() {
+  // expected-error @+2 {{expected '->' followed by function result list}}
+  cir.call @f_missing_result_arrow() : ()
+  cir.return
+}
+
+// -----
+
 !u32i = !cir.int<u, 32>
 
 cir.func private @f2()
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 58f569abff8ea..0cc007433ac0e 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
+      return failure();
     if (isDirect)
       return parser.emitError(trailingTypesLoc,
                               "expected direct call to have 1 trailing types");
diff --git a/mlir/lib/Interfaces/CallInterfaces.cpp b/mlir/lib/Interfaces/CallInterfaces.cpp
index e8ed4b339a0cb..af670cb62c1c6 100644
--- a/mlir/lib/Interfaces/CallInterfaces.cpp
+++ b/mlir/lib/Interfaces/CallInterfaces.cpp
@@ -72,7 +72,8 @@ ParseResult call_interface_impl::parseFunctionSignature(
     return call_interface_impl::parseFunctionResultList(parser, resultTypes,
                                                         resultAttrs);
   if (mustParseEmptyResult)
-    return failure();
+    return parser.emitError(parser.getCurrentLocation(),
+                            "expected '->' followed by function result list");
   return success();
 }
 
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index d5ea5c8de862e..1d0bb0b6ef999 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -241,6 +241,16 @@ func.func @call_missing_ptr_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 
 // -----
 
+func.func private @callee()
+
+func.func @call_missing_empty_result() {
+  // expected-error at +2 {{expected '->' followed by function result list}}
+  llvm.call @callee() : ()
+  llvm.return
+}
+
+// -----
+
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {

>From 7c84efca9a10c9f8cbbf974bea754e088388e5fd Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 02:49:36 +0000
Subject: [PATCH 2/2] [MLIR] Unify call signature diagnostics

---
 flang/lib/Optimizer/Dialect/FIROps.cpp        |  2 +-
 flang/test/Fir/invalid.fir                    |  8 ++++++++
 mlir/include/mlir/Interfaces/CallInterfaces.h |  1 +
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp    | 12 ++----------
 mlir/test/Dialect/LLVMIR/invalid.mlir         |  6 ++----
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Optimizer/Dialect/FIROps.cpp b/flang/lib/Optimizer/Dialect/FIROps.cpp
index ba047e71d6aa3..6f0c8eecfd462 100644
--- a/flang/lib/Optimizer/Dialect/FIROps.cpp
+++ b/flang/lib/Optimizer/Dialect/FIROps.cpp
@@ -1714,7 +1714,7 @@ mlir::ParseResult fir::CallOp::parse(mlir::OpAsmParser &parser,
   llvm::SmallVector<mlir::DictionaryAttr> resultAttrs;
   if (mlir::call_interface_impl::parseFunctionSignature(
           parser, argTypes, argAttrs, resTypes, resultAttrs))
-    return parser.emitError(parser.getNameLoc(), "expected function type");
+    return mlir::failure();
   mlir::FunctionType funcType =
       mlir::FunctionType::get(parser.getContext(), argTypes, resTypes);
   if (isDirect) {
diff --git a/flang/test/Fir/invalid.fir b/flang/test/Fir/invalid.fir
index 5471bcd8acc29..937ce891bfef0 100644
--- a/flang/test/Fir/invalid.fir
+++ b/flang/test/Fir/invalid.fir
@@ -1,5 +1,13 @@
 // RUN: fir-opt -split-input-file -verify-diagnostics --strict-fir-volatile-verifier %s
 
+func.func @call_missing_empty_result() {
+  // expected-error at +2{{expected '->' followed by function result list}}
+  fir.call @callee() : ()
+  return
+}
+
+// -----
+
 // expected-error at +1{{custom op 'fir.string_lit' must have character type}}
 %0 = fir.string_lit "Hello, World!"(13) : !fir.int<32>
 
diff --git a/mlir/include/mlir/Interfaces/CallInterfaces.h b/mlir/include/mlir/Interfaces/CallInterfaces.h
index 2bf3a3ca5f8a8..02c04c0f8dd1e 100644
--- a/mlir/include/mlir/Interfaces/CallInterfaces.h
+++ b/mlir/include/mlir/Interfaces/CallInterfaces.h
@@ -53,6 +53,7 @@ parseFunctionResultList(OpAsmParser &parser, SmallVectorImpl<Type> &resultTypes,
 /// signatures containing SSA region arguments (to parse these signatures, use
 /// function_interface_impl::parseFunctionSignature). When
 /// `mustParseEmptyResult`, `-> ()` is expected when there is no result type.
+/// Returns failure after emitting a diagnostic if the signature is malformed.
 ///
 ///   no-ssa-function-signature ::= `(` no-ssa-function-arg-list `)`
 ///                               -> function-result-list
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 0cc007433ac0e..223852d3828c4 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,17 +1388,9 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
-  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
-                                                  resTypes, resultAttrs)) {
-    if (parser.getCurrentLocation() != signatureLoc)
-      return failure();
-    if (isDirect)
-      return parser.emitError(trailingTypesLoc,
-                              "expected direct call to have 1 trailing types");
-    return parser.emitError(trailingTypesLoc,
-                            "expected trailing function type");
-  }
+                                                  resTypes, resultAttrs))
+    return failure();
 
   if (resTypes.size() > 1)
     return parser.emitError(trailingTypesLoc,
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index 1d0bb0b6ef999..d44fa7ebfc4b1 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -254,8 +254,7 @@ func.func @call_missing_empty_result() {
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected direct call to have 1 trailing type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call @standard_func_callee(%arg) : !llvm.ptr, (i8) -> (i8)
   llvm.return
 }
@@ -271,8 +270,7 @@ func.func @call_non_pointer_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 // -----
 
 func.func @call_non_function_type(%callee : !llvm.ptr, %arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected trailing function type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call %callee(%arg) : !llvm.ptr, !llvm.func<i8 (i8)>
   llvm.return
 }


From llvmlistbot at llvm.org  Sat Jul 11 12:37:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:37:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Coordinate branch argument cleanup (PR
	#208940)
Message-ID: <llvm.org/llvm/llvm-project/pull/208940@github.com>


https://github.com/mygitljf created https://github.com/llvm/llvm-project/pull/208940

I made branch argument cleanup consistent across all incoming control-flow paths, so unreachable paths no longer leave malformed IR. The change also preserves internally produced arguments and keeps surviving uses valid with poison values.
Fixes #205984 

>From b22ea028f463f0da3fdf71aa211bca11f6cf4f07 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 03:36:50 +0000
Subject: [PATCH] [mlir] Coordinate branch argument cleanup

---
 mlir/lib/Transforms/RemoveDeadValues.cpp     | 80 ++++++++++----------
 mlir/test/Transforms/remove-dead-values.mlir | 63 +++++++++++++++
 2 files changed, 101 insertions(+), 42 deletions(-)

diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index 6e55bc390be23..6c0f2f8b7bf1e 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -96,15 +96,10 @@ struct OperandsToCleanup {
   bool replaceWithPoison = false;
 };
 
-struct BlockArgsToCleanup {
-  Block *b;
-  BitVector nonLiveArgs;
-};
-
 struct SuccessorOperandsToCleanup {
   BranchOpInterface branch;
   unsigned successorIndex;
-  BitVector nonLiveOperands;
+  Block *successorBlock;
 };
 
 struct RDVFinalCleanupList {
@@ -112,7 +107,7 @@ struct RDVFinalCleanupList {
   SmallVector<FunctionToCleanUp> functions;
   SmallVector<OperandsToCleanup> operands;
   SmallVector<ResultsToCleanup> results;
-  SmallVector<BlockArgsToCleanup> blocks;
+  DenseMap<Block *, BitVector> branchBlockArgs;
   SmallVector<SuccessorOperandsToCleanup> successorOperands;
 };
 
@@ -465,11 +460,10 @@ static void processRegionBranchOp(RegionBranchOpInterface regionBranchOp,
 /// conditional branch op), the entire operation is dead.
 ///
 /// Otherwise, iterate through each successor block of `branchOp`.
-/// (1) For each successor block, gather all operands from all successors.
-/// (2) Fetch their associated liveness analysis data and collect for future
-///     removal.
-/// (3) Identify and collect the dead operands from the successor block
-///     as well as their corresponding arguments.
+/// (1) Fetch the successor block arguments' liveness analysis data.
+/// (2) Exclude arguments produced internally by the branch operation.
+/// (3) Collect dead forwarded operands and their corresponding block arguments
+///     for future removal.
 
 static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
                             DenseSet<Value> &nonLiveSet,
@@ -494,25 +488,18 @@ static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
 
   for (unsigned succIdx = 0; succIdx < numSuccessors; ++succIdx) {
     Block *successorBlock = branchOp->getSuccessor(succIdx);
-
-    // Do (1)
     SuccessorOperands successorOperands =
         branchOp.getSuccessorOperands(succIdx);
-    SmallVector<Value> operandValues;
-    for (unsigned operandIdx = 0; operandIdx < successorOperands.size();
-         ++operandIdx) {
-      operandValues.push_back(successorOperands[operandIdx]);
-    }
 
-    // Do (2)
     BitVector successorNonLive =
-        markLives(operandValues, nonLiveSet, la).flip();
-    collectNonLiveValues(nonLiveSet, successorBlock->getArguments(),
-                         successorNonLive);
-
-    // Do (3)
-    cl.blocks.push_back({successorBlock, successorNonLive});
-    cl.successorOperands.push_back({branchOp, succIdx, successorNonLive});
+        markLives(successorBlock->getArguments(), nonLiveSet, la).flip();
+    successorNonLive.reset(0, successorOperands.getProducedOperandCount());
+    auto [it, inserted] =
+        cl.branchBlockArgs.try_emplace(successorBlock, successorNonLive);
+    if (!inserted)
+      it->second &= successorNonLive;
+
+    cl.successorOperands.push_back({branchOp, succIdx, successorBlock});
   }
 }
 
@@ -576,26 +563,33 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
 
   // 2. Blocks, We must remove the block arguments and successor operands before
   // deleting the operation, as they may reside in the region operation.
-  LDBG() << "Cleaning up " << list.blocks.size() << " block argument lists";
-  for (auto &b : list.blocks) {
-    // blocks that are accessed via multiple codepaths processed once
-    if (b.b->getNumArguments() != b.nonLiveArgs.size())
+  LDBG() << "Cleaning up " << list.branchBlockArgs.size()
+         << " block argument lists";
+  for (auto &entry : list.branchBlockArgs) {
+    Block *block = entry.first;
+    BitVector &nonLiveArgs = entry.second;
+    if (block->getNumArguments() != nonLiveArgs.size())
       continue;
     LDBG_OS([&](raw_ostream &os) {
       os << "Erasing non-live arguments [";
-      llvm::interleaveComma(b.nonLiveArgs.set_bits(), os);
-      os << "] from block #" << b.b->computeBlockNumber() << " in region #"
-         << b.b->getParent()->getRegionNumber() << " of operation "
-         << OpWithFlags(b.b->getParent()->getParentOp(),
+      llvm::interleaveComma(nonLiveArgs.set_bits(), os);
+      os << "] from block #" << block->computeBlockNumber() << " in region #"
+         << block->getParent()->getRegionNumber() << " of operation "
+         << OpWithFlags(block->getParent()->getParentOp(),
                         OpPrintingFlags().skipRegions().printGenericOpForm());
     });
     // Note: Iterate from the end to make sure that that indices of not yet
     // processes arguments do not change.
-    for (int i = b.nonLiveArgs.size() - 1; i >= 0; --i) {
-      if (!b.nonLiveArgs[i])
+    for (int i = nonLiveArgs.size() - 1; i >= 0; --i) {
+      if (!nonLiveArgs[i])
         continue;
-      b.b->getArgument(i).dropAllUses();
-      b.b->eraseArgument(i);
+      BlockArgument argument = block->getArgument(i);
+      if (!argument.use_empty()) {
+        rewriter.setInsertionPointToStart(block);
+        Value poison = createPoisonedValue(rewriter, argument);
+        rewriter.replaceAllUsesWith(argument, poison);
+      }
+      block->eraseArgument(i);
     }
   }
 
@@ -605,19 +599,21 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
   for (auto &op : list.successorOperands) {
     SuccessorOperands successorOperands =
         op.branch.getSuccessorOperands(op.successorIndex);
+    const BitVector &nonLiveOperands =
+        list.branchBlockArgs.find(op.successorBlock)->second;
     // blocks that are accessed via multiple codepaths processed once
-    if (successorOperands.size() != op.nonLiveOperands.size())
+    if (successorOperands.size() != nonLiveOperands.size())
       continue;
     LDBG_OS([&](raw_ostream &os) {
       os << "Erasing non-live successor operands [";
-      llvm::interleaveComma(op.nonLiveOperands.set_bits(), os);
+      llvm::interleaveComma(nonLiveOperands.set_bits(), os);
       os << "] from successor " << op.successorIndex << " of branch: "
          << OpWithFlags(op.branch.getOperation(),
                         OpPrintingFlags().skipRegions().printGenericOpForm());
     });
     // it iterates backwards because erase invalidates all successor indexes
     for (int i = successorOperands.size() - 1; i >= 0; --i) {
-      if (!op.nonLiveOperands[i])
+      if (!nonLiveOperands[i])
         continue;
       successorOperands.erase(i);
     }
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 390a448060b7f..ed835d2d5fd91 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -895,3 +895,66 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Regression test for #205984. The loop header arguments are live on the entry
+// edge, even though the constant comparison makes the backedge unreachable.
+// CHECK-LABEL: func.func @keep_live_loop_header_args_with_unreachable_backedge
+// CHECK:         cf.br ^[[HEADER:bb[0-9]+]](%[[ZERO:.*]], %[[LIMIT:.*]] : index, index)
+// CHECK:       ^[[HEADER]](%[[IV:.*]]: index, %[[BOUND:.*]]: index):
+// CHECK-NEXT:    %[[CMP:.*]] = arith.cmpi slt, %[[IV]], %[[BOUND]] : index
+// CHECK-NEXT:    cf.cond_br %[[CMP]], ^[[BACKEDGE:bb[0-9]+]], ^[[EXIT:bb[0-9]+]]
+// CHECK:       ^[[BACKEDGE]]:
+// CHECK-NEXT:    %[[NEXT:.*]] = ub.poison : index
+// CHECK-NEXT:    %[[NEXT_BOUND:.*]] = ub.poison : index
+// CHECK-NEXT:    cf.br ^[[HEADER]](%[[NEXT]], %[[NEXT_BOUND]] : index, index)
+// CHECK:       ^[[EXIT]]:
+
+// CHECK-CANONICALIZE-LABEL: func.func @keep_live_loop_header_args_with_unreachable_backedge
+// CHECK-CANONICALIZE:         cf.br ^[[HEADER:bb[0-9]+]](%[[ZERO:.*]], %[[LIMIT:.*]] : index, index)
+// CHECK-CANONICALIZE:       ^[[HEADER]](%[[IV:.*]]: index, %[[BOUND:.*]]: index):
+// CHECK-CANONICALIZE-NEXT:    %[[CMP:.*]] = arith.cmpi slt, %[[IV]], %[[BOUND]] : index
+// CHECK-CANONICALIZE-NEXT:    cf.cond_br %[[CMP]], ^[[BACKEDGE:bb[0-9]+]], ^[[EXIT:bb[0-9]+]]
+// CHECK-CANONICALIZE:       ^[[BACKEDGE]]:
+// CHECK-CANONICALIZE-NEXT:    %[[NEXT:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:    %[[NEXT_BOUND:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:    cf.br ^[[HEADER]](%[[NEXT]], %[[NEXT_BOUND]] : index, index)
+// CHECK-CANONICALIZE:       ^[[EXIT]]:
+func.func @keep_live_loop_header_args_with_unreachable_backedge() {
+  %zero = arith.constant 0 : index
+  %limit = arith.constant 0 : index
+  cf.br ^bb1(%zero, %limit : index, index)
+^bb1(%iv: index, %bound: index):
+  %cmp = arith.cmpi slt, %iv, %bound : index
+  cf.cond_br %cmp, ^bb2(%limit, %limit : index, index), ^bb3
+^bb2(%next: index, %next_bound: index):
+  cf.br ^bb1(%next, %next_bound : index, index)
+^bb3:
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @keep_produced_successor_argument
+// CHECK:         "test.internal_br"()[^[[NORMAL:.*]], ^[[ERROR:.*]]]
+// CHECK:       ^[[NORMAL]]:
+// CHECK-NEXT:    "test.terminator"() : () -> ()
+// CHECK:       ^[[ERROR]](%{{.*}}: i32):
+// CHECK-NEXT:    "test.terminator"() : () -> ()
+
+// CHECK-CANONICALIZE-LABEL: func.func @keep_produced_successor_argument
+// CHECK-CANONICALIZE:         "test.internal_br"()[^[[NORMAL:.*]], ^[[ERROR:.*]]]
+// CHECK-CANONICALIZE:       ^[[NORMAL]]:
+// CHECK-CANONICALIZE-NEXT:    "test.terminator"() : () -> ()
+// CHECK-CANONICALIZE:       ^[[ERROR]](%{{.*}}: i32):
+// CHECK-CANONICALIZE-NEXT:    "test.terminator"() : () -> ()
+func.func @keep_produced_successor_argument() {
+  "test.internal_br"() [^bb0, ^bb1] {
+    operandSegmentSizes = array<i32: 0, 0>
+  } : () -> ()
+^bb0:
+  "test.terminator"() : () -> ()
+^bb1(%produced: i32):
+  "test.terminator"() : () -> ()
+}


From llvmlistbot at llvm.org  Sat Jul 11 12:37:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:37:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Coordinate branch argument cleanup (PR
	#208940)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208940@github.com>
Message-ID: <6a529b84.1905e68e.18ad68.de8b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I made branch argument cleanup consistent across all incoming control-flow paths, so unreachable paths no longer leave malformed IR. The change also preserves internally produced arguments and keeps surviving uses valid with poison values.
Fixes #<!-- -->205984 

---
Full diff: https://github.com/llvm/llvm-project/pull/208940.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/RemoveDeadValues.cpp (+38-42) 
- (modified) mlir/test/Transforms/remove-dead-values.mlir (+63) 


``````````diff
diff --git a/mlir/lib/Transforms/RemoveDeadValues.cpp b/mlir/lib/Transforms/RemoveDeadValues.cpp
index 6e55bc390be23..6c0f2f8b7bf1e 100644
--- a/mlir/lib/Transforms/RemoveDeadValues.cpp
+++ b/mlir/lib/Transforms/RemoveDeadValues.cpp
@@ -96,15 +96,10 @@ struct OperandsToCleanup {
   bool replaceWithPoison = false;
 };
 
-struct BlockArgsToCleanup {
-  Block *b;
-  BitVector nonLiveArgs;
-};
-
 struct SuccessorOperandsToCleanup {
   BranchOpInterface branch;
   unsigned successorIndex;
-  BitVector nonLiveOperands;
+  Block *successorBlock;
 };
 
 struct RDVFinalCleanupList {
@@ -112,7 +107,7 @@ struct RDVFinalCleanupList {
   SmallVector<FunctionToCleanUp> functions;
   SmallVector<OperandsToCleanup> operands;
   SmallVector<ResultsToCleanup> results;
-  SmallVector<BlockArgsToCleanup> blocks;
+  DenseMap<Block *, BitVector> branchBlockArgs;
   SmallVector<SuccessorOperandsToCleanup> successorOperands;
 };
 
@@ -465,11 +460,10 @@ static void processRegionBranchOp(RegionBranchOpInterface regionBranchOp,
 /// conditional branch op), the entire operation is dead.
 ///
 /// Otherwise, iterate through each successor block of `branchOp`.
-/// (1) For each successor block, gather all operands from all successors.
-/// (2) Fetch their associated liveness analysis data and collect for future
-///     removal.
-/// (3) Identify and collect the dead operands from the successor block
-///     as well as their corresponding arguments.
+/// (1) Fetch the successor block arguments' liveness analysis data.
+/// (2) Exclude arguments produced internally by the branch operation.
+/// (3) Collect dead forwarded operands and their corresponding block arguments
+///     for future removal.
 
 static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
                             DenseSet<Value> &nonLiveSet,
@@ -494,25 +488,18 @@ static void processBranchOp(BranchOpInterface branchOp, RunLivenessAnalysis &la,
 
   for (unsigned succIdx = 0; succIdx < numSuccessors; ++succIdx) {
     Block *successorBlock = branchOp->getSuccessor(succIdx);
-
-    // Do (1)
     SuccessorOperands successorOperands =
         branchOp.getSuccessorOperands(succIdx);
-    SmallVector<Value> operandValues;
-    for (unsigned operandIdx = 0; operandIdx < successorOperands.size();
-         ++operandIdx) {
-      operandValues.push_back(successorOperands[operandIdx]);
-    }
 
-    // Do (2)
     BitVector successorNonLive =
-        markLives(operandValues, nonLiveSet, la).flip();
-    collectNonLiveValues(nonLiveSet, successorBlock->getArguments(),
-                         successorNonLive);
-
-    // Do (3)
-    cl.blocks.push_back({successorBlock, successorNonLive});
-    cl.successorOperands.push_back({branchOp, succIdx, successorNonLive});
+        markLives(successorBlock->getArguments(), nonLiveSet, la).flip();
+    successorNonLive.reset(0, successorOperands.getProducedOperandCount());
+    auto [it, inserted] =
+        cl.branchBlockArgs.try_emplace(successorBlock, successorNonLive);
+    if (!inserted)
+      it->second &= successorNonLive;
+
+    cl.successorOperands.push_back({branchOp, succIdx, successorBlock});
   }
 }
 
@@ -576,26 +563,33 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
 
   // 2. Blocks, We must remove the block arguments and successor operands before
   // deleting the operation, as they may reside in the region operation.
-  LDBG() << "Cleaning up " << list.blocks.size() << " block argument lists";
-  for (auto &b : list.blocks) {
-    // blocks that are accessed via multiple codepaths processed once
-    if (b.b->getNumArguments() != b.nonLiveArgs.size())
+  LDBG() << "Cleaning up " << list.branchBlockArgs.size()
+         << " block argument lists";
+  for (auto &entry : list.branchBlockArgs) {
+    Block *block = entry.first;
+    BitVector &nonLiveArgs = entry.second;
+    if (block->getNumArguments() != nonLiveArgs.size())
       continue;
     LDBG_OS([&](raw_ostream &os) {
       os << "Erasing non-live arguments [";
-      llvm::interleaveComma(b.nonLiveArgs.set_bits(), os);
-      os << "] from block #" << b.b->computeBlockNumber() << " in region #"
-         << b.b->getParent()->getRegionNumber() << " of operation "
-         << OpWithFlags(b.b->getParent()->getParentOp(),
+      llvm::interleaveComma(nonLiveArgs.set_bits(), os);
+      os << "] from block #" << block->computeBlockNumber() << " in region #"
+         << block->getParent()->getRegionNumber() << " of operation "
+         << OpWithFlags(block->getParent()->getParentOp(),
                         OpPrintingFlags().skipRegions().printGenericOpForm());
     });
     // Note: Iterate from the end to make sure that that indices of not yet
     // processes arguments do not change.
-    for (int i = b.nonLiveArgs.size() - 1; i >= 0; --i) {
-      if (!b.nonLiveArgs[i])
+    for (int i = nonLiveArgs.size() - 1; i >= 0; --i) {
+      if (!nonLiveArgs[i])
         continue;
-      b.b->getArgument(i).dropAllUses();
-      b.b->eraseArgument(i);
+      BlockArgument argument = block->getArgument(i);
+      if (!argument.use_empty()) {
+        rewriter.setInsertionPointToStart(block);
+        Value poison = createPoisonedValue(rewriter, argument);
+        rewriter.replaceAllUsesWith(argument, poison);
+      }
+      block->eraseArgument(i);
     }
   }
 
@@ -605,19 +599,21 @@ static void cleanUpDeadVals(MLIRContext *ctx, RDVFinalCleanupList &list) {
   for (auto &op : list.successorOperands) {
     SuccessorOperands successorOperands =
         op.branch.getSuccessorOperands(op.successorIndex);
+    const BitVector &nonLiveOperands =
+        list.branchBlockArgs.find(op.successorBlock)->second;
     // blocks that are accessed via multiple codepaths processed once
-    if (successorOperands.size() != op.nonLiveOperands.size())
+    if (successorOperands.size() != nonLiveOperands.size())
       continue;
     LDBG_OS([&](raw_ostream &os) {
       os << "Erasing non-live successor operands [";
-      llvm::interleaveComma(op.nonLiveOperands.set_bits(), os);
+      llvm::interleaveComma(nonLiveOperands.set_bits(), os);
       os << "] from successor " << op.successorIndex << " of branch: "
          << OpWithFlags(op.branch.getOperation(),
                         OpPrintingFlags().skipRegions().printGenericOpForm());
     });
     // it iterates backwards because erase invalidates all successor indexes
     for (int i = successorOperands.size() - 1; i >= 0; --i) {
-      if (!op.nonLiveOperands[i])
+      if (!nonLiveOperands[i])
         continue;
       successorOperands.erase(i);
     }
diff --git a/mlir/test/Transforms/remove-dead-values.mlir b/mlir/test/Transforms/remove-dead-values.mlir
index 390a448060b7f..ed835d2d5fd91 100644
--- a/mlir/test/Transforms/remove-dead-values.mlir
+++ b/mlir/test/Transforms/remove-dead-values.mlir
@@ -895,3 +895,66 @@ module @func_with_non_call_users {
   }
   spirv.EntryPoint "GLCompute" @callee
 }
+
+// -----
+
+// Regression test for #205984. The loop header arguments are live on the entry
+// edge, even though the constant comparison makes the backedge unreachable.
+// CHECK-LABEL: func.func @keep_live_loop_header_args_with_unreachable_backedge
+// CHECK:         cf.br ^[[HEADER:bb[0-9]+]](%[[ZERO:.*]], %[[LIMIT:.*]] : index, index)
+// CHECK:       ^[[HEADER]](%[[IV:.*]]: index, %[[BOUND:.*]]: index):
+// CHECK-NEXT:    %[[CMP:.*]] = arith.cmpi slt, %[[IV]], %[[BOUND]] : index
+// CHECK-NEXT:    cf.cond_br %[[CMP]], ^[[BACKEDGE:bb[0-9]+]], ^[[EXIT:bb[0-9]+]]
+// CHECK:       ^[[BACKEDGE]]:
+// CHECK-NEXT:    %[[NEXT:.*]] = ub.poison : index
+// CHECK-NEXT:    %[[NEXT_BOUND:.*]] = ub.poison : index
+// CHECK-NEXT:    cf.br ^[[HEADER]](%[[NEXT]], %[[NEXT_BOUND]] : index, index)
+// CHECK:       ^[[EXIT]]:
+
+// CHECK-CANONICALIZE-LABEL: func.func @keep_live_loop_header_args_with_unreachable_backedge
+// CHECK-CANONICALIZE:         cf.br ^[[HEADER:bb[0-9]+]](%[[ZERO:.*]], %[[LIMIT:.*]] : index, index)
+// CHECK-CANONICALIZE:       ^[[HEADER]](%[[IV:.*]]: index, %[[BOUND:.*]]: index):
+// CHECK-CANONICALIZE-NEXT:    %[[CMP:.*]] = arith.cmpi slt, %[[IV]], %[[BOUND]] : index
+// CHECK-CANONICALIZE-NEXT:    cf.cond_br %[[CMP]], ^[[BACKEDGE:bb[0-9]+]], ^[[EXIT:bb[0-9]+]]
+// CHECK-CANONICALIZE:       ^[[BACKEDGE]]:
+// CHECK-CANONICALIZE-NEXT:    %[[NEXT:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:    %[[NEXT_BOUND:.*]] = ub.poison : index
+// CHECK-CANONICALIZE-NEXT:    cf.br ^[[HEADER]](%[[NEXT]], %[[NEXT_BOUND]] : index, index)
+// CHECK-CANONICALIZE:       ^[[EXIT]]:
+func.func @keep_live_loop_header_args_with_unreachable_backedge() {
+  %zero = arith.constant 0 : index
+  %limit = arith.constant 0 : index
+  cf.br ^bb1(%zero, %limit : index, index)
+^bb1(%iv: index, %bound: index):
+  %cmp = arith.cmpi slt, %iv, %bound : index
+  cf.cond_br %cmp, ^bb2(%limit, %limit : index, index), ^bb3
+^bb2(%next: index, %next_bound: index):
+  cf.br ^bb1(%next, %next_bound : index, index)
+^bb3:
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @keep_produced_successor_argument
+// CHECK:         "test.internal_br"()[^[[NORMAL:.*]], ^[[ERROR:.*]]]
+// CHECK:       ^[[NORMAL]]:
+// CHECK-NEXT:    "test.terminator"() : () -> ()
+// CHECK:       ^[[ERROR]](%{{.*}}: i32):
+// CHECK-NEXT:    "test.terminator"() : () -> ()
+
+// CHECK-CANONICALIZE-LABEL: func.func @keep_produced_successor_argument
+// CHECK-CANONICALIZE:         "test.internal_br"()[^[[NORMAL:.*]], ^[[ERROR:.*]]]
+// CHECK-CANONICALIZE:       ^[[NORMAL]]:
+// CHECK-CANONICALIZE-NEXT:    "test.terminator"() : () -> ()
+// CHECK-CANONICALIZE:       ^[[ERROR]](%{{.*}}: i32):
+// CHECK-CANONICALIZE-NEXT:    "test.terminator"() : () -> ()
+func.func @keep_produced_successor_argument() {
+  "test.internal_br"() [^bb0, ^bb1] {
+    operandSegmentSizes = array<i32: 0, 0>
+  } : () -> ()
+^bb0:
+  "test.terminator"() : () -> ()
+^bb1(%produced: i32):
+  "test.terminator"() : () -> ()
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208940

From llvmlistbot at llvm.org  Sat Jul 11 12:54:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:54:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a529f76.6537ae93.2f1c1b.6dac@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 3931 tests passed
* 424 tests skipped

All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.

<details>
<summary>tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenDecl.cpp.o</summary>

```
FAILED: tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenDecl.cpp.o
sccache /opt/llvm/bin/clang++ -DCLANG_EXPORTS -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/lib/CIR/CodeGen -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/../mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenDecl.cpp.o -MF tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenDecl.cpp.o.d -o tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenDecl.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen/CIRGenDecl.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen/CIRGenDecl.cpp:786:11: error: enumeration value 'CXXExpansionStmt' not handled in switch [-Werror,-Wswitch]
786 |   switch (d.getKind()) {
|           ^~~~~~~~~~~
1 error generated.
```
</details>
<details>
<summary>tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenStmt.cpp.o</summary>

```
FAILED: tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenStmt.cpp.o
sccache /opt/llvm/bin/clang++ -DCLANG_EXPORTS -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/lib/CIR/CodeGen -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen -I/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/../mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenStmt.cpp.o -MF tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenStmt.cpp.o.d -o tools/clang/lib/CIR/CodeGen/CMakeFiles/obj.clangCIR.dir/CIRGenStmt.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen/CIRGenStmt.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/clang/lib/CIR/CodeGen/CIRGenStmt.cpp:150:11: error: enumeration values 'CXXExpansionStmtPatternClass' and 'CXXExpansionStmtInstantiationClass' not handled in switch [-Werror,-Wswitch]
150 |   switch (s->getStmtClass()) {
|           ^~~~~~~~~~~~~~~~~
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Sat Jul 11 12:56:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 12:56:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a529ffa.af6ba3ed.36d9db.624c@mx.google.com>


================
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
----------------
mygitljf wrote:

You're right. parseFunctionSignature was emitting the diagnostic but still returning success, which made callers compensate for it. I've changed it to return failure after reporting the error, and the callers now propagate that failure directly.

https://github.com/llvm/llvm-project/pull/206215

From llvmlistbot at llvm.org  Sat Jul 11 13:10:25 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 13:10:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][mlir][tblgen] Refactor TableGen into a
 Cpp exposed lib (PR #189689)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/189689@github.com>
Message-ID: <6a52a331.824e11a6.f5f27.9530@mx.google.com>


https://github.com/makslevental approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/189689

From llvmlistbot at llvm.org  Sat Jul 11 15:52:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 15:52:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Handle dense iterators in
 sparse iteration lowering (PR #208963)
Message-ID: <llvm.org/llvm/llvm-project/pull/208963@github.com>


https://github.com/qyingwu created https://github.com/llvm/llvm-project/pull/208963

Fixes #205980.

`lower-sparse-iteration-to-scf` always called `linkNewScope()` when lowering
  iterators handled by `scf.for`. Dense levels are random-access iterators, and
  `linkNewScope()` asserts for random-access iterators because those should be
  traversed by coordinate.

  Use `locate()` for random-access iterators and keep `linkNewScope()` for
  non-random-access iterators.

  Verification:
  - `cmake --build /tmp/mlir-208198 --target mlir-opt`
  - `/tmp/mlir-208198/bin/mlir-opt -lower-sparse-iteration-to-scf /tmp/issue-
  205980.mlir`
  - `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/SparseTensor/
  sparse_iteration_to_scf.mlir mlir/test/Dialect/SparseTensor/
  sparse_kernels_to_iterator.mlir mlir/test/Dialect/SparseTensor/
  sparse_space_collapse.mlir`

>From cc7eade3da850dd6070b3468c2fcd236d920d7e7 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 11 Jul 2026 15:44:01 -0700
Subject: [PATCH] [mlir][sparse] Handle dense iterators in sparse iteration
 lowering

---
 .../Transforms/SparseIterationToScf.cpp           |  5 ++++-
 .../SparseTensor/sparse_iteration_to_scf.mlir     | 15 +++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
index 9cd489653a0f3..e4cecef2d44e5 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
@@ -134,8 +134,11 @@ static ValueRange genLoopWithIterator(
         });
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      it->linkNewScope(forOp.getInductionVar());
       rewriter.setInsertionPointToStart(forOp.getBody());
+      if (it->randomAccessible())
+        it->locate(rewriter, loc, forOp.getInductionVar());
+      else
+        it->linkNewScope(forOp.getInductionVar());
       SmallVector<Value> ret = bodyBuilder(rewriter, loc, forOp.getBodyRegion(),
                                            it, forOp.getRegionIterArgs());
 
diff --git a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
index 6d8f9018ad3a5..855f1e99e7396 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
@@ -50,3 +50,18 @@ func.func @sparse_iteration_to_scf(%sp : tensor<4x8xf32, #COO>) -> index {
   }
   return %r1 : index
 }
+
+#DenseCompressed = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : dense, d1 : compressed)
+}>
+
+// CHECK-LABEL:   @sparse_iteration_dense_level
+// CHECK:           scf.for
+func.func @sparse_iteration_dense_level(%sp: tensor<?x?xf64, #DenseCompressed>) {
+  %0 = sparse_tensor.extract_iteration_space %sp lvls = 0
+      : tensor<?x?xf64, #DenseCompressed> -> !sparse_tensor.iter_space<#DenseCompressed, lvls = 0>
+  sparse_tensor.iterate %it in %0 at(%i)
+      : !sparse_tensor.iter_space<#DenseCompressed, lvls = 0> {
+  }
+  return
+}


From llvmlistbot at llvm.org  Sat Jul 11 15:52:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 15:52:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Handle dense iterators in
 sparse iteration lowering (PR #208963)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208963@github.com>
Message-ID: <6a52c941.d1496011.26913c.de8f@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/208963

>From cc7eade3da850dd6070b3468c2fcd236d920d7e7 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 11 Jul 2026 15:44:01 -0700
Subject: [PATCH] [mlir][sparse] Handle dense iterators in sparse iteration
 lowering

---
 .../Transforms/SparseIterationToScf.cpp           |  5 ++++-
 .../SparseTensor/sparse_iteration_to_scf.mlir     | 15 +++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
index 9cd489653a0f3..e4cecef2d44e5 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
@@ -134,8 +134,11 @@ static ValueRange genLoopWithIterator(
         });
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      it->linkNewScope(forOp.getInductionVar());
       rewriter.setInsertionPointToStart(forOp.getBody());
+      if (it->randomAccessible())
+        it->locate(rewriter, loc, forOp.getInductionVar());
+      else
+        it->linkNewScope(forOp.getInductionVar());
       SmallVector<Value> ret = bodyBuilder(rewriter, loc, forOp.getBodyRegion(),
                                            it, forOp.getRegionIterArgs());
 
diff --git a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
index 6d8f9018ad3a5..855f1e99e7396 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
@@ -50,3 +50,18 @@ func.func @sparse_iteration_to_scf(%sp : tensor<4x8xf32, #COO>) -> index {
   }
   return %r1 : index
 }
+
+#DenseCompressed = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : dense, d1 : compressed)
+}>
+
+// CHECK-LABEL:   @sparse_iteration_dense_level
+// CHECK:           scf.for
+func.func @sparse_iteration_dense_level(%sp: tensor<?x?xf64, #DenseCompressed>) {
+  %0 = sparse_tensor.extract_iteration_space %sp lvls = 0
+      : tensor<?x?xf64, #DenseCompressed> -> !sparse_tensor.iter_space<#DenseCompressed, lvls = 0>
+  sparse_tensor.iterate %it in %0 at(%i)
+      : !sparse_tensor.iter_space<#DenseCompressed, lvls = 0> {
+  }
+  return
+}


From llvmlistbot at llvm.org  Sat Jul 11 15:53:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 15:53:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Handle dense iterators in
 sparse iteration lowering (PR #208963)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208963@github.com>
Message-ID: <6a52c953.e55dc341.1f19bd.a07d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: qyingwu

<details>
<summary>Changes</summary>

Fixes #<!-- -->205980.

`lower-sparse-iteration-to-scf` always called `linkNewScope()` when lowering
  iterators handled by `scf.for`. Dense levels are random-access iterators, and
  `linkNewScope()` asserts for random-access iterators because those should be
  traversed by coordinate.

  Use `locate()` for random-access iterators and keep `linkNewScope()` for
  non-random-access iterators.

  Verification:
  - `cmake --build /tmp/mlir-208198 --target mlir-opt`
  - `/tmp/mlir-208198/bin/mlir-opt -lower-sparse-iteration-to-scf /tmp/issue-
  205980.mlir`
  - `/tmp/mlir-208198/bin/llvm-lit -sv mlir/test/Dialect/SparseTensor/
  sparse_iteration_to_scf.mlir mlir/test/Dialect/SparseTensor/
  sparse_kernels_to_iterator.mlir mlir/test/Dialect/SparseTensor/
  sparse_space_collapse.mlir`

---
Full diff: https://github.com/llvm/llvm-project/pull/208963.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp (+4-1) 
- (modified) mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir (+15) 


``````````diff
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
index 9cd489653a0f3..e4cecef2d44e5 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseIterationToScf.cpp
@@ -134,8 +134,11 @@ static ValueRange genLoopWithIterator(
         });
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      it->linkNewScope(forOp.getInductionVar());
       rewriter.setInsertionPointToStart(forOp.getBody());
+      if (it->randomAccessible())
+        it->locate(rewriter, loc, forOp.getInductionVar());
+      else
+        it->linkNewScope(forOp.getInductionVar());
       SmallVector<Value> ret = bodyBuilder(rewriter, loc, forOp.getBodyRegion(),
                                            it, forOp.getRegionIterArgs());
 
diff --git a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
index 6d8f9018ad3a5..855f1e99e7396 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
@@ -50,3 +50,18 @@ func.func @sparse_iteration_to_scf(%sp : tensor<4x8xf32, #COO>) -> index {
   }
   return %r1 : index
 }
+
+#DenseCompressed = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : dense, d1 : compressed)
+}>
+
+// CHECK-LABEL:   @sparse_iteration_dense_level
+// CHECK:           scf.for
+func.func @sparse_iteration_dense_level(%sp: tensor<?x?xf64, #DenseCompressed>) {
+  %0 = sparse_tensor.extract_iteration_space %sp lvls = 0
+      : tensor<?x?xf64, #DenseCompressed> -> !sparse_tensor.iter_space<#DenseCompressed, lvls = 0>
+  sparse_tensor.iterate %it in %0 at(%i)
+      : !sparse_tensor.iter_space<#DenseCompressed, lvls = 0> {
+  }
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208963

From llvmlistbot at llvm.org  Sat Jul 11 18:16:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 18:16:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
Message-ID: <llvm.org/llvm/llvm-project/pull/208973@github.com>


https://github.com/pzoln created https://github.com/llvm/llvm-project/pull/208973

Fixes #202782

When folding bufferization.to_buffer(bufferization.to_tensor(...)), only create a memref.cast if the source and destination memref types are cast-compatible.

This avoids asserting on unranked memrefs with different memory spaces and leaves the original bufferization ops in place instead.

>From 71ea7d95a7564a43c08e05580f92f8cd39c65768 Mon Sep 17 00:00:00 2001
From: Pavel Zolnikov <pavel at avarivent.com>
Date: Sat, 11 Jul 2026 18:00:45 -0700
Subject: [PATCH] [mlir][bufferization] Avoid invalid memref.cast in to_buffer
 folding

When folding bufferization.to_buffer(bufferization.to_tensor(...)), only
create a memref.cast if the source and destination memref types are
cast-compatible.

This avoids asserting on unranked memrefs with different memory spaces
and leaves the original bufferization ops in place instead.

Fixes #202782.
---
 .../Bufferization/IR/BufferizationOps.cpp      |  9 +++++----
 .../Dialect/Bufferization/canonicalize.mlir    | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 45f5b3eaa5aea..80db89aa1b4bc 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -132,10 +132,11 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (unrankedSrcType && rankedDestType)
     return failure();
 
-  // Unranked memref -> unranked memref cast
-  // Ranked memref -> unranked memref cast: No copy needed.
-  assert(memref::CastOp::areCastCompatible(srcType, destType) &&
-         "expected that types are cast compatible");
+  // Unranked/ranked memref -> unranked memref cast: No copy needed if the types
+  // are cast-compatible.
+  if (!memref::CastOp::areCastCompatible(srcType, destType))
+    return failure();
+
   rewriter.replaceOpWithNewOp<memref::CastOp>(toBuffer, destType,
                                               bufferToTensor.getBuffer());
   return success();
diff --git a/mlir/test/Dialect/Bufferization/canonicalize.mlir b/mlir/test/Dialect/Bufferization/canonicalize.mlir
index df07511798b91..ab3d8df892bc1 100644
--- a/mlir/test/Dialect/Bufferization/canonicalize.mlir
+++ b/mlir/test/Dialect/Bufferization/canonicalize.mlir
@@ -50,6 +50,24 @@ func.func @canonicalize_buffer_cast_of_tensor_load_different_address_space(%arg0
 
 // -----
 
+// If unranked memrefs are not cast-compatible, don't fold them.
+// CHECK-LABEL: func @canonicalize_unranked_buffer_cast_of_tensor_load_different_address_space(
+//  CHECK-SAME:   %[[MEMREF:.*]]: memref<*xi64>)
+//  CHECK-SAME:     -> memref<*xi64, 1> {
+//  CHECK-NOT: memref.cast
+//      CHECK: %[[TENSOR:.*]] = bufferization.to_tensor %[[MEMREF]] : memref<*xi64> to tensor<*xi64>
+//      CHECK: %[[BUFFER:.*]] = bufferization.to_buffer %[[TENSOR]] : tensor<*xi64> to memref<*xi64, 1>
+//  CHECK-NOT: memref.cast
+//      CHECK: return %[[BUFFER]] : memref<*xi64, 1>
+func.func @canonicalize_unranked_buffer_cast_of_tensor_load_different_address_space(%arg0: memref<*xi64>)
+    -> memref<*xi64, 1> {
+  %0 = bufferization.to_tensor %arg0 : memref<*xi64> to tensor<*xi64>
+  %1 = bufferization.to_buffer %0 : tensor<*xi64> to memref<*xi64, 1>
+  return %1 : memref<*xi64, 1>
+}
+
+// -----
+
 // If the memrefs are definitely cast-compatible, canonicalize to
 //            cast.
 // CHECK-LABEL: func @canonicalize_buffer_cast_of_tensor_load(


From llvmlistbot at llvm.org  Sat Jul 11 18:16:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 18:16:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a52eaec.2020f6c2.243e78.4f92@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @pzoln :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 11 18:17:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 18:17:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a52eb12.7270d2fd.2d4879.bc78@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-bufferization

Author: pzoln

<details>
<summary>Changes</summary>

Fixes #<!-- -->202782

When folding bufferization.to_buffer(bufferization.to_tensor(...)), only create a memref.cast if the source and destination memref types are cast-compatible.

This avoids asserting on unranked memrefs with different memory spaces and leaves the original bufferization ops in place instead.

---
Full diff: https://github.com/llvm/llvm-project/pull/208973.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp (+5-4) 
- (modified) mlir/test/Dialect/Bufferization/canonicalize.mlir (+18) 


``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 45f5b3eaa5aea..80db89aa1b4bc 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -132,10 +132,11 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (unrankedSrcType && rankedDestType)
     return failure();
 
-  // Unranked memref -> unranked memref cast
-  // Ranked memref -> unranked memref cast: No copy needed.
-  assert(memref::CastOp::areCastCompatible(srcType, destType) &&
-         "expected that types are cast compatible");
+  // Unranked/ranked memref -> unranked memref cast: No copy needed if the types
+  // are cast-compatible.
+  if (!memref::CastOp::areCastCompatible(srcType, destType))
+    return failure();
+
   rewriter.replaceOpWithNewOp<memref::CastOp>(toBuffer, destType,
                                               bufferToTensor.getBuffer());
   return success();
diff --git a/mlir/test/Dialect/Bufferization/canonicalize.mlir b/mlir/test/Dialect/Bufferization/canonicalize.mlir
index df07511798b91..ab3d8df892bc1 100644
--- a/mlir/test/Dialect/Bufferization/canonicalize.mlir
+++ b/mlir/test/Dialect/Bufferization/canonicalize.mlir
@@ -50,6 +50,24 @@ func.func @canonicalize_buffer_cast_of_tensor_load_different_address_space(%arg0
 
 // -----
 
+// If unranked memrefs are not cast-compatible, don't fold them.
+// CHECK-LABEL: func @canonicalize_unranked_buffer_cast_of_tensor_load_different_address_space(
+//  CHECK-SAME:   %[[MEMREF:.*]]: memref<*xi64>)
+//  CHECK-SAME:     -> memref<*xi64, 1> {
+//  CHECK-NOT: memref.cast
+//      CHECK: %[[TENSOR:.*]] = bufferization.to_tensor %[[MEMREF]] : memref<*xi64> to tensor<*xi64>
+//      CHECK: %[[BUFFER:.*]] = bufferization.to_buffer %[[TENSOR]] : tensor<*xi64> to memref<*xi64, 1>
+//  CHECK-NOT: memref.cast
+//      CHECK: return %[[BUFFER]] : memref<*xi64, 1>
+func.func @canonicalize_unranked_buffer_cast_of_tensor_load_different_address_space(%arg0: memref<*xi64>)
+    -> memref<*xi64, 1> {
+  %0 = bufferization.to_tensor %arg0 : memref<*xi64> to tensor<*xi64>
+  %1 = bufferization.to_buffer %0 : tensor<*xi64> to memref<*xi64, 1>
+  return %1 : memref<*xi64, 1>
+}
+
+// -----
+
 // If the memrefs are definitely cast-compatible, canonicalize to
 //            cast.
 // CHECK-LABEL: func @canonicalize_buffer_cast_of_tensor_load(

``````````

</details>


https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 11 18:20:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 18:20:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a52ebca.c5504292.28b3b8.4017@mx.google.com>


https://github.com/pzoln edited https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 11 19:53:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 19:53:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a5301c7.d1496011.26913c.0d19@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206215

>From 2fc18628d1c3420e7399ad967866ce401fdf6fbe Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 12:12:37 +0000
Subject: [PATCH 1/2] [MLIR] Diagnose missing call result arrow

---
 clang/test/CIR/IR/invalid-call.cir         |  9 +++++++++
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp |  3 +++
 mlir/lib/Interfaces/CallInterfaces.cpp     |  3 ++-
 mlir/test/Dialect/LLVMIR/invalid.mlir      | 10 ++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/clang/test/CIR/IR/invalid-call.cir b/clang/test/CIR/IR/invalid-call.cir
index a9c7e38f73af6..11b5c4425549e 100644
--- a/clang/test/CIR/IR/invalid-call.cir
+++ b/clang/test/CIR/IR/invalid-call.cir
@@ -10,6 +10,15 @@ cir.func @f1() {
 
 // -----
 
+cir.func private @f_missing_result_arrow()
+cir.func @call_missing_empty_result() {
+  // expected-error @+2 {{expected '->' followed by function result list}}
+  cir.call @f_missing_result_arrow() : ()
+  cir.return
+}
+
+// -----
+
 !u32i = !cir.int<u, 32>
 
 cir.func private @f2()
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 58f569abff8ea..0cc007433ac0e 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
+      return failure();
     if (isDirect)
       return parser.emitError(trailingTypesLoc,
                               "expected direct call to have 1 trailing types");
diff --git a/mlir/lib/Interfaces/CallInterfaces.cpp b/mlir/lib/Interfaces/CallInterfaces.cpp
index e8ed4b339a0cb..af670cb62c1c6 100644
--- a/mlir/lib/Interfaces/CallInterfaces.cpp
+++ b/mlir/lib/Interfaces/CallInterfaces.cpp
@@ -72,7 +72,8 @@ ParseResult call_interface_impl::parseFunctionSignature(
     return call_interface_impl::parseFunctionResultList(parser, resultTypes,
                                                         resultAttrs);
   if (mustParseEmptyResult)
-    return failure();
+    return parser.emitError(parser.getCurrentLocation(),
+                            "expected '->' followed by function result list");
   return success();
 }
 
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index d5ea5c8de862e..1d0bb0b6ef999 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -241,6 +241,16 @@ func.func @call_missing_ptr_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 
 // -----
 
+func.func private @callee()
+
+func.func @call_missing_empty_result() {
+  // expected-error at +2 {{expected '->' followed by function result list}}
+  llvm.call @callee() : ()
+  llvm.return
+}
+
+// -----
+
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {

>From 7c84efca9a10c9f8cbbf974bea754e088388e5fd Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 02:49:36 +0000
Subject: [PATCH 2/2] [MLIR] Unify call signature diagnostics

---
 flang/lib/Optimizer/Dialect/FIROps.cpp        |  2 +-
 flang/test/Fir/invalid.fir                    |  8 ++++++++
 mlir/include/mlir/Interfaces/CallInterfaces.h |  1 +
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp    | 12 ++----------
 mlir/test/Dialect/LLVMIR/invalid.mlir         |  6 ++----
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Optimizer/Dialect/FIROps.cpp b/flang/lib/Optimizer/Dialect/FIROps.cpp
index ba047e71d6aa3..6f0c8eecfd462 100644
--- a/flang/lib/Optimizer/Dialect/FIROps.cpp
+++ b/flang/lib/Optimizer/Dialect/FIROps.cpp
@@ -1714,7 +1714,7 @@ mlir::ParseResult fir::CallOp::parse(mlir::OpAsmParser &parser,
   llvm::SmallVector<mlir::DictionaryAttr> resultAttrs;
   if (mlir::call_interface_impl::parseFunctionSignature(
           parser, argTypes, argAttrs, resTypes, resultAttrs))
-    return parser.emitError(parser.getNameLoc(), "expected function type");
+    return mlir::failure();
   mlir::FunctionType funcType =
       mlir::FunctionType::get(parser.getContext(), argTypes, resTypes);
   if (isDirect) {
diff --git a/flang/test/Fir/invalid.fir b/flang/test/Fir/invalid.fir
index 5471bcd8acc29..937ce891bfef0 100644
--- a/flang/test/Fir/invalid.fir
+++ b/flang/test/Fir/invalid.fir
@@ -1,5 +1,13 @@
 // RUN: fir-opt -split-input-file -verify-diagnostics --strict-fir-volatile-verifier %s
 
+func.func @call_missing_empty_result() {
+  // expected-error at +2{{expected '->' followed by function result list}}
+  fir.call @callee() : ()
+  return
+}
+
+// -----
+
 // expected-error at +1{{custom op 'fir.string_lit' must have character type}}
 %0 = fir.string_lit "Hello, World!"(13) : !fir.int<32>
 
diff --git a/mlir/include/mlir/Interfaces/CallInterfaces.h b/mlir/include/mlir/Interfaces/CallInterfaces.h
index 2bf3a3ca5f8a8..02c04c0f8dd1e 100644
--- a/mlir/include/mlir/Interfaces/CallInterfaces.h
+++ b/mlir/include/mlir/Interfaces/CallInterfaces.h
@@ -53,6 +53,7 @@ parseFunctionResultList(OpAsmParser &parser, SmallVectorImpl<Type> &resultTypes,
 /// signatures containing SSA region arguments (to parse these signatures, use
 /// function_interface_impl::parseFunctionSignature). When
 /// `mustParseEmptyResult`, `-> ()` is expected when there is no result type.
+/// Returns failure after emitting a diagnostic if the signature is malformed.
 ///
 ///   no-ssa-function-signature ::= `(` no-ssa-function-arg-list `)`
 ///                               -> function-result-list
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 0cc007433ac0e..223852d3828c4 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,17 +1388,9 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
-  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
-                                                  resTypes, resultAttrs)) {
-    if (parser.getCurrentLocation() != signatureLoc)
-      return failure();
-    if (isDirect)
-      return parser.emitError(trailingTypesLoc,
-                              "expected direct call to have 1 trailing types");
-    return parser.emitError(trailingTypesLoc,
-                            "expected trailing function type");
-  }
+                                                  resTypes, resultAttrs))
+    return failure();
 
   if (resTypes.size() > 1)
     return parser.emitError(trailingTypesLoc,
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index 1d0bb0b6ef999..d44fa7ebfc4b1 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -254,8 +254,7 @@ func.func @call_missing_empty_result() {
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected direct call to have 1 trailing type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call @standard_func_callee(%arg) : !llvm.ptr, (i8) -> (i8)
   llvm.return
 }
@@ -271,8 +270,7 @@ func.func @call_non_pointer_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 // -----
 
 func.func @call_non_function_type(%callee : !llvm.ptr, %arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected trailing function type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call %callee(%arg) : !llvm.ptr, !llvm.func<i8 (i8)>
   llvm.return
 }


From llvmlistbot at llvm.org  Sat Jul 11 20:47:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 20:47:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a530e67.2490598a.cf17c.cc7c@mx.google.com>


================

----------------
PragmaTwice wrote:

I checked the signature in C++ and it seems for target materialization it's more complicated?
```c++
  /// Generate a wrapper for the given target materialization callback.
  /// The callback may take any subclass of `Type` and the wrapper will check
  /// for the target type to be of the expected class before calling the
  /// callback.
  ///
  /// With callback of form:
  /// - Value(OpBuilder &, T, ValueRange, Location, Type)
  /// - SmallVector<Value>(OpBuilder &, TypeRange, ValueRange, Location, Type)
  template <typename T, typename FnT>
  std::enable_if_t<
      std::is_invocable_v<FnT, OpBuilder &, T, ValueRange, Location, Type>,
      TargetMaterializationCallbackFn>
  wrapTargetMaterialization(FnT &&callback) const {
```

It can accept an additional `Type` and return a list of `Value`s (for 1:N conversion?). Note that for source materialization this callback models 1:N conversion already (i.e. `intptr_t nInputs, MlirValue *inputs`).

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 22:15:32 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Sat, 11 Jul 2026 22:15:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve shared data operations
 during if-clause lowering (PR #208771)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208771@github.com>
Message-ID: <6a5322f4.d76d81af.39f9be.57f2@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Thank you! Looks good to me.

https://github.com/llvm/llvm-project/pull/208771

From llvmlistbot at llvm.org  Sat Jul 11 22:19:07 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 22:19:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5323cb.3cce8983.225732.a103@mx.google.com>


================

----------------
makslevental wrote:

Ya I split that out into another PR so that they're easier to review https://github.com/llvm/llvm-project/pull/208935

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 22:20:36 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 11 Jul 2026 22:20:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a532424.e5907290.281f16.d37a@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 11 22:26:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 22:26:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Skip escaping
	allocation users (PR #208979)
Message-ID: <llvm.org/llvm/llvm-project/pull/208979@github.com>


https://github.com/mygitljf created https://github.com/llvm/llvm-project/pull/208979

I made the liveness optimization bail out when an allocation user cannot be safely represented in the allocation block. In that case the pass leaves the dealloc in place instead of trying to compare or move across CFG boundaries.
Fixes #205982 

>From 66a6e3ca26f80461c1209f9caf5388cef97629f0 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 13:27:38 +0000
Subject: [PATCH] [mlir][bufferization] Skip escaping allocation users

---
 .../Transforms/OptimizeAllocationLiveness.cpp |  2 ++
 .../optimize-allocation-liveness.mlir         | 22 +++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp b/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
index 28ee5b8e32b99..cfa6fca9d601a 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
@@ -149,6 +149,8 @@ struct OptimizeAllocationLiveness
 
           // find the ancestor of user that is in the same block as the allocOp.
           auto *topUser = allocOp->getBlock()->findAncestorOpInBlock(*user);
+          if (!topUser || topUser->hasTrait<OpTrait::IsTerminator>())
+            return WalkResult::advance();
           if (!lastUser || happensBefore(lastUser, topUser)) {
             lastUser = topUser;
           }
diff --git a/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir b/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
index 63d33e3a88bed..4ba39d84d184f 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
@@ -234,3 +234,25 @@ func.func private @test_alloc_with_multiple_results() -> () {
   memref.dealloc %alloc2 : memref<64xf32>
   return
 }
+
+// -----
+
+// CHECK-LABEL: func.func @allocation_alias_in_successor_block(
+// CHECK:         %[[ALLOC:.*]] = memref.alloc()
+// CHECK-NEXT:    test.buffer_based
+// CHECK-NEXT:    memref.dealloc %[[ALLOC]]
+// CHECK-NEXT:    cf.br ^bb3(%[[ALLOC]] : memref<2xf32>)
+func.func @allocation_alias_in_successor_block(
+    %arg0: i1, %arg1: memref<2xf32>, %arg2: memref<2xf32>) {
+  cf.cond_br %arg0, ^bb1, ^bb2
+^bb1:
+  cf.br ^bb3(%arg1 : memref<2xf32>)
+^bb2:
+  %alloc = memref.alloc() : memref<2xf32>
+  test.buffer_based in(%arg1 : memref<2xf32>) out(%alloc : memref<2xf32>)
+  memref.dealloc %alloc : memref<2xf32>
+  cf.br ^bb3(%alloc : memref<2xf32>)
+^bb3(%0: memref<2xf32>):
+  test.copy(%0, %arg2) : (memref<2xf32>, memref<2xf32>)
+  return
+}


From llvmlistbot at llvm.org  Sat Jul 11 22:27:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 11 Jul 2026 22:27:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Skip escaping
	allocation users (PR #208979)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208979@github.com>
Message-ID: <6a5325c2.bd93e731.358318.cac7@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-bufferization

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I made the liveness optimization bail out when an allocation user cannot be safely represented in the allocation block. In that case the pass leaves the dealloc in place instead of trying to compare or move across CFG boundaries.
Fixes #<!-- -->205982 

---
Full diff: https://github.com/llvm/llvm-project/pull/208979.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp (+2) 
- (modified) mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir (+22) 


``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp b/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
index 28ee5b8e32b99..cfa6fca9d601a 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/OptimizeAllocationLiveness.cpp
@@ -149,6 +149,8 @@ struct OptimizeAllocationLiveness
 
           // find the ancestor of user that is in the same block as the allocOp.
           auto *topUser = allocOp->getBlock()->findAncestorOpInBlock(*user);
+          if (!topUser || topUser->hasTrait<OpTrait::IsTerminator>())
+            return WalkResult::advance();
           if (!lastUser || happensBefore(lastUser, topUser)) {
             lastUser = topUser;
           }
diff --git a/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir b/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
index 63d33e3a88bed..4ba39d84d184f 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/optimize-allocation-liveness.mlir
@@ -234,3 +234,25 @@ func.func private @test_alloc_with_multiple_results() -> () {
   memref.dealloc %alloc2 : memref<64xf32>
   return
 }
+
+// -----
+
+// CHECK-LABEL: func.func @allocation_alias_in_successor_block(
+// CHECK:         %[[ALLOC:.*]] = memref.alloc()
+// CHECK-NEXT:    test.buffer_based
+// CHECK-NEXT:    memref.dealloc %[[ALLOC]]
+// CHECK-NEXT:    cf.br ^bb3(%[[ALLOC]] : memref<2xf32>)
+func.func @allocation_alias_in_successor_block(
+    %arg0: i1, %arg1: memref<2xf32>, %arg2: memref<2xf32>) {
+  cf.cond_br %arg0, ^bb1, ^bb2
+^bb1:
+  cf.br ^bb3(%arg1 : memref<2xf32>)
+^bb2:
+  %alloc = memref.alloc() : memref<2xf32>
+  test.buffer_based in(%arg1 : memref<2xf32>) out(%alloc : memref<2xf32>)
+  memref.dealloc %alloc : memref<2xf32>
+  cf.br ^bb3(%alloc : memref<2xf32>)
+^bb3(%0: memref<2xf32>):
+  test.copy(%0, %arg2) : (memref<2xf32>, memref<2xf32>)
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/208979

From llvmlistbot at llvm.org  Sun Jul 12 00:13:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 00:13:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Remove Pure attribute from Linalg::IndexOp.
	(PR #68894)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/68894@github.com>
Message-ID: <6a533ea0.481afadf.196d57.4eca@mx.google.com>


jk20342 wrote:

bumping this

https://github.com/llvm/llvm-project/pull/68894

From llvmlistbot at llvm.org  Sun Jul 12 00:27:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 00:27:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5341ce.6537ae93.2f1c1b.5e76@mx.google.com>


https://github.com/PragmaTwice approved this pull request.

> Anyway I plan to land the second basically immediately after this one so it shouldn't be a big deal that there's this gap for a couple of days.

I'm good for this but I think we may be care of [the 23.x release branch](https://discourse.llvm.org/t/llvm-23-x-will-branch-next-week-tue-14th-july/91233) to avoid branching on an intermediate state.

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sun Jul 12 00:28:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 00:28:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a534210.a1accbb4.244acd.f28d@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sun Jul 12 00:29:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 00:29:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a534276.af6ba3ed.36d9db.25b8@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sun Jul 12 02:23:37 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:23:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d19.160324b0.195192.532b@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:23:38 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:23:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d1a.743400b0.2067c9.c961@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
----------------
FedericoBruzzone wrote:

This comment does not seem to apply correctly to the `pack_lhs` function.

Unfortunately, I notice the same issue in [ArmSVE/pack-unpack-mmt4d](https://github.com/llvm/llvm-project/blob/main/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir). I will submit a patch after this PR <3

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:23:38 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:23:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d1a.a87eb5dc.7d817.8f19@mx.google.com>


https://github.com/FedericoBruzzone commented:

Thanks a lot for your study in #201562 (together with @banach-space) and this PR @steplong 🫶

Overall, LGTM % nits :D

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:23:39 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:23:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d1b.6c5720b3.15e90c.5c15@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
----------------
FedericoBruzzone wrote:

A slight stylistic inconsistency: `pack_lhs` now fixes the outer dimension M as static at 1 (`tensor<1x16x?x1xf32>`), whereas `pack_acc` keeps it fully dynamic (`tensor<?x?x?x?xf32>`), even though the calculation is conceptually the same. 

Is there a reason why we can't align with something like [ArmSVE/pack-unpack-mmt4d](https://github.com/llvm/llvm-project/blob/main/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir)?

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:23:39 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:23:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d1b.af6ba3ed.36d9db.2087@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile sizes.
+  %c7 = arith.constant 7 : index
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
+  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
+
+  return %C_pack : tensor<?x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
----------------
FedericoBruzzone wrote:

It seems to be dead code, isn't it? :D

In [ArmSVE/pack-unpack-mmt4d](https://github.com/llvm/llvm-project/blob/main/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir)  seems to be used.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:24:07 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:24:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d37.bd93e731.358318.b1eb@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:24:28 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:24:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d4c.eb165e5c.1817a5.dbea@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 02:24:48 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 12 Jul 2026 02:24:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a535d60.824e11a6.f5f27.038b@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Sun Jul 12 05:58:48 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Sun, 12 Jul 2026 05:58:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
Message-ID: <llvm.org/llvm/llvm-project/pull/209011@github.com>


https://github.com/JerryShih created https://github.com/llvm/llvm-project/pull/209011

1. Add prefix `negative_` for the negative tests.
2. Add more test patterns.

>From ff4aa35fca37fba6f7b94c4202e61c58e3c484fa Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:51:20 -0700
Subject: [PATCH 1/2] [mlir][linalg][test] Rename the negative tests in
 `simplify-pack-unpack.mlir`.

Add prefix `negative_` for the negative tests.
---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 36 +++++++++----------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6979770154bab..4c1370abac947 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -12,11 +12,11 @@ func.func @single_dim_packing(%arg0: tensor<256xf32>) -> tensor<8x32xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @single_dim_packing_with_padding(
+// CHECK-LABEL: func.func @negative_single_dim_packing_with_padding(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<255xf32>)
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
+func.func @negative_single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
   %empty = tensor.empty() : tensor<8x32xf32>
   %cst = arith.constant 0.000000e+00 : f32
   %0 = linalg.pack %arg0 padding_value(%cst : f32) inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<255xf32> -> tensor<8x32xf32>
@@ -61,10 +61,10 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -72,10 +72,10 @@ func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<256x5xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -122,10 +122,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
   %empty = tensor.empty() : tensor<16x1x1x2xf32>
   %pack = linalg.pack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<32x1xf32> -> tensor<16x1x1x2xf32>
@@ -146,10 +146,10 @@ func.func @unpack_1d_to_collapse(%arg0: tensor<8x32xf32>) -> tensor<256xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_to_partial_slice
+// CHECK-LABEL: func.func @negative_unpack_to_partial_slice
 // CHECK-NOT:     tensor.collapse
 // CHECK:         linalg.unpack
-func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
+func.func @negative_unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
   %empty = tensor.empty() : tensor<255xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x32xf32> -> tensor<255xf32>
   return %0 : tensor<255xf32>
@@ -157,10 +157,10 @@ func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_dynamic
+// CHECK-LABEL: func.func @negative_unpack_dynamic
 // CHECK:     tensor.collapse
 // CHECK-NOT:         linalg.unpack
-func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
+func.func @negative_unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
   %c32 = arith.constant 32 : index
   %c0 = arith.constant 0 : index
   %d0 = tensor.dim %arg0, %c0 : tensor<?x32xf32>
@@ -196,10 +196,10 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
@@ -207,10 +207,10 @@ func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
   %empty = tensor.empty() : tensor<256x5xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<256x5xf32>
   return %0 : tensor<256x5xf32>
@@ -257,10 +257,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
   %empty = tensor.empty() : tensor<32x1xf32>
   %unpack = linalg.unpack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<16x1x1x2xf32> -> tensor<32x1xf32>

>From 4c79cf515c8213c6bd3a763dabafc5dd999fa3b2 Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:43:41 -0700
Subject: [PATCH 2/2] [mlir][linalg][test] Add more test patterns in
 `simplify-pack-unpack.mlir`.

---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 103 ++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 4c1370abac947..c8a0416c4c3c3 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x1x1x32x64xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x1x1x32x64xf32>) -> tensor<3x3x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x32x64xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x1x1x32x64xf32> -> tensor<3x3x32x64xf32>
+  return %0 : tensor<3x3x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
+  %empty = tensor.empty() : tensor<4x4xf32>
+  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
+  return %0 : tensor<4x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x32xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<1x1x32x32xf32>) -> tensor<32x32xf32> {
+  %empty = tensor.empty() : tensor<32x32xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
+  return %0 : tensor<32x32xf32>
+}
+
+// -----
+
+// Dynamic shape is not supported.
+// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
+  return %unpack : tensor<1x1000xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
+  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
+  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
+  return %result : tensor<1x1000xf32>
+}


From llvmlistbot at llvm.org  Sun Jul 12 05:59:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 05:59:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a538f99.160324b0.195192.e98e@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @JerryShih :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Sun Jul 12 05:59:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 05:59:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a538fbf.6c8668f2.e4f82.d5f0@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jerry Shih (JerryShih)

<details>
<summary>Changes</summary>

1. Add prefix `negative_` for the negative tests.
2. Add more test patterns.

---
Full diff: https://github.com/llvm/llvm-project/pull/209011.diff


1 Files Affected:

- (modified) mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir (+121-18) 


``````````diff
diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6979770154bab..c8a0416c4c3c3 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -12,11 +12,11 @@ func.func @single_dim_packing(%arg0: tensor<256xf32>) -> tensor<8x32xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @single_dim_packing_with_padding(
+// CHECK-LABEL: func.func @negative_single_dim_packing_with_padding(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<255xf32>)
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
+func.func @negative_single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
   %empty = tensor.empty() : tensor<8x32xf32>
   %cst = arith.constant 0.000000e+00 : f32
   %0 = linalg.pack %arg0 padding_value(%cst : f32) inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<255xf32> -> tensor<8x32xf32>
@@ -61,10 +61,10 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -72,10 +72,10 @@ func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<256x5xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -122,10 +122,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
   %empty = tensor.empty() : tensor<16x1x1x2xf32>
   %pack = linalg.pack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<32x1xf32> -> tensor<16x1x1x2xf32>
@@ -146,10 +146,10 @@ func.func @unpack_1d_to_collapse(%arg0: tensor<8x32xf32>) -> tensor<256xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_to_partial_slice
+// CHECK-LABEL: func.func @negative_unpack_to_partial_slice
 // CHECK-NOT:     tensor.collapse
 // CHECK:         linalg.unpack
-func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
+func.func @negative_unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
   %empty = tensor.empty() : tensor<255xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x32xf32> -> tensor<255xf32>
   return %0 : tensor<255xf32>
@@ -157,10 +157,10 @@ func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_dynamic
+// CHECK-LABEL: func.func @negative_unpack_dynamic
 // CHECK:     tensor.collapse
 // CHECK-NOT:         linalg.unpack
-func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
+func.func @negative_unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
   %c32 = arith.constant 32 : index
   %c0 = arith.constant 0 : index
   %d0 = tensor.dim %arg0, %c0 : tensor<?x32xf32>
@@ -196,10 +196,10 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
@@ -207,10 +207,10 @@ func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
   %empty = tensor.empty() : tensor<256x5xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<256x5xf32>
   return %0 : tensor<256x5xf32>
@@ -257,10 +257,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
   %empty = tensor.empty() : tensor<32x1xf32>
   %unpack = linalg.unpack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<16x1x1x2xf32> -> tensor<32x1xf32>
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x1x1x32x64xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x1x1x32x64xf32>) -> tensor<3x3x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x32x64xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x1x1x32x64xf32> -> tensor<3x3x32x64xf32>
+  return %0 : tensor<3x3x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
+  %empty = tensor.empty() : tensor<4x4xf32>
+  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
+  return %0 : tensor<4x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x32xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<1x1x32x32xf32>) -> tensor<32x32xf32> {
+  %empty = tensor.empty() : tensor<32x32xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
+  return %0 : tensor<32x32xf32>
+}
+
+// -----
+
+// Dynamic shape is not supported.
+// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
+  return %unpack : tensor<1x1000xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
+  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
+  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
+  return %result : tensor<1x1000xf32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Sun Jul 12 06:00:31 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Sun, 12 Jul 2026 06:00:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a538fef.4987c784.63cbc.498f@mx.google.com>


https://github.com/JerryShih updated https://github.com/llvm/llvm-project/pull/209011

>From f42bd22acf3e1b7f5efeb9db42bf29e77f70c26e Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:51:20 -0700
Subject: [PATCH 1/2] [mlir][linalg][test] Rename the negative tests in
 `simplify-pack-unpack.mlir`.

Add prefix `negative_` for the negative tests.
---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 36 +++++++++----------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6979770154bab..4c1370abac947 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -12,11 +12,11 @@ func.func @single_dim_packing(%arg0: tensor<256xf32>) -> tensor<8x32xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @single_dim_packing_with_padding(
+// CHECK-LABEL: func.func @negative_single_dim_packing_with_padding(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<255xf32>)
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
+func.func @negative_single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
   %empty = tensor.empty() : tensor<8x32xf32>
   %cst = arith.constant 0.000000e+00 : f32
   %0 = linalg.pack %arg0 padding_value(%cst : f32) inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<255xf32> -> tensor<8x32xf32>
@@ -61,10 +61,10 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -72,10 +72,10 @@ func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<256x5xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -122,10 +122,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
   %empty = tensor.empty() : tensor<16x1x1x2xf32>
   %pack = linalg.pack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<32x1xf32> -> tensor<16x1x1x2xf32>
@@ -146,10 +146,10 @@ func.func @unpack_1d_to_collapse(%arg0: tensor<8x32xf32>) -> tensor<256xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_to_partial_slice
+// CHECK-LABEL: func.func @negative_unpack_to_partial_slice
 // CHECK-NOT:     tensor.collapse
 // CHECK:         linalg.unpack
-func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
+func.func @negative_unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
   %empty = tensor.empty() : tensor<255xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x32xf32> -> tensor<255xf32>
   return %0 : tensor<255xf32>
@@ -157,10 +157,10 @@ func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_dynamic
+// CHECK-LABEL: func.func @negative_unpack_dynamic
 // CHECK:     tensor.collapse
 // CHECK-NOT:         linalg.unpack
-func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
+func.func @negative_unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
   %c32 = arith.constant 32 : index
   %c0 = arith.constant 0 : index
   %d0 = tensor.dim %arg0, %c0 : tensor<?x32xf32>
@@ -196,10 +196,10 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
@@ -207,10 +207,10 @@ func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
   %empty = tensor.empty() : tensor<256x5xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<256x5xf32>
   return %0 : tensor<256x5xf32>
@@ -257,10 +257,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
   %empty = tensor.empty() : tensor<32x1xf32>
   %unpack = linalg.unpack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<16x1x1x2xf32> -> tensor<32x1xf32>

>From f1ce1d19c346471ab7263cfd092f9be019228947 Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:43:41 -0700
Subject: [PATCH 2/2] [mlir][linalg][test] Add more test patterns in
 `simplify-pack-unpack.mlir`.

---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 103 ++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 4c1370abac947..c8a0416c4c3c3 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x1x1x32x64xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x1x1x32x64xf32>) -> tensor<3x3x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x32x64xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x1x1x32x64xf32> -> tensor<3x3x32x64xf32>
+  return %0 : tensor<3x3x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
+  %empty = tensor.empty() : tensor<4x4xf32>
+  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
+  return %0 : tensor<4x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x32xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<1x1x32x32xf32>) -> tensor<32x32xf32> {
+  %empty = tensor.empty() : tensor<32x32xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
+  return %0 : tensor<32x32xf32>
+}
+
+// -----
+
+// Dynamic shape is not supported.
+// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
+  return %unpack : tensor<1x1000xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
+  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
+  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
+  return %result : tensor<1x1000xf32>
+}


From llvmlistbot at llvm.org  Sun Jul 12 06:05:09 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Sun, 12 Jul 2026 06:05:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a539105.c56fb102.7c9cd.91fb@mx.google.com>


JerryShih wrote:

@banach-space 
please check this pr for https://github.com/llvm/llvm-project/pull/204971#issuecomment-4888625792

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Sun Jul 12 09:19:50 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 12 Jul 2026 09:19:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a53bea6.92d2873c.258297.d9d2@mx.google.com>


makslevental wrote:

> I'm good for this but I think it's better to be careful of [the 23.x release branch](https://discourse.llvm.org/t/llvm-23-x-will-branch-next-week-tue-14th-july/91233) to avoid branching on an intermediate state.

Ok I'll hold off until later this week


https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sun Jul 12 16:12:05 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Sun, 12 Jul 2026 16:12:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Remove Pure attribute from Linalg::IndexOp.
	(PR #68894)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/68894@github.com>
Message-ID: <6a541f45.492e23e4.14eda0.d0fc@mx.google.com>


rengolin wrote:

This PR was created to fix an issue in Linalg that isn't supported (nesting), and exposed a problem that is localize to a particular use (trying to CSE nested index operations without tracking from which region they are) that isn't expected to work anyway.

While there were many interesting discussions here on how to implement it, I believe they were all non-trivial, for a feature that is neither supported nor clearly desired. I recommend you start different discussions to make sure the wider community wants to support that functionality and if so, what would be the solution for this particular problem.

https://github.com/llvm/llvm-project/pull/68894

From llvmlistbot at llvm.org  Sun Jul 12 16:53:46 2026
From: llvmlistbot at llvm.org (David Ittah)
Date: Sun, 12 Jul 2026 16:53:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Remove Pure attribute from Linalg::IndexOp.
	(PR #68894)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/68894@github.com>
Message-ID: <6a54290a.e55dc341.1f19bd.153d@mx.google.com>


dime10 wrote:

Thank you for your response @rengolin. Respectfully, I have to disagree however. Nested linalg ops may have been how this issue was discovered, but that is not at the core of this issue. It is plain as day that the operation in its current form violates its declared semantics (as discussed at length above).

Unless we are happy to explicitly break the interface contract in an upstream dialect out of practical reasons, which seems to be the position some are arguing, I would say this is still an issue worth resolving.

https://github.com/llvm/llvm-project/pull/68894

From llvmlistbot at llvm.org  Sun Jul 12 21:01:57 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Sun, 12 Jul 2026 21:01:57 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a546335.f06f4994.1fc857.9630@mx.google.com>


================

----------------
madhur13490 wrote:

Hi @arsenm, Its been a month and I haven't heard from you. I will merge this first thing tomorrow IST morning. Please let me know if you have any final thoughts.

https://github.com/llvm/llvm-project/pull/201077

From llvmlistbot at llvm.org  Sun Jul 12 22:09:38 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Sun, 12 Jul 2026 22:09:38 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a547312.6c8668f2.e4f82.d4be@mx.google.com>


Saieiei wrote:

Ping @skatrak 

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Sun Jul 12 23:31:44 2026
From: llvmlistbot at llvm.org (Johnny Lin)
Date: Sun, 12 Jul 2026 23:31:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Lower subgroup query ops to
	NVVM (PR #209091)
Message-ID: <llvm.org/llvm/llvm-project/pull/209091@github.com>


https://github.com/johnny19436 created https://github.com/llvm/llvm-project/pull/209091

`convert-gpu-to-nvvm` currently lacks lowering coverage for `gpu.subgroup_id`, `gpu.num_subgroups`, and `gpu.subgroup_size`, while these subgroup query ops are already part of the GPU dialect.
This PR adds NVVM lowering for the three ops by mapping them to NVVM special registers (`warpid`, `nwarpid`, and `warpsize`), propagates `upper_bound` to range attributes where representable, and keeps index-bitwidth handling consistent with existing GPUToNVVM lowering patterns.
It also adds/extends `GPUToNVVM` conversion tests to cover:
- positive lowering of all three subgroup query ops,
- `upper_bound`/range behavior, and
- index-bitwidth=32 coverage.

Tested with:
- `ninja -C build mlir-opt`
- `build/bin/llvm-lit -sv mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir`

>From 483660e080a0487944b373b5087ea6be0cc167b7 Mon Sep 17 00:00:00 2001
From: johnny19436 <johnny194369672 at gmail.com>
Date: Mon, 13 Jul 2026 10:20:28 +0800
Subject: [PATCH 1/2] [mlir][gpu][nvvm] Lower subgroup query ops in GPUToNVVM

---
 .../GPUToNVVM/LowerGpuOpsToNVVMOps.cpp        | 123 ++++++++++++++++--
 .../Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir |   9 +-
 .../Conversion/GPUToNVVM/gpu-to-nvvm.mlir     |  27 +++-
 3 files changed, 137 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..939b933dfd1be 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -35,9 +35,12 @@
 #include "mlir/Transforms/DialectConversion.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
+#include "llvm/Support/MathExtras.h"
+
 #include "../GPUCommon/GPUOpsLowering.h"
 #include "../GPUCommon/IndexIntrinsicsOpLowering.h"
 #include "../GPUCommon/OpToFuncCallLowering.h"
+#include <limits>
 #include <optional>
 
 namespace mlir {
@@ -101,6 +104,31 @@ static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
 static constexpr int32_t kNVVMLastNamedBarrierId = 15;
 static constexpr int32_t kNVVMWarpSize = 32;
+static constexpr uint64_t kMaxDim = std::numeric_limits<uint32_t>::max();
+static constexpr uint64_t kMaxSubgroupSize = 128;
+
+// Truncate or extend the result depending on the index bitwidth specified by
+// the LLVMTypeConverter options.
+static Value truncOrExtToIndexBitwidth(ConversionPatternRewriter &rewriter,
+                                       Location loc, Value value,
+                                       const LLVMTypeConverter &converter) {
+  auto valueType = cast<IntegerType>(value.getType());
+  unsigned valueBitwidth = valueType.getWidth();
+  unsigned indexBitwidth = converter.getIndexTypeBitwidth();
+  if (indexBitwidth > valueBitwidth) {
+    return LLVM::SExtOp::create(rewriter, loc,
+                                IntegerType::get(rewriter.getContext(),
+                                                 indexBitwidth),
+                                value);
+  }
+  if (indexBitwidth < valueBitwidth) {
+    return LLVM::TruncOp::create(rewriter, loc,
+                                 IntegerType::get(rewriter.getContext(),
+                                                  indexBitwidth),
+                                 value);
+  }
+  return value;
+}
 
 static FailureOr<StringAttr>
 createNVVMNamedBarrierIdGlobal(gpu::InitializeNamedBarrierOp op,
@@ -256,7 +284,6 @@ struct GPULaneIdOpToNVVM : ConvertOpToLLVMPattern<gpu::LaneIdOp> {
   matchAndRewrite(gpu::LaneIdOp op, gpu::LaneIdOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = op->getLoc();
-    MLIRContext *context = rewriter.getContext();
     LLVM::ConstantRangeAttr bounds = nullptr;
     if (std::optional<APInt> upperBound = op.getUpperBound())
       bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
@@ -266,21 +293,89 @@ struct GPULaneIdOpToNVVM : ConvertOpToLLVMPattern<gpu::LaneIdOp> {
           /*bitWidth=*/32, /*lower=*/0, /*upper=*/kWarpSize);
     Value newOp =
         NVVM::LaneIdOp::create(rewriter, loc, rewriter.getI32Type(), bounds);
-    // Truncate or extend the result depending on the index bitwidth specified
-    // by the LLVMTypeConverter options.
-    const unsigned indexBitwidth = getTypeConverter()->getIndexTypeBitwidth();
-    if (indexBitwidth > 32) {
-      newOp = LLVM::SExtOp::create(
-          rewriter, loc, IntegerType::get(context, indexBitwidth), newOp);
-    } else if (indexBitwidth < 32) {
-      newOp = LLVM::TruncOp::create(
-          rewriter, loc, IntegerType::get(context, indexBitwidth), newOp);
-    }
+    newOp = truncOrExtToIndexBitwidth(rewriter, loc, newOp, *getTypeConverter());
     rewriter.replaceOp(op, {newOp});
     return success();
   }
 };
 
+struct GPUSubgroupSizeOpToNVVM
+    : ConvertOpToLLVMPattern<gpu::SubgroupSizeOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(gpu::SubgroupSizeOp op, gpu::SubgroupSizeOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    LLVM::ConstantRangeAttr bounds = nullptr;
+    if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1,
+          /*upper=*/llvm::SaturatingAdd(upperBound->getZExtValue(), 1ULL));
+    } else {
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1, /*upper=*/kMaxSubgroupSize + 1);
+    }
+
+    Value subgroupSize =
+        NVVM::WarpSizeOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                                 bounds);
+    subgroupSize = truncOrExtToIndexBitwidth(rewriter, op.getLoc(), subgroupSize,
+                                             *getTypeConverter());
+    rewriter.replaceOp(op, subgroupSize);
+    return success();
+  }
+};
+
+struct GPUSubgroupIdOpToNVVM : ConvertOpToLLVMPattern<gpu::SubgroupIdOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(gpu::SubgroupIdOp op, gpu::SubgroupIdOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // TODO: Derive tighter bounds from known launch dimensions when available.
+    uint64_t upperBound = kMaxDim;
+    if (std::optional<APInt> specifiedBound = op.getUpperBound())
+      upperBound = specifiedBound->getZExtValue();
+    LLVM::ConstantRangeAttr bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+        /*bitWidth=*/32, /*lower=*/0, /*upper=*/upperBound);
+
+    Value subgroupId =
+        NVVM::WarpIdOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                               bounds);
+    subgroupId = truncOrExtToIndexBitwidth(rewriter, op.getLoc(), subgroupId,
+                                           *getTypeConverter());
+    rewriter.replaceOp(op, subgroupId);
+    return success();
+  }
+};
+
+struct GPUNumSubgroupsOpToNVVM
+    : ConvertOpToLLVMPattern<gpu::NumSubgroupsOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult matchAndRewrite(gpu::NumSubgroupsOp op,
+                                gpu::NumSubgroupsOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter) const
+      override {
+    LLVM::ConstantRangeAttr bounds = nullptr;
+    if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1,
+          /*upper=*/llvm::SaturatingAdd(upperBound->getZExtValue(), 1ULL));
+    }
+    // Without an explicit upper bound, the target-independent default for
+    // gpu.num_subgroups cannot be encoded as a finite i32 upper-exclusive range.
+
+    Value numSubgroups =
+        NVVM::WarpDimOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                                bounds);
+    numSubgroups = truncOrExtToIndexBitwidth(rewriter, op.getLoc(),
+                                             numSubgroups, *getTypeConverter());
+    rewriter.replaceOp(op, numSubgroups);
+    return success();
+  }
+};
+
 struct GPUBallotOpToNVVM : public ConvertOpToLLVMPattern<gpu::BallotOp> {
   using ConvertOpToLLVMPattern<gpu::BallotOp>::ConvertOpToLLVMPattern;
 
@@ -666,8 +761,10 @@ void mlir::populateGpuToNVVMConversionPatterns(
   patterns.add<gpu::index_lowering::OpLowering<
       gpu::GridDimOp, NVVM::GridDimXOp, NVVM::GridDimYOp, NVVM::GridDimZOp>>(
       converter, IndexKind::Grid, IntrType::Dim, benefit);
-  patterns.add<GPULaneIdOpToNVVM, GPUBallotOpToNVVM, GPUShuffleOpLowering,
-               GPUReturnOpLowering>(converter, benefit);
+  patterns.add<GPULaneIdOpToNVVM, GPUSubgroupIdOpToNVVM,
+               GPUNumSubgroupsOpToNVVM, GPUSubgroupSizeOpToNVVM,
+               GPUBallotOpToNVVM, GPUShuffleOpLowering, GPUReturnOpLowering>(
+      converter, benefit);
 
   patterns.add<GPUDynamicSharedMemoryOpLowering>(
       converter, NVVM::kSharedMemoryAlignmentBit, benefit);
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
index 1bc7acfe46a4a..62bb55af5b026 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
@@ -7,7 +7,7 @@ gpu.module @test_module_0 {
   func.func @gpu_index_ops()
       -> (index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index) {
+          index, index, index, index) {
     %tIdX = gpu.thread_id x
     %tIdY = gpu.thread_id y
     %tIdZ = gpu.thread_id z
@@ -26,13 +26,16 @@ gpu.module @test_module_0 {
 
     // CHECK-NOT: = llvm.sext %{{.*}} : i32 to i64
     %laneId = gpu.lane_id
+    %subgroupSize = gpu.subgroup_size : index
+    %subgroupId = gpu.subgroup_id : index
+    %numSubgroups = gpu.num_subgroups : index
 
     func.return %tIdX, %tIdY, %tIdZ, %bDimX, %bDimY, %bDimZ,
                %bIdX, %bIdY, %bIdZ, %gDimX, %gDimY, %gDimZ,
-               %laneId
+               %laneId, %subgroupSize, %subgroupId, %numSubgroups
         : index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index
+          index, index, index, index
   }
 }
 
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..c3894d04741e6 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -9,7 +9,7 @@ gpu.module @test_module_0 {
   func.func @gpu_index_ops()
       -> (index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index) {
+          index, index, index, index) {
 
     // CHECK: = nvvm.read.ptx.sreg.tid.x : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
@@ -55,13 +55,22 @@ gpu.module @test_module_0 {
     // CHECK: = nvvm.read.ptx.sreg.laneid range <i32, 0, 32> : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
     %laneId = gpu.lane_id
+    // CHECK: = nvvm.read.ptx.sreg.warpsize range <i32, 1, 129> : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %subgroupSize = gpu.subgroup_size : index
+    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, 4294967295> : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %subgroupId = gpu.subgroup_id : index
+    // CHECK: = nvvm.read.ptx.sreg.nwarpid : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %numSubgroups = gpu.num_subgroups : index
 
     func.return %tIdX, %tIdY, %tIdZ, %bDimX, %bDimY, %bDimZ,
                %bIdX, %bIdY, %bIdZ, %gDimX, %gDimY, %gDimZ,
-               %laneId
+               %laneId, %subgroupSize, %subgroupId, %numSubgroups
         : index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index
+          index, index, index, index
   }
 }
 
@@ -998,15 +1007,21 @@ gpu.module @test_module_48 {
 
 gpu.module @test_module_49 {
 // CHECK-LABEL: func @explicit_id_bounds()
-  func.func @explicit_id_bounds() -> (index, index, index) {
+  func.func @explicit_id_bounds() -> (index, index, index, index, index, index) {
     // CHECK: = nvvm.read.ptx.sreg.tid.x range <i32, 0, 32> : i32
     %0 = gpu.thread_id x upper_bound 32
     // CHECK: = nvvm.read.ptx.sreg.ntid.x range <i32, 1, 33> : i32
     %1 = gpu.block_dim x upper_bound 32
     // CHECK: = nvvm.read.ptx.sreg.laneid range <i32, 0, 16> : i32
     %2 = gpu.lane_id upper_bound 16
-
-    return %0, %1, %2 : index, index, index
+    // CHECK: = nvvm.read.ptx.sreg.warpsize range <i32, 1, 33> : i32
+    %3 = gpu.subgroup_size upper_bound 32 : index
+    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, 16> : i32
+    %4 = gpu.subgroup_id upper_bound 16 : index
+    // CHECK: = nvvm.read.ptx.sreg.nwarpid range <i32, 1, 17> : i32
+    %5 = gpu.num_subgroups upper_bound 16 : index
+
+    return %0, %1, %2, %3, %4, %5 : index, index, index, index, index, index
   }
 }
 

>From 06235877c8f53f93492716c366017e4a77d4fc42 Mon Sep 17 00:00:00 2001
From: johnny19436 <johnny194369672 at gmail.com>
Date: Mon, 13 Jul 2026 14:31:08 +0800
Subject: [PATCH 2/2] fix

---
 .../GPUToNVVM/LowerGpuOpsToNVVMOps.cpp           | 16 ++++++++++------
 mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir  |  2 +-
 2 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 939b933dfd1be..52271ddd333f0 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -104,8 +104,8 @@ static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
 static constexpr int32_t kNVVMLastNamedBarrierId = 15;
 static constexpr int32_t kNVVMWarpSize = 32;
-static constexpr uint64_t kMaxDim = std::numeric_limits<uint32_t>::max();
-static constexpr uint64_t kMaxSubgroupSize = 128;
+static constexpr uint32_t kMaxDim = std::numeric_limits<uint32_t>::max();
+static constexpr uint32_t kMaxSubgroupSize = 128;
 
 // Truncate or extend the result depending on the index bitwidth specified by
 // the LLVMTypeConverter options.
@@ -308,9 +308,11 @@ struct GPUSubgroupSizeOpToNVVM
                   ConversionPatternRewriter &rewriter) const override {
     LLVM::ConstantRangeAttr bounds = nullptr;
     if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      uint32_t subgroupUpperBound =
+          static_cast<uint32_t>(upperBound->getLimitedValue(kMaxDim));
       bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
           /*bitWidth=*/32, /*lower=*/1,
-          /*upper=*/llvm::SaturatingAdd(upperBound->getZExtValue(), 1ULL));
+          /*upper=*/llvm::SaturatingAdd(subgroupUpperBound, uint32_t{1}));
     } else {
       bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
           /*bitWidth=*/32, /*lower=*/1, /*upper=*/kMaxSubgroupSize + 1);
@@ -333,9 +335,9 @@ struct GPUSubgroupIdOpToNVVM : ConvertOpToLLVMPattern<gpu::SubgroupIdOp> {
   matchAndRewrite(gpu::SubgroupIdOp op, gpu::SubgroupIdOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     // TODO: Derive tighter bounds from known launch dimensions when available.
-    uint64_t upperBound = kMaxDim;
+    uint32_t upperBound = kMaxDim;
     if (std::optional<APInt> specifiedBound = op.getUpperBound())
-      upperBound = specifiedBound->getZExtValue();
+      upperBound = static_cast<uint32_t>(specifiedBound->getLimitedValue(kMaxDim));
     LLVM::ConstantRangeAttr bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
         /*bitWidth=*/32, /*lower=*/0, /*upper=*/upperBound);
 
@@ -359,9 +361,11 @@ struct GPUNumSubgroupsOpToNVVM
       override {
     LLVM::ConstantRangeAttr bounds = nullptr;
     if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      uint32_t numSubgroupsUpperBound =
+          static_cast<uint32_t>(upperBound->getLimitedValue(kMaxDim));
       bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
           /*bitWidth=*/32, /*lower=*/1,
-          /*upper=*/llvm::SaturatingAdd(upperBound->getZExtValue(), 1ULL));
+          /*upper=*/llvm::SaturatingAdd(numSubgroupsUpperBound, uint32_t{1}));
     }
     // Without an explicit upper bound, the target-independent default for
     // gpu.num_subgroups cannot be encoded as a finite i32 upper-exclusive range.
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index c3894d04741e6..c18bb0e363d64 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -58,7 +58,7 @@ gpu.module @test_module_0 {
     // CHECK: = nvvm.read.ptx.sreg.warpsize range <i32, 1, 129> : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
     %subgroupSize = gpu.subgroup_size : index
-    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, 4294967295> : i32
+    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, -1> : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
     %subgroupId = gpu.subgroup_id : index
     // CHECK: = nvvm.read.ptx.sreg.nwarpid : i32


From llvmlistbot at llvm.org  Sun Jul 12 23:32:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 23:32:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Lower subgroup query ops to
	NVVM (PR #209091)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209091@github.com>
Message-ID: <6a548660.bd93e731.358318.1e83@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @johnny19436 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/209091

From llvmlistbot at llvm.org  Sun Jul 12 23:32:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 12 Jul 2026 23:32:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Lower subgroup query ops to
	NVVM (PR #209091)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209091@github.com>
Message-ID: <6a548684.bb839515.27d21e.c86d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Johnny Lin (johnny19436)

<details>
<summary>Changes</summary>

`convert-gpu-to-nvvm` currently lacks lowering coverage for `gpu.subgroup_id`, `gpu.num_subgroups`, and `gpu.subgroup_size`, while these subgroup query ops are already part of the GPU dialect.
This PR adds NVVM lowering for the three ops by mapping them to NVVM special registers (`warpid`, `nwarpid`, and `warpsize`), propagates `upper_bound` to range attributes where representable, and keeps index-bitwidth handling consistent with existing GPUToNVVM lowering patterns.
It also adds/extends `GPUToNVVM` conversion tests to cover:
- positive lowering of all three subgroup query ops,
- `upper_bound`/range behavior, and
- index-bitwidth=32 coverage.

Tested with:
- `ninja -C build mlir-opt`
- `build/bin/llvm-lit -sv mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir`

---
Full diff: https://github.com/llvm/llvm-project/pull/209091.diff


3 Files Affected:

- (modified) mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp (+114-13) 
- (modified) mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir (+6-3) 
- (modified) mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir (+21-6) 


``````````diff
diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..52271ddd333f0 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -35,9 +35,12 @@
 #include "mlir/Transforms/DialectConversion.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 
+#include "llvm/Support/MathExtras.h"
+
 #include "../GPUCommon/GPUOpsLowering.h"
 #include "../GPUCommon/IndexIntrinsicsOpLowering.h"
 #include "../GPUCommon/OpToFuncCallLowering.h"
+#include <limits>
 #include <optional>
 
 namespace mlir {
@@ -101,6 +104,31 @@ static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
 static constexpr int32_t kNVVMLastNamedBarrierId = 15;
 static constexpr int32_t kNVVMWarpSize = 32;
+static constexpr uint32_t kMaxDim = std::numeric_limits<uint32_t>::max();
+static constexpr uint32_t kMaxSubgroupSize = 128;
+
+// Truncate or extend the result depending on the index bitwidth specified by
+// the LLVMTypeConverter options.
+static Value truncOrExtToIndexBitwidth(ConversionPatternRewriter &rewriter,
+                                       Location loc, Value value,
+                                       const LLVMTypeConverter &converter) {
+  auto valueType = cast<IntegerType>(value.getType());
+  unsigned valueBitwidth = valueType.getWidth();
+  unsigned indexBitwidth = converter.getIndexTypeBitwidth();
+  if (indexBitwidth > valueBitwidth) {
+    return LLVM::SExtOp::create(rewriter, loc,
+                                IntegerType::get(rewriter.getContext(),
+                                                 indexBitwidth),
+                                value);
+  }
+  if (indexBitwidth < valueBitwidth) {
+    return LLVM::TruncOp::create(rewriter, loc,
+                                 IntegerType::get(rewriter.getContext(),
+                                                  indexBitwidth),
+                                 value);
+  }
+  return value;
+}
 
 static FailureOr<StringAttr>
 createNVVMNamedBarrierIdGlobal(gpu::InitializeNamedBarrierOp op,
@@ -256,7 +284,6 @@ struct GPULaneIdOpToNVVM : ConvertOpToLLVMPattern<gpu::LaneIdOp> {
   matchAndRewrite(gpu::LaneIdOp op, gpu::LaneIdOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = op->getLoc();
-    MLIRContext *context = rewriter.getContext();
     LLVM::ConstantRangeAttr bounds = nullptr;
     if (std::optional<APInt> upperBound = op.getUpperBound())
       bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
@@ -266,21 +293,93 @@ struct GPULaneIdOpToNVVM : ConvertOpToLLVMPattern<gpu::LaneIdOp> {
           /*bitWidth=*/32, /*lower=*/0, /*upper=*/kWarpSize);
     Value newOp =
         NVVM::LaneIdOp::create(rewriter, loc, rewriter.getI32Type(), bounds);
-    // Truncate or extend the result depending on the index bitwidth specified
-    // by the LLVMTypeConverter options.
-    const unsigned indexBitwidth = getTypeConverter()->getIndexTypeBitwidth();
-    if (indexBitwidth > 32) {
-      newOp = LLVM::SExtOp::create(
-          rewriter, loc, IntegerType::get(context, indexBitwidth), newOp);
-    } else if (indexBitwidth < 32) {
-      newOp = LLVM::TruncOp::create(
-          rewriter, loc, IntegerType::get(context, indexBitwidth), newOp);
-    }
+    newOp = truncOrExtToIndexBitwidth(rewriter, loc, newOp, *getTypeConverter());
     rewriter.replaceOp(op, {newOp});
     return success();
   }
 };
 
+struct GPUSubgroupSizeOpToNVVM
+    : ConvertOpToLLVMPattern<gpu::SubgroupSizeOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(gpu::SubgroupSizeOp op, gpu::SubgroupSizeOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    LLVM::ConstantRangeAttr bounds = nullptr;
+    if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      uint32_t subgroupUpperBound =
+          static_cast<uint32_t>(upperBound->getLimitedValue(kMaxDim));
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1,
+          /*upper=*/llvm::SaturatingAdd(subgroupUpperBound, uint32_t{1}));
+    } else {
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1, /*upper=*/kMaxSubgroupSize + 1);
+    }
+
+    Value subgroupSize =
+        NVVM::WarpSizeOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                                 bounds);
+    subgroupSize = truncOrExtToIndexBitwidth(rewriter, op.getLoc(), subgroupSize,
+                                             *getTypeConverter());
+    rewriter.replaceOp(op, subgroupSize);
+    return success();
+  }
+};
+
+struct GPUSubgroupIdOpToNVVM : ConvertOpToLLVMPattern<gpu::SubgroupIdOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(gpu::SubgroupIdOp op, gpu::SubgroupIdOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // TODO: Derive tighter bounds from known launch dimensions when available.
+    uint32_t upperBound = kMaxDim;
+    if (std::optional<APInt> specifiedBound = op.getUpperBound())
+      upperBound = static_cast<uint32_t>(specifiedBound->getLimitedValue(kMaxDim));
+    LLVM::ConstantRangeAttr bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+        /*bitWidth=*/32, /*lower=*/0, /*upper=*/upperBound);
+
+    Value subgroupId =
+        NVVM::WarpIdOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                               bounds);
+    subgroupId = truncOrExtToIndexBitwidth(rewriter, op.getLoc(), subgroupId,
+                                           *getTypeConverter());
+    rewriter.replaceOp(op, subgroupId);
+    return success();
+  }
+};
+
+struct GPUNumSubgroupsOpToNVVM
+    : ConvertOpToLLVMPattern<gpu::NumSubgroupsOp> {
+  using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
+
+  LogicalResult matchAndRewrite(gpu::NumSubgroupsOp op,
+                                gpu::NumSubgroupsOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter) const
+      override {
+    LLVM::ConstantRangeAttr bounds = nullptr;
+    if (std::optional<APInt> upperBound = op.getUpperBound()) {
+      uint32_t numSubgroupsUpperBound =
+          static_cast<uint32_t>(upperBound->getLimitedValue(kMaxDim));
+      bounds = rewriter.getAttr<LLVM::ConstantRangeAttr>(
+          /*bitWidth=*/32, /*lower=*/1,
+          /*upper=*/llvm::SaturatingAdd(numSubgroupsUpperBound, uint32_t{1}));
+    }
+    // Without an explicit upper bound, the target-independent default for
+    // gpu.num_subgroups cannot be encoded as a finite i32 upper-exclusive range.
+
+    Value numSubgroups =
+        NVVM::WarpDimOp::create(rewriter, op.getLoc(), rewriter.getI32Type(),
+                                bounds);
+    numSubgroups = truncOrExtToIndexBitwidth(rewriter, op.getLoc(),
+                                             numSubgroups, *getTypeConverter());
+    rewriter.replaceOp(op, numSubgroups);
+    return success();
+  }
+};
+
 struct GPUBallotOpToNVVM : public ConvertOpToLLVMPattern<gpu::BallotOp> {
   using ConvertOpToLLVMPattern<gpu::BallotOp>::ConvertOpToLLVMPattern;
 
@@ -666,8 +765,10 @@ void mlir::populateGpuToNVVMConversionPatterns(
   patterns.add<gpu::index_lowering::OpLowering<
       gpu::GridDimOp, NVVM::GridDimXOp, NVVM::GridDimYOp, NVVM::GridDimZOp>>(
       converter, IndexKind::Grid, IntrType::Dim, benefit);
-  patterns.add<GPULaneIdOpToNVVM, GPUBallotOpToNVVM, GPUShuffleOpLowering,
-               GPUReturnOpLowering>(converter, benefit);
+  patterns.add<GPULaneIdOpToNVVM, GPUSubgroupIdOpToNVVM,
+               GPUNumSubgroupsOpToNVVM, GPUSubgroupSizeOpToNVVM,
+               GPUBallotOpToNVVM, GPUShuffleOpLowering, GPUReturnOpLowering>(
+      converter, benefit);
 
   patterns.add<GPUDynamicSharedMemoryOpLowering>(
       converter, NVVM::kSharedMemoryAlignmentBit, benefit);
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
index 1bc7acfe46a4a..62bb55af5b026 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm-32b.mlir
@@ -7,7 +7,7 @@ gpu.module @test_module_0 {
   func.func @gpu_index_ops()
       -> (index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index) {
+          index, index, index, index) {
     %tIdX = gpu.thread_id x
     %tIdY = gpu.thread_id y
     %tIdZ = gpu.thread_id z
@@ -26,13 +26,16 @@ gpu.module @test_module_0 {
 
     // CHECK-NOT: = llvm.sext %{{.*}} : i32 to i64
     %laneId = gpu.lane_id
+    %subgroupSize = gpu.subgroup_size : index
+    %subgroupId = gpu.subgroup_id : index
+    %numSubgroups = gpu.num_subgroups : index
 
     func.return %tIdX, %tIdY, %tIdZ, %bDimX, %bDimY, %bDimZ,
                %bIdX, %bIdY, %bIdZ, %gDimX, %gDimY, %gDimZ,
-               %laneId
+               %laneId, %subgroupSize, %subgroupId, %numSubgroups
         : index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index
+          index, index, index, index
   }
 }
 
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..c18bb0e363d64 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -9,7 +9,7 @@ gpu.module @test_module_0 {
   func.func @gpu_index_ops()
       -> (index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index) {
+          index, index, index, index) {
 
     // CHECK: = nvvm.read.ptx.sreg.tid.x : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
@@ -55,13 +55,22 @@ gpu.module @test_module_0 {
     // CHECK: = nvvm.read.ptx.sreg.laneid range <i32, 0, 32> : i32
     // CHECK: = llvm.sext %{{.*}} : i32 to i64
     %laneId = gpu.lane_id
+    // CHECK: = nvvm.read.ptx.sreg.warpsize range <i32, 1, 129> : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %subgroupSize = gpu.subgroup_size : index
+    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, -1> : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %subgroupId = gpu.subgroup_id : index
+    // CHECK: = nvvm.read.ptx.sreg.nwarpid : i32
+    // CHECK: = llvm.sext %{{.*}} : i32 to i64
+    %numSubgroups = gpu.num_subgroups : index
 
     func.return %tIdX, %tIdY, %tIdZ, %bDimX, %bDimY, %bDimZ,
                %bIdX, %bIdY, %bIdZ, %gDimX, %gDimY, %gDimZ,
-               %laneId
+               %laneId, %subgroupSize, %subgroupId, %numSubgroups
         : index, index, index, index, index, index,
           index, index, index, index, index, index,
-          index
+          index, index, index, index
   }
 }
 
@@ -998,15 +1007,21 @@ gpu.module @test_module_48 {
 
 gpu.module @test_module_49 {
 // CHECK-LABEL: func @explicit_id_bounds()
-  func.func @explicit_id_bounds() -> (index, index, index) {
+  func.func @explicit_id_bounds() -> (index, index, index, index, index, index) {
     // CHECK: = nvvm.read.ptx.sreg.tid.x range <i32, 0, 32> : i32
     %0 = gpu.thread_id x upper_bound 32
     // CHECK: = nvvm.read.ptx.sreg.ntid.x range <i32, 1, 33> : i32
     %1 = gpu.block_dim x upper_bound 32
     // CHECK: = nvvm.read.ptx.sreg.laneid range <i32, 0, 16> : i32
     %2 = gpu.lane_id upper_bound 16
-
-    return %0, %1, %2 : index, index, index
+    // CHECK: = nvvm.read.ptx.sreg.warpsize range <i32, 1, 33> : i32
+    %3 = gpu.subgroup_size upper_bound 32 : index
+    // CHECK: = nvvm.read.ptx.sreg.warpid range <i32, 0, 16> : i32
+    %4 = gpu.subgroup_id upper_bound 16 : index
+    // CHECK: = nvvm.read.ptx.sreg.nwarpid range <i32, 1, 17> : i32
+    %5 = gpu.num_subgroups upper_bound 16 : index
+
+    return %0, %1, %2, %3, %4, %5 : index, index, index, index, index, index
   }
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/209091

From llvmlistbot at llvm.org  Mon Jul 13 00:17:18 2026
From: llvmlistbot at llvm.org (Johnny Lin)
Date: Mon, 13 Jul 2026 00:17:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Add subgroup_reduce shuffle
 fallback for clustered and non-i32 cases (PR #209098)
Message-ID: <llvm.org/llvm/llvm-project/pull/209098@github.com>


https://github.com/johnny19436 created https://github.com/llvm/llvm-project/pull/209098

`gpu.subgroup_reduce` lowering in GPUToNVVM currently relies on the `nvvm.redux.sync`
fast path and rejects clustered reductions and non-i32 payloads.

This patch keeps the existing redux fast path unchanged for supported uniform i32
cases, and adds a fallback path for unsupported uniform subgroup reductions by
reusing existing GPU subgroup-reduce rewrite patterns:
- `populateGpuBreakDownSubgroupReducePatterns`
- `populateGpuLowerSubgroupReduceToShufflePatterns`
- `populateGpuLowerClusteredSubgroupReduceToShufflePatterns`

The fallback is applied before LLVM conversion and lowers these cases to
`gpu.shuffle`-based reductions, which are then legalized by existing NVVM
shuffle lowering.

Behavior after this patch:
- uniform + i32 + non-clustered + redux-supported op: still lowers to `nvvm.redux.sync`
- uniform + non-i32: now lowers via shuffle fallback
- uniform + clustered: now lowers via shuffle fallback
- non-uniform subgroup_reduce behavior is unchanged in this patch

Tests:
- Updated `mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir` with focused cases for:
  - existing redux fast path
  - non-i32 fallback
  - clustered fallback

Commands run:
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1 allow-pattern-rollback=0' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`

>From 2de9afd3768ec6fd8e72c59cb30b9f4d68387530 Mon Sep 17 00:00:00 2001
From: johnny19436 <johnny194369672 at gmail.com>
Date: Mon, 13 Jul 2026 15:19:33 +0800
Subject: [PATCH] [mlir][gpu][nvvm] Add subgroup_reduce shuffle fallback for
 clustered and non-i32 cases

---
 .../GPUToNVVM/LowerGpuOpsToNVVMOps.cpp        | 35 +++++++++++++++++++
 .../Conversion/GPUToNVVM/gpu-to-nvvm.mlir     | 22 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..59b150deb546d 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -96,6 +96,14 @@ convertToNVVMReductionKind(gpu::AllReduceOperation mode) {
   return std::nullopt;
 }
 
+static bool canLowerSubgroupReduceToNVVMRedux(gpu::SubgroupReduceOp op) {
+  if (op.getClusterSize() || !op.getUniform())
+    return false;
+  if (!op.getValue().getType().isInteger(32))
+    return false;
+  return convertToNVVMReductionKind(op.getOp()).has_value();
+}
+
 static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
     "__named_barrier_id";
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
@@ -529,6 +537,33 @@ struct LowerGpuOpsToNVVMOpsPass final
     // ops which need to be lowered further, which is not supported by a
     // single conversion pass.
     {
+      // Lower subgroup reductions that cannot use nvvm.redux to shuffles
+      // before conversion. Keep redux-compatible cases untouched so the
+      // dedicated conversion pattern still applies.
+      SmallVector<Operation *> subgroupReduceOpsToLower;
+      m.walk([&](gpu::SubgroupReduceOp op) {
+        if (!op.getUniform())
+          return;
+        if (!this->hasRedux || !canLowerSubgroupReduceToNVVMRedux(op))
+          subgroupReduceOpsToLower.push_back(op.getOperation());
+      });
+      if (!subgroupReduceOpsToLower.empty()) {
+        RewritePatternSet subgroupReducePatterns(m.getContext());
+        populateGpuBreakDownSubgroupReducePatterns(
+            subgroupReducePatterns, /*maxShuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerClusteredSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        if (failed(applyOpPatternsGreedily(subgroupReduceOpsToLower,
+                                           std::move(subgroupReducePatterns))))
+          return signalPassFailure();
+      }
+
       RewritePatternSet patterns(m.getContext());
       populateGpuRewritePatterns(patterns);
       // Transform N-D vector.from_elements to 1-D vector.from_elements before
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..8c76285b3e47e 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -739,6 +739,28 @@ gpu.module @test_module_30 {
   }
 }
 
+gpu.module @test_module_30_fallback {
+  // CHECK-LABEL: func @subgroup_reduce_add_f32_fallback
+  gpu.func @subgroup_reduce_add_f32_fallback(%arg0 : f32, %buf : memref<f32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform {} : (f32) -> (f32)
+    memref.store %result, %buf[] : memref<f32>
+    gpu.return
+  }
+
+  // CHECK-LABEL: func @subgroup_reduce_clustered_i32_fallback
+  gpu.func @subgroup_reduce_clustered_i32_fallback(%arg0 : i32,
+                                                   %buf : memref<i32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform cluster(size = 8) :
+        (i32) -> (i32)
+    memref.store %result, %buf[] : memref<i32>
+    gpu.return
+  }
+}
+
 gpu.module @test_module_31 {
   // CHECK: llvm.func @__nv_fmodf(f32, f32) -> f32
   // CHECK: llvm.func @__nv_fmod(f64, f64) -> f64


From llvmlistbot at llvm.org  Mon Jul 13 00:17:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 00:17:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Add subgroup_reduce shuffle
 fallback for clustered and non-i32 cases (PR #209098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209098@github.com>
Message-ID: <6a549110.743400b0.2067c9.374e@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @johnny19436 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/209098

From llvmlistbot at llvm.org  Mon Jul 13 00:18:00 2026
From: llvmlistbot at llvm.org (Johnny Lin)
Date: Mon, 13 Jul 2026 00:18:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Add subgroup_reduce shuffle
 fallback for clustered and non-i32 cases (PR #209098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209098@github.com>
Message-ID: <6a549128.92d2873c.258297.db6d@mx.google.com>


https://github.com/johnny19436 updated https://github.com/llvm/llvm-project/pull/209098

>From 4fab19a557f8eefcfc40fdb5263ded9ca27c31f6 Mon Sep 17 00:00:00 2001
From: johnny19436 <johnny194369672 at gmail.com>
Date: Mon, 13 Jul 2026 15:19:33 +0800
Subject: [PATCH] [mlir][gpu][nvvm] Add subgroup_reduce shuffle fallback for
 clustered and non-i32 cases

---
 .../GPUToNVVM/LowerGpuOpsToNVVMOps.cpp        | 35 +++++++++++++++++++
 .../Conversion/GPUToNVVM/gpu-to-nvvm.mlir     | 22 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..59b150deb546d 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -96,6 +96,14 @@ convertToNVVMReductionKind(gpu::AllReduceOperation mode) {
   return std::nullopt;
 }
 
+static bool canLowerSubgroupReduceToNVVMRedux(gpu::SubgroupReduceOp op) {
+  if (op.getClusterSize() || !op.getUniform())
+    return false;
+  if (!op.getValue().getType().isInteger(32))
+    return false;
+  return convertToNVVMReductionKind(op.getOp()).has_value();
+}
+
 static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
     "__named_barrier_id";
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
@@ -529,6 +537,33 @@ struct LowerGpuOpsToNVVMOpsPass final
     // ops which need to be lowered further, which is not supported by a
     // single conversion pass.
     {
+      // Lower subgroup reductions that cannot use nvvm.redux to shuffles
+      // before conversion. Keep redux-compatible cases untouched so the
+      // dedicated conversion pattern still applies.
+      SmallVector<Operation *> subgroupReduceOpsToLower;
+      m.walk([&](gpu::SubgroupReduceOp op) {
+        if (!op.getUniform())
+          return;
+        if (!this->hasRedux || !canLowerSubgroupReduceToNVVMRedux(op))
+          subgroupReduceOpsToLower.push_back(op.getOperation());
+      });
+      if (!subgroupReduceOpsToLower.empty()) {
+        RewritePatternSet subgroupReducePatterns(m.getContext());
+        populateGpuBreakDownSubgroupReducePatterns(
+            subgroupReducePatterns, /*maxShuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerClusteredSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        if (failed(applyOpPatternsGreedily(subgroupReduceOpsToLower,
+                                           std::move(subgroupReducePatterns))))
+          return signalPassFailure();
+      }
+
       RewritePatternSet patterns(m.getContext());
       populateGpuRewritePatterns(patterns);
       // Transform N-D vector.from_elements to 1-D vector.from_elements before
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..8c76285b3e47e 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -739,6 +739,28 @@ gpu.module @test_module_30 {
   }
 }
 
+gpu.module @test_module_30_fallback {
+  // CHECK-LABEL: func @subgroup_reduce_add_f32_fallback
+  gpu.func @subgroup_reduce_add_f32_fallback(%arg0 : f32, %buf : memref<f32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform {} : (f32) -> (f32)
+    memref.store %result, %buf[] : memref<f32>
+    gpu.return
+  }
+
+  // CHECK-LABEL: func @subgroup_reduce_clustered_i32_fallback
+  gpu.func @subgroup_reduce_clustered_i32_fallback(%arg0 : i32,
+                                                   %buf : memref<i32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform cluster(size = 8) :
+        (i32) -> (i32)
+    memref.store %result, %buf[] : memref<i32>
+    gpu.return
+  }
+}
+
 gpu.module @test_module_31 {
   // CHECK: llvm.func @__nv_fmodf(f32, f32) -> f32
   // CHECK: llvm.func @__nv_fmod(f64, f64) -> f64


From llvmlistbot at llvm.org  Mon Jul 13 00:18:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 00:18:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Add subgroup_reduce shuffle
 fallback for clustered and non-i32 cases (PR #209098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209098@github.com>
Message-ID: <6a549137.bd93e731.358318.f3a3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Johnny Lin (johnny19436)

<details>
<summary>Changes</summary>

`gpu.subgroup_reduce` lowering in GPUToNVVM currently relies on the `nvvm.redux.sync`
fast path and rejects clustered reductions and non-i32 payloads.

This patch keeps the existing redux fast path unchanged for supported uniform i32
cases, and adds a fallback path for unsupported uniform subgroup reductions by
reusing existing GPU subgroup-reduce rewrite patterns:
- `populateGpuBreakDownSubgroupReducePatterns`
- `populateGpuLowerSubgroupReduceToShufflePatterns`
- `populateGpuLowerClusteredSubgroupReduceToShufflePatterns`

The fallback is applied before LLVM conversion and lowers these cases to
`gpu.shuffle`-based reductions, which are then legalized by existing NVVM
shuffle lowering.

Behavior after this patch:
- uniform + i32 + non-clustered + redux-supported op: still lowers to `nvvm.redux.sync`
- uniform + non-i32: now lowers via shuffle fallback
- uniform + clustered: now lowers via shuffle fallback
- non-uniform subgroup_reduce behavior is unchanged in this patch

Tests:
- Updated `mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir` with focused cases for:
  - existing redux fast path
  - non-i32 fallback
  - clustered fallback

Commands run:
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1 allow-pattern-rollback=0' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`

---
Full diff: https://github.com/llvm/llvm-project/pull/209098.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp (+35) 
- (modified) mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir (+22) 


``````````diff
diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..59b150deb546d 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -96,6 +96,14 @@ convertToNVVMReductionKind(gpu::AllReduceOperation mode) {
   return std::nullopt;
 }
 
+static bool canLowerSubgroupReduceToNVVMRedux(gpu::SubgroupReduceOp op) {
+  if (op.getClusterSize() || !op.getUniform())
+    return false;
+  if (!op.getValue().getType().isInteger(32))
+    return false;
+  return convertToNVVMReductionKind(op.getOp()).has_value();
+}
+
 static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
     "__named_barrier_id";
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
@@ -529,6 +537,33 @@ struct LowerGpuOpsToNVVMOpsPass final
     // ops which need to be lowered further, which is not supported by a
     // single conversion pass.
     {
+      // Lower subgroup reductions that cannot use nvvm.redux to shuffles
+      // before conversion. Keep redux-compatible cases untouched so the
+      // dedicated conversion pattern still applies.
+      SmallVector<Operation *> subgroupReduceOpsToLower;
+      m.walk([&](gpu::SubgroupReduceOp op) {
+        if (!op.getUniform())
+          return;
+        if (!this->hasRedux || !canLowerSubgroupReduceToNVVMRedux(op))
+          subgroupReduceOpsToLower.push_back(op.getOperation());
+      });
+      if (!subgroupReduceOpsToLower.empty()) {
+        RewritePatternSet subgroupReducePatterns(m.getContext());
+        populateGpuBreakDownSubgroupReducePatterns(
+            subgroupReducePatterns, /*maxShuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerClusteredSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        if (failed(applyOpPatternsGreedily(subgroupReduceOpsToLower,
+                                           std::move(subgroupReducePatterns))))
+          return signalPassFailure();
+      }
+
       RewritePatternSet patterns(m.getContext());
       populateGpuRewritePatterns(patterns);
       // Transform N-D vector.from_elements to 1-D vector.from_elements before
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..8c76285b3e47e 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -739,6 +739,28 @@ gpu.module @test_module_30 {
   }
 }
 
+gpu.module @test_module_30_fallback {
+  // CHECK-LABEL: func @subgroup_reduce_add_f32_fallback
+  gpu.func @subgroup_reduce_add_f32_fallback(%arg0 : f32, %buf : memref<f32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform {} : (f32) -> (f32)
+    memref.store %result, %buf[] : memref<f32>
+    gpu.return
+  }
+
+  // CHECK-LABEL: func @subgroup_reduce_clustered_i32_fallback
+  gpu.func @subgroup_reduce_clustered_i32_fallback(%arg0 : i32,
+                                                   %buf : memref<i32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform cluster(size = 8) :
+        (i32) -> (i32)
+    memref.store %result, %buf[] : memref<i32>
+    gpu.return
+  }
+}
+
 gpu.module @test_module_31 {
   // CHECK: llvm.func @__nv_fmodf(f32, f32) -> f32
   // CHECK: llvm.func @__nv_fmod(f64, f64) -> f64

``````````

</details>


https://github.com/llvm/llvm-project/pull/209098

From llvmlistbot at llvm.org  Mon Jul 13 00:18:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 00:18:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu][nvvm] Add subgroup_reduce shuffle
 fallback for clustered and non-i32 cases (PR #209098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209098@github.com>
Message-ID: <6a549138.0c05e68e.1e82f1.da69@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Johnny Lin (johnny19436)

<details>
<summary>Changes</summary>

`gpu.subgroup_reduce` lowering in GPUToNVVM currently relies on the `nvvm.redux.sync`
fast path and rejects clustered reductions and non-i32 payloads.

This patch keeps the existing redux fast path unchanged for supported uniform i32
cases, and adds a fallback path for unsupported uniform subgroup reductions by
reusing existing GPU subgroup-reduce rewrite patterns:
- `populateGpuBreakDownSubgroupReducePatterns`
- `populateGpuLowerSubgroupReduceToShufflePatterns`
- `populateGpuLowerClusteredSubgroupReduceToShufflePatterns`

The fallback is applied before LLVM conversion and lowers these cases to
`gpu.shuffle`-based reductions, which are then legalized by existing NVVM
shuffle lowering.

Behavior after this patch:
- uniform + i32 + non-clustered + redux-supported op: still lowers to `nvvm.redux.sync`
- uniform + non-i32: now lowers via shuffle fallback
- uniform + clustered: now lowers via shuffle fallback
- non-uniform subgroup_reduce behavior is unchanged in this patch

Tests:
- Updated `mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir` with focused cases for:
  - existing redux fast path
  - non-i32 fallback
  - clustered fallback

Commands run:
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`
- `mlir-opt mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir -convert-gpu-to-nvvm='has-redux=1 allow-pattern-rollback=0' -split-input-file | FileCheck mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir`

---
Full diff: https://github.com/llvm/llvm-project/pull/209098.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp (+35) 
- (modified) mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir (+22) 


``````````diff
diff --git a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
index 80420c26537c3..59b150deb546d 100644
--- a/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
+++ b/mlir/lib/Conversion/GPUToNVVM/LowerGpuOpsToNVVMOps.cpp
@@ -96,6 +96,14 @@ convertToNVVMReductionKind(gpu::AllReduceOperation mode) {
   return std::nullopt;
 }
 
+static bool canLowerSubgroupReduceToNVVMRedux(gpu::SubgroupReduceOp op) {
+  if (op.getClusterSize() || !op.getUniform())
+    return false;
+  if (!op.getValue().getType().isInteger(32))
+    return false;
+  return convertToNVVMReductionKind(op.getOp()).has_value();
+}
+
 static constexpr llvm::StringLiteral kNVVMNamedBarrierIdPrefix =
     "__named_barrier_id";
 static constexpr int32_t kNVVMFirstNamedBarrierId = 1;
@@ -529,6 +537,33 @@ struct LowerGpuOpsToNVVMOpsPass final
     // ops which need to be lowered further, which is not supported by a
     // single conversion pass.
     {
+      // Lower subgroup reductions that cannot use nvvm.redux to shuffles
+      // before conversion. Keep redux-compatible cases untouched so the
+      // dedicated conversion pattern still applies.
+      SmallVector<Operation *> subgroupReduceOpsToLower;
+      m.walk([&](gpu::SubgroupReduceOp op) {
+        if (!op.getUniform())
+          return;
+        if (!this->hasRedux || !canLowerSubgroupReduceToNVVMRedux(op))
+          subgroupReduceOpsToLower.push_back(op.getOperation());
+      });
+      if (!subgroupReduceOpsToLower.empty()) {
+        RewritePatternSet subgroupReducePatterns(m.getContext());
+        populateGpuBreakDownSubgroupReducePatterns(
+            subgroupReducePatterns, /*maxShuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        populateGpuLowerClusteredSubgroupReduceToShufflePatterns(
+            subgroupReducePatterns,
+            /*subgroupSize=*/kNVVMWarpSize,
+            /*shuffleBitwidth=*/kNVVMWarpSize);
+        if (failed(applyOpPatternsGreedily(subgroupReduceOpsToLower,
+                                           std::move(subgroupReducePatterns))))
+          return signalPassFailure();
+      }
+
       RewritePatternSet patterns(m.getContext());
       populateGpuRewritePatterns(patterns);
       // Transform N-D vector.from_elements to 1-D vector.from_elements before
diff --git a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
index b96069ac41a44..8c76285b3e47e 100644
--- a/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir
@@ -739,6 +739,28 @@ gpu.module @test_module_30 {
   }
 }
 
+gpu.module @test_module_30_fallback {
+  // CHECK-LABEL: func @subgroup_reduce_add_f32_fallback
+  gpu.func @subgroup_reduce_add_f32_fallback(%arg0 : f32, %buf : memref<f32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform {} : (f32) -> (f32)
+    memref.store %result, %buf[] : memref<f32>
+    gpu.return
+  }
+
+  // CHECK-LABEL: func @subgroup_reduce_clustered_i32_fallback
+  gpu.func @subgroup_reduce_clustered_i32_fallback(%arg0 : i32,
+                                                   %buf : memref<i32>) {
+    // CHECK-NOT: nvvm.redux.sync
+    // CHECK: nvvm.shfl.sync bfly
+    %result = gpu.subgroup_reduce add %arg0 uniform cluster(size = 8) :
+        (i32) -> (i32)
+    memref.store %result, %buf[] : memref<i32>
+    gpu.return
+  }
+}
+
 gpu.module @test_module_31 {
   // CHECK: llvm.func @__nv_fmodf(f32, f32) -> f32
   // CHECK: llvm.func @__nv_fmod(f64, f64) -> f64

``````````

</details>


https://github.com/llvm/llvm-project/pull/209098

From llvmlistbot at llvm.org  Mon Jul 13 00:29:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 00:29:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] QuantileType relax quantileType conditions
 and inheritance issue (PR #204793)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204793@github.com>
Message-ID: <6a5493e5.d76d81af.39f9be.908a@mx.google.com>


vsimion26 wrote:

@javedabsar1 gentle ping on this PR. Please take a look when you have time 

https://github.com/llvm/llvm-project/pull/204793

From llvmlistbot at llvm.org  Mon Jul 13 01:19:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:19:15 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a549f83.d681cab9.333b2b.562f@mx.google.com>


https://github.com/ShashwathiNavada updated https://github.com/llvm/llvm-project/pull/206642

>From d6cdc9ec1ffc9ad1b70b2c8cbfb0430548c3b724 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Thu, 25 Jun 2026 01:40:11 -0500
Subject: [PATCH 1/2] Implement support for partial in unroll construct

---
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    |  9 ++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 24 +++++++--
 flang/test/Lower/OpenMP/unroll-partial01.f90  | 54 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 35 ++++++++++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 50 +++++++++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 27 ++++++++++
 7 files changed, 195 insertions(+), 5 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/unroll-partial01.f90

diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index e2a2c8575ffc8..9031f6ddf0744 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -795,6 +795,15 @@ bool ClauseProcessor::processOrdered(
   return false;
 }
 
+bool ClauseProcessor::processPartial(std::optional<int64_t> &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Partial>()) {
+    if (clause->v.has_value())
+      result = evaluate::ToInt64(*clause->v);
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processPriority(
     lower::StatementContext &stmtCtx,
     mlir::omp::PriorityClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1fc221b721ebf..e92db70ca707b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -111,6 +111,7 @@ class ClauseProcessor {
                          mlir::omp::NumThreadsClauseOps &result) const;
   bool processOrder(mlir::omp::OrderClauseOps &result) const;
   bool processOrdered(mlir::omp::OrderedClauseOps &result) const;
+  bool processPartial(std::optional<int64_t> &result) const;
   bool processPriority(lower::StatementContext &stmtCtx,
                        mlir::omp::PriorityClauseOps &result) const;
   bool processProcBind(mlir::omp::ProcBindClauseOps &result) const;
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 266b06f353675..c81e4a59de197 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2539,10 +2539,17 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
                         ConstructQueue::const_iterator item) {
   fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
 
-  // Clauses for unrolling not yet implemnted
   ClauseProcessor cp(converter, semaCtx, item->clauses);
-  cp.processTODO<clause::Partial, clause::Full>(
-      loc, llvm::omp::Directive::OMPD_unroll);
+
+  // The `full` clause is not yet implemented.
+  cp.processTODO<clause::Full>(loc, llvm::omp::Directive::OMPD_unroll);
+
+  // Process the `partial` clause. If present, it may carry a constant unroll
+  // factor.
+  std::optional<int64_t> partialFactor;
+  bool hasPartial = cp.processPartial(partialFactor);
+  if (hasPartial && !partialFactor.has_value())
+    TODO(loc, "PARTIAL clause on UNROLL without a constant factor");
 
   // Emit the associated loop
   llvm::SmallVector<mlir::omp::CanonicalLoopOp, 1> canonLoops;
@@ -2554,9 +2561,16 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
   for (auto &&canonLoop : canonLoops)
     applyees.push_back(canonLoop.getCli());
 
-  // Apply unrolling to it
   auto cli = llvm::getSingleElement(canonLoops).getCli();
-  mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+
+  if (partialFactor.has_value()) {
+    // Partially unroll the loop by the given constant factor.
+    mlir::omp::UnrollPartialOp::create(firOpBuilder, loc, cli,
+                                       static_cast<uint64_t>(*partialFactor));
+  } else {
+    // Apply heuristic unrolling to it.
+    mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+  }
 }
 
 static mlir::omp::MaskedOp
diff --git a/flang/test/Lower/OpenMP/unroll-partial01.f90 b/flang/test/Lower/OpenMP/unroll-partial01.f90
new file mode 100644
index 0000000000000..fe76d8f077753
--- /dev/null
+++ b/flang/test/Lower/OpenMP/unroll-partial01.f90
@@ -0,0 +1,54 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+
+subroutine omp_unroll_partial01(lb, ub, inc)
+  integer res, i, lb, ub, inc
+
+  !$omp unroll partial(4)
+  do i = lb, ub, inc
+    res = i
+  end do
+  !$omp end unroll
+
+end subroutine omp_unroll_partial01
+
+
+! CHECK-LABEL:   func.func @_QPomp_unroll_partial01(
+! CHECK-SAME:      %[[ARG0:.*]]: !fir.ref<i32> {fir.bindc_name = "lb"},
+! CHECK-SAME:      %[[ARG1:.*]]: !fir.ref<i32> {fir.bindc_name = "ub"},
+! CHECK-SAME:      %[[ARG2:.*]]: !fir.ref<i32> {fir.bindc_name = "inc"}) {
+! CHECK:           %[[VAL_0:.*]] = fir.dummy_scope : !fir.dscope
+! CHECK:           %[[VAL_1:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFomp_unroll_partial01Ei"}
+! CHECK:           %[[VAL_2:.*]]:2 = hlfir.declare %[[VAL_1]] {uniq_name = "_QFomp_unroll_partial01Ei"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_3:.*]]:2 = hlfir.declare %[[ARG2]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Einc"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_4:.*]]:2 = hlfir.declare %[[ARG0]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Elb"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_5:.*]] = fir.alloca i32 {bindc_name = "res", uniq_name = "_QFomp_unroll_partial01Eres"}
+! CHECK:           %[[VAL_6:.*]]:2 = hlfir.declare %[[VAL_5]] {uniq_name = "_QFomp_unroll_partial01Eres"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_7:.*]]:2 = hlfir.declare %[[ARG1]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Eub"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_8:.*]] = fir.load %[[VAL_4]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_9:.*]] = fir.load %[[VAL_7]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_10:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+! CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+! CHECK:           %[[VAL_13:.*]] = arith.cmpi slt, %[[VAL_10]], %[[VAL_11]] : i32
+! CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_15:.*]] = arith.select %[[VAL_13]], %[[VAL_14]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_16:.*]] = arith.select %[[VAL_13]], %[[VAL_9]], %[[VAL_8]] : i32
+! CHECK:           %[[VAL_17:.*]] = arith.select %[[VAL_13]], %[[VAL_8]], %[[VAL_9]] : i32
+! CHECK:           %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_16]] overflow<nuw> : i32
+! CHECK:           %[[VAL_19:.*]] = arith.divui %[[VAL_18]], %[[VAL_15]] : i32
+! CHECK:           %[[VAL_20:.*]] = arith.addi %[[VAL_19]], %[[VAL_12]] overflow<nuw> : i32
+! CHECK:           %[[VAL_21:.*]] = arith.cmpi slt, %[[VAL_17]], %[[VAL_16]] : i32
+! CHECK:           %[[VAL_22:.*]] = arith.select %[[VAL_21]], %[[VAL_11]], %[[VAL_20]] : i32
+! CHECK:           %[[VAL_23:.*]] = omp.new_cli
+! CHECK:           omp.canonical_loop(%[[VAL_23]]) %[[VAL_24:.*]] : i32 in range(%[[VAL_22]]) {
+! CHECK:             %[[VAL_25:.*]] = arith.muli %[[VAL_24]], %[[VAL_10]] : i32
+! CHECK:             %[[VAL_26:.*]] = arith.addi %[[VAL_8]], %[[VAL_25]] : i32
+! CHECK:             hlfir.assign %[[VAL_26]] to %[[VAL_2]]#0 : i32, !fir.ref<i32>
+! CHECK:             %[[VAL_27:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:             hlfir.assign %[[VAL_27]] to %[[VAL_6]]#0 : i32, !fir.ref<i32>
+! CHECK:             omp.terminator
+! CHECK:           }
+! CHECK:           omp.unroll_partial(%[[VAL_23]]) {unroll_factor = 4 : i64}
+! CHECK:           return
+! CHECK:         }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..b5e4cefa1ce72 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -556,6 +556,41 @@ def UnrollHeuristicOp : OpenMPTransform_Op<"unroll_heuristic", []> {
   let hasCustomAssemblyFormat = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMP unroll_partial operation
+//===----------------------------------------------------------------------===//
+
+def UnrollPartialOp : OpenMPTransform_Op<"unroll_partial", []> {
+  let summary = "OpenMP partial unroll operation";
+  let description = [{
+    Represents a `#pragma omp unroll partial` construct introduced in OpenMP
+    5.1.
+
+    The operation has one applyee and no generatees. The applyee is partially
+    unrolled by the factor given in the `unroll_factor` attribute, which must be
+    a positive constant. That is, the operation replaces `unroll_factor`
+    consecutive logical iterations of the applyee by a single iteration that
+    executes the original body `unroll_factor` times.
+
+    The resulting unrolled loop is currently not accessible as a generatee, so
+    no further loop-associated transformations can be chained onto it.
+
+    The lowering is implemented using `OpenMPIRBuilder::unrollLoopPartial`.
+
+    Assembly formats:
+      omp.unroll_partial(%cli) {unroll_factor = 4 : i64}
+  }];
+
+  let arguments = (ins CanonicalLoopInfoType:$applyee,
+                       ConfinedAttr<I64Attr, [IntPositive]>:$unroll_factor);
+
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$cli, "uint64_t":$unrollFactor)>,
+  ];
+
+  let hasCustomAssemblyFormat = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // OpenMP tile operation
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..27296190d6ac3 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4193,6 +4193,56 @@ UnrollHeuristicOp::getGenerateesODSOperandIndexAndLength() {
   return {0, 0};
 }
 
+//===----------------------------------------------------------------------===//
+// UnrollPartialOp
+//===----------------------------------------------------------------------===//
+
+void UnrollPartialOp::build(::mlir::OpBuilder &odsBuilder,
+                            ::mlir::OperationState &odsState, ::mlir::Value cli,
+                            uint64_t unrollFactor) {
+  odsState.addOperands(cli);
+  Properties &props = odsState.getOrAddProperties<Properties>();
+  props.unroll_factor = odsBuilder.getI64IntegerAttr(unrollFactor);
+}
+
+void UnrollPartialOp::print(OpAsmPrinter &p) {
+  p << '(' << getApplyee() << ')';
+
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+mlir::ParseResult UnrollPartialOp::parse(::mlir::OpAsmParser &parser,
+                                         ::mlir::OperationState &result) {
+  auto cliType = CanonicalLoopInfoType::get(parser.getContext());
+
+  if (parser.parseLParen())
+    return failure();
+
+  OpAsmParser::UnresolvedOperand applyee;
+  if (parser.parseOperand(applyee) ||
+      parser.resolveOperand(applyee, cliType, result.operands))
+    return failure();
+
+  if (parser.parseRParen())
+    return failure();
+
+  // The unroll factor is carried by the `unroll_factor` attribute.
+  if (parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return mlir::success();
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getApplyeesODSOperandIndexAndLength() {
+  return getODSOperandIndexAndLength(odsIndex_applyee);
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getGenerateesODSOperandIndexAndLength() {
+  return {0, 0};
+}
+
 //===----------------------------------------------------------------------===//
 // TileOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f0511bb4be7dd..d15c7f01c3674 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4490,6 +4490,30 @@ applyUnrollHeuristic(omp::UnrollHeuristicOp op, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Apply a `#pragma omp unroll partial` / `!$omp unroll partial`
+/// transformation using the OpenMPIRBuilder.
+static LogicalResult
+applyUnrollPartial(omp::UnrollPartialOp op, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  Value applyee = op.getApplyee();
+  assert(applyee && "Loop to apply unrolling on required");
+
+  llvm::CanonicalLoopInfo *consBuilderCLI =
+      moduleTranslation.lookupOMPLoop(applyee);
+  llvm::OpenMPIRBuilder::LocationDescription loc(builder);
+
+  // No generatee is supported yet, so the unrolled loop's CanonicalLoopInfo is
+  // not requested and unrolling is deferred to LLVM's LoopUnroll pass.
+  int32_t factor = static_cast<int32_t>(op.getUnrollFactor());
+  ompBuilder->unrollLoopPartial(loc.DL, consBuilderCLI, factor,
+                                /*UnrolledCLI=*/nullptr);
+
+  moduleTranslation.invalidateOmpLoop(applyee);
+  return success();
+}
+
 /// Apply a `#pragma omp tile` / `!$omp tile` transformation using the
 /// OpenMPIRBuilder.
 static LogicalResult applyTile(omp::TileOp op, llvm::IRBuilderBase &builder,
@@ -8889,6 +8913,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             // the omp.canonical_loop.
             return applyUnrollHeuristic(op, builder, moduleTranslation);
           })
+          .Case([&](omp::UnrollPartialOp op) {
+            return applyUnrollPartial(op, builder, moduleTranslation);
+          })
           .Case([&](omp::TileOp op) {
             return applyTile(op, builder, moduleTranslation);
           })

>From cac9adbd34fa5b2e0dee49d6e9997fb0c7fa0332 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Mon, 6 Jul 2026 09:11:26 -0500
Subject: [PATCH 2/2] testcases

---
 .../Dialect/OpenMP/cli-unroll-partial.mlir    | 59 +++++++++++
 mlir/test/Dialect/OpenMP/invalid.mlir         | 16 +++
 .../LLVMIR/openmp-unroll-partial01.mlir       | 57 +++++++++++
 .../LLVMIR/openmp-unroll-partial02.mlir       | 97 +++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100644 mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir

diff --git a/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
new file mode 100644
index 0000000000000..5ac3a1557ba79
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
@@ -0,0 +1,59 @@
+// RUN: mlir-opt %s            | FileCheck %s --enable-var-scope
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --enable-var-scope
+
+
+// CHECK-LABEL: @omp_unroll_partial_raw(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_raw(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %canonloop = "omp.new_cli" () : () -> (!omp.cli)
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  "omp.canonical_loop" (%tc, %canonloop) ({
+    ^bb0(%iv: i32):
+      omp.terminator
+  }) : (i32, !omp.cli) -> ()
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  "omp.unroll_partial" (%canonloop) {unroll_factor = 4 : i64} : (!omp.cli) -> ()
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %[[CANONLOOP:.+]] = omp.new_cli
+  %canonloop = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%canonloop) %iv : i32 in range(%tc) {
+    omp.terminator
+  }
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_nested_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_nested_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %cli_outer = omp.new_cli
+  // CHECK-NEXT: %canonloop_d1 = omp.new_cli
+  %cli_inner = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%cli_outer) %iv_outer : i32 in range(%tc) {
+    // CHECK-NEXT: omp.canonical_loop(%canonloop_d1) %iv_d1 : i32 in range(%[[tc]]) {
+    omp.canonical_loop(%cli_inner) %iv_inner : i32 in range(%tc) {
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    // CHECK: omp.terminator
+    omp.terminator
+  }
+
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  omp.unroll_partial(%cli_outer) {unroll_factor = 4 : i64}
+  // CHECK-NEXT: omp.unroll_partial(%canonloop_d1) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%cli_inner) {unroll_factor = 2 : i64}
+  return
+}
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 7ff4755174053..bc14d791340cf 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4522,3 +4522,19 @@ func.func @free_shared_mem_invalid_alignment2(%n: i32, %ptr : !llvm.ptr) -> () {
   omp.free_shared_mem [%n x i64 : (i32) align(3)] %ptr : !llvm.ptr
   return
 }
+
+// -----
+// unroll_factor = 0 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_zero(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = 0 : i64}
+  return
+}
+
+// -----
+// unroll_factor = -1 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
+  return
+}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
new file mode 100644
index 0000000000000..b9938afea0c30
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
@@ -0,0 +1,57 @@
+// Test lowering of the omp.unroll_partial (single loop)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_trivial_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_trivial_loop(%ptr: !llvm.ptr, %tc: i32) -> () {
+  %literal_cli = omp.new_cli
+  omp.canonical_loop(%literal_cli) %iv : i32 in range(%tc) {
+    %val = llvm.mlir.constant(42.0 : f32) : f32
+    llvm.store %val, %ptr : f32, !llvm.ptr
+    omp.terminator
+  }
+  omp.unroll_partial(%literal_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
new file mode 100644
index 0000000000000..a62bc499255db
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
@@ -0,0 +1,97 @@
+// Test lowering of the omp.unroll_partial (nested loops)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_nested_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[outer_tc:.+]], i32 %[[inner_tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[outer_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   br label %omp_omp.loop.preheader1
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader1:
+// CHECK-NEXT:   br label %omp_omp.loop.header2
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header2:
+// CHECK-NEXT:   %omp_omp.loop.iv8 = phi i32 [ 0, %omp_omp.loop.preheader1 ], [ %omp_omp.loop.next10, %omp_omp.loop.inc5 ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond3
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond3:
+// CHECK-NEXT:   %omp_omp.loop.cmp9 = icmp ult i32 %omp_omp.loop.iv8, %[[inner_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp9, label %omp_omp.loop.body4, label %omp_omp.loop.exit6
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body4:
+// CHECK-NEXT:   br label %omp.loop.region12
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region12:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont11
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont11:
+// CHECK-NEXT:   br label %omp_omp.loop.inc5
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc5:
+// CHECK-NEXT:   %omp_omp.loop.next10 = add nuw i32 %omp_omp.loop.iv8, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header2, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit6:
+// CHECK-NEXT:   br label %omp_omp.loop.after7
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after7:
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD4:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_nested_loop(%ptr: !llvm.ptr, %outer_tc: i32, %inner_tc: i32) -> () {
+  %outer_cli = omp.new_cli
+  %inner_cli = omp.new_cli
+  omp.canonical_loop(%outer_cli) %outer_iv : i32 in range(%outer_tc) {
+    omp.canonical_loop(%inner_cli) %inner_iv : i32 in range(%inner_tc) {
+      %val = llvm.mlir.constant(42.0 : f32) : f32
+      llvm.store %val, %ptr : f32, !llvm.ptr
+      omp.terminator
+    }
+    omp.terminator
+  }
+  omp.unroll_partial(%outer_cli) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%inner_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// The inner loop back-edge (header2) is encountered first in the IR dump, so
+// its metadata node gets a lower ID than the outer loop's metadata node.
+// Both loops share the same llvm.loop.unroll.enable node (uniqued by content).
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
+// CHECK: ![[$MD4]] = distinct !{![[$MD4]], ![[$MD2]], ![[$MD5:[0-9]+]]}
+// CHECK: ![[$MD5]] = !{!"llvm.loop.unroll.count", i32 2}


From llvmlistbot at llvm.org  Mon Jul 13 01:28:26 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Mon, 13 Jul 2026 01:28:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
Message-ID: <llvm.org/llvm/llvm-project/pull/209106@github.com>


https://github.com/KrxGu created https://github.com/llvm/llvm-project/pull/209106

The static memory planner currently skips any allocation that doesn't have a direct `memref.dealloc` user. This is overly conservative, after running `ownership-based-buffer-deallocation`, it's common to see patterns like:

  `%2 = arith.select %c, %0, %1 : memref<1024xf32>`
  `memref.dealloc %2 : memref<1024xf32>`

where both `%0` and `%1` get skipped with `++numSkipNoDealloc` even though their lifetimes are well-defined.

This patch teaches `collectCandidates` to follow `arith.select` chains when looking for potential deallocs. We traverse the use-def graph forward from each alloc, collecting any `memref.dealloc` ops reachable through select results.

Since a single select-based dealloc can conditionally free one of several allocs, we enforce a group constraint: all allocs that share a dealloc via a select must either all go into the arena or all be skipped. Without this, we could end up with a `memref.view` (an arena slice) and a raw alloc being fed into the same select, making the resulting dealloc invalid.

The group constraint is computed with a simple fixpoint iteration , if any member of a group is ineligible, the whole group is dropped.

The lifetime indices (`timeStart`/`timeEnd`) in `buildAllocInfos` are also fixed: the old code did one block scan per candidate (O(n×m)). This replaces it with a single pass upfront using a `DenseMap`, and sets `timeEnd` conservatively to the latest dealloc index across all potential deallocs for an alloc.

Tests added for:
- Single alloc freed via a self-select dealloc
- Two allocs sharing one select-based dealloc (group constraint active)
- The two-select two-dealloc pattern from the design discussion

>From 9f168b6147acb33fa0f6df1e23617ac42a82f251 Mon Sep 17 00:00:00 2001
From: KrxGu <krishom70 at gmail.com>
Date: Mon, 13 Jul 2026 13:40:50 +0530
Subject: [PATCH] [mlir][bufferization] Handle arith.select-based deallocs in
 static memory planner
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Allocs freed indirectly via arith.select chains were previously skipped.
This adds forward select-chain traversal so patterns like:
  %2 = arith.select %c, %0, %1
  memref.dealloc %2
are now handled correctly.

A group constraint ensures that all allocs sharing a select-based
dealloc are either all placed in the arena or all skipped — putting
one alloc in while leaving its peer out would break the dealloc.

Also fixes the O(n*m) block scan in buildAllocInfos by doing a single
upfront pass with a DenseMap index.

Tests added for single-alloc select, shared select-dealloc, and the
two-select two-dealloc pattern.
---
 .../StaticMemoryPlannerAnalysis.cpp           | 151 +++++++++++++-----
 .../static-memory-planner-analysis.mlir       |  61 +++++++
 2 files changed, 174 insertions(+), 38 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index c2ac40a8427e7..feb7594ac43c9 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/Support/Debug.h"
 #include <numeric>
 
@@ -34,10 +35,12 @@ using namespace mlir;
 
 namespace {
 
-/// A candidate allocation with its matching deallocation and assigned offset.
+/// A candidate allocation with its matching deallocation(s) and assigned
+/// offset. An alloc may be freed indirectly through arith.select chains,
+/// yielding multiple potential deallocs — all must be in the same block.
 struct AllocationCandidate {
   memref::AllocOp alloc;
-  memref::DeallocOp dealloc;
+  SmallVector<memref::DeallocOp> deallocs;
   int64_t offset = 0; // Offset in bytes from arena start (assigned by planner)
   int64_t sizeInBytes = 0; // Size in bytes
   int64_t alignment = 1;   // Required alignment in bytes
@@ -47,18 +50,25 @@ struct AllocationCandidate {
 // Helper utilities
 //===----------------------------------------------------------------------===//
 
-/// Finds the unique dealloc operation for a given alloc value.
-/// Returns nullptr if there are zero or multiple deallocs.
-static memref::DeallocOp findUniqueDealloc(Value allocValue) {
-  memref::DeallocOp deallocOp = nullptr;
-  for (Operation *user : allocValue.getUsers()) {
+/// Collect all dealloc ops that might free the given value, following
+/// arith.select chains. For example:
+///   %0 = memref.alloc()
+///   %2 = arith.select %c, %0, %1
+///   memref.dealloc %2    <- this covers %0 conditionally
+/// `visited` prevents cycles in the use-def graph.
+static void findPotentialDeallocs(Value value,
+                                  SmallVectorImpl<memref::DeallocOp> &deallocs,
+                                  SmallPtrSetImpl<Value> &visited) {
+  if (!visited.insert(value).second)
+    return;
+  for (Operation *user : value.getUsers()) {
     if (auto dealloc = dyn_cast<memref::DeallocOp>(user)) {
-      if (deallocOp)
-        return nullptr; // Multiple deallocs found
-      deallocOp = dealloc;
+      deallocs.push_back(dealloc);
+    } else if (auto select = dyn_cast<arith::SelectOp>(user)) {
+      if (isa<MemRefType>(select.getType()))
+        findPotentialDeallocs(select.getResult(), deallocs, visited);
     }
   }
-  return deallocOp;
 }
 
 /// Compute the size in bytes for a memref type.
@@ -70,25 +80,31 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
 
 /// Build lifetime-annotated allocation descriptors from candidates.
 /// Returns the arena alignment (LCM of all individual alignments).
+/// Uses a single block scan (O(n+m)) instead of one scan per candidate.
 static int64_t buildAllocInfos(
     MutableArrayRef<AllocationCandidate> candidates,
     SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  // Build an op-index map with a single pass over the block.
+  DenseMap<Operation *, int64_t> opIndex;
+  if (!candidates.empty()) {
+    Block *block = candidates.front().alloc->getBlock();
+    int64_t idx = 0;
+    for (Operation &op : *block)
+      opIndex[&op] = idx++;
+  }
+
   int64_t arenaAlignment = 1;
   for (auto &candidate : candidates) {
     bufferization::MemoryPlannerAlloc info;
     info.sizeInBytes = candidate.sizeInBytes;
     info.alignment = candidate.alignment;
-
-    Block *block = candidate.alloc->getBlock();
-    int64_t opIdx = 0;
-    for (Operation &op : *block) {
-      if (&op == candidate.alloc.getOperation())
-        info.timeStart = opIdx;
-      if (&op == candidate.dealloc.getOperation())
-        info.timeEnd = opIdx;
-      ++opIdx;
-    }
-
+    info.timeStart = opIndex.lookup(candidate.alloc.getOperation());
+    // Conservative: timeEnd = latest dealloc index among all potential
+    // deallocs.
+    int64_t timeEnd = 0;
+    for (memref::DeallocOp d : candidate.deallocs)
+      timeEnd = std::max(timeEnd, opIndex.lookup(d.getOperation()));
+    info.timeEnd = timeEnd;
     allocInfos.push_back(info);
     arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
   }
@@ -96,45 +112,90 @@ static int64_t buildAllocInfos(
 }
 
 /// Collect alloc/dealloc pairs eligible for arena placement.
-/// An allocation is eligible if it has a static shape and a unique dealloc
-/// in the same block.
+/// An allocation is eligible if it has a static shape and all of its
+/// potential deallocs (including those reached via arith.select chains)
+/// are in the same block. A group constraint ensures that all allocs
+/// sharing a select-based dealloc are either all eligible or all skipped.
 static SmallVector<AllocationCandidate>
 collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
                   llvm::Statistic &numSkipNoDealloc,
                   llvm::Statistic &numEligible) {
-  SmallVector<AllocationCandidate> candidates;
+  // Phase 1: walk allocs, find all potential deallocs via select chains.
+  SmallVector<AllocationCandidate> potentialCandidates;
+  // Track which allocs share each dealloc (for the group constraint).
+  DenseMap<Operation *, SmallVector<Value>> deallocToAllocs;
 
   funcOp->walk([&](memref::AllocOp allocOp) {
     MemRefType memrefType = allocOp.getType();
-
-    // Skip dynamic shapes
     if (!memrefType.hasStaticShape()) {
       ++numSkipDynamic;
       return;
     }
 
-    // Find unique dealloc in the same block
-    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-    if (!deallocOp) {
+    SmallVector<memref::DeallocOp> deallocs;
+    SmallPtrSet<Value, 8> visited;
+    findPotentialDeallocs(allocOp.getResult(), deallocs, visited);
+
+    if (deallocs.empty()) {
       ++numSkipNoDealloc;
       return;
     }
 
-    if (deallocOp->getBlock() != allocOp->getBlock()) {
+    // All deallocs must be in the same block as the alloc.
+    bool allSameBlock = llvm::all_of(deallocs, [&](memref::DeallocOp d) {
+      return d->getBlock() == allocOp->getBlock();
+    });
+    if (!allSameBlock) {
       ++numSkipNoDealloc;
       return;
     }
 
-    // This allocation is eligible
-    ++numEligible;
     AllocationCandidate candidate;
     candidate.alloc = allocOp;
-    candidate.dealloc = deallocOp;
+    candidate.deallocs = deallocs;
     candidate.sizeInBytes = computeSizeInBytes(memrefType);
     candidate.alignment = allocOp.getAlignment().value_or(1);
-    candidates.push_back(candidate);
+    potentialCandidates.push_back(candidate);
+
+    for (memref::DeallocOp d : deallocs)
+      deallocToAllocs[d.getOperation()].push_back(allocOp.getResult());
   });
 
+  // Phase 2: enforce group constraint — if a dealloc covers multiple allocs,
+  // all of them must be eligible or none are. Iterate to fixpoint.
+  SmallPtrSet<Value, 16> validAllocs;
+  for (auto &c : potentialCandidates)
+    validAllocs.insert(c.alloc.getResult());
+
+  bool changed = true;
+  while (changed) {
+    changed = false;
+    for (auto &c : potentialCandidates) {
+      if (!validAllocs.contains(c.alloc.getResult()))
+        continue;
+      for (memref::DeallocOp d : c.deallocs) {
+        for (Value peer : deallocToAllocs[d.getOperation()]) {
+          if (!validAllocs.contains(peer)) {
+            validAllocs.erase(c.alloc.getResult());
+            changed = true;
+            break;
+          }
+        }
+        if (!validAllocs.contains(c.alloc.getResult()))
+          break;
+      }
+    }
+  }
+
+  SmallVector<AllocationCandidate> candidates;
+  for (auto &c : potentialCandidates) {
+    if (validAllocs.contains(c.alloc.getResult())) {
+      ++numEligible;
+      candidates.push_back(c);
+    } else {
+      ++numSkipNoDealloc;
+    }
+  }
   return candidates;
 }
 
@@ -180,8 +241,14 @@ static FailureOr<Value> createArena(OpBuilder &builder,
 }
 
 /// Replace each alloc/dealloc pair with a memref.view into the arena.
+/// Handles select-chained deallocs: a single dealloc may cover multiple allocs,
+/// so we track erased deallocs to avoid double-erase.
 static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
                                Value arenaValue) {
+  SmallPtrSet<Operation *, 8> erasedDeallocs;
+  SmallVector<Operation *> allocsToErase;
+
+  // First replace all alloc results (rewires selects too), collect for erase.
   for (auto &candidate : candidates) {
     OpBuilder builder(candidate.alloc);
     Location loc = candidate.alloc.getLoc();
@@ -191,11 +258,19 @@ static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
         arith::ConstantIndexOp::create(builder, loc, candidate.offset);
     auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
                                        offsetIndex, SmallVector<Value>{});
-
     candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-    candidate.alloc.erase();
-    candidate.dealloc.erase();
+    allocsToErase.push_back(candidate.alloc.getOperation());
   }
+
+  // Erase deallocs first (they may reference alloc results via selects).
+  for (auto &candidate : candidates)
+    for (memref::DeallocOp d : candidate.deallocs)
+      if (erasedDeallocs.insert(d.getOperation()).second)
+        d.erase();
+
+  // Erase allocs last (no users remain after replaceAllUsesWith).
+  for (Operation *allocOp : allocsToErase)
+    allocOp->erase();
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
index a80c0e13adc21..14c8dbf8fd0fc 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
@@ -190,3 +190,64 @@ func.func @lcm_alignment() {
   memref.dealloc %alloc1 : memref<3xi32>
   return
 }
+
+// -----
+
+// Test 10: Single alloc freed via arith.select-based dealloc.
+// CHECK-LABEL: func @select_single_alloc
+func.func @select_single_alloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<4096xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %alloc = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %alloc, %alloc : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 11: Two allocs freed via a shared select-based dealloc.
+// Group constraint: both must be eligible together or neither is.
+// CHECK-LABEL: func @select_shared_dealloc
+func.func @select_shared_dealloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V0:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %[[V1:.*]] = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 12: Two allocs, two select-based deallocs (mentor's canonical example).
+// %a freed via dealloc(%sel1) or dealloc(%sel2), %b likewise.
+// CHECK-LABEL: func @select_two_deallocs
+func.func @select_two_deallocs() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel1 = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel1 : memref<1024xf32>
+  %sel2 = arith.select %c, %b, %a : memref<1024xf32>
+  memref.dealloc %sel2 : memref<1024xf32>
+  return
+}


From llvmlistbot at llvm.org  Mon Jul 13 01:29:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:29:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a54a1d0.9edb5c9d.208698.6e73@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-bufferization

@llvm/pr-subscribers-mlir

Author: Krish Gupta (KrxGu)

<details>
<summary>Changes</summary>

The static memory planner currently skips any allocation that doesn't have a direct `memref.dealloc` user. This is overly conservative, after running `ownership-based-buffer-deallocation`, it's common to see patterns like:

  `%2 = arith.select %c, %0, %1 : memref&lt;1024xf32&gt;`
  `memref.dealloc %2 : memref&lt;1024xf32&gt;`

where both `%0` and `%1` get skipped with `++numSkipNoDealloc` even though their lifetimes are well-defined.

This patch teaches `collectCandidates` to follow `arith.select` chains when looking for potential deallocs. We traverse the use-def graph forward from each alloc, collecting any `memref.dealloc` ops reachable through select results.

Since a single select-based dealloc can conditionally free one of several allocs, we enforce a group constraint: all allocs that share a dealloc via a select must either all go into the arena or all be skipped. Without this, we could end up with a `memref.view` (an arena slice) and a raw alloc being fed into the same select, making the resulting dealloc invalid.

The group constraint is computed with a simple fixpoint iteration , if any member of a group is ineligible, the whole group is dropped.

The lifetime indices (`timeStart`/`timeEnd`) in `buildAllocInfos` are also fixed: the old code did one block scan per candidate (O(n×m)). This replaces it with a single pass upfront using a `DenseMap`, and sets `timeEnd` conservatively to the latest dealloc index across all potential deallocs for an alloc.

Tests added for:
- Single alloc freed via a self-select dealloc
- Two allocs sharing one select-based dealloc (group constraint active)
- The two-select two-dealloc pattern from the design discussion

---
Full diff: https://github.com/llvm/llvm-project/pull/209106.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp (+113-38) 
- (modified) mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir (+61) 


``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index c2ac40a8427e7..feb7594ac43c9 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/Support/Debug.h"
 #include <numeric>
 
@@ -34,10 +35,12 @@ using namespace mlir;
 
 namespace {
 
-/// A candidate allocation with its matching deallocation and assigned offset.
+/// A candidate allocation with its matching deallocation(s) and assigned
+/// offset. An alloc may be freed indirectly through arith.select chains,
+/// yielding multiple potential deallocs — all must be in the same block.
 struct AllocationCandidate {
   memref::AllocOp alloc;
-  memref::DeallocOp dealloc;
+  SmallVector<memref::DeallocOp> deallocs;
   int64_t offset = 0; // Offset in bytes from arena start (assigned by planner)
   int64_t sizeInBytes = 0; // Size in bytes
   int64_t alignment = 1;   // Required alignment in bytes
@@ -47,18 +50,25 @@ struct AllocationCandidate {
 // Helper utilities
 //===----------------------------------------------------------------------===//
 
-/// Finds the unique dealloc operation for a given alloc value.
-/// Returns nullptr if there are zero or multiple deallocs.
-static memref::DeallocOp findUniqueDealloc(Value allocValue) {
-  memref::DeallocOp deallocOp = nullptr;
-  for (Operation *user : allocValue.getUsers()) {
+/// Collect all dealloc ops that might free the given value, following
+/// arith.select chains. For example:
+///   %0 = memref.alloc()
+///   %2 = arith.select %c, %0, %1
+///   memref.dealloc %2    <- this covers %0 conditionally
+/// `visited` prevents cycles in the use-def graph.
+static void findPotentialDeallocs(Value value,
+                                  SmallVectorImpl<memref::DeallocOp> &deallocs,
+                                  SmallPtrSetImpl<Value> &visited) {
+  if (!visited.insert(value).second)
+    return;
+  for (Operation *user : value.getUsers()) {
     if (auto dealloc = dyn_cast<memref::DeallocOp>(user)) {
-      if (deallocOp)
-        return nullptr; // Multiple deallocs found
-      deallocOp = dealloc;
+      deallocs.push_back(dealloc);
+    } else if (auto select = dyn_cast<arith::SelectOp>(user)) {
+      if (isa<MemRefType>(select.getType()))
+        findPotentialDeallocs(select.getResult(), deallocs, visited);
     }
   }
-  return deallocOp;
 }
 
 /// Compute the size in bytes for a memref type.
@@ -70,25 +80,31 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
 
 /// Build lifetime-annotated allocation descriptors from candidates.
 /// Returns the arena alignment (LCM of all individual alignments).
+/// Uses a single block scan (O(n+m)) instead of one scan per candidate.
 static int64_t buildAllocInfos(
     MutableArrayRef<AllocationCandidate> candidates,
     SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  // Build an op-index map with a single pass over the block.
+  DenseMap<Operation *, int64_t> opIndex;
+  if (!candidates.empty()) {
+    Block *block = candidates.front().alloc->getBlock();
+    int64_t idx = 0;
+    for (Operation &op : *block)
+      opIndex[&op] = idx++;
+  }
+
   int64_t arenaAlignment = 1;
   for (auto &candidate : candidates) {
     bufferization::MemoryPlannerAlloc info;
     info.sizeInBytes = candidate.sizeInBytes;
     info.alignment = candidate.alignment;
-
-    Block *block = candidate.alloc->getBlock();
-    int64_t opIdx = 0;
-    for (Operation &op : *block) {
-      if (&op == candidate.alloc.getOperation())
-        info.timeStart = opIdx;
-      if (&op == candidate.dealloc.getOperation())
-        info.timeEnd = opIdx;
-      ++opIdx;
-    }
-
+    info.timeStart = opIndex.lookup(candidate.alloc.getOperation());
+    // Conservative: timeEnd = latest dealloc index among all potential
+    // deallocs.
+    int64_t timeEnd = 0;
+    for (memref::DeallocOp d : candidate.deallocs)
+      timeEnd = std::max(timeEnd, opIndex.lookup(d.getOperation()));
+    info.timeEnd = timeEnd;
     allocInfos.push_back(info);
     arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
   }
@@ -96,45 +112,90 @@ static int64_t buildAllocInfos(
 }
 
 /// Collect alloc/dealloc pairs eligible for arena placement.
-/// An allocation is eligible if it has a static shape and a unique dealloc
-/// in the same block.
+/// An allocation is eligible if it has a static shape and all of its
+/// potential deallocs (including those reached via arith.select chains)
+/// are in the same block. A group constraint ensures that all allocs
+/// sharing a select-based dealloc are either all eligible or all skipped.
 static SmallVector<AllocationCandidate>
 collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
                   llvm::Statistic &numSkipNoDealloc,
                   llvm::Statistic &numEligible) {
-  SmallVector<AllocationCandidate> candidates;
+  // Phase 1: walk allocs, find all potential deallocs via select chains.
+  SmallVector<AllocationCandidate> potentialCandidates;
+  // Track which allocs share each dealloc (for the group constraint).
+  DenseMap<Operation *, SmallVector<Value>> deallocToAllocs;
 
   funcOp->walk([&](memref::AllocOp allocOp) {
     MemRefType memrefType = allocOp.getType();
-
-    // Skip dynamic shapes
     if (!memrefType.hasStaticShape()) {
       ++numSkipDynamic;
       return;
     }
 
-    // Find unique dealloc in the same block
-    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-    if (!deallocOp) {
+    SmallVector<memref::DeallocOp> deallocs;
+    SmallPtrSet<Value, 8> visited;
+    findPotentialDeallocs(allocOp.getResult(), deallocs, visited);
+
+    if (deallocs.empty()) {
       ++numSkipNoDealloc;
       return;
     }
 
-    if (deallocOp->getBlock() != allocOp->getBlock()) {
+    // All deallocs must be in the same block as the alloc.
+    bool allSameBlock = llvm::all_of(deallocs, [&](memref::DeallocOp d) {
+      return d->getBlock() == allocOp->getBlock();
+    });
+    if (!allSameBlock) {
       ++numSkipNoDealloc;
       return;
     }
 
-    // This allocation is eligible
-    ++numEligible;
     AllocationCandidate candidate;
     candidate.alloc = allocOp;
-    candidate.dealloc = deallocOp;
+    candidate.deallocs = deallocs;
     candidate.sizeInBytes = computeSizeInBytes(memrefType);
     candidate.alignment = allocOp.getAlignment().value_or(1);
-    candidates.push_back(candidate);
+    potentialCandidates.push_back(candidate);
+
+    for (memref::DeallocOp d : deallocs)
+      deallocToAllocs[d.getOperation()].push_back(allocOp.getResult());
   });
 
+  // Phase 2: enforce group constraint — if a dealloc covers multiple allocs,
+  // all of them must be eligible or none are. Iterate to fixpoint.
+  SmallPtrSet<Value, 16> validAllocs;
+  for (auto &c : potentialCandidates)
+    validAllocs.insert(c.alloc.getResult());
+
+  bool changed = true;
+  while (changed) {
+    changed = false;
+    for (auto &c : potentialCandidates) {
+      if (!validAllocs.contains(c.alloc.getResult()))
+        continue;
+      for (memref::DeallocOp d : c.deallocs) {
+        for (Value peer : deallocToAllocs[d.getOperation()]) {
+          if (!validAllocs.contains(peer)) {
+            validAllocs.erase(c.alloc.getResult());
+            changed = true;
+            break;
+          }
+        }
+        if (!validAllocs.contains(c.alloc.getResult()))
+          break;
+      }
+    }
+  }
+
+  SmallVector<AllocationCandidate> candidates;
+  for (auto &c : potentialCandidates) {
+    if (validAllocs.contains(c.alloc.getResult())) {
+      ++numEligible;
+      candidates.push_back(c);
+    } else {
+      ++numSkipNoDealloc;
+    }
+  }
   return candidates;
 }
 
@@ -180,8 +241,14 @@ static FailureOr<Value> createArena(OpBuilder &builder,
 }
 
 /// Replace each alloc/dealloc pair with a memref.view into the arena.
+/// Handles select-chained deallocs: a single dealloc may cover multiple allocs,
+/// so we track erased deallocs to avoid double-erase.
 static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
                                Value arenaValue) {
+  SmallPtrSet<Operation *, 8> erasedDeallocs;
+  SmallVector<Operation *> allocsToErase;
+
+  // First replace all alloc results (rewires selects too), collect for erase.
   for (auto &candidate : candidates) {
     OpBuilder builder(candidate.alloc);
     Location loc = candidate.alloc.getLoc();
@@ -191,11 +258,19 @@ static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
         arith::ConstantIndexOp::create(builder, loc, candidate.offset);
     auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
                                        offsetIndex, SmallVector<Value>{});
-
     candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-    candidate.alloc.erase();
-    candidate.dealloc.erase();
+    allocsToErase.push_back(candidate.alloc.getOperation());
   }
+
+  // Erase deallocs first (they may reference alloc results via selects).
+  for (auto &candidate : candidates)
+    for (memref::DeallocOp d : candidate.deallocs)
+      if (erasedDeallocs.insert(d.getOperation()).second)
+        d.erase();
+
+  // Erase allocs last (no users remain after replaceAllUsesWith).
+  for (Operation *allocOp : allocsToErase)
+    allocOp->erase();
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
index a80c0e13adc21..14c8dbf8fd0fc 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
@@ -190,3 +190,64 @@ func.func @lcm_alignment() {
   memref.dealloc %alloc1 : memref<3xi32>
   return
 }
+
+// -----
+
+// Test 10: Single alloc freed via arith.select-based dealloc.
+// CHECK-LABEL: func @select_single_alloc
+func.func @select_single_alloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<4096xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %alloc = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %alloc, %alloc : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 11: Two allocs freed via a shared select-based dealloc.
+// Group constraint: both must be eligible together or neither is.
+// CHECK-LABEL: func @select_shared_dealloc
+func.func @select_shared_dealloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V0:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %[[V1:.*]] = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 12: Two allocs, two select-based deallocs (mentor's canonical example).
+// %a freed via dealloc(%sel1) or dealloc(%sel2), %b likewise.
+// CHECK-LABEL: func @select_two_deallocs
+func.func @select_two_deallocs() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel1 = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel1 : memref<1024xf32>
+  %sel2 = arith.select %c, %b, %a : memref<1024xf32>
+  memref.dealloc %sel2 : memref<1024xf32>
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Mon Jul 13 01:33:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:33:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1651159 - [MLIR] Install missing standalone
	helper tools (#205066)
Message-ID: <6a54a2e6.d1496011.26913c.be55@mx.google.com>


Author: Momchil Velikov
Date: 2026-07-13T09:33:38+01:00
New Revision: 1651159f32cfcf82ecbb364dbf879e97b53d8060

URL: https://github.com/llvm/llvm-project/commit/1651159f32cfcf82ecbb364dbf879e97b53d8060
DIFF: https://github.com/llvm/llvm-project/commit/1651159f32cfcf82ecbb364dbf879e97b53d8060.diff

LOG: [MLIR] Install missing standalone helper tools (#205066)

Standalone MLIR installs CMake package files that can reference helper
executables needed by downstream builds. Most helper tools already have
install paths available through existing LLVM tool and utility install
options, but `mlir-src-sharder` does not pass an install destination to
`add_tablegen` and `mlir-irdl-to-cpp` does not define an install rule.

Install those two missing helpers.

Co-authored-by: Luca Fancellu <luca.fancellu at arm.com>

Added: 
    

Modified: 
    mlir/tools/mlir-irdl-to-cpp/CMakeLists.txt
    mlir/tools/mlir-src-sharder/CMakeLists.txt

Removed: 
    


################################################################################
diff  --git a/mlir/tools/mlir-irdl-to-cpp/CMakeLists.txt b/mlir/tools/mlir-irdl-to-cpp/CMakeLists.txt
index 27a7e22db7d7b..5b69b364c6b32 100644
--- a/mlir/tools/mlir-irdl-to-cpp/CMakeLists.txt
+++ b/mlir/tools/mlir-irdl-to-cpp/CMakeLists.txt
@@ -7,3 +7,14 @@ mlir_target_link_libraries(mlir-irdl-to-cpp
   )
 
 setup_host_tool(mlir-irdl-to-cpp MLIR_IRDL_TO_CPP MLIR_IRDL_TO_CPP_EXE MLIR_IRDL_TO_CPP_TARGET)
+
+if(NOT LLVM_INSTALL_TOOLCHAIN_ONLY)
+  install(TARGETS mlir-irdl-to-cpp
+          RUNTIME DESTINATION "${MLIR_TOOLS_INSTALL_DIR}"
+          COMPONENT mlir-irdl-to-cpp)
+  if(NOT LLVM_ENABLE_IDE)
+    add_llvm_install_targets(install-mlir-irdl-to-cpp
+                             DEPENDS mlir-irdl-to-cpp
+                             COMPONENT mlir-irdl-to-cpp)
+  endif()
+endif()

diff  --git a/mlir/tools/mlir-src-sharder/CMakeLists.txt b/mlir/tools/mlir-src-sharder/CMakeLists.txt
index bc4a11bd4c0cb..b5d90cfe238ed 100644
--- a/mlir/tools/mlir-src-sharder/CMakeLists.txt
+++ b/mlir/tools/mlir-src-sharder/CMakeLists.txt
@@ -2,6 +2,8 @@ set(LLVM_LINK_COMPONENTS Support)
 set(LIBS MLIRSupport)
 
 add_tablegen(mlir-src-sharder MLIR_SRC_SHARDER
+  DESTINATION "${MLIR_TOOLS_INSTALL_DIR}"
+  EXPORT MLIR
   mlir-src-sharder.cpp)
 
 set_target_properties(mlir-src-sharder PROPERTIES FOLDER "MLIR/Tablegenning")


        

From llvmlistbot at llvm.org  Mon Jul 13 01:33:45 2026
From: llvmlistbot at llvm.org (Momchil Velikov)
Date: Mon, 13 Jul 2026 01:33:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Install missing standalone helper
	tools (PR #205066)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205066@github.com>
Message-ID: <6a54a2e9.6c5720b3.15e90c.26e5@mx.google.com>


https://github.com/momchil-velikov closed https://github.com/llvm/llvm-project/pull/205066

From llvmlistbot at llvm.org  Mon Jul 13 01:35:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:35:06 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a54a33a.c5504292.28b3b8.3533@mx.google.com>


https://github.com/ShashwathiNavada updated https://github.com/llvm/llvm-project/pull/206642

>From d6cdc9ec1ffc9ad1b70b2c8cbfb0430548c3b724 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Thu, 25 Jun 2026 01:40:11 -0500
Subject: [PATCH 1/2] Implement support for partial in unroll construct

---
 flang/lib/Lower/OpenMP/ClauseProcessor.cpp    |  9 ++++
 flang/lib/Lower/OpenMP/ClauseProcessor.h      |  1 +
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 24 +++++++--
 flang/test/Lower/OpenMP/unroll-partial01.f90  | 54 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 35 ++++++++++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 50 +++++++++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 27 ++++++++++
 7 files changed, 195 insertions(+), 5 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/unroll-partial01.f90

diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index e2a2c8575ffc8..9031f6ddf0744 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -795,6 +795,15 @@ bool ClauseProcessor::processOrdered(
   return false;
 }
 
+bool ClauseProcessor::processPartial(std::optional<int64_t> &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Partial>()) {
+    if (clause->v.has_value())
+      result = evaluate::ToInt64(*clause->v);
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processPriority(
     lower::StatementContext &stmtCtx,
     mlir::omp::PriorityClauseOps &result) const {
diff --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1fc221b721ebf..e92db70ca707b 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -111,6 +111,7 @@ class ClauseProcessor {
                          mlir::omp::NumThreadsClauseOps &result) const;
   bool processOrder(mlir::omp::OrderClauseOps &result) const;
   bool processOrdered(mlir::omp::OrderedClauseOps &result) const;
+  bool processPartial(std::optional<int64_t> &result) const;
   bool processPriority(lower::StatementContext &stmtCtx,
                        mlir::omp::PriorityClauseOps &result) const;
   bool processProcBind(mlir::omp::ProcBindClauseOps &result) const;
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 266b06f353675..c81e4a59de197 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2539,10 +2539,17 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
                         ConstructQueue::const_iterator item) {
   fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
 
-  // Clauses for unrolling not yet implemnted
   ClauseProcessor cp(converter, semaCtx, item->clauses);
-  cp.processTODO<clause::Partial, clause::Full>(
-      loc, llvm::omp::Directive::OMPD_unroll);
+
+  // The `full` clause is not yet implemented.
+  cp.processTODO<clause::Full>(loc, llvm::omp::Directive::OMPD_unroll);
+
+  // Process the `partial` clause. If present, it may carry a constant unroll
+  // factor.
+  std::optional<int64_t> partialFactor;
+  bool hasPartial = cp.processPartial(partialFactor);
+  if (hasPartial && !partialFactor.has_value())
+    TODO(loc, "PARTIAL clause on UNROLL without a constant factor");
 
   // Emit the associated loop
   llvm::SmallVector<mlir::omp::CanonicalLoopOp, 1> canonLoops;
@@ -2554,9 +2561,16 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
   for (auto &&canonLoop : canonLoops)
     applyees.push_back(canonLoop.getCli());
 
-  // Apply unrolling to it
   auto cli = llvm::getSingleElement(canonLoops).getCli();
-  mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+
+  if (partialFactor.has_value()) {
+    // Partially unroll the loop by the given constant factor.
+    mlir::omp::UnrollPartialOp::create(firOpBuilder, loc, cli,
+                                       static_cast<uint64_t>(*partialFactor));
+  } else {
+    // Apply heuristic unrolling to it.
+    mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+  }
 }
 
 static mlir::omp::MaskedOp
diff --git a/flang/test/Lower/OpenMP/unroll-partial01.f90 b/flang/test/Lower/OpenMP/unroll-partial01.f90
new file mode 100644
index 0000000000000..fe76d8f077753
--- /dev/null
+++ b/flang/test/Lower/OpenMP/unroll-partial01.f90
@@ -0,0 +1,54 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+
+subroutine omp_unroll_partial01(lb, ub, inc)
+  integer res, i, lb, ub, inc
+
+  !$omp unroll partial(4)
+  do i = lb, ub, inc
+    res = i
+  end do
+  !$omp end unroll
+
+end subroutine omp_unroll_partial01
+
+
+! CHECK-LABEL:   func.func @_QPomp_unroll_partial01(
+! CHECK-SAME:      %[[ARG0:.*]]: !fir.ref<i32> {fir.bindc_name = "lb"},
+! CHECK-SAME:      %[[ARG1:.*]]: !fir.ref<i32> {fir.bindc_name = "ub"},
+! CHECK-SAME:      %[[ARG2:.*]]: !fir.ref<i32> {fir.bindc_name = "inc"}) {
+! CHECK:           %[[VAL_0:.*]] = fir.dummy_scope : !fir.dscope
+! CHECK:           %[[VAL_1:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFomp_unroll_partial01Ei"}
+! CHECK:           %[[VAL_2:.*]]:2 = hlfir.declare %[[VAL_1]] {uniq_name = "_QFomp_unroll_partial01Ei"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_3:.*]]:2 = hlfir.declare %[[ARG2]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Einc"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_4:.*]]:2 = hlfir.declare %[[ARG0]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Elb"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_5:.*]] = fir.alloca i32 {bindc_name = "res", uniq_name = "_QFomp_unroll_partial01Eres"}
+! CHECK:           %[[VAL_6:.*]]:2 = hlfir.declare %[[VAL_5]] {uniq_name = "_QFomp_unroll_partial01Eres"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_7:.*]]:2 = hlfir.declare %[[ARG1]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Eub"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_8:.*]] = fir.load %[[VAL_4]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_9:.*]] = fir.load %[[VAL_7]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_10:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+! CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+! CHECK:           %[[VAL_13:.*]] = arith.cmpi slt, %[[VAL_10]], %[[VAL_11]] : i32
+! CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_15:.*]] = arith.select %[[VAL_13]], %[[VAL_14]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_16:.*]] = arith.select %[[VAL_13]], %[[VAL_9]], %[[VAL_8]] : i32
+! CHECK:           %[[VAL_17:.*]] = arith.select %[[VAL_13]], %[[VAL_8]], %[[VAL_9]] : i32
+! CHECK:           %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_16]] overflow<nuw> : i32
+! CHECK:           %[[VAL_19:.*]] = arith.divui %[[VAL_18]], %[[VAL_15]] : i32
+! CHECK:           %[[VAL_20:.*]] = arith.addi %[[VAL_19]], %[[VAL_12]] overflow<nuw> : i32
+! CHECK:           %[[VAL_21:.*]] = arith.cmpi slt, %[[VAL_17]], %[[VAL_16]] : i32
+! CHECK:           %[[VAL_22:.*]] = arith.select %[[VAL_21]], %[[VAL_11]], %[[VAL_20]] : i32
+! CHECK:           %[[VAL_23:.*]] = omp.new_cli
+! CHECK:           omp.canonical_loop(%[[VAL_23]]) %[[VAL_24:.*]] : i32 in range(%[[VAL_22]]) {
+! CHECK:             %[[VAL_25:.*]] = arith.muli %[[VAL_24]], %[[VAL_10]] : i32
+! CHECK:             %[[VAL_26:.*]] = arith.addi %[[VAL_8]], %[[VAL_25]] : i32
+! CHECK:             hlfir.assign %[[VAL_26]] to %[[VAL_2]]#0 : i32, !fir.ref<i32>
+! CHECK:             %[[VAL_27:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:             hlfir.assign %[[VAL_27]] to %[[VAL_6]]#0 : i32, !fir.ref<i32>
+! CHECK:             omp.terminator
+! CHECK:           }
+! CHECK:           omp.unroll_partial(%[[VAL_23]]) {unroll_factor = 4 : i64}
+! CHECK:           return
+! CHECK:         }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..b5e4cefa1ce72 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -556,6 +556,41 @@ def UnrollHeuristicOp : OpenMPTransform_Op<"unroll_heuristic", []> {
   let hasCustomAssemblyFormat = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMP unroll_partial operation
+//===----------------------------------------------------------------------===//
+
+def UnrollPartialOp : OpenMPTransform_Op<"unroll_partial", []> {
+  let summary = "OpenMP partial unroll operation";
+  let description = [{
+    Represents a `#pragma omp unroll partial` construct introduced in OpenMP
+    5.1.
+
+    The operation has one applyee and no generatees. The applyee is partially
+    unrolled by the factor given in the `unroll_factor` attribute, which must be
+    a positive constant. That is, the operation replaces `unroll_factor`
+    consecutive logical iterations of the applyee by a single iteration that
+    executes the original body `unroll_factor` times.
+
+    The resulting unrolled loop is currently not accessible as a generatee, so
+    no further loop-associated transformations can be chained onto it.
+
+    The lowering is implemented using `OpenMPIRBuilder::unrollLoopPartial`.
+
+    Assembly formats:
+      omp.unroll_partial(%cli) {unroll_factor = 4 : i64}
+  }];
+
+  let arguments = (ins CanonicalLoopInfoType:$applyee,
+                       ConfinedAttr<I64Attr, [IntPositive]>:$unroll_factor);
+
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$cli, "uint64_t":$unrollFactor)>,
+  ];
+
+  let hasCustomAssemblyFormat = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // OpenMP tile operation
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..27296190d6ac3 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4193,6 +4193,56 @@ UnrollHeuristicOp::getGenerateesODSOperandIndexAndLength() {
   return {0, 0};
 }
 
+//===----------------------------------------------------------------------===//
+// UnrollPartialOp
+//===----------------------------------------------------------------------===//
+
+void UnrollPartialOp::build(::mlir::OpBuilder &odsBuilder,
+                            ::mlir::OperationState &odsState, ::mlir::Value cli,
+                            uint64_t unrollFactor) {
+  odsState.addOperands(cli);
+  Properties &props = odsState.getOrAddProperties<Properties>();
+  props.unroll_factor = odsBuilder.getI64IntegerAttr(unrollFactor);
+}
+
+void UnrollPartialOp::print(OpAsmPrinter &p) {
+  p << '(' << getApplyee() << ')';
+
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+mlir::ParseResult UnrollPartialOp::parse(::mlir::OpAsmParser &parser,
+                                         ::mlir::OperationState &result) {
+  auto cliType = CanonicalLoopInfoType::get(parser.getContext());
+
+  if (parser.parseLParen())
+    return failure();
+
+  OpAsmParser::UnresolvedOperand applyee;
+  if (parser.parseOperand(applyee) ||
+      parser.resolveOperand(applyee, cliType, result.operands))
+    return failure();
+
+  if (parser.parseRParen())
+    return failure();
+
+  // The unroll factor is carried by the `unroll_factor` attribute.
+  if (parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return mlir::success();
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getApplyeesODSOperandIndexAndLength() {
+  return getODSOperandIndexAndLength(odsIndex_applyee);
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getGenerateesODSOperandIndexAndLength() {
+  return {0, 0};
+}
+
 //===----------------------------------------------------------------------===//
 // TileOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f0511bb4be7dd..d15c7f01c3674 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4490,6 +4490,30 @@ applyUnrollHeuristic(omp::UnrollHeuristicOp op, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Apply a `#pragma omp unroll partial` / `!$omp unroll partial`
+/// transformation using the OpenMPIRBuilder.
+static LogicalResult
+applyUnrollPartial(omp::UnrollPartialOp op, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  Value applyee = op.getApplyee();
+  assert(applyee && "Loop to apply unrolling on required");
+
+  llvm::CanonicalLoopInfo *consBuilderCLI =
+      moduleTranslation.lookupOMPLoop(applyee);
+  llvm::OpenMPIRBuilder::LocationDescription loc(builder);
+
+  // No generatee is supported yet, so the unrolled loop's CanonicalLoopInfo is
+  // not requested and unrolling is deferred to LLVM's LoopUnroll pass.
+  int32_t factor = static_cast<int32_t>(op.getUnrollFactor());
+  ompBuilder->unrollLoopPartial(loc.DL, consBuilderCLI, factor,
+                                /*UnrolledCLI=*/nullptr);
+
+  moduleTranslation.invalidateOmpLoop(applyee);
+  return success();
+}
+
 /// Apply a `#pragma omp tile` / `!$omp tile` transformation using the
 /// OpenMPIRBuilder.
 static LogicalResult applyTile(omp::TileOp op, llvm::IRBuilderBase &builder,
@@ -8889,6 +8913,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             // the omp.canonical_loop.
             return applyUnrollHeuristic(op, builder, moduleTranslation);
           })
+          .Case([&](omp::UnrollPartialOp op) {
+            return applyUnrollPartial(op, builder, moduleTranslation);
+          })
           .Case([&](omp::TileOp op) {
             return applyTile(op, builder, moduleTranslation);
           })

>From cac9adbd34fa5b2e0dee49d6e9997fb0c7fa0332 Mon Sep 17 00:00:00 2001
From: ShashwathiNavada <shashwathinavada at gmail.com>
Date: Mon, 6 Jul 2026 09:11:26 -0500
Subject: [PATCH 2/2] testcases

---
 .../Dialect/OpenMP/cli-unroll-partial.mlir    | 59 +++++++++++
 mlir/test/Dialect/OpenMP/invalid.mlir         | 16 +++
 .../LLVMIR/openmp-unroll-partial01.mlir       | 57 +++++++++++
 .../LLVMIR/openmp-unroll-partial02.mlir       | 97 +++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100644 mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir

diff --git a/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
new file mode 100644
index 0000000000000..5ac3a1557ba79
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
@@ -0,0 +1,59 @@
+// RUN: mlir-opt %s            | FileCheck %s --enable-var-scope
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --enable-var-scope
+
+
+// CHECK-LABEL: @omp_unroll_partial_raw(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_raw(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %canonloop = "omp.new_cli" () : () -> (!omp.cli)
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  "omp.canonical_loop" (%tc, %canonloop) ({
+    ^bb0(%iv: i32):
+      omp.terminator
+  }) : (i32, !omp.cli) -> ()
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  "omp.unroll_partial" (%canonloop) {unroll_factor = 4 : i64} : (!omp.cli) -> ()
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %[[CANONLOOP:.+]] = omp.new_cli
+  %canonloop = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%canonloop) %iv : i32 in range(%tc) {
+    omp.terminator
+  }
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_nested_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_nested_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %cli_outer = omp.new_cli
+  // CHECK-NEXT: %canonloop_d1 = omp.new_cli
+  %cli_inner = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%cli_outer) %iv_outer : i32 in range(%tc) {
+    // CHECK-NEXT: omp.canonical_loop(%canonloop_d1) %iv_d1 : i32 in range(%[[tc]]) {
+    omp.canonical_loop(%cli_inner) %iv_inner : i32 in range(%tc) {
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    // CHECK: omp.terminator
+    omp.terminator
+  }
+
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  omp.unroll_partial(%cli_outer) {unroll_factor = 4 : i64}
+  // CHECK-NEXT: omp.unroll_partial(%canonloop_d1) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%cli_inner) {unroll_factor = 2 : i64}
+  return
+}
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 7ff4755174053..bc14d791340cf 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4522,3 +4522,19 @@ func.func @free_shared_mem_invalid_alignment2(%n: i32, %ptr : !llvm.ptr) -> () {
   omp.free_shared_mem [%n x i64 : (i32) align(3)] %ptr : !llvm.ptr
   return
 }
+
+// -----
+// unroll_factor = 0 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_zero(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = 0 : i64}
+  return
+}
+
+// -----
+// unroll_factor = -1 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
+  return
+}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
new file mode 100644
index 0000000000000..b9938afea0c30
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
@@ -0,0 +1,57 @@
+// Test lowering of the omp.unroll_partial (single loop)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_trivial_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_trivial_loop(%ptr: !llvm.ptr, %tc: i32) -> () {
+  %literal_cli = omp.new_cli
+  omp.canonical_loop(%literal_cli) %iv : i32 in range(%tc) {
+    %val = llvm.mlir.constant(42.0 : f32) : f32
+    llvm.store %val, %ptr : f32, !llvm.ptr
+    omp.terminator
+  }
+  omp.unroll_partial(%literal_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
diff --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
new file mode 100644
index 0000000000000..a62bc499255db
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
@@ -0,0 +1,97 @@
+// Test lowering of the omp.unroll_partial (nested loops)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_nested_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[outer_tc:.+]], i32 %[[inner_tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[outer_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   br label %omp_omp.loop.preheader1
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader1:
+// CHECK-NEXT:   br label %omp_omp.loop.header2
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header2:
+// CHECK-NEXT:   %omp_omp.loop.iv8 = phi i32 [ 0, %omp_omp.loop.preheader1 ], [ %omp_omp.loop.next10, %omp_omp.loop.inc5 ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond3
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond3:
+// CHECK-NEXT:   %omp_omp.loop.cmp9 = icmp ult i32 %omp_omp.loop.iv8, %[[inner_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp9, label %omp_omp.loop.body4, label %omp_omp.loop.exit6
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body4:
+// CHECK-NEXT:   br label %omp.loop.region12
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region12:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont11
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont11:
+// CHECK-NEXT:   br label %omp_omp.loop.inc5
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc5:
+// CHECK-NEXT:   %omp_omp.loop.next10 = add nuw i32 %omp_omp.loop.iv8, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header2, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit6:
+// CHECK-NEXT:   br label %omp_omp.loop.after7
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after7:
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD4:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_nested_loop(%ptr: !llvm.ptr, %outer_tc: i32, %inner_tc: i32) -> () {
+  %outer_cli = omp.new_cli
+  %inner_cli = omp.new_cli
+  omp.canonical_loop(%outer_cli) %outer_iv : i32 in range(%outer_tc) {
+    omp.canonical_loop(%inner_cli) %inner_iv : i32 in range(%inner_tc) {
+      %val = llvm.mlir.constant(42.0 : f32) : f32
+      llvm.store %val, %ptr : f32, !llvm.ptr
+      omp.terminator
+    }
+    omp.terminator
+  }
+  omp.unroll_partial(%outer_cli) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%inner_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// The inner loop back-edge (header2) is encountered first in the IR dump, so
+// its metadata node gets a lower ID than the outer loop's metadata node.
+// Both loops share the same llvm.loop.unroll.enable node (uniqued by content).
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
+// CHECK: ![[$MD4]] = distinct !{![[$MD4]], ![[$MD2]], ![[$MD5:[0-9]+]]}
+// CHECK: ![[$MD5]] = !{!"llvm.loop.unroll.count", i32 2}


From llvmlistbot at llvm.org  Mon Jul 13 01:39:03 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 13 Jul 2026 01:39:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54a427.510f576d.1b3cb9.8e83@mx.google.com>


https://github.com/IgWod edited https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Mon Jul 13 01:39:03 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 13 Jul 2026 01:39:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54a427.160324b0.195192.da0f@mx.google.com>


https://github.com/IgWod approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Mon Jul 13 01:39:04 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 13 Jul 2026 01:39:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54a428.f1c1923d.991d1.bb22@mx.google.com>


================
@@ -90,17 +90,22 @@ static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
   return builder.getIntegerAttr(integerType, -1);
 }
 
+/// Creates `llvm.mlir.constant` with a scalar or vector integer value,
+/// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
+static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
+                                   PatternRewriter &rewriter,
+                                   IntegerAttr scalarAttr) {
+  if (auto vecType = dyn_cast<VectorType>(srcType))
+    return LLVM::ConstantOp::create(
+        rewriter, loc, dstType, SplatElementsAttr::get(vecType, scalarAttr));
+  return LLVM::ConstantOp::create(rewriter, loc, dstType, scalarAttr);
+}
+
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
-  if (isa<VectorType>(srcType)) {
-    return LLVM::ConstantOp::create(
-        rewriter, loc, dstType,
-        SplatElementsAttr::get(cast<ShapedType>(srcType),
-                               minusOneIntegerAttribute(srcType, rewriter)));
-  }
-  return LLVM::ConstantOp::create(rewriter, loc, dstType,
-                                  minusOneIntegerAttribute(srcType, rewriter));
+  return createIntegerConstant(loc, srcType, dstType, rewriter,
+                               minusOneIntegerAttribute(srcType, rewriter));
----------------
IgWod wrote:

Maybe it'd make sense to inline `minusOneIntegerAttribute` now (unless it's used anywhere else); but that's something for a future cleanup.

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Mon Jul 13 01:43:27 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 01:43:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a54a52f.6c8668f2.e4f82.3fc3@mx.google.com>


================

----------------
banach-space wrote:

[nit] What does "varying dimension" mean?

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Mon Jul 13 01:44:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:44:42 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
Message-ID: <llvm.org/llvm/llvm-project/pull/209111@github.com>


https://github.com/LouisLu060211 created https://github.com/llvm/llvm-project/pull/209111

The `affine.for` folder crashes when it runs inside dialect conversion. The test pass `func.func` to `llvm.func`, which detaches the old entry block while values inside the loop still point to the old block arguments. The old block has no parent region at this point, which is a dialect conversion problem. The loop yields one of these old block arguments, and the assertion in the folder calls `isDefinedOutsideOfLoop` on it. The crash is fixed by adding a check before the assertion. So the folder bails out when the value has no parent region. The loop cannot be folded in this state anyway, so the conversion driver leaves the op alone and the pass finishes without crashing.

Assisted by: GPT 5.5


>From 4087c0ff57a953a3984e9f6d5f17684b82a944c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/3] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 1434c4b3e9a81abbe1c5ec3a4f35fe817e9e01c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/3] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 7e5e80a6cae5d1546f817f6468759045879d0e66 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:38:13 +0800
Subject: [PATCH 3/3] Fix AffineForOP creash

---
 a.mlir                                   | 17 +++++++++++++++++
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp |  3 +++
 2 files changed, 20 insertions(+)
 create mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
new file mode 100644
index 0000000000000..899e405a1dabb
--- /dev/null
+++ b/a.mlir
@@ -0,0 +1,17 @@
+module {
+  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;


From llvmlistbot at llvm.org  Mon Jul 13 01:44:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:44:57 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54a589.e624eb8d.39cd4e.b4a4@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @LouisLu060211 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/209111

From llvmlistbot at llvm.org  Mon Jul 13 01:45:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:45:31 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54a5ab.f06f4994.1fc857.75eb@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: LouisLu060211

<details>
<summary>Changes</summary>

The `affine.for` folder crashes when it runs inside dialect conversion. The test pass `func.func` to `llvm.func`, which detaches the old entry block while values inside the loop still point to the old block arguments. The old block has no parent region at this point, which is a dialect conversion problem. The loop yields one of these old block arguments, and the assertion in the folder calls `isDefinedOutsideOfLoop` on it. The crash is fixed by adding a check before the assertion. So the folder bails out when the value has no parent region. The loop cannot be folded in this state anyway, so the conversion driver leaves the op alone and the pass finishes without crashing.

Assisted by: GPT 5.5


---
Full diff: https://github.com/llvm/llvm-project/pull/209111.diff


6 Files Affected:

- (added) a.mlir (+17) 
- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+3) 
- (modified) mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp (+9-4) 
- (modified) mlir/test/Analysis/test-alias-analysis-modref.mlir (+14) 
- (modified) mlir/test/Dialect/MemRef/mem2reg.mlir (+15) 
- (modified) mlir/test/lib/Analysis/TestAliasAnalysis.cpp (+40-10) 


``````````diff
diff --git a/a.mlir b/a.mlir
new file mode 100644
index 0000000000000..899e405a1dabb
--- /dev/null
+++ b/a.mlir
@@ -0,0 +1,17 @@
+module {
+  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

``````````

</details>


https://github.com/llvm/llvm-project/pull/209111

From llvmlistbot at llvm.org  Mon Jul 13 01:46:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 01:46:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a54a5ec.e624eb8d.39cd4e.f41a@mx.google.com>


================

----------------
banach-space wrote:

This comment has not been addressed.

https://github.com/llvm/llvm-project/pull/203873

From llvmlistbot at llvm.org  Mon Jul 13 01:50:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:50:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54a6cb.eb165e5c.1817a5.f98b@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 4087c0ff57a953a3984e9f6d5f17684b82a944c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/4] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 1434c4b3e9a81abbe1c5ec3a4f35fe817e9e01c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/4] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 7e5e80a6cae5d1546f817f6468759045879d0e66 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:38:13 +0800
Subject: [PATCH 3/4] Fix AffineForOP creash

---
 a.mlir                                   | 17 +++++++++++++++++
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp |  3 +++
 2 files changed, 20 insertions(+)
 create mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
new file mode 100644
index 0000000000000..899e405a1dabb
--- /dev/null
+++ b/a.mlir
@@ -0,0 +1,17 @@
+module {
+  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;

>From ff058946ea2e5b8f9ef0d84446d7d0b02bd8d3bd Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:49:54 +0800
Subject: [PATCH 4/4] remove

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 9a8acbb61cfab..6748e2cf71804 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,9 +67,7 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  std::optional<int64_t> numElements =
-      ShapedType::tryGetNumElements(type.getShape());
-  if (!numElements || *numElements != 1)
+  if (type.getNumElements() != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -292,12 +290,9 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape())
-      return {};
-    std::optional<int64_t> numElements =
-        ShapedType::tryGetNumElements(memrefType.getShape());
-    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
-        *numElements == 0)
+    if (!memrefType.hasStaticShape() ||
+        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
+        memrefType.getNumElements() == 1)
       return {};
 
     DenseMap<Attribute, Type> destructured;


From llvmlistbot at llvm.org  Mon Jul 13 01:56:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 01:56:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9e8b030 - [MLIR][Linalg] Fix
 FlattenElementwiseOp on broadcasted linalgs (#207005)
Message-ID: <6a54a829.27b3b003.3b871.e290@mx.google.com>


Author: Hugo Trachino
Date: 2026-07-13T09:56:04+01:00
New Revision: 9e8b030aa80b78abaa054b5f75c36f51d1995088

URL: https://github.com/llvm/llvm-project/commit/9e8b030aa80b78abaa054b5f75c36f51d1995088
DIFF: https://github.com/llvm/llvm-project/commit/9e8b030aa80b78abaa054b5f75c36f51d1995088.diff

LOG: [MLIR][Linalg] Fix FlattenElementwiseOp on broadcasted linalgs (#207005)

Applying FlattenElementwiseLinalgOp on Broadcasted linalg would break.
This transformation is not valid for those usecase, so I added a
condition to exit gracefully instead.

Added: 
    

Modified: 
    mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
    mlir/test/Dialect/Linalg/flatten-elementwise.mlir
    mlir/test/Dialect/Linalg/flatten-unsupported.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index 1eb668637c7e64..42ce13b6f214c4 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4310,6 +4310,14 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
     return mlir::emitSilenceableFailure(target->getLoc())
            << "only elementwise flattening is supported";
 
+  if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
+        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+      })) {
+    results.push_back(target);
+    return mlir::emitSilenceableFailure(target->getLoc())
+           << "operators with broadcasting semantics are not supported";
+  }
+
   // If rank <= 1, do nothing
   if (target.getNumLoops() <= 1) {
     results.push_back(target);

diff  --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index 9fe50a521d2d81..f9f8f9e2b8b449 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -1,11 +1,11 @@
 // RUN: mlir-opt %s -transform-interpreter -split-input-file | FileCheck %s
 
-// CHECK-LABEL: func.func @fill(
+// CHECK-LABEL: func.func @fill_memref(
 // CHECK-SAME:                  %[[ARG0:.*]]: f32,
 // CHECK-SAME:                  %[[ARG1:.*]]: memref<32x7xf32>
 // CHECK-NEXT:    %[[FLATTENED:.*]] = memref.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
 // CHECK-NEXT:    linalg.fill ins(%[[ARG0]] : f32) outs(%[[FLATTENED]] : memref<224xf32>)
-func.func @fill(%cst: f32, %arg: memref<32x7xf32>) {
+func.func @fill_memref(%cst: f32, %arg: memref<32x7xf32>) {
     linalg.fill ins(%cst: f32) outs(%arg: memref<32x7xf32>)
     return
 }
@@ -43,7 +43,7 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
-// CHECK-LABEL: func.func @map(
+// CHECK-LABEL: func.func @map_memref(
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32x7xf32>
@@ -51,7 +51,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK-NEXT:    %[[FLATTENED_1:.*]] = memref.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
 // CHECK-NEXT:    %[[FLATTENED_2:.*]] = memref.collapse_shape %[[ARG2]] {{\[}}[0, 1]]
 // CHECK-NEXT:    linalg.map { arith.addf } ins(%[[FLATTENED_0]], %[[FLATTENED_1]] : memref<224xf32>, memref<224xf32>) outs(%[[FLATTENED_2]] : memref<224xf32>)
-func.func @map(%arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
+func.func @map_memref(%arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
     linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32>, memref<32x7xf32>) outs(%arg2: memref<32x7xf32>)
     return
 }
@@ -67,12 +67,12 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
-// CHECK-LABEL: func.func @map_already_flat(
+// CHECK-LABEL: func.func @map_already_flat_memref(
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32xf32>
 // CHECK-NEXT:    linalg.map { arith.addf } ins(%[[ARG0]], %[[ARG1]] : memref<32xf32>, memref<32xf32>) outs(%[[ARG2]] : memref<32xf32>)
-func.func @map_already_flat(%arg0: memref<32xf32>, %arg1: memref<32xf32>, %arg2: memref<32xf32>) {
+func.func @map_already_flat_memref(%arg0: memref<32xf32>, %arg1: memref<32xf32>, %arg2: memref<32xf32>) {
     linalg.map {arith.addf} ins(%arg0, %arg1: memref<32xf32>, memref<32xf32>) outs(%arg2: memref<32xf32>)
     return
 }
@@ -89,7 +89,7 @@ module attributes {transform.with_named_sequence} {
 // -----
 
 // CHECK: #[[$MAP0:.*]] = affine_map<(d0) -> (d0)>
-// CHECK-LABEL: func.func @generic
+// CHECK-LABEL: func.func @elementwise_memref
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG2:[a-zA-Z0-9_]*]]: memref<32x7xf32>
@@ -101,7 +101,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK-NEXT:         %[[SUM:.*]] = arith.addf %[[A]], %[[B]]
 // CHECK-NEXT:         linalg.yield %[[SUM]]
 #map = affine_map<(d0, d1) -> (d0, d1)>
-func.func @generic( %arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
+func.func @elementwise_memref( %arg0: memref<32x7xf32>, %arg1: memref<32x7xf32>, %arg2: memref<32x7xf32>) {
     linalg.generic {indexing_maps = [#map, #map, #map], iterator_types = ["parallel", "parallel"]} ins(%arg0, %arg1: memref<32x7xf32>, memref<32x7xf32>) outs(%arg2: memref<32x7xf32>) {
         ^bb0(%a: f32, %b: f32, %c: f32):
             %0 = arith.addf %a, %b : f32

diff  --git a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
index 499db4cfb3294d..f0b8b8b153bbb4 100644
--- a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -transform-interpreter -split-input-file -verify-diagnostics
 
-func.func @non_elementwise(%arg0: memref<2x3xf32>, %arg1: memref<3x4xf32>, %arg2: memref<2x4xf32>) {
+func.func @unsupported_non_elementwise(%arg0: memref<2x3xf32>, %arg1: memref<3x4xf32>, %arg2: memref<2x4xf32>) {
   // expected-error @below {{only elementwise flattening is supported}}
   linalg.matmul ins(%arg0, %arg1 : memref<2x3xf32>, memref<3x4xf32>) outs(%arg2: memref<2x4xf32>)
   return
@@ -19,8 +19,53 @@ module attributes {transform.with_named_sequence} {
 
 func.func @unsupported_memref(%arg0: memref<32x7xf32, strided<[7, 2]>>, %arg1: memref<32x7xf32, strided<[7, 2]>>, %arg2: memref<32x7xf32, strided<[7, 2]>>) {
   // expected-error @below {{attempted to flatten, but failed}}
-    linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32, strided<[7, 2]>>, memref<32x7xf32, strided<[7, 2]>>) outs(%arg2: memref<32x7xf32, strided<[7, 2]>>)
-    return
+  linalg.map {arith.addf} ins(%arg0, %arg1: memref<32x7xf32, strided<[7, 2]>>, memref<32x7xf32, strided<[7, 2]>>) outs(%arg2: memref<32x7xf32, strided<[7, 2]>>)
+  return
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+#map1 = affine_map<(d0, d1, d2) -> (0, d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @unsupported_broadcasting_elementwise(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+    ^bb0(%in: i32, %out: i32):
+      linalg.yield %in : i32
+  } -> tensor<32x2x2xi32>
+  return %0 : tensor<32x2x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
+#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
+
+func.func @unsupported_rank_expanding_broadcasting_elementwise(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
+    ^bb0(%in: i32, %out: i32):
+      linalg.yield %in : i32
+  } -> tensor<32x2x2xi32>
+  return %0 : tensor<32x2x2xi32>
 }
 
 module attributes {transform.with_named_sequence} {


        

From llvmlistbot at llvm.org  Mon Jul 13 01:56:13 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Mon, 13 Jul 2026 01:56:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp on
 broadcasted linalgs (PR #207005)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207005@github.com>
Message-ID: <6a54a82d.13be6df8.145c.5b4d@mx.google.com>


https://github.com/nujaa closed https://github.com/llvm/llvm-project/pull/207005

From llvmlistbot at llvm.org  Mon Jul 13 02:00:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:00:27 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54a92b.66c0679a.3aa103.1bf8@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 4087c0ff57a953a3984e9f6d5f17684b82a944c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/5] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 1434c4b3e9a81abbe1c5ec3a4f35fe817e9e01c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/5] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 7e5e80a6cae5d1546f817f6468759045879d0e66 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:38:13 +0800
Subject: [PATCH 3/5] Fix AffineForOP creash

---
 a.mlir                                   | 17 +++++++++++++++++
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp |  3 +++
 2 files changed, 20 insertions(+)
 create mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
new file mode 100644
index 0000000000000..899e405a1dabb
--- /dev/null
+++ b/a.mlir
@@ -0,0 +1,17 @@
+module {
+  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;

>From ff058946ea2e5b8f9ef0d84446d7d0b02bd8d3bd Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:49:54 +0800
Subject: [PATCH 4/5] remove

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 9a8acbb61cfab..6748e2cf71804 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,9 +67,7 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  std::optional<int64_t> numElements =
-      ShapedType::tryGetNumElements(type.getShape());
-  if (!numElements || *numElements != 1)
+  if (type.getNumElements() != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -292,12 +290,9 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape())
-      return {};
-    std::optional<int64_t> numElements =
-        ShapedType::tryGetNumElements(memrefType.getShape());
-    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
-        *numElements == 0)
+    if (!memrefType.hasStaticShape() ||
+        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
+        memrefType.getNumElements() == 1)
       return {};
 
     DenseMap<Attribute, Type> destructured;

>From 8bd4d3ff49e0120be740eb50bc30b906406b3ad4 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 17:00:08 +0800
Subject: [PATCH 5/5] fix

---
 a.mlir                                        | 17 -------
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ------
 mlir/test/Dialect/MemRef/mem2reg.mlir         | 15 ------
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 48 +++++--------------
 4 files changed, 11 insertions(+), 83 deletions(-)
 delete mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
deleted file mode 100644
index 899e405a1dabb..0000000000000
--- a/a.mlir
+++ /dev/null
@@ -1,17 +0,0 @@
-module {
-  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
-    %cst = arith.constant 0.000000e+00 : f32
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c10 = arith.constant 10 : index
-    scf.for %arg4 = %c0 to %c10 step %c1 {
-      %0 = memref.load %arg1[%arg4] : memref<?xf32>
-      gpu.barrier
-    }
-    scf.for %arg4 = %c0 to %arg3 step %c1 {
-      %0 = memref.load %arg1[%arg4] : memref<?xf32>
-      gpu.barrier
-    }
-    return
-  }
-}
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 9c1fa17c1249a..2a9c612317d53 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,17 +112,3 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
-
-// -----
-
-// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
-// CHECK: func.func -> func.func.region0#0: ModRef
-module {
-  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
-    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
-    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
-    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
-    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
-    return
-  }
-}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 9327db24ff39c..8f937c4efe75e 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,18 +309,3 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
-
-
-// -----
-
-// Make sure mem2reg does not crash on an alloca whose element count overflows
-// int64. https://github.com/llvm/llvm-project/issues/204297
-// CHECK-LABEL: func.func @alloca_element_count_overflow
-func.func @alloca_element_count_overflow() {
-  return
-^bb1(%0: index):
-  // The alloca must be left untouched (not promoted).
-  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
-  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
-  return
-}
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 8b33a08db4ab1..d59e9854e925a 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,63 +16,36 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
-#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-namespace mlir {
-namespace test {
-
-/// Print an operation that is used as an operand of an alias query.
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  if (!op) {
-    llvm::errs() << "<<NULL OPERATION>>";
-    return;
-  }
-
-  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
-    llvm::errs() << attr.getValue();
-    return;
-  }
-
-  llvm::errs() << op->getName().getStringRef();
+  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
 }
 
 /// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
-    Operation *parentOp = region ? region->getParentOp() : nullptr;
-
-    if (parentOp) {
-      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
-        llvm::errs() << attr.getValue();
-      else
-        llvm::errs() << parentOp->getName().getStringRef();
-    } else {
-      llvm::errs() << "<<UNKNOWN PARENT>>";
-    }
-
-    if (region)
-      llvm::errs() << ".region" << region->getRegionNumber();
-
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
+    llvm::errs() << region->getParentOp()
+                        ->getAttrOfType<StringAttr>("test.ptr")
+                        .getValue()
+                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
 
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-
-  Operation *owner = value.getDefiningOp();
-  assert(owner && "expected non-block argument value to have a defining op");
-
-  printAliasOperand(owner);
-
-  auto result = cast<mlir::OpResult>(value);
+  OpResult result = cast<OpResult>(value);
+  printAliasOperand(result.getOwner());
   llvm::errs() << "#" << result.getResultNumber();
 }
 
+namespace mlir {
+namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";
@@ -253,3 +226,4 @@ void registerTestAliasAnalysisPass() {
 }
 } // namespace test
 } // namespace mlir
+


From llvmlistbot at llvm.org  Mon Jul 13 02:02:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:02:11 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54a993.9edb5c9d.208698.0df9@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 4087c0ff57a953a3984e9f6d5f17684b82a944c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/6] [mlir][memref] Avoid mem2reg crash on element count
 overflow

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 +++++++++----
 mlir/test/Dialect/MemRef/mem2reg.mlir           | 15 +++++++++++++++
 2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..9a8acbb61cfab 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,7 +67,9 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
+  std::optional<int64_t> numElements =
+      ShapedType::tryGetNumElements(type.getShape());
+  if (!numElements || *numElements != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -290,9 +292,12 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape() ||
-        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
-        memrefType.getNumElements() == 1)
+    if (!memrefType.hasStaticShape())
+      return {};
+    std::optional<int64_t> numElements =
+        ShapedType::tryGetNumElements(memrefType.getShape());
+    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
+        *numElements == 0)
       return {};
 
     DenseMap<Attribute, Type> destructured;
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 8f937c4efe75e..9327db24ff39c 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,3 +309,18 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
+
+
+// -----
+
+// Make sure mem2reg does not crash on an alloca whose element count overflows
+// int64. https://github.com/llvm/llvm-project/issues/204297
+// CHECK-LABEL: func.func @alloca_element_count_overflow
+func.func @alloca_element_count_overflow() {
+  return
+^bb1(%0: index):
+  // The alloca must be left untouched (not promoted).
+  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
+  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
+  return
+}

>From 1434c4b3e9a81abbe1c5ec3a4f35fe817e9e01c6 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 30 Jun 2026 16:56:00 +0800
Subject: [PATCH 2/6] Avoid crash in test alias analysis operand printing

---
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ++++++
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 50 +++++++++++++++----
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 2a9c612317d53..9c1fa17c1249a 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,3 +112,17 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
+
+// -----
+
+// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
+// CHECK: func.func -> func.func.region0#0: ModRef
+module {
+  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
+    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
+    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
+    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
+    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
+    return
+  }
+}
\ No newline at end of file
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 0125e403272a8..8b33a08db4ab1 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,33 +16,63 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
+#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-/// Print a value that is used as an operand of an alias query.
+namespace mlir {
+namespace test {
+
+/// Print an operation that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
+  if (!op) {
+    llvm::errs() << "<<NULL OPERATION>>";
+    return;
+  }
+
+  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
+    llvm::errs() << attr.getValue();
+    return;
+  }
+
+  llvm::errs() << op->getName().getStringRef();
 }
+
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
+    Operation *parentOp = region ? region->getParentOp() : nullptr;
+
+    if (parentOp) {
+      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
+        llvm::errs() << attr.getValue();
+      else
+        llvm::errs() << parentOp->getName().getStringRef();
+    } else {
+      llvm::errs() << "<<UNKNOWN PARENT>>";
+    }
+
+    if (region)
+      llvm::errs() << ".region" << region->getRegionNumber();
+
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
-    llvm::errs() << region->getParentOp()
-                        ->getAttrOfType<StringAttr>("test.ptr")
-                        .getValue()
-                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
+
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-  OpResult result = cast<OpResult>(value);
-  printAliasOperand(result.getOwner());
+
+  Operation *owner = value.getDefiningOp();
+  assert(owner && "expected non-block argument value to have a defining op");
+
+  printAliasOperand(owner);
+
+  auto result = cast<mlir::OpResult>(value);
   llvm::errs() << "#" << result.getResultNumber();
 }
 
-namespace mlir {
-namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";

>From 7e5e80a6cae5d1546f817f6468759045879d0e66 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:38:13 +0800
Subject: [PATCH 3/6] Fix AffineForOP creash

---
 a.mlir                                   | 17 +++++++++++++++++
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp |  3 +++
 2 files changed, 20 insertions(+)
 create mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
new file mode 100644
index 0000000000000..899e405a1dabb
--- /dev/null
+++ b/a.mlir
@@ -0,0 +1,17 @@
+module {
+  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
+    %cst = arith.constant 0.000000e+00 : f32
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c10 = arith.constant 10 : index
+    scf.for %arg4 = %c0 to %c10 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    scf.for %arg4 = %c0 to %arg3 step %c1 {
+      %0 = memref.load %arg1[%arg4] : memref<?xf32>
+      gpu.barrier
+    }
+    return
+  }
+}
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;

>From ff058946ea2e5b8f9ef0d84446d7d0b02bd8d3bd Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 16:49:54 +0800
Subject: [PATCH 4/6] remove

---
 mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 9a8acbb61cfab..6748e2cf71804 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -67,9 +67,7 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   if (!type.hasStaticShape())
     return {};
   // Make sure the memref contains only a single element.
-  std::optional<int64_t> numElements =
-      ShapedType::tryGetNumElements(type.getShape());
-  if (!numElements || *numElements != 1)
+  if (type.getNumElements() != 1)
     return {};
 
   return {MemorySlot{getResult(), type.getElementType()}};
@@ -292,12 +290,9 @@ struct MemRefDestructurableTypeExternalModel
   getSubelementIndexMap(Type type) const {
     auto memrefType = llvm::cast<MemRefType>(type);
     constexpr int64_t maxMemrefSizeForDestructuring = 16;
-    if (!memrefType.hasStaticShape())
-      return {};
-    std::optional<int64_t> numElements =
-        ShapedType::tryGetNumElements(memrefType.getShape());
-    if (!numElements || *numElements > maxMemrefSizeForDestructuring ||
-        *numElements == 0)
+    if (!memrefType.hasStaticShape() ||
+        memrefType.getNumElements() > maxMemrefSizeForDestructuring ||
+        memrefType.getNumElements() == 1)
       return {};
 
     DenseMap<Attribute, Type> destructured;

>From 8bd4d3ff49e0120be740eb50bc30b906406b3ad4 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 17:00:08 +0800
Subject: [PATCH 5/6] fix

---
 a.mlir                                        | 17 -------
 .../Analysis/test-alias-analysis-modref.mlir  | 14 ------
 mlir/test/Dialect/MemRef/mem2reg.mlir         | 15 ------
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp  | 48 +++++--------------
 4 files changed, 11 insertions(+), 83 deletions(-)
 delete mode 100644 a.mlir

diff --git a/a.mlir b/a.mlir
deleted file mode 100644
index 899e405a1dabb..0000000000000
--- a/a.mlir
+++ /dev/null
@@ -1,17 +0,0 @@
-module {
-  func.func private @fold_vector_transfer(%arg0: index, %arg1: memref<?xf32>, %arg2: index, %arg3: index) {
-    %cst = arith.constant 0.000000e+00 : f32
-    %c0 = arith.constant 0 : index
-    %c1 = arith.constant 1 : index
-    %c10 = arith.constant 10 : index
-    scf.for %arg4 = %c0 to %c10 step %c1 {
-      %0 = memref.load %arg1[%arg4] : memref<?xf32>
-      gpu.barrier
-    }
-    scf.for %arg4 = %c0 to %arg3 step %c1 {
-      %0 = memref.load %arg1[%arg4] : memref<?xf32>
-      gpu.barrier
-    }
-    return
-  }
-}
diff --git a/mlir/test/Analysis/test-alias-analysis-modref.mlir b/mlir/test/Analysis/test-alias-analysis-modref.mlir
index 9c1fa17c1249a..2a9c612317d53 100644
--- a/mlir/test/Analysis/test-alias-analysis-modref.mlir
+++ b/mlir/test/Analysis/test-alias-analysis-modref.mlir
@@ -112,17 +112,3 @@ func.func @conditional_all_effects(%arg: memref<2xf32>) attributes {test.ptr = "
   "test.conditional_side_effect_op"() {has_effects = true, test.ptr = "conditional_side_effect_op"} : () -> i32
   return {test.ptr = "return"}
 }
-
-// -----
-
-// CHECK-LABEL: Testing : "block_argument_with_unit_test_ptr"
-// CHECK: func.func -> func.func.region0#0: ModRef
-module {
-  func.func @block_argument_with_unit_test_ptr(%arg0: f32 {test._ptr}) attributes {test.ptr} {
-    %0 = "test.ptr"(%arg0) {test._ptr} : (f32) -> f32
-    %1 = "test.ptr"(%arg0) {test.inclusive} : (f32) -> f32
-    %2 = "test.ptr"(%arg0) {test.exclusive} : (f32) -> f32
-    %3 = "test.ptr"(%arg0) {test.c_ptr} : (f32) -> f32
-    return
-  }
-}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/mem2reg.mlir b/mlir/test/Dialect/MemRef/mem2reg.mlir
index 9327db24ff39c..8f937c4efe75e 100644
--- a/mlir/test/Dialect/MemRef/mem2reg.mlir
+++ b/mlir/test/Dialect/MemRef/mem2reg.mlir
@@ -309,18 +309,3 @@ func.func @two_consecutive_merge_points(%cond1: i1, %cond2: i1) -> i32 {
   // CHECK: return %[[RESULT]] : i32
   return %result : i32
 }
-
-
-// -----
-
-// Make sure mem2reg does not crash on an alloca whose element count overflows
-// int64. https://github.com/llvm/llvm-project/issues/204297
-// CHECK-LABEL: func.func @alloca_element_count_overflow
-func.func @alloca_element_count_overflow() {
-  return
-^bb1(%0: index):
-  // The alloca must be left untouched (not promoted).
-  // CHECK: memref.alloca() : memref<9223372036854775807x3xi32>
-  %alloca = memref.alloca() : memref<9223372036854775807x3xi32>
-  return
-}
diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index 8b33a08db4ab1..d59e9854e925a 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -16,63 +16,36 @@
 #include "mlir/Analysis/AliasAnalysis/LocalAliasAnalysis.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Pass/Pass.h"
-#include "mlir/IR/Value.h"
 
 using namespace mlir;
 
-namespace mlir {
-namespace test {
-
-/// Print an operation that is used as an operand of an alias query.
+/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Operation *op) {
-  if (!op) {
-    llvm::errs() << "<<NULL OPERATION>>";
-    return;
-  }
-
-  if (StringAttr attr = op->getAttrOfType<StringAttr>("test.ptr")) {
-    llvm::errs() << attr.getValue();
-    return;
-  }
-
-  llvm::errs() << op->getName().getStringRef();
+  llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
 }
 
 /// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
-    Operation *parentOp = region ? region->getParentOp() : nullptr;
-
-    if (parentOp) {
-      if (StringAttr attr = parentOp->getAttrOfType<StringAttr>("test.ptr"))
-        llvm::errs() << attr.getValue();
-      else
-        llvm::errs() << parentOp->getName().getStringRef();
-    } else {
-      llvm::errs() << "<<UNKNOWN PARENT>>";
-    }
-
-    if (region)
-      llvm::errs() << ".region" << region->getRegionNumber();
-
     unsigned parentBlockNumber = arg.getOwner()->computeBlockNumber();
+    llvm::errs() << region->getParentOp()
+                        ->getAttrOfType<StringAttr>("test.ptr")
+                        .getValue()
+                 << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
 
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
-
-  Operation *owner = value.getDefiningOp();
-  assert(owner && "expected non-block argument value to have a defining op");
-
-  printAliasOperand(owner);
-
-  auto result = cast<mlir::OpResult>(value);
+  OpResult result = cast<OpResult>(value);
+  printAliasOperand(result.getOwner());
   llvm::errs() << "#" << result.getResultNumber();
 }
 
+namespace mlir {
+namespace test {
 void printAliasResult(AliasResult result, Value lhs, Value rhs) {
   printAliasOperand(lhs);
   llvm::errs() << " <-> ";
@@ -253,3 +226,4 @@ void registerTestAliasAnalysisPass() {
 }
 } // namespace test
 } // namespace mlir
+

>From 40f41ecd6a705fcaa1790d68ab705f250fc5380e Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Mon, 13 Jul 2026 17:01:50 +0800
Subject: [PATCH 6/6] fix issue

---
 mlir/test/lib/Analysis/TestAliasAnalysis.cpp | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
index d59e9854e925a..0125e403272a8 100644
--- a/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
+++ b/mlir/test/lib/Analysis/TestAliasAnalysis.cpp
@@ -23,8 +23,6 @@ using namespace mlir;
 static void printAliasOperand(Operation *op) {
   llvm::errs() << op->getAttrOfType<StringAttr>("test.ptr").getValue();
 }
-
-/// Print a value that is used as an operand of an alias query.
 static void printAliasOperand(Value value) {
   if (BlockArgument arg = dyn_cast<BlockArgument>(value)) {
     Region *region = arg.getParentRegion();
@@ -35,7 +33,6 @@ static void printAliasOperand(Value value) {
                  << ".region" << region->getRegionNumber();
     if (parentBlockNumber != 0)
       llvm::errs() << ".block" << parentBlockNumber;
-
     llvm::errs() << "#" << arg.getArgNumber();
     return;
   }
@@ -226,4 +223,3 @@ void registerTestAliasAnalysisPass() {
 }
 } // namespace test
 } // namespace mlir
-


From llvmlistbot at llvm.org  Mon Jul 13 02:06:44 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 02:06:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54aaa4.510f576d.1b3cb9.4e5f@mx.google.com>


================
@@ -72,3 +72,51 @@ func.func @memref_load_store(%buff0: memref<2xf32>,
   memref.store %v, %buff1[%i, %j] : memref<4x8xf32>
   return
 }
+
+/// Rank-0 alloc-backed load/store lower through pointer subscript at index 0.
+// CHECK-LABEL: emitc.func private @memref_rank0_alloc_store(
+// CHECK-SAME:  %[[VAL:.*]]: i32)
+func.func private @memref_rank0_alloc_store(%v : i32) {
----------------
banach-space wrote:

[nit] `@memref_alloc_store` -> `@memref_alloc_store_rank0` (add new info at the end rather than in the middle). Similar comment for other tests.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 02:06:44 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 02:06:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54aaa4.2490598a.cf17c.7ec4@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you!

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 02:06:44 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 02:06:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54aaa4.c574b9b4.21ead0.c895@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 02:06:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:06:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 13dbff3 - [Flang][Openmp] Implement support
 for partial in unroll construct (#206642)
Message-ID: <6a54aaad.160324b0.195192.60d8@mx.google.com>


Author: ShashwathiNavada
Date: 2026-07-13T14:36:47+05:30
New Revision: 13dbff365eabc45fcc26a36ef077f52b6e168347

URL: https://github.com/llvm/llvm-project/commit/13dbff365eabc45fcc26a36ef077f52b6e168347
DIFF: https://github.com/llvm/llvm-project/commit/13dbff365eabc45fcc26a36ef077f52b6e168347.diff

LOG: [Flang][Openmp] Implement support for partial in unroll construct (#206642)

As described in section 9.2.2 of openmp 5.2 spec, the patch implements
support for partial in unroll construct.

Added: 
    flang/test/Lower/OpenMP/unroll-partial01.f90
    mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
    mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
    mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir

Modified: 
    flang/lib/Lower/OpenMP/ClauseProcessor.cpp
    flang/lib/Lower/OpenMP/ClauseProcessor.h
    flang/lib/Lower/OpenMP/OpenMP.cpp
    mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
    mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Dialect/OpenMP/invalid.mlir

Removed: 
    


################################################################################
diff  --git a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
index 05e7edc8bdc6a..26438ef968008 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.cpp
@@ -818,6 +818,15 @@ bool ClauseProcessor::processOrdered(
   return false;
 }
 
+bool ClauseProcessor::processPartial(std::optional<int64_t> &result) const {
+  if (auto *clause = findUniqueClause<omp::clause::Partial>()) {
+    if (clause->v.has_value())
+      result = evaluate::ToInt64(*clause->v);
+    return true;
+  }
+  return false;
+}
+
 bool ClauseProcessor::processPriority(
     lower::StatementContext &stmtCtx,
     mlir::omp::PriorityClauseOps &result) const {

diff  --git a/flang/lib/Lower/OpenMP/ClauseProcessor.h b/flang/lib/Lower/OpenMP/ClauseProcessor.h
index 1435af3e844e9..ff62d0ead8dc9 100644
--- a/flang/lib/Lower/OpenMP/ClauseProcessor.h
+++ b/flang/lib/Lower/OpenMP/ClauseProcessor.h
@@ -110,6 +110,7 @@ class ClauseProcessor {
                          mlir::omp::NumThreadsClauseOps &result) const;
   bool processOrder(mlir::omp::OrderClauseOps &result) const;
   bool processOrdered(mlir::omp::OrderedClauseOps &result) const;
+  bool processPartial(std::optional<int64_t> &result) const;
   bool processPriority(lower::StatementContext &stmtCtx,
                        mlir::omp::PriorityClauseOps &result) const;
   bool processProcBind(mlir::omp::ProcBindClauseOps &result) const;

diff  --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 9c6352b73a585..31587c624153c 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -3162,10 +3162,17 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
                         ConstructQueue::const_iterator item) {
   fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
 
-  // Clauses for unrolling not yet implemnted
   ClauseProcessor cp(converter, semaCtx, item->clauses);
-  cp.processTODO<clause::Partial, clause::Full>(
-      loc, llvm::omp::Directive::OMPD_unroll);
+
+  // The `full` clause is not yet implemented.
+  cp.processTODO<clause::Full>(loc, llvm::omp::Directive::OMPD_unroll);
+
+  // Process the `partial` clause. If present, it may carry a constant unroll
+  // factor.
+  std::optional<int64_t> partialFactor;
+  bool hasPartial = cp.processPartial(partialFactor);
+  if (hasPartial && !partialFactor.has_value())
+    TODO(loc, "PARTIAL clause on UNROLL without a constant factor");
 
   // Emit the associated loop
   llvm::SmallVector<mlir::omp::CanonicalLoopOp, 1> canonLoops;
@@ -3177,9 +3184,16 @@ static void genUnrollOp(Fortran::lower::AbstractConverter &converter,
   for (auto &&canonLoop : canonLoops)
     applyees.push_back(canonLoop.getCli());
 
-  // Apply unrolling to it
   auto cli = llvm::getSingleElement(canonLoops).getCli();
-  mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+
+  if (partialFactor.has_value()) {
+    // Partially unroll the loop by the given constant factor.
+    mlir::omp::UnrollPartialOp::create(firOpBuilder, loc, cli,
+                                       static_cast<uint64_t>(*partialFactor));
+  } else {
+    // Apply heuristic unrolling to it.
+    mlir::omp::UnrollHeuristicOp::create(firOpBuilder, loc, cli);
+  }
 }
 
 static mlir::omp::MaskedOp

diff  --git a/flang/test/Lower/OpenMP/unroll-partial01.f90 b/flang/test/Lower/OpenMP/unroll-partial01.f90
new file mode 100644
index 0000000000000..fe76d8f077753
--- /dev/null
+++ b/flang/test/Lower/OpenMP/unroll-partial01.f90
@@ -0,0 +1,54 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+
+subroutine omp_unroll_partial01(lb, ub, inc)
+  integer res, i, lb, ub, inc
+
+  !$omp unroll partial(4)
+  do i = lb, ub, inc
+    res = i
+  end do
+  !$omp end unroll
+
+end subroutine omp_unroll_partial01
+
+
+! CHECK-LABEL:   func.func @_QPomp_unroll_partial01(
+! CHECK-SAME:      %[[ARG0:.*]]: !fir.ref<i32> {fir.bindc_name = "lb"},
+! CHECK-SAME:      %[[ARG1:.*]]: !fir.ref<i32> {fir.bindc_name = "ub"},
+! CHECK-SAME:      %[[ARG2:.*]]: !fir.ref<i32> {fir.bindc_name = "inc"}) {
+! CHECK:           %[[VAL_0:.*]] = fir.dummy_scope : !fir.dscope
+! CHECK:           %[[VAL_1:.*]] = fir.alloca i32 {bindc_name = "i", uniq_name = "_QFomp_unroll_partial01Ei"}
+! CHECK:           %[[VAL_2:.*]]:2 = hlfir.declare %[[VAL_1]] {uniq_name = "_QFomp_unroll_partial01Ei"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_3:.*]]:2 = hlfir.declare %[[ARG2]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Einc"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_4:.*]]:2 = hlfir.declare %[[ARG0]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Elb"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_5:.*]] = fir.alloca i32 {bindc_name = "res", uniq_name = "_QFomp_unroll_partial01Eres"}
+! CHECK:           %[[VAL_6:.*]]:2 = hlfir.declare %[[VAL_5]] {uniq_name = "_QFomp_unroll_partial01Eres"} : (!fir.ref<i32>) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_7:.*]]:2 = hlfir.declare %[[ARG1]] dummy_scope %[[VAL_0]] arg {{[0-9]+}} {uniq_name = "_QFomp_unroll_partial01Eub"} : (!fir.ref<i32>, !fir.dscope) -> (!fir.ref<i32>, !fir.ref<i32>)
+! CHECK:           %[[VAL_8:.*]] = fir.load %[[VAL_4]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_9:.*]] = fir.load %[[VAL_7]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_10:.*]] = fir.load %[[VAL_3]]#0 : !fir.ref<i32>
+! CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+! CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+! CHECK:           %[[VAL_13:.*]] = arith.cmpi slt, %[[VAL_10]], %[[VAL_11]] : i32
+! CHECK:           %[[VAL_14:.*]] = arith.subi %[[VAL_11]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_15:.*]] = arith.select %[[VAL_13]], %[[VAL_14]], %[[VAL_10]] : i32
+! CHECK:           %[[VAL_16:.*]] = arith.select %[[VAL_13]], %[[VAL_9]], %[[VAL_8]] : i32
+! CHECK:           %[[VAL_17:.*]] = arith.select %[[VAL_13]], %[[VAL_8]], %[[VAL_9]] : i32
+! CHECK:           %[[VAL_18:.*]] = arith.subi %[[VAL_17]], %[[VAL_16]] overflow<nuw> : i32
+! CHECK:           %[[VAL_19:.*]] = arith.divui %[[VAL_18]], %[[VAL_15]] : i32
+! CHECK:           %[[VAL_20:.*]] = arith.addi %[[VAL_19]], %[[VAL_12]] overflow<nuw> : i32
+! CHECK:           %[[VAL_21:.*]] = arith.cmpi slt, %[[VAL_17]], %[[VAL_16]] : i32
+! CHECK:           %[[VAL_22:.*]] = arith.select %[[VAL_21]], %[[VAL_11]], %[[VAL_20]] : i32
+! CHECK:           %[[VAL_23:.*]] = omp.new_cli
+! CHECK:           omp.canonical_loop(%[[VAL_23]]) %[[VAL_24:.*]] : i32 in range(%[[VAL_22]]) {
+! CHECK:             %[[VAL_25:.*]] = arith.muli %[[VAL_24]], %[[VAL_10]] : i32
+! CHECK:             %[[VAL_26:.*]] = arith.addi %[[VAL_8]], %[[VAL_25]] : i32
+! CHECK:             hlfir.assign %[[VAL_26]] to %[[VAL_2]]#0 : i32, !fir.ref<i32>
+! CHECK:             %[[VAL_27:.*]] = fir.load %[[VAL_2]]#0 : !fir.ref<i32>
+! CHECK:             hlfir.assign %[[VAL_27]] to %[[VAL_6]]#0 : i32, !fir.ref<i32>
+! CHECK:             omp.terminator
+! CHECK:           }
+! CHECK:           omp.unroll_partial(%[[VAL_23]]) {unroll_factor = 4 : i64}
+! CHECK:           return
+! CHECK:         }

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 5b4ce7cc214bf..932a4a91cc6ab 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -557,6 +557,41 @@ def UnrollHeuristicOp : OpenMPTransform_Op<"unroll_heuristic", []> {
   let hasCustomAssemblyFormat = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// OpenMP unroll_partial operation
+//===----------------------------------------------------------------------===//
+
+def UnrollPartialOp : OpenMPTransform_Op<"unroll_partial", []> {
+  let summary = "OpenMP partial unroll operation";
+  let description = [{
+    Represents a `#pragma omp unroll partial` construct introduced in OpenMP
+    5.1.
+
+    The operation has one applyee and no generatees. The applyee is partially
+    unrolled by the factor given in the `unroll_factor` attribute, which must be
+    a positive constant. That is, the operation replaces `unroll_factor`
+    consecutive logical iterations of the applyee by a single iteration that
+    executes the original body `unroll_factor` times.
+
+    The resulting unrolled loop is currently not accessible as a generatee, so
+    no further loop-associated transformations can be chained onto it.
+
+    The lowering is implemented using `OpenMPIRBuilder::unrollLoopPartial`.
+
+    Assembly formats:
+      omp.unroll_partial(%cli) {unroll_factor = 4 : i64}
+  }];
+
+  let arguments = (ins CanonicalLoopInfoType:$applyee,
+                       ConfinedAttr<I64Attr, [IntPositive]>:$unroll_factor);
+
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$cli, "uint64_t":$unrollFactor)>,
+  ];
+
+  let hasCustomAssemblyFormat = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // OpenMP tile operation
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 41d3c547ca618..acf5500205b6f 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4230,6 +4230,56 @@ UnrollHeuristicOp::getGenerateesODSOperandIndexAndLength() {
   return {0, 0};
 }
 
+//===----------------------------------------------------------------------===//
+// UnrollPartialOp
+//===----------------------------------------------------------------------===//
+
+void UnrollPartialOp::build(::mlir::OpBuilder &odsBuilder,
+                            ::mlir::OperationState &odsState, ::mlir::Value cli,
+                            uint64_t unrollFactor) {
+  odsState.addOperands(cli);
+  Properties &props = odsState.getOrAddProperties<Properties>();
+  props.unroll_factor = odsBuilder.getI64IntegerAttr(unrollFactor);
+}
+
+void UnrollPartialOp::print(OpAsmPrinter &p) {
+  p << '(' << getApplyee() << ')';
+
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+mlir::ParseResult UnrollPartialOp::parse(::mlir::OpAsmParser &parser,
+                                         ::mlir::OperationState &result) {
+  auto cliType = CanonicalLoopInfoType::get(parser.getContext());
+
+  if (parser.parseLParen())
+    return failure();
+
+  OpAsmParser::UnresolvedOperand applyee;
+  if (parser.parseOperand(applyee) ||
+      parser.resolveOperand(applyee, cliType, result.operands))
+    return failure();
+
+  if (parser.parseRParen())
+    return failure();
+
+  // The unroll factor is carried by the `unroll_factor` attribute.
+  if (parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return mlir::success();
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getApplyeesODSOperandIndexAndLength() {
+  return getODSOperandIndexAndLength(odsIndex_applyee);
+}
+
+std::pair<unsigned, unsigned>
+UnrollPartialOp::getGenerateesODSOperandIndexAndLength() {
+  return {0, 0};
+}
+
 //===----------------------------------------------------------------------===//
 // TileOp
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 6929c75af1e2e..7bb15b74aeac3 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5188,6 +5188,30 @@ applyUnrollHeuristic(omp::UnrollHeuristicOp op, llvm::IRBuilderBase &builder,
   return success();
 }
 
+/// Apply a `#pragma omp unroll partial` / `!$omp unroll partial`
+/// transformation using the OpenMPIRBuilder.
+static LogicalResult
+applyUnrollPartial(omp::UnrollPartialOp op, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+
+  Value applyee = op.getApplyee();
+  assert(applyee && "Loop to apply unrolling on required");
+
+  llvm::CanonicalLoopInfo *consBuilderCLI =
+      moduleTranslation.lookupOMPLoop(applyee);
+  llvm::OpenMPIRBuilder::LocationDescription loc(builder);
+
+  // No generatee is supported yet, so the unrolled loop's CanonicalLoopInfo is
+  // not requested and unrolling is deferred to LLVM's LoopUnroll pass.
+  int32_t factor = static_cast<int32_t>(op.getUnrollFactor());
+  ompBuilder->unrollLoopPartial(loc.DL, consBuilderCLI, factor,
+                                /*UnrolledCLI=*/nullptr);
+
+  moduleTranslation.invalidateOmpLoop(applyee);
+  return success();
+}
+
 /// Apply a `#pragma omp tile` / `!$omp tile` transformation using the
 /// OpenMPIRBuilder.
 static LogicalResult applyTile(omp::TileOp op, llvm::IRBuilderBase &builder,
@@ -9688,6 +9712,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             // the omp.canonical_loop.
             return applyUnrollHeuristic(op, builder, moduleTranslation);
           })
+          .Case([&](omp::UnrollPartialOp op) {
+            return applyUnrollPartial(op, builder, moduleTranslation);
+          })
           .Case([&](omp::TileOp op) {
             return applyTile(op, builder, moduleTranslation);
           })

diff  --git a/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
new file mode 100644
index 0000000000000..5ac3a1557ba79
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/cli-unroll-partial.mlir
@@ -0,0 +1,59 @@
+// RUN: mlir-opt %s            | FileCheck %s --enable-var-scope
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s --enable-var-scope
+
+
+// CHECK-LABEL: @omp_unroll_partial_raw(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_raw(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %canonloop = "omp.new_cli" () : () -> (!omp.cli)
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  "omp.canonical_loop" (%tc, %canonloop) ({
+    ^bb0(%iv: i32):
+      omp.terminator
+  }) : (i32, !omp.cli) -> ()
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  "omp.unroll_partial" (%canonloop) {unroll_factor = 4 : i64} : (!omp.cli) -> ()
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %[[CANONLOOP:.+]] = omp.new_cli
+  %canonloop = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%canonloop) %iv : i32 in range(%tc) {
+    omp.terminator
+  }
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  omp.unroll_partial(%canonloop) {unroll_factor = 8 : i64}
+  return
+}
+
+
+// CHECK-LABEL: @omp_unroll_partial_nested_pretty(
+// CHECK-SAME: %[[tc:.+]]: i32) {
+func.func @omp_unroll_partial_nested_pretty(%tc : i32) -> () {
+  // CHECK-NEXT: %canonloop = omp.new_cli
+  %cli_outer = omp.new_cli
+  // CHECK-NEXT: %canonloop_d1 = omp.new_cli
+  %cli_inner = omp.new_cli
+  // CHECK-NEXT: omp.canonical_loop(%canonloop) %iv : i32 in range(%[[tc]]) {
+  omp.canonical_loop(%cli_outer) %iv_outer : i32 in range(%tc) {
+    // CHECK-NEXT: omp.canonical_loop(%canonloop_d1) %iv_d1 : i32 in range(%[[tc]]) {
+    omp.canonical_loop(%cli_inner) %iv_inner : i32 in range(%tc) {
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    // CHECK: omp.terminator
+    omp.terminator
+  }
+
+  // CHECK: omp.unroll_partial(%canonloop) {unroll_factor = 4 : i64}
+  omp.unroll_partial(%cli_outer) {unroll_factor = 4 : i64}
+  // CHECK-NEXT: omp.unroll_partial(%canonloop_d1) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%cli_inner) {unroll_factor = 2 : i64}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 2db7f953fa3bb..668c9fb6dc2b3 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4812,3 +4812,19 @@ func.func @free_shared_mem_invalid_alignment2(%n: i32, %ptr : !llvm.ptr) -> () {
   omp.free_shared_mem [%n x i64 : (i32) align(3)] %ptr : !llvm.ptr
   return
 }
+
+// -----
+// unroll_factor = 0 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_zero(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = 0 : i64}
+  return
+}
+
+// -----
+// unroll_factor = -1 is not a strictly positive integer.
+func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
+  // expected-error @below {{op attribute 'unroll_factor' failed to satisfy constraint: 64-bit signless integer attribute whose value is positive}}
+  omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
+  return
+}

diff  --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
new file mode 100644
index 0000000000000..b9938afea0c30
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial01.mlir
@@ -0,0 +1,57 @@
+// Test lowering of the omp.unroll_partial (single loop)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_trivial_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_trivial_loop(%ptr: !llvm.ptr, %tc: i32) -> () {
+  %literal_cli = omp.new_cli
+  omp.canonical_loop(%literal_cli) %iv : i32 in range(%tc) {
+    %val = llvm.mlir.constant(42.0 : f32) : f32
+    llvm.store %val, %ptr : f32, !llvm.ptr
+    omp.terminator
+  }
+  omp.unroll_partial(%literal_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}

diff  --git a/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
new file mode 100644
index 0000000000000..a62bc499255db
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-unroll-partial02.mlir
@@ -0,0 +1,97 @@
+// Test lowering of the omp.unroll_partial (nested loops)
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+
+// CHECK-LABEL: define void @unroll_partial_nested_loop(
+// CHECK-SAME:    ptr %[[ptr:.+]], i32 %[[outer_tc:.+]], i32 %[[inner_tc:.+]]) {
+// CHECK-NEXT:   br label %omp_omp.loop.preheader
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader:
+// CHECK-NEXT:   br label %omp_omp.loop.header
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header:
+// CHECK-NEXT:   %omp_omp.loop.iv = phi i32 [ 0, %omp_omp.loop.preheader ], [ %omp_omp.loop.next, %omp_omp.loop.inc ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond:
+// CHECK-NEXT:   %omp_omp.loop.cmp = icmp ult i32 %omp_omp.loop.iv, %[[outer_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp, label %omp_omp.loop.body, label %omp_omp.loop.exit
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body:
+// CHECK-NEXT:   br label %omp.loop.region
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region:
+// CHECK-NEXT:   br label %omp_omp.loop.preheader1
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.preheader1:
+// CHECK-NEXT:   br label %omp_omp.loop.header2
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.header2:
+// CHECK-NEXT:   %omp_omp.loop.iv8 = phi i32 [ 0, %omp_omp.loop.preheader1 ], [ %omp_omp.loop.next10, %omp_omp.loop.inc5 ]
+// CHECK-NEXT:   br label %omp_omp.loop.cond3
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.cond3:
+// CHECK-NEXT:   %omp_omp.loop.cmp9 = icmp ult i32 %omp_omp.loop.iv8, %[[inner_tc]]
+// CHECK-NEXT:   br i1 %omp_omp.loop.cmp9, label %omp_omp.loop.body4, label %omp_omp.loop.exit6
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.body4:
+// CHECK-NEXT:   br label %omp.loop.region12
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.loop.region12:
+// CHECK-NEXT:   store float 4.200000e+01, ptr %[[ptr]], align 4
+// CHECK-NEXT:   br label %omp.region.cont11
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont11:
+// CHECK-NEXT:   br label %omp_omp.loop.inc5
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc5:
+// CHECK-NEXT:   %omp_omp.loop.next10 = add nuw i32 %omp_omp.loop.iv8, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header2, !llvm.loop ![[$MD1:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit6:
+// CHECK-NEXT:   br label %omp_omp.loop.after7
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after7:
+// CHECK-NEXT:   br label %omp.region.cont
+// CHECK-EMPTY:
+// CHECK-NEXT: omp.region.cont:
+// CHECK-NEXT:   br label %omp_omp.loop.inc
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.inc:
+// CHECK-NEXT:   %omp_omp.loop.next = add nuw i32 %omp_omp.loop.iv, 1
+// CHECK-NEXT:   br label %omp_omp.loop.header, !llvm.loop ![[$MD4:[0-9]+]]
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.exit:
+// CHECK-NEXT:   br label %omp_omp.loop.after
+// CHECK-EMPTY:
+// CHECK-NEXT: omp_omp.loop.after:
+// CHECK-NEXT:   ret void
+// CHECK-NEXT: }
+llvm.func @unroll_partial_nested_loop(%ptr: !llvm.ptr, %outer_tc: i32, %inner_tc: i32) -> () {
+  %outer_cli = omp.new_cli
+  %inner_cli = omp.new_cli
+  omp.canonical_loop(%outer_cli) %outer_iv : i32 in range(%outer_tc) {
+    omp.canonical_loop(%inner_cli) %inner_iv : i32 in range(%inner_tc) {
+      %val = llvm.mlir.constant(42.0 : f32) : f32
+      llvm.store %val, %ptr : f32, !llvm.ptr
+      omp.terminator
+    }
+    omp.terminator
+  }
+  omp.unroll_partial(%outer_cli) {unroll_factor = 2 : i64}
+  omp.unroll_partial(%inner_cli) {unroll_factor = 4 : i64}
+  llvm.return
+}
+
+
+// Start of metadata
+// CHECK-LABEL: !llvm.module.flags
+
+// The inner loop back-edge (header2) is encountered first in the IR dump, so
+// its metadata node gets a lower ID than the outer loop's metadata node.
+// Both loops share the same llvm.loop.unroll.enable node (uniqued by content).
+// CHECK: ![[$MD1]] = distinct !{![[$MD1]], ![[$MD2:[0-9]+]], ![[$MD3:[0-9]+]]}
+// CHECK: ![[$MD2]] = !{!"llvm.loop.unroll.enable"}
+// CHECK: ![[$MD3]] = !{!"llvm.loop.unroll.count", i32 4}
+// CHECK: ![[$MD4]] = distinct !{![[$MD4]], ![[$MD2]], ![[$MD5:[0-9]+]]}
+// CHECK: ![[$MD5]] = !{!"llvm.loop.unroll.count", i32 2}


        

From llvmlistbot at llvm.org  Mon Jul 13 02:06:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:06:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [Flang][Openmp] Implement support for
 partial in unroll construct (PR #206642)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206642@github.com>
Message-ID: <6a54aaaf.0c05e68e.1e82f1.23f4@mx.google.com>


https://github.com/ShashwathiNavada closed https://github.com/llvm/llvm-project/pull/206642

From llvmlistbot at llvm.org  Mon Jul 13 02:10:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:10:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a54ab9d.9edb5c9d.208698.5912@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 3a9b774ca84ccb1395764dac6683a553c6e751e8 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH] [mlir][memref] Avoid mem2reg crash on element count overflow

Avoid crash in test alias analysis operand printing

Fix AffineForOP creash

remove

fix

fix issue
---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;


From llvmlistbot at llvm.org  Mon Jul 13 02:21:01 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 13 Jul 2026 02:21:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54adfd.8e75e0cb.3dbce4.c230@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206950

>From 187cd538f98c32b897f1d8839a38a3b6915917a1 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 13:12:30 +0200
Subject: [PATCH 1/3] [mlir][SPIR-V] Add SPIRVToLLVM conversion for SNegate

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 31 ++++++++++++++++++-
 .../SPIRVToLLVM/arithmetic-ops-to-llvm.mlir   | 20 ++++++++++++
 2 files changed, 50 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..610c052b64ea7 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -921,6 +921,35 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero;
+    if (vecSrcType)
+      zero = LLVM::ConstantOp::create(
+          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
+    else
+      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts `spirv.Load` and `spirv.Store` to LLVM dialect.
 template <typename SPIRVOp>
 class LoadStorePattern : public SPIRVToLLVMConversion<SPIRVOp> {
@@ -1824,7 +1853,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}

>From 204b7a856af639ca4bb46501848cabdcef02e393 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 10 Jul 2026 13:45:23 +0200
Subject: [PATCH 2/3] Address comments

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 34 +++++++++----------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 96acea0f4f127..a8bee01b08d26 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -90,17 +90,22 @@ static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
   return builder.getIntegerAttr(integerType, -1);
 }
 
+/// Creates `llvm.mlir.constant` with a scalar or vector integer value,
+/// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
+static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
+                                   PatternRewriter &rewriter,
+                                   IntegerAttr scalarAttr) {
+  if (auto vecType = dyn_cast<VectorType>(srcType))
+    return LLVM::ConstantOp::create(
+        rewriter, loc, dstType, SplatElementsAttr::get(vecType, scalarAttr));
+  return LLVM::ConstantOp::create(rewriter, loc, dstType, scalarAttr);
+}
+
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
-  if (isa<VectorType>(srcType)) {
-    return LLVM::ConstantOp::create(
-        rewriter, loc, dstType,
-        SplatElementsAttr::get(cast<ShapedType>(srcType),
-                               minusOneIntegerAttribute(srcType, rewriter)));
-  }
-  return LLVM::ConstantOp::create(rewriter, loc, dstType,
-                                  minusOneIntegerAttribute(srcType, rewriter));
+  return createIntegerConstant(loc, srcType, dstType, rewriter,
+                               minusOneIntegerAttribute(srcType, rewriter));
 }
 
 /// Creates `llvm.mlir.constant` with a floating-point scalar or vector value.
@@ -929,21 +934,16 @@ class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
   LogicalResult
   matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto srcType = op.getType();
-    auto dstType = getTypeConverter()->convertType(srcType);
+    Type srcType = op.getType();
+    Type dstType = getTypeConverter()->convertType(srcType);
     if (!dstType)
       return rewriter.notifyMatchFailure(op, "type conversion failed");
 
     Location loc = op.getLoc();
-    auto vecSrcType = dyn_cast<VectorType>(srcType);
     IntegerAttr zeroAttr = rewriter.getIntegerAttr(
         cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
-    Value zero;
-    if (vecSrcType)
-      zero = LLVM::ConstantOp::create(
-          rewriter, loc, dstType, SplatElementsAttr::get(vecSrcType, zeroAttr));
-    else
-      zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+    Value zero =
+        createIntegerConstant(loc, srcType, dstType, rewriter, zeroAttr);
     rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
                                              adaptor.getOperand());
     return success();

>From 5db1760d4cf1fa7e6be63fea92b68add78361a74 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Mon, 13 Jul 2026 11:20:50 +0200
Subject: [PATCH 3/3] Address comment

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 23 ++++---------------
 1 file changed, 5 insertions(+), 18 deletions(-)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index a8bee01b08d26..96669e609b246 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -80,16 +80,6 @@ static unsigned getLLVMTypeBitWidth(Type type) {
   return cast<IntegerType>(type).getWidth();
 }
 
-/// Creates `IntegerAttribute` with all bits set for given type
-static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
-  if (auto vecType = dyn_cast<VectorType>(type)) {
-    auto integerType = cast<IntegerType>(vecType.getElementType());
-    return builder.getIntegerAttr(integerType, -1);
-  }
-  auto integerType = cast<IntegerType>(type);
-  return builder.getIntegerAttr(integerType, -1);
-}
-
 /// Creates `llvm.mlir.constant` with a scalar or vector integer value,
 /// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
 static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
@@ -104,8 +94,11 @@ static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
+  auto integerType = cast<IntegerType>(
+      isa<VectorType>(srcType) ? cast<VectorType>(srcType).getElementType()
+                               : srcType);
   return createIntegerConstant(loc, srcType, dstType, rewriter,
-                               minusOneIntegerAttribute(srcType, rewriter));
+                               rewriter.getIntegerAttr(integerType, -1));
 }
 
 /// Creates `llvm.mlir.constant` with a floating-point scalar or vector value.
@@ -1024,13 +1017,7 @@ class NotPattern : public SPIRVToLLVMConversion<SPIRVOp> {
       return rewriter.notifyMatchFailure(notOp, "type conversion failed");
 
     Location loc = notOp.getLoc();
-    IntegerAttr minusOne = minusOneIntegerAttribute(srcType, rewriter);
-    auto mask =
-        isa<VectorType>(srcType)
-            ? LLVM::ConstantOp::create(
-                  rewriter, loc, dstType,
-                  SplatElementsAttr::get(cast<VectorType>(srcType), minusOne))
-            : LLVM::ConstantOp::create(rewriter, loc, dstType, minusOne);
+    Value mask = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
     rewriter.template replaceOpWithNewOp<LLVM::XOrOp>(notOp, dstType,
                                                       notOp.getOperand(), mask);
     return success();


From llvmlistbot at llvm.org  Mon Jul 13 02:21:14 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 13 Jul 2026 02:21:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54ae0a.824e11a6.f5f27.843a@mx.google.com>


================
@@ -90,17 +90,22 @@ static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
   return builder.getIntegerAttr(integerType, -1);
 }
 
+/// Creates `llvm.mlir.constant` with a scalar or vector integer value,
+/// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
+static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
+                                   PatternRewriter &rewriter,
+                                   IntegerAttr scalarAttr) {
+  if (auto vecType = dyn_cast<VectorType>(srcType))
+    return LLVM::ConstantOp::create(
+        rewriter, loc, dstType, SplatElementsAttr::get(vecType, scalarAttr));
+  return LLVM::ConstantOp::create(rewriter, loc, dstType, scalarAttr);
+}
+
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
-  if (isa<VectorType>(srcType)) {
-    return LLVM::ConstantOp::create(
-        rewriter, loc, dstType,
-        SplatElementsAttr::get(cast<ShapedType>(srcType),
-                               minusOneIntegerAttribute(srcType, rewriter)));
-  }
-  return LLVM::ConstantOp::create(rewriter, loc, dstType,
-                                  minusOneIntegerAttribute(srcType, rewriter));
+  return createIntegerConstant(loc, srcType, dstType, rewriter,
+                               minusOneIntegerAttribute(srcType, rewriter));
----------------
aobolensk wrote:

Done

https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Mon Jul 13 02:27:43 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 02:27:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54af8f.743400b0.2067c9.11c3@mx.google.com>


================

----------------
hockyy wrote:

can we just use this
```mlir
#sparse = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
module {
  func.func @main() -> tensor<77xi1, #sparse> {
    %0 = tensor.empty() : tensor<77xi1, #sparse>
    return %0 : tensor<77xi1, #sparse>
  }
}
```
the rest lgtm

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 02:28:07 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 02:28:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54afa7.c574b9b4.21ead0.64ea@mx.google.com>


https://github.com/hockyy edited https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 02:31:06 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 02:31:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a54b05a.824e11a6.f5f27.9b70@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/207851

>From 11a36a87f72267a4ccf12afc53ed410cfe73eb03 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 3 Jun 2026 17:06:26 +0100
Subject: [PATCH 1/2] [mlir][tosa] Add support for block scaled types in
 matmul_t

Builds on https://github.com/llvm/llvm-project/pull/203894
to enable MXFP block scaled fotmats within matmul_t.

Change-Id: I3674ee6315f56934409c11d02d3063eb054aa724
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 561 ++++++++++++++++--
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |   8 +-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |  22 +-
 .../Tosa/Transforms/TosaDowngrade1p1To1p0.cpp |   3 +
 .../Tosa/Transforms/TosaProfileCompliance.cpp |   2 +
 mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp    |  16 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  11 +
 mlir/test/Dialect/Tosa/ops.mlir               |  27 +
 .../Tosa/tosa-downgrade-1-1-to-1-0.mlir       |  10 +
 .../tosa-validation-version-1p1-valid.mlir    |  30 +
 mlir/test/Dialect/Tosa/verifier.mlir          |  10 +
 11 files changed, 646 insertions(+), 54 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..c1fb71110d4a1 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
      {{{Profile::pro_int}, {{{i8T, i8T}, SpecificationVersion::V_1_0}}},
       {{Profile::pro_fp},
@@ -696,52 +699,528 @@ extensionComplianceMap = {
        {{{bf16T, bf16T, bf16T, bf16T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Extension::int16},
-       {{{i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i16T, i16T, i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
-       {{{fp8e4m3T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3, Extension::fp8e5m2},
-       {{{fp8e4m3T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16, Extension::fp8e4m3},
-       {{{fp8e4m3T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::fp8e5m2},
-       {{{fp8e5m2T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16, Extension::fp8e5m2},
-       {{{fp8e5m2T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16},
-       {{{fp16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.matmul_t_block_scaled",
      {{{Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, fp4e2m1T, fp8ue8m0T, fp32T},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..9d36477ba6758 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -475,7 +475,9 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
   list<Availability> availability =
       [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
        Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
+                  Tosa_EXT_BF16, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -494,6 +496,8 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: matmul_t_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled", [NoMemoryEffect]> {
   let summary = "Performs two dimensional matrix multiplications using block scaled tensors.";
@@ -506,6 +510,8 @@ def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled",
 
     This operation is not pure. Undefined behaviour may occur if the accumulated
     result overflows.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..3d0f10a27353d 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -2128,16 +2128,26 @@ template <typename T>
 static LogicalResult verifyMatMulZeroPointType(T op, Value input, Value zp,
                                                StringRef inputName,
                                                StringRef zpName) {
+  const Type inputElementType = getElementTypeOrSelf(input.getType());
   const Type inputStorageElementType = getStorageElementTypeOrSelf(input);
   const Type zpElementType = getStorageElementTypeOrSelf(zp);
+  Type expectedElementType = inputStorageElementType;
 
-  if (inputStorageElementType != zpElementType)
-    return op.emitOpError("expect input ")
-           << inputName << " and " << zpName
-           << " have the same element type, got " << inputStorageElementType
-           << " and " << zpElementType;
+  if (isa<BlockScaledType>(inputElementType))
+    expectedElementType = Float32Type::get(op.getContext());
 
-  return success();
+  if (expectedElementType == zpElementType)
+    return success();
+
+  InFlightDiagnostic diag = op.emitOpError("expect input ");
+  diag << inputName << " and " << zpName;
+  if (isa<BlockScaledType>(inputElementType))
+    diag << " have compatible element types, got " << inputElementType
+         << " and " << zpElementType;
+  else
+    diag << " have the same element type, got " << inputStorageElementType
+         << " and " << zpElementType;
+  return diag;
 }
 
 LogicalResult MatMulOp::verify() {
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
index 163850f741166..2bc99d7fab8a4 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
@@ -151,6 +151,9 @@ static LogicalResult isMatMulTTypeCompatibleForDowngrade(tosa::MatMulTOp op) {
   if (aElementType != bElementType)
     return failure();
 
+  if (isa<BlockScaledType>(aElementType) || isa<BlockScaledType>(bElementType))
+    return failure();
+
   if ((aElementType.isF16() && outputElementType.isF16()) ||
       (aElementType.isF16() && outputElementType.isF32()) ||
       (aElementType.isF32() && outputElementType.isF32()) ||
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 348e4ec0ad44c..7ea64385c2415 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -291,6 +291,8 @@ template <>
 LogicalResult ProfileInfoDepot::populateProfileInfo(tosa::MatMulTOp op) {
   addValue(op.getA());
   addValue(op.getB());
+  addValue(op.getAZp());
+  addValue(op.getBZp());
   addValue(op.getOutput());
   return success();
 }
diff --git a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
index c55b13dc98cc5..b3fe95eefe13f 100644
--- a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
+++ b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
@@ -162,10 +162,14 @@ std::pair<Value, Value>
 mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   std::int64_t inputZp, weightZp;
 
-  auto inputEType = getElementTypeOrSelf(input.getType());
-  auto weightEType = getElementTypeOrSelf(weight.getType());
-
-  if (mlir::isa<FloatType>(inputEType) && mlir::isa<FloatType>(weightEType)) {
+  Type inputZpType = getElementTypeOrSelf(input.getType());
+  if (isa<BlockScaledType>(inputZpType))
+    inputZpType = builder.getF32Type();
+  Type weightZpType = getElementTypeOrSelf(weight.getType());
+  if (isa<BlockScaledType>(weightZpType))
+    weightZpType = builder.getF32Type();
+
+  if (mlir::isa<FloatType>(inputZpType) && mlir::isa<FloatType>(weightZpType)) {
     inputZp = 0;
     weightZp = 0;
   } else {
@@ -178,12 +182,12 @@ mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   }
 
   auto maybeInputZpValue =
-      createZeroPointTensor(builder, input.getLoc(), inputEType, inputZp);
+      createZeroPointTensor(builder, input.getLoc(), inputZpType, inputZp);
   if (!maybeInputZpValue.has_value())
     return {};
 
   auto maybeWeightZpValue =
-      createZeroPointTensor(builder, weight.getLoc(), weightEType, weightZp);
+      createZeroPointTensor(builder, weight.getLoc(), weightZpType, weightZp);
   if (!maybeWeightZpValue.has_value())
     return {};
 
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..c68487169da63 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -332,6 +332,17 @@ func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x
 }
 
 // -----
+
+func.func @test_matmul_t_mixed_block_scaled_operands(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.matmul_t' op illegal: requires all of [mx_common, mx_fp6e2m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}
   %0, %1 = tosa.fft2d %arg0, %arg1 {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..41d87410e2950 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -286,6 +286,15 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
   return %0 : tensor<2x14x28xf16>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_block_scaled
+func.func @test_matmul_t_block_scaled(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<2x14x28xf32>
+  return %0 : tensor<2x14x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_dynamic_dims
 func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<?x28x19xf32>) -> tensor<?x14x28xf32> {
@@ -295,6 +304,15 @@ func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<
   return %0 : tensor<?x14x28xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_dynamic_block_scaled_broadcast
+func.func @test_matmul_t_dynamic_block_scaled_broadcast(%arg0: tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<?x?x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<?x?x28xf32>
+  return %0 : tensor<?x?x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_static_broadcast
 func.func @test_matmul_t_static_broadcast(%arg0: tensor<4x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<4x14x28xf32> {
@@ -313,6 +331,15 @@ func.func @test_matmul_t_unranked(%arg0: tensor<*xf32>, %arg1: tensor<*xf32>) ->
   return %0 : tensor<*xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_unranked_block_scaled
+func.func @test_matmul_t_unranked_block_scaled(%arg0: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<*xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<*xf32>
+  return %0 : tensor<*xf32>
+}
+
 // -----
 // CHECK-LABEL: max_pool2d_f32
 func.func @test_max_pool2d_f32(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
index 378e5eb7b0147..1532ec91d53e8 100644
--- a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
@@ -150,3 +150,13 @@ func.func @test_preserve_matmul_t_dynamic_broadcast(%arg0: tensor<4x14x19xf32>,
   %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<4x14x19xf32>, tensor<?x28x19xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x14x28xf32>
   return %0 : tensor<4x14x28xf32>
 }
+
+// -----
+
+// CHECK-LABEL: @test_preserve_matmul_t_block_scaled
+// CHECK: %[[MATMUL_T:.+]] = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+// CHECK: return %[[MATMUL_T]]
+func.func @test_preserve_matmul_t_block_scaled(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg2: tensor<1xf32>, %arg3: tensor<1xf32>) -> tensor<1x14x28xf32> {
+  %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9ccf5dea8daf0 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -32,6 +32,36 @@ func.func @test_matmul_t_fp8_input_fp32_acc_type(%arg0: tensor<1x14x19xf8E4M3FN>
 
 // -----
 
+// CHECK-LABEL: test_matmul_t_block_scaled_type_fp8e4m3
+func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3
+func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_f16_block_scaled_type_fp6e3m2
+func.func @test_matmul_t_f16_block_scaled_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 // CHECK-LABEL: test_conv2d_fp8_acc32
 func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x1x1x4xf8E5M2>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..c2661e58d5586 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1488,6 +1488,16 @@ func.func @test_matmul_t_a_zp_same_element_type(%arg0: tensor<1x14x19xf32>, %arg
 
 // -----
 
+func.func @test_matmul_t_b_zp_block_scaled_requires_fp32(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+// expected-error at +1 {{'tosa.matmul_t' op expect input b and b_zp have compatible element types, got '!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>' and 'f16'}}
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf16>)  -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}
+
+// -----
+
 func.func @test_matmul_t_a_zp_non_zero(%arg0: tensor<1x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<1x14x28xf32> {
 %azp0 = "tosa.const"() <{values = dense<1.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>

>From 5df3ded0ccc3b88de60c2c66c4486c7df49acc53 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 13 Jul 2026 10:30:10 +0100
Subject: [PATCH 2/2] Address review comments

* Rename unit tests
* Fixup profile compliance

Change-Id: I497af9a9c6d9557dcb372738011f88522e6bcff5
---
 .../mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc  | 14 ++++++--------
 mlir/test/Dialect/Tosa/ops.mlir                    |  4 ++--
 .../Tosa/tosa-validation-version-1p1-valid.mlir    | 12 ++++++------
 3 files changed, 14 insertions(+), 16 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index c1fb71110d4a1..d16444b767fb5 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -64,12 +64,12 @@ profileComplianceMap = {
      {{{Profile::pro_int},
        {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
-         SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp16T, fp16T, fp32T},
-         SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T, fp32T, fp32T},
-         SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT}, 
+        {{fp16T, fp16T, fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}, 
+        {{fp16T, fp32T, fp16T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}, 
+        {{fp32T, fp16T, fp32T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},  
+        {{fp16T, fp16T, fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}, 
+        {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
      {{{Profile::pro_int}, {{{i8T, i8T}, SpecificationVersion::V_1_0}}},
       {{Profile::pro_fp},
@@ -895,8 +895,6 @@ extensionComplianceMap = {
          SpecificationVersion::V_1_1_DRAFT},
         {{bf16T, fp16T, bf16T, fp16T, fp32T},
          SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, bf16T, bf16T, fp32T},
-         SpecificationVersion::V_1_1_DRAFT},
         {{bf16T, fp32T, bf16T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT},
         {{fp32T, bf16T, fp32T, bf16T, fp32T},
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 41d87410e2950..5cc03ab206975 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -287,8 +287,8 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
 }
 
 // -----
-// CHECK-LABEL: test_matmul_t_block_scaled
-func.func @test_matmul_t_block_scaled(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs
+func.func @test_matmul_t_with_block_scaled_inputs(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
 %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<2x14x28xf32>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 9ccf5dea8daf0..a23f1bc149a47 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -32,8 +32,8 @@ func.func @test_matmul_t_fp8_input_fp32_acc_type(%arg0: tensor<1x14x19xf8E4M3FN>
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_block_scaled_type_fp8e4m3
-func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp8e4m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
@@ -42,8 +42,8 @@ func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.bl
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3
-func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
@@ -52,8 +52,8 @@ func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_f16_block_scaled_type_fp6e3m2
-func.func @test_matmul_t_f16_block_scaled_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
+// CHECK-LABEL: test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2
+func.func @test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>


From llvmlistbot at llvm.org  Mon Jul 13 02:31:39 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 02:31:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a54b07b.b1416da1.78d1e.7a74@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/207851

>From 11a36a87f72267a4ccf12afc53ed410cfe73eb03 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 3 Jun 2026 17:06:26 +0100
Subject: [PATCH 1/2] [mlir][tosa] Add support for block scaled types in
 matmul_t

Builds on https://github.com/llvm/llvm-project/pull/203894
to enable MXFP block scaled fotmats within matmul_t.

Change-Id: I3674ee6315f56934409c11d02d3063eb054aa724
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 561 ++++++++++++++++--
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |   8 +-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |  22 +-
 .../Tosa/Transforms/TosaDowngrade1p1To1p0.cpp |   3 +
 .../Tosa/Transforms/TosaProfileCompliance.cpp |   2 +
 mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp    |  16 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  11 +
 mlir/test/Dialect/Tosa/ops.mlir               |  27 +
 .../Tosa/tosa-downgrade-1-1-to-1-0.mlir       |  10 +
 .../tosa-validation-version-1p1-valid.mlir    |  30 +
 mlir/test/Dialect/Tosa/verifier.mlir          |  10 +
 11 files changed, 646 insertions(+), 54 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..c1fb71110d4a1 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
      {{{Profile::pro_int}, {{{i8T, i8T}, SpecificationVersion::V_1_0}}},
       {{Profile::pro_fp},
@@ -696,52 +699,528 @@ extensionComplianceMap = {
        {{{bf16T, bf16T, bf16T, bf16T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Extension::int16},
-       {{{i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i16T, i16T, i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
-       {{{fp8e4m3T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3, Extension::fp8e5m2},
-       {{{fp8e4m3T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16, Extension::fp8e4m3},
-       {{{fp8e4m3T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::fp8e5m2},
-       {{{fp8e5m2T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16, Extension::fp8e5m2},
-       {{{fp8e5m2T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16},
-       {{{fp16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.matmul_t_block_scaled",
      {{{Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, fp4e2m1T, fp8ue8m0T, fp32T},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..9d36477ba6758 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -475,7 +475,9 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
   list<Availability> availability =
       [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
        Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
+                  Tosa_EXT_BF16, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -494,6 +496,8 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: matmul_t_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled", [NoMemoryEffect]> {
   let summary = "Performs two dimensional matrix multiplications using block scaled tensors.";
@@ -506,6 +510,8 @@ def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled",
 
     This operation is not pure. Undefined behaviour may occur if the accumulated
     result overflows.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..3d0f10a27353d 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -2128,16 +2128,26 @@ template <typename T>
 static LogicalResult verifyMatMulZeroPointType(T op, Value input, Value zp,
                                                StringRef inputName,
                                                StringRef zpName) {
+  const Type inputElementType = getElementTypeOrSelf(input.getType());
   const Type inputStorageElementType = getStorageElementTypeOrSelf(input);
   const Type zpElementType = getStorageElementTypeOrSelf(zp);
+  Type expectedElementType = inputStorageElementType;
 
-  if (inputStorageElementType != zpElementType)
-    return op.emitOpError("expect input ")
-           << inputName << " and " << zpName
-           << " have the same element type, got " << inputStorageElementType
-           << " and " << zpElementType;
+  if (isa<BlockScaledType>(inputElementType))
+    expectedElementType = Float32Type::get(op.getContext());
 
-  return success();
+  if (expectedElementType == zpElementType)
+    return success();
+
+  InFlightDiagnostic diag = op.emitOpError("expect input ");
+  diag << inputName << " and " << zpName;
+  if (isa<BlockScaledType>(inputElementType))
+    diag << " have compatible element types, got " << inputElementType
+         << " and " << zpElementType;
+  else
+    diag << " have the same element type, got " << inputStorageElementType
+         << " and " << zpElementType;
+  return diag;
 }
 
 LogicalResult MatMulOp::verify() {
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
index 163850f741166..2bc99d7fab8a4 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
@@ -151,6 +151,9 @@ static LogicalResult isMatMulTTypeCompatibleForDowngrade(tosa::MatMulTOp op) {
   if (aElementType != bElementType)
     return failure();
 
+  if (isa<BlockScaledType>(aElementType) || isa<BlockScaledType>(bElementType))
+    return failure();
+
   if ((aElementType.isF16() && outputElementType.isF16()) ||
       (aElementType.isF16() && outputElementType.isF32()) ||
       (aElementType.isF32() && outputElementType.isF32()) ||
diff --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 348e4ec0ad44c..7ea64385c2415 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -291,6 +291,8 @@ template <>
 LogicalResult ProfileInfoDepot::populateProfileInfo(tosa::MatMulTOp op) {
   addValue(op.getA());
   addValue(op.getB());
+  addValue(op.getAZp());
+  addValue(op.getBZp());
   addValue(op.getOutput());
   return success();
 }
diff --git a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
index c55b13dc98cc5..b3fe95eefe13f 100644
--- a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
+++ b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
@@ -162,10 +162,14 @@ std::pair<Value, Value>
 mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   std::int64_t inputZp, weightZp;
 
-  auto inputEType = getElementTypeOrSelf(input.getType());
-  auto weightEType = getElementTypeOrSelf(weight.getType());
-
-  if (mlir::isa<FloatType>(inputEType) && mlir::isa<FloatType>(weightEType)) {
+  Type inputZpType = getElementTypeOrSelf(input.getType());
+  if (isa<BlockScaledType>(inputZpType))
+    inputZpType = builder.getF32Type();
+  Type weightZpType = getElementTypeOrSelf(weight.getType());
+  if (isa<BlockScaledType>(weightZpType))
+    weightZpType = builder.getF32Type();
+
+  if (mlir::isa<FloatType>(inputZpType) && mlir::isa<FloatType>(weightZpType)) {
     inputZp = 0;
     weightZp = 0;
   } else {
@@ -178,12 +182,12 @@ mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   }
 
   auto maybeInputZpValue =
-      createZeroPointTensor(builder, input.getLoc(), inputEType, inputZp);
+      createZeroPointTensor(builder, input.getLoc(), inputZpType, inputZp);
   if (!maybeInputZpValue.has_value())
     return {};
 
   auto maybeWeightZpValue =
-      createZeroPointTensor(builder, weight.getLoc(), weightEType, weightZp);
+      createZeroPointTensor(builder, weight.getLoc(), weightZpType, weightZp);
   if (!maybeWeightZpValue.has_value())
     return {};
 
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..c68487169da63 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -332,6 +332,17 @@ func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x
 }
 
 // -----
+
+func.func @test_matmul_t_mixed_block_scaled_operands(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.matmul_t' op illegal: requires all of [mx_common, mx_fp6e2m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}
   %0, %1 = tosa.fft2d %arg0, %arg1 {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..41d87410e2950 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -286,6 +286,15 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
   return %0 : tensor<2x14x28xf16>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_block_scaled
+func.func @test_matmul_t_block_scaled(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<2x14x28xf32>
+  return %0 : tensor<2x14x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_dynamic_dims
 func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<?x28x19xf32>) -> tensor<?x14x28xf32> {
@@ -295,6 +304,15 @@ func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<
   return %0 : tensor<?x14x28xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_dynamic_block_scaled_broadcast
+func.func @test_matmul_t_dynamic_block_scaled_broadcast(%arg0: tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<?x?x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<?x?x28xf32>
+  return %0 : tensor<?x?x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_static_broadcast
 func.func @test_matmul_t_static_broadcast(%arg0: tensor<4x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<4x14x28xf32> {
@@ -313,6 +331,15 @@ func.func @test_matmul_t_unranked(%arg0: tensor<*xf32>, %arg1: tensor<*xf32>) ->
   return %0 : tensor<*xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_unranked_block_scaled
+func.func @test_matmul_t_unranked_block_scaled(%arg0: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<*xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<*xf32>
+  return %0 : tensor<*xf32>
+}
+
 // -----
 // CHECK-LABEL: max_pool2d_f32
 func.func @test_max_pool2d_f32(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
index 378e5eb7b0147..1532ec91d53e8 100644
--- a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
@@ -150,3 +150,13 @@ func.func @test_preserve_matmul_t_dynamic_broadcast(%arg0: tensor<4x14x19xf32>,
   %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<4x14x19xf32>, tensor<?x28x19xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x14x28xf32>
   return %0 : tensor<4x14x28xf32>
 }
+
+// -----
+
+// CHECK-LABEL: @test_preserve_matmul_t_block_scaled
+// CHECK: %[[MATMUL_T:.+]] = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+// CHECK: return %[[MATMUL_T]]
+func.func @test_preserve_matmul_t_block_scaled(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg2: tensor<1xf32>, %arg3: tensor<1xf32>) -> tensor<1x14x28xf32> {
+  %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9ccf5dea8daf0 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -32,6 +32,36 @@ func.func @test_matmul_t_fp8_input_fp32_acc_type(%arg0: tensor<1x14x19xf8E4M3FN>
 
 // -----
 
+// CHECK-LABEL: test_matmul_t_block_scaled_type_fp8e4m3
+func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3
+func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_f16_block_scaled_type_fp6e3m2
+func.func @test_matmul_t_f16_block_scaled_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 // CHECK-LABEL: test_conv2d_fp8_acc32
 func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x1x1x4xf8E5M2>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..c2661e58d5586 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1488,6 +1488,16 @@ func.func @test_matmul_t_a_zp_same_element_type(%arg0: tensor<1x14x19xf32>, %arg
 
 // -----
 
+func.func @test_matmul_t_b_zp_block_scaled_requires_fp32(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+// expected-error at +1 {{'tosa.matmul_t' op expect input b and b_zp have compatible element types, got '!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>' and 'f16'}}
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf16>)  -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}
+
+// -----
+
 func.func @test_matmul_t_a_zp_non_zero(%arg0: tensor<1x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<1x14x28xf32> {
 %azp0 = "tosa.const"() <{values = dense<1.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>

>From c4a62bd81e8d64e09a44b79a8a0c083da6ac5b26 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 13 Jul 2026 10:30:10 +0100
Subject: [PATCH 2/2] Address review comments

* Rename unit tests
* Fixup profile compliance

Change-Id: I497af9a9c6d9557dcb372738011f88522e6bcff5
---
 .../mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc    |  8 ++++++--
 mlir/test/Dialect/Tosa/ops.mlir                      |  4 ++--
 .../Tosa/tosa-validation-version-1p1-valid.mlir      | 12 ++++++------
 3 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index c1fb71110d4a1..803a2535bab7f 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -68,6 +68,12 @@ profileComplianceMap = {
          SpecificationVersion::V_1_1_DRAFT},
         {{fp16T, fp16T, fp16T, fp16T, fp32T},
          SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp32T, fp32T, fp32T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
@@ -895,8 +901,6 @@ extensionComplianceMap = {
          SpecificationVersion::V_1_1_DRAFT},
         {{bf16T, fp16T, bf16T, fp16T, fp32T},
          SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, bf16T, bf16T, fp32T},
-         SpecificationVersion::V_1_1_DRAFT},
         {{bf16T, fp32T, bf16T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT},
         {{fp32T, bf16T, fp32T, bf16T, fp32T},
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 41d87410e2950..5cc03ab206975 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -287,8 +287,8 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
 }
 
 // -----
-// CHECK-LABEL: test_matmul_t_block_scaled
-func.func @test_matmul_t_block_scaled(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs
+func.func @test_matmul_t_with_block_scaled_inputs(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
 %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<2x14x28xf32>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 9ccf5dea8daf0..a23f1bc149a47 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -32,8 +32,8 @@ func.func @test_matmul_t_fp8_input_fp32_acc_type(%arg0: tensor<1x14x19xf8E4M3FN>
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_block_scaled_type_fp8e4m3
-func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp8e4m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
@@ -42,8 +42,8 @@ func.func @test_matmul_t_block_scaled_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.bl
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3
-func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
@@ -52,8 +52,8 @@ func.func @test_matmul_t_block_scaled_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x
 
 // -----
 
-// CHECK-LABEL: test_matmul_t_f16_block_scaled_type_fp6e3m2
-func.func @test_matmul_t_f16_block_scaled_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
+// CHECK-LABEL: test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2
+func.func @test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
   %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
   %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>


From llvmlistbot at llvm.org  Mon Jul 13 02:35:48 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 02:35:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a54b174.a1accbb4.244acd.f241@mx.google.com>


https://github.com/lhutton1 approved this pull request.

LGTM thanks @mygitljf!

https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Mon Jul 13 02:44:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 02:44:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] dc8ee15 - [mlir][SPIR-V] Add SPIRVToLLVM
 conversion for SNegate (#206950)
Message-ID: <6a54b382.510f576d.1b3cb9.5fe7@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-13T11:44:29+02:00
New Revision: dc8ee15f2bbc5b18316f071c5ae746bd95609ade

URL: https://github.com/llvm/llvm-project/commit/dc8ee15f2bbc5b18316f071c5ae746bd95609ade
DIFF: https://github.com/llvm/llvm-project/commit/dc8ee15f2bbc5b18316f071c5ae746bd95609ade.diff

LOG: [mlir][SPIR-V] Add SPIRVToLLVM conversion for SNegate (#206950)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index 0822c44c10f1d..f8c4405157f94 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -80,27 +80,25 @@ static unsigned getLLVMTypeBitWidth(Type type) {
   return cast<IntegerType>(type).getWidth();
 }
 
-/// Creates `IntegerAttribute` with all bits set for given type
-static IntegerAttr minusOneIntegerAttribute(Type type, Builder builder) {
-  if (auto vecType = dyn_cast<VectorType>(type)) {
-    auto integerType = cast<IntegerType>(vecType.getElementType());
-    return builder.getIntegerAttr(integerType, -1);
-  }
-  auto integerType = cast<IntegerType>(type);
-  return builder.getIntegerAttr(integerType, -1);
+/// Creates `llvm.mlir.constant` with a scalar or vector integer value,
+/// broadcasting `scalarAttr` across the vector if `srcType` is a vector.
+static Value createIntegerConstant(Location loc, Type srcType, Type dstType,
+                                   PatternRewriter &rewriter,
+                                   IntegerAttr scalarAttr) {
+  if (auto vecType = dyn_cast<VectorType>(srcType))
+    return LLVM::ConstantOp::create(
+        rewriter, loc, dstType, SplatElementsAttr::get(vecType, scalarAttr));
+  return LLVM::ConstantOp::create(rewriter, loc, dstType, scalarAttr);
 }
 
 /// Creates `llvm.mlir.constant` with all bits set for the given type.
 static Value createConstantAllBitsSet(Location loc, Type srcType, Type dstType,
                                       PatternRewriter &rewriter) {
-  if (isa<VectorType>(srcType)) {
-    return LLVM::ConstantOp::create(
-        rewriter, loc, dstType,
-        SplatElementsAttr::get(cast<ShapedType>(srcType),
-                               minusOneIntegerAttribute(srcType, rewriter)));
-  }
-  return LLVM::ConstantOp::create(rewriter, loc, dstType,
-                                  minusOneIntegerAttribute(srcType, rewriter));
+  auto integerType = cast<IntegerType>(
+      isa<VectorType>(srcType) ? cast<VectorType>(srcType).getElementType()
+                               : srcType);
+  return createIntegerConstant(loc, srcType, dstType, rewriter,
+                               rewriter.getIntegerAttr(integerType, -1));
 }
 
 /// Creates `llvm.mlir.constant` with a floating-point scalar or vector value.
@@ -921,6 +919,30 @@ class InverseSqrtPattern
   }
 };
 
+/// Converts `spirv.SNegate` to `0 - x`.
+class SNegatePattern : public SPIRVToLLVMConversion<spirv::SNegateOp> {
+public:
+  using SPIRVToLLVMConversion<spirv::SNegateOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(spirv::SNegateOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Type srcType = op.getType();
+    Type dstType = getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    IntegerAttr zeroAttr = rewriter.getIntegerAttr(
+        cast<IntegerType>(getElementTypeOrSelf(srcType)), 0);
+    Value zero =
+        createIntegerConstant(loc, srcType, dstType, rewriter, zeroAttr);
+    rewriter.replaceOpWithNewOp<LLVM::SubOp>(op, dstType, zero,
+                                             adaptor.getOperand());
+    return success();
+  }
+};
+
 /// Converts the GLSL clamp ops (FClamp, SClamp, UClamp) into a nested
 /// min/max sequence, following the op semantics `min(max(x, minVal), maxVal)`.
 template <typename SPIRVOp, typename LLVMMinOp, typename LLVMMaxOp>
@@ -995,13 +1017,7 @@ class NotPattern : public SPIRVToLLVMConversion<SPIRVOp> {
       return rewriter.notifyMatchFailure(notOp, "type conversion failed");
 
     Location loc = notOp.getLoc();
-    IntegerAttr minusOne = minusOneIntegerAttribute(srcType, rewriter);
-    auto mask =
-        isa<VectorType>(srcType)
-            ? LLVM::ConstantOp::create(
-                  rewriter, loc, dstType,
-                  SplatElementsAttr::get(cast<VectorType>(srcType), minusOne))
-            : LLVM::ConstantOp::create(rewriter, loc, dstType, minusOne);
+    Value mask = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
     rewriter.template replaceOpWithNewOp<LLVM::XOrOp>(notOp, dstType,
                                                       notOp.getOperand(), mask);
     return success();
@@ -1957,7 +1973,7 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::SDivOp, LLVM::SDivOp>,
       DirectConversionPattern<spirv::SRemOp, LLVM::SRemOp>,
       DirectConversionPattern<spirv::UDivOp, LLVM::UDivOp>,
-      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>,
+      DirectConversionPattern<spirv::UModOp, LLVM::URemOp>, SNegatePattern,
 
       // Bitwise ops
       BitFieldInsertPattern, BitFieldUExtractPattern, BitFieldSExtractPattern,

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
index dbbf8610afb4d..ee9c06fbb10e1 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/arithmetic-ops-to-llvm.mlir
@@ -233,3 +233,23 @@ spirv.func @srem_vector(%arg0: vector<4xi32>, %arg1: vector<4xi32>) "None" {
   %0 = spirv.SRem %arg0, %arg1 : vector<4xi32>
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.SNegate
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @snegate_scalar
+spirv.func @snegate_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : i32
+  %0 = spirv.SNegate %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @snegate_vector
+spirv.func @snegate_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(dense<0> : vector<4xi32>) : vector<4xi32>
+  // CHECK: llvm.sub %[[ZERO]], %{{.*}} : vector<4xi32>
+  %0 = spirv.SNegate %arg0 : vector<4xi32>
+  spirv.Return
+}


        

From llvmlistbot at llvm.org  Mon Jul 13 02:44:37 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 13 Jul 2026 02:44:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversion for
	SNegate (PR #206950)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206950@github.com>
Message-ID: <6a54b385.510f576d.1b3cb9.6211@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/206950

From llvmlistbot at llvm.org  Mon Jul 13 02:46:19 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 02:46:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a54b3eb.13be6df8.145c.7c45@mx.google.com>


================
@@ -62,11 +62,14 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
----------------
lhutton1 wrote:

I've corrected this manually for now, but will look at fixing the generation in the specification shortly

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Mon Jul 13 03:00:28 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 03:00:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54b73c.92d2873c.258297.8bf9@mx.google.com>


================

----------------
hockyy wrote:

I just generated some other ops btw, it seems like these are failing too
```

// SparseTensorAllocConverter
func.func @alloc_tensor_i1_elem_type(%sz: index) -> tensor<?xi1, #sparse1d> {
  %0 = bufferization.alloc_tensor(%sz) : tensor<?xi1, #sparse1d>
  return %0 : tensor<?xi1, #sparse1d>
}

// SparseTensorReorderCOOConverter
func.func @reorder_coo_i1_elem_type(%arg0: tensor<?x?xi1, #unordered_coo>)
    -> tensor<?x?xi1, #ordered_coo> {
  %0 = sparse_tensor.reorder_coo quick_sort %arg0
    : tensor<?x?xi1, #unordered_coo> to tensor<?x?xi1, #ordered_coo>
  return %0 : tensor<?x?xi1, #ordered_coo>
}

// SparseTensorAssembleConverter
func.func @assemble_i1_elem_type(%pos: tensor<2xindex>, %coords: tensor<4x1xindex>,
                                 %vals: tensor<4xi1>) -> tensor<8xi1, #sparse1d> {
  %0 = sparse_tensor.assemble (%pos, %coords), %vals
    : (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1> to tensor<8xi1, #sparse1d>
  return %0 : tensor<8xi1, #sparse1d>
}

// SparseTensorToValuesConverter
func.func @values_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> memref<?xi1> {
  %0 = sparse_tensor.values %arg0 : tensor<8xi1, #sparse1d> to memref<?xi1>
  return %0 : memref<?xi1>
}

// SparseNumberOfEntriesConverter
func.func @number_of_entries_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> index {
  %0 = sparse_tensor.number_of_entries %arg0 : tensor<8xi1, #sparse1d>
  return %0 : index
}

// SparseTensorDisassembleConverter
func.func @disassemble_i1_elem_type(%sp: tensor<8xi1, #sparse1d>,
                                      %op: tensor<2xindex>, %oi: tensor<4x1xindex>,
                                      %od: tensor<4xi1>)
    -> (tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>) {
  %rp, %ri, %d, %rpl, %ril, %dl = sparse_tensor.disassemble %sp
    : tensor<8xi1, #sparse1d>
    out_lvls(%op, %oi : tensor<2xindex>, tensor<4x1xindex>)
    out_vals(%od : tensor<4xi1>)
    -> (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1>, (index, index), index
  return %rp, %ri, %d : tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>
}

// SparseTensorInsertConverter
func.func @insert_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>, %idx: index, %val: i1)
    -> tensor<8xi1, #sparse1d> {
  %0 = tensor.insert %val into %arg0[%idx] : tensor<8xi1, #sparse1d>
  return %0 : tensor<8xi1, #sparse1d>
}

// SparseTensorCompressConverter
func.func @compress_i1_elem_type(%tensor: tensor<8x8xi1, #csr>,
                                 %values: memref<?xi1>, %filled: memref<?xi1>,
                                 %added: memref<?xindex>, %count: index, %i: index)
    -> tensor<8x8xi1, #csr> {
  %0 = sparse_tensor.compress %values, %filled, %added, %count into %tensor[%i]
    : memref<?xi1>, memref<?xi1>, memref<?xindex>, tensor<8x8xi1, #csr>
  return %0 : tensor<8x8xi1, #csr>
}

```

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:00:36 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 03:00:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54b744.a87eb5dc.7d817.a18a@mx.google.com>


https://github.com/hockyy edited https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:09:43 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 03:09:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Fix crash in sparse_tensor.new
 with unsupported element type (PR #183898)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183898@github.com>
Message-ID: <6a54b967.27b3b003.3b871.6de1@mx.google.com>


hockyy wrote:

IMO, this doesn't look like a very robust fix.

```
bool mlir::sparse_tensor::isValidPrimaryType(Type elemTp) {
  if (elemTp.isF64() || elemTp.isF32() || elemTp.isF16() || elemTp.isBF16() ||
      elemTp.isInteger(64) || elemTp.isInteger(32) || elemTp.isInteger(16) ||
      elemTp.isInteger(8))
    return true;
  if (auto complexTp = dyn_cast<ComplexType>(elemTp)) {
    Type elt = complexTp.getElementType();
    return elt.isF64() || elt.isF32();
  }
  return false;
}
```

https://github.com/llvm/llvm-project/pull/208913#discussion_r3569696086 <- There are errors in other patterns that needs this check in prior

I'm thinking of changing this function's signature `PrimaryType mlir::sparse_tensor::primaryTypeEncoding(Type elemTp)` instead to `failureOr<PrimaryType>`

so we don't need to update both the same time and adding validPrimaryType check everywhere

@joker-eph wdyt?

https://github.com/llvm/llvm-project/pull/183898

From llvmlistbot at llvm.org  Mon Jul 13 03:14:34 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 13 Jul 2026 03:14:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54ba8a.92d2873c.258297.5bff@mx.google.com>


================

----------------
secona wrote:

Yeah, I think all those also need the valid primary type check

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:16:51 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 13 Jul 2026 03:16:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54bb13.e5907290.281f16.fdb6@mx.google.com>


================

----------------
secona wrote:

...or your idea of changing the type signature. That looks to be elegant

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:17:06 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 13 Jul 2026 03:17:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54bb22.481afadf.196d57.a21e@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:17:55 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 03:17:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54bb53.af6ba3ed.1dc10f.1035@mx.google.com>


================

----------------
hockyy wrote:

> ...or your idea of changing the function signature. That looks to be elegant

do you have other opinions? I haven't looked fully into the code base. So I don't know how much of the code we'll change.


https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:18:19 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 13 Jul 2026 03:18:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54bb6b.c5504292.28b3b8.c1c8@mx.google.com>


https://github.com/hockyy edited https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 03:38:55 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 13 Jul 2026 03:38:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54c03f.d1496011.26913c.3bdb@mx.google.com>


================

----------------
secona wrote:

I can investigate. We may also need @aartbik's view on this as the sparse tensor dialect maintainer

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 04:15:11 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Mon, 13 Jul 2026 04:15:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a54c8bf.a87eb5dc.7d817.5662@mx.google.com>


Devjiu wrote:

@joker-eph

Ok, thanks for the clarification. Sorry, I didn't want to skip this issue; I thought that static verification could be sufficient. I think this actually splits into two very different situations, and I think conflating them is where the "inconsistency" comes from.

**Category 1** - patterns that only insert/fold a `tensor.cast` and never change the runtime value: `InsertSliceOpConstantArgumentFolder`, `ConvertToStaticExpandShape`, `FoldSourceTensorCast`. These recover static knowledge about a tensor whose actual shape never changes - the cast target describes the *same* runtime tensor, just more precisely. If an encoding was valid for that tensor before, it's valid after, unconditionally - no verify call needed, because nothing about the value changed, only what the compiler knows about it. I think unconditional propagation is actually
correct here, not a heuristic.

**Category 2** - patterns where the op genuinely transforms data: `CollapseShapeOp` (rank change), `ConcatOp` (merges tensors), `PadOp` (adds elements). Here the shape really does change, and you're right that `verifyEncoding` can't reliably answer whether the encoding still holds when the new shape has dynamic dims - it can only catch what's statically
provable, which is exactly the gap you're pointing at with the concat example.

For category 2 I don't have a fully general answer - bailing out when an encoding is present is one option, though it also disables refinement for cases that would've been fine. 

Would it help to split the scope along these lines: land category 1 here (this PR, `InsertSliceOpConstantArgumentFolder`/`ConvertToStaticExpandShape`/ `FoldSourceTensorCast`) with unconditional propagation, and keep category 2 (`CollapseShapeOp`, `ConcatOp`, `PadOp`, tracked separately) open for
further discussion - possibly landing on bail-out there if we can't find something better?



https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Mon Jul 13 04:37:09 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Mon, 13 Jul 2026 04:37:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a54cde5.d681cab9.333b2b.fc2e@mx.google.com>


https://github.com/Wolfram70 updated https://github.com/llvm/llvm-project/pull/199700

>From 1246175402141857ea45fec146e1a9f35770d2e5 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 10:09:58 +0000
Subject: [PATCH 01/12] [MLIR][NVGPU] Add convert.fpext and convert.fptrunc Ops

This change adds the `convert.fpext` and `convert.fptrunc` Ops to the
NVGPU dialect to support floating-point conversion operations.

These Ops support scalar, vector and tensor inputs and lower to the
corresponding NVVM Ops after padding and chunking the input into i32
registers.

For tensor inputs, the tensors must be converted to vectors by other means
before they can be successfully lowered.
---
 mlir/include/mlir/Conversion/Passes.td        |   3 +-
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  14 +
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  76 ++
 .../lib/Conversion/NVGPUToNVVM/CMakeLists.txt |   1 +
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 670 ++++++++++++++++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 156 ++++
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      | 466 ++++++++++++
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 327 +++++++++
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 106 +++
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 126 ++++
 10 files changed, 1944 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index b92bc1cd31723..a675ce431f567 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -1106,7 +1106,8 @@ def ConvertNVGPUToNVVMPass : Pass<"convert-nvgpu-to-nvvm"> {
     "arith::ArithDialect",
     "LLVM::LLVMDialect",
     "memref::MemRefDialect",
-    "NVVM::NVVMDialect"
+    "NVVM::NVVMDialect",
+    "vector::VectorDialect"
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 1c0d7bd1113ea..e56aca779cd1e 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,6 +101,20 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
+def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
+def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
+def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
+def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
+    "Sub-bytes packed kind type",
+    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
+     SubBytesPackedU6UnpackU8E2M3]> {
+  let genSpecializedAttr = 0;
+  let cppNamespace = "::mlir::nvgpu";
+}
+def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
+  let assemblyFormat = "`<` $value `>`";
+}
+
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 5a1771eecd0f6..07408ac18f95b 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -671,4 +671,80 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU Conversion Ops
+//===----------------------------------------------------------------------===//
+
+def Int8OrFloatLike : TypeConstraint<
+    Or<[FloatLike.predicate,
+        I8.predicate,
+        ValueSemanticsContainerOf<[I8]>.predicate]>,
+    "scalar, vector, or tensor of i8 or floats">;
+def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
+
+def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
+    [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
+  let description = [{
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
+
+    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
+    f16->f8, f16->f4, bf16->f8, bf16->f4.
+
+    The `random_bits` operand enables stochastic rounding (RS mode)
+    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+
+    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
+    padding; use `packed_kind` to specify the f6 variant. `compact` is not
+    a supported `packed_kind` for f6 types.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : f32 to f16
+    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu,
+                       Optional<I32>:$random_bits
+                       );
+  let results = (outs Int8OrFloatLike:$out);
+  let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
+def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
+    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
+    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
+    (f32 goes through bf16 intermediate).
+
+    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
+
+    Example:
+    ```mlir
+    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins Int8OrFloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
index a050749eb7da8..2e0a417bd823d 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/NVGPUToNVVM/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_conversion_library(MLIRNVGPUToNVVM
   MLIRNVGPUDialect
   MLIRNVVMDialect
   MLIRArithDialect
+  MLIRVectorDialect
   MLIRPass
   MLIRSCFTransforms
   MLIRTransforms
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index e566449ffadff..8b992b67aed29 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/NVGPU/IR/NVGPUDialect.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/TypeUtilities.h"
@@ -461,6 +462,7 @@ struct ConvertNVGPUToNVVMPass
     target.addLegalDialect<::mlir::arith::ArithDialect>();
     target.addLegalDialect<::mlir::memref::MemRefDialect>();
     target.addLegalDialect<::mlir::NVVM::NVVMDialect>();
+    target.addLegalDialect<::mlir::vector::VectorDialect>();
     mlir::scf::populateSCFStructuralTypeConversionsAndLegality(
         converter, patterns, target);
     if (failed(applyPartialConversion(getOperation(), target,
@@ -1709,6 +1711,669 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
         rewriter);
   }
 };
+
+//===----------------------------------------------------------------------===//
+// FPTruncOp Lowering
+//===----------------------------------------------------------------------===//
+
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+enum class TruncConvOp {
+  F32x2_TO_F16x2,
+  F32x2_TO_BF16x2,
+  F32x2_TO_F8x2,
+  F32x2_TO_F6x2,
+  F32x2_TO_F4x2,
+  F16x2_TO_F8x2,
+  F16x2_TO_F4x2,
+  BF16x2_TO_F8x2,
+  BF16x2_TO_F4x2,
+};
+
+enum class TruncSrcKind { F32, F16, BF16 };
+
+enum class TruncDstKind { F16, BF16, F8, F6, F4 };
+
+struct TruncTableEntry {
+  TruncSrcKind src;
+  TruncDstKind dst;
+  TruncConvOp convOp;
+  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+};
+
+static constexpr TruncTableEntry kTruncTable[] = {
+    // f32 source
+    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
+    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    // f16 source
+    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
+    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    // bf16 source
+    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
+    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+};
+
+static bool isConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<TruncSrcKind> classifySrcType(Type t) {
+  if (t.isF32())
+    return TruncSrcKind::F32;
+  if (t.isF16())
+    return TruncSrcKind::F16;
+  if (t.isBF16())
+    return TruncSrcKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<TruncDstKind> classifyDstType(Type t) {
+  if (t.isF16())
+    return TruncDstKind::F16;
+  if (t.isBF16())
+    return TruncDstKind::BF16;
+  if (isConvertibleF8Type(t))
+    return TruncDstKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return TruncDstKind::F6;
+  if (bitWidth == 4)
+    return TruncDstKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<std::pair<TruncConvOp, int>>
+lookupTruncConvOp(Type srcElemType, Type dstElemType) {
+  auto srcKind = classifySrcType(srcElemType);
+  auto dstKind = classifyDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kTruncTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return {{entry.convOp, entry.srcStepDecrement}};
+  }
+  return std::nullopt;
+}
+
+static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  IntegerType i64Ty = b.getI64Type();
+  return b.create<LLVM::ExtractElementOp>(
+      srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
+}
+
+/// Extract a pair of f32 values from an i32 vector at the given base index.
+/// Returns {f32_lo (lower index), f32_hi (higher index)}.
+static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
+                                              Value srcI32Vec, int baseIdx) {
+  FloatType f32Ty = b.getF32Type();
+  Value elem0 = extractElement(b, srcI32Vec, baseIdx);
+  Value elem1 = extractElement(b, srcI32Vec, baseIdx + 1);
+  return {b.create<LLVM::BitcastOp>(f32Ty, elem0),
+          b.create<LLVM::BitcastOp>(f32Ty, elem1)};
+}
+
+static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int idx, VectorType vecTy) {
+  Value elem = extractElement(b, srcI32Vec, idx);
+  return b.create<LLVM::BitcastOp>(vecTy, elem);
+}
+
+/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
+/// extract source operands, and return the native result.
+/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
+/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
+/// - f4 destinations: returns i8 (packed 2 x f4)
+/// Create a sub-byte conversion from an f32 pair source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                                int srcBaseIdx, Type resultTy, Args &&...args) {
+  auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+  return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
+}
+
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+template <typename ConvertOp, typename... Args>
+static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
+                               int srcBaseIdx, Type srcElemTy, Type resultTy,
+                               Args &&...args) {
+  Value src = extractAndBitcast(b, srcI32Vec, srcBaseIdx,
+                                VectorType::get(2, srcElemTy));
+  return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
+}
+
+static Value createTruncConversion(
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
+    NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
+    Type actualDstFloatType, Value randomBits = Value()) {
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  auto dstTyAttr = TypeAttr::get(dstElemType);
+  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+
+  switch (convOp) {
+  case TruncConvOp::F32x2_TO_F16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
+        VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_BF16x2: {
+    auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
+    Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
+        VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
+        satAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case TruncConvOp::F32x2_TO_F8x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
+  case TruncConvOp::F32x2_TO_F6x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
+  case TruncConvOp::F32x2_TO_F4x2:
+    return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
+  case TruncConvOp::F16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  case TruncConvOp::BF16x2_TO_F8x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
+        reluAttr, dstTyAttr);
+  case TruncConvOp::BF16x2_TO_F4x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
+        actualDstTyAttr);
+  }
+  llvm_unreachable("unhandled TruncConvOp");
+}
+
+struct NVGPUFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
+
+  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
+                                    nvgpu::SubBytesPackedKind packedKind) {
+    if (isa<IntegerType>(elemType)) {
+      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+        return Float6E3M2FNType::get(ctx);
+      return Float6E2M3FNType::get(ctx);
+    }
+    return elemType;
+  }
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+    auto reluBoolAttr = op.getReluAttr();
+    Value randomBits = adaptor.getRandomBits();
+    Type actualDstFloatType =
+        getActualDstFloatType(ctx, dstElemType, packedKind);
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
+    Value input = adaptor.getIn();
+    if (srcBW == 64) {
+      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+      srcType = f32VecTy;
+      srcElemType = b.getF32Type();
+      srcBW = 32;
+      if (dstBW == 32) {
+        rewriter.replaceOp(op, input);
+        return success();
+      }
+    }
+
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * dstBW / regBits;
+    Value srcI32Vec =
+        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+    if (!convEntry)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for truncation");
+    auto [convOp, srcStepDecrement] = *convEntry;
+
+    // STEP 3: pack conversion results into destination i32 vector.
+    const int srcStep = srcBW / dstBW;
+    const int resultBW =
+        dstBW * 2; // each conversion produces 2 (packed) elements
+    const int numConvsPerI32 = regBits / resultBW;
+
+    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+         srcIdx += srcStep, dstIdx++) {
+      Value dstIdxConst =
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+      Value dstValue;
+
+      if (numConvsPerI32 == 1) {
+        // f16/bf16 destinations
+        dstValue = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+      } else {
+        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        auto subResultType = IntegerType::get(ctx, resultBW);
+        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+        int insertIdx = numConvsPerI32 - 1;
+        int curStep = srcStep;
+        while (curStep > 0) {
+          curStep -= srcStepDecrement;
+          Value subResult = createTruncConversion(
+              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+              /*randomBits=*/Value());
+          subVec = b.create<LLVM::InsertElementOp>(
+              subVec, subResult,
+              b.create<LLVM::ConstantOp>(i64Ty,
+                                         b.getI64IntegerAttr(insertIdx)));
+          insertIdx--;
+        }
+
+        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+      }
+
+      dstI32Vec =
+          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+    }
+
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+    return success();
+  }
+};
+
+//===----------------------------------------------------------------------===//
+// NVGPUFPExtOpLowering
+//===----------------------------------------------------------------------===//
+
+/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+enum class ExtConvOp {
+  F8x2_TO_F16x2,
+  F8x2_TO_BF16x2,
+  F6x2_TO_F16x2,
+  F4x2_TO_F16x2,
+};
+
+enum class ExtSrcKind { F8, F6, F4 };
+enum class ExtDstKind { F16, BF16 };
+
+struct ExtTableEntry {
+  ExtSrcKind src;
+  ExtDstKind dst;
+  ExtConvOp convOp;
+};
+
+static constexpr ExtTableEntry kExtTable[] = {
+    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
+    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
+    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
+    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+};
+
+static bool isExtConvertibleF8Type(Type t) {
+  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+}
+
+static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
+  if (isExtConvertibleF8Type(t))
+    return ExtSrcKind::F8;
+  int bitWidth = t.getIntOrFloatBitWidth();
+  if (isa<IntegerType>(t) && bitWidth == 8)
+    return ExtSrcKind::F6;
+  if (bitWidth == 4)
+    return ExtSrcKind::F4;
+  return std::nullopt;
+}
+
+static std::optional<ExtDstKind> classifyExtDstType(Type t) {
+  if (t.isF16())
+    return ExtDstKind::F16;
+  if (t.isBF16())
+    return ExtDstKind::BF16;
+  return std::nullopt;
+}
+
+static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
+                                                Type dstElemType) {
+  auto srcKind = classifyExtSrcType(srcElemType);
+  auto dstKind = classifyExtDstType(dstElemType);
+  if (!srcKind || !dstKind)
+    return std::nullopt;
+  for (const auto &entry : kExtTable) {
+    if (entry.src == *srcKind && entry.dst == *dstKind)
+      return entry.convOp;
+  }
+  return std::nullopt;
+}
+
+static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
+                                  nvgpu::SubBytesPackedKind packedKind) {
+  if (isa<IntegerType>(elemType)) {
+    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
+      return Float6E3M2FNType::get(ctx);
+    return Float6E2M3FNType::get(ctx);
+  }
+  return elemType;
+}
+
+/// Create a typed NVVM extension conversion.
+/// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
+/// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
+static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
+                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
+                                 Type actualSrcFloatType,
+                                 Value extScaleFactor = Value()) {
+  IntegerType i32Ty = b.getI32Type();
+  auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
+
+  switch (convOp) {
+  case ExtConvOp::F8x2_TO_F16x2: {
+    Value r = NVVM::ConvertF8x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F8x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F6x2_TO_F16x2: {
+    Value r = NVVM::ConvertF6x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case ExtConvOp::F4x2_TO_F16x2: {
+    Value r = NVVM::ConvertF4x2ToF16x2Op::create(
+        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  }
+  llvm_unreachable("unhandled ExtConvOp");
+}
+
+struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
+
+    MLIRContext *ctx = getContext();
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    IntegerType i8Ty = b.getI8Type();
+    IntegerType i16Ty = b.getI16Type();
+    IntegerType i32Ty = b.getI32Type();
+    IntegerType i64Ty = b.getI64Type();
+
+    static constexpr int regBits = 32;
+    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+    if (!srcType || srcType.getRank() != 1 || !dstType ||
+        dstType.getRank() != 1)
+      return rewriter.notifyMatchFailure(
+          op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+    auto srcElemType = srcType.getElementType();
+    auto dstElemType = dstType.getElementType();
+    int srcBW = srcType.getElementTypeBitWidth();
+    int dstBW = dstType.getElementTypeBitWidth();
+    int numElems = srcType.getNumElements();
+
+    auto packedKind = op.getPackedKind();
+    auto reluBoolAttr = op.getReluAttr();
+    Type actualSrcFloatType =
+        getActualSrcFloatType(ctx, srcElemType, packedKind);
+
+    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+    if (srcBW >= 16 && dstBW >= 32) {
+      Value result = adaptor.getIn();
+      if (srcElemType != dstElemType) {
+        Type convertedType = getTypeConverter()->convertType(dstType);
+        assert(convertedType && "failed to convert type");
+        result = b.create<LLVM::FPExtOp>(convertedType, result);
+      }
+      rewriter.replaceOp(op, result);
+      return success();
+    }
+
+    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+    // followed by FPExt to the final f32/f64 destination.
+    bool needsFinalFPExt = (dstBW >= 32);
+    // e8m0 must go through bf16 (only available NVVM op); others use f16.
+    Type intermediateDstElem = dstElemType;
+    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+      intermediateDstElem = b.getBF16Type();
+    else if (needsFinalFPExt)
+      intermediateDstElem = b.getF16Type();
+    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+    // STEP 1: bitcast input vector to i32 register vector.
+    int srcI32Elems = numElems * srcBW / regBits;
+    int dstI32Elems = numElems * intermediateDstBW / regBits;
+    Value srcI32Vec = b.create<LLVM::BitcastOp>(
+        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+    Value dstI32Vec =
+        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+    // STEP 2: look up the conversion op from the (srcType, dstType) table.
+    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convOpOpt)
+      return rewriter.notifyMatchFailure(
+          op, "unsupported type combination for extension");
+    ExtConvOp convOp = *convOpOpt;
+    Value extScaleFactor;
+
+    // STEP 3: iterate over source i32 elements, producing destination i32s.
+    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+      Value srcI32 = b.create<LLVM::ExtractElementOp>(
+          srcI32Vec,
+          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+      if (srcBW == 8) {
+        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+        Value i16Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+        for (int half = 0; half < 2; half++) {
+          Value halfI16 = b.create<LLVM::ExtractElementOp>(
+              i16Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+          Value src =
+              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      } else {
+        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+        Value i8Vec =
+            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+          Value src = b.create<LLVM::ExtractElementOp>(
+              i8Vec,
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+          Value dstValue =
+              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                  actualSrcFloatType, extScaleFactor);
+          Value dstIdxConst =
+              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+          dstI32Vec =
+              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+          dstIdx++;
+        }
+      }
+    }
+
+    // STEP 4: produce final result.
+    Type convertedType = getTypeConverter()->convertType(dstType);
+    assert(convertedType && "failed to convert type");
+    Value result;
+    if (needsFinalFPExt) {
+      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+      Value intermediateVec =
+          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+    } else {
+      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
+                                  int step) {
+  static constexpr int regBits = 32;
+  auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
+  int64_t padded =
+      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
+               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+  return ceilDiv(padded, step) * step;
+}
+
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
+/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
+/// an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(CvtOp op,
+                                PatternRewriter &rewriter) const override {
+    Type inType = op.getIn().getType();
+    Type outType = op.getOut().getType();
+
+    // Tensor inputs are not handled here; they must be converted to vectors
+    // by other means before this pattern runs.
+    if (isa<RankedTensorType>(inType))
+      return failure();
+
+    Type srcElemTy = getElementTypeOrSelf(inType);
+    Type dstElemTy = getElementTypeOrSelf(outType);
+    int srcBW = srcElemTy.getIntOrFloatBitWidth();
+    int dstBW = dstElemTy.getIntOrFloatBitWidth();
+
+    bool isScalar = !isa<VectorType>(inType);
+    auto srcVecTy = dyn_cast<VectorType>(inType);
+    bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
+    int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
+    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
+    bool needsPad = (paddedElems != numElems);
+
+    if (!isScalar && !isMultiRank && !needsPad)
+      return failure();
+
+    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+    Value input = op.getIn();
+
+    if (isScalar)
+      input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
+                                          input);
+
+    if (isMultiRank)
+      input = vector::ShapeCastOp::create(
+          b, VectorType::get({numElems}, srcElemTy), input);
+
+    if (needsPad) {
+      auto paddedTy = VectorType::get({paddedElems}, srcElemTy);
+      Value zero = arith::ConstantOp::create(
+          b, DenseElementsAttr::get(paddedTy, b.getZeroAttr(srcElemTy)));
+      input = vector::InsertStridedSliceOp::create(
+          b, input, zero, SmallVector<int64_t>{0}, SmallVector<int64_t>{1});
+    }
+
+    auto cvtDstTy =
+        VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
+                          op.getPackedKindAttr(), op.getReluAttr());
+    Value result = cvt;
+
+    if (needsPad)
+      result = vector::ExtractStridedSliceOp::create(
+          b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
+          SmallVector<int64_t>{1});
+
+    if (isMultiRank)
+      result =
+          vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+
+    if (isScalar)
+      result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+};
+
+using NVGPUFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
+using NVGPUFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -1753,7 +2418,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
+      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
+      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
+
+  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
+      patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 2b2686a9163bf..5e3b7c8df515e 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -700,6 +700,162 @@ LogicalResult RcpOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPTruncOp
+//===----------------------------------------------------------------------===//
+
+static bool isShapedContainerType(Type t) {
+  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
+}
+
+static LogicalResult verifyConversionShapes(Operation *op, Type inType,
+                                            Type outType) {
+  if (llvm::isa<UnrankedTensorType>(inType) ||
+      llvm::isa<UnrankedTensorType>(outType))
+    return op->emitOpError("unranked tensor types are not supported, got ")
+           << inType << " and " << outType;
+  bool srcIsShaped = isShapedContainerType(inType);
+  bool dstIsShaped = isShapedContainerType(outType);
+  if (srcIsShaped != dstIsShaped)
+    return op->emitOpError("input and output must both be scalars or both be "
+                           "vectors/tensors, got ")
+           << inType << " and " << outType;
+  if (srcIsShaped) {
+    auto srcShaped = llvm::cast<ShapedType>(inType);
+    auto dstShaped = llvm::cast<ShapedType>(outType);
+    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
+      return op->emitOpError("rank-0 shaped types are not supported, use "
+                             "scalar type instead");
+    if (srcShaped.getShape() != dstShaped.getShape())
+      return op->emitOpError("input and output shapes must match, got ")
+             << inType << " and " << outType;
+    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
+      return op->emitOpError("input and output must be the same container "
+                             "type (both vector or both tensor), got ")
+             << inType << " and " << outType;
+  }
+  return success();
+}
+
+LogicalResult FPTruncOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (dstBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact result type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !dstType.isInteger(8))
+    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(dstType))
+    return emitOpError("result type expects float type with `compact` packed "
+                       "kind, but got ")
+           << dstType;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
+
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
+
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_CvtFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult FPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  SubBytesPackedKind packedKind = getPackedKind();
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
+
+  if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
+    return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
+           << dstBitWidth;
+
+  if (srcBitWidth == 6)
+    return emitOpError("currently doesn't support fp6 compact input type");
+
+  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
+       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
+      !srcType.isInteger(8))
+    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (packedKind == SubBytesPackedKind::COMPACT &&
+      !llvm::isa<FloatType>(srcType))
+    return emitOpError("input type expects float type with `compact` packed "
+                       "kind, but got ")
+           << srcType;
+
+  if (llvm::isa<Float8E8M0FNUType>(srcType) &&
+      !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
+    return emitOpError("expects bf16 or f32 output type when input type is "
+                       "e8m0.");
+
+  if (rnd != mlir::NVVM::FPRoundingMode::RN)
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+
+  if (getRelu() && llvm::isa<BFloat16Type>(dstType))
+    return emitOpError("relu is not supported for bf16 destination");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd dialect, type, and op definitions
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
new file mode 100644
index 0000000000000..08407fe4bebf1
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -0,0 +1,466 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector extension to f16/bf16.
+
+// CHECK-LABEL: @cvt_float_e4m3fn_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E4M3FN>
+func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E4M3FN> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E4M3FN)
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e5m2_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E5M2>
+func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E5M2> to vector<8xi8>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f8E5M2)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e8m0_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf8E8M0FNU>
+func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf8E8M0FNU> to vector<8xi8>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xbf16> to i32
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f8E8M0FNU)
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  return 
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e3m2_to_f16(
+// CHECK: %[[IN:.+]]: vector<8xi8>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK-SAME: : vector<2xi8>(f6E3M2FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<4xi32>
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<8xi32>
+  // CHECK: llvm.extractelement %[[IN_I32]]
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xf16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.extractelement {{.*}} : vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.insertelement {{.*}} : vector<8xi32>
+  // CHECK: llvm.bitcast {{.*}} : i32 to vector<4xi8>
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  return 
+}
+
+// Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
+
+// -----
+
+// CHECK-LABEL: @fpext_e4m3fn_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E4M3FN>
+func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf8E4M3FN> to vector<4xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi8> to vector<1xi32>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e5m2_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E5M2>
+func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e8m0_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf8E8M0FNU>
+func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m3_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
+func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_e2m1_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf4E2M1FN>
+func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  return %out : vector<8xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf16>
+func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_bf16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
+func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
+  // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// Extension to f64.
+
+// CHECK-LABEL: @fpext_f16_to_f64
+func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_bf16_to_f64
+func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
+  // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
+  // CHECK-NOT: llvm.fpext
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f32_to_f64
+func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
+  // CHECK-NOT: nvvm
+  // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f8_to_f64
+func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// CHECK-LABEL: @fpext_f4_to_f64
+func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
+  // CHECK: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  return %out : vector<8xf64>
+}
+
+// CHECK-LABEL: @fpext_e2m3_to_f64
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+  // CHECK: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} to {{.*}}f64
+  // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
+  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  return %out : vector<4xf64>
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f8E4M3FN to vector<1xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_scalar_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: f8E4M3FN
+func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v2x4_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf8E4M3FN>
+func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  // CHECK: vector.shape_cast {{.*}} to vector<8xf8E4M3FN>
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fpext_v3f8_to_v3f16
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// -----
+
+// CHECK-LABEL: @fpext_v3_f8_to_f32
+// CHECK-SAME: %[[IN:.+]]: vector<3xf8E5M2>
+func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.fpext
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fpext_v3x1_f8_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf8E4M3FN>
+func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf8E4M3FN> to vector<3xf8E4M3FN>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// Relu attribute.
+
+// CHECK-LABEL: @fpext_f8_to_f16_relu
+func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
+  // CHECK: nvvm.convert.f8x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fpext %in {relu = true}
+      : vector<8xf8E4M3FN> to vector<8xf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f16
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  return %out : vector<3xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_scalar_f8_to_f32
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  return %out : f32
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f8_to_f32
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  return %out : vector<2x4xf32>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f8_to_v3f32
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f8x2.to.f16x2
+// CHECK-E2E: llvm.fpext
+// CHECK-E2E: return
+func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  return %out : vector<3xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
new file mode 100644
index 0000000000000..e8a5c27f22917
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -0,0 +1,327 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -convert-vector-to-llvm | FileCheck %s --check-prefix=CHECK-E2E
+
+// Basic aligned vector inputs.
+
+// CHECK-LABEL: @cvt_float_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xf16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_f16_v8(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<4xf32>
+func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<4xf32> to vector<4xi32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
+  // CHECK-SAME: : vector<2xbf16>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e4m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
+  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<2xi32>
+  // CHECK: %[[IDX_0:.+]] = llvm.mlir.constant(0 : i64) : i64
+  // CHECK: %[[SUB_VEC_0:.+]] = llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: : i16(f8E4M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return 
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf32> to vector<8xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK-SAME: : i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  return 
+}
+
+// Scalar inputs (canonicalize: broadcast + pad + extract).
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
+  // CHECK: vector.broadcast %[[IN]] : f32 to vector<1xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f32_to_bf16
+// CHECK-SAME: %[[IN:.+]]: f32
+func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
+  // CHECK: vector.broadcast %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  return %out : bf16
+}
+
+// CHECK-LABEL: @fptrunc_scalar_f64_to_f32
+func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
+  // CHECK: vector.broadcast
+  // CHECK: llvm.fptrunc
+  // CHECK: vector.extract
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  return %out : f32
+}
+
+// Multi-rank vectors (canonicalize: shape_cast flatten).
+
+// CHECK-LABEL: @fptrunc_v2x4_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<2x4xf32>
+func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v4x2_f32_to_f8
+// CHECK-SAME: %[[IN:.+]]: vector<4x2xf32>
+func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  return %out : vector<4x2xf8E4M3FN>
+}
+
+// Non-aligned 1-D vectors (canonicalize: pad via insert/extract_strided_slice).
+
+// CHECK-LABEL: @fptrunc_v1f32_to_v1f16
+// CHECK-SAME: %[[IN:.+]]: vector<1xf32>
+func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  return %out : vector<1xf16>
+}
+
+// CHECK-LABEL: @fptrunc_v3f16_to_v3f8
+// CHECK-SAME: %[[IN:.+]]: vector<3xf16>
+func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice %[[IN]]
+  // CHECK: nvvm.convert.f16x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
+
+// Multi-rank + padding combined.
+
+// CHECK-LABEL: @fptrunc_v3x1_f32_to_f16
+// CHECK-SAME: %[[IN:.+]]: vector<3x1xf32>
+func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
+  // CHECK: vector.shape_cast %[[IN]] : vector<3x1xf32> to vector<3xf32>
+  // CHECK: vector.insert_strided_slice
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  return %out : vector<3x1xf16>
+}
+
+// f64 source truncation.
+
+// CHECK-LABEL: @fptrunc_f64_to_f32
+func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK-NOT: nvvm
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  return %out : vector<4xf32>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f16
+func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  return %out : vector<2xf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_bf16
+func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  return %out : vector<2xbf16>
+}
+
+// CHECK-LABEL: @fptrunc_f64_to_f8
+func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
+  // CHECK: vector.insert_strided_slice
+  // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK: vector.extract_strided_slice
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  return %out : vector<4xf8E4M3FN>
+}
+
+// Saturation and relu attributes.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_satfinite
+func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_relu
+func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: relu = true
+  %out = nvgpu.convert.fptrunc %in {relu = true}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Default SATFINITE behavior.
+
+// CHECK-LABEL: @fptrunc_f32_to_f8_default_sat
+func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_default_sat
+func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_explicit_none
+func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: satfinite
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// Stochastic rounding (RS + random_bits).
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rs
+func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rs
+func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// End-to-end: no residual vector ops after full lowering.
+
+// CHECK-E2E-LABEL: @e2e_scalar_f32_to_f16
+// CHECK-E2E-NOT: vector.broadcast
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E-NOT: vector.extract
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  return %out : f16
+}
+
+// CHECK-E2E-LABEL: @e2e_v2x4_f32_to_f16
+// CHECK-E2E-NOT: vector.shape_cast
+// CHECK-E2E: nvvm.convert.f32x2.to.f16x2
+// CHECK-E2E: return
+func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  return %out : vector<2x4xf16>
+}
+
+// CHECK-E2E-LABEL: @e2e_v3f16_to_v3f8
+// CHECK-E2E-NOT: vector.insert_strided_slice
+// CHECK-E2E-NOT: vector.extract_strided_slice
+// CHECK-E2E: nvvm.convert.f16x2.to.f8x2
+// CHECK-E2E: return
+func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  return %out : vector<3xf8E4M3FN>
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
new file mode 100644
index 0000000000000..a2f8fd6ec48f8
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -0,0 +1,106 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
+  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_compact_not_float(%in : vector<16xi8>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
new file mode 100644
index 0000000000000..9e295b4ad25a0
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -0,0 +1,126 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  return
+}
+
+// -----
+
+func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
+  return
+}
+
+// -----
+
+func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+      : vector<16xf32> to vector<16xf8E8M0FNU>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  return
+}

>From 1cff6f6c7644ebb0467cffb4a64b11928874c8a4 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 26 May 2026 14:49:27 +0000
Subject: [PATCH 02/12] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 8b992b67aed29..08cbefee015c2 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2289,9 +2289,9 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext: handles
-/// scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors. Runs as
-/// an OpRewritePattern on MLIR types before LLVM type conversion.
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
 struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
   using OpRewritePattern<CvtOp>::OpRewritePattern;

>From cae401fd4789b033cb9e96c2d5ba3514f476dc02 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 13:37:59 +0000
Subject: [PATCH 03/12] address comments and refactor

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  13 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  21 +-
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 345 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  34 --
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  33 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  36 +-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  24 --
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  24 --
 8 files changed, 225 insertions(+), 305 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index e56aca779cd1e..38c8bb5c9aa2c 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,19 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-def SubBytesPackedCompact         : I32EnumAttrCase<"COMPACT", 0, "compact">;
-def SubBytesPackedU6UnpackU8E3M2  : I32EnumAttrCase<"U6_UNPACK_U8_E3M2", 1, "u6_unpack_u8_e3m2">;
-def SubBytesPackedU6UnpackU8E2M3  : I32EnumAttrCase<"U6_UNPACK_U8_E2M3", 2, "u6_unpack_u8_e2m3">;
-def SubBytesPackedKind : I32EnumAttr<"SubBytesPackedKind",
-    "Sub-bytes packed kind type",
-    [SubBytesPackedCompact, SubBytesPackedU6UnpackU8E3M2,   
-     SubBytesPackedU6UnpackU8E2M3]> {
-  let genSpecializedAttr = 0;
-  let cppNamespace = "::mlir::nvgpu";
-}
-def SubBytesPackedKindAttr : EnumAttr<NVGPU_Dialect, SubBytesPackedKind, "subbytes_packedkind"> {
-  let assemblyFormat = "`<` $value `>`";
-}
 
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 07408ac18f95b..4ffca888c0260 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -675,11 +675,6 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 // NVGPU Conversion Ops
 //===----------------------------------------------------------------------===//
 
-def Int8OrFloatLike : TypeConstraint<
-    Or<[FloatLike.predicate,
-        I8.predicate,
-        ValueSemanticsContainerOf<[I8]>.predicate]>,
-    "scalar, vector, or tensor of i8 or floats">;
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
 def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
@@ -690,30 +685,26 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Destination must be strictly narrower than source.
 
     Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f4, bf16->f8, bf16->f4.
+    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
 
-    For f6 types (f6E3M2FN/f6E2M3FN), result is i8 with 2-bit MSB
-    padding; use `packed_kind` to specify the f6 variant. `compact` is not
-    a supported `packed_kind` for f6 types.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
     %r = nvgpu.cvt_fptrunc %in : f32 to f16
     %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs Int8OrFloatLike:$out);
+  let results = (outs FloatLike:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -729,18 +720,16 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
     (f32 goes through bf16 intermediate).
 
-    For f6 types, input is i8 with `packed_kind` specifying the f6 variant.
-
     Example:
     ```mlir
     %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
     %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
     %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins Int8OrFloatLike:$in,
+  let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SubBytesPackedKindAttr, "SubBytesPackedKind::COMPACT">:$packed_kind,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
   let results = (outs FloatLike:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 08cbefee015c2..99ab025a6a27e 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,90 +1713,99 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// FPTruncOp Lowering
+// NVGPUFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
+enum class FPKind { F32, BF16, F16, F8, F6, F4 };
+
+static int getEffectiveBitWidth(int bitWidth) {
+  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
+  return bitWidth == 6 ? 8 : bitWidth;
+}
+
+static std::optional<FPKind> classifyFPType(Type t) {
+  static constexpr auto isConvertibleF8Type = [](Type t) {
+    return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
+  };
+  static constexpr auto isConvertibleF6Type = [](Type t) {
+    return isa<Float6E2M3FNType, Float6E3M2FNType>(t);
+  };
+  static constexpr auto isConvertibleF4Type = [](Type t) {
+    return isa<Float4E2M1FNType>(t);
+  };
+
+  if (t.isF32())
+    return FPKind::F32;
+  if (t.isBF16())
+    return FPKind::BF16;
+  if (t.isF16())
+    return FPKind::F16;
+  if (isConvertibleF8Type(t))
+    return FPKind::F8;
+  if (isConvertibleF6Type(t))
+    return FPKind::F6;
+  if (isConvertibleF4Type(t))
+    return FPKind::F4;
+  return std::nullopt;
+}
+
+/// Number of source-side i32 register slots consumed by each NVVM convert Op.
+static int getNumSrcI32PerConv(FPKind src) {
+  return src == FPKind::F32 ? 2 : 1;
+}
+
 /// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
-enum class TruncConvOp {
+enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
   F32x2_TO_F8x2,
   F32x2_TO_F6x2,
   F32x2_TO_F4x2,
   F16x2_TO_F8x2,
+  F16x2_TO_F6x2,
   F16x2_TO_F4x2,
   BF16x2_TO_F8x2,
+  BF16x2_TO_F6x2,
   BF16x2_TO_F4x2,
 };
 
-enum class TruncSrcKind { F32, F16, BF16 };
-
-enum class TruncDstKind { F16, BF16, F8, F6, F4 };
-
-struct TruncTableEntry {
-  TruncSrcKind src;
-  TruncDstKind dst;
-  TruncConvOp convOp;
-  int srcStepDecrement; // 2 for f32 pairs, 1 for f16x2/bf16x2
+struct FPTruncTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPTruncConvOp convOp;
 };
 
-static constexpr TruncTableEntry kTruncTable[] = {
+static constexpr FPTruncTableEntry kFPTruncTable[] = {
     // f32 source
-    {TruncSrcKind::F32, TruncDstKind::F16, TruncConvOp::F32x2_TO_F16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::BF16, TruncConvOp::F32x2_TO_BF16x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F8, TruncConvOp::F32x2_TO_F8x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F6, TruncConvOp::F32x2_TO_F6x2, 2},
-    {TruncSrcKind::F32, TruncDstKind::F4, TruncConvOp::F32x2_TO_F4x2, 2},
+    {FPKind::F32, FPKind::F16, FPTruncConvOp::F32x2_TO_F16x2},
+    {FPKind::F32, FPKind::BF16, FPTruncConvOp::F32x2_TO_BF16x2},
+    {FPKind::F32, FPKind::F8, FPTruncConvOp::F32x2_TO_F8x2},
+    {FPKind::F32, FPKind::F6, FPTruncConvOp::F32x2_TO_F6x2},
+    {FPKind::F32, FPKind::F4, FPTruncConvOp::F32x2_TO_F4x2},
     // f16 source
-    {TruncSrcKind::F16, TruncDstKind::F8, TruncConvOp::F16x2_TO_F8x2, 1},
-    {TruncSrcKind::F16, TruncDstKind::F4, TruncConvOp::F16x2_TO_F4x2, 1},
+    {FPKind::F16, FPKind::F8, FPTruncConvOp::F16x2_TO_F8x2},
+    {FPKind::F16, FPKind::F6, FPTruncConvOp::F16x2_TO_F6x2},
+    {FPKind::F16, FPKind::F4, FPTruncConvOp::F16x2_TO_F4x2},
     // bf16 source
-    {TruncSrcKind::BF16, TruncDstKind::F8, TruncConvOp::BF16x2_TO_F8x2, 1},
-    {TruncSrcKind::BF16, TruncDstKind::F4, TruncConvOp::BF16x2_TO_F4x2, 1},
+    {FPKind::BF16, FPKind::F8, FPTruncConvOp::BF16x2_TO_F8x2},
+    {FPKind::BF16, FPKind::F6, FPTruncConvOp::BF16x2_TO_F6x2},
+    {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static bool isConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<TruncSrcKind> classifySrcType(Type t) {
-  if (t.isF32())
-    return TruncSrcKind::F32;
-  if (t.isF16())
-    return TruncSrcKind::F16;
-  if (t.isBF16())
-    return TruncSrcKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<TruncDstKind> classifyDstType(Type t) {
-  if (t.isF16())
-    return TruncDstKind::F16;
-  if (t.isBF16())
-    return TruncDstKind::BF16;
-  if (isConvertibleF8Type(t))
-    return TruncDstKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return TruncDstKind::F6;
-  if (bitWidth == 4)
-    return TruncDstKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<std::pair<TruncConvOp, int>>
-lookupTruncConvOp(Type srcElemType, Type dstElemType) {
-  auto srcKind = classifySrcType(srcElemType);
-  auto dstKind = classifyDstType(dstElemType);
+static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
+                                                          Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kTruncTable) {
+  for (const auto &entry : kFPTruncTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return {{entry.convOp, entry.srcStepDecrement}};
+      return entry;
   }
   return std::nullopt;
 }
 
+/// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
@@ -1804,7 +1813,6 @@ static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
 }
 
 /// Extract a pair of f32 values from an i32 vector at the given base index.
-/// Returns {f32_lo (lower index), f32_hi (higher index)}.
 static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
                                               Value srcI32Vec, int baseIdx) {
   FloatType f32Ty = b.getF32Type();
@@ -1814,18 +1822,16 @@ static std::pair<Value, Value> extractF32Pair(ImplicitLocOpBuilder &b,
           b.create<LLVM::BitcastOp>(f32Ty, elem1)};
 }
 
+/// Extract a vector of elements of size i32 from an i32 vector and bitcast to
+/// the specified vector type.
 static Value extractAndBitcast(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int idx, VectorType vecTy) {
   Value elem = extractElement(b, srcI32Vec, idx);
   return b.create<LLVM::BitcastOp>(vecTy, elem);
 }
 
-/// Dispatch to the specific NVVM conversion op based on TruncConvOp,
-/// extract source operands, and return the native result.
-/// - f16/bf16 destinations: returns i32 (bitcast from vector<2xf16/bf16>)
-/// - f8/f6 destinations: returns i16 (packed 2 x f8/f6)
-/// - f4 destinations: returns i8 (packed 2 x f4)
-/// Create a sub-byte conversion from an f32 pair source.
+/// Create a sub-byte conversion from an f32 pair source and return the native
+/// result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                 int srcBaseIdx, Type resultTy, Args &&...args) {
@@ -1833,7 +1839,8 @@ static Value convertFromF32Pair(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, hi, lo, std::forward<Args>(args)...);
 }
 
-/// Create a sub-byte conversion from a packed f16x2/bf16x2 source.
+/// Create a sub-byte conversion from a packed f16x2/bf16x2 source and return
+/// the native result.
 template <typename ConvertOp, typename... Args>
 static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
                                int srcBaseIdx, Type srcElemTy, Type resultTy,
@@ -1843,8 +1850,9 @@ static Value convertFromPacked(ImplicitLocOpBuilder &b, Value srcI32Vec,
   return b.create<ConvertOp>(resultTy, src, std::forward<Args>(args)...);
 }
 
+/// Create a typed NVVM truncation conversion.
 static Value createTruncConversion(
-    ImplicitLocOpBuilder &b, MLIRContext *ctx, TruncConvOp convOp,
+    ImplicitLocOpBuilder &b, MLIRContext *ctx, FPTruncConvOp convOp,
     Value srcI32Vec, int srcBaseIdx, NVVM::FPRoundingModeAttr rndAttr,
     NVVM::SaturationModeAttr satAttr, BoolAttr reluAttr, Type dstElemType,
     Type actualDstFloatType, Value randomBits = Value()) {
@@ -1855,67 +1863,63 @@ static Value createTruncConversion(
   auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
-  case TruncConvOp::F32x2_TO_F16x2: {
+  case FPTruncConvOp::F32x2_TO_F16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToF16x2Op>(
         VectorType::get(2, b.getF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_BF16x2: {
+  case FPTruncConvOp::F32x2_TO_BF16x2: {
     auto [lo, hi] = extractF32Pair(b, srcI32Vec, srcBaseIdx);
     Value r = b.create<NVVM::ConvertF32x2ToBF16x2Op>(
         VectorType::get(2, b.getBF16Type()), hi, lo, randomBits, rndAttr,
         satAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case TruncConvOp::F32x2_TO_F8x2:
+  case FPTruncConvOp::F32x2_TO_F8x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, rndAttr, satAttr, reluAttr, dstTyAttr);
-  case TruncConvOp::F32x2_TO_F6x2:
+  case FPTruncConvOp::F32x2_TO_F6x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF6x2Op>(
         b, srcI32Vec, srcBaseIdx, i16Ty, reluAttr, actualDstTyAttr);
-  case TruncConvOp::F32x2_TO_F4x2:
+  case FPTruncConvOp::F32x2_TO_F4x2:
     return convertFromF32Pair<NVVM::ConvertF32x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, i8Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F8x2:
+  case FPTruncConvOp::F16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr, dstTyAttr);
-  case TruncConvOp::F16x2_TO_F4x2:
+  case FPTruncConvOp::F16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::F16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
-  case TruncConvOp::BF16x2_TO_F8x2:
+  case FPTruncConvOp::BF16x2_TO_F8x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF8x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, rndAttr, satAttr,
         reluAttr, dstTyAttr);
-  case TruncConvOp::BF16x2_TO_F4x2:
+  case FPTruncConvOp::BF16x2_TO_F6x2:
+    return convertFromPacked<NVVM::ConvertBF16x2ToF6x2Op>(
+        b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i16Ty, reluAttr,
+        actualDstTyAttr);
+  case FPTruncConvOp::BF16x2_TO_F4x2:
     return convertFromPacked<NVVM::ConvertBF16x2ToF4x2Op>(
         b, srcI32Vec, srcBaseIdx, b.getBF16Type(), i8Ty, reluAttr,
         actualDstTyAttr);
   }
-  llvm_unreachable("unhandled TruncConvOp");
+  llvm_unreachable("unhandled FPTruncConvOp");
 }
 
 struct NVGPUFPTruncOpLowering
     : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
   using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
 
-  static Type getActualDstFloatType(MLIRContext *ctx, Type elemType,
-                                    nvgpu::SubBytesPackedKind packedKind) {
-    if (isa<IntegerType>(elemType)) {
-      if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-        return Float6E3M2FNType::get(ctx);
-      return Float6E2M3FNType::get(ctx);
-    }
-    return elemType;
-  }
-
   LogicalResult
   matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -1924,8 +1928,8 @@ struct NVGPUFPTruncOpLowering
     ImplicitLocOpBuilder b(op->getLoc(), rewriter);
     IntegerType i32Ty = b.getI32Type();
     IntegerType i64Ty = b.getI64Type();
-
     static constexpr int regBits = 32;
+
     auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
     auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
     if (!srcType || srcType.getRank() != 1 || !dstType ||
@@ -1939,13 +1943,11 @@ struct NVGPUFPTruncOpLowering
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
     NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
     auto reluBoolAttr = op.getReluAttr();
     Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType =
-        getActualDstFloatType(ctx, dstElemType, packedKind);
+    Type actualDstFloatType = dstElemType;
 
     // STEP 1: bitcast input vector to i32 register vector.
     // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
@@ -1962,8 +1964,11 @@ struct NVGPUFPTruncOpLowering
       }
     }
 
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
     int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * dstBW / regBits;
+    int dstI32Elems = numElems * effectiveDstBW / regBits;
     Value srcI32Vec =
         b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
     Value dstI32Vec =
@@ -1974,12 +1979,13 @@ struct NVGPUFPTruncOpLowering
     if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for truncation");
-    auto [convOp, srcStepDecrement] = *convEntry;
+    FPTruncConvOp convOp = convEntry->convOp;
+    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
 
     // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / dstBW;
+    const int srcStep = srcBW / effectiveDstBW;
     const int resultBW =
-        dstBW * 2; // each conversion produces 2 (packed) elements
+        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
     const int numConvsPerI32 = regBits / resultBW;
 
     for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
@@ -1994,7 +2000,7 @@ struct NVGPUFPTruncOpLowering
             b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
             reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
       } else {
-        // f8/f6 destinations: pack sub-results via vector insert + bitcast.
+        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
         auto subResultType = IntegerType::get(ctx, resultBW);
         auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
         Value subVec = b.create<LLVM::UndefOp>(subVecTy);
@@ -2002,7 +2008,7 @@ struct NVGPUFPTruncOpLowering
         int insertIdx = numConvsPerI32 - 1;
         int curStep = srcStep;
         while (curStep > 0) {
-          curStep -= srcStepDecrement;
+          curStep -= numSrcI32PerConv;
           Value subResult = createTruncConversion(
               b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
               satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
@@ -2021,10 +2027,19 @@ struct NVGPUFPTruncOpLowering
           b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
     }
 
+    // STEP 4: produce final result.
     Type convertedType = getTypeConverter()->convertType(dstType);
     assert(convertedType && "failed to convert type");
-    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    rewriter.replaceOp(op, dstVec);
+    if (convEntry->dst == FPKind::F6) {
+      IntegerType i8Ty = b.getI8Type();
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+      rewriter.replaceOp(op, truncVec);
+    } else {
+      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+      rewriter.replaceOp(op, dstVec);
+    }
     return success();
   }
 };
@@ -2034,108 +2049,72 @@ struct NVGPUFPTruncOpLowering
 //===----------------------------------------------------------------------===//
 
 /// Extension conversion op identifier for nvgpu.convert.fpext lowering.
-enum class ExtConvOp {
+enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
   F4x2_TO_F16x2,
 };
 
-enum class ExtSrcKind { F8, F6, F4 };
-enum class ExtDstKind { F16, BF16 };
-
-struct ExtTableEntry {
-  ExtSrcKind src;
-  ExtDstKind dst;
-  ExtConvOp convOp;
+struct FPExtTableEntry {
+  FPKind src;
+  FPKind dst;
+  FPExtConvOp convOp;
 };
 
-static constexpr ExtTableEntry kExtTable[] = {
-    {ExtSrcKind::F8, ExtDstKind::F16, ExtConvOp::F8x2_TO_F16x2},
-    {ExtSrcKind::F8, ExtDstKind::BF16, ExtConvOp::F8x2_TO_BF16x2},
-    {ExtSrcKind::F6, ExtDstKind::F16, ExtConvOp::F6x2_TO_F16x2},
-    {ExtSrcKind::F4, ExtDstKind::F16, ExtConvOp::F4x2_TO_F16x2},
+static constexpr FPExtTableEntry kFPExtTable[] = {
+    {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
+    {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
+    {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
 };
 
-static bool isExtConvertibleF8Type(Type t) {
-  return isa<Float8E4M3FNType, Float8E5M2Type, Float8E8M0FNUType>(t);
-}
-
-static std::optional<ExtSrcKind> classifyExtSrcType(Type t) {
-  if (isExtConvertibleF8Type(t))
-    return ExtSrcKind::F8;
-  int bitWidth = t.getIntOrFloatBitWidth();
-  if (isa<IntegerType>(t) && bitWidth == 8)
-    return ExtSrcKind::F6;
-  if (bitWidth == 4)
-    return ExtSrcKind::F4;
-  return std::nullopt;
-}
-
-static std::optional<ExtDstKind> classifyExtDstType(Type t) {
-  if (t.isF16())
-    return ExtDstKind::F16;
-  if (t.isBF16())
-    return ExtDstKind::BF16;
-  return std::nullopt;
-}
-
-static std::optional<ExtConvOp> lookupExtConvOp(Type srcElemType,
-                                                Type dstElemType) {
-  auto srcKind = classifyExtSrcType(srcElemType);
-  auto dstKind = classifyExtDstType(dstElemType);
+static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
+                                                      Type dstElemType) {
+  auto srcKind = classifyFPType(srcElemType);
+  auto dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kExtTable) {
+  for (const auto &entry : kFPExtTable) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry.convOp;
+      return entry;
   }
   return std::nullopt;
 }
 
-static Type getActualSrcFloatType(MLIRContext *ctx, Type elemType,
-                                  nvgpu::SubBytesPackedKind packedKind) {
-  if (isa<IntegerType>(elemType)) {
-    if (packedKind == nvgpu::SubBytesPackedKind::U6_UNPACK_U8_E3M2)
-      return Float6E3M2FNType::get(ctx);
-    return Float6E2M3FNType::get(ctx);
-  }
-  return elemType;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
 static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
-                                 ExtConvOp convOp, Value src, BoolAttr reluAttr,
-                                 Type actualSrcFloatType,
+                                 FPExtConvOp convOp, Value src,
+                                 BoolAttr reluAttr, Type actualSrcFloatType,
                                  Value extScaleFactor = Value()) {
   IntegerType i32Ty = b.getI32Type();
   auto srcTyAttr = TypeAttr::get(actualSrcFloatType);
 
   switch (convOp) {
-  case ExtConvOp::F8x2_TO_F16x2: {
+  case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F8x2_TO_BF16x2: {
+  case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
         b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F6x2_TO_F16x2: {
+  case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
-  case ExtConvOp::F4x2_TO_F16x2: {
+  case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
         b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
-  llvm_unreachable("unhandled ExtConvOp");
+  llvm_unreachable("unhandled FPExtConvOp");
 }
 
 struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
@@ -2144,8 +2123,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
   LogicalResult
   matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
@@ -2171,10 +2148,8 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     int dstBW = dstType.getElementTypeBitWidth();
     int numElems = srcType.getNumElements();
 
-    auto packedKind = op.getPackedKind();
     auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType =
-        getActualSrcFloatType(ctx, srcElemType, packedKind);
+    Type actualSrcFloatType = srcElemType;
 
     assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
 
@@ -2193,7 +2168,6 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
     // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
     // followed by FPExt to the final f32/f64 destination.
     bool needsFinalFPExt = (dstBW >= 32);
-    // e8m0 must go through bf16 (only available NVVM op); others use f16.
     Type intermediateDstElem = dstElemType;
     if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
       intermediateDstElem = b.getBF16Type();
@@ -2201,20 +2175,30 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
       intermediateDstElem = b.getF16Type();
     int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
 
-    // STEP 1: bitcast input vector to i32 register vector.
-    int srcI32Elems = numElems * srcBW / regBits;
+    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+    // STEP 1: prepare input as i32 register vector.
+    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+    Value inputVec = adaptor.getIn();
+    if (srcBW == 6) {
+      auto i8VecTy = VectorType::get(numElems, i8Ty);
+      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+    }
+
+    int srcI32Elems = numElems * effectiveSrcBW / regBits;
     int dstI32Elems = numElems * intermediateDstBW / regBits;
     Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), adaptor.getIn());
+        VectorType::get(srcI32Elems, i32Ty), inputVec);
     Value dstI32Vec =
         b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
     // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convOpOpt = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convOpOpt)
+    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+    if (!convEntry)
       return rewriter.notifyMatchFailure(
           op, "unsupported type combination for extension");
-    ExtConvOp convOp = *convOpOpt;
+    FPExtConvOp convOp = convEntry->convOp;
     Value extScaleFactor;
 
     // STEP 3: iterate over source i32 elements, producing destination i32s.
@@ -2223,7 +2207,7 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
           srcI32Vec,
           b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
 
-      if (srcBW == 8) {
+      if (effectiveSrcBW == 8) {
         // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
         Value i16Vec =
             b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
@@ -2282,10 +2266,12 @@ struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
 static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
                                   int step) {
   static constexpr int regBits = 32;
+  int effSrcBW = getEffectiveBitWidth(srcBW);
+  int effDstBW = getEffectiveBitWidth(dstBW);
   auto ceilDiv = [](int64_t x, int64_t y) { return (x + y - 1) / y; };
   int64_t padded =
-      std::max(ceilDiv(numElems * srcBW, regBits) * regBits / srcBW,
-               ceilDiv(numElems * dstBW, regBits) * regBits / dstBW);
+      std::max(ceilDiv(numElems * effSrcBW, regBits) * regBits / effSrcBW,
+               ceilDiv(numElems * effDstBW, regBits) * regBits / effDstBW);
   return ceilDiv(padded, step) * step;
 }
 
@@ -2301,8 +2287,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    // Tensor inputs are not handled here; they must be converted to vectors
-    // by other means before this pattern runs.
     if (isa<RankedTensorType>(inType))
       return failure();
 
@@ -2310,12 +2294,14 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
     int dstBW = dstElemTy.getIntOrFloatBitWidth();
+    int effSrcBW = getEffectiveBitWidth(srcBW);
+    int effDstBW = getEffectiveBitWidth(dstBW);
 
     bool isScalar = !isa<VectorType>(inType);
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? srcBW / dstBW : dstBW / srcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2345,12 +2331,11 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
     if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getSatAttr(),
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
     else
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(),
-                          op.getPackedKindAttr(), op.getReluAttr());
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
     Value result = cvt;
 
     if (needsPad)
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 5e3b7c8df515e..7fb68681da4c0 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -742,7 +742,6 @@ LogicalResult FPTruncOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -759,22 +758,6 @@ LogicalResult FPTruncOp::verify() {
     return emitOpError("input type must be 64/32/16 bitwidth, but got ")
            << srcBitWidth;
 
-  if (dstBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact result type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !dstType.isInteger(8))
-    return emitOpError("result type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << dstType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(dstType))
-    return emitOpError("result type expects float type with `compact` packed "
-                       "kind, but got ")
-           << dstType;
-
   if (llvm::isa<Float8E8M0FNUType>(dstType)) {
     if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
         rnd != mlir::NVVM::FPRoundingMode::RP)
@@ -809,7 +792,6 @@ LogicalResult FPExtOp::verify() {
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
   Type dstType = getElementTypeOrSelf(outType);
-  SubBytesPackedKind packedKind = getPackedKind();
   int srcBitWidth = srcType.getIntOrFloatBitWidth();
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
@@ -826,22 +808,6 @@ LogicalResult FPExtOp::verify() {
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
 
-  if (srcBitWidth == 6)
-    return emitOpError("currently doesn't support fp6 compact input type");
-
-  if ((packedKind == SubBytesPackedKind::U6_UNPACK_U8_E3M2 ||
-       packedKind == SubBytesPackedKind::U6_UNPACK_U8_E2M3) &&
-      !srcType.isInteger(8))
-    return emitOpError("input type expects `i8` with `u6_unpack_u8` packed "
-                       "kind, but got ")
-           << srcType;
-
-  if (packedKind == SubBytesPackedKind::COMPACT &&
-      !llvm::isa<FloatType>(srcType))
-    return emitOpError("input type expects float type with `compact` packed "
-                       "kind, but got ")
-           << srcType;
-
   if (llvm::isa<Float8E8M0FNUType>(srcType) &&
       !llvm::isa<BFloat16Type>(dstType) && !dstType.isF32())
     return emitOpError("expects bf16 or f32 output type when input type is "
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 08407fe4bebf1..73aaf63641e97 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -77,9 +77,11 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
 // -----
 
 // CHECK-LABEL: @cvt_float_e2m3_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -90,16 +92,18 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
 // -----
 
 // CHECK-LABEL: @cvt_float_e3m2_to_f16(
-// CHECK: %[[IN:.+]]: vector<8xi8>
-func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
-  // CHECK: %[[IN_I32:.+]] = llvm.bitcast %[[IN]] : vector<8xi8> to vector<2xi32>
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E3M2FN>
+func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E3M2FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
   // CHECK: %[[OUT_I32:.+]] = llvm.mlir.undef : vector<4xi32>
   // CHECK: llvm.bitcast {{.*}} : i32 to vector<2xi16>
   // CHECK: llvm.bitcast {{.*}} : i16 to vector<2xi8>
@@ -113,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xi8>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<8xi8> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -196,13 +200,15 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
 // -----
 
 // CHECK-LABEL: @fpext_e2m3_to_f32(
-// CHECK-SAME: %[[IN:.+]]: vector<4xi8>
-func.func @fpext_e2m3_to_f32(%in : vector<4xi8>) -> vector<4xf32> {
+// CHECK-SAME: %[[IN:.+]]: vector<4xf6E2M3FN>
+func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<4xf6E2M3FN> to vector<4xi6>
+  // CHECK: llvm.zext {{.*}} : vector<4xi6> to vector<4xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -284,11 +290,12 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
 }
 
 // CHECK-LABEL: @fpext_e2m3_to_f64
-func.func @fpext_e2m3_to_f64(%arg0: vector<4xi8>) -> vector<4xf64> {
+func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
+  // CHECK: llvm.zext {{.*}} : vector<8xi6> to vector<8xi8>
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>} : vector<4xi8> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index e8a5c27f22917..bee520a61152a 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -70,6 +70,39 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f16_to_e2m3(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf16>
+func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xf16> to vector<4xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi32>
+  // CHECK: llvm.mlir.undef : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: nvvm.convert.f16x2.to.f6x2
+  // CHECK-SAME: : vector<2xf16> -> i16(f6E2M3FN)
+  // CHECK: llvm.insertelement {{.*}} : vector<2xi16>
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_bf16_to_e3m2(
+// CHECK-SAME: %[[IN:.+]]: vector<8xbf16>
+func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
+  // CHECK: llvm.bitcast %[[IN]] : vector<8xbf16> to vector<4xi32>
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
+  // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  return
+}
+
 // CHECK-LABEL: @cvt_float_f32_to_e2m3(
 // CHECK-SAME: %[[IN:.+]]: vector<8xf32>
 func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
@@ -90,7 +123,8 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e2m3>}: vector<8xf32> to vector<8xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index a2f8fd6ec48f8..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
 
 // -----
 
-func.func @fpext_fp6_compact(%in : vector<16xf6E2M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op currently doesn't support fp6 compact input type}}
-  %out = nvgpu.convert.fpext %in : vector<16xf6E2M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_u6_packed_not_i8(%in : vector<16xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fpext %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf8E4M3FN> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_compact_not_float(%in : vector<16xi8>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fpext %in : vector<16xi8> to vector<16xf16>
-  return
-}
-
-// -----
-
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
   // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
   %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 9e295b4ad25a0..ff899d341c99a 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -18,30 +18,6 @@ func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
 
 // -----
 
-func.func @fptrunc_fp6_compact(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op currently doesn't support fp6 compact result type}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xf6E2M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_u6_packed_not_i8(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects `i8` with `u6_unpack_u8` packed kind, but got 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {packed_kind = #nvgpu.subbytes_packedkind<u6_unpack_u8_e3m2>} : vector<16xf32> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fptrunc_compact_not_float(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type expects float type with `compact` packed kind, but got 'i8'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<16xi8>
-  return
-}
-
-// -----
-
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
   // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
   %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}

>From 16d21ca1d5b31bed58998ff867d95ee20d20913e Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Wed, 27 May 2026 14:06:28 +0000
Subject: [PATCH 04/12] address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 19 +++--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  3 +
 .../nvgpu-convert-fpext-large.mlir            | 78 +++++++++++++++++++
 .../nvgpu-convert-fptrunc-large.mlir          | 76 ++++++++++++++++++
 4 files changed, 170 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 4ffca888c0260..14929e6187d54 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -684,8 +684,10 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
-    Supported paths: f32->f16, f32->bf16, f32->f8, f32->f6, f32->f4,
-    f16->f8, f16->f6, f16->f4, bf16->f8, bf16->f6, bf16->f4.
+    Supported paths:
+      f32 -> f16, bf16, f8, f6, f4
+      f16 -> f8, f6, f4
+      bf16 -> f8, f6, f4
 
     The `random_bits` operand enables stochastic rounding (RS mode)
     for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
@@ -715,10 +717,15 @@ def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Extend a floating-point value to a wider floating-point type.
     Destination must be strictly wider than source.
 
-    Supported paths: f8->f16, f8->bf16, f6->f16, f4->f16, f16->f32,
-    bf16->f32. Narrow->f32 uses two-step lowering (NVVM op to f16/bf16
-    intermediate, then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported
-    (f32 goes through bf16 intermediate).
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16
+      f4 -> f16
+      f16 -> f32
+      bf16 -> f32
+    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
+    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
+    bf16 intermediate).
 
     Example:
     ```mlir
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 99ab025a6a27e..46c7783d9f43c 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,6 +1807,9 @@ static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
+  auto vecTy = cast<VectorType>(srcVec.getType());
+  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+         "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(
       srcVec, b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(idx)));
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
new file mode 100644
index 0000000000000..35379861f8f5d
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -0,0 +1,78 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fpext
+
+// CHECK-LABEL: @cvt_large_f8_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
+func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf8E4M3FN> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_e8m0_to_bf16(
+func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf8E8M0FNU> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf6E2M3FN>
+func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %[[IN]] : vector<400xf6E2M3FN> to vector<400xi6>
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi8> to vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f4_to_f16(
+func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
+  // CHECK: builtin.unrealized_conversion_cast %{{.*}} : vector<400xf4E2M1FN> to vector<400xi4>
+  // CHECK: llvm.bitcast {{.*}} : vector<400xi4> to vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f8_to_f32(
+func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
+  // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f6_to_f32(
+func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
+  // CHECK: llvm.zext {{.*}} : vector<400xi6> to vector<400xi8>
+  // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
+  // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  return %out : vector<400xf32>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f32(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf16>
+func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
+  // CHECK-NOT: nvvm.convert
+  // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  return %out : vector<400xf32>
+}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
new file mode 100644
index 0000000000000..d6c862fa5a7a6
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
+
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
+
+// CHECK-LABEL: @cvt_large_f32_to_f16(
+// CHECK-SAME: %[[IN:.+]]: vector<400xf32>
+func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
+  // CHECK: llvm.bitcast %[[IN]] : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  return %out : vector<400xf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_bf16(
+func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  return %out : vector<400xbf16>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f8(
+func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<100xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f6(
+func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
+  // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  return %out : vector<400xf6E2M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_f32_to_f4(
+func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf32> to vector<400xi32>
+  // CHECK: llvm.mlir.undef : vector<50xi32>
+  // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
+  // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  return %out : vector<400xf4E2M1FN>
+}
+
+// CHECK-LABEL: @cvt_large_f16_to_f8(
+func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
+  // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  return %out : vector<400xf8E4M3FN>
+}
+
+// CHECK-LABEL: @cvt_large_bf16_to_f6(
+func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN> {
+  // CHECK: llvm.bitcast %{{.*}} : vector<400xbf16> to vector<200xi32>
+  // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
+  // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
+  // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  return %out : vector<400xf6E3M2FN>
+}

>From 8345ebf05e15824210ea60c40901795f2c5dd69f Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 11 Jun 2026 12:49:26 +0000
Subject: [PATCH 05/12] combine Ops and address comments

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  73 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 589 +++++++++---------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 103 ++-
 .../nvgpu-convert-fpext-large.mlir            |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  99 ++-
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +-
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 106 +++-
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 +-
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  77 ++-
 9 files changed, 611 insertions(+), 520 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 14929e6187d54..0c5bd29e38336 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,32 +677,47 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
-    [Pure]> {
-  let summary = "Truncate floating-point to narrower floating-point";
+// nvgpu.convert.float only supports the satfinite and none saturation modes.
+def NVGPU_SaturationModeSatfiniteOrNone :
+  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
+                [SaturationModeNone, SaturationModeFinite]>]>;
+
+def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
+  let summary = "Convert between floating-point types of different widths";
   let description = [{
-    Truncate a floating-point value to a smaller floating-point type.
-    Destination must be strictly narrower than source.
+    Convert a floating-point value to a floating-point type of a different
+    width. The direction is inferred from the bitwidths: a narrower result is a
+    truncation, a wider result is an extension. Source and destination must have
+    different bitwidths.
 
-    Supported paths:
+    Supported truncation paths:
+      f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    The `random_bits` operand enables stochastic rounding (RS mode)
-    for f32->f16/bf16 conversions. When provided, `rnd` must be RS.
+    Supported extension paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    The `sat` and `random_bits` operands apply to truncation only. The
+    `random_bits` operand enables stochastic rounding (RS mode) for
+    f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.cvt_fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.cvt_fptrunc %in : f32 to f16
-    %r = nvgpu.cvt_fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.float %in : f32 to f16
+    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<SaturationModeAttr, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -711,36 +726,4 @@ def NVGPU_FPTruncOp : NVGPU_Op<"convert.fptrunc",
   let hasVerifier = 1;
 }
 
-def NVGPU_FPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
-  let summary = "Extend floating-point to wider floating-point";
-  let description = [{
-    Extend a floating-point value to a wider floating-point type.
-    Destination must be strictly wider than source.
-
-    Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16
-      f4 -> f16
-      f16 -> f32
-      bf16 -> f32
-    Narrow->f32 uses two-step lowering (NVVM op to f16/bf16 intermediate,
-    then LLVM FPExt). e8m0->bf16 and e8m0->f32 supported (f32 goes through
-    bf16 intermediate).
-
-    Example:
-    ```mlir
-    %r = nvgpu.cvt_fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.cvt_fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.cvt_fpext %in : f8E4M3FN to f32
-    ```
-  }];
-  let arguments = (ins FloatLike:$in,
-                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
-  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
-  let hasVerifier = 1;
-}
-
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 46c7783d9f43c..b4443b30c9d7a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPTruncOp Lowering
+// NVGPUConvertFloatOp Lowering (truncation)
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,147 +1916,144 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-struct NVGPUFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::FPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPTruncOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::FPTruncOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
+                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                  ConversionPatternRewriter &rewriter,
+                                  const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
+  static constexpr int regBits = 32;
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-    static constexpr int regBits = 32;
-
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
-    NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
-    auto reluBoolAttr = op.getReluAttr();
-    Value randomBits = adaptor.getRandomBits();
-    Type actualDstFloatType = dstElemType;
-
-    // STEP 1: bitcast input vector to i32 register vector.
-    // f64 source: decompose to f64->f32 (LLVM fptrunc), then lower f32->dst.
-    Value input = adaptor.getIn();
-    if (srcBW == 64) {
-      auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
-      input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
-      srcType = f32VecTy;
-      srcElemType = b.getF32Type();
-      srcBW = 32;
-      if (dstBW == 32) {
-        rewriter.replaceOp(op, input);
-        return success();
-      }
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  NVVM::FPRoundingModeAttr rndModeAttr = op.getRndAttr();
+  NVVM::SaturationModeAttr satModeAttr = op.getSatAttr();
+  auto reluBoolAttr = op.getReluAttr();
+  Value randomBits = adaptor.getRandomBits();
+  Type actualDstFloatType = dstElemType;
+
+  // STEP 1: bitcast input vector to i32 register vector.
+  // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
+  // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
+  //        conversion path below.
+  Value input = adaptor.getIn();
+  if (srcBW == 64) {
+    if (dstBW >= 16) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      Value result = b.create<LLVM::FPTruncOp>(convertedType, input);
+      rewriter.replaceOp(op, result);
+      return success();
     }
+    auto f32VecTy = VectorType::get(srcType.getShape(), b.getF32Type());
+    input = b.create<LLVM::FPTruncOp>(f32VecTy, input);
+    srcType = f32VecTy;
+    srcElemType = b.getF32Type();
+    srcBW = 32;
+  }
 
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveDstBW = getEffectiveBitWidth(dstBW);
-
-    int srcI32Elems = numElems * srcBW / regBits;
-    int dstI32Elems = numElems * effectiveDstBW / regBits;
-    Value srcI32Vec =
-        b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
-
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
-    if (!convEntry)
-      return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for truncation");
-    FPTruncConvOp convOp = convEntry->convOp;
-    int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
-
-    // STEP 3: pack conversion results into destination i32 vector.
-    const int srcStep = srcBW / effectiveDstBW;
-    const int resultBW =
-        effectiveDstBW * 2; // each conversion produces 2 (packed) elements
-    const int numConvsPerI32 = regBits / resultBW;
-
-    for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
-         srcIdx += srcStep, dstIdx++) {
-      Value dstIdxConst =
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-      Value dstValue;
-
-      if (numConvsPerI32 == 1) {
-        // f16/bf16 destinations
-        dstValue = createTruncConversion(
-            b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
-            reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
-      } else {
-        // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
-        auto subResultType = IntegerType::get(ctx, resultBW);
-        auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
-        Value subVec = b.create<LLVM::UndefOp>(subVecTy);
-
-        int insertIdx = numConvsPerI32 - 1;
-        int curStep = srcStep;
-        while (curStep > 0) {
-          curStep -= numSrcI32PerConv;
-          Value subResult = createTruncConversion(
-              b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
-              satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
-              /*randomBits=*/Value());
-          subVec = b.create<LLVM::InsertElementOp>(
-              subVec, subResult,
-              b.create<LLVM::ConstantOp>(i64Ty,
-                                         b.getI64IntegerAttr(insertIdx)));
-          insertIdx--;
-        }
-
-        dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveDstBW = getEffectiveBitWidth(dstBW);
+
+  int srcI32Elems = numElems * srcBW / regBits;
+  int dstI32Elems = numElems * effectiveDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), input);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for truncation");
+  FPTruncConvOp convOp = convEntry->convOp;
+  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+
+  // STEP 3: pack conversion results into destination i32 vector.
+  const int srcStep = srcBW / effectiveDstBW;
+  const int resultBW =
+      effectiveDstBW * 2; // each conversion produces 2 (packed) elements
+  const int numConvsPerI32 = regBits / resultBW;
+
+  for (int srcIdx = 0, dstIdx = 0; dstIdx < dstI32Elems;
+       srcIdx += srcStep, dstIdx++) {
+    Value dstIdxConst =
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+    Value dstValue;
+
+    if (numConvsPerI32 == 1) {
+      // f16/bf16 destinations
+      dstValue = createTruncConversion(
+          b, ctx, convOp, srcI32Vec, srcIdx, rndModeAttr, satModeAttr,
+          reluBoolAttr, dstElemType, actualDstFloatType, randomBits);
+    } else {
+      // f8/f6/f4 destinations: pack sub-results via vector insert + bitcast.
+      auto subResultType = IntegerType::get(ctx, resultBW);
+      auto subVecTy = VectorType::get(numConvsPerI32, subResultType);
+      Value subVec = b.create<LLVM::UndefOp>(subVecTy);
+
+      int insertIdx = numConvsPerI32 - 1;
+      int curStep = srcStep;
+      while (curStep > 0) {
+        curStep -= numSrcI32PerConv;
+        Value subResult = createTruncConversion(
+            b, ctx, convOp, srcI32Vec, srcIdx + curStep, rndModeAttr,
+            satModeAttr, reluBoolAttr, dstElemType, actualDstFloatType,
+            /*randomBits=*/Value());
+        subVec = b.create<LLVM::InsertElementOp>(
+            subVec, subResult,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(insertIdx)));
+        insertIdx--;
       }
 
-      dstI32Vec =
-          b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+      dstValue = b.create<LLVM::BitcastOp>(i32Ty, subVec);
     }
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    if (convEntry->dst == FPKind::F6) {
-      IntegerType i8Ty = b.getI8Type();
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
-      Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
-      rewriter.replaceOp(op, truncVec);
-    } else {
-      auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-      rewriter.replaceOp(op, dstVec);
-    }
-    return success();
+    dstI32Vec =
+        b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
   }
-};
+
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  if (convEntry->dst == FPKind::F6) {
+    IntegerType i8Ty = b.getI8Type();
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    Value i8Vec = b.create<LLVM::BitcastOp>(i8VecTy, dstI32Vec);
+    Value truncVec = b.create<LLVM::TruncOp>(convertedType, i8Vec);
+    rewriter.replaceOp(op, truncVec);
+  } else {
+    auto dstVec = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+    rewriter.replaceOp(op, dstVec);
+  }
+  return success();
+}
 
 //===----------------------------------------------------------------------===//
-// NVGPUFPExtOpLowering
+// NVGPUConvertFloatOp Lowering (extension)
 //===----------------------------------------------------------------------===//
 
-/// Extension conversion op identifier for nvgpu.convert.fpext lowering.
+/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
   F6x2_TO_F16x2,
+  F6x2_TO_BF16x2,
   F4x2_TO_F16x2,
+  F4x2_TO_BF16x2,
 };
 
 struct FPExtTableEntry {
@@ -2069,7 +2066,9 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F8, FPKind::F16, FPExtConvOp::F8x2_TO_F16x2},
     {FPKind::F8, FPKind::BF16, FPExtConvOp::F8x2_TO_BF16x2},
     {FPKind::F6, FPKind::F16, FPExtConvOp::F6x2_TO_F16x2},
+    {FPKind::F6, FPKind::BF16, FPExtConvOp::F6x2_TO_BF16x2},
     {FPKind::F4, FPKind::F16, FPExtConvOp::F4x2_TO_F16x2},
+    {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
 static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
@@ -2098,171 +2097,195 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   switch (convOp) {
   case FPExtConvOp::F8x2_TO_F16x2: {
     Value r = NVVM::ConvertF8x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F8x2_TO_BF16x2: {
     Value r = NVVM::ConvertF8x2ToBF16x2Op::create(
-        b, VectorType::get(2, b.getBF16Type()), src, srcTyAttr);
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F6x2_TO_F16x2: {
     Value r = NVVM::ConvertF6x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F6x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF6x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   case FPExtConvOp::F4x2_TO_F16x2: {
     Value r = NVVM::ConvertF4x2ToF16x2Op::create(
-        b, VectorType::get(2, b.getF16Type()), src, reluAttr, srcTyAttr);
+        b, VectorType::get(2, b.getF16Type()), src, srcTyAttr, reluAttr);
+    return b.create<LLVM::BitcastOp>(i32Ty, r);
+  }
+  case FPExtConvOp::F4x2_TO_BF16x2: {
+    Value r = NVVM::ConvertF4x2ToBF16x2Op::create(
+        b, VectorType::get(2, b.getBF16Type()), src, extScaleFactor, srcTyAttr);
     return b.create<LLVM::BitcastOp>(i32Ty, r);
   }
   }
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-struct NVGPUFPExtOpLowering : public ConvertOpToLLVMPattern<nvgpu::FPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::FPExtOp>::ConvertOpToLLVMPattern;
+static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
+                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+                                ConversionPatternRewriter &rewriter,
+                                const LLVMTypeConverter *typeConverter) {
+  MLIRContext *ctx = op.getContext();
+  ImplicitLocOpBuilder b(op->getLoc(), rewriter);
+  IntegerType i8Ty = b.getI8Type();
+  IntegerType i16Ty = b.getI16Type();
+  IntegerType i32Ty = b.getI32Type();
+  IntegerType i64Ty = b.getI64Type();
 
-  LogicalResult
-  matchAndRewrite(nvgpu::FPExtOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
+  static constexpr int regBits = 32;
+  auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
+  auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
+  if (!srcType || srcType.getRank() != 1 || !dstType || dstType.getRank() != 1)
+    return rewriter.notifyMatchFailure(
+        op, "expected 1-D vector; canonicalize pattern handles other shapes");
+
+  auto srcElemType = srcType.getElementType();
+  auto dstElemType = dstType.getElementType();
+  int srcBW = srcType.getElementTypeBitWidth();
+  int dstBW = dstType.getElementTypeBitWidth();
+  int numElems = srcType.getNumElements();
+
+  auto reluBoolAttr = op.getReluAttr();
+  Type actualSrcFloatType = srcElemType;
+
+  assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
+
+  // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
+  if (srcBW >= 16 && dstBW >= 32) {
+    Value result = adaptor.getIn();
+    if (srcElemType != dstElemType) {
+      Type convertedType = typeConverter->convertType(dstType);
+      assert(convertedType && "failed to convert type");
+      result = b.create<LLVM::FPExtOp>(convertedType, result);
+    }
+    rewriter.replaceOp(op, result);
+    return success();
+  }
 
-    MLIRContext *ctx = getContext();
-    ImplicitLocOpBuilder b(op->getLoc(), rewriter);
-    IntegerType i8Ty = b.getI8Type();
-    IntegerType i16Ty = b.getI16Type();
-    IntegerType i32Ty = b.getI32Type();
-    IntegerType i64Ty = b.getI64Type();
-
-    static constexpr int regBits = 32;
-    auto srcType = llvm::dyn_cast<VectorType>(op.getIn().getType());
-    auto dstType = llvm::dyn_cast<VectorType>(op.getOut().getType());
-    if (!srcType || srcType.getRank() != 1 || !dstType ||
-        dstType.getRank() != 1)
-      return rewriter.notifyMatchFailure(
-          op, "expected 1-D vector; canonicalize pattern handles other shapes");
-
-    auto srcElemType = srcType.getElementType();
-    auto dstElemType = dstType.getElementType();
-    int srcBW = srcType.getElementTypeBitWidth();
-    int dstBW = dstType.getElementTypeBitWidth();
-    int numElems = srcType.getNumElements();
-
-    auto reluBoolAttr = op.getReluAttr();
-    Type actualSrcFloatType = srcElemType;
-
-    assert(dstBW == 16 || dstBW == 32 || dstBW == 64);
-
-    // Wide source (f16/bf16/f32) to wide destination (f32/f64): single FPExt.
-    if (srcBW >= 16 && dstBW >= 32) {
-      Value result = adaptor.getIn();
-      if (srcElemType != dstElemType) {
-        Type convertedType = getTypeConverter()->convertType(dstType);
-        assert(convertedType && "failed to convert type");
-        result = b.create<LLVM::FPExtOp>(convertedType, result);
+  // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
+  // followed by FPExt to the final f32/f64 destination.
+  bool needsFinalFPExt = (dstBW >= 32);
+  Type intermediateDstElem = dstElemType;
+  if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
+    intermediateDstElem = b.getBF16Type();
+  else if (needsFinalFPExt)
+    intermediateDstElem = b.getF16Type();
+  int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
+
+  // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
+  int effectiveSrcBW = getEffectiveBitWidth(srcBW);
+
+  // STEP 1: prepare input as i32 register vector.
+  // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
+  Value inputVec = adaptor.getIn();
+  if (srcBW == 6) {
+    auto i8VecTy = VectorType::get(numElems, i8Ty);
+    inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
+  }
+
+  int srcI32Elems = numElems * effectiveSrcBW / regBits;
+  int dstI32Elems = numElems * intermediateDstBW / regBits;
+  Value srcI32Vec =
+      b.create<LLVM::BitcastOp>(VectorType::get(srcI32Elems, i32Ty), inputVec);
+  Value dstI32Vec =
+      b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+
+  // STEP 2: look up the conversion op from the (srcType, dstType) table.
+  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  if (!convEntry)
+    return rewriter.notifyMatchFailure(
+        op, "unsupported type combination for extension");
+  FPExtConvOp convOp = convEntry->convOp;
+  Value extScaleFactor;
+
+  // STEP 3: iterate over source i32 elements, producing destination i32s.
+  for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
+    Value srcI32 = b.create<LLVM::ExtractElementOp>(
+        srcI32Vec,
+        b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
+
+    if (effectiveSrcBW == 8) {
+      // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
+      Value i16Vec =
+          b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
+      for (int half = 0; half < 2; half++) {
+        Value halfI16 = b.create<LLVM::ExtractElementOp>(
+            i16Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
+        Value src =
+            b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
+      }
+    } else {
+      // f4: one i32 holds 4 bytes -> each byte is one conversion input.
+      Value i8Vec = b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
+      for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
+        Value src = b.create<LLVM::ExtractElementOp>(
+            i8Vec,
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
+        Value dstValue =
+            createExtConversion(b, ctx, convOp, src, reluBoolAttr,
+                                actualSrcFloatType, extScaleFactor);
+        Value dstIdxConst =
+            b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
+        dstI32Vec =
+            b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
+        dstIdx++;
       }
-      rewriter.replaceOp(op, result);
-      return success();
     }
+  }
 
-    // Narrow source (f8/f6/f4): NVVM typed op produces f16/bf16; optionally
-    // followed by FPExt to the final f32/f64 destination.
-    bool needsFinalFPExt = (dstBW >= 32);
-    Type intermediateDstElem = dstElemType;
-    if (needsFinalFPExt && llvm::isa<Float8E8M0FNUType>(srcElemType))
-      intermediateDstElem = b.getBF16Type();
-    else if (needsFinalFPExt)
-      intermediateDstElem = b.getF16Type();
-    int intermediateDstBW = needsFinalFPExt ? 16 : dstBW;
-
-    // f6 types are 6-bit in MLIR but NVVM uses 8-bit containers.
-    int effectiveSrcBW = getEffectiveBitWidth(srcBW);
-
-    // STEP 1: prepare input as i32 register vector.
-    // For f6: zext from vector<Nxi6> to vector<Nxi8>, then bitcast to i32s.
-    Value inputVec = adaptor.getIn();
-    if (srcBW == 6) {
-      auto i8VecTy = VectorType::get(numElems, i8Ty);
-      inputVec = b.create<LLVM::ZExtOp>(i8VecTy, inputVec);
-    }
+  // STEP 4: produce final result.
+  Type convertedType = typeConverter->convertType(dstType);
+  assert(convertedType && "failed to convert type");
+  Value result;
+  if (needsFinalFPExt) {
+    auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
+    Value intermediateVec =
+        b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
+    result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
+  } else {
+    result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
+  }
+  rewriter.replaceOp(op, result);
+  return success();
+}
 
-    int srcI32Elems = numElems * effectiveSrcBW / regBits;
-    int dstI32Elems = numElems * intermediateDstBW / regBits;
-    Value srcI32Vec = b.create<LLVM::BitcastOp>(
-        VectorType::get(srcI32Elems, i32Ty), inputVec);
-    Value dstI32Vec =
-        b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
+/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
+/// helper based on the source/destination bitwidths.
+struct NVGPUConvertFloatOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
 
-    // STEP 2: look up the conversion op from the (srcType, dstType) table.
-    auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
-    if (!convEntry)
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
-          op, "unsupported type combination for extension");
-    FPExtConvOp convOp = convEntry->convOp;
-    Value extScaleFactor;
-
-    // STEP 3: iterate over source i32 elements, producing destination i32s.
-    for (int srcIdx = 0, dstIdx = 0; srcIdx < srcI32Elems; srcIdx++) {
-      Value srcI32 = b.create<LLVM::ExtractElementOp>(
-          srcI32Vec,
-          b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(srcIdx)));
-
-      if (effectiveSrcBW == 8) {
-        // f8/f6: one i32 holds 4 bytes -> split into 2 pairs of i16 -> 2 convs.
-        Value i16Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(2, i16Ty), srcI32);
-        for (int half = 0; half < 2; half++) {
-          Value halfI16 = b.create<LLVM::ExtractElementOp>(
-              i16Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(half)));
-          Value src =
-              b.create<LLVM::BitcastOp>(VectorType::get(2, i8Ty), halfI16);
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      } else {
-        // f4: one i32 holds 4 bytes -> each byte is one conversion input.
-        Value i8Vec =
-            b.create<LLVM::BitcastOp>(VectorType::get(4, i8Ty), srcI32);
-        for (int byteIdx = 0; byteIdx < 4; byteIdx++) {
-          Value src = b.create<LLVM::ExtractElementOp>(
-              i8Vec,
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(byteIdx)));
-          Value dstValue =
-              createExtConversion(b, ctx, convOp, src, reluBoolAttr,
-                                  actualSrcFloatType, extScaleFactor);
-          Value dstIdxConst =
-              b.create<LLVM::ConstantOp>(i64Ty, b.getI64IntegerAttr(dstIdx));
-          dstI32Vec =
-              b.create<LLVM::InsertElementOp>(dstI32Vec, dstValue, dstIdxConst);
-          dstIdx++;
-        }
-      }
-    }
+          op, "tensor inputs not handled; type converter should lower first");
 
-    // STEP 4: produce final result.
-    Type convertedType = getTypeConverter()->convertType(dstType);
-    assert(convertedType && "failed to convert type");
-    Value result;
-    if (needsFinalFPExt) {
-      auto intermediateVecTy = VectorType::get(numElems, intermediateDstElem);
-      Value intermediateVec =
-          b.create<LLVM::BitcastOp>(intermediateVecTy, dstI32Vec);
-      result = b.create<LLVM::FPExtOp>(convertedType, intermediateVec);
-    } else {
-      result = b.create<LLVM::BitcastOp>(convertedType, dstI32Vec);
-    }
-    rewriter.replaceOp(op, result);
-    return success();
+    int srcBW =
+        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
+    int dstBW =
+        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
+    if (srcBW > dstBW)
+      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2278,14 +2301,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
-/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
-/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
-template <typename CvtOp, bool IsTrunc>
-struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
-  using OpRewritePattern<CvtOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
+/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
+/// OpRewritePattern on MLIR types before LLVM type conversion.
+struct NVGPUConvertFloatCanonicalizePattern
+    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
+  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(CvtOp op,
+  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2304,7 +2327,7 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2332,14 +2355,9 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value cvt;
-    if constexpr (IsTrunc)
-      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
-                          op.getReluAttr(), op.getRandomBits());
-    else
-      cvt =
-          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
-    Value result = cvt;
+    Value result = nvgpu::ConvertFloatOp::create(
+        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
+        op.getRandomBits());
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2357,11 +2375,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     return success();
   }
 };
-
-using NVGPUFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPTruncOp, true>;
-using NVGPUFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::FPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2406,12 +2419,10 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUFPTruncOpLowering,                     // nvgpu.convert.fptrunc
-      NVGPUFPExtOpLowering,                       // nvgpu.convert.fpext
+      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUFPTruncCanonicalizePattern, NVGPUFPExtCanonicalizePattern>(
-      patterns.getContext());
+  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 7fb68681da4c0..8c96eb7843de4 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_CvtFPTruncOp
+// NVGPU_ConvertFloatOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult FPTruncOp::verify() {
+LogicalResult ConvertFloatOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,60 +750,54 @@ LogicalResult FPTruncOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth <= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be narrower than operand type " << srcType;
-
-  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-           << srcBitWidth;
-
-  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-        rnd != mlir::NVVM::FPRoundingMode::RP)
-      return emitOpError("expects RZ or RP rounding mode when result type is "
-                         "e8m0, but got ")
-             << getRndAttr();
-  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-    // TODO: Currently, we only support conversions which fit into a single i32
-    // register. Support f32->f8/f6/f4 conversions with RS rounding.
-    if (!(srcBitWidth == 32 && (dstBitWidth == 16)))
-      return emitOpError("RS (stochastic) rounding is only supported for "
-                         "f32->f16/bf16, got ")
-             << srcType << " -> " << dstType;
-    if (!getRandomBits())
-      return emitOpError("random_bits operand is required with RS rounding");
-  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-  }
-
-  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-    return emitOpError("random_bits can only be used with RS rounding mode");
-
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// NVGPU_CvtFPExtOp
-//===----------------------------------------------------------------------===//
-
-LogicalResult FPExtOp::verify() {
-  Type inType = getIn().getType();
-  Type outType = getType();
-  Type srcType = getElementTypeOrSelf(inType);
-  Type dstType = getElementTypeOrSelf(outType);
-  int srcBitWidth = srcType.getIntOrFloatBitWidth();
-  int dstBitWidth = dstType.getIntOrFloatBitWidth();
-  auto rnd = getRnd();
+  if (srcBitWidth == dstBitWidth)
+    return emitOpError("input and output types must have different bitwidths, "
+                       "got ")
+           << srcType << " and " << dstType;
+
+  // Truncation: destination is narrower than source.
+  if (srcBitWidth > dstBitWidth) {
+    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+             << srcBitWidth;
+
+    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+          rnd != mlir::NVVM::FPRoundingMode::RP)
+        return emitOpError("expects RZ or RP rounding mode when result type is "
+                           "e8m0, but got ")
+               << getRndAttr();
+    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+      // TODO: Currently, we only support conversions which fit into a single
+      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
+      if (!(srcBitWidth == 32 && dstBitWidth == 16))
+        return emitOpError("RS (stochastic) rounding is only supported for "
+                           "f32->f16/bf16, got ")
+               << srcType << " -> " << dstType;
+      if (!getRandomBits())
+        return emitOpError("random_bits operand is required with RS rounding");
+    } else if (srcType.isF64() && dstBitWidth >= 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN)
+        return emitOpError("expects RN rounding mode for f64 input, but got ")
+               << getRndAttr();
+    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+          rnd != mlir::NVVM::FPRoundingMode::RZ)
+        return emitOpError(
+                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
+                   "but got ")
+               << getRndAttr();
+    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+    }
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+      return emitOpError("random_bits can only be used with RS rounding mode");
 
-  if (srcBitWidth >= dstBitWidth)
-    return emitOpError("result type ")
-           << dstType << " must be wider than operand type " << srcType;
+    return success();
+  }
 
+  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -819,6 +813,9 @@ LogicalResult FPExtOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
+  if (getRandomBits())
+    return emitOpError("random_bits is only supported for truncation");
+
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index 35379861f8f5d..c5a4f37cdf949 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.convert.float
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index 73aaf63641e97..dcf090955df3c 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,10 +150,43 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m3_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf6E2M3FN>
+func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<8xf6E2M3FN> to vector<8xi6>
+  // CHECK: llvm.zext %[[CAST]] : vector<8xi6> to vector<8xi8>
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi8> to vector<2xi32>
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK-SAME: : vector<2xi8>(f6E2M3FN)
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: nvvm.convert.f6x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
+  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @cvt_float_e2m1_to_bf16(
+// CHECK-SAME: %[[IN:.+]]: vector<16xf4E2M1FN>
+func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
+  // CHECK: %[[CAST:.*]] = builtin.unrealized_conversion_cast %[[IN]] : vector<16xf4E2M1FN> to vector<16xi4>
+  // CHECK: llvm.bitcast %[[CAST]] : vector<16xi4> to vector<2xi32>
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK-SAME: : i8(f4E2M1FN)
+  // CHECK: nvvm.convert.f4x2.to.bf16x2
+  // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  return
+}
+
 // Extension to f32 (two-step: narrow -> f16/bf16 -> f32).
 
 // -----
@@ -167,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -180,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -193,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -208,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -220,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -230,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -240,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -250,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -258,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -266,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -276,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -285,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -295,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -324,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -336,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -349,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -361,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -374,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -388,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -398,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -414,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.convert.float %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -423,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -433,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -447,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.convert.float %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -457,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -468,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..2216f0e9b5c80 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.convert.float.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index bee520a61152a..8eaccebf56e00 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,10 +124,32 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rz(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
+// CHECK-LABEL: @cvt_float_f32_to_e8m0_rp(
+// CHECK-SAME: %[[IN:.+]]: vector<8xf32>
+func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f8x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
+  // CHECK-SAME: : i16(f8E8M0FNU)
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+      : vector<8xf32> to vector<8xf8E8M0FNU>
+  return
+}
+
 // Scalar inputs (canonicalize: broadcast + pad + extract).
 
 // CHECK-LABEL: @fptrunc_scalar_f32_to_f16
@@ -138,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -148,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.convert.float %in : f32 to bf16
   return %out : bf16
 }
 
@@ -157,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.convert.float %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -169,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -179,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -191,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -201,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -215,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -225,27 +247,27 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_f16
 func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
 // CHECK-LABEL: @fptrunc_f64_to_bf16
 func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: vector.insert_strided_slice
-  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
-  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
+  // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -255,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -265,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -274,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.convert.float %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -285,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -293,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -301,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -312,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -321,7 +343,25 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+      : vector<4xf32> to vector<4xbf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_f16_rz
+func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.f16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf32> to vector<4xf16>
+  return
+}
+
+// CHECK-LABEL: @fptrunc_f32_to_bf16_rz
+func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
+  // CHECK: nvvm.convert.f32x2.to.bf16x2
+  // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -337,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.convert.float %in : f32 to f16
   return %out : f16
 }
 
@@ -346,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -356,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 26c3afc5da358..9e1798456379f 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,33 +2,25 @@
 
 // -----
 
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
 func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
 func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf8E5M2> to vector<16xf16>
   return
 }
@@ -36,47 +28,55 @@ func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
+  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index ff899d341c99a..2ef54221e4e21 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,34 +2,61 @@
 
 // -----
 
-func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
+  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
 
 // -----
 
+func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+      : vector<8xf32> to vector<8xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf32> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<4xf64> to vector<4xf16>
+  return
+}
+
+// -----
+
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -37,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -46,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -55,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.float %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -64,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
+  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From d68e1aa32f62c683726d2827460e805bc4772968 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 15 Jun 2026 13:57:39 +0000
Subject: [PATCH 06/12] split Op back to convert.fptrunc and convert.fpext

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  61 ++++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  83 ++++++++-----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 113 ++++++++++--------
 .../nvgpu-convert-fpext-large.mlir            |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fpext.mlir      |  70 +++++------
 .../nvgpu-convert-fptrunc-large.mlir          |  16 +--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    |  66 +++++-----
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    |  52 ++++----
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  |  62 +++++-----
 9 files changed, 294 insertions(+), 245 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 0c5bd29e38336..9f27db4c9dcc2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,42 +677,32 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.float only supports the satfinite and none saturation modes.
+// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
 def NVGPU_SaturationModeSatfiniteOrNone :
   ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
                 [SaturationModeNone, SaturationModeFinite]>]>;
 
-def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
-  let summary = "Convert between floating-point types of different widths";
+def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+  let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
-    Convert a floating-point value to a floating-point type of a different
-    width. The direction is inferred from the bitwidths: a narrower result is a
-    truncation, a wider result is an extension. Source and destination must have
-    different bitwidths.
+    Truncate a floating-point value to a smaller floating-point type.
+    Destination must be strictly narrower than source.
 
-    Supported truncation paths:
+    Supported paths:
       f64 -> f32, f16, bf16
       f32 -> f16, bf16, f8, f6, f4
       f16 -> f8, f6, f4
       bf16 -> f8, f6, f4
 
-    Supported extension paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
-      f16 -> f32
-      bf16 -> f32
-
-    The `sat` and `random_bits` operands apply to truncation only. The
-    `random_bits` operand enables stochastic rounding (RS mode) for
+    The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.float %in : f32 to f16
-    %r = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.convert.fptrunc %in : f32 to f16
+    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
@@ -726,4 +716,33 @@ def NVGPU_ConvertFloatOp : NVGPU_Op<"convert.float", [Pure]> {
   let hasVerifier = 1;
 }
 
+def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+  let summary = "Extend floating-point to wider floating-point";
+  let description = [{
+    Extend a floating-point value to a wider floating-point type.
+    Destination must be strictly wider than source.
+
+    Supported paths:
+      f8 -> f16, bf16
+      f6 -> f16, bf16
+      f4 -> f16, bf16
+      f16 -> f32
+      bf16 -> f32
+
+    Example:
+    ```mlir
+    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    ```
+  }];
+  let arguments = (ins FloatLike:$in,
+                       DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
+                       DefaultValuedAttr<BoolAttr, "false">:$relu);
+  let results = (outs FloatLike:$out);
+  let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
+  let hasVerifier = 1;
+}
+
 #endif // MLIR_DIALECT_NVGPU_IR_NVGPUOPS_TD
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index b4443b30c9d7a..abd6ac7ff8fe4 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,7 +1713,7 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (truncation)
+// NVGPUConvertFPTruncOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
@@ -1754,7 +1754,7 @@ static int getNumSrcI32PerConv(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.float truncation dispatch table.
+/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1916,8 +1916,8 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
-                                  nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
+                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
                                   ConversionPatternRewriter &rewriter,
                                   const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2043,10 +2043,10 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFloatOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFloatOp Lowering (extension)
+// NVGPUConvertFPExtOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.float extension dispatch table.
+/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2129,8 +2129,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
-                                nvgpu::ConvertFloatOp::Adaptor adaptor,
+static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
+                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2266,25 +2266,30 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFloatOp op,
   return success();
 }
 
-/// Lowers nvgpu.convert.float by dispatching to the truncation or extension
-/// helper based on the source/destination bitwidths.
-struct NVGPUConvertFloatOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFloatOp>::ConvertOpToLLVMPattern;
+struct NVGPUConvertFPTruncOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFloatOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
+    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
+struct NVGPUConvertFPExtOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
 
-    int srcBW =
-        getElementTypeOrSelf(op.getIn().getType()).getIntOrFloatBitWidth();
-    int dstBW =
-        getElementTypeOrSelf(op.getOut().getType()).getIntOrFloatBitWidth();
-    if (srcBW > dstBW)
-      return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+  LogicalResult
+  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (isa<RankedTensorType>(op.getIn().getType()))
+      return rewriter.notifyMatchFailure(
+          op, "tensor inputs not handled; type converter should lower first");
     return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2301,14 +2306,14 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.float: handles scalar inputs,
-/// non-32-bit-aligned vectors, and multi-rank vectors. Runs as an
-/// OpRewritePattern on MLIR types before LLVM type conversion.
-struct NVGPUConvertFloatCanonicalizePattern
-    : public OpRewritePattern<nvgpu::ConvertFloatOp> {
-  using OpRewritePattern<nvgpu::ConvertFloatOp>::OpRewritePattern;
+/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
+/// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
+template <typename CvtOp, bool IsTrunc>
+struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
+  using OpRewritePattern<CvtOp>::OpRewritePattern;
 
-  LogicalResult matchAndRewrite(nvgpu::ConvertFloatOp op,
+  LogicalResult matchAndRewrite(CvtOp op,
                                 PatternRewriter &rewriter) const override {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
@@ -2327,7 +2332,7 @@ struct NVGPUConvertFloatCanonicalizePattern
     auto srcVecTy = dyn_cast<VectorType>(inType);
     bool isMultiRank = srcVecTy && srcVecTy.getRank() > 1;
     int64_t numElems = isScalar ? 1 : srcVecTy.getNumElements();
-    int step = srcBW > dstBW ? effSrcBW / effDstBW : effDstBW / effSrcBW;
+    int step = IsTrunc ? effSrcBW / effDstBW : effDstBW / effSrcBW;
     int64_t paddedElems = computePaddedElems(numElems, srcBW, dstBW, step);
     bool needsPad = (paddedElems != numElems);
 
@@ -2355,9 +2360,14 @@ struct NVGPUConvertFloatCanonicalizePattern
 
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
-    Value result = nvgpu::ConvertFloatOp::create(
-        b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(), op.getReluAttr(),
-        op.getRandomBits());
+    Value cvt;
+    if constexpr (IsTrunc)
+      cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
+                          op.getReluAttr(), op.getRandomBits());
+    else
+      cvt =
+          CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    Value result = cvt;
 
     if (needsPad)
       result = vector::ExtractStridedSliceOp::create(
@@ -2375,6 +2385,11 @@ struct NVGPUConvertFloatCanonicalizePattern
     return success();
   }
 };
+
+using NVGPUConvertFPTruncCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUConvertFPExtCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2419,10 +2434,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFloatOpLowering,                // nvgpu.convert.float
+      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFloatCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8c96eb7843de4..98973a733f063 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFloatOp
+// NVGPU_ConvertFPTruncOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFloatOp::verify() {
+LogicalResult ConvertFPTruncOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -750,54 +750,70 @@ LogicalResult ConvertFloatOp::verify() {
       failed(result))
     return result;
 
-  if (srcBitWidth == dstBitWidth)
-    return emitOpError("input and output types must have different bitwidths, "
-                       "got ")
-           << srcType << " and " << dstType;
-
-  // Truncation: destination is narrower than source.
-  if (srcBitWidth > dstBitWidth) {
-    if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
-      return emitOpError("input type must be 64/32/16 bitwidth, but got ")
-             << srcBitWidth;
-
-    if (llvm::isa<Float8E8M0FNUType>(dstType)) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
-          rnd != mlir::NVVM::FPRoundingMode::RP)
-        return emitOpError("expects RZ or RP rounding mode when result type is "
-                           "e8m0, but got ")
-               << getRndAttr();
-    } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
-      // TODO: Currently, we only support conversions which fit into a single
-      // i32 register. Support f32->f8/f6/f4 conversions with RS rounding.
-      if (!(srcBitWidth == 32 && dstBitWidth == 16))
-        return emitOpError("RS (stochastic) rounding is only supported for "
-                           "f32->f16/bf16, got ")
-               << srcType << " -> " << dstType;
-      if (!getRandomBits())
-        return emitOpError("random_bits operand is required with RS rounding");
-    } else if (srcType.isF64() && dstBitWidth >= 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN)
-        return emitOpError("expects RN rounding mode for f64 input, but got ")
-               << getRndAttr();
-    } else if (srcBitWidth == 32 && dstBitWidth == 16) {
-      if (rnd != mlir::NVVM::FPRoundingMode::RN &&
-          rnd != mlir::NVVM::FPRoundingMode::RZ)
-        return emitOpError(
-                   "expects RN or RZ rounding mode for f32 to f16/bf16, "
-                   "but got ")
-               << getRndAttr();
-    } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
-      return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
-    }
+  if (srcBitWidth <= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be narrower than operand type " << srcType;
+
+  if (!(srcBitWidth == 64 || srcBitWidth == 32 || srcBitWidth == 16))
+    return emitOpError("input type must be 64/32/16 bitwidth, but got ")
+           << srcBitWidth;
+
+  if (llvm::isa<Float8E8M0FNUType>(dstType)) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RZ &&
+        rnd != mlir::NVVM::FPRoundingMode::RP)
+      return emitOpError("expects RZ or RP rounding mode when result type is "
+                         "e8m0, but got ")
+             << getRndAttr();
+  } else if (rnd == mlir::NVVM::FPRoundingMode::RS) {
+    // TODO: Currently, we only support conversions which fit into a single i32
+    // register. Support f32->f8/f6/f4 conversions with RS rounding.
+    if (!(srcBitWidth == 32 && dstBitWidth == 16))
+      return emitOpError("RS (stochastic) rounding is only supported for "
+                         "f32->f16/bf16, got ")
+             << srcType << " -> " << dstType;
+    if (!getRandomBits())
+      return emitOpError("random_bits operand is required with RS rounding");
+  } else if (srcType.isF64() && dstBitWidth >= 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN)
+      return emitOpError("expects RN rounding mode for f64 input, but got ")
+             << getRndAttr();
+  } else if (srcBitWidth == 32 && dstBitWidth == 16) {
+    if (rnd != mlir::NVVM::FPRoundingMode::RN &&
+        rnd != mlir::NVVM::FPRoundingMode::RZ)
+      return emitOpError("expects RN or RZ rounding mode for f32 to f16/bf16, "
+                         "but got ")
+             << getRndAttr();
+  } else if (rnd != mlir::NVVM::FPRoundingMode::RN) {
+    return emitOpError("expects RN rounding mode, but got ") << getRndAttr();
+  }
 
-    if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
-      return emitOpError("random_bits can only be used with RS rounding mode");
+  if (getRandomBits() && rnd != mlir::NVVM::FPRoundingMode::RS)
+    return emitOpError("random_bits can only be used with RS rounding mode");
 
-    return success();
-  }
+  return success();
+}
+
+//===----------------------------------------------------------------------===//
+// NVGPU_ConvertFPExtOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult ConvertFPExtOp::verify() {
+  Type inType = getIn().getType();
+  Type outType = getType();
+  Type srcType = getElementTypeOrSelf(inType);
+  Type dstType = getElementTypeOrSelf(outType);
+  int srcBitWidth = srcType.getIntOrFloatBitWidth();
+  int dstBitWidth = dstType.getIntOrFloatBitWidth();
+  auto rnd = getRnd();
+
+  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
+      failed(result))
+    return result;
+
+  if (srcBitWidth >= dstBitWidth)
+    return emitOpError("result type ")
+           << dstType << " must be wider than operand type " << srcType;
 
-  // Extension: destination is wider than source.
   if (dstBitWidth != 16 && dstBitWidth != 32 && dstBitWidth != 64)
     return emitOpError("result type must be 16, 32, or 64 bitwidth, but got ")
            << dstBitWidth;
@@ -813,9 +829,6 @@ LogicalResult ConvertFloatOp::verify() {
   if (getRelu() && llvm::isa<BFloat16Type>(dstType))
     return emitOpError("relu is not supported for bf16 destination");
 
-  if (getRandomBits())
-    return emitOpError("random_bits is only supported for truncation");
-
   return success();
 }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
index c5a4f37cdf949..35379861f8f5d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float
+// Large-vector smoke tests for nvgpu.convert.fpext
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
index dcf090955df3c..1d711540f7aff 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.float %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.float %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.float %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.float %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.float %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.float %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fpext %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f16
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.float %in : f8E4M3FN to f32
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.float %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.float %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index 2216f0e9b5c80..d6c862fa5a7a6 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.float.
+// Large-vector smoke tests for nvgpu.convert.fptrunc.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.float %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.float %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.float %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8eaccebf56e00..8d1c71e22f366 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %in : f32 to bf16
+  %out = nvgpu.convert.fptrunc %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.float %arg0 : f64 to f32
+  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.float %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.float %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.float %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.float %in {relu = true}
+  %out = nvgpu.convert.fptrunc %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.float %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.float %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.float %in : f32 to f16
+  %out = nvgpu.convert.fptrunc %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.float %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.float %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
index 9e1798456379f..26c3afc5da358 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
@@ -2,81 +2,81 @@
 
 // -----
 
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.float %in : vector<16xf8E8M0FNU> to vector<16xf16>
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
   return
 }
 
 // -----
 
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.float %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
   return
 }
 
 // -----
 
-func.func @fpext_random_bits(%in : vector<8xf8E5M2>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits is only supported for truncation}}
-  %out = nvgpu.convert.float %in, %rbits : vector<8xf8E5M2> to vector<8xf16>
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
+  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
   return
 }
 
 // -----
 
 func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<8xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
+  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
   return
 }
 
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f8E4M3FN to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f8E4M3FN> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 2ef54221e4e21..93d58a1d5dd72 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -2,25 +2,25 @@
 
 // -----
 
-func.func @convert_float_same_bitwidth(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output types must have different bitwidths, got 'f16' and 'bf16'}}
-  %out = nvgpu.convert.float %in : vector<16xf16> to vector<16xbf16>
+func.func @fptrunc_narrower(%in : vector<16xf16>) {
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.float %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.float %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.float %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.float %in, %rbits
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.convert.fptrunc %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output shapes must match}}
-  %out = nvgpu.convert.float %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
+  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.float %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.float %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.float %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.float' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.float %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
+  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From 467746b1a968238fc1771e00feb745f7a24580a2 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:28:13 +0000
Subject: [PATCH 07/12] address comments

---
 mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td   |  1 -
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 37 ++++----
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 90 +++++++++----------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  9 +-
 .../nvgpu-convert-fptrunc-large.mlir          | 16 ++--
 .../NVGPUToNVVM/nvgpu-convert-fptrunc.mlir    | 66 +++++++-------
 .../NVGPU/nvgpu-convert-fptrunc-invalid.mlir  | 60 ++++++-------
 7 files changed, 136 insertions(+), 143 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
index 38c8bb5c9aa2c..1c0d7bd1113ea 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPU.td
@@ -101,7 +101,6 @@ def TensorMapInterleaveKind : I32EnumAttr<"TensorMapInterleaveKind",
   let cppNamespace = "::mlir::nvgpu";
 }
 
-
 def RcpApprox : I32EnumAttrCase<"APPROX", 0, "approx">;
 def RcpRN     : I32EnumAttrCase<"RN", 1, "rn">;
 def RcpRZ     : I32EnumAttrCase<"RZ", 2, "rz">;
diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 9f27db4c9dcc2..85a35187eb54d 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -677,37 +677,38 @@ def NVGPU_RcpOp : NVGPU_Op<"rcp", [Pure,
 
 def AnyI32Like : TypeOrValueSemanticsContainer<I32, "scalar i32 or vector of i32">;
 
-// nvgpu.convert.fptrunc only supports the satfinite and none saturation modes.
-def NVGPU_SaturationModeSatfiniteOrNone :
-  ConfinedAttr<SaturationModeAttr, [EnumAttrIsOneOf<SaturationModeAttr,
-                [SaturationModeNone, SaturationModeFinite]>]>;
-
-def NVGPU_ConvertFPTruncOp : NVGPU_Op<"convert.fptrunc", [Pure]> {
+def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let summary = "Truncate floating-point to narrower floating-point";
   let description = [{
     Truncate a floating-point value to a smaller floating-point type.
     Destination must be strictly narrower than source.
 
     Supported paths:
-      f64 -> f32, f16, bf16
-      f32 -> f16, bf16, f8, f6, f4
-      f16 -> f8, f6, f4
-      bf16 -> f8, f6, f4
+      f16  -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      bf16 -> f8 (f8E4M3FN, f8E5M2, f8E8M0FNU), f6 (f6E2M3FN, f6E3M2FN), 
+              f4 (f4E2M1FN)
+      f32  -> f16, bf16, f8 (f8E4M3FN, f8E5M2, f8E8M0FNU),
+              f6 (f6E2M3FN, f6E3M2FN), f4 (f4E2M1FN)
+      f64  -> f32, f16, bf16
 
     The `random_bits` operand enables stochastic rounding (RS mode) for
     f32->f16/bf16 conversions; when provided, `rnd` must be RS.
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
-    %r = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
-    %r = nvgpu.convert.fptrunc %in : f32 to f16
-    %r = nvgpu.convert.fptrunc %in : vector<2x4xf16> to vector<2x4xf8E5M2>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
+    %r = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
+    %r = nvgpu.truncf %in : f32 to f16
+    %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
   let arguments = (ins FloatLike:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
-                       DefaultValuedAttr<NVGPU_SaturationModeSatfiniteOrNone, "NVVM::SaturationMode::SATFINITE">:$sat,
+                       DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
+                         [EnumAttrIsOneOf<SaturationModeAttr, 
+                           [SaturationModeNone, SaturationModeFinite]>]>, 
+                         "NVVM::SaturationMode::SATFINITE">:$sat,
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
@@ -723,9 +724,9 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
     Destination must be strictly wider than source.
 
     Supported paths:
-      f8 -> f16, bf16
-      f6 -> f16, bf16
-      f4 -> f16, bf16
+      f4 (f4E2M1FN) -> f16, bf16
+      f6 (f6E2M3FN, f6E3M2FN) -> f16, bf16
+      f8 (f8E4M3FN, f8E5M2, f8E8M0FNU) -> f16, bf16
       f16 -> f32
       bf16 -> f32
 
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index abd6ac7ff8fe4..877356d5903dd 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1713,13 +1713,14 @@ struct NVGPURcpOpLowering : public ConvertOpToLLVMPattern<nvgpu::RcpOp> {
 };
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPTruncOp Lowering
+// NVGPUTruncfOp Lowering
 //===----------------------------------------------------------------------===//
 
 enum class FPKind { F32, BF16, F16, F8, F6, F4 };
 
+/// Get the effective bit width of a floating-point type.
+/// f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
 static int getEffectiveBitWidth(int bitWidth) {
-  // f6 types are 6-bit but NVVM Ops expect 8-bit (i8) containers.
   return bitWidth == 6 ? 8 : bitWidth;
 }
 
@@ -1746,15 +1747,16 @@ static std::optional<FPKind> classifyFPType(Type t) {
     return FPKind::F6;
   if (isConvertibleF4Type(t))
     return FPKind::F4;
+
   return std::nullopt;
 }
 
 /// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConv(FPKind src) {
+static int getNumSrcI32PerConvert(FPKind src) {
   return src == FPKind::F32 ? 2 : 1;
 }
 
-/// Conversion op identifier for nvgpu.convert.fptrunc lowering dispatch table.
+/// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
   F32x2_TO_BF16x2,
@@ -1792,13 +1794,16 @@ static constexpr FPTruncTableEntry kFPTruncTable[] = {
     {FPKind::BF16, FPKind::F4, FPTruncConvOp::BF16x2_TO_F4x2},
 };
 
-static std::optional<FPTruncTableEntry> lookupTruncConvOp(Type srcElemType,
-                                                          Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
+/// Find the conversion table entry whose source/destination `FPKind`s match the
+/// given element types.
+template <typename TableEntry, size_t N>
+static std::optional<TableEntry>
+lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
+  std::optional<FPKind> srcKind = classifyFPType(srcElemType);
+  std::optional<FPKind> dstKind = classifyFPType(dstElemType);
   if (!srcKind || !dstKind)
     return std::nullopt;
-  for (const auto &entry : kFPTruncTable) {
+  for (const TableEntry &entry : table) {
     if (entry.src == *srcKind && entry.dst == *dstKind)
       return entry;
   }
@@ -1862,8 +1867,8 @@ static Value createTruncConversion(
   IntegerType i8Ty = b.getI8Type();
   IntegerType i16Ty = b.getI16Type();
   IntegerType i32Ty = b.getI32Type();
-  auto dstTyAttr = TypeAttr::get(dstElemType);
-  auto actualDstTyAttr = TypeAttr::get(actualDstFloatType);
+  TypeAttr dstTyAttr = TypeAttr::get(dstElemType);
+  TypeAttr actualDstTyAttr = TypeAttr::get(actualDstFloatType);
 
   switch (convOp) {
   case FPTruncConvOp::F32x2_TO_F16x2: {
@@ -1916,10 +1921,10 @@ static Value createTruncConversion(
   llvm_unreachable("unhandled FPTruncConvOp");
 }
 
-static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
-                                  nvgpu::ConvertFPTruncOp::Adaptor adaptor,
-                                  ConversionPatternRewriter &rewriter,
-                                  const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
+                                 nvgpu::TruncfOp::Adaptor adaptor,
+                                 ConversionPatternRewriter &rewriter,
+                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i32Ty = b.getI32Type();
@@ -1944,7 +1949,7 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
   Value randomBits = adaptor.getRandomBits();
   Type actualDstFloatType = dstElemType;
 
-  // STEP 1: bitcast input vector to i32 register vector.
+  // STEP 1: bitcast input vector to i32 vector type.
   // f64 -> f32/f16/bf16 lowers to a single direct LLVM fptrunc
   // f64 -> f8/f6/f4 first truncates to f32 and then reuses the narrow
   //        conversion path below.
@@ -1975,12 +1980,12 @@ static LogicalResult lowerFPTrunc(nvgpu::ConvertFPTruncOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupTruncConvOp(srcElemType, dstElemType);
+  auto convEntry = lookupConvOp(kFPTruncTable, srcElemType, dstElemType);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
-  int numSrcI32PerConv = getNumSrcI32PerConv(convEntry->src);
+  int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.
   const int srcStep = srcBW / effectiveDstBW;
@@ -2071,19 +2076,6 @@ static constexpr FPExtTableEntry kFPExtTable[] = {
     {FPKind::F4, FPKind::BF16, FPExtConvOp::F4x2_TO_BF16x2},
 };
 
-static std::optional<FPExtTableEntry> lookupExtConvOp(Type srcElemType,
-                                                      Type dstElemType) {
-  auto srcKind = classifyFPType(srcElemType);
-  auto dstKind = classifyFPType(dstElemType);
-  if (!srcKind || !dstKind)
-    return std::nullopt;
-  for (const auto &entry : kFPExtTable) {
-    if (entry.src == *srcKind && entry.dst == *dstKind)
-      return entry;
-  }
-  return std::nullopt;
-}
-
 /// Create a typed NVVM extension conversion.
 /// For f8/f6: src is vector<2xi8>.  For f4: src is i8.
 /// Returns i32 (bitcast from vector<2xf16> or vector<2xbf16>).
@@ -2199,7 +2191,7 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
       b.create<LLVM::UndefOp>(VectorType::get(dstI32Elems, i32Ty));
 
   // STEP 2: look up the conversion op from the (srcType, dstType) table.
-  auto convEntry = lookupExtConvOp(srcElemType, intermediateDstElem);
+  auto convEntry = lookupConvOp(kFPExtTable, srcElemType, intermediateDstElem);
   if (!convEntry)
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for extension");
@@ -2266,17 +2258,16 @@ static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
   return success();
 }
 
-struct NVGPUConvertFPTruncOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPTruncOp>::ConvertOpToLLVMPattern;
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPTruncOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPTrunc(op, adaptor, rewriter, getTypeConverter());
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2306,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.convert.fptrunc / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2345,7 +2336,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     if (isScalar)
       input = vector::BroadcastOp::create(b, VectorType::get({1}, srcElemTy),
                                           input);
-
     if (isMultiRank)
       input = vector::ShapeCastOp::create(
           b, VectorType::get({numElems}, srcElemTy), input);
@@ -2361,33 +2351,37 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     auto cvtDstTy =
         VectorType::get({needsPad ? paddedElems : numElems}, dstElemTy);
     Value cvt;
-    if constexpr (IsTrunc)
+    if constexpr (IsTrunc) {
       cvt = CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getSatAttr(),
                           op.getReluAttr(), op.getRandomBits());
-    else
+    } else {
       cvt =
           CvtOp::create(b, cvtDstTy, input, op.getRndAttr(), op.getReluAttr());
+    }
     Value result = cvt;
 
-    if (needsPad)
+    if (needsPad) {
       result = vector::ExtractStridedSliceOp::create(
           b, result, SmallVector<int64_t>{0}, SmallVector<int64_t>{numElems},
           SmallVector<int64_t>{1});
+    }
 
-    if (isMultiRank)
+    if (isMultiRank) {
       result =
           vector::ShapeCastOp::create(b, cast<VectorType>(outType), result);
+    }
 
-    if (isScalar)
+    if (isScalar) {
       result = vector::ExtractOp::create(b, result, SmallVector<int64_t>{0});
+    }
 
     rewriter.replaceOp(op, result);
     return success();
   }
 };
 
-using NVGPUConvertFPTruncCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPTruncOp, true>;
+using NVGPUTruncfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
 using NVGPUConvertFPExtCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
 } // namespace
@@ -2434,12 +2428,12 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaOpLowering,              // nvgpu.warpgroup.mma
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
-      NVGPUConvertFPTruncOpLowering,              // nvgpu.convert.fptrunc
+      NVGPUTruncfOpLowering,                      // nvgpu.truncf
       NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUConvertFPTruncCanonicalizePattern,
+  patterns.add<NVGPUTruncfCanonicalizePattern,
                NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 98973a733f063..607888b678df1 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -701,7 +701,7 @@ LogicalResult RcpOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPTruncOp
+// NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
 static bool isShapedContainerType(Type t) {
@@ -737,7 +737,7 @@ static LogicalResult verifyConversionShapes(Operation *op, Type inType,
   return success();
 }
 
-LogicalResult ConvertFPTruncOp::verify() {
+LogicalResult TruncfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
@@ -806,9 +806,8 @@ LogicalResult ConvertFPExtOp::verify() {
   int dstBitWidth = dstType.getIntOrFloatBitWidth();
   auto rnd = getRnd();
 
-  if (auto result = verifyConversionShapes(getOperation(), inType, outType);
-      failed(result))
-    return result;
+  if (failed(verifyConversionShapes(getOperation(), inType, outType)))
+    return failure();
 
   if (srcBitWidth >= dstBitWidth)
     return emitOpError("result type ")
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
index d6c862fa5a7a6..c5a66ccc3751f 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fptrunc.
+// Large-vector smoke tests for nvgpu.truncf.
 
 // CHECK-LABEL: @cvt_large_f32_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf32>
@@ -10,7 +10,7 @@ func.func @cvt_large_f32_to_f16(%in : vector<400xf32>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_f32_to_bf16(%in : vector<400xf32>) -> vector<400xbf16> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xbf16>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -31,7 +31,7 @@ func.func @cvt_large_f32_to_f8(%in : vector<400xf32>) -> vector<400xf8E4M3FN> {
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f8x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -42,7 +42,7 @@ func.func @cvt_large_f32_to_f6(%in : vector<400xf32>) -> vector<400xf6E2M3FN> {
   // CHECK-NOT: nvvm.convert.f32x2.to.f6x2
   // CHECK: llvm.bitcast {{.*}} : vector<100xi32> to vector<400xi8>
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf6E2M3FN>
   return %out : vector<400xf6E2M3FN>
 }
 
@@ -52,7 +52,7 @@ func.func @cvt_large_f32_to_f4(%in : vector<400xf32>) -> vector<400xf4E2M1FN> {
   // CHECK: llvm.mlir.undef : vector<50xi32>
   // CHECK-COUNT-200: nvvm.convert.f32x2.to.f4x2
   // CHECK-NOT: nvvm.convert.f32x2.to.f4x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf32> to vector<400xf4E2M1FN>
+  %out = nvgpu.truncf %in : vector<400xf32> to vector<400xf4E2M1FN>
   return %out : vector<400xf4E2M1FN>
 }
 
@@ -61,7 +61,7 @@ func.func @cvt_large_f16_to_f8(%in : vector<400xf16>) -> vector<400xf8E4M3FN> {
   // CHECK: llvm.bitcast %{{.*}} : vector<400xf16> to vector<200xi32>
   // CHECK-COUNT-200: nvvm.convert.f16x2.to.f8x2
   // CHECK-NOT: nvvm.convert.f16x2.to.f8x2
-  %out = nvgpu.convert.fptrunc %in : vector<400xf16> to vector<400xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<400xf16> to vector<400xf8E4M3FN>
   return %out : vector<400xf8E4M3FN>
 }
 
@@ -71,6 +71,6 @@ func.func @cvt_large_bf16_to_f6(%in : vector<400xbf16>) -> vector<400xf6E3M2FN>
   // CHECK-COUNT-200: nvvm.convert.bf16x2.to.f6x2
   // CHECK-NOT: nvvm.convert.bf16x2.to.f6x2
   // CHECK: llvm.trunc {{.*}} : vector<400xi8> to vector<400xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<400xbf16> to vector<400xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<400xbf16> to vector<400xf6E3M2FN>
   return %out : vector<400xf6E3M2FN>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
index 8d1c71e22f366..d34402fcf1ce0 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
@@ -14,7 +14,7 @@ func.func @cvt_float_f32_to_f16(%in : vector<4xf32>) {
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rn>
   // CHECK-SAME: : vector<2xf16>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return 
 }
 
@@ -27,7 +27,7 @@ func.func @cvt_float_f32_to_f16_v8(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf16>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf16>
   return 
 }
 
@@ -40,7 +40,7 @@ func.func @cvt_float_f32_to_bf16(%in : vector<4xf32>) {
   // CHECK-SAME: : vector<2xbf16>
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xbf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xbf16>
   return 
 }
 
@@ -66,7 +66,7 @@ func.func @cvt_float_f32_to_e4m3(%in : vector<8xf32>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return 
 }
 
@@ -85,7 +85,7 @@ func.func @cvt_float_f16_to_e2m3(%in : vector<8xf16>) {
   // CHECK: llvm.bitcast {{.*}} : vector<2xi16> to i32
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf16> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf16> to vector<8xf6E2M3FN>
   return
 }
 
@@ -99,7 +99,7 @@ func.func @cvt_float_bf16_to_e3m2(%in : vector<8xbf16>) {
   // CHECK-SAME: : vector<2xbf16> -> i16(f6E3M2FN)
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xbf16> to vector<8xf6E3M2FN>
+  %out = nvgpu.truncf %in : vector<8xbf16> to vector<8xf6E3M2FN>
   return
 }
 
@@ -124,7 +124,7 @@ func.func @cvt_float_f32_to_e2m3(%in : vector<8xf32>) {
   // CHECK: llvm.insertelement {{.*}} : vector<2xi32>
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<8xi8>
   // CHECK: llvm.trunc {{.*}} : vector<8xi8> to vector<8xi6>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf6E2M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf6E2M3FN>
   return 
 }
 
@@ -134,7 +134,7 @@ func.func @cvt_float_f32_to_e8m0_rz(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -145,7 +145,7 @@ func.func @cvt_float_f32_to_e8m0_rp(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rp>
   // CHECK-SAME: : i16(f8E8M0FNU)
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<8xf32> to vector<8xf8E8M0FNU>
   return
 }
@@ -160,7 +160,7 @@ func.func @fptrunc_scalar_f32_to_f16(%in : f32) -> f16 {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -170,7 +170,7 @@ func.func @fptrunc_scalar_f32_to_bf16(%in : f32) -> bf16 {
   // CHECK: vector.broadcast %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %in : f32 to bf16
+  %out = nvgpu.truncf %in : f32 to bf16
   return %out : bf16
 }
 
@@ -179,7 +179,7 @@ func.func @fptrunc_scalar_f64_to_f32(%arg0: f64) -> f32 {
   // CHECK: vector.broadcast
   // CHECK: llvm.fptrunc
   // CHECK: vector.extract
-  %out = nvgpu.convert.fptrunc %arg0 : f64 to f32
+  %out = nvgpu.truncf %arg0 : f64 to f32
   return %out : f32
 }
 
@@ -191,7 +191,7 @@ func.func @fptrunc_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -201,7 +201,7 @@ func.func @fptrunc_v4x2_f32_to_f8(%in : vector<4x2xf32>) -> vector<4x2xf8E4M3FN>
   // CHECK: vector.shape_cast %[[IN]] : vector<4x2xf32> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.shape_cast {{.*}} to vector<4x2xf8E4M3FN>
-  %out = nvgpu.convert.fptrunc %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<4x2xf32> to vector<4x2xf8E4M3FN>
   return %out : vector<4x2xf8E4M3FN>
 }
 
@@ -213,7 +213,7 @@ func.func @fptrunc_v1f32_to_v1f16(%in : vector<1xf32>) -> vector<1xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<1xf32> to vector<1xf16>
+  %out = nvgpu.truncf %in : vector<1xf32> to vector<1xf16>
   return %out : vector<1xf16>
 }
 
@@ -223,7 +223,7 @@ func.func @fptrunc_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f16x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
 
@@ -237,7 +237,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fptrunc %in : vector<3x1xf32> to vector<3x1xf16>
+  %out = nvgpu.truncf %in : vector<3x1xf32> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -247,7 +247,7 @@ func.func @fptrunc_v3x1_f32_to_f16(%in : vector<3x1xf32>) -> vector<3x1xf16> {
 func.func @fptrunc_f64_to_f32(%arg0: vector<4xf64>) -> vector<4xf32> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf32>
   // CHECK-NOT: nvvm
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf32>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -257,7 +257,7 @@ func.func @fptrunc_f64_to_f16(%arg0: vector<2xf64>) -> vector<2xf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xf16>
   return %out : vector<2xf16>
 }
 
@@ -267,7 +267,7 @@ func.func @fptrunc_f64_to_bf16(%arg0: vector<2xf64>) -> vector<2xbf16> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<4xf64> to vector<4xbf16>
   // CHECK-NOT: nvvm.convert
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<2xf64> to vector<2xbf16>
+  %out = nvgpu.truncf %arg0 : vector<2xf64> to vector<2xbf16>
   return %out : vector<2xbf16>
 }
 
@@ -277,7 +277,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
   // CHECK: llvm.fptrunc %{{.*}} : vector<8xf64> to vector<8xf32>
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fptrunc %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
+  %out = nvgpu.truncf %arg0 : vector<4xf64> to vector<4xf8E4M3FN>
   return %out : vector<4xf8E4M3FN>
 }
 
@@ -287,7 +287,7 @@ func.func @fptrunc_f64_to_f8(%arg0: vector<4xf64>) -> vector<4xf8E4M3FN> {
 func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<satfinite>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<satfinite>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -296,7 +296,7 @@ func.func @fptrunc_f32_to_f8_satfinite(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fptrunc %in {relu = true}
+  %out = nvgpu.truncf %in {relu = true}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -307,7 +307,7 @@ func.func @fptrunc_f32_to_f16_relu(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f8x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<8xf32> to vector<8xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
 
@@ -315,7 +315,7 @@ func.func @fptrunc_f32_to_f8_default_sat(%in : vector<8xf32>) {
 func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: sat = #nvvm.sat_mode<satfinite>
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to vector<4xf16>
+  %out = nvgpu.truncf %in : vector<4xf32> to vector<4xf16>
   return
 }
 
@@ -323,7 +323,7 @@ func.func @fptrunc_f32_to_f16_default_sat(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-NOT: satfinite
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<none>}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<none>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -334,7 +334,7 @@ func.func @fptrunc_f32_to_f16_explicit_none(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -343,7 +343,7 @@ func.func @fptrunc_f32_to_f16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rs>
-  %out = nvgpu.convert.fptrunc %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
+  %out = nvgpu.truncf %in, %rbits {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -352,7 +352,7 @@ func.func @fptrunc_f32_to_bf16_rs(%in : vector<4xf32>, %rbits : i32) {
 func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.f16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -361,7 +361,7 @@ func.func @fptrunc_f32_to_f16_rz(%in : vector<4xf32>) {
 func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
   // CHECK: nvvm.convert.f32x2.to.bf16x2
   // CHECK-SAME: rnd = #nvvm.fp_rnd_mode<rz>
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf32> to vector<4xbf16>
   return
 }
@@ -377,7 +377,7 @@ func.func @fptrunc_f32_to_bf16_rz(%in : vector<4xf32>) {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
-  %out = nvgpu.convert.fptrunc %in : f32 to f16
+  %out = nvgpu.truncf %in : f32 to f16
   return %out : f16
 }
 
@@ -386,7 +386,7 @@ func.func @e2e_scalar_f32_to_f16(%in : f32) -> f16 {
 // CHECK-E2E: nvvm.convert.f32x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fptrunc %in : vector<2x4xf32> to vector<2x4xf16>
+  %out = nvgpu.truncf %in : vector<2x4xf32> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -396,6 +396,6 @@ func.func @e2e_v2x4_f32_to_f16(%in : vector<2x4xf32>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f16x2.to.f8x2
 // CHECK-E2E: return
 func.func @e2e_v3f16_to_v3f8(%in : vector<3xf16>) -> vector<3xf8E4M3FN> {
-  %out = nvgpu.convert.fptrunc %in : vector<3xf16> to vector<3xf8E4M3FN>
+  %out = nvgpu.truncf %in : vector<3xf16> to vector<3xf8E4M3FN>
   return %out : vector<3xf8E4M3FN>
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
index 93d58a1d5dd72..9cee9720bddda 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
@@ -3,24 +3,24 @@
 // -----
 
 func.func @fptrunc_narrower(%in : vector<16xf16>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op result type 'f32' must be narrower than operand type 'f16'}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf16> to vector<16xf32>
+  // expected-error @+1 {{'nvgpu.truncf' op result type 'f32' must be narrower than operand type 'f16'}}
+  %out = nvgpu.truncf %in : vector<16xf16> to vector<16xf32>
   return
 }
 
 // -----
 
 func.func @fptrunc_src_bitwidth(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input type must be 64/32/16 bitwidth, but got 8}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input type must be 64/32/16 bitwidth, but got 8}}
+  %out = nvgpu.truncf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rn>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RZ or RP rounding mode when result type is e8m0, but got #nvvm.fp_rnd_mode<rn>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rn>}
       : vector<16xf32> to vector<16xf8E8M0FNU>
   return
 }
@@ -28,8 +28,8 @@ func.func @fptrunc_e8m0_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
-  %out = nvgpu.convert.fptrunc %in {sat = #nvvm.sat_mode<sat>}
+  // expected-error @+1 {{'nvgpu.truncf' op attribute 'sat' failed to satisfy constraint: Describes the saturation mode whose value is one of {none, satfinite}}}
+  %out = nvgpu.truncf %in {sat = #nvvm.sat_mode<sat>}
       : vector<8xf32> to vector<8xf8E4M3FN>
   return
 }
@@ -37,8 +37,8 @@ func.func @fptrunc_unsupported_sat_mode(%in : vector<8xf32>) {
 // -----
 
 func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -46,8 +46,8 @@ func.func @fptrunc_f32_to_f8_rz(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rz>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode for f64 input, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rz>}
       : vector<4xf64> to vector<4xf16>
   return
 }
@@ -55,8 +55,8 @@ func.func @fptrunc_f64_to_f16_rz(%in : vector<4xf64>) {
 // -----
 
 func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op RS (stochastic) rounding is only supported for f32->f16/bf16, got 'f32' -> 'f8E4M3FN'}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -64,8 +64,8 @@ func.func @fptrunc_rs_unsupported_types(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits operand is required with RS rounding}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rs>}
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits operand is required with RS rounding}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rs>}
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -73,8 +73,8 @@ func.func @fptrunc_rs_no_random_bits(%in : vector<4xf32>) {
 // -----
 
 func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
-  %out = nvgpu.convert.fptrunc %in {rnd = #nvvm.fp_rnd_mode<rp>}
+  // expected-error @+1 {{'nvgpu.truncf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rp>}}
+  %out = nvgpu.truncf %in {rnd = #nvvm.fp_rnd_mode<rp>}
       : vector<16xf32> to vector<16xf8E4M3FN>
   return
 }
@@ -82,8 +82,8 @@ func.func @fptrunc_bad_rounding(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op random_bits can only be used with RS rounding mode}}
-  %out = nvgpu.convert.fptrunc %in, %rbits
+  // expected-error @+1 {{'nvgpu.truncf' op random_bits can only be used with RS rounding mode}}
+  %out = nvgpu.truncf %in, %rbits
       : vector<4xf32> to vector<4xf16>
   return
 }
@@ -91,39 +91,39 @@ func.func @fptrunc_random_bits_no_rs(%in : vector<4xf32>, %rbits : i32) {
 // -----
 
 func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output shapes must match}}
-  %out = nvgpu.convert.fptrunc %in : vector<16xf32> to vector<8xf8E4M3FN>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output shapes must match}}
+  %out = nvgpu.truncf %in : vector<16xf32> to vector<8xf8E4M3FN>
   return
 }
 
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fptrunc %in : f32 to vector<1xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fptrunc %in : tensor<f32> to tensor<f16>
+  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
   return
 }
 
 // -----
 
 func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fptrunc %in : vector<4xf32> to tensor<4xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
   return
 }
 
 // -----
 
 func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.convert.fptrunc' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fptrunc %in : tensor<*xf32> to tensor<*xf16>
+  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
+  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
   return
 }

>From bd1e5df96af41323971ac778226419735aee779b Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:38:22 +0000
Subject: [PATCH 08/12] rename convert.fpext to extf and rename test files

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td | 10 +--
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 28 +++----
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    |  4 +-
 ...fpext-large.mlir => nvgpu-extf-large.mlir} | 16 ++--
 ...gpu-convert-fpext.mlir => nvgpu-extf.mlir} | 70 ++++++++--------
 ...unc-large.mlir => nvgpu-truncf-large.mlir} |  0
 ...convert-fptrunc.mlir => nvgpu-truncf.mlir} |  0
 .../NVGPU/nvgpu-convert-fpext-invalid.mlir    | 82 -------------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 82 +++++++++++++++++++
 ...invalid.mlir => nvgpu-truncf-invalid.mlir} |  0
 10 files changed, 146 insertions(+), 146 deletions(-)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext-large.mlir => nvgpu-extf-large.mlir} (84%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fpext.mlir => nvgpu-extf.mlir} (87%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc-large.mlir => nvgpu-truncf-large.mlir} (100%)
 rename mlir/test/Conversion/NVGPUToNVVM/{nvgpu-convert-fptrunc.mlir => nvgpu-truncf.mlir} (100%)
 delete mode 100644 mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
 create mode 100644 mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
 rename mlir/test/Dialect/NVGPU/{nvgpu-convert-fptrunc-invalid.mlir => nvgpu-truncf-invalid.mlir} (100%)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 85a35187eb54d..88ffc69c31cb2 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -717,7 +717,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
   let hasVerifier = 1;
 }
 
-def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
+def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
   let summary = "Extend floating-point to wider floating-point";
   let description = [{
     Extend a floating-point value to a wider floating-point type.
@@ -732,10 +732,10 @@ def NVGPU_ConvertFPExtOp : NVGPU_Op<"convert.fpext", [Pure]> {
 
     Example:
     ```mlir
-    %r = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
-    %r = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
-    %r = nvgpu.convert.fpext %in : f8E4M3FN to f32
+    %r = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
+    %r = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
+    %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
   let arguments = (ins FloatLike:$in,
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 877356d5903dd..bb0bbc2d0a284 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2048,10 +2048,10 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPUConvertFPExtOp Lowering
+// NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
 
-/// Conversion op identifier for nvgpu.convert.fpext lowering dispatch table.
+/// Conversion op identifier for nvgpu.extf lowering dispatch table.
 enum class FPExtConvOp {
   F8x2_TO_F16x2,
   F8x2_TO_BF16x2,
@@ -2121,8 +2121,8 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerFPExt(nvgpu::ConvertFPExtOp op,
-                                nvgpu::ConvertFPExtOp::Adaptor adaptor,
+static LogicalResult lowerExtf(nvgpu::ExtfOp op,
+                               nvgpu::ExtfOp::Adaptor adaptor,
                                 ConversionPatternRewriter &rewriter,
                                 const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
@@ -2271,17 +2271,17 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUConvertFPExtOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp> {
-  using ConvertOpToLLVMPattern<nvgpu::ConvertFPExtOp>::ConvertOpToLLVMPattern;
+struct NVGPUExtfOpLowering
+    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
-  matchAndRewrite(nvgpu::ConvertFPExtOp op, OpAdaptor adaptor,
+  matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     if (isa<RankedTensorType>(op.getIn().getType()))
       return rewriter.notifyMatchFailure(
           op, "tensor inputs not handled; type converter should lower first");
-    return lowerFPExt(op, adaptor, rewriter, getTypeConverter());
+    return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
 
@@ -2297,7 +2297,7 @@ static int64_t computePaddedElems(int64_t numElems, int srcBW, int dstBW,
   return ceilDiv(padded, step) * step;
 }
 
-/// Canonicalization pattern for nvgpu.truncf / nvgpu.convert.fpext:
+/// Canonicalization pattern for nvgpu.truncf / nvgpu.extf:
 /// handles scalar inputs, non-32-bit-aligned vectors, and multi-rank vectors.
 /// Runs as an OpRewritePattern on MLIR types before LLVM type conversion.
 template <typename CvtOp, bool IsTrunc>
@@ -2382,8 +2382,8 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
 
 using NVGPUTruncfCanonicalizePattern =
     NVGPUFPCanonicalizePattern<nvgpu::TruncfOp, true>;
-using NVGPUConvertFPExtCanonicalizePattern =
-    NVGPUFPCanonicalizePattern<nvgpu::ConvertFPExtOp, false>;
+using NVGPUExtfCanonicalizePattern =
+    NVGPUFPCanonicalizePattern<nvgpu::ExtfOp, false>;
 } // namespace
 
 void mlir::nvgpu::populateCommonGPUTypeAndAttributeConversions(
@@ -2429,11 +2429,11 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUWarpgroupMmaStoreOpLowering,         // nvgpu.warpgroup.mma.store
       NVGPUWarpgroupMmaInitAccumulatorOpLowering, // nvgpu.warpgroup.mma.init.accumulator
       NVGPUTruncfOpLowering,                      // nvgpu.truncf
-      NVGPUConvertFPExtOpLowering,                // nvgpu.convert.fpext
+      NVGPUExtfOpLowering,                        // nvgpu.extf
       MmaSyncOptoNVVM, MmaLdMatrixOpToNVVM, NVGPUAsyncCopyLowering,
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
   patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUConvertFPExtCanonicalizePattern>(patterns.getContext());
+               NVGPUExtfCanonicalizePattern>(patterns.getContext());
 }
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 607888b678df1..8dd67411292dd 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -794,10 +794,10 @@ LogicalResult TruncfOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// NVGPU_ConvertFPExtOp
+// NVGPU_ExtfOp
 //===----------------------------------------------------------------------===//
 
-LogicalResult ConvertFPExtOp::verify() {
+LogicalResult ExtfOp::verify() {
   Type inType = getIn().getType();
   Type outType = getType();
   Type srcType = getElementTypeOrSelf(inType);
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
similarity index 84%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
index 35379861f8f5d..9f2ddc5270c50 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext-large.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf-large.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt %s -convert-nvgpu-to-nvvm | FileCheck %s
 
-// Large-vector smoke tests for nvgpu.convert.fpext
+// Large-vector smoke tests for nvgpu.extf
 
 // CHECK-LABEL: @cvt_large_f8_to_f16(
 // CHECK-SAME: %[[IN:.+]]: vector<400xf8E4M3FN>
@@ -10,7 +10,7 @@ func.func @cvt_large_f8_to_f16(%in : vector<400xf8E4M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E4M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf8E4M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -20,7 +20,7 @@ func.func @cvt_large_e8m0_to_bf16(%in : vector<400xf8E8M0FNU>) -> vector<400xbf1
   // CHECK-COUNT-200: nvvm.convert.f8x2.to.bf16x2
   // CHECK-NOT: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xbf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E8M0FNU> to vector<400xbf16>
+  %out = nvgpu.extf %in : vector<400xf8E8M0FNU> to vector<400xbf16>
   return %out : vector<400xbf16>
 }
 
@@ -33,7 +33,7 @@ func.func @cvt_large_f6_to_f16(%in : vector<400xf6E2M3FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -44,7 +44,7 @@ func.func @cvt_large_f4_to_f16(%in : vector<400xf4E2M1FN>) -> vector<400xf16> {
   // CHECK-COUNT-200: nvvm.convert.f4x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
-  %out = nvgpu.convert.fpext %in : vector<400xf4E2M1FN> to vector<400xf16>
+  %out = nvgpu.extf %in : vector<400xf4E2M1FN> to vector<400xf16>
   return %out : vector<400xf16>
 }
 
@@ -54,7 +54,7 @@ func.func @cvt_large_f8_to_f32(%in : vector<400xf8E5M2>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<200xi32> to vector<400xf16>
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf8E5M2> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf8E5M2> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -64,7 +64,7 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
   // CHECK-COUNT-200: nvvm.convert.f6x2.to.f16x2
   // CHECK-NOT: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf6E2M3FN> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf6E2M3FN> to vector<400xf32>
   return %out : vector<400xf32>
 }
 
@@ -73,6 +73,6 @@ func.func @cvt_large_f6_to_f32(%in : vector<400xf6E2M3FN>) -> vector<400xf32> {
 func.func @cvt_large_f16_to_f32(%in : vector<400xf16>) -> vector<400xf32> {
   // CHECK-NOT: nvvm.convert
   // CHECK: llvm.fpext %[[IN]] : vector<400xf16> to vector<400xf32>
-  %out = nvgpu.convert.fpext %in : vector<400xf16> to vector<400xf32>
+  %out = nvgpu.extf %in : vector<400xf16> to vector<400xf32>
   return %out : vector<400xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
similarity index 87%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
index 1d711540f7aff..5924f984f870d 100644
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fpext.mlir
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-extf.mlir
@@ -18,7 +18,7 @@ func.func @cvt_float_e4m3fn_to_f16(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E4M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E4M3FN> to vector<8xf16>
   return 
 }
 
@@ -48,7 +48,7 @@ func.func @cvt_float_e5m2_to_f16(%in : vector<8xf8E5M2>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E5M2> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf8E5M2> to vector<8xf16>
   return 
 }
 
@@ -70,7 +70,7 @@ func.func @cvt_float_e8m0_to_bf16(%in : vector<8xf8E8M0FNU>) {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf8E8M0FNU> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf8E8M0FNU> to vector<8xbf16>
   return 
 }
 
@@ -92,7 +92,7 @@ func.func @cvt_float_e2m3_to_f16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xf16>
   return 
 }
 
@@ -117,7 +117,7 @@ func.func @cvt_float_e3m2_to_f16(%in : vector<8xf6E3M2FN>) {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E3M2FN> to vector<8xf16>
+  %out = nvgpu.extf %in : vector<8xf6E3M2FN> to vector<8xf16>
   return
 }
 
@@ -150,7 +150,7 @@ func.func @cvt_float_e2m1_to_f16(%in : vector<16xf4E2M1FN>) {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf16>
   return 
 }
 
@@ -168,7 +168,7 @@ func.func @cvt_float_e2m3_to_bf16(%in : vector<8xf6E2M3FN>) {
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: nvvm.convert.f6x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xbf16>
-  %out = nvgpu.convert.fpext %in : vector<8xf6E2M3FN> to vector<8xbf16>
+  %out = nvgpu.extf %in : vector<8xf6E2M3FN> to vector<8xbf16>
   return
 }
 
@@ -183,7 +183,7 @@ func.func @cvt_float_e2m1_to_bf16(%in : vector<16xf4E2M1FN>) {
   // CHECK-SAME: : i8(f4E2M1FN)
   // CHECK: nvvm.convert.f4x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<8xi32> to vector<16xbf16>
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xbf16>
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xbf16>
   return
 }
 
@@ -200,7 +200,7 @@ func.func @fpext_e4m3fn_to_f32(%in : vector<4xf8E4M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -213,7 +213,7 @@ func.func @fpext_e5m2_to_f32(%in : vector<4xf8E5M2>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E5M2> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E5M2> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -226,7 +226,7 @@ func.func @fpext_e8m0_to_f32(%in : vector<4xf8E8M0FNU>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f8x2.to.bf16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xbf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf8E8M0FNU> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf8E8M0FNU> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -241,7 +241,7 @@ func.func @fpext_e2m3_to_f32(%in : vector<4xf6E2M3FN>) -> vector<4xf32> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<2xi32> to vector<4xf16>
   // CHECK: llvm.fpext {{.*}} : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf6E2M3FN> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf6E2M3FN> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -253,7 +253,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.bitcast {{.*}} : vector<4xi32> to vector<8xf16>
   // CHECK: llvm.fpext {{.*}} : vector<8xf16> to vector<8xf32>
-  %out = nvgpu.convert.fpext %in : vector<8xf4E2M1FN> to vector<8xf32>
+  %out = nvgpu.extf %in : vector<8xf4E2M1FN> to vector<8xf32>
   return %out : vector<8xf32>
 }
 
@@ -263,7 +263,7 @@ func.func @fpext_e2m1_to_f32(%in : vector<8xf4E2M1FN>) -> vector<8xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xf16>
 func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -273,7 +273,7 @@ func.func @fpext_f16_to_f32(%in : vector<4xf16>) -> vector<4xf32> {
 // CHECK-SAME: %[[IN:.+]]: vector<4xbf16>
 func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
   // CHECK: llvm.fpext %[[IN]] : vector<4xbf16> to vector<4xf32>
-  %out = nvgpu.convert.fpext %in : vector<4xbf16> to vector<4xf32>
+  %out = nvgpu.extf %in : vector<4xbf16> to vector<4xf32>
   return %out : vector<4xf32>
 }
 
@@ -283,7 +283,7 @@ func.func @fpext_bf16_to_f32(%in : vector<4xbf16>) -> vector<4xf32> {
 func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -291,7 +291,7 @@ func.func @fpext_f16_to_f64(%arg0: vector<4xf16>) -> vector<4xf64> {
 func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
   // CHECK: llvm.fpext %{{.*}} : vector<4xbf16> to vector<4xf64>
   // CHECK-NOT: llvm.fpext
-  %out = nvgpu.convert.fpext %arg0 : vector<4xbf16> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xbf16> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -299,7 +299,7 @@ func.func @fpext_bf16_to_f64(%arg0: vector<4xbf16>) -> vector<4xf64> {
 func.func @fpext_f32_to_f64(%arg0: vector<4xf32>) -> vector<4xf64> {
   // CHECK-NOT: nvvm
   // CHECK: llvm.fpext %{{.*}} : vector<4xf32> to vector<4xf64>
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf32> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf32> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -309,7 +309,7 @@ func.func @fpext_f8_to_f64(%arg0: vector<4xf8E4M3FN>) -> vector<4xf64> {
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf8E4M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -318,7 +318,7 @@ func.func @fpext_f4_to_f64(%arg0: vector<8xf4E2M1FN>) -> vector<8xf64> {
   // CHECK: nvvm.convert.f4x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
+  %out = nvgpu.extf %arg0 : vector<8xf4E2M1FN> to vector<8xf64>
   return %out : vector<8xf64>
 }
 
@@ -328,7 +328,7 @@ func.func @fpext_e2m3_to_f64(%arg0: vector<4xf6E2M3FN>) -> vector<4xf64> {
   // CHECK: nvvm.convert.f6x2.to.f16x2
   // CHECK: llvm.fpext {{.*}} to {{.*}}f64
   // CHECK-NOT: llvm.fpext {{.*}} to {{.*}}f32
-  %out = nvgpu.convert.fpext %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
+  %out = nvgpu.extf %arg0 : vector<4xf6E2M3FN> to vector<4xf64>
   return %out : vector<4xf64>
 }
 
@@ -342,7 +342,7 @@ func.func @fpext_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f16 from vector<1xf16>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -357,7 +357,7 @@ func.func @fpext_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.extract {{.*}}[0] : f32 from vector<1xf32>
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -369,7 +369,7 @@ func.func @fpext_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
   // CHECK: vector.shape_cast %[[IN]] : vector<2x4xf8E4M3FN> to vector<8xf8E4M3FN>
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf16>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -382,7 +382,7 @@ func.func @fpext_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.shape_cast {{.*}} to vector<2x4xf32>
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -394,7 +394,7 @@ func.func @fpext_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
   // CHECK: vector.insert_strided_slice %[[IN]]
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -407,7 +407,7 @@ func.func @fpext_v3_f8_to_f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: llvm.fpext
   // CHECK: vector.extract_strided_slice
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
 
@@ -421,7 +421,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK: vector.extract_strided_slice
   // CHECK: vector.shape_cast {{.*}} to vector<3x1xf16>
-  %out = nvgpu.convert.fpext %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
+  %out = nvgpu.extf %in : vector<3x1xf8E4M3FN> to vector<3x1xf16>
   return %out : vector<3x1xf16>
 }
 
@@ -431,7 +431,7 @@ func.func @fpext_v3x1_f8_to_f16(%in : vector<3x1xf8E4M3FN>) -> vector<3x1xf16> {
 func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
   // CHECK: nvvm.convert.f8x2.to.f16x2
   // CHECK-SAME: relu = true
-  %out = nvgpu.convert.fpext %in {relu = true}
+  %out = nvgpu.extf %in {relu = true}
       : vector<8xf8E4M3FN> to vector<8xf16>
   return
 }
@@ -447,7 +447,7 @@ func.func @fpext_f8_to_f16_relu(%in : vector<8xf8E4M3FN>) {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f16
+  %out = nvgpu.extf %in : f8E4M3FN to f16
   return %out : f16
 }
 
@@ -456,7 +456,7 @@ func.func @e2e_scalar_f8_to_f16(%in : f8E4M3FN) -> f16 {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf16>
   return %out : vector<2x4xf16>
 }
 
@@ -466,7 +466,7 @@ func.func @e2e_v2x4_f8_to_f16(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf16> {
 // CHECK-E2E: nvvm.convert.f8x2.to.f16x2
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf16>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf16>
   return %out : vector<3xf16>
 }
 
@@ -480,7 +480,7 @@ func.func @e2e_v3f8_to_v3f16(%in : vector<3xf8E5M2>) -> vector<3xf16> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to f32
+  %out = nvgpu.extf %in : f8E4M3FN to f32
   return %out : f32
 }
 
@@ -490,7 +490,7 @@ func.func @e2e_scalar_f8_to_f32(%in : f8E4M3FN) -> f32 {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
-  %out = nvgpu.convert.fpext %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
+  %out = nvgpu.extf %in : vector<2x4xf8E4M3FN> to vector<2x4xf32>
   return %out : vector<2x4xf32>
 }
 
@@ -501,6 +501,6 @@ func.func @e2e_v2x4_f8_to_f32(%in : vector<2x4xf8E4M3FN>) -> vector<2x4xf32> {
 // CHECK-E2E: llvm.fpext
 // CHECK-E2E: return
 func.func @e2e_v3f8_to_v3f32(%in : vector<3xf8E5M2>) -> vector<3xf32> {
-  %out = nvgpu.convert.fpext %in : vector<3xf8E5M2> to vector<3xf32>
+  %out = nvgpu.extf %in : vector<3xf8E5M2> to vector<3xf32>
   return %out : vector<3xf32>
 }
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc-large.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf-large.mlir
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
similarity index 100%
rename from mlir/test/Conversion/NVGPUToNVVM/nvgpu-convert-fptrunc.mlir
rename to mlir/test/Conversion/NVGPUToNVVM/nvgpu-truncf.mlir
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
deleted file mode 100644
index 26c3afc5da358..0000000000000
--- a/mlir/test/Dialect/NVGPU/nvgpu-convert-fpext-invalid.mlir
+++ /dev/null
@@ -1,82 +0,0 @@
-// RUN: mlir-opt -split-input-file -verify-diagnostics %s
-
-// -----
-
-func.func @fpext_wider(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
-  return
-}
-
-// -----
-
-func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op result type must be 16, 32, or 64 bitwidth, but got 8}}
-  %out = nvgpu.convert.fpext %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
-  return
-}
-
-// -----
-
-func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects bf16 or f32 output type when input type is e8m0.}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E8M0FNU> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
-  %out = nvgpu.convert.fpext %in {rnd = #nvvm.fp_rnd_mode<rz>}
-      : vector<16xf8E5M2> to vector<16xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op relu is not supported for bf16 destination}}
-  %out = nvgpu.convert.fpext %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
-  return
-}
-
-// -----
-
-func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output shapes must match}}
-  %out = nvgpu.convert.fpext %in : vector<16xf8E5M2> to vector<8xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must both be scalars or both be vectors/tensors}}
-  %out = nvgpu.convert.fpext %in : f8E4M3FN to vector<1xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.convert.fpext %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.convert.fpext %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.convert.fpext' op unranked tensor types are not supported}}
-  %out = nvgpu.convert.fpext %in : tensor<*xf8E4M3FN> to tensor<*xf16>
-  return
-}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
new file mode 100644
index 0000000000000..874ea7ab0e6fc
--- /dev/null
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt -split-input-file -verify-diagnostics %s
+
+// -----
+
+func.func @fpext_wider(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type 'f4E2M1FN' must be wider than operand type 'f8E5M2'}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<16xf4E2M1FN>
+  return
+}
+
+// -----
+
+func.func @fpext_dst_bitwidth(%in : vector<16xf4E2M1FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op result type must be 16, 32, or 64 bitwidth, but got 8}}
+  %out = nvgpu.extf %in : vector<16xf4E2M1FN> to vector<16xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @fpext_e8m0_to_f16(%in : vector<16xf8E8M0FNU>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects bf16 or f32 output type when input type is e8m0.}}
+  %out = nvgpu.extf %in : vector<16xf8E8M0FNU> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_bad_rounding(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op expects RN rounding mode, but got #nvvm.fp_rnd_mode<rz>}}
+  %out = nvgpu.extf %in {rnd = #nvvm.fp_rnd_mode<rz>}
+      : vector<16xf8E5M2> to vector<16xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_relu_bf16(%in : vector<8xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op relu is not supported for bf16 destination}}
+  %out = nvgpu.extf %in {relu = true} : vector<8xf8E5M2> to vector<8xbf16>
+  return
+}
+
+// -----
+
+func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output shapes must match}}
+  %out = nvgpu.extf %in : vector<16xf8E5M2> to vector<8xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
+  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
+  return
+}
+
+// -----
+
+func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
+  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
+  return
+}
+
+// -----
+
+func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
+  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+  return
+}
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
similarity index 100%
rename from mlir/test/Dialect/NVGPU/nvgpu-convert-fptrunc-invalid.mlir
rename to mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir

>From 9da2835665afefdd93f3adc06cb50edec612e24c Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:46:13 +0000
Subject: [PATCH 09/12] fix formatting

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index bb0bbc2d0a284..d2f7fe768930b 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2121,10 +2121,9 @@ static Value createExtConversion(ImplicitLocOpBuilder &b, MLIRContext *ctx,
   llvm_unreachable("unhandled FPExtConvOp");
 }
 
-static LogicalResult lowerExtf(nvgpu::ExtfOp op,
-                               nvgpu::ExtfOp::Adaptor adaptor,
-                                ConversionPatternRewriter &rewriter,
-                                const LLVMTypeConverter *typeConverter) {
+static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
+                               ConversionPatternRewriter &rewriter,
+                               const LLVMTypeConverter *typeConverter) {
   MLIRContext *ctx = op.getContext();
   ImplicitLocOpBuilder b(op->getLoc(), rewriter);
   IntegerType i8Ty = b.getI8Type();
@@ -2271,8 +2270,7 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   }
 };
 
-struct NVGPUExtfOpLowering
-    : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
+struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 
   LogicalResult
@@ -2434,6 +2432,6 @@ void mlir::populateNVGPUToNVVMConversionPatterns(
       NVGPUAsyncCreateGroupLowering, NVGPUAsyncWaitLowering,
       NVGPUMmaSparseSyncLowering, NVGPURcpOpLowering>(converter);
 
-  patterns.add<NVGPUTruncfCanonicalizePattern,
-               NVGPUExtfCanonicalizePattern>(patterns.getContext());
+  patterns.add<NVGPUTruncfCanonicalizePattern, NVGPUExtfCanonicalizePattern>(
+      patterns.getContext());
 }

>From 901c37448621ec25ebc7d5204a0bfa23388b38f3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 3 Jul 2026 12:56:56 +0000
Subject: [PATCH 10/12] convert single use function to lambda

---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index d2f7fe768930b..67d3b2e86293a 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1751,11 +1751,6 @@ static std::optional<FPKind> classifyFPType(Type t) {
   return std::nullopt;
 }
 
-/// Number of source-side i32 register slots consumed by each NVVM convert Op.
-static int getNumSrcI32PerConvert(FPKind src) {
-  return src == FPKind::F32 ? 2 : 1;
-}
-
 /// Conversion op identifier for nvgpu.truncf lowering dispatch table.
 enum class FPTruncConvOp {
   F32x2_TO_F16x2,
@@ -1985,6 +1980,11 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
     return rewriter.notifyMatchFailure(
         op, "unsupported type combination for truncation");
   FPTruncConvOp convOp = convEntry->convOp;
+
+  // Number of source-side i32 register slots consumed by each NVVM convert Op.
+  auto getNumSrcI32PerConvert = [](FPKind src) {
+    return src == FPKind::F32 ? 2 : 1;
+  };
   int numSrcI32PerConv = getNumSrcI32PerConvert(convEntry->src);
 
   // STEP 3: pack conversion results into destination i32 vector.

>From e647f5b131237e4e17acfcb2e9966b0ddf38dcd8 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Thu, 9 Jul 2026 13:43:34 +0000
Subject: [PATCH 11/12] remove tensor type support

---
 .../include/mlir/Dialect/NVGPU/IR/NVGPUOps.td |  8 ++---
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    |  9 ------
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp    | 31 +++++--------------
 .../Dialect/NVGPU/nvgpu-extf-invalid.mlir     | 24 +++-----------
 .../Dialect/NVGPU/nvgpu-truncf-invalid.mlir   | 24 +++-----------
 5 files changed, 20 insertions(+), 76 deletions(-)

diff --git a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
index 88ffc69c31cb2..d0b07de6f01bf 100644
--- a/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
+++ b/mlir/include/mlir/Dialect/NVGPU/IR/NVGPUOps.td
@@ -703,7 +703,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
     %r = nvgpu.truncf %in : vector<2x4xf16> to vector<2x4xf8E5M2>
     ```
   }];
-  let arguments = (ins FloatLike:$in,
+  let arguments = (ins AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
                        DefaultValuedAttr<ConfinedAttr<SaturationModeAttr, 
                          [EnumAttrIsOneOf<SaturationModeAttr, 
@@ -712,7 +712,7 @@ def NVGPU_TruncfOp : NVGPU_Op<"truncf", [Pure]> {
                        DefaultValuedAttr<BoolAttr, "false">:$relu,
                        Optional<I32>:$random_bits
                        );
-  let results = (outs FloatLike:$out);
+  let results = (outs AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$out);
   let assemblyFormat = "$in (`,` $random_bits^)? attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
@@ -738,10 +738,10 @@ def NVGPU_ExtfOp : NVGPU_Op<"extf", [Pure]> {
     %r = nvgpu.extf %in : f8E4M3FN to f32
     ```
   }];
-  let arguments = (ins FloatLike:$in,
+  let arguments = (ins AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$in,
                        DefaultValuedAttr<FPRoundingModeAttr, "NVVM::FPRoundingMode::RN">:$rnd,
                        DefaultValuedAttr<BoolAttr, "false">:$relu);
-  let results = (outs FloatLike:$out);
+  let results = (outs AnyTypeOf<[AnyFloat, VectorOfNonZeroRankOf<[AnyFloat]>]>:$out);
   let assemblyFormat = "$in attr-dict `:` type($in) `to` type($out)";
   let hasVerifier = 1;
 }
diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 67d3b2e86293a..0642ad95d0f61 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2263,9 +2263,6 @@ struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
   LogicalResult
   matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
     return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2276,9 +2273,6 @@ struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   LogicalResult
   matchAndRewrite(nvgpu::ExtfOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (isa<RankedTensorType>(op.getIn().getType()))
-      return rewriter.notifyMatchFailure(
-          op, "tensor inputs not handled; type converter should lower first");
     return lowerExtf(op, adaptor, rewriter, getTypeConverter());
   }
 };
@@ -2307,9 +2301,6 @@ struct NVGPUFPCanonicalizePattern : public OpRewritePattern<CvtOp> {
     Type inType = op.getIn().getType();
     Type outType = op.getOut().getType();
 
-    if (isa<RankedTensorType>(inType))
-      return failure();
-
     Type srcElemTy = getElementTypeOrSelf(inType);
     Type dstElemTy = getElementTypeOrSelf(outType);
     int srcBW = srcElemTy.getIntOrFloatBitWidth();
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index 8dd67411292dd..c77ecb8431ad8 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -704,35 +704,20 @@ LogicalResult RcpOp::verify() {
 // NVGPU_TruncfOp
 //===----------------------------------------------------------------------===//
 
-static bool isShapedContainerType(Type t) {
-  return llvm::isa<VectorType, RankedTensorType, UnrankedTensorType>(t);
-}
-
 static LogicalResult verifyConversionShapes(Operation *op, Type inType,
                                             Type outType) {
-  if (llvm::isa<UnrankedTensorType>(inType) ||
-      llvm::isa<UnrankedTensorType>(outType))
-    return op->emitOpError("unranked tensor types are not supported, got ")
-           << inType << " and " << outType;
-  bool srcIsShaped = isShapedContainerType(inType);
-  bool dstIsShaped = isShapedContainerType(outType);
-  if (srcIsShaped != dstIsShaped)
+  bool srcIsVector = llvm::isa<VectorType>(inType);
+  bool dstIsVector = llvm::isa<VectorType>(outType);
+  if (srcIsVector != dstIsVector)
     return op->emitOpError("input and output must both be scalars or both be "
-                           "vectors/tensors, got ")
+                           "vectors, got ")
            << inType << " and " << outType;
-  if (srcIsShaped) {
-    auto srcShaped = llvm::cast<ShapedType>(inType);
-    auto dstShaped = llvm::cast<ShapedType>(outType);
-    if (srcShaped.getRank() == 0 || dstShaped.getRank() == 0)
-      return op->emitOpError("rank-0 shaped types are not supported, use "
-                             "scalar type instead");
-    if (srcShaped.getShape() != dstShaped.getShape())
+  if (srcIsVector) {
+    auto srcVector = llvm::cast<VectorType>(inType);
+    auto dstVector = llvm::cast<VectorType>(outType);
+    if (srcVector.getShape() != dstVector.getShape())
       return op->emitOpError("input and output shapes must match, got ")
              << inType << " and " << outType;
-    if (llvm::isa<VectorType>(inType) != llvm::isa<VectorType>(outType))
-      return op->emitOpError("input and output must be the same container "
-                             "type (both vector or both tensor), got ")
-             << inType << " and " << outType;
   }
   return success();
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
index 874ea7ab0e6fc..ef659efb96f8e 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-extf-invalid.mlir
@@ -52,31 +52,15 @@ func.func @fpext_shape_mismatch(%in : vector<16xf8E5M2>) {
 // -----
 
 func.func @fpext_scalar_vector_mismatch(%in : f8E4M3FN) {
-  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors/tensors}}
+  // expected-error @+1 {{'nvgpu.extf' op input and output must both be scalars or both be vectors}}
   %out = nvgpu.extf %in : f8E4M3FN to vector<1xf16>
   return
 }
 
 // -----
 
-func.func @fpext_rank0_tensor(%in : tensor<f8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.extf %in : tensor<f8E4M3FN> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fpext_container_mismatch(%in : vector<4xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.extf %in : vector<4xf8E4M3FN> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fpext_unranked_tensor(%in : tensor<*xf8E4M3FN>) {
-  // expected-error @+1 {{'nvgpu.extf' op unranked tensor types are not supported}}
-  %out = nvgpu.extf %in : tensor<*xf8E4M3FN> to tensor<*xf16>
+func.func @fpext_rank0_vector(%in : vector<f8E4M3FN>) {
+  // expected-error @+1 {{'nvgpu.extf' op operand #0 must be floating-point or vector of floating-point values, but got 'vector<f8E4M3FN>'}}
+  %out = nvgpu.extf %in : vector<f8E4M3FN> to vector<f16>
   return
 }
diff --git a/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
index 9cee9720bddda..1b2fb3db5ceca 100644
--- a/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
+++ b/mlir/test/Dialect/NVGPU/nvgpu-truncf-invalid.mlir
@@ -99,31 +99,15 @@ func.func @fptrunc_shape_mismatch(%in : vector<16xf32>) {
 // -----
 
 func.func @fptrunc_scalar_vector_mismatch(%in : f32) {
-  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors/tensors}}
+  // expected-error @+1 {{'nvgpu.truncf' op input and output must both be scalars or both be vectors}}
   %out = nvgpu.truncf %in : f32 to vector<1xf16>
   return
 }
 
 // -----
 
-func.func @fptrunc_rank0_tensor(%in : tensor<f32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op rank-0 shaped types are not supported, use scalar type instead}}
-  %out = nvgpu.truncf %in : tensor<f32> to tensor<f16>
-  return
-}
-
-// -----
-
-func.func @fptrunc_container_mismatch(%in : vector<4xf32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op input and output must be the same container type (both vector or both tensor)}}
-  %out = nvgpu.truncf %in : vector<4xf32> to tensor<4xf16>
-  return
-}
-
-// -----
-
-func.func @fptrunc_unranked_tensor(%in : tensor<*xf32>) {
-  // expected-error @+1 {{'nvgpu.truncf' op unranked tensor types are not supported}}
-  %out = nvgpu.truncf %in : tensor<*xf32> to tensor<*xf16>
+func.func @fptrunc_rank0_vector(%in : vector<f32>) {
+  // expected-error @+1 {{'nvgpu.truncf' op operand #0 must be floating-point or vector of floating-point values, but got 'vector<f32>'}}
+  %out = nvgpu.truncf %in : vector<f32> to vector<f16>
   return
 }

>From 81c469fd11c6be068970f51bf0a051324a7c3bc3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Mon, 13 Jul 2026 11:31:54 +0000
Subject: [PATCH 12/12] clean-up

---
 .../Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp    | 20 +++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 0642ad95d0f61..948229bb53328 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -2047,6 +2047,16 @@ static LogicalResult lowerTruncf(nvgpu::TruncfOp op,
   return success();
 }
 
+struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
+  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
+  }
+};
+
 //===----------------------------------------------------------------------===//
 // NVGPUExtfOp Lowering
 //===----------------------------------------------------------------------===//
@@ -2257,16 +2267,6 @@ static LogicalResult lowerExtf(nvgpu::ExtfOp op, nvgpu::ExtfOp::Adaptor adaptor,
   return success();
 }
 
-struct NVGPUTruncfOpLowering : public ConvertOpToLLVMPattern<nvgpu::TruncfOp> {
-  using ConvertOpToLLVMPattern<nvgpu::TruncfOp>::ConvertOpToLLVMPattern;
-
-  LogicalResult
-  matchAndRewrite(nvgpu::TruncfOp op, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    return lowerTruncf(op, adaptor, rewriter, getTypeConverter());
-  }
-};
-
 struct NVGPUExtfOpLowering : public ConvertOpToLLVMPattern<nvgpu::ExtfOp> {
   using ConvertOpToLLVMPattern<nvgpu::ExtfOp>::ConvertOpToLLVMPattern;
 


From llvmlistbot at llvm.org  Mon Jul 13 04:53:54 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Mon, 13 Jul 2026 04:53:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a54d1d2.c56fb102.7c9cd.70f0@mx.google.com>


https://github.com/Devjiu updated https://github.com/llvm/llvm-project/pull/207241

>From 65e00b4915e4894a77b7fd083c1d6aa06a964d2d Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Thu, 2 Jul 2026 16:32:59 +0000
Subject: [PATCH 1/3] [mlir][tensor] Preserve encoding in more canonicalizers
 (concat, reshape, pad)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This commit fixes encoding drop in similar with InsertSliceOpConstantArgumentFolder cases.
The same anti-pattern — silently producing a `RankedTensorType` without an
encoding while a source value with an encoding flows through the
refined type — is present in several other canonicalizers/inferResultType
implementations:

* ConcatOp::inferResultType built the result type with no encoding. When all
  inputs share an encoding, propagate it; when they differ, drop it (concat
  semantics are undefined across differing encodings, so we do not pick one).

* InferConcatOperandTypes built per-operand refined types with the result
  element type but no encoding. The cast targets an operand value, so it
  must carry that operand's own encoding.

* CollapseShapeOp::inferCollapsedType dropped the encoding. Since collapse
  changes the tensor rank, `VerifiableTensorEncoding` attrs (e.g. sparse
  tensor encoding) would become invalid on the new shape; those are dropped
  as before. Encodings that do not implement `VerifiableTensorEncoding`
  (opaque dict/string/bounds attrs used by downstream dialects) are
  rank-agnostic and are now preserved.

* ConvertToStaticExpandShape built the inserted `tensor.cast` target and
  the refined `expand_shape` result with no encoding. Both types preserve
  the rank of the value flowing through them, so they now carry the source's
  and result's encodings respectively.

* PadOp::inferResultType dropped the encoding. Pad preserves rank, so the
  source encoding stays valid. FoldSourceTensorCast and FoldStaticPadding
  inherit the fix through inferResultType; FoldStaticPadding additionally
  rebuilt the type directly and is updated to propagate the pad's result
  encoding.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 47 ++++++++---
 mlir/test/Dialect/Tensor/canonicalize.mlir | 93 +++++++++++++++++++++-
 2 files changed, 127 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 637366a289ac9..2c94a10a8e870 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -24,6 +24,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/TensorEncoding.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/DestinationStyleOpInterface.h"
 #include "mlir/Interfaces/InferIntRangeInterface.h"
@@ -577,7 +578,18 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  return RankedTensorType::get(sizes, tensorTypes[0].getElementType());
+  // Preserve the encoding when all inputs share it; otherwise drop it (the
+  // semantics of concatenating tensors with differing encodings are undefined
+  // at this level, so we don't try to pick one).
+  Attribute encoding = tensorTypes[0].getEncoding();
+  for (auto tensorType : llvm::drop_begin(tensorTypes)) {
+    if (tensorType.getEncoding() != encoding) {
+      encoding = Attribute();
+      break;
+    }
+  }
+  return RankedTensorType::get(sizes, tensorTypes[0].getElementType(),
+                               encoding);
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -811,11 +823,14 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
     SmallVector<int64_t> inferredOperandShape(inferredResultType.getShape());
     for (auto [operandIdx, operandType] :
          llvm::enumerate(concatOp->getOperandTypes())) {
-      // Compute inferred type for operand.
-      inferredOperandShape[dim] =
-          cast<RankedTensorType>(operandType).getDimSize(dim);
+      // Compute inferred type for operand. The refined type is applied to the
+      // operand itself, so it must carry the operand's own encoding rather
+      // than the (potentially different or missing) result encoding.
+      auto operandRankedType = cast<RankedTensorType>(operandType);
+      inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, inferredResultType.getElementType());
+          inferredOperandShape, inferredResultType.getElementType(),
+          operandRankedType.getEncoding());
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2023,7 +2038,10 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  return RankedTensorType::get(newShape, type.getElementType());
+  Attribute encoding = type.getEncoding();
+  if (llvm::isa_and_present<VerifiableTensorEncoding>(encoding))
+    encoding = {};
+  return RankedTensorType::get(newShape, type.getElementType(), encoding);
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
@@ -2273,10 +2291,13 @@ struct ConvertToStaticExpandShape : public OpRewritePattern<ExpandShapeOp> {
 
     SmallVector<OpFoldResult> outputOfr =
         getMixedValues(newOutputShape, dynamicOutputShape, rewriter);
+    // The refined types keep the ranks of the src / result respectively
     auto inputType = RankedTensorType::get(
-        newInputShape, expandOp.getSrcType().getElementType());
+        newInputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getSrcType().getEncoding());
     auto outputType = RankedTensorType::get(
-        newOutputShape, expandOp.getSrcType().getElementType());
+        newOutputShape, expandOp.getSrcType().getElementType(),
+        expandOp.getResultType().getEncoding());
     auto inputCast = CastOp::create(rewriter, expandOp.getLoc(), inputType,
                                     expandOp.getSrc());
     auto newExpand = ExpandShapeOp::create(
@@ -2385,7 +2406,8 @@ RankedTensorType ExtractSliceOp::inferCanonicalRankReducedResultType(
       if (!dimsToProject.test(pos))
         projectedShape.push_back(shape[pos]);
     inferredType =
-        RankedTensorType::get(projectedShape, inferredType.getElementType());
+        RankedTensorType::get(projectedShape, inferredType.getElementType(),
+                              inferredType.getEncoding());
   }
   return inferredType;
 }
@@ -3323,7 +3345,8 @@ RankedTensorType PadOp::inferResultType(RankedTensorType sourceType,
     }
   }
 
-  return RankedTensorType::get(inferredShape, sourceType.getElementType());
+  return RankedTensorType::get(inferredShape, sourceType.getElementType(),
+                               sourceType.getEncoding());
 }
 
 void PadOp::build(OpBuilder &b, OperationState &result, Type resultType,
@@ -3730,9 +3753,9 @@ struct FoldStaticPadding : public OpRewritePattern<PadOp> {
                      [&](int64_t x) { return x == ShapedType::kDynamic; }))
       return failure();
 
-    // Rewrite the op using the new static type.
     auto newResultType = RankedTensorType::get(
-        newOutDims, padTensorOp.getType().getElementType());
+        newOutDims, padTensorOp.getType().getElementType(),
+        padTensorOp.getType().getEncoding());
     auto newOp = PadOp::create(
         rewriter, padTensorOp->getLoc(), newResultType, input, staticLow,
         staticHigh, newLows, newHighs, padTensorOp.getNofold(),
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index 67b7ab99c5d18..c5fc71e6c7bd5 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -162,6 +162,22 @@ func.func @infer_concat_return_type(%arg0: tensor<5x12xi32>, %arg1: tensor<?x12x
 
 // -----
 
+// ConcatOp::inferResultType must carry the (uniformly-shared) operand encoding
+// onto both the refined operand cast and the refined ConcatOp result.
+// CHECK-LABEL: concat_preserves_uniform_encoding
+//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?xi32, "abc">
+//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?xi32, "abc">
+//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?xi32, "abc"> to tensor<3x?xi32, "abc">
+//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]] : (tensor<3x?xi32, "abc">, tensor<3x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+func.func @concat_preserves_uniform_encoding(
+    %a: tensor<3x?xi32, "abc">, %b: tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc"> {
+  %r = tensor.concat dim(1) %a, %b
+      : (tensor<3x?xi32, "abc">, tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc">
+  return %r : tensor<3x?xi32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1092,6 +1108,22 @@ func.func @collapse_of_cast(%t: tensor<8x12x32xf32>) -> tensor<?x32xf32> {
 
 // -----
 
+// A user-defined (non-VerifiableTensorEncoding) encoding must be preserved
+// through the collapse_of_cast folder; inferCollapsedType propagates it
+// alongside the refined shape.
+// CHECK-LABEL: func.func @collapse_of_cast_preserves_encoding(
+//  CHECK-SAME:     %[[IN:.*]]: tensor<8x12x32xf32, "abc">
+//       CHECK:   %[[COLLAPSE:.*]] = tensor.collapse_shape %[[IN]] {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, "abc"> into tensor<96x32xf32, "abc">
+//       CHECK:   tensor.cast %[[COLLAPSE]] : tensor<96x32xf32, "abc"> to tensor<?x32xf32>
+func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -> tensor<?x32xf32> {
+  %0 = tensor.cast %t : tensor<8x12x32xf32, "abc"> to tensor<?x?x?xf32, "abc">
+  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, "abc"> into tensor<?x?xf32, "abc">
+  %2 = tensor.cast %1 : tensor<?x?xf32, "abc"> to tensor<?x32xf32>
+  return %2 : tensor<?x32xf32>
+}
+
+// -----
+
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>
@@ -1773,6 +1805,28 @@ func.func @pad_nofold_same_static_shape(%arg0: tensor<5x6xf32>, %a: index)
 
 // -----
 
+// FoldSourceTensorCast (via PadOp::inferResultType) must preserve the source
+// encoding onto the refined pad result and the inserted result cast.
+// CHECK-LABEL:   func @pad_after_cast_preserves_encoding(
+//  CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[PADDED:.*]] = tensor.pad %[[INPUT]]
+//       CHECK:       : tensor<?x64x?x?xf32, "abc"> to tensor<?x64x?x?xf32, "abc">
+//       CHECK:     %[[CAST:.*]] = tensor.cast %[[PADDED]] : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+//       CHECK:     return %[[CAST]]
+func.func @pad_after_cast_preserves_encoding(
+    %arg0: tensor<?x64x?x?xf32, "abc">) -> tensor<?x?x?x?xf32, "abc"> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %dynamic = tensor.cast %arg0
+      : tensor<?x64x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  %padded = tensor.pad %dynamic low[0, 0, 1, 1] high[0, 0, 1, 1] {
+    ^bb0(%a: index, %b: index, %c: index, %d: index):
+      tensor.yield %cst: f32
+  } : tensor<?x?x?x?xf32, "abc"> to tensor<?x?x?x?xf32, "abc">
+  return %padded: tensor<?x?x?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL:   func @pad_after_cast_different_shape(
 // CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32>) -> tensor<?x?x?x?xf32> {
 // CHECK:           %[[CST:.*]] = arith.constant 0.000000e+00 : f32
@@ -1921,6 +1975,26 @@ func.func @pad_static_zero_cast(%arg0: tensor<?x?x?xf32>, %pad_value: f32) -> te
 
 // -----
 
+// FoldStaticPadding must preserve the pad's result encoding on the refined
+// (more-static) pad and the inserted result cast.
+// CHECK-LABEL: func @fold_static_padding_preserves_encoding(
+//  CHECK-SAME:     %[[SRC:.*]]: tensor<8x?xf32, "abc">
+//       CHECK:   %[[PADDED:.*]] = tensor.pad %[[SRC]] low[1, 2] high[1, 2]
+//       CHECK:     : tensor<8x?xf32, "abc"> to tensor<10x?xf32, "abc">
+//       CHECK:   tensor.cast %[[PADDED]] : tensor<10x?xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @fold_static_padding_preserves_encoding(
+    %arg0: tensor<8x?xf32, "abc">, %pv: f32) -> tensor<?x?xf32, "abc"> {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %r = tensor.pad %arg0 low[%c1, %c2] high[%c1, %c2] {
+    ^bb0(%a: index, %b: index):
+      tensor.yield %pv: f32
+  } : tensor<8x?xf32, "abc"> to tensor<?x?xf32, "abc">
+  return %r : tensor<?x?xf32, "abc">
+}
+
+// -----
+
 // CHECK-LABEL: func @pad_nofold_static_zero(
 //  CHECK-SAME:                  %[[ARG0:.*]]: tensor<?x?x?xf32>
 //       CHECK:   %[[PAD:.*]] = tensor.pad
@@ -2641,8 +2715,25 @@ func.func @partial_sink_expand_of_cast(%arg0 : tensor<10x10xf32>, %arg1 : index,
 // CHECK-LABEL:  func.func @partial_sink_expand_of_cast
 //       CHECK:   %[[CAST:.+]] = tensor.cast
 //  CHECK-SAME:     tensor<10x10xf32> to tensor<?x10xf32>
-//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]] 
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape %{{.*}} {{\[}}[0, 1], [2]]
 //  CHECK-SAME:     output_shape [%{{.*}}, %{{.*}}, 10]
 //       CHECK:   %[[RES:.+]] = tensor.cast %[[EXPAND]]
 //  CHECK-SAME:     tensor<?x?x10xf32> to tensor<?x?x?xf32>
 //       CHECK:   return %[[RES]]
+
+// -----
+
+// ConvertToStaticExpandShape must carry the source's encoding onto the refined
+// cast-target and the (source-side) encoding of the new ExpandShapeOp result.
+// CHECK-LABEL:  func.func @sink_expand_of_cast_preserves_encoding
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape
+//  CHECK-SAME:     tensor<64xf32, "abc"> into tensor<8x8xf32, "abc">
+//       CHECK:   tensor.cast %[[EXPAND]] : tensor<8x8xf32, "abc"> to tensor<?x?xf32, "abc">
+func.func @sink_expand_of_cast_preserves_encoding(%t: tensor<64xf32, "abc">) -> tensor<?x?xf32, "abc"> {
+  %c = tensor.cast %t : tensor<64xf32, "abc"> to tensor<?xf32, "abc">
+  %c8 = arith.constant 8 : index
+  %c8b = arith.constant 8 : index
+  %e = tensor.expand_shape %c [[0, 1]] output_shape [%c8, %c8b]
+      : tensor<?xf32, "abc"> into tensor<?x?xf32, "abc">
+  return %e : tensor<?x?xf32, "abc">
+}

>From 8e19f97a147696c5cd2a6f30a64f93f584359a23 Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Fri, 3 Jul 2026 14:40:16 +0000
Subject: [PATCH 2/3] Add VerifiableTensorEncoding check

This commit checks tensor encoding for `VerifiableTensorEncoding::verifyEncoding`
if possible before assigning it source type, updated by canonicalizer.

Signed-off-by: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 73 +++++++++++++-----
 mlir/test/Dialect/Tensor/canonicalize.mlir | 90 ++++++++++++++++++++++
 2 files changed, 142 insertions(+), 21 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 2c94a10a8e870..903eb5f6c411d 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -45,6 +45,25 @@
 using namespace mlir;
 using namespace mlir::tensor;
 
+/// The intent is to keep canonicalizers from silently erasing downstream
+/// metadata: a `RankedTensorType` encoding is a per-value property, and the
+/// canonicalizer is not a place to guess whether it still applies — the
+/// encoding's own contract is.
+static Attribute propagateEncoding(Attribute encoding, ArrayRef<int64_t> shape,
+                                   Type elementType) {
+  auto verifiable = dyn_cast_or_null<VerifiableTensorEncoding>(encoding);
+  if (!verifiable)
+    return encoding;
+
+  MLIRContext *ctx = encoding.getContext();
+  // to avoid user's error stream
+  ScopedDiagnosticHandler swallow(ctx, [](Diagnostic &) { return success(); });
+  auto emit = [ctx]() { return mlir::emitError(UnknownLoc::get(ctx)); };
+  return succeeded(verifiable.verifyEncoding(shape, elementType, emit))
+             ? encoding
+             : Attribute{};
+}
+
 /// Materialize a single constant operation from a given attribute value with
 /// the desired resultant type.
 Operation *TensorDialect::materializeConstant(OpBuilder &builder,
@@ -578,9 +597,9 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  // Preserve the encoding when all inputs share it; otherwise drop it (the
-  // semantics of concatenating tensors with differing encodings are undefined
-  // at this level, so we don't try to pick one).
+  // Only propagate an encoding when all inputs agree on it (concat semantics
+  // across differing encodings are undefined at this level). Let the encoding
+  // itself decide whether it still holds on the new shape.
   Attribute encoding = tensorTypes[0].getEncoding();
   for (auto tensorType : llvm::drop_begin(tensorTypes)) {
     if (tensorType.getEncoding() != encoding) {
@@ -588,8 +607,9 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
       break;
     }
   }
-  return RankedTensorType::get(sizes, tensorTypes[0].getElementType(),
-                               encoding);
+  Type elementType = tensorTypes[0].getElementType();
+  return RankedTensorType::get(sizes, elementType,
+                               propagateEncoding(encoding, sizes, elementType));
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -825,12 +845,15 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
          llvm::enumerate(concatOp->getOperandTypes())) {
       // Compute inferred type for operand. The refined type is applied to the
       // operand itself, so it must carry the operand's own encoding rather
-      // than the (potentially different or missing) result encoding.
+      // than the (potentially different or missing) result encoding, subject
+      // to the encoding still holding on the refined shape.
       auto operandRankedType = cast<RankedTensorType>(operandType);
       inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
+      Type elementType = inferredResultType.getElementType();
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, inferredResultType.getElementType(),
-          operandRankedType.getEncoding());
+          inferredOperandShape, elementType,
+          propagateEncoding(operandRankedType.getEncoding(),
+                            inferredOperandShape, elementType));
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2038,10 +2061,9 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  Attribute encoding = type.getEncoding();
-  if (llvm::isa_and_present<VerifiableTensorEncoding>(encoding))
-    encoding = {};
-  return RankedTensorType::get(newShape, type.getElementType(), encoding);
+  return RankedTensorType::get(
+      newShape, type.getElementType(),
+      propagateEncoding(type.getEncoding(), newShape, type.getElementType()));
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
@@ -2291,13 +2313,18 @@ struct ConvertToStaticExpandShape : public OpRewritePattern<ExpandShapeOp> {
 
     SmallVector<OpFoldResult> outputOfr =
         getMixedValues(newOutputShape, dynamicOutputShape, rewriter);
-    // The refined types keep the ranks of the src / result respectively
+    // The refined types are still applied to the same src/result values, so
+    // propagate their encodings, letting each encoding self-decide whether it
+    // still holds on the more-static shape.
+    Type elementType = expandOp.getSrcType().getElementType();
     auto inputType = RankedTensorType::get(
-        newInputShape, expandOp.getSrcType().getElementType(),
-        expandOp.getSrcType().getEncoding());
+        newInputShape, elementType,
+        propagateEncoding(expandOp.getSrcType().getEncoding(), newInputShape,
+                          elementType));
     auto outputType = RankedTensorType::get(
-        newOutputShape, expandOp.getSrcType().getElementType(),
-        expandOp.getResultType().getEncoding());
+        newOutputShape, elementType,
+        propagateEncoding(expandOp.getResultType().getEncoding(),
+                          newOutputShape, elementType));
     auto inputCast = CastOp::create(rewriter, expandOp.getLoc(), inputType,
                                     expandOp.getSrc());
     auto newExpand = ExpandShapeOp::create(
@@ -3345,8 +3372,10 @@ RankedTensorType PadOp::inferResultType(RankedTensorType sourceType,
     }
   }
 
-  return RankedTensorType::get(inferredShape, sourceType.getElementType(),
-                               sourceType.getEncoding());
+  Type elementType = sourceType.getElementType();
+  return RankedTensorType::get(
+      inferredShape, elementType,
+      propagateEncoding(sourceType.getEncoding(), inferredShape, elementType));
 }
 
 void PadOp::build(OpBuilder &b, OperationState &result, Type resultType,
@@ -3753,9 +3782,11 @@ struct FoldStaticPadding : public OpRewritePattern<PadOp> {
                      [&](int64_t x) { return x == ShapedType::kDynamic; }))
       return failure();
 
+    Type elementType = padTensorOp.getType().getElementType();
     auto newResultType = RankedTensorType::get(
-        newOutDims, padTensorOp.getType().getElementType(),
-        padTensorOp.getType().getEncoding());
+        newOutDims, elementType,
+        propagateEncoding(padTensorOp.getType().getEncoding(), newOutDims,
+                          elementType));
     auto newOp = PadOp::create(
         rewriter, padTensorOp->getLoc(), newResultType, input, staticLow,
         staticHigh, newLows, newHighs, padTensorOp.getNofold(),
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index c5fc71e6c7bd5..a92c94f4cc652 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -178,6 +178,30 @@ func.func @concat_preserves_uniform_encoding(
 
 // -----
 
+// `VerifiableTensorEncoding` case: concat preserves rank, so the sparse
+// encoding's rank invariant still holds on the refined shape and it is kept.
+// The operand refinement inserts a cast whose target must carry the same
+// sparse encoding (not `null`, which would need re-encoding back later).
+#sparse_dense4 = #sparse_tensor.encoding<{
+    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
+}>
+// CHECK-LABEL: concat_preserves_sparse_encoding
+//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}> to tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]]
+//  CHECK-SAME:     -> tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
+func.func @concat_preserves_sparse_encoding(
+    %a: tensor<3x?x8x8xf32, #sparse_dense4>,
+    %b: tensor<?x?x8x8xf32, #sparse_dense4>) -> tensor<3x?x8x8xf32, #sparse_dense4> {
+  %r = tensor.concat dim(1) %a, %b
+      : (tensor<3x?x8x8xf32, #sparse_dense4>, tensor<?x?x8x8xf32, #sparse_dense4>)
+     -> tensor<3x?x8x8xf32, #sparse_dense4>
+  return %r : tensor<3x?x8x8xf32, #sparse_dense4>
+}
+
+// -----
+
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1124,6 +1148,25 @@ func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -
 
 // -----
 
+// `VerifiableTensorEncoding` case: collapse changes the rank, so the sparse
+// encoding's rank invariant no longer holds; propagateEncoding asks the
+// encoding via `verifyEncoding` and drops it from the refined type.
+#sparse_dense3 = #sparse_tensor.encoding<{
+    map = (d0, d1, d2) -> (d0 : dense, d1 : dense, d2 : dense)
+}>
+// CHECK-LABEL: func.func @collapse_of_cast_drops_rank_invalid_encoding(
+//   CHECK-NOT:   collapse_shape {{.*}}#{{[a-z_0-9]+}}
+//       CHECK:   tensor.collapse_shape %{{.*}} {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, #{{[a-z_0-9]+}}> into tensor<96x32xf32>
+func.func @collapse_of_cast_drops_rank_invalid_encoding(
+    %t: tensor<8x12x32xf32, #sparse_dense3>) -> tensor<?x32xf32> {
+  %0 = tensor.cast %t : tensor<8x12x32xf32, #sparse_dense3> to tensor<?x?x?xf32, #sparse_dense3>
+  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, #sparse_dense3> into tensor<?x?xf32>
+  %2 = tensor.cast %1 : tensor<?x?xf32> to tensor<?x32xf32>
+  return %2 : tensor<?x32xf32>
+}
+
+// -----
+
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>
@@ -1827,6 +1870,30 @@ func.func @pad_after_cast_preserves_encoding(
 
 // -----
 
+// `VerifiableTensorEncoding` case: pad preserves rank, so the sparse encoding
+// stays valid on the refined shape and PadOp::inferResultType keeps it.
+#sparse_dense4b = #sparse_tensor.encoding<{
+    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
+}>
+// CHECK-LABEL:   func @pad_after_cast_preserves_sparse_encoding(
+//  CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}>
+//       CHECK:     %[[PADDED:.*]] = tensor.pad %[[INPUT]]
+//       CHECK:       : tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}> to tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}>
+//       CHECK:     tensor.cast %[[PADDED]] : tensor<?x64x?x?xf32, #{{[a-z_0-9]+}}> to tensor<?x?x?x?xf32, #{{[a-z_0-9]+}}>
+func.func @pad_after_cast_preserves_sparse_encoding(
+    %arg0: tensor<?x64x?x?xf32, #sparse_dense4b>) -> tensor<?x?x?x?xf32, #sparse_dense4b> {
+  %cst = arith.constant 0.000000e+00 : f32
+  %dynamic = tensor.cast %arg0
+      : tensor<?x64x?x?xf32, #sparse_dense4b> to tensor<?x?x?x?xf32, #sparse_dense4b>
+  %padded = tensor.pad %dynamic low[0, 0, 1, 1] high[0, 0, 1, 1] {
+    ^bb0(%a: index, %b: index, %c: index, %d: index):
+      tensor.yield %cst: f32
+  } : tensor<?x?x?x?xf32, #sparse_dense4b> to tensor<?x?x?x?xf32, #sparse_dense4b>
+  return %padded: tensor<?x?x?x?xf32, #sparse_dense4b>
+}
+
+// -----
+
 // CHECK-LABEL:   func @pad_after_cast_different_shape(
 // CHECK-SAME:      %[[INPUT:.*]]: tensor<?x64x?x?xf32>) -> tensor<?x?x?x?xf32> {
 // CHECK:           %[[CST:.*]] = arith.constant 0.000000e+00 : f32
@@ -2737,3 +2804,26 @@ func.func @sink_expand_of_cast_preserves_encoding(%t: tensor<64xf32, "abc">) ->
       : tensor<?xf32, "abc"> into tensor<?x?xf32, "abc">
   return %e : tensor<?x?xf32, "abc">
 }
+
+// -----
+
+// `VerifiableTensorEncoding` case: expand_shape refined shapes keep the src's
+// and result's ranks respectively, so both sparse encodings pass their
+// (rank-based) `verifyEncoding` and are preserved.
+#sparse_v = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+#sparse_m = #sparse_tensor.encoding<{
+    map = (d0, d1) -> (d0 : compressed, d1 : compressed)
+}>
+// CHECK-LABEL:  func.func @sink_expand_of_cast_preserves_sparse_encoding
+//       CHECK:   %[[EXPAND:.+]] = tensor.expand_shape
+//  CHECK-SAME:     tensor<64xf32, #{{[a-z_0-9]+}}> into tensor<8x8xf32, #{{[a-z_0-9]+}}>
+//       CHECK:   tensor.cast %[[EXPAND]] : tensor<8x8xf32, #{{[a-z_0-9]+}}> to tensor<?x?xf32, #{{[a-z_0-9]+}}>
+func.func @sink_expand_of_cast_preserves_sparse_encoding(
+    %t: tensor<64xf32, #sparse_v>) -> tensor<?x?xf32, #sparse_m> {
+  %c = tensor.cast %t : tensor<64xf32, #sparse_v> to tensor<?xf32, #sparse_v>
+  %c8a = arith.constant 8 : index
+  %c8b = arith.constant 8 : index
+  %e = tensor.expand_shape %c [[0, 1]] output_shape [%c8a, %c8b]
+      : tensor<?xf32, #sparse_v> into tensor<?x?xf32, #sparse_m>
+  return %e : tensor<?x?xf32, #sparse_m>
+}

>From b7578cda978b392be786b132818211e92de912d2 Mon Sep 17 00:00:00 2001
From: Dmitrii Makarenko <dmitrii.makarenko at intel.com>
Date: Mon, 13 Jul 2026 11:52:40 +0000
Subject: [PATCH 3/3] bailout

---
 mlir/lib/Dialect/Tensor/IR/TensorOps.cpp   | 32 ++--------
 mlir/test/Dialect/Tensor/canonicalize.mlir | 74 ++++++----------------
 2 files changed, 24 insertions(+), 82 deletions(-)

diff --git a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
index 903eb5f6c411d..56a93af3209fd 100644
--- a/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
+++ b/mlir/lib/Dialect/Tensor/IR/TensorOps.cpp
@@ -597,19 +597,7 @@ RankedTensorType ConcatOp::inferResultType(int64_t dim, TypeRange inputTypes) {
     concatSize =
         concatSize + SaturatedInteger::wrap(tensorType.getDimSize(dim));
   sizes[dim] = concatSize.asInteger();
-  // Only propagate an encoding when all inputs agree on it (concat semantics
-  // across differing encodings are undefined at this level). Let the encoding
-  // itself decide whether it still holds on the new shape.
-  Attribute encoding = tensorTypes[0].getEncoding();
-  for (auto tensorType : llvm::drop_begin(tensorTypes)) {
-    if (tensorType.getEncoding() != encoding) {
-      encoding = Attribute();
-      break;
-    }
-  }
-  Type elementType = tensorTypes[0].getElementType();
-  return RankedTensorType::get(sizes, elementType,
-                               propagateEncoding(encoding, sizes, elementType));
+  return RankedTensorType::get(sizes, tensorTypes[0].getElementType());
 }
 
 void ConcatOp::build(OpBuilder &builder, OperationState &result, int64_t dim,
@@ -843,17 +831,11 @@ struct InferConcatOperandTypes : public OpRewritePattern<ConcatOp> {
     SmallVector<int64_t> inferredOperandShape(inferredResultType.getShape());
     for (auto [operandIdx, operandType] :
          llvm::enumerate(concatOp->getOperandTypes())) {
-      // Compute inferred type for operand. The refined type is applied to the
-      // operand itself, so it must carry the operand's own encoding rather
-      // than the (potentially different or missing) result encoding, subject
-      // to the encoding still holding on the refined shape.
-      auto operandRankedType = cast<RankedTensorType>(operandType);
-      inferredOperandShape[dim] = operandRankedType.getDimSize(dim);
-      Type elementType = inferredResultType.getElementType();
+      // Compute inferred type for operand.
+      inferredOperandShape[dim] =
+          cast<RankedTensorType>(operandType).getDimSize(dim);
       auto inferredOperandType = RankedTensorType::get(
-          inferredOperandShape, elementType,
-          propagateEncoding(operandRankedType.getEncoding(),
-                            inferredOperandShape, elementType));
+          inferredOperandShape, inferredResultType.getElementType());
 
       // Check if inferred type is more static.
       if (!preservesStaticInformation(inferredOperandType, operandType)) {
@@ -2061,9 +2043,7 @@ CollapseShapeOp::inferCollapsedType(RankedTensorType type,
     currentDim += dim;
   }
 
-  return RankedTensorType::get(
-      newShape, type.getElementType(),
-      propagateEncoding(type.getEncoding(), newShape, type.getElementType()));
+  return RankedTensorType::get(newShape, type.getElementType());
 }
 
 void CollapseShapeOp::build(OpBuilder &b, OperationState &result, Value src,
diff --git a/mlir/test/Dialect/Tensor/canonicalize.mlir b/mlir/test/Dialect/Tensor/canonicalize.mlir
index a92c94f4cc652..26c40863e70dc 100644
--- a/mlir/test/Dialect/Tensor/canonicalize.mlir
+++ b/mlir/test/Dialect/Tensor/canonicalize.mlir
@@ -162,14 +162,18 @@ func.func @infer_concat_return_type(%arg0: tensor<5x12xi32>, %arg1: tensor<?x12x
 
 // -----
 
-// ConcatOp::inferResultType must carry the (uniformly-shared) operand encoding
-// onto both the refined operand cast and the refined ConcatOp result.
-// CHECK-LABEL: concat_preserves_uniform_encoding
+// ConcatOp genuinely merges data from multiple operands (unlike a plain
+// shape-refining cast), so `ConcatOp::inferResultType` / `InferConcatOperandTypes`
+// intentionally do NOT try to propagate an operand's encoding onto the
+// refined types
+// CHECK-LABEL: concat_drops_encoding
 //  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?xi32, "abc">
 //  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?xi32, "abc">
-//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?xi32, "abc"> to tensor<3x?xi32, "abc">
-//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]] : (tensor<3x?xi32, "abc">, tensor<3x?xi32, "abc">) -> tensor<3x?xi32, "abc">
-func.func @concat_preserves_uniform_encoding(
+//       CHECK:   %[[CAST0:.+]] = tensor.cast %[[ARG0]] : tensor<3x?xi32, "abc"> to tensor<3x?xi32>
+//       CHECK:   %[[CAST1:.+]] = tensor.cast %[[ARG1]] : tensor<?x?xi32, "abc"> to tensor<3x?xi32>
+//       CHECK:   %[[CONCAT:.+]] = tensor.concat dim(1) %[[CAST0]], %[[CAST1]] : (tensor<3x?xi32>, tensor<3x?xi32>) -> tensor<3x?xi32>
+//       CHECK:   tensor.cast %[[CONCAT]] : tensor<3x?xi32> to tensor<3x?xi32, "abc">
+func.func @concat_drops_encoding(
     %a: tensor<3x?xi32, "abc">, %b: tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc"> {
   %r = tensor.concat dim(1) %a, %b
       : (tensor<3x?xi32, "abc">, tensor<?x?xi32, "abc">) -> tensor<3x?xi32, "abc">
@@ -178,30 +182,6 @@ func.func @concat_preserves_uniform_encoding(
 
 // -----
 
-// `VerifiableTensorEncoding` case: concat preserves rank, so the sparse
-// encoding's rank invariant still holds on the refined shape and it is kept.
-// The operand refinement inserts a cast whose target must carry the same
-// sparse encoding (not `null`, which would need re-encoding back later).
-#sparse_dense4 = #sparse_tensor.encoding<{
-    map = (d0, d1, d2, d3) -> (d0 : dense, d1 : dense, d2 : dense, d3 : dense)
-}>
-// CHECK-LABEL: concat_preserves_sparse_encoding
-//  CHECK-SAME:     %[[ARG0:[a-zA-Z0-9_]+]]: tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
-//  CHECK-SAME:     %[[ARG1:[a-zA-Z0-9_]+]]: tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}>
-//       CHECK:   %[[CAST:.+]] = tensor.cast %[[ARG1]] : tensor<?x?x8x8xf32, #{{[a-z_0-9]+}}> to tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
-//       CHECK:   tensor.concat dim(1) %[[ARG0]], %[[CAST]]
-//  CHECK-SAME:     -> tensor<3x?x8x8xf32, #{{[a-z_0-9]+}}>
-func.func @concat_preserves_sparse_encoding(
-    %a: tensor<3x?x8x8xf32, #sparse_dense4>,
-    %b: tensor<?x?x8x8xf32, #sparse_dense4>) -> tensor<3x?x8x8xf32, #sparse_dense4> {
-  %r = tensor.concat dim(1) %a, %b
-      : (tensor<3x?x8x8xf32, #sparse_dense4>, tensor<?x?x8x8xf32, #sparse_dense4>)
-     -> tensor<3x?x8x8xf32, #sparse_dense4>
-  return %r : tensor<3x?x8x8xf32, #sparse_dense4>
-}
-
-// -----
-
 // CHECK-LABEL: func @fold_extract
 func.func @fold_extract(%arg0 : index) -> (f32, f16, f16, i32, complex<f32>, i32) {
   %const_0 = arith.constant 0 : index
@@ -1132,14 +1112,15 @@ func.func @collapse_of_cast(%t: tensor<8x12x32xf32>) -> tensor<?x32xf32> {
 
 // -----
 
-// A user-defined (non-VerifiableTensorEncoding) encoding must be preserved
-// through the collapse_of_cast folder; inferCollapsedType propagates it
-// alongside the refined shape.
-// CHECK-LABEL: func.func @collapse_of_cast_preserves_encoding(
+// CollapseShapeOp::inferCollapsedType changes the tensor rank, and (like
+// concat) we can't statically verify an arbitrary encoding still holds on
+// the collapsed shape when dynamic dims are involved, so the encoding is
+// dropped 
+// CHECK-LABEL: func.func @collapse_of_cast_drops_encoding(
 //  CHECK-SAME:     %[[IN:.*]]: tensor<8x12x32xf32, "abc">
-//       CHECK:   %[[COLLAPSE:.*]] = tensor.collapse_shape %[[IN]] {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, "abc"> into tensor<96x32xf32, "abc">
-//       CHECK:   tensor.cast %[[COLLAPSE]] : tensor<96x32xf32, "abc"> to tensor<?x32xf32>
-func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -> tensor<?x32xf32> {
+//       CHECK:   %[[COLLAPSE:.*]] = tensor.collapse_shape %[[IN]] {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, "abc"> into tensor<96x32xf32>
+//       CHECK:   tensor.cast %[[COLLAPSE]] : tensor<96x32xf32> to tensor<?x32xf32>
+func.func @collapse_of_cast_drops_encoding(%t: tensor<8x12x32xf32, "abc">) -> tensor<?x32xf32> {
   %0 = tensor.cast %t : tensor<8x12x32xf32, "abc"> to tensor<?x?x?xf32, "abc">
   %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, "abc"> into tensor<?x?xf32, "abc">
   %2 = tensor.cast %1 : tensor<?x?xf32, "abc"> to tensor<?x32xf32>
@@ -1148,25 +1129,6 @@ func.func @collapse_of_cast_preserves_encoding(%t: tensor<8x12x32xf32, "abc">) -
 
 // -----
 
-// `VerifiableTensorEncoding` case: collapse changes the rank, so the sparse
-// encoding's rank invariant no longer holds; propagateEncoding asks the
-// encoding via `verifyEncoding` and drops it from the refined type.
-#sparse_dense3 = #sparse_tensor.encoding<{
-    map = (d0, d1, d2) -> (d0 : dense, d1 : dense, d2 : dense)
-}>
-// CHECK-LABEL: func.func @collapse_of_cast_drops_rank_invalid_encoding(
-//   CHECK-NOT:   collapse_shape {{.*}}#{{[a-z_0-9]+}}
-//       CHECK:   tensor.collapse_shape %{{.*}} {{\[}}[0, 1], [2]] : tensor<8x12x32xf32, #{{[a-z_0-9]+}}> into tensor<96x32xf32>
-func.func @collapse_of_cast_drops_rank_invalid_encoding(
-    %t: tensor<8x12x32xf32, #sparse_dense3>) -> tensor<?x32xf32> {
-  %0 = tensor.cast %t : tensor<8x12x32xf32, #sparse_dense3> to tensor<?x?x?xf32, #sparse_dense3>
-  %1 = tensor.collapse_shape %0 [[0, 1], [2]] : tensor<?x?x?xf32, #sparse_dense3> into tensor<?x?xf32>
-  %2 = tensor.cast %1 : tensor<?x?xf32> to tensor<?x32xf32>
-  return %2 : tensor<?x32xf32>
-}
-
-// -----
-
 func.func @fold_collapse_of_expand(%arg0 : tensor<12x4xf32>) -> tensor<12x4xf32> {
   %0 = tensor.expand_shape %arg0 [[0, 1], [2]] output_shape [3, 4, 4]
       : tensor<12x4xf32> into tensor<3x4x4xf32>


From llvmlistbot at llvm.org  Mon Jul 13 05:01:24 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Mon, 13 Jul 2026 05:01:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a54d394.b1416da1.78d1e.7ed6@mx.google.com>


Devjiu wrote:

Returned the previous behavior for CollapseShapeOp and ConcatOp, but with Pad - it's a little bit tricky as "FoldStaticPadding" depends on "PadOp::inferResultType" - I left it with encoding propagation, for now. I am not sure that it's the same issue with Concat, but feel free to share your thoughts. 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Mon Jul 13 05:09:09 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Mon, 13 Jul 2026 05:09:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a54d565.f732ad81.3a4994.9673@mx.google.com>


https://github.com/nujaa updated https://github.com/llvm/llvm-project/pull/208466

>From df521a8ba9b7907c65f42787e38a620c71a68406 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 21:41:33 +0800
Subject: [PATCH 1/4] [MLIR][Linalg] Fix LinalgSpecializeGenericOpsPass for
 scalar inputs

---
 .../DecomposeGenericByUnfoldingPermutation.cpp |  6 +++---
 .../Linalg/specialize-generic-ops-fail.mlir    | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 9015cbb096f88..450cfb2641cb5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -164,8 +164,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
   // out which operand can supply that runtime-value (tensor.dim).
   // Leaving it as a future TODO.
   if (llvm::any_of(op->getOpOperands(), [](OpOperand &oper) {
-        auto opType = cast<RankedTensorType>(oper.get().getType());
-        return ShapedType::isDynamicShape(opType.getShape());
+        auto opType = dyn_cast<RankedTensorType>(oper.get().getType());
+        return !opType || ShapedType::isDynamicShape(opType.getShape());
       }))
     return failure();
 
@@ -210,7 +210,7 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
       Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  inputRTType.getElementType());
+                                                  elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);
diff --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index 5d66837fca510..bb2c2edfa20f2 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -46,3 +46,21 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
   } -> tensor<8xi32>
   return %res : tensor<8xi32>
 }
+
+
+// -----
+
+// CHECK-LABEL: @scalar_input
+// CHECK: linalg.generic
+#map = affine_map<(d0) -> (d0)>
+#map1 = affine_map<(d0) -> ()>
+func.func @scalar_input(%arg0: tensor<128xi64>, %arg1: f32) -> tensor<128xf32> {
+  %0 = tensor.empty() : tensor<128xf32>
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xi64>, f32) outs(%0 : tensor<128xf32>) {
+  ^bb0(%in: i64, %in_1: f32, %out: f32):
+    %2 = arith.sitofp %in : i64 to f32
+    %3 = arith.divf %2, %in_1 : f32
+    linalg.yield %3 : f32
+  } -> tensor<128xf32>
+  return %1 : tensor<128xf32>
+}
\ No newline at end of file

>From d9d23dbb6aac024374ed5fb3cf44712e04eca049 Mon Sep 17 00:00:00 2001
From: Hugo Trachino <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 15:57:34 +0100
Subject: [PATCH 2/4] formatting

---
 .../Transforms/DecomposeGenericByUnfoldingPermutation.cpp    | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 450cfb2641cb5..7ee89c0f06f56 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -209,8 +209,9 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     // Does it require broadcast?
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
-      Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  elType);
+      Value emptyTensor =
+          tensor::EmptyOp::create(rewriter, loc, outputShape, elType);
+
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);

>From 91f3300bf5b6aaa948a10f2cf8b77438841d8871 Mon Sep 17 00:00:00 2001
From: Hugo Trachino <hugo.trachino at huawei.com>
Date: Thu, 9 Jul 2026 16:42:32 +0100
Subject: [PATCH 3/4] Update DecomposeGenericByUnfoldingPermutation.cpp

---
 .../Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 7ee89c0f06f56..8ab0f67d9b7de 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -212,7 +212,6 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
       Value emptyTensor =
           tensor::EmptyOp::create(rewriter, loc, outputShape, elType);
 
-
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);
 

>From 3b22e2ce1b7c0ed5f1a951d17365924fbbefb5e2 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Mon, 13 Jul 2026 20:07:37 +0800
Subject: [PATCH 4/4] Simplify test

---
 .../Linalg/specialize-generic-ops-fail.mlir        | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index bb2c2edfa20f2..f0792d7a696de 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -54,13 +54,11 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
 // CHECK: linalg.generic
 #map = affine_map<(d0) -> (d0)>
 #map1 = affine_map<(d0) -> ()>
-func.func @scalar_input(%arg0: tensor<128xi64>, %arg1: f32) -> tensor<128xf32> {
-  %0 = tensor.empty() : tensor<128xf32>
-  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xi64>, f32) outs(%0 : tensor<128xf32>) {
-  ^bb0(%in: i64, %in_1: f32, %out: f32):
-    %2 = arith.sitofp %in : i64 to f32
-    %3 = arith.divf %2, %in_1 : f32
-    linalg.yield %3 : f32
+func.func @scalar_input(%arg0: tensor<128xf32>, %arg1: f32) -> tensor<128xf32> {
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xf32>, f32) outs(%arg0 : tensor<128xf32>) {
+  ^bb0(%in: f32, %in_1: f32, %out: f32):
+    %2 = arith.addf %in_1, %in_1 : f32
+    linalg.yield %2 : f32
   } -> tensor<128xf32>
   return %1 : tensor<128xf32>
-}
\ No newline at end of file
+}


From llvmlistbot at llvm.org  Mon Jul 13 05:13:04 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Mon, 13 Jul 2026 05:13:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a54d650.7270d2fd.2d4879.31a4@mx.google.com>


================

----------------
nujaa wrote:

Removed `%0` and unnecessary operator inside `linalg.generic`. 
> Make it require at most 2 indexing maps.

This error happens only if there are multiple inputs. so there is necessarily 2 maps for the inputs and 1 for the output. (shared with input). I am not sure I understand correctly what you meant.

> Perhaps reduce to linalg.generic on scalars?

You mean having both inputs scalars ? 

https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Mon Jul 13 05:20:18 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:18 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d802.07d7ca98.9dee3.87a2@mx.google.com>


https://github.com/skatrak commented:

Thank you @Saieiei, only minor nits left from me. Otherwise, this looks almost ready.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:18 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:18 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d802.4a342cbc.ca844.7d7b@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:19 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:19 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d803.d681cab9.333b2b.e015@mx.google.com>


================
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
----------------
skatrak wrote:

Nit: This is a general comment that involves changes across all files in this patch, and the PR title/description. Host fallback refers to the host version of device code that gets executed when e.g. the device is unavailable, but `in_reduction` is a host-only clause with no associated target device logic. I'd like to ask you to update your references to "`in_reduction` host fallback" and either remove that clarification or more generally refer to the host, depending on what works better for each case.
```suggestion
  /// Emit the host runtime lookups that redirect the `in_reduction`
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.a1accbb4.244acd.6a1e@mx.google.com>


================
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
----------------
skatrak wrote:

```suggestion
    function_ref<void(unsigned, Value *)> MapPrivateCB) {
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.6c5720b3.15e90c.8731@mx.google.com>


================
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivate.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivate   Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivate);
----------------
skatrak wrote:

```suggestion
      function_ref<void(unsigned, Value *)> MapPrivateCB);
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.492e23e4.14eda0.bb0d@mx.google.com>


================
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
----------------
skatrak wrote:

Nit: This description currently goes too in depth and re-states the checks explained in the `targetInReductionCapturedBy` function.
```suggestion
  // An `in_reduction` operand on `omp.target` has no dedicated entry block
  // argument; inside the region it is accessed through the block argument of a
  // matching `map_entries` entry, and the host fallback rewrites that map
  // argument to the reduction-private storage. Require every `in_reduction`
  // operand to be captured by at least one `map_entries` entry.
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.8e75e0cb.3dbce4.9718@mx.google.com>


================
@@ -8435,6 +8458,20 @@ getDynGroupprivateFallbackType(omp::FallbackModifierAttr fallbackAttr) {
   llvm_unreachable("unexpected dyn_groupprivate fallback type");
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `omp::TargetOp::verify()` in OpenMPDialect.cpp so
+/// translation matches exactly what the verifier accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
----------------
skatrak wrote:

At this point in the compilation, there are no longer any `hlfir` operations left. We can actually just compare `inReductionVar` and `mapVarPtr` directly, so adding this helper function is unnecessary.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.07d7ca98.9dee3.88b7@mx.google.com>


================
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (mapInfo &&
+          targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
----------------
skatrak wrote:

```suggestion
      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
      if (targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
        captured = true;
        break;
      }
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.bd93e731.358318.b6a6@mx.google.com>


================
@@ -8493,6 +8530,59 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand,
+      // using the same predicate as omp::TargetOp::verify(): exact same value,
+      // or another result of the same defining op. The verifier guarantees at
+      // least one match exists; more than one matching entry is a lowering
+      // ambiguity (the redirect cannot pick which map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (!mapInfo || !targetInReductionCapturedBy(v, mapInfo.getVarPtr()))
+          continue;
----------------
skatrak wrote:

`map_vars` being defined by `omp.map.info` ops is an invariant checked by the `verifyMapClause` helper called by the `omp.target` verifier, so we know here it won't be null. This can be simplified:
```suggestion
        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
        if (v != mapInfo.getVarPtr())
          continue;
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:21 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:21 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d805.d76d81af.39f9be.b6cb@mx.google.com>


================
@@ -8605,6 +8702,42 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // OpenMPIRBuilder owns the runtime-generation path: it computes the
+      // target body gtid once (so several in_reduction items do not each emit a
+      // __kmpc_global_thread_num) and emits the per-item
+      // __kmpc_task_reduction_get_th_data lookup with the required
+      // address-space casts. The MLIR-specific work stays here: collect, per
+      // item, the type the private pointer must have (the map block argument's
+      // type), and, through the callback, rebind the map block argument that
+      // stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
----------------
skatrak wrote:

```suggestion
      // Collect, per
      // item, the type the private pointer must have (the map block argument's
      // type), and, through the callback, rebind the map block argument that
      // stands in for each in_reduction list item to the per-task
      // reduction-private storage the runtime returns.
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:27 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:27 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d80b.8e75e0cb.3dbce4.9a41@mx.google.com>


================
@@ -2631,6 +2700,21 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `targetInReductionCapturedBy` in
+/// OpenMPToLLVMIRTranslation.cpp so translation matches what the verifier
+/// accepts.
----------------
skatrak wrote:

```suggestion
/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
/// result of the same defining op. We can only enforce exact value identity for block arguments, which have no defining op, because Flang emits `hlfir.declare`#0 for
/// `in_reduction` operands and `hlfir.declare`#1 for `map` var_ptr.
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:20:29 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 13 Jul 2026 05:20:29 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54d80d.6537ae93.2f1c1b.d11a@mx.google.com>


================
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item. Owning the gtid setup and the lookup loop here keeps the runtime
+  // generation inside OpenMPIRBuilder rather than the MLIR translation layer.
----------------
skatrak wrote:

Nit: Please make sure to always review your LLM's output to avoid redundant/irrelevant information.
```suggestion
  // item.
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 05:38:39 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Mon, 13 Jul 2026 05:38:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Remove Pure attribute from Linalg::IndexOp.
	(PR #68894)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/68894@github.com>
Message-ID: <6a54dc4f.1a08c196.2f87d2.0bce@mx.google.com>


rengolin wrote:

> Unless we are happy to explicitly break the interface contract in an upstream dialect out of practical reasons, which seems to be the position some are arguing, I would say this is still an issue worth resolving.

Since this is the second necro-bump in three years, it's clear some still care about it. But the original framing is outdated and unsupported. 

All I'm asking is to reframe the problem in a way that exposes what's broken without reference to the unsupported usage (the issue the PR claims to fix). 

The best way forward is to create a new RFC on it and a new PR, and not to bump again a discussion that happened over three years.

https://github.com/llvm/llvm-project/pull/68894

From llvmlistbot at llvm.org  Mon Jul 13 05:42:22 2026
From: llvmlistbot at llvm.org (Andrey Pavlenko)
Date: Mon, 13 Jul 2026 05:42:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a54dd2e.a3128543.625c1.ecfb@mx.google.com>


https://github.com/AndreyPavlenko updated https://github.com/llvm/llvm-project/pull/197281

>From c119c3a5b3f6b34b21a8f90e615051f194725ea4 Mon Sep 17 00:00:00 2001
From: Andrey Pavlenko <andrey.a.pavlenko at gmail.com>
Date: Tue, 12 May 2026 19:35:41 +0000
Subject: [PATCH] Fixed transposed transfer_read/write

The problem was that adjustStridesForPermutation() permutes strides,
but they are required to compute base offset from transfer read indices.
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 18 ++++----
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 45 +++++++++++++++++++
 2 files changed, 54 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 75ffbe58edd6c..9a994e87697f6 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -256,13 +256,8 @@ computeMemrefMeta(OpType xferOp, PatternRewriter &rewriter) {
       offsetVal = meta.getOffset();
   }
 
-  if constexpr (llvm::is_one_of<std::decay_t<OpType>, vector::TransferReadOp,
-                                vector::TransferWriteOp>::value) {
-    AffineMap permMap = xferOp.getPermutationMap();
-    // Adjust strides according to the permutation map (e.g., for transpose)
-    adjustStridesForPermutation(permMap, strides);
-  }
-
+  // Strides are returned in original memref order; permutation is applied in
+  // computeOffsets only where offsets are indexed in vector order.
   return {strides, offsetVal};
 }
 
@@ -312,13 +307,18 @@ static Value computeOffsets(VectorTransferOpInterface xferOp,
     return stepOp;
   });
 
+  // Local offsets are indexed in vector order, so permute strides; the base
+  // offset below uses the original memref-order strides.
+  SmallVector<Value> permutedStrides(strides.begin(), strides.end());
+  adjustStridesForPermutation(xferOp.getPermutationMap(), permutedStrides);
+
   // Multiply step vectors by corresponding strides
-  size_t memrefRank = strides.size();
+  size_t memrefRank = permutedStrides.size();
   size_t vectorRank = vectorShape.size();
   SmallVector<Value> strideMultiplied;
   for (size_t i = 0; i < vectorRank; ++i) {
     size_t memrefDim = memrefRank - vectorRank + i;
-    Value strideValue = strides[memrefDim];
+    Value strideValue = permutedStrides[memrefDim];
     auto mulType = dyn_cast<VectorType>(stepVectors[i].getType());
     auto bcastOp =
         vector::BroadcastOp::create(rewriter, loc, mulType, strideValue);
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index db7928724f082..3354868cd6cb9 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -703,3 +703,48 @@ gpu.func @load_0D_vector_unsupported(%source: memref<3xf32>,
 // LOAD-GATHER: vector.transfer_read
 
 }
+
+// -----
+gpu.module @xevm_module {
+gpu.func @transpose_1x1024x24x64(
+    %arg0: memref<1x1024x24x64xf16>,
+    %arg1: memref<1x24x1024x64xf16>) kernel
+    attributes {known_block_size = array<i32: 256, 1, 1>} {
+  %pad = ub.poison : f16
+  %c0 = arith.constant 0 : index
+  %block_id_x = gpu.block_id x
+  %block_id_y = gpu.block_id y
+  %block_id_z = gpu.block_id z
+  %seq_off = affine.apply affine_map<()[s0] -> (s0 * 16)>()[%block_id_y]
+  %hid_off = affine.apply affine_map<()[s0] -> (s0 * 8)>()[%block_id_z]
+  %vec = vector.transfer_read %arg0[%c0, %seq_off, %block_id_x, %hid_off], %pad
+    {in_bounds = [true, true, true, true]}
+    : memref<1x1024x24x64xf16>, vector<1x16x1x8xf16>
+  %transposed = vector.transpose %vec, [0, 2, 1, 3]
+    : vector<1x16x1x8xf16> to vector<1x1x16x8xf16>
+  vector.transfer_write %transposed, %arg1[%c0, %block_id_x, %seq_off, %hid_off]
+    {in_bounds = [true, true, true, true]}
+    : vector<1x1x16x8xf16>, memref<1x24x1024x64xf16>
+  gpu.return
+}
+
+// CHECK-LABEL: @transpose_1x1024x24x64
+// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
+// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+
+// Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
+// Scalar base offset: seq_off * 1536 (original dim1 stride),
+//                    block_id_x * 64  (original dim2 stride).
+// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
+// CHECK:     arith.muli %block_id_x, %[[C64]] : index
+// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+
+// Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
+// Scalar base offset: block_id_x * 65536 (original dim1 stride),
+//                    seq_off * 64        (original dim2 stride).
+// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
+// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
+// CHECK:     xegpu.store {{.*}}
+
+}


From llvmlistbot at llvm.org  Mon Jul 13 05:44:02 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Mon, 13 Jul 2026 05:44:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Normalize path separator in
	MLIRConfig.cmake (PR #208659)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208659@github.com>
Message-ID: <6a54dd92.b097c8a7.177164.11a8@mx.google.com>


https://github.com/Meinersbur updated https://github.com/llvm/llvm-project/pull/208659

>From 6c136abba23ee01dfe44aab1e46214c2984c73fd Mon Sep 17 00:00:00 2001
From: Michael Kruse <llvm-project at meinersbur.de>
Date: Fri, 10 Jul 2026 10:12:22 +0200
Subject: [PATCH] [MLIR] Use TO_CMAKE_PATH

---
 mlir/cmake/modules/CMakeLists.txt | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 47349f0c7eb62..e7d7d3a526b80 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -55,10 +55,12 @@ set(MLIR_CONFIG_INCLUDE_DIRS
   "${MLIR_SOURCE_DIR}/include"
   "${MLIR_BINARY_DIR}/include"
   )
-# Refer to the best host mlir-tblgen, which might be a host-optimized version
-set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
-set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
-set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+# Refer to the best host mlir-tblgen, which might be a host-optimized version.
+# TO_CMAKE_PATH normalizes path separators across platforms. Backslashes would
+# cause a syntax error in MLIRConfig.cmake.
+file(TO_CMAKE_PATH "${MLIR_TABLEGEN_EXE}" MLIR_CONFIG_TABLEGEN_EXE)
+file(TO_CMAKE_PATH "${MLIR_PDLL_TABLEGEN_EXE}" MLIR_CONFIG_PDLL_TABLEGEN_EXE)
+file(TO_CMAKE_PATH "${MLIR_SRC_SHARDER_TABLEGEN_EXE}" MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE)
 
 configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/MLIRConfig.cmake.in


From llvmlistbot at llvm.org  Mon Jul 13 05:45:37 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Mon, 13 Jul 2026 05:45:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Normalize path separator in
	MLIRConfig.cmake (PR #208659)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208659@github.com>
Message-ID: <6a54ddf1.b097c8a7.177164.2061@mx.google.com>


https://github.com/Meinersbur auto_merge_enabled https://github.com/llvm/llvm-project/pull/208659

From llvmlistbot at llvm.org  Mon Jul 13 05:52:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 05:52:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9325f22 - [MLIR] Normalize path separator in
 MLIRConfig.cmake (#208659)
Message-ID: <6a54df9a.8f540080.209be2.608b@mx.google.com>


Author: Michael Kruse
Date: 2026-07-13T12:52:37Z
New Revision: 9325f22f1cf17a6f192519163dd449d684700d2a

URL: https://github.com/llvm/llvm-project/commit/9325f22f1cf17a6f192519163dd449d684700d2a
DIFF: https://github.com/llvm/llvm-project/commit/9325f22f1cf17a6f192519163dd449d684700d2a.diff

LOG: [MLIR] Normalize path separator in MLIRConfig.cmake (#208659)

Under Windows, the paths may contain `\` path separators.
`MLIR_TABLEGEN_EXE` is added into MLIRConfig.cmake where backslashes are
interpreted as escape characters. This results in syntax errors during
`find_package(MLIR)`.

Fix by using CMake-normalized paths (`/` separator under all platforms).
LLVM/Clang do not put their tablegen paths in their config files.

Added: 
    

Modified: 
    mlir/cmake/modules/CMakeLists.txt

Removed: 
    


################################################################################
diff  --git a/mlir/cmake/modules/CMakeLists.txt b/mlir/cmake/modules/CMakeLists.txt
index 8a00a3d4de98c..42362d864874c 100644
--- a/mlir/cmake/modules/CMakeLists.txt
+++ b/mlir/cmake/modules/CMakeLists.txt
@@ -56,9 +56,11 @@ set(MLIR_CONFIG_INCLUDE_DIRS
   "${MLIR_BINARY_DIR}/include"
   )
 # Refer to the best host mlir-tblgen, which might be a host-optimized version
-set(MLIR_CONFIG_TABLEGEN_EXE "${MLIR_TABLEGEN_EXE}")
-set(MLIR_CONFIG_PDLL_TABLEGEN_EXE "${MLIR_PDLL_TABLEGEN_EXE}")
-set(MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE "${MLIR_SRC_SHARDER_TABLEGEN_EXE}")
+# TO_CMAKE_PATH normalizes path separators across platforms. Backslashes would
+# cause a syntax error in MLIRConfig.cmake.
+file(TO_CMAKE_PATH "${MLIR_TABLEGEN_EXE}" MLIR_CONFIG_TABLEGEN_EXE)
+file(TO_CMAKE_PATH "${MLIR_PDLL_TABLEGEN_EXE}" MLIR_CONFIG_PDLL_TABLEGEN_EXE)
+file(TO_CMAKE_PATH "${MLIR_SRC_SHARDER_TABLEGEN_EXE}" MLIR_CONFIG_SRC_SHARDER_TABLEGEN_EXE)
 set(MLIR_CONFIG_LINK_MLIR_DYLIB ${MLIR_LINK_MLIR_DYLIB})
 llvm_canonicalize_cmake_booleans(
   MLIR_CONFIG_LINK_MLIR_DYLIB


        

From llvmlistbot at llvm.org  Mon Jul 13 05:52:45 2026
From: llvmlistbot at llvm.org (Michael Kruse)
Date: Mon, 13 Jul 2026 05:52:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Normalize path separator in
	MLIRConfig.cmake (PR #208659)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208659@github.com>
Message-ID: <6a54df9d.ff8885e8.33de06.7dad@mx.google.com>


https://github.com/Meinersbur closed https://github.com/llvm/llvm-project/pull/208659

From llvmlistbot at llvm.org  Mon Jul 13 05:54:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 05:54:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1884c75 - [mlir][tosa] Validate variable
	element types (#205873)
Message-ID: <6a54e006.ff8885e8.33de06.8c5b@mx.google.com>


Author: lianjinfeng2003
Date: 2026-07-13T13:54:26+01:00
New Revision: 1884c7565155b603ec3946718801f53b276876bb

URL: https://github.com/llvm/llvm-project/commit/1884c7565155b603ec3946718801f53b276876bb
DIFF: https://github.com/llvm/llvm-project/commit/1884c7565155b603ec3946718801f53b276876bb.diff

LOG: [mlir][tosa] Validate variable element types (#205873)

I changed the TOSA validation path so unsupported types report a normal
diagnostic instead of crashing.
Fixes #205710

Added: 
    

Modified: 
    mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
index a3d5d21294b0d..83367fdf4e5f3 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaValidation.cpp
@@ -669,7 +669,10 @@ struct TosaValidation : public tosa::impl::TosaValidationBase<TosaValidation> {
 
   LogicalResult CheckVariable(Operation *op);
   LogicalResult CheckVariableReadOrWrite(Operation *op);
-  bool isValidElementType(Type type, const bool allowUnsigned = false);
+  LogicalResult validateValidElementType(Operation *op, Type type,
+                                         bool allowUnsigned = false);
+  LogicalResult validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned = false);
 
   SmallVector<
       std::function<LogicalResult(Operation *, const tosa::TargetEnv &)>>
@@ -1468,11 +1471,13 @@ LogicalResult TosaValidation::applyFunctionSignatureCheck(func::FuncOp op) {
   return success();
 }
 
-bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
+LogicalResult TosaValidation::validateValidElementType(Operation *op, Type type,
+                                                       bool allowUnsigned) {
   if (isa<FloatType>(type)) {
-    return isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
-               Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
-               Float6E3M2FNType, Float8E8M0FNUType>(type);
+    if (isa<Float32Type, Float16Type, BFloat16Type, Float8E4M3FNType,
+            Float8E5M2Type, Float4E2M1FNType, Float6E2M3FNType,
+            Float6E3M2FNType, Float8E8M0FNUType>(type))
+      return success();
   } else if (auto intTy = dyn_cast<IntegerType>(type)) {
     if (intTy.isSignless()) {
       switch (intTy.getWidth()) {
@@ -1483,21 +1488,47 @@ bool TosaValidation::isValidElementType(Type type, const bool allowUnsigned) {
       case 32:
       case 48:
       case 64:
-        return true;
+        return success();
       }
     } else if (allowUnsigned && intTy.isUnsigned()) {
       switch (intTy.getWidth()) {
       case 8:
       case 16:
       case 32:
-        return true;
+        return success();
       }
     }
   } else if (isa<tosa::shapeType>(type))
-    return true;
+    return success();
   else if (isa<tosa::mxint8Type, tosa::BlockScaledType>(type))
-    return true;
-  return false;
+    return success();
+
+  return op->emitOpError() << "is not profile-aligned: element type " << type
+                           << " is not legal";
+}
+
+LogicalResult
+TosaValidation::validateOperationElementTypes(Operation *op,
+                                              bool allowUnsigned) {
+  for (Value operand : op->getOperands()) {
+    Type elementTy = getElementTypeOrSelf(operand);
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  for (Type resultTy : op->getResultTypes()) {
+    Type elementTy = getElementTypeOrSelf(resultTy);
+    if (failed(validateValidElementType(op, elementTy, allowUnsigned)))
+      return failure();
+  }
+
+  if (auto variableOp = dyn_cast<tosa::VariableOp>(op)) {
+    if (failed(
+            validateValidElementType(op, variableOp.getType(), allowUnsigned)))
+      return failure();
+  }
+
+  return success();
 }
 
 void TosaValidation::runOnOperation() {
@@ -1530,22 +1561,8 @@ void TosaValidation::runOnOperation() {
     //   protect rest of code against quantized element types
     const bool allowUnsigned =
         !strictOpSpecAlignment && isa<tosa::RescaleOp>(op);
-    for (Value operand : op->getOperands()) {
-      auto elementTy = getElementTypeOrSelf(operand);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
-    for (Type resultTy : op->getResultTypes()) {
-      auto elementTy = getElementTypeOrSelf(resultTy);
-      if (!isValidElementType(elementTy, allowUnsigned)) {
-        op->emitOpError() << "is not profile-aligned: element type "
-                          << elementTy << " is not legal";
-        return signalPassFailure();
-      }
-    }
+    if (failed(validateOperationElementTypes(op, allowUnsigned)))
+      return signalPassFailure();
 
     if (strictOpSpecAlignment &&
         failed(profileComp.checkProfile(op, targetEnv)))

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
index 5ce54980c8d61..20e004077d291 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-invalid.mlir
@@ -25,3 +25,10 @@ func.func @test_avg_pool2d_adaptive_non_const_output_zp(%arg0: tensor<1x32x32x8x
        (tensor<1x32x32x8xf32>, tensor<1xf32>, tensor<1xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+module {
+  // expected-error at +1 {{'tosa.variable' op is not profile-aligned: element type 'f64' is not legal}}
+  tosa.variable @dynamic_var : tensor<2x?xf64>
+}


        

From llvmlistbot at llvm.org  Mon Jul 13 05:54:33 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 05:54:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Validate variable element types
	(PR #205873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205873@github.com>
Message-ID: <6a54e009.510f576d.1b3cb9.eaea@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/205873

From llvmlistbot at llvm.org  Mon Jul 13 06:00:48 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Mon, 13 Jul 2026 06:00:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Remove Pure attribute from Linalg::IndexOp.
	(PR #68894)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/68894@github.com>
Message-ID: <6a54e180.ece797f7.20fbae.45a4@mx.google.com>


matthias-springer wrote:

Multiple comments above ([example](https://github.com/llvm/llvm-project/pull/68894#issuecomment-1765549641)) have suggested using block arguments. We have recently added a new token type to MLIR. Another approach that won't regress CSE/DCE/... would be to add a new token block argument to `linalg.generic`, and let `linalg.index` take that token as an operand. That way, `linalg.index` could remain `Pure`.


https://github.com/llvm/llvm-project/pull/68894

From llvmlistbot at llvm.org  Mon Jul 13 06:12:05 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 13 Jul 2026 06:12:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Fix crash in
 SparseTensorEmptyConverter (PR #208913)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208913@github.com>
Message-ID: <6a54e425.6d3ea74f.12cc6.1f2d@mx.google.com>


================

----------------
secona wrote:

Looks like many functions assume that the type is a valid primary type, so changing the function signatures requires changing a lot of lines and also I don't think the benefits are significant, in contrast to adding primary type checks at the top of each matchAndRewrite.

https://github.com/llvm/llvm-project/pull/208913

From llvmlistbot at llvm.org  Mon Jul 13 06:12:32 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Mon, 13 Jul 2026 06:12:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [memref] Support non-scalar copies in
 `reinterpret_cast` elision (PR #203873)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203873@github.com>
Message-ID: <6a54e440.a2448446.879a1.cb51@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/203873

>From 365f02115a63122f456b894ffb334a4aab52390c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 12 Jun 2026 20:00:44 +0200
Subject: [PATCH 01/17] [memref] Support non-scalar copies in reinterpret_cast
 elision

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElideReinterpretCast.cpp       | 465 +++++++++++++-----
 .../MemRef/elide-reinterpret-cast.mlir        | 258 +++++++++-
 4 files changed, 582 insertions(+), 145 deletions(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 915667190a3d3..7b54be85db340 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -19,6 +19,9 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     operations to obtain compatible shapes with equivalent ops that operate on
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
+  let dependentDialects = [
+      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+  ];
 }
 
 def ExpandOpsPass : Pass<"memref-expand"> {
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1d409b6f06242..1a8b03dabcfb7 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -36,6 +36,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   MLIRMemRefDialect
   MLIRMemRefUtils
   MLIRPass
+  MLIRSCFDialect
   MLIRTensorDialect
   MLIRTransforms
   MLIRValueBoundsOpInterface
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 0fd1ff83f15b5..f8ee5d27b7f89 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -11,10 +11,10 @@
 #include "mlir/Dialect/Arith/Utils/Utils.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
-#include "llvm/ADT/Repeated.h"
 #include <cassert>
 #include <optional>
 
@@ -29,128 +29,268 @@ using namespace mlir;
 
 namespace {
 
-/// Returns true if `rc` represents a scalar view (all sizes == 1)
-/// into a memref that has exactly one non-unit dimension located at
-/// either the first or last position (i.e. a "row" or "column").
+static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
+  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
+    return std::nullopt;
+
+  SmallVector<int64_t> strides(type.getRank(), 1);
+  int64_t stride = 1;
+  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
+    strides[dim] = stride;
+    stride *= type.getDimSize(dim);
+  }
+  return strides;
+}
+
+static std::optional<unsigned>
+findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
+                         int64_t viewSize) {
+  std::optional<unsigned> fallback;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    if (usedBaseDims[idx] || stride != viewStride ||
+        baseType.getDimSize(idx) < viewSize)
+      continue;
+
+    // Prefer an exact shape match. Otherwise, use the first dimension large
+    // enough to contain the copied logical vector.
+    if (baseType.getDimSize(idx) == viewSize)
+      return idx;
+    if (!fallback)
+      fallback = idx;
+  }
+  return fallback;
+}
+
+static std::optional<SmallVector<int64_t>>
+delinearizeStaticOffset(int64_t offset, MemRefType baseType,
+                        ArrayRef<int64_t> baseStrides) {
+  if (offset < 0)
+    return std::nullopt;
+
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = offset;
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+    indices[idx] = remainder / stride;
+    if (indices[idx] >= baseType.getDimSize(idx))
+      return std::nullopt;
+    remainder %= stride;
+  }
+
+  if (remainder != 0)
+    return std::nullopt;
+  return indices;
+}
+
+static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
+  if (!type.hasStaticShape() || type.getRank() == 0)
+    return std::nullopt;
+
+  std::optional<unsigned> nonUnitDim;
+  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+    if (dim == 1)
+      continue;
+    if (nonUnitDim)
+      return std::nullopt;
+    nonUnitDim = idx;
+  }
+  return nonUnitDim;
+}
+
+struct CopyLoopDimInfo {
+  unsigned viewDim;
+  unsigned dstLoopDim;
+  int64_t loopSize;
+};
+
+struct CopyToLoadStoreInfo {
+  SmallVector<CopyLoopDimInfo> loopDims;
+  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Builds the index mapping needed to replace a copy into a reinterpret_cast
+/// view with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Checklist:
+/// - The copy destination must be a `memref.reinterpret_cast`.
+/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
+///   be ranked memrefs with static shapes.
+/// - The reinterpret_cast source/result ranks must match.
+/// - The reinterpret_cast source must have static identity layout.
+/// - Each non-unit copied view dimension must have a static stride that maps to
+///   an identity-layout base dimension.
+/// - Static offsets, dynamic only in scalar or effectively-1D copies
+///   where the offset can be used directly as one base index.
 ///
-///   // Row-major slice (last dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x1x8xi32> to memref<1x1x1xi32>
+/// Examples that return true:
 ///
-///   // Column-major slice (first dim is non-unit)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<2x1xf32> to memref<1x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
+///   // dynamic strides in the scalar view do not affect index mapping.
+///   copy memref<1x...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Random strides
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [10, 100]
-///     : memref<2x1xf32, strided<[10, 100]>>
-///         to memref<1x1xf32>
+///   // Effectively-1D copy. The single non-unit view dimension is mapped to
+///   // an identity-layout base dimension by its static stride.
+///   copy memref<1x...xNx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
 ///
-///   // Rank-1 case
-///   memref.reinterpret_cast %buf to offset: [%off],
-///     sizes: [1], strides: [1]
-///     : memref<8xi32> to memref<1xi32>
+///   // Multidimensional copy with static offset. Each non-unit view dimension
+///   // is mapped independently by its static stride.
+///   copy memref<1x...xNx...xKx...x1xf32>
+///     to reinterpret_cast memref<base-shape>
+///       to memref<1x...xNx...xKx...x1xf32,
+///                 strided<[..., S0, ..., S1, ...], offset: O>>
 ///
 /// Examples that return false:
 ///
-///   // More non-unit dims
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1, 1], strides: [1, 1, 1]
-///     : memref<1x2x8xi32> to memref<1x1x1xi32>
+///   // Dynamic stride on a copied view dimension.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxf32, strided<[?, ?]>>
 ///
-///   // View is not scalar (size != 1)
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [2, 1], strides: [1, 1]
-///     : memref<1x2xf32> to memref<2x1xf32>
-///
-///   // Base has non-identity layout
-///   %buff = memref.alloc() : memref<1x2xf32, strided<[1, 3]>>
-///   memref.reinterpret_cast %buff to offset: [%off],
-///     sizes: [1, 1], strides: [1, 1]
-///     : memref<1x2xf32, strided<[1, 3]>> to memref<1x1xf32>
-static bool isScalarSlice(memref::ReinterpretCastOp rc) {
-  auto rcInputTy = dyn_cast<MemRefType>(rc.getSource().getType());
-  auto rcOutputTy = dyn_cast<MemRefType>(rc.getType());
-
-  // Reject strided base - logic for computing linear idx is TODO
-  if (!rcInputTy.getLayout().isIdentity())
-    return false;
-
-  // Reject non-matching ranks
-  unsigned srcRank = rcInputTy.getRank();
-  if (srcRank != rcOutputTy.getRank())
-    return false;
-
-  ArrayRef<int64_t> sizes = rc.getStaticSizes();
-
-  // View must be scalar: memref<1x...x1>
-  if (!llvm::all_of(rcOutputTy.getShape(),
-                    [](int64_t dim) { return dim == 1; }))
-    return false;
-
-  // Sizes must all be statically 1
-  if (!llvm::all_of(sizes, [](int64_t size) {
-        return !ShapedType::isDynamic(size) && size == 1;
-      }))
-    return false;
-
-  // Rank-1 special case
-  if (srcRank == 1) {
-    // Reject non-scalar output
-    if (rcOutputTy.getDimSize(0) > 1)
-      return false;
+///   // Multidimensional copy with dynamic linear offset.
+///   copy memref<1xNxKxf32>
+///     to reinterpret_cast memref<1xNxMxf32>
+///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
+static std::optional<CopyToLoadStoreInfo>
+getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
+  // TODO: Support unranked copy sources or reinterpret_cast sources.
+  if (!srcType || !baseType || !viewType)
+    return std::nullopt;
+
+  // TODO: Support rank-changing reinterpret_casts by converting the
+  // destination view indices to base indices. For example, a copy to a
+  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
+  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
+  // indexing the rank-1 base memref.
+  if (baseType.getRank() != viewType.getRank())
+    return std::nullopt;
+
+  // TODO: Support dynamic shapes with mixed size operands as loop bounds.
+  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
+        viewType.hasStaticShape()))
+    return std::nullopt;
+
+  assert(srcType.getShape() == viewType.getShape() &&
+         "copy source and destination are expected to have the same shape");
+
+  // Store indices are formed in the reinterpret_cast source layout.
+  std::optional<SmallVector<int64_t>> baseStrides =
+      getIdentityStrides(baseType);
+  // TODO: Support non-identity reinterpret_cast source layouts by using the
+  // source layout strides as base strides.
+  if (!baseStrides)
+    return std::nullopt;
+
+  CopyToLoadStoreInfo info;
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
+  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
+    if (viewSize == 1)
+      continue;
+
+    // TODO: Support dynamic strides on copied view dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+      return std::nullopt;
+
+    std::optional<unsigned> dstLoopDim =
+        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
+                                 rc.getStaticStrides()[viewDim], viewSize);
+    assert(dstLoopDim &&
+           "static reinterpret_cast stride must map to an identity base "
+           "dimension");
+
+    usedBaseDims[*dstLoopDim] = true;
+    info.loopDims.push_back(
+        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
   }
 
-  int nonUnitCount =
-      std::count_if(rcInputTy.getShape().begin(), rcInputTy.getShape().end(),
-                    [](int dim) { return dim != 1; });
-  return nonUnitCount == 1;
+  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
+  assert(staticOffsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(staticOffsets[0])) {
+    // Static offsets are converted to base indices.
+    std::optional<SmallVector<int64_t>> offsetIndices =
+        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+    assert(offsetIndices &&
+           "static reinterpret_cast offset must delinearize to in-bounds base "
+           "indices");
+
+    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
+      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
+                 baseType.getDimSize(loopDim.dstLoopDim) &&
+             "reinterpret_cast metadata describes an invalid accessible "
+             "region");
+    }
+    info.staticOffsetIndices = std::move(*offsetIndices);
+    return info;
+  }
+
+  // Dynamic offsets are kept only when they can be used as a single base index.
+  // TODO: Support multidimensional dynamic offsets with div/mod
+  // delinearization.
+  if (info.loopDims.size() > 1)
+    return std::nullopt;
+
+  if (info.loopDims.empty()) {
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions, and all-unit bases with a provably zero offset.
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    if (!nonUnitDim)
+      return std::nullopt;
+
+    info.dynamicOffsetDim = *nonUnitDim;
+    return info;
+  }
+
+  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  info.dynamicOffsetDim =
+      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+  return info;
 }
 
-/// Rewrites `memref.copy` of a 1-element MemRef as a scalar load-store pair
+/// Rewrites supported copy operations through `memref.reinterpret_cast` to
+/// scalar load/store operations.
 ///
-/// The pattern matches a reinterpret_cast that creates a scalar view
-/// (`sizes = [1, ..., 1]`) into a memref with a single non-unit dimension.
-/// Since the view contains only one element, the accessed address is
-/// determined solely by the base pointer and the offset.
+///   // BEFORE (scalar copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, ..., 1], strides: [...]
+///   memref.copy %src, %view
 ///
-/// Two layouts are supported:
-///   * row-major slice  (stride pattern [N, ..., 1])
-///   * column-major slice (stride pattern [1, ..., N])
+///   // AFTER
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(O)]
 ///
-/// BEFORE (row-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [N, ..., 1]
-///       : memref<1x...xNxf32>
-///         to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
+///   // BEFORE (effectively-1D copy)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, 1], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[N, ..., 1], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%c0, ..., %off] : memref<1x...xNxf32>
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
+///   }
 ///
-/// BEFORE (column-major slice)
-///   %view = memref.reinterpret_cast %base
-///     to offset: [%off], sizes: [1, ..., 1], strides: [1, ..., N]
-///       : memref<Nx...x1xf32>
-///         to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
+///   // BEFORE (multidimensional copy with static offset)
+///   %view = memref.reinterpret_cast %dst
+///     to offset: [O], sizes: [1, N, K], strides: [...]
 ///   memref.copy %src, %view
-///     : memref<1x...x1xf32>
-///       to memref<1x...x1xf32, strided<[1, ..., N], offset: ?>>
 ///
-/// AFTER
-///   %c0 = arith.constant 0 : index
-///   %v  = memref.load %src[%c0, ..., %c0] : memref<1x...x1xf32>
-///   memref.store %v, %base[%off, ..., %c0] : memref<Nx...x1xf32>
-struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
+///   // AFTER
+///   scf.for %i = 0 to N step 1 {
+///     scf.for %j = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %j]
+///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
+///     }
+///   }
+struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
@@ -162,38 +302,112 @@ struct CopyToScalarLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    if (!isScalarSlice(rc))
+    std::optional<CopyToLoadStoreInfo> copyInfo =
+        getCopyToLoadStoreInfo(op, rc);
+    if (!copyInfo)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar slice");
+          op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    auto dstType = cast<MemRefType>(dst.getType());
-    unsigned dstRank = dstType.getRank();
+    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType dstType = cast<MemRefType>(dst.getType());
 
     Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    Value one;
+    // Reuse common index constants across bounds, steps, and static offsets.
+    // Keep `%c1` lazy so scalar copies without loops do not create an unused
+    // loop-step constant.
+    auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
+      if (value == 0)
+        return zero;
+      if (value == 1) {
+        if (!one)
+          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        return one;
+      }
+      return arith::ConstantIndexOp::create(rewriter, loc, value);
+    };
+
+    // Materialize all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound may be created inside an outer loop body.
+    SmallVector<Value> upperBounds;
+    upperBounds.reserve(copyInfo->loopDims.size());
+    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+
+    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
+    // Static offsets were already delinearized into base indices. Materialize
+    // the non-zero starting indices before creating loop bodies.
+    if (!copyInfo->staticOffsetIndices.empty()) {
+      for (auto [idx, offset] :
+           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+        if (offset == 0)
+          continue;
+        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+      }
+    } else if (copyInfo->dynamicOffsetDim) {
+      // Supported dynamic offsets are used directly in exactly one base
+      // dimension selected by getCopyToLoadStoreInfo.
+      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      assert(offsets.size() == 1 && "Expecting single offset");
+      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+    }
 
-    auto srcType = cast<MemRefType>(src.getType());
-    Repeated<Value> loadIndices(srcType.getRank(), zero);
-    auto offsets = rc.getMixedOffsets();
-    assert(offsets.size() == 1 && "Expecting single offset");
-    OpFoldResult offset = offsets[0];
-    Value storeOffset = getValueOrCreateConstantIndexOp(rewriter, loc, offset);
-    unsigned offsetDim = dstType.getDimSize(0) == 1 ? dstRank - 1 : 0;
-    SmallVector<Value> storeIndices(dstRank, zero);
-    storeIndices[offsetDim] = storeOffset;
-    // If the only user of `rc` is the current Op (which is about to be erased),
-    // we can safely erase it.
-    if (rcOutput.hasOneUse())
-      rewriter.eraseOp(rc);
+    // Scope for OpBuilder::InsertionGuard.
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      Value step;
+      if (!upperBounds.empty())
+        step = getOrCreateIndexConstant(1);
+
+      SmallVector<Value> loopIvs;
+      loopIvs.reserve(copyInfo->loopDims.size());
+
+      // Build one nested loop per non-unit copied view dimension.
+      for (Value upperBound : upperBounds) {
+        scf::ForOp loop =
+            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+        loopIvs.push_back(loop.getInductionVar());
+        rewriter.setInsertionPointToStart(loop.getBody());
+      }
 
-    Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-    memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      // Load indices are zero except for copied view dimensions, which use the
+      // corresponding loop induction variables.
+      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      unsigned loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
+        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+
+      // Store indices start from the offset-derived base indices. Add each loop
+      // IV to the mapped base dimension.
+      SmallVector<Value> storeIndices(baseStoreIndices);
+      loopIndex = 0;
+      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+        Value iv = loopIvs[loopIndex++];
+        if (storeIndices[loopDim.dstLoopDim] == zero) {
+          storeIndices[loopDim.dstLoopDim] = iv;
+        } else {
+          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+        }
+      }
 
+      // Emit the scalar load/store at the innermost loop body, or directly at
+      // the original copy location for scalar copies.
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+    }
+
+    // If the only user of `rc` is the current Op (which is about to be erased),
+    // we can safely erase it.
+    bool eraseRc = rcOutput.hasOneUse();
     rewriter.eraseOp(op);
+    if (eraseRc)
+      rewriter.eraseOp(rc);
     return success();
   }
 };
@@ -398,7 +612,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !isScalarSlice(rc);
+      return !getCopyToLoadStoreInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
@@ -406,7 +620,8 @@ struct ElideReinterpretCastPass
         return true;
       return !getNonUnitDimMapping(rc);
     });
-    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect>();
+    target.addLegalDialect<arith::ArithDialect, memref::MemRefDialect,
+                           scf::SCFDialect>();
     if (failed(applyPartialConversion(getOperation(), target,
                                       std::move(patterns))))
       signalPassFailure();
@@ -417,6 +632,6 @@ struct ElideReinterpretCastPass
 
 void mlir::memref::populateElideReinterpretCastPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<CopyToScalarLoadAndStore, RewriteLoadFromReinterpretCast>(
+  patterns.add<CopyToLoadAndStore, RewriteLoadFromReinterpretCast>(
       patterns.getContext());
 }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 0e23372d754a5..1cd6a1dc9756a 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -19,9 +19,8 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   /// Ensure copy was replaced
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
   return
@@ -85,14 +84,14 @@ func.func private @concat_strided(%src : memref<1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
+// Dynamic strides are irrelevant because all view indices are zero.
 // CHECK-LABEL: func.func private @concat_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -108,10 +107,9 @@ func.func private @concat_dynamic_stride(%stride0: index,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   /// Dynamic offset used in store
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0_0]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -129,9 +127,8 @@ func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0_0]]] : memref<108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1xf32> to memref<1xf32>
   return
@@ -149,14 +146,165 @@ func.func private @concat_rank3(%src : memref<1x1x1xf32>,
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C0_0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0_0]]] : memref<1x1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   return
 }
 
+// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_0d(
+  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x33x42xf32>
+      to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_zero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_nonzero_offset(
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
+func.func private @concat_1d_vector_dynamic_offset_same_dim(
+  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+  %offset : index, %src : memref<1x33x1xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @concat_2d_vector_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+func.func private @concat_2d_vector_offset(
+  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
@@ -179,8 +327,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_rank_change(
-func.func private @negative_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_concat_rank_change(
+func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -195,19 +343,90 @@ func.func private @negative_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_multiple_non_unit_dims(
-func.func private @negative_concat_multiple_non_unit_dims(
-  %src : memref<1x1xf32>, %dst : memref<2x108xf32>) {
+// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
+func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [1, 1]
-    : memref<2x108xf32>
-      to memref<1x1xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
+
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32>
+    : memref<?xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
+func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [%dim], strides: [1]
+    : memref<?xf32> to memref<?xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<?xf32, strided<[1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
+func.func private @negative_concat_dynamic_offset_multi_dim_base(
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
+    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32>
+      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
+func.func private @negative_concat_2d_dynamic_offset(
+  %offset : index, %src : memref<1x33x4xf32>,
+  %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x4xf32>
+      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+  return
+}
+
+/// Non-unit copied dimension needs stride-based address computation.
+// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
+func.func private @negative_concat_dynamic_rc_stride(%stride : index,
+  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
+    : memref<1x33x42xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x33x1xf32>
+      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
   return
 }
 
@@ -222,7 +441,6 @@ func.func private @negative_plain_copy(%src : memref<1x1xf32>,
   return
 }
 
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From c2298438ffd349272002757840954512d91aa1c0 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 17 Jun 2026 17:10:59 +0200
Subject: [PATCH 02/17] Address first round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 268 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        | 264 ++++++++---------
 2 files changed, 264 insertions(+), 268 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index f8ee5d27b7f89..b62ac4703b80f 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
+#include <array>
 #include <cassert>
 #include <optional>
 
@@ -29,6 +30,10 @@ using namespace mlir;
 
 namespace {
 
+/// Copy rewrite helpers
+
+/// The copy rewrite stores directly into the reinterpret_cast source, so it
+/// needs concrete base strides instead of layout-map reasoning.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -42,19 +47,22 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
+/// Non-unit strided memref dimensions are lowered to loops over base memref
+/// dimensions. Static reinterpret_cast strides connect those dimension spaces
+/// in the currently supported cases.
 static std::optional<unsigned>
-findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                         ArrayRef<bool> usedBaseDims, int64_t viewStride,
-                         int64_t viewSize) {
+findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
+                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
+                           int64_t resultSize) {
   std::optional<unsigned> fallback;
   for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != viewStride ||
-        baseType.getDimSize(idx) < viewSize)
+    if (usedBaseDims[idx] || stride != resultStride ||
+        baseType.getDimSize(idx) < resultSize)
       continue;
 
     // Prefer an exact shape match. Otherwise, use the first dimension large
     // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == viewSize)
+    if (baseType.getDimSize(idx) == resultSize)
       return idx;
     if (!fallback)
       fallback = idx;
@@ -62,6 +70,8 @@ findBaseDimForViewStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
   return fallback;
 }
 
+/// reinterpret_cast offsets are linear element offsets, while `memref.store`
+/// needs one index per base dimension.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticOffset(int64_t offset, MemRefType baseType,
                         ArrayRef<int64_t> baseStrides) {
@@ -82,6 +92,8 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
+/// Dynamic scalar offsets cannot be delinearized statically. They can be used
+/// directly only when the base has a single non-unit dimension to receive them.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   if (!type.hasStaticShape() || type.getRank() == 0)
     return std::nullopt;
@@ -97,56 +109,48 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
+/// Per-dimension loop nest info.
 struct CopyLoopDimInfo {
-  unsigned viewDim;
-  unsigned dstLoopDim;
-  int64_t loopSize;
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
 };
 
-struct CopyToLoadStoreInfo {
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
   SmallVector<CopyLoopDimInfo> loopDims;
-  SmallVector<int64_t> staticOffsetIndices;
+  std::optional<SmallVector<int64_t>> staticOffsetIndices;
   std::optional<unsigned> dynamicOffsetDim;
 };
 
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// view with scalar stores into the reinterpret_cast base.
-///
-/// Checklist:
-/// - The copy destination must be a `memref.reinterpret_cast`.
-/// - The copy source, reinterpret_cast source, and reinterpret_cast result must
-///   be ranked memrefs with static shapes.
-/// - The reinterpret_cast source/result ranks must match.
-/// - The reinterpret_cast source must have static identity layout.
-/// - Each non-unit copied view dimension must have a static stride that maps to
-///   an identity-layout base dimension.
-/// - Static offsets, dynamic only in scalar or effectively-1D copies
-///   where the offset can be used directly as one base index.
+/// strided memref with scalar stores into the reinterpret_cast base.
 ///
-/// Examples that return true:
+/// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so
-///   // dynamic strides in the scalar view do not affect index mapping.
-///   copy memref<1x...x1xf32>
+///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
+///   // strides in the strided memref do not affect index mapping.
+///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...x1xf32, strided<[?, ..., ?], offset: ?>>
+///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
-///   // Effectively-1D copy. The single non-unit view dimension is mapped to
-///   // an identity-layout base dimension by its static stride.
-///   copy memref<1x...xNx...x1xf32>
+///   // Effectively-1D copy. The single non-unit strided memref dimension is
+///   // mapped to an identity-layout base dimension by its static stride.
+///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...x1xf32, strided<[..., S, ...]>>
+///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit view dimension
-///   // is mapped independently by its static stride.
-///   copy memref<1x...xNx...xKx...x1xf32>
+///   // Multidimensional copy with static offset. Each non-unit strided memref
+///   // dimension is mapped independently by its static stride.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<base-shape>
-///       to memref<1x...xNx...xKx...x1xf32,
-///                 strided<[..., S0, ..., S1, ...], offset: O>>
+///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
+///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
-/// Examples that return false:
+/// Examples that return no info:
 ///
-///   // Dynamic stride on a copied view dimension.
+///   // Dynamic stride on a copied strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -155,32 +159,35 @@ struct CopyToLoadStoreInfo {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyToLoadStoreInfo>
-getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
+static std::optional<CopyFromReinterCastInfo>
+getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
   MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType viewType = dyn_cast<MemRefType>(rc.getType());
-  // TODO: Support unranked copy sources or reinterpret_cast sources.
-  if (!srcType || !baseType || !viewType)
+  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+
+  // Ranked memref types are required to statically build load/store index
+  // lists.
+  if (!srcType || !baseType || !resultType)
+    return std::nullopt;
+
+  if (srcType.getShape() != resultType.getShape())
     return std::nullopt;
 
   // TODO: Support rank-changing reinterpret_casts by converting the
-  // destination view indices to base indices. For example, a copy to a
-  // memref<2x3xf32> view of memref<6xf32> needs to linearize the view indices
-  // as `i * 3 + j`, then combine that with the reinterpret_cast offset before
-  // indexing the rank-1 base memref.
-  if (baseType.getRank() != viewType.getRank())
+  // strided memref indices to base indices. For example, a copy to
+  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
+  // strided memref indices as `i * 3 + j`, then combine that with the
+  // reinterpret_cast offset before indexing the rank-1 base memref.
+  if (baseType.getRank() != resultType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
   if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        viewType.hasStaticShape()))
+        resultType.hasStaticShape()))
     return std::nullopt;
 
-  assert(srcType.getShape() == viewType.getShape() &&
-         "copy source and destination are expected to have the same shape");
-
-  // Store indices are formed in the reinterpret_cast source layout.
+  // Map reinterpret_cast strided memref dimensions to source dimensions by
+  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
   // TODO: Support non-identity reinterpret_cast source layouts by using the
@@ -188,60 +195,68 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   if (!baseStrides)
     return std::nullopt;
 
-  CopyToLoadStoreInfo info;
+  CopyFromReinterCastInfo info;
   SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  // Non-unit view dimensions become loop dimensions in the scalar rewrite.
-  for (auto [viewDim, viewSize] : llvm::enumerate(viewType.getShape())) {
-    if (viewSize == 1)
+  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied view dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[viewDim]))
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    std::optional<unsigned> dstLoopDim =
-        findBaseDimForViewStride(baseType, *baseStrides, usedBaseDims,
-                                 rc.getStaticStrides()[viewDim], viewSize);
-    assert(dstLoopDim &&
+    // Non-unit strided memref dimensions become loop dimensions in the scalar
+    // rewrite.
+    std::optional<unsigned> baseDim = findBaseDimForResultStride(
+        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
+        resultSize);
+    assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
 
-    usedBaseDims[*dstLoopDim] = true;
-    info.loopDims.push_back(
-        CopyLoopDimInfo{static_cast<unsigned>(viewDim), *dstLoopDim, viewSize});
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
 
-  ArrayRef<int64_t> staticOffsets = rc.getStaticOffsets();
-  assert(staticOffsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(staticOffsets[0])) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  if (!ShapedType::isDynamic(offsets[0])) {
     // Static offsets are converted to base indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(staticOffsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
     for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.dstLoopDim] + loopDim.loopSize <=
-                 baseType.getDimSize(loopDim.dstLoopDim) &&
+      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
+                 baseType.getDimSize(loopDim.baseDim) &&
              "reinterpret_cast metadata describes an invalid accessible "
              "region");
     }
-    info.staticOffsetIndices = std::move(*offsetIndices);
+    info.staticOffsetIndices = std::move(offsetIndices);
     return info;
   }
 
   // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support multidimensional dynamic offsets with div/mod
-  // delinearization.
+  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
+  // delinearizing the offset into base start indices at runtime before adding
+  // loop IVs.
   if (info.loopDims.size() > 1)
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions, and all-unit bases with a provably zero offset.
+    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
+    // offset can be used directly only when the base has exactly one non-unit
+    // dimension.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
+    // dimensions by delinearizing the single accessed element offset at
+    // runtime.
     if (!nonUnitDim)
       return std::nullopt;
 
@@ -249,9 +264,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned dstLoopDim = info.loopDims.front().dstLoopDim;
+  unsigned baseDim = info.loopDims.front().baseDim;
   info.dynamicOffsetDim =
-      (*baseStrides)[dstLoopDim] == 1 ? dstLoopDim : baseStrides->size() - 1;
+      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
   return info;
 }
 
@@ -259,18 +274,18 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 /// scalar load/store operations.
 ///
 ///   // BEFORE (scalar copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(O)]
 ///
 ///   // BEFORE (effectively-1D copy)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, 1], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -279,9 +294,9 @@ getCopyToLoadStoreInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
 ///   }
 ///
 ///   // BEFORE (multidimensional copy with static offset)
-///   %view = memref.reinterpret_cast %dst
+///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [O], sizes: [1, N, K], strides: [...]
-///   memref.copy %src, %view
+///   memref.copy %src, %strided
 ///
 ///   // AFTER
 ///   scf.for %i = 0 to N step 1 {
@@ -302,8 +317,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyToLoadStoreInfo> copyInfo =
-        getCopyToLoadStoreInfo(op, rc);
+    std::optional<CopyFromReinterCastInfo> copyInfo =
+        getCopyFromReinterCastInfo(op, rc);
     if (!copyInfo)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
@@ -315,43 +330,51 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType srcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
-    Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
-    Value one;
-    // Reuse common index constants across bounds, steps, and static offsets.
-    // Keep `%c1` lazy so scalar copies without loops do not create an unused
-    // loop-step constant.
+    // Reuse common index constants across bounds, steps, and static offsets,
+    // but avoid creating them for rank-0 copies.
+    std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
-      if (value == 0)
-        return zero;
-      if (value == 1) {
-        if (!one)
-          one = arith::ConstantIndexOp::create(rewriter, loc, 1);
-        return one;
+      if (value == 0 || value == 1) {
+        Value &cached = cachedIndexConstants[value];
+        if (!cached)
+          cached = arith::ConstantIndexOp::create(rewriter, loc, value);
+        return cached;
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+    auto getZeroIndices = [&](int64_t rank) {
+      SmallVector<Value> indices;
+      indices.reserve(rank);
+      if (rank != 0)
+        indices.append(rank, getOrCreateIndexConstant(0));
+      return indices;
+    };
 
-    // Materialize all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound may be created inside an outer loop body.
+    // Create all loop bounds before building the loop nest. Otherwise an
+    // inner-loop bound can be inserted inside an outer loop body.
     SmallVector<Value> upperBounds;
     upperBounds.reserve(copyInfo->loopDims.size());
     for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.loopSize));
+      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
 
-    SmallVector<Value> baseStoreIndices(dstType.getRank(), zero);
-    // Static offsets were already delinearized into base indices. Materialize
-    // the non-zero starting indices before creating loop bodies.
-    if (!copyInfo->staticOffsetIndices.empty()) {
+    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
+    // Static offsets were already delinearized into base indices. Fill the
+    // non-zero starting indices before creating loop bodies.
+    if (copyInfo->staticOffsetIndices) {
       for (auto [idx, offset] :
-           llvm::enumerate(copyInfo->staticOffsetIndices)) {
+           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
       }
-    } else if (copyInfo->dynamicOffsetDim) {
+    } else {
       // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyToLoadStoreInfo.
+      // dimension selected by getCopyFromReinterCastInfo.
+      assert(copyInfo->dynamicOffsetDim &&
+             "expected dynamic offset dimension for dynamic offset");
       SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
+      // only a single offset. That should be fixed at the op definition level.
       assert(offsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
           getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
@@ -360,27 +383,30 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Scope for OpBuilder::InsertionGuard.
     {
       OpBuilder::InsertionGuard guard(rewriter);
+      Value lowerBound;
       Value step;
-      if (!upperBounds.empty())
+      if (!upperBounds.empty()) {
+        lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
+      }
 
       SmallVector<Value> loopIvs;
       loopIvs.reserve(copyInfo->loopDims.size());
 
-      // Build one nested loop per non-unit copied view dimension.
+      // Build one nested loop per non-unit copied strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, zero, upperBound, step);
+            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
         loopIvs.push_back(loop.getInductionVar());
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied view dimensions, which use the
-      // corresponding loop induction variables.
-      SmallVector<Value> loadIndices(srcType.getRank(), zero);
+      // Load indices are zero except for copied strided memref dimensions,
+      // which use the corresponding loop induction variables.
+      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.viewDim] = loopIvs[loopIndex++];
+        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
 
       // Store indices start from the offset-derived base indices. Add each loop
       // IV to the mapped base dimension.
@@ -388,11 +414,11 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.dstLoopDim] == zero) {
-          storeIndices[loopDim.dstLoopDim] = iv;
+        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
+          storeIndices[loopDim.baseDim] = iv;
         } else {
-          storeIndices[loopDim.dstLoopDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.dstLoopDim], iv);
+          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
+              rewriter, loc, storeIndices[loopDim.baseDim], iv);
         }
       }
 
@@ -612,7 +638,7 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyToLoadStoreInfo(op, rc);
+      return !getCopyFromReinterCastInfo(op, rc);
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 1cd6a1dc9756a..cd8f23ae4e045 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,10 +5,10 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @concat_zero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
@@ -23,13 +23,14 @@ func.func private @concat_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32> to memref<1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -45,14 +46,15 @@ func.func private @concat_nonzero_offset(%src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -69,35 +71,17 @@ func.func private @concat_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_strided(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @concat_strided(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
   return
 }
 
-// Dynamic strides are irrelevant because all view indices are zero.
-// CHECK-LABEL: func.func private @concat_dynamic_stride(
+// Dynamic strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @concat_dynamic_stride(%stride0: index,
+func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -113,50 +97,30 @@ func.func private @concat_dynamic_stride(%stride0: index,
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
-  return
-}
-
-// CHECK-LABEL: func.func private @concat_rank1(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @concat_rank1(%src : memref<1xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1], strides: [1]
-    : memref<108xf32> to memref<1xf32>
-
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]]] : memref<1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]]] : memref<108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1xf32> to memref<1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @concat_rank3(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x1x108xf32>
-func.func private @concat_rank3(%src : memref<1x1x1xf32>,
-  %dst : memref<1x1x108xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x1x108xf32> to memref<1x1x1xf32>
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
-  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x108xf32>
-  memref.copy %src, %reinterpret_cast
-    : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_0d(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_0d(
+func.func private @copy_to_strided_0d_2d_base(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -169,65 +133,68 @@ func.func private @concat_0d(
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1x1xf32> to memref<1x1x1xf32>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_zero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_zero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_nonzero_offset(
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_nonzero_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [41], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK-DAG:  %[[C41:.*]] = arith.constant 41 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C41]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: 41>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_same_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @concat_1d_vector_dynamic_offset_same_dim(
+func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -245,63 +212,66 @@ func.func private @concat_1d_vector_dynamic_offset_same_dim(
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_1d_vector_dynamic_offset_separate_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
 // CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_1d_vector_dynamic_offset_separate_dim(
-  %offset : index, %src : memref<1x33x1xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 1], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x33x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x33x42xf32>
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @concat_2d_vector_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x33x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @concat_2d_vector_offset(
-  %src : memref<1x33x4xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_to_strided_2d_vector_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [16], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C33:.*]] = arith.constant 33 : index
+  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C16:.*]] = arith.constant 16 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C33]] step %[[C1]] {
+  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C16]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x33x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x33x42xf32>
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: 16>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -309,8 +279,8 @@ func.func private @concat_2d_vector_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_concat_strided_base(
-func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
+func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -327,8 +297,8 @@ func.func private @negative_concat_strided_base(%src: memref<1x1xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_rank_change(
-func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
+func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -343,8 +313,8 @@ func.func private @negative_concat_rank_change(%src : memref<2x3xf32>,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_copy_source_shape(
-func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -359,8 +329,8 @@ func.func private @negative_concat_dynamic_copy_source_shape(%src : memref<?xf32
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_shapes(
-func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
+func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -375,8 +345,8 @@ func.func private @negative_concat_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_dynamic_offset_multi_dim_base(
-func.func private @negative_concat_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -392,41 +362,41 @@ func.func private @negative_concat_dynamic_offset_multi_dim_base(
   return
 }
 
-// CHECK-LABEL: func.func private @negative_concat_2d_dynamic_offset(
-func.func private @negative_concat_2d_dynamic_offset(
-  %offset : index, %src : memref<1x33x4xf32>,
-  %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x4xf32>,
+  %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 33, 4], strides: [1386, 42, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x4xf32>
-      to memref<1x33x4xf32, strided<[1386, 42, 1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_concat_dynamic_rc_stride(
-func.func private @negative_concat_dynamic_rc_stride(%stride : index,
-  %src : memref<1x33x1xf32>, %dst : memref<1x33x42xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
+func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 33, 1], strides: [1386, %stride, 1]
-    : memref<1x33x42xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %reinterpret_cast
-    : memref<1x33x1xf32>
-      to memref<1x33x1xf32, strided<[1386, ?, 1]>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return
 }
 

>From 6d8331af83799c082f0d3546e1c5c596aae0ea77 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 10:45:19 +0200
Subject: [PATCH 03/17] Refine helper function comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 108 ++++++++++--------
 1 file changed, 60 insertions(+), 48 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index b62ac4703b80f..af8f8e1e6b5e3 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -30,10 +30,11 @@ using namespace mlir;
 
 namespace {
 
-/// Copy rewrite helpers
+//===----------------------------------------------------------------------===//
+// Copy Rewrite Helpers
+//===----------------------------------------------------------------------===//
 
-/// The copy rewrite stores directly into the reinterpret_cast source, so it
-/// needs concrete base strides instead of layout-map reasoning.
+/// Returns row-major strides for static identity-layout memref type.
 static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   if (!type.getLayout().isIdentity() || !type.hasStaticShape())
     return std::nullopt;
@@ -47,42 +48,54 @@ static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
   return strides;
 }
 
-/// Non-unit strided memref dimensions are lowered to loops over base memref
-/// dimensions. Static reinterpret_cast strides connect those dimension spaces
-/// in the currently supported cases.
-static std::optional<unsigned>
-findBaseDimForResultStride(MemRefType baseType, ArrayRef<int64_t> baseStrides,
-                           ArrayRef<bool> usedBaseDims, int64_t resultStride,
-                           int64_t resultSize) {
-  std::optional<unsigned> fallback;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    if (usedBaseDims[idx] || stride != resultStride ||
-        baseType.getDimSize(idx) < resultSize)
+/// Finds the source dimension for a static reinterpret_cast result dimension.
+/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
+/// source dimension whose size is at least the result dimension size, with the
+/// same stride.
+static std::optional<unsigned> findSourceDimForResultDim(
+    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
+    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
+  MemRefType resultType = cast<MemRefType>(rc.getType());
+  assert(resultDim < resultType.getRank() && "result dimension out of range");
+  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
+         sourceStrides.size() == usedSourceDims.size() &&
+         "expected same-rank source type, strides, and used-dimension mask");
+  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
+         "expected static result stride");
+
+  int64_t resultStride = rc.getStaticStrides()[resultDim];
+  int64_t resultSize = resultType.getDimSize(resultDim);
+  std::optional<unsigned> sourceDim;
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+    if (usedSourceDims[idx] || stride != resultStride ||
+        sourceType.getDimSize(idx) < resultSize)
       continue;
 
-    // Prefer an exact shape match. Otherwise, use the first dimension large
-    // enough to contain the copied logical vector.
-    if (baseType.getDimSize(idx) == resultSize)
-      return idx;
-    if (!fallback)
-      fallback = idx;
+    if (!sourceDim ||
+        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
+      sourceDim = idx;
   }
-  return fallback;
+  return sourceDim;
 }
 
-/// reinterpret_cast offsets are linear element offsets, while `memref.store`
-/// needs one index per base dimension.
+/// Returns source indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(int64_t offset, MemRefType baseType,
-                        ArrayRef<int64_t> baseStrides) {
-  if (offset < 0)
+delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
+                        ArrayRef<int64_t> sourceStrides) {
+  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(offsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+
+  if (offsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
-  int64_t remainder = offset;
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
+  SmallVector<int64_t> indices(sourceType.getRank(), 0);
+  int64_t remainder = offsets[0];
+  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
+    if (indices[idx] >= sourceType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -92,14 +105,15 @@ delinearizeStaticOffset(int64_t offset, MemRefType baseType,
   return indices;
 }
 
-/// Dynamic scalar offsets cannot be delinearized statically. They can be used
-/// directly only when the base has a single non-unit dimension to receive them.
+/// Returns the dimension whose static size is not one if it is unique.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
-  if (!type.hasStaticShape() || type.getRank() == 0)
+  assert(type.hasStaticShape() && "expected static shape");
+  ArrayRef<int64_t> shape = type.getShape();
+  if (shape.empty())
     return std::nullopt;
 
   std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(type.getShape())) {
+  for (auto [idx, dim] : llvm::enumerate(shape)) {
     if (dim == 1)
       continue;
     if (nonUnitDim)
@@ -186,12 +200,10 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
         resultType.hasStaticShape()))
     return std::nullopt;
 
-  // Map reinterpret_cast strided memref dimensions to source dimensions by
-  // stride.
   std::optional<SmallVector<int64_t>> baseStrides =
       getIdentityStrides(baseType);
-  // TODO: Support non-identity reinterpret_cast source layouts by using the
-  // source layout strides as base strides.
+  // TODO: Support non-identity source layouts by computing source strides from
+  // the layout map.
   if (!baseStrides)
     return std::nullopt;
 
@@ -206,11 +218,11 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return std::nullopt;
 
-    // Non-unit strided memref dimensions become loop dimensions in the scalar
-    // rewrite.
-    std::optional<unsigned> baseDim = findBaseDimForResultStride(
-        baseType, *baseStrides, usedBaseDims, rc.getStaticStrides()[resultDim],
-        resultSize);
+    // Each copied result dimension must map by stride to a source dimension
+    // whose static size >= result dimension size.
+    std::optional<unsigned> baseDim =
+        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
+                                  baseType, *baseStrides, usedBaseDims);
     assert(baseDim &&
            "static reinterpret_cast stride must map to an identity base "
            "dimension");
@@ -225,9 +237,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
   // only a single offset. That should be fixed at the op definition level.
   assert(offsets.size() == 1 && "Expecting single offset");
   if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offsets are converted to base indices.
+    // Static offset must delinearize to in-bounds source indices.
     std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(offsets[0], baseType, *baseStrides);
+        delinearizeStaticOffset(rc, baseType, *baseStrides);
     assert(offsetIndices &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
@@ -250,9 +262,9 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return std::nullopt;
 
   if (info.loopDims.empty()) {
-    // Scalar-shaped strided memrefs copy one element, so a dynamic linear
-    // offset can be used directly only when the base has exactly one non-unit
-    // dimension.
+    // Dynamic scalar offsets cannot be delinearized statically. They can be
+    // used directly only when the base has a single non-unit dimension to
+    // receive them.
     std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at

>From 928c1903f52498ef60af7f2e4b4dcb29750b5477 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 19 Jun 2026 13:05:33 +0200
Subject: [PATCH 04/17] Remove redundant dialect dependency

---
 mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 7b54be85db340..c6be600247696 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -20,7 +20,7 @@ def ElideReinterpretCastPass : Pass<"memref-elide-reinterpret-cast"> {
     compatible shapes directly. This simplifies conversion to EmitC.
 }];
   let dependentDialects = [
-      "arith::ArithDialect", "memref::MemRefDialect", "scf::SCFDialect"
+      "arith::ArithDialect", "scf::SCFDialect"
   ];
 }
 

>From ebaa70e636d8edbc3807d5ef69c3c6826c240e3b Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 18:01:38 +0200
Subject: [PATCH 05/17] Address second round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 243 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  16 ++
 2 files changed, 133 insertions(+), 126 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index af8f8e1e6b5e3..f1246da1daeba 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/MemRef/Transforms/Transforms.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Transforms/DialectConversion.h"
@@ -34,68 +35,87 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Returns row-major strides for static identity-layout memref type.
-static std::optional<SmallVector<int64_t>> getIdentityStrides(MemRefType type) {
-  if (!type.getLayout().isIdentity() || !type.hasStaticShape())
-    return std::nullopt;
+/// Per-dimension loop nest info.
+struct CopyLoopDimInfo {
+  unsigned copyDim;
+  unsigned baseDim;
+  int64_t size;
+};
 
-  SmallVector<int64_t> strides(type.getRank(), 1);
-  int64_t stride = 1;
-  for (int64_t dim = type.getRank() - 1; dim >= 0; --dim) {
-    strides[dim] = stride;
-    stride *= type.getDimSize(dim);
-  }
-  return strides;
-}
+/// Rewrite info from reinterpret_cast layout, captured after passing legality
+/// checks.
+struct CopyFromReinterCastInfo {
+  // Loop bounds that non-scalar loads "lower" to
+  SmallVector<CopyLoopDimInfo> loopDims;
+  // Deinearized offsets to in-bounds base indices.
+  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
+  // reinterpret_cast dynamic offsets only supported for single non-unit
+  // dimension base, stored here to receive them.
+  std::optional<unsigned> dynamicOffsetDim;
+};
+
+/// Maps non-unit reinterpret_cast result dimensions to distinct base
+/// dimensions.
+static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
+                                    CopyFromReinterCastInfo &info) {
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> baseIdentityStrides =
+      computeStrides(baseType.getShape());
 
-/// Finds the source dimension for a static reinterpret_cast result dimension.
-/// Dimensions marked in `usedSourceDims` are skipped. Returns the smallest
-/// source dimension whose size is at least the result dimension size, with the
-/// same stride.
-static std::optional<unsigned> findSourceDimForResultDim(
-    memref::ReinterpretCastOp rc, unsigned resultDim, MemRefType sourceType,
-    ArrayRef<int64_t> sourceStrides, ArrayRef<bool> usedSourceDims) {
-  MemRefType resultType = cast<MemRefType>(rc.getType());
-  assert(resultDim < resultType.getRank() && "result dimension out of range");
-  assert(sourceType.getRank() == static_cast<int64_t>(sourceStrides.size()) &&
-         sourceStrides.size() == usedSourceDims.size() &&
-         "expected same-rank source type, strides, and used-dimension mask");
-  assert(!ShapedType::isDynamic(rc.getStaticStrides()[resultDim]) &&
-         "expected static result stride");
-
-  int64_t resultStride = rc.getStaticStrides()[resultDim];
-  int64_t resultSize = resultType.getDimSize(resultDim);
-  std::optional<unsigned> sourceDim;
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
-    if (usedSourceDims[idx] || stride != resultStride ||
-        sourceType.getDimSize(idx) < resultSize)
+  // Each result loop IV is added directly to one base index. Reusing a base
+  // dimension would require delinearizing the combined linear offset.
+  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+
+  // Populate one loop-dimension entry for each non-unit result dimension.
+  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize == 1)
       continue;
 
-    if (!sourceDim ||
-        sourceType.getDimSize(idx) < sourceType.getDimSize(*sourceDim))
-      sourceDim = idx;
+    // TODO: Support dynamic strides on copied dimensions.
+    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+      return false;
+
+    int64_t resultStride = rc.getStaticStrides()[resultDim];
+    std::optional<unsigned> baseDim;
+    // Find an unused base dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
+      if (usedBaseDims[idx] || stride != resultStride ||
+          baseType.getDimSize(idx) < resultSize)
+        continue;
+
+      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
+        baseDim = idx;
+    }
+    if (!baseDim)
+      return false;
+
+    usedBaseDims[*baseDim] = true;
+    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
+                                            *baseDim, resultSize});
   }
-  return sourceDim;
+  return true;
 }
 
-/// Returns source indices for a static reinterpret_cast offset.
+/// Returns base indices for a static reinterpret_cast offset.
 static std::optional<SmallVector<int64_t>>
-delinearizeStaticOffset(memref::ReinterpretCastOp rc, MemRefType sourceType,
-                        ArrayRef<int64_t> sourceStrides) {
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(offsets[0]) && "expected static offset");
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
 
-  if (offsets[0] < 0)
+  if (rcOffsets[0] < 0)
     return std::nullopt;
 
-  SmallVector<int64_t> indices(sourceType.getRank(), 0);
-  int64_t remainder = offsets[0];
-  for (auto [idx, stride] : llvm::enumerate(sourceStrides)) {
+  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  int64_t remainder = rcOffsets[0];
+  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
+  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
     indices[idx] = remainder / stride;
-    if (indices[idx] >= sourceType.getDimSize(idx))
+    if (indices[idx] >= baseType.getDimSize(idx))
       return std::nullopt;
     remainder %= stride;
   }
@@ -123,21 +143,6 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return nonUnitDim;
 }
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
-};
-
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  SmallVector<CopyLoopDimInfo> loopDims;
-  std::optional<SmallVector<int64_t>> staticOffsetIndices;
-  std::optional<unsigned> dynamicOffsetDim;
-};
-
 /// Builds the index mapping needed to replace a copy into a reinterpret_cast
 /// strided memref with scalar stores into the reinterpret_cast base.
 ///
@@ -174,86 +179,70 @@ struct CopyFromReinterCastInfo {
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
 static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
-  MemRefType srcType = dyn_cast<MemRefType>(op.getSource().getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType resultType = dyn_cast<MemRefType>(rc.getType());
+getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
+  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
+  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!srcType || !baseType || !resultType)
+  if (!cpSrcType || !rcBaseType || !rcResType)
     return std::nullopt;
 
-  if (srcType.getShape() != resultType.getShape())
+  if (cpSrcType.getShape() != rcResType.getShape())
     return std::nullopt;
 
-  // TODO: Support rank-changing reinterpret_casts by converting the
+  // TODO: Support rank-modifying reinterpret_casts by converting the
   // strided memref indices to base indices. For example, a copy to
   // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
   // strided memref indices as `i * 3 + j`, then combine that with the
   // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (baseType.getRank() != resultType.getRank())
+  if (rcBaseType.getRank() != rcResType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(srcType.hasStaticShape() && baseType.hasStaticShape() &&
-        resultType.hasStaticShape()))
+  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
+        rcResType.hasStaticShape()))
     return std::nullopt;
 
-  std::optional<SmallVector<int64_t>> baseStrides =
-      getIdentityStrides(baseType);
-  // TODO: Support non-identity source layouts by computing source strides from
+  // TODO: Support non-identity base layouts by computing base strides from
   // the layout map.
-  if (!baseStrides)
+  if (!rcBaseType.getLayout().isIdentity())
     return std::nullopt;
 
   CopyFromReinterCastInfo info;
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
 
-  for (auto [resultDim, resultSize] : llvm::enumerate(resultType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    // TODO: Support dynamic strides on copied dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
-      return std::nullopt;
-
-    // Each copied result dimension must map by stride to a source dimension
-    // whose static size >= result dimension size.
-    std::optional<unsigned> baseDim =
-        findSourceDimForResultDim(rc, static_cast<unsigned>(resultDim),
-                                  baseType, *baseStrides, usedBaseDims);
-    assert(baseDim &&
-           "static reinterpret_cast stride must map to an identity base "
-           "dimension");
-
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
-  }
+  // reinterpret_cast result dimensions must map to distinct base dimensions.
+  // The rewrite emits one loop per copied dimension and adds each IV to one
+  // base index.
+  if (!findBaseDimForResultDim(rc, info))
+    return std::nullopt;
 
-  ArrayRef<int64_t> offsets = rc.getStaticOffsets();
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
-  assert(offsets.size() == 1 && "Expecting single offset");
-  if (!ShapedType::isDynamic(offsets[0])) {
-    // Static offset must delinearize to in-bounds source indices.
-    std::optional<SmallVector<int64_t>> offsetIndices =
-        delinearizeStaticOffset(rc, baseType, *baseStrides);
-    assert(offsetIndices &&
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+  // CASE 1: Static ReinterpretCast offset
+  if (!ShapedType::isDynamic(rcOffsets[0])) {
+    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+    // every base dimension).
+    std::optional<SmallVector<int64_t>> offsetIdxs =
+        delinearizeStaticRCOffset(rc);
+    assert(offsetIdxs &&
            "static reinterpret_cast offset must delinearize to in-bounds base "
            "indices");
 
-    for (const CopyLoopDimInfo &loopDim : info.loopDims) {
-      assert((*offsetIndices)[loopDim.baseDim] + loopDim.size <=
-                 baseType.getDimSize(loopDim.baseDim) &&
-             "reinterpret_cast metadata describes an invalid accessible "
-             "region");
-    }
-    info.staticOffsetIndices = std::move(offsetIndices);
+    assert(llvm::all_of(info.loopDims,
+                        [&](const CopyLoopDimInfo &loopDim) {
+                          return (*offsetIdxs)[loopDim.baseDim] +
+                                     loopDim.size <=
+                                 rcBaseType.getDimSize(loopDim.baseDim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    info.staticOffsetIdxs = std::move(offsetIdxs);
     return info;
   }
-
+  // CASE 2: Dynamic ReinterpretCast offset
   // Dynamic offsets are kept only when they can be used as a single base index.
   // TODO: Support dynamic offsets for copies with multiple loop dimensions by
   // delinearizing the offset into base start indices at runtime before adding
@@ -265,7 +254,7 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     // Dynamic scalar offsets cannot be delinearized statically. They can be
     // used directly only when the base has a single non-unit dimension to
     // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(baseType);
+    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
     // TODO: Support scalar dynamic offsets into bases with multiple non-unit
     // dimensions by delinearizing the single accessed element offset at
     // runtime.
@@ -276,9 +265,12 @@ getCopyFromReinterCastInfo(memref::CopyOp op, memref::ReinterpretCastOp rc) {
     return info;
   }
 
-  unsigned baseDim = info.loopDims.front().baseDim;
-  info.dynamicOffsetDim =
-      (*baseStrides)[baseDim] == 1 ? baseDim : baseStrides->size() - 1;
+  unsigned rcBaseDim = info.loopDims.front().baseDim;
+  SmallVector<int64_t> rcBaseIdentityStrides =
+      computeStrides(rcBaseType.getShape());
+  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
+                              ? rcBaseDim
+                              : rcBaseIdentityStrides.size() - 1;
   return info;
 }
 
@@ -339,7 +331,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     Value src = op.getSource();
     Value dst = rc.getSource();
 
-    MemRefType srcType = cast<MemRefType>(src.getType());
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -372,9 +364,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
     // Static offsets were already delinearized into base indices. Fill the
     // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIndices) {
-      for (auto [idx, offset] :
-           llvm::enumerate(*copyInfo->staticOffsetIndices)) {
+    if (copyInfo->staticOffsetIdxs) {
+      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
         if (offset == 0)
           continue;
         baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
@@ -384,12 +375,12 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // dimension selected by getCopyFromReinterCastInfo.
       assert(copyInfo->dynamicOffsetDim &&
              "expected dynamic offset dimension for dynamic offset");
-      SmallVector<OpFoldResult> offsets = rc.getMixedOffsets();
+      SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
-      assert(offsets.size() == 1 && "Expecting single offset");
+      assert(rcOffsets.size() == 1 && "Expecting single offset");
       baseStoreIndices[*copyInfo->dynamicOffsetDim] =
-          getValueOrCreateConstantIndexOp(rewriter, loc, offsets[0]);
+          getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
     // Scope for OpBuilder::InsertionGuard.
@@ -415,7 +406,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
       // Load indices are zero except for copied strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(srcType.getRank());
+      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
       unsigned loopIndex = 0;
       for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
         loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index cd8f23ae4e045..93d5770615b3e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -297,6 +297,22 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
+func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+  %dst : memref<4x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 3], strides: [1, 1]
+    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
 func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {

>From 7ab3ba7f1c97c9958866c057afbf1a50a0b88448 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 25 Jun 2026 09:41:58 +0200
Subject: [PATCH 06/17] Address third round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 378 +++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        |  76 ++--
 2 files changed, 230 insertions(+), 224 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index f1246da1daeba..a0e9810249f60 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,141 +35,154 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Per-dimension loop nest info.
-struct CopyLoopDimInfo {
-  unsigned copyDim;
-  unsigned baseDim;
-  int64_t size;
+/// Non-unit reinterpret_cast result dimension and the source dimension it
+/// advances through.
+struct NonUnitDimAssocMapForRC {
+  unsigned resultDimPos;
+  unsigned sourceDimPos;
 };
 
-/// Rewrite info from reinterpret_cast layout, captured after passing legality
-/// checks.
-struct CopyFromReinterCastInfo {
-  // Loop bounds that non-scalar loads "lower" to
-  SmallVector<CopyLoopDimInfo> loopDims;
-  // Deinearized offsets to in-bounds base indices.
-  std::optional<SmallVector<int64_t>> staticOffsetIdxs;
-  // reinterpret_cast dynamic offsets only supported for single non-unit
-  // dimension base, stored here to receive them.
-  std::optional<unsigned> dynamicOffsetDim;
+/// Copy-relevant information derived from a reinterpret_cast.
+struct AssocMapAndOffsetsForRC {
+  // Non-unit dimensions of the reinterpret_cast result.
+  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  // Delinearized offsets to in-bounds reinterpret_cast source indices.
+  // Optional since it is only supported for static offsets.
+  std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps non-unit reinterpret_cast result dimensions to distinct base
-/// dimensions.
-static bool findBaseDimForResultDim(memref::ReinterpretCastOp rc,
-                                    CopyFromReinterCastInfo &info) {
+/// Records the reinterpret_cast result dimensions that span more than one
+/// element and maps each one to its corresponding source dimension.
+static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
+                                      AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> baseIdentityStrides =
-      computeStrides(baseType.getShape());
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
 
-  // Each result loop IV is added directly to one base index. Reusing a base
-  // dimension would require delinearizing the combined linear offset.
-  SmallVector<bool> usedBaseDims(baseType.getRank(), false);
+  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+
+  // Reusing a source dimension would require delinearizing the combined linear
+  // offset, which is TODO.
+  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
-  // Populate one loop-dimension entry for each non-unit result dimension.
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on copied dimensions.
+    // TODO: Support dynamic strides on non-unit result dimensions.
     if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
       return false;
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> baseDim;
-    // Find an unused base dimension with matching stride and enough elements.
-    for (auto [idx, stride] : llvm::enumerate(baseIdentityStrides)) {
-      if (usedBaseDims[idx] || stride != resultStride ||
-          baseType.getDimSize(idx) < resultSize)
+    std::optional<unsigned> srcDim;
+    // Find an unused source dimension with matching stride and enough elements.
+    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
+      if (usedSrcDims[idx] || stride != resultStride ||
+          srcType.getDimSize(idx) < resultSize)
         continue;
 
-      if (!baseDim || baseType.getDimSize(idx) < baseType.getDimSize(*baseDim))
-        baseDim = idx;
+      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
+        srcDim = idx;
     }
-    if (!baseDim)
+    if (!srcDim)
       return false;
 
-    usedBaseDims[*baseDim] = true;
-    info.loopDims.push_back(CopyLoopDimInfo{static_cast<unsigned>(resultDim),
-                                            *baseDim, resultSize});
+    usedSrcDims[*srcDim] = true;
+    mapAndOffs.assocMap.push_back(
+        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
   }
   return true;
 }
 
-/// Returns base indices for a static reinterpret_cast offset.
+/// Returns source indices for a static reinterpret_cast offset of an
+/// identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(!ShapedType::isDynamic(rcOffsets[0]) && "expected static offset");
-
-  if (rcOffsets[0] < 0)
-    return std::nullopt;
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
+
+  assert(rcOffsets[0] >= 0 &&
+         "static reinterpret_cast offset must be non-negative");
+
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  assert(srcType.getLayout().isIdentity() &&
+         "Expecting identity source layout.");
+  if (srcType.getRank() == 0) {
+    assert(rcOffsets[0] == 0 &&
+           "non-zero static offset is invalid for rank-0 source memref");
+    return SmallVector<int64_t>{};
+  }
 
-  MemRefType baseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  SmallVector<int64_t> indices(baseType.getRank(), 0);
+  SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
-  SmallVector<int64_t> baseStrides = computeStrides(baseType.getShape());
-  for (auto [idx, stride] : llvm::enumerate(baseStrides)) {
-    indices[idx] = remainder / stride;
-    if (indices[idx] >= baseType.getDimSize(idx))
-      return std::nullopt;
+  SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
+  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // indices.
+  for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
+    offsetIdxs[dim] = remainder / stride;
+    assert(offsetIdxs[dim] < srcType.getDimSize(dim) &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "source indices");
     remainder %= stride;
   }
 
-  if (remainder != 0)
-    return std::nullopt;
-  return indices;
+  assert(remainder == 0 &&
+         "Assuming identity source layout, the trailing stride == 1 "
+         "so, the remainder should be 0 at the end of index calculation.");
+  return offsetIdxs;
 }
 
-/// Returns the dimension whose static size is not one if it is unique.
+/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
-  if (shape.empty())
+
+  // Find all non-unit dims
+  auto nonUnitDims = llvm::make_filter_range(
+      llvm::enumerate(shape), [](auto it) { return it.value() != 1; });
+
+  // Expect single non-unit dims
+  if (llvm::range_size(nonUnitDims) != 1)
     return std::nullopt;
 
-  std::optional<unsigned> nonUnitDim;
-  for (auto [idx, dim] : llvm::enumerate(shape)) {
-    if (dim == 1)
-      continue;
-    if (nonUnitDim)
-      return std::nullopt;
-    nonUnitDim = idx;
-  }
-  return nonUnitDim;
+  // Return the index of the unique non-unit dim.
+  return (*nonUnitDims.begin()).index();
 }
 
-/// Builds the index mapping needed to replace a copy into a reinterpret_cast
-/// strided memref with scalar stores into the reinterpret_cast base.
+/// Returns the copy-relevant reinterpret_cast information: non-unit result
+/// dimensions, their source-dimension mapping, and optional source starting
+/// indices for a static offset.
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy. There are no copied non-unit dimensions, so dynamic
-///   // strides in the strided memref do not affect index mapping.
+///   // Scalar-shaped copy into a source with at most one non-unit dimension.
+///   There are
+///   // no non-unit result dimensions, so dynamic strides in the strided memref
+///   // do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout base dimension by its static stride.
+///   // mapped to an identity-layout source dimension by its static stride.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
 ///   // dimension is mapped independently by its static stride.
 ///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<base-shape>
+///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a copied strided memref dimension.
+///   // Dynamic stride on a non-unit strided memref dimension.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -178,100 +191,73 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///   copy memref<1xNxKxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<CopyFromReinterCastInfo>
-getCopyFromReinterCastInfo(memref::CopyOp copy, memref::ReinterpretCastOp rc) {
-  MemRefType cpSrcType = dyn_cast<MemRefType>(copy.getSource().getType());
-  MemRefType rcBaseType = dyn_cast<MemRefType>(rc.getSource().getType());
-  MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+static std::optional<AssocMapAndOffsetsForRC>
+getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
   // Ranked memref types are required to statically build load/store index
   // lists.
-  if (!cpSrcType || !rcBaseType || !rcResType)
-    return std::nullopt;
-
-  if (cpSrcType.getShape() != rcResType.getShape())
+  if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts by converting the
-  // strided memref indices to base indices. For example, a copy to
-  // a strided memref<2x3xf32> of base memref<6xf32> needs to linearize the
-  // strided memref indices as `i * 3 + j`, then combine that with the
-  // reinterpret_cast offset before indexing the rank-1 base memref.
-  if (rcBaseType.getRank() != rcResType.getRank())
+  // TODO: Support rank-modifying reinterpret_casts
+  if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
   // TODO: Support dynamic shapes with mixed size operands as loop bounds.
-  if (!(cpSrcType.hasStaticShape() && rcBaseType.hasStaticShape() &&
-        rcResType.hasStaticShape()))
+  if (!(srcType.hasStaticShape() && resType.hasStaticShape()))
     return std::nullopt;
 
-  // TODO: Support non-identity base layouts by computing base strides from
+  // TODO: Support non-identity source layouts by computing source strides from
   // the layout map.
-  if (!rcBaseType.getLayout().isIdentity())
+  if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  CopyFromReinterCastInfo info;
+  AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct base dimensions.
-  // The rewrite emits one loop per copied dimension and adds each IV to one
-  // base index.
-  if (!findBaseDimForResultDim(rc, info))
+  // reinterpret_cast result dimensions must map to distinct source dimensions.
+  if (!findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // CASE 1: Static ReinterpretCast offset
-  if (!ShapedType::isDynamic(rcOffsets[0])) {
+  // Static ReinterpretCast offset
+  if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every base dimension).
-    std::optional<SmallVector<int64_t>> offsetIdxs =
-        delinearizeStaticRCOffset(rc);
-    assert(offsetIdxs &&
-           "static reinterpret_cast offset must delinearize to in-bounds base "
-           "indices");
-
-    assert(llvm::all_of(info.loopDims,
-                        [&](const CopyLoopDimInfo &loopDim) {
-                          return (*offsetIdxs)[loopDim.baseDim] +
-                                     loopDim.size <=
-                                 rcBaseType.getDimSize(loopDim.baseDim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
-    info.staticOffsetIdxs = std::move(offsetIdxs);
-    return info;
+    // every source dimension).
+    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(
+        mapAndOffs.delinearizedOffsets &&
+        "static reinterpret_cast offset must delinearize to in-bounds source "
+        "indices");
+
+    assert(
+        llvm::all_of(
+            mapAndOffs.assocMap,
+            [&](const NonUnitDimAssocMapForRC &assocMap) {
+              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
+                         resType.getDimSize(assocMap.resultDimPos) <=
+                     srcType.getDimSize(assocMap.sourceDimPos);
+            }) &&
+        "reinterpret_cast metadata describes an invalid accessible region");
+    return mapAndOffs;
   }
-  // CASE 2: Dynamic ReinterpretCast offset
-  // Dynamic offsets are kept only when they can be used as a single base index.
-  // TODO: Support dynamic offsets for copies with multiple loop dimensions by
-  // delinearizing the offset into base start indices at runtime before adding
-  // loop IVs.
-  if (info.loopDims.size() > 1)
-    return std::nullopt;
-
-  if (info.loopDims.empty()) {
-    // Dynamic scalar offsets cannot be delinearized statically. They can be
-    // used directly only when the base has a single non-unit dimension to
-    // receive them.
-    std::optional<unsigned> nonUnitDim = getSingleNonUnitDim(rcBaseType);
-    // TODO: Support scalar dynamic offsets into bases with multiple non-unit
-    // dimensions by delinearizing the single accessed element offset at
-    // runtime.
-    if (!nonUnitDim)
-      return std::nullopt;
 
-    info.dynamicOffsetDim = *nonUnitDim;
-    return info;
-  }
+  // Dynamic ReinterpretCast offset.
+  // TODO: Support dynamic offsets into sources with multiple non-unit
+  // dimensions by delinearizing the offset into source start indices at runtime
+  // before adding loop IVs.
+  if (mapAndOffs.assocMap.size() > 1)
+    return std::nullopt;
+  // Only sources with a single non-unit dimension can receive a dynamic offset
+  // directly.
+  if (!getSingleNonUnitDim(srcType))
+    return std::nullopt;
 
-  unsigned rcBaseDim = info.loopDims.front().baseDim;
-  SmallVector<int64_t> rcBaseIdentityStrides =
-      computeStrides(rcBaseType.getShape());
-  info.dynamicOffsetDim = rcBaseIdentityStrides[rcBaseDim] == 1
-                              ? rcBaseDim
-                              : rcBaseIdentityStrides.size() - 1;
-  return info;
+  return mapAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -315,23 +301,25 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 
   LogicalResult matchAndRewrite(memref::CopyOp op,
                                 PatternRewriter &rewriter) const final {
+    Value src = op.getSource();
+    MemRefType cpSrcType = cast<MemRefType>(src.getType());
+    if (!cpSrcType || !cpSrcType.hasStaticShape())
+      return rewriter.notifyMatchFailure(
+          op, "only ranked, static copy sources are supported.");
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<CopyFromReinterCastInfo> copyInfo =
-        getCopyFromReinterCastInfo(op, rc);
-    if (!copyInfo)
+    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
+        getAssocMapAndOffsetsForRC(rc);
+    if (!mapAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
-    Value src = op.getSource();
     Value dst = rc.getSource();
-
-    MemRefType cpSrcType = cast<MemRefType>(src.getType());
     MemRefType dstType = cast<MemRefType>(dst.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
@@ -346,40 +334,48 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
-    auto getZeroIndices = [&](int64_t rank) {
-      SmallVector<Value> indices;
-      indices.reserve(rank);
+    auto getZeroIdxs = [&](int64_t rank) {
+      SmallVector<Value> idxs;
+      idxs.reserve(rank);
       if (rank != 0)
-        indices.append(rank, getOrCreateIndexConstant(0));
-      return indices;
+        idxs.append(rank, getOrCreateIndexConstant(0));
+      return idxs;
     };
 
-    // Create all loop bounds before building the loop nest. Otherwise an
-    // inner-loop bound can be inserted inside an outer loop body.
+    // Create loop bounds before moving the insertion point into the loop nest,
+    // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(copyInfo->loopDims.size());
-    for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-      upperBounds.push_back(getOrCreateIndexConstant(loopDim.size));
-
-    SmallVector<Value> baseStoreIndices = getZeroIndices(dstType.getRank());
-    // Static offsets were already delinearized into base indices. Fill the
-    // non-zero starting indices before creating loop bodies.
-    if (copyInfo->staticOffsetIdxs) {
-      for (auto [idx, offset] : llvm::enumerate(*copyInfo->staticOffsetIdxs)) {
+    upperBounds.reserve(mapAndOffs->assocMap.size());
+    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
+    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+      upperBounds.push_back(getOrCreateIndexConstant(
+          rcResType.getDimSize(assocMap.resultDimPos)));
+
+    SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
+    std::optional<unsigned> srcNonUnitDimPos;
+    // Static offset has been delinearized in function gating rewrite.
+    if (mapAndOffs->delinearizedOffsets) {
+      for (auto [idx, offset] :
+           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
-        baseStoreIndices[idx] = getOrCreateIndexConstant(offset);
+        rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Supported dynamic offsets are used directly in exactly one base
-      // dimension selected by getCopyFromReinterCastInfo.
-      assert(copyInfo->dynamicOffsetDim &&
-             "expected dynamic offset dimension for dynamic offset");
+      // Without runtime delinearization, use the dynamic offset directly only
+      // when the source has a single non-unit dimension.
+      assert(mapAndOffs->assocMap.size() <= 1 &&
+             "Expecting single non-unit dimension mapping.");
+      srcNonUnitDimPos = getSingleNonUnitDim(dstType);
+      assert(srcNonUnitDimPos &&
+             "Expecting single non-unit dimension source to receive the "
+             "dynamic offset.");
+
       SmallVector<OpFoldResult> rcOffsets = rc.getMixedOffsets();
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
-      baseStoreIndices[*copyInfo->dynamicOffsetDim] =
+      rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
@@ -392,11 +388,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         lowerBound = getOrCreateIndexConstant(0);
         step = getOrCreateIndexConstant(1);
       }
-
       SmallVector<Value> loopIvs;
-      loopIvs.reserve(copyInfo->loopDims.size());
+      loopIvs.reserve(mapAndOffs->assocMap.size());
 
-      // Build one nested loop per non-unit copied strided memref dimension.
+      // Build one nested loop per non-unit strided memref dimension.
       for (Value upperBound : upperBounds) {
         scf::ForOp loop =
             scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
@@ -404,31 +399,32 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rewriter.setInsertionPointToStart(loop.getBody());
       }
 
-      // Load indices are zero except for copied strided memref dimensions,
+      // Load indices are zero except for non-unit strided memref dimensions,
       // which use the corresponding loop induction variables.
-      SmallVector<Value> loadIndices = getZeroIndices(cpSrcType.getRank());
+      SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
       unsigned loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims)
-        loadIndices[loopDim.copyDim] = loopIvs[loopIndex++];
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
+        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
 
-      // Store indices start from the offset-derived base indices. Add each loop
-      // IV to the mapped base dimension.
-      SmallVector<Value> storeIndices(baseStoreIndices);
+      // Store indices start from the offset-derived source indices. Add each
+      // loop IV to the mapped source dimension.
+      SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
       loopIndex = 0;
-      for (const CopyLoopDimInfo &loopDim : copyInfo->loopDims) {
+      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
         Value iv = loopIvs[loopIndex++];
-        if (storeIndices[loopDim.baseDim] == getOrCreateIndexConstant(0)) {
-          storeIndices[loopDim.baseDim] = iv;
+        // Add each IV to one source index.
+        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
+          storeIdxs[assocMap.sourceDimPos] = iv;
         } else {
-          storeIndices[loopDim.baseDim] = arith::AddIOp::create(
-              rewriter, loc, storeIndices[loopDim.baseDim], iv);
+          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
+              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
         }
       }
 
       // Emit the scalar load/store at the innermost loop body, or directly at
       // the original copy location for scalar copies.
-      Value val = memref::LoadOp::create(rewriter, loc, src, loadIndices);
-      memref::StoreOp::create(rewriter, loc, val, dst, storeIndices);
+      Value val = memref::LoadOp::create(rewriter, loc, src, loadIdxs);
+      memref::StoreOp::create(rewriter, loc, val, dst, storeIdxs);
     }
 
     // If the only user of `rc` is the current Op (which is about to be erased),
@@ -641,7 +637,11 @@ struct ElideReinterpretCastPass
       auto rc = op.getTarget().getDefiningOp<memref::ReinterpretCastOp>();
       if (!rc)
         return true;
-      return !getCopyFromReinterCastInfo(op, rc);
+      // Pattern applies only when the copy source shape is static and the
+      // reinterpret_cast result can be mapped back to base memref indices.
+      MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
+      return !(cpSrcType && cpSrcType.hasStaticShape() &&
+               getAssocMapAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 93d5770615b3e..dace611bffb7f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -92,7 +92,6 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
     : memref<1x1xf32>
@@ -117,10 +116,10 @@ func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>)
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_2d_base(
+// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_2d_base(
+func.func private @copy_to_strided_0d_base_2d(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -190,11 +189,11 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
+func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -216,34 +215,6 @@ func.func private @copy_to_strided_1d_vector_dynamic_offset_in_loop_dim(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_dynamic_offset_not_in_loop_dim(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
-  // CHECK:      }
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -361,8 +332,24 @@ func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
-func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
+// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
+func.func private @negative_copy_to_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<4xf32> to memref<4xf32>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -378,6 +365,25 @@ func.func private @negative_copy_to_strided_dynamic_offset_multi_dim_base(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>,
+  %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %reinterpret_cast
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 // CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
 func.func private @negative_copy_to_strided_2d_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,

>From d4e6a5174202b860ade26bedca8bd13fccb3871d Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 26 Jun 2026 17:50:06 +0200
Subject: [PATCH 07/17] Fixup tests

---
 .../MemRef/elide-reinterpret-cast.mlir        | 245 +++++++++++-------
 1 file changed, 145 insertions(+), 100 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index dace611bffb7f..6ab4f4e9f6fd8 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -5,14 +5,16 @@
 // Positive tests
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @copy_to_strided_zero_offset(
+/// Effectively copied scalar within a MemRef with rank >= 0
+/// to practically a 1D array within a MemRef with rank >= 1
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   /// reinterpret_cast removed
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32> to memref<1x1xf32>
 
@@ -21,19 +23,19 @@ func.func private @copy_to_strided_zero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [1], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -43,21 +45,21 @@ func.func private @copy_to_strided_nonzero_offset(%src : memref<1x1xf32>,
   // CHECK:      %[[C1:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [1, 1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
@@ -68,7 +70,7 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
   // CHECK-SAME: : memref<1x1xf32>
   /// Dynamic offset used in store
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: ?>>
   // CHECK-NOT:  memref.copy
@@ -76,15 +78,15 @@ func.func private @copy_to_strided_dynamic_offset(%offset: index, %src : memref<
 }
 
 // Dynamic strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_to_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_to_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1], strides: [%stride0, %stride1]
     : memref<1x108xf32>
       to memref<1x1xf32, strided<[?, ?]>>
@@ -93,36 +95,37 @@ func.func private @copy_to_strided_dynamic_stride(%stride0: index,
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[?, ?]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_rank0(
+// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_to_strided_rank0(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
     : memref<f32> to memref<f32>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %reinterpret_cast : memref<f32> to memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_0d_base_2d(
+/// Effectively a 2D array within a MemRef with rank >= 2
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
-func.func private @copy_to_strided_0d_base_2d(
+func.func private @copy_scalar_into_2D_strided(
   %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
     : memref<1x33x42xf32>
       to memref<1x1x1xf32>
@@ -130,19 +133,19 @@ func.func private @copy_to_strided_0d_base_2d(
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_zero_offset(
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
@@ -155,20 +158,20 @@ func.func private @copy_to_strided_1d_vector_zero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_1d_vector_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [10], sizes: [1, 3, 1], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
@@ -182,21 +185,21 @@ func.func private @copy_to_strided_1d_vector_nonzero_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // CHECK-SAME:   %[[OFF:.*]]: index
 // CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [4], strides: [1]
     : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
 
@@ -209,19 +212,19 @@ func.func private @copy_to_strided_1d_vector_1d_base_dynamic_offset(
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_to_strided_2d_vector_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_to_strided_2d_vector_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
@@ -239,7 +242,7 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
   // CHECK:      }
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
   // CHECK-NOT:  memref.copy
@@ -250,11 +253,11 @@ func.func private @copy_to_strided_2d_vector_nonzero_offset(
 // Negative tests (must NOT rewrite)
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_non_identity_base(
-func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
   %dst: memref<8x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [6], sizes: [1, 1], strides: [11, 80]
     : memref<8x1xf32, strided<[10, 2]>>
       to memref<1x1xf32, strided<[11, 80], offset: 6>>
@@ -262,153 +265,195 @@ func.func private @negative_copy_to_strided_non_identity_base(%src: memref<1x1xf
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
 
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_overlapping_result_dims(
-func.func private @negative_copy_to_strided_overlapping_result_dims(%src : memref<3x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
+func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
   %dst : memref<4x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [3, 3], strides: [1, 1]
     : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_rank_change(
-func.func private @negative_copy_to_strided_rank_change(%src : memref<2x3xf32>,
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
+    : memref<1x4x8xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
+  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x4x3xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+  return
+}
+
+/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
+/// the matching source dim has size 4. Directly using the loop IV as the source
+/// dim index would be out-of-bounds. Each loop IV value would need to be
+/// linearized with the result stride and then delinearized into source indices.
+// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
+  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 5], strides: [4, 1]
+    : memref<2x4xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x5xf32>
+      to memref<1x5xf32, strided<[4, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
   %dst : memref<6xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [2, 3], strides: [3, 1]
     : memref<6xf32> to memref<2x3xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<2x3xf32> to memref<2x3xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_to_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_shapes(
-func.func private @negative_copy_to_strided_dynamic_rc_shapes(%dim : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
+func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
   %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
     : memref<?xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<?xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_unranked_rc_base(
-func.func private @negative_copy_to_strided_unranked_rc_base(
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
   %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_0d_base_2d_dynamic_offset(
+// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [%offset], sizes: [1, 1], strides: [8, 1]
     : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[8, 1], offset: ?>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_1d_base_2d_dynamic_offset(
-  %offset : index, %src : memref<1x3x1xf32>,
-  %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_to_strided_2d_dynamic_offset(
-func.func private @negative_copy_to_strided_2d_dynamic_offset(
+/// For effectively-1D copies with dynamic offsets, the single non-unit result
+/// dimension must map to the single non-unit source dimension. Otherwise the
+/// dynamic linear offset would need runtime delinearization before adding IVs.
+// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
+func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
   %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x3x11xf32>) {
+  %dst : memref<1x12x1xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
+    : memref<1x12x1xf32>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: ?>>
+      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
   return
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_to_strided_dynamic_rc_stride(
-func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
+func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %reinterpret_cast = memref.reinterpret_cast %dst
+  %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
     : memref<1x3x11xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
@@ -416,7 +461,7 @@ func.func private @negative_copy_to_strided_dynamic_rc_stride(%stride : index,
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %reinterpret_cast
+  memref.copy %src, %rc
     : memref<1x3x1xf32>
       to memref<1x3x1xf32, strided<[33, ?, 1]>>
   return

>From 36ce2e6480b8a32bf84e64e91798cf874c2dae41 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 15:04:44 +0200
Subject: [PATCH 08/17] Address fourth round of comments

---
 .../Transforms/ElideReinterpretCast.cpp       |  24 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 304 ++++++++++++------
 2 files changed, 214 insertions(+), 114 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index a0e9810249f60..fc5ab79742b81 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,8 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Records the reinterpret_cast result dimensions that span more than one
-/// element and maps each one to its corresponding source dimension.
+/// Maps each non-unit result dimension to a source dimension. Returns false if
+/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
+/// source dimension matches.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,8 +63,9 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
 
-  // Reusing a source dimension would require delinearizing the combined linear
-  // offset, which is TODO.
+  // Each non-unit result dimension needs one source dimension to receive its
+  // loop IV. Combining multiple IVs into one source index would require
+  // linearizing the result position which is TODO.
   SmallVector<bool> usedSrcDims(srcType.getRank(), false);
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -76,14 +78,16 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
 
     int64_t resultStride = rc.getStaticStrides()[resultDim];
     std::optional<unsigned> srcDim;
-    // Find an unused source dimension with matching stride and enough elements.
+    // Pick the first unused source dimension with the same stride.
     for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride ||
-          srcType.getDimSize(idx) < resultSize)
+      if (usedSrcDims[idx] || stride != resultStride)
         continue;
 
-      if (!srcDim || srcType.getDimSize(idx) < srcType.getDimSize(*srcDim))
-        srcDim = idx;
+      assert(srcType.getDimSize(idx) >= resultSize &&
+             "reinterpret_cast result dimension does not fit in the matching "
+             "source dimension");
+      srcDim = idx;
+      break;
     }
     if (!srcDim)
       return false;
@@ -120,7 +124,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   SmallVector<int64_t> offsetIdxs(srcType.getRank(), 0);
   int64_t remainder = rcOffsets[0];
   SmallVector<int64_t> srcStrides = computeStrides(srcType.getShape());
-  // Convert the linear reinterpret_cast offset to per-dimension source starting
+  // Convert the scalar reinterpret_cast offset to per-dimension source starting
   // indices.
   for (auto [dim, stride] : llvm::enumerate(srcStrides)) {
     offsetIdxs[dim] = remainder / stride;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 6ab4f4e9f6fd8..5e40f9694bab8 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,10 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
+//
+// The destination is effectively a 1D array within a MemRef with rank >= 1 
 //===----------------------------------------------------------------------===//
 
-/// Effectively copied scalar within a MemRef with rank >= 0
-/// to practically a 1D array within a MemRef with rank >= 1
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -42,9 +42,9 @@ func.func private @copy_scalar_into_1D_strided_nonzero_offset(%src : memref<1x1x
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
-  // CHECK:      %[[C1:.*]] = arith.constant 1 : index
+  // CHECK:      %[[OFF:.*]] = arith.constant 1 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C1]]] : memref<1x108xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[OFF]]] : memref<1x108xf32>
   memref.copy %src, %rc
     : memref<1x1xf32>
       to memref<1x1xf32, strided<[1, 1], offset: 1>>
@@ -77,7 +77,26 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Dynamic strides are irrelevant because all strided memref indices are zero.
+// Strides are irrelevant because all strided memref indices are zero.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
+func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+  %dst : memref<1x108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %reinterpret_cast = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [107, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[C0:.*]] = arith.constant 0 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
+  memref.copy %src, %reinterpret_cast
+    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
@@ -102,9 +121,41 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_0D_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a scalar within a MemRef with rank == 0 
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -118,31 +169,36 @@ func.func private @copy_0D_into_0D_strided(%src : memref<f32>, %dst : memref<f32
   return
 }
 
-/// Effectively a 2D array within a MemRef with rank >= 2
+//===----------------------------------------------------------------------===//
+// Positive tests
+//
+// The destination is effectively a 2D array within a MemRef with rank >= 2 
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x33x42xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
 func.func private @copy_scalar_into_2D_strided(
-  %src : memref<1x1x1xf32>, %dst : memref<1x33x42xf32>) {
+  %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 1, 1], strides: [1, 1, 1]
-    : memref<1x33x42xf32>
+    : memref<1x3x11xf32>
       to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x33x42xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x3x11xf32>
   memref.copy %src, %rc
     : memref<1x1x1xf32> to memref<1x1x1xf32>
   // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_1D_strided_zero_offset(
+func.func private @copy_1D_into_2D_strided_zero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -153,8 +209,8 @@ func.func private @copy_1D_into_1D_strided_zero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
   // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
   // CHECK:      }
@@ -179,11 +235,11 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C10:.*]] = arith.constant 10 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
   // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C10]]] : memref<1x3x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[OFF]]] : memref<1x3x11xf32>
   // CHECK:      }
   memref.copy %src, %rc
     : memref<1x3x1xf32>
@@ -192,28 +248,31 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<42xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<42xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<42xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<42xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
@@ -232,12 +291,12 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[C3:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[C4:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[C7:.*]] = arith.constant 7 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[C3]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[C4]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[C7]], %[[IDX1]] : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
   // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
   // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
   // CHECK:        }
@@ -249,59 +308,102 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<8x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
+func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
+  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [11, 80]
-    : memref<8x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
+    : memref<1x3x4xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
+  // CHECK:        }
+  // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[11, 80], offset: 6>>
+    : memref<1x4x3xf32>
+      to memref<1x4x3xf32, strided<[12, 1, 4]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
 
+/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %{{[^:]+}}: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
+  %stride0 : index, %stride2 : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_overlapping_result_dims(
-func.func private @negative_copy_into_strided_overlapping_result_dims(%src : memref<3x3xf32>,
-  %dst : memref<4x4xf32>) {
+//===----------------------------------------------------------------------===//
+// Negative tests (must NOT rewrite)
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 3], strides: [1, 1]
-    : memref<4x4xf32> to memref<3x3xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x3xf32> to memref<3x3xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
 func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x8xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [32, 5, 1]
-    : memref<1x4x8xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[32, 5, 1]>>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
@@ -323,67 +425,61 @@ func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
   return
 }
 
-/// The linear accessed region is in-bounds, but result dim 1 has size 5 while
-/// the matching source dim has size 4. Directly using the loop IV as the source
-/// dim index would be out-of-bounds. Each loop IV value would need to be
-/// linearized with the result stride and then delinearized into source indices.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-func.func private @negative_copy_into_strided_crosses_source_dim_boundary(
-  %src : memref<1x5xf32>, %dst : memref<2x4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 5], strides: [4, 1]
-    : memref<2x4xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x5xf32>
-      to memref<1x5xf32, strided<[4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<2x3xf32>,
-  %dst : memref<6xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
+func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
+  %dst : memref<4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [2, 3], strides: [3, 1]
-    : memref<6xf32> to memref<2x3xf32>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<4xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<2x3xf32> to memref<2x3xf32>
+    : memref<?xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
-func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
-  %dst : memref<4xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_source_shape(
+  %src : memref<4xf32>, %dst : memref<?xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
-    : memref<4xf32> to memref<4xf32>
+    : memref<?xf32> to memref<4xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<?xf32> to memref<4xf32>
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_shapes(
-func.func private @negative_copy_into_strided_dynamic_rc_shapes(%dim : index,
-  %src : memref<4xf32>, %dst : memref<?xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_result_shape(
+func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : index,
+  %src : memref<4xf32>, %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [%dim], strides: [1]
-    : memref<?xf32> to memref<?xf32, strided<[1]>>
+    : memref<12xf32> to memref<?xf32, strided<[1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
@@ -411,18 +507,18 @@ func.func private @negative_copy_into_strided_unranked_rc_base(
 
 // CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
 func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<4x8xf32>) {
+  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [8, 1]
-    : memref<4x8xf32> to memref<1x1xf32, strided<[8, 1], offset: ?>>
+    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
+    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x1xf32>
-      to memref<1x1xf32, strided<[8, 1], offset: ?>>
+      to memref<1x1xf32, strided<[12, 1], offset: ?>>
   return
 }
 
@@ -449,21 +545,21 @@ func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
 }
 
 /// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_rc_stride(
-func.func private @negative_copy_into_strided_dynamic_rc_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
+func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, %stride, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
 
   // CHECK:      memref.copy %arg1, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, ?, 1]>>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
   return
 }
 

>From acab220b08c58e1fe5bf5e73a97fff8d060a6f92 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 14:13:18 +0200
Subject: [PATCH 09/17] Drop support for non-identical rc source and result
 strides

---
 .../Transforms/ElideReinterpretCast.cpp       |  94 +++---
 .../MemRef/elide-reinterpret-cast.mlir        | 295 +++++++-----------
 2 files changed, 156 insertions(+), 233 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index fc5ab79742b81..6d1c3e76ac14d 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -51,9 +51,9 @@ struct AssocMapAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension. Returns false if
-/// a stride of a non-unit dimension in the rc result is dynamic or no distinct
-/// source dimension matches.
+/// Maps each non-unit result dimension to a source dimension by stride. Returns
+/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
+/// result strides are not equivalent to rc source strides.
 static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
                                       AssocMapAndOffsetsForRC &mapAndOffs) {
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
@@ -62,40 +62,34 @@ static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
          "Expecting identity source layout.");
 
   SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
+  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-  // Each non-unit result dimension needs one source dimension to receive its
-  // loop IV. Combining multiple IVs into one source index would require
-  // linearizing the result position which is TODO.
-  SmallVector<bool> usedSrcDims(srcType.getRank(), false);
+  assert(srcType.getRank() == resType.getRank() &&
+         "Expecting rank-preserving reinterpret_casts");
 
   for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize == 1)
       continue;
 
-    // TODO: Support dynamic strides on non-unit result dimensions.
-    if (ShapedType::isDynamic(rc.getStaticStrides()[resultDim]))
+    int64_t resultStride = rcResultStrides[resultDim];
+
+    if (ShapedType::isDynamic(resultStride))
       return false;
 
-    int64_t resultStride = rc.getStaticStrides()[resultDim];
-    std::optional<unsigned> srcDim;
-    // Pick the first unused source dimension with the same stride.
-    for (auto [idx, stride] : llvm::enumerate(srcIdentityStrides)) {
-      if (usedSrcDims[idx] || stride != resultStride)
-        continue;
-
-      assert(srcType.getDimSize(idx) >= resultSize &&
-             "reinterpret_cast result dimension does not fit in the matching "
-             "source dimension");
-      srcDim = idx;
-      break;
-    }
-    if (!srcDim)
+    // For non-scalar strided memrefs, only support result strides identical to
+    // the identity strides of the source. This enables direct indexing into the
+    // same source dimensions, without linearization.
+    if (resultStride != srcIdentityStrides[resultDim])
       return false;
 
-    usedSrcDims[*srcDim] = true;
-    mapAndOffs.assocMap.push_back(
-        NonUnitDimAssocMapForRC{static_cast<unsigned>(resultDim), *srcDim});
+    assert(srcType.getDimSize(resultDim) >= resultSize &&
+           "reinterpret_cast result dimension does not fit in the matching "
+           "source dimension");
+
+    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
+        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
   }
+
   return true;
 }
 
@@ -164,29 +158,28 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return rewrite info:
 ///
 ///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   There are
-///   // no non-unit result dimensions, so dynamic strides in the strided memref
-///   // do not affect index mapping.
+///   // There are no non-unit result dimensions, so strides in the strided
+///   // memref do not affect index mapping.
 ///   copy memref<1 x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
 ///
 ///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to an identity-layout source dimension by its static stride.
+///   // mapped to its matching identity-layout stride source dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
 ///
 ///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped independently by its static stride.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///   // dimension is mapped in order, checking for matching source static
+///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
 ///     to reinterpret_cast memref<source-shape>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
 ///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Dynamic stride on a non-unit strided memref dimension.
+///   // Non-scalar copies: dynamic strides in strided memref.
 ///   copy memref<1xNxf32>
 ///     to reinterpret_cast memref<1xNxMxf32>
 ///       to memref<1xNxf32, strided<[?, ?]>>
@@ -220,24 +213,31 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   AssocMapAndOffsetsForRC mapAndOffs;
 
-  // reinterpret_cast result dimensions must map to distinct source dimensions.
-  if (!findSourceDimForResultDim(rc, mapAndOffs))
+  assert(resType.hasStaticShape() && "expected static shape");
+  // For scalar copies, result strides are irrelevant, including dynamic ones.
+  // For non-scalar copies, require static result strides identical to the
+  // identity strides of the reinterpret_cast source.
+  if (!llvm::all_of(resType.getShape(),
+                    [](int64_t size) { return size == 1; }) &&
+      !findSourceDimForResultDim(rc, mapAndOffs))
     return std::nullopt;
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  // Static ReinterpretCast offset
+
+  // CASE 1: Static ReinterpretCast offset
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
     mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(
-        mapAndOffs.delinearizedOffsets &&
-        "static reinterpret_cast offset must delinearize to in-bounds source "
-        "indices");
+    assert(mapAndOffs.delinearizedOffsets &&
+           "static reinterpret_cast offset must delinearize to in-bounds "
+           "reinterpret_cast source indices");
 
+    // Relevant for non-scalar copies: assert that the rectangular
+    // copied slice is in bounds.
     assert(
         llvm::all_of(
             mapAndOffs.assocMap,
@@ -250,17 +250,21 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     return mapAndOffs;
   }
 
-  // Dynamic ReinterpretCast offset.
+  // CASE 2: Dynamic ReinterpretCast offset.
   // TODO: Support dynamic offsets into sources with multiple non-unit
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
-  if (mapAndOffs.assocMap.size() > 1)
-    return std::nullopt;
-  // Only sources with a single non-unit dimension can receive a dynamic offset
-  // directly.
+
+  // With an effectively-1D source, a dynamic linear offset can be used directly
+  // as the index of the unique non-unit source dimension.
   if (!getSingleNonUnitDim(srcType))
     return std::nullopt;
 
+  // Non-scalar copies require identical strides and no rank-changing,
+  // so there can be at most one non-unit result dimension in this case.
+  assert(mapAndOffs.assocMap.size() <= 1 &&
+         "effectively-1D source cannot have multiple mapped non-unit dims");
+
   return mapAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 5e40f9694bab8..1a9208a75fee6 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -3,10 +3,28 @@
 
 //===----------------------------------------------------------------------===//
 // Positive tests
-//
-// The destination is effectively a 1D array within a MemRef with rank >= 1 
+// 
+// Scalar (0D) copy
 //===----------------------------------------------------------------------===//
 
+// The destination is effectively a scalar within a MemRef with rank == 0 
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
+func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [], strides: []
+    : memref<f32> to memref<f32>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
+  memref.copy %src, %rc : memref<f32> to memref<f32>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// The destination is effectively a 1D array within a MemRef with rank >= 1 
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -77,11 +95,12 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Strides are irrelevant because all strided memref indices are zero.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_static_stride(
+// Scalar copies have no varying result dimensions, so rc result strides do
+// not affect the copy destination address and are ignored.
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_static_stride(%src : memref<1x1xf32>,
+func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
@@ -121,64 +140,11 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a scalar within a MemRef with rank == 0 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [], strides: []
-    : memref<f32> to memref<f32>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][] : memref<f32>
-  // CHECK:      memref.store %[[VAL]], %[[DST]][] : memref<f32>
-  memref.copy %src, %rc : memref<f32> to memref<f32>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Positive tests
-//
-// The destination is effectively a 2D array within a MemRef with rank >= 2 
-//===----------------------------------------------------------------------===//
-
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided(
+/// The destination is effectively a 2D array within a MemRef with rank >= 2 
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided(
+func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -195,6 +161,12 @@ func.func private @copy_scalar_into_2D_strided(
   return
 }
 
+//===----------------------------------------------------------------------===//
+// Positive tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -308,139 +280,107 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x4x3xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x4xf32>
-func.func private @copy_2D_into_2D_strided_rc_non_identity_strides(
-  %src : memref<1x4x3xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 4, 3], strides: [12, 1, 4]
-    : memref<1x3x4xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x4x3xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX1]], %[[IDX0]]] : memref<1x3x4xf32>
-  // CHECK:        }
-  // CHECK:      }
-  memref.copy %src, %rc
-    : memref<1x4x3xf32>
-      to memref<1x4x3xf32, strided<[12, 1, 4]>>
-  // CHECK-NOT:  memref.copy
-  return
-}
-
-/// Only strides in non-unit dimensions of reinterpret_cast result are checked.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %{{[^:]+}}: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_rc_unit_dim_dynamic_stride(
-  %stride0 : index, %stride2 : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
+// CHECK-SAME:   %[[OFF:.*]]: index
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_dynamic_offset(
+  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [%stride0, 11, %stride2]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    to offset: [%offset], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
   // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x11xf32>
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[?, 11, ?]>>
+    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
   // CHECK-NOT:  memref.copy
   return
 }
 
 //===----------------------------------------------------------------------===//
 // Negative tests (must NOT rewrite)
+// 
+// Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
+/// Reject copies that don't target a strided memref
+// CHECK-LABEL: func.func private @negative_plain_copy(
+func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+  %dst : memref<1x1xf32>) {
+  // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
-
+  memref.copy %src, %dst
+  : memref<1x1xf32> to memref<1x1xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_1D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+/// Reject unranked memref operands
+// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
+func.func private @negative_copy_into_strided_unranked_rc_base(
+  %src : memref<4xf32>, %dst : memref<*xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<*xf32> to memref<4xf32>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    : memref<4xf32> to memref<4xf32>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-func.func private @negative_copy_2D_into_2D_strided_unmatched_strides(
-  %src : memref<1x3x4xf32>, %dst : memref<1x4x3xf32>) {
+/// Reject rank-changing reinterpet_casts
+// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
+func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
+  %dst : memref<12xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x4x3xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    to offset: [0], sizes: [3, 4], strides: [1, 1]
+    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1]>>
+    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rank_change(
-func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>,
-  %dst : memref<12xf32>) {
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
+func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
+  %dst: memref<108x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [3, 4], strides: [1, 1]
-    : memref<12xf32> to memref<3x4xf32, strided<[1, 1]>>
+    to offset: [6], sizes: [1, 1], strides: [10, 2]
+    : memref<108x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2], offset: 6>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<3x4xf32> to memref<3x4xf32, strided<[1, 1]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+
   return
 }
 
+/// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
   %dst : memref<4xf32>) {
@@ -489,24 +429,10 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_into_strided_unranked_rc_base(
-func.func private @negative_copy_into_strided_unranked_rc_base(
-  %src : memref<4xf32>, %dst : memref<*xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [4], strides: [1]
-    : memref<*xf32> to memref<4xf32>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32>
-  return
-}
-
-// CHECK-LABEL: func.func private @negative_copy_into_multidim_strided_dynamic_offset(
-func.func private @negative_copy_into_multidim_strided_dynamic_offset(
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
+func.func private @negative_copy_into_ND_strided_dynamic_offset(
   %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -522,31 +448,35 @@ func.func private @negative_copy_into_multidim_strided_dynamic_offset(
   return
 }
 
-/// For effectively-1D copies with dynamic offsets, the single non-unit result
-/// dimension must map to the single non-unit source dimension. Otherwise the
-/// dynamic linear offset would need runtime delinearization before adding IVs.
-// CHECK-LABEL: func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-func.func private @negative_copy_multidim_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<1x3x4xf32>,
-  %dst : memref<1x12x1xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+//===----------------------------------------------------------------------===//
+// Negative tests
+// 
+// Non-scalar (ND) copy
+//===----------------------------------------------------------------------===//
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
+func.func private @negative_copy_into_strided_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 3, 4], strides: [12, 4, 1]
-    : memref<1x12x1xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
 
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[12, 4, 1], offset: ?>>
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[33, 10, 1]>>
   return
 }
 
-/// Non-unit copied dimension needs stride-based address computation.
-// CHECK-LABEL: func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(
-func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%stride : index,
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
+func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
   %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
@@ -563,17 +493,6 @@ func.func private @negative_copy_into_strided_rc_non_unit_dim_dynamic_stride(%st
   return
 }
 
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
-  %dst : memref<1x1xf32>) {
-  // CHECK:      memref.copy %arg0, %arg1
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %dst
-  : memref<1x1xf32> to memref<1x1xf32>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 961944f75cce955a9c7270d774adb53049e7de66 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 09:47:56 +0200
Subject: [PATCH 10/17] Fixup

Remove reduntant dimension mapping, refine comments and tests
---
 .../Transforms/ElideReinterpretCast.cpp       | 304 ++++++++++--------
 .../MemRef/elide-reinterpret-cast.mlir        |  32 +-
 2 files changed, 178 insertions(+), 158 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6d1c3e76ac14d..3ffd834c2863b 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -35,64 +35,15 @@ namespace {
 // Copy Rewrite Helpers
 //===----------------------------------------------------------------------===//
 
-/// Non-unit reinterpret_cast result dimension and the source dimension it
-/// advances through.
-struct NonUnitDimAssocMapForRC {
-  unsigned resultDimPos;
-  unsigned sourceDimPos;
-};
-
 /// Copy-relevant information derived from a reinterpret_cast.
-struct AssocMapAndOffsetsForRC {
+struct ResultNonUnitDimsAndOffsetsForRC {
   // Non-unit dimensions of the reinterpret_cast result.
-  SmallVector<NonUnitDimAssocMapForRC> assocMap;
+  SmallVector<unsigned> nonUnitDimsPos;
   // Delinearized offsets to in-bounds reinterpret_cast source indices.
   // Optional since it is only supported for static offsets.
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Maps each non-unit result dimension to a source dimension by stride. Returns
-/// false if a stride of a non-unit dimension in the rc result is dynamic or rc
-/// result strides are not equivalent to rc source strides.
-static bool findSourceDimForResultDim(memref::ReinterpretCastOp rc,
-                                      AssocMapAndOffsetsForRC &mapAndOffs) {
-  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
-  assert(srcType.getLayout().isIdentity() &&
-         "Expecting identity source layout.");
-
-  SmallVector<int64_t> srcIdentityStrides = computeStrides(srcType.getShape());
-  ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
-
-  assert(srcType.getRank() == resType.getRank() &&
-         "Expecting rank-preserving reinterpret_casts");
-
-  for (auto [resultDim, resultSize] : llvm::enumerate(resType.getShape())) {
-    if (resultSize == 1)
-      continue;
-
-    int64_t resultStride = rcResultStrides[resultDim];
-
-    if (ShapedType::isDynamic(resultStride))
-      return false;
-
-    // For non-scalar strided memrefs, only support result strides identical to
-    // the identity strides of the source. This enables direct indexing into the
-    // same source dimensions, without linearization.
-    if (resultStride != srcIdentityStrides[resultDim])
-      return false;
-
-    assert(srcType.getDimSize(resultDim) >= resultSize &&
-           "reinterpret_cast result dimension does not fit in the matching "
-           "source dimension");
-
-    mapAndOffs.assocMap.push_back(NonUnitDimAssocMapForRC{
-        static_cast<unsigned>(resultDim), static_cast<unsigned>(resultDim)});
-  }
-
-  return true;
-}
-
 /// Returns source indices for a static reinterpret_cast offset of an
 /// identity-layout source.
 static std::optional<SmallVector<int64_t>>
@@ -157,39 +108,93 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///
 /// Examples that return rewrite info:
 ///
-///   // Scalar-shaped copy into a source with at most one non-unit dimension.
-///   // There are no non-unit result dimensions, so strides in the strided
-///   // memref do not affect index mapping.
+///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
+///   // strides do not affect index mapping - may be static or
+///   // dynamic.
+///   copy memref<1 x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: O>>
+///
+///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
+///   // The dynamic offset can be used directly as the index of the source's
+///   // unique non-unit dimension.
 ///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x 1 x f32, strided<[?, ..., ?], offset: ?>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x 1 x f32,
+///                 strided<[?, ..., ?], offset: ?>>
+///
+///   // Non-scalar effectively-1D copy with static offset. Result strides must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N x ... x 1 x f32>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: O>>
 ///
-///   // Effectively-1D copy. The single non-unit strided memref dimension is
-///   // mapped to its matching identity-layout stride source dimension.
+///   // Non-scalar effectively-1D copy with dynamic offset into an
+///   // effectively-1D source. Runtime delinearization is not needed because
+///   the
+///   // source has a unique non-unit dimension.
 ///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
-///       to memref<1 x ... x N x ... x 1 x f32, strided<[..., S, ...]>>
+///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
+///                               identity-layout>
+///       to memref<1 x ... x N x ... x 1 x f32,
+///                 strided<[source-identity-strides], offset: ?>>
 ///
-///   // Multidimensional copy with static offset. Each non-unit strided memref
-///   // dimension is mapped in order, checking for matching source static
-///   stride. copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape>
+///   // Non-scalar multidimensional copy with static offset. Result strides
+///   must
+///   // be static and identical to the identity strides of the source.
+///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
+///     to reinterpret_cast memref<source-shape, identity-layout>
 ///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[..., S_0, ..., S_1, ...], offset: O>>
+///                 strided<[source-identity-strides], offset: O>>
 ///
 /// Examples that return no info:
 ///
-///   // Non-scalar copies: dynamic strides in strided memref.
+///   // Rank-changing reinterpret_casts are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<Mxf32, identity-layout>
+///       to memref<1xNxf32, strided<[N, 1]>>
+///
+///   // Dynamic shapes are not supported.
+///   copy memref<?xNxf32>
+///     to reinterpret_cast memref<?xMxf32, identity-layout>
+///       to memref<?xNxf32, strided<[M, 1]>>
+///
+///   // Non-identity source layouts are not supported.
 ///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxf32, strided<[?, ?]>>
+///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
+///       to memref<1xNxf32, strided<[M, 1]>>
 ///
-///   // Multidimensional copy with dynamic linear offset.
+///   // Dynamic offset into a source with more than one non-unit dimension is
+///   // not supported because runtime delinearization is not implemented.
+///   copy memref<1x1xf32>
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
+///
+///   // Non-scalar copies with dynamic result strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[?, 1]>>
+///
+///   // Non-scalar copies with result strides different from the source
+///   identity
+///   // strides are not supported.
+///   copy memref<1xNxf32>
+///     to reinterpret_cast memref<1xMxf32, identity-layout>
+///       to memref<1xNxf32, strided<[S, 1]>>
+///
+///   // Multidimensional non-scalar copies with dynamic offset are not
+///   supported
+///   // unless the source is effectively 1D.
 ///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32>
-///       to memref<1xNxKxf32, strided<[N*M, M, 1], offset: ?>>
-static std::optional<AssocMapAndOffsetsForRC>
-getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
+///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
+///       to memref<1xNxKxf32,
+///                 strided<[N*M, M, 1], offset: ?>>
+static std::optional<ResultNonUnitDimsAndOffsetsForRC>
+getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
 
@@ -211,16 +216,34 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType.getLayout().isIdentity())
     return std::nullopt;
 
-  AssocMapAndOffsetsForRC mapAndOffs;
+  ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
   // For scalar copies, result strides are irrelevant, including dynamic ones.
   // For non-scalar copies, require static result strides identical to the
   // identity strides of the reinterpret_cast source.
   if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; }) &&
-      !findSourceDimForResultDim(rc, mapAndOffs))
-    return std::nullopt;
+                    [](int64_t size) { return size == 1; })) {
+    SmallVector<int64_t> srcIdentityStrides =
+        computeStrides(srcType.getShape());
+    ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
+
+    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
+           "Expecting same number of strides for rank-preserving "
+           "reinterpret_casts.");
+
+    if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
+                      [](auto pair) {
+                        auto [srcStride, resultStride] = pair;
+                        return !ShapedType::isDynamic(resultStride) &&
+                               srcStride == resultStride;
+                      }))
+      return std::nullopt;
+    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+      if (resultSize != 1)
+        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+    }
+  }
 
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
@@ -231,23 +254,23 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (ShapedType::isStatic(rcOffsets[0])) {
     // Delinearize static ReinterpretCast offset as in-bounds indices (one for
     // every source dimension).
-    mapAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(mapAndOffs.delinearizedOffsets &&
+    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+    assert(dimsAndOffs.delinearizedOffsets &&
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
     // Relevant for non-scalar copies: assert that the rectangular
     // copied slice is in bounds.
-    assert(
-        llvm::all_of(
-            mapAndOffs.assocMap,
-            [&](const NonUnitDimAssocMapForRC &assocMap) {
-              return (*mapAndOffs.delinearizedOffsets)[assocMap.sourceDimPos] +
-                         resType.getDimSize(assocMap.resultDimPos) <=
-                     srcType.getDimSize(assocMap.sourceDimPos);
-            }) &&
-        "reinterpret_cast metadata describes an invalid accessible region");
-    return mapAndOffs;
+    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
+                        [&](auto it) {
+                          unsigned dim = it.index();
+                          int64_t resultSize = it.value();
+                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
+                                     resultSize <=
+                                 srcType.getDimSize(dim);
+                        }) &&
+           "reinterpret_cast metadata describes an invalid accessible region");
+    return dimsAndOffs;
   }
 
   // CASE 2: Dynamic ReinterpretCast offset.
@@ -262,10 +285,10 @@ getAssocMapAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   // Non-scalar copies require identical strides and no rank-changing,
   // so there can be at most one non-unit result dimension in this case.
-  assert(mapAndOffs.assocMap.size() <= 1 &&
-         "effectively-1D source cannot have multiple mapped non-unit dims");
+  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
+         "effectively-1D source cannot have multiple non-unit result dims");
 
-  return mapAndOffs;
+  return dimsAndOffs;
 }
 
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
@@ -314,21 +337,23 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     if (!cpSrcType || !cpSrcType.hasStaticShape())
       return rewriter.notifyMatchFailure(
           op, "only ranked, static copy sources are supported.");
+
     Value rcOutput = op.getTarget();
     auto rc = rcOutput.getDefiningOp<memref::ReinterpretCastOp>();
     if (!rc)
       return rewriter.notifyMatchFailure(
           op, "target is not a memref.reinterpret_cast");
 
-    std::optional<AssocMapAndOffsetsForRC> mapAndOffs =
-        getAssocMapAndOffsetsForRC(rc);
-    if (!mapAndOffs)
+    std::optional<ResultNonUnitDimsAndOffsetsForRC> dimsAndOffs =
+        getResultNonUnitDimsAndOffsetsForRC(rc);
+    if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
           op, "reinterpret_cast does not match scalar or loop copy region");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
     MemRefType dstType = cast<MemRefType>(dst.getType());
+    MemRefType rcResType = cast<MemRefType>(rc.getType());
 
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
@@ -342,6 +367,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       }
       return arith::ConstantIndexOp::create(rewriter, loc, value);
     };
+
     auto getZeroIdxs = [&](int64_t rank) {
       SmallVector<Value> idxs;
       idxs.reserve(rank);
@@ -353,27 +379,28 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     // Create loop bounds before moving the insertion point into the loop nest,
     // so loop-invariant constants are emitted outside the generated loops.
     SmallVector<Value> upperBounds;
-    upperBounds.reserve(mapAndOffs->assocMap.size());
-    MemRefType rcResType = dyn_cast<MemRefType>(rc.getType());
-    for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-      upperBounds.push_back(getOrCreateIndexConstant(
-          rcResType.getDimSize(assocMap.resultDimPos)));
+    upperBounds.reserve(dimsAndOffs->nonUnitDimsPos.size());
+    for (unsigned dim : dimsAndOffs->nonUnitDimsPos) {
+      upperBounds.push_back(
+          getOrCreateIndexConstant(rcResType.getDimSize(dim)));
+    }
 
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
-    // Static offset has been delinearized in function gating rewrite.
-    if (mapAndOffs->delinearizedOffsets) {
+    if (dimsAndOffs->delinearizedOffsets) {
+      // Initialize store indices from the static reinterpret_cast offset,
+      // delinearized in function gating rewrite.
       for (auto [idx, offset] :
-           llvm::enumerate(*mapAndOffs->delinearizedOffsets)) {
+           llvm::enumerate(*dimsAndOffs->delinearizedOffsets)) {
         if (offset == 0)
           continue;
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Without runtime delinearization, use the dynamic offset directly only
-      // when the source has a single non-unit dimension.
-      assert(mapAndOffs->assocMap.size() <= 1 &&
-             "Expecting single non-unit dimension mapping.");
+      // Dynamic offsets are used directly only for effectively-1D sources.
+      assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
+             "Expecting at most one non-unit result dimension.");
+
       srcNonUnitDimPos = getSingleNonUnitDim(dstType);
       assert(srcNonUnitDimPos &&
              "Expecting single non-unit dimension source to receive the "
@@ -387,45 +414,38 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
 
-    // Scope for OpBuilder::InsertionGuard.
+    // Create the loop nest and emit the load/store at the innermost insertion
+    // point.
     {
       OpBuilder::InsertionGuard guard(rewriter);
-      Value lowerBound;
-      Value step;
-      if (!upperBounds.empty()) {
-        lowerBound = getOrCreateIndexConstant(0);
-        step = getOrCreateIndexConstant(1);
-      }
-      SmallVector<Value> loopIvs;
-      loopIvs.reserve(mapAndOffs->assocMap.size());
-
-      // Build one nested loop per non-unit strided memref dimension.
-      for (Value upperBound : upperBounds) {
-        scf::ForOp loop =
-            scf::ForOp::create(rewriter, loc, lowerBound, upperBound, step);
-        loopIvs.push_back(loop.getInductionVar());
-        rewriter.setInsertionPointToStart(loop.getBody());
-      }
 
-      // Load indices are zero except for non-unit strided memref dimensions,
-      // which use the corresponding loop induction variables.
       SmallVector<Value> loadIdxs = getZeroIdxs(cpSrcType.getRank());
-      unsigned loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap)
-        loadIdxs[assocMap.resultDimPos] = loopIvs[loopIndex++];
-
-      // Store indices start from the offset-derived source indices. Add each
-      // loop IV to the mapped source dimension.
       SmallVector<Value> storeIdxs(rcSrcStoreIdxs);
-      loopIndex = 0;
-      for (const NonUnitDimAssocMapForRC &assocMap : mapAndOffs->assocMap) {
-        Value iv = loopIvs[loopIndex++];
-        // Add each IV to one source index.
-        if (storeIdxs[assocMap.sourceDimPos] == getOrCreateIndexConstant(0)) {
-          storeIdxs[assocMap.sourceDimPos] = iv;
-        } else {
-          storeIdxs[assocMap.sourceDimPos] = arith::AddIOp::create(
-              rewriter, loc, storeIdxs[assocMap.sourceDimPos], iv);
+
+      if (!dimsAndOffs->nonUnitDimsPos.empty()) {
+        Value lowerBound = getOrCreateIndexConstant(0);
+        Value step = getOrCreateIndexConstant(1);
+
+        // Build one nested loop per non-unit reinterpret_cast result dimension.
+        for (auto [loopIndex, dim] :
+             llvm::enumerate(dimsAndOffs->nonUnitDimsPos)) {
+          scf::ForOp loop = scf::ForOp::create(rewriter, loc, lowerBound,
+                                               upperBounds[loopIndex], step);
+
+          rewriter.setInsertionPointToStart(loop.getBody());
+
+          Value iv = loop.getInductionVar();
+          // Since result strides match source identity strides dimension-wise,
+          // each IV indexes the same dimension in both the copy source and rc
+          // source.
+          loadIdxs[dim] = iv;
+
+          if (storeIdxs[dim] == getOrCreateIndexConstant(0)) {
+            storeIdxs[dim] = iv;
+          } else {
+            storeIdxs[dim] =
+                arith::AddIOp::create(rewriter, loc, storeIdxs[dim], iv);
+          }
         }
       }
 
@@ -649,7 +669,7 @@ struct ElideReinterpretCastPass
       // reinterpret_cast result can be mapped back to base memref indices.
       MemRefType cpSrcType = dyn_cast<MemRefType>(op.getSource().getType());
       return !(cpSrcType && cpSrcType.hasStaticShape() &&
-               getAssocMapAndOffsetsForRC(rc));
+               getResultNonUnitDimsAndOffsetsForRC(rc));
     });
     target.addDynamicallyLegalOp<memref::LoadOp>([](memref::LoadOp op) {
       auto rc = op.getMemRef().getDefiningOp<memref::ReinterpretCastOp>();
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 1a9208a75fee6..2f4be0a011d54 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -104,15 +104,15 @@ func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref
   %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [107, 2]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[107, 2]>>
+    to offset: [0], sizes: [1, 1], strides: [2, 54]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[107, 2]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
   return
 }
 
@@ -315,9 +315,9 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// Reject copies that don't target a strided memref
-// CHECK-LABEL: func.func private @negative_plain_copy(
-func.func private @negative_plain_copy(%src : memref<1x1xf32>,
+/// Reject copies that don't target a reinterpret_cast result
+// CHECK-LABEL: func.func private @negative_no_rc(
+func.func private @negative_no_rc(%src : memref<1x1xf32>,
   %dst : memref<1x1xf32>) {
   // CHECK:      memref.copy %arg0, %arg1
   // CHECK-NOT:  memref.load
@@ -364,18 +364,18 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
 /// Reject non-identity layout rc source strides
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
 func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<108x1xf32, strided<[10, 2]>>) {
+  %dst: memref<12x1xf32, strided<[10, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [6], sizes: [1, 1], strides: [10, 2]
-    : memref<108x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    to offset: [0], sizes: [1, 1], strides: [10, 2]
+    : memref<12x1xf32, strided<[10, 2]>>
+      to memref<1x1xf32, strided<[10, 2]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2], offset: 6>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
 
   return
 }
@@ -458,19 +458,19 @@ func.func private @negative_copy_into_ND_strided_dynamic_offset(
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
 func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [33, 10, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
 
   // CHECK:      memref.copy %arg0, %reinterpret_cast
   // CHECK-NOT:  memref.load
   // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[33, 10, 1]>>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
   return
 }
 

>From d524369429122b4235106e1acb1118820cdf8a80 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Thu, 2 Jul 2026 15:54:30 +0200
Subject: [PATCH 11/17] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 117 ++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 224 ++++++++++--------
 2 files changed, 146 insertions(+), 195 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 3ffd834c2863b..ba342ffe8c655 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -49,15 +49,14 @@ struct ResultNonUnitDimsAndOffsetsForRC {
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
   assert(rcOffsets.size() == 1 && "Expecting single offset");
-  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
 
+  assert(ShapedType::isStatic(rcOffsets[0]) && "expected static offset");
   assert(rcOffsets[0] >= 0 &&
          "static reinterpret_cast offset must be non-negative");
-
-  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   assert(srcType.getLayout().isIdentity() &&
          "Expecting identity source layout.");
   if (srcType.getRank() == 0) {
@@ -102,97 +101,29 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns the copy-relevant reinterpret_cast information: non-unit result
-/// dimensions, their source-dimension mapping, and optional source starting
-/// indices for a static offset.
-///
-/// Examples that return rewrite info:
+/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
+/// the corresponding source indices.
 ///
-///   // Scalar-shaped copy. There are no non-unit result dimensions, so result
-///   // strides do not affect index mapping - may be static or
-///   // dynamic.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: O>>
+/// Supports ranked, static-shape, rank-preserving reinterpret_casts from
+/// identity-layout sources. Non-scalar results must have static strides
+/// identical to the source identity strides. Dynamic offsets are supported only
+/// for effectively-1D sources.
 ///
-///   // Scalar-shaped copy with dynamic offset into an effectively-1D source.
-///   // The dynamic offset can be used directly as the index of the source's
-///   // unique non-unit dimension.
-///   copy memref<1 x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x 1 x f32,
-///                 strided<[?, ..., ?], offset: ?>>
+/// Examples that return info:
 ///
-///   // Non-scalar effectively-1D copy with static offset. Result strides must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
 ///
-///   // Non-scalar effectively-1D copy with dynamic offset into an
-///   // effectively-1D source. Runtime delinearization is not needed because
-///   the
-///   // source has a unique non-unit dimension.
-///   copy memref<1 x ... x N x ... x 1 x f32>
-///     to reinterpret_cast memref<1 x ... x M x ... x 1 x f32,
-///                               identity-layout>
-///       to memref<1 x ... x N x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: ?>>
-///
-///   // Non-scalar multidimensional copy with static offset. Result strides
-///   must
-///   // be static and identical to the identity strides of the source.
-///   copy memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32>
-///     to reinterpret_cast memref<source-shape, identity-layout>
-///       to memref<1 x ... x N_0 x ... x N_K x ... x 1 x f32,
-///                 strided<[source-identity-strides], offset: O>>
+///   reinterpret_cast memref<1xMxf32, identity-layout>
+///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   // Rank-changing reinterpret_casts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<Mxf32, identity-layout>
-///       to memref<1xNxf32, strided<[N, 1]>>
-///
-///   // Dynamic shapes are not supported.
-///   copy memref<?xNxf32>
-///     to reinterpret_cast memref<?xMxf32, identity-layout>
-///       to memref<?xNxf32, strided<[M, 1]>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
-///   // Non-identity source layouts are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, strided<[S, 1]>>
-///       to memref<1xNxf32, strided<[M, 1]>>
-///
-///   // Dynamic offset into a source with more than one non-unit dimension is
-///   // not supported because runtime delinearization is not implemented.
-///   copy memref<1x1xf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1x1xf32, strided<[?, ?], offset: ?>>
-///
-///   // Non-scalar copies with dynamic result strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[?, 1]>>
-///
-///   // Non-scalar copies with result strides different from the source
-///   identity
-///   // strides are not supported.
-///   copy memref<1xNxf32>
-///     to reinterpret_cast memref<1xMxf32, identity-layout>
-///       to memref<1xNxf32, strided<[S, 1]>>
-///
-///   // Multidimensional non-scalar copies with dynamic offset are not
-///   supported
-///   // unless the source is effectively 1D.
-///   copy memref<1xNxKxf32>
-///     to reinterpret_cast memref<1xNxMxf32, identity-layout>
-///       to memref<1xNxKxf32,
-///                 strided<[N*M, M, 1], offset: ?>>
+///   reinterpret_cast memref<1xNxMxf32, identity-layout>
+///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -219,11 +150,12 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
   assert(resType.hasStaticShape() && "expected static shape");
+
+  bool isScalarCopy =
+      llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
+
   // For scalar copies, result strides are irrelevant, including dynamic ones.
-  // For non-scalar copies, require static result strides identical to the
-  // identity strides of the reinterpret_cast source.
-  if (!llvm::all_of(resType.getShape(),
-                    [](int64_t size) { return size == 1; })) {
+  if (!isScalarCopy) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -231,7 +163,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
     assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
            "Expecting same number of strides for rank-preserving "
            "reinterpret_casts.");
-
+    // For non-scalar copies, require static result strides identical to the
+    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -239,6 +172,8 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
+    // Track result dimensions that produce varying indices; unit dimensions are
+    // always indexed at 0.
     for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
       if (resultSize != 1)
         dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 2f4be0a011d54..913cabbf9280f 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -2,15 +2,16 @@
 // RUN: | FileCheck %s
 
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Scalar (0D) copy
+//
+// No varying RC result dimensions =>
+//   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
 // The destination is effectively a scalar within a MemRef with rank == 0 
-// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided(
+// CHECK-LABEL: func.func private @copy_scalar_into_0D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<f32>, %[[DST:.*]]: memref<f32>
-func.func private @copy_scalar_into_0D_strided(%src : memref<f32>, %dst : memref<f32>) {
+func.func private @copy_scalar_into_0D_strided_zero_offset(%src : memref<f32>, %dst : memref<f32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [], strides: []
@@ -47,6 +48,25 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
   return
 }
 
+/// Reject non-identity layout rc source strides
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+  %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32, strided<[54, 2]>>
+      to memref<1x1xf32, strided<[54, 2]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
+
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
@@ -95,33 +115,31 @@ func.func private @copy_scalar_into_1D_strided_dynamic_offset(%offset: index, %s
   return
 }
 
-// Scalar copies have no varying result dimensions, so rc result strides do
-// not affect the copy destination address and are ignored.
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_non_identity_stride(%src : memref<1x1xf32>,
-  %dst : memref<1x108xf32>) {
+func.func private @copy_scalar_into_1D_strided_zero_offset_non_identity_stride(
+  %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %reinterpret_cast = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [2, 54]
-    : memref<1x108xf32> to memref<1x1xf32, strided<[2, 54]>>
+    to offset: [0], sizes: [1, 1], strides: [54, 2]
+    : memref<1x108xf32> to memref<1x1xf32, strided<[54, 2]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK:      %[[C0:.*]] = arith.constant 0 : index
   // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]]] : memref<1x1xf32>
   // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]]] : memref<1x108xf32>
   memref.copy %src, %reinterpret_cast
-    : memref<1x1xf32> to memref<1x1xf32, strided<[2, 54]>>
+    : memref<1x1xf32> to memref<1x1xf32, strided<[54, 2]>>
   return
 }
 
-// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_dynamic_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(
 // CHECK-SAME:   %[[STR0:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[STR1:[A-Za-z][A-Za-z0-9-]*]]: index
 // CHECK-SAME:   %[[SRC:[A-Za-z][A-Za-z0-9-]*]]: memref<1x1xf32>
 // CHECK-SAME:   %[[DST:[A-Za-z][A-Za-z0-9-]*]]: memref<1x108xf32>
-func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
+func.func private @copy_scalar_into_1D_strided_zero_offset_dynamic_stride(%stride0: index,
   %stride1: index, %src : memref<1x1xf32>, %dst : memref<1x108xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -141,10 +159,10 @@ func.func private @copy_scalar_into_1D_strided_dynamic_stride(%stride0: index,
 }
 
 /// The destination is effectively a 2D array within a MemRef with rank >= 2 
-// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_scalar_into_2D_strided_non_identity_stride(
+func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -161,9 +179,26 @@ func.func private @copy_scalar_into_2D_strided_non_identity_stride(
   return
 }
 
+/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
+/// runtime delinearization of these offsets is TODO.
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [%offset], sizes: [1, 1, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32> to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[33, 11, 1], offset: ?>>
+  return
+}
+
 //===----------------------------------------------------------------------===//
-// Positive tests
-// 
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -220,6 +255,72 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
+func.func private @copy_1D_into_2D_strided_delinearized_offset(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Reject rc result strides that not equal to rc source identity strides.
+/// (non-unit copied dimension needs stride-based address computation)
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, 4, 4]>>
+  return
+}
+
+/// Reject dynamic rc result strides.
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(
+func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%stride : index,
+  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
+    : memref<1x3x4xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[12, ?, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -309,9 +410,7 @@ func.func private @copy_1D_into_1D_strided_dynamic_offset(
   return
 }
 
-//===----------------------------------------------------------------------===//
-// Negative tests (must NOT rewrite)
-// 
+//===----------------------------------------------------------------------===// 
 // Either scalar (0D) OR non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
@@ -361,25 +460,6 @@ func.func private @negative_copy_into_strided_rank_change(%src : memref<3x4xf32>
   return
 }
 
-/// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity(
-func.func private @negative_copy_into_strided_non_identity(%src: memref<1x1xf32>,
-  %dst: memref<12x1xf32, strided<[10, 2]>>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 1], strides: [10, 2]
-    : memref<12x1xf32, strided<[10, 2]>>
-      to memref<1x1xf32, strided<[10, 2]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32> to memref<1x1xf32, strided<[10, 2]>>
-
-  return
-}
-
 /// Reject dynamic shapes
 // CHECK-LABEL: func.func private @negative_copy_into_strided_dynamic_copy_source_shape(
 func.func private @negative_copy_into_strided_dynamic_copy_source_shape(%src : memref<?xf32>,
@@ -429,70 +509,6 @@ func.func private @negative_copy_into_strided_dynamic_rc_result_shape(%dim : ind
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
-// CHECK-LABEL: func.func private @negative_copy_into_ND_strided_dynamic_offset(
-func.func private @negative_copy_into_ND_strided_dynamic_offset(
-  %offset : index, %src : memref<1x1xf32>, %dst : memref<3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [1, 1], strides: [12, 1]
-    : memref<3x4xf32> to memref<1x1xf32, strided<[12, 1], offset: ?>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x1xf32>
-      to memref<1x1xf32, strided<[12, 1], offset: ?>>
-  return
-}
-
-//===----------------------------------------------------------------------===//
-// Negative tests
-// 
-// Non-scalar (ND) copy
-//===----------------------------------------------------------------------===//
-
-/// Reject rc result strides that not equal to rc source identity strides.
-/// (non-unit copied dimension needs stride-based address computation)
-// CHECK-LABEL: func.func private @negative_copy_into_strided_non_identity_strides(
-func.func private @negative_copy_into_strided_non_identity_strides(
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, 4, 4]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, 4, 4]>>
-  return
-}
-
-/// Reject dynamic rc result strides.
-// CHECK-LABEL: func.func private @negative_copy_ND_into_strided_dynamic_stride(
-func.func private @negative_copy_ND_into_strided_dynamic_stride(%stride : index,
-  %src : memref<1x3x1xf32>, %dst : memref<1x3x4xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 1], strides: [12, %stride, 1]
-    : memref<1x3x4xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-
-  // CHECK:      memref.copy %arg1, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[12, ?, 1]>>
-  return
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//

>From 3da22cbf4ce36e92473f9a5bd34f2b9ef1db7392 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 3 Jul 2026 12:19:31 +0200
Subject: [PATCH 12/17] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 .../MemRef/elide-reinterpret-cast.mlir        | 12 +++--
 2 files changed, 36 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index ba342ffe8c655..307f95f7d0a82 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -107,7 +107,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
 /// identity-layout sources. Non-scalar results must have static strides
 /// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources.
+/// for effectively-1D sources. Returns nullopt for unsupported
+/// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
@@ -123,7 +124,8 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 ///     to memref<1xNxKxf32, strided<[?, M, 1]>>
 ///
 ///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[N*M, M, K]>>
+///     to memref<1xNx1xf32, strided<[K, M, L]>>
+///       ( identity-layout != [K, M, L] )
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -151,20 +153,23 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   assert(resType.hasStaticShape() && "expected static shape");
 
-  bool isScalarCopy =
+  // Track result dimensions that produce varying indices; unit dimensions are
+  // always indexed at 0.
+  for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
+    if (resultSize != 1)
+      dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
+  }
+
+  bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For scalar copies, result strides are irrelevant, including dynamic ones.
-  if (!isScalarCopy) {
+  // For non-scalar results, verify that strides do not introduce
+  // non-contiguity that would require extra logic.
+  if (!isScalarRes) {
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
 
-    assert((srcIdentityStrides.size() == rcResultStrides.size()) &&
-           "Expecting same number of strides for rank-preserving "
-           "reinterpret_casts.");
-    // For non-scalar copies, require static result strides identical to the
-    // identity strides of the reinterpret_cast source.
     if (!llvm::all_of(llvm::zip_equal(srcIdentityStrides, rcResultStrides),
                       [](auto pair) {
                         auto [srcStride, resultStride] = pair;
@@ -172,14 +177,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-    // Track result dimensions that produce varying indices; unit dimensions are
-    // always indexed at 0.
-    for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
-      if (resultSize != 1)
-        dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
-    }
   }
 
+  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
+
+  // A source with exactly one non-unit dimension cannot be indexed directly by
+  // multiple non-unit result dimensions.
+  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
+    return std::nullopt;
+
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
   // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
   // only a single offset. That should be fixed at the op definition level.
@@ -194,8 +200,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
            "static reinterpret_cast offset must delinearize to in-bounds "
            "reinterpret_cast source indices");
 
-    // Relevant for non-scalar copies: assert that the rectangular
-    // copied slice is in bounds.
+    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
+    // MemRef. Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
     assert(llvm::all_of(llvm::enumerate(resType.getShape()),
                         [&](auto it) {
                           unsigned dim = it.index();
@@ -213,16 +220,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // dimensions by delinearizing the offset into source start indices at runtime
   // before adding loop IVs.
 
-  // With an effectively-1D source, a dynamic linear offset can be used directly
-  // as the index of the unique non-unit source dimension.
-  if (!getSingleNonUnitDim(srcType))
+  // With an effectively-1D source, a dynamic offset can be mapped to the unique
+  // non-unit dim. For other cases, bail out.
+  if (!srcNonUnitDim)
     return std::nullopt;
 
-  // Non-scalar copies require identical strides and no rank-changing,
-  // so there can be at most one non-unit result dimension in this case.
-  assert(dimsAndOffs.nonUnitDimsPos.size() <= 1 &&
-         "effectively-1D source cannot have multiple non-unit result dims");
-
   return dimsAndOffs;
 }
 
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 913cabbf9280f..8dce4fd63fb97 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -228,10 +228,12 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
+/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -255,10 +257,12 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_delinearized_offset(
+/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
+/// both the looped source dimension and the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_delinearized_offset(
+func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst

>From 16b1cbfaad45996de5f21a46217810a73cc83728 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 7 Jul 2026 17:26:24 +0200
Subject: [PATCH 13/17] Constrain RC to either scalar OR single non-unit dim
 collapsed result

---
 .../Transforms/ElideReinterpretCast.cpp       | 179 +++++++------
 .../MemRef/elide-reinterpret-cast.mlir        | 242 ++++++++++++------
 2 files changed, 268 insertions(+), 153 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 307f95f7d0a82..aa760e71b6aae 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,35 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-/// Returns the unique non-unit dim or nullopt of # non-unit-dims != 1.
+static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
+                                             MemRefType resType) {
+  assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
+         "expected static shapes");
+  assert(srcType.getRank() == resType.getRank() &&
+         "expected rank-preserving reinterpret_cast");
+
+  unsigned collapsedDims = 0;
+
+  for (auto [srcSize, resSize] :
+       llvm::zip_equal(srcType.getShape(), resType.getShape())) {
+    if (srcSize == resSize)
+      continue;
+
+    // Only allow collapsing one non-unit source dim to a unit result dim.
+    if (srcSize != 1 && resSize == 1) {
+      ++collapsedDims;
+      continue;
+    }
+
+    // The sizes differ and both of them are non-unit - ATM not supported.
+    return false;
+  }
+
+  // Make sure there is only one collapsed dimension.
+  return collapsedDims == 1;
+}
+
+/// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
 static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   assert(type.hasStaticShape() && "expected static shape");
   ArrayRef<int64_t> shape = type.getShape();
@@ -105,27 +133,27 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// the corresponding source indices.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Non-scalar results must have static strides
-/// identical to the source identity strides. Dynamic offsets are supported only
-/// for effectively-1D sources. Returns nullopt for unsupported
+/// identity-layout sources. Scalar-shaped results may have arbitrary result
+/// strides. Non-scalar results must have static offsets, static result strides
+/// identical to the source identity strides, and exactly one non-unit source
+/// dimension collapsed to unit size. Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[N*M, M, 1], offset: O>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
 ///
 /// Examples that return no info:
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNxKxf32, strided<[?, M, 1]>>
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xMx1xf32, strided<[?, N, 1]>>
 ///
-///   reinterpret_cast memref<1xNxMxf32, identity-layout>
-///     to memref<1xNx1xf32, strided<[K, M, L]>>
-///       ( identity-layout != [K, M, L] )
+///   reinterpret_cast memref<1xMxNxf32, identity-layout>
+///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -136,7 +164,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   if (!srcType || !resType)
     return std::nullopt;
 
-  // TODO: Support rank-modifying reinterpret_casts
+  // TODO: Support rank-modifying reinterpret_casts.
   if (srcType.getRank() != resType.getRank())
     return std::nullopt;
 
@@ -151,8 +179,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  assert(resType.hasStaticShape() && "expected static shape");
-
   // Track result dimensions that produce varying indices; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
@@ -160,12 +186,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
       dimsAndOffs.nonUnitDimsPos.push_back(static_cast<unsigned>(dim));
   }
 
+  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
+  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
+  // only a single offset. That should be fixed at the op definition level.
+  assert(rcOffsets.size() == 1 && "Expecting single offset");
+
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
-  // For non-scalar results, verify that strides do not introduce
-  // non-contiguity that would require extra logic.
   if (!isScalarRes) {
+    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
+    // views with static metadata.
+    if (ShapedType::isDynamic(rcOffsets[0]))
+      return std::nullopt;
+
     SmallVector<int64_t> srcIdentityStrides =
         computeStrides(srcType.getShape());
     ArrayRef<int64_t> rcResultStrides = rc.getStaticStrides();
@@ -177,53 +211,32 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                                srcStride == resultStride;
                       }))
       return std::nullopt;
-  }
-
-  std::optional<unsigned> srcNonUnitDim = getSingleNonUnitDim(srcType);
 
-  // A source with exactly one non-unit dimension cannot be indexed directly by
-  // multiple non-unit result dimensions.
-  if (srcNonUnitDim && dimsAndOffs.nonUnitDimsPos.size() > 1)
-    return std::nullopt;
-
-  ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();
-  // FIXME: Despite what `getStaticOffsets` implies, `reinterpret_cast` takes
-  // only a single offset. That should be fixed at the op definition level.
-  assert(rcOffsets.size() == 1 && "Expecting single offset");
+    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+      return std::nullopt;
+  }
 
-  // CASE 1: Static ReinterpretCast offset
-  if (ShapedType::isStatic(rcOffsets[0])) {
-    // Delinearize static ReinterpretCast offset as in-bounds indices (one for
-    // every source dimension).
-    dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-    assert(dimsAndOffs.delinearizedOffsets &&
-           "static reinterpret_cast offset must delinearize to in-bounds "
-           "reinterpret_cast source indices");
+  // CASE 1: Dynamic ReinterpretCast offset.
+  //
+  // Dynamic offsets are supported only for scalar-shaped results, under the
+  // previous effectively-1D source restriction.
+  if (ShapedType::isDynamic(rcOffsets[0])) {
+    // With an effectively-1D source, a dynamic offset can be mapped to its
+    // unique non-unit dim. For other cases, bail out.
+    if (llvm::count_if(srcType.getShape(),
+                       [](int64_t size) { return size != 1; }) != 1)
+      return std::nullopt;
 
-    // Sanity check that the reinterpret_cast doesn't create an out-of-bounds
-    // MemRef. Such cases should probably be rejected by Op verifier.
-    // FIXME: Add run-time verification for cases like this.
-    assert(llvm::all_of(llvm::enumerate(resType.getShape()),
-                        [&](auto it) {
-                          unsigned dim = it.index();
-                          int64_t resultSize = it.value();
-                          return (*dimsAndOffs.delinearizedOffsets)[dim] +
-                                     resultSize <=
-                                 srcType.getDimSize(dim);
-                        }) &&
-           "reinterpret_cast metadata describes an invalid accessible region");
     return dimsAndOffs;
   }
 
-  // CASE 2: Dynamic ReinterpretCast offset.
-  // TODO: Support dynamic offsets into sources with multiple non-unit
-  // dimensions by delinearizing the offset into source start indices at runtime
-  // before adding loop IVs.
-
-  // With an effectively-1D source, a dynamic offset can be mapped to the unique
-  // non-unit dim. For other cases, bail out.
-  if (!srcNonUnitDim)
-    return std::nullopt;
+  // CASE 2: Static ReinterpretCast offset
+  // Delinearize static ReinterpretCast offset as in-bounds indices (one for
+  // every source dimension).
+  dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
+  assert(dimsAndOffs.delinearizedOffsets &&
+         "static reinterpret_cast offset must delinearize to in-bounds "
+         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -231,37 +244,36 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 /// Rewrites supported copy operations through `memref.reinterpret_cast` to
 /// scalar load/store operations.
 ///
-///   // BEFORE (scalar copy)
-///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, ..., 1], strides: [...]
-///   memref.copy %src, %strided
+/// Supported cases:
+///   1. Scalar-shaped reinterpret_cast results. Result strides are ignored;
+///      the store index is derived from the reinterpret_cast offset.
 ///
-///   // AFTER
-///   %v = memref.load %src[0, ..., 0]
-///   memref.store %v, %dst[delinearized(O)]
+///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
+///      dimensions except one collapsed-to-unit dimension. Result strides must
+///      be static and identical to the identity strides of the source, and the
+///      static offset selects the collapsed dimension.
 ///
-///   // BEFORE (effectively-1D copy)
+///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, 1], strides: [...]
+///     to offset: [OFF], sizes: [1, ..., 1], strides: [...]
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[delinearized(O) + mapped(%i)]
-///   }
+///   %v = memref.load %src[0, ..., 0]
+///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (multidimensional copy with static offset)
+///   // BEFORE (one collapsed non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [O], sizes: [1, N, K], strides: [...]
+///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     : memref<1xMxNxf32>
+///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   scf.for %i = 0 to N step 1 {
-///     scf.for %j = 0 to K step 1 {
-///       %v = memref.load %src[0, %i, %j]
-///       memref.store %v, %dst[delinearized(O) + mapped(%i, %j)]
-///     }
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   scf.for %i = 0 to M step 1 {
+///     %v = memref.load %src[0, %i, 0]
+///     memref.store %v, %dst[0, %i, OFF]
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
@@ -292,6 +304,19 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
     MemRefType dstType = cast<MemRefType>(dst.getType());
     MemRefType rcResType = cast<MemRefType>(rc.getType());
 
+    // Sanity check that the copy doesn't access strided MemRef out-of-bounds.
+    // Such cases should probably be rejected by Op verifier.
+    // FIXME: Add run-time verification for cases like this.
+    if (ShapedType::isStatic(rc.getStaticOffsets()[0]) &&
+        llvm::any_of(llvm::enumerate(rcResType.getShape()), [&](auto it) {
+          unsigned dim = it.index();
+          int64_t rcResultSize = it.value();
+          return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
+                 dstType.getDimSize(dim);
+        }))
+      return rewriter.notifyMatchFailure(
+          op, "copy accesses invalid accessible region");
+
     // Reuse common index constants across bounds, steps, and static offsets,
     // but avoid creating them for rank-0 copies.
     std::array<Value, 2> cachedIndexConstants;
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 8dce4fd63fb97..94d8f2a5f543c 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -179,8 +179,59 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Reject dynamic offsets for rc sources with > 1 non-unit dimension -
-/// runtime delinearization of these offsets is TODO.
+/// Offset delinearized to [0, 0, 10], therefore is only
+/// added to the trailing source dimension.
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [10], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[OFF]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 10>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// Offset delinearized into more dimensions: [0, 2, 1]
+// CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
+    %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [23], sizes: [1, 1, 1], strides: [1, 1, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[C2:.*]] = arith.constant 2 : index
+  // CHECK:      %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[C0]]] : memref<1x1x1xf32>
+  // CHECK:      memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C2]], %[[C1]]] : memref<1x3x11xf32>
+  memref.copy %src, %rc
+    : memref<1x1x1xf32>
+      to memref<1x1x1xf32, strided<[1, 1, 1], offset: 23>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// rc result dynamic offset:
+///    supported only for effectively-1D rc source
+///    (runtime delinearization not implemented)
 // CHECK-LABEL: func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
   %offset : index, %src : memref<1x1x1xf32>, %dst : memref<1x3x11xf32>) {
@@ -202,6 +253,23 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
+func.func private @negative_copy_1D_into_1D_strided(
+  %src : memref<4xf32>, %dst : memref<108xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [4], strides: [1]
+    : memref<108xf32> to memref<4xf32, strided<[1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<4xf32> to memref<4xf32, strided<[1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -228,12 +296,38 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
-/// Offset 10 delinearizes to [0, 0, 10], therefore is only
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
+  %src : memref<1x1x11xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 1, 11], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 11 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x1x11xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[C0]], %[[IDX]]] : memref<1x3x11xf32>
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x1x11xf32>
+      to memref<1x1x11xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+/// Offset delinearized to [0, 0, 10], therefore is only
 /// added to the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
+func.func private @copy_1D_into_2D_strided_nonzero_offset(
   %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -257,36 +351,25 @@ func.func private @copy_1D_into_2D_strided_nonzero_offset_delinierized_v1(
   return
 }
 
-/// Offset 12 delinearizes to [0, 1, 1], therefore is split across
-/// both the looped source dimension and the trailing source dimension.
-// CHECK-LABEL: func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x1xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x4x11xf32>
-func.func private @copy_1D_into_2D_strided_nonzero_offset_delinearized_v2(
-  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+func.func private @negative_copy_1D_into_2D_strided_dynamic_offset(
+  %offset : index, %src : memref<1x3x1xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg2
   %rc = memref.reinterpret_cast %dst
-    to offset: [12], sizes: [1, 3, 1], strides: [44, 11, 1]
-    : memref<1x4x11xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
+    to offset: [%offset], sizes: [1, 3, 1], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 3 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[C1]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX]], %[[C0]]] : memref<1x3x1xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[C1]]] : memref<1x4x11xf32>
-  // CHECK:      }
+  // CHECK:      memref.copy %arg1, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x1xf32>
-      to memref<1x3x1xf32, strided<[44, 11, 1], offset: 12>>
-  // CHECK-NOT:  memref.copy
+      to memref<1x3x1xf32, strided<[33, 11, 1], offset: ?>>
   return
 }
 
-/// Reject rc result strides that not equal to rc source identity strides.
+/// Reject rc result strides that are not equal to rc source identity strides.
 /// (non-unit copied dimension needs stride-based address computation)
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
 func.func private @negative_copy_1D_into_2D_strided_zero_offset_non_identity_strides(
@@ -325,91 +408,98 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+  %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 1], strides: [44, 11, 1]
+    : memref<1x4x11xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<1x3x1xf32>
+      to memref<1x3x1xf32, strided<[44, 11, 1]>>
+  return
+}
+
+/// No non-unit dimension collapsed
+// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
+func.func private @negative_copy_2D_into_2D_strided(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK-NOT:  memref.reinterpret_cast
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
     : memref<1x3x11xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
 
-  // CHECK-NOT:  memref.copy
-  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
-  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
-  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
-  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
-  // CHECK:        }
-  // CHECK:      }
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
   memref.copy %src, %rc
     : memref<1x3x4xf32>
       to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  // CHECK-NOT:  memref.copy
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
-// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_zero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [7], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    to offset: [0], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
   // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
   // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
-  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 7 : index
   // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
   // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
-  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
-  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
-  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x11xf32>
   // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 7>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1]>>
   // CHECK-NOT:  memref.copy
   return
 }
 
-/// rc result dynamic offset:
-///    supported only for effectively-1D rc source
-///    (runtime delinearization not implemented)
-// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_dynamic_offset(
-// CHECK-SAME:   %[[OFF:.*]]: index
-// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
-// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
-func.func private @copy_1D_into_1D_strided_dynamic_offset(
-  %offset : index, %src : memref<4xf32>, %dst : memref<108xf32>) {
+// CHECK-LABEL: func.func private @copy_2D_into_3D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>
+func.func private @copy_2D_into_3D_strided_nonzero_offset(
+  %src : memref<3x1x4x1xf32>, %dst : memref<3x1x4x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
-    to offset: [%offset], sizes: [4], strides: [1]
-    : memref<108xf32> to memref<4xf32, strided<[1], offset: ?>>
+    to offset: [10], sizes: [3, 1, 4, 1], strides: [44, 44, 11, 1]
+    : memref<3x1x4x11xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
 
   // CHECK-NOT:  memref.copy
   // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
   // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
-  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
-  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
-  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX]] : index
-  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
-  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[DST_IDX]]] : memref<108xf32>
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 10 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[C0]]] : memref<3x1x4x1xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[IDX0]], %[[C0]], %[[IDX1]], %[[OFF]]] : memref<3x1x4x11xf32>
+  // CHECK:        }
   // CHECK:      }
   memref.copy %src, %rc
-    : memref<4xf32> to memref<4xf32, strided<[1], offset: ?>>
+    : memref<3x1x4x1xf32>
+      to memref<3x1x4x1xf32, strided<[44, 44, 11, 1], offset: 10>>
   // CHECK-NOT:  memref.copy
   return
 }

>From d530b8711be6a272325b1a63395d0ad0ea97b769 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:13:32 +0200
Subject: [PATCH 14/17] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       | 54 ++++++++++---------
 1 file changed, 30 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index aa760e71b6aae..6a7ab58c3b980 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,8 +84,9 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(MemRefType srcType,
-                                             MemRefType resType) {
+static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+  MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
+  MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
          "expected static shapes");
   assert(srcType.getRank() == resType.getRank() &&
@@ -129,14 +130,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
   return (*nonUnitDims.begin()).index();
 }
 
-/// Returns reinterpret_cast result non-unit dimensions and, for static offsets,
-/// the corresponding source indices.
+/// Returns reinterpret_cast's result non-unit dimensions and, for static
+/// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. Scalar-shaped results may have arbitrary result
-/// strides. Non-scalar results must have static offsets, static result strides
-/// identical to the source identity strides, and exactly one non-unit source
-/// dimension collapsed to unit size. Returns nullopt for unsupported
+/// identity-layout sources. In addition:
+///     identical to the source identity strides, and exactly one non-unit
+///     source
+///  * Non-scalar results must have static offsets, static result strides
+///     dimension collapsed to unit size
+/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
+/// strides are effectively irrelevant).
+///
+/// Returns nullopt for unsupported
 /// reinterpret_casts.
 ///
 /// Examples that return info:
@@ -179,7 +185,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 
   ResultNonUnitDimsAndOffsetsForRC dimsAndOffs;
 
-  // Track result dimensions that produce varying indices; unit dimensions are
+  // Track non-unit result dimensions; unit dimensions are
   // always indexed at 0.
   for (auto [dim, resultSize] : llvm::enumerate(resType.getShape())) {
     if (resultSize != 1)
@@ -194,10 +200,13 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   bool isScalarRes =
       llvm::all_of(resType.getShape(), [](int64_t size) { return size == 1; });
 
+  bool isOffsetDynamic = ShapedType::isDynamic(rcOffsets[0]);
+
+  // Cases with at least one non-unit dimension in reinterpret_cast's result are
+  // restricted to preserving all but one dimension from the source, which
+  // collapsed to `1` in the result, and fully static metadata.
   if (!isScalarRes) {
-    // Non-scalar cases are restricted to same-dimension, one-collapsed-dim
-    // views with static metadata.
-    if (ShapedType::isDynamic(rcOffsets[0]))
+    if (isOffsetDynamic)
       return std::nullopt;
 
     SmallVector<int64_t> srcIdentityStrides =
@@ -212,15 +221,15 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(srcType, resType))
+    if (!hasExactlyOneCollapsedNonUnitDim(rc))
       return std::nullopt;
   }
 
   // CASE 1: Dynamic ReinterpretCast offset.
   //
-  // Dynamic offsets are supported only for scalar-shaped results, under the
-  // previous effectively-1D source restriction.
-  if (ShapedType::isDynamic(rcOffsets[0])) {
+  // Dynamic offsets are supported only for effectively-1D to scalar
+  // reinterpret_casts.
+  if (isOffsetDynamic) {
     // With an effectively-1D source, a dynamic offset can be mapped to its
     // unique non-unit dim. For other cases, bail out.
     if (llvm::count_if(srcType.getShape(),
@@ -234,9 +243,6 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   // Delinearize static ReinterpretCast offset as in-bounds indices (one for
   // every source dimension).
   dimsAndOffs.delinearizedOffsets = delinearizeStaticRCOffset(rc);
-  assert(dimsAndOffs.delinearizedOffsets &&
-         "static reinterpret_cast offset must delinearize to in-bounds "
-         "reinterpret_cast source indices");
 
   return dimsAndOffs;
 }
@@ -297,7 +303,8 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         getResultNonUnitDimsAndOffsetsForRC(rc);
     if (!dimsAndOffs)
       return rewriter.notifyMatchFailure(
-          op, "reinterpret_cast does not match scalar or loop copy region");
+          op,
+          "unsupported reinterpret_cast result dimensions, strides, or offset");
 
     Location loc = op.getLoc();
     Value dst = rc.getSource();
@@ -314,11 +321,10 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           return (*dimsAndOffs->delinearizedOffsets)[dim] + rcResultSize >
                  dstType.getDimSize(dim);
         }))
-      return rewriter.notifyMatchFailure(
-          op, "copy accesses invalid accessible region");
+      return rewriter.notifyMatchFailure(op, "copy accesses are OOB");
 
-    // Reuse common index constants across bounds, steps, and static offsets,
-    // but avoid creating them for rank-0 copies.
+    // Constant Op cache to reuse common index constants across bounds, steps,
+    // and static offsets: 0 is stored at index 0 and 1 is stored at index 1.
     std::array<Value, 2> cachedIndexConstants;
     auto getOrCreateIndexConstant = [&](int64_t value) -> Value {
       if (value == 0 || value == 1) {

>From 5a0dc9846d1d919cc90082b9c2784167d6b8e39a Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 14:59:34 +0200
Subject: [PATCH 15/17] Fixup

---
 mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 6a7ab58c3b980..d9952141dfee9 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -44,8 +44,8 @@ struct ResultNonUnitDimsAndOffsetsForRC {
   std::optional<SmallVector<int64_t>> delinearizedOffsets;
 };
 
-/// Returns source indices for a static reinterpret_cast offset of an
-/// identity-layout source.
+/// Returns delinearized offset indices for a static reinterpret_cast offset of
+/// an identity-layout source.
 static std::optional<SmallVector<int64_t>>
 delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   ArrayRef<int64_t> rcOffsets = rc.getStaticOffsets();

>From 7d62120b505e35036ec1ebc2a7ebc821facc0283 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 16:50:35 +0200
Subject: [PATCH 16/17] Fixup

---
 .../Transforms/ElideReinterpretCast.cpp       |  4 +-
 .../MemRef/elide-reinterpret-cast.mlir        | 40 ++++++++-----------
 2 files changed, 19 insertions(+), 25 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index d9952141dfee9..a24a0ccaa3d67 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -353,6 +353,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
           getOrCreateIndexConstant(rcResType.getDimSize(dim)));
     }
 
+    // All indices are initialised to zero.
     SmallVector<Value> rcSrcStoreIdxs = getZeroIdxs(dstType.getRank());
     std::optional<unsigned> srcNonUnitDimPos;
     if (dimsAndOffs->delinearizedOffsets) {
@@ -365,7 +366,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
         rcSrcStoreIdxs[idx] = getOrCreateIndexConstant(offset);
       }
     } else {
-      // Dynamic offsets are used directly only for effectively-1D sources.
+      // Dynamic offset is used directly only for effectively-1D sources.
       assert(dimsAndOffs->nonUnitDimsPos.size() <= 1 &&
              "Expecting at most one non-unit result dimension.");
 
@@ -378,6 +379,7 @@ struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
       // FIXME: Despite what `getMixedOffsets` implies, `reinterpret_cast` takes
       // only a single offset. That should be fixed at the op definition level.
       assert(rcOffsets.size() == 1 && "Expecting single offset");
+      // Only the index corresponding to the single non-unit dim is updated.
       rcSrcStoreIdxs[*srcNonUnitDimPos] =
           getValueOrCreateConstantIndexOp(rewriter, loc, rcOffsets[0]);
     }
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 94d8f2a5f543c..c6818b29e271e 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -49,8 +49,8 @@ func.func private @copy_scalar_into_1D_strided_zero_offset(%src : memref<1x1xf32
 }
 
 /// Reject non-identity layout rc source strides
-// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
-func.func private @negative_copy_scalar_into_1D_strided_zero_offset_non_identity_layout(
+// CHECK-LABEL: func.func private @negative_copy_scalar_into_1D_strided_zero_offset_base_non_identity_layout(
+func.func private @negative_copy_scalar_into_1D_strided_zero_offset_base_non_identity_layout(
   %src: memref<1x1xf32>, %dst: memref<1x108xf32, strided<[54, 2]>>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -179,8 +179,12 @@ func.func private @copy_scalar_into_2D_strided_zero_offset_non_identity_stride(
   return
 }
 
-/// Offset delinearized to [0, 0, 10], therefore is only
-/// added to the trailing source dimension.
+/// %dst has identity strides [33, 11, 1].
+/// Offset 10 delinearizes as:
+///   dim 0: 10 / 33 = 0, remainder 10
+///   dim 1: 10 / 11 = 0, remainder 10
+///   dim 2: 10 /  1 = 10, remainder 0
+/// Therefore the scalar is stored at %dst[0, 0, 10].
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -204,7 +208,12 @@ func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearize
   return
 }
 
-// Offset delinearized into more dimensions: [0, 2, 1]
+/// %dst has identity strides [33, 11, 1].
+/// Offset 23 delinearizes as:
+///   dim 0: 23 / 33 = 0, remainder 23
+///   dim 1: 23 / 11 = 2, remainder 1
+///   dim 2:  1 /  1 = 1, remainder 0
+/// Therefore the scalar is stored at %dst[0, 2, 1].
 // CHECK-LABEL: func.func private @copy_scalar_into_2D_scalar_strided_nonzero_offset_delinearized_v2(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -296,6 +305,7 @@ func.func private @copy_1D_into_2D_strided_zero_offset(
   return
 }
 
+/// Copied non-unit dimension is the right-most dimension, therefore the loop indices are placed there.
 // CHECK-LABEL: func.func private @copy_1D_into_2D_strided_zero_offset_loop_trailing_dim(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x1x11xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -408,6 +418,7 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
+/// 4 is collapsed to a non-unit dimension.
 // CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
 func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
@@ -426,25 +437,6 @@ func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   return
 }
 
-/// No non-unit dimension collapsed
-// CHECK-LABEL: func.func private @negative_copy_2D_into_2D_strided(
-func.func private @negative_copy_2D_into_2D_strided(
-  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
-  %rc = memref.reinterpret_cast %dst
-    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
-    : memref<1x3x11xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1]>>
-
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
-  memref.copy %src, %rc
-    : memref<1x3x4xf32>
-      to memref<1x3x4xf32, strided<[33, 11, 1]>>
-  return
-}
-
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>

>From 0cacbd3f31cf1e000b7686eb4a8d6fc0d856006e Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 13 Jul 2026 15:11:29 +0200
Subject: [PATCH 17/17] Address final comments

---
 mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index c6818b29e271e..0955d9638f093 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -4,7 +4,7 @@
 //===----------------------------------------------------------------------===//
 // Scalar (0D) copy
 //
-// No varying RC result dimensions =>
+// All RC result dimensions are unit (1) =>
 //   RC result strides do not affect copy destination address and are ignored.
 //===----------------------------------------------------------------------===//
 
@@ -262,7 +262,7 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// No non-unit dimension collapsed
+/// The result of collapsing dimensions is not a unit dimension.
 // CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
 func.func private @negative_copy_1D_into_1D_strided(
   %src : memref<4xf32>, %dst : memref<108xf32>) {


From llvmlistbot at llvm.org  Mon Jul 13 06:15:40 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:15:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a54e4fc.a1accbb4.244acd.8a7b@mx.google.com>


================

----------------
banach-space wrote:

The simplified version LGTM, thank you!

https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Mon Jul 13 06:23:38 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Mon, 13 Jul 2026 06:23:38 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a54e6da.481afadf.196d57.f7fa@mx.google.com>


https://github.com/tblah commented:

Thank you very much for all of the testing

https://github.com/llvm/llvm-project/pull/202315

From llvmlistbot at llvm.org  Mon Jul 13 06:23:38 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Mon, 13 Jul 2026 06:23:38 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a54e6da.c574b9b4.21ead0.428a@mx.google.com>


================
@@ -5370,6 +5370,99 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Result of matching the decomposed complex equality pattern inside an atomic
+/// compare region.
+struct ComplexComparePattern {
+  bool isComplex = false;
+  bool isNE = false;         // `or` of the field compares => NE (unsupported).
+  mlir::Value eAggregate;    // The complex expected value (`e`).
+  bool isXBinopExpr = false; // True if x is the first fcmp operand.
+};
+
+/// Detect a decomposed complex equality comparison in an atomic compare region:
+///   %re_x = llvm.extractvalue %xval[0]
+///   %re_e = llvm.extractvalue %eStruct[0]
+///   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
+///   %im_x = llvm.extractvalue %xval[1]
+///   %im_e = llvm.extractvalue %eStruct[1]
+///   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
+///   %cmp = llvm.and %cmp_re, %cmp_im   (llvm.or would be NE)
+/// It is recognised by an and/or whose operands are both fcmps operating on
+/// extractvalues, one chain rooted at the block argument (x) and the other at
+/// the expected complex value (e).
+static ComplexComparePattern detectComplexCompareEq(Block &block) {
+  ComplexComparePattern result;
+  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
+    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
+      return extractOp.getContainer();
+    return nullptr;
+  };
+  for (Operation &op : block.getOperations()) {
+    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
+      continue;
+    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
+    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
+    if (!lhsFcmp || !rhsFcmp)
+      continue;
+    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
+    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
+    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
+    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
+    if (!lhsXIsOp0 && !lhsXIsOp1)
+      continue;
+    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
+    if (!eAggregate)
+      continue;
+    result.isComplex = true;
+    result.isNE = isa<LLVM::OrOp>(op);
+    result.eAggregate = eAggregate;
+    result.isXBinopExpr = lhsXIsOp0;
+    break;
+  }
+  return result;
+}
+
+/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
+/// compare. The expected/desired complex values are spilled to allocas and
----------------
tblah wrote:

This has the same problem we discussed here: https://github.com/llvm/llvm-project/pull/184761#discussion_r3130007915

https://github.com/llvm/llvm-project/pull/202315

From llvmlistbot at llvm.org  Mon Jul 13 06:23:39 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Mon, 13 Jul 2026 06:23:39 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a54e6db.bd93e731.358318.0bd6@mx.google.com>


https://github.com/tblah edited https://github.com/llvm/llvm-project/pull/202315

From llvmlistbot at llvm.org  Mon Jul 13 06:38:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 06:38:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f88150e - [MLIR][XeGPU][VectorToXeGPU] Fixed
 transposed transfer_read/write (#197281)
Message-ID: <6a54ea70.e624eb8d.39cd4e.511d@mx.google.com>


Author: Andrey Pavlenko
Date: 2026-07-13T15:38:51+02:00
New Revision: f88150e1d5fc8591b0d71619d02e5b553161c1f8

URL: https://github.com/llvm/llvm-project/commit/f88150e1d5fc8591b0d71619d02e5b553161c1f8
DIFF: https://github.com/llvm/llvm-project/commit/f88150e1d5fc8591b0d71619d02e5b553161c1f8.diff

LOG: [MLIR][XeGPU][VectorToXeGPU] Fixed transposed transfer_read/write (#197281)

The problem was that adjustStridesForPermutation() permutes strides, but
they are required to compute base offset from transfer read indices.
Added optional argument to computeMemrefMeta() to preserve the original
strides.

Added: 
    

Modified: 
    mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
    mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 75ffbe58edd6c..9a994e87697f6 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -256,13 +256,8 @@ computeMemrefMeta(OpType xferOp, PatternRewriter &rewriter) {
       offsetVal = meta.getOffset();
   }
 
-  if constexpr (llvm::is_one_of<std::decay_t<OpType>, vector::TransferReadOp,
-                                vector::TransferWriteOp>::value) {
-    AffineMap permMap = xferOp.getPermutationMap();
-    // Adjust strides according to the permutation map (e.g., for transpose)
-    adjustStridesForPermutation(permMap, strides);
-  }
-
+  // Strides are returned in original memref order; permutation is applied in
+  // computeOffsets only where offsets are indexed in vector order.
   return {strides, offsetVal};
 }
 
@@ -312,13 +307,18 @@ static Value computeOffsets(VectorTransferOpInterface xferOp,
     return stepOp;
   });
 
+  // Local offsets are indexed in vector order, so permute strides; the base
+  // offset below uses the original memref-order strides.
+  SmallVector<Value> permutedStrides(strides.begin(), strides.end());
+  adjustStridesForPermutation(xferOp.getPermutationMap(), permutedStrides);
+
   // Multiply step vectors by corresponding strides
-  size_t memrefRank = strides.size();
+  size_t memrefRank = permutedStrides.size();
   size_t vectorRank = vectorShape.size();
   SmallVector<Value> strideMultiplied;
   for (size_t i = 0; i < vectorRank; ++i) {
     size_t memrefDim = memrefRank - vectorRank + i;
-    Value strideValue = strides[memrefDim];
+    Value strideValue = permutedStrides[memrefDim];
     auto mulType = dyn_cast<VectorType>(stepVectors[i].getType());
     auto bcastOp =
         vector::BroadcastOp::create(rewriter, loc, mulType, strideValue);

diff  --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index db7928724f082..3354868cd6cb9 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -703,3 +703,48 @@ gpu.func @load_0D_vector_unsupported(%source: memref<3xf32>,
 // LOAD-GATHER: vector.transfer_read
 
 }
+
+// -----
+gpu.module @xevm_module {
+gpu.func @transpose_1x1024x24x64(
+    %arg0: memref<1x1024x24x64xf16>,
+    %arg1: memref<1x24x1024x64xf16>) kernel
+    attributes {known_block_size = array<i32: 256, 1, 1>} {
+  %pad = ub.poison : f16
+  %c0 = arith.constant 0 : index
+  %block_id_x = gpu.block_id x
+  %block_id_y = gpu.block_id y
+  %block_id_z = gpu.block_id z
+  %seq_off = affine.apply affine_map<()[s0] -> (s0 * 16)>()[%block_id_y]
+  %hid_off = affine.apply affine_map<()[s0] -> (s0 * 8)>()[%block_id_z]
+  %vec = vector.transfer_read %arg0[%c0, %seq_off, %block_id_x, %hid_off], %pad
+    {in_bounds = [true, true, true, true]}
+    : memref<1x1024x24x64xf16>, vector<1x16x1x8xf16>
+  %transposed = vector.transpose %vec, [0, 2, 1, 3]
+    : vector<1x16x1x8xf16> to vector<1x1x16x8xf16>
+  vector.transfer_write %transposed, %arg1[%c0, %block_id_x, %seq_off, %hid_off]
+    {in_bounds = [true, true, true, true]}
+    : vector<1x1x16x8xf16>, memref<1x24x1024x64xf16>
+  gpu.return
+}
+
+// CHECK-LABEL: @transpose_1x1024x24x64
+// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
+// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+
+// Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
+// Scalar base offset: seq_off * 1536 (original dim1 stride),
+//                    block_id_x * 64  (original dim2 stride).
+// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
+// CHECK:     arith.muli %block_id_x, %[[C64]] : index
+// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+
+// Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
+// Scalar base offset: block_id_x * 65536 (original dim1 stride),
+//                    seq_off * 64        (original dim2 stride).
+// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
+// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
+// CHECK:     xegpu.store {{.*}}
+
+}


        

From llvmlistbot at llvm.org  Mon Jul 13 06:38:59 2026
From: llvmlistbot at llvm.org (Dmitry Chigarev)
Date: Mon, 13 Jul 2026 06:38:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU][VectorToXeGPU] Fixed transposed
 transfer_read/write (PR #197281)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/197281@github.com>
Message-ID: <6a54ea73.bb839515.27d21e.3116@mx.google.com>


https://github.com/dchigarev closed https://github.com/llvm/llvm-project/pull/197281

From llvmlistbot at llvm.org  Mon Jul 13 06:43:43 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Mon, 13 Jul 2026 06:43:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a54eb8f.481afadf.196d57.65d5@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 1f4d9bf3f135a032ffa4bff077cb1069cdfe0c9f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 01/14] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 4ec68510bfab58dea07a914a88960868c01f7fb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 02/14] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c9f1dc2d48cf6385d7fa22eef4e18b4750dabb70 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 03/14] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From a88b70384dc2dbfebc3dff61238efbb65eb6e1df Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 04/14] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From f0d134d38de12873a98a12810ed3f198aa49325e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 05/14] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From 1cb07564a0f3dd67cfdbc8a746e88f39d8e71286 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 06/14] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 2f506c83f24eda7f22f58019c7dee22751464f58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 07/14] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From 0752206c6433d322eba660ae2abd83491ecf067f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 08/14] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///

>From c6e524f3a855560ea31b282f0804da8906db6f6f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:35:29 -0700
Subject: [PATCH 09/14] Added create_named_contractions. Removing 2nd
 reduction_to_contract causes test to fail

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir        | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index d6a6bedc75a10..32984a04f6a99 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -252,9 +252,8 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
-    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
-    transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+      : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
     transform.apply_patterns to %loop_k {
@@ -270,7 +269,6 @@ module @transforms attributes { transform.with_named_sequence } {
     %func_pre = transform.structured.match ops{["func.func"]} in %module
       : (!transform.any_op) -> !transform.any_op
     transform.apply_patterns to %func_pre {
-      transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.transfer_permutation_patterns
       transform.apply_patterns.canonicalization
     } : !transform.any_op

>From 4f07f9de67b3d57d17dd64083b8f982453a60e1f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:47:50 -0700
Subject: [PATCH 10/14] Fix some comments

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir          | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 32984a04f6a99..06dae3ee89042 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -244,14 +244,14 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the M, N dim is scalable!)
+    // Tile parallel dims (note, the M, N dims are scalable!)
     %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
     %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
     transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
       : !transform.any_op
 

>From b244dc77fbf5fa74c4b257c0b4ea4c6fb8d52573 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 09:35:52 -0700
Subject: [PATCH 11/14] Make A_pack inner_tiles [%vscale_c8, 1]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir   | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 06dae3ee89042..7b8b3b4c74ff1 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -125,17 +125,21 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
   %A_pack = linalg.pack %A
     padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xf32>
+  return %A_pack : tensor<1x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -219,12 +223,12 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
 //===----------------------------------------------------------------------===//
 func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
   %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>

>From e71db1c7e119be8321bb88ee5079bce0b4909089 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 10:12:37 -0700
Subject: [PATCH 12/14] Make C_pack and C_out_unpack's inner tiles [%vs_c8,
 %vs_c8]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 +++++++++++--------
 1 file changed, 19 insertions(+), 14 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 7b8b3b4c74ff1..f0348c1679723 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -175,24 +175,26 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32
 
-  // Compute the outer tile size.
+  // Compute the outer tile sizes.
+  %c7 = arith.constant 7 : index
   %c13 = arith.constant 13 : index
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
-  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
+  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
   %C_pack = linalg.pack %C
     padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xf32>
+  return %C_pack : tensor<?x?x?x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -201,7 +203,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
@@ -210,8 +212,8 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
 
   return %C_out_unpack: tensor<7x13xf32>
 }
@@ -225,13 +227,13 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   // Pack input matrices
   %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
 
   return %C_out_unpack : tensor<7x13xf32>
 }
@@ -297,8 +299,11 @@ module @transforms attributes { transform.with_named_sequence } {
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
-    //    and other lower-level Ops (see step 2)
-    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+    //    and other lower-level Ops (see step 2). Tile by [8*vscale, 8*vscale]
+    //    (scalable, equal to the inner tile sizes) so the tiled outer dims are
+    //    statically 1 and DecomposeOuterUnitDimsUnPackOp can fire.
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [[8], [8]]
+        inner_tile_alignments = [Equal, Equal]
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 2.1. Decompose tiled PackOp into lower-level Ops + simplify

>From 2e370d3c7513decc790255201cf54b93be19d64e Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:38:23 -0700
Subject: [PATCH 13/14] Remove dead %mmt4d_func

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f0348c1679723..db15eff92cad3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -247,7 +247,6 @@ module @transforms attributes { transform.with_named_sequence } {
     // HANDLE MMT4D
     //==========================================================================
     %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
-    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
     // Tile parallel dims (note, the M, N dims are scalable!)

>From d3ea3efb935e38d2d17708703d4b71a05f96d2db Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:41:16 -0700
Subject: [PATCH 14/14] Fix comment for pack_lhs

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir         | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index db15eff92cad3..b599b95cd19fa 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -121,9 +121,10 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 }
 
 //===----------------------------------------------------------------------===//
-// @matmul_via_mmt4d
+// @pack_lhs
 //
-// Implements matrix-multiplication via linalg.mmt4d
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32


From llvmlistbot at llvm.org  Mon Jul 13 06:43:51 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb97.481afadf.196d57.6985@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb98.c5504292.28b3b8.10d7@mx.google.com>


https://github.com/banach-space commented:

Thank you!

Some tests are unclear to me, so I left inline comments asking for some clarification.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb98.bb839515.27d21e.b47e@mx.google.com>


================
@@ -122,10 +122,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
----------------
banach-space wrote:

Why does this one fail? 

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb98.27b3b003.3b871.54ce@mx.google.com>


================
@@ -257,10 +257,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
----------------
banach-space wrote:

Why does this one fail?

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.af6ba3ed.1dc10f.b861@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
----------------
banach-space wrote:

IIUC, the issue here is that the tiled outer-dims are non-unit? Definition is available here: https://github.com/llvm/llvm-project/blob/29489077663c5dd9922a66996dbf897e2710727d/mlir/include/mlir/Dialect/Linalg/IR/LinalgRelayoutOps.td?plain=1#L82-L85

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.160324b0.195192.c70f@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
----------------
banach-space wrote:

>From what I can tell, what follows are tests for `linalg.unpack` that are symmetrical to tests for `linalg.pack` that you added above?

Instead of mixing them, could you make sure that all tests for `pack` and `unpack`, respectively, are grouped together? It's sufficient if you just move these tests. Thank you!

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.13be6df8.145c.d69a@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
----------------
banach-space wrote:

Is this testing a code-path different to `@negative_packing_with_outer_dims_perm`?

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.bd93e731.358318.e341@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
----------------
banach-space wrote:

Aren't both inner and outer dims permuted?

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.f06f4994.1fc857.1f2d@mx.google.com>


================
@@ -196,21 +196,21 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
 }
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
----------------
banach-space wrote:

Why does this one fail?

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:43:53 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Mon, 13 Jul 2026 06:43:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a54eb99.e5907290.281f16.eb18@mx.google.com>


================
@@ -61,21 +61,21 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
 }
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
----------------
banach-space wrote:

IIUC, this is failing because the inner time is tiled, right? If yes, I'd rename as:
* `@negative_packing_inner_dim`

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Mon Jul 13 06:44:09 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Mon, 13 Jul 2026 06:44:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a54eba9.160324b0.195192.cf72@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile sizes.
+  %c7 = arith.constant 7 : index
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
+  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
+
+  return %C_pack : tensor<?x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
----------------
steplong wrote:

Done

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 06:44:17 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Mon, 13 Jul 2026 06:44:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a54ebb1.c574b9b4.21ead0.12b9@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
----------------
steplong wrote:

Done

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 06:45:45 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Mon, 13 Jul 2026 06:45:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a54ec09.eb165e5c.1817a5.c129@mx.google.com>


================
@@ -0,0 +1,362 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unpack/mmt4d ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
----------------
steplong wrote:

Hmm yeah you might be right. I'll have to think more carefully about the tiling sizes/dims for `pack_lhs`, `pack_rhs`, `pack_acc`, `unpack_acc` 

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 06:59:44 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Mon, 13 Jul 2026 06:59:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54ef50.266b0cc5.29d78.684e@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
----------------
ioghiban wrote:

replacing the type with `Value` does not work here: `error: no member named 'getResult' in 'mlir::Value'`. `auto` resolves to `!emitc.lvalue<>` here.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 07:00:33 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Mon, 13 Jul 2026 07:00:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54ef81.af6ba3ed.1dc10f.88c6@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
----------------
ioghiban wrote:

replacing the type with `Value` does not work here: `error: no member named 'getResult' in 'mlir::Value'`. `auto` resolves to the element type here.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 07:00:42 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Mon, 13 Jul 2026 07:00:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a54ef8a.92d2873c.258297.457b@mx.google.com>


================
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
+                                         srcLValue.getResult());
+
+      auto targetLValue = emitc::SubscriptOp::create(
----------------
ioghiban wrote:

replacing the type with `Value` does not work here: `error: no member named 'getResult' in 'mlir::Value'`. `auto` resolves to `!emitc.lvalue<>` here.

https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Mon Jul 13 07:01:50 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 07:01:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a54efce.92ea0e2a.13fd37.ff90@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/207933

>From 26ed8a15ec67e0efd28a71b04b42c6941c50c4f2 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 29 Jun 2026 16:00:18 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP row_gather

Adds profile compliance support for MXFP row_gather.

Change-Id: I3a256bd60e857955c5a849b951143fdb2df48fab
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 112 +++++++++++++-----
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  17 ++-
 mlir/test/Dialect/Tosa/availability.mlir      |   2 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |   8 ++
 mlir/test/Dialect/Tosa/ops.mlir               |   8 ++
 .../tosa-validation-version-1p0-invalid.mlir  |   9 ++
 .../tosa-validation-version-1p1-valid.mlir    |  18 +++
 7 files changed, 138 insertions(+), 36 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..3d9f40efac459 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -992,37 +992,87 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..ffa80a24b231e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2596,7 +2596,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2608,6 +2616,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2624,11 +2634,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..151a9a27302ae 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..e42650dc88cc1 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..81137921be19f 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1438,6 +1438,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..87d407c11e386 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9aee861c6a8d3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


From llvmlistbot at llvm.org  Mon Jul 13 07:02:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 07:02:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] ac370b8 - [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (#208466)
Message-ID: <6a54f005.7270d2fd.2d4879.3db8@mx.google.com>


Author: Hugo Trachino
Date: 2026-07-13T15:02:40+01:00
New Revision: ac370b8e838ada3dd482560c4b8a568e760f95b5

URL: https://github.com/llvm/llvm-project/commit/ac370b8e838ada3dd482560c4b8a568e760f95b5
DIFF: https://github.com/llvm/llvm-project/commit/ac370b8e838ada3dd482560c4b8a568e760f95b5.diff

LOG: [MLIR][Linalg] Fix LinalgSpecializeGenericOpsPass for scalar inputs (#208466)

The test case added would crash because the Ranked Tensor-ness of
operands was assumed but never checked. This crash only happened when there
were multiple inputs as it must bail out somewhere else when there is only
1 input.

Added: 
    

Modified: 
    mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
    mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
index 9015cbb096f88..8ab0f67d9b7de 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/DecomposeGenericByUnfoldingPermutation.cpp
@@ -164,8 +164,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
   // out which operand can supply that runtime-value (tensor.dim).
   // Leaving it as a future TODO.
   if (llvm::any_of(op->getOpOperands(), [](OpOperand &oper) {
-        auto opType = cast<RankedTensorType>(oper.get().getType());
-        return ShapedType::isDynamicShape(opType.getShape());
+        auto opType = dyn_cast<RankedTensorType>(oper.get().getType());
+        return !opType || ShapedType::isDynamicShape(opType.getShape());
       }))
     return failure();
 
@@ -209,8 +209,8 @@ LogicalResult DecomposeProjectedPermutation::matchAndRewrite(
     // Does it require broadcast?
     if (!broadcastedDims.empty()) {
       assert(!broadcastedDims.empty() && "should have non size broadcast");
-      Value emptyTensor = tensor::EmptyOp::create(rewriter, loc, outputShape,
-                                                  inputRTType.getElementType());
+      Value emptyTensor =
+          tensor::EmptyOp::create(rewriter, loc, outputShape, elType);
 
       auto broadcastOp = linalg::BroadcastOp::create(
           rewriter, loc, newInitValues[i], emptyTensor, broadcastedDims);

diff  --git a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
index 5d66837fca510..f0792d7a696de 100644
--- a/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
+++ b/mlir/test/Dialect/Linalg/specialize-generic-ops-fail.mlir
@@ -46,3 +46,19 @@ func.func @not_copy(%input: tensor<8xi32>, %init: tensor<8xi32>) -> tensor<8xi32
   } -> tensor<8xi32>
   return %res : tensor<8xi32>
 }
+
+
+// -----
+
+// CHECK-LABEL: @scalar_input
+// CHECK: linalg.generic
+#map = affine_map<(d0) -> (d0)>
+#map1 = affine_map<(d0) -> ()>
+func.func @scalar_input(%arg0: tensor<128xf32>, %arg1: f32) -> tensor<128xf32> {
+  %1 = linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel"]} ins(%arg0, %arg1 : tensor<128xf32>, f32) outs(%arg0 : tensor<128xf32>) {
+  ^bb0(%in: f32, %in_1: f32, %out: f32):
+    %2 = arith.addf %in_1, %in_1 : f32
+    linalg.yield %2 : f32
+  } -> tensor<128xf32>
+  return %1 : tensor<128xf32>
+}


        

From llvmlistbot at llvm.org  Mon Jul 13 07:02:50 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Mon, 13 Jul 2026 07:02:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix
 LinalgSpecializeGenericOpsPass for scalar inputs (PR #208466)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208466@github.com>
Message-ID: <6a54f00a.3cce8983.225732.6714@mx.google.com>


https://github.com/nujaa closed https://github.com/llvm/llvm-project/pull/208466

From llvmlistbot at llvm.org  Mon Jul 13 07:19:58 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Mon, 13 Jul 2026 07:19:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] add isEffectivelySerial() to
 acc.parallel and acc.kernels (PR #209192)
Message-ID: <llvm.org/llvm/llvm-project/pull/209192@github.com>


https://github.com/rscottmanley created https://github.com/llvm/llvm-project/pull/209192

Mirror the isEffectivelySerial() function in acc.compute_region to acc.parallel and acc.kernel so this check is unified anywhere it needs to be used, namely before ACCComputeLowering. Should be NFC

>From fd4c12a15c2a009640dc88cdb0e5a44c6008e013 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:13:39 -0700
Subject: [PATCH] [OpenACC] add isEffectivelySerial() to acc.parallel and
 acc.kernels

Mirror the isEffectivelySerial() function in acc.compute_region to
acc.parallel and acc.kernel so this check is unified anywhere it needs
to be used. Should be NFC
---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        |  6 +++
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp     | 35 ++++++++++++++++
 .../OpenACC/Transforms/ACCComputeLowering.cpp | 41 ++-----------------
 3 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 30298767c0f3e..db2f0e0605db5 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -1842,6 +1842,9 @@ def OpenACC_ParallelOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
@@ -2173,6 +2176,9 @@ def OpenACC_KernelsOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 7de59058ead0d..3b168b7631db8 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -14,6 +14,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
@@ -571,6 +572,40 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
+/// Strip index_cast operations from a value before checking for a constant.
+static Value stripIndexCasts(Value val) {
+  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+    val = castOp.getIn();
+  return val;
+}
+
+template <typename ComputeOpT>
+static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  auto numGangs = op.getNumGangsValues();
+  if (numGangs.empty())
+    return false;
+  for (Value gangSize : numGangs) {
+    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
+      return false;
+  }
+  Value numWorkers = op.getNumWorkersValue();
+  if (!numWorkers)
+    return false;
+  Value vectorLength = op.getVectorLengthValue();
+  if (!vectorLength)
+    return false;
+  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
+         isConstantIntValue(stripIndexCasts(vectorLength), 1);
+}
+
+bool ParallelOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
+bool KernelsOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool ComputeRegionOp::isEffectivelySerial() {
   auto *ctx = getContext();
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
index b3d54d2e05697..a275a794c9fd5 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
@@ -78,41 +78,6 @@ namespace {
 // Helper functions
 //===----------------------------------------------------------------------===//
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
-template <typename ComputeOpT>
-static bool isGangWorkerVectorAllOne(ComputeOpT op) {
-  auto numGangs = op.getNumGangsValues();
-  if (numGangs.empty())
-    return false;
-  for (Value gangSize : numGangs) {
-    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
-      return false;
-  }
-  Value numWorkers = op.getNumWorkersValue();
-  if (!numWorkers)
-    return false;
-  Value vectorLength = op.getVectorLengthValue();
-  if (!vectorLength)
-    return false;
-  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
-         isConstantIntValue(stripIndexCasts(vectorLength), 1);
-}
-
-/// A compute construct is "effectively serial" when it specifies
-/// num_gangs(1), num_workers(1), and vector_length(1). This is because
-/// these are the only parallelism dimensions expressible from OpenACC spec
-/// point-of-view and is consistent with how `serial` semantics are defined.
-template <typename ComputeOpT>
-static bool isEffectivelySerial(ComputeOpT op) {
-  return isGangWorkerVectorAllOne(op);
-}
-
 static bool isOpInComputeRegion(Operation *op) {
   Region *region = op->getBlock()->getParent();
   return getEnclosingComputeOp(*region) != nullptr;
@@ -120,9 +85,9 @@ static bool isOpInComputeRegion(Operation *op) {
 
 static bool isOpInSerialRegion(Operation *op) {
   if (auto parallelOp = op->getParentOfType<ParallelOp>())
-    return isEffectivelySerial(parallelOp);
+    return parallelOp.isEffectivelySerial();
   if (auto kernelsOp = op->getParentOfType<KernelsOp>())
-    return isEffectivelySerial(kernelsOp);
+    return kernelsOp.isEffectivelySerial();
   if (op->getParentOfType<SerialOp>())
     return true;
   if (auto computeRegion = op->getParentOfType<ComputeRegionOp>())
@@ -230,7 +195,7 @@ assignKnownLaunchArgs(ComputeConstructT computeOp, DeviceType deviceType,
     return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
   } else if constexpr (llvm::is_one_of<ComputeConstructT, ParallelOp,
                                        KernelsOp>::value) {
-    if (isEffectivelySerial(computeOp))
+    if (computeOp.isEffectivelySerial())
       return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
 
     deviceType = getParDimsDeviceType(computeOp, deviceType);


From llvmlistbot at llvm.org  Mon Jul 13 07:20:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 07:20:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] add isEffectivelySerial() to
 acc.parallel and acc.kernels (PR #209192)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209192@github.com>
Message-ID: <6a54f438.af6ba3ed.1dc10f.b532@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: Scott Manley (rscottmanley)

<details>
<summary>Changes</summary>

Mirror the isEffectivelySerial() function in acc.compute_region to acc.parallel and acc.kernel so this check is unified anywhere it needs to be used, namely before ACCComputeLowering. Should be NFC

---
Full diff: https://github.com/llvm/llvm-project/pull/209192.diff


3 Files Affected:

- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td (+6) 
- (modified) mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp (+35) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp (+3-38) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 30298767c0f3e..db2f0e0605db5 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -1842,6 +1842,9 @@ def OpenACC_ParallelOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
@@ -2173,6 +2176,9 @@ def OpenACC_KernelsOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 7de59058ead0d..3b168b7631db8 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -14,6 +14,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
@@ -571,6 +572,40 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
+/// Strip index_cast operations from a value before checking for a constant.
+static Value stripIndexCasts(Value val) {
+  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+    val = castOp.getIn();
+  return val;
+}
+
+template <typename ComputeOpT>
+static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  auto numGangs = op.getNumGangsValues();
+  if (numGangs.empty())
+    return false;
+  for (Value gangSize : numGangs) {
+    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
+      return false;
+  }
+  Value numWorkers = op.getNumWorkersValue();
+  if (!numWorkers)
+    return false;
+  Value vectorLength = op.getVectorLengthValue();
+  if (!vectorLength)
+    return false;
+  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
+         isConstantIntValue(stripIndexCasts(vectorLength), 1);
+}
+
+bool ParallelOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
+bool KernelsOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool ComputeRegionOp::isEffectivelySerial() {
   auto *ctx = getContext();
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
index b3d54d2e05697..a275a794c9fd5 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
@@ -78,41 +78,6 @@ namespace {
 // Helper functions
 //===----------------------------------------------------------------------===//
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
-template <typename ComputeOpT>
-static bool isGangWorkerVectorAllOne(ComputeOpT op) {
-  auto numGangs = op.getNumGangsValues();
-  if (numGangs.empty())
-    return false;
-  for (Value gangSize : numGangs) {
-    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
-      return false;
-  }
-  Value numWorkers = op.getNumWorkersValue();
-  if (!numWorkers)
-    return false;
-  Value vectorLength = op.getVectorLengthValue();
-  if (!vectorLength)
-    return false;
-  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
-         isConstantIntValue(stripIndexCasts(vectorLength), 1);
-}
-
-/// A compute construct is "effectively serial" when it specifies
-/// num_gangs(1), num_workers(1), and vector_length(1). This is because
-/// these are the only parallelism dimensions expressible from OpenACC spec
-/// point-of-view and is consistent with how `serial` semantics are defined.
-template <typename ComputeOpT>
-static bool isEffectivelySerial(ComputeOpT op) {
-  return isGangWorkerVectorAllOne(op);
-}
-
 static bool isOpInComputeRegion(Operation *op) {
   Region *region = op->getBlock()->getParent();
   return getEnclosingComputeOp(*region) != nullptr;
@@ -120,9 +85,9 @@ static bool isOpInComputeRegion(Operation *op) {
 
 static bool isOpInSerialRegion(Operation *op) {
   if (auto parallelOp = op->getParentOfType<ParallelOp>())
-    return isEffectivelySerial(parallelOp);
+    return parallelOp.isEffectivelySerial();
   if (auto kernelsOp = op->getParentOfType<KernelsOp>())
-    return isEffectivelySerial(kernelsOp);
+    return kernelsOp.isEffectivelySerial();
   if (op->getParentOfType<SerialOp>())
     return true;
   if (auto computeRegion = op->getParentOfType<ComputeRegionOp>())
@@ -230,7 +195,7 @@ assignKnownLaunchArgs(ComputeConstructT computeOp, DeviceType deviceType,
     return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
   } else if constexpr (llvm::is_one_of<ComputeConstructT, ParallelOp,
                                        KernelsOp>::value) {
-    if (isEffectivelySerial(computeOp))
+    if (computeOp.isEffectivelySerial())
       return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
 
     deviceType = getParDimsDeviceType(computeOp, deviceType);

``````````

</details>


https://github.com/llvm/llvm-project/pull/209192

From llvmlistbot at llvm.org  Mon Jul 13 07:21:59 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Mon, 13 Jul 2026 07:21:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] add isEffectivelySerial() to
 acc.parallel and acc.kernels (PR #209192)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209192@github.com>
Message-ID: <6a54f487.8e75e0cb.3dbce4.c53f@mx.google.com>


https://github.com/rscottmanley updated https://github.com/llvm/llvm-project/pull/209192

>From fd4c12a15c2a009640dc88cdb0e5a44c6008e013 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:13:39 -0700
Subject: [PATCH 1/2] [OpenACC] add isEffectivelySerial() to acc.parallel and
 acc.kernels

Mirror the isEffectivelySerial() function in acc.compute_region to
acc.parallel and acc.kernel so this check is unified anywhere it needs
to be used. Should be NFC
---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        |  6 +++
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp     | 35 ++++++++++++++++
 .../OpenACC/Transforms/ACCComputeLowering.cpp | 41 ++-----------------
 3 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 30298767c0f3e..db2f0e0605db5 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -1842,6 +1842,9 @@ def OpenACC_ParallelOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
@@ -2173,6 +2176,9 @@ def OpenACC_KernelsOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 7de59058ead0d..3b168b7631db8 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -14,6 +14,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
@@ -571,6 +572,40 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
+/// Strip index_cast operations from a value before checking for a constant.
+static Value stripIndexCasts(Value val) {
+  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+    val = castOp.getIn();
+  return val;
+}
+
+template <typename ComputeOpT>
+static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  auto numGangs = op.getNumGangsValues();
+  if (numGangs.empty())
+    return false;
+  for (Value gangSize : numGangs) {
+    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
+      return false;
+  }
+  Value numWorkers = op.getNumWorkersValue();
+  if (!numWorkers)
+    return false;
+  Value vectorLength = op.getVectorLengthValue();
+  if (!vectorLength)
+    return false;
+  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
+         isConstantIntValue(stripIndexCasts(vectorLength), 1);
+}
+
+bool ParallelOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
+bool KernelsOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool ComputeRegionOp::isEffectivelySerial() {
   auto *ctx = getContext();
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
index b3d54d2e05697..a275a794c9fd5 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
@@ -78,41 +78,6 @@ namespace {
 // Helper functions
 //===----------------------------------------------------------------------===//
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
-template <typename ComputeOpT>
-static bool isGangWorkerVectorAllOne(ComputeOpT op) {
-  auto numGangs = op.getNumGangsValues();
-  if (numGangs.empty())
-    return false;
-  for (Value gangSize : numGangs) {
-    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
-      return false;
-  }
-  Value numWorkers = op.getNumWorkersValue();
-  if (!numWorkers)
-    return false;
-  Value vectorLength = op.getVectorLengthValue();
-  if (!vectorLength)
-    return false;
-  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
-         isConstantIntValue(stripIndexCasts(vectorLength), 1);
-}
-
-/// A compute construct is "effectively serial" when it specifies
-/// num_gangs(1), num_workers(1), and vector_length(1). This is because
-/// these are the only parallelism dimensions expressible from OpenACC spec
-/// point-of-view and is consistent with how `serial` semantics are defined.
-template <typename ComputeOpT>
-static bool isEffectivelySerial(ComputeOpT op) {
-  return isGangWorkerVectorAllOne(op);
-}
-
 static bool isOpInComputeRegion(Operation *op) {
   Region *region = op->getBlock()->getParent();
   return getEnclosingComputeOp(*region) != nullptr;
@@ -120,9 +85,9 @@ static bool isOpInComputeRegion(Operation *op) {
 
 static bool isOpInSerialRegion(Operation *op) {
   if (auto parallelOp = op->getParentOfType<ParallelOp>())
-    return isEffectivelySerial(parallelOp);
+    return parallelOp.isEffectivelySerial();
   if (auto kernelsOp = op->getParentOfType<KernelsOp>())
-    return isEffectivelySerial(kernelsOp);
+    return kernelsOp.isEffectivelySerial();
   if (op->getParentOfType<SerialOp>())
     return true;
   if (auto computeRegion = op->getParentOfType<ComputeRegionOp>())
@@ -230,7 +195,7 @@ assignKnownLaunchArgs(ComputeConstructT computeOp, DeviceType deviceType,
     return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
   } else if constexpr (llvm::is_one_of<ComputeConstructT, ParallelOp,
                                        KernelsOp>::value) {
-    if (isEffectivelySerial(computeOp))
+    if (computeOp.isEffectivelySerial())
       return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
 
     deviceType = getParDimsDeviceType(computeOp, deviceType);

>From db3c29b6b0fd8452eebbab0c273d69668d9a7d1e Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:21:28 -0700
Subject: [PATCH 2/2] move strip index casts to lambda

---
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 3b168b7631db8..8cb2be720c29e 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -572,15 +572,15 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
 template <typename ComputeOpT>
 static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  // Strip index_cast operations from a value before checking for a constant.
+  auto stripIndexCasts = [](Value val) -> Value {
+    while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+      val = castOp.getIn();
+    return val;
+  };
+
   auto numGangs = op.getNumGangsValues();
   if (numGangs.empty())
     return false;


From llvmlistbot at llvm.org  Mon Jul 13 07:27:25 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:27:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
 in_reduction for host fallback (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54f5cd.bc28b987.90005.c2e4@mx.google.com>


https://github.com/Saieiei updated https://github.com/llvm/llvm-project/pull/199967

>From 03552bc49f3fab632048bad1e661c348c94a68ec Mon Sep 17 00:00:00 2001
From: Sairudra More <sairudra60 at gmail.com>
Date: Fri, 12 Jun 2026 04:57:07 -0500
Subject: [PATCH] [flang][OpenMP] Lower target in_reduction

Enable host lowering for target in_reduction in Flang and MLIR OpenMP
translation.

Model target in_reduction through the matching map entry, force
address-preserving implicit mapping for Flang in_reduction list items, and emit
the host-side task-reduction lookup with __kmpc_task_reduction_get_th_data. The
runtime entry point takes and returns a generic, default-address-space pointer,
so normalize a non-default-address-space captured pointer to the generic address
space before the call and cast the returned private pointer back to the map
block argument's address space, mirroring the in_reduction handling on
omp.taskloop. On the target device, in_reduction is handled as a regular
map(tofrom) variable. The byref modifier, two-argument initializers, cleanup
regions, and the remaining Flang COMMON/EQUIVALENCE/privatized-variable cases
continue to be diagnosed.

Add Flang lowering, MLIR verifier/translation, and LLVM IR tests for the
supported host path, including a non-default-address-space case, and the
remaining unsupported cases.
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 103 ++++++++++--
 .../OpenMP/target-inreduction-llvmir.f90      |  50 ++++++
 .../OpenMP/Todo/target-inreduction-common.f90 |  20 +++
 .../Todo/target-inreduction-equivalence.f90   |  19 +++
 .../Todo/target-inreduction-firstprivate.f90  |  19 +++
 .../Lower/OpenMP/Todo/target-inreduction.f90  |  15 --
 .../OpenMP/target-inreduction-unused.f90      |  27 ++++
 .../test/Lower/OpenMP/target-inreduction.f90  |  30 ++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  33 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  50 ++++++
 .../mlir/Dialect/OpenMP/OpenMPClauses.td      |  14 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  22 ++-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 149 +++++++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 141 +++++++++++++++--
 .../Dialect/OpenMP/host-op-filtering.mlir     |   8 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  85 ++++++++++
 .../openmp-target-in-reduction-device.mlir    |  51 ++++++
 .../openmp-target-in-reduction-multi.mlir     |  77 +++++++++
 .../LLVMIR/openmp-target-in-reduction.mlir    | 107 +++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  91 ++++++++++-
 20 files changed, 1032 insertions(+), 79 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction-unused.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 9c6352b73a585..23eb9d587d852 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2474,6 +2474,7 @@ genTargetClauses(lower::AbstractConverter &converter,
                  mlir::omp::TargetExtOperands &clauseOps,
                  DefaultMapsTy &defaultMaps,
                  llvm::SmallVectorImpl<Object> &hasDeviceAddrObjects,
+                 llvm::SmallVectorImpl<Object> &inReductionObjects,
                  llvm::SmallVectorImpl<Object> &isDevicePtrObjects,
                  llvm::SmallVectorImpl<Object> &mapObjects) {
   ClauseProcessor cp(converter, semaCtx, clauses);
@@ -2489,12 +2490,33 @@ genTargetClauses(lower::AbstractConverter &converter,
     hostEvalInfo->collectValues(clauseOps.hostEvalVars);
   }
   cp.processIf(llvm::omp::Directive::OMPD_target, clauseOps);
+  cp.processInReduction(loc, clauseOps, inReductionObjects);
+  // The host lowering of `target in_reduction` redirects each list item's
+  // dedicated implicit map entry to the per-task reduction-private storage
+  // returned by the runtime. A COMMON block member is accessed in the target
+  // body through the COMMON block storage map, not through its own member map
+  // entry, so the redirect would rewrite an unused map argument and the body
+  // would accumulate into the mapped original rather than the private copy.
+  // An EQUIVALENCE-associated variable has the same problem: it shares storage
+  // with another variable that may be mapped separately (e.g. `map(tofrom: j)`
+  // with `equivalence(i, j)`), so the body updates the aliased storage while
+  // the redirected runtime-private copy stays dead.
+  // Reject both cases until the redirect can target the storage map.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      if (semantics::FindCommonBlockContaining(sym->GetUltimate()))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a COMMON block member");
+      if (semantics::FindEquivalenceSet(sym->GetUltimate()))
+        TODO(loc, "TARGET construct with IN_REDUCTION of an EQUIVALENCE "
+                  "storage-associated variable");
+    }
   cp.processIsDevicePtr(stmtCtx, clauseOps, isDevicePtrObjects);
   cp.processMap(loc, stmtCtx, clauseOps, llvm::omp::Directive::OMPD_unknown,
                 &mapObjects);
   cp.processNowait(clauseOps);
   cp.processThreadLimit(stmtCtx, clauseOps);
-  cp.processTODO<clause::Allocate, clause::InReduction, clause::UsesAllocators>(
+  cp.processTODO<clause::Allocate, clause::UsesAllocators>(
       loc, llvm::omp::Directive::OMPD_target);
 
   // `target private(..)` is only supported in delayed privatization mode.
@@ -3556,10 +3578,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   mlir::omp::TargetExtOperands clauseOps;
   DefaultMapsTy defaultMaps;
   llvm::SmallVector<Object> mapObjects, hasDeviceAddrObjects,
-      isDevicePtrObjects;
+      inReductionObjects, isDevicePtrObjects;
   genTargetClauses(converter, semaCtx, symTable, stmtCtx, eval, item->clauses,
                    loc, clauseOps, defaultMaps, hasDeviceAddrObjects,
-                   isDevicePtrObjects, mapObjects);
+                   inReductionObjects, isDevicePtrObjects, mapObjects);
 
   KernelTypeVisitor visitor(semaCtx, converter.getModuleOp());
   clauseOps.kernelType = mlir::omp::TargetExecModeAttr::get(
@@ -3607,6 +3629,26 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
                            /*isTargetPrivitization=*/true);
   dsp.processStep1(&clauseOps);
 
+  // The host lowering of `target in_reduction` requires each list item to be
+  // implicitly mapped as an address-preserving entry whose map argument the
+  // redirect rewrites to the runtime reduction-private storage. When the same
+  // variable is also privatized on the target (implicitly, as with a
+  // host-associated scalar that defaults to firstprivate, or explicitly with
+  // `firstprivate(i) in_reduction(+:i)`), the force-map is suppressed because
+  // the symbol is already in the privatization set, and the firstprivate copy
+  // region then has no mold value for it, aborting translation. Reject until
+  // in_reduction can be reconciled with target privatization.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      const semantics::Symbol &ultimate = sym->GetUltimate();
+      if (llvm::any_of(dsp.getAllSymbolsToPrivatize(),
+                       [&](const semantics::Symbol *priv) {
+                         return priv->GetUltimate() == ultimate;
+                       }))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a privatized variable");
+    }
+
   // Collect symbols that have dynamic substring accesses
   llvm::SmallPtrSet<const semantics::Symbol *, 8> symbolsWithDynamicSubstring;
   collectSymbolsWithDynamicSubstring(semaCtx, eval,
@@ -3617,7 +3659,16 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   // symbols used inside the region that do not have explicit data-environment
   // attribute clauses (neither data-sharing; e.g. `private`, nor `map`
   // clauses).
-  auto captureImplicitMap = [&](const semantics::Symbol &sym) {
+  //
+  // When `forceAddressPreserving` is set, the symbol is force-mapped as an
+  // address-preserving `capture(ByRef)` with implicit `tofrom` flags,
+  // bypassing the scalar default capture rules. This is used for `target
+  // in_reduction` list items, whose mapped pointer is passed as the `orig`
+  // argument of `__kmpc_task_reduction_get_th_data`; a ByCopy scalar capture
+  // would break the runtime lookup against the enclosing taskgroup's
+  // task_reduction descriptor.
+  auto captureImplicitMap = [&](const semantics::Symbol &sym,
+                                bool forceAddressPreserving = false) {
     // Structure component symbols don't have bindings, and can only be
     // explicitly mapped individually. If a member is captured implicitly
     // we map the entirety of the derived type when we find its symbol.
@@ -3626,12 +3677,14 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
     // if the symbol is part of an already mapped common block, do not make a
     // map for it.
-    if (const Fortran::semantics::Symbol *common =
-            Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
-      if (llvm::any_of(mapObjects, [=](const Object &object) {
-            return object.sym() == common;
-          }))
-        return;
+    if (!forceAddressPreserving) {
+      if (const Fortran::semantics::Symbol *common =
+              Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
+        if (llvm::any_of(mapObjects, [=](const Object &object) {
+              return object.sym() == common;
+            }))
+          return;
+    }
 
     // If we come across a symbol without a symbol address, we
     // return as we cannot process it, this is intended as a
@@ -3681,13 +3734,21 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
         eleType = refType.getElementType();
 
       std::pair<mlir::omp::ClauseMapFlags, mlir::omp::VariableCaptureKind>
-          mapFlagAndKind = getImplicitMapTypeAndKind(
-              firOpBuilder, converter, defaultMaps, eleType, loc, sym);
+          mapFlagAndKind;
+      if (forceAddressPreserving)
+        mapFlagAndKind = {mlir::omp::ClauseMapFlags::implicit |
+                              mlir::omp::ClauseMapFlags::to |
+                              mlir::omp::ClauseMapFlags::from,
+                          mlir::omp::VariableCaptureKind::ByRef};
+      else
+        mapFlagAndKind = getImplicitMapTypeAndKind(
+            firOpBuilder, converter, defaultMaps, eleType, loc, sym);
 
       mlir::FlatSymbolRefAttr mapperId;
       auto defaultmapBehaviour = getDefaultmapIfPresent(defaultMaps, eleType);
-      if (defaultmapBehaviour ==
-          clause::Defaultmap::ImplicitBehavior::Default) {
+      if (!forceAddressPreserving &&
+          defaultmapBehaviour ==
+              clause::Defaultmap::ImplicitBehavior::Default) {
         const semantics::DerivedTypeSpec *typeSpec =
             sym.GetType() ? sym.GetType()->AsDerived() : nullptr;
         if (typeSpec) {
@@ -3741,6 +3802,15 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
           Object{const_cast<semantics::Symbol *>(&sym), std::nullopt});
     }
   };
+  // OpenMP requires `in_reduction` list items on `target` to be implicitly
+  // data-mapped. Force-map them as address-preserving captures before the
+  // generic implicit-map walk so that walk treats the symbols as already
+  // mapped via `isDuplicateMappedSymbol` and does not downgrade them to
+  // ByCopy.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym())
+      captureImplicitMap(*sym, /*forceAddressPreserving=*/true);
+
   lower::pft::visitAllSymbols(eval, captureImplicitMap);
 
   auto targetOp = mlir::omp::TargetOp::create(firOpBuilder, loc, clauseOps);
@@ -3753,7 +3823,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   args.hasDeviceAddr.objects = hasDeviceAddrObjects;
   args.hasDeviceAddr.vars = hasDeviceAddrBaseValues;
   args.hostEvalVars = clauseOps.hostEvalVars;
-  // TODO: Add in_reduction syms and vars.
+  // `in_reduction` list items do not get their own entry block argument on
+  // `omp.target`; they are implicitly mapped (see the force-map above) and the
+  // target body accesses them through their `map_entries` block argument. The
+  // `in_reduction` operands remain on the op as host-side metadata.
   args.map.objects = mapObjects;
   args.map.vars = mapBaseValues;
   args.priv.objects = makeObjects(dsp.getDelayedPrivSymbols());
diff --git a/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90 b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
new file mode 100644
index 0000000000000..1898e6605f7f8
--- /dev/null
+++ b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
@@ -0,0 +1,50 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-version=50 -o - %s | FileCheck %s
+
+! End-to-end coverage: Flang lowers a taskgroup task_reduction enclosing a
+! target in_reduction all the way to LLVM IR. Verifies that the host path
+! emits __kmpc_taskred_init for the enclosing taskgroup and
+! __kmpc_task_reduction_get_th_data for the target's in_reduction lookup,
+! with load/store through the returned private pointer and no direct update
+! through the original shared pointer.
+
+subroutine target_in_reduction_e2e()
+  integer :: i
+  i = 0
+  !$omp taskgroup task_reduction(+:i)
+    !$omp target in_reduction(+:i)
+    i = i + 1
+    !$omp end target
+  !$omp end taskgroup
+end subroutine target_in_reduction_e2e
+
+! CHECK-LABEL: define void @target_in_reduction_e2e_()
+! The enclosing taskgroup emits __kmpc_taskred_init to register the
+! task_reduction descriptor.
+! CHECK:         call ptr @__kmpc_taskred_init(i32 %{{.+}}, i32 1, ptr %{{.+}})
+
+! The host stub calls the outlined target body passing the captured pointer.
+! CHECK:         call void @__omp_offloading_{{.*}}_target_in_reduction_e2e_{{.*}}(ptr %{{.+}}, ptr null)
+
+! Inside the outlined target body, the in_reduction private pointer is
+! obtained from the runtime using the captured original pointer with a NULL
+! descriptor (the runtime walks enclosing taskgroups). All loads and stores
+! go through the returned private pointer.
+! CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_in_reduction_e2e_
+! CHECK-SAME:    (ptr %[[ORIG:.+]], ptr %{{.+}})
+! CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+! CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[ORIG]])
+! CHECK:         %[[VAL:.+]] = load i32, ptr %[[PRIV]]
+! CHECK:         %[[SUM:.+]] = add i32 %[[VAL]], 1
+! CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+! The outlined body must not store directly through the captured original
+! pointer; all updates go through the runtime-returned private copy.
+! CHECK-NOT:     store i32 %{{.+}}, ptr %[[ORIG]]
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
new file mode 100644
index 0000000000000..0b0f3e2e53dea
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is a COMMON block member is accessed in
+! the target body through the COMMON block storage map, not through its own
+! dedicated member map entry. The host-side redirect only rewrites the
+! member map argument, so the runtime-private pointer would be dead and the body
+! would accumulate into the mapped original instead of the reduction-private
+! copy. Until the redirect can target the storage map this case is rejected.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a COMMON block member
+
+subroutine omp_target_in_reduction_common()
+  integer :: i, k
+  common /cb/ i, k
+  i = 0
+  !$omp target map(tofrom: /cb/) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_common
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
new file mode 100644
index 0000000000000..43d03dba0c594
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is EQUIVALENCE storage-associated
+! shares storage with another variable that may be mapped separately (here `j`).
+! The body updates the aliased storage through `j`'s map while the host-side
+! redirect rewrites only `i`'s map argument, leaving the runtime-private copy
+! dead. Reject until the redirect can account for storage association.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of an EQUIVALENCE storage-associated variable
+
+subroutine omp_target_in_reduction_equivalence()
+  integer :: i, j
+  equivalence(i, j)
+  i = 0
+  !$omp target map(tofrom: j) in_reduction(+:i)
+  j = j + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_equivalence
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
new file mode 100644
index 0000000000000..075fb1a9135be
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! An explicit `firstprivate(i)` on the same `target` as `in_reduction(+:i)`
+! privatizes the reduction variable. The item is then privatized instead of
+! address-preserving-mapped, so the host-side redirect has no dedicated map
+! argument to rewrite and the firstprivate copy region has no mold value, which
+! previously aborted translation. Reject until in_reduction can be reconciled
+! with target privatization.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a privatized variable
+
+subroutine omp_target_in_reduction_firstprivate()
+  integer :: i
+  i = 0
+  !$omp target firstprivate(i) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_firstprivate
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
deleted file mode 100644
index e5a9cffac5a11..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-
-!===============================================================================
-! `mergeable` clause
-!===============================================================================
-
-! CHECK: not yet implemented: Unhandled clause IN_REDUCTION in TARGET construct
-subroutine omp_target_inreduction()
-  integer i
-  i = 0
-  !$omp target in_reduction(+:i)
-  i = i + 1
-  !$omp end target
-end subroutine omp_target_inreduction
diff --git a/flang/test/Lower/OpenMP/target-inreduction-unused.f90 b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
new file mode 100644
index 0000000000000..f9dfa4973610c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
@@ -0,0 +1,27 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Per the OpenMP spec, an in_reduction list item on a target construct is
+! implicitly data-mapped. The lowering must not rely on the variable being
+! referenced inside the target body to discover that map: here `i` only
+! appears in the in_reduction clause and is never read or written inside
+! the region. Verify that an omp.map.info for `i` is still emitted and
+! flows into the omp.target's map_entries.
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction_unused()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reduction_unusedEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@{{[^ ]+}} %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %{{[^ ]+}} : !fir.ref<i32>)
+
+subroutine omp_target_in_reduction_unused()
+  interface
+    subroutine sub()
+    end subroutine
+  end interface
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  call sub()
+  !$omp end target
+end subroutine omp_target_in_reduction_unused
diff --git a/flang/test/Lower/OpenMP/target-inreduction.f90 b/flang/test/Lower/OpenMP/target-inreduction.f90
new file mode 100644
index 0000000000000..74f0f003aab4f
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction.f90
@@ -0,0 +1,30 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Verify that in_reduction on a target construct is lowered to an
+! omp.target with both an in_reduction clause and an implicit map_entries
+! entry for the same variable. The in_reduction clause does not define an
+! entry block argument: inside the target body the variable is accessed
+! through its map_entries block argument. The implicit map also captures the
+! original pointer into the target region so the MLIR -> LLVM IR translation
+! can pass it to __kmpc_task_reduction_get_th_data.
+
+!CHECK-LABEL: omp.declare_reduction
+!CHECK-SAME:  @[[RED_I32_NAME:.*]] : i32 init {
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reductionEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@[[RED_I32_NAME]] %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %[[MAPARG:[^ ]+]] : !fir.ref<i32>)
+!CHECK:         hlfir.declare %[[MAPARG]]
+!CHECK:         omp.terminator
+!CHECK:       }
+
+subroutine omp_target_in_reduction()
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 965ac358c259e..54d14602fbdbf 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivateCB.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivateCB Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivateCB);
+
   /// Return the insertion point used by the underlying IRBuilder.
   InsertPointTy getInsertionPoint() { return Builder.saveIP(); }
 
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..0dc704a87e4ed 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -1218,6 +1218,56 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivateCB) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item.
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Gtid = getOrCreateThreadID(Ident);
+
+  // The runtime entry point takes (and returns) a generic, default-address-
+  // space `ptr`. A NULL descriptor makes the runtime walk the enclosing
+  // taskgroups to find the matching task_reduction registration for the item.
+  Type *PtrTy = PointerType::getUnqual(M.getContext());
+  Value *NullDesc = ConstantPointerNull::get(PtrTy);
+  FunctionCallee GetThData =
+      getOrCreateRuntimeFunction(M, OMPRTL___kmpc_task_reduction_get_th_data);
+
+  for (unsigned Idx = 0; Idx < OrigPtrs.size(); ++Idx) {
+    // Normalize a non-default-address-space original pointer to the generic
+    // address space before the call.
+    Value *OrigPtr = OrigPtrs[Idx];
+    if (auto *OrigPtrTy = dyn_cast<PointerType>(OrigPtr->getType());
+        OrigPtrTy && OrigPtrTy->getAddressSpace() != 0)
+      OrigPtr = Builder.CreateAddrSpaceCast(OrigPtr, PtrTy);
+
+    Value *Priv = Builder.CreateCall(GetThData, {Gtid, NullDesc, OrigPtr},
+                                     "omp.inred.priv");
+
+    // Cast the returned private pointer back to the requested address space
+    // when it differs.
+    if (auto *ResPtrTy = dyn_cast<PointerType>(ResultPtrTys[Idx]);
+        ResPtrTy && ResPtrTy->getAddressSpace() != 0)
+      Priv = Builder.CreateAddrSpaceCast(Priv, ResultPtrTys[Idx]);
+
+    MapPrivateCB(Idx, Priv);
+  }
+  return Builder.saveIP();
+}
+
 OpenMPIRBuilder::InsertPointOrErrorTy
 OpenMPIRBuilder::createBarrier(const LocationDescription &Loc, Directive Kind,
                                bool ForceSimpleCall, bool CheckCancelFlag) {
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
index 54e46b74c5b64..cd1223dc1702c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
@@ -764,9 +764,17 @@ class OpenMP_InReductionClauseSkip<
     OptionalAttr<SymbolRefArrayAttr>:$in_reduction_syms
   );
 
-  // Description varies depending on the operation. Assembly format not defined
-  // because this clause must be processed together with the first region of the
-  // operation, as it defines entry block arguments.
+  // This assembly format should only be used by operations where `in_reduction`
+  // does not define entry block arguments (e.g. `omp.target`). Otherwise, it
+  // must be printed and parsed together with the corresponding region, because
+  // it defines entry block arguments.
+  let optAssemblyFormat = [{
+    `in_reduction` `(`
+      custom<InReductionClause>($in_reduction_vars, type($in_reduction_vars),
+                                $in_reduction_byref, $in_reduction_syms) `)`
+  }];
+
+  // Description varies depending on the operation.
 }
 
 def OpenMP_InReductionClause : OpenMP_InReductionClauseSkip<>;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 5b4ce7cc214bf..fd700e081fa01 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -966,7 +966,8 @@ def TaskOp : OpenMP_Op<"task", traits = [
     OutlineableOpenMPOpInterface
   ], clauses = [
     OpenMP_AffinityClause, OpenMP_AllocateClause, OpenMP_DependClause,
-    OpenMP_FinalClause, OpenMP_IfClause, OpenMP_InReductionClause,
+    OpenMP_FinalClause, OpenMP_IfClause,
+    OpenMP_InReductionClauseSkip<assemblyFormat = true>,
     OpenMP_MergeableClause, OpenMP_PriorityClause, OpenMP_PrivateClause,
     OpenMP_UntiedClause, OpenMP_DetachClause
   ], singleRegion = true> {
@@ -1012,9 +1013,10 @@ def TaskloopContextOp : OpenMP_Op<"taskloop.context", traits = [
     DeclareOpInterfaceMethods<OutlineableOpenMPOpInterface>
   ], clauses = [
     OpenMP_AllocateClause, OpenMP_FinalClause, OpenMP_GrainsizeClause,
-    OpenMP_IfClause, OpenMP_InReductionClause, OpenMP_MergeableClause,
-    OpenMP_NogroupClause, OpenMP_NumTasksClause, OpenMP_PriorityClause,
-    OpenMP_PrivateClause, OpenMP_ReductionClause, OpenMP_UntiedClause
+    OpenMP_IfClause, OpenMP_InReductionClauseSkip<assemblyFormat = true>,
+    OpenMP_MergeableClause, OpenMP_NogroupClause, OpenMP_NumTasksClause,
+    OpenMP_PriorityClause, OpenMP_PrivateClause, OpenMP_ReductionClause,
+    OpenMP_UntiedClause
   ], singleRegion = true> {
   let summary = "OutlinableOpenMPOpInterface wrapper for taskloop construct";
   let description = [{
@@ -1627,6 +1629,15 @@ def TargetOp : OpenMP_Op<"target", traits = [
   ];
 
   let extraClassDeclaration = [{
+    // Override BlockArgOpenMPOpInterface method because `in_reduction` list
+    // items on `omp.target` do not define entry block arguments. The reduction
+    // variable is accessed inside the target body through its matching
+    // `map_entries` block argument; the `in_reduction` operands are kept only
+    // as host-side metadata used to look up the per-task private storage.
+    unsigned numInReductionBlockArgs() {
+      return 0;
+    }
+
     mlir::Value getMappedValueForPrivateVar(unsigned privVarIdx) {
       std::optional<DenseI64ArrayAttr> privateMapIdices = getPrivateMapsAttr();
 
@@ -1651,8 +1662,7 @@ def TargetOp : OpenMP_Op<"target", traits = [
     " | `map_iterated` `(` $map_iterated `:` type($map_iterated) `)`)" # [{
       custom<TargetOpRegion>(
           $region, $has_device_addr_vars, type($has_device_addr_vars),
-          $host_eval_vars, type($host_eval_vars), $in_reduction_vars,
-          type($in_reduction_vars), $in_reduction_byref, $in_reduction_syms,
+          $host_eval_vars, type($host_eval_vars),
           $map_vars, type($map_vars), $private_vars, type($private_vars),
           $private_syms, $private_needs_barrier, $private_maps) attr-dict
     }];
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 41d3c547ca618..adfb097c830a3 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1038,6 +1038,82 @@ static void printDynGroupprivateClause(OpAsmPrinter &printer, Operation *op,
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Parser and printer for in_reduction Clause
+//===----------------------------------------------------------------------===//
+
+/// Parses an `in_reduction` clause for an operation that does not give its
+/// list items entry block arguments (e.g. `omp.target`). The expected format is
+/// a comma-separated list of `[byref] @sym %var` followed by `: types`.
+static ParseResult parseInReductionClause(
+    OpAsmParser &parser,
+    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
+    SmallVectorImpl<Type> &inReductionTypes,
+    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms) {
+  SmallVector<SymbolRefAttr> symbolVec;
+  SmallVector<bool> isByRefVec;
+
+  if (parser.parseCommaSeparatedList([&]() {
+        isByRefVec.push_back(parser.parseOptionalKeyword("byref").succeeded());
+        if (parser.parseAttribute(symbolVec.emplace_back()) ||
+            parser.parseOperand(inReductionVars.emplace_back()))
+          return failure();
+        return success();
+      }))
+    return failure();
+
+  if (parser.parseColon())
+    return failure();
+
+  if (parser.parseCommaSeparatedList(
+          [&]() { return parser.parseType(inReductionTypes.emplace_back()); }))
+    return failure();
+
+  if (inReductionVars.size() != inReductionTypes.size())
+    return failure();
+
+  inReductionByref = makeDenseBoolArrayAttr(parser.getContext(), isByRefVec);
+  SmallVector<Attribute> symbolAttrs(symbolVec.begin(), symbolVec.end());
+  inReductionSyms = ArrayAttr::get(parser.getContext(), symbolAttrs);
+  return success();
+}
+
+/// Prints an `in_reduction` clause for an operation that does not give its list
+/// items entry block arguments (e.g. `omp.target`). Mirrors
+/// `parseInReductionClause`.
+static void printInReductionClause(OpAsmPrinter &p, Operation *op,
+                                   ValueRange inReductionVars,
+                                   TypeRange inReductionTypes,
+                                   DenseBoolArrayAttr inReductionByref,
+                                   ArrayAttr inReductionSyms) {
+  MLIRContext *ctx = op->getContext();
+
+  ArrayAttr syms = inReductionSyms;
+  if (!syms) {
+    SmallVector<Attribute> values(inReductionVars.size(), nullptr);
+    syms = ArrayAttr::get(ctx, values);
+  }
+
+  DenseBoolArrayAttr byref = inReductionByref;
+  if (!byref) {
+    SmallVector<bool> values(inReductionVars.size(), false);
+    byref = DenseBoolArrayAttr::get(ctx, values);
+  }
+
+  llvm::interleaveComma(
+      llvm::zip_equal(inReductionVars, syms.getValue(), byref.asArrayRef()), p,
+      [&p](auto t) {
+        auto [var, sym, isByRef] = t;
+        if (isByRef)
+          p << "byref ";
+        if (sym)
+          p << sym << " ";
+        p << var;
+      });
+  p << " : ";
+  llvm::interleaveComma(inReductionTypes, p);
+}
+
 //===----------------------------------------------------------------------===//
 // Parsers for operations including clauses that define entry block arguments.
 //===----------------------------------------------------------------------===//
@@ -1305,9 +1381,6 @@ static ParseResult parseTargetOpRegion(
     SmallVectorImpl<Type> &hasDeviceAddrTypes,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &hostEvalVars,
     SmallVectorImpl<Type> &hostEvalTypes,
-    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
-    SmallVectorImpl<Type> &inReductionTypes,
-    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &mapVars,
     SmallVectorImpl<Type> &mapTypes,
     llvm::SmallVectorImpl<OpAsmParser::UnresolvedOperand> &privateVars,
@@ -1316,8 +1389,6 @@ static ParseResult parseTargetOpRegion(
   AllRegionParseArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, &privateMaps);
@@ -1568,20 +1639,18 @@ static void printBlockArgRegion(OpAsmPrinter &p, Operation *op, Region &region,
 
 // These parseXyz functions correspond to the custom<Xyz> definitions
 // in the .td file(s).
-static void printTargetOpRegion(
-    OpAsmPrinter &p, Operation *op, Region &region,
-    ValueRange hasDeviceAddrVars, TypeRange hasDeviceAddrTypes,
-    ValueRange hostEvalVars, TypeRange hostEvalTypes,
-    ValueRange inReductionVars, TypeRange inReductionTypes,
-    DenseBoolArrayAttr inReductionByref, ArrayAttr inReductionSyms,
-    ValueRange mapVars, TypeRange mapTypes, ValueRange privateVars,
-    TypeRange privateTypes, ArrayAttr privateSyms, UnitAttr privateNeedsBarrier,
-    DenseI64ArrayAttr privateMaps) {
+static void printTargetOpRegion(OpAsmPrinter &p, Operation *op, Region &region,
+                                ValueRange hasDeviceAddrVars,
+                                TypeRange hasDeviceAddrTypes,
+                                ValueRange hostEvalVars,
+                                TypeRange hostEvalTypes, ValueRange mapVars,
+                                TypeRange mapTypes, ValueRange privateVars,
+                                TypeRange privateTypes, ArrayAttr privateSyms,
+                                UnitAttr privateNeedsBarrier,
+                                DenseI64ArrayAttr privateMaps) {
   AllRegionPrintArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, privateMaps);
@@ -2579,8 +2648,7 @@ LogicalResult TargetUpdateOp::verify() {
 void TargetOp::build(OpBuilder &builder, OperationState &state,
                      const TargetExtOperands &clauses) {
   MLIRContext *ctx = builder.getContext();
-  // TODO Store clauses in op: allocateVars, allocatorVars, inReductionVars,
-  // inReductionByref, inReductionSyms.
+  // TODO Store clauses in op: allocateVars, allocatorVars.
   TargetOp::build(
       builder, state, /*allocate_vars=*/{}, /*allocator_vars=*/{},
       makeArrayAttr(ctx, clauses.dependKinds), clauses.dependVars,
@@ -2588,9 +2656,10 @@ void TargetOp::build(OpBuilder &builder, OperationState &state,
       clauses.device, clauses.dynGroupprivateAccessGroup,
       clauses.dynGroupprivateFallback, clauses.dynGroupprivateSize,
       clauses.hasDeviceAddrVars, clauses.hostEvalVars, clauses.ifExpr,
-      /*in_reduction_vars=*/{}, /*in_reduction_byref=*/nullptr,
-      /*in_reduction_syms=*/nullptr, clauses.isDevicePtrVars, clauses.mapVars,
-      clauses.mapIterated, clauses.nowait, clauses.privateVars,
+      clauses.inReductionVars,
+      makeDenseBoolArrayAttr(ctx, clauses.inReductionByref),
+      makeArrayAttr(ctx, clauses.inReductionSyms), clauses.isDevicePtrVars,
+      clauses.mapVars, clauses.mapIterated, clauses.nowait, clauses.privateVars,
       makeArrayAttr(ctx, clauses.privateSyms), clauses.privateNeedsBarrier,
       clauses.threadLimitVars, /*private_maps=*/nullptr, clauses.kernelType);
 }
@@ -2631,6 +2700,19 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op. Block arguments have no defining op, so they
+/// match only by exact identity. Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr, so exact
+/// value identity is not required.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 LogicalResult TargetOp::verify() {
   if (getKernelType() == TargetExecMode::bare && !isCombined())
     return emitOpError() << "bare kernel requires 'omp.combined'";
@@ -2655,6 +2737,31 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host rewrites that map argument to
+  // the reduction-private storage. Require every `in_reduction` operand to be
+  // captured by at least one `map_entries` entry.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      // verifyMapClause guarantees every map operand is defined by a MapInfoOp.
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
+    }
+    if (!captured)
+      return emitOpError() << "in_reduction variable must be captured by a "
+                              "matching map_entries entry";
+  }
+
   return verifyPrivateVarsMapping(*this);
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b2d2ef6d8f770..b27685492926a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -364,9 +364,40 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       op.emitWarning("hint clause discarded");
   };
   auto checkInReduction = [&todo](auto op, LogicalResult &result) {
-    if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
-        op.getInReductionSyms())
+    if (isa<omp::TargetOp, omp::TaskOp, omp::TaskloopContextOp>(
+            op.getOperation())) {
+      if (auto byrefAttr = op.getInReductionByref()) {
+        for (bool isByRef : *byrefAttr) {
+          if (isByRef) {
+            result = todo("in_reduction with byref modifier");
+            return;
+          }
+        }
+      }
+      if (isa<omp::TargetOp>(op.getOperation())) {
+        if (auto inReductionSyms = op.getInReductionSyms()) {
+          for (auto sym :
+               (*inReductionSyms).template getAsRange<SymbolRefAttr>()) {
+            auto decl =
+                SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                    op, sym);
+            assert(decl &&
+                   "symbol resolution should be guaranteed by the op verifier");
+            if (decl.getInitializerRegion().front().getNumArguments() != 1) {
+              result = todo("in_reduction with two-argument initializer");
+              return;
+            }
+            if (!decl.getCleanupRegion().empty()) {
+              result = todo("in_reduction with cleanup region");
+              return;
+            }
+          }
+        }
+      }
+    } else if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
+               op.getInReductionSyms()) {
       result = todo("in_reduction");
+    }
   };
   auto checkNowait = [&todo](auto op, LogicalResult &result) {
     if (op.getNowait())
@@ -423,14 +454,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
           return;
         }
   };
-  auto checkInReductionByref = [&todo](auto op, LogicalResult &result) {
-    if (auto byrefAttr = op.getInReductionByref())
-      for (bool isByRef : *byrefAttr)
-        if (isByRef) {
-          result = todo("in_reduction with byref modifier");
-          return;
-        }
-  };
   auto checkNumTeams = [&todo](auto op, LogicalResult &result) {
     if (op.hasNumTeamsMultiDim())
       result = todo("num_teams with multi-dimensional values");
@@ -482,7 +505,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
       })
       .Case([&](omp::TaskgroupOp op) {
         checkAllocate(op, result);
@@ -494,7 +517,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
         checkReduction(op, result);
         checkReductionByref(op, result);
       })
@@ -8493,6 +8516,58 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand.
+      // By this translation stage HLFIR has been lowered away, so the
+      // in_reduction operand and the capturing entry's map var_ptr are the same
+      // SSA value; compare them directly. verifyMapClause guarantees every map
+      // operand is defined by an omp.map.info. The verifier guarantees at least
+      // one match exists; more than one matching entry is a lowering ambiguity
+      // (the redirect cannot pick which map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (v != mapInfo.getVarPtr())
+          continue;
+        if (matchIdx)
+          return targetOp.emitError()
+                 << "in_reduction variable on omp.target has multiple matching "
+                    "map_entries entries; the redirect target is ambiguous";
+        matchIdx = idx;
+      }
+      // The verifier requires a capturing map entry for every in_reduction
+      // operand, so a match must exist here.
+      assert(matchIdx &&
+             "TargetOp verifier guarantees a matching map_entries entry for "
+             "each in_reduction variable");
+      inRedMapArgIdx.push_back(*matchIdx);
+      // The runtime `orig` pointer is the in_reduction operand itself, the
+      // reduction variable the enclosing taskgroup registered.
+      inRedOrigPtrs.push_back(moduleTranslation.lookupValue(v));
+    }
+  }
+
   // For some private variables, the MapsForPrivatizedVariablesPass
   // creates MapInfoOp instances. Go through the private variables and
   // the mapped variables so that during codegeneration we are able
@@ -8568,8 +8643,15 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
         attr.isStringAttribute())
       llvmOutlinedFn->addFnAttr(attr);
 
-    for (auto [arg, mapOp] : llvm::zip_equal(mapBlockArgs, mapVars)) {
-      auto mapInfoOp = cast<omp::MapInfoOp>(mapOp.getDefiningOp());
+    for (auto [idx, arg] : llvm::enumerate(mapBlockArgs)) {
+      // in_reduction list items on omp.target are accessed through their
+      // map_entries block argument, which is redirected below to the per-task
+      // reduction-private storage returned by the runtime. Skip the default
+      // host-value mapping for those block arguments so the write-once
+      // mapValue mapping is free to be set to the private pointer.
+      if (llvm::is_contained(inRedMapArgIdx, idx))
+        continue;
+      auto mapInfoOp = cast<omp::MapInfoOp>(mapVars[idx].getDefiningOp());
       llvm::Value *mapOpValue =
           moduleTranslation.lookupValue(mapInfoOp.getVarPtr());
       moduleTranslation.mapValue(arg, mapOpValue);
@@ -8605,6 +8687,37 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // Collect, per item, the type the private pointer must have (the map
+      // block argument's type), and, through the callback, rebind the map block
+      // argument that stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
+      SmallVector<llvm::Type *> inRedResultPtrTys;
+      inRedResultPtrTys.reserve(inRedMapArgIdx.size());
+      for (unsigned mapArgIdx : inRedMapArgIdx)
+        inRedResultPtrTys.push_back(
+            moduleTranslation.convertType(mapBlockArgs[mapArgIdx].getType()));
+
+      llvm::OpenMPIRBuilder::LocationDescription bodyLoc(builder);
+      llvm::OpenMPIRBuilder::InsertPointTy redIP =
+          ompBuilder->createTargetInReduction(
+              bodyLoc, inRedOrigPtrs, inRedResultPtrTys,
+              [&](unsigned idx, llvm::Value *priv) {
+                moduleTranslation.mapValue(mapBlockArgs[inRedMapArgIdx[idx]],
+                                           priv);
+              });
+      builder.restoreIP(redIP);
+    }
+
     LLVM::ModuleTranslation::SaveStack<OpenMPAllocStackFrame> frame(
         moduleTranslation, allocaIP, deallocBlocks);
     llvm::Expected<llvm::BasicBlock *> exitBlock = convertOmpOpRegions(
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
index 015e901126a76..8108a1af31e21 100644
--- a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -348,8 +348,12 @@ module attributes {omp.is_target_device = true} {
       omp.terminator
     }
 
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
-    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+    // The `in_reduction` list item on `omp.target` has no dedicated block
+    // argument; it is captured by a matching `map_entries` entry.
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 : !llvm.ptr) map_entries(%1 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
       omp.terminator
     }
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 2db7f953fa3bb..57270ec0694c3 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3255,6 +3255,91 @@ func.func @omp_target_depend(%data_var: memref<i32>) {
 
 // -----
 
+func.func @omp_target_in_reduction_unresolved(%ptr: !llvm.ptr) {
+  // expected-error @below {{op expected symbol reference @add_f32 to point to a reduction declaration}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_duplicate(%ptr: !llvm.ptr) {
+  // expected-error @below {{op accumulator variable used more than once}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr, @add_f32 %ptr : !llvm.ptr, !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg: i32):
+  %0 = arith.constant 0 : i32
+  omp.yield (%0 : i32)
+}
+combiner {
+^bb1(%arg0: i32, %arg1: i32):
+  %1 = arith.addi %arg0, %arg1 : i32
+  omp.yield (%1 : i32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> i32
+  llvm.atomicrmw add %arg2, %2 monotonic : !llvm.ptr, i32
+  omp.yield
+}
+
+func.func @omp_target_in_reduction_type_mismatch(%mem: memref<1xf32>) {
+  // expected-error @below {{op expected accumulator ('memref<1xf32>') to be the same type as reduction declaration ('!llvm.ptr')}}
+  omp.target kernel_type(generic) in_reduction(@add_i32 %mem : memref<1xf32>) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_missing_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_missing_map(%ptr: !llvm.ptr) {
+  // An in_reduction operand on omp.target has no dedicated block argument; it
+  // is accessed inside the body through a matching map_entries block argument.
+  // With no map_entries entry capturing it, the verifier must reject the op.
+  // expected-error @below {{in_reduction variable must be captured by a matching map_entries entry}}
+  omp.target kernel_type(generic) in_reduction(@add_missing_map_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
 func.func @omp_distribute_schedule(%chunk_size : i32, %lb : i32, %ub : i32, %step : i32) -> () {
   // expected-error @below {{op chunk size set without dist_schedule_static being present}}
   "omp.distribute"(%chunk_size) <{operandSegmentSizes = array<i32: 0, 0, 1, 0>}> ({
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
new file mode 100644
index 0000000000000..0e9611c27ddcc
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
@@ -0,0 +1,51 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// in_reduction on omp.target compiled for the target *device*. From the
+// device's perspective an in_reduction list item behaves as a regular mapped
+// variable: the host-only redirect to the per-task reduction-private storage
+// (the __kmpc_global_thread_num + __kmpc_task_reduction_get_th_data lookup) is
+// guarded by !isTargetDevice and is therefore not emitted. Verify the device
+// body accesses the mapped variable directly and that none of the host
+// runtime lookup is present.
+
+module attributes {dlti.dl_spec = #dlti.dl_spec<"dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  omp.declare_reduction @add_i32 : i32
+  init {
+  ^bb0(%arg0: i32):
+    %c0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%c0 : i32)
+  }
+  combiner {
+  ^bb0(%arg0: i32, %arg1: i32):
+    %s = llvm.add %arg0, %arg1 : i32
+    omp.yield(%s : i32)
+  }
+
+  llvm.func @target_inreduction_device(%x : !llvm.ptr) {
+    %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+      %v = llvm.load %marg : !llvm.ptr -> i32
+      %c1 = llvm.mlir.constant(1 : i32) : i32
+      %s = llvm.add %v, %c1 : i32
+      llvm.store %s, %marg : i32, !llvm.ptr
+      omp.terminator
+    }
+    llvm.return
+  }
+}
+
+// The device kernel receives the mapped pointer as a captured argument and
+// accesses the in_reduction variable directly through it.
+// CHECK-LABEL: define {{.*}}amdgpu_kernel void @__omp_offloading_{{.*}}_target_inreduction_device_
+// CHECK-SAME:    (ptr %[[ARG:.+]], ptr %{{.+}})
+// CHECK:         store ptr %[[ARG]], ptr %[[ALLOCA:.+]]
+// CHECK:         %[[MAPPED:.+]] = load ptr, ptr %[[ALLOCA]]
+// The host-only in_reduction redirect must not run on the device: there is no
+// gtid lookup, no __kmpc_task_reduction_get_th_data call and no private copy.
+// CHECK-NOT:     @__kmpc_global_thread_num
+// CHECK-NOT:     @__kmpc_task_reduction_get_th_data
+// CHECK-NOT:     omp.inred.priv
+// The body reads and writes the mapped variable in place.
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[MAPPED]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[MAPPED]]
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
new file mode 100644
index 0000000000000..c0df1a4e1d332
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Multiple in_reduction items on omp.target. Each item is captured into the
+// target region through its own map_entries entry and accessed inside the
+// body via the corresponding map_entries block argument. On the host, every
+// item performs an independent
+// __kmpc_task_reduction_get_th_data lookup using its own captured original
+// pointer, and the returned per-task private pointer is bound to that item's
+// map block argument. This test pins down the pairing so it cannot pass if the
+// two items were swapped or collapsed onto a single pointer.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_multi(%x : !llvm.ptr, %y : !llvm.ptr) {
+  %mx = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %my = omp.map.info var_ptr(%y : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x, @add_i32 %y : !llvm.ptr, !llvm.ptr)
+      map_entries(%mx -> %mxarg, %my -> %myarg : !llvm.ptr, !llvm.ptr) {
+    // First item (x): load, += 1, store back.
+    %vx = llvm.load %mxarg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %sx = llvm.add %vx, %c1 : i32
+    llvm.store %sx, %mxarg : i32, !llvm.ptr
+    // Second item (y): load, += 2, store back.
+    %vy = llvm.load %myarg : !llvm.ptr -> i32
+    %c2 = llvm.mlir.constant(2 : i32) : i32
+    %sy = llvm.add %vy, %c2 : i32
+    llvm.store %sy, %myarg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards both captured pointers into the outlined target
+// kernel (the trailing argument is the unused descriptor slot).
+// CHECK-LABEL: define void @target_inreduction_multi(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_multi_{{.*}}(ptr %{{.+}}, ptr %{{.+}}, ptr null)
+
+// The two captured original pointers arrive as distinct kernel arguments.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_multi_
+// CHECK-SAME:    (ptr %[[CAPTX:.+]], ptr %[[CAPTY:.+]], ptr %{{.+}})
+
+// A single gtid is computed once for the whole target body and shared by both
+// lookups; each item then performs its own __kmpc_task_reduction_get_th_data
+// call against its own captured pointer, and the returned per-task private
+// pointer is bound to that item's map block argument.
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIVX:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTX]])
+// CHECK:         %[[PRIVY:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTY]])
+
+// The first item's private storage is the base of the +1 load/store; the
+// CHECK-NOT below ensures the second item's pointer is not touched until the
+// first item's accumulation has completed (i.e. the items are not swapped or
+// merged onto a single private pointer).
+// CHECK:         %[[LX:.+]] = load i32, ptr %[[PRIVX]]
+// CHECK:         %[[SX:.+]] = add i32 %[[LX]], 1
+// CHECK-NOT:     %[[PRIVY]]
+// CHECK:         store i32 %[[SX]], ptr %[[PRIVX]]
+
+// The second item's private storage is the base of the +2 load/store.
+// CHECK:         %[[LY:.+]] = load i32, ptr %[[PRIVY]]
+// CHECK:         %[[SY:.+]] = add i32 %[[LY]], 2
+// CHECK:         store i32 %[[SY]], ptr %[[PRIVY]]
+
+// Exactly two reduction lookups are emitted; no third call sneaks in. The
+// `call` form is used so this does not match the runtime declaration.
+// CHECK-NOT:     call ptr @__kmpc_task_reduction_get_th_data
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
new file mode 100644
index 0000000000000..5e52ee00efafa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+// in_reduction on omp.target: the in_reduction variable is also captured
+// into the target region as a map entry (a frontend is expected to emit this
+// implicit map). The in_reduction clause does not define an entry block
+// argument; inside the target body the variable is accessed through its
+// map_entries block argument. The captured pointer is passed to
+// __kmpc_task_reduction_get_th_data with a NULL descriptor; the runtime
+// walks enclosing taskgroups to locate the matching task_reduction
+// registration. The returned per-task private pointer is bound to the
+// map_entries block argument so subsequent loads/stores inside the region
+// use the private copy.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+    %v = llvm.load %marg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards the captured pointer into the outlined target
+// kernel.
+// CHECK-LABEL: define void @target_inreduction(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_{{.*}}(ptr %{{.+}}, ptr null)
+
+// In the outlined target body the in_reduction private pointer is
+// obtained from the runtime using the captured original pointer; that
+// pointer is then the base of the load and store inside the region.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_
+// CHECK-SAME:    (ptr %[[CAPT:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPT]])
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[PRIV]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+// -----
+
+// Same as the first case but the in_reduction variable lives in a non-default
+// address space (addrspace(1)). __kmpc_task_reduction_get_th_data is declared
+// to take and return a generic (default-AS) `ptr`, so the host
+// lowering must (1) addrspacecast the captured addrspace(1) original pointer to
+// generic `ptr` before the runtime lookup, and (2) addrspacecast the returned
+// generic private pointer back to addrspace(1) so the body's load/store use the
+// private copy in the right address space. This pins down the address-space
+// handling so a regression that passed the addrspace(1) pointer straight into
+// the runtime call (a bad-signature crash) cannot pass.
+
+omp.declare_reduction @add_i32_as1 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_as1(%x : !llvm.ptr<1>) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr<1>, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr<1>
+  omp.target kernel_type(generic) in_reduction(@add_i32_as1 %x : !llvm.ptr<1>) map_entries(%m -> %marg : !llvm.ptr<1>) {
+    %v = llvm.load %marg : !llvm.ptr<1> -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr<1>
+    omp.terminator
+  }
+  llvm.return
+}
+
+// In the outlined target body the addrspace(1) original pointer is normalized
+// to generic `ptr` before the NULL-descriptor in_reduction lookup, and the
+// returned generic private pointer is cast back to addrspace(1) for the body's
+// load and store. The original captured pointer arrives as the addrspace(1)
+// kernel argument and is only used to derive the generic lookup pointer.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_as1_
+// CHECK-SAME:    (ptr addrspace(1) %[[CAPT_AS1:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID_AS1:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[ORIG_GEN:.+]] = addrspacecast ptr addrspace(1) %[[CAPT_AS1]] to ptr
+// CHECK:         %[[PRIV_AS1_GEN:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID_AS1]], ptr null, ptr %[[ORIG_GEN]])
+// CHECK:         %[[PRIV_AS1:.+]] = addrspacecast ptr %[[PRIV_AS1_GEN]] to ptr addrspace(1)
+// CHECK:         %[[LOADED_AS1:.+]] = load i32, ptr addrspace(1) %[[PRIV_AS1]]
+// CHECK:         %[[SUM_AS1:.+]] = add i32 %[[LOADED_AS1]], 1
+// CHECK:         store i32 %[[SUM_AS1]], ptr addrspace(1) %[[PRIV_AS1]]
+
+// The body must not load or store through the original captured addrspace(1)
+// pointer; all accesses go through the runtime-returned private copy.
+// CHECK-NOT:     store i32 %{{.+}}, ptr addrspace(1) %[[CAPT_AS1]]
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..943b5bad6970f 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -218,10 +218,95 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @target_in_reduction(%x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause in_reduction in omp.target operation}}
+llvm.func @target_in_reduction_byref(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with byref modifier in omp.target operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.target}}
-  omp.target kernel_type(generic) in_reduction(@add_f32 %x -> %prv : !llvm.ptr) {
+  omp.target kernel_type(generic) in_reduction(byref @add_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_cleanup_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+cleanup {
+^bb2(%arg2: f32):
+  omp.yield
+}
+llvm.func @target_in_reduction_cleanup(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with cleanup region in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_cleanup_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_two_arg_init_i32 : !llvm.ptr alloc {
+^bb0(%arg: !llvm.ptr):
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+  omp.yield(%1 : !llvm.ptr)
+} init {
+^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.store %0, %arg1 : i32, !llvm.ptr
+  omp.yield(%arg1 : !llvm.ptr)
+} combiner {
+^bb1(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.load %arg0 : !llvm.ptr -> i32
+  %1 = llvm.load %arg1 : !llvm.ptr -> i32
+  %2 = llvm.add %0, %1 : i32
+  llvm.store %2, %arg0 : i32, !llvm.ptr
+  omp.yield(%arg0 : !llvm.ptr)
+}
+llvm.func @target_in_reduction_two_arg_init(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with two-argument initializer in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_two_arg_init_i32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_dup_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb0(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+llvm.func @target_in_reduction_duplicate_map(%x : !llvm.ptr) {
+  // The in_reduction variable %x has two matching map_entries entries for the
+  // same var_ptr. The translation cannot disambiguate which map block argument
+  // to redirect, so it must reject this as ambiguous.
+  %m1 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %m2 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{in_reduction variable on omp.target has multiple matching map_entries entries; the redirect target is ambiguous}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_dup_map_f32 %x : !llvm.ptr) map_entries(%m1 -> %arg1, %m2 -> %arg2 : !llvm.ptr, !llvm.ptr) {
     omp.terminator
   }
   llvm.return


From llvmlistbot at llvm.org  Mon Jul 13 07:28:09 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Mon, 13 Jul 2026 07:28:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a54f5f9.824e11a6.f5f27.7a8a@mx.google.com>


AGindinson wrote:

I would merge this PR tomorrow at the latest, given that it has been approved by 2 maintainers.

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Mon Jul 13 07:29:33 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:29:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54f64d.743400b0.2067c9.d840@mx.google.com>


https://github.com/Saieiei edited https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:29:53 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Mon, 13 Jul 2026 07:29:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] add isEffectivelySerial() to
 acc.parallel and acc.kernels (PR #209192)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209192@github.com>
Message-ID: <6a54f661.d1496011.26913c.800c@mx.google.com>


https://github.com/rscottmanley updated https://github.com/llvm/llvm-project/pull/209192

>From fd4c12a15c2a009640dc88cdb0e5a44c6008e013 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:13:39 -0700
Subject: [PATCH 1/3] [OpenACC] add isEffectivelySerial() to acc.parallel and
 acc.kernels

Mirror the isEffectivelySerial() function in acc.compute_region to
acc.parallel and acc.kernel so this check is unified anywhere it needs
to be used. Should be NFC
---
 .../mlir/Dialect/OpenACC/OpenACCOps.td        |  6 +++
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp     | 35 ++++++++++++++++
 .../OpenACC/Transforms/ACCComputeLowering.cpp | 41 ++-----------------
 3 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
index 30298767c0f3e..db2f0e0605db5 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOps.td
@@ -1842,6 +1842,9 @@ def OpenACC_ParallelOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
@@ -2173,6 +2176,9 @@ def OpenACC_KernelsOp
     /// clause for the given device_type.
     bool hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType);
 
+    /// Check whether num_gangs, num_workers, and vector_length are all 1.
+    bool isEffectivelySerial();
+
     /// Return true if the op has the wait attribute for the
     /// mlir::acc::DeviceType::None device_type.
     bool hasWaitOnly();
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 7de59058ead0d..3b168b7631db8 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -14,6 +14,7 @@
 //
 //===----------------------------------------------------------------------===//
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
@@ -571,6 +572,40 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
+/// Strip index_cast operations from a value before checking for a constant.
+static Value stripIndexCasts(Value val) {
+  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+    val = castOp.getIn();
+  return val;
+}
+
+template <typename ComputeOpT>
+static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  auto numGangs = op.getNumGangsValues();
+  if (numGangs.empty())
+    return false;
+  for (Value gangSize : numGangs) {
+    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
+      return false;
+  }
+  Value numWorkers = op.getNumWorkersValue();
+  if (!numWorkers)
+    return false;
+  Value vectorLength = op.getVectorLengthValue();
+  if (!vectorLength)
+    return false;
+  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
+         isConstantIntValue(stripIndexCasts(vectorLength), 1);
+}
+
+bool ParallelOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
+bool KernelsOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool ComputeRegionOp::isEffectivelySerial() {
   auto *ctx = getContext();
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
index b3d54d2e05697..a275a794c9fd5 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCComputeLowering.cpp
@@ -78,41 +78,6 @@ namespace {
 // Helper functions
 //===----------------------------------------------------------------------===//
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
-template <typename ComputeOpT>
-static bool isGangWorkerVectorAllOne(ComputeOpT op) {
-  auto numGangs = op.getNumGangsValues();
-  if (numGangs.empty())
-    return false;
-  for (Value gangSize : numGangs) {
-    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
-      return false;
-  }
-  Value numWorkers = op.getNumWorkersValue();
-  if (!numWorkers)
-    return false;
-  Value vectorLength = op.getVectorLengthValue();
-  if (!vectorLength)
-    return false;
-  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
-         isConstantIntValue(stripIndexCasts(vectorLength), 1);
-}
-
-/// A compute construct is "effectively serial" when it specifies
-/// num_gangs(1), num_workers(1), and vector_length(1). This is because
-/// these are the only parallelism dimensions expressible from OpenACC spec
-/// point-of-view and is consistent with how `serial` semantics are defined.
-template <typename ComputeOpT>
-static bool isEffectivelySerial(ComputeOpT op) {
-  return isGangWorkerVectorAllOne(op);
-}
-
 static bool isOpInComputeRegion(Operation *op) {
   Region *region = op->getBlock()->getParent();
   return getEnclosingComputeOp(*region) != nullptr;
@@ -120,9 +85,9 @@ static bool isOpInComputeRegion(Operation *op) {
 
 static bool isOpInSerialRegion(Operation *op) {
   if (auto parallelOp = op->getParentOfType<ParallelOp>())
-    return isEffectivelySerial(parallelOp);
+    return parallelOp.isEffectivelySerial();
   if (auto kernelsOp = op->getParentOfType<KernelsOp>())
-    return isEffectivelySerial(kernelsOp);
+    return kernelsOp.isEffectivelySerial();
   if (op->getParentOfType<SerialOp>())
     return true;
   if (auto computeRegion = op->getParentOfType<ComputeRegionOp>())
@@ -230,7 +195,7 @@ assignKnownLaunchArgs(ComputeConstructT computeOp, DeviceType deviceType,
     return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
   } else if constexpr (llvm::is_one_of<ComputeConstructT, ParallelOp,
                                        KernelsOp>::value) {
-    if (isEffectivelySerial(computeOp))
+    if (computeOp.isEffectivelySerial())
       return {ParWidthOp::create(rewriter, loc, Value(), policy.seqDim(ctx))};
 
     deviceType = getParDimsDeviceType(computeOp, deviceType);

>From db3c29b6b0fd8452eebbab0c273d69668d9a7d1e Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:21:28 -0700
Subject: [PATCH 2/3] move strip index casts to lambda

---
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 3b168b7631db8..8cb2be720c29e 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -572,15 +572,15 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
-/// Strip index_cast operations from a value before checking for a constant.
-static Value stripIndexCasts(Value val) {
-  while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-    val = castOp.getIn();
-  return val;
-}
-
 template <typename ComputeOpT>
 static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  // Strip index_cast operations from a value before checking for a constant.
+  auto stripIndexCasts = [](Value val) -> Value {
+    while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+      val = castOp.getIn();
+    return val;
+  };
+
   auto numGangs = op.getNumGangsValues();
   if (numGangs.empty())
     return false;

>From 0cc933527e3865a07a046be1a98156e46134e593 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:29:17 -0700
Subject: [PATCH 3/3] move isGangWorkerVectorAllOne into OpenACC.h

---
 mlir/include/mlir/Dialect/OpenACC/OpenACC.h | 26 ++++++++++++++++
 mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp     |  8 +++++
 mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp   | 34 ---------------------
 3 files changed, 34 insertions(+), 34 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACC.h b/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
index 650f6f780273f..4abc3971f12c0 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACC.h
@@ -238,6 +238,32 @@ struct CurrentDeviceIdResource
   bool isAddressable() const override { return false; }
 };
 
+template <typename ComputeOpT>
+static bool isGangWorkerVectorAllOne(ComputeOpT op) {
+  // Strip index_cast operations from a value before checking for a constant.
+  auto stripIndexCasts = [](Value val) -> Value {
+    while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
+      val = castOp.getIn();
+    return val;
+  };
+
+  auto numGangs = op.getNumGangsValues();
+  if (numGangs.empty())
+    return false;
+  for (Value gangSize : numGangs) {
+    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
+      return false;
+  }
+  Value numWorkers = op.getNumWorkersValue();
+  if (!numWorkers)
+    return false;
+  Value vectorLength = op.getVectorLengthValue();
+  if (!vectorLength)
+    return false;
+  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
+         isConstantIntValue(stripIndexCasts(vectorLength), 1);
+}
+
 } // namespace acc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
index d793645e49eca..0205afc1dfedb 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
@@ -2234,6 +2234,10 @@ bool acc::ParallelOp::hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType) {
       getVectorLength(), deviceType);
 }
 
+bool acc::ParallelOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool acc::ParallelOp::hasWaitOnly() {
   return hasWaitOnly(mlir::acc::DeviceType::None);
 }
@@ -3107,6 +3111,10 @@ bool acc::KernelsOp::hasAnyGangWorkerVector(mlir::acc::DeviceType deviceType) {
       getVectorLength(), deviceType);
 }
 
+bool acc::KernelsOp::isEffectivelySerial() {
+  return isGangWorkerVectorAllOne(*this);
+}
+
 bool acc::KernelsOp::hasWaitOnly() {
   return hasWaitOnly(mlir::acc::DeviceType::None);
 }
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
index 8cb2be720c29e..4f112b4002427 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACCCG.cpp
@@ -572,40 +572,6 @@ ComputeRegionOp::wireHoistedValueThroughIns(Value value) {
   return arg;
 }
 
-template <typename ComputeOpT>
-static bool isGangWorkerVectorAllOne(ComputeOpT op) {
-  // Strip index_cast operations from a value before checking for a constant.
-  auto stripIndexCasts = [](Value val) -> Value {
-    while (auto castOp = val.getDefiningOp<arith::IndexCastOp>())
-      val = castOp.getIn();
-    return val;
-  };
-
-  auto numGangs = op.getNumGangsValues();
-  if (numGangs.empty())
-    return false;
-  for (Value gangSize : numGangs) {
-    if (!isConstantIntValue(stripIndexCasts(gangSize), 1))
-      return false;
-  }
-  Value numWorkers = op.getNumWorkersValue();
-  if (!numWorkers)
-    return false;
-  Value vectorLength = op.getVectorLengthValue();
-  if (!vectorLength)
-    return false;
-  return isConstantIntValue(stripIndexCasts(numWorkers), 1) &&
-         isConstantIntValue(stripIndexCasts(vectorLength), 1);
-}
-
-bool ParallelOp::isEffectivelySerial() {
-  return isGangWorkerVectorAllOne(*this);
-}
-
-bool KernelsOp::isEffectivelySerial() {
-  return isGangWorkerVectorAllOne(*this);
-}
-
 bool ComputeRegionOp::isEffectivelySerial() {
   auto *ctx = getContext();
 


From llvmlistbot at llvm.org  Mon Jul 13 07:46:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 07:46:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a54fa3b.66c0679a.3aa103.27ee@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Kigyosi Alexandru (akigyosi)

<details>
<summary>Changes</summary>

Adds a Rerun control value to ExecutionContext::Control that allows the re-execution of the current action immediately after it completes, without restarting the full compilation pipeline. This is analogous to GDB's ability to restart execution from a breakpoint. When the callback returns Rerun, the action is executed normally, then re-dispatched through the full ExecutionContext::operator() pipeline, including breakpoint matching, so the user gets a fresh opportunity to inspect or control the re-execution. As a practical usage example, a breakpoint + an observer can be added, to save and restore IR between runs to check if each run produces the same IR or something different each time. A depth-keyed structure is used, so rerun requests survive nested action dispatch and are consumed by the correct stack frame.

---
Full diff: https://github.com/llvm/llvm-project/pull/209197.diff


3 Files Affected:

- (modified) mlir/include/mlir/Debug/ExecutionContext.h (+5-1) 
- (modified) mlir/lib/Debug/ExecutionContext.cpp (+21-1) 
- (modified) mlir/unittests/Debug/ExecutionContextTest.cpp (+27) 


``````````diff
diff --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
index fbad04b9a2f1d..e42d61087f22d 100644
--- a/mlir/include/mlir/Debug/ExecutionContext.h
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -65,7 +65,8 @@ class ExecutionContext {
   /// - Finish: The action is executed and the execution is paused only when we
   ///           reach the parent/enclosing operation. If there are no enclosing
   ///           operation, the execution continues without stopping.
-  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5 };
+  /// - Rerun: The action gets executed again immediately after it finishes.
+  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5, Rerun = 6 };
 
   /// The type of the callback that is used to control the execution.
   /// The callback is passed the current action.
@@ -134,6 +135,9 @@ class ExecutionContext {
 
   /// The list of managers that are queried for breakpoints.
   SmallVector<BreakpointManager *> breakpoints;
+
+  /// Stack of depths of actions that should be rerun.
+  SmallVector<int> rerunControlStack;
 };
 
 } // namespace tracing
diff --git a/mlir/lib/Debug/ExecutionContext.cpp b/mlir/lib/Debug/ExecutionContext.cpp
index 7fc5c165d391e..cc72bac549ab0 100644
--- a/mlir/lib/Debug/ExecutionContext.cpp
+++ b/mlir/lib/Debug/ExecutionContext.cpp
@@ -84,10 +84,25 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
     case ExecutionContext::Finish:
       depthToBreak = depth - 1;
       return true;
+    case ExecutionContext::Rerun:
+      depthToBreak = std::nullopt;
+      rerunControlStack.push_back(depth);
+      return true;
     }
     llvm::report_fatal_error("Unknown control request");
   };
 
+  auto rerunCurrentActionIfRequested = [&]() -> bool {
+    if (!rerunControlStack.empty() && rerunControlStack.back() == depth) {
+      // If the user requested to rerun this action, we do it here.
+      rerunControlStack.pop_back();
+      actionStack = info.getParent();
+      (*this)(transform, action);
+      return true;
+    }
+    return false;
+  };
+
   // Try to find a breakpoint that would hit on this action.
   // Right now there is no way to collect them all, we stop at the first one.
   for (auto *breakpointManager : breakpoints) {
@@ -116,6 +131,11 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
       observer->afterExecute(actionStack);
   }
 
-  if (depthToBreak && depth <= depthToBreak)
+  if (rerunCurrentActionIfRequested())
+    return;
+
+  if (depthToBreak && depth <= depthToBreak) {
     handleUserInput();
+    rerunCurrentActionIfRequested();
+  }
 }
diff --git a/mlir/unittests/Debug/ExecutionContextTest.cpp b/mlir/unittests/Debug/ExecutionContextTest.cpp
index 642adff51002a..16c785b6f826e 100644
--- a/mlir/unittests/Debug/ExecutionContextTest.cpp
+++ b/mlir/unittests/Debug/ExecutionContextTest.cpp
@@ -349,4 +349,31 @@ TEST(ExecutionContext, EnableDisableBreakpointOnCallback) {
   executionCtx(original, DebuggerAction{});
   EXPECT_EQ(counter, 4);
 }
+
+TEST(ExecutionContext, RerunRequestedFromPostActionCallback) {
+  // If rerun is requested from the post-action callback path,
+  // the same action must be rerun immediately.
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Next, ExecutionContext::Rerun, ExecutionContext::Apply};
+  int idx = 0;
+  int callbackCounter = 0;
+  int executionCounter = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++callbackCounter;
+    EXPECT_EQ(backtrace->getAction().getTag(), DebuggerAction::tag);
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  auto callback = [&]() { ++executionCounter; };
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(callbackCounter, 3);
+  EXPECT_EQ(executionCounter, 2);
+}
 } // namespace

``````````

</details>


https://github.com/llvm/llvm-project/pull/209197

From llvmlistbot at llvm.org  Mon Jul 13 07:47:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 07:47:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a54fa7f.e55dc341.1f19bd.e816@mx.google.com>


weimin023 wrote:

@AGindinson Thanks a lot!

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Mon Jul 13 07:47:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 07:47:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a54fa8a.e624eb8d.39cd4e.7f04@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp,h -- mlir/include/mlir/Debug/ExecutionContext.h mlir/lib/Debug/ExecutionContext.cpp mlir/unittests/Debug/ExecutionContextTest.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
index e42d61087..93c26e0ce 100644
--- a/mlir/include/mlir/Debug/ExecutionContext.h
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -66,7 +66,14 @@ public:
   ///           reach the parent/enclosing operation. If there are no enclosing
   ///           operation, the execution continues without stopping.
   /// - Rerun: The action gets executed again immediately after it finishes.
-  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5, Rerun = 6 };
+  enum Control {
+    Apply = 1,
+    Skip = 2,
+    Step = 3,
+    Next = 4,
+    Finish = 5,
+    Rerun = 6
+  };
 
   /// The type of the callback that is used to control the execution.
   /// The callback is passed the current action.

``````````

</details>


https://github.com/llvm/llvm-project/pull/209197

From llvmlistbot at llvm.org  Mon Jul 13 07:48:39 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:48:39 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fac7.481afadf.196d57.e51d@mx.google.com>


================
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
----------------
Saieiei wrote:

Done. Updated the target `in_reduction` terminology across the patch, commit message, PR title, and description to use host/host-side lowering instead of host fallback.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:48:55 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:48:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fad7.6537ae93.2f1c1b.7ea2@mx.google.com>


================
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host-fallback runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivate.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivate   Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivate);
----------------
Saieiei wrote:

Done!


https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:49:06 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:49:06 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fae2.9edb5c9d.208698.8c9b@mx.google.com>


================
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:50:30 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:50:30 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb36.d76d81af.39f9be.467e@mx.google.com>


================
@@ -1218,6 +1218,57 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivate) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item. Owning the gtid setup and the lookup loop here keeps the runtime
+  // generation inside OpenMPIRBuilder rather than the MLIR translation layer.
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:51:19 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:51:19 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb67.92d2873c.258297.153b@mx.google.com>


================
@@ -8435,6 +8458,20 @@ getDynGroupprivateFallbackType(omp::FallbackModifierAttr fallbackAttr) {
   llvm_unreachable("unexpected dyn_groupprivate fallback type");
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `omp::TargetOp::verify()` in OpenMPDialect.cpp so
+/// translation matches exactly what the verifier accepts.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:51:25 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:51:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb6d.a1accbb4.244acd.63da@mx.google.com>


================
@@ -8493,6 +8530,59 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand,
+      // using the same predicate as omp::TargetOp::verify(): exact same value,
+      // or another result of the same defining op. The verifier guarantees at
+      // least one match exists; more than one matching entry is a lowering
+      // ambiguity (the redirect cannot pick which map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (!mapInfo || !targetInReductionCapturedBy(v, mapInfo.getVarPtr()))
+          continue;
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:51:52 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:51:52 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb88.c5504292.28b3b8.aee0@mx.google.com>


================
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:51:55 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:51:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb8b.d76d81af.39f9be.68ee@mx.google.com>


================
@@ -2631,6 +2700,21 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op (block arguments, which have no defining op,
+/// match only by exact identity). Flang emits `hlfir.declare`#0 for the
+/// `in_reduction` operand and `hlfir.declare`#1 for the map var_ptr. Keep this
+/// predicate in sync with `targetInReductionCapturedBy` in
+/// OpenMPToLLVMIRTranslation.cpp so translation matches what the verifier
+/// accepts.
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:51:59 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:51:59 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb8f.a1accbb4.244acd.81d1@mx.google.com>


================
@@ -2655,6 +2739,35 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host fallback rewrites that map
+  // argument to the reduction-private storage. Require every `in_reduction`
+  // operand to be captured by at least one `map_entries` entry whose
+  // `MapInfoOp` var_ptr is the same value, or another result of the same
+  // defining op. Flang emits e.g. `hlfir.declare`#0 for the `in_reduction`
+  // operand and `hlfir.declare`#1 for the map var_ptr, so exact value identity
+  // is not required; block arguments (no defining op) must match exactly.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (mapInfo &&
+          targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 07:52:04 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Mon, 13 Jul 2026 07:52:04 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a54fb94.c5504292.28b3b8.b841@mx.google.com>


================
@@ -8605,6 +8702,42 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // OpenMPIRBuilder owns the runtime-generation path: it computes the
+      // target body gtid once (so several in_reduction items do not each emit a
+      // __kmpc_global_thread_num) and emits the per-item
+      // __kmpc_task_reduction_get_th_data lookup with the required
+      // address-space casts. The MLIR-specific work stays here: collect, per
+      // item, the type the private pointer must have (the map block argument's
+      // type), and, through the callback, rebind the map block argument that
+      // stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Mon Jul 13 08:51:24 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Mon, 13 Jul 2026 08:51:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a55097c.6c5720b3.15e90c.602f@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
----------------
egebeysel wrote:

That's cool that we can use masking and lower to SME with mmt4d, thanks for working on the patch that enabled it!

The thing is, these tests usually act as a documentation for people who are curious to know how we would lower these things. So, unless you have a specific use case where you would, I would very much prefer to have it with the flag passed, and without masking, since packing the operands already has padding semantics.

A natural use case where you might actually need to use masking despite packing would be if you were to bubble up the pack operation, i.e. propagate the layout to the producers. If you have something like this, we can add it as an additional test, but I'd rather keep this one lean and canonical if you don't mind.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 08:52:04 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Mon, 13 Jul 2026 08:52:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5509a4.6c5720b3.15e90c.70a7@mx.google.com>


https://github.com/egebeysel edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 08:53:32 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Mon, 13 Jul 2026 08:53:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5509fc.8e75e0cb.3dbce4.cdc7@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
----------------
steplong wrote:

Makes sense, I'll add the flag

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Mon Jul 13 09:48:42 2026
From: llvmlistbot at llvm.org (Krzysztof Drewniak)
Date: Mon, 13 Jul 2026 09:48:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a5516ea.743400b0.2067c9.1767@mx.google.com>


krzysz00 wrote:

False positive undef detector

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Mon Jul 13 10:46:21 2026
From: llvmlistbot at llvm.org (Matt Arsenault)
Date: Mon, 13 Jul 2026 10:46:21 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a55246d.492e23e4.14eda0.cbf1@mx.google.com>


================
@@ -1485,6 +1485,33 @@ def int_amdgcn_s_buffer_load : DefaultAttrsIntrinsic <
   [IntrNoMem, ImmArg<ArgIndex<2>>]>,
   AMDGPURsrcIntrinsic<0>;
 
+// Generate an s_buffer_load instruction, falling back to buffer_load if the offset
+// is not uniform. This operates like the s.buffer.load intrinsic, but uses a buffer
----------------
arsenm wrote:

the scalar cache isn't coherent with vector, it's difficult to migrate that direction 

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Mon Jul 13 11:00:41 2026
From: llvmlistbot at llvm.org (Krzysztof Drewniak)
Date: Mon, 13 Jul 2026 11:00:41 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a5527c9.ff037943.2e959a.02e6@mx.google.com>


================
@@ -1485,6 +1485,33 @@ def int_amdgcn_s_buffer_load : DefaultAttrsIntrinsic <
   [IntrNoMem, ImmArg<ArgIndex<2>>]>,
   AMDGPURsrcIntrinsic<0>;
 
+// Generate an s_buffer_load instruction, falling back to buffer_load if the offset
+// is not uniform. This operates like the s.buffer.load intrinsic, but uses a buffer
----------------
krzysz00 wrote:

I think s_buffer_load is documented weird, given that it's got those "no memory" semantics and all.

Both of these are an s_buffer_load when all the arguments are uniform and a buffer_load otherwise, I just wrote the docs on this one as "you're explicitly requesting a uniform load".

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Mon Jul 13 11:02:01 2026
From: llvmlistbot at llvm.org (Fabian Mora)
Date: Mon, 13 Jul 2026 11:02:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ROCm] Build mlir_rocm_runtime
	STANDALONE (PR #205651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205651@github.com>
Message-ID: <6a552819.92d2873c.258297.711a@mx.google.com>


https://github.com/fabianmcg approved this pull request.


https://github.com/llvm/llvm-project/pull/205651

From llvmlistbot at llvm.org  Mon Jul 13 12:48:23 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 13 Jul 2026 12:48:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Canonicalize `addf(negf(x),
 y)` to `subf(y, x)` (PR #209277)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209277@github.com>
Message-ID: <6a554107.92d2873c.258297.a8d3@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/209277

>From bcc58dcd530ad06ab73ee6c164ea16ce455804a6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Mon, 13 Jul 2026 07:25:08 -0700
Subject: [PATCH] [mlir][arith] Canonicalize addf(negf(x), y) to subf(y, x)

arith.addf had no canonicalization patterns at all. This adds a
canonicalizer rewriting addf(negf(x), y) -> subf(y, x) (and the commuted
operand order).

The rewrite is valid under any rounding mode, which is propagated to the
subf: negf is an exact sign flip, IEEE addition is commutative, and y - x
is the correctly rounded y + (-x), so subf(y, x) and addf(negf(x), y)
are bit-identical with identical exception behavior for every rounding
mode. Fast-math flags are propagated to the new subf.
---
 .../include/mlir/Dialect/Arith/IR/ArithOps.td |  1 +
 .../Dialect/Arith/IR/ArithCanonicalization.td | 18 +++++
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        |  5 ++
 mlir/test/Dialect/Arith/canonicalize.mlir     | 74 +++++++++++++++++++
 4 files changed, 98 insertions(+)

diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..54481d3232483 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -1113,6 +1113,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
     %a = arith.addf %b, %c to_nearest_even : f64
     ```
   }];
+  let hasCanonicalizer = 1;
   let hasFolder = 1;
 }
 
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index 86c9b6d3551d8..84deb2b920a85 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -521,6 +521,24 @@ def UIToFPOfExtUI :
     Pat<(Arith_UIToFPOp (Arith_ExtUIOp $x, $nneg1), $nneg2),
         (Arith_UIToFPOp $x, $nneg1)>;
 
+//===----------------------------------------------------------------------===//
+// AddFOp
+//===----------------------------------------------------------------------===//
+
+// addf(negf(x), y) -> subf(y, x)
+// addf(y, negf(x)) -> subf(y, x)
+//
+// Valid for any rounding mode, which is propagated to the subf: negf is an
+// exact sign flip, IEEE addition is commutative, and y - x is the correctly
+// rounded y + (-x), so subf(y, x) and addf(negf(x), y) are bit-identical.
+def AddFOfNegFLhs :
+    Pat<(Arith_AddFOp (Arith_NegFOp $x, $_), $y, $fmf, $rm),
+        (Arith_SubFOp $y, $x, $fmf, $rm)>;
+
+def AddFOfNegFRhs :
+    Pat<(Arith_AddFOp $y, (Arith_NegFOp $x, $_), $fmf, $rm),
+        (Arith_SubFOp $y, $x, $fmf, $rm)>;
+
 //===----------------------------------------------------------------------===//
 // SubFOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 397fa28227831..674577e5764d8 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -1235,6 +1235,11 @@ OpFoldResult arith::AddFOp::fold(FoldAdaptor adaptor) {
       });
 }
 
+void arith::AddFOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
+                                                MLIRContext *context) {
+  patterns.add<AddFOfNegFLhs, AddFOfNegFRhs>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // SubFOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index c12befc76675a..e273ff58921ef 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3005,6 +3005,80 @@ func.func @test_addf_rounding_mode(%arg0 : f32) -> (f32, f32, f32) {
 
 // -----
 
+// CHECK-LABEL: @test_addf_negf(
+//  CHECK-SAME: %[[ARG0:.+]]: f32, %[[ARG1:.+]]: f32
+func.func @test_addf_negf(%arg0 : f32, %arg1 : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[X:.+]] = arith.subf %[[ARG1]], %[[ARG0]] : f32
+  // CHECK-DAG:  %[[Y:.+]] = arith.subf %[[ARG0]], %[[ARG1]] : f32
+  // CHECK:      return %[[X]], %[[Y]]
+  %0 = arith.negf %arg0 : f32
+  %1 = arith.addf %0, %arg1 : f32
+  %2 = arith.negf %arg1 : f32
+  %3 = arith.addf %arg0, %2 : f32
+  return %1, %3 : f32, f32
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_types
+//  CHECK-SAME: %[[F16:.+]]: f16, %[[G16:.+]]: f16, %[[F64:.+]]: f64, %[[G64:.+]]: f64, %[[BF:.+]]: bf16, %[[GBF:.+]]: bf16,
+//  CHECK-SAME: %[[VEC:.+]]: vector<4xf32>, %[[GVEC:.+]]: vector<4xf32>,
+//  CHECK-SAME: %[[TEN:.+]]: tensor<2x3xf32>, %[[GTEN:.+]]: tensor<2x3xf32>
+func.func @test_addf_negf_types(
+    %f16 : f16, %g16 : f16, %f64 : f64, %g64 : f64, %bf : bf16, %gbf : bf16,
+    %vec : vector<4xf32>, %gvec : vector<4xf32>,
+    %ten : tensor<2x3xf32>, %gten : tensor<2x3xf32>)
+    -> (f16, f64, bf16, vector<4xf32>, tensor<2x3xf32>) {
+  // CHECK-DAG:  arith.subf %[[G16]], %[[F16]] : f16
+  // CHECK-DAG:  arith.subf %[[G64]], %[[F64]] : f64
+  // CHECK-DAG:  arith.subf %[[GBF]], %[[BF]] : bf16
+  // CHECK-DAG:  arith.subf %[[GVEC]], %[[VEC]] : vector<4xf32>
+  // CHECK-DAG:  arith.subf %[[GTEN]], %[[TEN]] : tensor<2x3xf32>
+  // CHECK-NOT:  arith.negf
+  // CHECK-NOT:  arith.addf
+  %n16 = arith.negf %f16 : f16
+  %0 = arith.addf %n16, %g16 : f16
+  %n64 = arith.negf %f64 : f64
+  %1 = arith.addf %g64, %n64 : f64
+  %nbf = arith.negf %bf : bf16
+  %2 = arith.addf %nbf, %gbf : bf16
+  %nvec = arith.negf %vec : vector<4xf32>
+  %3 = arith.addf %gvec, %nvec : vector<4xf32>
+  %nten = arith.negf %ten : tensor<2x3xf32>
+  %4 = arith.addf %nten, %gten : tensor<2x3xf32>
+  return %0, %1, %2, %3, %4 : f16, f64, bf16, vector<4xf32>, tensor<2x3xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_fastmath(
+//  CHECK-SAME: %[[X:.+]]: f32, %[[Y:.+]]: f32
+func.func @test_addf_negf_fastmath(%x : f32, %y : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[A:.+]] = arith.subf %[[Y]], %[[X]] fastmath<nnan,nsz> : f32
+  // CHECK-DAG:  %[[B:.+]] = arith.subf %[[Y]], %[[X]] fastmath<reassoc> : f32
+  // CHECK:      return %[[A]], %[[B]]
+  %n = arith.negf %x : f32
+  %0 = arith.addf %n, %y fastmath<nnan,nsz> : f32
+  %1 = arith.addf %y, %n fastmath<reassoc> : f32
+  return %0, %1 : f32, f32
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_rounding_mode(
+//  CHECK-SAME: %[[X:.+]]: f32, %[[Y:.+]]: f32
+func.func @test_addf_negf_rounding_mode(%x : f32, %y : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[A:.+]] = arith.subf %[[Y]], %[[X]] downward : f32
+  // CHECK-DAG:  %[[B:.+]] = arith.subf %[[Y]], %[[X]] upward : f32
+  // CHECK:      return %[[A]], %[[B]]
+  %n = arith.negf %x : f32
+  %0 = arith.addf %n, %y downward : f32
+  %1 = arith.addf %y, %n upward : f32
+  return %0, %1 : f32, f32
+}
+
+// -----
+
 // CHECK-LABEL: @test_subf_rounding_mode(
 // CHECK-SAME: %[[ARG0:.+]]: f32
 func.func @test_subf_rounding_mode(%arg0 : f32) -> (f32, f32, f32, f32) {


From llvmlistbot at llvm.org  Mon Jul 13 12:51:47 2026
From: llvmlistbot at llvm.org (Paul Kirth)
Date: Mon, 13 Jul 2026 12:51:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5541d3.f1c1923d.991d1.f153@mx.google.com>


================
@@ -0,0 +1,58 @@
+// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+
+
+// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+// Test that a reflection map is created for fields with a certain named attribute
+// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
----------------
ilovepi wrote:

nit: EOF

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Mon Jul 13 12:52:40 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 13 Jul 2026 12:52:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a554208.f732ad81.3a4994.c42f@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/207933

>From 4678904eb7d36b9e1c189467b6df364438166d15 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 29 Jun 2026 16:00:18 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP row_gather

Adds profile compliance support for MXFP row_gather.

Change-Id: I3a256bd60e857955c5a849b951143fdb2df48fab
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 107 +++++++++++++-----
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  17 ++-
 mlir/test/Dialect/Tosa/availability.mlir      |   2 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |   8 ++
 mlir/test/Dialect/Tosa/ops.mlir               |   8 ++
 .../tosa-validation-version-1p0-invalid.mlir  |   9 ++
 .../tosa-validation-version-1p1-valid.mlir    |  18 +++
 7 files changed, 133 insertions(+), 36 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..59c333f8cc408 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -992,37 +992,82 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..ffa80a24b231e 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2596,7 +2596,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2608,6 +2616,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2624,11 +2634,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..151a9a27302ae 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..e42650dc88cc1 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..81137921be19f 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1438,6 +1438,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..87d407c11e386 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..9aee861c6a8d3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -260,6 +260,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


From llvmlistbot at llvm.org  Mon Jul 13 12:54:39 2026
From: llvmlistbot at llvm.org (Andy Kaylor)
Date: Mon, 13 Jul 2026 12:54:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
Message-ID: <llvm.org/llvm/llvm-project/pull/209287@github.com>


https://github.com/andykaylor created https://github.com/llvm/llvm-project/pull/209287

This is a proposed implementation of strict floating-point handling for Arith and Math dialects. If this direction is accepted, I will split the individual commits in this PR out for proper review. I have combined them here so that the entire design can be considered as a whole.

This is a follow-up to my prior [RFC on Discourse](https://discourse.llvm.org/t/rfc-mlir-new-representation-of-floating-point-constraints/91123) proposing a new MLIR representation of floating-point constraints. Based on the feedback I received in https://github.com/llvm/llvm-project/pull/205158, I am moving the floating-point environment attribute out to the CIR and Arith dialects and lowering these to existing operations in the LLVM dialect that continue to map directly to the current form of LLVM IR.

This PR shows the Arith and Math implementation. I have posted a separate [PR for the CIR dialect changes](https://github.com/llvm/llvm-project/pull/209275).

This is intended to replace the existing rounding-mode handling in the Arith dialect. I have left the old handling in place here but added comments that it should be considered deprecated. I am open to suggestions on how that should be handled.

The implementation here has one significant difference from the CIR implementation. In the CIR implementation, I am lowering operations with the Fenv attribute to general intrinsic calls to the constrained floating-point intrinsics. Here I have added specialized constrained operations to the LLVM dialect where they were missing and lower to those. I did this to be consistent with the existing Arith rounding-mode handling, though I think it may be more consistent with the LLVM dialect design philosophy to remove the specialized constrained operations there entirely and to lower to the generalized intrinsic calls as I have done in CIR.

Assisted-by: Cursor / claude-opus-4.8

>From 67b0393ec879f961ddec5c55efe65ba00d095b6a Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 16:44:22 -0700
Subject: [PATCH 1/7] [mlir][llvm] Add more constrained FP operations

This change adds special constrained forms of transcendental operations
that may lower to contrained fp intrinsic calls. These will be used by
the Math and CIR dialects in later changes.

WIP! Needs a test.
---
 .../mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td   | 68 +++++++++++++++++++
 .../LLVMIR/Import/intrinsic-unregistered.ll   | 10 +--
 2 files changed, 73 insertions(+), 5 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index 76a3afad9a360..b912cef1386dd 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -504,6 +504,42 @@ class LLVM_ConstrainedIntr<string mnem, int numArgs,
   }];
 }
 
+// Convenience base class for constrained intrinsics that take a single
+// floating-point operand of the same type as the result and support a rounding
+// mode (e.g. the transcendental math functions).
+class LLVM_ConstrainedUnaryIntrRM<string mnem>
+    : LLVM_ConstrainedIntr<mnem, /*numArgs=*/1,
+        /*overloadedResult=*/1, /*overloadedOperands=*/[],
+        /*hasRoundingMode=*/1, [SameOperandsAndResultType]> {
+  let assemblyFormat = [{
+    $arg_0 $roundingmode $fpExceptionBehavior attr-dict `:` type($arg_0)
+  }];
+}
+
+// Convenience base class for constrained intrinsics that take a single
+// floating-point operand of the same type as the result and do not support a
+// rounding mode (e.g. the rounding math functions such as `ceil`).
+class LLVM_ConstrainedUnaryIntrNoRM<string mnem>
+    : LLVM_ConstrainedIntr<mnem, /*numArgs=*/1,
+        /*overloadedResult=*/1, /*overloadedOperands=*/[],
+        /*hasRoundingMode=*/0, [SameOperandsAndResultType]> {
+  let assemblyFormat = [{
+    $arg_0 $fpExceptionBehavior attr-dict `:` type($arg_0)
+  }];
+}
+
+// Convenience base class for constrained intrinsics that take two
+// floating-point operands of the same type as the result and support a rounding
+// mode (e.g. `pow` and `atan2`).
+class LLVM_ConstrainedBinaryIntrRM<string mnem>
+    : LLVM_ConstrainedIntr<mnem, /*numArgs=*/2,
+        /*overloadedResult=*/1, /*overloadedOperands=*/[],
+        /*hasRoundingMode=*/1, [SameOperandsAndResultType]> {
+  let assemblyFormat = [{
+    $arg_0 `,` $arg_1 $roundingmode $fpExceptionBehavior attr-dict `:` type($arg_0)
+  }];
+}
+
 def LLVM_ConstrainedFAddIntr
     : LLVM_ConstrainedIntr<"fadd", /*numArgs=*/2,
         /*overloadedResult=*/1, /*overloadedOperands=*/[],
@@ -567,6 +603,38 @@ def LLVM_ConstrainedFMulAddIntr
   }];
 }
 
+// Constrained intrinsics for math library functions. These correspond to the
+// like-named operations in the Math dialect and are produced when the source
+// operation carries a `#arith.fenv` floating-point environment attribute.
+
+// Unary transcendental functions (support a rounding mode).
+def LLVM_ConstrainedSqrtIntr  : LLVM_ConstrainedUnaryIntrRM<"sqrt">;
+def LLVM_ConstrainedSinIntr   : LLVM_ConstrainedUnaryIntrRM<"sin">;
+def LLVM_ConstrainedCosIntr   : LLVM_ConstrainedUnaryIntrRM<"cos">;
+def LLVM_ConstrainedTanIntr   : LLVM_ConstrainedUnaryIntrRM<"tan">;
+def LLVM_ConstrainedASinIntr  : LLVM_ConstrainedUnaryIntrRM<"asin">;
+def LLVM_ConstrainedACosIntr  : LLVM_ConstrainedUnaryIntrRM<"acos">;
+def LLVM_ConstrainedATanIntr  : LLVM_ConstrainedUnaryIntrRM<"atan">;
+def LLVM_ConstrainedSinhIntr  : LLVM_ConstrainedUnaryIntrRM<"sinh">;
+def LLVM_ConstrainedCoshIntr  : LLVM_ConstrainedUnaryIntrRM<"cosh">;
+def LLVM_ConstrainedTanhIntr  : LLVM_ConstrainedUnaryIntrRM<"tanh">;
+def LLVM_ConstrainedExpIntr   : LLVM_ConstrainedUnaryIntrRM<"exp">;
+def LLVM_ConstrainedExp2Intr  : LLVM_ConstrainedUnaryIntrRM<"exp2">;
+def LLVM_ConstrainedLogIntr   : LLVM_ConstrainedUnaryIntrRM<"log">;
+def LLVM_ConstrainedLog10Intr : LLVM_ConstrainedUnaryIntrRM<"log10">;
+def LLVM_ConstrainedLog2Intr  : LLVM_ConstrainedUnaryIntrRM<"log2">;
+
+// Unary rounding functions (do not support a rounding mode).
+def LLVM_ConstrainedCeilIntr      : LLVM_ConstrainedUnaryIntrNoRM<"ceil">;
+def LLVM_ConstrainedFloorIntr     : LLVM_ConstrainedUnaryIntrNoRM<"floor">;
+def LLVM_ConstrainedRoundIntr     : LLVM_ConstrainedUnaryIntrNoRM<"round">;
+def LLVM_ConstrainedRoundEvenIntr : LLVM_ConstrainedUnaryIntrNoRM<"roundeven">;
+def LLVM_ConstrainedTruncIntr     : LLVM_ConstrainedUnaryIntrNoRM<"trunc">;
+
+// Binary functions (support a rounding mode).
+def LLVM_ConstrainedPowIntr   : LLVM_ConstrainedBinaryIntrRM<"pow">;
+def LLVM_ConstrainedATan2Intr : LLVM_ConstrainedBinaryIntrRM<"atan2">;
+
 def LLVM_ConstrainedUIToFP
     : LLVM_ConstrainedIntr<"uitofp", /*numArgs=*/1,
         /*overloadedResult=*/1, /*overloadedOperands=*/[0],
diff --git a/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll b/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
index da58d48d995be..556803f3a0dbb 100644
--- a/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
+++ b/mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll
@@ -85,14 +85,14 @@ define signext i32 @test_intrin_arg_attr(i32 signext %a) nounwind {
 ; imported as `llvm.mlir.metadata_as_value` ops wrapping the corresponding
 ; `#llvm.md_string` attribute.
 
-declare float @llvm.experimental.constrained.sqrt.f32(float, metadata, metadata)
+declare float @llvm.experimental.constrained.nearbyint.f32(float, metadata, metadata)
 
-; CHECK-LABEL: llvm.func @constrained_sqrt
-define float @constrained_sqrt(float %a) {
+; CHECK-LABEL: llvm.func @constrained_nearbyint
+define float @constrained_nearbyint(float %a) {
   ; CHECK: %[[RM:.*]] = llvm.mlir.metadata_as_value #llvm.md_string<"round.tonearest">
   ; CHECK: %[[EB:.*]] = llvm.mlir.metadata_as_value #llvm.md_string<"fpexcept.strict">
-  ; CHECK: %{{.*}} = llvm.call_intrinsic "llvm.experimental.constrained.sqrt.f32"(%{{.*}}, %[[RM]], %[[EB]]) : (f32, !llvm.metadata, !llvm.metadata) -> f32
-  %r = call float @llvm.experimental.constrained.sqrt.f32(float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
+  ; CHECK: %{{.*}} = llvm.call_intrinsic "llvm.experimental.constrained.nearbyint.f32"(%{{.*}}, %[[RM]], %[[EB]]) : (f32, !llvm.metadata, !llvm.metadata) -> f32
+  %r = call float @llvm.experimental.constrained.nearbyint.f32(float %a, metadata !"round.tonearest", metadata !"fpexcept.strict")
   ret float %r
 }
 

>From 382229113f7af27560812b61787ee28e78f6c375 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Fri, 26 Jun 2026 08:47:24 -0700
Subject: [PATCH 2/7] [mlir][arith] Introduce fenv attribute to arith dialect

This adds a new attribute, arith::FenvAttr, and adds it to all operations
that currently support arith::RoundingModeAttr. The fenv attribute is
intended as a replacement for the rounding mode attribute, extending it
to also describe exception behavior requirements.

WIP!
---
 .../mlir/Dialect/Arith/IR/ArithBase.td        | 148 +++++++++++++++++-
 .../include/mlir/Dialect/Arith/IR/ArithOps.td |  89 ++++++++++-
 .../Dialect/Arith/IR/ArithOpsInterfaces.td    |  83 ++++++++++
 mlir/include/mlir/Dialect/Math/IR/MathOps.td  |  53 ++++++-
 .../ArithCommon/AttrToLLVMConverter.cpp       |   2 +
 .../Dialect/Arith/IR/ArithCanonicalization.td |  40 +++--
 .../Dialect/Arith/Transforms/ExpandOps.cpp    |  14 +-
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir |   2 +
 mlir/test/Dialect/Arith/invalid.mlir          |   8 +
 mlir/test/Dialect/Arith/ops.mlir              |  23 +++
 10 files changed, 432 insertions(+), 30 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
index 985ae01008002..4132a4cfb3c55 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
@@ -186,12 +186,158 @@ def Arith_RTowardZero                // Round toward 0
     : I32EnumAttrCase<"toward_zero", 3>;
 def Arith_RToNearestTiesAwayFromZero // Round to nearest, ties away from zero
     : I32EnumAttrCase<"to_nearest_away", 4>;
+def Arith_RDynamic                   // Rounding mode unknown at compile time
+    : I32EnumAttrCase<"unknown", 7>;  // Mirrors `llvm::RoundingMode::Dynamic`.
 
 def Arith_RoundingModeAttr : I32EnumAttr<
     "RoundingMode", "Floating point rounding mode",
     [Arith_RToNearestTiesToEven, Arith_RDownward, Arith_RUpward,
-     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero]> {
+     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero, Arith_RDynamic]> {
   let cppNamespace = "::mlir::arith";
 }
 
+//===----------------------------------------------------------------------===//
+// Floating-point environment enums (used by the `#arith.fenv` attribute)
+//===----------------------------------------------------------------------===//
+
+def FPExceptionMode : I32EnumAttr<
+    "FPExceptionMode", "floating-point exception mode", [
+  I32EnumAttrCase<"Unknown", 0, "unknown">,
+  I32EnumAttrCase<"Masked", 1, "masked">,
+  I32EnumAttrCase<"Unmasked", 2, "unmasked">
+]> {
+  let description = [{
+    The known floating-point exception mode at the point the instruction is
+    executed. If the actual exception mode differs from this value, the
+    behavior is undefined.
+  }];
+  let cppNamespace = "::mlir::arith";
+  let genSpecializedAttr = 0;
+}
+
+//===----------------------------------------------------------------------===//
+// FenvAttr
+//===----------------------------------------------------------------------===//
+
+def Arith_FenvAttr : AttrDef<Arith_Dialect, "Fenv"> {
+  let mnemonic = "fenv";
+  let summary = "Describes floating-point environment constraints";
+  let description = [{
+    The `#arith.fenv` attribute describes constraints on the floating-point
+    handling of a floating-point operation. It can be attached to
+    floating-point operations to capture rounding and exception behavior. All
+    of its parameters are optional.
+
+    - `dynamic_rounding_mode`: the known dynamic rounding mode at the point the
+      instruction is executed. If the actual dynamic rounding mode when the
+      instruction is executed does not match the assumed value, the behavior is
+      undefined.
+    - `except_mode`: the known exception mode at the point the instruction is
+      executed. If the actual state of exception masking when the instruction is
+      executed does not match this value, the behavior is undefined.
+    - `strict_except`: if `false`, any case that would produce an FP exception
+      produces it non-deterministically instead (i.e. it may or may not occur).
+      This means the FP status is written non-deterministically, and, if
+      exceptions are unmasked, the instruction traps non-deterministically.
+
+    When this attribute is present on an operation, the operation is lowered to
+    the corresponding `llvm.experimental.constrained.*` intrinsic during
+    translation to LLVM IR.
+
+    The default setting for each parameter (`unknown` for
+    `dynamic_rounding_mode`, `masked` for exception_mode, and `false` for
+    `strict_except`) is canonically represented as an unset parameter.
+    Constructing an attribute whose parameters are all set to their defaults
+    therefore yields the same canonical, empty attribute as `#arith.fenv<>`.
+
+    Carrying the (possibly empty) attribute is *not* the same as having no
+    attribute at all. An operation that carries the attribute is lowered to the
+    corresponding constrained intrinsic operation in LLVM, even when every
+    parameter is at its default value. An operation with no `#arith.fenv`
+    attribute is lowered to the regular, unconstrained operation.
+
+    Examples:
+    ```mlir
+    fenv<dynamic_rounding_mode = to_nearest_even>
+    fenv<except_mode = unmasked>
+    fenv<strict_except = true>
+    fenv<dynamic_rounding_mode = upward, except_mode = unknown, strict_except = true>
+    ```
+  }];
+
+  let parameters = (ins
+    OptionalParameter<
+        "std::optional<::mlir::arith::RoundingMode>">:$dynamic_rounding_mode,
+    OptionalParameter<"std::optional<::mlir::arith::FPExceptionMode>">:$except_mode,
+    OptionalParameter<"mlir::BoolAttr">:$strict_except);
+
+  let assemblyFormat = [{
+    `<` struct($dynamic_rounding_mode, $except_mode, $strict_except) `>`
+  }];
+
+  // Replace the default builder so that constructing the attribute normalizes
+  // any parameter that is set to its default value to "unset". This keeps a
+  // single canonical representation for the default floating-point environment.
+  // Because `skipDefaultBuilders` suppresses the auto-synthesized const builder
+  // call, provide it explicitly. The `DefaultValuedAttr` form below uses a null
+  // attribute as its default, so this simply forwards its argument (a null
+  // attribute when the operation does not carry a `#arith.fenv` attribute).
+  let skipDefaultBuilders = 1;
+  let constBuilderCall = "$0";
+  let builders = [
+    AttrBuilder<(ins
+        CArg<"std::optional<::mlir::arith::RoundingMode>",
+             "std::nullopt">:$dynamicRoundingMode,
+        CArg<"std::optional<::mlir::arith::FPExceptionMode>",
+             "std::nullopt">:$exceptMode,
+        CArg<"mlir::BoolAttr", "{}">:$strictExcept), [{
+      if (dynamicRoundingMode == ::mlir::arith::RoundingMode::unknown)
+        dynamicRoundingMode = std::nullopt;
+      if (exceptMode == ::mlir::arith::FPExceptionMode::Masked)
+        exceptMode = std::nullopt;
+      if (strictExcept && !strictExcept.getValue())
+        strictExcept = {};
+      return Base::get($_ctxt, dynamicRoundingMode, exceptMode, strictExcept);
+    }]>
+  ];
+
+  let extraClassDeclaration = [{
+    /// The default dynamic rounding mode used when `dynamic_rounding_mode` is
+    /// unset.
+    static constexpr ::mlir::arith::RoundingMode
+    getDefaultDynamicRoundingMode() {
+      return ::mlir::arith::RoundingMode::unknown;
+    }
+
+    /// The default exception mode used when `except_mode` is unset.
+    static constexpr ::mlir::arith::FPExceptionMode getDefaultExceptionMode() {
+      return ::mlir::arith::FPExceptionMode::Masked;
+    }
+
+    /// Returns the effective dynamic rounding mode, substituting the default
+    /// when unset.
+    ::mlir::arith::RoundingMode getDynamicRoundingModeOrDefault() const {
+      return getDynamicRoundingMode().value_or(getDefaultDynamicRoundingMode());
+    }
+    /// Returns the effective exception mode, substituting the default when
+    /// unset.
+    ::mlir::arith::FPExceptionMode getExceptionModeOrDefault() const {
+      return getExceptMode().value_or(getDefaultExceptionMode());
+    }
+    /// Returns the effective `strict_except` flag.
+    bool getStrictExceptOrDefault() const {
+      ::mlir::BoolAttr attr = getStrictExcept();
+      return attr ? attr.getValue() : false;
+    }
+
+    /// Returns true if every parameter is at its default value, i.e. the
+    /// attribute describes the default floating-point environment and imposes
+    /// no constraints. Because the builder normalizes defaults to "unset", this
+    /// is equivalent to all parameters being unset.
+    bool isDefault() const {
+      return !getDynamicRoundingMode() && !getExceptMode() && !getStrictExcept();
+    }
+  }];
+}
+
 #endif // ARITH_BASE
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..c383e8d8cd1b9 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -93,6 +93,11 @@ class Arith_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
 
 // Base class for floating point binary operations with an optional rounding
 // mode.
+//
+// Deprecated: use `Arith_FloatBinaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
 class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                                           list<Trait> traits = []> :
     Arith_FloatBinaryOp<mnemonic,
@@ -128,6 +133,50 @@ class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                           attr-dict `:` type($result) }];
 }
 
+// Base class for floating point binary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Arith_FloatBinaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Arith_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Arith_FloatBinaryOpWithRoundingMode<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+      DefaultValuedAttr<
+        Arith_FastMathAttr, "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With two trailing optional attributes (`roundingmode` and `fenv`), the
+  // generated builders no longer provide C++ default arguments for them, so the
+  // convenience builders must pass both explicitly.
+  let builders = [
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Type":$type, "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, type, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $lhs `,` $rhs ($roundingmode^)?
+                          (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Checks that tensor input and outputs have identical shapes. This is stricker
 // than the verification done in `SameOperandsAndResultShape` that allows for
 // tensor dimensions to be 'compatible' (e.g., dynamic dimensions being
@@ -1089,7 +1138,7 @@ def Arith_FlushDenormalsOp : Arith_FloatUnaryOp<"flush_denormals"> {
 // AddFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
+def Arith_AddFOp : Arith_FloatBinaryOpWithFenv<"addf", [Commutative]> {
   let summary = "floating point addition operation";
   let description = [{
     The `addf` operation takes two operands and returns one result, each of
@@ -1120,7 +1169,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
 // SubFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_SubFOp : Arith_FloatBinaryOpWithRoundingMode<"subf"> {
+def Arith_SubFOp : Arith_FloatBinaryOpWithFenv<"subf"> {
   let summary = "floating point subtraction operation";
   let description = [{
     The `subf` operation takes two operands and returns one result, each of
@@ -1276,7 +1325,7 @@ def Arith_MinUIOp : Arith_TotalIntBinaryOp<"minui",
 // MulFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
+def Arith_MulFOp : Arith_FloatBinaryOpWithFenv<"mulf", [Commutative]> {
   let summary = "floating point multiplication operation";
   let description = [{
     The `mulf` operation takes two operands and returns one result, each of
@@ -1308,7 +1357,7 @@ def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
 // DivFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_DivFOp : Arith_FloatBinaryOpWithRoundingMode<"divf"> {
+def Arith_DivFOp : Arith_FloatBinaryOpWithFenv<"divf"> {
   let summary = "floating point division operation";
   let description = [{
     The `divf` operation takes two operands and returns one result, each of
@@ -1566,10 +1615,17 @@ def Arith_TruncFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast from floating-point to narrower floating-point";
   let description = [{
@@ -1589,6 +1645,7 @@ def Arith_TruncFOp :
   let hasVerifier = 1;
   let assemblyFormat = [{ $in ($roundingmode^)?
                           (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($in) `to` type($out) }];
 }
 
@@ -1601,10 +1658,17 @@ def Arith_ConvertFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast between floating-point types of the same bitwidth";
   let description = [{
@@ -1621,6 +1685,7 @@ def Arith_ConvertFOp :
   let hasVerifier = 1;
   let assemblyFormat = [{ $in ($roundingmode^)?
                           (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($in) `to` type($out) }];
 }
 
@@ -1633,10 +1698,17 @@ def Arith_ScalingTruncFOp
                [Pure, SameInputOutputTensorDims,
                 DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
                 DeclareOpInterfaceMethods<ArithFastMathInterface>,
+                DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
                 DeclareOpInterfaceMethods<CastOpInterface>]>,
       Arguments<(ins FloatLike:$in, FloatLike:$scale,
+          // Deprecated: represent the rounding mode through the `fenv`
+          // attribute instead. This attribute is retained temporarily while
+          // floating-point operations migrate to the `#arith.fenv`
+          // floating-point environment attribute, and will be removed once the
+          // migration is complete.
           OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-          OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+          OptionalAttr<Arith_FastMathAttr>:$fastmath,
+          OptionalAttr<Arith_FenvAttr>:$fenv)>,
       Results<(outs FloatLike:$out)> {
   let summary = "Downcasts input floating point values using provided scales "
                 "values following OCP MXFP Spec";
@@ -1690,7 +1762,8 @@ def Arith_ScalingTruncFOp
   }];
   let hasVerifier = 1;
   let assemblyFormat =
-      [{ $in `,` $scale ($roundingmode^)? (`fastmath` `` $fastmath^)? attr-dict `:`
+      [{ $in `,` $scale ($roundingmode^)? (`fastmath` `` $fastmath^)?
+      (`fenv` `` $fenv^)? attr-dict `:`
       type($in) `,` type($scale) `to` type($out)}];
 }
 
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
index d1b8e250cdb59..204bb32f89f4a 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td
@@ -186,4 +186,87 @@ def ArithRoundingModeInterface : OpInterface<"ArithRoundingModeInterface"> {
   ];
 }
 
+def FPEnvConstrainedOpInterface : OpInterface<"FPEnvConstrainedOpInterface"> {
+  let description = [{
+    This interface is implemented by floating-point operations that can carry an
+    optional `#arith.fenv` attribute describing constraints on their
+    floating-point environment. It provides a uniform API for querying the
+    attribute and its individual fields.
+
+    When the attribute is present, the operation is lowered to the corresponding
+    `llvm.experimental.constrained.*` intrinsic during translation to LLVM IR.
+  }];
+
+  let cppNamespace = "::mlir::arith";
+
+  let verify = [{
+    auto fenvOp = ::llvm::cast<::mlir::arith::FPEnvConstrainedOpInterface>($_op);
+    if (fenvOp.getFenvAttr()) {
+      if (auto roundingOp =
+              ::llvm::dyn_cast<::mlir::arith::ArithRoundingModeInterface>($_op))
+        if (roundingOp.getRoundingModeAttr())
+          return $_op->emitOpError("The `roundingmode` attribute is deprecated."
+                                   " Represent the rounding mode through the "
+                                   "`fenv` attribute instead.");
+    }
+    return ::mlir::success();
+  }];
+
+  let methods = [
+    InterfaceMethod<
+      /*desc=*/        [{Returns the `#arith.fenv` attribute attached to the
+                         operation, or a null attribute if none is present.}],
+      /*returnType=*/  "::mlir::arith::FenvAttr",
+      /*methodName=*/  "getFenvAttr",
+      /*args=*/        (ins),
+      /*methodBody=*/  [{}],
+      /*defaultImpl=*/ [{
+        auto op = cast<ConcreteOp>(this->getOperation());
+        return op.getFenvAttr();
+      }]
+      >,
+    InterfaceMethod<
+      /*desc=*/        [{Returns the effective known dynamic rounding mode
+                         requested by the attribute, substituting the default
+                         (`unknown`) when the attribute is absent or does not set
+                         the field.}],
+      /*returnType=*/  "::mlir::arith::RoundingMode",
+      /*methodName=*/  "getFenvDynamicRoundingMode",
+      /*args=*/        (ins),
+      /*methodBody=*/  [{}],
+      /*defaultImpl=*/ [{
+        ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+        return attr ? attr.getDynamicRoundingModeOrDefault()
+                    : ::mlir::arith::FenvAttr::getDefaultDynamicRoundingMode();
+      }]
+      >,
+    InterfaceMethod<
+      /*desc=*/        [{Returns the effective known exception mode requested by
+                         the attribute, substituting the default (`masked`) when
+                         the attribute is absent or does not set the field.}],
+      /*returnType=*/  "::mlir::arith::FPExceptionMode",
+      /*methodName=*/  "getFenvExceptionMode",
+      /*args=*/        (ins),
+      /*methodBody=*/  [{}],
+      /*defaultImpl=*/ [{
+        ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+        return attr ? attr.getExceptionModeOrDefault()
+                    : ::mlir::arith::FenvAttr::getDefaultExceptionMode();
+      }]
+      >,
+    InterfaceMethod<
+      /*desc=*/        [{Returns the effective strict-exception flag requested
+                         by the attribute.}],
+      /*returnType=*/  "bool",
+      /*methodName=*/  "getFenvStrictExcept",
+      /*args=*/        (ins),
+      /*methodBody=*/  [{}],
+      /*defaultImpl=*/ [{
+        ::mlir::arith::FenvAttr attr = $_op.getFenvAttr();
+        return attr ? attr.getStrictExceptOrDefault() : false;
+      }]
+      >
+  ];
+}
+
 #endif // ARITH_OPS_INTERFACES
diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 926bac1de2b0d..21c49484610a0 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -112,6 +112,11 @@ class Math_FloatTernaryOp<string mnemonic, list<Trait> traits = []> :
 
 // Base class for floating point ternary operations with an optional rounding
 // mode.
+//
+// Deprecated: use `Math_FloatTernaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
 class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
                                           list<Trait> traits = []> :
     Math_FloatTernaryOp<mnemonic,
@@ -147,6 +152,52 @@ class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
                           attr-dict `:` type($result) }];
 }
 
+// Base class for floating point ternary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Math_FloatTernaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Math_FloatTernaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Math_FloatTernaryOpWithRoundingMode<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$a, FloatLike:$b, FloatLike:$c,
+      DefaultValuedAttr<Arith_FastMathAttr,
+                        "::mlir::arith::FastMathFlags::none">:$fastmath,
+      // Deprecated: represent the rounding mode through the `fenv` attribute
+      // instead.
+      OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With two trailing optional attributes (`roundingmode` and `fenv`), the
+  // generated builders no longer provide C++ default arguments for them, so the
+  // convenience builders must pass both explicitly.
+  let builders = [
+    OpBuilder<(ins "Value":$a, "Value":$b, "Value":$c,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, a, b, c, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Value":$a, "Value":$b, "Value":$c,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, a, b, c, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Type":$type, "Value":$a, "Value":$b, "Value":$c,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, type, a, b, c,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $a `,` $b `,` $c ($roundingmode^)?
+                          (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 //===----------------------------------------------------------------------===//
 // AbsFOp
 //===----------------------------------------------------------------------===//
@@ -787,7 +838,7 @@ def Math_FloorOp : Math_FloatUnaryOp<"floor"> {
 // FmaOp
 //===----------------------------------------------------------------------===//
 
-def Math_FmaOp : Math_FloatTernaryOpWithRoundingMode<"fma"> {
+def Math_FmaOp : Math_FloatTernaryOpWithFenv<"fma"> {
   let summary = "floating point fused multipy-add operation";
   let description = [{
     The `fma` operation takes three operands and returns one result, each of
diff --git a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
index cf60a048f782c..b3f5705601cfb 100644
--- a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
+++ b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
@@ -62,6 +62,8 @@ mlir::arith::convertArithRoundingModeToLLVM(arith::RoundingMode roundingMode) {
     return LLVM::RoundingMode::TowardZero;
   case arith::RoundingMode::upward:
     return LLVM::RoundingMode::TowardPositive;
+  case arith::RoundingMode::unknown:
+    return LLVM::RoundingMode::Dynamic;
   }
   llvm_unreachable("Unhandled rounding mode");
 }
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index 86c9b6d3551d8..74f1b2ab44dde 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -486,17 +486,21 @@ def TruncIShrSIToTrunciShrUI :
 // TruncFOp
 //===----------------------------------------------------------------------===//
 
-// truncf(sitofp(x)) -> sitofp(x) if default rounding mode.
+// truncf(sitofp(x)) -> sitofp(x) if default rounding mode and no fenv
+// constraints.
 def TruncFSIToFPToSIToFP :
-    Pat<(Arith_TruncFOp:$tr (Arith_SIToFPOp:$fp $x), $rmf, $fmf),
+    Pat<(Arith_TruncFOp:$tr (Arith_SIToFPOp:$fp $x), $rmf, $fmf, $fenv),
         (Arith_SIToFPOp $x),
-        [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf)]>;
+        [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf),
+         (Constraint<CPred<"$0 == nullptr">, "no fenv constraints"> $fenv)]>;
 
-// truncf(uitofp(x)) -> uitofp(x) if default rounding mode.
+// truncf(uitofp(x)) -> uitofp(x) if default rounding mode and no fenv
+// constraints.
 def TruncFUIToFPToUIToFP :
-    Pat<(Arith_TruncFOp:$tr (Arith_UIToFPOp:$fp $x, $nneg), $rmf, $fmf),
+    Pat<(Arith_TruncFOp:$tr (Arith_UIToFPOp:$fp $x, $nneg), $rmf, $fmf, $fenv),
         (Arith_UIToFPOp $x, $nneg),
-        [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf)]>;
+        [(Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rmf),
+         (Constraint<CPred<"$0 == nullptr">, "no fenv constraints"> $fenv)]>;
 
 //===----------------------------------------------------------------------===//
 // SIToFPOp
@@ -534,10 +538,12 @@ def IsNegZeroFloat :
 // TODO: Verify if this canonicalization is safe when a rounding mode is
 // specified. For the moment, bail on custom rounding modes.
 def SubFOfNegZero :
-    Pat<(Arith_SubFOp (ConstantLikeMatcher AnyAttr:$c), $x, $fmf, $rm),
+    Pat<(Arith_SubFOp (ConstantLikeMatcher AnyAttr:$c), $x, $fmf, $rm, $fenv),
         (Arith_NegFOp $x, $fmf),
         [(IsNegZeroFloat $c),
-         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+         (Constraint<CPred<"$0 == nullptr">,
+                     "default floating-point environment"> $fenv)]>;
 
 //===----------------------------------------------------------------------===//
 // MulFOp
@@ -547,10 +553,13 @@ def SubFOfNegZero :
 // TODO: Verify if this canonicalization is safe when a rounding mode is
 // specified. For the moment, bail on custom rounding modes.
 def MulFOfNegF :
-    Pat<(Arith_MulFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm),
-        (Arith_MulFOp $x, $y, $fmf, $rm),
+    Pat<(Arith_MulFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm,
+                      $fenv),
+        (Arith_MulFOp $x, $y, $fmf, $rm, $fenv),
         [(TypesMatch $x, $y),
-         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+         (Constraint<CPred<"$0 == nullptr">,
+                     "default floating-point environment"> $fenv)]>;
 
 //===----------------------------------------------------------------------===//
 // DivFOp
@@ -560,9 +569,12 @@ def MulFOfNegF :
 // TODO: Verify if this canonicalization is safe when a rounding mode is
 // specified. For the moment, bail on custom rounding modes.
 def DivFOfNegF :
-    Pat<(Arith_DivFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm),
-        (Arith_DivFOp $x, $y, $fmf, $rm),
+    Pat<(Arith_DivFOp (Arith_NegFOp $x, $_), (Arith_NegFOp $y, $_), $fmf, $rm,
+                      $fenv),
+        (Arith_DivFOp $x, $y, $fmf, $rm, $fenv),
         [(TypesMatch $x, $y),
-         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm)]>;
+         (Constraint<CPred<"$0 == nullptr">, "default rounding mode"> $rm),
+         (Constraint<CPred<"$0 == nullptr">,
+                     "default floating-point environment"> $fenv)]>;
 
 #endif // ARITH_PATTERNS
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..85c6d82cffe5c 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -485,7 +485,7 @@ struct F8E8M0ExtFOpConverter : public OpRewritePattern<arith::ExtFOp> {
     Value result = arith::BitcastOp::create(b, f32Ty, f32Bits);
     if (resultETy.getIntOrFloatBitWidth() < 32) {
       result = arith::TruncFOp::create(b, resultTy, result, nullptr,
-                                       op.getFastmathAttr());
+                                       op.getFastmathAttr(), nullptr);
     } else if (resultETy.getIntOrFloatBitWidth() > 32) {
       result = arith::ExtFOp::create(b, resultTy, result, op.getFastmathAttr());
     }
@@ -649,8 +649,9 @@ struct F8E8M0TruncFOpConverter : public OpRewritePattern<arith::TruncFOp> {
     if (operandETy.getIntOrFloatBitWidth() < 32) {
       operand = arith::ExtFOp::create(b, f32Ty, operand, op.getFastmathAttr());
     } else if (operandETy.getIntOrFloatBitWidth() > 32) {
-      operand = arith::TruncFOp::create(
-          b, f32Ty, operand, op.getRoundingmodeAttr(), op.getFastmathAttr());
+      operand = arith::TruncFOp::create(b, f32Ty, operand,
+                                        op.getRoundingmodeAttr(),
+                                        op.getFastmathAttr(), op.getFenvAttr());
     }
     Value f32Bits = arith::BitcastOp::create(b, i32Ty, operand);
     Value cF32MantissaWidth = createConst(op->getLoc(), i32Ty, 23, rewriter);
@@ -676,7 +677,7 @@ struct ScalingExtFOpConverter : public OpRewritePattern<arith::ScalingExtFOp> {
       scaleETy = b.getF8E8M0Type();
       scaleTy = cloneToShapedType(scaleTy, scaleETy);
       scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
-                                             op.getFastmathAttr());
+                                             op.getFastmathAttr(), nullptr);
     }
     // Catch scale types like f8E5M2.
     if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
@@ -718,7 +719,7 @@ struct ScalingTruncFOpConverter
       scaleETy = b.getF8E8M0Type();
       scaleTy = cloneToShapedType(scaleTy, scaleETy);
       scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
-                                             op.getFastmathAttr());
+                                             op.getFastmathAttr(), nullptr);
     }
     if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
       return rewriter.notifyMatchFailure(
@@ -734,7 +735,8 @@ struct ScalingTruncFOpConverter
     Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
                                          op.getFastmathAttr());
     Value resultCast = arith::TruncFOp::create(
-        b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr());
+        b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
+        op.getFenvAttr());
     rewriter.replaceOp(op, resultCast);
     return success();
   }
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..5d24ab17ee8c3 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -396,6 +396,8 @@ func.func @experimental_constrained_fptrunc(%arg0 : f64) {
   %3 = arith.truncf %arg0 toward_zero : f64 to f32
 // CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc {{.*}} tonearestaway ignore : f64 to f32
   %4 = arith.truncf %arg0 to_nearest_away : f64 to f32
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc {{.*}} dynamic ignore : f64 to f32
+  %5 = arith.truncf %arg0 unknown : f64 to f32
   return
 }
 
diff --git a/mlir/test/Dialect/Arith/invalid.mlir b/mlir/test/Dialect/Arith/invalid.mlir
index 49f55e855663d..4829402c4e6a9 100644
--- a/mlir/test/Dialect/Arith/invalid.mlir
+++ b/mlir/test/Dialect/Arith/invalid.mlir
@@ -1152,3 +1152,11 @@ func.func @convertf_int_output(%arg0 : f32) {
   %0 = arith.convertf %arg0 : f32 to i32
   return
 }
+
+// -----
+
+func.func @fenv_with_rounding_mode(%arg0 : f32, %arg1 : f32) {
+  // expected-error @+1 {{'arith.addf' op The `roundingmode` attribute is deprecated. Represent the rounding mode through the `fenv` attribute instead.}}
+  %0 = arith.addf %arg0, %arg1 to_nearest_even fenv<except_mode = unmasked> : f32
+  return
+}
diff --git a/mlir/test/Dialect/Arith/ops.mlir b/mlir/test/Dialect/Arith/ops.mlir
index 76aef203a64fb..ba416c78fc3df 100644
--- a/mlir/test/Dialect/Arith/ops.mlir
+++ b/mlir/test/Dialect/Arith/ops.mlir
@@ -1305,6 +1305,29 @@ func.func @roundingmode(%arg0: f32, %arg1: f32) {
   return
 }
 
+// CHECK-LABEL: @fenv
+func.func @fenv(%arg0: f32, %arg1: f32) {
+// CHECK: {{.*}} = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward> : f32
+  %0 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward> : f32
+// CHECK: {{.*}} = arith.subf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = true> : f32
+  %1 = arith.subf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = true> : f32
+// CHECK: {{.*}} = arith.mulf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unknown> : f32
+  %2 = arith.mulf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unknown> : f32
+// CHECK: {{.*}} = arith.divf %arg0, %arg1 fenv<> : f32
+  %3 = arith.divf %arg0, %arg1 fenv<> : f32
+
+  // Parameters at their default values are normalized to an unset (canonical
+  // empty) attribute.
+// CHECK: {{.*}} = arith.addf %arg0, %arg1 fenv<> : f32
+  %4 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = unknown> : f32
+// CHECK: {{.*}} = arith.subf %arg0, %arg1 fenv<> : f32
+  %5 = arith.subf %arg0, %arg1 fenv<except_mode = masked> : f32
+// CHECK: {{.*}} = arith.mulf %arg0, %arg1 fenv<except_mode = unmasked> : f32
+  %6 = arith.mulf %arg0, %arg1 fenv<except_mode = unmasked, strict_except = false> : f32
+
+  return
+}
+
 // CHECK-LABEL: @select_tensor
 func.func @select_tensor(%arg0 : tensor<8xi1>, %arg1 : tensor<8xi32>, %arg2 : tensor<8xi32>) -> tensor<8xi32> {
   // CHECK: = arith.select %{{.*}}, %{{.*}}, %{{.*}} : tensor<8xi1>, tensor<8xi32>

>From 95bf042c894277b2c0c106427500dc0994a2a3e4 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Fri, 26 Jun 2026 17:45:32 -0700
Subject: [PATCH 3/7] [mlir][arith] Lower operations with the fenv attribute

This change adds handling to lower operations in the Arith or Math dialects
with the fenv attribute to constrained fp operations in the LLVM dialect.

WIP!
---
 .../ArithCommon/AttrToLLVMConverter.h         | 59 ++++++++++--
 .../ArithCommon/AttrToLLVMConverter.cpp       | 15 +++
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    | 92 ++++++++++++++-----
 mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 23 +++--
 .../Dialect/Arith/Transforms/ExpandOps.cpp    |  6 +-
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir | 80 ++++++++++++++++
 .../Conversion/MathToLLVM/math-to-llvm.mlir   | 18 ++++
 mlir/test/Dialect/Arith/expand-ops.mlir       | 13 +++
 8 files changed, 259 insertions(+), 47 deletions(-)

diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index feb74c86e349f..31d94daf4bb92 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -45,6 +45,18 @@ convertArithRoundingModeAttrToLLVM(arith::RoundingModeAttr roundingModeAttr);
 LLVM::FPExceptionBehaviorAttr
 getLLVMDefaultFPExceptionBehavior(MLIRContext &context);
 
+/// Maps an arith floating-point exception mode and strict-exception flag, as
+/// described by an `#arith.fenv` attribute, to the corresponding LLVM
+/// constrained-intrinsic exception behavior:
+///
+///   masked            + strict_except=false -> fpexcept.ignore
+///   masked            + strict_except=true  -> fpexcept.strict
+///   unmasked|unknown  + strict_except=false -> fpexcept.maytrap
+///   unmasked|unknown  + strict_except=true  -> fpexcept.strict
+LLVM::FPExceptionBehavior
+convertArithFPExceptionBehaviorToLLVM(arith::FPExceptionMode exceptionMode,
+                                      bool strictExcept);
+
 // Attribute converter that populates a NamedAttrList by removing the fastmath
 // attribute from the source operation attributes, and replacing it with an
 // equivalent LLVM fastmath attribute.
@@ -145,25 +157,52 @@ class AttrConverterConstrainedFPToLLVM {
   AttrConverterConstrainedFPToLLVM(SourceOp srcOp) {
     // Copy the source attributes.
     convertedAttr = NamedAttrList{srcOp->getAttrs()};
+    MLIRContext *ctx = srcOp->getContext();
+
+    // The floating-point environment may be described either by the deprecated
+    // `roundingmode` attribute or by the `#arith.fenv` attribute. Collect both
+    // (the verifier guarantees they are never set at the same time) and remove
+    // them from the attributes carried over to the target op.
+    auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+        convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+    arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
+    convertedAttr.erase(srcOp.getFenvAttrName());
+
+    // Determine the rounding mode. The `fenv` attribute takes precedence; when
+    // neither source carries one, fall back to the default dynamic rounding
+    // mode.
+    [[maybe_unused]] arith::RoundingMode roundingMode =
+        arith::FenvAttr::getDefaultDynamicRoundingMode();
+    if (fenvAttr)
+      roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
+    else if (roundingModeAttr)
+      roundingMode = roundingModeAttr.getValue();
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
-      // Get the name of the rounding mode attribute.
-      StringRef arithAttrName = srcOp.getRoundingModeAttrName();
-      // Remove the source attribute.
-      auto arithAttr =
-          cast<arith::RoundingModeAttr>(convertedAttr.erase(arithAttrName));
-      // Set the target attribute.
-      convertedAttr.set(TargetOp::getRoundingModeAttrName(),
-                        convertArithRoundingModeAttrToLLVM(arithAttr));
+      convertedAttr.set(
+          TargetOp::getRoundingModeAttrName(),
+          LLVM::RoundingModeAttr::get(
+              ctx, convertArithRoundingModeToLLVM(roundingMode)));
     }
     // Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
     // support fastmath flags. Remove the arith fastmath attribute if present.
     if constexpr (SourceOp::template hasTrait<
                       arith::ArithFastMathInterface::Trait>())
       convertedAttr.erase(srcOp.getFastMathAttrName());
-    convertedAttr.set(TargetOp::getFPExceptionBehaviorAttrName(),
-                      getLLVMDefaultFPExceptionBehavior(*srcOp->getContext()));
+
+    // Determine the exception behavior from the `fenv` attribute, defaulting to
+    // `ignore` when no environment is specified (e.g. only a `roundingmode`
+    // attribute is present).
+    LLVM::FPExceptionBehavior exceptionBehavior =
+        LLVM::FPExceptionBehavior::Ignore;
+    if (fenvAttr)
+      exceptionBehavior = convertArithFPExceptionBehaviorToLLVM(
+          fenvAttr.getExceptionModeOrDefault(),
+          fenvAttr.getStrictExceptOrDefault());
+    convertedAttr.set(
+        TargetOp::getFPExceptionBehaviorAttrName(),
+        LLVM::FPExceptionBehaviorAttr::get(ctx, exceptionBehavior));
   }
 
   ArrayRef<NamedAttribute> getAttrs() const { return convertedAttr.getAttrs(); }
diff --git a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
index b3f5705601cfb..b964acdf7a137 100644
--- a/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
+++ b/mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp
@@ -81,3 +81,18 @@ mlir::arith::getLLVMDefaultFPExceptionBehavior(MLIRContext &context) {
   return LLVM::FPExceptionBehaviorAttr::get(&context,
                                             LLVM::FPExceptionBehavior::Ignore);
 }
+
+LLVM::FPExceptionBehavior mlir::arith::convertArithFPExceptionBehaviorToLLVM(
+    arith::FPExceptionMode exceptionMode, bool strictExcept) {
+  // A strict-exception requirement always maps to `strict`, which preserves
+  // both the side effects and the exact exception semantics regardless of the
+  // exception mode.
+  if (strictExcept)
+    return LLVM::FPExceptionBehavior::Strict;
+  // Without the strict requirement, a masked environment may ignore exceptions,
+  // while an unmasked or unknown environment must conservatively assume that
+  // traps may occur.
+  if (exceptionMode == arith::FPExceptionMode::Masked)
+    return LLVM::FPExceptionBehavior::Ignore;
+  return LLVM::FPExceptionBehavior::MayTrap;
+}
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..fed991c7074d2 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -29,20 +29,21 @@ using namespace mlir;
 
 namespace {
 
-/// Lowering pattern that matches only when the source op's rounding mode
-/// presence agrees with `HasRoundingMode`. This allows registering two
-/// instances of the same pattern for one source op: one that handles the
-/// unconstrained case (no rounding mode, lowering to a regular LLVM op) and
-/// one that handles the constrained case (rounding mode present, lowering to
-/// a constrained LLVM intrinsic).
+/// Lowering pattern that matches only when the source op's constrained
+/// floating-point environment presence agrees with `IsConstrained`. A source
+/// op is considered constrained when it carries either the deprecated
+/// `roundingmode` attribute or the `#arith.fenv` attribute. This allows
+/// registering two instances of the same pattern for one source op: one that
+/// handles the unconstrained case (lowering to a regular LLVM op) and one that
+/// handles the constrained case (lowering to a constrained LLVM intrinsic).
 ///
-/// * `HasRoundingMode`: the pattern matches if and only if the source op has
-///   a rounding mode attribute.
+/// * `IsConstrained`: the pattern matches if and only if the source op carries
+///   a `roundingmode` or `#arith.fenv` attribute.
 /// * `AttrConvert`: attribute converter to translate source attributes to
 ///   target attributes.
 /// * `FailOnUnsupportedFP`: whether to fail if the source op has unsupported
 ///   floating point types.
-template <typename SourceOp, typename TargetOp, bool HasRoundingMode,
+template <typename SourceOp, typename TargetOp, bool IsConstrained,
           template <typename, typename> typename AttrConvert =
               AttrConvertPassThrough,
           bool FailOnUnsupportedFP = false>
@@ -56,7 +57,8 @@ struct ConstrainedVectorConvertToLLVMPattern
   LogicalResult
   matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (HasRoundingMode != static_cast<bool>(op.getRoundingModeAttr()))
+    bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+    if (IsConstrained != opIsConstrained)
       return failure();
     return VectorConvertToLLVMPattern<
         SourceOp, TargetOp, AttrConvert,
@@ -89,11 +91,11 @@ struct IdentityBitcastLowering final
 
 using AddFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::AddFOp, LLVM::FAddOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedAddFOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    arith::AddFOp, LLVM::ConstrainedFAddIntr, /*HasRoundingMode=*/true,
+    arith::AddFOp, LLVM::ConstrainedFAddIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using AddIOpLowering =
     VectorConvertToLLVMPattern<arith::AddIOp, LLVM::AddOp,
@@ -103,11 +105,11 @@ using BitcastOpLowering =
     VectorConvertToLLVMPattern<arith::BitcastOp, LLVM::BitcastOp>;
 using DivFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::DivFOp, LLVM::FDivOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedDivFOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    arith::DivFOp, LLVM::ConstrainedFDivIntr, /*HasRoundingMode=*/true,
+    arith::DivFOp, LLVM::ConstrainedFDivIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using DivSIOpLowering =
     VectorConvertToLLVMPattern<arith::DivSIOp, LLVM::SDivOp>;
@@ -155,11 +157,11 @@ using MinUIOpLowering =
     VectorConvertToLLVMPattern<arith::MinUIOp, LLVM::UMinOp>;
 using MulFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::MulFOp, LLVM::FMulOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedMulFOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    arith::MulFOp, LLVM::ConstrainedFMulIntr, /*HasRoundingMode=*/true,
+    arith::MulFOp, LLVM::ConstrainedFMulIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using MulIOpLowering =
     VectorConvertToLLVMPattern<arith::MulIOp, LLVM::MulOp,
@@ -190,22 +192,22 @@ using SIToFPOpLowering =
     VectorConvertToLLVMPattern<arith::SIToFPOp, LLVM::SIToFPOp>;
 using SubFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::SubFOp, LLVM::FSubOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedSubFOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    arith::SubFOp, LLVM::ConstrainedFSubIntr, /*HasRoundingMode=*/true,
+    arith::SubFOp, LLVM::ConstrainedFSubIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using SubIOpLowering =
     VectorConvertToLLVMPattern<arith::SubIOp, LLVM::SubOp,
                                arith::AttrConvertOverflowToLLVM>;
 using TruncFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::TruncFOp, LLVM::FPTruncOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           AttrConvertPassThrough,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedTruncFOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*HasRoundingMode=*/true,
+    arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using TruncIOpLowering =
     VectorConvertToLLVMPattern<arith::TruncIOp, LLVM::TruncOp,
@@ -300,6 +302,12 @@ struct CmpFOpLowering : public ConvertOpToLLVMPattern<arith::CmpFOp> {
 /// Extends to f32 via llvm.fpext, then truncates to the target type via
 /// llvm.fptrunc. This handles bf16 <-> f16, which is the only same-bitwidth
 /// pair of LLVM-supported FP types.
+///
+/// When the op carries a constrained floating-point environment (through the
+/// deprecated `roundingmode` attribute or the `#arith.fenv` attribute), the two
+/// steps lower to the constrained intrinsics instead: the widening `fpext` is
+/// exact and only carries the exception behavior, while the narrowing `fptrunc`
+/// carries both the rounding mode and the exception behavior.
 struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
   using ConvertOpToLLVMPattern::ConvertOpToLLVMPattern;
 
@@ -322,12 +330,33 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
     if (!convertedType)
       return rewriter.notifyMatchFailure(op, "failed to convert result type");
 
+    // Derive the constrained floating-point environment, if any, from either
+    // the deprecated `roundingmode` attribute or the `#arith.fenv` attribute.
+    // The rounding mode only applies to the narrowing step; the exception
+    // behavior, when present, applies to both steps.
+    MLIRContext *ctx = rewriter.getContext();
+    LLVM::RoundingModeAttr roundingModeAttr;
+    LLVM::FPExceptionBehaviorAttr exceptionBehaviorAttr;
+    if (arith::FenvAttr fenvAttr = op.getFenvAttr()) {
+      roundingModeAttr = LLVM::RoundingModeAttr::get(
+          ctx, arith::convertArithRoundingModeToLLVM(
+                   fenvAttr.getDynamicRoundingModeOrDefault()));
+      exceptionBehaviorAttr = LLVM::FPExceptionBehaviorAttr::get(
+          ctx, arith::convertArithFPExceptionBehaviorToLLVM(
+                   fenvAttr.getExceptionModeOrDefault(),
+                   fenvAttr.getStrictExceptOrDefault()));
+    } else if (arith::RoundingModeAttr rmAttr = op.getRoundingmodeAttr()) {
+      roundingModeAttr = arith::convertArithRoundingModeAttrToLLVM(rmAttr);
+      exceptionBehaviorAttr = arith::getLLVMDefaultFPExceptionBehavior(*ctx);
+    }
+
     Value input = adaptor.getIn();
     Location loc = op.getLoc();
 
     if (!isa<LLVM::LLVMArrayType>(input.getType())) {
-      rewriter.replaceOp(op,
-                         emitConversion(rewriter, loc, input, convertedType));
+      rewriter.replaceOp(op, emitConversion(rewriter, loc, input, convertedType,
+                                            roundingModeAttr,
+                                            exceptionBehaviorAttr));
       return success();
     }
 
@@ -337,20 +366,33 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
     return LLVM::detail::handleMultidimensionalVectors(
         op.getOperation(), adaptor.getOperands(), *getTypeConverter(),
         [&](Type llvm1DVectorTy, ValueRange operands) -> Value {
-          return emitConversion(rewriter, loc, operands.front(),
-                                llvm1DVectorTy);
+          return emitConversion(rewriter, loc, operands.front(), llvm1DVectorTy,
+                                roundingModeAttr, exceptionBehaviorAttr);
         },
         rewriter);
   }
 
 private:
   static Value emitConversion(ConversionPatternRewriter &rewriter, Location loc,
-                              Value input, Type targetType) {
+                              Value input, Type targetType,
+                              LLVM::RoundingModeAttr roundingModeAttr,
+                              LLVM::FPExceptionBehaviorAttr
+                                  exceptionBehaviorAttr) {
     Type f32Scalar = Float32Type::get(rewriter.getContext());
     Type f32Ty = f32Scalar;
     if (auto vecTy = dyn_cast<VectorType>(targetType))
       f32Ty = VectorType::get(vecTy.getShape(), f32Scalar);
 
+    // A constrained floating-point environment was requested: emit the
+    // constrained intrinsics carrying the rounding mode and exception behavior.
+    if (exceptionBehaviorAttr) {
+      Value ext = LLVM::ConstrainedFPExtIntr::create(rewriter, loc, f32Ty, input,
+                                                     exceptionBehaviorAttr);
+      return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType, ext,
+                                                  roundingModeAttr,
+                                                  exceptionBehaviorAttr);
+    }
+
     Value ext = LLVM::FPExtOp::create(rewriter, loc, f32Ty, input);
     return LLVM::FPTruncOp::create(rewriter, loc, targetType, ext);
   }
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 1ed1ffd005d49..aca6973ab5956 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -37,13 +37,15 @@ using ConvertFMFMathToLLVMPattern =
     VectorConvertToLLVMPattern<SourceOp, TargetOp, ConvertFastMath,
                                FailOnUnsupportedFP>;
 
-/// Lowering pattern that matches only when the source op's rounding mode
-/// presence agrees with `HasRoundingMode`. Mirrors the helper of the same
-/// name in `mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp`. This lets us
-/// register two patterns for one math op: an unconstrained one that lowers
-/// to a regular LLVM op, and a constrained one (rounding mode present) that
-/// lowers to an `llvm.intr.experimental.constrained.*` intrinsic.
-template <typename SourceOp, typename TargetOp, bool HasRoundingMode,
+/// Lowering pattern that matches only when the source op's constrained
+/// floating-point environment presence agrees with `IsConstrained`. A source
+/// op is considered constrained when it carries either the deprecated
+/// `roundingmode` attribute or the `#arith.fenv` attribute. Mirrors the helper
+/// of the same name in `mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp`. This
+/// lets us register two patterns for one math op: an unconstrained one that
+/// lowers to a regular LLVM op, and a constrained one that lowers to an
+/// `llvm.intr.experimental.constrained.*` intrinsic.
+template <typename SourceOp, typename TargetOp, bool IsConstrained,
           template <typename, typename> typename AttrConvert =
               AttrConvertPassThrough,
           bool FailOnUnsupportedFP = true>
@@ -57,7 +59,8 @@ struct ConstrainedVectorConvertToLLVMPattern
   LogicalResult
   matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    if (HasRoundingMode != static_cast<bool>(op.getRoundingModeAttr()))
+    bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+    if (IsConstrained != opIsConstrained)
       return failure();
     return VectorConvertToLLVMPattern<
         SourceOp, TargetOp, AttrConvert,
@@ -84,11 +87,11 @@ using FloorOpLowering =
     ConvertFMFMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
 using FmaOpLowering =
     ConstrainedVectorConvertToLLVMPattern<math::FmaOp, LLVM::FMAOp,
-                                          /*HasRoundingMode=*/false,
+                                          /*IsConstrained=*/false,
                                           ConvertFastMath,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedFmaOpLowering = ConstrainedVectorConvertToLLVMPattern<
-    math::FmaOp, LLVM::ConstrainedFMAIntr, /*HasRoundingMode=*/true,
+    math::FmaOp, LLVM::ConstrainedFMAIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using Log10OpLowering =
     ConvertFMFMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index 85c6d82cffe5c..b104a3b9711e9 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -719,7 +719,8 @@ struct ScalingTruncFOpConverter
       scaleETy = b.getF8E8M0Type();
       scaleTy = cloneToShapedType(scaleTy, scaleETy);
       scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
-                                             op.getFastmathAttr(), nullptr);
+                                             op.getFastmathAttr(),
+                                             op.getFenvAttr());
     }
     if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
       return rewriter.notifyMatchFailure(
@@ -733,7 +734,8 @@ struct ScalingTruncFOpConverter
     scaleOperand =
         arith::ExtFOp::create(b, inputTy, scaleOperand, op.getFastmathAttr());
     Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
-                                         op.getFastmathAttr());
+                                         op.getFastmathAttr(), nullptr,
+                                         op.getFenvAttr());
     Value resultCast = arith::TruncFOp::create(
         b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
         op.getFenvAttr());
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 5d24ab17ee8c3..20e92d826c6b3 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -465,6 +465,47 @@ func.func @constrained_addf_with_fastmath(%arg0 : f64, %arg1 : f64) {
 
 // -----
 
+// Verify that the `#arith.fenv` attribute lowers to a constrained intrinsic,
+// mapping the dynamic rounding mode and the exception behavior. The exception
+// behavior depends on both `except_mode` and `strict_except`:
+//   masked           + strict_except=false -> ignore
+//   masked           + strict_except=true  -> strict
+//   unmasked|unknown + strict_except=false -> maytrap
+//   unmasked|unknown + strict_except=true  -> strict
+// CHECK-LABEL: experimental_constrained_fenv
+func.func @experimental_constrained_fenv(%arg0 : f64, %arg1 : f64) {
+// An empty fenv uses all defaults: dynamic rounding and ignored exceptions.
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 dynamic ignore
+  %0 = arith.addf %arg0, %arg1 fenv<> : f64
+// masked (default) + non-strict (default) -> ignore
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 tonearest ignore
+  %1 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = to_nearest_even> : f64
+// masked + strict -> strict
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 upward strict
+  %2 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = upward, strict_except = true> : f64
+// unmasked + non-strict -> maytrap
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 downward maytrap
+  %3 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = downward, except_mode = unmasked> : f64
+// unknown + non-strict -> maytrap
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 towardzero maytrap
+  %4 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+// unmasked + strict -> strict
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fadd %arg0, %arg1 tonearestaway strict
+  %5 = arith.addf %arg0, %arg1 fenv<dynamic_rounding_mode = to_nearest_away, except_mode = unmasked, strict_except = true> : f64
+  return
+}
+
+// -----
+
+// CHECK-LABEL: experimental_constrained_fptrunc_fenv
+func.func @experimental_constrained_fptrunc_fenv(%arg0 : f64) {
+// CHECK-NEXT: = llvm.intr.experimental.constrained.fptrunc %arg0 upward maytrap : f64 to f32
+  %0 = arith.truncf %arg0 fenv<dynamic_rounding_mode = upward, except_mode = unmasked> : f64 to f32
+  return
+}
+
+// -----
+
 // CHECK-LABEL: @convertf_f16_to_bf16
 func.func @convertf_f16_to_bf16(%arg0 : f16) -> bf16 {
 // CHECK-NEXT: %[[EXT:.*]] = llvm.fpext %arg0 : f16 to f32
@@ -498,6 +539,45 @@ func.func @convertf_vector(%arg0 : vector<2xf16>) -> vector<2xbf16> {
 
 // -----
 
+// A constrained floating-point environment lowers both conversion steps to the
+// constrained intrinsics: the widening `fpext` is exact and only carries the
+// exception behavior, while the narrowing `fptrunc` carries both the rounding
+// mode and the exception behavior.
+// CHECK-LABEL: @convertf_fenv
+func.func @convertf_fenv(%arg0 : f16) -> bf16 {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 maytrap : f16 to f32
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] upward maytrap : f32 to bf16
+  %0 = arith.convertf %arg0 fenv<dynamic_rounding_mode = upward, except_mode = unmasked> : f16 to bf16
+// CHECK-NEXT: return %[[TRUNC]]
+  return %0 : bf16
+}
+
+// -----
+
+// The deprecated `roundingmode` attribute also lowers to the constrained
+// intrinsics, using the default (`ignore`) exception behavior.
+// CHECK-LABEL: @convertf_roundingmode
+func.func @convertf_roundingmode(%arg0 : bf16) -> f16 {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 ignore : bf16 to f32
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] tonearest ignore : f32 to f16
+  %0 = arith.convertf %arg0 to_nearest_even : bf16 to f16
+// CHECK-NEXT: return %[[TRUNC]]
+  return %0 : f16
+}
+
+// -----
+
+// CHECK-LABEL: @convertf_fenv_vector
+func.func @convertf_fenv_vector(%arg0 : vector<2xf16>) -> vector<2xbf16> {
+// CHECK-NEXT: %[[EXT:.*]] = llvm.intr.experimental.constrained.fpext %arg0 strict : vector<2xf16> to vector<2xf32>
+// CHECK-NEXT: %[[TRUNC:.*]] = llvm.intr.experimental.constrained.fptrunc %[[EXT]] towardzero strict : vector<2xf32> to vector<2xbf16>
+  %0 = arith.convertf %arg0 fenv<dynamic_rounding_mode = toward_zero, strict_except = true> : vector<2xf16> to vector<2xbf16>
+// CHECK-NEXT: return %[[TRUNC]]
+  return %0 : vector<2xbf16>
+}
+
+// -----
+
 // Check sign and zero extension and truncation of integers.
 // CHECK-LABEL: @integer_extension_and_truncation
 func.func @integer_extension_and_truncation(%arg0 : i3) {
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index 638261bc6bde3..e0a89893eeb3a 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -692,3 +692,21 @@ func.func @constrained_fma_with_fastmath(%a : f64, %b : f64, %c : f64) {
   %0 = math.fma %a, %b, %c to_nearest_even fastmath<fast> : f64
   return
 }
+
+// -----
+
+// The `#arith.fenv` attribute lowers to the constrained fma intrinsic, mapping
+// the dynamic rounding mode and the exception behavior derived from
+// `except_mode` and `strict_except`.
+// CHECK-LABEL: func @experimental_constrained_fma_fenv
+func.func @experimental_constrained_fma_fenv(%a : f64, %b : f64, %c : f64) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} dynamic ignore : f64
+  %0 = math.fma %a, %b, %c fenv<> : f64
+  // masked + strict -> strict
+  // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} tonearest strict : f64
+  %1 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f64
+  // unknown + non-strict -> maytrap
+  // CHECK-NEXT: llvm.intr.experimental.constrained.fma %{{.*}}, %{{.*}}, %{{.*}} towardzero maytrap : f64
+  %2 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+  return
+}
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 75c4de2168761..ca0de33533ec1 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -349,6 +349,19 @@ func.func @scaling_truncf_propagate_rounding_mode_fast_math(%arg0 : vector<4xf16
 
 // -----
 
+func.func @scaling_truncf_propagate_fenv(%arg0 : vector<4xf16>, %arg1: vector<4xf16>) -> vector<4xf6E3M2FN> {
+    %0 = arith.scaling_truncf %arg0, %arg1 fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16>, vector<4xf16> to vector<4xf6E3M2FN>
+    return %0 : vector<4xf6E3M2FN>
+}
+// SCHECK-LABEL: @scaling_truncf_propagate_fenv
+// SCHECK: %[[SCALEF8:.+]] = arith.truncf %arg1 fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16> to vector<4xf8E8M0FNU>
+// SCHECK: %[[SCALEINTY:.+]] = arith.extf %[[SCALEF8]] : vector<4xf8E8M0FNU> to vector<4xf16>
+// SCHECK: %[[DIVF:.+]] = arith.divf %arg0, %[[SCALEINTY]] fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16>
+// SCHECK: %[[TRUNCF:.+]] = arith.truncf [[_:%[a-zA-Z0-9_]+]] fenv<dynamic_rounding_mode = upward, except_mode = unmasked, strict_except = true> : vector<4xf16> to vector<4xf6E3M2FN>
+// SCHECK: return %[[TRUNCF]] : vector<4xf6E3M2FN>
+
+// -----
+
 func.func @scaling_truncf_f16_to_f4E2M1FN_using_f16_scales(%arg0: f16, %arg1 : f16) -> f4E2M1FN {
     %0 = arith.scaling_truncf %arg0, %arg1 : f16, f16 to f4E2M1FN
     return %0 : f4E2M1FN

>From dc79401c1926a5f4bc205dff9965ee4a06c23f5b Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 16:46:41 -0700
Subject: [PATCH 4/7] [mlir][math] Add strict fp support to more operations

This change adds the arith.fenv attribute to transcendental operations
that may be affected by rounding mode or floating-point exception
constraints.
---
 .../ArithCommon/AttrToLLVMConverter.h         |  20 ++-
 mlir/include/mlir/Dialect/Math/IR/MathOps.td  | 169 +++++++++++++++---
 mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 160 ++++++++++++++---
 .../Conversion/MathToLLVM/math-to-llvm.mlir   |  87 +++++++++
 mlir/test/Dialect/Math/ops.mlir               |  18 ++
 5 files changed, 396 insertions(+), 58 deletions(-)

diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index 31d94daf4bb92..007c62481372c 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -159,12 +159,11 @@ class AttrConverterConstrainedFPToLLVM {
     convertedAttr = NamedAttrList{srcOp->getAttrs()};
     MLIRContext *ctx = srcOp->getContext();
 
-    // The floating-point environment may be described either by the deprecated
-    // `roundingmode` attribute or by the `#arith.fenv` attribute. Collect both
-    // (the verifier guarantees they are never set at the same time) and remove
-    // them from the attributes carried over to the target op.
-    auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
-        convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+    // The floating-point environment is described by the `#arith.fenv`
+    // attribute and, on operations that predate it, may also be described by
+    // the deprecated `roundingmode` attribute. Collect both (the verifier
+    // guarantees they are never set at the same time) and remove them from the
+    // attributes carried over to the target op.
     arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
     convertedAttr.erase(srcOp.getFenvAttrName());
 
@@ -173,10 +172,15 @@ class AttrConverterConstrainedFPToLLVM {
     // mode.
     [[maybe_unused]] arith::RoundingMode roundingMode =
         arith::FenvAttr::getDefaultDynamicRoundingMode();
+    if constexpr (SourceOp::template hasTrait<
+                      arith::ArithRoundingModeInterface::Trait>()) {
+      auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+          convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+      if (!fenvAttr && roundingModeAttr)
+        roundingMode = roundingModeAttr.getValue();
+    }
     if (fenvAttr)
       roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
-    else if (roundingModeAttr)
-      roundingMode = roundingModeAttr.getValue();
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 21c49484610a0..8c86436e514d4 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -67,6 +67,65 @@ class Math_FloatUnaryOp<string mnemonic, list<Trait> traits = []> :
                           attr-dict `:` type($result) }];
 }
 
+// Base class for unary floating point operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute.
+class Math_FloatUnaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Math_FloatUnaryOp<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$operand,
+      DefaultValuedAttr<Arith_FastMathAttr,
+                        "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With a trailing optional `fenv` attribute following the `fastmath`
+  // attribute, the generated builders no longer provide C++ default arguments,
+  // so the convenience builders that predate the `fenv` attribute must be
+  // provided explicitly.
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$operand,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, operand,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Value":$operand,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, operand, fastmath, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$operand,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, result, operand,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$operand,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, result, operand, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$operand,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, resultTypes, operand,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$operand,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, resultTypes, operand, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $operand (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Base class for binary math operations on integer types. Require two
 // operands and one result of the same type. This type can be an integer
 // type, vector or tensor thereof.
@@ -94,6 +153,67 @@ class Math_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
                           attr-dict `:` type($result) }];
 }
 
+// Base class for binary floating point operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute.
+class Math_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Math_FloatBinaryOp<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+      DefaultValuedAttr<Arith_FastMathAttr,
+                        "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With a trailing optional `fenv` attribute following the `fastmath`
+  // attribute, the generated builders no longer provide C++ default arguments,
+  // so the convenience builders that predate the `fenv` attribute must be
+  // provided explicitly.
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, result, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, result, lhs, rhs, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, resultTypes, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, resultTypes, lhs, rhs, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $lhs `,` $rhs (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Base class for floating point ternary operations. Require three operands and
 // one result of the same type. This type can be a floating point type, vector
 // or tensor thereof.
@@ -153,10 +273,7 @@ class Math_FloatTernaryOpWithRoundingMode<string mnemonic,
 }
 
 // Base class for floating point ternary operations that can carry an optional
-// floating-point environment (`#arith.fenv`) attribute. It extends
-// `Math_FloatTernaryOpWithRoundingMode` with the `fenv` attribute and the
-// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
-// floating-point environment constraints.
+// floating-point environment (`#arith.fenv`) attribute.
 class Math_FloatTernaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
     Math_FloatTernaryOpWithRoundingMode<mnemonic,
       !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
@@ -271,7 +388,7 @@ def Math_AcoshOp : Math_FloatUnaryOp<"acosh">{
 // AsinOp
 //===----------------------------------------------------------------------===//
 
-def Math_AsinOp : Math_FloatUnaryOp<"asin">{
+def Math_AsinOp : Math_FloatUnaryOpWithFenv<"asin">{
   let summary = "arcus sine of the given value";
   let description = [{
     Syntax:
@@ -325,7 +442,7 @@ def Math_AsinhOp : Math_FloatUnaryOp<"asinh">{
 // AtanOp
 //===----------------------------------------------------------------------===//
 
-def Math_AtanOp : Math_FloatUnaryOp<"atan">{
+def Math_AtanOp : Math_FloatUnaryOpWithFenv<"atan">{
   let summary = "arcus tangent of the given value";
   let description = [{
     The `atan` operation computes the arcus tangent of a given value.  It takes
@@ -373,7 +490,7 @@ def Math_AtanhOp : Math_FloatUnaryOp<"atanh">{
 // Atan2Op
 //===----------------------------------------------------------------------===//
 
-def Math_Atan2Op : Math_FloatBinaryOp<"atan2">{
+def Math_Atan2Op : Math_FloatBinaryOpWithFenv<"atan2">{
   let summary = "2-argument arcus tangent of the given values";
   let description = [{
     The `atan2` operation takes two operands and returns one result, all of
@@ -425,7 +542,7 @@ def Math_CbrtOp : Math_FloatUnaryOp<"cbrt"> {
 // CeilOp
 //===----------------------------------------------------------------------===//
 
-def Math_CeilOp : Math_FloatUnaryOp<"ceil"> {
+def Math_CeilOp : Math_FloatUnaryOpWithFenv<"ceil"> {
   let summary = "ceiling of the specified value";
   let description = [{
     The `ceil` operation computes the ceiling of a given value. It takes one
@@ -500,7 +617,7 @@ def Math_CopySignOp : Math_FloatBinaryOp<"copysign"> {
 // CosOp
 //===----------------------------------------------------------------------===//
 
-def Math_CosOp : Math_FloatUnaryOp<"cos"> {
+def Math_CosOp : Math_FloatUnaryOpWithFenv<"cos"> {
   let summary = "cosine of the specified value";
   let description = [{
     The `cos` operation computes the cosine of a given value. It takes one
@@ -521,7 +638,7 @@ def Math_CosOp : Math_FloatUnaryOp<"cos"> {
 // AcosOp
 //===----------------------------------------------------------------------===//
 
-def Math_AcosOp : Math_FloatUnaryOp<"acos"> {
+def Math_AcosOp : Math_FloatUnaryOpWithFenv<"acos"> {
   let summary = "arcus cosine of the specified value";
   let description = [{
     The `acos` operation computes the arcus cosine of a given value. It takes one
@@ -542,7 +659,7 @@ def Math_AcosOp : Math_FloatUnaryOp<"acos"> {
 // CoshOp
 //===----------------------------------------------------------------------===//
 
-def Math_CoshOp : Math_FloatUnaryOp<"cosh"> {
+def Math_CoshOp : Math_FloatUnaryOpWithFenv<"cosh"> {
   let summary = "hyperbolic cosine of the specified value";
   let description = [{
     The `cosh` operation computes the hyperbolic cosine. It takes one operand
@@ -563,7 +680,7 @@ def Math_CoshOp : Math_FloatUnaryOp<"cosh"> {
 // SinOp
 //===----------------------------------------------------------------------===//
 
-def Math_SinOp : Math_FloatUnaryOp<"sin"> {
+def Math_SinOp : Math_FloatUnaryOpWithFenv<"sin"> {
   let summary = "sine of the specified value";
   let description = [{
     The `sin` operation computes the sine of a given value. It takes one
@@ -584,7 +701,7 @@ def Math_SinOp : Math_FloatUnaryOp<"sin"> {
 // SinhOp
 //===----------------------------------------------------------------------===//
 
-def Math_SinhOp : Math_FloatUnaryOp<"sinh"> {
+def Math_SinhOp : Math_FloatUnaryOpWithFenv<"sinh"> {
   let summary = "hyperbolic sine of the specified value";
   let description = [{
     The `sinh` operation computes the hyperbolic sine. It takes one operand
@@ -750,7 +867,7 @@ def Math_ErfcOp : Math_FloatUnaryOp<"erfc"> {
 // ExpOp
 //===----------------------------------------------------------------------===//
 
-def Math_ExpOp : Math_FloatUnaryOp<"exp"> {
+def Math_ExpOp : Math_FloatUnaryOpWithFenv<"exp"> {
   let summary = "base-e exponential of the specified value";
   let description = [{
     The `exp` operation takes one operand of floating point type (i.e., scalar,
@@ -771,7 +888,7 @@ def Math_ExpOp : Math_FloatUnaryOp<"exp"> {
 // Exp2Op
 //===----------------------------------------------------------------------===//
 
-def Math_Exp2Op : Math_FloatUnaryOp<"exp2"> {
+def Math_Exp2Op : Math_FloatUnaryOpWithFenv<"exp2"> {
   let summary = "base-2 exponential of the specified value";
 
   let description = [{
@@ -816,7 +933,7 @@ def Math_ExpM1Op : Math_FloatUnaryOp<"expm1"> {
 // FloorOp
 //===----------------------------------------------------------------------===//
 
-def Math_FloorOp : Math_FloatUnaryOp<"floor"> {
+def Math_FloorOp : Math_FloatUnaryOpWithFenv<"floor"> {
   let summary = "floor of the specified value";
   let description = [{
     The `floor` operation computes the floor of a given value. It takes one
@@ -965,7 +1082,7 @@ def Math_IsNormalOp : Math_FloatClassificationOp<"isnormal"> {
 // LogOp
 //===----------------------------------------------------------------------===//
 
-def Math_LogOp : Math_FloatUnaryOp<"log"> {
+def Math_LogOp : Math_FloatUnaryOpWithFenv<"log"> {
   let summary = "base-e logarithm of the specified value";
 
   let description = [{
@@ -987,7 +1104,7 @@ def Math_LogOp : Math_FloatUnaryOp<"log"> {
 // Log10Op
 //===----------------------------------------------------------------------===//
 
-def Math_Log10Op : Math_FloatUnaryOp<"log10"> {
+def Math_Log10Op : Math_FloatUnaryOpWithFenv<"log10"> {
   let summary = "base-10 logarithm of the specified value";
 
   let description = [{
@@ -1033,7 +1150,7 @@ def Math_Log1pOp : Math_FloatUnaryOp<"log1p"> {
 // Log2Op
 //===----------------------------------------------------------------------===//
 
-def Math_Log2Op : Math_FloatUnaryOp<"log2"> {
+def Math_Log2Op : Math_FloatUnaryOpWithFenv<"log2"> {
   let summary = "base-2 logarithm of the specified value";
 
   let description = [{
@@ -1055,7 +1172,7 @@ def Math_Log2Op : Math_FloatUnaryOp<"log2"> {
 // PowFOp
 //===----------------------------------------------------------------------===//
 
-def Math_PowFOp : Math_FloatBinaryOp<"powf"> {
+def Math_PowFOp : Math_FloatBinaryOpWithFenv<"powf"> {
   let summary = "floating point raised to the power of operation";
   let description = [{
     The `powf` operation takes two operands of floating point type (i.e.,
@@ -1097,7 +1214,7 @@ def Math_RsqrtOp : Math_FloatUnaryOp<"rsqrt"> {
 // SqrtOp
 //===----------------------------------------------------------------------===//
 
-def Math_SqrtOp : Math_FloatUnaryOp<"sqrt"> {
+def Math_SqrtOp : Math_FloatUnaryOpWithFenv<"sqrt"> {
   let summary = "sqrt of the specified value";
   let description = [{
     The `sqrt` operation computes the square root. It takes one operand of
@@ -1118,7 +1235,7 @@ def Math_SqrtOp : Math_FloatUnaryOp<"sqrt"> {
 // TanOp
 //===----------------------------------------------------------------------===//
 
-def Math_TanOp : Math_FloatUnaryOp<"tan"> {
+def Math_TanOp : Math_FloatUnaryOpWithFenv<"tan"> {
   let summary = "tangent of the specified value";
   let description = [{
     The `tan` operation computes the tangent. It takes one operand
@@ -1139,7 +1256,7 @@ def Math_TanOp : Math_FloatUnaryOp<"tan"> {
 // TanhOp
 //===----------------------------------------------------------------------===//
 
-def Math_TanhOp : Math_FloatUnaryOp<"tanh"> {
+def Math_TanhOp : Math_FloatUnaryOpWithFenv<"tanh"> {
   let summary = "hyperbolic tangent of the specified value";
   let description = [{
     The `tanh` operation computes the hyperbolic tangent. It takes one operand
@@ -1160,7 +1277,7 @@ def Math_TanhOp : Math_FloatUnaryOp<"tanh"> {
 // RoundEvenOp
 //===----------------------------------------------------------------------===//
 
-def Math_RoundEvenOp : Math_FloatUnaryOp<"roundeven"> {
+def Math_RoundEvenOp : Math_FloatUnaryOpWithFenv<"roundeven"> {
   let summary = "round of the specified value with halfway cases to even";
   let description = [{
     The `roundeven` operation returns the operand rounded to the nearest integer
@@ -1184,7 +1301,7 @@ def Math_RoundEvenOp : Math_FloatUnaryOp<"roundeven"> {
 // RoundOp
 //===----------------------------------------------------------------------===//
 
-def Math_RoundOp : Math_FloatUnaryOp<"round"> {
+def Math_RoundOp : Math_FloatUnaryOpWithFenv<"round"> {
   let summary = "round of the specified value";
   let description = [{
     The `round` operation returns the operand rounded to the nearest integer
@@ -1208,7 +1325,7 @@ def Math_RoundOp : Math_FloatUnaryOp<"round"> {
 // TruncOp
 //===----------------------------------------------------------------------===//
 
-def Math_TruncOp : Math_FloatUnaryOp<"trunc"> {
+def Math_TruncOp : Math_FloatUnaryOpWithFenv<"trunc"> {
   let summary = "trunc of the specified value";
   let description = [{
     The `trunc` operation returns the operand rounded to the nearest integer
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index aca6973ab5956..4786a3f0cc17e 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -59,7 +59,13 @@ struct ConstrainedVectorConvertToLLVMPattern
   LogicalResult
   matchAndRewrite(SourceOp op, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    bool opIsConstrained = op.getRoundingModeAttr() || op.getFenvAttr();
+    bool opIsConstrained = static_cast<bool>(op.getFenvAttr());
+    // Operations that predate the `#arith.fenv` attribute may also carry the
+    // deprecated `roundingmode` attribute.
+    if constexpr (SourceOp::template hasTrait<
+                      arith::ArithRoundingModeInterface::Trait>())
+      opIsConstrained =
+          opIsConstrained || static_cast<bool>(op.getRoundingModeAttr());
     if (IsConstrained != opIsConstrained)
       return failure();
     return VectorConvertToLLVMPattern<
@@ -68,23 +74,62 @@ struct ConstrainedVectorConvertToLLVMPattern
   }
 };
 
+// Convenience alias for the pattern that lowers a math op carrying no
+// floating-point environment constraint to a regular LLVM intrinsic op,
+// converting the fastmath flags.
+template <typename SourceOp, typename TargetOp>
+using ConvertUnconstrainedMathToLLVMPattern =
+    ConstrainedVectorConvertToLLVMPattern<SourceOp, TargetOp,
+                                          /*IsConstrained=*/false, ConvertFastMath,
+                                          /*FailOnUnsupportedFP=*/true>;
+
+// Convenience alias for the pattern that lowers a math op carrying a
+// floating-point environment constraint (the `#arith.fenv` attribute) to the
+// matching `llvm.intr.experimental.constrained.*` intrinsic.
+template <typename SourceOp, typename TargetOp>
+using ConvertConstrainedMathToLLVMPattern =
+    ConstrainedVectorConvertToLLVMPattern<
+        SourceOp, TargetOp, /*IsConstrained=*/true,
+        arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
+
 using AbsFOpLowering =
     ConvertFMFMathToLLVMPattern<math::AbsFOp, LLVM::FAbsOp,
                                 /*FailOnUnsupportedFP=*/true>;
-using CeilOpLowering = ConvertFMFMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
+using CeilOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
+using ConstrainedCeilOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::CeilOp, LLVM::ConstrainedCeilIntr>;
 using CopySignOpLowering =
     ConvertFMFMathToLLVMPattern<math::CopySignOp, LLVM::CopySignOp>;
-using CosOpLowering = ConvertFMFMathToLLVMPattern<math::CosOp, LLVM::CosOp>;
-using CoshOpLowering = ConvertFMFMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
-using AcosOpLowering = ConvertFMFMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
+using CosOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::CosOp, LLVM::CosOp>;
+using ConstrainedCosOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::CosOp, LLVM::ConstrainedCosIntr>;
+using CoshOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
+using ConstrainedCoshOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::CoshOp, LLVM::ConstrainedCoshIntr>;
+using AcosOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
+using ConstrainedAcosOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::AcosOp, LLVM::ConstrainedACosIntr>;
 using CtPopFOpLowering =
     VectorConvertToLLVMPattern<math::CtPopOp, LLVM::CtPopOp,
                                AttrConvertPassThrough,
                                /*FailOnUnsupportedFP=*/true>;
-using Exp2OpLowering = ConvertFMFMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
-using ExpOpLowering = ConvertFMFMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
+using Exp2OpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
+using ConstrainedExp2OpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::Exp2Op, LLVM::ConstrainedExp2Intr>;
+using ExpOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
+using ConstrainedExpOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::ExpOp, LLVM::ConstrainedExpIntr>;
 using FloorOpLowering =
-    ConvertFMFMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::FloorOp, LLVM::FFloorOp>;
+using ConstrainedFloorOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::FloorOp,
+                                        LLVM::ConstrainedFloorIntr>;
 using FmaOpLowering =
     ConstrainedVectorConvertToLLVMPattern<math::FmaOp, LLVM::FMAOp,
                                           /*IsConstrained=*/false,
@@ -94,27 +139,72 @@ using ConstrainedFmaOpLowering = ConstrainedVectorConvertToLLVMPattern<
     math::FmaOp, LLVM::ConstrainedFMAIntr, /*IsConstrained=*/true,
     arith::AttrConverterConstrainedFPToLLVM, /*FailOnUnsupportedFP=*/true>;
 using Log10OpLowering =
-    ConvertFMFMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
-using Log2OpLowering = ConvertFMFMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
-using LogOpLowering = ConvertFMFMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
-using PowFOpLowering = ConvertFMFMathToLLVMPattern<math::PowFOp, LLVM::PowOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::Log10Op, LLVM::Log10Op>;
+using ConstrainedLog10OpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::Log10Op,
+                                        LLVM::ConstrainedLog10Intr>;
+using Log2OpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
+using ConstrainedLog2OpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::Log2Op, LLVM::ConstrainedLog2Intr>;
+using LogOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
+using ConstrainedLogOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::LogOp, LLVM::ConstrainedLogIntr>;
+using PowFOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::PowFOp, LLVM::PowOp>;
+using ConstrainedPowFOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::PowFOp, LLVM::ConstrainedPowIntr>;
 using FPowIOpLowering =
     ConvertFMFMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
 using RoundEvenOpLowering =
-    ConvertFMFMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
+using ConstrainedRoundEvenOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::RoundEvenOp,
+                                        LLVM::ConstrainedRoundEvenIntr>;
 using RoundOpLowering =
-    ConvertFMFMathToLLVMPattern<math::RoundOp, LLVM::RoundOp>;
-using SinOpLowering = ConvertFMFMathToLLVMPattern<math::SinOp, LLVM::SinOp>;
-using SinhOpLowering = ConvertFMFMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
-using ASinOpLowering = ConvertFMFMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
-using SqrtOpLowering = ConvertFMFMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::RoundOp, LLVM::RoundOp>;
+using ConstrainedRoundOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::RoundOp,
+                                        LLVM::ConstrainedRoundIntr>;
+using SinOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::SinOp, LLVM::SinOp>;
+using ConstrainedSinOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::SinOp, LLVM::ConstrainedSinIntr>;
+using SinhOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
+using ConstrainedSinhOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::SinhOp, LLVM::ConstrainedSinhIntr>;
+using ASinOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
+using ConstrainedASinOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::AsinOp, LLVM::ConstrainedASinIntr>;
+using SqrtOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
+using ConstrainedSqrtOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::SqrtOp, LLVM::ConstrainedSqrtIntr>;
 using FTruncOpLowering =
-    ConvertFMFMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
-using TanOpLowering = ConvertFMFMathToLLVMPattern<math::TanOp, LLVM::TanOp>;
-using TanhOpLowering = ConvertFMFMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
-using ATanOpLowering = ConvertFMFMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
+using ConstrainedFTruncOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::TruncOp,
+                                        LLVM::ConstrainedTruncIntr>;
+using TanOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::TanOp, LLVM::TanOp>;
+using ConstrainedTanOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::TanOp, LLVM::ConstrainedTanIntr>;
+using TanhOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
+using ConstrainedTanhOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::TanhOp, LLVM::ConstrainedTanhIntr>;
+using ATanOpLowering =
+    ConvertUnconstrainedMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
+using ConstrainedATanOpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::AtanOp, LLVM::ConstrainedATanIntr>;
 using ATan2OpLowering =
-    ConvertFMFMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
+    ConvertUnconstrainedMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
+using ConstrainedATan2OpLowering =
+    ConvertConstrainedMathToLLVMPattern<math::Atan2Op,
+                                        LLVM::ConstrainedATan2Intr>;
 // A `CtLz/CtTz/absi(a)` is converted into `CtLz/CtTz/absi(a, false)`.
 // TODO: Result and operand types match for `absi` as opposed to `ct*z`, so it
 // may be better to separate the patterns.
@@ -468,37 +558,59 @@ void mlir::populateMathToLLVMConversionPatterns(
     AbsFOpLowering,
     AbsIOpLowering,
     CeilOpLowering,
+    ConstrainedCeilOpLowering,
     CopySignOpLowering,
     CosOpLowering,
+    ConstrainedCosOpLowering,
     CoshOpLowering,
+    ConstrainedCoshOpLowering,
     AcosOpLowering,
+    ConstrainedAcosOpLowering,
     CountLeadingZerosOpLowering,
     CountTrailingZerosOpLowering,
     CtPopFOpLowering,
     Exp2OpLowering,
+    ConstrainedExp2OpLowering,
     ExpM1OpLowering,
     ExpOpLowering,
+    ConstrainedExpOpLowering,
     FPowIOpLowering,
     FloorOpLowering,
+    ConstrainedFloorOpLowering,
     FmaOpLowering,
     ConstrainedFmaOpLowering,
     Log10OpLowering,
+    ConstrainedLog10OpLowering,
     Log2OpLowering,
+    ConstrainedLog2OpLowering,
     LogOpLowering,
+    ConstrainedLogOpLowering,
     PowFOpLowering,
+    ConstrainedPowFOpLowering,
     RoundEvenOpLowering,
+    ConstrainedRoundEvenOpLowering,
     RoundOpLowering,
+    ConstrainedRoundOpLowering,
     RsqrtOpLowering,
     SincosOpLowering,
     SinOpLowering,
+    ConstrainedSinOpLowering,
     SinhOpLowering,
+    ConstrainedSinhOpLowering,
     ASinOpLowering,
+    ConstrainedASinOpLowering,
     SqrtOpLowering,
+    ConstrainedSqrtOpLowering,
     FTruncOpLowering,
+    ConstrainedFTruncOpLowering,
     TanOpLowering,
+    ConstrainedTanOpLowering,
     TanhOpLowering,
+    ConstrainedTanhOpLowering,
     ATanOpLowering,
-    ATan2OpLowering
+    ConstrainedATanOpLowering,
+    ATan2OpLowering,
+    ConstrainedATan2OpLowering
   >(converter, benefit);
   // clang-format on
 }
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index e0a89893eeb3a..67edabe4f7c31 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -710,3 +710,90 @@ func.func @experimental_constrained_fma_fenv(%a : f64, %b : f64, %c : f64) {
   %2 = math.fma %a, %b, %c fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
   return
 }
+
+// -----
+
+// Unary transcendental math ops with the `#arith.fenv` attribute lower to the
+// matching constrained intrinsic, carrying a rounding mode and exception
+// behavior.
+// CHECK-LABEL: func @experimental_constrained_unary
+func.func @experimental_constrained_unary(%a : f64) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.sqrt %{{.*}} dynamic ignore : f64
+  %0 = math.sqrt %a fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.exp %{{.*}} tonearest strict : f64
+  %1 = math.exp %a fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.log %{{.*}} towardzero maytrap : f64
+  %2 = math.log %a fenv<dynamic_rounding_mode = toward_zero, except_mode = unknown> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.sin %{{.*}} dynamic ignore : f64
+  %3 = math.sin %a fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.cos %{{.*}} dynamic ignore : f64
+  %4 = math.cos %a fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.tanh %{{.*}} dynamic ignore : f64
+  %5 = math.tanh %a fenv<> : f64
+  return
+}
+
+// -----
+
+// Without the `#arith.fenv` attribute, the unconstrained lowering is used.
+// CHECK-LABEL: func @unconstrained_sqrt
+func.func @unconstrained_sqrt(%a : f64) {
+  // CHECK-NEXT: llvm.intr.sqrt(%{{.*}}) : (f64) -> f64
+  // CHECK-NOT: constrained
+  %0 = math.sqrt %a : f64
+  return
+}
+
+// -----
+
+// Rounding math ops (ceil/floor/round/roundeven/trunc) map to constrained
+// intrinsics that only carry the exception behavior (no rounding mode).
+// CHECK-LABEL: func @experimental_constrained_rounding
+func.func @experimental_constrained_rounding(%a : f64) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.ceil %{{.*}} ignore : f64
+  %0 = math.ceil %a fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.floor %{{.*}} maytrap : f64
+  %1 = math.floor %a fenv<except_mode = unknown> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.trunc %{{.*}} strict : f64
+  %2 = math.trunc %a fenv<strict_except = true> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.round %{{.*}} ignore : f64
+  %3 = math.round %a fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.roundeven %{{.*}} ignore : f64
+  %4 = math.roundeven %a fenv<> : f64
+  return
+}
+
+// -----
+
+// Binary math ops with the `#arith.fenv` attribute lower to the matching
+// constrained intrinsic.
+// CHECK-LABEL: func @experimental_constrained_binary
+func.func @experimental_constrained_binary(%a : f64, %b : f64) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.pow %{{.*}}, %{{.*}} dynamic ignore : f64
+  %0 = math.powf %a, %b fenv<> : f64
+  // CHECK-NEXT: llvm.intr.experimental.constrained.atan2 %{{.*}}, %{{.*}} tonearest ignore : f64
+  %1 = math.atan2 %a, %b fenv<dynamic_rounding_mode = to_nearest_even> : f64
+  return
+}
+
+// -----
+
+// The constrained lowering also applies element-wise to vector operands.
+// CHECK-LABEL: func @experimental_constrained_vector
+func.func @experimental_constrained_vector(%a : vector<4xf32>) {
+  // CHECK: llvm.intr.experimental.constrained.sqrt {{.*}} dynamic ignore : vector<4xf32>
+  %0 = math.sqrt %a fenv<> : vector<4xf32>
+  return
+}
+
+// -----
+
+// Constrained intrinsics do not carry fastmath flags. The fastmath attribute is
+// dropped during the lowering.
+// CHECK-LABEL: func @constrained_unary_with_fastmath
+func.func @constrained_unary_with_fastmath(%a : f64) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.sqrt %{{.*}} dynamic ignore : f64
+  // CHECK-NOT: fastmath
+  %0 = math.sqrt %a fastmath<fast> fenv<> : f64
+  return
+}
diff --git a/mlir/test/Dialect/Math/ops.mlir b/mlir/test/Dialect/Math/ops.mlir
index b33f3d1eeea97..30912a7cf597a 100644
--- a/mlir/test/Dialect/Math/ops.mlir
+++ b/mlir/test/Dialect/Math/ops.mlir
@@ -317,6 +317,24 @@ func.func @fastmath(%f: f32, %i: i32, %v: vector<4xf32>, %t: tensor<4x4x?xf32>)
   return
 }
 
+// CHECK-LABEL: func @fenv(
+// CHECK-SAME:      %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>)
+func.func @fenv(%f: f32, %v: vector<4xf32>) {
+  // CHECK: math.sqrt %[[F]] fenv<> : f32
+  %0 = math.sqrt %f fenv<> : f32
+  // CHECK: math.exp %[[F]] fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
+  %1 = math.exp %f fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
+  // CHECK: math.sin %[[V]] fastmath<contract> fenv<except_mode = unknown> : vector<4xf32>
+  %2 = math.sin %v fastmath<contract> fenv<except_mode = unknown> : vector<4xf32>
+  // CHECK: math.ceil %[[F]] fenv<> : f32
+  %3 = math.ceil %f fenv<> : f32
+  // CHECK: math.powf %[[F]], %[[F]] fenv<dynamic_rounding_mode = toward_zero> : f32
+  %4 = math.powf %f, %f fenv<dynamic_rounding_mode = toward_zero> : f32
+  // CHECK: math.atan2 %[[F]], %[[F]] fenv<> : f32
+  %5 = math.atan2 %f, %f fenv<> : f32
+  return
+}
+
 // CHECK-LABEL: func @fpclassify(
 // CHECK-SAME:    %[[F:.+]]: f32, %[[D:.+]]: f64,
 // CHECK-SAME:    %[[V:.+]]: vector<4xf32>, %[[T:.+]]: tensor<4x?xf32>

>From 17921ffeb45f8956a39c3cfaa127594cba0e74b7 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 17:31:01 -0700
Subject: [PATCH 5/7] [mlir][llvm] Add a constrained version of the powi
 operation

This adds a version of the powi operation that maps to the constrained
intrinsic.

WIP! Need a test.
---
 mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
index b912cef1386dd..5840f80dfe704 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td
@@ -635,6 +635,16 @@ def LLVM_ConstrainedTruncIntr     : LLVM_ConstrainedUnaryIntrNoRM<"trunc">;
 def LLVM_ConstrainedPowIntr   : LLVM_ConstrainedBinaryIntrRM<"pow">;
 def LLVM_ConstrainedATan2Intr : LLVM_ConstrainedBinaryIntrRM<"atan2">;
 
+def LLVM_ConstrainedPowIIntr
+    : LLVM_ConstrainedIntr<"powi", /*numArgs=*/2,
+        /*overloadedResult=*/1, /*overloadedOperands=*/[],
+        /*hasRoundingMode=*/1> {
+  let assemblyFormat = [{
+    $arg_0 `,` $arg_1 $roundingmode $fpExceptionBehavior attr-dict `:`
+    functional-type(operands, results)
+  }];
+}
+
 def LLVM_ConstrainedUIToFP
     : LLVM_ConstrainedIntr<"uitofp", /*numArgs=*/1,
         /*overloadedResult=*/1, /*overloadedOperands=*/[0],

>From b97d67c990e481e989af4f62fd8154eaf9027b23 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Thu, 2 Jul 2026 17:32:06 -0700
Subject: [PATCH 6/7] [mlir][math] Add fenv support for the powi operation

This adds the arith.fenv attribute (optionally) to the math.powi
operation and adds handling to lower to the corresponding
constrained LLVM operation when it is present.

Note that the math.powi operation currently requires the floating-point
input operand and the integer exponent operand to have the same number
of elements if the input value is a vector. This is a mismatch with the
llvm.powi operation (and the documented requirements of the LLVM IR
llvm.powi intrinsic), and so that form can't be successfully lowered.
---
 mlir/include/mlir/Dialect/Math/IR/MathOps.td  | 56 ++++++++++++++++++-
 mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 43 +++++++++++++-
 mlir/lib/Dialect/Math/IR/MathOps.cpp          |  5 ++
 .../Conversion/MathToLLVM/math-to-llvm.mlir   | 23 ++++++++
 mlir/test/Dialect/Math/ops.mlir               |  6 +-
 5 files changed, 128 insertions(+), 5 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Math/IR/MathOps.td b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
index 8c86436e514d4..8d03cafcdfe5a 100644
--- a/mlir/include/mlir/Dialect/Math/IR/MathOps.td
+++ b/mlir/include/mlir/Dialect/Math/IR/MathOps.td
@@ -1350,7 +1350,8 @@ def Math_TruncOp : Math_FloatUnaryOpWithFenv<"trunc"> {
 
 def Math_FPowIOp : Math_Op<"fpowi",
     [SameOperandsAndResultShape, AllTypesMatch<["lhs", "result"]>,
-     DeclareOpInterfaceMethods<ArithFastMathInterface>]> {
+     DeclareOpInterfaceMethods<ArithFastMathInterface>,
+     DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>]> {
   let summary = "floating point raised to the signed integer power";
   let description = [{
     The `fpowi` operation takes a `base` operand of floating point type
@@ -1375,13 +1376,64 @@ def Math_FPowIOp : Math_Op<"fpowi",
     // Scalar exponentiation.
     %a = math.fpowi %base, %power : f64, i32
     ```
+
+    An optional `#arith.fenv` attribute may be attached to describe constraints
+    on the floating-point environment.
   }];
 
   let arguments = (ins FloatLike:$lhs, SignlessIntegerOrIndexLike:$rhs,
       DefaultValuedAttr<Arith_FastMathAttr,
-                        "::mlir::arith::FastMathFlags::none">:$fastmath);
+                        "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
   let results = (outs FloatLike:$result);
+  // With a trailing optional `fenv` attribute following the `fastmath`
+  // attribute, the generated builders no longer provide C++ default arguments,
+  // so the convenience builders that predate the `fenv` attribute must be
+  // provided explicitly.
+  let builders = [
+    OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Value":$lhs, "::mlir::Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, result, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::Type":$result, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, result, lhs, rhs, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, resultTypes, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "::mlir::TypeRange":$resultTypes, "::mlir::Value":$lhs,
+      "::mlir::Value":$rhs, "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, resultTypes, lhs, rhs, fastmath,
+            ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
   let assemblyFormat = [{ $lhs `,` $rhs (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($lhs) `,` type($rhs) }];
 
   let hasFolder = 1;
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 4786a3f0cc17e..84b716f578568 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -156,7 +156,7 @@ using PowFOpLowering =
 using ConstrainedPowFOpLowering =
     ConvertConstrainedMathToLLVMPattern<math::PowFOp, LLVM::ConstrainedPowIntr>;
 using FPowIOpLowering =
-    ConvertFMFMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
+    ConvertUnconstrainedMathToLLVMPattern<math::FPowIOp, LLVM::PowIOp>;
 using RoundEvenOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::RoundEvenOp, LLVM::RoundEvenOp>;
 using ConstrainedRoundEvenOpLowering =
@@ -205,6 +205,46 @@ using ATan2OpLowering =
 using ConstrainedATan2OpLowering =
     ConvertConstrainedMathToLLVMPattern<math::Atan2Op,
                                         LLVM::ConstrainedATan2Intr>;
+
+// A `math.fpowi` carrying a floating-point environment (`#arith.fenv`) lowers
+// to the constrained `powi` intrinsic.
+struct ConstrainedFPowIOpLowering
+    : public ConvertOpToLLVMPattern<math::FPowIOp,
+                                    /*FailOnUnsupportedFP=*/true> {
+  using ConvertOpToLLVMPattern<
+      math::FPowIOp, /*FailOnUnsupportedFP=*/true>::ConvertOpToLLVMPattern;
+
+  LogicalResult
+  matchAndRewrite(math::FPowIOp op, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (!op.getFenvAttr())
+      return rewriter.notifyMatchFailure(
+          op, "unconstrained fpowi is handled by a separate pattern");
+
+    // `llvm.intr.experimental.constrained.powi` requires a scalar operand with
+    // an `i32` exponent.
+    Type resultType = op.getResult().getType();
+    if (!isa<FloatType>(resultType))
+      return rewriter.notifyMatchFailure(
+          op, "constrained fpowi only supports scalar operands");
+    if (!op.getRhs().getType().isInteger(32))
+      return rewriter.notifyMatchFailure(
+          op, "constrained fpowi requires an i32 exponent");
+
+    Type llvmResultType = getTypeConverter()->convertType(resultType);
+    if (!llvmResultType)
+      return failure();
+
+    arith::AttrConverterConstrainedFPToLLVM<math::FPowIOp,
+                                            LLVM::ConstrainedPowIIntr>
+        attrConvert(op);
+    rewriter.replaceOpWithNewOp<LLVM::ConstrainedPowIIntr>(
+        op, llvmResultType, ValueRange{adaptor.getLhs(), adaptor.getRhs()},
+        attrConvert.getAttrs());
+    return success();
+  }
+};
+
 // A `CtLz/CtTz/absi(a)` is converted into `CtLz/CtTz/absi(a, false)`.
 // TODO: Result and operand types match for `absi` as opposed to `ct*z`, so it
 // may be better to separate the patterns.
@@ -575,6 +615,7 @@ void mlir::populateMathToLLVMConversionPatterns(
     ExpOpLowering,
     ConstrainedExpOpLowering,
     FPowIOpLowering,
+    ConstrainedFPowIOpLowering,
     FloorOpLowering,
     ConstrainedFloorOpLowering,
     FmaOpLowering,
diff --git a/mlir/lib/Dialect/Math/IR/MathOps.cpp b/mlir/lib/Dialect/Math/IR/MathOps.cpp
index b900cb1911759..2faf3fe3cd6b8 100644
--- a/mlir/lib/Dialect/Math/IR/MathOps.cpp
+++ b/mlir/lib/Dialect/Math/IR/MathOps.cpp
@@ -828,6 +828,11 @@ OpFoldResult math::TruncOp::fold(FoldAdaptor adaptor) {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult math::FPowIOp::fold(FoldAdaptor adaptor) {
+  // Do not fold when a constrained floating-point environment is requested.
+  // In theory, we could fold this when we know the rounding mode and we can
+  // verify that the operation wouldn't raise an exception.
+  if (getFenvAttr())
+    return {};
   return constFoldBinaryOpConditional<FloatAttr, IntegerAttr>(
       adaptor.getOperands(),
       [](const APFloat &base, const APInt &exp) -> std::optional<APFloat> {
diff --git a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
index 67edabe4f7c31..6e4f8fe1ffc0d 100644
--- a/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
+++ b/mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir
@@ -778,6 +778,29 @@ func.func @experimental_constrained_binary(%a : f64, %b : f64) {
 
 // -----
 
+// `math.fpowi` with a floating-point environment lowers to the constrained
+// `powi` intrinsic, which takes a scalar `i32` exponent.
+// CHECK-LABEL: func @experimental_constrained_fpowi
+func.func @experimental_constrained_fpowi(%a : f64, %b : i32) {
+  // CHECK-NEXT: llvm.intr.experimental.constrained.powi %{{.*}}, %{{.*}} tonearest maytrap : (f64, i32) -> f64
+  %0 = math.fpowi %a, %b fenv<dynamic_rounding_mode = to_nearest_even, except_mode = unknown> : f64, i32
+  return
+}
+
+// -----
+
+// Without the `#arith.fenv` attribute, `math.fpowi` uses the unconstrained
+// `llvm.intr.powi` lowering.
+// CHECK-LABEL: func @unconstrained_fpowi
+func.func @unconstrained_fpowi(%a : f64, %b : i32) {
+  // CHECK-NEXT: llvm.intr.powi(%{{.*}}, %{{.*}}) : (f64, i32) -> f64
+  // CHECK-NOT: constrained
+  %0 = math.fpowi %a, %b : f64, i32
+  return
+}
+
+// -----
+
 // The constrained lowering also applies element-wise to vector operands.
 // CHECK-LABEL: func @experimental_constrained_vector
 func.func @experimental_constrained_vector(%a : vector<4xf32>) {
diff --git a/mlir/test/Dialect/Math/ops.mlir b/mlir/test/Dialect/Math/ops.mlir
index 30912a7cf597a..fd60dd80086cb 100644
--- a/mlir/test/Dialect/Math/ops.mlir
+++ b/mlir/test/Dialect/Math/ops.mlir
@@ -318,8 +318,8 @@ func.func @fastmath(%f: f32, %i: i32, %v: vector<4xf32>, %t: tensor<4x4x?xf32>)
 }
 
 // CHECK-LABEL: func @fenv(
-// CHECK-SAME:      %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>)
-func.func @fenv(%f: f32, %v: vector<4xf32>) {
+// CHECK-SAME:      %[[F:.*]]: f32, %[[V:.*]]: vector<4xf32>, %[[I:.*]]: i32)
+func.func @fenv(%f: f32, %v: vector<4xf32>, %i: i32) {
   // CHECK: math.sqrt %[[F]] fenv<> : f32
   %0 = math.sqrt %f fenv<> : f32
   // CHECK: math.exp %[[F]] fenv<dynamic_rounding_mode = to_nearest_even, strict_except = true> : f32
@@ -332,6 +332,8 @@ func.func @fenv(%f: f32, %v: vector<4xf32>) {
   %4 = math.powf %f, %f fenv<dynamic_rounding_mode = toward_zero> : f32
   // CHECK: math.atan2 %[[F]], %[[F]] fenv<> : f32
   %5 = math.atan2 %f, %f fenv<> : f32
+  // CHECK: math.fpowi %[[F]], %[[I]] fenv<dynamic_rounding_mode = toward_zero> : f32, i32
+  %6 = math.fpowi %f, %i fenv<dynamic_rounding_mode = toward_zero> : f32, i32
   return
 }
 

>From 2c9169da738b3e760cfe2f18791b2db69a527d37 Mon Sep 17 00:00:00 2001
From: Andy Kaylor <akaylor at nvidia.com>
Date: Mon, 13 Jul 2026 12:45:07 -0700
Subject: [PATCH 7/7] Fix formatting

---
 .../ArithCommon/AttrToLLVMConverter.h         |  7 ++--
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    | 17 +++++-----
 mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp | 33 ++++++++++++-------
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 24 +++++++-------
 4 files changed, 45 insertions(+), 36 deletions(-)

diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index 007c62481372c..7f4bb80cc4f07 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -184,10 +184,9 @@ class AttrConverterConstrainedFPToLLVM {
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
-      convertedAttr.set(
-          TargetOp::getRoundingModeAttrName(),
-          LLVM::RoundingModeAttr::get(
-              ctx, convertArithRoundingModeToLLVM(roundingMode)));
+      convertedAttr.set(TargetOp::getRoundingModeAttrName(),
+                        LLVM::RoundingModeAttr::get(
+                            ctx, convertArithRoundingModeToLLVM(roundingMode)));
     }
     // Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
     // support fastmath flags. Remove the arith fastmath attribute if present.
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index fed991c7074d2..39672cf447816 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -373,11 +373,10 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
   }
 
 private:
-  static Value emitConversion(ConversionPatternRewriter &rewriter, Location loc,
-                              Value input, Type targetType,
-                              LLVM::RoundingModeAttr roundingModeAttr,
-                              LLVM::FPExceptionBehaviorAttr
-                                  exceptionBehaviorAttr) {
+  static Value
+  emitConversion(ConversionPatternRewriter &rewriter, Location loc, Value input,
+                 Type targetType, LLVM::RoundingModeAttr roundingModeAttr,
+                 LLVM::FPExceptionBehaviorAttr exceptionBehaviorAttr) {
     Type f32Scalar = Float32Type::get(rewriter.getContext());
     Type f32Ty = f32Scalar;
     if (auto vecTy = dyn_cast<VectorType>(targetType))
@@ -386,10 +385,10 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
     // A constrained floating-point environment was requested: emit the
     // constrained intrinsics carrying the rounding mode and exception behavior.
     if (exceptionBehaviorAttr) {
-      Value ext = LLVM::ConstrainedFPExtIntr::create(rewriter, loc, f32Ty, input,
-                                                     exceptionBehaviorAttr);
-      return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType, ext,
-                                                  roundingModeAttr,
+      Value ext = LLVM::ConstrainedFPExtIntr::create(
+          rewriter, loc, f32Ty, input, exceptionBehaviorAttr);
+      return LLVM::ConstrainedFPTruncIntr::create(rewriter, loc, targetType,
+                                                  ext, roundingModeAttr,
                                                   exceptionBehaviorAttr);
     }
 
diff --git a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
index 84b716f578568..73528707bde49 100644
--- a/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
+++ b/mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp
@@ -80,7 +80,8 @@ struct ConstrainedVectorConvertToLLVMPattern
 template <typename SourceOp, typename TargetOp>
 using ConvertUnconstrainedMathToLLVMPattern =
     ConstrainedVectorConvertToLLVMPattern<SourceOp, TargetOp,
-                                          /*IsConstrained=*/false, ConvertFastMath,
+                                          /*IsConstrained=*/false,
+                                          ConvertFastMath,
                                           /*FailOnUnsupportedFP=*/true>;
 
 // Convenience alias for the pattern that lowers a math op carrying a
@@ -98,7 +99,8 @@ using AbsFOpLowering =
 using CeilOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::CeilOp, LLVM::FCeilOp>;
 using ConstrainedCeilOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::CeilOp, LLVM::ConstrainedCeilIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::CeilOp,
+                                        LLVM::ConstrainedCeilIntr>;
 using CopySignOpLowering =
     ConvertFMFMathToLLVMPattern<math::CopySignOp, LLVM::CopySignOp>;
 using CosOpLowering =
@@ -108,11 +110,13 @@ using ConstrainedCosOpLowering =
 using CoshOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::CoshOp, LLVM::CoshOp>;
 using ConstrainedCoshOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::CoshOp, LLVM::ConstrainedCoshIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::CoshOp,
+                                        LLVM::ConstrainedCoshIntr>;
 using AcosOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::AcosOp, LLVM::ACosOp>;
 using ConstrainedAcosOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::AcosOp, LLVM::ConstrainedACosIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::AcosOp,
+                                        LLVM::ConstrainedACosIntr>;
 using CtPopFOpLowering =
     VectorConvertToLLVMPattern<math::CtPopOp, LLVM::CtPopOp,
                                AttrConvertPassThrough,
@@ -120,7 +124,8 @@ using CtPopFOpLowering =
 using Exp2OpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::Exp2Op, LLVM::Exp2Op>;
 using ConstrainedExp2OpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::Exp2Op, LLVM::ConstrainedExp2Intr>;
+    ConvertConstrainedMathToLLVMPattern<math::Exp2Op,
+                                        LLVM::ConstrainedExp2Intr>;
 using ExpOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::ExpOp, LLVM::ExpOp>;
 using ConstrainedExpOpLowering =
@@ -146,7 +151,8 @@ using ConstrainedLog10OpLowering =
 using Log2OpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::Log2Op, LLVM::Log2Op>;
 using ConstrainedLog2OpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::Log2Op, LLVM::ConstrainedLog2Intr>;
+    ConvertConstrainedMathToLLVMPattern<math::Log2Op,
+                                        LLVM::ConstrainedLog2Intr>;
 using LogOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::LogOp, LLVM::LogOp>;
 using ConstrainedLogOpLowering =
@@ -174,15 +180,18 @@ using ConstrainedSinOpLowering =
 using SinhOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::SinhOp, LLVM::SinhOp>;
 using ConstrainedSinhOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::SinhOp, LLVM::ConstrainedSinhIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::SinhOp,
+                                        LLVM::ConstrainedSinhIntr>;
 using ASinOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::AsinOp, LLVM::ASinOp>;
 using ConstrainedASinOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::AsinOp, LLVM::ConstrainedASinIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::AsinOp,
+                                        LLVM::ConstrainedASinIntr>;
 using SqrtOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::SqrtOp, LLVM::SqrtOp>;
 using ConstrainedSqrtOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::SqrtOp, LLVM::ConstrainedSqrtIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::SqrtOp,
+                                        LLVM::ConstrainedSqrtIntr>;
 using FTruncOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::TruncOp, LLVM::FTruncOp>;
 using ConstrainedFTruncOpLowering =
@@ -195,11 +204,13 @@ using ConstrainedTanOpLowering =
 using TanhOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::TanhOp, LLVM::TanhOp>;
 using ConstrainedTanhOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::TanhOp, LLVM::ConstrainedTanhIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::TanhOp,
+                                        LLVM::ConstrainedTanhIntr>;
 using ATanOpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::AtanOp, LLVM::ATanOp>;
 using ConstrainedATanOpLowering =
-    ConvertConstrainedMathToLLVMPattern<math::AtanOp, LLVM::ConstrainedATanIntr>;
+    ConvertConstrainedMathToLLVMPattern<math::AtanOp,
+                                        LLVM::ConstrainedATanIntr>;
 using ATan2OpLowering =
     ConvertUnconstrainedMathToLLVMPattern<math::Atan2Op, LLVM::ATan2Op>;
 using ConstrainedATan2OpLowering =
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index b104a3b9711e9..df10228ac864b 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -649,9 +649,9 @@ struct F8E8M0TruncFOpConverter : public OpRewritePattern<arith::TruncFOp> {
     if (operandETy.getIntOrFloatBitWidth() < 32) {
       operand = arith::ExtFOp::create(b, f32Ty, operand, op.getFastmathAttr());
     } else if (operandETy.getIntOrFloatBitWidth() > 32) {
-      operand = arith::TruncFOp::create(b, f32Ty, operand,
-                                        op.getRoundingmodeAttr(),
-                                        op.getFastmathAttr(), op.getFenvAttr());
+      operand =
+          arith::TruncFOp::create(b, f32Ty, operand, op.getRoundingmodeAttr(),
+                                  op.getFastmathAttr(), op.getFenvAttr());
     }
     Value f32Bits = arith::BitcastOp::create(b, i32Ty, operand);
     Value cF32MantissaWidth = createConst(op->getLoc(), i32Ty, 23, rewriter);
@@ -718,9 +718,9 @@ struct ScalingTruncFOpConverter
     if (scaleETy.getIntOrFloatBitWidth() >= 16) {
       scaleETy = b.getF8E8M0Type();
       scaleTy = cloneToShapedType(scaleTy, scaleETy);
-      scaleOperand = arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
-                                             op.getFastmathAttr(),
-                                             op.getFenvAttr());
+      scaleOperand =
+          arith::TruncFOp::create(b, scaleTy, scaleOperand, nullptr,
+                                  op.getFastmathAttr(), op.getFenvAttr());
     }
     if (!llvm::isa<Float8E8M0FNUType>(scaleETy)) {
       return rewriter.notifyMatchFailure(
@@ -733,12 +733,12 @@ struct ScalingTruncFOpConverter
     // inputTy that is 2^scale and will also propagate NaNs
     scaleOperand =
         arith::ExtFOp::create(b, inputTy, scaleOperand, op.getFastmathAttr());
-    Value result = arith::DivFOp::create(b, inputOperand, scaleOperand,
-                                         op.getFastmathAttr(), nullptr,
-                                         op.getFenvAttr());
-    Value resultCast = arith::TruncFOp::create(
-        b, resultTy, result, op.getRoundingmodeAttr(), op.getFastmathAttr(),
-        op.getFenvAttr());
+    Value result =
+        arith::DivFOp::create(b, inputOperand, scaleOperand,
+                              op.getFastmathAttr(), nullptr, op.getFenvAttr());
+    Value resultCast =
+        arith::TruncFOp::create(b, resultTy, result, op.getRoundingmodeAttr(),
+                                op.getFastmathAttr(), op.getFenvAttr());
     rewriter.replaceOp(op, resultCast);
     return success();
   }


From llvmlistbot at llvm.org  Mon Jul 13 12:55:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 12:55:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a5542a4.c56fb102.7c9cd.c36b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Andy Kaylor (andykaylor)

<details>
<summary>Changes</summary>

This is a proposed implementation of strict floating-point handling for Arith and Math dialects. If this direction is accepted, I will split the individual commits in this PR out for proper review. I have combined them here so that the entire design can be considered as a whole.

This is a follow-up to my prior [RFC on Discourse](https://discourse.llvm.org/t/rfc-mlir-new-representation-of-floating-point-constraints/91123) proposing a new MLIR representation of floating-point constraints. Based on the feedback I received in https://github.com/llvm/llvm-project/pull/205158, I am moving the floating-point environment attribute out to the CIR and Arith dialects and lowering these to existing operations in the LLVM dialect that continue to map directly to the current form of LLVM IR.

This PR shows the Arith and Math implementation. I have posted a separate [PR for the CIR dialect changes](https://github.com/llvm/llvm-project/pull/209275).

This is intended to replace the existing rounding-mode handling in the Arith dialect. I have left the old handling in place here but added comments that it should be considered deprecated. I am open to suggestions on how that should be handled.

The implementation here has one significant difference from the CIR implementation. In the CIR implementation, I am lowering operations with the Fenv attribute to general intrinsic calls to the constrained floating-point intrinsics. Here I have added specialized constrained operations to the LLVM dialect where they were missing and lower to those. I did this to be consistent with the existing Arith rounding-mode handling, though I think it may be more consistent with the LLVM dialect design philosophy to remove the specialized constrained operations there entirely and to lower to the generalized intrinsic calls as I have done in CIR.

Assisted-by: Cursor / claude-opus-4.8

---

Patch is 107.11 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209287.diff


19 Files Affected:

- (modified) mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h (+51-9) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithBase.td (+147-1) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOps.td (+81-8) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td (+83) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td (+78) 
- (modified) mlir/include/mlir/Dialect/Math/IR/MathOps.td (+245-25) 
- (modified) mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp (+17) 
- (modified) mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp (+67-26) 
- (modified) mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp (+201-34) 
- (modified) mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td (+26-14) 
- (modified) mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp (+14-10) 
- (modified) mlir/lib/Dialect/Math/IR/MathOps.cpp (+5) 
- (modified) mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir (+82) 
- (modified) mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir (+128) 
- (modified) mlir/test/Dialect/Arith/expand-ops.mlir (+13) 
- (modified) mlir/test/Dialect/Arith/invalid.mlir (+8) 
- (modified) mlir/test/Dialect/Arith/ops.mlir (+23) 
- (modified) mlir/test/Dialect/Math/ops.mlir (+20) 
- (modified) mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index feb74c86e349f..7f4bb80cc4f07 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -45,6 +45,18 @@ convertArithRoundingModeAttrToLLVM(arith::RoundingModeAttr roundingModeAttr);
 LLVM::FPExceptionBehaviorAttr
 getLLVMDefaultFPExceptionBehavior(MLIRContext &context);
 
+/// Maps an arith floating-point exception mode and strict-exception flag, as
+/// described by an `#arith.fenv` attribute, to the corresponding LLVM
+/// constrained-intrinsic exception behavior:
+///
+///   masked            + strict_except=false -> fpexcept.ignore
+///   masked            + strict_except=true  -> fpexcept.strict
+///   unmasked|unknown  + strict_except=false -> fpexcept.maytrap
+///   unmasked|unknown  + strict_except=true  -> fpexcept.strict
+LLVM::FPExceptionBehavior
+convertArithFPExceptionBehaviorToLLVM(arith::FPExceptionMode exceptionMode,
+                                      bool strictExcept);
+
 // Attribute converter that populates a NamedAttrList by removing the fastmath
 // attribute from the source operation attributes, and replacing it with an
 // equivalent LLVM fastmath attribute.
@@ -145,25 +157,55 @@ class AttrConverterConstrainedFPToLLVM {
   AttrConverterConstrainedFPToLLVM(SourceOp srcOp) {
     // Copy the source attributes.
     convertedAttr = NamedAttrList{srcOp->getAttrs()};
+    MLIRContext *ctx = srcOp->getContext();
+
+    // The floating-point environment is described by the `#arith.fenv`
+    // attribute and, on operations that predate it, may also be described by
+    // the deprecated `roundingmode` attribute. Collect both (the verifier
+    // guarantees they are never set at the same time) and remove them from the
+    // attributes carried over to the target op.
+    arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
+    convertedAttr.erase(srcOp.getFenvAttrName());
+
+    // Determine the rounding mode. The `fenv` attribute takes precedence; when
+    // neither source carries one, fall back to the default dynamic rounding
+    // mode.
+    [[maybe_unused]] arith::RoundingMode roundingMode =
+        arith::FenvAttr::getDefaultDynamicRoundingMode();
+    if constexpr (SourceOp::template hasTrait<
+                      arith::ArithRoundingModeInterface::Trait>()) {
+      auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+          convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+      if (!fenvAttr && roundingModeAttr)
+        roundingMode = roundingModeAttr.getValue();
+    }
+    if (fenvAttr)
+      roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
-      // Get the name of the rounding mode attribute.
-      StringRef arithAttrName = srcOp.getRoundingModeAttrName();
-      // Remove the source attribute.
-      auto arithAttr =
-          cast<arith::RoundingModeAttr>(convertedAttr.erase(arithAttrName));
-      // Set the target attribute.
       convertedAttr.set(TargetOp::getRoundingModeAttrName(),
-                        convertArithRoundingModeAttrToLLVM(arithAttr));
+                        LLVM::RoundingModeAttr::get(
+                            ctx, convertArithRoundingModeToLLVM(roundingMode)));
     }
     // Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
     // support fastmath flags. Remove the arith fastmath attribute if present.
     if constexpr (SourceOp::template hasTrait<
                       arith::ArithFastMathInterface::Trait>())
       convertedAttr.erase(srcOp.getFastMathAttrName());
-    convertedAttr.set(TargetOp::getFPExceptionBehaviorAttrName(),
-                      getLLVMDefaultFPExceptionBehavior(*srcOp->getContext()));
+
+    // Determine the exception behavior from the `fenv` attribute, defaulting to
+    // `ignore` when no environment is specified (e.g. only a `roundingmode`
+    // attribute is present).
+    LLVM::FPExceptionBehavior exceptionBehavior =
+        LLVM::FPExceptionBehavior::Ignore;
+    if (fenvAttr)
+      exceptionBehavior = convertArithFPExceptionBehaviorToLLVM(
+          fenvAttr.getExceptionModeOrDefault(),
+          fenvAttr.getStrictExceptOrDefault());
+    convertedAttr.set(
+        TargetOp::getFPExceptionBehaviorAttrName(),
+        LLVM::FPExceptionBehaviorAttr::get(ctx, exceptionBehavior));
   }
 
   ArrayRef<NamedAttribute> getAttrs() const { return convertedAttr.getAttrs(); }
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
index 985ae01008002..4132a4cfb3c55 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
@@ -186,12 +186,158 @@ def Arith_RTowardZero                // Round toward 0
     : I32EnumAttrCase<"toward_zero", 3>;
 def Arith_RToNearestTiesAwayFromZero // Round to nearest, ties away from zero
     : I32EnumAttrCase<"to_nearest_away", 4>;
+def Arith_RDynamic                   // Rounding mode unknown at compile time
+    : I32EnumAttrCase<"unknown", 7>;  // Mirrors `llvm::RoundingMode::Dynamic`.
 
 def Arith_RoundingModeAttr : I32EnumAttr<
     "RoundingMode", "Floating point rounding mode",
     [Arith_RToNearestTiesToEven, Arith_RDownward, Arith_RUpward,
-     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero]> {
+     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero, Arith_RDynamic]> {
   let cppNamespace = "::mlir::arith";
 }
 
+//===----------------------------------------------------------------------===//
+// Floating-point environment enums (used by the `#arith.fenv` attribute)
+//===----------------------------------------------------------------------===//
+
+def FPExceptionMode : I32EnumAttr<
+    "FPExceptionMode", "floating-point exception mode", [
+  I32EnumAttrCase<"Unknown", 0, "unknown">,
+  I32EnumAttrCase<"Masked", 1, "masked">,
+  I32EnumAttrCase<"Unmasked", 2, "unmasked">
+]> {
+  let description = [{
+    The known floating-point exception mode at the point the instruction is
+    executed. If the actual exception mode differs from this value, the
+    behavior is undefined.
+  }];
+  let cppNamespace = "::mlir::arith";
+  let genSpecializedAttr = 0;
+}
+
+//===----------------------------------------------------------------------===//
+// FenvAttr
+//===----------------------------------------------------------------------===//
+
+def Arith_FenvAttr : AttrDef<Arith_Dialect, "Fenv"> {
+  let mnemonic = "fenv";
+  let summary = "Describes floating-point environment constraints";
+  let description = [{
+    The `#arith.fenv` attribute describes constraints on the floating-point
+    handling of a floating-point operation. It can be attached to
+    floating-point operations to capture rounding and exception behavior. All
+    of its parameters are optional.
+
+    - `dynamic_rounding_mode`: the known dynamic rounding mode at the point the
+      instruction is executed. If the actual dynamic rounding mode when the
+      instruction is executed does not match the assumed value, the behavior is
+      undefined.
+    - `except_mode`: the known exception mode at the point the instruction is
+      executed. If the actual state of exception masking when the instruction is
+      executed does not match this value, the behavior is undefined.
+    - `strict_except`: if `false`, any case that would produce an FP exception
+      produces it non-deterministically instead (i.e. it may or may not occur).
+      This means the FP status is written non-deterministically, and, if
+      exceptions are unmasked, the instruction traps non-deterministically.
+
+    When this attribute is present on an operation, the operation is lowered to
+    the corresponding `llvm.experimental.constrained.*` intrinsic during
+    translation to LLVM IR.
+
+    The default setting for each parameter (`unknown` for
+    `dynamic_rounding_mode`, `masked` for exception_mode, and `false` for
+    `strict_except`) is canonically represented as an unset parameter.
+    Constructing an attribute whose parameters are all set to their defaults
+    therefore yields the same canonical, empty attribute as `#arith.fenv<>`.
+
+    Carrying the (possibly empty) attribute is *not* the same as having no
+    attribute at all. An operation that carries the attribute is lowered to the
+    corresponding constrained intrinsic operation in LLVM, even when every
+    parameter is at its default value. An operation with no `#arith.fenv`
+    attribute is lowered to the regular, unconstrained operation.
+
+    Examples:
+    ```mlir
+    fenv<dynamic_rounding_mode = to_nearest_even>
+    fenv<except_mode = unmasked>
+    fenv<strict_except = true>
+    fenv<dynamic_rounding_mode = upward, except_mode = unknown, strict_except = true>
+    ```
+  }];
+
+  let parameters = (ins
+    OptionalParameter<
+        "std::optional<::mlir::arith::RoundingMode>">:$dynamic_rounding_mode,
+    OptionalParameter<"std::optional<::mlir::arith::FPExceptionMode>">:$except_mode,
+    OptionalParameter<"mlir::BoolAttr">:$strict_except);
+
+  let assemblyFormat = [{
+    `<` struct($dynamic_rounding_mode, $except_mode, $strict_except) `>`
+  }];
+
+  // Replace the default builder so that constructing the attribute normalizes
+  // any parameter that is set to its default value to "unset". This keeps a
+  // single canonical representation for the default floating-point environment.
+  // Because `skipDefaultBuilders` suppresses the auto-synthesized const builder
+  // call, provide it explicitly. The `DefaultValuedAttr` form below uses a null
+  // attribute as its default, so this simply forwards its argument (a null
+  // attribute when the operation does not carry a `#arith.fenv` attribute).
+  let skipDefaultBuilders = 1;
+  let constBuilderCall = "$0";
+  let builders = [
+    AttrBuilder<(ins
+        CArg<"std::optional<::mlir::arith::RoundingMode>",
+             "std::nullopt">:$dynamicRoundingMode,
+        CArg<"std::optional<::mlir::arith::FPExceptionMode>",
+             "std::nullopt">:$exceptMode,
+        CArg<"mlir::BoolAttr", "{}">:$strictExcept), [{
+      if (dynamicRoundingMode == ::mlir::arith::RoundingMode::unknown)
+        dynamicRoundingMode = std::nullopt;
+      if (exceptMode == ::mlir::arith::FPExceptionMode::Masked)
+        exceptMode = std::nullopt;
+      if (strictExcept && !strictExcept.getValue())
+        strictExcept = {};
+      return Base::get($_ctxt, dynamicRoundingMode, exceptMode, strictExcept);
+    }]>
+  ];
+
+  let extraClassDeclaration = [{
+    /// The default dynamic rounding mode used when `dynamic_rounding_mode` is
+    /// unset.
+    static constexpr ::mlir::arith::RoundingMode
+    getDefaultDynamicRoundingMode() {
+      return ::mlir::arith::RoundingMode::unknown;
+    }
+
+    /// The default exception mode used when `except_mode` is unset.
+    static constexpr ::mlir::arith::FPExceptionMode getDefaultExceptionMode() {
+      return ::mlir::arith::FPExceptionMode::Masked;
+    }
+
+    /// Returns the effective dynamic rounding mode, substituting the default
+    /// when unset.
+    ::mlir::arith::RoundingMode getDynamicRoundingModeOrDefault() const {
+      return getDynamicRoundingMode().value_or(getDefaultDynamicRoundingMode());
+    }
+    /// Returns the effective exception mode, substituting the default when
+    /// unset.
+    ::mlir::arith::FPExceptionMode getExceptionModeOrDefault() const {
+      return getExceptMode().value_or(getDefaultExceptionMode());
+    }
+    /// Returns the effective `strict_except` flag.
+    bool getStrictExceptOrDefault() const {
+      ::mlir::BoolAttr attr = getStrictExcept();
+      return attr ? attr.getValue() : false;
+    }
+
+    /// Returns true if every parameter is at its default value, i.e. the
+    /// attribute describes the default floating-point environment and imposes
+    /// no constraints. Because the builder normalizes defaults to "unset", this
+    /// is equivalent to all parameters being unset.
+    bool isDefault() const {
+      return !getDynamicRoundingMode() && !getExceptMode() && !getStrictExcept();
+    }
+  }];
+}
+
 #endif // ARITH_BASE
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..c383e8d8cd1b9 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -93,6 +93,11 @@ class Arith_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
 
 // Base class for floating point binary operations with an optional rounding
 // mode.
+//
+// Deprecated: use `Arith_FloatBinaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
 class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                                           list<Trait> traits = []> :
     Arith_FloatBinaryOp<mnemonic,
@@ -128,6 +133,50 @@ class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                           attr-dict `:` type($result) }];
 }
 
+// Base class for floating point binary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Arith_FloatBinaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Arith_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Arith_FloatBinaryOpWithRoundingMode<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+      DefaultValuedAttr<
+        Arith_FastMathAttr, "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With two trailing optional attributes (`roundingmode` and `fenv`), the
+  // generated builders no longer provide C++ default arguments for them, so the
+  // convenience builders must pass both explicitly.
+  let builders = [
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Type":$type, "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, type, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $lhs `,` $rhs ($roundingmode^)?
+                          (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Checks that tensor input and outputs have identical shapes. This is stricker
 // than the verification done in `SameOperandsAndResultShape` that allows for
 // tensor dimensions to be 'compatible' (e.g., dynamic dimensions being
@@ -1089,7 +1138,7 @@ def Arith_FlushDenormalsOp : Arith_FloatUnaryOp<"flush_denormals"> {
 // AddFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
+def Arith_AddFOp : Arith_FloatBinaryOpWithFenv<"addf", [Commutative]> {
   let summary = "floating point addition operation";
   let description = [{
     The `addf` operation takes two operands and returns one result, each of
@@ -1120,7 +1169,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
 // SubFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_SubFOp : Arith_FloatBinaryOpWithRoundingMode<"subf"> {
+def Arith_SubFOp : Arith_FloatBinaryOpWithFenv<"subf"> {
   let summary = "floating point subtraction operation";
   let description = [{
     The `subf` operation takes two operands and returns one result, each of
@@ -1276,7 +1325,7 @@ def Arith_MinUIOp : Arith_TotalIntBinaryOp<"minui",
 // MulFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
+def Arith_MulFOp : Arith_FloatBinaryOpWithFenv<"mulf", [Commutative]> {
   let summary = "floating point multiplication operation";
   let description = [{
     The `mulf` operation takes two operands and returns one result, each of
@@ -1308,7 +1357,7 @@ def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
 // DivFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_DivFOp : Arith_FloatBinaryOpWithRoundingMode<"divf"> {
+def Arith_DivFOp : Arith_FloatBinaryOpWithFenv<"divf"> {
   let summary = "floating point division operation";
   let description = [{
     The `divf` operation takes two operands and returns one result, each of
@@ -1566,10 +1615,17 @@ def Arith_TruncFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast from floating-point to narrower floating-point";
   let description = [{
@@ -1589,6 +1645,7 @@ def Arith_TruncFOp :
   let hasVerifier = 1;
   let assemblyFormat = [{ $in ($roundingmode^)?
                           (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($in) `to` type($out) }];
 }
 
@@ -1601,10 +1658,17 @@ def Arith_ConvertFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast between floating-point types of the same bitwidth";
   let description = [{
@@ -1621,6 ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Mon Jul 13 12:55:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 12:55:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a5542a5.481afadf.196d57.a196@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-math

@llvm/pr-subscribers-mlir-arith

Author: Andy Kaylor (andykaylor)

<details>
<summary>Changes</summary>

This is a proposed implementation of strict floating-point handling for Arith and Math dialects. If this direction is accepted, I will split the individual commits in this PR out for proper review. I have combined them here so that the entire design can be considered as a whole.

This is a follow-up to my prior [RFC on Discourse](https://discourse.llvm.org/t/rfc-mlir-new-representation-of-floating-point-constraints/91123) proposing a new MLIR representation of floating-point constraints. Based on the feedback I received in https://github.com/llvm/llvm-project/pull/205158, I am moving the floating-point environment attribute out to the CIR and Arith dialects and lowering these to existing operations in the LLVM dialect that continue to map directly to the current form of LLVM IR.

This PR shows the Arith and Math implementation. I have posted a separate [PR for the CIR dialect changes](https://github.com/llvm/llvm-project/pull/209275).

This is intended to replace the existing rounding-mode handling in the Arith dialect. I have left the old handling in place here but added comments that it should be considered deprecated. I am open to suggestions on how that should be handled.

The implementation here has one significant difference from the CIR implementation. In the CIR implementation, I am lowering operations with the Fenv attribute to general intrinsic calls to the constrained floating-point intrinsics. Here I have added specialized constrained operations to the LLVM dialect where they were missing and lower to those. I did this to be consistent with the existing Arith rounding-mode handling, though I think it may be more consistent with the LLVM dialect design philosophy to remove the specialized constrained operations there entirely and to lower to the generalized intrinsic calls as I have done in CIR.

Assisted-by: Cursor / claude-opus-4.8

---

Patch is 107.11 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209287.diff


19 Files Affected:

- (modified) mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h (+51-9) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithBase.td (+147-1) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOps.td (+81-8) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td (+83) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td (+78) 
- (modified) mlir/include/mlir/Dialect/Math/IR/MathOps.td (+245-25) 
- (modified) mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp (+17) 
- (modified) mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp (+67-26) 
- (modified) mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp (+201-34) 
- (modified) mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td (+26-14) 
- (modified) mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp (+14-10) 
- (modified) mlir/lib/Dialect/Math/IR/MathOps.cpp (+5) 
- (modified) mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir (+82) 
- (modified) mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir (+128) 
- (modified) mlir/test/Dialect/Arith/expand-ops.mlir (+13) 
- (modified) mlir/test/Dialect/Arith/invalid.mlir (+8) 
- (modified) mlir/test/Dialect/Arith/ops.mlir (+23) 
- (modified) mlir/test/Dialect/Math/ops.mlir (+20) 
- (modified) mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index feb74c86e349f..7f4bb80cc4f07 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -45,6 +45,18 @@ convertArithRoundingModeAttrToLLVM(arith::RoundingModeAttr roundingModeAttr);
 LLVM::FPExceptionBehaviorAttr
 getLLVMDefaultFPExceptionBehavior(MLIRContext &context);
 
+/// Maps an arith floating-point exception mode and strict-exception flag, as
+/// described by an `#arith.fenv` attribute, to the corresponding LLVM
+/// constrained-intrinsic exception behavior:
+///
+///   masked            + strict_except=false -> fpexcept.ignore
+///   masked            + strict_except=true  -> fpexcept.strict
+///   unmasked|unknown  + strict_except=false -> fpexcept.maytrap
+///   unmasked|unknown  + strict_except=true  -> fpexcept.strict
+LLVM::FPExceptionBehavior
+convertArithFPExceptionBehaviorToLLVM(arith::FPExceptionMode exceptionMode,
+                                      bool strictExcept);
+
 // Attribute converter that populates a NamedAttrList by removing the fastmath
 // attribute from the source operation attributes, and replacing it with an
 // equivalent LLVM fastmath attribute.
@@ -145,25 +157,55 @@ class AttrConverterConstrainedFPToLLVM {
   AttrConverterConstrainedFPToLLVM(SourceOp srcOp) {
     // Copy the source attributes.
     convertedAttr = NamedAttrList{srcOp->getAttrs()};
+    MLIRContext *ctx = srcOp->getContext();
+
+    // The floating-point environment is described by the `#arith.fenv`
+    // attribute and, on operations that predate it, may also be described by
+    // the deprecated `roundingmode` attribute. Collect both (the verifier
+    // guarantees they are never set at the same time) and remove them from the
+    // attributes carried over to the target op.
+    arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
+    convertedAttr.erase(srcOp.getFenvAttrName());
+
+    // Determine the rounding mode. The `fenv` attribute takes precedence; when
+    // neither source carries one, fall back to the default dynamic rounding
+    // mode.
+    [[maybe_unused]] arith::RoundingMode roundingMode =
+        arith::FenvAttr::getDefaultDynamicRoundingMode();
+    if constexpr (SourceOp::template hasTrait<
+                      arith::ArithRoundingModeInterface::Trait>()) {
+      auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+          convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+      if (!fenvAttr && roundingModeAttr)
+        roundingMode = roundingModeAttr.getValue();
+    }
+    if (fenvAttr)
+      roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
-      // Get the name of the rounding mode attribute.
-      StringRef arithAttrName = srcOp.getRoundingModeAttrName();
-      // Remove the source attribute.
-      auto arithAttr =
-          cast<arith::RoundingModeAttr>(convertedAttr.erase(arithAttrName));
-      // Set the target attribute.
       convertedAttr.set(TargetOp::getRoundingModeAttrName(),
-                        convertArithRoundingModeAttrToLLVM(arithAttr));
+                        LLVM::RoundingModeAttr::get(
+                            ctx, convertArithRoundingModeToLLVM(roundingMode)));
     }
     // Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
     // support fastmath flags. Remove the arith fastmath attribute if present.
     if constexpr (SourceOp::template hasTrait<
                       arith::ArithFastMathInterface::Trait>())
       convertedAttr.erase(srcOp.getFastMathAttrName());
-    convertedAttr.set(TargetOp::getFPExceptionBehaviorAttrName(),
-                      getLLVMDefaultFPExceptionBehavior(*srcOp->getContext()));
+
+    // Determine the exception behavior from the `fenv` attribute, defaulting to
+    // `ignore` when no environment is specified (e.g. only a `roundingmode`
+    // attribute is present).
+    LLVM::FPExceptionBehavior exceptionBehavior =
+        LLVM::FPExceptionBehavior::Ignore;
+    if (fenvAttr)
+      exceptionBehavior = convertArithFPExceptionBehaviorToLLVM(
+          fenvAttr.getExceptionModeOrDefault(),
+          fenvAttr.getStrictExceptOrDefault());
+    convertedAttr.set(
+        TargetOp::getFPExceptionBehaviorAttrName(),
+        LLVM::FPExceptionBehaviorAttr::get(ctx, exceptionBehavior));
   }
 
   ArrayRef<NamedAttribute> getAttrs() const { return convertedAttr.getAttrs(); }
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
index 985ae01008002..4132a4cfb3c55 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
@@ -186,12 +186,158 @@ def Arith_RTowardZero                // Round toward 0
     : I32EnumAttrCase<"toward_zero", 3>;
 def Arith_RToNearestTiesAwayFromZero // Round to nearest, ties away from zero
     : I32EnumAttrCase<"to_nearest_away", 4>;
+def Arith_RDynamic                   // Rounding mode unknown at compile time
+    : I32EnumAttrCase<"unknown", 7>;  // Mirrors `llvm::RoundingMode::Dynamic`.
 
 def Arith_RoundingModeAttr : I32EnumAttr<
     "RoundingMode", "Floating point rounding mode",
     [Arith_RToNearestTiesToEven, Arith_RDownward, Arith_RUpward,
-     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero]> {
+     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero, Arith_RDynamic]> {
   let cppNamespace = "::mlir::arith";
 }
 
+//===----------------------------------------------------------------------===//
+// Floating-point environment enums (used by the `#arith.fenv` attribute)
+//===----------------------------------------------------------------------===//
+
+def FPExceptionMode : I32EnumAttr<
+    "FPExceptionMode", "floating-point exception mode", [
+  I32EnumAttrCase<"Unknown", 0, "unknown">,
+  I32EnumAttrCase<"Masked", 1, "masked">,
+  I32EnumAttrCase<"Unmasked", 2, "unmasked">
+]> {
+  let description = [{
+    The known floating-point exception mode at the point the instruction is
+    executed. If the actual exception mode differs from this value, the
+    behavior is undefined.
+  }];
+  let cppNamespace = "::mlir::arith";
+  let genSpecializedAttr = 0;
+}
+
+//===----------------------------------------------------------------------===//
+// FenvAttr
+//===----------------------------------------------------------------------===//
+
+def Arith_FenvAttr : AttrDef<Arith_Dialect, "Fenv"> {
+  let mnemonic = "fenv";
+  let summary = "Describes floating-point environment constraints";
+  let description = [{
+    The `#arith.fenv` attribute describes constraints on the floating-point
+    handling of a floating-point operation. It can be attached to
+    floating-point operations to capture rounding and exception behavior. All
+    of its parameters are optional.
+
+    - `dynamic_rounding_mode`: the known dynamic rounding mode at the point the
+      instruction is executed. If the actual dynamic rounding mode when the
+      instruction is executed does not match the assumed value, the behavior is
+      undefined.
+    - `except_mode`: the known exception mode at the point the instruction is
+      executed. If the actual state of exception masking when the instruction is
+      executed does not match this value, the behavior is undefined.
+    - `strict_except`: if `false`, any case that would produce an FP exception
+      produces it non-deterministically instead (i.e. it may or may not occur).
+      This means the FP status is written non-deterministically, and, if
+      exceptions are unmasked, the instruction traps non-deterministically.
+
+    When this attribute is present on an operation, the operation is lowered to
+    the corresponding `llvm.experimental.constrained.*` intrinsic during
+    translation to LLVM IR.
+
+    The default setting for each parameter (`unknown` for
+    `dynamic_rounding_mode`, `masked` for exception_mode, and `false` for
+    `strict_except`) is canonically represented as an unset parameter.
+    Constructing an attribute whose parameters are all set to their defaults
+    therefore yields the same canonical, empty attribute as `#arith.fenv<>`.
+
+    Carrying the (possibly empty) attribute is *not* the same as having no
+    attribute at all. An operation that carries the attribute is lowered to the
+    corresponding constrained intrinsic operation in LLVM, even when every
+    parameter is at its default value. An operation with no `#arith.fenv`
+    attribute is lowered to the regular, unconstrained operation.
+
+    Examples:
+    ```mlir
+    fenv<dynamic_rounding_mode = to_nearest_even>
+    fenv<except_mode = unmasked>
+    fenv<strict_except = true>
+    fenv<dynamic_rounding_mode = upward, except_mode = unknown, strict_except = true>
+    ```
+  }];
+
+  let parameters = (ins
+    OptionalParameter<
+        "std::optional<::mlir::arith::RoundingMode>">:$dynamic_rounding_mode,
+    OptionalParameter<"std::optional<::mlir::arith::FPExceptionMode>">:$except_mode,
+    OptionalParameter<"mlir::BoolAttr">:$strict_except);
+
+  let assemblyFormat = [{
+    `<` struct($dynamic_rounding_mode, $except_mode, $strict_except) `>`
+  }];
+
+  // Replace the default builder so that constructing the attribute normalizes
+  // any parameter that is set to its default value to "unset". This keeps a
+  // single canonical representation for the default floating-point environment.
+  // Because `skipDefaultBuilders` suppresses the auto-synthesized const builder
+  // call, provide it explicitly. The `DefaultValuedAttr` form below uses a null
+  // attribute as its default, so this simply forwards its argument (a null
+  // attribute when the operation does not carry a `#arith.fenv` attribute).
+  let skipDefaultBuilders = 1;
+  let constBuilderCall = "$0";
+  let builders = [
+    AttrBuilder<(ins
+        CArg<"std::optional<::mlir::arith::RoundingMode>",
+             "std::nullopt">:$dynamicRoundingMode,
+        CArg<"std::optional<::mlir::arith::FPExceptionMode>",
+             "std::nullopt">:$exceptMode,
+        CArg<"mlir::BoolAttr", "{}">:$strictExcept), [{
+      if (dynamicRoundingMode == ::mlir::arith::RoundingMode::unknown)
+        dynamicRoundingMode = std::nullopt;
+      if (exceptMode == ::mlir::arith::FPExceptionMode::Masked)
+        exceptMode = std::nullopt;
+      if (strictExcept && !strictExcept.getValue())
+        strictExcept = {};
+      return Base::get($_ctxt, dynamicRoundingMode, exceptMode, strictExcept);
+    }]>
+  ];
+
+  let extraClassDeclaration = [{
+    /// The default dynamic rounding mode used when `dynamic_rounding_mode` is
+    /// unset.
+    static constexpr ::mlir::arith::RoundingMode
+    getDefaultDynamicRoundingMode() {
+      return ::mlir::arith::RoundingMode::unknown;
+    }
+
+    /// The default exception mode used when `except_mode` is unset.
+    static constexpr ::mlir::arith::FPExceptionMode getDefaultExceptionMode() {
+      return ::mlir::arith::FPExceptionMode::Masked;
+    }
+
+    /// Returns the effective dynamic rounding mode, substituting the default
+    /// when unset.
+    ::mlir::arith::RoundingMode getDynamicRoundingModeOrDefault() const {
+      return getDynamicRoundingMode().value_or(getDefaultDynamicRoundingMode());
+    }
+    /// Returns the effective exception mode, substituting the default when
+    /// unset.
+    ::mlir::arith::FPExceptionMode getExceptionModeOrDefault() const {
+      return getExceptMode().value_or(getDefaultExceptionMode());
+    }
+    /// Returns the effective `strict_except` flag.
+    bool getStrictExceptOrDefault() const {
+      ::mlir::BoolAttr attr = getStrictExcept();
+      return attr ? attr.getValue() : false;
+    }
+
+    /// Returns true if every parameter is at its default value, i.e. the
+    /// attribute describes the default floating-point environment and imposes
+    /// no constraints. Because the builder normalizes defaults to "unset", this
+    /// is equivalent to all parameters being unset.
+    bool isDefault() const {
+      return !getDynamicRoundingMode() && !getExceptMode() && !getStrictExcept();
+    }
+  }];
+}
+
 #endif // ARITH_BASE
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..c383e8d8cd1b9 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -93,6 +93,11 @@ class Arith_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
 
 // Base class for floating point binary operations with an optional rounding
 // mode.
+//
+// Deprecated: use `Arith_FloatBinaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
 class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                                           list<Trait> traits = []> :
     Arith_FloatBinaryOp<mnemonic,
@@ -128,6 +133,50 @@ class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                           attr-dict `:` type($result) }];
 }
 
+// Base class for floating point binary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Arith_FloatBinaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Arith_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Arith_FloatBinaryOpWithRoundingMode<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+      DefaultValuedAttr<
+        Arith_FastMathAttr, "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With two trailing optional attributes (`roundingmode` and `fenv`), the
+  // generated builders no longer provide C++ default arguments for them, so the
+  // convenience builders must pass both explicitly.
+  let builders = [
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Type":$type, "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, type, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $lhs `,` $rhs ($roundingmode^)?
+                          (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Checks that tensor input and outputs have identical shapes. This is stricker
 // than the verification done in `SameOperandsAndResultShape` that allows for
 // tensor dimensions to be 'compatible' (e.g., dynamic dimensions being
@@ -1089,7 +1138,7 @@ def Arith_FlushDenormalsOp : Arith_FloatUnaryOp<"flush_denormals"> {
 // AddFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
+def Arith_AddFOp : Arith_FloatBinaryOpWithFenv<"addf", [Commutative]> {
   let summary = "floating point addition operation";
   let description = [{
     The `addf` operation takes two operands and returns one result, each of
@@ -1120,7 +1169,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
 // SubFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_SubFOp : Arith_FloatBinaryOpWithRoundingMode<"subf"> {
+def Arith_SubFOp : Arith_FloatBinaryOpWithFenv<"subf"> {
   let summary = "floating point subtraction operation";
   let description = [{
     The `subf` operation takes two operands and returns one result, each of
@@ -1276,7 +1325,7 @@ def Arith_MinUIOp : Arith_TotalIntBinaryOp<"minui",
 // MulFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
+def Arith_MulFOp : Arith_FloatBinaryOpWithFenv<"mulf", [Commutative]> {
   let summary = "floating point multiplication operation";
   let description = [{
     The `mulf` operation takes two operands and returns one result, each of
@@ -1308,7 +1357,7 @@ def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
 // DivFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_DivFOp : Arith_FloatBinaryOpWithRoundingMode<"divf"> {
+def Arith_DivFOp : Arith_FloatBinaryOpWithFenv<"divf"> {
   let summary = "floating point division operation";
   let description = [{
     The `divf` operation takes two operands and returns one result, each of
@@ -1566,10 +1615,17 @@ def Arith_TruncFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast from floating-point to narrower floating-point";
   let description = [{
@@ -1589,6 +1645,7 @@ def Arith_TruncFOp :
   let hasVerifier = 1;
   let assemblyFormat = [{ $in ($roundingmode^)?
                           (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($in) `to` type($out) }];
 }
 
@@ -1601,10 +1658,17 @@ def Arith_ConvertFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast between floating-point types of the same bitwidth";
   let description = [{
@@ -1621,6 ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Mon Jul 13 12:55:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 12:55:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a5542a5.bb839515.27d21e.979c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andy Kaylor (andykaylor)

<details>
<summary>Changes</summary>

This is a proposed implementation of strict floating-point handling for Arith and Math dialects. If this direction is accepted, I will split the individual commits in this PR out for proper review. I have combined them here so that the entire design can be considered as a whole.

This is a follow-up to my prior [RFC on Discourse](https://discourse.llvm.org/t/rfc-mlir-new-representation-of-floating-point-constraints/91123) proposing a new MLIR representation of floating-point constraints. Based on the feedback I received in https://github.com/llvm/llvm-project/pull/205158, I am moving the floating-point environment attribute out to the CIR and Arith dialects and lowering these to existing operations in the LLVM dialect that continue to map directly to the current form of LLVM IR.

This PR shows the Arith and Math implementation. I have posted a separate [PR for the CIR dialect changes](https://github.com/llvm/llvm-project/pull/209275).

This is intended to replace the existing rounding-mode handling in the Arith dialect. I have left the old handling in place here but added comments that it should be considered deprecated. I am open to suggestions on how that should be handled.

The implementation here has one significant difference from the CIR implementation. In the CIR implementation, I am lowering operations with the Fenv attribute to general intrinsic calls to the constrained floating-point intrinsics. Here I have added specialized constrained operations to the LLVM dialect where they were missing and lower to those. I did this to be consistent with the existing Arith rounding-mode handling, though I think it may be more consistent with the LLVM dialect design philosophy to remove the specialized constrained operations there entirely and to lower to the generalized intrinsic calls as I have done in CIR.

Assisted-by: Cursor / claude-opus-4.8

---

Patch is 107.11 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209287.diff


19 Files Affected:

- (modified) mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h (+51-9) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithBase.td (+147-1) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOps.td (+81-8) 
- (modified) mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.td (+83) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/LLVMIntrinsicOps.td (+78) 
- (modified) mlir/include/mlir/Dialect/Math/IR/MathOps.td (+245-25) 
- (modified) mlir/lib/Conversion/ArithCommon/AttrToLLVMConverter.cpp (+17) 
- (modified) mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp (+67-26) 
- (modified) mlir/lib/Conversion/MathToLLVM/MathToLLVM.cpp (+201-34) 
- (modified) mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td (+26-14) 
- (modified) mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp (+14-10) 
- (modified) mlir/lib/Dialect/Math/IR/MathOps.cpp (+5) 
- (modified) mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir (+82) 
- (modified) mlir/test/Conversion/MathToLLVM/math-to-llvm.mlir (+128) 
- (modified) mlir/test/Dialect/Arith/expand-ops.mlir (+13) 
- (modified) mlir/test/Dialect/Arith/invalid.mlir (+8) 
- (modified) mlir/test/Dialect/Arith/ops.mlir (+23) 
- (modified) mlir/test/Dialect/Math/ops.mlir (+20) 
- (modified) mlir/test/Target/LLVMIR/Import/intrinsic-unregistered.ll (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
index feb74c86e349f..7f4bb80cc4f07 100644
--- a/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
+++ b/mlir/include/mlir/Conversion/ArithCommon/AttrToLLVMConverter.h
@@ -45,6 +45,18 @@ convertArithRoundingModeAttrToLLVM(arith::RoundingModeAttr roundingModeAttr);
 LLVM::FPExceptionBehaviorAttr
 getLLVMDefaultFPExceptionBehavior(MLIRContext &context);
 
+/// Maps an arith floating-point exception mode and strict-exception flag, as
+/// described by an `#arith.fenv` attribute, to the corresponding LLVM
+/// constrained-intrinsic exception behavior:
+///
+///   masked            + strict_except=false -> fpexcept.ignore
+///   masked            + strict_except=true  -> fpexcept.strict
+///   unmasked|unknown  + strict_except=false -> fpexcept.maytrap
+///   unmasked|unknown  + strict_except=true  -> fpexcept.strict
+LLVM::FPExceptionBehavior
+convertArithFPExceptionBehaviorToLLVM(arith::FPExceptionMode exceptionMode,
+                                      bool strictExcept);
+
 // Attribute converter that populates a NamedAttrList by removing the fastmath
 // attribute from the source operation attributes, and replacing it with an
 // equivalent LLVM fastmath attribute.
@@ -145,25 +157,55 @@ class AttrConverterConstrainedFPToLLVM {
   AttrConverterConstrainedFPToLLVM(SourceOp srcOp) {
     // Copy the source attributes.
     convertedAttr = NamedAttrList{srcOp->getAttrs()};
+    MLIRContext *ctx = srcOp->getContext();
+
+    // The floating-point environment is described by the `#arith.fenv`
+    // attribute and, on operations that predate it, may also be described by
+    // the deprecated `roundingmode` attribute. Collect both (the verifier
+    // guarantees they are never set at the same time) and remove them from the
+    // attributes carried over to the target op.
+    arith::FenvAttr fenvAttr = srcOp.getFenvAttr();
+    convertedAttr.erase(srcOp.getFenvAttrName());
+
+    // Determine the rounding mode. The `fenv` attribute takes precedence; when
+    // neither source carries one, fall back to the default dynamic rounding
+    // mode.
+    [[maybe_unused]] arith::RoundingMode roundingMode =
+        arith::FenvAttr::getDefaultDynamicRoundingMode();
+    if constexpr (SourceOp::template hasTrait<
+                      arith::ArithRoundingModeInterface::Trait>()) {
+      auto roundingModeAttr = dyn_cast_if_present<arith::RoundingModeAttr>(
+          convertedAttr.erase(srcOp.getRoundingModeAttrName()));
+      if (!fenvAttr && roundingModeAttr)
+        roundingMode = roundingModeAttr.getValue();
+    }
+    if (fenvAttr)
+      roundingMode = fenvAttr.getDynamicRoundingModeOrDefault();
 
     if constexpr (TargetOp::template hasTrait<
                       LLVM::RoundingModeOpInterface::Trait>()) {
-      // Get the name of the rounding mode attribute.
-      StringRef arithAttrName = srcOp.getRoundingModeAttrName();
-      // Remove the source attribute.
-      auto arithAttr =
-          cast<arith::RoundingModeAttr>(convertedAttr.erase(arithAttrName));
-      // Set the target attribute.
       convertedAttr.set(TargetOp::getRoundingModeAttrName(),
-                        convertArithRoundingModeAttrToLLVM(arithAttr));
+                        LLVM::RoundingModeAttr::get(
+                            ctx, convertArithRoundingModeToLLVM(roundingMode)));
     }
     // Constrained intrinsics (llvm.intr.experimental.constrained.*) do not
     // support fastmath flags. Remove the arith fastmath attribute if present.
     if constexpr (SourceOp::template hasTrait<
                       arith::ArithFastMathInterface::Trait>())
       convertedAttr.erase(srcOp.getFastMathAttrName());
-    convertedAttr.set(TargetOp::getFPExceptionBehaviorAttrName(),
-                      getLLVMDefaultFPExceptionBehavior(*srcOp->getContext()));
+
+    // Determine the exception behavior from the `fenv` attribute, defaulting to
+    // `ignore` when no environment is specified (e.g. only a `roundingmode`
+    // attribute is present).
+    LLVM::FPExceptionBehavior exceptionBehavior =
+        LLVM::FPExceptionBehavior::Ignore;
+    if (fenvAttr)
+      exceptionBehavior = convertArithFPExceptionBehaviorToLLVM(
+          fenvAttr.getExceptionModeOrDefault(),
+          fenvAttr.getStrictExceptOrDefault());
+    convertedAttr.set(
+        TargetOp::getFPExceptionBehaviorAttrName(),
+        LLVM::FPExceptionBehaviorAttr::get(ctx, exceptionBehavior));
   }
 
   ArrayRef<NamedAttribute> getAttrs() const { return convertedAttr.getAttrs(); }
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
index 985ae01008002..4132a4cfb3c55 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithBase.td
@@ -186,12 +186,158 @@ def Arith_RTowardZero                // Round toward 0
     : I32EnumAttrCase<"toward_zero", 3>;
 def Arith_RToNearestTiesAwayFromZero // Round to nearest, ties away from zero
     : I32EnumAttrCase<"to_nearest_away", 4>;
+def Arith_RDynamic                   // Rounding mode unknown at compile time
+    : I32EnumAttrCase<"unknown", 7>;  // Mirrors `llvm::RoundingMode::Dynamic`.
 
 def Arith_RoundingModeAttr : I32EnumAttr<
     "RoundingMode", "Floating point rounding mode",
     [Arith_RToNearestTiesToEven, Arith_RDownward, Arith_RUpward,
-     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero]> {
+     Arith_RTowardZero, Arith_RToNearestTiesAwayFromZero, Arith_RDynamic]> {
   let cppNamespace = "::mlir::arith";
 }
 
+//===----------------------------------------------------------------------===//
+// Floating-point environment enums (used by the `#arith.fenv` attribute)
+//===----------------------------------------------------------------------===//
+
+def FPExceptionMode : I32EnumAttr<
+    "FPExceptionMode", "floating-point exception mode", [
+  I32EnumAttrCase<"Unknown", 0, "unknown">,
+  I32EnumAttrCase<"Masked", 1, "masked">,
+  I32EnumAttrCase<"Unmasked", 2, "unmasked">
+]> {
+  let description = [{
+    The known floating-point exception mode at the point the instruction is
+    executed. If the actual exception mode differs from this value, the
+    behavior is undefined.
+  }];
+  let cppNamespace = "::mlir::arith";
+  let genSpecializedAttr = 0;
+}
+
+//===----------------------------------------------------------------------===//
+// FenvAttr
+//===----------------------------------------------------------------------===//
+
+def Arith_FenvAttr : AttrDef<Arith_Dialect, "Fenv"> {
+  let mnemonic = "fenv";
+  let summary = "Describes floating-point environment constraints";
+  let description = [{
+    The `#arith.fenv` attribute describes constraints on the floating-point
+    handling of a floating-point operation. It can be attached to
+    floating-point operations to capture rounding and exception behavior. All
+    of its parameters are optional.
+
+    - `dynamic_rounding_mode`: the known dynamic rounding mode at the point the
+      instruction is executed. If the actual dynamic rounding mode when the
+      instruction is executed does not match the assumed value, the behavior is
+      undefined.
+    - `except_mode`: the known exception mode at the point the instruction is
+      executed. If the actual state of exception masking when the instruction is
+      executed does not match this value, the behavior is undefined.
+    - `strict_except`: if `false`, any case that would produce an FP exception
+      produces it non-deterministically instead (i.e. it may or may not occur).
+      This means the FP status is written non-deterministically, and, if
+      exceptions are unmasked, the instruction traps non-deterministically.
+
+    When this attribute is present on an operation, the operation is lowered to
+    the corresponding `llvm.experimental.constrained.*` intrinsic during
+    translation to LLVM IR.
+
+    The default setting for each parameter (`unknown` for
+    `dynamic_rounding_mode`, `masked` for exception_mode, and `false` for
+    `strict_except`) is canonically represented as an unset parameter.
+    Constructing an attribute whose parameters are all set to their defaults
+    therefore yields the same canonical, empty attribute as `#arith.fenv<>`.
+
+    Carrying the (possibly empty) attribute is *not* the same as having no
+    attribute at all. An operation that carries the attribute is lowered to the
+    corresponding constrained intrinsic operation in LLVM, even when every
+    parameter is at its default value. An operation with no `#arith.fenv`
+    attribute is lowered to the regular, unconstrained operation.
+
+    Examples:
+    ```mlir
+    fenv<dynamic_rounding_mode = to_nearest_even>
+    fenv<except_mode = unmasked>
+    fenv<strict_except = true>
+    fenv<dynamic_rounding_mode = upward, except_mode = unknown, strict_except = true>
+    ```
+  }];
+
+  let parameters = (ins
+    OptionalParameter<
+        "std::optional<::mlir::arith::RoundingMode>">:$dynamic_rounding_mode,
+    OptionalParameter<"std::optional<::mlir::arith::FPExceptionMode>">:$except_mode,
+    OptionalParameter<"mlir::BoolAttr">:$strict_except);
+
+  let assemblyFormat = [{
+    `<` struct($dynamic_rounding_mode, $except_mode, $strict_except) `>`
+  }];
+
+  // Replace the default builder so that constructing the attribute normalizes
+  // any parameter that is set to its default value to "unset". This keeps a
+  // single canonical representation for the default floating-point environment.
+  // Because `skipDefaultBuilders` suppresses the auto-synthesized const builder
+  // call, provide it explicitly. The `DefaultValuedAttr` form below uses a null
+  // attribute as its default, so this simply forwards its argument (a null
+  // attribute when the operation does not carry a `#arith.fenv` attribute).
+  let skipDefaultBuilders = 1;
+  let constBuilderCall = "$0";
+  let builders = [
+    AttrBuilder<(ins
+        CArg<"std::optional<::mlir::arith::RoundingMode>",
+             "std::nullopt">:$dynamicRoundingMode,
+        CArg<"std::optional<::mlir::arith::FPExceptionMode>",
+             "std::nullopt">:$exceptMode,
+        CArg<"mlir::BoolAttr", "{}">:$strictExcept), [{
+      if (dynamicRoundingMode == ::mlir::arith::RoundingMode::unknown)
+        dynamicRoundingMode = std::nullopt;
+      if (exceptMode == ::mlir::arith::FPExceptionMode::Masked)
+        exceptMode = std::nullopt;
+      if (strictExcept && !strictExcept.getValue())
+        strictExcept = {};
+      return Base::get($_ctxt, dynamicRoundingMode, exceptMode, strictExcept);
+    }]>
+  ];
+
+  let extraClassDeclaration = [{
+    /// The default dynamic rounding mode used when `dynamic_rounding_mode` is
+    /// unset.
+    static constexpr ::mlir::arith::RoundingMode
+    getDefaultDynamicRoundingMode() {
+      return ::mlir::arith::RoundingMode::unknown;
+    }
+
+    /// The default exception mode used when `except_mode` is unset.
+    static constexpr ::mlir::arith::FPExceptionMode getDefaultExceptionMode() {
+      return ::mlir::arith::FPExceptionMode::Masked;
+    }
+
+    /// Returns the effective dynamic rounding mode, substituting the default
+    /// when unset.
+    ::mlir::arith::RoundingMode getDynamicRoundingModeOrDefault() const {
+      return getDynamicRoundingMode().value_or(getDefaultDynamicRoundingMode());
+    }
+    /// Returns the effective exception mode, substituting the default when
+    /// unset.
+    ::mlir::arith::FPExceptionMode getExceptionModeOrDefault() const {
+      return getExceptMode().value_or(getDefaultExceptionMode());
+    }
+    /// Returns the effective `strict_except` flag.
+    bool getStrictExceptOrDefault() const {
+      ::mlir::BoolAttr attr = getStrictExcept();
+      return attr ? attr.getValue() : false;
+    }
+
+    /// Returns true if every parameter is at its default value, i.e. the
+    /// attribute describes the default floating-point environment and imposes
+    /// no constraints. Because the builder normalizes defaults to "unset", this
+    /// is equivalent to all parameters being unset.
+    bool isDefault() const {
+      return !getDynamicRoundingMode() && !getExceptMode() && !getStrictExcept();
+    }
+  }];
+}
+
 #endif // ARITH_BASE
diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..c383e8d8cd1b9 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -93,6 +93,11 @@ class Arith_FloatBinaryOp<string mnemonic, list<Trait> traits = []> :
 
 // Base class for floating point binary operations with an optional rounding
 // mode.
+//
+// Deprecated: use `Arith_FloatBinaryOpWithFenv` instead. This class is retained
+// temporarily while floating-point operations migrate to the `#arith.fenv`
+// floating-point environment attribute, and will be removed once the migration
+// is complete.
 class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                                           list<Trait> traits = []> :
     Arith_FloatBinaryOp<mnemonic,
@@ -128,6 +133,50 @@ class Arith_FloatBinaryOpWithRoundingMode<string mnemonic,
                           attr-dict `:` type($result) }];
 }
 
+// Base class for floating point binary operations that can carry an optional
+// floating-point environment (`#arith.fenv`) attribute. It extends
+// `Arith_FloatBinaryOpWithRoundingMode` with the `fenv` attribute and the
+// `FPEnvConstrainedOpInterface`, which provides a uniform API for querying the
+// floating-point environment constraints.
+class Arith_FloatBinaryOpWithFenv<string mnemonic, list<Trait> traits = []> :
+    Arith_FloatBinaryOpWithRoundingMode<mnemonic,
+      !listconcat([DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>],
+                  traits)> {
+  let arguments = (ins FloatLike:$lhs, FloatLike:$rhs,
+      DefaultValuedAttr<
+        Arith_FastMathAttr, "::mlir::arith::FastMathFlags::none">:$fastmath,
+      OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
+      OptionalAttr<Arith_FenvAttr>:$fenv);
+  // With two trailing optional attributes (`roundingmode` and `fenv`), the
+  // generated builders no longer provide C++ default arguments for them, so the
+  // convenience builders must pass both explicitly.
+  let builders = [
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Value":$lhs, "Value":$rhs,
+      "::mlir::arith::FastMathFlagsAttr":$fastmath), [{
+      build($_builder, $_state, lhs, rhs, fastmath,
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+    OpBuilder<(ins "Type":$type, "Value":$lhs, "Value":$rhs,
+      CArg<"::mlir::arith::FastMathFlags",
+           "::mlir::arith::FastMathFlags::none">:$fastmath), [{
+      build($_builder, $_state, type, lhs, rhs,
+            ::mlir::arith::FastMathFlagsAttr::get(
+                $_builder.getContext(), fastmath),
+            ::mlir::arith::RoundingModeAttr{}, ::mlir::arith::FenvAttr{});
+    }]>,
+  ];
+  let assemblyFormat = [{ $lhs `,` $rhs ($roundingmode^)?
+                          (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
+                          attr-dict `:` type($result) }];
+}
+
 // Checks that tensor input and outputs have identical shapes. This is stricker
 // than the verification done in `SameOperandsAndResultShape` that allows for
 // tensor dimensions to be 'compatible' (e.g., dynamic dimensions being
@@ -1089,7 +1138,7 @@ def Arith_FlushDenormalsOp : Arith_FloatUnaryOp<"flush_denormals"> {
 // AddFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
+def Arith_AddFOp : Arith_FloatBinaryOpWithFenv<"addf", [Commutative]> {
   let summary = "floating point addition operation";
   let description = [{
     The `addf` operation takes two operands and returns one result, each of
@@ -1120,7 +1169,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
 // SubFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_SubFOp : Arith_FloatBinaryOpWithRoundingMode<"subf"> {
+def Arith_SubFOp : Arith_FloatBinaryOpWithFenv<"subf"> {
   let summary = "floating point subtraction operation";
   let description = [{
     The `subf` operation takes two operands and returns one result, each of
@@ -1276,7 +1325,7 @@ def Arith_MinUIOp : Arith_TotalIntBinaryOp<"minui",
 // MulFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
+def Arith_MulFOp : Arith_FloatBinaryOpWithFenv<"mulf", [Commutative]> {
   let summary = "floating point multiplication operation";
   let description = [{
     The `mulf` operation takes two operands and returns one result, each of
@@ -1308,7 +1357,7 @@ def Arith_MulFOp : Arith_FloatBinaryOpWithRoundingMode<"mulf", [Commutative]> {
 // DivFOp
 //===----------------------------------------------------------------------===//
 
-def Arith_DivFOp : Arith_FloatBinaryOpWithRoundingMode<"divf"> {
+def Arith_DivFOp : Arith_FloatBinaryOpWithFenv<"divf"> {
   let summary = "floating point division operation";
   let description = [{
     The `divf` operation takes two operands and returns one result, each of
@@ -1566,10 +1615,17 @@ def Arith_TruncFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast from floating-point to narrower floating-point";
   let description = [{
@@ -1589,6 +1645,7 @@ def Arith_TruncFOp :
   let hasVerifier = 1;
   let assemblyFormat = [{ $in ($roundingmode^)?
                           (`fastmath` `` $fastmath^)?
+                          (`fenv` `` $fenv^)?
                           attr-dict `:` type($in) `to` type($out) }];
 }
 
@@ -1601,10 +1658,17 @@ def Arith_ConvertFOp :
       [Pure, SameOperandsAndResultShape, SameInputOutputTensorDims,
        DeclareOpInterfaceMethods<ArithRoundingModeInterface>,
        DeclareOpInterfaceMethods<ArithFastMathInterface>,
+       DeclareOpInterfaceMethods<FPEnvConstrainedOpInterface>,
        DeclareOpInterfaceMethods<CastOpInterface>]>,
     Arguments<(ins FloatLike:$in,
+                   // Deprecated: represent the rounding mode through the `fenv`
+                   // attribute instead. This attribute is retained temporarily
+                   // while floating-point operations migrate to the
+                   // `#arith.fenv` floating-point environment attribute, and
+                   // will be removed once the migration is complete.
                    OptionalAttr<Arith_RoundingModeAttr>:$roundingmode,
-                   OptionalAttr<Arith_FastMathAttr>:$fastmath)>,
+                   OptionalAttr<Arith_FastMathAttr>:$fastmath,
+                   OptionalAttr<Arith_FenvAttr>:$fenv)>,
     Results<(outs FloatLike:$out)> {
   let summary = "cast between floating-point types of the same bitwidth";
   let description = [{
@@ -1621,6 ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Mon Jul 13 13:54:57 2026
From: llvmlistbot at llvm.org (Carlos Alberto Enciso)
Date: Mon, 13 Jul 2026 13:54:57 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [Support] Remove virtual
 functions from formatv (PR #207516)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207516@github.com>
Message-ID: <6a5550a1.160324b0.195192.2bec@mx.google.com>


CarlosAlbertoEnciso wrote:

@aengelke This change causes multiple compile errors on Windows when trying to build any MLIR executable or tools, using MS Visual Studio 2019.

https://github.com/llvm/llvm-project/pull/207516

From llvmlistbot at llvm.org  Mon Jul 13 14:51:12 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Mon, 13 Jul 2026 14:51:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555dd0.a1accbb4.244acd.6d0a@mx.google.com>


https://github.com/vzakhari edited https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:51:12 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Mon, 13 Jul 2026 14:51:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555dd0.160324b0.195192.b941@mx.google.com>


https://github.com/vzakhari approved this pull request.

Looks good to me.

https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:51:12 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Mon, 13 Jul 2026 14:51:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555dd0.d1496011.26913c.af84@mx.google.com>


================
@@ -206,5 +213,215 @@ void copyParDimsAttr(Operation *from, Operation *to) {
   setParDimsAttr(to, getParDimsAttr(from));
 }
 
+int64_t SharedMemoryBudget::alignOffset(int64_t offset) {
+  return (offset + 15) & ~static_cast<int64_t>(15);
+}
+
+bool SharedMemoryBudget::tryAllocate(int64_t bytes) {
+  int64_t aligned = alignOffset(bytesUsed_);
+  if (aligned + bytes > maxTotalBytes_) {
+    return false;
+  }
+  bytesUsed_ = aligned + bytes;
+  return true;
+}
+
+int64_t sumExistingSharedMemoryBytes(Region &region) {
+  int64_t total = 0;
+  region.walk([&](GPUSharedMemoryOp op) {
+    int64_t upperBound = op.getStaticUpperBoundBytes();
+    total = SharedMemoryBudget::alignOffset(total) + upperBound;
+  });
+  return total;
+}
+
+PrivatizeOp getPrivatizeOp(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion) {
+  Value value = privateLocal.getPrivatized();
+  if (BlockArgument blockArg = dyn_cast<BlockArgument>(value)) {
+    auto owner = dyn_cast<ComputeRegionOp>(blockArg.getOwner()->getParentOp());
+    value = (owner ? owner : computeRegion).getOperand(blockArg);
+  }
+  PrivatizeOp privatizeOp = value.getDefiningOp<PrivatizeOp>();
+  assert(privatizeOp && "expected privatize op to be the defining op");
+  return privatizeOp;
+}
+
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+MemRefType getPrivateBaseMemRefType(Type baseTy, ModuleOp module) {
+  auto memrefTy = cast<PointerLikeType>(baseTy).getAsMemRefType(module);
+  assert(memrefTy && "private base type must be convertible to memref");
+  return memrefTy;
+}
+
+SmallVector<GPUParallelDimAttr>
+collectPrivateLocalParDims(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion) {
+  SmallVector<GPUParallelDimAttr> parDims;
+  auto parentLoop = privateLocal->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDims, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
----------------
vzakhari wrote:

I wonder if we can pass `acc.compute_region` here and start processing its parent `scf.parallel` operations.

It should not be a big deal, as long as the parent loops won't have any par-dims, but we may as well stop at the compute boundary.

https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:52:13 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 13 Jul 2026 14:52:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555e0d.449bfb44.172ed9.f2d1@mx.google.com>


https://github.com/razvanlupusoru updated https://github.com/llvm/llvm-project/pull/209313

>From c686ac8dacf5f04e808cb746433d28e9a21460e8 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Mon, 13 Jul 2026 14:25:06 -0700
Subject: [PATCH 1/3] [mlir][acc] Add utilities for acc to gpu lowering

In preparation for the pass that converts `acc.compute_region`
to GPU dialect, this PR adds several utilities which are used
in that pass. Doing so to simplify review and to ensure that
unit testing is added for each of the utilities.
---
 .../mlir/Dialect/OpenACC/OpenACCUtilsCG.h     |  55 +++
 .../mlir/Dialect/OpenACC/OpenACCUtilsGPU.h    |  12 +
 .../Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp  | 223 ++++++++++
 .../Dialect/OpenACC/Utils/OpenACCUtilsGPU.cpp |  60 +++
 .../Dialect/OpenACC/OpenACCUtilsCGTest.cpp    | 418 +++++++++++++++++-
 .../Dialect/OpenACC/OpenACCUtilsGPUTest.cpp   |  59 ++-
 6 files changed, 825 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index 64de4eda7366a..cc854c3238c8a 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -15,14 +15,19 @@
 #define MLIR_DIALECT_OPENACC_OPENACCUTILSCG_H_
 
 #include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
 #include "mlir/IR/IRMapping.h"
+#include "mlir/IR/Value.h"
 #include "mlir/Interfaces/DataLayoutInterfaces.h"
+#include "mlir/Support/LogicalResult.h"
 #include "llvm/ADT/SmallVector.h"
 #include <optional>
 
 namespace mlir {
 namespace acc {
 
+class OpenACCSupport;
+
 /// Get the data layout for an operation.
 ///
 /// Attempts to get the data layout from the operation or its parent module.
@@ -90,6 +95,56 @@ void updateParDimsAttr(Operation *op, GPUParallelDimsAttr attr);
 /// Copy parallel dimensions from \p from to \p to.
 void copyParDimsAttr(Operation *from, Operation *to);
 
+/// Tracks aligned byte consumption against a configurable shared memory cap.
+class SharedMemoryBudget {
+public:
+  SharedMemoryBudget(int64_t maxTotalBytes, int64_t initialBytesUsed = 0)
+      : bytesUsed_(initialBytesUsed), maxTotalBytes_(maxTotalBytes) {}
+
+  bool tryAllocate(int64_t bytes);
+  int64_t bytesUsed() const { return bytesUsed_; }
+  int64_t maxTotalBytes() const { return maxTotalBytes_; }
+  void setMaxTotalBytes(int64_t maxTotalBytes) {
+    maxTotalBytes_ = maxTotalBytes;
+  }
+
+  static int64_t alignOffset(int64_t offset);
+
+private:
+  int64_t bytesUsed_ = 0;
+  int64_t maxTotalBytes_ = 0;
+};
+
+/// Sum aligned static_upper_bound_bytes for all acc.gpu_shared_memory in \p
+/// region.
+int64_t sumExistingSharedMemoryBytes(Region &region);
+
+/// Resolve the acc.privatize operation associated with a private local.
+PrivatizeOp getPrivatizeOp(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion);
+
+/// Returns the ranked MemRef type used to allocate privatized storage.
+///
+/// \p baseTy is the `baseTy` parameter of `acc.private_type` (the privatized
+/// variable's type).
+MemRefType getPrivateBaseMemRefType(Type baseTy, ModuleOp module);
+
+/// Collect parallel dimensions that govern privatization of \p privateLocal.
+SmallVector<GPUParallelDimAttr>
+collectPrivateLocalParDims(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion);
+
+/// True when \p privateLocal may be placed in shared memory.
+FailureOr<bool> isPrivateLocalSharedMemoryCandidate(
+    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion, ModuleOp module,
+    const ACCToGPUMappingPolicy &policy, OpenACCSupport *support = nullptr);
+
+/// Upper-bound byte size for a shared-memory private_local candidate, or
+/// std::nullopt when not eligible or not statically computable.
+std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
+    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion, ModuleOp module,
+    const ACCToGPUMappingPolicy &policy, OpenACCSupport *support = nullptr);
+
 } // namespace acc
 } // namespace mlir
 
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsGPU.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsGPU.h
index aebd15daca060..71e14a78a6925 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsGPU.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsGPU.h
@@ -16,6 +16,8 @@
 
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/Value.h"
+#include "llvm/ADT/DenseMap.h"
 #include <optional>
 
 namespace mlir {
@@ -39,6 +41,16 @@ std::optional<gpu::GPUModuleOp>
 getOrCreateGPUModule(ModuleOp mod, bool create = true,
                      llvm::StringRef name = kDefaultGPUModuleName);
 
+/// Return the launch dimension for \p processor from \p launch, or from
+/// \p dimensionOps when \p launch is null.
+Value getGPUSize(gpu::Processor processor, gpu::LaunchOp launch,
+                 const llvm::DenseMap<gpu::Processor, Value> &dimensionOps);
+
+/// Return the thread/block index for \p processor from \p launch, or from
+/// \p indexOps when \p launch is null.
+Value getGPUThreadId(gpu::Processor processor, gpu::LaunchOp launch,
+                     const llvm::DenseMap<gpu::Processor, Value> &indexOps);
+
 } // namespace acc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 06605371bdf42..06740c48a1eaf 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -11,10 +11,17 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/OpenACC/OpenACCUtilsLoop.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsType.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/IRMapping.h"
+#include "mlir/Interfaces/FunctionInterfaces.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -206,5 +213,221 @@ void copyParDimsAttr(Operation *from, Operation *to) {
   setParDimsAttr(to, getParDimsAttr(from));
 }
 
+int64_t SharedMemoryBudget::alignOffset(int64_t offset) {
+  return (offset + 15) & ~static_cast<int64_t>(15);
+}
+
+bool SharedMemoryBudget::tryAllocate(int64_t bytes) {
+  int64_t aligned = alignOffset(bytesUsed_);
+  if (aligned + bytes > maxTotalBytes_) {
+    return false;
+  }
+  bytesUsed_ = aligned + bytes;
+  return true;
+}
+
+int64_t sumExistingSharedMemoryBytes(Region &region) {
+  int64_t total = 0;
+  region.walk([&](GPUSharedMemoryOp op) {
+    int64_t upperBound = op.getStaticUpperBoundBytes();
+    total = SharedMemoryBudget::alignOffset(total) + upperBound;
+  });
+  return total;
+}
+
+PrivatizeOp getPrivatizeOp(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion) {
+  Value value = privateLocal.getPrivatized();
+  if (BlockArgument blockArg = dyn_cast<BlockArgument>(value)) {
+    auto owner = dyn_cast<ComputeRegionOp>(blockArg.getOwner()->getParentOp());
+    value = (owner ? owner : computeRegion).getOperand(blockArg);
+  }
+  PrivatizeOp privatizeOp = value.getDefiningOp<PrivatizeOp>();
+  assert(privatizeOp && "expected privatize op to be the defining op");
+  return privatizeOp;
+}
+
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+MemRefType getPrivateBaseMemRefType(Type baseTy, ModuleOp module) {
+  auto memrefTy = cast<PointerLikeType>(baseTy).getAsMemRefType(module);
+  assert(memrefTy && "private base type must be convertible to memref");
+  return memrefTy;
+}
+
+SmallVector<GPUParallelDimAttr>
+collectPrivateLocalParDims(PrivateLocalOp privateLocal,
+                           ComputeRegionOp computeRegion) {
+  SmallVector<GPUParallelDimAttr> parDims;
+  auto parentLoop = privateLocal->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDims, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDims, parDim);
+  if (parDims.empty()) {
+    for (GPUParallelDimAttr parDim : computeRegion.getLaunchParDims()) {
+      if (parDim.isAnyBlock())
+        insertParDim(parDims, parDim);
+    }
+  }
+
+  for (Operation *user : privateLocal.getResult().getUsers()) {
+    if (auto accumulateOp = dyn_cast<ReductionAccumulateOp>(user)) {
+      if (accumulateOp.getMemref() == privateLocal.getResult())
+        for (GPUParallelDimAttr parDim : accumulateOp.getParDims().getArray())
+          insertParDim(parDims, parDim);
+    }
+    if (auto combineOp = dyn_cast<ReductionCombineOp>(user)) {
+      if (combineOp.getSrcMemref() == privateLocal.getResult())
+        for (GPUParallelDimAttr parDim : getReductionCombineParDims(combineOp))
+          insertParDim(parDims, parDim);
+    }
+    if (auto combineRegionOp = dyn_cast<ReductionCombineRegionOp>(user)) {
+      if (combineRegionOp.getSrcVar() == privateLocal.getResult())
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp))
+          insertParDim(parDims, parDim);
+    }
+  }
+  return parDims;
+}
+
+static FailureOr<std::optional<int64_t>>
+getWorkerPrivateSharedMemoryNumCopies(PrivateLocalOp privateLocal,
+                                      ComputeRegionOp computeRegion,
+                                      bool isWorkerPrivate, bool isGangPrivate,
+                                      OpenACCSupport *support) {
+  if (!isWorkerPrivate || isGangPrivate)
+    return std::optional<int64_t>(1);
+
+  GPUParallelDimAttr threadY =
+      GPUParallelDimAttr::threadYDim(privateLocal.getContext());
+  std::optional<Value> workerArg = computeRegion.getKnownLaunchArg(threadY);
+  if (!workerArg)
+    return std::optional<int64_t>();
+
+  auto workerArgConst = workerArg->getDefiningOp<arith::ConstantIndexOp>();
+  if (!workerArgConst) {
+    if (support) {
+      (void)support->emitNYI(privateLocal.getLoc(),
+                             "worker-private variables in shared memory "
+                             "require compile-time constant num_workers");
+      return failure();
+    }
+    return std::optional<int64_t>();
+  }
+  return std::optional<int64_t>(workerArgConst.value());
+}
+
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  auto funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && isSpecializedAccRoutine(funcOp);
+}
+
+FailureOr<bool>
+isPrivateLocalSharedMemoryCandidate(PrivateLocalOp privateLocal,
+                                    ComputeRegionOp computeRegion,
+                                    ModuleOp module,
+                                    const ACCToGPUMappingPolicy &policy,
+                                    OpenACCSupport *support) {
+  if (isInsideACCSpecializedRoutine(computeRegion))
+    return false;
+
+  if (isThreadXPrivatize(getPrivatizeOp(privateLocal, computeRegion)))
+    return false;
+
+  bool isReductionAccumulator =
+      llvm::any_of(privateLocal.getResult().getUsers(), [](Operation *user) {
+        return isa<ReductionAccumulateOp>(user);
+      });
+
+  SmallVector<GPUParallelDimAttr> parDims =
+      collectPrivateLocalParDims(privateLocal, computeRegion);
+  bool isGangPrivate =
+      llvm::any_of(parDims, [&](auto parDim) { return policy.isGang(parDim); });
+  bool isWorkerPrivate = llvm::any_of(
+      parDims, [&](auto parDim) { return policy.isWorker(parDim); });
+  bool isVectorPrivate = llvm::any_of(
+      parDims, [&](auto parDim) { return policy.isVector(parDim); });
+
+  auto baseTy = getPrivateBaseMemRefType(
+      cast<PrivateType>(privateLocal.getPrivatized().getType()).getBaseTy(),
+      module);
+
+  bool isBlockLevelPrivate =
+      !isVectorPrivate &&
+      (isGangPrivate ||
+       (isWorkerPrivate && baseTy.getRank() > 0 && !isReductionAccumulator));
+  if (!isBlockLevelPrivate)
+    return false;
+
+  for (int64_t dim : baseTy.getShape())
+    if (dim == ShapedType::kDynamic)
+      return false;
+
+  auto resultMemRefTy = dyn_cast<MemRefType>(privateLocal.getType());
+  if (!resultMemRefTy || !resultMemRefTy.getLayout().isIdentity() ||
+      resultMemRefTy.getMemorySpace())
+    return false;
+
+  if (isGangPrivate && isWorkerPrivate && !isReductionAccumulator)
+    return false;
+
+  FailureOr<std::optional<int64_t>> numCopies =
+      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
+                                            isWorkerPrivate, isGangPrivate,
+                                            support);
+  if (failed(numCopies))
+    return failure();
+  return numCopies->has_value();
+}
+
+std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
+    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion, ModuleOp module,
+    const ACCToGPUMappingPolicy &policy, OpenACCSupport *support) {
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, policy);
+  if (failed(isCandidate) || !*isCandidate)
+    return std::nullopt;
+
+  SmallVector<GPUParallelDimAttr> parDims =
+      collectPrivateLocalParDims(privateLocal, computeRegion);
+  bool isGangPrivate =
+      llvm::any_of(parDims, [&](auto parDim) { return policy.isGang(parDim); });
+  bool isWorkerPrivate = llvm::any_of(
+      parDims, [&](auto parDim) { return policy.isWorker(parDim); });
+
+  FailureOr<std::optional<int64_t>> numCopies =
+      getWorkerPrivateSharedMemoryNumCopies(
+          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate,
+          /*support=*/nullptr);
+  if (failed(numCopies) || !numCopies->has_value())
+    return std::nullopt;
+
+  auto baseTy = getPrivateBaseMemRefType(
+      cast<PrivateType>(privateLocal.getPrivatized().getType()).getBaseTy(),
+      module);
+  std::optional<TypeSizeAndAlignment> elementSizeAndAlignment =
+      getTypeSizeAndAlignment(baseTy.getElementType(), module, support);
+  if (!elementSizeAndAlignment)
+    return std::nullopt;
+
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape())
+    numElements *= dim;
+  return elementSizeAndAlignment->first.getFixedValue() * numElements *
+         numCopies->value();
+}
+
 } // namespace acc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsGPU.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsGPU.cpp
index fb1e34f514da9..d0f454302a2a8 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsGPU.cpp
@@ -43,5 +43,65 @@ std::optional<gpu::GPUModuleOp> getOrCreateGPUModule(ModuleOp mod, bool create,
   return gpuMod;
 }
 
+static Value getGPUSizeFromLaunch(gpu::LaunchOp launch,
+                                  gpu::Processor processor) {
+  gpu::KernelDim3 gridSize = launch.getGridSize();
+  gpu::KernelDim3 blockSize = launch.getBlockSize();
+  switch (processor) {
+  case gpu::Processor::ThreadX:
+    return blockSize.x;
+  case gpu::Processor::ThreadY:
+    return blockSize.y;
+  case gpu::Processor::ThreadZ:
+    return blockSize.z;
+  case gpu::Processor::BlockX:
+    return gridSize.x;
+  case gpu::Processor::BlockY:
+    return gridSize.y;
+  case gpu::Processor::BlockZ:
+    return gridSize.z;
+  default:
+    return {};
+  }
+}
+
+Value getGPUSize(gpu::Processor processor, gpu::LaunchOp launch,
+                 const llvm::DenseMap<gpu::Processor, Value> &dimensionOps) {
+  if (launch)
+    return getGPUSizeFromLaunch(launch, processor);
+  assert(!dimensionOps.empty() && "dimension map is empty");
+  return dimensionOps.lookup(processor);
+}
+
+static Value getGPUThreadIdFromLaunch(gpu::LaunchOp launch,
+                                      gpu::Processor processor) {
+  gpu::KernelDim3 blockIds = launch.getBlockIds();
+  gpu::KernelDim3 threadIds = launch.getThreadIds();
+  switch (processor) {
+  case gpu::Processor::ThreadX:
+    return threadIds.x;
+  case gpu::Processor::ThreadY:
+    return threadIds.y;
+  case gpu::Processor::ThreadZ:
+    return threadIds.z;
+  case gpu::Processor::BlockX:
+    return blockIds.x;
+  case gpu::Processor::BlockY:
+    return blockIds.y;
+  case gpu::Processor::BlockZ:
+    return blockIds.z;
+  default:
+    return {};
+  }
+}
+
+Value getGPUThreadId(gpu::Processor processor, gpu::LaunchOp launch,
+                     const llvm::DenseMap<gpu::Processor, Value> &indexOps) {
+  if (launch)
+    return getGPUThreadIdFromLaunch(launch, processor);
+  assert(!indexOps.empty() && "index map is empty");
+  return indexOps.lookup(processor);
+}
+
 } // namespace acc
 } // namespace mlir
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 75cf385af096f..244845e58e7e8 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -11,13 +11,17 @@
 #include "mlir/Dialect/DLTI/DLTI.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
 #include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
 #include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/IRMapping.h"
 #include "mlir/IR/MLIRContext.h"
 #include "mlir/IR/OwningOpRef.h"
+#include "mlir/IR/PatternMatch.h"
 #include "gtest/gtest.h"
 
 using namespace mlir;
@@ -32,7 +36,72 @@ class OpenACCUtilsCGTest : public ::testing::Test {
   OpenACCUtilsCGTest() : b(&context), loc(UnknownLoc::get(&context)) {
     context.loadDialect<acc::OpenACCDialect, arith::ArithDialect,
                         func::FuncDialect, scf::SCFDialect, gpu::GPUDialect,
-                        DLTIDialect>();
+                        memref::MemRefDialect, DLTIDialect>();
+  }
+
+  static ComputeRegionOp buildComputeRegionWithPrivateLocal(
+      MLIRContext &context, OpBuilder &b, Location loc, ModuleOp module,
+      GPUParallelDimsAttr privatizeParDims, ValueRange launchArgs,
+      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut) {
+    IRRewriter rewriter(&context);
+    rewriter.setInsertionPointToStart(module.getBody());
+
+    MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+    Type privateTy = PrivateType::get(&context, memTy);
+    privatizeOut = PrivatizeOp::create(rewriter, loc, privateTy, ValueRange{},
+                                       privatizeParDims);
+
+    Region sourceRegion;
+    Block *srcBlock = new Block();
+    sourceRegion.push_back(srcBlock);
+    BlockArgument privArg = srcBlock->addArgument(privateTy, loc);
+    OpBuilder srcBuilder(&context);
+    srcBuilder.setInsertionPointToStart(srcBlock);
+
+    Value c0 = arith::ConstantIndexOp::create(srcBuilder, loc, 0);
+    Value c1 = arith::ConstantIndexOp::create(srcBuilder, loc, 1);
+    YieldOp::create(srcBuilder, loc);
+
+    // Nest one scf.parallel per privatization dimension so that
+    // collectPrivateLocalParDims recovers them from the parent loops.
+    srcBuilder.setInsertionPoint(srcBlock->getTerminator());
+    for (GPUParallelDimAttr parDim : privatizeParDims.getArray()) {
+      auto par = scf::ParallelOp::create(srcBuilder, loc, ValueRange{c0},
+                                         ValueRange{c1}, ValueRange{c1});
+      setParDimsAttr(par, GPUParallelDimsAttr::get(&context, {parDim}));
+      srcBuilder.setInsertionPoint(par.getBody()->getTerminator());
+    }
+    PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+
+    IRMapping mapping;
+    auto cr = buildComputeRegion(
+        loc, launchArgs, ValueRange{privatizeOut.getResult()},
+        ParallelOp::getOperationName(), sourceRegion, rewriter, mapping,
+        /*output=*/{}, /*kernelFuncName=*/{}, /*kernelModuleName=*/{},
+        /*stream=*/{}, ValueRange{privArg});
+
+    privateLocalOut = {};
+    cr.walk([&](PrivateLocalOp op) { privateLocalOut = op; });
+    return cr;
+  }
+
+  static ComputeRegionOp createEmptyComputeRegion(MLIRContext &context,
+                                                  Location loc,
+                                                  ModuleOp module) {
+    IRRewriter rewriter(&context);
+    rewriter.setInsertionPointToStart(module.getBody());
+
+    Region sourceRegion;
+    Block *block = new Block();
+    sourceRegion.push_back(block);
+    OpBuilder regionBuilder(&context);
+    regionBuilder.setInsertionPointToStart(block);
+    YieldOp::create(regionBuilder, loc);
+
+    IRMapping mapping;
+    return buildComputeRegion(loc, ValueRange{}, ValueRange{},
+                              ParallelOp::getOperationName(), sourceRegion,
+                              rewriter, mapping);
   }
 
   MLIRContext context;
@@ -312,3 +381,350 @@ TEST_F(OpenACCUtilsCGTest, buildComputeRegionWithInputArgsToMap) {
 
   func::ReturnOp::create(rewriter, loc);
 }
+
+//===----------------------------------------------------------------------===//
+// SharedMemoryBudget Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, SharedMemoryBudgetAlignAndAllocate) {
+  EXPECT_EQ(SharedMemoryBudget::alignOffset(0), 0);
+  EXPECT_EQ(SharedMemoryBudget::alignOffset(1), 16);
+  EXPECT_EQ(SharedMemoryBudget::alignOffset(16), 16);
+
+  SharedMemoryBudget budget(100);
+  EXPECT_TRUE(budget.tryAllocate(50));
+  EXPECT_EQ(budget.bytesUsed(), 50);
+  EXPECT_TRUE(budget.tryAllocate(34));
+  EXPECT_EQ(budget.bytesUsed(), 98);
+  EXPECT_FALSE(budget.tryAllocate(10));
+}
+
+TEST_F(OpenACCUtilsCGTest, SharedMemoryBudgetInitialBytesUsed) {
+  SharedMemoryBudget budget(64, /*initialBytesUsed=*/48);
+  EXPECT_FALSE(budget.tryAllocate(32));
+  EXPECT_TRUE(budget.tryAllocate(16));
+  EXPECT_EQ(budget.bytesUsed(), 64);
+}
+
+//===----------------------------------------------------------------------===//
+// sumExistingSharedMemoryBytes Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, sumExistingSharedMemoryBytes) {
+  Region region;
+  Block *block = new Block();
+  region.push_back(block);
+  b.setInsertionPointToStart(block);
+
+  MemRefType ty = MemRefType::get({4}, b.getI32Type());
+  GPUSharedMemoryOp::create(b, loc, ty, b.getI64IntegerAttr(1),
+                            b.getI64IntegerAttr(100), ValueRange{},
+                            IntegerAttr{}, IntegerAttr{});
+  GPUSharedMemoryOp::create(b, loc, ty, b.getI64IntegerAttr(1),
+                            b.getI64IntegerAttr(50), ValueRange{},
+                            IntegerAttr{}, IntegerAttr{});
+
+  EXPECT_EQ(sumExistingSharedMemoryBytes(region), 162);
+}
+
+//===----------------------------------------------------------------------===//
+// getPrivateBaseMemRefType Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, getPrivateBaseMemRefType) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  MemRefType memTy = MemRefType::get({10}, b.getI64Type());
+
+  EXPECT_EQ(getPrivateBaseMemRefType(memTy, *module), memTy);
+}
+
+//===----------------------------------------------------------------------===//
+// getPrivatizeOp Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, getPrivatizeOpFromHandle) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+  Type privateTy = PrivateType::get(&context, memTy);
+  auto privatize =
+      PrivatizeOp::create(b, loc, privateTy, /*dynamicSizes=*/ValueRange{});
+  auto privateLocal = PrivateLocalOp::create(b, loc, memTy, privatize);
+  auto computeRegion = createEmptyComputeRegion(context, loc, *module);
+
+  EXPECT_EQ(getPrivatizeOp(privateLocal, computeRegion), privatize);
+}
+
+TEST_F(OpenACCUtilsCGTest, getPrivatizeOpFromComputeRegionBlockArg) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangDims, ValueRange{bx.getResult()},
+      privateLocal, privatize);
+
+  EXPECT_EQ(getPrivatizeOp(privateLocal, cr), privatize);
+}
+
+//===----------------------------------------------------------------------===//
+// collectPrivateLocalParDims Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromParentLoops) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  IRRewriter rewriter(&context);
+  rewriter.setInsertionPointToStart(module->getBody());
+
+  auto c1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  auto bx = ParWidthOp::create(rewriter, loc, c1,
+                               GPUParallelDimAttr::blockXDim(&context));
+  auto tx = ParWidthOp::create(rewriter, loc, c1,
+                               GPUParallelDimAttr::threadXDim(&context));
+
+  MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+  Type privateTy = PrivateType::get(&context, memTy);
+  auto privatize = PrivatizeOp::create(rewriter, loc, privateTy, ValueRange{});
+
+  Region sourceRegion;
+  Block *srcBlock = new Block();
+  sourceRegion.push_back(srcBlock);
+  BlockArgument privArg = srcBlock->addArgument(privateTy, loc);
+  OpBuilder srcBuilder(&context);
+  srcBuilder.setInsertionPointToStart(srcBlock);
+  auto c0 = arith::ConstantIndexOp::create(srcBuilder, loc, 0);
+  auto c1Body = arith::ConstantIndexOp::create(srcBuilder, loc, 1);
+  YieldOp::create(srcBuilder, loc);
+  srcBuilder.setInsertionPoint(srcBlock->getTerminator());
+
+  // Outer gang (block_x) loop containing an inner vector (thread_x) loop.
+  auto gangLoop = scf::ParallelOp::create(srcBuilder, loc, ValueRange{c0},
+                                          ValueRange{c1Body}, ValueRange{c1Body});
+  setParDimsAttr(gangLoop, GPUParallelDimsAttr::get(
+                               &context, {GPUParallelDimAttr::blockXDim(&context)}));
+  srcBuilder.setInsertionPoint(gangLoop.getBody()->getTerminator());
+  auto vectorLoop = scf::ParallelOp::create(
+      srcBuilder, loc, ValueRange{c0}, ValueRange{c1Body}, ValueRange{c1Body});
+  setParDimsAttr(vectorLoop,
+                 GPUParallelDimsAttr::get(
+                     &context, {GPUParallelDimAttr::threadXDim(&context)}));
+  srcBuilder.setInsertionPoint(vectorLoop.getBody()->getTerminator());
+  PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+
+  IRMapping mapping;
+  auto cr = buildComputeRegion(
+      loc, ValueRange{bx.getResult(), tx.getResult()},
+      ValueRange{privatize.getResult()}, ParallelOp::getOperationName(),
+      sourceRegion, rewriter, mapping, /*output=*/{}, /*kernelFuncName=*/{},
+      /*kernelModuleName=*/{}, /*stream=*/{}, ValueRange{privArg});
+
+  PrivateLocalOp clonedLocal;
+  cr.walk([&](PrivateLocalOp op) { clonedLocal = op; });
+
+  DefaultACCToGPUMappingPolicy policy;
+  SmallVector<GPUParallelDimAttr> parDims =
+      collectPrivateLocalParDims(clonedLocal, cr);
+  ASSERT_EQ(parDims.size(), 2u);
+  EXPECT_TRUE(policy.isGang(parDims[0]));
+  EXPECT_TRUE(policy.isVector(parDims[1]));
+}
+
+TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromLaunchFallback) {
+  // With no enclosing parallel loops, collectPrivateLocalParDims falls back to
+  // the block-level launch dimensions.
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+
+  MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+  Type privateTy = PrivateType::get(&context, memTy);
+  auto privatize = PrivatizeOp::create(b, loc, privateTy, ValueRange{});
+
+  Region sourceRegion;
+  Block *srcBlock = new Block();
+  sourceRegion.push_back(srcBlock);
+  BlockArgument privArg = srcBlock->addArgument(privateTy, loc);
+  OpBuilder srcBuilder(&context);
+  srcBuilder.setInsertionPointToStart(srcBlock);
+  PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+  YieldOp::create(srcBuilder, loc);
+
+  IRRewriter rewriter(&context);
+  rewriter.setInsertionPointToStart(module->getBody());
+  IRMapping mapping;
+  auto cr = buildComputeRegion(
+      loc, ValueRange{bx.getResult()}, ValueRange{privatize.getResult()},
+      ParallelOp::getOperationName(), sourceRegion, rewriter, mapping,
+      /*output=*/{}, /*kernelFuncName=*/{}, /*kernelModuleName=*/{},
+      /*stream=*/{}, ValueRange{privArg});
+  (void)gangDims;
+
+  PrivateLocalOp clonedLocal;
+  cr.walk([&](PrivateLocalOp op) { clonedLocal = op; });
+
+  DefaultACCToGPUMappingPolicy policy;
+  SmallVector<GPUParallelDimAttr> parDims =
+      collectPrivateLocalParDims(clonedLocal, cr);
+  ASSERT_EQ(parDims.size(), 1u);
+  EXPECT_TRUE(policy.isGang(parDims[0]));
+}
+
+TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromReductionUsers) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  MemRefType memTy = MemRefType::get({}, b.getI32Type());
+  Type privateTy = PrivateType::get(&context, memTy);
+  auto privatize =
+      PrivatizeOp::create(b, loc, privateTy, /*dynamicSizes=*/ValueRange{});
+  auto privateLocal = PrivateLocalOp::create(b, loc, memTy, privatize);
+  auto computeRegion = createEmptyComputeRegion(context, loc, *module);
+
+  GPUParallelDimsAttr accDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context),
+                 GPUParallelDimAttr::threadXDim(&context)});
+  Value partial = arith::ConstantIntOp::create(b, loc, b.getI32Type(), 1);
+  ReductionAccumulateOp::create(b, loc, partial, privateLocal.getResult(),
+                                ReductionOperator::AccAdd, accDims);
+
+  DefaultACCToGPUMappingPolicy policy;
+  SmallVector<GPUParallelDimAttr> parDims =
+      collectPrivateLocalParDims(privateLocal, computeRegion);
+  ASSERT_EQ(parDims.size(), 2u);
+  EXPECT_TRUE(policy.isGang(parDims[0]));
+  EXPECT_TRUE(policy.isVector(parDims[1]));
+}
+
+//===----------------------------------------------------------------------===//
+// Shared-memory private_local eligibility Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateGangPrivate) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangDims, ValueRange{bx.getResult()},
+      privateLocal, privatize);
+
+  DefaultACCToGPUMappingPolicy policy;
+  FailureOr<bool> isCandidate =
+      isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
+  ASSERT_TRUE(succeeded(isCandidate));
+  EXPECT_TRUE(*isCandidate);
+}
+
+TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateThreadXPrivate) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr vectorDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::threadXDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto tx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::threadXDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, vectorDims, ValueRange{tx.getResult()},
+      privateLocal, privatize);
+
+  DefaultACCToGPUMappingPolicy policy;
+  FailureOr<bool> isCandidate =
+      isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
+  ASSERT_TRUE(succeeded(isCandidate));
+  EXPECT_FALSE(*isCandidate);
+}
+
+TEST_F(OpenACCUtilsCGTest,
+       isPrivateLocalSharedMemoryCandidateWorkerPrivateConstant) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::threadYDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+  auto ty =
+      ParWidthOp::create(b, loc, c4, GPUParallelDimAttr::threadYDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, workerDims,
+      ValueRange{bx.getResult(), ty.getResult()}, privateLocal, privatize);
+
+  DefaultACCToGPUMappingPolicy policy;
+  FailureOr<bool> isCandidate =
+      isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
+  ASSERT_TRUE(succeeded(isCandidate));
+  EXPECT_TRUE(*isCandidate);
+}
+
+TEST_F(OpenACCUtilsCGTest,
+       isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::threadYDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+  // A non-constant num_workers: the launch operand exists but is not an
+  // arith.constant, which is what triggers the diagnostic / failure path.
+  auto dynNumWorkers = arith::AddIOp::create(b, loc, c1, c1);
+  auto ty = ParWidthOp::create(b, loc, dynNumWorkers,
+                               GPUParallelDimAttr::threadYDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, workerDims,
+      ValueRange{bx.getResult(), ty.getResult()}, privateLocal, privatize);
+
+  DefaultACCToGPUMappingPolicy policy;
+  OpenACCSupport support;
+  FailureOr<bool> silent =
+      isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
+  ASSERT_TRUE(succeeded(silent));
+  EXPECT_FALSE(*silent);
+
+  FailureOr<bool> diagnosed = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, cr, *module, policy, &support);
+  EXPECT_TRUE(failed(diagnosed));
+}
+
+TEST_F(OpenACCUtilsCGTest, getPrivateLocalSharedMemoryUpperBoundBytes) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangDims, ValueRange{bx.getResult()},
+      privateLocal, privatize);
+
+  DefaultACCToGPUMappingPolicy policy;
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 16);
+}
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
index e9d94edde3a92..49090f4d657d1 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/MLIRContext.h"
@@ -23,7 +24,7 @@ using namespace mlir::acc;
 class OpenACCUtilsGPUTest : public ::testing::Test {
 protected:
   OpenACCUtilsGPUTest() : b(&context), loc(UnknownLoc::get(&context)) {
-    context.loadDialect<gpu::GPUDialect>();
+    context.loadDialect<arith::ArithDialect, gpu::GPUDialect>();
   }
 
   MLIRContext context;
@@ -87,3 +88,59 @@ TEST_F(OpenACCUtilsGPUTest, getOrCreateGPUModuleEmptyNameUsesDefault) {
   ASSERT_TRUE(gpuMod.has_value());
   EXPECT_EQ(gpuMod->getName(), kDefaultGPUModuleName);
 }
+
+//===----------------------------------------------------------------------===//
+// getGPUSize / getGPUThreadId Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsGPUTest, getGPUSizeFromLaunch) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  auto c128 = arith::ConstantIndexOp::create(b, loc, 128);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto launch = gpu::LaunchOp::create(b, loc, c4, c1, c1, c128, c1, c1);
+  llvm::DenseMap<gpu::Processor, Value> dimensionOps;
+
+  // getGPUSize returns the in-kernel size SSA values (block arguments), which
+  // are the values exposed by getBlockSize()/getGridSize().
+  EXPECT_EQ(getGPUSize(gpu::Processor::ThreadX, launch, dimensionOps),
+            launch.getBlockSize().x);
+  EXPECT_EQ(getGPUSize(gpu::Processor::BlockX, launch, dimensionOps),
+            launch.getGridSize().x);
+}
+
+TEST_F(OpenACCUtilsGPUTest, getGPUSizeFromDimensionMap) {
+  auto c128 = arith::ConstantIndexOp::create(b, loc, 128);
+  llvm::DenseMap<gpu::Processor, Value> dimensionOps;
+  dimensionOps[gpu::Processor::ThreadX] = c128.getResult();
+
+  EXPECT_EQ(getGPUSize(gpu::Processor::ThreadX, nullptr, dimensionOps),
+            c128.getResult());
+}
+
+TEST_F(OpenACCUtilsGPUTest, getGPUThreadIdFromLaunch) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+
+  auto c128 = arith::ConstantIndexOp::create(b, loc, 128);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto launch = gpu::LaunchOp::create(b, loc, c4, c1, c1, c128, c1, c1);
+  llvm::DenseMap<gpu::Processor, Value> indexOps;
+
+  EXPECT_EQ(getGPUThreadId(gpu::Processor::ThreadX, launch, indexOps),
+            launch.getThreadIds().x);
+  EXPECT_EQ(getGPUThreadId(gpu::Processor::BlockX, launch, indexOps),
+            launch.getBlockIds().x);
+}
+
+TEST_F(OpenACCUtilsGPUTest, getGPUThreadIdFromIndexMap) {
+  auto c7 = arith::ConstantIndexOp::create(b, loc, 7);
+  llvm::DenseMap<gpu::Processor, Value> indexOps;
+  indexOps[gpu::Processor::BlockX] = c7.getResult();
+
+  EXPECT_EQ(getGPUThreadId(gpu::Processor::BlockX, nullptr, indexOps),
+            c7.getResult());
+}

>From 25652c08d21cceecb313ee92fcb5b753c1f9ca82 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Mon, 13 Jul 2026 14:30:26 -0700
Subject: [PATCH 2/3] Fix formatting

---
 .../Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp  | 28 ++++++++-----------
 .../Dialect/OpenACC/OpenACCUtilsCGTest.cpp    |  9 +++---
 2 files changed, 16 insertions(+), 21 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 06740c48a1eaf..1c33f6fee48e8 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -302,11 +302,9 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
   return parDims;
 }
 
-static FailureOr<std::optional<int64_t>>
-getWorkerPrivateSharedMemoryNumCopies(PrivateLocalOp privateLocal,
-                                      ComputeRegionOp computeRegion,
-                                      bool isWorkerPrivate, bool isGangPrivate,
-                                      OpenACCSupport *support) {
+static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
+    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
+    bool isWorkerPrivate, bool isGangPrivate, OpenACCSupport *support) {
   if (!isWorkerPrivate || isGangPrivate)
     return std::optional<int64_t>(1);
 
@@ -334,12 +332,9 @@ static bool isInsideACCSpecializedRoutine(Operation *op) {
   return funcOp && isSpecializedAccRoutine(funcOp);
 }
 
-FailureOr<bool>
-isPrivateLocalSharedMemoryCandidate(PrivateLocalOp privateLocal,
-                                    ComputeRegionOp computeRegion,
-                                    ModuleOp module,
-                                    const ACCToGPUMappingPolicy &policy,
-                                    OpenACCSupport *support) {
+FailureOr<bool> isPrivateLocalSharedMemoryCandidate(
+    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion, ModuleOp module,
+    const ACCToGPUMappingPolicy &policy, OpenACCSupport *support) {
   if (isInsideACCSpecializedRoutine(computeRegion))
     return false;
 
@@ -384,9 +379,8 @@ isPrivateLocalSharedMemoryCandidate(PrivateLocalOp privateLocal,
     return false;
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
-                                            isWorkerPrivate, isGangPrivate,
-                                            support);
+      getWorkerPrivateSharedMemoryNumCopies(
+          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate, support);
   if (failed(numCopies))
     return failure();
   return numCopies->has_value();
@@ -408,9 +402,9 @@ std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
       parDims, [&](auto parDim) { return policy.isWorker(parDim); });
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(
-          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate,
-          /*support=*/nullptr);
+      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
+                                            isWorkerPrivate, isGangPrivate,
+                                            /*support=*/nullptr);
   if (failed(numCopies) || !numCopies->has_value())
     return std::nullopt;
 
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 244845e58e7e8..abf2294db114d 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -504,10 +504,11 @@ TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromParentLoops) {
   srcBuilder.setInsertionPoint(srcBlock->getTerminator());
 
   // Outer gang (block_x) loop containing an inner vector (thread_x) loop.
-  auto gangLoop = scf::ParallelOp::create(srcBuilder, loc, ValueRange{c0},
-                                          ValueRange{c1Body}, ValueRange{c1Body});
-  setParDimsAttr(gangLoop, GPUParallelDimsAttr::get(
-                               &context, {GPUParallelDimAttr::blockXDim(&context)}));
+  auto gangLoop = scf::ParallelOp::create(
+      srcBuilder, loc, ValueRange{c0}, ValueRange{c1Body}, ValueRange{c1Body});
+  setParDimsAttr(gangLoop,
+                 GPUParallelDimsAttr::get(
+                     &context, {GPUParallelDimAttr::blockXDim(&context)}));
   srcBuilder.setInsertionPoint(gangLoop.getBody()->getTerminator());
   auto vectorLoop = scf::ParallelOp::create(
       srcBuilder, loc, ValueRange{c0}, ValueRange{c1Body}, ValueRange{c1Body});

>From 09c9bcc80a5c5e944c11f13b8adafb168b2eecbf Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Mon, 13 Jul 2026 14:52:01 -0700
Subject: [PATCH 3/3] Avoid hardcoded alignment

---
 mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h | 14 ++++++++++++--
 mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp  | 11 +++++++----
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index cc854c3238c8a..01be7760aad77 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -98,17 +98,27 @@ void copyParDimsAttr(Operation *from, Operation *to);
 /// Tracks aligned byte consumption against a configurable shared memory cap.
 class SharedMemoryBudget {
 public:
+  /// Default allocation alignment (bytes).
+  static constexpr int64_t kDefaultAlignmentBytes = 16;
+
   SharedMemoryBudget(int64_t maxTotalBytes, int64_t initialBytesUsed = 0)
       : bytesUsed_(initialBytesUsed), maxTotalBytes_(maxTotalBytes) {}
 
-  bool tryAllocate(int64_t bytes);
+  /// Reserve \p bytes, rounding the current offset up to \p alignment first.
+  /// Returns false without mutating state if the reservation would exceed the
+  /// cap. \p alignment must be a power of two.
+  bool tryAllocate(int64_t bytes,
+                   int64_t alignment = kDefaultAlignmentBytes);
   int64_t bytesUsed() const { return bytesUsed_; }
   int64_t maxTotalBytes() const { return maxTotalBytes_; }
   void setMaxTotalBytes(int64_t maxTotalBytes) {
     maxTotalBytes_ = maxTotalBytes;
   }
 
-  static int64_t alignOffset(int64_t offset);
+  /// Round \p offset up to the next multiple of \p alignment, which must be a
+  /// power of two.
+  static int64_t alignOffset(int64_t offset,
+                             int64_t alignment = kDefaultAlignmentBytes);
 
 private:
   int64_t bytesUsed_ = 0;
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 1c33f6fee48e8..1bc79531523aa 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -25,6 +25,7 @@
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/TypeSwitch.h"
+#include "llvm/Support/MathExtras.h"
 
 namespace mlir {
 namespace acc {
@@ -213,12 +214,14 @@ void copyParDimsAttr(Operation *from, Operation *to) {
   setParDimsAttr(to, getParDimsAttr(from));
 }
 
-int64_t SharedMemoryBudget::alignOffset(int64_t offset) {
-  return (offset + 15) & ~static_cast<int64_t>(15);
+int64_t SharedMemoryBudget::alignOffset(int64_t offset, int64_t alignment) {
+  assert(alignment > 0 && llvm::isPowerOf2_64(alignment) &&
+         "alignment must be a power of two");
+  return (offset + alignment - 1) & ~(alignment - 1);
 }
 
-bool SharedMemoryBudget::tryAllocate(int64_t bytes) {
-  int64_t aligned = alignOffset(bytesUsed_);
+bool SharedMemoryBudget::tryAllocate(int64_t bytes, int64_t alignment) {
+  int64_t aligned = alignOffset(bytesUsed_, alignment);
   if (aligned + bytes > maxTotalBytes_) {
     return false;
   }


From llvmlistbot at llvm.org  Mon Jul 13 14:53:16 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Mon, 13 Jul 2026 14:53:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555e4c.d76d81af.39f9be.72ba@mx.google.com>


https://github.com/clementval approved this pull request.


https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:53:23 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 13 Jul 2026 14:53:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555e53.481afadf.196d57.9bf3@mx.google.com>


================
@@ -206,5 +213,221 @@ void copyParDimsAttr(Operation *from, Operation *to) {
   setParDimsAttr(to, getParDimsAttr(from));
 }
 
+int64_t SharedMemoryBudget::alignOffset(int64_t offset) {
+  return (offset + 15) & ~static_cast<int64_t>(15);
----------------
razvanlupusoru wrote:

Added a default alignment and updated API to allow for non-default alignment. Thanks for this suggestion - I like this better.

https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:54:22 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Mon, 13 Jul 2026 14:54:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add utilities for acc to gpu
	lowering (PR #209313)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209313@github.com>
Message-ID: <6a555e8e.d1496011.26913c.dfb2@mx.google.com>


================
@@ -206,5 +213,221 @@ void copyParDimsAttr(Operation *from, Operation *to) {
   setParDimsAttr(to, getParDimsAttr(from));
 }
 
+int64_t SharedMemoryBudget::alignOffset(int64_t offset) {
+  return (offset + 15) & ~static_cast<int64_t>(15);
----------------
clementval wrote:

Looks great! Thanks for the change. 

https://github.com/llvm/llvm-project/pull/209313

From llvmlistbot at llvm.org  Mon Jul 13 14:55:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 14:55:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 0661f66 - [mlir][acc] Add reduction utilities
 for acc to gpu lowering (#209316)
Message-ID: <6a555ebe.492e23e4.14eda0.a8a3@mx.google.com>


Author: Razvan Lupusoru
Date: 2026-07-13T21:55:05Z
New Revision: 0661f6636a0a5a79635e99d98a239c2bba4ea2ed

URL: https://github.com/llvm/llvm-project/commit/0661f6636a0a5a79635e99d98a239c2bba4ea2ed
DIFF: https://github.com/llvm/llvm-project/commit/0661f6636a0a5a79635e99d98a239c2bba4ea2ed.diff

LOG: [mlir][acc] Add reduction utilities for acc to gpu lowering (#209316)

In preparation for the pass that converts `acc.compute_region` to GPU
dialect, this PR adds several utilities which are used in that pass
related to reductions. Doing so to simplify review and to ensure that
unit testing is added for the utilities.

---------

Co-authored-by: Scott Manley <rscottmanley at gmail.com>

Added: 
    mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsReduction.h
    mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsReduction.cpp
    mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp

Modified: 
    mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
    mlir/unittests/Dialect/OpenACC/CMakeLists.txt

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsReduction.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsReduction.h
new file mode 100644
index 0000000000000..ba4c09c0aaf5f
--- /dev/null
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsReduction.h
@@ -0,0 +1,66 @@
+//===- OpenACCUtilsReduction.h - OpenACC reduction utilities ----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines utility functions for OpenACC reductions.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_OPENACC_OPENACCUTILSREDUCTION_H_
+#define MLIR_DIALECT_OPENACC_OPENACCUTILSREDUCTION_H_
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/Value.h"
+#include "llvm/ADT/SmallVector.h"
+
+#include <optional>
+
+namespace mlir {
+namespace acc {
+
+/// Returns the parallel dimensions that participate in \p op's combine step.
+///
+/// Used when lowering reductions to determine which GPU parallelism levels must
+/// be synchronized before combining partial results.
+SmallVector<GPUParallelDimAttr>
+getReductionCombineParDims(ReductionCombineOp op);
+
+/// Returns the parallel dimensions that participate in \p op's combine step.
+///
+/// Prefers dimensions from an `acc.reduction_accumulate` user of the source
+/// variable; otherwise falls back to \p op's `acc.par_dims` attribute.
+SmallVector<GPUParallelDimAttr>
+getReductionCombineParDims(ReductionCombineRegionOp op);
+
+/// Maps an `arith` atomic RMW kind to the corresponding acc reduction operator.
+ReductionOperator translateAtomicRMWKind(arith::AtomicRMWKind kind);
+
+/// Maps an acc reduction operator to the `arith` atomic RMW kind for \p type.
+///
+/// Returns `std::nullopt` when \p redOp is not supported for \p type.
+std::optional<arith::AtomicRMWKind>
+translateACCReductionOperator(ReductionOperator redOp, Type type);
+
+/// Creates the identity (neutral) value for a reduction of \p type and \p kind.
+///
+/// When \p useOnlyFiniteValue is true, floating-point identities avoid
+/// non-finite sentinel values where applicable.
+Value createIdentityValue(OpBuilder &b, Location loc, Type type,
+                          arith::AtomicRMWKind kind,
+                          bool useOnlyFiniteValue = true);
+
+/// Combines two reduction partial values using the operator for \p kind.
+Value generateReductionOp(OpBuilder &b, Location loc, Value lhs, Value rhs,
+                          arith::AtomicRMWKind kind);
+
+} // namespace acc
+} // namespace mlir
+
+#endif // MLIR_DIALECT_OPENACC_OPENACCUTILSREDUCTION_H_

diff  --git a/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt b/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
index 67e1fc269ffa4..fc30625964c43 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenACC/Utils/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_dialect_library(MLIROpenACCUtils
   OpenACCUtilsCG.cpp
   OpenACCUtilsGPU.cpp
   OpenACCUtilsLoop.cpp
+  OpenACCUtilsReduction.cpp
   OpenACCUtilsTiling.cpp
   OpenACCUtilsType.cpp
 
@@ -21,8 +22,10 @@ add_mlir_dialect_library(MLIROpenACCUtils
   LINK_LIBS PUBLIC
   MLIRArithDialect
   MLIRArithUtils
+  MLIRComplexDialect
   MLIRDataLayoutInterfaces
   MLIRGPUDialect
+  MLIRMemRefDialect
   MLIROpenACCDialect
   MLIRIR
   MLIRSCFDialect

diff  --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsReduction.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsReduction.cpp
new file mode 100644
index 0000000000000..df69714344f14
--- /dev/null
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsReduction.cpp
@@ -0,0 +1,221 @@
+//===- OpenACCUtilsReduction.cpp - OpenACC reduction utilities ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace mlir;
+
+namespace mlir {
+namespace acc {
+
+static bool isFloatOrComplexType(Type ty) {
+  return isa<FloatType, ComplexType>(ty);
+}
+
+SmallVector<GPUParallelDimAttr>
+getReductionCombineParDims(ReductionCombineOp reductionCombineOp) {
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(reductionCombineOp))
+    return SmallVector<GPUParallelDimAttr>(parDimsAttr.getArray());
+  llvm_unreachable(
+      "expected parallel dimensions attribute for reduction combine op");
+}
+
+SmallVector<GPUParallelDimAttr>
+getReductionCombineParDims(ReductionCombineRegionOp combineRegionOp) {
+  for (Operation *user : combineRegionOp.getSrcVar().getUsers()) {
+    if (auto accumulateOp = dyn_cast<ReductionAccumulateOp>(user))
+      return SmallVector<GPUParallelDimAttr>(
+          accumulateOp.getParDims().getArray());
+  }
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(combineRegionOp))
+    return SmallVector<GPUParallelDimAttr>(parDimsAttr.getArray());
+  return {};
+}
+
+ReductionOperator translateAtomicRMWKind(arith::AtomicRMWKind kind) {
+  switch (kind) {
+  case arith::AtomicRMWKind::addf:
+  case arith::AtomicRMWKind::addi:
+    return ReductionOperator::AccAdd;
+  case arith::AtomicRMWKind::mulf:
+  case arith::AtomicRMWKind::muli:
+    return ReductionOperator::AccMul;
+  case arith::AtomicRMWKind::maxs:
+  case arith::AtomicRMWKind::maxu:
+  case arith::AtomicRMWKind::maximumf:
+  case arith::AtomicRMWKind::maxnumf:
+    return ReductionOperator::AccMax;
+  case arith::AtomicRMWKind::minu:
+  case arith::AtomicRMWKind::mins:
+  case arith::AtomicRMWKind::minimumf:
+  case arith::AtomicRMWKind::minnumf:
+    return ReductionOperator::AccMin;
+  case arith::AtomicRMWKind::andi:
+    return ReductionOperator::AccIand;
+  case arith::AtomicRMWKind::ori:
+    return ReductionOperator::AccIor;
+  case arith::AtomicRMWKind::xori:
+    return ReductionOperator::AccXor;
+  case arith::AtomicRMWKind::assign:
+    break;
+  }
+  llvm_unreachable("unsupported atomic kind");
+}
+
+std::optional<arith::AtomicRMWKind>
+translateACCReductionOperator(ReductionOperator redOp, Type type) {
+  if (type.isInteger() && type.isUnsignedInteger())
+    return std::nullopt;
+
+  if (auto reducible = dyn_cast<ReducibleType>(type)) {
+    if (std::optional<arith::AtomicRMWKind> kind =
+            reducible.getAtomicRMWKind(redOp))
+      return kind;
+    return std::nullopt;
+  }
+
+  switch (redOp) {
+  case ReductionOperator::AccAdd:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::addi;
+    if (isFloatOrComplexType(type))
+      return arith::AtomicRMWKind::addf;
+    break;
+  case ReductionOperator::AccMul:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::muli;
+    if (isFloatOrComplexType(type))
+      return arith::AtomicRMWKind::mulf;
+    break;
+  case ReductionOperator::AccMax:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::maxs;
+    if (type.isFloat())
+      return arith::AtomicRMWKind::maxnumf;
+    break;
+  case ReductionOperator::AccMaximumf:
+    return arith::AtomicRMWKind::maximumf;
+  case ReductionOperator::AccMaxnumf:
+    return arith::AtomicRMWKind::maxnumf;
+  case ReductionOperator::AccMin:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::mins;
+    if (type.isFloat())
+      return arith::AtomicRMWKind::minnumf;
+    break;
+  case ReductionOperator::AccMinimumf:
+    return arith::AtomicRMWKind::minimumf;
+  case ReductionOperator::AccMinnumf:
+    return arith::AtomicRMWKind::minnumf;
+  case ReductionOperator::AccIand:
+  case ReductionOperator::AccLand:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::andi;
+    break;
+  case ReductionOperator::AccIor:
+  case ReductionOperator::AccLor:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::ori;
+    break;
+  case ReductionOperator::AccXor:
+  case ReductionOperator::AccNeqv:
+    if (type.isInteger())
+      return arith::AtomicRMWKind::xori;
+    break;
+  case ReductionOperator::AccEqv:
+  case ReductionOperator::AccNone:
+    break;
+  }
+  return std::nullopt;
+}
+
+static TypedAttr getReductionIdentityValueAttr(arith::AtomicRMWKind kind,
+                                               Type type, OpBuilder &builder,
+                                               Location loc,
+                                               bool useOnlyFiniteValue) {
+  if (type.isIntOrIndexOrFloat()) {
+    TypedAttr attr = arith::getIdentityValueAttr(kind, type, builder, loc,
+                                                 useOnlyFiniteValue);
+    if (!attr)
+      emitError(loc) << "reduction identity: operator not supported " << kind;
+    return attr;
+  }
+  if (auto complexTy = dyn_cast<ComplexType>(type)) {
+    auto eltTy = dyn_cast<FloatType>(complexTy.getElementType());
+    if (!eltTy) {
+      emitError(loc) << "reduction identity: complex with non-floating "
+                        "element type";
+      return nullptr;
+    }
+    switch (kind) {
+    case arith::AtomicRMWKind::addf: {
+      TypedAttr scalarAttr = arith::getIdentityValueAttr(
+          kind, eltTy, builder, loc, useOnlyFiniteValue);
+      assert(scalarAttr && "expected scalar identity for complex reduction");
+      double d = cast<FloatAttr>(scalarAttr).getValue().convertToDouble();
+      return complex::NumberAttr::get(complexTy, d, d);
+    }
+    case arith::AtomicRMWKind::mulf: {
+      TypedAttr scalarAttr = arith::getIdentityValueAttr(
+          kind, eltTy, builder, loc, useOnlyFiniteValue);
+      assert(scalarAttr &&
+             "expected scalar identity for complex mulf reduction");
+      auto realPart = cast<FloatAttr>(scalarAttr).getValue();
+      return complex::NumberAttr::get(complexTy, realPart.convertToDouble(),
+                                      0.0);
+    }
+    default:
+      emitError(loc)
+          << "reduction identity: operator not supported for complex " << kind;
+      return nullptr;
+    }
+  }
+  emitError(loc) << "reduction identity: type not supported " << type;
+  return nullptr;
+}
+
+Value createIdentityValue(OpBuilder &b, Location loc, Type type,
+                          arith::AtomicRMWKind kind, bool useOnlyFiniteValue) {
+  TypedAttr typedAttr =
+      getReductionIdentityValueAttr(kind, type, b, loc, useOnlyFiniteValue);
+  assert(typedAttr && "expected identity attribute");
+  if (auto numAttr = dyn_cast<complex::NumberAttr>(typedAttr)) {
+    auto complexTy = cast<ComplexType>(numAttr.getType());
+    auto floatElt = cast<FloatType>(complexTy.getElementType());
+    Value realVal = arith::ConstantOp::create(
+        b, loc, b.getFloatAttr(floatElt, numAttr.getReal()));
+    Value imagVal = arith::ConstantOp::create(
+        b, loc, b.getFloatAttr(floatElt, numAttr.getImag()));
+    return complex::CreateOp::create(b, loc, complexTy, realVal, imagVal);
+  }
+  return arith::ConstantOp::create(b, loc, typedAttr);
+}
+
+Value generateReductionOp(OpBuilder &b, Location loc, Value lhs, Value rhs,
+                          arith::AtomicRMWKind kind) {
+  assert(lhs.getType() == rhs.getType() &&
+         "expected same type for lhs and rhs");
+  if (isa<ComplexType>(lhs.getType())) {
+    switch (kind) {
+    case arith::AtomicRMWKind::addf:
+      return complex::AddOp::create(b, loc, lhs, rhs);
+    case arith::AtomicRMWKind::mulf:
+      return complex::MulOp::create(b, loc, lhs, rhs);
+    default:
+      llvm_unreachable("unsupported complex atomic reduction kind");
+    }
+  }
+  return arith::getReductionOp(kind, b, loc, lhs, rhs);
+}
+
+} // namespace acc
+} // namespace mlir

diff  --git a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
index 3a14d3e6796cb..c3b56ec877827 100644
--- a/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
+++ b/mlir/unittests/Dialect/OpenACC/CMakeLists.txt
@@ -5,6 +5,7 @@ add_mlir_unittest(MLIROpenACCTests
   OpenACCTypeInterfacesTest.cpp
   OpenACCUtilsCGTest.cpp
   OpenACCUtilsGPUTest.cpp
+  OpenACCUtilsReductionTest.cpp
   OpenACCUtilsTest.cpp
   OpenACCUtilsTypeTest.cpp
   OpenACCUtilsTilingTest.cpp
@@ -14,12 +15,14 @@ mlir_target_link_libraries(MLIROpenACCTests
   PRIVATE
   MLIRIR
   MLIRAffineDialect
+  MLIRComplexDialect
   MLIRDLTIDialect
   MLIRFuncDialect
   MLIRGPUDialect
   MLIRLLVMDialect
   MLIRMemRefDialect
   MLIRArithDialect
+  MLIROpenACCAnalysis
   MLIROpenACCDialect
   MLIROpenACCUtils
   MLIRSCFDialect

diff  --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp
new file mode 100644
index 0000000000000..566f905ff24d5
--- /dev/null
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp
@@ -0,0 +1,192 @@
+//===- OpenACCUtilsReductionTest.cpp - OpenACC reduction utility tests ----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/MLIRContext.h"
+#include "mlir/IR/OwningOpRef.h"
+#include "gtest/gtest.h"
+
+using namespace mlir;
+using namespace mlir::acc;
+
+//===----------------------------------------------------------------------===//
+// Test Fixture
+//===----------------------------------------------------------------------===//
+
+class OpenACCUtilsReductionTest : public ::testing::Test {
+protected:
+  OpenACCUtilsReductionTest() : b(&context), loc(UnknownLoc::get(&context)) {
+    context.loadDialect<acc::OpenACCDialect, arith::ArithDialect,
+                        complex::ComplexDialect, memref::MemRefDialect>();
+  }
+
+  MLIRContext context;
+  OpBuilder b;
+  Location loc;
+};
+
+//===----------------------------------------------------------------------===//
+// translateAtomicRMWKind / translateACCReductionOperator Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsReductionTest, translateAtomicRMWKind) {
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::addi),
+            ReductionOperator::AccAdd);
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::addf),
+            ReductionOperator::AccAdd);
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::muli),
+            ReductionOperator::AccMul);
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::maxnumf),
+            ReductionOperator::AccMax);
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::minnumf),
+            ReductionOperator::AccMin);
+  EXPECT_EQ(translateAtomicRMWKind(arith::AtomicRMWKind::andi),
+            ReductionOperator::AccIand);
+}
+
+TEST_F(OpenACCUtilsReductionTest, translateACCReductionOperator) {
+  EXPECT_EQ(
+      *translateACCReductionOperator(ReductionOperator::AccAdd, b.getI32Type()),
+      arith::AtomicRMWKind::addi);
+  EXPECT_EQ(
+      *translateACCReductionOperator(ReductionOperator::AccAdd, b.getF32Type()),
+      arith::AtomicRMWKind::addf);
+  EXPECT_EQ(
+      *translateACCReductionOperator(ReductionOperator::AccMul, b.getI64Type()),
+      arith::AtomicRMWKind::muli);
+  EXPECT_EQ(
+      *translateACCReductionOperator(ReductionOperator::AccMax, b.getF32Type()),
+      arith::AtomicRMWKind::maxnumf);
+  EXPECT_FALSE(translateACCReductionOperator(ReductionOperator::AccAdd,
+                                             b.getIntegerType(32, false)));
+}
+
+//===----------------------------------------------------------------------===//
+// getReductionCombineParDims Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsReductionTest, getReductionCombineParDimsFromCombineOp) {
+  MemRefType memTy = MemRefType::get({}, b.getI32Type());
+  auto dest = memref::AllocaOp::create(b, loc, memTy);
+  auto src = memref::AllocaOp::create(b, loc, memTy);
+  auto combine =
+      ReductionCombineOp::create(b, loc, dest, src, ReductionOperator::AccAdd);
+  GPUParallelDimsAttr parDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context),
+                 GPUParallelDimAttr::threadXDim(&context)});
+  setParDimsAttr(combine, parDims);
+
+  SmallVector<GPUParallelDimAttr> result = getReductionCombineParDims(combine);
+  ASSERT_EQ(result.size(), 2u);
+  EXPECT_EQ(result[0], GPUParallelDimAttr::blockXDim(&context));
+  EXPECT_EQ(result[1], GPUParallelDimAttr::threadXDim(&context));
+}
+
+TEST_F(OpenACCUtilsReductionTest,
+       getReductionCombineParDimsFromCombineRegionViaAccumulate) {
+  MemRefType memTy = MemRefType::get({}, b.getI32Type());
+  auto dest = memref::AllocaOp::create(b, loc, memTy);
+  auto src = memref::AllocaOp::create(b, loc, memTy);
+  auto combineRegion = ReductionCombineRegionOp::create(b, loc, dest, src);
+  combineRegion.getRegion().emplaceBlock();
+  b.setInsertionPointToStart(&combineRegion.getRegion().front());
+  YieldOp::create(b, loc);
+
+  GPUParallelDimsAttr accDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::threadYDim(&context)});
+  Value partial = arith::ConstantIntOp::create(b, loc, b.getI32Type(), 1);
+  ReductionAccumulateOp::create(b, loc, partial, src.getResult(),
+                                ReductionOperator::AccAdd, accDims);
+
+  SmallVector<GPUParallelDimAttr> result =
+      getReductionCombineParDims(combineRegion);
+  ASSERT_EQ(result.size(), 1u);
+  EXPECT_EQ(result[0], GPUParallelDimAttr::threadYDim(&context));
+}
+
+TEST_F(OpenACCUtilsReductionTest,
+       getReductionCombineParDimsFromCombineRegionAttribute) {
+  MemRefType memTy = MemRefType::get({}, b.getI32Type());
+  auto dest = memref::AllocaOp::create(b, loc, memTy);
+  auto src = memref::AllocaOp::create(b, loc, memTy);
+  auto combineRegion = ReductionCombineRegionOp::create(b, loc, dest, src);
+  GPUParallelDimsAttr parDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockZDim(&context)});
+  setParDimsAttr(combineRegion, parDims);
+  combineRegion.getRegion().emplaceBlock();
+  b.setInsertionPointToStart(&combineRegion.getRegion().front());
+  YieldOp::create(b, loc);
+
+  SmallVector<GPUParallelDimAttr> result =
+      getReductionCombineParDims(combineRegion);
+  ASSERT_EQ(result.size(), 1u);
+  EXPECT_EQ(result[0], GPUParallelDimAttr::blockZDim(&context));
+}
+
+//===----------------------------------------------------------------------===//
+// createIdentityValue / generateReductionOp Tests
+//===----------------------------------------------------------------------===//
+
+TEST_F(OpenACCUtilsReductionTest, createIdentityValueIntegerAdd) {
+  Value ident =
+      createIdentityValue(b, loc, b.getI32Type(), arith::AtomicRMWKind::addi);
+  auto cst = ident.getDefiningOp<arith::ConstantOp>();
+  ASSERT_TRUE(cst);
+  EXPECT_EQ(cast<IntegerAttr>(cst.getValue()).getInt(), 0);
+}
+
+TEST_F(OpenACCUtilsReductionTest, createIdentityValueFloatMul) {
+  Value ident =
+      createIdentityValue(b, loc, b.getF32Type(), arith::AtomicRMWKind::mulf);
+  auto cst = ident.getDefiningOp<arith::ConstantOp>();
+  ASSERT_TRUE(cst);
+  EXPECT_EQ(cast<FloatAttr>(cst.getValue()).getValueAsDouble(), 1.0);
+}
+
+TEST_F(OpenACCUtilsReductionTest, createIdentityValueComplexAdd) {
+  auto complexTy = ComplexType::get(b.getF32Type());
+  Value ident =
+      createIdentityValue(b, loc, complexTy, arith::AtomicRMWKind::addf);
+  auto createOp = ident.getDefiningOp<complex::CreateOp>();
+  ASSERT_TRUE(createOp);
+  auto realCst = createOp.getReal().getDefiningOp<arith::ConstantOp>();
+  auto imagCst = createOp.getImaginary().getDefiningOp<arith::ConstantOp>();
+  ASSERT_TRUE(realCst);
+  ASSERT_TRUE(imagCst);
+  EXPECT_EQ(cast<FloatAttr>(realCst.getValue()).getValueAsDouble(), 0.0);
+  EXPECT_EQ(cast<FloatAttr>(imagCst.getValue()).getValueAsDouble(), 0.0);
+}
+
+TEST_F(OpenACCUtilsReductionTest, generateReductionOpIntegerAdd) {
+  Value lhs = arith::ConstantIntOp::create(b, loc, b.getI32Type(), 3);
+  Value rhs = arith::ConstantIntOp::create(b, loc, b.getI32Type(), 5);
+  Value sum = generateReductionOp(b, loc, lhs, rhs, arith::AtomicRMWKind::addi);
+  EXPECT_TRUE(isa<arith::AddIOp>(sum.getDefiningOp()));
+}
+
+TEST_F(OpenACCUtilsReductionTest, generateReductionOpComplexMul) {
+  auto complexTy = ComplexType::get(b.getF32Type());
+  Value lhs = complex::CreateOp::create(
+      b, loc, complexTy,
+      arith::ConstantOp::create(b, loc, b.getF32FloatAttr(2.0)),
+      arith::ConstantOp::create(b, loc, b.getF32FloatAttr(1.0)));
+  Value rhs = complex::CreateOp::create(
+      b, loc, complexTy,
+      arith::ConstantOp::create(b, loc, b.getF32FloatAttr(3.0)),
+      arith::ConstantOp::create(b, loc, b.getF32FloatAttr(0.0)));
+  Value product =
+      generateReductionOp(b, loc, lhs, rhs, arith::AtomicRMWKind::mulf);
+  EXPECT_TRUE(isa<complex::MulOp>(product.getDefiningOp()));
+}


        

From llvmlistbot at llvm.org  Mon Jul 13 14:55:13 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 13 Jul 2026 14:55:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add reduction utilities for acc
 to gpu lowering (PR #209316)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209316@github.com>
Message-ID: <6a555ec1.9edb5c9d.208698.af33@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/209316

From llvmlistbot at llvm.org  Mon Jul 13 16:46:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 13 Jul 2026 16:46:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 96ad6fc - [mlir][acc] Add inline
	implementation for emitNYI (#209337)
Message-ID: <6a5578ba.92d2873c.258297.9c04@mx.google.com>


Author: Razvan Lupusoru
Date: 2026-07-13T16:45:57-07:00
New Revision: 96ad6fc9e86d7cb2dbd0ad1646b40219fd6a8fda

URL: https://github.com/llvm/llvm-project/commit/96ad6fc9e86d7cb2dbd0ad1646b40219fd6a8fda
DIFF: https://github.com/llvm/llvm-project/commit/96ad6fc9e86d7cb2dbd0ad1646b40219fd6a8fda.diff

LOG: [mlir][acc] Add inline implementation for emitNYI (#209337)

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
    mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
index 03bb6fb81d1ed..b004a5f83e380 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
+++ b/mlir/include/mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h
@@ -274,7 +274,11 @@ class OpenACCSupport {
   /// \param message The message to report.
   /// \return An in-flight diagnostic object that can be used to report the
   ///         unsupported case.
-  InFlightDiagnostic emitNYI(Location loc, const Twine &message);
+  InFlightDiagnostic emitNYI(Location loc, const Twine &message) {
+    if (impl)
+      return impl->emitNYI(loc, message);
+    return mlir::emitError(loc, "not yet implemented: " + message);
+  }
 
   /// Emit an OpenACC remark with lazy message generation.
   ///

diff  --git a/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp b/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
index 0f2f9d02ef2b3..be97b2bb2fbec 100644
--- a/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
+++ b/mlir/lib/Dialect/OpenACC/Analysis/OpenACCSupport.cpp
@@ -37,12 +37,6 @@ std::string OpenACCSupport::getRecipeName(RecipeKind kind, Type type,
   return recipeName;
 }
 
-InFlightDiagnostic OpenACCSupport::emitNYI(Location loc, const Twine &message) {
-  if (impl)
-    return impl->emitNYI(loc, message);
-  return mlir::emitError(loc, "not yet implemented: " + message);
-}
-
 remark::detail::InFlightRemark
 OpenACCSupport::emitRemark(Operation *op,
                            std::function<std::string()> messageFn,


        

From llvmlistbot at llvm.org  Mon Jul 13 16:46:04 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 13 Jul 2026 16:46:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Add inline implementation for
	emitNYI (PR #209337)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209337@github.com>
Message-ID: <6a5578bc.824e11a6.f5f27.544f@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/209337

From llvmlistbot at llvm.org  Mon Jul 13 16:51:30 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 13 Jul 2026 16:51:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Fix leaks in cg unit tests (PR
	#209339)
Message-ID: <llvm.org/llvm/llvm-project/pull/209339@github.com>


https://github.com/razvanlupusoru created https://github.com/llvm/llvm-project/pull/209339

Newly added tests are failing valgrind --leak-check=full: https://github.com/llvm/llvm-project/pull/209313
https://github.com/llvm/llvm-project/pull/209316

This PR resolves the problem by ensuring that the insertion point is set into an operation that is owned and deleted by the test.

>From 3e6edffc2c5bd9629ce1c574bce325a377215745 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Mon, 13 Jul 2026 16:49:23 -0700
Subject: [PATCH] [mlir][acc] Fix leaks in cg unit tests

Newly added tests are failing valgrind --leak-check=full:
https://github.com/llvm/llvm-project/pull/209313
https://github.com/llvm/llvm-project/pull/209316

This PR resolves the problem by ensuring that the insertion
point is set into an operation that is owned and deleted by the
test.
---
 mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp      | 7 +++++++
 mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp     | 6 ++++++
 .../Dialect/OpenACC/OpenACCUtilsReductionTest.cpp          | 6 ++++++
 3 files changed, 19 insertions(+)

diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 6de57c9801f98..66e500c4d2100 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -458,6 +458,7 @@ TEST_F(OpenACCUtilsCGTest, getPrivatizeOpFromHandle) {
 
 TEST_F(OpenACCUtilsCGTest, getPrivatizeOpFromComputeRegionBlockArg) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::blockXDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -540,6 +541,7 @@ TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromLaunchFallback) {
   // With no enclosing parallel loops, collectPrivateLocalParDims falls back to
   // the block-level launch dimensions.
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::blockXDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -678,6 +680,7 @@ TEST_F(OpenACCUtilsCGTest, collectPrivateLocalParDimsFromReductionUsers) {
 
 TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateGangPrivate) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::blockXDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -699,6 +702,7 @@ TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateGangPrivate) {
 
 TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateThreadXPrivate) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr vectorDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::threadXDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -721,6 +725,7 @@ TEST_F(OpenACCUtilsCGTest, isPrivateLocalSharedMemoryCandidateThreadXPrivate) {
 TEST_F(OpenACCUtilsCGTest,
        isPrivateLocalSharedMemoryCandidateWorkerPrivateConstant) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::threadYDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -746,6 +751,7 @@ TEST_F(OpenACCUtilsCGTest,
 TEST_F(OpenACCUtilsCGTest,
        isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::threadYDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
@@ -777,6 +783,7 @@ TEST_F(OpenACCUtilsCGTest,
 
 TEST_F(OpenACCUtilsCGTest, getPrivateLocalSharedMemoryUpperBoundBytes) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr gangDims = GPUParallelDimsAttr::get(
       &context, {GPUParallelDimAttr::blockXDim(&context)});
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
index 49090f4d657d1..e7ee0655d41f0 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsGPUTest.cpp
@@ -27,9 +27,15 @@ class OpenACCUtilsGPUTest : public ::testing::Test {
     context.loadDialect<arith::ArithDialect, gpu::GPUDialect>();
   }
 
+  void SetUp() override {
+    module = ModuleOp::create(b, loc);
+    b.setInsertionPointToStart(module->getBody());
+  }
+
   MLIRContext context;
   OpBuilder b;
   Location loc;
+  OwningOpRef<ModuleOp> module;
 };
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp
index 566f905ff24d5..690b174b5ba03 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsReductionTest.cpp
@@ -32,9 +32,15 @@ class OpenACCUtilsReductionTest : public ::testing::Test {
                         complex::ComplexDialect, memref::MemRefDialect>();
   }
 
+  void SetUp() override {
+    module = ModuleOp::create(b, loc);
+    b.setInsertionPointToStart(module->getBody());
+  }
+
   MLIRContext context;
   OpBuilder b;
   Location loc;
+  OwningOpRef<ModuleOp> module;
 };
 
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Mon Jul 13 19:06:13 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Mon, 13 Jul 2026 19:06:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a559995.07d7ca98.9dee3.3b8c@mx.google.com>


https://github.com/jacquesguan updated https://github.com/llvm/llvm-project/pull/208648

>From 8211ec080fa7d8436e533a33caecdc365c6e9d63 Mon Sep 17 00:00:00 2001
From: Jianjian GUAN <jacquesguan at me.com>
Date: Fri, 10 Jul 2026 14:59:57 +0800
Subject: [PATCH 1/2] [mlir][emitc] Add increment and decrement ops

---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td | 45 +++++++++++++++++++
 mlir/lib/Target/Cpp/TranslateToCpp.cpp      | 48 ++++++++++++++++++++-
 mlir/test/Dialect/EmitC/invalid_ops.mlir    | 16 +++++++
 mlir/test/Dialect/EmitC/ops.mlir            |  9 ++++
 mlir/test/Target/Cpp/common-cpp.mlir        | 35 +++++++++++++++
 5 files changed, 152 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..c52f5b5f909c4 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -65,6 +65,39 @@ class EmitC_BinaryOp<string mnemonic, list<Trait> traits = []> :
   }];
 }
 
+// Base class for increment and decrement operations.
+class EmitC_IncDecOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, [
+        CExpressionInterface,
+        TypesMatchWith<"input and result reference the same type",
+                       "operand", "result",
+                       "::llvm::cast<emitc::LValueType>($_self).getValueType()">
+      ]> {
+  let summary = summaryStr;
+  let description = [{
+    This operation models a C/C++ increment or decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+
+  let arguments = (ins
+    Res<EmitC_LValueType, "",
+        [MemRead<DefaultResource, 0, FullEffect>,
+         MemWrite<DefaultResource, 0, FullEffect>]>:$operand);
+  let results = (outs EmitCType:$result);
+  let assemblyFormat = "$operand attr-dict `:` type($operand)";
+
+  let extraClassDeclaration = [{
+    bool hasSideEffects() {
+      return true;
+    }
+  }];
+}
+
 // Types only used in binary arithmetic operations.
 def IntegerIndexOrOpaqueType : Type<CPred<"emitc::isIntegerIndexOrOpaqueType($_self)">,
 "integer, index or opaque type supported by EmitC">;
@@ -1259,6 +1292,18 @@ def EmitC_ConditionalOp : EmitC_Op<"conditional",
   }];
 }
 
+def EmitC_PreIncrementOp
+    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation">;
+
+def EmitC_PostIncrementOp
+    : EmitC_IncDecOp<"post_increment", "Post-increment operation">;
+
+def EmitC_PreDecrementOp
+    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation">;
+
+def EmitC_PostDecrementOp
+    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation">;
+
 def EmitC_UnaryMinusOp : EmitC_UnaryOp<"unary_minus", []> {
   let summary = "Unary minus operation";
   let description = [{
diff --git a/mlir/lib/Target/Cpp/TranslateToCpp.cpp b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
index 1bf87e4994d67..5ae51bd580d31 100644
--- a/mlir/lib/Target/Cpp/TranslateToCpp.cpp
+++ b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
@@ -112,6 +112,10 @@ static FailureOr<int> getOperatorPrecedence(Operation *operation) {
       .Case([&](emitc::MemberOfPtrOp op) { return 17; })
       .Case([&](emitc::MemberOp op) { return 17; })
       .Case([&](emitc::MulOp op) { return 13; })
+      .Case([&](emitc::PostDecrementOp op) { return 16; })
+      .Case([&](emitc::PostIncrementOp op) { return 16; })
+      .Case([&](emitc::PreDecrementOp op) { return 15; })
+      .Case([&](emitc::PreIncrementOp op) { return 15; })
       .Case([&](emitc::RemOp op) { return 13; })
       .Case([&](emitc::SubOp op) { return 12; })
       .Case([&](emitc::SubscriptOp op) { return 17; })
@@ -646,6 +650,22 @@ static LogicalResult printUnaryOperation(CppEmitter &emitter,
   return success();
 }
 
+static LogicalResult printPostfixUnaryOperation(CppEmitter &emitter,
+                                                Operation *operation,
+                                                StringRef unaryOperator) {
+  raw_ostream &os = emitter.ostream();
+
+  if (failed(emitter.emitAssignPrefix(*operation)))
+    return failure();
+
+  if (failed(emitter.emitOperand(operation->getOperand(0))))
+    return failure();
+
+  os << unaryOperator;
+
+  return success();
+}
+
 static LogicalResult printOperation(CppEmitter &emitter, emitc::AddOp addOp) {
   Operation *operation = addOp.getOperation();
 
@@ -1038,6 +1058,30 @@ static LogicalResult printOperation(CppEmitter &emitter,
   return printBinaryOperation(emitter, operation, "^");
 }
 
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreIncrementOp preIncrementOp) {
+  Operation *operation = preIncrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostIncrementOp postIncrementOp) {
+  Operation *operation = postIncrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreDecrementOp preDecrementOp) {
+  Operation *operation = preDecrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "--");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostDecrementOp postDecrementOp) {
+  Operation *operation = postDecrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "--");
+}
+
 static LogicalResult printOperation(CppEmitter &emitter,
                                     emitc::UnaryPlusOp unaryPlusOp) {
   Operation *operation = unaryPlusOp.getOperation();
@@ -1896,7 +1940,9 @@ LogicalResult CppEmitter::emitOperation(Operation &op, bool trailingSemicolon) {
                 emitc::IncludeOp, emitc::LiteralOp, emitc::LoadOp,
                 emitc::LogicalAndOp, emitc::LogicalNotOp, emitc::LogicalOrOp,
                 emitc::MemberCallOpaqueOp, emitc::MemberOfPtrOp,
-                emitc::MemberOp, emitc::MulOp, emitc::RemOp, emitc::ReturnOp,
+                emitc::MemberOp, emitc::MulOp, emitc::PostDecrementOp,
+                emitc::PostIncrementOp, emitc::PreDecrementOp,
+                emitc::PreIncrementOp, emitc::RemOp, emitc::ReturnOp,
                 emitc::SubscriptOp, emitc::SubOp, emitc::SwitchOp,
                 emitc::UnaryMinusOp, emitc::UnaryPlusOp, emitc::VariableOp,
                 emitc::VerbatimOp>(
diff --git a/mlir/test/Dialect/EmitC/invalid_ops.mlir b/mlir/test/Dialect/EmitC/invalid_ops.mlir
index eacf4fdedcbd3..3048160d1590c 100644
--- a/mlir/test/Dialect/EmitC/invalid_ops.mlir
+++ b/mlir/test/Dialect/EmitC/invalid_ops.mlir
@@ -981,3 +981,19 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   %1 = "emitc.dereference"(%arg0) : (!emitc.ptr<i32>) -> !emitc.lvalue<i8>
   return
 }
+
+// -----
+
+func.func @pre_increment_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.pre_increment"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
+
+// -----
+
+func.func @post_decrement_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.post_decrement"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
diff --git a/mlir/test/Dialect/EmitC/ops.mlir b/mlir/test/Dialect/EmitC/ops.mlir
index e456db4634892..e5c4af67c4896 100644
--- a/mlir/test/Dialect/EmitC/ops.mlir
+++ b/mlir/test/Dialect/EmitC/ops.mlir
@@ -148,6 +148,15 @@ func.func @unary(%arg0: i32) {
   return
 }
 
+func.func @inc_dec(%arg0: !emitc.ptr<i32>) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
 func.func @test_if(%arg0: i1, %arg1: f32) {
   emitc.if %arg0 {
      %0 = emitc.call_opaque "func_const"(%arg1) : (f32) -> i32
diff --git a/mlir/test/Target/Cpp/common-cpp.mlir b/mlir/test/Target/Cpp/common-cpp.mlir
index 8e84433eaa2f3..1bffd59d4d44c 100644
--- a/mlir/test/Target/Cpp/common-cpp.mlir
+++ b/mlir/test/Target/Cpp/common-cpp.mlir
@@ -107,6 +107,41 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   return
 }
 
+// CHECK-LABEL: void inc_dec() {
+func.func @inc_dec() {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = ++[[V1]];
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]++;
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = --[[V1]];
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V5:[^ ]*]] = [[V1]]--;
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
+// CHECK-LABEL: int32_t inc_dec_expression() {
+func.func @inc_dec_expression() -> i32 {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %inc = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_increment %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = [[V1]]++;
+  %dec = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_decrement %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]--;
+  %r = emitc.add %inc, %dec : (i32, i32) -> i32
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = [[V2]] + [[V3]];
+  // CHECK-NEXT: return [[V4]];
+  return %r : i32
+}
+
 // CHECK: void array_type(int32_t v1[3], float v2[10][20])
 func.func @array_type(%arg0: !emitc.array<3xi32>, %arg1: !emitc.array<10x20xf32>) {
   return

>From b87881459866f405e1fa87bae1d024349ed816e3 Mon Sep 17 00:00:00 2001
From: Jianjian GUAN <jacquesguan at me.com>
Date: Tue, 14 Jul 2026 10:05:40 +0800
Subject: [PATCH 2/2] Address comment

---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td | 58 ++++++++++++++++-----
 1 file changed, 44 insertions(+), 14 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index c52f5b5f909c4..c8417ce2f3de7 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -74,16 +74,6 @@ class EmitC_IncDecOp<string mnemonic, string summaryStr>
                        "::llvm::cast<emitc::LValueType>($_self).getValueType()">
       ]> {
   let summary = summaryStr;
-  let description = [{
-    This operation models a C/C++ increment or decrement operator on an lvalue.
-
-    Example:
-
-    ```mlir
-    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
-    ```
-  }];
-
   let arguments = (ins
     Res<EmitC_LValueType, "",
         [MemRead<DefaultResource, 0, FullEffect>,
@@ -1293,16 +1283,56 @@ def EmitC_ConditionalOp : EmitC_Op<"conditional",
 }
 
 def EmitC_PreIncrementOp
-    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation">;
+    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation"> {
+  let description = [{
+    This operation models the C/C++ pre-increment operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
 
 def EmitC_PostIncrementOp
-    : EmitC_IncDecOp<"post_increment", "Post-increment operation">;
+    : EmitC_IncDecOp<"post_increment", "Post-increment operation"> {
+  let description = [{
+    This operation models the C/C++ post-increment operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.post_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
 
 def EmitC_PreDecrementOp
-    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation">;
+    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation"> {
+  let description = [{
+    This operation models the C/C++ pre-decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_decrement %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
 
 def EmitC_PostDecrementOp
-    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation">;
+    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation"> {
+  let description = [{
+    This operation models the C/C++ post-decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.post_decrement %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
 
 def EmitC_UnaryMinusOp : EmitC_UnaryOp<"unary_minus", []> {
   let summary = "Unary minus operation";


From llvmlistbot at llvm.org  Mon Jul 13 19:10:36 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Mon, 13 Jul 2026 19:10:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a559a9c.a1accbb4.244acd.c8b1@mx.google.com>


================
@@ -65,6 +65,39 @@ class EmitC_BinaryOp<string mnemonic, list<Trait> traits = []> :
   }];
 }
 
+// Base class for increment and decrement operations.
+class EmitC_IncDecOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, [
+        CExpressionInterface,
+        TypesMatchWith<"input and result reference the same type",
+                       "operand", "result",
+                       "::llvm::cast<emitc::LValueType>($_self).getValueType()">
+      ]> {
+  let summary = summaryStr;
+  let description = [{
+    This operation models a C/C++ increment or decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
----------------
jacquesguan wrote:

Addressed.

https://github.com/llvm/llvm-project/pull/208648

From llvmlistbot at llvm.org  Mon Jul 13 21:00:46 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Mon, 13 Jul 2026 21:00:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55b46e.c56fb102.7c9cd.4863@mx.google.com>


================
@@ -61,21 +61,21 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
 }
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
----------------
JerryShih wrote:

This one is the existed test case.
I think the name here is that there is only a single inner tile and the tiled dim is at first(0) dim.
I think the name `negative_packing_inner_dim` is so generic which can't show more info about this case.

The expand_shape op can't handle the following case which needs transpose.
```
tensor<256x5xf32> -> tensor<8x5x32xf32>
```

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 00:07:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 00:07:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 69ee6b1 - [NVVM] Add FP8 (e4m3/e5m2) support
 to dense mma.sync (#207307)
Message-ID: <6a55e017.bd93e731.358318.5447@mx.google.com>


Author: WMC
Date: 2026-07-14T09:06:57+02:00
New Revision: 69ee6b114ac3d3c6652b39d0d57b292bcfa3a81c

URL: https://github.com/llvm/llvm-project/commit/69ee6b114ac3d3c6652b39d0d57b292bcfa3a81c
DIFF: https://github.com/llvm/llvm-project/commit/69ee6b114ac3d3c6652b39d0d57b292bcfa3a81c.diff

LOG: [NVVM] Add FP8 (e4m3/e5m2) support to dense mma.sync (#207307)

## Motivation
I'm adding NVIDIA FP8 mma.sync support in IREE for sm_89/sm_120, and
IREE's codegen lowers through `nvgpu.mma.sync` on this path, which in
turn lowers to `nvvm.mma.sync`. This MLIR-level gap is blocking that
work, so I'm upstreaming the fix here first. This is split into two PRs
per review feedback — this one covers `nvvm.mma.sync` only; a follow-up
PR will add the corresponding `nvgpu.mma.sync` support built on top of
this.

## Description:

`nvvm.mma.sync` (warp-level, dense) don't support FP8 (e4m3/e5m2)
operands, even though the underlying PTX/NVPTX intrinsics have supported
FP8 `mma.sync.m16n8k32` since sm_89. FP8 currently only exists on the
sparse mma path and on warpgroup-level `wgmma`; the warp-level dense
path was never updated.

This adds FP8 (e4m3/e5m2) support to `nvvm.mma.sync`, covering
`m16n8k16` and `m16n8k32`.

## Verification
`$ mlir-opt mlir/test/Dialect/LLVMIR/nvvm.mlir -split-input-file
-verify-diagnostics | FileCheck mlir/test/Dialect/LLVMIR/nvvm.mlir
`

```
%0 = nvvm.mma.sync A[...] B[...] C[...]
     {multiplicandAPtxType = #nvvm.mma_type<e4m3>,
      multiplicandBPtxType = #nvvm.mma_type<e4m3>,
      shape = #nvvm.shape<m = 16, n = 8, k = 32>}
     : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>

```

`$ mlir-translate -mlir-to-llvmir mlir/test/Target/LLVMIR/nvvmir.mlir
-split-input-file -verify-diagnostics | FileCheck
mlir/test/Target/LLVMIR/nvvmir.mlir
`

```
%8 = call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32(i32 %0, i32 %1, i32 %2, float %3, float %4, float %5, float %6)
%11 = call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32(i32 %0, i32 %1, i32 %2, i32 %3, i32 %4, i32 %5, float %6, float %7, float %8, float %9)

```

All existing tests in nvvm.mlir (including -verify-diagnostics error
cases) still pass — no regression.

Co-authored with Claude Sonnet 5

Signed-off-by: weimin023 <tnwilly at gmail.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
    mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
    mlir/test/Dialect/LLVMIR/nvvm.mlir
    mlir/test/Target/LLVMIR/nvvmir.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 6dd32615451a7..1f76218936e16 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -2705,9 +2705,13 @@ class NVVM_MMA_OPS {
   list<list<WMMA_REGS>> bit_mma_ops = MMA_OPS<
             [GEOM<8,8,128>, GEOM<16,8,128>, GEOM<16,8,256>],
             ["b1"], [], ["s32"], []>.ret;
+  list<list<WMMA_REGS>> fp8_mma_ops = MMA_OPS<
+            [GEOM<16,8,16>, GEOM<16,8,32>],
+            ["e4m3", "e5m2"], ["e4m3", "e5m2"], ["f16", "f32"], []>.ret;
   list<list<WMMA_REGS>> all_mma_sync_ops = !listconcat(
             tf32_mma_ops, bf16_mma_ops, f64_mma_ops,
-            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops);
+            fp_mma_ops, int_mma_ops, subint_mma_ops, bit_mma_ops,
+            fp8_mma_ops);
 
   list<list<WMMA_REGS>> bf16_mma_sp_ops = MMA_OPS<
             [GEOM<16,8,16>, GEOM<16,8,32>],

diff  --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index 14406ab55f138..b29cba96a410f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -1007,6 +1007,15 @@ LogicalResult MmaOp::verify() {
       expectedResult.push_back(f16x2x2StructTy);
       expectedResult.push_back(f32x4StructTy);
       break;
+    case MMATypes::e4m3:
+    case MMATypes::e5m2:
+      // FP8 (m16n8k16 / m16n8k32) packs 4 values per 32-bit register, same
+      // as s8/u8, but the accumulator is f16 or f32 (not integer).
+      kFactor = 16;
+      multiplicandFragType = i32Ty;
+      expectedResult.push_back(f16x2x2StructTy);
+      expectedResult.push_back(f32x4StructTy);
+      break;
     case MMATypes::s4:
     case MMATypes::u4:
       kFactor = 32;

diff  --git a/mlir/test/Dialect/LLVMIR/nvvm.mlir b/mlir/test/Dialect/LLVMIR/nvvm.mlir
index 72935ffd7b7ce..ae3d18ec46812 100644
--- a/mlir/test/Dialect/LLVMIR/nvvm.mlir
+++ b/mlir/test/Dialect/LLVMIR/nvvm.mlir
@@ -210,6 +210,54 @@ func.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3_f16
+func.func @nvvm_mma_m16n8k16_e4m3_e4m3_f16(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>, shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2_f16
+func.func @nvvm_mma_m16n8k32_e4m3_e5m2_f16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: nvvm.mma.sync A[{{.*}}] B[{{.*}}] C[{{.*}}] {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>, multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>, shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
 // CHECK-LABEL: @nvvm_mma_m8n8k16_s8_s8
 func.func @nvvm_mma_m8n8k16_s8_s8(%a0 : i32, %b0 : i32,
                              %c0 : i32, %c1 : i32) -> !llvm.struct<(i32, i32)> {

diff  --git a/mlir/test/Target/LLVMIR/nvvmir.mlir b/mlir/test/Target/LLVMIR/nvvmir.mlir
index f2888025d8a08..778a5c2229225 100644
--- a/mlir/test/Target/LLVMIR/nvvmir.mlir
+++ b/mlir/test/Target/LLVMIR/nvvmir.mlir
@@ -293,6 +293,54 @@ llvm.func @nvvm_mma_m16n8k16_bf16_bf16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i3
   llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
 }
 
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k16.row.col.f32.e4m3.e4m3.f32
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : f32, %c1 : f32, %c2 : f32, %c3 : f32) -> !llvm.struct<(f32, f32, f32, f32)> {
+  // CHECK: call { float, float, float, float } @llvm.nvvm.mma.m16n8k32.row.col.f32.e4m3.e5m2.f32
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1, %c2, %c3]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, f32) -> !llvm.struct<(f32, f32, f32, f32)>
+  llvm.return %0 : !llvm.struct<(f32, f32, f32, f32)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k16_e4m3_e4m3_f16
+llvm.func @nvvm_mma_m16n8k16_e4m3_e4m3_f16(%a0 : i32, %a1 : i32,
+                              %b0 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: call { <2 x half>, <2 x half> } @llvm.nvvm.mma.m16n8k16.row.col.f16.e4m3.e4m3.f16
+  %0 = nvvm.mma.sync A[%a0, %a1] B[%b0] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e4m3>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 16>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
+// CHECK-LABEL: @nvvm_mma_m16n8k32_e4m3_e5m2_f16
+llvm.func @nvvm_mma_m16n8k32_e4m3_e5m2_f16(%a0 : i32, %a1 : i32, %a2 : i32, %a3 : i32,
+                              %b0 : i32, %b1 : i32,
+                              %c0 : vector<2xf16>, %c1 : vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)> {
+  // CHECK: call { <2 x half>, <2 x half> } @llvm.nvvm.mma.m16n8k32.row.col.f16.e4m3.e5m2.f16
+  %0 = nvvm.mma.sync A[%a0, %a1, %a2, %a3] B[%b0, %b1] C[%c0, %c1]
+    {layoutA = #nvvm.mma_layout<row>, layoutB = #nvvm.mma_layout<col>,
+     multiplicandAPtxType = #nvvm.mma_type<e4m3>, multiplicandBPtxType = #nvvm.mma_type<e5m2>,
+     shape = #nvvm.shape<m = 16, n = 8, k = 32>} : (i32, i32, vector<2xf16>) -> !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+  llvm.return %0 : !llvm.struct<(vector<2xf16>, vector<2xf16>)>
+}
+
 // f32 return type, f32 accumulate type
 // CHECK-LABEL: @nvvm_mma_m16n8k16_f32_f32
 llvm.func @nvvm_mma_m16n8k16_f32_f32(%a0 : vector<2xf16>, %a1 : vector<2xf16>,


        

From llvmlistbot at llvm.org  Tue Jul 14 00:07:05 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Tue, 14 Jul 2026 00:07:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a55e019.6537ae93.2f1c1b.501e@mx.google.com>


https://github.com/AGindinson closed https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Tue Jul 14 00:07:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 00:07:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a55e029.4a342cbc.ca844.e884@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@weimin023 Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Tue Jul 14 00:49:30 2026
From: llvmlistbot at llvm.org (Reinhard Stahn)
Date: Tue, 14 Jul 2026 00:49:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToAffine] Raise scf.for to
	affine.for (PR #200851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200851@github.com>
Message-ID: <6a55ea0a.4987c784.63cbc.e201@mx.google.com>


rainij wrote:

Ping

https://github.com/llvm/llvm-project/pull/200851

From llvmlistbot at llvm.org  Tue Jul 14 01:09:46 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Tue, 14 Jul 2026 01:09:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55eeca.92ea0e2a.13fd37.84ca@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
----------------
JerryShih wrote:

ohh...
I intend to test the non-tiled dim with perm. I will rename this test with the prefix like `non-tiled-dim-xxx`.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 01:13:21 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Tue, 14 Jul 2026 01:13:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55efa1.d681cab9.333b2b.8a0a@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
----------------
JerryShih wrote:

It's valid for pack op def.
But this valid pack op can't be replaced by expand_shape.
This case need transpose which can't be covered by expand_shape.

4x4
=>
2x2_inner0x2x2_inner1
=>
with transpose
2x2x2_inner0x2_inner1

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 01:15:11 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Tue, 14 Jul 2026 01:15:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55f00f.92ea0e2a.13fd37.a5f2@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
----------------
JerryShih wrote:

I will update this test with `outer_dims_perm = [0, 1] inner_dims_pos = [1, 0]`

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 01:19:10 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Tue, 14 Jul 2026 01:19:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55f0fe.66c0679a.3aa103.fc11@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
----------------
JerryShih wrote:

do you mean that `all` pack and unpack tests in this file are grouped together?
like:
```
pack_test1
pack_test2
...
pack_testn
unpack_test1
unpack_test2
...
unpack_testn
```

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 01:19:27 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Tue, 14 Jul 2026 01:19:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a55f10f.b1416da1.78d1e.6902@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
----------------
JerryShih wrote:

if yes, i will do that in next commit.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Tue Jul 14 01:47:01 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Tue, 14 Jul 2026 01:47:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a55f785.492e23e4.14eda0.4ca3@mx.google.com>


https://github.com/IgWod edited https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Tue Jul 14 01:47:01 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Tue, 14 Jul 2026 01:47:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a55f785.3cce8983.225732.dd39@mx.google.com>


================

----------------
IgWod wrote:

Maybe merge with the case above? They both return `std::nullopt` and you could write a comment that captures both cases.

Also is it possible to add a test case?

https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Tue Jul 14 01:47:01 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Tue, 14 Jul 2026 01:47:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith][RFC] Add new strict FP handling
	in Arith (PR #209287)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209287@github.com>
Message-ID: <6a55f785.6365b89b.8575e.c5c6@mx.google.com>


https://github.com/IgWod commented:

A small comment on the SPIR-V part.

https://github.com/llvm/llvm-project/pull/209287

From llvmlistbot at llvm.org  Tue Jul 14 01:48:56 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Tue, 14 Jul 2026 01:48:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Collapse
 TanPattern/TanhPattern into DirectConversionPattern (PR #203830)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203830@github.com>
Message-ID: <6a55f7f8.f3151ff8.305c37.0ec3@mx.google.com>


https://github.com/IgWod approved this pull request.

Thanks!

https://github.com/llvm/llvm-project/pull/203830

From llvmlistbot at llvm.org  Tue Jul 14 01:49:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 01:49:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55f803.eb165e5c.1817a5.0b0d@mx.google.com>


================

----------------
mygitljf wrote:

okay

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 01:52:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 01:52:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55f8c7.f1c1923d.991d1.7e2a@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206249

>From 3663396b2155970b6a4982ec42a0108c749df095 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 20:57:43 +0000
Subject: [PATCH 1/8] [mlir] Fix arith tensor constants

---
 mlir/include/mlir/Conversion/Passes.td        |  2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.td   |  2 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |  3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |  1 +
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 29 +++++++--
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    | 10 +++
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  8 +++
 mlir/test/Dialect/Arith/expand-ops.mlir       | 63 +++++++++++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |  2 +
 9 files changed, 111 insertions(+), 9 deletions(-)
 create mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 550e6b853057d..23463342314b0 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
index 27e9146ec3606..2cee05914d6b5 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
@@ -13,7 +13,7 @@ include "mlir/Pass/PassBase.td"
 
 def ArithExpandOpsPass : Pass<"arith-expand"> {
   let summary = "Legalize Arith ops to be convertible to LLVM.";
-  let dependentDialects = ["vector::VectorDialect"];
+  let dependentDialects = ["tensor::TensorDialect", "vector::VectorDialect"];
   let options =
       [Option<"includeBf16", "include-bf16", "bool", /*default=*/"false",
               "Enable the BF16 expansion patterns">,
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..77e36b45a84af 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -736,7 +737,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect>();
+    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 0a0e25e18b47a..955fcc397dd37 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,4 +16,5 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
+  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..d9cfa6bc78dc6 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Arith/Transforms/Passes.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/Location.h"
@@ -34,6 +35,21 @@ static Value createConst(Location loc, Type type, int value,
   return arith::ConstantOp::create(rewriter, loc, attr);
 }
 
+/// Create an integer or index constant, using source as the dynamic shape
+/// source for ranked tensors whose splat cannot be represented as a dense
+/// attribute.
+static Value createConst(Location loc, Type type, int value,
+                         PatternRewriter &rewriter, Value source) {
+  auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
+    return createConst(loc, type, value, rewriter);
+
+  Value scalar =
+      createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
+  return tensor::SplatOp::create(rewriter, loc, scalar,
+                                 tensor::getMixedSizes(rewriter, loc, source));
+}
+
 /// Create an integer constant from an APInt.
 static Value createAPIntConst(Location loc, Type type, const APInt &value,
                               PatternRewriter &rewriter) {
@@ -76,10 +92,10 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter);
+    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
     Value compare =
         arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter);
+    Value one = createConst(loc, a.getType(), 1, rewriter, a);
     Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
     Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
@@ -104,8 +120,8 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter);
-    Value one = createConst(loc, type, 1, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
+    Value one = createConst(loc, type, 1, rewriter, a);
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -150,7 +166,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter);
+    Value zero = createConst(loc, type, 0, rewriter, a);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
                                        a, zero);
@@ -162,7 +178,7 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter);
+    Value minusOne = createConst(loc, type, -1, rewriter, a);
     Value quotientMinusOne =
         arith::AddIOp::create(rewriter, loc, quotient, minusOne);
 
@@ -826,6 +842,7 @@ struct ArithExpandOpsPass
     arith::populateArithExpandOpsPatterns(patterns);
 
     target.addLegalDialect<arith::ArithDialect>();
+    target.addLegalDialect<tensor::TensorDialect>();
     target.addLegalDialect<vector::VectorDialect>();
 
     // clang-format off
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
new file mode 100644
index 0000000000000..9d169bc60cade
--- /dev/null
+++ b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
@@ -0,0 +1,10 @@
+// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
+// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
+
+// CHECK-LABEL: @ceildivui_dynamic_tensor
+// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
+func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
+  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
+  return %0: tensor<8x4x?xi64>
+}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index fa204c4df8ace..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,14 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
 // -----
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 75c4de2168761..80b6592417cc2 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -144,6 +144,69 @@ func.func @ceildivui_index(%arg0: index, %arg1: index) -> (index) {
 
 // -----
 
+// CHECK-LABEL:   func.func @ceildivui_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x4x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+func.func @ceildivui_dynamic_tensor(%arg0: tensor<8x4x?xi64>, %arg1: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+  %res = arith.ceildivui %arg0, %arg1 : tensor<8x4x?xi64>
+  return %res : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C2_IDX:.*]] = arith.constant 2 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX]] : tensor<8x4x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[ISZERO:.*]] = arith.cmpi eq, %[[ARG0]], %[[ZERO]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C2_IDX_1:.*]] = arith.constant 2 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX_1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x4x?xi64>
+// CHECK:       %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divui %[[SUB]], %[[ARG1]] : tensor<8x4x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x4x?xi64>
+// CHECK:       %[[RES:.*]] = arith.select %[[ISZERO]], %[[ZERO]], %[[ADD]] : tensor<8x4x?xi1>, tensor<8x4x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @ceildivsi_dynamic_tensor(
+// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x?xi64>,
+// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x?xi64>) -> tensor<8x?xi64> {
+func.func @ceildivsi_dynamic_tensor(%arg0: tensor<8x?xi64>, %arg1: tensor<8x?xi64>) -> tensor<8x?xi64> {
+  %res = arith.ceildivsi %arg0, %arg1 : tensor<8x?xi64>
+  return %res : tensor<8x?xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C1_IDX:.*]] = arith.constant 1 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX]] : tensor<8x?xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
+// CHECK:       %[[C1_IDX_1:.*]] = arith.constant 1 : index
+// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX_1]] : tensor<8x?xi64>
+// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x?xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<8x?xi64>
+// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x?xi64>
+}
+
+// -----
+
+// CHECK-LABEL:   func.func @floordivsi_dynamic_tensor(
+// CHECK-SAME:                                       %[[ARG0:.*]]: tensor<?x4xi64>,
+// CHECK-SAME:                                       %[[ARG1:.*]]: tensor<?x4xi64>) -> tensor<?x4xi64> {
+func.func @floordivsi_dynamic_tensor(%arg0: tensor<?x4xi64>, %arg1: tensor<?x4xi64>) -> tensor<?x4xi64> {
+  %res = arith.floordivsi %arg0, %arg1 : tensor<?x4xi64>
+  return %res : tensor<?x4xi64>
+// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
+// CHECK:       %[[C0_IDX:.*]] = arith.constant 0 : index
+// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX]] : tensor<?x4xi64>
+// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE_SCALAR:.*]] = arith.constant -1 : i64
+// CHECK:       %[[C0_IDX_1:.*]] = arith.constant 0 : index
+// CHECK:       %[[NEG_ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX_1]] : tensor<?x4xi64>
+// CHECK:       %[[NEG_ONE:.*]] = tensor.splat %[[NEG_ONE_SCALAR]]{{\[}}%[[NEG_ONE_DIM]]] : tensor<?x4xi64>
+// CHECK:       %[[SUB_ONE:.*]] = arith.addi %[[DIV]], %[[NEG_ONE]] : tensor<?x4xi64>
+}
+
+// -----
+
 // CHECK-LABEL: func @maximumf
 func.func @maximumf(%a: f32, %b: f32) -> f32 {
   %result = arith.maximumf %a, %b : f32
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 77863de1c79a7c5ae3d6f1b4cc8f62f9caf98498 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 20:47:24 +0000
Subject: [PATCH 2/8] [mlir] Refine tensor splat expansion

---
 mlir/include/mlir/Conversion/Passes.td        |   2 +-
 .../mlir/Dialect/Arith/Transforms/Passes.h    |   3 +-
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |   3 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |   1 -
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 117 ++++++++++++++----
 .../ArithToLLVM/ceil-floor-div-tensor.mlir    |  10 --
 .../test/Conversion/ConvertToSPIRV/arith.mlir |  10 --
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp |   2 -
 8 files changed, 96 insertions(+), 52 deletions(-)
 delete mode 100644 mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir

diff --git a/mlir/include/mlir/Conversion/Passes.td b/mlir/include/mlir/Conversion/Passes.td
index 23463342314b0..550e6b853057d 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -180,7 +180,7 @@ def ArithToLLVMConversionPass : Pass<"convert-arith-to-llvm"> {
   let description = [{
     This pass converts supported Arith ops to LLVM dialect instructions.
   }];
-  let dependentDialects = ["LLVM::LLVMDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["LLVM::LLVMDialect"];
   let options = [
     Option<"indexBitwidth", "index-bitwidth", "unsigned",
            /*default=kDeriveIndexBitwidthFromDataLayout*/"0",
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
index 5a07a01d0928a..11081342c2244 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
@@ -54,7 +54,8 @@ void populateEmulateUnsupportedFloatsPatterns(RewritePatternSet &patterns,
 void populateEmulateUnsupportedFloatsLegality(ConversionTarget &target,
                                               const TypeConverter &converter);
 /// Add patterns to expand Arith ceil/floor division ops.
-void populateCeilFloorDivExpandOpsPatterns(RewritePatternSet &patterns);
+void populateCeilFloorDivExpandOpsPatterns(
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat = false);
 
 /// Add patterns to expand Arith bf16 patterns to lower level bitcasts/shifts.
 void populateExpandBFloat16Patterns(RewritePatternSet &patterns);
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 77e36b45a84af..6bf0fe85bb62f 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -17,7 +17,6 @@
 #include "mlir/Dialect/LLVMIR/FunctionCallUtils.h"
 #include "mlir/Dialect/LLVMIR/LLVMAttrs.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/TypeUtilities.h"
 #include <type_traits>
 
@@ -737,7 +736,7 @@ struct ArithToLLVMDialectInterface : public ConvertToLLVMPatternInterface {
       : ConvertToLLVMPatternInterface(dialect) {}
 
   void loadDependentDialects(MLIRContext *context) const final {
-    context->loadDialect<LLVM::LLVMDialect, tensor::TensorDialect>();
+    context->loadDialect<LLVM::LLVMDialect>();
   }
 
   /// Hook for derived dialect interface to provide conversion patterns
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 955fcc397dd37..0a0e25e18b47a 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -16,5 +16,4 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithTransforms
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
-  MLIRTensorDialect
   )
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index d9cfa6bc78dc6..b8c0e39dfddda 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -24,6 +24,10 @@ namespace arith {
 
 using namespace mlir;
 
+namespace {
+
+enum class DynamicTensorConstantMaterialization { Disabled, TensorSplat };
+
 /// Create an integer or index constant.
 static Value createConst(Location loc, Type type, int value,
                          PatternRewriter &rewriter) {
@@ -38,16 +42,27 @@ static Value createConst(Location loc, Type type, int value,
 /// Create an integer or index constant, using source as the dynamic shape
 /// source for ranked tensors whose splat cannot be represented as a dense
 /// attribute.
-static Value createConst(Location loc, Type type, int value,
-                         PatternRewriter &rewriter, Value source) {
+static FailureOr<Value>
+createConst(Location loc, Type type, int value, PatternRewriter &rewriter,
+            Value source,
+            DynamicTensorConstantMaterialization dynamicTensorMaterialization) {
   auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
+  auto sourceTensorTy = dyn_cast<RankedTensorType>(source.getType());
   if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
     return createConst(loc, type, value, rewriter);
 
+  if (!sourceTensorTy ||
+      rankedTensorTy.getShape() != sourceTensorTy.getShape() ||
+      dynamicTensorMaterialization !=
+          DynamicTensorConstantMaterialization::TensorSplat)
+    return failure();
+
+  // Dynamic shaped splats need runtime dimension operands from the source.
   Value scalar =
       createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
   return tensor::SplatOp::create(rewriter, loc, scalar,
-                                 tensor::getMixedSizes(rewriter, loc, source));
+                                 tensor::getMixedSizes(rewriter, loc, source))
+      .getResult();
 }
 
 /// Create an integer constant from an APInt.
@@ -81,27 +96,42 @@ static Type cloneToShapedType(Type cloneFrom, Type cloneTo) {
   return cloneTo;
 }
 
-namespace {
-
 /// Expands CeilDivUIOp (n, m) into
 ///  n == 0 ? 0 : ((n-1) / m) + 1
 struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
+  CeilDivUIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivUIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivUIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
     Value a = op.getLhs();
     Value b = op.getRhs();
-    Value zero = createConst(loc, a.getType(), 0, rewriter, a);
-    Value compare =
-        arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq, a, zero);
-    Value one = createConst(loc, a.getType(), 1, rewriter, a);
-    Value minusOne = arith::SubIOp::create(rewriter, loc, a, one);
+    FailureOr<Value> zero = createConst(loc, a.getType(), 0, rewriter, a,
+                                        dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value compare = arith::CmpIOp::create(rewriter, loc,
+                                          arith::CmpIPredicate::eq, a, *zero);
+    FailureOr<Value> one = createConst(loc, a.getType(), 1, rewriter, a,
+                                       dynamicTensorMaterialization);
+    if (failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
+    Value minusOne = arith::SubIOp::create(rewriter, loc, a, *one);
     Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
-    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, zero, plusOne);
+    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, *one);
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, *zero, plusOne);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands CeilDivSIOp (a, b) into
@@ -112,6 +142,12 @@ struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
 ///   return z;
 /// }
 struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
+  CeilDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::CeilDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -120,8 +156,13 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
     Value a = op.getLhs();
     Value b = op.getRhs();
 
-    Value zero = createConst(loc, type, 0, rewriter, a);
-    Value one = createConst(loc, type, 1, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    FailureOr<Value> one =
+        createConst(loc, type, 1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero) || failed(one))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
@@ -129,21 +170,25 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signEqual = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::eq, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signEqual);
 
-    Value quotientPlusOne = arith::AddIOp::create(rewriter, loc, quotient, one);
+    Value quotientPlusOne =
+        arith::AddIOp::create(rewriter, loc, quotient, *one);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientPlusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 /// Expands FloorDivSIOp (x, y) into
@@ -154,6 +199,12 @@ struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
 ///   return z;
 /// }
 struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
+  FloorDivSIOpConverter(
+      MLIRContext *context,
+      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
+      : OpRewritePattern<arith::FloorDivSIOp>(context),
+        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+
   using Base::Base;
   LogicalResult matchAndRewrite(arith::FloorDivSIOp op,
                                 PatternRewriter &rewriter) const final {
@@ -166,26 +217,37 @@ struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
     Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
     Value notEqualDivisor = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    Value zero = createConst(loc, type, 0, rewriter, a);
+    FailureOr<Value> zero =
+        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
+    if (failed(zero))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
 
     Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, zero);
+                                       a, *zero);
     Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, zero);
+                                       b, *zero);
 
     Value signOpposite = arith::CmpIOp::create(
         rewriter, loc, arith::CmpIPredicate::ne, aNeg, bNeg);
     Value cond =
         arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
 
-    Value minusOne = createConst(loc, type, -1, rewriter, a);
+    FailureOr<Value> minusOne =
+        createConst(loc, type, -1, rewriter, a, dynamicTensorMaterialization);
+    if (failed(minusOne))
+      return rewriter.notifyMatchFailure(
+          op, "cannot materialize dynamically shaped tensor constants");
     Value quotientMinusOne =
-        arith::AddIOp::create(rewriter, loc, quotient, minusOne);
+        arith::AddIOp::create(rewriter, loc, quotient, *minusOne);
 
     rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientMinusOne,
                                                  quotient);
     return success();
   }
+
+private:
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 template <typename OpTy, arith::CmpIPredicate pred>
@@ -921,10 +983,14 @@ struct ArithExpandOpsPass
 } // namespace
 
 void mlir::arith::populateCeilFloorDivExpandOpsPatterns(
-    RewritePatternSet &patterns) {
+    RewritePatternSet &patterns, bool enableDynamicTensorSplat) {
+  DynamicTensorConstantMaterialization dynamicTensorMaterialization =
+      enableDynamicTensorSplat
+          ? DynamicTensorConstantMaterialization::TensorSplat
+          : DynamicTensorConstantMaterialization::Disabled;
   patterns
       .add<CeilDivSIOpConverter, CeilDivUIOpConverter, FloorDivSIOpConverter>(
-          patterns.getContext());
+          patterns.getContext(), dynamicTensorMaterialization);
 }
 
 void mlir::arith::populateExpandBFloat16Patterns(RewritePatternSet &patterns) {
@@ -954,7 +1020,8 @@ void mlir::arith::populateExpandFlushDenormalsPatterns(
 }
 
 void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
-  populateCeilFloorDivExpandOpsPatterns(patterns);
+  populateCeilFloorDivExpandOpsPatterns(patterns,
+                                        /*enableDynamicTensorSplat=*/true);
   populateExpandScalingExtTruncPatterns(patterns);
   // clang-format off
   patterns.add<
diff --git a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir b/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
deleted file mode 100644
index 9d169bc60cade..0000000000000
--- a/mlir/test/Conversion/ArithToLLVM/ceil-floor-div-tensor.mlir
+++ /dev/null
@@ -1,10 +0,0 @@
-// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
-// RUN: mlir-opt --convert-to-llvm="filter-dialects=arith" --split-input-file %s | FileCheck %s
-
-// CHECK-LABEL: @ceildivui_dynamic_tensor
-// CHECK-SAME: %[[ARG0:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64>
-func.func @ceildivui_dynamic_tensor(%arg0 : tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
-// CHECK: arith.ceildivui %[[ARG0]], %[[ARG0]] : tensor<8x4x?xi64>
-  %0 = arith.ceildivui %arg0, %arg0 : tensor<8x4x?xi64>
-  return %0: tensor<8x4x?xi64>
-}
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 343c34c9f0ba8..65a8180e480b7 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,16 +89,6 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
-// CHECK-LABEL: @dynamic_tensor_ceildivui
-// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
-func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
-  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
-  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
-  return %0 : tensor<8x4x?xi32>
-}
-
-// -----
-
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 58c66ab6f74c7..3c99d3c5b60ce 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,7 +18,6 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -94,7 +93,6 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
-    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From ae1131363388ffca29bb2d1c70e2d379a716701c Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 21:30:10 +0000
Subject: [PATCH 3/8] [mlir] Restore SPIR-V test

---
 mlir/test/Conversion/ConvertToSPIRV/arith.mlir | 10 ++++++++++
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp  |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 65a8180e480b7..343c34c9f0ba8 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,6 +89,16 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
+// CHECK-LABEL: @dynamic_tensor_ceildivui
+// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
+func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
+  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
+  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
+  return %0 : tensor<8x4x?xi32>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // arith bit ops
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 3c99d3c5b60ce..58c66ab6f74c7 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
+#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -93,6 +94,7 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
+    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 4f6a92bef6a65f92a103adb2c4a6c8211ea31352 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 10 Jul 2026 21:54:16 +0000
Subject: [PATCH 4/8] [mlir] Drop SPIR-V tensor test

---
 mlir/test/Conversion/ConvertToSPIRV/arith.mlir | 8 --------
 mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp  | 2 --
 2 files changed, 10 deletions(-)

diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index 343c34c9f0ba8..fa204c4df8ace 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -89,14 +89,6 @@ func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<
   return %0 : vector<4xi32>
 }
 
-// CHECK-LABEL: @dynamic_tensor_ceildivui
-// CHECK-SAME: (%[[LHS:.+]]: tensor<8x4x?xi32>, %[[RHS:.+]]: tensor<8x4x?xi32>)
-func.func @dynamic_tensor_ceildivui(%lhs: tensor<8x4x?xi32>, %rhs: tensor<8x4x?xi32>) -> tensor<8x4x?xi32> {
-  // CHECK: arith.ceildivui %[[LHS]], %[[RHS]] : tensor<8x4x?xi32>
-  %0 = arith.ceildivui %lhs, %rhs : tensor<8x4x?xi32>
-  return %0 : tensor<8x4x?xi32>
-}
-
 // -----
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
index 58c66ab6f74c7..3c99d3c5b60ce 100644
--- a/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
+++ b/mlir/test/lib/Pass/TestConvertToSPIRVPass.cpp
@@ -18,7 +18,6 @@
 #include "mlir/Dialect/GPU/IR/GPUDialect.h"
 #include "mlir/Dialect/SPIRV/IR/SPIRVAttributes.h"
 #include "mlir/Dialect/SPIRV/Transforms/SPIRVConversion.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/LoweringPatterns.h"
 #include "mlir/Dialect/Vector/Transforms/VectorRewritePatterns.h"
@@ -94,7 +93,6 @@ struct TestConvertToSPIRVPass final
   }
   void getDependentDialects(DialectRegistry &registry) const override {
     registry.insert<spirv::SPIRVDialect>();
-    registry.insert<tensor::TensorDialect>();
     registry.insert<vector::VectorDialect>();
   }
 

>From 2d6c5dced435f347fb4b604e2a4b12c73a395502 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Tue, 14 Jul 2026 16:49:52 +0800
Subject: [PATCH 5/8] [MLIR][Arith] Generalize division expansion

---
 .../mlir/Dialect/Arith/Transforms/Passes.h    |   3 +-
 .../mlir/Dialect/Arith/Transforms/Passes.td   |   2 +-
 .../Dialect/Arith/Transforms/ExpandOps.cpp    | 229 ++-----
 mlir/test/Dialect/Arith/expand-ops.mlir       | 561 +++++++++++++-----
 4 files changed, 464 insertions(+), 331 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
index 11081342c2244..5a07a01d0928a 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
@@ -54,8 +54,7 @@ void populateEmulateUnsupportedFloatsPatterns(RewritePatternSet &patterns,
 void populateEmulateUnsupportedFloatsLegality(ConversionTarget &target,
                                               const TypeConverter &converter);
 /// Add patterns to expand Arith ceil/floor division ops.
-void populateCeilFloorDivExpandOpsPatterns(
-    RewritePatternSet &patterns, bool enableDynamicTensorSplat = false);
+void populateCeilFloorDivExpandOpsPatterns(RewritePatternSet &patterns);
 
 /// Add patterns to expand Arith bf16 patterns to lower level bitcasts/shifts.
 void populateExpandBFloat16Patterns(RewritePatternSet &patterns);
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
index 2cee05914d6b5..27e9146ec3606 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
@@ -13,7 +13,7 @@ include "mlir/Pass/PassBase.td"
 
 def ArithExpandOpsPass : Pass<"arith-expand"> {
   let summary = "Legalize Arith ops to be convertible to LLVM.";
-  let dependentDialects = ["tensor::TensorDialect", "vector::VectorDialect"];
+  let dependentDialects = ["vector::VectorDialect"];
   let options =
       [Option<"includeBf16", "include-bf16", "bool", /*default=*/"false",
               "Enable the BF16 expansion patterns">,
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index b8c0e39dfddda..e165bb2f2c0f1 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -8,7 +8,6 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Arith/Transforms/Passes.h"
-#include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/BuiltinTypeInterfaces.h"
 #include "mlir/IR/Location.h"
@@ -24,10 +23,6 @@ namespace arith {
 
 using namespace mlir;
 
-namespace {
-
-enum class DynamicTensorConstantMaterialization { Disabled, TensorSplat };
-
 /// Create an integer or index constant.
 static Value createConst(Location loc, Type type, int value,
                          PatternRewriter &rewriter) {
@@ -39,32 +34,6 @@ static Value createConst(Location loc, Type type, int value,
   return arith::ConstantOp::create(rewriter, loc, attr);
 }
 
-/// Create an integer or index constant, using source as the dynamic shape
-/// source for ranked tensors whose splat cannot be represented as a dense
-/// attribute.
-static FailureOr<Value>
-createConst(Location loc, Type type, int value, PatternRewriter &rewriter,
-            Value source,
-            DynamicTensorConstantMaterialization dynamicTensorMaterialization) {
-  auto rankedTensorTy = dyn_cast<RankedTensorType>(type);
-  auto sourceTensorTy = dyn_cast<RankedTensorType>(source.getType());
-  if (!rankedTensorTy || rankedTensorTy.hasStaticShape())
-    return createConst(loc, type, value, rewriter);
-
-  if (!sourceTensorTy ||
-      rankedTensorTy.getShape() != sourceTensorTy.getShape() ||
-      dynamicTensorMaterialization !=
-          DynamicTensorConstantMaterialization::TensorSplat)
-    return failure();
-
-  // Dynamic shaped splats need runtime dimension operands from the source.
-  Value scalar =
-      createConst(loc, rankedTensorTy.getElementType(), value, rewriter);
-  return tensor::SplatOp::create(rewriter, loc, scalar,
-                                 tensor::getMixedSizes(rewriter, loc, source))
-      .getResult();
-}
-
 /// Create an integer constant from an APInt.
 static Value createAPIntConst(Location loc, Type type, const APInt &value,
                               PatternRewriter &rewriter) {
@@ -96,158 +65,94 @@ static Type cloneToShapedType(Type cloneFrom, Type cloneTo) {
   return cloneTo;
 }
 
-/// Expands CeilDivUIOp (n, m) into
-///  n == 0 ? 0 : ((n-1) / m) + 1
-struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
-  CeilDivUIOpConverter(
-      MLIRContext *context,
-      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
-      : OpRewritePattern<arith::CeilDivUIOp>(context),
-        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
+static Value extendBoolToType(Location loc, Value value, Type type,
+                              PatternRewriter &rewriter) {
+  if (value.getType() == type)
+    return value;
+  if (getElementTypeOrSelf(type).isIndex())
+    return arith::IndexCastUIOp::create(rewriter, loc, type, value);
+  return arith::ExtUIOp::create(rewriter, loc, type, value);
+}
+
+namespace {
 
+/// Expands CeilDivUIOp (lhs, rhs) into
+///   q = lhs / rhs
+///   q * rhs != lhs ? q + 1 : q
+struct CeilDivUIOpConverter : public OpRewritePattern<arith::CeilDivUIOp> {
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivUIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
-    Value a = op.getLhs();
-    Value b = op.getRhs();
-    FailureOr<Value> zero = createConst(loc, a.getType(), 0, rewriter, a,
-                                        dynamicTensorMaterialization);
-    if (failed(zero))
-      return rewriter.notifyMatchFailure(
-          op, "cannot materialize dynamically shaped tensor constants");
-    Value compare = arith::CmpIOp::create(rewriter, loc,
-                                          arith::CmpIPredicate::eq, a, *zero);
-    FailureOr<Value> one = createConst(loc, a.getType(), 1, rewriter, a,
-                                       dynamicTensorMaterialization);
-    if (failed(one))
-      return rewriter.notifyMatchFailure(
-          op, "cannot materialize dynamically shaped tensor constants");
-    Value minusOne = arith::SubIOp::create(rewriter, loc, a, *one);
-    Value quotient = arith::DivUIOp::create(rewriter, loc, minusOne, b);
-    Value plusOne = arith::AddIOp::create(rewriter, loc, quotient, *one);
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, compare, *zero, plusOne);
+    Value lhs = op.getLhs();
+    Value rhs = op.getRhs();
+
+    Value quotient = arith::DivUIOp::create(rewriter, loc, lhs, rhs);
+    Value product = arith::MulIOp::create(rewriter, loc, quotient, rhs);
+    Value inexact = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ne, product, lhs);
+    Value adjustment = extendBoolToType(loc, inexact, op.getType(), rewriter);
+    rewriter.replaceOpWithNewOp<arith::AddIOp>(op, quotient, adjustment);
     return success();
   }
-
-private:
-  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
-/// Expands CeilDivSIOp (a, b) into
-/// z = a / b
-/// if (z * b != a && (a < 0) == (b < 0)) {
-///   return z + 1;
-/// } else {
-///   return z;
-/// }
+/// Expands CeilDivSIOp (lhs, rhs) by rounding an inexact quotient when the
+/// signed and unsigned operand orderings agree (the operands have equal sign).
 struct CeilDivSIOpConverter : public OpRewritePattern<arith::CeilDivSIOp> {
-  CeilDivSIOpConverter(
-      MLIRContext *context,
-      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
-      : OpRewritePattern<arith::CeilDivSIOp>(context),
-        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
-
   using Base::Base;
   LogicalResult matchAndRewrite(arith::CeilDivSIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
-    Type type = op.getType();
-    Value a = op.getLhs();
-    Value b = op.getRhs();
-
-    FailureOr<Value> zero =
-        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
-    FailureOr<Value> one =
-        createConst(loc, type, 1, rewriter, a, dynamicTensorMaterialization);
-    if (failed(zero) || failed(one))
-      return rewriter.notifyMatchFailure(
-          op, "cannot materialize dynamically shaped tensor constants");
-
-    Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
-    Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
-    Value notEqualDivisor = arith::CmpIOp::create(
-        rewriter, loc, arith::CmpIPredicate::ne, a, product);
-
-    Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, *zero);
-    Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, *zero);
-
-    Value signEqual = arith::CmpIOp::create(
-        rewriter, loc, arith::CmpIPredicate::eq, aNeg, bNeg);
-    Value cond =
-        arith::AndIOp::create(rewriter, loc, notEqualDivisor, signEqual);
-
-    Value quotientPlusOne =
-        arith::AddIOp::create(rewriter, loc, quotient, *one);
+    Value lhs = op.getLhs();
+    Value rhs = op.getRhs();
 
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientPlusOne,
-                                                 quotient);
+    Value quotient = arith::DivSIOp::create(rewriter, loc, lhs, rhs);
+    Value product = arith::MulIOp::create(rewriter, loc, quotient, rhs);
+    Value inexact = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ne, product, lhs);
+    Value signedLess = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::slt, lhs, rhs);
+    Value unsignedLess = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ult, lhs, rhs);
+    Value sameSign = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::eq, signedLess, unsignedLess);
+    Value shouldRound = arith::AndIOp::create(rewriter, loc, inexact, sameSign);
+    Value adjustment =
+        extendBoolToType(loc, shouldRound, op.getType(), rewriter);
+    rewriter.replaceOpWithNewOp<arith::AddIOp>(op, quotient, adjustment);
     return success();
   }
-
-private:
-  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
-/// Expands FloorDivSIOp (x, y) into
-/// z = x / y
-/// if (z * y != x && (x < 0) != (y < 0)) {
-///   return  z - 1;
-/// } else {
-///   return z;
-/// }
+/// Expands FloorDivSIOp (lhs, rhs) by rounding an inexact quotient when the
+/// signed and unsigned operand orderings differ (the operands have opposite
+/// signs).
 struct FloorDivSIOpConverter : public OpRewritePattern<arith::FloorDivSIOp> {
-  FloorDivSIOpConverter(
-      MLIRContext *context,
-      DynamicTensorConstantMaterialization dynamicTensorMaterialization)
-      : OpRewritePattern<arith::FloorDivSIOp>(context),
-        dynamicTensorMaterialization(dynamicTensorMaterialization) {}
-
   using Base::Base;
   LogicalResult matchAndRewrite(arith::FloorDivSIOp op,
                                 PatternRewriter &rewriter) const final {
     Location loc = op.getLoc();
-    Type type = op.getType();
-    Value a = op.getLhs();
-    Value b = op.getRhs();
-
-    Value quotient = arith::DivSIOp::create(rewriter, loc, a, b);
-    Value product = arith::MulIOp::create(rewriter, loc, quotient, b);
-    Value notEqualDivisor = arith::CmpIOp::create(
-        rewriter, loc, arith::CmpIPredicate::ne, a, product);
-    FailureOr<Value> zero =
-        createConst(loc, type, 0, rewriter, a, dynamicTensorMaterialization);
-    if (failed(zero))
-      return rewriter.notifyMatchFailure(
-          op, "cannot materialize dynamically shaped tensor constants");
-
-    Value aNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       a, *zero);
-    Value bNeg = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::slt,
-                                       b, *zero);
-
-    Value signOpposite = arith::CmpIOp::create(
-        rewriter, loc, arith::CmpIPredicate::ne, aNeg, bNeg);
-    Value cond =
-        arith::AndIOp::create(rewriter, loc, notEqualDivisor, signOpposite);
-
-    FailureOr<Value> minusOne =
-        createConst(loc, type, -1, rewriter, a, dynamicTensorMaterialization);
-    if (failed(minusOne))
-      return rewriter.notifyMatchFailure(
-          op, "cannot materialize dynamically shaped tensor constants");
-    Value quotientMinusOne =
-        arith::AddIOp::create(rewriter, loc, quotient, *minusOne);
+    Value lhs = op.getLhs();
+    Value rhs = op.getRhs();
 
-    rewriter.replaceOpWithNewOp<arith::SelectOp>(op, cond, quotientMinusOne,
-                                                 quotient);
+    Value quotient = arith::DivSIOp::create(rewriter, loc, lhs, rhs);
+    Value product = arith::MulIOp::create(rewriter, loc, quotient, rhs);
+    Value inexact = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ne, product, lhs);
+    Value signedLess = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::slt, lhs, rhs);
+    Value unsignedLess = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ult, lhs, rhs);
+    Value oppositeSign = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::ne, signedLess, unsignedLess);
+    Value shouldRound =
+        arith::AndIOp::create(rewriter, loc, inexact, oppositeSign);
+    Value adjustment =
+        extendBoolToType(loc, shouldRound, op.getType(), rewriter);
+    rewriter.replaceOpWithNewOp<arith::SubIOp>(op, quotient, adjustment);
     return success();
   }
-
-private:
-  DynamicTensorConstantMaterialization dynamicTensorMaterialization;
 };
 
 template <typename OpTy, arith::CmpIPredicate pred>
@@ -904,7 +809,6 @@ struct ArithExpandOpsPass
     arith::populateArithExpandOpsPatterns(patterns);
 
     target.addLegalDialect<arith::ArithDialect>();
-    target.addLegalDialect<tensor::TensorDialect>();
     target.addLegalDialect<vector::VectorDialect>();
 
     // clang-format off
@@ -983,14 +887,10 @@ struct ArithExpandOpsPass
 } // namespace
 
 void mlir::arith::populateCeilFloorDivExpandOpsPatterns(
-    RewritePatternSet &patterns, bool enableDynamicTensorSplat) {
-  DynamicTensorConstantMaterialization dynamicTensorMaterialization =
-      enableDynamicTensorSplat
-          ? DynamicTensorConstantMaterialization::TensorSplat
-          : DynamicTensorConstantMaterialization::Disabled;
+    RewritePatternSet &patterns) {
   patterns
       .add<CeilDivSIOpConverter, CeilDivUIOpConverter, FloorDivSIOpConverter>(
-          patterns.getContext(), dynamicTensorMaterialization);
+          patterns.getContext());
 }
 
 void mlir::arith::populateExpandBFloat16Patterns(RewritePatternSet &patterns) {
@@ -1020,8 +920,7 @@ void mlir::arith::populateExpandFlushDenormalsPatterns(
 }
 
 void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
-  populateCeilFloorDivExpandOpsPatterns(patterns,
-                                        /*enableDynamicTensorSplat=*/true);
+  populateCeilFloorDivExpandOpsPatterns(patterns);
   populateExpandScalingExtTruncPatterns(patterns);
   // clang-format off
   patterns.add<
diff --git a/mlir/test/Dialect/Arith/expand-ops.mlir b/mlir/test/Dialect/Arith/expand-ops.mlir
index 80b6592417cc2..f8432a6a78627 100644
--- a/mlir/test/Dialect/Arith/expand-ops.mlir
+++ b/mlir/test/Dialect/Arith/expand-ops.mlir
@@ -1,208 +1,443 @@
 // RUN: mlir-opt %s -arith-expand="include-bf16=true include-f8e8m0=true include-f4e2m1=true" -verify-diagnostics -split-input-file | FileCheck %s
 // RUN: mlir-opt %s -arith-expand -split-input-file -verify-diagnostics | FileCheck %s --check-prefix=SCHECK
-
-// Test ceil divide with signed integer
-// CHECK-LABEL:       func @ceildivi
-// CHECK-SAME:     ([[ARG0:%.+]]: i32, [[ARG1:%.+]]: i32) -> i32 {
-func.func @ceildivi(%arg0: i32, %arg1: i32) -> (i32) {
-  %res = arith.ceildivsi %arg0, %arg1 : i32
+// RUN: mlir-opt %s -arith-expand="include-bf16=true include-f8e8m0=true include-f4e2m1=true" -canonicalize -verify-diagnostics -split-input-file | FileCheck %s --check-prefix=VALUES
+
+// CHECK-LABEL: func.func @ceildivui(
+// CHECK-SAME: %[[LHS:.*]]: i32, %[[RHS:.*]]: i32) -> i32 {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i32
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i32
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[INEXACT]] : i1 to i32
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : i32
+// CHECK-NEXT: return %[[RESULT]] : i32
+func.func @ceildivui(%arg0: i32, %arg1: i32) -> i32 {
+  %res = arith.ceildivui %arg0, %arg1 : i32
   return %res : i32
-
-// CHECK:           [[ZERO:%.+]] = arith.constant 0 : i32
-// CHECK:           [[ONE:%.+]] = arith.constant 1 : i32
-// CHECK:           [[DIV:%.+]] = arith.divsi %arg0, %arg1 : i32
-// CHECK:           [[MUL:%.+]] = arith.muli [[DIV]], %arg1 : i32
-// CHECK:           [[NEXACT:%.+]] = arith.cmpi ne, %arg0, [[MUL]] : i32
-// CHECK:           [[NNEG:%.+]] = arith.cmpi slt, %arg0, [[ZERO]] : i32
-// CHECK:           [[MNEG:%.+]] = arith.cmpi slt, %arg1, [[ZERO]] : i32
-// CHECK:           [[SAMESIGN:%.+]] = arith.cmpi eq, [[NNEG]], [[MNEG]] : i1
-// CHECK:           [[SHOULDROUND:%.+]] = arith.andi [[NEXACT]], [[SAMESIGN]] : i1
-// CHECK:           [[CEIL:%.+]] = arith.addi [[DIV]], [[ONE]] : i32
-// CHECK:           [[RES:%.+]] = arith.select [[SHOULDROUND]], [[CEIL]], [[DIV]] : i32
 }
 
 // -----
 
-// Test ceil divide with index type
-// CHECK-LABEL:       func @ceildivi_index
-// CHECK-SAME:     ([[ARG0:%.+]]: index, [[ARG1:%.+]]: index) -> index {
-func.func @ceildivi_index(%arg0: index, %arg1: index) -> (index) {
-  %res = arith.ceildivsi %arg0, %arg1 : index
+// CHECK-LABEL: func.func @ceildivui_index(
+// CHECK-SAME: %[[LHS:.*]]: index, %[[RHS:.*]]: index) -> index {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : index
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : index
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.index_castui %[[INEXACT]] : i1 to index
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : index
+// CHECK-NEXT: return %[[RESULT]] : index
+func.func @ceildivui_index(%arg0: index, %arg1: index) -> index {
+  %res = arith.ceildivui %arg0, %arg1 : index
   return %res : index
+}
 
-// CHECK:           [[ZERO:%.+]] = arith.constant 0 : index
-// CHECK:           [[ONE:%.+]] = arith.constant 1 : index
-// CHECK:           [[DIV:%.+]] = arith.divsi %arg0, %arg1 : index
-// CHECK:           [[MUL:%.+]] = arith.muli [[DIV]], %arg1 : index
-// CHECK:           [[NEXACT:%.+]] = arith.cmpi ne, %arg0, [[MUL]] : index
-// CHECK:           [[NNEG:%.+]] = arith.cmpi slt, %arg0, [[ZERO]] : index
-// CHECK:           [[MNEG:%.+]] = arith.cmpi slt, %arg1, [[ZERO]] : index
-// CHECK:           [[SAMESIGN:%.+]] = arith.cmpi eq, [[NNEG]], [[MNEG]] : i1
-// CHECK:           [[SHOULDROUND:%.+]] = arith.andi [[NEXACT]], [[SAMESIGN]] : i1
-// CHECK:           [[CEIL:%.+]] = arith.addi [[DIV]], [[ONE]] : index
-// CHECK:           [[RES:%.+]] = arith.select [[SHOULDROUND]], [[CEIL]], [[DIV]] : index
+// -----
 
+// CHECK-LABEL: func.func @ceildivui_vec(
+// CHECK-SAME: %[[LHS:.*]]: vector<4xi32>, %[[RHS:.*]]: vector<4xi32>) -> vector<4xi32> {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : vector<4xi32>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[INEXACT]] : vector<4xi1> to vector<4xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : vector<4xi32>
+// CHECK-NEXT: return %[[RESULT]] : vector<4xi32>
+func.func @ceildivui_vec(%arg0: vector<4xi32>, %arg1: vector<4xi32>) -> vector<4xi32> {
+  %res = arith.ceildivui %arg0, %arg1 : vector<4xi32>
+  return %res : vector<4xi32>
 }
 
 // -----
 
-// Test floor divide with signed integer
-// CHECK-LABEL:       func @floordivi
-// CHECK-SAME:     ([[ARG0:%.+]]: i32, [[ARG1:%.+]]: i32) -> i32 {
-func.func @floordivi(%arg0: i32, %arg1: i32) -> (i32) {
-  %res = arith.floordivsi %arg0, %arg1 : i32
-  return %res : i32
-// CHECK:   %[[QUOTIENT:.*]] = arith.divsi %arg0, %arg1 : i32
-// CHECK:   %[[PRODUCT:.*]] = arith.muli %[[QUOTIENT]], %arg1 : i32
-// CHECK:   %[[NOT_EQ_PRODUCT:.*]] = arith.cmpi ne, %arg0, %[[PRODUCT]] : i32
-// CHECK-DAG:   %[[ZERO:.*]] = arith.constant 0 : i32
-// CHECK:   %[[NEG_DIVISOR:.*]] = arith.cmpi slt, %arg0, %[[ZERO]] : i32
-// CHECK:   %[[NEG_DIVIDEND:.*]] = arith.cmpi slt, %arg1, %[[ZERO]] : i32
-// CHECK:   %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[NEG_DIVISOR]], %[[NEG_DIVIDEND]] : i1
-// CHECK:   %[[CONDITION:.*]] = arith.andi %[[NOT_EQ_PRODUCT]], %[[OPPOSITE_SIGN]] : i1
-// CHECK-DAG:   %[[NEG_ONE:.*]] = arith.constant -1 : i32
-// CHECK:   %[[MINUS_ONE:.*]] = arith.addi %[[QUOTIENT]], %[[NEG_ONE]] : i32
-// CHECK:   %[[RES:.*]] = arith.select %[[CONDITION]], %[[MINUS_ONE]], %[[QUOTIENT]] : i32
+// CHECK-LABEL: func.func @ceildivui_static_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<2x3xi32>, %[[RHS:.*]]: tensor<2x3xi32>) -> tensor<2x3xi32> {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[INEXACT]] : tensor<2x3xi1> to tensor<2x3xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : tensor<2x3xi32>
+// CHECK-NEXT: return %[[RESULT]] : tensor<2x3xi32>
+func.func @ceildivui_static_tensor(%arg0: tensor<2x3xi32>, %arg1: tensor<2x3xi32>) -> tensor<2x3xi32> {
+  %res = arith.ceildivui %arg0, %arg1 : tensor<2x3xi32>
+  return %res : tensor<2x3xi32>
 }
 
 // -----
 
-// Test floor divide with index type
-// CHECK-LABEL:       func @floordivi_index
-// CHECK-SAME:     ([[ARG0:%.+]]: index, [[ARG1:%.+]]: index) -> index {
-func.func @floordivi_index(%arg0: index, %arg1: index) -> (index) {
-  %res = arith.floordivsi %arg0, %arg1 : index
-  return %res : index
-// CHECK:   %[[QUOTIENT:.*]] = arith.divsi %arg0, %arg1 : index
-// CHECK:   %[[PRODUCT:.*]] = arith.muli %[[QUOTIENT]], %arg1 : index
-// CHECK:   %[[NOT_EQ_PRODUCT:.*]] = arith.cmpi ne, %arg0, %[[PRODUCT]] : index
-// CHECK-DAG:   %[[ZERO:.*]] = arith.constant 0 : index
-// CHECK:   %[[NEG_DIVISOR:.*]] = arith.cmpi slt, %arg0, %[[ZERO]] : index
-// CHECK:   %[[NEG_DIVIDEND:.*]] = arith.cmpi slt, %arg1, %[[ZERO]] : index
-// CHECK:   %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[NEG_DIVISOR]], %[[NEG_DIVIDEND]] : i1
-// CHECK:   %[[CONDITION:.*]] = arith.andi %[[NOT_EQ_PRODUCT]], %[[OPPOSITE_SIGN]] : i1
-// CHECK:   %[[NEG_ONE:.*]] = arith.constant -1 : index
-// CHECK-DAG:   %[[MINUS_ONE:.*]] = arith.addi %[[QUOTIENT]], %[[NEG_ONE]] : index
-// CHECK:   %[[RES:.*]] = arith.select %[[CONDITION]], %[[MINUS_ONE]], %[[QUOTIENT]] : index
-}
-
-// -----
-
-// Test floor divide with vector
-// CHECK-LABEL:   func.func @floordivi_vec(
-// CHECK-SAME:                             %[[VAL_0:.*]]: vector<4xi32>,
-// CHECK-SAME:                             %[[VAL_1:.*]]: vector<4xi32>) -> vector<4xi32> {
-func.func @floordivi_vec(%arg0: vector<4xi32>, %arg1: vector<4xi32>) -> (vector<4xi32>) {
-  %res = arith.floordivsi %arg0, %arg1 : vector<4xi32>
-  return %res : vector<4xi32>
-// CHECK:   %[[QUOTIENT:.*]] = arith.divsi %arg0, %arg1 : vector<4xi32>
-// CHECK:   %[[PRODUCT:.*]] = arith.muli %[[QUOTIENT]], %arg1 : vector<4xi32>
-// CHECK:   %[[NOT_EQ_PRODUCT:.*]] = arith.cmpi ne, %arg0, %[[PRODUCT]] : vector<4xi32>
-// CHECK-DAG:   %[[ZERO:.*]] = arith.constant dense<0> : vector<4xi32>
-// CHECK:   %[[NEG_DIVISOR:.*]] = arith.cmpi slt, %arg0, %[[ZERO]] : vector<4xi32>
-// CHECK:   %[[NEG_DIVIDEND:.*]] = arith.cmpi slt, %arg1, %[[ZERO]] : vector<4xi32>
-// CHECK:   %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[NEG_DIVISOR]], %[[NEG_DIVIDEND]] : vector<4xi1>
-// CHECK:   %[[CONDITION:.*]] = arith.andi %[[NOT_EQ_PRODUCT]], %[[OPPOSITE_SIGN]] : vector<4xi1>
-// CHECK-DAG:   %[[NEG_ONE:.*]] = arith.constant dense<-1> : vector<4xi32>
-// CHECK:   %[[MINUS_ONE:.*]] = arith.addi %[[QUOTIENT]], %[[NEG_ONE]] : vector<4xi32>
-// CHECK:   %[[RES:.*]] = arith.select %[[CONDITION]], %[[MINUS_ONE]], %[[QUOTIENT]] : vector<4xi1>, vector<4xi32>
+// CHECK-LABEL: func.func @ceildivui_dynamic_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<8x4x?xi64>, %[[RHS:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : tensor<8x4x?xi64>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<8x4x?xi64>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<8x4x?xi64>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[INEXACT]] : tensor<8x4x?xi1> to tensor<8x4x?xi64>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : tensor<8x4x?xi64>
+// CHECK-NEXT: return %[[RESULT]] : tensor<8x4x?xi64>
+func.func @ceildivui_dynamic_tensor(%arg0: tensor<8x4x?xi64>, %arg1: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
+  %res = arith.ceildivui %arg0, %arg1 : tensor<8x4x?xi64>
+  return %res : tensor<8x4x?xi64>
 }
 
 // -----
 
-// Test ceil divide with unsigned integer
-// CHECK-LABEL:       func @ceildivui
-// CHECK-SAME:     ([[ARG0:%.+]]: i32, [[ARG1:%.+]]: i32) -> i32 {
-func.func @ceildivui(%arg0: i32, %arg1: i32) -> (i32) {
-  %res = arith.ceildivui %arg0, %arg1 : i32
+// CHECK-LABEL: func.func @ceildivui_i1(
+// CHECK-SAME: %[[LHS:.*]]: i1, %[[RHS:.*]]: i1) -> i1 {
+// CHECK-NOT: arith.ceildivui
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divui %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i1
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i1
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[INEXACT]] : i1
+// CHECK-NEXT: return %[[RESULT]] : i1
+func.func @ceildivui_i1(%arg0: i1, %arg1: i1) -> i1 {
+  %res = arith.ceildivui %arg0, %arg1 : i1
+  return %res : i1
+}
+
+// -----
+
+// CHECK-LABEL: func.func @ceildivi(
+// CHECK-SAME: %[[LHS:.*]]: i32, %[[RHS:.*]]: i32) -> i32 {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i32
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i32
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : i1
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : i1 to i32
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : i32
+// CHECK-NEXT: return %[[RESULT]] : i32
+func.func @ceildivi(%arg0: i32, %arg1: i32) -> i32 {
+  %res = arith.ceildivsi %arg0, %arg1 : i32
   return %res : i32
-// CHECK:           [[ZERO:%.+]] = arith.constant 0 : i32
-// CHECK:           [[ISZERO:%.+]] = arith.cmpi eq, %arg0, [[ZERO]] : i32
-// CHECK:           [[ONE:%.+]] = arith.constant 1 : i32
-// CHECK:           [[SUB:%.+]] = arith.subi %arg0, [[ONE]] : i32
-// CHECK:           [[DIV:%.+]] = arith.divui [[SUB]], %arg1 : i32
-// CHECK:           [[REM:%.+]] = arith.addi [[DIV]], [[ONE]] : i32
-// CHECK:           [[RES:%.+]] = arith.select [[ISZERO]], [[ZERO]], [[REM]] : i32
 }
 
 // -----
 
-// Test unsigned ceil divide with index
-// CHECK-LABEL:       func @ceildivui_index
-// CHECK-SAME:     ([[ARG0:%.+]]: index, [[ARG1:%.+]]: index) -> index {
-func.func @ceildivui_index(%arg0: index, %arg1: index) -> (index) {
-  %res = arith.ceildivui %arg0, %arg1 : index
+// CHECK-LABEL: func.func @ceildivi_index(
+// CHECK-SAME: %[[LHS:.*]]: index, %[[RHS:.*]]: index) -> index {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : index
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : index
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : i1
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.index_castui %[[ROUND]] : i1 to index
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : index
+// CHECK-NEXT: return %[[RESULT]] : index
+func.func @ceildivi_index(%arg0: index, %arg1: index) -> index {
+  %res = arith.ceildivsi %arg0, %arg1 : index
   return %res : index
-// CHECK:           [[ZERO:%.+]] = arith.constant 0 : index
-// CHECK:           [[ISZERO:%.+]] = arith.cmpi eq, %arg0, [[ZERO]] : index
-// CHECK:           [[ONE:%.+]] = arith.constant 1 : index
-// CHECK:           [[SUB:%.+]] = arith.subi %arg0, [[ONE]] : index
-// CHECK:           [[DIV:%.+]] = arith.divui [[SUB]], %arg1 : index
-// CHECK:           [[REM:%.+]] = arith.addi [[DIV]], [[ONE]] : index
-// CHECK:           [[RES:%.+]] = arith.select [[ISZERO]], [[ZERO]], [[REM]] : index
 }
 
 // -----
 
-// CHECK-LABEL:   func.func @ceildivui_dynamic_tensor(
-// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x4x?xi64>,
-// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
-func.func @ceildivui_dynamic_tensor(%arg0: tensor<8x4x?xi64>, %arg1: tensor<8x4x?xi64>) -> tensor<8x4x?xi64> {
-  %res = arith.ceildivui %arg0, %arg1 : tensor<8x4x?xi64>
-  return %res : tensor<8x4x?xi64>
-// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
-// CHECK:       %[[C2_IDX:.*]] = arith.constant 2 : index
-// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX]] : tensor<8x4x?xi64>
-// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x4x?xi64>
-// CHECK:       %[[ISZERO:.*]] = arith.cmpi eq, %[[ARG0]], %[[ZERO]] : tensor<8x4x?xi64>
-// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
-// CHECK:       %[[C2_IDX_1:.*]] = arith.constant 2 : index
-// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C2_IDX_1]] : tensor<8x4x?xi64>
-// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x4x?xi64>
-// CHECK:       %[[SUB:.*]] = arith.subi %[[ARG0]], %[[ONE]] : tensor<8x4x?xi64>
-// CHECK:       %[[DIV:.*]] = arith.divui %[[SUB]], %[[ARG1]] : tensor<8x4x?xi64>
-// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x4x?xi64>
-// CHECK:       %[[RES:.*]] = arith.select %[[ISZERO]], %[[ZERO]], %[[ADD]] : tensor<8x4x?xi1>, tensor<8x4x?xi64>
-}
-
-// -----
-
-// CHECK-LABEL:   func.func @ceildivsi_dynamic_tensor(
-// CHECK-SAME:                                      %[[ARG0:.*]]: tensor<8x?xi64>,
-// CHECK-SAME:                                      %[[ARG1:.*]]: tensor<8x?xi64>) -> tensor<8x?xi64> {
+// CHECK-LABEL: func.func @ceildivsi_vec(
+// CHECK-SAME: %[[LHS:.*]]: vector<4xi32>, %[[RHS:.*]]: vector<4xi32>) -> vector<4xi32> {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : vector<4xi32>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : vector<4xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : vector<4xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : vector<4xi1> to vector<4xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : vector<4xi32>
+// CHECK-NEXT: return %[[RESULT]] : vector<4xi32>
+func.func @ceildivsi_vec(%arg0: vector<4xi32>, %arg1: vector<4xi32>) -> vector<4xi32> {
+  %res = arith.ceildivsi %arg0, %arg1 : vector<4xi32>
+  return %res : vector<4xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @ceildivsi_static_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<2x3xi32>, %[[RHS:.*]]: tensor<2x3xi32>) -> tensor<2x3xi32> {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : tensor<2x3xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : tensor<2x3xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : tensor<2x3xi1> to tensor<2x3xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : tensor<2x3xi32>
+// CHECK-NEXT: return %[[RESULT]] : tensor<2x3xi32>
+func.func @ceildivsi_static_tensor(%arg0: tensor<2x3xi32>, %arg1: tensor<2x3xi32>) -> tensor<2x3xi32> {
+  %res = arith.ceildivsi %arg0, %arg1 : tensor<2x3xi32>
+  return %res : tensor<2x3xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @ceildivsi_dynamic_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<8x?xi64>, %[[RHS:.*]]: tensor<8x?xi64>) -> tensor<8x?xi64> {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : tensor<8x?xi64>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<8x?xi64>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<8x?xi64>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : tensor<8x?xi64>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : tensor<8x?xi64>
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : tensor<8x?xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : tensor<8x?xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : tensor<8x?xi1> to tensor<8x?xi64>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ADJUSTMENT]] : tensor<8x?xi64>
+// CHECK-NEXT: return %[[RESULT]] : tensor<8x?xi64>
 func.func @ceildivsi_dynamic_tensor(%arg0: tensor<8x?xi64>, %arg1: tensor<8x?xi64>) -> tensor<8x?xi64> {
   %res = arith.ceildivsi %arg0, %arg1 : tensor<8x?xi64>
   return %res : tensor<8x?xi64>
-// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
-// CHECK:       %[[C1_IDX:.*]] = arith.constant 1 : index
-// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX]] : tensor<8x?xi64>
-// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<8x?xi64>
-// CHECK:       %[[ONE_SCALAR:.*]] = arith.constant 1 : i64
-// CHECK:       %[[C1_IDX_1:.*]] = arith.constant 1 : index
-// CHECK:       %[[ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C1_IDX_1]] : tensor<8x?xi64>
-// CHECK:       %[[ONE:.*]] = tensor.splat %[[ONE_SCALAR]]{{\[}}%[[ONE_DIM]]] : tensor<8x?xi64>
-// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<8x?xi64>
-// CHECK:       %[[ADD:.*]] = arith.addi %[[DIV]], %[[ONE]] : tensor<8x?xi64>
 }
 
 // -----
 
-// CHECK-LABEL:   func.func @floordivsi_dynamic_tensor(
-// CHECK-SAME:                                       %[[ARG0:.*]]: tensor<?x4xi64>,
-// CHECK-SAME:                                       %[[ARG1:.*]]: tensor<?x4xi64>) -> tensor<?x4xi64> {
+// CHECK-LABEL: func.func @ceildivsi_i1(
+// CHECK-SAME: %[[LHS:.*]]: i1, %[[RHS:.*]]: i1) -> i1 {
+// CHECK-NOT: arith.ceildivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i1
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i1
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[SAME_SIGN:.*]] = arith.cmpi eq, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[SAME_SIGN]] : i1
+// CHECK-NEXT: %[[RESULT:.*]] = arith.addi %[[Q]], %[[ROUND]] : i1
+// CHECK-NEXT: return %[[RESULT]] : i1
+func.func @ceildivsi_i1(%arg0: i1, %arg1: i1) -> i1 {
+  %res = arith.ceildivsi %arg0, %arg1 : i1
+  return %res : i1
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivi(
+// CHECK-SAME: %[[LHS:.*]]: i32, %[[RHS:.*]]: i32) -> i32 {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i32
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i32
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : i32
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : i1
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : i1 to i32
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ADJUSTMENT]] : i32
+// CHECK-NEXT: return %[[RESULT]] : i32
+func.func @floordivi(%arg0: i32, %arg1: i32) -> i32 {
+  %res = arith.floordivsi %arg0, %arg1 : i32
+  return %res : i32
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivi_index(
+// CHECK-SAME: %[[LHS:.*]]: index, %[[RHS:.*]]: index) -> index {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : index
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : index
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : index
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : i1
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.index_castui %[[ROUND]] : i1 to index
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ADJUSTMENT]] : index
+// CHECK-NEXT: return %[[RESULT]] : index
+func.func @floordivi_index(%arg0: index, %arg1: index) -> index {
+  %res = arith.floordivsi %arg0, %arg1 : index
+  return %res : index
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivi_vec(
+// CHECK-SAME: %[[LHS:.*]]: vector<4xi32>, %[[RHS:.*]]: vector<4xi32>) -> vector<4xi32> {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : vector<4xi32>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : vector<4xi32>
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : vector<4xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : vector<4xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : vector<4xi1> to vector<4xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ADJUSTMENT]] : vector<4xi32>
+// CHECK-NEXT: return %[[RESULT]] : vector<4xi32>
+func.func @floordivi_vec(%arg0: vector<4xi32>, %arg1: vector<4xi32>) -> vector<4xi32> {
+  %res = arith.floordivsi %arg0, %arg1 : vector<4xi32>
+  return %res : vector<4xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivsi_static_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<2x3xi32>, %[[RHS:.*]]: tensor<2x3xi32>) -> tensor<2x3xi32> {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : tensor<2x3xi32>
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : tensor<2x3xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : tensor<2x3xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : tensor<2x3xi1> to tensor<2x3xi32>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ADJUSTMENT]] : tensor<2x3xi32>
+// CHECK-NEXT: return %[[RESULT]] : tensor<2x3xi32>
+func.func @floordivsi_static_tensor(%arg0: tensor<2x3xi32>, %arg1: tensor<2x3xi32>) -> tensor<2x3xi32> {
+  %res = arith.floordivsi %arg0, %arg1 : tensor<2x3xi32>
+  return %res : tensor<2x3xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivsi_dynamic_tensor(
+// CHECK-SAME: %[[LHS:.*]]: tensor<?x4xi64>, %[[RHS:.*]]: tensor<?x4xi64>) -> tensor<?x4xi64> {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : tensor<?x4xi64>
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : tensor<?x4xi64>
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : tensor<?x4xi64>
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : tensor<?x4xi64>
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : tensor<?x4xi64>
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : tensor<?x4xi1>
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : tensor<?x4xi1>
+// CHECK-NEXT: %[[ADJUSTMENT:.*]] = arith.extui %[[ROUND]] : tensor<?x4xi1> to tensor<?x4xi64>
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ADJUSTMENT]] : tensor<?x4xi64>
+// CHECK-NEXT: return %[[RESULT]] : tensor<?x4xi64>
 func.func @floordivsi_dynamic_tensor(%arg0: tensor<?x4xi64>, %arg1: tensor<?x4xi64>) -> tensor<?x4xi64> {
   %res = arith.floordivsi %arg0, %arg1 : tensor<?x4xi64>
   return %res : tensor<?x4xi64>
-// CHECK:       %[[DIV:.*]] = arith.divsi %[[ARG0]], %[[ARG1]] : tensor<?x4xi64>
-// CHECK:       %[[ZERO_SCALAR:.*]] = arith.constant 0 : i64
-// CHECK:       %[[C0_IDX:.*]] = arith.constant 0 : index
-// CHECK:       %[[ZERO_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX]] : tensor<?x4xi64>
-// CHECK:       %[[ZERO:.*]] = tensor.splat %[[ZERO_SCALAR]]{{\[}}%[[ZERO_DIM]]] : tensor<?x4xi64>
-// CHECK:       %[[NEG_ONE_SCALAR:.*]] = arith.constant -1 : i64
-// CHECK:       %[[C0_IDX_1:.*]] = arith.constant 0 : index
-// CHECK:       %[[NEG_ONE_DIM:.*]] = tensor.dim %[[ARG0]], %[[C0_IDX_1]] : tensor<?x4xi64>
-// CHECK:       %[[NEG_ONE:.*]] = tensor.splat %[[NEG_ONE_SCALAR]]{{\[}}%[[NEG_ONE_DIM]]] : tensor<?x4xi64>
-// CHECK:       %[[SUB_ONE:.*]] = arith.addi %[[DIV]], %[[NEG_ONE]] : tensor<?x4xi64>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @floordivsi_i1(
+// CHECK-SAME: %[[LHS:.*]]: i1, %[[RHS:.*]]: i1) -> i1 {
+// CHECK-NOT: arith.floordivsi
+// CHECK-NOT: arith.constant
+// CHECK-NOT: tensor.
+// CHECK-NOT: arith.extui
+// CHECK-NOT: arith.index_castui
+// CHECK-NEXT: %[[Q:.*]] = arith.divsi %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[PRODUCT:.*]] = arith.muli %[[Q]], %[[RHS]] : i1
+// CHECK-NEXT: %[[INEXACT:.*]] = arith.cmpi ne, %[[PRODUCT]], %[[LHS]] : i1
+// CHECK-NEXT: %[[SIGNED_LT:.*]] = arith.cmpi slt, %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[UNSIGNED_LT:.*]] = arith.cmpi ult, %[[LHS]], %[[RHS]] : i1
+// CHECK-NEXT: %[[OPPOSITE_SIGN:.*]] = arith.cmpi ne, %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+// CHECK-NEXT: %[[ROUND:.*]] = arith.andi %[[INEXACT]], %[[OPPOSITE_SIGN]] : i1
+// CHECK-NEXT: %[[RESULT:.*]] = arith.subi %[[Q]], %[[ROUND]] : i1
+// CHECK-NEXT: return %[[RESULT]] : i1
+func.func @floordivsi_i1(%arg0: i1, %arg1: i1) -> i1 {
+  %res = arith.floordivsi %arg0, %arg1 : i1
+  return %res : i1
+}
+
+// -----
+
+// VALUES-LABEL: func.func @ceildivui_values()
+// VALUES-DAG: %[[THREE:.*]] = arith.constant 3 : i8
+// VALUES-DAG: %[[TWO:.*]] = arith.constant 2 : i8
+// VALUES-DAG: %[[ZERO:.*]] = arith.constant 0 : i8
+// VALUES: return %[[THREE]], %[[TWO]], %[[ZERO]] : i8, i8, i8
+func.func @ceildivui_values() -> (i8, i8, i8) {
+  %zero = arith.constant 0 : i8
+  %two = arith.constant 2 : i8
+  %four = arith.constant 4 : i8
+  %five = arith.constant 5 : i8
+  %inexact = arith.ceildivui %five, %two : i8
+  %exact = arith.ceildivui %four, %two : i8
+  %zeroDividend = arith.ceildivui %zero, %two : i8
+  return %inexact, %exact, %zeroDividend : i8, i8, i8
+}
+
+// VALUES-LABEL: func.func @ceildivsi_values()
+// VALUES-DAG: %[[THREE:.*]] = arith.constant 3 : i8
+// VALUES-DAG: %[[NEG_TWO:.*]] = arith.constant -2 : i8
+// VALUES: return %[[THREE]], %[[NEG_TWO]], %[[NEG_TWO]], %[[THREE]] : i8, i8, i8, i8
+func.func @ceildivsi_values() -> (i8, i8, i8, i8) {
+  %negFive = arith.constant -5 : i8
+  %negTwo = arith.constant -2 : i8
+  %two = arith.constant 2 : i8
+  %five = arith.constant 5 : i8
+  %posPos = arith.ceildivsi %five, %two : i8
+  %negPos = arith.ceildivsi %negFive, %two : i8
+  %posNeg = arith.ceildivsi %five, %negTwo : i8
+  %negNeg = arith.ceildivsi %negFive, %negTwo : i8
+  return %posPos, %negPos, %posNeg, %negNeg : i8, i8, i8, i8
+}
+
+// VALUES-LABEL: func.func @floordivsi_values()
+// VALUES-DAG: %[[TWO:.*]] = arith.constant 2 : i8
+// VALUES-DAG: %[[NEG_THREE:.*]] = arith.constant -3 : i8
+// VALUES: return %[[TWO]], %[[NEG_THREE]], %[[NEG_THREE]], %[[TWO]] : i8, i8, i8, i8
+func.func @floordivsi_values() -> (i8, i8, i8, i8) {
+  %negFive = arith.constant -5 : i8
+  %negTwo = arith.constant -2 : i8
+  %two = arith.constant 2 : i8
+  %five = arith.constant 5 : i8
+  %posPos = arith.floordivsi %five, %two : i8
+  %negPos = arith.floordivsi %negFive, %two : i8
+  %posNeg = arith.floordivsi %five, %negTwo : i8
+  %negNeg = arith.floordivsi %negFive, %negTwo : i8
+  return %posPos, %negPos, %posNeg, %negNeg : i8, i8, i8, i8
 }
 
 // -----

>From e05cd53e02b5ab91d98317901d835c37c1b2070b Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Tue, 14 Jul 2026 16:50:11 +0800
Subject: [PATCH 6/8] [MLIR] Update division consumer checks

---
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir | 52 ++++++++-----------
 .../test/Conversion/ConvertToSPIRV/arith.mlir | 51 ++++++++++--------
 2 files changed, 52 insertions(+), 51 deletions(-)

diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..133c10eeb6ed8 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -764,17 +764,15 @@ func.func @select_complex(%arg0 : i1, %arg1 : complex<f32>, %arg2 : complex<f32>
 // CHECK-LABEL: @ceildivsi
 // CHECK-SAME: %[[ARG0:.*]]: i64, %[[ARG1:.*]]: i64) -> i64
 func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
-  // CHECK: %[[ZERO:.+]] = llvm.mlir.constant(0 : i64) : i64
-  // CHECK: %[[ONE:.+]] = llvm.mlir.constant(1 : i64) : i64
-  // CHECK: %[[DIV:.+]] = llvm.sdiv %[[ARG0]], %[[ARG1]] : i64
-  // CHECK: %[[MUL:.+]] = llvm.mul %[[DIV]], %[[ARG1]] : i64
-  // CHECK: %[[NEXACT:.+]] = llvm.icmp "ne" %[[ARG0]], %[[MUL]] : i64
-  // CHECK: %[[NNEG:.+]] = llvm.icmp "slt" %[[ARG0]], %[[ZERO]] : i64
-  // CHECK: %[[MNEG:.+]] = llvm.icmp "slt" %[[ARG1]], %[[ZERO]] : i64
-  // CHECK: %[[SAMESIGN:.+]] = llvm.icmp "eq" %[[NNEG]], %[[MNEG]] : i1
-  // CHECK: %[[SHOULDROUND:.+]] = llvm.and %[[NEXACT]], %[[SAMESIGN]] : i1
-  // CHECK: %[[CEIL:.+]] = llvm.add %[[DIV]], %[[ONE]] : i64
-  // CHECK: %[[RES:.+]] = llvm.select %[[SHOULDROUND]], %[[CEIL]], %[[DIV]] : i1, i64
+  // CHECK: %[[Q:.+]] = llvm.sdiv %[[ARG0]], %[[ARG1]] : i64
+  // CHECK: %[[PRODUCT:.+]] = llvm.mul %[[Q]], %[[ARG1]] : i64
+  // CHECK: %[[INEXACT:.+]] = llvm.icmp "ne" %[[PRODUCT]], %[[ARG0]] : i64
+  // CHECK: %[[SIGNED_LT:.+]] = llvm.icmp "slt" %[[ARG0]], %[[ARG1]] : i64
+  // CHECK: %[[UNSIGNED_LT:.+]] = llvm.icmp "ult" %[[ARG0]], %[[ARG1]] : i64
+  // CHECK: %[[SAME_SIGN:.+]] = llvm.icmp "eq" %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+  // CHECK: %[[ROUND:.+]] = llvm.and %[[INEXACT]], %[[SAME_SIGN]] : i1
+  // CHECK: %[[ADJUSTMENT:.+]] = llvm.zext %[[ROUND]] : i1 to i64
+  // CHECK: %[[RES:.+]] = llvm.add %[[Q]], %[[ADJUSTMENT]] : i64
   %0 = arith.ceildivsi %arg0, %arg1 : i64
   return %0: i64
 }
@@ -782,13 +780,11 @@ func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
 // CHECK-LABEL: @ceildivui
 // CHECK-SAME: %[[ARG0:.*]]: i32) -> i32
 func.func @ceildivui(%arg0 : i32) -> i32 {
-// CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
-// CHECK: %[[CMP0:.*]] = llvm.icmp "eq" %[[ARG0]], %[[CST0]] : i32
-// CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
-// CHECK: %[[SUB0:.*]] = llvm.sub %[[ARG0]], %[[CST1]] : i32
-// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG0]] : i32
-// CHECK: %[[ADD0:.*]] = llvm.add %[[DIV0]], %[[CST1]] : i32
-// CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[CST0]], %[[ADD0]] : i1, i32
+// CHECK: %[[Q:.*]] = llvm.udiv %[[ARG0]], %[[ARG0]] : i32
+// CHECK: %[[PRODUCT:.*]] = llvm.mul %[[Q]], %[[ARG0]] : i32
+// CHECK: %[[INEXACT:.*]] = llvm.icmp "ne" %[[PRODUCT]], %[[ARG0]] : i32
+// CHECK: %[[ADJUSTMENT:.*]] = llvm.zext %[[INEXACT]] : i1 to i32
+// CHECK: %[[RES:.*]] = llvm.add %[[Q]], %[[ADJUSTMENT]] : i32
   %0 = arith.ceildivui %arg0, %arg0 : i32
   return %0: i32
 }
@@ -798,17 +794,15 @@ func.func @ceildivui(%arg0 : i32) -> i32 {
 // CHECK-LABEL: @floordivsi
 // CHECK-SAME: %[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32) -> i32
 func.func @floordivsi(%arg0 : i32, %arg1 : i32) -> i32 {
-  // CHECK: %[[SDIV:.*]] = llvm.sdiv %[[ARG0]], %[[ARG1]] : i32
-  // CHECK: %[[MUL0:.*]] = llvm.mul %[[SDIV]], %[[ARG1]] : i32
-  // CHECK: %[[CMP0:.*]] = llvm.icmp "ne" %[[ARG0]], %[[MUL0]] : i32
-  // CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
-  // CHECK: %[[CMP1:.*]] = llvm.icmp "slt" %[[ARG0]], %[[CST0]] : i32
-  // CHECK: %[[CMP2:.*]] = llvm.icmp "slt" %[[ARG1]], %[[CST0]] : i32
-  // CHECK: %[[CMP3:.*]] = llvm.icmp "ne" %[[CMP1]], %[[CMP2]] : i1
-  // CHECK: %[[AND:.*]] = llvm.and %[[CMP0]], %[[CMP3]] : i1
-  // CHECK: %[[CST1:.*]] = llvm.mlir.constant(-1 : i32) : i32
-  // CHECK: %[[ADD:.*]] = llvm.add %[[SDIV]], %[[CST1]] : i32
-  // CHECK: %[[SEL:.*]] = llvm.select %[[AND]], %[[ADD]], %[[SDIV]] : i1, i32
+  // CHECK: %[[Q:.*]] = llvm.sdiv %[[ARG0]], %[[ARG1]] : i32
+  // CHECK: %[[PRODUCT:.*]] = llvm.mul %[[Q]], %[[ARG1]] : i32
+  // CHECK: %[[INEXACT:.*]] = llvm.icmp "ne" %[[PRODUCT]], %[[ARG0]] : i32
+  // CHECK: %[[SIGNED_LT:.*]] = llvm.icmp "slt" %[[ARG0]], %[[ARG1]] : i32
+  // CHECK: %[[UNSIGNED_LT:.*]] = llvm.icmp "ult" %[[ARG0]], %[[ARG1]] : i32
+  // CHECK: %[[OPPOSITE_SIGN:.*]] = llvm.icmp "ne" %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+  // CHECK: %[[ROUND:.*]] = llvm.and %[[INEXACT]], %[[OPPOSITE_SIGN]] : i1
+  // CHECK: %[[ADJUSTMENT:.*]] = llvm.zext %[[ROUND]] : i1 to i32
+  // CHECK: %[[RES:.*]] = llvm.sub %[[Q]], %[[ADJUSTMENT]] : i32
   %0 = arith.floordivsi %arg0, %arg1 : i32
   return %0 : i32
 }
diff --git a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
index fa204c4df8ace..58e5ac10d3a1b 100644
--- a/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
+++ b/mlir/test/Conversion/ConvertToSPIRV/arith.mlir
@@ -37,11 +37,13 @@ func.func @int32_scalar_srem(%lhs: i32, %rhs: i32) {
 // CHECK-LABEL: @scalar_ceildivui
 // CHECK-SAME: (%[[LHS:.+]]: i32, %[[RHS:.+]]: i32)
 func.func @scalar_ceildivui(%lhs: i32, %rhs: i32) -> i32 {
-  // CHECK:     %[[ISZERO:.+]] = spirv.IEqual %[[LHS]], %{{.+}} : i32
-  // CHECK:     %[[MINUSONE:.+]] = spirv.ISub %[[LHS]], %{{.+}} : i32
-  // CHECK:     %[[Q:.+]] = spirv.UDiv %[[MINUSONE]], %[[RHS]] : i32
-  // CHECK:     %[[PLUSONE:.+]] = spirv.IAdd %[[Q]], %{{.+}} : i32
-  // CHECK:     %[[R:.+]] = spirv.Select %[[ISZERO]], %{{.+}}, %[[PLUSONE]] : i1, i32
+  // CHECK:     %[[Q:.+]] = spirv.UDiv %[[LHS]], %[[RHS]] : i32
+  // CHECK:     %[[PRODUCT:.+]] = spirv.IMul %[[Q]], %[[RHS]] : i32
+  // CHECK:     %[[INEXACT:.+]] = spirv.INotEqual %[[PRODUCT]], %[[LHS]] : i32
+  // CHECK:     %[[ZERO:.+]] = spirv.Constant 0 : i32
+  // CHECK:     %[[ONE:.+]] = spirv.Constant 1 : i32
+  // CHECK:     %[[ADJUSTMENT:.+]] = spirv.Select %[[INEXACT]], %[[ONE]], %[[ZERO]] : i1, i32
+  // CHECK:     %[[R:.+]] = spirv.IAdd %[[Q]], %[[ADJUSTMENT]] : i32
   // CHECK:     spirv.ReturnValue %[[R]]
   %0 = arith.ceildivui %lhs, %rhs : i32
   return %0 : i32
@@ -51,14 +53,16 @@ func.func @scalar_ceildivui(%lhs: i32, %rhs: i32) -> i32 {
 // CHECK-SAME: (%[[LHS:.+]]: i32, %[[RHS:.+]]: i32)
 func.func @scalar_ceildivsi(%lhs: i32, %rhs: i32) -> i32 {
   // CHECK:     %[[Q:.+]] = spirv.SDiv %[[LHS]], %[[RHS]] : i32
-  // CHECK:     %[[PROD:.+]] = spirv.IMul %[[Q]], %[[RHS]] : i32
-  // CHECK:     %[[NE:.+]] = spirv.INotEqual %[[LHS]], %[[PROD]] : i32
-  // CHECK:     %[[LNEG:.+]] = spirv.SLessThan %[[LHS]], %{{.+}} : i32
-  // CHECK:     %[[RNEG:.+]] = spirv.SLessThan %[[RHS]], %{{.+}} : i32
-  // CHECK:     %[[SAMESIGN:.+]] = spirv.LogicalEqual %[[LNEG]], %[[RNEG]] : i1
-  // CHECK:     %[[COND:.+]] = spirv.LogicalAnd %[[NE]], %[[SAMESIGN]] : i1
-  // CHECK:     %[[QP1:.+]] = spirv.IAdd %[[Q]], %{{.+}} : i32
-  // CHECK:     %[[R:.+]] = spirv.Select %[[COND]], %[[QP1]], %[[Q]] : i1, i32
+  // CHECK:     %[[PRODUCT:.+]] = spirv.IMul %[[Q]], %[[RHS]] : i32
+  // CHECK:     %[[INEXACT:.+]] = spirv.INotEqual %[[PRODUCT]], %[[LHS]] : i32
+  // CHECK:     %[[SIGNED_LT:.+]] = spirv.SLessThan %[[LHS]], %[[RHS]] : i32
+  // CHECK:     %[[UNSIGNED_LT:.+]] = spirv.ULessThan %[[LHS]], %[[RHS]] : i32
+  // CHECK:     %[[SAME_SIGN:.+]] = spirv.LogicalEqual %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+  // CHECK:     %[[ROUND:.+]] = spirv.LogicalAnd %[[INEXACT]], %[[SAME_SIGN]] : i1
+  // CHECK:     %[[ZERO:.+]] = spirv.Constant 0 : i32
+  // CHECK:     %[[ONE:.+]] = spirv.Constant 1 : i32
+  // CHECK:     %[[ADJUSTMENT:.+]] = spirv.Select %[[ROUND]], %[[ONE]], %[[ZERO]] : i1, i32
+  // CHECK:     %[[R:.+]] = spirv.IAdd %[[Q]], %[[ADJUSTMENT]] : i32
   // CHECK:     spirv.ReturnValue %[[R]]
   %0 = arith.ceildivsi %lhs, %rhs : i32
   return %0 : i32
@@ -68,14 +72,16 @@ func.func @scalar_ceildivsi(%lhs: i32, %rhs: i32) -> i32 {
 // CHECK-SAME: (%[[LHS:.+]]: i32, %[[RHS:.+]]: i32)
 func.func @scalar_floordivsi(%lhs: i32, %rhs: i32) -> i32 {
   // CHECK:     %[[Q:.+]] = spirv.SDiv %[[LHS]], %[[RHS]] : i32
-  // CHECK:     %[[PROD:.+]] = spirv.IMul %[[Q]], %[[RHS]] : i32
-  // CHECK:     %[[NE:.+]] = spirv.INotEqual %[[LHS]], %[[PROD]] : i32
-  // CHECK:     %[[LNEG:.+]] = spirv.SLessThan %[[LHS]], %{{.+}} : i32
-  // CHECK:     %[[RNEG:.+]] = spirv.SLessThan %[[RHS]], %{{.+}} : i32
-  // CHECK:     %[[DIFFSIGN:.+]] = spirv.LogicalNotEqual %[[LNEG]], %[[RNEG]] : i1
-  // CHECK:     %[[COND:.+]] = spirv.LogicalAnd %[[NE]], %[[DIFFSIGN]] : i1
-  // CHECK:     %[[QM1:.+]] = spirv.IAdd %[[Q]], %{{.+}} : i32
-  // CHECK:     %[[R:.+]] = spirv.Select %[[COND]], %[[QM1]], %[[Q]] : i1, i32
+  // CHECK:     %[[PRODUCT:.+]] = spirv.IMul %[[Q]], %[[RHS]] : i32
+  // CHECK:     %[[INEXACT:.+]] = spirv.INotEqual %[[PRODUCT]], %[[LHS]] : i32
+  // CHECK:     %[[SIGNED_LT:.+]] = spirv.SLessThan %[[LHS]], %[[RHS]] : i32
+  // CHECK:     %[[UNSIGNED_LT:.+]] = spirv.ULessThan %[[LHS]], %[[RHS]] : i32
+  // CHECK:     %[[OPPOSITE_SIGN:.+]] = spirv.LogicalNotEqual %[[SIGNED_LT]], %[[UNSIGNED_LT]] : i1
+  // CHECK:     %[[ROUND:.+]] = spirv.LogicalAnd %[[INEXACT]], %[[OPPOSITE_SIGN]] : i1
+  // CHECK:     %[[ZERO:.+]] = spirv.Constant 0 : i32
+  // CHECK:     %[[ONE:.+]] = spirv.Constant 1 : i32
+  // CHECK:     %[[ADJUSTMENT:.+]] = spirv.Select %[[ROUND]], %[[ONE]], %[[ZERO]] : i1, i32
+  // CHECK:     %[[R:.+]] = spirv.ISub %[[Q]], %[[ADJUSTMENT]] : i32
   // CHECK:     spirv.ReturnValue %[[R]]
   %0 = arith.floordivsi %lhs, %rhs : i32
   return %0 : i32
@@ -84,7 +90,8 @@ func.func @scalar_floordivsi(%lhs: i32, %rhs: i32) -> i32 {
 // CHECK-LABEL: @vector_ceildivsi
 func.func @vector_ceildivsi(%lhs: vector<4xi32>, %rhs: vector<4xi32>) -> vector<4xi32> {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}} : vector<4xi32>
-  // CHECK: spirv.Select %{{.*}}, %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: %[[ADJUSTMENT:.+]] = spirv.Select %{{.*}}, %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: spirv.IAdd %{{.*}}, %[[ADJUSTMENT]] : vector<4xi32>
   %0 = arith.ceildivsi %lhs, %rhs : vector<4xi32>
   return %0 : vector<4xi32>
 }

>From 9595a0c00124be170b8512a650dbb0db871d4256 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Tue, 14 Jul 2026 16:50:27 +0800
Subject: [PATCH 7/8] [MLIR][SPIR-V] Support boolean index casts

---
 .../Conversion/ArithToSPIRV/ArithToSPIRV.cpp  |  3 +-
 .../ArithToSPIRV/arith-to-spirv.mlir          | 28 ++++++++++++
 .../ArithToSPIRV/ceil-floor-div-index.mlir    | 45 +++++++++++++++++++
 3 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/ArithToSPIRV/ceil-floor-div-index.mlir

diff --git a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
index 9a6d330db72fe..afe8eaab861b0 100644
--- a/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
+++ b/mlir/lib/Conversion/ArithToSPIRV/ArithToSPIRV.cpp
@@ -675,7 +675,7 @@ struct ShRSIBoolPattern final : public OpConversionPattern<arith::ShRSIOp> {
 
 /// Converts an op whose i1 (or vector of i1) source selects between one and
 /// zero of the destination type, i.e. spirv.Select(src, one, zero). Shared by
-/// arith.uitofp, arith.extui, and arith.index_cast on boolean sources.
+/// arith.uitofp, arith.extui, and index casts on boolean sources.
 template <typename ArithOp>
 struct BoolToValuePattern final : public OpConversionPattern<ArithOp> {
   using OpConversionPattern<ArithOp>::OpConversionPattern;
@@ -1508,6 +1508,7 @@ void mlir::arith::populateArithToSPIRVPatterns(
     TypeCastingOpPattern<arith::IndexCastOp, spirv::SConvertOp>,
     IndexCastIndexI1Pattern, BoolToValuePattern<arith::IndexCastOp>,
     TypeCastingOpPattern<arith::IndexCastUIOp, spirv::UConvertOp>,
+    BoolToValuePattern<arith::IndexCastUIOp>,
     TypeCastingOpPattern<arith::BitcastOp, spirv::BitcastOp>,
     CmpIOpBooleanPattern, CmpIOpPattern,
     CmpFOpNanNonePattern, CmpFOpPattern,
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index b6a488f0dad73..b8dbce76331d8 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -893,6 +893,34 @@ func.func @index_casti1index_3(%arg0 : vector<3xi1>) {
   return
 }
 
+// CHECK-LABEL: index_castuii1index_1
+func.func @index_castuii1index_1(%arg0 : i1) {
+  // CHECK: %[[ZERO:.+]] = spirv.Constant 0 : i32
+  // CHECK: %[[ONE:.+]] = spirv.Constant 1 : i32
+  // CHECK: spirv.Select %{{.+}}, %[[ONE]], %[[ZERO]] : i1, i32
+  %0 = arith.index_castui %arg0 : i1 to index
+  return
+}
+
+// CHECK-LABEL: index_castuii1index_2
+func.func @index_castuii1index_2(%arg0 : vector<1xi1>) -> vector<1xindex> {
+  // Single-element vectors do not exist in SPIRV.
+  // CHECK: %[[ZERO:.+]] = spirv.Constant 0 : i32
+  // CHECK: %[[ONE:.+]] = spirv.Constant 1 : i32
+  // CHECK: spirv.Select %{{.+}}, %[[ONE]], %[[ZERO]] : i1, i32
+  %0 = arith.index_castui %arg0 : vector<1xi1> to vector<1xindex>
+  return %0 : vector<1xindex>
+}
+
+// CHECK-LABEL: index_castuii1index_3
+func.func @index_castuii1index_3(%arg0 : vector<3xi1>) {
+  // CHECK: %[[ZERO:.+]] = spirv.Constant dense<0> : vector<3xi32>
+  // CHECK: %[[ONE:.+]] = spirv.Constant dense<1> : vector<3xi32>
+  // CHECK: spirv.Select %{{.+}}, %[[ONE]], %[[ZERO]] : vector<3xi1>, vector<3xi32>
+  %0 = arith.index_castui %arg0 : vector<3xi1> to vector<3xindex>
+  return
+}
+
 // CHECK-LABEL: @bit_cast
 func.func @bit_cast(%arg0: vector<2xf32>, %arg1: i64) {
   // CHECK: spirv.Bitcast %{{.+}} : vector<2xf32> to vector<2xi32>
diff --git a/mlir/test/Conversion/ArithToSPIRV/ceil-floor-div-index.mlir b/mlir/test/Conversion/ArithToSPIRV/ceil-floor-div-index.mlir
new file mode 100644
index 0000000000000..961127306468a
--- /dev/null
+++ b/mlir/test/Conversion/ArithToSPIRV/ceil-floor-div-index.mlir
@@ -0,0 +1,45 @@
+// RUN: mlir-opt %s -arith-expand -convert-arith-to-spirv | FileCheck %s
+
+// CHECK-LABEL: func.func @ceildivui_index
+// CHECK: %[[Q:.*]] = spirv.UDiv
+// CHECK: %[[PRODUCT:.*]] = spirv.IMul %[[Q]]
+// CHECK: %[[INEXACT:.*]] = spirv.INotEqual %[[PRODUCT]]
+// CHECK: %[[ADJUSTMENT:.*]] = spirv.Select %[[INEXACT]]
+// CHECK: %[[RESULT:.*]] = spirv.IAdd %[[Q]], %[[ADJUSTMENT]]
+// CHECK: return %{{.*}} : index
+func.func @ceildivui_index(%lhs: index, %rhs: index) -> index {
+  %result = arith.ceildivui %lhs, %rhs : index
+  return %result : index
+}
+
+// CHECK-LABEL: func.func @ceildivsi_index
+// CHECK: %[[Q:.*]] = spirv.SDiv
+// CHECK: %[[PRODUCT:.*]] = spirv.IMul %[[Q]]
+// CHECK: %[[INEXACT:.*]] = spirv.INotEqual %[[PRODUCT]]
+// CHECK: %[[SIGNED_LT:.*]] = spirv.SLessThan
+// CHECK: %[[UNSIGNED_LT:.*]] = spirv.ULessThan
+// CHECK: %[[SAME_SIGN:.*]] = spirv.LogicalEqual %[[SIGNED_LT]], %[[UNSIGNED_LT]]
+// CHECK: %[[ROUND:.*]] = spirv.LogicalAnd %[[INEXACT]], %[[SAME_SIGN]]
+// CHECK: %[[ADJUSTMENT:.*]] = spirv.Select %[[ROUND]]
+// CHECK: %[[RESULT:.*]] = spirv.IAdd %[[Q]], %[[ADJUSTMENT]]
+// CHECK: return %{{.*}} : index
+func.func @ceildivsi_index(%lhs: index, %rhs: index) -> index {
+  %result = arith.ceildivsi %lhs, %rhs : index
+  return %result : index
+}
+
+// CHECK-LABEL: func.func @floordivsi_index
+// CHECK: %[[Q:.*]] = spirv.SDiv
+// CHECK: %[[PRODUCT:.*]] = spirv.IMul %[[Q]]
+// CHECK: %[[INEXACT:.*]] = spirv.INotEqual %[[PRODUCT]]
+// CHECK: %[[SIGNED_LT:.*]] = spirv.SLessThan
+// CHECK: %[[UNSIGNED_LT:.*]] = spirv.ULessThan
+// CHECK: %[[OPPOSITE_SIGN:.*]] = spirv.LogicalNotEqual %[[SIGNED_LT]], %[[UNSIGNED_LT]]
+// CHECK: %[[ROUND:.*]] = spirv.LogicalAnd %[[INEXACT]], %[[OPPOSITE_SIGN]]
+// CHECK: %[[ADJUSTMENT:.*]] = spirv.Select %[[ROUND]]
+// CHECK: %[[RESULT:.*]] = spirv.ISub %[[Q]], %[[ADJUSTMENT]]
+// CHECK: return %{{.*}} : index
+func.func @floordivsi_index(%lhs: index, %rhs: index) -> index {
+  %result = arith.floordivsi %lhs, %rhs : index
+  return %result : index
+}

>From 9a358571792d420092cbfb666700a3ff491b86b2 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Tue, 14 Jul 2026 16:50:48 +0800
Subject: [PATCH 8/8] [MLIR][Affine] Update division rationale

---
 mlir/lib/Dialect/Affine/Utils/Utils.cpp | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/Utils/Utils.cpp b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
index 7043083298615..4c4f6981fac74 100644
--- a/mlir/lib/Dialect/Affine/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
@@ -113,12 +113,9 @@ class AffineApplyExpander
   ///            let quotient = absolute / b in
   ///                negative ? -quotient - 1 : quotient
   ///
-  /// Note: this lowering does not use arith.floordivsi because the lowering of
-  /// that to arith.divsi (see populateCeilFloorDivExpandOpsPatterns) generates
-  /// not one but two arith.divsi. That could be changed to one divsi, but one
-  /// way or another, going through arith.floordivsi will result in more complex
-  /// IR because arith.floordivsi is more general than affine floordiv in that
-  /// it supports negative RHS.
+  /// Note: this lowering does not use arith.floordivsi because that operation
+  /// is more general than affine floordiv: it also supports negative RHS. The
+  /// dedicated positive-divisor lowering therefore produces simpler IR.
   Value visitFloorDivExpr(AffineBinaryOpExpr expr) {
     if (auto rhsConst = dyn_cast<AffineConstantExpr>(expr.getRHS())) {
       if (rhsConst.getValue() <= 0) {


From llvmlistbot at llvm.org  Tue Jul 14 02:04:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:04:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix arith tensor constants (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fbab.f732ad81.3a4994.fa0f@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:04:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:04:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fbb6.6c8668f2.e4f82.cf2d@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:08:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:08:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] d4568c9 - [mlir][SPIR-V] Collapse
 TanPattern/TanhPattern into DirectConversionPattern (#203830)
Message-ID: <6a55fc83.92d2873c.258297.9d07@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-14T11:08:14+02:00
New Revision: d4568c9fd15235bd4545a5c57f23bfcb42673f39

URL: https://github.com/llvm/llvm-project/commit/d4568c9fd15235bd4545a5c57f23bfcb42673f39
DIFF: https://github.com/llvm/llvm-project/commit/d4568c9fd15235bd4545a5c57f23bfcb42673f39.diff

LOG: [mlir][SPIR-V] Collapse TanPattern/TanhPattern into DirectConversionPattern (#203830)

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index f8c4405157f94..c5f9305a5c226 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1560,41 +1560,6 @@ class ShiftPattern : public SPIRVToLLVMConversion<SPIRVOp> {
   }
 };
 
-class TanPattern : public SPIRVToLLVMConversion<spirv::GLTanOp> {
-public:
-  using SPIRVToLLVMConversion<spirv::GLTanOp>::SPIRVToLLVMConversion;
-
-  LogicalResult
-  matchAndRewrite(spirv::GLTanOp tanOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto dstType = getTypeConverter()->convertType(tanOp.getType());
-    if (!dstType)
-      return rewriter.notifyMatchFailure(tanOp, "type conversion failed");
-
-    rewriter.replaceOpWithNewOp<LLVM::TanOp>(tanOp, dstType,
-                                             adaptor.getOperands());
-    return success();
-  }
-};
-
-class TanhPattern : public SPIRVToLLVMConversion<spirv::GLTanhOp> {
-public:
-  using SPIRVToLLVMConversion<spirv::GLTanhOp>::SPIRVToLLVMConversion;
-
-  LogicalResult
-  matchAndRewrite(spirv::GLTanhOp tanhOp, OpAdaptor adaptor,
-                  ConversionPatternRewriter &rewriter) const override {
-    auto srcType = tanhOp.getType();
-    auto dstType = getTypeConverter()->convertType(srcType);
-    if (!dstType)
-      return rewriter.notifyMatchFailure(tanhOp, "type conversion failed");
-
-    rewriter.replaceOpWithNewOp<LLVM::TanhOp>(tanhOp, dstType,
-                                              adaptor.getOperands());
-    return success();
-  }
-};
-
 // `llvm.intr.abs` requires an `is_int_min_poison` immarg that `spirv.GL.SAbs`
 // does not carry; default to `false` to preserve SPIR-V's well-defined
 // behavior on INT_MIN.
@@ -2070,8 +2035,9 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAsinOp, LLVM::ASinOp>,
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
-      InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern, FractPattern,
-      GLFMixPattern,
+      DirectConversionPattern<spirv::GLTanOp, LLVM::TanOp>,
+      DirectConversionPattern<spirv::GLTanhOp, LLVM::TanhOp>,
+      InverseSqrtPattern, SAbsPattern, FractPattern, GLFMixPattern,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,


        

From llvmlistbot at llvm.org  Tue Jul 14 02:08:22 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Tue, 14 Jul 2026 02:08:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Collapse
 TanPattern/TanhPattern into DirectConversionPattern (PR #203830)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203830@github.com>
Message-ID: <6a55fc86.510f576d.1b3cb9.d6ea@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/203830

From llvmlistbot at llvm.org  Tue Jul 14 02:09:05 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 02:09:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a55fcb1.481afadf.196d57.6554@mx.google.com>


================
@@ -6787,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
----------------
banach-space wrote:

> Would it be possible that we return true for a case whose types look like a broadcast but it's actually representing something else under the shape cast semantics?

I don't think so and that's for two reasons:
* `ShapeCastOp` always preserves the number of elements.
* `BroadcastOp` will only preserve the number of elements if it is merely adding leading unit dims (i.e. it's quite an edge case).

Let me add more checks and comments to make it clearer. I don't that's going to make the method stricter, but definitely clearer.

https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Tue Jul 14 02:10:56 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 02:10:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a55fd20.f1c1923d.991d1.b821@mx.google.com>


https://github.com/banach-space updated https://github.com/llvm/llvm-project/pull/208752

>From 643775665538881e17a0ddb2337ae7f170a30bcb Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 10 Jul 2026 14:56:23 +0000
Subject: [PATCH 1/2] [mlir][vector] Extend `combineContractAndBroadcast` to
 accept `vector.shape_cast`

`combineContractAndBroadcast` is a pattern that matches `vector.contract`
operations whose inputs are defined by `vector.broadcast`, e.g.:

```mlir
%0 = vector.broadcast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.broadcast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

The pattern folds away the reduction over a unit dimension introduced by
the defining operations, producing a simplified `vector.contract`:

```mlir
vector.contract {
  indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
  iterator_types = ["parallel", "parallel", "reduction"]
} %0, %1, %acc : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
```

This change extends the pattern to also accept broadcast-like
`vector.shape_cast` operations:

```mlir
%0 = vector.shape_cast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.shape_cast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

Note that `mlir::vector::isBroadcastableTo` was only moved to make it
available to the newly introduced `ShapeCastOp` helper,
`ShapeCastOp::isBroadcastLike()`.
---
 .../mlir/Dialect/Vector/IR/VectorOps.h        |   2 +-
 .../mlir/Dialect/Vector/IR/VectorOps.td       |   3 +
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 121 ++++++++++--------
 .../Vector/Transforms/VectorTransforms.cpp    |  36 ++++--
 .../Vector/vector-reduce-to-contract.mlir     | 103 ++++++++++++++-
 5 files changed, 194 insertions(+), 71 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
index b3a0653b90765..1a7d37d989f5a 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
@@ -67,7 +67,7 @@ enum class ConstantMaskKind { AllFalse = 0, AllTrue };
 /// arguments.
 void buildTerminatedBody(OpBuilder &builder, Location loc);
 
-/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// Models whether `srcType` can be broadcast to `dstVectorType` under the
 /// semantics of the `vector.broadcast` op.
 enum class BroadcastableToResult {
   Success = 0,
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index a519e6d89a472..f2a8ce2951106 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2484,6 +2484,9 @@ def Vector_ShapeCastOp :
     VectorType getResultVectorType() {
       return ::llvm::cast<VectorType>(getResult().getType());
     }
+    // Return true if this Op is effectively a vector.broadcast (i.e. the input
+    // and output shapes satisfy the vector.broadcast constraints).
+    bool isBroadcastLike();
   }];
   let assemblyFormat = "$source attr-dict `:` type($source) `to` type($result)";
   let hasFolder = 1;
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index e0cb9923b1dd4..fb535cdae461f 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -429,6 +429,63 @@ static Attribute convertNumericAttr(Attribute attr, Type expectedType) {
   return attr;
 }
 
+/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// semantics of the `vector.broadcast` op.
+BroadcastableToResult mlir::vector::isBroadcastableTo(
+    Type srcType, VectorType dstVectorType,
+    std::pair<VectorDim, VectorDim> *mismatchingDims) {
+  // Broadcast scalar to vector of the same element type.
+  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
+      srcType == getElementTypeOrSelf(dstVectorType))
+    return BroadcastableToResult::Success;
+  // From now on, only vectors broadcast.
+  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
+  if (!srcVectorType)
+    return BroadcastableToResult::SourceTypeNotAVector;
+
+  int64_t srcRank = srcVectorType.getRank();
+  int64_t dstRank = dstVectorType.getRank();
+  if (srcRank > dstRank)
+    return BroadcastableToResult::SourceRankHigher;
+  // Source has an exact match or singleton value for all trailing dimensions
+  // (all leading dimensions are simply duplicated).
+  int64_t lead = dstRank - srcRank;
+  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
+    // Have mismatching dims (in the sense of vector.broadcast semantics) been
+    // encountered?
+    bool foundMismatchingDims = false;
+
+    // Check fixed-width dims.
+    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
+    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
+    if (srcDim != 1 && srcDim != dstDim)
+      foundMismatchingDims = true;
+
+    // Check scalable flags.
+    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
+    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
+    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
+        // 1 -> [N] is fine, everything else should be rejected when mixing
+        // fixed-width and scalable dims
+        (srcDimScalableFlag != dstDimScalableFlag &&
+         (srcDim != 1 || srcDimScalableFlag)))
+      foundMismatchingDims = true;
+
+    if (foundMismatchingDims) {
+      if (mismatchingDims != nullptr) {
+        mismatchingDims->first.dim = srcDim;
+        mismatchingDims->first.isScalable = srcDimScalableFlag;
+
+        mismatchingDims->second.dim = dstDim;
+        mismatchingDims->second.isScalable = dstDimScalableFlag;
+      }
+      return BroadcastableToResult::DimensionMismatch;
+    }
+  }
+
+  return BroadcastableToResult::Success;
+}
+
 //===----------------------------------------------------------------------===//
 // CombiningKindAttr
 //===----------------------------------------------------------------------===//
@@ -3101,61 +3158,6 @@ Value BroadcastOp::createOrFoldBroadcastOp(
   return res;
 }
 
-BroadcastableToResult mlir::vector::isBroadcastableTo(
-    Type srcType, VectorType dstVectorType,
-    std::pair<VectorDim, VectorDim> *mismatchingDims) {
-  // Broadcast scalar to vector of the same element type.
-  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
-      srcType == getElementTypeOrSelf(dstVectorType))
-    return BroadcastableToResult::Success;
-  // From now on, only vectors broadcast.
-  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
-  if (!srcVectorType)
-    return BroadcastableToResult::SourceTypeNotAVector;
-
-  int64_t srcRank = srcVectorType.getRank();
-  int64_t dstRank = dstVectorType.getRank();
-  if (srcRank > dstRank)
-    return BroadcastableToResult::SourceRankHigher;
-  // Source has an exact match or singleton value for all trailing dimensions
-  // (all leading dimensions are simply duplicated).
-  int64_t lead = dstRank - srcRank;
-  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
-    // Have mismatching dims (in the sense of vector.broadcast semantics) been
-    // encountered?
-    bool foundMismatchingDims = false;
-
-    // Check fixed-width dims.
-    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
-    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
-    if (srcDim != 1 && srcDim != dstDim)
-      foundMismatchingDims = true;
-
-    // Check scalable flags.
-    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
-    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
-    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
-        // 1 -> [N] is fine, everything else should be rejected when mixing
-        // fixed-width and scalable dims
-        (srcDimScalableFlag != dstDimScalableFlag &&
-         (srcDim != 1 || srcDimScalableFlag)))
-      foundMismatchingDims = true;
-
-    if (foundMismatchingDims) {
-      if (mismatchingDims != nullptr) {
-        mismatchingDims->first.dim = srcDim;
-        mismatchingDims->first.isScalable = srcDimScalableFlag;
-
-        mismatchingDims->second.dim = dstDim;
-        mismatchingDims->second.isScalable = dstDimScalableFlag;
-      }
-      return BroadcastableToResult::DimensionMismatch;
-    }
-  }
-
-  return BroadcastableToResult::Success;
-}
-
 LogicalResult BroadcastOp::verify() {
   std::pair<VectorDim, VectorDim> mismatchingDims;
   BroadcastableToResult res = isBroadcastableTo(
@@ -6757,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
+         BroadcastableToResult::Success;
+}
+
 /// Return true if `transpose` does not permute a pair of non-unit dims.
 /// By `order preserving` we mean that the flattened versions of the input and
 /// output vectors are (numerically) identical. In other words `transpose` is
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 752610efc6992..22e7742128ff2 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -282,21 +282,32 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   bool changed = false;
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
+
+    // Make sure that the source
+    auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
-    if (!broadcast)
+    if (!broadcast && !sc)
       continue;
+
+    if (sc && !sc.isBroadcastLike())
+      return rewriter.notifyMatchFailure(
+          contractOp, "Operand defined via vector.shape_cast that has "
+                      "non-broadcast semantics");
+
+    VectorType srcType = (sc) ? sc.getSourceVectorType()
+                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    VectorType resType =
+        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+
     // contractionOp can only take vector as operands.
-    auto srcType = dyn_cast<VectorType>(broadcast.getSourceType());
-    if (!srcType ||
-        srcType.getRank() == broadcast.getResultVectorType().getRank())
+    // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());
+    if (!srcType || srcType.getRank() >= resType.getRank())
       continue;
-    int64_t rankDiff =
-        broadcast.getResultVectorType().getRank() - srcType.getRank();
+    int64_t rankDiff = resType.getRank() - srcType.getRank();
     bool innerDimBroadcast = false;
     SmallVector<AffineExpr> originalDims;
     for (const auto &dim : llvm::enumerate(srcType.getShape())) {
-      if (dim.value() !=
-          broadcast.getResultVectorType().getDimSize(rankDiff + dim.index())) {
+      if (dim.value() != resType.getDimSize(rankDiff + dim.index())) {
         innerDimBroadcast = true;
         break;
       }
@@ -311,7 +322,7 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     // of non-unit size.
     bool nonUnitDimReductionBroadcast = false;
     for (int64_t i = 0; i < rankDiff; ++i) {
-      if (broadcast.getResultVectorType().getDimSize(i) != 1 &&
+      if (resType.getDimSize(i) != 1 &&
           isReductionIterator(contractOp.getIteratorTypes()
                                   .getValue()[map.getDimPosition(i)])) {
         nonUnitDimReductionBroadcast = true;
@@ -321,11 +332,10 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     if (nonUnitDimReductionBroadcast)
       continue;
 
-    AffineMap broadcastMap =
-        AffineMap::get(broadcast.getResultVectorType().getRank(), 0,
-                       originalDims, contractOp.getContext());
+    AffineMap broadcastMap = AffineMap::get(resType.getRank(), 0, originalDims,
+                                            contractOp.getContext());
     map = broadcastMap.compose(map);
-    *operand = broadcast.getSource();
+    *operand = broadcast ? broadcast.getSource() : sc.getSource();
     changed = true;
   }
 
diff --git a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
index 3b51e6b1e1b6f..8e53a815f70f8 100644
--- a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
+++ b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
@@ -202,7 +202,37 @@ func.func @contract_broadcast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1
 
 // -----
 
-// Same as above, but with a mask.
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but
+// `vector.broadcast` is replaced with `vector.shape_cast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>)
+//  CHECK: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but with a mask.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -235,7 +265,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>,
 
 // -----
 
-// Same as above, but with a scalable dim.
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with
+// `vector.shape_cast` instead of `vector.broadcast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>, %[[MASK:.+]]: vector<1x8x8x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x8x4xi1> to vector<8x8x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>, %mask: vector<1x8x8x4xi1>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    } : vector<1x8x8x4xi1> -> vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with a scalable dim.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -267,6 +331,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked_scalable(%arg0 : vector<
 }
 
 // -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked_scalable`), but with
+// vector.shape_cast instead of vector.broadcast.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked_scalable
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<[8]x4xi32>, %[[ARG2:.+]]: vector<8x[8]xi32>, %[[MASK:.+]]: vector<1x8x[8]x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x[8]x4xi1> to vector<8x[8]x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<[8]x4xi32> into vector<8x[8]xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked_scalable(%arg0 : vector<8x4xi32>, %arg1 : vector<[8]x4xi32>, %arg2 : vector<8x[8]xi32>, %mask: vector<1x8x[8]x4xi1>) -> vector<8x[8]xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<[8]x4xi32> to vector<1x[8]x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x[8]x4xi32> into vector<8x[8]xi32>
+    } : vector<1x8x[8]x4xi1> -> vector<8x[8]xi32>
+    return %result : vector<8x[8]xi32>
+}
+
+// -----
+
 // Test that CombineContractBroadcast will not combine a broadcast that creates
 // a non-unit dim that is consumed by a reduction iterator.
 // Moreover, the affine_map's are permuting the position of that reduction

>From 5a9232f71f80268e93cfd7790a2229b839c43fae Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Tue, 14 Jul 2026 09:09:30 +0000
Subject: [PATCH 2/2] Address PR comments

---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 28 +++++++++++++++++--
 .../Vector/Transforms/VectorTransforms.cpp    | 10 ++++---
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index fb535cdae461f..dd1a3895ad669 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6759,13 +6759,37 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+/// Check whether this ShapeCastOp is effectively a BroadcastOp.
+///
+/// The only case in which this method can return `true` is when the underlying
+/// op merely adds leading unit dimensions, e.g.:
+///   %res = vector.shape_cast %src : vector<8x4xi32> to vector<1x8x4xi32>
+///
 bool ShapeCastOp::isBroadcastLike() {
   auto srcType = getSourceVectorType();
   auto resType = getResultVectorType();
 
+  // Is srcType broadcastable to resType?
   std::pair<VectorDim, VectorDim> mismatchingDims;
-  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
-         BroadcastableToResult::Success;
+  if (isBroadcastableTo(srcType, resType, &mismatchingDims) !=
+      BroadcastableToResult::Success)
+    return false;
+
+  // Do ranks mismatch?
+  //
+  // The only case where ranks match and this ShapeCastOp is also a broadcast,
+  // is when it's effectively a NOp, but that's an uninteresting edge case.
+  size_t rankDiff = resType.getRank() - srcType.getRank();
+  if (rankDiff == 0)
+    return false;
+
+  // Are all newly added leading dims unit?
+  if (!llvm::all_of(resType.getShape().take_front(rankDiff),
+                    [](int64_t dim) { return dim == 1; }))
+    return false;
+
+  // Do all trailing dims match?
+  return resType.getShape().take_back(srcType.getRank()) == srcType.getShape();
 }
 
 /// Return true if `transpose` does not permute a pair of non-unit dims.
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 22e7742128ff2..ad35465179e94 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -283,7 +283,8 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
 
-    // Make sure that the source
+    // Accept operands defined by vector.broadcast and broadcast-like
+    // vector.shape_cast.
     auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
     if (!broadcast && !sc)
@@ -294,10 +295,11 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
           contractOp, "Operand defined via vector.shape_cast that has "
                       "non-broadcast semantics");
 
-    VectorType srcType = (sc) ? sc.getSourceVectorType()
-                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    // Get the source and the result types.
+    VectorType srcType = sc ? sc.getSourceVectorType()
+                            : dyn_cast<VectorType>(broadcast.getSourceType());
     VectorType resType =
-        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+        sc ? sc.getResultVectorType() : broadcast.getResultVectorType();
 
     // contractionOp can only take vector as operands.
     // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());


From llvmlistbot at llvm.org  Tue Jul 14 02:11:14 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 02:11:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a55fd32.bd93e731.358318.1364@mx.google.com>


banach-space wrote:

Thank you for reviewing @dcaballe , PTAL :)

https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Tue Jul 14 02:15:28 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Tue, 14 Jul 2026 02:15:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for GL.FSign and GL.SSign (PR #206934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206934@github.com>
Message-ID: <6a55fe30.d1496011.26913c.8e4f@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206934

>From 2e1952b741d34c2b0dfc1887493498283a21a93c Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 1 Jul 2026 12:06:39 +0200
Subject: [PATCH] [mlir][SPIR-V] Add SPIRVToLLVM conversions for GL.FSign and
 GL.SSign

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    | 64 +++++++++++++++++++
 .../SPIRVToLLVM/gl-ops-to-llvm.mlir           | 54 ++++++++++++++++
 2 files changed, 118 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c43415b27b1b3..eb86d0e5c417f 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1564,6 +1564,68 @@ class SAbsPattern : public SPIRVToLLVMConversion<spirv::GLSAbsOp> {
   }
 };
 
+/// Converts `spirv.GL.FSign`/`spirv.GL.SSign` to a sign(x) sequence that maps
+/// the operand to -1/0/1 using two comparisons and two selects. The `isFloat`
+/// flag selects between floating-point and integer comparisons/constants.
+template <typename SPIRVOp, bool isFloat>
+class SignPattern : public SPIRVToLLVMConversion<SPIRVOp> {
+public:
+  using SPIRVToLLVMConversion<SPIRVOp>::SPIRVToLLVMConversion;
+
+  LogicalResult
+  matchAndRewrite(SPIRVOp op, typename SPIRVOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto srcType = op.getType();
+    auto dstType = this->getTypeConverter()->convertType(srcType);
+    if (!dstType)
+      return rewriter.notifyMatchFailure(op, "type conversion failed");
+
+    Location loc = op.getLoc();
+    Value operand = adaptor.getOperand();
+    auto vecSrcType = dyn_cast<VectorType>(srcType);
+    Type i1Type = rewriter.getI1Type();
+    Type cmpType =
+        vecSrcType ? VectorType::get(vecSrcType.getShape(), i1Type) : i1Type;
+
+    Value zero, one, minusOne, gt, lt;
+    if constexpr (isFloat) {
+      zero = createFPConstant(loc, srcType, dstType, rewriter, 0.0);
+      one = createFPConstant(loc, srcType, dstType, rewriter, 1.0);
+      minusOne = createFPConstant(loc, srcType, dstType, rewriter, -1.0);
+      gt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::ogt, operand, zero);
+      lt = LLVM::FCmpOp::create(rewriter, loc, cmpType,
+                                LLVM::FCmpPredicate::olt, operand, zero);
+    } else {
+      auto intElemType = cast<IntegerType>(getElementTypeOrSelf(srcType));
+      IntegerAttr zeroAttr = rewriter.getIntegerAttr(intElemType, 0);
+      IntegerAttr oneAttr = rewriter.getIntegerAttr(intElemType, 1);
+      if (vecSrcType) {
+        zero = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, zeroAttr));
+        one = LLVM::ConstantOp::create(
+            rewriter, loc, dstType,
+            SplatElementsAttr::get(vecSrcType, oneAttr));
+      } else {
+        zero = LLVM::ConstantOp::create(rewriter, loc, dstType, zeroAttr);
+        one = LLVM::ConstantOp::create(rewriter, loc, dstType, oneAttr);
+      }
+      minusOne = createConstantAllBitsSet(loc, srcType, dstType, rewriter);
+      gt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::sgt, operand, zero);
+      lt = LLVM::ICmpOp::create(rewriter, loc, cmpType,
+                                LLVM::ICmpPredicate::slt, operand, zero);
+    }
+
+    Value negOrZero =
+        LLVM::SelectOp::create(rewriter, loc, dstType, lt, minusOne, zero);
+    rewriter.replaceOpWithNewOp<LLVM::SelectOp>(op, dstType, gt, one,
+                                                negOrZero);
+    return success();
+  }
+};
+
 class VariablePattern : public SPIRVToLLVMConversion<spirv::VariableOp> {
 public:
   using SPIRVToLLVMConversion<spirv::VariableOp>::SPIRVToLLVMConversion;
@@ -1917,6 +1979,8 @@ void mlir::populateSPIRVToLLVMConversionPatterns(
       DirectConversionPattern<spirv::GLAcosOp, LLVM::ACosOp>,
       DirectConversionPattern<spirv::GLAtanOp, LLVM::ATanOp>,
       InverseSqrtPattern, SAbsPattern, TanPattern, TanhPattern,
+      SignPattern<spirv::GLFSignOp, /*isFloat=*/true>,
+      SignPattern<spirv::GLSSignOp, /*isFloat=*/false>,
 
       // OpenCL extended instruction set ops
       DirectConversionPattern<spirv::CLCeilOp, LLVM::FCeilOp>,
diff --git a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
index ffa47efbf9213..99153da1e9509 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/gl-ops-to-llvm.mlir
@@ -361,3 +361,57 @@ spirv.func @asin_acos_atan(%arg0: f32, %arg1: vector<3xf16>) "None" {
   %2 = spirv.GL.Atan %arg0 : f32
   spirv.Return
 }
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.FSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @fsign_scalar
+spirv.func @fsign_scalar(%arg0: f32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0.000000e+00 : f32) : f32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1.000000e+00 : f32) : f32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1.000000e+00 : f32) : f32
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %[[ZERO]] : f32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, f32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, f32
+  %0 = spirv.GL.FSign %arg0 : f32
+  spirv.Return
+}
+
+// CHECK-LABEL: @fsign_vector
+spirv.func @fsign_vector(%arg0: vector<4xf32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.fcmp "ogt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[LT:.*]] = llvm.fcmp "olt" %{{.*}}, %{{.*}} : vector<4xf32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xf32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xf32>
+  %0 = spirv.GL.FSign %arg0 : vector<4xf32>
+  spirv.Return
+}
+
+//===----------------------------------------------------------------------===//
+// spirv.GL.SSign
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: @ssign_scalar
+spirv.func @ssign_scalar(%arg0: i32) "None" {
+  // CHECK: %[[ZERO:.*]] = llvm.mlir.constant(0 : i32) : i32
+  // CHECK: %[[ONE:.*]] = llvm.mlir.constant(1 : i32) : i32
+  // CHECK: %[[MONE:.*]] = llvm.mlir.constant(-1 : i32) : i32
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %[[ZERO]] : i32
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %[[MONE]], %[[ZERO]] : i1, i32
+  // CHECK: llvm.select %[[GT]], %[[ONE]], %[[SEL0]] : i1, i32
+  %0 = spirv.GL.SSign %arg0 : i32
+  spirv.Return
+}
+
+// CHECK-LABEL: @ssign_vector
+spirv.func @ssign_vector(%arg0: vector<4xi32>) "None" {
+  // CHECK: %[[GT:.*]] = llvm.icmp "sgt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[LT:.*]] = llvm.icmp "slt" %{{.*}}, %{{.*}} : vector<4xi32>
+  // CHECK: %[[SEL0:.*]] = llvm.select %[[LT]], %{{.*}}, %{{.*}} : vector<4xi1>, vector<4xi32>
+  // CHECK: llvm.select %[[GT]], %{{.*}}, %[[SEL0]] : vector<4xi1>, vector<4xi32>
+  %0 = spirv.GL.SSign %arg0 : vector<4xi32>
+  spirv.Return
+}


From llvmlistbot at llvm.org  Tue Jul 14 02:15:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:15:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fe41.6c5720b3.15e90c.c1e9@mx.google.com>


================

----------------
mygitljf wrote:

I removed this test file from the current patch.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:16:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:16:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fe61.743400b0.2067c9.1d91@mx.google.com>


================

----------------
mygitljf wrote:

I removed this test file from the current patch.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:16:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:16:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fe7c.824e11a6.f5f27.805a@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:18:08 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:18:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55fed0.a1accbb4.244acd.0380@mx.google.com>


================

----------------
mygitljf wrote:

I changed the implementation to avoid this issue. The ceil/floor expansions are now pure Arith: I compute one quotient, derive an `i1` adjustment, convert it to the result type, and add or subtract it. Nothing materializes Tensor operations or requires a tensor lowering pipeline anymore.

https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:19:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 02:19:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Generalize ceil and floor
 division expansion (PR #206249)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206249@github.com>
Message-ID: <6a55ff1b.3cce8983.225732.728b@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206249

From llvmlistbot at llvm.org  Tue Jul 14 02:24:28 2026
From: llvmlistbot at llvm.org (Ian Tayler Lessa)
Date: Tue, 14 Jul 2026 02:24:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a56004c.13be6df8.145c.d63d@mx.google.com>


IanTaylerLessa-arm wrote:

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Tue Jul 14 02:32:32 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:32:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a560230.07d7ca98.9dee3.e3cd@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
----------------
ioghiban wrote:

[nit]: use three slashes in test comments

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 02:33:04 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:33:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a560250.0c05e68e.1e82f1.46e9@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
----------------
ioghiban wrote:

[nit]: split into two lines with `CHECK-SAME`

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 02:33:29 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:33:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a560269.c5504292.28b3b8.9664@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 02:35:36 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:35:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5602e8.27b3b003.3b871.8e75@mx.google.com>


================
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
----------------
ioghiban wrote:

[nit]: This comment implies that the pass is still applied, but in a different way. Is that the case? Or the pass bails out aka - never applied

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 02:39:32 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:39:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5603d4.743400b0.2067c9.b4c5@mx.google.com>


================
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @actionClassNoFields {
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if the ClassOp doesn't have
+/// a FunctionOp named operator()
+
+// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
----------------
ioghiban wrote:

[nit]: move tests with diagnostic verifiers to a different file

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 02:44:52 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 02:44:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a560514.92ea0e2a.13fd37.abbf@mx.google.com>


================
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
----------------
ioghiban wrote:

The comment above mentions that the pass "leaves IR unchanged". That to me suggests gracefully aborting the rewrite, without throwing errors. Would that be the expected behaviour?

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 03:10:54 2026
From: llvmlistbot at llvm.org (Aayush Shrivastava)
Date: Tue, 14 Jul 2026 03:10:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
Message-ID: <llvm.org/llvm/llvm-project/pull/209426@github.com>


https://github.com/iamaayushrivastava created https://github.com/llvm/llvm-project/pull/209426

Fixes #209065

`createInsertAndCast/createExtractAndCast` in `LLVMMemorySlot.cpp` queried the bit size of the source/target types before checking whether they were already identical. For scalable vector types (e.g. `vector<[4]xi1>`), this size query implicitly converts a scalable TypeSize to a scalar, which aborts. `mem2reg` hits this path for any store or load whose value type exactly matches the slot's element type, since `getStored/removeBlockingUses` run even when no cast is needed.

This fix adds an early return when the types are already equal, skipping the size query, consistent with the existing short-circuit in `areConversionCompatible/castSameSizedTypes`.

>From 50df936c86ffc4dc1e8ae1a254775b61a7c9c56c Mon Sep 17 00:00:00 2001
From: iamaayushrivastava <iamaayushrivastava at gmail.com>
Date: Tue, 14 Jul 2026 15:35:31 +0530
Subject: [PATCH] [mlir] Fix mem2reg crash on scalable vector store/load with
 matching type

---
 mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp | 13 ++++++++
 mlir/test/Dialect/LLVMIR/mem2reg.mlir         | 32 +++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
index 66d6a592a93df..1186e64b8934d 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
@@ -256,6 +256,12 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcType == targetType)
+    return srcValue;
+
   uint64_t srcTypeSize = dataLayout.getTypeSizeInBits(srcType);
   uint64_t targetTypeSize = dataLayout.getTypeSizeInBits(targetType);
   if (srcTypeSize == targetTypeSize)
@@ -291,6 +297,13 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
                                  srcValue.getType(),
                                  /*narrowingConversion=*/false) &&
          "expected that the compatibility was checked before");
+
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcValue.getType() == reachingDef.getType())
+    return srcValue;
+
   uint64_t valueTypeSize = dataLayout.getTypeSizeInBits(srcValue.getType());
   uint64_t slotTypeSize = dataLayout.getTypeSizeInBits(reachingDef.getType());
   if (slotTypeSize == valueTypeSize)
diff --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index cabbd1c1bc013..07ad0b86043e9 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1035,6 +1035,38 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// Storing into a scalable vector slot with a type matching the slot's
+// element type must not query the type's bit size, as this is not possible
+// for scalable vectors. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store
+llvm.func @scalable_vector_matching_store(%arg: vector<[4]xi1>) {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  // CHECK-NOT: llvm.store
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  llvm.return
+}
+
+// -----
+
+// Loading back a scalable vector value that was stored with a matching type
+// must also avoid querying the type's bit size. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store_load
+llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  %2 = llvm.load %1 : !llvm.ptr -> vector<[4]xi1>
+  // CHECK: llvm.return %[[ARG:.*]] : vector<[4]xi1>
+  llvm.return %2 : vector<[4]xi1>
+}
+
+// -----
+
 // CHECK-LABEL: @smaller_store_forwarding
 // CHECK-SAME: %[[ARG:.+]]: i16
 llvm.func @smaller_store_forwarding(%arg : i16) {


From llvmlistbot at llvm.org  Tue Jul 14 03:11:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 03:11:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a560b58.0c05e68e.1e82f1.d4b6@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Aayush Shrivastava (iamaayushrivastava)

<details>
<summary>Changes</summary>

Fixes #<!-- -->209065

`createInsertAndCast/createExtractAndCast` in `LLVMMemorySlot.cpp` queried the bit size of the source/target types before checking whether they were already identical. For scalable vector types (e.g. `vector&lt;[4]xi1&gt;`), this size query implicitly converts a scalable TypeSize to a scalar, which aborts. `mem2reg` hits this path for any store or load whose value type exactly matches the slot's element type, since `getStored/removeBlockingUses` run even when no cast is needed.

This fix adds an early return when the types are already equal, skipping the size query, consistent with the existing short-circuit in `areConversionCompatible/castSameSizedTypes`.

---
Full diff: https://github.com/llvm/llvm-project/pull/209426.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp (+13) 
- (modified) mlir/test/Dialect/LLVMIR/mem2reg.mlir (+32) 


``````````diff
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
index 66d6a592a93df..1186e64b8934d 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
@@ -256,6 +256,12 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcType == targetType)
+    return srcValue;
+
   uint64_t srcTypeSize = dataLayout.getTypeSizeInBits(srcType);
   uint64_t targetTypeSize = dataLayout.getTypeSizeInBits(targetType);
   if (srcTypeSize == targetTypeSize)
@@ -291,6 +297,13 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
                                  srcValue.getType(),
                                  /*narrowingConversion=*/false) &&
          "expected that the compatibility was checked before");
+
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcValue.getType() == reachingDef.getType())
+    return srcValue;
+
   uint64_t valueTypeSize = dataLayout.getTypeSizeInBits(srcValue.getType());
   uint64_t slotTypeSize = dataLayout.getTypeSizeInBits(reachingDef.getType());
   if (slotTypeSize == valueTypeSize)
diff --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index cabbd1c1bc013..07ad0b86043e9 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1035,6 +1035,38 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// Storing into a scalable vector slot with a type matching the slot's
+// element type must not query the type's bit size, as this is not possible
+// for scalable vectors. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store
+llvm.func @scalable_vector_matching_store(%arg: vector<[4]xi1>) {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  // CHECK-NOT: llvm.store
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  llvm.return
+}
+
+// -----
+
+// Loading back a scalable vector value that was stored with a matching type
+// must also avoid querying the type's bit size. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store_load
+llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  %2 = llvm.load %1 : !llvm.ptr -> vector<[4]xi1>
+  // CHECK: llvm.return %[[ARG:.*]] : vector<[4]xi1>
+  llvm.return %2 : vector<[4]xi1>
+}
+
+// -----
+
 // CHECK-LABEL: @smaller_store_forwarding
 // CHECK-SAME: %[[ARG:.+]]: i16
 llvm.func @smaller_store_forwarding(%arg : i16) {

``````````

</details>


https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 04:21:52 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 04:21:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a561bd0.2490598a.cf17c.d2ce@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/205774

>From f741504ef4239958e3b7d1af170d3bc9af6d809c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 22 Jun 2026 14:05:19 +0200
Subject: [PATCH 1/5] [mlir][EmitC] Add rank-0 MemRef conversion

---
 .../Conversion/EmitCCommon/TypeConverter.cpp  |  3 +-
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 72 ++++++++++++++-----
 .../FuncToEmitC/func-to-emitc-failed.mlir     | 15 ----
 .../memref-to-emitc-alloc-dealloc.mlir        | 26 +++++++
 .../memref-to-emitc-alloc-load-store.mlir     | 48 +++++++++++++
 .../MemRefToEmitC/memref-to-emitc-copy.mlir   | 21 +++++-
 6 files changed, 150 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
index a65edcf8306b8..1722fb95524d5 100644
--- a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
+++ b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
@@ -18,7 +18,6 @@ namespace {
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -50,6 +49,8 @@ EmitCTypeConverter::EmitCTypeConverter(MLIRContext *ctx) {
     if (!convertedElementType)
       return {};
 
+    if (memRefType.getRank() == 0)
+      return emitc::PointerType::get(convertedElementType);
     return emitc::ArrayType::get(memRefType.getShape(), convertedElementType);
   });
 
diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 8acb737f0e9b8..3af48fe23d140 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -30,7 +30,6 @@ using namespace mlir;
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -73,6 +72,11 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with dynamic shape");
     }
 
+    if (op.getType().getRank() == 0) {
+      return rewriter.notifyMatchFailure(
+          op.getLoc(), "cannot transform alloca with zero rank");
+    }
+
     if (op.getAlignment().value_or(1) > 1) {
       // TODO: Allow alignment if it is not more than the natural alignment
       // of the C array.
@@ -142,9 +146,12 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-// If `v` is defined through an unrealized cast and the source of that cast
-// is `emitc.ptr`, return the pointer.
-static Value stripPointerUnrealizedCast(Value v) {
+static Value getPointerOrStripUnrealizedCast(Value v) {
+  if (isa<emitc::PointerType>(v.getType()))
+    return v;
+
+  // If `v` is defined through an unrealized cast and the source of that cast
+  // is `emitc.ptr`, return the pointer.
   if (auto cast = v.getDefiningOp<UnrealizedConversionCastOp>())
     if (cast.getNumOperands() == 1 &&
         isa<emitc::PointerType>(cast.getOperand(0).getType()))
@@ -165,6 +172,9 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
+  if (shape.empty())
+    return linearIndex;
+
   for (auto [dim, idx] : llvm::zip(shape.drop_front(), indices.drop_front())) {
     Value dimSize =
         emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(dim));
@@ -238,10 +248,9 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = deallocOp.getLoc();
     // `free` can only be emitted when the dealloc operand is recoverable as an
-    // `emitc.ptr<T>`. In the current conversion, that happens via an
-    // unrealized_conversion_cast from the pointer-backed EmitC form.
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr) {
+    // `emitc.ptr<T>`.
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr) {
       return rewriter.notifyMatchFailure(
           loc, "expected pointer-backed memref for EmitC deallocation");
     }
@@ -252,7 +261,7 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
     Type opaqueVoidPtrType = emitc::PointerType::get(
         emitc::OpaqueType::get(rewriter.getContext(), "void"));
     Value freeArg =
-        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, strippedPtr);
+        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, ptr);
     emitc::CallOpaqueOp freeCall = emitc::CallOpaqueOp::create(
         rewriter, loc, TypeRange{}, rewriter.getStringAttr(freeFunctionName),
         ValueRange{freeArg});
@@ -280,6 +289,35 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getPointerOrStripUnrealizedCast(operands.getSource());
+      Value targetPtr = getPointerOrStripUnrealizedCast(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      auto srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
+                                         srcLValue.getResult());
+
+      auto targetLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(targetPtr),
+          zeroIndex);
+      rewriter.replaceOpWithNewOp<emitc::AssignOp>(
+          copyOp, targetLValue.getResult(), value.getResult());
+      return success();
+    }
+
     auto srcArrayValue =
         cast<TypedValue<emitc::ArrayType>>(operands.getSource());
     emitc::AddressOfOp srcPtr =
@@ -414,8 +452,8 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
 
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
 
@@ -423,14 +461,14 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 
@@ -448,8 +486,8 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
     Location loc = op.getLoc();
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
       rewriter.replaceOpWithNewOp<emitc::AssignOp>(op, subscript,
@@ -457,14 +495,14 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 
diff --git a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
index 46e5319d7d17c..85410ec0134e6 100644
--- a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
@@ -7,21 +7,6 @@ func.func @unsuppoted_emitc_type(%arg0: i4) -> i4 {
 
 // -----
 
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_alloc() -> memref<i32> {
-  %alloc = memref.alloc() : memref<i32>
-  return %alloc : memref<i32>
-}
-
-// -----
-
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_arg(%arg0: memref<i32>) -> memref<i32> {
-  return %arg0 : memref<i32>
-}
-
-// -----
-
 // expected-error at +1 {{failed to legalize operation 'func.func'}}
 func.func private @return_rank1_alloc() -> memref<1xi32> {
   %alloc = memref.alloc() : memref<1xi32>
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
index 9d20667a6490c..3194a40c16eeb 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
@@ -89,3 +89,29 @@ func.func @allocating_and_deallocating_multi() {
 // NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
 // NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
 // NOCPP-NEXT: return
+
+func.func @alloc_and_dealloc_rank0() {
+  %alloc = memref.alloc() : memref<i32>
+  memref.dealloc %alloc : memref<i32>
+  return
+}
+
+// CPP-LABEL: alloc_and_dealloc_rank0
+// CPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// CPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// CPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// CPP-NEXT: return
+
+// NOCPP-LABEL: alloc_and_dealloc_rank0
+// NOCPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// NOCPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// NOCPP-NEXT: return
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
index 07cad3b0c4dc2..5e466630a66a0 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
@@ -72,3 +72,51 @@ func.func @memref_load_store(%buff0: memref<2xf32>,
   memref.store %v, %buff1[%i, %j] : memref<4x8xf32>
   return
 }
+
+/// Rank-0 alloc-backed load/store lower through pointer subscript at index 0.
+// CHECK-LABEL: emitc.func private @memref_rank0_alloc_store(
+// CHECK-SAME:  %[[VAL:.*]]: i32)
+func.func private @memref_rank0_alloc_store(%v : i32) {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[ELEM_LVALUE]] : <i32>
+  memref.store %v, %alloc[] : memref<i32>
+  return
+}
+
+// CHECK-LABEL: emitc.func private @memref_rank0_alloc_load() -> i32
+func.func private @memref_rank0_alloc_load() -> i32 {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[LOADED_VAL:.*]] = load %[[ELEM_LVALUE]] : <i32>
+  %v = memref.load %alloc[] : memref<i32>
+  // CHECK:     return %[[LOADED_VAL]] : i32
+  return %v : i32
+}
+
+// CHECK-LABEL: emitc.func @memref_rank0_load_store(
+// CHECK-SAME:  %[[SRC:.*]]: !emitc.ptr<i32>,
+// CHECK-SAME:  %[[DST:.*]]: !emitc.ptr<i32>)
+func.func @memref_rank0_load_store(%src: memref<i32>, %dst: memref<i32>) {
+  // CHECK:     %[[LOAD_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[SRC_LVALUE:.*]] = subscript %[[SRC]]{{\[}}%[[LOAD_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[VAL:.*]] = load %[[SRC_LVALUE]] : <i32>
+  %v = memref.load %src[] : memref<i32>
+  // CHECK:     %[[STORE_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[DST_LVALUE:.*]] = subscript %[[DST]]{{\[}}%[[STORE_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[DST_LVALUE]] : <i32>
+  memref.store %v, %dst[] : memref<i32>
+  return
+}
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
index 04e6edd5b6981..de3741e83c01c 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
@@ -1,5 +1,5 @@
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s -split-input-file | FileCheck %s --check-prefixes=CPP,CHECK
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s -split-input-file | FileCheck %s --check-prefixes=NOCPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s | FileCheck %s --check-prefixes=CPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s | FileCheck %s --check-prefixes=NOCPP,CHECK
 
 func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
   memref.copy %arg0, %arg1 : memref<9x4x5x7xf32> to memref<9x4x5x7xf32>
@@ -28,3 +28,20 @@ func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
 // CHECK:           return
 // CHECK:         }
 
+func.func @copying_rank0(%arg0 : memref<i32>, %arg1 : memref<i32>) {
+  memref.copy %arg0, %arg1 : memref<i32> to memref<i32>
+  return
+}
+
+// CHECK-LABEL:   func.func @copying_rank0(
+// CHECK-SAME:      %[[ARG0:.*]]: memref<i32>,
+// CHECK-SAME:      %[[ARG1:.*]]: memref<i32>) {
+// CHECK:           %[[TARGET_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG1]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[SOURCE_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG0]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+// CHECK:           %[[SOURCE_LVALUE:.*]] = emitc.subscript %[[SOURCE_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           %[[VALUE:.*]] = emitc.load %[[SOURCE_LVALUE]] : <i32>
+// CHECK:           %[[TARGET_LVALUE:.*]] = emitc.subscript %[[TARGET_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           emitc.assign %[[VALUE]] : i32 to %[[TARGET_LVALUE]] : <i32>
+// CHECK:           return
+// CHECK:         }

>From 83fcab31a7bc8d1e28498da25bcd3fe338032436 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 29 Jun 2026 17:31:08 +0200
Subject: [PATCH 2/5] Support lowering rank-0 AllocaOp

---
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 44 +++++++++++--------
 .../MemRefToEmitC/memref-to-emitc-failed.mlir |  8 ----
 .../MemRefToEmitC/memref-to-emitc.mlir        |  9 ++++
 3 files changed, 35 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 3af48fe23d140..6a76ddddd4eeb 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -59,6 +59,16 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
+Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
+  Type resultTy;
+  if (opTy.getRank() == 0) {
+    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
+  } else {
+    resultTy = typeConverter->convertType(opTy);
+  }
+  return resultTy;
+}
+
 namespace {
 struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
   using OpConversionPattern::OpConversionPattern;
@@ -72,11 +82,6 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with dynamic shape");
     }
 
-    if (op.getType().getRank() == 0) {
-      return rewriter.notifyMatchFailure(
-          op.getLoc(), "cannot transform alloca with zero rank");
-    }
-
     if (op.getAlignment().value_or(1) > 1) {
       // TODO: Allow alignment if it is not more than the natural alignment
       // of the C array.
@@ -84,26 +89,29 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = getTypeConverter()->convertType(op.getType());
-    if (!resultTy) {
+    auto resultTy = convertMemRefType(op.getType(), getTypeConverter());
+    if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
-    }
+
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
+    // Rank-0 path
+    if (op.getType().getRank() == 0) {
+      auto var = emitc::VariableOp::create(
+          rewriter, op.getLoc(), emitc::LValueType::get(resultTy), noInit);
+
+      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(),
+                                            emitc::PointerType::get(resultTy),
+                                            var.getResult());
+
+      rewriter.replaceOp(op, ptr.getResult());
+      return success();
+    }
+    // Rank > 0 path
     rewriter.replaceOpWithNewOp<emitc::VariableOp>(op, resultTy, noInit);
     return success();
   }
 };
 
-Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
-  Type resultTy;
-  if (opTy.getRank() == 0) {
-    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  } else {
-    resultTy = typeConverter->convertType(opTy);
-  }
-  return resultTy;
-}
-
 static Value calculateMemrefTotalSizeBytes(Location loc, MemRefType memrefType,
                                            OpBuilder &builder,
                                            Type convertedElementType) {
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
index 753f749fabbd6..f2d0e9a5eee66 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
@@ -72,14 +72,6 @@ func.func @non_identity_layout() {
 
 // -----
 
-func.func @zero_rank() {
-  // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
-  %0 = memref.alloca() : memref<f32>
-  return
-}
-
-// -----
-
 func.func @zero_dim_rank_1() {
   // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
   %0 = memref.alloca() : memref<0xf32>
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
index 5ae22490ff05e..9749906e0c98a 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
@@ -10,6 +10,15 @@ func.func @alloca() {
 
 // -----
 
+func.func @alloca_zero_rank() {
+  // CHECK: %[[LVALUE:.*]] = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<f32>
+  // CHECK: %[[PTR:.*]] = emitc.address_of %[[LVALUE]] : !emitc.lvalue<f32>
+  %0 = memref.alloca() : memref<f32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: memref_store
 // CHECK-SAME:  %[[buff:.*]]: memref<4x8xf32>, %[[v:.*]]: f32, %[[i:.*]]: index, %[[j:.*]]: index
 func.func @memref_store(%buff : memref<4x8xf32>, %v : f32, %i: index, %j: index) {

>From 917305f0abe40582faebffa6cbc93e8cc90213d9 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 1 Jul 2026 15:00:58 +0200
Subject: [PATCH 3/5] Address first round of comments

---
 mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp       | 7 +++----
 .../Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir    | 8 ++++++--
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 6a76ddddd4eeb..b87bcd5476596 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -61,11 +61,10 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 
 Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
   Type resultTy;
-  if (opTy.getRank() == 0) {
+  if (opTy.getRank() == 0)
     resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  } else {
+  else
     resultTy = typeConverter->convertType(opTy);
-  }
   return resultTy;
 }
 
@@ -180,7 +179,7 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
-  if (shape.empty())
+  if (indices.empty())
     return linearIndex;
 
   for (auto [dim, idx] : llvm::zip(shape.drop_front(), indices.drop_front())) {
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
index de3741e83c01c..6105521d9326d 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
@@ -1,5 +1,5 @@
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s | FileCheck %s --check-prefixes=CPP,CHECK
-// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s | FileCheck %s --check-prefixes=NOCPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=true" %s -split-input-file | FileCheck %s --check-prefixes=CPP,CHECK
+// RUN: mlir-opt -convert-memref-to-emitc="lower-to-cpp=false" %s -split-input-file | FileCheck %s --check-prefixes=NOCPP,CHECK
 
 func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
   memref.copy %arg0, %arg1 : memref<9x4x5x7xf32> to memref<9x4x5x7xf32>
@@ -28,11 +28,15 @@ func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
 // CHECK:           return
 // CHECK:         }
 
+// -----
+
 func.func @copying_rank0(%arg0 : memref<i32>, %arg1 : memref<i32>) {
   memref.copy %arg0, %arg1 : memref<i32> to memref<i32>
   return
 }
 
+// CHECK: module {
+
 // CHECK-LABEL:   func.func @copying_rank0(
 // CHECK-SAME:      %[[ARG0:.*]]: memref<i32>,
 // CHECK-SAME:      %[[ARG1:.*]]: memref<i32>) {

>From db8b6c19b2437acf68da60388363450c6a42a049 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 8 Jul 2026 13:11:58 +0200
Subject: [PATCH 4/5] Remove wrapper for rank-0 conversion

---
 .../MemRefToEmitC/MemRefToEmitC.cpp           | 46 +++++++++----------
 1 file changed, 22 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index b87bcd5476596..8275749144f8d 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -59,15 +59,6 @@ void mlir::registerConvertMemRefToEmitCInterface(DialectRegistry &registry) {
 // Conversion Patterns
 //===----------------------------------------------------------------------===//
 
-Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
-  Type resultTy;
-  if (opTy.getRank() == 0)
-    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  else
-    resultTy = typeConverter->convertType(opTy);
-  return resultTy;
-}
-
 namespace {
 struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
   using OpConversionPattern::OpConversionPattern;
@@ -88,18 +79,20 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = convertMemRefType(op.getType(), getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(op.getType());
     if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
 
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
     // Rank-0 path
     if (op.getType().getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
       auto var = emitc::VariableOp::create(
-          rewriter, op.getLoc(), emitc::LValueType::get(resultTy), noInit);
+          rewriter, op.getLoc(), emitc::LValueType::get(elemTy), noInit);
 
-      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(),
-                                            emitc::PointerType::get(resultTy),
+      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(), resultTy,
                                             var.getResult());
 
       rewriter.replaceOp(op, ptr.getResult());
@@ -153,7 +146,7 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-static Value getPointerOrStripUnrealizedCast(Value v) {
+static Value getMemRefPointer(Value v) {
   if (isa<emitc::PointerType>(v.getType()))
     return v;
 
@@ -256,7 +249,7 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
     Location loc = deallocOp.getLoc();
     // `free` can only be emitted when the dealloc operand is recoverable as an
     // `emitc.ptr<T>`.
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr) {
       return rewriter.notifyMatchFailure(
           loc, "expected pointer-backed memref for EmitC deallocation");
@@ -304,8 +297,8 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       if (!elementType)
         return rewriter.notifyMatchFailure(loc, "cannot convert element type");
 
-      Value srcPtr = getPointerOrStripUnrealizedCast(operands.getSource());
-      Value targetPtr = getPointerOrStripUnrealizedCast(operands.getTarget());
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
       if (!srcPtr || !targetPtr)
         return rewriter.notifyMatchFailure(loc, "expected pointer operands");
 
@@ -376,7 +369,7 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
                        "currently not supported");
     }
 
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -397,6 +390,9 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
 
     Attribute initialValue = operands.getInitialValueAttr();
     if (opTy.getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      resultTy = pointerTy.getPointee();
       // special case for `variable : memref<i32> = dense<-1>`
       if (std::optional<Attribute> initValueAttr = op.getInitialValue()) {
         if (auto elementsAttr = llvm::dyn_cast<ElementsAttr>(*initValueAttr)) {
@@ -423,7 +419,7 @@ struct ConvertGetGlobal final
                   ConversionPatternRewriter &rewriter) const override {
 
     MemRefType opTy = op.getType();
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -431,11 +427,13 @@ struct ConvertGetGlobal final
     }
 
     if (opTy.getRank() == 0) {
-      emitc::LValueType lvalueType = emitc::LValueType::get(resultTy);
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
+      emitc::LValueType lvalueType = emitc::LValueType::get(elemTy);
       emitc::GetGlobalOp globalLValue = emitc::GetGlobalOp::create(
           rewriter, op.getLoc(), lvalueType, operands.getNameAttr());
-      emitc::PointerType pointerType = emitc::PointerType::get(resultTy);
-      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, pointerType,
+      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, resultTy,
                                                       globalLValue);
       return success();
     }
@@ -459,7 +457,7 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
 
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
@@ -493,7 +491,7 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
     Location loc = op.getLoc();
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value ptr = getPointerOrStripUnrealizedCast(operands.getMemref());
+    Value ptr = getMemRefPointer(operands.getMemref());
     if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());

>From cfea80626a09901047301b07cdbf687cdbaa0709 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Mon, 13 Jul 2026 15:19:38 +0200
Subject: [PATCH 5/5] Address final comments

---
 mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp  | 11 +++++------
 .../memref-to-emitc-alloc-load-store.mlir            | 12 ++++++------
 .../Conversion/MemRefToEmitC/memref-to-emitc.mlir    |  2 +-
 3 files changed, 12 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 8275749144f8d..9ac49e2eb1a14 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -304,17 +304,16 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
 
       Value zeroIndex = emitc::ConstantOp::create(
           rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
-      auto srcLValue = emitc::SubscriptOp::create(
+      Value srcLValue = emitc::SubscriptOp::create(
           rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
           zeroIndex);
-      auto value = emitc::LoadOp::create(rewriter, loc, elementType,
-                                         srcLValue.getResult());
+      Value value =
+          emitc::LoadOp::create(rewriter, loc, elementType, srcLValue);
 
-      auto targetLValue = emitc::SubscriptOp::create(
+      Value targetLValue = emitc::SubscriptOp::create(
           rewriter, loc, cast<TypedValue<emitc::PointerType>>(targetPtr),
           zeroIndex);
-      rewriter.replaceOpWithNewOp<emitc::AssignOp>(
-          copyOp, targetLValue.getResult(), value.getResult());
+      rewriter.replaceOpWithNewOp<emitc::AssignOp>(copyOp, targetLValue, value);
       return success();
     }
 
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
index 5e466630a66a0..653220470bb5a 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
@@ -74,9 +74,9 @@ func.func @memref_load_store(%buff0: memref<2xf32>,
 }
 
 /// Rank-0 alloc-backed load/store lower through pointer subscript at index 0.
-// CHECK-LABEL: emitc.func private @memref_rank0_alloc_store(
+// CHECK-LABEL: emitc.func private @memref_alloc_store_rank0(
 // CHECK-SAME:  %[[VAL:.*]]: i32)
-func.func private @memref_rank0_alloc_store(%v : i32) {
+func.func private @memref_alloc_store_rank0(%v : i32) {
   // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
   // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
   // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
@@ -90,8 +90,8 @@ func.func private @memref_rank0_alloc_store(%v : i32) {
   return
 }
 
-// CHECK-LABEL: emitc.func private @memref_rank0_alloc_load() -> i32
-func.func private @memref_rank0_alloc_load() -> i32 {
+// CHECK-LABEL: emitc.func private @memref_alloc_load_rank0() -> i32
+func.func private @memref_alloc_load_rank0() -> i32 {
   // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
   // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
   // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
@@ -106,10 +106,10 @@ func.func private @memref_rank0_alloc_load() -> i32 {
   return %v : i32
 }
 
-// CHECK-LABEL: emitc.func @memref_rank0_load_store(
+// CHECK-LABEL: emitc.func @memref_load_store_rank0(
 // CHECK-SAME:  %[[SRC:.*]]: !emitc.ptr<i32>,
 // CHECK-SAME:  %[[DST:.*]]: !emitc.ptr<i32>)
-func.func @memref_rank0_load_store(%src: memref<i32>, %dst: memref<i32>) {
+func.func @memref_load_store_rank0(%src: memref<i32>, %dst: memref<i32>) {
   // CHECK:     %[[LOAD_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
   // CHECK:     %[[SRC_LVALUE:.*]] = subscript %[[SRC]]{{\[}}%[[LOAD_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
   // CHECK:     %[[VAL:.*]] = load %[[SRC_LVALUE]] : <i32>
diff --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
index 9749906e0c98a..1c832b3fba6ec 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
@@ -10,7 +10,7 @@ func.func @alloca() {
 
 // -----
 
-func.func @alloca_zero_rank() {
+func.func @alloca_rank0() {
   // CHECK: %[[LVALUE:.*]] = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<f32>
   // CHECK: %[[PTR:.*]] = emitc.address_of %[[LVALUE]] : !emitc.lvalue<f32>
   %0 = memref.alloca() : memref<f32>


From llvmlistbot at llvm.org  Tue Jul 14 04:28:59 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Tue, 14 Jul 2026 04:28:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NFC] Fix tgfmt compilation with MSVC
	(PR #209446)
Message-ID: <llvm.org/llvm/llvm-project/pull/209446@github.com>


https://github.com/aengelke created https://github.com/llvm/llvm-project/pull/209446

Apparently older MSVC has problems with inferring template argument
types. Provide them explicitly.

Fixes https://github.com/llvm/llvm-project/issues/209303.


>From d2804df44de484dc3aea62b310ef7c9c61657012 Mon Sep 17 00:00:00 2001
From: Alexis Engelke <engelke at in.tum.de>
Date: Tue, 14 Jul 2026 11:28:19 +0000
Subject: [PATCH] [spr] initial version

Created using spr 1.3.8-wip
---
 mlir/include/mlir/TableGen/Format.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/TableGen/Format.h b/mlir/include/mlir/TableGen/Format.h
index daadf8e636feb..bdecf491faf21 100644
--- a/mlir/include/mlir/TableGen/Format.h
+++ b/mlir/include/mlir/TableGen/Format.h
@@ -254,13 +254,13 @@ class FmtStrVecObject : public FmtObjectBase {
 template <typename... Ts>
 inline auto tgfmt(StringRef fmt, const FmtContext *ctx, Ts &&...vals)
     -> FmtObject<decltype(std::make_tuple(
-        llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...))> {
+        llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...))> {
   using ParamTuple = decltype(std::make_tuple(
-      llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
+      llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
   return FmtObject<ParamTuple>(
       fmt, ctx,
       std::make_tuple(
-          llvm::support::detail::FormatFunctor(std::forward<Ts>(vals))...));
+          llvm::support::detail::FormatFunctor<Ts>(std::forward<Ts>(vals))...));
 }
 
 /// Like tgfmt, but take replacement parameters as an array of strings instead.


From llvmlistbot at llvm.org  Tue Jul 14 04:29:17 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Tue, 14 Jul 2026 04:29:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NFC] Fix tgfmt compilation with MSVC
	(PR #209446)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209446@github.com>
Message-ID: <6a561d8d.f1c1923d.991d1.84cf@mx.google.com>


https://github.com/aengelke auto_merge_enabled https://github.com/llvm/llvm-project/pull/209446

From llvmlistbot at llvm.org  Tue Jul 14 04:48:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 04:48:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 5ae6537 - [mlir][EmitC] Add rank-0 MemRef
	conversion (#205774)
Message-ID: <6a5621fd.492e23e4.14eda0.ba7c@mx.google.com>


Author: ioana ghiban
Date: 2026-07-14T13:48:08+02:00
New Revision: 5ae6537bbaf577889fb81f5f8397463b0d0499c3

URL: https://github.com/llvm/llvm-project/commit/5ae6537bbaf577889fb81f5f8397463b0d0499c3
DIFF: https://github.com/llvm/llvm-project/commit/5ae6537bbaf577889fb81f5f8397463b0d0499c3.diff

LOG: [mlir][EmitC] Add rank-0 MemRef conversion (#205774)

Add rank-0 memref support to MemRefToEmitC and the EmitC TypeConverter,
needed for lowering the models generated by:
- `llvm/lib/Analysis/models/gen-inline-oz-test-model.py`
- `llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py`

Rank-0 memrefs are no longer rejected by `isMemRefTypeLegalForEmitC`.
The EmitC type converter maps `memref<T>` to `!emitc.ptr<T>`, giving
rank-0 memrefs addressable scalar storage.

`memref.alloc` now allocates one element for rank-0 memrefs.
`memref.dealloc` frees the pointer-backed value. `memref.load` and
`memref.store` lower rank-0 accesses through `emitc.subscript %ptr[0]`,
followed by `emitc.load` or `emitc.assign`. `memref.copy` lowers rank-0
copies as scalar load plus assign.

`memref.alloca` still rejects rank-0 memrefs because this patch only
supports pointer-backed rank-0 lowering.

The pointer recovery helper now accepts both direct `emitc.ptr<T>`
values and pointer values behind `builtin.unrealized_conversion_cast`,
which is needed by dealloc, copy, load, and store depending on where the
value comes from in conversion.

Tests cover rank-0 alloc/dealloc, load/store, alloc-backed load/store,
and copy. Removed cases this patch adds support for (rank-0) from
`func-to-emitc-failed`.

Assisted-by: Codex (refine implementation + tests). I reviewed all code
and tests before submission.

Added: 
    

Modified: 
    mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
    mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
    mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
    mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
    mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
    mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
    mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
    mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
index a65edcf8306b8..1722fb95524d5 100644
--- a/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
+++ b/mlir/lib/Conversion/EmitCCommon/TypeConverter.cpp
@@ -18,7 +18,6 @@ namespace {
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -50,6 +49,8 @@ EmitCTypeConverter::EmitCTypeConverter(MLIRContext *ctx) {
     if (!convertedElementType)
       return {};
 
+    if (memRefType.getRank() == 0)
+      return emitc::PointerType::get(convertedElementType);
     return emitc::ArrayType::get(memRefType.getShape(), convertedElementType);
   });
 

diff  --git a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
index 8acb737f0e9b8..9ac49e2eb1a14 100644
--- a/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
+++ b/mlir/lib/Conversion/MemRefToEmitC/MemRefToEmitC.cpp
@@ -30,7 +30,6 @@ using namespace mlir;
 
 static bool isMemRefTypeLegalForEmitC(MemRefType memRefType) {
   return memRefType.hasStaticShape() && memRefType.getLayout().isIdentity() &&
-         memRefType.getRank() != 0 &&
          !llvm::is_contained(memRefType.getShape(), 0);
 }
 
@@ -80,26 +79,31 @@ struct ConvertAlloca final : public OpConversionPattern<memref::AllocaOp> {
           op.getLoc(), "cannot transform alloca with alignment requirement");
     }
 
-    auto resultTy = getTypeConverter()->convertType(op.getType());
-    if (!resultTy) {
+    Type resultTy = getTypeConverter()->convertType(op.getType());
+    if (!resultTy)
       return rewriter.notifyMatchFailure(op.getLoc(), "cannot convert type");
-    }
+
     auto noInit = emitc::OpaqueAttr::get(getContext(), "");
+    // Rank-0 path
+    if (op.getType().getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
+      auto var = emitc::VariableOp::create(
+          rewriter, op.getLoc(), emitc::LValueType::get(elemTy), noInit);
+
+      auto ptr = emitc::AddressOfOp::create(rewriter, op.getLoc(), resultTy,
+                                            var.getResult());
+
+      rewriter.replaceOp(op, ptr.getResult());
+      return success();
+    }
+    // Rank > 0 path
     rewriter.replaceOpWithNewOp<emitc::VariableOp>(op, resultTy, noInit);
     return success();
   }
 };
 
-Type convertMemRefType(MemRefType opTy, const TypeConverter *typeConverter) {
-  Type resultTy;
-  if (opTy.getRank() == 0) {
-    resultTy = typeConverter->convertType(mlir::getElementTypeOrSelf(opTy));
-  } else {
-    resultTy = typeConverter->convertType(opTy);
-  }
-  return resultTy;
-}
-
 static Value calculateMemrefTotalSizeBytes(Location loc, MemRefType memrefType,
                                            OpBuilder &builder,
                                            Type convertedElementType) {
@@ -142,9 +146,12 @@ createPointerFromEmitcArray(Location loc, OpBuilder &builder,
   return ptr;
 }
 
-// If `v` is defined through an unrealized cast and the source of that cast
-// is `emitc.ptr`, return the pointer.
-static Value stripPointerUnrealizedCast(Value v) {
+static Value getMemRefPointer(Value v) {
+  if (isa<emitc::PointerType>(v.getType()))
+    return v;
+
+  // If `v` is defined through an unrealized cast and the source of that cast
+  // is `emitc.ptr`, return the pointer.
   if (auto cast = v.getDefiningOp<UnrealizedConversionCastOp>())
     if (cast.getNumOperands() == 1 &&
         isa<emitc::PointerType>(cast.getOperand(0).getType()))
@@ -165,6 +172,9 @@ static Value computeRowMajorLinearIndex(ImplicitLocOpBuilder &builder,
           ? emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(0))
           : indices[0];
 
+  if (indices.empty())
+    return linearIndex;
+
   for (auto [dim, idx] : llvm::zip(shape.drop_front(), indices.drop_front())) {
     Value dimSize =
         emitc::ConstantOp::create(builder, idxType, builder.getIndexAttr(dim));
@@ -238,10 +248,9 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = deallocOp.getLoc();
     // `free` can only be emitted when the dealloc operand is recoverable as an
-    // `emitc.ptr<T>`. In the current conversion, that happens via an
-    // unrealized_conversion_cast from the pointer-backed EmitC form.
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr) {
+    // `emitc.ptr<T>`.
+    Value ptr = getMemRefPointer(operands.getMemref());
+    if (!ptr) {
       return rewriter.notifyMatchFailure(
           loc, "expected pointer-backed memref for EmitC deallocation");
     }
@@ -252,7 +261,7 @@ struct ConvertDealloc final : public OpConversionPattern<memref::DeallocOp> {
     Type opaqueVoidPtrType = emitc::PointerType::get(
         emitc::OpaqueType::get(rewriter.getContext(), "void"));
     Value freeArg =
-        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, strippedPtr);
+        emitc::CastOp::create(rewriter, loc, opaqueVoidPtrType, ptr);
     emitc::CallOpaqueOp freeCall = emitc::CallOpaqueOp::create(
         rewriter, loc, TypeRange{}, rewriter.getStringAttr(freeFunctionName),
         ValueRange{freeArg});
@@ -280,6 +289,34 @@ struct ConvertCopy final : public OpConversionPattern<memref::CopyOp> {
       return rewriter.notifyMatchFailure(
           loc, "incompatible target memref type for EmitC conversion");
 
+    if (srcMemrefType.getRank() == 0) {
+      assert(targetMemrefType.getRank() == 0 &&
+             "target must have same rank as source");
+      Type elementType =
+          getTypeConverter()->convertType(srcMemrefType.getElementType());
+      if (!elementType)
+        return rewriter.notifyMatchFailure(loc, "cannot convert element type");
+
+      Value srcPtr = getMemRefPointer(operands.getSource());
+      Value targetPtr = getMemRefPointer(operands.getTarget());
+      if (!srcPtr || !targetPtr)
+        return rewriter.notifyMatchFailure(loc, "expected pointer operands");
+
+      Value zeroIndex = emitc::ConstantOp::create(
+          rewriter, loc, rewriter.getIndexType(), rewriter.getIndexAttr(0));
+      Value srcLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(srcPtr),
+          zeroIndex);
+      Value value =
+          emitc::LoadOp::create(rewriter, loc, elementType, srcLValue);
+
+      Value targetLValue = emitc::SubscriptOp::create(
+          rewriter, loc, cast<TypedValue<emitc::PointerType>>(targetPtr),
+          zeroIndex);
+      rewriter.replaceOpWithNewOp<emitc::AssignOp>(copyOp, targetLValue, value);
+      return success();
+    }
+
     auto srcArrayValue =
         cast<TypedValue<emitc::ArrayType>>(operands.getSource());
     emitc::AddressOfOp srcPtr =
@@ -331,7 +368,7 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
                        "currently not supported");
     }
 
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -352,6 +389,9 @@ struct ConvertGlobal final : public OpConversionPattern<memref::GlobalOp> {
 
     Attribute initialValue = operands.getInitialValueAttr();
     if (opTy.getRank() == 0) {
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      resultTy = pointerTy.getPointee();
       // special case for `variable : memref<i32> = dense<-1>`
       if (std::optional<Attribute> initValueAttr = op.getInitialValue()) {
         if (auto elementsAttr = llvm::dyn_cast<ElementsAttr>(*initValueAttr)) {
@@ -378,7 +418,7 @@ struct ConvertGetGlobal final
                   ConversionPatternRewriter &rewriter) const override {
 
     MemRefType opTy = op.getType();
-    Type resultTy = convertMemRefType(opTy, getTypeConverter());
+    Type resultTy = getTypeConverter()->convertType(opTy);
 
     if (!resultTy) {
       return rewriter.notifyMatchFailure(op.getLoc(),
@@ -386,11 +426,13 @@ struct ConvertGetGlobal final
     }
 
     if (opTy.getRank() == 0) {
-      emitc::LValueType lvalueType = emitc::LValueType::get(resultTy);
+      auto pointerTy = dyn_cast<emitc::PointerType>(resultTy);
+      assert(pointerTy && "expected rank-0 MemRef to convert to pointer");
+      Type elemTy = pointerTy.getPointee();
+      emitc::LValueType lvalueType = emitc::LValueType::get(elemTy);
       emitc::GetGlobalOp globalLValue = emitc::GetGlobalOp::create(
           rewriter, op.getLoc(), lvalueType, operands.getNameAttr());
-      emitc::PointerType pointerType = emitc::PointerType::get(resultTy);
-      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, pointerType,
+      rewriter.replaceOpWithNewOp<emitc::AddressOfOp>(op, resultTy,
                                                       globalLValue);
       return success();
     }
@@ -414,8 +456,8 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
 
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getMemRefPointer(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
 
@@ -423,14 +465,14 @@ struct ConvertLoad final : public OpConversionPattern<memref::LoadOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 
@@ -448,8 +490,8 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
     Location loc = op.getLoc();
     auto arrayValue =
         dyn_cast<TypedValue<emitc::ArrayType>>(operands.getMemref());
-    Value strippedPtr = stripPointerUnrealizedCast(operands.getMemref());
-    if (!strippedPtr && arrayValue) {
+    Value ptr = getMemRefPointer(operands.getMemref());
+    if (!ptr && arrayValue) {
       auto subscript = emitc::SubscriptOp::create(rewriter, loc, arrayValue,
                                                   operands.getIndices());
       rewriter.replaceOpWithNewOp<emitc::AssignOp>(op, subscript,
@@ -457,14 +499,14 @@ struct ConvertStore final : public OpConversionPattern<memref::StoreOp> {
       return success();
     }
 
-    if (!strippedPtr)
+    if (!ptr)
       return rewriter.notifyMatchFailure(loc, "expected array or pointer type");
     MemRefType opMemrefType = cast<MemRefType>(op.getMemref().getType());
     ValueRange indices = operands.getIndices();
 
     ImplicitLocOpBuilder b(loc, rewriter);
     Value linearIndex = computeRowMajorLinearIndex(b, opMemrefType, indices);
-    auto typedPtr = cast<TypedValue<emitc::PointerType>>(strippedPtr);
+    auto typedPtr = cast<TypedValue<emitc::PointerType>>(ptr);
     auto subscript =
         emitc::SubscriptOp::create(rewriter, loc, typedPtr, linearIndex);
 

diff  --git a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
index 46e5319d7d17c..85410ec0134e6 100644
--- a/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/FuncToEmitC/func-to-emitc-failed.mlir
@@ -7,21 +7,6 @@ func.func @unsuppoted_emitc_type(%arg0: i4) -> i4 {
 
 // -----
 
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_alloc() -> memref<i32> {
-  %alloc = memref.alloc() : memref<i32>
-  return %alloc : memref<i32>
-}
-
-// -----
-
-// expected-error at +1 {{failed to legalize operation 'func.func'}}
-func.func private @return_rank0_arg(%arg0: memref<i32>) -> memref<i32> {
-  return %arg0 : memref<i32>
-}
-
-// -----
-
 // expected-error at +1 {{failed to legalize operation 'func.func'}}
 func.func private @return_rank1_alloc() -> memref<1xi32> {
   %alloc = memref.alloc() : memref<1xi32>

diff  --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
index 9d20667a6490c..3194a40c16eeb 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-dealloc.mlir
@@ -89,3 +89,29 @@ func.func @allocating_and_deallocating_multi() {
 // NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
 // NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
 // NOCPP-NEXT: return
+
+func.func @alloc_and_dealloc_rank0() {
+  %alloc = memref.alloc() : memref<i32>
+  memref.dealloc %alloc : memref<i32>
+  return
+}
+
+// CPP-LABEL: alloc_and_dealloc_rank0
+// CPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// CPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// CPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// CPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// CPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// CPP-NEXT: return
+
+// NOCPP-LABEL: alloc_and_dealloc_rank0
+// NOCPP-NEXT: %[[ALLOC:.*]] = emitc.call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_SIZE:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+// NOCPP-NEXT: %[[ALLOC_TOTAL_SIZE:.*]] = emitc.mul %[[ALLOC]], %[[ALLOC_SIZE]] : (!emitc.size_t, index) -> !emitc.size_t
+// NOCPP-NEXT: %[[ALLOC_PTR:.*]] = emitc.call_opaque "malloc"(%[[ALLOC_TOTAL_SIZE]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: %[[ALLOC_CAST:.*]] = emitc.cast %[[ALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+// NOCPP-NEXT: %[[FREE_PTR:.*]] = emitc.cast %[[ALLOC_CAST]] : !emitc.ptr<i32> to !emitc.ptr<!emitc.opaque<"void">>
+// NOCPP-NEXT: emitc.call_opaque "free"(%[[FREE_PTR]]) : (!emitc.ptr<!emitc.opaque<"void">>) -> ()
+// NOCPP-NEXT: return

diff  --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
index 07cad3b0c4dc2..653220470bb5a 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-alloc-load-store.mlir
@@ -72,3 +72,51 @@ func.func @memref_load_store(%buff0: memref<2xf32>,
   memref.store %v, %buff1[%i, %j] : memref<4x8xf32>
   return
 }
+
+/// Rank-0 alloc-backed load/store lower through pointer subscript at index 0.
+// CHECK-LABEL: emitc.func private @memref_alloc_store_rank0(
+// CHECK-SAME:  %[[VAL:.*]]: i32)
+func.func private @memref_alloc_store_rank0(%v : i32) {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[ELEM_LVALUE]] : <i32>
+  memref.store %v, %alloc[] : memref<i32>
+  return
+}
+
+// CHECK-LABEL: emitc.func private @memref_alloc_load_rank0() -> i32
+func.func private @memref_alloc_load_rank0() -> i32 {
+  // CHECK:     %[[SIZEOF_I32:.*]] = call_opaque "sizeof"() <{args = [i32]}> : () -> !emitc.size_t
+  // CHECK:     %[[NUM_ELEMS:.*]] = "emitc.constant"() <{value = 1 : index}> : () -> index
+  // CHECK:     %[[TOTAL_BYTES:.*]] = mul %[[SIZEOF_I32]], %[[NUM_ELEMS]] : (!emitc.size_t, index) -> !emitc.size_t
+  // CHECK:     %[[MALLOC_PTR:.*]] = call_opaque "malloc"(%[[TOTAL_BYTES]]) : (!emitc.size_t) -> !emitc.ptr<!emitc.opaque<"void">>
+  // CHECK:     %[[ELEM_PTR:.*]] = cast %[[MALLOC_PTR]] : !emitc.ptr<!emitc.opaque<"void">> to !emitc.ptr<i32>
+  %alloc = memref.alloc() : memref<i32>
+  // CHECK:     %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[ELEM_LVALUE:.*]] = subscript %[[ELEM_PTR]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[LOADED_VAL:.*]] = load %[[ELEM_LVALUE]] : <i32>
+  %v = memref.load %alloc[] : memref<i32>
+  // CHECK:     return %[[LOADED_VAL]] : i32
+  return %v : i32
+}
+
+// CHECK-LABEL: emitc.func @memref_load_store_rank0(
+// CHECK-SAME:  %[[SRC:.*]]: !emitc.ptr<i32>,
+// CHECK-SAME:  %[[DST:.*]]: !emitc.ptr<i32>)
+func.func @memref_load_store_rank0(%src: memref<i32>, %dst: memref<i32>) {
+  // CHECK:     %[[LOAD_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[SRC_LVALUE:.*]] = subscript %[[SRC]]{{\[}}%[[LOAD_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     %[[VAL:.*]] = load %[[SRC_LVALUE]] : <i32>
+  %v = memref.load %src[] : memref<i32>
+  // CHECK:     %[[STORE_ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+  // CHECK:     %[[DST_LVALUE:.*]] = subscript %[[DST]]{{\[}}%[[STORE_ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+  // CHECK:     assign %[[VAL]] : i32 to %[[DST_LVALUE]] : <i32>
+  memref.store %v, %dst[] : memref<i32>
+  return
+}

diff  --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
index 04e6edd5b6981..6105521d9326d 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-copy.mlir
@@ -28,3 +28,24 @@ func.func @copying(%arg0 : memref<9x4x5x7xf32>, %arg1 : memref<9x4x5x7xf32>) {
 // CHECK:           return
 // CHECK:         }
 
+// -----
+
+func.func @copying_rank0(%arg0 : memref<i32>, %arg1 : memref<i32>) {
+  memref.copy %arg0, %arg1 : memref<i32> to memref<i32>
+  return
+}
+
+// CHECK: module {
+
+// CHECK-LABEL:   func.func @copying_rank0(
+// CHECK-SAME:      %[[ARG0:.*]]: memref<i32>,
+// CHECK-SAME:      %[[ARG1:.*]]: memref<i32>) {
+// CHECK:           %[[TARGET_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG1]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[SOURCE_CAST:.*]] = builtin.unrealized_conversion_cast %[[ARG0]] : memref<i32> to !emitc.ptr<i32>
+// CHECK:           %[[ZERO:.*]] = "emitc.constant"() <{value = 0 : index}> : () -> index
+// CHECK:           %[[SOURCE_LVALUE:.*]] = emitc.subscript %[[SOURCE_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           %[[VALUE:.*]] = emitc.load %[[SOURCE_LVALUE]] : <i32>
+// CHECK:           %[[TARGET_LVALUE:.*]] = emitc.subscript %[[TARGET_CAST]]{{\[}}%[[ZERO]]] : (!emitc.ptr<i32>, index) -> !emitc.lvalue<i32>
+// CHECK:           emitc.assign %[[VALUE]] : i32 to %[[TARGET_LVALUE]] : <i32>
+// CHECK:           return
+// CHECK:         }

diff  --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
index 753f749fabbd6..f2d0e9a5eee66 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc-failed.mlir
@@ -72,14 +72,6 @@ func.func @non_identity_layout() {
 
 // -----
 
-func.func @zero_rank() {
-  // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
-  %0 = memref.alloca() : memref<f32>
-  return
-}
-
-// -----
-
 func.func @zero_dim_rank_1() {
   // expected-error at +1 {{failed to legalize operation 'memref.alloca'}}
   %0 = memref.alloca() : memref<0xf32>

diff  --git a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
index 5ae22490ff05e..1c832b3fba6ec 100644
--- a/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
+++ b/mlir/test/Conversion/MemRefToEmitC/memref-to-emitc.mlir
@@ -10,6 +10,15 @@ func.func @alloca() {
 
 // -----
 
+func.func @alloca_rank0() {
+  // CHECK: %[[LVALUE:.*]] = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<f32>
+  // CHECK: %[[PTR:.*]] = emitc.address_of %[[LVALUE]] : !emitc.lvalue<f32>
+  %0 = memref.alloca() : memref<f32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: memref_store
 // CHECK-SAME:  %[[buff:.*]]: memref<4x8xf32>, %[[v:.*]]: f32, %[[i:.*]]: index, %[[j:.*]]: index
 func.func @memref_store(%buff : memref<4x8xf32>, %v : f32, %i: index, %j: index) {


        

From llvmlistbot at llvm.org  Tue Jul 14 04:48:18 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Tue, 14 Jul 2026 04:48:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add rank-0 MemRef conversion
	(PR #205774)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205774@github.com>
Message-ID: <6a562202.8e75e0cb.3dbce4.4c05@mx.google.com>


https://github.com/ioghiban closed https://github.com/llvm/llvm-project/pull/205774

From llvmlistbot at llvm.org  Tue Jul 14 05:11:40 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Tue, 14 Jul 2026 05:11:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a56277c.d681cab9.333b2b.156f@mx.google.com>


================
@@ -6787,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
----------------
dcaballe wrote:

SG. I think what @Groverkss mentioned goes along this line but in the opposite direction and it summarizes my overall concern: I'm not sure we want to tie this up to broadcast semantics specifically. We may want to build something a bit more flexible: any unit dims, unit-dim transposes, etc.

https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Tue Jul 14 05:30:31 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Tue, 14 Jul 2026 05:30:31 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a562be7.0c05e68e.1e82f1.21fd@mx.google.com>


https://github.com/skatrak approved this pull request.

LGTM, no need for another review by me. However, please do address the last nits before merging. Thank you!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 05:30:31 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Tue, 14 Jul 2026 05:30:31 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a562be7.bc28b987.90005.2b85@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 05:30:31 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Tue, 14 Jul 2026 05:30:31 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a562be7.c5504292.28b3b8.ffc3@mx.google.com>


================

----------------
skatrak wrote:

This is not the first time I've had to point out in this PR that we're not supposed to make references to Flang in the omp dialect. Admittedly, there are some already around, but they are generally giving an example of a situation that justifies an edge case. This is not the situation here, this reads like what an LLM adds as a comment to justify a change based on the prompt given to it, resulting in an explanation that seems arbitrary without the context of the review discussion. The same comment block also for no reason explains what `verifyMapClause` does, also because it was part of an earlier review comment.

When I previously suggested to "make sure to always review your LLM's output to avoid redundant/irrelevant information" I was referring to this kind of thing. By doing so, I get to complete the review and you get to merge the PR faster. I also doubt we would have gone through so many rounds of review for this PR if it wasn't for how little attention appears to have been put into de-slopifying each version of the patch before asking for another review.

We're all constrained by review throughput, so anything you can do as a contributor to reduce reviewer workload benefits everyone. I hope this doesn't come across as accusatory, I'm only trying to share some perspective about the costs of not paying enough attention to what an LLM generates before asking another person to check it for you.

Also, I think the PR description is missing an "Assisted-by" tag, as per the [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html).

```suggestion
      // Select the map_entries entry that captures this in_reduction operand.
      // The verifier guarantees at least
      // one match exists; more than one matching entry is a lowering ambiguity
      // (the redirect cannot pick which map argument to rebind).
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 05:30:32 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Tue, 14 Jul 2026 05:30:32 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a562be8.510f576d.1b3cb9.0b38@mx.google.com>


================

----------------
skatrak wrote:

This is a suggestion to better explain the logic of this. Feel free to rewrite it in a different way, but it's important to convey the fact that it is a workaround and not that we intend to generally allow matching different values.
```suggestion
/// result of the same defining op. At this stage, we can only enforce exact value
/// identity for block arguments, which have no defining op, because Flang emits
/// `hlfir.declare`#0 for `in_reduction` operands and `hlfir.declare`#1 for `map` var_ptr.
/// These eventually collapse into the same value when lowered, but we cannot enforce
/// it here.
```

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 05:30:32 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Tue, 14 Jul 2026 05:30:32 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a562be8.0c05e68e.1e82f1.2356@mx.google.com>


================

----------------
skatrak wrote:

Nit: Please remove this redundant comment.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 05:31:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 05:31:41 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [mlir] [mlir] Diagnose missing call
	result arrow (PR #206215)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206215@github.com>
Message-ID: <6a562c2d.3cce8983.225732.9930@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206215

>From 2fc18628d1c3420e7399ad967866ce401fdf6fbe Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 27 Jun 2026 12:12:37 +0000
Subject: [PATCH 1/2] [MLIR] Diagnose missing call result arrow

---
 clang/test/CIR/IR/invalid-call.cir         |  9 +++++++++
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp |  3 +++
 mlir/lib/Interfaces/CallInterfaces.cpp     |  3 ++-
 mlir/test/Dialect/LLVMIR/invalid.mlir      | 10 ++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/clang/test/CIR/IR/invalid-call.cir b/clang/test/CIR/IR/invalid-call.cir
index a9c7e38f73af6..11b5c4425549e 100644
--- a/clang/test/CIR/IR/invalid-call.cir
+++ b/clang/test/CIR/IR/invalid-call.cir
@@ -10,6 +10,15 @@ cir.func @f1() {
 
 // -----
 
+cir.func private @f_missing_result_arrow()
+cir.func @call_missing_empty_result() {
+  // expected-error @+2 {{expected '->' followed by function result list}}
+  cir.call @f_missing_result_arrow() : ()
+  cir.return
+}
+
+// -----
+
 !u32i = !cir.int<u, 32>
 
 cir.func private @f2()
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 58f569abff8ea..0cc007433ac0e 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,8 +1388,11 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
+  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
                                                   resTypes, resultAttrs)) {
+    if (parser.getCurrentLocation() != signatureLoc)
+      return failure();
     if (isDirect)
       return parser.emitError(trailingTypesLoc,
                               "expected direct call to have 1 trailing types");
diff --git a/mlir/lib/Interfaces/CallInterfaces.cpp b/mlir/lib/Interfaces/CallInterfaces.cpp
index e8ed4b339a0cb..af670cb62c1c6 100644
--- a/mlir/lib/Interfaces/CallInterfaces.cpp
+++ b/mlir/lib/Interfaces/CallInterfaces.cpp
@@ -72,7 +72,8 @@ ParseResult call_interface_impl::parseFunctionSignature(
     return call_interface_impl::parseFunctionResultList(parser, resultTypes,
                                                         resultAttrs);
   if (mustParseEmptyResult)
-    return failure();
+    return parser.emitError(parser.getCurrentLocation(),
+                            "expected '->' followed by function result list");
   return success();
 }
 
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index d5ea5c8de862e..1d0bb0b6ef999 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -241,6 +241,16 @@ func.func @call_missing_ptr_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 
 // -----
 
+func.func private @callee()
+
+func.func @call_missing_empty_result() {
+  // expected-error at +2 {{expected '->' followed by function result list}}
+  llvm.call @callee() : ()
+  llvm.return
+}
+
+// -----
+
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {

>From 7c84efca9a10c9f8cbbf974bea754e088388e5fd Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sun, 12 Jul 2026 02:49:36 +0000
Subject: [PATCH 2/2] [MLIR] Unify call signature diagnostics

---
 flang/lib/Optimizer/Dialect/FIROps.cpp        |  2 +-
 flang/test/Fir/invalid.fir                    |  8 ++++++++
 mlir/include/mlir/Interfaces/CallInterfaces.h |  1 +
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp    | 12 ++----------
 mlir/test/Dialect/LLVMIR/invalid.mlir         |  6 ++----
 5 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Optimizer/Dialect/FIROps.cpp b/flang/lib/Optimizer/Dialect/FIROps.cpp
index ba047e71d6aa3..6f0c8eecfd462 100644
--- a/flang/lib/Optimizer/Dialect/FIROps.cpp
+++ b/flang/lib/Optimizer/Dialect/FIROps.cpp
@@ -1714,7 +1714,7 @@ mlir::ParseResult fir::CallOp::parse(mlir::OpAsmParser &parser,
   llvm::SmallVector<mlir::DictionaryAttr> resultAttrs;
   if (mlir::call_interface_impl::parseFunctionSignature(
           parser, argTypes, argAttrs, resTypes, resultAttrs))
-    return parser.emitError(parser.getNameLoc(), "expected function type");
+    return mlir::failure();
   mlir::FunctionType funcType =
       mlir::FunctionType::get(parser.getContext(), argTypes, resTypes);
   if (isDirect) {
diff --git a/flang/test/Fir/invalid.fir b/flang/test/Fir/invalid.fir
index 5471bcd8acc29..937ce891bfef0 100644
--- a/flang/test/Fir/invalid.fir
+++ b/flang/test/Fir/invalid.fir
@@ -1,5 +1,13 @@
 // RUN: fir-opt -split-input-file -verify-diagnostics --strict-fir-volatile-verifier %s
 
+func.func @call_missing_empty_result() {
+  // expected-error at +2{{expected '->' followed by function result list}}
+  fir.call @callee() : ()
+  return
+}
+
+// -----
+
 // expected-error at +1{{custom op 'fir.string_lit' must have character type}}
 %0 = fir.string_lit "Hello, World!"(13) : !fir.int<32>
 
diff --git a/mlir/include/mlir/Interfaces/CallInterfaces.h b/mlir/include/mlir/Interfaces/CallInterfaces.h
index 2bf3a3ca5f8a8..02c04c0f8dd1e 100644
--- a/mlir/include/mlir/Interfaces/CallInterfaces.h
+++ b/mlir/include/mlir/Interfaces/CallInterfaces.h
@@ -53,6 +53,7 @@ parseFunctionResultList(OpAsmParser &parser, SmallVectorImpl<Type> &resultTypes,
 /// signatures containing SSA region arguments (to parse these signatures, use
 /// function_interface_impl::parseFunctionSignature). When
 /// `mustParseEmptyResult`, `-> ()` is expected when there is no result type.
+/// Returns failure after emitting a diagnostic if the signature is malformed.
 ///
 ///   no-ssa-function-signature ::= `(` no-ssa-function-arg-list `)`
 ///                               -> function-result-list
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 0cc007433ac0e..223852d3828c4 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -1388,17 +1388,9 @@ static ParseResult parseCallTypeAndResolveOperands(
   }
   SmallVector<Type> argTypes;
   SmallVector<Type> resTypes;
-  SMLoc signatureLoc = parser.getCurrentLocation();
   if (call_interface_impl::parseFunctionSignature(parser, argTypes, argAttrs,
-                                                  resTypes, resultAttrs)) {
-    if (parser.getCurrentLocation() != signatureLoc)
-      return failure();
-    if (isDirect)
-      return parser.emitError(trailingTypesLoc,
-                              "expected direct call to have 1 trailing types");
-    return parser.emitError(trailingTypesLoc,
-                            "expected trailing function type");
-  }
+                                                  resTypes, resultAttrs))
+    return failure();
 
   if (resTypes.size() > 1)
     return parser.emitError(trailingTypesLoc,
diff --git a/mlir/test/Dialect/LLVMIR/invalid.mlir b/mlir/test/Dialect/LLVMIR/invalid.mlir
index 1d0bb0b6ef999..d44fa7ebfc4b1 100644
--- a/mlir/test/Dialect/LLVMIR/invalid.mlir
+++ b/mlir/test/Dialect/LLVMIR/invalid.mlir
@@ -254,8 +254,7 @@ func.func @call_missing_empty_result() {
 func.func private @standard_func_callee()
 
 func.func @call_missing_ptr_type(%arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected direct call to have 1 trailing type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call @standard_func_callee(%arg) : !llvm.ptr, (i8) -> (i8)
   llvm.return
 }
@@ -271,8 +270,7 @@ func.func @call_non_pointer_type(%callee : !llvm.func<i8 (i8)>, %arg : i8) {
 // -----
 
 func.func @call_non_function_type(%callee : !llvm.ptr, %arg : i8) {
-  // expected-error at +2 {{expected '('}}
-  // expected-error at +1 {{expected trailing function type}}
+  // expected-error at +1 {{expected '('}}
   llvm.call %callee(%arg) : !llvm.ptr, !llvm.func<i8 (i8)>
   llvm.return
 }


From llvmlistbot at llvm.org  Tue Jul 14 05:35:07 2026
From: llvmlistbot at llvm.org (Kunwar Grover)
Date: Tue, 14 Jul 2026 05:35:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a562cfb.9edb5c9d.208698.06e5@mx.google.com>


https://github.com/Groverkss edited https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Tue Jul 14 05:37:39 2026
From: llvmlistbot at llvm.org (Kirill Vedernikov)
Date: Tue, 14 Jul 2026 05:37:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a562d93.27b3b003.3b871.dfd3@mx.google.com>


kvederni wrote:

Could you please update this PR? It seems this PR has changes already merged to llvm:main from [PR207307](https://github.com/llvm/llvm-project/pull/207307).

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Tue Jul 14 05:40:22 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Tue, 14 Jul 2026 05:40:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] apply seq par dims to reduction
	variables (PR #209261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209261@github.com>
Message-ID: <6a562e36.f1c1923d.991d1.7cf8@mx.google.com>


https://github.com/rscottmanley updated https://github.com/llvm/llvm-project/pull/209261

>From 3d19a90e554316bd98233abc0602c911a944b12f Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Mon, 13 Jul 2026 07:00:57 -0700
Subject: [PATCH] [OpenACC] apply seq par dims to reduction variables

When the acc.parallel op is "effectively sequential" apply the correct
par dims during recipe materialization of reduction variables
---
 .../mlir/Dialect/OpenACC/OpenACCUtilsCG.h     | 13 ++++++++
 .../Transforms/ACCRecipeMaterialization.cpp   | 16 ++++++----
 .../acc-recipe-materialization-reduction.mlir | 32 +++++++++++++++++++
 3 files changed, 55 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index 8cfc4dac4ab36..9ab5365405d5d 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -95,6 +95,19 @@ void updateParDimsAttr(Operation *op, GPUParallelDimsAttr attr);
 /// Copy parallel dimensions from \p from to \p to.
 void copyParDimsAttr(Operation *from, Operation *to);
 
+/// Create a gang dim 1 GPUParallelDimsAttr based on the mapping policy.
+inline GPUParallelDimsAttr
+getGangDim1ParDimsAttr(MLIRContext *ctx, ACCToGPUMappingPolicy &policy) {
+  return GPUParallelDimsAttr::get(
+      ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+}
+
+/// Create a sequential GPUParallelDimsAttr based on the mapping policy.
+inline GPUParallelDimsAttr getSeqParDimsAttr(MLIRContext *ctx,
+                                             ACCToGPUMappingPolicy &policy) {
+  return GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
+}
+
 /// Tracks aligned byte consumption against a configurable shared memory cap.
 class SharedMemoryBudget {
 public:
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index d2e8d77c168d1..e666bcd9a6242 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -383,25 +383,29 @@ LogicalResult ACCRecipeMaterialization::materialize(
     cloneRegionIntoAccRegion(&combinerRegion, &combineRegionOp.getRegion(),
                              /*hasResult=*/false);
 
-    auto ctx = b.getContext();
+    auto *ctx = b.getContext();
 
     // For reductions that come from parallel constructs, explicitly set the
     // GPU parallel dimensions attribute to blockXDim since they will always be
     // gang private. GPU parallel dimensions cannot be determined for acc.loop
     // at this point.
     if constexpr (std::is_same_v<AccOpTy, acc::ParallelOp>) {
-      auto parDimsAttr = acc::GPUParallelDimsAttr::get(
-          ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+      acc::GPUParallelDimsAttr parDimsAttr;
+      if (accOp.isEffectivelySerial()) {
+        // If acc.serial has been lowered to a parallel op that is effectively
+        // sequential
+        parDimsAttr = acc::getSeqParDimsAttr(ctx, policy);
+      } else {
+        parDimsAttr = acc::getGangDim1ParDimsAttr(ctx, policy);
+      }
       acc::setParDimsAttr(reductionOp, parDimsAttr);
       acc::setParDimsAttr(combineRegionOp, parDimsAttr);
     }
 
     // Set sequential parallel dimensions attribute for loops in the recipe.
-    auto seqParDimsAttr =
-        acc::GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
     auto setSeqParDimsForRecipeLoops = [&](Region *r) {
       r->walk([&](LoopLikeOpInterface loopLike) {
-        acc::setParDimsAttr(loopLike, seqParDimsAttr);
+        acc::setParDimsAttr(loopLike, acc::getSeqParDimsAttr(ctx, policy));
       });
     };
     setSeqParDimsForRecipeLoops(&reductionOp.getRegion());
diff --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
index 6f2bafb2b6165..e5dc884d37d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
@@ -51,3 +51,35 @@ func.func @par_reduction_clause_(%arg0: memref<f64>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @par_reduction_clause_serial
+// CHECK:       acc.parallel {{.*}} {
+// CHECK:       [[PRIVATE:%.*]] = acc.reduction_init {{.*}} <add>
+// CHECK-NEXT:  [[ZERO:%.*]] = arith.constant 0.000000e+00 : f64
+// CHECK-NEXT:  [[ALLOCA:%.*]] = memref.alloca() : memref<f64>
+// CHECK-NEXT:  memref.store [[ZERO]], [[ALLOCA]][]
+// CHECK-NEXT:  acc.yield {{.*}}
+// CHECK:       } {{.*}}acc.par_dims = #acc<par_dims[sequential]>, acc.var_name = #acc.var_name<"tmp">
+// CHECK:       memref.load [[PRIVATE]][]
+// CHECK:       memref.store {{.*}}, [[PRIVATE]][]
+// CHECK:       acc.reduction_combine_region [[PRIVATE]] into [[REDUCVAR:%.*]] : memref<f64> {
+// CHECK:       [[LOADVAR:%.*]] = memref.load [[REDUCVAR]][]
+// CHECK-NEXT:  [[LOADPRIV:%.*]] = memref.load [[PRIVATE]][]
+// CHECK-NEXT:  [[COMBINE:%.*]] = arith.addf [[LOADVAR]], [[LOADPRIV]]
+// CHECK-NEXT:  memref.store [[COMBINE]], [[REDUCVAR]][]
+// CHECK-NEXT:  } {acc.par_dims = #acc<par_dims[sequential]>}
+// CHECK-NEXT:  memref.dealloc [[PRIVATE]] : memref<f64>
+// CHECK:       acc.yield
+
+func.func @par_reduction_clause_serial(%arg0: memref<f64>) {
+  %c1_i32 = arith.constant 1 : i32
+  %cst = arith.constant 1.000000e+00 : f64
+  %red = acc.reduction varPtr(%arg0 : memref<f64>) recipe(@reduction_add_memref_f64) -> memref<f64> {name = "tmp"}
+  acc.parallel num_gangs({%c1_i32 : i32}) num_workers(%c1_i32 : i32) vector_length(%c1_i32 : i32) reduction(%red : memref<f64>) {
+    %3 = memref.load %red[] : memref<f64>
+    %4 = arith.addf %3, %cst fastmath<contract> : f64
+    memref.store %4, %red[] : memref<f64>
+    acc.yield
+  }
+  return
+}


From llvmlistbot at llvm.org  Tue Jul 14 06:07:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 06:07:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2e86804 - [OpenACC] apply seq par dims to
 reduction variables (#209261)
Message-ID: <6a56349b.15d54cb3.1d1d4a.01fe@mx.google.com>


Author: Scott Manley
Date: 2026-07-14T08:07:33-05:00
New Revision: 2e86804e612fc6d1bace4e08b8404d0ea383d368

URL: https://github.com/llvm/llvm-project/commit/2e86804e612fc6d1bace4e08b8404d0ea383d368
DIFF: https://github.com/llvm/llvm-project/commit/2e86804e612fc6d1bace4e08b8404d0ea383d368.diff

LOG: [OpenACC] apply seq par dims to reduction variables (#209261)

When the acc.parallel op is "effectively sequential" apply the correct
par dims during recipe materialization of reduction variables

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
    mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
    mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
index 8cfc4dac4ab36..9ab5365405d5d 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsCG.h
@@ -95,6 +95,19 @@ void updateParDimsAttr(Operation *op, GPUParallelDimsAttr attr);
 /// Copy parallel dimensions from \p from to \p to.
 void copyParDimsAttr(Operation *from, Operation *to);
 
+/// Create a gang dim 1 GPUParallelDimsAttr based on the mapping policy.
+inline GPUParallelDimsAttr
+getGangDim1ParDimsAttr(MLIRContext *ctx, ACCToGPUMappingPolicy &policy) {
+  return GPUParallelDimsAttr::get(
+      ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+}
+
+/// Create a sequential GPUParallelDimsAttr based on the mapping policy.
+inline GPUParallelDimsAttr getSeqParDimsAttr(MLIRContext *ctx,
+                                             ACCToGPUMappingPolicy &policy) {
+  return GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
+}
+
 /// Tracks aligned byte consumption against a configurable shared memory cap.
 class SharedMemoryBudget {
 public:

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index d2e8d77c168d1..e666bcd9a6242 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -383,25 +383,29 @@ LogicalResult ACCRecipeMaterialization::materialize(
     cloneRegionIntoAccRegion(&combinerRegion, &combineRegionOp.getRegion(),
                              /*hasResult=*/false);
 
-    auto ctx = b.getContext();
+    auto *ctx = b.getContext();
 
     // For reductions that come from parallel constructs, explicitly set the
     // GPU parallel dimensions attribute to blockXDim since they will always be
     // gang private. GPU parallel dimensions cannot be determined for acc.loop
     // at this point.
     if constexpr (std::is_same_v<AccOpTy, acc::ParallelOp>) {
-      auto parDimsAttr = acc::GPUParallelDimsAttr::get(
-          ctx, {policy.gangDim(ctx, acc::ParLevel::gang_dim1)});
+      acc::GPUParallelDimsAttr parDimsAttr;
+      if (accOp.isEffectivelySerial()) {
+        // If acc.serial has been lowered to a parallel op that is effectively
+        // sequential
+        parDimsAttr = acc::getSeqParDimsAttr(ctx, policy);
+      } else {
+        parDimsAttr = acc::getGangDim1ParDimsAttr(ctx, policy);
+      }
       acc::setParDimsAttr(reductionOp, parDimsAttr);
       acc::setParDimsAttr(combineRegionOp, parDimsAttr);
     }
 
     // Set sequential parallel dimensions attribute for loops in the recipe.
-    auto seqParDimsAttr =
-        acc::GPUParallelDimsAttr::get(ctx, {policy.seqDim(ctx)});
     auto setSeqParDimsForRecipeLoops = [&](Region *r) {
       r->walk([&](LoopLikeOpInterface loopLike) {
-        acc::setParDimsAttr(loopLike, seqParDimsAttr);
+        acc::setParDimsAttr(loopLike, acc::getSeqParDimsAttr(ctx, policy));
       });
     };
     setSeqParDimsForRecipeLoops(&reductionOp.getRegion());

diff  --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
index 6f2bafb2b6165..e5dc884d37d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-reduction.mlir
@@ -51,3 +51,35 @@ func.func @par_reduction_clause_(%arg0: memref<f64>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @par_reduction_clause_serial
+// CHECK:       acc.parallel {{.*}} {
+// CHECK:       [[PRIVATE:%.*]] = acc.reduction_init {{.*}} <add>
+// CHECK-NEXT:  [[ZERO:%.*]] = arith.constant 0.000000e+00 : f64
+// CHECK-NEXT:  [[ALLOCA:%.*]] = memref.alloca() : memref<f64>
+// CHECK-NEXT:  memref.store [[ZERO]], [[ALLOCA]][]
+// CHECK-NEXT:  acc.yield {{.*}}
+// CHECK:       } {{.*}}acc.par_dims = #acc<par_dims[sequential]>, acc.var_name = #acc.var_name<"tmp">
+// CHECK:       memref.load [[PRIVATE]][]
+// CHECK:       memref.store {{.*}}, [[PRIVATE]][]
+// CHECK:       acc.reduction_combine_region [[PRIVATE]] into [[REDUCVAR:%.*]] : memref<f64> {
+// CHECK:       [[LOADVAR:%.*]] = memref.load [[REDUCVAR]][]
+// CHECK-NEXT:  [[LOADPRIV:%.*]] = memref.load [[PRIVATE]][]
+// CHECK-NEXT:  [[COMBINE:%.*]] = arith.addf [[LOADVAR]], [[LOADPRIV]]
+// CHECK-NEXT:  memref.store [[COMBINE]], [[REDUCVAR]][]
+// CHECK-NEXT:  } {acc.par_dims = #acc<par_dims[sequential]>}
+// CHECK-NEXT:  memref.dealloc [[PRIVATE]] : memref<f64>
+// CHECK:       acc.yield
+
+func.func @par_reduction_clause_serial(%arg0: memref<f64>) {
+  %c1_i32 = arith.constant 1 : i32
+  %cst = arith.constant 1.000000e+00 : f64
+  %red = acc.reduction varPtr(%arg0 : memref<f64>) recipe(@reduction_add_memref_f64) -> memref<f64> {name = "tmp"}
+  acc.parallel num_gangs({%c1_i32 : i32}) num_workers(%c1_i32 : i32) vector_length(%c1_i32 : i32) reduction(%red : memref<f64>) {
+    %3 = memref.load %red[] : memref<f64>
+    %4 = arith.addf %3, %cst fastmath<contract> : f64
+    memref.store %4, %red[] : memref<f64>
+    acc.yield
+  }
+  return
+}


        

From llvmlistbot at llvm.org  Tue Jul 14 06:07:42 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Tue, 14 Jul 2026 06:07:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] apply seq par dims to reduction
	variables (PR #209261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209261@github.com>
Message-ID: <6a56349e.a2599b5a.2f2332.d0a4@mx.google.com>


https://github.com/rscottmanley closed https://github.com/llvm/llvm-project/pull/209261

From llvmlistbot at llvm.org  Tue Jul 14 06:08:18 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c2.a0beed4e.2c939e.a7f7@mx.google.com>


https://github.com/gysit commented:

Thanks for the fix! 

I think this works modulo comments. Adding @Moxinilian to to have a look. Maybe there are other places where scalable vectors break assumptions. 

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:08:18 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c2.2ae50a66.24523.7833@mx.google.com>


https://github.com/gysit edited https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:08:18 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c2.18d803df.1026d8.61a5@mx.google.com>


================
@@ -256,6 +256,12 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcType == targetType)
----------------
gysit wrote:

A possible alternative would have to been to extend the `getTypeSizeInBits` calls below to check if the `TypeSize` has a fixed size. That would be more conservative and failsafe since the current code assumes we never get here with two different scalable vector types (which is probably ensured by an earlier cast compatibility check). 

However, I believe your fix works for the case where all loads and stores are of the same scalable vector type and it sort of complements the implementation of `areConversionCompatible`. 



https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:08:19 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c3.dda25a5e.1db851.5c94@mx.google.com>


================
@@ -256,6 +256,12 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
----------------
gysit wrote:

```suggestion
  // Nothing has to be done if the types are already the same. This also
  // avoids querying the bit size of scalable vector types below.
```
nit: it is fine to query the type size of scalable vectors (the implicit cast of the return value is the issue).

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:08:19 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c3.15d54cb3.1d1d4a.0a73@mx.google.com>


================
@@ -1035,6 +1035,38 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// Storing into a scalable vector slot with a type matching the slot's
+// element type must not query the type's bit size, as this is not possible
+// for scalable vectors. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store
+llvm.func @scalable_vector_matching_store(%arg: vector<[4]xi1>) {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  // CHECK-NOT: llvm.store
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  llvm.return
+}
+
+// -----
+
+// Loading back a scalable vector value that was stored with a matching type
+// must also avoid querying the type's bit size. See #209065.
----------------
gysit wrote:

```suggestion
```

I think the second test is good enough? Or is there something the first test exercises which is not exercised by the second test?

nit: I would drop the comment. It is completely fine to check the type size and the main purpose of the test is checking the scalable vector promotion which is already reflected by the test name?

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:08:20 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:08:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5634c4.8df1d642.1d0119.18f3@mx.google.com>


================
@@ -291,6 +297,13 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
                                  srcValue.getType(),
                                  /*narrowingConversion=*/false) &&
          "expected that the compatibility was checked before");
+
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
----------------
gysit wrote:

```suggestion
  // Nothing has to be done if the types are already the same. This also
  // avoids querying the bit size of scalable vector types below.
```
nit: same

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Tue Jul 14 06:12:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 06:12:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8518c74 - [mlir][llvm] Preserve function
 entry count metadata (#204707)
Message-ID: <6a5635c5.bb286e2c.394855.b09a@mx.google.com>


Author: 1sgtpepper
Date: 2026-07-14T15:12:33+02:00
New Revision: 8518c74b5ef8950b9b604df588f29d9037fe1380

URL: https://github.com/llvm/llvm-project/commit/8518c74b5ef8950b9b604df588f29d9037fe1380
DIFF: https://github.com/llvm/llvm-project/commit/8518c74b5ef8950b9b604df588f29d9037fe1380.diff

LOG: [mlir][llvm] Preserve function entry count metadata (#204707)

Fixes #202374.

Preserve LLVM function `!prof` metadata for:
- `synthetic_function_entry_count`
- `function_entry_count` with import GUID operands

The importer represents the entry count, count kind, and trailing import
GUID operands in a single `#llvm.function_entry_count` attribute,
validates malformed operands before mutating the operation, and exports
the metadata through `llvm::Function::setEntryCount`.

Tests cover import, export, malformed metadata operands, synthetic
counts, synthetic counts with trailing operands, import GUID
round-tripping, and raw i64 bit-pattern preservation.

Checks:
- `git diff --check`
- `git clang-format --diff origin/main -- ...`
- `llvm-as -disable-output` smoke check

Codex was used in making this PR.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
    mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
    mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
    mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
    mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
    mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
    mlir/test/Target/LLVMIR/Import/function-attributes.ll
    mlir/test/Target/LLVMIR/llvmir.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 72497a5328f78..6e2d4628c6173 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -96,6 +96,28 @@ def FramePointerKindAttr : LLVM_Attr<"FramePointerKind", "framePointerKind"> {
   let assemblyFormat = "`<` $framePointerKind `>`";
 }
 
+//===----------------------------------------------------------------------===//
+// FunctionEntryCountAttr
+//===----------------------------------------------------------------------===//
+
+def LLVM_FunctionEntryCountAttr
+    : LLVM_Attr<"FunctionEntryCount", "function_entry_count"> {
+  let summary = "LLVM function entry count profile metadata";
+  let description = [{
+    Models function-level `!prof` entry-count metadata. The `entry_count` field
+    stores the unsigned 64-bit counter bit pattern. The `count_type` field
+    defaults to real and selects whether the metadata is emitted as
+    `"function_entry_count"` or `"synthetic_function_entry_count"`. The optional
+    `imports` field stores the trailing import GUID operands used by ThinLTO
+    sample PGO.
+  }];
+  let parameters = (ins "uint64_t":$entry_count,
+                        DefaultValuedParameter<"ProfileCountType",
+                                               "ProfileCountType::Real">:$count_type,
+                        OptionalArrayRefParameter<"uint64_t">:$imports);
+  let assemblyFormat = "`<` struct(params) `>`";
+}
+
 //===----------------------------------------------------------------------===//
 // Loop Attributes
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
index 51ac465000341..5a8f195a91a10 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMEnums.td
@@ -83,6 +83,21 @@ def AsmATTOrIntel : LLVM_EnumAttr<
   let cppNamespace = "::mlir::LLVM";
 }
 
+//===----------------------------------------------------------------------===//
+// ProfileCountType
+//===----------------------------------------------------------------------===//
+
+def ProfileCountReal : I64EnumAttrCase<
+  /*string sym=*/"Real", /*int val=*/0, /*string str=*/"real">;
+def ProfileCountSynthetic : I64EnumAttrCase<
+  /*string sym=*/"Synthetic", /*int val=*/1, /*string str=*/"synthetic">;
+def ProfileCountTypeAttr : I64EnumAttr<
+  /*string name=*/"ProfileCountType",
+  /*string description=*/"real or synthetic function entry count",
+  /*list<I64EnumAttrCase> cases=*/[ProfileCountReal, ProfileCountSynthetic]> {
+  let cppNamespace = "::mlir::LLVM";
+}
+
 //===----------------------------------------------------------------------===//
 // Atomic Operations
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
index 9d112e5ea227e..4bdcf5f6c1cc5 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMOps.td
@@ -2018,6 +2018,11 @@ def LLVM_LLVMFuncOp : LLVM_Op<"func", [
       llvm.return
     }
     ```
+
+    The `function_entry_count` attribute models function-level `!prof`
+    entry-count metadata. It stores the entry count, whether the count is real
+    or synthetic, and any trailing import GUID operands in a single
+    `#llvm.function_entry_count` attribute.
   }];
 
   let arguments = (ins
@@ -2034,7 +2039,7 @@ def LLVM_LLVMFuncOp : LLVM_Op<"func", [
     OptionalAttr<ArrayAttr>:$passthrough,
     OptionalAttr<DictArrayAttr>:$arg_attrs,
     OptionalAttr<DictArrayAttr>:$res_attrs,
-    OptionalAttr<I64Attr>:$function_entry_count,
+    OptionalAttr<LLVM_FunctionEntryCountAttr>:$function_entry_count,
     OptionalAttr<LLVM_MemoryEffectsAttr>:$memory_effects,
     DefaultValuedAttr<Visibility, "mlir::LLVM::Visibility::Default">:$visibility_,
     UnitAttr:$arm_streaming, UnitAttr:$arm_locally_streaming,

diff  --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index 58f569abff8ea..f3dc00e177b0f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -26,6 +26,7 @@
 
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/Support/Error.h"
@@ -3044,7 +3045,9 @@ void LLVMFuncOp::build(OpBuilder &builder, OperationState &result,
     result.addAttribute(getComdatAttrName(result.name), comdat);
   if (functionEntryCount)
     result.addAttribute(getFunctionEntryCountAttrName(result.name),
-                        builder.getI64IntegerAttr(functionEntryCount.value()));
+                        FunctionEntryCountAttr::get(
+                            builder.getContext(), *functionEntryCount,
+                            ProfileCountType::Real, ArrayRef<uint64_t>{}));
 #ifndef NDEBUG
   std::optional<NamedAttribute> duplicate = result.attributes.findDuplicate();
   if (duplicate.has_value()) {

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
index e9cd335835263..b6dee4e89ee4c 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMIRToLLVMTranslation.cpp
@@ -23,6 +23,7 @@
 #include "llvm/IR/Instructions.h"
 #include "llvm/IR/IntrinsicInst.h"
 #include "llvm/IR/MemoryModelRelaxationAnnotations.h"
+#include <optional>
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -99,6 +100,14 @@ getSupportedMetadataImpl(llvm::LLVMContext &llvmContext) {
   return convertibleMetadata;
 }
 
+/// Extracts an LLVM metadata constant as an unsigned 64-bit integer.
+static std::optional<uint64_t> getUInt64Metadata(llvm::Metadata *metadata) {
+  auto *constant = llvm::mdconst::dyn_extract<llvm::ConstantInt>(metadata);
+  if (!constant)
+    return std::nullopt;
+  return constant->getValue().tryZExtValue();
+}
+
 /// Converts the given profiling metadata `node` to an MLIR profiling attribute
 /// and attaches it to the imported operation if the translation succeeds.
 /// Returns failure otherwise.
@@ -112,27 +121,45 @@ static LogicalResult setProfilingAttr(OpBuilder &builder, llvm::MDNode *node,
   auto *name = dyn_cast<llvm::MDString>(node->getOperand(0));
   if (!name)
     return failure();
+  StringRef profName = name->getString();
 
   // Handle function entry count metadata.
-  if (name->getString() == llvm::MDProfLabels::FunctionEntryCount) {
-
-    // TODO support function entry count metadata with GUID fields.
-    if (node->getNumOperands() != 2)
+  if (profName == llvm::MDProfLabels::FunctionEntryCount ||
+      profName == llvm::MDProfLabels::SyntheticFunctionEntryCount) {
+    if (node->getNumOperands() < 2)
       return failure();
 
-    llvm::ConstantInt *entryCount =
-        llvm::mdconst::dyn_extract<llvm::ConstantInt>(node->getOperand(1));
-    if (!entryCount)
+    bool isSynthetic =
+        profName == llvm::MDProfLabels::SyntheticFunctionEntryCount;
+    ProfileCountType profileCountType =
+        isSynthetic ? ProfileCountType::Synthetic : ProfileCountType::Real;
+
+    std::optional<uint64_t> entryCountValue =
+        getUInt64Metadata(node->getOperand(1));
+    if (!entryCountValue)
       return failure();
+
+    SmallVector<uint64_t> importGUIDValues;
+    importGUIDValues.reserve(node->getNumOperands() - 2);
+    for (unsigned idx = 2, e = node->getNumOperands(); idx < e; ++idx) {
+      std::optional<uint64_t> guidValue =
+          getUInt64Metadata(node->getOperand(idx));
+      if (!guidValue)
+        return failure();
+      importGUIDValues.push_back(*guidValue);
+    }
+
     if (auto funcOp = dyn_cast<LLVMFuncOp>(op)) {
-      funcOp.setFunctionEntryCount(entryCount->getZExtValue());
+      funcOp.setFunctionEntryCountAttr(
+          FunctionEntryCountAttr::get(builder.getContext(), *entryCountValue,
+                                      profileCountType, importGUIDValues));
       return success();
     }
     return op->emitWarning()
            << "expected function_entry_count to be attached to a function";
   }
 
-  if (name->getString() != llvm::MDProfLabels::BranchWeights)
+  if (profName != llvm::MDProfLabels::BranchWeights)
     return failure();
   // The branch_weights metadata must have at least 2 operands.
   if (node->getNumOperands() < 2)

diff  --git a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
index a9ca4a57e8e8a..b76a803f93ad9 100644
--- a/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp
@@ -31,6 +31,7 @@
 #include "mlir/Target/LLVMIR/LLVMTranslationInterface.h"
 #include "mlir/Target/LLVMIR/TypeToLLVM.h"
 
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/TypeSwitch.h"
@@ -2018,8 +2019,19 @@ LogicalResult ModuleTranslation::convertFunctionSignatures() {
     convertFunctionKernelAttributes(function, llvmFunc, *this);
 
     // Convert function_entry_count attribute to metadata.
-    if (std::optional<uint64_t> entryCount = function.getFunctionEntryCount())
-      llvmFunc->setEntryCount(entryCount.value());
+    if (auto entryCount = function.getFunctionEntryCountAttr()) {
+      ArrayRef<uint64_t> imports = entryCount.getImports();
+      llvm::DenseSet<llvm::GlobalValue::GUID> importGUIDs;
+      if (!imports.empty())
+        importGUIDs.insert(imports.begin(), imports.end());
+      llvm::MDBuilder metadataBuilder(llvmFunc->getContext());
+      llvmFunc->setMetadata(
+          llvm::LLVMContext::MD_prof,
+          metadataBuilder.createFunctionEntryCount(
+              entryCount.getEntryCount(),
+              entryCount.getCountType() == ProfileCountType::Synthetic,
+              imports.empty() ? nullptr : &importGUIDs));
+    }
 
     // Convert result attributes.
     if (ArrayAttr allResultAttrs = function.getAllResultAttrs()) {

diff  --git a/mlir/test/Target/LLVMIR/Import/function-attributes.ll b/mlir/test/Target/LLVMIR/Import/function-attributes.ll
index 5d664519e100c..1c9c616ce3c5b 100644
--- a/mlir/test/Target/LLVMIR/Import/function-attributes.ll
+++ b/mlir/test/Target/LLVMIR/Import/function-attributes.ll
@@ -171,7 +171,7 @@ declare range(i64 0, 4097) i64 @func_res_attr_range()
 ; // -----
 
 ; CHECK-LABEL: @entry_count
-; CHECK-SAME:  attributes {function_entry_count = 4242 : i64}
+; CHECK-SAME:  attributes {function_entry_count = #llvm.function_entry_count<entry_count = 4242>}
 define void @entry_count() !prof !1 {
   ret void
 }
@@ -180,6 +180,80 @@ define void @entry_count() !prof !1 {
 
 ; // -----
 
+; CHECK-LABEL: @synthetic_entry_count
+; CHECK-SAME:  attributes {function_entry_count = #llvm.function_entry_count<entry_count = 7, count_type = synthetic>}
+define void @synthetic_entry_count() !prof !2 {
+  ret void
+}
+
+!2 = !{!"synthetic_function_entry_count", i64 7}
+
+; // -----
+
+; CHECK-LABEL: @entry_count_imports
+; CHECK-SAME:  attributes {function_entry_count = #llvm.function_entry_count<entry_count = 7, imports = 1234, 18446744073709551615, 4, 1234>}
+define void @entry_count_imports() !prof !3 {
+  ret void
+}
+
+!3 = !{!"function_entry_count", i64 7, i64 1234, i64 -1, i64 4, i64 1234}
+
+; // -----
+
+; CHECK-LABEL: @synthetic_entry_count_imports
+; CHECK-SAME:  attributes {function_entry_count = #llvm.function_entry_count<entry_count = 7, count_type = synthetic, imports = 1234>}
+define void @synthetic_entry_count_imports() !prof !4 {
+  ret void
+}
+
+!4 = !{!"synthetic_function_entry_count", i64 7, i64 1234}
+
+; // -----
+
+; CHECK-LABEL: @entry_count_malformed_import
+; CHECK-NOT: function_entry_count
+; expected-warning @unknown {{unhandled function metadata}}
+define void @entry_count_malformed_import() !prof !5 {
+  ret void
+}
+
+!5 = !{!"function_entry_count", i64 7, !"bad"}
+
+; // -----
+
+; CHECK-LABEL: @entry_count_too_wide_count
+; CHECK-NOT: function_entry_count
+; expected-warning @unknown {{unhandled function metadata}}
+define void @entry_count_too_wide_count() !prof !6 {
+  ret void
+}
+
+!6 = !{!"function_entry_count", i128 18446744073709551616}
+
+; // -----
+
+; CHECK-LABEL: @entry_count_too_wide_import
+; CHECK-NOT: function_entry_count
+; expected-warning @unknown {{unhandled function metadata}}
+define void @entry_count_too_wide_import() !prof !7 {
+  ret void
+}
+
+!7 = !{!"function_entry_count", i64 7, i128 18446744073709551616}
+
+; // -----
+
+; Preserve the raw i64 metadata bit pattern.
+; CHECK-LABEL: @entry_count_negative_count
+; CHECK-SAME:  attributes {function_entry_count = #llvm.function_entry_count<entry_count = 18446744073709551615>}
+define void @entry_count_negative_count() !prof !8 {
+  ret void
+}
+
+!8 = !{!"function_entry_count", i64 -1}
+
+; // -----
+
 ; CHECK-LABEL: @func_memory
 ; CHECK-SAME:  attributes {memory_effects = #llvm.memory_effects<other = readwrite, argMem = none, inaccessibleMem = readwrite, errnoMem = readwrite, targetMem0 = readwrite, targetMem1 = readwrite>}
 ; CHECK:   llvm.return

diff  --git a/mlir/test/Target/LLVMIR/llvmir.mlir b/mlir/test/Target/LLVMIR/llvmir.mlir
index 5eca8f19154a1..a665c72f4f54c 100644
--- a/mlir/test/Target/LLVMIR/llvmir.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir.mlir
@@ -1893,12 +1893,62 @@ llvm.func @my_allocator(i64) attributes {passthrough = [["allocsize", "429496729
 // -----
 
 // CHECK-LABEL: @functionEntryCount
-// CHECK-SAME: !prof ![[PROF_ID:[0-9]*]]
-llvm.func @functionEntryCount() attributes {function_entry_count = 4242 : i64} {
+// CHECK-SAME: !prof ![[PROF_ID:[0-9]+]]
+llvm.func @functionEntryCount() attributes {
+  function_entry_count = #llvm.function_entry_count<entry_count = 4242>
+} {
   llvm.return
 }
 
-// CHECK: ![[PROF_ID]] = !{!"function_entry_count", i64 4242}
+// CHECK-DAG: ![[PROF_ID]] = !{!"function_entry_count", i64 4242}
+
+// -----
+
+// CHECK-LABEL: @syntheticFunctionEntryCount
+// CHECK-SAME: !prof ![[SYNTH_PROF_ID:[0-9]+]]
+llvm.func @syntheticFunctionEntryCount() attributes {
+  function_entry_count = #llvm.function_entry_count<entry_count = 7, count_type = synthetic>
+} {
+  llvm.return
+}
+
+// CHECK-DAG: ![[SYNTH_PROF_ID]] = !{!"synthetic_function_entry_count", i64 7}
+
+// -----
+
+// CHECK-LABEL: @syntheticFunctionEntryCountWithImports
+// CHECK-SAME: !prof ![[SYNTH_IMPORTS_PROF_ID:[0-9]+]]
+llvm.func @syntheticFunctionEntryCountWithImports() attributes {
+  function_entry_count = #llvm.function_entry_count<entry_count = 7, count_type = synthetic, imports = 1234, 4, 1234>
+} {
+  llvm.return
+}
+
+// CHECK-DAG: ![[SYNTH_IMPORTS_PROF_ID]] = !{!"synthetic_function_entry_count", i64 7, i64 4, i64 1234}
+
+// -----
+
+// CHECK-LABEL: @functionEntryCountWithImports
+// CHECK-SAME: !prof ![[IMPORTS_PROF_ID:[0-9]+]]
+llvm.func @functionEntryCountWithImports() attributes {
+  function_entry_count = #llvm.function_entry_count<entry_count = 7, imports = 1234, 4, 18446744073709551615, 1234>
+} {
+  llvm.return
+}
+
+// CHECK-DAG: ![[IMPORTS_PROF_ID]] = !{!"function_entry_count", i64 7, i64 4, i64 1234, i64 -1}
+
+// -----
+
+// CHECK-LABEL: @functionEntryCountNegativeCount
+// CHECK-SAME: !prof ![[NEG_PROF_ID:[0-9]+]]
+llvm.func @functionEntryCountNegativeCount() attributes {
+  function_entry_count = #llvm.function_entry_count<entry_count = 18446744073709551615>
+} {
+  llvm.return
+}
+
+// CHECK-DAG: ![[NEG_PROF_ID]] = !{!"function_entry_count", i64 -1}
 
 // -----
 


        

From llvmlistbot at llvm.org  Tue Jul 14 06:12:45 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Tue, 14 Jul 2026 06:12:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a5635cd.d84a4299.3853e5.2fb9@mx.google.com>


https://github.com/gysit closed https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Tue Jul 14 06:13:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 06:13:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][llvm] Preserve function entry count
	metadata (PR #204707)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204707@github.com>
Message-ID: <6a5635dd.267a80c6.18c8fb.70c7@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@1sgtpepper Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/204707

From llvmlistbot at llvm.org  Tue Jul 14 06:19:05 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Tue, 14 Jul 2026 06:19:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] apply seq par dims to reduction
	variables (PR #209261)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209261@github.com>
Message-ID: <6a563749.91d695f4.18551.15ce@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-01` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/12880

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8681/8706] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-bins && /usr/local/therock-tools/bin/cmake --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_CXX_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_Fortran_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_ASM_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=OFF -DLLVM_INCLUDE_TESTS=ON -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON -DLLVM_RUNTIMES_TARGET=amdgpu-amd-amdhsa -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_DEFAULT_TARGET_TRIPLE=amdgpu-amd-amdhsa -DFORTRAN_SUPPORTS_REAL16=0 "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DLLVM_USE_LINKER= -DLLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes-amdgpu-amd-amdhsa/tmp/runtimes-amdgpu-amd-amdhsa-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure
[8683/8706] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8684/8706] Linking CXX executable bin/tco
[8685/8706] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8681/8706] Creating library symlink lib/libflangPasses.so
FAILED: runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-bins && /usr/local/therock-tools/bin/cmake --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_CXX_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_Fortran_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_ASM_COMPILER_TARGET=amdgpu-amd-amdhsa -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=23.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=OFF -DLLVM_INCLUDE_TESTS=ON -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON -DLLVM_RUNTIMES_TARGET=amdgpu-amd-amdhsa -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_DEFAULT_TARGET_TRIPLE=amdgpu-amd-amdhsa -DFORTRAN_SUPPORTS_REAL16=0 "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DLLVM_USE_LINKER= -DLLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes-amdgpu-amd-amdhsa/tmp/runtimes-amdgpu-amd-amdhsa-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-amdgpu-amd-amdhsa-stamps/runtimes-amdgpu-amd-amdhsa-configure
[8683/8706] Linking CXX shared library lib/libflangFrontend.so.23.0git
[8684/8706] Linking CXX executable bin/tco
[8685/8706] Linking CXX executable bin/fir-opt
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=78.244973

```

</details>

https://github.com/llvm/llvm-project/pull/209261

From llvmlistbot at llvm.org  Tue Jul 14 06:26:41 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Tue, 14 Jul 2026 06:26:41 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [IR] Make semantics of strictfp
	consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a563911.94ba4530.280c00.1ba3@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From dda490564946867c1de66df3149d8210eb0e0467 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir |  8 +--
 20 files changed, 120 insertions(+), 113 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index f7e3ca8db80e1..0c913a254854c 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26572,9 +26574,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index fa0fe7a2e6092..2b4f0048cae4f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..dc184fa87f7f3 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strict {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..8a97798de67fb 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) strict {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) strict {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) strict {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) strict {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64


From llvmlistbot at llvm.org  Tue Jul 14 06:27:08 2026
From: llvmlistbot at llvm.org (Leandro Lupori)
Date: Tue, 14 Jul 2026 06:27:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a56392c.28821aa3.319535.73da@mx.google.com>


luporl wrote:

@mjklemm @kparzysz is there anything still blocking this PR?

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Tue Jul 14 06:48:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 06:48:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a563e28.8e75e0cb.3dbce4.76f9@mx.google.com>


https://github.com/agozillon approved this pull request.

LGTM, thank you for the fix! :-)

https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Tue Jul 14 06:57:27 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 06:57:27 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a564047.481afadf.196d57.73bb@mx.google.com>


================

----------------
Saieiei wrote:

Thank you for the feedback, @skatrak , and I’m sorry for the unnecessary review burden. I relied too heavily on LLM-generated comments. I’ll tighten my workflow before requesting future reviews. I’ll address the remaining nits and add the required Assisted-by tag. Thank you for your time and patience.

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 06:58:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 06:58:12 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [llvm] [mlir] Apply
 fixes from bugprone-container-bounds-check-overflow check (PR #206484)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206484@github.com>
Message-ID: <6a564074.4a342cbc.ca844.a937@mx.google.com>


https://github.com/Harald-R closed https://github.com/llvm/llvm-project/pull/206484

From llvmlistbot at llvm.org  Tue Jul 14 06:59:01 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 06:59:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVGPU] Add truncf and extf Ops (PR
	#199700)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199700@github.com>
Message-ID: <6a5640a5.d681cab9.333b2b.cadf@mx.google.com>


gburgessiv wrote:

Hi, looks like this broke builders: https://lab.llvm.org/buildbot/#/builders/228/builds/4754 . Working on a fix-forward now.

https://github.com/llvm/llvm-project/pull/199700

From llvmlistbot at llvm.org  Tue Jul 14 07:01:30 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Tue, 14 Jul 2026 07:01:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a56413a.2490598a.cf17c.dbe4@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 15c26a195567d9aa8b03929fdf1f25c6d7f1e87a Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 .../cl20-device-side-enqueue-attributes.cl    |  2 +-
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 llvm/test/CodeGen/SystemZ/zos-ppa2.ll         |  2 +-
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 22 +++---
 22 files changed, 129 insertions(+), 122 deletions(-)

diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..b4b4ad1143766 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index f7e3ca8db80e1..0c913a254854c 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26572,9 +26574,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index fa0fe7a2e6092..2b4f0048cae4f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
index a4c1d50df5543..4e6de83d7b9c2 100644
--- a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
+++ b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
@@ -21,7 +21,7 @@
 ; CHECK:  DC XL2'0000'
 ; CHECK: L#DVS DS 0H
 ; CHECK:  DC XL14'F1F9F7F0F0F1F0F1F0F0F0F0F0F0'
-; CHECK:  DC XL6'F2F3F0F0F0F0'
+; CHECK:  DC XL6'F2F4F0F0F0F0'
 ; CHECK:  DC XL2'0000'
 
 ; CHECK: C_@@QPPA2 CATTR ALIGN(3),FILL(0),NOTEXECUTABLE,READONLY,RMODE(64),PART(.
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..39b8e8c03996e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthru = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64


From llvmlistbot at llvm.org  Tue Jul 14 07:02:19 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 07:02:19 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a56416b.d76d81af.39f9be.09fc@mx.google.com>


https://github.com/Saieiei edited https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 07:07:52 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:07:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] mlir: fix builders by moving var into assert
	(PR #209494)
Message-ID: <llvm.org/llvm/llvm-project/pull/209494@github.com>


https://github.com/gburgessiv created https://github.com/llvm/llvm-project/pull/209494

non-asserts builders are failing since `vecTy` is unused aside from this one assertion: https://lab.llvm.org/buildbot/#/builders/228/builds/4754

Since side-effects here are uninteresting, move the entire expr into the assert, per CodingStandards.md

Fix-forward for #199700

>From 300b04bf455db87ccbbabbfd8fc52efa912c53b1 Mon Sep 17 00:00:00 2001
From: George Burgess IV <gbiv at google.com>
Date: Tue, 14 Jul 2026 07:56:20 -0600
Subject: [PATCH] mlir: fix builders by moving var into assert

non-asserts builders are failing since `vecTy` is unused aside from this
one assertion: https://lab.llvm.org/buildbot/#/builders/228/builds/4754

Since side-effects here are uninteresting, move the entire expr into the
assert, per CodingStandards.md

Fix-forward for #199700
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 948229bb53328..e97546f43c159 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,8 +1807,8 @@ lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
-  auto vecTy = cast<VectorType>(srcVec.getType());
-  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+  assert(idx >= 0 &&
+         idx < cast<VectorType>(srcVec.getType()).getNumElements() &&
          "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(


From llvmlistbot at llvm.org  Tue Jul 14 07:08:33 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:08:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] mlir: fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a5642e1.9edb5c9d.208698.0185@mx.google.com>


https://github.com/gburgessiv updated https://github.com/llvm/llvm-project/pull/209494

>From c641d0ce2c4b75358b6a91e890efd29a8eaded11 Mon Sep 17 00:00:00 2001
From: George Burgess IV <gbiv at google.com>
Date: Tue, 14 Jul 2026 07:56:20 -0600
Subject: [PATCH] mlir: fix builders by moving var into assert

non-asserts builders are failing since `vecTy` is unused aside from this
one assertion: https://lab.llvm.org/buildbot/#/builders/228/builds/4754

Since side-effects here are uninteresting, move the entire expr into the
assert, per CodingStandards.md

Fix-forward for #199700
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 948229bb53328..e97546f43c159 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -1807,8 +1807,8 @@ lookupConvOp(const TableEntry (&table)[N], Type srcElemType, Type dstElemType) {
 
 /// Extract a single element from a vector.
 static Value extractElement(ImplicitLocOpBuilder &b, Value srcVec, int idx) {
-  auto vecTy = cast<VectorType>(srcVec.getType());
-  assert(idx >= 0 && idx < vecTy.getNumElements() &&
+  assert(idx >= 0 &&
+         idx < cast<VectorType>(srcVec.getType()).getNumElements() &&
          "extractElement: index out of bounds");
   IntegerType i64Ty = b.getI64Type();
   return b.create<LLVM::ExtractElementOp>(


From llvmlistbot at llvm.org  Tue Jul 14 07:08:51 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:08:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] mlir: fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a5642f3.ff037943.2e959a.8c7b@mx.google.com>


gburgessiv wrote:

Waiting for local verification to finish, then will find a reviewer

https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:19:51 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:19:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] mlir: fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a564587.e5907290.281f16.4ae4@mx.google.com>


gburgessiv wrote:

Local build passed.

https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:19:59 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:19:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] mlir: fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a56458f.6c5720b3.15e90c.4d9a@mx.google.com>


https://github.com/gburgessiv ready_for_review https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:21:02 2026
From: llvmlistbot at llvm.org (George Burgess IV)
Date: Tue, 14 Jul 2026 07:21:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a5645ce.c56fb102.7c9cd.e8da@mx.google.com>


https://github.com/gburgessiv edited https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:22:47 2026
From: llvmlistbot at llvm.org (Aiden Grossman)
Date: Tue, 14 Jul 2026 07:22:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a564637.ff037943.2e959a.d7a4@mx.google.com>


https://github.com/boomanaiden154 approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:23:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 07:23:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a56464a.510f576d.1b3cb9.0266@mx.google.com>


https://github.com/cmtice approved this pull request.


https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:23:11 2026
From: llvmlistbot at llvm.org (Derek Schuff)
Date: Tue, 14 Jul 2026 07:23:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] fix builders by moving var into assert
	(PR #209494)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209494@github.com>
Message-ID: <6a56464f.c5504292.28b3b8.8201@mx.google.com>


https://github.com/dschuff approved this pull request.


https://github.com/llvm/llvm-project/pull/209494

From llvmlistbot at llvm.org  Tue Jul 14 07:31:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 07:31:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a564827.d681cab9.333b2b.ed1b@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207342

>From aca6098aff076bee1214c0a85be73b9648e44711 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:43:32 +0000
Subject: [PATCH] [NVGPU] Add FP8 (e4m3/e5m2) support to nvgpu.mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 948229bb53328..89cc2e5328a84 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -330,6 +330,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index c77ecb8431ad8..642e588488641 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) || aType.isF8E4M3FN() ||
+             aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Tue Jul 14 07:36:08 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 07:36:08 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a564958.27b3b003.3b871.1507@mx.google.com>


https://github.com/Saieiei updated https://github.com/llvm/llvm-project/pull/199967

>From 3992543570d9f3df84aefc3c38a735df79eec589 Mon Sep 17 00:00:00 2001
From: Sairudra More <sairudra60 at gmail.com>
Date: Fri, 12 Jun 2026 04:57:07 -0500
Subject: [PATCH] [flang][OpenMP] Lower target in_reduction

Enable host lowering for target in_reduction in Flang and MLIR OpenMP
translation.

Model target in_reduction through the matching map entry, force
address-preserving implicit mapping for Flang in_reduction list items, and emit
the host-side task-reduction lookup with __kmpc_task_reduction_get_th_data. The
runtime entry point takes and returns a generic, default-address-space pointer,
so normalize a non-default-address-space captured pointer to the generic address
space before the call and cast the returned private pointer back to the map
block argument's address space, mirroring the in_reduction handling on
omp.taskloop. On the target device, in_reduction is handled as a regular
map(tofrom) variable. The byref modifier, two-argument initializers, cleanup
regions, and the remaining Flang COMMON/EQUIVALENCE/privatized-variable cases
continue to be diagnosed.

Add Flang lowering, MLIR verifier/translation, and LLVM IR tests for the
supported host path, including a non-default-address-space case, and the
remaining unsupported cases.
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 103 ++++++++++--
 .../OpenMP/target-inreduction-llvmir.f90      |  50 ++++++
 .../OpenMP/Todo/target-inreduction-common.f90 |  20 +++
 .../Todo/target-inreduction-equivalence.f90   |  19 +++
 .../Todo/target-inreduction-firstprivate.f90  |  19 +++
 .../Lower/OpenMP/Todo/target-inreduction.f90  |  15 --
 .../OpenMP/target-inreduction-unused.f90      |  27 ++++
 .../test/Lower/OpenMP/target-inreduction.f90  |  30 ++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  33 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  50 ++++++
 .../mlir/Dialect/OpenMP/OpenMPClauses.td      |  14 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  22 ++-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 149 +++++++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 138 ++++++++++++++--
 .../Dialect/OpenMP/host-op-filtering.mlir     |   8 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  85 ++++++++++
 .../openmp-target-in-reduction-device.mlir    |  51 ++++++
 .../openmp-target-in-reduction-multi.mlir     |  77 +++++++++
 .../LLVMIR/openmp-target-in-reduction.mlir    | 107 +++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  91 ++++++++++-
 20 files changed, 1029 insertions(+), 79 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction-unused.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 9c6352b73a585..23eb9d587d852 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2474,6 +2474,7 @@ genTargetClauses(lower::AbstractConverter &converter,
                  mlir::omp::TargetExtOperands &clauseOps,
                  DefaultMapsTy &defaultMaps,
                  llvm::SmallVectorImpl<Object> &hasDeviceAddrObjects,
+                 llvm::SmallVectorImpl<Object> &inReductionObjects,
                  llvm::SmallVectorImpl<Object> &isDevicePtrObjects,
                  llvm::SmallVectorImpl<Object> &mapObjects) {
   ClauseProcessor cp(converter, semaCtx, clauses);
@@ -2489,12 +2490,33 @@ genTargetClauses(lower::AbstractConverter &converter,
     hostEvalInfo->collectValues(clauseOps.hostEvalVars);
   }
   cp.processIf(llvm::omp::Directive::OMPD_target, clauseOps);
+  cp.processInReduction(loc, clauseOps, inReductionObjects);
+  // The host lowering of `target in_reduction` redirects each list item's
+  // dedicated implicit map entry to the per-task reduction-private storage
+  // returned by the runtime. A COMMON block member is accessed in the target
+  // body through the COMMON block storage map, not through its own member map
+  // entry, so the redirect would rewrite an unused map argument and the body
+  // would accumulate into the mapped original rather than the private copy.
+  // An EQUIVALENCE-associated variable has the same problem: it shares storage
+  // with another variable that may be mapped separately (e.g. `map(tofrom: j)`
+  // with `equivalence(i, j)`), so the body updates the aliased storage while
+  // the redirected runtime-private copy stays dead.
+  // Reject both cases until the redirect can target the storage map.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      if (semantics::FindCommonBlockContaining(sym->GetUltimate()))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a COMMON block member");
+      if (semantics::FindEquivalenceSet(sym->GetUltimate()))
+        TODO(loc, "TARGET construct with IN_REDUCTION of an EQUIVALENCE "
+                  "storage-associated variable");
+    }
   cp.processIsDevicePtr(stmtCtx, clauseOps, isDevicePtrObjects);
   cp.processMap(loc, stmtCtx, clauseOps, llvm::omp::Directive::OMPD_unknown,
                 &mapObjects);
   cp.processNowait(clauseOps);
   cp.processThreadLimit(stmtCtx, clauseOps);
-  cp.processTODO<clause::Allocate, clause::InReduction, clause::UsesAllocators>(
+  cp.processTODO<clause::Allocate, clause::UsesAllocators>(
       loc, llvm::omp::Directive::OMPD_target);
 
   // `target private(..)` is only supported in delayed privatization mode.
@@ -3556,10 +3578,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   mlir::omp::TargetExtOperands clauseOps;
   DefaultMapsTy defaultMaps;
   llvm::SmallVector<Object> mapObjects, hasDeviceAddrObjects,
-      isDevicePtrObjects;
+      inReductionObjects, isDevicePtrObjects;
   genTargetClauses(converter, semaCtx, symTable, stmtCtx, eval, item->clauses,
                    loc, clauseOps, defaultMaps, hasDeviceAddrObjects,
-                   isDevicePtrObjects, mapObjects);
+                   inReductionObjects, isDevicePtrObjects, mapObjects);
 
   KernelTypeVisitor visitor(semaCtx, converter.getModuleOp());
   clauseOps.kernelType = mlir::omp::TargetExecModeAttr::get(
@@ -3607,6 +3629,26 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
                            /*isTargetPrivitization=*/true);
   dsp.processStep1(&clauseOps);
 
+  // The host lowering of `target in_reduction` requires each list item to be
+  // implicitly mapped as an address-preserving entry whose map argument the
+  // redirect rewrites to the runtime reduction-private storage. When the same
+  // variable is also privatized on the target (implicitly, as with a
+  // host-associated scalar that defaults to firstprivate, or explicitly with
+  // `firstprivate(i) in_reduction(+:i)`), the force-map is suppressed because
+  // the symbol is already in the privatization set, and the firstprivate copy
+  // region then has no mold value for it, aborting translation. Reject until
+  // in_reduction can be reconciled with target privatization.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      const semantics::Symbol &ultimate = sym->GetUltimate();
+      if (llvm::any_of(dsp.getAllSymbolsToPrivatize(),
+                       [&](const semantics::Symbol *priv) {
+                         return priv->GetUltimate() == ultimate;
+                       }))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a privatized variable");
+    }
+
   // Collect symbols that have dynamic substring accesses
   llvm::SmallPtrSet<const semantics::Symbol *, 8> symbolsWithDynamicSubstring;
   collectSymbolsWithDynamicSubstring(semaCtx, eval,
@@ -3617,7 +3659,16 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   // symbols used inside the region that do not have explicit data-environment
   // attribute clauses (neither data-sharing; e.g. `private`, nor `map`
   // clauses).
-  auto captureImplicitMap = [&](const semantics::Symbol &sym) {
+  //
+  // When `forceAddressPreserving` is set, the symbol is force-mapped as an
+  // address-preserving `capture(ByRef)` with implicit `tofrom` flags,
+  // bypassing the scalar default capture rules. This is used for `target
+  // in_reduction` list items, whose mapped pointer is passed as the `orig`
+  // argument of `__kmpc_task_reduction_get_th_data`; a ByCopy scalar capture
+  // would break the runtime lookup against the enclosing taskgroup's
+  // task_reduction descriptor.
+  auto captureImplicitMap = [&](const semantics::Symbol &sym,
+                                bool forceAddressPreserving = false) {
     // Structure component symbols don't have bindings, and can only be
     // explicitly mapped individually. If a member is captured implicitly
     // we map the entirety of the derived type when we find its symbol.
@@ -3626,12 +3677,14 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
     // if the symbol is part of an already mapped common block, do not make a
     // map for it.
-    if (const Fortran::semantics::Symbol *common =
-            Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
-      if (llvm::any_of(mapObjects, [=](const Object &object) {
-            return object.sym() == common;
-          }))
-        return;
+    if (!forceAddressPreserving) {
+      if (const Fortran::semantics::Symbol *common =
+              Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
+        if (llvm::any_of(mapObjects, [=](const Object &object) {
+              return object.sym() == common;
+            }))
+          return;
+    }
 
     // If we come across a symbol without a symbol address, we
     // return as we cannot process it, this is intended as a
@@ -3681,13 +3734,21 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
         eleType = refType.getElementType();
 
       std::pair<mlir::omp::ClauseMapFlags, mlir::omp::VariableCaptureKind>
-          mapFlagAndKind = getImplicitMapTypeAndKind(
-              firOpBuilder, converter, defaultMaps, eleType, loc, sym);
+          mapFlagAndKind;
+      if (forceAddressPreserving)
+        mapFlagAndKind = {mlir::omp::ClauseMapFlags::implicit |
+                              mlir::omp::ClauseMapFlags::to |
+                              mlir::omp::ClauseMapFlags::from,
+                          mlir::omp::VariableCaptureKind::ByRef};
+      else
+        mapFlagAndKind = getImplicitMapTypeAndKind(
+            firOpBuilder, converter, defaultMaps, eleType, loc, sym);
 
       mlir::FlatSymbolRefAttr mapperId;
       auto defaultmapBehaviour = getDefaultmapIfPresent(defaultMaps, eleType);
-      if (defaultmapBehaviour ==
-          clause::Defaultmap::ImplicitBehavior::Default) {
+      if (!forceAddressPreserving &&
+          defaultmapBehaviour ==
+              clause::Defaultmap::ImplicitBehavior::Default) {
         const semantics::DerivedTypeSpec *typeSpec =
             sym.GetType() ? sym.GetType()->AsDerived() : nullptr;
         if (typeSpec) {
@@ -3741,6 +3802,15 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
           Object{const_cast<semantics::Symbol *>(&sym), std::nullopt});
     }
   };
+  // OpenMP requires `in_reduction` list items on `target` to be implicitly
+  // data-mapped. Force-map them as address-preserving captures before the
+  // generic implicit-map walk so that walk treats the symbols as already
+  // mapped via `isDuplicateMappedSymbol` and does not downgrade them to
+  // ByCopy.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym())
+      captureImplicitMap(*sym, /*forceAddressPreserving=*/true);
+
   lower::pft::visitAllSymbols(eval, captureImplicitMap);
 
   auto targetOp = mlir::omp::TargetOp::create(firOpBuilder, loc, clauseOps);
@@ -3753,7 +3823,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   args.hasDeviceAddr.objects = hasDeviceAddrObjects;
   args.hasDeviceAddr.vars = hasDeviceAddrBaseValues;
   args.hostEvalVars = clauseOps.hostEvalVars;
-  // TODO: Add in_reduction syms and vars.
+  // `in_reduction` list items do not get their own entry block argument on
+  // `omp.target`; they are implicitly mapped (see the force-map above) and the
+  // target body accesses them through their `map_entries` block argument. The
+  // `in_reduction` operands remain on the op as host-side metadata.
   args.map.objects = mapObjects;
   args.map.vars = mapBaseValues;
   args.priv.objects = makeObjects(dsp.getDelayedPrivSymbols());
diff --git a/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90 b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
new file mode 100644
index 0000000000000..1898e6605f7f8
--- /dev/null
+++ b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
@@ -0,0 +1,50 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-version=50 -o - %s | FileCheck %s
+
+! End-to-end coverage: Flang lowers a taskgroup task_reduction enclosing a
+! target in_reduction all the way to LLVM IR. Verifies that the host path
+! emits __kmpc_taskred_init for the enclosing taskgroup and
+! __kmpc_task_reduction_get_th_data for the target's in_reduction lookup,
+! with load/store through the returned private pointer and no direct update
+! through the original shared pointer.
+
+subroutine target_in_reduction_e2e()
+  integer :: i
+  i = 0
+  !$omp taskgroup task_reduction(+:i)
+    !$omp target in_reduction(+:i)
+    i = i + 1
+    !$omp end target
+  !$omp end taskgroup
+end subroutine target_in_reduction_e2e
+
+! CHECK-LABEL: define void @target_in_reduction_e2e_()
+! The enclosing taskgroup emits __kmpc_taskred_init to register the
+! task_reduction descriptor.
+! CHECK:         call ptr @__kmpc_taskred_init(i32 %{{.+}}, i32 1, ptr %{{.+}})
+
+! The host stub calls the outlined target body passing the captured pointer.
+! CHECK:         call void @__omp_offloading_{{.*}}_target_in_reduction_e2e_{{.*}}(ptr %{{.+}}, ptr null)
+
+! Inside the outlined target body, the in_reduction private pointer is
+! obtained from the runtime using the captured original pointer with a NULL
+! descriptor (the runtime walks enclosing taskgroups). All loads and stores
+! go through the returned private pointer.
+! CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_in_reduction_e2e_
+! CHECK-SAME:    (ptr %[[ORIG:.+]], ptr %{{.+}})
+! CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+! CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[ORIG]])
+! CHECK:         %[[VAL:.+]] = load i32, ptr %[[PRIV]]
+! CHECK:         %[[SUM:.+]] = add i32 %[[VAL]], 1
+! CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+! The outlined body must not store directly through the captured original
+! pointer; all updates go through the runtime-returned private copy.
+! CHECK-NOT:     store i32 %{{.+}}, ptr %[[ORIG]]
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
new file mode 100644
index 0000000000000..0b0f3e2e53dea
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is a COMMON block member is accessed in
+! the target body through the COMMON block storage map, not through its own
+! dedicated member map entry. The host-side redirect only rewrites the
+! member map argument, so the runtime-private pointer would be dead and the body
+! would accumulate into the mapped original instead of the reduction-private
+! copy. Until the redirect can target the storage map this case is rejected.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a COMMON block member
+
+subroutine omp_target_in_reduction_common()
+  integer :: i, k
+  common /cb/ i, k
+  i = 0
+  !$omp target map(tofrom: /cb/) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_common
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
new file mode 100644
index 0000000000000..43d03dba0c594
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is EQUIVALENCE storage-associated
+! shares storage with another variable that may be mapped separately (here `j`).
+! The body updates the aliased storage through `j`'s map while the host-side
+! redirect rewrites only `i`'s map argument, leaving the runtime-private copy
+! dead. Reject until the redirect can account for storage association.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of an EQUIVALENCE storage-associated variable
+
+subroutine omp_target_in_reduction_equivalence()
+  integer :: i, j
+  equivalence(i, j)
+  i = 0
+  !$omp target map(tofrom: j) in_reduction(+:i)
+  j = j + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_equivalence
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
new file mode 100644
index 0000000000000..075fb1a9135be
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! An explicit `firstprivate(i)` on the same `target` as `in_reduction(+:i)`
+! privatizes the reduction variable. The item is then privatized instead of
+! address-preserving-mapped, so the host-side redirect has no dedicated map
+! argument to rewrite and the firstprivate copy region has no mold value, which
+! previously aborted translation. Reject until in_reduction can be reconciled
+! with target privatization.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a privatized variable
+
+subroutine omp_target_in_reduction_firstprivate()
+  integer :: i
+  i = 0
+  !$omp target firstprivate(i) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_firstprivate
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
deleted file mode 100644
index e5a9cffac5a11..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-
-!===============================================================================
-! `mergeable` clause
-!===============================================================================
-
-! CHECK: not yet implemented: Unhandled clause IN_REDUCTION in TARGET construct
-subroutine omp_target_inreduction()
-  integer i
-  i = 0
-  !$omp target in_reduction(+:i)
-  i = i + 1
-  !$omp end target
-end subroutine omp_target_inreduction
diff --git a/flang/test/Lower/OpenMP/target-inreduction-unused.f90 b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
new file mode 100644
index 0000000000000..f9dfa4973610c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
@@ -0,0 +1,27 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Per the OpenMP spec, an in_reduction list item on a target construct is
+! implicitly data-mapped. The lowering must not rely on the variable being
+! referenced inside the target body to discover that map: here `i` only
+! appears in the in_reduction clause and is never read or written inside
+! the region. Verify that an omp.map.info for `i` is still emitted and
+! flows into the omp.target's map_entries.
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction_unused()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reduction_unusedEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@{{[^ ]+}} %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %{{[^ ]+}} : !fir.ref<i32>)
+
+subroutine omp_target_in_reduction_unused()
+  interface
+    subroutine sub()
+    end subroutine
+  end interface
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  call sub()
+  !$omp end target
+end subroutine omp_target_in_reduction_unused
diff --git a/flang/test/Lower/OpenMP/target-inreduction.f90 b/flang/test/Lower/OpenMP/target-inreduction.f90
new file mode 100644
index 0000000000000..74f0f003aab4f
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction.f90
@@ -0,0 +1,30 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Verify that in_reduction on a target construct is lowered to an
+! omp.target with both an in_reduction clause and an implicit map_entries
+! entry for the same variable. The in_reduction clause does not define an
+! entry block argument: inside the target body the variable is accessed
+! through its map_entries block argument. The implicit map also captures the
+! original pointer into the target region so the MLIR -> LLVM IR translation
+! can pass it to __kmpc_task_reduction_get_th_data.
+
+!CHECK-LABEL: omp.declare_reduction
+!CHECK-SAME:  @[[RED_I32_NAME:.*]] : i32 init {
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reductionEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@[[RED_I32_NAME]] %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %[[MAPARG:[^ ]+]] : !fir.ref<i32>)
+!CHECK:         hlfir.declare %[[MAPARG]]
+!CHECK:         omp.terminator
+!CHECK:       }
+
+subroutine omp_target_in_reduction()
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 965ac358c259e..54d14602fbdbf 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivateCB.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivateCB Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivateCB);
+
   /// Return the insertion point used by the underlying IRBuilder.
   InsertPointTy getInsertionPoint() { return Builder.saveIP(); }
 
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..0dc704a87e4ed 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -1218,6 +1218,56 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivateCB) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item.
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Gtid = getOrCreateThreadID(Ident);
+
+  // The runtime entry point takes (and returns) a generic, default-address-
+  // space `ptr`. A NULL descriptor makes the runtime walk the enclosing
+  // taskgroups to find the matching task_reduction registration for the item.
+  Type *PtrTy = PointerType::getUnqual(M.getContext());
+  Value *NullDesc = ConstantPointerNull::get(PtrTy);
+  FunctionCallee GetThData =
+      getOrCreateRuntimeFunction(M, OMPRTL___kmpc_task_reduction_get_th_data);
+
+  for (unsigned Idx = 0; Idx < OrigPtrs.size(); ++Idx) {
+    // Normalize a non-default-address-space original pointer to the generic
+    // address space before the call.
+    Value *OrigPtr = OrigPtrs[Idx];
+    if (auto *OrigPtrTy = dyn_cast<PointerType>(OrigPtr->getType());
+        OrigPtrTy && OrigPtrTy->getAddressSpace() != 0)
+      OrigPtr = Builder.CreateAddrSpaceCast(OrigPtr, PtrTy);
+
+    Value *Priv = Builder.CreateCall(GetThData, {Gtid, NullDesc, OrigPtr},
+                                     "omp.inred.priv");
+
+    // Cast the returned private pointer back to the requested address space
+    // when it differs.
+    if (auto *ResPtrTy = dyn_cast<PointerType>(ResultPtrTys[Idx]);
+        ResPtrTy && ResPtrTy->getAddressSpace() != 0)
+      Priv = Builder.CreateAddrSpaceCast(Priv, ResultPtrTys[Idx]);
+
+    MapPrivateCB(Idx, Priv);
+  }
+  return Builder.saveIP();
+}
+
 OpenMPIRBuilder::InsertPointOrErrorTy
 OpenMPIRBuilder::createBarrier(const LocationDescription &Loc, Directive Kind,
                                bool ForceSimpleCall, bool CheckCancelFlag) {
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
index 54e46b74c5b64..cd1223dc1702c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
@@ -764,9 +764,17 @@ class OpenMP_InReductionClauseSkip<
     OptionalAttr<SymbolRefArrayAttr>:$in_reduction_syms
   );
 
-  // Description varies depending on the operation. Assembly format not defined
-  // because this clause must be processed together with the first region of the
-  // operation, as it defines entry block arguments.
+  // This assembly format should only be used by operations where `in_reduction`
+  // does not define entry block arguments (e.g. `omp.target`). Otherwise, it
+  // must be printed and parsed together with the corresponding region, because
+  // it defines entry block arguments.
+  let optAssemblyFormat = [{
+    `in_reduction` `(`
+      custom<InReductionClause>($in_reduction_vars, type($in_reduction_vars),
+                                $in_reduction_byref, $in_reduction_syms) `)`
+  }];
+
+  // Description varies depending on the operation.
 }
 
 def OpenMP_InReductionClause : OpenMP_InReductionClauseSkip<>;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 5b4ce7cc214bf..fd700e081fa01 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -966,7 +966,8 @@ def TaskOp : OpenMP_Op<"task", traits = [
     OutlineableOpenMPOpInterface
   ], clauses = [
     OpenMP_AffinityClause, OpenMP_AllocateClause, OpenMP_DependClause,
-    OpenMP_FinalClause, OpenMP_IfClause, OpenMP_InReductionClause,
+    OpenMP_FinalClause, OpenMP_IfClause,
+    OpenMP_InReductionClauseSkip<assemblyFormat = true>,
     OpenMP_MergeableClause, OpenMP_PriorityClause, OpenMP_PrivateClause,
     OpenMP_UntiedClause, OpenMP_DetachClause
   ], singleRegion = true> {
@@ -1012,9 +1013,10 @@ def TaskloopContextOp : OpenMP_Op<"taskloop.context", traits = [
     DeclareOpInterfaceMethods<OutlineableOpenMPOpInterface>
   ], clauses = [
     OpenMP_AllocateClause, OpenMP_FinalClause, OpenMP_GrainsizeClause,
-    OpenMP_IfClause, OpenMP_InReductionClause, OpenMP_MergeableClause,
-    OpenMP_NogroupClause, OpenMP_NumTasksClause, OpenMP_PriorityClause,
-    OpenMP_PrivateClause, OpenMP_ReductionClause, OpenMP_UntiedClause
+    OpenMP_IfClause, OpenMP_InReductionClauseSkip<assemblyFormat = true>,
+    OpenMP_MergeableClause, OpenMP_NogroupClause, OpenMP_NumTasksClause,
+    OpenMP_PriorityClause, OpenMP_PrivateClause, OpenMP_ReductionClause,
+    OpenMP_UntiedClause
   ], singleRegion = true> {
   let summary = "OutlinableOpenMPOpInterface wrapper for taskloop construct";
   let description = [{
@@ -1627,6 +1629,15 @@ def TargetOp : OpenMP_Op<"target", traits = [
   ];
 
   let extraClassDeclaration = [{
+    // Override BlockArgOpenMPOpInterface method because `in_reduction` list
+    // items on `omp.target` do not define entry block arguments. The reduction
+    // variable is accessed inside the target body through its matching
+    // `map_entries` block argument; the `in_reduction` operands are kept only
+    // as host-side metadata used to look up the per-task private storage.
+    unsigned numInReductionBlockArgs() {
+      return 0;
+    }
+
     mlir::Value getMappedValueForPrivateVar(unsigned privVarIdx) {
       std::optional<DenseI64ArrayAttr> privateMapIdices = getPrivateMapsAttr();
 
@@ -1651,8 +1662,7 @@ def TargetOp : OpenMP_Op<"target", traits = [
     " | `map_iterated` `(` $map_iterated `:` type($map_iterated) `)`)" # [{
       custom<TargetOpRegion>(
           $region, $has_device_addr_vars, type($has_device_addr_vars),
-          $host_eval_vars, type($host_eval_vars), $in_reduction_vars,
-          type($in_reduction_vars), $in_reduction_byref, $in_reduction_syms,
+          $host_eval_vars, type($host_eval_vars),
           $map_vars, type($map_vars), $private_vars, type($private_vars),
           $private_syms, $private_needs_barrier, $private_maps) attr-dict
     }];
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 41d3c547ca618..a7abbe26c745d 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1038,6 +1038,82 @@ static void printDynGroupprivateClause(OpAsmPrinter &printer, Operation *op,
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Parser and printer for in_reduction Clause
+//===----------------------------------------------------------------------===//
+
+/// Parses an `in_reduction` clause for an operation that does not give its
+/// list items entry block arguments (e.g. `omp.target`). The expected format is
+/// a comma-separated list of `[byref] @sym %var` followed by `: types`.
+static ParseResult parseInReductionClause(
+    OpAsmParser &parser,
+    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
+    SmallVectorImpl<Type> &inReductionTypes,
+    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms) {
+  SmallVector<SymbolRefAttr> symbolVec;
+  SmallVector<bool> isByRefVec;
+
+  if (parser.parseCommaSeparatedList([&]() {
+        isByRefVec.push_back(parser.parseOptionalKeyword("byref").succeeded());
+        if (parser.parseAttribute(symbolVec.emplace_back()) ||
+            parser.parseOperand(inReductionVars.emplace_back()))
+          return failure();
+        return success();
+      }))
+    return failure();
+
+  if (parser.parseColon())
+    return failure();
+
+  if (parser.parseCommaSeparatedList(
+          [&]() { return parser.parseType(inReductionTypes.emplace_back()); }))
+    return failure();
+
+  if (inReductionVars.size() != inReductionTypes.size())
+    return failure();
+
+  inReductionByref = makeDenseBoolArrayAttr(parser.getContext(), isByRefVec);
+  SmallVector<Attribute> symbolAttrs(symbolVec.begin(), symbolVec.end());
+  inReductionSyms = ArrayAttr::get(parser.getContext(), symbolAttrs);
+  return success();
+}
+
+/// Prints an `in_reduction` clause for an operation that does not give its list
+/// items entry block arguments (e.g. `omp.target`). Mirrors
+/// `parseInReductionClause`.
+static void printInReductionClause(OpAsmPrinter &p, Operation *op,
+                                   ValueRange inReductionVars,
+                                   TypeRange inReductionTypes,
+                                   DenseBoolArrayAttr inReductionByref,
+                                   ArrayAttr inReductionSyms) {
+  MLIRContext *ctx = op->getContext();
+
+  ArrayAttr syms = inReductionSyms;
+  if (!syms) {
+    SmallVector<Attribute> values(inReductionVars.size(), nullptr);
+    syms = ArrayAttr::get(ctx, values);
+  }
+
+  DenseBoolArrayAttr byref = inReductionByref;
+  if (!byref) {
+    SmallVector<bool> values(inReductionVars.size(), false);
+    byref = DenseBoolArrayAttr::get(ctx, values);
+  }
+
+  llvm::interleaveComma(
+      llvm::zip_equal(inReductionVars, syms.getValue(), byref.asArrayRef()), p,
+      [&p](auto t) {
+        auto [var, sym, isByRef] = t;
+        if (isByRef)
+          p << "byref ";
+        if (sym)
+          p << sym << " ";
+        p << var;
+      });
+  p << " : ";
+  llvm::interleaveComma(inReductionTypes, p);
+}
+
 //===----------------------------------------------------------------------===//
 // Parsers for operations including clauses that define entry block arguments.
 //===----------------------------------------------------------------------===//
@@ -1305,9 +1381,6 @@ static ParseResult parseTargetOpRegion(
     SmallVectorImpl<Type> &hasDeviceAddrTypes,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &hostEvalVars,
     SmallVectorImpl<Type> &hostEvalTypes,
-    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
-    SmallVectorImpl<Type> &inReductionTypes,
-    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &mapVars,
     SmallVectorImpl<Type> &mapTypes,
     llvm::SmallVectorImpl<OpAsmParser::UnresolvedOperand> &privateVars,
@@ -1316,8 +1389,6 @@ static ParseResult parseTargetOpRegion(
   AllRegionParseArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, &privateMaps);
@@ -1568,20 +1639,18 @@ static void printBlockArgRegion(OpAsmPrinter &p, Operation *op, Region &region,
 
 // These parseXyz functions correspond to the custom<Xyz> definitions
 // in the .td file(s).
-static void printTargetOpRegion(
-    OpAsmPrinter &p, Operation *op, Region &region,
-    ValueRange hasDeviceAddrVars, TypeRange hasDeviceAddrTypes,
-    ValueRange hostEvalVars, TypeRange hostEvalTypes,
-    ValueRange inReductionVars, TypeRange inReductionTypes,
-    DenseBoolArrayAttr inReductionByref, ArrayAttr inReductionSyms,
-    ValueRange mapVars, TypeRange mapTypes, ValueRange privateVars,
-    TypeRange privateTypes, ArrayAttr privateSyms, UnitAttr privateNeedsBarrier,
-    DenseI64ArrayAttr privateMaps) {
+static void printTargetOpRegion(OpAsmPrinter &p, Operation *op, Region &region,
+                                ValueRange hasDeviceAddrVars,
+                                TypeRange hasDeviceAddrTypes,
+                                ValueRange hostEvalVars,
+                                TypeRange hostEvalTypes, ValueRange mapVars,
+                                TypeRange mapTypes, ValueRange privateVars,
+                                TypeRange privateTypes, ArrayAttr privateSyms,
+                                UnitAttr privateNeedsBarrier,
+                                DenseI64ArrayAttr privateMaps) {
   AllRegionPrintArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, privateMaps);
@@ -2579,8 +2648,7 @@ LogicalResult TargetUpdateOp::verify() {
 void TargetOp::build(OpBuilder &builder, OperationState &state,
                      const TargetExtOperands &clauses) {
   MLIRContext *ctx = builder.getContext();
-  // TODO Store clauses in op: allocateVars, allocatorVars, inReductionVars,
-  // inReductionByref, inReductionSyms.
+  // TODO Store clauses in op: allocateVars, allocatorVars.
   TargetOp::build(
       builder, state, /*allocate_vars=*/{}, /*allocator_vars=*/{},
       makeArrayAttr(ctx, clauses.dependKinds), clauses.dependVars,
@@ -2588,9 +2656,10 @@ void TargetOp::build(OpBuilder &builder, OperationState &state,
       clauses.device, clauses.dynGroupprivateAccessGroup,
       clauses.dynGroupprivateFallback, clauses.dynGroupprivateSize,
       clauses.hasDeviceAddrVars, clauses.hostEvalVars, clauses.ifExpr,
-      /*in_reduction_vars=*/{}, /*in_reduction_byref=*/nullptr,
-      /*in_reduction_syms=*/nullptr, clauses.isDevicePtrVars, clauses.mapVars,
-      clauses.mapIterated, clauses.nowait, clauses.privateVars,
+      clauses.inReductionVars,
+      makeDenseBoolArrayAttr(ctx, clauses.inReductionByref),
+      makeArrayAttr(ctx, clauses.inReductionSyms), clauses.isDevicePtrVars,
+      clauses.mapVars, clauses.mapIterated, clauses.nowait, clauses.privateVars,
       makeArrayAttr(ctx, clauses.privateSyms), clauses.privateNeedsBarrier,
       clauses.threadLimitVars, /*private_maps=*/nullptr, clauses.kernelType);
 }
@@ -2631,6 +2700,20 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op. At this stage, exact identity can only be
+/// required for block arguments, which have no defining op. Flang emits
+/// `hlfir.declare` #0 for the `in_reduction` operand and #1 for the map
+/// `var_ptr`; these collapse to the same value after lowering, but that cannot
+/// be enforced here.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 LogicalResult TargetOp::verify() {
   if (getKernelType() == TargetExecMode::bare && !isCombined())
     return emitOpError() << "bare kernel requires 'omp.combined'";
@@ -2655,6 +2738,30 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host rewrites that map argument to
+  // the reduction-private storage. Require every `in_reduction` operand to be
+  // captured by at least one `map_entries` entry.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
+    }
+    if (!captured)
+      return emitOpError() << "in_reduction variable must be captured by a "
+                              "matching map_entries entry";
+  }
+
   return verifyPrivateVarsMapping(*this);
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b2d2ef6d8f770..61cbdf7b18d74 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -364,9 +364,40 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       op.emitWarning("hint clause discarded");
   };
   auto checkInReduction = [&todo](auto op, LogicalResult &result) {
-    if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
-        op.getInReductionSyms())
+    if (isa<omp::TargetOp, omp::TaskOp, omp::TaskloopContextOp>(
+            op.getOperation())) {
+      if (auto byrefAttr = op.getInReductionByref()) {
+        for (bool isByRef : *byrefAttr) {
+          if (isByRef) {
+            result = todo("in_reduction with byref modifier");
+            return;
+          }
+        }
+      }
+      if (isa<omp::TargetOp>(op.getOperation())) {
+        if (auto inReductionSyms = op.getInReductionSyms()) {
+          for (auto sym :
+               (*inReductionSyms).template getAsRange<SymbolRefAttr>()) {
+            auto decl =
+                SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                    op, sym);
+            assert(decl &&
+                   "symbol resolution should be guaranteed by the op verifier");
+            if (decl.getInitializerRegion().front().getNumArguments() != 1) {
+              result = todo("in_reduction with two-argument initializer");
+              return;
+            }
+            if (!decl.getCleanupRegion().empty()) {
+              result = todo("in_reduction with cleanup region");
+              return;
+            }
+          }
+        }
+      }
+    } else if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
+               op.getInReductionSyms()) {
       result = todo("in_reduction");
+    }
   };
   auto checkNowait = [&todo](auto op, LogicalResult &result) {
     if (op.getNowait())
@@ -423,14 +454,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
           return;
         }
   };
-  auto checkInReductionByref = [&todo](auto op, LogicalResult &result) {
-    if (auto byrefAttr = op.getInReductionByref())
-      for (bool isByRef : *byrefAttr)
-        if (isByRef) {
-          result = todo("in_reduction with byref modifier");
-          return;
-        }
-  };
   auto checkNumTeams = [&todo](auto op, LogicalResult &result) {
     if (op.hasNumTeamsMultiDim())
       result = todo("num_teams with multi-dimensional values");
@@ -482,7 +505,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
       })
       .Case([&](omp::TaskgroupOp op) {
         checkAllocate(op, result);
@@ -494,7 +517,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
         checkReduction(op, result);
         checkReductionByref(op, result);
       })
@@ -8493,6 +8516,55 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand.
+      // The verifier guarantees at least one match exists; more than one
+      // matching entry is a lowering ambiguity (the redirect cannot pick which
+      // map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (v != mapInfo.getVarPtr())
+          continue;
+        if (matchIdx)
+          return targetOp.emitError()
+                 << "in_reduction variable on omp.target has multiple matching "
+                    "map_entries entries; the redirect target is ambiguous";
+        matchIdx = idx;
+      }
+      // The verifier requires a capturing map entry for every in_reduction
+      // operand, so a match must exist here.
+      assert(matchIdx &&
+             "TargetOp verifier guarantees a matching map_entries entry for "
+             "each in_reduction variable");
+      inRedMapArgIdx.push_back(*matchIdx);
+      // The runtime `orig` pointer is the in_reduction operand itself, the
+      // reduction variable the enclosing taskgroup registered.
+      inRedOrigPtrs.push_back(moduleTranslation.lookupValue(v));
+    }
+  }
+
   // For some private variables, the MapsForPrivatizedVariablesPass
   // creates MapInfoOp instances. Go through the private variables and
   // the mapped variables so that during codegeneration we are able
@@ -8568,8 +8640,15 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
         attr.isStringAttribute())
       llvmOutlinedFn->addFnAttr(attr);
 
-    for (auto [arg, mapOp] : llvm::zip_equal(mapBlockArgs, mapVars)) {
-      auto mapInfoOp = cast<omp::MapInfoOp>(mapOp.getDefiningOp());
+    for (auto [idx, arg] : llvm::enumerate(mapBlockArgs)) {
+      // in_reduction list items on omp.target are accessed through their
+      // map_entries block argument, which is redirected below to the per-task
+      // reduction-private storage returned by the runtime. Skip the default
+      // host-value mapping for those block arguments so the write-once
+      // mapValue mapping is free to be set to the private pointer.
+      if (llvm::is_contained(inRedMapArgIdx, idx))
+        continue;
+      auto mapInfoOp = cast<omp::MapInfoOp>(mapVars[idx].getDefiningOp());
       llvm::Value *mapOpValue =
           moduleTranslation.lookupValue(mapInfoOp.getVarPtr());
       moduleTranslation.mapValue(arg, mapOpValue);
@@ -8605,6 +8684,37 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // Collect, per item, the type the private pointer must have (the map
+      // block argument's type), and, through the callback, rebind the map block
+      // argument that stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
+      SmallVector<llvm::Type *> inRedResultPtrTys;
+      inRedResultPtrTys.reserve(inRedMapArgIdx.size());
+      for (unsigned mapArgIdx : inRedMapArgIdx)
+        inRedResultPtrTys.push_back(
+            moduleTranslation.convertType(mapBlockArgs[mapArgIdx].getType()));
+
+      llvm::OpenMPIRBuilder::LocationDescription bodyLoc(builder);
+      llvm::OpenMPIRBuilder::InsertPointTy redIP =
+          ompBuilder->createTargetInReduction(
+              bodyLoc, inRedOrigPtrs, inRedResultPtrTys,
+              [&](unsigned idx, llvm::Value *priv) {
+                moduleTranslation.mapValue(mapBlockArgs[inRedMapArgIdx[idx]],
+                                           priv);
+              });
+      builder.restoreIP(redIP);
+    }
+
     LLVM::ModuleTranslation::SaveStack<OpenMPAllocStackFrame> frame(
         moduleTranslation, allocaIP, deallocBlocks);
     llvm::Expected<llvm::BasicBlock *> exitBlock = convertOmpOpRegions(
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
index 015e901126a76..8108a1af31e21 100644
--- a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -348,8 +348,12 @@ module attributes {omp.is_target_device = true} {
       omp.terminator
     }
 
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
-    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+    // The `in_reduction` list item on `omp.target` has no dedicated block
+    // argument; it is captured by a matching `map_entries` entry.
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 : !llvm.ptr) map_entries(%1 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
       omp.terminator
     }
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 2db7f953fa3bb..57270ec0694c3 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3255,6 +3255,91 @@ func.func @omp_target_depend(%data_var: memref<i32>) {
 
 // -----
 
+func.func @omp_target_in_reduction_unresolved(%ptr: !llvm.ptr) {
+  // expected-error @below {{op expected symbol reference @add_f32 to point to a reduction declaration}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_duplicate(%ptr: !llvm.ptr) {
+  // expected-error @below {{op accumulator variable used more than once}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr, @add_f32 %ptr : !llvm.ptr, !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg: i32):
+  %0 = arith.constant 0 : i32
+  omp.yield (%0 : i32)
+}
+combiner {
+^bb1(%arg0: i32, %arg1: i32):
+  %1 = arith.addi %arg0, %arg1 : i32
+  omp.yield (%1 : i32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> i32
+  llvm.atomicrmw add %arg2, %2 monotonic : !llvm.ptr, i32
+  omp.yield
+}
+
+func.func @omp_target_in_reduction_type_mismatch(%mem: memref<1xf32>) {
+  // expected-error @below {{op expected accumulator ('memref<1xf32>') to be the same type as reduction declaration ('!llvm.ptr')}}
+  omp.target kernel_type(generic) in_reduction(@add_i32 %mem : memref<1xf32>) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_missing_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_missing_map(%ptr: !llvm.ptr) {
+  // An in_reduction operand on omp.target has no dedicated block argument; it
+  // is accessed inside the body through a matching map_entries block argument.
+  // With no map_entries entry capturing it, the verifier must reject the op.
+  // expected-error @below {{in_reduction variable must be captured by a matching map_entries entry}}
+  omp.target kernel_type(generic) in_reduction(@add_missing_map_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
 func.func @omp_distribute_schedule(%chunk_size : i32, %lb : i32, %ub : i32, %step : i32) -> () {
   // expected-error @below {{op chunk size set without dist_schedule_static being present}}
   "omp.distribute"(%chunk_size) <{operandSegmentSizes = array<i32: 0, 0, 1, 0>}> ({
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
new file mode 100644
index 0000000000000..0e9611c27ddcc
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
@@ -0,0 +1,51 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// in_reduction on omp.target compiled for the target *device*. From the
+// device's perspective an in_reduction list item behaves as a regular mapped
+// variable: the host-only redirect to the per-task reduction-private storage
+// (the __kmpc_global_thread_num + __kmpc_task_reduction_get_th_data lookup) is
+// guarded by !isTargetDevice and is therefore not emitted. Verify the device
+// body accesses the mapped variable directly and that none of the host
+// runtime lookup is present.
+
+module attributes {dlti.dl_spec = #dlti.dl_spec<"dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  omp.declare_reduction @add_i32 : i32
+  init {
+  ^bb0(%arg0: i32):
+    %c0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%c0 : i32)
+  }
+  combiner {
+  ^bb0(%arg0: i32, %arg1: i32):
+    %s = llvm.add %arg0, %arg1 : i32
+    omp.yield(%s : i32)
+  }
+
+  llvm.func @target_inreduction_device(%x : !llvm.ptr) {
+    %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+      %v = llvm.load %marg : !llvm.ptr -> i32
+      %c1 = llvm.mlir.constant(1 : i32) : i32
+      %s = llvm.add %v, %c1 : i32
+      llvm.store %s, %marg : i32, !llvm.ptr
+      omp.terminator
+    }
+    llvm.return
+  }
+}
+
+// The device kernel receives the mapped pointer as a captured argument and
+// accesses the in_reduction variable directly through it.
+// CHECK-LABEL: define {{.*}}amdgpu_kernel void @__omp_offloading_{{.*}}_target_inreduction_device_
+// CHECK-SAME:    (ptr %[[ARG:.+]], ptr %{{.+}})
+// CHECK:         store ptr %[[ARG]], ptr %[[ALLOCA:.+]]
+// CHECK:         %[[MAPPED:.+]] = load ptr, ptr %[[ALLOCA]]
+// The host-only in_reduction redirect must not run on the device: there is no
+// gtid lookup, no __kmpc_task_reduction_get_th_data call and no private copy.
+// CHECK-NOT:     @__kmpc_global_thread_num
+// CHECK-NOT:     @__kmpc_task_reduction_get_th_data
+// CHECK-NOT:     omp.inred.priv
+// The body reads and writes the mapped variable in place.
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[MAPPED]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[MAPPED]]
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
new file mode 100644
index 0000000000000..c0df1a4e1d332
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Multiple in_reduction items on omp.target. Each item is captured into the
+// target region through its own map_entries entry and accessed inside the
+// body via the corresponding map_entries block argument. On the host, every
+// item performs an independent
+// __kmpc_task_reduction_get_th_data lookup using its own captured original
+// pointer, and the returned per-task private pointer is bound to that item's
+// map block argument. This test pins down the pairing so it cannot pass if the
+// two items were swapped or collapsed onto a single pointer.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_multi(%x : !llvm.ptr, %y : !llvm.ptr) {
+  %mx = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %my = omp.map.info var_ptr(%y : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x, @add_i32 %y : !llvm.ptr, !llvm.ptr)
+      map_entries(%mx -> %mxarg, %my -> %myarg : !llvm.ptr, !llvm.ptr) {
+    // First item (x): load, += 1, store back.
+    %vx = llvm.load %mxarg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %sx = llvm.add %vx, %c1 : i32
+    llvm.store %sx, %mxarg : i32, !llvm.ptr
+    // Second item (y): load, += 2, store back.
+    %vy = llvm.load %myarg : !llvm.ptr -> i32
+    %c2 = llvm.mlir.constant(2 : i32) : i32
+    %sy = llvm.add %vy, %c2 : i32
+    llvm.store %sy, %myarg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards both captured pointers into the outlined target
+// kernel (the trailing argument is the unused descriptor slot).
+// CHECK-LABEL: define void @target_inreduction_multi(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_multi_{{.*}}(ptr %{{.+}}, ptr %{{.+}}, ptr null)
+
+// The two captured original pointers arrive as distinct kernel arguments.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_multi_
+// CHECK-SAME:    (ptr %[[CAPTX:.+]], ptr %[[CAPTY:.+]], ptr %{{.+}})
+
+// A single gtid is computed once for the whole target body and shared by both
+// lookups; each item then performs its own __kmpc_task_reduction_get_th_data
+// call against its own captured pointer, and the returned per-task private
+// pointer is bound to that item's map block argument.
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIVX:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTX]])
+// CHECK:         %[[PRIVY:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTY]])
+
+// The first item's private storage is the base of the +1 load/store; the
+// CHECK-NOT below ensures the second item's pointer is not touched until the
+// first item's accumulation has completed (i.e. the items are not swapped or
+// merged onto a single private pointer).
+// CHECK:         %[[LX:.+]] = load i32, ptr %[[PRIVX]]
+// CHECK:         %[[SX:.+]] = add i32 %[[LX]], 1
+// CHECK-NOT:     %[[PRIVY]]
+// CHECK:         store i32 %[[SX]], ptr %[[PRIVX]]
+
+// The second item's private storage is the base of the +2 load/store.
+// CHECK:         %[[LY:.+]] = load i32, ptr %[[PRIVY]]
+// CHECK:         %[[SY:.+]] = add i32 %[[LY]], 2
+// CHECK:         store i32 %[[SY]], ptr %[[PRIVY]]
+
+// Exactly two reduction lookups are emitted; no third call sneaks in. The
+// `call` form is used so this does not match the runtime declaration.
+// CHECK-NOT:     call ptr @__kmpc_task_reduction_get_th_data
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
new file mode 100644
index 0000000000000..5e52ee00efafa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+// in_reduction on omp.target: the in_reduction variable is also captured
+// into the target region as a map entry (a frontend is expected to emit this
+// implicit map). The in_reduction clause does not define an entry block
+// argument; inside the target body the variable is accessed through its
+// map_entries block argument. The captured pointer is passed to
+// __kmpc_task_reduction_get_th_data with a NULL descriptor; the runtime
+// walks enclosing taskgroups to locate the matching task_reduction
+// registration. The returned per-task private pointer is bound to the
+// map_entries block argument so subsequent loads/stores inside the region
+// use the private copy.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+    %v = llvm.load %marg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards the captured pointer into the outlined target
+// kernel.
+// CHECK-LABEL: define void @target_inreduction(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_{{.*}}(ptr %{{.+}}, ptr null)
+
+// In the outlined target body the in_reduction private pointer is
+// obtained from the runtime using the captured original pointer; that
+// pointer is then the base of the load and store inside the region.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_
+// CHECK-SAME:    (ptr %[[CAPT:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPT]])
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[PRIV]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+// -----
+
+// Same as the first case but the in_reduction variable lives in a non-default
+// address space (addrspace(1)). __kmpc_task_reduction_get_th_data is declared
+// to take and return a generic (default-AS) `ptr`, so the host
+// lowering must (1) addrspacecast the captured addrspace(1) original pointer to
+// generic `ptr` before the runtime lookup, and (2) addrspacecast the returned
+// generic private pointer back to addrspace(1) so the body's load/store use the
+// private copy in the right address space. This pins down the address-space
+// handling so a regression that passed the addrspace(1) pointer straight into
+// the runtime call (a bad-signature crash) cannot pass.
+
+omp.declare_reduction @add_i32_as1 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_as1(%x : !llvm.ptr<1>) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr<1>, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr<1>
+  omp.target kernel_type(generic) in_reduction(@add_i32_as1 %x : !llvm.ptr<1>) map_entries(%m -> %marg : !llvm.ptr<1>) {
+    %v = llvm.load %marg : !llvm.ptr<1> -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr<1>
+    omp.terminator
+  }
+  llvm.return
+}
+
+// In the outlined target body the addrspace(1) original pointer is normalized
+// to generic `ptr` before the NULL-descriptor in_reduction lookup, and the
+// returned generic private pointer is cast back to addrspace(1) for the body's
+// load and store. The original captured pointer arrives as the addrspace(1)
+// kernel argument and is only used to derive the generic lookup pointer.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_as1_
+// CHECK-SAME:    (ptr addrspace(1) %[[CAPT_AS1:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID_AS1:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[ORIG_GEN:.+]] = addrspacecast ptr addrspace(1) %[[CAPT_AS1]] to ptr
+// CHECK:         %[[PRIV_AS1_GEN:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID_AS1]], ptr null, ptr %[[ORIG_GEN]])
+// CHECK:         %[[PRIV_AS1:.+]] = addrspacecast ptr %[[PRIV_AS1_GEN]] to ptr addrspace(1)
+// CHECK:         %[[LOADED_AS1:.+]] = load i32, ptr addrspace(1) %[[PRIV_AS1]]
+// CHECK:         %[[SUM_AS1:.+]] = add i32 %[[LOADED_AS1]], 1
+// CHECK:         store i32 %[[SUM_AS1]], ptr addrspace(1) %[[PRIV_AS1]]
+
+// The body must not load or store through the original captured addrspace(1)
+// pointer; all accesses go through the runtime-returned private copy.
+// CHECK-NOT:     store i32 %{{.+}}, ptr addrspace(1) %[[CAPT_AS1]]
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..943b5bad6970f 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -218,10 +218,95 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @target_in_reduction(%x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause in_reduction in omp.target operation}}
+llvm.func @target_in_reduction_byref(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with byref modifier in omp.target operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.target}}
-  omp.target kernel_type(generic) in_reduction(@add_f32 %x -> %prv : !llvm.ptr) {
+  omp.target kernel_type(generic) in_reduction(byref @add_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_cleanup_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+cleanup {
+^bb2(%arg2: f32):
+  omp.yield
+}
+llvm.func @target_in_reduction_cleanup(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with cleanup region in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_cleanup_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_two_arg_init_i32 : !llvm.ptr alloc {
+^bb0(%arg: !llvm.ptr):
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+  omp.yield(%1 : !llvm.ptr)
+} init {
+^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.store %0, %arg1 : i32, !llvm.ptr
+  omp.yield(%arg1 : !llvm.ptr)
+} combiner {
+^bb1(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.load %arg0 : !llvm.ptr -> i32
+  %1 = llvm.load %arg1 : !llvm.ptr -> i32
+  %2 = llvm.add %0, %1 : i32
+  llvm.store %2, %arg0 : i32, !llvm.ptr
+  omp.yield(%arg0 : !llvm.ptr)
+}
+llvm.func @target_in_reduction_two_arg_init(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with two-argument initializer in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_two_arg_init_i32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_dup_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb0(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+llvm.func @target_in_reduction_duplicate_map(%x : !llvm.ptr) {
+  // The in_reduction variable %x has two matching map_entries entries for the
+  // same var_ptr. The translation cannot disambiguate which map block argument
+  // to redirect, so it must reject this as ambiguous.
+  %m1 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %m2 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{in_reduction variable on omp.target has multiple matching map_entries entries; the redirect target is ambiguous}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_dup_map_f32 %x : !llvm.ptr) map_entries(%m1 -> %arg1, %m2 -> %arg2 : !llvm.ptr, !llvm.ptr) {
     omp.terminator
   }
   llvm.return


From llvmlistbot at llvm.org  Tue Jul 14 07:37:20 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 07:37:20 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a5649a0.6c8668f2.e4f82.1e23@mx.google.com>


================

----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 07:37:24 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 07:37:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a5649a4.6c5720b3.15e90c.0ffd@mx.google.com>


================

----------------
Saieiei wrote:

Done!

https://github.com/llvm/llvm-project/pull/199967

From llvmlistbot at llvm.org  Tue Jul 14 07:47:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 07:47:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a564c14.8e75e0cb.3dbce4.6b63@mx.google.com>


weimin023 wrote:

Hi @kvederni, I've rebased it. 

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Tue Jul 14 07:48:17 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Tue, 14 Jul 2026 07:48:17 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a564c31.4987c784.63cbc.2bf0@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 700359a9382d5a38d6b2790de9dfd84f91ef38a1 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 .../cl20-device-side-enqueue-attributes.cl    |  6 +-
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 llvm/test/CodeGen/SystemZ/zos-ppa2.ll         |  2 +-
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 30 ++++-----
 22 files changed, 135 insertions(+), 128 deletions(-)

diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..e21c39536db45 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -10,7 +10,7 @@
 typedef void (^bl_t)(local void *);
 typedef struct {int a;} ndrange_t;
 
-kernel void device_side_enqueue(global float *a, global float *b, int i) {
+kernel void device_side_enqueue(global float *a, global float *b, int i) strictfp {
   queue_t default_queue;
   unsigned flags = 0;
   ndrange_t ndrange;
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index f7e3ca8db80e1..0c913a254854c 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26572,9 +26574,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index fa0fe7a2e6092..2b4f0048cae4f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
index a4c1d50df5543..4e6de83d7b9c2 100644
--- a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
+++ b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
@@ -21,7 +21,7 @@
 ; CHECK:  DC XL2'0000'
 ; CHECK: L#DVS DS 0H
 ; CHECK:  DC XL14'F1F9F7F0F0F1F0F1F0F0F0F0F0F0'
-; CHECK:  DC XL6'F2F3F0F0F0F0'
+; CHECK:  DC XL6'F2F4F0F0F0F0'
 ; CHECK:  DC XL2'0000'
 
 ; CHECK: C_@@QPPA2 CATTR ALIGN(3),FILL(0),NOTEXECUTABLE,READONLY,RMODE(64),PART(.
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..4e111d2b459f9 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64


From llvmlistbot at llvm.org  Tue Jul 14 08:03:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 08:03:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 2a1200e - [Flang][OpenMP] Fix Fortran automap
	handling (#162501)
Message-ID: <6a564fb0.e5907290.281f16.f4b9@mx.google.com>


Author: Akash Banerjee
Date: 2026-07-14T16:03:05+01:00
New Revision: 2a1200e08cd1cb0e6744e38a7ec4611d983fb44f

URL: https://github.com/llvm/llvm-project/commit/2a1200e08cd1cb0e6744e38a7ec4611d983fb44f
DIFF: https://github.com/llvm/llvm-project/commit/2a1200e08cd1cb0e6744e38a7ec4611d983fb44f.diff

LOG: [Flang][OpenMP] Fix Fortran automap handling (#162501)

This fixes Fortran `declare target enter(automap:...)` handling for
allocatable
descriptors.

- Replace the previous `has_device_addr` target-region rewrite with
explicit
  target data operations around allocation/deallocation.
- On allocation, emit:
  - descriptor `map(always,to)` to keep the Fortran descriptor present
  - storage `map(storage)` for the allocated array data
- `map(attach)` to update the device descriptor base address to the
device data
- On deallocation, delete both the mapped storage and descriptor.
- Add a weak visible offload-entry alias for local/hidden `declare
target to/enter`
globals so libomptarget can associate the host descriptor with the real
device
  global without changing the original symbol linkage.

This fixes `offload/test/offloading/fortran/declare-target-automap.f90`
reported broken in #161265.

Co-authored-by: Codex <codex at openai.com>

Added: 
    flang/test/Lower/OpenMP/declare-target-automap.f90

Modified: 
    flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
    flang/test/Transforms/omp-automap-to-target-data.fir
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
    offload/test/offloading/fortran/declare-target-automap.f90

Removed: 
    


################################################################################
diff  --git a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
index eeb08ebf51191..bf269b5dd3ff2 100644
--- a/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
+++ b/flang/lib/Optimizer/OpenMP/AutomapToTargetData.cpp
@@ -15,11 +15,12 @@
 #include "flang/Optimizer/HLFIR/HLFIROps.h"
 
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
-#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/Pass/Pass.h"
 
+#include "llvm/Frontend/OpenMP/OMPConstants.h"
+
 namespace flangomp {
 #define GEN_PASS_DEF_AUTOMAPTOTARGETDATAPASS
 #include "flang/Optimizer/OpenMP/Passes.h.inc"
@@ -86,11 +87,7 @@ class AutomapToTargetDataPass
   }
 
   void runOnOperation() override {
-    ModuleOp module = getOperation()->getParentOfType<ModuleOp>();
-    if (!module)
-      module = dyn_cast<ModuleOp>(getOperation());
-    if (!module)
-      return;
+    ModuleOp module = getOperation();
 
     // Build FIR builder for helper utilities.
     fir::KindMapping kindMap = fir::getKindMapping(module);
@@ -113,16 +110,63 @@ class AutomapToTargetDataPass
       if (needsBoundsOps(memOp.getMemref()))
         genBoundsOps(builder, memOp.getMemref(), bounds);
 
+      mlir::Value boxValue;
+      if (auto storeOp = mlir::dyn_cast<fir::StoreOp>(memOp.getOperation()))
+        boxValue = storeOp.getValue();
+      else
+        boxValue = mlir::cast<fir::LoadOp>(memOp.getOperation()).getResult();
+
+      mlir::Value baseAddr =
+          fir::BoxAddrOp::create(builder, memOp.getLoc(), boxValue);
+      mlir::Value dataAddr = builder.createConvert(
+          memOp.getLoc(),
+          builder.getRefType(fir::unwrapRefType(baseAddr.getType())), baseAddr);
+      mlir::Type baseTy = fir::unwrapRefType(dataAddr.getType());
+      if (mlir::Type eleTy = fir::dyn_cast_ptrOrBoxEleTy(baseTy))
+        baseTy = eleTy;
+      if (auto seqTy = mlir::dyn_cast<fir::SequenceType>(baseTy))
+        if (seqTy.hasDynamicExtents())
+          baseTy = seqTy.getEleTy();
+
       omp::TargetEnterExitUpdateDataOperands clauses;
+      bool isAlloc = isa<fir::StoreOp>(memOp);
+
+      auto createDescriptorMap =
+          [&](mlir::omp::ClauseMapFlags mapType) -> mlir::omp::MapInfoOp {
+        mlir::Type descriptorTy =
+            fir::unwrapRefType(memOp.getMemref().getType());
+        // Keep the descriptor itself present. MapInfoFinalization expands this
+        // ref_ptr map and emits the attach map when the descriptor is created.
+        mapType |= omp::ClauseMapFlags::ref_ptr;
+        if (!isAlloc)
+          mapType |= omp::ClauseMapFlags::attach_never;
+        return mlir::omp::MapInfoOp::create(
+            builder, memOp.getLoc(), memOp.getMemref().getType(),
+            memOp.getMemref(), TypeAttr::get(descriptorTy),
+            builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
+            builder.getAttr<omp::VariableCaptureKindAttr>(
+                omp::VariableCaptureKind::ByRef),
+            /*var_ptr_ptr=*/mlir::Value{},
+            /*var_ptr_ptr_type=*/mlir::TypeAttr{},
+            /*members=*/SmallVector<Value>{},
+            /*members_index=*/ArrayAttr{},
+            /*bounds=*/SmallVector<Value>{},
+            /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
+            builder.getBoolAttr(false));
+      };
+
+      if (isAlloc)
+        clauses.mapVars.push_back(createDescriptorMap(
+            omp::ClauseMapFlags::to | omp::ClauseMapFlags::always));
+
+      mlir::omp::ClauseMapFlags mapType =
+          isAlloc ? omp::ClauseMapFlags::storage : omp::ClauseMapFlags::del;
       mlir::omp::MapInfoOp mapInfo = mlir::omp::MapInfoOp::create(
-          builder, memOp.getLoc(), memOp.getMemref().getType(),
-          memOp.getMemref(),
-          TypeAttr::get(fir::unwrapRefType(memOp.getMemref().getType())),
-          builder.getAttr<omp::ClauseMapFlagsAttr>(
-              isa<fir::StoreOp>(memOp) ? omp::ClauseMapFlags::to
-                                       : omp::ClauseMapFlags::del),
+          builder, memOp.getLoc(), dataAddr.getType(), dataAddr,
+          TypeAttr::get(baseTy),
+          builder.getAttr<omp::ClauseMapFlagsAttr>(mapType),
           builder.getAttr<omp::VariableCaptureKindAttr>(
-              omp::VariableCaptureKind::ByCopy),
+              omp::VariableCaptureKind::ByRef),
           /*var_ptr_ptr=*/mlir::Value{},
           /*var_ptr_ptr_type=*/mlir::TypeAttr{},
           /*members=*/SmallVector<Value>{},
@@ -130,9 +174,13 @@ class AutomapToTargetDataPass
           /*mapperId=*/mlir::FlatSymbolRefAttr(), globalOp.getSymNameAttr(),
           builder.getBoolAttr(false));
       clauses.mapVars.push_back(mapInfo);
-      isa<fir::StoreOp>(memOp)
-          ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
-          : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
+
+      if (!isAlloc)
+        clauses.mapVars.push_back(
+            createDescriptorMap(omp::ClauseMapFlags::del));
+
+      isAlloc ? omp::TargetEnterDataOp::create(builder, memOp.getLoc(), clauses)
+              : omp::TargetExitDataOp::create(builder, memOp.getLoc(), clauses);
     };
 
     for (fir::GlobalOp globalOp : automapGlobals) {

diff  --git a/flang/test/Lower/OpenMP/declare-target-automap.f90 b/flang/test/Lower/OpenMP/declare-target-automap.f90
new file mode 100644
index 0000000000000..63014cbf36725
--- /dev/null
+++ b/flang/test/Lower/OpenMP/declare-target-automap.f90
@@ -0,0 +1,28 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=60 %s -o - | FileCheck %s
+
+program automap_program
+   integer, allocatable, target :: automap_array(:)
+   !$omp declare target enter(automap:automap_array)
+
+   allocate (automap_array(10))
+
+   !$omp target
+      automap_array(1) = 1
+   !$omp end target
+
+   deallocate (automap_array)
+end program
+
+! CHECK-LABEL: func.func @_QQmain()
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[DESC:.*]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_BASE:.*]] = fir.box_offset %[[DESC]] base_addr : (!fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>) -> !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>
+! CHECK: %[[ATTACH_MAP:.*]] = omp.map.info var_ptr(%[[DESC]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(attach, ref_ptr) capture(ByRef) var_ptr_ptr(%[[DESC_BASE]] : !fir.llvm_ptr<!fir.ref<!fir.array<?xi32>>>, i32) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
+! CHECK: %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]], %[[ATTACH_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
+! CHECK-NOT: has_device_addr
+! CHECK: omp.target {{.*}}map_entries(
+! CHECK-NOT: has_device_addr
+! CHECK: %[[DELETE_MAP:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef) bounds(%{{.*}}) -> !fir.ref<!fir.array<?xi32>> {name = "_QFEautomap_array"}
+! CHECK: %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%{{.*}} : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QFEautomap_array"}
+! CHECK: omp.target_exit_data map_entries(%[[DELETE_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)

diff  --git a/flang/test/Transforms/omp-automap-to-target-data.fir b/flang/test/Transforms/omp-automap-to-target-data.fir
index 7a19705a248b4..37187203369a4 100644
--- a/flang/test/Transforms/omp-automap-to-target-data.fir
+++ b/flang/test/Transforms/omp-automap-to-target-data.fir
@@ -38,21 +38,25 @@ module {
 // CHECK-NEXT:      fir.shape
 // CHECK-NEXT:      %[[ARR_BOXED:.*]] = fir.embox %[[ALLOC_MEM]]
 // CHECK-NEXT:      fir.store %[[ARR_BOXED]]
-// CHECK-NEXT:      %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK-NEXT:      %[[ARR_HEAP_PTR:.*]] = fir.box_addr %[[ARR_BOXED_LOADED]]
-// CHECK-NEXT:      %[[DIM0:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
-// CHECK-NEXT:      %[[ONE:.*]] = arith.constant 1 : index
-// CHECK-NEXT:      %[[ZERO:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
-// CHECK-NEXT:      %[[LOWER_BOUND:.*]] = arith.constant 0 : index
-// CHECK-NEXT:      %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
-// CHECK-NEXT:      omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
-// CHECK-NEXT:      arith.muli %[[BOX_DIMS2]]#2, %[[BOX_DIMS2]]#1 : index
-// CHECK-NEXT:      %[[MAP_INFO:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(to) capture(ByCopy)
-// CHECK-NEXT:      omp.target_enter_data map_entries(%[[MAP_INFO]]
+// CHECK:           %[[ARR_BOXED_LOADED:.*]] = fir.load %[[AUTOMAP_DECL]]#0
+// CHECK:           %[[DIM0:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[DIM0]]
+// CHECK:           %[[ONE:.*]] = arith.constant 1 : index
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : index
+// CHECK:           %[[BOX_DIMS2:.*]]:3 = fir.box_dims %[[ARR_BOXED_LOADED]], %[[ZERO]]
+// CHECK:           %[[LOWER_BOUND:.*]] = arith.constant 0 : index
+// CHECK:           %[[UPPER_BOUND:.*]] = arith.subi %[[BOX_DIMS2]]#1, %[[ONE]] : index
+// CHECK:           %[[BOUNDS:.*]] = omp.map.bounds lower_bound(%[[LOWER_BOUND]] : index) upper_bound(%[[UPPER_BOUND]] : index) extent(%[[BOX_DIMS2]]#1 : index) stride(%[[BOX_DIMS2]]#2 : index) start_idx(%[[BOX_DIMS]]#0 : index) {stride_in_bytes = true}
+// CHECK:           %[[DATA_ADDR:.*]] = fir.box_addr %[[ARR_BOXED]]
+// CHECK-NEXT:      %[[DATA_REF:.*]] = fir.convert %[[DATA_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[DESC_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(always, to, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      %[[STORAGE_MAP:.*]] = omp.map.info var_ptr(%[[DATA_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(storage) capture(ByRef) bounds(%[[BOUNDS]]) -> !fir.ref<!fir.array<?xi32>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      omp.target_enter_data map_entries(%[[DESC_MAP]], %[[STORAGE_MAP]] : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.ref<!fir.array<?xi32>>)
 // CHECK:           %[[LOAD:.*]] = fir.load %[[AUTOMAP_DECL]]#0
-// CHECK:           %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 {{.*}} map_clauses(delete) capture(ByCopy)
-// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]]
+// CHECK:           %[[EXIT_ADDR:.*]] = fir.box_addr %[[LOAD]]
+// CHECK-NEXT:      %[[EXIT_REF:.*]] = fir.convert %[[EXIT_ADDR]] : (!fir.heap<!fir.array<?xi32>>) -> !fir.ref<!fir.array<?xi32>>
+// CHECK-NEXT:      %[[EXIT_MAP:.*]] = omp.map.info var_ptr(%[[EXIT_REF]] : !fir.ref<!fir.array<?xi32>>, i32) map_clauses(delete) capture(ByRef)
+// CHECK-NEXT:      %[[DESC_DELETE:.*]] = omp.map.info var_ptr(%[[AUTOMAP_DECL]]#0 : !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>, !fir.box<!fir.heap<!fir.array<?xi32>>>) map_clauses(delete, attach_never, ref_ptr) capture(ByRef) -> !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>> {name = "_QMtestEarr"}
+// CHECK-NEXT:      omp.target_exit_data map_entries(%[[EXIT_MAP]], %[[DESC_DELETE]] : !fir.ref<!fir.array<?xi32>>, !fir.ref<!fir.box<!fir.heap<!fir.array<?xi32>>>>)
 // CHECK-NEXT:      %[[BOXADDR:.*]] = fir.box_addr %[[LOAD]]
 // CHECK-NEXT:      fir.freemem %[[BOXADDR]]

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 7bb15b74aeac3..af1ec17563806 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -8922,6 +8922,54 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
       auto captureClauseKind = convertToCaptureClauseKind(captureClause);
       auto deviceClause =
           convertToDeviceClauseKind(attribute.getDeviceType().getValue());
+      llvm::StringRef entryMangledName = mangledName;
+      llvm::Constant *entryAddr = llvm::cast<llvm::Constant>(gVal);
+      std::function<llvm::GlobalValue::LinkageTypes()> variableLinkage;
+      llvm::SmallString<128> entryNameStorage;
+      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
+      bool isToOrEnter =
+          captureClause == omp::DeclareTargetCaptureClause::to ||
+          captureClause == omp::DeclareTargetCaptureClause::enter;
+      bool isHostOnly = attribute.getDeviceType().getValue() ==
+                        omp::DeclareTargetDeviceType::host;
+
+      // A to/enter declare-target variable needs a device-resident,
+      // name-resolvable copy and a host offloading entry. A local-linkage
+      // global provides neither, so we promote it to external.
+      if (isToOrEnter && !isHostOnly && !requiresUSM &&
+          gVar->hasLocalLinkage()) {
+        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
+        isExternallyVisible = true;
+
+        // Clear the stale dso_local flag so it is referenced like a
+        // module-scope declare target global.
+        if (ompBuilder->Config.isTargetDevice())
+          gVar->setDSOLocal(false);
+      }
+
+      if (isToOrEnter &&
+          deviceClause ==
+              llvm::OffloadEntriesInfoManager::OMPTargetDeviceClauseAny &&
+          !requiresUSM && !isDeclaration &&
+          (gVal->hasLocalLinkage() || gVal->hasHiddenVisibility())) {
+        // Keep the original symbol as-is for target code, but create a visible
+        // alias for the offload entry so libomptarget can associate the host
+        // global with the actual device global.
+        entryNameStorage = (mangledName + llvm::Twine("_decl_tgt_entry")).str();
+        entryMangledName = entryNameStorage;
+        if (llvm::GlobalValue *existing =
+                llvmModule->getNamedValue(entryMangledName)) {
+          entryAddr = llvm::cast<llvm::Constant>(existing);
+        } else {
+          entryAddr = llvm::GlobalAlias::create(
+              gVal->getValueType(), gVal->getAddressSpace(),
+              llvm::GlobalValue::WeakAnyLinkage, entryMangledName, entryAddr,
+              llvmModule);
+          llvm::cast<llvm::GlobalAlias>(entryAddr)->setVisibility(
+              llvm::GlobalValue::DefaultVisibility);
+        }
+        variableLinkage = [] { return llvm::GlobalValue::WeakAnyLinkage; };
+      }
       // unused for MLIR at the moment, required in Clang for book
       // keeping
       std::vector<llvm::GlobalVariable *> generatedRefs;
@@ -8946,34 +8994,13 @@ convertDeclareTargetAttr(Operation *op, mlir::omp::DeclareTargetAttr attribute,
                                                      lineNo);
       };
 
-      bool requiresUSM = ompBuilder->Config.hasRequiresUnifiedSharedMemory();
-      bool isToOrEnter =
-          captureClause == omp::DeclareTargetCaptureClause::to ||
-          captureClause == omp::DeclareTargetCaptureClause::enter;
-      bool isHostOnly = attribute.getDeviceType().getValue() ==
-                        omp::DeclareTargetDeviceType::host;
-
-      // A to/enter declare-target variable needs a device-resident,
-      // name-resolvable copy and a host offloading entry. A local-linkage
-      // global provides neither, so we promote it to external.
-      if (isToOrEnter && !isHostOnly && !requiresUSM &&
-          gVar->hasLocalLinkage()) {
-        gVar->setLinkage(llvm::GlobalValue::ExternalLinkage);
-        isExternallyVisible = true;
-
-        // Clear the stale dso_local flag so it is referenced like a
-        // module-scope declare target global.
-        if (ompBuilder->Config.isTargetDevice())
-          gVar->setDSOLocal(false);
-      }
-
       llvm::vfs::FileSystem &vfs = moduleTranslation.getFileSystem();
       ompBuilder->registerTargetGlobalVariable(
           captureClauseKind, deviceClause, isDeclaration, isExternallyVisible,
           ompBuilder->getTargetEntryUniqueInfo(fileInfoCallBack, vfs),
-          mangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
-          /*GlobalInitializer*/ nullptr, /*VariableLinkage*/ nullptr,
-          gVal->getType(), gVal);
+          entryMangledName, generatedRefs, /*OpenMPSimd*/ false, targetTriple,
+          /*GlobalInitializer*/ nullptr, variableLinkage, gVal->getType(),
+          entryAddr);
 
       if (ompBuilder->Config.isTargetDevice() &&
           (captureClause == omp::DeclareTargetCaptureClause::link ||

diff  --git a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
index 92c85738dbc72..0d795eefa1f9d 100644
--- a/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
+++ b/mlir/test/Target/LLVMIR/omptarget-declare-target-llvm-host.mlir
@@ -133,6 +133,25 @@ module attributes {llvm.target_triple = "x86_64-unknown-linux-gnu", omp.is_targe
     llvm.return %0 : i32
   }
 
+  // CHECK-DAG: @_QFEinternal_enter = dso_local global i32 7
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [19 x i8] c"_QFEinternal_enter\00"
+  // CHECK-DAG: @.offloading.entry._QFEinternal_enter = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEinternal_enter, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEinternal_enter", i32 {{.*}}, i32 {{.*}}}
+  llvm.mlir.global internal @_QFEinternal_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
+    %0 = llvm.mlir.constant(7 : i32) : i32
+    llvm.return %0 : i32
+  }
+
+  // CHECK-DAG: @_QFEhidden_enter = hidden global i32 8
+  // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [32 x i8] c"_QFEhidden_enter_decl_tgt_entry\00"
+  // CHECK-DAG: @.offloading.entry._QFEhidden_enter_decl_tgt_entry = weak constant %struct.__tgt_offload_entry { i64 0, i16 1, i16 1, i32 0, ptr @_QFEhidden_enter_decl_tgt_entry, ptr @.offloading.entry_name{{.*}}, i64 4, i64 0, ptr null }, section "llvm_offload_entries"
+  // CHECK-DAG: @_QFEhidden_enter_decl_tgt_entry = weak alias i32, ptr @_QFEhidden_enter
+  // CHECK-DAG: !{{.*}} = !{i32 {{.*}}, !"_QFEhidden_enter_decl_tgt_entry", i32 {{.*}}, i32 {{.*}}}
+  llvm.mlir.global external hidden @_QFEhidden_enter() {addr_space = 0 : i32, omp.declare_target = #omp.declaretarget<device_type = (any), capture_clause = (enter)>} : i32 {
+    %0 = llvm.mlir.constant(8 : i32) : i32
+    llvm.return %0 : i32
+  }
+
   // CHECK-DAG: @_QMtest_0Ept1 = global { ptr, i64, i32, i8, i8, i8, i8 } { ptr null, i64 4, i32 20180515, i8 0, i8 9, i8 1, i8 0 }
   // CHECK-DAG: @_QMtest_0Ept1_decl_tgt_ref_ptr = weak global ptr @_QMtest_0Ept1
   // CHECK-DAG: @.offloading.entry_name{{.*}} = internal unnamed_addr constant [31 x i8] c"_QMtest_0Ept1_decl_tgt_ref_ptr\00"

diff  --git a/offload/test/offloading/fortran/declare-target-automap.f90 b/offload/test/offloading/fortran/declare-target-automap.f90
index b44c0b2815274..b9c2d34c834fa 100644
--- a/offload/test/offloading/fortran/declare-target-automap.f90
+++ b/offload/test/offloading/fortran/declare-target-automap.f90
@@ -1,9 +1,6 @@
 !Offloading test for AUTOMAP modifier in declare target enter
 ! REQUIRES: flang, amdgpu
 
-! FIXME: https://github.com/llvm/llvm-project/issues/161265
-! XFAIL: amdgpu
-
 ! RUN: %libomptarget-compile-fortran-run-and-check-generic
 program automap_program
    use iso_c_binding, only: c_loc


        

From llvmlistbot at llvm.org  Tue Jul 14 08:03:18 2026
From: llvmlistbot at llvm.org (Akash Banerjee)
Date: Tue, 14 Jul 2026 08:03:18 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Fix Fortran
 automap handling (PR #162501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/162501@github.com>
Message-ID: <6a564fb6.eb165e5c.1817a5.f283@mx.google.com>


https://github.com/TIFitis closed https://github.com/llvm/llvm-project/pull/162501

From llvmlistbot at llvm.org  Tue Jul 14 08:47:35 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 08:47:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Refine pack/unpack
 simplification checks (NFC) (PR #209522)
Message-ID: <llvm.org/llvm/llvm-project/pull/209522@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/209522

Update `isPackOn1D`, which is used by both
`SimplifyPackToExpandShape` and `SimplifyUnPackToCollapseShape`:

* Rename it to `isPackOnEffectively1D` to better reflect its
  functionality: the underlying pack can be multi-dimensional.
* Add checks to ensure that the unique non-unit inner tile is used to tile
  the unique non-unit unpacked dimension (that was previously left as an
  unchecked assumption).
* Add comments to the test file for these patterns, grouping the tests
  according to the functionality/cases being tested.


>From 72293635a5bfdd02eb5903ea7b152401b8378ecb Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Tue, 14 Jul 2026 13:51:42 +0100
Subject: [PATCH] [mlir][linalg] Refine pack/unpack simplification checks (NFC)

Update `isPackOn1D`, which is used by both
`SimplifyPackToExpandShape` and `SimplifyUnPackToCollapseShape`:

* Rename it to `isPackOnEffectively1D` to better reflect its
  functionality: the underlying pack can be multi-dimensional.
* Add checks to ensure that the unique non-unit inner tile is used to tile
  the unique non-unit unpacked dimension (that was previously left as an
  unchecked assumption).
* Add comments to the test file for these patterns, grouping the tests
  according to the functionality/cases being tested.
---
 .../Transforms/PackAndUnpackPatterns.cpp      | 77 ++++++++++++++-----
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 58 +++++++++++---
 2 files changed, 105 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp b/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
index 993eae62535c3..c8e9f021938e5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
@@ -24,24 +24,67 @@ static int64_t getNumGtOneDims(ArrayRef<int64_t> shape) {
       shape, [](int64_t v) { return ShapedType::isDynamic(v) || v > 1; });
 }
 
-/// Returns success() if there is only 1 dimension size in non-packed domain
-/// being greater than 1 and packing only happens on the dimension.
-/// Note: this method should only be used by pack/unpack to reshape conversion.
-/// It assumes that non-unit inner tile size must be used by the non-unit
-/// dimension.
-static LogicalResult isPackOn1D(RewriterBase &rewriter, Operation *op,
-                                ArrayRef<int64_t> srcShape,
-                                ArrayRef<int64_t> innerPackTileSize) {
-  if (getNumGtOneDims(srcShape) > 1) {
+/// Returns the index of the first non-unit size in `sizes`. Returns -1 if
+/// there are no non-unit sizes.
+static int64_t getFirstNonUnitSizeIdx(ArrayRef<int64_t> sizes) {
+  const auto *it = llvm::find_if(sizes, [](int64_t dim) { return dim != 1; });
+  return (it != sizes.end()) ? std::distance(sizes.begin(), it) : -1;
+}
+
+/// Check whether `op` is effectively a 1D pack/unpack. Example:
+///
+///  %pack = linalg.pack %src
+///     inner_dims_pos = [0, 1]
+///     inner_tiles = [1, 2] into %dest
+///   : tensor<1x32xf32> -> tensor<1x16x1x2xf32>
+///
+/// Returns success() if there is:
+///   * only 1 non-unit dim in the un-packed domain,
+///   * only 1 non-unit inner tile size, and
+///   * the unique non-unit tile size is applied to the unique non-unit
+///     un-packed dim.
+template <typename PackOrUnpackOp>
+static LogicalResult isPackOnEffectively1D(RewriterBase &rewriter,
+                                           PackOrUnpackOp *op) {
+  // Obtain the unpacked shape.
+  auto pack = dyn_cast<linalg::PackOp>(op);
+  auto unpack = dyn_cast<linalg::UnPackOp>(op);
+
+  ArrayRef<int64_t> unpackedShape = pack ? pack->getSourceType().getShape()
+                                         : unpack->getDestType().getShape();
+
+  // Obtain the inner tile sizes.
+  ArrayRef<int64_t> innerTileSizes = op->getStaticInnerTiles();
+
+  // Make sure that there is exactly single non-unit unpacked dim.
+  if (getNumGtOneDims(unpackedShape) != 1) {
     return rewriter.notifyMatchFailure(
-        op, "expects non-packed domain to have at most one non-unit dims");
+        *op, "expects non-packed domain to have at most one non-unit dims");
   }
-  // Non-unit inner tile size must be used by the non-unit dimension. If not, it
-  // will faill on getting reassociation maps.
-  if (getNumGtOneDims(innerPackTileSize) > 1) {
+
+  // Make sure that there is at most one non-unit inner tile size.
+  auto numNonUnitInnerTiles = getNumGtOneDims(innerTileSizes);
+  if (numNonUnitInnerTiles > 1) {
     return rewriter.notifyMatchFailure(
-        op, "expects at most one non-unit inner tiles");
+        *op, "expects at most one non-unit inner tiles");
   }
+
+  // If there are no non-unit tiles, there is nothing else to check.
+  if (numNonUnitInnerTiles == 0)
+    return success();
+
+  // Get the index of the unique non-unit unpacked dim.
+  int64_t nonUnitDimIdx = getFirstNonUnitSizeIdx(unpackedShape);
+
+  // Get the index of the dim that the unique non-unit tile is applied to.
+  int64_t nonUnitTileDestDimIdx = getFirstNonUnitSizeIdx(innerTileSizes);
+
+  // Make sure that the unique non-unit tile is applied to the unique unit dim.
+  if (nonUnitTileDestDimIdx != nonUnitDimIdx) {
+    return rewriter.notifyMatchFailure(
+        *op, "expects at most one non-unit inner tiles");
+  }
+
   return success();
 }
 
@@ -117,8 +160,7 @@ struct SimplifyPackToExpandShape : public OpRewritePattern<PackOp> {
 
     ShapedType sourceType = packOp.getSourceType();
     if (failed(isPackOnInnerMostDim(rewriter, packOp)) &&
-        failed(isPackOn1D(rewriter, packOp, sourceType.getShape(),
-                          packOp.getStaticTiles())) &&
+        failed(isPackOnEffectively1D(rewriter, &packOp)) &&
         !packOp.isLikePad()) {
       return failure();
     }
@@ -183,8 +225,7 @@ struct SimplifyUnPackToCollapseShape : public OpRewritePattern<UnPackOp> {
 
     ShapedType destType = unpackOp.getDestType();
     if (failed(isUnpackOnInnerMostDim(rewriter, unpackOp)) &&
-        failed(isPackOn1D(rewriter, unpackOp, destType.getShape(),
-                          unpackOp.getStaticTiles())) &&
+        failed(isPackOnEffectively1D(rewriter, &unpackOp)) &&
         !unpackOp.isLikeUnPad()) {
       return failure();
     }
diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6979770154bab..f4f3cb233d45e 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -1,5 +1,9 @@
 // RUN: mlir-opt -split-input-file -test-linalg-transform-patterns="test-simplify-pack-unpack-patterns" %s | FileCheck %s
 
+//===========================================================================//
+// Packing: 1D unpacked source
+//===========================================================================//
+
 // CHECK-LABEL: func.func @single_dim_packing(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<256xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]] output_shape [8, 32] : tensor<256xf32> into tensor<8x32xf32>
@@ -25,18 +29,6 @@ func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x3
 
 // -----
 
-// CHECK-LABEL: func.func @single_last_inner_dim_packing(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
-// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
-// CHECK:         return %[[EXPANDED]] : tensor<5x8x32xf32>
-func.func @single_last_inner_dim_packing(%arg0: tensor<5x256xf32>) -> tensor<5x8x32xf32> {
-  %empty = tensor.empty() : tensor<5x8x32xf32>
-  %0 = linalg.pack %arg0 inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<5x8x32xf32>
-  return %0 : tensor<5x8x32xf32>
-}
-
-// -----
-
 // CHECK-LABEL: func.func @pack_1d_with_outer_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<64xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]] output_shape [2, 32] : tensor<64xf32> into tensor<2x32xf32>
@@ -49,6 +41,22 @@ func.func @pack_1d_with_outer_dims_perm(%arg0: tensor<64xf32>) -> tensor<2x32xf3
 
 // -----
 
+//===========================================================================//
+// Packing: 2D unpacked source
+//===========================================================================//
+
+// CHECK-LABEL: func.func @single_last_inner_dim_packing(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
+// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
+// CHECK:         return %[[EXPANDED]] : tensor<5x8x32xf32>
+func.func @single_last_inner_dim_packing(%arg0: tensor<5x256xf32>) -> tensor<5x8x32xf32> {
+  %empty = tensor.empty() : tensor<5x8x32xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<5x8x32xf32>
+  return %0 : tensor<5x8x32xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
@@ -83,6 +91,10 @@ func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x
 
 // -----
 
+//===========================================================================//
+// Packing: Multi-dim unpacked source
+//===========================================================================//
+
 // CHECK-LABEL: func.func @pack_1x32_to_1x32x1x1
 // CHECK-SAME:    %[[ARG0:[0-9a-zA-Z]+]]
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2, 3]] output_shape [1, 32, 1, 1]
@@ -134,6 +146,10 @@ func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf3
 
 // -----
 
+//===========================================================================//
+// Un-Packing: 1D unpacked dest
+//===========================================================================//
+
 // CHECK-LABEL: func.func @unpack_1d_to_collapse
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<8x32xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]] : tensor<8x32xf32> into tensor<256xf32>
@@ -172,6 +188,10 @@ func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
 
 // -----
 
+//===========================================================================//
+// Un-Packing: 2D unpacked dest
+//===========================================================================//
+
 // CHECK-LABEL: func.func @single_last_inner_dim_unpacking(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<5x8x32xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0], [1, 2]] : tensor<5x8x32xf32> into tensor<5x256xf32>
@@ -269,6 +289,13 @@ func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1x
 
 // -----
 
+//===========================================================================//
+// Packing: Pad-like pack
+//
+// FIXME: With no single element added, there is no padding here. Improve
+// naming.
+//===========================================================================//
+
 // CHECK-LABEL: func.func @pad_like_pack(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<32x64xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1, 2], [3]] output_shape [1, 1, 32, 64] : tensor<32x64xf32> into tensor<1x1x32x64xf32>
@@ -333,6 +360,13 @@ func.func @pad_like_pack_with_transpose(%arg0: tensor<32x64x16xf32>) -> tensor<3
 
 // -----
 
+//===========================================================================//
+// Un-Packing: Un-Pad-like unpack
+//
+// FIXME: With no single element added, there is no padding here. Improve
+// naming.
+//===========================================================================//
+
 // CHECK-LABEL: func.func @unpad_like_unpack(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x64xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2], [3]] : tensor<1x1x32x64xf32> into tensor<32x64xf32>


From llvmlistbot at llvm.org  Tue Jul 14 08:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 08:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Refine pack/unpack
 simplification checks (NFC) (PR #209522)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209522@github.com>
Message-ID: <6a565a54.7270d2fd.2d4879.3a96@mx.google.com>


banach-space wrote:

CC @JerryShih

https://github.com/llvm/llvm-project/pull/209522

From llvmlistbot at llvm.org  Tue Jul 14 08:48:49 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Tue, 14 Jul 2026 08:48:49 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a565a61.e55dc341.1f19bd.46b7@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 819f04decf3b21e8bc6a83feadb481f1d56a0907 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 .../cl20-device-side-enqueue-attributes.cl    | 148 +++++++-----------
 llvm/docs/LangRef.md                          |   7 +-
 llvm/include/llvm/IR/Function.h               |   3 +
 llvm/lib/IR/Verifier.cpp                      |   5 +
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |   4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |   2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |   4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |   4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |   2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      |  16 +-
 .../CodeGen/RISCV/double-intrinsics-strict.ll |   2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |   2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |   2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |   2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    |  12 +-
 llvm/test/CodeGen/SystemZ/zos-ppa2.ll         |   2 +-
 .../EarlyCSE/replace-calls-def-attrs.ll       |  67 ++++----
 llvm/test/Transforms/InstCombine/erf.ll       |  40 ++---
 llvm/test/Transforms/InstCombine/fdim.ll      |   7 +-
 .../InstSimplify/disable_folding.ll           |  14 +-
 .../merge-compatible-invokes-of-landingpad.ll |  30 ++--
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir |  50 +++---
 22 files changed, 199 insertions(+), 226 deletions(-)

diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c73f381c0ab1c 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -20,31 +20,14 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
                    a[i] = 4.0f * b[i] + 1.0f;
                  });
 }
-// SPIR32: Function Attrs: convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign)
+// SPIR32: Function Attrs: convergent noinline norecurse nounwind optnone
 // SPIR32-LABEL: define dso_local spir_kernel void @device_side_enqueue(
-// SPIR32-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0:[0-9]+]] !kernel_arg_addr_space [[META1:![0-9]+]] !kernel_arg_access_qual [[META2:![0-9]+]] !kernel_arg_type [[META3:![0-9]+]] !kernel_arg_base_type [[META3]] !kernel_arg_type_qual [[META4:![0-9]+]] {
+// SPIR32-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0:[0-9]+]] !kernel_arg_addr_space [[META2:![0-9]+]] !kernel_arg_access_qual [[META3:![0-9]+]] !kernel_arg_type [[META4:![0-9]+]] !kernel_arg_base_type [[META4]] !kernel_arg_type_qual [[META5:![0-9]+]] {
 // SPIR32-NEXT:  [[ENTRY:.*:]]
 // SPIR32-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // SPIR32-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // SPIR32-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// SPIR32-NEXT:    store ptr addrspace(1) [[A]], ptr [[A_ADDR]], align 4
-// SPIR32-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
-// SPIR32-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
-// SPIR32-NEXT:    [[TMP0:%.*]] = load ptr addrspace(1), ptr [[A_ADDR]], align 4
-// SPIR32-NEXT:    [[TMP1:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
-// SPIR32-NEXT:    [[TMP2:%.*]] = load i32, ptr [[I_ADDR]], align 4
-// SPIR32-NEXT:    call spir_func void @__clang_ocl_kern_imp_device_side_enqueue(ptr addrspace(1) align 4 [[TMP0]], ptr addrspace(1) align 4 [[TMP1]], i32 [[TMP2]]) #[[ATTR5:[0-9]+]]
-// SPIR32-NEXT:    ret void
-//
-//
-// SPIR32: Function Attrs: convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign)
-// SPIR32-LABEL: define dso_local spir_func void @__clang_ocl_kern_imp_device_side_enqueue(
-// SPIR32-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0]] !kernel_arg_addr_space [[META1]] !kernel_arg_access_qual [[META2]] !kernel_arg_type [[META3]] !kernel_arg_base_type [[META3]] !kernel_arg_type_qual [[META4]] {
-// SPIR32-NEXT:  [[ENTRY:.*:]]
-// SPIR32-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
-// SPIR32-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
-// SPIR32-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// SPIR32-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca target("spirv.Queue"), align 4
+// SPIR32-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca ptr, align 4
 // SPIR32-NEXT:    [[FLAGS:%.*]] = alloca i32, align 4
 // SPIR32-NEXT:    [[NDRANGE:%.*]] = alloca [[STRUCT_NDRANGE_T:%.*]], align 4
 // SPIR32-NEXT:    [[TMP:%.*]] = alloca [[STRUCT_NDRANGE_T]], align 4
@@ -53,30 +36,30 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // SPIR32-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
 // SPIR32-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
 // SPIR32-NEXT:    store i32 0, ptr [[FLAGS]], align 4
-// SPIR32-NEXT:    [[TMP0:%.*]] = load target("spirv.Queue"), ptr [[DEFAULT_QUEUE]], align 4
+// SPIR32-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[DEFAULT_QUEUE]], align 4
 // SPIR32-NEXT:    [[TMP1:%.*]] = load i32, ptr [[FLAGS]], align 4
 // SPIR32-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[TMP]], ptr align 4 [[NDRANGE]], i32 4, i1 false)
-// SPIR32-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
+// SPIR32-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
 // SPIR32-NEXT:    store i32 24, ptr [[BLOCK_SIZE]], align 4
-// SPIR32-NEXT:    [[BLOCK_ALIGN:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 1
+// SPIR32-NEXT:    [[BLOCK_ALIGN:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 1
 // SPIR32-NEXT:    store i32 4, ptr [[BLOCK_ALIGN]], align 4
-// SPIR32-NEXT:    [[BLOCK_INVOKE:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 2
+// SPIR32-NEXT:    [[BLOCK_INVOKE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 2
 // SPIR32-NEXT:    store ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke to ptr addrspace(4)), ptr [[BLOCK_INVOKE]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURED:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 3
+// SPIR32-NEXT:    [[BLOCK_CAPTURED:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 3
 // SPIR32-NEXT:    [[TMP2:%.*]] = load ptr addrspace(1), ptr [[A_ADDR]], align 4
 // SPIR32-NEXT:    store ptr addrspace(1) [[TMP2]], ptr [[BLOCK_CAPTURED]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURED1:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 4
+// SPIR32-NEXT:    [[BLOCK_CAPTURED1:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 4
 // SPIR32-NEXT:    [[TMP3:%.*]] = load i32, ptr [[I_ADDR]], align 4
 // SPIR32-NEXT:    store i32 [[TMP3]], ptr [[BLOCK_CAPTURED1]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURED2:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 5
+// SPIR32-NEXT:    [[BLOCK_CAPTURED2:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 5
 // SPIR32-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
 // SPIR32-NEXT:    store ptr addrspace(1) [[TMP4]], ptr [[BLOCK_CAPTURED2]], align 4
 // SPIR32-NEXT:    [[TMP5:%.*]] = addrspacecast ptr [[BLOCK]] to ptr addrspace(4)
-// SPIR32-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(target("spirv.Queue") [[TMP0]], i32 [[TMP1]], ptr [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
+// SPIR32-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(ptr [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
 // SPIR32-NEXT:    ret void
 //
 //
-// SPIR32: Function Attrs: convergent noinline nounwind optnone denormal_fpenv(float: preservesign)
+// SPIR32: Function Attrs: convergent noinline nounwind optnone
 // SPIR32-LABEL: define internal spir_func void @__device_side_enqueue_block_invoke(
 // SPIR32-SAME: ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR:%.*]]) #[[ATTR2:[0-9]+]] {
 // SPIR32-NEXT:  [[ENTRY:.*:]]
@@ -84,23 +67,23 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // SPIR32-NEXT:    [[BLOCK_ADDR:%.*]] = alloca ptr addrspace(4), align 4
 // SPIR32-NEXT:    store ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], ptr [[DOTBLOCK_DESCRIPTOR_ADDR]], align 4
 // SPIR32-NEXT:    store ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], ptr [[BLOCK_ADDR]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 5
+// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 5
 // SPIR32-NEXT:    [[TMP0:%.*]] = load ptr addrspace(1), ptr addrspace(4) [[BLOCK_CAPTURE_ADDR]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR1:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
+// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR1:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
 // SPIR32-NEXT:    [[TMP1:%.*]] = load i32, ptr addrspace(4) [[BLOCK_CAPTURE_ADDR1]], align 4
 // SPIR32-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[TMP0]], i32 [[TMP1]]
 // SPIR32-NEXT:    [[TMP2:%.*]] = load float, ptr addrspace(1) [[ARRAYIDX]], align 4
 // SPIR32-NEXT:    [[TMP3:%.*]] = call float @llvm.fmuladd.f32(float 4.000000e+00, float [[TMP2]], float 1.000000e+00)
-// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR2:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 3
+// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR2:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 3
 // SPIR32-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr addrspace(4) [[BLOCK_CAPTURE_ADDR2]], align 4
-// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR3:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
+// SPIR32-NEXT:    [[BLOCK_CAPTURE_ADDR3:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
 // SPIR32-NEXT:    [[TMP5:%.*]] = load i32, ptr addrspace(4) [[BLOCK_CAPTURE_ADDR3]], align 4
 // SPIR32-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[TMP4]], i32 [[TMP5]]
 // SPIR32-NEXT:    store float [[TMP3]], ptr addrspace(1) [[ARRAYIDX4]], align 4
 // SPIR32-NEXT:    ret void
 //
 //
-// SPIR32: Function Attrs: convergent nounwind denormal_fpenv(float: preservesign)
+// SPIR32: Function Attrs: nounwind
 // SPIR32-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // SPIR32-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // SPIR32-NEXT:  [[ENTRY:.*:]]
@@ -110,29 +93,12 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 //
 // STRICTFP: Function Attrs: convergent noinline norecurse nounwind optnone strictfp
 // STRICTFP-LABEL: define dso_local spir_kernel void @device_side_enqueue(
-// STRICTFP-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0:[0-9]+]] !kernel_arg_addr_space [[META1:![0-9]+]] !kernel_arg_access_qual [[META2:![0-9]+]] !kernel_arg_type [[META3:![0-9]+]] !kernel_arg_base_type [[META3]] !kernel_arg_type_qual [[META4:![0-9]+]] {
-// STRICTFP-NEXT:  [[ENTRY:.*:]]
-// STRICTFP-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
-// STRICTFP-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
-// STRICTFP-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// STRICTFP-NEXT:    store ptr addrspace(1) [[A]], ptr [[A_ADDR]], align 4
-// STRICTFP-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
-// STRICTFP-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
-// STRICTFP-NEXT:    [[TMP0:%.*]] = load ptr addrspace(1), ptr [[A_ADDR]], align 4
-// STRICTFP-NEXT:    [[TMP1:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
-// STRICTFP-NEXT:    [[TMP2:%.*]] = load i32, ptr [[I_ADDR]], align 4
-// STRICTFP-NEXT:    call spir_func void @__clang_ocl_kern_imp_device_side_enqueue(ptr addrspace(1) align 4 [[TMP0]], ptr addrspace(1) align 4 [[TMP1]], i32 [[TMP2]]) #[[ATTR5:[0-9]+]]
-// STRICTFP-NEXT:    ret void
-//
-//
-// STRICTFP: Function Attrs: convergent noinline norecurse nounwind optnone strictfp
-// STRICTFP-LABEL: define dso_local spir_func void @__clang_ocl_kern_imp_device_side_enqueue(
-// STRICTFP-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0]] !kernel_arg_addr_space [[META1]] !kernel_arg_access_qual [[META2]] !kernel_arg_type [[META3]] !kernel_arg_base_type [[META3]] !kernel_arg_type_qual [[META4]] {
+// STRICTFP-SAME: ptr addrspace(1) align 4 [[A:%.*]], ptr addrspace(1) align 4 [[B:%.*]], i32 [[I:%.*]]) #[[ATTR0:[0-9]+]] !kernel_arg_addr_space [[META2:![0-9]+]] !kernel_arg_access_qual [[META3:![0-9]+]] !kernel_arg_type [[META4:![0-9]+]] !kernel_arg_base_type [[META4]] !kernel_arg_type_qual [[META5:![0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
 // STRICTFP-NEXT:    [[A_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // STRICTFP-NEXT:    [[B_ADDR:%.*]] = alloca ptr addrspace(1), align 4
 // STRICTFP-NEXT:    [[I_ADDR:%.*]] = alloca i32, align 4
-// STRICTFP-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca target("spirv.Queue"), align 4
+// STRICTFP-NEXT:    [[DEFAULT_QUEUE:%.*]] = alloca ptr, align 4
 // STRICTFP-NEXT:    [[FLAGS:%.*]] = alloca i32, align 4
 // STRICTFP-NEXT:    [[NDRANGE:%.*]] = alloca [[STRUCT_NDRANGE_T:%.*]], align 4
 // STRICTFP-NEXT:    [[TMP:%.*]] = alloca [[STRUCT_NDRANGE_T]], align 4
@@ -141,26 +107,26 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    store ptr addrspace(1) [[B]], ptr [[B_ADDR]], align 4
 // STRICTFP-NEXT:    store i32 [[I]], ptr [[I_ADDR]], align 4
 // STRICTFP-NEXT:    store i32 0, ptr [[FLAGS]], align 4
-// STRICTFP-NEXT:    [[TMP0:%.*]] = load target("spirv.Queue"), ptr [[DEFAULT_QUEUE]], align 4
+// STRICTFP-NEXT:    [[TMP0:%.*]] = load ptr, ptr [[DEFAULT_QUEUE]], align 4
 // STRICTFP-NEXT:    [[TMP1:%.*]] = load i32, ptr [[FLAGS]], align 4
-// STRICTFP-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[TMP]], ptr align 4 [[NDRANGE]], i32 4, i1 false) #[[ATTR6:[0-9]+]]
-// STRICTFP-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
+// STRICTFP-NEXT:    call void @llvm.memcpy.p0.p0.i32(ptr align 4 [[TMP]], ptr align 4 [[NDRANGE]], i32 4, i1 false) #[[ATTR5:[0-9]+]]
+// STRICTFP-NEXT:    [[BLOCK_SIZE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 0
 // STRICTFP-NEXT:    store i32 24, ptr [[BLOCK_SIZE]], align 4
-// STRICTFP-NEXT:    [[BLOCK_ALIGN:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 1
+// STRICTFP-NEXT:    [[BLOCK_ALIGN:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 1
 // STRICTFP-NEXT:    store i32 4, ptr [[BLOCK_ALIGN]], align 4
-// STRICTFP-NEXT:    [[BLOCK_INVOKE:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 2
+// STRICTFP-NEXT:    [[BLOCK_INVOKE:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 2
 // STRICTFP-NEXT:    store ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke to ptr addrspace(4)), ptr [[BLOCK_INVOKE]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURED:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 3
+// STRICTFP-NEXT:    [[BLOCK_CAPTURED:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 3
 // STRICTFP-NEXT:    [[TMP2:%.*]] = load ptr addrspace(1), ptr [[A_ADDR]], align 4
 // STRICTFP-NEXT:    store ptr addrspace(1) [[TMP2]], ptr [[BLOCK_CAPTURED]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURED1:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 4
+// STRICTFP-NEXT:    [[BLOCK_CAPTURED1:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 4
 // STRICTFP-NEXT:    [[TMP3:%.*]] = load i32, ptr [[I_ADDR]], align 4
 // STRICTFP-NEXT:    store i32 [[TMP3]], ptr [[BLOCK_CAPTURED1]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURED2:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 5
+// STRICTFP-NEXT:    [[BLOCK_CAPTURED2:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr [[BLOCK]], i32 0, i32 5
 // STRICTFP-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr [[B_ADDR]], align 4
 // STRICTFP-NEXT:    store ptr addrspace(1) [[TMP4]], ptr [[BLOCK_CAPTURED2]], align 4
 // STRICTFP-NEXT:    [[TMP5:%.*]] = addrspacecast ptr [[BLOCK]] to ptr addrspace(4)
-// STRICTFP-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(target("spirv.Queue") [[TMP0]], i32 [[TMP1]], ptr [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]]) #[[ATTR6]]
+// STRICTFP-NEXT:    [[TMP6:%.*]] = call spir_func i32 @__enqueue_kernel_basic(ptr [[TMP0]], i32 [[TMP1]], ptr byval([[STRUCT_NDRANGE_T]]) [[TMP]], ptr addrspace(4) addrspacecast (ptr @__device_side_enqueue_block_invoke_kernel to ptr addrspace(4)), ptr addrspace(4) [[TMP5]])
 // STRICTFP-NEXT:    ret void
 //
 //
@@ -172,54 +138,54 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    [[BLOCK_ADDR:%.*]] = alloca ptr addrspace(4), align 4
 // STRICTFP-NEXT:    store ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], ptr [[DOTBLOCK_DESCRIPTOR_ADDR]], align 4
 // STRICTFP-NEXT:    store ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], ptr [[BLOCK_ADDR]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 5
+// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 5
 // STRICTFP-NEXT:    [[TMP0:%.*]] = load ptr addrspace(1), ptr addrspace(4) [[BLOCK_CAPTURE_ADDR]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR1:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
+// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR1:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
 // STRICTFP-NEXT:    [[TMP1:%.*]] = load i32, ptr addrspace(4) [[BLOCK_CAPTURE_ADDR1]], align 4
 // STRICTFP-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[TMP0]], i32 [[TMP1]]
 // STRICTFP-NEXT:    [[TMP2:%.*]] = load float, ptr addrspace(1) [[ARRAYIDX]], align 4
-// STRICTFP-NEXT:    [[TMP3:%.*]] = call float @llvm.experimental.constrained.fmuladd.f32(float 4.000000e+00, float [[TMP2]], float 1.000000e+00, metadata !"round.tonearest", metadata !"fpexcept.strict") #[[ATTR6]]
-// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR2:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 3
+// STRICTFP-NEXT:    [[TMP3:%.*]] = call float @llvm.experimental.constrained.fmuladd.f32(float 4.000000e+00, float [[TMP2]], float 1.000000e+00, metadata !"round.tonearest", metadata !"fpexcept.strict") #[[ATTR5]]
+// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR2:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 3
 // STRICTFP-NEXT:    [[TMP4:%.*]] = load ptr addrspace(1), ptr addrspace(4) [[BLOCK_CAPTURE_ADDR2]], align 4
-// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR3:%.*]] = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
+// STRICTFP-NEXT:    [[BLOCK_CAPTURE_ADDR3:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 4
 // STRICTFP-NEXT:    [[TMP5:%.*]] = load i32, ptr addrspace(4) [[BLOCK_CAPTURE_ADDR3]], align 4
 // STRICTFP-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds float, ptr addrspace(1) [[TMP4]], i32 [[TMP5]]
 // STRICTFP-NEXT:    store float [[TMP3]], ptr addrspace(1) [[ARRAYIDX4]], align 4
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: nounwind
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
-// STRICTFP-NEXT:    call spir_func void @__device_side_enqueue_block_invoke(ptr addrspace(4) [[TMP0]]) #[[ATTR6]]
+// STRICTFP-NEXT:    call spir_func void @__device_side_enqueue_block_invoke(ptr addrspace(4) [[TMP0]]) #[[ATTR5]]
 // STRICTFP-NEXT:    ret void
 //
 //.
-// SPIR32: attributes #[[ATTR0]] = { convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size" }
-// SPIR32: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
-// SPIR32: attributes #[[ATTR2]] = { convergent noinline nounwind optnone denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size" }
-// SPIR32: attributes #[[ATTR3:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-// SPIR32: attributes #[[ATTR4]] = { convergent nounwind denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
-// SPIR32: attributes #[[ATTR5]] = { convergent nounwind "uniform-work-group-size" }
+// SPIR32: attributes #[[ATTR0]] = { convergent noinline norecurse nounwind optnone "denormal-fp-math-f32"="preserve-sign,preserve-sign" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size"="true" }
+// SPIR32: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
+// SPIR32: attributes #[[ATTR2]] = { convergent noinline nounwind optnone "denormal-fp-math-f32"="preserve-sign,preserve-sign" "no-trapping-math"="true" "stack-protector-buffer-size"="8" }
+// SPIR32: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind speculatable willreturn memory(none) }
+// SPIR32: attributes #[[ATTR4]] = { nounwind }
 //.
-// STRICTFP: attributes #[[ATTR0]] = { convergent noinline norecurse nounwind optnone strictfp "stack-protector-buffer-size"="8" }
-// STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
+// STRICTFP: attributes #[[ATTR0]] = { convergent noinline norecurse nounwind optnone strictfp "stack-protector-buffer-size"="8" "strictfp" "uniform-work-group-size"="false" }
+// STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
-// STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
-// STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
-// STRICTFP: attributes #[[ATTR6]] = { strictfp }
+// STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(inaccessiblemem: readwrite) }
+// STRICTFP: attributes #[[ATTR4]] = { nounwind }
+// STRICTFP: attributes #[[ATTR5]] = { strictfp }
 //.
-// SPIR32: [[META0:![0-9]+]] = !{i32 2, i32 0}
-// SPIR32: [[META1]] = !{i32 1, i32 1, i32 0}
-// SPIR32: [[META2]] = !{!"none", !"none", !"none"}
-// SPIR32: [[META3]] = !{!"float*", !"float*", !"int"}
-// SPIR32: [[META4]] = !{!"", !"", !""}
+// SPIR32: [[META0:![0-9]+]] = !{i32 1, !"wchar_size", i32 4}
+// SPIR32: [[META1:![0-9]+]] = !{i32 2, i32 0}
+// SPIR32: [[META2]] = !{i32 1, i32 1, i32 0}
+// SPIR32: [[META3]] = !{!"none", !"none", !"none"}
+// SPIR32: [[META4]] = !{!"float*", !"float*", !"int"}
+// SPIR32: [[META5]] = !{!"", !"", !""}
 //.
-// STRICTFP: [[META0:![0-9]+]] = !{i32 2, i32 0}
-// STRICTFP: [[META1]] = !{i32 1, i32 1, i32 0}
-// STRICTFP: [[META2]] = !{!"none", !"none", !"none"}
-// STRICTFP: [[META3]] = !{!"float*", !"float*", !"int"}
-// STRICTFP: [[META4]] = !{!"", !"", !""}
+// STRICTFP: [[META0:![0-9]+]] = !{i32 1, !"wchar_size", i32 4}
+// STRICTFP: [[META1:![0-9]+]] = !{i32 2, i32 0}
+// STRICTFP: [[META2]] = !{i32 1, i32 1, i32 0}
+// STRICTFP: [[META3]] = !{!"none", !"none", !"none"}
+// STRICTFP: [[META4]] = !{!"float*", !"float*", !"int"}
+// STRICTFP: [[META5]] = !{!"", !"", !""}
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index f555ca543ffdc..ab5960c4cbad3 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26573,9 +26575,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index fa0fe7a2e6092..2b4f0048cae4f 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
index a4c1d50df5543..4e6de83d7b9c2 100644
--- a/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
+++ b/llvm/test/CodeGen/SystemZ/zos-ppa2.ll
@@ -21,7 +21,7 @@
 ; CHECK:  DC XL2'0000'
 ; CHECK: L#DVS DS 0H
 ; CHECK:  DC XL14'F1F9F7F0F0F1F0F1F0F0F0F0F0F0'
-; CHECK:  DC XL6'F2F3F0F0F0F0'
+; CHECK:  DC XL6'F2F4F0F0F0F0'
 ; CHECK:  DC XL2'0000'
 
 ; CHECK: C_@@QPPA2 CATTR ALIGN(3),FILL(0),NOTEXECUTABLE,READONLY,RMODE(64),PART(.
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)


From llvmlistbot at llvm.org  Tue Jul 14 09:52:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 09:52:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 83dc9c6 - [mlir][openacc] Remove
	OpenACCToLLVMIRTranslation (#209295)
Message-ID: <6a566951.d76d81af.39f9be.fa02@mx.google.com>


Author: Valentin Clement (バレンタイン クレメン)
Date: 2026-07-14T09:52:29-07:00
New Revision: 83dc9c6c9a56bc90faa810512e0577e59dc46033

URL: https://github.com/llvm/llvm-project/commit/83dc9c6c9a56bc90faa810512e0577e59dc46033
DIFF: https://github.com/llvm/llvm-project/commit/83dc9c6c9a56bc90faa810512e0577e59dc46033.diff

LOG: [mlir][openacc] Remove OpenACCToLLVMIRTranslation (#209295)

OpenACCToLLVMIRTranslation was designed more than 5 years ago and does
not align with the current upstreaming effort. OpenACC will be converted
to LLVM IR dialect before the translation to LLVM IR so this translation
is obsolete. Remove it to avoid confusion.

Added: 
    

Modified: 
    flang/lib/Optimizer/Support/CMakeLists.txt
    flang/lib/Optimizer/Support/InitFIR.cpp
    flang/lib/Optimizer/Transforms/CMakeLists.txt
    mlir/include/mlir/Target/LLVMIR/Dialect/All.h
    mlir/lib/Target/LLVMIR/CMakeLists.txt
    mlir/lib/Target/LLVMIR/Dialect/CMakeLists.txt
    utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Support/BUILD.bazel
    utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Transforms/BUILD.bazel
    utils/bazel/llvm-project-overlay/mlir/BUILD.bazel

Removed: 
    mlir/include/mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h
    mlir/lib/Target/LLVMIR/Dialect/OpenACC/CMakeLists.txt
    mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/openacc-llvm.mlir


################################################################################
diff  --git a/flang/lib/Optimizer/Support/CMakeLists.txt b/flang/lib/Optimizer/Support/CMakeLists.txt
index 6f3652b1a86af..9f6069be6323d 100644
--- a/flang/lib/Optimizer/Support/CMakeLists.txt
+++ b/flang/lib/Optimizer/Support/CMakeLists.txt
@@ -24,7 +24,6 @@ add_flang_library(FIRSupport
   MLIR_LIBS
   MLIRBuiltinToLLVMIRTranslation
   MLIRLLVMToLLVMIRTranslation
-  MLIROpenACCToLLVMIRTranslation
   MLIROpenMPToLLVMIRTranslation
   MLIRRegisterAllDialects
   MLIRRegisterAllExtensions

diff  --git a/flang/lib/Optimizer/Support/InitFIR.cpp b/flang/lib/Optimizer/Support/InitFIR.cpp
index 0753c4511d9c6..09852b1add372 100644
--- a/flang/lib/Optimizer/Support/InitFIR.cpp
+++ b/flang/lib/Optimizer/Support/InitFIR.cpp
@@ -9,13 +9,10 @@
 #include "flang/Optimizer/Support/InitFIR.h"
 #include "mlir/Target/LLVMIR/Dialect/Builtin/BuiltinToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.h"
-#include "mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 
 void fir::support::registerLLVMTranslation(mlir::MLIRContext &context) {
   mlir::DialectRegistry registry;
-  // Register OpenACC dialect interface here as well.
-  registerOpenACCDialectTranslation(registry);
   // Register OpenMP dialect interface here as well.
   registerOpenMPDialectTranslation(registry);
   // Register LLVM-IR dialect interface.

diff  --git a/flang/lib/Optimizer/Transforms/CMakeLists.txt b/flang/lib/Optimizer/Transforms/CMakeLists.txt
index 8ff67d901b31d..1a4940af95d3e 100644
--- a/flang/lib/Optimizer/Transforms/CMakeLists.txt
+++ b/flang/lib/Optimizer/Transforms/CMakeLists.txt
@@ -91,7 +91,6 @@ add_flang_library(FIRTransforms
   MLIRMathTransforms
   MLIRNVVMDialect
   MLIROpenACCDialect
-  MLIROpenACCToLLVMIRTranslation
   MLIROpenMPDialect
   MLIRTransformUtils
 )

diff  --git a/mlir/include/mlir/Target/LLVMIR/Dialect/All.h b/mlir/include/mlir/Target/LLVMIR/Dialect/All.h
index 05b66ace902ad..95e90789e9580 100644
--- a/mlir/include/mlir/Target/LLVMIR/Dialect/All.h
+++ b/mlir/include/mlir/Target/LLVMIR/Dialect/All.h
@@ -23,7 +23,6 @@
 #include "mlir/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/NVVM/LLVMIRToNVVMTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/NVVM/NVVMToLLVMIRTranslation.h"
-#include "mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/Ptr/PtrToLLVMIRTranslation.h"
 #include "mlir/Target/LLVMIR/Dialect/ROCDL/ROCDLToLLVMIRTranslation.h"
@@ -44,7 +43,6 @@ static inline void registerAllToLLVMIRTranslations(DialectRegistry &registry) {
   registerGPUDialectTranslation(registry);
   registerLLVMDialectTranslation(registry);
   registerNVVMDialectTranslation(registry);
-  registerOpenACCDialectTranslation(registry);
   registerOpenMPDialectTranslation(registry);
   registerPtrDialectTranslation(registry);
   registerROCDLDialectTranslation(registry);

diff  --git a/mlir/include/mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h b/mlir/include/mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h
deleted file mode 100644
index d7e847f9cd32a..0000000000000
--- a/mlir/include/mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h
+++ /dev/null
@@ -1,31 +0,0 @@
-//===- OpenACCToLLVMIRTranslation.h - OpenACC Dialect to LLVM IR -- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This provides registration calls for OpenACC dialect to LLVM IR translation.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef MLIR_TARGET_LLVMIR_DIALECT_OPENACC_OPENACCTOLLVMIRTRANSLATION_H
-#define MLIR_TARGET_LLVMIR_DIALECT_OPENACC_OPENACCTOLLVMIRTRANSLATION_H
-
-namespace mlir {
-
-class DialectRegistry;
-class MLIRContext;
-
-/// Register the OpenACC dialect and the translation to the LLVM IR in
-/// the given registry;
-void registerOpenACCDialectTranslation(DialectRegistry &registry);
-
-/// Register the OpenACC dialect and the translation in the registry
-/// associated with the given context.
-void registerOpenACCDialectTranslation(MLIRContext &context);
-
-} // namespace mlir
-
-#endif // MLIR_TARGET_LLVMIR_DIALECT_OPENACC_OPENACCTOLLVMIRTRANSLATION_H

diff  --git a/mlir/lib/Target/LLVMIR/CMakeLists.txt b/mlir/lib/Target/LLVMIR/CMakeLists.txt
index db221697edd05..69fd5583fac2a 100644
--- a/mlir/lib/Target/LLVMIR/CMakeLists.txt
+++ b/mlir/lib/Target/LLVMIR/CMakeLists.txt
@@ -56,7 +56,6 @@ add_mlir_translation_library(MLIRToLLVMIRTranslationRegistration
   MLIRGPUToLLVMIRTranslation
   MLIRLLVMToLLVMIRTranslation
   MLIRNVVMToLLVMIRTranslation
-  MLIROpenACCToLLVMIRTranslation
   MLIROpenMPToLLVMIRTranslation
   MLIRROCDLToLLVMIRTranslation
   MLIRPtrToLLVMIRTranslation

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/CMakeLists.txt b/mlir/lib/Target/LLVMIR/Dialect/CMakeLists.txt
index a102c4323075b..d590b236cfc91 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/CMakeLists.txt
+++ b/mlir/lib/Target/LLVMIR/Dialect/CMakeLists.txt
@@ -5,7 +5,6 @@ add_subdirectory(Builtin)
 add_subdirectory(GPU)
 add_subdirectory(LLVMIR)
 add_subdirectory(NVVM)
-add_subdirectory(OpenACC)
 add_subdirectory(OpenMP)
 add_subdirectory(ROCDL)
 add_subdirectory(Ptr)

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/CMakeLists.txt b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/CMakeLists.txt
deleted file mode 100644
index e43581e28c77a..0000000000000
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/CMakeLists.txt
+++ /dev/null
@@ -1,14 +0,0 @@
-add_mlir_translation_library(MLIROpenACCToLLVMIRTranslation
-  OpenACCToLLVMIRTranslation.cpp
-
-  LINK_COMPONENTS
-  Core
-
-  LINK_LIBS PUBLIC
-  MLIRIR
-  MLIRLLVMDialect
-  MLIROpenACCDialect
-  MLIRSupport
-  MLIRTargetLLVMIRExport
-  MLIRTransformUtils
-  )

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
deleted file mode 100644
index 9bb6939043621..0000000000000
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.cpp
+++ /dev/null
@@ -1,541 +0,0 @@
-//===- OpenACCToLLVMIRTranslation.cpp -------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-//
-// This file implements a translation between the MLIR OpenACC dialect and LLVM
-// IR.
-//
-//===----------------------------------------------------------------------===//
-
-#include "mlir/Target/LLVMIR/Dialect/OpenACC/OpenACCToLLVMIRTranslation.h"
-#include "mlir/Analysis/TopologicalSortUtils.h"
-#include "mlir/Dialect/LLVMIR/LLVMDialect.h"
-#include "mlir/Dialect/OpenACC/OpenACC.h"
-#include "mlir/IR/Operation.h"
-#include "mlir/Support/LLVM.h"
-#include "mlir/Target/LLVMIR/Dialect/OpenMPCommon.h"
-#include "mlir/Target/LLVMIR/ModuleTranslation.h"
-
-#include "llvm/ADT/TypeSwitch.h"
-#include "llvm/Frontend/OpenMP/OMPConstants.h"
-
-using namespace mlir;
-
-using OpenACCIRBuilder = llvm::OpenMPIRBuilder;
-
-//===----------------------------------------------------------------------===//
-// Utility functions
-//===----------------------------------------------------------------------===//
-
-/// Flag values are extracted from openmp/libomptarget/include/omptarget.h and
-/// mapped to corresponding OpenACC flags.
-static constexpr uint64_t kCreateFlag = 0x000;
-static constexpr uint64_t kDeviceCopyinFlag = 0x001;
-static constexpr uint64_t kHostCopyoutFlag = 0x002;
-static constexpr uint64_t kPresentFlag = 0x1000;
-static constexpr uint64_t kDeleteFlag = 0x008;
-// Runtime extension to implement the OpenACC second reference counter.
-static constexpr uint64_t kHoldFlag = 0x2000;
-
-/// Default value for the device id
-static constexpr int64_t kDefaultDevice = -1;
-
-/// Create the location struct from the operation location information.
-static llvm::Value *createSourceLocationInfo(OpenACCIRBuilder &builder,
-                                             Operation *op) {
-  auto loc = op->getLoc();
-  auto funcOp = op->getParentOfType<LLVM::LLVMFuncOp>();
-  StringRef funcName = funcOp ? funcOp.getName() : "unknown";
-  uint32_t strLen;
-  llvm::Constant *locStr = mlir::LLVM::createSourceLocStrFromLocation(
-      loc, builder, funcName, strLen);
-  return builder.getOrCreateIdent(locStr, strLen);
-}
-
-/// Return the runtime function used to lower the given operation.
-static llvm::Function *getAssociatedFunction(OpenACCIRBuilder &builder,
-                                             Operation *op) {
-  return llvm::TypeSwitch<Operation *, llvm::Function *>(op)
-      .Case([&](acc::EnterDataOp) {
-        return builder.getOrCreateRuntimeFunctionPtr(
-            llvm::omp::OMPRTL___tgt_target_data_begin_mapper);
-      })
-      .Case([&](acc::ExitDataOp) {
-        return builder.getOrCreateRuntimeFunctionPtr(
-            llvm::omp::OMPRTL___tgt_target_data_end_mapper);
-      })
-      .Case([&](acc::UpdateOp) {
-        return builder.getOrCreateRuntimeFunctionPtr(
-            llvm::omp::OMPRTL___tgt_target_data_update_mapper);
-      });
-  llvm_unreachable("Unknown OpenACC operation");
-}
-
-/// Extract pointer, size and mapping information from operands
-/// to populate the future functions arguments.
-static LogicalResult
-processOperands(llvm::IRBuilderBase &builder,
-                LLVM::ModuleTranslation &moduleTranslation, Operation *op,
-                ValueRange operands, unsigned totalNbOperand,
-                uint64_t operandFlag, SmallVector<uint64_t> &flags,
-                SmallVectorImpl<llvm::Constant *> &names, unsigned &index,
-                struct OpenACCIRBuilder::MapperAllocas &mapperAllocas) {
-  OpenACCIRBuilder *accBuilder = moduleTranslation.getOpenMPBuilder();
-  llvm::LLVMContext &ctx = builder.getContext();
-  auto *i8PtrTy = llvm::PointerType::getUnqual(ctx);
-  auto *arrI8PtrTy = llvm::ArrayType::get(i8PtrTy, totalNbOperand);
-  auto *i64Ty = llvm::Type::getInt64Ty(ctx);
-  auto *arrI64Ty = llvm::ArrayType::get(i64Ty, totalNbOperand);
-
-  for (Value data : operands) {
-    llvm::Value *dataValue = moduleTranslation.lookupValue(data);
-
-    llvm::Value *dataPtrBase;
-    llvm::Value *dataPtr;
-    llvm::Value *dataSize;
-
-    if (isa<LLVM::LLVMPointerType>(data.getType())) {
-      dataPtrBase = dataValue;
-      dataPtr = dataValue;
-      dataSize = accBuilder->getSizeInBytes(dataValue);
-    } else {
-      return op->emitOpError()
-             << "Data operand must be legalized before translation."
-             << "Unsupported type: " << data.getType();
-    }
-
-    // Store base pointer extracted from operand into the i-th position of
-    // argBase.
-    llvm::Value *ptrBaseGEP = builder.CreateInBoundsGEP(
-        arrI8PtrTy, mapperAllocas.ArgsBase,
-        {builder.getInt32(0), builder.getInt32(index)});
-    builder.CreateStore(dataPtrBase, ptrBaseGEP);
-
-    // Store pointer extracted from operand into the i-th position of args.
-    llvm::Value *ptrGEP = builder.CreateInBoundsGEP(
-        arrI8PtrTy, mapperAllocas.Args,
-        {builder.getInt32(0), builder.getInt32(index)});
-    builder.CreateStore(dataPtr, ptrGEP);
-
-    // Store size extracted from operand into the i-th position of argSizes.
-    llvm::Value *sizeGEP = builder.CreateInBoundsGEP(
-        arrI64Ty, mapperAllocas.ArgSizes,
-        {builder.getInt32(0), builder.getInt32(index)});
-    builder.CreateStore(dataSize, sizeGEP);
-
-    flags.push_back(operandFlag);
-    llvm::Constant *mapName =
-        mlir::LLVM::createMappingInformation(data.getLoc(), *accBuilder);
-    names.push_back(mapName);
-    ++index;
-  }
-  return success();
-}
-
-/// Process data operands from acc::EnterDataOp
-static LogicalResult
-processDataOperands(llvm::IRBuilderBase &builder,
-                    LLVM::ModuleTranslation &moduleTranslation,
-                    acc::EnterDataOp op, SmallVector<uint64_t> &flags,
-                    SmallVectorImpl<llvm::Constant *> &names,
-                    struct OpenACCIRBuilder::MapperAllocas &mapperAllocas) {
-  // TODO add `create_zero` and `attach` operands
-
-  unsigned index = 0;
-
-  // Create operands are handled as `alloc` call.
-  // Copyin operands are handled as `to` call.
-  llvm::SmallVector<mlir::Value> create, copyin;
-  for (mlir::Value dataOp : op.getDataClauseOperands()) {
-    if (auto createOp = dataOp.getDefiningOp<acc::CreateOp>()) {
-      create.push_back(createOp.getVarPtr());
-    } else if (auto copyinOp = mlir::dyn_cast_or_null<acc::CopyinOp>(
-                   dataOp.getDefiningOp())) {
-      copyin.push_back(copyinOp.getVarPtr());
-    }
-  }
-
-  auto nbTotalOperands = create.size() + copyin.size();
-
-  // Create operands are handled as `alloc` call.
-  if (failed(processOperands(builder, moduleTranslation, op, create,
-                             nbTotalOperands, kCreateFlag, flags, names, index,
-                             mapperAllocas)))
-    return failure();
-
-  // Copyin operands are handled as `to` call.
-  if (failed(processOperands(builder, moduleTranslation, op, copyin,
-                             nbTotalOperands, kDeviceCopyinFlag, flags, names,
-                             index, mapperAllocas)))
-    return failure();
-
-  return success();
-}
-
-/// Process data operands from acc::ExitDataOp
-static LogicalResult
-processDataOperands(llvm::IRBuilderBase &builder,
-                    LLVM::ModuleTranslation &moduleTranslation,
-                    acc::ExitDataOp op, SmallVector<uint64_t> &flags,
-                    SmallVectorImpl<llvm::Constant *> &names,
-                    struct OpenACCIRBuilder::MapperAllocas &mapperAllocas) {
-  // TODO add `detach` operands
-
-  unsigned index = 0;
-
-  llvm::SmallVector<mlir::Value> deleteOperands, copyoutOperands;
-  for (mlir::Value dataOp : op.getDataClauseOperands()) {
-    if (auto devicePtrOp = mlir::dyn_cast_or_null<acc::GetDevicePtrOp>(
-            dataOp.getDefiningOp())) {
-      for (auto &u : devicePtrOp.getAccPtr().getUses()) {
-        if (mlir::dyn_cast_or_null<acc::DeleteOp>(u.getOwner()))
-          deleteOperands.push_back(devicePtrOp.getVarPtr());
-        else if (mlir::dyn_cast_or_null<acc::CopyoutOp>(u.getOwner()))
-          copyoutOperands.push_back(devicePtrOp.getVarPtr());
-      }
-    }
-  }
-
-  auto nbTotalOperands = deleteOperands.size() + copyoutOperands.size();
-
-  // Delete operands are handled as `delete` call.
-  if (failed(processOperands(builder, moduleTranslation, op, deleteOperands,
-                             nbTotalOperands, kDeleteFlag, flags, names, index,
-                             mapperAllocas)))
-    return failure();
-
-  // Copyout operands are handled as `from` call.
-  if (failed(processOperands(builder, moduleTranslation, op, copyoutOperands,
-                             nbTotalOperands, kHostCopyoutFlag, flags, names,
-                             index, mapperAllocas)))
-    return failure();
-
-  return success();
-}
-
-/// Process data operands from acc::UpdateOp
-static LogicalResult
-processDataOperands(llvm::IRBuilderBase &builder,
-                    LLVM::ModuleTranslation &moduleTranslation,
-                    acc::UpdateOp op, SmallVector<uint64_t> &flags,
-                    SmallVectorImpl<llvm::Constant *> &names,
-                    struct OpenACCIRBuilder::MapperAllocas &mapperAllocas) {
-  unsigned index = 0;
-
-  // Host operands are handled as `from` call.
-  // Device operands are handled as `to` call.
-  llvm::SmallVector<mlir::Value> from, to;
-  for (mlir::Value dataOp : op.getDataClauseOperands()) {
-    if (auto getDevicePtrOp = mlir::dyn_cast_or_null<acc::GetDevicePtrOp>(
-            dataOp.getDefiningOp())) {
-      from.push_back(getDevicePtrOp.getVarPtr());
-    } else if (auto updateDeviceOp =
-                   mlir::dyn_cast_or_null<acc::UpdateDeviceOp>(
-                       dataOp.getDefiningOp())) {
-      to.push_back(updateDeviceOp.getVarPtr());
-    }
-  }
-
-  if (failed(processOperands(builder, moduleTranslation, op, from, from.size(),
-                             kHostCopyoutFlag, flags, names, index,
-                             mapperAllocas)))
-    return failure();
-
-  if (failed(processOperands(builder, moduleTranslation, op, to, to.size(),
-                             kDeviceCopyinFlag, flags, names, index,
-                             mapperAllocas)))
-    return failure();
-  return success();
-}
-
-//===----------------------------------------------------------------------===//
-// Conversion functions
-//===----------------------------------------------------------------------===//
-
-/// Converts an OpenACC data operation into LLVM IR.
-static LogicalResult convertDataOp(acc::DataOp &op,
-                                   llvm::IRBuilderBase &builder,
-                                   LLVM::ModuleTranslation &moduleTranslation) {
-  llvm::LLVMContext &ctx = builder.getContext();
-  auto enclosingFuncOp = op.getOperation()->getParentOfType<LLVM::LLVMFuncOp>();
-  llvm::Function *enclosingFunction =
-      moduleTranslation.lookupFunction(enclosingFuncOp.getName());
-
-  OpenACCIRBuilder *accBuilder = moduleTranslation.getOpenMPBuilder();
-
-  llvm::Value *srcLocInfo = createSourceLocationInfo(*accBuilder, op);
-
-  llvm::Function *beginMapperFunc = accBuilder->getOrCreateRuntimeFunctionPtr(
-      llvm::omp::OMPRTL___tgt_target_data_begin_mapper);
-
-  llvm::Function *endMapperFunc = accBuilder->getOrCreateRuntimeFunctionPtr(
-      llvm::omp::OMPRTL___tgt_target_data_end_mapper);
-
-  // Number of arguments in the data operation.
-  unsigned totalNbOperand = op.getNumDataOperands();
-
-  struct OpenACCIRBuilder::MapperAllocas mapperAllocas;
-  OpenACCIRBuilder::InsertPointTy allocaIP(
-      &enclosingFunction->getEntryBlock(),
-      enclosingFunction->getEntryBlock().getFirstInsertionPt());
-  accBuilder->createMapperAllocas(builder.saveIP(), allocaIP, totalNbOperand,
-                                  mapperAllocas);
-
-  SmallVector<uint64_t> flags;
-  SmallVector<llvm::Constant *> names;
-  unsigned index = 0;
-
-  // TODO handle no_create, deviceptr and attach operands.
-
-  llvm::SmallVector<mlir::Value> copyin, copyout, create, present,
-      deleteOperands;
-  for (mlir::Value dataOp : op.getDataClauseOperands()) {
-    if (auto devicePtrOp = mlir::dyn_cast_or_null<acc::GetDevicePtrOp>(
-            dataOp.getDefiningOp())) {
-      for (auto &u : devicePtrOp.getAccPtr().getUses()) {
-        if (mlir::dyn_cast_or_null<acc::DeleteOp>(u.getOwner())) {
-          deleteOperands.push_back(devicePtrOp.getVarPtr());
-        } else if (mlir::dyn_cast_or_null<acc::CopyoutOp>(u.getOwner())) {
-          // TODO copyout zero currenlty handled as copyout. Update when
-          // extension available.
-          copyout.push_back(devicePtrOp.getVarPtr());
-        }
-      }
-    } else if (auto copyinOp = mlir::dyn_cast_or_null<acc::CopyinOp>(
-                   dataOp.getDefiningOp())) {
-      // TODO copyin readonly currenlty handled as copyin. Update when extension
-      // available.
-      copyin.push_back(copyinOp.getVarPtr());
-    } else if (auto createOp = mlir::dyn_cast_or_null<acc::CreateOp>(
-                   dataOp.getDefiningOp())) {
-      // TODO create zero currenlty handled as create. Update when extension
-      // available.
-      create.push_back(createOp.getVarPtr());
-    } else if (auto presentOp = mlir::dyn_cast_or_null<acc::PresentOp>(
-                   dataOp.getDefiningOp())) {
-      present.push_back(createOp.getVarPtr());
-    }
-  }
-
-  auto nbTotalOperands = copyin.size() + copyout.size() + create.size() +
-                         present.size() + deleteOperands.size();
-
-  // Copyin operands are handled as `to` call.
-  if (failed(processOperands(builder, moduleTranslation, op, copyin,
-                             nbTotalOperands, kDeviceCopyinFlag | kHoldFlag,
-                             flags, names, index, mapperAllocas)))
-    return failure();
-
-  // Delete operands are handled as `delete` call.
-  if (failed(processOperands(builder, moduleTranslation, op, deleteOperands,
-                             nbTotalOperands, kDeleteFlag, flags, names, index,
-                             mapperAllocas)))
-    return failure();
-
-  // Copyout operands are handled as `from` call.
-  if (failed(processOperands(builder, moduleTranslation, op, copyout,
-                             nbTotalOperands, kHostCopyoutFlag | kHoldFlag,
-                             flags, names, index, mapperAllocas)))
-    return failure();
-
-  // Create operands are handled as `alloc` call.
-  if (failed(processOperands(builder, moduleTranslation, op, create,
-                             nbTotalOperands, kCreateFlag | kHoldFlag, flags,
-                             names, index, mapperAllocas)))
-    return failure();
-
-  if (failed(processOperands(builder, moduleTranslation, op, present,
-                             nbTotalOperands, kPresentFlag | kHoldFlag, flags,
-                             names, index, mapperAllocas)))
-    return failure();
-
-  llvm::GlobalVariable *maptypes =
-      accBuilder->createOffloadMaptypes(flags, ".offload_maptypes");
-  llvm::Value *maptypesArg = builder.CreateConstInBoundsGEP2_32(
-      llvm::ArrayType::get(llvm::Type::getInt64Ty(ctx), totalNbOperand),
-      maptypes, /*Idx0=*/0, /*Idx1=*/0);
-
-  llvm::GlobalVariable *mapnames =
-      accBuilder->createOffloadMapnames(names, ".offload_mapnames");
-  llvm::Value *mapnamesArg = builder.CreateConstInBoundsGEP2_32(
-      llvm::ArrayType::get(llvm::PointerType::getUnqual(ctx), totalNbOperand),
-      mapnames, /*Idx0=*/0, /*Idx1=*/0);
-
-  // Create call to start the data region.
-  accBuilder->emitMapperCall(builder.saveIP(), beginMapperFunc, srcLocInfo,
-                             maptypesArg, mapnamesArg, mapperAllocas,
-                             kDefaultDevice, totalNbOperand);
-
-  // Convert the region.
-  llvm::BasicBlock *entryBlock = nullptr;
-
-  for (Block &bb : op.getRegion()) {
-    llvm::BasicBlock *llvmBB = llvm::BasicBlock::Create(
-        ctx, "acc.data", builder.GetInsertBlock()->getParent());
-    if (entryBlock == nullptr)
-      entryBlock = llvmBB;
-    moduleTranslation.mapBlock(&bb, llvmBB);
-  }
-
-  auto afterDataRegion = builder.saveIP();
-
-  llvm::UncondBrInst *sourceTerminator = builder.CreateBr(entryBlock);
-
-  builder.restoreIP(afterDataRegion);
-  llvm::BasicBlock *endDataBlock = llvm::BasicBlock::Create(
-      ctx, "acc.end_data", builder.GetInsertBlock()->getParent());
-
-  SetVector<Block *> blocks = getBlocksSortedByDominance(op.getRegion());
-  for (Block *bb : blocks) {
-    llvm::BasicBlock *llvmBB = moduleTranslation.lookupBlock(bb);
-    if (bb->isEntryBlock()) {
-      assert(sourceTerminator->getNumSuccessors() == 1 &&
-             "provided entry block has multiple successors");
-      sourceTerminator->setSuccessor(0, llvmBB);
-    }
-
-    if (failed(
-            moduleTranslation.convertBlock(*bb, bb->isEntryBlock(), builder))) {
-      return failure();
-    }
-
-    if (isa<acc::TerminatorOp, acc::YieldOp>(bb->getTerminator()))
-      builder.CreateBr(endDataBlock);
-  }
-
-  // Create call to end the data region.
-  builder.SetInsertPoint(endDataBlock);
-  accBuilder->emitMapperCall(builder.saveIP(), endMapperFunc, srcLocInfo,
-                             maptypesArg, mapnamesArg, mapperAllocas,
-                             kDefaultDevice, totalNbOperand);
-
-  return success();
-}
-
-/// Converts an OpenACC standalone data operation into LLVM IR.
-template <typename OpTy>
-static LogicalResult
-convertStandaloneDataOp(OpTy &op, llvm::IRBuilderBase &builder,
-                        LLVM::ModuleTranslation &moduleTranslation) {
-  auto enclosingFuncOp =
-      op.getOperation()->template getParentOfType<LLVM::LLVMFuncOp>();
-  llvm::Function *enclosingFunction =
-      moduleTranslation.lookupFunction(enclosingFuncOp.getName());
-
-  OpenACCIRBuilder *accBuilder = moduleTranslation.getOpenMPBuilder();
-
-  auto *srcLocInfo = createSourceLocationInfo(*accBuilder, op);
-  auto *mapperFunc = getAssociatedFunction(*accBuilder, op);
-
-  // Number of arguments in the enter_data operation.
-  unsigned totalNbOperand = op.getNumDataOperands();
-
-  llvm::LLVMContext &ctx = builder.getContext();
-
-  struct OpenACCIRBuilder::MapperAllocas mapperAllocas;
-  OpenACCIRBuilder::InsertPointTy allocaIP(
-      &enclosingFunction->getEntryBlock(),
-      enclosingFunction->getEntryBlock().getFirstInsertionPt());
-  accBuilder->createMapperAllocas(builder.saveIP(), allocaIP, totalNbOperand,
-                                  mapperAllocas);
-
-  SmallVector<uint64_t> flags;
-  SmallVector<llvm::Constant *> names;
-
-  if (failed(processDataOperands(builder, moduleTranslation, op, flags, names,
-                                 mapperAllocas)))
-    return failure();
-
-  llvm::GlobalVariable *maptypes =
-      accBuilder->createOffloadMaptypes(flags, ".offload_maptypes");
-  llvm::Value *maptypesArg = builder.CreateConstInBoundsGEP2_32(
-      llvm::ArrayType::get(llvm::Type::getInt64Ty(ctx), totalNbOperand),
-      maptypes, /*Idx0=*/0, /*Idx1=*/0);
-
-  llvm::GlobalVariable *mapnames =
-      accBuilder->createOffloadMapnames(names, ".offload_mapnames");
-  llvm::Value *mapnamesArg = builder.CreateConstInBoundsGEP2_32(
-      llvm::ArrayType::get(llvm::PointerType::getUnqual(ctx), totalNbOperand),
-      mapnames, /*Idx0=*/0, /*Idx1=*/0);
-
-  accBuilder->emitMapperCall(builder.saveIP(), mapperFunc, srcLocInfo,
-                             maptypesArg, mapnamesArg, mapperAllocas,
-                             kDefaultDevice, totalNbOperand);
-
-  return success();
-}
-
-namespace {
-
-/// Implementation of the dialect interface that converts operations belonging
-/// to the OpenACC dialect to LLVM IR.
-class OpenACCDialectLLVMIRTranslationInterface
-    : public LLVMTranslationDialectInterface {
-public:
-  using LLVMTranslationDialectInterface::LLVMTranslationDialectInterface;
-
-  /// Translates the given operation to LLVM IR using the provided IR builder
-  /// and saving the state in `moduleTranslation`.
-  LogicalResult
-  convertOperation(Operation *op, llvm::IRBuilderBase &builder,
-                   LLVM::ModuleTranslation &moduleTranslation) const final;
-};
-
-} // namespace
-
-/// Given an OpenACC MLIR operation, create the corresponding LLVM IR
-/// (including OpenACC runtime calls).
-LogicalResult OpenACCDialectLLVMIRTranslationInterface::convertOperation(
-    Operation *op, llvm::IRBuilderBase &builder,
-    LLVM::ModuleTranslation &moduleTranslation) const {
-
-  return llvm::TypeSwitch<Operation *, LogicalResult>(op)
-      .Case([&](acc::DataOp dataOp) {
-        return convertDataOp(dataOp, builder, moduleTranslation);
-      })
-      .Case([&](acc::EnterDataOp enterDataOp) {
-        return convertStandaloneDataOp<acc::EnterDataOp>(enterDataOp, builder,
-                                                         moduleTranslation);
-      })
-      .Case([&](acc::ExitDataOp exitDataOp) {
-        return convertStandaloneDataOp<acc::ExitDataOp>(exitDataOp, builder,
-                                                        moduleTranslation);
-      })
-      .Case([&](acc::UpdateOp updateOp) {
-        return convertStandaloneDataOp<acc::UpdateOp>(updateOp, builder,
-                                                      moduleTranslation);
-      })
-      .Case<acc::TerminatorOp, acc::YieldOp>([](auto op) {
-        // `yield` and `terminator` can be just omitted. The block structure was
-        // created in the function that handles their parent operation.
-        assert(op->getNumOperands() == 0 &&
-               "unexpected OpenACC terminator with operands");
-        return success();
-      })
-      .Case<acc::CreateOp, acc::CopyinOp, acc::CopyoutOp, acc::DeleteOp,
-            acc::UpdateDeviceOp, acc::GetDevicePtrOp>([](auto op) {
-        // NOP
-        return success();
-      })
-      .Default([&](Operation *op) {
-        return op->emitError("unsupported OpenACC operation: ")
-               << op->getName();
-      });
-}
-
-void mlir::registerOpenACCDialectTranslation(DialectRegistry &registry) {
-  registry.insert<acc::OpenACCDialect>();
-  registry.addExtension(+[](MLIRContext *ctx, acc::OpenACCDialect *dialect) {
-    dialect->addInterfaces<OpenACCDialectLLVMIRTranslationInterface>();
-  });
-}
-
-void mlir::registerOpenACCDialectTranslation(MLIRContext &context) {
-  DialectRegistry registry;
-  registerOpenACCDialectTranslation(registry);
-  context.appendDialectRegistry(registry);
-}

diff  --git a/mlir/test/Target/LLVMIR/openacc-llvm.mlir b/mlir/test/Target/LLVMIR/openacc-llvm.mlir
deleted file mode 100644
index c4b0b6a424417..0000000000000
--- a/mlir/test/Target/LLVMIR/openacc-llvm.mlir
+++ /dev/null
@@ -1,191 +0,0 @@
-// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
-
-llvm.func @testenterdataop(%arg0: !llvm.ptr, %arg1 : !llvm.ptr) {
-  %0 = acc.create varPtr(%arg0 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  %1 = acc.copyin varPtr(%arg1 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  acc.enter_data dataOperands(%0, %1 : !llvm.ptr, !llvm.ptr)
-  llvm.return
-}
-
-// CHECK: %struct.ident_t = type { i32, i32, i32, i32, ptr }
-
-// CHECK: @[[LOCSTR:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";{{.*}};testenterdataop;{{[0-9]*}};{{[0-9]*}};;\00", align 1
-// CHECK: @[[LOCGLOBAL:.*]] = private unnamed_addr constant %struct.ident_t { i32 0, i32 2, i32 0, i32 {{[0-9]*}}, ptr @[[LOCSTR]] }, align 8
-// CHECK: @[[MAPNAME1:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00", align 1
-// CHECK: @[[MAPNAME2:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00", align 1
-// CHECK: @[[MAPTYPES:.*]] = private unnamed_addr constant [{{[0-9]*}} x i64] [i64 0, i64 1]
-// CHECK: @[[MAPNAMES:.*]] = private constant [{{[0-9]*}} x ptr] [ptr @[[MAPNAME1]], ptr @[[MAPNAME2]]]
-
-// CHECK: define void @testenterdataop(ptr %[[PTR0:.*]], ptr %[[PTR1:.*]])
-// CHECK: %[[OFFLOAD_BASEPTR:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAD_PTRS:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAD_SIZES:.*]] = alloca [{{[0-9]*}} x i64]
-
-// CHECK: %[[OFFLOAD_BASEPTR_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTR]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_BASEPTR_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAD_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAD_SIZES]], i32 0, i32 0
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAD_SIZES_GEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTR_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTR]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_BASEPTR_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAD_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAD_SIZES]], i32 0, i32 1
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAD_SIZES_GEP]]
-
-
-// CHECK: %[[OFFLOAD_BASEPTR_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTR]], i32 0, i32 0
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAD_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAD_SIZES]], i32 0, i32 0
-
-// CHECK: call void @__tgt_target_data_begin_mapper(ptr @[[LOCGLOBAL]], i64 -1, i32 2, ptr %[[OFFLOAD_BASEPTR_GEP]], ptr %[[OFFLOAD_PTRS_GEP]], ptr %[[OFFLOAD_SIZES_GEP]], ptr @[[MAPTYPES]], ptr @[[MAPNAMES]], ptr null)
-
-// CHECK: declare void @__tgt_target_data_begin_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr) #0
-
-// -----
-
-
-llvm.func @testexitdataop(%arg0: !llvm.ptr, %arg1: !llvm.ptr) {
-  %arg0_devptr = acc.getdeviceptr varPtr(%arg0 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  %1 = acc.getdeviceptr varPtr(%arg1 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  acc.exit_data dataOperands(%arg0_devptr, %1 : !llvm.ptr, !llvm.ptr)
-  acc.delete accPtr(%arg0_devptr : !llvm.ptr)
-  acc.copyout accPtr(%1 : !llvm.ptr) to varPtr(%arg1 : !llvm.ptr) varType(f32)
-  llvm.return
-}
-
-// CHECK: %struct.ident_t = type { i32, i32, i32, i32, ptr }
-
-// CHECK: @[[LOCSTR:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";{{.*}};testexitdataop;{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[LOCGLOBAL:.*]] = private unnamed_addr constant %struct.ident_t { i32 0, i32 2, i32 0, i32 {{[0-9]*}}, ptr @[[LOCSTR]] }
-// CHECK: @[[MAPNAME1:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[MAPNAME2:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[MAPTYPES:.*]] = private unnamed_addr constant [{{[0-9]*}} x i64] [i64 8, i64 2]
-// CHECK: @[[MAPNAMES:.*]] = private constant [{{[0-9]*}} x ptr] [ptr @[[MAPNAME1]], ptr @[[MAPNAME2]]]
-
-// CHECK: define void @testexitdataop(ptr %[[PTR0:.*]], ptr %[[PTR1:.*]])
-// CHECK: %[[OFFLOAD_BASEPTRS:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAD_PTRS:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAS_SIZES:.*]] = alloca [{{[0-9]*}} x i64]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_BASEPTRS_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAS_SIZES_GEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_BASEPTRS_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 1
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAS_SIZES_GEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-
-// CHECK: call void @__tgt_target_data_end_mapper(ptr @[[LOCGLOBAL]], i64 -1, i32 2, ptr %[[OFFLOAD_BASEPTRS_GEP]], ptr %[[OFFLOAD_PTRS_GEP]], ptr %[[OFFLOAS_SIZES_GEP]], ptr @[[MAPTYPES]], ptr @[[MAPNAMES]], ptr null)
-
-// CHECK: declare void @__tgt_target_data_end_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr) #0
-
-// -----
-
-llvm.func @testupdateop(%arg1: !llvm.ptr) {
-  %0 = acc.update_device varPtr(%arg1 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  acc.update dataOperands(%0 : !llvm.ptr)
-  llvm.return
-}
-
-// CHECK: %struct.ident_t = type { i32, i32, i32, i32, ptr }
-
-// CHECK: [[LOCSTR:@.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";{{.*}};testupdateop;{{[0-9]*}};{{[0-9]*}};;\00", align 1
-// CHECK: [[LOCGLOBAL:@.*]] = private unnamed_addr constant %struct.ident_t { i32 0, i32 2, i32 0, i32 {{[0-9]*}}, ptr [[LOCSTR]] }, align 8
-// CHECK: [[MAPNAME1:@.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00", align 1
-// CHECK: [[MAPTYPES:@.*]] = private unnamed_addr constant [{{[0-9]*}} x i64] [i64 1]
-// CHECK: [[MAPNAMES:@.*]] = private constant [{{[0-9]*}} x ptr] [ptr [[MAPNAME1]]]
-
-// CHECK: define void @testupdateop(ptr %[[SIMPLEPTR:.*]])
-// CHECK: %[[OFFLOAD_BASEPTRS:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAD_PTRS:.*]] = alloca [{{[0-9]*}} x ptr]
-// CHECK: %[[OFFLOAS_SIZES:.*]] = alloca [{{[0-9]*}} x i64]
-
-// CHECK: %[[BASEGEP:.*]] = getelementptr inbounds [1 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: store ptr %[[SIMPLEPTR]], ptr %[[BASEGEP]]
-// CHECK: %[[ARGGEP:.*]] = getelementptr inbounds [1 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: store ptr %[[SIMPLEPTR]], ptr %[[ARGGEP]]
-// CHECK: %[[SIZEGEP:.*]] = getelementptr inbounds [1 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[SIZEGEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [1 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [1 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [1 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-
-// CHECK: call void @__tgt_target_data_update_mapper(ptr [[LOCGLOBAL]], i64 -1, i32 1, ptr %[[OFFLOAD_BASEPTRS_GEP]], ptr %[[OFFLOAD_PTRS_GEP]], ptr %[[OFFLOAS_SIZES_GEP]], ptr [[MAPTYPES]], ptr [[MAPNAMES]], ptr null)
-
-// CHECK: declare void @__tgt_target_data_update_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr) #0
-
-// -----
-
-llvm.func @testdataop(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: !llvm.ptr) {
-  
-  %0 = acc.copyin varPtr(%arg0 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  %1 = acc.create varPtr(%arg1 : !llvm.ptr) varType(f32) -> !llvm.ptr
-  acc.data dataOperands(%0, %1 : !llvm.ptr, !llvm.ptr) {
-    %9 = llvm.mlir.constant(2 : i32) : i32
-    llvm.store %9, %arg2 : i32, !llvm.ptr
-    acc.terminator
-  }
-  acc.copyout accPtr(%0 : !llvm.ptr) to varPtr(%arg0 : !llvm.ptr) varType(f32)
-  acc.copyout accPtr(%1 : !llvm.ptr) to varPtr(%arg1 : !llvm.ptr) varType(f32)
-  llvm.return
-}
-
-// CHECK: %struct.ident_t = type { i32, i32, i32, i32, ptr }
-// CHECK: @[[LOCSTR:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";{{.*}};testdataop;{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[LOCGLOBAL:.*]] = private unnamed_addr constant %struct.ident_t { i32 0, i32 2, i32 0, i32 {{[0-9]*}}, ptr @[[LOCSTR]] }
-// CHECK: @[[MAPNAME1:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[MAPNAME2:.*]] = private unnamed_addr constant [{{[0-9]*}} x i8] c";unknown;{{.*}};{{[0-9]*}};{{[0-9]*}};;\00"
-// CHECK: @[[MAPTYPES:.*]] = private unnamed_addr constant [{{[0-9]*}} x i64] [i64 8193, i64 8192]
-// CHECK: @[[MAPNAMES:.*]] = private constant [{{[0-9]*}} x ptr] [ptr @[[MAPNAME1]], ptr @[[MAPNAME2]]]
-
-// CHECK: define void @testdataop(ptr %[[PTR0:.*]], ptr %[[PTR1:.*]], ptr %[[PTR2:.*]])
-// CHECK: %[[OFFLOAD_BASEPTRS:.*]] = alloca [{{[0-9]*}} x ptr], align 8
-// CHECK: %[[OFFLOAD_PTRS:.*]] = alloca [{{[0-9]*}} x ptr], align 8
-// CHECK: %[[OFFLOAS_SIZES:.*]] = alloca [{{[0-9]*}} x i64], align 8
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_BASEPTRS_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: store ptr %[[PTR0]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAS_SIZES_GEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_BASEPTRS_GEP]]
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 1
-// CHECK: store ptr %[[PTR1]], ptr %[[OFFLOAD_PTRS_GEP]]
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 1
-// CHECK: store i64 ptrtoint (ptr getelementptr (ptr, ptr null, i32 1) to i64), ptr %[[OFFLOAS_SIZES_GEP]]
-
-// CHECK: %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK: %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-// CHECK: call void @__tgt_target_data_begin_mapper(ptr @[[LOCGLOBAL]], i64 -1, i32 2, ptr %[[OFFLOAD_BASEPTRS_GEP]], ptr %[[OFFLOAD_PTRS_GEP]], ptr %[[OFFLOAS_SIZES_GEP]], ptr @[[MAPTYPES]], ptr @[[MAPNAMES]], ptr null)
-// CHECK: br label %acc.data
-
-// CHECK:      acc.data:
-// CHECK-NEXT:   store i32 2, ptr %{{.*}}
-// CHECK-NEXT:   br label %acc.end_data
-
-// CHECK: acc.end_data:
-// CHECK:   %[[OFFLOAD_BASEPTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_BASEPTRS]], i32 0, i32 0
-// CHECK:   %[[OFFLOAD_PTRS_GEP:.*]] = getelementptr inbounds [2 x ptr], ptr %[[OFFLOAD_PTRS]], i32 0, i32 0
-// CHECK:   %[[OFFLOAS_SIZES_GEP:.*]] = getelementptr inbounds [2 x i64], ptr %[[OFFLOAS_SIZES]], i32 0, i32 0
-// CHECK:   call void @__tgt_target_data_end_mapper(ptr @[[LOCGLOBAL]], i64 -1, i32 2, ptr %[[OFFLOAD_BASEPTRS_GEP]], ptr %[[OFFLOAD_PTRS_GEP]], ptr %[[OFFLOAS_SIZES_GEP]], ptr @[[MAPTYPES]], ptr @[[MAPNAMES]], ptr null)
-
-// CHECK: declare void @__tgt_target_data_begin_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr)
-// CHECK: declare void @__tgt_target_data_end_mapper(ptr, i64, i32, ptr, ptr, ptr, ptr, ptr, ptr)

diff  --git a/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Support/BUILD.bazel b/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Support/BUILD.bazel
index 831bdd09f9549..c40b59926ef67 100644
--- a/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Support/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Support/BUILD.bazel
@@ -52,7 +52,6 @@ cc_library(
         "//mlir:IR",
         "//mlir:LLVMDialect",
         "//mlir:LLVMToLLVMIRTranslation",
-        "//mlir:OpenACCToLLVMIRTranslation",
         "//mlir:OpenACCTransforms",
         "//mlir:OpenMPToLLVMIRTranslation",
         "//mlir:OpenMPTransforms",

diff  --git a/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Transforms/BUILD.bazel b/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Transforms/BUILD.bazel
index ecde032f32fb2..d6d6224d28387 100644
--- a/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Transforms/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/flang/lib/Optimizer/Transforms/BUILD.bazel
@@ -77,7 +77,6 @@ cc_library(
         "//mlir:MemRefDialect",
         "//mlir:NVVMDialect",
         "//mlir:OpenACCDialect",
-        "//mlir:OpenACCToLLVMIRTranslation",
         "//mlir:OpenMPDialect",
         "//mlir:Pass",
         "//mlir:SCFDialect",

diff  --git a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
index 74771db2af525..96e2b9b466ecc 100644
--- a/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
+++ b/utils/bazel/llvm-project-overlay/mlir/BUILD.bazel
@@ -9380,27 +9380,6 @@ cc_library(
     ],
 )
 
-cc_library(
-    name = "OpenACCToLLVMIRTranslation",
-    srcs = glob(["lib/Target/LLVMIR/Dialect/OpenACC/*.cpp"]),
-    hdrs = glob(["include/mlir/Target/LLVMIR/Dialect/OpenACC/*.h"]),
-    includes = ["include"],
-    deps = [
-        ":Analysis",
-        ":IR",
-        ":LLVMDialect",
-        ":OpenACCDialect",
-        ":OpenMPCommon",
-        ":Support",
-        ":ToLLVMIRTranslation",
-        ":TransformUtils",
-        "//llvm:Core",
-        "//llvm:FrontendOpenMP",
-        "//llvm:Support",
-        "//llvm:TargetParser",
-    ],
-)
-
 cc_library(
     name = "OpenMPToLLVMIRTranslation",
     srcs = glob(["lib/Target/LLVMIR/Dialect/OpenMP/*.cpp"]),
@@ -9471,7 +9450,6 @@ cc_library(
         ":LLVMIRToNVVMTranslation",
         ":LLVMToLLVMIRTranslation",
         ":NVVMToLLVMIRTranslation",
-        ":OpenACCToLLVMIRTranslation",
         ":OpenMPToLLVMIRTranslation",
         ":PtrToLLVMIRTranslation",
         ":ROCDLToLLVMIRTranslation",
@@ -10071,7 +10049,6 @@ cc_library(
         ":IR",
         ":LLVMDialect",
         ":LLVMToLLVMIRTranslation",
-        ":OpenACCToLLVMIRTranslation",
         ":OpenMPToLLVMIRTranslation",
         ":ParseUtilities",
         ":Parser",
@@ -10266,7 +10243,6 @@ cc_binary(
         ":LLVMDialect",
         ":LLVMToLLVMIRTranslation",
         ":MlirJitRunner",
-        ":OpenACCToLLVMIRTranslation",
         ":OpenMPToLLVMIRTranslation",
         ":ToLLVMIRTranslation",
         "//llvm:AsmParser",


        

From llvmlistbot at llvm.org  Tue Jul 14 09:52:36 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 09:52:36 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [mlir][openacc] Remove
 OpenACCToLLVMIRTranslation (PR #209295)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209295@github.com>
Message-ID: <6a566954.6537ae93.2f1c1b.7087@mx.google.com>


https://github.com/clementval closed https://github.com/llvm/llvm-project/pull/209295

From llvmlistbot at llvm.org  Tue Jul 14 09:53:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 09:53:07 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a566973.c5504292.28b3b8.28b3@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 178937 tests passed
* 3540 tests skipped
* 2 tests failed

## Failed Tests
(click on a test name to see its output)

### lldb-api
<details>
<summary>lldb-api.lang/cpp/std-function-step-into-callable/TestStdFunctionStepIntoCallable.py</summary>

```
Script:
--
/usr/bin/python3 /home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/test/API/dotest.py -u CXXFLAGS -u CFLAGS --env LLVM_LIBS_DIR=/home/gha/actions-runner/_work/llvm-project/llvm-project/build/./lib --env LLVM_INCLUDE_DIR=/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include --env LLVM_TOOLS_DIR=/home/gha/actions-runner/_work/llvm-project/llvm-project/build/./bin --libcxx-include-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/include/c++/v1 --libcxx-include-target-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/include/x86_64-unknown-linux-gnu/c++/v1 --libcxx-library-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./lib/x86_64-unknown-linux-gnu --triple x86_64-unknown-linux-gnu --build-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lldb-test-build --lldb-module-cache-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lldb-test-build/module-cache-lldb/lldb-api --clang-module-cache-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lldb-test-build/module-cache-clang/lldb-api --executable /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./bin/lldb --lldb-python-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/local/lib/python3.12/dist-packages --compiler /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./bin/clang --dsymutil /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./bin/dsymutil --make /usr/bin/gmake --llvm-tools-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./bin --lldb-obj-root /home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/lldb --lldb-libs-dir /home/gha/actions-runner/_work/llvm-project/llvm-project/build/./lib --cmake-build-type Release /home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/test/API/lang/cpp/std-function-step-into-callable -p TestStdFunctionStepIntoCallable.py
--
Exit Code: 1

Command Output (stdout):
--
Skipping the following test categories: msvcstl, dsym, pdb, gmodules, debugserver, objc

--
Command Output (stderr):
--
FAIL: LLDB (/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/clang-x86_64) :: test (TestStdFunctionStepIntoCallable.LibCxxFunctionSteppingIntoCallableTestCase.test)
Log Files:
 - /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lldb-test-build/lang/cpp/std-function-step-into-callable/TestStdFunctionStepIntoCallable/Failure_test.log
======================================================================
FAIL: test (TestStdFunctionStepIntoCallable.LibCxxFunctionSteppingIntoCallableTestCase.test)
   Test that std::function as defined by libc++ is correctly printed by LLDB
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/test/API/lang/cpp/std-function-step-into-callable/TestStdFunctionStepIntoCallable.py", line 54, in test
    self.assertEqual(
AssertionError: 32 != 4
Config=x86_64-/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/clang
----------------------------------------------------------------------
Ran 1 test in 0.640s

FAILED (failures=1)

--

```
</details>

### Clang
<details>
<summary>Clang.CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/clang -cc1 -internal-isystem /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lib/clang/24/include -nostdsysteminc -fno-ident -no-enable-noundef-analysis /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple "spir-unknown-unknown" -fdenormal-fp-math-f32=preserve-sign -cl-uniform-work-group-size | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck --check-prefix=SPIR32 /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/clang -cc1 -internal-isystem /home/gha/actions-runner/_work/llvm-project/llvm-project/build/lib/clang/24/include -nostdsysteminc -fno-ident -no-enable-noundef-analysis /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl -cl-std=CL2.0 -ffake-address-space-map -O0 -emit-llvm -o - -triple spir-unknown-unknown -fdenormal-fp-math-f32=preserve-sign -cl-uniform-work-group-size
# note: command had no output on stdout or stderr
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck --check-prefix=SPIR32 /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl:30:17: error: SPIR32-NEXT: expected string not found in input
# | // SPIR32-NEXT: [[DEFAULT_QUEUE:%.*]] = alloca ptr, align 4
# |                 ^
# | <stdin>:13:31: note: scanning from here
# |  %i.addr = alloca i32, align 4
# |                               ^
# | <stdin>:13:31: note: pattern attempts to capture variables: "DEFAULT_QUEUE"
# |  %i.addr = alloca i32, align 4
# |                               ^
# | <stdin>:29:4: note: possible intended match here
# |  %i.addr = alloca i32, align 4
# |    ^
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl:70:17: error: SPIR32-NEXT: expected string not found in input
# | // SPIR32-NEXT: [[BLOCK_CAPTURE_ADDR:%.*]] = getelementptr inbounds <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) [[DOTBLOCK_DESCRIPTOR]], i32 0, i32 5
# |                 ^
# | <stdin>:71:69: note: scanning from here
# |  store ptr addrspace(4) %.block_descriptor, ptr %block.addr, align 4
# |                                                                     ^
# | <stdin>:71:69: note: with "DOTBLOCK_DESCRIPTOR" equal to "%.block_descriptor"
# |  store ptr addrspace(4) %.block_descriptor, ptr %block.addr, align 4
# |                                                                     ^
# | <stdin>:71:69: note: pattern attempts to capture variables: "BLOCK_CAPTURE_ADDR"
# |  store ptr addrspace(4) %.block_descriptor, ptr %block.addr, align 4
# |                                                                     ^
# | <stdin>:72:14: note: possible intended match here
# |  %block.capture.addr = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) %.block_descriptor, i32 0, i32 5
# |              ^
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl:165:12: error: SPIR32: expected string not found in input
# | // SPIR32: attributes #[[ATTR0]] = { convergent noinline norecurse nounwind optnone "denormal-fp-math-f32"="preserve-sign,preserve-sign" "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size"="true" }
# |            ^
# | <stdin>:95:10: note: scanning from here
# |  ret void
# |          ^
# | <stdin>:95:10: note: with "ATTR0" equal to "0"
# |  ret void
# |          ^
# | <stdin>:100:1: note: possible intended match here
# | attributes #0 = { convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size" }
# | ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |              .
# |              .
# |              .
# |              8: ; Function Attrs: convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign) 
# |              9: define dso_local spir_kernel void @device_side_enqueue(ptr addrspace(1) align 4 %a, ptr addrspace(1) align 4 %b, i32 %i) #0 !kernel_arg_addr_space !1 !kernel_arg_access_qual !2 !kernel_arg_type !3 !kernel_arg_base_type !3 !kernel_arg_type_qual !4 { 
# |             10: entry: 
# |             11:  %a.addr = alloca ptr addrspace(1), align 4 
# |             12:  %b.addr = alloca ptr addrspace(1), align 4 
# |             13:  %i.addr = alloca i32, align 4 
# | next:30'0                                    {   search range start (exclusive)
# | next:30'1                                        error: no match found in search range
# | next:30'2                                        pattern attempts to capture variables: "DEFAULT_QUEUE"
# |             14:  store ptr addrspace(1) %a, ptr %a.addr, align 4 
# |             15:  store ptr addrspace(1) %b, ptr %b.addr, align 4 
# |             16:  store i32 %i, ptr %i.addr, align 4 
# |             17:  %0 = load ptr addrspace(1), ptr %a.addr, align 4 
# |             18:  %1 = load ptr addrspace(1), ptr %b.addr, align 4 
# |              .
# |              .
# |              .
# |             24: ; Function Attrs: convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign) 
# |             25: define dso_local spir_func void @__clang_ocl_kern_imp_device_side_enqueue(ptr addrspace(1) align 4 %a, ptr addrspace(1) align 4 %b, i32 %i) #0 !kernel_arg_addr_space !1 !kernel_arg_access_qual !2 !kernel_arg_type !3 !kernel_arg_base_type !3 !kernel_arg_type_qual !4 { 
# |             26: entry: 
# |             27:  %a.addr = alloca ptr addrspace(1), align 4 
# |             28:  %b.addr = alloca ptr addrspace(1), align 4 
# |             29:  %i.addr = alloca i32, align 4 
# | next:30'3          ?                             possible intended match
# |             30:  %default_queue = alloca target("spirv.Queue"), align 4 
# |             31:  %flags = alloca i32, align 4 
# |             32:  %ndrange = alloca %struct.ndrange_t, align 4 
# |             33:  %tmp = alloca %struct.ndrange_t, align 4 
# |             34:  %block = alloca <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, align 4 
# |              .
# |              .
# |              .
# |             61:  
# |             62: ; Function Attrs: nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) 
# |             63: declare void @llvm.memcpy.p0.p0.i32(ptr noalias writeonly captures(none), ptr noalias readonly captures(none), i32, i1 immarg) #1 
# |             64:  
# |             65: ; Function Attrs: convergent noinline nounwind optnone denormal_fpenv(float: preservesign) 
# |             66: define internal spir_func void @__device_side_enqueue_block_invoke(ptr addrspace(4) %.block_descriptor) #2 { 
# | next:30'4                                                                          }                                           search range end (exclusive)
# |             67: entry: 
# |             68:  %.block_descriptor.addr = alloca ptr addrspace(4), align 4 
# |             69:  %block.addr = alloca ptr addrspace(4), align 4 
# |             70:  store ptr addrspace(4) %.block_descriptor, ptr %.block_descriptor.addr, align 4 
# |             71:  store ptr addrspace(4) %.block_descriptor, ptr %block.addr, align 4 
# | next:70'0                                                                          {   search range start (exclusive)
# | next:70'1                                                                              error: no match found in search range
# | next:70'2                                                                              with "DOTBLOCK_DESCRIPTOR" equal to "%.block_descriptor"
# | next:70'3                                                                              pattern attempts to capture variables: "BLOCK_CAPTURE_ADDR"
# |             72:  %block.capture.addr = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) %.block_descriptor, i32 0, i32 5 
# | next:70'4                    ?                                                                                                                                                                   possible intended match
# |             73:  %0 = load ptr addrspace(1), ptr addrspace(4) %block.capture.addr, align 4 
# |             74:  %block.capture.addr1 = getelementptr inbounds nuw <{ i32, i32, ptr addrspace(4), ptr addrspace(1), i32, ptr addrspace(1) }>, ptr addrspace(4) %.block_descriptor, i32 0, i32 4 
# |             75:  %1 = load i32, ptr addrspace(4) %block.capture.addr1, align 4 
# |             76:  %arrayidx = getelementptr inbounds float, ptr addrspace(1) %0, i32 %1 
# |             77:  %2 = load float, ptr addrspace(1) %arrayidx, align 4 
# |              .
# |              .
# |              .
# |             87:  
# |             88: ; Function Attrs: nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) 
# |             89: declare float @llvm.fmuladd.f32(float, float, float) #3 
# |             90:  
# |             91: ; Function Attrs: convergent nounwind denormal_fpenv(float: preservesign) 
# |             92: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(ptr addrspace(4) %0) #4 { 
# | next:70'5                                                                          }                           search range end (exclusive)
# |             93: entry: 
# |             94:  call spir_func void @__device_side_enqueue_block_invoke(ptr addrspace(4) %0) 
# |             95:  ret void 
# | check:165'0             {   search range start (exclusive)
# | check:165'1                 error: no match found in search range
# | check:165'2                 with "ATTR0" equal to "0"
# |             96: } 
# |             97:  
# |             98: declare spir_func i32 @__enqueue_kernel_basic(target("spirv.Queue"), i32, ptr, ptr addrspace(4), ptr addrspace(4)) 
# |             99:  
# |            100: attributes #0 = { convergent noinline norecurse nounwind optnone denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size" } 
# | check:165'3     ?                                                                                                                                                                                              possible intended match
# |            101: attributes #1 = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) } 
# |            102: attributes #2 = { convergent noinline nounwind optnone denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" "uniform-work-group-size" } 
# |            103: attributes #3 = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) } 
# |            104: attributes #4 = { convergent nounwind denormal_fpenv(float: preservesign) "no-trapping-math"="true" "stack-protector-buffer-size"="8" } 
# |            105: attributes #5 = { convergent nounwind "uniform-work-group-size" } 
# |            106:  
# |            107: !opencl.ocl.version = !{!0} 
# |            108: !opencl.spir.version = !{!0} 
# |            109:  
# |            110: !0 = !{i32 2, i32 0} 
# |            111: !1 = !{i32 1, i32 1, i32 0} 
# |            112: !2 = !{!"none", !"none", !"none"} 
# |            113: !3 = !{!"float*", !"float*", !
# | ...
# `---data was truncated (10240/10363) (change limit with -D output_limit=N)
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/209465

From llvmlistbot at llvm.org  Tue Jul 14 09:54:23 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Tue, 14 Jul 2026 09:54:23 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Lower target
	in_reduction (PR #199967)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199967@github.com>
Message-ID: <6a5669bf.f1c1923d.991d1.f6b1@mx.google.com>


https://github.com/Saieiei updated https://github.com/llvm/llvm-project/pull/199967

>From cdb40663477c09fbefd5eff8e14893eccc7e240b Mon Sep 17 00:00:00 2001
From: Sairudra More <sairudra60 at gmail.com>
Date: Fri, 12 Jun 2026 04:57:07 -0500
Subject: [PATCH] [flang][OpenMP] Lower target in_reduction

Enable host lowering for target in_reduction in Flang and MLIR OpenMP
translation.

Model target in_reduction through the matching map entry, force
address-preserving implicit mapping for Flang in_reduction list items, and emit
the host-side task-reduction lookup with __kmpc_task_reduction_get_th_data. The
runtime entry point takes and returns a generic, default-address-space pointer,
so normalize a non-default-address-space captured pointer to the generic address
space before the call and cast the returned private pointer back to the map
block argument's address space, mirroring the in_reduction handling on
omp.taskloop. On the target device, in_reduction is handled as a regular
map(tofrom) variable. The byref modifier, two-argument initializers, cleanup
regions, and the remaining Flang COMMON/EQUIVALENCE/privatized-variable cases
continue to be diagnosed.

Add Flang lowering, MLIR verifier/translation, and LLVM IR tests for the
supported host path, including a non-default-address-space case, and the
remaining unsupported cases.
---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 103 ++++++++++--
 .../OpenMP/target-inreduction-llvmir.f90      |  50 ++++++
 .../OpenMP/Todo/target-inreduction-common.f90 |  20 +++
 .../Todo/target-inreduction-equivalence.f90   |  19 +++
 .../Todo/target-inreduction-firstprivate.f90  |  19 +++
 .../Lower/OpenMP/Todo/target-inreduction.f90  |  15 --
 .../OpenMP/target-inreduction-unused.f90      |  27 ++++
 .../test/Lower/OpenMP/target-inreduction.f90  |  30 ++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  33 ++++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  50 ++++++
 .../mlir/Dialect/OpenMP/OpenMPClauses.td      |  14 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  22 ++-
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 149 +++++++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 138 ++++++++++++++--
 .../Dialect/OpenMP/host-op-filtering.mlir     |   8 +-
 mlir/test/Dialect/OpenMP/invalid.mlir         |  85 ++++++++++
 .../openmp-target-in-reduction-device.mlir    |  51 ++++++
 .../openmp-target-in-reduction-multi.mlir     |  77 +++++++++
 .../LLVMIR/openmp-target-in-reduction.mlir    | 107 +++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  91 ++++++++++-
 20 files changed, 1029 insertions(+), 79 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/target-inreduction.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction-unused.f90
 create mode 100644 flang/test/Lower/OpenMP/target-inreduction.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ed3e989daab7e..8a7dde1063ea1 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2474,6 +2474,7 @@ genTargetClauses(lower::AbstractConverter &converter,
                  mlir::omp::TargetExtOperands &clauseOps,
                  DefaultMapsTy &defaultMaps,
                  llvm::SmallVectorImpl<Object> &hasDeviceAddrObjects,
+                 llvm::SmallVectorImpl<Object> &inReductionObjects,
                  llvm::SmallVectorImpl<Object> &isDevicePtrObjects,
                  llvm::SmallVectorImpl<Object> &mapObjects) {
   ClauseProcessor cp(converter, semaCtx, clauses);
@@ -2489,12 +2490,33 @@ genTargetClauses(lower::AbstractConverter &converter,
     hostEvalInfo->collectValues(clauseOps.hostEvalVars);
   }
   cp.processIf(llvm::omp::Directive::OMPD_target, clauseOps);
+  cp.processInReduction(loc, clauseOps, inReductionObjects);
+  // The host lowering of `target in_reduction` redirects each list item's
+  // dedicated implicit map entry to the per-task reduction-private storage
+  // returned by the runtime. A COMMON block member is accessed in the target
+  // body through the COMMON block storage map, not through its own member map
+  // entry, so the redirect would rewrite an unused map argument and the body
+  // would accumulate into the mapped original rather than the private copy.
+  // An EQUIVALENCE-associated variable has the same problem: it shares storage
+  // with another variable that may be mapped separately (e.g. `map(tofrom: j)`
+  // with `equivalence(i, j)`), so the body updates the aliased storage while
+  // the redirected runtime-private copy stays dead.
+  // Reject both cases until the redirect can target the storage map.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      if (semantics::FindCommonBlockContaining(sym->GetUltimate()))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a COMMON block member");
+      if (semantics::FindEquivalenceSet(sym->GetUltimate()))
+        TODO(loc, "TARGET construct with IN_REDUCTION of an EQUIVALENCE "
+                  "storage-associated variable");
+    }
   cp.processIsDevicePtr(stmtCtx, clauseOps, isDevicePtrObjects);
   cp.processMap(loc, stmtCtx, clauseOps, llvm::omp::Directive::OMPD_unknown,
                 &mapObjects);
   cp.processNowait(clauseOps);
   cp.processThreadLimit(stmtCtx, clauseOps);
-  cp.processTODO<clause::Allocate, clause::InReduction, clause::UsesAllocators>(
+  cp.processTODO<clause::Allocate, clause::UsesAllocators>(
       loc, llvm::omp::Directive::OMPD_target);
 
   // `target private(..)` is only supported in delayed privatization mode.
@@ -3570,10 +3592,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   mlir::omp::TargetExtOperands clauseOps;
   DefaultMapsTy defaultMaps;
   llvm::SmallVector<Object> mapObjects, hasDeviceAddrObjects,
-      isDevicePtrObjects;
+      inReductionObjects, isDevicePtrObjects;
   genTargetClauses(converter, semaCtx, symTable, stmtCtx, eval, item->clauses,
                    loc, clauseOps, defaultMaps, hasDeviceAddrObjects,
-                   isDevicePtrObjects, mapObjects);
+                   inReductionObjects, isDevicePtrObjects, mapObjects);
 
   KernelTypeVisitor visitor(semaCtx, converter.getModuleOp());
   clauseOps.kernelType = mlir::omp::TargetExecModeAttr::get(
@@ -3621,6 +3643,26 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
                            /*isTargetPrivitization=*/true);
   dsp.processStep1(&clauseOps);
 
+  // The host lowering of `target in_reduction` requires each list item to be
+  // implicitly mapped as an address-preserving entry whose map argument the
+  // redirect rewrites to the runtime reduction-private storage. When the same
+  // variable is also privatized on the target (implicitly, as with a
+  // host-associated scalar that defaults to firstprivate, or explicitly with
+  // `firstprivate(i) in_reduction(+:i)`), the force-map is suppressed because
+  // the symbol is already in the privatization set, and the firstprivate copy
+  // region then has no mold value for it, aborting translation. Reject until
+  // in_reduction can be reconciled with target privatization.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym()) {
+      const semantics::Symbol &ultimate = sym->GetUltimate();
+      if (llvm::any_of(dsp.getAllSymbolsToPrivatize(),
+                       [&](const semantics::Symbol *priv) {
+                         return priv->GetUltimate() == ultimate;
+                       }))
+        TODO(loc,
+             "TARGET construct with IN_REDUCTION of a privatized variable");
+    }
+
   // Collect symbols that have dynamic substring accesses
   llvm::SmallPtrSet<const semantics::Symbol *, 8> symbolsWithDynamicSubstring;
   collectSymbolsWithDynamicSubstring(semaCtx, eval,
@@ -3631,7 +3673,16 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   // symbols used inside the region that do not have explicit data-environment
   // attribute clauses (neither data-sharing; e.g. `private`, nor `map`
   // clauses).
-  auto captureImplicitMap = [&](const semantics::Symbol &sym) {
+  //
+  // When `forceAddressPreserving` is set, the symbol is force-mapped as an
+  // address-preserving `capture(ByRef)` with implicit `tofrom` flags,
+  // bypassing the scalar default capture rules. This is used for `target
+  // in_reduction` list items, whose mapped pointer is passed as the `orig`
+  // argument of `__kmpc_task_reduction_get_th_data`; a ByCopy scalar capture
+  // would break the runtime lookup against the enclosing taskgroup's
+  // task_reduction descriptor.
+  auto captureImplicitMap = [&](const semantics::Symbol &sym,
+                                bool forceAddressPreserving = false) {
     // Structure component symbols don't have bindings, and can only be
     // explicitly mapped individually. If a member is captured implicitly
     // we map the entirety of the derived type when we find its symbol.
@@ -3640,12 +3691,14 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
     // if the symbol is part of an already mapped common block, do not make a
     // map for it.
-    if (const Fortran::semantics::Symbol *common =
-            Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
-      if (llvm::any_of(mapObjects, [=](const Object &object) {
-            return object.sym() == common;
-          }))
-        return;
+    if (!forceAddressPreserving) {
+      if (const Fortran::semantics::Symbol *common =
+              Fortran::semantics::FindCommonBlockContaining(sym.GetUltimate()))
+        if (llvm::any_of(mapObjects, [=](const Object &object) {
+              return object.sym() == common;
+            }))
+          return;
+    }
 
     // If we come across a symbol without a symbol address, we
     // return as we cannot process it, this is intended as a
@@ -3695,13 +3748,21 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
         eleType = refType.getElementType();
 
       std::pair<mlir::omp::ClauseMapFlags, mlir::omp::VariableCaptureKind>
-          mapFlagAndKind = getImplicitMapTypeAndKind(
-              firOpBuilder, converter, defaultMaps, eleType, loc, sym);
+          mapFlagAndKind;
+      if (forceAddressPreserving)
+        mapFlagAndKind = {mlir::omp::ClauseMapFlags::implicit |
+                              mlir::omp::ClauseMapFlags::to |
+                              mlir::omp::ClauseMapFlags::from,
+                          mlir::omp::VariableCaptureKind::ByRef};
+      else
+        mapFlagAndKind = getImplicitMapTypeAndKind(
+            firOpBuilder, converter, defaultMaps, eleType, loc, sym);
 
       mlir::FlatSymbolRefAttr mapperId;
       auto defaultmapBehaviour = getDefaultmapIfPresent(defaultMaps, eleType);
-      if (defaultmapBehaviour ==
-          clause::Defaultmap::ImplicitBehavior::Default) {
+      if (!forceAddressPreserving &&
+          defaultmapBehaviour ==
+              clause::Defaultmap::ImplicitBehavior::Default) {
         const semantics::DerivedTypeSpec *typeSpec =
             sym.GetType() ? sym.GetType()->AsDerived() : nullptr;
         if (typeSpec) {
@@ -3755,6 +3816,15 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
           Object{const_cast<semantics::Symbol *>(&sym), std::nullopt});
     }
   };
+  // OpenMP requires `in_reduction` list items on `target` to be implicitly
+  // data-mapped. Force-map them as address-preserving captures before the
+  // generic implicit-map walk so that walk treats the symbols as already
+  // mapped via `isDuplicateMappedSymbol` and does not downgrade them to
+  // ByCopy.
+  for (const Object &object : inReductionObjects)
+    if (const semantics::Symbol *sym = object.sym())
+      captureImplicitMap(*sym, /*forceAddressPreserving=*/true);
+
   lower::pft::visitAllSymbols(eval, captureImplicitMap);
 
   auto targetOp = mlir::omp::TargetOp::create(firOpBuilder, loc, clauseOps);
@@ -3767,7 +3837,10 @@ genTargetOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   args.hasDeviceAddr.objects = hasDeviceAddrObjects;
   args.hasDeviceAddr.vars = hasDeviceAddrBaseValues;
   args.hostEvalVars = clauseOps.hostEvalVars;
-  // TODO: Add in_reduction syms and vars.
+  // `in_reduction` list items do not get their own entry block argument on
+  // `omp.target`; they are implicitly mapped (see the force-map above) and the
+  // target body accesses them through their `map_entries` block argument. The
+  // `in_reduction` operands remain on the op as host-side metadata.
   args.map.objects = mapObjects;
   args.map.vars = mapBaseValues;
   args.priv.objects = makeObjects(dsp.getDelayedPrivSymbols());
diff --git a/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90 b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
new file mode 100644
index 0000000000000..1898e6605f7f8
--- /dev/null
+++ b/flang/test/Integration/OpenMP/target-inreduction-llvmir.f90
@@ -0,0 +1,50 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+! RUN: %flang_fc1 -emit-llvm -fopenmp -fopenmp-version=50 -o - %s | FileCheck %s
+
+! End-to-end coverage: Flang lowers a taskgroup task_reduction enclosing a
+! target in_reduction all the way to LLVM IR. Verifies that the host path
+! emits __kmpc_taskred_init for the enclosing taskgroup and
+! __kmpc_task_reduction_get_th_data for the target's in_reduction lookup,
+! with load/store through the returned private pointer and no direct update
+! through the original shared pointer.
+
+subroutine target_in_reduction_e2e()
+  integer :: i
+  i = 0
+  !$omp taskgroup task_reduction(+:i)
+    !$omp target in_reduction(+:i)
+    i = i + 1
+    !$omp end target
+  !$omp end taskgroup
+end subroutine target_in_reduction_e2e
+
+! CHECK-LABEL: define void @target_in_reduction_e2e_()
+! The enclosing taskgroup emits __kmpc_taskred_init to register the
+! task_reduction descriptor.
+! CHECK:         call ptr @__kmpc_taskred_init(i32 %{{.+}}, i32 1, ptr %{{.+}})
+
+! The host stub calls the outlined target body passing the captured pointer.
+! CHECK:         call void @__omp_offloading_{{.*}}_target_in_reduction_e2e_{{.*}}(ptr %{{.+}}, ptr null)
+
+! Inside the outlined target body, the in_reduction private pointer is
+! obtained from the runtime using the captured original pointer with a NULL
+! descriptor (the runtime walks enclosing taskgroups). All loads and stores
+! go through the returned private pointer.
+! CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_in_reduction_e2e_
+! CHECK-SAME:    (ptr %[[ORIG:.+]], ptr %{{.+}})
+! CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+! CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[ORIG]])
+! CHECK:         %[[VAL:.+]] = load i32, ptr %[[PRIV]]
+! CHECK:         %[[SUM:.+]] = add i32 %[[VAL]], 1
+! CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+! The outlined body must not store directly through the captured original
+! pointer; all updates go through the runtime-returned private copy.
+! CHECK-NOT:     store i32 %{{.+}}, ptr %[[ORIG]]
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
new file mode 100644
index 0000000000000..0b0f3e2e53dea
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-common.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is a COMMON block member is accessed in
+! the target body through the COMMON block storage map, not through its own
+! dedicated member map entry. The host-side redirect only rewrites the
+! member map argument, so the runtime-private pointer would be dead and the body
+! would accumulate into the mapped original instead of the reduction-private
+! copy. Until the redirect can target the storage map this case is rejected.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a COMMON block member
+
+subroutine omp_target_in_reduction_common()
+  integer :: i, k
+  common /cb/ i, k
+  i = 0
+  !$omp target map(tofrom: /cb/) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_common
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
new file mode 100644
index 0000000000000..43d03dba0c594
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-equivalence.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! A `target in_reduction` list item that is EQUIVALENCE storage-associated
+! shares storage with another variable that may be mapped separately (here `j`).
+! The body updates the aliased storage through `j`'s map while the host-side
+! redirect rewrites only `i`'s map argument, leaving the runtime-private copy
+! dead. Reject until the redirect can account for storage association.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of an EQUIVALENCE storage-associated variable
+
+subroutine omp_target_in_reduction_equivalence()
+  integer :: i, j
+  equivalence(i, j)
+  i = 0
+  !$omp target map(tofrom: j) in_reduction(+:i)
+  j = j + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_equivalence
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
new file mode 100644
index 0000000000000..075fb1a9135be
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/target-inreduction-firstprivate.f90
@@ -0,0 +1,19 @@
+! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! An explicit `firstprivate(i)` on the same `target` as `in_reduction(+:i)`
+! privatizes the reduction variable. The item is then privatized instead of
+! address-preserving-mapped, so the host-side redirect has no dedicated map
+! argument to rewrite and the firstprivate copy region has no mold value, which
+! previously aborted translation. Reject until in_reduction can be reconciled
+! with target privatization.
+
+! CHECK: not yet implemented: TARGET construct with IN_REDUCTION of a privatized variable
+
+subroutine omp_target_in_reduction_firstprivate()
+  integer :: i
+  i = 0
+  !$omp target firstprivate(i) in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction_firstprivate
diff --git a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90 b/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
deleted file mode 100644
index e5a9cffac5a11..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/target-inreduction.f90
+++ /dev/null
@@ -1,15 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
-
-!===============================================================================
-! `mergeable` clause
-!===============================================================================
-
-! CHECK: not yet implemented: Unhandled clause IN_REDUCTION in TARGET construct
-subroutine omp_target_inreduction()
-  integer i
-  i = 0
-  !$omp target in_reduction(+:i)
-  i = i + 1
-  !$omp end target
-end subroutine omp_target_inreduction
diff --git a/flang/test/Lower/OpenMP/target-inreduction-unused.f90 b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
new file mode 100644
index 0000000000000..f9dfa4973610c
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction-unused.f90
@@ -0,0 +1,27 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Per the OpenMP spec, an in_reduction list item on a target construct is
+! implicitly data-mapped. The lowering must not rely on the variable being
+! referenced inside the target body to discover that map: here `i` only
+! appears in the in_reduction clause and is never read or written inside
+! the region. Verify that an omp.map.info for `i` is still emitted and
+! flows into the omp.target's map_entries.
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction_unused()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reduction_unusedEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@{{[^ ]+}} %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %{{[^ ]+}} : !fir.ref<i32>)
+
+subroutine omp_target_in_reduction_unused()
+  interface
+    subroutine sub()
+    end subroutine
+  end interface
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  call sub()
+  !$omp end target
+end subroutine omp_target_in_reduction_unused
diff --git a/flang/test/Lower/OpenMP/target-inreduction.f90 b/flang/test/Lower/OpenMP/target-inreduction.f90
new file mode 100644
index 0000000000000..74f0f003aab4f
--- /dev/null
+++ b/flang/test/Lower/OpenMP/target-inreduction.f90
@@ -0,0 +1,30 @@
+! RUN: bbc -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 -o - %s 2>&1 | FileCheck %s
+
+! Verify that in_reduction on a target construct is lowered to an
+! omp.target with both an in_reduction clause and an implicit map_entries
+! entry for the same variable. The in_reduction clause does not define an
+! entry block argument: inside the target body the variable is accessed
+! through its map_entries block argument. The implicit map also captures the
+! original pointer into the target region so the MLIR -> LLVM IR translation
+! can pass it to __kmpc_task_reduction_get_th_data.
+
+!CHECK-LABEL: omp.declare_reduction
+!CHECK-SAME:  @[[RED_I32_NAME:.*]] : i32 init {
+
+!CHECK-LABEL: func.func @_QPomp_target_in_reduction()
+!CHECK:       %[[IDECL:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = "_QFomp_target_in_reductionEi"}
+!CHECK:       %[[IMAP:.*]] = omp.map.info var_ptr(%[[IDECL]]#1 : !fir.ref<i32>, i32) map_clauses(implicit, tofrom) capture(ByRef) -> !fir.ref<i32> {name = "i"}
+!CHECK:       omp.target kernel_type(generic) in_reduction(@[[RED_I32_NAME]] %[[IDECL]]#0 : !fir.ref<i32>)
+!CHECK-SAME:    map_entries(%[[IMAP]] -> %[[MAPARG:[^ ]+]] : !fir.ref<i32>)
+!CHECK:         hlfir.declare %[[MAPARG]]
+!CHECK:         omp.terminator
+!CHECK:       }
+
+subroutine omp_target_in_reduction()
+  integer i
+  i = 0
+  !$omp target in_reduction(+:i)
+  i = i + 1
+  !$omp end target
+end subroutine omp_target_in_reduction
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 965ac358c259e..54d14602fbdbf 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -2440,6 +2440,39 @@ class OpenMPIRBuilder {
 
   ///}
 
+  /// Emit the host runtime lookups that redirect the `in_reduction`
+  /// list items of an `omp.target` region to their per-task reduction-private
+  /// storage.
+  ///
+  /// This owns the complete runtime-generation path for target `in_reduction`.
+  /// It computes the executing thread's gtid once for the whole target body (so
+  /// a target with several in_reduction items does not emit a redundant
+  /// `__kmpc_global_thread_num` per item), then for each item emits
+  /// `__kmpc_task_reduction_get_th_data(gtid, /*descriptor=*/null, origPtr)`
+  /// with the address-space normalization required by the runtime entry point.
+  /// The NULL descriptor makes the runtime walk the enclosing taskgroups to
+  /// find the matching `task_reduction` registration for the item. The lookups
+  /// are emitted at \p Loc, which must be inside the target task body.
+  ///
+  /// The front-end-specific work (matching each `in_reduction` item to its
+  /// mapped storage and binding the generated private pointer back to the
+  /// right value) stays with the caller: each generated private pointer is
+  /// handed back through \p MapPrivateCB.
+  ///
+  /// \param Loc          Insertion point for the target body.
+  /// \param OrigPtrs     Per item, the mapped original pointer used as the
+  ///                     runtime `orig` argument.
+  /// \param ResultPtrTys Per item, the type the returned private pointer must
+  ///                     have (for address-space normalization). Must have the
+  ///                     same length as \p OrigPtrs.
+  /// \param MapPrivateCB Called once per item, in list order, with the item
+  ///                     index and the generated per-task private pointer.
+  /// \returns The insertion point after the emitted lookups.
+  LLVM_ABI InsertPointTy createTargetInReduction(
+      const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+      ArrayRef<Type *> ResultPtrTys,
+      function_ref<void(unsigned, Value *)> MapPrivateCB);
+
   /// Return the insertion point used by the underlying IRBuilder.
   InsertPointTy getInsertionPoint() { return Builder.saveIP(); }
 
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..0dc704a87e4ed 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -1218,6 +1218,56 @@ Value *OpenMPIRBuilder::getOrCreateThreadID(Value *Ident) {
       "omp_global_thread_num");
 }
 
+OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createTargetInReduction(
+    const LocationDescription &Loc, ArrayRef<Value *> OrigPtrs,
+    ArrayRef<Type *> ResultPtrTys,
+    function_ref<void(unsigned, Value *)> MapPrivateCB) {
+  assert(OrigPtrs.size() == ResultPtrTys.size() &&
+         "expected one result pointer type per in_reduction item");
+  if (!updateToLocation(Loc))
+    return Loc.IP;
+  if (OrigPtrs.empty())
+    return Builder.saveIP();
+
+  // Compute the executing thread's gtid once for the whole target body and
+  // reuse it for every in_reduction lookup, so a target with several
+  // in_reduction items does not emit a redundant __kmpc_global_thread_num per
+  // item.
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Gtid = getOrCreateThreadID(Ident);
+
+  // The runtime entry point takes (and returns) a generic, default-address-
+  // space `ptr`. A NULL descriptor makes the runtime walk the enclosing
+  // taskgroups to find the matching task_reduction registration for the item.
+  Type *PtrTy = PointerType::getUnqual(M.getContext());
+  Value *NullDesc = ConstantPointerNull::get(PtrTy);
+  FunctionCallee GetThData =
+      getOrCreateRuntimeFunction(M, OMPRTL___kmpc_task_reduction_get_th_data);
+
+  for (unsigned Idx = 0; Idx < OrigPtrs.size(); ++Idx) {
+    // Normalize a non-default-address-space original pointer to the generic
+    // address space before the call.
+    Value *OrigPtr = OrigPtrs[Idx];
+    if (auto *OrigPtrTy = dyn_cast<PointerType>(OrigPtr->getType());
+        OrigPtrTy && OrigPtrTy->getAddressSpace() != 0)
+      OrigPtr = Builder.CreateAddrSpaceCast(OrigPtr, PtrTy);
+
+    Value *Priv = Builder.CreateCall(GetThData, {Gtid, NullDesc, OrigPtr},
+                                     "omp.inred.priv");
+
+    // Cast the returned private pointer back to the requested address space
+    // when it differs.
+    if (auto *ResPtrTy = dyn_cast<PointerType>(ResultPtrTys[Idx]);
+        ResPtrTy && ResPtrTy->getAddressSpace() != 0)
+      Priv = Builder.CreateAddrSpaceCast(Priv, ResultPtrTys[Idx]);
+
+    MapPrivateCB(Idx, Priv);
+  }
+  return Builder.saveIP();
+}
+
 OpenMPIRBuilder::InsertPointOrErrorTy
 OpenMPIRBuilder::createBarrier(const LocationDescription &Loc, Directive Kind,
                                bool ForceSimpleCall, bool CheckCancelFlag) {
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
index 54e46b74c5b64..cd1223dc1702c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPClauses.td
@@ -764,9 +764,17 @@ class OpenMP_InReductionClauseSkip<
     OptionalAttr<SymbolRefArrayAttr>:$in_reduction_syms
   );
 
-  // Description varies depending on the operation. Assembly format not defined
-  // because this clause must be processed together with the first region of the
-  // operation, as it defines entry block arguments.
+  // This assembly format should only be used by operations where `in_reduction`
+  // does not define entry block arguments (e.g. `omp.target`). Otherwise, it
+  // must be printed and parsed together with the corresponding region, because
+  // it defines entry block arguments.
+  let optAssemblyFormat = [{
+    `in_reduction` `(`
+      custom<InReductionClause>($in_reduction_vars, type($in_reduction_vars),
+                                $in_reduction_byref, $in_reduction_syms) `)`
+  }];
+
+  // Description varies depending on the operation.
 }
 
 def OpenMP_InReductionClause : OpenMP_InReductionClauseSkip<>;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 932a4a91cc6ab..98242a66e411f 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1001,7 +1001,8 @@ def TaskOp : OpenMP_Op<"task", traits = [
     OutlineableOpenMPOpInterface
   ], clauses = [
     OpenMP_AffinityClause, OpenMP_AllocateClause, OpenMP_DependClause,
-    OpenMP_FinalClause, OpenMP_IfClause, OpenMP_InReductionClause,
+    OpenMP_FinalClause, OpenMP_IfClause,
+    OpenMP_InReductionClauseSkip<assemblyFormat = true>,
     OpenMP_MergeableClause, OpenMP_PriorityClause, OpenMP_PrivateClause,
     OpenMP_UntiedClause, OpenMP_DetachClause
   ], singleRegion = true> {
@@ -1047,9 +1048,10 @@ def TaskloopContextOp : OpenMP_Op<"taskloop.context", traits = [
     DeclareOpInterfaceMethods<OutlineableOpenMPOpInterface>
   ], clauses = [
     OpenMP_AllocateClause, OpenMP_FinalClause, OpenMP_GrainsizeClause,
-    OpenMP_IfClause, OpenMP_InReductionClause, OpenMP_MergeableClause,
-    OpenMP_NogroupClause, OpenMP_NumTasksClause, OpenMP_PriorityClause,
-    OpenMP_PrivateClause, OpenMP_ReductionClause, OpenMP_UntiedClause
+    OpenMP_IfClause, OpenMP_InReductionClauseSkip<assemblyFormat = true>,
+    OpenMP_MergeableClause, OpenMP_NogroupClause, OpenMP_NumTasksClause,
+    OpenMP_PriorityClause, OpenMP_PrivateClause, OpenMP_ReductionClause,
+    OpenMP_UntiedClause
   ], singleRegion = true> {
   let summary = "OutlinableOpenMPOpInterface wrapper for taskloop construct";
   let description = [{
@@ -1662,6 +1664,15 @@ def TargetOp : OpenMP_Op<"target", traits = [
   ];
 
   let extraClassDeclaration = [{
+    // Override BlockArgOpenMPOpInterface method because `in_reduction` list
+    // items on `omp.target` do not define entry block arguments. The reduction
+    // variable is accessed inside the target body through its matching
+    // `map_entries` block argument; the `in_reduction` operands are kept only
+    // as host-side metadata used to look up the per-task private storage.
+    unsigned numInReductionBlockArgs() {
+      return 0;
+    }
+
     mlir::Value getMappedValueForPrivateVar(unsigned privVarIdx) {
       std::optional<DenseI64ArrayAttr> privateMapIdices = getPrivateMapsAttr();
 
@@ -1686,8 +1697,7 @@ def TargetOp : OpenMP_Op<"target", traits = [
     " | `map_iterated` `(` $map_iterated `:` type($map_iterated) `)`)" # [{
       custom<TargetOpRegion>(
           $region, $has_device_addr_vars, type($has_device_addr_vars),
-          $host_eval_vars, type($host_eval_vars), $in_reduction_vars,
-          type($in_reduction_vars), $in_reduction_byref, $in_reduction_syms,
+          $host_eval_vars, type($host_eval_vars),
           $map_vars, type($map_vars), $private_vars, type($private_vars),
           $private_syms, $private_needs_barrier, $private_maps) attr-dict
     }];
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index acf5500205b6f..3a3798c3ad00b 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -1038,6 +1038,82 @@ static void printDynGroupprivateClause(OpAsmPrinter &printer, Operation *op,
   }
 }
 
+//===----------------------------------------------------------------------===//
+// Parser and printer for in_reduction Clause
+//===----------------------------------------------------------------------===//
+
+/// Parses an `in_reduction` clause for an operation that does not give its
+/// list items entry block arguments (e.g. `omp.target`). The expected format is
+/// a comma-separated list of `[byref] @sym %var` followed by `: types`.
+static ParseResult parseInReductionClause(
+    OpAsmParser &parser,
+    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
+    SmallVectorImpl<Type> &inReductionTypes,
+    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms) {
+  SmallVector<SymbolRefAttr> symbolVec;
+  SmallVector<bool> isByRefVec;
+
+  if (parser.parseCommaSeparatedList([&]() {
+        isByRefVec.push_back(parser.parseOptionalKeyword("byref").succeeded());
+        if (parser.parseAttribute(symbolVec.emplace_back()) ||
+            parser.parseOperand(inReductionVars.emplace_back()))
+          return failure();
+        return success();
+      }))
+    return failure();
+
+  if (parser.parseColon())
+    return failure();
+
+  if (parser.parseCommaSeparatedList(
+          [&]() { return parser.parseType(inReductionTypes.emplace_back()); }))
+    return failure();
+
+  if (inReductionVars.size() != inReductionTypes.size())
+    return failure();
+
+  inReductionByref = makeDenseBoolArrayAttr(parser.getContext(), isByRefVec);
+  SmallVector<Attribute> symbolAttrs(symbolVec.begin(), symbolVec.end());
+  inReductionSyms = ArrayAttr::get(parser.getContext(), symbolAttrs);
+  return success();
+}
+
+/// Prints an `in_reduction` clause for an operation that does not give its list
+/// items entry block arguments (e.g. `omp.target`). Mirrors
+/// `parseInReductionClause`.
+static void printInReductionClause(OpAsmPrinter &p, Operation *op,
+                                   ValueRange inReductionVars,
+                                   TypeRange inReductionTypes,
+                                   DenseBoolArrayAttr inReductionByref,
+                                   ArrayAttr inReductionSyms) {
+  MLIRContext *ctx = op->getContext();
+
+  ArrayAttr syms = inReductionSyms;
+  if (!syms) {
+    SmallVector<Attribute> values(inReductionVars.size(), nullptr);
+    syms = ArrayAttr::get(ctx, values);
+  }
+
+  DenseBoolArrayAttr byref = inReductionByref;
+  if (!byref) {
+    SmallVector<bool> values(inReductionVars.size(), false);
+    byref = DenseBoolArrayAttr::get(ctx, values);
+  }
+
+  llvm::interleaveComma(
+      llvm::zip_equal(inReductionVars, syms.getValue(), byref.asArrayRef()), p,
+      [&p](auto t) {
+        auto [var, sym, isByRef] = t;
+        if (isByRef)
+          p << "byref ";
+        if (sym)
+          p << sym << " ";
+        p << var;
+      });
+  p << " : ";
+  llvm::interleaveComma(inReductionTypes, p);
+}
+
 //===----------------------------------------------------------------------===//
 // Parsers for operations including clauses that define entry block arguments.
 //===----------------------------------------------------------------------===//
@@ -1305,9 +1381,6 @@ static ParseResult parseTargetOpRegion(
     SmallVectorImpl<Type> &hasDeviceAddrTypes,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &hostEvalVars,
     SmallVectorImpl<Type> &hostEvalTypes,
-    SmallVectorImpl<OpAsmParser::UnresolvedOperand> &inReductionVars,
-    SmallVectorImpl<Type> &inReductionTypes,
-    DenseBoolArrayAttr &inReductionByref, ArrayAttr &inReductionSyms,
     SmallVectorImpl<OpAsmParser::UnresolvedOperand> &mapVars,
     SmallVectorImpl<Type> &mapTypes,
     llvm::SmallVectorImpl<OpAsmParser::UnresolvedOperand> &privateVars,
@@ -1316,8 +1389,6 @@ static ParseResult parseTargetOpRegion(
   AllRegionParseArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, &privateMaps);
@@ -1568,20 +1639,18 @@ static void printBlockArgRegion(OpAsmPrinter &p, Operation *op, Region &region,
 
 // These parseXyz functions correspond to the custom<Xyz> definitions
 // in the .td file(s).
-static void printTargetOpRegion(
-    OpAsmPrinter &p, Operation *op, Region &region,
-    ValueRange hasDeviceAddrVars, TypeRange hasDeviceAddrTypes,
-    ValueRange hostEvalVars, TypeRange hostEvalTypes,
-    ValueRange inReductionVars, TypeRange inReductionTypes,
-    DenseBoolArrayAttr inReductionByref, ArrayAttr inReductionSyms,
-    ValueRange mapVars, TypeRange mapTypes, ValueRange privateVars,
-    TypeRange privateTypes, ArrayAttr privateSyms, UnitAttr privateNeedsBarrier,
-    DenseI64ArrayAttr privateMaps) {
+static void printTargetOpRegion(OpAsmPrinter &p, Operation *op, Region &region,
+                                ValueRange hasDeviceAddrVars,
+                                TypeRange hasDeviceAddrTypes,
+                                ValueRange hostEvalVars,
+                                TypeRange hostEvalTypes, ValueRange mapVars,
+                                TypeRange mapTypes, ValueRange privateVars,
+                                TypeRange privateTypes, ArrayAttr privateSyms,
+                                UnitAttr privateNeedsBarrier,
+                                DenseI64ArrayAttr privateMaps) {
   AllRegionPrintArgs args;
   args.hasDeviceAddrArgs.emplace(hasDeviceAddrVars, hasDeviceAddrTypes);
   args.hostEvalArgs.emplace(hostEvalVars, hostEvalTypes);
-  args.inReductionArgs.emplace(inReductionVars, inReductionTypes,
-                               inReductionByref, inReductionSyms);
   args.mapArgs.emplace(mapVars, mapTypes);
   args.privateArgs.emplace(privateVars, privateTypes, privateSyms,
                            privateNeedsBarrier, privateMaps);
@@ -2579,8 +2648,7 @@ LogicalResult TargetUpdateOp::verify() {
 void TargetOp::build(OpBuilder &builder, OperationState &state,
                      const TargetExtOperands &clauses) {
   MLIRContext *ctx = builder.getContext();
-  // TODO Store clauses in op: allocateVars, allocatorVars, inReductionVars,
-  // inReductionByref, inReductionSyms.
+  // TODO Store clauses in op: allocateVars, allocatorVars.
   TargetOp::build(
       builder, state, /*allocate_vars=*/{}, /*allocator_vars=*/{},
       makeArrayAttr(ctx, clauses.dependKinds), clauses.dependVars,
@@ -2588,9 +2656,10 @@ void TargetOp::build(OpBuilder &builder, OperationState &state,
       clauses.device, clauses.dynGroupprivateAccessGroup,
       clauses.dynGroupprivateFallback, clauses.dynGroupprivateSize,
       clauses.hasDeviceAddrVars, clauses.hostEvalVars, clauses.ifExpr,
-      /*in_reduction_vars=*/{}, /*in_reduction_byref=*/nullptr,
-      /*in_reduction_syms=*/nullptr, clauses.isDevicePtrVars, clauses.mapVars,
-      clauses.mapIterated, clauses.nowait, clauses.privateVars,
+      clauses.inReductionVars,
+      makeDenseBoolArrayAttr(ctx, clauses.inReductionByref),
+      makeArrayAttr(ctx, clauses.inReductionSyms), clauses.isDevicePtrVars,
+      clauses.mapVars, clauses.mapIterated, clauses.nowait, clauses.privateVars,
       makeArrayAttr(ctx, clauses.privateSyms), clauses.privateNeedsBarrier,
       clauses.threadLimitVars, /*private_maps=*/nullptr, clauses.kernelType);
 }
@@ -2631,6 +2700,20 @@ bool TargetOp::hasHostEvalTripCount() {
   return false;
 }
 
+/// An `omp.target` `in_reduction` operand is captured by a `map_entries` entry
+/// when the entry's `MapInfoOp` var_ptr is the same SSA value, or another
+/// result of the same defining op. At this stage, exact identity can only be
+/// required for block arguments, which have no defining op. Flang emits
+/// `hlfir.declare` #0 for the `in_reduction` operand and #1 for the map
+/// `var_ptr`; these collapse to the same value after lowering, but that cannot
+/// be enforced here.
+static bool targetInReductionCapturedBy(Value inReductionVar, Value mapVarPtr) {
+  if (mapVarPtr == inReductionVar)
+    return true;
+  Operation *def = inReductionVar.getDefiningOp();
+  return def && mapVarPtr.getDefiningOp() == def;
+}
+
 LogicalResult TargetOp::verify() {
   if (getKernelType() == TargetExecMode::bare && !isCombined())
     return emitOpError() << "bare kernel requires 'omp.combined'";
@@ -2655,6 +2738,30 @@ LogicalResult TargetOp::verify() {
   if (failed(verifyPrivateVarList(*this)))
     return failure();
 
+  if (failed(verifyReductionVarList(*this, getInReductionSyms(),
+                                    getInReductionVars(),
+                                    getInReductionByref())))
+    return failure();
+
+  // An `in_reduction` operand on `omp.target` has no dedicated entry block
+  // argument; inside the region it is accessed through the block argument of a
+  // matching `map_entries` entry, and the host rewrites that map argument to
+  // the reduction-private storage. Require every `in_reduction` operand to be
+  // captured by at least one `map_entries` entry.
+  for (Value inReductionVar : getInReductionVars()) {
+    bool captured = false;
+    for (Value mapVar : getMapVars()) {
+      auto mapInfo = mapVar.getDefiningOp<MapInfoOp>();
+      if (targetInReductionCapturedBy(inReductionVar, mapInfo.getVarPtr())) {
+        captured = true;
+        break;
+      }
+    }
+    if (!captured)
+      return emitOpError() << "in_reduction variable must be captured by a "
+                              "matching map_entries entry";
+  }
+
   return verifyPrivateVarsMapping(*this);
 }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index af1ec17563806..bd773662b8bbd 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -364,9 +364,40 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       op.emitWarning("hint clause discarded");
   };
   auto checkInReduction = [&todo](auto op, LogicalResult &result) {
-    if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
-        op.getInReductionSyms())
+    if (isa<omp::TargetOp, omp::TaskOp, omp::TaskloopContextOp>(
+            op.getOperation())) {
+      if (auto byrefAttr = op.getInReductionByref()) {
+        for (bool isByRef : *byrefAttr) {
+          if (isByRef) {
+            result = todo("in_reduction with byref modifier");
+            return;
+          }
+        }
+      }
+      if (isa<omp::TargetOp>(op.getOperation())) {
+        if (auto inReductionSyms = op.getInReductionSyms()) {
+          for (auto sym :
+               (*inReductionSyms).template getAsRange<SymbolRefAttr>()) {
+            auto decl =
+                SymbolTable::lookupNearestSymbolFrom<omp::DeclareReductionOp>(
+                    op, sym);
+            assert(decl &&
+                   "symbol resolution should be guaranteed by the op verifier");
+            if (decl.getInitializerRegion().front().getNumArguments() != 1) {
+              result = todo("in_reduction with two-argument initializer");
+              return;
+            }
+            if (!decl.getCleanupRegion().empty()) {
+              result = todo("in_reduction with cleanup region");
+              return;
+            }
+          }
+        }
+      }
+    } else if (!op.getInReductionVars().empty() || op.getInReductionByref() ||
+               op.getInReductionSyms()) {
       result = todo("in_reduction");
+    }
   };
   auto checkNowait = [&todo](auto op, LogicalResult &result) {
     if (op.getNowait())
@@ -423,14 +454,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
           return;
         }
   };
-  auto checkInReductionByref = [&todo](auto op, LogicalResult &result) {
-    if (auto byrefAttr = op.getInReductionByref())
-      for (bool isByRef : *byrefAttr)
-        if (isByRef) {
-          result = todo("in_reduction with byref modifier");
-          return;
-        }
-  };
   auto checkNumTeams = [&todo](auto op, LogicalResult &result) {
     if (op.hasNumTeamsMultiDim())
       result = todo("num_teams with multi-dimensional values");
@@ -482,7 +505,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
       })
       .Case([&](omp::TaskgroupOp op) {
         checkAllocate(op, result);
@@ -494,7 +517,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
       })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
-        checkInReductionByref(op, result);
+        checkInReduction(op, result);
         checkReduction(op, result);
         checkReductionByref(op, result);
       })
@@ -8560,6 +8583,55 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isOffloadEntry =
       isTargetDevice || !ompBuilder->Config.TargetTriples.empty();
 
+  // Resolve in_reduction clauses on omp.target for the host. From the target
+  // device's perspective an in_reduction list item behaves as a regular
+  // map(tofrom) variable, so no special handling is needed there; only the
+  // host redirects the mapped value to the per-task reduction-private storage
+  // returned by __kmpc_task_reduction_get_th_data (emitted inside the
+  // to-be-outlined target task body). This applies to both offloading and
+  // non-offloading host modules.
+  //
+  // The target body has no dedicated in_reduction block argument: each
+  // in_reduction variable is accessed through its map_entries block argument.
+  // So each in_reduction variable must also be captured by a matching
+  // map_entries entry (guaranteed by the verifier); without one the outlined
+  // body would reference a value defined in the host function. Record, for each
+  // in_reduction variable, the position of that map entry so the corresponding
+  // map block argument can be redirected inside the body. The in_reduction
+  // operand itself is used as the `orig` argument of the runtime lookup.
+  SmallVector<llvm::Value *> inRedOrigPtrs;
+  SmallVector<unsigned> inRedMapArgIdx;
+  if (!targetOp.getInReductionVars().empty() && !isTargetDevice) {
+    inRedOrigPtrs.reserve(targetOp.getInReductionVars().size());
+    inRedMapArgIdx.reserve(targetOp.getInReductionVars().size());
+    for (Value v : targetOp.getInReductionVars()) {
+      // Select the map_entries entry that captures this in_reduction operand.
+      // The verifier guarantees at least one match exists; more than one
+      // matching entry is a lowering ambiguity (the redirect cannot pick which
+      // map argument to rebind).
+      std::optional<unsigned> matchIdx;
+      for (auto [idx, mapV] : llvm::enumerate(targetOp.getMapVars())) {
+        auto mapInfo = mapV.getDefiningOp<omp::MapInfoOp>();
+        if (v != mapInfo.getVarPtr())
+          continue;
+        if (matchIdx)
+          return targetOp.emitError()
+                 << "in_reduction variable on omp.target has multiple matching "
+                    "map_entries entries; the redirect target is ambiguous";
+        matchIdx = idx;
+      }
+      // The verifier requires a capturing map entry for every in_reduction
+      // operand, so a match must exist here.
+      assert(matchIdx &&
+             "TargetOp verifier guarantees a matching map_entries entry for "
+             "each in_reduction variable");
+      inRedMapArgIdx.push_back(*matchIdx);
+      // The runtime `orig` pointer is the in_reduction operand itself, the
+      // reduction variable the enclosing taskgroup registered.
+      inRedOrigPtrs.push_back(moduleTranslation.lookupValue(v));
+    }
+  }
+
   // For some private variables, the MapsForPrivatizedVariablesPass
   // creates MapInfoOp instances. Go through the private variables and
   // the mapped variables so that during codegeneration we are able
@@ -8635,8 +8707,15 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
         attr.isStringAttribute())
       llvmOutlinedFn->addFnAttr(attr);
 
-    for (auto [arg, mapOp] : llvm::zip_equal(mapBlockArgs, mapVars)) {
-      auto mapInfoOp = cast<omp::MapInfoOp>(mapOp.getDefiningOp());
+    for (auto [idx, arg] : llvm::enumerate(mapBlockArgs)) {
+      // in_reduction list items on omp.target are accessed through their
+      // map_entries block argument, which is redirected below to the per-task
+      // reduction-private storage returned by the runtime. Skip the default
+      // host-value mapping for those block arguments so the write-once
+      // mapValue mapping is free to be set to the private pointer.
+      if (llvm::is_contained(inRedMapArgIdx, idx))
+        continue;
+      auto mapInfoOp = cast<omp::MapInfoOp>(mapVars[idx].getDefiningOp());
       llvm::Value *mapOpValue =
           moduleTranslation.lookupValue(mapInfoOp.getVarPtr());
       moduleTranslation.mapValue(arg, mapOpValue);
@@ -8672,6 +8751,37 @@ convertOmpTarget(Operation &opInst, llvm::IRBuilderBase &builder,
             targetOp.getPrivateNeedsBarrier(), &mappedPrivateVars)))
       return llvm::make_error<PreviouslyReportedError>();
 
+    // The target body accesses each in_reduction variable through its
+    // map_entries block argument. Redirect that block argument to the per-task
+    // private storage returned by __kmpc_task_reduction_get_th_data so the body
+    // accumulates into the reduction-private copy rather than the mapped
+    // original. The lookup must run inside the target task body so the gtid
+    // corresponds to the executing thread. The descriptor argument is NULL: the
+    // runtime walks enclosing taskgroups to locate the matching task_reduction
+    // registration for `origPtr`. Mirrors the in_reduction handling on
+    // omp.taskloop.context.
+    if (!inRedOrigPtrs.empty()) {
+      // Collect, per item, the type the private pointer must have (the map
+      // block argument's type), and, through the callback, rebind the map block
+      // argument that stands in for each in_reduction list item to the per-task
+      // reduction-private storage the runtime returns.
+      SmallVector<llvm::Type *> inRedResultPtrTys;
+      inRedResultPtrTys.reserve(inRedMapArgIdx.size());
+      for (unsigned mapArgIdx : inRedMapArgIdx)
+        inRedResultPtrTys.push_back(
+            moduleTranslation.convertType(mapBlockArgs[mapArgIdx].getType()));
+
+      llvm::OpenMPIRBuilder::LocationDescription bodyLoc(builder);
+      llvm::OpenMPIRBuilder::InsertPointTy redIP =
+          ompBuilder->createTargetInReduction(
+              bodyLoc, inRedOrigPtrs, inRedResultPtrTys,
+              [&](unsigned idx, llvm::Value *priv) {
+                moduleTranslation.mapValue(mapBlockArgs[inRedMapArgIdx[idx]],
+                                           priv);
+              });
+      builder.restoreIP(redIP);
+    }
+
     LLVM::ModuleTranslation::SaveStack<OpenMPAllocStackFrame> frame(
         moduleTranslation, allocaIP, deallocBlocks);
     llvm::Expected<llvm::BasicBlock *> exitBlock = convertOmpOpRegions(
diff --git a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
index 015e901126a76..8108a1af31e21 100644
--- a/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
+++ b/mlir/test/Dialect/OpenMP/host-op-filtering.mlir
@@ -348,8 +348,12 @@ module attributes {omp.is_target_device = true} {
       omp.terminator
     }
 
-    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
-    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
+    // The `in_reduction` list item on `omp.target` has no dedicated block
+    // argument; it is captured by a matching `map_entries` entry.
+    // CHECK-NEXT: %[[MAP:.*]] = omp.map.info var_ptr(%[[ARG0]] : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    // CHECK-NEXT: omp.target kernel_type(generic) allocate(%[[ARG0]] : !llvm.ptr -> %[[ARG0]] : !llvm.ptr) thread_limit(%[[ARG1]] : i32) map_entries(%[[MAP]] -> %{{.*}} : !llvm.ptr) private(@privatizer %[[ARG0]] -> %{{.*}} : !llvm.ptr)
+    omp.target kernel_type(generic) allocate(%arg0 : !llvm.ptr -> %arg0 : !llvm.ptr) depend(taskdependin -> %arg0 : !llvm.ptr) device(%arg1 : i32) if(%arg2) thread_limit(%arg1 : i32) in_reduction(@reduction %arg0 : !llvm.ptr) map_entries(%1 -> %arg3 : !llvm.ptr) private(@privatizer %arg0 -> %arg4 : !llvm.ptr) {
       omp.terminator
     }
 
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index 668c9fb6dc2b3..cf7be4eb638cc 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -3255,6 +3255,91 @@ func.func @omp_target_depend(%data_var: memref<i32>) {
 
 // -----
 
+func.func @omp_target_in_reduction_unresolved(%ptr: !llvm.ptr) {
+  // expected-error @below {{op expected symbol reference @add_f32 to point to a reduction declaration}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_duplicate(%ptr: !llvm.ptr) {
+  // expected-error @below {{op accumulator variable used more than once}}
+  omp.target kernel_type(generic) in_reduction(@add_f32 %ptr, @add_f32 %ptr : !llvm.ptr, !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg: i32):
+  %0 = arith.constant 0 : i32
+  omp.yield (%0 : i32)
+}
+combiner {
+^bb1(%arg0: i32, %arg1: i32):
+  %1 = arith.addi %arg0, %arg1 : i32
+  omp.yield (%1 : i32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> i32
+  llvm.atomicrmw add %arg2, %2 monotonic : !llvm.ptr, i32
+  omp.yield
+}
+
+func.func @omp_target_in_reduction_type_mismatch(%mem: memref<1xf32>) {
+  // expected-error @below {{op expected accumulator ('memref<1xf32>') to be the same type as reduction declaration ('!llvm.ptr')}}
+  omp.target kernel_type(generic) in_reduction(@add_i32 %mem : memref<1xf32>) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
+omp.declare_reduction @add_missing_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = arith.constant 0.0 : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = arith.addf %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+
+func.func @omp_target_in_reduction_missing_map(%ptr: !llvm.ptr) {
+  // An in_reduction operand on omp.target has no dedicated block argument; it
+  // is accessed inside the body through a matching map_entries block argument.
+  // With no map_entries entry capturing it, the verifier must reject the op.
+  // expected-error @below {{in_reduction variable must be captured by a matching map_entries entry}}
+  omp.target kernel_type(generic) in_reduction(@add_missing_map_f32 %ptr : !llvm.ptr) {
+    omp.terminator
+  }
+  return
+}
+
+// -----
+
 func.func @omp_distribute_schedule(%chunk_size : i32, %lb : i32, %ub : i32, %step : i32) -> () {
   // expected-error @below {{op chunk size set without dist_schedule_static being present}}
   "omp.distribute"(%chunk_size) <{operandSegmentSizes = array<i32: 0, 0, 1, 0>}> ({
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
new file mode 100644
index 0000000000000..0e9611c27ddcc
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-device.mlir
@@ -0,0 +1,51 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// in_reduction on omp.target compiled for the target *device*. From the
+// device's perspective an in_reduction list item behaves as a regular mapped
+// variable: the host-only redirect to the per-task reduction-private storage
+// (the __kmpc_global_thread_num + __kmpc_task_reduction_get_th_data lookup) is
+// guarded by !isTargetDevice and is therefore not emitted. Verify the device
+// body accesses the mapped variable directly and that none of the host
+// runtime lookup is present.
+
+module attributes {dlti.dl_spec = #dlti.dl_spec<"dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true} {
+  omp.declare_reduction @add_i32 : i32
+  init {
+  ^bb0(%arg0: i32):
+    %c0 = llvm.mlir.constant(0 : i32) : i32
+    omp.yield(%c0 : i32)
+  }
+  combiner {
+  ^bb0(%arg0: i32, %arg1: i32):
+    %s = llvm.add %arg0, %arg1 : i32
+    omp.yield(%s : i32)
+  }
+
+  llvm.func @target_inreduction_device(%x : !llvm.ptr) {
+    %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+    omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+      %v = llvm.load %marg : !llvm.ptr -> i32
+      %c1 = llvm.mlir.constant(1 : i32) : i32
+      %s = llvm.add %v, %c1 : i32
+      llvm.store %s, %marg : i32, !llvm.ptr
+      omp.terminator
+    }
+    llvm.return
+  }
+}
+
+// The device kernel receives the mapped pointer as a captured argument and
+// accesses the in_reduction variable directly through it.
+// CHECK-LABEL: define {{.*}}amdgpu_kernel void @__omp_offloading_{{.*}}_target_inreduction_device_
+// CHECK-SAME:    (ptr %[[ARG:.+]], ptr %{{.+}})
+// CHECK:         store ptr %[[ARG]], ptr %[[ALLOCA:.+]]
+// CHECK:         %[[MAPPED:.+]] = load ptr, ptr %[[ALLOCA]]
+// The host-only in_reduction redirect must not run on the device: there is no
+// gtid lookup, no __kmpc_task_reduction_get_th_data call and no private copy.
+// CHECK-NOT:     @__kmpc_global_thread_num
+// CHECK-NOT:     @__kmpc_task_reduction_get_th_data
+// CHECK-NOT:     omp.inred.priv
+// The body reads and writes the mapped variable in place.
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[MAPPED]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[MAPPED]]
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
new file mode 100644
index 0000000000000..c0df1a4e1d332
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction-multi.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Multiple in_reduction items on omp.target. Each item is captured into the
+// target region through its own map_entries entry and accessed inside the
+// body via the corresponding map_entries block argument. On the host, every
+// item performs an independent
+// __kmpc_task_reduction_get_th_data lookup using its own captured original
+// pointer, and the returned per-task private pointer is bound to that item's
+// map block argument. This test pins down the pairing so it cannot pass if the
+// two items were swapped or collapsed onto a single pointer.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_multi(%x : !llvm.ptr, %y : !llvm.ptr) {
+  %mx = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %my = omp.map.info var_ptr(%y : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x, @add_i32 %y : !llvm.ptr, !llvm.ptr)
+      map_entries(%mx -> %mxarg, %my -> %myarg : !llvm.ptr, !llvm.ptr) {
+    // First item (x): load, += 1, store back.
+    %vx = llvm.load %mxarg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %sx = llvm.add %vx, %c1 : i32
+    llvm.store %sx, %mxarg : i32, !llvm.ptr
+    // Second item (y): load, += 2, store back.
+    %vy = llvm.load %myarg : !llvm.ptr -> i32
+    %c2 = llvm.mlir.constant(2 : i32) : i32
+    %sy = llvm.add %vy, %c2 : i32
+    llvm.store %sy, %myarg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards both captured pointers into the outlined target
+// kernel (the trailing argument is the unused descriptor slot).
+// CHECK-LABEL: define void @target_inreduction_multi(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_multi_{{.*}}(ptr %{{.+}}, ptr %{{.+}}, ptr null)
+
+// The two captured original pointers arrive as distinct kernel arguments.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_multi_
+// CHECK-SAME:    (ptr %[[CAPTX:.+]], ptr %[[CAPTY:.+]], ptr %{{.+}})
+
+// A single gtid is computed once for the whole target body and shared by both
+// lookups; each item then performs its own __kmpc_task_reduction_get_th_data
+// call against its own captured pointer, and the returned per-task private
+// pointer is bound to that item's map block argument.
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIVX:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTX]])
+// CHECK:         %[[PRIVY:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPTY]])
+
+// The first item's private storage is the base of the +1 load/store; the
+// CHECK-NOT below ensures the second item's pointer is not touched until the
+// first item's accumulation has completed (i.e. the items are not swapped or
+// merged onto a single private pointer).
+// CHECK:         %[[LX:.+]] = load i32, ptr %[[PRIVX]]
+// CHECK:         %[[SX:.+]] = add i32 %[[LX]], 1
+// CHECK-NOT:     %[[PRIVY]]
+// CHECK:         store i32 %[[SX]], ptr %[[PRIVX]]
+
+// The second item's private storage is the base of the +2 load/store.
+// CHECK:         %[[LY:.+]] = load i32, ptr %[[PRIVY]]
+// CHECK:         %[[SY:.+]] = add i32 %[[LY]], 2
+// CHECK:         store i32 %[[SY]], ptr %[[PRIVY]]
+
+// Exactly two reduction lookups are emitted; no third call sneaks in. The
+// `call` form is used so this does not match the runtime declaration.
+// CHECK-NOT:     call ptr @__kmpc_task_reduction_get_th_data
diff --git a/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
new file mode 100644
index 0000000000000..5e52ee00efafa
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-target-in-reduction.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+// in_reduction on omp.target: the in_reduction variable is also captured
+// into the target region as a map entry (a frontend is expected to emit this
+// implicit map). The in_reduction clause does not define an entry block
+// argument; inside the target body the variable is accessed through its
+// map_entries block argument. The captured pointer is passed to
+// __kmpc_task_reduction_get_th_data with a NULL descriptor; the runtime
+// walks enclosing taskgroups to locate the matching task_reduction
+// registration. The returned per-task private pointer is bound to the
+// map_entries block argument so subsequent loads/stores inside the region
+// use the private copy.
+
+omp.declare_reduction @add_i32 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  omp.target kernel_type(generic) in_reduction(@add_i32 %x : !llvm.ptr) map_entries(%m -> %marg : !llvm.ptr) {
+    %v = llvm.load %marg : !llvm.ptr -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr
+    omp.terminator
+  }
+  llvm.return
+}
+
+// The host stub forwards the captured pointer into the outlined target
+// kernel.
+// CHECK-LABEL: define void @target_inreduction(
+// CHECK:         call void @__omp_offloading_{{.*}}_target_inreduction_{{.*}}(ptr %{{.+}}, ptr null)
+
+// In the outlined target body the in_reduction private pointer is
+// obtained from the runtime using the captured original pointer; that
+// pointer is then the base of the load and store inside the region.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_
+// CHECK-SAME:    (ptr %[[CAPT:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[PRIV:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID]], ptr null, ptr %[[CAPT]])
+// CHECK:         %[[LOADED:.+]] = load i32, ptr %[[PRIV]]
+// CHECK:         %[[SUM:.+]] = add i32 %[[LOADED]], 1
+// CHECK:         store i32 %[[SUM]], ptr %[[PRIV]]
+
+// -----
+
+// Same as the first case but the in_reduction variable lives in a non-default
+// address space (addrspace(1)). __kmpc_task_reduction_get_th_data is declared
+// to take and return a generic (default-AS) `ptr`, so the host
+// lowering must (1) addrspacecast the captured addrspace(1) original pointer to
+// generic `ptr` before the runtime lookup, and (2) addrspacecast the returned
+// generic private pointer back to addrspace(1) so the body's load/store use the
+// private copy in the right address space. This pins down the address-space
+// handling so a regression that passed the addrspace(1) pointer straight into
+// the runtime call (a bad-signature crash) cannot pass.
+
+omp.declare_reduction @add_i32_as1 : i32
+init {
+^bb0(%arg0: i32):
+  %c0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%c0 : i32)
+}
+combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %s = llvm.add %arg0, %arg1 : i32
+  omp.yield(%s : i32)
+}
+
+llvm.func @target_inreduction_as1(%x : !llvm.ptr<1>) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr<1>, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr<1>
+  omp.target kernel_type(generic) in_reduction(@add_i32_as1 %x : !llvm.ptr<1>) map_entries(%m -> %marg : !llvm.ptr<1>) {
+    %v = llvm.load %marg : !llvm.ptr<1> -> i32
+    %c1 = llvm.mlir.constant(1 : i32) : i32
+    %s = llvm.add %v, %c1 : i32
+    llvm.store %s, %marg : i32, !llvm.ptr<1>
+    omp.terminator
+  }
+  llvm.return
+}
+
+// In the outlined target body the addrspace(1) original pointer is normalized
+// to generic `ptr` before the NULL-descriptor in_reduction lookup, and the
+// returned generic private pointer is cast back to addrspace(1) for the body's
+// load and store. The original captured pointer arrives as the addrspace(1)
+// kernel argument and is only used to derive the generic lookup pointer.
+// CHECK-LABEL: define internal void @__omp_offloading_{{.*}}_target_inreduction_as1_
+// CHECK-SAME:    (ptr addrspace(1) %[[CAPT_AS1:.+]], ptr %{{.+}})
+// CHECK:         %[[GTID_AS1:.+]] = call i32 @__kmpc_global_thread_num(
+// CHECK:         %[[ORIG_GEN:.+]] = addrspacecast ptr addrspace(1) %[[CAPT_AS1]] to ptr
+// CHECK:         %[[PRIV_AS1_GEN:.+]] = call ptr @__kmpc_task_reduction_get_th_data(i32 %[[GTID_AS1]], ptr null, ptr %[[ORIG_GEN]])
+// CHECK:         %[[PRIV_AS1:.+]] = addrspacecast ptr %[[PRIV_AS1_GEN]] to ptr addrspace(1)
+// CHECK:         %[[LOADED_AS1:.+]] = load i32, ptr addrspace(1) %[[PRIV_AS1]]
+// CHECK:         %[[SUM_AS1:.+]] = add i32 %[[LOADED_AS1]], 1
+// CHECK:         store i32 %[[SUM_AS1]], ptr addrspace(1) %[[PRIV_AS1]]
+
+// The body must not load or store through the original captured addrspace(1)
+// pointer; all accesses go through the runtime-returned private copy.
+// CHECK-NOT:     store i32 %{{.+}}, ptr addrspace(1) %[[CAPT_AS1]]
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..943b5bad6970f 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -218,10 +218,95 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @target_in_reduction(%x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause in_reduction in omp.target operation}}
+llvm.func @target_in_reduction_byref(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with byref modifier in omp.target operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.target}}
-  omp.target kernel_type(generic) in_reduction(@add_f32 %x -> %prv : !llvm.ptr) {
+  omp.target kernel_type(generic) in_reduction(byref @add_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_cleanup_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+cleanup {
+^bb2(%arg2: f32):
+  omp.yield
+}
+llvm.func @target_in_reduction_cleanup(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with cleanup region in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_cleanup_f32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_two_arg_init_i32 : !llvm.ptr alloc {
+^bb0(%arg: !llvm.ptr):
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 : (i64) -> !llvm.ptr
+  omp.yield(%1 : !llvm.ptr)
+} init {
+^bb0(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.store %0, %arg1 : i32, !llvm.ptr
+  omp.yield(%arg1 : !llvm.ptr)
+} combiner {
+^bb1(%arg0: !llvm.ptr, %arg1: !llvm.ptr):
+  %0 = llvm.load %arg0 : !llvm.ptr -> i32
+  %1 = llvm.load %arg1 : !llvm.ptr -> i32
+  %2 = llvm.add %0, %1 : i32
+  llvm.store %2, %arg0 : i32, !llvm.ptr
+  omp.yield(%arg0 : !llvm.ptr)
+}
+llvm.func @target_in_reduction_two_arg_init(%x : !llvm.ptr) {
+  %m = omp.map.info var_ptr(%x : !llvm.ptr, i32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{not yet implemented: Unhandled clause in_reduction with two-argument initializer in omp.target operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_two_arg_init_i32 %x : !llvm.ptr) map_entries(%m -> %arg : !llvm.ptr) {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
+omp.declare_reduction @add_dup_map_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb0(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+llvm.func @target_in_reduction_duplicate_map(%x : !llvm.ptr) {
+  // The in_reduction variable %x has two matching map_entries entries for the
+  // same var_ptr. The translation cannot disambiguate which map block argument
+  // to redirect, so it must reject this as ambiguous.
+  %m1 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  %m2 = omp.map.info var_ptr(%x : !llvm.ptr, f32) map_clauses(tofrom) capture(ByRef) -> !llvm.ptr
+  // expected-error at below {{in_reduction variable on omp.target has multiple matching map_entries entries; the redirect target is ambiguous}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.target}}
+  omp.target kernel_type(generic) in_reduction(@add_dup_map_f32 %x : !llvm.ptr) map_entries(%m1 -> %arg1, %m2 -> %arg2 : !llvm.ptr, !llvm.ptr) {
     omp.terminator
   }
   llvm.return


From llvmlistbot at llvm.org  Tue Jul 14 11:50:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 11:50:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a568502.6c8668f2.e4f82.2bae@mx.google.com>


================
@@ -0,0 +1,354 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+
+  return %A_pack : tensor<1x16x8x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+
+  return %C_pack : tensor<1x?x8x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
----------------
banach-space wrote:

The problem is that we don't have any other TD op that would run `CombineContractBroadcastMask`. Could you add a TODO here? Something like this:

```mlir
// TODO: Lowering directly to a named contraction. For that to work, `CombineContractBroadcastMask` will have to be moved to a dedicated TD Op so that we don't require `transform.apply_patterns.vector.reduction_to_contract`.
``` 

In fact, you could still lower directly to `vector.contract` and then run `transform.apply_patterns.vector.reduction_to_contract` to get `CombineContractBroadcastMask`. Could you try that? That would already be an improvement.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 14 11:54:59 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Tue, 14 Jul 2026 11:54:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
Message-ID: <llvm.org/llvm/llvm-project/pull/209591@github.com>


https://github.com/secona created https://github.com/llvm/llvm-project/pull/209591

SparseTensorConversion rewrite patterns assume element types are valid primary types, but the dialect does not actually enforce this restriction. This change makes it so that sparse tensor encodings gracefully reject invalid primary types by verifying the encoding.

>From c3ecdc74839369e26f73f60e3733e6f070cc4429 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Wed, 15 Jul 2026 01:42:06 +0700
Subject: [PATCH] [mlir][SparseTensor] Reject sparse encoding on non-primary
 types

---
 .../SparseTensor/IR/SparseTensorDialect.cpp   |  14 ++
 .../Transforms/SparseTensorConversion.cpp     |   3 -
 .../Transforms/Utils/CodegenUtils.cpp         |  12 --
 .../Transforms/Utils/CodegenUtils.h           |   5 -
 .../SparseTensor/constant_index_map.mlir      |  44 ++---
 .../SparseTensor/conversion_invalid.mlir      |  99 ++++++++++-
 .../SparseTensor/vectorize_reduction.mlir     | 160 +++++++++---------
 7 files changed, 214 insertions(+), 123 deletions(-)

diff --git a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
index 7c9f552d62255..6c26a31bf88a0 100644
--- a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
+++ b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
@@ -919,6 +919,18 @@ LogicalResult SparseTensorEncodingAttr::verify(
   return success();
 }
 
+static bool isValidPrimaryType(Type elemTp) {
+  if (elemTp.isF64() || elemTp.isF32() || elemTp.isF16() || elemTp.isBF16() ||
+      elemTp.isInteger(64) || elemTp.isInteger(32) || elemTp.isInteger(16) ||
+      elemTp.isInteger(8))
+    return true;
+  if (auto complexTp = dyn_cast<ComplexType>(elemTp)) {
+    Type elt = complexTp.getElementType();
+    return elt.isF64() || elt.isF32();
+  }
+  return false;
+}
+
 LogicalResult SparseTensorEncodingAttr::verifyEncoding(
     ArrayRef<Size> dimShape, Type elementType,
     function_ref<InFlightDiagnostic()> emitError) const {
@@ -964,6 +976,8 @@ LogicalResult SparseTensorEncodingAttr::verifyEncoding(
       return emitError() << "implicit value must be zero";
     }
   }
+  if (!isValidPrimaryType(elementType))
+    return emitError() << "invalid primary type";
   return success();
 }
 
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
index c9bcefe92de79..0e88d31dae8e8 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
@@ -383,9 +383,6 @@ class SparseTensorNewConverter : public OpConversionPattern<NewOp> {
     const auto stt = getSparseTensorType(op);
     if (!stt.hasEncoding())
       return failure();
-    // Verify that the element type is supported by the runtime library.
-    if (!isValidPrimaryType(stt.getElementType()))
-      return rewriter.notifyMatchFailure(op, "unsupported element type");
     // Construct the `reader` opening method calls.
     SmallVector<Value> dimSizesValues;
     Value dimSizesBuffer;
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
index 614c29ba68481..3c6a905ebb696 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
@@ -99,18 +99,6 @@ StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix(Type tp) {
   return overheadTypeFunctionSuffix(overheadTypeEncoding(tp));
 }
 
-bool mlir::sparse_tensor::isValidPrimaryType(Type elemTp) {
-  if (elemTp.isF64() || elemTp.isF32() || elemTp.isF16() || elemTp.isBF16() ||
-      elemTp.isInteger(64) || elemTp.isInteger(32) || elemTp.isInteger(16) ||
-      elemTp.isInteger(8))
-    return true;
-  if (auto complexTp = dyn_cast<ComplexType>(elemTp)) {
-    Type elt = complexTp.getElementType();
-    return elt.isF64() || elt.isF32();
-  }
-  return false;
-}
-
 PrimaryType mlir::sparse_tensor::primaryTypeEncoding(Type elemTp) {
   if (elemTp.isF64())
     return PrimaryType::kF64;
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
index 98e351a371ac1..1c10dd5566184 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
@@ -60,11 +60,6 @@ StringRef overheadTypeFunctionSuffix(OverheadType ot);
 /// Converts an overhead storage type to its function-name suffix.
 StringRef overheadTypeFunctionSuffix(Type overheadTp);
 
-/// Returns true if the given type is a valid sparse tensor element type
-/// supported by the runtime library (i.e., maps to a PrimaryType).
-/// Use this to guard calls to primaryTypeEncoding() with invalid types.
-bool isValidPrimaryType(Type elemTp);
-
 /// Converts a primary storage type to its internal type-encoding.
 PrimaryType primaryTypeEncoding(Type elemTp);
 
diff --git a/mlir/test/Dialect/SparseTensor/constant_index_map.mlir b/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
index cf1eb3e9e44f5..6c6e4579d4f11 100644
--- a/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
+++ b/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
@@ -8,34 +8,34 @@
 #SpVec = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
 
 // CHECK-LABEL:   func.func @main(
-// CHECK-SAME:      %[[VAL_0:.*0]]: tensor<1x77xi1>,
-// CHECK-SAME:      %[[VAL_1:.*1]]: tensor<1x77xi1>) -> tensor<77xi1, #{{.*}}> {
+// CHECK-SAME:      %[[VAL_0:.*0]]: tensor<1x77xi32>,
+// CHECK-SAME:      %[[VAL_1:.*1]]: tensor<1x77xi32>) -> tensor<77xi32, #{{.*}}> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 77 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = tensor.empty() : tensor<77xi1, #{{.*}}>
-// CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<1x77xi1>
-// CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_buffer %[[VAL_1]] : tensor<1x77xi1>
-// CHECK:           %[[VAL_8:.*]] = scf.for %[[VAL_9:.*]] = %[[VAL_3]] to %[[VAL_2]] step %[[VAL_4]] iter_args(%[[VAL_10:.*]] = %[[VAL_5]]) -> (tensor<77xi1, #{{.*}}>) {
-// CHECK:             %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi1>
-// CHECK:             %[[VAL_12:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi1>
-// CHECK:             %[[VAL_13:.*]] = arith.addi %[[VAL_11]], %[[VAL_12]] : i1
-// CHECK:             %[[VAL_14:.*]] = tensor.insert %[[VAL_13]] into %[[VAL_10]]{{\[}}%[[VAL_9]]] : tensor<77xi1, #{{.*}}>
-// CHECK:             scf.yield %[[VAL_14]] : tensor<77xi1, #{{.*}}>
+// CHECK-DAG:       %[[VAL_5:.*]] = tensor.empty() : tensor<77xi32, #{{.*}}>
+// CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<1x77xi32>
+// CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_buffer %[[VAL_1]] : tensor<1x77xi32>
+// CHECK:           %[[VAL_8:.*]] = scf.for %[[VAL_9:.*]] = %[[VAL_3]] to %[[VAL_2]] step %[[VAL_4]] iter_args(%[[VAL_10:.*]] = %[[VAL_5]]) -> (tensor<77xi32, #{{.*}}>) {
+// CHECK:             %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi32>
+// CHECK:             %[[VAL_12:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi32>
+// CHECK:             %[[VAL_13:.*]] = arith.addi %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:             %[[VAL_14:.*]] = tensor.insert %[[VAL_13]] into %[[VAL_10]]{{\[}}%[[VAL_9]]] : tensor<77xi32, #{{.*}}>
+// CHECK:             scf.yield %[[VAL_14]] : tensor<77xi32, #{{.*}}>
 // CHECK:           }
-// CHECK:           %[[VAL_15:.*]] = sparse_tensor.load %[[VAL_16:.*]] hasInserts : tensor<77xi1, #{{.*}}>
-// CHECK:           return %[[VAL_15]] : tensor<77xi1, #{{.*}}>
+// CHECK:           %[[VAL_15:.*]] = sparse_tensor.load %[[VAL_16:.*]] hasInserts : tensor<77xi32, #{{.*}}>
+// CHECK:           return %[[VAL_15]] : tensor<77xi32, #{{.*}}>
 // CHECK:         }
-func.func @main(%arg0: tensor<1x77xi1>, %arg1: tensor<1x77xi1>) -> tensor<77xi1, #SpVec> {
-  %0 = tensor.empty() : tensor<77xi1, #SpVec>
+func.func @main(%arg0: tensor<1x77xi32>, %arg1: tensor<1x77xi32>) -> tensor<77xi32, #SpVec> {
+  %0 = tensor.empty() : tensor<77xi32, #SpVec>
   %1 = linalg.generic {
     indexing_maps = [#map1, #map1, #map2],
     iterator_types = ["parallel"]}
-    ins(%arg0, %arg1 : tensor<1x77xi1>, tensor<1x77xi1>)
-    outs(%0 : tensor<77xi1, #SpVec>) {
-  ^bb0(%in: i1, %in_0: i1, %out: i1):
-    %2 = arith.addi %in, %in_0 : i1
-    linalg.yield %2 : i1
-  } -> tensor<77xi1, #SpVec>
-  return %1 : tensor<77xi1, #SpVec>
+    ins(%arg0, %arg1 : tensor<1x77xi32>, tensor<1x77xi32>)
+    outs(%0 : tensor<77xi32, #SpVec>) {
+  ^bb0(%in: i32, %in_0: i32, %out: i32):
+    %2 = arith.addi %in, %in_0 : i32
+    linalg.yield %2 : i32
+  } -> tensor<77xi32, #SpVec>
+  return %1 : tensor<77xi32, #SpVec>
 }
diff --git a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
index 456c1898a9311..7e95be495b333 100644
--- a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
+++ b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
@@ -8,7 +8,104 @@
 #sparse = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
 
 func.func @new_index_elem_type(%arg0: index) {
-  // expected-error at +1 {{failed to legalize operation 'sparse_tensor.new'}}
+  // expected-error at +1 {{invalid primary type}}
   %0 = sparse_tensor.new %arg0 : index to tensor<?xindex, #sparse>
   return
 }
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @alloc_tensor_i1_elem_type(%sz: index) -> tensor<?xi1, #sparse1d> {
+  %0 = bufferization.alloc_tensor(%sz) : tensor<?xi1, #sparse1d>
+  return %0 : tensor<?xi1, #sparse1d>
+}
+
+// -----
+
+#unordered_coo = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : compressed(nonunique, nonordered), d1 : singleton(nonordered))}>
+#ordered_coo = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton)}>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @reorder_coo_i1_elem_type(%arg0: tensor<?x?xi1, #unordered_coo>)
+    -> tensor<?x?xi1, #ordered_coo> {
+  %0 = sparse_tensor.reorder_coo quick_sort %arg0
+    : tensor<?x?xi1, #unordered_coo> to tensor<?x?xi1, #ordered_coo>
+  return %0 : tensor<?x?xi1, #ordered_coo>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +2 {{invalid primary type}}
+func.func @assemble_i1_elem_type(%pos: tensor<2xindex>, %coords: tensor<4x1xindex>,
+                                 %vals: tensor<4xi1>) -> tensor<8xi1, #sparse1d> {
+  %0 = sparse_tensor.assemble (%pos, %coords), %vals
+    : (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1> to tensor<8xi1, #sparse1d>
+  return %0 : tensor<8xi1, #sparse1d>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @values_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> memref<?xi1> {
+  %0 = sparse_tensor.values %arg0 : tensor<8xi1, #sparse1d> to memref<?xi1>
+  return %0 : memref<?xi1>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @number_of_entries_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> index {
+  %0 = sparse_tensor.number_of_entries %arg0 : tensor<8xi1, #sparse1d>
+  return %0 : index
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @disassemble_i1_elem_type(%sp: tensor<8xi1, #sparse1d>,
+                                      %op: tensor<2xindex>, %oi: tensor<4x1xindex>,
+                                      %od: tensor<4xi1>)
+    -> (tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>) {
+  %rp, %ri, %d, %rpl, %ril, %dl = sparse_tensor.disassemble %sp
+    : tensor<8xi1, #sparse1d>
+    out_lvls(%op, %oi : tensor<2xindex>, tensor<4x1xindex>)
+    out_vals(%od : tensor<4xi1>)
+    -> (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1>, (index, index), index
+  return %rp, %ri, %d : tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @insert_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>, %idx: index, %val: i1)
+    -> tensor<8xi1, #sparse1d> {
+  %0 = tensor.insert %val into %arg0[%idx] : tensor<8xi1, #sparse1d>
+  return %0 : tensor<8xi1, #sparse1d>
+}
+
+// -----
+
+#csr = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : dense, d1 : compressed)}>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @compress_i1_elem_type(%tensor: tensor<8x8xi1, #csr>,
+                                 %values: memref<?xi1>, %filled: memref<?xi1>,
+                                 %added: memref<?xindex>, %count: index, %i: index)
+    -> tensor<8x8xi1, #csr> {
+  %0 = sparse_tensor.compress %values, %filled, %added, %count into %tensor[%i]
+    : memref<?xi1>, memref<?xi1>, memref<?xindex>, tensor<8x8xi1, #csr>
+  return %0 : tensor<8x8xi1, #csr>
+}
diff --git a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
index 6effbbf98abb7..5eb428f197a75 100644
--- a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
@@ -8,52 +8,52 @@
 // Check that we vectorize reductions with ori.
 
 // CHECK-ON-LABEL:   func.func @sparse_reduction_ori(
-// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-ON-DAG:       %[[VAL_2:.*]] = arith.constant 8 : index
-// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
+// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
+// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
 // CHECK-ON:           %[[VAL_10:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i13 to vector<8xi13>
-// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi13>) {
+// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i32 to vector<8xi32>
+// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi32>) {
 // CHECK-ON:             %[[VAL_16:.*]] = affine.min #map(%[[VAL_11]], %[[VAL_14]]){{\[}}%[[VAL_2]]]
 // CHECK-ON:             %[[VAL_17:.*]] = vector.create_mask %[[VAL_16]] : vector<8xi1>
-// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi13>, vector<8xi1>, vector<8xi13> into vector<8xi13>
-// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_15]], %[[VAL_18]] : vector<8xi13>
-// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi13>
-// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi13>
+// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi32>, vector<8xi1>, vector<8xi32> into vector<8xi32>
+// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_15]], %[[VAL_18]] : vector<8xi32>
+// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi32>
+// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi32>
 // CHECK-ON:           } {"Emitted from" = "linalg.generic"}
-// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi13> into i13
-// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i13>
-// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i13>
-// CHECK-ON:           return %[[VAL_23]] : tensor<i13>
+// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi32> into i32
+// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i32>
+// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i32>
+// CHECK-ON:           return %[[VAL_23]] : tensor<i32>
 // CHECK-ON:         }
 //
 // CHECK-OFF-LABEL:   func.func @sparse_reduction_ori(
-// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
+// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK-OFF:           %[[VAL_9:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_3]]] : memref<?xindex>
-// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i13) {
-// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi13>
-// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_12]], %[[VAL_13]] : i13
-// CHECK-OFF:             scf.yield %[[VAL_14]] : i13
+// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i32) {
+// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi32>
+// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK-OFF:             scf.yield %[[VAL_14]] : i32
 // CHECK-OFF:           } {"Emitted from" = "linalg.generic"}
-// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i13>
-// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i13>
-// CHECK-OFF:           return %[[VAL_16]] : tensor<i13>
+// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i32>
+// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i32>
+// CHECK-OFF:           return %[[VAL_16]] : tensor<i32>
 // CHECK-OFF:         }
 #SparseVector = #sparse_tensor.encoding<{map = (d0) -> (d0 : compressed)}>
 
@@ -65,17 +65,17 @@
   iterator_types = ["reduction"]
 }
 
-func.func @sparse_reduction_ori(%argx: tensor<i13>,
-                                %arga: tensor<?xi13, #SparseVector>)
- -> tensor<i13> {
+func.func @sparse_reduction_ori(%argx: tensor<i32>,
+                                %arga: tensor<?xi32, #SparseVector>)
+ -> tensor<i32> {
   %0 = linalg.generic #trait
-     ins(%arga: tensor<?xi13, #SparseVector>)
-      outs(%argx: tensor<i13>) {
-      ^bb(%a: i13, %x: i13):
-        %t = arith.ori %x, %a: i13
-        linalg.yield %t : i13
-  } -> tensor<i13>
-  return %0 : tensor<i13>
+     ins(%arga: tensor<?xi32, #SparseVector>)
+      outs(%argx: tensor<i32>) {
+      ^bb(%a: i32, %x: i32):
+        %t = arith.ori %x, %a: i32
+        linalg.yield %t : i32
+  } -> tensor<i32>
+  return %0 : tensor<i32>
 }
 
 // -----
@@ -85,52 +85,52 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
 // irrespective to where the accumulator appears on commutative operations.
 
 // CHECK-ON-LABEL:   func.func @sparse_reduction_ori_accumulator_on_rhs(
-// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-ON-DAG:       %[[VAL_2:.*]] = arith.constant 8 : index
-// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
+// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
+// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
 // CHECK-ON:           %[[VAL_10:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i13 to vector<8xi13>
-// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi13>) {
+// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i32 to vector<8xi32>
+// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi32>) {
 // CHECK-ON:             %[[VAL_16:.*]] = affine.min #map(%[[VAL_11]], %[[VAL_14]]){{\[}}%[[VAL_2]]]
 // CHECK-ON:             %[[VAL_17:.*]] = vector.create_mask %[[VAL_16]] : vector<8xi1>
-// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi13>, vector<8xi1>, vector<8xi13> into vector<8xi13>
-// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_18]], %[[VAL_15]] : vector<8xi13>
-// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi13>
-// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi13>
+// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi32>, vector<8xi1>, vector<8xi32> into vector<8xi32>
+// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_18]], %[[VAL_15]] : vector<8xi32>
+// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi32>
+// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi32>
 // CHECK-ON:           } {"Emitted from" = "linalg.generic"}
-// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi13> into i13
-// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i13>
-// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i13>
-// CHECK-ON:           return %[[VAL_23]] : tensor<i13>
+// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi32> into i32
+// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i32>
+// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i32>
+// CHECK-ON:           return %[[VAL_23]] : tensor<i32>
 // CHECK-ON:         }
 //
 // CHECK-OFF-LABEL:   func.func @sparse_reduction_ori_accumulator_on_rhs(
-// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
+// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK-OFF:           %[[VAL_9:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_3]]] : memref<?xindex>
-// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i13) {
-// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi13>
-// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_13]], %[[VAL_12]] : i13
-// CHECK-OFF:             scf.yield %[[VAL_14]] : i13
+// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i32) {
+// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi32>
+// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_13]], %[[VAL_12]] : i32
+// CHECK-OFF:             scf.yield %[[VAL_14]] : i32
 // CHECK-OFF:           } {"Emitted from" = "linalg.generic"}
-// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i13>
-// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i13>
-// CHECK-OFF:           return %[[VAL_16]] : tensor<i13>
+// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i32>
+// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i32>
+// CHECK-OFF:           return %[[VAL_16]] : tensor<i32>
 // CHECK-OFF:         }
 #SparseVector = #sparse_tensor.encoding<{map = (d0) -> (d0 : compressed)}>
 
@@ -142,17 +142,17 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
   iterator_types = ["reduction"]
 }
 
-func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i13>,
-                                                   %arga: tensor<?xi13, #SparseVector>)
- -> tensor<i13> {
+func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i32>,
+                                                   %arga: tensor<?xi32, #SparseVector>)
+ -> tensor<i32> {
   %0 = linalg.generic #trait
-     ins(%arga: tensor<?xi13, #SparseVector>)
-      outs(%argx: tensor<i13>) {
-      ^bb(%a: i13, %x: i13):
-        %t = arith.ori %a, %x: i13
-        linalg.yield %t : i13
-  } -> tensor<i13>
-  return %0 : tensor<i13>
+     ins(%arga: tensor<?xi32, #SparseVector>)
+      outs(%argx: tensor<i32>) {
+      ^bb(%a: i32, %x: i32):
+        %t = arith.ori %a, %x: i32
+        linalg.yield %t : i32
+  } -> tensor<i32>
+  return %0 : tensor<i32>
 }
 
 // -----


From llvmlistbot at llvm.org  Tue Jul 14 12:46:42 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 12:46:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a569222.f3151ff8.305c37.f226@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
----------------
beamandala wrote:

They're used to match the variable names that are automatically generated (v1, v2, etc.). I renamed the first to `ARG` and the second to `VAR` to specify that they represent the generated argument and variable names.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 12:53:15 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 12:53:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5693ab.af6ba3ed.1dc10f.8c7a@mx.google.com>


================
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
----------------
beamandala wrote:

There's a match failure in this case so none of the transformations are applied.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 12:53:47 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 12:53:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5693cb.743400b0.2067c9.d182@mx.google.com>


https://github.com/beamandala edited https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 12:56:21 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 12:56:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a569465.66c0679a.3aa103.f0b8@mx.google.com>


================
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
----------------
beamandala wrote:

I also realized while looking at this that the map and string includes are added regardless of if a pattern is matched or not. I updated the pass so that they're only added if there wasn't a match failure.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 12:56:33 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 12:56:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a569471.07d7ca98.9dee3.6a35@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
----------------
banach-space wrote:

The auto-generated names are meant to be refined to better match the context. 

In general, in MLIR we encourage well documented tests: https://mlir.llvm.org/getting_started/TestingGuide/#test-formatting-best-practices

It's worth reading the whole document.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 12:59:10 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 12:59:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a56950e.d76d81af.39f9be.95bf@mx.google.com>


================
@@ -60,43 +60,37 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
-def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
     This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
     EmitC classes, enabling runtime lookup of class fields by name. 
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
     `field-attr-name` option) that is an array containing a single string
-    attribute.
-
-
+    attribute. If a field possesses both the attribute specified by
+    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
+    the `field-attr-name` attribute takes precedence and the field will be
+    included in the reflection map.
+    
     Example:
 
     ```mlir
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.func @execute() { ... }
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      emitc.func @operator()() { ... }
----------------
beamandala wrote:

I changed it to `@operator()()` because that's the function that's generated by the `wrap-emitc-func-in-class` pass which is run before this pass in our use case. This isn't a hard pre req to running this pass though so I'll change it back to `execute()`.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 13:00:04 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 13:00:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a569544.f06f4994.1fc857.abb6@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/205464

>From ce76fb7aa44a3320fdb45acc6e011758ed3afc77 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 1/7] [mlir][EmitC] Create a pass to add a reflection map to a
 class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |   2 +
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  52 +++++
 .../Dialect/EmitC/Transforms/Transforms.h     |   7 +
 .../EmitC/Transforms/AddReflectionMap.cpp     | 181 ++++++++++++++++++
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |   1 +
 .../Dialect/EmitC/add-reflection-map.mlir     |  58 ++++++
 6 files changed, 301 insertions(+)
 create mode 100644 mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
 create mode 100644 mlir/test/Dialect/EmitC/add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 1af4aa06fa811..56596110a1e90 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,6 +16,8 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
+#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index c34c3303a6ab3..6da57c07a8047 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,4 +60,56 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `field-attr-name` option) that is an array containing a single string
+    attribute.
+
+
+    Example:
+
+    ```mlir
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.func @execute() { ... }
+    }
+
+    Becomes:
+
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
+      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
+          -> !emitc.ptr<!emitc.opaque<"char">> {
+        %0 = get_field @reflectionMap
+          : !emitc.opaque<"const std::map<std::string, char*>">
+        %1 = member_call_opaque %0 "at"(%arg0)
+          : !emitc.opaque<"const std::map<std::string, char*>">,
+            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      }
+      emitc.func @execute() { ... }
+    }
+    ```
+  }];
+  let dependentDialects = ["emitc::EmitCDialect"];
+  let options = [
+    Option<"fieldAttrName", "field-attr-name", "std::string",
+      /*default=*/"",
+      "Attribute key used to extract field names from a field's "
+      "attribute dictionary">,
+    ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
+      "Attribute keys that, if present on a field, exclude it "
+      "from the reflection map">
+  ];
+}
+
 #endif // MLIR_DIALECT_EMITC_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 791e545a8edcf..8eadfe9a2460d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/EmitC/IR/EmitC.h"
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -37,6 +38,12 @@ void populateWrapFuncInClass(
     RewritePatternSet &patterns, StringRef funcName,
     DenseMap<FuncOp, llvm::DenseSet<GlobalOp>> &globalsToMove);
 
+//===----------------------------------------------------------------------===//
+// The AddReflectionMap pass.
+//===----------------------------------------------------------------------===//
+
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+
 } // namespace emitc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
new file mode 100644
index 0000000000000..c1588463ffb7a
--- /dev/null
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -0,0 +1,181 @@
+//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(
+      builder, module.getLoc(), includeAttr,
+      /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class AddReflectionMapPass
+    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
+  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+
+    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+        
+        hasMapHdr = include == mapLibraryHeader;
+        hasStringHdr = include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect all field names
+    std::vector<std::pair<std::string, std::string>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
+        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
+          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
+          fieldNames.emplace_back(stringAttr.getValue().str(),
+                                  fieldOp.getName().str());
+          return;
+        }
+      }
+
+      bool shouldIgnore = false;
+      for (const std::string &ignoreAttr : excludedFieldAttrs) {
+        if (fieldOp->hasAttr(ignoreAttr)) {
+          shouldIgnore = true;
+          break;
+        }
+      }
+
+      if (shouldIgnore)
+        return;
+
+      fieldOp.emitError()
+          << "FieldOp must have a dictionary attribute named '"
+          << fieldAttrName << "'"
+          << "with an array containing a string attribute";
+    });
+
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
+          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+    }
+    reflectionMapContents += " }";
+
+    if (FuncOp executeFunc =
+            classOp.lookupSymbol<FuncOp>("operator()"))
+      rewriter.setInsertionPoint(executeFunc);
+    else {
+      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
+                             "to add reflection map";
+      return failure();
+    }
+
+    FieldOp reflectionMapField = FieldOp::create(
+        rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
+        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+
+    // Create getBufferForName method
+    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    FuncOp getBufferForNameFunc = FuncOp::create(
+        rewriter, reflectionMapField->getLoc(), "getBufferForName",
+        FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
+
+    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    rewriter.setInsertionPointToStart(body);
+    GetFieldOp mapField = GetFieldOp::create(
+        rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
+    Value nameArg = body->getArgument(0);
+    MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
+        rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
+        "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+
+    return success();
+  }
+
+private:
+  /// The name of the attribute on FieldOps that contains the field name
+  /// metadata for the reflection map.
+  StringRef fieldAttrName;
+
+  /// Attributes that, if present on a field, exclude it from the
+  /// reflection map.
+  llvm::SmallVector<std::string> excludedFieldAttrs;
+};
+
+void mlir::emitc::populateAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    llvm::ArrayRef<std::string> excludedFieldAttrs) {
+  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                      excludedFieldAttrs);
+}
\ No newline at end of file
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index baf67afc30072..dd8f014dc4737 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
+  AddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
new file mode 100644
index 0000000000000..a1ec0356e19f4
--- /dev/null
+++ b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+
+
+// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+// Test that a reflection map is created for fields with a certain named attribute
+// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
\ No newline at end of file

>From 60d4d88576ecc713db0a2cf7ae184308c3cd745f Mon Sep 17 00:00:00 2001
From: Bhavesh M <85327930+beamandala at users.noreply.github.com>
Date: Tue, 23 Jun 2026 02:37:42 +0530
Subject: [PATCH 2/7] [mlir][EmitC] Make `GlobalOps` `FieldOps` in
 wrap-emitc-func-in-class pass (#203641)

Update the `WrapFuncInClassPass` pass so that `GlobalOp`s are moved into
the `ClassOp` as `FieldOps`. This respects MLIR's behavior of resolving
references to the closest parent operation that defines a symbol table
which is the `ClassOp` that we are creating in this pass.

Without this change, references to a `GlobalOp` in `GetGlobalOp` are
failing to resolve.

Details:
- Identify `GlobalOp`s
- Create a `FieldOp` within the `ClassOp` for each `GlobalOp`
- Delete the `GlobalOp`s after all functions have been wrapped in a
class. Doing this after every function can cause an error when multiple
functions refer to the same `GlobalOp`(s) which would be deleted after
the first function is wrapped in a class.

Also renamed `fName` parameter in `populateWrapFuncInClass` to
`funcName` to match naming in `WrapFuncInClass`.

Based on PR #153452. Key differences:
- No size is set for the `globalsToMove` `SmallVector` type because I'm
not sure if the number of global variables is consistent across
different models.
- `GlobalOp`s are deleted after all functions have been processed.
- Instead of directly cloning the `GlobalOp`, an equivalent `FieldOp` is
created
- `GetGlobalOp`s are translated to `GetFieldOp`s

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8eadfe9a2460d..4c8d3978af0f1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,6 +13,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace mlir {
 namespace emitc {

>From b6af183f868036d825f4133a2671b67e5a9b643e Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 3/7] [mlir][EmitC] Create a pass to add a reflection map to a
 class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 4c8d3978af0f1..5a4781dc31ea8 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {

>From 8d7429a032c6e0d91d1857ff5ef5c324ac35999f Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 24 Jun 2026 10:31:17 -0700
Subject: [PATCH 4/7] Formatting

---
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +--
 .../EmitC/Transforms/AddReflectionMap.cpp     | 47 +++++++++++--------
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 5a4781dc31ea8..8f1784f61f540 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,8 +13,6 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -44,7 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
+                                      StringRef fieldAttrName,
+                                      ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
index c1588463ffb7a..d444f3abbf4b1 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -31,9 +31,8 @@ constexpr const char *stringLibraryHeader = "string";
 
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
-  return IncludeOp::create(
-      builder, module.getLoc(), includeAttr,
-      /*is_standard_include=*/builder.getUnitAttr());
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
 }
 
 class AddReflectionMapPass
@@ -43,7 +42,8 @@ class AddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+    populateAddReflectionMapPatterns(patterns, fieldAttrName,
+                                     excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -55,7 +55,7 @@ class AddReflectionMapPass
 
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
-        
+
         hasMapHdr = include == mapLibraryHeader;
         hasStringHdr = include == stringLibraryHeader;
       }
@@ -82,7 +82,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   AddReflectionMapClass(MLIRContext *context, StringRef attrName,
                         llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
-        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+        excludedFieldAttrs(excludedFieldAttrs.begin(),
+                           excludedFieldAttrs.end()) {}
 
   LogicalResult matchAndRewrite(ClassOp classOp,
                                 PatternRewriter &rewriter) const override {
@@ -114,10 +115,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError()
-          << "FieldOp must have a dictionary attribute named '"
-          << fieldAttrName << "'"
-          << "with an array containing a string attribute";
+      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
+                          << fieldAttrName << "'"
+                          << "with an array containing a string attribute";
     });
 
     std::string reflectionMapContents;
@@ -129,28 +129,34 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc =
-            classOp.lookupSymbol<FuncOp>("operator()"))
+    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
-                             "to add reflection map";
+      classOp.emitError()
+          << "ClassOp must contain a function named 'operator()' "
+             "to add reflection map";
       return failure();
     }
 
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
-        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+        TypeAttr::get(mapType),
+        emitc::OpaqueAttr::get(context, reflectionMapContents));
 
     // Create getBufferForName method
-    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
-    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
-    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    emitc::OpaqueType nameType =
+        emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType =
+        emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType =
+        emitc::PointerType::get(rewriter.getContext(), charType);
     FuncOp getBufferForNameFunc = FuncOp::create(
         rewriter, reflectionMapField->getLoc(), "getBufferForName",
         FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
 
-    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    Block *body =
+        rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType},
+                             {reflectionMapField->getLoc()});
     rewriter.setInsertionPointToStart(body);
     GetFieldOp mapField = GetFieldOp::create(
         rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
@@ -158,7 +164,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
         rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
         "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
-    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(),
+                     lookupCall.getResult(0));
 
     return success();
   }

>From 266f21b78f0534b8bdf8354587cc883a81464a48 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Fri, 26 Jun 2026 16:04:13 -0700
Subject: [PATCH 5/7] Address review comments

---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |  2 +-
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 10 ++-
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +-
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |  2 +-
 ...ectionMap.cpp => MLGOAddReflectionMap.cpp} | 76 +++++++++--------
 ...-map.mlir => mlgo-add-reflection-map.mlir} | 84 +++++++++++++++++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 57 +++++++++++++
 7 files changed, 188 insertions(+), 49 deletions(-)
 rename mlir/lib/Dialect/EmitC/Transforms/{AddReflectionMap.cpp => MLGOAddReflectionMap.cpp} (71%)
 rename mlir/test/Dialect/EmitC/{add-reflection-map.mlir => mlgo-add-reflection-map.mlir} (52%)
 create mode 100644 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 56596110a1e90..4f46d63774625 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,7 +16,7 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
-#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DECL_MLGOADDREFLECTIONMAPPASS
 
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 6da57c07a8047..3d17f53032a7a 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,7 +60,7 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
-def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
     This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
@@ -68,9 +68,11 @@ def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
     `field-attr-name` option) that is an array containing a single string
-    attribute.
-
-
+    attribute. If a field possesses both the attribute specified by
+    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
+    the `field-attr-name` attribute takes precedence and the field will be
+    included in the reflection map.
+    
     Example:
 
     ```mlir
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8f1784f61f540..6425614e1fbcd 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -42,9 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
-                                      StringRef fieldAttrName,
-                                      ArrayRef<std::string> excludedFieldAttrs);
+void populateMLGOAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index dd8f014dc4737..ede2cb5e208f0 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,7 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
-  AddReflectionMap.cpp
+  MLGOAddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
similarity index 71%
rename from mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
rename to mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index d444f3abbf4b1..9f506b10c722a 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -1,7 +1,7 @@
-//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt license information.
+// See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
@@ -22,7 +22,7 @@ using namespace emitc;
 
 namespace mlir {
 namespace emitc {
-#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 namespace {
@@ -35,15 +35,15 @@ IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
                            /*is_standard_include=*/builder.getUnitAttr());
 }
 
-class AddReflectionMapPass
-    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
-  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
   void runOnOperation() override {
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName,
-                                     excludedFieldAttrs);
+    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+                                         excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -56,8 +56,8 @@ class AddReflectionMapPass
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
 
-        hasMapHdr = include == mapLibraryHeader;
-        hasStringHdr = include == stringLibraryHeader;
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
       }
 
       if (hasMapHdr && hasStringHdr)
@@ -77,10 +77,10 @@ class AddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
-class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
-                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                            llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
@@ -93,33 +93,35 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
         context, "const std::map<std::string, char*>");
 
     // Collect all field names
-    std::vector<std::pair<std::string, std::string>> fieldNames;
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
-        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
-          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
-          fieldNames.emplace_back(stringAttr.getValue().str(),
-                                  fieldOp.getName().str());
-          return;
-        }
-      }
-
-      bool shouldIgnore = false;
-      for (const std::string &ignoreAttr : excludedFieldAttrs) {
-        if (fieldOp->hasAttr(ignoreAttr)) {
-          shouldIgnore = true;
-          break;
+      if (auto arrayAttr =
+              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
+        if (!arrayAttr.empty()) {
+          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
+            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
+            return;
+          }
         }
       }
 
+      bool shouldIgnore =
+          llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
+            return fieldOp->hasAttr(ignoreAttr);
+          });
       if (shouldIgnore)
         return;
 
       fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "'"
+                          << fieldAttrName << "' "
                           << "with an array containing a string attribute";
+      hasError = true;
     });
 
+    if (hasError || fieldNames.empty())
+      return failure();
+
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
@@ -138,6 +140,14 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       return failure();
     }
 
+    // To generate the following C++ code
+    // const std::map<std::string, char*> reflectionMap = {
+    //  { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+    //  { "some_feature", reinterpret_cast<char*>(&fieldName1) },
+    //  ...
+    // };
+    // This can be used to retrieve a pointer to the field's contents given the
+    // attribute string identifying the field
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
         TypeAttr::get(mapType),
@@ -180,9 +190,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   llvm::SmallVector<std::string> excludedFieldAttrs;
 };
 
-void mlir::emitc::populateAddReflectionMapPatterns(
+void mlir::emitc::populateMLGOAddReflectionMapPatterns(
     RewritePatternSet &patterns, StringRef fieldAttrName,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                      excludedFieldAttrs);
-}
\ No newline at end of file
+  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                          excludedFieldAttrs);
+}
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
similarity index 52%
rename from mlir/test/Dialect/EmitC/add-reflection-map.mlir
rename to mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index a1ec0356e19f4..2bea6cfaf2677 100644
--- a/mlir/test/Dialect/EmitC/add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,7 +1,7 @@
-// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
 
-
-// Tests that a reflection map is created for fields with a certain attribute.
+/// Tests that a reflection map is created for fields with a certain attribute.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -29,10 +29,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:  }
 
 // -----
-// Test that a reflection map is created for fields with a certain named attribute
-// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClass {
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -41,7 +42,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @actionClassExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @actionClassNoFields {
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if the ClassOp doesn't have
+/// a FunctionOp named operator()
+
+// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
+emitc.class @actionClassNoOperator {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if a FieldOp has the specified
+/// dictionary attribute with an array containing a type other than string
+
+emitc.class @actionClassNonStringAttr {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
\ No newline at end of file
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
new file mode 100644
index 0000000000000..eb8a8e58c8b77
--- /dev/null
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 648cdfad719256977718ce44ced1ee78766430ce Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Mon, 6 Jul 2026 12:50:33 -0700
Subject: [PATCH 6/7] update MLGOAddReflectionMap to use notifyMatchFailure,
 update example to be more minimal and clear about the transformation, use
 range based for loop in reflection map construction

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 24 ++++++------------
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 25 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        |  5 +---
 3 files changed, 24 insertions(+), 30 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 3d17f53032a7a..29b053f3469c7 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -77,28 +77,20 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
 
     ```mlir
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.func @execute() { ... }
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      emitc.func @operator()() { ... }
     }
 
     Becomes:
 
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
-      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
-          -> !emitc.ptr<!emitc.opaque<"char">> {
-        %0 = get_field @reflectionMap
-          : !emitc.opaque<"const std::map<std::string, char*>">
-        %1 = member_call_opaque %0 "at"(%arg0)
-          : !emitc.opaque<"const std::map<std::string, char*>">,
-            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
-        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      // reflectionMap maps the string "feature0" to the address of @field0
+      emitc.field @reflectionMap : const std::map<string, char*> = {{"feature0", &field0}}
+      emitc.func @getBufferForName(%name: string) -> char* {
+        return reflectionMap.at(%name)
       }
-      emitc.func @execute() { ... }
+      emitc.func @operator()() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 9f506b10c722a..05a850c0e7551 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -113,9 +113,11 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "' "
-                          << "with an array containing a string attribute";
+      (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
+        diag << "FieldOp must have a dictionary attribute named '"
+             << fieldAttrName << "' "
+             << "with an array containing a string attribute";
+      });
       hasError = true;
     });
 
@@ -124,20 +126,23 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
-    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+
+      first = false;
       reflectionMapContents += llvm::formatv(
-          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
-          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
     }
     reflectionMapContents += " }";
 
     if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError()
-          << "ClassOp must contain a function named 'operator()' "
-             "to add reflection map";
-      return failure();
+      return rewriter.notifyMatchFailure(
+          classOp, "ClassOp must contain a function named 'operator()' "
+                   "to add reflection map");
     }
 
     // To generate the following C++ code
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2bea6cfaf2677..1f8647910ada9 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -63,7 +63,6 @@ emitc.class @actionClassExcluded {
 /// Test that the pass leaves IR unchanged if fields don't have any attributes
 
 emitc.class @actionClassNoAttrs {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -98,7 +97,6 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
 emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
@@ -113,7 +111,6 @@ emitc.class @actionClassNoOperator {
 /// dictionary attribute with an array containing a type other than string
 
 emitc.class @actionClassNonStringAttr {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return

>From c25c939e5c4a5e0e8687e26562dbb9d6230c73b2 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 12:59:46 -0700
Subject: [PATCH 7/7] Update tests with formatting and wording changes, update
 pass description, only add headers if there was a match

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  4 +--
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 16 ++++++++-
 .../EmitC/mlgo-add-reflection-map.mlir        |  8 +++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 34 +++++++++++++++----
 4 files changed, 51 insertions(+), 11 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 29b053f3469c7..1e245e00c9b2d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -78,7 +78,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
     ```mlir
     emitc.class @MyClass {
       emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
 
     Becomes:
@@ -90,7 +90,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
       emitc.func @getBufferForName(%name: string) -> char* {
         return reflectionMap.at(%name)
       }
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 05a850c0e7551..666d9bc01dfdf 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -29,6 +29,15 @@ namespace {
 constexpr const char *mapLibraryHeader = "map";
 constexpr const char *stringLibraryHeader = "string";
 
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
   return IncludeOp::create(builder, module.getLoc(), includeAttr,
@@ -45,7 +54,12 @@ class MLGOAddReflectionMapPass
     populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
                                          excludedFieldAttrs);
 
-    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    if (!listener.patternApplied)
+      return;
+
     bool hasMapHdr = false;
     bool hasStringHdr = false;
     for (auto &op : *moduleOp.getBody()) {
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 1f8647910ada9..030de796b308c 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -60,7 +60,7 @@ emitc.class @actionClassExcluded {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if fields don't have any attributes
+/// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
 
 emitc.class @actionClassNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
@@ -69,6 +69,7 @@ emitc.class @actionClassNoAttrs {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @"operator()"() {
@@ -78,7 +79,7 @@ emitc.class @actionClassNoAttrs {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
 emitc.class @actionClassNoFields {
   emitc.func @"operator()"() {
@@ -86,6 +87,7 @@ emitc.class @actionClassNoFields {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoFields {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
@@ -101,6 +103,7 @@ emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:  }
@@ -117,6 +120,7 @@ emitc.class @actionClassNonStringAttr {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @"operator()"() {
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index eb8a8e58c8b77..e37b09bc6d611 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
-// Test that a reflection map and lookup function are generated in the class.
+/// Test that a reflection map and lookup function are generated in the class.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -17,10 +17,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
-// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
-// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
-// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
-// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+// CHECK-SAME:                                                         { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[ARG:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAR:v[0-9]+]] = reflectionMap.at([[ARG]]);
+// CHECK-NEXT:      return [[VAR]];
 // CHECK-NEXT:    }
 // CHECK-NEXT:    void operator()() {
 // CHECK-NEXT:      return;
@@ -29,7 +30,7 @@ emitc.class @actionClass {
 
 // -----
 
-// Test that fields with excluded attributes are ignored.
+/// Test that fields with excluded attributes are ignored.
 
 emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -55,3 +56,24 @@ emitc.class @actionClassExcluded {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
+
+// -----
+
+/// Test that translation doesn't add headers if the class does not match the pass.
+
+emitc.class @actionClassNoAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-NOT:   #include <map>
+// CHECK-NOT:   #include <string>
+// CHECK:       class actionClassNoAttrs {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };


From llvmlistbot at llvm.org  Tue Jul 14 13:00:31 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 13:00:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a56955f.af6ba3ed.1dc10f.0849@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
----------------
beamandala wrote:

Fixed

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 13:00:39 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 14 Jul 2026 13:00:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a569567.c56fb102.7c9cd.92ab@mx.google.com>


================
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
----------------
beamandala wrote:

Fixed

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Tue Jul 14 13:46:02 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a00a.af6ba3ed.1dc10f.ac1b@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:10 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a012.2490598a.cf17c.0696@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:22 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a01e.d76d81af.39f9be.67d9@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:30 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a026.c56fb102.7c9cd.e098@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:37 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a02d.a1accbb4.244acd.8a5e@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:44 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a034.e55dc341.1f19bd.fccb@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:46:54 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:46:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a03e.eb165e5c.1817a5.58d6@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:05 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a049.4987c784.63cbc.fe66@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:15 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a053.a1accbb4.244acd.a138@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp) {
+        return WalkResult::advance();
+      }
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop) {
+      return;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:24 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a05c.f732ad81.3a4994.cc15@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp) {
+        return WalkResult::advance();
+      }
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop) {
+      return;
+    }
+    scf::ParallelOp threadLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+    if (!threadLoop) {
+      return;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:29 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a061.92ea0e2a.13fd37.f6d2@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp) {
+        return WalkResult::advance();
+      }
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop) {
+      return;
+    }
+    scf::ParallelOp threadLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+    if (!threadLoop) {
+      return;
+    }
+    scf::ParallelOp blockLoop = threadLoop->getParentOfType<scf::ParallelOp>();
+    if (!blockLoop) {
+      return;
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:37 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a069.6c8668f2.e4f82.2956@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp) {
+        return WalkResult::advance();
+      }
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop) {
+      return;
+    }
+    scf::ParallelOp threadLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+    if (!threadLoop) {
+      return;
+    }
+    scf::ParallelOp blockLoop = threadLoop->getParentOfType<scf::ParallelOp>();
+    if (!blockLoop) {
+      return;
+    }
+    mlir::acc::GPUParallelDimsAttr parDimsAttr =
+        mlir::acc::getParDimsAttr(blockLoop);
+    if (parDimsAttr.hasOnlyBlockLevel()) {
+      createBarrier(loopOp->getLoc(), parDimsAttr);
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:47:45 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:47:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a071.92ea0e2a.13fd37.facc@mx.google.com>


================
@@ -0,0 +1,3757 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName) {
+      name.push_back(llvm::toUpper(c));
+    }
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() ==
+          computeRegion) {
+        return;
+      }
+      for (Value extent : privatizeOp.getDynamicSizes()) {
+        getOrAppendInsBlockArg(extent);
+      }
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape()) {
+    if (dim == ShapedType::kDynamic) {
+      return false;
+    }
+  }
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1)) {
+      return false;
+    }
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr())) {
+    return true;
+  }
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop))) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second) {
+      continue;
+    }
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray()) {
+          hasThread |= pd.isAnyThread();
+        }
+        if (hasThread && reductionHasBlockContext(accArr)) {
+          return accArr;
+        }
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through)) {
+        worklist.append(through.begin(), through.end());
+      } else if (isa<ViewLikeOpInterface>(user)) {
+        worklist.append(user->result_begin(), user->result_end());
+      }
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult())) {
+    return std::nullopt;
+  }
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value()) {
+    return std::nullopt;
+  }
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    numElements *= dim;
+  }
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg) {
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+    }
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal))) {
+      staticBlockDimX = bdxVal.getSExtValue();
+    }
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr()) {
+      launch.setFunctionAttr(kernelFuncName);
+    }
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr()) {
+      launch.setModuleAttr(kernelModuleName);
+    }
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i) {
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+  }
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op :
+       computeRegion.getRegion().getBlocks().front().getOperations()) {
+    processOp(&op);
+  }
+
+  for (auto &parLoop : loopReductions) {
+    postprocessLoopReduction(parLoop);
+  }
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end()) {
+        continue;
+      }
+      if (!domInfo.dominates(it->second, loadOp.getOperation())) {
+        continue;
+      }
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed) {
+    return failure();
+  }
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref) {
+    return false;
+  }
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref) {
+      continue;
+    }
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims) {
+    if (parDim.isAnyBlock()) {
+      hasBlock = true;
+    }
+  }
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray()) {
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+            }
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ()) {
+            lowestParDim = parDim;
+          } else {
+            lowestParDim = parDim.getOneHigher();
+          }
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder()) {
+      break;
+    }
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate) {
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    } else {
+      predicate = cmp;
+    }
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock()) {
+      hasAnyBlock = true;
+    }
+    if (parDim.isThreadY()) {
+      hasThreadY = true;
+    }
+    if (parDim.isThreadX()) {
+      hasThreadX = true;
+    }
+  }
+
+  if (hasAnyBlock || hasThreadY) {
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  } else if (hasThreadX) {
+    createPerRowBarrier(loc);
+  }
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at different program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next)) {
+      return true;
+    }
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested) {
+      return true;
+    }
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty()) {
+      return false;
+    }
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p)) {
+      return cast<LoopLikeOpInterface>(p);
+    }
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp)) {
+        return cast<LoopLikeOpInterface>(p);
+      }
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op)) {
+    return true;
+  }
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op)) {
+    return true;
+  }
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp) {
+        return WalkResult::advance();
+      }
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop) {
+      return;
+    }
+    scf::ParallelOp threadLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+    if (!threadLoop) {
+      return;
+    }
+    scf::ParallelOp blockLoop = threadLoop->getParentOfType<scf::ParallelOp>();
+    if (!blockLoop) {
+      return;
+    }
+    mlir::acc::GPUParallelDimsAttr parDimsAttr =
+        mlir::acc::getParDimsAttr(blockLoop);
+    if (parDimsAttr.hasOnlyBlockLevel()) {
+      createBarrier(loopOp->getLoc(), parDimsAttr);
+    }
+    return;
+  }
+
+  // Parent is a non-sequential parallel loop.  Walk up to find the block-level
+  // ancestor and insert a barrier there.
+  scf::ParallelOp seqLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+  if (!seqLoop) {
+    // wsLoop is a worksharing loop directly under the compute region with no
+    // gang ancestor: a gang-redundant init loop (e.g. a thread-level loop that
+    // every gang runs to fill its own copy of gang-private shared memory). If
+    // it writes gang-private (block-only) memory that a later sibling loop
+    // reads, insert a workgroup barrier between them. The barrier sits just
+    // after the top-level loop where all threads converge, so it cannot
+    // deadlock.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(wsLoop)) {
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    }
----------------
clementval wrote:

no braces

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 13:48:23 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 14 Jul 2026 13:48:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a56a097.3cce8983.225732.7e3e@mx.google.com>


https://github.com/clementval commented:

I flagged couple of if/for statements that have braces where it should be omitted. There are more in the code. 

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Tue Jul 14 15:49:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 15:49:49 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][flang][acc] Preserve source
 array extents in data bounds` (PR #209618)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209618@github.com>
Message-ID: <6a56bd0d.b4a331f0.271e68.32a5@mx.google.com>


khaki3 wrote:

```fortran
real :: a(100,100)
integer :: i, j

!$acc parallel loop collapse(2) copy(a)
do j = 20, 29
  do i = 3, 5
    a(i,j) = 1.0
  end do
end do
```

Before specialization, `copy(a)` maps the whole array:

```text
dim 0: lb=0, ub=99, extent=100, sourceExtent=100, stride=1
dim 1: lb=0, ub=99, extent=100, sourceExtent=100, stride=100
```

Specialization narrows the mapped bounds:

```text
dim 0: lb=2,  ub=4,  extent=3,  sourceExtent=100, stride=1
dim 1: lb=19, ub=28, extent=10, sourceExtent=100, stride=100
```

`extent` changes because it describes the selected range. Leaving `extent=100` with `lb=2, ub=4` would produce inconsistent bounds. `sourceExtent` remains unchanged because specialization does not change the source shape.

A stride cannot generally recover the source extent. A stride of `100` may represent 100 contiguous elements or 50 elements with spacing two. The final dimension also has no following stride from which its extent could be inferred. Therefore extent and stride represent independent information.

https://github.com/llvm/llvm-project/pull/209618

From llvmlistbot at llvm.org  Tue Jul 14 15:57:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 15:57:59 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][flang][acc] Preserve source
 array extents in data bounds` (PR #209618)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209618@github.com>
Message-ID: <6a56bef7.4987c784.63cbc.c008@mx.google.com>


khaki3 wrote:

On second thought, I realized sourceExtent is redundant. I'll close this PR and try to calculate similar values from exiting parameters.

https://github.com/llvm/llvm-project/pull/209618

From llvmlistbot at llvm.org  Tue Jul 14 15:57:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 15:57:59 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][flang][acc] Preserve source
 array extents in data bounds` (PR #209618)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209618@github.com>
Message-ID: <6a56bef7.4987c784.63cbc.c00a@mx.google.com>


https://github.com/khaki3 closed https://github.com/llvm/llvm-project/pull/209618

From llvmlistbot at llvm.org  Tue Jul 14 19:11:05 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Tue, 14 Jul 2026 19:11:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Change inst-data of
 load/load_matrix to fit lane layout (PR #209661)
Message-ID: <llvm.org/llvm/llvm-project/pull/209661@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/209661

The consumer layout may have multiple distribution of lane data, but load_gather/load_matrix only allow one single distribution, change their inst_data according to the single distribution of lane data. 

>From 8d4af3f7516a86beb8634bff744bbc020bee84be Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Wed, 15 Jul 2026 02:02:16 +0000
Subject: [PATCH] change the inst-data layout of load to fit lane layout

---
 .../XeGPU/Transforms/XeGPULayoutImpl.cpp      | 15 ++----
 .../XeGPU/propagate-layout-inst-data.mlir     | 47 +++++++++++++++----
 2 files changed, 41 insertions(+), 21 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 8da151a333fb3..d32d53fa39613 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -1686,9 +1686,6 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   SmallVector<int64_t> consumerLaneData =
       consumerLayout.getEffectiveLaneDataAsInt();
 
-  // Pick lane_layout / lane_data: prefer consumer's, fall back to the
-  // scatter-store default (subgroupSize lanes on innermost dim, per-lane
-  // vector capped by maxChunkSize).
   SmallVector<int64_t> laneLayout;
   SmallVector<int64_t> laneData;
   assert(!consumerLaneLayout.empty() && !consumerLaneData.empty() &&
@@ -1697,16 +1694,10 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   laneData.assign(consumerLaneData.begin(), consumerLaneData.end());
 
   if (layoutKind == xegpu::LayoutKind::InstData) {
-    // Take consumer's inst_data as-is. If the consumer doesn't have one,
-    // fall back to lane_layout * lane_data per dim.
     SmallVector<int64_t> instData;
-    if (!consumerInstData.empty()) {
-      instData.assign(consumerInstData.begin(), consumerInstData.end());
-    } else {
-      instData.resize(resShape.size());
-      for (size_t i = 0; i < resShape.size(); ++i)
-        instData[i] = laneLayout[i] * laneData[i];
-    }
+    instData.resize(resShape.size());
+    for (size_t i = 0; i < resShape.size(); ++i)
+      instData[i] = laneLayout[i] * laneData[i];
     return buildInstDataLayoutWithLane(context, instData, laneLayout, laneData);
   }
   if (layoutKind == xegpu::LayoutKind::Lane)
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index c72e917b6fafa..ea2a7f395b764 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -128,37 +128,65 @@ func.func @store_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
 }
 }
 
+// TODO: colaesce_chunksize tests should be adjusted when we enable colasce optimization
 // -----
+gpu.module @test {
+// CHECK-LABEL: func.func @load_matrix_with_coalesce_chunksize(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<2048x512xf32>, %[[ARG1:[0-9a-zA-Z]+]]: memref<2048x512xf32>)
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<128xf32, 3>
+// CHECK: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+// CHECK: %[[LOAD:.*]] = xegpu.load_matrix %[[MDESC]][%[[C0]]] <{layout = #xegpu.layout<inst_data = [1], lane_layout = [1], lane_data = [1]>}>: !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<16x16xf32>
+// CHECK: %[[TRANS:.*]] = vector.transpose %[[BCAST]], [1, 0] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32> to vector<16x16xf32>
+// CHECK: %[[EXP:.*]] = math.exp %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[DIV:.*]] = arith.divf %[[EXP]], %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[TDESC:.*]] = xegpu.create_nd_tdesc %[[ARG1]] : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+// CHECK: xegpu.store_nd %[[DIV]], %[[TDESC]][%[[C0]], %[[C0]]] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+func.func @load_matrix_with_coalesce_chunksize(%arg0: memref<2048x512xf32>, %arg1: memref<2048x512xf32>) {
+  %c0 = arith.constant 0 : index
+  %alloca = memref.alloca() : memref<128xf32, 3>
+  %6 = xegpu.create_mem_desc %alloca : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+  %13 = xegpu.load_matrix %6[%c0] : !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+  %14 = vector.broadcast %13 : vector<16xf32> to vector<16x16xf32>
+  %15 = vector.transpose %14, [1, 0] : vector<16x16xf32> to vector<16x16xf32>
+  %17 = math.exp %15 : vector<16x16xf32>
+  %18 = arith.divf %17, %15 : vector<16x16xf32>
+  %22 = xegpu.create_nd_tdesc %arg1 : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  xegpu.store_nd %18, %22[%c0, %c0]  : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  return
+}
+}
 
+// -----
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<512xf32>) {
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<true> : vector<16x32xi1>
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<12> : vector<16x32xindex>
-// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<true> : vector<16x32xi1>
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<12> : vector<16x32xindex>
+// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
 func.func @scatter_ops_coalesce_chunksize(%src: memref<512xf32>) {
   %1 = arith.constant dense<1>: vector<16x32xi1>
   %offset = arith.constant dense<12> : vector<16x32xindex>
   %3 = xegpu.load %src[%offset], %1
       : memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [16, 16]>}>
+  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [1, 32]>}>
       : vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
   return
 }
 }
 
 // -----
-
 gpu.module @test {
 // CHECK-LABEL: func.func @load_gather_with_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x16xf16>, %[[ARG1:[0-9a-zA-Z]+]]: memref<256xf16>, %[[ARG2:[0-9a-zA-Z]+]]: memref<8x16xf32>) {
-// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}
+// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}
 // CHECK-SAME:  dense<0> : vector<16x16xindex>
-// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
-// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
+// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
+// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
 func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: memref<256xf16>, %arg2: memref<8x16xf32>) {
   %c0 = arith.constant 0 : index
   %0 = xegpu.create_nd_tdesc %arg0 : memref<8x16xf16> -> !xegpu.tensor_desc<8x16xf16>
@@ -176,6 +204,7 @@ func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: m
 }
 
 // -----
+
 gpu.module @test {
 // CHECK-LABEL: func.func @vector_shape_cast_expand_non_unit_dims(
 // CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[STEP:.*]]], %[[CST:.*]] <{layout = #xegpu.layout<inst_data = [16], lane_layout = [16], lane_data = [1]>}> : memref<1024xf16>, vector<1024xindex>, vector<1024xi1> -> vector<1024xf16>


From llvmlistbot at llvm.org  Tue Jul 14 19:11:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 19:11:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Change inst-data of
 load/load_matrix to fit lane layout (PR #209661)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209661@github.com>
Message-ID: <6a56ec5d.ff037943.2e959a.afc2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

The consumer layout may have multiple distribution of lane data, but load_gather/load_matrix only allow one single distribution, change their inst_data according to the single distribution of lane data. 

---
Full diff: https://github.com/llvm/llvm-project/pull/209661.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp (+3-12) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir (+38-9) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 8da151a333fb3..d32d53fa39613 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -1686,9 +1686,6 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   SmallVector<int64_t> consumerLaneData =
       consumerLayout.getEffectiveLaneDataAsInt();
 
-  // Pick lane_layout / lane_data: prefer consumer's, fall back to the
-  // scatter-store default (subgroupSize lanes on innermost dim, per-lane
-  // vector capped by maxChunkSize).
   SmallVector<int64_t> laneLayout;
   SmallVector<int64_t> laneData;
   assert(!consumerLaneLayout.empty() && !consumerLaneData.empty() &&
@@ -1697,16 +1694,10 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   laneData.assign(consumerLaneData.begin(), consumerLaneData.end());
 
   if (layoutKind == xegpu::LayoutKind::InstData) {
-    // Take consumer's inst_data as-is. If the consumer doesn't have one,
-    // fall back to lane_layout * lane_data per dim.
     SmallVector<int64_t> instData;
-    if (!consumerInstData.empty()) {
-      instData.assign(consumerInstData.begin(), consumerInstData.end());
-    } else {
-      instData.resize(resShape.size());
-      for (size_t i = 0; i < resShape.size(); ++i)
-        instData[i] = laneLayout[i] * laneData[i];
-    }
+    instData.resize(resShape.size());
+    for (size_t i = 0; i < resShape.size(); ++i)
+      instData[i] = laneLayout[i] * laneData[i];
     return buildInstDataLayoutWithLane(context, instData, laneLayout, laneData);
   }
   if (layoutKind == xegpu::LayoutKind::Lane)
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index c72e917b6fafa..ea2a7f395b764 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -128,37 +128,65 @@ func.func @store_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
 }
 }
 
+// TODO: colaesce_chunksize tests should be adjusted when we enable colasce optimization
 // -----
+gpu.module @test {
+// CHECK-LABEL: func.func @load_matrix_with_coalesce_chunksize(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<2048x512xf32>, %[[ARG1:[0-9a-zA-Z]+]]: memref<2048x512xf32>)
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<128xf32, 3>
+// CHECK: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+// CHECK: %[[LOAD:.*]] = xegpu.load_matrix %[[MDESC]][%[[C0]]] <{layout = #xegpu.layout<inst_data = [1], lane_layout = [1], lane_data = [1]>}>: !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<16x16xf32>
+// CHECK: %[[TRANS:.*]] = vector.transpose %[[BCAST]], [1, 0] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32> to vector<16x16xf32>
+// CHECK: %[[EXP:.*]] = math.exp %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[DIV:.*]] = arith.divf %[[EXP]], %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[TDESC:.*]] = xegpu.create_nd_tdesc %[[ARG1]] : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+// CHECK: xegpu.store_nd %[[DIV]], %[[TDESC]][%[[C0]], %[[C0]]] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+func.func @load_matrix_with_coalesce_chunksize(%arg0: memref<2048x512xf32>, %arg1: memref<2048x512xf32>) {
+  %c0 = arith.constant 0 : index
+  %alloca = memref.alloca() : memref<128xf32, 3>
+  %6 = xegpu.create_mem_desc %alloca : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+  %13 = xegpu.load_matrix %6[%c0] : !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+  %14 = vector.broadcast %13 : vector<16xf32> to vector<16x16xf32>
+  %15 = vector.transpose %14, [1, 0] : vector<16x16xf32> to vector<16x16xf32>
+  %17 = math.exp %15 : vector<16x16xf32>
+  %18 = arith.divf %17, %15 : vector<16x16xf32>
+  %22 = xegpu.create_nd_tdesc %arg1 : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  xegpu.store_nd %18, %22[%c0, %c0]  : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  return
+}
+}
 
+// -----
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<512xf32>) {
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<true> : vector<16x32xi1>
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<12> : vector<16x32xindex>
-// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<true> : vector<16x32xi1>
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<12> : vector<16x32xindex>
+// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
 func.func @scatter_ops_coalesce_chunksize(%src: memref<512xf32>) {
   %1 = arith.constant dense<1>: vector<16x32xi1>
   %offset = arith.constant dense<12> : vector<16x32xindex>
   %3 = xegpu.load %src[%offset], %1
       : memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [16, 16]>}>
+  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [1, 32]>}>
       : vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
   return
 }
 }
 
 // -----
-
 gpu.module @test {
 // CHECK-LABEL: func.func @load_gather_with_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x16xf16>, %[[ARG1:[0-9a-zA-Z]+]]: memref<256xf16>, %[[ARG2:[0-9a-zA-Z]+]]: memref<8x16xf32>) {
-// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}
+// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}
 // CHECK-SAME:  dense<0> : vector<16x16xindex>
-// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
-// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
+// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
+// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
 func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: memref<256xf16>, %arg2: memref<8x16xf32>) {
   %c0 = arith.constant 0 : index
   %0 = xegpu.create_nd_tdesc %arg0 : memref<8x16xf16> -> !xegpu.tensor_desc<8x16xf16>
@@ -176,6 +204,7 @@ func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: m
 }
 
 // -----
+
 gpu.module @test {
 // CHECK-LABEL: func.func @vector_shape_cast_expand_non_unit_dims(
 // CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[STEP:.*]]], %[[CST:.*]] <{layout = #xegpu.layout<inst_data = [16], lane_layout = [16], lane_data = [1]>}> : memref<1024xf16>, vector<1024xindex>, vector<1024xi1> -> vector<1024xf16>

``````````

</details>


https://github.com/llvm/llvm-project/pull/209661

From llvmlistbot at llvm.org  Tue Jul 14 19:11:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 19:11:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Change inst-data of
 load/load_matrix to fit lane layout (PR #209661)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209661@github.com>
Message-ID: <6a56ec5e.ff037943.2e959a.b010@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

The consumer layout may have multiple distribution of lane data, but load_gather/load_matrix only allow one single distribution, change their inst_data according to the single distribution of lane data. 

---
Full diff: https://github.com/llvm/llvm-project/pull/209661.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp (+3-12) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir (+38-9) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 8da151a333fb3..d32d53fa39613 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -1686,9 +1686,6 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   SmallVector<int64_t> consumerLaneData =
       consumerLayout.getEffectiveLaneDataAsInt();
 
-  // Pick lane_layout / lane_data: prefer consumer's, fall back to the
-  // scatter-store default (subgroupSize lanes on innermost dim, per-lane
-  // vector capped by maxChunkSize).
   SmallVector<int64_t> laneLayout;
   SmallVector<int64_t> laneData;
   assert(!consumerLaneLayout.empty() && !consumerLaneData.empty() &&
@@ -1697,16 +1694,10 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   laneData.assign(consumerLaneData.begin(), consumerLaneData.end());
 
   if (layoutKind == xegpu::LayoutKind::InstData) {
-    // Take consumer's inst_data as-is. If the consumer doesn't have one,
-    // fall back to lane_layout * lane_data per dim.
     SmallVector<int64_t> instData;
-    if (!consumerInstData.empty()) {
-      instData.assign(consumerInstData.begin(), consumerInstData.end());
-    } else {
-      instData.resize(resShape.size());
-      for (size_t i = 0; i < resShape.size(); ++i)
-        instData[i] = laneLayout[i] * laneData[i];
-    }
+    instData.resize(resShape.size());
+    for (size_t i = 0; i < resShape.size(); ++i)
+      instData[i] = laneLayout[i] * laneData[i];
     return buildInstDataLayoutWithLane(context, instData, laneLayout, laneData);
   }
   if (layoutKind == xegpu::LayoutKind::Lane)
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index c72e917b6fafa..ea2a7f395b764 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -128,37 +128,65 @@ func.func @store_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
 }
 }
 
+// TODO: colaesce_chunksize tests should be adjusted when we enable colasce optimization
 // -----
+gpu.module @test {
+// CHECK-LABEL: func.func @load_matrix_with_coalesce_chunksize(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<2048x512xf32>, %[[ARG1:[0-9a-zA-Z]+]]: memref<2048x512xf32>)
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<128xf32, 3>
+// CHECK: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+// CHECK: %[[LOAD:.*]] = xegpu.load_matrix %[[MDESC]][%[[C0]]] <{layout = #xegpu.layout<inst_data = [1], lane_layout = [1], lane_data = [1]>}>: !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<16x16xf32>
+// CHECK: %[[TRANS:.*]] = vector.transpose %[[BCAST]], [1, 0] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32> to vector<16x16xf32>
+// CHECK: %[[EXP:.*]] = math.exp %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[DIV:.*]] = arith.divf %[[EXP]], %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[TDESC:.*]] = xegpu.create_nd_tdesc %[[ARG1]] : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+// CHECK: xegpu.store_nd %[[DIV]], %[[TDESC]][%[[C0]], %[[C0]]] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+func.func @load_matrix_with_coalesce_chunksize(%arg0: memref<2048x512xf32>, %arg1: memref<2048x512xf32>) {
+  %c0 = arith.constant 0 : index
+  %alloca = memref.alloca() : memref<128xf32, 3>
+  %6 = xegpu.create_mem_desc %alloca : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+  %13 = xegpu.load_matrix %6[%c0] : !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+  %14 = vector.broadcast %13 : vector<16xf32> to vector<16x16xf32>
+  %15 = vector.transpose %14, [1, 0] : vector<16x16xf32> to vector<16x16xf32>
+  %17 = math.exp %15 : vector<16x16xf32>
+  %18 = arith.divf %17, %15 : vector<16x16xf32>
+  %22 = xegpu.create_nd_tdesc %arg1 : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  xegpu.store_nd %18, %22[%c0, %c0]  : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  return
+}
+}
 
+// -----
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<512xf32>) {
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<true> : vector<16x32xi1>
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<12> : vector<16x32xindex>
-// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<true> : vector<16x32xi1>
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<12> : vector<16x32xindex>
+// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
 func.func @scatter_ops_coalesce_chunksize(%src: memref<512xf32>) {
   %1 = arith.constant dense<1>: vector<16x32xi1>
   %offset = arith.constant dense<12> : vector<16x32xindex>
   %3 = xegpu.load %src[%offset], %1
       : memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [16, 16]>}>
+  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [1, 32]>}>
       : vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
   return
 }
 }
 
 // -----
-
 gpu.module @test {
 // CHECK-LABEL: func.func @load_gather_with_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x16xf16>, %[[ARG1:[0-9a-zA-Z]+]]: memref<256xf16>, %[[ARG2:[0-9a-zA-Z]+]]: memref<8x16xf32>) {
-// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}
+// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}
 // CHECK-SAME:  dense<0> : vector<16x16xindex>
-// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
-// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
+// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
+// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
 func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: memref<256xf16>, %arg2: memref<8x16xf32>) {
   %c0 = arith.constant 0 : index
   %0 = xegpu.create_nd_tdesc %arg0 : memref<8x16xf16> -> !xegpu.tensor_desc<8x16xf16>
@@ -176,6 +204,7 @@ func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: m
 }
 
 // -----
+
 gpu.module @test {
 // CHECK-LABEL: func.func @vector_shape_cast_expand_non_unit_dims(
 // CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[STEP:.*]]], %[[CST:.*]] <{layout = #xegpu.layout<inst_data = [16], lane_layout = [16], lane_data = [1]>}> : memref<1024xf16>, vector<1024xindex>, vector<1024xi1> -> vector<1024xf16>

``````````

</details>


https://github.com/llvm/llvm-project/pull/209661

From llvmlistbot at llvm.org  Tue Jul 14 19:21:36 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Tue, 14 Jul 2026 19:21:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Add local forward layout
	propagation (PR #208932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208932@github.com>
Message-ID: <6a56eeb0.b1416da1.78d1e.65da@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/208932

>From d83861877d11a1f6e2d57b933be7cbc1a769269b Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 11 Jul 2026 18:16:28 +0000
Subject: [PATCH] [MLIR][XeGPU] Add local forward layout propagation

Backward layout propagation only assigns layouts to values that are
(transitively) consumed by an anchor op. A value whose only consumer is,
e.g., the next iteration of a loop is left without a layout.

Add a local forward-fill step, run after the backward materialization walk
in propagateLayouts(): it visits ops in producer-first order and, for any
un-laid-out vector result, infers the layout from the op's already-known
operand layouts via a new inferResultLayoutFromSourceForNonAnchorOp
dispatcher (covering elementwise, transpose, and shape_cast; other ops are
left as TODO), then stamps it with setDistributeLayoutAttr.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPULayoutImpl.h        |  43 ++++++
 .../XeGPU/Transforms/XeGPULayoutImpl.cpp      | 123 ++++++++++++++++++
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  56 ++++++++
 .../XeGPU/propagate-layout-subgroup.mlir      |  35 +++++
 4 files changed, 257 insertions(+)

diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
index 75fcb156a9d0d..040227996f0ae 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/XeGPULayoutImpl.h
@@ -83,6 +83,15 @@ dropSgLayoutAndDataOnAttrs(ArrayRef<NamedAttribute> attrs);
 /// any DistributeLayoutAttr found.
 SmallVector<NamedAttribute> dropInstDataOnAttrs(ArrayRef<NamedAttribute> attrs);
 
+//===----------------------------------------------------------------------===//
+// Backward layout inference (result layout -> source layout)
+//===----------------------------------------------------------------------===//
+//
+// The infer*SourceLayout helpers below derive the layout of an operation's
+// source operand from the layout of its result. They implement the per-op
+// transfer functions used by the backward layout propagation analysis, which
+// flows layouts from anchor ops (dpas, store_nd, ...) back to their producers.
+
 /// Infers the source layout attribute for a broadcast operation given the
 /// result layout attribute, result shape, and source shape.
 DistributeLayoutAttr inferBroadcastSourceLayout(DistributeLayoutAttr resLayout,
@@ -159,6 +168,40 @@ DistributeLayoutAttr
 inferSourceLayoutFromResultForNonAnchorOp(OpOperand &operand,
                                           DistributeLayoutAttr resLayout);
 
+//===----------------------------------------------------------------------===//
+// Forward layout inference (source layout -> result layout)
+//===----------------------------------------------------------------------===//
+//
+// The infer*ResultLayout helpers below are the forward counterparts of the
+// infer*SourceLayout helpers above: given the layout of an operation's source
+// operand they derive the layout of its result. They are used by the local
+// forward-fill step in XeGPUPropagateLayout that assigns layouts to values not
+// reached by the backward propagation analysis (e.g. loop-carried values whose
+// only consumer is the next iteration).
+
+/// Infers the result layout attribute for a transpose operation given the
+/// source layout attribute and permutation. Inverse of
+/// inferTransposeSourceLayout.
+DistributeLayoutAttr inferTransposeResultLayout(DistributeLayoutAttr srcLayout,
+                                                ArrayRef<int64_t> permutation);
+
+/// Infers the result layout attribute for a shape cast operation given the
+/// source layout attribute, source shape, and result shape. Inverse of
+/// inferShapeCastSourceLayout. Returns nullptr for shape-cast patterns whose
+/// forward direction is ambiguous (e.g. unit-dim expansion).
+DistributeLayoutAttr inferShapeCastResultLayout(DistributeLayoutAttr srcLayout,
+                                                ArrayRef<int64_t> srcShape,
+                                                ArrayRef<int64_t> resShape);
+
+/// Infers the result layout attribute for a non-anchor operation from the
+/// layouts of its source operands (the forward counterpart of
+/// inferSourceLayoutFromResultForNonAnchorOp). `operandLayouts` is indexed by
+/// operand number; entries may be null for operands without a known layout.
+/// Returns nullptr when no forward rule applies (the result is then left
+/// un-laid-out).
+DistributeLayoutAttr inferResultLayoutFromSourceForNonAnchorOp(
+    Operation *op, ArrayRef<DistributeLayoutAttr> operandLayouts);
+
 /// Note on the `consumerLayout` argument used by the consumer-driven setup* /
 /// complete* helpers below:
 ///
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 8da151a333fb3..d68f81c7fd3cd 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -835,6 +835,129 @@ xegpu::inferShapeCastSourceLayout(xegpu::DistributeLayoutAttr resLayout,
   return nullptr;
 }
 
+//===----------------------------------------------------------------------===//
+// Forward layout inference (source layout -> result layout)
+//===----------------------------------------------------------------------===//
+
+/// Infers the result layout attribute for a transpose operation given the
+/// source layout attribute and permutation.
+///
+/// vector.transpose semantics is `result[i] = source[permutation[i]]`, so
+/// `result_layout[i] = source_layout[permutation[i]]`, which is exactly
+/// `srcLayout.transposeDims(permutation)`. This is the inverse of
+/// inferTransposeSourceLayout (which applies the inverse permutation).
+xegpu::DistributeLayoutAttr
+xegpu::inferTransposeResultLayout(xegpu::DistributeLayoutAttr srcLayout,
+                                  ArrayRef<int64_t> permutation) {
+  return srcLayout.transposeDims(permutation);
+}
+
+/// Infers the result layout attribute for a shape cast operation given the
+/// source layout attribute, source shape, and result shape. This is the
+/// inverse of inferShapeCastSourceLayout: a dim-split (src -> res) is undone by
+/// collapsing the split groups, and a dim-collapse (src -> res) is undone by
+/// expanding the collapsed groups. The unit-dim-expansion case is not inverted
+/// here because recovering which result dims are the expanded unit dims would
+/// require the SliceAttr the backward direction produces; such patterns return
+/// nullptr (leaving the result un-laid-out).
+xegpu::DistributeLayoutAttr
+xegpu::inferShapeCastResultLayout(xegpu::DistributeLayoutAttr srcLayout,
+                                  ArrayRef<int64_t> srcShape,
+                                  ArrayRef<int64_t> resShape) {
+  // Case: source dims were split into result dims (forward of use case 2 in
+  // inferShapeCastSourceLayout). Undo by expanding each source dim into its
+  // group of result dims.
+  SmallVector<SmallVector<int64_t>> splitDimGroups;
+  if (xegpu::matchSplitDimExpansion(srcShape, resShape, splitDimGroups)) {
+    auto resLayout = srcLayout;
+    // Process source dims from innermost to outermost so that expanding a dim
+    // does not shift the indices of dims not yet processed.
+    for (int64_t srcIdx = static_cast<int64_t>(splitDimGroups.size()) - 1;
+         srcIdx >= 0; --srcIdx) {
+      ArrayRef<int64_t> resDims = splitDimGroups[srcIdx];
+      if (resDims.size() <= 1)
+        continue;
+      SmallVector<int64_t> targetShape;
+      targetShape.reserve(resDims.size());
+      for (int64_t d : resDims)
+        targetShape.push_back(resShape[d]);
+      resLayout = resLayout.expandDim(srcIdx, targetShape);
+    }
+    return resLayout;
+  }
+
+  // Case: source dims were collapsed into result dims (forward of use case 3).
+  // Undo by collapsing each group of source dims into its single result dim.
+  SmallVector<SmallVector<int64_t>> collapseDims;
+  if (xegpu::matchDimCollapse(srcShape, resShape, collapseDims)) {
+    auto resLayout = srcLayout;
+    // Process result dims from innermost to outermost so that collapsing a
+    // group does not shift the indices of groups not yet processed.
+    for (int64_t dstIdx = static_cast<int64_t>(collapseDims.size()) - 1;
+         dstIdx >= 0; --dstIdx) {
+      ArrayRef<int64_t> srcDims = collapseDims[dstIdx];
+      // A result dim with no backing source dims is a trailing/leading unit
+      // dim; its forward inference is ambiguous, so bail out.
+      if (srcDims.empty())
+        return nullptr;
+      if (srcDims.size() == 1)
+        continue;
+      resLayout = resLayout.collapseDims(llvm::to_vector(srcDims));
+    }
+    return resLayout;
+  }
+
+  return nullptr;
+}
+
+/// Infers the result layout attribute for a non-anchor operation from the
+/// layouts of its source operands. Forward counterpart of
+/// inferSourceLayoutFromResultForNonAnchorOp.
+xegpu::DistributeLayoutAttr xegpu::inferResultLayoutFromSourceForNonAnchorOp(
+    Operation *op, ArrayRef<xegpu::DistributeLayoutAttr> operandLayouts) {
+  if (op->getNumResults() != 1)
+    return nullptr;
+
+  // For vector::TransposeOp, infer the result layout from the source layout.
+  if (auto transpose = dyn_cast<vector::TransposeOp>(op)) {
+    if (!operandLayouts[0])
+      return nullptr;
+    return xegpu::inferTransposeResultLayout(operandLayouts[0],
+                                             transpose.getPermutation());
+  }
+
+  // For vector::ShapeCastOp, infer the result layout from the source layout.
+  if (auto shapeCast = dyn_cast<vector::ShapeCastOp>(op)) {
+    if (!operandLayouts[0])
+      return nullptr;
+    return xegpu::inferShapeCastResultLayout(
+        operandLayouts[0], shapeCast.getSourceVectorType().getShape(),
+        shapeCast.getResultVectorType().getShape());
+  }
+
+  // For elementwise operations, all operands and the result share the same
+  // layout. Use the first operand that carries a layout.
+  if (OpTrait::hasElementwiseMappableTraits(op)) {
+    for (xegpu::DistributeLayoutAttr layout : operandLayouts)
+      if (layout)
+        return layout;
+    return nullptr;
+  }
+
+  // TODO: add forward inference rules for the remaining ops; their result is
+  // left un-laid-out until then.
+  //  - vector::BroadcastOp: the forward direction is under-determined. The
+  //    backward rule (inferBroadcastSourceLayout) either sets broadcast dims to
+  //    unit data (losing the original data on those dims) or wraps the result
+  //    in a SliceAttr; neither is generally invertible from the source layout
+  //    alone, so a forward rule must decide how to distribute the new/stretched
+  //    dims.
+  //  - vector::BitCastOp, vector::MultiDimReductionOp / vector::ReductionOp,
+  //    vector::InterleaveOp / vector::DeinterleaveOp, and the insert / extract
+  //    / strided-slice family.
+  return nullptr;
+}
+
 /// Infers the layout attribute for mask and offset operand for Chunked load
 /// and store, given the anchor layout attribute for the value being load/store.
 xegpu::DistributeLayoutAttr xegpu::inferMaskOffsetLayoutForScatterIO(
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1b1abcc4b0e4f..68f8abd1ed92d 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1695,6 +1695,57 @@ ResolveLayoutConflicts::resolveTensorDescConsumer(OpOperand &operand) {
 }
 
 using GetLayoutFnTy = function_ref<xegpu::DistributeLayoutAttr(Value)>;
+
+/// Local forward layout propagation, run after the backward propagation
+/// analysis has materialized its layouts. Backward propagation only assigns
+/// layouts to values that are (transitively) consumed by an anchor op; a value
+/// whose only consumer is, e.g., the next iteration of a loop (its producing op
+/// is not on any anchor's backward slice) is left without a layout. This walk
+/// visits ops in producer-first (forward) order and, for any vector result that
+/// still lacks a layout, infers it from the already-known layouts of the op's
+/// operands via inferResultLayoutFromSourceForNonAnchorOp, then stamps it with
+/// setDistributeLayoutAttr.
+///
+/// A single forward pass suffices: forward-only values form producer ->
+/// consumer chains that the pre-order walk visits in dependency order, so an
+/// operand's layout is already assigned by the time its consumer is visited.
+static void forwardFillLayouts(Operation *root) {
+  root->walk([&](Operation *op) {
+    // Anchor ops carry authoritative layouts; region ops and their terminators
+    // are handled by the control-flow propagation; function ops carry no result
+    // layouts here.
+    if (isa<xegpu::AnchorLayoutInterface, RegionBranchOpInterface,
+            RegionBranchTerminatorOpInterface, FunctionOpInterface>(op))
+      return;
+    if (op->getNumResults() != 1)
+      return;
+    OpResult result = op->getResult(0);
+    if (!isa<VectorType>(result.getType()))
+      return;
+    // Skip results that already have a layout (from backward propagation or an
+    // earlier forward-fill step).
+    if (xegpu::getDistributeLayoutAttr(result))
+      return;
+
+    // Gather operand layouts, indexed by operand number.
+    SmallVector<xegpu::DistributeLayoutAttr> operandLayouts;
+    operandLayouts.reserve(op->getNumOperands());
+    bool anyAssigned = false;
+    for (Value operand : op->getOperands()) {
+      auto layout = xegpu::getDistributeLayoutAttr(operand);
+      operandLayouts.push_back(layout);
+      anyAssigned |= (layout != nullptr);
+    }
+    if (!anyAssigned)
+      return;
+
+    xegpu::DistributeLayoutAttr layout =
+        xegpu::inferResultLayoutFromSourceForNonAnchorOp(op, operandLayouts);
+    if (layout)
+      xegpu::setDistributeLayoutAttr(result, layout);
+  });
+}
+
 /// Update an operation with the layout of its results. If the result type is
 /// a vector type, a temporary layout attribute is added to the operation. If
 /// the result type is a tensor descriptor type, the type is updated with the
@@ -1918,6 +1969,11 @@ LogicalResult xegpu::propagateLayouts(OpBuilder &builder, Operation *target,
   if (walkResult.wasInterrupted())
     return failure();
 
+  // Backward propagation only reaches values consumed by anchor ops. Run a
+  // local forward pass to fill in layouts for values it could not reach (e.g.
+  // loop-carried values whose only consumer is the next iteration).
+  forwardFillLayouts(op);
+
   return success();
 }
 
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..5c29695bafa31 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -672,3 +672,38 @@ gpu.module @test {
     gpu.return
   }
 }
+
+// -----
+// Forward layout fill: %sum is a loop-carried value whose only consumer is the
+// next iteration (via iter_arg %acc), so backward propagation from anchor ops
+// never reaches it. The local forward pass derives its layout from the operand
+// %acc (elementwise rule), and the scf.for's second result / yield inherit it.
+gpu.module @test {
+// CHECK-LABEL: gpu.func @forward_fill_loop_carried(
+gpu.func @forward_fill_loop_carried(%arg0: memref<128x64xf16>, %arg1: memref<64x64xf16>, %arg2: memref<128x64xf32>) kernel attributes {known_block_size = array<i32: 128, 1, 1>} {
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c128 = arith.constant 128 : index
+  %cst = arith.constant dense<0.000000e+00> : vector<128x64xf32>
+  %cst_0 = arith.constant dense<0.000000e+00> : vector<128xf32>
+  %0 = xegpu.create_nd_tdesc %arg0 : memref<128x64xf16> -> !xegpu.tensor_desc<128x64xf16>
+  %1 = xegpu.load_nd %0[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>}> : !xegpu.tensor_desc<128x64xf16> -> vector<128x64xf16>
+  %2 = xegpu.create_nd_tdesc %arg1 : memref<64x64xf16> -> !xegpu.tensor_desc<64x64xf16>
+  %3 = xegpu.load_nd %2[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [1, 1], sg_data = [64, 64]>}> : !xegpu.tensor_desc<64x64xf16> -> vector<64x64xf16>
+  // CHECK: scf.for
+  %4:2 = scf.for %arg4 = %c0 to %c128 step %c64 iter_args(%acc0 = %cst, %acc = %cst_0) -> (vector<128x64xf32>, vector<128xf32>) {
+    %8 = xegpu.dpas %1, %3, %acc0 {layout_a = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, layout_b = #xegpu.layout<sg_layout = [1, 1], sg_data = [64, 64]>, layout_cd = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>} : vector<128x64xf16>, vector<64x64xf16>, vector<128x64xf32> -> vector<128x64xf32>
+    %9 = vector.broadcast %acc : vector<128xf32> to vector<64x128xf32>
+    %10 = vector.transpose %9, [1, 0] : vector<64x128xf32> to vector<128x64xf32>
+    %11 = arith.mulf %8, %10 : vector<128x64xf32>
+    // The forward pass assigns %sum a layout inherited from %acc.
+    // CHECK: arith.addf
+    // CHECK-SAME: layout_result_0 = #xegpu.slice<#xegpu.layout<sg_layout = [1, 8], sg_data = [64, 16], order = [0, 1]>, dims = [0]>
+    %sum = arith.addf %acc, %acc : vector<128xf32>
+    scf.yield %11, %sum : vector<128x64xf32>, vector<128xf32>
+  }
+  %5 = xegpu.create_nd_tdesc %arg2 : memref<128x64xf32> -> !xegpu.tensor_desc<128x64xf32>
+  xegpu.store_nd %4#0, %5[%c0, %c0] <{layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>}> : vector<128x64xf32>, !xegpu.tensor_desc<128x64xf32>
+  gpu.return
+}
+}


From llvmlistbot at llvm.org  Tue Jul 14 19:59:47 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Tue, 14 Jul 2026 19:59:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][func] Fix a crash in
 DuplicateFunctionEliminationPass (PR #209667)
Message-ID: <llvm.org/llvm/llvm-project/pull/209667@github.com>


https://github.com/CoTinker created https://github.com/llvm/llvm-project/pull/209667

Previously, we used `SymbolUserMap::replaceAllUsesWith` to replace symbols, but this could not update the symbol table cached by `SymbolUserMap` during traversal, leading to a crash. This PR switches to `SymbolTable::replaceAllSymbolUses`, which always operates on the latest symbol table and avoids the crash. Fixes #209648.

>From 9a90651c9299881f1d44a1eb5f0d1ec0c626c9d6 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Wed, 15 Jul 2026 10:55:53 +0800
Subject: [PATCH 1/2] [mlir][func] Fix a crash in
 DuplicateFunctionEliminationPass

Previously, we used `SymbolUserMap::replaceAllUsesWith` to replace symbols, but this could not update the symbol table cached by `SymbolUserMap` during traversal, leading to a crash. This PR switches to `SymbolTable::replaceAllSymbolUses`, which always operates on the latest symbol table and avoids the crash.
---
 .../Func/Transforms/DuplicateFunctionElimination.cpp        | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp b/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
index a2186e433f544..c480e469fdcc8 100644
--- a/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
+++ b/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
@@ -101,12 +101,12 @@ struct DuplicateFunctionEliminationPass
 
     // Update all symbol uses to reference unique func op
     // representants and erase redundant func ops.
-    SymbolTableCollection symbolTable;
-    SymbolUserMap userMap(symbolTable, module);
     for (auto it : toBeErased) {
       StringAttr oldSymbol = it.getSymNameAttr();
       StringAttr newSymbol = getRepresentant[oldSymbol].getSymNameAttr();
-      userMap.replaceAllUsesWith(it, newSymbol);
+      if (failed(
+              SymbolTable::replaceAllSymbolUses(oldSymbol, newSymbol, module)))
+        return signalPassFailure();
       it.erase();
     }
   }

>From ba869255ca1fb1df2ce972a580326558c6f07ec7 Mon Sep 17 00:00:00 2001
From: Longsheng Mou <longshengmou at gmail.com>
Date: Wed, 15 Jul 2026 10:58:18 +0800
Subject: [PATCH 2/2] add test

---
 .../Func/duplicate-function-elimination.mlir  | 26 +++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/mlir/test/Dialect/Func/duplicate-function-elimination.mlir b/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
index 4d00d8a954d17..b6588b39c5a87 100644
--- a/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
+++ b/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
@@ -412,3 +412,29 @@ func.func @user(%arg0: tensor<f32>) -> tensor<f32> {
 // CHECK:     @user
 // CHECK-2:     constant @identity
 // CHECK:       call @identity
+
+// -----
+
+func.func @callee0() {
+  return
+}
+
+func.func @caller0() {
+  call @callee1() : () -> ()
+  return
+}
+
+func.func @caller1() {
+  call @callee1() : () -> ()
+  return
+}
+
+func.func @callee1() {
+  return
+}
+
+// CHECK:     @callee0
+// CHECK:     @caller0
+// CHECK:       call @callee0()
+// CHECK-NOT: @caller1
+// CHECK-NOT: @callee1


From llvmlistbot at llvm.org  Tue Jul 14 20:00:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 20:00:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][func] Fix a crash in
 DuplicateFunctionEliminationPass (PR #209667)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209667@github.com>
Message-ID: <6a56f7c5.492e23e4.14eda0.d09f@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Longsheng Mou (CoTinker)

<details>
<summary>Changes</summary>

Previously, we used `SymbolUserMap::replaceAllUsesWith` to replace symbols, but this could not update the symbol table cached by `SymbolUserMap` during traversal, leading to a crash. This PR switches to `SymbolTable::replaceAllSymbolUses`, which always operates on the latest symbol table and avoids the crash. Fixes #<!-- -->209648.

---
Full diff: https://github.com/llvm/llvm-project/pull/209667.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp (+3-3) 
- (modified) mlir/test/Dialect/Func/duplicate-function-elimination.mlir (+26) 


``````````diff
diff --git a/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp b/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
index a2186e433f544..c480e469fdcc8 100644
--- a/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
+++ b/mlir/lib/Dialect/Func/Transforms/DuplicateFunctionElimination.cpp
@@ -101,12 +101,12 @@ struct DuplicateFunctionEliminationPass
 
     // Update all symbol uses to reference unique func op
     // representants and erase redundant func ops.
-    SymbolTableCollection symbolTable;
-    SymbolUserMap userMap(symbolTable, module);
     for (auto it : toBeErased) {
       StringAttr oldSymbol = it.getSymNameAttr();
       StringAttr newSymbol = getRepresentant[oldSymbol].getSymNameAttr();
-      userMap.replaceAllUsesWith(it, newSymbol);
+      if (failed(
+              SymbolTable::replaceAllSymbolUses(oldSymbol, newSymbol, module)))
+        return signalPassFailure();
       it.erase();
     }
   }
diff --git a/mlir/test/Dialect/Func/duplicate-function-elimination.mlir b/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
index 4d00d8a954d17..b6588b39c5a87 100644
--- a/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
+++ b/mlir/test/Dialect/Func/duplicate-function-elimination.mlir
@@ -412,3 +412,29 @@ func.func @user(%arg0: tensor<f32>) -> tensor<f32> {
 // CHECK:     @user
 // CHECK-2:     constant @identity
 // CHECK:       call @identity
+
+// -----
+
+func.func @callee0() {
+  return
+}
+
+func.func @caller0() {
+  call @callee1() : () -> ()
+  return
+}
+
+func.func @caller1() {
+  call @callee1() : () -> ()
+  return
+}
+
+func.func @callee1() {
+  return
+}
+
+// CHECK:     @callee0
+// CHECK:     @caller0
+// CHECK:       call @callee0()
+// CHECK-NOT: @caller1
+// CHECK-NOT: @callee1

``````````

</details>


https://github.com/llvm/llvm-project/pull/209667

From llvmlistbot at llvm.org  Tue Jul 14 21:07:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 21:07:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Avoid crash on non-affine
	memory effects (PR #205865)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205865@github.com>
Message-ID: <6a57078d.a1accbb4.244acd.ea4c@mx.google.com>


mygitljf wrote:

Hi @FedericoBruzzone , just checking in — is there anything else I should change here? If everything looks good, I’d appreciate a review and help getting this merged. Thanks!

https://github.com/llvm/llvm-project/pull/205865

From llvmlistbot at llvm.org  Tue Jul 14 21:13:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 21:13:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Validate function insertions (PR
	#206009)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206009@github.com>
Message-ID: <6a570905.a1accbb4.244acd.53c1@mx.google.com>


mygitljf wrote:

Hi @joker-eph
Could take a moment to look at this? Thanks.

https://github.com/llvm/llvm-project/pull/206009

From llvmlistbot at llvm.org  Tue Jul 14 21:15:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 21:15:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Handle repeated EmitC operands (PR
	#206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a57096f.af6ba3ed.1dc10f.22a1@mx.google.com>


mygitljf wrote:

Hi @aniragil and @simon-camp
Could take a moment to look at this? Thanks.

https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Tue Jul 14 23:56:03 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 14 Jul 2026 23:56:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Avoid crash on non-affine
	memory effects (PR #205865)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205865@github.com>
Message-ID: <6a572f03.824e11a6.f5f27.7e74@mx.google.com>


FedericoBruzzone wrote:

> Hi @FedericoBruzzone , just checking in — is there anything else I should change here? If everything looks good, I’d appreciate it if you could merge this pr. Thanks!

Hi @mygitljf!

Unfortunately, I don't have the permissions to merge the PRs. I suggest you check which maintainers have recently worked on the affine dialect and then ping them :D

https://github.com/llvm/llvm-project/pull/205865

From llvmlistbot at llvm.org  Tue Jul 14 23:57:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 14 Jul 2026 23:57:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 7881a37 - [mlir][linalg] Refine pack/unpack
 simplification checks (NFC) (#209522)
Message-ID: <6a572f77.bd93e731.358318.8503@mx.google.com>


Author: Andrzej Warzyński
Date: 2026-07-15T07:57:55+01:00
New Revision: 7881a375884c3793d2fa7c31ab97781a43547f31

URL: https://github.com/llvm/llvm-project/commit/7881a375884c3793d2fa7c31ab97781a43547f31
DIFF: https://github.com/llvm/llvm-project/commit/7881a375884c3793d2fa7c31ab97781a43547f31.diff

LOG: [mlir][linalg] Refine pack/unpack simplification checks (NFC) (#209522)

Update `isPackOn1D`, which is used by both
`SimplifyPackToExpandShape` and `SimplifyUnPackToCollapseShape`:

* Rename it to `isPackOnEffectively1D` to better reflect its
   functionality: the underlying pack can be multi-dimensional.
* Add checks to ensure that the unique non-unit inner tile is used to
   tile the unique non-unit unpacked dimension (that was previously
   left as an unchecked assumption).
* Add comments to the test file for these patterns, grouping the tests
   according to the functionality/cases being tested.

Added: 
    

Modified: 
    mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
    mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp b/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
index 993eae62535c3..c8e9f021938e5 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/PackAndUnpackPatterns.cpp
@@ -24,24 +24,67 @@ static int64_t getNumGtOneDims(ArrayRef<int64_t> shape) {
       shape, [](int64_t v) { return ShapedType::isDynamic(v) || v > 1; });
 }
 
-/// Returns success() if there is only 1 dimension size in non-packed domain
-/// being greater than 1 and packing only happens on the dimension.
-/// Note: this method should only be used by pack/unpack to reshape conversion.
-/// It assumes that non-unit inner tile size must be used by the non-unit
-/// dimension.
-static LogicalResult isPackOn1D(RewriterBase &rewriter, Operation *op,
-                                ArrayRef<int64_t> srcShape,
-                                ArrayRef<int64_t> innerPackTileSize) {
-  if (getNumGtOneDims(srcShape) > 1) {
+/// Returns the index of the first non-unit size in `sizes`. Returns -1 if
+/// there are no non-unit sizes.
+static int64_t getFirstNonUnitSizeIdx(ArrayRef<int64_t> sizes) {
+  const auto *it = llvm::find_if(sizes, [](int64_t dim) { return dim != 1; });
+  return (it != sizes.end()) ? std::distance(sizes.begin(), it) : -1;
+}
+
+/// Check whether `op` is effectively a 1D pack/unpack. Example:
+///
+///  %pack = linalg.pack %src
+///     inner_dims_pos = [0, 1]
+///     inner_tiles = [1, 2] into %dest
+///   : tensor<1x32xf32> -> tensor<1x16x1x2xf32>
+///
+/// Returns success() if there is:
+///   * only 1 non-unit dim in the un-packed domain,
+///   * only 1 non-unit inner tile size, and
+///   * the unique non-unit tile size is applied to the unique non-unit
+///     un-packed dim.
+template <typename PackOrUnpackOp>
+static LogicalResult isPackOnEffectively1D(RewriterBase &rewriter,
+                                           PackOrUnpackOp *op) {
+  // Obtain the unpacked shape.
+  auto pack = dyn_cast<linalg::PackOp>(op);
+  auto unpack = dyn_cast<linalg::UnPackOp>(op);
+
+  ArrayRef<int64_t> unpackedShape = pack ? pack->getSourceType().getShape()
+                                         : unpack->getDestType().getShape();
+
+  // Obtain the inner tile sizes.
+  ArrayRef<int64_t> innerTileSizes = op->getStaticInnerTiles();
+
+  // Make sure that there is exactly single non-unit unpacked dim.
+  if (getNumGtOneDims(unpackedShape) != 1) {
     return rewriter.notifyMatchFailure(
-        op, "expects non-packed domain to have at most one non-unit dims");
+        *op, "expects non-packed domain to have at most one non-unit dims");
   }
-  // Non-unit inner tile size must be used by the non-unit dimension. If not, it
-  // will faill on getting reassociation maps.
-  if (getNumGtOneDims(innerPackTileSize) > 1) {
+
+  // Make sure that there is at most one non-unit inner tile size.
+  auto numNonUnitInnerTiles = getNumGtOneDims(innerTileSizes);
+  if (numNonUnitInnerTiles > 1) {
     return rewriter.notifyMatchFailure(
-        op, "expects at most one non-unit inner tiles");
+        *op, "expects at most one non-unit inner tiles");
   }
+
+  // If there are no non-unit tiles, there is nothing else to check.
+  if (numNonUnitInnerTiles == 0)
+    return success();
+
+  // Get the index of the unique non-unit unpacked dim.
+  int64_t nonUnitDimIdx = getFirstNonUnitSizeIdx(unpackedShape);
+
+  // Get the index of the dim that the unique non-unit tile is applied to.
+  int64_t nonUnitTileDestDimIdx = getFirstNonUnitSizeIdx(innerTileSizes);
+
+  // Make sure that the unique non-unit tile is applied to the unique unit dim.
+  if (nonUnitTileDestDimIdx != nonUnitDimIdx) {
+    return rewriter.notifyMatchFailure(
+        *op, "expects at most one non-unit inner tiles");
+  }
+
   return success();
 }
 
@@ -117,8 +160,7 @@ struct SimplifyPackToExpandShape : public OpRewritePattern<PackOp> {
 
     ShapedType sourceType = packOp.getSourceType();
     if (failed(isPackOnInnerMostDim(rewriter, packOp)) &&
-        failed(isPackOn1D(rewriter, packOp, sourceType.getShape(),
-                          packOp.getStaticTiles())) &&
+        failed(isPackOnEffectively1D(rewriter, &packOp)) &&
         !packOp.isLikePad()) {
       return failure();
     }
@@ -183,8 +225,7 @@ struct SimplifyUnPackToCollapseShape : public OpRewritePattern<UnPackOp> {
 
     ShapedType destType = unpackOp.getDestType();
     if (failed(isUnpackOnInnerMostDim(rewriter, unpackOp)) &&
-        failed(isPackOn1D(rewriter, unpackOp, destType.getShape(),
-                          unpackOp.getStaticTiles())) &&
+        failed(isPackOnEffectively1D(rewriter, &unpackOp)) &&
         !unpackOp.isLikeUnPad()) {
       return failure();
     }

diff  --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6979770154bab..f4f3cb233d45e 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -1,5 +1,9 @@
 // RUN: mlir-opt -split-input-file -test-linalg-transform-patterns="test-simplify-pack-unpack-patterns" %s | FileCheck %s
 
+//===========================================================================//
+// Packing: 1D unpacked source
+//===========================================================================//
+
 // CHECK-LABEL: func.func @single_dim_packing(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<256xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]] output_shape [8, 32] : tensor<256xf32> into tensor<8x32xf32>
@@ -25,18 +29,6 @@ func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x3
 
 // -----
 
-// CHECK-LABEL: func.func @single_last_inner_dim_packing(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
-// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
-// CHECK:         return %[[EXPANDED]] : tensor<5x8x32xf32>
-func.func @single_last_inner_dim_packing(%arg0: tensor<5x256xf32>) -> tensor<5x8x32xf32> {
-  %empty = tensor.empty() : tensor<5x8x32xf32>
-  %0 = linalg.pack %arg0 inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<5x8x32xf32>
-  return %0 : tensor<5x8x32xf32>
-}
-
-// -----
-
 // CHECK-LABEL: func.func @pack_1d_with_outer_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<64xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1]] output_shape [2, 32] : tensor<64xf32> into tensor<2x32xf32>
@@ -49,6 +41,22 @@ func.func @pack_1d_with_outer_dims_perm(%arg0: tensor<64xf32>) -> tensor<2x32xf3
 
 // -----
 
+//===========================================================================//
+// Packing: 2D unpacked source
+//===========================================================================//
+
+// CHECK-LABEL: func.func @single_last_inner_dim_packing(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
+// CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
+// CHECK:         return %[[EXPANDED]] : tensor<5x8x32xf32>
+func.func @single_last_inner_dim_packing(%arg0: tensor<5x256xf32>) -> tensor<5x8x32xf32> {
+  %empty = tensor.empty() : tensor<5x8x32xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<5x8x32xf32>
+  return %0 : tensor<5x8x32xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<5x256xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2]] output_shape [5, 8, 32] : tensor<5x256xf32> into tensor<5x8x32xf32>
@@ -83,6 +91,10 @@ func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x
 
 // -----
 
+//===========================================================================//
+// Packing: Multi-dim unpacked source
+//===========================================================================//
+
 // CHECK-LABEL: func.func @pack_1x32_to_1x32x1x1
 // CHECK-SAME:    %[[ARG0:[0-9a-zA-Z]+]]
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0], [1, 2, 3]] output_shape [1, 32, 1, 1]
@@ -134,6 +146,10 @@ func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf3
 
 // -----
 
+//===========================================================================//
+// Un-Packing: 1D unpacked dest
+//===========================================================================//
+
 // CHECK-LABEL: func.func @unpack_1d_to_collapse
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<8x32xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1]] : tensor<8x32xf32> into tensor<256xf32>
@@ -172,6 +188,10 @@ func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
 
 // -----
 
+//===========================================================================//
+// Un-Packing: 2D unpacked dest
+//===========================================================================//
+
 // CHECK-LABEL: func.func @single_last_inner_dim_unpacking(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<5x8x32xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0], [1, 2]] : tensor<5x8x32xf32> into tensor<5x256xf32>
@@ -269,6 +289,13 @@ func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1x
 
 // -----
 
+//===========================================================================//
+// Packing: Pad-like pack
+//
+// FIXME: With no single element added, there is no padding here. Improve
+// naming.
+//===========================================================================//
+
 // CHECK-LABEL: func.func @pad_like_pack(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<32x64xf32>)
 // CHECK:         %[[EXPANDED:.+]] = tensor.expand_shape %[[ARG0]] {{\[}}[0, 1, 2], [3]] output_shape [1, 1, 32, 64] : tensor<32x64xf32> into tensor<1x1x32x64xf32>
@@ -333,6 +360,13 @@ func.func @pad_like_pack_with_transpose(%arg0: tensor<32x64x16xf32>) -> tensor<3
 
 // -----
 
+//===========================================================================//
+// Un-Packing: Un-Pad-like unpack
+//
+// FIXME: With no single element added, there is no padding here. Improve
+// naming.
+//===========================================================================//
+
 // CHECK-LABEL: func.func @unpad_like_unpack(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x64xf32>)
 // CHECK:         %[[COLLAPSED:.+]] = tensor.collapse_shape %[[ARG0]] {{\[}}[0, 1, 2], [3]] : tensor<1x1x32x64xf32> into tensor<32x64xf32>


        

From llvmlistbot at llvm.org  Tue Jul 14 23:58:02 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 23:58:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Refine pack/unpack
 simplification checks (NFC) (PR #209522)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209522@github.com>
Message-ID: <6a572f7a.449bfb44.172ed9.9785@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/209522

From llvmlistbot at llvm.org  Tue Jul 14 23:59:39 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 14 Jul 2026 23:59:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a572fdb.bc28b987.90005.3493@mx.google.com>


banach-space wrote:

> @banach-space I will rebase on top of #209522 later.

That's now merged, so you can rebase on top of `main`. Thank you!

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Wed Jul 15 00:12:15 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 15 Jul 2026 00:12:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5732cf.8e75e0cb.3dbce4.a5fd@mx.google.com>


================

----------------
banach-space wrote:

> How would I check for the match failure? 

You don't :) 

We know that the pass is run, that's implicit from the `RUN` line:
```mlir
// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref 
```

If the IR does not change and the test does not "crash" (it does not, otherwise `RUN` would fail), then it follows that a "match failure" took place. There is just no other option. However, it's more idiomatic to say "it bails out" rather than "match failure", because:
* The "class" Op would still be matched, but
* The pattern bails out as there are no fields.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Wed Jul 15 01:26:14 2026
From: llvmlistbot at llvm.org (Aayush Shrivastava)
Date: Wed, 15 Jul 2026 01:26:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a574426.6365b89b.8575e.2b00@mx.google.com>


https://github.com/iamaayushrivastava updated https://github.com/llvm/llvm-project/pull/209426

>From 50df936c86ffc4dc1e8ae1a254775b61a7c9c56c Mon Sep 17 00:00:00 2001
From: iamaayushrivastava <iamaayushrivastava at gmail.com>
Date: Tue, 14 Jul 2026 15:35:31 +0530
Subject: [PATCH 1/3] [mlir] Fix mem2reg crash on scalable vector store/load
 with matching type

---
 mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp | 13 ++++++++
 mlir/test/Dialect/LLVMIR/mem2reg.mlir         | 32 +++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
index 66d6a592a93df..1186e64b8934d 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
@@ -256,6 +256,12 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcType == targetType)
+    return srcValue;
+
   uint64_t srcTypeSize = dataLayout.getTypeSizeInBits(srcType);
   uint64_t targetTypeSize = dataLayout.getTypeSizeInBits(targetType);
   if (srcTypeSize == targetTypeSize)
@@ -291,6 +297,13 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
                                  srcValue.getType(),
                                  /*narrowingConversion=*/false) &&
          "expected that the compatibility was checked before");
+
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of types that may not have one, such as
+  // scalable vectors.
+  if (srcValue.getType() == reachingDef.getType())
+    return srcValue;
+
   uint64_t valueTypeSize = dataLayout.getTypeSizeInBits(srcValue.getType());
   uint64_t slotTypeSize = dataLayout.getTypeSizeInBits(reachingDef.getType());
   if (slotTypeSize == valueTypeSize)
diff --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index cabbd1c1bc013..07ad0b86043e9 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1035,6 +1035,38 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// Storing into a scalable vector slot with a type matching the slot's
+// element type must not query the type's bit size, as this is not possible
+// for scalable vectors. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store
+llvm.func @scalable_vector_matching_store(%arg: vector<[4]xi1>) {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  // CHECK-NOT: llvm.store
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  llvm.return
+}
+
+// -----
+
+// Loading back a scalable vector value that was stored with a matching type
+// must also avoid querying the type's bit size. See #209065.
+
+// CHECK-LABEL: @scalable_vector_matching_store_load
+llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  %2 = llvm.load %1 : !llvm.ptr -> vector<[4]xi1>
+  // CHECK: llvm.return %[[ARG:.*]] : vector<[4]xi1>
+  llvm.return %2 : vector<[4]xi1>
+}
+
+// -----
+
 // CHECK-LABEL: @smaller_store_forwarding
 // CHECK-SAME: %[[ARG:.+]]: i16
 llvm.func @smaller_store_forwarding(%arg : i16) {

>From 4710e803e7180e7d4320cccf0b2eefe835385a5e Mon Sep 17 00:00:00 2001
From: iamaayushrivastava <iamaayushrivastava at gmail.com>
Date: Tue, 14 Jul 2026 21:03:31 +0530
Subject: [PATCH 2/3] [mlir] Address review comments on mem2reg scalable vector
 fix

---
 mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp |  6 ++----
 mlir/test/Dialect/LLVMIR/mem2reg.mlir         | 19 -------------------
 2 files changed, 2 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
index 1186e64b8934d..2416e01a67955 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
@@ -257,8 +257,7 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
          "expected that the compatibility was checked before");
 
   // Nothing has to be done if the types are already the same. This also
-  // avoids querying the bit size of types that may not have one, such as
-  // scalable vectors.
+  // avoids querying the bit size of scalable vector types below.
   if (srcType == targetType)
     return srcValue;
 
@@ -299,8 +298,7 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
          "expected that the compatibility was checked before");
 
   // Nothing has to be done if the types are already the same. This also
-  // avoids querying the bit size of types that may not have one, such as
-  // scalable vectors.
+  // avoids querying the bit size of scalable vector types below.
   if (srcValue.getType() == reachingDef.getType())
     return srcValue;
 
diff --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index 07ad0b86043e9..bf7da45bfa97e 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1035,25 +1035,6 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
-// Storing into a scalable vector slot with a type matching the slot's
-// element type must not query the type's bit size, as this is not possible
-// for scalable vectors. See #209065.
-
-// CHECK-LABEL: @scalable_vector_matching_store
-llvm.func @scalable_vector_matching_store(%arg: vector<[4]xi1>) {
-  %0 = llvm.mlir.constant(1 : i32) : i32
-  // CHECK-NOT: llvm.alloca
-  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
-  // CHECK-NOT: llvm.store
-  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
-  llvm.return
-}
-
-// -----
-
-// Loading back a scalable vector value that was stored with a matching type
-// must also avoid querying the type's bit size. See #209065.
-
 // CHECK-LABEL: @scalable_vector_matching_store_load
 llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
   %0 = llvm.mlir.constant(1 : i32) : i32

>From 37654ffd85b1c4ad02f24336b57779e2d2590cc0 Mon Sep 17 00:00:00 2001
From: iamaayushrivastava <iamaayushrivastava at gmail.com>
Date: Wed, 15 Jul 2026 13:55:47 +0530
Subject: [PATCH 3/3] [mlir] Bind test function argument via CHECK-SAME in
 mem2reg scalable vector test

---
 mlir/test/Dialect/LLVMIR/mem2reg.mlir | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index bf7da45bfa97e..5f1029625b9b4 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1036,13 +1036,14 @@ llvm.func @scalable_llvm_vector() -> i16 {
 // -----
 
 // CHECK-LABEL: @scalable_vector_matching_store_load
+// CHECK-SAME: %[[ARG:.+]]: vector<[4]xi1>
 llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
   %0 = llvm.mlir.constant(1 : i32) : i32
   // CHECK-NOT: llvm.alloca
   %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
   llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
   %2 = llvm.load %1 : !llvm.ptr -> vector<[4]xi1>
-  // CHECK: llvm.return %[[ARG:.*]] : vector<[4]xi1>
+  // CHECK: llvm.return %[[ARG]] : vector<[4]xi1>
   llvm.return %2 : vector<[4]xi1>
 }
 


From llvmlistbot at llvm.org  Wed Jul 15 01:27:14 2026
From: llvmlistbot at llvm.org (Aayush Shrivastava)
Date: Wed, 15 Jul 2026 01:27:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a574462.1728eaf3.eafd0.464f@mx.google.com>


================
@@ -1035,6 +1035,19 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// CHECK-LABEL: @scalable_vector_matching_store_load
+llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
----------------
iamaayushrivastava wrote:

Done. Thank you for pointing that out!

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Wed Jul 15 02:00:21 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 02:00:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a574c25.492e23e4.14eda0.0ced@mx.google.com>


================
@@ -418,9 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-/// 4 is collapsed to a non-unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
-func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_more_reduced_dims(
----------------
ioghiban wrote:

Added `@negative_copy_into_strided_no_truncated_dims` test to show the rewrite being skipped if reinterpret_cast changes no dimensions.

https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 02:01:02 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 02:01:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a574c4e.a1accbb4.244acd.be2a@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 02:02:50 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 02:02:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a574cba.e5907290.281f16.766b@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/209536

>From 5905cd319ad0a104d1316e05f53d4a40eff839b3 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 14 Jul 2026 17:43:06 +0200
Subject: [PATCH 1/2] [memref] Support reducing copied dims to non-unit in
 reinterpret_cast elision

---
 .../Transforms/ElideReinterpretCast.cpp       | 62 ++++++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 86 ++++++++++++++++---
 2 files changed, 108 insertions(+), 40 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index a24a0ccaa3d67..eb5d673c27a70 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
@@ -92,25 +92,26 @@ static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
   assert(srcType.getRank() == resType.getRank() &&
          "expected rank-preserving reinterpret_cast");
 
-  unsigned collapsedDims = 0;
+  unsigned reducedDims = 0;
 
   for (auto [srcSize, resSize] :
        llvm::zip_equal(srcType.getShape(), resType.getShape())) {
     if (srcSize == resSize)
       continue;
 
-    // Only allow collapsing one non-unit source dim to a unit result dim.
-    if (srcSize != 1 && resSize == 1) {
-      ++collapsedDims;
+    // Only one non-unit source dimension may be reduced.
+    if (srcSize != 1 && resSize < srcSize) {
+      ++reducedDims;
       continue;
     }
 
-    // The sizes differ and both of them are non-unit - ATM not supported.
+    // The size change is not a supported single non-unit source dimension
+    // reduction.
     return false;
   }
 
-  // Make sure there is only one collapsed dimension.
-  return collapsedDims == 1;
+  // Make sure there is only one reduced dimension.
+  return reducedDims == 1;
 }
 
 /// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
@@ -134,21 +135,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. In addition:
-///     identical to the source identity strides, and exactly one non-unit
-///     source
-///  * Non-scalar results must have static offsets, static result strides
-///     dimension collapsed to unit size
-/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
-/// strides are effectively irrelevant).
+/// identity-layout sources.
+/// * Scalar-shaped results may have arbitrary result strides.
+/// * Non-scalar results must have static offsets, static result strides
+///   identical to the source identity strides, and exactly one non-unit
+///   source dimension size reduced.
 ///
-/// Returns nullopt for unsupported
-/// reinterpret_casts.
+/// Returns nullopt for unsupported reinterpret_casts.
 ///
 /// Examples that return info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where K < N
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
@@ -156,10 +155,11 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return no info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[?, N, 1]>>
+///     to memref<1xMxKxf32, strided<[?, N, 1]>>
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xNxPxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where M != N && N != P
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -221,7 +221,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(rc))
+    if (!hasExactlyOneReducedNonUnitDim(rc))
       return std::nullopt;
   }
 
@@ -255,9 +255,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///      the store index is derived from the reinterpret_cast offset.
 ///
 ///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
-///      dimensions except one collapsed-to-unit dimension. Result strides must
-///      be static and identical to the identity strides of the source, and the
-///      static offset selects the collapsed dimension.
+///      dimensions sizes except one. Result
+///      strides must be static and identical to the identity strides of the
+///      source, and the offset must be static.
 ///
 ///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
@@ -268,18 +268,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (one collapsed non-unit dimension)
+///   // BEFORE (same-dimension static slice)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
-///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///       to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   // Assuming OFF delinearizes to [0, 0, j]:
 ///   scf.for %i = 0 to M step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[0, %i, OFF]
+///     scf.for %k = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %k]
+///       memref.store %v, %dst[0, %i, j + %k]
+///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 0955d9638f093..1145c7b010ff1 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -262,20 +262,27 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// The result of collapsing dimensions is not a unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
-func.func private @negative_copy_1D_into_1D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<108xf32> to memref<4xf32, strided<[1]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[IDX]]] : memref<108xf32>
+  // CHECK:      }
   memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -418,9 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-/// 4 is collapsed to a non-unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
-func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_more_reduced_dims(
+func.func private @negative_copy_1D_into_2D_more_reduced_dims(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -437,6 +443,66 @@ func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [6], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 6 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>

>From c96967b11aa5c6fbecdbd1fd4021c8285c8f62aa Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 15 Jul 2026 11:01:41 +0200
Subject: [PATCH 2/2] Address comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 22 ++++----
 .../MemRef/elide-reinterpret-cast.mlir        | 55 +++++++++++++++++--
 2 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index eb5d673c27a70..47180db268d78 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
+static bool hasExactlyOneTruncatedNonUnitDim(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
@@ -92,16 +92,16 @@ static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
   assert(srcType.getRank() == resType.getRank() &&
          "expected rank-preserving reinterpret_cast");
 
-  unsigned reducedDims = 0;
+  unsigned truncatedDims = 0;
 
   for (auto [srcSize, resSize] :
        llvm::zip_equal(srcType.getShape(), resType.getShape())) {
     if (srcSize == resSize)
       continue;
 
-    // Only one non-unit source dimension may be reduced.
+    // Only one non-unit source dimension may be truncated.
     if (srcSize != 1 && resSize < srcSize) {
-      ++reducedDims;
+      ++truncatedDims;
       continue;
     }
 
@@ -110,8 +110,8 @@ static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
     return false;
   }
 
-  // Make sure there is only one reduced dimension.
-  return reducedDims == 1;
+  // Make sure there is only one truncated dimension.
+  return truncatedDims == 1;
 }
 
 /// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
@@ -139,7 +139,7 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// * Scalar-shaped results may have arbitrary result strides.
 /// * Non-scalar results must have static offsets, static result strides
 ///   identical to the source identity strides, and exactly one non-unit
-///   source dimension size reduced.
+///   source dimension size truncated.
 ///
 /// Returns nullopt for unsupported reinterpret_casts.
 ///
@@ -221,7 +221,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneReducedNonUnitDim(rc))
+    if (!hasExactlyOneTruncatedNonUnitDim(rc))
       return std::nullopt;
   }
 
@@ -268,7 +268,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (same-dimension static slice)
+///   // BEFORE (one truncated non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
@@ -276,11 +276,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, j]:
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
 ///   scf.for %i = 0 to M step 1 {
 ///     scf.for %k = 0 to K step 1 {
 ///       %v = memref.load %src[0, %i, %k]
-///       memref.store %v, %dst[0, %i, j + %k]
+///       memref.store %v, %dst[0, %i, OFF + %k]
 ///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 1145c7b010ff1..f305de1ccd0d8 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -425,8 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_more_reduced_dims(
-func.func private @negative_copy_1D_into_2D_more_reduced_dims(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
+func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -443,6 +443,22 @@ func.func private @negative_copy_1D_into_2D_more_reduced_dims(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_into_strided_no_truncated_dims(
+func.func private @negative_copy_into_strided_no_truncated_dims(%src : memref<3x4xf32>,
+  %dst : memref<3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 4], strides: [12, 1]
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -472,10 +488,10 @@ func.func private @copy_2D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -503,6 +519,37 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x10x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+  %src : memref<1x3x11xf32>, %dst : memref<1x10x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [44], sizes: [1, 3, 11], strides: [110, 11, 1]
+    : memref<1x10x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 11 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX0]] : index
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[IDX1]]] : memref<1x10x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>


From llvmlistbot at llvm.org  Wed Jul 15 02:03:25 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 02:03:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a574cdd.b1416da1.78d1e.0c26@mx.google.com>


https://github.com/ioghiban deleted https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 02:03:38 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 02:03:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a574cea.4987c784.63cbc.e23c@mx.google.com>


================
@@ -134,32 +135,31 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. In addition:
-///     identical to the source identity strides, and exactly one non-unit
-///     source
-///  * Non-scalar results must have static offsets, static result strides
-///     dimension collapsed to unit size
-/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
-/// strides are effectively irrelevant).
+/// identity-layout sources.
+/// * Scalar-shaped results may have arbitrary result strides.
+/// * Non-scalar results must have static offsets, static result strides
+///   identical to the source identity strides, and exactly one non-unit
----------------
ioghiban wrote:

Added `@negative_copy_into_strided_no_truncated_dims` test to show the rewrite being skipped if reinterpret_cast changes no dimensions.

https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 02:10:25 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Wed, 15 Jul 2026 02:10:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a574e81.7270d2fd.2d4879.8338@mx.google.com>


https://github.com/gysit approved this pull request.

LGTM!

Feel free to land the fix.

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Wed Jul 15 02:33:41 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 15 Jul 2026 02:33:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a5753f5.c56fb102.7c9cd.b24f@mx.google.com>


================
@@ -268,18 +268,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (one collapsed non-unit dimension)
+///   // BEFORE (same-dimension static slice)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
-///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///       to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
----------------
banach-space wrote:

> Therefore, reducing a non-trailing dimension can delinearize OFF to a different index.

Sure, but that's the case for more general examples. For the example from the comment, `OFF -> [0, 0, OFF]` would be correct - it's worth noting that. Also, `OFF -> j` feels off (i.e. switching from ALL_CAPS to lower_case). I suggest `OFF` --> `DELIN_OFF`.

https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 02:41:59 2026
From: llvmlistbot at llvm.org (Muyao Xiao)
Date: Wed, 15 Jul 2026 02:41:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenMP][MLIR] RegionBranchOpInterface for
 TargetOp, TargetDataOp (PR #209710)
Message-ID: <llvm.org/llvm/llvm-project/pull/209710@github.com>


https://github.com/mooxiu created https://github.com/llvm/llvm-project/pull/209710

This adds RegionBranchOpInterface support to `omp.target` and `omp.target_data`, and adds RegionBranchTerminatorOpInterface to `omp.terminator`.

Both `omp.target` and `omp.target_data` contain structured regions whose control flow enters the region and eventually returns to the parent operation. Without RegionBranchOpInterface, generic region-aware data-flow analyses cannot propagate lattice state across these region boundaries.

For example, an analysis that tracks the host and device states of mapped values cannot propagate its state into an `omp.target` region and back to the parent operation after a mapped value is modified and copied out.

Changes:
- Add RegionBranchOpInterface to omp.target.
- Add RegionBranchOpInterface to omp.target_data.
- Add RegionBranchTerminatorOpInterface to omp.terminator.
- Implement the corresponding region successor methods.

This was previously raised on [LLVM Discussion](https://discourse.llvm.org/t/mlir-openmp-should-omp-target-target-data-implement-regionbranchopinterface/91227).

Similar PR: https://github.com/llvm/llvm-project/pull/202418

>From b874e89f382535c46ceef67f3dc4d2231520124f Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 18:19:00 +0900
Subject: [PATCH] implement regional branch interface for omp::TargetOp,
 omp::TargetDataOp, and omp::terminatorOp

---
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  7 +++---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 22 +++++++++++++++++++
 2 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 98242a66e411f..e1840a3bee317 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -222,7 +222,7 @@ def ParallelOp : OpenMP_Op<"parallel", traits = [
   let hasRegionVerifier = 1;
 }
 
-def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure]> {
+def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure, RegionBranchTerminatorOpInterface]> {
   let summary = "terminator for OpenMP regions";
   let description = [{
     A terminator operation for regions that appear in the body of OpenMP
@@ -1457,7 +1457,8 @@ def MapInfoOp : OpenMP_Op<"map.info", [AttrSizedOperandSegments]> {
 //===---------------------------------------------------------------------===//
 
 def TargetDataOp: OpenMP_Op<"target_data", traits = [
-    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>
+    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     OpenMP_DeviceClause, OpenMP_IfClause, OpenMP_MapClause,
     OpenMP_UseDeviceAddrClause, OpenMP_UseDevicePtrClause
@@ -1621,7 +1622,7 @@ def TargetUpdateOp: OpenMP_Op<"target_update", traits = [
 def TargetOp : OpenMP_Op<"target", traits = [
     AttrSizedOperandSegments, BlockArgOpenMPOpInterface,
     DeclareOpInterfaceMethods<ComposableOpInterface>, IsolatedFromAbove,
-    OutlineableOpenMPOpInterface
+    OutlineableOpenMPOpInterface, DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     // TODO: Complete clause list (defaultmap, uses_allocators).
     OpenMP_AllocateClause, OpenMP_DependClause, OpenMP_DeviceClause,
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 3a3798c3ad00b..d50f96c8b0669 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2571,6 +2571,17 @@ LogicalResult TargetDataOp::verify() {
   return verifyMapClause(*this, getMapVars(), getMapIterated());
 }
 
+// Adapted from fir.if implementation.
+void TargetDataOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // TargetEnterDataOp
 //===----------------------------------------------------------------------===//
@@ -2853,6 +2864,17 @@ LogicalResult TargetOp::verifyRegions() {
   return success();
 }
 
+// Copy from `TargetDataOp::getSuccessorRegions`
+void TargetOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // ParallelOp
 //===----------------------------------------------------------------------===//


From llvmlistbot at llvm.org  Wed Jul 15 02:43:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 02:43:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenMP][MLIR] RegionBranchOpInterface for
 TargetOp, TargetDataOp (PR #209710)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209710@github.com>
Message-ID: <6a57564b.d681cab9.333b2b.17e7@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index d50f96c8b..9e55391d6 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2573,8 +2573,8 @@ LogicalResult TargetDataOp::verify() {
 
 // Adapted from fir.if implementation.
 void TargetDataOp::getSuccessorRegions(
-  mlir::RegionBranchPoint point, 
-  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+    mlir::RegionBranchPoint point,
+    llvm::SmallVectorImpl<::mlir::RegionSuccessor> &regions) {
   if (!point.isParent()) {
     regions.push_back(mlir::RegionSuccessor::parent());
     return;
@@ -2866,8 +2866,8 @@ LogicalResult TargetOp::verifyRegions() {
 
 // Copy from `TargetDataOp::getSuccessorRegions`
 void TargetOp::getSuccessorRegions(
-  mlir::RegionBranchPoint point, 
-  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+    mlir::RegionBranchPoint point,
+    llvm::SmallVectorImpl<::mlir::RegionSuccessor> &regions) {
   if (!point.isParent()) {
     regions.push_back(mlir::RegionSuccessor::parent());
     return;

``````````

</details>


https://github.com/llvm/llvm-project/pull/209710

From llvmlistbot at llvm.org  Wed Jul 15 02:49:27 2026
From: llvmlistbot at llvm.org (Aayush Shrivastava)
Date: Wed, 15 Jul 2026 02:49:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5757a7.1905e68e.18ad68.83b3@mx.google.com>


iamaayushrivastava wrote:

> LGTM!
> 
> Feel free to land the fix.

Thanks @gysit for the review and approval.



https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Wed Jul 15 03:23:19 2026
From: llvmlistbot at llvm.org (Muyao Xiao)
Date: Wed, 15 Jul 2026 03:23:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenMP][MLIR] RegionBranchOpInterface for
 TargetOp, TargetDataOp (PR #209710)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209710@github.com>
Message-ID: <6a575f97.f3151ff8.305c37.985b@mx.google.com>


https://github.com/mooxiu updated https://github.com/llvm/llvm-project/pull/209710

>From b874e89f382535c46ceef67f3dc4d2231520124f Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 18:19:00 +0900
Subject: [PATCH 1/2] implement regional branch interface for omp::TargetOp,
 omp::TargetDataOp, and omp::terminatorOp

---
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  7 +++---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 22 +++++++++++++++++++
 2 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 98242a66e411f..e1840a3bee317 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -222,7 +222,7 @@ def ParallelOp : OpenMP_Op<"parallel", traits = [
   let hasRegionVerifier = 1;
 }
 
-def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure]> {
+def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure, RegionBranchTerminatorOpInterface]> {
   let summary = "terminator for OpenMP regions";
   let description = [{
     A terminator operation for regions that appear in the body of OpenMP
@@ -1457,7 +1457,8 @@ def MapInfoOp : OpenMP_Op<"map.info", [AttrSizedOperandSegments]> {
 //===---------------------------------------------------------------------===//
 
 def TargetDataOp: OpenMP_Op<"target_data", traits = [
-    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>
+    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     OpenMP_DeviceClause, OpenMP_IfClause, OpenMP_MapClause,
     OpenMP_UseDeviceAddrClause, OpenMP_UseDevicePtrClause
@@ -1621,7 +1622,7 @@ def TargetUpdateOp: OpenMP_Op<"target_update", traits = [
 def TargetOp : OpenMP_Op<"target", traits = [
     AttrSizedOperandSegments, BlockArgOpenMPOpInterface,
     DeclareOpInterfaceMethods<ComposableOpInterface>, IsolatedFromAbove,
-    OutlineableOpenMPOpInterface
+    OutlineableOpenMPOpInterface, DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     // TODO: Complete clause list (defaultmap, uses_allocators).
     OpenMP_AllocateClause, OpenMP_DependClause, OpenMP_DeviceClause,
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 3a3798c3ad00b..d50f96c8b0669 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2571,6 +2571,17 @@ LogicalResult TargetDataOp::verify() {
   return verifyMapClause(*this, getMapVars(), getMapIterated());
 }
 
+// Adapted from fir.if implementation.
+void TargetDataOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // TargetEnterDataOp
 //===----------------------------------------------------------------------===//
@@ -2853,6 +2864,17 @@ LogicalResult TargetOp::verifyRegions() {
   return success();
 }
 
+// Copy from `TargetDataOp::getSuccessorRegions`
+void TargetOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // ParallelOp
 //===----------------------------------------------------------------------===//

>From b81004d4bfbce9be9742c40c7a28e884433f6669 Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 19:22:52 +0900
Subject: [PATCH 2/2] update to latest API: no parent() anymore

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index d50f96c8b0669..ca201aba48a03 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2576,7 +2576,7 @@ void TargetDataOp::getSuccessorRegions(
   mlir::RegionBranchPoint point, 
   llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
   if (!point.isParent()) {
-    regions.push_back(mlir::RegionSuccessor::parent());
+    regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;
   }
   regions.push_back(mlir::RegionSuccessor(&getRegion()));
@@ -2869,7 +2869,7 @@ void TargetOp::getSuccessorRegions(
   mlir::RegionBranchPoint point, 
   llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
   if (!point.isParent()) {
-    regions.push_back(mlir::RegionSuccessor::parent());
+    regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;
   }
   regions.push_back(mlir::RegionSuccessor(&getRegion()));


From llvmlistbot at llvm.org  Wed Jul 15 03:25:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 03:25:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP in CONCAT
	(PR #209719)
Message-ID: <llvm.org/llvm/llvm-project/pull/209719@github.com>


https://github.com/Philip742 created https://github.com/llvm/llvm-project/pull/209719

Adds support to profile compliance for MXFP variants of CONCAT


Change-Id: I1e0c6569df1188e5e5fff6dde12eb9f9f60f169c

>From e4d2e91e5706194faf3468050a6a2886fd5b21b5 Mon Sep 17 00:00:00 2001
From: Philip Wilkinson <philip.wilkinson at arm.com>
Date: Wed, 24 Jun 2026 14:40:09 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP in CONCAT

Adds support to profile compliance for MXFP variants of CONCAT

Signed-off-by: Philip Wilkinson <philip.wilkinson at arm.com>
Change-Id: I1e0c6569df1188e5e5fff6dde12eb9f9f60f169c
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 26 ++++++++++++++++++-
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  6 ++++-
 mlir/test/Dialect/Tosa/availability.mlir      |  2 +-
 mlir/test/Dialect/Tosa/ops.mlir               |  8 ++++++
 .../tosa-validation-version-1p0-invalid.mlir  |  7 +++++
 .../tosa-validation-version-1p1-valid.mlir    |  8 ++++++
 6 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 291097a7f7514..a8bf2d36a49ed 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -922,7 +922,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.pad",
      {{{Extension::int64},
        {{{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index e3b74f5a07d3f..18d48d15b07a4 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2241,7 +2241,11 @@ def Tosa_ConcatOp : Tosa_InferTensorTypeOp<"concat", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT16, Tosa_EXT_INT64,
+               Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+               Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index bc014fb9b928e..cd5efc5995e4c 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -521,7 +521,7 @@ func.func @test_reduce_sum(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
 // CHECK-LABEL: concat
 func.func @test_concat(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<26x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x3xf32>, tensor<13x21x3xf32>) -> tensor<26x21x3xf32>
   return %0 : tensor<26x21x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 4e71d512def2d..d5df6ff1842c5 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -2010,3 +2010,11 @@ func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x
   %1 = tosa.cast %0 : (tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<2x32xf32>
   return %1 : tensor<2x32xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index bea262a11a391..952de03319b0f 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -246,3 +246,10 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{'tosa.concat' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp8e4m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index d290ff71f052d..2c04f0451bc11 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -518,3 +518,11 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}


From llvmlistbot at llvm.org  Wed Jul 15 03:26:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 03:26:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP in CONCAT
	(PR #209719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209719@github.com>
Message-ID: <6a57603a.92ea0e2a.13fd37.ae24@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Philip742

<details>
<summary>Changes</summary>

Adds support to profile compliance for MXFP variants of CONCAT


Change-Id: I1e0c6569df1188e5e5fff6dde12eb9f9f60f169c

---
Full diff: https://github.com/llvm/llvm-project/pull/209719.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+25-1) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+5-1) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+8) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+7) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+8) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 291097a7f7514..a8bf2d36a49ed 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -922,7 +922,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.pad",
      {{{Extension::int64},
        {{{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index e3b74f5a07d3f..18d48d15b07a4 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2241,7 +2241,11 @@ def Tosa_ConcatOp : Tosa_InferTensorTypeOp<"concat", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT16, Tosa_EXT_INT64,
+               Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+               Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index bc014fb9b928e..cd5efc5995e4c 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -521,7 +521,7 @@ func.func @test_reduce_sum(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
 // CHECK-LABEL: concat
 func.func @test_concat(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<26x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x3xf32>, tensor<13x21x3xf32>) -> tensor<26x21x3xf32>
   return %0 : tensor<26x21x3xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 4e71d512def2d..d5df6ff1842c5 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -2010,3 +2010,11 @@ func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x
   %1 = tosa.cast %0 : (tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<2x32xf32>
   return %1 : tensor<2x32xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index bea262a11a391..952de03319b0f 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -246,3 +246,10 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{'tosa.concat' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp8e4m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index d290ff71f052d..2c04f0451bc11 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -518,3 +518,11 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/209719

From llvmlistbot at llvm.org  Wed Jul 15 03:48:23 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Wed, 15 Jul 2026 03:48:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Canonicalize `addf(negf(x),
 y)` to `subf(y, x)` (PR #209277)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209277@github.com>
Message-ID: <6a576577.66c0679a.3a40e5.d129@mx.google.com>


MaxGraey wrote:

> Does instcombine perform the same rewrite?

Yes, this legit for safe math

https://github.com/llvm/llvm-project/pull/209277

From llvmlistbot at llvm.org  Wed Jul 15 03:51:04 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 03:51:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a576618.4987c784.a8336.8a03@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/209536

>From 5905cd319ad0a104d1316e05f53d4a40eff839b3 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Tue, 14 Jul 2026 17:43:06 +0200
Subject: [PATCH 1/3] [memref] Support reducing copied dims to non-unit in
 reinterpret_cast elision

---
 .../Transforms/ElideReinterpretCast.cpp       | 62 ++++++-------
 .../MemRef/elide-reinterpret-cast.mlir        | 86 ++++++++++++++++---
 2 files changed, 108 insertions(+), 40 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index a24a0ccaa3d67..eb5d673c27a70 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
@@ -92,25 +92,26 @@ static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
   assert(srcType.getRank() == resType.getRank() &&
          "expected rank-preserving reinterpret_cast");
 
-  unsigned collapsedDims = 0;
+  unsigned reducedDims = 0;
 
   for (auto [srcSize, resSize] :
        llvm::zip_equal(srcType.getShape(), resType.getShape())) {
     if (srcSize == resSize)
       continue;
 
-    // Only allow collapsing one non-unit source dim to a unit result dim.
-    if (srcSize != 1 && resSize == 1) {
-      ++collapsedDims;
+    // Only one non-unit source dimension may be reduced.
+    if (srcSize != 1 && resSize < srcSize) {
+      ++reducedDims;
       continue;
     }
 
-    // The sizes differ and both of them are non-unit - ATM not supported.
+    // The size change is not a supported single non-unit source dimension
+    // reduction.
     return false;
   }
 
-  // Make sure there is only one collapsed dimension.
-  return collapsedDims == 1;
+  // Make sure there is only one reduced dimension.
+  return reducedDims == 1;
 }
 
 /// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
@@ -134,21 +135,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. In addition:
-///     identical to the source identity strides, and exactly one non-unit
-///     source
-///  * Non-scalar results must have static offsets, static result strides
-///     dimension collapsed to unit size
-/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
-/// strides are effectively irrelevant).
+/// identity-layout sources.
+/// * Scalar-shaped results may have arbitrary result strides.
+/// * Non-scalar results must have static offsets, static result strides
+///   identical to the source identity strides, and exactly one non-unit
+///   source dimension size reduced.
 ///
-/// Returns nullopt for unsupported
-/// reinterpret_casts.
+/// Returns nullopt for unsupported reinterpret_casts.
 ///
 /// Examples that return info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where K < N
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
@@ -156,10 +155,11 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return no info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[?, N, 1]>>
+///     to memref<1xMxKxf32, strided<[?, N, 1]>>
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xNxPxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where M != N && N != P
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -221,7 +221,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(rc))
+    if (!hasExactlyOneReducedNonUnitDim(rc))
       return std::nullopt;
   }
 
@@ -255,9 +255,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///      the store index is derived from the reinterpret_cast offset.
 ///
 ///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
-///      dimensions except one collapsed-to-unit dimension. Result strides must
-///      be static and identical to the identity strides of the source, and the
-///      static offset selects the collapsed dimension.
+///      dimensions sizes except one. Result
+///      strides must be static and identical to the identity strides of the
+///      source, and the offset must be static.
 ///
 ///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
@@ -268,18 +268,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (one collapsed non-unit dimension)
+///   // BEFORE (same-dimension static slice)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
-///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///       to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   // Assuming OFF delinearizes to [0, 0, j]:
 ///   scf.for %i = 0 to M step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[0, %i, OFF]
+///     scf.for %k = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %k]
+///       memref.store %v, %dst[0, %i, j + %k]
+///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 0955d9638f093..1145c7b010ff1 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -262,20 +262,27 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// The result of collapsing dimensions is not a unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
-func.func private @negative_copy_1D_into_1D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<108xf32> to memref<4xf32, strided<[1]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[IDX]]] : memref<108xf32>
+  // CHECK:      }
   memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -418,9 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-/// 4 is collapsed to a non-unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
-func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_more_reduced_dims(
+func.func private @negative_copy_1D_into_2D_more_reduced_dims(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -437,6 +443,66 @@ func.func private @negative_copy_1D_into_2D_strided_diff_dim_sizes(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [6], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 6 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>

>From c96967b11aa5c6fbecdbd1fd4021c8285c8f62aa Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 15 Jul 2026 11:01:41 +0200
Subject: [PATCH 2/3] Address comments

---
 .../Transforms/ElideReinterpretCast.cpp       | 22 ++++----
 .../MemRef/elide-reinterpret-cast.mlir        | 55 +++++++++++++++++--
 2 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index eb5d673c27a70..47180db268d78 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
+static bool hasExactlyOneTruncatedNonUnitDim(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
@@ -92,16 +92,16 @@ static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
   assert(srcType.getRank() == resType.getRank() &&
          "expected rank-preserving reinterpret_cast");
 
-  unsigned reducedDims = 0;
+  unsigned truncatedDims = 0;
 
   for (auto [srcSize, resSize] :
        llvm::zip_equal(srcType.getShape(), resType.getShape())) {
     if (srcSize == resSize)
       continue;
 
-    // Only one non-unit source dimension may be reduced.
+    // Only one non-unit source dimension may be truncated.
     if (srcSize != 1 && resSize < srcSize) {
-      ++reducedDims;
+      ++truncatedDims;
       continue;
     }
 
@@ -110,8 +110,8 @@ static bool hasExactlyOneReducedNonUnitDim(memref::ReinterpretCastOp rc) {
     return false;
   }
 
-  // Make sure there is only one reduced dimension.
-  return reducedDims == 1;
+  // Make sure there is only one truncated dimension.
+  return truncatedDims == 1;
 }
 
 /// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
@@ -139,7 +139,7 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// * Scalar-shaped results may have arbitrary result strides.
 /// * Non-scalar results must have static offsets, static result strides
 ///   identical to the source identity strides, and exactly one non-unit
-///   source dimension size reduced.
+///   source dimension size truncated.
 ///
 /// Returns nullopt for unsupported reinterpret_casts.
 ///
@@ -221,7 +221,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneReducedNonUnitDim(rc))
+    if (!hasExactlyOneTruncatedNonUnitDim(rc))
       return std::nullopt;
   }
 
@@ -268,7 +268,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (same-dimension static slice)
+///   // BEFORE (one truncated non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
 ///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
@@ -276,11 +276,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, j]:
+///   // Assuming OFF delinearizes to [0, 0, OFF]:
 ///   scf.for %i = 0 to M step 1 {
 ///     scf.for %k = 0 to K step 1 {
 ///       %v = memref.load %src[0, %i, %k]
-///       memref.store %v, %dst[0, %i, j + %k]
+///       memref.store %v, %dst[0, %i, OFF + %k]
 ///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
diff --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 1145c7b010ff1..f305de1ccd0d8 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -425,8 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_more_reduced_dims(
-func.func private @negative_copy_1D_into_2D_more_reduced_dims(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
+func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -443,6 +443,22 @@ func.func private @negative_copy_1D_into_2D_more_reduced_dims(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_into_strided_no_truncated_dims(
+func.func private @negative_copy_into_strided_no_truncated_dims(%src : memref<3x4xf32>,
+  %dst : memref<3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 4], strides: [12, 1]
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
@@ -472,10 +488,10 @@ func.func private @copy_2D_into_2D_strided_zero_offset(
   return
 }
 
-// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset(
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
 // CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
-func.func private @copy_2D_into_2D_strided_nonzero_offset(
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
   %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
   // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
@@ -503,6 +519,37 @@ func.func private @copy_2D_into_2D_strided_nonzero_offset(
   return
 }
 
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x10x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+  %src : memref<1x3x11xf32>, %dst : memref<1x10x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [44], sizes: [1, 3, 11], strides: [110, 11, 1]
+    : memref<1x10x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 11 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX0]] : index
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[IDX1]]] : memref<1x10x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>

>From e5f9dd3ccc3810a742649077616c1ddccfcdce8c Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Wed, 15 Jul 2026 12:50:31 +0200
Subject: [PATCH 3/3] Fixup

---
 mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index 47180db268d78..49f30a594e0ee 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -276,11 +276,11 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   // Assuming OFF delinearizes to [0, 0, DELIN_OFF]:
 ///   scf.for %i = 0 to M step 1 {
 ///     scf.for %k = 0 to K step 1 {
 ///       %v = memref.load %src[0, %i, %k]
-///       memref.store %v, %dst[0, %i, OFF + %k]
+///       memref.store %v, %dst[0, %i, DELIN_OFF + %k]
 ///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {


From llvmlistbot at llvm.org  Wed Jul 15 03:51:44 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 03:51:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a576640.c574b9b4.2aa0bf.b808@mx.google.com>


================
@@ -268,18 +268,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (one collapsed non-unit dimension)
+///   // BEFORE (same-dimension static slice)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
-///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///       to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
----------------
ioghiban wrote:

Changed to `DELIN_OFF`. PTAL. Thank you.

https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 04:15:46 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Wed, 15 Jul 2026 04:15:46 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a576be2.f1c1923d.1bf95f.c4b5@mx.google.com>


https://github.com/jsjodin updated https://github.com/llvm/llvm-project/pull/209592

>From d6d61356abf6768b91fb1c4fe5094417d1dd39b9 Mon Sep 17 00:00:00 2001
From: Jan Leyonberg <jan_sjodin at yahoo.com>
Date: Tue, 14 Jul 2026 14:54:30 -0400
Subject: [PATCH 1/2] [CIR][OpenMP] Add host op filtering pass to CIR pipeline

This patch adds the host op filtering pass which prevents host code being
lowered when compiling for the target device.
---
 .../CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp |  1 +
 .../CodeGenOpenMP/target-host-op-filtering.c  | 35 +++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100644 clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c

diff --git a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
index 0e412090a16da..67406d261ce1a 100644
--- a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
+++ b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
@@ -5134,6 +5134,7 @@ void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
   mlir::populateCIRPreLoweringPasses(pm);
   pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
+  pm.addPass(mlir::omp::createHostOpFilteringPass());
 }
 
 std::unique_ptr<llvm::Module>
diff --git a/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c b/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c
new file mode 100644
index 0000000000000..ab20b285c79ed
--- /dev/null
+++ b/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c
@@ -0,0 +1,35 @@
+// Verifies the omp-host-op-filtering pass runs in the CIR-to-LLVM device
+// pipeline.//
+// RUN: %clang_cc1 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa \
+// RUN:   -fclangir -emit-llvm-bc %s -o %t-cir-host.bc
+// RUN: %clang_cc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-is-target-device \
+// RUN:   -fopenmp-host-ir-file-path %t-cir-host.bc \
+// RUN:   -fclangir -emit-llvm %s -o - \
+// RUN:   | FileCheck %s --check-prefix=LLVM
+
+void use(int);
+int host_helper(int);
+
+void f(int x) {
+  // Host-only computation that must not leak into the device module.
+  int y = host_helper(x);
+  use(y);
+#pragma omp target map(tofrom : x)
+  {
+    x = x + 1;
+  }
+}
+
+// LLVM-LABEL: define hidden void @f(
+// LLVM-NOT:     call i32 @host_helper
+// LLVM-NOT:     call void @use
+// LLVM:         ret void
+
+// LLVM-LABEL: define weak_odr protected amdgpu_kernel void @__omp_offloading_{{.*}}_f_l
+// LLVM:         call i32 @__kmpc_target_init(
+// LLVM:       user_code.entry:
+// LLVM:         %[[LD:.*]] = load i32, ptr %{{.*}}, align 4
+// LLVM:         %[[ADD:.*]] = add nsw i32 %[[LD]], 1
+// LLVM:         store i32 %[[ADD]], ptr %{{.*}}, align 4
+// LLVM:         call void @__kmpc_target_deinit()
+// LLVM:         ret void

>From 9ce9d47344b8b493b0c6077b692cb7eaf967f2c6 Mon Sep 17 00:00:00 2001
From: Jan Leyonberg <jan_sjodin at yahoo.com>
Date: Wed, 15 Jul 2026 07:13:31 -0400
Subject: [PATCH 2/2] Add guard for adding OpenMP passes.

---
 clang/include/clang/CIR/LowerToLLVM.h                |  2 +-
 clang/include/clang/CIR/Passes.h                     |  5 +++--
 clang/lib/CIR/FrontendAction/CIRGenAction.cpp        |  9 +++++----
 clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp  | 12 +++++++-----
 clang/tools/cir-opt/cir-opt.cpp                      |  9 +++++++--
 clang/tools/cir-translate/cir-translate.cpp          |  7 ++++++-
 .../mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td       |  9 +++++++++
 7 files changed, 38 insertions(+), 15 deletions(-)

diff --git a/clang/include/clang/CIR/LowerToLLVM.h b/clang/include/clang/CIR/LowerToLLVM.h
index df5f9221842ea..305f54017ebe0 100644
--- a/clang/include/clang/CIR/LowerToLLVM.h
+++ b/clang/include/clang/CIR/LowerToLLVM.h
@@ -32,7 +32,7 @@ namespace cir {
 namespace direct {
 std::unique_ptr<llvm::Module>
 lowerDirectlyFromCIRToLLVMIR(mlir::ModuleOp mlirModule,
-                             llvm::LLVMContext &llvmCtx,
+                             llvm::LLVMContext &llvmCtx, bool enableOpenMP,
                              llvm::StringRef mlirSaveTempsOutFile = {},
                              llvm::vfs::FileSystem *fs = nullptr);
 } // namespace direct
diff --git a/clang/include/clang/CIR/Passes.h b/clang/include/clang/CIR/Passes.h
index 1d202ce5bfb52..a42f20405af24 100644
--- a/clang/include/clang/CIR/Passes.h
+++ b/clang/include/clang/CIR/Passes.h
@@ -22,8 +22,9 @@ namespace direct {
 /// Create a pass that fully lowers CIR to the LLVMIR dialect.
 std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass();
 
-/// Adds passes that fully lower CIR to the LLVMIR dialect.
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm);
+/// Adds passes that fully lower CIR to the LLVMIR dialect. When enableOpenMP
+/// is set (-fopenmp), the OpenMP lowering passes are also added.
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP);
 
 } // namespace direct
 } // end namespace cir
diff --git a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
index 92c9ef5499693..28229ac8c58fd 100644
--- a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
+++ b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
@@ -57,9 +57,10 @@ getBackendActionFromOutputType(CIRGenAction::OutputType Action) {
 
 static std::unique_ptr<llvm::Module>
 lowerFromCIRToLLVMIR(mlir::ModuleOp MLIRModule, llvm::LLVMContext &LLVMCtx,
+                     bool EnableOpenMP,
                      llvm::StringRef mlirSaveTempsOutFile = {},
                      llvm::vfs::FileSystem *fs = nullptr) {
-  return direct::lowerDirectlyFromCIRToLLVMIR(MLIRModule, LLVMCtx,
+  return direct::lowerDirectlyFromCIRToLLVMIR(MLIRModule, LLVMCtx, EnableOpenMP,
                                               mlirSaveTempsOutFile, fs);
 }
 
@@ -179,9 +180,9 @@ class CIRGenConsumer : public clang::ASTConsumer {
           MlirModule->print(out);
       }
 
-      std::unique_ptr<llvm::Module> LLVMModule =
-          lowerFromCIRToLLVMIR(MlirModule, LLVMCtx, mlirSaveTempsOutFile,
-                               &CI.getVirtualFileSystem());
+      std::unique_ptr<llvm::Module> LLVMModule = lowerFromCIRToLLVMIR(
+          MlirModule, LLVMCtx, C.getLangOpts().OpenMP, mlirSaveTempsOutFile,
+          &CI.getVirtualFileSystem());
 
       if (linkInModules(*LLVMModule))
         return;
diff --git a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
index 67406d261ce1a..ebe74399dda62 100644
--- a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
+++ b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
@@ -5130,23 +5130,25 @@ std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass() {
   return std::make_unique<ConvertCIRToLLVMPass>();
 }
 
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP) {
   mlir::populateCIRPreLoweringPasses(pm);
-  pm.addPass(mlir::omp::createMarkDeclareTargetPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
-  pm.addPass(mlir::omp::createHostOpFilteringPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
 }
 
 std::unique_ptr<llvm::Module>
 lowerDirectlyFromCIRToLLVMIR(mlir::ModuleOp mlirModule, LLVMContext &llvmCtx,
-                             StringRef mlirSaveTempsOutFile,
+                             bool enableOpenMP, StringRef mlirSaveTempsOutFile,
                              llvm::vfs::FileSystem *fs) {
   llvm::TimeTraceScope scope("lower from CIR to LLVM directly");
 
   mlir::MLIRContext *mlirCtx = mlirModule.getContext();
 
   mlir::PassManager pm(mlirCtx);
-  populateCIRToLLVMPasses(pm);
+  populateCIRToLLVMPasses(pm, enableOpenMP);
 
   (void)mlir::applyPassManagerCLOptions(pm);
 
diff --git a/clang/tools/cir-opt/cir-opt.cpp b/clang/tools/cir-opt/cir-opt.cpp
index 6742985e149e8..a76e4ca6a70e9 100644
--- a/clang/tools/cir-opt/cir-opt.cpp
+++ b/clang/tools/cir-opt/cir-opt.cpp
@@ -26,7 +26,12 @@
 #include "clang/CIR/Passes.h"
 
 struct CIRToLLVMPipelineOptions
-    : public mlir::PassPipelineOptions<CIRToLLVMPipelineOptions> {};
+    : public mlir::PassPipelineOptions<CIRToLLVMPipelineOptions> {
+  Option<bool> enableOpenMP{
+      *this, "enable-openmp",
+      llvm::cl::desc("Add OpenMP-specific CIR-to-LLVM lowering passes"),
+      llvm::cl::init(false)};
+};
 
 int main(int argc, char **argv) {
   // TODO: register needed MLIR passes for CIR?
@@ -44,7 +49,7 @@ int main(int argc, char **argv) {
   mlir::PassPipelineRegistration<CIRToLLVMPipelineOptions> pipeline(
       "cir-to-llvm", "",
       [](mlir::OpPassManager &pm, const CIRToLLVMPipelineOptions &options) {
-        cir::direct::populateCIRToLLVMPasses(pm);
+        cir::direct::populateCIRToLLVMPasses(pm, options.enableOpenMP);
       });
 
   ::mlir::registerPass([]() -> std::unique_ptr<::mlir::Pass> {
diff --git a/clang/tools/cir-translate/cir-translate.cpp b/clang/tools/cir-translate/cir-translate.cpp
index 26adf2cf47d17..9bbbb3939a62f 100644
--- a/clang/tools/cir-translate/cir-translate.cpp
+++ b/clang/tools/cir-translate/cir-translate.cpp
@@ -159,8 +159,13 @@ void registerToLLVMTranslation() {
           return mlir::failure();
 
         llvm::LLVMContext llvmContext;
+        bool enableOpenMP = false;
+        if (auto offloadMod = llvm::dyn_cast<mlir::omp::OffloadModuleInterface>(
+                cirModule.getOperation()))
+          enableOpenMP = offloadMod.getIsOpenMP();
         std::unique_ptr<llvm::Module> llvmModule =
-            cir::direct::lowerDirectlyFromCIRToLLVMIR(cirModule, llvmContext);
+            cir::direct::lowerDirectlyFromCIRToLLVMIR(cirModule, llvmContext,
+                                                      enableOpenMP);
         if (!llvmModule)
           return mlir::failure();
         llvmModule->print(output, nullptr);
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index 51f925b17f47e..5ba66c323845a 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
     InterfaceMethod<
       /*description=*/[{
       Set the attribute on the current module with the specified boolean


From llvmlistbot at llvm.org  Wed Jul 15 04:27:37 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 15 Jul 2026 04:27:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add SPIRVToLLVM conversions
 for IAddCarry and ISubBorrow (PR #203605)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203605@github.com>
Message-ID: <6a576ea9.510f576d.d8fd6.bec0@mx.google.com>


https://github.com/aobolensk edited https://github.com/llvm/llvm-project/pull/203605

From llvmlistbot at llvm.org  Wed Jul 15 04:47:36 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 04:47:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa][WIP] More informative error
 messages on block scaled types (PR #209736)
Message-ID: <llvm.org/llvm/llvm-project/pull/209736@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/209736

This draft PR demonstrates how error messages can be added to block scaled types. This aims to restore previous functionality that was reverted by https://github.com/llvm/llvm-project/pull/207995, but uses the ODS string interpolation mechanism instead to prevent incompatibility with PDLL.

I plan to create separate PR's for the 3 commits in this PR to make it easier for reviewers.

>From 82a25847092d71d6ae16f9a2ac456b2d3b8e7111 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 9 Jul 2026 01:00:27 +0100
Subject: [PATCH 1/3] [mlir][tosa] Combine unranked/ranked tensor types into
 single type

This commit refactors the defined TOSA types to combine unranked and
ranked tensor types into a single type `TosaTensorOf`. This helps
simplify the type definitions and allows all tensor types to support
both unranked and ranked tensors.

Change-Id: I9866f2e26db44401cbb0e99ce1dbaad131ab574e
---
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 108 +++++++-----------
 .../TosaToLinalg/tosa-to-linalg-pipeline.mlir |   2 +-
 mlir/test/Dialect/Tosa/invalid.mlir           |  40 +++----
 mlir/test/Dialect/Tosa/verifier.mlir          |  10 +-
 4 files changed, 66 insertions(+), 94 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 1518024ecaa19..2ab135befaa96 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>],
-      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tensor">;
+      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tosa-conformant tensor">;
 
 class TosaScalarTensorOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>, AllDimensionsAreSizeOne],
       "tosa-conformant scalar tensor">;
 
@@ -217,9 +212,6 @@ def Tosa_FloatTensor : TosaTensorOf<[AnyFloat]>;
 // Either ranked or unranked tensor of TOSA supported element types.
 def Tosa_Tensor : TosaTensorOf<[Tosa_AnyNumber]>;
 
-// Must be ranked but no further constraints
-def Tosa_RankedTensor : TosaRankedTensorOf<[Tosa_AnyNumber]>;
-
 // Any tensor element type allowed in Tosa ops.
 def Tosa_ElementType : Type<Or<[Tosa_Int.predicate, Tosa_QuantizedInt.predicate,
                                 AnyFloat.predicate]>, "tosa.dtype">;
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
+def Tosa_1DInt16Or32Tensor : TosaTensorRankOf<[Tosa_Int16Or32], [1]>;
 
-// Ranked tensors up to given rank.
-def Tosa_Tensor1Dto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>]>;
-def Tosa_Tensor1Dto6D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>]>;
+// Tensors up to a given rank. Unranked tensors are accepted.
+def Tosa_Tensor1Dto4D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>;
+def Tosa_Tensor1Dto6D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>;
 
-def Tosa_TensorUpto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>]>;
+def Tosa_TensorUpto4D : TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>;
 
-def Tosa_IndexTensor1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>]>;
-def Tosa_IndexTensor2D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>]>;
+def Tosa_IndexTensor1D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>;
+def Tosa_IndexTensor2D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>;
 
-def Tosa_TensorAtLeast1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
+def Tosa_TensorAtLeast1D
+    : TosaTensorOf<[Tosa_AnyNumber], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 def Tosa_MXFPDataTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [3]>;
 def Tosa_MXFPScaleTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [3]>;
 def Tosa_MXFPDataTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [4]>;
 def Tosa_MXFPScaleTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [4]>;
 def Tosa_MXFPDataTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPValue], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 def Tosa_MXFPScaleTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPScale], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
index 67b6aa63f2293..373b5b71cb68b 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
@@ -14,7 +14,7 @@ func.func @tensor_with_unknown_rank(%arg0: tensor<*xi32>) -> tensor<*xi32> {
 
 // check that tosa verify kick in
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..0e66cfa02b67f 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -317,7 +317,7 @@ func.func @test_concat_input_output_rank_mismatch(%arg0: tensor<2x2xf32>, %arg1:
 func.func @test_pad_invalid_padConst_rank(%arg0: tensor<13x21xf32>) {
   %0 = tosa.const_shape {values = dense<1> : tensor<4xindex>} : () -> !tosa.shape<4>
   %1 = "tosa.const"() {values = dense<3.14> : tensor<2xf32>} : () -> tensor<2xf32>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
   %2 = tosa.pad %arg0, %0, %1 : (tensor<13x21xf32>, !tosa.shape<4>, tensor<2xf32>) -> tensor<13x21xf32>
   return
 }
@@ -543,7 +543,7 @@ func.func @test_const_attribute_type_mismatch() -> tensor<100x100xf32> {
 func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x29x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x29x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -554,7 +554,7 @@ func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1:
 func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x?x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x?x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x?x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -564,7 +564,7 @@ func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<
 // -----
 
 func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x7x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x7x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x7x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -573,7 +573,7 @@ func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %ar
 // -----
 
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -673,7 +673,7 @@ func.func @test_tile_io_rank_mismatch() {
 
 // CHECK-LABEL: test_table_rank0_table
 func.func @test_table_rank0_table(%arg0: tensor<64xi16>, %arg1: tensor<i16>) {
-  // expected-error at +1 {{'tosa.table' op operand #1 must be 1-d tosa-conformant tensor, but got 'tensor<i16>'}}
+  // expected-error at +1 {{'tosa.table' op operand #1 must be 1D tosa-conformant tensor of number values, but got 'tensor<i16>'}}
   %0 = tosa.table %arg0, %arg1 : (tensor<64xi16>, tensor<i16>) -> tensor<64xi16>
   return
 }
@@ -1011,7 +1011,7 @@ func.func @test_non_tosa_ops() {
 func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<13x21x3xf8E5M2> {
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   %cst = "tosa.const"() { values = dense<-0.0> : tensor<f8E4M3FN> } : () -> tensor<f8E4M3FN>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
   %0 = tosa.pad %arg0, %padding, %cst : (tensor<13x21x3xf8E4M3FN>, !tosa.shape<6>, tensor<f8E4M3FN>) -> tensor<13x21x3xf8E5M2>
   return %0 : tensor<13x21x3xf8E5M2>
 }
@@ -1021,7 +1021,7 @@ func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<1
 func.func @test_conv2d_rank0_zp(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<i8>}> : () -> tensor<i8>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<i8>, tensor<1xi8>) -> tensor<1x27x27x16xi32>
   return %0 : tensor<1x27x27x16xi32>
@@ -1130,7 +1130,7 @@ func.func @test_sub_with_unequal_result_ranks(%arg0: tensor<1x21x3xf32>, %arg1:
 // CHECK-LABEL: test_mul_non_scalar_shift_2d
 func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<1x1xi8>}> : () -> tensor<1x1xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<1x1xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1139,7 +1139,7 @@ func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tenso
 // CHECK-LABEL: test_mul_non_scalar_shift_1d
 func.func @test_mul_non_scalar_shift_1d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<2xi8>}> : () -> tensor<2xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<2xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1500,7 +1500,7 @@ func.func @test_rescale_invalid_multiplier_type(%arg0: tensor<13x21x3xi32>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi48>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1512,7 +1512,7 @@ func.func @test_rescale_invalid_shift_type(%arg0: tensor<13x21x3xi32>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit signless integer values, but got 'tensor<1xi16>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tosa-conformant tensor of 8-bit signless integer values, but got 'tensor<1xi16>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi32>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1621,7 +1621,7 @@ func.func @test_rescale_invalid_multiplier_rank(%arg0: tensor<13x21x3xi16>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi8> } : () -> tensor<1xi8>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1x1xi32>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1x1xi32>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = true} : (tensor<13x21x3xi16>, tensor<1x1xi32>, tensor<1xi8>, tensor<1xi16>, tensor<1xi16>) -> tensor<13x21x3xi16>
   return %0 : tensor<13x21x3xi16>
 }
@@ -1633,7 +1633,7 @@ func.func @test_rescale_invalid_shift_rank(%arg0: tensor<13x21x3xi16>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1x1xi8> } : () -> tensor<1x1xi8>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi16>} : () -> tensor<1xi16>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tosa-conformant tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = true} : (tensor<13x21x3xi16>, tensor<1xi32>, tensor<1x1xi8>, tensor<1xi16>, tensor<1xi16>) -> tensor<13x21x3xi16>
   return %0 : tensor<13x21x3xi16>
 }
@@ -2057,7 +2057,7 @@ func.func @test_maxpool2d_unexpected_output_width(%arg0: tensor<1x32x32x8xf32>)
 // -----
 
 func.func @test_scalar_argmax(%arg0: tensor<i32>) -> tensor<i32> {
-  // expected-error at +1 {{'tosa.argmax' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i32>'}}
+  // expected-error at +1 {{'tosa.argmax' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i32>'}}
   %0 = tosa.argmax %arg0 {axis = 0 : i32} : (tensor<i32>) -> tensor<i32>
   return %0 : tensor<i32>
 }
@@ -2065,7 +2065,7 @@ func.func @test_scalar_argmax(%arg0: tensor<i32>) -> tensor<i32> {
 // -----
 
 func.func @test_scalar_reduce_all(%arg0: tensor<i1>) -> tensor<i1> {
-  // expected-error at +1 {{'tosa.reduce_all' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i1>'}}
+  // expected-error at +1 {{'tosa.reduce_all' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i1>'}}
   %0 = tosa.reduce_all %arg0 {axis = 0 : i32} : (tensor<i1>) -> tensor<i1>
   return %0 : tensor<i1>
 }
@@ -2073,7 +2073,7 @@ func.func @test_scalar_reduce_all(%arg0: tensor<i1>) -> tensor<i1> {
 // -----
 
 func.func @test_scalar_inputs_concat(%arg0: tensor<f32>, %arg1: tensor<f32>) -> tensor<2xf32> {
-  // expected-error at +1 {{'tosa.concat' op operand #0 must be variadic of tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.concat' op operand #0 must be variadic of tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<f32>, tensor<f32>) -> tensor<2xf32>
   return %0 : tensor<2xf32>
 }
@@ -2083,7 +2083,7 @@ func.func @test_scalar_inputs_concat(%arg0: tensor<f32>, %arg1: tensor<f32>) ->
 func.func @test_scalar_pad(%arg0: tensor<f32>) -> tensor<f32> {
   %0 = "tosa.const"() {values = dense<3.14> : tensor<1xf32>} : () -> tensor<1xf32>
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
-  // expected-error at +1 {{'tosa.pad' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %1 = tosa.pad %arg0, %padding, %0 : (tensor<f32>, !tosa.shape<6>, tensor<1xf32>) -> tensor<f32>
   return %1 : tensor<f32>
 }
@@ -2091,7 +2091,7 @@ func.func @test_scalar_pad(%arg0: tensor<f32>) -> tensor<f32> {
 // -----
 
 func.func @test_scalar_reverse(%arg0: tensor<f32>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.reverse' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.reverse' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.reverse %arg0 {axis = 0: i32} : (tensor<f32>) -> tensor<f32>
   return %arg0 : tensor<f32>
 }
@@ -2100,7 +2100,7 @@ func.func @test_scalar_reverse(%arg0: tensor<f32>) -> tensor<f32> {
 
 func.func @test_scalar_tile(%arg0: tensor<f32>) -> tensor<*xf32> {
   %cst = tosa.const_shape { values = dense<[]> : tensor<0xindex> } : () -> !tosa.shape<0>
-  // expected-error at +1 {{'tosa.tile' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.tile' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.tile %arg0, %cst: (tensor<f32>, !tosa.shape<0>) -> tensor<*xf32>
   return %0 : tensor<*xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index c31effeb6a1d2..2fda4a966ec9b 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -120,7 +120,7 @@ func.func @test_large_constant_permutation() {
 // -----
 
 func.func @test_scalar_output_transpose(%arg0: tensor<*xf32>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %1 = tosa.transpose %arg0 {perms = array<i32: 2, 0, 1>} : (tensor<*xf32>) -> tensor<f32>
   return %1 : tensor<f32>
 }
@@ -187,7 +187,7 @@ func.func @test_slice_invalid_size() {
 func.func @test_scalar_slice(%arg0: tensor<f32>) -> tensor<f32> {
   %0 = tosa.const_shape {values = dense<[]> : tensor<0xindex>} : () -> !tosa.shape<0>
   %1 = tosa.const_shape {values = dense<[]> : tensor<0xindex>} : () -> !tosa.shape<0>
-  // expected-error at +1 {{'tosa.slice' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.slice' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %2 = tosa.slice %arg0, %0, %1 : (tensor<f32>, !tosa.shape<0>, !tosa.shape<0>) -> tensor<f32>
   return %2 : tensor<f32>
 }
@@ -1563,7 +1563,7 @@ func.func @cast_from_block_scaled_incompatible_input_output_shape(%arg0: tensor<
 // -----
 
 func.func @cast_from_block_scaled_not_scalar(%arg0: tensor<f4E2M1FN>, %arg1: tensor<f8E8M0FNU>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cast_from_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f4E2M1FN>'}}
+  // expected-error at +1 {{'tosa.cast_from_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1 of micro-scaling format number values, but got 'tensor<f4E2M1FN>'}}
   %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<f4E2M1FN>, tensor<f8E8M0FNU>) -> tensor<f32>
   return %0 : tensor<f32>
 }
@@ -1611,7 +1611,7 @@ func.func @test_cast_to_block_scaled_incompatible_input_output_shape(%arg0: tens
 // -----
 
 func.func @test_cast_to_block_scaled_not_scalar(%arg0: tensor<f32>) -> (tensor<f4E2M1FN>, tensor<f8E8M0FNU>) {
-  // expected-error at +1 {{'tosa.cast_to_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.cast_to_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0:2 = tosa.cast_to_block_scaled %arg0 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<f32>) -> (tensor<f4E2M1FN>, tensor<f8E8M0FNU>)
   return %0#0, %0#1 : tensor<f4E2M1FN>, tensor<f8E8M0FNU>
 }
@@ -1731,7 +1731,7 @@ func.func @test_dim_invalid_axis(%arg0: tensor<1x2x3xi32>) -> !tosa.shape<1> {
 // -----
 
 func.func @test_dim_scalar(%arg0: tensor<i32>) -> !tosa.shape<1> {
-  // expected-error at +1 {{'tosa.dim' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i32>'}}
+  // expected-error at +1 {{'tosa.dim' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i32>'}}
   %0 = tosa.dim %arg0 {axis = 4 : i32} : (tensor<i32>) -> !tosa.shape<1>
   return %0 : !tosa.shape<1>
 }

>From 9608c88559b4ad29e4debe6e40df922d31c9f2b9 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 9 Jul 2026 23:18:38 +0100
Subject: [PATCH 2/3] [mlir][ods] Fix string interpolation at end of
 description

Updates the error streaming string logic to handle the case
where string interpolation used at the end of the description.
Previously this could generate malformed code that would not
compile e.g.:
```
"' failed to satisfy constraint: another attribute " << reformat(attr)";
```
With this change the above example would now generate:
```
"' failed to satisfy constraint: another attribute " << reformat(attr) << "";
```

Change-Id: Iea9bd52064b5e19a9a258f955b3b70eb01f26415
---
 mlir/lib/TableGen/CodeGenHelpers.cpp       | 14 ++++++--------
 mlir/test/mlir-tblgen/constraint-unique.td |  4 ++--
 2 files changed, 8 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/TableGen/CodeGenHelpers.cpp b/mlir/lib/TableGen/CodeGenHelpers.cpp
index c8c419669f3d4..19439eddcb333 100644
--- a/mlir/lib/TableGen/CodeGenHelpers.cpp
+++ b/mlir/lib/TableGen/CodeGenHelpers.cpp
@@ -157,18 +157,16 @@ std::string mlir::tblgen::buildErrorStreamingString(
 
     os << " << " << tgfmt(var, &ctx);
 
-    if (rest.empty())
-      break;
-
-    split = rest.split("{{");
-    if (split.second.empty() &&
-        errorStreamType == ErrorStreamType::InsideOpError) {
+    if (errorStreamType == ErrorStreamType::InsideOpError) {
       // To enable having part of string post, this adds a parenthesis before
       // the last string segment to match the existing one.
-      os << " << (\"" << split.first;
+      os << " << (\"";
     } else {
-      os << " << \"" << split.first;
+      os << " << \"";
     }
+
+    split = rest.split("{{");
+    os << split.first;
     msg = split.second;
   }
 
diff --git a/mlir/test/mlir-tblgen/constraint-unique.td b/mlir/test/mlir-tblgen/constraint-unique.td
index 5fdaf368c323a..55d6a56e3fc80 100644
--- a/mlir/test/mlir-tblgen/constraint-unique.td
+++ b/mlir/test/mlir-tblgen/constraint-unique.td
@@ -17,7 +17,7 @@ def OtherType : Type<ATypePred, "another type">;
 
 def AnAttrPred : CPred<"attrPred($_self, $_op)">;
 def AnAttr : Attr<AnAttrPred, "an attribute (got {{reformat($_self)}})">;
-def OtherAttr : Attr<AnAttrPred, "another attribute">;
+def OtherAttr : Attr<AnAttrPred, "another attribute {{reformat($_self)}}">;
 
 def ASuccessorPred : CPred<"successorPred($_self, $_op)">;
 def ASuccessor : Successor<ASuccessorPred, "a successor">;
@@ -81,7 +81,7 @@ def OpC : NS_Op<"op_c"> {
 // CHECK:    static ::llvm::LogicalResult [[$O_ATTR_CONSTRAINT:__mlir_ods_local_attr_constraint.*]](
 // CHECK:      if (attr && !((attrPred(attr, *op))))
 // CHECK-NEXT:   return emitError() << "attribute '" << attrName
-// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute";
+// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute " << reformat(attr) << "";
 
 /// Test that a successor contraint was generated.
 // CHECK:    static ::llvm::LogicalResult [[$A_SUCCESSOR_CONSTRAINT:__mlir_ods_local_successor_constraint.*]](

>From 8990668f2ccc5f2e48b9bc84a0f7bc225aac12a5 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 15 Jul 2026 11:31:41 +0100
Subject: [PATCH 3/3] [mlir][tosa] Add more informative error messages to block
 scaled types

This commit improves block scaled tensor type verification to provide
a specific reason about why type verification failed. Previously the
error message was a very generic
"must be tosa-conformant tensor of number values".

This commit resores previous functionality that was reverted by
https://github.com/llvm/llvm-project/pull/207995, but uses the ODS
string interpolation mechanism instead to prevent incompatibility
with PDLL.

Change-Id: I07f74efe8da9b3bdbae46fd14704d47ab96ec174
---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h   | 13 ++++-
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 22 +++++--
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          | 58 ++++++++++++++++---
 mlir/test/Dialect/Tosa/verifier.mlir          | 10 ++--
 4 files changed, 81 insertions(+), 22 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index b1404d9c700ae..2c4d3b7446544 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -25,6 +25,8 @@
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Interfaces/VectorInterfaces.h"
 
+#include <string>
+
 //===----------------------------------------------------------------------===//
 // TOSA dialect and structs includes.
 //===----------------------------------------------------------------------===//
@@ -141,9 +143,14 @@ Type getStorageElementTypeOrSelf(Type type);
 // Returns the storage element type for a given value
 Type getStorageElementTypeOrSelf(Value value);
 
-// Verify a block scaled tensor type is valid
-LogicalResult verifyBlockScaledTensorType(mlir::Type type,
-                                          bool allowScaleValues);
+// Verify that a given type is a valid block scaled tensor type
+LogicalResult verifyBlockScaledTensorType(
+    mlir::Type type,
+    llvm::function_ref<mlir::InFlightDiagnostic()> emitError = nullptr,
+    bool allowScaleValues = false);
+
+// Collect error messages for a given type
+std::string getTosaTensorTypeErrorMessage(mlir::Type type);
 
 } // namespace tosa
 } // namespace mlir
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 2ab135befaa96..0ab0da14e2ba2 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -172,7 +172,19 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::"
+            "verifyBlockScaledTensorType($_self))">;
+
+// Analogous to ShapedContainerType, but with additional 'summary' information.
+// This is used to provide more informative error messages when a type predicate
+// is false.
+class TosaShapedContainerType<list<Type> allowedTypes, Pred containerPred, string descr, string cppType = "::mlir::Type"> :
+    Type<And<[containerPred,
+              Concat<"[](::mlir::Type elementType) { return ",
+                SubstLeaves<"$_self", "elementType",
+                AnyTypeOf<allowedTypes>.predicate>,
+                "; }(::llvm::cast<::mlir::ShapedType>($_self).getElementType())">]>,
+         descr # " of " # AnyTypeOf<allowedTypes>.summary # " values" # "{{::mlir::tosa::getTosaTensorTypeErrorMessage($_self)}}", cppType>;
 
 // We include unranked tensors as a supported type for all possible tosa
 // Tensors as unranked does not guarantee invalid. If unranked tensors exist
@@ -181,13 +193,13 @@ def IsValidBlockScaledTensorType
 class TosaTensorOf<list<Type> allowedTypes,
                    list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes,
-               [Or<[
+    : TosaShapedContainerType<allowedTypes,
+               And<[Or<[
                     IsUnrankedTensorTypePred, 
                     And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
                 ]>,
-                IsValidBlockScaledTensorType],
-               summary>;
+                IsValidBlockScaledTensorType]>,
+               summary, "::mlir::TensorType">;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaTensorOf<allowedTypes,
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index bf9d98ffe35a0..215366ade3aba 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -741,22 +741,31 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
-                                                      bool allowScaleValues) {
+LogicalResult mlir::tosa::verifyBlockScaledTensorType(
+    mlir::Type type, llvm::function_ref<mlir::InFlightDiagnostic()> emitError,
+    bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
-  if (!allowScaleValues && elemType.hasScaleValues())
+  if (!allowScaleValues && elemType.hasScaleValues()) {
+    if (emitError)
+      emitError()
+          << "block scaled tensor type with scale values is not allowed";
     return failure();
+  }
 
   if (!tensorType.hasRank())
     return success();
 
-  if (tensorType.getRank() == 0)
+  if (tensorType.getRank() == 0) {
+    if (emitError)
+      emitError() << "block scaled tensor type must have rank greater than "
+                     "zero";
     return failure();
+  }
 
   const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
@@ -765,19 +774,47 @@ LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
   if (allowScaleValues && elemType.hasScaleValues() &&
       tensorType.hasStaticShape()) {
     const size_t numBlocks = tensorType.getNumElements() / blockSize;
-    if (elemType.getScaleValues().size() != numBlocks)
+    if (elemType.getScaleValues().size() != numBlocks) {
+      if (emitError)
+        emitError() << "block scaled tensor type with scale values must have "
+                       "scale values for each block, expected "
+                    << numBlocks << ", got "
+                    << elemType.getScaleValues().size();
       return failure();
+    }
   }
 
   const int64_t blockedDimension = tensorShape.back();
   if (ShapedType::isDynamic(blockedDimension))
     return success();
-  if (blockedDimension % blockSize != 0)
+
+  if (blockedDimension % blockSize != 0) {
+    if (emitError)
+      emitError() << "last dimension of block scaled tensor type ("
+                  << blockedDimension << ") must be divisible by block size ("
+                  << blockSize << ")";
+
     return failure();
+  }
 
   return success();
 }
 
+std::string mlir::tosa::getTosaTensorTypeErrorMessage(mlir::Type type) {
+  MLIRContext *ctx = type.getContext();
+  std::string message;
+  ScopedDiagnosticHandler handler(
+      ctx, [&](Diagnostic &diag) { message = diag.str(); });
+
+  if (failed(verifyBlockScaledTensorType(
+          type, [ctx] { return emitError(UnknownLoc::get(ctx)); })) &&
+      !message.empty()) {
+    return ": " + message;
+  }
+
+  return "";
+}
+
 static ParseResult parseScaleValues(AsmParser &parser,
                                     SmallVector<Attribute> &scaleValues,
                                     Type scaleType) {
@@ -959,9 +996,12 @@ LogicalResult tosa::ConstOp::verify() {
       return op.emitOpError(
           "attribute block scaled type must have scale values");
 
-    if (failed(verifyBlockScaledTensorType(attrType, true)))
-      return op.emitOpError("block scaled attribute type is not valid, got ")
-             << attrType;
+    const auto emitAttributeError = [&op]() {
+      return op.emitOpError("attribute block scaled type is invalid ");
+    };
+
+    if (failed(verifyBlockScaledTensorType(attrType, emitAttributeError, true)))
+      return failure();
 
     const BlockScaledType resultBlockScaledType =
         llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 2fda4a966ec9b..30eadf18dc620 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1675,7 +1675,7 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: last dimension of block scaled tensor type (31) must be divisible by block size (32), but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
 }
@@ -1683,7 +1683,7 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: block scaled tensor type must have rank greater than zero, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>
 }
@@ -2316,7 +2316,7 @@ func.func @test_block_scaled_const_scale_value_non_float_explicit_type() -> tens
 !mxint8_scale = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0, 4.0}>
 
 func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid block scaled tensor type with scale values must have scale values for each block, expected 2, got 3}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!mxint8_scale> : 0 : i8>}> : () -> tensor<2x32x!mxint8>
   return %0 : tensor<2x32x!mxint8>
 }
@@ -2324,7 +2324,7 @@ func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8>
 // -----
 
 func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid block scaled tensor type with scale values must have scale values for each block, expected 4, got 2}}
   %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0}>> : 0 : i8>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
   return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
@@ -2332,7 +2332,7 @@ func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor
 // -----
 
 func.func @test_block_scaled_const_cast_scale_values_propagate() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> {
-  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values: block scaled tensor type with scale values is not allowed, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
   return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
 }


From llvmlistbot at llvm.org  Wed Jul 15 04:47:53 2026
From: llvmlistbot at llvm.org (Udaya Ranga)
Date: Wed, 15 Jul 2026 04:47:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a577369.824e11a6.1389c5.e6d1@mx.google.com>


https://github.com/udaya-ranga approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Wed Jul 15 04:48:53 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 04:48:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
Message-ID: <llvm.org/llvm/llvm-project/pull/209737@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/209737

This commit refactors the defined TOSA types to combine unranked and ranked tensor types into a single type `TosaTensorOf`. This helps simplify the type definitions and allows all tensor types to support both unranked and ranked tensors.

>From 82a25847092d71d6ae16f9a2ac456b2d3b8e7111 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 9 Jul 2026 01:00:27 +0100
Subject: [PATCH] [mlir][tosa] Combine unranked/ranked tensor types into single
 type

This commit refactors the defined TOSA types to combine unranked and
ranked tensor types into a single type `TosaTensorOf`. This helps
simplify the type definitions and allows all tensor types to support
both unranked and ranked tensors.

Change-Id: I9866f2e26db44401cbb0e99ce1dbaad131ab574e
---
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 108 +++++++-----------
 .../TosaToLinalg/tosa-to-linalg-pipeline.mlir |   2 +-
 mlir/test/Dialect/Tosa/invalid.mlir           |  40 +++----
 mlir/test/Dialect/Tosa/verifier.mlir          |  10 +-
 4 files changed, 66 insertions(+), 94 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 1518024ecaa19..2ab135befaa96 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>],
-      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tensor">;
+      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tosa-conformant tensor">;
 
 class TosaScalarTensorOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>, AllDimensionsAreSizeOne],
       "tosa-conformant scalar tensor">;
 
@@ -217,9 +212,6 @@ def Tosa_FloatTensor : TosaTensorOf<[AnyFloat]>;
 // Either ranked or unranked tensor of TOSA supported element types.
 def Tosa_Tensor : TosaTensorOf<[Tosa_AnyNumber]>;
 
-// Must be ranked but no further constraints
-def Tosa_RankedTensor : TosaRankedTensorOf<[Tosa_AnyNumber]>;
-
 // Any tensor element type allowed in Tosa ops.
 def Tosa_ElementType : Type<Or<[Tosa_Int.predicate, Tosa_QuantizedInt.predicate,
                                 AnyFloat.predicate]>, "tosa.dtype">;
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
+def Tosa_1DInt16Or32Tensor : TosaTensorRankOf<[Tosa_Int16Or32], [1]>;
 
-// Ranked tensors up to given rank.
-def Tosa_Tensor1Dto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>]>;
-def Tosa_Tensor1Dto6D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>]>;
+// Tensors up to a given rank. Unranked tensors are accepted.
+def Tosa_Tensor1Dto4D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>;
+def Tosa_Tensor1Dto6D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>;
 
-def Tosa_TensorUpto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>]>;
+def Tosa_TensorUpto4D : TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>;
 
-def Tosa_IndexTensor1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>]>;
-def Tosa_IndexTensor2D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>]>;
+def Tosa_IndexTensor1D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>;
+def Tosa_IndexTensor2D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>;
 
-def Tosa_TensorAtLeast1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
+def Tosa_TensorAtLeast1D
+    : TosaTensorOf<[Tosa_AnyNumber], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 def Tosa_MXFPDataTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [3]>;
 def Tosa_MXFPScaleTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [3]>;
 def Tosa_MXFPDataTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [4]>;
 def Tosa_MXFPScaleTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [4]>;
 def Tosa_MXFPDataTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPValue], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 def Tosa_MXFPScaleTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPScale], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
index 67b6aa63f2293..373b5b71cb68b 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
@@ -14,7 +14,7 @@ func.func @tensor_with_unknown_rank(%arg0: tensor<*xi32>) -> tensor<*xi32> {
 
 // check that tosa verify kick in
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..0e66cfa02b67f 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -317,7 +317,7 @@ func.func @test_concat_input_output_rank_mismatch(%arg0: tensor<2x2xf32>, %arg1:
 func.func @test_pad_invalid_padConst_rank(%arg0: tensor<13x21xf32>) {
   %0 = tosa.const_shape {values = dense<1> : tensor<4xindex>} : () -> !tosa.shape<4>
   %1 = "tosa.const"() {values = dense<3.14> : tensor<2xf32>} : () -> tensor<2xf32>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
   %2 = tosa.pad %arg0, %0, %1 : (tensor<13x21xf32>, !tosa.shape<4>, tensor<2xf32>) -> tensor<13x21xf32>
   return
 }
@@ -543,7 +543,7 @@ func.func @test_const_attribute_type_mismatch() -> tensor<100x100xf32> {
 func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x29x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x29x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -554,7 +554,7 @@ func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1:
 func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x?x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x?x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x?x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -564,7 +564,7 @@ func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<
 // -----
 
 func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x7x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x7x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x7x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -573,7 +573,7 @@ func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %ar
 // -----
 
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -673,7 +673,7 @@ func.func @test_tile_io_rank_mismatch() {
 
 // CHECK-LABEL: test_table_rank0_table
 func.func @test_table_rank0_table(%arg0: tensor<64xi16>, %arg1: tensor<i16>) {
-  // expected-error at +1 {{'tosa.table' op operand #1 must be 1-d tosa-conformant tensor, but got 'tensor<i16>'}}
+  // expected-error at +1 {{'tosa.table' op operand #1 must be 1D tosa-conformant tensor of number values, but got 'tensor<i16>'}}
   %0 = tosa.table %arg0, %arg1 : (tensor<64xi16>, tensor<i16>) -> tensor<64xi16>
   return
 }
@@ -1011,7 +1011,7 @@ func.func @test_non_tosa_ops() {
 func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<13x21x3xf8E5M2> {
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   %cst = "tosa.const"() { values = dense<-0.0> : tensor<f8E4M3FN> } : () -> tensor<f8E4M3FN>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
   %0 = tosa.pad %arg0, %padding, %cst : (tensor<13x21x3xf8E4M3FN>, !tosa.shape<6>, tensor<f8E4M3FN>) -> tensor<13x21x3xf8E5M2>
   return %0 : tensor<13x21x3xf8E5M2>
 }
@@ -1021,7 +1021,7 @@ func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<1
 func.func @test_conv2d_rank0_zp(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<i8>}> : () -> tensor<i8>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<i8>, tensor<1xi8>) -> tensor<1x27x27x16xi32>
   return %0 : tensor<1x27x27x16xi32>
@@ -1130,7 +1130,7 @@ func.func @test_sub_with_unequal_result_ranks(%arg0: tensor<1x21x3xf32>, %arg1:
 // CHECK-LABEL: test_mul_non_scalar_shift_2d
 func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<1x1xi8>}> : () -> tensor<1x1xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<1x1xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1139,7 +1139,7 @@ func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tenso
 // CHECK-LABEL: test_mul_non_scalar_shift_1d
 func.func @test_mul_non_scalar_shift_1d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<2xi8>}> : () -> tensor<2xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<2xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1500,7 +1500,7 @@ func.func @test_rescale_invalid_multiplier_type(%arg0: tensor<13x21x3xi32>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi48>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1512,7 +1512,7 @@ func.func @test_rescale_invalid_shift_type(%arg0: tensor<13x21x3xi32>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit signless integer values, but got 'tensor<1xi16>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tosa-conformant tensor of 8-bit signless integer values, but got 'tensor<1xi16>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi32>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1621,7 +1621,7 @@ func.func @test_rescale_invalid_multiplier_rank(%arg0: tensor<13x21x3xi16>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi8> } : () -> tensor<1xi8>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1x1xi32>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1x1xi32>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = true} : (tensor<13x21x3xi16>, tensor<1x1xi32>, tensor<1xi8>, tensor<1xi16>, tensor<1xi16>) -> tensor<13x21x3xi16>
   return %0 : tensor<13x21x3xi16>
 }
@@ -1633,7 +1633,7 @@ func.func @test_rescale_invalid_shift_rank(%arg0: tensor<13x21x3xi16>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1x1xi8> } : () -> tensor<1x1xi8>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi16>} : () -> tensor<1xi16>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tosa-conformant tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = true} : (tensor<13x21x3xi16>, tensor<1xi32>, tensor<1x1xi8>, tensor<1xi16>, tensor<1xi16>) -> tensor<13x21x3xi16>
   return %0 : tensor<13x21x3xi16>
 }
@@ -2057,7 +2057,7 @@ func.func @test_maxpool2d_unexpected_output_width(%arg0: tensor<1x32x32x8xf32>)
 // -----
 
 func.func @test_scalar_argmax(%arg0: tensor<i32>) -> tensor<i32> {
-  // expected-error at +1 {{'tosa.argmax' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i32>'}}
+  // expected-error at +1 {{'tosa.argmax' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i32>'}}
   %0 = tosa.argmax %arg0 {axis = 0 : i32} : (tensor<i32>) -> tensor<i32>
   return %0 : tensor<i32>
 }
@@ -2065,7 +2065,7 @@ func.func @test_scalar_argmax(%arg0: tensor<i32>) -> tensor<i32> {
 // -----
 
 func.func @test_scalar_reduce_all(%arg0: tensor<i1>) -> tensor<i1> {
-  // expected-error at +1 {{'tosa.reduce_all' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i1>'}}
+  // expected-error at +1 {{'tosa.reduce_all' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i1>'}}
   %0 = tosa.reduce_all %arg0 {axis = 0 : i32} : (tensor<i1>) -> tensor<i1>
   return %0 : tensor<i1>
 }
@@ -2073,7 +2073,7 @@ func.func @test_scalar_reduce_all(%arg0: tensor<i1>) -> tensor<i1> {
 // -----
 
 func.func @test_scalar_inputs_concat(%arg0: tensor<f32>, %arg1: tensor<f32>) -> tensor<2xf32> {
-  // expected-error at +1 {{'tosa.concat' op operand #0 must be variadic of tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.concat' op operand #0 must be variadic of tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<f32>, tensor<f32>) -> tensor<2xf32>
   return %0 : tensor<2xf32>
 }
@@ -2083,7 +2083,7 @@ func.func @test_scalar_inputs_concat(%arg0: tensor<f32>, %arg1: tensor<f32>) ->
 func.func @test_scalar_pad(%arg0: tensor<f32>) -> tensor<f32> {
   %0 = "tosa.const"() {values = dense<3.14> : tensor<1xf32>} : () -> tensor<1xf32>
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
-  // expected-error at +1 {{'tosa.pad' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %1 = tosa.pad %arg0, %padding, %0 : (tensor<f32>, !tosa.shape<6>, tensor<1xf32>) -> tensor<f32>
   return %1 : tensor<f32>
 }
@@ -2091,7 +2091,7 @@ func.func @test_scalar_pad(%arg0: tensor<f32>) -> tensor<f32> {
 // -----
 
 func.func @test_scalar_reverse(%arg0: tensor<f32>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.reverse' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.reverse' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.reverse %arg0 {axis = 0: i32} : (tensor<f32>) -> tensor<f32>
   return %arg0 : tensor<f32>
 }
@@ -2100,7 +2100,7 @@ func.func @test_scalar_reverse(%arg0: tensor<f32>) -> tensor<f32> {
 
 func.func @test_scalar_tile(%arg0: tensor<f32>) -> tensor<*xf32> {
   %cst = tosa.const_shape { values = dense<[]> : tensor<0xindex> } : () -> !tosa.shape<0>
-  // expected-error at +1 {{'tosa.tile' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.tile' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0 = tosa.tile %arg0, %cst: (tensor<f32>, !tosa.shape<0>) -> tensor<*xf32>
   return %0 : tensor<*xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index c31effeb6a1d2..2fda4a966ec9b 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -120,7 +120,7 @@ func.func @test_large_constant_permutation() {
 // -----
 
 func.func @test_scalar_output_transpose(%arg0: tensor<*xf32>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.transpose' op result #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %1 = tosa.transpose %arg0 {perms = array<i32: 2, 0, 1>} : (tensor<*xf32>) -> tensor<f32>
   return %1 : tensor<f32>
 }
@@ -187,7 +187,7 @@ func.func @test_slice_invalid_size() {
 func.func @test_scalar_slice(%arg0: tensor<f32>) -> tensor<f32> {
   %0 = tosa.const_shape {values = dense<[]> : tensor<0xindex>} : () -> !tosa.shape<0>
   %1 = tosa.const_shape {values = dense<[]> : tensor<0xindex>} : () -> !tosa.shape<0>
-  // expected-error at +1 {{'tosa.slice' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.slice' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %2 = tosa.slice %arg0, %0, %1 : (tensor<f32>, !tosa.shape<0>, !tosa.shape<0>) -> tensor<f32>
   return %2 : tensor<f32>
 }
@@ -1563,7 +1563,7 @@ func.func @cast_from_block_scaled_incompatible_input_output_shape(%arg0: tensor<
 // -----
 
 func.func @cast_from_block_scaled_not_scalar(%arg0: tensor<f4E2M1FN>, %arg1: tensor<f8E8M0FNU>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cast_from_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f4E2M1FN>'}}
+  // expected-error at +1 {{'tosa.cast_from_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1 of micro-scaling format number values, but got 'tensor<f4E2M1FN>'}}
   %0 = tosa.cast_from_block_scaled %arg0, %arg1 {block_size = #tosa.block_size<BLOCK_SIZE_32> : i32} : (tensor<f4E2M1FN>, tensor<f8E8M0FNU>) -> tensor<f32>
   return %0 : tensor<f32>
 }
@@ -1611,7 +1611,7 @@ func.func @test_cast_to_block_scaled_incompatible_input_output_shape(%arg0: tens
 // -----
 
 func.func @test_cast_to_block_scaled_not_scalar(%arg0: tensor<f32>) -> (tensor<f4E2M1FN>, tensor<f8E8M0FNU>) {
-  // expected-error at +1 {{'tosa.cast_to_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<f32>'}}
+  // expected-error at +1 {{'tosa.cast_to_block_scaled' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<f32>'}}
   %0:2 = tosa.cast_to_block_scaled %arg0 {block_size = #tosa.block_size<BLOCK_SIZE_32>} : (tensor<f32>) -> (tensor<f4E2M1FN>, tensor<f8E8M0FNU>)
   return %0#0, %0#1 : tensor<f4E2M1FN>, tensor<f8E8M0FNU>
 }
@@ -1731,7 +1731,7 @@ func.func @test_dim_invalid_axis(%arg0: tensor<1x2x3xi32>) -> !tosa.shape<1> {
 // -----
 
 func.func @test_dim_scalar(%arg0: tensor<i32>) -> !tosa.shape<1> {
-  // expected-error at +1 {{'tosa.dim' op operand #0 must be tosa-conformant tensor of at least rank 1, but got 'tensor<i32>'}}
+  // expected-error at +1 {{'tosa.dim' op operand #0 must be tosa-conformant tensor of at least rank 1 of number values, but got 'tensor<i32>'}}
   %0 = tosa.dim %arg0 {axis = 4 : i32} : (tensor<i32>) -> !tosa.shape<1>
   return %0 : !tosa.shape<1>
 }


From llvmlistbot at llvm.org  Wed Jul 15 04:49:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 04:49:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5773d1.1728eaf3.2532c3.03e1@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

This commit refactors the defined TOSA types to combine unranked and ranked tensor types into a single type `TosaTensorOf`. This helps simplify the type definitions and allows all tensor types to support both unranked and ranked tensors.

---

Patch is 29.52 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209737.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td (+40-68) 
- (modified) mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid.mlir (+20-20) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 1518024ecaa19..2ab135befaa96 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>],
-      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tensor">;
+      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tosa-conformant tensor">;
 
 class TosaScalarTensorOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>, AllDimensionsAreSizeOne],
       "tosa-conformant scalar tensor">;
 
@@ -217,9 +212,6 @@ def Tosa_FloatTensor : TosaTensorOf<[AnyFloat]>;
 // Either ranked or unranked tensor of TOSA supported element types.
 def Tosa_Tensor : TosaTensorOf<[Tosa_AnyNumber]>;
 
-// Must be ranked but no further constraints
-def Tosa_RankedTensor : TosaRankedTensorOf<[Tosa_AnyNumber]>;
-
 // Any tensor element type allowed in Tosa ops.
 def Tosa_ElementType : Type<Or<[Tosa_Int.predicate, Tosa_QuantizedInt.predicate,
                                 AnyFloat.predicate]>, "tosa.dtype">;
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
+def Tosa_1DInt16Or32Tensor : TosaTensorRankOf<[Tosa_Int16Or32], [1]>;
 
-// Ranked tensors up to given rank.
-def Tosa_Tensor1Dto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>]>;
-def Tosa_Tensor1Dto6D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>]>;
+// Tensors up to a given rank. Unranked tensors are accepted.
+def Tosa_Tensor1Dto4D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>;
+def Tosa_Tensor1Dto6D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>;
 
-def Tosa_TensorUpto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>]>;
+def Tosa_TensorUpto4D : TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>;
 
-def Tosa_IndexTensor1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>]>;
-def Tosa_IndexTensor2D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>]>;
+def Tosa_IndexTensor1D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>;
+def Tosa_IndexTensor2D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>;
 
-def Tosa_TensorAtLeast1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
+def Tosa_TensorAtLeast1D
+    : TosaTensorOf<[Tosa_AnyNumber], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 def Tosa_MXFPDataTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [3]>;
 def Tosa_MXFPScaleTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [3]>;
 def Tosa_MXFPDataTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [4]>;
 def Tosa_MXFPScaleTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [4]>;
 def Tosa_MXFPDataTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPValue], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 def Tosa_MXFPScaleTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPScale], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
index 67b6aa63f2293..373b5b71cb68b 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
@@ -14,7 +14,7 @@ func.func @tensor_with_unknown_rank(%arg0: tensor<*xi32>) -> tensor<*xi32> {
 
 // check that tosa verify kick in
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..0e66cfa02b67f 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -317,7 +317,7 @@ func.func @test_concat_input_output_rank_mismatch(%arg0: tensor<2x2xf32>, %arg1:
 func.func @test_pad_invalid_padConst_rank(%arg0: tensor<13x21xf32>) {
   %0 = tosa.const_shape {values = dense<1> : tensor<4xindex>} : () -> !tosa.shape<4>
   %1 = "tosa.const"() {values = dense<3.14> : tensor<2xf32>} : () -> tensor<2xf32>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
   %2 = tosa.pad %arg0, %0, %1 : (tensor<13x21xf32>, !tosa.shape<4>, tensor<2xf32>) -> tensor<13x21xf32>
   return
 }
@@ -543,7 +543,7 @@ func.func @test_const_attribute_type_mismatch() -> tensor<100x100xf32> {
 func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x29x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x29x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -554,7 +554,7 @@ func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1:
 func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x?x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x?x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x?x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -564,7 +564,7 @@ func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<
 // -----
 
 func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x7x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x7x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x7x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -573,7 +573,7 @@ func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %ar
 // -----
 
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -673,7 +673,7 @@ func.func @test_tile_io_rank_mismatch() {
 
 // CHECK-LABEL: test_table_rank0_table
 func.func @test_table_rank0_table(%arg0: tensor<64xi16>, %arg1: tensor<i16>) {
-  // expected-error at +1 {{'tosa.table' op operand #1 must be 1-d tosa-conformant tensor, but got 'tensor<i16>'}}
+  // expected-error at +1 {{'tosa.table' op operand #1 must be 1D tosa-conformant tensor of number values, but got 'tensor<i16>'}}
   %0 = tosa.table %arg0, %arg1 : (tensor<64xi16>, tensor<i16>) -> tensor<64xi16>
   return
 }
@@ -1011,7 +1011,7 @@ func.func @test_non_tosa_ops() {
 func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<13x21x3xf8E5M2> {
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   %cst = "tosa.const"() { values = dense<-0.0> : tensor<f8E4M3FN> } : () -> tensor<f8E4M3FN>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
   %0 = tosa.pad %arg0, %padding, %cst : (tensor<13x21x3xf8E4M3FN>, !tosa.shape<6>, tensor<f8E4M3FN>) -> tensor<13x21x3xf8E5M2>
   return %0 : tensor<13x21x3xf8E5M2>
 }
@@ -1021,7 +1021,7 @@ func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<1
 func.func @test_conv2d_rank0_zp(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<i8>}> : () -> tensor<i8>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<i8>, tensor<1xi8>) -> tensor<1x27x27x16xi32>
   return %0 : tensor<1x27x27x16xi32>
@@ -1130,7 +1130,7 @@ func.func @test_sub_with_unequal_result_ranks(%arg0: tensor<1x21x3xf32>, %arg1:
 // CHECK-LABEL: test_mul_non_scalar_shift_2d
 func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<1x1xi8>}> : () -> tensor<1x1xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<1x1xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1139,7 +1139,7 @@ func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tenso
 // CHECK-LABEL: test_mul_non_scalar_shift_1d
 func.func @test_mul_non_scalar_shift_1d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<2xi8>}> : () -> tensor<2xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<2xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1500,7 +1500,7 @@ func.func @test_rescale_invalid_multiplier_type(%arg0: tensor<13x21x3xi32>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi48>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1512,7 +1512,7 @@ func.func @test_rescale_invalid_shift_type(%arg0: tensor<13x21x3xi32>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit si...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 15 04:49:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 04:49:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5773d2.b1416da1.26c0f8.bc44@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-tosa

@llvm/pr-subscribers-mlir-linalg

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

This commit refactors the defined TOSA types to combine unranked and ranked tensor types into a single type `TosaTensorOf`. This helps simplify the type definitions and allows all tensor types to support both unranked and ranked tensors.

---

Patch is 29.52 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209737.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td (+40-68) 
- (modified) mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid.mlir (+20-20) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 1518024ecaa19..2ab135befaa96 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>],
-      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tensor">;
+      !interleave(!foreach(rank, ranks, rank # "D"), "/") # " tosa-conformant tensor">;
 
 class TosaScalarTensorOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
       [HasAnyRankOfPred<ranks>, AllDimensionsAreSizeOne],
       "tosa-conformant scalar tensor">;
 
@@ -217,9 +212,6 @@ def Tosa_FloatTensor : TosaTensorOf<[AnyFloat]>;
 // Either ranked or unranked tensor of TOSA supported element types.
 def Tosa_Tensor : TosaTensorOf<[Tosa_AnyNumber]>;
 
-// Must be ranked but no further constraints
-def Tosa_RankedTensor : TosaRankedTensorOf<[Tosa_AnyNumber]>;
-
 // Any tensor element type allowed in Tosa ops.
 def Tosa_ElementType : Type<Or<[Tosa_Int.predicate, Tosa_QuantizedInt.predicate,
                                 AnyFloat.predicate]>, "tosa.dtype">;
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
+def Tosa_1DInt16Or32Tensor : TosaTensorRankOf<[Tosa_Int16Or32], [1]>;
 
-// Ranked tensors up to given rank.
-def Tosa_Tensor1Dto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>]>;
-def Tosa_Tensor1Dto6D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>]>;
+// Tensors up to a given rank. Unranked tensors are accepted.
+def Tosa_Tensor1Dto4D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4]>;
+def Tosa_Tensor1Dto6D : TosaTensorRankOf<[Tosa_AnyNumber], [1,2,3,4,5,6]>;
 
-def Tosa_TensorUpto4D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>]>;
+def Tosa_TensorUpto4D : TosaTensorRankOf<[Tosa_AnyNumber], [0,1,2,3,4]>;
 
-def Tosa_IndexTensor1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>]>;
-def Tosa_IndexTensor2D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>]>;
+def Tosa_IndexTensor1D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [1]>;
+def Tosa_IndexTensor2D : TosaTensorRankOf<[Tosa_Int32, Tosa_Int64], [2]>;
 
-def Tosa_TensorAtLeast1D : AnyTypeOf<[
-  Tosa_UnrankedTensor, TosaRankedTensorOf<[Tosa_AnyNumber], [AtLeastRankOne]>], "tosa-conformant tensor of at least rank 1", "::mlir::TensorType">;
+def Tosa_TensorAtLeast1D
+    : TosaTensorOf<[Tosa_AnyNumber], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 def Tosa_MXFPDataTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [3]>;
 def Tosa_MXFPScaleTensor3D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [3]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [3]>;
 def Tosa_MXFPDataTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaTensorRankOf<[Tosa_MXFPValue], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPValue], [4]>;
 def Tosa_MXFPScaleTensor4D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaTensorRankOf<[Tosa_MXFPScale], [4]>]>;
+    : TosaTensorRankOf<[Tosa_MXFPScale], [4]>;
 def Tosa_MXFPDataTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPValue]>,
-                 TosaRankedTensorOf<[Tosa_MXFPValue], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPValue], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 def Tosa_MXFPScaleTensorAtLeast1D
-    : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_MXFPScale]>,
-                 TosaRankedTensorOf<[Tosa_MXFPScale], [AtLeastRankOne]>],
-                "tosa-conformant tensor of at least rank 1",
-                "::mlir::TensorType">;
+    : TosaTensorOf<[Tosa_MXFPScale], [AtLeastRankOne],
+                   "tosa-conformant tensor of at least rank 1">;
 
 //===----------------------------------------------------------------------===//
 // Generic scalar, vector, or tensor of a particular type.
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
index 67b6aa63f2293..373b5b71cb68b 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg-pipeline.mlir
@@ -14,7 +14,7 @@ func.func @tensor_with_unknown_rank(%arg0: tensor<*xi32>) -> tensor<*xi32> {
 
 // check that tosa verify kick in
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..0e66cfa02b67f 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -317,7 +317,7 @@ func.func @test_concat_input_output_rank_mismatch(%arg0: tensor<2x2xf32>, %arg1:
 func.func @test_pad_invalid_padConst_rank(%arg0: tensor<13x21xf32>) {
   %0 = tosa.const_shape {values = dense<1> : tensor<4xindex>} : () -> !tosa.shape<4>
   %1 = "tosa.const"() {values = dense<3.14> : tensor<2xf32>} : () -> tensor<2xf32>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<2xf32>'}}
   %2 = tosa.pad %arg0, %0, %1 : (tensor<13x21xf32>, !tosa.shape<4>, tensor<2xf32>) -> tensor<13x21xf32>
   return
 }
@@ -543,7 +543,7 @@ func.func @test_const_attribute_type_mismatch() -> tensor<100x100xf32> {
 func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x29x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x29x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -554,7 +554,7 @@ func.func @test_conv2d_static_zero_dim_input(%arg0: tensor<1x29x0x4xf32>, %arg1:
 func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi32>}> : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x?x0x4xf32>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x?x0x4xf32>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x?x0x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xi32>, tensor<1xi32>) -> tensor<1x27x27x16xf32>
   return %0 : tensor<1x27x27x16xf32>
@@ -564,7 +564,7 @@ func.func @test_conv2d_zero_dim_input(%arg0: tensor<1x?x0x4xf32>, %arg1: tensor<
 // -----
 
 func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x7x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x7x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x7x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -573,7 +573,7 @@ func.func @test_avg_pool2d_static_zero_dim_input(%arg0: tensor<1x0x7x9xf32>, %ar
 // -----
 
 func.func @test_avg_pool2d_zero_dim_input(%arg0: tensor<1x0x?x9xf32>, %arg1: tensor<1xf32>, %arg2: tensor<1xf32>) -> tensor<1x7x7x9xf32> {
-  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4-d tosa-conformant tensor, but got 'tensor<1x0x?x9xf32>'}}
+  // expected-error at +1 {{'tosa.avg_pool2d' op operand #0 must be 4D tosa-conformant tensor of number values, but got 'tensor<1x0x?x9xf32>'}}
     %0 = "tosa.avg_pool2d"(%arg0, %arg1, %arg2) {acc_type = f32, kernel = array<i64: 2, 2>, pad = array<i64: 0, 1, 0, 1>, stride = array<i64: 1, 1>}
       : (tensor<1x0x?x9xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x7x7x9xf32>
     return %0 : tensor<1x7x7x9xf32>
@@ -673,7 +673,7 @@ func.func @test_tile_io_rank_mismatch() {
 
 // CHECK-LABEL: test_table_rank0_table
 func.func @test_table_rank0_table(%arg0: tensor<64xi16>, %arg1: tensor<i16>) {
-  // expected-error at +1 {{'tosa.table' op operand #1 must be 1-d tosa-conformant tensor, but got 'tensor<i16>'}}
+  // expected-error at +1 {{'tosa.table' op operand #1 must be 1D tosa-conformant tensor of number values, but got 'tensor<i16>'}}
   %0 = tosa.table %arg0, %arg1 : (tensor<64xi16>, tensor<i16>) -> tensor<64xi16>
   return
 }
@@ -1011,7 +1011,7 @@ func.func @test_non_tosa_ops() {
 func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<13x21x3xf8E5M2> {
   %padding = tosa.const_shape {values = dense<0> : tensor<6xindex>} : () -> !tosa.shape<6>
   %cst = "tosa.const"() { values = dense<-0.0> : tensor<f8E4M3FN> } : () -> tensor<f8E4M3FN>
-  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
+  // expected-error at +1 {{'tosa.pad' op operand #2 must be tosa-conformant scalar tensor of number values, but got 'tensor<f8E4M3FN>'}}
   %0 = tosa.pad %arg0, %padding, %cst : (tensor<13x21x3xf8E4M3FN>, !tosa.shape<6>, tensor<f8E4M3FN>) -> tensor<13x21x3xf8E5M2>
   return %0 : tensor<13x21x3xf8E5M2>
 }
@@ -1021,7 +1021,7 @@ func.func @test_pad_rank0_pad_const(%arg0: tensor<13x21x3xf8E4M3FN>) -> tensor<1
 func.func @test_conv2d_rank0_zp(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi32> {
   %input_zp = "tosa.const"() <{values = dense<0> : tensor<i8>}> : () -> tensor<i8>
   %weight_zp = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant unranked tensor of unsigned integer or signless integer or floating-point values or tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
+  // expected-error at +1 {{'tosa.conv2d' op operand #3 must be tosa-conformant scalar tensor of unsigned integer or signless integer or floating-point values, but got 'tensor<i8>'}}
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
            : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<i8>, tensor<1xi8>) -> tensor<1x27x27x16xi32>
   return %0 : tensor<1x27x27x16xi32>
@@ -1130,7 +1130,7 @@ func.func @test_sub_with_unequal_result_ranks(%arg0: tensor<1x21x3xf32>, %arg1:
 // CHECK-LABEL: test_mul_non_scalar_shift_2d
 func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<1x1xi8>}> : () -> tensor<1x1xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<1x1xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<1x1xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1139,7 +1139,7 @@ func.func @test_mul_non_scalar_shift_2d(%arg0: tensor<13x21x3xf32>, %arg1: tenso
 // CHECK-LABEL: test_mul_non_scalar_shift_1d
 func.func @test_mul_non_scalar_shift_1d(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x1x3xf32>) -> tensor<13x21x3xf32> {
   %shift = "tosa.const"() <{values = dense<0> : tensor<2xi8>}> : () -> tensor<2xi8>
-  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant unranked tensor of 8-bit signless integer values or tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
+  // expected-error at +1 {{'tosa.mul' op operand #2 must be tosa-conformant scalar tensor of 8-bit signless integer values, but got 'tensor<2xi8>'}}
   %0 = tosa.mul %arg0, %arg1, %shift : (tensor<13x21x3xf32>, tensor<13x1x3xf32>, tensor<2xi8>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1500,7 +1500,7 @@ func.func @test_rescale_invalid_multiplier_type(%arg0: tensor<13x21x3xi32>) -> t
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
+  // expected-error at +1 {{'tosa.rescale' op operand #1 must be 1D tosa-conformant tensor of 16-bit signless integer or 32-bit signless integer values, but got 'tensor<1xi48>'}}
   %0 = tosa.rescale %arg0, %multiplier, %shift, %input_zp, %output_zp {rounding_mode = SINGLE_ROUND, per_channel = false, scale32 = true, input_unsigned = false, output_unsigned = false} : (tensor<13x21x3xi32>, tensor<1xi48>, tensor<1xi16>, tensor<1xi32>, tensor<1xi32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -1512,7 +1512,7 @@ func.func @test_rescale_invalid_shift_type(%arg0: tensor<13x21x3xi32>) -> tensor
   %shift = "tosa.const"() {values = dense<30> : tensor<1xi16> } : () -> tensor<1xi16>
   %input_zp = "tosa.const"() {values = dense<1> : tensor<1xi32>} : () -> tensor<1xi32>
   %output_zp = "tosa.const"() {values = dense<0> : tensor<1xi32>} : () -> tensor<1xi32>
-  // expected-error at +1 {{'tosa.rescale' op operand #2 must be 1D tensor of 8-bit si...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 15 04:50:11 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 04:50:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5773f3.4a342cbc.cf320.4bb4@mx.google.com>


lhutton1 wrote:

Please see https://github.com/llvm/llvm-project/pull/209736 for an overview of how this refactor helps error diagnostics.

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 15 04:51:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 04:51:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8c9a6fd - [mlir][tosa] Add support for block
 scaled types in matmul_t (#207851)
Message-ID: <6a577440.c574b9b4.2aa0bf.bd1e@mx.google.com>


Author: Luke Hutton
Date: 2026-07-15T12:51:24+01:00
New Revision: 8c9a6fd876677a69b3a08ce8d868f21730dd6915

URL: https://github.com/llvm/llvm-project/commit/8c9a6fd876677a69b3a08ce8d868f21730dd6915
DIFF: https://github.com/llvm/llvm-project/commit/8c9a6fd876677a69b3a08ce8d868f21730dd6915.diff

LOG: [mlir][tosa] Add support for block scaled types in matmul_t (#207851)

Builds on https://github.com/llvm/llvm-project/pull/203894 to enable
MXFP block scaled fotmats within matmul_t.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
    mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
    mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
    mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
    mlir/test/Dialect/Tosa/invalid_extension.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
    mlir/test/Dialect/Tosa/verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 291097a7f7514..4e23d8dd2fbbb 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -62,11 +62,20 @@ profileComplianceMap = {
         {{fp32T, fp32T, fp32T, fp32T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Profile::pro_int},
-       {{{i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i8T, i8T, i8T, i8T, i32T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Profile::pro_fp},
-       {{{fp16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, fp16T, fp16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp16T, fp16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}}}},
     {"tosa.max_pool2d",
      {{{Profile::pro_int}, {{{i8T, i8T}, SpecificationVersion::V_1_0}}},
       {{Profile::pro_fp},
@@ -696,52 +705,526 @@ extensionComplianceMap = {
        {{{bf16T, bf16T, bf16T, bf16T, fp32T}, SpecificationVersion::V_1_0}}}}},
     {"tosa.matmul_t",
      {{{Extension::int16},
-       {{{i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{i16T, i16T, i16T, i16T, i48T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
-       {{{fp8e4m3T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e4m3T, fp8e4m3T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp8e4m3T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3, Extension::fp8e5m2},
-       {{{fp8e4m3T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp8e4m3T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp8e5m2T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16, Extension::fp8e4m3},
-       {{{fp8e4m3T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e4m3T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e4m3T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp8e4m3T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, bf16T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp8e4m3T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::fp8e5m2},
-       {{{fp8e5m2T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+       {{{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e5m2T, fp8e5m2T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp8e5m2T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16, Extension::fp8e5m2},
-       {{{fp8e5m2T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp8e5m2T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp8e5m2T, fp32T}, SpecificationVersion::V_1_1_DRAFT}},
+       {{{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp8e5m2T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, bf16T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp8e5m2T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
        allOf},
       {{Extension::bf16},
-       {{{fp16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, fp32T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{fp32T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT},
-        {{bf16T, bf16T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}}}},
+       {{{fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, bf16T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, bf16T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, bf16T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.matmul_t_block_scaled",
      {{{Extension::mxfp},
        {{{fp4e2m1T, fp8ue8m0T, fp4e2m1T, fp8ue8m0T, fp32T},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index e3b74f5a07d3f..24de011e90c92 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -475,7 +475,9 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
   list<Availability> availability =
       [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
        Extension<[Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2,
-                  Tosa_EXT_BF16]>,
+                  Tosa_EXT_BF16, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+                  Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+                  Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -494,6 +496,8 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: matmul_t_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled", [NoMemoryEffect]> {
   let summary = "Performs two dimensional matrix multiplications using block scaled tensors.";
@@ -506,6 +510,8 @@ def Tosa_MatmulTBlockScaledOp : Tosa_InferShapedTypeOp<"matmul_t_block_scaled",
 
     This operation is not pure. Undefined behaviour may occur if the accumulated
     result overflows.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index bf9d98ffe35a0..0b6abc2df3a70 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -2238,16 +2238,26 @@ template <typename T>
 static LogicalResult verifyMatMulZeroPointType(T op, Value input, Value zp,
                                                StringRef inputName,
                                                StringRef zpName) {
+  const Type inputElementType = getElementTypeOrSelf(input.getType());
   const Type inputStorageElementType = getStorageElementTypeOrSelf(input);
   const Type zpElementType = getStorageElementTypeOrSelf(zp);
+  Type expectedElementType = inputStorageElementType;
 
-  if (inputStorageElementType != zpElementType)
-    return op.emitOpError("expect input ")
-           << inputName << " and " << zpName
-           << " have the same element type, got " << inputStorageElementType
-           << " and " << zpElementType;
+  if (isa<BlockScaledType>(inputElementType))
+    expectedElementType = Float32Type::get(op.getContext());
 
-  return success();
+  if (expectedElementType == zpElementType)
+    return success();
+
+  InFlightDiagnostic diag = op.emitOpError("expect input ");
+  diag << inputName << " and " << zpName;
+  if (isa<BlockScaledType>(inputElementType))
+    diag << " have compatible element types, got " << inputElementType
+         << " and " << zpElementType;
+  else
+    diag << " have the same element type, got " << inputStorageElementType
+         << " and " << zpElementType;
+  return diag;
 }
 
 LogicalResult MatMulOp::verify() {

diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
index 163850f741166..2bc99d7fab8a4 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaDowngrade1p1To1p0.cpp
@@ -151,6 +151,9 @@ static LogicalResult isMatMulTTypeCompatibleForDowngrade(tosa::MatMulTOp op) {
   if (aElementType != bElementType)
     return failure();
 
+  if (isa<BlockScaledType>(aElementType) || isa<BlockScaledType>(bElementType))
+    return failure();
+
   if ((aElementType.isF16() && outputElementType.isF16()) ||
       (aElementType.isF16() && outputElementType.isF32()) ||
       (aElementType.isF32() && outputElementType.isF32()) ||

diff  --git a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
index 348e4ec0ad44c..7ea64385c2415 100644
--- a/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
+++ b/mlir/lib/Dialect/Tosa/Transforms/TosaProfileCompliance.cpp
@@ -291,6 +291,8 @@ template <>
 LogicalResult ProfileInfoDepot::populateProfileInfo(tosa::MatMulTOp op) {
   addValue(op.getA());
   addValue(op.getB());
+  addValue(op.getAZp());
+  addValue(op.getBZp());
   addValue(op.getOutput());
   return success();
 }

diff  --git a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
index c55b13dc98cc5..b3fe95eefe13f 100644
--- a/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
+++ b/mlir/lib/Dialect/Tosa/Utils/QuantUtils.cpp
@@ -162,10 +162,14 @@ std::pair<Value, Value>
 mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   std::int64_t inputZp, weightZp;
 
-  auto inputEType = getElementTypeOrSelf(input.getType());
-  auto weightEType = getElementTypeOrSelf(weight.getType());
-
-  if (mlir::isa<FloatType>(inputEType) && mlir::isa<FloatType>(weightEType)) {
+  Type inputZpType = getElementTypeOrSelf(input.getType());
+  if (isa<BlockScaledType>(inputZpType))
+    inputZpType = builder.getF32Type();
+  Type weightZpType = getElementTypeOrSelf(weight.getType());
+  if (isa<BlockScaledType>(weightZpType))
+    weightZpType = builder.getF32Type();
+
+  if (mlir::isa<FloatType>(inputZpType) && mlir::isa<FloatType>(weightZpType)) {
     inputZp = 0;
     weightZp = 0;
   } else {
@@ -178,12 +182,12 @@ mlir::tosa::createZPsAsConst(OpBuilder &builder, Value input, Value weight) {
   }
 
   auto maybeInputZpValue =
-      createZeroPointTensor(builder, input.getLoc(), inputEType, inputZp);
+      createZeroPointTensor(builder, input.getLoc(), inputZpType, inputZp);
   if (!maybeInputZpValue.has_value())
     return {};
 
   auto maybeWeightZpValue =
-      createZeroPointTensor(builder, weight.getLoc(), weightEType, weightZp);
+      createZeroPointTensor(builder, weight.getLoc(), weightZpType, weightZp);
   if (!maybeWeightZpValue.has_value())
     return {};
 

diff  --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..c68487169da63 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -332,6 +332,17 @@ func.func @test_cast_fp8_block_scaled(%arg0: tensor<4x32xf8E4M3FN>) -> tensor<4x
 }
 
 // -----
+
+func.func @test_matmul_t_mixed_block_scaled_operands(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.matmul_t' op illegal: requires all of [mx_common, mx_fp6e2m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 func.func @test_fft2d(%arg0: tensor<1x4x8xf32>, %arg1: tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>) {
   // expected-error at +1 {{'tosa.fft2d' op illegal: requires any of [fft] profiles/extensions to be specified in the target environment}}
   %0, %1 = tosa.fft2d %arg0, %arg1 {inverse = false} : (tensor<1x4x8xf32>, tensor<1x4x8xf32>) -> (tensor<1x4x8xf32>, tensor<1x4x8xf32>)

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 4e71d512def2d..74ba92bdb25ab 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -286,6 +286,15 @@ func.func @test_matmul_t_fp8_mixed(%arg0: tensor<2x14x19xf8E4M3FN>, %arg1: tenso
   return %0 : tensor<2x14x28xf16>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs
+func.func @test_matmul_t_with_block_scaled_inputs(%arg0: tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, %arg1: tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<2x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<2x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<2x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<2x14x28xf32>
+  return %0 : tensor<2x14x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_dynamic_dims
 func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<?x28x19xf32>) -> tensor<?x14x28xf32> {
@@ -295,6 +304,15 @@ func.func @test_matmul_t_dynamic_dims(%arg0: tensor<?x14x19xf32>, %arg1: tensor<
   return %0 : tensor<?x14x28xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_dynamic_block_scaled_broadcast
+func.func @test_matmul_t_dynamic_block_scaled_broadcast(%arg0: tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<?x?x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<?x?x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<?x?x28xf32>
+  return %0 : tensor<?x?x28xf32>
+}
+
 // -----
 // CHECK-LABEL: test_matmul_t_static_broadcast
 func.func @test_matmul_t_static_broadcast(%arg0: tensor<4x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<4x14x28xf32> {
@@ -313,6 +331,15 @@ func.func @test_matmul_t_unranked(%arg0: tensor<*xf32>, %arg1: tensor<*xf32>) ->
   return %0 : tensor<*xf32>
 }
 
+// -----
+// CHECK-LABEL: test_matmul_t_unranked_block_scaled
+func.func @test_matmul_t_unranked_block_scaled(%arg0: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<*xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<*x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>)  -> tensor<*xf32>
+  return %0 : tensor<*xf32>
+}
+
 // -----
 // CHECK-LABEL: max_pool2d_f32
 func.func @test_max_pool2d_f32(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {

diff  --git a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
index 378e5eb7b0147..1532ec91d53e8 100644
--- a/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-downgrade-1-1-to-1-0.mlir
@@ -150,3 +150,13 @@ func.func @test_preserve_matmul_t_dynamic_broadcast(%arg0: tensor<4x14x19xf32>,
   %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<4x14x19xf32>, tensor<?x28x19xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x14x28xf32>
   return %0 : tensor<4x14x28xf32>
 }
+
+// -----
+
+// CHECK-LABEL: @test_preserve_matmul_t_block_scaled
+// CHECK: %[[MATMUL_T:.+]] = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+// CHECK: return %[[MATMUL_T]]
+func.func @test_preserve_matmul_t_block_scaled(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg2: tensor<1xf32>, %arg3: tensor<1xf32>) -> tensor<1x14x28xf32> {
+  %0 = tosa.matmul_t %arg0, %arg1, %arg2, %arg3 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index d290ff71f052d..facb49379248a 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -32,6 +32,36 @@ func.func @test_matmul_t_fp8_input_fp32_acc_type(%arg0: tensor<1x14x19xf8E4M3FN>
 
 // -----
 
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp8e4m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp8e4m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3
+func.func @test_matmul_t_with_block_scaled_inputs_type_fp4e2m1_fp6e2m3(%arg0: tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>) -> tensor<4x8x16xf32> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E2M3FN>>, tensor<1xf32>, tensor<1xf32>) -> tensor<4x8x16xf32>
+  return %0 : tensor<4x8x16xf32>
+}
+
+// -----
+
+// CHECK-LABEL: test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2
+func.func @test_matmul_t_f16_with_block_scaled_inputs_type_fp6e3m2(%arg0: tensor<4x8x32xf16>, %arg1: tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>) -> tensor<4x8x16xf16> {
+  %azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<4x8x32xf16>, tensor<4x16x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f6E3M2FN>>, tensor<1xf16>, tensor<1xf32>) -> tensor<4x8x16xf16>
+  return %0 : tensor<4x8x16xf16>
+}
+
+// -----
+
 // CHECK-LABEL: test_conv2d_fp8_acc32
 func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x1x1x4xf8E5M2>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>

diff  --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index c31effeb6a1d2..6fc88e4bd7694 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1488,6 +1488,16 @@ func.func @test_matmul_t_a_zp_same_element_type(%arg0: tensor<1x14x19xf32>, %arg
 
 // -----
 
+func.func @test_matmul_t_b_zp_block_scaled_requires_fp32(%arg0: tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x14x28xf32> {
+%azp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+%bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+// expected-error at +1 {{'tosa.matmul_t' op expect input b and b_zp have compatible element types, got '!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>' and 'f16'}}
+%0 = tosa.matmul_t %arg0, %arg1, %azp0, %bzp0 : (tensor<1x14x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1x28x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<1xf32>, tensor<1xf16>)  -> tensor<1x14x28xf32>
+  return %0 : tensor<1x14x28xf32>
+}
+
+// -----
+
 func.func @test_matmul_t_a_zp_non_zero(%arg0: tensor<1x14x19xf32>, %arg1: tensor<1x28x19xf32>) -> tensor<1x14x28xf32> {
 %azp0 = "tosa.const"() <{values = dense<1.0> : tensor<1xf32>}> : () -> tensor<1xf32>
 %bzp0 = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>


        

From llvmlistbot at llvm.org  Wed Jul 15 04:51:32 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 04:51:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a577444.bc28b987.3d8071.4466@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Wed Jul 15 04:55:53 2026
From: llvmlistbot at llvm.org (Michael Klemm)
Date: Wed, 15 Jul 2026 04:55:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a577549.af6ba3ed.3db413.9423@mx.google.com>


https://github.com/mjklemm approved this pull request.

I'm fine with the changes,

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Wed Jul 15 04:56:37 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 15 Jul 2026 04:56:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][OpenACC] fix doc formatting for
	reduction ops (PR #209739)
Message-ID: <llvm.org/llvm/llvm-project/pull/209739@github.com>


https://github.com/rscottmanley created https://github.com/llvm/llvm-project/pull/209739

Was not printing correctly due to the matching `$`

>From 232be9cd7502176fb8fb6c1264e583adbb4a2d4b Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Wed, 15 Jul 2026 04:54:21 -0700
Subject: [PATCH] [NFC][OpenACC] fix doc formatting for reduction ops

Was not printing correctly due to the matching `$`
---
 mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index cd64fba564a56..53adb91fc2983 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -42,8 +42,8 @@ def OpenACC_ReductionInitOp
     contains the recipe's init code and must yield a single value (the private
     reduction storage) via `acc.yield`.
 
-    The var operand is the original/shared reduction variable. The
-    reduction_operator specifies the reduction kind (e.g. add, mul).
+    The `var` operand is the original/shared reduction variable. The
+    `reduction_operator` specifies the reduction kind (e.g. add, mul).
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$var,
                        OpenACC_ReductionOperatorAttr:$reductionOperator);
@@ -74,8 +74,8 @@ def OpenACC_ReductionCombineRegionOp
     value(s) in the original/shared reduction variable. The region is
     terminated by `acc.yield` with no operands.
 
-    The destVar operand is the original/shared reduction variable.
-    The srcVar operand is typically the result of acc.reduction_init.
+    The `destVar` operand is the original/shared reduction variable.
+    The `srcVar` operand is typically the result of acc.reduction_init.
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$destVar,
                        OpenACC_AnyPointerOrMappableType:$srcVar);
@@ -113,11 +113,11 @@ def OpenACC_ReductionCombineOp: OpenACC_Op<"reduction_combine",
       memref.store %combine, %dest[] : memref<i32>
     ```
     
-    The $destMemref operand is a "pointer" to the original reduction 
-    variable (typically shared). The $srcMemref operand is a "pointer"
+    The `destMemref` operand is a "pointer" to the original reduction 
+    variable (typically shared). The `srcMemref` operand is a "pointer"
     to the partial sum of the reduction (typically private).
 
-    The $kind is the OpenACC reduction operator that determines how to 
+    The `kind` is the OpenACC reduction operator that determines how to 
     accumulate the two values.
   }];
 


From llvmlistbot at llvm.org  Wed Jul 15 04:57:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 04:57:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][OpenACC] fix doc formatting for
	reduction ops (PR #209739)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209739@github.com>
Message-ID: <6a57759b.c574b9b4.2aa0bf.ad0b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: Scott Manley (rscottmanley)

<details>
<summary>Changes</summary>

Was not printing correctly due to the matching `$`

---
Full diff: https://github.com/llvm/llvm-project/pull/209739.diff


1 Files Affected:

- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td (+7-7) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index cd64fba564a56..53adb91fc2983 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -42,8 +42,8 @@ def OpenACC_ReductionInitOp
     contains the recipe's init code and must yield a single value (the private
     reduction storage) via `acc.yield`.
 
-    The var operand is the original/shared reduction variable. The
-    reduction_operator specifies the reduction kind (e.g. add, mul).
+    The `var` operand is the original/shared reduction variable. The
+    `reduction_operator` specifies the reduction kind (e.g. add, mul).
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$var,
                        OpenACC_ReductionOperatorAttr:$reductionOperator);
@@ -74,8 +74,8 @@ def OpenACC_ReductionCombineRegionOp
     value(s) in the original/shared reduction variable. The region is
     terminated by `acc.yield` with no operands.
 
-    The destVar operand is the original/shared reduction variable.
-    The srcVar operand is typically the result of acc.reduction_init.
+    The `destVar` operand is the original/shared reduction variable.
+    The `srcVar` operand is typically the result of acc.reduction_init.
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$destVar,
                        OpenACC_AnyPointerOrMappableType:$srcVar);
@@ -113,11 +113,11 @@ def OpenACC_ReductionCombineOp: OpenACC_Op<"reduction_combine",
       memref.store %combine, %dest[] : memref<i32>
     ```
     
-    The $destMemref operand is a "pointer" to the original reduction 
-    variable (typically shared). The $srcMemref operand is a "pointer"
+    The `destMemref` operand is a "pointer" to the original reduction 
+    variable (typically shared). The `srcMemref` operand is a "pointer"
     to the partial sum of the reduction (typically private).
 
-    The $kind is the OpenACC reduction operator that determines how to 
+    The `kind` is the OpenACC reduction operator that determines how to 
     accumulate the two values.
   }];
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/209739

From llvmlistbot at llvm.org  Wed Jul 15 04:58:16 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 15 Jul 2026 04:58:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a5775d8.8e75e0cb.1225b0.8beb@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/13041

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8623/8705] Creating library symlink lib/libCUFDialect.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8625/8705] Linking CXX executable bin/mlir-runner
[8626/8705] Linking CXX executable bin/mlir-translate
[8627/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
[8628/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8629/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8634/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8623/8705] Creating library symlink lib/libCUFDialect.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8625/8705] Linking CXX executable bin/mlir-runner
[8626/8705] Linking CXX executable bin/mlir-translate
[8627/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
[8628/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8629/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8634/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=131.227126

```

</details>

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Wed Jul 15 04:59:01 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 15 Jul 2026 04:59:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for block scaled
 types in matmul_t (PR #207851)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207851@github.com>
Message-ID: <6a577605.3cce8983.2f9901.0b57@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/13041

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8623/8705] Creating library symlink lib/libCUFDialect.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8625/8705] Linking CXX executable bin/mlir-runner
[8626/8705] Linking CXX executable bin/mlir-translate
[8627/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
[8628/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8629/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8634/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8623/8705] Creating library symlink lib/libCUFDialect.so
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8625/8705] Linking CXX executable bin/mlir-runner
[8626/8705] Linking CXX executable bin/mlir-translate
[8627/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
[8628/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8629/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8634/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=131.227126

```

</details>

https://github.com/llvm/llvm-project/pull/207851

From llvmlistbot at llvm.org  Wed Jul 15 05:10:46 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 05:10:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ods] Fix string interpolation at end
 of description (PR #209744)
Message-ID: <llvm.org/llvm/llvm-project/pull/209744@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/209744

Updates the error streaming string logic to handle the case where string interpolation used at the end of the description. Previously this could generate malformed code that would not compile e.g.:
```
"' failed to satisfy constraint: another attribute " << reformat(attr)";
```
With this change the above example would now generate:
```
"' failed to satisfy constraint: another attribute " << reformat(attr) << "";
```

>From 43c7776c342274861c25a69100ecac05ba64e312 Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Thu, 9 Jul 2026 23:18:38 +0100
Subject: [PATCH] [mlir][ods] Fix string interpolation at end of description

Updates the error streaming string logic to handle the case
where string interpolation used at the end of the description.
Previously this could generate malformed code that would not
compile e.g.:
```
"' failed to satisfy constraint: another attribute " << reformat(attr)";
```
With this change the above example would now generate:
```
"' failed to satisfy constraint: another attribute " << reformat(attr) << "";
```

Change-Id: Iea9bd52064b5e19a9a258f955b3b70eb01f26415
---
 mlir/lib/TableGen/CodeGenHelpers.cpp       | 14 ++++++--------
 mlir/test/mlir-tblgen/constraint-unique.td |  4 ++--
 2 files changed, 8 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/TableGen/CodeGenHelpers.cpp b/mlir/lib/TableGen/CodeGenHelpers.cpp
index c8c419669f3d4..19439eddcb333 100644
--- a/mlir/lib/TableGen/CodeGenHelpers.cpp
+++ b/mlir/lib/TableGen/CodeGenHelpers.cpp
@@ -157,18 +157,16 @@ std::string mlir::tblgen::buildErrorStreamingString(
 
     os << " << " << tgfmt(var, &ctx);
 
-    if (rest.empty())
-      break;
-
-    split = rest.split("{{");
-    if (split.second.empty() &&
-        errorStreamType == ErrorStreamType::InsideOpError) {
+    if (errorStreamType == ErrorStreamType::InsideOpError) {
       // To enable having part of string post, this adds a parenthesis before
       // the last string segment to match the existing one.
-      os << " << (\"" << split.first;
+      os << " << (\"";
     } else {
-      os << " << \"" << split.first;
+      os << " << \"";
     }
+
+    split = rest.split("{{");
+    os << split.first;
     msg = split.second;
   }
 
diff --git a/mlir/test/mlir-tblgen/constraint-unique.td b/mlir/test/mlir-tblgen/constraint-unique.td
index 5fdaf368c323a..55d6a56e3fc80 100644
--- a/mlir/test/mlir-tblgen/constraint-unique.td
+++ b/mlir/test/mlir-tblgen/constraint-unique.td
@@ -17,7 +17,7 @@ def OtherType : Type<ATypePred, "another type">;
 
 def AnAttrPred : CPred<"attrPred($_self, $_op)">;
 def AnAttr : Attr<AnAttrPred, "an attribute (got {{reformat($_self)}})">;
-def OtherAttr : Attr<AnAttrPred, "another attribute">;
+def OtherAttr : Attr<AnAttrPred, "another attribute {{reformat($_self)}}">;
 
 def ASuccessorPred : CPred<"successorPred($_self, $_op)">;
 def ASuccessor : Successor<ASuccessorPred, "a successor">;
@@ -81,7 +81,7 @@ def OpC : NS_Op<"op_c"> {
 // CHECK:    static ::llvm::LogicalResult [[$O_ATTR_CONSTRAINT:__mlir_ods_local_attr_constraint.*]](
 // CHECK:      if (attr && !((attrPred(attr, *op))))
 // CHECK-NEXT:   return emitError() << "attribute '" << attrName
-// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute";
+// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute " << reformat(attr) << "";
 
 /// Test that a successor contraint was generated.
 // CHECK:    static ::llvm::LogicalResult [[$A_SUCCESSOR_CONSTRAINT:__mlir_ods_local_successor_constraint.*]](


From llvmlistbot at llvm.org  Wed Jul 15 05:11:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 05:11:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ods] Fix string interpolation at end
 of description (PR #209744)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209744@github.com>
Message-ID: <6a5778eb.0c05e68e.b358c.235a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

Updates the error streaming string logic to handle the case where string interpolation used at the end of the description. Previously this could generate malformed code that would not compile e.g.:
```
"' failed to satisfy constraint: another attribute " &lt;&lt; reformat(attr)";
```
With this change the above example would now generate:
```
"' failed to satisfy constraint: another attribute " &lt;&lt; reformat(attr) &lt;&lt; "";
```

---
Full diff: https://github.com/llvm/llvm-project/pull/209744.diff


2 Files Affected:

- (modified) mlir/lib/TableGen/CodeGenHelpers.cpp (+6-8) 
- (modified) mlir/test/mlir-tblgen/constraint-unique.td (+2-2) 


``````````diff
diff --git a/mlir/lib/TableGen/CodeGenHelpers.cpp b/mlir/lib/TableGen/CodeGenHelpers.cpp
index c8c419669f3d4..19439eddcb333 100644
--- a/mlir/lib/TableGen/CodeGenHelpers.cpp
+++ b/mlir/lib/TableGen/CodeGenHelpers.cpp
@@ -157,18 +157,16 @@ std::string mlir::tblgen::buildErrorStreamingString(
 
     os << " << " << tgfmt(var, &ctx);
 
-    if (rest.empty())
-      break;
-
-    split = rest.split("{{");
-    if (split.second.empty() &&
-        errorStreamType == ErrorStreamType::InsideOpError) {
+    if (errorStreamType == ErrorStreamType::InsideOpError) {
       // To enable having part of string post, this adds a parenthesis before
       // the last string segment to match the existing one.
-      os << " << (\"" << split.first;
+      os << " << (\"";
     } else {
-      os << " << \"" << split.first;
+      os << " << \"";
     }
+
+    split = rest.split("{{");
+    os << split.first;
     msg = split.second;
   }
 
diff --git a/mlir/test/mlir-tblgen/constraint-unique.td b/mlir/test/mlir-tblgen/constraint-unique.td
index 5fdaf368c323a..55d6a56e3fc80 100644
--- a/mlir/test/mlir-tblgen/constraint-unique.td
+++ b/mlir/test/mlir-tblgen/constraint-unique.td
@@ -17,7 +17,7 @@ def OtherType : Type<ATypePred, "another type">;
 
 def AnAttrPred : CPred<"attrPred($_self, $_op)">;
 def AnAttr : Attr<AnAttrPred, "an attribute (got {{reformat($_self)}})">;
-def OtherAttr : Attr<AnAttrPred, "another attribute">;
+def OtherAttr : Attr<AnAttrPred, "another attribute {{reformat($_self)}}">;
 
 def ASuccessorPred : CPred<"successorPred($_self, $_op)">;
 def ASuccessor : Successor<ASuccessorPred, "a successor">;
@@ -81,7 +81,7 @@ def OpC : NS_Op<"op_c"> {
 // CHECK:    static ::llvm::LogicalResult [[$O_ATTR_CONSTRAINT:__mlir_ods_local_attr_constraint.*]](
 // CHECK:      if (attr && !((attrPred(attr, *op))))
 // CHECK-NEXT:   return emitError() << "attribute '" << attrName
-// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute";
+// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute " << reformat(attr) << "";
 
 /// Test that a successor contraint was generated.
 // CHECK:    static ::llvm::LogicalResult [[$A_SUCCESSOR_CONSTRAINT:__mlir_ods_local_successor_constraint.*]](

``````````

</details>


https://github.com/llvm/llvm-project/pull/209744

From llvmlistbot at llvm.org  Wed Jul 15 05:12:22 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 05:12:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix regression in mlir StableHLO
 tests due to PR #203583 (PR #207995)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207995@github.com>
Message-ID: <6a577926.c574b9b4.2aa0bf.1164@mx.google.com>


lhutton1 wrote:

Just following up here - I've created https://github.com/llvm/llvm-project/pull/209736 to explore re-enabling these error messages using the approach @jpienaar suggested

https://github.com/llvm/llvm-project/pull/207995

From llvmlistbot at llvm.org  Wed Jul 15 05:13:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 05:13:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ods] Fix string interpolation at end
 of description (PR #209744)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209744@github.com>
Message-ID: <6a577951.d681cab9.120c5c.df96@mx.google.com>


lhutton1 wrote:

Please see https://github.com/llvm/llvm-project/pull/209736 for an overview of how this fix helps error diagnostics.

https://github.com/llvm/llvm-project/pull/209744

From llvmlistbot at llvm.org  Wed Jul 15 05:47:31 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Wed, 15 Jul 2026 05:47:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a578163.af6ba3ed.3db413.c224@mx.google.com>


banach-space wrote:

LGTM, thank you!

https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 05:49:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 05:49:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 42605e7 - [mlir][memref] Support truncating
 copied dims to non-unit in reinterpret_cast elision (#209536)
Message-ID: <6a5781de.f3151ff8.b27a.50da@mx.google.com>


Author: ioana ghiban
Date: 2026-07-15T14:49:28+02:00
New Revision: 42605e7b341ccbc6b2486e2ced82386f099c476a

URL: https://github.com/llvm/llvm-project/commit/42605e7b341ccbc6b2486e2ced82386f099c476a
DIFF: https://github.com/llvm/llvm-project/commit/42605e7b341ccbc6b2486e2ced82386f099c476a.diff

LOG: [mlir][memref] Support truncating copied dims to non-unit in reinterpret_cast elision (#209536)

Relax the `memref.copy`through `memref.reinterpret_cast` rewrite to
support same-dimension slices where strictly one non-unit source
dimension size is truncated to a smaller non-unit result dimension.

Previously, the rewrite only supported truncating to unit size:
```mlir
memref<1xMxNxf32>
  to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
```

This change relaxes constraints to also support:
```mlir
memref<1xMxNxf32>
  to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
```
where `K < N`.

The rewrite remains conservative:
* source must have identity layout, 
* result strides must match the source identity strides, 
* offset must be static for non-scalar results, and
* copied slice must be in bounds.


Assisted-by: Codex (refine PR description).

Added: 
    

Modified: 
    mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
    mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
index a24a0ccaa3d67..49f30a594e0ee 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElideReinterpretCast.cpp
@@ -84,7 +84,7 @@ delinearizeStaticRCOffset(memref::ReinterpretCastOp rc) {
   return offsetIdxs;
 }
 
-static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
+static bool hasExactlyOneTruncatedNonUnitDim(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
   MemRefType resType = dyn_cast<MemRefType>(rc.getType());
   assert(srcType.hasStaticShape() && resType.hasStaticShape() &&
@@ -92,25 +92,26 @@ static bool hasExactlyOneCollapsedNonUnitDim(memref::ReinterpretCastOp rc) {
   assert(srcType.getRank() == resType.getRank() &&
          "expected rank-preserving reinterpret_cast");
 
-  unsigned collapsedDims = 0;
+  unsigned truncatedDims = 0;
 
   for (auto [srcSize, resSize] :
        llvm::zip_equal(srcType.getShape(), resType.getShape())) {
     if (srcSize == resSize)
       continue;
 
-    // Only allow collapsing one non-unit source dim to a unit result dim.
-    if (srcSize != 1 && resSize == 1) {
-      ++collapsedDims;
+    // Only one non-unit source dimension may be truncated.
+    if (srcSize != 1 && resSize < srcSize) {
+      ++truncatedDims;
       continue;
     }
 
-    // The sizes 
diff er and both of them are non-unit - ATM not supported.
+    // The size change is not a supported single non-unit source dimension
+    // reduction.
     return false;
   }
 
-  // Make sure there is only one collapsed dimension.
-  return collapsedDims == 1;
+  // Make sure there is only one truncated dimension.
+  return truncatedDims == 1;
 }
 
 /// Returns the unique non-unit dim or nullopt if # non-unit-dims != 1.
@@ -134,21 +135,19 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// offsets, delinearized offset.
 ///
 /// Supports ranked, static-shape, rank-preserving reinterpret_casts from
-/// identity-layout sources. In addition:
-///     identical to the source identity strides, and exactly one non-unit
-///     source
-///  * Non-scalar results must have static offsets, static result strides
-///     dimension collapsed to unit size
-/// Scalar-shaped results may have arbitrary result strides (i.e. for scalars,
-/// strides are effectively irrelevant).
+/// identity-layout sources.
+/// * Scalar-shaped results may have arbitrary result strides.
+/// * Non-scalar results must have static offsets, static result strides
+///   identical to the source identity strides, and exactly one non-unit
+///   source dimension size truncated.
 ///
-/// Returns nullopt for unsupported
-/// reinterpret_casts.
+/// Returns nullopt for unsupported reinterpret_casts.
 ///
 /// Examples that return info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where K < N
 ///
 ///   reinterpret_cast memref<1xMxf32, identity-layout>
 ///     to memref<1x1xf32, strided<[?, ?], offset: ?>>
@@ -156,10 +155,11 @@ static std::optional<unsigned> getSingleNonUnitDim(MemRefType type) {
 /// Examples that return no info:
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xMx1xf32, strided<[?, N, 1]>>
+///     to memref<1xMxKxf32, strided<[?, N, 1]>>
 ///
 ///   reinterpret_cast memref<1xMxNxf32, identity-layout>
-///     to memref<1xKx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///     to memref<1xNxPxf32, strided<[M*N, N, 1], offset: OFF>>
+///     where M != N && N != P
 static std::optional<ResultNonUnitDimsAndOffsetsForRC>
 getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
   MemRefType srcType = dyn_cast<MemRefType>(rc.getSource().getType());
@@ -221,7 +221,7 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
                       }))
       return std::nullopt;
 
-    if (!hasExactlyOneCollapsedNonUnitDim(rc))
+    if (!hasExactlyOneTruncatedNonUnitDim(rc))
       return std::nullopt;
   }
 
@@ -255,9 +255,9 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///      the store index is derived from the reinterpret_cast offset.
 ///
 ///   2. Non-scalar reinterpret_cast results that preserve all non-unit source
-///      dimensions except one collapsed-to-unit dimension. Result strides must
-///      be static and identical to the identity strides of the source, and the
-///      static offset selects the collapsed dimension.
+///      dimensions sizes except one. Result
+///      strides must be static and identical to the identity strides of the
+///      source, and the offset must be static.
 ///
 ///   // BEFORE (scalar-shaped result)
 ///   %strided = memref.reinterpret_cast %dst
@@ -268,18 +268,20 @@ getResultNonUnitDimsAndOffsetsForRC(memref::ReinterpretCastOp rc) {
 ///   %v = memref.load %src[0, ..., 0]
 ///   memref.store %v, %dst[delinearized(OFF)]
 ///
-///   // BEFORE (one collapsed non-unit dimension)
+///   // BEFORE (one truncated non-unit dimension)
 ///   %strided = memref.reinterpret_cast %dst
-///     to offset: [OFF], sizes: [1, M, 1], strides: [M*N, N, 1]
+///     to offset: [OFF], sizes: [1, M, K], strides: [M*N, N, 1]
 ///     : memref<1xMxNxf32>
-///       to memref<1xMx1xf32, strided<[M*N, N, 1], offset: OFF>>
+///       to memref<1xMxKxf32, strided<[M*N, N, 1], offset: OFF>>
 ///   memref.copy %src, %strided
 ///
 ///   // AFTER
-///   // Assuming OFF delinearizes to [0, 0, OFF]:
+///   // Assuming OFF delinearizes to [0, 0, DELIN_OFF]:
 ///   scf.for %i = 0 to M step 1 {
-///     %v = memref.load %src[0, %i, 0]
-///     memref.store %v, %dst[0, %i, OFF]
+///     scf.for %k = 0 to K step 1 {
+///       %v = memref.load %src[0, %i, %k]
+///       memref.store %v, %dst[0, %i, DELIN_OFF + %k]
+///     }
 ///   }
 struct CopyToLoadAndStore : public OpRewritePattern<memref::CopyOp> {
 public:

diff  --git a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
index 0955d9638f093..f305de1ccd0d8 100644
--- a/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
+++ b/mlir/test/Dialect/MemRef/elide-reinterpret-cast.mlir
@@ -262,20 +262,27 @@ func.func private @negative_copy_scalar_into_2D_strided_dynamic_offset(
 // Non-scalar (ND) copy
 //===----------------------------------------------------------------------===//
 
-/// The result of collapsing dimensions is not a unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_1D_strided(
-func.func private @negative_copy_1D_into_1D_strided(
+// CHECK-LABEL: func.func private @copy_1D_into_1D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<108xf32>
+func.func private @copy_1D_into_1D_strided_zero_offset(
   %src : memref<4xf32>, %dst : memref<108xf32>) {
-  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  // CHECK-NOT:  memref.reinterpret_cast
   %rc = memref.reinterpret_cast %dst
     to offset: [0], sizes: [4], strides: [1]
     : memref<108xf32> to memref<4xf32, strided<[1]>>
 
-  // CHECK:      memref.copy %arg0, %reinterpret_cast
-  // CHECK-NOT:  memref.load
-  // CHECK-NOT:  memref.store
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX:.*]] = %[[C0]] to %[[UB]] step %[[C1]] {
+  // CHECK:        %[[VAL:.*]] = memref.load %[[SRC]][%[[IDX]]] : memref<4xf32>
+  // CHECK:        memref.store %[[VAL]], %[[DST]][%[[IDX]]] : memref<108xf32>
+  // CHECK:      }
   memref.copy %src, %rc
     : memref<4xf32> to memref<4xf32, strided<[1]>>
+  // CHECK-NOT:  memref.copy
   return
 }
 
@@ -418,9 +425,8 @@ func.func private @negative_copy_1D_into_2D_strided_zero_offset_dynamic_stride(%
   return
 }
 
-/// 4 is collapsed to a non-unit dimension.
-// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_strided_
diff _dim_sizes(
-func.func private @negative_copy_1D_into_2D_strided_
diff _dim_sizes(
+// CHECK-LABEL: func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
+func.func private @negative_copy_1D_into_2D_multiple_truncated_dims(
   %src : memref<1x3x1xf32>, %dst : memref<1x4x11xf32>) {
   // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
   %rc = memref.reinterpret_cast %dst
@@ -437,6 +443,113 @@ func.func private @negative_copy_1D_into_2D_strided_
diff _dim_sizes(
   return
 }
 
+// CHECK-LABEL: func.func private @negative_copy_into_strided_no_truncated_dims(
+func.func private @negative_copy_into_strided_no_truncated_dims(%src : memref<3x4xf32>,
+  %dst : memref<3x4xf32>) {
+  // CHECK:      %reinterpret_cast = memref.reinterpret_cast %arg1
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [3, 4], strides: [12, 1]
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+
+  // CHECK:      memref.copy %arg0, %reinterpret_cast
+  // CHECK-NOT:  memref.load
+  // CHECK-NOT:  memref.store
+  memref.copy %src, %rc
+    : memref<3x4xf32> to memref<3x4xf32, strided<[12, 1]>>
+  return
+}
+
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_zero_offset(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_zero_offset(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [0], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1]>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x4xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x3x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v1(
+  %src : memref<1x3x4xf32>, %dst : memref<1x3x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [6], sizes: [1, 3, 4], strides: [33, 11, 1]
+    : memref<1x3x11xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 4 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 6 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX1]] : index
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x4xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[IDX0]], %[[DST_IDX]]] : memref<1x3x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x4xf32>
+      to memref<1x3x4xf32, strided<[33, 11, 1], offset: 6>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
+// CHECK-LABEL: func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+// CHECK-SAME:   %[[SRC:.*]]: memref<1x3x11xf32>
+// CHECK-SAME:   %[[DST:.*]]: memref<1x10x11xf32>
+func.func private @copy_2D_into_2D_strided_nonzero_offset_delinearized_v2(
+  %src : memref<1x3x11xf32>, %dst : memref<1x10x11xf32>) {
+  // CHECK-NOT:  memref.reinterpret_cast
+  %rc = memref.reinterpret_cast %dst
+    to offset: [44], sizes: [1, 3, 11], strides: [110, 11, 1]
+    : memref<1x10x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+
+  // CHECK-NOT:  memref.copy
+  // CHECK-DAG:  %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-DAG:  %[[C1:.*]] = arith.constant 1 : index
+  // CHECK-DAG:  %[[UB0:.*]] = arith.constant 3 : index
+  // CHECK-DAG:  %[[UB1:.*]] = arith.constant 11 : index
+  // CHECK-DAG:  %[[OFF:.*]] = arith.constant 4 : index
+  // CHECK:      scf.for %[[IDX0:.*]] = %[[C0]] to %[[UB0]] step %[[C1]] {
+  // CHECK:        %[[DST_IDX:.*]] = arith.addi %[[OFF]], %[[IDX0]] : index
+  // CHECK:        scf.for %[[IDX1:.*]] = %[[C0]] to %[[UB1]] step %[[C1]] {
+  // CHECK:          %[[VAL:.*]] = memref.load %[[SRC]][%[[C0]], %[[IDX0]], %[[IDX1]]] : memref<1x3x11xf32>
+  // CHECK:          memref.store %[[VAL]], %[[DST]][%[[C0]], %[[DST_IDX]], %[[IDX1]]] : memref<1x10x11xf32>
+  // CHECK:        }
+  // CHECK:      }
+  memref.copy %src, %rc
+    : memref<1x3x11xf32>
+      to memref<1x3x11xf32, strided<[110, 11, 1], offset: 44>>
+  // CHECK-NOT:  memref.copy
+  return
+}
+
 // CHECK-LABEL: func.func private @copy_2D_into_3D_strided_zero_offset(
 // CHECK-SAME:   %[[SRC:.*]]: memref<3x1x4x1xf32>
 // CHECK-SAME:   %[[DST:.*]]: memref<3x1x4x11xf32>


        

From llvmlistbot at llvm.org  Wed Jul 15 05:49:39 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Wed, 15 Jul 2026 05:49:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][memref] Support reducing copied dims
 to non-unit in reinterpret_cast elision (PR #209536)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209536@github.com>
Message-ID: <6a5781e3.92d2873c.3511c3.0e45@mx.google.com>


https://github.com/ioghiban closed https://github.com/llvm/llvm-project/pull/209536

From llvmlistbot at llvm.org  Wed Jul 15 05:59:07 2026
From: llvmlistbot at llvm.org (Udaya Ranga)
Date: Wed, 15 Jul 2026 05:59:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a57841b.e624eb8d.3065e1.f358@mx.google.com>


https://github.com/udaya-ranga approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207944

From llvmlistbot at llvm.org  Wed Jul 15 05:59:15 2026
From: llvmlistbot at llvm.org (Udaya Ranga)
Date: Wed, 15 Jul 2026 05:59:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a578423.c5504292.a1395.8066@mx.google.com>


https://github.com/udaya-ranga approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/207933

From llvmlistbot at llvm.org  Wed Jul 15 06:11:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 06:11:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 706db1e - [mlir][tosa] Add support for MXFP
	row_gather (#207933)
Message-ID: <6a578719.481afadf.34be4b.a223@mx.google.com>


Author: Luke Hutton
Date: 2026-07-15T14:11:47+01:00
New Revision: 706db1e0959f45d6392edd2ad196a4ea38bb99d3

URL: https://github.com/llvm/llvm-project/commit/706db1e0959f45d6392edd2ad196a4ea38bb99d3
DIFF: https://github.com/llvm/llvm-project/commit/706db1e0959f45d6392edd2ad196a4ea38bb99d3.diff

LOG: [mlir][tosa] Add support for MXFP row_gather (#207933)

Adds profile compliance support for MXFP row_gather.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/invalid_extension.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 4e23d8dd2fbbb..9bbbed98402f0 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1475,37 +1475,82 @@ extensionComplianceMap = {
        {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
        allOf}}},
     {"tosa.row_gather",
-     {
-         {{Extension::int16},
-          {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int64},
-          {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
-           {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
-           {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::int16, Extension::int64},
-          {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e4m3},
-          {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e5m2},
-          {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::bf16},
-          {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
-         {{Extension::fp8e4m3, Extension::int64},
-          {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::fp8e5m2, Extension::int64},
-          {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-         {{Extension::bf16, Extension::int64},
-          {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
-          allOf},
-     }},
+     {{{Extension::int16},
+       {{{i16T, i32T, i16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int64},
+       {{{i64T, i32T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{boolT, i64T, boolT}, SpecificationVersion::V_1_1_DRAFT},
+        {{i8T, i64T, i8T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i32T, i64T, i32T}, SpecificationVersion::V_1_1_DRAFT},
+        {{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, i64T, fp16T}, SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, i64T, fp32T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::int16, Extension::int64},
+       {{{i16T, i64T, i16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3},
+       {{{fp8e4m3T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::fp8e5m2},
+       {{{fp8e5m2T, i32T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16},
+       {{{bf16T, i32T, bf16T}, SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i32T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i32T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i32T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i32T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i32T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i32T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::int64},
+       {{{fp8e4m3T, i64T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::int64},
+       {{{fp8e5m2T, i64T, fp8e5m2T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::int64},
+       {{{bf16T, i64T, bf16T}, SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, i64T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, i64T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, i64T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, i64T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, i64T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::int64, Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, i64T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.row_gather_block_scaled",
      {{{Extension::fp8e4m3},
        {{{fp8e4m3T, i32T, i32T, fp8e4m3T}, SpecificationVersion::V_1_1_DRAFT}}},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 24de011e90c92..46ddffd245a15 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2603,7 +2603,15 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
   list<Availability> availability = [Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
                                      Extension<[Tosa_EXT_FP8E4M3,
                                                 Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
-                                                Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+                                                Tosa_EXT_INT16,
+                                                Tosa_EXT_INT64,
+                                                Tosa_EXT_MX_COMMON,
+                                                Tosa_EXT_MX_FP4E2M1,
+                                                Tosa_EXT_MX_FP6E2M3,
+                                                Tosa_EXT_MX_FP6E3M2,
+                                                Tosa_EXT_MX_FP8E4M3,
+                                                Tosa_EXT_MX_FP8E5M2,
+                                                Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;
@@ -2615,6 +2623,8 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: row_gather_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_RowGatherBlockScaledOp
     : Tosa_InferShapedTypeOp<"row_gather_block_scaled", [NoMemoryEffect]> {
@@ -2631,11 +2641,10 @@ def Tosa_RowGatherBlockScaledOp
     - non-block-scaled: 1 input value tensor and 1 result tensor
     - block-scaled: data + scale input tensors and data + scale result tensors
 
-    This operation follows the TOSA 1.1 draft specification and may evolve as
-    the specification is updated.
-
     This operation is not pure. Undefined behaviour may occur if the specified
     indices are out of range.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins Variadic<Tosa_Tensor3D>:$values,

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index bc014fb9b928e..9d982395b9407 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -600,7 +600,7 @@ func.func @test_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) ->
 func.func @test_row_gather(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xf32> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x3xf32>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x3xf32>
   return %0 : tensor<13x52x3xf32>
 }

diff  --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index c68487169da63..ffd60ba5e62ad 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -257,6 +257,14 @@ func.func @test_row_gather(%arg0: tensor<13x21x3xbf16>, %arg1: tensor<13x26xi32>
   return %0 : tensor<13x52x3xbf16>
 }
 
+// -----
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_row_gather_block_scaled(%arg0: tensor<13x21x32xf4E2M1FN>, %arg1: tensor<13x21x1xf8E8M0FNU>, %arg2: tensor<13x26xi32>) -> (tensor<13x52x32xf4E2M1FN>, tensor<13x52x1xf8E8M0FNU>) {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 74ba92bdb25ab..643c04588b6ef 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -1465,6 +1465,14 @@ func.func @test_row_gather_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>, %arg1: tensor<1
   return %0 : tensor<13x52x3xf8E5M2>
 }
 
+// -----
+// CHECK-LABEL: row_gather_mxfp
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: pad_f8E5M2
 func.func @test_pad_f8E5M2(%arg0: tensor<13x21x3xf8E5M2>) -> tensor<13x21x3xf8E5M2> {

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index bea262a11a391..a46da2ab9abd3 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -97,6 +97,15 @@ func.func @test_row_gather_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26
 
 // -----
 
+func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  // expected-error at +1 {{'tosa.row_gather' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index facb49379248a..70a65c3448f39 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -303,6 +303,24 @@ func.func @test_row_gather_f8e5m2_i32_indices(%arg0: tensor<13x21x3xf8E5M2>, %ar
 
 // -----
 
+// CHECK-LABEL: test_row_gather_mxfp_i32_indices
+func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<13x26xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_row_gather_mxfp_i64_indices
+func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, %arg1: tensor<13x26xi64>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
+  %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, tensor<13x26xi64>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


        

From llvmlistbot at llvm.org  Wed Jul 15 06:11:55 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 06:11:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP row_gather
	(PR #207933)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207933@github.com>
Message-ID: <6a57871b.92ea0e2a.9a5e7.c199@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/207933

From llvmlistbot at llvm.org  Wed Jul 15 06:27:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 06:27:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP in CONCAT
	(PR #209719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209719@github.com>
Message-ID: <6a578aa9.f1c1923d.1bf95f.f774@mx.google.com>


https://github.com/lhutton1 approved this pull request.


https://github.com/llvm/llvm-project/pull/209719

From llvmlistbot at llvm.org  Wed Jul 15 06:27:36 2026
From: llvmlistbot at llvm.org (Aiden Grossman)
Date: Wed, 15 Jul 2026 06:27:36 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [compiler-rt] [flang]
 [lld] [lldb] [llvm] [mlir] [polly] [lldb] Drop support for using the
 external shell (PR #209627)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209627@github.com>
Message-ID: <6a578ac8.d681cab9.120c5c.aa0a@mx.google.com>


https://github.com/boomanaiden154 edited https://github.com/llvm/llvm-project/pull/209627

From llvmlistbot at llvm.org  Wed Jul 15 06:28:06 2026
From: llvmlistbot at llvm.org (Aiden Grossman)
Date: Wed, 15 Jul 2026 06:28:06 -0700 (PDT)
Subject: [Mlir-commits] [clang] [clang-tools-extra] [compiler-rt] [flang]
 [lld] [lldb] [llvm] [mlir] [polly] [lldb] Drop support for using the
 external shell (PR #209627)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209627@github.com>
Message-ID: <6a578ae6.66c0679a.3a40e5.a703@mx.google.com>


https://github.com/boomanaiden154 closed https://github.com/llvm/llvm-project/pull/209627

From llvmlistbot at llvm.org  Wed Jul 15 06:36:35 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 06:36:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a578ce3.dc971d6c.33bddd.de93@mx.google.com>


https://github.com/lhutton1 updated https://github.com/llvm/llvm-project/pull/207944

>From b7be91c2d1c8f87b9ba6c75ff60f8feb2f42953f Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Mon, 29 Jun 2026 17:07:15 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP reshape

Adds profile compliance support for MXFP reshape.

Change-Id: I91354f8db555fc000835efe18dd846506025c781
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 26 ++++++++++++++++++-
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  | 10 ++++++-
 mlir/test/Dialect/Tosa/availability.mlir      |  2 +-
 mlir/test/Dialect/Tosa/invalid_extension.mlir |  8 ++++++
 mlir/test/Dialect/Tosa/ops.mlir               |  8 ++++++
 .../tosa-validation-version-1p0-invalid.mlir  |  9 +++++++
 .../tosa-validation-version-1p1-valid.mlir    | 19 +++++++++++++-
 7 files changed, 78 insertions(+), 4 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 9bbbed98402f0..e71907376e0eb 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1421,7 +1421,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.reverse",
      {{{Extension::int64}, {{{i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 46ddffd245a15..536b6cb470bd7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2340,7 +2340,11 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2,
+               Tosa_EXT_MX_FP8E4M3, Tosa_EXT_MX_FP8E5M2,
+               Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -2354,6 +2358,8 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: reshape_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_ReshapeBlockScaledOp
     : Tosa_InferTensorTypeOp<"reshape_block_scaled", [Pure]> {
@@ -2366,6 +2372,8 @@ def Tosa_ReshapeBlockScaledOp
     single non-block-scaled tensor of any rank. No data conversion happens during
     a reshape operation. Reshape must retain the relationship between values
     and their scale in a block for block-scaled content.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let hasVerifier = 1;
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 9d982395b9407..30c5f32d20a29 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -542,7 +542,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
 func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xf32>, !tosa.shape<2>) -> tensor<1x819xf32>
   return %0 : tensor<1x819xf32>
 }
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index ffd60ba5e62ad..94574dea21d01 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -211,6 +211,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xbf16>) -> tensor<1x819xbf16> {
   return %0 : tensor<1x819xbf16>
 }
 
+// -----
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_reverse(%arg0: tensor<13x21x3xbf16>) -> tensor<13x21x3xbf16> {
   // expected-error at +1 {{'tosa.reverse' op illegal: requires any of [bf16] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 643c04588b6ef..41a17375482b2 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -931,6 +931,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   return %0 : tensor<1x819xf32>
 }
 
+// -----
+// CHECK-LABEL: reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: reshape_unranked_output
 func.func @test_reshape_unranked_output(%arg0: tensor<13x21x3xf32>) -> tensor<*xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a46da2ab9abd3..00a5a23cbe767 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -106,6 +106,15 @@ func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_
 
 // -----
 
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 70a65c3448f39..d2d6fe3e9126a 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -309,7 +309,6 @@ func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_s
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
   return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
-
 // -----
 
 // CHECK-LABEL: test_row_gather_mxfp_i64_indices
@@ -321,6 +320,24 @@ func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_s
 
 // -----
 
+// CHECK-LABEL: test_reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_reshape_mxfp_mxint8
+func.func @test_reshape_mxfp_mxint8(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


From llvmlistbot at llvm.org  Wed Jul 15 06:49:40 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Wed, 15 Jul 2026 06:49:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Canonicalize `addf(negf(x),
 y)` to `subf(y, x)` (PR #209277)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209277@github.com>
Message-ID: <6a578ff4.e208124d.1b0258.e95d@mx.google.com>


victor-eds wrote:

> Does instcombine perform the same rewrite?

Yes :)

https://github.com/llvm/llvm-project/pull/209277

From llvmlistbot at llvm.org  Wed Jul 15 06:51:15 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Wed, 15 Jul 2026 06:51:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Canonicalize `addf(negf(x),
 y)` to `subf(y, x)` (PR #209277)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209277@github.com>
Message-ID: <6a579053.cd9bf2f4.18ec1d.749b@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/209277

>From bcc58dcd530ad06ab73ee6c164ea16ce455804a6 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Mon, 13 Jul 2026 07:25:08 -0700
Subject: [PATCH 1/2] [mlir][arith] Canonicalize addf(negf(x), y) to subf(y, x)

arith.addf had no canonicalization patterns at all. This adds a
canonicalizer rewriting addf(negf(x), y) -> subf(y, x) (and the commuted
operand order).

The rewrite is valid under any rounding mode, which is propagated to the
subf: negf is an exact sign flip, IEEE addition is commutative, and y - x
is the correctly rounded y + (-x), so subf(y, x) and addf(negf(x), y)
are bit-identical with identical exception behavior for every rounding
mode. Fast-math flags are propagated to the new subf.
---
 .../include/mlir/Dialect/Arith/IR/ArithOps.td |  1 +
 .../Dialect/Arith/IR/ArithCanonicalization.td | 18 +++++
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        |  5 ++
 mlir/test/Dialect/Arith/canonicalize.mlir     | 74 +++++++++++++++++++
 4 files changed, 98 insertions(+)

diff --git a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
index 423948c8734af..54481d3232483 100644
--- a/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
+++ b/mlir/include/mlir/Dialect/Arith/IR/ArithOps.td
@@ -1113,6 +1113,7 @@ def Arith_AddFOp : Arith_FloatBinaryOpWithRoundingMode<"addf", [Commutative]> {
     %a = arith.addf %b, %c to_nearest_even : f64
     ```
   }];
+  let hasCanonicalizer = 1;
   let hasFolder = 1;
 }
 
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index 86c9b6d3551d8..84deb2b920a85 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -521,6 +521,24 @@ def UIToFPOfExtUI :
     Pat<(Arith_UIToFPOp (Arith_ExtUIOp $x, $nneg1), $nneg2),
         (Arith_UIToFPOp $x, $nneg1)>;
 
+//===----------------------------------------------------------------------===//
+// AddFOp
+//===----------------------------------------------------------------------===//
+
+// addf(negf(x), y) -> subf(y, x)
+// addf(y, negf(x)) -> subf(y, x)
+//
+// Valid for any rounding mode, which is propagated to the subf: negf is an
+// exact sign flip, IEEE addition is commutative, and y - x is the correctly
+// rounded y + (-x), so subf(y, x) and addf(negf(x), y) are bit-identical.
+def AddFOfNegFLhs :
+    Pat<(Arith_AddFOp (Arith_NegFOp $x, $_), $y, $fmf, $rm),
+        (Arith_SubFOp $y, $x, $fmf, $rm)>;
+
+def AddFOfNegFRhs :
+    Pat<(Arith_AddFOp $y, (Arith_NegFOp $x, $_), $fmf, $rm),
+        (Arith_SubFOp $y, $x, $fmf, $rm)>;
+
 //===----------------------------------------------------------------------===//
 // SubFOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 397fa28227831..674577e5764d8 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -1235,6 +1235,11 @@ OpFoldResult arith::AddFOp::fold(FoldAdaptor adaptor) {
       });
 }
 
+void arith::AddFOp::getCanonicalizationPatterns(RewritePatternSet &patterns,
+                                                MLIRContext *context) {
+  patterns.add<AddFOfNegFLhs, AddFOfNegFRhs>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // SubFOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index c12befc76675a..e273ff58921ef 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3005,6 +3005,80 @@ func.func @test_addf_rounding_mode(%arg0 : f32) -> (f32, f32, f32) {
 
 // -----
 
+// CHECK-LABEL: @test_addf_negf(
+//  CHECK-SAME: %[[ARG0:.+]]: f32, %[[ARG1:.+]]: f32
+func.func @test_addf_negf(%arg0 : f32, %arg1 : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[X:.+]] = arith.subf %[[ARG1]], %[[ARG0]] : f32
+  // CHECK-DAG:  %[[Y:.+]] = arith.subf %[[ARG0]], %[[ARG1]] : f32
+  // CHECK:      return %[[X]], %[[Y]]
+  %0 = arith.negf %arg0 : f32
+  %1 = arith.addf %0, %arg1 : f32
+  %2 = arith.negf %arg1 : f32
+  %3 = arith.addf %arg0, %2 : f32
+  return %1, %3 : f32, f32
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_types
+//  CHECK-SAME: %[[F16:.+]]: f16, %[[G16:.+]]: f16, %[[F64:.+]]: f64, %[[G64:.+]]: f64, %[[BF:.+]]: bf16, %[[GBF:.+]]: bf16,
+//  CHECK-SAME: %[[VEC:.+]]: vector<4xf32>, %[[GVEC:.+]]: vector<4xf32>,
+//  CHECK-SAME: %[[TEN:.+]]: tensor<2x3xf32>, %[[GTEN:.+]]: tensor<2x3xf32>
+func.func @test_addf_negf_types(
+    %f16 : f16, %g16 : f16, %f64 : f64, %g64 : f64, %bf : bf16, %gbf : bf16,
+    %vec : vector<4xf32>, %gvec : vector<4xf32>,
+    %ten : tensor<2x3xf32>, %gten : tensor<2x3xf32>)
+    -> (f16, f64, bf16, vector<4xf32>, tensor<2x3xf32>) {
+  // CHECK-DAG:  arith.subf %[[G16]], %[[F16]] : f16
+  // CHECK-DAG:  arith.subf %[[G64]], %[[F64]] : f64
+  // CHECK-DAG:  arith.subf %[[GBF]], %[[BF]] : bf16
+  // CHECK-DAG:  arith.subf %[[GVEC]], %[[VEC]] : vector<4xf32>
+  // CHECK-DAG:  arith.subf %[[GTEN]], %[[TEN]] : tensor<2x3xf32>
+  // CHECK-NOT:  arith.negf
+  // CHECK-NOT:  arith.addf
+  %n16 = arith.negf %f16 : f16
+  %0 = arith.addf %n16, %g16 : f16
+  %n64 = arith.negf %f64 : f64
+  %1 = arith.addf %g64, %n64 : f64
+  %nbf = arith.negf %bf : bf16
+  %2 = arith.addf %nbf, %gbf : bf16
+  %nvec = arith.negf %vec : vector<4xf32>
+  %3 = arith.addf %gvec, %nvec : vector<4xf32>
+  %nten = arith.negf %ten : tensor<2x3xf32>
+  %4 = arith.addf %nten, %gten : tensor<2x3xf32>
+  return %0, %1, %2, %3, %4 : f16, f64, bf16, vector<4xf32>, tensor<2x3xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_fastmath(
+//  CHECK-SAME: %[[X:.+]]: f32, %[[Y:.+]]: f32
+func.func @test_addf_negf_fastmath(%x : f32, %y : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[A:.+]] = arith.subf %[[Y]], %[[X]] fastmath<nnan,nsz> : f32
+  // CHECK-DAG:  %[[B:.+]] = arith.subf %[[Y]], %[[X]] fastmath<reassoc> : f32
+  // CHECK:      return %[[A]], %[[B]]
+  %n = arith.negf %x : f32
+  %0 = arith.addf %n, %y fastmath<nnan,nsz> : f32
+  %1 = arith.addf %y, %n fastmath<reassoc> : f32
+  return %0, %1 : f32, f32
+}
+
+// -----
+
+// CHECK-LABEL: @test_addf_negf_rounding_mode(
+//  CHECK-SAME: %[[X:.+]]: f32, %[[Y:.+]]: f32
+func.func @test_addf_negf_rounding_mode(%x : f32, %y : f32) -> (f32, f32) {
+  // CHECK-DAG:  %[[A:.+]] = arith.subf %[[Y]], %[[X]] downward : f32
+  // CHECK-DAG:  %[[B:.+]] = arith.subf %[[Y]], %[[X]] upward : f32
+  // CHECK:      return %[[A]], %[[B]]
+  %n = arith.negf %x : f32
+  %0 = arith.addf %n, %y downward : f32
+  %1 = arith.addf %y, %n upward : f32
+  return %0, %1 : f32, f32
+}
+
+// -----
+
 // CHECK-LABEL: @test_subf_rounding_mode(
 // CHECK-SAME: %[[ARG0:.+]]: f32
 func.func @test_subf_rounding_mode(%arg0 : f32) -> (f32, f32, f32, f32) {

>From 5449fcc36af75d0c29d7c2bb16ba787743011763 Mon Sep 17 00:00:00 2001
From: Victor Perez <victor.pc.upm at gmail.com>
Date: Wed, 15 Jul 2026 09:51:06 -0400
Subject: [PATCH 2/2] Tweak comment, preserving x y ordering

---
 mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index 84deb2b920a85..dcc8b8371dba8 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -526,7 +526,7 @@ def UIToFPOfExtUI :
 //===----------------------------------------------------------------------===//
 
 // addf(negf(x), y) -> subf(y, x)
-// addf(y, negf(x)) -> subf(y, x)
+// addf(x, negf(y)) -> subf(x, y)
 //
 // Valid for any rounding mode, which is propagated to the subf: negf is an
 // exact sign flip, IEEE addition is commutative, and y - x is the correctly


From llvmlistbot at llvm.org  Wed Jul 15 06:52:55 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Wed, 15 Jul 2026 06:52:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Canonicalize `addf(negf(x),
 y)` to `subf(y, x)` (PR #209277)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209277@github.com>
Message-ID: <6a5790b7.30b32922.2578be.e109@mx.google.com>


================
@@ -521,6 +521,24 @@ def UIToFPOfExtUI :
     Pat<(Arith_UIToFPOp (Arith_ExtUIOp $x, $nneg1), $nneg2),
         (Arith_UIToFPOp $x, $nneg1)>;
 
+//===----------------------------------------------------------------------===//
+// AddFOp
+//===----------------------------------------------------------------------===//
+
+// addf(negf(x), y) -> subf(y, x)
+// addf(y, negf(x)) -> subf(y, x)
----------------
victor-eds wrote:

Fixed (`addf(x, negf(y)) -> subf(x, y)` actually)

https://github.com/llvm/llvm-project/pull/209277

From llvmlistbot at llvm.org  Wed Jul 15 06:56:44 2026
From: llvmlistbot at llvm.org (Kirill Vedernikov)
Date: Wed, 15 Jul 2026 06:56:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a57919c.bc28b987.3d8071.3973@mx.google.com>


================
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
----------------
kvederni wrote:

I like that new mma fp8 tests are added to a separate file. Thanks!
Could you please rename the filename to something like nvgpu-to-nvvm-mma-fp8.mlir and add more combinations here for e5m2 type and shapes?

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Wed Jul 15 07:01:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 07:01:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8a41c1f - [mlir][tosa] Add support for MXFP
	reshape (#207944)
Message-ID: <6a5792bb.eb165e5c.3844ee.cd29@mx.google.com>


Author: Luke Hutton
Date: 2026-07-15T15:01:26+01:00
New Revision: 8a41c1f77732d18eae12409b572fc926df29062b

URL: https://github.com/llvm/llvm-project/commit/8a41c1f77732d18eae12409b572fc926df29062b
DIFF: https://github.com/llvm/llvm-project/commit/8a41c1f77732d18eae12409b572fc926df29062b.diff

LOG: [mlir][tosa] Add support for MXFP reshape (#207944)

Adds profile compliance support for MXFP reshape.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/invalid_extension.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index 9bbbed98402f0..e71907376e0eb 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1421,7 +1421,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.reverse",
      {{{Extension::int64}, {{{i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e4m3},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 46ddffd245a15..536b6cb470bd7 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2340,7 +2340,11 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT64, Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2,
+               Tosa_EXT_MX_FP8E4M3, Tosa_EXT_MX_FP8E5M2,
+               Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -2354,6 +2358,8 @@ def Tosa_ReshapeOp : Tosa_InferTensorTypeOp<"reshape", [Pure]> {
 
 //===----------------------------------------------------------------------===//
 // Operator: reshape_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_ReshapeBlockScaledOp
     : Tosa_InferTensorTypeOp<"reshape_block_scaled", [Pure]> {
@@ -2366,6 +2372,8 @@ def Tosa_ReshapeBlockScaledOp
     single non-block-scaled tensor of any rank. No data conversion happens during
     a reshape operation. Reshape must retain the relationship between values
     and their scale in a block for block-scaled content.
+
+    Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let hasVerifier = 1;

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 9d982395b9407..30c5f32d20a29 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -542,7 +542,7 @@ func.func @test_pad(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
 func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   %1 = tosa.const_shape {values = dense<[1, 819]> : tensor<2xindex>} : () -> !tosa.shape<2>
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.reshape %arg0, %1 : (tensor<13x21x3xf32>, !tosa.shape<2>) -> tensor<1x819xf32>
   return %0 : tensor<1x819xf32>
 }

diff  --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index ffd60ba5e62ad..94574dea21d01 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -211,6 +211,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xbf16>) -> tensor<1x819xbf16> {
   return %0 : tensor<1x819xbf16>
 }
 
+// -----
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 func.func @test_reverse(%arg0: tensor<13x21x3xbf16>) -> tensor<13x21x3xbf16> {
   // expected-error at +1 {{'tosa.reverse' op illegal: requires any of [bf16] profiles/extensions to be specified in the target environment}}

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 643c04588b6ef..41a17375482b2 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -931,6 +931,14 @@ func.func @test_reshape(%arg0: tensor<13x21x3xf32>) -> tensor<1x819xf32> {
   return %0 : tensor<1x819xf32>
 }
 
+// -----
+// CHECK-LABEL: reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
 // -----
 // CHECK-LABEL: reshape_unranked_output
 func.func @test_reshape_unranked_output(%arg0: tensor<13x21x3xf32>) -> tensor<*xf32> {

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a46da2ab9abd3..00a5a23cbe767 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -106,6 +106,15 @@ func.func @test_row_gather_mxfp(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_
 
 // -----
 
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  // expected-error at +1 {{'tosa.reshape' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
 func.func @test_row_gather_block_scaled_i8_i32(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>
   // expected-error at +1 {{'tosa.row_gather_block_scaled' op illegal: requires specification version compatible with 1.1.draft (got 1.0) OR requires specification version compatible with 1.1.draft (got 1.0) to be specified in the target environment}}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index 70a65c3448f39..d2d6fe3e9126a 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -309,7 +309,6 @@ func.func @test_row_gather_mxfp_i32_indices(%arg0: tensor<13x21x32x!tosa.block_s
   %0 = tosa.row_gather %arg0, %arg1, %row_count : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<13x26xi32>, tensor<1xi32>) -> tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
   return %0 : tensor<13x52x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
 }
-
 // -----
 
 // CHECK-LABEL: test_row_gather_mxfp_i64_indices
@@ -321,6 +320,24 @@ func.func @test_row_gather_mxfp_i64_indices(%arg0: tensor<13x21x32x!tosa.block_s
 
 // -----
 
+// CHECK-LABEL: test_reshape_mxfp
+func.func @test_reshape_mxfp(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>
+}
+
+// -----
+
+// CHECK-LABEL: test_reshape_mxfp_mxint8
+func.func @test_reshape_mxfp_mxint8(%arg0: tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
+  %1 = tosa.const_shape {values = dense<[2, 64]> : tensor<2xindex>} : () -> !tosa.shape<2>
+  %0 = tosa.reshape %arg0, %1 : (tensor<4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>, !tosa.shape<2>) -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+  return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
+}
+
+// -----
+
 // CHECK-LABEL: test_row_gather_block_scaled_i8_i32_indices
 func.func @test_row_gather_block_scaled_i8_i32_indices(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x26xi32>) -> tensor<13x52x3xi8> {
   %row_count = "tosa.const"() {values = dense<2> : tensor<1xi32>} : () -> tensor<1xi32>


        

From llvmlistbot at llvm.org  Wed Jul 15 07:01:34 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 15 Jul 2026 07:01:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a5792be.6c5720b3.5d850.fa03@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/207944

From llvmlistbot at llvm.org  Wed Jul 15 07:03:40 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 15 Jul 2026 07:03:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][OpenACC] fix doc formatting for
	reduction ops (PR #209739)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209739@github.com>
Message-ID: <6a57933c.d1496011.289bf0.977e@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Nice catch. Thank you.

https://github.com/llvm/llvm-project/pull/209739

From llvmlistbot at llvm.org  Wed Jul 15 07:21:42 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 15 Jul 2026 07:21:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a579776.fae8f0c5.8c6dc.9db7@mx.google.com>


https://github.com/rscottmanley approved this pull request.

LGTM, nice work!

https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Wed Jul 15 07:34:06 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Wed, 15 Jul 2026 07:34:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP reshape (PR
	#207944)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207944@github.com>
Message-ID: <6a579a5e.92d2873c.3511c3.91f3@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-offload-build-only` running on `AMD-bb-w-02` while building `mlir` at step 4 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/226/builds/13068

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 4 (annotate) failure: '../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py --jobs=32' (failure)
...
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)


-- Configuring incomplete, errors occurred!
[8624/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8626/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8627/8705] Linking CXX executable bin/mlir-runner
[8628/8705] Linking CXX executable bin/mlir-translate
[8629/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8634/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
@@@STEP_FAILURE@@@
Step 7 (build cmake config) failure: build cmake config (failure)
...
Missing variable is:
CMAKE_Fortran_PREPROCESS_SOURCE
CMake Error at /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /usr/local/therock-tools/share/cmake-3.27/Modules/CMakeTestFortranCompiler.cmake:20 (CMAKE_DETERMINE_COMPILER_ABI)
  cmake/config-Fortran.cmake:157 (enable_language)
  CMakeLists.txt:297 (include)
-- Configuring incomplete, errors occurred!
[8624/8705] Building CXX object tools/flang/lib/Optimizer/Passes/CMakeFiles/flangPasses.dir/Pipelines.cpp.o
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure 
cd /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt --no-warn-unused-cli -DCMAKE_C_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/clang -DCMAKE_Fortran_COMPILER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/flang -DCMAKE_LINKER=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/ld.lld -DCMAKE_AR=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-ranlib -DCMAKE_NM=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-strip -DCMAKE_READELF=/home/botworker/bbot/amdgpu-offload-build-only/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_Fortran_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=x86_64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/tmp/llvm.install.test -DLLVM_BINARY_DIR=/home/botworker/bbot/amdgpu-offload-build-only/build -DLLVM_CONFIG_PATH=/home/botworker/bbot/amdgpu-offload-build-only/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF "-DLLVM_LIT_ARGS=-v --show-unsupported --timeout 100 --show-xfail -j 16" -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=24.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/local/bin/ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_Fortran_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON -DCLANG_RESOURCE_DIR= -DLLVM_LIBDIR_SUFFIX= -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt -DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../openmp -DLLVM_EXTERNAL_OFFLOAD_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../offload -DLLVM_EXTERNAL_FLANG_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../flang-rt -DFORTRAN_SUPPORTS_REAL16=0 -DCOMPILER_RT_TEST_EXTERNAL_BUILTINS=ON "-DLLVM_ENABLE_RUNTIMES=compiler-rt;openmp;offload;flang-rt" -DFLANG_RUNTIME_F128_MATH_LIB= -DLIBOMPTEST_BUILD_UNITTESTS=ON -DLIBOMP_ENABLE_ARM64X=OFF "-DRUNTIMES_CMAKE_ARGS=-DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=/home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../compiler-rt;-DLLVM_EXTERNAL_OPENMP_SOURCE_DIR=/home/botworker/bbot/amdgpu
d-only/llvm-project/llvm/runtimes/../../flang-rt" -DRUNTIMES_amdgpu-amd-amdhsa_LLVM_ENABLE_RUNTIMES=openmp -GNinja -C/home/botworker/bbot/amdgpu-offload-build-only/build/projects/runtimes/tmp/runtimes-cache-Release.cmake -S /home/botworker/bbot/amdgpu-offload-build-only/llvm-project/llvm/runtimes/../../runtimes -B /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-bins && /usr/local/therock-tools/bin/cmake -E touch /home/botworker/bbot/amdgpu-offload-build-only/build/runtimes/runtimes-stamps/runtimes-configure
[8626/8705] Building CXX object tools/flang/lib/Optimizer/Support/CMakeFiles/FIRSupport.dir/InitFIR.cpp.o
[8627/8705] Linking CXX executable bin/mlir-runner
[8628/8705] Linking CXX executable bin/mlir-translate
[8629/8705] Linking CXX shared library lib/libHLFIRDialect.so.24.0git
[8630/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllPasses.dir/RegisterAllPasses.cpp.o
[8631/8705] Building CXX object tools/flang/tools/fir-lsp-server/CMakeFiles/fir-lsp-server.dir/fir-lsp-server.cpp.o
[8632/8705] Building CXX object tools/flang/tools/fir-opt/CMakeFiles/fir-opt.dir/fir-opt.cpp.o
[8633/8705] Building CXX object tools/flang/tools/tco/CMakeFiles/tco.dir/tco.cpp.o
[8634/8705] Building CXX object tools/mlir/lib/CMakeFiles/obj.MLIRRegisterAllDialects.dir/RegisterAllDialects.cpp.o
[8635/8705] Building CXX object tools/flang/tools/bbc/CMakeFiles/bbc.dir/bbc.cpp.o
[8636/8705] Building CXX object tools/flang/lib/Frontend/CMakeFiles/flangFrontend.dir/FrontendActions.cpp.o
ninja: build stopped: subcommand failed.
['ninja'] exited with return code 1.
The build step threw an exception...
Traceback (most recent call last):
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 62, in step
    yield
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 53, in main
    run_command(["ninja"])
  File "../llvm-zorg/zorg/buildbot/builders/annotated/amdgpu-offload-cmake.py", line 75, in run_command
    util.report_run_cmd(cmd, cwd=directory)
  File "/home/botworker/bbot/amdgpu-offload-build-only/llvm-zorg/zorg/buildbot/builders/annotated/util.py", line 49, in report_run_cmd
    subprocess.check_call(cmd, shell=shell, *args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ninja']' returned non-zero exit status 1.
program finished with exit code 0
elapsedTime=129.916658

```

</details>

https://github.com/llvm/llvm-project/pull/207944

From llvmlistbot at llvm.org  Wed Jul 15 07:39:32 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 15 Jul 2026 07:39:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Introduce acc to gpu codegen pass
	(PR #209606)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209606@github.com>
Message-ID: <6a579ba4.510f576d.d8fd6.c3d7@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/209606

From llvmlistbot at llvm.org  Wed Jul 15 07:39:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 07:39:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f8f37a6 - [mlir][acc] Introduce acc to gpu
	codegen pass (#209606)
Message-ID: <6a579ba1.1905e68e.cdbe2.fe3f@mx.google.com>


Author: Razvan Lupusoru
Date: 2026-07-15T07:39:24-07:00
New Revision: f8f37a65d472e65561994fb7ae55f6b539c357d4

URL: https://github.com/llvm/llvm-project/commit/f8f37a65d472e65561994fb7ae55f6b539c357d4
DIFF: https://github.com/llvm/llvm-project/commit/f8f37a65d472e65561994fb7ae55f6b539c357d4.diff

LOG: [mlir][acc] Introduce acc to gpu codegen pass (#209606)

Introduce pass that lowers `acc.compute_region` to the GPU dialect. For
host-side kernels it wraps the region in `gpu.launch`; for specialized
ACC routines already inside a `gpu.func`, the body is lowered in place.

The pass maps nested `scf.parallel` / `scf.for` loops carrying
`acc.par_dims` to GPU block and thread parallelism, materializes
privatization and reductions for the device, and handles
synchronization.

Example input IR:
```
func.func @two_sibling_loops() {
  %c4 = arith.constant 4 : index
  %c32 = arith.constant 32 : index
  %grid = acc.par_width %c4 {par_dim = #acc.par_dim<block_x>}
  %block = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}

  acc.kernel_environment {
    %priv = acc.privatize [#acc<par_dims[block_x]>] : () ->
!acc.private_type<memref<32xi32>>
    acc.compute_region launch(%g = %grid, %t = %block) ins(%p = %priv)
        : (!acc.private_type<memref<32xi32>>) {
      %slot = acc.private_local %p : (!acc.private_type<memref<32xi32>>)
-> memref<32xi32>
      %c0 = arith.constant 0 : index
      %c1 = arith.constant 1 : index
      %c32_len = arith.constant 32 : index

      scf.parallel (%tx) = (%c0) to (%t) step (%c1) {
        scf.parallel (%i) = (%c0) to (%c32_len) step (%c1) {
          %v = arith.index_cast %i : index to i32
          memref.store %v, %slot[%i] : memref<32xi32>
          scf.reduce
        } {acc.par_dims = #acc<par_dims[sequential]>}
        scf.reduce
      } {acc.par_dims = #acc<par_dims[thread_x]>}

      scf.parallel (%bx) = (%c0) to (%g) step (%c1) {
        scf.parallel (%tx2) = (%c0) to (%t) step (%c1) {
          scf.parallel (%j) = (%c0) to (%c32_len) step (%c1) {
            %loaded = memref.load %slot[%j] : memref<32xi32>
            memref.store %loaded, %slot[%j] : memref<32xi32>
            scf.reduce
          } {acc.par_dims = #acc<par_dims[sequential]>}
          scf.reduce
        } {acc.par_dims = #acc<par_dims[thread_x]>}
        scf.reduce
      } {acc.par_dims = #acc<par_dims[block_x]>}

      acc.yield
    } {origin = "acc.parallel"}
  }
  return
}
```

Output IR:
```
module {
  func.func @two_sibling_loops() {
    %c4 = arith.constant 4 : index
    %c32 = arith.constant 32 : index
    acc.kernel_environment {
      %0 = acc.privatize [#acc<par_dims[block_x]>] : () ->
!acc.private_type<memref<32xi32>>
      %c1 = arith.constant 1 : index
      gpu.launch blocks(%arg0, %arg1, %arg2) in (%arg6 = %c4, %arg7 =
%c1, %arg8 = %c1)
                   threads(%arg3, %arg4, %arg5) in (%arg9 = %c32, %arg10
= %c1, %arg11 = %c1) {
        %grid_dim_x = gpu.grid_dim x
        %grid_dim_y = gpu.grid_dim y
        %grid_dim_z = gpu.grid_dim z
        %block_dim_x = gpu.block_dim x
        %block_dim_y = gpu.block_dim y
        %block_dim_z = gpu.block_dim z
        %1 = acc.gpu_shared_memory {num_copies = 1 : i64,
static_upper_bound_bytes = 128 : i64}
             : () -> memref<32xi32, #gpu.address_space<workgroup>>
        %memspacecast = memref.memory_space_cast %1
             : memref<32xi32, #gpu.address_space<workgroup>> to
memref<32xi32>
        %c0 = arith.constant 0 : index
        %c1_0 = arith.constant 1 : index
        %c32_1 = arith.constant 32 : index

        scf.parallel (%arg12) = (%c0) to (%c32_1) step (%c1_0) {
          %2 = arith.index_cast %arg12 : index to i32
          memref.store %2, %memspacecast[%arg12] : memref<32xi32>
          scf.reduce
        } {acc.par_dims = #acc<par_dims[sequential]>}

        gpu.barrier

        scf.parallel (%arg12) = (%c0) to (%c32_1) step (%c1_0) {
          %2 = memref.load %memspacecast[%arg12] : memref<32xi32>
          memref.store %2, %memspacecast[%arg12] : memref<32xi32>
          scf.reduce
        } {acc.par_dims = #acc<par_dims[sequential]>}

        gpu.barrier
        gpu.terminator
      }
    }
    return
  }
}
```

---------

Co-authored-by: Matsu <kmatsumura at nvidia.com>

Added: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-barrier-gang-private-init.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-launch-mapping.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region-reuse-barrier.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-private-local-gang-redundant.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-local.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-routine-seq.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array-shared.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-block-combine-no-reload.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-combine-region-private-dest.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reuse-barrier-sibling-region-privatize.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-call-nested-if.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-worker-call-with-thread-y-reduction.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-reuse-barrier.mlir
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-subgroup-align.mlir

Modified: 
    mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
    mlir/lib/Dialect/OpenACC/Transforms/CMakeLists.txt

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
index 5921b8030432f..04723c5db2d12 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
@@ -541,4 +541,33 @@ def ACCBindRoutine
   let options = [ AccDeviceTypeOption ];
 }
 
+def ACCCGToGPU : InterfacePass<"acc-cg-to-gpu", "mlir::FunctionOpInterface"> {
+  let summary = "Lower acc.compute_region to gpu.launch";
+  let description = [{
+    Generate GPU code for acc.compute_region loop nests by converting them to
+    gpu.launch and mapping acc.par_dims parallelism to GPU block and thread
+    dimensions.
+  }];
+  let options = [
+    AccDeviceTypeOption,
+    Option<"maxWorkgroupSharedMemory", "max-workgroup-shared-memory", "int64_t",
+        "49152",
+        "Maximum workgroup shared memory budget in bytes.">,
+    Option<"maxThreadPrivateStack", "max-thread-private-stack", "int64_t",
+        "16384",
+        "Maximum thread-private stack allocation in bytes.">,
+    Option<"subgroupSize", "subgroup-size", "int64_t", "32",
+        "Subgroup size used for block-dimension alignment and reductions.">
+  ];
+  let dependentDialects = [
+    "mlir::arith::ArithDialect",
+    "mlir::complex::ComplexDialect",
+    "mlir::gpu::GPUDialect",
+    "mlir::acc::OpenACCDialect",
+    "mlir::scf::SCFDialect",
+    "mlir::NVVM::NVVMDialect",
+    "mlir::memref::MemRefDialect"
+  ];
+}
+
 #endif // MLIR_DIALECT_OPENACC_TRANSFORMS_PASSES

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
new file mode 100644
index 0000000000000..87375e0291528
--- /dev/null
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -0,0 +1,3617 @@
+//===- ACCCGToGPU.cpp - Lower acc.compute_region to gpu.launch ------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass lowers `acc.compute_region` to the GPU dialect. For host-side
+// kernels it wraps the region in `gpu.launch`; for specialized acc routines
+// already inside a `gpu.func`, the body is lowered in place without emitting
+// a launch.
+//
+// Overview:
+// ---------
+// `acc.compute_region` is the compute-body representation produced after
+// OpenACC compute constructs are decomposed and parallelism has been assigned.
+// This pass is the final ACC-to-GPU lowering step for that body: it converts
+// nested `scf.parallel` / `scf.for` loops marked with `acc.par_dims` into GPU
+// block and thread parallelism, materializes privatization and reductions for
+// the device, inserts synchronization where shared state is observed across
+// threads, and erases the ACC scaffolding (`acc.compute_region`,
+// `acc.par_width`).
+//
+// Transformations:
+// ----------------
+// 1. Launch creation: outside a `gpu.func`, each `acc.compute_region` becomes a
+//    `gpu.launch` whose grid and block sizes come from `acc.par_width` launch
+//    operands (defaulting to 1). Kernel/module name attributes are preserved.
+//    Inside a `gpu.func` (specialized acc routine), no launch is emitted.
+//
+// 2. Parallel loops: `scf.parallel` with a single `acc.par_dims` entry is
+//    mapped to the corresponding GPU dimension (`block_*` or `thread_*`).
+//    Sequential dimensions remain as `scf.parallel`/`scf.for` loops in the
+//    generated kernel body.
+//
+// 3. Privatization: `acc.privatize` / `acc.private_local` storage is
+//    materialized as one of: a per-thread `memref.alloca` (thread-private
+//    arrays within the stack budget), an `acc.gpu_shared_memory` buffer
+//    (gang-/worker-private arrays that fit the shared-memory budget), or a
+//    `memref.alloc` whose pointer is broadcast to the block through a small
+//    shared-memory slot (the data lives in global memory; shared memory only
+//    holds the broadcast pointer).
+//
+// 4. Predication: `acc.predicate_region` becomes `scf.if` guarded by active
+//    thread/block indices derived from `acc.par_dims` and launch dimensions.
+//
+// 5. Reductions: `acc.reduction_*` ops are lowered to GPU reduction and
+//    synchronization primitives according to each reduction's parallel
+//    dimensions and accumulator storage class.
+//
+// Example:
+// --------
+// Before:
+//   %c128 = arith.constant 128 : index
+//   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+//   acc.compute_region launch(%arg0 = %tx) {
+//     %c0 = arith.constant 0 : index
+//     %c1 = arith.constant 1 : index
+//     scf.parallel (%iv) = (%c0) to (%c128) step (%c1) {
+//       ...
+//       scf.reduce
+//     } {acc.par_dims = #acc<par_dims[thread_x]>}
+//     acc.yield
+//   } {origin = "acc.parallel"}
+//
+// After:
+//   gpu.launch blocks(%bidx, %bidy, %bidz) in (%gdimx = %c1, ...)
+//                threads(%tidx, %tidy, %tidz) in (%bdimx = %c128, ...) {
+//     ...
+//   }
+//
+// Requirements:
+// -------------
+// - Must run on a GPU device type (`device-type` option); host and multicore
+//   targets are rejected.
+// - Input must already be in the `acc.compute_region` form: nested SCF loops
+//   carry `acc.par_dims`, privatization is expressed via `acc.privatize` /
+//   `acc.private_local`, and reductions use the `acc.reduction_*` ops.
+// - Each `scf.parallel` processed by this pass is expected to have exactly
+//   one parallel dimension and one induction variable.
+// - For acc routines, the `acc.compute_region` must live inside a `gpu.func`
+//   in the GPU module.
+// - Uses `acc::OpenACCSupport` for NYI reporting and compiler remarks.
+// - Pass options: `max-workgroup-shared-memory`, `max-thread-private-stack`,
+//   and `subgroup-size` (used for reductions and block-dimension alignment).
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h"
+
+#include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/Complex/IR/Complex.h"
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/Dialect/GPU/IR/GPUDialect.h"
+#include "mlir/Dialect/GPU/Utils/GPUUtils.h"
+#include "mlir/Dialect/LLVMIR/NVVMDialect.h"
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/OpenACC/Analysis/OpenACCSupport.h"
+#include "mlir/Dialect/OpenACC/OpenACC.h"
+#include "mlir/Dialect/OpenACC/OpenACCParMapping.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsCG.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsGPU.h"
+#include "mlir/Dialect/OpenACC/OpenACCUtilsReduction.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/IR/Block.h"
+#include "mlir/IR/BuiltinAttributeInterfaces.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/IR/Diagnostics.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/IRMapping.h"
+#include "mlir/IR/OpDefinition.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/IR/Value.h"
+#include "mlir/Interfaces/CallInterfaces.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ViewLikeInterface.h"
+#include "mlir/Support/LLVM.h"
+#include "mlir/Transforms/DialectConversion.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/StringExtras.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/Debug.h"
+#include <algorithm>
+#include <optional>
+#include <utility>
+
+namespace mlir {
+namespace acc {
+#define GEN_PASS_DEF_ACCCGTOGPU
+#include "mlir/Dialect/OpenACC/Transforms/Passes.h.inc"
+} // namespace acc
+} // namespace mlir
+
+#define DEBUG_TYPE "acc-cg-to-gpu"
+
+namespace {
+using namespace mlir;
+using namespace mlir::acc;
+
+enum class PrivateMemScope { Thread, Worker, Gang, None };
+
+/// Device label used in compiler remarks (e.g. "NVIDIA GPU").
+static std::string getDeviceRemarkQualifier(DeviceType deviceType) {
+  switch (deviceType) {
+  case DeviceType::None:
+  case DeviceType::Star:
+  case DeviceType::Default:
+    return "GPU";
+  default: {
+    std::string name;
+    llvm::StringRef deviceName = stringifyDeviceType(deviceType);
+    name.reserve(deviceName.size());
+    for (char c : deviceName)
+      name.push_back(llvm::toUpper(c));
+    return name + " GPU";
+  }
+  }
+}
+
+/// True when \p op is inside a specialized acc routine function.
+static bool isInsideACCSpecializedRoutine(Operation *op) {
+  FunctionOpInterface funcOp = op->getParentOfType<FunctionOpInterface>();
+  return funcOp && acc::isSpecializedAccRoutine(funcOp);
+}
+
+/// Maps an acc.routine's parallelism clauses to a GPU parallel dimension.
+static GPUParallelDimAttr
+getAccRoutineParDim(RoutineOp routineOp, MLIRContext *ctx,
+                    const ACCToGPUMappingPolicy &policy) {
+  if (routineOp.getGangDimValue() ||
+      routineOp.getGangDimValue(DeviceType::Nvidia)) {
+    int64_t gangDimValue = routineOp.getGangDimValue(DeviceType::Nvidia)
+                               ? *routineOp.getGangDimValue(DeviceType::Nvidia)
+                               : *routineOp.getGangDimValue();
+    ParLevel gangLevel = getGangParLevel(gangDimValue);
+    return policy.gangDim(ctx, gangLevel);
+  }
+  if (routineOp.hasGang() || routineOp.hasGang(DeviceType::Nvidia))
+    return policy.gangDim(ctx, ParLevel::gang_dim1);
+  if (routineOp.hasWorker() || routineOp.hasWorker(DeviceType::Nvidia))
+    return policy.workerDim(ctx);
+  if (routineOp.hasVector() || routineOp.hasVector(DeviceType::Nvidia))
+    return policy.vectorDim(ctx);
+  return policy.seqDim(ctx);
+}
+
+/// Looks up the acc.routine symbol associated with \p funcOp.
+static RoutineOp getRoutineOpForAccRoutineFunction(FunctionOpInterface funcOp,
+                                                   const SymbolTable &symTab) {
+  if (isSpecializedAccRoutine(funcOp)) {
+    SpecializedRoutineAttr attr = funcOp->getAttrOfType<SpecializedRoutineAttr>(
+        getSpecializedRoutineAttrName());
+    return symTab.lookup<RoutineOp>(attr.getRoutine().getLeafReference());
+  }
+  RoutineInfoAttr routineInfo =
+      funcOp->getAttrOfType<RoutineInfoAttr>(getRoutineInfoAttrName());
+  if (!routineInfo || routineInfo.getAccRoutines().empty())
+    return nullptr;
+  return symTab.lookup<RoutineOp>(
+      routineInfo.getAccRoutines().front().getLeafReference());
+}
+
+/// Returns the parallelism level of a specialized acc routine function.
+static GPUParallelDimAttr
+getSpecializedRoutineDim(FunctionOpInterface funcOp,
+                         const ACCToGPUMappingPolicy &policy) {
+  SpecializedRoutineAttr specAttr =
+      funcOp->getAttrOfType<SpecializedRoutineAttr>(
+          getSpecializedRoutineAttrName());
+  assert(specAttr && "expected specialized routine attribute");
+  return policy.map(funcOp->getContext(), specAttr.getLevel().getValue());
+}
+
+/// Returns the parallelism dimension of a callee acc routine, if any.
+static GPUParallelDimAttr
+getAccRoutineCallParDim(CallOpInterface callOp,
+                        const ACCToGPUMappingPolicy &policy) {
+  std::optional<CallInterfaceCallable> callee = callOp.getCallableForCallee();
+  if (!callee)
+    return nullptr;
+  SymbolRefAttr calleeSymbolRef = dyn_cast<SymbolRefAttr>(*callee);
+  if (!calleeSymbolRef)
+    return nullptr;
+  ModuleOp moduleOp = callOp->getParentOfType<ModuleOp>();
+  if (!moduleOp)
+    return nullptr;
+
+  SymbolTable symTab(moduleOp);
+  FunctionOpInterface funcOp =
+      symTab.lookup<FunctionOpInterface>(calleeSymbolRef.getLeafReference());
+  if (!funcOp)
+    return nullptr;
+
+  if (isSpecializedAccRoutine(funcOp))
+    return getSpecializedRoutineDim(funcOp, policy);
+  if (RoutineOp routineOp = getRoutineOpForAccRoutineFunction(funcOp, symTab))
+    return getAccRoutineParDim(routineOp, funcOp.getContext(), policy);
+  return nullptr;
+}
+
+/// Collects parallel dimensions from enclosing loops and the compute region.
+static SmallVector<GPUParallelDimAttr> getAncestorParDims(Operation *op) {
+  SmallVector<GPUParallelDimAttr> parDimsArray;
+  scf::ParallelOp parentLoop = op->getParentOfType<scf::ParallelOp>();
+  while (parentLoop) {
+    if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(parentLoop))
+      for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+        insertParDim(parDimsArray, parDim);
+    parentLoop = parentLoop->getParentOfType<scf::ParallelOp>();
+  }
+
+  ComputeRegionOp computeRegion = op->getParentOfType<ComputeRegionOp>();
+  assert(computeRegion && "missing enclosing acc.compute_region");
+  if (GPUParallelDimsAttr parDimsAttr = getParDimsAttr(computeRegion))
+    for (GPUParallelDimAttr parDim : parDimsAttr.getArray())
+      insertParDim(parDimsArray, parDim);
+  return parDimsArray;
+}
+
+/// Strips index casts to reach the underlying defining value.
+static Value stripIndexCastsFromValue(Value x) {
+  Operation *op = x.getDefiningOp();
+  if (!op)
+    return x;
+  while (arith::IndexCastOp castOp = dyn_cast<arith::IndexCastOp>(op)) {
+    op = castOp->getOperand(0).getDefiningOp();
+    if (!op)
+      return x;
+  }
+  return op->getResult(0);
+}
+
+/// Extracts a compile-time integer constant from \p x, when known.
+static FailureOr<int64_t> extractIntConst(Value x,
+                                          bool stripIndexCasts = false) {
+  if (stripIndexCasts)
+    x = stripIndexCastsFromValue(x);
+  Operation *op = x.getDefiningOp();
+  if (op) {
+    if (arith::ConstantIntOp constOp = dyn_cast<arith::ConstantIntOp>(op)) {
+      assert(constOp.getType().getIntOrFloatBitWidth() <= 64);
+      return constOp.value();
+    }
+    if (arith::ConstantIndexOp constOp = dyn_cast<arith::ConstantIndexOp>(op))
+      return constOp.value();
+  }
+  return failure();
+}
+
+/// True when \p x is a constant equal to \p y (modulo index casts).
+static bool sameEffectiveValue(Value x, int64_t y) {
+  x = stripIndexCastsFromValue(x);
+  FailureOr<int64_t> conX = extractIntConst(x);
+  if (failed(conX))
+    return false;
+  return *conX == y;
+}
+
+/// Continues tracking a memref through view-like and partial-access ops.
+static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
+                                  SmallVectorImpl<Value> &passThroughResults) {
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
+      return true;
+    }
+    return false;
+  }
+
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
+      return true;
+    }
+    return false;
+  }
+  return false;
+}
+
+/// Skips memref view/cast chains to reach the underlying buffer.
+static Value unwrapMemRefConversion(Value v) {
+  while (Operation *op = v.getDefiningOp()) {
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op)) {
+      if (isa<MemRefType>(viewLike.getViewSource().getType()) ||
+          isa<MemRefType>(viewLike.getViewDest().getType())) {
+        v = viewLike.getViewSource();
+        continue;
+      }
+    }
+    break;
+  }
+  return v;
+}
+
+/// Casts between pointer-like private types when lowering requires it.
+static Value castPointerLikeTypeIfNeeded(OpBuilder &builder, Location loc,
+                                         Value value, Type resultType) {
+  if (value.getType() == resultType)
+    return value;
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(value.getType())) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  if (PointerLikeType ptrLike = dyn_cast<PointerLikeType>(resultType)) {
+    if (Value casted = ptrLike.genCast(builder, loc, value, resultType))
+      return casted;
+  }
+  emitError(loc) << "unsupported pointer-like type cast from "
+                 << value.getType() << " to " << resultType;
+  return value;
+}
+
+/// Returns the sole user of \p v, or null if it has zero or multiple uses.
+static Operation *getOnlyUser(Value v) {
+  if (!v.hasOneUse())
+    return nullptr;
+  return *v.user_begin();
+}
+
+/// True when \p privatize is privatized at thread_x parallelism.
+static bool isThreadXPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isThreadX(); });
+  return false;
+}
+
+/// Emits a workgroup-wide GPU barrier.
+static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc);
+}
+
+/// Emits a subgroup-scoped GPU barrier.
+static void emitGPUBarrierSubgroup(OpBuilder &builder, Location loc) {
+  gpu::BarrierOp::create(builder, loc, /*address_spaces=*/ArrayAttr{},
+                         /*named_barrier=*/Value{},
+                         gpu::BarrierScope::Subgroup);
+}
+
+/// Lowers a single `acc.compute_region` to GPU dialect IR.
+class ACCCGToGPULowering {
+public:
+  explicit ACCCGToGPULowering(acc::ComputeRegionOp computeRegion,
+                              RewriterBase &rewriter,
+                              acc::OpenACCSupport &accSupport,
+                              const ACCCGToGPUOptions &options)
+      : rewriter(rewriter), computeRegion(computeRegion),
+        accSupport(accSupport), options(options),
+        sharedMemBudget(
+            options.maxWorkgroupSharedMemory,
+            sumExistingSharedMemoryBytes(computeRegion.getRegion())) {}
+
+  /// Main entry point: emit launch (if needed) and lower the region body.
+  LogicalResult rewrite();
+
+  gpu::LaunchOp getLaunch() const { return launch; }
+
+  bool hasFailed = false;
+  bool insideAccumulateGridStride = false;
+  Value reductionSharedBuf;
+  // Reduction-accumulator slot (memref) -> the block-reduced value stored into
+  // it; lets a block combine use the register instead of reloading.
+  llvm::DenseMap<Value, Value> reductionAccumValue;
+  // Combine reloads recorded before accumulates are lowered, patched up after.
+  llvm::SmallVector<std::pair<Value, memref::LoadOp>> pendingCombineReloads;
+
+private:
+  /// Lower a parallel loop to the GPU dimension given by its `acc.par_dims`.
+  void processParallelOp(scf::ParallelOp parallelOp);
+  /// Lower a sequential loop, including any required post-loop barriers.
+  template <typename LoopOp>
+  void processSeqLoop(LoopOp loopOp);
+  /// Lower an `acc.predicate_region` to a predicated `scf.if`.
+  void processPredicateRegion(acc::PredicateRegionOp interOp);
+  /// Materialize storage for an `acc.private_local`.
+  void
+  processPrivateLocal(acc::PrivateLocalOp privateLocal,
+                      std::optional<int64_t> sharedMemCopies = std::nullopt);
+  /// Lower an `acc.privatize` to device storage.
+  Value processPrivatize(acc::PrivatizeOp privatize);
+  /// Clone and lower an `scf.execute_region`.
+  void processExecuteRegion(scf::ExecuteRegionOp op);
+  /// Lower `acc.reduction_accumulate`.
+  void processAccumulateOp(acc::ReductionAccumulateOp op);
+  /// Lower `acc.reduction_accumulate_array`.
+  void processAccumulateArrayOp(acc::ReductionAccumulateArrayOp op);
+  /// Lower `acc.reduction_init`.
+  void processReductionOp(acc::ReductionInitOp op);
+  /// Lower `acc.reduction_combine`.
+  void processReductionCombineOp(acc::ReductionCombineOp op);
+  /// Lower `acc.reduction_combine_region`.
+  void processCombineRegionOp(acc::ReductionCombineRegionOp op);
+  /// Clone a leaf operation into the lowered region.
+  void processGenericOp(Operation *op);
+  /// Clone and recursively lower an operation with nested regions.
+  void processGenericOpWithRegions(Operation *op);
+  /// Dispatch lowering for one operation in the compute-region body.
+  void processOp(Operation *op);
+
+  /// Emit an atomic reduction update to \p memref.
+  void constructAtomicAccumulation(Location loc, Value memref,
+                                   ValueRange indices, Value input,
+                                   arith::AtomicRMWKind kind);
+
+  /// Map an ACC reduction operator to an atomic RMW kind.
+  FailureOr<arith::AtomicRMWKind> getReductionKind(acc::ReductionOperator redOp,
+                                                   Type type, Location loc);
+
+  /// Split launch dimensions into those that execute \p op and those that do
+  /// not, for predication and barrier placement.
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+  computeActiveAndInactiveParDims(Operation *op, Block *block);
+
+  /// Build a predicate that is true only on inactive parallel dimensions.
+  Value
+  emitPredicate(Location loc,
+                SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims);
+
+  /// True when \p privateLocal may be placed in shared memory; returns the
+  /// number of copies needed, or nullopt if ineligible.
+  std::optional<int64_t>
+  isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                            MemRefType baseTy);
+
+  /// Reserve \p bytes from the shared-memory budget.
+  bool tryAllocateSharedMemory(int64_t bytes);
+
+  /// Element size in bytes for \p elementType .
+  int64_t getElementSizeInBytes(Location loc, Type elementType) const;
+
+  /// True when a static privatization fits in the per-thread stack budget.
+  bool canUseStackAlloca(MemRefType baseTy, Location loc,
+                         int64_t maxThreadPrivateStack) const;
+
+  /// Emit a barrier scoped to the parallel dimensions in \p parDimsAttr.
+  void createBarrier(Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr);
+
+  /// Emit a per-row (per-worker) barrier.
+  /// Runtime branch on blockDim.y == 1 (workgroup-wide); compile-time choice
+  /// between gpu.barrier scope<subgroup> (staticBlockDimX <= subgroupSize)
+  /// and a named gpu.barrier (staticBlockDimX > subgroupSize) with tid.y+1.
+  void createPerRowBarrier(Location loc);
+
+  /// Insert barriers after a sequential loop when shared private state must be
+  /// visible to later loops.
+  void createBarrierAfterSeqLoop(Operation *loopOp);
+
+  /// Flush any deferred post-loop barriers that precede \p beforeOp.
+  void flushDeferredBarriersBefore(Operation *beforeOp);
+
+  /// True when \p loopOp may write shared memory read by a later sibling loop.
+  bool mayWriteSharedMemory(Operation *loopOp);
+
+  /// Parallelism scope (thread, worker, or gang) of a privatized variable.
+  PrivateMemScope getPrivateMemScope(acc::PrivatizeOp privatizeOp);
+
+  /// Parallelism scope of the private buffer backing \p memref.
+  PrivateMemScope getPrivateScopeForMemref(Value memref);
+
+  /// `acc.privatize` that materialized the private buffer for \p memref.
+  acc::PrivatizeOp getPrivatizeForMemref(Value memref);
+
+  /// Whether a predicate region needs a barrier before stores that will be read
+  /// by a later parallel loop over the same private memory.
+  PrivateMemScope needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp);
+
+  /// Emit `gpu.all_reduce` for a reduction partial.
+  void createGPUAllReduceOp(Location loc, Value input, Value memref,
+                            arith::AtomicRMWKind kind,
+                            mlir::acc::GPUParallelDimsAttr parDimsAttr,
+                            ValueRange indices = {});
+
+  /// Finish lowering a deferred `acc.reduction_accumulate`.
+  void postprocessAccumulateOp(acc::ReductionAccumulateOp op);
+
+  /// Finish lowering reductions attached to a parallel loop.
+  void postprocessLoopReduction(scf::ParallelOp parLoop);
+
+  /// Populate block/thread id and grid/block dimension maps for device
+  /// routines.
+  static void
+  createForAllDimensions(RewriterBase &rewriter, Location loc,
+                         llvm::DenseMap<gpu::Processor, Value> &ids,
+                         llvm::DenseMap<gpu::Processor, Value> &dims) {
+    ids[gpu::Processor::BlockX] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::BlockY] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::BlockZ] = gpu::BlockIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    ids[gpu::Processor::ThreadX] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    ids[gpu::Processor::ThreadY] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    ids[gpu::Processor::ThreadZ] = gpu::ThreadIdOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::BlockX] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::BlockY] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::BlockZ] = gpu::GridDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+    dims[gpu::Processor::ThreadX] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+    dims[gpu::Processor::ThreadY] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+    dims[gpu::Processor::ThreadZ] = gpu::BlockDimOp::create(
+        rewriter, loc, rewriter.getIndexType(), gpu::Dimension::z);
+  }
+
+  /// Return the compute-region block argument for \p outside, adding an `ins`
+  /// operand when needed.
+  BlockArgument getOrAppendInsBlockArg(Value outside) {
+    if (std::optional<BlockArgument> blockArg =
+            computeRegion.getBlockArg(outside)) {
+      return *blockArg;
+    }
+    return computeRegion.appendInputArg(outside);
+  }
+
+  /// Wire dynamic privatization extents into the compute region as `ins` args.
+  void preparePrivatizeExtentInsOperands() {
+    computeRegion.walk([&](acc::PrivateLocalOp privateLocal) {
+      acc::PrivatizeOp privatizeOp =
+          getPrivatizeOp(privateLocal, computeRegion);
+      if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() == computeRegion)
+        return;
+      for (Value extent : privatizeOp.getDynamicSizes())
+        getOrAppendInsBlockArg(extent);
+    });
+  }
+
+  /// Resolve dynamic size operands for a privatized array.
+  SmallVector<Value>
+  resolvePrivateLocalDynamicExtents(acc::PrivateLocalOp privateLocal) {
+    acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+    SmallVector<Value> extents;
+    for (Value extent : privatizeOp.getDynamicSizes()) {
+      if (std::optional<BlockArgument> blockArg =
+              computeRegion.getBlockArg(extent)) {
+        extents.push_back(mapping.lookupOrDefault(*blockArg));
+        continue;
+      }
+      extents.push_back(mapping.lookupOrDefault(extent));
+    }
+    return extents;
+  }
+
+  RewriterBase &rewriter;
+  acc::ComputeRegionOp computeRegion;
+
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+  gpu::LaunchOp launch;
+  IRMapping mapping;
+  llvm::SmallVector<scf::ParallelOp> loopReductions;
+  llvm::DenseMap<gpu::Processor, Value> threadIdMap;
+  llvm::DenseMap<gpu::Processor, Value> dimensionMap;
+  // True if ThreadY reduction exists, which triggers subgroup alignment
+  bool hasThreadYReduction = false;
+  // True if any ThreadX routine call exists in the kernel
+  bool hasThreadLevelRoutineCall = false;
+  // True when a per-row ThreadY barrier is emitted
+  bool hasThreadYBarrier = false;
+
+  // Reusable privatize broadcast slots per type; disabled for kernels.
+  llvm::DenseMap<Type, Value> privatizeBroadcastCache;
+
+  int64_t staticBlockDimX = 1024;
+  acc::DefaultACCToGPUMappingPolicy defaultPolicy;
+  SharedMemoryBudget sharedMemBudget;
+  SmallVector<std::string> sharedMemPrivateVarNames;
+  llvm::SmallVector<Operation *, 4> deferredBarrierSeqLoops;
+
+  Value getThreadId(Location loc, gpu::Dimension dim) {
+    return gpu::ThreadIdOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  Value getBlockDim(Location loc, gpu::Dimension dim) {
+    return gpu::BlockDimOp::create(rewriter, loc, rewriter.getIndexType(), dim);
+  }
+
+  /// Thread id for \p proc, from the launch op or the routine context map.
+  Value getGPUThreadIdFor(gpu::Processor proc) {
+    return getGPUThreadId(proc, getLaunch(), threadIdMap);
+  }
+
+  /// Grid/block dimension for \p proc, from the launch op or routine map.
+  Value getGPUSizeFor(gpu::Processor proc) {
+    return getGPUSize(proc, getLaunch(), dimensionMap);
+  }
+};
+
+int64_t ACCCGToGPULowering::getElementSizeInBytes(Location loc,
+                                                  Type elementType) const {
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  if (std::optional<acc::TypeSizeAndAlignment> sizeAndAlignment =
+          accSupport.getTypeSizeAndAlignment(elementType, module)) {
+    return sizeAndAlignment->first.getFixedValue();
+  }
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "element size computation for unsupported type: " << elementType;
+  (void)accSupport.emitNYI(loc, os.str());
+  return 0;
+}
+
+bool ACCCGToGPULowering::canUseStackAlloca(
+    MemRefType baseTy, Location loc, int64_t maxThreadPrivateStack) const {
+  for (int64_t dim : baseTy.getShape())
+    if (dim == ShapedType::kDynamic)
+      return false;
+  int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape()) {
+    if (numElements > maxThreadPrivateStack / std::max<int64_t>(dim, 1))
+      return false;
+    numElements *= dim;
+  }
+  return elementSize * numElements < maxThreadPrivateStack;
+}
+
+/// True if the accumulate spans a block dim or is nested in a block-mapped
+/// loop, i.e. each block owns the elements it reduces across threads. A
+/// thread-only accumulate with no block context grid-strides its element loop
+/// onto blocks, so per-thread partials would be dropped; such reductions must
+/// stay shared.
+static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
+  auto hasBlock = [](mlir::acc::GPUParallelDimsAttr parDims) {
+    return parDims && llvm::any_of(parDims.getArray(),
+                                   [](auto pd) { return pd.isAnyBlock(); });
+  };
+  if (hasBlock(accArr.getParDimsAttr()))
+    return true;
+  for (scf::ParallelOp loop = accArr->getParentOfType<scf::ParallelOp>(); loop;
+       loop = loop->getParentOfType<scf::ParallelOp>()) {
+    if (hasBlock(mlir::acc::getParDimsAttr(loop)))
+      return true;
+  }
+  return false;
+}
+
+/// Returns the array reduction accumulate (through cast/view ops) that \p v
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// and it has block context so the cross-thread all_reduce is well defined.
+static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
+  SmallVector<Value> worklist{v};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value cur = worklist.pop_back_val();
+    if (!seen.insert(cur).second)
+      continue;
+    for (Operation *user : cur.getUsers()) {
+      if (acc::ReductionAccumulateArrayOp accArr =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
+        bool hasThread = false;
+        for (auto pd : accArr.getParDims().getArray())
+          hasThread |= pd.isAnyThread();
+        if (hasThread && reductionHasBlockContext(accArr))
+          return accArr;
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, cur, through))
+        worklist.append(through.begin(), through.end());
+      else if (isa<ViewLikeOpInterface>(user))
+        worklist.append(user->result_begin(), user->result_end());
+    }
+  }
+  return nullptr;
+}
+
+/// Store the reduction identity to every element of a freshly allocated
+/// per-thread array accumulator so all lanes start from identity (the original
+/// init loop may only run on one lane).
+static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
+                                    MemRefType baseTy,
+                                    arith::AtomicRMWKind kind) {
+  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static rank-1");
+  Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
+                                    /*useOnlyFiniteValue=*/true);
+  Value lb = arith::ConstantIndexOp::create(b, loc, 0);
+  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
+  Value step = arith::ConstantIndexOp::create(b, loc, 1);
+  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+  OpBuilder::InsertionGuard g(b);
+  b.setInsertionPoint(forOp.getBody()->getTerminator());
+  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+}
+
+std::optional<int64_t>
+ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
+                                              MemRefType baseTy) {
+  // Cross-thread array reduction accumulators must stay per-thread.
+  if (perThreadArrayReductionAccum(privateLocal.getResult()))
+    return std::nullopt;
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  FailureOr<bool> isCandidate = isPrivateLocalSharedMemoryCandidate(
+      privateLocal, computeRegion, module, defaultPolicy, &accSupport);
+  if (failed(isCandidate)) {
+    hasFailed = true;
+    return std::nullopt;
+  }
+  if (!isCandidate.value())
+    return std::nullopt;
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, computeRegion,
+                                                 module, defaultPolicy);
+  assert(upperBound && "candidate private_local must have an upper bound");
+  int64_t elementSize =
+      getElementSizeInBytes(privateLocal.getLoc(), baseTy.getElementType());
+  int64_t numElements = 1;
+  for (int64_t dim : baseTy.getShape())
+    numElements *= dim;
+  return *upperBound / (elementSize * numElements);
+}
+
+bool ACCCGToGPULowering::tryAllocateSharedMemory(int64_t bytes) {
+  return sharedMemBudget.tryAllocate(bytes);
+}
+
+FailureOr<arith::AtomicRMWKind>
+ACCCGToGPULowering::getReductionKind(acc::ReductionOperator redOp, Type type,
+                                     Location loc) {
+  if (std::optional<arith::AtomicRMWKind> kind =
+          translateACCReductionOperator(redOp, type))
+    return *kind;
+
+  std::string msg;
+  llvm::raw_string_ostream os(msg);
+  os << "reduction operator (" << redOp << ") for type " << type;
+  (void)accSupport.emitNYI(loc, os.str());
+  return failure();
+}
+
+LogicalResult ACCCGToGPULowering::rewrite() {
+
+  // Pre-compute if thread-level reductions exist. ThreadY reduction generates
+  // shuffles which require subgroup alignment (blockDim.x = subgroupSize),
+  // meaning ThreadX lanes exist even without explicit ThreadX parallelism.
+  computeRegion->walk([&](acc::ReductionAccumulateOp op) -> WalkResult {
+    for (auto parDim : op.getParDimsAttr().getArray()) {
+      if (parDim.isThreadY()) {
+        hasThreadYReduction = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  // Pre-compute if any thread-level (vector or worker) routine call exists.
+  // Such routines partition work across ThreadX/ThreadY and emit workgroup-wide
+  // barriers internally (e.g. for shared memory alloca synchronization), so all
+  // workgroup threads must reach the call site for those barriers to converge.
+  computeRegion->walk([&](CallOpInterface callOp) -> WalkResult {
+    if (mlir::acc::GPUParallelDimAttr parDim =
+            getAccRoutineCallParDim(callOp, defaultPolicy)) {
+      if (parDim.isThreadX() || parDim.isThreadY()) {
+        hasThreadLevelRoutineCall = true;
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+
+  Location loc = computeRegion->getLoc();
+  Value constantOne = arith::ConstantIndexOp::create(rewriter, loc, 1);
+
+  auto launchArgument = [&](gpu::Processor processor) -> Value {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> maybeLaunchArg =
+        computeRegion.getKnownLaunchArg(parDim);
+    LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: launch-arg: "
+                            << " parDim: " << parDim << " gpu: " << processor
+                            << " widthValue: "
+                            << maybeLaunchArg.value_or(constantOne) << "\n");
+
+    return getValueOrCreateCastToIndexLike(
+        rewriter, loc, rewriter.getIndexType(),
+        maybeLaunchArg.value_or(constantOne));
+  };
+  LLVM_DEBUG(llvm::dbgs() << "ACCCGToGPU: creating gpu launch op: \n");
+
+  // acc.compute_region keeps launch argument as block argument, for rewriting
+  // we now replace these with gpu.launch dimensions.
+  auto mapLaunchArguments = [&](gpu::Processor processor, Value launchArg) {
+    mlir::acc::GPUParallelDimAttr parDim = mlir::acc::GPUParallelDimAttr::get(
+        computeRegion->getContext(), processor);
+    std::optional<Value> kernelArg = computeRegion.getLaunchArg(parDim);
+    if (kernelArg)
+      mapping.map(computeRegion.gpuParWidth(processor), launchArg);
+  };
+
+  llvm::StringRef blockDimXName = "blockDim.x";
+  llvm::StringRef blockDimYName = "blockDim.y";
+  std::string deviceLabel = getDeviceRemarkQualifier(options.deviceType);
+
+  if (!computeRegion->getParentOfType<gpu::GPUFuncOp>()) {
+    Value blockDimX = launchArgument(gpu::Processor::ThreadX);
+    APInt bdxVal;
+    if (matchPattern(blockDimX, m_ConstantInt(&bdxVal)))
+      staticBlockDimX = bdxVal.getSExtValue();
+    Value blockDimY = launchArgument(gpu::Processor::ThreadY);
+    Value blockDimZ = launchArgument(gpu::Processor::ThreadZ);
+    Value gridDimX = launchArgument(gpu::Processor::BlockX);
+    Value gridDimY = launchArgument(gpu::Processor::BlockY);
+    Value gridDimZ = launchArgument(gpu::Processor::BlockZ);
+
+    // The format of the message is:
+    // Generating [serial] {deviceLabel} code with gridDim=32x1x1
+    // blockDim=256x1x1
+    accSupport.emitRemark(computeRegion, [&]() {
+      auto getName = [&](Value val) -> std::string {
+        std::string name = accSupport.getVariableName(val);
+        return name.empty() ? "(*)" : name;
+      };
+      bool isEffectivelySerial =
+          sameEffectiveValue(blockDimX, 1) &&
+          sameEffectiveValue(blockDimY, 1) &&
+          sameEffectiveValue(blockDimZ, 1) && sameEffectiveValue(gridDimX, 1) &&
+          sameEffectiveValue(gridDimY, 1) && sameEffectiveValue(gridDimZ, 1);
+      return (llvm::Twine("Generating ") +
+              llvm::Twine(isEffectivelySerial ? "serial " : "") + deviceLabel +
+              " code with gridDim=" + getName(gridDimX) + "x" +
+              getName(gridDimY) + "x" + getName(gridDimZ) +
+              " blockDim=" + getName(blockDimX) + "x" + getName(blockDimY) +
+              "x" + getName(blockDimZ))
+          .str();
+    });
+
+    // Check if kernel has a stream operand for async execution
+    if (mlir::Value streamValue = computeRegion.getStream()) {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: Creating async gpu.launch with stream: "
+                 << streamValue << "\n");
+      launch = gpu::LaunchOp::create(
+          rewriter, loc, gridDimX, gridDimY, gridDimZ, blockDimX, blockDimY,
+          blockDimZ,
+          /*dynamicSharedMemorySize=*/mlir::Value{},
+          /*asyncTokenType=*/
+          mlir::gpu::AsyncTokenType::get(rewriter.getContext()));
+      // Add the stream as an async dependency
+      launch.getAsyncDependenciesMutable().append(streamValue);
+    } else {
+      LLVM_DEBUG(llvm::dbgs()
+                 << "\nDEBUG: No stream, creating sync gpu.launch\n");
+      launch = gpu::LaunchOp::create(rewriter, loc, gridDimX, gridDimY,
+                                     gridDimZ, blockDimX, blockDimY, blockDimZ);
+    }
+
+    // Transfer kernel function name and module name from acc.compute_region to
+    // gpu.launch if present
+    if (auto kernelFuncName = computeRegion.getKernelFuncNameAttr())
+      launch.setFunctionAttr(kernelFuncName);
+    if (auto kernelModuleName = computeRegion.getKernelModuleNameAttr())
+      launch.setModuleAttr(kernelModuleName);
+
+    rewriter.setInsertionPointToEnd(&launch.getBody().front());
+    gpu::TerminatorOp::create(rewriter, loc);
+    rewriter.setInsertionPointToStart(&launch.getBody().front());
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       gpu::GridDimOp::create(rewriter, loc,
+                                              rewriter.getIndexType(),
+                                              gpu::Dimension::z));
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::x));
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::y));
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       gpu::BlockDimOp::create(rewriter, loc,
+                                               rewriter.getIndexType(),
+                                               gpu::Dimension::z));
+  } else {
+    // Do not create gpu.launch for acc routine and map
+    // to block/thread index and block/grid size instead
+    // of launch arguments, using created maps.
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPointToStart(computeRegion->getBlock());
+    createForAllDimensions(rewriter, loc, threadIdMap, dimensionMap);
+    mapLaunchArguments(gpu::Processor::BlockX,
+                       dimensionMap[gpu::Processor::BlockX]);
+    mapLaunchArguments(gpu::Processor::BlockY,
+                       dimensionMap[gpu::Processor::BlockY]);
+    mapLaunchArguments(gpu::Processor::BlockZ,
+                       dimensionMap[gpu::Processor::BlockZ]);
+    mapLaunchArguments(gpu::Processor::ThreadX,
+                       dimensionMap[gpu::Processor::ThreadX]);
+    mapLaunchArguments(gpu::Processor::ThreadY,
+                       dimensionMap[gpu::Processor::ThreadY]);
+    mapLaunchArguments(gpu::Processor::ThreadZ,
+                       dimensionMap[gpu::Processor::ThreadZ]);
+  }
+
+  // Map input arguments for compute region; we go from an IsolatedFromAbove
+  // operation to gpu.launch which is not IsolatedFromAbove.
+  preparePrivatizeExtentInsOperands();
+  Block *body = computeRegion.getBody();
+  unsigned numLaunchArgs = computeRegion.getLaunchArgs().size();
+  ValueRange inputArgs = computeRegion.getInputArgs();
+  for (unsigned i = numLaunchArgs; i < body->getNumArguments(); ++i)
+    mapping.map(body->getArgument(i), inputArgs[i - numLaunchArgs]);
+
+  assert(computeRegion.getRegion().hasOneBlock() &&
+         "compute region only supports one block region for now");
+  // process all operations inside kernel region
+  for (auto &op : computeRegion.getRegion().getBlocks().front().getOperations())
+    processOp(&op);
+
+  for (auto &parLoop : loopReductions)
+    postprocessLoopReduction(parLoop);
+
+  // Replace combine reloads of a reduction slot with the block-reduced value.
+  // Only when it dominates the reload; otherwise keep the reload.
+  if (!pendingCombineReloads.empty() && launch) {
+    DominanceInfo domInfo(launch);
+    for (auto &[slot, loadOp] : pendingCombineReloads) {
+      llvm::DenseMap<Value, Value>::iterator it =
+          reductionAccumValue.find(slot);
+      if (it == reductionAccumValue.end())
+        continue;
+      if (!domInfo.dominates(it->second, loadOp.getOperation()))
+        continue;
+      rewriter.replaceOp(loadOp, ValueRange{it->second});
+    }
+  }
+
+  if (launch) {
+    const int64_t subgroupSize = options.subgroupSize;
+    const int64_t subgroupAlignMask = subgroupSize - 1;
+
+    // Adjust blockDim.x to be a multiple of subgroupSize. This is required
+    // because:
+    // - Subgroup reductions (gpu.all_reduce) require full subgroups
+    // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
+    bool isShuffleEnabled = false;
+
+    launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+      ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+          mlir::acc::getParDimsAttr(allReduce).getArray();
+      for (auto parDim : parDims) {
+        if (parDim.isThreadX() || parDim.isThreadY()) {
+          // Shuffle are enabled. Need to adjust the ThreadX length.
+          isShuffleEnabled = true;
+          return WalkResult::interrupt();
+        }
+      }
+      return WalkResult::advance();
+    });
+    // Also check if called routines have ThreadY reductions
+    if (!isShuffleEnabled) {
+      launch.walk([&](func::CallOp callOp) -> WalkResult {
+        if (gpu::GPUFuncOp callee =
+                callOp->getParentOfType<ModuleOp>()
+                    .lookupSymbol<gpu::GPUFuncOp>(callOp.getCallee())) {
+          callee.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
+            ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
+                mlir::acc::getParDimsAttr(allReduce).getArray();
+            for (auto parDim : parDims) {
+              if (parDim.isThreadX() || parDim.isThreadY()) {
+                isShuffleEnabled = true;
+                return WalkResult::interrupt();
+              }
+            }
+            return WalkResult::advance();
+          });
+        }
+        return isShuffleEnabled ? WalkResult::interrupt()
+                                : WalkResult::advance();
+      });
+    }
+
+    if (isShuffleEnabled || hasThreadYBarrier) {
+      rewriter.setInsertionPoint(launch);
+
+      Value curBlockDimX = launch.getBlockSizeX();
+      Value curBlockDimY = launch.getBlockSizeY();
+
+      // Emit a report on changing parallelism.
+      accSupport.emitRemark(computeRegion, [&]() {
+        auto getName = [&](Value val) -> std::string {
+          std::string name = accSupport.getVariableName(val);
+          return name.empty() ? "(*)" : name;
+        };
+        std::string blockDimXValStr = getName(curBlockDimX);
+        std::string blockDimYValStr = getName(curBlockDimY);
+        llvm::StringRef kind =
+            isShuffleEnabled ? "Shuffle reduction" : "ThreadY barrier";
+        return (llvm::Twine(kind) +
+                " is generated while adjusting the number of threads into "
+                "groups of " +
+                llvm::Twine(subgroupSize) + ".\n\t" + blockDimXName + ": `" +
+                blockDimXValStr + "` to `((" + blockDimXValStr + " + " +
+                llvm::Twine(subgroupAlignMask) + ") / " +
+                llvm::Twine(subgroupSize) + ") * " + llvm::Twine(subgroupSize) +
+                "`\n" + "\t" + blockDimYName + ": `" + blockDimYValStr +
+                "` to `max(1, (new-" + blockDimXName + " * " + blockDimYValStr +
+                ") / new-" + blockDimXName + ")`")
+            .str();
+      });
+
+      std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
+      std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+
+      // Skip subgroup alignment only when the total thread count is already
+      // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
+      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
+      // blockDim.x to a subgroup is still required so subgroups don't cross
+      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      bool skipAlign = false;
+      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
+          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+        skipAlign = true;
+      }
+
+      // Update both the ThreadX length and the number of ThreadY.
+      // When the original blockDim.x and blockDim.y are compile-time
+      // constants, compute the adjusted dimensions as constants directly so
+      // that the GpuKernelOutliningPass can set `known_block_size` on the
+      // outlined gpu.func.
+      Value newBlockDimX, newBlockDimY;
+      if (constBlockDimX && constBlockDimY) {
+        int64_t bdx = *constBlockDimX;
+        int64_t bdy = *constBlockDimY;
+        int64_t alignedBdx =
+            ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
+        int64_t numThreads = bdx * bdy;
+        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        newBlockDimX =
+            arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
+        newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+      } else {
+        // numThreads = blockDim.x * blockDim.y
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
+        Value cstMask =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
+        Value cstSubgroupSize =
+            arith::ConstantIndexOp::create(rewriter, loc, subgroupSize);
+        Value padded =
+            arith::AddIOp::create(rewriter, loc, curBlockDimX, cstMask);
+        Value subgroupsRequired =
+            arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
+        newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
+                                             cstSubgroupSize);
+        // blockDim.y = max(1, numThreads / blockDim.x)
+        Value quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+        Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+        newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+      }
+
+      if (!skipAlign) {
+        launch.getBlockSizeXMutable().assign(newBlockDimX);
+        launch.getBlockSizeYMutable().assign(newBlockDimY);
+      }
+    }
+  }
+
+  if (hasFailed)
+    return failure();
+
+  if (!sharedMemPrivateVarNames.empty()) {
+    accSupport.emitRemark(computeRegion, [&]() {
+      return (llvm::Twine("GPU shared memory used for ") +
+              llvm::join(sharedMemPrivateVarNames, ","))
+          .str();
+    });
+  }
+
+  rewriter.eraseOp(computeRegion);
+  return success();
+}
+
+/// True when this accumulate is redundant in a nested reduction chain: the
+/// value is a load of the destination memref and a sibling
+/// acc.reduction_combine with block par_dims has already reduced %M across
+/// threads in the block.
+///
+///   %v = memref.load %M[]
+///   acc.reduction_accumulate %v to %M ...
+///   acc.reduction_combine %M into %parent ... {block par_dims}
+///
+/// Lowering the accumulate again would double-count. Detection is structural;
+/// nested reductions into per-thread privates do not match because their
+/// combines are not block-scoped.
+static bool isRedundantChainAccumulate(acc::ReductionAccumulateOp op) {
+  Value memref = op.getMemref();
+  memref::LoadOp loadOp = op.getValue().getDefiningOp<memref::LoadOp>();
+  if (!loadOp || loadOp.getMemRef() != memref)
+    return false;
+  for (Operation *user : memref.getUsers()) {
+    acc::ReductionCombineOp combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp || combineOp.getDestMemref() != memref)
+      continue;
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        })) {
+      return true;
+    }
+  }
+  return false;
+}
+
+std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+          SmallVector<mlir::acc::GPUParallelDimAttr>>
+ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
+                                                    Block *block) {
+  MLIRContext *ctx = computeRegion->getContext();
+  SmallVector<mlir::acc::GPUParallelDimAttr> ancestorParDims =
+      getAncestorParDims(op);
+  // Preserve whether there were any structural ancestor par-dims before
+  // we start augmenting them based on inner uses (e.g. private_local).
+  // This is needed for gang redundancy check - stores to worker-indexed
+  // private_local should not disable redundant gang execution.
+  bool noStructuralAncestorParDims =
+      llvm::none_of(ancestorParDims, [](auto pd) { return !pd.isSeq(); });
+
+  mlir::acc::GPUParallelDimAttr routineParDim;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    routineParDim = getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      mlir::acc::insertParDim(ancestorParDims,
+                              mlir::acc::GPUParallelDimAttr::threadYDim(ctx));
+    }
+    mlir::acc::insertParDim(ancestorParDims,
+                            mlir::acc::GPUParallelDimAttr::blockXDim(ctx));
+  }
+
+  // acc.private_local should use the same par_dims as acc.reduction_accumulate.
+  if (acc::PrivateLocalOp privateLocalOp = dyn_cast<acc::PrivateLocalOp>(op)) {
+    for (Operation *user : privateLocalOp.getResult().getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        if (accumulateOp.getMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               accumulateOp.getParDims().getArray()) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // For decomposed complex reductions, the private_local is consumed
+      // by an acc.reduction_combine{,_region} (no acc.reduction_accumulate
+      // user). Mirror the par_dims so this private_local is treated as the
+      // accumulator at the same parallelism level as a scalar reduction
+      // would be (per-thread, not block-shared).
+      if (acc::ReductionCombineOp combineOp =
+              dyn_cast<acc::ReductionCombineOp>(user)) {
+        if (combineOp.getSrcMemref() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      if (auto combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(user)) {
+        if (combineRegionOp.getSrcVar() == privateLocalOp.getResult()) {
+          for (mlir::acc::GPUParallelDimAttr parDim :
+               getReductionCombineParDims(combineRegionOp)) {
+            mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+    }
+  }
+
+  bool hasBlock = false;
+  for (mlir::acc::GPUParallelDimAttr parDim : ancestorParDims)
+    if (parDim.isAnyBlock())
+      hasBlock = true;
+
+  mlir::acc::GPUParallelDimAttr lowestParDim =
+      mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+  if (block) {
+    block->walk([&](Operation *op) {
+      // Check stores to acc.private_local - add the privatize's par_dims
+      // as active dims so predication is correct for per-worker/gang memory.
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
+          if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
+                  privatizeOp.getParDimsAttr()) {
+            for (auto parDim : parDimsAttr.getArray())
+              mlir::acc::insertParDim(ancestorParDims, parDim);
+          }
+        }
+      }
+      // Consider ACC routine calls; routine calls should be predicated up to
+      // one level above the parallel dimension of the callee.
+      if (CallOpInterface callOp = dyn_cast<CallOpInterface>(op)) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isBlockZ())
+            lowestParDim = parDim;
+          else
+            lowestParDim = parDim.getOneHigher();
+        }
+      }
+      // acc.reduction_combine_region should be predicated with the par_dims of
+      // acc.reduction_accumulate. This is required when using combine between
+      // kernel and loop in combined constructs.
+      if (acc::ReductionCombineOp reductionCombineOp =
+              dyn_cast<acc::ReductionCombineOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      if (acc::ReductionCombineRegionOp combineRegionOp =
+              dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp)) {
+          mlir::acc::removeParDim(ancestorParDims, parDim);
+        }
+      }
+      // An array accumulate reduces across its par_dims via gpu.all_reduce, so
+      // all those threads must execute it - treat them as active (unlike the
+      // scalar accumulate, which is active through its enclosing scf.parallel).
+      if (acc::ReductionAccumulateArrayOp accArrayOp =
+              dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+        for (mlir::acc::GPUParallelDimAttr parDim :
+             accArrayOp.getParDims().getArray()) {
+          mlir::acc::insertParDim(ancestorParDims, parDim);
+        }
+      }
+      return WalkResult::advance();
+    });
+  }
+
+  // Obtain launch dimensions
+  SmallVector<mlir::acc::GPUParallelDimAttr> launchParDims;
+  if (routineParDim) {
+    for (mlir::acc::GPUParallelDimAttr parDim = routineParDim;
+         parDim.getOrder() >= lowestParDim.getOrder();
+         parDim = parDim.getOneLower()) {
+      mlir::acc::insertParDim(launchParDims, parDim);
+    }
+  } else {
+    launchParDims = computeRegion.getLaunchParDims();
+  }
+
+  // Compute dimensions that execute op
+  SmallVector<mlir::acc::GPUParallelDimAttr> activeParDims, inactiveParDims;
+  for (mlir::acc::GPUParallelDimAttr launchParDim : launchParDims) {
+    if (launchParDim.getOrder() < lowestParDim.getOrder())
+      break;
+    if (llvm::find(ancestorParDims, launchParDim) != ancestorParDims.end() ||
+        (launchParDim.isAnyBlock() &&
+         (noStructuralAncestorParDims || hasBlock))) {
+      activeParDims.push_back(launchParDim);
+    } else {
+      inactiveParDims.push_back(launchParDim);
+    }
+  }
+
+  return std::pair{activeParDims, inactiveParDims};
+}
+
+Value ACCCGToGPULowering::emitPredicate(
+    Location loc, SmallVector<mlir::acc::GPUParallelDimAttr> &inactiveParDims) {
+  Value predicate;
+  for (mlir::acc::GPUParallelDimAttr inactiveParDim : inactiveParDims) {
+    Value threadId = getGPUThreadIdFor(inactiveParDim.getProcessor());
+    TypedAttr zeroAttr = rewriter.getZeroAttr(threadId.getType());
+    Value zero = arith::ConstantOp::create(rewriter, loc, zeroAttr);
+    Value cmp = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::eq,
+                                      threadId, zero);
+    if (predicate)
+      predicate = arith::AndIOp::create(rewriter, loc, cmp, predicate);
+    else
+      predicate = cmp;
+  }
+  return predicate;
+}
+
+void ACCCGToGPULowering::createBarrier(
+    Location loc, mlir::acc::GPUParallelDimsAttr parDimsAttr) {
+  bool hasAnyBlock = false, hasThreadY = false, hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock())
+      hasAnyBlock = true;
+    if (parDim.isThreadY())
+      hasThreadY = true;
+    if (parDim.isThreadX())
+      hasThreadX = true;
+  }
+
+  if (hasAnyBlock || hasThreadY)
+    emitGPUBarrierWorkgroup(rewriter, loc);
+  else if (hasThreadX)
+    createPerRowBarrier(loc);
+}
+
+void ACCCGToGPULowering::createPerRowBarrier(Location loc) {
+  hasThreadYBarrier = true;
+
+  if (staticBlockDimX <= options.subgroupSize) {
+    emitGPUBarrierSubgroup(rewriter, loc);
+    return;
+  }
+
+  if (options.deviceType != mlir::acc::DeviceType::Nvidia) {
+    (void)accSupport.emitNYI(
+        loc,
+        "per-row barrier to support worker parallelism on non-NVIDIA device");
+  }
+
+  // Per-row barrier with fully runtime branching.
+  // Three mutually exclusive paths:
+  //   blockDim.y == 1    -> gpu.barrier (workgroup-wide, only one worker)
+  //   blockDim.x <= subgroupSize -> gpu.barrier scope<subgroup>
+  //   blockDim.x > subgroupSize  -> nvvm.barrier (tid.y + 1), blockDim.x
+  //   (named)
+  //
+  // Per-row barriers use tid.y+1 so IDs start at 1, avoiding clash with
+  // barrier 0. When blockDim.y >= 16, worker 15's ID (16) wraps to
+  // physical barrier 0; this is safe because named barriers are reusable
+  // resources - workgroup-wide and per-row barriers on the same physical
+  // barrier execute at 
diff erent program points and never overlap.
+  Value blockDimX = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+  Value blockDimY = gpu::BlockDimOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value isSingleWorker = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::eq, blockDimY, cst1);
+
+  auto outerIf = scf::IfOp::create(rewriter, loc, isSingleWorker,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.y == 1 -> workgroup-wide barrier (safe, only one worker)
+  rewriter.setInsertionPointToStart(&outerIf.getThenRegion().front());
+  emitGPUBarrierWorkgroup(rewriter, loc);
+
+  // Else: blockDim.y > 1 - choose between subgroup sync and named barrier
+  rewriter.setInsertionPointToStart(&outerIf.getElseRegion().front());
+  Value cstSubgroupSize =
+      arith::ConstantIndexOp::create(rewriter, loc, options.subgroupSize);
+  Value isSubgroupSized = arith::CmpIOp::create(
+      rewriter, loc, arith::CmpIPredicate::ule, blockDimX, cstSubgroupSize);
+
+  auto innerIf = scf::IfOp::create(rewriter, loc, isSubgroupSized,
+                                   /*withElseRegion=*/true);
+
+  // Then: blockDim.x <= subgroupSize -> subgroup barrier (one worker per
+  // subgroup)
+  rewriter.setInsertionPointToStart(&innerIf.getThenRegion().front());
+  emitGPUBarrierSubgroup(rewriter, loc);
+
+  // Else: blockDim.x > subgroupSize -> per-row named barrier with tid.y + 1.
+  // The 1024-thread-per-block hardware limit with subgroup-aligned blockDim.x
+  // (>= 64 here) guarantees blockDim.y <= 16, so IDs span at most 16
+  // physical barriers (0-15) with no aliasing across workers.
+  rewriter.setInsertionPointToStart(&innerIf.getElseRegion().front());
+  Value threadYId = gpu::ThreadIdOp::create(
+      rewriter, loc, rewriter.getIndexType(), gpu::Dimension::y);
+  Value barrierId = arith::AddIOp::create(rewriter, loc, threadYId, cst1);
+  Type i32Ty = rewriter.getI32Type();
+  Value barrierId32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, barrierId);
+  Value numberOfThreads32 =
+      arith::IndexCastOp::create(rewriter, loc, i32Ty, blockDimX);
+
+  // GPU dialect named barriers do not have a means to create a custom barrier
+  // id. Thus use nvvm directly.
+  assert(options.deviceType == mlir::acc::DeviceType::Nvidia);
+  NVVM::BarrierOp::create(rewriter, loc, barrierId32, numberOfThreads32);
+
+  rewriter.setInsertionPointAfter(outerIf);
+}
+
+/// Whether any later sibling of \p loopOp (or a loop nested inside one) is a
+/// loop, i.e. whether some subsequent loop in the same region may read what
+/// \p loopOp wrote. Used to skip a barrier after the last loop, where nothing
+/// reads the data afterward.
+static bool hasSubsequentLoopSibling(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    if (isa<scf::ParallelOp, scf::ForOp>(next))
+      return true;
+    bool nested = false;
+    next->walk([&](Operation *op) {
+      if (isa<scf::ParallelOp, scf::ForOp>(op)) {
+        nested = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (nested)
+      return true;
+  }
+  return false;
+}
+
+/// Nearest enclosing sequential loop ancestor of \p op.
+static LoopLikeOpInterface findFirstSequentialLoop(Operation *op) {
+  auto isAllSequentialParDims = [](scf::ParallelOp par) -> bool {
+    mlir::acc::GPUParallelDimsAttr pd = mlir::acc::getParDimsAttr(par);
+    if (!pd || pd.getArray().empty())
+      return false;
+    return llvm::all_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+      return d.isSeq();
+    });
+  };
+
+  for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) {
+    // Do not need to check scf.for op's parents
+    if (isa<scf::ForOp>(p))
+      return cast<LoopLikeOpInterface>(p);
+    if (scf::ParallelOp parOp = dyn_cast<scf::ParallelOp>(p)) {
+      if (isAllSequentialParDims(parOp))
+        return cast<LoopLikeOpInterface>(p);
+    }
+  }
+  return nullptr;
+}
+
+// A sequential loop that uses gang-private shared memory needs a
+// workgroup-wide barrier afterward so every thread in the block observes the
+// same state. When more work still lies between the loop and the next
+// thread-reconvergence point in the same block, that barrier must follow that
+// work; not sit immediately after the loop.
+//
+// The helpers below mark loop-body closure and other reconvergence points
+// where any postponed barrier must be inserted.
+
+/// Marks the end of a loop body's iteration in the current block.
+static bool isLoopBodyClosureOp(Operation *op) {
+  return isa<scf::ReduceOp, scf::YieldOp, acc::YieldOp>(op);
+}
+
+/// Thread-reconvergence point where any postponed post-loop barrier for earlier
+/// loops in this block must be inserted before proceeding.
+static bool isDeferredBarrierFlushPoint(Operation *op) {
+  if (isLoopBodyClosureOp(op))
+    return true;
+  // The next sequential loop may consume shared state produced by the prior
+  // one.
+  if (isa<scf::ForOp>(op))
+    return true;
+  if (scf::ParallelOp parallelOp = dyn_cast<scf::ParallelOp>(op)) {
+    if (mlir::acc::hasParDimsAttr(parallelOp)) {
+      if (mlir::acc::GPUParallelDimsAttr parDims =
+              mlir::acc::getParDimsAttr(parallelOp)) {
+        if (parDims.getArray().size() == 1 &&
+            parDims.getArray().front().isSeq()) {
+          return true;
+        }
+      }
+    }
+  }
+  return false;
+}
+
+/// True when a loop is followed by other work in the same block before the
+/// loop body closes; the post-loop barrier must wait for that reconvergence
+/// point instead of being placed right after the loop.
+static bool hasTrailingSideEffectSiblings(Operation *loopOp) {
+  for (Operation *next = loopOp->getNextNode(); next;
+       next = next->getNextNode()) {
+    return !isLoopBodyClosureOp(next);
+  }
+  return false;
+}
+
+// Insert a barrier after a sequential loop when the surrounding parallel
+// structure requires it, so that block threads observe the loop's writes to
+// gang-/block-private shared memory before a later loop reads them. The barrier
+// must land at an enclosing collective (block- or worker-level) point where all
+// threads converge, so it cannot deadlock.
+//
+// The function walks the parent hierarchy of the loop to decide where (if at
+// all) to place the barrier:
+//   - No enclosing parallel loop (top-level worksharing loop): emit a
+//     workgroup barrier when the loop writes shared memory and a later sibling
+//     loop may read it.
+//   - Nearest scf.parallel parent is itself sequential: a barrier *after* it
+//     would be unsafe when threads have varying iteration counts. However, when
+//     the loop has thread-level sub-loops collaborating on block-private memory
+//     (all threads participate every iteration), walk up to the block-level
+//     ancestor and insert the barrier there.
+//   - Nearest scf.parallel parent is a non-sequential parallel loop: walk up to
+//     the block-level (or worker/thread-y) ancestor and insert the barrier
+//     there, handling gang-redundant init loops and grid-stride remainders.
+void ACCCGToGPULowering::createBarrierAfterSeqLoop(Operation *loopOp) {
+  scf::ParallelOp wsLoop = loopOp->getParentOfType<scf::ParallelOp>();
+  if (!wsLoop) {
+    // loopOp is a worksharing loop at the kernel-body top level (no enclosing
+    // parallel loop). When it writes gang-private shared memory
+    // that a later sibling loop reads, a workgroup barrier is needed between
+    // them. The point just after a top-level loop is uniform (all threads reach
+    // it), so a workgroup-wide barrier here cannot deadlock. Loops writing only
+    // global/thread-private memory, or with no subsequent reader, get none.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(loopOp))
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    return;
+  }
+
+  bool parentIsSeq = false;
+  if (mlir::acc::GPUParallelDimsAttr wsParDims =
+          mlir::acc::getParDimsAttr(wsLoop)) {
+    if (wsParDims.getArray().size() == 1 &&
+        wsParDims.getArray().front().isSeq()) {
+      parentIsSeq = true;
+    }
+  }
+
+  if (parentIsSeq) {
+    // loopOp is nested inside a sequential parent loop.
+    // When the loop contains thread-level sub-loops,
+    // multiple threads collaborate on shared (block-private) memory
+    // within each iteration.  Insert a block-level barrier.
+    bool hasThreadSubLoop = false;
+    loopOp->walk([&](scf::ParallelOp innerPar) -> WalkResult {
+      if (innerPar.getOperation() == loopOp)
+        return WalkResult::advance();
+      if (mlir::acc::GPUParallelDimsAttr dims =
+              mlir::acc::getParDimsAttr(innerPar)) {
+        for (auto d : dims.getArray()) {
+          if (d.isThreadX() || d.isThreadY()) {
+            hasThreadSubLoop = true;
+            return WalkResult::interrupt();
+          }
+        }
+      }
+      return WalkResult::advance();
+    });
+    if (!hasThreadSubLoop)
+      return;
+    scf::ParallelOp threadLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+    if (!threadLoop)
+      return;
+    scf::ParallelOp blockLoop = threadLoop->getParentOfType<scf::ParallelOp>();
+    if (!blockLoop)
+      return;
+    mlir::acc::GPUParallelDimsAttr parDimsAttr =
+        mlir::acc::getParDimsAttr(blockLoop);
+    if (parDimsAttr.hasOnlyBlockLevel())
+      createBarrier(loopOp->getLoc(), parDimsAttr);
+    return;
+  }
+
+  // Parent is a non-sequential parallel loop.  Walk up to find the block-level
+  // ancestor and insert a barrier there.
+  scf::ParallelOp seqLoop = wsLoop->getParentOfType<scf::ParallelOp>();
+  if (!seqLoop) {
+    // wsLoop is a worksharing loop directly under the compute region with no
+    // gang ancestor: a gang-redundant init loop (e.g. a thread-level loop that
+    // every gang runs to fill its own copy of gang-private shared memory). If
+    // it writes gang-private (block-only) memory that a later sibling loop
+    // reads, insert a workgroup barrier between them. The barrier sits just
+    // after the top-level loop where all threads converge, so it cannot
+    // deadlock.
+    if (mayWriteSharedMemory(loopOp) && hasSubsequentLoopSibling(wsLoop))
+      emitGPUBarrierWorkgroup(rewriter, loopOp->getLoc());
+    return;
+  }
+  if (scf::ParallelOp outerParLoop =
+          seqLoop->getParentOfType<scf::ParallelOp>()) {
+    mlir::acc::GPUParallelDimsAttr parDimsAttr =
+        mlir::acc::getParDimsAttr(outerParLoop);
+    if (parDimsAttr.hasOnlyBlockLevel()) {
+      createBarrier(loopOp->getLoc(), parDimsAttr);
+    } else if (parDimsAttr.hasOnlyThreadYLevel()) {
+      createPerRowBarrier(loopOp->getLoc());
+    } else if (parDimsAttr && parDimsAttr.isSeq()) {
+      // outerParLoop is a sequential grid-stride remainder of a partitioned
+      // gang loop, not the gang. Walk past the remainder(s) to the block-level
+      // gang and barrier there.
+      for (Operation *gangLoop =
+               outerParLoop->getParentOfType<scf::ParallelOp>();
+           gangLoop; gangLoop = gangLoop->getParentOfType<scf::ParallelOp>()) {
+        mlir::acc::GPUParallelDimsAttr gangDims =
+            mlir::acc::getParDimsAttr(gangLoop);
+        if (!gangDims)
+          break;
+        if (gangDims.hasOnlyBlockLevel()) {
+          createBarrier(loopOp->getLoc(), gangDims);
+          break;
+        }
+        if (!gangDims.isSeq())
+          break;
+      }
+    }
+    return;
+  }
+
+  // No block-level ancestor above seqLoop: the gang loop is seqLoop itself, a
+  // gang(+vector) loop directly under the compute region. The fixed-depth
+  // lookup above misses it (issue: a gang+vector init loop that writes
+  // gang-private shared memory must be followed by a barrier before another
+  // loop reads it). Only needed when the loop writes such shared memory; loops
+  // writing global/thread-private memory (e.g. a plain gang-vector saxpy) do
+  // not need one here.
+  mlir::acc::GPUParallelDimsAttr parDimsAttr =
+      mlir::acc::getParDimsAttr(seqLoop);
+  if (parDimsAttr && parDimsAttr.hasOnlyBlockLevel() &&
+      mayWriteSharedMemory(loopOp)) {
+    createBarrier(loopOp->getLoc(), parDimsAttr);
+  }
+}
+
+bool ACCCGToGPULowering::mayWriteSharedMemory(Operation *loopOp) {
+  bool found = false;
+  loopOp->walk([&](memref::StoreOp storeOp) {
+    // Trace the store target back to its backing acc.private_local through
+    // view/cast/box ops, then check the privatization is gang-level.
+    llvm::SmallVector<Value, 8> worklist{storeOp.getMemref()};
+    llvm::SmallPtrSet<Value, 8> seen;
+    while (!worklist.empty()) {
+      Value v = worklist.pop_back_val();
+      if (!seen.insert(v).second)
+        continue;
+      Operation *def = v.getDefiningOp();
+      if (!def)
+        continue;
+      if (acc::PrivateLocalOp privateLocal =
+              dyn_cast<acc::PrivateLocalOp>(def)) {
+        acc::PrivatizeOp privatizeOp =
+            getPrivatizeOp(privateLocal, computeRegion);
+        // Only gang-level (block, no thread) private memory is a single copy
+        // shared across the workgroup's threads, so a write needs a workgroup
+        // barrier before another thread reads it. A [block_x, thread_x]
+        // privatization is thread-private (one copy per thread); barriering on
+        // it would deadlock when threads take 
diff erent grid-stride iteration
+        // counts.
+        if (mlir::acc::GPUParallelDimsAttr parDims =
+                privatizeOp.getParDimsAttr()) {
+          bool hasBlock = false, hasThread = false;
+          for (mlir::acc::GPUParallelDimAttr d : parDims.getArray()) {
+            if (d.isAnyBlock())
+              hasBlock = true;
+            if (d.isThreadX() || d.isThreadY())
+              hasThread = true;
+          }
+          if (hasBlock && !hasThread) {
+            found = true;
+            return WalkResult::interrupt();
+          }
+        }
+        continue;
+      }
+      worklist.append(def->getOperands().begin(), def->getOperands().end());
+    }
+    return WalkResult::advance();
+  });
+  return found;
+}
+
+PrivateMemScope
+ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
+  bool hasBlock = false;
+  bool hasThreadX = false;
+  bool hasThreadY = false;
+  if (mlir::acc::GPUParallelDimsAttr parDims = privatizeOp.getParDimsAttr()) {
+    for (mlir::acc::GPUParallelDimAttr d : parDims.getArray()) {
+      if (d.isAnyBlock())
+        hasBlock = true;
+      if (d.isThreadX())
+        hasThreadX = true;
+      if (d.isThreadY())
+        hasThreadY = true;
+    }
+  } else {
+    for (mlir::acc::GPUParallelDimAttr d : computeRegion.getLaunchParDims())
+      if (d.isAnyBlock())
+        hasBlock = true;
+    if (hasBlock)
+      return PrivateMemScope::Gang;
+    return PrivateMemScope::Thread;
+  }
+  if (hasThreadX)
+    return PrivateMemScope::Thread;
+  if (hasBlock && hasThreadY)
+    return PrivateMemScope::Worker;
+  if (hasBlock)
+    return PrivateMemScope::Gang;
+  return PrivateMemScope::Thread;
+}
+
+/// Walks back from a memref use to its defining `acc.private_local`, if any.
+static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
+  llvm::SmallVector<Value, 8> worklist{memref};
+  llvm::SmallPtrSet<Value, 8> seen;
+  while (!worklist.empty()) {
+    Value v = worklist.pop_back_val();
+    if (!seen.insert(v).second)
+      continue;
+    Operation *def = v.getDefiningOp();
+    if (!def)
+      continue;
+    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
+      return privateLocal;
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
+  }
+  return nullptr;
+}
+
+PrivateMemScope ACCCGToGPULowering::getPrivateScopeForMemref(Value memref) {
+  if (auto privateLocal = getPrivateLocalForMemref(memref))
+    return getPrivateMemScope(getPrivatizeOp(privateLocal, computeRegion));
+  return PrivateMemScope::None;
+}
+
+acc::PrivatizeOp ACCCGToGPULowering::getPrivatizeForMemref(Value memref) {
+  if (auto privateLocal = getPrivateLocalForMemref(memref))
+    return getPrivatizeOp(privateLocal, computeRegion);
+  return acc::PrivatizeOp();
+}
+
+PrivateMemScope
+ACCCGToGPULowering::needsPreStoreReuseBarrier(acc::PredicateRegionOp interOp) {
+
+  // Check if we need a pre-predicate barrier first.
+  // Next, check if the barrier should be gang- or worker-level.
+  LoopLikeOpInterface seqLoopOp = findFirstSequentialLoop(interOp);
+  if (!seqLoopOp)
+    return PrivateMemScope::None;
+
+  // Check if any op in the predicate region stores to gang- or worker-private
+  // memory. If not, no barrier is needed.
+  PrivateMemScope storeScope = PrivateMemScope::None;
+  llvm::SmallPtrSet<Operation *, 4> storePrivatizes;
+  interOp.getRegion().walk([&](memref::StoreOp storeOp) {
+    PrivateMemScope scope = getPrivateScopeForMemref(storeOp.getMemref());
+    if (scope != PrivateMemScope::Gang && scope != PrivateMemScope::Worker)
+      return WalkResult::advance();
+    if (auto privatize = getPrivatizeForMemref(storeOp.getMemref()))
+      storePrivatizes.insert(privatize.getOperation());
+    if (storeScope == PrivateMemScope::None)
+      storeScope = scope;
+    return WalkResult::advance();
+  });
+  if (storeScope == PrivateMemScope::None || storePrivatizes.empty())
+    return PrivateMemScope::None;
+
+  // Check that there is a subsequent parallel region that uses private memory
+  bool hasParallelPrivateUse = false;
+  seqLoopOp.getOperation()->walk([&](Operation *op) {
+    // Ignore loads inside the predicate.
+    if (interOp->isAncestor(op))
+      return WalkResult::advance();
+
+    Value memref;
+    if (memref::LoadOp loadOp = dyn_cast<memref::LoadOp>(op))
+      memref = loadOp.getMemref();
+    else if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op))
+      memref = storeOp.getMemref();
+    else
+      return WalkResult::advance();
+
+    PrivateMemScope scope = getPrivateScopeForMemref(memref);
+    if (scope != storeScope)
+      return WalkResult::advance();
+
+    acc::PrivatizeOp usePrivatize = getPrivatizeForMemref(memref);
+    if (!usePrivatize || !storePrivatizes.contains(usePrivatize.getOperation()))
+      return WalkResult::advance();
+
+    bool insideNestedParallel = false;
+    for (Operation *p = op->getParentOp(); p && p != seqLoopOp.getOperation();
+         p = p->getParentOp()) {
+      if (scf::ParallelOp par = dyn_cast<scf::ParallelOp>(p)) {
+        if (mlir::acc::GPUParallelDimsAttr pd =
+                mlir::acc::getParDimsAttr(par)) {
+          if (llvm::any_of(pd.getArray(), [](mlir::acc::GPUParallelDimAttr d) {
+                return !d.isSeq();
+              })) {
+            insideNestedParallel = true;
+            break;
+          }
+        }
+      }
+    }
+    if (!insideNestedParallel)
+      return WalkResult::advance();
+    // An inner parallel region uses the same private memory.
+    hasParallelPrivateUse = true;
+    return WalkResult::interrupt();
+  });
+
+  if (!hasParallelPrivateUse)
+    return PrivateMemScope::None;
+
+  return storeScope;
+}
+
+void ACCCGToGPULowering::processPredicateRegion(
+    acc::PredicateRegionOp interOp) {
+  LLVM_DEBUG(llvm::dbgs() << "processing predicate region: ";
+             interOp->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Location loc = interOp->getLoc();
+
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+      parDimsPair = computeActiveAndInactiveParDims(
+          interOp, &interOp.getRegion().front());
+
+  // If ThreadY reduction exists, subgroup alignment is applied
+  // (blockDim.x = subgroupSize), so ThreadX lanes exist even without explicit
+  // ThreadX parallelism. Add ThreadX to inactiveParDims if not already present.
+  // Exception: if this region contains a thread-level (vector or worker)
+  // routine call, all ThreadX threads must reach the call so the routine's
+  // workgroup-wide barriers (e.g. shared memory alloca sync) converge.
+  if (hasThreadYReduction) {
+    MLIRContext *ctx = computeRegion->getContext();
+    mlir::acc::GPUParallelDimAttr threadXParDim =
+        mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
+    bool hasThreadXInActive =
+        llvm::any_of(parDimsPair.first, [](mlir::acc::GPUParallelDimAttr pd) {
+          return pd.isThreadX();
+        });
+    bool hasThreadXInInactive =
+        llvm::any_of(parDimsPair.second, [](mlir::acc::GPUParallelDimAttr pd) {
+          return pd.isThreadX();
+        });
+
+    // Check if THIS predicate region contains a thread-level routine call.
+    // We use the pre-computed hasThreadLevelRoutineCall as an early-out
+    // optimization.
+    bool regionHasThreadLevelRoutineCall = false;
+    if (hasThreadLevelRoutineCall) {
+      interOp.getRegion().walk([&](CallOpInterface callOp) {
+        if (mlir::acc::GPUParallelDimAttr parDim =
+                getAccRoutineCallParDim(callOp, defaultPolicy)) {
+          if (parDim.isThreadX() || parDim.isThreadY()) {
+            regionHasThreadLevelRoutineCall = true;
+            return WalkResult::interrupt();
+          }
+        }
+        return WalkResult::advance();
+      });
+    }
+
+    if (!hasThreadXInActive && !hasThreadXInInactive &&
+        !regionHasThreadLevelRoutineCall) {
+      parDimsPair.second.push_back(threadXParDim);
+    }
+  }
+
+  if (Value predicate = emitPredicate(loc, parDimsPair.second)) {
+    LLVM_DEBUG(llvm::dbgs() << "predicate: " << predicate << "\n");
+    bool isInsideThreadXLoop = false;
+    bool isInsideThreadYLoop = false;
+    for (auto parDim : parDimsPair.first) {
+      if (parDim.isThreadX())
+        isInsideThreadXLoop = true;
+      if (parDim.isThreadY())
+        isInsideThreadYLoop = true;
+    }
+    // Emits the reconvergence barrier matching this region's predication level,
+    // at the current insertion point. Called for both before and after the
+    // predicated store. Below is the pre-predicate barrier; the post-predicate
+    // barrier is emitted after the ifOp.
+    auto emitReconvergenceBarrier = [&]() {
+      if (isInsideThreadXLoop) {
+        // Inside ThreadX loop - skip barrier
+      } else if (isInsideThreadYLoop) {
+        // Inside ThreadY loop
+        if (!isInsideACCSpecializedRoutine(computeRegion)) {
+          // Add barrier if ThreadX is predicated (lane 0 writes must be
+          // visible to all lanes before they read).
+          bool predicatesThreadX = llvm::any_of(
+              parDimsPair.second,
+              [](mlir::acc::GPUParallelDimAttr pd) { return pd.isThreadX(); });
+          if (predicatesThreadX) {
+            createBarrier(loc, mlir::acc::GPUParallelDimsAttr::get(
+                                   interOp->getContext(), parDimsPair.second));
+          }
+        }
+        // For acc routine ThreadY routines, skip barrier
+      } else if (!parDimsPair.first.empty()) {
+        // Inside block loop
+        createBarrier(loc, mlir::acc::GPUParallelDimsAttr::get(
+                               interOp->getContext(), parDimsPair.first));
+      } else {
+        // Top level
+        createBarrier(loc, mlir::acc::GPUParallelDimsAttr::get(
+                               interOp->getContext(), parDimsPair.second));
+      }
+    };
+
+    // A gang-level (block, no thread) shared slot written here and reused on
+    // the next iteration of an enclosing sequential loop must not be
+    // overwritten before all threads read the current value. Emit the
+    // reconvergence barrier before the store too (the post-store barrier below
+    // only orders this iteration's store->read). Restricted to the block-level
+    // case: such loops are run uniformly by every workgroup thread, so the
+    // workgroup barrier cannot deadlock; worker/vector (thread_y/thread_x)
+    // loops may have divergent trip counts.
+    PrivateMemScope scope = needsPreStoreReuseBarrier(interOp);
+    if (scope == PrivateMemScope::Gang)
+      emitGPUBarrierWorkgroup(rewriter, loc);
+    else if (scope == PrivateMemScope::Worker)
+      createPerRowBarrier(loc);
+
+    auto ifOp = scf::IfOp::create(rewriter, loc, predicate,
+                                  /*withElseRegion=*/false);
+    Region &thenRegion = ifOp.getThenRegion();
+    Block &thenBlock = thenRegion.back();
+    rewriter.setInsertionPoint(thenBlock.getTerminator());
+    // Ops in a predicate region may need to be further processed, recurse
+    for (auto &bodyOp : interOp.getRegion().front().getOperations()) {
+      // If the store's value loads from a block-level reduction
+      // memref, convert to atomic for cross-block correctness.
+      // Only at kernel top level (no active thread dims).
+      if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(&bodyOp)) {
+        std::optional<arith::AtomicRMWKind> blockReduceKind;
+        bool failedReductionKind = false;
+        Value storeVal = storeOp.getValueToStore();
+        if (storeVal.getDefiningOp()) {
+          // Walk the epilogue def-chain within the enclosing block to find
+          // the block-level accumulate load feeding this store.  Epilogue ops
+          // (type conversions, arithmetic, etc.) are traversed transparently.
+          // Non-acc loads and values defined outside the block are treated as
+          // loop-invariant and return nullopt, bounding the search naturally.
+          Block *epilogueBlock = interOp->getBlock();
+          auto findBlockAccLoad =
+              [&](auto &self,
+                  Value val) -> std::optional<arith::AtomicRMWKind> {
+            Operation *def = val.getDefiningOp();
+            if (!def || def->getBlock() != epilogueBlock)
+              return std::nullopt;
+            if (memref::LoadOp loadOp = dyn_cast<memref::LoadOp>(def)) {
+              for (auto *user : loadOp.getMemRef().getUsers()) {
+                if (acc::ReductionAccumulateOp accOp =
+                        dyn_cast<acc::ReductionAccumulateOp>(user)) {
+                  if (llvm::any_of(accOp.getParDims().getArray(),
+                                   [](mlir::acc::GPUParallelDimAttr pd) {
+                                     return pd.isAnyBlock();
+                                   })) {
+                    FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
+                        accOp.getReductionOperator(),
+                        accOp.getValue().getType(), accOp.getLoc());
+                    if (failed(kind)) {
+                      failedReductionKind = true;
+                      return std::nullopt;
+                    }
+                    return *kind;
+                  }
+                }
+              }
+              return std::nullopt;
+            }
+            for (Value operand : def->getOperands())
+              if (auto kind = self(self, operand))
+                return kind;
+            return std::nullopt;
+          };
+          blockReduceKind = findBlockAccLoad(findBlockAccLoad, storeVal);
+        }
+        if (failedReductionKind)
+          return;
+        if (blockReduceKind) {
+          Value input = mapping.lookupOrDefault(storeOp.getValueToStore());
+          Value memref = mapping.lookupOrDefault(storeOp.getMemref());
+          bool threadIsActive = llvm::any_of(
+              parDimsPair.first, [](mlir::acc::GPUParallelDimAttr pd) {
+                return !pd.isAnyBlock();
+              });
+          if (!threadIsActive &&
+              !isa_and_nonnull<memref::AllocaOp>(
+                  unwrapMemRefConversion(memref).getDefiningOp())) {
+            // Initialize the destination to the reduction identity before
+            // cross-block atomics so that the final result reflects pure
+            // assignment semantics (e.g. r = sum(a)), not accumulation
+            // on top of the pre-kernel value.
+            if (launch) {
+              MemRefType memrefTy = cast<MemRefType>(memref.getType());
+              // Map the store indices for ranked memrefs.
+              SmallVector<Value> initIndices;
+              for (Value idx : storeOp.getIndices())
+                initIndices.push_back(mapping.lookupOrDefault(idx));
+
+              OpBuilder::InsertionGuard guard(rewriter);
+              Block &launchBody = launch.getBody().front();
+              Operation *insertBefore = nullptr;
+
+              launchBody.walk([&](scf::ParallelOp parOp) -> WalkResult {
+                for (Operation *parent = parOp->getParentOp(); parent;
+                     parent = parent->getParentOp()) {
+                  if (parent == launch.getOperation())
+                    break;
+                  if (isa<scf::ParallelOp>(parent))
+                    return WalkResult::advance();
+                }
+                insertBefore = parOp.getOperation();
+                return WalkResult::interrupt();
+              });
+              if (insertBefore)
+                rewriter.setInsertionPoint(insertBefore);
+              else
+                rewriter.setInsertionPointToStart(&launchBody);
+              // Recursively re-materialize operations whose definitions
+              // do not dominate the insertion point.  A single-level clone
+              // is insufficient when the value is produced by a chain of
+              // operations (e.g. reinterpret_cast depending on box_dims,
+              // divsi, convert, etc.) that are all defined after the
+              // insertion point.
+              DominanceInfo domInfo(launch);
+              IRMapping initMapping;
+              std::function<Value(Value)> materialize =
+                  [&](Value val) -> Value {
+                Operation *defOp = val.getDefiningOp();
+                if (!defOp)
+                  return val;
+                if (domInfo.dominates(defOp, &*rewriter.getInsertionPoint()))
+                  return val;
+                if (auto mapped = initMapping.lookupOrNull(val))
+                  return mapped;
+                // Recurse on operands; the recursive call seeds
+                // initMapping for any operand it clones, which the
+                // subsequent rewriter.clone(..., initMapping) picks up.
+                for (Value operand : defOp->getOperands())
+                  materialize(operand);
+                Operation *cloned = rewriter.clone(*defOp, initMapping);
+                for (auto [orig, clonedRes] :
+                     llvm::zip(defOp->getResults(), cloned->getResults())) {
+                  initMapping.map(orig, clonedRes);
+                }
+                return initMapping.lookup(val);
+              };
+              Value initMemref = materialize(memref);
+              for (auto &idx : initIndices)
+                idx = materialize(idx);
+              Value identityVal = createIdentityValue(
+                  rewriter, loc, memrefTy.getElementType(), *blockReduceKind,
+                  /*useOnlyFiniteValue=*/true);
+              Value blockId = gpu::BlockIdOp::create(
+                  rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+              Value threadId = gpu::ThreadIdOp::create(
+                  rewriter, loc, rewriter.getIndexType(), gpu::Dimension::x);
+              Value zero = arith::ConstantIndexOp::create(rewriter, loc, 0);
+              Value isBlock0 = arith::CmpIOp::create(
+                  rewriter, loc, arith::CmpIPredicate::eq, blockId, zero);
+              Value isThread0 = arith::CmpIOp::create(
+                  rewriter, loc, arith::CmpIPredicate::eq, threadId, zero);
+              Value isFirstThread =
+                  arith::AndIOp::create(rewriter, loc, isBlock0, isThread0);
+              auto initIf = scf::IfOp::create(rewriter, loc, isFirstThread,
+                                              /*withElseRegion=*/false);
+              rewriter.setInsertionPoint(
+                  initIf.getThenRegion().back().getTerminator());
+              memref::StoreOp::create(rewriter, loc, identityVal, initMemref,
+                                      initIndices);
+              rewriter.setInsertionPointAfter(initIf);
+              gpu::BarrierOp::create(rewriter, loc);
+            }
+            SmallVector<Value> atomicIndices;
+            for (Value idx : storeOp.getIndices())
+              atomicIndices.push_back(mapping.lookupOrDefault(idx));
+            constructAtomicAccumulation(loc, memref, atomicIndices, input,
+                                        *blockReduceKind);
+            continue;
+          }
+        }
+      }
+      processOp(&bodyOp);
+    }
+    rewriter.setInsertionPointAfter(ifOp);
+    emitReconvergenceBarrier();
+  } else {
+    // Ops in a predicate region may need to be further processed, recurse
+    for (auto &bodyOp : interOp.getRegion().front().getOperations())
+      processOp(&bodyOp);
+  }
+}
+
+// clang-format off
+//
+// Allocates private storage and broadcasts its pointer through a shared
+// memref-of-memref slot, using the same predication utility as predicate
+// regions.
+//
+//      %0 = arith.cmpi eq, %thread_id_y, %c0 : index
+//      %1 = arith.cmpi eq, %thread_id_x, %c0 : index
+//      %2 = arith.andi %1, %0 : i1
+//      scf.if %2 {
+//        %alloc = memref.alloc() : memref<10xi32>
+//        memref.store %alloc, %arg1[] : memref<memref<10xi32>, #gpu.address_space<workgroup>>
+//      }
+//      gpu.barrier scope<subgroup>
+//      %4 = memref.load %arg1[] : memref<memref<10xi32>, #gpu.address_space<workgroup>>
+//
+// clang-format on
+
+Value ACCCGToGPULowering::processPrivatize(acc::PrivatizeOp privatize) {
+  LLVM_DEBUG(llvm::dbgs() << "processing privatize: ";
+             privatize->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Value tracked = privatize.getResult();
+  if (acc::ComputeRegionOp insUser =
+          dyn_cast<acc::ComputeRegionOp>(getOnlyUser(tracked))) {
+    assert(privatize->hasOneUse() &&
+           "expected acc.privatize op to have one use");
+    tracked = insUser.getBody()->getArgument(
+        privatize->use_begin()->getOperandNumber());
+  }
+  Operation *privatizeUser = getOnlyUser(tracked);
+  assert(privatizeUser && "expected PrivateLocalOp user for privatize");
+
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+      parDimsPair = computeActiveAndInactiveParDims(privatizeUser, nullptr);
+  // Set `par_dims` only when this `acc.privatize` does not already carry it.
+  if (!privatize.getParDimsAttr()) {
+    privatize.setParDimsAttr(mlir::acc::GPUParallelDimsAttr::get(
+        rewriter.getContext(), parDimsPair.first));
+  }
+
+  Location loc = privatize->getLoc();
+  acc::PrivateType privTy = cast<acc::PrivateType>(privatize.getType());
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  MemRefType baseTy = getPrivateBaseMemRefType(privTy.getBaseTy(), module);
+
+  gpu::GPUFuncOp gpuFuncOp = computeRegion->getParentOfType<gpu::GPUFuncOp>();
+  // acc.privatize is outside this compute_region (e.g. passed via ins).
+  // Leave the op unchanged here; processPrivateLocal materializes
+  // storage when acc.private_local is lowered.
+  if (!gpuFuncOp &&
+      privatize->getParentOfType<acc::ComputeRegionOp>() != computeRegion) {
+    return privatize.getResult();
+  }
+
+  for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    if (parDim.isThreadX() &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
+      auto alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+      mapping.map(privatize.getResult(), alloca.getResult());
+      return alloca.getResult();
+    }
+  }
+
+  if (!gpuFuncOp)
+    return privatize.getResult();
+
+  // When ThreadY is active, shared memory must be indexed by ThreadY ID to
+  // avoid races between ThreadY threads.
+  // For ThreadX acc routines, assume ThreadY may be active since the routine
+  // can be called from a ThreadY loop at runtime.
+  // For ThreadY/block acc routines, do not force ThreadY indexing - variables
+  // outside the ThreadY loop should be shared across ThreadY threads.
+  bool threadYIsActive =
+      llvm::any_of(parDimsPair.first, [](mlir::acc::GPUParallelDimAttr parDim) {
+        return parDim.isThreadY();
+      });
+  // For routines with multiple ThreadY threads, need a workgroup barrier
+  // instead of a subgroup barrier.
+  // - ThreadX routines: called independently by 
diff erent ThreadY threads, need
+  //   per-ThreadY slots (override threadYIsActive)
+  // - ThreadY routines: workgroup barrier, but respect parDimsPair for
+  // threadYIsActive
+  //   (variables before ThreadY loop are shared, inside are per-ThreadY)
+  // - Block routines: single call, ThreadY threads cooperate within the
+  // routine, so
+  //   variables at routine level are shared across ThreadY threads (single
+  //   slot)
+  bool needsWorkgroupBarrier = false;
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    FunctionOpInterface funcOp =
+        computeRegion->getParentOfType<FunctionOpInterface>();
+    mlir::acc::GPUParallelDimAttr routineParDim =
+        getSpecializedRoutineDim(funcOp, defaultPolicy);
+    if (routineParDim.isThreadX()) {
+      // ThreadX routine: always per-ThreadY slots since called from ThreadY
+      // loops
+      threadYIsActive = true;
+    } else if (routineParDim.isThreadY()) {
+      // ThreadY routine: workgroup barrier, but keep original threadYIsActive
+      // (variables before ThreadY loop shared, inside per-ThreadY)
+      needsWorkgroupBarrier = true;
+    } else if (routineParDim.isAnyBlock()) {
+      needsWorkgroupBarrier = true;
+    }
+  }
+
+  llvm::SmallVector<Value> mappedDynamicSizes;
+  for (auto dynamicSize : privatize.getDynamicSizes()) {
+    Value mappedDynamicSize = mapping.lookupOrDefault(dynamicSize);
+    mappedDynamicSizes.push_back(mappedDynamicSize);
+  }
+  if (isInsideACCSpecializedRoutine(computeRegion) &&
+      computeRegion.isEffectivelySerial()) {
+    if (mappedDynamicSizes.empty()) {
+      // Static sizes: use alloca (stack allocation)
+      auto alloca =
+          memref::AllocaOp::create(rewriter, privatize->getLoc(), baseTy);
+      mapping.map(privatize.getResult(), alloca.getResult());
+      return alloca.getResult();
+    }
+    // Dynamic sizes: use alloc (heap allocation) with dealloc
+    auto alloc = memref::AllocOp::create(rewriter, privatize->getLoc(), baseTy,
+                                         mappedDynamicSizes);
+
+    // Insert dealloc (free) before the function return
+    OpBuilder::InsertPoint currentInsertPoint = rewriter.saveInsertionPoint();
+    Block &parentBlock = *alloc->getBlock();
+    if (parentBlock.mightHaveTerminator()) {
+      rewriter.setInsertionPoint(parentBlock.getTerminator());
+      memref::DeallocOp::create(rewriter, privatize->getLoc(), alloc);
+    }
+    rewriter.restoreInsertionPoint(currentInsertPoint);
+
+    mapping.map(privatize.getResult(), alloc.getResult());
+    return alloc.getResult();
+  }
+
+  // Predication - when threadYIsActive, don't predicate on ThreadY dimension
+  // since each ThreadY needs to execute the allocation for its own slot
+  SmallVector<mlir::acc::GPUParallelDimAttr> predicateDims;
+  for (auto parDim : parDimsPair.second) {
+    // Skip ThreadY if threadYIsActive - each ThreadY needs to allocate
+    if (threadYIsActive && parDim.isThreadY())
+      continue;
+    predicateDims.push_back(parDim);
+  }
+  Value predicate = emitPredicate(loc, predicateDims);
+  if (!predicate) {
+    predicate = arith::ConstantOp::create(
+        rewriter, loc, rewriter.getIntegerAttr(rewriter.getI1Type(), 1));
+  }
+  auto ifOp = scf::IfOp::create(rewriter, loc, predicate,
+                                /*withElseRegion=*/false);
+  Region &thenRegion = ifOp.getThenRegion();
+  Block &thenBlock = thenRegion.back();
+  rewriter.setInsertionPoint(thenBlock.getTerminator());
+  auto mem = memref::AllocOp::create(rewriter, privatize->getLoc(), baseTy,
+                                     mappedDynamicSizes);
+  // Shared memory allocation
+  gpu::AddressSpaceAttr sharedMemoryAddressSpace = gpu::AddressSpaceAttr::get(
+      computeRegion->getContext(), gpu::GPUDialect::getWorkgroupAddressSpace());
+  // When ThreadY is active, create a shared memory array indexed by ThreadY ID.
+  // Each ThreadY stores to its own slot; ThreadX lanes within a ThreadY share
+  // it.
+  constexpr int64_t kMaxThreadY = 32;
+  MemRefType sharedMemTy =
+      threadYIsActive
+          ? MemRefType::get({kMaxThreadY}, baseTy, MemRefLayoutAttrInterface{},
+                            sharedMemoryAddressSpace)
+          : MemRefType::get({}, baseTy, MemRefLayoutAttrInterface{},
+                            sharedMemoryAddressSpace);
+  // The slot only transiently broadcasts the storage pointer, so reuse one per
+  // type across privatizes, barriering (before the predicated store) on reuse.
+  bool reuseBroadcast = !gpuFuncOp.isKernel();
+  Value alloca;
+  llvm::DenseMap<Type, Value>::iterator cachedSlot =
+      reuseBroadcast ? privatizeBroadcastCache.find(sharedMemTy)
+                     : privatizeBroadcastCache.end();
+  if (reuseBroadcast && cachedSlot != privatizeBroadcastCache.end()) {
+    alloca = cachedSlot->second;
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPoint(ifOp);
+    mlir::acc::GPUParallelDimAttr dim =
+        needsWorkgroupBarrier
+            ? mlir::acc::GPUParallelDimAttr::threadYDim(rewriter.getContext())
+            : mlir::acc::GPUParallelDimAttr::threadXDim(rewriter.getContext());
+    createBarrier(
+        loc, mlir::acc::GPUParallelDimsAttr::get(rewriter.getContext(), {dim}));
+  } else {
+    alloca = gpuFuncOp.addWorkgroupAttribution(sharedMemTy,
+                                               rewriter.getUnknownLoc());
+    // Setting the alignment to 16 because of a bug in the gpu toolchain.
+    // The default alignment is 8, but optimizations create a packed store of 16
+    // bytes which cause a misalignment error at runtime.
+    unsigned index = gpuFuncOp.getNumWorkgroupAttributions() - 1;
+    gpuFuncOp.setWorkgroupAttributionAttr(index,
+                                          LLVM::LLVMDialect::getAlignAttrName(),
+                                          rewriter.getI32IntegerAttr(16));
+    if (reuseBroadcast)
+      privatizeBroadcastCache[sharedMemTy] = alloca;
+  }
+  // Store to shared memory, indexed by ThreadY ID when ThreadY is active.
+  if (threadYIsActive) {
+    Value threadYId = getThreadId(loc, gpu::Dimension::y);
+    memref::StoreOp::create(rewriter, privatize->getLoc(), mem, alloca,
+                            ValueRange{threadYId});
+  } else {
+    memref::StoreOp::create(rewriter, privatize->getLoc(), mem, alloca);
+  }
+
+  // Sync and load - use workgroup barrier for Block/ThreadY routines,
+  // ThreadX barrier for ThreadX-only routines
+  rewriter.setInsertionPointAfter(ifOp);
+  if (needsWorkgroupBarrier) {
+    mlir::acc::GPUParallelDimsAttr threadYDimsAttr =
+        mlir::acc::GPUParallelDimsAttr::get(
+            rewriter.getContext(),
+            {mlir::acc::GPUParallelDimAttr::threadYDim(rewriter.getContext())});
+    createBarrier(loc, threadYDimsAttr);
+  } else {
+    // ThreadX-only: use per-ThreadY barrier
+    mlir::acc::GPUParallelDimsAttr threadXDimsAttr =
+        mlir::acc::GPUParallelDimsAttr::get(
+            rewriter.getContext(),
+            {mlir::acc::GPUParallelDimAttr::threadXDim(rewriter.getContext())});
+    createBarrier(loc, threadXDimsAttr);
+  }
+  // Load from shared memory, indexed by ThreadY ID when ThreadY is active.
+  Value load;
+  if (threadYIsActive) {
+    Value threadYId = getThreadId(loc, gpu::Dimension::y);
+    load = memref::LoadOp::create(rewriter, privatize->getLoc(), baseTy, alloca,
+                                  ValueRange{threadYId});
+  } else {
+    load =
+        memref::LoadOp::create(rewriter, privatize->getLoc(), baseTy, alloca);
+  }
+  rewriter.setInsertionPointAfter(load.getDefiningOp());
+  mapping.map(privatize.getResult(), load);
+
+  // Operations inside the kernel are all rewritten from scratch.
+  // But if the privatize op is outside the kernel, it needs to be replaced.
+  if (!privatize->getParentOfType<acc::ComputeRegionOp>())
+    rewriter.replaceOp(privatize, load);
+  // Deallocate
+  rewriter.setInsertionPoint(ifOp->getBlock()->getTerminator());
+  if (needsWorkgroupBarrier) {
+    mlir::acc::GPUParallelDimsAttr workerDimsAttr =
+        mlir::acc::GPUParallelDimsAttr::get(
+            rewriter.getContext(),
+            {mlir::acc::GPUParallelDimAttr::threadYDim(rewriter.getContext())});
+    createBarrier(loc, workerDimsAttr);
+  } else {
+    mlir::acc::GPUParallelDimsAttr vectorDimsAttr =
+        mlir::acc::GPUParallelDimsAttr::get(
+            rewriter.getContext(),
+            {mlir::acc::GPUParallelDimAttr::threadXDim(rewriter.getContext())});
+    createBarrier(loc, vectorDimsAttr);
+  }
+  auto ifOp2 = scf::IfOp::create(rewriter, loc, predicate,
+                                 /*withElseRegion=*/false);
+  Region &thenRegion2 = ifOp2.getThenRegion();
+  Block &thenBlock2 = thenRegion2.back();
+  rewriter.setInsertionPoint(thenBlock2.getTerminator());
+  memref::DeallocOp::create(rewriter, privatize->getLoc(), load);
+
+  // Return the private memory
+  rewriter.setInsertionPointAfter(load.getDefiningOp());
+
+  return load;
+}
+
+// Materialize acc.private_local storage from acc.privatize: per-thread alloca
+// when possible, otherwise a shared broadcast slot or acc.gpu_shared_memory.
+
+void ACCCGToGPULowering::processPrivateLocal(
+    acc::PrivateLocalOp privateLocal, std::optional<int64_t> sharedMemCopies) {
+  LLVM_DEBUG(llvm::dbgs() << "processing private local: ";
+             privateLocal->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Location loc = privateLocal.getLoc();
+  acc::PrivateType privTy =
+      cast<acc::PrivateType>(privateLocal.getPrivatized().getType());
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  MemRefType baseTy = getPrivateBaseMemRefType(privTy.getBaseTy(), module);
+  MemRefType byteMemrefTy =
+      MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
+
+  acc::PrivatizeOp privatizeOp = getPrivatizeOp(privateLocal, computeRegion);
+  Value inputMem;
+  if (privatizeOp->getParentOfType<acc::ComputeRegionOp>() == computeRegion) {
+    inputMem = mapping.lookupOrNull(privatizeOp);
+    if (inputMem) {
+      Value result = castPointerLikeTypeIfNeeded(rewriter, loc, inputMem,
+                                                 privateLocal.getType());
+      mapping.map(privateLocal.getResult(), result);
+      return;
+    }
+  } else {
+    // Hoisted acc.privatize: allocate per-thread stack storage in the launch
+    // body. Cross-thread array reduction accumulators are per-thread too, so
+    // the accumulate can reduce each element across threads.
+    acc::ReductionAccumulateArrayOp arrayAccum =
+        perThreadArrayReductionAccum(privateLocal.getResult());
+    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+      if (arrayAccum) {
+        FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
+            arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
+        if (failed(kind))
+          return;
+        initPerThreadArrayAccum(rewriter, loc, alloca, baseTy, *kind);
+      }
+      Value mem = castPointerLikeTypeIfNeeded(rewriter, loc, alloca,
+                                              privateLocal.getType());
+      mapping.map(privateLocal.getResult(), mem);
+      return;
+    }
+
+    // If acc.privatize is outside the kernel, it needs to be converted
+    // explicitly.
+    std::optional<int64_t> copies =
+        sharedMemCopies ? sharedMemCopies
+                        : isEligibleForSharedMemory(privateLocal, baseTy);
+    if (copies) {
+      int64_t numCopies = *copies;
+      int64_t elementSize = getElementSizeInBytes(loc, baseTy.getElementType());
+      int64_t numElements = 1;
+      for (int64_t dim : baseTy.getShape())
+        numElements *= dim;
+      int64_t upperBound = elementSize * numElements * numCopies;
+
+      if (tryAllocateSharedMemory(upperBound)) {
+        std::string varName =
+            accSupport.getVariableName(privateLocal.getResult());
+        sharedMemPrivateVarNames.push_back(varName.empty() ? "(*)" : varName);
+
+        gpu::AddressSpaceAttr workgroupAS = gpu::AddressSpaceAttr::get(
+            computeRegion->getContext(),
+            gpu::GPUDialect::getWorkgroupAddressSpace());
+        MemRefType sharedMemTy =
+            MemRefType::get(baseTy.getShape(), baseTy.getElementType(),
+                            MemRefLayoutAttrInterface{}, workgroupAS);
+        Value sharedMem = acc::GPUSharedMemoryOp::create(
+            rewriter, loc, sharedMemTy, rewriter.getI64IntegerAttr(numCopies),
+            rewriter.getI64IntegerAttr(upperBound), ValueRange{}, IntegerAttr{},
+            IntegerAttr{});
+
+        Value mem =
+            castPointerLikeTypeIfNeeded(rewriter, loc, sharedMem, baseTy);
+        Value result = castPointerLikeTypeIfNeeded(rewriter, loc, mem,
+                                                   privateLocal.getType());
+
+        mapping.map(privateLocal.getResult(), result);
+        return;
+      }
+    }
+
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPoint(privatizeOp);
+    inputMem = processPrivatize(privatizeOp);
+  }
+
+  if (isInsideACCSpecializedRoutine(computeRegion)) {
+    assert(inputMem && "expected input mem to be mapped");
+    Value result = castPointerLikeTypeIfNeeded(rewriter, loc, inputMem,
+                                               privateLocal.getType());
+    mapping.map(privateLocal.getResult(), result);
+    return;
+  }
+
+  // The private element shape augmented with a dimension for each level of
+  // parallelism
+  SmallVector<int64_t> viewShape;
+  // The dynamic sizes of the view (num_gangs, num_workers, vector_length)
+  SmallVector<Value> viewDynSizes;
+  // The offset of the subview from gpu.block_id/gpu.thread_id dimensions.
+  SmallVector<OpFoldResult> subviewOffset;
+  // The sizes of the subview where the dimensionality is brought back to the
+  // private element. That is size 1 for each active block/thread dimension.
+  SmallVector<OpFoldResult> subviewSizes;
+  // The strides of the subview
+  SmallVector<int64_t> subviewStrides;
+  // The shape of the subview
+  SmallVector<int64_t> subviewShape;
+
+  std::pair<SmallVector<mlir::acc::GPUParallelDimAttr>,
+            SmallVector<mlir::acc::GPUParallelDimAttr>>
+      parDimsPair = computeActiveAndInactiveParDims(privateLocal, nullptr);
+  acc::ReductionAccumulateArrayOp arrayAccum =
+      perThreadArrayReductionAccum(privateLocal.getResult());
+  for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    if ((parDim.isThreadX() || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+      if (arrayAccum) {
+        FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
+            arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
+        if (failed(kind))
+          return;
+        initPerThreadArrayAccum(rewriter, loc, alloca, baseTy, *kind);
+      }
+      Value mem = castPointerLikeTypeIfNeeded(rewriter, loc, alloca,
+                                              privateLocal.getType());
+      mapping.map(privateLocal.getResult(), mem);
+      return;
+    }
+  }
+  if (parDimsPair.first.empty()) {
+    // No parallelism is found above. It's single block execution.
+    mlir::acc::insertParDim(
+        parDimsPair.first,
+        mlir::acc::GPUParallelDimAttr::blockXDim(privateLocal.getContext()));
+  }
+  for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    gpu::Processor gpuProc = parDim.getProcessor();
+    Value gpuSize = getGPUSizeFor(gpuProc);
+    viewDynSizes.push_back(gpuSize);
+    viewShape.push_back(ShapedType::kDynamic);
+    subviewOffset.push_back(getGPUThreadIdFor(gpuProc));
+    subviewSizes.push_back(rewriter.getIndexAttr(1));
+  }
+
+  SmallVector<Value> innerDynSizes =
+      resolvePrivateLocalDynamicExtents(privateLocal);
+
+  unsigned dynIdx = 0;
+  for (auto innerDim : baseTy.getShape()) {
+    subviewOffset.push_back(rewriter.getIndexAttr(0));
+    viewShape.push_back(innerDim);
+    subviewShape.push_back(innerDim);
+    if (innerDim == ShapedType::kDynamic) {
+      assert(dynIdx < innerDynSizes.size() &&
+             "not enough dynamic sizes for inner dimensions");
+      viewDynSizes.push_back(innerDynSizes[dynIdx]);
+      subviewSizes.push_back(innerDynSizes[dynIdx]);
+      ++dynIdx;
+    } else {
+      subviewSizes.push_back(rewriter.getIndexAttr(innerDim));
+    }
+  }
+
+  // Do the strides in reverse order.
+  int64_t stride = 1;
+  for (auto innerDimIt = baseTy.getShape().rbegin();
+       innerDimIt != baseTy.getShape().rend(); ++innerDimIt) {
+    int64_t innerDim = *innerDimIt;
+    subviewStrides.insert(subviewStrides.begin(), stride);
+    if (innerDim == ShapedType::kDynamic)
+      stride = ShapedType::kDynamic;
+    if (stride != ShapedType::kDynamic)
+      stride *= innerDim;
+  }
+
+  Value memBuffer =
+      castPointerLikeTypeIfNeeded(rewriter, loc, inputMem, byteMemrefTy);
+  auto c0 = arith::ConstantIndexOp::create(rewriter, loc, 0);
+  MemRefType viewType = MemRefType::get(viewShape, baseTy.getElementType());
+  auto view = memref::ViewOp::create(rewriter, loc, viewType, memBuffer,
+                                     c0.getResult(), viewDynSizes);
+
+  // memref.subview
+  StridedLayoutAttr stridedLayout = StridedLayoutAttr::get(
+      computeRegion->getContext(), ShapedType::kDynamic, subviewStrides);
+  MemRefType subviewType =
+      MemRefType::get(subviewShape, baseTy.getElementType(), stridedLayout);
+  SmallVector<OpFoldResult> ones(viewType.getRank(), rewriter.getIndexAttr(1));
+  Value subview = memref::SubViewOp::create(rewriter, loc, subviewType, view,
+                                            subviewOffset, subviewSizes, ones);
+
+  // Cast subview to the target type, preserving the dynamic offset.
+  // Do NOT cast to a plain memref (offset: 0) - the LLVM optimizer
+  // would fold the gang offset to zero, making all gangs share memory.
+  Value result = castPointerLikeTypeIfNeeded(rewriter, loc, subview,
+                                             privateLocal.getType());
+  mapping.map(privateLocal.getResult(), result);
+}
+
+// Could be scf::for or scf::parallel
+template <typename LoopOp>
+void ACCCGToGPULowering::processSeqLoop(LoopOp loopOp) {
+  // Pre-process shared-memory-eligible private_local ops.  Only direct-child
+  // ops are considered; nested private_local ops (e.g. inside predicate_region)
+  // are handled by recursive body processing.
+  LLVM_DEBUG(llvm::dbgs() << "processing seq loop: ";
+             loopOp->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  llvm::SmallPtrSet<Operation *, 4> preProcessedPrivateLocals;
+  ModuleOp module = computeRegion->getParentOfType<ModuleOp>();
+  for (auto &bodyOp : loopOp.getBody()->getOperations()) {
+    if (acc::PrivateLocalOp privateLocal =
+            dyn_cast<acc::PrivateLocalOp>(&bodyOp)) {
+      acc::PrivateType privTy =
+          cast<acc::PrivateType>(privateLocal.getPrivatized().getType());
+      MemRefType baseTy = getPrivateBaseMemRefType(privTy.getBaseTy(), module);
+      if (auto copies = isEligibleForSharedMemory(privateLocal, baseTy)) {
+        processPrivateLocal(privateLocal, copies);
+        preProcessedPrivateLocals.insert(privateLocal.getOperation());
+      }
+    }
+  }
+
+  LoopOp newLoop =
+      dyn_cast<LoopOp>(rewriter.cloneWithoutRegions(*loopOp, mapping));
+  rewriter.createBlock(
+      &newLoop.getRegion(), newLoop.getRegion().begin(),
+      loopOp.getBody()->getArgumentTypes(),
+      SmallVector<Location>(loopOp.getBody()->getArgumentTypes().size(),
+                            loopOp->getLoc()));
+  rewriter.setInsertionPointToStart(&newLoop.getRegion().front());
+
+  // Need to clone all block arguments
+  Block::BlockArgListType blockArgs = loopOp.getBody()->getArguments();
+  assert(blockArgs.size() && "expected block arguments for loop");
+  mapping.map(blockArgs, newLoop.getBody()->getArguments());
+
+  for (auto &bodyOp : loopOp.getBody()->getOperations()) {
+    if (preProcessedPrivateLocals.contains(&bodyOp))
+      continue;
+    processOp(&bodyOp);
+  }
+
+  mapping.map(loopOp.getResults(), newLoop.getResults());
+  rewriter.setInsertionPointAfter(newLoop);
+
+  // Postpone the barrier when trailing work in this block still separates the
+  // loop from the next reconvergence point.
+  if (hasTrailingSideEffectSiblings(loopOp.getOperation()))
+    deferredBarrierSeqLoops.push_back(loopOp.getOperation());
+  else
+    createBarrierAfterSeqLoop(loopOp.getOperation());
+}
+
+void ACCCGToGPULowering::flushDeferredBarriersBefore(Operation *beforeOp) {
+  Block *block = beforeOp->getBlock();
+  SmallVector<Operation *, 4> toFlush;
+  for (Operation *loopOp : deferredBarrierSeqLoops)
+    if (loopOp->getBlock() == block && loopOp->isBeforeInBlock(beforeOp))
+      toFlush.push_back(loopOp);
+  if (toFlush.empty())
+    return;
+  llvm::sort(toFlush,
+             [](Operation *a, Operation *b) { return a->isBeforeInBlock(b); });
+  for (Operation *loopOp : toFlush)
+    createBarrierAfterSeqLoop(loopOp);
+  deferredBarrierSeqLoops.erase(
+      std::remove_if(deferredBarrierSeqLoops.begin(),
+                     deferredBarrierSeqLoops.end(),
+                     [&](Operation *loopOp) {
+                       return loopOp->getBlock() == block &&
+                              loopOp->isBeforeInBlock(beforeOp);
+                     }),
+      deferredBarrierSeqLoops.end());
+}
+
+// try to process op as a loop mapped to a gpu parallelism
+// failure signifies not a loop and needs 
diff erent processing
+void ACCCGToGPULowering::processParallelOp(scf::ParallelOp parallelOp) {
+  LLVM_DEBUG(llvm::dbgs() << "processing par loop: ";
+             parallelOp->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  assert(mlir::acc::hasParDimsAttr(parallelOp) &&
+         "requires parallel dimensions attribute");
+  mlir::acc::GPUParallelDimsAttr pDimsAttr =
+      mlir::acc::getParDimsAttr(parallelOp);
+  // both of these should be dealt with before compiler reaches ACCCGToGPU
+  // Invalid parallel-loop structure should be rejected before acc-cg-to-gpu.
+  assert(pDimsAttr.getArray().size() == 1 &&
+         "expected a single par dim in acc-cg-to-gpu");
+  assert(parallelOp.getInductionVars().size() == 1 &&
+         "expected a single induction variable in acc-cg-to-gpu");
+
+  mlir::acc::GPUParallelDimAttr parDim = pDimsAttr.getArray().front();
+
+  bool savedGridStrideFlag = insideAccumulateGridStride;
+  Value savedReductionBuf = reductionSharedBuf;
+  if (parDim.isThreadX()) {
+    bool found = false;
+    parallelOp.getBody()->walk([&](acc::ReductionAccumulateOp accOp) {
+      bool hasBlockDim = false;
+      bool hasThreadDim = false;
+      for (auto d : accOp.getParDims().getArray()) {
+        if (d.isAnyBlock())
+          hasBlockDim = true;
+        if (d.isThreadX() || d.isThreadY())
+          hasThreadDim = true;
+      }
+      if (hasThreadDim && !hasBlockDim) {
+        found = true;
+        return WalkResult::interrupt();
+      }
+      return WalkResult::advance();
+    });
+    if (found)
+      insideAccumulateGridStride = true;
+  }
+
+  // common loop body processing for both par0 and 1+
+  auto processLoopBody = [&]() {
+    // process inner ops recursively
+    for (auto &bodyOp : parallelOp.getBody()->getOperations()) {
+      if (bodyOp.hasTrait<OpTrait::IsTerminator>()) {
+        // Non-cloned parallel loops reconverge at their terminator.
+        flushDeferredBarriersBefore(&bodyOp);
+        continue;
+      }
+      // Non-terminator body ops are processed recursively.
+      processOp(&bodyOp);
+    }
+  };
+
+  if (parDim.isSeq()) {
+    LLVM_DEBUG(llvm::dbgs() << "loop: parDim: " << parDim << " as gpu seq\n");
+    // Sequential loops here are remainder loops from partitioned parallel
+    // loops - clone them as-is but process the body as a parallel region.
+    // When blockDim.x >= subgroupSize and the loop contains a thread-level
+    // accumulate, inactive grid-stride threads cannot participate in subgroup
+    // reductions, so we use atomic-to-shared-memory reduction instead.
+    bool needsAtomicReduction = false;
+    bool hasAccumulateSibling = false;
+    if (scf::ParallelOp parentPar =
+            parallelOp->getParentOfType<scf::ParallelOp>()) {
+      if (mlir::acc::GPUParallelDimsAttr parentDims =
+              mlir::acc::getParDimsAttr(parentPar);
+          parentDims && llvm::any_of(parentDims.getArray(),
+                                     [](auto d) { return d.isThreadX(); })) {
+        for (auto &op : parentPar.getBody()->getOperations()) {
+          if (acc::ReductionAccumulateOp acc =
+                  dyn_cast<acc::ReductionAccumulateOp>(op)) {
+            bool hasBlockDim = false;
+            bool hasThreadDim = false;
+            for (auto d : acc.getParDims().getArray()) {
+              if (d.isAnyBlock())
+                hasBlockDim = true;
+              if (d.isThreadX() || d.isThreadY())
+                hasThreadDim = true;
+            }
+            if (hasThreadDim && !hasBlockDim)
+              hasAccumulateSibling = true;
+          }
+        }
+      }
+    }
+    if (insideAccumulateGridStride || hasAccumulateSibling) {
+      for (auto launchArg : computeRegion.getLaunchArgs()) {
+        if (acc::ParWidthOp pw = launchArg.getDefiningOp<acc::ParWidthOp>()) {
+          if (pw.getParDim().isThreadX()) {
+            if (auto cval = getConstantIntValue(pw.getLaunchArg()))
+              needsAtomicReduction = (*cval >= options.subgroupSize);
+            else
+              needsAtomicReduction = true;
+            break;
+          }
+        }
+      }
+    }
+    if (needsAtomicReduction && !reductionSharedBuf) {
+      Type elemTy;
+      parallelOp.getBody()->walk([&](acc::ReductionAccumulateOp accOp) {
+        Type t = accOp.getValue().getType();
+        if (isa<FloatType, IntegerType>(t))
+          elemTy = t;
+        return elemTy ? WalkResult::interrupt() : WalkResult::advance();
+      });
+      if (!elemTy)
+        needsAtomicReduction = false;
+    }
+    if (needsAtomicReduction && !reductionSharedBuf) {
+      Location seqLoc = parallelOp->getLoc();
+      gpu::AddressSpaceAttr workgroupAS = gpu::AddressSpaceAttr::get(
+          computeRegion->getContext(),
+          gpu::GPUDialect::getWorkgroupAddressSpace());
+      Type elemTy;
+      parallelOp.getBody()->walk([&](acc::ReductionAccumulateOp accOp) {
+        Type t = accOp.getValue().getType();
+        if (isa<FloatType, IntegerType>(t))
+          elemTy = t;
+        return elemTy ? WalkResult::interrupt() : WalkResult::advance();
+      });
+      assert(elemTy && "expected scalar reduction element type");
+      unsigned elemBytes = elemTy.getIntOrFloatBitWidth() / 8;
+      MemRefType bufTy = MemRefType::get({options.subgroupSize}, elemTy,
+                                         AffineMap{}, workgroupAS);
+      reductionSharedBuf = acc::GPUSharedMemoryOp::create(
+          rewriter, seqLoc, bufTy, rewriter.getI64IntegerAttr(1),
+          rewriter.getI64IntegerAttr(options.subgroupSize * elemBytes),
+          ValueRange{}, IntegerAttr{}, IntegerAttr{});
+      Value tidY = getThreadId(seqLoc, gpu::Dimension::y);
+      Value identity;
+      if (isa<FloatType>(elemTy)) {
+        identity = arith::ConstantOp::create(
+            rewriter, seqLoc, elemTy, rewriter.getFloatAttr(elemTy, 0.0));
+      } else {
+        identity = arith::ConstantIntOp::create(rewriter, seqLoc, elemTy, 0);
+      }
+      memref::StoreOp::create(rewriter, seqLoc, identity, reductionSharedBuf,
+                              tidY);
+      createPerRowBarrier(seqLoc);
+    }
+    processSeqLoop(parallelOp);
+    loopReductions.push_back(parallelOp);
+  } else {
+    LLVM_DEBUG(llvm::dbgs()
+               << "processing loop: parDim: " << parDim << " as gpu par\n");
+    // actual parallel loops get their iv mapped to gpu hierarchy
+    // and the loop construct is not cloned to gpu kernel only the
+    // ops are cloned with mapping of gpu id for original loop iv
+    Value gpuThreadId = getGPUThreadIdFor(parDim.getProcessor());
+    mapping.map(parallelOp.getInductionVars()[0], gpuThreadId);
+
+    processLoopBody();
+
+    // Since the loop is not copied over, create dummy mappings for_each
+    // of the loop results. These will ultimately by replaced with a
+    // reduction
+    llvm::for_each(parallelOp.getResults(), [&](Value v) {
+      Type valTy = v.getType();
+      TypedAttr zeroAttr = rewriter.getZeroAttr(valTy);
+      auto zero = arith::ConstantOp::create(rewriter, parallelOp->getLoc(),
+                                            valTy, zeroAttr);
+      mapping.map(v, zero);
+    });
+    loopReductions.push_back(parallelOp);
+  }
+  insideAccumulateGridStride = savedGridStrideFlag;
+  if (!insideAccumulateGridStride && !savedReductionBuf)
+    reductionSharedBuf = Value();
+}
+
+/// Map an atomic RMW kind to the corresponding `gpu.all_reduce` operation.
+static gpu::AllReduceOperation
+getAllReduceOperation(arith::AtomicRMWKind kind) {
+  switch (kind) {
+  case arith::AtomicRMWKind::addf:
+  case arith::AtomicRMWKind::addi:
+    return gpu::AllReduceOperation::ADD;
+  case arith::AtomicRMWKind::mulf:
+  case arith::AtomicRMWKind::muli:
+    return gpu::AllReduceOperation::MUL;
+  case arith::AtomicRMWKind::minu:
+    return gpu::AllReduceOperation::MINUI;
+  case arith::AtomicRMWKind::mins:
+    return gpu::AllReduceOperation::MINSI;
+  case arith::AtomicRMWKind::minnumf:
+    return gpu::AllReduceOperation::MINNUMF;
+  case arith::AtomicRMWKind::maxu:
+    return gpu::AllReduceOperation::MAXUI;
+  case arith::AtomicRMWKind::maxs:
+    return gpu::AllReduceOperation::MAXSI;
+  case arith::AtomicRMWKind::maxnumf:
+    return gpu::AllReduceOperation::MAXNUMF;
+  case arith::AtomicRMWKind::ori:
+    return gpu::AllReduceOperation::OR;
+  case arith::AtomicRMWKind::andi:
+    return gpu::AllReduceOperation::AND;
+  case arith::AtomicRMWKind::xori:
+    return gpu::AllReduceOperation::XOR;
+  case arith::AtomicRMWKind::minimumf:
+    return gpu::AllReduceOperation::MINIMUMF;
+  case arith::AtomicRMWKind::maximumf:
+    return gpu::AllReduceOperation::MAXIMUMF;
+  case arith::AtomicRMWKind::assign:
+    break;
+  }
+  llvm_unreachable("unsupported atomic kind");
+}
+
+void ACCCGToGPULowering::constructAtomicAccumulation(
+    Location loc, Value memref, ValueRange indices, Value input,
+    arith::AtomicRMWKind kind) {
+  assert(!memref.getDefiningOp<memref::AllocaOp>() &&
+         "cannot lower atomic accumulation on an stack variable");
+
+  // acc.atomic.update derives the element address from the memref descriptor's
+  // base pointer and offset field; it has no subscript operand. When the store
+  // being lowered targets a specific array element (e.g. result(idx) =
+  // max(...)), fold the indices into the descriptor offset with a subview so
+  // the atomic updates the intended element. Otherwise the atomic always hits
+  // element 0, so any reduction whose destination index is non-zero is
+  // miscompiled (the result lands in element 0 while the intended element keeps
+  // its identity-init value).
+  Value target = memref;
+  if (!indices.empty()) {
+    MemRefType memrefTy = cast<MemRefType>(memref.getType());
+    unsigned rank = memrefTy.getRank();
+    assert(indices.size() == rank && "expected one index per memref dimension");
+    SmallVector<OpFoldResult> offsets(indices.begin(), indices.end());
+    SmallVector<OpFoldResult> sizes(rank, rewriter.getIndexAttr(1));
+    SmallVector<OpFoldResult> strides(rank, rewriter.getIndexAttr(1));
+    target = memref::SubViewOp::create(rewriter, loc, memref, offsets, sizes,
+                                       strides);
+  }
+
+  auto atomicUpdateOp =
+      acc::AtomicUpdateOp::create(rewriter, loc, target, /*ifCond=*/Value());
+  Region &region = atomicUpdateOp->getRegion(0);
+  Block *block =
+      rewriter.createBlock(&region, region.begin(), {input.getType()}, {loc});
+  rewriter.setInsertionPointToStart(block);
+  Value reductionExpr =
+      generateReductionOp(rewriter, loc, input, block->getArgument(0), kind);
+  acc::YieldOp::create(rewriter, loc, reductionExpr);
+  rewriter.setInsertionPointAfter(atomicUpdateOp);
+}
+
+void ACCCGToGPULowering::createGPUAllReduceOp(
+    Location loc, Value input, Value memref, arith::AtomicRMWKind kind,
+    mlir::acc::GPUParallelDimsAttr parDimsAttr, ValueRange indices) {
+  gpu::AllReduceOperationAttr attr = gpu::AllReduceOperationAttr::get(
+      computeRegion->getContext(), getAllReduceOperation(kind));
+  auto allReduceOp = gpu::AllReduceOp::create(rewriter, loc, input, attr, true);
+  mlir::acc::setParDimsAttr(allReduceOp, parDimsAttr);
+  // Predicate the store on the thread-level dimensions being reduced so that
+  // only one thread per reduced group writes the result. Only dimensions in
+  // parDimsAttr are included; sweeping over all dimensions between the highest
+  // par_dim and thread_x would incorrectly add unrelated dimensions (e.g.
+  // thread_y for a thread_x-only reduction), preventing other rows from
+  // storing their independent results.
+  SmallVector<mlir::acc::GPUParallelDimAttr> inactiveParDims;
+  MLIRContext *ctx = computeRegion->getContext();
+  bool hasThreadX = false;
+  for (auto parDim : parDimsAttr.getArray()) {
+    if (parDim.isAnyBlock())
+      continue;
+    if (parDim.isThreadX())
+      hasThreadX = true;
+    if (computeRegion.getLaunchArg(parDim) ||
+        isInsideACCSpecializedRoutine(computeRegion)) {
+      inactiveParDims.push_back(parDim);
+    }
+  }
+  // Subgroup alignment may introduce extra ThreadX lanes even when ThreadX is
+  // not part of the reduction. Predicate on ThreadX so only one lane stores.
+  if (!hasThreadX)
+    inactiveParDims.push_back(mlir::acc::GPUParallelDimAttr::threadXDim(ctx));
+  Value predicate = emitPredicate(loc, inactiveParDims);
+  // Predication is only needed when the store target is visible to
+  // multiple threads (shared/global memory). Per-thread targets like
+  // memref.alloca are thread-private: gpu.all_reduce returns the same
+  // value on all threads, so each can safely store to its own copy.
+  // Detect per-thread storage by walking through conversion ops to
+  // find the underlying allocation.
+  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(
+      unwrapMemRefConversion(memref).getDefiningOp());
+  // combine
+  scf::IfOp ifOp;
+  if (predicate && !isPerThreadPrivate) {
+    ifOp =
+        scf::IfOp::create(rewriter, loc, predicate, /*withElseRegion=*/false);
+    Region &thenRegion = ifOp.getThenRegion();
+    Block &thenBlock = thenRegion.back();
+    rewriter.setInsertionPoint(thenBlock.getTerminator());
+  }
+  memref::StoreOp::create(rewriter, loc, allReduceOp, memref, indices);
+  if (predicate && !isPerThreadPrivate)
+    rewriter.setInsertionPointAfter(ifOp);
+  // A later block combine reuses this instead of reloading the slot. This only
+  // applies to scalar accumulators; array elements are indexed individually.
+  if (indices.empty())
+    reductionAccumValue[memref] = allReduceOp;
+}
+
+void ACCCGToGPULowering::postprocessAccumulateOp(
+    acc::ReductionAccumulateOp op) {
+  Location loc = op->getLoc();
+
+  rewriter.setInsertionPoint(op);
+
+  // Check whether this accumulate has only block-level par dims (no thread
+  // dims).  gpu.all_reduce reduces across threads within a block, which is
+  // wrong for block-only reductions - the loop result is already per-thread
+  // and only needs a predicated store to the memref.
+  bool hasThreadDim = false;
+  SmallVector<mlir::acc::GPUParallelDimAttr> threadParDims;
+  for (auto parDim : op.getParDims().getArray()) {
+    if (!parDim.isAnyBlock()) {
+      hasThreadDim = true;
+      threadParDims.push_back(parDim);
+    }
+  }
+
+  std::optional<arith::AtomicRMWKind> kind;
+  if (hasThreadDim) {
+    FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
+        op.getReductionOperator(), op.getValue().getType(), loc);
+    if (failed(kindOr))
+      return;
+    kind = *kindOr;
+  }
+
+  if (hasThreadDim && reductionSharedBuf &&
+      op.getValue().getType() ==
+          cast<MemRefType>(reductionSharedBuf.getType()).getElementType()) {
+    Value val = op.getValue();
+    Value mem = op.getMemref();
+    Value tidY = getThreadId(loc, gpu::Dimension::y);
+    memref::AtomicRMWOp::create(rewriter, loc, *kind, val, reductionSharedBuf,
+                                ValueRange{tidY});
+    createPerRowBarrier(loc);
+    Value result =
+        memref::LoadOp::create(rewriter, loc, reductionSharedBuf, tidY);
+    memref::StoreOp::create(rewriter, loc, result, mem);
+    reductionAccumValue[mem] = result;
+  } else if (hasThreadDim) {
+    createGPUAllReduceOp(loc, op.getValue(), op.getMemref(), *kind,
+                         op.getParDims());
+  } else {
+    // Block-only: no gpu.all_reduce needed (all threads have the same
+    // value after broadcast).  Just store the value to the memref.
+    // Predicate on all thread dims when the target is shared memory.
+    Value val = mapping.lookupOrDefault(op.getValue());
+    Value mem = mapping.lookupOrDefault(op.getMemref());
+    bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(
+        unwrapMemRefConversion(mem).getDefiningOp());
+    if (!isPerThreadPrivate) {
+      SmallVector<mlir::acc::GPUParallelDimAttr> predDims;
+      for (auto parDim : computeRegion.getLaunchParDims())
+        if (!parDim.isAnyBlock())
+          predDims.push_back(parDim);
+      if (predDims.empty()) {
+        predDims.push_back(mlir::acc::GPUParallelDimAttr::threadXDim(
+            computeRegion->getContext()));
+      }
+      Value predicate = emitPredicate(loc, predDims);
+      auto ifOp =
+          scf::IfOp::create(rewriter, loc, predicate, /*withElseRegion=*/false);
+      rewriter.setInsertionPoint(ifOp.getThenRegion().back().getTerminator());
+      memref::StoreOp::create(rewriter, loc, val, mem);
+      rewriter.setInsertionPointAfter(ifOp);
+    } else {
+      memref::StoreOp::create(rewriter, loc, val, mem);
+    }
+  }
+
+  // erase acc.reduction_accumulate
+  rewriter.eraseOp(op);
+}
+
+void ACCCGToGPULowering::postprocessLoopReduction(scf::ParallelOp parLoop) {
+  if (parLoop.getNumReductions() == 0)
+    return;
+
+  for (unsigned i = 0; i < parLoop.getNumResults(); ++i) {
+    for (Operation *user :
+         mapping.lookupOrDefault(parLoop.getResult(i)).getUsers()) {
+      if (acc::ReductionAccumulateOp accumulateOp =
+              dyn_cast<acc::ReductionAccumulateOp>(user)) {
+        postprocessAccumulateOp(accumulateOp);
+      }
+    }
+  }
+}
+
+void ACCCGToGPULowering::processExecuteRegion(scf::ExecuteRegionOp op) {
+  LLVM_DEBUG(llvm::dbgs() << "processing execute region op: ";
+             op->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Location loc = op->getLoc();
+  auto types = op.getResultTypes();
+  Region &oldRegion = op.getRegion();
+  // create the executeRegion op inside gpu launch
+  auto executeRegionOp = scf::ExecuteRegionOp::create(rewriter, loc, types);
+  Region &region = executeRegionOp.getRegion();
+  rewriter.createBlock(&region);
+  rewriter.setInsertionPointToEnd(&region.front());
+
+  llvm::DenseMap<Block *, Block *> blockMap;
+  blockMap[&oldRegion.front()] = &region.front();
+
+  // Create blocks in the new operation corresponding to all blocks in the
+  // original op
+  for (auto &oldBlock : llvm::drop_begin(oldRegion.getBlocks())) {
+    TypeRange argTypes = oldBlock.getArgumentTypes();
+    size_t numArgs = argTypes.size();
+    // Create new block with same argument types
+    Block *newBlock = rewriter.createBlock(&region, region.end(), argTypes,
+                                           SmallVector<Location>(numArgs, loc));
+    blockMap[&oldBlock] = newBlock;
+    // Map block arguments
+    mapping.map(oldBlock.getArguments(), newBlock->getArguments());
+  }
+
+  // Iterate over all blocks of oldRegion and all operations inside them
+  // process all the ops except the terminator
+  for (auto [oldBlock, newBlock] :
+       llvm::zip(oldRegion.getBlocks(), region.getBlocks())) {
+    OpBuilder::InsertionGuard blockGuard(rewriter);
+    rewriter.setInsertionPointToStart(&newBlock);
+    for (auto &bodyOp : oldBlock.getOperations()) {
+      // Skip terminators during normal iteration - handle them separately
+      if (bodyOp.hasTrait<OpTrait::IsTerminator>())
+        continue;
+      processOp(&bodyOp);
+    }
+
+    // Copy the terminator from old block to new block
+    Operation *oldTerminator = oldBlock.getTerminator();
+    rewriter.setInsertionPointToEnd(&newBlock);
+    Operation *newTerminator = rewriter.clone(*oldTerminator, mapping);
+
+    // Replace successors with mapped blocks
+    for (unsigned i = 0; i < oldTerminator->getNumSuccessors(); ++i) {
+      Block *oldDest = oldTerminator->getSuccessor(i);
+      Block *newDest = blockMap.lookup(oldDest);
+      assert(newDest && "Successor block must be in blockMap");
+      newTerminator->setSuccessor(newDest, i);
+    }
+  }
+  mapping.map(op->getResults(), executeRegionOp->getResults());
+  rewriter.setInsertionPointAfter(executeRegionOp);
+}
+
+void ACCCGToGPULowering::processAccumulateOp(acc::ReductionAccumulateOp op) {
+  LLVM_DEBUG(llvm::dbgs() << "processing accumulate op: " << *op << "\n");
+  Value accumulateValue = op.getValue();
+  if (reductionSharedBuf &&
+      mapping.lookupOrDefault(accumulateValue).getType() ==
+          cast<MemRefType>(reductionSharedBuf.getType()).getElementType()) {
+    Location loc = op->getLoc();
+    FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
+        op.getReductionOperator(), accumulateValue.getType(), loc);
+    if (failed(kind))
+      return;
+    Value mappedValue = mapping.lookupOrDefault(accumulateValue);
+    Value memref = mapping.lookupOrDefault(op.getMemref());
+    Value tidY = getThreadId(loc, gpu::Dimension::y);
+    memref::AtomicRMWOp::create(rewriter, loc, *kind, mappedValue,
+                                reductionSharedBuf, ValueRange{tidY});
+    createPerRowBarrier(loc);
+    Value result =
+        memref::LoadOp::create(rewriter, loc, reductionSharedBuf, tidY);
+    memref::StoreOp::create(rewriter, loc, result, memref);
+    reductionAccumValue[memref] = result;
+    return;
+  }
+  if (accumulateValue.getDefiningOp<scf::ParallelOp>()) {
+    Operation *newOp = rewriter.clone(*op, mapping);
+    mapping.map(op->getResults(), newOp->getResults());
+  } else if (isRedundantChainAccumulate(op)) {
+    // The destination memref already holds the correctly aggregated value
+    // (atomically reduced by a preceding acc.reduction_combine with a block
+    // par_dim). Skip the redundant gpu.all_reduce + atomic.update; emit a
+    // Workgroup-wide barrier so downstream readers see all prior atomic
+    // updates.
+    LLVM_DEBUG(llvm::dbgs() << "  skipped: redundant chain accumulate\n");
+    gpu::BarrierOp::create(rewriter, op->getLoc());
+  } else {
+    Value mappedValue = mapping.lookupOrDefault(accumulateValue);
+    Value memref = mapping.lookupOrDefault(op.getMemref());
+    FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
+        op.getReductionOperator(), accumulateValue.getType(), op.getLoc());
+    if (failed(kind))
+      return;
+    createGPUAllReduceOp(op->getLoc(), mappedValue, memref, *kind,
+                         op.getParDims());
+  }
+}
+
+void ACCCGToGPULowering::processAccumulateArrayOp(
+    acc::ReductionAccumulateArrayOp op) {
+  LLVM_DEBUG(llvm::dbgs() << "processing accumulate array op: " << *op << "\n");
+  Location loc = op.getLoc();
+
+  Value memref = mapping.lookupOrDefault(op.getMemref());
+  MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
+  assert(memrefTy && memrefTy.getRank() == 1 &&
+         "array reduction accumulate expects a rank-1 memref");
+
+  FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
+      op.getReductionOperator(), memrefTy.getElementType(), loc);
+  if (failed(kindOr))
+    return;
+  arith::AtomicRMWKind kind = *kindOr;
+
+  // The (already mapped/cloned) acc.bounds op describes the element range; it
+  // is dead after lowering since we read its operands directly.
+  acc::DataBoundsOp boundsOp = mapping.lookupOrDefault(op.getBounds())
+                                   .getDefiningOp<acc::DataBoundsOp>();
+  assert(boundsOp && "expected acc.bounds defining op for array accumulate");
+  auto eraseDeadBounds = [&] {
+    if (boundsOp->use_empty())
+      rewriter.eraseOp(boundsOp);
+  };
+
+  bool hasThreadDim = false;
+  bool hasBlockDim = false;
+  for (auto pd : op.getParDims().getArray()) {
+    hasThreadDim |= pd.isAnyThread();
+    hasBlockDim |= pd.isAnyBlock();
+  }
+
+  // Block-only (gang) reduction: each element is produced by one gang, so the
+  // per-gang copy already holds the result and the combine does the rest.
+  if (hasBlockDim && !hasThreadDim) {
+    eraseDeadBounds();
+    return;
+  }
+
+  // A thread-level reduction with no block owner for its elements cannot merge
+  // the cross-thread partials, so report NYI.
+  if (!reductionHasBlockContext(op)) {
+    (void)accSupport.emitNYI(
+        loc, "reduction: thread-only array reduction accumulate");
+    return;
+  }
+
+  // Per-element gpu.all_reduce is only correct for a per-thread alloca; mirror
+  // processPrivateLocal's stack-fit decision rather than inspect the memref.
+  bool isPerThreadPrivate =
+      canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
+  if (!isPerThreadPrivate) {
+    // Block-shared accumulator: no-op only when the accumulate spans a block
+    // dim (threads distribute distinct elements, so the block partial is in
+    // place and the atomic combine finishes it). A thread-only shared
+    // reduction, where several threads reduce into the same element, is not yet
+    // supported.
+    if (hasBlockDim) {
+      eraseDeadBounds();
+    } else {
+      (void)accSupport.emitNYI(
+          loc, "reduction: shared-memory array reduction accumulate");
+    }
+    return;
+  }
+
+  // Bounds are normalized to be zero-based.
+  auto toIndex = [&](Value v) -> Value {
+    if (v.getType().isIndex())
+      return v;
+    return arith::IndexCastOp::create(rewriter, loc, rewriter.getIndexType(),
+                                      v);
+  };
+
+  Value lb = boundsOp.getLowerbound()
+                 ? toIndex(boundsOp.getLowerbound())
+                 : arith::ConstantIndexOp::create(rewriter, loc, 0);
+  Value step = boundsOp.getStride()
+                   ? toIndex(boundsOp.getStride())
+                   : arith::ConstantIndexOp::create(rewriter, loc, 1);
+  Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
+  // Exclusive upper bound: prefer extent (count of elements), fall back to the
+  // inclusive upperbound.
+  Value ub;
+  if (boundsOp.getExtent()) {
+    ub =
+        arith::AddIOp::create(rewriter, loc, lb, toIndex(boundsOp.getExtent()));
+  } else {
+    assert(boundsOp.getUpperbound() &&
+           "acc.bounds must specify an extent or upperbound");
+    ub = arith::AddIOp::create(rewriter, loc, toIndex(boundsOp.getUpperbound()),
+                               one);
+  }
+
+  // Reduce each array element across the requested parallel dimensions.
+  auto forOp = scf::ForOp::create(rewriter, loc, lb, ub, step);
+  {
+    OpBuilder::InsertionGuard guard(rewriter);
+    rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
+    Value iv = forOp.getInductionVar();
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
+                         ValueRange{iv});
+  }
+
+  eraseDeadBounds();
+}
+
+void ACCCGToGPULowering::processReductionOp(acc::ReductionInitOp op) {
+  // Clone the inner ops of the reduction op only
+  op.getRegion().walk<WalkOrder::PreOrder>([&](Operation *innerOp) {
+    if (acc::YieldOp yieldOp = dyn_cast<acc::YieldOp>(innerOp)) {
+      op.getResult().replaceAllUsesWith(mapping.lookup(yieldOp.getOperand(0)));
+      return WalkResult::interrupt();
+    }
+    if (innerOp->getNumRegions() > 0) {
+      processOp(innerOp);
+      return WalkResult::skip();
+    }
+    rewriter.clone(*innerOp, mapping);
+    return WalkResult::advance();
+  });
+}
+
+void ACCCGToGPULowering::processReductionCombineOp(acc::ReductionCombineOp op) {
+  LLVM_DEBUG(llvm::dbgs() << "processing reduction combine op: ";
+             op->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Location loc = op.getLoc();
+  MemRefType memrefType = dyn_cast<MemRefType>(op.getSrcMemref().getType());
+  assert(memrefType && "expected memref type for reduction combine op");
+  assert(memrefType.getRank() == 0 &&
+         "expected scalar memref type for reduction combine op");
+  Type elTy = memrefType.getElementType();
+  FailureOr<arith::AtomicRMWKind> kindOr =
+      getReductionKind(op.getReductionOperator(), elTy, loc);
+  if (failed(kindOr))
+    return;
+  arith::AtomicRMWKind kind = *kindOr;
+
+  Value srcMemref = mapping.lookupOrDefault(op.getSrcMemref());
+  Value destMemref = mapping.lookupOrDefault(op.getDestMemref());
+
+  // A block par_dim normally means the accumulator is shared across blocks and
+  // must be updated atomically. But when the destination resolves to a
+  // thread-private stack alloca (e.g. an inner-loop reduction combining one
+  // private accumulator into another private accumulator for the same thread),
+  // the update is not visible to other threads and must not be atomic. Treat
+  // such destinations as a plain load/combine/store below.
+  bool destIsPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(
+      unwrapMemRefConversion(destMemref).getDefiningOp());
+
+  SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+      getReductionCombineParDims(op);
+  for (auto parDim : parDims) {
+    if (parDim.isAnyBlock() && !destIsPerThreadPrivate) {
+      // Block reduction directly stores to the accumulator using atomic.
+      // The predication (tid.x == 0 when subgroup-aligned) is already handled
+      // by the parent predicate_region processing.
+      // Reloading a grid-shared slot races with other blocks; record
+      // it and replace with the block-reduced register value in the fixup.
+      auto srcLoad = memref::LoadOp::create(rewriter, loc, srcMemref);
+      pendingCombineReloads.push_back({srcMemref, srcLoad});
+      constructAtomicAccumulation(loc, destMemref, /*indices=*/{}, srcLoad,
+                                  kind);
+      return;
+    }
+  }
+
+  // Atomic construction is not needed; lower this operation to typical
+  // reduction update operations. E.g. dest = dest <kind> src
+  auto srcLoad = memref::LoadOp::create(rewriter, loc, srcMemref, ValueRange{});
+  auto destLoad =
+      memref::LoadOp::create(rewriter, loc, destMemref, ValueRange{});
+  Value combine = generateReductionOp(rewriter, loc, srcLoad, destLoad, kind);
+  memref::StoreOp::create(rewriter, loc, combine, destMemref, ValueRange{});
+}
+
+void ACCCGToGPULowering::processCombineRegionOp(
+    acc::ReductionCombineRegionOp op) {
+  LLVM_DEBUG(llvm::dbgs() << "processing combine region op: ";
+             op->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  // A block par_dim on a combine into a thread-private stack alloca is not a
+  // real cross-block accumulation (the alloca is not shared across blocks), so
+  // it must use a plain load/combine/store rather than an atomic update.
+  bool destIsPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(
+      unwrapMemRefConversion(mapping.lookupOrDefault(op.getDestVar()))
+          .getDefiningOp());
+  SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+      getReductionCombineParDims(op);
+  for (auto parDim : parDims) {
+    if (parDim.isAnyBlock() && !destIsPerThreadPrivate) {
+      // Block reduction directly stores to the accumulator using atomic.
+      // The predication (tid.x == 0 when subgroup-aligned) is already handled
+      // by the parent predicate_region processing.
+      for (Operation *user : op.getSrcVar().getUsers()) {
+        if (acc::ReductionAccumulateOp accumulateOp =
+                dyn_cast<acc::ReductionAccumulateOp>(user)) {
+          Location loc = accumulateOp.getLoc();
+          FailureOr<arith::AtomicRMWKind> kind =
+              getReductionKind(accumulateOp.getReductionOperator(),
+                               accumulateOp.getValue().getType(), loc);
+          if (failed(kind))
+            return;
+          Value srcMemref = mapping.lookupOrDefault(accumulateOp.getMemref());
+          // Recorded and patched in the fixup to avoid the reload race.
+          auto reductionLoad = memref::LoadOp::create(rewriter, loc, srcMemref);
+          pendingCombineReloads.push_back({srcMemref, reductionLoad});
+          constructAtomicAccumulation(loc,
+                                      mapping.lookupOrDefault(op.getDestVar()),
+                                      /*indices=*/{}, reductionLoad, *kind);
+          return;
+        }
+      }
+      // For decomposed complex reductions, the AccumulateOp was replaced
+      // with real/imag AccumulateOps. Load from the private memref which
+      // holds the reconstructed complex value.
+      Value privateMemref = mapping.lookupOrDefault(op.getSrcVar());
+      MemRefType memrefTy = cast<MemRefType>(privateMemref.getType());
+      if (isa<ComplexType>(memrefTy.getElementType())) {
+        Location loc = op.getLoc();
+        Value reductionResult =
+            memref::LoadOp::create(rewriter, loc, privateMemref);
+        arith::AtomicRMWKind kind = arith::AtomicRMWKind::addf;
+        op.getRegion().walk([&](Operation *innerOp) {
+          if (isa<complex::MulOp>(innerOp))
+            kind = arith::AtomicRMWKind::mulf;
+        });
+        constructAtomicAccumulation(loc,
+                                    mapping.lookupOrDefault(op.getDestVar()),
+                                    /*indices=*/{}, reductionResult, kind);
+        return;
+      }
+    }
+  }
+  op.getRegion().walk<WalkOrder::PreOrder>([&](Operation *innerOp) {
+    if (acc::YieldOp yieldOp = dyn_cast<acc::YieldOp>(innerOp))
+      return WalkResult::interrupt();
+    if (innerOp->getNumRegions() > 0) {
+      processOp(innerOp);
+      return WalkResult::skip();
+    }
+    rewriter.clone(*innerOp, mapping);
+    return WalkResult::advance();
+  });
+}
+
+void ACCCGToGPULowering::processGenericOp(Operation *op) {
+  // Operations with no regions or operations for which we know
+  // no recursive processing is needed can be fully cloned.
+  LLVM_DEBUG(llvm::dbgs() << "processing generic op, cloning: ";
+             op->print(llvm::dbgs()); llvm::dbgs() << "\n");
+  Operation *newOp = rewriter.clone(*op, mapping);
+  // update mapping as cloning creates 
diff erent result values
+  mapping.map(op->getResults(), newOp->getResults());
+}
+
+void ACCCGToGPULowering::processGenericOpWithRegions(Operation *op) {
+  // Generic handling for operations with regions
+  LLVM_DEBUG(llvm::dbgs() << "processing generic op with regions: ";
+             op->print(llvm::dbgs()); llvm::dbgs() << "\n");
+
+  // Clone the operation structure without its regions
+  Operation *newOp = rewriter.cloneWithoutRegions(*op, mapping);
+
+  // Process each region recursively
+  for (auto [oldRegion, newRegion] :
+       llvm::zip(op->getRegions(), newOp->getRegions())) {
+    // Create blocks in the new region corresponding to old region blocks
+    for (auto &oldBlock : oldRegion.getBlocks()) {
+      TypeRange argTypes = oldBlock.getArgumentTypes();
+      size_t numArgs = argTypes.size();
+      // Create new block with same argument types
+      Block *newBlock =
+          rewriter.createBlock(&newRegion, newRegion.end(), argTypes,
+                               SmallVector<Location>(numArgs, op->getLoc()));
+
+      // Map block arguments
+      mapping.map(oldBlock.getArguments(), newBlock->getArguments());
+
+      // Process each operation in the block
+      for (auto &innerOp : oldBlock.getOperations()) {
+        OpBuilder::InsertionGuard guard(rewriter);
+        rewriter.setInsertionPointToEnd(newBlock);
+        processOp(&innerOp);
+      }
+    }
+  }
+  rewriter.setInsertionPointAfter(newOp);
+
+  // Update mapping for results
+  mapping.map(op->getResults(), newOp->getResults());
+}
+
+// thread through par dim to verify redundant/0 execution modes
+void ACCCGToGPULowering::processOp(Operation *op) {
+  if (isDeferredBarrierFlushPoint(op))
+    flushDeferredBarriersBefore(op);
+  if (mlir::acc::hasParDimsAttr(op) && isa<scf::ParallelOp>(op)) {
+    // parallel loops require special processing based on parallel dimension
+    // this is mutually recursive with processOp
+    scf::ParallelOp parallelOp = cast<scf::ParallelOp>(op);
+    processParallelOp(parallelOp);
+  } else if (scf::ForOp seqLoop = dyn_cast<scf::ForOp>(op)) {
+    processSeqLoop(seqLoop);
+  } else if (acc::PrivatizeOp privatize = dyn_cast<acc::PrivatizeOp>(op)) {
+    processPrivatize(privatize);
+  } else if (acc::PrivateLocalOp privateLocal =
+                 dyn_cast<acc::PrivateLocalOp>(op)) {
+    processPrivateLocal(privateLocal);
+  } else if (acc::PredicateRegionOp predicateRegionOp =
+                 dyn_cast<acc::PredicateRegionOp>(op)) {
+    processPredicateRegion(predicateRegionOp);
+  } else if (acc::ReductionAccumulateOp accumulateOp =
+                 dyn_cast<acc::ReductionAccumulateOp>(op)) {
+    processAccumulateOp(accumulateOp);
+  } else if (auto accumulateArrayOp =
+                 dyn_cast<acc::ReductionAccumulateArrayOp>(op)) {
+    processAccumulateArrayOp(accumulateArrayOp);
+  } else if (acc::ReductionInitOp reductionInitOp =
+                 dyn_cast<acc::ReductionInitOp>(op)) {
+    processReductionOp(reductionInitOp);
+  } else if (acc::ReductionCombineOp reductionCombineOp =
+                 dyn_cast<acc::ReductionCombineOp>(op)) {
+    processReductionCombineOp(reductionCombineOp);
+  } else if (auto combineRegionOp =
+                 dyn_cast<acc::ReductionCombineRegionOp>(op)) {
+    processCombineRegionOp(combineRegionOp);
+  } else if (acc::ReductionOp accReductionOp = dyn_cast<acc::ReductionOp>(op)) {
+    mapping.map(accReductionOp->getResult(0), accReductionOp.getVarPtr());
+  } else if (mapping.contains(op)) {
+    // do nothing, operation in mapping signals it is already taken care of
+    LLVM_DEBUG(llvm::dbgs() << "skipping mapped op: " << *op << "\n");
+  } else if (isa<acc::YieldOp>(op)) {
+    for (auto [operand, result] :
+         llvm::zip(op->getOperands(), op->getParentOp()->getResults())) {
+      result.replaceAllUsesWith(mapping.lookup(operand));
+    }
+  } else if (isa<scf::ExecuteRegionOp>(op)) {
+    processExecuteRegion(cast<scf::ExecuteRegionOp>(op));
+  } else if (op->getNumRegions() == 0 ||
+             isa<acc::OpenACCDialect>(op->getDialect())) {
+    processGenericOp(op);
+  } else {
+    processGenericOpWithRegions(op);
+  }
+}
+
+/// Fold `acc.par_width` to its launch operand or constant one.
+class RemoveParWidth : public OpRewritePattern<acc::ParWidthOp> {
+  using OpRewritePattern<acc::ParWidthOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(acc::ParWidthOp op,
+                                PatternRewriter &rewriter) const override {
+    if (Value launchArg = op.getLaunchArg()) {
+      rewriter.replaceOp(op, launchArg);
+    } else {
+      Value one = arith::ConstantIndexOp::create(rewriter, op.getLoc(), 1);
+      rewriter.replaceOp(op, one);
+    }
+    return success();
+  }
+};
+
+/// Rewrite pattern that lowers `acc.compute_region` via ACCCGToGPULowering.
+class ACCComputeRegionToGPUPattern
+    : public OpRewritePattern<acc::ComputeRegionOp> {
+public:
+  ACCComputeRegionToGPUPattern(MLIRContext *context,
+                               acc::OpenACCSupport &accSupport,
+                               const ACCCGToGPUOptions &options)
+      : OpRewritePattern<acc::ComputeRegionOp>(context), accSupport(accSupport),
+        options(options) {}
+
+  LogicalResult matchAndRewrite(acc::ComputeRegionOp op,
+                                PatternRewriter &rewriter) const override {
+    ACCCGToGPULowering kernelOpRewriter(op, rewriter, accSupport, options);
+    return kernelOpRewriter.rewrite();
+  }
+
+private:
+  acc::OpenACCSupport &accSupport;
+  const ACCCGToGPUOptions &options;
+};
+
+class ACCCGToGPU : public acc::impl::ACCCGToGPUBase<ACCCGToGPU> {
+public:
+  using acc::impl::ACCCGToGPUBase<ACCCGToGPU>::ACCCGToGPUBase;
+
+  void runOnOperation() override {
+    FunctionOpInterface funcOp = getOperation();
+    MLIRContext *context = funcOp->getContext();
+
+    assert(deviceType != mlir::acc::DeviceType::Host &&
+           deviceType != mlir::acc::DeviceType::Multicore &&
+           "ACCCGToGPU only supports GPU device types");
+    ACCCGToGPUOptions options;
+    options.deviceType = deviceType;
+    options.maxWorkgroupSharedMemory = maxWorkgroupSharedMemory;
+    options.maxThreadPrivateStack = maxThreadPrivateStack;
+    options.subgroupSize = subgroupSize;
+
+    // Try to get cached parent analysis first, fall back to local analysis.
+    std::optional<std::reference_wrapper<acc::OpenACCSupport>> cachedAnalysis =
+        getCachedParentAnalysis<acc::OpenACCSupport>(funcOp->getParentOp());
+    acc::OpenACCSupport &accSupport = cachedAnalysis
+                                          ? cachedAnalysis->get()
+                                          : getAnalysis<acc::OpenACCSupport>();
+
+    RewritePatternSet patterns(context);
+    patterns.insert<ACCComputeRegionToGPUPattern>(context, accSupport, options);
+    patterns.insert<RemoveParWidth>(context);
+    ConversionTarget target(getContext());
+    target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
+    target.addIllegalOp<acc::ComputeRegionOp, acc::ParWidthOp>();
+    if (failed(applyPartialConversion(getOperation(), target,
+                                      std::move(patterns)))) {
+      signalPassFailure();
+    }
+  }
+};
+
+} // namespace

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/OpenACC/Transforms/CMakeLists.txt
index d977028c283e6..11675640965b6 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/OpenACC/Transforms/CMakeLists.txt
@@ -1,5 +1,6 @@
 add_mlir_dialect_library(MLIROpenACCTransforms
   ACCBindRoutine.cpp
+  ACCCGToGPU.cpp
   ACCComputeLowering.cpp
   ACCRoutineLowering.cpp
   ACCRoutineToGPUFunc.cpp
@@ -35,11 +36,16 @@ add_mlir_dialect_library(MLIROpenACCTransforms
   MLIRAnalysis
   MLIRArithDialect
   MLIRArithUtils
+  MLIRComplexDialect
   MLIROpenACCAnalysis
   MLIROpenACCDialect
   MLIROpenACCUtils
   MLIRFuncDialect
+  MLIRGPUDialect
+  MLIRGPUUtils
   MLIRIR
+  MLIRMemRefDialect
+  MLIRNVVMDialect
   MLIRPass
   MLIRSCFDialect
   MLIRSupport

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-barrier-gang-private-init.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-barrier-gang-private-init.mlir
new file mode 100644
index 0000000000000..dc77d77e61a7e
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-barrier-gang-private-init.mlir
@@ -0,0 +1,107 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" --split-input-file | FileCheck %s
+
+// A gang-redundant init loop that writes a gang-private array, followed by a
+// loop that reads it, must be separated by a workgroup barrier.
+
+// CHECK-LABEL: @test_gang_private_init_barrier
+// CHECK: gpu.launch
+// CHECK: memref.store {{.*}} memref<8xf32>
+// CHECK: gpu.barrier
+// CHECK: scf.parallel
+
+func.func @test_gang_private_init_barrier(%arg0: memref<100x100xf32>) {
+  %c128 = arith.constant 128 : index
+  %c10 = arith.constant 10 : index
+  %c8 = arith.constant 8 : index
+  %c100 = arith.constant 100 : index
+  %12 = acc.copyin varPtr(%arg0 : memref<100x100xf32>) -> memref<100x100xf32> {dataClause = #acc<data_clause acc_copy>, name = "gp"}
+  %13 = acc.par_width %c10 {par_dim = #acc.par_dim<block_x>}
+  %14 = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment dataOperands(%12 : memref<100x100xf32>) {
+    %15 = acc.privatize [#acc<par_dims[block_x]>] : () -> !acc.private_type<memref<8xf32>>
+    acc.compute_region launch(%arg1 = %13, %arg2 = %14) ins(%arg10 = %12, %arg11 = %15) : (memref<100x100xf32>, !acc.private_type<memref<8xf32>>) {
+      %c8_b = arith.constant 8 : index
+      %c100_b = arith.constant 100 : index
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %19 = acc.private_local %arg11 : (!acc.private_type<memref<8xf32>>) -> memref<8xf32>
+      scf.parallel (%tx) = (%c0) to (%c8_b) step (%c1) {
+        scf.parallel (%s) = (%c0) to (%c8_b) step (%c1) {
+          %v = arith.index_cast %s : index to i32
+          %vf = arith.sitofp %v : i32 to f32
+          memref.store %vf, %19[%s] : memref<8xf32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_x]>}
+      scf.parallel (%bx2) = (%c0) to (%c100_b) step (%c1) {
+        scf.parallel (%tx2) = (%c0) to (%c100_b) step (%c1) {
+          scf.parallel (%s2) = (%c0) to (%c8_b) step (%c1) {
+            %r = memref.load %19[%s2] : memref<8xf32>
+            acc.predicate_region {
+              memref.store %r, %arg10[%bx2, %tx2] : memref<100x100xf32>
+            }
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// -----
+
+// A thread-private array must not introduce a workgroup barrier between init and read.
+
+// CHECK-LABEL: @test_thread_private_init_no_barrier
+// CHECK: memref.store {{.*}} memref<8xf32>
+// CHECK-NOT: gpu.barrier
+// CHECK: scf.parallel
+
+func.func @test_thread_private_init_no_barrier(%arg0: memref<100x100xf32>) {
+  %c128 = arith.constant 128 : index
+  %c10 = arith.constant 10 : index
+  %c8 = arith.constant 8 : index
+  %c100 = arith.constant 100 : index
+  %12 = acc.copyin varPtr(%arg0 : memref<100x100xf32>) -> memref<100x100xf32> {dataClause = #acc<data_clause acc_copy>, name = "gp"}
+  %13 = acc.par_width %c10 {par_dim = #acc.par_dim<block_x>}
+  %14 = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment dataOperands(%12 : memref<100x100xf32>) {
+    %15 = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<8xf32>>
+    acc.compute_region launch(%arg1 = %13, %arg2 = %14) ins(%arg10 = %12, %arg11 = %15) : (memref<100x100xf32>, !acc.private_type<memref<8xf32>>) {
+      %c8_b = arith.constant 8 : index
+      %c100_b = arith.constant 100 : index
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %19 = acc.private_local %arg11 : (!acc.private_type<memref<8xf32>>) -> memref<8xf32>
+      scf.parallel (%tx) = (%c0) to (%c8_b) step (%c1) {
+        scf.parallel (%s) = (%c0) to (%c8_b) step (%c1) {
+          %v = arith.index_cast %s : index to i32
+          %vf = arith.sitofp %v : i32 to f32
+          memref.store %vf, %19[%s] : memref<8xf32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_x]>}
+      scf.parallel (%bx2) = (%c0) to (%c100_b) step (%c1) {
+        scf.parallel (%tx2) = (%c0) to (%c100_b) step (%c1) {
+          scf.parallel (%s2) = (%c0) to (%c8_b) step (%c1) {
+            %r = memref.load %19[%s2] : memref<8xf32>
+            acc.predicate_region {
+              memref.store %r, %arg10[%bx2, %tx2] : memref<100x100xf32>
+            }
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-launch-mapping.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-launch-mapping.mlir
new file mode 100644
index 0000000000000..54b6b19529caf
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-launch-mapping.mlir
@@ -0,0 +1,277 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// ** tests for mapping par loops to gpu dimensions **
+
+// CHECK-LABEL: @par0_loop
+// CHECK: gpu.launch
+// CHECK: %[[c1:.*]] = arith.constant 1 : index
+// CHECK: %[[c4:.*]] = arith.constant 4 : index
+// CHECK: scf.parallel (%[[iv:.*]]) = (%[[c1]]) to (%[[c4]]) step (%[[c1]])
+func.func @par0_loop() {
+  acc.compute_region {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv) = (%c1) to (%c4) step (%c1) {
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[sequential]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @par1_loop
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+// CHECK-NOT: scf.parallel
+// CHECK: arith.addi %[[tidx]], %[[c4:.*]] : index
+func.func @par1_loop() {
+  acc.compute_region {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv) = (%c1) to (%c4) step (%c1) {
+      %i = arith.addi %iv, %c4 : index
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @par1_0_loop
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+// CHECK: scf.parallel (%[[iv2:.*]]) = (%[[c1:.*]]) to (%[[c4:.*]]) step (%[[c1]])
+// CHECK: arith.addi %[[tidx]], %[[c1]]
+// CHECK: arith.addi %[[iv2]], %[[c4]]
+func.func @par1_0_loop() {
+  acc.compute_region {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv1) = (%c1) to (%c4) step (%c1) {
+      scf.parallel (%iv2) = (%c1) to (%c4) step (%c1) {
+        %i1 = arith.addi %iv1, %c1 : index
+        %i2 = arith.addi %iv2, %c4 : index
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[sequential]>}
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @par0_1_loop
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+// CHECK: scf.parallel (%[[iv1:.*]]) = (%[[c1:.*]]) to (%[[c4:.*]]) step (%[[c1]])
+// CHECK: arith.addi %[[iv1]], %[[c1]]
+// CHECK: arith.addi %[[tidx]], %[[c4]]
+func.func @par0_1_loop() {
+  acc.compute_region {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv1) = (%c1) to (%c4) step (%c1) {
+      scf.parallel (%iv2) = (%c1) to (%c4) step (%c1) {
+        %i1 = arith.addi %iv1, %c1 : index
+        %i2 = arith.addi %iv2, %c4 : index
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_x]>}
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[sequential]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @par2_1_loop
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+// CHECK-NOT: scf.parallel
+// CHECK: arith.addi %[[tidy]], %[[c1:.*]] : index
+// CHECK: arith.addi %[[tidx]], %[[c4:.*]] : index
+func.func @par2_1_loop() {
+  acc.compute_region {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv1) = (%c1) to (%c4) step (%c1) {
+      scf.parallel (%iv2) = (%c1) to (%c4) step (%c1) {
+        %i1 = arith.addi %iv1, %c1 : index
+        %i2 = arith.addi %iv2, %c4 : index
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_x]>}
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @par2_0_1_loop
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+// CHECK: arith.addi %[[tidy]], %[[c1:.*]] : index
+// CHECK: scf.parallel (%[[iv2:.*]]) = (%[[c1]]) to (%[[c4:.*]]) step (%[[c1]])
+// CHECK: arith.addi %[[iv2]], %[[tidx]]
+func.func @par2_0_1_loop() {
+  %par_dim1 = acc.par_width {par_dim = #acc.par_dim<thread_x>}
+  %par_dim2 = acc.par_width {par_dim = #acc.par_dim<thread_y>}
+  acc.compute_region launch(%arg0 = %par_dim1, %arg1 = %par_dim2) {
+    %c1 = arith.constant 1 : index
+    %c4 = arith.constant 4 : index
+    scf.parallel (%iv1) = (%c1) to (%c4) step (%c1) {
+      %i1 = arith.addi %iv1, %c1 : index
+      scf.parallel (%iv2) = (%c1) to (%c4) step (%c1) {
+        scf.parallel (%iv3) = (%c1) to (%c4) step (%c1) {
+          %i2 = arith.addi %iv2, %iv3 : index
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[sequential]>}
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// ** tests for launch arguments **
+
+// CHECK-LABEL: @empty
+// CHECK-NOT: acc.compute_region
+// CHECK-NOT: acc.yield
+
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val:[a-z0-9_]+]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+func.func @empty() {
+  acc.compute_region {
+  ^bb1:
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @empty_some_known_launch_arg
+
+// CHECK: %[[bdimx_val:.*]] = arith.constant 32 : index
+// CHECK-NOT: acc.compute_region
+// CHECK-NOT: acc.yield
+
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val:[a-z0-9_]+]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val:[a-z0-9_]+]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val:[a-z0-9_]+]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[bdimx_val]], %[[bdimy:[a-z0-9_]+]] = %[[bdimy_val:[a-z0-9_]+]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val:[a-z0-9_]+]])
+func.func @empty_some_known_launch_arg() {
+  %c32 = arith.constant 32 : index
+  %par_dim1 = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%arg0 = %par_dim1) {
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+
+// CHECK-LABEL: @empty_all_known_launch_arg
+// CHECK: %[[VL:.*]] = arith.constant 2 : index
+// CHECK: %[[NW:.*]] = arith.constant 4 : index
+// CHECK: %[[bdimz_val:.*]] = arith.constant 8 : index
+
+// CHECK: %[[gdimx_val:.*]] = arith.constant 16 : index
+// CHECK: %[[gdimy_val:.*]] = arith.constant 32 : index
+// CHECK: %[[gdimz_val:.*]] = arith.constant 128 : index
+
+// CHECK-NOT: acc.compute_region
+// CHECK-NOT: acc.yield
+
+// 2D block: blockDimX = VL, blockDimY = NW
+// CHECK: gpu.launch
+// CHECK-SAME: blocks(%[[bidx:[a-z0-9_]+]], %[[bidy:[a-z0-9_]+]], %[[bidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[gdimx:[a-z0-9_]+]] = %[[gdimx_val]], %[[gdimy:[a-z0-9_]+]] = %[[gdimy_val]], %[[gdimz:[a-z0-9_]+]] = %[[gdimz_val]])
+// CHECK-SAME: threads(%[[tidx:[a-z0-9_]+]], %[[tidy:[a-z0-9_]+]], %[[tidz:[a-z0-9_]+]])
+// CHECK-SAME: in (%[[bdimx:[a-z0-9_]+]] = %[[VL]], %[[bdimy:[a-z0-9_]+]] = %[[NW]], %[[bdimz:[a-z0-9_]+]] = %[[bdimz_val]])
+func.func @empty_all_known_launch_arg() {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+  %c128 = arith.constant 128 : index
+
+  %par_dim1 = acc.par_width %c2 {par_dim = #acc.par_dim<thread_x>}
+  %par_dim2 = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %par_dim3 = acc.par_width %c8 {par_dim = #acc.par_dim<thread_z>}
+  %par_dim4 = acc.par_width %c16 {par_dim = #acc.par_dim<block_x>}
+  %par_dim5 = acc.par_width %c32 {par_dim = #acc.par_dim<block_y>}
+  %par_dim6 = acc.par_width %c128 {par_dim = #acc.par_dim<block_z>}
+  acc.compute_region launch(%tx = %par_dim1, %ty = %par_dim2, %tz = %par_dim3, %bx = %par_dim4, %by = %par_dim5, %bz = %par_dim6) {
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: @using_block_args
+// 2D block: blockDimX = VL, blockDimY = NW
+// CHECK: gpu.launch
+// CHECK-SAME: threads({{.*}}) in (%[[bdimx2:[a-z0-9_]+]] = %c2, %[[bdimy2:[a-z0-9_]+]] = %c4, %[[bdimz2:[a-z0-9_]+]] = %c8)
+// CHECK: %[[gdimx:.*]] = gpu.grid_dim x
+// CHECK: %[[gdimy:.*]] = gpu.grid_dim y
+// CHECK: %[[gdimz:.*]] = gpu.grid_dim z
+// CHECK: arith.addi %[[gdimx]], %[[gdimy]] : index
+// CHECK: arith.subi %[[gdimz]], %c2{{.*}} : index
+func.func @using_block_args(%arr : memref<?xf32>) {
+  %c2_pw = arith.constant 2 : index
+  %c4_pw = arith.constant 4 : index
+  %c8_pw = arith.constant 8 : index
+  %c16_pw = arith.constant 16 : index
+  %c32_pw = arith.constant 32 : index
+  %c128_pw = arith.constant 128 : index
+
+  %par_dim1 = acc.par_width %c2_pw {par_dim = #acc.par_dim<thread_x>}
+  %par_dim2 = acc.par_width %c4_pw {par_dim = #acc.par_dim<thread_y>}
+  %par_dim3 = acc.par_width %c8_pw {par_dim = #acc.par_dim<thread_z>}
+  %par_dim4 = acc.par_width %c16_pw {par_dim = #acc.par_dim<block_x>}
+  %par_dim5 = acc.par_width %c32_pw {par_dim = #acc.par_dim<block_y>}
+  %par_dim6 = acc.par_width %c128_pw {par_dim = #acc.par_dim<block_z>}
+  acc.compute_region launch(%tx = %par_dim1, %ty = %par_dim2, %tz = %par_dim3, %bx = %par_dim4, %by = %par_dim5, %bz = %par_dim6) ins(%arg10 = %arr) : (memref<?xf32>) {
+    %c0 = arith.constant 0 : index
+    %c2 = arith.constant 2 : index
+    %cst = arith.constant 1.000000e+01 : f32
+    scf.parallel (%arg8) = (%c0) to (%tx) step (%tx) {
+      %22 = arith.muli %ty, %tz : index
+      %23 = arith.addi %bx, %by : index
+      %24 = arith.subi %bz, %c2 : index
+      scf.parallel (%arg9) = (%22) to (%23) step (%24) {
+        memref.store %cst, %arg10[%arg9] : memref<?xf32>
+        scf.reduce
+      }
+    }
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region-reuse-barrier.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region-reuse-barrier.mlir
new file mode 100644
index 0000000000000..891ad37e39ab7
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region-reuse-barrier.mlir
@@ -0,0 +1,83 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A scalar bridged out of an acc.predicate_region is stored to gang-shared
+// memory by the predicated thread and read by all threads after a barrier.
+// When the predicate region sits in a gang/block-level sequential loop the
+// shared slot is reused every iteration, so a reuse barrier must also be
+// emitted BEFORE the predicated store: otherwise the next iteration's store
+// can clobber the slot before all threads have read the current value (WAR).
+
+// Inside a block-level sequential loop: barrier BEFORE and AFTER the store.
+// CHECK-LABEL: func.func @reuse_barrier_in_block_seq_loop
+// CHECK:       scf.parallel
+// CHECK:         gpu.barrier
+// CHECK-NEXT:    scf.if
+// CHECK:         gpu.barrier
+// CHECK:         memref.load
+func.func @reuse_barrier_in_block_seq_loop() {
+  %c256_pw = arith.constant 256 : index
+  %c1024_pw = arith.constant 1024 : index
+  %par_bx = acc.par_width %c256_pw {par_dim = #acc.par_dim<block_x>}
+  %par_tx = acc.par_width %c1024_pw {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv0 = acc.privatize : () -> !acc.private_type<memref<i64>>
+    acc.compute_region launch(%grid = %par_bx, %block = %par_tx) ins(%arg10 = %priv0) : (!acc.private_type<memref<i64>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c256 = arith.constant 256 : index
+      scf.parallel (%gang_iv) = (%c0) to (%grid) step (%c1) {
+        scf.parallel (%seq_iv) = (%c0) to (%c256) step (%c1) {
+          %pl0 = acc.private_local %arg10 : (!acc.private_type<memref<i64>>) -> memref<i64>
+          acc.predicate_region {
+            %val = arith.index_cast %seq_iv : index to i64
+            memref.store %val, %pl0[] : memref<i64>
+          }
+          %v0 = memref.load %pl0[] : memref<i64>
+          scf.parallel (%vec_iv) = (%c0) to (%block) step (%c1) {
+            memref.store %v0, %pl0[] : memref<i64>
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[thread_x]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// Not inside a sequential loop: only the post-store barrier, no pre-store one.
+// CHECK-LABEL: func.func @no_reuse_barrier_outside_seq_loop
+// CHECK-NOT:   gpu.barrier
+// CHECK:       scf.if
+// CHECK:       gpu.barrier
+// CHECK:       memref.load
+func.func @no_reuse_barrier_outside_seq_loop() {
+  %c256_pw = arith.constant 256 : index
+  %c1024_pw = arith.constant 1024 : index
+  %par_bx = acc.par_width %c256_pw {par_dim = #acc.par_dim<block_x>}
+  %par_tx = acc.par_width %c1024_pw {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv0 = acc.privatize : () -> !acc.private_type<memref<i64>>
+    acc.compute_region launch(%grid = %par_bx, %block = %par_tx) ins(%arg10 = %priv0) : (!acc.private_type<memref<i64>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      scf.parallel (%gang_iv) = (%c0) to (%grid) step (%c1) {
+        %pl0 = acc.private_local %arg10 : (!acc.private_type<memref<i64>>) -> memref<i64>
+        acc.predicate_region {
+          %val = arith.index_cast %gang_iv : index to i64
+          memref.store %val, %pl0[] : memref<i64>
+        }
+        %v0 = memref.load %pl0[] : memref<i64>
+        scf.parallel (%vec_iv) = (%c0) to (%block) step (%c1) {
+          memref.store %v0, %pl0[] : memref<i64>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region.mlir
new file mode 100644
index 0000000000000..8fc9832424222
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-predicate-region.mlir
@@ -0,0 +1,27 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// CHECK-LABEL: func.func @predicate_region_reduction
+// CHECK-NOT:   acc.predicate_region
+// CHECK:       scf.if
+// CHECK-NEXT:  memref.store %c0_i32, %alloca[] : memref<i32>
+
+func.func @predicate_region_reduction(%arg0: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %0 = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %1 = acc.par_width %c1 {par_dim = #acc.par_dim<thread_y>}
+  %2 = acc.par_width %c1 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    acc.compute_region launch(%arg1 = %0, %arg2 = %1, %arg3 = %2) ins(%arg10 = %arg0) : (memref<i32>) {
+      %c0_i32 = arith.constant 0 : i32
+      %4 = acc.reduction_init %arg10 <add> : memref<i32> {
+        %alloca = memref.alloca() : memref<i32>
+        acc.predicate_region {
+          memref.store %c0_i32, %alloca[] : memref<i32>
+        }
+        acc.yield %alloca : memref<i32>
+      }
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-private-local-gang-redundant.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-private-local-gang-redundant.mlir
new file mode 100644
index 0000000000000..1fb4b444fe5c8
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-private-local-gang-redundant.mlir
@@ -0,0 +1,36 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// Test that stores to worker-indexed private_local at top level preserve
+// redundant gang execution. The store should NOT be guarded by block_id.x
+// predicate - all gangs should execute the store.
+//
+// With the bug: arith.andi combines block_id.x==0 and thread_id.x==0
+// With the fix: only thread_id.x==0 check (no arith.andi needed)
+
+// CHECK-LABEL: func.func @gang_redundant_worker_private
+// CHECK:       gpu.launch
+// CHECK-NOT:   arith.andi
+// CHECK:       memref.store
+
+func.func @gang_redundant_worker_private() {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %0 = acc.par_width %c4 {par_dim = #acc.par_dim<block_x>}
+  %1 = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %2 = acc.par_width %c1 {par_dim = #acc.par_dim<thread_x>}
+  %3 = acc.privatize [#acc<par_dims[thread_y]>] : () -> !acc.private_type<memref<1xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%arg0 = %0, %arg1 = %1, %arg2 = %2) ins(%arg10 = %3) : (!acc.private_type<memref<1xi32>>) {
+      %loc = acc.private_local %arg10 : (!acc.private_type<memref<1xi32>>) -> memref<1xi32>
+      // Top-level predicate_region storing into the worker-private buffer.
+      // This should execute redundantly across gangs (no block_id.x predicate).
+      acc.predicate_region {
+        %c0 = arith.constant 0 : index
+        %c42 = arith.constant 42 : i32
+        memref.store %c42, %loc[%c0] : memref<1xi32>
+      }
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-local.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-local.mlir
new file mode 100644
index 0000000000000..9ac5385ec9674
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-local.mlir
@@ -0,0 +1,23 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(gpu.module(gpu.func(acc-cg-to-gpu)))" | FileCheck %s
+
+// CHECK-LABEL: gpu.func @test()
+// CHECK: %[[PRIVATIZED_MEMREF:[0-9]+]] = memref.load {{.*}} : memref<memref<4xi8>, #gpu.address_space<workgroup>>
+
+module attributes {gpu.container_module} {
+  acc.routine @test_routine func(@test) worker
+
+  gpu.module @cuda_device_mod {
+    gpu.func @test()
+        attributes {acc.specialized_routine = #acc.specialized_routine<@test_routine, <worker>, "test">} {
+      %c1 = arith.constant 1 : index
+      %1 = acc.par_width %c1 {par_dim = #acc.par_dim<thread_x>}
+      %2 = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+      %4 = acc.privatize : () -> !acc.private_type<memref<4xi8>>
+      acc.compute_region launch(%arg3 = %1, %arg4 = %2) ins(%arg10 = %4) : (!acc.private_type<memref<4xi8>>) {
+        %8 = acc.private_local %arg10 : (!acc.private_type<memref<4xi8>>) -> memref<f32>
+        acc.yield
+      } {origin = "acc.routine"}
+      gpu.return
+    }
+  }
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-routine-seq.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-routine-seq.mlir
new file mode 100644
index 0000000000000..2f67033adc03d
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-routine-seq.mlir
@@ -0,0 +1,37 @@
+
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(gpu.module(gpu.func(acc-cg-to-gpu)))" | FileCheck %s
+
+// Verify that a seq routine uses stack allocation (memref.alloca) for
+// privatized variables instead of workgroup shared memory + barriers.
+// This is a regression test for barriers in a sequential (single-thread)
+// kernel: workgroup barriers there are unnecessary and can miscompile.
+//
+// After ParallelDimAssignment, a seq kernel keeps seqDim as the sole
+// launch arg (internal scf.parallel loops get sequential par_dims).
+
+module attributes {gpu.container_module} {
+
+  acc.routine @routine_seq func(@seq_routine) seq
+
+  gpu.module @cuda_device_mod {
+
+    // CHECK-LABEL: gpu.func @seq_routine
+    // CHECK-NOT: workgroup
+    // CHECK-NOT: nvvm.barrier
+    // CHECK-NOT: gpu.barrier
+    gpu.func @seq_routine()
+        attributes {acc.specialized_routine = #acc.specialized_routine<@routine_seq, <seq>, "seq_routine">} {
+      %c1 = arith.constant 1 : index
+      %0 = acc.par_width %c1 {par_dim = #acc.par_dim<sequential>}
+      // CHECK: memref.alloca() : memref<10xi32>
+      // CHECK-NOT: memref.store {{.*}} #gpu.address_space<workgroup>
+      %1 = acc.privatize : () -> !acc.private_type<memref<10xi32>>
+      acc.compute_region launch(%arg0 = %0) ins(%arg10 = %1) : (!acc.private_type<memref<10xi32>>) {
+        %2 = acc.private_local %arg10 : (!acc.private_type<memref<10xi32>>) -> memref<10xi32>
+        acc.yield
+      } {origin = "acc.routine"}
+      gpu.return
+    }
+
+  }
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
new file mode 100644
index 0000000000000..281eae4b5b0f7
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// CHECK-LABEL: func.func @threadprivate
+// CHECK:       acc.privatize [#acc<par_dims[thread_x]>] : () -> !acc.private_type<memref<i32>>
+// CHECK:       gpu.launch
+// CHECK:         memref.alloca() : memref<i32>
+// CHECK-NOT:     acc.gpu_shared_memory
+
+func.func @threadprivate(%host: memref<i32>) {
+  %c99 = arith.constant 99 : i32
+  memref.store %c99, %host[] : memref<i32>
+  %init = memref.load %host[] : memref<i32>
+  %priv = acc.privatize [#acc<par_dims[thread_x]>] : () -> !acc.private_type<memref<i32>>
+
+  acc.compute_region ins(%priv_in = %priv, %init_in = %init) :
+      (!acc.private_type<memref<i32>>, i32) {
+    %local = acc.private_local %priv_in : (!acc.private_type<memref<i32>>) -> memref<i32>
+    memref.store %init_in, %local[] : memref<i32>
+    %v = memref.load %local[] : memref<i32>
+    %next = arith.addi %v, %v : i32
+    memref.store %next, %local[] : memref<i32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array-shared.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array-shared.mlir
new file mode 100644
index 0000000000000..aeb4539be83eb
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array-shared.mlir
@@ -0,0 +1,60 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A block-shared array accumulator (too large for a per-thread stack alloca)
+// must NOT run a per-element gpu.all_reduce: all threads share the slot, so
+// all_reduce would scale the block partial by the thread count. The block
+// partial is already in place and the atomic combine merges across blocks, so
+// the accumulate lowers away to nothing (like the block-only case).
+
+// CHECK-LABEL: func.func @array_reduction_shared
+// CHECK: gpu.launch
+// CHECK-NOT: gpu.all_reduce
+// CHECK-NOT: acc.reduction_accumulate_array
+// CHECK-NOT: acc.bounds
+
+func.func @array_reduction_shared(%arg0: memref<8192xi32>) {
+  %0 = acc.copyin varPtr(%arg0 : memref<8192xi32>) -> memref<8192xi32> {dataClause = #acc<data_clause acc_reduction>, implicit = true, name = "r"}
+  acc.kernel_environment dataOperands(%0 : memref<8192xi32>) {
+    %c1_pw = arith.constant 1 : index
+    %c128 = arith.constant 128 : index
+    %bx = acc.par_width %c1_pw {par_dim = #acc.par_dim<block_x>}
+    %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg2 = %0) : (memref<8192xi32>) {
+      %c8192 = arith.constant 8192 : index
+      %c0_i32 = arith.constant 0 : i32
+      %c1_i32 = arith.constant 1 : i32
+      %c1 = arith.constant 1 : index
+      %c0 = arith.constant 0 : index
+      %2 = acc.reduction_init %arg2 <add> : memref<8192xi32> {
+        %alloc = memref.alloc() : memref<8192xi32>
+        scf.parallel (%i) = (%c0) to (%c8192) step (%c1) {
+          memref.store %c0_i32, %alloc[%i] : memref<8192xi32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        acc.yield %alloc : memref<8192xi32>
+      }
+      scf.parallel (%bx_iv) = (%c0) to (%kbx) step (%c1) {
+        scf.parallel (%tx_iv) = (%c0) to (%ktx) step (%c1) {
+          %3 = memref.load %2[%c0] : memref<8192xi32>
+          %4 = arith.addi %3, %c1_i32 : i32
+          memref.store %4, %2[%c0] : memref<8192xi32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      %b = acc.bounds extent(%c8192 : index)
+      acc.reduction_accumulate_array %2 bounds(%b) <add> : memref<8192xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.reduction_combine_region %2 into %arg2 : memref<8192xi32> {
+        scf.for %i = %c0 to %c8192 step %c1 {
+          %3 = memref.load %2[%i] : memref<8192xi32>
+          %4 = memref.load %arg2[%i] : memref<8192xi32>
+          %5 = arith.addi %3, %4 : i32
+          memref.store %5, %arg2[%i] : memref<8192xi32>
+        }
+      }
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  acc.copyout accPtr(%0 : memref<8192xi32>) to varPtr(%arg0 : memref<8192xi32>) {dataClause = #acc<data_clause acc_reduction>, implicit = true, name = "r"}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
new file mode 100644
index 0000000000000..0b94fa033441e
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -0,0 +1,64 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A per-thread array accumulator (memref.alloca) for a block+thread reduction is
+// reduced element-by-element across the parallel dimensions with gpu.all_reduce -
+// the array analog of the scalar acc.reduction_accumulate.
+
+// CHECK-LABEL: func.func @array_reduction
+// CHECK: gpu.launch
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<2xi32>
+// CHECK-NOT: acc.reduction_accumulate_array
+// CHECK-NOT: acc.bounds
+// CHECK: scf.for %[[IV:.*]] = %{{.*}} to %{{.*}} step %{{.*}} {
+// CHECK:   %[[ELT:.*]] = memref.load %[[ALLOCA]][%[[IV]]] : memref<2xi32>
+// CHECK:   %[[RED:.*]] = gpu.all_reduce add %[[ELT]]
+// Per-thread alloca: the all_reduce result is stored unpredicated.
+// CHECK:   memref.store %[[RED]], %[[ALLOCA]][%[[IV]]] : memref<2xi32>
+// CHECK: }
+
+func.func @array_reduction(%arg0: memref<2xi32>) {
+  %0 = acc.copyin varPtr(%arg0 : memref<2xi32>) -> memref<2xi32> {dataClause = #acc<data_clause acc_reduction>, implicit = true, name = "r"}
+  acc.kernel_environment dataOperands(%0 : memref<2xi32>) {
+    %c1_pw = arith.constant 1 : index
+    %c128 = arith.constant 128 : index
+    %bx = acc.par_width %c1_pw {par_dim = #acc.par_dim<block_x>}
+    %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg2 = %0) : (memref<2xi32>) {
+      %c2 = arith.constant 2 : index
+      %c0_i32 = arith.constant 0 : i32
+      %c1_i32 = arith.constant 1 : i32
+      %c1 = arith.constant 1 : index
+      %c0 = arith.constant 0 : index
+      %2 = acc.reduction_init %arg2 <add> : memref<2xi32> {
+        %alloca = memref.alloca() : memref<2xi32>
+        scf.parallel (%i) = (%c0) to (%c2) step (%c1) {
+          memref.store %c0_i32, %alloca[%i] : memref<2xi32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        acc.yield %alloca : memref<2xi32>
+      }
+      scf.parallel (%bx_iv) = (%c0) to (%kbx) step (%c1) {
+        scf.parallel (%tx_iv) = (%c0) to (%ktx) step (%c1) {
+          %3 = memref.load %2[%c0] : memref<2xi32>
+          %4 = arith.addi %3, %c1_i32 : i32
+          memref.store %4, %2[%c0] : memref<2xi32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      %b = acc.bounds extent(%c2 : index)
+      acc.reduction_accumulate_array %2 bounds(%b) <add> : memref<2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.reduction_combine_region %2 into %arg2 : memref<2xi32> {
+        scf.for %i = %c0 to %c2 step %c1 {
+          %3 = memref.load %2[%i] : memref<2xi32>
+          %4 = memref.load %arg2[%i] : memref<2xi32>
+          %5 = arith.addi %3, %4 : i32
+          memref.store %5, %arg2[%i] : memref<2xi32>
+        }
+      }
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  acc.copyout accPtr(%0 : memref<2xi32>) to varPtr(%arg0 : memref<2xi32>) {dataClause = #acc<data_clause acc_reduction>, implicit = true, name = "r"}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-block-combine-no-reload.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-block-combine-no-reload.mlir
new file mode 100644
index 0000000000000..70506c6b719b6
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-block-combine-no-reload.mlir
@@ -0,0 +1,62 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A [block_x, thread_x] reduction whose accumulator is a grid-shared global
+// slot (not a per-thread alloca). The block-level reduction_combine must
+// atomically add the block-reduced gpu.all_reduce value directly into the
+// result, NOT reload the global slot first.
+
+// CHECK-LABEL: func.func @test_block_combine_no_reload
+// CHECK:       gpu.launch
+// CHECK:       %[[RED:.*]] = gpu.all_reduce add
+// CHECK-NEXT:  } {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+// CHECK-NOT:   memref.load
+// CHECK:       acc.atomic.update %[[RESULT:.*]] : memref<i32> {
+// CHECK-NEXT:  ^bb0(%[[ARG:.*]]: i32):
+// CHECK-NEXT:    %{{.*}} = arith.addi %[[RED]], %[[ARG]]
+
+module attributes {gpu.container_module} {
+  gpu.module @cuda_device_mod {
+    gpu.func @test_block_combine_no_reload_kernel() kernel {
+      gpu.return
+    }
+  }
+
+  func.func @test_block_combine_no_reload(%arg_slot: memref<i32>, %arg_res: memref<i32>) {
+    %c1_pw = arith.constant 1 : index
+    %c128 = arith.constant 128 : index
+    %bx = acc.par_width %c1_pw {par_dim = #acc.par_dim<block_x>}
+    %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%a_slot = %arg_slot, %a_res = %arg_res) : (memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c8 = arith.constant 8 : index
+      %c0_i32 = arith.constant 0 : i32
+      %c1_i32 = arith.constant 1 : i32
+      scf.parallel (%bx_iv) = (%c0) to (%kbx) step (%c1) {
+        scf.parallel (%tx_iv) = (%c0) to (%ktx) step (%c1) {
+          %row_step = arith.muli %c1, %ktx : index
+          %row_start = arith.muli %tx_iv, %c1 : index
+          %row_off = arith.addi %c0, %row_start : index
+          %row_stride = arith.muli %row_step, %kbx : index
+          %row_block = arith.muli %bx_iv, %row_step : index
+          %row_idx = arith.addi %row_off, %row_block : index
+          %inner_red = scf.parallel (%i) = (%row_idx) to (%c8) step (%row_stride) init (%c0_i32) -> i32 {
+            scf.reduce(%c1_i32 : i32) {
+            ^bb0(%lhs: i32, %rhs: i32):
+              %sum = arith.addi %lhs, %rhs : i32
+              scf.reduce.return %sum : i32
+            }
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          acc.reduction_accumulate %inner_red to %a_slot <add> : i32 -> memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.predicate_region {
+        acc.reduction_combine %a_slot into %a_res <add> : memref<i32> {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+      }
+      acc.yield
+    } {kernel_func_name = @test_block_combine_no_reload_kernel, kernel_module_name = @cuda_device_mod, origin = "acc.parallel"}
+    return
+  }
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-combine-region-private-dest.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-combine-region-private-dest.mlir
new file mode 100644
index 0000000000000..79137a42d9619
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-combine-region-private-dest.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A gang-vector reduction with an inner seq-loop reduction stages the
+// outer-scoped accumulator through an extra inner-loop private accumulator,
+// producing an acc.reduction_combine_region that writes one thread-private
+// accumulator into another while still carrying a block par_dim. That combine
+// must lower to a plain load/combine/store, not acc.atomic.update.
+
+// CHECK-LABEL: func.func @combine_region_private_dest
+// CHECK:         gpu.launch
+// CHECK:         %[[OUTER:.*]] = memref.alloca() : memref<i32>
+// CHECK:         memref.load %[[OUTER]][] : memref<i32>
+// CHECK-NEXT:    memref.load %{{.*}}[] : memref<i32>
+// CHECK-NEXT:    arith.addi
+// CHECK-NEXT:    memref.store %{{.*}}, %[[OUTER]][] : memref<i32>
+// CHECK:         acc.atomic.update %[[DEV:.*]] : memref<i32>
+// CHECK-NOT:     acc.atomic.update
+
+func.func @combine_region_private_dest(%arg: memref<i32>) {
+  %c1_pw = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1_pw {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %pv_outer = acc.privatize {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : () -> !acc.private_type<memref<i32>>
+  %pv_inner = acc.privatize {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : () -> !acc.private_type<memref<i32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%a_res = %arg, %po = %pv_outer, %pi = %pv_inner) : (memref<i32>, !acc.private_type<memref<i32>>, !acc.private_type<memref<i32>>) {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c8 = arith.constant 8 : index
+    %c0_i32 = arith.constant 0 : i32
+    %c1_i32 = arith.constant 1 : i32
+    %outer = acc.private_local %po {acc.par_dims = #acc<par_dims[block_x, thread_x]>, acc.var_name = #acc.var_name<"r">} : (!acc.private_type<memref<i32>>) -> memref<i32>
+    acc.predicate_region {
+      memref.store %c0_i32, %outer[] : memref<i32>
+    }
+    scf.parallel (%bx_iv) = (%c0) to (%kbx) step (%c1) {
+      scf.parallel (%tx_iv) = (%c0) to (%ktx) step (%c1) {
+        %row_step = arith.muli %c1, %ktx : index
+        %row_off = arith.muli %tx_iv, %c1 : index
+        %row_stride = arith.muli %row_step, %kbx : index
+        %row_block = arith.muli %bx_iv, %row_step : index
+        %row_idx = arith.addi %row_off, %row_block : index
+        scf.parallel (%i) = (%row_idx) to (%c8) step (%row_stride) {
+          %inner = acc.private_local %pi {acc.par_dims = #acc<par_dims[block_x, thread_x]>, acc.var_name = #acc.var_name<"r">} : (!acc.private_type<memref<i32>>) -> memref<i32>
+          acc.predicate_region {
+            memref.store %c0_i32, %inner[] : memref<i32>
+          }
+          acc.predicate_region {
+            memref.store %c1_i32, %inner[] : memref<i32>
+          }
+          %v = memref.load %inner[] : memref<i32>
+          acc.reduction_accumulate %v to %inner <add> : i32 -> memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+          acc.predicate_region {
+            acc.reduction_combine_region %inner into %outer : memref<i32> {
+              %la = memref.load %outer[] : memref<i32>
+              %lb = memref.load %inner[] : memref<i32>
+              %s = arith.addi %la, %lb : i32
+              memref.store %s, %outer[] : memref<i32>
+              acc.yield
+            } {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          }
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        %ov = memref.load %outer[] : memref<i32>
+        acc.reduction_accumulate %ov to %outer <add> : i32 -> memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_x]>}
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[block_x]>}
+    acc.predicate_region {
+      acc.reduction_combine_region %outer into %a_res : memref<i32> {
+        %da = memref.load %a_res[] : memref<i32>
+        %db = memref.load %outer[] : memref<i32>
+        %ds = arith.addi %da, %db : i32
+        memref.store %ds, %a_res[] : memref<i32>
+        acc.yield
+      } {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+    }
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reuse-barrier-sibling-region-privatize.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reuse-barrier-sibling-region-privatize.mlir
new file mode 100644
index 0000000000000..e0f8d3391c054
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reuse-barrier-sibling-region-privatize.mlir
@@ -0,0 +1,67 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// Two sibling acc.compute_regions share one enclosing sequential scf.for loop.
+// When lowering the region that stores to gang-private memory,
+// needsPreStoreReuseBarrier calls findFirstSequentialLoop, which walks up past
+// the compute region to the shared scf.for and then walks its whole body. That
+// walk descends into the sibling compute region and reaches a memref.load whose
+// acc.private_local is defined there: its getPrivatized() value is a block
+// argument of the sibling region, not the region being lowered. getPrivatizeOp
+// used to map that block arg through the region being lowered, so
+// ComputeRegionOp::getOperand returned a null Value and getDefiningOp() aborted
+// (dyn_cast on a non-existent value). Both regions must lower without crashing.
+
+// CHECK-LABEL: func.func @sibling_regions_share_seq_loop
+// CHECK:         scf.for
+// CHECK:           gpu.launch
+// CHECK:             memref.load
+// CHECK:           gpu.launch
+// CHECK:             gpu.barrier
+// CHECK:             gpu.barrier
+// CHECK:             memref.load
+func.func @sibling_regions_share_seq_loop() {
+  %c256_pw = arith.constant 256 : index
+  %c1024_pw = arith.constant 1024 : index
+  %par_bx = acc.par_width %c256_pw {par_dim = #acc.par_dim<block_x>}
+  %par_tx = acc.par_width %c1024_pw {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv0 = acc.privatize : () -> !acc.private_type<memref<i64>>
+    %priv1 = acc.privatize : () -> !acc.private_type<memref<i64>>
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c8 = arith.constant 8 : index
+    scf.for %t = %c0 to %c8 step %c1 {
+      // Sibling region: reads its own gang-private local.
+      acc.compute_region launch(%gridB = %par_bx, %blockB = %par_tx) ins(%argB = %priv1) : (!acc.private_type<memref<i64>>) {
+        %cb0 = arith.constant 0 : index
+        %cb1 = arith.constant 1 : index
+        scf.parallel (%gang_iv2) = (%cb0) to (%gridB) step (%cb1) {
+          %plB = acc.private_local %argB : (!acc.private_type<memref<i64>>) -> memref<i64>
+          %vb = memref.load %plB[] : memref<i64>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[block_x]>}
+        acc.yield
+      } {origin = "acc.parallel"}
+      // Region with a gang-private store inside a block-level predicate region.
+      acc.compute_region launch(%gridA = %par_bx, %blockA = %par_tx) ins(%argA = %priv0) : (!acc.private_type<memref<i64>>) {
+        %ca0 = arith.constant 0 : index
+        %ca1 = arith.constant 1 : index
+        scf.parallel (%gang_iv) = (%ca0) to (%gridA) step (%ca1) {
+          %plA = acc.private_local %argA : (!acc.private_type<memref<i64>>) -> memref<i64>
+          acc.predicate_region {
+            %val = arith.index_cast %gang_iv : index to i64
+            memref.store %val, %plA[] : memref<i64>
+          }
+          %v0 = memref.load %plA[] : memref<i64>
+          scf.parallel (%vec_iv) = (%ca0) to (%blockA) step (%ca1) {
+            memref.store %v0, %plA[] : memref<i64>
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[thread_x]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[block_x]>}
+        acc.yield
+      } {origin = "acc.parallel"}
+    }
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-call-nested-if.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-call-nested-if.mlir
new file mode 100644
index 0000000000000..c4e5a0559f5ec
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-call-nested-if.mlir
@@ -0,0 +1,35 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// Test that vector routine calls inside nested control flow (scf.if) are
+// properly detected. Without the fix, the routine call would not be found
+// and the predication would incorrectly add thread_id.x == 0, causing a hang.
+
+// CHECK-LABEL: @routine_call_in_nested_if
+// CHECK: gpu.launch
+// The routine calls should be inside the user's scf.if (based on block index),
+// NOT wrapped in an additional thread_id.x == 0 predicate.
+// CHECK: scf.if
+// CHECK: func.call @vector_routine
+// CHECK: } else {
+// CHECK: func.call @vector_routine
+func.func @routine_call_in_nested_if(%arg0: memref<4xf32>, %arg1: memref<4xf32>) {
+  acc.compute_region ins(%arg10 = %arg0, %arg11 = %arg1) : (memref<4xf32>, memref<4xf32>) {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    %c2 = arith.constant 2 : index
+    scf.parallel (%iv) = (%c0) to (%c2) step (%c1) {
+      %cond = arith.cmpi eq, %iv, %c0 : index
+      scf.if %cond {
+        func.call @vector_routine(%arg10) : (memref<4xf32>) -> ()
+      } else {
+        func.call @vector_routine(%arg11) : (memref<4xf32>) -> ()
+      }
+      scf.reduce
+    } {acc.par_dims = #acc<par_dims[block_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+func.func private @vector_routine(memref<4xf32>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_vector]>}
+acc.routine @acc_routine_vector func(@vector_routine) vector

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-worker-call-with-thread-y-reduction.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-worker-call-with-thread-y-reduction.mlir
new file mode 100644
index 0000000000000..26bc974d45892
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-routine-worker-call-with-thread-y-reduction.mlir
@@ -0,0 +1,74 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// Regression test for a workgroup-barrier deadlock when an `acc routine worker`
+// call sits inside an `acc.predicate_region` of a kernel that also has a
+// ThreadY reduction.  `processPredicateRegion`'s `hasThreadYReduction`
+// branch used to add ThreadX to the region's inactive parDims unless the
+// region contained a ThreadX (vector) routine call.  That exception did
+// not cover ThreadY (worker) routine calls, so the call ended up gated
+// on `thread_id.x == 0` — only 4 of the 128 workgroup threads entered the routine,
+// while its body still emitted workgroup-wide `gpu.barrier`s, deadlocking
+// across gang-loop iterations.
+//
+// After the fix, the worker routine call is NOT wrapped in a
+// `thread_id.x == 0` predicate: it sits directly under the gang scf.parallel
+// (i.e. the gpu.launch body, with no enclosing scf.if).
+
+// CHECK-LABEL: func.func @test_worker_routine_with_thread_y_reduction
+// CHECK:       gpu.launch blocks(%{{.*}}, %{{.*}}, %{{.*}}) in {{.*}} threads(%[[TIDX:.*]], %{{.*}}, %{{.*}}) in
+// The worker routine call appears at top level of the launch body, with
+// no `arith.cmpi eq, %tidx, ...` / `scf.if` sequence preceding it.
+// CHECK-NOT:   arith.cmpi eq, %[[TIDX]]
+// CHECK:       func.call @worker_routine
+
+module attributes {gpu.container_module} {
+  func.func private @worker_routine(memref<16xi32>) attributes {acc.routine_info = #acc.routine_info<[@acc_routine_worker]>}
+  acc.routine @acc_routine_worker func(@worker_routine) worker
+
+  gpu.module @cuda_device_mod {
+    gpu.func @test_worker_routine_kernel() kernel {
+      gpu.return
+    }
+  }
+
+  func.func @test_worker_routine_with_thread_y_reduction(%arg0: memref<16xi32>) {
+    %c4 = arith.constant 4 : index
+    %c32 = arith.constant 32 : index
+    %bx = acc.par_width %c4 {par_dim = #acc.par_dim<block_x>}
+    %tx = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+    %ty = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+    %priv = acc.privatize : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx, %kty = %ty)
+        ins(%arg10 = %priv, %arg11 = %arg0)
+        : (!acc.private_type<memref<i32>>, memref<16xi32>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c16 = arith.constant 16 : index
+      %c0_i32 = arith.constant 0 : i32
+      %c1_i32 = arith.constant 1 : i32
+      %out_priv = acc.private_local %arg10 : (!acc.private_type<memref<i32>>) -> memref<i32>
+      // Gang loop: contains the worker-routine call (in an
+      // acc.predicate_region as it would be after InsertPredicateRegions)
+      // followed by a worker reduction that sets hasThreadYReduction.
+      scf.parallel (%bx_iv) = (%c0) to (%kbx) step (%c1) {
+        acc.predicate_region {
+          func.call @worker_routine(%arg11) : (memref<16xi32>) -> ()
+        }
+        scf.parallel (%ty_iv) = (%c0) to (%kty) step (%c1) {
+          %loop_red = scf.parallel (%j) = (%ty_iv) to (%c16) step (%kty) init (%c0_i32) -> i32 {
+            scf.reduce(%c1_i32 : i32) {
+            ^bb0(%lhs: i32, %rhs: i32):
+              %sum = arith.addi %lhs, %rhs : i32
+              scf.reduce.return %sum : i32
+            }
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          acc.reduction_accumulate %loop_red to %out_priv <add> : i32 -> memref<i32> {par_dims = #acc<par_dims[thread_y]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {kernel_func_name = @test_worker_routine_kernel, kernel_module_name = @cuda_device_mod, origin = "acc.parallel"}
+    return
+  }
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
new file mode 100644
index 0000000000000..5df3d986fa5b0
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
@@ -0,0 +1,29 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" --verify-diagnostics
+
+// Verify that worker-private shared memory with dynamic num_workers emits
+// a diagnostic instead of silently miscompiling.
+
+func.func @test_worker_private_dynamic_nw(%nw: index) {
+  %c32 = arith.constant 32 : index
+  %c5 = arith.constant 5 : index
+  %block_x = acc.par_width %c5 {par_dim = #acc.par_dim<block_x>}
+  %thread_y = acc.par_width %nw {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv = acc.privatize : () -> !acc.private_type<memref<2xi32>>
+    // expected-error @below {{failed to legalize operation 'acc.compute_region'}}
+    acc.compute_region launch(%arg0 = %block_x, %arg1 = %thread_y, %arg2 = %thread_x) ins(%arg10 = %priv) : (!acc.private_type<memref<2xi32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%iv) = (%c0) to (%arg1) step (%c1) {
+        // expected-error @below {{worker-private variables in shared memory require compile-time constant num_workers}}
+        %local = acc.private_local %arg10 : (!acc.private_type<memref<2xi32>>) -> memref<2xi32>
+        memref.store %c0_i32, %local[%c0] : memref<2xi32>
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[thread_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-reuse-barrier.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-reuse-barrier.mlir
new file mode 100644
index 0000000000000..48670229b28f2
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-reuse-barrier.mlir
@@ -0,0 +1,64 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A vector (thread_x) worksharing loop nested in a SEQUENTIAL loop nested in a
+// worker (thread_y) loop reuses worker-private memory across the sequential
+// loop's iterations. The vector lanes of each worker row must reconverge before
+// the next iteration overwrites the shared slot (WAR). Because worker rows can
+// have divergent trip counts, the reconvergence barrier must be PER-ROW
+// (gpu.barrier scope<subgroup>), NOT a workgroup-wide barrier (which could deadlock).
+//
+// On a worker-level parent (no gang ancestor between the seq loop and its
+// parent) there is no block-level reconvergence path, so without the worker
+// per-row barrier no barrier is emitted here at all.
+
+// CHECK-LABEL: func.func @worker_seq_vector_reuse
+// CHECK:       gpu.launch
+// Enclosing sequential loop, then the vector loop, then a PER-ROW barrier
+// before the next sequential iteration. The barrier must be subgroup-scoped.
+// CHECK:         scf.parallel
+// CHECK:           scf.parallel
+// CHECK:             memref.load
+// CHECK:             memref.store
+// CHECK:           {{.*}}par_dims{{.*}}sequential
+// CHECK:           gpu.barrier scope <subgroup>
+// CHECK-NOT:       gpu.barrier{{[[:space:]]}}
+func.func @worker_seq_vector_reuse() {
+  %c256_pw = arith.constant 256 : index
+  %c4_pw = arith.constant 4 : index
+  %c32_pw = arith.constant 32 : index
+  %par_bx = acc.par_width %c256_pw {par_dim = #acc.par_dim<block_x>}
+  %par_ty = acc.par_width %c4_pw {par_dim = #acc.par_dim<thread_y>}
+  %par_tx = acc.par_width %c32_pw {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv0 = acc.privatize : () -> !acc.private_type<memref<32xf32>>
+    acc.compute_region launch(%grid = %par_bx, %worker = %par_ty, %block = %par_tx) ins(%arg10 = %priv0) : (!acc.private_type<memref<32xf32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c8 = arith.constant 8 : index
+      %c32 = arith.constant 32 : index
+      %cst = arith.constant 1.0 : f32
+      scf.parallel (%gang_iv) = (%c0) to (%grid) step (%c1) {
+        scf.parallel (%worker_iv) = (%c0) to (%worker) step (%c1) {
+          %pl0 = acc.private_local %arg10 : (!acc.private_type<memref<32xf32>>) -> memref<32xf32>
+          // Enclosing sequential loop: the slot is reused across its iterations.
+          scf.parallel (%k) = (%c0) to (%c8) step (%c1) {
+            // Vector (thread_x) loop with cross-lane reuse of worker-private memory.
+            scf.parallel (%vec_iv0) = (%c0) to (%block) step (%c1) {
+              scf.parallel (%seq0) = (%vec_iv0) to (%c32) step (%block) {
+                %v = memref.load %pl0[%seq0] : memref<32xf32>
+                memref.store %v, %pl0[%seq0] : memref<32xf32>
+                scf.reduce
+              } {acc.par_dims = #acc<par_dims[sequential]>}
+              scf.reduce
+            } {acc.par_dims = #acc<par_dims[thread_x]>}
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-subgroup-align.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-subgroup-align.mlir
new file mode 100644
index 0000000000000..86db31c9f8136
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-vector-subgroup-align.mlir
@@ -0,0 +1,55 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A worker per-row reconvergence barrier requires blockDim.x to be
+// subgroup-aligned, otherwise a warp spans two worker rows and the warp-scoped
+// barrier deadlocks under row-divergent control flow.
+//
+// Here thread_x = 16 (< subgroupSize 32) and thread_y = 4, and the vector loop
+// is in a sequential loop with a worker-row-dependent trip count. The fast-path
+// per-row barrier (gpu.barrier scope<subgroup>) must trigger the blockDim.x
+// alignment: blockDim.x is padded to 32 and blockDim.y reduced to 2, so each
+// warp holds exactly one worker row.
+
+// CHECK-LABEL: func.func @worker_divergent_subgroup_barrier
+// CHECK:       gpu.launch
+// CHECK-SAME:  threads({{.*}}) in (%{{[a-z0-9_]+}} = %c32, %{{[a-z0-9_]+}} = %c2,
+// CHECK:       gpu.barrier scope <subgroup>
+func.func @worker_divergent_subgroup_barrier() {
+  %c256_pw = arith.constant 256 : index
+  %c4_pw = arith.constant 4 : index
+  %c16_pw = arith.constant 16 : index
+  %par_bx = acc.par_width %c256_pw {par_dim = #acc.par_dim<block_x>}
+  %par_ty = acc.par_width %c4_pw {par_dim = #acc.par_dim<thread_y>}
+  %par_tx = acc.par_width %c16_pw {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %priv0 = acc.privatize : () -> !acc.private_type<memref<64xf32>>
+    acc.compute_region launch(%grid = %par_bx, %worker = %par_ty, %block = %par_tx) ins(%arg10 = %priv0) : (!acc.private_type<memref<64xf32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c64 = arith.constant 64 : index
+      %cst = arith.constant 1.0 : f32
+      scf.parallel (%gang_iv) = (%c0) to (%grid) step (%c1) {
+        scf.parallel (%worker_iv) = (%c0) to (%worker) step (%c1) {
+          %pl0 = acc.private_local %arg10 : (!acc.private_type<memref<64xf32>>) -> memref<64xf32>
+          // Worker-row-dependent trip count => rows sharing a warp diverge.
+          %ub = arith.addi %worker_iv, %c1 : index
+          scf.parallel (%k) = (%c0) to (%ub) step (%c1) {
+            scf.parallel (%vec_iv0) = (%c0) to (%block) step (%c1) {
+              scf.parallel (%seq0) = (%vec_iv0) to (%c64) step (%block) {
+                %v = memref.load %pl0[%seq0] : memref<64xf32>
+                memref.store %v, %pl0[%seq0] : memref<64xf32>
+                scf.reduce
+              } {acc.par_dims = #acc<par_dims[sequential]>}
+              scf.reduce
+            } {acc.par_dims = #acc<par_dims[thread_x]>}
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[sequential]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}


        

From llvmlistbot at llvm.org  Wed Jul 15 07:39:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 07:39:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 703e9b3 - [Flang][OpenMP] Implement DEPEND
 clause for TASKWAIT directive (#193568)
Message-ID: <6a579bb4.5a0412d2.82e8e.2816@mx.google.com>


Author: Phoebe Linck
Date: 2026-07-15T15:39:44+01:00
New Revision: 703e9b3b79f1520d05fceebabfacf31af4ad4e14

URL: https://github.com/llvm/llvm-project/commit/703e9b3b79f1520d05fceebabfacf31af4ad4e14
DIFF: https://github.com/llvm/llvm-project/commit/703e9b3b79f1520d05fceebabfacf31af4ad4e14.diff

LOG: [Flang][OpenMP] Implement DEPEND clause for TASKWAIT directive (#193568)

Implements support for the `DEPEND` clause on `TASKWAIT`, added in
OpenMP 5.0.

Added: 
    flang/test/Lower/OpenMP/taskwait-depend.f90
    mlir/test/Target/LLVMIR/openmp-taskwait-depend.mlir

Modified: 
    flang/lib/Lower/OpenMP/OpenMP.cpp
    llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
    llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
    mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Target/LLVMIR/openmp-todo.mlir

Removed: 
    flang/test/Lower/OpenMP/Todo/taskwait-depend.f90


################################################################################
diff  --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 8a7dde1063ea1..0d58703f3f15b 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2627,8 +2627,10 @@ static void genTaskwaitClauses(lower::AbstractConverter &converter,
                                const List<Clause> &clauses, mlir::Location loc,
                                mlir::omp::TaskwaitOperands &clauseOps) {
   ClauseProcessor cp(converter, semaCtx, clauses);
-  cp.processTODO<clause::Depend, clause::Nowait>(
-      loc, llvm::omp::Directive::OMPD_taskwait);
+  lower::StatementContext stmtCtx;
+  lower::SymMap &symTable = converter.getSymbolMap();
+  cp.processDepend(symTable, stmtCtx, clauseOps);
+  cp.processTODO<clause::Nowait>(loc, llvm::omp::Directive::OMPD_taskwait);
 }
 
 static void genWorkshareClauses(lower::AbstractConverter &converter,

diff  --git a/flang/test/Lower/OpenMP/Todo/taskwait-depend.f90 b/flang/test/Lower/OpenMP/Todo/taskwait-depend.f90
deleted file mode 100644
index d1f953be8802f..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/taskwait-depend.f90
+++ /dev/null
@@ -1,10 +0,0 @@
-! RUN: %not_todo_cmd bbc -emit-fir -fopenmp -o - %s -fopenmp-version=50 2>&1 | FileCheck %s
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -o - %s -fopenmp-version=50 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: Unhandled clause DEPEND in TASKWAIT construct
-subroutine omp_tw_depend
-  integer :: res
-  !$omp taskwait depend(out: res)
-  res = res + 1
-end subroutine omp_tw_depend
-

diff  --git a/flang/test/Lower/OpenMP/taskwait-depend.f90 b/flang/test/Lower/OpenMP/taskwait-depend.f90
new file mode 100644
index 0000000000000..21e3423fa4ae7
--- /dev/null
+++ b/flang/test/Lower/OpenMP/taskwait-depend.f90
@@ -0,0 +1,8 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=50 %s -o - | FileCheck %s
+
+!CHECK-LABEL: @_QPomp_taskwait_depend
+subroutine omp_taskwait_depend
+  integer :: x
+  !CHECK: omp.taskwait depend(taskdependin -> %{{.+}} : !fir.ref<i32>)
+  !$omp taskwait depend(in: x)
+end subroutine omp_taskwait_depend

diff  --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index 54d14602fbdbf..fee8c712998b5 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -1497,11 +1497,6 @@ class OpenMPIRBuilder {
   /// \param Loc The location where the flush directive was encountered
   LLVM_ABI void createFlush(const LocationDescription &Loc);
 
-  /// Generator for '#omp taskwait'
-  ///
-  /// \param Loc The location where the taskwait directive was encountered.
-  LLVM_ABI void createTaskwait(const LocationDescription &Loc);
-
   /// Generator for '#omp taskyield'
   ///
   /// \param Loc The location where the taskyield directive was encountered.
@@ -1540,7 +1535,14 @@ class OpenMPIRBuilder {
   LLVM_ABI void emitTaskDependency(IRBuilderBase &Builder, Value *Entry,
                                    const DependData &Dep);
 
-  /// Return the LLVM struct type matching runtime `kmp_task_affinity_info_t`.
+  /// Generator for '#omp taskwait'
+  ///
+  /// \param Loc The location where the taskwait directive was encountered.
+  /// \param Dependencies dependencies as specified by the 'depend' clause.
+  LLVM_ABI void createTaskwait(const LocationDescription &Loc,
+                               DependenciesInfo Dependencies = {});
+
+  ///  Return the LLVM struct type matching runtime `kmp_task_affinity_info_t`.
   /// `{ kmp_intptr_t base_addr; size_t len; flags (bitfield storage as i32) }`
   LLVM_ABI llvm::StructType *getKmpTaskAffinityInfoTy();
 

diff  --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 0dc704a87e4ed..dd00a3fae26e3 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2184,25 +2184,6 @@ void OpenMPIRBuilder::createFlush(const LocationDescription &Loc) {
   emitFlush(Loc);
 }
 
-void OpenMPIRBuilder::emitTaskwaitImpl(const LocationDescription &Loc) {
-  // Build call kmp_int32 __kmpc_omp_taskwait(ident_t *loc, kmp_int32
-  // global_tid);
-  uint32_t SrcLocStrSize;
-  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
-  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
-  Value *Args[] = {Ident, getOrCreateThreadID(Ident)};
-
-  // Ignore return result until untied tasks are supported.
-  createRuntimeFunctionCall(
-      getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_omp_taskwait), Args);
-}
-
-void OpenMPIRBuilder::createTaskwait(const LocationDescription &Loc) {
-  if (!updateToLocation(Loc))
-    return;
-  emitTaskwaitImpl(Loc);
-}
-
 void OpenMPIRBuilder::emitTaskyieldImpl(const LocationDescription &Loc) {
   // Build call __kmpc_omp_taskyield(loc, thread_id, 0);
   uint32_t SrcLocStrSize;
@@ -2293,6 +2274,69 @@ static Value *emitTaskDependencies(
   return DepArray;
 }
 
+void OpenMPIRBuilder::emitTaskwaitImpl(const LocationDescription &Loc) {
+  // Build call kmp_int32 __kmpc_omp_taskwait(ident_t *loc, kmp_int32
+  // global_tid);
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  Value *Args[] = {Ident, getOrCreateThreadID(Ident)};
+
+  // Ignore return result until untied tasks are supported.
+  createRuntimeFunctionCall(
+      getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_omp_taskwait), Args);
+}
+
+void OpenMPIRBuilder::createTaskwait(const LocationDescription &Loc,
+                                     DependenciesInfo Dependencies) {
+  if (!updateToLocation(Loc))
+    return;
+
+  Value *DepArray = nullptr;
+  Type *DepArrayTy = nullptr;
+  Value *NumDeps = nullptr;
+  if (Dependencies.DepArray) {
+    DepArray = Dependencies.DepArray;
+    NumDeps = Dependencies.NumDeps;
+  } else if (!Dependencies.Deps.empty()) {
+    InsertPointTy OldIP = Builder.saveIP();
+    BasicBlock &entryBB =
+        Builder.GetInsertBlock()->getParent()->getEntryBlock();
+    Builder.SetInsertPoint(&entryBB, entryBB.getFirstInsertionPt());
+
+    DepArrayTy = ArrayType::get(DependInfo, Dependencies.Deps.size());
+    DepArray = Builder.CreateAlloca(DepArrayTy, nullptr, ".dep.arr.addr");
+    NumDeps = Builder.getInt32(Dependencies.Deps.size());
+
+    Builder.restoreIP(OldIP);
+    for (const auto &[DepIdx, Dep] : enumerate(Dependencies.Deps)) {
+      Value *Base =
+          Builder.CreateConstInBoundsGEP2_64(DepArrayTy, DepArray, 0, DepIdx);
+      this->emitTaskDependency(Builder, Base, Dep);
+    }
+  }
+
+  if (DepArray) {
+    uint32_t SrcLocStrSize;
+    Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+    Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+    Value *Args[] = {
+        Ident,
+        getOrCreateThreadID(Ident),
+        NumDeps,
+        DepArray,
+        ConstantInt::get(Builder.getInt32Ty(), 0),
+        ConstantPointerNull::get(PointerType::getUnqual(M.getContext())),
+        ConstantInt::get(Builder.getInt32Ty(), false)};
+    createRuntimeFunctionCall(
+        getOrCreateRuntimeFunctionPtr(
+            omp::RuntimeFunction::OMPRTL___kmpc_omp_taskwait_deps_51),
+        Args);
+  } else {
+    emitTaskwaitImpl(Loc);
+  }
+}
+
 /// Create the task duplication function passed to kmpc_taskloop.
 Expected<Value *> OpenMPIRBuilder::createTaskDuplicationFunction(
     Type *PrivatesTy, int32_t PrivatesIndex, TaskDupCallbackTy DupCB) {

diff  --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 3a3798c3ad00b..67bc2bdf35619 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4687,10 +4687,15 @@ LogicalResult OrderedRegionOp::verify() { return verifyOrderedParent(**this); }
 
 void TaskwaitOp::build(OpBuilder &builder, OperationState &state,
                        const TaskwaitOperands &clauses) {
-  // TODO Store clauses in op: dependKinds, dependVars, nowait.
-  TaskwaitOp::build(builder, state, /*depend_kinds=*/nullptr,
-                    /*depend_vars=*/{}, /*depend_iterated_kinds=*/nullptr,
-                    /*depend_iterated=*/{}, /*nowait=*/nullptr);
+  // TODO Store clauses in op: depend_iterated_kinds, depend_iterated, nowait.
+  MLIRContext *ctx = builder.getContext();
+  TaskwaitOp::build(
+      builder, state,
+      /*depend_kinds=*/makeArrayAttr(ctx, clauses.dependKinds),
+      /*depend_vars=*/clauses.dependVars,
+      /*depend_iterated_kinds=*/makeArrayAttr(ctx, clauses.dependIteratedKinds),
+      /*depend_iterated=*/ValueRange(clauses.dependIterated),
+      /*nowait=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index bd773662b8bbd..f2fefa1f5a53f 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -511,10 +511,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkAllocate(op, result);
         checkTaskReductionByref(op, result);
       })
-      .Case([&](omp::TaskwaitOp op) {
-        checkDepend(op, result);
-        checkNowait(op, result);
-      })
+      .Case([&](omp::TaskwaitOp op) { checkNowait(op, result); })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
         checkInReduction(op, result);
@@ -4373,7 +4370,18 @@ convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
   if (failed(checkImplementationStatus(*twOp)))
     return failure();
 
-  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder.saveIP());
+  llvm::OpenMPIRBuilder::DependenciesInfo dds;
+  if (failed(buildDependData(
+          twOp.getDependVars(), twOp.getDependKinds(), twOp.getDependIterated(),
+          twOp.getDependIteratedKinds(), builder, moduleTranslation, dds))) {
+    return failure();
+  }
+
+  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder.saveIP(), dds);
+  if (dds.DepArray) {
+    builder.CreateFree(dds.DepArray);
+  }
+
   return success();
 }
 

diff  --git a/mlir/test/Target/LLVMIR/openmp-taskwait-depend.mlir b/mlir/test/Target/LLVMIR/openmp-taskwait-depend.mlir
new file mode 100644
index 0000000000000..24a26805df618
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-taskwait-depend.mlir
@@ -0,0 +1,53 @@
+// RUN: mlir-translate --mlir-to-llvmir -split-input-file %s | FileCheck %s
+
+llvm.func @taskwait_depend(%x: !llvm.ptr) {
+  omp.taskwait depend(taskdependout -> %x : !llvm.ptr)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @taskwait_depend
+// CHECK-SAME: (ptr[[xaddr:.+]])
+// CHECK: %[[dep_arr_addr:.+]] = alloca [1 x %struct.kmp_dep_info], align 8
+// CHECK: %[[omp_global_thread_num:.+]] = call i32 @__kmpc_global_thread_num({{.+}})
+// CHECK: call void @__kmpc_omp_taskwait_deps_51(ptr @{{.+}}, i32 %[[omp_global_thread_num]], i32 1, ptr %[[dep_arr_addr]], i32 0, ptr null, i32 0)
+
+llvm.func @taskwait_depend_iterator(%x: !llvm.ptr) {
+  %c1 = llvm.mlir.constant(1 : i64) : i64
+  %c10 = llvm.mlir.constant(10 : i64) : i64
+  %step = llvm.mlir.constant(1 : i64) : i64
+
+  %ix = omp.iterator(%i: i64) = (%c1 to %c10 step %step) {
+    omp.yield(%x : !llvm.ptr)
+  } -> !omp.iterated<!llvm.ptr>
+  omp.taskwait depend(taskdependin -> %ix : !omp.iterated<!llvm.ptr>)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @taskwait_depend_iterator
+// CHECK-SAME: (ptr[[xaddr:.+]])
+// CHECK: %[[dep_arr_addr:.+]] = tail call ptr @malloc(i64 %mallocsize)
+//
+// CHECK: omp_dep_iterator.header:
+// CHECK: %[[iv:.*]] = phi i64 [ 0, %omp_dep_iterator.preheader ], [ %[[next:.*]], %omp_dep_iterator.inc ]
+//
+// CHECK: omp_dep_iterator.cond:
+// CHECK: %[[cmp:.*]] = icmp ult i64 %[[iv]], 10
+// CHECK: br i1 %[[cmp]], label %omp_dep_iterator.body, label %omp_dep_iterator.exit
+//
+// CHECK: omp_dep_iterator.body:
+// CHECK: %[[idx:.*]] = add i64 0, %[[iv]]
+// CHECK: %[[entry:.*]] = getelementptr inbounds %struct.kmp_dep_info, ptr %[[dep_arr_addr]], i64 %[[idx]]
+// CHECK: %[[base_gep:.*]] = getelementptr inbounds nuw %struct.kmp_dep_info, ptr %[[entry]], i32 0, i32 0
+// CHECK: %[[ptrint:.*]] = ptrtoint ptr[[xaddr]] to i64
+// CHECK: store i64 %[[ptrint]], ptr %[[base_gep]]
+// CHECK: %[[len_gep:.*]] = getelementptr inbounds nuw %struct.kmp_dep_info, ptr %[[entry]], i32 0, i32 1
+// CHECK: store i64 8, ptr %[[len_gep]]
+// CHECK: %[[flags_gep:.*]] = getelementptr inbounds nuw %struct.kmp_dep_info, ptr %[[entry]], i32 0, i32 2
+// CHECK: store i8 1, ptr %[[flags_gep]]
+//
+// CHECK: omp_dep_iterator.inc:
+// CHECK: %[[next]] = add nuw i64 %[[iv]], 1
+//
+// CHECK: %[[omp_global_thread_num:.+]] = call i32 @__kmpc_global_thread_num({{.+}})
+// CHECK: call void @__kmpc_omp_taskwait_deps_51(ptr @{{.+}}, i32 %[[omp_global_thread_num]], i32 10, ptr %[[dep_arr_addr]], i32 0, ptr null, i32 0)
+// CHECK: tail call void @free(ptr %[[dep_arr_addr:.+]])

diff  --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 943b5bad6970f..e29659255f690 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -624,17 +624,6 @@ llvm.func @taskloop_reduction_two_arg_init(%lb : i32, %ub : i32, %step : i32, %x
 
 // -----
 
-llvm.func @taskwait_depend(%x: !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause depend in omp.taskwait operation}}
-  // expected-error at below {{LLVM Translation failed for operation: omp.taskwait}}
-  omp.taskwait depend(taskdependin -> %x : !llvm.ptr) {
-    omp.terminator
-  }
-  llvm.return
-}
-
-// -----
-
 llvm.func @taskwait_nowait() {
   // expected-error at below {{not yet implemented: Unhandled clause nowait in omp.taskwait operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.taskwait}}


        

From llvmlistbot at llvm.org  Wed Jul 15 07:39:51 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Wed, 15 Jul 2026 07:39:51 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [Flang][OpenMP] Implement
 DEPEND clause for TASKWAIT directive (PR #193568)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/193568@github.com>
Message-ID: <6a579bb7.e55dc341.1dd5fc.26cf@mx.google.com>


https://github.com/tblah closed https://github.com/llvm/llvm-project/pull/193568

From llvmlistbot at llvm.org  Wed Jul 15 07:47:21 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Wed, 15 Jul 2026 07:47:21 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a579d79.6365b89b.1d8070.f9a7@mx.google.com>


================
@@ -5134,6 +5134,7 @@ void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
   mlir::populateCIRPreLoweringPasses(pm);
   pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
+  pm.addPass(mlir::omp::createHostOpFilteringPass());
----------------
jsjodin wrote:

We still need to get the information passed through. We could bundle things in a struct but I think having an extra parameter might be more clear since it has OpenMP in the parameter name instead of having to look inside a struct. I'm open to either solution if you have a preference. 

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Wed Jul 15 07:47:40 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Wed, 15 Jul 2026 07:47:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a579d8c.492e23e4.304475.1ad4@mx.google.com>


================
@@ -189,6 +190,26 @@ class LinearClauseProcessor {
     }
   }
 
+  // Find linear iteration variable and save it for later updates
+  void initLinearIV(omp::SimdOp simdOp) {
+    auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
+    // NOTE iteration variables can only be linear in non-nested loops.
+    if (loopOp.getIVs().size() != 1)
+      return;
+    // The linear IV is the loop IV's store address.
+    BlockArgument arg = loopOp.getIVs().front();
+    for (const Operation *user : arg.getUsers()) {
+      if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
+        for (Value linearVar : simdOp.getLinearVars()) {
+          if (linearVar == storeOp.getAddr()) {
+            linearLoopIV = linearVar;
+            break;
----------------
tblah wrote:

What if there are multiple stores from different variables to the IV? Codex gave me this example:
```
// x then y
omp.simd linear(%x : !llvm.ptr = %step : i32,
                %y : !llvm.ptr = %step : i32) {
  omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
    llvm.store %iv, %x : i32, !llvm.ptr
    llvm.store %iv, %y : i32, !llvm.ptr
    omp.yield
  }
} {linear_var_types = [i32, i32]}
```

If x and y were flipped then the other one would be inferred as the linear result variable.

This is at least valid MLIR (if this can't be generated from Fortran maybe we need an MLIR verifier check to disallow this construction).

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Wed Jul 15 07:47:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 07:47:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 649466a - [NFC][OpenACC] fix doc formatting
 for reduction ops (#209739)
Message-ID: <6a579d9f.bc28b987.3d8071.2528@mx.google.com>


Author: Scott Manley
Date: 2026-07-15T09:47:54-05:00
New Revision: 649466a54f920970e99574d0d982b8ed35465c8f

URL: https://github.com/llvm/llvm-project/commit/649466a54f920970e99574d0d982b8ed35465c8f
DIFF: https://github.com/llvm/llvm-project/commit/649466a54f920970e99574d0d982b8ed35465c8f.diff

LOG: [NFC][OpenACC] fix doc formatting for reduction ops (#209739)

Was not printing correctly due to the matching `$`

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index cd64fba564a56..53adb91fc2983 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -42,8 +42,8 @@ def OpenACC_ReductionInitOp
     contains the recipe's init code and must yield a single value (the private
     reduction storage) via `acc.yield`.
 
-    The var operand is the original/shared reduction variable. The
-    reduction_operator specifies the reduction kind (e.g. add, mul).
+    The `var` operand is the original/shared reduction variable. The
+    `reduction_operator` specifies the reduction kind (e.g. add, mul).
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$var,
                        OpenACC_ReductionOperatorAttr:$reductionOperator);
@@ -74,8 +74,8 @@ def OpenACC_ReductionCombineRegionOp
     value(s) in the original/shared reduction variable. The region is
     terminated by `acc.yield` with no operands.
 
-    The destVar operand is the original/shared reduction variable.
-    The srcVar operand is typically the result of acc.reduction_init.
+    The `destVar` operand is the original/shared reduction variable.
+    The `srcVar` operand is typically the result of acc.reduction_init.
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$destVar,
                        OpenACC_AnyPointerOrMappableType:$srcVar);
@@ -113,11 +113,11 @@ def OpenACC_ReductionCombineOp: OpenACC_Op<"reduction_combine",
       memref.store %combine, %dest[] : memref<i32>
     ```
     
-    The $destMemref operand is a "pointer" to the original reduction 
-    variable (typically shared). The $srcMemref operand is a "pointer"
+    The `destMemref` operand is a "pointer" to the original reduction 
+    variable (typically shared). The `srcMemref` operand is a "pointer"
     to the partial sum of the reduction (typically private).
 
-    The $kind is the OpenACC reduction operator that determines how to 
+    The `kind` is the OpenACC reduction operator that determines how to 
     accumulate the two values.
   }];
 


        

From llvmlistbot at llvm.org  Wed Jul 15 07:48:02 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 15 Jul 2026 07:48:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NFC][OpenACC] fix doc formatting for
	reduction ops (PR #209739)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209739@github.com>
Message-ID: <6a579da2.9edb5c9d.5e50a.5a26@mx.google.com>


https://github.com/rscottmanley closed https://github.com/llvm/llvm-project/pull/209739

From llvmlistbot at llvm.org  Wed Jul 15 09:47:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 09:47:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a57b989.d681cab9.120c5c.e910@mx.google.com>


https://github.com/weimin023 updated https://github.com/llvm/llvm-project/pull/207342

>From aca6098aff076bee1214c0a85be73b9648e44711 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 3 Jul 2026 07:43:32 +0000
Subject: [PATCH 1/2] [NVGPU] Add FP8 (e4m3/e5m2) support to nvgpu.mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp      |  4 ++++
 mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp           |  7 ++++---
 .../Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir    | 12 ++++++++++++
 3 files changed, 20 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir

diff --git a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
index 948229bb53328..89cc2e5328a84 100644
--- a/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
+++ b/mlir/lib/Conversion/NVGPUToNVVM/NVGPUToNVVM.cpp
@@ -330,6 +330,10 @@ static FailureOr<NVVM::MMATypes> getNvvmMmaType(Type t) {
     return NVVM::MMATypes::f64;
   if (elType.isF32())
     return NVVM::MMATypes::tf32;
+  if (elType.isF8E4M3FN())
+    return NVVM::MMATypes::e4m3;
+  if (elType.isF8E5M2())
+    return NVVM::MMATypes::e5m2;
   return failure();
 }
 
diff --git a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
index c77ecb8431ad8..642e588488641 100644
--- a/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
+++ b/mlir/lib/Dialect/NVGPU/IR/NVGPUDialect.cpp
@@ -184,7 +184,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementA = 1;
     numElementB = 1;
   } else if (aType.isF32() || aType.isBF16() || aType.isF16() ||
-             aType.isInteger(8) || aType.isInteger(4)) {
+             aType.isInteger(8) || aType.isInteger(4) || aType.isF8E4M3FN() ||
+             aType.isF8E5M2()) {
     // 8-by-8-128b fundamental tensor core tile size
     int operandBitwidth = aType.getIntOrFloatBitWidth();
     shapeK = 128 / operandBitwidth; // 128b wide shapeK
@@ -193,8 +194,8 @@ static LogicalResult verifyMmaSyncOp(Operation *op,
     numElementB = 32 / operandBitwidth; // 32b wide operand B
   } else {
     return op->emitError()
-           << "expected input data type (i4,i8,f16,bf16,tf32,f64) "
-              "supported by "
+           << "expected input data type (i4,i8,f16,bf16,tf32,f64,"
+              "f8E4M3FN,f8E5M2) supported by "
            << op->getName();
   }
 
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
new file mode 100644
index 0000000000000..90c81af872f84
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}

>From df07327e29790c7fbf6516637d4f543ed883122f Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Wed, 15 Jul 2026 16:46:44 +0000
Subject: [PATCH 2/2] [NVGPU] Expand FP8 MMA lowering test coverage

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 .../NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir        | 12 -----
 .../NVGPUToNVVM/nvgpu-to-nvvm-mma-fp8.mlir    | 47 +++++++++++++++++++
 2 files changed, 47 insertions(+), 12 deletions(-)
 delete mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
 create mode 100644 mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-mma-fp8.mlir

diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
deleted file mode 100644
index 90c81af872f84..0000000000000
--- a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-fp8.mlir
+++ /dev/null
@@ -1,12 +0,0 @@
-// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
-
-// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
-// correct multiplicand PTX type.
-func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
-  // CHECK: nvvm.mma.sync
-  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
-  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
-  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
-  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
-  return %0 : vector<2x2xf32>
-}
diff --git a/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-mma-fp8.mlir b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-mma-fp8.mlir
new file mode 100644
index 0000000000000..5e7c7f8109f9a
--- /dev/null
+++ b/mlir/test/Conversion/NVGPUToNVVM/nvgpu-to-nvvm-mma-fp8.mlir
@@ -0,0 +1,47 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// CHECK-LABEL: @fp8_mma_e4m3_e4m3_m16n8k16
+func.func @fp8_mma_e4m3_e4m3_m16n8k16(%arg0: vector<2x4xf8E4M3FN>, %arg1: vector<1x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 16>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 16]} : (vector<2x4xf8E4M3FN>, vector<1x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fp8_mma_e4m3_e4m3_m16n8k32
+func.func @fp8_mma_e4m3_e4m3_m16n8k32(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e4m3>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E4M3FN>, vector<2x4xf8E4M3FN>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fp8_mma_e5m2_e5m2_m16n8k16
+func.func @fp8_mma_e5m2_e5m2_m16n8k16(%arg0: vector<2x4xf8E5M2>, %arg1: vector<1x4xf8E5M2>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e5m2>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e5m2>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 16>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 16]} : (vector<2x4xf8E5M2>, vector<1x4xf8E5M2>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @fp8_mma_e5m2_e5m2_m16n8k32
+func.func @fp8_mma_e5m2_e5m2_m16n8k32(%arg0: vector<4x4xf8E5M2>, %arg1: vector<2x4xf8E5M2>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
+  // CHECK: nvvm.mma.sync
+  // CHECK-SAME: multiplicandAPtxType = #nvvm.mma_type<e5m2>
+  // CHECK-SAME: multiplicandBPtxType = #nvvm.mma_type<e5m2>
+  // CHECK-SAME: shape = #nvvm.shape<m = 16, n = 8, k = 32>
+  %0 = nvgpu.mma.sync(%arg0, %arg1, %arg2) {mmaShape = [16, 8, 32]} : (vector<4x4xf8E5M2>, vector<2x4xf8E5M2>, vector<2x2xf32>) -> vector<2x2xf32>
+  return %0 : vector<2x2xf32>
+}


From llvmlistbot at llvm.org  Wed Jul 15 09:52:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 09:52:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a57bac6.66c0679a.3a40e5.1f3c@mx.google.com>


================
@@ -0,0 +1,12 @@
+// RUN: mlir-opt %s -convert-nvgpu-to-nvvm -split-input-file | FileCheck %s
+
+// Test that FP8 (e4m3) nvgpu.mma.sync lowers to nvvm.mma.sync with the
+// correct multiplicand PTX type.
+func.func @fp8_mma_sync(%arg0: vector<4x4xf8E4M3FN>, %arg1: vector<2x4xf8E4M3FN>, %arg2: vector<2x2xf32>) -> vector<2x2xf32> {
----------------
weimin023 wrote:

Hi @kvederni, 
Got it! I renamed the test file and added E4M3 and E5M2 coverage for both m16n8k16 and m16n8k32.

I did not add mixed E4M3/E5M2 A/B cases because `nvgpu.mma.sync` currently has an existing verifier constraint requiring matrixA and matrixB to have the same element type. 

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Wed Jul 15 12:46:39 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Wed, 15 Jul 2026 12:46:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a57e39f.4a342cbc.cf320.91f6@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/205464

>From ad1f5134c8a574c016d2dd2c0241afef08781a48 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 01/18] [mlir][EmitC] Create a pass to add a reflection map to
 a class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |   2 +
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  52 +++++
 .../Dialect/EmitC/Transforms/Transforms.h     |   7 +
 .../EmitC/Transforms/AddReflectionMap.cpp     | 181 ++++++++++++++++++
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |   1 +
 .../Dialect/EmitC/add-reflection-map.mlir     |  58 ++++++
 6 files changed, 301 insertions(+)
 create mode 100644 mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
 create mode 100644 mlir/test/Dialect/EmitC/add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 1af4aa06fa811..56596110a1e90 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,6 +16,8 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
+#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index c34c3303a6ab3..6da57c07a8047 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,4 +60,56 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `field-attr-name` option) that is an array containing a single string
+    attribute.
+
+
+    Example:
+
+    ```mlir
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.func @execute() { ... }
+    }
+
+    Becomes:
+
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
+      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
+          -> !emitc.ptr<!emitc.opaque<"char">> {
+        %0 = get_field @reflectionMap
+          : !emitc.opaque<"const std::map<std::string, char*>">
+        %1 = member_call_opaque %0 "at"(%arg0)
+          : !emitc.opaque<"const std::map<std::string, char*>">,
+            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      }
+      emitc.func @execute() { ... }
+    }
+    ```
+  }];
+  let dependentDialects = ["emitc::EmitCDialect"];
+  let options = [
+    Option<"fieldAttrName", "field-attr-name", "std::string",
+      /*default=*/"",
+      "Attribute key used to extract field names from a field's "
+      "attribute dictionary">,
+    ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
+      "Attribute keys that, if present on a field, exclude it "
+      "from the reflection map">
+  ];
+}
+
 #endif // MLIR_DIALECT_EMITC_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 791e545a8edcf..8eadfe9a2460d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/EmitC/IR/EmitC.h"
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -37,6 +38,12 @@ void populateWrapFuncInClass(
     RewritePatternSet &patterns, StringRef funcName,
     DenseMap<FuncOp, llvm::DenseSet<GlobalOp>> &globalsToMove);
 
+//===----------------------------------------------------------------------===//
+// The AddReflectionMap pass.
+//===----------------------------------------------------------------------===//
+
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+
 } // namespace emitc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
new file mode 100644
index 0000000000000..c1588463ffb7a
--- /dev/null
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -0,0 +1,181 @@
+//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(
+      builder, module.getLoc(), includeAttr,
+      /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class AddReflectionMapPass
+    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
+  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+
+    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+        
+        hasMapHdr = include == mapLibraryHeader;
+        hasStringHdr = include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect all field names
+    std::vector<std::pair<std::string, std::string>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
+        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
+          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
+          fieldNames.emplace_back(stringAttr.getValue().str(),
+                                  fieldOp.getName().str());
+          return;
+        }
+      }
+
+      bool shouldIgnore = false;
+      for (const std::string &ignoreAttr : excludedFieldAttrs) {
+        if (fieldOp->hasAttr(ignoreAttr)) {
+          shouldIgnore = true;
+          break;
+        }
+      }
+
+      if (shouldIgnore)
+        return;
+
+      fieldOp.emitError()
+          << "FieldOp must have a dictionary attribute named '"
+          << fieldAttrName << "'"
+          << "with an array containing a string attribute";
+    });
+
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
+          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+    }
+    reflectionMapContents += " }";
+
+    if (FuncOp executeFunc =
+            classOp.lookupSymbol<FuncOp>("operator()"))
+      rewriter.setInsertionPoint(executeFunc);
+    else {
+      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
+                             "to add reflection map";
+      return failure();
+    }
+
+    FieldOp reflectionMapField = FieldOp::create(
+        rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
+        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+
+    // Create getBufferForName method
+    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    FuncOp getBufferForNameFunc = FuncOp::create(
+        rewriter, reflectionMapField->getLoc(), "getBufferForName",
+        FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
+
+    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    rewriter.setInsertionPointToStart(body);
+    GetFieldOp mapField = GetFieldOp::create(
+        rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
+    Value nameArg = body->getArgument(0);
+    MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
+        rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
+        "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+
+    return success();
+  }
+
+private:
+  /// The name of the attribute on FieldOps that contains the field name
+  /// metadata for the reflection map.
+  StringRef fieldAttrName;
+
+  /// Attributes that, if present on a field, exclude it from the
+  /// reflection map.
+  llvm::SmallVector<std::string> excludedFieldAttrs;
+};
+
+void mlir::emitc::populateAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    llvm::ArrayRef<std::string> excludedFieldAttrs) {
+  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                      excludedFieldAttrs);
+}
\ No newline at end of file
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index baf67afc30072..dd8f014dc4737 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
+  AddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
new file mode 100644
index 0000000000000..a1ec0356e19f4
--- /dev/null
+++ b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+
+
+// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+// Test that a reflection map is created for fields with a certain named attribute
+// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
\ No newline at end of file

>From fce6ac458f4eaeb0a701082864f0f9dda68fb422 Mon Sep 17 00:00:00 2001
From: Bhavesh M <85327930+beamandala at users.noreply.github.com>
Date: Tue, 23 Jun 2026 02:37:42 +0530
Subject: [PATCH 02/18] [mlir][EmitC] Make `GlobalOps` `FieldOps` in
 wrap-emitc-func-in-class pass (#203641)

Update the `WrapFuncInClassPass` pass so that `GlobalOp`s are moved into
the `ClassOp` as `FieldOps`. This respects MLIR's behavior of resolving
references to the closest parent operation that defines a symbol table
which is the `ClassOp` that we are creating in this pass.

Without this change, references to a `GlobalOp` in `GetGlobalOp` are
failing to resolve.

Details:
- Identify `GlobalOp`s
- Create a `FieldOp` within the `ClassOp` for each `GlobalOp`
- Delete the `GlobalOp`s after all functions have been wrapped in a
class. Doing this after every function can cause an error when multiple
functions refer to the same `GlobalOp`(s) which would be deleted after
the first function is wrapped in a class.

Also renamed `fName` parameter in `populateWrapFuncInClass` to
`funcName` to match naming in `WrapFuncInClass`.

Based on PR #153452. Key differences:
- No size is set for the `globalsToMove` `SmallVector` type because I'm
not sure if the number of global variables is consistent across
different models.
- `GlobalOp`s are deleted after all functions have been processed.
- Instead of directly cloning the `GlobalOp`, an equivalent `FieldOp` is
created
- `GetGlobalOp`s are translated to `GetFieldOp`s

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8eadfe9a2460d..4c8d3978af0f1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,6 +13,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace mlir {
 namespace emitc {

>From 66bed5c47b37810c550c08f63a8f6dcf918c621d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 03/18] [mlir][EmitC] Create a pass to add a reflection map to
 a class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 4c8d3978af0f1..5a4781dc31ea8 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {

>From 1b949e08c522bf1befdba3fc0612dcad5d04a02a Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 24 Jun 2026 10:31:17 -0700
Subject: [PATCH 04/18] Formatting

---
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +--
 .../EmitC/Transforms/AddReflectionMap.cpp     | 47 +++++++++++--------
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 5a4781dc31ea8..8f1784f61f540 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,8 +13,6 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -44,7 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
+                                      StringRef fieldAttrName,
+                                      ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
index c1588463ffb7a..d444f3abbf4b1 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -31,9 +31,8 @@ constexpr const char *stringLibraryHeader = "string";
 
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
-  return IncludeOp::create(
-      builder, module.getLoc(), includeAttr,
-      /*is_standard_include=*/builder.getUnitAttr());
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
 }
 
 class AddReflectionMapPass
@@ -43,7 +42,8 @@ class AddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+    populateAddReflectionMapPatterns(patterns, fieldAttrName,
+                                     excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -55,7 +55,7 @@ class AddReflectionMapPass
 
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
-        
+
         hasMapHdr = include == mapLibraryHeader;
         hasStringHdr = include == stringLibraryHeader;
       }
@@ -82,7 +82,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   AddReflectionMapClass(MLIRContext *context, StringRef attrName,
                         llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
-        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+        excludedFieldAttrs(excludedFieldAttrs.begin(),
+                           excludedFieldAttrs.end()) {}
 
   LogicalResult matchAndRewrite(ClassOp classOp,
                                 PatternRewriter &rewriter) const override {
@@ -114,10 +115,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError()
-          << "FieldOp must have a dictionary attribute named '"
-          << fieldAttrName << "'"
-          << "with an array containing a string attribute";
+      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
+                          << fieldAttrName << "'"
+                          << "with an array containing a string attribute";
     });
 
     std::string reflectionMapContents;
@@ -129,28 +129,34 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc =
-            classOp.lookupSymbol<FuncOp>("operator()"))
+    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
-                             "to add reflection map";
+      classOp.emitError()
+          << "ClassOp must contain a function named 'operator()' "
+             "to add reflection map";
       return failure();
     }
 
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
-        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+        TypeAttr::get(mapType),
+        emitc::OpaqueAttr::get(context, reflectionMapContents));
 
     // Create getBufferForName method
-    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
-    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
-    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    emitc::OpaqueType nameType =
+        emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType =
+        emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType =
+        emitc::PointerType::get(rewriter.getContext(), charType);
     FuncOp getBufferForNameFunc = FuncOp::create(
         rewriter, reflectionMapField->getLoc(), "getBufferForName",
         FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
 
-    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    Block *body =
+        rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType},
+                             {reflectionMapField->getLoc()});
     rewriter.setInsertionPointToStart(body);
     GetFieldOp mapField = GetFieldOp::create(
         rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
@@ -158,7 +164,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
         rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
         "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
-    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(),
+                     lookupCall.getResult(0));
 
     return success();
   }

>From ab36f41166fd803ff45b2be3ca958125d82e940d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Fri, 26 Jun 2026 16:04:13 -0700
Subject: [PATCH 05/18] Address review comments

---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |  2 +-
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 10 ++-
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +-
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |  2 +-
 ...ectionMap.cpp => MLGOAddReflectionMap.cpp} | 76 +++++++++--------
 ...-map.mlir => mlgo-add-reflection-map.mlir} | 84 +++++++++++++++++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 57 +++++++++++++
 7 files changed, 188 insertions(+), 49 deletions(-)
 rename mlir/lib/Dialect/EmitC/Transforms/{AddReflectionMap.cpp => MLGOAddReflectionMap.cpp} (71%)
 rename mlir/test/Dialect/EmitC/{add-reflection-map.mlir => mlgo-add-reflection-map.mlir} (52%)
 create mode 100644 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 56596110a1e90..4f46d63774625 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,7 +16,7 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
-#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DECL_MLGOADDREFLECTIONMAPPASS
 
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 6da57c07a8047..3d17f53032a7a 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,7 +60,7 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
-def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
     This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
@@ -68,9 +68,11 @@ def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
     `field-attr-name` option) that is an array containing a single string
-    attribute.
-
-
+    attribute. If a field possesses both the attribute specified by
+    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
+    the `field-attr-name` attribute takes precedence and the field will be
+    included in the reflection map.
+    
     Example:
 
     ```mlir
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8f1784f61f540..6425614e1fbcd 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -42,9 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
-                                      StringRef fieldAttrName,
-                                      ArrayRef<std::string> excludedFieldAttrs);
+void populateMLGOAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index dd8f014dc4737..ede2cb5e208f0 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,7 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
-  AddReflectionMap.cpp
+  MLGOAddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
similarity index 71%
rename from mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
rename to mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index d444f3abbf4b1..9f506b10c722a 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -1,7 +1,7 @@
-//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt license information.
+// See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
@@ -22,7 +22,7 @@ using namespace emitc;
 
 namespace mlir {
 namespace emitc {
-#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 namespace {
@@ -35,15 +35,15 @@ IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
                            /*is_standard_include=*/builder.getUnitAttr());
 }
 
-class AddReflectionMapPass
-    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
-  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
   void runOnOperation() override {
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName,
-                                     excludedFieldAttrs);
+    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+                                         excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -56,8 +56,8 @@ class AddReflectionMapPass
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
 
-        hasMapHdr = include == mapLibraryHeader;
-        hasStringHdr = include == stringLibraryHeader;
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
       }
 
       if (hasMapHdr && hasStringHdr)
@@ -77,10 +77,10 @@ class AddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
-class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
-                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                            llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
@@ -93,33 +93,35 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
         context, "const std::map<std::string, char*>");
 
     // Collect all field names
-    std::vector<std::pair<std::string, std::string>> fieldNames;
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
-        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
-          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
-          fieldNames.emplace_back(stringAttr.getValue().str(),
-                                  fieldOp.getName().str());
-          return;
-        }
-      }
-
-      bool shouldIgnore = false;
-      for (const std::string &ignoreAttr : excludedFieldAttrs) {
-        if (fieldOp->hasAttr(ignoreAttr)) {
-          shouldIgnore = true;
-          break;
+      if (auto arrayAttr =
+              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
+        if (!arrayAttr.empty()) {
+          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
+            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
+            return;
+          }
         }
       }
 
+      bool shouldIgnore =
+          llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
+            return fieldOp->hasAttr(ignoreAttr);
+          });
       if (shouldIgnore)
         return;
 
       fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "'"
+                          << fieldAttrName << "' "
                           << "with an array containing a string attribute";
+      hasError = true;
     });
 
+    if (hasError || fieldNames.empty())
+      return failure();
+
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
@@ -138,6 +140,14 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       return failure();
     }
 
+    // To generate the following C++ code
+    // const std::map<std::string, char*> reflectionMap = {
+    //  { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+    //  { "some_feature", reinterpret_cast<char*>(&fieldName1) },
+    //  ...
+    // };
+    // This can be used to retrieve a pointer to the field's contents given the
+    // attribute string identifying the field
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
         TypeAttr::get(mapType),
@@ -180,9 +190,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   llvm::SmallVector<std::string> excludedFieldAttrs;
 };
 
-void mlir::emitc::populateAddReflectionMapPatterns(
+void mlir::emitc::populateMLGOAddReflectionMapPatterns(
     RewritePatternSet &patterns, StringRef fieldAttrName,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                      excludedFieldAttrs);
-}
\ No newline at end of file
+  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                          excludedFieldAttrs);
+}
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
similarity index 52%
rename from mlir/test/Dialect/EmitC/add-reflection-map.mlir
rename to mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index a1ec0356e19f4..2bea6cfaf2677 100644
--- a/mlir/test/Dialect/EmitC/add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,7 +1,7 @@
-// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
 
-
-// Tests that a reflection map is created for fields with a certain attribute.
+/// Tests that a reflection map is created for fields with a certain attribute.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -29,10 +29,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:  }
 
 // -----
-// Test that a reflection map is created for fields with a certain named attribute
-// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClass {
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -41,7 +42,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @actionClassExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @actionClassNoFields {
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if the ClassOp doesn't have
+/// a FunctionOp named operator()
+
+// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
+emitc.class @actionClassNoOperator {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if a FieldOp has the specified
+/// dictionary attribute with an array containing a type other than string
+
+emitc.class @actionClassNonStringAttr {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
\ No newline at end of file
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
new file mode 100644
index 0000000000000..eb8a8e58c8b77
--- /dev/null
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 0b94b71ff7e2b25032b3203b9a58d9fee2780ae4 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Mon, 6 Jul 2026 12:50:33 -0700
Subject: [PATCH 06/18] update MLGOAddReflectionMap to use notifyMatchFailure,
 update example to be more minimal and clear about the transformation, use
 range based for loop in reflection map construction

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 24 ++++++------------
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 25 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        |  5 +---
 3 files changed, 24 insertions(+), 30 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 3d17f53032a7a..29b053f3469c7 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -77,28 +77,20 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
 
     ```mlir
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.func @execute() { ... }
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      emitc.func @operator()() { ... }
     }
 
     Becomes:
 
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
-      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
-          -> !emitc.ptr<!emitc.opaque<"char">> {
-        %0 = get_field @reflectionMap
-          : !emitc.opaque<"const std::map<std::string, char*>">
-        %1 = member_call_opaque %0 "at"(%arg0)
-          : !emitc.opaque<"const std::map<std::string, char*>">,
-            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
-        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      // reflectionMap maps the string "feature0" to the address of @field0
+      emitc.field @reflectionMap : const std::map<string, char*> = {{"feature0", &field0}}
+      emitc.func @getBufferForName(%name: string) -> char* {
+        return reflectionMap.at(%name)
       }
-      emitc.func @execute() { ... }
+      emitc.func @operator()() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 9f506b10c722a..05a850c0e7551 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -113,9 +113,11 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "' "
-                          << "with an array containing a string attribute";
+      (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
+        diag << "FieldOp must have a dictionary attribute named '"
+             << fieldAttrName << "' "
+             << "with an array containing a string attribute";
+      });
       hasError = true;
     });
 
@@ -124,20 +126,23 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
-    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+
+      first = false;
       reflectionMapContents += llvm::formatv(
-          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
-          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
     }
     reflectionMapContents += " }";
 
     if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError()
-          << "ClassOp must contain a function named 'operator()' "
-             "to add reflection map";
-      return failure();
+      return rewriter.notifyMatchFailure(
+          classOp, "ClassOp must contain a function named 'operator()' "
+                   "to add reflection map");
     }
 
     // To generate the following C++ code
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2bea6cfaf2677..1f8647910ada9 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -63,7 +63,6 @@ emitc.class @actionClassExcluded {
 /// Test that the pass leaves IR unchanged if fields don't have any attributes
 
 emitc.class @actionClassNoAttrs {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -98,7 +97,6 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
 emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
@@ -113,7 +111,6 @@ emitc.class @actionClassNoOperator {
 /// dictionary attribute with an array containing a type other than string
 
 emitc.class @actionClassNonStringAttr {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return

>From d5e8301536841a90c281fd46a84414bb2507bd42 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 12:59:46 -0700
Subject: [PATCH 07/18] Update tests with formatting and wording changes,
 update pass description, only add headers if there was a match

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  4 +--
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 16 ++++++++-
 .../EmitC/mlgo-add-reflection-map.mlir        |  8 +++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 34 +++++++++++++++----
 4 files changed, 51 insertions(+), 11 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 29b053f3469c7..1e245e00c9b2d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -78,7 +78,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
     ```mlir
     emitc.class @MyClass {
       emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
 
     Becomes:
@@ -90,7 +90,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
       emitc.func @getBufferForName(%name: string) -> char* {
         return reflectionMap.at(%name)
       }
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 05a850c0e7551..666d9bc01dfdf 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -29,6 +29,15 @@ namespace {
 constexpr const char *mapLibraryHeader = "map";
 constexpr const char *stringLibraryHeader = "string";
 
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
   return IncludeOp::create(builder, module.getLoc(), includeAttr,
@@ -45,7 +54,12 @@ class MLGOAddReflectionMapPass
     populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
                                          excludedFieldAttrs);
 
-    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    if (!listener.patternApplied)
+      return;
+
     bool hasMapHdr = false;
     bool hasStringHdr = false;
     for (auto &op : *moduleOp.getBody()) {
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 1f8647910ada9..030de796b308c 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -60,7 +60,7 @@ emitc.class @actionClassExcluded {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if fields don't have any attributes
+/// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
 
 emitc.class @actionClassNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
@@ -69,6 +69,7 @@ emitc.class @actionClassNoAttrs {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @"operator()"() {
@@ -78,7 +79,7 @@ emitc.class @actionClassNoAttrs {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
 emitc.class @actionClassNoFields {
   emitc.func @"operator()"() {
@@ -86,6 +87,7 @@ emitc.class @actionClassNoFields {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoFields {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
@@ -101,6 +103,7 @@ emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:  }
@@ -117,6 +120,7 @@ emitc.class @actionClassNonStringAttr {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @"operator()"() {
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index eb8a8e58c8b77..e37b09bc6d611 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
-// Test that a reflection map and lookup function are generated in the class.
+/// Test that a reflection map and lookup function are generated in the class.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -17,10 +17,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
-// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
-// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
-// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
-// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+// CHECK-SAME:                                                         { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[ARG:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAR:v[0-9]+]] = reflectionMap.at([[ARG]]);
+// CHECK-NEXT:      return [[VAR]];
 // CHECK-NEXT:    }
 // CHECK-NEXT:    void operator()() {
 // CHECK-NEXT:      return;
@@ -29,7 +30,7 @@ emitc.class @actionClass {
 
 // -----
 
-// Test that fields with excluded attributes are ignored.
+/// Test that fields with excluded attributes are ignored.
 
 emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -55,3 +56,24 @@ emitc.class @actionClassExcluded {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
+
+// -----
+
+/// Test that translation doesn't add headers if the class does not match the pass.
+
+emitc.class @actionClassNoAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-NOT:   #include <map>
+// CHECK-NOT:   #include <string>
+// CHECK:       class actionClassNoAttrs {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 72d872a2e29df289955aedc5d85733a5f2d50b49 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 13:20:12 -0700
Subject: [PATCH 08/18] Update naming of classes in tests to be generic

---
 .../EmitC/mlgo-add-reflection-map.mlir        | 24 +++++++++----------
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 12 +++++-----
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 030de796b308c..bd6733f982fe0 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -3,7 +3,7 @@
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
-emitc.class @actionClass {
+emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -12,7 +12,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -33,7 +33,7 @@ emitc.class @actionClass {
 /// Test that a reflection map is created for fields with a certain named attribute
 /// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClassExcluded {
+emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -42,7 +42,7 @@ emitc.class @actionClassExcluded {
   }
 }
 
-// CHECK:       emitc.class @actionClassExcluded {
+// CHECK:       emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -62,7 +62,7 @@ emitc.class @actionClassExcluded {
 
 /// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
 
-emitc.class @actionClassNoAttrs {
+emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -70,7 +70,7 @@ emitc.class @actionClassNoAttrs {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-LABEL: emitc.class @fooNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
@@ -81,14 +81,14 @@ emitc.class @actionClassNoAttrs {
 
 /// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
-emitc.class @actionClassNoFields {
+emitc.class @fooNoFields {
   emitc.func @"operator()"() {
     return
   }
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-LABEL: emitc.class @fooNoFields {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
@@ -99,12 +99,12 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-emitc.class @actionClassNoOperator {
+emitc.class @fooNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-LABEL: emitc.class @fooNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:  }
 
@@ -113,7 +113,7 @@ emitc.class @actionClassNoOperator {
 /// Test that the pass returns with a match failure if a FieldOp has the specified
 /// dictionary attribute with an array containing a type other than string
 
-emitc.class @actionClassNonStringAttr {
+emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return
@@ -121,7 +121,7 @@ emitc.class @actionClassNonStringAttr {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-LABEL: emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index e37b09bc6d611..d0eab613672a7 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -2,7 +2,7 @@
 
 /// Test that a reflection map and lookup function are generated in the class.
 
-emitc.class @actionClass {
+emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -13,7 +13,7 @@ emitc.class @actionClass {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:  class foo {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
@@ -32,7 +32,7 @@ emitc.class @actionClass {
 
 /// Test that fields with excluded attributes are ignored.
 
-emitc.class @actionClassExcluded {
+emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -43,7 +43,7 @@ emitc.class @actionClassExcluded {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:  class fooExcluded {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
@@ -61,7 +61,7 @@ emitc.class @actionClassExcluded {
 
 /// Test that translation doesn't add headers if the class does not match the pass.
 
-emitc.class @actionClassNoAttrs {
+emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -70,7 +70,7 @@ emitc.class @actionClassNoAttrs {
 
 // CHECK-NOT:   #include <map>
 // CHECK-NOT:   #include <string>
-// CHECK:       class actionClassNoAttrs {
+// CHECK:       class fooNoAttrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    void operator()() {

>From d3c18f1c82a794408eea80cf63e8ca4fbcdd2fae Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 13:29:53 -0700
Subject: [PATCH 09/18] Set a filecheck pattern variable for \22 escape
 character to make test more readable

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index bd6733f982fe0..2220d419b677c 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s -DQUOTE="\[[QUOTE]]"
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -16,7 +16,7 @@ emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[QUOTE]]some_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -46,7 +46,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>

>From 076af93ded9db57ab7e0f3663ae7883ef6c6f05b Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:20:05 -0700
Subject: [PATCH 10/18] Relax the restriction where pass only matched classes
 with operator()() present

---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td   |  2 +-
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 13 ++++++------
 .../EmitC/mlgo-add-reflection-map.mlir        | 20 ++++++++++++-------
 3 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..0636232906f1b 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -1801,7 +1801,7 @@ def EmitC_ClassOp
 
   let extraClassDeclaration = [{
     // Returns the body block containing class members and methods.
-    Block &getBlock();
+    Block &getBlock() { return getBody().front(); }
   }];
 
   let assemblyFormat =
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 666d9bc01dfdf..12b173074e5bb 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -151,13 +151,12 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
-      rewriter.setInsertionPoint(executeFunc);
-    else {
-      return rewriter.notifyMatchFailure(
-          classOp, "ClassOp must contain a function named 'operator()' "
-                   "to add reflection map");
-    }
+    auto funcs = classOp.getBlock().getOps<FuncOp>();
+    auto it = funcs.begin();
+    if (it != funcs.end())
+      rewriter.setInsertionPoint(*it);
+    else
+      rewriter.setInsertionPointToEnd(&classOp.getBlock());
 
     // To generate the following C++ code
     // const std::map<std::string, char*> reflectionMap = {
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2220d419b677c..3374838aeac34 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s -DQUOTE="\[[QUOTE]]"
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -16,7 +16,7 @@ emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[QUOTE]]some_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -46,7 +46,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -96,16 +96,22 @@ emitc.class @fooNoFields {
 
 // -----
 
-/// Test that the pass returns with a match failure if the ClassOp doesn't have
-/// a FunctionOp named operator()
+/// Test that a reflection map is still created in the case that there are no
+/// functions in the class
 
 emitc.class @fooNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
-// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
 
 // -----
@@ -126,4 +132,4 @@ emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
-// CHECK-NEXT:  }
\ No newline at end of file
+// CHECK-NEXT:  }

>From ef2ab7ac726afd849958ce3f97fdb48a4e3f3f0f Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:23:11 -0700
Subject: [PATCH 11/18] Simplify test

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 3374838aeac34..5e7780dc23f78 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -7,7 +7,6 @@ emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
-    %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
 }
@@ -23,7 +22,6 @@ emitc.class @foo {
 // CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
 // CHECK-NEXT:    emitc.func @"operator()"() {
-// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }

>From aeeb53b5ff87882f41a42a0f4b4eaca9db7d6fbe Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:25:12 -0700
Subject: [PATCH 12/18] Update function name in tests to be general

---
 .../EmitC/mlgo-add-reflection-map.mlir        | 20 +++++++++----------
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 12 +++++------
 2 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 5e7780dc23f78..d18c1583cfd38 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -6,7 +6,7 @@
 emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -21,7 +21,7 @@ emitc.class @foo {
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -34,7 +34,7 @@ emitc.class @foo {
 emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -50,7 +50,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
@@ -62,7 +62,7 @@ emitc.class @fooExcluded {
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -70,7 +70,7 @@ emitc.class @fooNoAttrs {
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -80,14 +80,14 @@ emitc.class @fooNoAttrs {
 /// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
 emitc.class @fooNoFields {
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
 
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoFields {
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -119,7 +119,7 @@ emitc.class @fooNoOperator {
 
 emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -127,7 +127,7 @@ emitc.class @fooNonStringAttr {
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index d0eab613672a7..8aca1583f6313 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -5,7 +5,7 @@
 emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -23,7 +23,7 @@ emitc.class @foo {
 // CHECK-NEXT:      char* [[VAR:v[0-9]+]] = reflectionMap.at([[ARG]]);
 // CHECK-NEXT:      return [[VAR]];
 // CHECK-NEXT:    }
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
@@ -35,7 +35,7 @@ emitc.class @foo {
 emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -52,7 +52,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
 // CHECK-NEXT:      return [[VAL_2]];
 // CHECK-NEXT:    }
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
@@ -63,7 +63,7 @@ emitc.class @fooExcluded {
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -73,7 +73,7 @@ emitc.class @fooNoAttrs {
 // CHECK:       class fooNoAttrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };

>From 835b1bebfe57f56923b339f64b7cb84e0c927066 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 18:26:20 -0700
Subject: [PATCH 13/18] Switched pass input for field attr name to list to
 match the exclude list.

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  5 +--
 .../Dialect/EmitC/Transforms/Transforms.h     |  2 +-
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 41 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        | 29 ++++++++++++-
 .../Target/Cpp/mlgo-add-reflection-map.mlir   |  2 +-
 5 files changed, 56 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 1e245e00c9b2d..328852a369e28 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -96,9 +96,8 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   }];
   let dependentDialects = ["emitc::EmitCDialect"];
   let options = [
-    Option<"fieldAttrName", "field-attr-name", "std::string",
-      /*default=*/"",
-      "Attribute key used to extract field names from a field's "
+    ListOption<"includedFieldAttrs", "included-field-attrs", "std::string",
+      "Attribute keys used to extract field names from a field's "
       "attribute dictionary">,
     ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
       "Attribute keys that, if present on a field, exclude it "
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 6425614e1fbcd..b3899a2b070d9 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -43,7 +43,7 @@ void populateWrapFuncInClass(
 //===----------------------------------------------------------------------===//
 
 void populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, StringRef fieldAttrName,
+    RewritePatternSet &patterns, ArrayRef<std::string> includedFieldAttrs,
     ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 12b173074e5bb..afd1bf5b9b56b 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Support/FormatVariadic.h"
 
 using namespace mlir;
@@ -51,7 +52,7 @@ class MLGOAddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs,
                                          excludedFieldAttrs);
 
     PatternMatchListener listener;
@@ -93,9 +94,11 @@ class MLGOAddReflectionMapPass
 
 class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+  MLGOAddReflectionMapClass(MLIRContext *context,
+                            llvm::ArrayRef<std::string> includedFieldAttrs,
                             llvm::ArrayRef<std::string> excludedFieldAttrs)
-      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+      : OpRewritePattern<ClassOp>(context),
+        includedFieldAttrs(includedFieldAttrs),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
 
@@ -110,13 +113,16 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     std::vector<std::pair<StringRef, StringRef>> fieldNames;
     bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (auto arrayAttr =
-              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
-        if (!arrayAttr.empty()) {
-          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
-            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
-            return;
-          }
+      for (const auto &attr : includedFieldAttrs) {
+        auto arrayAttr = dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(attr));
+
+        if (!arrayAttr)
+          continue;
+
+        if (!arrayAttr.empty() && isa<StringAttr>(arrayAttr[0])) {
+          fieldNames.emplace_back(cast<StringAttr>(arrayAttr[0]).getValue(),
+                                  fieldOp.getName());
+          return;
         }
       }
 
@@ -129,7 +135,7 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
       (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
         diag << "FieldOp must have a dictionary attribute named '"
-             << fieldAttrName << "' "
+             << includedFieldAttrs << "' "
              << "with an array containing a string attribute";
       });
       hasError = true;
@@ -199,9 +205,10 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
   }
 
 private:
-  /// The name of the attribute on FieldOps that contains the field name
-  /// metadata for the reflection map.
-  StringRef fieldAttrName;
+  /// The names of the attributes on FieldOps that contain the field name
+  /// metadata for the reflection map. The pass matches the first attribute
+  /// present in the order they are specified in this list.
+  llvm::SmallVector<std::string> includedFieldAttrs;
 
   /// Attributes that, if present on a field, exclude it from the
   /// reflection map.
@@ -209,8 +216,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 };
 
 void mlir::emitc::populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, StringRef fieldAttrName,
+    RewritePatternSet &patterns, llvm::ArrayRef<std::string> includedFieldAttrs,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                          excludedFieldAttrs);
+  patterns.add<MLGOAddReflectionMapClass>(
+      patterns.getContext(), includedFieldAttrs, excludedFieldAttrs);
 }
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index d18c1583cfd38..ff5f44a1ac959 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
 // RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
 
 /// Tests that a reflection map is created for fields with a certain attribute.
@@ -131,3 +131,30 @@ emitc.class @fooNonStringAttr {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass matches one of the multiple included attributes.
+
+emitc.class @fooMultipleAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+  emitc.func @bar() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @fooMultipleAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> =
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL:.*]] = member_call_opaque %[[MAP]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index 8aca1583f6313..41329771562da 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
 /// Test that a reflection map and lookup function are generated in the class.
 

>From 35c5a45e4714fe5bba82a1cdac20a62078acd414 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:09:54 -0700
Subject: [PATCH 14/18] Add comments

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp        | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index afd1bf5b9b56b..881da3f4aee87 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -58,9 +58,12 @@ class MLGOAddReflectionMapPass
     PatternMatchListener listener;
     walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
 
+    // If nothing was matched, no reflection maps were added, removing the need
+    // to add include headers for map and string
     if (!listener.patternApplied)
       return;
 
+    // Check if the map and/or string headers are already present
     bool hasMapHdr = false;
     bool hasStringHdr = false;
     for (auto &op : *moduleOp.getBody()) {
@@ -109,7 +112,9 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
         context, "const std::map<std::string, char*>");
 
-    // Collect all field names
+    // Collect the names of all FieldOps that have one of the attributes in
+    // includedFieldAttrs to use the first element of the array attribute
+    // as the reflection map key
     std::vector<std::pair<StringRef, StringRef>> fieldNames;
     bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
@@ -126,6 +131,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
         }
       }
 
+      // If one of the attributes in excludedFieldAttrs is present,
+      // don't add this FieldOp
       bool shouldIgnore =
           llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
             return fieldOp->hasAttr(ignoreAttr);
@@ -144,6 +151,7 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     if (hasError || fieldNames.empty())
       return failure();
 
+    // Create reflection map contents
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     bool first = true;
@@ -157,6 +165,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
+    // Set insertion point before the first function or after all fields
+    // if there are no functions within the class
     auto funcs = classOp.getBlock().getOps<FuncOp>();
     auto it = funcs.begin();
     if (it != funcs.end())

>From e8af11de937a1571b252c3d85b9bf8894eaa9209 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:28:08 -0700
Subject: [PATCH 15/18] Update test comments

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 8 ++++----
 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir    | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index ff5f44a1ac959..f46bf4528048f 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -58,7 +58,7 @@ emitc.class @fooExcluded {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
+/// Test that the pass bails out and leaves IR unchanged if fields don't have any attributes
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
@@ -77,7 +77,7 @@ emitc.class @fooNoAttrs {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
+/// Test that the pass bails out and leaves IR unchanged if the ClassOp doesn't have any fields
 
 emitc.class @fooNoFields {
   emitc.func @bar() {
@@ -114,8 +114,8 @@ emitc.class @fooNoOperator {
 
 // -----
 
-/// Test that the pass returns with a match failure if a FieldOp has the specified
-/// dictionary attribute with an array containing a type other than string
+/// Test that the pass bails out if a FieldOp has the specified dictionary attribute
+/// with an array containing a type other than string
 
 emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index 41329771562da..e4c851b2889ff 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -59,7 +59,7 @@ emitc.class @fooExcluded {
 
 // -----
 
-/// Test that translation doesn't add headers if the class does not match the pass.
+/// Test that headers aren't added if the pass bails out.
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>

>From 1e7499d5aad7c38d328740c96b6b530129ca2823 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:37:17 -0700
Subject: [PATCH 16/18] Document OpRewritePattern

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 881da3f4aee87..23ade8ded7568 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -95,6 +95,36 @@ class MLGOAddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
+/// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
+/// and a lookup method.
+///
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes (e.g.,
+/// `emitc.field_ref`). Fields containing `excludedFieldAttrs` are skipped. All
+/// other fields must match one of the inclusion/exclusion filters, otherwise the
+/// pattern fails.
+///
+/// Before:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+///   emitc.func @bar() { return }
+/// }
+/// ```
+///
+/// After:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+///   emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+///     #emitc.opaque<"{ { \"another_feature\", reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+///     %ptr = member_call_opaque %map "at"(%name) : ...
+///     return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   }
+///   emitc.func @bar() { return }
+/// }
+/// ```
 class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
   MLGOAddReflectionMapClass(MLIRContext *context,

>From 654a2088ef8503ac1df77eb98381c688dac7818e Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:39:59 -0700
Subject: [PATCH 17/18] Formatting

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 29 ++++++++++---------
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 23ade8ded7568..adc6bcbe7f8be 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -98,29 +98,32 @@ class MLGOAddReflectionMapPass
 /// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
 /// and a lookup method.
 ///
-/// Fields to be mapped are identified via `includedFieldAttrs` attributes (e.g.,
-/// `emitc.field_ref`). Fields containing `excludedFieldAttrs` are skipped. All
-/// other fields must match one of the inclusion/exclusion filters, otherwise the
-/// pattern fails.
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes
+/// (e.g., `emitc.field_ref`). Fields containing `excludedFieldAttrs` are
+/// skipped. All other fields must match one of the inclusion/exclusion filters,
+/// otherwise the pattern fails.
 ///
 /// Before:
 /// ```mlir
 /// emitc.class @foo {
-///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-///   emitc.func @bar() { return }
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.func @bar() { return }
 /// }
 /// ```
 ///
 /// After:
 /// ```mlir
 /// emitc.class @foo {
-///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-///   emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-///     #emitc.opaque<"{ { \"another_feature\", reinterpret_cast<char*>(&fieldName0) } }">
-///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
-///     %map = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
-///     %ptr = member_call_opaque %map "at"(%name) : ...
-///     return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.field @reflectionMap : !emitc.opaque<"const
+///   std::map<std::string, char*>"> =
+///     #emitc.opaque<"{ { \"another_feature\",
+///     reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) ->
+///   !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const
+///     std::map<std::string, char*>"> %ptr = member_call_opaque %map
+///     "at"(%name) : ... return %ptr : !emitc.ptr<!emitc.opaque<"char">>
 ///   }
 ///   emitc.func @bar() { return }
 /// }

>From 3c32f1e81cc3ea002d79d2a02c8c5a3e20f6f735 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:43:52 -0700
Subject: [PATCH 18/18] Update pass documentation

---
 mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 328852a369e28..5c0dfe7bf69c1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -67,10 +67,10 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
     EmitC classes, enabling runtime lookup of class fields by name. 
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
-    `field-attr-name` option) that is an array containing a single string
-    attribute. If a field possesses both the attribute specified by
-    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
-    the `field-attr-name` attribute takes precedence and the field will be
+    `included-field-attrs` option) that is an array containing a single string
+    attribute. If a field possesses both an attribute specified by
+    `included-field-attrs` and an attribute specified in `excluded-field-attrs`,
+    the `included-field-attrs` attribute takes precedence and the field will be
     included in the reflection map.
     
     Example:


From llvmlistbot at llvm.org  Wed Jul 15 12:54:19 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Wed, 15 Jul 2026 12:54:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Handle repeated EmitC operands (PR
	#206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a57e56b.eb165e5c.3844ee.9e8b@mx.google.com>


aniragil wrote:

> Hi @marbre , @aniragil and @simon-camp Could take a moment to look at this? Thanks.

Sure, will take a look.

https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Wed Jul 15 19:36:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 19:36:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 31b3a0a - [mlir][emitc] Add increment and
	decrement ops (#208648)
Message-ID: <6a58439c.e55dc341.1dd5fc.a433@mx.google.com>


Author: Jianjian Guan
Date: 2026-07-16T10:36:07+08:00
New Revision: 31b3a0ab54e21a5d38fa83bdcce3e3aa07e2cb1e

URL: https://github.com/llvm/llvm-project/commit/31b3a0ab54e21a5d38fa83bdcce3e3aa07e2cb1e
DIFF: https://github.com/llvm/llvm-project/commit/31b3a0ab54e21a5d38fa83bdcce3e3aa07e2cb1e.diff

LOG: [mlir][emitc] Add increment and decrement ops (#208648)

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
    mlir/lib/Target/Cpp/TranslateToCpp.cpp
    mlir/test/Dialect/EmitC/invalid_ops.mlir
    mlir/test/Dialect/EmitC/ops.mlir
    mlir/test/Target/Cpp/common-cpp.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..c8417ce2f3de7 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -65,6 +65,29 @@ class EmitC_BinaryOp<string mnemonic, list<Trait> traits = []> :
   }];
 }
 
+// Base class for increment and decrement operations.
+class EmitC_IncDecOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, [
+        CExpressionInterface,
+        TypesMatchWith<"input and result reference the same type",
+                       "operand", "result",
+                       "::llvm::cast<emitc::LValueType>($_self).getValueType()">
+      ]> {
+  let summary = summaryStr;
+  let arguments = (ins
+    Res<EmitC_LValueType, "",
+        [MemRead<DefaultResource, 0, FullEffect>,
+         MemWrite<DefaultResource, 0, FullEffect>]>:$operand);
+  let results = (outs EmitCType:$result);
+  let assemblyFormat = "$operand attr-dict `:` type($operand)";
+
+  let extraClassDeclaration = [{
+    bool hasSideEffects() {
+      return true;
+    }
+  }];
+}
+
 // Types only used in binary arithmetic operations.
 def IntegerIndexOrOpaqueType : Type<CPred<"emitc::isIntegerIndexOrOpaqueType($_self)">,
 "integer, index or opaque type supported by EmitC">;
@@ -1259,6 +1282,58 @@ def EmitC_ConditionalOp : EmitC_Op<"conditional",
   }];
 }
 
+def EmitC_PreIncrementOp
+    : EmitC_IncDecOp<"pre_increment", "Pre-increment operation"> {
+  let description = [{
+    This operation models the C/C++ pre-increment operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_PostIncrementOp
+    : EmitC_IncDecOp<"post_increment", "Post-increment operation"> {
+  let description = [{
+    This operation models the C/C++ post-increment operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.post_increment %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_PreDecrementOp
+    : EmitC_IncDecOp<"pre_decrement", "Pre-decrement operation"> {
+  let description = [{
+    This operation models the C/C++ pre-decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.pre_decrement %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_PostDecrementOp
+    : EmitC_IncDecOp<"post_decrement", "Post-decrement operation"> {
+  let description = [{
+    This operation models the C/C++ post-decrement operator on an lvalue.
+
+    Example:
+
+    ```mlir
+    %0 = emitc.post_decrement %arg0 : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
 def EmitC_UnaryMinusOp : EmitC_UnaryOp<"unary_minus", []> {
   let summary = "Unary minus operation";
   let description = [{

diff  --git a/mlir/lib/Target/Cpp/TranslateToCpp.cpp b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
index 1bf87e4994d67..5ae51bd580d31 100644
--- a/mlir/lib/Target/Cpp/TranslateToCpp.cpp
+++ b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
@@ -112,6 +112,10 @@ static FailureOr<int> getOperatorPrecedence(Operation *operation) {
       .Case([&](emitc::MemberOfPtrOp op) { return 17; })
       .Case([&](emitc::MemberOp op) { return 17; })
       .Case([&](emitc::MulOp op) { return 13; })
+      .Case([&](emitc::PostDecrementOp op) { return 16; })
+      .Case([&](emitc::PostIncrementOp op) { return 16; })
+      .Case([&](emitc::PreDecrementOp op) { return 15; })
+      .Case([&](emitc::PreIncrementOp op) { return 15; })
       .Case([&](emitc::RemOp op) { return 13; })
       .Case([&](emitc::SubOp op) { return 12; })
       .Case([&](emitc::SubscriptOp op) { return 17; })
@@ -646,6 +650,22 @@ static LogicalResult printUnaryOperation(CppEmitter &emitter,
   return success();
 }
 
+static LogicalResult printPostfixUnaryOperation(CppEmitter &emitter,
+                                                Operation *operation,
+                                                StringRef unaryOperator) {
+  raw_ostream &os = emitter.ostream();
+
+  if (failed(emitter.emitAssignPrefix(*operation)))
+    return failure();
+
+  if (failed(emitter.emitOperand(operation->getOperand(0))))
+    return failure();
+
+  os << unaryOperator;
+
+  return success();
+}
+
 static LogicalResult printOperation(CppEmitter &emitter, emitc::AddOp addOp) {
   Operation *operation = addOp.getOperation();
 
@@ -1038,6 +1058,30 @@ static LogicalResult printOperation(CppEmitter &emitter,
   return printBinaryOperation(emitter, operation, "^");
 }
 
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreIncrementOp preIncrementOp) {
+  Operation *operation = preIncrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostIncrementOp postIncrementOp) {
+  Operation *operation = postIncrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "++");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PreDecrementOp preDecrementOp) {
+  Operation *operation = preDecrementOp.getOperation();
+  return printUnaryOperation(emitter, operation, "--");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::PostDecrementOp postDecrementOp) {
+  Operation *operation = postDecrementOp.getOperation();
+  return printPostfixUnaryOperation(emitter, operation, "--");
+}
+
 static LogicalResult printOperation(CppEmitter &emitter,
                                     emitc::UnaryPlusOp unaryPlusOp) {
   Operation *operation = unaryPlusOp.getOperation();
@@ -1896,7 +1940,9 @@ LogicalResult CppEmitter::emitOperation(Operation &op, bool trailingSemicolon) {
                 emitc::IncludeOp, emitc::LiteralOp, emitc::LoadOp,
                 emitc::LogicalAndOp, emitc::LogicalNotOp, emitc::LogicalOrOp,
                 emitc::MemberCallOpaqueOp, emitc::MemberOfPtrOp,
-                emitc::MemberOp, emitc::MulOp, emitc::RemOp, emitc::ReturnOp,
+                emitc::MemberOp, emitc::MulOp, emitc::PostDecrementOp,
+                emitc::PostIncrementOp, emitc::PreDecrementOp,
+                emitc::PreIncrementOp, emitc::RemOp, emitc::ReturnOp,
                 emitc::SubscriptOp, emitc::SubOp, emitc::SwitchOp,
                 emitc::UnaryMinusOp, emitc::UnaryPlusOp, emitc::VariableOp,
                 emitc::VerbatimOp>(

diff  --git a/mlir/test/Dialect/EmitC/invalid_ops.mlir b/mlir/test/Dialect/EmitC/invalid_ops.mlir
index eacf4fdedcbd3..3048160d1590c 100644
--- a/mlir/test/Dialect/EmitC/invalid_ops.mlir
+++ b/mlir/test/Dialect/EmitC/invalid_ops.mlir
@@ -981,3 +981,19 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   %1 = "emitc.dereference"(%arg0) : (!emitc.ptr<i32>) -> !emitc.lvalue<i8>
   return
 }
+
+// -----
+
+func.func @pre_increment_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.pre_increment"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}
+
+// -----
+
+func.func @post_decrement_unmatch_type(%arg0: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{failed to verify that input and result reference the same type}}
+  %1 = "emitc.post_decrement"(%arg0) : (!emitc.lvalue<i32>) -> i8
+  return
+}

diff  --git a/mlir/test/Dialect/EmitC/ops.mlir b/mlir/test/Dialect/EmitC/ops.mlir
index e456db4634892..e5c4af67c4896 100644
--- a/mlir/test/Dialect/EmitC/ops.mlir
+++ b/mlir/test/Dialect/EmitC/ops.mlir
@@ -148,6 +148,15 @@ func.func @unary(%arg0: i32) {
   return
 }
 
+func.func @inc_dec(%arg0: !emitc.ptr<i32>) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
 func.func @test_if(%arg0: i1, %arg1: f32) {
   emitc.if %arg0 {
      %0 = emitc.call_opaque "func_const"(%arg1) : (f32) -> i32

diff  --git a/mlir/test/Target/Cpp/common-cpp.mlir b/mlir/test/Target/Cpp/common-cpp.mlir
index 8e84433eaa2f3..1bffd59d4d44c 100644
--- a/mlir/test/Target/Cpp/common-cpp.mlir
+++ b/mlir/test/Target/Cpp/common-cpp.mlir
@@ -107,6 +107,41 @@ func.func @dereference(%arg0: !emitc.ptr<i32>) {
   return
 }
 
+// CHECK-LABEL: void inc_dec() {
+func.func @inc_dec() {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = ++[[V1]];
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]++;
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = --[[V1]];
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: int32_t [[V5:[^ ]*]] = [[V1]]--;
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
+  return
+}
+
+// CHECK-LABEL: int32_t inc_dec_expression() {
+func.func @inc_dec_expression() -> i32 {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %inc = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_increment %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V2:[^ ]*]] = [[V1]]++;
+  %dec = emitc.expression %v : (!emitc.lvalue<i32>) -> i32 {
+    %0 = emitc.post_decrement %v : !emitc.lvalue<i32>
+    emitc.yield %0 : i32
+  }
+  // CHECK-NEXT: int32_t [[V3:[^ ]*]] = [[V1]]--;
+  %r = emitc.add %inc, %dec : (i32, i32) -> i32
+  // CHECK-NEXT: int32_t [[V4:[^ ]*]] = [[V2]] + [[V3]];
+  // CHECK-NEXT: return [[V4]];
+  return %r : i32
+}
+
 // CHECK: void array_type(int32_t v1[3], float v2[10][20])
 func.func @array_type(%arg0: !emitc.array<3xi32>, %arg1: !emitc.array<10x20xf32>) {
   return


        

From llvmlistbot at llvm.org  Wed Jul 15 19:36:14 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Wed, 15 Jul 2026 19:36:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a58439e.f1c1923d.1bf95f.a391@mx.google.com>


https://github.com/jacquesguan closed https://github.com/llvm/llvm-project/pull/208648

From llvmlistbot at llvm.org  Wed Jul 15 19:40:08 2026
From: llvmlistbot at llvm.org (Pradeep Kumar)
Date: Wed, 15 Jul 2026 19:40:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVGPU] Add FP8 (e4m3/e5m2) support to
	nvgpu.mma.sync (PR #207342)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207342@github.com>
Message-ID: <6a584488.5a0412d2.82e8e.9c78@mx.google.com>


https://github.com/schwarzschild-radius approved this pull request.

LGTM, Thanks!

https://github.com/llvm/llvm-project/pull/207342

From llvmlistbot at llvm.org  Wed Jul 15 21:54:36 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 15 Jul 2026 21:54:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a58640c.510f576d.d8fd6.b532@mx.google.com>


https://github.com/nbpatel ready_for_review https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Wed Jul 15 21:55:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 21:55:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a586436.3cce8983.2f9901.d120@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Nishant Patel (nbpatel)

<details>
<summary>Changes</summary>

This PR adds rewriteWithRegrouping to unroll ConvertLayoutOp by extracting at the input inst_data granularity and inserting at the target inst_data granularity for cancellation during canonicalization.

Assisted by Claude

---
Full diff: https://github.com/llvm/llvm-project/pull/209822.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp (+110-14) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-blocking.mlir (+27-5) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..bcb9946373298 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -957,8 +957,92 @@ struct UnrollStoreMatrixOp : public UnrollPattern<xegpu::StoreMatrixOp> {
 /// after inst_data stripped. If it does, it will unroll the vector into
 /// multiple smaller vectors according to the target shape, and create multiple
 /// ConvertLayoutOp with the unrolled vectors and the stripped layouts.
+///
+/// When the input and target layouts have different inst_data, the source is
+/// extracted at the input inst_data granularity and the result is inserted at
+/// the target inst_data granularity, enabling slice cancellation during
+/// canonicalization.
 struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
   using UnrollPattern<xegpu::ConvertLayoutOp>::UnrollPattern;
+
+  /// Extracts source in `inTile` slices, regroups into `convTile`-sized
+  /// ConvertLayoutOps, and inserts the result in `outTile` slices.
+  /// Returns failure if the tiles do not evenly divide.
+  LogicalResult
+  rewriteWithRegrouping(xegpu::ConvertLayoutOp op, VectorType valueTy,
+                        ArrayRef<int64_t> convTile, ArrayRef<int64_t> inTile,
+                        ArrayRef<int64_t> outTile,
+                        xegpu::DistributeLayoutAttr inputLayout,
+                        xegpu::DistributeLayoutAttr targetLayout, Location loc,
+                        PatternRewriter &rewriter) const {
+    ArrayRef<int64_t> vecShape = valueTy.getShape();
+    if (!computeShapeRatio(vecShape, convTile) ||
+        !computeShapeRatio(convTile, inTile) ||
+        !computeShapeRatio(convTile, outTile))
+      return failure();
+
+    Type elemTy = valueTy.getElementType();
+    int64_t rank = valueTy.getRank();
+    VectorType convTy = VectorType::get(convTile, elemTy);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value source = op.getSource();
+    auto zeroOf = [&](VectorType ty) -> Value {
+      return arith::ConstantOp::create(
+          rewriter, loc, ty,
+          DenseElementsAttr::get(ty, rewriter.getZeroAttr(elemTy)));
+    };
+    auto addOffsets = [](ArrayRef<int64_t> a,
+                         ArrayRef<int64_t> b) -> SmallVector<int64_t> {
+      SmallVector<int64_t> res(a);
+      for (auto [r, v] : llvm::zip_equal(res, b))
+        r += v;
+      return res;
+    };
+
+    Value result = zeroOf(valueTy);
+    for (SmallVector<int64_t> convOff :
+         StaticTileOffsetRange(vecShape, convTile)) {
+      // Build the convert tile from inTile-sized slices of the source.
+      Value conv;
+      if (convTile == inTile) {
+        conv = vector::ExtractStridedSliceOp::create(
+            rewriter, loc, source, convOff, convTile, strides);
+      } else {
+        conv = zeroOf(convTy);
+        for (SmallVector<int64_t> inLocal :
+             StaticTileOffsetRange(convTile, inTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, source, addOffsets(convOff, inLocal), inTile,
+              strides);
+          conv = vector::InsertStridedSliceOp::create(rewriter, loc, piece,
+                                                      conv, inLocal, strides);
+        }
+      }
+
+      conv = xegpu::ConvertLayoutOp::create(rewriter, loc, convTy, conv,
+                                            inputLayout, targetLayout);
+
+      // Write the converted tile into the result as outTile-sized slices.
+      if (convTile == outTile) {
+        result = vector::InsertStridedSliceOp::create(rewriter, loc, conv,
+                                                      result, convOff, strides);
+      } else {
+        for (SmallVector<int64_t> outLocal :
+             StaticTileOffsetRange(convTile, outTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, conv, outLocal, outTile, strides);
+          result = vector::InsertStridedSliceOp::create(
+              rewriter, loc, piece, result, addOffsets(convOff, outLocal),
+              strides);
+        }
+      }
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+
   LogicalResult matchAndRewrite(xegpu::ConvertLayoutOp op,
                                 PatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
@@ -974,8 +1058,10 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
       return success();
     }
 
-    if (inputLayout.getEffectiveInstDataAsInt().empty() ||
-        targetLayout.getEffectiveInstDataAsInt().empty())
+    // Capture inst_data granularities before stripping them.
+    SmallVector<int64_t> inTile = inputLayout.getEffectiveInstDataAsInt();
+    SmallVector<int64_t> outTile = targetLayout.getEffectiveInstDataAsInt();
+    if (inTile.empty() || outTile.empty())
       return rewriter.notifyMatchFailure(op, "Not a target ConvertLayoutOp.");
 
     inputLayout = inputLayout.dropInstData();
@@ -988,20 +1074,30 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     if (!targetShape || targetShape->size() != (size_t)valueTy.getRank())
       return failure();
 
-    Value newSource = op.getSource();
+    // Nothing to convert if layouts match after stripping inst_data.
+    if (!inputLayout || !targetLayout || inputLayout.isEqualTo(targetLayout)) {
+      rewriter.replaceOp(op, op.getSource());
+      return success();
+    }
+
+    // Try regrouping: extract at inTile, convert, insert at outTile.
+    if (succeeded(rewriteWithRegrouping(op, valueTy, *targetShape, inTile,
+                                        outTile, inputLayout, targetLayout, loc,
+                                        rewriter)))
+      return success();
+
+    // Fallback: pack/unpack at the convert tile granularity.
+    SmallVector<Type> convertedValTypes =
+        getUnrolledTypes(valueTy, *targetShape);
+    SmallVector<Value> convertedValues =
+        pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
     SmallVector<Value> newOps;
-    if (inputLayout && targetLayout && !inputLayout.isEqualTo(targetLayout)) {
-      SmallVector<Type> convertedValTypes =
-          getUnrolledTypes(valueTy, *targetShape);
-      SmallVector<Value> convertedValues =
-          pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
-      for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
-        auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
-                                                    inputLayout, targetLayout);
-        newOps.push_back(newOp);
-      }
-      newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
+    for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
+      auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
+                                                  inputLayout, targetLayout);
+      newOps.push_back(newOp);
     }
+    Value newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
 
     rewriter.replaceOp(op, newSource);
     return success();
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..27a88efcf443c 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -463,19 +463,19 @@ gpu.module @test_kernel {
   //CHECK: gpu.func @convert_layout([[arg0:%.+]]: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
   //CHECK: [[cst:%.+]] = arith.constant dense<0.000000e+00> : vector<8x32x2xf16>
   //CHECK: [[e0:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [0, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
-  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
   //CHECK: [[c0:%.+]] = xegpu.convert_layout [[e0]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
-  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
   //CHECK: [[e2:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
+  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
+  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m0:%.+]] = math.exp [[e2]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i0:%.+]] = vector.insert_strided_slice [[m0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m1:%.+]] = math.exp [[e3]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i1:%.+]] = vector.insert_strided_slice [[m1]], [[i0]] {offsets = [0, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m2:%.+]] = math.exp [[e4]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i2:%.+]] = vector.insert_strided_slice [[m2]], [[i1]] {offsets = [4, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m3:%.+]] = math.exp [[e5]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i3:%.+]] = vector.insert_strided_slice [[m3]], [[i2]] {offsets = [4, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
   //CHECK: gpu.return [[i3]] : vector<8x32x2xf16>
@@ -488,6 +488,28 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 16] vs [8, 64]).
+#in = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#tgt = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 16]
+  //CHECK: math.exp {{.*}} : vector<8x16xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 64]
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<8x64xbf16>
+  //CHECK: xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x64xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix

``````````

</details>


https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Wed Jul 15 21:55:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 21:55:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a586437.c574b9b4.2aa0bf.b771@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Nishant Patel (nbpatel)

<details>
<summary>Changes</summary>

This PR adds rewriteWithRegrouping to unroll ConvertLayoutOp by extracting at the input inst_data granularity and inserting at the target inst_data granularity for cancellation during canonicalization.

Assisted by Claude

---
Full diff: https://github.com/llvm/llvm-project/pull/209822.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp (+110-14) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-blocking.mlir (+27-5) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..bcb9946373298 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -957,8 +957,92 @@ struct UnrollStoreMatrixOp : public UnrollPattern<xegpu::StoreMatrixOp> {
 /// after inst_data stripped. If it does, it will unroll the vector into
 /// multiple smaller vectors according to the target shape, and create multiple
 /// ConvertLayoutOp with the unrolled vectors and the stripped layouts.
+///
+/// When the input and target layouts have different inst_data, the source is
+/// extracted at the input inst_data granularity and the result is inserted at
+/// the target inst_data granularity, enabling slice cancellation during
+/// canonicalization.
 struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
   using UnrollPattern<xegpu::ConvertLayoutOp>::UnrollPattern;
+
+  /// Extracts source in `inTile` slices, regroups into `convTile`-sized
+  /// ConvertLayoutOps, and inserts the result in `outTile` slices.
+  /// Returns failure if the tiles do not evenly divide.
+  LogicalResult
+  rewriteWithRegrouping(xegpu::ConvertLayoutOp op, VectorType valueTy,
+                        ArrayRef<int64_t> convTile, ArrayRef<int64_t> inTile,
+                        ArrayRef<int64_t> outTile,
+                        xegpu::DistributeLayoutAttr inputLayout,
+                        xegpu::DistributeLayoutAttr targetLayout, Location loc,
+                        PatternRewriter &rewriter) const {
+    ArrayRef<int64_t> vecShape = valueTy.getShape();
+    if (!computeShapeRatio(vecShape, convTile) ||
+        !computeShapeRatio(convTile, inTile) ||
+        !computeShapeRatio(convTile, outTile))
+      return failure();
+
+    Type elemTy = valueTy.getElementType();
+    int64_t rank = valueTy.getRank();
+    VectorType convTy = VectorType::get(convTile, elemTy);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value source = op.getSource();
+    auto zeroOf = [&](VectorType ty) -> Value {
+      return arith::ConstantOp::create(
+          rewriter, loc, ty,
+          DenseElementsAttr::get(ty, rewriter.getZeroAttr(elemTy)));
+    };
+    auto addOffsets = [](ArrayRef<int64_t> a,
+                         ArrayRef<int64_t> b) -> SmallVector<int64_t> {
+      SmallVector<int64_t> res(a);
+      for (auto [r, v] : llvm::zip_equal(res, b))
+        r += v;
+      return res;
+    };
+
+    Value result = zeroOf(valueTy);
+    for (SmallVector<int64_t> convOff :
+         StaticTileOffsetRange(vecShape, convTile)) {
+      // Build the convert tile from inTile-sized slices of the source.
+      Value conv;
+      if (convTile == inTile) {
+        conv = vector::ExtractStridedSliceOp::create(
+            rewriter, loc, source, convOff, convTile, strides);
+      } else {
+        conv = zeroOf(convTy);
+        for (SmallVector<int64_t> inLocal :
+             StaticTileOffsetRange(convTile, inTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, source, addOffsets(convOff, inLocal), inTile,
+              strides);
+          conv = vector::InsertStridedSliceOp::create(rewriter, loc, piece,
+                                                      conv, inLocal, strides);
+        }
+      }
+
+      conv = xegpu::ConvertLayoutOp::create(rewriter, loc, convTy, conv,
+                                            inputLayout, targetLayout);
+
+      // Write the converted tile into the result as outTile-sized slices.
+      if (convTile == outTile) {
+        result = vector::InsertStridedSliceOp::create(rewriter, loc, conv,
+                                                      result, convOff, strides);
+      } else {
+        for (SmallVector<int64_t> outLocal :
+             StaticTileOffsetRange(convTile, outTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, conv, outLocal, outTile, strides);
+          result = vector::InsertStridedSliceOp::create(
+              rewriter, loc, piece, result, addOffsets(convOff, outLocal),
+              strides);
+        }
+      }
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+
   LogicalResult matchAndRewrite(xegpu::ConvertLayoutOp op,
                                 PatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
@@ -974,8 +1058,10 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
       return success();
     }
 
-    if (inputLayout.getEffectiveInstDataAsInt().empty() ||
-        targetLayout.getEffectiveInstDataAsInt().empty())
+    // Capture inst_data granularities before stripping them.
+    SmallVector<int64_t> inTile = inputLayout.getEffectiveInstDataAsInt();
+    SmallVector<int64_t> outTile = targetLayout.getEffectiveInstDataAsInt();
+    if (inTile.empty() || outTile.empty())
       return rewriter.notifyMatchFailure(op, "Not a target ConvertLayoutOp.");
 
     inputLayout = inputLayout.dropInstData();
@@ -988,20 +1074,30 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     if (!targetShape || targetShape->size() != (size_t)valueTy.getRank())
       return failure();
 
-    Value newSource = op.getSource();
+    // Nothing to convert if layouts match after stripping inst_data.
+    if (!inputLayout || !targetLayout || inputLayout.isEqualTo(targetLayout)) {
+      rewriter.replaceOp(op, op.getSource());
+      return success();
+    }
+
+    // Try regrouping: extract at inTile, convert, insert at outTile.
+    if (succeeded(rewriteWithRegrouping(op, valueTy, *targetShape, inTile,
+                                        outTile, inputLayout, targetLayout, loc,
+                                        rewriter)))
+      return success();
+
+    // Fallback: pack/unpack at the convert tile granularity.
+    SmallVector<Type> convertedValTypes =
+        getUnrolledTypes(valueTy, *targetShape);
+    SmallVector<Value> convertedValues =
+        pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
     SmallVector<Value> newOps;
-    if (inputLayout && targetLayout && !inputLayout.isEqualTo(targetLayout)) {
-      SmallVector<Type> convertedValTypes =
-          getUnrolledTypes(valueTy, *targetShape);
-      SmallVector<Value> convertedValues =
-          pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
-      for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
-        auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
-                                                    inputLayout, targetLayout);
-        newOps.push_back(newOp);
-      }
-      newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
+    for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
+      auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
+                                                  inputLayout, targetLayout);
+      newOps.push_back(newOp);
     }
+    Value newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
 
     rewriter.replaceOp(op, newSource);
     return success();
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..27a88efcf443c 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -463,19 +463,19 @@ gpu.module @test_kernel {
   //CHECK: gpu.func @convert_layout([[arg0:%.+]]: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
   //CHECK: [[cst:%.+]] = arith.constant dense<0.000000e+00> : vector<8x32x2xf16>
   //CHECK: [[e0:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [0, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
-  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
   //CHECK: [[c0:%.+]] = xegpu.convert_layout [[e0]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
-  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
   //CHECK: [[e2:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
+  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
+  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m0:%.+]] = math.exp [[e2]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i0:%.+]] = vector.insert_strided_slice [[m0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m1:%.+]] = math.exp [[e3]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i1:%.+]] = vector.insert_strided_slice [[m1]], [[i0]] {offsets = [0, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m2:%.+]] = math.exp [[e4]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i2:%.+]] = vector.insert_strided_slice [[m2]], [[i1]] {offsets = [4, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m3:%.+]] = math.exp [[e5]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i3:%.+]] = vector.insert_strided_slice [[m3]], [[i2]] {offsets = [4, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
   //CHECK: gpu.return [[i3]] : vector<8x32x2xf16>
@@ -488,6 +488,28 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 16] vs [8, 64]).
+#in = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#tgt = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 16]
+  //CHECK: math.exp {{.*}} : vector<8x16xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 64]
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<8x64xbf16>
+  //CHECK: xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x64xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix

``````````

</details>


https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Wed Jul 15 22:57:31 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Wed, 15 Jul 2026 22:57:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5872cb.9edb5c9d.5e50a.25c6@mx.google.com>


https://github.com/gysit commented:

Thanks for looking into this.

I believe the correct approach here would be to make DOLocalVariableAttr self recursive similar to the subprogram. That way we use the same machinery for all recursive nodes and we can handle arbitrary debug info graphs.

See https://github.com/llvm/llvm-project/pull/190808 for an example PR that makes a debug info node self recursive. 

https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Wed Jul 15 23:18:52 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 15 Jul 2026 23:18:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Guard ShlOp/OrOp folds against
 APInt bitwidth mismatch (PR #205304)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205304@github.com>
Message-ID: <6a5877cc.507fc713.2da8f.eba2@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/205304

>From b2c7da13e465e02d40fa7fe5e1dc0da56da781b4 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Tue, 23 Jun 2026 15:04:03 +0530
Subject: [PATCH] [mlir][LLVMIR] Guard ShlOp/OrOp folds against APInt bitwidth
 mismatch

---
 mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp    | 12 +++++++++++
 .../test/Dialect/LLVMIR/constant-folding.mlir | 20 +++++++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
index f3dc00e177b0f..2b1635ef7eea4 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMDialect.cpp
@@ -4004,6 +4004,12 @@ OpFoldResult LLVM::ShlOp::fold(FoldAdaptor adaptor) {
   if (!lhs)
     return {};
 
+  // Bail out if APInt bitwidths don't match the result type.
+  unsigned width = getType().getIntOrFloatBitWidth();
+  if (lhs.getValue().getBitWidth() != width ||
+      rhs.getValue().getBitWidth() != width)
+    return {};
+
   return IntegerAttr::get(getType(), lhs.getValue().shl(rhs.getValue()));
 }
 
@@ -4020,6 +4026,12 @@ OpFoldResult LLVM::OrOp::fold(FoldAdaptor adaptor) {
   if (!rhs)
     return {};
 
+  // Same bitwidth guard as ShlOp::fold.
+  unsigned width = getType().getIntOrFloatBitWidth();
+  if (lhs.getValue().getBitWidth() != width ||
+      rhs.getValue().getBitWidth() != width)
+    return {};
+
   return IntegerAttr::get(getType(), lhs.getValue() | rhs.getValue());
 }
 
diff --git a/mlir/test/Dialect/LLVMIR/constant-folding.mlir b/mlir/test/Dialect/LLVMIR/constant-folding.mlir
index 0616f19b8fddb..7ae017e1ba896 100644
--- a/mlir/test/Dialect/LLVMIR/constant-folding.mlir
+++ b/mlir/test/Dialect/LLVMIR/constant-folding.mlir
@@ -211,3 +211,23 @@ llvm.func @blockaddress_select(%arg: i1) -> !llvm.ptr {
   llvm.blocktag <id = 1>
   llvm.return %1 : !llvm.ptr
 }
+
+// -----
+
+// CHECK-LABEL: llvm.func @shl_bitwidth_mismatch
+llvm.func @shl_bitwidth_mismatch() -> i64 {
+  %0 = llvm.mlir.constant(1 : i8) : i64
+  // CHECK: llvm.shl
+  %1 = llvm.shl %0, %0 : i64
+  llvm.return %1 : i64
+}
+
+// -----
+
+// CHECK-LABEL: llvm.func @or_bitwidth_mismatch
+llvm.func @or_bitwidth_mismatch() -> i64 {
+  %0 = llvm.mlir.constant(1 : i8) : i64
+  // CHECK: llvm.or
+  %1 = llvm.or %0, %0 : i64
+  llvm.return %1 : i64
+}


From llvmlistbot at llvm.org  Wed Jul 15 23:20:37 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 15 Jul 2026 23:20:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Guard ShlOp/OrOp folds against
 APInt bitwidth mismatch (PR #205304)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205304@github.com>
Message-ID: <6a587835.07a3e4bc.43de0.804c@mx.google.com>


anutosh491 wrote:

Hey @krzysz00 @kuhar @joker-eph 

Could y'all help me with a review here. Thanks !

https://github.com/llvm/llvm-project/pull/205304

From llvmlistbot at llvm.org  Wed Jul 15 23:46:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 23:46:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve dropped conversion mappings
	(PR #209982)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209982@github.com>
Message-ID: <6a587e2c.c20dae4b.1fa811.a2e0@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-core

Author: lianjinfeng2003 (mygitljf)

<details>
<summary>Changes</summary>

I separated missing mappings from values that were explicitly dropped during dialect conversion. This keeps folded replacement chains from reusing values whose defining operations are removed.
I added coverage for folded live uses and partial 1:N replacements. The original reproducer now reports a normal conversion failure instead of crashing.
Fixes #<!-- -->207353 

---
Full diff: https://github.com/llvm/llvm-project/pull/209982.diff


3 Files Affected:

- (modified) mlir/lib/Transforms/Utils/DialectConversion.cpp (+46-31) 
- (modified) mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir (+25-1) 
- (modified) mlir/test/lib/Dialect/Test/TestPatterns.cpp (+18-1) 


``````````diff
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..da6bd32a96693 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -138,8 +138,10 @@ struct ConversionValueMapping {
   /// false positives.
   bool isMappedTo(Value value) const { return mappedTo.contains(value); }
 
-  /// Lookup a value in the mapping.
-  ValueVector lookup(const ValueVector &from) const;
+  /// Lookup a value in the mapping. A missing optional means that there is no
+  /// mapping, an empty vector is an explicit mapping to no values, and a
+  /// non-empty vector contains the mapped values.
+  std::optional<ValueVector> lookup(const ValueVector &from) const;
 
   template <typename T>
   struct IsValueVector : std::is_same<std::decay_t<T>, ValueVector> {};
@@ -220,11 +222,12 @@ static bool isPureTypeConversion(const ValueVector &values) {
   return op && op->hasAttr(kPureTypeConversionMarker);
 }
 
-ValueVector ConversionValueMapping::lookup(const ValueVector &from) const {
+std::optional<ValueVector>
+ConversionValueMapping::lookup(const ValueVector &from) const {
   auto it = mapping.find(from);
   if (it == mapping.end()) {
     // No mapping found: The lookup stops here.
-    return {};
+    return std::nullopt;
   }
   return it->second;
 }
@@ -945,12 +948,16 @@ struct ConversionPatternRewriterImpl : public RewriterBase::Listener {
   ///
   /// If `skipPureTypeConversions` is "true", materializations that are pure
   /// type conversions are not considered.
+  ///
+  /// If `followDroppedValues` is "false", explicit mappings to no values are
+  /// treated as leaves. Otherwise, they are followed as dropped values.
   ValueVector lookupOrDefault(Value from, TypeRange desiredTypes = {},
-                              bool skipPureTypeConversions = false) const;
+                              bool skipPureTypeConversions = false,
+                              bool followDroppedValues = false) const;
 
   /// Lookup the given value within the map, or return an empty vector if the
-  /// value is not mapped. If it is mapped, this follows the same behavior
-  /// as `lookupOrDefault`.
+  /// value is not mapped. Unlike the default `lookupOrDefault` behavior, this
+  /// follows explicit mappings to no values.
   ValueVector lookupOrNull(Value from, TypeRange desiredTypes = {}) const;
 
   //===--------------------------------------------------------------------===//
@@ -1362,9 +1369,10 @@ void ConversionPatternRewriterImpl::applyRewrites() {
 //===----------------------------------------------------------------------===//
 
 ValueVector ConversionPatternRewriterImpl::lookupOrDefault(
-    Value from, TypeRange desiredTypes, bool skipPureTypeConversions) const {
+    Value from, TypeRange desiredTypes, bool skipPureTypeConversions,
+    bool followDroppedValues) const {
   // Helper function that looks up a single value.
-  auto lookup = [&](const ValueVector &values) -> ValueVector {
+  auto lookup = [&](const ValueVector &values) -> std::optional<ValueVector> {
     assert(!values.empty() && "expected non-empty value vector");
 
     // If the pattern rollback is enabled, use the mapping to look up the
@@ -1376,32 +1384,37 @@ ValueVector ConversionPatternRewriterImpl::lookupOrDefault(
     // already been materialized in IR.
     Operation *op = getCommonDefiningOp(values);
     if (!op)
-      return {};
+      return std::nullopt;
     auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
     if (!castOp)
-      return {};
+      return std::nullopt;
     if (!this->unresolvedMaterializations.contains(castOp))
-      return {};
+      return std::nullopt;
     if (castOp.getOutputs() != values)
-      return {};
-    return castOp.getInputs();
+      return std::nullopt;
+    if (castOp.getInputs().empty())
+      return std::nullopt;
+    return ValueVector(castOp.getInputs());
   };
 
   // Helper function that looks up each value in `values` individually and then
-  // composes the results. If that fails, it tries to look up the entire vector
-  // at once.
-  auto composedLookup = [&](const ValueVector &values) -> ValueVector {
+  // composes the results. If no singleton mapping can be followed, it tries to
+  // look up the entire vector at once.
+  auto composedLookup =
+      [&](const ValueVector &values) -> std::optional<ValueVector> {
     // If possible, replace each value with (one or multiple) mapped values.
     ValueVector next;
+    bool foundMapping = false;
     for (Value v : values) {
-      ValueVector r = lookup({v});
-      if (!r.empty()) {
-        llvm::append_range(next, r);
-      } else {
+      std::optional<ValueVector> r = lookup({v});
+      if (!r || (r->empty() && !followDroppedValues)) {
         next.push_back(v);
+        continue;
       }
+      foundMapping = true;
+      llvm::append_range(next, *r);
     }
-    if (next != values) {
+    if (foundMapping) {
       // At least one value was replaced.
       return next;
     }
@@ -1414,11 +1427,9 @@ ValueVector ConversionPatternRewriterImpl::lookupOrDefault(
     // be stored (and looked up) in the mapping. But for performance reasons,
     // we choose to reuse existing IR (when possible) instead of creating it
     // multiple times.
-    ValueVector r = lookup(values);
-    if (r.empty()) {
-      // No mapping found: The lookup stops here.
-      return {};
-    }
+    std::optional<ValueVector> r = lookup(values);
+    if (r && r->empty() && !followDroppedValues)
+      return std::nullopt;
     return r;
   };
 
@@ -1443,10 +1454,12 @@ ValueVector ConversionPatternRewriterImpl::lookupOrDefault(
       desiredValue = current;
 
     // Lookup next value in the mapping.
-    ValueVector next = composedLookup(current);
-    if (next.empty())
+    std::optional<ValueVector> next = composedLookup(current);
+    if (!next)
       break;
-    current = std::move(next);
+    if (next->empty())
+      return {};
+    current = std::move(*next);
   } while (true);
 
   // If the desired values were found use them, otherwise default to the leaf
@@ -1461,7 +1474,9 @@ ValueVector ConversionPatternRewriterImpl::lookupOrDefault(
 ValueVector
 ConversionPatternRewriterImpl::lookupOrNull(Value from,
                                             TypeRange desiredTypes) const {
-  ValueVector result = lookupOrDefault(from, desiredTypes);
+  ValueVector result = lookupOrDefault(from, desiredTypes,
+                                       /*skipPureTypeConversions=*/false,
+                                       /*followDroppedValues=*/true);
   if (result == ValueVector{from} ||
       (!desiredTypes.empty() && TypeRange(ValueRange(result)) != desiredTypes))
     return {};
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..dea5e2eb6f7e1 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics | FileCheck %s
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,27 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that folding an identity cast does not hide a live use of an explicitly
+// erased result.
+func.func @remove_op_through_folded_identity_cast() -> i32 {
+  %0 = "test.illegal_op_a"() : () -> i32
+  // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+  %1 = builtin.unrealized_conversion_cast %0 : i32 to i32
+  // expected-note at below {{see existing live user here}}
+  return %1 : i32
+}
+
+// -----
+
+// CHECK-LABEL: func.func @compose_partial_1_to_n_erasure
+// CHECK-SAME:  (%[[ARG:.*]]: i32) -> i32 {
+// CHECK-NEXT:    return %[[ARG]] : i32
+// CHECK-NEXT:  }
+func.func @compose_partial_1_to_n_erasure(%arg0: i32) -> i32 {
+  %0 = "test.illegal_op_b"() : () -> i32
+  %1 = "test.cast"(%0, %arg0) : (i32, i32) -> i32
+  return %1 : i32
+}
diff --git a/mlir/test/lib/Dialect/Test/TestPatterns.cpp b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
index 552a1a473c9fd..451fe299adf13 100644
--- a/mlir/test/lib/Dialect/Test/TestPatterns.cpp
+++ b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
@@ -1881,6 +1881,22 @@ struct RemoveTestDialectOps : public RewritePattern {
   }
 };
 
+/// This pattern replaces the single result of test.cast with its two operands
+/// to test composing a partial 1:N replacement with an erasure.
+struct ReplaceTestDialectOpWithOperands : public ConversionPattern {
+  ReplaceTestDialectOpWithOperands(MLIRContext *context)
+      : ConversionPattern("test.cast", /*benefit=*/2, context) {}
+
+  LogicalResult
+  matchAndRewrite(Operation *op, ArrayRef<ValueRange> operands,
+                  ConversionPatternRewriter &rewriter) const override {
+    if (op->getNumOperands() != 2 || op->getNumResults() != 1)
+      return failure();
+    rewriter.replaceOpWithMultiple(op, {op->getOperands()});
+    return success();
+  }
+};
+
 struct TestUnknownRootOpDriver
     : public mlir::PassWrapper<TestUnknownRootOpDriver, OperationPass<>> {
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TestUnknownRootOpDriver)
@@ -1893,7 +1909,8 @@ struct TestUnknownRootOpDriver
   }
   void runOnOperation() override {
     mlir::RewritePatternSet patterns(&getContext());
-    patterns.add<RemoveTestDialectOps>(&getContext());
+    patterns.add<RemoveTestDialectOps, ReplaceTestDialectOpWithOperands>(
+        &getContext());
 
     mlir::ConversionTarget target(getContext());
     target.addIllegalDialect<TestDialect>();

``````````

</details>


https://github.com/llvm/llvm-project/pull/209982

From llvmlistbot at llvm.org  Wed Jul 15 23:52:18 2026
From: llvmlistbot at llvm.org (Jesse Rosenstock)
Date: Wed, 15 Jul 2026 23:52:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][quant] Skip non-finite scales in fake
 quant bounding (PR #209985)
Message-ID: <llvm.org/llvm/llvm-project/pull/209985@github.com>


https://github.com/jmr created https://github.com/llvm/llvm-project/pull/209985

Skip quantization if the dynamically generated scale is not finite. This occurs frequently in quantized JAX models where padding values bounded at -jnp.inf trigger overflow during fake quantization bounding.

Note: FakeQuantSupport.cpp acts as a legacy translation utility whose primary consumers live downstream, and there are currently no in-tree tests covering it. Therefore, no tests were added for this fix.

Assisted-by: Gemini

>From 93cb7b224d47dc732ca3757adfe52818a4fe535f Mon Sep 17 00:00:00 2001
From: Jesse Rosenstock <jmr at google.com>
Date: Thu, 16 Jul 2026 08:28:54 +0200
Subject: [PATCH] [mlir][quant] Skip non-finite scales in fake quant bounding

Skip quantization if the dynamically generated scale is not finite.
This occurs frequently in quantized JAX models where padding values
bounded at -jnp.inf trigger overflow during fake quantization bounding.

Note: FakeQuantSupport.cpp acts as a legacy translation utility whose
primary consumers live downstream, and there are currently no in-tree
tests covering it. Therefore, no tests were added for this fix.

Assisted-by: Gemini
---
 mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
index fb27640bfd278..97637ab71ff38 100644
--- a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
+++ b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
@@ -131,6 +131,12 @@ mlir::quant::fakeQuantAttrsToType(Location loc, unsigned numBits, double rmin,
   int64_t nudgedZeroPoint;
   getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
 
+  // Skip quantization if the dynamically generated scale is not finite.
+  // This occurs frequently in quantized JAX models where padding values
+  // bounded at -jnp.inf trigger overflow during fake quantization bounding.
+  if (!std::isfinite(scale))
+    return nullptr;
+
   return UniformQuantizedType::getChecked(loc, flags, storageType,
                                           expressedType, scale, nudgedZeroPoint,
                                           qmin, qmax);
@@ -173,6 +179,8 @@ UniformQuantizedPerAxisType mlir::quant::fakeQuantAttrsToType(
     double scale;
     int64_t nudgedZeroPoint;
     getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
+    if (!std::isfinite(scale))
+      return nullptr;
     scales.push_back(scale);
     zeroPoints.push_back(nudgedZeroPoint);
   }


From llvmlistbot at llvm.org  Wed Jul 15 23:52:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 23:52:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][quant] Skip non-finite scales in fake
 quant bounding (PR #209985)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209985@github.com>
Message-ID: <6a587fb4.824e11a6.1389c5.f135@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @jmr :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/209985

From llvmlistbot at llvm.org  Wed Jul 15 23:53:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 23:53:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][quant] Skip non-finite scales in fake
 quant bounding (PR #209985)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209985@github.com>
Message-ID: <6a587fdf.e55dc341.1dd5fc.e2d7@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-quant

Author: Jesse Rosenstock (jmr)

<details>
<summary>Changes</summary>

Skip quantization if the dynamically generated scale is not finite. This occurs frequently in quantized JAX models where padding values bounded at -jnp.inf trigger overflow during fake quantization bounding.

Note: FakeQuantSupport.cpp acts as a legacy translation utility whose primary consumers live downstream, and there are currently no in-tree tests covering it. Therefore, no tests were added for this fix.

Assisted-by: Gemini

---
Full diff: https://github.com/llvm/llvm-project/pull/209985.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp (+8) 


``````````diff
diff --git a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
index fb27640bfd278..97637ab71ff38 100644
--- a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
+++ b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
@@ -131,6 +131,12 @@ mlir::quant::fakeQuantAttrsToType(Location loc, unsigned numBits, double rmin,
   int64_t nudgedZeroPoint;
   getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
 
+  // Skip quantization if the dynamically generated scale is not finite.
+  // This occurs frequently in quantized JAX models where padding values
+  // bounded at -jnp.inf trigger overflow during fake quantization bounding.
+  if (!std::isfinite(scale))
+    return nullptr;
+
   return UniformQuantizedType::getChecked(loc, flags, storageType,
                                           expressedType, scale, nudgedZeroPoint,
                                           qmin, qmax);
@@ -173,6 +179,8 @@ UniformQuantizedPerAxisType mlir::quant::fakeQuantAttrsToType(
     double scale;
     int64_t nudgedZeroPoint;
     getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
+    if (!std::isfinite(scale))
+      return nullptr;
     scales.push_back(scale);
     zeroPoints.push_back(nudgedZeroPoint);
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/209985

From llvmlistbot at llvm.org  Wed Jul 15 23:53:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 15 Jul 2026 23:53:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][quant] Skip non-finite scales in fake
 quant bounding (PR #209985)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209985@github.com>
Message-ID: <6a587fe1.af6ba3ed.3db413.a97e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jesse Rosenstock (jmr)

<details>
<summary>Changes</summary>

Skip quantization if the dynamically generated scale is not finite. This occurs frequently in quantized JAX models where padding values bounded at -jnp.inf trigger overflow during fake quantization bounding.

Note: FakeQuantSupport.cpp acts as a legacy translation utility whose primary consumers live downstream, and there are currently no in-tree tests covering it. Therefore, no tests were added for this fix.

Assisted-by: Gemini

---
Full diff: https://github.com/llvm/llvm-project/pull/209985.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp (+8) 


``````````diff
diff --git a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
index fb27640bfd278..97637ab71ff38 100644
--- a/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
+++ b/mlir/lib/Dialect/Quant/Utils/FakeQuantSupport.cpp
@@ -131,6 +131,12 @@ mlir::quant::fakeQuantAttrsToType(Location loc, unsigned numBits, double rmin,
   int64_t nudgedZeroPoint;
   getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
 
+  // Skip quantization if the dynamically generated scale is not finite.
+  // This occurs frequently in quantized JAX models where padding values
+  // bounded at -jnp.inf trigger overflow during fake quantization bounding.
+  if (!std::isfinite(scale))
+    return nullptr;
+
   return UniformQuantizedType::getChecked(loc, flags, storageType,
                                           expressedType, scale, nudgedZeroPoint,
                                           qmin, qmax);
@@ -173,6 +179,8 @@ UniformQuantizedPerAxisType mlir::quant::fakeQuantAttrsToType(
     double scale;
     int64_t nudgedZeroPoint;
     getNudgedScaleAndZeroPoint(qmin, qmax, rmin, rmax, scale, nudgedZeroPoint);
+    if (!std::isfinite(scale))
+      return nullptr;
     scales.push_back(scale);
     zeroPoints.push_back(nudgedZeroPoint);
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/209985

From llvmlistbot at llvm.org  Wed Jul 15 23:56:35 2026
From: llvmlistbot at llvm.org (Jesse Rosenstock)
Date: Wed, 15 Jul 2026 23:56:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][quant] Skip non-finite scales in fake
 quant bounding (PR #209985)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209985@github.com>
Message-ID: <6a5880a3.743400b0.d1365.f626@mx.google.com>


jmr wrote:

> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

Confirmed.

https://github.com/llvm/llvm-project/pull/209985

From llvmlistbot at llvm.org  Thu Jul 16 00:49:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:49:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][NVVM] Diagnose non-NVVM target ops
 before serialization (PR #210000)
Message-ID: <llvm.org/llvm/llvm-project/pull/210000@github.com>


https://github.com/shubhamnarlawar created https://github.com/llvm/llvm-project/pull/210000

When a gpu.module targeted with #nvvm.target contains operations from other GPU target dialects (e.g. rocdl, xevm), those ops are translated to their LLVM intrinsics (e.g. llvm.amdgcn.*)

Add a walk at the top of NVVMTargetAttrImpl::serializeToObject that rejects operations from foreign GPU target dialects with a clear diagnostic before any LLVM IR is produced

This PR fixes #205274

>From 7c0242975051440dd4db56ea4c469ce52d5e4d15 Mon Sep 17 00:00:00 2001
From: Shubham Narlawar <shubham.narlawar at rrlogic.co.in>
Date: Thu, 16 Jul 2026 13:17:11 +0530
Subject: [PATCH] [mlir][NVVM] Diagnose non-NVVM target ops before
 serialization

When a gpu.module targeted with #nvvm.target contains operations from
other GPU target dialects (e.g. rocdl, xevm), those ops are translated
to their LLVM intrinsics (e.g. llvm.amdgcn.*)

Add a walk at the top of NVVMTargetAttrImpl::serializeToObject that
rejects operations from foreign GPU target dialects with a clear
diagnostic before any LLVM IR is produced
---
 mlir/lib/Target/LLVM/NVVM/Target.cpp          | 14 ++++++++++
 .../Dialect/GPU/module-to-binary-invalid.mlir | 28 ++++++++++++++++++-
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVM/NVVM/Target.cpp b/mlir/lib/Target/LLVM/NVVM/Target.cpp
index 2d197a162a7af..41351dc766d1c 100644
--- a/mlir/lib/Target/LLVM/NVVM/Target.cpp
+++ b/mlir/lib/Target/LLVM/NVVM/Target.cpp
@@ -760,6 +760,20 @@ NVVMTargetAttrImpl::serializeToObject(Attribute attribute, Operation *module,
     module->emitError("Module must be a GPU module.");
     return std::nullopt;
   }
+
+  // Reject operations from other GPU target dialects.
+  WalkResult foreignOp = module->walk([&](Operation *op) {
+    StringRef dialect = op->getDialect()->getNamespace();
+    if (dialect == "rocdl" || dialect == "xevm") {
+      op->emitError() << "'" << op->getName() << "' from '" << dialect
+                      << "' dialect is not compatible with the NVVM target";
+      return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  if (foreignOp.wasInterrupted())
+    return std::nullopt;
+
   NVPTXSerializer serializer(*module, cast<NVVMTargetAttr>(attribute), options);
   serializer.init();
   std::optional<SmallVector<char, 0>> result = serializer.run();
diff --git a/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir b/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
index 7043c4fe49e1a..e7eb75f11388b 100644
--- a/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
+++ b/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s --gpu-module-to-binary --verify-diagnostics
+// RUN: mlir-opt %s --gpu-module-to-binary --verify-diagnostics --split-input-file
 
 module attributes {gpu.container_module} {
   // expected-error @below {{the module has no target attributes}}
@@ -10,3 +10,29 @@ module attributes {gpu.container_module} {
     }
   }
 }
+
+// -----
+
+module attributes {gpu.container_module} {
+  // expected-error @below {{An error happened while serializing the module}}
+  gpu.module @kernel_module_nvvm_rocdl_op [#nvvm.target] {
+    llvm.func @kernel() attributes {gpu.kernel} {
+      // expected-error @below {{'rocdl.workitem.id.x' from 'rocdl' dialect is not compatible with the NVVM target}}
+      %tx = rocdl.workitem.id.x : i32
+      llvm.return
+    }
+  }
+}
+
+// -----
+
+module attributes {gpu.container_module} {
+  // expected-error @below {{An error happened while serializing the module}}
+  gpu.module @kernel_module_nvvm_rocdl_barrier [#nvvm.target] {
+    llvm.func @kernel() attributes {gpu.kernel} {
+      // expected-error @below {{'rocdl.barrier' from 'rocdl' dialect is not compatible with the NVVM target}}
+      rocdl.barrier
+      llvm.return
+    }
+  }
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Thu Jul 16 00:49:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:49:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][NVVM] Diagnose non-NVVM target ops
 before serialization (PR #210000)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210000@github.com>
Message-ID: <6a588d1d.eb165e5c.3844ee.19e2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: shubhamnarlawar

<details>
<summary>Changes</summary>

When a gpu.module targeted with #nvvm.target contains operations from other GPU target dialects (e.g. rocdl, xevm), those ops are translated to their LLVM intrinsics (e.g. llvm.amdgcn.*)

Add a walk at the top of NVVMTargetAttrImpl::serializeToObject that rejects operations from foreign GPU target dialects with a clear diagnostic before any LLVM IR is produced

This PR fixes #<!-- -->205274

---
Full diff: https://github.com/llvm/llvm-project/pull/210000.diff


2 Files Affected:

- (modified) mlir/lib/Target/LLVM/NVVM/Target.cpp (+14) 
- (modified) mlir/test/Dialect/GPU/module-to-binary-invalid.mlir (+27-1) 


``````````diff
diff --git a/mlir/lib/Target/LLVM/NVVM/Target.cpp b/mlir/lib/Target/LLVM/NVVM/Target.cpp
index 2d197a162a7af..41351dc766d1c 100644
--- a/mlir/lib/Target/LLVM/NVVM/Target.cpp
+++ b/mlir/lib/Target/LLVM/NVVM/Target.cpp
@@ -760,6 +760,20 @@ NVVMTargetAttrImpl::serializeToObject(Attribute attribute, Operation *module,
     module->emitError("Module must be a GPU module.");
     return std::nullopt;
   }
+
+  // Reject operations from other GPU target dialects.
+  WalkResult foreignOp = module->walk([&](Operation *op) {
+    StringRef dialect = op->getDialect()->getNamespace();
+    if (dialect == "rocdl" || dialect == "xevm") {
+      op->emitError() << "'" << op->getName() << "' from '" << dialect
+                      << "' dialect is not compatible with the NVVM target";
+      return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  if (foreignOp.wasInterrupted())
+    return std::nullopt;
+
   NVPTXSerializer serializer(*module, cast<NVVMTargetAttr>(attribute), options);
   serializer.init();
   std::optional<SmallVector<char, 0>> result = serializer.run();
diff --git a/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir b/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
index 7043c4fe49e1a..e7eb75f11388b 100644
--- a/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
+++ b/mlir/test/Dialect/GPU/module-to-binary-invalid.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s --gpu-module-to-binary --verify-diagnostics
+// RUN: mlir-opt %s --gpu-module-to-binary --verify-diagnostics --split-input-file
 
 module attributes {gpu.container_module} {
   // expected-error @below {{the module has no target attributes}}
@@ -10,3 +10,29 @@ module attributes {gpu.container_module} {
     }
   }
 }
+
+// -----
+
+module attributes {gpu.container_module} {
+  // expected-error @below {{An error happened while serializing the module}}
+  gpu.module @kernel_module_nvvm_rocdl_op [#nvvm.target] {
+    llvm.func @kernel() attributes {gpu.kernel} {
+      // expected-error @below {{'rocdl.workitem.id.x' from 'rocdl' dialect is not compatible with the NVVM target}}
+      %tx = rocdl.workitem.id.x : i32
+      llvm.return
+    }
+  }
+}
+
+// -----
+
+module attributes {gpu.container_module} {
+  // expected-error @below {{An error happened while serializing the module}}
+  gpu.module @kernel_module_nvvm_rocdl_barrier [#nvvm.target] {
+    llvm.func @kernel() attributes {gpu.kernel} {
+      // expected-error @below {{'rocdl.barrier' from 'rocdl' dialect is not compatible with the NVVM target}}
+      rocdl.barrier
+      llvm.return
+    }
+  }
+}
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/210000

From llvmlistbot at llvm.org  Thu Jul 16 00:55:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:55:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588e5a.92d2873c.3511c3.6188@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/209911

>From c02fc926263c282624cbdae458400b14c86f3b2a Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Wed, 15 Jul 2026 14:58:04 -0700
Subject: [PATCH] [mlir][LLVMIR] Avoid duplicate retained local variables

Use recursive DILocalVariableAttr for retained locals that also appear in debug intrinsics. This avoids importing the same LLVM metadata node twice.
---
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 26 ++++++++--
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |  4 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |  3 +-
 mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp      | 19 ++++++++
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      | 42 +++++++++++++----
 mlir/lib/Target/LLVMIR/DebugImporter.h        | 13 +++++
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   | 47 ++++++++++++++++++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     | 10 ++++
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 33 +++++++++++++
 mlir/test/Target/LLVMIR/Import/debug-info.ll  | 33 +++++++++++++
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     | 40 ++++++++++++++++
 11 files changed, 255 insertions(+), 15 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 6e2d4628c6173..fb17f1efb7c94 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -696,9 +696,14 @@ def LLVM_DILexicalBlockFile : LLVM_Attr<"DILexicalBlockFile", "di_lexical_block_
 //===----------------------------------------------------------------------===//
 
 def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
-                                         /*traits=*/[], "DINodeAttr"> {
+                                         [LLVM_DIRecursiveTypeAttrInterface],
+                                         "DINodeAttr"> {
   let parameters = (ins
-    "DIScopeAttr":$scope,
+    // DIRecursiveTypeAttrInterface specific parameters.
+    OptionalParameter<"DistinctAttr">:$recId,
+    OptionalParameter<"bool">:$isRecSelf,
+    // DILocalVariableAttr specific parameters.
+    OptionalParameter<"DIScopeAttr">:$scope,
     OptionalParameter<"StringAttr">:$name,
     OptionalParameter<"DIFileAttr">:$file,
     OptionalParameter<"unsigned">:$line,
@@ -714,11 +719,24 @@ def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
       "DITypeAttr":$type, "DIFlags":$flags
     ), [{
       MLIRContext *ctx = scope.getContext();
-      return $_get(ctx, scope, StringAttr::get(ctx, name), file, line,
-                   arg, alignInBits, type, flags);
+      return $_get(ctx, /*recId=*/nullptr, /*isRecSelf=*/false, scope,
+                   StringAttr::get(ctx, name), file, line, arg, alignInBits,
+                   type, flags);
     }]>
   ];
   let assemblyFormat = "`<` struct(params) `>`";
+  let extraClassDeclaration = [{
+    /// Requirements of DIRecursiveTypeAttrInterface.
+    /// @{
+
+    /// Get a copy of this attr but with the recursive ID set to `recId`.
+    DIRecursiveTypeAttrInterface withRecId(DistinctAttr recId);
+
+    /// Build a rec-self instance using the provided `recId`.
+    static DIRecursiveTypeAttrInterface getRecSelf(DistinctAttr recId);
+
+    /// @}
+  }];
 
   // Generate mnemonic alias for the attribute.
   let genMnemonicAlias = 1;
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index b25a8c8a9829e..ef7df3c204841 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -99,7 +99,9 @@ def DIFileAttr : DialectAttribute<(attr
 //===----------------------------------------------------------------------===//
 
 def DILocalVariableAttr : DialectAttribute<(attr
-  Attr<"DIScopeAttr">:$scope,
+  OptionalAttribute<"DistinctAttr">:$recId,
+  Bool:$isRecSelf,
+  OptionalAttribute<"DIScopeAttr">:$scope,
   OptionalAttribute<"StringAttr">:$name,
   OptionalAttribute<"DIFileAttr">:$file,
   VarInt:$line,
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index 8082de122e74e..eb7e969254962 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -414,7 +414,8 @@ MlirAttribute mlirLLVMDILocalVariableAttrGet(
     MlirAttribute diFile, unsigned int line, unsigned int arg,
     unsigned int alignInBits, MlirAttribute diType, int64_t flags) {
   return wrap(DILocalVariableAttr::get(
-      unwrap(ctx), cast<DIScopeAttr>(unwrap(scope)),
+      unwrap(ctx), /*recId=*/DistinctAttr{}, /*isRecSelf=*/false,
+      cast<DIScopeAttr>(unwrap(scope)),
       cast<StringAttr>(unwrap(name)), cast<DIFileAttr>(unwrap(diFile)), line,
       arg, alignInBits, cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
 }
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
index 266faeca4553b..7917ac8b1ff10 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
@@ -325,6 +325,25 @@ void printExpressionArg(AsmPrinter &printer, uint64_t opcode,
   });
 }
 
+//===----------------------------------------------------------------------===//
+// DILocalVariableAttr
+//===----------------------------------------------------------------------===//
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::withRecId(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      getContext(), recId, getIsRecSelf(), getScope(), getName(), getFile(),
+      getLine(), getArg(), getAlignInBits(), getType(), getFlags());
+}
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::getRecSelf(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      recId.getContext(), recId, /*isRecSelf=*/true, /*scope=*/{},
+      /*name=*/{}, /*file=*/{}, /*line=*/0, /*arg=*/0, /*alignInBits=*/0,
+      /*type=*/{}, DIFlags::Zero);
+}
+
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 85df880d2712a..f8ccbc5bb6637 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -17,6 +17,7 @@
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Metadata.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -193,13 +194,21 @@ DebugImporter::translateImpl(llvm::DIGlobalVariable *node) {
 
 DILocalVariableAttr DebugImporter::translateImpl(llvm::DILocalVariable *node) {
   // Return nullptr if the scope or type is a cyclic dependency.
-  DIScopeAttr scope = translate(node->getScope());
+  DIScopeAttr scope;
+  {
+    bool preferLocalVariableCycleBreakForNode =
+        static_cast<bool>(nodeToRecId.lookup(node));
+    llvm::SaveAndRestore preferLocalVariableCycleBreakGuard(
+        preferLocalVariableCycleBreak, preferLocalVariableCycleBreakForNode);
+    scope = translate(node->getScope());
+  }
   if (node->getScope() && !scope)
     return nullptr;
   return DILocalVariableAttr::get(
-      context, scope, getStringAttrOrNull(node->getRawName()),
-      translate(node->getFile()), node->getLine(), node->getArg(),
-      node->getAlignInBits(), translate(node->getType()),
+      context, /*recId=*/DistinctAttr{}, /*isRecSelf=*/false, scope,
+      getStringAttrOrNull(node->getRawName()), translate(node->getFile()),
+      node->getLine(), node->getArg(), node->getAlignInBits(),
+      translate(node->getType()),
       symbolizeDIFlags(node->getFlags()).value_or(DIFlags::Zero));
 }
 
@@ -261,6 +270,11 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   // Convert the retained nodes but drop all of them if one of them is invalid.
   SmallVector<Attribute> retainedNodes;
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
+    if (auto *local = dyn_cast<llvm::DILocalVariable>(retainedNode);
+        local && nodeToRecId.contains(local)) {
+      retainedNodes.push_back(*createRecSelf(local));
+      return;
+    }
     retainedNodes.push_back(translate(retainedNode));
   };
   auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
@@ -446,6 +460,9 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
       DistinctAttr recId = nodeToRecId.lookup(node);
       auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
       attr = cast<DINodeAttr>(recType.withRecId(recId));
+    } else if (DistinctAttr recId = nodeToRecId.lookup(node)) {
+      auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
+      attr = cast<DINodeAttr>(recType.withRecId(recId));
     }
     cacheEntry.resolve(attr);
     return attr;
@@ -454,22 +471,31 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
   return nullptr;
 }
 
+void DebugImporter::markLocalVariableAsRecursive(llvm::DILocalVariable *node) {
+  if (!nodeToRecId.contains(node))
+    nodeToRecId[node] = DistinctAttr::create(UnitAttr::get(context));
+}
+
 /// Get the `getRecSelf` constructor for the translated node if it participates
-/// in CyclicReplacerCache cycle breaking (recursive composite types,
-/// subprograms, or compile units).
-static function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
-getRecSelfConstructor(llvm::DINode *node) {
+/// in CyclicReplacerCache cycle breaking.
+function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+DebugImporter::getRecSelfConstructor(llvm::DINode *node) {
   using CtorType = function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>;
   return TypeSwitch<llvm::DINode *, CtorType>(node)
       .Case([&](llvm::DICompositeType *) {
         return CtorType(DICompositeTypeAttr::getRecSelf);
       })
       .Case([&](llvm::DISubprogram *) {
+        if (preferLocalVariableCycleBreak)
+          return CtorType();
         return CtorType(DISubprogramAttr::getRecSelf);
       })
       .Case([&](llvm::DICompileUnit *) {
         return CtorType(DICompileUnitAttr::getRecSelf);
       })
+      .Case([&](llvm::DILocalVariable *) {
+        return CtorType(DILocalVariableAttr::getRecSelf);
+      })
       .Default(CtorType());
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.h b/mlir/lib/Target/LLVMIR/DebugImporter.h
index b224ad3376be7..40bf0b113e28f 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.h
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.h
@@ -47,6 +47,9 @@ class DebugImporter {
   /// Returns UnknownLoc if `func` has no debug information attached to it.
   Location translateFuncLocation(llvm::Function *func);
 
+  /// Marks the local variable as requiring a recursive representation.
+  void markLocalVariableAsRecursive(llvm::DILocalVariable *node);
+
   /// Translates the given LLVM debug metadata to MLIR.
   DINodeAttr translate(llvm::DINode *node);
 
@@ -92,6 +95,11 @@ class DebugImporter {
   /// for it, or create a new one if not.
   DistinctAttr getOrCreateDistinctID(llvm::DINode *node);
 
+  /// Get the `getRecSelf` constructor for the translated node if it
+  /// participates in CyclicReplacerCache cycle breaking.
+  function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+  getRecSelfConstructor(llvm::DINode *node);
+
   std::optional<DINodeAttr> createRecSelf(llvm::DINode *node);
 
   /// A mapping between distinct LLVM debug metadata nodes and the corresponding
@@ -103,6 +111,11 @@ class DebugImporter {
   /// same ID and be deduplicated.
   DenseMap<llvm::DINode *, DistinctAttr> nodeToRecId;
 
+  /// True while translating the scope of a DILocalVariable. This lets a
+  /// local-variable/subprogram cycle be broken at the local variable instead of
+  /// at the subprogram.
+  bool preferLocalVariableCycleBreak = false;
+
   CyclicReplacerCache<llvm::DINode *, DINodeAttr> cache;
 
   MLIRContext *context;
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 442f5b6b137e6..fc387c0eb7714 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -16,6 +16,7 @@
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -329,6 +330,20 @@ DebugTranslation::translateImpl(DILocalVariableAttr attr) {
       /*Annotations=*/nullptr);
 }
 
+llvm::TempDILocalVariable
+DebugTranslation::translateTemporaryImpl(DILocalVariableAttr attr) {
+  llvm::DILocalScope *scope =
+      attr.getScope() ? cast<llvm::DILocalScope>(translate(attr.getScope()))
+                      : localVariablePlaceholderScope;
+  assert(scope && "expected local variable placeholder scope");
+  return llvm::DILocalVariable::getTemporary(
+      llvmCtx, scope, /*Name=*/"", /*File=*/nullptr, attr.getLine(),
+      /*Type=*/static_cast<llvm::DIType *>(nullptr), attr.getArg(),
+      static_cast<llvm::DINode::DIFlags>(attr.getFlags()),
+      attr.getAlignInBits(),
+      /*Annotations=*/nullptr);
+}
+
 llvm::DIGlobalVariable *
 DebugTranslation::translateImpl(DIGlobalVariableAttr attr) {
   return llvm::DIGlobalVariable::getDistinct(
@@ -345,6 +360,24 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
       iter != recursiveNodeMap.end()) {
     return iter->second;
   }
+
+  if (auto localVariable = dyn_cast<DILocalVariableAttr>(attr)) {
+    if (localVariable.getIsRecSelf()) {
+      auto iter = forwardLocalVariablePlaceholders
+                      .try_emplace(recursiveId,
+                                   translateTemporaryImpl(localVariable))
+                      .first;
+      return iter->second.get();
+    }
+    if (auto iter = forwardLocalVariablePlaceholders.find(recursiveId);
+        iter != forwardLocalVariablePlaceholders.end()) {
+      llvm::DILocalVariable *concrete = translateImpl(localVariable);
+      iter->second->replaceAllUsesWith(concrete);
+      forwardLocalVariablePlaceholders.erase(iter);
+      return concrete;
+    }
+  }
+
   assert(!attr.getIsRecSelf() && "unbound DI recursive self reference");
 
   auto setRecursivePlaceholder = [&](llvm::DINode *placeholder) {
@@ -377,6 +410,13 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
             auto *concrete = translateImpl(attr);
             temporary->replaceAllUsesWith(concrete);
             return concrete;
+          })
+          .Case([&](DILocalVariableAttr attr) {
+            auto temporary = translateTemporaryImpl(attr);
+            setRecursivePlaceholder(temporary.get());
+            auto *concrete = translateImpl(attr);
+            temporary->replaceAllUsesWith(concrete);
+            return concrete;
           });
 
   assert(recursiveNodeMap.back().first == recursiveId &&
@@ -417,10 +457,15 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
+      /*RetainedNodes=*/nullptr, /*ThrownTypes=*/nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
+  if (!attr.getRetainedNodes().empty()) {
+    llvm::SaveAndRestore guard(localVariablePlaceholderScope,
+                               cast<llvm::DILocalScope>(node));
+    node->replaceRetainedNodes(getRetainedNodesOrNull(attr.getRetainedNodes()));
+  }
   return node;
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index c1fdbd89087da..2db7f401f576d 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -105,6 +105,7 @@ class DebugTranslation {
   llvm::TempDICompositeType translateTemporaryImpl(DICompositeTypeAttr attr);
   llvm::TempDISubprogram translateTemporaryImpl(DISubprogramAttr attr);
   llvm::TempDICompileUnit translateTemporaryImpl(DICompileUnitAttr attr);
+  llvm::TempDILocalVariable translateTemporaryImpl(DILocalVariableAttr attr);
 
   /// Constructs a string metadata node from the string attribute. Returns
   /// nullptr if `stringAttr` is null or contains and empty string.
@@ -135,6 +136,15 @@ class DebugTranslation {
   /// A mapping between recursive ID and the translated DINode.
   llvm::MapVector<DistinctAttr, llvm::DINode *> recursiveNodeMap;
 
+  /// A mapping between recursive IDs and local-variable placeholders created
+  /// from forward self references in retainedNodes.
+  DenseMap<DistinctAttr, llvm::TempDILocalVariable>
+      forwardLocalVariablePlaceholders;
+
+  /// Scope to use for forward local-variable placeholders while translating a
+  /// DISubprogram's retainedNodes.
+  llvm::DILocalScope *localVariablePlaceholderScope = nullptr;
+
   /// A mapping between a distinct ID and the translated LLVM metadata node.
   /// This helps identify attrs that should translate into the same LLVM debug
   /// node.
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index bb55678f91005..fc4ad29d2d747 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -35,6 +35,7 @@
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugProgramInstruction.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/InstIterator.h"
@@ -246,6 +247,38 @@ ModuleImport::ModuleImport(ModuleOp mlirModule,
       emitExpensiveWarnings(emitExpensiveWarnings),
       preferUnregisteredIntrinsics(preferUnregisteredIntrinsics) {
   builder.setInsertionPointToStart(mlirModule.getBody());
+
+  // Variables that are both retained and used by a debug intrinsic need to
+  // break cycles at the variable itself to avoid creating duplicate attrs with
+  // different subprogram scopes.
+  llvm::SmallPtrSet<llvm::DILocalVariable *, 8> retainedLocalVariables;
+  for (llvm::Function &func : *this->llvmModule) {
+    llvm::DISubprogram *subprogram = func.getSubprogram();
+    if (!subprogram)
+      continue;
+    auto addLocalVariable = [&](llvm::DILocalVariable *localVariable) {
+      retainedLocalVariables.insert(localVariable);
+    };
+    auto skip = [](auto *) {};
+    subprogram->forEachRetainedNode(addLocalVariable, skip, skip, skip, skip);
+  }
+
+  auto markIfRetained = [&](llvm::DILocalVariable *localVariable) {
+    if (localVariable && retainedLocalVariables.contains(localVariable))
+      debugImporter->markLocalVariableAsRecursive(localVariable);
+  };
+  for (llvm::Function &func : *this->llvmModule) {
+    for (llvm::Instruction &inst : llvm::instructions(func)) {
+      if (auto *dbgIntrinsic = dyn_cast<llvm::DbgVariableIntrinsic>(&inst))
+        markIfRetained(dbgIntrinsic->getVariable());
+      if (!inst.DebugMarker)
+        continue;
+      for (llvm::DbgRecord &dbgRecord : inst.DebugMarker->getDbgRecordRange())
+        if (auto *dbgVariableRecord =
+                dyn_cast<llvm::DbgVariableRecord>(&dbgRecord))
+          markIfRetained(dbgVariableRecord->getVariable());
+    }
+  }
 }
 
 ComdatOp ModuleImport::getGlobalComdatOp() {
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 28c25809002cd..38b250f00b958 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1046,3 +1046,36 @@ define void @fn_with_static_local() !dbg !3 {
 !12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
 !13 = !DISubroutineType(types: !14)
 !14 = !{null}
+
+; // -----
+
+; Test that a DILocalVariable listed in DISubprogram's retainedNodes and used by
+; a debug intrinsic does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR_SELF:.+]] = #llvm.di_local_variable<recId = [[REC_ID:distinct.+<>]], isRecSelf = true>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_retained_local"{{.*}}retainedNodes = [#[[LVAR_SELF]]]>
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<recId = [[REC_ID]], scope = #[[SP]], name = "n"{{.*}}>
+; CHECK-DAG-COUNT-2: {{#llvm\.di_local_variable<}}
+; CHECK: llvm.intr.dbg.value #[[LVAR]] = %{{.*}} : i32
+
+define void @fn_with_retained_local(i32 %n) !dbg !3 {
+entry:
+  call void @llvm.dbg.value(metadata i32 %n, metadata !12, metadata !DIExpression()), !dbg !15
+  ret void
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!3 = distinct !DISubprogram(name: "fn_with_retained_local", scope: !4, file: !4, line: 1, type: !13, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!12}
+!12 = !DILocalVariable(name: "n", arg: 1, scope: !3, file: !4, line: 1, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null, !9}
+!15 = !DILocation(line: 1, column: 1, scope: !3)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 72dadb2602c52..77eade68ae9e3 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -421,6 +421,46 @@ llvm.func @imp_fn() {
 
 // -----
 
+// Test that a recursive local variable retained by a subprogram is exported
+// without duplicating the DILocalVariable metadata node.
+
+#file = #llvm.di_file<"test.c" in "">
+#cu = #llvm.di_compile_unit<
+  id = distinct[0]<>, sourceLanguage = DW_LANG_C, file = #file,
+  isOptimized = false, emissionKind = Full
+>
+#void = #llvm.di_null_type
+#void_type = #llvm.di_subroutine_type<types = #void>
+#int = #llvm.di_basic_type<
+  tag = DW_TAG_base_type, name = "int",
+  sizeInBits = 32, encoding = DW_ATE_signed
+>
+#local_self = #llvm.di_local_variable<recId = distinct[1]<>, isRecSelf = true>
+#sp = #llvm.di_subprogram<
+  id = distinct[2]<>, compileUnit = #cu, scope = #file,
+  name = "fn_with_retained_local", file = #file,
+  subprogramFlags = Definition, type = #void_type,
+  retainedNodes = [#local_self]
+>
+#local = #llvm.di_local_variable<
+  recId = distinct[1]<>, scope = #sp, name = "n", file = #file,
+  line = 1, arg = 1, type = #int
+>
+
+// CHECK-LABEL: define void @fn_with_retained_local(
+// CHECK-SAME: i32 %[[ARG:.*]]) !dbg ![[SP:[0-9]+]]
+llvm.func @fn_with_retained_local(%arg0: i32) {
+  // RECORDS: #dbg_value(i32 %[[ARG]], ![[LOCAL:[0-9]+]], !DIExpression(), !{{.*}})
+  llvm.intr.dbg.value #local = %arg0 : i32 loc(fused<#sp>["test.c":1:1])
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])
+
+// CHECK-DAG: ![[SP]] = distinct !DISubprogram(name: "fn_with_retained_local"{{.*}}retainedNodes: ![[NODES:[0-9]+]])
+// CHECK-DAG: ![[LOCAL]] = !DILocalVariable(name: "n", arg: 1, scope: ![[SP]], file: !{{.*}}, line: 1, type: !{{.*}})
+// CHECK-DAG: ![[NODES]] = !{![[LOCAL]]}
+
+// -----
+
 // Nameless and scopeless global constant.
 
 // CHECK-LABEL: @.str.1 = external constant [10 x i8]


From llvmlistbot at llvm.org  Thu Jul 16 00:55:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:55:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588e85.824e11a6.1389c5.d3c1@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Thu Jul 16 00:56:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:56:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588ebc.5a0412d2.82e8e.1470@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- mlir/lib/CAPI/Dialect/LLVM.cpp mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp mlir/lib/Target/LLVMIR/DebugImporter.cpp mlir/lib/Target/LLVMIR/DebugImporter.h mlir/lib/Target/LLVMIR/DebugTranslation.cpp mlir/lib/Target/LLVMIR/DebugTranslation.h mlir/lib/Target/LLVMIR/ModuleImport.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index eb7e96925..d59fb3c86 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -415,9 +415,9 @@ MlirAttribute mlirLLVMDILocalVariableAttrGet(
     unsigned int alignInBits, MlirAttribute diType, int64_t flags) {
   return wrap(DILocalVariableAttr::get(
       unwrap(ctx), /*recId=*/DistinctAttr{}, /*isRecSelf=*/false,
-      cast<DIScopeAttr>(unwrap(scope)),
-      cast<StringAttr>(unwrap(name)), cast<DIFileAttr>(unwrap(diFile)), line,
-      arg, alignInBits, cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
+      cast<DIScopeAttr>(unwrap(scope)), cast<StringAttr>(unwrap(name)),
+      cast<DIFileAttr>(unwrap(diFile)), line, arg, alignInBits,
+      cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
 }
 
 MlirStringRef mlirLLVMDILocalVariableAttrGetName(void) {
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index fc387c0eb..473935fcf 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -363,10 +363,10 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
 
   if (auto localVariable = dyn_cast<DILocalVariableAttr>(attr)) {
     if (localVariable.getIsRecSelf()) {
-      auto iter = forwardLocalVariablePlaceholders
-                      .try_emplace(recursiveId,
-                                   translateTemporaryImpl(localVariable))
-                      .first;
+      auto iter =
+          forwardLocalVariablePlaceholders
+              .try_emplace(recursiveId, translateTemporaryImpl(localVariable))
+              .first;
       return iter->second.get();
     }
     if (auto iter = forwardLocalVariablePlaceholders.find(recursiveId);

``````````

</details>


https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Thu Jul 16 00:58:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 00:58:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588f43.f1c1923d.1bf95f.3bd8@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/209911

>From 16737080d14aaca751ab19e622c9b6f22e27c903 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Wed, 15 Jul 2026 14:58:04 -0700
Subject: [PATCH] [mlir][LLVMIR] Avoid duplicate retained local variables

Use recursive DILocalVariableAttr for retained locals that also appear in debug intrinsics. This avoids importing the same LLVM metadata node twice.
---
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 26 ++++++++--
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |  4 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |  7 +--
 mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp      | 19 ++++++++
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      | 42 +++++++++++++----
 mlir/lib/Target/LLVMIR/DebugImporter.h        | 13 +++++
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   | 47 ++++++++++++++++++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     | 10 ++++
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 33 +++++++++++++
 mlir/test/Target/LLVMIR/Import/debug-info.ll  | 33 +++++++++++++
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     | 40 ++++++++++++++++
 11 files changed, 257 insertions(+), 17 deletions(-)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 6e2d4628c6173..fb17f1efb7c94 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -696,9 +696,14 @@ def LLVM_DILexicalBlockFile : LLVM_Attr<"DILexicalBlockFile", "di_lexical_block_
 //===----------------------------------------------------------------------===//
 
 def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
-                                         /*traits=*/[], "DINodeAttr"> {
+                                         [LLVM_DIRecursiveTypeAttrInterface],
+                                         "DINodeAttr"> {
   let parameters = (ins
-    "DIScopeAttr":$scope,
+    // DIRecursiveTypeAttrInterface specific parameters.
+    OptionalParameter<"DistinctAttr">:$recId,
+    OptionalParameter<"bool">:$isRecSelf,
+    // DILocalVariableAttr specific parameters.
+    OptionalParameter<"DIScopeAttr">:$scope,
     OptionalParameter<"StringAttr">:$name,
     OptionalParameter<"DIFileAttr">:$file,
     OptionalParameter<"unsigned">:$line,
@@ -714,11 +719,24 @@ def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
       "DITypeAttr":$type, "DIFlags":$flags
     ), [{
       MLIRContext *ctx = scope.getContext();
-      return $_get(ctx, scope, StringAttr::get(ctx, name), file, line,
-                   arg, alignInBits, type, flags);
+      return $_get(ctx, /*recId=*/nullptr, /*isRecSelf=*/false, scope,
+                   StringAttr::get(ctx, name), file, line, arg, alignInBits,
+                   type, flags);
     }]>
   ];
   let assemblyFormat = "`<` struct(params) `>`";
+  let extraClassDeclaration = [{
+    /// Requirements of DIRecursiveTypeAttrInterface.
+    /// @{
+
+    /// Get a copy of this attr but with the recursive ID set to `recId`.
+    DIRecursiveTypeAttrInterface withRecId(DistinctAttr recId);
+
+    /// Build a rec-self instance using the provided `recId`.
+    static DIRecursiveTypeAttrInterface getRecSelf(DistinctAttr recId);
+
+    /// @}
+  }];
 
   // Generate mnemonic alias for the attribute.
   let genMnemonicAlias = 1;
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index b25a8c8a9829e..ef7df3c204841 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -99,7 +99,9 @@ def DIFileAttr : DialectAttribute<(attr
 //===----------------------------------------------------------------------===//
 
 def DILocalVariableAttr : DialectAttribute<(attr
-  Attr<"DIScopeAttr">:$scope,
+  OptionalAttribute<"DistinctAttr">:$recId,
+  Bool:$isRecSelf,
+  OptionalAttribute<"DIScopeAttr">:$scope,
   OptionalAttribute<"StringAttr">:$name,
   OptionalAttribute<"DIFileAttr">:$file,
   VarInt:$line,
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index 8082de122e74e..d59fb3c86d903 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -414,9 +414,10 @@ MlirAttribute mlirLLVMDILocalVariableAttrGet(
     MlirAttribute diFile, unsigned int line, unsigned int arg,
     unsigned int alignInBits, MlirAttribute diType, int64_t flags) {
   return wrap(DILocalVariableAttr::get(
-      unwrap(ctx), cast<DIScopeAttr>(unwrap(scope)),
-      cast<StringAttr>(unwrap(name)), cast<DIFileAttr>(unwrap(diFile)), line,
-      arg, alignInBits, cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
+      unwrap(ctx), /*recId=*/DistinctAttr{}, /*isRecSelf=*/false,
+      cast<DIScopeAttr>(unwrap(scope)), cast<StringAttr>(unwrap(name)),
+      cast<DIFileAttr>(unwrap(diFile)), line, arg, alignInBits,
+      cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
 }
 
 MlirStringRef mlirLLVMDILocalVariableAttrGetName(void) {
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
index 266faeca4553b..7917ac8b1ff10 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
@@ -325,6 +325,25 @@ void printExpressionArg(AsmPrinter &printer, uint64_t opcode,
   });
 }
 
+//===----------------------------------------------------------------------===//
+// DILocalVariableAttr
+//===----------------------------------------------------------------------===//
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::withRecId(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      getContext(), recId, getIsRecSelf(), getScope(), getName(), getFile(),
+      getLine(), getArg(), getAlignInBits(), getType(), getFlags());
+}
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::getRecSelf(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      recId.getContext(), recId, /*isRecSelf=*/true, /*scope=*/{},
+      /*name=*/{}, /*file=*/{}, /*line=*/0, /*arg=*/0, /*alignInBits=*/0,
+      /*type=*/{}, DIFlags::Zero);
+}
+
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 85df880d2712a..f8ccbc5bb6637 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -17,6 +17,7 @@
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Metadata.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -193,13 +194,21 @@ DebugImporter::translateImpl(llvm::DIGlobalVariable *node) {
 
 DILocalVariableAttr DebugImporter::translateImpl(llvm::DILocalVariable *node) {
   // Return nullptr if the scope or type is a cyclic dependency.
-  DIScopeAttr scope = translate(node->getScope());
+  DIScopeAttr scope;
+  {
+    bool preferLocalVariableCycleBreakForNode =
+        static_cast<bool>(nodeToRecId.lookup(node));
+    llvm::SaveAndRestore preferLocalVariableCycleBreakGuard(
+        preferLocalVariableCycleBreak, preferLocalVariableCycleBreakForNode);
+    scope = translate(node->getScope());
+  }
   if (node->getScope() && !scope)
     return nullptr;
   return DILocalVariableAttr::get(
-      context, scope, getStringAttrOrNull(node->getRawName()),
-      translate(node->getFile()), node->getLine(), node->getArg(),
-      node->getAlignInBits(), translate(node->getType()),
+      context, /*recId=*/DistinctAttr{}, /*isRecSelf=*/false, scope,
+      getStringAttrOrNull(node->getRawName()), translate(node->getFile()),
+      node->getLine(), node->getArg(), node->getAlignInBits(),
+      translate(node->getType()),
       symbolizeDIFlags(node->getFlags()).value_or(DIFlags::Zero));
 }
 
@@ -261,6 +270,11 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   // Convert the retained nodes but drop all of them if one of them is invalid.
   SmallVector<Attribute> retainedNodes;
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
+    if (auto *local = dyn_cast<llvm::DILocalVariable>(retainedNode);
+        local && nodeToRecId.contains(local)) {
+      retainedNodes.push_back(*createRecSelf(local));
+      return;
+    }
     retainedNodes.push_back(translate(retainedNode));
   };
   auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
@@ -446,6 +460,9 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
       DistinctAttr recId = nodeToRecId.lookup(node);
       auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
       attr = cast<DINodeAttr>(recType.withRecId(recId));
+    } else if (DistinctAttr recId = nodeToRecId.lookup(node)) {
+      auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
+      attr = cast<DINodeAttr>(recType.withRecId(recId));
     }
     cacheEntry.resolve(attr);
     return attr;
@@ -454,22 +471,31 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
   return nullptr;
 }
 
+void DebugImporter::markLocalVariableAsRecursive(llvm::DILocalVariable *node) {
+  if (!nodeToRecId.contains(node))
+    nodeToRecId[node] = DistinctAttr::create(UnitAttr::get(context));
+}
+
 /// Get the `getRecSelf` constructor for the translated node if it participates
-/// in CyclicReplacerCache cycle breaking (recursive composite types,
-/// subprograms, or compile units).
-static function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
-getRecSelfConstructor(llvm::DINode *node) {
+/// in CyclicReplacerCache cycle breaking.
+function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+DebugImporter::getRecSelfConstructor(llvm::DINode *node) {
   using CtorType = function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>;
   return TypeSwitch<llvm::DINode *, CtorType>(node)
       .Case([&](llvm::DICompositeType *) {
         return CtorType(DICompositeTypeAttr::getRecSelf);
       })
       .Case([&](llvm::DISubprogram *) {
+        if (preferLocalVariableCycleBreak)
+          return CtorType();
         return CtorType(DISubprogramAttr::getRecSelf);
       })
       .Case([&](llvm::DICompileUnit *) {
         return CtorType(DICompileUnitAttr::getRecSelf);
       })
+      .Case([&](llvm::DILocalVariable *) {
+        return CtorType(DILocalVariableAttr::getRecSelf);
+      })
       .Default(CtorType());
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.h b/mlir/lib/Target/LLVMIR/DebugImporter.h
index b224ad3376be7..40bf0b113e28f 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.h
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.h
@@ -47,6 +47,9 @@ class DebugImporter {
   /// Returns UnknownLoc if `func` has no debug information attached to it.
   Location translateFuncLocation(llvm::Function *func);
 
+  /// Marks the local variable as requiring a recursive representation.
+  void markLocalVariableAsRecursive(llvm::DILocalVariable *node);
+
   /// Translates the given LLVM debug metadata to MLIR.
   DINodeAttr translate(llvm::DINode *node);
 
@@ -92,6 +95,11 @@ class DebugImporter {
   /// for it, or create a new one if not.
   DistinctAttr getOrCreateDistinctID(llvm::DINode *node);
 
+  /// Get the `getRecSelf` constructor for the translated node if it
+  /// participates in CyclicReplacerCache cycle breaking.
+  function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+  getRecSelfConstructor(llvm::DINode *node);
+
   std::optional<DINodeAttr> createRecSelf(llvm::DINode *node);
 
   /// A mapping between distinct LLVM debug metadata nodes and the corresponding
@@ -103,6 +111,11 @@ class DebugImporter {
   /// same ID and be deduplicated.
   DenseMap<llvm::DINode *, DistinctAttr> nodeToRecId;
 
+  /// True while translating the scope of a DILocalVariable. This lets a
+  /// local-variable/subprogram cycle be broken at the local variable instead of
+  /// at the subprogram.
+  bool preferLocalVariableCycleBreak = false;
+
   CyclicReplacerCache<llvm::DINode *, DINodeAttr> cache;
 
   MLIRContext *context;
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 442f5b6b137e6..473935fcf38d3 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -16,6 +16,7 @@
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -329,6 +330,20 @@ DebugTranslation::translateImpl(DILocalVariableAttr attr) {
       /*Annotations=*/nullptr);
 }
 
+llvm::TempDILocalVariable
+DebugTranslation::translateTemporaryImpl(DILocalVariableAttr attr) {
+  llvm::DILocalScope *scope =
+      attr.getScope() ? cast<llvm::DILocalScope>(translate(attr.getScope()))
+                      : localVariablePlaceholderScope;
+  assert(scope && "expected local variable placeholder scope");
+  return llvm::DILocalVariable::getTemporary(
+      llvmCtx, scope, /*Name=*/"", /*File=*/nullptr, attr.getLine(),
+      /*Type=*/static_cast<llvm::DIType *>(nullptr), attr.getArg(),
+      static_cast<llvm::DINode::DIFlags>(attr.getFlags()),
+      attr.getAlignInBits(),
+      /*Annotations=*/nullptr);
+}
+
 llvm::DIGlobalVariable *
 DebugTranslation::translateImpl(DIGlobalVariableAttr attr) {
   return llvm::DIGlobalVariable::getDistinct(
@@ -345,6 +360,24 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
       iter != recursiveNodeMap.end()) {
     return iter->second;
   }
+
+  if (auto localVariable = dyn_cast<DILocalVariableAttr>(attr)) {
+    if (localVariable.getIsRecSelf()) {
+      auto iter =
+          forwardLocalVariablePlaceholders
+              .try_emplace(recursiveId, translateTemporaryImpl(localVariable))
+              .first;
+      return iter->second.get();
+    }
+    if (auto iter = forwardLocalVariablePlaceholders.find(recursiveId);
+        iter != forwardLocalVariablePlaceholders.end()) {
+      llvm::DILocalVariable *concrete = translateImpl(localVariable);
+      iter->second->replaceAllUsesWith(concrete);
+      forwardLocalVariablePlaceholders.erase(iter);
+      return concrete;
+    }
+  }
+
   assert(!attr.getIsRecSelf() && "unbound DI recursive self reference");
 
   auto setRecursivePlaceholder = [&](llvm::DINode *placeholder) {
@@ -377,6 +410,13 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
             auto *concrete = translateImpl(attr);
             temporary->replaceAllUsesWith(concrete);
             return concrete;
+          })
+          .Case([&](DILocalVariableAttr attr) {
+            auto temporary = translateTemporaryImpl(attr);
+            setRecursivePlaceholder(temporary.get());
+            auto *concrete = translateImpl(attr);
+            temporary->replaceAllUsesWith(concrete);
+            return concrete;
           });
 
   assert(recursiveNodeMap.back().first == recursiveId &&
@@ -417,10 +457,15 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
+      /*RetainedNodes=*/nullptr, /*ThrownTypes=*/nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
+  if (!attr.getRetainedNodes().empty()) {
+    llvm::SaveAndRestore guard(localVariablePlaceholderScope,
+                               cast<llvm::DILocalScope>(node));
+    node->replaceRetainedNodes(getRetainedNodesOrNull(attr.getRetainedNodes()));
+  }
   return node;
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index c1fdbd89087da..2db7f401f576d 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -105,6 +105,7 @@ class DebugTranslation {
   llvm::TempDICompositeType translateTemporaryImpl(DICompositeTypeAttr attr);
   llvm::TempDISubprogram translateTemporaryImpl(DISubprogramAttr attr);
   llvm::TempDICompileUnit translateTemporaryImpl(DICompileUnitAttr attr);
+  llvm::TempDILocalVariable translateTemporaryImpl(DILocalVariableAttr attr);
 
   /// Constructs a string metadata node from the string attribute. Returns
   /// nullptr if `stringAttr` is null or contains and empty string.
@@ -135,6 +136,15 @@ class DebugTranslation {
   /// A mapping between recursive ID and the translated DINode.
   llvm::MapVector<DistinctAttr, llvm::DINode *> recursiveNodeMap;
 
+  /// A mapping between recursive IDs and local-variable placeholders created
+  /// from forward self references in retainedNodes.
+  DenseMap<DistinctAttr, llvm::TempDILocalVariable>
+      forwardLocalVariablePlaceholders;
+
+  /// Scope to use for forward local-variable placeholders while translating a
+  /// DISubprogram's retainedNodes.
+  llvm::DILocalScope *localVariablePlaceholderScope = nullptr;
+
   /// A mapping between a distinct ID and the translated LLVM metadata node.
   /// This helps identify attrs that should translate into the same LLVM debug
   /// node.
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index bb55678f91005..fc4ad29d2d747 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -35,6 +35,7 @@
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugProgramInstruction.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/InstIterator.h"
@@ -246,6 +247,38 @@ ModuleImport::ModuleImport(ModuleOp mlirModule,
       emitExpensiveWarnings(emitExpensiveWarnings),
       preferUnregisteredIntrinsics(preferUnregisteredIntrinsics) {
   builder.setInsertionPointToStart(mlirModule.getBody());
+
+  // Variables that are both retained and used by a debug intrinsic need to
+  // break cycles at the variable itself to avoid creating duplicate attrs with
+  // different subprogram scopes.
+  llvm::SmallPtrSet<llvm::DILocalVariable *, 8> retainedLocalVariables;
+  for (llvm::Function &func : *this->llvmModule) {
+    llvm::DISubprogram *subprogram = func.getSubprogram();
+    if (!subprogram)
+      continue;
+    auto addLocalVariable = [&](llvm::DILocalVariable *localVariable) {
+      retainedLocalVariables.insert(localVariable);
+    };
+    auto skip = [](auto *) {};
+    subprogram->forEachRetainedNode(addLocalVariable, skip, skip, skip, skip);
+  }
+
+  auto markIfRetained = [&](llvm::DILocalVariable *localVariable) {
+    if (localVariable && retainedLocalVariables.contains(localVariable))
+      debugImporter->markLocalVariableAsRecursive(localVariable);
+  };
+  for (llvm::Function &func : *this->llvmModule) {
+    for (llvm::Instruction &inst : llvm::instructions(func)) {
+      if (auto *dbgIntrinsic = dyn_cast<llvm::DbgVariableIntrinsic>(&inst))
+        markIfRetained(dbgIntrinsic->getVariable());
+      if (!inst.DebugMarker)
+        continue;
+      for (llvm::DbgRecord &dbgRecord : inst.DebugMarker->getDbgRecordRange())
+        if (auto *dbgVariableRecord =
+                dyn_cast<llvm::DbgVariableRecord>(&dbgRecord))
+          markIfRetained(dbgVariableRecord->getVariable());
+    }
+  }
 }
 
 ComdatOp ModuleImport::getGlobalComdatOp() {
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 28c25809002cd..38b250f00b958 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1046,3 +1046,36 @@ define void @fn_with_static_local() !dbg !3 {
 !12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
 !13 = !DISubroutineType(types: !14)
 !14 = !{null}
+
+; // -----
+
+; Test that a DILocalVariable listed in DISubprogram's retainedNodes and used by
+; a debug intrinsic does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR_SELF:.+]] = #llvm.di_local_variable<recId = [[REC_ID:distinct.+<>]], isRecSelf = true>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_retained_local"{{.*}}retainedNodes = [#[[LVAR_SELF]]]>
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<recId = [[REC_ID]], scope = #[[SP]], name = "n"{{.*}}>
+; CHECK-DAG-COUNT-2: {{#llvm\.di_local_variable<}}
+; CHECK: llvm.intr.dbg.value #[[LVAR]] = %{{.*}} : i32
+
+define void @fn_with_retained_local(i32 %n) !dbg !3 {
+entry:
+  call void @llvm.dbg.value(metadata i32 %n, metadata !12, metadata !DIExpression()), !dbg !15
+  ret void
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!3 = distinct !DISubprogram(name: "fn_with_retained_local", scope: !4, file: !4, line: 1, type: !13, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!12}
+!12 = !DILocalVariable(name: "n", arg: 1, scope: !3, file: !4, line: 1, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null, !9}
+!15 = !DILocation(line: 1, column: 1, scope: !3)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 72dadb2602c52..77eade68ae9e3 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -421,6 +421,46 @@ llvm.func @imp_fn() {
 
 // -----
 
+// Test that a recursive local variable retained by a subprogram is exported
+// without duplicating the DILocalVariable metadata node.
+
+#file = #llvm.di_file<"test.c" in "">
+#cu = #llvm.di_compile_unit<
+  id = distinct[0]<>, sourceLanguage = DW_LANG_C, file = #file,
+  isOptimized = false, emissionKind = Full
+>
+#void = #llvm.di_null_type
+#void_type = #llvm.di_subroutine_type<types = #void>
+#int = #llvm.di_basic_type<
+  tag = DW_TAG_base_type, name = "int",
+  sizeInBits = 32, encoding = DW_ATE_signed
+>
+#local_self = #llvm.di_local_variable<recId = distinct[1]<>, isRecSelf = true>
+#sp = #llvm.di_subprogram<
+  id = distinct[2]<>, compileUnit = #cu, scope = #file,
+  name = "fn_with_retained_local", file = #file,
+  subprogramFlags = Definition, type = #void_type,
+  retainedNodes = [#local_self]
+>
+#local = #llvm.di_local_variable<
+  recId = distinct[1]<>, scope = #sp, name = "n", file = #file,
+  line = 1, arg = 1, type = #int
+>
+
+// CHECK-LABEL: define void @fn_with_retained_local(
+// CHECK-SAME: i32 %[[ARG:.*]]) !dbg ![[SP:[0-9]+]]
+llvm.func @fn_with_retained_local(%arg0: i32) {
+  // RECORDS: #dbg_value(i32 %[[ARG]], ![[LOCAL:[0-9]+]], !DIExpression(), !{{.*}})
+  llvm.intr.dbg.value #local = %arg0 : i32 loc(fused<#sp>["test.c":1:1])
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])
+
+// CHECK-DAG: ![[SP]] = distinct !DISubprogram(name: "fn_with_retained_local"{{.*}}retainedNodes: ![[NODES:[0-9]+]])
+// CHECK-DAG: ![[LOCAL]] = !DILocalVariable(name: "n", arg: 1, scope: ![[SP]], file: !{{.*}}, line: 1, type: !{{.*}})
+// CHECK-DAG: ![[NODES]] = !{![[LOCAL]]}
+
+// -----
+
 // Nameless and scopeless global constant.
 
 // CHECK-LABEL: @.str.1 = external constant [10 x i8]


From llvmlistbot at llvm.org  Thu Jul 16 01:01:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 01:01:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588fcf.fe2ec69d.1e87df.a8c7@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Thu Jul 16 01:01:53 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 01:01:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a588ff1.65514cde.1dc38f.ab00@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Thu Jul 16 01:06:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 01:06:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a5890ee.f06f4994.1c712e.39c7@mx.google.com>


LouisLu060211 wrote:

> Hello @LouisLu060211 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community



https://github.com/llvm/llvm-project/pull/207670

From llvmlistbot at llvm.org  Thu Jul 16 01:14:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 01:14:58 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][LLVMIR] Avoid duplicate
 retained local variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a589302.481afadf.34be4b.370e@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/209911

>From 818ce27934a2b483f0d0864243721b7492a55ddd Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Wed, 15 Jul 2026 14:58:04 -0700
Subject: [PATCH] [mlir][LLVMIR] Avoid duplicate retained local variables

Use recursive DILocalVariableAttr for retained locals that also appear in debug intrinsics. This avoids importing the same LLVM metadata node twice.
---
 .../lib/Optimizer/Transforms/AddDebugInfo.cpp |  5 +-
 .../Transforms/DebugTypeGenerator.cpp         |  2 +-
 .../mlir/Dialect/LLVMIR/LLVMAttrDefs.td       | 26 ++++++++--
 .../Dialect/LLVMIR/LLVMDialectBytecode.td     |  4 +-
 mlir/lib/CAPI/Dialect/LLVM.cpp                |  7 +--
 mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp      | 19 ++++++++
 mlir/lib/Target/LLVMIR/DebugImporter.cpp      | 42 +++++++++++++----
 mlir/lib/Target/LLVMIR/DebugImporter.h        | 13 +++++
 mlir/lib/Target/LLVMIR/DebugTranslation.cpp   | 47 ++++++++++++++++++-
 mlir/lib/Target/LLVMIR/DebugTranslation.h     | 10 ++++
 mlir/lib/Target/LLVMIR/ModuleImport.cpp       | 33 +++++++++++++
 mlir/test/Target/LLVMIR/Import/debug-info.ll  | 33 +++++++++++++
 mlir/test/Target/LLVMIR/llvmir-debug.mlir     | 40 ++++++++++++++++
 13 files changed, 260 insertions(+), 21 deletions(-)

diff --git a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
index 7c0b32e48832e..6004e5509e6e6 100644
--- a/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
+++ b/flang/lib/Optimizer/Transforms/AddDebugInfo.cpp
@@ -363,9 +363,8 @@ void AddDebugInfoPass::handleLocalVariable(Op declOp, llvm::StringRef name,
   }
 
   auto localVarAttr = mlir::LLVM::DILocalVariableAttr::get(
-      context, scopeAttr, mlir::StringAttr::get(context, name), fileAttr,
-      getLineFromLoc(declOp.getLoc()), argNo, /* alignInBits*/ 0, tyAttr,
-      mlir::LLVM::DIFlags::Zero);
+      scopeAttr, name, fileAttr, getLineFromLoc(declOp.getLoc()), argNo,
+      /*alignInBits=*/0, tyAttr, mlir::LLVM::DIFlags::Zero);
   declOp->setLoc(builder.getFusedLoc({declOp->getLoc()}, localVarAttr));
 }
 
diff --git a/flang/lib/Optimizer/Transforms/DebugTypeGenerator.cpp b/flang/lib/Optimizer/Transforms/DebugTypeGenerator.cpp
index d45aee26470a7..b3f3186288685 100644
--- a/flang/lib/Optimizer/Transforms/DebugTypeGenerator.cpp
+++ b/flang/lib/Optimizer/Transforms/DebugTypeGenerator.cpp
@@ -114,7 +114,7 @@ mlir::LLVM::DILocalVariableAttr DebugTypeGenerator::generateArtificialVariable(
   }
   mlir::LLVM::DITypeAttr Ty = convertType(type, fileAttr, scope, declOp);
   auto lvAttr = mlir::LLVM::DILocalVariableAttr::get(
-      context, scope, name, fileAttr, /*line=*/0, /*argNo=*/0,
+      scope, name.getValue(), fileAttr, /*line=*/0, /*argNo=*/0,
       /*alignInBits=*/0, Ty, mlir::LLVM::DIFlags::Artificial);
   mlir::LLVM::DbgValueOp::create(builder, declOp.getLoc(), val, lvAttr,
                                  nullptr);
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
index 6e2d4628c6173..fb17f1efb7c94 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMAttrDefs.td
@@ -696,9 +696,14 @@ def LLVM_DILexicalBlockFile : LLVM_Attr<"DILexicalBlockFile", "di_lexical_block_
 //===----------------------------------------------------------------------===//
 
 def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
-                                         /*traits=*/[], "DINodeAttr"> {
+                                         [LLVM_DIRecursiveTypeAttrInterface],
+                                         "DINodeAttr"> {
   let parameters = (ins
-    "DIScopeAttr":$scope,
+    // DIRecursiveTypeAttrInterface specific parameters.
+    OptionalParameter<"DistinctAttr">:$recId,
+    OptionalParameter<"bool">:$isRecSelf,
+    // DILocalVariableAttr specific parameters.
+    OptionalParameter<"DIScopeAttr">:$scope,
     OptionalParameter<"StringAttr">:$name,
     OptionalParameter<"DIFileAttr">:$file,
     OptionalParameter<"unsigned">:$line,
@@ -714,11 +719,24 @@ def LLVM_DILocalVariableAttr : LLVM_Attr<"DILocalVariable", "di_local_variable",
       "DITypeAttr":$type, "DIFlags":$flags
     ), [{
       MLIRContext *ctx = scope.getContext();
-      return $_get(ctx, scope, StringAttr::get(ctx, name), file, line,
-                   arg, alignInBits, type, flags);
+      return $_get(ctx, /*recId=*/nullptr, /*isRecSelf=*/false, scope,
+                   StringAttr::get(ctx, name), file, line, arg, alignInBits,
+                   type, flags);
     }]>
   ];
   let assemblyFormat = "`<` struct(params) `>`";
+  let extraClassDeclaration = [{
+    /// Requirements of DIRecursiveTypeAttrInterface.
+    /// @{
+
+    /// Get a copy of this attr but with the recursive ID set to `recId`.
+    DIRecursiveTypeAttrInterface withRecId(DistinctAttr recId);
+
+    /// Build a rec-self instance using the provided `recId`.
+    static DIRecursiveTypeAttrInterface getRecSelf(DistinctAttr recId);
+
+    /// @}
+  }];
 
   // Generate mnemonic alias for the attribute.
   let genMnemonicAlias = 1;
diff --git a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
index b25a8c8a9829e..ef7df3c204841 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/LLVMDialectBytecode.td
@@ -99,7 +99,9 @@ def DIFileAttr : DialectAttribute<(attr
 //===----------------------------------------------------------------------===//
 
 def DILocalVariableAttr : DialectAttribute<(attr
-  Attr<"DIScopeAttr">:$scope,
+  OptionalAttribute<"DistinctAttr">:$recId,
+  Bool:$isRecSelf,
+  OptionalAttribute<"DIScopeAttr">:$scope,
   OptionalAttribute<"StringAttr">:$name,
   OptionalAttribute<"DIFileAttr">:$file,
   VarInt:$line,
diff --git a/mlir/lib/CAPI/Dialect/LLVM.cpp b/mlir/lib/CAPI/Dialect/LLVM.cpp
index 8082de122e74e..d59fb3c86d903 100644
--- a/mlir/lib/CAPI/Dialect/LLVM.cpp
+++ b/mlir/lib/CAPI/Dialect/LLVM.cpp
@@ -414,9 +414,10 @@ MlirAttribute mlirLLVMDILocalVariableAttrGet(
     MlirAttribute diFile, unsigned int line, unsigned int arg,
     unsigned int alignInBits, MlirAttribute diType, int64_t flags) {
   return wrap(DILocalVariableAttr::get(
-      unwrap(ctx), cast<DIScopeAttr>(unwrap(scope)),
-      cast<StringAttr>(unwrap(name)), cast<DIFileAttr>(unwrap(diFile)), line,
-      arg, alignInBits, cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
+      unwrap(ctx), /*recId=*/DistinctAttr{}, /*isRecSelf=*/false,
+      cast<DIScopeAttr>(unwrap(scope)), cast<StringAttr>(unwrap(name)),
+      cast<DIFileAttr>(unwrap(diFile)), line, arg, alignInBits,
+      cast<DITypeAttr>(unwrap(diType)), DIFlags(flags)));
 }
 
 MlirStringRef mlirLLVMDILocalVariableAttrGetName(void) {
diff --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
index 266faeca4553b..7917ac8b1ff10 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMAttrs.cpp
@@ -325,6 +325,25 @@ void printExpressionArg(AsmPrinter &printer, uint64_t opcode,
   });
 }
 
+//===----------------------------------------------------------------------===//
+// DILocalVariableAttr
+//===----------------------------------------------------------------------===//
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::withRecId(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      getContext(), recId, getIsRecSelf(), getScope(), getName(), getFile(),
+      getLine(), getArg(), getAlignInBits(), getType(), getFlags());
+}
+
+DIRecursiveTypeAttrInterface
+DILocalVariableAttr::getRecSelf(DistinctAttr recId) {
+  return DILocalVariableAttr::get(
+      recId.getContext(), recId, /*isRecSelf=*/true, /*scope=*/{},
+      /*name=*/{}, /*file=*/{}, /*line=*/0, /*arg=*/0, /*alignInBits=*/0,
+      /*type=*/{}, DIFlags::Zero);
+}
+
 //===----------------------------------------------------------------------===//
 // DICompositeTypeAttr
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.cpp b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
index 85df880d2712a..f8ccbc5bb6637 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.cpp
@@ -17,6 +17,7 @@
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/Metadata.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -193,13 +194,21 @@ DebugImporter::translateImpl(llvm::DIGlobalVariable *node) {
 
 DILocalVariableAttr DebugImporter::translateImpl(llvm::DILocalVariable *node) {
   // Return nullptr if the scope or type is a cyclic dependency.
-  DIScopeAttr scope = translate(node->getScope());
+  DIScopeAttr scope;
+  {
+    bool preferLocalVariableCycleBreakForNode =
+        static_cast<bool>(nodeToRecId.lookup(node));
+    llvm::SaveAndRestore preferLocalVariableCycleBreakGuard(
+        preferLocalVariableCycleBreak, preferLocalVariableCycleBreakForNode);
+    scope = translate(node->getScope());
+  }
   if (node->getScope() && !scope)
     return nullptr;
   return DILocalVariableAttr::get(
-      context, scope, getStringAttrOrNull(node->getRawName()),
-      translate(node->getFile()), node->getLine(), node->getArg(),
-      node->getAlignInBits(), translate(node->getType()),
+      context, /*recId=*/DistinctAttr{}, /*isRecSelf=*/false, scope,
+      getStringAttrOrNull(node->getRawName()), translate(node->getFile()),
+      node->getLine(), node->getArg(), node->getAlignInBits(),
+      translate(node->getType()),
       symbolizeDIFlags(node->getFlags()).value_or(DIFlags::Zero));
 }
 
@@ -261,6 +270,11 @@ DISubprogramAttr DebugImporter::translateImpl(llvm::DISubprogram *node) {
   // Convert the retained nodes but drop all of them if one of them is invalid.
   SmallVector<Attribute> retainedNodes;
   auto add = [this, &retainedNodes](llvm::DINode *retainedNode) {
+    if (auto *local = dyn_cast<llvm::DILocalVariable>(retainedNode);
+        local && nodeToRecId.contains(local)) {
+      retainedNodes.push_back(*createRecSelf(local));
+      return;
+    }
     retainedNodes.push_back(translate(retainedNode));
   };
   auto addGVE = [](llvm::DIGlobalVariableExpression *GVE) {
@@ -446,6 +460,9 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
       DistinctAttr recId = nodeToRecId.lookup(node);
       auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
       attr = cast<DINodeAttr>(recType.withRecId(recId));
+    } else if (DistinctAttr recId = nodeToRecId.lookup(node)) {
+      auto recType = cast<DIRecursiveTypeAttrInterface>(attr);
+      attr = cast<DINodeAttr>(recType.withRecId(recId));
     }
     cacheEntry.resolve(attr);
     return attr;
@@ -454,22 +471,31 @@ DINodeAttr DebugImporter::translate(llvm::DINode *node) {
   return nullptr;
 }
 
+void DebugImporter::markLocalVariableAsRecursive(llvm::DILocalVariable *node) {
+  if (!nodeToRecId.contains(node))
+    nodeToRecId[node] = DistinctAttr::create(UnitAttr::get(context));
+}
+
 /// Get the `getRecSelf` constructor for the translated node if it participates
-/// in CyclicReplacerCache cycle breaking (recursive composite types,
-/// subprograms, or compile units).
-static function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
-getRecSelfConstructor(llvm::DINode *node) {
+/// in CyclicReplacerCache cycle breaking.
+function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+DebugImporter::getRecSelfConstructor(llvm::DINode *node) {
   using CtorType = function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>;
   return TypeSwitch<llvm::DINode *, CtorType>(node)
       .Case([&](llvm::DICompositeType *) {
         return CtorType(DICompositeTypeAttr::getRecSelf);
       })
       .Case([&](llvm::DISubprogram *) {
+        if (preferLocalVariableCycleBreak)
+          return CtorType();
         return CtorType(DISubprogramAttr::getRecSelf);
       })
       .Case([&](llvm::DICompileUnit *) {
         return CtorType(DICompileUnitAttr::getRecSelf);
       })
+      .Case([&](llvm::DILocalVariable *) {
+        return CtorType(DILocalVariableAttr::getRecSelf);
+      })
       .Default(CtorType());
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugImporter.h b/mlir/lib/Target/LLVMIR/DebugImporter.h
index b224ad3376be7..40bf0b113e28f 100644
--- a/mlir/lib/Target/LLVMIR/DebugImporter.h
+++ b/mlir/lib/Target/LLVMIR/DebugImporter.h
@@ -47,6 +47,9 @@ class DebugImporter {
   /// Returns UnknownLoc if `func` has no debug information attached to it.
   Location translateFuncLocation(llvm::Function *func);
 
+  /// Marks the local variable as requiring a recursive representation.
+  void markLocalVariableAsRecursive(llvm::DILocalVariable *node);
+
   /// Translates the given LLVM debug metadata to MLIR.
   DINodeAttr translate(llvm::DINode *node);
 
@@ -92,6 +95,11 @@ class DebugImporter {
   /// for it, or create a new one if not.
   DistinctAttr getOrCreateDistinctID(llvm::DINode *node);
 
+  /// Get the `getRecSelf` constructor for the translated node if it
+  /// participates in CyclicReplacerCache cycle breaking.
+  function_ref<DIRecursiveTypeAttrInterface(DistinctAttr)>
+  getRecSelfConstructor(llvm::DINode *node);
+
   std::optional<DINodeAttr> createRecSelf(llvm::DINode *node);
 
   /// A mapping between distinct LLVM debug metadata nodes and the corresponding
@@ -103,6 +111,11 @@ class DebugImporter {
   /// same ID and be deduplicated.
   DenseMap<llvm::DINode *, DistinctAttr> nodeToRecId;
 
+  /// True while translating the scope of a DILocalVariable. This lets a
+  /// local-variable/subprogram cycle be broken at the local variable instead of
+  /// at the subprogram.
+  bool preferLocalVariableCycleBreak = false;
+
   CyclicReplacerCache<llvm::DINode *, DINodeAttr> cache;
 
   MLIRContext *context;
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
index 442f5b6b137e6..473935fcf38d3 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.cpp
@@ -16,6 +16,7 @@
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/SaveAndRestore.h"
 
 using namespace mlir;
 using namespace mlir::LLVM;
@@ -329,6 +330,20 @@ DebugTranslation::translateImpl(DILocalVariableAttr attr) {
       /*Annotations=*/nullptr);
 }
 
+llvm::TempDILocalVariable
+DebugTranslation::translateTemporaryImpl(DILocalVariableAttr attr) {
+  llvm::DILocalScope *scope =
+      attr.getScope() ? cast<llvm::DILocalScope>(translate(attr.getScope()))
+                      : localVariablePlaceholderScope;
+  assert(scope && "expected local variable placeholder scope");
+  return llvm::DILocalVariable::getTemporary(
+      llvmCtx, scope, /*Name=*/"", /*File=*/nullptr, attr.getLine(),
+      /*Type=*/static_cast<llvm::DIType *>(nullptr), attr.getArg(),
+      static_cast<llvm::DINode::DIFlags>(attr.getFlags()),
+      attr.getAlignInBits(),
+      /*Annotations=*/nullptr);
+}
+
 llvm::DIGlobalVariable *
 DebugTranslation::translateImpl(DIGlobalVariableAttr attr) {
   return llvm::DIGlobalVariable::getDistinct(
@@ -345,6 +360,24 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
       iter != recursiveNodeMap.end()) {
     return iter->second;
   }
+
+  if (auto localVariable = dyn_cast<DILocalVariableAttr>(attr)) {
+    if (localVariable.getIsRecSelf()) {
+      auto iter =
+          forwardLocalVariablePlaceholders
+              .try_emplace(recursiveId, translateTemporaryImpl(localVariable))
+              .first;
+      return iter->second.get();
+    }
+    if (auto iter = forwardLocalVariablePlaceholders.find(recursiveId);
+        iter != forwardLocalVariablePlaceholders.end()) {
+      llvm::DILocalVariable *concrete = translateImpl(localVariable);
+      iter->second->replaceAllUsesWith(concrete);
+      forwardLocalVariablePlaceholders.erase(iter);
+      return concrete;
+    }
+  }
+
   assert(!attr.getIsRecSelf() && "unbound DI recursive self reference");
 
   auto setRecursivePlaceholder = [&](llvm::DINode *placeholder) {
@@ -377,6 +410,13 @@ DebugTranslation::translateRecursive(DIRecursiveTypeAttrInterface attr) {
             auto *concrete = translateImpl(attr);
             temporary->replaceAllUsesWith(concrete);
             return concrete;
+          })
+          .Case([&](DILocalVariableAttr attr) {
+            auto temporary = translateTemporaryImpl(attr);
+            setRecursivePlaceholder(temporary.get());
+            auto *concrete = translateImpl(attr);
+            temporary->replaceAllUsesWith(concrete);
+            return concrete;
           });
 
   assert(recursiveNodeMap.back().first == recursiveId &&
@@ -417,10 +457,15 @@ llvm::DISubprogram *DebugTranslation::translateImpl(DISubprogramAttr attr) {
       /*ThisAdjustment=*/0, llvm::DINode::FlagZero,
       static_cast<llvm::DISubprogram::DISPFlags>(attr.getSubprogramFlags()),
       compileUnit, /*TemplateParams=*/nullptr, /*Declaration=*/nullptr,
-      getRetainedNodesOrNull(attr.getRetainedNodes()), nullptr,
+      /*RetainedNodes=*/nullptr, /*ThrownTypes=*/nullptr,
       getMDTupleOrNull(attr.getAnnotations()));
   if (attr.getId())
     distinctAttrToNode.try_emplace(attr.getId(), node);
+  if (!attr.getRetainedNodes().empty()) {
+    llvm::SaveAndRestore guard(localVariablePlaceholderScope,
+                               cast<llvm::DILocalScope>(node));
+    node->replaceRetainedNodes(getRetainedNodesOrNull(attr.getRetainedNodes()));
+  }
   return node;
 }
 
diff --git a/mlir/lib/Target/LLVMIR/DebugTranslation.h b/mlir/lib/Target/LLVMIR/DebugTranslation.h
index c1fdbd89087da..2db7f401f576d 100644
--- a/mlir/lib/Target/LLVMIR/DebugTranslation.h
+++ b/mlir/lib/Target/LLVMIR/DebugTranslation.h
@@ -105,6 +105,7 @@ class DebugTranslation {
   llvm::TempDICompositeType translateTemporaryImpl(DICompositeTypeAttr attr);
   llvm::TempDISubprogram translateTemporaryImpl(DISubprogramAttr attr);
   llvm::TempDICompileUnit translateTemporaryImpl(DICompileUnitAttr attr);
+  llvm::TempDILocalVariable translateTemporaryImpl(DILocalVariableAttr attr);
 
   /// Constructs a string metadata node from the string attribute. Returns
   /// nullptr if `stringAttr` is null or contains and empty string.
@@ -135,6 +136,15 @@ class DebugTranslation {
   /// A mapping between recursive ID and the translated DINode.
   llvm::MapVector<DistinctAttr, llvm::DINode *> recursiveNodeMap;
 
+  /// A mapping between recursive IDs and local-variable placeholders created
+  /// from forward self references in retainedNodes.
+  DenseMap<DistinctAttr, llvm::TempDILocalVariable>
+      forwardLocalVariablePlaceholders;
+
+  /// Scope to use for forward local-variable placeholders while translating a
+  /// DISubprogram's retainedNodes.
+  llvm::DILocalScope *localVariablePlaceholderScope = nullptr;
+
   /// A mapping between a distinct ID and the translated LLVM metadata node.
   /// This helps identify attrs that should translate into the same LLVM debug
   /// node.
diff --git a/mlir/lib/Target/LLVMIR/ModuleImport.cpp b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
index bb55678f91005..fc4ad29d2d747 100644
--- a/mlir/lib/Target/LLVMIR/ModuleImport.cpp
+++ b/mlir/lib/Target/LLVMIR/ModuleImport.cpp
@@ -35,6 +35,7 @@
 #include "llvm/ADT/TypeSwitch.h"
 #include "llvm/IR/Comdat.h"
 #include "llvm/IR/Constants.h"
+#include "llvm/IR/DebugInfoMetadata.h"
 #include "llvm/IR/DebugProgramInstruction.h"
 #include "llvm/IR/InlineAsm.h"
 #include "llvm/IR/InstIterator.h"
@@ -246,6 +247,38 @@ ModuleImport::ModuleImport(ModuleOp mlirModule,
       emitExpensiveWarnings(emitExpensiveWarnings),
       preferUnregisteredIntrinsics(preferUnregisteredIntrinsics) {
   builder.setInsertionPointToStart(mlirModule.getBody());
+
+  // Variables that are both retained and used by a debug intrinsic need to
+  // break cycles at the variable itself to avoid creating duplicate attrs with
+  // different subprogram scopes.
+  llvm::SmallPtrSet<llvm::DILocalVariable *, 8> retainedLocalVariables;
+  for (llvm::Function &func : *this->llvmModule) {
+    llvm::DISubprogram *subprogram = func.getSubprogram();
+    if (!subprogram)
+      continue;
+    auto addLocalVariable = [&](llvm::DILocalVariable *localVariable) {
+      retainedLocalVariables.insert(localVariable);
+    };
+    auto skip = [](auto *) {};
+    subprogram->forEachRetainedNode(addLocalVariable, skip, skip, skip, skip);
+  }
+
+  auto markIfRetained = [&](llvm::DILocalVariable *localVariable) {
+    if (localVariable && retainedLocalVariables.contains(localVariable))
+      debugImporter->markLocalVariableAsRecursive(localVariable);
+  };
+  for (llvm::Function &func : *this->llvmModule) {
+    for (llvm::Instruction &inst : llvm::instructions(func)) {
+      if (auto *dbgIntrinsic = dyn_cast<llvm::DbgVariableIntrinsic>(&inst))
+        markIfRetained(dbgIntrinsic->getVariable());
+      if (!inst.DebugMarker)
+        continue;
+      for (llvm::DbgRecord &dbgRecord : inst.DebugMarker->getDbgRecordRange())
+        if (auto *dbgVariableRecord =
+                dyn_cast<llvm::DbgVariableRecord>(&dbgRecord))
+          markIfRetained(dbgVariableRecord->getVariable());
+    }
+  }
 }
 
 ComdatOp ModuleImport::getGlobalComdatOp() {
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info.ll b/mlir/test/Target/LLVMIR/Import/debug-info.ll
index 28c25809002cd..38b250f00b958 100644
--- a/mlir/test/Target/LLVMIR/Import/debug-info.ll
+++ b/mlir/test/Target/LLVMIR/Import/debug-info.ll
@@ -1046,3 +1046,36 @@ define void @fn_with_static_local() !dbg !3 {
 !12 = !DILocalVariable(name: "unused_local", scope: !3, file: !4, line: 122, type: !9)
 !13 = !DISubroutineType(types: !14)
 !14 = !{null}
+
+; // -----
+
+; Test that a DILocalVariable listed in DISubprogram's retainedNodes and used by
+; a debug intrinsic does not get duplicated after import.
+
+; CHECK-DAG: #[[LVAR_SELF:.+]] = #llvm.di_local_variable<recId = [[REC_ID:distinct.+<>]], isRecSelf = true>
+; CHECK-DAG: #[[SP:.+]] = #llvm.di_subprogram<{{.*}}name = "fn_with_retained_local"{{.*}}retainedNodes = [#[[LVAR_SELF]]]>
+; CHECK-DAG: #[[LVAR:.+]] = #llvm.di_local_variable<recId = [[REC_ID]], scope = #[[SP]], name = "n"{{.*}}>
+; CHECK-DAG-COUNT-2: {{#llvm\.di_local_variable<}}
+; CHECK: llvm.intr.dbg.value #[[LVAR]] = %{{.*}} : i32
+
+define void @fn_with_retained_local(i32 %n) !dbg !3 {
+entry:
+  call void @llvm.dbg.value(metadata i32 %n, metadata !12, metadata !DIExpression()), !dbg !15
+  ret void
+}
+
+declare void @llvm.dbg.value(metadata, metadata, metadata)
+
+!llvm.dbg.cu = !{!7}
+!llvm.module.flags = !{!10}
+
+!3 = distinct !DISubprogram(name: "fn_with_retained_local", scope: !4, file: !4, line: 1, type: !13, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !7, retainedNodes: !11)
+!4 = !DIFile(filename: "test.c", directory: "")
+!7 = distinct !DICompileUnit(language: DW_LANG_C, file: !4)
+!9 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)
+!10 = !{i32 2, !"Debug Info Version", i32 3}
+!11 = !{!12}
+!12 = !DILocalVariable(name: "n", arg: 1, scope: !3, file: !4, line: 1, type: !9)
+!13 = !DISubroutineType(types: !14)
+!14 = !{null, !9}
+!15 = !DILocation(line: 1, column: 1, scope: !3)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 72dadb2602c52..77eade68ae9e3 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -421,6 +421,46 @@ llvm.func @imp_fn() {
 
 // -----
 
+// Test that a recursive local variable retained by a subprogram is exported
+// without duplicating the DILocalVariable metadata node.
+
+#file = #llvm.di_file<"test.c" in "">
+#cu = #llvm.di_compile_unit<
+  id = distinct[0]<>, sourceLanguage = DW_LANG_C, file = #file,
+  isOptimized = false, emissionKind = Full
+>
+#void = #llvm.di_null_type
+#void_type = #llvm.di_subroutine_type<types = #void>
+#int = #llvm.di_basic_type<
+  tag = DW_TAG_base_type, name = "int",
+  sizeInBits = 32, encoding = DW_ATE_signed
+>
+#local_self = #llvm.di_local_variable<recId = distinct[1]<>, isRecSelf = true>
+#sp = #llvm.di_subprogram<
+  id = distinct[2]<>, compileUnit = #cu, scope = #file,
+  name = "fn_with_retained_local", file = #file,
+  subprogramFlags = Definition, type = #void_type,
+  retainedNodes = [#local_self]
+>
+#local = #llvm.di_local_variable<
+  recId = distinct[1]<>, scope = #sp, name = "n", file = #file,
+  line = 1, arg = 1, type = #int
+>
+
+// CHECK-LABEL: define void @fn_with_retained_local(
+// CHECK-SAME: i32 %[[ARG:.*]]) !dbg ![[SP:[0-9]+]]
+llvm.func @fn_with_retained_local(%arg0: i32) {
+  // RECORDS: #dbg_value(i32 %[[ARG]], ![[LOCAL:[0-9]+]], !DIExpression(), !{{.*}})
+  llvm.intr.dbg.value #local = %arg0 : i32 loc(fused<#sp>["test.c":1:1])
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])
+
+// CHECK-DAG: ![[SP]] = distinct !DISubprogram(name: "fn_with_retained_local"{{.*}}retainedNodes: ![[NODES:[0-9]+]])
+// CHECK-DAG: ![[LOCAL]] = !DILocalVariable(name: "n", arg: 1, scope: ![[SP]], file: !{{.*}}, line: 1, type: !{{.*}})
+// CHECK-DAG: ![[NODES]] = !{![[LOCAL]]}
+
+// -----
+
 // Nameless and scopeless global constant.
 
 // CHECK-LABEL: @.str.1 = external constant [10 x i8]


From llvmlistbot at llvm.org  Thu Jul 16 01:52:17 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 16 Jul 2026 01:52:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a589bc1.481afadf.34be4b.db46@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
----------------
lforg37 wrote:

I didn't know that issue, thanks for bringing it up. IIUC this will not pose problem as we're going to define the specialization in th esame enclosing namespace. 
Subparser will be able to be defined in their own namespaces.

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul 16 02:00:49 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 16 Jul 2026 02:00:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a589dc1.66c0679a.3a40e5.8cf9@mx.google.com>


https://github.com/lforg37 deleted https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul 16 02:33:23 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 16 Jul 2026 02:33:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a58a563.492e23e4.304475.85a4@mx.google.com>


================
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
----------------
lforg37 wrote:

OK I think we will handle that with a macro. 
This is a bit sad that we can't use proper C++ -_-  

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Thu Jul 16 03:15:14 2026
From: llvmlistbot at llvm.org (Andrei Sabalenka)
Date: Thu, 16 Jul 2026 03:15:14 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [libcxx] [lldb] [llvm] [mlir] [NFC]
 Silence -Wbool-integral-comparison warnings across LLVM (PR #195246)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195246@github.com>
Message-ID: <6a58af32.c56fb102.1bc57.71c6@mx.google.com>


https://github.com/mechakotik updated https://github.com/llvm/llvm-project/pull/195246

>From 1bb6eded870f0fc32c3ce0cc5bcd3f7c9420a9a6 Mon Sep 17 00:00:00 2001
From: Andrei Sabalenka <mechakotik at gmail.com>
Date: Fri, 1 May 2026 12:21:37 +0300
Subject: [PATCH] [NFC] Silence -Wbool-integral-comparison warnings across LLVM

---
 bolt/lib/Profile/BoltAddressTranslation.cpp        |  4 ++--
 clang/lib/CodeGen/CodeGenFunction.cpp              |  2 +-
 clang/lib/Serialization/ASTReaderStmt.cpp          |  4 ++--
 flang/unittests/Evaluate/uint128.cpp               | 14 +++++++-------
 .../utilities/const.wrap.class/unary_ops.pass.cpp  |  2 +-
 lldb/source/ValueObject/ValueObject.cpp            |  3 ++-
 llvm/include/llvm/CodeGen/DIE.h                    |  4 ++--
 llvm/include/llvm/Support/DataExtractor.h          |  2 +-
 llvm/lib/ExecutionEngine/JITLink/aarch32.cpp       |  2 +-
 llvm/lib/Support/DataExtractor.cpp                 |  4 ++--
 llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp   |  3 ++-
 .../Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp    |  2 +-
 .../AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp      |  2 +-
 llvm/lib/Target/RISCV/RISCVISelLowering.cpp        |  3 ++-
 llvm/lib/Transforms/IPO/AttributorAttributes.cpp   |  2 +-
 mlir/lib/Rewrite/ByteCode.cpp                      |  2 +-
 16 files changed, 29 insertions(+), 26 deletions(-)

diff --git a/bolt/lib/Profile/BoltAddressTranslation.cpp b/bolt/lib/Profile/BoltAddressTranslation.cpp
index 2068c9efbcb44..cc404e1aef70f 100644
--- a/bolt/lib/Profile/BoltAddressTranslation.cpp
+++ b/bolt/lib/Profile/BoltAddressTranslation.cpp
@@ -185,7 +185,7 @@ template <bool Cold>
 void BoltAddressTranslation::writeMaps(uint64_t &PrevAddress, raw_ostream &OS) {
   const uint32_t NumFuncs =
       llvm::count_if(llvm::make_first_range(Maps), [&](const uint64_t Address) {
-        return Cold == ColdPartSource.count(Address);
+        return Cold == (ColdPartSource.count(Address) != 0);
       });
   encodeULEB128(NumFuncs, OS);
   LLVM_DEBUG(dbgs() << "Writing " << NumFuncs << (Cold ? " cold" : "")
@@ -194,7 +194,7 @@ void BoltAddressTranslation::writeMaps(uint64_t &PrevAddress, raw_ostream &OS) {
   for (auto &MapEntry : Maps) {
     const uint64_t Address = MapEntry.first;
     // Only process cold fragments in cold mode, and vice versa.
-    if (Cold != ColdPartSource.count(Address))
+    if (Cold != (ColdPartSource.count(Address) != 0))
       continue;
     // NB: in `writeMaps` we use the input address because hashes are saved
     // early in `saveMetadata` before output addresses are assigned.
diff --git a/clang/lib/CodeGen/CodeGenFunction.cpp b/clang/lib/CodeGen/CodeGenFunction.cpp
index fc4c7ea40f03d..8d8476956b65d 100644
--- a/clang/lib/CodeGen/CodeGenFunction.cpp
+++ b/clang/lib/CodeGen/CodeGenFunction.cpp
@@ -176,7 +176,7 @@ void CodeGenFunction::CGFPOptionsRAII::ConstructorHelper(FPOptions FPFeatures) {
   auto mergeFnAttrValue = [&](StringRef Name, bool Value) {
     auto OldValue =
         CGF.CurFn->getFnAttribute(Name).getValueAsBool();
-    auto NewValue = OldValue & Value;
+    bool NewValue = OldValue && Value;
     if (OldValue != NewValue)
       CGF.CurFn->addFnAttr(Name, llvm::toStringRef(NewValue));
   };
diff --git a/clang/lib/Serialization/ASTReaderStmt.cpp b/clang/lib/Serialization/ASTReaderStmt.cpp
index 87cec16a76323..4ac60cd28b9ae 100644
--- a/clang/lib/Serialization/ASTReaderStmt.cpp
+++ b/clang/lib/Serialization/ASTReaderStmt.cpp
@@ -154,7 +154,7 @@ void ASTStmtReader::VisitCompoundStmt(CompoundStmt *S) {
   VisitStmt(S);
   SmallVector<Stmt *, 16> Stmts;
   unsigned NumStmts = Record.readInt();
-  unsigned HasFPFeatures = Record.readInt();
+  bool HasFPFeatures = Record.readBool();
   assert(S->hasStoredFPFeatures() == HasFPFeatures);
   while (NumStmts--)
     Stmts.push_back(Record.readSubStmt());
@@ -1159,7 +1159,7 @@ void ASTStmtReader::VisitCastExpr(CastExpr *E) {
 
   CurrentUnpackingBits.emplace(Record.readInt());
   E->setCastKind((CastKind)CurrentUnpackingBits->getNextBits(/*Width=*/7));
-  unsigned HasFPFeatures = CurrentUnpackingBits->getNextBit();
+  bool HasFPFeatures = CurrentUnpackingBits->getNextBit();
   assert(E->hasStoredFPFeatures() == HasFPFeatures);
 
   E->setSubExpr(Record.readSubExpr());
diff --git a/flang/unittests/Evaluate/uint128.cpp b/flang/unittests/Evaluate/uint128.cpp
index 0b749abe1c080..3d0e1b7e6e84c 100644
--- a/flang/unittests/Evaluate/uint128.cpp
+++ b/flang/unittests/Evaluate/uint128.cpp
@@ -66,13 +66,13 @@ static void TestVsNative(__uint128_t x, __uint128_t y) {
   TEST(ToNative(n) == y);
   TEST(ToNative(~m) == ~x);
   TEST(ToNative(-m) == -x);
-  TEST(ToNative(!m) == !x);
-  TEST(ToNative(m < n) == (x < y));
-  TEST(ToNative(m <= n) == (x <= y));
-  TEST(ToNative(m == n) == (x == y));
-  TEST(ToNative(m != n) == (x != y));
-  TEST(ToNative(m >= n) == (x >= y));
-  TEST(ToNative(m > n) == (x > y));
+  TEST(ToNative(!m) == static_cast<__uint128_t>(!x));
+  TEST(ToNative(m < n) == static_cast<__uint128_t>(x < y));
+  TEST(ToNative(m <= n) == static_cast<__uint128_t>(x <= y));
+  TEST(ToNative(m == n) == static_cast<__uint128_t>(x == y));
+  TEST(ToNative(m != n) == static_cast<__uint128_t>(x != y));
+  TEST(ToNative(m >= n) == static_cast<__uint128_t>(x >= y));
+  TEST(ToNative(m > n) == static_cast<__uint128_t>(x > y));
   TEST(ToNative(m & n) == (x & y));
   TEST(ToNative(m | n) == (x | y));
   TEST(ToNative(m ^ n) == (x ^ y));
diff --git a/libcxx/test/std/utilities/const.wrap.class/unary_ops.pass.cpp b/libcxx/test/std/utilities/const.wrap.class/unary_ops.pass.cpp
index 8cd27b75f64c2..4ab30566a07d8 100644
--- a/libcxx/test/std/utilities/const.wrap.class/unary_ops.pass.cpp
+++ b/libcxx/test/std/utilities/const.wrap.class/unary_ops.pass.cpp
@@ -231,7 +231,7 @@ constexpr bool test() {
     assert(result3.get() == ~42);
 
     std::same_as<NonStructural> decltype(auto) result4 = !cwOpsReturnNonStructural;
-    assert(result4.get() == !42);
+    assert(result4.get() == static_cast<int>(!42));
 
     std::same_as<NonStructural> decltype(auto) result5 = &cwOpsReturnNonStructural;
     assert(result5.get() == 84);
diff --git a/lldb/source/ValueObject/ValueObject.cpp b/lldb/source/ValueObject/ValueObject.cpp
index cac4933c64325..78bc3ea131c63 100644
--- a/lldb/source/ValueObject/ValueObject.cpp
+++ b/lldb/source/ValueObject/ValueObject.cpp
@@ -358,8 +358,9 @@ bool ValueObject::IsLogicalTrue(Status &error) {
     LazyBool is_logical_true = language->IsLogicalTrue(*this, error);
     switch (is_logical_true) {
     case eLazyBoolYes:
+      return true;
     case eLazyBoolNo:
-      return (is_logical_true == true);
+      return false;
     case eLazyBoolCalculate:
       break;
     }
diff --git a/llvm/include/llvm/CodeGen/DIE.h b/llvm/include/llvm/CodeGen/DIE.h
index c01ea4cfedbae..76bfb52c1ee9d 100644
--- a/llvm/include/llvm/CodeGen/DIE.h
+++ b/llvm/include/llvm/CodeGen/DIE.h
@@ -547,7 +547,7 @@ struct IntrusiveBackListBase {
 
   void push_back(Node &N) {
     assert(N.Next.getPointer() == &N && "Expected unlinked node");
-    assert(N.Next.getInt() == true && "Expected unlinked node");
+    assert(N.Next.getInt() && "Expected unlinked node");
 
     if (Last) {
       N.Next = Last->Next;
@@ -558,7 +558,7 @@ struct IntrusiveBackListBase {
 
   void push_front(Node &N) {
     assert(N.Next.getPointer() == &N && "Expected unlinked node");
-    assert(N.Next.getInt() == true && "Expected unlinked node");
+    assert(N.Next.getInt() && "Expected unlinked node");
 
     if (Last) {
       N.Next.setPointerAndInt(Last->Next.getPointer(), false);
diff --git a/llvm/include/llvm/Support/DataExtractor.h b/llvm/include/llvm/Support/DataExtractor.h
index ccabd70ed042a..426ada2f64bbd 100644
--- a/llvm/include/llvm/Support/DataExtractor.h
+++ b/llvm/include/llvm/Support/DataExtractor.h
@@ -91,7 +91,7 @@ class DataExtractor {
   /// Get the data pointed to by this extractor.
   StringRef getData() const { return Data; }
   /// Get the endianness for this extractor.
-  bool isLittleEndian() const { return IsLittleEndian; }
+  bool isLittleEndian() const { return IsLittleEndian != 0; }
 
   /// Extract a C string from \a *offset_ptr.
   ///
diff --git a/llvm/lib/ExecutionEngine/JITLink/aarch32.cpp b/llvm/lib/ExecutionEngine/JITLink/aarch32.cpp
index cc606b8278d26..f8342896254f7 100644
--- a/llvm/lib/ExecutionEngine/JITLink/aarch32.cpp
+++ b/llvm/lib/ExecutionEngine/JITLink/aarch32.cpp
@@ -899,7 +899,7 @@ bool StubsManager_v7::visitEdge(LinkGraph &G, Block *B, Edge &E) {
     });
   }
 
-  assert(MakeThumb == (StubSymbol->getTargetFlags() & ThumbSymbol) &&
+  assert(MakeThumb == hasTargetFlags(*StubSymbol, ThumbSymbol) &&
          "Instruction set states of stub and relocation site should be equal");
   LLVM_DEBUG({
     dbgs() << "    Using " << (MakeThumb ? "Thumb" : "Arm") << " entry "
diff --git a/llvm/lib/Support/DataExtractor.cpp b/llvm/lib/Support/DataExtractor.cpp
index 6b58b492002fd..8385888c3cb9c 100644
--- a/llvm/lib/Support/DataExtractor.cpp
+++ b/llvm/lib/Support/DataExtractor.cpp
@@ -48,7 +48,7 @@ T DataExtractor::getU(uint64_t *offset_ptr, Error *Err) const {
   if (!prepareRead(offset, sizeof(T), Err))
     return val;
   std::memcpy(&val, &Data.data()[offset], sizeof(val));
-  if (sys::IsLittleEndianHost != IsLittleEndian)
+  if (sys::IsLittleEndianHost != isLittleEndian())
     sys::swapByteOrder(val);
 
   // Advance the offset
@@ -150,7 +150,7 @@ uint64_t DataExtractor::getUnsigned(uint64_t *offset_ptr, uint32_t byte_size,
                   (sys::IsLittleEndianHost ? 0 : 8 - byte_size),
               &Data.data()[offset], byte_size);
   // Swap the least significant bytes of val if endianness doesn't match.
-  if (sys::IsLittleEndianHost != IsLittleEndian)
+  if (sys::IsLittleEndianHost != isLittleEndian())
     val = sys::getSwappedBytes(val) >> (8 * (8 - byte_size));
 
   *offset_ptr += byte_size;
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
index cad1513f280fb..60ed0d2adc456 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUNextUseAnalysis.cpp
@@ -567,7 +567,8 @@ class llvm::AMDGPUNextUseAnalysisImpl {
       const auto &ToInit = R ? ReachablePaths : UnreachablePaths;
       for (const Path &P : ToInit) {
         PathInfo &Slot = getOrInitPathInfo(P.src(), P.dst());
-        assert(Slot.isForwardReachableUnset() || Slot.ForwardReachable == R);
+        assert(Slot.isForwardReachableUnset() ||
+               Slot.ForwardReachable == static_cast<int>(R));
         Slot.ForwardReachable = R;
       }
     }
diff --git a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
index f019c280997d6..15f06049994aa 100644
--- a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
+++ b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
@@ -6393,7 +6393,7 @@ bool AMDGPUAsmParser::ParseDirectiveAMDHSAKernel() {
         return Error(IDRange.Start, "directive requires gfx8+", IDRange);
       if (!isUInt<1>(Val))
         return OutOfRangeError(ValRange);
-      if (Val != getTargetStreamer().getTargetID()->isXnackOnOrAny())
+      if ((Val != 0) != getTargetStreamer().getTargetID()->isXnackOnOrAny())
         return getParser().Error(IDRange.Start, ".amdhsa_reserve_xnack_mask does not match target id",
                                  IDRange);
     } else if (ID == ".amdhsa_float_round_mode_32") {
diff --git a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp
index 2764ff2d68ce0..092bb0e27131b 100644
--- a/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp
+++ b/llvm/lib/Target/AMDGPU/MCTargetDesc/AMDGPUInstPrinter.cpp
@@ -1311,7 +1311,7 @@ void AMDGPUInstPrinter::printExpTgt(const MCInst *MI, unsigned OpNo,
 
 static bool allOpsDefaultValue(const int* Ops, int NumOps, int Mod,
                                bool IsPacked, bool HasDstSel) {
-  int DefaultValue = IsPacked && (Mod == SISrcMods::OP_SEL_1);
+  bool DefaultValue = IsPacked && (Mod == SISrcMods::OP_SEL_1);
 
   for (int I = 0; I < NumOps; ++I) {
     if (!!(Ops[I] & Mod) != DefaultValue)
diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index 2df58fc2ca9f0..614eff71c67da 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -5458,7 +5458,8 @@ static bool isAlternating(const std::array<std::pair<int, int>, 2> &SrcInfo,
     bool C = Src == SrcInfo[1].first && Diff == SrcInfo[1].second;
     assert(C != (Src == SrcInfo[0].first && Diff == SrcInfo[0].second) &&
            "Must match exactly one of the two slides");
-    if (RequiredPolarity != (C == (Idx / Factor) % 2))
+    bool OddGroup = ((Idx / Factor) % 2) != 0;
+    if (RequiredPolarity != (C == OddGroup))
       return false;
   }
   return true;
diff --git a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
index 66a05a077b208..02ea34a60f269 100644
--- a/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
+++ b/llvm/lib/Transforms/IPO/AttributorAttributes.cpp
@@ -13326,7 +13326,7 @@ struct AANoAliasAddrSpaceImpl : public AANoAliasAddrSpace {
 
   ChangeStatus updateImpl(Attributor &A) override {
     unsigned FlatAS = A.getInfoCache().getFlatAddressSpace().value();
-    uint32_t OldAssumed = getAssumed();
+    bool OldAssumed = getAssumed();
 
     auto CheckAddressSpace = [&](Value &Obj) {
       if (isa<PoisonValue>(&Obj))
diff --git a/mlir/lib/Rewrite/ByteCode.cpp b/mlir/lib/Rewrite/ByteCode.cpp
index 2daf2635d96d5..e6f8950cb11b1 100644
--- a/mlir/lib/Rewrite/ByteCode.cpp
+++ b/mlir/lib/Rewrite/ByteCode.cpp
@@ -1421,7 +1421,7 @@ void ByteCodeExecutor::executeApplyConstraint(PatternRewriter &rewriter) {
 
   LDBG() << "  * Arguments: " << llvm::interleaved(args);
 
-  ByteCodeField isNegated = read();
+  bool isNegated = (read() != 0);
   LDBG() << "  * isNegated: " << isNegated;
 
   ByteCodeField numResults = read();


From llvmlistbot at llvm.org  Thu Jul 16 03:39:53 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Thu, 16 Jul 2026 03:39:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a58b4f9.e5907290.27bf36.ff01@mx.google.com>


https://github.com/flemairen6 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH 1/2] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {

>From 7be327e24300bcd5da269c9bb161fbc1733b0443 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 16 Jul 2026 19:38:41 +0900
Subject: [PATCH 2/2] [mlir][wasm] Avoid failure to build with old gcc by not
 relying on sfinae for instantiating parsers

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 160 ++++++++-------------
 1 file changed, 59 insertions(+), 101 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 872e91179a76c..92bc71e8d55fa 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -425,8 +425,10 @@ class ExpressionParser {
   class TopLevelInstParserRegistry {
   public:
     template <std::byte opCode>
-    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
-                                              ExpressionParser &) = delete;
+    static constexpr bool hasParserForOpcode = false;
+
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &, ExpressionParser &) = delete;
   };
 
 private:
@@ -446,26 +448,20 @@ unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
 template <typename ParserRegistry>
 class InstDispatcher {
 private:
-  /// used to check if a parser is registered in the parser registry for a
-  /// given opcode.
-  /// Default to unreachable handler.
-  template <std::byte opCode, typename = void>
-  struct ParserInfo : std::false_type {
-    static constexpr auto handler = unreachableHandler;
-  };
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
   template <std::byte opCode>
-  struct ParserInfo<
-      opCode,
-      std::void_t<
-          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
-      : std::true_type {
-    static constexpr auto handler =
-        ParserRegistry::template parseInstrWithOpCode<opCode>;
-  };
+  static constexpr dispatch_t getHandlerForOpCode() {
+    if constexpr (ParserRegistry::template hasParserForOpcode<opCode>)
+      return ParserRegistry::template parseInstrWithOpCode<opCode>;
+    else
+      return unreachableHandler;
+  }
 
 public:
   template <std::byte opCode>
-  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+  static constexpr bool isValidInst =
+      ParserRegistry::template hasParserForOpcode<opCode>;
 
 private:
   static inline parsed_inst_t
@@ -476,8 +472,6 @@ class InstDispatcher {
            << static_cast<int>(opCode);
   }
 
-  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
-
   template <std::byte... opCodes>
   static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
                                            ExpressionParser &exprParser,
@@ -485,7 +479,7 @@ class InstDispatcher {
     static constexpr std::array<bool, 256> opcodeValidityMap{
         isValidInst<opCodes>...};
     static constexpr std::array<dispatch_t, 256> dispatchTable{
-        ParserInfo<opCodes>::handler...};
+        getHandlerForOpCode<opCodes>()...};
     if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
       return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
     }
@@ -1059,27 +1053,24 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
-                                       ExpressionParser &exprParser) {
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+  template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<opcode> = true;                                       \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
+      OpBuilder & builder, ExpressionParser & exprParser)
+
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1124,11 +1115,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1155,11 +1142,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {*branchArgs};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1177,11 +1160,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {callOp.getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1194,11 +1173,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
                .getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1235,19 +1210,11 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1319,35 +1286,19 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1371,6 +1322,9 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX> = true;   \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1475,24 +1429,19 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
-                                                ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::SOURCE_OP> = true;        \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1528,6 +1477,11 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S> =   \
+          true;                                                                \
+  template <>                                                                  \
   parsed_inst_t                                                                \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
@@ -1546,16 +1500,16 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
-                                                 ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
@@ -1565,6 +1519,10 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   }                                                                            \
                                                                                \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \


From llvmlistbot at llvm.org  Thu Jul 16 04:15:04 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 04:15:04 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58bd38.d681cab9.120c5c.f8c2@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 04:15:04 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 04:15:04 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58bd38.92ea0e2a.9a5e7.454e@mx.google.com>


https://github.com/skatrak commented:

Thank you Jan, just one comment from me.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 04:15:05 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 04:15:05 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58bd39.d2cb4adc.69be2.0ffd@mx.google.com>


================
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
----------------
skatrak wrote:

I'm not too convinced about this. The problem is that this attribute is optional and has a default, so a valid OpenMP module can skip it. It's also not something that would belong to the `OffloadModuleInterface`, as it's not offloading-specific.

How about setting an `omp.module` unit module attribute or similar in the frontend and just check for presence where needed?

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 04:15:05 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 04:15:05 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58bd39.6c5720b3.5d850.fb10@mx.google.com>


================
@@ -5130,22 +5130,25 @@ std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass() {
   return std::make_unique<ConvertCIRToLLVMPass>();
 }
 
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP) {
   mlir::populateCIRPreLoweringPasses(pm);
-  pm.addPass(mlir::omp::createMarkDeclareTargetPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createMarkDeclareTargetPass());
----------------
skatrak wrote:

Not a change request for this patch, only a related comment: I see there's no function filtering between this and the host op filtering pass. Is the plan to handle that in CIR or in LLVM dialect? The question here is mostly to figure out whether we might want to move that pass to the omp dialect as well.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 04:17:30 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 04:17:30 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58bdca.0c05e68e.b358c.3081@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 04:54:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 04:54:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3d87de8 - [mlir][tosa] Add support for MXFP
	in CONCAT (#209719)
Message-ID: <6a58c65a.510f576d.d8fd6.54a2@mx.google.com>


Author: Philip742
Date: 2026-07-16T12:53:58+01:00
New Revision: 3d87de86a24d1bff2821e4eb2f2d28e58605a394

URL: https://github.com/llvm/llvm-project/commit/3d87de86a24d1bff2821e4eb2f2d28e58605a394
DIFF: https://github.com/llvm/llvm-project/commit/3d87de86a24d1bff2821e4eb2f2d28e58605a394.diff

LOG: [mlir][tosa] Add support for MXFP in CONCAT (#209719)

Adds support to profile compliance for MXFP variants of CONCAT

Signed-off-by: Philip Wilkinson <philip.wilkinson at arm.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index e71907376e0eb..f191da6583584 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -1405,7 +1405,31 @@ extensionComplianceMap = {
        {{{fp8e4m3T, fp8e4m3T}, SpecificationVersion::V_1_0}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T}, SpecificationVersion::V_1_0}}},
-      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}}}},
+      {{Extension::bf16}, {{{bf16T, bf16T}, SpecificationVersion::V_1_0}}},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.pad",
      {{{Extension::int64},
        {{{i64T, i64T, i64T}, SpecificationVersion::V_1_1_DRAFT}}},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 536b6cb470bd7..66521cbf73db8 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -2247,7 +2247,11 @@ def Tosa_ConcatOp : Tosa_InferTensorTypeOp<"concat", [Pure]> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_INT16, Tosa_EXT_INT64]>,
+    Extension<[Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16,
+               Tosa_EXT_INT16, Tosa_EXT_INT64,
+               Tosa_EXT_MX_COMMON, Tosa_EXT_MX_FP4E2M1,
+               Tosa_EXT_MX_FP6E2M3, Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+               Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let hasCanonicalizer = 1;

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 30c5f32d20a29..04277c4ea9bfb 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -521,7 +521,7 @@ func.func @test_reduce_sum(%arg0: tensor<13x21x3xf32>) -> tensor<1x21x3xf32> {
 // CHECK-LABEL: concat
 func.func @test_concat(%arg0: tensor<13x21x3xf32>, %arg1: tensor<13x21x3xf32>) -> tensor<26x21x3xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64] ]
+  // CHECK: extensions: [ [fp8e4m3, fp8e5m2, bf16, int16, int64, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x3xf32>, tensor<13x21x3xf32>) -> tensor<26x21x3xf32>
   return %0 : tensor<26x21x3xf32>
 }

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index 41a17375482b2..b2c9bcfdb2be1 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -2053,3 +2053,11 @@ func.func @test_block_scaled_const_cast_scale_values_no_propagate() -> tensor<2x
   %1 = tosa.cast %0 : (tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<2x32xf32>
   return %1 : tensor<2x32xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index 00a5a23cbe767..b80705348b46b 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -264,3 +264,10 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  // expected-error at +1 {{'tosa.concat' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [mx_common, mx_fp8e4m3] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index d2d6fe3e9126a..e3eb88b10bded 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -583,3 +583,11 @@ func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x
          (tensor<1x32x32x8xf32>, !tosa.shape<2>, !tosa.shape<2>, !tosa.shape<4>) -> tensor<1x32x32x8xf32>
   return %0 : tensor<1x32x32x8xf32>
 }
+
+// -----
+
+// CHECK-LABEL: test_block_scaled_concat
+func.func @test_block_scaled_concat(%arg0: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, %arg1: tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>> {
+  %0 = tosa.concat %arg0, %arg1 {axis = 0 : i32} : (tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>, tensor<13x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+  return %0 : tensor<26x21x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>
+}


        

From llvmlistbot at llvm.org  Thu Jul 16 04:54:05 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 16 Jul 2026 04:54:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP in CONCAT
	(PR #209719)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209719@github.com>
Message-ID: <6a58c65d.0c05e68e.b358c.105e@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/209719

From llvmlistbot at llvm.org  Thu Jul 16 06:14:14 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Thu, 16 Jul 2026 06:14:14 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58d926.f3151ff8.b27a.3ccb@mx.google.com>


================
@@ -5130,22 +5130,25 @@ std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass() {
   return std::make_unique<ConvertCIRToLLVMPass>();
 }
 
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP) {
   mlir::populateCIRPreLoweringPasses(pm);
-  pm.addPass(mlir::omp::createMarkDeclareTargetPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createMarkDeclareTargetPass());
----------------
jsjodin wrote:

I think we should move this to the omp dialect as well.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 06:17:36 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Thu, 16 Jul 2026 06:17:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP conv2d (PR
	#210054)
Message-ID: <llvm.org/llvm/llvm-project/pull/210054@github.com>


https://github.com/iliyan-georgiev-arm created https://github.com/llvm/llvm-project/pull/210054

- Adds profile compliance support for MXFP conv2d
- Relax conv2d accumulator constraits. They are checked as part of validation pass
and this would allow experimentation with different types.

Signed-off-by: Iliyan Georgiev <iliyan.georgiev at arm.com>

>From 58cfca71a90233ffe2162d98b3697fa9912cdd6d Mon Sep 17 00:00:00 2001
From: Iliyan Georgiev <iliyan.georgiev at arm.com>
Date: Fri, 10 Jul 2026 10:20:51 +0100
Subject: [PATCH] [mlir][tosa] Add support for MXFP conv2d

- Adds profile compliance support for MXFP conv2d
- Relax conv2d accumulator constraits.
They are checked as part of validation pass
and this would allow experimentation with different types.

Signed-off-by: Iliyan Georgiev <iliyan.georgiev at arm.com>
Change-Id: I6bc866d50fb7303b200fdc3d366492971b5edf95
---
 .../Dialect/Tosa/IR/TosaComplianceData.h.inc  | 977 +++++++++++++++++-
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td  |  16 +-
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          |  60 +-
 mlir/test/Dialect/Tosa/availability.mlir      |   2 +-
 mlir/test/Dialect/Tosa/invalid.mlir           | 108 --
 mlir/test/Dialect/Tosa/invalid_extension.mlir |   9 +
 mlir/test/Dialect/Tosa/ops.mlir               |   9 +
 .../tosa-validation-version-1p0-invalid.mlir  |  10 +
 .../tosa-validation-version-1p1-valid.mlir    |  20 +-
 mlir/test/Dialect/Tosa/verifier.mlir          |  20 +
 10 files changed, 1075 insertions(+), 156 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..fa1ff76b1ed04 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -606,16 +606,991 @@ extensionComplianceMap = {
       {{Extension::fp8e4m3},
        {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp32T, fp16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp32T, fp16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16},
        {{{bf16T, bf16T, bf16T, bf16T, bf16T, fp32T, bf16T},
-         SpecificationVersion::V_1_0}}}}},
+         SpecificationVersion::V_1_0},
+        {{fp16T, fp16T, fp16T, fp16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp32T, fp16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp32T, bf16T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp32T, bf16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16, Extension::fp8e4m3},
+       {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp32T, fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp32T, bf16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp32T, fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp32T, fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2},
+       {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp32T, fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp32T, bf16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.conv2d_block_scaled",
      {{{Extension::mxfp_conv},
        {{{fp4e2m1T, fp8ue8m0T, fp4e2m1T, fp8ue8m0T, fp32T, fp32T},
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 3518a62800963..665faf41b3652 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -68,7 +68,7 @@ def Tosa_ArgMaxOp : Tosa_InferShapedTypeOp<"argmax", [NoMemoryEffect]> {
 // Accumulator types.
 //===----------------------------------------------------------------------===//
 
-def Tosa_AccType : AnyTypeOf<[I<32>, I<48>, F16, F32]>;
+def Tosa_AccType : AnyTypeOf<[I<32>, I<48>, F16, F32, BF16]>;
 
 //===----------------------------------------------------------------------===//
 // Operator: avg_pool2d
@@ -203,7 +203,11 @@ def Tosa_Conv2DOp : Tosa_ConvOp<"conv2d"> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
+    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3,
+               Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MX_COMMON,
+               Tosa_EXT_MX_FP4E2M1, Tosa_EXT_MX_FP6E2M3,
+               Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+               Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -221,6 +225,8 @@ def Tosa_Conv2DOp : Tosa_ConvOp<"conv2d"> {
 
 //===----------------------------------------------------------------------===//
 // Operator: conv2d_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_Conv2DBlockScaledOp : Tosa_InferShapedTypeOp<"conv2d_block_scaled", [NoMemoryEffect]> {
   let summary = "Performs two dimensional convolution using block scaled tensors.";
@@ -232,6 +238,8 @@ def Tosa_Conv2DBlockScaledOp : Tosa_InferShapedTypeOp<"conv2d_block_scaled", [No
 
     This operation is not pure. Undefined behaviour may occur if the accumulated
     result overflows.
+
+    // Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
@@ -463,7 +471,7 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
   let description = [{
     Performs two dimensional matrix multiplications. `A` matrix is of shape
     `N x H x C`. `B` matrix is of shape `D x W x C`. This is effectively a
-    matrix multiply of `A` by the transposed `B` matrix. If the batched 
+    matrix multiply of `A` by the transposed `B` matrix. If the batched
     dimension of input `B` is of size 1, the `B` matrix is broadcast.
   }];
 
@@ -2578,7 +2586,7 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
   let description = [{
     Generate a tensor based on the input indices and row_count. The number of
-    consecutive rows gathered for each index is specified in row_count. N is 
+    consecutive rows gathered for each index is specified in row_count. N is
     the number of batches, W is the number of indices in each batch, K is
     the range of each index, and C is the number of data channels for each
     index. The values tensor has shape [N, K, C] and the output tensor has shape
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index e25774856f59d..2b2fbf4b33f97 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -810,21 +810,16 @@ static LogicalResult verifyConvOp(T op) {
     return failure();
   }
 
-  if (isa<Float8E5M2Type>(inputEType) || isa<Float8E4M3FNType>(inputEType) ||
-      isa<Float8E5M2Type>(weightEType) || isa<Float8E4M3FNType>(weightEType)) {
-    if (inputEType != weightEType) {
-      op.emitOpError(
-          "expect both input and weight to have same element type, got ")
-          << inputEType << " and " << weightEType;
-      return failure();
-    }
-  }
+  const bool isInputBlockScaled = llvm::isa<BlockScaledType>(inputEType);
+  const bool isWeightBlockScaled = llvm::isa<BlockScaledType>(weightEType);
+  const bool isInputFloat = llvm::isa<FloatType>(inputEType);
+  const bool isWeightFloat = llvm::isa<FloatType>(weightEType);
 
-  bool inputIsFloat = llvm::isa<FloatType>(inputEType);
-  bool weightIsFloat = llvm::isa<FloatType>(weightEType);
+  const bool isInputBSorFloat = isInputBlockScaled || isInputFloat;
+  const bool isWeightBSorFloat = isWeightBlockScaled || isWeightFloat;
 
   // Either both must be float or both non-float.
-  if (inputIsFloat != weightIsFloat) {
+  if (isInputBSorFloat != isWeightBSorFloat) {
     op.emitOpError(
         "expect both input and weight to be float or not together, got ")
         << inputEType << " and " << weightEType;
@@ -832,18 +827,28 @@ static LogicalResult verifyConvOp(T op) {
   }
 
   auto inputZpEType = getStorageElementTypeOrSelf(op.getInputZp().getType());
-  if (inputEType != inputZpEType) {
+  if (!isInputBlockScaled && inputEType != inputZpEType) {
     return op.emitOpError("expect both input and its zero point are the same "
                           "element type, got ")
            << inputEType << " and " << inputZpEType;
   }
+  if (isInputBlockScaled && !llvm::isa<Float32Type>(inputZpEType)) {
+    return op.emitOpError(
+               "expect block scaled input to have fp32 zero point, got ")
+           << inputEType << " and " << inputZpEType;
+  }
 
   auto weightZpEType = getStorageElementTypeOrSelf(op.getWeightZp().getType());
-  if (weightEType != weightZpEType) {
+  if (!isWeightBlockScaled && weightEType != weightZpEType) {
     return op.emitOpError("expect both weight and its zero point are the same "
                           "element type, got ")
            << weightEType << " and " << weightZpEType;
   }
+  if (isWeightBlockScaled && !llvm::isa<Float32Type>(weightZpEType)) {
+    return op.emitOpError(
+               "expect block scaled weight to have fp32 zero point, got ")
+           << weightEType << " and " << weightZpEType;
+  }
 
   FailureOr<int64_t> maybeIZp = op.getInputZeroPoint();
   if (succeeded(maybeIZp) && op.verifyInputZeroPoint(*maybeIZp).failed())
@@ -888,33 +893,6 @@ static LogicalResult verifyConvOpModes(T op) {
   if (auto quantType = llvm::dyn_cast<mlir::quant::QuantizedType>(inputEType))
     inputEType = getStorageElementTypeFromQuantized(quantType);
 
-  auto accType = op.getAccType();
-  if (inputEType.isInteger(8) && !accType.isInteger(32))
-    return op.emitOpError("accumulator type for i8 tensor is not i32, got ")
-           << accType;
-
-  if (inputEType.isInteger(16) && !accType.isInteger(48))
-    return op.emitOpError("accumulator type for i16 tensor is not i48, got ")
-           << accType;
-
-  if (isa<Float8E5M2Type, Float8E4M3Type>(inputEType) &&
-      !(accType.isF16() || accType.isF32()))
-    return op.emitOpError("accumulator type for f8 tensor is not f16/f32, got ")
-           << accType;
-
-  if (inputEType.isF16() && !(accType.isF16() || accType.isF32()))
-    return op.emitOpError(
-               "accumulator type for f16 tensor is not f16/f32, got ")
-           << accType;
-
-  if (inputEType.isBF16() && !accType.isF32())
-    return op.emitOpError("accumulator type for bf16 tensor is not f32, got ")
-           << accType;
-
-  if (inputEType.isF32() && !accType.isF32())
-    return op.emitOpError("accumulator type for f32 tensor is not f32, got ")
-           << accType;
-
   auto resultEType =
       llvm::cast<ShapedType>(op.getResult().getType()).getElementType();
 
diff --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index c358460bbc662..39dd4080845e1 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -43,7 +43,7 @@ func.func @test_avg_pool2d_adaptive(%arg0: tensor<1x7x7x9xf32>) -> tensor<1x7x7x
 // CHECK-LABEL: conv2d
 func.func @test_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<8x1x1x4xf32>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
+  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x4xf32>, tensor<8x1x1x4xf32>, tensor<8xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf32>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..e559ea5f3c0ab 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -74,105 +74,6 @@ func.func @test_conv2d_weight_zp(%arg0: tensor<1x29x29x4xf16>, %arg1: tensor<16x
 
 // -----
 
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x27x27x16xi8>
-  return %0 : tensor<1x27x27x16xi8>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xi16>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi16>) -> tensor<1x27x27x16xi16> {
-  %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  %weight_zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i16 tensor is not i48, got 'f16'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xi16>, tensor<16x3x3x4xi8>, tensor<16xi16>, tensor<1xi16>, tensor<1xi8>) -> tensor<1x27x27x16xi16>
-  return %0 : tensor<1x27x27x16xi16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf8E5M2>, %arg1: tensor<16x3x3x4xf8E5M2>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf8E5M2>} : () -> tensor<1xf8E5M2>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f8 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf8E5M2>, tensor<16x3x3x4xf8E5M2>, tensor<16xf16>, tensor<1xf8E5M2>, tensor<1xf8E5M2>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf8E4M3>, %arg1: tensor<16x3x3x4xf8E4M3>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf8E4M3>} : () -> tensor<1xf8E4M3>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f8 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf8E4M3>, tensor<16x3x3x4xf8E4M3>, tensor<16xf16>, tensor<1xf8E4M3>, tensor<1xf8E4M3>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf16>, %arg1: tensor<16x3x3x4xf16>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf16>} : () -> tensor<1xf16>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f16 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf16>, tensor<16x3x3x4xf16>, tensor<16xf16>, tensor<1xf16>, tensor<1xf16>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xbf16>, %arg1: tensor<16x3x3x4xbf16>, %arg2: tensor<16xbf16>) -> tensor<1x27x27x16xbf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xbf16>} : () -> tensor<1xbf16>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for bf16 tensor is not f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xbf16>, tensor<16x3x3x4xbf16>, tensor<16xbf16>, tensor<1xbf16>, tensor<1xbf16>) -> tensor<1x27x27x16xbf16>
-  return %0 : tensor<1x27x27x16xbf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf32>} : () -> tensor<1xf32>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f32 tensor is not f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x27x27x16xf32>
-  return %0 : tensor<1x27x27x16xf32>
-}
-
-// -----
-
-func.func @test_conv3d_acc_type(%arg0: tensor<1x4x8x21x17xi8>, %arg1: tensor<34x1x1x1x17xi8>, %arg2: tensor<34xi8>) -> tensor<1x4x8x21x34xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv3d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.conv3d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1, 1>, pad = array<i64: 0, 0, 0, 0, 0, 0>, stride = array<i64: 1, 1, 1>}
-           : (tensor<1x4x8x21x17xi8>, tensor<34x1x1x1x17xi8>, tensor<34xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x8x21x34xi8>
-  return %0 : tensor<1x4x8x21x34xi8>
-}
-
-// -----
-
-func.func @test_depthwise_conv2d_acc_type(%arg0: tensor<1x4x4x4xi8>, %arg1: tensor<1x1x4x2xi8>, %arg2: tensor<8xi8>) -> tensor<1x4x4x8xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.depthwise_conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.depthwise_conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x4x4x4xi8>, tensor<1x1x4x2xi8>, tensor<8xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x4x8xi8>
-  return %0 : tensor<1x4x4x8xi8>
-}
-
-// -----
-
-func.func @test_transpose_conv2d(%arg0: tensor<1x32x32x8xi8>, %arg1: tensor<16x1x1x8xi8>, %arg2: tensor<16xi8>) -> tensor<1x32x32x16xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.transpose_conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.transpose_conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, out_pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xi8>, tensor<16x1x1x8xi8>, tensor<16xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x32x32x16xi8>
-  return %0 : tensor<1x32x32x16xi8>
-}
-
-// -----
-
 func.func @test_transpose_conv2d_invalid_padding_top(%arg0: tensor<1x32x32x8xf32>, %arg1: tensor<16x1x1x8xf32>, %arg2: tensor<16xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x32x32x16xf32> {
   // expected-error at +1 {{'tosa.transpose_conv2d' op expected out_pad_top > -KH, but got: out_pad_top=-3 and KH=1}}
   %0 = tosa.transpose_conv2d %arg0, %arg1, %arg2, %arg3, %arg4 {acc_type = f32, out_pad = array<i64: -3, 0, 0, 0>, out_shape = array<i64: 1, 32, 32, 16>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xf32>, tensor<16x1x1x8xf32>, tensor<16xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x32x32x16xf32>
@@ -243,15 +144,6 @@ func.func @test_transpose_conv2d_invalid_bias(%arg0: tensor<1x32x32x8xf32>, %arg
   return %0 : tensor<1x32x32x16xf32>
 }
 
-// -----
-// CHECK-LABEL: conv2d_quant_any_acc
-func.func @test_conv2d_quant_any_acc(%arg0: tensor<1x4x4x4x!quant.any<i8<-8:7>>>, %arg1: tensor<8x1x1x4x!quant.any<i8<-8:7>>>, %arg2: tensor<8x!quant.any<i8<-8:7>>>) -> tensor<1x4x4x8x!quant.any<i8<-8:7>>> {
-  %zp = "tosa.const" () { values = dense<0> : tensor<1xi8> } : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i8 tensor is not i32, got 'f32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x4x!quant.any<i8<-8:7>>>, tensor<8x1x1x4x!quant.any<i8<-8:7>>>, tensor<8x!quant.any<i8<-8:7>>>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x4x8x!quant.any<i8<-8:7>>>
-  return %0 : tensor<1x4x4x8x!quant.any<i8<-8:7>>>
-}
-
 // -----
 // CHECK-LABEL: conv2d_quant_any
 func.func @test_conv2d_quant_any(%arg0: tensor<1x4x4x4x!quant.any<i8<-8:7>>>, %arg1: tensor<8x1x1x4x!quant.any<i8<-8:7>>>, %arg2: tensor<8x!quant.any<i32<-8:7>>>) -> tensor<1x4x4x8x!quant.any<i32<-8:7>>> {
diff --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 5e4dd17b24090..98357910adeaf 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -25,6 +25,15 @@ func.func @test_conv2d(%arg0: tensor<1x4x4x4xi8>, %arg1: tensor<8x1x1x4xi4>, %ar
   return %0 : tensor<1x4x4x8xi32>
 }
 
+// -----
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
 // -----
 func.func @test_conv3d(%arg0: tensor<1x4x8x21x17xi16>, %arg1: tensor<34x1x1x1x17xi8>, %arg2: tensor<34xi48>, %arg3: tensor<1xi16>, %arg4: tensor<1xi8>) -> tensor<1x4x8x21x34xi48> {
   // expected-error at +1 {{'tosa.conv3d' op illegal: requires any of [int16] profiles/extensions to be specified in the target environment}}
diff --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index ec3d0c881d857..3a1bd17b85c73 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -142,6 +142,15 @@ func.func @test_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<8x1x1x4xf32>, %
   return %0 : tensor<1x4x4x8xf32>
 }
 
+// -----
+// CHECK-LABEL: conv2d_mxfp
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
 // -----
 // CHECK-LABEL: conv2d_unranked_input
 func.func @test_conv2d_unranked_input(%arg0: tensor<*xf32>, %arg1: tensor<8x1x1x4xf32>, %arg2: tensor<8xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x4x4x8xf32> {
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index a5e4d4397f290..c7f061f2e80f0 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -42,6 +42,16 @@ func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x
 
 // -----
 
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 func.func @test_conv3d_fp8_acc32(%arg0: tensor<1x4x8x21x17xf8E5M2>, %arg1: tensor<34x1x1x1x17xf8E5M2>, %arg2: tensor<34xf32>) -> tensor<1x4x8x21x34xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
   %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
diff --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e7a4fe6735457..cdf9ae1be63fe 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -42,6 +42,24 @@ func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x
 
 // -----
 
+// CHECK-LABEL: test_conv2d_mxfp
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// CHECK-LABEL: test_conv2d_mxfp_acc32
+func.func @test_conv2d_mxfp_acc32(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 // CHECK-LABEL: test_conv3d_fp8_acc32
 func.func @test_conv3d_fp8_acc32(%arg0: tensor<1x4x8x21x17xf8E5M2>, %arg1: tensor<34x1x1x1x17xf8E5M2>, %arg2: tensor<34xf32>) -> tensor<1x4x8x21x34xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
@@ -497,7 +515,7 @@ func.func @test_assert_equal_shape() {
 }
 
 // -----
-func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> { 
+func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {
   %kernel = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
   %stride = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
   %pad = tosa.const_shape {values = dense<[0, 0, 0, 0]> : tensor<4xindex>} : () -> !tosa.shape<4>
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 20c0dfd2a094c..c7d889ffb58f2 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -244,6 +244,26 @@ func.func @test_slice_output_shape_mismatch_dynamic(%arg0: tensor<?x5x6xf32>) {
 
 // -----
 
+func.func @test_conv2d_mxfp_invalid_weight_zp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  // expected-error at +1 {{'tosa.conv2d' op expect block scaled weight to have fp32 zero point}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf16>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
+func.func @test_conv2d_mxfp_invalid_input_zp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op expect block scaled input to have fp32 zero point}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf16>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 func.func @test_depthwise_conv2d_invalid_padding(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<1x1x8x4xf32>, %arg2: tensor<8xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x4x4x8xf32> {
   // expected-error at +1 {{'tosa.depthwise_conv2d' op expect all padding values to be >= 0, got 0, 0, -1, 0}}
   %0 = tosa.depthwise_conv2d %arg0, %arg1, %arg2, %arg3, %arg4 {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, -1, 0>, stride = array<i64: 1, 1>, local_bound = true}


From llvmlistbot at llvm.org  Thu Jul 16 06:18:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 06:18:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP conv2d (PR
	#210054)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210054@github.com>
Message-ID: <6a58da16.b4a331f0.261d49.b642@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Iliyan Georgiev (iliyan-georgiev-arm)

<details>
<summary>Changes</summary>

- Adds profile compliance support for MXFP conv2d
- Relax conv2d accumulator constraits. They are checked as part of validation pass
and this would allow experimentation with different types.

Signed-off-by: Iliyan Georgiev &lt;iliyan.georgiev@<!-- -->arm.com&gt;

---

Patch is 77.60 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210054.diff


10 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc (+976-1) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td (+12-4) 
- (modified) mlir/lib/Dialect/Tosa/IR/TosaOps.cpp (+19-41) 
- (modified) mlir/test/Dialect/Tosa/availability.mlir (+1-1) 
- (modified) mlir/test/Dialect/Tosa/invalid.mlir (-108) 
- (modified) mlir/test/Dialect/Tosa/invalid_extension.mlir (+9) 
- (modified) mlir/test/Dialect/Tosa/ops.mlir (+9) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir (+10) 
- (modified) mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir (+19-1) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+20) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index ef644845ed937..fa1ff76b1ed04 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -606,16 +606,991 @@ extensionComplianceMap = {
       {{Extension::fp8e4m3},
        {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp32T, fp16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp32T, fp16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16},
        {{{bf16T, bf16T, bf16T, bf16T, bf16T, fp32T, bf16T},
-         SpecificationVersion::V_1_0}}}}},
+         SpecificationVersion::V_1_0},
+        {{fp16T, fp16T, fp16T, fp16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp32T, fp16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp32T, bf16T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp32T, bf16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16, Extension::fp8e4m3},
+       {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp32T, fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp32T, bf16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp32T, fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp32T, fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2},
+       {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp32T, fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp32T, bf16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210054

From llvmlistbot at llvm.org  Thu Jul 16 06:23:33 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Thu, 16 Jul 2026 06:23:33 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58db55.c5504292.a1395.5d0c@mx.google.com>


================
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
----------------
jsjodin wrote:

Is there a reason to have this optional? I guess it would affect tests, but adding an extra attribute would do the same I thnk. This is an indirect way of checking if OpenMP is used so it looks a bit odd, although having two attributes is also not great since they can be inconsistent with each other if something goes wrong.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 06:23:58 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Thu, 16 Jul 2026 06:23:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a58db6e.492e23e4.304475.d65e@mx.google.com>


https://github.com/flemairen6 updated https://github.com/llvm/llvm-project/pull/205990

>From f294988d80afcf50d52102adf91aa52dd2e9199e Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 12:48:24 +0900
Subject: [PATCH 1/7] [mlir][wasm] RaiseWasmMLIRPass: Add suport for comparison
 operators

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 139 ++++++
 .../wasm-comparisons-to-arith-cmp.mlir        | 421 ++++++++++++++++++
 .../RaiseWasm/wasm-eqz-to-arith-cmp.mlir      |  28 ++
 3 files changed, 588 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 83bfde7032ef8..1fb4564032421 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,104 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+template <typename SourceOp, typename TargetOp, typename AttrType,
+          typename ValType, ValType flag>
+struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto cmpRes =
+        rewriter
+            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                              AttrType::get(rewriter.getContext(), flag),
+                              adaptor.getLhs(), adaptor.getRhs())
+            .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
+template <typename SourceOp, arith::CmpFPredicate compFlag>
+using FPComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpFOp, arith::CmpFPredicateAttr,
+                           arith::CmpFPredicate, compFlag>;
+
+template <typename SourceOp, arith::CmpIPredicate compFlag>
+using IntComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpIOp, arith::CmpIPredicateAttr,
+                           arith::CmpIPredicate, compFlag>;
+
+using WasmLtSIOpConversion =
+    IntComparisonConversion<LtSIOp, arith::CmpIPredicate::slt>;
+using WasmLeSIOpConversion =
+    IntComparisonConversion<LeSIOp, arith::CmpIPredicate::sle>;
+using WasmGtSIOpConversion =
+    IntComparisonConversion<GtSIOp, arith::CmpIPredicate::sgt>;
+using WasmGeSIOpConversion =
+    IntComparisonConversion<GeSIOp, arith::CmpIPredicate::sge>;
+using WasmLtUIOpConversion =
+    IntComparisonConversion<LtUIOp, arith::CmpIPredicate::ult>;
+using WasmLeUIOpConversion =
+    IntComparisonConversion<LeUIOp, arith::CmpIPredicate::ule>;
+using WasmGtUIOpConversion =
+    IntComparisonConversion<GtUIOp, arith::CmpIPredicate::ugt>;
+using WasmGeUIOpConversion =
+    IntComparisonConversion<GeUIOp, arith::CmpIPredicate::uge>;
+using WasmLtOpConversion =
+    FPComparisonConversion<LtOp, arith::CmpFPredicate::OLT>;
+using WasmLeOpConversion =
+    FPComparisonConversion<LeOp, arith::CmpFPredicate::OLE>;
+using WasmGtOpConversion =
+    FPComparisonConversion<GtOp, arith::CmpFPredicate::OGT>;
+using WasmGeOpConversion =
+    FPComparisonConversion<GeOp, arith::CmpFPredicate::OGE>;
+
+template <typename SourceOp, arith::CmpIPredicate IntFlag,
+          arith::CmpFPredicate FloatFlag>
+struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Value comparisonResult;
+    if (srcOp.getLhs().getType().isInteger())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpIOp>(
+                  srcOp.getLoc(), rewriter.getI1Type(),
+                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                  adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else if (srcOp.getLhs().getType().isFloat())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                                     arith::CmpFPredicateAttr::get(
+                                         rewriter.getContext(), FloatFlag),
+                                     adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else
+      return rewriter.notifyMatchFailure(
+          srcOp.getLoc(), "Unsupported datatype for comparison OP.");
+
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                comparisonResult);
+    return success();
+  }
+};
+
+using WasmEqOpConversion =
+    IntFpComparisonOpConversion<EqOp, arith::CmpIPredicate::eq,
+                                arith::CmpFPredicate::OEQ>;
+using WasmNeOpConversion =
+    IntFpComparisonOpConversion<NeOp, arith::CmpIPredicate::ne,
+                                arith::CmpFPredicate::ONE>;
+
 struct WasmCallOpConversion : OpConversionPattern<FuncCallOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -148,6 +246,32 @@ struct WasmConstOpConversion : OpConversionPattern<ConstOp> {
   }
 };
 
+struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = eqzOp->getLoc();
+    auto zero =
+        rewriter
+            .create<arith::ConstantOp>(
+                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+            .getResult();
+    auto cmpRes = rewriter
+                      .create<arith::CmpIOp>(
+                          loc, rewriter.getI1Type(),
+                          arith::CmpIPredicateAttr::get(
+                              rewriter.getContext(), arith::CmpIPredicate::eq),
+                          adaptor.getInput(), zero)
+                      .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -429,21 +553,36 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivFPOpConversion,
            WasmDivSIOpConversion,
            WasmDivUIOpConversion,
+           WasmEqOpConversion,
+           WasmEqzOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
            WasmFuncImportOpConversion,
            WasmFuncOpConversion,
+           WasmGeOpConversion,
+           WasmGeSIOpConversion,
+           WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
+           WasmGtOpConversion,
+           WasmGtSIOpConversion,
+           WasmGtUIOpConversion,
+           WasmLeOpConversion,
+           WasmLeSIOpConversion,
+           WasmLeUIOpConversion,
            WasmLocalConversion,
            WasmLocalGetConversion,
            WasmLocalSetConversion,
            WasmLocalTeeConversion,
+           WasmLtOpConversion,
+           WasmLtSIOpConversion,
+           WasmLtUIOpConversion,
            WasmMaxOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
+           WasmNeOpConversion,
            WasmNegOpConversion,
            WasmOrOpConversion,
            WasmPopCntOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..bc952b47520ca
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -0,0 +1,421 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+
+module {
+  wasmssa.func nested @func_lt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+
+  wasmssa.func nested @func_le_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.lt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.le %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.gt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ge %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.lt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.le %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.gt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.ge %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.eq %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.eq %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.eq %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.eq %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+    wasmssa.func nested @func_ne_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ne %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.ne %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ne %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.ne %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_lt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..083a24c5f36a8
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
@@ -0,0 +1,28 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+module {
+  wasmssa.func @eqz_i32() -> i32 {
+    %0 = wasmssa.const 13 : i32
+    %1 = wasmssa.eqz %0 : i32 -> i32
+    wasmssa.return %1 : i32
+  }
+  wasmssa.func @eqz_i64() -> i32 {
+    %0 = wasmssa.const 13 : i64
+    %1 = wasmssa.eqz %0 : i64 -> i32
+    wasmssa.return %1 : i32
+  }
+}
+
+// CHECK-LABEL:   func.func @eqz_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @eqz_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32

>From 49944e5c6ec9e356338dfaabbd248fb969ce49b8 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:06:50 +0900
Subject: [PATCH 2/7] [mlir][wasm] RaiseWasmMLIRPass: Add rotl and extend
 support

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 87 +++++++++++++++++++
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   | 76 ++++++++++++++++
 .../RaiseWasm/wasm-rotl-to-arith.mlir         | 75 ++++++++++++++++
 .../RaiseWasm/wasm-rotr-to-arith.mlir         | 77 ++++++++++++++++
 4 files changed, 315 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 1fb4564032421..bf958aad5012b 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,72 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+/// Lower a rotate to a series of bitwise operations. Intended for us
+/// in dialects that do not natively support rotate operations.
+///
+/// Result stays in the wasm dialect. It will then subsequently be lowered to
+/// the target dialect.
+///
+/// The rotate will be lowered to a pattern like so:
+///
+/// (val LHSShiftOp (bits & (width-1))) | (val RHSShiftOp (-bits & (width-1)))
+///
+/// Where LHSShiftOp and RHSShiftOp are shift operations. Concretely,
+///
+/// rotr = (val >> (bits & (width - 1))) | (val << (-bits & (width - 1)))
+/// rotl = (val << (bits & (width - 1))) | (val >> (-bits & (width - 1)))
+///
+/// Using this variant ensures that our rotate is defined in the target dialect.
+///
+/// \p SourceOp - Rotate operation to replace.
+/// \p LHSShiftOp - Shift operation to use on the left-hand side of the OR.
+/// \p RHSShiftOp - Shift operation to use on the right-hand side of the OR.
+template <typename SourceOp, typename LHSShiftOp, typename RHSShiftOp>
+struct RotateOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    const Type ty = srcOp->getResultTypes()[0];
+    const Location loc = srcOp->getLoc();
+    const Value val = adaptor.getVal();
+    const Value bits = adaptor.getBits();
+    const unsigned width = ty.getIntOrFloatBitWidth();
+
+    // Materialize (width - 1) for use in both sides of the expression.
+    auto cstWidthMinusOne =
+        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+
+    // Form the left-hand side of the OR:
+    // (val (lhs shift op) (bits & (width - 1)))
+    auto orLHS = rewriter.create<LHSShiftOp>(
+        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+
+    // Form the right-hand side of the OR:
+    // (val (rhs shift op) (-bits & (width - 1)))
+    auto orRHS = rewriter.create<RHSShiftOp>(
+        loc, val,
+        // (-bits & (width - 1))
+        rewriter.create<AndOp>(
+            loc,
+            // 0 - bits == -bits
+            rewriter.create<SubOp>(
+                loc,
+                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
+                bits),
+            cstWidthMinusOne));
+
+    // OR together the two shifts and replace the rotate with the new
+    // expression.
+    rewriter.replaceOpWithNewOp<OrOp>(srcOp, orLHS, orRHS);
+    return success();
+  }
+};
+
+using WasmRotrOpConversion = RotateOpConversion<RotrOp, ShRUOp, ShLOp>;
+using WasmRotlOpConversion = RotateOpConversion<RotlOp, ShLOp, ShRUOp>;
+
 template <typename SourceOp, typename TargetOp, typename AttrType,
           typename ValType, ValType flag>
 struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
@@ -272,6 +338,24 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   }
 };
 
+struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(ExtendLowBitsSOp extendLowBytesSOp,
+                  ExtendLowBitsSOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
+    auto truncation = rewriter.create<arith::TruncIOp>(
+        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
+        adaptor.getInput());
+    rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
+        extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
+        truncation.getResult());
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -555,6 +639,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivUIOpConversion,
            WasmEqOpConversion,
            WasmEqzOpConversion,
+           WasmExtendLowBitsOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
@@ -591,6 +676,8 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmRemSIOpConversion,
            WasmRemUIOpConversion,
            WasmReturnOpConversion,
+           WasmRotlOpConversion,
+           WasmRotrOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
new file mode 100644
index 0000000000000..90c92001a78d6
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @func_0() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_0() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_s %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_1() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_1() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_u %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 8 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 16 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_4() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_4() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 8 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_5() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_5() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 16 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_6() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_6() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 32 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
new file mode 100644
index 0000000000000..91e929bd5ff35
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -0,0 +1,75 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotl %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotl_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val << (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i32
+
+// (val >> (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotl_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val << (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i64
+
+// (val >> (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
new file mode 100644
index 0000000000000..77e301c3221e3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotr %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotr_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val >> (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i32
+
+// (val << (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+// CHECK:         }
+wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotr_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val >> (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i64
+
+// (val << (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+// CHECK:         }
+wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}

>From 6431a7494a5edc89de997962d92b1803d6aed43f Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:24:21 +0900
Subject: [PATCH 3/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for control
 flow operations in functions

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 184 ++++++++-
 .../RaiseWasm/wasm-blocks-to-cf.mlir          | 366 ++++++++++++++++++
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 206 ++++++++++
 3 files changed, 755 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index bf958aad5012b..d9dcb756790f8 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -371,6 +371,187 @@ struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
 
 struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   using OpConversionPattern::OpConversionPattern;
+  ///
+  /// Control flow conversion needs shared state for tracking which block
+  /// corresponds to which operation at which level.
+  ///
+  /// This class handles such tracking and performs the conversion of control
+  /// flow related ops contained in a function.
+  class CFRewriterVisitor {
+  private:
+    using branch_to_dest_t =
+        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+    Value getCompResultAsI1(Value compResult,
+                            ConversionPatternRewriter &rewriter) {
+      auto testValue = rewriter.create<arith::ConstantOp>(
+          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = rewriter
+                      .create<arith::CmpIOp>(
+                          compResult.getLoc(), rewriter.getIntegerType(1),
+                          arith::CmpIPredicate::ne, compResult, testValue)
+                      .getResult();
+      return flag;
+    }
+
+    void replaceNestLevelWithBranch(BlockOp blockOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(blockOp, regionsToEntry[0],
+                                                blockOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(LoopOp loopOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(loopOp, regionsToEntry[0],
+                                                loopOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(IfOp ifOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      Block *falseDest =
+          regionsToEntry.size() == 2 ? regionsToEntry[1] : ifOp.getTarget();
+      auto flag = getCompResultAsI1(ifOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          ifOp, flag, regionsToEntry[0], ifOp.getInputs(), falseDest,
+          ifOp.getInputs());
+    }
+
+    template <typename LevelType>
+    LogicalResult
+    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+                                      llvm::ArrayRef<Block *> regionsToEntry,
+                                      ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
+      if (!cast)
+        return failure();
+      replaceNestLevelWithBranch(cast, regionsToEntry, rewriter);
+      return success();
+    }
+
+    template <typename... LevelTypes>
+    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+                                       ConversionPatternRewriter &rewriter) {
+      auto sip = rewriter.saveInsertionPoint();
+      Block *blockSuccessor = nestingOp->getSuccessor(0);
+      llvm::SmallVector<Block *, 2> regionEntries;
+      LLVM_DEBUG(llvm::dbgs()
+                     << "Starting inlining blocks for " << nestingOp << "\n";);
+      for (auto &region : nestingOp->getRegions()) {
+        if (region.empty())
+          continue;
+        regionEntries.push_back(&region.front());
+        /// Inline blocks of nested ops
+        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
+            region.getOps<WasmSSALabelLevelInterface>()};
+        for (auto nestedOp : nestedOps) {
+          LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
+          if (failed(inlineBlocks(nestedOp, rewriter)))
+            return failure();
+        }
+        rewriter.inlineRegionBefore(region, blockSuccessor);
+      }
+      LLVM_DEBUG(llvm::dbgs() << "End of region inlining\n");
+      LLVM_DEBUG(llvm::dbgs() << "Replacing initial op with branching\n");
+      rewriter.setInsertionPoint(nestingOp);
+      auto res = success(
+          (... || succeeded(replaceNestLevelWithBranchWrapper<LevelTypes>(
+                      nestingOp, regionEntries, rewriter))));
+      rewriter.restoreInsertionPoint(sip);
+      if (failed(res))
+        return emitError(nestingOp->getLoc(),
+                         "Unable to inline the operation regions.");
+      return success();
+    }
+
+    /// Take a nesting level defining op and inline it in the parent region.
+    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+                               ConversionPatternRewriter &rewriter) {
+      return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
+    }
+
+    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+      auto destIter = branchToDest.find(branchOp);
+      if (destIter == branchToDest.end())
+        return branchOp->emitError("No indexed label op for this operation: ")
+               << branchOp;
+      return destIter->second;
+    }
+
+    inline void convertBranch(BranchIfOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      auto flag = getCompResultAsI1(brOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          brOp, flag, dest, brOp.getInputs(), brOp.getElseSuccessor(),
+          ValueRange{});
+    }
+
+    inline void convertBranch(BlockReturnOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(brOp, dest, brOp.getInputs());
+    }
+
+    template <typename LevelInterfaceT>
+    inline LogicalResult
+    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+                         ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
+      if (!cast)
+        return failure();
+      auto sip = rewriter.saveInsertionPoint();
+      rewriter.setInsertionPoint(branchOp);
+      convertBranch(cast, dest, rewriter);
+      rewriter.restoreInsertionPoint(sip);
+      return success();
+    }
+
+    template <typename... BranchInterfaceT>
+    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+                                        ConversionPatternRewriter &rewriter) {
+      auto dest = getBlockFor(branchOp);
+      if (failed(dest))
+        return failure();
+      auto res =
+          success((... || succeeded(convertBranchWrapper<BranchInterfaceT>(
+                              branchOp, *dest, rewriter))));
+      if (failed(res))
+        return emitError(branchOp->getLoc(), "No known converter for op ")
+               << branchOp;
+      return res;
+    }
+
+    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+                                ConversionPatternRewriter &rewriter) {
+      return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
+                                                              rewriter);
+    }
+
+    func::FuncOp func;
+    branch_to_dest_t branchToDest;
+
+  public:
+    CFRewriterVisitor(func::FuncOp func) : func{func} {
+      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+        branchToDest.insert({branchOp, branchOp.getTarget()});
+      });
+    }
+    LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
+      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
+          func.getOps<WasmSSALabelLevelInterface>()};
+      for (auto nestingOp : nestingOps)
+        if (failed(inlineBlocks(nestingOp, rewriter)))
+          return failure();
+
+      auto res =
+          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+            if (failed(convertBranch(branchOp, rewriter)))
+              return WalkResult::interrupt();
+            return WalkResult::advance();
+          });
+      return failure(res.wasInterrupted());
+    }
+  };
 
   LogicalResult
   matchAndRewrite(FuncOp funcOp, FuncOp::Adaptor adaptor,
@@ -393,7 +574,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     rewriter.applySignatureConversion(oldEntryBlock, sC, getTypeConverter());
     rewriter.replaceOp(funcOp, newFunc);
-    return success();
+    CFRewriterVisitor cfRewriter{newFunc};
+    return cfRewriter.rewrite(rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
new file mode 100644
index 0000000000000..3b275573bcbd3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -0,0 +1,366 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK_CANONICALIZED %s
+
+// CHECK-LABEL:   func.func @i_am_a_block(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 17 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 42 : i32
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_3:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @i_am_a_block(
+// CHECK_CANONICALIZED-SAME:      %[[VAL_0:.*]]: i32) -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_1:.*]] = arith.constant 42 : i32
+// CHECK_CANONICALIZED:           %[[VAL_3:.*]] = memref.alloca() : memref<i32>
+// CHECK_CANONICALIZED:           memref.store %[[VAL_1]], %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED-NOT:       memref.store
+// CHECK_CANONICALIZED:           %[[VAL_4:.*]] = memref.load %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED:           return %[[VAL_4]] : i32
+wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.const 17 : i32
+  wasmssa.local_set %arg0 : ref to i32 to %1 : i32
+  wasmssa.block : {
+    %2 = wasmssa.const 42 : i32
+    wasmssa.local_set %arg0 : ref to i32 to %2 : i32
+    wasmssa.block_return
+  }> ^bb1
+  ^bb1:
+  %res = wasmssa.local_get %arg0 : ref to i32
+  wasmssa.return %res : i32
+}
+
+
+// CHECK-LABEL:   func.func @func_0(
+// CHECK-SAME:                      %[[ARG0:.*]]: i32) {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block : {
+      %3 = wasmssa.const 3: i32
+      wasmssa.block : {
+        %4 = wasmssa.const 4: i32
+        wasmssa.block_return
+      }> ^bb1
+    ^bb1:  // pred: ^bb0
+      %5 = wasmssa.const 5: i32
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_1() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_1() {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %3 = wasmssa.const 3: i32
+  wasmssa.block : {
+    %4 = wasmssa.const 4: i32
+    wasmssa.block_return
+  }> ^bb2
+^bb2:  // pred: ^bb1
+  %5 = wasmssa.const 5: i32
+  wasmssa.block : {
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb3
+^bb3:  // pred: ^bb2
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 14 : i32
+// CHECK:           cf.br ^bb1(%[[VAL_0]] : i32)
+// CHECK:         ^bb1(%[[VAL_1:.*]]: i32):
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.addi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
+// CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
+// CHECK:           return %[[VAL_4]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 14 : i32
+  wasmssa.block(%0) : i32 : {
+  ^bb0(%arg0: i32):
+    %2 = wasmssa.const 1 : i32
+    %3 = wasmssa.add %arg0 %2 : i32
+    wasmssa.block_return %3 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
+// CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
+// CHECK:           return %[[VAL_1]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 17 : i32
+    wasmssa.block_return %1 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+//// ============= Branch instructions etc ==========
+
+// CHECK-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+
+wasmssa.func nested @branch_if_taken() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 2 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+// CHECK_CANONICALIZED:         }
+wasmssa.func nested @branch_if_continue() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 0 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @if(
+// CHECK-SAME:                  %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.andi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.cmpi ne, %[[VAL_3]], %[[VAL_4]] : i32
+// CHECK:           cf.cond_br %[[VAL_5]], ^bb1, ^bb2
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 3 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.muli %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_10:.*]] = arith.addi %[[VAL_8]], %[[VAL_9]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_10]] : i32)
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_13:.*]] = arith.shrui %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
+// CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
+// CHECK:           return %[[VAL_14]] : i32
+wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.const 1 : i32
+  %3 = wasmssa.and %1 %2 : i32
+  "wasmssa.if"(%3)[^bb1] ({
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 3 : i32
+    %7 = wasmssa.mul %5 %6 : i32
+    %8 = wasmssa.const 1 : i32
+    %9 = wasmssa.add %7 %8 : i32
+    wasmssa.block_return %9 : i32
+  }, {
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    wasmssa.block_return %7 : i32
+  }) : (i32) -> ()
+^bb1(%4: i32):  // pred: ^bb0
+  wasmssa.return %4 : i32
+}
+
+// CHECK-LABEL:   func.func @if_else(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.andi %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.cmpi ne, %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           cf.cond_br %[[VAL_6]], ^bb1(%[[VAL_1]] : i32), ^bb2(%[[VAL_1]] : i32)
+// CHECK:         ^bb1(%[[VAL_7:.*]]: i32):
+// CHECK:           %[[VAL_8:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
+// CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
+// CHECK:           return %[[VAL_10]] : i32
+wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.local_get %arg0 : ref to i32
+  %3 = wasmssa.const 1 : i32
+  %4 = wasmssa.and %2 %3 : i32
+  "wasmssa.if"(%4, %1)[^bb1] ({
+  ^bb0(%arg1: i32):
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.add %arg1 %6 : i32
+    wasmssa.block_return %7 : i32
+  }, {
+  }) : (i32, i32) -> ()
+^bb1(%5: i32):  // pred: ^bb0
+  wasmssa.return %5 : i32
+}
+
+// CHECK-LABEL:   func.func @if_if(
+// CHECK-SAME:                     %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = math.cttz %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi ne, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           cf.cond_br %[[VAL_4]], ^bb1, ^bb4
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.shrui %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = math.cttz %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_11:.*]] = arith.cmpi ne, %[[VAL_9]], %[[VAL_10]] : i32
+// CHECK:           cf.cond_br %[[VAL_11]], ^bb2(%[[VAL_5]] : i32), ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_12:.*]]: i32):
+// CHECK:           %[[VAL_13:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_14:.*]] = arith.addi %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_14]] : i32)
+// CHECK:         ^bb3(%[[VAL_15:.*]]: i32):
+// CHECK:           cf.br ^bb5(%[[VAL_15]] : i32)
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_16:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
+// CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
+// CHECK:           return %[[VAL_17]] : i32
+wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.ctz %1 : i32
+  "wasmssa.if"(%2)[^bb1] ({
+    %4 = wasmssa.const 2 : i32
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    %8 = wasmssa.ctz %7 : i32
+    "wasmssa.if"(%8, %4)[^bb1] ({
+    ^bb0(%arg1: i32):
+      %10 = wasmssa.const 2 : i32
+      %11 = wasmssa.add %arg1 %10 : i32
+      wasmssa.block_return %11 : i32
+    }, {
+    }) : (i32, i32) -> ()
+  ^bb1(%9: i32):  // pred: ^bb0
+    wasmssa.block_return %9 : i32
+  }, {
+    %4 = wasmssa.const 1 : i32
+    wasmssa.block_return %4 : i32
+  }) : (i32) -> ()
+^bb1(%3: i32):  // pred: ^bb0
+  wasmssa.return %3 : i32
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
new file mode 100644
index 0000000000000..e49909cc9417e
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -0,0 +1,206 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
+
+module {
+  wasmssa.func nested @func_0() {
+    wasmssa.loop : {
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   module {
+// CHECK:           func.func @func_0() {
+// CHECK:             cf.br ^bb1
+// CHECK:           ^bb1:
+// CHECK:             cf.br ^bb2
+// CHECK:           ^bb2:
+// CHECK:             return
+// CHECK:           }
+
+// CHECK-CANONICAL-LABEL:  func.func @func_0() {
+// CHECK-CANONICAL:             return
+// CHECK-CANONICAL:           }
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() -> i32 {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.block_return %3 : i32
+    }> ^bb1
+  ^bb1(%1: i32):  // pred: ^bb0
+    wasmssa.return %1 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_0() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           cf.br ^bb2(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() -> i32 {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK-CANONICAL:           return %[[VAL_5]] : i32
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.branch_if %3 to level 0 else ^bb1
+    ^bb1:  // pred: ^bb0
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_7:.*]] = arith.cmpi ne, %[[VAL_5]], %[[VAL_6]] : i32
+// CHECK:           cf.cond_br %[[VAL_7]], ^bb1, ^bb2
+// CHECK:         ^bb2:
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_4]], ^bb1, ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           return
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    %1 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %2 = wasmssa.local_get %0 : ref to i32
+      %3 = wasmssa.const 1 : i32
+      %4 = wasmssa.add %2 %3 : i32
+      wasmssa.local_set %0 : ref to i32 to %4 : i32
+      wasmssa.loop : {
+        %8 = wasmssa.const 12 : i32
+        %9 = wasmssa.local_get %0 : ref to i32
+        %10 = wasmssa.gt_si %8 %9 : i32 -> i32
+        wasmssa.branch_if %10 to level 0 else ^bb1
+      ^bb1:  // pred: ^bb0
+        wasmssa.block_return %8 : i32
+      }> ^bb1
+    ^bb1(%5: i32):  // pred: ^bb0
+      %6 = wasmssa.const 10 : i32
+      %7 = wasmssa.lt_si %5 %6 : i32 -> i32
+      wasmssa.branch_if %7 to level 0 else ^bb2
+    ^bb2:  // pred: ^bb1
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_3]], %[[VAL_2]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_4:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_5:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.addi %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           memref.store %[[VAL_6]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_7:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_9:.*]] = arith.cmpi sgt, %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.extui %[[VAL_9]] : i1 to i32
+// CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_12:.*]] = arith.cmpi ne, %[[VAL_10]], %[[VAL_11]] : i32
+// CHECK:           cf.cond_br %[[VAL_12]], ^bb2, ^bb3
+// CHECK:         ^bb3:
+// CHECK:           cf.br ^bb4(%[[VAL_7]] : i32)
+// CHECK:         ^bb4(%[[VAL_13:.*]]: i32):
+// CHECK:           %[[VAL_14:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_15:.*]] = arith.cmpi slt, %[[VAL_13]], %[[VAL_14]] : i32
+// CHECK:           %[[VAL_16:.*]] = arith.extui %[[VAL_15]] : i1 to i32
+// CHECK:           %[[VAL_17:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_18:.*]] = arith.cmpi ne, %[[VAL_16]], %[[VAL_17]] : i32
+// CHECK:           cf.cond_br %[[VAL_18]], ^bb1, ^bb5
+// CHECK:         ^bb5:
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
+// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
+// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
+// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
+// CHECK-CANONICAL:         ^bb4:
+// CHECK-CANONICAL:           return

>From 2af7f4e6300c16db66abf0ad83efa9e338ca21bd Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:36:04 +0900
Subject: [PATCH 4/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for memory

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 41 +++++++++++++++++++
 .../RaiseWasm/wasm-memory-to-memref.mlir      | 14 +++++++
 2 files changed, 55 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index d9dcb756790f8..53a8900b965a3 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -677,6 +677,46 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(MemOp memOp, MemOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = memOp.getLoc();
+    auto bufferType =
+        MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
+    auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
+        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
+    auto initializerName = (memPtr.getSymName() + "::initializer").str();
+    auto memInitializer = rewriter.create<func::FuncOp>(
+        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    memInitializer->setAttr(rewriter.getStringAttr("initializer"),
+                            rewriter.getUnitAttr());
+    auto *initializerBody = memInitializer.addEntryBlock();
+    auto sip = rewriter.saveInsertionPoint();
+    rewriter.setInsertionPointToStart(initializerBody);
+    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
+        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = rewriter.create<memref::AllocOp>(
+        loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto castOp =
+        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
+    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
+    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
+                                     memRefPtr.getResult(),
+                                     ValueRange{idx.getResult()});
+    rewriter.create<func::ReturnOp>(loc);
+    rewriter.restoreInsertionPoint(sip);
+    rewriter.create<func::CallOp>(loc, memInitializer);
+    return success();
+  }
+};
+
 inline TypedAttr getInitializerAttr(Type t) {
   assert(t.isIntOrFloat() &&
          "This helper is intended to use with int and float types");
@@ -847,6 +887,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmLtSIOpConversion,
            WasmLtUIOpConversion,
            WasmMaxOpConversion,
+           WasmMemoryOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
            WasmNeOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
new file mode 100644
index 0000000000000..09b07184d97fa
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
@@ -0,0 +1,14 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+"wasmssa.memory"() <{limits = !wasmssa<limit[17:]>, sym_name = "mem_0", sym_visibility = "nested"}> : () -> ()
+
+// CHECK-LABEL:   memref.global "nested" @mem_0 : memref<1xmemref<?xi8>>
+
+// CHECK-LABEL:   func.func @"mem_0::initializer"() attributes {initializer} {
+// CHECK:           %[[VAL_0:.*]] = memref.get_global @mem_0 : memref<1xmemref<?xi8>>
+// CHECK:           %[[VAL_1:.*]] = memref.alloc() : memref<17xi8>
+// CHECK:           %[[VAL_2:.*]] = memref.cast %[[VAL_1]] : memref<17xi8> to memref<?xi8>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]]{{\[}}%[[VAL_3]]] : memref<1xmemref<?xi8>>
+// CHECK:           return
+// CHECK:         func.call @"mem_0::initializer"() : () -> ()

>From ec4763983eff03797ce2a4f1d60e296b250450a1 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 30 Apr 2026 21:00:48 +0900
Subject: [PATCH 5/7] [mlir][wasm] Deprecate rewriter.create in favor of
 <op>::create and cleanup the tests

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 123 ++++++++----------
 .../RaiseWasm/wasm-blocks-to-cf.mlir          |  18 +--
 .../wasm-comparisons-to-arith-cmp.mlir        |  64 ++++-----
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   |  14 +-
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir |   8 +-
 .../RaiseWasm/wasm-rotl-to-arith.mlir         |   4 +-
 .../RaiseWasm/wasm-rotr-to-arith.mlir         |   4 +-
 7 files changed, 112 insertions(+), 123 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 53a8900b965a3..0e6e0c05b79b6 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
+#include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinDialect.h"
 #include "mlir/IR/ValueRange.h"
@@ -159,25 +160,22 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
 
     // Materialize (width - 1) for use in both sides of the expression.
     auto cstWidthMinusOne =
-        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+        ConstOp::create(rewriter, loc, IntegerAttr::get(ty, width - 1));
 
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
-    auto orLHS = rewriter.create<LHSShiftOp>(
-        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+    auto orLHS = LHSShiftOp::create(
+        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
-    auto orRHS = rewriter.create<RHSShiftOp>(
-        loc, val,
+    auto orRHS = RHSShiftOp::create(
+        rewriter, loc, val,
         // (-bits & (width - 1))
-        rewriter.create<AndOp>(
-            loc,
+        AndOp::create(
+            rewriter, loc,
             // 0 - bits == -bits
-            rewriter.create<SubOp>(
-                loc,
-                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
-                bits),
+            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
             cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
@@ -199,8 +197,7 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
   matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
-        rewriter
-            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+        TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
                               AttrType::get(rewriter.getContext(), flag),
                               adaptor.getLhs(), adaptor.getRhs())
             .getResult();
@@ -257,19 +254,15 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          rewriter
-              .create<arith::CmpIOp>(
-                  srcOp.getLoc(), rewriter.getI1Type(),
-                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                  adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          rewriter
-              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
-                                     arith::CmpFPredicateAttr::get(
-                                         rewriter.getContext(), FloatFlag),
-                                     adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -320,13 +313,9 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
     auto zero =
-        rewriter
-            .create<arith::ConstantOp>(
-                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
             .getResult();
-    auto cmpRes = rewriter
-                      .create<arith::CmpIOp>(
-                          loc, rewriter.getI1Type(),
+    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
                           arith::CmpIPredicateAttr::get(
                               rewriter.getContext(), arith::CmpIPredicate::eq),
                           adaptor.getInput(), zero)
@@ -346,9 +335,7 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = rewriter.create<arith::TruncIOp>(
-        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
-        adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -380,15 +367,11 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   class CFRewriterVisitor {
   private:
     using branch_to_dest_t =
-        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = rewriter.create<arith::ConstantOp>(
-          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = rewriter
-                      .create<arith::CmpIOp>(
-                          compResult.getLoc(), rewriter.getIntegerType(1),
-                          arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -420,7 +403,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelType>
     LogicalResult
-    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+    replaceNestLevelWithBranchWrapper(LabelLevelOpInterface nestingOp,
                                       llvm::ArrayRef<Block *> regionsToEntry,
                                       ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
@@ -431,7 +414,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... LevelTypes>
-    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineNestDispatcher(LabelLevelOpInterface nestingOp,
                                        ConversionPatternRewriter &rewriter) {
       auto sip = rewriter.saveInsertionPoint();
       Block *blockSuccessor = nestingOp->getSuccessor(0);
@@ -443,8 +426,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
           continue;
         regionEntries.push_back(&region.front());
         /// Inline blocks of nested ops
-        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
-            region.getOps<WasmSSALabelLevelInterface>()};
+        llvm::SmallVector<LabelLevelOpInterface> nestedOps{
+            region.getOps<LabelLevelOpInterface>()};
         for (auto nestedOp : nestedOps) {
           LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
           if (failed(inlineBlocks(nestedOp, rewriter)))
@@ -466,12 +449,12 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     /// Take a nesting level defining op and inline it in the parent region.
-    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineBlocks(LabelLevelOpInterface nestingOp,
                                ConversionPatternRewriter &rewriter) {
       return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
     }
 
-    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+    llvm::FailureOr<Block *> getBlockFor(LabelBranchingOpInterface branchOp) {
       auto destIter = branchToDest.find(branchOp);
       if (destIter == branchToDest.end())
         return branchOp->emitError("No indexed label op for this operation: ")
@@ -494,7 +477,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelInterfaceT>
     inline LogicalResult
-    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+    convertBranchWrapper(LabelBranchingOpInterface branchOp, Block *dest,
                          ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
       if (!cast)
@@ -507,7 +490,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... BranchInterfaceT>
-    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranchDispatch(LabelBranchingOpInterface branchOp,
                                         ConversionPatternRewriter &rewriter) {
       auto dest = getBlockFor(branchOp);
       if (failed(dest))
@@ -521,7 +504,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
       return res;
     }
 
-    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranch(LabelBranchingOpInterface branchOp,
                                 ConversionPatternRewriter &rewriter) {
       return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
                                                               rewriter);
@@ -532,19 +515,19 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
   public:
     CFRewriterVisitor(func::FuncOp func) : func{func} {
-      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+      func.walk([this](LabelBranchingOpInterface branchOp) {
         branchToDest.insert({branchOp, branchOp.getTarget()});
       });
     }
     LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
-      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
-          func.getOps<WasmSSALabelLevelInterface>()};
+      llvm::SmallVector<LabelLevelOpInterface> nestingOps{
+          func.getOps<LabelLevelOpInterface>()};
       for (auto nestingOp : nestingOps)
         if (failed(inlineBlocks(nestingOp, rewriter)))
           return failure();
 
       auto res =
-          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+          func->walk([this, &rewriter](LabelBranchingOpInterface branchOp) {
             if (failed(convertBranch(branchOp, rewriter)))
               return WalkResult::interrupt();
             return WalkResult::advance();
@@ -687,32 +670,38 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
     auto bufferType =
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memVisibility = memOp.getVisibility();
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    mlir::StringAttr visAttr;
+    if (memVisibility == mlir::SymbolTable::Visibility::Public)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+    else if (memVisibility == mlir::SymbolTable::Visibility::Private)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+    else
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr,
         TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = rewriter.create<func::FuncOp>(
-        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
+                                               FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
-        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = rewriter.create<memref::AllocOp>(
-        loc,
-        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
-        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
-    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
-    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
-                                     memRefPtr.getResult(),
-                                     ValueRange{idx.getResult()});
-    rewriter.create<func::ReturnOp>(loc);
+        memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    memref::StoreOp::create(rewriter, loc, castOp.getResult(),
+                            memRefPtr.getResult(), ValueRange{idx.getResult()});
+    func::ReturnOp::create(rewriter, loc);
     rewriter.restoreInsertionPoint(sip);
-    rewriter.create<func::CallOp>(loc, memInitializer);
+    func::CallOp::create(rewriter, loc, memInitializer);
     return success();
   }
 };
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
index 3b275573bcbd3..fdce71cab096b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -60,7 +60,7 @@ wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+wasmssa.func @func_0(%arg0: !wasmssa<local ref to i32>) {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -104,7 +104,7 @@ wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_1() {
+wasmssa.func @func_1() {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -136,7 +136,7 @@ wasmssa.func nested @func_1() {
 // CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
 // CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
 // CHECK:           return %[[VAL_4]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 14 : i32
   wasmssa.block(%0) : i32 : {
   ^bb0(%arg0: i32):
@@ -155,7 +155,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
 // CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
 // CHECK:           return %[[VAL_1]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 17 : i32
     wasmssa.block_return %1 : i32
@@ -185,7 +185,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 
-wasmssa.func nested @branch_if_taken() -> i32 {
+wasmssa.func @branch_if_taken() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 2 : i32
@@ -218,7 +218,7 @@ wasmssa.func nested @branch_if_taken() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 // CHECK_CANONICALIZED:         }
-wasmssa.func nested @branch_if_continue() -> i32 {
+wasmssa.func @branch_if_continue() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 0 : i32
@@ -256,7 +256,7 @@ wasmssa.func nested @branch_if_continue() -> i32 {
 // CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
 // CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
 // CHECK:           return %[[VAL_14]] : i32
-wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.const 1 : i32
   %3 = wasmssa.and %1 %2 : i32
@@ -294,7 +294,7 @@ wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
 // CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
 // CHECK:           return %[[VAL_10]] : i32
-wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.local_get %arg0 : ref to i32
   %3 = wasmssa.const 1 : i32
@@ -339,7 +339,7 @@ wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
 // CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
 // CHECK:           return %[[VAL_17]] : i32
-wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.ctz %1 : i32
   "wasmssa.if"(%2)[^bb1] ({
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
index bc952b47520ca..6643b0389f405 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -2,194 +2,194 @@
 
 
 module {
-  wasmssa.func nested @func_lt_si32() -> i32 {
+  wasmssa.func @func_lt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
 
-  wasmssa.func nested @func_le_si32() -> i32 {
+  wasmssa.func @func_le_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui32() -> i32 {
+  wasmssa.func @func_lt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui32() -> i32 {
+  wasmssa.func @func_le_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si32() -> i32 {
+  wasmssa.func @func_gt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui32() -> i32 {
+  wasmssa.func @func_gt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si32() -> i32 {
+  wasmssa.func @func_ge_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui32() -> i32 {
+  wasmssa.func @func_ge_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_si64() -> i32 {
+  wasmssa.func @func_lt_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_si64() -> i32 {
+  wasmssa.func @func_le_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+  wasmssa.func @func_lt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui_i64() -> i32 {
+  wasmssa.func @func_le_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si_i64() -> i32 {
+  wasmssa.func @func_gt_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+  wasmssa.func @func_gt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si_i64() -> i32 {
+  wasmssa.func @func_ge_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+  wasmssa.func @func_ge_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f32() -> i32 {
+  wasmssa.func @func_lt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.lt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f32() -> i32 {
+  wasmssa.func @func_le_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.le %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f32() -> i32 {
+  wasmssa.func @func_gt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.gt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f32() -> i32 {
+  wasmssa.func @func_ge_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ge %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f64() -> i32 {
+  wasmssa.func @func_lt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.lt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f64() -> i32 {
+  wasmssa.func @func_le_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.le %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f64() -> i32 {
+  wasmssa.func @func_gt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.gt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f64() -> i32 {
+  wasmssa.func @func_ge_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.ge %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i32() -> i32 {
+  wasmssa.func @func_eq_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.eq %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i64() -> i32 {
+  wasmssa.func @func_eq_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.eq %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f32() -> i32 {
+  wasmssa.func @func_eq_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.eq %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f64() -> i32 {
+  wasmssa.func @func_eq_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.eq %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-    wasmssa.func nested @func_ne_i32() -> i32 {
+    wasmssa.func @func_ne_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ne %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_i64() -> i32 {
+  wasmssa.func @func_ne_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.ne %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f32() -> i32 {
+  wasmssa.func @func_ne_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ne %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f64() -> i32 {
+  wasmssa.func @func_ne_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.ne %0 %1 : f64 -> i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
index 90c92001a78d6..2b8bb651bbe5b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -4,7 +4,7 @@
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_0() -> i64 {
+wasmssa.func @func_0() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_s %0 to i64
   wasmssa.return %1 : i64
@@ -14,7 +14,7 @@ wasmssa.func nested @func_0() -> i64 {
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_1() -> i64 {
+wasmssa.func @func_1() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_u %0 to i64
   wasmssa.return %1 : i64
@@ -25,7 +25,7 @@ wasmssa.func nested @func_1() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 8 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -36,7 +36,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 16 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -47,7 +47,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_4() -> i64 {
+wasmssa.func @func_4() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 8 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -58,7 +58,7 @@ wasmssa.func nested @func_4() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_5() -> i64 {
+wasmssa.func @func_5() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 16 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -69,7 +69,7 @@ wasmssa.func nested @func_5() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_6() -> i64 {
+wasmssa.func @func_6() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 32 low bits from %0: i64
   wasmssa.return %1 : i64
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index e49909cc9417e..6ab812008a2f8 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -2,7 +2,7 @@
 // RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     wasmssa.loop : {
       wasmssa.block_return
     }> ^bb1
@@ -27,7 +27,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() -> i32 {
+  wasmssa.func @func_0() -> i32 {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -67,7 +67,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -117,7 +117,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     %1 = wasmssa.local of type i32
     wasmssa.loop : {
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
index 91e929bd5ff35..b69f88afd4d7c 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -31,7 +31,7 @@
 
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
-wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -66,7 +66,7 @@ wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // Form final result.
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
-wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
index 77e301c3221e3..f0b77271c86dd 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -32,7 +32,7 @@
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
 // CHECK:         }
-wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -68,7 +68,7 @@ wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
 // CHECK:         }
-wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 

>From a249e73f27672c4d5dfb8552cd0057da55e9e3c0 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 16:03:07 +0900
Subject: [PATCH 6/7] [mlir][wasm] Fix format

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 88 +++++++++++--------
 1 file changed, 52 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 0e6e0c05b79b6..b4b5006352c11 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -165,18 +165,21 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
     auto orLHS = LHSShiftOp::create(
-        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
+        rewriter, loc, val,
+        AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
     auto orRHS = RHSShiftOp::create(
         rewriter, loc, val,
         // (-bits & (width - 1))
-        AndOp::create(
-            rewriter, loc,
-            // 0 - bits == -bits
-            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
-            cstWidthMinusOne));
+        AndOp::create(rewriter, loc,
+                      // 0 - bits == -bits
+                      SubOp::create(rewriter, loc,
+                                    ConstOp::create(rewriter, loc,
+                                                    IntegerAttr::get(ty, 0)),
+                                    bits),
+                      cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
     // expression.
@@ -198,8 +201,8 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
         TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                              AttrType::get(rewriter.getContext(), flag),
-                              adaptor.getLhs(), adaptor.getRhs())
+                         AttrType::get(rewriter.getContext(), flag),
+                         adaptor.getLhs(), adaptor.getRhs())
             .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -254,15 +257,17 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -312,13 +317,15 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
-    auto zero =
-        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
-            .getResult();
-    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
-                          arith::CmpIPredicateAttr::get(
-                              rewriter.getContext(), arith::CmpIPredicate::eq),
-                          adaptor.getInput(), zero)
+    auto zero = arith::ConstantOp::create(
+                    rewriter, loc,
+                    rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+                    .getResult();
+    auto cmpRes = arith::CmpIOp::create(
+                      rewriter, loc, rewriter.getI1Type(),
+                      arith::CmpIPredicateAttr::get(rewriter.getContext(),
+                                                    arith::CmpIPredicate::eq),
+                      adaptor.getInput(), zero)
                       .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -335,7 +342,9 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(
+        rewriter, extendLowBytesSOp->getLoc(),
+        rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -366,12 +375,14 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   /// flow related ops contained in a function.
   class CFRewriterVisitor {
   private:
-    using branch_to_dest_t =
-        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
+    using branch_to_dest_t = llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(),
+                                                 rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(
+                      rewriter, compResult.getLoc(), rewriter.getIntegerType(1),
+                      arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -671,29 +682,34 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
     auto memVisibility = memOp.getVisibility();
-    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a
+    // string attribute.
     mlir::StringAttr visAttr;
     if (memVisibility == mlir::SymbolTable::Visibility::Public)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
     else if (memVisibility == mlir::SymbolTable::Visibility::Private)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
     else
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
 
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), visAttr,
-        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr, TypeAttr::get(bufferPtrType),
+        /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
-                                               FunctionType::get(getContext(), {}, {}));
+    auto memInitializer =
+        func::FuncOp::create(rewriter, loc, initializerName,
+                             FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(
+        rewriter, loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
         memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
     auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);

>From 528e3fc46245cf2e62904b6d60508b8aa301b5ca Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 21:26:54 +0900
Subject: [PATCH 7/7] [mlir][wasm] Update tests references

---
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 29 ++++++++-----------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index 6ab812008a2f8..cdaf249d9ccf6 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -183,24 +183,19 @@ module {
 // CHECK:         }
 
 // CHECK-CANONICAL-LABEL:   func.func @func_0() {
-// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
-// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
-// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
-// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
-// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
-// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CONSTANT_0:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_1:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_2:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[ALLOCA_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[CONSTANT_2]], %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[LOAD_0:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[ADDI_0:.*]] = arith.addi %[[LOAD_0]], %[[CONSTANT_1]] : i32
+// CHECK-CANONICAL:           memref.store %[[ADDI_0]], %[[ALLOCA_0]][] : memref<i32>
 // CHECK-CANONICAL:           cf.br ^bb1
 // CHECK-CANONICAL:         ^bb1:
-// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
-// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:           %[[LOAD_1:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CMPI_0:.*]] = arith.cmpi slt, %[[LOAD_1]], %[[CONSTANT_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[CMPI_0]], ^bb1, ^bb2
 // CHECK-CANONICAL:         ^bb2:
-// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
-// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
-// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
-// CHECK-CANONICAL:         ^bb4:
 // CHECK-CANONICAL:           return
+// CHECK-CANONICAL:         }


From llvmlistbot at llvm.org  Thu Jul 16 06:24:45 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Thu, 16 Jul 2026 06:24:45 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58db9d.f3151ff8.b27a.73ca@mx.google.com>


https://github.com/jsjodin edited https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 06:24:59 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Thu, 16 Jul 2026 06:24:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58dbab.e5907290.27bf36.63bb@mx.google.com>


https://github.com/jsjodin edited https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 06:59:31 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Thu, 16 Jul 2026 06:59:31 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a58e3c3.bd93e731.18e8ee.7772@mx.google.com>


================
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
----------------
skatrak wrote:

Yeah, adding a new attribute just for that doesn't make a lot of sense, but I think using `omp.version` as the indirect way of checking whether this is being compiled for OpenMP would be a good solution.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Thu Jul 16 07:59:45 2026
From: llvmlistbot at llvm.org (Krzysztof Drewniak)
Date: Thu, 16 Jul 2026 07:59:45 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a58f1e1.449bfb44.1b765.5c2f@mx.google.com>


krzysz00 wrote:

Ping

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 08:00:50 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 16 Jul 2026 08:00:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP conv2d (PR
	#210054)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210054@github.com>
Message-ID: <6a58f222.d2cb4adc.69be2.e2f3@mx.google.com>


https://github.com/lhutton1 approved this pull request.

LGTM, thanks @iliyan-georgiev-arm!

At some point, I think we'll need to consider how the compliance data can be condensed, but this is a separate task IMO.

https://github.com/llvm/llvm-project/pull/210054

From llvmlistbot at llvm.org  Thu Jul 16 09:04:29 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Thu, 16 Jul 2026 09:04:29 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a59010d.824e11a6.1389c5.2204@mx.google.com>


https://github.com/jsjodin updated https://github.com/llvm/llvm-project/pull/209592

>From d6d61356abf6768b91fb1c4fe5094417d1dd39b9 Mon Sep 17 00:00:00 2001
From: Jan Leyonberg <jan_sjodin at yahoo.com>
Date: Tue, 14 Jul 2026 14:54:30 -0400
Subject: [PATCH 1/3] [CIR][OpenMP] Add host op filtering pass to CIR pipeline

This patch adds the host op filtering pass which prevents host code being
lowered when compiling for the target device.
---
 .../CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp |  1 +
 .../CodeGenOpenMP/target-host-op-filtering.c  | 35 +++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100644 clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c

diff --git a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
index 0e412090a16da..67406d261ce1a 100644
--- a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
+++ b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
@@ -5134,6 +5134,7 @@ void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
   mlir::populateCIRPreLoweringPasses(pm);
   pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
+  pm.addPass(mlir::omp::createHostOpFilteringPass());
 }
 
 std::unique_ptr<llvm::Module>
diff --git a/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c b/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c
new file mode 100644
index 0000000000000..ab20b285c79ed
--- /dev/null
+++ b/clang/test/CIR/CodeGenOpenMP/target-host-op-filtering.c
@@ -0,0 +1,35 @@
+// Verifies the omp-host-op-filtering pass runs in the CIR-to-LLVM device
+// pipeline.//
+// RUN: %clang_cc1 -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa \
+// RUN:   -fclangir -emit-llvm-bc %s -o %t-cir-host.bc
+// RUN: %clang_cc1 -triple amdgcn-amd-amdhsa -fopenmp -fopenmp-is-target-device \
+// RUN:   -fopenmp-host-ir-file-path %t-cir-host.bc \
+// RUN:   -fclangir -emit-llvm %s -o - \
+// RUN:   | FileCheck %s --check-prefix=LLVM
+
+void use(int);
+int host_helper(int);
+
+void f(int x) {
+  // Host-only computation that must not leak into the device module.
+  int y = host_helper(x);
+  use(y);
+#pragma omp target map(tofrom : x)
+  {
+    x = x + 1;
+  }
+}
+
+// LLVM-LABEL: define hidden void @f(
+// LLVM-NOT:     call i32 @host_helper
+// LLVM-NOT:     call void @use
+// LLVM:         ret void
+
+// LLVM-LABEL: define weak_odr protected amdgpu_kernel void @__omp_offloading_{{.*}}_f_l
+// LLVM:         call i32 @__kmpc_target_init(
+// LLVM:       user_code.entry:
+// LLVM:         %[[LD:.*]] = load i32, ptr %{{.*}}, align 4
+// LLVM:         %[[ADD:.*]] = add nsw i32 %[[LD]], 1
+// LLVM:         store i32 %[[ADD]], ptr %{{.*}}, align 4
+// LLVM:         call void @__kmpc_target_deinit()
+// LLVM:         ret void

>From 9ce9d47344b8b493b0c6077b692cb7eaf967f2c6 Mon Sep 17 00:00:00 2001
From: Jan Leyonberg <jan_sjodin at yahoo.com>
Date: Wed, 15 Jul 2026 07:13:31 -0400
Subject: [PATCH 2/3] Add guard for adding OpenMP passes.

---
 clang/include/clang/CIR/LowerToLLVM.h                |  2 +-
 clang/include/clang/CIR/Passes.h                     |  5 +++--
 clang/lib/CIR/FrontendAction/CIRGenAction.cpp        |  9 +++++----
 clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp  | 12 +++++++-----
 clang/tools/cir-opt/cir-opt.cpp                      |  9 +++++++--
 clang/tools/cir-translate/cir-translate.cpp          |  7 ++++++-
 .../mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td       |  9 +++++++++
 7 files changed, 38 insertions(+), 15 deletions(-)

diff --git a/clang/include/clang/CIR/LowerToLLVM.h b/clang/include/clang/CIR/LowerToLLVM.h
index df5f9221842ea..305f54017ebe0 100644
--- a/clang/include/clang/CIR/LowerToLLVM.h
+++ b/clang/include/clang/CIR/LowerToLLVM.h
@@ -32,7 +32,7 @@ namespace cir {
 namespace direct {
 std::unique_ptr<llvm::Module>
 lowerDirectlyFromCIRToLLVMIR(mlir::ModuleOp mlirModule,
-                             llvm::LLVMContext &llvmCtx,
+                             llvm::LLVMContext &llvmCtx, bool enableOpenMP,
                              llvm::StringRef mlirSaveTempsOutFile = {},
                              llvm::vfs::FileSystem *fs = nullptr);
 } // namespace direct
diff --git a/clang/include/clang/CIR/Passes.h b/clang/include/clang/CIR/Passes.h
index 1d202ce5bfb52..a42f20405af24 100644
--- a/clang/include/clang/CIR/Passes.h
+++ b/clang/include/clang/CIR/Passes.h
@@ -22,8 +22,9 @@ namespace direct {
 /// Create a pass that fully lowers CIR to the LLVMIR dialect.
 std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass();
 
-/// Adds passes that fully lower CIR to the LLVMIR dialect.
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm);
+/// Adds passes that fully lower CIR to the LLVMIR dialect. When enableOpenMP
+/// is set (-fopenmp), the OpenMP lowering passes are also added.
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP);
 
 } // namespace direct
 } // end namespace cir
diff --git a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
index 92c9ef5499693..28229ac8c58fd 100644
--- a/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
+++ b/clang/lib/CIR/FrontendAction/CIRGenAction.cpp
@@ -57,9 +57,10 @@ getBackendActionFromOutputType(CIRGenAction::OutputType Action) {
 
 static std::unique_ptr<llvm::Module>
 lowerFromCIRToLLVMIR(mlir::ModuleOp MLIRModule, llvm::LLVMContext &LLVMCtx,
+                     bool EnableOpenMP,
                      llvm::StringRef mlirSaveTempsOutFile = {},
                      llvm::vfs::FileSystem *fs = nullptr) {
-  return direct::lowerDirectlyFromCIRToLLVMIR(MLIRModule, LLVMCtx,
+  return direct::lowerDirectlyFromCIRToLLVMIR(MLIRModule, LLVMCtx, EnableOpenMP,
                                               mlirSaveTempsOutFile, fs);
 }
 
@@ -179,9 +180,9 @@ class CIRGenConsumer : public clang::ASTConsumer {
           MlirModule->print(out);
       }
 
-      std::unique_ptr<llvm::Module> LLVMModule =
-          lowerFromCIRToLLVMIR(MlirModule, LLVMCtx, mlirSaveTempsOutFile,
-                               &CI.getVirtualFileSystem());
+      std::unique_ptr<llvm::Module> LLVMModule = lowerFromCIRToLLVMIR(
+          MlirModule, LLVMCtx, C.getLangOpts().OpenMP, mlirSaveTempsOutFile,
+          &CI.getVirtualFileSystem());
 
       if (linkInModules(*LLVMModule))
         return;
diff --git a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
index 67406d261ce1a..ebe74399dda62 100644
--- a/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
+++ b/clang/lib/CIR/Lowering/DirectToLLVM/LowerToLLVM.cpp
@@ -5130,23 +5130,25 @@ std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass() {
   return std::make_unique<ConvertCIRToLLVMPass>();
 }
 
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP) {
   mlir::populateCIRPreLoweringPasses(pm);
-  pm.addPass(mlir::omp::createMarkDeclareTargetPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
-  pm.addPass(mlir::omp::createHostOpFilteringPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
 }
 
 std::unique_ptr<llvm::Module>
 lowerDirectlyFromCIRToLLVMIR(mlir::ModuleOp mlirModule, LLVMContext &llvmCtx,
-                             StringRef mlirSaveTempsOutFile,
+                             bool enableOpenMP, StringRef mlirSaveTempsOutFile,
                              llvm::vfs::FileSystem *fs) {
   llvm::TimeTraceScope scope("lower from CIR to LLVM directly");
 
   mlir::MLIRContext *mlirCtx = mlirModule.getContext();
 
   mlir::PassManager pm(mlirCtx);
-  populateCIRToLLVMPasses(pm);
+  populateCIRToLLVMPasses(pm, enableOpenMP);
 
   (void)mlir::applyPassManagerCLOptions(pm);
 
diff --git a/clang/tools/cir-opt/cir-opt.cpp b/clang/tools/cir-opt/cir-opt.cpp
index 6742985e149e8..a76e4ca6a70e9 100644
--- a/clang/tools/cir-opt/cir-opt.cpp
+++ b/clang/tools/cir-opt/cir-opt.cpp
@@ -26,7 +26,12 @@
 #include "clang/CIR/Passes.h"
 
 struct CIRToLLVMPipelineOptions
-    : public mlir::PassPipelineOptions<CIRToLLVMPipelineOptions> {};
+    : public mlir::PassPipelineOptions<CIRToLLVMPipelineOptions> {
+  Option<bool> enableOpenMP{
+      *this, "enable-openmp",
+      llvm::cl::desc("Add OpenMP-specific CIR-to-LLVM lowering passes"),
+      llvm::cl::init(false)};
+};
 
 int main(int argc, char **argv) {
   // TODO: register needed MLIR passes for CIR?
@@ -44,7 +49,7 @@ int main(int argc, char **argv) {
   mlir::PassPipelineRegistration<CIRToLLVMPipelineOptions> pipeline(
       "cir-to-llvm", "",
       [](mlir::OpPassManager &pm, const CIRToLLVMPipelineOptions &options) {
-        cir::direct::populateCIRToLLVMPasses(pm);
+        cir::direct::populateCIRToLLVMPasses(pm, options.enableOpenMP);
       });
 
   ::mlir::registerPass([]() -> std::unique_ptr<::mlir::Pass> {
diff --git a/clang/tools/cir-translate/cir-translate.cpp b/clang/tools/cir-translate/cir-translate.cpp
index 26adf2cf47d17..9bbbb3939a62f 100644
--- a/clang/tools/cir-translate/cir-translate.cpp
+++ b/clang/tools/cir-translate/cir-translate.cpp
@@ -159,8 +159,13 @@ void registerToLLVMTranslation() {
           return mlir::failure();
 
         llvm::LLVMContext llvmContext;
+        bool enableOpenMP = false;
+        if (auto offloadMod = llvm::dyn_cast<mlir::omp::OffloadModuleInterface>(
+                cirModule.getOperation()))
+          enableOpenMP = offloadMod.getIsOpenMP();
         std::unique_ptr<llvm::Module> llvmModule =
-            cir::direct::lowerDirectlyFromCIRToLLVMIR(cirModule, llvmContext);
+            cir::direct::lowerDirectlyFromCIRToLLVMIR(cirModule, llvmContext,
+                                                      enableOpenMP);
         if (!llvmModule)
           return mlir::failure();
         llvmModule->print(output, nullptr);
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index 51f925b17f47e..5ba66c323845a 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
     InterfaceMethod<
       /*description=*/[{
       Set the attribute on the current module with the specified boolean

>From 3eb5ebe97171062ec310da246edb65bb8489ef76 Mon Sep 17 00:00:00 2001
From: Jan Leyonberg <jan_sjodin at yahoo.com>
Date: Thu, 16 Jul 2026 12:03:44 -0400
Subject: [PATCH 3/3] Change how OpenMP is checked in the module.

---
 clang/lib/CIR/CodeGen/CIRGenModule.cpp              |  3 ++-
 clang/test/CIR/CodeGenOpenMP/omp-module-attrs.c     |  1 +
 clang/tools/cir-translate/cir-translate.cpp         |  6 ++----
 mlir/include/mlir/Dialect/OpenMP/OpenMPDialect.h    |  2 +-
 .../mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td      |  9 ---------
 .../OpenMP/{OpenMPOffloadUtils.h => OpenMPUtils.h}  | 13 +++++++++----
 6 files changed, 15 insertions(+), 19 deletions(-)
 rename mlir/include/mlir/Dialect/OpenMP/{OpenMPOffloadUtils.h => OpenMPUtils.h} (91%)

diff --git a/clang/lib/CIR/CodeGen/CIRGenModule.cpp b/clang/lib/CIR/CodeGen/CIRGenModule.cpp
index 66a9dc9ea605c..de322c261a8cc 100644
--- a/clang/lib/CIR/CodeGen/CIRGenModule.cpp
+++ b/clang/lib/CIR/CodeGen/CIRGenModule.cpp
@@ -16,7 +16,7 @@
 #include "CIRGenConstantEmitter.h"
 #include "CIRGenFunction.h"
 
-#include "mlir/Dialect/OpenMP/OpenMPOffloadUtils.h"
+#include "mlir/Dialect/OpenMP/OpenMPUtils.h"
 #include "mlir/IR/SymbolTable.h"
 #include "clang/AST/ASTContext.h"
 #include "clang/AST/ASTLambda.h"
@@ -151,6 +151,7 @@ CIRGenModule::CIRGenModule(mlir::MLIRContext &mlirContext,
         getTriple().isGPU(), langOpts.OpenMPForceUSM, langOpts.OpenMP,
         langOpts.OMPHostIRFile, langOpts.OMPTargetTriples, langOpts.NoGPULib);
     mlir::omp::setOffloadModuleInterfaceAttributes(theModule, ompOpts);
+    mlir::omp::setOpenMPVersionAttribute(theModule, langOpts.OpenMP);
   }
 
   if (langOpts.CUDA)
diff --git a/clang/test/CIR/CodeGenOpenMP/omp-module-attrs.c b/clang/test/CIR/CodeGenOpenMP/omp-module-attrs.c
index 833268b02d312..7481975a6df7e 100644
--- a/clang/test/CIR/CodeGenOpenMP/omp-module-attrs.c
+++ b/clang/test/CIR/CodeGenOpenMP/omp-module-attrs.c
@@ -7,6 +7,7 @@
 // HOST: module {{.*}} attributes {
 // HOST-SAME: omp.is_gpu = false
 // HOST-SAME: omp.is_target_device = false
+// HOST-SAME: omp.version = #omp.version<version = {{[0-9]+}}>
 
 // Host with target triples
 // RUN: %clang_cc1 -triple x86_64-unknown-linux-gnu -fopenmp -fclangir -emit-cir \
diff --git a/clang/tools/cir-translate/cir-translate.cpp b/clang/tools/cir-translate/cir-translate.cpp
index 9bbbb3939a62f..cc3b7dec291d1 100644
--- a/clang/tools/cir-translate/cir-translate.cpp
+++ b/clang/tools/cir-translate/cir-translate.cpp
@@ -14,6 +14,7 @@
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
+#include "mlir/Dialect/OpenMP/OpenMPUtils.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/MLIRContext.h"
 #include "mlir/InitAllTranslations.h"
@@ -159,10 +160,7 @@ void registerToLLVMTranslation() {
           return mlir::failure();
 
         llvm::LLVMContext llvmContext;
-        bool enableOpenMP = false;
-        if (auto offloadMod = llvm::dyn_cast<mlir::omp::OffloadModuleInterface>(
-                cirModule.getOperation()))
-          enableOpenMP = offloadMod.getIsOpenMP();
+        const bool enableOpenMP = mlir::omp::isOpenMPModule(cirModule);
         std::unique_ptr<llvm::Module> llvmModule =
             cir::direct::lowerDirectlyFromCIRToLLVMIR(cirModule, llvmContext,
                                                       enableOpenMP);
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPDialect.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPDialect.h
index c61e9415327f0..2507a5a36b514 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPDialect.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPDialect.h
@@ -17,7 +17,7 @@
 #include "mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.h"
 #include "mlir/Dialect/OpenACCMPCommon/Interfaces/OpenACCMPOpsInterfaces.h"
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
-#include "mlir/Dialect/OpenMP/OpenMPOffloadUtils.h"
+#include "mlir/Dialect/OpenMP/OpenMPUtils.h"
 #include "mlir/IR/Dialect.h"
 #include "mlir/IR/OpDefinition.h"
 #include "mlir/IR/PatternMatch.h"
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
index 5ba66c323845a..51f925b17f47e 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOpsInterfaces.td
@@ -473,15 +473,6 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
-    InterfaceMethod<
-      /*description=*/[{
-        Return true if the current module was generated with OpenMP enabled.
-        }],
-      /*retTy=*/"bool",
-      /*methodName=*/"getIsOpenMP",
-      (ins), [{}], [{
-        return $_op->hasAttr("omp.is_target_device");
-      }]>,
     InterfaceMethod<
       /*description=*/[{
       Set the attribute on the current module with the specified boolean
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h b/mlir/include/mlir/Dialect/OpenMP/OpenMPUtils.h
similarity index 91%
rename from mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h
rename to mlir/include/mlir/Dialect/OpenMP/OpenMPUtils.h
index f1f2950d2325b..002377ae6d24a 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOffloadUtils.h
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPUtils.h
@@ -1,4 +1,4 @@
-//===- OpenMPOffloadUtils.h - OpenMP offload utilities ----------*- C++ -*-===//
+//===- OpenMPUtils.h - OpenMP utilities -------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
@@ -12,8 +12,8 @@
 //
 //===----------------------------------------------------------------------===//
 
-#ifndef MLIR_DIALECT_OPENMP_OPENMPOFFLOADUTILS_H_
-#define MLIR_DIALECT_OPENMP_OPENMPOFFLOADUTILS_H_
+#ifndef MLIR_DIALECT_OPENMP_OPENMPUTILS_H_
+#define MLIR_DIALECT_OPENMP_OPENMPUTILS_H_
 
 #include "mlir/Dialect/OpenMP/OpenMPInterfaces.h"
 #include "mlir/Dialect/OpenMP/OpenMPOpsAttributes.h"
@@ -99,6 +99,11 @@ getOpenMPVersionAttribute(ModuleOp module, int64_t fallback = -1) {
   return fallback;
 }
 
+/// Returns true if the given module contains OpenMP code.
+[[maybe_unused]] static bool isOpenMPModule(ModuleOp module) {
+  return module->hasAttr("omp.version");
+}
+
 } // namespace mlir::omp
 
-#endif // MLIR_DIALECT_OPENMP_OPENMPOFFLOADUTILS_H_
+#endif // MLIR_DIALECT_OPENMP_OPENMPUTILS_H_


From llvmlistbot at llvm.org  Thu Jul 16 09:13:18 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 09:13:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
Message-ID: <llvm.org/llvm/llvm-project/pull/210090@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/210090

Add comments, unify naming, remove redundant printing hooks.


>From 1119aaf78a23e98ff63b844ca20ae1421675394b Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Thu, 16 Jul 2026 15:22:34 +0000
Subject: [PATCH] [mlir][vector][nfc] Update integration tests

Add comments, unify naming, remove redundant printing hooks.
---
 .../Dialect/Vector/CPU/compress.mlir          | 91 +++++++++++-------
 .../Dialect/Vector/CPU/expand.mlir            | 42 ++++++---
 .../Dialect/Vector/CPU/maskedload.mlir        | 24 ++++-
 .../Dialect/Vector/CPU/maskedstore.mlir       | 94 +++++++++++--------
 .../Dialect/Vector/CPU/scatter.mlir           | 71 ++++++++------
 5 files changed, 202 insertions(+), 120 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
index 1683fa504e6e2..2c8680c9fa369 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @compress16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xf32>,
                  %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @compress16(%base: memref<?xf32>,
   return
 }
 
-func.func @compress16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xf32>,
                      %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.compressstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @compress16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -55,50 +67,57 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  call @compress16(%A, %none, %value)
+  call @compress_16(%A, %none, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @compress16(%A, %all, %value)
+  call @compress_16(%A, %all, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some3, %value)
+  call @compress_16(%A, %some3, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some2, %value)
+  call @compress_16(%A, %some2, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some1, %value)
+  call @compress_16(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16_at8(%A, %some1, %value)
+  call @compress_16_at_8(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
index 8c994f83c5fdb..48f8862240776 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @expand16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16(%base: memref<?xf32>,
                %mask: vector<16xi1>,
                %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -12,7 +17,12 @@ func.func @expand16(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @expand16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16_at_8
+//
+// Same as @expand_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16_at_8(%base: memref<?xf32>,
                    %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -21,7 +31,12 @@ func.func @expand16_at8(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -41,41 +56,46 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  %e1 = call @expand16(%A, %none, %pass)
+  %e1 = call @expand_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e2 = call @expand16(%A, %all, %pass)
+  %e2 = call @expand_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e2 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %e3 = call @expand16(%A, %some1, %pass)
+  %e3 = call @expand_16(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e3 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e4 = call @expand16(%A, %some2, %pass)
+  %e4 = call @expand_16(%A, %some2, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e4 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 1, 2, -7, -7, -7, 3, -7, -7, -7, 4, -7, 5, -7, 6 )
 
-  %e5 = call @expand16(%A, %some3, %pass)
+  %e5 = call @expand_16(%A, %some3, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e5 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, -7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, -7, 5 )
@@ -83,12 +103,12 @@ func.func @entry() {
   %4 = vector.insert %v, %pass[1] : f32 into vector<16xf32>
   %5 = vector.insert %v, %4[2] : f32 into vector<16xf32>
   %alt_pass = vector.insert %v, %5[14] : f32 into vector<16xf32>
-  %e6 = call @expand16(%A, %some3, %alt_pass)
+  %e6 = call @expand_16(%A, %some3, %alt_pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e6 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 7.7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, 7.7, 5 )
 
-  %e7 = call @expand16_at8(%A, %some1, %pass)
+  %e7 = call @expand_16_at_8(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e7 : vector<16xf32>
   // CHECK-NEXT: ( 8, 9, 10, 11, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
index b9f9f8674d412..82eb6e28386d5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
@@ -1,8 +1,13 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -11,6 +16,12 @@ func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16_at_8
+//
+// Same as @maskedload_16, but the insertion index is hard-coded to 8 instead
+// of 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
                        %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -19,7 +30,12 @@ func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -38,12 +54,16 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
   %some = vector.constant_mask [8] : vector<16xi1>
   %0 = vector.insert %f, %some[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[13] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[14] : i1 into vector<16xi1>
+  // %other = [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1]
   %other = vector.insert %t, %2[14] : i1 into vector<16xi1>
 
   //
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
index 826da5309035e..38935f436d1c0 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedstore16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16(%base: memref<?xf32>,
                     %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.maskedstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @maskedstore16(%base: memref<?xf32>,
   return
 }
 
-func.func @maskedstore16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16_at_8
+//
+// Same as @maskedstore_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16_at_8(%base: memref<?xf32>,
                         %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.maskedstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @maskedstore16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %f0 = arith.constant 0.0: f32
   %c0 = arith.constant 0: index
@@ -70,34 +82,36 @@ func.func @entry() {
   vector.print %val : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A): (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %none, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %none, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %more, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0 )
+  call @maskedstore_16(%A, %more, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0]
 
-  call @maskedstore16(%A, %all, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @maskedstore_16(%A, %all, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @maskedstore16_at8(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @maskedstore_16_at_8(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
index 22b5eef12a202..5c2062303978f 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @scatter8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @scatter_8
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @scatter_8(%base: memref<?xf32>,
                %indices: vector<8xi32>,
                %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0: index
@@ -12,23 +17,25 @@ func.func @scatter8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem8(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c8 = arith.constant 8: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<8xf32>
-  %mem = scf.for %i = %c0 to %c8 step %c1
-    iter_args(%m_iter = %m) -> (vector<8xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<8xf32>
-    scf.yield %m_new : vector<8xf32>
-  }
-  vector.print %mem : vector<8xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -90,29 +97,31 @@ func.func @entry() {
   vector.print %idx : vector<8xi32>
   // CHECK: ( 7, 0, 1, 6, 2, 4, 5, 3 )
 
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %none, %val)
+  call @scatter_8(%A, %idx, %none, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %some, %val)
+  call @scatter_8(%A, %idx, %some, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 3, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 3, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %more, %val)
+  call @scatter_8(%A, %idx, %more, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 7, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 7, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %all, %val)
+  call @scatter_8(%A, %idx, %all, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 4, 7, 5, 6, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 4, 7, 5, 6, 3, 0]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)


From llvmlistbot at llvm.org  Thu Jul 16 09:13:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 09:13:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a590344.0c05e68e.b358c.f019@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-vector

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Add comments, unify naming, remove redundant printing hooks.


---

Patch is 23.64 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210090.diff


5 Files Affected:

- (modified) mlir/test/Integration/Dialect/Vector/CPU/compress.mlir (+55-36) 
- (modified) mlir/test/Integration/Dialect/Vector/CPU/expand.mlir (+31-11) 
- (modified) mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir (+22-2) 
- (modified) mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir (+54-40) 
- (modified) mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir (+40-31) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
index 1683fa504e6e2..2c8680c9fa369 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @compress16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xf32>,
                  %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @compress16(%base: memref<?xf32>,
   return
 }
 
-func.func @compress16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xf32>,
                      %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.compressstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @compress16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -55,50 +67,57 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  call @compress16(%A, %none, %value)
+  call @compress_16(%A, %none, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @compress16(%A, %all, %value)
+  call @compress_16(%A, %all, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some3, %value)
+  call @compress_16(%A, %some3, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some2, %value)
+  call @compress_16(%A, %some2, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some1, %value)
+  call @compress_16(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16_at8(%A, %some1, %value)
+  call @compress_16_at_8(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
index 8c994f83c5fdb..48f8862240776 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @expand16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16(%base: memref<?xf32>,
                %mask: vector<16xi1>,
                %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -12,7 +17,12 @@ func.func @expand16(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @expand16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16_at_8
+//
+// Same as @expand_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16_at_8(%base: memref<?xf32>,
                    %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -21,7 +31,12 @@ func.func @expand16_at8(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -41,41 +56,46 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  %e1 = call @expand16(%A, %none, %pass)
+  %e1 = call @expand_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e2 = call @expand16(%A, %all, %pass)
+  %e2 = call @expand_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e2 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %e3 = call @expand16(%A, %some1, %pass)
+  %e3 = call @expand_16(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e3 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e4 = call @expand16(%A, %some2, %pass)
+  %e4 = call @expand_16(%A, %some2, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e4 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 1, 2, -7, -7, -7, 3, -7, -7, -7, 4, -7, 5, -7, 6 )
 
-  %e5 = call @expand16(%A, %some3, %pass)
+  %e5 = call @expand_16(%A, %some3, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e5 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, -7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, -7, 5 )
@@ -83,12 +103,12 @@ func.func @entry() {
   %4 = vector.insert %v, %pass[1] : f32 into vector<16xf32>
   %5 = vector.insert %v, %4[2] : f32 into vector<16xf32>
   %alt_pass = vector.insert %v, %5[14] : f32 into vector<16xf32>
-  %e6 = call @expand16(%A, %some3, %alt_pass)
+  %e6 = call @expand_16(%A, %some3, %alt_pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e6 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 7.7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, 7.7, 5 )
 
-  %e7 = call @expand16_at8(%A, %some1, %pass)
+  %e7 = call @expand_16_at_8(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e7 : vector<16xf32>
   // CHECK-NEXT: ( 8, 9, 10, 11, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
index b9f9f8674d412..82eb6e28386d5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
@@ -1,8 +1,13 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -11,6 +16,12 @@ func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16_at_8
+//
+// Same as @maskedload_16, but the insertion index is hard-coded to 8 instead
+// of 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
                        %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -19,7 +30,12 @@ func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -38,12 +54,16 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
   %some = vector.constant_mask [8] : vector<16xi1>
   %0 = vector.insert %f, %some[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[13] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[14] : i1 into vector<16xi1>
+  // %other = [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1]
   %other = vector.insert %t, %2[14] : i1 into vector<16xi1>
 
   //
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
index 826da5309035e..38935f436d1c0 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedstore16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16(%base: memref<?xf32>,
                     %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.maskedstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @maskedstore16(%base: memref<?xf32>,
   return
 }
 
-func.func @maskedstore16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16_at_8
+//
+// Same as @maskedstore_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16_at_8(%base: memref<?xf32>,
                         %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.maskedstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @maskedstore16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %f0 = arith.constant 0.0: f32
   %c0 = arith.constant 0: index
@@ -70,34 +82,36 @@ func.func @entry() {
   vector.print %val : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A): (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %none, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %none, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %more, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0 )
+  call @maskedstore_16(%A, %more, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0]
 
-  call @maskedstore16(%A, %all, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @maskedstore_16(%A, %all, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @maskedstore16_at8(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @maskedstore_16_at_8(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
index 22b5eef12a202..5c2062303978f 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-resul...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Thu Jul 16 09:17:38 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 09:17:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a590422.1e1ac41d.15a4a3.9516@mx.google.com>


https://github.com/banach-space updated https://github.com/llvm/llvm-project/pull/210090

>From 7071fa25fd92ca1e4e836237b4d7949a21ec4ba6 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Thu, 16 Jul 2026 15:22:34 +0000
Subject: [PATCH] [mlir][vector][nfc] Update integration tests

Add comments, unify naming, remove redundant printing hooks.
---
 .../Dialect/Vector/CPU/compress.mlir          | 91 +++++++++++-------
 .../Dialect/Vector/CPU/expand.mlir            | 42 ++++++---
 .../Dialect/Vector/CPU/gather.mlir            | 26 +++--
 .../Dialect/Vector/CPU/maskedload.mlir        | 24 ++++-
 .../Dialect/Vector/CPU/maskedstore.mlir       | 94 +++++++++++--------
 .../Dialect/Vector/CPU/scatter.mlir           | 71 ++++++++------
 6 files changed, 220 insertions(+), 128 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
index 1683fa504e6e2..2c8680c9fa369 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @compress16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xf32>,
                  %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @compress16(%base: memref<?xf32>,
   return
 }
 
-func.func @compress16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xf32>,
                      %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.compressstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @compress16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -55,50 +67,57 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  call @compress16(%A, %none, %value)
+  call @compress_16(%A, %none, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @compress16(%A, %all, %value)
+  call @compress_16(%A, %all, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some3, %value)
+  call @compress_16(%A, %some3, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some2, %value)
+  call @compress_16(%A, %some2, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some1, %value)
+  call @compress_16(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16_at8(%A, %some1, %value)
+  call @compress_16_at_8(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
index 8c994f83c5fdb..48f8862240776 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @expand16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16(%base: memref<?xf32>,
                %mask: vector<16xi1>,
                %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -12,7 +17,12 @@ func.func @expand16(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @expand16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16_at_8
+//
+// Same as @expand_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16_at_8(%base: memref<?xf32>,
                    %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -21,7 +31,12 @@ func.func @expand16_at8(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -41,41 +56,46 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  %e1 = call @expand16(%A, %none, %pass)
+  %e1 = call @expand_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e2 = call @expand16(%A, %all, %pass)
+  %e2 = call @expand_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e2 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %e3 = call @expand16(%A, %some1, %pass)
+  %e3 = call @expand_16(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e3 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e4 = call @expand16(%A, %some2, %pass)
+  %e4 = call @expand_16(%A, %some2, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e4 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 1, 2, -7, -7, -7, 3, -7, -7, -7, 4, -7, 5, -7, 6 )
 
-  %e5 = call @expand16(%A, %some3, %pass)
+  %e5 = call @expand_16(%A, %some3, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e5 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, -7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, -7, 5 )
@@ -83,12 +103,12 @@ func.func @entry() {
   %4 = vector.insert %v, %pass[1] : f32 into vector<16xf32>
   %5 = vector.insert %v, %4[2] : f32 into vector<16xf32>
   %alt_pass = vector.insert %v, %5[14] : f32 into vector<16xf32>
-  %e6 = call @expand16(%A, %some3, %alt_pass)
+  %e6 = call @expand_16(%A, %some3, %alt_pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e6 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 7.7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, 7.7, 5 )
 
-  %e7 = call @expand16_at8(%A, %some1, %pass)
+  %e7 = call @expand_16_at_8(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e7 : vector<16xf32>
   // CHECK-NEXT: ( 8, 9, 10, 11, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 110a46e2d89d8..078826522bcb5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
-// DEFINE: %{entry_point} = main
-// DEFINE: %{run} = mlir-runner -e entry -entry-point-result=void \
+// DEFINE: %{main_point} = main
+// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)
@@ -16,7 +16,12 @@
 // REDEFINE: %{compile} = mlir-opt %s --test-vector-gather-lowering
 // RUN: %{compile} | FileCheck %s -check-prefix CHECK-IR 
 
-func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
+//===----------------------------------------------------------------------===//
+// @gather_8
+//
+// Gather 8 elements
+//===----------------------------------------------------------------------===//
+func.func @gather_8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
               %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
   %c0 = arith.constant 0: index
   /// Verify that the lowering via vector.load does indeed generate vector.load
@@ -26,7 +31,12 @@ func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
   return %g : vector<8xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -78,25 +88,25 @@ func.func @entry() {
   // Gather tests.
   //
 
-  %g1 = call @gather8(%A, %idx, %all, %pass)
+  %g1 = call @gather_8(%A, %idx, %all, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g1 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, 10, 84, 34, 42 )
 
-  %g2 = call @gather8(%A, %idx, %none, %pass)
+  %g2 = call @gather_8(%A, %idx, %none, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g2 : vector<8xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %g3 = call @gather8(%A, %idx, %some, %pass)
+  %g3 = call @gather_8(%A, %idx, %some, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g3 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, -7, -7, -7, -7 )
 
-  %g4 = call @gather8(%A, %idx, %more, %pass)
+  %g4 = call @gather_8(%A, %idx, %more, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g4 : vector<8xf32>
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
index b9f9f8674d412..82eb6e28386d5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
@@ -1,8 +1,13 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -11,6 +16,12 @@ func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
+//===----------------------------------------------------------------------===//
+// @maskedload_16_at_8
+//
+// Same as @maskedload_16, but the insertion index is hard-coded to 8 instead
+// of 0
+//===----------------------------------------------------------------------===//
 func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
                        %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -19,7 +30,12 @@ func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -38,12 +54,16 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
   %some = vector.constant_mask [8] : vector<16xi1>
   %0 = vector.insert %f, %some[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[13] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[14] : i1 into vector<16xi1>
+  // %other = [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1]
   %other = vector.insert %t, %2[14] : i1 into vector<16xi1>
 
   //
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
index 826da5309035e..38935f436d1c0 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedstore16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16(%base: memref<?xf32>,
                     %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.maskedstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @maskedstore16(%base: memref<?xf32>,
   return
 }
 
-func.func @maskedstore16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16_at_8
+//
+// Same as @maskedstore_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16_at_8(%base: memref<?xf32>,
                         %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.maskedstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @maskedstore16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %f0 = arith.constant 0.0: f32
   %c0 = arith.constant 0: index
@@ -70,34 +82,36 @@ func.func @entry() {
   vector.print %val : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A): (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %none, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %none, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %more, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0 )
+  call @maskedstore_16(%A, %more, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0]
 
-  call @maskedstore16(%A, %all, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @maskedstore_16(%A, %all, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @maskedstore16_at8(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @maskedstore_16_at_8(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
index 22b5eef12a202..1f5410716d05a 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @scatter8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @scatter_8
+//
+// Scatter 8 elements
+//===----------------------------------------------------------------------===//
+func.func @scatter_8(%base: memref<?xf32>,
                %indices: vector<8xi32>,
                %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0: index
@@ -12,23 +17,25 @@ func.func @scatter8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem8(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c8 = arith.constant 8: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<8xf32>
-  %mem = scf.for %i = %c0 to %c8 step %c1
-    iter_args(%m_iter = %m) -> (vector<8xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<8xf32>
-    scf.yield %m_new : vector<8xf32>
-  }
-  vector.print %mem : vector<8xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -90,29 +97,31 @@ func.func @entry() {
   vector.print %idx : vector<8xi32>
   // CHECK: ( 7, 0, 1, 6, 2, 4, 5, 3 )
 
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %none, %val)
+  call @scatter_8(%A, %idx, %none, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %some, %val)
+  call @scatter_8(%A, %idx, %some, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 3, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 3, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %more, %val)
+  call @scatter_8(%A, %idx, %more, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 7, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 7, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %all, %val)
+  call @scatter_8(%A, %idx, %all, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 4, 7, 5, 6, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 4, 7, 5, 6, 3, 0]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)


From llvmlistbot at llvm.org  Thu Jul 16 09:18:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 09:18:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a590471.c5504292.a1395.b16c@mx.google.com>


banach-space wrote:

These are merely editorial changes and I am happy to land it without a review. I will till tomorrow should anyone decide to leave comments.

https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Thu Jul 16 09:20:29 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 09:20:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a5904cd.e624eb8d.3065e1.5814@mx.google.com>


https://github.com/banach-space updated https://github.com/llvm/llvm-project/pull/210090

>From 3e73efb38971459c1a89ce4d9378c8269c5fda37 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Thu, 16 Jul 2026 15:22:34 +0000
Subject: [PATCH] [mlir][vector][nfc] Update integration tests

Add comments, unify naming, remove redundant printing hooks.
---
 .../Dialect/Vector/CPU/compress.mlir          | 91 +++++++++++-------
 .../Dialect/Vector/CPU/expand.mlir            | 42 ++++++---
 .../Dialect/Vector/CPU/gather.mlir            | 26 +++--
 .../Dialect/Vector/CPU/maskedload.mlir        | 38 ++++++--
 .../Dialect/Vector/CPU/maskedstore.mlir       | 94 +++++++++++--------
 .../Dialect/Vector/CPU/scatter.mlir           | 71 ++++++++------
 6 files changed, 227 insertions(+), 135 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
index 1683fa504e6e2..2c8680c9fa369 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @compress16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xf32>,
                  %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @compress16(%base: memref<?xf32>,
   return
 }
 
-func.func @compress16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xf32>,
                      %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.compressstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @compress16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -55,50 +67,57 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  call @compress16(%A, %none, %value)
+  call @compress_16(%A, %none, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @compress16(%A, %all, %value)
+  call @compress_16(%A, %all, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some3, %value)
+  call @compress_16(%A, %some3, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some2, %value)
+  call @compress_16(%A, %some2, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some1, %value)
+  call @compress_16(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16_at8(%A, %some1, %value)
+  call @compress_16_at_8(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
index 8c994f83c5fdb..48f8862240776 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @expand16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16(%base: memref<?xf32>,
                %mask: vector<16xi1>,
                %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -12,7 +17,12 @@ func.func @expand16(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @expand16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16_at_8
+//
+// Same as @expand_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16_at_8(%base: memref<?xf32>,
                    %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -21,7 +31,12 @@ func.func @expand16_at8(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -41,41 +56,46 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  %e1 = call @expand16(%A, %none, %pass)
+  %e1 = call @expand_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e2 = call @expand16(%A, %all, %pass)
+  %e2 = call @expand_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e2 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %e3 = call @expand16(%A, %some1, %pass)
+  %e3 = call @expand_16(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e3 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e4 = call @expand16(%A, %some2, %pass)
+  %e4 = call @expand_16(%A, %some2, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e4 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 1, 2, -7, -7, -7, 3, -7, -7, -7, 4, -7, 5, -7, 6 )
 
-  %e5 = call @expand16(%A, %some3, %pass)
+  %e5 = call @expand_16(%A, %some3, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e5 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, -7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, -7, 5 )
@@ -83,12 +103,12 @@ func.func @entry() {
   %4 = vector.insert %v, %pass[1] : f32 into vector<16xf32>
   %5 = vector.insert %v, %4[2] : f32 into vector<16xf32>
   %alt_pass = vector.insert %v, %5[14] : f32 into vector<16xf32>
-  %e6 = call @expand16(%A, %some3, %alt_pass)
+  %e6 = call @expand_16(%A, %some3, %alt_pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e6 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 7.7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, 7.7, 5 )
 
-  %e7 = call @expand16_at8(%A, %some1, %pass)
+  %e7 = call @expand_16_at_8(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e7 : vector<16xf32>
   // CHECK-NEXT: ( 8, 9, 10, 11, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 110a46e2d89d8..078826522bcb5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
-// DEFINE: %{entry_point} = main
-// DEFINE: %{run} = mlir-runner -e entry -entry-point-result=void \
+// DEFINE: %{main_point} = main
+// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)
@@ -16,7 +16,12 @@
 // REDEFINE: %{compile} = mlir-opt %s --test-vector-gather-lowering
 // RUN: %{compile} | FileCheck %s -check-prefix CHECK-IR 
 
-func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
+//===----------------------------------------------------------------------===//
+// @gather_8
+//
+// Gather 8 elements
+//===----------------------------------------------------------------------===//
+func.func @gather_8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
               %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
   %c0 = arith.constant 0: index
   /// Verify that the lowering via vector.load does indeed generate vector.load
@@ -26,7 +31,12 @@ func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
   return %g : vector<8xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -78,25 +88,25 @@ func.func @entry() {
   // Gather tests.
   //
 
-  %g1 = call @gather8(%A, %idx, %all, %pass)
+  %g1 = call @gather_8(%A, %idx, %all, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g1 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, 10, 84, 34, 42 )
 
-  %g2 = call @gather8(%A, %idx, %none, %pass)
+  %g2 = call @gather_8(%A, %idx, %none, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g2 : vector<8xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %g3 = call @gather8(%A, %idx, %some, %pass)
+  %g3 = call @gather_8(%A, %idx, %some, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g3 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, -7, -7, -7, -7 )
 
-  %g4 = call @gather8(%A, %idx, %more, %pass)
+  %g4 = call @gather_8(%A, %idx, %more, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g4 : vector<8xf32>
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
index b9f9f8674d412..ce06dbbbf4cac 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
+//===----------------------------------------------------------------------===//
+// @maskedload_16
+//
+// Load 16 elements. Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedload_16(%base: memref<?xf32>, %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
   %ld = vector.maskedload %base[%c0], %mask, %pass_thru
@@ -11,7 +16,13 @@ func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
+//===----------------------------------------------------------------------===//
+// @maskedload_16_at_8
+//
+// Same as @maskedload_16, but the insertion index is hard-coded to 8 instead
+// of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedload_16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
                        %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
   %ld = vector.maskedload %base[%c8], %mask, %pass_thru
@@ -19,7 +30,12 @@ func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -38,39 +54,43 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
   %some = vector.constant_mask [8] : vector<16xi1>
   %0 = vector.insert %f, %some[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[13] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[14] : i1 into vector<16xi1>
+  // %other = [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1]
   %other = vector.insert %t, %2[14] : i1 into vector<16xi1>
 
   //
   // Masked load tests.
   //
 
-  %l1 = call @maskedload16(%A, %none, %pass)
+  %l1 = call @maskedload_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %l2 = call @maskedload16(%A, %all, %pass)
+  %l2 = call @maskedload_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l2 : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %l3 = call @maskedload16(%A, %some, %pass)
+  %l3 = call @maskedload_16(%A, %some, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l3 : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %l4 = call @maskedload16(%A, %other, %pass)
+  %l4 = call @maskedload_16(%A, %other, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l4 : vector<16xf32>
   // CHECK: ( -7, 1, 2, 3, 4, 5, 6, 7, -7, -7, -7, -7, -7, 13, 14, -7 )
 
-  %l5 = call @maskedload16_at8(%A, %some, %pass)
+  %l5 = call @maskedload_16_at8(%A, %some, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l5 : vector<16xf32>
   // CHECK: ( 8, 9, 10, 11, 12, 13, 14, 15, -7, -7, -7, -7, -7, -7, -7, -7 )
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
index 826da5309035e..b4d3b2ac5d6e4 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedstore16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16
+//
+// Store 16 elements. Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16(%base: memref<?xf32>,
                     %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.maskedstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @maskedstore16(%base: memref<?xf32>,
   return
 }
 
-func.func @maskedstore16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16_at_8
+//
+// Same as @maskedstore_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16_at_8(%base: memref<?xf32>,
                         %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.maskedstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @maskedstore16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %f0 = arith.constant 0.0: f32
   %c0 = arith.constant 0: index
@@ -70,34 +82,36 @@ func.func @entry() {
   vector.print %val : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A): (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %none, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %none, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %more, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0 )
+  call @maskedstore_16(%A, %more, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0]
 
-  call @maskedstore16(%A, %all, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @maskedstore_16(%A, %all, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @maskedstore16_at8(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @maskedstore_16_at_8(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
index 22b5eef12a202..1f5410716d05a 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @scatter8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @scatter_8
+//
+// Scatter 8 elements
+//===----------------------------------------------------------------------===//
+func.func @scatter_8(%base: memref<?xf32>,
                %indices: vector<8xi32>,
                %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0: index
@@ -12,23 +17,25 @@ func.func @scatter8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem8(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c8 = arith.constant 8: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<8xf32>
-  %mem = scf.for %i = %c0 to %c8 step %c1
-    iter_args(%m_iter = %m) -> (vector<8xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<8xf32>
-    scf.yield %m_new : vector<8xf32>
-  }
-  vector.print %mem : vector<8xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -90,29 +97,31 @@ func.func @entry() {
   vector.print %idx : vector<8xi32>
   // CHECK: ( 7, 0, 1, 6, 2, 4, 5, 3 )
 
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %none, %val)
+  call @scatter_8(%A, %idx, %none, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %some, %val)
+  call @scatter_8(%A, %idx, %some, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 3, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 3, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %more, %val)
+  call @scatter_8(%A, %idx, %more, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 7, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 7, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %all, %val)
+  call @scatter_8(%A, %idx, %all, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 4, 7, 5, 6, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 4, 7, 5, 6, 3, 0]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)


From llvmlistbot at llvm.org  Thu Jul 16 09:41:31 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Thu, 16 Jul 2026 09:41:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
Message-ID: <llvm.org/llvm/llvm-project/pull/210105@github.com>


https://github.com/secona created https://github.com/llvm/llvm-project/pull/210105

Fixes #205228

This change fixes a use-after-free bug in the BufferDeallocationSimplification pipeline caused by the greedy pattern rewriter deleting operation tracked by the BufferOriginAnalysis. BufferViewFlowAnalysis (internally used by BufferOriginAnalysis) creates the dependencies map once during initialization. In this specific case, the folder removed the `memref.cast` ops making later uses a use-after-free bug.

Below is the snippet of running the pass using `-debug-only=greedy-rewriter`. It erases a `memref.cast` and crashes in RemoveDeallocMemrefsContainedInRetained.

```
...
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] Processing operation : 'memref.cast'(0x654d43d18670) {
[greedy-rewriter:1]   %4 = "memref.cast"(%3) : (memref<?x?xf32, 1>) -> memref<4x4xf32, 1>
[greedy-rewriter:1] 
[greedy-rewriter:1] } -> success : operation was folded
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] ** Replace : 'memref.cast'(0x654d43d18670)
[greedy-rewriter:1] ** Modified: 'scf.yield'(0x654d43c97bb0)
[greedy-rewriter:1] ** Erase   : 'memref.cast'(0x654d43d18670)
...
[greedy-rewriter:1] Processing operation : 'bufferization.dealloc'(0x654d43cbd7d0) {
[greedy-rewriter:1]   %5 = "bufferization.dealloc"(%4#0, %1, %3) <{operandSegmentSizes = array<i32: 1, 1, 1>}> : (memref<f32, 1>, i1, memref<4x4xf32, 1>) -> i1
[greedy-rewriter:1] 
[greedy-rewriter:1] 
[greedy-rewriter:1]   * Pattern (anonymous namespace)::RemoveDeallocMemrefsContainedInRetained : 'bufferization.dealloc -> ()' {
```

The fix is by recreating BufferOriginAnalysis on-the-fly inside matchAndRewrite of each rewrite patterns, where it was previously created once in BufferDeallocationSimplificationPass::runOnOperation.

>From 25ecc63471781c47bcc016802e4c596d32d90fa0 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Thu, 16 Jul 2026 23:09:37 +0700
Subject: [PATCH] [mlir][bufferization] Fix stale BufferOriginAnalysis in
 BufferDeallocationSimplification

---
 .../BufferDeallocationSimplification.cpp      | 57 +++++++++----------
 .../buffer-deallocation-simplification.mlir   | 29 ++++++++++
 2 files changed, 56 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..64b8f6d1fbfa8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -46,6 +46,14 @@ static Value getViewBase(Value value) {
   return value;
 }
 
+static BufferOriginAnalysis getBufferOriginAnalysis(Operation *op) {
+  // dataflow boundary; we want to the origin of all internral buffers.
+  Operation *parent = op->getParentWithTrait<OpTrait::IsIsolatedFromAbove>();
+  assert(parent && "expected dealloc inside an IsIsolatedFromAbove region");
+  BufferOriginAnalysis analysis(parent);
+  return analysis;
+}
+
 static LogicalResult updateDeallocIfChanged(DeallocOp deallocOp,
                                             ValueRange memrefs,
                                             ValueRange conditions,
@@ -134,9 +142,8 @@ namespace {
 /// given that `%r0` and `%r1` may not alias with `%m0`.
 struct RemoveDeallocMemrefsContainedInRetained
     : public OpRewritePattern<DeallocOp> {
-  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context,
-                                          BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   /// The passed 'memref' must not have a may-alias relation to any retained
   /// memref, and at least one must-alias relation. If there is no must-aliasing
@@ -145,7 +152,8 @@ struct RemoveDeallocMemrefsContainedInRetained
   /// no-alias, then just proceed, if it's must-alias we need to update the
   /// updated condition returned by the dealloc operation for that alias.
   LogicalResult handleOneMemref(DeallocOp deallocOp, Value memref, Value cond,
-                                PatternRewriter &rewriter) const {
+                                PatternRewriter &rewriter,
+                                BufferOriginAnalysis analysis) const {
     rewriter.setInsertionPointAfter(deallocOp);
 
     // Check that there is no may-aliasing memref and that at least one memref
@@ -183,6 +191,8 @@ struct RemoveDeallocMemrefsContainedInRetained
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
+
     // There must not be any duplicates in the retain list anymore because we
     // would miss updating one of the result values otherwise.
     DenseSet<Value> retained(deallocOp.getRetained().begin(),
@@ -194,13 +204,14 @@ struct RemoveDeallocMemrefsContainedInRetained
     for (auto [memref, cond] :
          llvm::zip(deallocOp.getMemrefs(), deallocOp.getConditions())) {
 
-      if (succeeded(handleOneMemref(deallocOp, memref, cond, rewriter)))
+      if (succeeded(
+              handleOneMemref(deallocOp, memref, cond, rewriter, analysis)))
         continue;
 
       if (auto extractOp =
               memref.getDefiningOp<memref::ExtractStridedMetadataOp>())
         if (succeeded(handleOneMemref(deallocOp, extractOp.getOperand(), cond,
-                                      rewriter)))
+                                      rewriter, analysis)))
           continue;
 
       newMemrefs.push_back(memref);
@@ -212,9 +223,6 @@ struct RemoveDeallocMemrefsContainedInRetained
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Remove memrefs from the `retained` list which are guaranteed to not alias
@@ -235,12 +243,12 @@ struct RemoveDeallocMemrefsContainedInRetained
 /// ```
 struct RemoveRetainedMemrefsGuaranteedToNotAlias
     : public OpRewritePattern<DeallocOp> {
-  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context,
-                                            BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     SmallVector<Value> newRetainedMemrefs, replacements;
 
     for (auto retainedMemref : deallocOp.getRetained()) {
@@ -270,9 +278,6 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Split off memrefs to separate dealloc operations to reduce the number of
@@ -304,12 +309,12 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
 /// ```
 struct SplitDeallocWhenNotAliasingAnyOther
     : public OpRewritePattern<DeallocOp> {
-  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context,
-                                      BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     Location loc = deallocOp.getLoc();
     if (deallocOp.getMemrefs().size() <= 1)
       return failure();
@@ -359,9 +364,6 @@ struct SplitDeallocWhenNotAliasingAnyOther
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Check for every retained memref if a must-aliasing memref exists in the
@@ -389,12 +391,12 @@ struct SplitDeallocWhenNotAliasingAnyOther
 /// don't have uses anymore.
 struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     : public OpRewritePattern<DeallocOp> {
-  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context,
-                                                BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     BitVector aliasesWithConstTrueMemref(deallocOp.getRetained().size());
     SmallVector<Value> newMemrefs, newConditions;
     for (auto [memref, cond] :
@@ -441,9 +443,6 @@ struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 } // namespace
@@ -461,13 +460,11 @@ struct BufferDeallocationSimplificationPass
     : public bufferization::impl::BufferDeallocationSimplificationPassBase<
           BufferDeallocationSimplificationPass> {
   void runOnOperation() override {
-    BufferOriginAnalysis analysis(getOperation());
     RewritePatternSet patterns(&getContext());
     patterns.add<RemoveDeallocMemrefsContainedInRetained,
                  RemoveRetainedMemrefsGuaranteedToNotAlias,
                  SplitDeallocWhenNotAliasingAnyOther,
-                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
-                                                                analysis);
+                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext());
 
     populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
diff --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)


From llvmlistbot at llvm.org  Thu Jul 16 09:42:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 09:42:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5909dd.9edb5c9d.5e50a.5b1d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Vito Secona (secona)

<details>
<summary>Changes</summary>

Fixes #<!-- -->205228

This change fixes a use-after-free bug in the BufferDeallocationSimplification pipeline caused by the greedy pattern rewriter deleting operation tracked by the BufferOriginAnalysis. BufferViewFlowAnalysis (internally used by BufferOriginAnalysis) creates the dependencies map once during initialization. In this specific case, the folder removed the `memref.cast` ops making later uses a use-after-free bug.

Below is the snippet of running the pass using `-debug-only=greedy-rewriter`. It erases a `memref.cast` and crashes in RemoveDeallocMemrefsContainedInRetained.

```
...
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] Processing operation : 'memref.cast'(0x654d43d18670) {
[greedy-rewriter:1]   %4 = "memref.cast"(%3) : (memref&lt;?x?xf32, 1&gt;) -&gt; memref&lt;4x4xf32, 1&gt;
[greedy-rewriter:1] 
[greedy-rewriter:1] } -&gt; success : operation was folded
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] ** Replace : 'memref.cast'(0x654d43d18670)
[greedy-rewriter:1] ** Modified: 'scf.yield'(0x654d43c97bb0)
[greedy-rewriter:1] ** Erase   : 'memref.cast'(0x654d43d18670)
...
[greedy-rewriter:1] Processing operation : 'bufferization.dealloc'(0x654d43cbd7d0) {
[greedy-rewriter:1]   %5 = "bufferization.dealloc"(%4#<!-- -->0, %1, %3) &lt;{operandSegmentSizes = array&lt;i32: 1, 1, 1&gt;}&gt; : (memref&lt;f32, 1&gt;, i1, memref&lt;4x4xf32, 1&gt;) -&gt; i1
[greedy-rewriter:1] 
[greedy-rewriter:1] 
[greedy-rewriter:1]   * Pattern (anonymous namespace)::RemoveDeallocMemrefsContainedInRetained : 'bufferization.dealloc -&gt; ()' {
```

The fix is by recreating BufferOriginAnalysis on-the-fly inside matchAndRewrite of each rewrite patterns, where it was previously created once in BufferDeallocationSimplificationPass::runOnOperation.

---
Full diff: https://github.com/llvm/llvm-project/pull/210105.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp (+27-30) 
- (modified) mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir (+29) 


``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..64b8f6d1fbfa8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -46,6 +46,14 @@ static Value getViewBase(Value value) {
   return value;
 }
 
+static BufferOriginAnalysis getBufferOriginAnalysis(Operation *op) {
+  // dataflow boundary; we want to the origin of all internral buffers.
+  Operation *parent = op->getParentWithTrait<OpTrait::IsIsolatedFromAbove>();
+  assert(parent && "expected dealloc inside an IsIsolatedFromAbove region");
+  BufferOriginAnalysis analysis(parent);
+  return analysis;
+}
+
 static LogicalResult updateDeallocIfChanged(DeallocOp deallocOp,
                                             ValueRange memrefs,
                                             ValueRange conditions,
@@ -134,9 +142,8 @@ namespace {
 /// given that `%r0` and `%r1` may not alias with `%m0`.
 struct RemoveDeallocMemrefsContainedInRetained
     : public OpRewritePattern<DeallocOp> {
-  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context,
-                                          BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   /// The passed 'memref' must not have a may-alias relation to any retained
   /// memref, and at least one must-alias relation. If there is no must-aliasing
@@ -145,7 +152,8 @@ struct RemoveDeallocMemrefsContainedInRetained
   /// no-alias, then just proceed, if it's must-alias we need to update the
   /// updated condition returned by the dealloc operation for that alias.
   LogicalResult handleOneMemref(DeallocOp deallocOp, Value memref, Value cond,
-                                PatternRewriter &rewriter) const {
+                                PatternRewriter &rewriter,
+                                BufferOriginAnalysis analysis) const {
     rewriter.setInsertionPointAfter(deallocOp);
 
     // Check that there is no may-aliasing memref and that at least one memref
@@ -183,6 +191,8 @@ struct RemoveDeallocMemrefsContainedInRetained
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
+
     // There must not be any duplicates in the retain list anymore because we
     // would miss updating one of the result values otherwise.
     DenseSet<Value> retained(deallocOp.getRetained().begin(),
@@ -194,13 +204,14 @@ struct RemoveDeallocMemrefsContainedInRetained
     for (auto [memref, cond] :
          llvm::zip(deallocOp.getMemrefs(), deallocOp.getConditions())) {
 
-      if (succeeded(handleOneMemref(deallocOp, memref, cond, rewriter)))
+      if (succeeded(
+              handleOneMemref(deallocOp, memref, cond, rewriter, analysis)))
         continue;
 
       if (auto extractOp =
               memref.getDefiningOp<memref::ExtractStridedMetadataOp>())
         if (succeeded(handleOneMemref(deallocOp, extractOp.getOperand(), cond,
-                                      rewriter)))
+                                      rewriter, analysis)))
           continue;
 
       newMemrefs.push_back(memref);
@@ -212,9 +223,6 @@ struct RemoveDeallocMemrefsContainedInRetained
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Remove memrefs from the `retained` list which are guaranteed to not alias
@@ -235,12 +243,12 @@ struct RemoveDeallocMemrefsContainedInRetained
 /// ```
 struct RemoveRetainedMemrefsGuaranteedToNotAlias
     : public OpRewritePattern<DeallocOp> {
-  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context,
-                                            BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     SmallVector<Value> newRetainedMemrefs, replacements;
 
     for (auto retainedMemref : deallocOp.getRetained()) {
@@ -270,9 +278,6 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Split off memrefs to separate dealloc operations to reduce the number of
@@ -304,12 +309,12 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
 /// ```
 struct SplitDeallocWhenNotAliasingAnyOther
     : public OpRewritePattern<DeallocOp> {
-  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context,
-                                      BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     Location loc = deallocOp.getLoc();
     if (deallocOp.getMemrefs().size() <= 1)
       return failure();
@@ -359,9 +364,6 @@ struct SplitDeallocWhenNotAliasingAnyOther
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Check for every retained memref if a must-aliasing memref exists in the
@@ -389,12 +391,12 @@ struct SplitDeallocWhenNotAliasingAnyOther
 /// don't have uses anymore.
 struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     : public OpRewritePattern<DeallocOp> {
-  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context,
-                                                BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     BitVector aliasesWithConstTrueMemref(deallocOp.getRetained().size());
     SmallVector<Value> newMemrefs, newConditions;
     for (auto [memref, cond] :
@@ -441,9 +443,6 @@ struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 } // namespace
@@ -461,13 +460,11 @@ struct BufferDeallocationSimplificationPass
     : public bufferization::impl::BufferDeallocationSimplificationPassBase<
           BufferDeallocationSimplificationPass> {
   void runOnOperation() override {
-    BufferOriginAnalysis analysis(getOperation());
     RewritePatternSet patterns(&getContext());
     patterns.add<RemoveDeallocMemrefsContainedInRetained,
                  RemoveRetainedMemrefsGuaranteedToNotAlias,
                  SplitDeallocWhenNotAliasingAnyOther,
-                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
-                                                                analysis);
+                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext());
 
     populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
diff --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)

``````````

</details>


https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Thu Jul 16 09:42:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 09:42:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5909de.b4a331f0.261d49.3e24@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-bufferization

Author: Vito Secona (secona)

<details>
<summary>Changes</summary>

Fixes #<!-- -->205228

This change fixes a use-after-free bug in the BufferDeallocationSimplification pipeline caused by the greedy pattern rewriter deleting operation tracked by the BufferOriginAnalysis. BufferViewFlowAnalysis (internally used by BufferOriginAnalysis) creates the dependencies map once during initialization. In this specific case, the folder removed the `memref.cast` ops making later uses a use-after-free bug.

Below is the snippet of running the pass using `-debug-only=greedy-rewriter`. It erases a `memref.cast` and crashes in RemoveDeallocMemrefsContainedInRetained.

```
...
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] Processing operation : 'memref.cast'(0x654d43d18670) {
[greedy-rewriter:1]   %4 = "memref.cast"(%3) : (memref&lt;?x?xf32, 1&gt;) -&gt; memref&lt;4x4xf32, 1&gt;
[greedy-rewriter:1] 
[greedy-rewriter:1] } -&gt; success : operation was folded
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] ** Replace : 'memref.cast'(0x654d43d18670)
[greedy-rewriter:1] ** Modified: 'scf.yield'(0x654d43c97bb0)
[greedy-rewriter:1] ** Erase   : 'memref.cast'(0x654d43d18670)
...
[greedy-rewriter:1] Processing operation : 'bufferization.dealloc'(0x654d43cbd7d0) {
[greedy-rewriter:1]   %5 = "bufferization.dealloc"(%4#<!-- -->0, %1, %3) &lt;{operandSegmentSizes = array&lt;i32: 1, 1, 1&gt;}&gt; : (memref&lt;f32, 1&gt;, i1, memref&lt;4x4xf32, 1&gt;) -&gt; i1
[greedy-rewriter:1] 
[greedy-rewriter:1] 
[greedy-rewriter:1]   * Pattern (anonymous namespace)::RemoveDeallocMemrefsContainedInRetained : 'bufferization.dealloc -&gt; ()' {
```

The fix is by recreating BufferOriginAnalysis on-the-fly inside matchAndRewrite of each rewrite patterns, where it was previously created once in BufferDeallocationSimplificationPass::runOnOperation.

---
Full diff: https://github.com/llvm/llvm-project/pull/210105.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp (+27-30) 
- (modified) mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir (+29) 


``````````diff
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..64b8f6d1fbfa8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -46,6 +46,14 @@ static Value getViewBase(Value value) {
   return value;
 }
 
+static BufferOriginAnalysis getBufferOriginAnalysis(Operation *op) {
+  // dataflow boundary; we want to the origin of all internral buffers.
+  Operation *parent = op->getParentWithTrait<OpTrait::IsIsolatedFromAbove>();
+  assert(parent && "expected dealloc inside an IsIsolatedFromAbove region");
+  BufferOriginAnalysis analysis(parent);
+  return analysis;
+}
+
 static LogicalResult updateDeallocIfChanged(DeallocOp deallocOp,
                                             ValueRange memrefs,
                                             ValueRange conditions,
@@ -134,9 +142,8 @@ namespace {
 /// given that `%r0` and `%r1` may not alias with `%m0`.
 struct RemoveDeallocMemrefsContainedInRetained
     : public OpRewritePattern<DeallocOp> {
-  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context,
-                                          BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveDeallocMemrefsContainedInRetained(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   /// The passed 'memref' must not have a may-alias relation to any retained
   /// memref, and at least one must-alias relation. If there is no must-aliasing
@@ -145,7 +152,8 @@ struct RemoveDeallocMemrefsContainedInRetained
   /// no-alias, then just proceed, if it's must-alias we need to update the
   /// updated condition returned by the dealloc operation for that alias.
   LogicalResult handleOneMemref(DeallocOp deallocOp, Value memref, Value cond,
-                                PatternRewriter &rewriter) const {
+                                PatternRewriter &rewriter,
+                                BufferOriginAnalysis analysis) const {
     rewriter.setInsertionPointAfter(deallocOp);
 
     // Check that there is no may-aliasing memref and that at least one memref
@@ -183,6 +191,8 @@ struct RemoveDeallocMemrefsContainedInRetained
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
+
     // There must not be any duplicates in the retain list anymore because we
     // would miss updating one of the result values otherwise.
     DenseSet<Value> retained(deallocOp.getRetained().begin(),
@@ -194,13 +204,14 @@ struct RemoveDeallocMemrefsContainedInRetained
     for (auto [memref, cond] :
          llvm::zip(deallocOp.getMemrefs(), deallocOp.getConditions())) {
 
-      if (succeeded(handleOneMemref(deallocOp, memref, cond, rewriter)))
+      if (succeeded(
+              handleOneMemref(deallocOp, memref, cond, rewriter, analysis)))
         continue;
 
       if (auto extractOp =
               memref.getDefiningOp<memref::ExtractStridedMetadataOp>())
         if (succeeded(handleOneMemref(deallocOp, extractOp.getOperand(), cond,
-                                      rewriter)))
+                                      rewriter, analysis)))
           continue;
 
       newMemrefs.push_back(memref);
@@ -212,9 +223,6 @@ struct RemoveDeallocMemrefsContainedInRetained
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Remove memrefs from the `retained` list which are guaranteed to not alias
@@ -235,12 +243,12 @@ struct RemoveDeallocMemrefsContainedInRetained
 /// ```
 struct RemoveRetainedMemrefsGuaranteedToNotAlias
     : public OpRewritePattern<DeallocOp> {
-  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context,
-                                            BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RemoveRetainedMemrefsGuaranteedToNotAlias(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     SmallVector<Value> newRetainedMemrefs, replacements;
 
     for (auto retainedMemref : deallocOp.getRetained()) {
@@ -270,9 +278,6 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Split off memrefs to separate dealloc operations to reduce the number of
@@ -304,12 +309,12 @@ struct RemoveRetainedMemrefsGuaranteedToNotAlias
 /// ```
 struct SplitDeallocWhenNotAliasingAnyOther
     : public OpRewritePattern<DeallocOp> {
-  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context,
-                                      BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  SplitDeallocWhenNotAliasingAnyOther(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     Location loc = deallocOp.getLoc();
     if (deallocOp.getMemrefs().size() <= 1)
       return failure();
@@ -359,9 +364,6 @@ struct SplitDeallocWhenNotAliasingAnyOther
     rewriter.replaceOp(deallocOp, replacements);
     return success();
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 /// Check for every retained memref if a must-aliasing memref exists in the
@@ -389,12 +391,12 @@ struct SplitDeallocWhenNotAliasingAnyOther
 /// don't have uses anymore.
 struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     : public OpRewritePattern<DeallocOp> {
-  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context,
-                                                BufferOriginAnalysis &analysis)
-      : OpRewritePattern<DeallocOp>(context), analysis(analysis) {}
+  RetainedMemrefAliasingAlwaysDeallocatedMemref(MLIRContext *context)
+      : OpRewritePattern<DeallocOp>(context) {}
 
   LogicalResult matchAndRewrite(DeallocOp deallocOp,
                                 PatternRewriter &rewriter) const override {
+    auto analysis = getBufferOriginAnalysis(deallocOp);
     BitVector aliasesWithConstTrueMemref(deallocOp.getRetained().size());
     SmallVector<Value> newMemrefs, newConditions;
     for (auto [memref, cond] :
@@ -441,9 +443,6 @@ struct RetainedMemrefAliasingAlwaysDeallocatedMemref
     return updateDeallocIfChanged(deallocOp, newMemrefs, newConditions,
                                   rewriter);
   }
-
-private:
-  BufferOriginAnalysis &analysis;
 };
 
 } // namespace
@@ -461,13 +460,11 @@ struct BufferDeallocationSimplificationPass
     : public bufferization::impl::BufferDeallocationSimplificationPassBase<
           BufferDeallocationSimplificationPass> {
   void runOnOperation() override {
-    BufferOriginAnalysis analysis(getOperation());
     RewritePatternSet patterns(&getContext());
     patterns.add<RemoveDeallocMemrefsContainedInRetained,
                  RemoveRetainedMemrefsGuaranteedToNotAlias,
                  SplitDeallocWhenNotAliasingAnyOther,
-                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
-                                                                analysis);
+                 RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext());
 
     populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
diff --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)

``````````

</details>


https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Thu Jul 16 09:43:58 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Thu, 16 Jul 2026 09:43:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a590a4e.4987c784.a8336.a143@mx.google.com>


Devjiu wrote:

@joker-eph Sorry for pinging, but could you please check if it's fine to reduce the scope as I proposed 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Thu Jul 16 09:53:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 09:53:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Openacc bootstrap (PR #209691)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209691@github.com>
Message-ID: <6a590c93.c5504292.a1395.e3ed@mx.google.com>


https://github.com/wyzfighter closed https://github.com/llvm/llvm-project/pull/209691

From llvmlistbot at llvm.org  Thu Jul 16 10:13:41 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:13:41 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591145.d681cab9.120c5c.48d6@mx.google.com>


================
@@ -1485,6 +1485,33 @@ def int_amdgcn_s_buffer_load : DefaultAttrsIntrinsic <
   [IntrNoMem, ImmArg<ArgIndex<2>>]>,
   AMDGPURsrcIntrinsic<0>;
 
+// Generate an s_buffer_load instruction, falling back to buffer_load if the offset
+// is not uniform. This operates like the s.buffer.load intrinsic, but uses a buffer
+// resource pointer (`ptr addrspace(8)`) to hold the resource (which should be
----------------
slinder1 wrote:

Is the "should" here a requirement, or a suggestion?

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:13:41 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:13:41 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591145.6c8668f2.26584f.0798@mx.google.com>


================
@@ -644,19 +644,16 @@ bool RegBankLegalizeHelper::lowerSBufToBuf(MachineInstr &MI,
     NumLoads = LoadSize / 128;
     Ty = Ty.divide(NumLoads);
   }
-  for (int i = 0; i < NumLoads; ++i)
-    LoadParts.emplace_back(MRI.createVirtualRegister({VgprRB, Ty}));
   MachineMemOperand *OrigMMO = *MI.memoperands_begin();
+  for (int I = 0; I < NumLoads; ++I)
----------------
slinder1 wrote:

Is there a side-effect here that makes moving this snippet meaningful?

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:13:42 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:13:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591146.510f576d.d8fd6.7619@mx.google.com>


================
@@ -1485,6 +1485,33 @@ def int_amdgcn_s_buffer_load : DefaultAttrsIntrinsic <
   [IntrNoMem, ImmArg<ArgIndex<2>>]>,
   AMDGPURsrcIntrinsic<0>;
 
+// Generate an s_buffer_load instruction, falling back to buffer_load if the offset
+// is not uniform. This operates like the s.buffer.load intrinsic, but uses a buffer
----------------
slinder1 wrote:

We should probably update the old intrinsic wording too, then?

Alternatively it seems like we can define one in terms of the other. I.e. keep your now wording, and update the comment for the old intrinsic to be "an int_amdgcn_ptr_s_buffer_load except <describe the difference>"

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:13:42 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:13:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591146.0c05e68e.b358c.383d@mx.google.com>


================
@@ -10705,33 +10707,58 @@ SDValue SITargetLowering::lowerImage(SDValue Op,
                            NumVDataDwords, IsAtomicPacked16Bit, DL);
 }
 
-SDValue SITargetLowering::lowerSBuffer(EVT VT, SDLoc DL, SDValue Rsrc,
+SDValue SITargetLowering::lowerSBuffer(EVT VT, EVT MemVT, SDLoc DL,
+                                       SDValue Chain, SDValue Rsrc,
                                        SDValue Offset, SDValue CachePolicy,
-                                       SelectionDAG &DAG) const {
+                                       SelectionDAG &DAG,
+                                       MachineMemOperand *MMO) const {
   MachineFunction &MF = DAG.getMachineFunction();
+  bool HasChainResult = MMO != nullptr;
 
-  const DataLayout &DataLayout = DAG.getDataLayout();
-  Align Alignment =
-      DataLayout.getABITypeAlign(VT.getTypeForEVT(*DAG.getContext()));
+  if (!MMO) {
+    const DataLayout &DataLayout = DAG.getDataLayout();
+    Align Alignment =
+        DataLayout.getABITypeAlign(MemVT.getTypeForEVT(*DAG.getContext()));
 
-  MachineMemOperand *MMO = MF.getMachineMemOperand(
-      MachinePointerInfo(),
-      MachineMemOperand::MOLoad | MachineMemOperand::MODereferenceable |
-          MachineMemOperand::MOInvariant,
-      VT.getStoreSize(), Alignment);
+    MMO = MF.getMachineMemOperand(MachinePointerInfo(),
+                                  MachineMemOperand::MOLoad |
+                                      MachineMemOperand::MODereferenceable |
+                                      MachineMemOperand::MOInvariant,
+                                  MemVT.getStoreSize(), Alignment);
+  }
 
   if (!Offset->isDivergent()) {
-    SDValue Ops[] = {Rsrc, Offset, CachePolicy};
+    SDValue Ops[] = {Chain, Rsrc, Offset, CachePolicy};
+
+    // Lower llvm.amdgcn.*s.buffer.load.(i8, u8) intrinsics. First, generate
+    // s_buffer_load_u8 for signed and unsigned load instructions. Next, DAG
+    // combiner tries to merge the s_buffer_load_u8 with a sext instruction
+    // (performSignExtendInRegCombine()) and it replaces s_buffer_load_u8 with
+    // s_buffer_load_i8.
+    if (MemVT == MVT::i8 && Subtarget->hasScalarSubwordLoads()) {
+      SDValue BufferLoad = DAG.getMemIntrinsicNode(
+          AMDGPUISD::SBUFFER_LOAD_UBYTE, DL,
+          DAG.getVTList(MVT::i32, MVT::Other), Ops, MemVT, MMO);
+      SDValue LoadVal = DAG.getAnyExtOrTrunc(
+          DAG.getNode(ISD::TRUNCATE, DL, MemVT, BufferLoad), DL, VT);
+      if (HasChainResult)
+        return DAG.getMergeValues({LoadVal, BufferLoad.getValue(1)}, DL);
+      return LoadVal;
+    }
 
     // Lower llvm.amdgcn.s.buffer.load.{i16, u16} intrinsics. Initially, the
     // s_buffer_load_u16 instruction is emitted for both signed and unsigned
     // loads. Later, DAG combiner tries to combine s_buffer_load_u16 with sext
     // and generates s_buffer_load_i16 (performSignExtendInRegCombine).
-    if (VT == MVT::i16 && Subtarget->hasScalarSubwordLoads()) {
-      SDValue BufferLoad =
-          DAG.getMemIntrinsicNode(AMDGPUISD::SBUFFER_LOAD_USHORT, DL,
-                                  DAG.getVTList(MVT::i32), Ops, VT, MMO);
-      return DAG.getNode(ISD::TRUNCATE, DL, VT, BufferLoad);
+    if (MemVT == MVT::i16 && Subtarget->hasScalarSubwordLoads()) {
+      SDValue BufferLoad = DAG.getMemIntrinsicNode(
+          AMDGPUISD::SBUFFER_LOAD_USHORT, DL,
+          DAG.getVTList(MVT::i32, MVT::Other), Ops, MemVT, MMO);
+      SDValue LoadVal = DAG.getAnyExtOrTrunc(
+          DAG.getNode(ISD::TRUNCATE, DL, MemVT, BufferLoad), DL, VT);
+      if (HasChainResult)
+        return DAG.getMergeValues({LoadVal, BufferLoad.getValue(1)}, DL);
+      return LoadVal;
----------------
slinder1 wrote:

This seems unrelated to the broader change? A patch could handle `i8` here separately, right?

It could be factored out with the existing `MVT::i16` handling?

```suggestion
    // Lower llvm.amdgcn.*s.buffer.load.{i,u}* intrinsics. First, generate
    // s_buffer_load_u* for signed and unsigned load instructions. Next, DAG
    // combiner tries to merge the s_buffer_load_u* with a sext instruction
    // (performSignExtendInRegCombine()) and it replaces s_buffer_load_u* with
    // s_buffer_load_i*.
    auto HandleScalarSubwordLoads = [VT, MemVT, DL, &DAG, MMO, &Ops](unsigned Opcode) {
      SDValue BufferLoad = DAG.getMemIntrinsicNode(
          Opcode, DL,
          DAG.getVTList(MVT::i32, MVT::Other), Ops, MemVT, MMO);
      SDValue LoadVal = DAG.getAnyExtOrTrunc(
          DAG.getNode(ISD::TRUNCATE, DL, MemVT, BufferLoad), DL, VT);
      if (HasChainResult)
        return DAG.getMergeValues({LoadVal, BufferLoad.getValue(1)}, DL);
      return LoadVal;
    }

    if (Subtarget->hasScalarSubwordLoads()) {
      if (MemVT == MVT::i8)
        return HandleScalarSubwordLoads(AMDGPUISD::SBUFFER_LOAD_UBYTE);
      if (MemVT == MVT::i16)
        return HandleScalarSubwordLoads(AMDGPUISD::SBUFFER_LOAD_USHORT);
    }
```

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:13:42 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:13:42 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591146.449bfb44.1b765.cd8b@mx.google.com>


================
@@ -10705,33 +10707,58 @@ SDValue SITargetLowering::lowerImage(SDValue Op,
                            NumVDataDwords, IsAtomicPacked16Bit, DL);
 }
 
-SDValue SITargetLowering::lowerSBuffer(EVT VT, SDLoc DL, SDValue Rsrc,
+SDValue SITargetLowering::lowerSBuffer(EVT VT, EVT MemVT, SDLoc DL,
+                                       SDValue Chain, SDValue Rsrc,
                                        SDValue Offset, SDValue CachePolicy,
-                                       SelectionDAG &DAG) const {
+                                       SelectionDAG &DAG,
+                                       MachineMemOperand *MMO) const {
   MachineFunction &MF = DAG.getMachineFunction();
+  bool HasChainResult = MMO != nullptr;
----------------
slinder1 wrote:

Nit: pick either `MMO != nullptr` or `!MMO` for this and the next `if`s test

https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:14:26 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:14:26 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a591172.5a0412d2.82e8e.b057@mx.google.com>


https://github.com/slinder1 edited https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 10:14:51 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Thu, 16 Jul 2026 10:14:51 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [AMDGPU] Add ptr.s.buffer.load
 intrinsic, use it from Clang (PR #209243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209243@github.com>
Message-ID: <6a59118b.eb165e5c.3844ee.d7f0@mx.google.com>


https://github.com/slinder1 edited https://github.com/llvm/llvm-project/pull/209243

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.d1496011.289bf0.5de4@mx.google.com>


https://github.com/banach-space commented:

Thank you for the updates. I feel that there might be more scope for simplification - please see my comments inline.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.6c8668f2.26584f.2c0e@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.8e75e0cb.1225b0.651c@mx.google.com>


================
@@ -0,0 +1,160 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
+
+/// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @foo {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @bar() {
+    return
+  }
+}
+
+// CHECK:       emitc.class @foo {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @fooExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @bar() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
----------------
banach-space wrote:

[nit] If the first test contains only "included" attributes, then I'd expect the second to only contain "excluded" attributes. And then some other test could mix both.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.f3151ff8.b27a.a1c1@mx.google.com>


================
@@ -0,0 +1,160 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
+
+/// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @foo {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @bar() {
+    return
+  }
----------------
banach-space wrote:

IIUC, the code path _with_ and _without_ member methods is identical? If yes, lets remove them - this is effectively duplicating tests.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.c5504292.a1395.3e61@mx.google.com>


================
@@ -0,0 +1,160 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
+
+/// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @foo {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @bar() {
+    return
+  }
+}
+
+// CHECK:       emitc.class @foo {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @fooExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @bar() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @fooExcluded {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass bails out and leaves IR unchanged if fields don't have any attributes
+
+emitc.class @fooNoAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @bar() {
+    return
+  }
+}
+
+// CHECK-NOT:     emitc.include
+// CHECK-LABEL: emitc.class @fooNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass bails out and leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @fooNoFields {
+  emitc.func @bar() {
+    return
+  }
+}
----------------
banach-space wrote:

OK, so this is a negative test, right? Please add a prefix: `@negative_{}`. See https://mlir.llvm.org/getting_started/TestingGuide/#test-naming-convention

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:36 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592784.f1c1923d.1bf95f.23b2@mx.google.com>


================
@@ -60,4 +60,49 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `included-field-attrs` option) that is an array containing a single string
+    attribute. If a field possesses both an attribute specified by
+    `included-field-attrs` and an attribute specified in `excluded-field-attrs`,
+    the `included-field-attrs` attribute takes precedence and the field will be
+    included in the reflection map.
----------------
banach-space wrote:

This is a bit confusing.

1. Why is `included-field-attrs` given precedence? That's very arbitrary. Why doesn't the pass prioritise in the order of specification? Or, select the right-most one (i.e. the last one that was added).
2. Do we really need both attributes? I would expect `included-field-attrs` to be sufficient to model everything that is required. 

My suggestion - remove `excluded-field-attrs` and keep `included-field-attrs`. Based on the tests taht I have seen, one set should be enough. Everything else that is not listed in `included-field-attrs` should be ignored. Will that work?

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 11:48:38 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 16 Jul 2026 11:48:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a592786.a1accbb4.3afc30.9518@mx.google.com>


================

----------------
banach-space wrote:

What will happen if a field is decorated with multiple attributes ("included" and "excluded")?

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Thu Jul 16 12:47:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 16 Jul 2026 12:47:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 152fdb8 - [OpenACC] emit NYI for multi-rank
 or non-memref type array reduction (#210143)
Message-ID: <6a593554.510f576d.d8fd6.4476@mx.google.com>


Author: Scott Manley
Date: 2026-07-16T14:47:27-05:00
New Revision: 152fdb8ae8725dab7e529c5cedec1d5151506307

URL: https://github.com/llvm/llvm-project/commit/152fdb8ae8725dab7e529c5cedec1d5151506307
DIFF: https://github.com/llvm/llvm-project/commit/152fdb8ae8725dab7e529c5cedec1d5151506307.diff

LOG: [OpenACC] emit NYI for multi-rank or non-memref type array reduction (#210143)

This should be an NYI and not an assertion

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6e8f38698ca67..24aeed6e23bda 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -3205,8 +3205,10 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  assert(memrefTy && memrefTy.getRank() == 1 &&
-         "array reduction accumulate expects a rank-1 memref");
+  if (!memref)
+    (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
+  if (memrefTy.getRank() != 1)
+    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);


        

From llvmlistbot at llvm.org  Thu Jul 16 12:47:36 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Thu, 16 Jul 2026 12:47:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] emit NYI for multi-rank or
 non-memref type array reduction (PR #210143)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210143@github.com>
Message-ID: <6a593558.a1accbb4.3afc30.2363@mx.google.com>


https://github.com/rscottmanley closed https://github.com/llvm/llvm-project/pull/210143

From llvmlistbot at llvm.org  Thu Jul 16 19:02:31 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Thu, 16 Jul 2026 19:02:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ptr] Add nontemporal field to
	ptr.masked_load (PR #204987)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204987@github.com>
Message-ID: <6a598d37.160324b0.1f653e.a63b@mx.google.com>


linuxlonelyeagle wrote:

You can also see this PR https://github.com/llvm/llvm-project/pull/205566. It's a similar case; I think it actually does have this field, but it's just not documented.

https://github.com/llvm/llvm-project/pull/204987

From llvmlistbot at llvm.org  Thu Jul 16 22:27:47 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Thu, 16 Jul 2026 22:27:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a59bd53.92d2873c.aa1ff.5705@mx.google.com>


https://github.com/lforg37 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH 1/4] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {

>From 7be327e24300bcd5da269c9bb161fbc1733b0443 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 16 Jul 2026 19:38:41 +0900
Subject: [PATCH 2/4] [mlir][wasm] Avoid failure to build with old gcc by not
 relying on sfinae for instantiating parsers

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 160 ++++++++-------------
 1 file changed, 59 insertions(+), 101 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 872e91179a76c..92bc71e8d55fa 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -425,8 +425,10 @@ class ExpressionParser {
   class TopLevelInstParserRegistry {
   public:
     template <std::byte opCode>
-    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
-                                              ExpressionParser &) = delete;
+    static constexpr bool hasParserForOpcode = false;
+
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &, ExpressionParser &) = delete;
   };
 
 private:
@@ -446,26 +448,20 @@ unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
 template <typename ParserRegistry>
 class InstDispatcher {
 private:
-  /// used to check if a parser is registered in the parser registry for a
-  /// given opcode.
-  /// Default to unreachable handler.
-  template <std::byte opCode, typename = void>
-  struct ParserInfo : std::false_type {
-    static constexpr auto handler = unreachableHandler;
-  };
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
   template <std::byte opCode>
-  struct ParserInfo<
-      opCode,
-      std::void_t<
-          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
-      : std::true_type {
-    static constexpr auto handler =
-        ParserRegistry::template parseInstrWithOpCode<opCode>;
-  };
+  static constexpr dispatch_t getHandlerForOpCode() {
+    if constexpr (ParserRegistry::template hasParserForOpcode<opCode>)
+      return ParserRegistry::template parseInstrWithOpCode<opCode>;
+    else
+      return unreachableHandler;
+  }
 
 public:
   template <std::byte opCode>
-  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+  static constexpr bool isValidInst =
+      ParserRegistry::template hasParserForOpcode<opCode>;
 
 private:
   static inline parsed_inst_t
@@ -476,8 +472,6 @@ class InstDispatcher {
            << static_cast<int>(opCode);
   }
 
-  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
-
   template <std::byte... opCodes>
   static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
                                            ExpressionParser &exprParser,
@@ -485,7 +479,7 @@ class InstDispatcher {
     static constexpr std::array<bool, 256> opcodeValidityMap{
         isValidInst<opCodes>...};
     static constexpr std::array<dispatch_t, 256> dispatchTable{
-        ParserInfo<opCodes>::handler...};
+        getHandlerForOpCode<opCodes>()...};
     if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
       return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
     }
@@ -1059,27 +1053,24 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
-                                       ExpressionParser &exprParser) {
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+  template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<opcode> = true;                                       \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
+      OpBuilder & builder, ExpressionParser & exprParser)
+
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1124,11 +1115,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1155,11 +1142,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {*branchArgs};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1177,11 +1160,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {callOp.getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1194,11 +1173,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
                .getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1235,19 +1210,11 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1319,35 +1286,19 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1371,6 +1322,9 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX> = true;   \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1475,24 +1429,19 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
-                                                ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::SOURCE_OP> = true;        \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1528,6 +1477,11 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S> =   \
+          true;                                                                \
+  template <>                                                                  \
   parsed_inst_t                                                                \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
@@ -1546,16 +1500,16 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
-                                                 ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
@@ -1565,6 +1519,10 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   }                                                                            \
                                                                                \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \

>From ed54961e33dc45ed1f54086b3c8dcb2244fb3929 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:17:25 +0900
Subject: [PATCH 3/4] [mlir][wasm]  Centralize check of presence of
 currentOpLoc in wasm parser

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 92bc71e8d55fa..b7cb1c9e97f54 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -27,6 +27,7 @@
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/LogicalResult.h"
 
+#include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <variant>
@@ -420,7 +421,11 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
-  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+  Location getCurrentOpLoc() {
+    assert(currentOpLoc.has_value() &&
+           "expects current opcode location to be set");
+    return *currentOpLoc;
+  }
 
   class TopLevelInstParserRegistry {
   public:
@@ -467,7 +472,7 @@ class InstDispatcher {
   static inline parsed_inst_t
   invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
                     std::byte opCode) {
-    return emitError(*(expressionParser.getCurrentOpLoc()),
+    return emitError(expressionParser.getCurrentOpLoc(),
                      "unknown instruction opcode: ")
            << static_cast<int>(opCode);
   }
@@ -1089,11 +1094,11 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
+  locations.resize(resTypes.size(), exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+  auto ifOp = IfOp::create(builder, exprParser.getCurrentOpLoc(),
                            conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
@@ -1135,7 +1140,7 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
+  BranchIfOp::create(builder, exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);

>From 736689f5f7ae6fda09ea1846a5dfef39bc71a39e Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:32:10 +0900
Subject: [PATCH 4/4] [wasm][mlir] Add one level of parser macro to prepare
 registration of multi byte opcodes.

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 68 +++++++++++++++-------
 1 file changed, 46 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index b7cb1c9e97f54..0e1c67a730b69 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -1058,24 +1058,35 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+// We can't use SFINAE in combination with deleted default instantiation
+// to identify which parsers are registered due to GCC < 14.1 bug, so we
+// use a constexpr variable to register them.
+// This is in order to avoid having to have only one "registration" of the
+// opcode.
+#define REGISTER_PARSER_OPCODE_PARSER(parserType, opcode, builderName,         \
+                                      parserName)                              \
   template <>                                                                  \
-  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
-      hasParserForOpcode<opcode> = true;                                       \
+  constexpr bool parserType::hasParserForOpcode<opcode> = true;                \
   template <>                                                                  \
-  inline parsed_inst_t                                                         \
-  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
-      OpBuilder & builder, ExpressionParser & exprParser)
+  inline parsed_inst_t parserType::parseInstrWithOpCode<opcode>(               \
+      OpBuilder & (builderName), ExpressionParser & (parserName))
+
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode, builderName, parserName)     \
+  REGISTER_PARSER_OPCODE_PARSER(ExpressionParser::TopLevelInstParserRegistry,  \
+                                opcode, builderName, parserName)
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode, builder,
+                                  exprParser) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1120,7 +1131,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   return {ValueRange{successor->getArguments()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf, builder,
+                                  exprParser) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1147,7 +1159,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   return {*branchArgs};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call, builder,
+                                  exprParser) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1165,7 +1178,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   return {callOp.getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet, builder,
+                                  exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1178,7 +1192,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
                .getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet,
+                                  builder, exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1215,11 +1230,13 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1291,19 +1308,23 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1434,11 +1455,13 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap, builder,
+                                  exprParser) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
@@ -1505,7 +1528,8 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 


From llvmlistbot at llvm.org  Thu Jul 16 22:55:28 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 16 Jul 2026 22:55:28 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a59c3d0.1905e68e.30d957.08d6@mx.google.com>


================
@@ -4527,6 +4527,46 @@ void OrderedRegionOp::build(OpBuilder &builder, OperationState &state,
 
 LogicalResult OrderedRegionOp::verify() { return verifyOrderedParent(**this); }
 
+//===----------------------------------------------------------------------===//
+// InteropInitOp
+//===----------------------------------------------------------------------===//
+
+void InteropInitOp::build(OpBuilder &builder, OperationState &state,
----------------
Saieiei wrote:

Thanks, this is addressed.

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Thu Jul 16 22:56:43 2026
From: llvmlistbot at llvm.org (Sairudra More)
Date: Thu, 16 Jul 2026 22:56:43 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a59c41b.f1c1923d.2d2b59.0a50@mx.google.com>


https://github.com/Saieiei approved this pull request.

Thanks for addressing the feedback! LGTM

https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Fri Jul 17 01:03:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 01:03:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] b69dd33 - [mlir][vector][nfc] Update
	integration tests (#210090)
Message-ID: <6a59e1de.0c05e68e.3c2f5d.6b69@mx.google.com>


Author: Andrzej Warzyński
Date: 2026-07-17T09:03:37+01:00
New Revision: b69dd3366863acff56b0b2efdc4c4938097bb43e

URL: https://github.com/llvm/llvm-project/commit/b69dd3366863acff56b0b2efdc4c4938097bb43e
DIFF: https://github.com/llvm/llvm-project/commit/b69dd3366863acff56b0b2efdc4c4938097bb43e.diff

LOG: [mlir][vector][nfc] Update integration tests (#210090)

Add comments, unify naming, remove redundant printing hooks.

Added: 
    

Modified: 
    mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
    mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
    mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
    mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
    mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
    mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
index 1683fa504e6e2..2c8680c9fa369 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/compress.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @compress16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xf32>,
                  %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @compress16(%base: memref<?xf32>,
   return
 }
 
-func.func @compress16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xf32>,
                      %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.compressstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @compress16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -55,50 +67,57 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  call @compress16(%A, %none, %value)
+  call @compress_16(%A, %none, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @compress16(%A, %all, %value)
+  call @compress_16(%A, %all, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some3, %value)
+  call @compress_16(%A, %some3, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 3, 7, 11, 13, 15, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some2, %value)
+  call @compress_16(%A, %some2, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 3, 7, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16(%A, %some1, %value)
+  call @compress_16(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @compress16_at8(%A, %some1, %value)
+  call @compress_16_at_8(%A, %some1, %value)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK-NEXT: ( 0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 11, 13, 15, 7, 0, 1, 2, 3, 12, 13, 14, 15]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
index 8c994f83c5fdb..48f8862240776 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/expand.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @expand16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16
+//
+// Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16(%base: memref<?xf32>,
                %mask: vector<16xi1>,
                %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
@@ -12,7 +17,12 @@ func.func @expand16(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @expand16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @expand_16_at_8
+//
+// Same as @expand_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @expand_16_at_8(%base: memref<?xf32>,
                    %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
@@ -21,7 +31,12 @@ func.func @expand16_at8(%base: memref<?xf32>,
   return %e : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -41,41 +56,46 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some1 = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %some1 = vector.constant_mask [4] : vector<16xi1>
   %0 = vector.insert %f, %some1[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[7] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[11] : i1 into vector<16xi1>
   %3 = vector.insert %t, %2[13] : i1 into vector<16xi1>
+  // %some2 = [0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some2 = vector.insert %t, %3[15] : i1 into vector<16xi1>
+  // %some3 = [0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0]
   %some3 = vector.insert %f, %some2[2] : i1 into vector<16xi1>
 
   //
   // Expanding load tests.
   //
 
-  %e1 = call @expand16(%A, %none, %pass)
+  %e1 = call @expand_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e2 = call @expand16(%A, %all, %pass)
+  %e2 = call @expand_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e2 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %e3 = call @expand16(%A, %some1, %pass)
+  %e3 = call @expand_16(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e3 : vector<16xf32>
   // CHECK-NEXT: ( 0, 1, 2, 3, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %e4 = call @expand16(%A, %some2, %pass)
+  %e4 = call @expand_16(%A, %some2, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e4 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 1, 2, -7, -7, -7, 3, -7, -7, -7, 4, -7, 5, -7, 6 )
 
-  %e5 = call @expand16(%A, %some3, %pass)
+  %e5 = call @expand_16(%A, %some3, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e5 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, -7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, -7, 5 )
@@ -83,12 +103,12 @@ func.func @entry() {
   %4 = vector.insert %v, %pass[1] : f32 into vector<16xf32>
   %5 = vector.insert %v, %4[2] : f32 into vector<16xf32>
   %alt_pass = vector.insert %v, %5[14] : f32 into vector<16xf32>
-  %e6 = call @expand16(%A, %some3, %alt_pass)
+  %e6 = call @expand_16(%A, %some3, %alt_pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e6 : vector<16xf32>
   // CHECK-NEXT: ( -7, 0, 7.7, 1, -7, -7, -7, 2, -7, -7, -7, 3, -7, 4, 7.7, 5 )
 
-  %e7 = call @expand16_at8(%A, %some1, %pass)
+  %e7 = call @expand_16_at_8(%A, %some1, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %e7 : vector<16xf32>
   // CHECK-NEXT: ( 8, 9, 10, 11, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 110a46e2d89d8..078826522bcb5 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
-// DEFINE: %{entry_point} = main
-// DEFINE: %{run} = mlir-runner -e entry -entry-point-result=void \
+// DEFINE: %{main_point} = main
+// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)
@@ -16,7 +16,12 @@
 // REDEFINE: %{compile} = mlir-opt %s --test-vector-gather-lowering
 // RUN: %{compile} | FileCheck %s -check-prefix CHECK-IR 
 
-func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
+//===----------------------------------------------------------------------===//
+// @gather_8
+//
+// Gather 8 elements
+//===----------------------------------------------------------------------===//
+func.func @gather_8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
               %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
   %c0 = arith.constant 0: index
   /// Verify that the lowering via vector.load does indeed generate vector.load
@@ -26,7 +31,12 @@ func.func @gather8(%base: memref<?x?xf32>, %indices: vector<8xi32>,
   return %g : vector<8xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -78,25 +88,25 @@ func.func @entry() {
   // Gather tests.
   //
 
-  %g1 = call @gather8(%A, %idx, %all, %pass)
+  %g1 = call @gather_8(%A, %idx, %all, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g1 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, 10, 84, 34, 42 )
 
-  %g2 = call @gather8(%A, %idx, %none, %pass)
+  %g2 = call @gather_8(%A, %idx, %none, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g2 : vector<8xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %g3 = call @gather8(%A, %idx, %some, %pass)
+  %g3 = call @gather_8(%A, %idx, %some, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g3 : vector<8xf32>
   // CHECK: ( 0, 31, 21, 63, -7, -7, -7, -7 )
 
-  %g4 = call @gather8(%A, %idx, %more, %pass)
+  %g4 = call @gather_8(%A, %idx, %more, %pass)
     : (memref<?x?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>)
     -> (vector<8xf32>)
   vector.print %g4 : vector<8xf32>

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
index b9f9f8674d412..ce06dbbbf4cac 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedload.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
+// RUN: mlir-runner -e main -entry-point-result=void \
 // RUN:   -shared-libs=%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
+//===----------------------------------------------------------------------===//
+// @maskedload_16
+//
+// Load 16 elements. Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedload_16(%base: memref<?xf32>, %mask: vector<16xi1>,
                    %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c0 = arith.constant 0: index
   %ld = vector.maskedload %base[%c0], %mask, %pass_thru
@@ -11,7 +16,13 @@ func.func @maskedload16(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
+//===----------------------------------------------------------------------===//
+// @maskedload_16_at_8
+//
+// Same as @maskedload_16, but the insertion index is hard-coded to 8 instead
+// of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedload_16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
                        %pass_thru: vector<16xf32>) -> vector<16xf32> {
   %c8 = arith.constant 8: index
   %ld = vector.maskedload %base[%c8], %mask, %pass_thru
@@ -19,7 +30,12 @@ func.func @maskedload16_at8(%base: memref<?xf32>, %mask: vector<16xi1>,
   return %ld : vector<16xf32>
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -38,39 +54,43 @@ func.func @entry() {
   // Set up masks.
   %f = arith.constant 0: i1
   %t = arith.constant 1: i1
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
   %none = vector.constant_mask [0] : vector<16xi1>
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
   %all = vector.constant_mask [16] : vector<16xi1>
+  // %some = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]
   %some = vector.constant_mask [8] : vector<16xi1>
   %0 = vector.insert %f, %some[0] : i1 into vector<16xi1>
   %1 = vector.insert %t, %0[13] : i1 into vector<16xi1>
   %2 = vector.insert %t, %1[14] : i1 into vector<16xi1>
+  // %other = [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1]
   %other = vector.insert %t, %2[14] : i1 into vector<16xi1>
 
   //
   // Masked load tests.
   //
 
-  %l1 = call @maskedload16(%A, %none, %pass)
+  %l1 = call @maskedload_16(%A, %none, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l1 : vector<16xf32>
   // CHECK: ( -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %l2 = call @maskedload16(%A, %all, %pass)
+  %l2 = call @maskedload_16(%A, %all, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l2 : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  %l3 = call @maskedload16(%A, %some, %pass)
+  %l3 = call @maskedload_16(%A, %some, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l3 : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, -7, -7, -7, -7, -7, -7, -7, -7 )
 
-  %l4 = call @maskedload16(%A, %other, %pass)
+  %l4 = call @maskedload_16(%A, %other, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l4 : vector<16xf32>
   // CHECK: ( -7, 1, 2, 3, 4, 5, 6, 7, -7, -7, -7, -7, -7, 13, 14, -7 )
 
-  %l5 = call @maskedload16_at8(%A, %some, %pass)
+  %l5 = call @maskedload_16_at8(%A, %some, %pass)
     : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> (vector<16xf32>)
   vector.print %l5 : vector<16xf32>
   // CHECK: ( 8, 9, 10, 11, 12, 13, 14, 15, -7, -7, -7, -7, -7, -7, -7, -7 )

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
index 826da5309035e..b4d3b2ac5d6e4 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/maskedstore.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @maskedstore16(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16
+//
+// Store 16 elements. Insertion index is hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16(%base: memref<?xf32>,
                     %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0: index
   vector.maskedstore %base[%c0], %mask, %value
@@ -11,7 +16,12 @@ func.func @maskedstore16(%base: memref<?xf32>,
   return
 }
 
-func.func @maskedstore16_at8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @maskedstore_16_at_8
+//
+// Same as @maskedstore_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @maskedstore_16_at_8(%base: memref<?xf32>,
                         %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c8 = arith.constant 8: index
   vector.maskedstore %base[%c8], %mask, %value
@@ -19,23 +29,25 @@ func.func @maskedstore16_at8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem16(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c16 = arith.constant 16: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<16xf32>
-  %mem = scf.for %i = %c0 to %c16 step %c1
-    iter_args(%m_iter = %m) -> (vector<16xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<16xf32>
-    scf.yield %m_new : vector<16xf32>
-  }
-  vector.print %mem : vector<16xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %f0 = arith.constant 0.0: f32
   %c0 = arith.constant 0: index
@@ -70,34 +82,36 @@ func.func @entry() {
   vector.print %val : vector<16xf32>
   // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
 
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @print1DMemRef(%A): (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %none, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %none, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0 )
+  call @maskedstore_16(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0, 0]
 
-  call @maskedstore16(%A, %more, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0 )
+  call @maskedstore_16(%A, %more, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 13, 0, 0]
 
-  call @maskedstore16(%A, %all, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 )
+  call @maskedstore_16(%A, %all, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
 
-  call @maskedstore16_at8(%A, %some, %val)
-    : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
-  call @printmem16(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @maskedstore_16_at_8(%A, %some, %val)
+  : (memref<?xf32>, vector<16xi1>, vector<16xf32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
index 22b5eef12a202..1f5410716d05a 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/scatter.mlir
@@ -1,9 +1,14 @@
 // RUN: mlir-opt %s -test-lower-to-llvm  | \
-// RUN: mlir-runner -e entry -entry-point-result=void \
-// RUN:   -shared-libs=%mlir_c_runner_utils | \
+// RUN: mlir-runner -e main -entry-point-result=void \
+// RUN:   -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils | \
 // RUN: FileCheck %s
 
-func.func @scatter8(%base: memref<?xf32>,
+//===----------------------------------------------------------------------===//
+// @scatter_8
+//
+// Scatter 8 elements
+//===----------------------------------------------------------------------===//
+func.func @scatter_8(%base: memref<?xf32>,
                %indices: vector<8xi32>,
                %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0: index
@@ -12,23 +17,25 @@ func.func @scatter8(%base: memref<?xf32>,
   return
 }
 
-func.func @printmem8(%A: memref<?xf32>) {
-  %c0 = arith.constant 0: index
-  %c1 = arith.constant 1: index
-  %c8 = arith.constant 8: index
-  %z = arith.constant 0.0: f32
-  %m = vector.broadcast %z : f32 to vector<8xf32>
-  %mem = scf.for %i = %c0 to %c8 step %c1
-    iter_args(%m_iter = %m) -> (vector<8xf32>) {
-    %c = memref.load %A[%i] : memref<?xf32>
-    %m_new = vector.insert %c, %m_iter[%i] : f32 into vector<8xf32>
-    scf.yield %m_new : vector<8xf32>
-  }
-  vector.print %mem : vector<8xf32>
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xf32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xf32> to memref<*xf32>
+
+  call @printMemrefF32(%cast): (memref<*xf32>) -> ()
+
   return
 }
 
-func.func @entry() {
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point.
+//===----------------------------------------------------------------------===//
+func.func @main() {
   // Set up memory.
   %c0 = arith.constant 0: index
   %c1 = arith.constant 1: index
@@ -90,29 +97,31 @@ func.func @entry() {
   vector.print %idx : vector<8xi32>
   // CHECK: ( 7, 0, 1, 6, 2, 4, 5, 3 )
 
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %none, %val)
+  call @scatter_8(%A, %idx, %none, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 0, 1, 2, 3, 4, 5, 6, 7 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7]
 
-  call @scatter8(%A, %idx, %some, %val)
+  call @scatter_8(%A, %idx, %some, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 3, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 3, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %more, %val)
+  call @scatter_8(%A, %idx, %more, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 2, 7, 4, 5, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 2, 7, 4, 5, 3, 0]
 
-  call @scatter8(%A, %idx, %all, %val)
+  call @scatter_8(%A, %idx, %all, %val)
     : (memref<?xf32>, vector<8xi32>, vector<8xi1>, vector<8xf32>) -> ()
-  call @printmem8(%A) : (memref<?xf32>) -> ()
-  // CHECK: ( 1, 2, 4, 7, 5, 6, 3, 0 )
+  call @print1DMemRef(%A) : (memref<?xf32>) -> ()
+  // CHECK: [1, 2, 4, 7, 5, 6, 3, 0]
 
   memref.dealloc %A : memref<?xf32>
   return
 }
+
+func.func private @printMemrefF32(%ptr : memref<*xf32>)


        

From llvmlistbot at llvm.org  Fri Jul 17 01:03:44 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:03:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a59e1e0.d1496011.199a59.3639@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Fri Jul 17 01:07:42 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Fri, 17 Jul 2026 01:07:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ptr] Add nontemporal field to
	ptr.masked_load (PR #204987)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204987@github.com>
Message-ID: <6a59e2ce.92ea0e2a.4a9a5.f749@mx.google.com>


linuxlonelyeagle wrote:

> This is indeed a bit confusing. The nontemporal flag is just metadata so nothing prevents us from adding it to masked loads / stores. However, it is indeed a good question if this is something that the LLVM backends take into account?
> 
> I see some SVE tests that use nontemporal metadata but it is unclear to me if that is intentional or just a random artifact of these tests.
> 
> @linuxlonelyeagle do you have a concrete use case for this? Do you know of a backend that uses the metadata?


```
define void @test_x86_store_nontemporal(<4 x float>* %dst, <4 x float> %val) {
  store <4 x float> %val, <4 x float>* %dst, align 16, !nontemporal !0
  ret void
}

!0 = !{i32 1}
```
we use opt, it will be 
```
test_x86_store_nontemporal:             # @test_x86_store_nontemporal
        vmovntps        xmmword ptr [rdi], xmm0
        ret
```

```
define void @test_x86_store(<4 x float>* %dst, <4 x float> %val) {
  store <4 x float> %val, <4 x float>* %dst, align 16
  ret void
}

!0 = !{i32 1}
```
it will be
```
test_x86_store:                         # @test_x86_store
        vmovaps xmmword ptr [rdi], xmm0
        ret
```
You can see https://www.felixcloutier.com/x86/movntps.

For the record, the successful test case was an unmasked store. I was unable to generate any non-temporal instructions for the masked counterpart, even with strict alignment requirements.
My understanding is that this behavior is highly target-dependent. If the target ISA supports the masked non-temporal instruction and the LLVM backend has the matching pattern implemented, it will be generated. Otherwise, the nontemporal metadata is simply ignored.


https://github.com/llvm/llvm-project/pull/204987

From llvmlistbot at llvm.org  Fri Jul 17 01:08:59 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 17 Jul 2026 01:08:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a59e31b.a1accbb4.5eca2.fbb2@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `mlir-rocm-mi200` running on `mi200-buildbot` while building `mlir` at step 7 "test-build-check-mlir-build-only-check-mlir".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/177/builds/37626

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 7 (test-build-check-mlir-build-only-check-mlir) failure: test (failure)
******************** TEST 'MLIR :: Integration/Dialect/Vector/CPU/gather.mlir' FAILED ********************
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 7
/vol/worker/mi200-buildbot/mlir-rocm-mi200/build/bin/mlir-opt /vol/worker/mi200-buildbot/mlir-rocm-mi200/llvm-project/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm | /vol/worker/mi200-buildbot/mlir-rocm-mi200/build/bin/mlir-runner -e main -main-point-result=void -shared-libs=/vol/worker/mi200-buildbot/mlir-rocm-mi200/build/lib/libmlir_runner_utils.so,/vol/worker/mi200-buildbot/mlir-rocm-mi200/build/lib/libmlir_c_runner_utils.so | /vol/worker/mi200-buildbot/mlir-rocm-mi200/build/bin/FileCheck /vol/worker/mi200-buildbot/mlir-rocm-mi200/llvm-project/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
# executed command: /vol/worker/mi200-buildbot/mlir-rocm-mi200/build/bin/mlir-opt /vol/worker/mi200-buildbot/mlir-rocm-mi200/llvm-project/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm
# .---command stdout------------
# | module {
# |   llvm.func @free(!llvm.ptr)
# |   llvm.func @malloc(i64) -> !llvm.ptr
# |   llvm.func @printNewline()
# |   llvm.func @printClose()
# |   llvm.func @printComma()
# |   llvm.func @printF32(f32)
# |   llvm.func @printOpen()
# |   llvm.func @gather_8(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: i64, %arg3: i64, %arg4: i64, %arg5: i64, %arg6: i64, %arg7: vector<8xi32>, %arg8: vector<8xi1>, %arg9: vector<8xf32>) -> vector<8xf32> {
# |     %0 = llvm.mlir.poison : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
# |     %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %3 = llvm.insertvalue %arg2, %2[2] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %4 = llvm.insertvalue %arg3, %3[3, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %5 = llvm.insertvalue %arg5, %4[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %6 = llvm.insertvalue %arg4, %5[3, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %7 = llvm.insertvalue %arg6, %6[4, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.extractvalue %7[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %10 = llvm.extractvalue %7[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %11 = llvm.mul %8, %10 : i64
# |     %12 = llvm.add %11, %8 : i64
# |     %13 = llvm.getelementptr %9[%12] : (!llvm.ptr, i64) -> !llvm.ptr, f32
# |     %14 = llvm.getelementptr %13[%arg7] : (!llvm.ptr, vector<8xi32>) -> vector<8x!llvm.ptr>, f32
# |     %15 = llvm.intr.masked.gather %14, %arg8, %arg9 {alignment = 4 : i32} : (vector<8x!llvm.ptr>, vector<8xi1>, vector<8xf32>) -> vector<8xf32>
# |     llvm.return %15 : vector<8xf32>
# |   }
# |   llvm.func @main() {
# |     %0 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, true]> : vector<8xi1>) : vector<8xi1>
# |     %1 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, false]> : vector<8xi1>) : vector<8xi1>
# |     %2 = llvm.mlir.constant(7 : index) : i64
# |     %3 = llvm.mlir.constant(8 : index) : i64
# |     %4 = llvm.mlir.constant(dense<true> : vector<8xi1>) : vector<8xi1>
# |     %5 = llvm.mlir.constant(dense<false> : vector<8xi1>) : vector<8xi1>
# |     %6 = llvm.mlir.constant(dense<-7.000000e+00> : vector<8xf32>) : vector<8xf32>
# |     %7 = llvm.mlir.constant(dense<[0, 16, 11, 33, 5, 44, 19, 22]> : vector<8xi32>) : vector<8xi32>
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.mlir.constant(1 : index) : i64
# |     %10 = llvm.mlir.constant(10 : index) : i64
# |     %11 = llvm.mlir.constant(5 : index) : i64
# |     %12 = llvm.mlir.constant(10 : index) : i64
...

```

</details>

https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Fri Jul 17 01:10:34 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 17 Jul 2026 01:10:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a59e37a.510f576d.2f694c.e8e1@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `mlir-nvidia-gcc7` running on `mlir-nvidia` while building `mlir` at step 7 "test-build-check-mlir-build-only-check-mlir".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/116/builds/30570

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 7 (test-build-check-mlir-build-only-check-mlir) failure: test (failure)
******************** TEST 'MLIR :: Integration/Dialect/Vector/CPU/gather.mlir' FAILED ********************
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 7
/vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/bin/mlir-opt /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm | /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/bin/mlir-runner -e main -main-point-result=void -shared-libs=/vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/lib/libmlir_runner_utils.so,/vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/lib/libmlir_c_runner_utils.so | /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/bin/FileCheck /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
# executed command: /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.obj/bin/mlir-opt /vol/worker/mlir-nvidia/mlir-nvidia-gcc7/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm
# .---command stdout------------
# | module {
# |   llvm.func @free(!llvm.ptr)
# |   llvm.func @malloc(i64) -> !llvm.ptr
# |   llvm.func @printNewline()
# |   llvm.func @printClose()
# |   llvm.func @printComma()
# |   llvm.func @printF32(f32)
# |   llvm.func @printOpen()
# |   llvm.func @gather_8(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: i64, %arg3: i64, %arg4: i64, %arg5: i64, %arg6: i64, %arg7: vector<8xi32>, %arg8: vector<8xi1>, %arg9: vector<8xf32>) -> vector<8xf32> {
# |     %0 = llvm.mlir.poison : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
# |     %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %3 = llvm.insertvalue %arg2, %2[2] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %4 = llvm.insertvalue %arg3, %3[3, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %5 = llvm.insertvalue %arg5, %4[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %6 = llvm.insertvalue %arg4, %5[3, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %7 = llvm.insertvalue %arg6, %6[4, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.extractvalue %7[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %10 = llvm.extractvalue %7[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %11 = llvm.mul %8, %10 : i64
# |     %12 = llvm.add %11, %8 : i64
# |     %13 = llvm.getelementptr %9[%12] : (!llvm.ptr, i64) -> !llvm.ptr, f32
# |     %14 = llvm.getelementptr %13[%arg7] : (!llvm.ptr, vector<8xi32>) -> vector<8x!llvm.ptr>, f32
# |     %15 = llvm.intr.masked.gather %14, %arg8, %arg9 {alignment = 4 : i32} : (vector<8x!llvm.ptr>, vector<8xi1>, vector<8xf32>) -> vector<8xf32>
# |     llvm.return %15 : vector<8xf32>
# |   }
# |   llvm.func @main() {
# |     %0 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, true]> : vector<8xi1>) : vector<8xi1>
# |     %1 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, false]> : vector<8xi1>) : vector<8xi1>
# |     %2 = llvm.mlir.constant(7 : index) : i64
# |     %3 = llvm.mlir.constant(8 : index) : i64
# |     %4 = llvm.mlir.constant(dense<true> : vector<8xi1>) : vector<8xi1>
# |     %5 = llvm.mlir.constant(dense<false> : vector<8xi1>) : vector<8xi1>
# |     %6 = llvm.mlir.constant(dense<-7.000000e+00> : vector<8xf32>) : vector<8xf32>
# |     %7 = llvm.mlir.constant(dense<[0, 16, 11, 33, 5, 44, 19, 22]> : vector<8xi32>) : vector<8xi32>
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.mlir.constant(1 : index) : i64
# |     %10 = llvm.mlir.constant(10 : index) : i64
# |     %11 = llvm.mlir.constant(5 : index) : i64
# |     %12 = llvm.mlir.constant(10 : index) : i64
...

```

</details>

https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Fri Jul 17 01:12:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:12:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a59e404.eb165e5c.4506c.d679@mx.google.com>


banach-space wrote:

Gentle ping :) 

I would like to land this and continue with https://github.com/llvm/llvm-project/pull/199546 before disappearing for a break 😅 

https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Fri Jul 17 01:17:01 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:17:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix e2e test post #210090 (PR
	#210270)
Message-ID: <llvm.org/llvm/llvm-project/pull/210270@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/210270

Failing bots:
  * https://lab.llvm.org/buildbot/#/builders/177/builds/37626
  * https://lab.llvm.org/buildbot/#/builders/116/builds/30570


>From 33c921222d9f3edfc9c81f9caba526ae41aa12c3 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 17 Jul 2026 08:14:48 +0000
Subject: [PATCH] [mlir][vector] Fix e2e test post #210090

Failing bots:
  * https://lab.llvm.org/buildbot/#/builders/177/builds/37626
  * https://lab.llvm.org/buildbot/#/builders/116/builds/30570
---
 mlir/test/Integration/Dialect/Vector/CPU/gather.mlir | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 078826522bcb5..03bf82a630478 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
 // DEFINE: %{main_point} = main
-// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
+// DEFINE: %{run} = mlir-runner -e main -entry-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)


From llvmlistbot at llvm.org  Fri Jul 17 01:17:18 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:17:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a59e50e.66c0679a.24976c.862b@mx.google.com>


banach-space wrote:

Buildbot failures fixed in https://github.com/llvm/llvm-project/pull/210270

https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Fri Jul 17 01:17:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 01:17:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix e2e test post #210090 (PR
	#210270)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210270@github.com>
Message-ID: <6a59e520.e5907290.281687.f4eb@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Failing bots:
  * https://lab.llvm.org/buildbot/#/builders/177/builds/37626
  * https://lab.llvm.org/buildbot/#/builders/116/builds/30570


---
Full diff: https://github.com/llvm/llvm-project/pull/210270.diff


1 Files Affected:

- (modified) mlir/test/Integration/Dialect/Vector/CPU/gather.mlir (+1-1) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 078826522bcb5..03bf82a630478 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
 // DEFINE: %{main_point} = main
-// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
+// DEFINE: %{run} = mlir-runner -e main -entry-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)

``````````

</details>


https://github.com/llvm/llvm-project/pull/210270

From llvmlistbot at llvm.org  Fri Jul 17 01:17:40 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:17:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix e2e test post #210090 (PR
	#210270)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210270@github.com>
Message-ID: <6a59e524.160324b0.1f653e.adcb@mx.google.com>


https://github.com/banach-space auto_merge_enabled https://github.com/llvm/llvm-project/pull/210270

From llvmlistbot at llvm.org  Fri Jul 17 01:19:29 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Fri, 17 Jul 2026 01:19:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector][nfc] Update integration tests
	(PR #210090)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210090@github.com>
Message-ID: <6a59e591.7270d2fd.a9bd6.b213@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `mlir-nvidia` running on `mlir-nvidia` while building `mlir` at step 7 "test-build-check-mlir-build-only-check-mlir".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/138/builds/31912

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 7 (test-build-check-mlir-build-only-check-mlir) failure: test (failure)
******************** TEST 'MLIR :: Integration/Dialect/Vector/CPU/gather.mlir' FAILED ********************
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 7
/vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/bin/mlir-opt /vol/worker/mlir-nvidia/mlir-nvidia/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm | /vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/bin/mlir-runner -e main -main-point-result=void -shared-libs=/vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/lib/libmlir_runner_utils.so,/vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/lib/libmlir_c_runner_utils.so | /vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/bin/FileCheck /vol/worker/mlir-nvidia/mlir-nvidia/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
# executed command: /vol/worker/mlir-nvidia/mlir-nvidia/llvm.obj/bin/mlir-opt /vol/worker/mlir-nvidia/mlir-nvidia/llvm.src/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir -test-lower-to-llvm
# .---command stdout------------
# | module {
# |   llvm.func @free(!llvm.ptr)
# |   llvm.func @malloc(i64) -> !llvm.ptr
# |   llvm.func @printNewline()
# |   llvm.func @printClose()
# |   llvm.func @printComma()
# |   llvm.func @printF32(f32)
# |   llvm.func @printOpen()
# |   llvm.func @gather_8(%arg0: !llvm.ptr, %arg1: !llvm.ptr, %arg2: i64, %arg3: i64, %arg4: i64, %arg5: i64, %arg6: i64, %arg7: vector<8xi32>, %arg8: vector<8xi1>, %arg9: vector<8xf32>) -> vector<8xf32> {
# |     %0 = llvm.mlir.poison : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)>
# |     %1 = llvm.insertvalue %arg0, %0[0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %2 = llvm.insertvalue %arg1, %1[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %3 = llvm.insertvalue %arg2, %2[2] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %4 = llvm.insertvalue %arg3, %3[3, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %5 = llvm.insertvalue %arg5, %4[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %6 = llvm.insertvalue %arg4, %5[3, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %7 = llvm.insertvalue %arg6, %6[4, 1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.extractvalue %7[1] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %10 = llvm.extractvalue %7[4, 0] : !llvm.struct<(ptr, ptr, i64, array<2 x i64>, array<2 x i64>)> 
# |     %11 = llvm.mul %8, %10 : i64
# |     %12 = llvm.add %11, %8 : i64
# |     %13 = llvm.getelementptr %9[%12] : (!llvm.ptr, i64) -> !llvm.ptr, f32
# |     %14 = llvm.getelementptr %13[%arg7] : (!llvm.ptr, vector<8xi32>) -> vector<8x!llvm.ptr>, f32
# |     %15 = llvm.intr.masked.gather %14, %arg8, %arg9 {alignment = 4 : i32} : (vector<8x!llvm.ptr>, vector<8xi1>, vector<8xf32>) -> vector<8xf32>
# |     llvm.return %15 : vector<8xf32>
# |   }
# |   llvm.func @main() {
# |     %0 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, true]> : vector<8xi1>) : vector<8xi1>
# |     %1 = llvm.mlir.constant(dense<[true, true, true, true, false, false, false, false]> : vector<8xi1>) : vector<8xi1>
# |     %2 = llvm.mlir.constant(7 : index) : i64
# |     %3 = llvm.mlir.constant(8 : index) : i64
# |     %4 = llvm.mlir.constant(dense<true> : vector<8xi1>) : vector<8xi1>
# |     %5 = llvm.mlir.constant(dense<false> : vector<8xi1>) : vector<8xi1>
# |     %6 = llvm.mlir.constant(dense<-7.000000e+00> : vector<8xf32>) : vector<8xf32>
# |     %7 = llvm.mlir.constant(dense<[0, 16, 11, 33, 5, 44, 19, 22]> : vector<8xi32>) : vector<8xi32>
# |     %8 = llvm.mlir.constant(0 : index) : i64
# |     %9 = llvm.mlir.constant(1 : index) : i64
# |     %10 = llvm.mlir.constant(10 : index) : i64
# |     %11 = llvm.mlir.constant(5 : index) : i64
# |     %12 = llvm.mlir.constant(10 : index) : i64
...

```

</details>

https://github.com/llvm/llvm-project/pull/210090

From llvmlistbot at llvm.org  Fri Jul 17 01:30:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 01:30:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9ee8b18 - [mlir][vector] Fix e2e test post
	#210090 (#210270)
Message-ID: <6a59e842.481afadf.9e047.32dd@mx.google.com>


Author: Andrzej Warzyński
Date: 2026-07-17T08:30:53Z
New Revision: 9ee8b18b6c0758e57cedfa57e99ba85dfd247d35

URL: https://github.com/llvm/llvm-project/commit/9ee8b18b6c0758e57cedfa57e99ba85dfd247d35
DIFF: https://github.com/llvm/llvm-project/commit/9ee8b18b6c0758e57cedfa57e99ba85dfd247d35.diff

LOG: [mlir][vector] Fix e2e test post #210090 (#210270)

Failing bots:
  * https://lab.llvm.org/buildbot/#/builders/177/builds/37626
  * https://lab.llvm.org/buildbot/#/builders/116/builds/30570

Added: 
    

Modified: 
    mlir/test/Integration/Dialect/Vector/CPU/gather.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
index 078826522bcb5..03bf82a630478 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/gather.mlir
@@ -1,5 +1,5 @@
 // DEFINE: %{main_point} = main
-// DEFINE: %{run} = mlir-runner -e main -main-point-result=void \
+// DEFINE: %{run} = mlir-runner -e main -entry-point-result=void \
 // DEFINE:         -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
 
 /// TEST 1. Verify default compilation (direct lowering of `vector.gather` to LLVM)


        

From llvmlistbot at llvm.org  Fri Jul 17 01:31:01 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 01:31:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Fix e2e test post #210090 (PR
	#210270)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210270@github.com>
Message-ID: <6a59e845.4a342cbc.3a2f36.6a65@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/210270

From llvmlistbot at llvm.org  Fri Jul 17 02:57:16 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 02:57:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp` +
 ExpandloadOp support scalable vectors (PR #210288)
Message-ID: <llvm.org/llvm/llvm-project/pull/210288@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/210288

Extends `vector.compressstore` + `vector.expandload` to support scalable
vectors and updates relevant tests.


>From d9c4ebfc9363b1c9fd3af3f7dbb04815dbf9d6a1 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 17 Jul 2026 09:54:59 +0000
Subject: [PATCH] [mlir][vector] Make CompressstoreOp` + ExpandloadOp support
 scalable vectors

Extends `vector.compressstore` + `vector.expandload` to support scalable
vectors and updates relevant tests.
---
 .../mlir/Dialect/Vector/IR/VectorOps.td       |   4 +-
 .../vector-to-llvm-interface.mlir             |  53 ++++-
 mlir/test/Dialect/Vector/invalid.mlir         |  16 --
 mlir/test/Dialect/Vector/ops.mlir             |  20 ++
 .../vector-dropleadunitdim-transforms.mlir    |  24 ++
 .../Dialect/Vector/vector-mem-transforms.mlir |  50 ++++
 .../Dialect/Vector/CPU/ArmSVE/compress.mlir   | 214 ++++++++++++++++++
 7 files changed, 351 insertions(+), 30 deletions(-)
 create mode 100644 mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index 6e0134fc0cdc6..ebfb379684dd3 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2289,7 +2289,7 @@ def Vector_ExpandLoadOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$pass_thru,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)>,
     Results<(outs AnyVectorOfNonZeroRank:$result)> {
@@ -2381,7 +2381,7 @@ def Vector_CompressStoreOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$valueToStore,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)> {
 
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
index cc35c6600d12c..4cbea04e8076e 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
@@ -1944,13 +1944,13 @@ func.func @negative_scatter_on_strided_memref(%arg0: memref<?xf32, strided<[2],
 // vector.expandload
 //===----------------------------------------------------------------------===//
 
-func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
+func.func @expandload(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32> into vector<11xf32>
   return %0 : vector<11xf32>
 }
 
-// CHECK-LABEL: func @expand_load_op
+// CHECK-LABEL: func @expandload
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1959,21 +1959,36 @@ func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vec
 
 // -----
 
-func.func @expand_load_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
+func.func @expandload_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) -> vector<[11]xf32> {
+  %c0 = arith.constant 0: index
+  %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32> into vector<[11]xf32>
+  return %0 : vector<[11]xf32>
+}
+
+// CHECK-LABEL: func @expandload_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: %[[E:.*]] = "llvm.intr.masked.expandload"(%[[P]], %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<[11]xi1>, vector<[11]xf32>) -> vector<[11]xf32>
+// CHECK: return %[[E]] : vector<[11]xf32>
+
+// -----
+
+func.func @expandload_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_index
+// CHECK-LABEL: func @expandload_index
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
 
-func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
+func.func @expandload_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_with_alignment
+// CHECK-LABEL: func @expandload_with_alignment
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{llvm.align = 8 : i64}, {}, {}]}> : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
@@ -1982,13 +1997,13 @@ func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<1
 // vector.compressstore
 //===----------------------------------------------------------------------===//
 
-func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
+func.func @compressstore(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32>
   return
 }
 
-// CHECK-LABEL: func @compress_store_op
+// CHECK-LABEL: func @compressstore
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1996,21 +2011,35 @@ func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2:
 
 // -----
 
-func.func @compress_store_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
+func.func @compressstore_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32>
+  return
+}
+
+// CHECK-LABEL: func @compressstore_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %[[P]], %{{.*}}) : (vector<[11]xf32>, !llvm.ptr, vector<[11]xi1>) -> ()
+
+// -----
+
+func.func @compressstore_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_index
+// CHECK-LABEL: func @compressstore_index
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
 
-func.func @compress_store_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
+func.func @compressstore_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
   vector.compressstore %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_with_alignment
+// CHECK-LABEL: func @compressstore_with_alignment
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{}, {llvm.align = 8 : i64}, {}]}> : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..d4f305f20b595 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1688,14 +1688,6 @@ func.func @expand_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>,
 
 // -----
 
-func.func @expand_base_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.expandload' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
-}
-
-// -----
-
 func.func @expand_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.expandload' op expected result shape to match mask shape}}
@@ -1750,14 +1742,6 @@ func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1
 
 // -----
 
-func.func @compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %value: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.compressstore' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
-}
-
-// -----
-
 func.func @compress_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op expected valueToStore shape to match mask shape}}
diff --git a/mlir/test/Dialect/Vector/ops.mlir b/mlir/test/Dialect/Vector/ops.mlir
index e84bd3f1dce17..1b4109a3368ea 100644
--- a/mlir/test/Dialect/Vector/ops.mlir
+++ b/mlir/test/Dialect/Vector/ops.mlir
@@ -885,6 +885,16 @@ func.func @expand_and_compress(%base: memref<?xf32>, %mask: vector<16xi1>, %pass
   return
 }
 
+// CHECK-LABEL: @expand_and_compress_scalable
+func.func @expand_and_compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}], %{{.*}}, %{{.*}} : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}], %{{.*}}, %[[X]] : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0], %mask, %0 : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @expand_and_compress2d
 func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -895,6 +905,16 @@ func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %
   return
 }
 
+// CHECK-LABEL: @expand_and_compress2d_scalable
+func.func @expand_and_compress2d_scalable(%base: memref<?x?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %{{.*}} : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0, %c0], %mask, %pass_thru : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %[[X]] : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0, %c0], %mask, %0 : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @multi_reduction
 func.func @multi_reduction(%0: vector<4x8x16x32xf32>, %acc0: vector<4x16xf32>,
                            %acc1: f32) -> f32 {
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..ff978665125ed 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -733,6 +733,19 @@ func.func @cast_away_expandload_leading_one_dims(%base: memref<16xf32>, %i: inde
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_expandload_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[P:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         %[[L:.+]] = vector.expandload %{{.*}}[%{{.*}}], %[[M]], %[[P]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32> into vector<[4]xf32>
+// CHECK:         %[[B:.+]] = vector.broadcast %[[L]] : vector<[4]xf32> to vector<1x[4]xf32>
+// CHECK:         return %[[B]] : vector<1x[4]xf32>
+func.func @cast_away_expandload_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %pass: vector<1x[4]xf32>) -> vector<1x[4]xf32> {
+  %0 = vector.expandload %base[%i], %mask, %pass : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32> into vector<1x[4]xf32>
+  return %0 : vector<1x[4]xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_gather_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
@@ -779,6 +792,17 @@ func.func @cast_away_compressstore_leading_one_dims(%base: memref<16xf32>, %i: i
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_compressstore_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[V:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         vector.compressstore %{{.*}}[%{{.*}}], %[[M]], %[[V]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32>
+func.func @cast_away_compressstore_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %val: vector<1x[4]xf32>) {
+  vector.compressstore %base[%i], %mask, %val : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_scatter_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
diff --git a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
index 2004a47851e2e..cd7ea84cdb2eb 100644
--- a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
@@ -175,6 +175,20 @@ func.func @fold_expandload_all_true(%base: memref<16xf32>, %pass_thru: vector<16
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_true_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-DAG:       %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      %[[T:.*]] = vector.load %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return %[[T]] : vector<[16]xf32>
+func.func @fold_expandload_all_true_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 // CHECK-LABEL:   func @fold_expandload_all_false(
 // CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<16xf32>) -> vector<16xf32> {
@@ -187,6 +201,18 @@ func.func @fold_expandload_all_false(%base: memref<16xf32>, %pass_thru: vector<1
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_false_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-NEXT:      return %[[PASS_THRU]] : vector<[16]xf32>
+func.func @fold_expandload_all_false_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 //-----------------------------------------------------------------------------
 // [Pattern: CompressStoreFolder]
 //-----------------------------------------------------------------------------
@@ -204,6 +230,19 @@ func.func @fold_compressstore_all_true(%base: memref<16xf32>, %value: vector<16x
   return
 }
 
+// CHECK-LABEL:   func @fold_compressstore_all_true_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      vector.store %[[VALUE]], %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_true_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value  : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL:   func @fold_compressstore_all_false(
 // CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                    %[[VALUE:.*]]: vector<16xf32>) {
@@ -214,3 +253,14 @@ func.func @fold_compressstore_all_false(%base: memref<16xf32>, %value: vector<16
   vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<16xi1>, vector<16xf32>
   return
 }
+
+// CHECK-LABEL:   func @fold_compressstore_all_false_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_false_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
new file mode 100644
index 0000000000000..d8752cfb7fc92
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
@@ -0,0 +1,214 @@
+// REQUIRES: arm-emulator
+
+/// End-to-end test for vector.compressstore for SVE
+
+// In order to demonstrate the impact of using scalable vectors, vscale is set
+// to 2 so that vector<[16]xi32> constains 32 rather than 16 elements at
+// run-time
+//
+// Note that you can also tweak the size of vscale by passing this flag to
+// QEMU:
+//  * -cpu max,sve-max-vq=[1-16]
+// (select the value between 1 and 16).
+
+// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm
+// DEFINE: %{run} = %mcr_aarch64_cmd %t -e main -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils,%native_mlir_arm_runner_utils
+
+// RUN: rm -f %t && %{compile} &&  %{run} |  FileCheck %s
+
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// The number of inserted elements is 16 x vscale. Insertion index is
+// hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xi32>,
+                 %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %base[%c0], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xi32>,
+                     %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c8 = arith.constant 8: index
+  vector.compressstore %base[%c8], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xi32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xi32> to memref<*xi32>
+
+  call @printMemrefI32(%cast): (memref<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @reset_mem_i32
+//
+// Resets the input memory to 0.
+
+// TODO: Create a run-time utility funcion.
+//===----------------------------------------------------------------------===//
+func.func @reset_mem_i32(%ptr: memref<?xi32>, %size: index) {
+  %c0_idx = arith.constant 0: index
+  %c0 = arith.constant 0: i32
+  %step = arith.constant 1: index
+
+  scf.for %i = %c0_idx to %size step %step {
+    memref.store %c0, %ptr[%i] : memref<?xi32>
+  }
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point - sets the value of vscale.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Set vscale to 2 (vector width = 256). This will have identical effect to:
+  //  * qemu-aarch64 -cpu max,sve-max-vq=2 (...)
+  %c256 = arith.constant 256 : i32
+  func.call @setArmVLBits(%c256) : (i32) -> ()
+
+  // Run the tests.
+  func.call @test() : () -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @test
+//
+// Set-up and run tests.
+//===----------------------------------------------------------------------===//
+func.func @test() {
+  //
+  // Shared constants.
+  //
+  %vs = vector.vscale
+
+  //
+  // Set up memory.
+  //
+  %c16 = arith.constant 16: index
+  %vs_16 = arith.muli %vs, %c16 : index
+  %A = memref.alloc(%vs_16) : memref<?xi32>
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+
+  //
+  // Set the input vector.
+  //
+  %value = vector.step : vector<[16]xi32>
+  vector.print %value : vector<[16]xi32>
+
+  //
+  // Set up masks.
+  //
+
+  %f = arith.constant 0: i1
+  %t = arith.constant 1: i1
+
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %none = vector.constant_mask [0] : vector<[16]xi1>
+
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ..., 1]
+  %all = vector.constant_mask [16] : vector<[16]xi1>
+  vector.print %all : vector<[16]xi1>
+
+  // %first_vscale_4 = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %c4 = arith.constant 4 : index
+  %vs_4 = arith.muli %vs, %c4 : index
+  %first_vscale_4 = vector.create_mask %vs_4 : vector<[16]xi1>
+  vector.print %first_vscale_4 : vector<[16]xi1>
+
+  // %odd = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, ..., 0]
+  %0 = vector.insert %t, %none[1] : i1 into vector<[16]xi1>
+  %1 = vector.insert %t, %0[3] : i1 into vector<[16]xi1>
+  %2 = vector.insert %t, %1[5] : i1 into vector<[16]xi1>
+  %3 = vector.insert %t, %2[7] : i1 into vector<[16]xi1>
+  %4 = vector.insert %t, %3[9] : i1 into vector<[16]xi1>
+  %5 = vector.insert %t, %4[11] : i1 into vector<[16]xi1>
+  %odd = vector.insert %t, %5[13] : i1 into vector<[16]xi1>
+
+  // %even = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, ..., 0]
+  %6 = vector.insert %t, %none[0] : i1 into vector<[16]xi1>
+  %7 = vector.insert %t, %6[2] : i1 into vector<[16]xi1>
+  %8 = vector.insert %t, %7[4] : i1 into vector<[16]xi1>
+  %9 = vector.insert %t, %8[6] : i1 into vector<[16]xi1>
+  %10 = vector.insert %t, %9[8] : i1 into vector<[16]xi1>
+  %11 = vector.insert %t, %10[10] : i1 into vector<[16]xi1>
+  %even = vector.insert %t, %11[12] : i1 into vector<[16]xi1>
+
+
+  //
+  // Tests.
+  //
+
+  call @compress_16(%A, %none, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %all, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
+  /// (...)
+  // CHECK-SAME: 30, 31]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16_at_8(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %odd, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [1,  3,  5,  7,  9,  11,  13,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %even, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0,  2,  4,  6,  8,  10,  12,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  memref.dealloc %A : memref<?xi32>
+  return
+}
+
+func.func private @printMemrefI32(%ptr : memref<*xi32>)
+func.func private @setArmVLBits(%bits : i32)


From llvmlistbot at llvm.org  Fri Jul 17 02:58:02 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 02:58:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp` +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a59fcaa.e55dc341.1c6a0.15cc@mx.google.com>


banach-space wrote:

CC @FedericoBruzzone 

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Fri Jul 17 03:11:08 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 03:11:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp` +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a59ffbc.c56fb102.2d07b8.0168@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Extends `vector.compressstore` + `vector.expandload` to support scalable
vectors and updates relevant tests.


---

Patch is 25.68 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210288.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/Vector/IR/VectorOps.td (+2-2) 
- (modified) mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir (+41-12) 
- (modified) mlir/test/Dialect/Vector/invalid.mlir (-16) 
- (modified) mlir/test/Dialect/Vector/ops.mlir (+20) 
- (modified) mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir (+24) 
- (modified) mlir/test/Dialect/Vector/vector-mem-transforms.mlir (+50) 
- (added) mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir (+214) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index 6e0134fc0cdc6..ebfb379684dd3 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2289,7 +2289,7 @@ def Vector_ExpandLoadOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$pass_thru,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)>,
     Results<(outs AnyVectorOfNonZeroRank:$result)> {
@@ -2381,7 +2381,7 @@ def Vector_CompressStoreOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$valueToStore,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)> {
 
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
index cc35c6600d12c..4cbea04e8076e 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
@@ -1944,13 +1944,13 @@ func.func @negative_scatter_on_strided_memref(%arg0: memref<?xf32, strided<[2],
 // vector.expandload
 //===----------------------------------------------------------------------===//
 
-func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
+func.func @expandload(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32> into vector<11xf32>
   return %0 : vector<11xf32>
 }
 
-// CHECK-LABEL: func @expand_load_op
+// CHECK-LABEL: func @expandload
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1959,21 +1959,36 @@ func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vec
 
 // -----
 
-func.func @expand_load_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
+func.func @expandload_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) -> vector<[11]xf32> {
+  %c0 = arith.constant 0: index
+  %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32> into vector<[11]xf32>
+  return %0 : vector<[11]xf32>
+}
+
+// CHECK-LABEL: func @expandload_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: %[[E:.*]] = "llvm.intr.masked.expandload"(%[[P]], %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<[11]xi1>, vector<[11]xf32>) -> vector<[11]xf32>
+// CHECK: return %[[E]] : vector<[11]xf32>
+
+// -----
+
+func.func @expandload_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_index
+// CHECK-LABEL: func @expandload_index
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
 
-func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
+func.func @expandload_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_with_alignment
+// CHECK-LABEL: func @expandload_with_alignment
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{llvm.align = 8 : i64}, {}, {}]}> : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
@@ -1982,13 +1997,13 @@ func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<1
 // vector.compressstore
 //===----------------------------------------------------------------------===//
 
-func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
+func.func @compressstore(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32>
   return
 }
 
-// CHECK-LABEL: func @compress_store_op
+// CHECK-LABEL: func @compressstore
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1996,21 +2011,35 @@ func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2:
 
 // -----
 
-func.func @compress_store_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
+func.func @compressstore_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32>
+  return
+}
+
+// CHECK-LABEL: func @compressstore_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %[[P]], %{{.*}}) : (vector<[11]xf32>, !llvm.ptr, vector<[11]xi1>) -> ()
+
+// -----
+
+func.func @compressstore_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_index
+// CHECK-LABEL: func @compressstore_index
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
 
-func.func @compress_store_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
+func.func @compressstore_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
   vector.compressstore %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_with_alignment
+// CHECK-LABEL: func @compressstore_with_alignment
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{}, {llvm.align = 8 : i64}, {}]}> : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..d4f305f20b595 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1688,14 +1688,6 @@ func.func @expand_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>,
 
 // -----
 
-func.func @expand_base_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.expandload' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
-}
-
-// -----
-
 func.func @expand_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.expandload' op expected result shape to match mask shape}}
@@ -1750,14 +1742,6 @@ func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1
 
 // -----
 
-func.func @compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %value: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.compressstore' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
-}
-
-// -----
-
 func.func @compress_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op expected valueToStore shape to match mask shape}}
diff --git a/mlir/test/Dialect/Vector/ops.mlir b/mlir/test/Dialect/Vector/ops.mlir
index e84bd3f1dce17..1b4109a3368ea 100644
--- a/mlir/test/Dialect/Vector/ops.mlir
+++ b/mlir/test/Dialect/Vector/ops.mlir
@@ -885,6 +885,16 @@ func.func @expand_and_compress(%base: memref<?xf32>, %mask: vector<16xi1>, %pass
   return
 }
 
+// CHECK-LABEL: @expand_and_compress_scalable
+func.func @expand_and_compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}], %{{.*}}, %{{.*}} : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}], %{{.*}}, %[[X]] : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0], %mask, %0 : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @expand_and_compress2d
 func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -895,6 +905,16 @@ func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %
   return
 }
 
+// CHECK-LABEL: @expand_and_compress2d_scalable
+func.func @expand_and_compress2d_scalable(%base: memref<?x?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %{{.*}} : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0, %c0], %mask, %pass_thru : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %[[X]] : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0, %c0], %mask, %0 : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @multi_reduction
 func.func @multi_reduction(%0: vector<4x8x16x32xf32>, %acc0: vector<4x16xf32>,
                            %acc1: f32) -> f32 {
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..ff978665125ed 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -733,6 +733,19 @@ func.func @cast_away_expandload_leading_one_dims(%base: memref<16xf32>, %i: inde
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_expandload_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[P:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         %[[L:.+]] = vector.expandload %{{.*}}[%{{.*}}], %[[M]], %[[P]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32> into vector<[4]xf32>
+// CHECK:         %[[B:.+]] = vector.broadcast %[[L]] : vector<[4]xf32> to vector<1x[4]xf32>
+// CHECK:         return %[[B]] : vector<1x[4]xf32>
+func.func @cast_away_expandload_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %pass: vector<1x[4]xf32>) -> vector<1x[4]xf32> {
+  %0 = vector.expandload %base[%i], %mask, %pass : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32> into vector<1x[4]xf32>
+  return %0 : vector<1x[4]xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_gather_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
@@ -779,6 +792,17 @@ func.func @cast_away_compressstore_leading_one_dims(%base: memref<16xf32>, %i: i
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_compressstore_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[V:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         vector.compressstore %{{.*}}[%{{.*}}], %[[M]], %[[V]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32>
+func.func @cast_away_compressstore_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %val: vector<1x[4]xf32>) {
+  vector.compressstore %base[%i], %mask, %val : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_scatter_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
diff --git a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
index 2004a47851e2e..cd7ea84cdb2eb 100644
--- a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
@@ -175,6 +175,20 @@ func.func @fold_expandload_all_true(%base: memref<16xf32>, %pass_thru: vector<16
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_true_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-DAG:       %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      %[[T:.*]] = vector.load %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return %[[T]] : vector<[16]xf32>
+func.func @fold_expandload_all_true_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 // CHECK-LABEL:   func @fold_expandload_all_false(
 // CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<16xf32>) -> vector<16xf32> {
@@ -187,6 +201,18 @@ func.func @fold_expandload_all_false(%base: memref<16xf32>, %pass_thru: vector<1
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_false_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-NEXT:      return %[[PASS_THRU]] : vector<[16]xf32>
+func.func @fold_expandload_all_false_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 //-----------------------------------------------------------------------------
 // [Pattern: CompressStoreFolder]
 //-----------------------------------------------------------------------------
@@ -204,6 +230,19 @@ func.func @fold_compressstore_all_true(%base: memref<16xf32>, %value: vector<16x
   return
 }
 
+// CHECK-LABEL:   func @fold_compressstore_all_true_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      vector.store %[[VALUE]], %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_true_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value  : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL:   func @fold_compressstore_all_false(
 // CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                    %[[VALUE:.*]]: vector<16xf32>) {
@@ -214,3 +253,14 @@ func.func @fold_compressstore_all_false(%base: memref<16xf32>, %value: vector<16
   vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<16xi1>, vector<16xf32>
   return
 }
+
+// CHECK-LABEL:   func @fold_compressstore_all_false_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_false_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
new file mode 100644
index 0000000000000..d8752cfb7fc92
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
@@ -0,0 +1,214 @@
+// REQUIRES: arm-emulator
+
+/// End-to-end test for vector.compressstore for SVE
+
+// In order to demonstrate the impact of using scalable vectors, vscale is set
+// to 2 so that vector<[16]xi32> constains 32 rather than 16 elements at
+// run-time
+//
+// Note that you can also tweak the size of vscale by passing this flag to
+// QEMU:
+//  * -cpu max,sve-max-vq=[1-16]
+// (select the value between 1 and 16).
+
+// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm
+// DEFINE: %{run} = %mcr_aarch64_cmd %t -e main -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils,%native_mlir_arm_runner_utils
+
+// RUN: rm -f %t && %{compile} &&  %{run} |  FileCheck %s
+
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// The number of inserted elements is 16 x vscale. Insertion index is
+// hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xi32>,
+                 %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %base[%c0], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xi32>,
+                     %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c8 = arith.constant 8: index
+  vector.compre...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Fri Jul 17 03:13:10 2026
From: llvmlistbot at llvm.org (Max Graey)
Date: Fri, 17 Jul 2026 03:13:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a5a0036.dfa3d646.12b476.6219@mx.google.com>


https://github.com/MaxGraey commented:

LGTM

https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Fri Jul 17 03:44:28 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Fri, 17 Jul 2026 03:44:28 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5a078c.d1496011.199a59.c0b9@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 188886c54fe3febb5c784bfff2164da1d0f01d0e Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH 1/3] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 clang/lib/CodeGen/TargetInfo.cpp              |  3 +
 .../cl20-device-side-enqueue-attributes.cl    |  4 +-
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 50 +++++++-------
 22 files changed, 146 insertions(+), 136 deletions(-)

diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..af90be4b1a47b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26587,9 +26589,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 51c315cb13668..353e6aaedd0b6 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)

>From fe4c2b705dc9e7719d2acb19cd32e8bc7a865b6d Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 18:42:48 +0100
Subject: [PATCH 2/3] [LangRef/Verfier] Drop bad propagation terminology

---
 llvm/docs/LangRef.md     | 5 ++---
 llvm/lib/IR/Verifier.cpp | 2 +-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index af90be4b1a47b..9526cd020d724 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,9 +2794,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap. The
-    attribute propagates from callee to caller, and all function definitions
-    that contain strictfp calls must be marked strictfp.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 353e6aaedd0b6..a584f65f55eb5 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3891,7 +3891,7 @@ void Verifier::visitCallBase(CallBase &Call) {
 
   Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
             Call.getFunction()->isStrictFP(),
-        "strictfp attribute must propagate from the call site to the caller",
+        "call site marked strictfp without caller function marked strictfp",
         Call);
 
   // Verify call attributes.

>From 4de63ddcc7936fcf1227fa32bfc4875dee64d305 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Fri, 17 Jul 2026 11:01:45 +0100
Subject: [PATCH 3/3] [IR] Add Verifier test

---
 llvm/unittests/IR/VerifierTest.cpp | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/llvm/unittests/IR/VerifierTest.cpp b/llvm/unittests/IR/VerifierTest.cpp
index 48d461cab330a..f6a89cde0901b 100644
--- a/llvm/unittests/IR/VerifierTest.cpp
+++ b/llvm/unittests/IR/VerifierTest.cpp
@@ -699,4 +699,28 @@ TEST(VerifierTest, IntrinsicRetInvalidStruct) {
   }
 }
 
+TEST(VerifierTest, InvalidStrictFPAttribute) {
+  LLVMContext Ctx;
+  Module M("M", Ctx);
+  FunctionType *FuncTy =
+      FunctionType::get(Type::getVoidTy(Ctx), /*isVarArg=*/false);
+  Function *F =
+      Function::Create(FuncTy, Function::ExternalLinkage, "strictfp_test", M);
+  BasicBlock *Entry = BasicBlock::Create(Ctx, "entry", F);
+  Type *FloatTy = Type::getFloatTy(Ctx);
+
+  Function *CosF =
+      Intrinsic::getOrInsertDeclaration(&M, Intrinsic::cos, FloatTy);
+  CallInst *CI = CallInst::Create(CosF, ConstantFP::getNullValue(FloatTy),
+                                  "strictfp_call", Entry);
+  CI->addFnAttr(Attribute::StrictFP);
+  ReturnInst::Create(Ctx, Entry);
+
+  std::string Error;
+  raw_string_ostream ErrorOS(Error);
+  EXPECT_TRUE(verifyModule(M, &ErrorOS));
+  EXPECT_TRUE(StringRef(Error).starts_with(
+      "call site marked strictfp without caller function marked strictfp"))
+      << Error;
+}
 } // end anonymous namespace


From llvmlistbot at llvm.org  Fri Jul 17 03:48:52 2026
From: llvmlistbot at llvm.org (Fabian Mora)
Date: Fri, 17 Jul 2026 03:48:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ptr] Add nontemporal field to
	ptr.masked_load (PR #204987)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204987@github.com>
Message-ID: <6a5a0894.449bfb44.7b862.db86@mx.google.com>


fabianmcg wrote:

> This is indeed a bit confusing. The nontemporal flag is just metadata so nothing prevents us from adding it to masked loads / stores. However, it is indeed a good question if this is something that the LLVM backends take into account?
> 
> I see some SVE tests that use nontemporal metadata but it is unclear to me if that is intentional or just a random artifact of these tests.

I would argue the even if some backends support the metadata for optimizations, it's not a first class member and as such it shouldn't live as intrinsic attributes in the op. However, I'm not opposed to amend translation such that if it's found in the discardable attributes it gets passthrough LLVM. Having said that, IMO that's also fragile. I would try to rally LLVM to add the nontemporal flag as first class or get more information from LLVM before making a change.


https://github.com/llvm/llvm-project/pull/204987

From llvmlistbot at llvm.org  Fri Jul 17 04:15:55 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 17 Jul 2026 04:15:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Handle repeated EmitC operands (PR
	#206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a0eeb.1a3c332b.1edb2f.e3a4@mx.google.com>


https://github.com/aniragil commented:

Thanks for addressing this issue @mygitljf !

The bug is that the firstIndexOf each unique operand gets is its index in the original expression, not its index in the unique expression's. A simpler/smaller/immediate fix would be to just fix the `i` count:

```C++
    int i = 0;
    for (auto operand : expressionOp.getDefs()) {
      if (uniqueOperands.contains(operand))
        continue;
      uniqueOperands.insert(operand);
      firstIndexOf[operand] = i++;
    }
```

Your new mapping is more elegant than the indexed one, but since it's it's not a trivial fix as the above but replacing part of the mechanism it's hard to understand from the PR description what the bug was. So it would be great if you either:

- Commit the (above) more direct fix to the bug in this PR (assuming you agree it fixes the bug), then open another PR for replacing the indexed mechanism with your simpler mapping, or

- Keep this PR as-is, but elaborate in the PR description and commit message about the bug and about the new mapping that replaces the index-based mechanism, so future readers can easily follow the rationale.

it's not a major rewrite, so I'm ok with either option.


https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Fri Jul 17 04:37:50 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 17 Jul 2026 04:37:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a5a140e.492e23e4.d19bc.ca81@mx.google.com>


Devjiu wrote:

@joker-eph

Good point, and I think the code already does exactly what you're describing - for the patterns that fold an existing cast.

`FoldSourceTensorCast`, `ConvertToStaticExpandShape`, `FoldCollapseOfCastOp` all gate on `canFoldIntoConsumerOp` -> `preservesStaticInformation`, which requires the cast's source and target to have the *same* encoding:

```cpp
// Requires same encoding.
if (sourceType.getEncoding() != targetType.getEncoding())
  return false;
```

So the two situations are mutually exclusive: 
if a cast erases the encoding (source has it, target doesn't), the encodings differ, the gate returns false, and the cast is left in place - the erasure is preserved, exactly as you want. 

The fold only applies when both sides already carry the same encoding, i.e. when nothing was erased and there's nothing to respect. I confirmed this: a `tensor.cast` dropping the encoding in front of a tensor.pad is untouched by canonicalization.

That leaves `InsertSliceOpConstantArgumentFolder`, which is a different shape: there's no incoming cast at all - it applies because one of the `mixedSizes` became constant, and inserts a new cast from the original source operand. There's no prior erasure to respect, so propagating the source's own encoding is the only correct choice.

So maybe the split isn't category-1-vs-2 by "changes runtime value", but:
- fold-an-existing-cast patterns: already correct via `preservesStaticInformation` (same-encoding requirement), no change needed;
- insert-a-new-cast patterns (`InsertSliceOpConstantArgumentFolder`): propagate the operand's encoding, since there's no prior cast to defer to.

Does that reframing address the concern?

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul 17 04:58:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 04:58:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix repeated operand remapping in
	expressions (PR #206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a18d7.b4a331f0.d9742.2e3a@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Fri Jul 17 04:58:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 04:58:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix repeated operand remapping in
	expressions (PR #206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a18f3.481afadf.9e047.6d83@mx.google.com>


https://github.com/mygitljf edited https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Fri Jul 17 04:59:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 04:59:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix repeated operand remapping in
	expressions (PR #206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a1927.481afadf.9e047.94db@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/206001

>From 82bcf39326f28e527d15a8eafb2b12093b0f305b Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 26 Jun 2026 15:26:02 +0000
Subject: [PATCH 1/2] Handle repeated EmitC operands

---
 mlir/lib/Dialect/EmitC/IR/EmitC.cpp           | 11 +++++++----
 mlir/test/Dialect/EmitC/form-expressions.mlir | 15 +++++++++++++++
 2 files changed, 22 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
index 6e68a3d0c5a41..e0a9a556b6893 100644
--- a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
+++ b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
@@ -417,14 +417,12 @@ struct RemoveRecurringExpressionOperands
   LogicalResult matchAndRewrite(ExpressionOp expressionOp,
                                 PatternRewriter &rewriter) const override {
     SetVector<Value> uniqueOperands;
-    DenseMap<Value, int> firstIndexOf;
 
     // Collect duplicate operands and prepare to remove excessive copies.
-    for (auto [i, operand] : llvm::enumerate(expressionOp.getDefs())) {
+    for (Value operand : expressionOp.getDefs()) {
       if (uniqueOperands.contains(operand))
         continue;
       uniqueOperands.insert(operand);
-      firstIndexOf[operand] = i;
     }
 
     // If every operand is unique, bail out.
@@ -438,13 +436,18 @@ struct RemoveRecurringExpressionOperands
         uniqueOperands.getArrayRef(), expressionOp.getDoNotInline());
     Block &uniqueExpressionBody = uniqueExpression.createBody();
 
+    DenseMap<Value, BlockArgument> uniqueArgFor;
+    for (auto [operand, arg] : llvm::zip(uniqueExpression.getOperands(),
+                                         uniqueExpressionBody.getArguments()))
+      uniqueArgFor[operand] = arg;
+
     // Map each original block arguments to the unique block argument taking
     // the same operand.
     IRMapping mapper;
     Block *expressionBody = expressionOp.getBody();
     for (auto [operand, arg] :
          llvm::zip(expressionOp.getOperands(), expressionBody->getArguments()))
-      mapper.map(arg, uniqueExpressionBody.getArgument(firstIndexOf[operand]));
+      mapper.map(arg, uniqueArgFor[operand]);
 
     rewriter.setInsertionPointToStart(&uniqueExpressionBody);
     for (Operation &opToClone : *expressionOp.getBody())
diff --git a/mlir/test/Dialect/EmitC/form-expressions.mlir b/mlir/test/Dialect/EmitC/form-expressions.mlir
index b233766b65a1c..bc4c5106a85cb 100644
--- a/mlir/test/Dialect/EmitC/form-expressions.mlir
+++ b/mlir/test/Dialect/EmitC/form-expressions.mlir
@@ -39,6 +39,21 @@ func.func @expression_recurring_args(%arg0: i32, %arg1: i32) -> i1 {
   return %c : i1
 }
 
+// CHECK-LABEL:   func.func @expression_with_call_recurring_args(
+// CHECK-SAME:      %[[ARG0:.*]]: i32,
+// CHECK-SAME:      %[[ARG1:.*]]: i32) -> i32 {
+// CHECK:           %[[EXPRESSION_0:.*]] = emitc.expression %[[ARG0]], %[[ARG1]] : (i32, i32) -> i32 {
+// CHECK:             %[[VAL_0:.*]] = call_opaque "minsi"(%[[ARG0]], %[[ARG0]], %[[ARG1]]) : (i32, i32, i32) -> i32
+// CHECK:             yield %[[VAL_0]] : i32
+// CHECK:           }
+// CHECK:           return %[[EXPRESSION_0]] : i32
+// CHECK:         }
+
+func.func @expression_with_call_recurring_args(%arg0: i32, %arg1: i32) -> i32 {
+  %0 = emitc.call_opaque "minsi"(%arg0, %arg0, %arg1) : (i32, i32, i32) -> i32
+  return %0 : i32
+}
+
 // CHECK-LABEL: func.func @multiple_expressions(
 // CHECK-SAME:      %[[VAL_0:.*]]: i32, %[[VAL_1:.*]]: i32, %[[VAL_2:.*]]: i32, %[[VAL_3:.*]]: i32) -> (i32, i32) {
 // CHECK:         %[[VAL_4:.*]] = emitc.expression %[[VAL_2]], %[[VAL_0]], %[[VAL_1]] : (i32, i32, i32) -> i32 {

>From 615753bd08ac3d04a68df722b63ba3642a951fb3 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Fri, 17 Jul 2026 19:58:51 +0800
Subject: [PATCH 2/2] Explain EmitC operand mapping

---
 mlir/lib/Dialect/EmitC/IR/EmitC.cpp | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
index e0a9a556b6893..187b508ea1283 100644
--- a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
+++ b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
@@ -436,13 +436,15 @@ struct RemoveRecurringExpressionOperands
         uniqueOperands.getArrayRef(), expressionOp.getDoNotInline());
     Block &uniqueExpressionBody = uniqueExpression.createBody();
 
+    // Map operands directly because removing duplicates changes block argument
+    // indices, so indices from the original expression cannot be reused.
     DenseMap<Value, BlockArgument> uniqueArgFor;
     for (auto [operand, arg] : llvm::zip(uniqueExpression.getOperands(),
                                          uniqueExpressionBody.getArguments()))
       uniqueArgFor[operand] = arg;
 
-    // Map each original block arguments to the unique block argument taking
-    // the same operand.
+    // Map each original block argument to the unique block argument taking the
+    // same operand.
     IRMapping mapper;
     Block *expressionBody = expressionOp.getBody();
     for (auto [operand, arg] :


From llvmlistbot at llvm.org  Fri Jul 17 05:04:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 05:04:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix repeated operand remapping in
	expressions (PR #206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a1a67.b4a331f0.d9742.5cc3@mx.google.com>


mygitljf wrote:

Thanks @aniragil .
I kept the current mapping-based implementation. I've updated the PR description and commit message to better explain the bug and how the new mapping replaces the index-based mechanism.
If you would like me to use an immediate fix and submit a new pull request, please let me know.

https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Fri Jul 17 05:24:19 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Fri, 17 Jul 2026 05:24:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a5a1ef3.d76d81af.23b347.3462@mx.google.com>


joker-eph wrote:

I think that is reasonable, with one caveat: making an operation more static (turning an SSA value into an attribute) can generate invalid IR (the verifier would check the attribute where it didn't check the SSA value, I consider operation accepting either attributes or SSA value to be an anti-pattern by the way). 
The patterns are responsible to preserve correctness (that is checking the folding is valid), but with encoding you can't really know I think.

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul 17 07:02:00 2026
From: llvmlistbot at llvm.org (Jan Leyonberg)
Date: Fri, 17 Jul 2026 07:02:00 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a5a35d8.a1accbb4.5eca2.1aba@mx.google.com>


================
@@ -473,6 +473,15 @@ def OffloadModuleInterface : OpInterface<"OffloadModuleInterface"> {
            return ::llvm::dyn_cast<BoolAttr>(isTargetDevice).getValue();
         return false;
       }]>,
+    InterfaceMethod<
+      /*description=*/[{
+        Return true if the current module was generated with OpenMP enabled.
+        }],
+      /*retTy=*/"bool",
+      /*methodName=*/"getIsOpenMP",
+      (ins), [{}], [{
+        return $_op->hasAttr("omp.is_target_device");
+      }]>,
----------------
jsjodin wrote:

I updated the code to use the omp.version attribute and renamed the offloading utils file since it is also handling non-offloading things.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Fri Jul 17 07:02:03 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Fri, 17 Jul 2026 07:02:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a5a35db.3cce8983.3cf542.0747@mx.google.com>


Devjiu wrote:

@joker-eph

Right - folding an SSA size into the type means the verifier now checks what it couldn't before, and the pattern has to keep that valid.

For `VerifiableTensorEncoding`, `propagateEncoding` does exactly that: it runs `verifyEncoding` against the *refined* (more-static) shape - the one the verifier will now see - and drops the encoding if it no longer holds, so we can't emit invalid IR.

For opaque encodings there's nothing to check, but there's also nothing upstream *can* check: `RankedTensorType::verify` has no predicate for them, so they never make the type invalid regardless. Whether the metadata is still semantically right is the downstream's call - not implementing the interface is effectively the opt-out ("shape-agnostic / I own correctness").

Concrete proposal to unblock:
1. keep propagation for the insert-a-new-cast case ([this PR](https://github.com/llvm/llvm-project/pull/207239));
2. document the contract on `VerifiableTensorEncoding`: implement it =
   upstream verifies on transforms; don't = propagated as-is, author owns it.

If you'd rather bail out for opaque encodings by default, I can leave that - but it silently drops valid metadata in the common case. WDYT?

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul 17 07:18:50 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 17 Jul 2026 07:18:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a5a39ca.7d702ed2.13d102.250c@mx.google.com>


https://github.com/dcaballe approved this pull request.


https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Fri Jul 17 07:19:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 07:19:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1f2772f - [mlir][tosa] Add support for MXFP
	conv2d (#210054)
Message-ID: <6a5a39ef.b1416da1.3242c8.84f4@mx.google.com>


Author: Iliyan Georgiev
Date: 2026-07-17T15:19:23+01:00
New Revision: 1f2772f8e26beb909c2a559f1fb08697e3e06909

URL: https://github.com/llvm/llvm-project/commit/1f2772f8e26beb909c2a559f1fb08697e3e06909
DIFF: https://github.com/llvm/llvm-project/commit/1f2772f8e26beb909c2a559f1fb08697e3e06909.diff

LOG: [mlir][tosa] Add support for MXFP conv2d (#210054)

- Adds profile compliance support for MXFP conv2d
- Relax conv2d accumulator constraits. They are checked as part of
validation pass
and this would allow experimentation with different types.

Signed-off-by: Iliyan Georgiev <iliyan.georgiev at arm.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
    mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
    mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
    mlir/test/Dialect/Tosa/availability.mlir
    mlir/test/Dialect/Tosa/invalid.mlir
    mlir/test/Dialect/Tosa/invalid_extension.mlir
    mlir/test/Dialect/Tosa/ops.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
    mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
    mlir/test/Dialect/Tosa/verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
index f191da6583584..54cf204624937 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaComplianceData.h.inc
@@ -615,16 +615,991 @@ extensionComplianceMap = {
       {{Extension::fp8e4m3},
        {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e4m3T, fp8e4m3T, fp32T, fp8e4m3T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp32T, fp16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::fp8e5m2},
        {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp16T, fp16T},
          SpecificationVersion::V_1_0},
+        {{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
         {{fp8e5m2T, fp8e5m2T, fp32T, fp8e5m2T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp32T, fp16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T, fp32T},
          SpecificationVersion::V_1_1_DRAFT}}},
       {{Extension::bf16},
        {{{bf16T, bf16T, bf16T, bf16T, bf16T, fp32T, bf16T},
-         SpecificationVersion::V_1_0}}}}},
+         SpecificationVersion::V_1_0},
+        {{fp16T, fp16T, fp16T, fp16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp16T, fp16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp16T, bf16T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp16T, bf16T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bf16T, fp32T, fp16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp16T, fp32T, bf16T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bf16T, fp32T, bf16T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp32T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}}},
+      {{Extension::bf16, Extension::fp8e4m3},
+       {{{fp8e4m3T, fp8e4m3T, fp16T, fp8e4m3T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp16T, fp16T, fp8e4m3T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp16T, fp8e4m3T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e4m3T, fp16T, fp16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp16T, bf16T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bf16T, fp32T, fp8e4m3T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e4m3T, fp32T, bf16T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::fp8e5m2},
+       {{{fp8e4m3T, fp8e5m2T, fp16T, fp8e4m3T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp16T, fp8e5m2T, fp8e4m3T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, fp8e5m2T, fp32T, fp8e4m3T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp8e4m3T, fp32T, fp8e5m2T, fp8e4m3T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e4m3, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e4m3, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e4m3T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp16T, fp32T, fp8e4m3T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e4m3T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e4m3T, fp32T, fp32T, fp8e4m3T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2},
+       {{{fp8e5m2T, fp8e5m2T, fp16T, fp8e5m2T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, fp16T, fp16T, fp8e5m2T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp16T, fp8e5m2T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, fp8e5m2T, fp16T, fp16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp16T, bf16T, fp8e5m2T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bf16T, fp32T, fp8e5m2T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, fp8e5m2T, fp32T, bf16T, fp8e5m2T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::fp8e5m2, Extension::mx_common,
+        Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, bf16T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::fp8e5m2, Extension::mx_common, Extension::mx_int8},
+       {{{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp8e5m2T, fp32T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp16T, fp32T, fp8e5m2T, fp32T,
+          fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp8e5m2T, fp32T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp8e5m2T, fp32T, fp32T, fp8e5m2T, fp32T,
+          fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e4m3T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3},
+       {{{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp8e5m2T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2},
+       {{{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e3m2T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2},
+       {{{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp6e2m3T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3},
+       {{{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_fp4e2m1T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1},
+       {{{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp16T, bf16T, fp32T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp16T, bf16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp16T, fp32T, bf16T, bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp16T, fp32T, bf16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bf16T, bs32_fp8ue8m0_mxint8T, fp32T, bf16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bf16T, fp32T, fp32T, bf16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_int8},
+       {{{fp16T, bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp16T, fp32T, fp16T, fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp16T, bs32_fp8ue8m0_mxint8T, fp32T, fp16T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{fp32T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T, fp16T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T, fp32T, fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e4m3},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e4m3,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e4m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e4m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e4m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp8e5m2},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp8e5m2,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp8e5m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp8e5m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp8e5m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e3m2},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e3m2,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e3m2, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e3m2T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e3m2T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_fp6e2m3},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp6e2m3,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp6e2m3, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp6e2m3T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp6e2m3T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::bf16, Extension::mx_common, Extension::mx_fp4e2m1,
+        Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          bf16T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf},
+      {{Extension::mx_common, Extension::mx_fp4e2m1, Extension::mx_int8},
+       {{{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp16T, fp32T, fp32T,
+          fp32T, fp16T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_fp4e2m1T, bs32_fp8ue8m0_mxint8T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT},
+        {{bs32_fp8ue8m0_mxint8T, bs32_fp8ue8m0_fp4e2m1T, fp32T, fp32T, fp32T,
+          fp32T, fp32T},
+         SpecificationVersion::V_1_1_DRAFT}},
+       allOf}}},
     {"tosa.conv2d_block_scaled",
      {{{Extension::mxfp_conv},
        {{{fp4e2m1T, fp8ue8m0T, fp4e2m1T, fp8ue8m0T, fp32T, fp32T},

diff  --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
index 66521cbf73db8..7136d629a751c 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.td
@@ -68,7 +68,7 @@ def Tosa_ArgMaxOp : Tosa_InferShapedTypeOp<"argmax", [NoMemoryEffect]> {
 // Accumulator types.
 //===----------------------------------------------------------------------===//
 
-def Tosa_AccType : AnyTypeOf<[I<32>, I<48>, F16, F32]>;
+def Tosa_AccType : AnyTypeOf<[I<32>, I<48>, F16, F32, BF16]>;
 
 //===----------------------------------------------------------------------===//
 // Operator: avg_pool2d
@@ -203,7 +203,11 @@ def Tosa_Conv2DOp : Tosa_ConvOp<"conv2d"> {
 
   list<Availability> availability = [
     Profile<[Tosa_PRO_INT, Tosa_PRO_FP]>,
-    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3, Tosa_EXT_FP8E5M2, Tosa_EXT_BF16]>,
+    Extension<[Tosa_EXT_INT4, Tosa_EXT_INT16, Tosa_EXT_FP8E4M3,
+               Tosa_EXT_FP8E5M2, Tosa_EXT_BF16, Tosa_EXT_MX_COMMON,
+               Tosa_EXT_MX_FP4E2M1, Tosa_EXT_MX_FP6E2M3,
+               Tosa_EXT_MX_FP6E3M2, Tosa_EXT_MX_FP8E4M3,
+               Tosa_EXT_MX_FP8E5M2, Tosa_EXT_MX_INT8]>,
   ];
 
   let extraClassDeclaration = [{
@@ -221,6 +225,8 @@ def Tosa_Conv2DOp : Tosa_ConvOp<"conv2d"> {
 
 //===----------------------------------------------------------------------===//
 // Operator: conv2d_block_scaled
+//
+// Note: This operation is deprecated. It will be removed in the future.
 //===----------------------------------------------------------------------===//
 def Tosa_Conv2DBlockScaledOp : Tosa_InferShapedTypeOp<"conv2d_block_scaled", [NoMemoryEffect]> {
   let summary = "Performs two dimensional convolution using block scaled tensors.";
@@ -232,6 +238,8 @@ def Tosa_Conv2DBlockScaledOp : Tosa_InferShapedTypeOp<"conv2d_block_scaled", [No
 
     This operation is not pure. Undefined behaviour may occur if the accumulated
     result overflows.
+
+    // Note: This operation is deprecated. It will be removed in the future.
   }];
 
   let arguments = (ins
@@ -463,7 +471,7 @@ def Tosa_MatMulTOp : Tosa_InferShapedTypeOp<"matmul_t", [NoMemoryEffect]> {
   let description = [{
     Performs two dimensional matrix multiplications. `A` matrix is of shape
     `N x H x C`. `B` matrix is of shape `D x W x C`. This is effectively a
-    matrix multiply of `A` by the transposed `B` matrix. If the batched 
+    matrix multiply of `A` by the transposed `B` matrix. If the batched
     dimension of input `B` is of size 1, the `B` matrix is broadcast.
   }];
 
@@ -2597,7 +2605,7 @@ def Tosa_RowGatherOp : Tosa_InferShapedTypeOp<"row_gather", [NoMemoryEffect]> {
 
   let description = [{
     Generate a tensor based on the input indices and row_count. The number of
-    consecutive rows gathered for each index is specified in row_count. N is 
+    consecutive rows gathered for each index is specified in row_count. N is
     the number of batches, W is the number of indices in each batch, K is
     the range of each index, and C is the number of data channels for each
     index. The values tensor has shape [N, K, C] and the output tensor has shape

diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index 0b6abc2df3a70..0dfcd23504019 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -888,21 +888,16 @@ static LogicalResult verifyConvOp(T op) {
     return failure();
   }
 
-  if (isa<Float8E5M2Type>(inputEType) || isa<Float8E4M3FNType>(inputEType) ||
-      isa<Float8E5M2Type>(weightEType) || isa<Float8E4M3FNType>(weightEType)) {
-    if (inputEType != weightEType) {
-      op.emitOpError(
-          "expect both input and weight to have same element type, got ")
-          << inputEType << " and " << weightEType;
-      return failure();
-    }
-  }
+  const bool isInputBlockScaled = llvm::isa<BlockScaledType>(inputEType);
+  const bool isWeightBlockScaled = llvm::isa<BlockScaledType>(weightEType);
+  const bool isInputFloat = llvm::isa<FloatType>(inputEType);
+  const bool isWeightFloat = llvm::isa<FloatType>(weightEType);
 
-  bool inputIsFloat = llvm::isa<FloatType>(inputEType);
-  bool weightIsFloat = llvm::isa<FloatType>(weightEType);
+  const bool isInputBSorFloat = isInputBlockScaled || isInputFloat;
+  const bool isWeightBSorFloat = isWeightBlockScaled || isWeightFloat;
 
   // Either both must be float or both non-float.
-  if (inputIsFloat != weightIsFloat) {
+  if (isInputBSorFloat != isWeightBSorFloat) {
     op.emitOpError(
         "expect both input and weight to be float or not together, got ")
         << inputEType << " and " << weightEType;
@@ -910,18 +905,28 @@ static LogicalResult verifyConvOp(T op) {
   }
 
   auto inputZpEType = getStorageElementTypeOrSelf(op.getInputZp().getType());
-  if (inputEType != inputZpEType) {
+  if (!isInputBlockScaled && inputEType != inputZpEType) {
     return op.emitOpError("expect both input and its zero point are the same "
                           "element type, got ")
            << inputEType << " and " << inputZpEType;
   }
+  if (isInputBlockScaled && !llvm::isa<Float32Type>(inputZpEType)) {
+    return op.emitOpError(
+               "expect block scaled input to have fp32 zero point, got ")
+           << inputEType << " and " << inputZpEType;
+  }
 
   auto weightZpEType = getStorageElementTypeOrSelf(op.getWeightZp().getType());
-  if (weightEType != weightZpEType) {
+  if (!isWeightBlockScaled && weightEType != weightZpEType) {
     return op.emitOpError("expect both weight and its zero point are the same "
                           "element type, got ")
            << weightEType << " and " << weightZpEType;
   }
+  if (isWeightBlockScaled && !llvm::isa<Float32Type>(weightZpEType)) {
+    return op.emitOpError(
+               "expect block scaled weight to have fp32 zero point, got ")
+           << weightEType << " and " << weightZpEType;
+  }
 
   FailureOr<int64_t> maybeIZp = op.getInputZeroPoint();
   if (succeeded(maybeIZp) && op.verifyInputZeroPoint(*maybeIZp).failed())
@@ -998,33 +1003,6 @@ static LogicalResult verifyConvOpModes(T op) {
   if (auto quantType = llvm::dyn_cast<mlir::quant::QuantizedType>(inputEType))
     inputEType = getStorageElementTypeFromQuantized(quantType);
 
-  auto accType = op.getAccType();
-  if (inputEType.isInteger(8) && !accType.isInteger(32))
-    return op.emitOpError("accumulator type for i8 tensor is not i32, got ")
-           << accType;
-
-  if (inputEType.isInteger(16) && !accType.isInteger(48))
-    return op.emitOpError("accumulator type for i16 tensor is not i48, got ")
-           << accType;
-
-  if (isa<Float8E5M2Type, Float8E4M3Type>(inputEType) &&
-      !(accType.isF16() || accType.isF32()))
-    return op.emitOpError("accumulator type for f8 tensor is not f16/f32, got ")
-           << accType;
-
-  if (inputEType.isF16() && !(accType.isF16() || accType.isF32()))
-    return op.emitOpError(
-               "accumulator type for f16 tensor is not f16/f32, got ")
-           << accType;
-
-  if (inputEType.isBF16() && !accType.isF32())
-    return op.emitOpError("accumulator type for bf16 tensor is not f32, got ")
-           << accType;
-
-  if (inputEType.isF32() && !accType.isF32())
-    return op.emitOpError("accumulator type for f32 tensor is not f32, got ")
-           << accType;
-
   auto resultEType =
       llvm::cast<ShapedType>(op.getResult().getType()).getElementType();
 

diff  --git a/mlir/test/Dialect/Tosa/availability.mlir b/mlir/test/Dialect/Tosa/availability.mlir
index 04277c4ea9bfb..eb571f9a163d7 100644
--- a/mlir/test/Dialect/Tosa/availability.mlir
+++ b/mlir/test/Dialect/Tosa/availability.mlir
@@ -43,7 +43,7 @@ func.func @test_avg_pool2d_adaptive(%arg0: tensor<1x7x7x9xf32>) -> tensor<1x7x7x
 // CHECK-LABEL: conv2d
 func.func @test_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<8x1x1x4xf32>, %arg2: tensor<8xf32>) -> tensor<1x4x4x8xf32> {
   // CHECK: profiles: [ [pro_int, pro_fp] ]
-  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16] ]
+  // CHECK: extensions: [ [int4, int16, fp8e4m3, fp8e5m2, bf16, mx_common, mx_fp4e2m1, mx_fp6e2m3, mx_fp6e3m2, mx_fp8e4m3, mx_fp8e5m2, mx_int8] ]
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
   %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x4xf32>, tensor<8x1x1x4xf32>, tensor<8xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf32>

diff  --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..e559ea5f3c0ab 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -74,105 +74,6 @@ func.func @test_conv2d_weight_zp(%arg0: tensor<1x29x29x4xf16>, %arg1: tensor<16x
 
 // -----
 
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xi8>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi8>) -> tensor<1x27x27x16xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xi8>, tensor<16x3x3x4xi8>, tensor<16xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x27x27x16xi8>
-  return %0 : tensor<1x27x27x16xi8>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xi16>, %arg1: tensor<16x3x3x4xi8>, %arg2: tensor<16xi16>) -> tensor<1x27x27x16xi16> {
-  %input_zp = "tosa.const"() {values = dense<0> : tensor<1xi16>} : () -> tensor<1xi16>
-  %weight_zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i16 tensor is not i48, got 'f16'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xi16>, tensor<16x3x3x4xi8>, tensor<16xi16>, tensor<1xi16>, tensor<1xi8>) -> tensor<1x27x27x16xi16>
-  return %0 : tensor<1x27x27x16xi16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf8E5M2>, %arg1: tensor<16x3x3x4xf8E5M2>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf8E5M2>} : () -> tensor<1xf8E5M2>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f8 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf8E5M2>, tensor<16x3x3x4xf8E5M2>, tensor<16xf16>, tensor<1xf8E5M2>, tensor<1xf8E5M2>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf8E4M3>, %arg1: tensor<16x3x3x4xf8E4M3>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf8E4M3>} : () -> tensor<1xf8E4M3>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f8 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf8E4M3>, tensor<16x3x3x4xf8E4M3>, tensor<16xf16>, tensor<1xf8E4M3>, tensor<1xf8E4M3>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf16>, %arg1: tensor<16x3x3x4xf16>, %arg2: tensor<16xf16>) -> tensor<1x27x27x16xf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf16>} : () -> tensor<1xf16>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f16 tensor is not f16/f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf16>, tensor<16x3x3x4xf16>, tensor<16xf16>, tensor<1xf16>, tensor<1xf16>) -> tensor<1x27x27x16xf16>
-  return %0 : tensor<1x27x27x16xf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xbf16>, %arg1: tensor<16x3x3x4xbf16>, %arg2: tensor<16xbf16>) -> tensor<1x27x27x16xbf16> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xbf16>} : () -> tensor<1xbf16>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for bf16 tensor is not f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xbf16>, tensor<16x3x3x4xbf16>, tensor<16xbf16>, tensor<1xbf16>, tensor<1xbf16>) -> tensor<1x27x27x16xbf16>
-  return %0 : tensor<1x27x27x16xbf16>
-}
-
-// -----
-
-func.func @test_conv2d_acc_type(%arg0: tensor<1x29x29x4xf32>, %arg1: tensor<16x3x3x4xf32>, %arg2: tensor<16xf32>) -> tensor<1x27x27x16xf32> {
-  %zp = "tosa.const"() {values = dense<0.0> : tensor<1xf32>} : () -> tensor<1xf32>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for f32 tensor is not f32, got 'i32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = i32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>}
-           : (tensor<1x29x29x4xf32>, tensor<16x3x3x4xf32>, tensor<16xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x27x27x16xf32>
-  return %0 : tensor<1x27x27x16xf32>
-}
-
-// -----
-
-func.func @test_conv3d_acc_type(%arg0: tensor<1x4x8x21x17xi8>, %arg1: tensor<34x1x1x1x17xi8>, %arg2: tensor<34xi8>) -> tensor<1x4x8x21x34xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv3d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.conv3d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1, 1>, pad = array<i64: 0, 0, 0, 0, 0, 0>, stride = array<i64: 1, 1, 1>}
-           : (tensor<1x4x8x21x17xi8>, tensor<34x1x1x1x17xi8>, tensor<34xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x8x21x34xi8>
-  return %0 : tensor<1x4x8x21x34xi8>
-}
-
-// -----
-
-func.func @test_depthwise_conv2d_acc_type(%arg0: tensor<1x4x4x4xi8>, %arg1: tensor<1x1x4x2xi8>, %arg2: tensor<8xi8>) -> tensor<1x4x4x8xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.depthwise_conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.depthwise_conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x4x4x4xi8>, tensor<1x1x4x2xi8>, tensor<8xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x4x8xi8>
-  return %0 : tensor<1x4x4x8xi8>
-}
-
-// -----
-
-func.func @test_transpose_conv2d(%arg0: tensor<1x32x32x8xi8>, %arg1: tensor<16x1x1x8xi8>, %arg2: tensor<16xi8>) -> tensor<1x32x32x16xi8> {
-  %zp = "tosa.const"() {values = dense<0> : tensor<1xi8>} : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.transpose_conv2d' op accumulator type for i8 tensor is not i32, got 'f16'}}
-  %0 = tosa.transpose_conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f16, out_pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xi8>, tensor<16x1x1x8xi8>, tensor<16xi8>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x32x32x16xi8>
-  return %0 : tensor<1x32x32x16xi8>
-}
-
-// -----
-
 func.func @test_transpose_conv2d_invalid_padding_top(%arg0: tensor<1x32x32x8xf32>, %arg1: tensor<16x1x1x8xf32>, %arg2: tensor<16xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x32x32x16xf32> {
   // expected-error at +1 {{'tosa.transpose_conv2d' op expected out_pad_top > -KH, but got: out_pad_top=-3 and KH=1}}
   %0 = tosa.transpose_conv2d %arg0, %arg1, %arg2, %arg3, %arg4 {acc_type = f32, out_pad = array<i64: -3, 0, 0, 0>, out_shape = array<i64: 1, 32, 32, 16>, stride = array<i64: 1, 1>} : (tensor<1x32x32x8xf32>, tensor<16x1x1x8xf32>, tensor<16xf32>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x32x32x16xf32>
@@ -243,15 +144,6 @@ func.func @test_transpose_conv2d_invalid_bias(%arg0: tensor<1x32x32x8xf32>, %arg
   return %0 : tensor<1x32x32x16xf32>
 }
 
-// -----
-// CHECK-LABEL: conv2d_quant_any_acc
-func.func @test_conv2d_quant_any_acc(%arg0: tensor<1x4x4x4x!quant.any<i8<-8:7>>>, %arg1: tensor<8x1x1x4x!quant.any<i8<-8:7>>>, %arg2: tensor<8x!quant.any<i8<-8:7>>>) -> tensor<1x4x4x8x!quant.any<i8<-8:7>>> {
-  %zp = "tosa.const" () { values = dense<0> : tensor<1xi8> } : () -> tensor<1xi8>
-  // expected-error at +1 {{'tosa.conv2d' op accumulator type for i8 tensor is not i32, got 'f32'}}
-  %0 = tosa.conv2d %arg0, %arg1, %arg2, %zp, %zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x4x!quant.any<i8<-8:7>>>, tensor<8x1x1x4x!quant.any<i8<-8:7>>>, tensor<8x!quant.any<i8<-8:7>>>, tensor<1xi8>, tensor<1xi8>) -> tensor<1x4x4x8x!quant.any<i8<-8:7>>>
-  return %0 : tensor<1x4x4x8x!quant.any<i8<-8:7>>>
-}
-
 // -----
 // CHECK-LABEL: conv2d_quant_any
 func.func @test_conv2d_quant_any(%arg0: tensor<1x4x4x4x!quant.any<i8<-8:7>>>, %arg1: tensor<8x1x1x4x!quant.any<i8<-8:7>>>, %arg2: tensor<8x!quant.any<i32<-8:7>>>) -> tensor<1x4x4x8x!quant.any<i32<-8:7>>> {

diff  --git a/mlir/test/Dialect/Tosa/invalid_extension.mlir b/mlir/test/Dialect/Tosa/invalid_extension.mlir
index 94574dea21d01..fe17b67465fb3 100644
--- a/mlir/test/Dialect/Tosa/invalid_extension.mlir
+++ b/mlir/test/Dialect/Tosa/invalid_extension.mlir
@@ -25,6 +25,15 @@ func.func @test_conv2d(%arg0: tensor<1x4x4x4xi8>, %arg1: tensor<8x1x1x4xi4>, %ar
   return %0 : tensor<1x4x4x8xi32>
 }
 
+// -----
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op illegal: requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
 // -----
 func.func @test_conv3d(%arg0: tensor<1x4x8x21x17xi16>, %arg1: tensor<34x1x1x1x17xi8>, %arg2: tensor<34xi48>, %arg3: tensor<1xi16>, %arg4: tensor<1xi8>) -> tensor<1x4x8x21x34xi48> {
   // expected-error at +1 {{'tosa.conv3d' op illegal: requires any of [int16] profiles/extensions to be specified in the target environment}}

diff  --git a/mlir/test/Dialect/Tosa/ops.mlir b/mlir/test/Dialect/Tosa/ops.mlir
index b2c9bcfdb2be1..998826da567f3 100644
--- a/mlir/test/Dialect/Tosa/ops.mlir
+++ b/mlir/test/Dialect/Tosa/ops.mlir
@@ -142,6 +142,15 @@ func.func @test_conv2d(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<8x1x1x4xf32>, %
   return %0 : tensor<1x4x4x8xf32>
 }
 
+// -----
+// CHECK-LABEL: conv2d_mxfp
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
 // -----
 // CHECK-LABEL: conv2d_unranked_input
 func.func @test_conv2d_unranked_input(%arg0: tensor<*xf32>, %arg1: tensor<8x1x1x4xf32>, %arg2: tensor<8xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x4x4x8xf32> {

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
index b80705348b46b..4623ed167a1b8 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p0-invalid.mlir
@@ -42,6 +42,16 @@ func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x
 
 // -----
 
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op illegal: requires specification version compatible with 1.1.draft (got 1.0) and requires all of [bf16, mx_common, mx_fp4e2m1] profiles/extensions to be specified in the target environment}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 func.func @test_conv3d_fp8_acc32(%arg0: tensor<1x4x8x21x17xf8E5M2>, %arg1: tensor<34x1x1x1x17xf8E5M2>, %arg2: tensor<34xf32>) -> tensor<1x4x8x21x34xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
   %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>

diff  --git a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
index e3eb88b10bded..4c48d3f12d645 100644
--- a/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
+++ b/mlir/test/Dialect/Tosa/tosa-validation-version-1p1-valid.mlir
@@ -72,6 +72,24 @@ func.func @test_conv2d_fp8_acc32(%arg0: tensor<1x4x4x4xf8E5M2>, %arg1: tensor<8x
 
 // -----
 
+// CHECK-LABEL: test_conv2d_mxfp
+func.func @test_conv2d_mxfp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// CHECK-LABEL: test_conv2d_mxfp_acc32
+func.func @test_conv2d_mxfp_acc32(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 // CHECK-LABEL: test_conv3d_fp8_acc32
 func.func @test_conv3d_fp8_acc32(%arg0: tensor<1x4x8x21x17xf8E5M2>, %arg1: tensor<34x1x1x1x17xf8E5M2>, %arg2: tensor<34xf32>) -> tensor<1x4x8x21x34xf32> {
   %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf8E5M2>}> : () -> tensor<1xf8E5M2>
@@ -575,7 +593,7 @@ func.func @test_assert_equal_shape() {
 }
 
 // -----
-func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> { 
+func.func @test_maxpool2d_adaptive(%arg0: tensor<1x32x32x8xf32>) -> tensor<1x32x32x8xf32> {
   %kernel = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
   %stride = tosa.const_shape {values = dense<[1, 1]> : tensor<2xindex>} : () -> !tosa.shape<2>
   %pad = tosa.const_shape {values = dense<[0, 0, 0, 0]> : tensor<4xindex>} : () -> !tosa.shape<4>

diff  --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index 6fc88e4bd7694..96ce19f34aee5 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -244,6 +244,26 @@ func.func @test_slice_output_shape_mismatch_dynamic(%arg0: tensor<?x5x6xf32>) {
 
 // -----
 
+func.func @test_conv2d_mxfp_invalid_weight_zp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  // expected-error at +1 {{'tosa.conv2d' op expect block scaled weight to have fp32 zero point}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf32>, tensor<1xf16>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
+func.func @test_conv2d_mxfp_invalid_input_zp(%arg0: tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg1: tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, %arg2: tensor<8xf16>) -> tensor<1x4x4x8xf16> {
+  %input_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf16>}> : () -> tensor<1xf16>
+  %weight_zp = "tosa.const"() <{values = dense<0.0> : tensor<1xf32>}> : () -> tensor<1xf32>
+  // expected-error at +1 {{'tosa.conv2d' op expect block scaled input to have fp32 zero point}}
+  %0 = tosa.conv2d %arg0, %arg1, %arg2, %input_zp, %weight_zp {acc_type = bf16, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, 0, 0>, stride = array<i64: 1, 1>, local_bound = true} : (tensor<1x4x4x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8x1x1x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f4E2M1FN>>, tensor<8xf16>, tensor<1xf16>, tensor<1xf32>) -> tensor<1x4x4x8xf16>
+  return %0 : tensor<1x4x4x8xf16>
+}
+
+// -----
+
 func.func @test_depthwise_conv2d_invalid_padding(%arg0: tensor<1x4x4x4xf32>, %arg1: tensor<1x1x8x4xf32>, %arg2: tensor<8xf32>, %arg3: tensor<1xf32>, %arg4: tensor<1xf32>) -> tensor<1x4x4x8xf32> {
   // expected-error at +1 {{'tosa.depthwise_conv2d' op expect all padding values to be >= 0, got 0, 0, -1, 0}}
   %0 = tosa.depthwise_conv2d %arg0, %arg1, %arg2, %arg3, %arg4 {acc_type = f32, dilation = array<i64: 1, 1>, pad = array<i64: 0, 0, -1, 0>, stride = array<i64: 1, 1>, local_bound = true}


        

From llvmlistbot at llvm.org  Fri Jul 17 07:19:30 2026
From: llvmlistbot at llvm.org (Iliyan Georgiev)
Date: Fri, 17 Jul 2026 07:19:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add support for MXFP conv2d (PR
	#210054)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210054@github.com>
Message-ID: <6a5a39f2.66c0679a.24976c.09ee@mx.google.com>


https://github.com/iliyan-georgiev-arm closed https://github.com/llvm/llvm-project/pull/210054

From llvmlistbot at llvm.org  Fri Jul 17 08:07:50 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 17 Jul 2026 08:07:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a5a4546.bd93e731.30f155.f27f@mx.google.com>


================
@@ -112,6 +112,10 @@ static FailureOr<int> getOperatorPrecedence(Operation *operation) {
       .Case([&](emitc::MemberOfPtrOp op) { return 17; })
       .Case([&](emitc::MemberOp op) { return 17; })
       .Case([&](emitc::MulOp op) { return 13; })
+      .Case([&](emitc::PostDecrementOp op) { return 16; })
+      .Case([&](emitc::PostIncrementOp op) { return 16; })
+      .Case([&](emitc::PreDecrementOp op) { return 15; })
+      .Case([&](emitc::PreIncrementOp op) { return 15; })
----------------
aniragil wrote:

Shouldn't post-ops' precedence be 17?


https://github.com/llvm/llvm-project/pull/208648

From llvmlistbot at llvm.org  Fri Jul 17 08:07:50 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 17 Jul 2026 08:07:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add increment and decrement ops
	(PR #208648)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208648@github.com>
Message-ID: <6a5a4546.e55dc341.1c6a0.87e5@mx.google.com>


================
@@ -148,6 +148,15 @@ func.func @unary(%arg0: i32) {
   return
 }
 
+func.func @inc_dec(%arg0: !emitc.ptr<i32>) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %0 = emitc.pre_increment %v : !emitc.lvalue<i32>
+  %1 = emitc.post_increment %v : !emitc.lvalue<i32>
+  %2 = emitc.pre_decrement %v : !emitc.lvalue<i32>
+  %3 = emitc.post_decrement %v : !emitc.lvalue<i32>
----------------
aniragil wrote:

IINM the pre-operators in C++ return an lvalue. I'm note sure if/how that's a problem ATM, but it may be worth documenting in the op's tablegen comment.

https://github.com/llvm/llvm-project/pull/208648

From llvmlistbot at llvm.org  Fri Jul 17 08:22:53 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Fri, 17 Jul 2026 08:22:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a5a48cd.8cba1354.350ceb.44fc@mx.google.com>


joker-eph wrote:

> If you'd rather bail out for opaque encodings by default, I can leave that - but it silently drops valid metadata in the common case. WDYT?


Can you clarify how bailing out would drop anything? I would think that bailing out leaves the IR totally unchanged?

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Fri Jul 17 08:28:23 2026
From: llvmlistbot at llvm.org (Gil Rapaport)
Date: Fri, 17 Jul 2026 08:28:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix repeated operand remapping in
	expressions (PR #206001)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206001@github.com>
Message-ID: <6a5a4a17.fae8f0c5.6d323.5012@mx.google.com>


https://github.com/aniragil approved this pull request.

New description looks great, thanks @mygitljf !

https://github.com/llvm/llvm-project/pull/206001

From llvmlistbot at llvm.org  Fri Jul 17 08:33:06 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 08:33:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a5a4b32.b3e79337.dbe49.f670@mx.google.com>


https://github.com/banach-space updated https://github.com/llvm/llvm-project/pull/208752

>From 643775665538881e17a0ddb2337ae7f170a30bcb Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 10 Jul 2026 14:56:23 +0000
Subject: [PATCH 1/3] [mlir][vector] Extend `combineContractAndBroadcast` to
 accept `vector.shape_cast`

`combineContractAndBroadcast` is a pattern that matches `vector.contract`
operations whose inputs are defined by `vector.broadcast`, e.g.:

```mlir
%0 = vector.broadcast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.broadcast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

The pattern folds away the reduction over a unit dimension introduced by
the defining operations, producing a simplified `vector.contract`:

```mlir
vector.contract {
  indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
  iterator_types = ["parallel", "parallel", "reduction"]
} %0, %1, %acc : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
```

This change extends the pattern to also accept broadcast-like
`vector.shape_cast` operations:

```mlir
%0 = vector.shape_cast %lhs : vector<8x4xi32> to vector<1x8x4xi32>
%1 = vector.shape_cast %rhs : vector<8x4xi32> to vector<1x8x4xi32>
%result = vector.contract {
    indexing_maps = [#map0, #map1, #map2],
    iterator_types = ["reduction", "parallel", "parallel", "reduction"],
    kind = #vector.kind<add>
  } %0, %1, %acc : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
```

Note that `mlir::vector::isBroadcastableTo` was only moved to make it
available to the newly introduced `ShapeCastOp` helper,
`ShapeCastOp::isBroadcastLike()`.
---
 .../mlir/Dialect/Vector/IR/VectorOps.h        |   2 +-
 .../mlir/Dialect/Vector/IR/VectorOps.td       |   3 +
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 121 ++++++++++--------
 .../Vector/Transforms/VectorTransforms.cpp    |  36 ++++--
 .../Vector/vector-reduce-to-contract.mlir     | 103 ++++++++++++++-
 5 files changed, 194 insertions(+), 71 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
index b3a0653b90765..1a7d37d989f5a 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.h
@@ -67,7 +67,7 @@ enum class ConstantMaskKind { AllFalse = 0, AllTrue };
 /// arguments.
 void buildTerminatedBody(OpBuilder &builder, Location loc);
 
-/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// Models whether `srcType` can be broadcast to `dstVectorType` under the
 /// semantics of the `vector.broadcast` op.
 enum class BroadcastableToResult {
   Success = 0,
diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index a519e6d89a472..f2a8ce2951106 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2484,6 +2484,9 @@ def Vector_ShapeCastOp :
     VectorType getResultVectorType() {
       return ::llvm::cast<VectorType>(getResult().getType());
     }
+    // Return true if this Op is effectively a vector.broadcast (i.e. the input
+    // and output shapes satisfy the vector.broadcast constraints).
+    bool isBroadcastLike();
   }];
   let assemblyFormat = "$source attr-dict `:` type($source) `to` type($result)";
   let hasFolder = 1;
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index e0cb9923b1dd4..fb535cdae461f 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -429,6 +429,63 @@ static Attribute convertNumericAttr(Attribute attr, Type expectedType) {
   return attr;
 }
 
+/// Return whether `srcType` can be broadcast to `dstVectorType` under the
+/// semantics of the `vector.broadcast` op.
+BroadcastableToResult mlir::vector::isBroadcastableTo(
+    Type srcType, VectorType dstVectorType,
+    std::pair<VectorDim, VectorDim> *mismatchingDims) {
+  // Broadcast scalar to vector of the same element type.
+  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
+      srcType == getElementTypeOrSelf(dstVectorType))
+    return BroadcastableToResult::Success;
+  // From now on, only vectors broadcast.
+  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
+  if (!srcVectorType)
+    return BroadcastableToResult::SourceTypeNotAVector;
+
+  int64_t srcRank = srcVectorType.getRank();
+  int64_t dstRank = dstVectorType.getRank();
+  if (srcRank > dstRank)
+    return BroadcastableToResult::SourceRankHigher;
+  // Source has an exact match or singleton value for all trailing dimensions
+  // (all leading dimensions are simply duplicated).
+  int64_t lead = dstRank - srcRank;
+  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
+    // Have mismatching dims (in the sense of vector.broadcast semantics) been
+    // encountered?
+    bool foundMismatchingDims = false;
+
+    // Check fixed-width dims.
+    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
+    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
+    if (srcDim != 1 && srcDim != dstDim)
+      foundMismatchingDims = true;
+
+    // Check scalable flags.
+    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
+    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
+    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
+        // 1 -> [N] is fine, everything else should be rejected when mixing
+        // fixed-width and scalable dims
+        (srcDimScalableFlag != dstDimScalableFlag &&
+         (srcDim != 1 || srcDimScalableFlag)))
+      foundMismatchingDims = true;
+
+    if (foundMismatchingDims) {
+      if (mismatchingDims != nullptr) {
+        mismatchingDims->first.dim = srcDim;
+        mismatchingDims->first.isScalable = srcDimScalableFlag;
+
+        mismatchingDims->second.dim = dstDim;
+        mismatchingDims->second.isScalable = dstDimScalableFlag;
+      }
+      return BroadcastableToResult::DimensionMismatch;
+    }
+  }
+
+  return BroadcastableToResult::Success;
+}
+
 //===----------------------------------------------------------------------===//
 // CombiningKindAttr
 //===----------------------------------------------------------------------===//
@@ -3101,61 +3158,6 @@ Value BroadcastOp::createOrFoldBroadcastOp(
   return res;
 }
 
-BroadcastableToResult mlir::vector::isBroadcastableTo(
-    Type srcType, VectorType dstVectorType,
-    std::pair<VectorDim, VectorDim> *mismatchingDims) {
-  // Broadcast scalar to vector of the same element type.
-  if (isa<VectorElementTypeInterface>(srcType) && dstVectorType &&
-      srcType == getElementTypeOrSelf(dstVectorType))
-    return BroadcastableToResult::Success;
-  // From now on, only vectors broadcast.
-  VectorType srcVectorType = llvm::dyn_cast<VectorType>(srcType);
-  if (!srcVectorType)
-    return BroadcastableToResult::SourceTypeNotAVector;
-
-  int64_t srcRank = srcVectorType.getRank();
-  int64_t dstRank = dstVectorType.getRank();
-  if (srcRank > dstRank)
-    return BroadcastableToResult::SourceRankHigher;
-  // Source has an exact match or singleton value for all trailing dimensions
-  // (all leading dimensions are simply duplicated).
-  int64_t lead = dstRank - srcRank;
-  for (int64_t dimIdx = 0; dimIdx < srcRank; ++dimIdx) {
-    // Have mismatching dims (in the sense of vector.broadcast semantics) been
-    // encountered?
-    bool foundMismatchingDims = false;
-
-    // Check fixed-width dims.
-    int64_t srcDim = srcVectorType.getDimSize(dimIdx);
-    int64_t dstDim = dstVectorType.getDimSize(lead + dimIdx);
-    if (srcDim != 1 && srcDim != dstDim)
-      foundMismatchingDims = true;
-
-    // Check scalable flags.
-    bool srcDimScalableFlag = srcVectorType.getScalableDims()[dimIdx];
-    bool dstDimScalableFlag = dstVectorType.getScalableDims()[lead + dimIdx];
-    if ((srcDim == 1 && srcDimScalableFlag && dstDim != 1) ||
-        // 1 -> [N] is fine, everything else should be rejected when mixing
-        // fixed-width and scalable dims
-        (srcDimScalableFlag != dstDimScalableFlag &&
-         (srcDim != 1 || srcDimScalableFlag)))
-      foundMismatchingDims = true;
-
-    if (foundMismatchingDims) {
-      if (mismatchingDims != nullptr) {
-        mismatchingDims->first.dim = srcDim;
-        mismatchingDims->first.isScalable = srcDimScalableFlag;
-
-        mismatchingDims->second.dim = dstDim;
-        mismatchingDims->second.isScalable = dstDimScalableFlag;
-      }
-      return BroadcastableToResult::DimensionMismatch;
-    }
-  }
-
-  return BroadcastableToResult::Success;
-}
-
 LogicalResult BroadcastOp::verify() {
   std::pair<VectorDim, VectorDim> mismatchingDims;
   BroadcastableToResult res = isBroadcastableTo(
@@ -6757,6 +6759,15 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+bool ShapeCastOp::isBroadcastLike() {
+  auto srcType = getSourceVectorType();
+  auto resType = getResultVectorType();
+
+  std::pair<VectorDim, VectorDim> mismatchingDims;
+  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
+         BroadcastableToResult::Success;
+}
+
 /// Return true if `transpose` does not permute a pair of non-unit dims.
 /// By `order preserving` we mean that the flattened versions of the input and
 /// output vectors are (numerically) identical. In other words `transpose` is
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 752610efc6992..22e7742128ff2 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -282,21 +282,32 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   bool changed = false;
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
+
+    // Make sure that the source
+    auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
-    if (!broadcast)
+    if (!broadcast && !sc)
       continue;
+
+    if (sc && !sc.isBroadcastLike())
+      return rewriter.notifyMatchFailure(
+          contractOp, "Operand defined via vector.shape_cast that has "
+                      "non-broadcast semantics");
+
+    VectorType srcType = (sc) ? sc.getSourceVectorType()
+                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    VectorType resType =
+        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+
     // contractionOp can only take vector as operands.
-    auto srcType = dyn_cast<VectorType>(broadcast.getSourceType());
-    if (!srcType ||
-        srcType.getRank() == broadcast.getResultVectorType().getRank())
+    // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());
+    if (!srcType || srcType.getRank() >= resType.getRank())
       continue;
-    int64_t rankDiff =
-        broadcast.getResultVectorType().getRank() - srcType.getRank();
+    int64_t rankDiff = resType.getRank() - srcType.getRank();
     bool innerDimBroadcast = false;
     SmallVector<AffineExpr> originalDims;
     for (const auto &dim : llvm::enumerate(srcType.getShape())) {
-      if (dim.value() !=
-          broadcast.getResultVectorType().getDimSize(rankDiff + dim.index())) {
+      if (dim.value() != resType.getDimSize(rankDiff + dim.index())) {
         innerDimBroadcast = true;
         break;
       }
@@ -311,7 +322,7 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     // of non-unit size.
     bool nonUnitDimReductionBroadcast = false;
     for (int64_t i = 0; i < rankDiff; ++i) {
-      if (broadcast.getResultVectorType().getDimSize(i) != 1 &&
+      if (resType.getDimSize(i) != 1 &&
           isReductionIterator(contractOp.getIteratorTypes()
                                   .getValue()[map.getDimPosition(i)])) {
         nonUnitDimReductionBroadcast = true;
@@ -321,11 +332,10 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
     if (nonUnitDimReductionBroadcast)
       continue;
 
-    AffineMap broadcastMap =
-        AffineMap::get(broadcast.getResultVectorType().getRank(), 0,
-                       originalDims, contractOp.getContext());
+    AffineMap broadcastMap = AffineMap::get(resType.getRank(), 0, originalDims,
+                                            contractOp.getContext());
     map = broadcastMap.compose(map);
-    *operand = broadcast.getSource();
+    *operand = broadcast ? broadcast.getSource() : sc.getSource();
     changed = true;
   }
 
diff --git a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
index 3b51e6b1e1b6f..8e53a815f70f8 100644
--- a/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
+++ b/mlir/test/Dialect/Vector/vector-reduce-to-contract.mlir
@@ -202,7 +202,37 @@ func.func @contract_broadcast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1
 
 // -----
 
-// Same as above, but with a mask.
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but
+// `vector.broadcast` is replaced with `vector.shape_cast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>)
+//  CHECK: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction`), but with a mask.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -235,7 +265,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>,
 
 // -----
 
-// Same as above, but with a scalable dim.
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with
+// `vector.shape_cast` instead of `vector.broadcast`.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<8x4xi32>, %[[ARG2:.+]]: vector<8x8xi32>, %[[MASK:.+]]: vector<1x8x8x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x8x4xi1> to vector<8x8x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<8x4xi32> into vector<8x8xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked(%arg0 : vector<8x4xi32>, %arg1 : vector<8x4xi32>, %arg2 : vector<8x8xi32>, %mask: vector<1x8x8x4xi1>) -> vector<8x8xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<8x4xi32> to vector<1x8x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x8x4xi32> into vector<8x8xi32>
+    } : vector<1x8x8x4xi1> -> vector<8x8xi32>
+    return %result : vector<8x8xi32>
+}
+
+// -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked`), but with a scalable dim.
 
 #map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
 #map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
@@ -267,6 +331,41 @@ func.func @contract_broadcast_unit_dim_reduction_masked_scalable(%arg0 : vector<
 }
 
 // -----
+
+// Same as above (`@contract_broadcast_unit_dim_reduction_masked_scalable`), but with
+// vector.shape_cast instead of vector.broadcast.
+
+#map0 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d2, d3)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+// CHECK-DAG: #[[$MAP0:.*]] = affine_map<(d0, d1, d2) -> (d0, d2)>
+// CHECK-DAG: #[[$MAP1:.*]] = affine_map<(d0, d1, d2) -> (d1, d2)>
+// CHECK-DAG: #[[$MAP2:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
+
+// CHECK-LABEL: contract_shape_cast_unit_dim_reduction_masked_scalable
+//  CHECK-SAME: (%[[ARG0:.+]]: vector<8x4xi32>, %[[ARG1:.+]]: vector<[8]x4xi32>, %[[ARG2:.+]]: vector<8x[8]xi32>, %[[MASK:.+]]: vector<1x8x[8]x4xi1>)
+//  CHECK:      %[[MASK_SC:.*]] = vector.shape_cast %[[MASK]] : vector<1x8x[8]x4xi1> to vector<8x[8]x4xi1>
+//  CHECK:      %[[R:.*]] = vector.mask %[[MASK_SC]] {
+//  CHECK-SAME: vector.contract
+//  CHECK-SAME: indexing_maps = [#[[$MAP0]], #[[$MAP1]], #[[$MAP2]]]
+//  CHECK-SAME: iterator_types = ["parallel", "parallel", "reduction"]
+//  CHECK-SAME: %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<8x4xi32>, vector<[8]x4xi32> into vector<8x[8]xi32>
+func.func @contract_shape_cast_unit_dim_reduction_masked_scalable(%arg0 : vector<8x4xi32>, %arg1 : vector<[8]x4xi32>, %arg2 : vector<8x[8]xi32>, %mask: vector<1x8x[8]x4xi1>) -> vector<8x[8]xi32> {
+    %0 = vector.shape_cast %arg0 : vector<8x4xi32> to vector<1x8x4xi32>
+    %1 = vector.shape_cast %arg1 : vector<[8]x4xi32> to vector<1x[8]x4xi32>
+    %result = vector.mask %mask {
+      vector.contract {
+        indexing_maps = [#map0, #map1, #map2],
+        iterator_types = ["reduction", "parallel", "parallel", "reduction"],
+        kind = #vector.kind<add>
+      } %0, %1, %arg2 : vector<1x8x4xi32>, vector<1x[8]x4xi32> into vector<8x[8]xi32>
+    } : vector<1x8x[8]x4xi1> -> vector<8x[8]xi32>
+    return %result : vector<8x[8]xi32>
+}
+
+// -----
+
 // Test that CombineContractBroadcast will not combine a broadcast that creates
 // a non-unit dim that is consumed by a reduction iterator.
 // Moreover, the affine_map's are permuting the position of that reduction

>From 5a9232f71f80268e93cfd7790a2229b839c43fae Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Tue, 14 Jul 2026 09:09:30 +0000
Subject: [PATCH 2/3] Address PR comments

---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 28 +++++++++++++++++--
 .../Vector/Transforms/VectorTransforms.cpp    | 10 ++++---
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index fb535cdae461f..dd1a3895ad669 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6759,13 +6759,37 @@ LogicalResult ShapeCastOp::verify() {
   return success();
 }
 
+/// Check whether this ShapeCastOp is effectively a BroadcastOp.
+///
+/// The only case in which this method can return `true` is when the underlying
+/// op merely adds leading unit dimensions, e.g.:
+///   %res = vector.shape_cast %src : vector<8x4xi32> to vector<1x8x4xi32>
+///
 bool ShapeCastOp::isBroadcastLike() {
   auto srcType = getSourceVectorType();
   auto resType = getResultVectorType();
 
+  // Is srcType broadcastable to resType?
   std::pair<VectorDim, VectorDim> mismatchingDims;
-  return isBroadcastableTo(srcType, resType, &mismatchingDims) ==
-         BroadcastableToResult::Success;
+  if (isBroadcastableTo(srcType, resType, &mismatchingDims) !=
+      BroadcastableToResult::Success)
+    return false;
+
+  // Do ranks mismatch?
+  //
+  // The only case where ranks match and this ShapeCastOp is also a broadcast,
+  // is when it's effectively a NOp, but that's an uninteresting edge case.
+  size_t rankDiff = resType.getRank() - srcType.getRank();
+  if (rankDiff == 0)
+    return false;
+
+  // Are all newly added leading dims unit?
+  if (!llvm::all_of(resType.getShape().take_front(rankDiff),
+                    [](int64_t dim) { return dim == 1; }))
+    return false;
+
+  // Do all trailing dims match?
+  return resType.getShape().take_back(srcType.getRank()) == srcType.getShape();
 }
 
 /// Return true if `transpose` does not permute a pair of non-unit dims.
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 22e7742128ff2..ad35465179e94 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -283,7 +283,8 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
   for (Value *operand : {&lhs, &rhs}) {
     AffineMap &map = maps[index++];
 
-    // Make sure that the source
+    // Accept operands defined by vector.broadcast and broadcast-like
+    // vector.shape_cast.
     auto sc = operand->getDefiningOp<vector::ShapeCastOp>();
     auto broadcast = operand->getDefiningOp<vector::BroadcastOp>();
     if (!broadcast && !sc)
@@ -294,10 +295,11 @@ FailureOr<Value> combineContractAndBroadcast(vector::ContractionOp contractOp,
           contractOp, "Operand defined via vector.shape_cast that has "
                       "non-broadcast semantics");
 
-    VectorType srcType = (sc) ? sc.getSourceVectorType()
-                              : dyn_cast<VectorType>(broadcast.getSourceType());
+    // Get the source and the result types.
+    VectorType srcType = sc ? sc.getSourceVectorType()
+                            : dyn_cast<VectorType>(broadcast.getSourceType());
     VectorType resType =
-        (sc) ? sc.getResultVectorType() : broadcast.getResultVectorType();
+        sc ? sc.getResultVectorType() : broadcast.getResultVectorType();
 
     // contractionOp can only take vector as operands.
     // auto srcType = dyn_cast<VectorType>(broadcast.getSourceVectorType());

>From 03d048350a2e17078b8986aa3096b3ba2fc3eb15 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 17 Jul 2026 15:32:37 +0000
Subject: [PATCH 3/3] Update comment

---
 mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index ad35465179e94..17e6ad640cada 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -244,7 +244,7 @@ struct CombineContractResultTranspose final
   }
 };
 
-/// Merge BroadcastOp into ContractionOp user.
+/// Merge BroadcastOp (and broadcast-like ShapeCastOp) into ContractionOp user.
 /// Ex:
 /// ```
 ///   %0 = vector.broadcast %arg0 : vector<32x16xf32> to vector<8x32x16xf32>


From llvmlistbot at llvm.org  Fri Jul 17 08:34:03 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 08:34:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5a4b6b.32420449.289d.be41@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Fri Jul 17 08:38:59 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 08:38:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Extend
 `combineContractAndBroadcast` to accept `vector.shape_cast` (PR #208752)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208752@github.com>
Message-ID: <6a5a4c93.dcf3bad1.3ca4c9.a7a7@mx.google.com>


https://github.com/banach-space auto_merge_enabled https://github.com/llvm/llvm-project/pull/208752

From llvmlistbot at llvm.org  Fri Jul 17 09:07:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 09:07:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5a535b.1b0626d0.f869d.5ae7@mx.google.com>


================
@@ -0,0 +1,240 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs);
+
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    // If nothing was matched, no reflection maps were added, removing the need
+    // to add include headers for map and string
+    if (!listener.patternApplied)
+      return;
+
+    // Check if the map and/or string headers are already present
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+/// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
+/// and a lookup method.
+///
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes
+/// (e.g., `emitc.field_ref`). Fields that do not have a matching attribute
+/// are omitted from the reflection map.
+///
+/// Before:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.func @bar() { return }
----------------
banach-space wrote:

Here and in `After`, the formatting is off:
```suggestion
///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
///      ["another_feature"]} 
///   emitc.func @bar() { return }
```

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Fri Jul 17 09:09:46 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 09:09:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5a53ca.481afadf.9e047.247e@mx.google.com>


================
@@ -0,0 +1,240 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs);
+
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    // If nothing was matched, no reflection maps were added, removing the need
+    // to add include headers for map and string
+    if (!listener.patternApplied)
+      return;
+
+    // Check if the map and/or string headers are already present
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+/// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
+/// and a lookup method.
+///
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes
+/// (e.g., `emitc.field_ref`). Fields that do not have a matching attribute
+/// are omitted from the reflection map.
+///
+/// Before:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.func @bar() { return }
+/// }
+/// ```
+///
+/// After:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.field @reflectionMap : !emitc.opaque<"const
+///   std::map<std::string, char*>"> =
+///     #emitc.opaque<"{ { \"another_feature\",
+///     reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) ->
+///   !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const
+///     std::map<std::string, char*>"> %ptr = member_call_opaque %map
+///     "at"(%name) : ... return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   }
+///   emitc.func @bar() { return }
+/// }
+/// ```
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  MLGOAddReflectionMapClass(MLIRContext *context,
+                            llvm::ArrayRef<std::string> includedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context),
+        includedFieldAttrs(includedFieldAttrs) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect the names of all FieldOps that have one of the attributes in
+    // includedFieldAttrs to use the first element of the array attribute
+    // as the reflection map key
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      for (const auto &attr : includedFieldAttrs) {
+        auto arrayAttr = dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(attr));
+
+        if (!arrayAttr)
+          continue;
+
+        if (!arrayAttr.empty() && isa<StringAttr>(arrayAttr[0])) {
+          fieldNames.emplace_back(cast<StringAttr>(arrayAttr[0]).getValue(),
+                                  fieldOp.getName());
+          return;
+        }
+      }
+    });
+
+    if (fieldNames.empty())
+      return failure();
+
+    // Create reflection map contents
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+
+      first = false;
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
+    }
+    reflectionMapContents += " }";
+
+    // Set insertion point before the first function or after all fields
+    // if there are no functions within the class
+    auto funcs = classOp.getBlock().getOps<FuncOp>();
+    auto it = funcs.begin();
+    if (it != funcs.end())
+      rewriter.setInsertionPoint(*it);
+    else
+      rewriter.setInsertionPointToEnd(&classOp.getBlock());
----------------
banach-space wrote:

What's the benefit of this conditionality? Why not always use `rewriter.setInsertionPointToEnd(&classOp.getBlock());`?

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Fri Jul 17 09:11:21 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 09:11:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5a5429.92d2873c.aa1ff.cdeb@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM % minor comments, thank you!

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Fri Jul 17 09:12:26 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 17 Jul 2026 09:12:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5a546a.4a342cbc.3a2f36.10dd@mx.google.com>


================
@@ -60,4 +60,49 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `included-field-attrs` option) that is an array containing a single string
+    attribute. If a field possesses both an attribute specified by
+    `included-field-attrs` and an attribute specified in `excluded-field-attrs`,
+    the `included-field-attrs` attribute takes precedence and the field will be
+    included in the reflection map.
----------------
banach-space wrote:

> We also wouldn't need https://github.com/llvm/llvm-project/pull/207069 anymore either.

Then it's a win-win :)

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Fri Jul 17 10:01:07 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:01:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a5fd3.d76d81af.23b347.df65@mx.google.com>


https://github.com/JerryShih updated https://github.com/llvm/llvm-project/pull/209011

>From 93774b2a280c0f9078a3abf6bfb9d97528e14fb2 Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:51:20 -0700
Subject: [PATCH 1/4] Rename the negative tests in `simplify-pack-unpack.mlir`.

Add prefix `negative_` for the negative tests.
---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 36 +++++++++----------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index f4f3cb233d45e..6f6da9c41a974 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -16,11 +16,11 @@ func.func @single_dim_packing(%arg0: tensor<256xf32>) -> tensor<8x32xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @single_dim_packing_with_padding(
+// CHECK-LABEL: func.func @negative_single_dim_packing_with_padding(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<255xf32>)
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
+func.func @negative_single_dim_packing_with_padding(%arg0: tensor<255xf32>) -> tensor<8x32xf32> {
   %empty = tensor.empty() : tensor<8x32xf32>
   %cst = arith.constant 0.000000e+00 : f32
   %0 = linalg.pack %arg0 padding_value(%cst : f32) inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<255xf32> -> tensor<8x32xf32>
@@ -69,10 +69,10 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -80,10 +80,10 @@ func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<256x5xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -134,10 +134,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
   %empty = tensor.empty() : tensor<16x1x1x2xf32>
   %pack = linalg.pack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<32x1xf32> -> tensor<16x1x1x2xf32>
@@ -162,10 +162,10 @@ func.func @unpack_1d_to_collapse(%arg0: tensor<8x32xf32>) -> tensor<256xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_to_partial_slice
+// CHECK-LABEL: func.func @negative_unpack_to_partial_slice
 // CHECK-NOT:     tensor.collapse
 // CHECK:         linalg.unpack
-func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
+func.func @negative_unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
   %empty = tensor.empty() : tensor<255xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x32xf32> -> tensor<255xf32>
   return %0 : tensor<255xf32>
@@ -173,10 +173,10 @@ func.func @unpack_to_partial_slice(%arg0: tensor<8x32xf32>) -> tensor<255xf32> {
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_dynamic
+// CHECK-LABEL: func.func @negative_unpack_dynamic
 // CHECK:     tensor.collapse
 // CHECK-NOT:         linalg.unpack
-func.func @unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
+func.func @negative_unpack_dynamic(%arg0: tensor<?x32xf32>) -> tensor<?xf32> {
   %c32 = arith.constant 32 : index
   %c0 = arith.constant 0 : index
   %d0 = tensor.dim %arg0, %c0 : tensor<?x32xf32>
@@ -216,10 +216,10 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
@@ -227,10 +227,10 @@ func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
   %empty = tensor.empty() : tensor<256x5xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<256x5xf32>
   return %0 : tensor<256x5xf32>
@@ -277,10 +277,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
   %empty = tensor.empty() : tensor<32x1xf32>
   %unpack = linalg.unpack %arg0 inner_dims_pos = [1, 0] inner_tiles = [1, 2] into %empty
     : tensor<16x1x1x2xf32> -> tensor<32x1xf32>

>From 0603e88408c312005f24aa926b6e3da705888599 Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Sun, 12 Jul 2026 03:43:41 -0700
Subject: [PATCH 2/4] Add more test patterns in `simplify-pack-unpack.mlir`.

---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 103 ++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 6f6da9c41a974..875e153f5af56 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -428,3 +428,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x1x1x32x64xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x1x1x32x64xf32>) -> tensor<3x3x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x32x64xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x1x1x32x64xf32> -> tensor<3x3x32x64xf32>
+  return %0 : tensor<3x3x32x64xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
+  %empty = tensor.empty() : tensor<4x4xf32>
+  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
+  return %0 : tensor<4x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x32xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<1x1x32x32xf32>) -> tensor<32x32xf32> {
+  %empty = tensor.empty() : tensor<32x32xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
+  return %0 : tensor<32x32xf32>
+}
+
+// -----
+
+// Dynamic shape is not supported.
+// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
+  return %unpack : tensor<1x1000xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
+  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
+  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
+  return %result : tensor<1x1000xf32>
+}

>From 3c0c3b28c75d8f6b324b7d652b571890a999e1cf Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Fri, 17 Jul 2026 07:50:33 -0700
Subject: [PATCH 3/4] Address review comments.

---
 .../Dialect/Linalg/simplify-pack-unpack.mlir     | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index 875e153f5af56..c80de4734657a 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -80,10 +80,11 @@ func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> te
 
 // -----
 
-// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
+// This sort of packing requires transposition.
+// CHECK-LABEL: func.func @negative_packing_non_trailing_dim(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_non_trailing_dim(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<256x5xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
@@ -134,6 +135,7 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
+// Impossible to compute a re-association map.
 // CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
@@ -227,10 +229,10 @@ func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) ->
 
 // -----
 
-// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_packed_non_trailing_dim(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_packed_non_trailing_dim(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
   %empty = tensor.empty() : tensor<256x5xf32>
   %0 = linalg.unpack %arg0 inner_dims_pos = [0] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<256x5xf32>
   return %0 : tensor<256x5xf32>
@@ -432,11 +434,11 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
 // -----
 
 // The non-packed outer dims are not unit-size and permuted.
-// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
   %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
   return %0 : tensor<3x3x1x1x32x64xf32>
@@ -464,7 +466,7 @@ func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> te
 // CHECK:         linalg.pack
 func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
   %empty = tensor.empty() : tensor<1x1x32x32xf32>
-  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
   return %0 : tensor<1x1x32x32xf32>
 }
 

>From b6da105d40b09a0ee127e8b3da4d0ac91ef25a25 Mon Sep 17 00:00:00 2001
From: Jerry Shih <bignose1007 at gmail.com>
Date: Fri, 17 Jul 2026 09:51:26 -0700
Subject: [PATCH 4/4] Reorder test case.

---
 .../Dialect/Linalg/simplify-pack-unpack.mlir  | 160 +++++++++---------
 1 file changed, 80 insertions(+), 80 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
index c80de4734657a..39d1398a131c3 100644
--- a/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
+++ b/mlir/test/Dialect/Linalg/simplify-pack-unpack.mlir
@@ -92,6 +92,18 @@ func.func @negative_packing_non_trailing_dim(%arg0: tensor<256x5xf32>) -> tensor
 
 // -----
 
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
 //===========================================================================//
 // Packing: Multi-dim unpacked source
 //===========================================================================//
@@ -148,6 +160,20 @@ func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16
 
 // -----
 
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
 //===========================================================================//
 // Un-Packing: 1D unpacked dest
 //===========================================================================//
@@ -291,6 +317,46 @@ func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> ten
 
 // -----
 
+// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
+  %empty = tensor.empty() : tensor<4x4xf32>
+  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
+  return %0 : tensor<4x4xf32>
+}
+
+// -----
+
+// Dynamic shape is not supported.
+// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
+  return %unpack : tensor<1x1000xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
+// CHECK-NOT:     tensor.collapse_shape
+// CHECK:         linalg.unpack
+func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
+  %empty = tensor.empty() : tensor<1x1000xf32>
+  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
+  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
+  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
+  return %result : tensor<1x1000xf32>
+}
+
+// -----
+
 //===========================================================================//
 // Packing: Pad-like pack
 //
@@ -362,6 +428,19 @@ func.func @pad_like_pack_with_transpose(%arg0: tensor<32x64x16xf32>) -> tensor<3
 
 // -----
 
+// The non-unit inner dims are permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
+  %empty = tensor.empty() : tensor<1x1x32x32xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
+  return %0 : tensor<1x1x32x32xf32>
+}
+
+// -----
+
 //===========================================================================//
 // Un-Packing: Un-Pad-like unpack
 //
@@ -433,45 +512,6 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
 
 // -----
 
-// The non-packed outer dims are not unit-size and permuted.
-// CHECK-LABEL: func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
-// CHECK-NOT:     tensor.expand_shape
-// CHECK:         linalg.pack
-func.func @negative_pack_with_non_unit_non_packed_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
-  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
-  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
-  return %0 : tensor<3x3x1x1x32x64xf32>
-
-}
-
-// -----
-
-// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
-// CHECK-NOT:     tensor.expand_shape
-// CHECK:         linalg.pack
-func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
-  %empty = tensor.empty() : tensor<2x2x2x2xf32>
-  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
-  return %0 : tensor<2x2x2x2xf32>
-}
-
-// -----
-
-// The non-unit inner dims are permuted.
-// CHECK-LABEL: func.func @negative_pack_with_non_unit_inner_tile_dims_perm(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<32x32xf32>)
-// CHECK-NOT:     tensor.expand_shape
-// CHECK:         linalg.pack
-func.func @negative_pack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<32x32xf32>) -> tensor<1x1x32x32xf32> {
-  %empty = tensor.empty() : tensor<1x1x32x32xf32>
-  %0 = linalg.pack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
-  return %0 : tensor<1x1x32x32xf32>
-}
-
-// -----
-
 // CHECK-LABEL: func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x1x1x32x64xf32>)
 // CHECK-NOT:     tensor.collapse_shape
@@ -484,52 +524,12 @@ func.func @negative_unpack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x
 
 // -----
 
-// CHECK-LABEL: func.func @negative_unpack_with_non_unit_packed_dims(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<2x2x2x2xf32>)
-// CHECK-NOT:     tensor.collapse_shape
-// CHECK:         linalg.unpack
-func.func @negative_unpack_with_non_unit_packed_dims(%arg0: tensor<2x2x2x2xf32>) -> tensor<4x4xf32> {
-  %empty = tensor.empty() : tensor<4x4xf32>
-  %0 = linalg.unpack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<2x2x2x2xf32> -> tensor<4x4xf32>
-  return %0 : tensor<4x4xf32>
-}
-
-// -----
-
 // CHECK-LABEL: func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(
 // CHECK-SAME:    %[[ARG0:.+]]: tensor<1x1x32x32xf32>)
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
 func.func @negative_unpack_with_non_unit_inner_tile_dims_perm(%arg0: tensor<1x1x32x32xf32>) -> tensor<32x32xf32> {
   %empty = tensor.empty() : tensor<32x32xf32>
-  %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
+  %0 = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<1x1x32x32xf32> -> tensor<32x32xf32>
   return %0 : tensor<32x32xf32>
 }
-
-// -----
-
-// Dynamic shape is not supported.
-// CHECK-LABEL: func.func @negative_unpack_dynamic_input_shape(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
-// CHECK-NOT:     tensor.collapse_shape
-// CHECK:         linalg.unpack
-func.func @negative_unpack_dynamic_input_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
-  %dynamic_arg0 = tensor.cast %arg0 : tensor<1x63x1x16xf32> to tensor<1x?x1x16xf32>
-  %empty = tensor.empty() : tensor<1x1000xf32>
-  %unpack = linalg.unpack %dynamic_arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %empty : tensor<1x?x1x16xf32> -> tensor<1x1000xf32>
-  return %unpack : tensor<1x1000xf32>
-}
-
-// -----
-
-// CHECK-LABEL: func.func @negative_unpack_dynamic_output_shape(
-// CHECK-SAME:    %[[ARG0:.+]]: tensor<1x63x1x16xf32>)
-// CHECK-NOT:     tensor.collapse_shape
-// CHECK:         linalg.unpack
-func.func @negative_unpack_dynamic_output_shape(%arg0: tensor<1x63x1x16xf32>) -> tensor<1x1000xf32> {
-  %empty = tensor.empty() : tensor<1x1000xf32>
-  %dynamic_empty = tensor.cast %empty : tensor<1x1000xf32> to tensor<1x?xf32>
-  %unpack = linalg.unpack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [1, 16] into %dynamic_empty : tensor<1x63x1x16xf32> -> tensor<1x?xf32>
-  %result = tensor.cast %unpack : tensor<1x?xf32> to tensor<1x1000xf32>
-  return %result : tensor<1x1000xf32>
-}


From llvmlistbot at llvm.org  Fri Jul 17 10:03:36 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:03:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a6068.9edb5c9d.280be3.c022@mx.google.com>


JerryShih wrote:

@banach-space 
I reorder my new test cases into the pack/unpack categories
https://github.com/llvm/llvm-project/pull/209011/changes/b6da105d40b09a0ee127e8b3da4d0ac91ef25a25

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:08:27 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:08:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a618b.e624eb8d.4a8a7.336c@mx.google.com>


================
@@ -61,21 +61,21 @@ func.func @single_last_inner_dim_packing_with_identity_outer_dims_perm(%arg0: te
 
 // -----
 
-// CHECK-LABEL: func.func @packing_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_packing_with_outer_dims_perm(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_packing_with_outer_dims_perm(%arg0: tensor<5x256xf32>) -> tensor<8x5x32xf32> {
   %empty = tensor.empty() : tensor<8x5x32xf32>
   %0 = linalg.pack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<5x256xf32> -> tensor<8x5x32xf32>
   return %0 : tensor<8x5x32xf32>
 }
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_packing(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_packing(
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
+func.func @negative_single_first_inner_dim_packing(%arg0: tensor<256x5xf32>) -> tensor<8x5x32xf32> {
----------------
JerryShih wrote:

checked.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:08:51 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:08:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a61a3.510f576d.2f694c.5ba6@mx.google.com>


================
@@ -122,10 +122,10 @@ func.func @pack_32x1_to_16x1x2x1(%arg0 : tensor<32x1xf32>) -> tensor<1x16x2x1xf3
 
 // -----
 
-// CHECK-LABEL: func.func @pack_32x1_to_16x1x1x2
+// CHECK-LABEL: func.func @negative_pack_32x1_to_16x1x1x2
 // CHECK-NOT:     tensor.expand_shape
 // CHECK:         linalg.pack
-func.func @pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
+func.func @negative_pack_32x1_to_16x1x1x2(%arg0 : tensor<32x1xf32>) -> tensor<16x1x1x2xf32> {
----------------
JerryShih wrote:

checked.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:09:20 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:09:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a61c0.b1416da1.3242c8.4ed3@mx.google.com>


================
@@ -196,21 +196,21 @@ func.func @single_last_inner_dim_unpacking_with_identity_outer_dims_perm(%arg0:
 
 // -----
 
-// CHECK-LABEL: func.func @unpacking_with_outer_dims_perm(
+// CHECK-LABEL: func.func @negative_unpacking_with_outer_dims_perm(
 // CHECK-NOT:     tensor.collpase_shape
 // CHECK:         linalg.unpack
-func.func @unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
+func.func @negative_unpacking_with_outer_dims_perm(%arg0: tensor<8x5x32xf32>) -> tensor<5x256xf32> {
   %empty = tensor.empty() : tensor<5x256xf32>
   %0 = linalg.unpack %arg0 outer_dims_perm = [1, 0] inner_dims_pos = [1] inner_tiles = [32] into %empty : tensor<8x5x32xf32> -> tensor<5x256xf32>
   return %0 : tensor<5x256xf32>
 }
 
 // -----
 
-// CHECK-LABEL: func.func @single_first_inner_dim_unpacking(
+// CHECK-LABEL: func.func @negative_single_first_inner_dim_unpacking(
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
+func.func @negative_single_first_inner_dim_unpacking(%arg0: tensor<8x5x32xf32>) -> tensor<256x5xf32> {
----------------
JerryShih wrote:

checked.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:09:44 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:09:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a61d8.d681cab9.169270.a657@mx.google.com>


================
@@ -257,10 +257,10 @@ func.func @unpack_16x1x2x1_to_32x1(%arg0 : tensor<1x16x2x1xf32>) -> tensor<32x1x
 
 // -----
 
-// CHECK-LABEL: func.func @unpack_16x1x1x2_to_32x1
+// CHECK-LABEL: func.func @negative_unpack_16x1x1x2_to_32x1
 // CHECK-NOT:     tensor.collapse_shape
 // CHECK:         linalg.unpack
-func.func @unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
+func.func @negative_unpack_16x1x1x2_to_32x1(%arg0 : tensor<16x1x1x2xf32>) -> tensor<32x1xf32> {
----------------
JerryShih wrote:

checked.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:10:47 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:10:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a6217.492e23e4.d19bc.21b5@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
----------------
JerryShih wrote:

The new name is `negative_pack_with_non_unit_non_packed_outer_tile_dims_perm`.

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 10:12:19 2026
From: llvmlistbot at llvm.org (Jerry Shih)
Date: Fri, 17 Jul 2026 10:12:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg][test] Update
	`simplify-pack-unpack.mlir`. (PR #209011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209011@github.com>
Message-ID: <6a5a6273.743400b0.23a7b2.75b6@mx.google.com>


================
@@ -394,3 +394,106 @@ func.func @unpad_like_unpack_with_transpose(%arg0: tensor<32x1x16x64xf32>) -> te
   %0 = linalg.unpack %arg0 inner_dims_pos = [1] inner_tiles = [64] into %empty : tensor<32x1x16x64xf32> -> tensor<32x64x16xf32>
   return %0 : tensor<32x64x16xf32>
 }
+
+// -----
+
+// The non-packed outer dims are not unit-size and permuted.
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_outer_tile_dims_perm(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<3x3x32x64xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_outer_tile_dims_perm(%arg0: tensor<3x3x32x64xf32>) -> tensor<3x3x1x1x32x64xf32> {
+  %empty = tensor.empty() : tensor<3x3x1x1x32x64xf32>
+  %0 = linalg.pack %arg0 outer_dims_perm = [1, 0, 2, 3] inner_dims_pos = [2, 3] inner_tiles = [32, 64] into %empty : tensor<3x3x32x64xf32> -> tensor<3x3x1x1x32x64xf32>
+  return %0 : tensor<3x3x1x1x32x64xf32>
+
+}
+
+// -----
+
+// CHECK-LABEL: func.func @negative_pack_with_non_unit_packed_dims(
+// CHECK-SAME:    %[[ARG0:.+]]: tensor<4x4xf32>)
+// CHECK-NOT:     tensor.expand_shape
+// CHECK:         linalg.pack
+func.func @negative_pack_with_non_unit_packed_dims(%arg0: tensor<4x4xf32>) -> tensor<2x2x2x2xf32> {
+  %empty = tensor.empty() : tensor<2x2x2x2xf32>
+  %0 = linalg.pack %arg0 inner_dims_pos = [0, 1] inner_tiles = [2, 2] into %empty : tensor<4x4xf32> -> tensor<2x2x2x2xf32>
+  return %0 : tensor<2x2x2x2xf32>
+}
+
+// -----
+
+// The non-unit inner dims are permuted.
----------------
JerryShih wrote:

new test:
```
  %0 = linalg.pack %arg0 outer_dims_perm = [0, 1] inner_dims_pos = [1, 0] inner_tiles = [32, 32] into %empty : tensor<32x32xf32> -> tensor<1x1x32x32xf32>
```

https://github.com/llvm/llvm-project/pull/209011

From llvmlistbot at llvm.org  Fri Jul 17 12:52:22 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 17 Jul 2026 12:52:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Support human-readable float literals
 in the textual IR (PR #210422)
Message-ID: <llvm.org/llvm/llvm-project/pull/210422@github.com>


https://github.com/victor-eds created https://github.com/llvm/llvm-project/pull/210422

Add LLVM-style textual floating-point literals to the MLIR assembly format, both parsed and printed:

```
+inf / -inf
+qnan / -qnan
+nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
0x1.8p3                                  (C-style hexadecimal float)
```

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary identifiers. C-style hex floats are distinguished from the existing bit-pattern form by a fractional part and/or a binary exponent; the plain `0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's semantics: `Token::getFloatingPointValue` takes the target `fltSemantics` (and sign) and returns `std::optional<APFloat>` via `APFloat::convertFromString`, instead of first converting to double. This preserves NaN payloads and fixes a precision bug for types wider than double: e.g. `0.1 : f128` was previously stored as the double approximation of 0.1 widened to f128 rather than the correctly-rounded f128 value. `convertFromString` accepts inf/nan/snan spellings but not "qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is handled explicitly.

Also thread the float literal's location into the `FloatAttr` parse diagnostics so "not valid for specified type" points at the constant rather than the following token.

>From 13437b54cc55f8906677a00d92e79a503adc6b83 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 mlir/docs/LangRef.md                          |  11 +++
 mlir/lib/AsmParser/AttributeParser.cpp        |  22 +++--
 mlir/lib/AsmParser/Lexer.cpp                  |  80 ++++++++++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  10 +--
 mlir/lib/AsmParser/Token.cpp                  |  36 ++++++--
 mlir/lib/AsmParser/Token.h                    |   9 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  37 +++++++-
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++++-----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 ++--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +--
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +--
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 ++--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              |  64 ++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  43 ++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++++--------
 38 files changed, 424 insertions(+), 161 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..6ada4081b54ea 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,19 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics());
+  if (!result) {
+    emitError(loc, "invalid floating point literal");
+    return nullptr;
+  }
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..9adc6296dc48b 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,61 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal (infinity or NaN).
+/// `tokStart` spans from the (already consumed) sign character to the end of the
+/// buffer. On success returns a `floatliteral` (or `error`) token and advances
+/// `curPtr`; otherwise returns std::nullopt with `curPtr` unchanged so the
+/// caller can fall back to a plain `plus`/`minus` token. The sign is mandatory
+/// and folded into the token spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef tokStart) {
+  constexpr auto isIdentifierChar = [](char c) {
+    return isalpha(static_cast<unsigned char>(c)) ||
+           isdigit(static_cast<unsigned char>(c)) || c == '_' || c == '$' ||
+           c == '.';
+  };
+
+  // Scan the alphabetic keyword following the sign character.
+  StringRef rest = tokStart.drop_front();
+  size_t wordLen = 0;
+  while (wordLen < rest.size() &&
+         isalpha(static_cast<unsigned char>(rest[wordLen])))
+    ++wordLen;
+  StringRef word = rest.take_front(wordLen);
+  StringRef afterWord = rest.drop_front(wordLen);
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (word == "inf" || word == "qnan") {
+    if (!afterWord.empty() && isIdentifierChar(afterWord.front()))
+      return std::nullopt;
+    curPtr = afterWord.data();
+    return formToken(Token::floatliteral, tokStart.data());
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((word == "nan" || word == "snan") && afterWord.starts_with("(")) {
+    size_t close = afterWord.find(')');
+    if (close == StringRef::npos) {
+      curPtr = tokStart.end();
+      return emitError(tokStart.data(), "expected ')' in NaN literal");
+    }
+    curPtr = afterWord.data() + close + 1; // Consume through ')'.
+    return formToken(Token::floatliteral, tokStart.data());
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +384,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..c61606190e8ba 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef tokStart);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d5439bd192bc6 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,13 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
+    result = tok.getFloatingPointValue(isNegative, semantics);
+    if (!result)
+      return emitError(tok.getLoc()) << "invalid floating point literal";
     return success();
   }
 
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..53091a8b8d085 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -13,6 +13,7 @@
 #include "Token.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +52,37 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. Returns std::nullopt if the
+/// literal cannot be parsed for the given semantics.
+std::optional<APFloat>
+Token::getFloatingPointValue(bool isNegative,
+                             const llvm::fltSemantics &semantics) const {
+  StringRef str = spelling;
+  bool isNeg = isNegative;
+  if (str.consume_front("-"))
+    isNeg = !isNeg;
+  else
+    str.consume_front("+");
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics, matching LLVM's textual
+  // float literals: this covers decimal, C-style hexadecimal (`0x1.fp13`), inf,
+  // and `nan(0x..)`/`snan(0x..)` forms, preserving NaN payloads and avoiding a
+  // detour through `double`. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
     return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..f27a9a6e0b85d 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,6 +10,7 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
@@ -84,9 +85,11 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. Returns std::nullopt if
+  /// the literal cannot be parsed for the given semantics.
+  std::optional<APFloat> getFloatingPointValue(
+      bool isNegative, const llvm::fltSemantics &semantics) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..0a4e54f54a283 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,44 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
+  // Print special values in a human-readable, typeless form. The sign is always
+  // included, mirroring LLVM's textual float literals. Because these forms carry
+  // no type, signal the caller (via printedHex) to keep the trailing type.
+  if (apValue.isInfinity()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+  if (apValue.isNaN()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value.
+    if (payload.isSignMask()) {
+      os << "qnan";
+    } else {
+      if (apValue.isSignaling())
+        os << 's';
+      os << "nan(";
+      // Clear the signaling/quiet bit and trim leading zeros for output.
+      payload.clearBit(payload.getBitWidth() - 1);
+      SmallVector<char, 16> str;
+      payload.trunc(std::max(payload.getActiveBits(), 1u))
+          .toString(str, /*Radix=*/16, /*Signed=*/false,
+                    /*formatAsCLiteral=*/true);
+      os << str << ')';
+    }
+    return;
+  }
+
   // We would like to output the FP constant value in exponential notation,
   // but we cannot do this if doing so will lose precision.  Check here to
   // make sure that we only output it in exponential format if we can parse
   // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..6980514de9b92
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,64 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %1 = arith.constant -0x1.8p3 : f64
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %2 = arith.constant 0x1p4 : f32
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..67dc752e1b598 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,35 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Fri Jul 17 13:05:31 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 17 Jul 2026 13:05:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Support human-readable float literals
 in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5a8b0b.d681cab9.169270.6671@mx.google.com>


https://github.com/victor-eds edited https://github.com/llvm/llvm-project/pull/210422

From llvmlistbot at llvm.org  Fri Jul 17 13:10:45 2026
From: llvmlistbot at llvm.org (Aart Bik)
Date: Fri, 17 Jul 2026 13:10:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5a8c45.507fc713.239434.8eb1@mx.google.com>


https://github.com/aartbik commented:

At first glance, this looks exactly what I had in mind, since this centralizes the rejection of unsupported types (rather than adding them ad-hoc into conversions and rewriting cases). However, I am traveling right now to a conference, so let me have another more detailed look when I am back (next week)

https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Fri Jul 17 13:49:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 13:49:55 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openacc] Implicitly map external
 constant globals (PR #210408)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210408@github.com>
Message-ID: <6a5a9573.6c8668f2.1f24.9c3c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: Valentin Clement (バレンタイン クレメン) (clementval)

<details>
<summary>Changes</summary>

Hoist initializer-less constant globals out of OpenACC compute regions so implicit data mapping handles them. This avoids unresolved device symbols for PARAMETER arrays defined in separately compiled modules, while retaining implicit declare for initialized constants. Add regression coverage and a global initializer query to the OpenACC interface.

---
Full diff: https://github.com/llvm/llvm-project/pull/210408.diff


6 Files Affected:

- (modified) flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h (+1) 
- (modified) flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp (+5) 
- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td (+4) 
- (modified) mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp (+5) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCImplicitDeclare.cpp (+20-12) 
- (modified) mlir/test/Dialect/OpenACC/acc-implicit-declare.mlir (+24) 


``````````diff
diff --git a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
index 31e1593e11325..f5307c6b4f046 100644
--- a/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
+++ b/flang/include/flang/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.h
@@ -68,6 +68,7 @@ struct GlobalVariableModel
     : public mlir::acc::GlobalVariableOpInterface::ExternalModel<
           GlobalVariableModel, fir::GlobalOp> {
   bool isConstant(mlir::Operation *op) const;
+  bool hasInitializer(mlir::Operation *op) const;
   mlir::Region *getInitRegion(mlir::Operation *op) const;
   bool isDeviceData(mlir::Operation *op) const;
 };
diff --git a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
index 8e4b55fe5f4a8..3fdfb0302fb74 100644
--- a/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
+++ b/flang/lib/Optimizer/OpenACC/Support/FIROpenACCOpsInterfaces.cpp
@@ -121,6 +121,11 @@ bool GlobalVariableModel::isConstant(mlir::Operation *op) const {
   return globalOp.getConstant().has_value();
 }
 
+bool GlobalVariableModel::hasInitializer(mlir::Operation *op) const {
+  auto globalOp = mlir::cast<fir::GlobalOp>(op);
+  return globalOp.getInitVal().has_value() || globalOp.hasInitializationBody();
+}
+
 mlir::Region *GlobalVariableModel::getInitRegion(mlir::Operation *op) const {
   auto globalOp = mlir::cast<fir::GlobalOp>(op);
   return globalOp.hasInitializationBody() ? &globalOp.getRegion() : nullptr;
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index fa44fe7a49158..eddd21a7e917b 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -73,6 +73,10 @@ def GlobalVariableOpInterface : OpInterface<"GlobalVariableOpInterface"> {
       "isConstant", (ins), [{
         return false;
       }]>,
+    InterfaceMethod<"Check if the global variable has an initializer", "bool",
+      "hasInitializer", (ins), [{
+        return false;
+      }]>,
     InterfaceMethod<"Get the initialization region (returns nullptr if none)",
       "::mlir::Region*", "getInitRegion", (ins)>,
     InterfaceMethod<"Check if the global variable is device data",
diff --git a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
index 0205afc1dfedb..cdf34234f06df 100644
--- a/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
+++ b/mlir/lib/Dialect/OpenACC/IR/OpenACC.cpp
@@ -394,6 +394,11 @@ struct MemrefGlobalVariableModel
     return globalOp.getConstant();
   }
 
+  bool hasInitializer(Operation *op) const {
+    auto globalOp = cast<memref::GlobalOp>(op);
+    return globalOp.getInitialValue().has_value();
+  }
+
   Region *getInitRegion(Operation *op) const {
     // GlobalOp uses attributes for initialization, not regions
     return nullptr;
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCImplicitDeclare.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCImplicitDeclare.cpp
index 3b8bf86fee11c..4d77353c20f56 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCImplicitDeclare.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCImplicitDeclare.cpp
@@ -45,10 +45,11 @@
 //
 // The pass performs two main optimizations:
 //
-// 1. Hoisting: For non-constant globals referenced in compute regions, the
-//    pass hoists the address-of operation out of the region when possible,
-//    allowing them to be implicitly mapped through normal data clause
-//    mechanisms rather than requiring declare marking.
+// 1. Hoisting: For non-constant globals and constant globals without a local
+//    initializer referenced in compute regions, the pass hoists the address-of
+//    operation out of the region when possible, allowing them to be implicitly
+//    mapped through normal data clause mechanisms rather than requiring
+//    declare marking.
 //
 // 2. Declaration: For globals that must be available on the device (constants,
 //    globals in routines, globals in recipe operations), the pass adds the
@@ -214,20 +215,27 @@ static bool isGlobalUseCandidateForHoisting(Operation *globalOp,
   if (accSupport.isValidSymbolUse(user, symbol))
     return false;
 
-  bool isConstant = false;
+  bool isInitializedConstant = false;
   bool isFunction = false;
 
   if (auto globalVarOp = dyn_cast<acc::GlobalVariableOpInterface>(globalOp))
-    isConstant = globalVarOp.isConstant();
+    isInitializedConstant =
+        globalVarOp.isConstant() && globalVarOp.hasInitializer();
 
   if (isa<FunctionOpInterface>(globalOp))
     isFunction = true;
 
-  // Constants should be kept in device code to ensure they are duplicated.
+  // Initialized constants should be kept in device code so their definitions
+  // can be duplicated in the device image. An initializer-less constant is an
+  // external declaration, so keeping its address-of in device code would
+  // create a device symbol that a separately compiled defining translation
+  // unit may not provide. Hoist it so normal implicit mapping passes its host
+  // definition to the compute region instead.
+  //
   // Function references should be kept in device code to ensure their device
   // addresses are computed. Everything else should be hoisted since we already
-  // proved they are not valid symbols in GPU region.
-  return !isConstant && !isFunction;
+  // proved it is not a valid symbol in the GPU region.
+  return !isInitializedConstant && !isFunction;
 }
 
 /// Checks whether it is valid to use acc.declare marking on the global.
@@ -259,9 +267,9 @@ static bool hasRelevantRecipeUse(RecipeOpT &recipeOp, ModuleOp &mod) {
   return use.getUser() != recipeOp.getOperation();
 }
 
-// Hoists addr_of operations for non-constant globals out of OpenACC regions.
-// This way - they are implicitly mapped instead of being considered for
-// implicit declare.
+// Hoists addr_of operations for globals that cannot be defined in this
+// translation unit out of OpenACC regions. This way they are implicitly mapped
+// instead of being considered for implicit declare.
 template <typename AccConstructT>
 static void hoistNonConstantDirectUses(AccConstructT accOp,
                                        acc::OpenACCSupport &accSupport) {
diff --git a/mlir/test/Dialect/OpenACC/acc-implicit-declare.mlir b/mlir/test/Dialect/OpenACC/acc-implicit-declare.mlir
index b1c9aa9016dc3..b8ab5c89cd7a5 100644
--- a/mlir/test/Dialect/OpenACC/acc-implicit-declare.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-implicit-declare.mlir
@@ -43,6 +43,30 @@ func.func @test_constant_in_serial() {
 
 // -----
 
+// Test that an external constant global is hoisted for implicit mapping. Its
+// definition may be in a separately compiled translation unit that does not
+// contain an OpenACC construct and therefore does not emit a device definition.
+
+memref.global constant @gexternalconstant : memref<4xf32>
+
+func.func @test_external_constant_in_serial() {
+  acc.serial {
+    %c0 = arith.constant 0 : index
+    %addr = memref.get_global @gexternalconstant : memref<4xf32>
+    %load = memref.load %addr[%c0] : memref<4xf32>
+    acc.yield
+  }
+  return
+}
+
+// CHECK: memref.global constant @gexternalconstant : memref<4xf32>
+// CHECK-NOT: acc.declare
+// CHECK-LABEL: func.func @test_external_constant_in_serial
+// CHECK: memref.get_global @gexternalconstant
+// CHECK-NEXT: acc.serial
+
+// -----
+
 // Test globals referenced in acc routine functions
 
 memref.global @gscalar_routine : memref<f32> = dense<0.0>

``````````

</details>


https://github.com/llvm/llvm-project/pull/210408

From llvmlistbot at llvm.org  Fri Jul 17 13:56:02 2026
From: llvmlistbot at llvm.org (Aiden Grossman)
Date: Fri, 17 Jul 2026 13:56:02 -0700 (PDT)
Subject: [Mlir-commits] [clang] [libcxx] [lld] [lldb] [llvm] [mlir] [polly]
 [lit] Delete the external shell (PR #209571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209571@github.com>
Message-ID: <6a5a96e2.5a0412d2.3daf32.5f5c@mx.google.com>


https://github.com/boomanaiden154 closed https://github.com/llvm/llvm-project/pull/209571

From llvmlistbot at llvm.org  Fri Jul 17 13:56:46 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Fri, 17 Jul 2026 13:56:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5a970e.6c8668f2.1f24.22dd@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 3248a8fc7c2b36246d7b73ff7738a3fc322c8762 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 01/23] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From d19c4ffe85f0222faea19d8e678c43547ed8c8b1 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 02/23] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From ca829ba5ed1ba4b34f5863d308d9a20b846a2e8e Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 03/23] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From d8683ccca9ac6eefd0a5e49614ba84838f28f935 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 04/23] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From 180b5755488334941ed6a75d8714a21fed5ea458 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 05/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From b002c15b4bd7e246f2851ab1f3925e6653443e2f Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 06/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From cde91391d1799651551b261ba7b8f54503ce4eaf Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 07/23] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From ae48f6fb2082091b953130160b333e8b3394555d Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 08/23] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///

>From 0363ce89b63e2b848d7e2aad8d0fa272ebebc09d Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:35:29 -0700
Subject: [PATCH 09/23] Added create_named_contractions. Removing 2nd
 reduction_to_contract causes test to fail

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir        | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index d6a6bedc75a10..32984a04f6a99 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -252,9 +252,8 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
-    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
-    transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+      : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
     transform.apply_patterns to %loop_k {
@@ -270,7 +269,6 @@ module @transforms attributes { transform.with_named_sequence } {
     %func_pre = transform.structured.match ops{["func.func"]} in %module
       : (!transform.any_op) -> !transform.any_op
     transform.apply_patterns to %func_pre {
-      transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.transfer_permutation_patterns
       transform.apply_patterns.canonicalization
     } : !transform.any_op

>From c167923a6286adafacfbb921eb2b40f0f4d015f3 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:47:50 -0700
Subject: [PATCH 10/23] Fix some comments

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir          | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 32984a04f6a99..06dae3ee89042 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -244,14 +244,14 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the M, N dim is scalable!)
+    // Tile parallel dims (note, the M, N dims are scalable!)
     %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
     %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
     transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
       : !transform.any_op
 

>From 9d06ccdd278126459aa7b4934960fde8a15faf2d Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 09:35:52 -0700
Subject: [PATCH 11/23] Make A_pack inner_tiles [%vscale_c8, 1]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir   | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 06dae3ee89042..7b8b3b4c74ff1 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -125,17 +125,21 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
   %A_pack = linalg.pack %A
     padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xf32>
+  return %A_pack : tensor<1x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -219,12 +223,12 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
 //===----------------------------------------------------------------------===//
 func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
   %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>

>From fd8300be13f723aa360db77ce9a648b86af542b6 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 10:12:37 -0700
Subject: [PATCH 12/23] Make C_pack and C_out_unpack's inner tiles [%vs_c8,
 %vs_c8]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 +++++++++++--------
 1 file changed, 19 insertions(+), 14 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 7b8b3b4c74ff1..f0348c1679723 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -175,24 +175,26 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32
 
-  // Compute the outer tile size.
+  // Compute the outer tile sizes.
+  %c7 = arith.constant 7 : index
   %c13 = arith.constant 13 : index
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
-  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
+  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
   %C_pack = linalg.pack %C
     padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xf32>
+  return %C_pack : tensor<?x?x?x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -201,7 +203,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
@@ -210,8 +212,8 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
 
   return %C_out_unpack: tensor<7x13xf32>
 }
@@ -225,13 +227,13 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   // Pack input matrices
   %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
 
   return %C_out_unpack : tensor<7x13xf32>
 }
@@ -297,8 +299,11 @@ module @transforms attributes { transform.with_named_sequence } {
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
-    //    and other lower-level Ops (see step 2)
-    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+    //    and other lower-level Ops (see step 2). Tile by [8*vscale, 8*vscale]
+    //    (scalable, equal to the inner tile sizes) so the tiled outer dims are
+    //    statically 1 and DecomposeOuterUnitDimsUnPackOp can fire.
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [[8], [8]]
+        inner_tile_alignments = [Equal, Equal]
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 2.1. Decompose tiled PackOp into lower-level Ops + simplify

>From 1edb7fae2e931cb2e558210bb913113b6fd86463 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:38:23 -0700
Subject: [PATCH 13/23] Remove dead %mmt4d_func

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f0348c1679723..db15eff92cad3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -247,7 +247,6 @@ module @transforms attributes { transform.with_named_sequence } {
     // HANDLE MMT4D
     //==========================================================================
     %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
-    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
     // Tile parallel dims (note, the M, N dims are scalable!)

>From 6f3cbd34f38a29857f1c1dcb1ded6cc85eb9db11 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:41:16 -0700
Subject: [PATCH 14/23] Fix comment for pack_lhs

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir         | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index db15eff92cad3..b599b95cd19fa 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -121,9 +121,10 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 }
 
 //===----------------------------------------------------------------------===//
-// @matmul_via_mmt4d
+// @pack_lhs
 //
-// Implements matrix-multiplication via linalg.mmt4d
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From ece92ed8eee70deb23dd9ab65d6d8845ffed5e8b Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 08:54:55 -0700
Subject: [PATCH 15/23] Add flag assume_dynamic_dims_match_vec_sizes when
 vectorizing

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b599b95cd19fa..0db52af86c17e 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -258,7 +258,7 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
-    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
       : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask

>From fa8f9442c3d6dce7fac487986ba4cbd46dbd262f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 08:58:12 -0700
Subject: [PATCH 16/23] Remove TODO about fusion

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir           | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 0db52af86c17e..a520816bce2f0 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,9 +40,6 @@
 /// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
 ///    the corresponding lowering and lowering configs.
 /// 
-/// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unpack/mmt4d ops into one function:
-///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751
 //===----------------------------------------------------------------------===//

>From a8c3dfd5b86220e6f139c12473f09a0fce8fbd7a Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:19:44 -0400
Subject: [PATCH 17/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index a520816bce2f0..a516148c80a7e 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -32,7 +32,7 @@
 ///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.

>From d94acd569cf6af29f00b283e1b90b0a5aac4f640 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:20:26 -0400
Subject: [PATCH 18/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index a516148c80a7e..7923ff09aa9d9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -171,7 +171,7 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // @pack_acc
 //
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
-// The inner tile size is "scalable": 8 * vscale
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
 func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32

>From be4415d02eb95cd53c7ac04d0c6ca93e04261121 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:20:44 -0400
Subject: [PATCH 19/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 7923ff09aa9d9..78bcd5cf63ce5 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -199,7 +199,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
 // @unpack_acc
 //
 // Implements unpacking for the C matrix (accumulator) in matrix
-// multiplication. The inner tile size is "scalable": 8 * vscale
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
 func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale

>From 4b73bcc8a350793fc0e5c2a6b06948b936919617 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:21:01 -0400
Subject: [PATCH 20/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 78bcd5cf63ce5..34abbdf31ab47 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -121,7 +121,7 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 // @pack_lhs
 //
 // Implements packing for the A matrix (LHS) in matrix multiplication. The
-// inner tile size is "scalable": 8 * vscale.
+// inner tile size for dim M is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From e4e9d39bf934c7fae88f4b7fb093bd17201325ae Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:21:19 -0400
Subject: [PATCH 21/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 34abbdf31ab47..27322db143b30 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -144,7 +144,7 @@ func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
 // @pack_rhs
 //
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
-// inner tile size is "scalable": 8 * vscale.
+// inner tile size for dim N is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From aec6f290ca8dbcf066d010604dda903227732ff9 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Fri, 17 Jul 2026 12:37:19 -0700
Subject: [PATCH 22/23] Add comment about create_named_contraction

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir           | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 27322db143b30..ed8c12992b3a2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -276,6 +276,9 @@ module @transforms attributes { transform.with_named_sequence } {
       transform.apply_patterns.canonicalization
     } : !transform.any_op
     transform.apply_patterns to %func_pre {
+      // TODO: `create_named_contraction` above lowers directly to a named contraction. If we want to remove
+      // `transform.apply_patterns.vector.reduction_to_contract` below, `CombineContractBroadcastMask`
+      // will have to be moved to a dedicated TD Op.
       transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.cast_away_vector_leading_one_dim
       transform.apply_patterns.vector.lower_contraction

>From c201468fb883e2ada1f22bc54e430561a70d9551 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Fri, 17 Jul 2026 13:55:58 -0700
Subject: [PATCH 23/23] Fix up dims of pack_lhs, pack_rhs, pack_acc, unpack_acc

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 24 +++++++++----------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index ed8c12992b3a2..73e286f960008 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -173,26 +173,24 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32
 
-  // Compute the outer tile sizes.
-  %c7 = arith.constant 7 : index
+  // Compute the outer tile size.
   %c13 = arith.constant 13 : index
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
-  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
-  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
   %C_pack = linalg.pack %C
     padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
 
-  return %C_pack : tensor<?x?x?x?xf32>
+  return %C_pack : tensor<1x?x?x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -201,7 +199,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
@@ -211,7 +209,7 @@ func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [%vs_c8, %vs_c8]
-    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
 
   return %C_out_unpack: tensor<7x13xf32>
 }
@@ -225,13 +223,13 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   // Pack input matrices
   %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
 
   return %C_out_unpack : tensor<7x13xf32>
 }


From llvmlistbot at llvm.org  Fri Jul 17 14:50:59 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Fri, 17 Jul 2026 14:50:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve section bounds on
	acc.reduction_init (PR #210443)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210443@github.com>
Message-ID: <6a5aa3c3.4987c784.2c60a9.8227@mx.google.com>


https://github.com/razvanlupusoru updated https://github.com/llvm/llvm-project/pull/210443

>From 01f966dab9321dbc120fe9e4f12c73d5e059b266 Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Fri, 17 Jul 2026 14:30:43 -0700
Subject: [PATCH 1/2] [mlir][acc] Preserve section bounds on acc.reduction_init

When `acc.reduction_init` is created from `acc.reduction`, its
bounds are lost because they only live as values folded into the
init region's body. Those bounds describe the legal indexing of
the section: the produced view of a (possibly base-adjusted,
compact) reduction temporary must be indexed with the same bounds
as the original array.

Carry the acc.bounds as optional operands on acc.reduction_init
so this indexing information is directly available. The operands
are optional, so scalar/whole-array reductions and existing
IR are unaffected.
---
 .../mlir/Dialect/OpenACC/OpenACCCGOps.td      | 21 +++++++++++++++++--
 .../Transforms/ACCRecipeMaterialization.cpp   |  4 +++-
 mlir/test/Dialect/OpenACC/ops-cg.mlir         | 14 +++++++++++++
 3 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index 53adb91fc2983..730ea1f73a3ac 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -30,7 +30,7 @@ include "mlir/IR/CommonAttrConstraints.td"
 
 def OpenACC_ReductionInitOp
     : OpenACC_Op<"reduction_init",
-                 [SameOperandsAndResultType, RecursiveMemoryEffects,
+                 [AllTypesMatch<["var", "result"]>, RecursiveMemoryEffects,
                   DeclareOpInterfaceMethods<RegionBranchOpInterface,
                                             ["getRegionInvocationBounds",
                                              "getSuccessorInputs"]>,
@@ -44,14 +44,31 @@ def OpenACC_ReductionInitOp
 
     The `var` operand is the original/shared reduction variable. The
     `reduction_operator` specifies the reduction kind (e.g. add, mul).
+
+    The optional `bounds` operands describe the element range of the reduction
+    variable (as `acc.bounds` ops) when it refers to an array section. They
+    carry the section's bound information.
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$var,
+                       Variadic<OpenACC_DataBoundsType>:$bounds,
                        OpenACC_ReductionOperatorAttr:$reductionOperator);
   let results = (outs OpenACC_AnyPointerOrMappableType:$result);
   let regions = (region AnyRegion:$region);
   let assemblyFormat = [{
-    $var $reductionOperator `:` type($result) $region attr-dict
+    $var (`bounds` `(` $bounds^ `)`)? $reductionOperator `:` type($result)
+    $region attr-dict
   }];
+  let builders = [
+    // Convenience builder for reductions without array-section bounds.
+    OpBuilder<(ins "::mlir::Value":$var,
+                   "::mlir::acc::ReductionOperatorAttr":$reductionOperator), [{
+      build($_builder, $_state, var, ::mlir::ValueRange{}, reductionOperator);
+    }]>,
+    OpBuilder<(ins "::mlir::Value":$var,
+                   "::mlir::acc::ReductionOperator":$reductionOperator), [{
+      build($_builder, $_state, var, ::mlir::ValueRange{}, reductionOperator);
+    }]>,
+  ];
   let hasVerifier = 1;
 }
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index e666bcd9a6242..856ecba953b7a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -344,8 +344,10 @@ LogicalResult ACCRecipeMaterialization::materialize(
     else
       llvm_unreachable("unexpected acc op with reduction recipe");
 
+    SmallVector<Value> reductionBounds(acc::getBounds(op));
     auto reductionOp = acc::ReductionInitOp::create(
-        b, op.getLoc(), origPtr, recipe.getReductionOperatorAttr());
+        b, op.getLoc(), origPtr, reductionBounds,
+        recipe.getReductionOperatorAttr());
     saveVarName(op.getAccVar(), reductionOp.getResult());
     cloneRegionIntoAccRegion(&initRegion, &reductionOp.getRegion(),
                              /*hasResult=*/true);
diff --git a/mlir/test/Dialect/OpenACC/ops-cg.mlir b/mlir/test/Dialect/OpenACC/ops-cg.mlir
index 765552e8577a7..cf1c3340870a5 100644
--- a/mlir/test/Dialect/OpenACC/ops-cg.mlir
+++ b/mlir/test/Dialect/OpenACC/ops-cg.mlir
@@ -160,6 +160,20 @@ func.func @compute_region_two_dims(%data: memref<8xi32>,
 
 // -----
 
+// CHECK-LABEL: func @reduction_init_with_bounds
+func.func @reduction_init_with_bounds(%arg0: memref<?xi32>, %lb: index, %ext: index) {
+  %c1 = arith.constant 1 : index
+  %bnd = acc.bounds lowerbound(%lb : index) extent(%ext : index) stride(%c1 : index)
+  %0 = acc.reduction_init %arg0 bounds(%bnd) <add> : memref<?xi32> {
+    acc.yield %arg0 : memref<?xi32>
+  }
+  return
+}
+// CHECK: %[[BND:.*]] = acc.bounds
+// CHECK: acc.reduction_init %{{.*}} bounds(%[[BND]]) <add> : memref<?xi32>
+
+// -----
+
 // CHECK-LABEL: func @compute_region_unknown_width
 func.func @compute_region_unknown_width(%data: memref<100xf32>) {
   %copyin = acc.copyin varPtr(%data : memref<100xf32>) -> memref<100xf32>

>From f7b59b656b807f23c47246cdf5c4e9631019758f Mon Sep 17 00:00:00 2001
From: Razvan Lupusoru <rlupusoru at nvidia.com>
Date: Fri, 17 Jul 2026 14:50:47 -0700
Subject: [PATCH 2/2] Fix formatting

---
 .../Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index 856ecba953b7a..70aacd251c329 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -345,9 +345,9 @@ LogicalResult ACCRecipeMaterialization::materialize(
       llvm_unreachable("unexpected acc op with reduction recipe");
 
     SmallVector<Value> reductionBounds(acc::getBounds(op));
-    auto reductionOp = acc::ReductionInitOp::create(
-        b, op.getLoc(), origPtr, reductionBounds,
-        recipe.getReductionOperatorAttr());
+    auto reductionOp =
+        acc::ReductionInitOp::create(b, op.getLoc(), origPtr, reductionBounds,
+                                     recipe.getReductionOperatorAttr());
     saveVarName(op.getAccVar(), reductionOp.getResult());
     cloneRegionIntoAccRegion(&initRegion, &reductionOp.getRegion(),
                              /*hasResult=*/true);


From llvmlistbot at llvm.org  Fri Jul 17 14:51:44 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Fri, 17 Jul 2026 14:51:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve section bounds on
	acc.reduction_init (PR #210443)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210443@github.com>
Message-ID: <6a5aa3f0.eb165e5c.4506c.4d67@mx.google.com>


https://github.com/razvanlupusoru auto_merge_enabled https://github.com/llvm/llvm-project/pull/210443

From llvmlistbot at llvm.org  Fri Jul 17 14:52:57 2026
From: llvmlistbot at llvm.org (Andrei Sabalenka)
Date: Fri, 17 Jul 2026 14:52:57 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [libcxx] [lldb] [llvm] [mlir] [NFC]
 Silence -Wbool-integral-comparison warnings across LLVM (PR #195246)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195246@github.com>
Message-ID: <6a5aa439.c4d3d80a.849c3.f305@mx.google.com>


mechakotik wrote:

Ping @arsenm @aaupov @maksfb @rafaelauler @ayermolo @yota9 @paschalis-mpeis @yozhu @JDevlieghere


https://github.com/llvm/llvm-project/pull/195246

From llvmlistbot at llvm.org  Fri Jul 17 14:54:28 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Fri, 17 Jul 2026 14:54:28 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openacc] Implicitly map external
 constant globals (PR #210408)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210408@github.com>
Message-ID: <6a5aa494.7270d2fd.a9bd6.24f3@mx.google.com>


================
@@ -43,6 +43,30 @@ func.func @test_constant_in_serial() {
 
 // -----
 
+// Test that an external constant global is hoisted for implicit mapping. Its
----------------
razvanlupusoru wrote:

Possible to also add a test for memref with initial value?

https://github.com/llvm/llvm-project/pull/210408

From llvmlistbot at llvm.org  Fri Jul 17 14:54:37 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Fri, 17 Jul 2026 14:54:37 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [openacc] Implicitly map external
 constant globals (PR #210408)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210408@github.com>
Message-ID: <6a5aa49d.92ea0e2a.4a9a5.9eaf@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Great work.

https://github.com/llvm/llvm-project/pull/210408

From llvmlistbot at llvm.org  Fri Jul 17 14:59:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 14:59:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 35bf123 - [mlir][acc] Preserve section bounds
 on acc.reduction_init (#210443)
Message-ID: <6a5aa5b9.c56fb102.2d07b8.b28e@mx.google.com>


Author: Razvan Lupusoru
Date: 2026-07-17T21:59:16Z
New Revision: 35bf12351740521f86fd406d01165e1c90b32ed8

URL: https://github.com/llvm/llvm-project/commit/35bf12351740521f86fd406d01165e1c90b32ed8
DIFF: https://github.com/llvm/llvm-project/commit/35bf12351740521f86fd406d01165e1c90b32ed8.diff

LOG: [mlir][acc] Preserve section bounds on acc.reduction_init (#210443)

When `acc.reduction_init` is created from `acc.reduction`, its bounds
are lost because they only live as values folded into the init region's
body. Those bounds describe the legal indexing of the section: the
produced view of a (possibly base-adjusted, compact) reduction temporary
must be indexed with the same bounds as the original array.

Carry the acc.bounds as optional operands on acc.reduction_init so this
indexing information is directly available. The operands are optional,
so scalar/whole-array reductions and existing IR are unaffected.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
    mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
    mlir/test/Dialect/OpenACC/ops-cg.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index 53adb91fc2983..730ea1f73a3ac 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -30,7 +30,7 @@ include "mlir/IR/CommonAttrConstraints.td"
 
 def OpenACC_ReductionInitOp
     : OpenACC_Op<"reduction_init",
-                 [SameOperandsAndResultType, RecursiveMemoryEffects,
+                 [AllTypesMatch<["var", "result"]>, RecursiveMemoryEffects,
                   DeclareOpInterfaceMethods<RegionBranchOpInterface,
                                             ["getRegionInvocationBounds",
                                              "getSuccessorInputs"]>,
@@ -44,14 +44,31 @@ def OpenACC_ReductionInitOp
 
     The `var` operand is the original/shared reduction variable. The
     `reduction_operator` specifies the reduction kind (e.g. add, mul).
+
+    The optional `bounds` operands describe the element range of the reduction
+    variable (as `acc.bounds` ops) when it refers to an array section. They
+    carry the section's bound information.
   }];
   let arguments = (ins OpenACC_AnyPointerOrMappableType:$var,
+                       Variadic<OpenACC_DataBoundsType>:$bounds,
                        OpenACC_ReductionOperatorAttr:$reductionOperator);
   let results = (outs OpenACC_AnyPointerOrMappableType:$result);
   let regions = (region AnyRegion:$region);
   let assemblyFormat = [{
-    $var $reductionOperator `:` type($result) $region attr-dict
+    $var (`bounds` `(` $bounds^ `)`)? $reductionOperator `:` type($result)
+    $region attr-dict
   }];
+  let builders = [
+    // Convenience builder for reductions without array-section bounds.
+    OpBuilder<(ins "::mlir::Value":$var,
+                   "::mlir::acc::ReductionOperatorAttr":$reductionOperator), [{
+      build($_builder, $_state, var, ::mlir::ValueRange{}, reductionOperator);
+    }]>,
+    OpBuilder<(ins "::mlir::Value":$var,
+                   "::mlir::acc::ReductionOperator":$reductionOperator), [{
+      build($_builder, $_state, var, ::mlir::ValueRange{}, reductionOperator);
+    }]>,
+  ];
   let hasVerifier = 1;
 }
 

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index e666bcd9a6242..70aacd251c329 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -344,8 +344,10 @@ LogicalResult ACCRecipeMaterialization::materialize(
     else
       llvm_unreachable("unexpected acc op with reduction recipe");
 
-    auto reductionOp = acc::ReductionInitOp::create(
-        b, op.getLoc(), origPtr, recipe.getReductionOperatorAttr());
+    SmallVector<Value> reductionBounds(acc::getBounds(op));
+    auto reductionOp =
+        acc::ReductionInitOp::create(b, op.getLoc(), origPtr, reductionBounds,
+                                     recipe.getReductionOperatorAttr());
     saveVarName(op.getAccVar(), reductionOp.getResult());
     cloneRegionIntoAccRegion(&initRegion, &reductionOp.getRegion(),
                              /*hasResult=*/true);

diff  --git a/mlir/test/Dialect/OpenACC/ops-cg.mlir b/mlir/test/Dialect/OpenACC/ops-cg.mlir
index 765552e8577a7..cf1c3340870a5 100644
--- a/mlir/test/Dialect/OpenACC/ops-cg.mlir
+++ b/mlir/test/Dialect/OpenACC/ops-cg.mlir
@@ -160,6 +160,20 @@ func.func @compute_region_two_dims(%data: memref<8xi32>,
 
 // -----
 
+// CHECK-LABEL: func @reduction_init_with_bounds
+func.func @reduction_init_with_bounds(%arg0: memref<?xi32>, %lb: index, %ext: index) {
+  %c1 = arith.constant 1 : index
+  %bnd = acc.bounds lowerbound(%lb : index) extent(%ext : index) stride(%c1 : index)
+  %0 = acc.reduction_init %arg0 bounds(%bnd) <add> : memref<?xi32> {
+    acc.yield %arg0 : memref<?xi32>
+  }
+  return
+}
+// CHECK: %[[BND:.*]] = acc.bounds
+// CHECK: acc.reduction_init %{{.*}} bounds(%[[BND]]) <add> : memref<?xi32>
+
+// -----
+
 // CHECK-LABEL: func @compute_region_unknown_width
 func.func @compute_region_unknown_width(%data: memref<100xf32>) {
   %copyin = acc.copyin varPtr(%data : memref<100xf32>) -> memref<100xf32>


        

From llvmlistbot at llvm.org  Fri Jul 17 14:59:24 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Fri, 17 Jul 2026 14:59:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Preserve section bounds on
	acc.reduction_init (PR #210443)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210443@github.com>
Message-ID: <6a5aa5bc.d681cab9.169270.4415@mx.google.com>


https://github.com/razvanlupusoru closed https://github.com/llvm/llvm-project/pull/210443

From llvmlistbot at llvm.org  Fri Jul 17 15:46:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 15:46:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Privatize worker reduction
 accumulators per worker (PR #210456)
Message-ID: <llvm.org/llvm/llvm-project/pull/210456@github.com>


https://github.com/khaki3 created https://github.com/llvm/llvm-project/pull/210456

Example:
```fortran
!$acc parallel
!$acc loop gang reduction(+:sum)
do k = 1, p
  !$acc loop worker reduction(+:sum)
  do j = 1, n
    !$acc loop vector reduction(+:sum)
    do i = 1, m
      sum = sum + a(i,j,k)
    end do
  end do
end do
```

In this code, the worker accumulator is both gang- and worker-scoped. Shared
memory allocated only one copy, so workers raced while updating the same
value.

Fix: allocate one shared-memory accumulator per worker whenever worker
parallelism is present.

>From 8c75f15b36cd9624373bbeaf5754b51642029340 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Fri, 17 Jul 2026 15:34:48 -0700
Subject: [PATCH] [mlir][OpenACC] Privatize worker reduction accumulators per
 worker

Allocate one shared-memory slot per worker even when the accumulator is also gang-scoped, preventing workers from racing on one block-wide value.
---
 .../Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp  | 15 +++---
 ...cc-cg-to-gpu-worker-reduction-private.mlir | 41 +++++++++++++++
 .../Dialect/OpenACC/OpenACCUtilsCGTest.cpp    | 50 +++++++++++++++++--
 3 files changed, 94 insertions(+), 12 deletions(-)
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index cd2e732dbc817..5f4af6a5bcc5b 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -319,8 +319,8 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
 
 static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
     PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
-    bool isWorkerPrivate, bool isGangPrivate, OpenACCSupport *support) {
-  if (!isWorkerPrivate || isGangPrivate)
+    bool isWorkerPrivate, OpenACCSupport *support) {
+  if (!isWorkerPrivate)
     return std::optional<int64_t>(1);
 
   GPUParallelDimAttr threadY =
@@ -394,8 +394,8 @@ FailureOr<bool> isPrivateLocalSharedMemoryCandidate(
     return false;
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(
-          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate, support);
+      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
+                                            isWorkerPrivate, support);
   if (failed(numCopies))
     return failure();
   return numCopies->has_value();
@@ -411,15 +411,12 @@ std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
 
   SmallVector<GPUParallelDimAttr> parDims =
       collectPrivateLocalParDims(privateLocal, computeRegion);
-  bool isGangPrivate =
-      llvm::any_of(parDims, [&](auto parDim) { return policy.isGang(parDim); });
   bool isWorkerPrivate = llvm::any_of(
       parDims, [&](auto parDim) { return policy.isWorker(parDim); });
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
-                                            isWorkerPrivate, isGangPrivate,
-                                            /*support=*/nullptr);
+      getWorkerPrivateSharedMemoryNumCopies(
+          privateLocal, computeRegion, isWorkerPrivate, /*support=*/nullptr);
   if (failed(numCopies) || !numCopies->has_value())
     return std::nullopt;
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
new file mode 100644
index 0000000000000..3866c8057f129
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
@@ -0,0 +1,41 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// CHECK-LABEL: func.func @worker_reduction_private
+// CHECK: acc.gpu_shared_memory
+// CHECK-SAME: num_copies = 4
+
+func.func @worker_reduction_private() {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block = acc.par_width %c2 {par_dim = #acc.par_dim<block_x>}
+  %worker = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %vector = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>] : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%bx = %block, %wy = %worker, %vx = %vector)
+        ins(%private_arg = %private) : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%bx) step (%c1) {
+        scf.parallel (%k) = (%block_iv) to (%bx) step (%bx) {
+          %local = acc.private_local %private_arg
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+              : (!acc.private_type<memref<i32>>) -> memref<i32>
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.parallel (%worker_iv) = (%c0) to (%wy) step (%c1) {
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[thread_y]>}
+          %value = memref.load %local[] : memref<i32>
+          acc.reduction_accumulate %value to %local <add>
+              : i32 -> memref<i32> {par_dims = #acc<par_dims[thread_y]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 66e500c4d2100..8c9f057f15c50 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -42,11 +42,13 @@ class OpenACCUtilsCGTest : public ::testing::Test {
   static ComputeRegionOp buildComputeRegionWithPrivateLocal(
       MLIRContext &context, OpBuilder &b, Location loc, ModuleOp module,
       GPUParallelDimsAttr privatizeParDims, ValueRange launchArgs,
-      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut) {
+      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut,
+      MemRefType memTy = {}, bool addReductionAccumulator = false) {
     IRRewriter rewriter(&context);
     rewriter.setInsertionPointToStart(module.getBody());
 
-    MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+    if (!memTy)
+      memTy = MemRefType::get({4}, b.getI32Type());
     Type privateTy = PrivateType::get(&context, memTy);
     privatizeOut = PrivatizeOp::create(rewriter, loc, privateTy, ValueRange{},
                                        privatizeParDims);
@@ -71,7 +73,20 @@ class OpenACCUtilsCGTest : public ::testing::Test {
       setParDimsAttr(par, GPUParallelDimsAttr::get(&context, {parDim}));
       srcBuilder.setInsertionPoint(par.getBody()->getTerminator());
     }
-    PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    PrivateLocalOp privateLocal =
+        PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    if (addReductionAccumulator) {
+      Value partial =
+          arith::ConstantIntOp::create(srcBuilder, loc, b.getI32Type(), 1);
+      SmallVector<GPUParallelDimAttr> reductionDims;
+      for (GPUParallelDimAttr parDim : privatizeParDims.getArray())
+        if (!parDim.isAnyBlock())
+          reductionDims.push_back(parDim);
+      ReductionAccumulateOp::create(
+          srcBuilder, loc, partial, privateLocal.getResult(),
+          ReductionOperator::AccAdd,
+          GPUParallelDimsAttr::get(&context, reductionDims));
+    }
 
     IRMapping mapping;
     auto cr = buildComputeRegion(
@@ -748,6 +763,35 @@ TEST_F(OpenACCUtilsCGTest,
   EXPECT_TRUE(*isCandidate);
 }
 
+TEST_F(OpenACCUtilsCGTest, getSharedMemoryBytesGangWorkerReductionAccumulator) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+  GPUParallelDimsAttr gangWorkerDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context),
+                 GPUParallelDimAttr::threadYDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+  auto ty =
+      ParWidthOp::create(b, loc, c4, GPUParallelDimAttr::threadYDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  MemRefType scalarTy = MemRefType::get({}, b.getI32Type());
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangWorkerDims,
+      ValueRange{bx.getResult(), ty.getResult()}, privateLocal, privatize,
+      scalarTy, /*addReductionAccumulator=*/true);
+
+  DefaultACCToGPUMappingPolicy policy;
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 16);
+}
+
 TEST_F(OpenACCUtilsCGTest,
        isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);


From llvmlistbot at llvm.org  Fri Jul 17 15:47:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 15:47:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Privatize worker reduction
 accumulators per worker (PR #210456)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210456@github.com>
Message-ID: <6a5ab106.743400b0.23a7b2.3042@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Matsu (khaki3)

<details>
<summary>Changes</summary>

Example:
```fortran
!$acc parallel
!$acc loop gang reduction(+:sum)
do k = 1, p
  !$acc loop worker reduction(+:sum)
  do j = 1, n
    !$acc loop vector reduction(+:sum)
    do i = 1, m
      sum = sum + a(i,j,k)
    end do
  end do
end do
```

In this code, the worker accumulator is both gang- and worker-scoped. Shared
memory allocated only one copy, so workers raced while updating the same
value.

Fix: allocate one shared-memory accumulator per worker whenever worker
parallelism is present.

---
Full diff: https://github.com/llvm/llvm-project/pull/210456.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp (+6-9) 
- (added) mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir (+41) 
- (modified) mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp (+47-3) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index cd2e732dbc817..5f4af6a5bcc5b 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -319,8 +319,8 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
 
 static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
     PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
-    bool isWorkerPrivate, bool isGangPrivate, OpenACCSupport *support) {
-  if (!isWorkerPrivate || isGangPrivate)
+    bool isWorkerPrivate, OpenACCSupport *support) {
+  if (!isWorkerPrivate)
     return std::optional<int64_t>(1);
 
   GPUParallelDimAttr threadY =
@@ -394,8 +394,8 @@ FailureOr<bool> isPrivateLocalSharedMemoryCandidate(
     return false;
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(
-          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate, support);
+      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
+                                            isWorkerPrivate, support);
   if (failed(numCopies))
     return failure();
   return numCopies->has_value();
@@ -411,15 +411,12 @@ std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
 
   SmallVector<GPUParallelDimAttr> parDims =
       collectPrivateLocalParDims(privateLocal, computeRegion);
-  bool isGangPrivate =
-      llvm::any_of(parDims, [&](auto parDim) { return policy.isGang(parDim); });
   bool isWorkerPrivate = llvm::any_of(
       parDims, [&](auto parDim) { return policy.isWorker(parDim); });
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
-                                            isWorkerPrivate, isGangPrivate,
-                                            /*support=*/nullptr);
+      getWorkerPrivateSharedMemoryNumCopies(
+          privateLocal, computeRegion, isWorkerPrivate, /*support=*/nullptr);
   if (failed(numCopies) || !numCopies->has_value())
     return std::nullopt;
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
new file mode 100644
index 0000000000000..3866c8057f129
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
@@ -0,0 +1,41 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// CHECK-LABEL: func.func @worker_reduction_private
+// CHECK: acc.gpu_shared_memory
+// CHECK-SAME: num_copies = 4
+
+func.func @worker_reduction_private() {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block = acc.par_width %c2 {par_dim = #acc.par_dim<block_x>}
+  %worker = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %vector = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>] : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%bx = %block, %wy = %worker, %vx = %vector)
+        ins(%private_arg = %private) : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%bx) step (%c1) {
+        scf.parallel (%k) = (%block_iv) to (%bx) step (%bx) {
+          %local = acc.private_local %private_arg
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+              : (!acc.private_type<memref<i32>>) -> memref<i32>
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.parallel (%worker_iv) = (%c0) to (%wy) step (%c1) {
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[thread_y]>}
+          %value = memref.load %local[] : memref<i32>
+          acc.reduction_accumulate %value to %local <add>
+              : i32 -> memref<i32> {par_dims = #acc<par_dims[thread_y]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 66e500c4d2100..8c9f057f15c50 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -42,11 +42,13 @@ class OpenACCUtilsCGTest : public ::testing::Test {
   static ComputeRegionOp buildComputeRegionWithPrivateLocal(
       MLIRContext &context, OpBuilder &b, Location loc, ModuleOp module,
       GPUParallelDimsAttr privatizeParDims, ValueRange launchArgs,
-      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut) {
+      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut,
+      MemRefType memTy = {}, bool addReductionAccumulator = false) {
     IRRewriter rewriter(&context);
     rewriter.setInsertionPointToStart(module.getBody());
 
-    MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+    if (!memTy)
+      memTy = MemRefType::get({4}, b.getI32Type());
     Type privateTy = PrivateType::get(&context, memTy);
     privatizeOut = PrivatizeOp::create(rewriter, loc, privateTy, ValueRange{},
                                        privatizeParDims);
@@ -71,7 +73,20 @@ class OpenACCUtilsCGTest : public ::testing::Test {
       setParDimsAttr(par, GPUParallelDimsAttr::get(&context, {parDim}));
       srcBuilder.setInsertionPoint(par.getBody()->getTerminator());
     }
-    PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    PrivateLocalOp privateLocal =
+        PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    if (addReductionAccumulator) {
+      Value partial =
+          arith::ConstantIntOp::create(srcBuilder, loc, b.getI32Type(), 1);
+      SmallVector<GPUParallelDimAttr> reductionDims;
+      for (GPUParallelDimAttr parDim : privatizeParDims.getArray())
+        if (!parDim.isAnyBlock())
+          reductionDims.push_back(parDim);
+      ReductionAccumulateOp::create(
+          srcBuilder, loc, partial, privateLocal.getResult(),
+          ReductionOperator::AccAdd,
+          GPUParallelDimsAttr::get(&context, reductionDims));
+    }
 
     IRMapping mapping;
     auto cr = buildComputeRegion(
@@ -748,6 +763,35 @@ TEST_F(OpenACCUtilsCGTest,
   EXPECT_TRUE(*isCandidate);
 }
 
+TEST_F(OpenACCUtilsCGTest, getSharedMemoryBytesGangWorkerReductionAccumulator) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+  GPUParallelDimsAttr gangWorkerDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context),
+                 GPUParallelDimAttr::threadYDim(&context)});
+  auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto bx =
+      ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
+  auto ty =
+      ParWidthOp::create(b, loc, c4, GPUParallelDimAttr::threadYDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  MemRefType scalarTy = MemRefType::get({}, b.getI32Type());
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangWorkerDims,
+      ValueRange{bx.getResult(), ty.getResult()}, privateLocal, privatize,
+      scalarTy, /*addReductionAccumulator=*/true);
+
+  DefaultACCToGPUMappingPolicy policy;
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 16);
+}
+
 TEST_F(OpenACCUtilsCGTest,
        isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);

``````````

</details>


https://github.com/llvm/llvm-project/pull/210456

From llvmlistbot at llvm.org  Fri Jul 17 16:49:54 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Fri, 17 Jul 2026 16:49:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Extend WrapFuncInClassAdd by
 adding source attribute (PR #207069)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207069@github.com>
Message-ID: <6a5abfa2.d681cab9.169270.f1fd@mx.google.com>


https://github.com/beamandala closed https://github.com/llvm/llvm-project/pull/207069

From llvmlistbot at llvm.org  Fri Jul 17 16:55:53 2026
From: llvmlistbot at llvm.org (Jared Hoberock)
Date: Fri, 17 Jul 2026 16:55:53 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][IR] Add SymbolUserTypeInterface
	(PR #198435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198435@github.com>
Message-ID: <6a5ac109.7270d2fd.a9bd6.6309@mx.google.com>


jaredhoberock wrote:

@joker-eph @jpienaar has approved. Would you please merge it for me?

https://github.com/llvm/llvm-project/pull/198435

From llvmlistbot at llvm.org  Fri Jul 17 19:29:24 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Fri, 17 Jul 2026 19:29:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a5ae504.13be6df8.22eac4.9d0e@mx.google.com>


https://github.com/lforg37 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH 1/4] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {

>From 7be327e24300bcd5da269c9bb161fbc1733b0443 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 16 Jul 2026 19:38:41 +0900
Subject: [PATCH 2/4] [mlir][wasm] Avoid failure to build with old gcc by not
 relying on sfinae for instantiating parsers

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 160 ++++++++-------------
 1 file changed, 59 insertions(+), 101 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 872e91179a76c..92bc71e8d55fa 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -425,8 +425,10 @@ class ExpressionParser {
   class TopLevelInstParserRegistry {
   public:
     template <std::byte opCode>
-    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
-                                              ExpressionParser &) = delete;
+    static constexpr bool hasParserForOpcode = false;
+
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &, ExpressionParser &) = delete;
   };
 
 private:
@@ -446,26 +448,20 @@ unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
 template <typename ParserRegistry>
 class InstDispatcher {
 private:
-  /// used to check if a parser is registered in the parser registry for a
-  /// given opcode.
-  /// Default to unreachable handler.
-  template <std::byte opCode, typename = void>
-  struct ParserInfo : std::false_type {
-    static constexpr auto handler = unreachableHandler;
-  };
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
   template <std::byte opCode>
-  struct ParserInfo<
-      opCode,
-      std::void_t<
-          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
-      : std::true_type {
-    static constexpr auto handler =
-        ParserRegistry::template parseInstrWithOpCode<opCode>;
-  };
+  static constexpr dispatch_t getHandlerForOpCode() {
+    if constexpr (ParserRegistry::template hasParserForOpcode<opCode>)
+      return ParserRegistry::template parseInstrWithOpCode<opCode>;
+    else
+      return unreachableHandler;
+  }
 
 public:
   template <std::byte opCode>
-  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+  static constexpr bool isValidInst =
+      ParserRegistry::template hasParserForOpcode<opCode>;
 
 private:
   static inline parsed_inst_t
@@ -476,8 +472,6 @@ class InstDispatcher {
            << static_cast<int>(opCode);
   }
 
-  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
-
   template <std::byte... opCodes>
   static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
                                            ExpressionParser &exprParser,
@@ -485,7 +479,7 @@ class InstDispatcher {
     static constexpr std::array<bool, 256> opcodeValidityMap{
         isValidInst<opCodes>...};
     static constexpr std::array<dispatch_t, 256> dispatchTable{
-        ParserInfo<opCodes>::handler...};
+        getHandlerForOpCode<opCodes>()...};
     if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
       return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
     }
@@ -1059,27 +1053,24 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
-                                       ExpressionParser &exprParser) {
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+  template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<opcode> = true;                                       \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
+      OpBuilder & builder, ExpressionParser & exprParser)
+
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1124,11 +1115,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1155,11 +1142,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {*branchArgs};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1177,11 +1160,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {callOp.getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1194,11 +1173,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
                .getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1235,19 +1210,11 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1319,35 +1286,19 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1371,6 +1322,9 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX> = true;   \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1475,24 +1429,19 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
-                                                ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::SOURCE_OP> = true;        \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1528,6 +1477,11 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S> =   \
+          true;                                                                \
+  template <>                                                                  \
   parsed_inst_t                                                                \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
@@ -1546,16 +1500,16 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
-                                                 ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
@@ -1565,6 +1519,10 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   }                                                                            \
                                                                                \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \

>From ed54961e33dc45ed1f54086b3c8dcb2244fb3929 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:17:25 +0900
Subject: [PATCH 3/4] [mlir][wasm]  Centralize check of presence of
 currentOpLoc in wasm parser

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 92bc71e8d55fa..b7cb1c9e97f54 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -27,6 +27,7 @@
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/LogicalResult.h"
 
+#include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <variant>
@@ -420,7 +421,11 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
-  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+  Location getCurrentOpLoc() {
+    assert(currentOpLoc.has_value() &&
+           "expects current opcode location to be set");
+    return *currentOpLoc;
+  }
 
   class TopLevelInstParserRegistry {
   public:
@@ -467,7 +472,7 @@ class InstDispatcher {
   static inline parsed_inst_t
   invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
                     std::byte opCode) {
-    return emitError(*(expressionParser.getCurrentOpLoc()),
+    return emitError(expressionParser.getCurrentOpLoc(),
                      "unknown instruction opcode: ")
            << static_cast<int>(opCode);
   }
@@ -1089,11 +1094,11 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
+  locations.resize(resTypes.size(), exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+  auto ifOp = IfOp::create(builder, exprParser.getCurrentOpLoc(),
                            conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
@@ -1135,7 +1140,7 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
+  BranchIfOp::create(builder, exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);

>From 736689f5f7ae6fda09ea1846a5dfef39bc71a39e Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:32:10 +0900
Subject: [PATCH 4/4] [wasm][mlir] Add one level of parser macro to prepare
 registration of multi byte opcodes.

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 68 +++++++++++++++-------
 1 file changed, 46 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index b7cb1c9e97f54..0e1c67a730b69 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -1058,24 +1058,35 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+// We can't use SFINAE in combination with deleted default instantiation
+// to identify which parsers are registered due to GCC < 14.1 bug, so we
+// use a constexpr variable to register them.
+// This is in order to avoid having to have only one "registration" of the
+// opcode.
+#define REGISTER_PARSER_OPCODE_PARSER(parserType, opcode, builderName,         \
+                                      parserName)                              \
   template <>                                                                  \
-  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
-      hasParserForOpcode<opcode> = true;                                       \
+  constexpr bool parserType::hasParserForOpcode<opcode> = true;                \
   template <>                                                                  \
-  inline parsed_inst_t                                                         \
-  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
-      OpBuilder & builder, ExpressionParser & exprParser)
+  inline parsed_inst_t parserType::parseInstrWithOpCode<opcode>(               \
+      OpBuilder & (builderName), ExpressionParser & (parserName))
+
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode, builderName, parserName)     \
+  REGISTER_PARSER_OPCODE_PARSER(ExpressionParser::TopLevelInstParserRegistry,  \
+                                opcode, builderName, parserName)
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode, builder,
+                                  exprParser) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1120,7 +1131,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   return {ValueRange{successor->getArguments()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf, builder,
+                                  exprParser) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1147,7 +1159,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   return {*branchArgs};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call, builder,
+                                  exprParser) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1165,7 +1178,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   return {callOp.getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet, builder,
+                                  exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1178,7 +1192,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
                .getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet,
+                                  builder, exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1215,11 +1230,13 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1291,19 +1308,23 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1434,11 +1455,13 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap, builder,
+                                  exprParser) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
@@ -1505,7 +1528,8 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 


From llvmlistbot at llvm.org  Fri Jul 17 19:35:46 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Fri, 17 Jul 2026 19:35:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a5ae682.1e1ac41d.3969c3.3a2d@mx.google.com>


https://github.com/lforg37 updated https://github.com/llvm/llvm-project/pull/195500

>From 1b1b49f5d325b4c43ceb51e856925317bd4eeeb9 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sun, 3 May 2026 14:42:38 +0900
Subject: [PATCH 1/5] [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA
 importer

Refactored the WasmSSA importer mechanism which dispatches control
flow to the relevant instruction parser based on op code.

This is to prepare for instructions with multi-bytes opcodes
(e.g. vector instruction and some scalar instructions extensions)
which will be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.
---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 354 +++++++++++++--------
 1 file changed, 224 insertions(+), 130 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 048e964037558..5a5b2e204dd4f 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -237,6 +237,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +254,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +286,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +420,15 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +437,78 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  /// used to check if a parser is registered in the parser registry for a
+  /// given opcode.
+  /// Default to unreachable handler.
+  template <std::byte opCode, typename = void>
+  struct ParserInfo : std::false_type {
+    static constexpr auto handler = unreachableHandler;
+  };
+  template <std::byte opCode>
+  struct ParserInfo<
+      opCode,
+      std::void_t<
+          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
+      : std::true_type {
+    static constexpr auto handler =
+        ParserRegistry::template parseInstrWithOpCode<opCode>;
+  };
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(*(expressionParser.getCurrentOpLoc()),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        ParserInfo<opCodes>::handler...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1010,32 +1061,36 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
+                                       ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1098,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1070,16 +1125,18 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,10 +1145,10 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
@@ -1100,18 +1157,19 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1120,30 +1178,36 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1172,15 +1236,19 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1252,27 +1320,35 @@ parsed_inst_t ExpressionParser::parseConstInst(
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
+                                          ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
+                                           ExpressionParser &exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1371,10 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,23 +1475,27 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
 }
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
+                                                ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
 template <>
 inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
+                                      ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1435,12 +1516,14 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1452,24 +1535,30 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 #undef BUILD_SLICE_EXTEND_PARSER
 
 template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+inline parsed_inst_t
+ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
+    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
+                                                 ExpressionParser &exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1477,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {

>From 7be327e24300bcd5da269c9bb161fbc1733b0443 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 16 Jul 2026 19:38:41 +0900
Subject: [PATCH 2/5] [mlir][wasm] Avoid failure to build with old gcc by not
 relying on sfinae for instantiating parsers

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 160 ++++++++-------------
 1 file changed, 59 insertions(+), 101 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 872e91179a76c..92bc71e8d55fa 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -425,8 +425,10 @@ class ExpressionParser {
   class TopLevelInstParserRegistry {
   public:
     template <std::byte opCode>
-    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
-                                              ExpressionParser &) = delete;
+    static constexpr bool hasParserForOpcode = false;
+
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &, ExpressionParser &) = delete;
   };
 
 private:
@@ -446,26 +448,20 @@ unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
 template <typename ParserRegistry>
 class InstDispatcher {
 private:
-  /// used to check if a parser is registered in the parser registry for a
-  /// given opcode.
-  /// Default to unreachable handler.
-  template <std::byte opCode, typename = void>
-  struct ParserInfo : std::false_type {
-    static constexpr auto handler = unreachableHandler;
-  };
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
   template <std::byte opCode>
-  struct ParserInfo<
-      opCode,
-      std::void_t<
-          decltype(&ParserRegistry::template parseInstrWithOpCode<opCode>)>>
-      : std::true_type {
-    static constexpr auto handler =
-        ParserRegistry::template parseInstrWithOpCode<opCode>;
-  };
+  static constexpr dispatch_t getHandlerForOpCode() {
+    if constexpr (ParserRegistry::template hasParserForOpcode<opCode>)
+      return ParserRegistry::template parseInstrWithOpCode<opCode>;
+    else
+      return unreachableHandler;
+  }
 
 public:
   template <std::byte opCode>
-  static constexpr bool isValidInst = ParserInfo<opCode>::value;
+  static constexpr bool isValidInst =
+      ParserRegistry::template hasParserForOpcode<opCode>;
 
 private:
   static inline parsed_inst_t
@@ -476,8 +472,6 @@ class InstDispatcher {
            << static_cast<int>(opCode);
   }
 
-  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
-
   template <std::byte... opCodes>
   static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
                                            ExpressionParser &exprParser,
@@ -485,7 +479,7 @@ class InstDispatcher {
     static constexpr std::array<bool, 256> opcodeValidityMap{
         isValidInst<opCodes>...};
     static constexpr std::array<dispatch_t, 256> dispatchTable{
-        ParserInfo<opCodes>::handler...};
+        getHandlerForOpCode<opCodes>()...};
     if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
       return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
     }
@@ -1059,27 +1053,24 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::block>(OpBuilder &builder,
-                                       ExpressionParser &exprParser) {
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+  template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<opcode> = true;                                       \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
+      OpBuilder & builder, ExpressionParser & exprParser)
+
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::loop>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1124,11 +1115,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1155,11 +1142,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {*branchArgs};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::call>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1177,11 +1160,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   return {callOp.getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1194,11 +1173,7 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
                .getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1235,19 +1210,11 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1319,35 +1286,19 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder,
-                                          ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder,
-                                           ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1371,6 +1322,9 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX> = true;   \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1475,24 +1429,19 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder,
-                                                ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::wrap>(OpBuilder &builder,
-                                      ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::SOURCE_OP> = true;        \
+  template <>                                                                  \
   inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
       parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
           OpBuilder & builder, ExpressionParser & exprParser) {                \
@@ -1528,6 +1477,11 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S> =   \
+          true;                                                                \
+  template <>                                                                  \
   parsed_inst_t                                                                \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
@@ -1546,16 +1500,16 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-template <>
-inline parsed_inst_t
-ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder,
-                                                 ExpressionParser &exprParser) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
@@ -1565,6 +1519,10 @@ ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<
   }                                                                            \
                                                                                \
   template <>                                                                  \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH> = true; \
+  template <>                                                                  \
   inline parsed_inst_t                                                         \
   ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \

>From ed54961e33dc45ed1f54086b3c8dcb2244fb3929 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:17:25 +0900
Subject: [PATCH 3/5] [mlir][wasm]  Centralize check of presence of
 currentOpLoc in wasm parser

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 92bc71e8d55fa..b7cb1c9e97f54 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -27,6 +27,7 @@
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/LogicalResult.h"
 
+#include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <variant>
@@ -420,7 +421,11 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
-  std::optional<Location> getCurrentOpLoc() { return currentOpLoc; }
+  Location getCurrentOpLoc() {
+    assert(currentOpLoc.has_value() &&
+           "expects current opcode location to be set");
+    return *currentOpLoc;
+  }
 
   class TopLevelInstParserRegistry {
   public:
@@ -467,7 +472,7 @@ class InstDispatcher {
   static inline parsed_inst_t
   invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
                     std::byte opCode) {
-    return emitError(*(expressionParser.getCurrentOpLoc()),
+    return emitError(expressionParser.getCurrentOpLoc(),
                      "unknown instruction opcode: ")
            << static_cast<int>(opCode);
   }
@@ -1089,11 +1094,11 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *exprParser.getCurrentOpLoc());
+  locations.resize(resTypes.size(), exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *exprParser.getCurrentOpLoc(),
+  auto ifOp = IfOp::create(builder, exprParser.getCurrentOpLoc(),
                            conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
@@ -1135,7 +1140,7 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *exprParser.getCurrentOpLoc(), condition->front(),
+  BranchIfOp::create(builder, exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);

>From 736689f5f7ae6fda09ea1846a5dfef39bc71a39e Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Fri, 17 Jul 2026 08:32:10 +0900
Subject: [PATCH 4/5] [wasm][mlir] Add one level of parser macro to prepare
 registration of multi byte opcodes.

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 68 +++++++++++++++-------
 1 file changed, 46 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index b7cb1c9e97f54..0e1c67a730b69 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -1058,24 +1058,35 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode)                              \
+// We can't use SFINAE in combination with deleted default instantiation
+// to identify which parsers are registered due to GCC < 14.1 bug, so we
+// use a constexpr variable to register them.
+// This is in order to avoid having to have only one "registration" of the
+// opcode.
+#define REGISTER_PARSER_OPCODE_PARSER(parserType, opcode, builderName,         \
+                                      parserName)                              \
   template <>                                                                  \
-  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
-      hasParserForOpcode<opcode> = true;                                       \
+  constexpr bool parserType::hasParserForOpcode<opcode> = true;                \
   template <>                                                                  \
-  inline parsed_inst_t                                                         \
-  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<opcode>(  \
-      OpBuilder & builder, ExpressionParser & exprParser)
+  inline parsed_inst_t parserType::parseInstrWithOpCode<opcode>(               \
+      OpBuilder & (builderName), ExpressionParser & (parserName))
+
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode, builderName, parserName)     \
+  REGISTER_PARSER_OPCODE_PARSER(ExpressionParser::TopLevelInstParserRegistry,  \
+                                opcode, builderName, parserName)
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop, builder,
+                                  exprParser) {
   return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode, builder,
+                                  exprParser) {
   auto opLoc = exprParser.currentOpLoc;
   auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
@@ -1120,7 +1131,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode) {
   return {ValueRange{successor->getArguments()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf, builder,
+                                  exprParser) {
   auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
@@ -1147,7 +1159,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf) {
   return {*branchArgs};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call, builder,
+                                  exprParser) {
   auto loc = *exprParser.currentOpLoc;
   auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
@@ -1165,7 +1178,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call) {
   return {callOp.getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet, builder,
+                                  exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1178,7 +1192,8 @@ REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet) {
                .getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet,
+                                  builder, exprParser) {
   FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
   Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
@@ -1215,11 +1230,13 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee, builder,
+                                  exprParser) {
   return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
@@ -1291,19 +1308,23 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int32_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64, builder,
+                                  exprParser) {
   return exprParser.parseConstInst<int64_t>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64,
+                                  builder, exprParser) {
   return exprParser.parseConstInst<double>(builder);
 }
 
@@ -1434,11 +1455,13 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap, builder,
+                                  exprParser) {
   return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
@@ -1505,7 +1528,8 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64) {
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64,
+                                  builder, exprParser) {
   return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 

>From 49069514d9176fcad66c1520aadb3dd402c78fd4 Mon Sep 17 00:00:00 2001
From: Luc Forget <dev at alias.lforget.fr>
Date: Sat, 18 Jul 2026 11:34:15 +0900
Subject: [PATCH 5/5] [mlir][wasm] Formatting

---
 mlir/lib/Target/Wasm/TranslateFromWasm.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 0e1c67a730b69..cea6bd2816dc1 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -433,7 +433,8 @@ class ExpressionParser {
     static constexpr bool hasParserForOpcode = false;
 
     template <std::byte opCode>
-    static parsed_inst_t parseInstrWithOpCode(OpBuilder &, ExpressionParser &) = delete;
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
   };
 
 private:


From llvmlistbot at llvm.org  Fri Jul 17 19:39:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 19:39:51 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][LLVMIR] Avoid duplicate
 retained local variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5ae777.e55dc341.1c6a0.dafe@mx.google.com>


qyingwu wrote:

> It seems like you implemented some logic to break the cycle at the local variable instead of at the subprogram. That should not be necessary? Note that self recursive machinery is extremely complex and fragile. We should avoid changing it and stick as close as possible to the PR I linked.
> 
> What was the reason for breaking the cycle at the local variable rather at the subprogram? Did the imported example debug info look unexpected?
> 
> Note I didn't get to the end with the review. I think it makes sense to move the closer to the PR I linked first.

I tried the suggested approach locally, but it does not fix #206713.

With that approach, the issue test imports roughly as:

  ```mlir
  #di_subprogram = #llvm.di_subprogram<recId = distinct[0]<>, isRecSelf = true>

  #di_local_variable = #llvm.di_local_variable<
    scope = #di_subprogram, name = "n", ...
  >

  #di_subprogram1 = #llvm.di_subprogram<
    recId = distinct[0]<>, id = distinct[2]<>, ...,
    retainedNodes = [#di_local_variable]
  >

  #di_local_variable1 = #llvm.di_local_variable<
    scope = #di_subprogram1, name = "n", ...
  >

So the retainedNodes import creates one local variable scoped by the self-recursive subprogram placeholder, and the
later dbg.value import creates another local variable scoped by the finalized subprogram. It is the same LLVM DILocalVariable, but it becomes two different MLIR attrs.

That is why I broke the cycle at the local variable for retainedNodes that are also referenced by debug intrinsics. I’ll keep looking for a way to simplify the patch and stay closer to the existing recursive machinery.

https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Fri Jul 17 19:59:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 17 Jul 2026 19:59:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1ef8ba9 - [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (#195500)
Message-ID: <6a5aec0c.c4d3d80a.849c3.832f@mx.google.com>


Author: Luc Forget
Date: 2026-07-18T11:59:18+09:00
New Revision: 1ef8ba9a00e16423d711ddbf16c7a45fb7fd570b

URL: https://github.com/llvm/llvm-project/commit/1ef8ba9a00e16423d711ddbf16c7a45fb7fd570b
DIFF: https://github.com/llvm/llvm-project/commit/1ef8ba9a00e16423d711ddbf16c7a45fb7fd570b.diff

LOG: [MLIR][WasmSSA] Instruction parser refactoring of WasmSSA importer (#195500)

Refactored WasmSSA importer mechanism to dispatch control flow to
relevant parser based on op code.
This is to prepare for instructions with multi-bytes opcodes (e.g.
vector instruction and some scalar instructions extensions) which will
be able to reuse the same mechanism.

It also replaces the bit tree to find the address by a jump table.

---------

Co-authored-by: Luc Forget <lforg37 at users.noreply.github.com>
Co-authored-by: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Co-authored-by: Ferdinand Lemaire <flscminecraft at gmail.com>

Added: 
    

Modified: 
    mlir/lib/Target/Wasm/TranslateFromWasm.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
index 1a0d53f1207f6..cea6bd2816dc1 100644
--- a/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
+++ b/mlir/lib/Target/Wasm/TranslateFromWasm.cpp
@@ -27,6 +27,7 @@
 #include "llvm/Support/LEB128.h"
 #include "llvm/Support/LogicalResult.h"
 
+#include <cassert>
 #include <cstddef>
 #include <cstdint>
 #include <variant>
@@ -237,6 +238,15 @@ class ValueStack {
 
 using local_val_t = TypedValue<wasmssa::LocalRefType>;
 
+template <size_t... IS>
+constexpr ByteSequence<std::byte{IS}...>
+    castIndexSequenceToBytes(std::index_sequence<IS...>) {
+  return {};
+}
+
+constexpr auto all8bitsBytes =
+    castIndexSequenceToBytes(std::make_index_sequence<256>());
+
 class ExpressionParser {
 public:
   using locals_t = SmallVector<local_val_t>;
@@ -245,9 +255,6 @@ class ExpressionParser {
       : parser{parser}, symbols{symbols}, locals{initLocal} {}
 
 private:
-  template <std::byte opCode>
-  inline parsed_inst_t parseSpecificInstruction(OpBuilder &builder);
-
   template <typename valueT>
   parsed_inst_t
   parseConstInst(OpBuilder &builder,
@@ -280,38 +287,8 @@ class ExpressionParser {
             typename... extraArgsT>
   inline parsed_inst_t buildConvertOp(OpBuilder &builder, extraArgsT...);
 
-  /// This function generates a dispatch tree to associate an opcode with a
-  /// parser. Parsers are registered by specialising the
-  /// `parseSpecificInstruction` function for the op code to handle.
-  ///
-  /// The dispatcher is generated by recursively creating all possible patterns
-  /// for an opcode and calling the relevant parser on the leaf.
-  ///
-  /// @tparam patternBitSize is the first bit for which the pattern is not fixed
-  ///
-  /// @tparam highBitPattern is the fixed pattern that this instance handles for
-  /// the 8-patternBitSize bits
-  template <size_t patternBitSize = 0, std::byte highBitPattern = std::byte{0}>
   inline parsed_inst_t dispatchToInstParser(std::byte opCode,
-                                            OpBuilder &builder) {
-    static_assert(patternBitSize <= 8,
-                  "PatternBitSize is outside of range of opcode space! "
-                  "(expected at most 8 bits)");
-    if constexpr (patternBitSize < 8) {
-      constexpr std::byte bitSelect{1 << (7 - patternBitSize)};
-      constexpr std::byte nextHighBitPatternStem = highBitPattern << 1;
-      constexpr size_t nextPatternBitSize = patternBitSize + 1;
-      if ((opCode & bitSelect) != std::byte{0})
-        return dispatchToInstParser<nextPatternBitSize,
-                                    nextHighBitPatternStem | std::byte{1}>(
-            opCode, builder);
-      return dispatchToInstParser<nextPatternBitSize, nextHighBitPatternStem>(
-          opCode, builder);
-    } else {
-      return parseSpecificInstruction<highBitPattern>(builder);
-    }
-  }
-
+                                            OpBuilder &builder);
   ///
   /// RAII guard class for creating a nesting level
   ///
@@ -444,6 +421,22 @@ class ExpressionParser {
   template <typename OpToCreate>
   parsed_inst_t parseBlockLikeOp(OpBuilder &);
 
+  Location getCurrentOpLoc() {
+    assert(currentOpLoc.has_value() &&
+           "expects current opcode location to be set");
+    return *currentOpLoc;
+  }
+
+  class TopLevelInstParserRegistry {
+  public:
+    template <std::byte opCode>
+    static constexpr bool hasParserForOpcode = false;
+
+    template <std::byte opCode>
+    static parsed_inst_t parseInstrWithOpCode(OpBuilder &,
+                                              ExpressionParser &) = delete;
+  };
+
 private:
   std::optional<Location> currentOpLoc;
   ParserHead &parser;
@@ -452,6 +445,70 @@ class ExpressionParser {
   ValueStack valueStack;
 };
 
+static inline parsed_inst_t
+unreachableHandler(OpBuilder &, ExpressionParser &expressionParser) {
+  llvm_unreachable("Failure in opcode parser dispatch logic.");
+  return mlir::failure();
+}
+
+template <typename ParserRegistry>
+class InstDispatcher {
+private:
+  using dispatch_t = parsed_inst_t (*)(OpBuilder &, ExpressionParser &);
+
+  template <std::byte opCode>
+  static constexpr dispatch_t getHandlerForOpCode() {
+    if constexpr (ParserRegistry::template hasParserForOpcode<opCode>)
+      return ParserRegistry::template parseInstrWithOpCode<opCode>;
+    else
+      return unreachableHandler;
+  }
+
+public:
+  template <std::byte opCode>
+  static constexpr bool isValidInst =
+      ParserRegistry::template hasParserForOpcode<opCode>;
+
+private:
+  static inline parsed_inst_t
+  invalidOpcodeDiag(OpBuilder &, ExpressionParser &expressionParser,
+                    std::byte opCode) {
+    return emitError(expressionParser.getCurrentOpLoc(),
+                     "unknown instruction opcode: ")
+           << static_cast<int>(opCode);
+  }
+
+  template <std::byte... opCodes>
+  static inline parsed_inst_t dispatchImpl(std::byte opCode, OpBuilder &builder,
+                                           ExpressionParser &exprParser,
+                                           ByteSequence<opCodes...>) {
+    static constexpr std::array<bool, 256> opcodeValidityMap{
+        isValidInst<opCodes>...};
+    static constexpr std::array<dispatch_t, 256> dispatchTable{
+        getHandlerForOpCode<opCodes>()...};
+    if (opcodeValidityMap[static_cast<size_t>(opCode)]) {
+      return dispatchTable[static_cast<size_t>(opCode)](builder, exprParser);
+    }
+    return invalidOpcodeDiag(builder, exprParser, opCode);
+  }
+
+public:
+  ///
+  /// @brief dispatch control flow to the sub parser registered for opCode in
+  /// `ParserRegistry`
+  ///
+  /// @param opCode opCode of the instruction to be Parsed
+  /// @param builder builder that will be passed to the parser
+  /// @param exprParser the generic parser passed to the sub parser
+  ///
+  /// @return the result of the parser or an error if there is no parser
+  /// registered for the opcode (emits a diagnostic)
+  static parsed_inst_t dispatch(std::byte opCode, OpBuilder &builder,
+                                ExpressionParser &exprParser) {
+    return dispatchImpl(opCode, builder, exprParser, all8bitsBytes);
+  }
+};
+
 class ParserHead {
 public:
   ParserHead(StringRef src, StringAttr name) : head{src}, locName{name} {}
@@ -850,12 +907,6 @@ inline FailureOr<int64_t> ParserHead::parseI64() {
   return parseLiteral<int64_t>();
 }
 
-template <std::byte opCode>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction(OpBuilder &) {
-  return emitError(*currentOpLoc, "unknown instruction opcode: ")
-         << static_cast<int>(opCode);
-}
-
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 void ValueStack::dump() const {
   llvm::dbgs() << "================= Wasm ValueStack =======================\n";
@@ -1008,34 +1059,46 @@ parsed_inst_t ExpressionParser::parseBlockLikeOp(OpBuilder &builder) {
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::block>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<BlockOp>(builder);
+// We can't use SFINAE in combination with deleted default instantiation
+// to identify which parsers are registered due to GCC < 14.1 bug, so we
+// use a constexpr variable to register them.
+// This is in order to avoid having to have only one "registration" of the
+// opcode.
+#define REGISTER_PARSER_OPCODE_PARSER(parserType, opcode, builderName,         \
+                                      parserName)                              \
+  template <>                                                                  \
+  constexpr bool parserType::hasParserForOpcode<opcode> = true;                \
+  template <>                                                                  \
+  inline parsed_inst_t parserType::parseInstrWithOpCode<opcode>(               \
+      OpBuilder & (builderName), ExpressionParser & (parserName))
+
+#define REGISTER_PRIMARY_WASM_INST_PARSER(opcode, builderName, parserName)     \
+  REGISTER_PARSER_OPCODE_PARSER(ExpressionParser::TopLevelInstParserRegistry,  \
+                                opcode, builderName, parserName)
+
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::block, builder,
+                                  exprParser) {
+  return exprParser.parseBlockLikeOp<BlockOp>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::loop>(
-    OpBuilder &builder) {
-  return parseBlockLikeOp<LoopOp>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::loop, builder,
+                                  exprParser) {
+  return exprParser.parseBlockLikeOp<LoopOp>(builder);
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::ifOpCode>(OpBuilder &builder) {
-  auto opLoc = currentOpLoc;
-  auto funcType = parseBlockFuncType(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::ifOpCode, builder,
+                                  exprParser) {
+  auto opLoc = exprParser.currentOpLoc;
+  auto funcType = exprParser.parseBlockFuncType(builder);
   if (failed(funcType))
     return failure();
 
   LDBG() << "Parsing an if instruction of type " << *funcType;
   auto inputTypes = funcType->getInputs();
-  auto conditionValue = popOperands(builder.getI32Type());
+  auto conditionValue = exprParser.popOperands(builder.getI32Type());
   if (failed(conditionValue))
     return failure();
-  auto inputOps = popOperands(inputTypes);
+  auto inputOps = exprParser.popOperands(inputTypes);
   if (failed(inputOps))
     return failure();
 
@@ -1043,25 +1106,25 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   Region *curRegion = curBlock->getParent();
   auto resTypes = funcType->getResults();
   llvm::SmallVector<Location> locations{};
-  locations.resize(resTypes.size(), *currentOpLoc);
+  locations.resize(resTypes.size(), exprParser.getCurrentOpLoc());
   auto *successor =
       builder.createBlock(curRegion, curRegion->end(), resTypes, locations);
   builder.setInsertionPointToEnd(curBlock);
-  auto ifOp = IfOp::create(builder, *currentOpLoc, conditionValue->front(),
-                           *inputOps, successor);
+  auto ifOp = IfOp::create(builder, exprParser.getCurrentOpLoc(),
+                           conditionValue->front(), *inputOps, successor);
   auto *ifEntryBlock = ifOp.createIfBlock();
   constexpr auto ifElseFilter =
       ByteSequence<WasmBinaryEncoding::endByte,
                    WasmBinaryEncoding::OpCode::elseOpCode>{};
-  auto parseIfRes = parseBlockContent(builder, ifEntryBlock, resTypes, *opLoc,
-                                      ifOp, ifElseFilter);
+  auto parseIfRes = exprParser.parseBlockContent(
+      builder, ifEntryBlock, resTypes, *opLoc, ifOp, ifElseFilter);
   if (failed(parseIfRes))
     return failure();
   if (*parseIfRes == WasmBinaryEncoding::OpCode::elseOpCode) {
     LDBG() << "  else block is present.";
     Block *elseEntryBlock = ifOp.createElseBlock();
-    auto parseElseRes =
-        parseBlockContent(builder, elseEntryBlock, resTypes, *opLoc, ifOp);
+    auto parseElseRes = exprParser.parseBlockContent(builder, elseEntryBlock,
+                                                     resTypes, *opLoc, ifOp);
     if (failed(parseElseRes))
       return failure();
   }
@@ -1069,17 +1132,16 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   return {ValueRange{successor->getArguments()}};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::branchIf>(OpBuilder &builder) {
-  auto level = parser.parseLiteral<uint32_t>();
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::branchIf, builder,
+                                  exprParser) {
+  auto level = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(level))
     return failure();
   Block *curBlock = builder.getBlock();
   Region *curRegion = curBlock->getParent();
   auto sip = builder.saveInsertionPoint();
   Block *elseBlock = builder.createBlock(curRegion, curRegion->end());
-  auto condition = popOperands(builder.getI32Type());
+  auto condition = exprParser.popOperands(builder.getI32Type());
   if (failed(condition))
     return failure();
   builder.restoreInsertionPoint(sip);
@@ -1088,30 +1150,28 @@ inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
   if (failed(targetOp))
     return failure();
   auto inputTypes = targetOp->getLabelTarget()->getArgumentTypes();
-  auto branchArgs = popOperands(inputTypes);
+  auto branchArgs = exprParser.popOperands(inputTypes);
   if (failed(branchArgs))
     return failure();
-  BranchIfOp::create(builder, *currentOpLoc, condition->front(),
+  BranchIfOp::create(builder, exprParser.getCurrentOpLoc(), condition->front(),
                      builder.getUI32IntegerAttr(*level), *branchArgs,
                      elseBlock);
   builder.setInsertionPointToStart(elseBlock);
   return {*branchArgs};
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
-    OpBuilder &builder) {
-  auto loc = *currentOpLoc;
-  auto funcIdx = parser.parseLiteral<uint32_t>();
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::call, builder,
+                                  exprParser) {
+  auto loc = *exprParser.currentOpLoc;
+  auto funcIdx = exprParser.parser.parseLiteral<uint32_t>();
   if (failed(funcIdx))
     return failure();
-  if (*funcIdx >= symbols.funcSymbols.size())
+  if (*funcIdx >= exprParser.symbols.funcSymbols.size())
     return emitError(loc, "Invalid function index: ") << *funcIdx;
-  auto callee = symbols.funcSymbols[*funcIdx];
+  auto callee = exprParser.symbols.funcSymbols[*funcIdx];
   llvm::ArrayRef<Type> inTypes = callee.functionType.getInputs();
   llvm::ArrayRef<Type> resTypes = callee.functionType.getResults();
-  parsed_inst_t inOperands = popOperands(inTypes);
+  parsed_inst_t inOperands = exprParser.popOperands(inTypes);
   if (failed(inOperands))
     return failure();
   auto callOp =
@@ -1119,31 +1179,31 @@ ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::call>(
   return {callOp.getResults()};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localGet, builder,
+                                  exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= locals.size())
+  if (*id >= exprParser.locals.size())
     return emitError(instLoc, "invalid local index. function has ")
-           << locals.size() << " accessible locals, received index " << *id;
-  return {{LocalGetOp::create(builder, instLoc, locals[*id]).getResult()}};
+           << exprParser.locals.size() << " accessible locals, received index "
+           << *id;
+  return {{LocalGetOp::create(builder, instLoc, exprParser.locals[*id])
+               .getResult()}};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::globalGet>(OpBuilder &builder) {
-  FailureOr<uint32_t> id = parser.parseLiteral<uint32_t>();
-  Location instLoc = *currentOpLoc;
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::globalGet,
+                                  builder, exprParser) {
+  FailureOr<uint32_t> id = exprParser.parser.parseLiteral<uint32_t>();
+  Location instLoc = *exprParser.currentOpLoc;
   if (failed(id))
     return failure();
-  if (*id >= symbols.globalSymbols.size())
+  if (*id >= exprParser.symbols.globalSymbols.size())
     return emitError(instLoc, "invalid global index. function has ")
-           << symbols.globalSymbols.size()
+           << exprParser.symbols.globalSymbols.size()
            << " accessible globals, received index " << *id;
-  GlobalSymbolRefContainer globalVar = symbols.globalSymbols[*id];
+  GlobalSymbolRefContainer globalVar = exprParser.symbols.globalSymbols[*id];
   auto globalOp = GlobalGetOp::create(builder, instLoc, globalVar.globalType,
                                       globalVar.symbol);
 
@@ -1171,16 +1231,14 @@ parsed_inst_t ExpressionParser::parseSetOrTee(OpBuilder &builder) {
           ->getResults()};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localSet>(OpBuilder &builder) {
-  return parseSetOrTee<LocalSetOp>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localSet, builder,
+                                  exprParser) {
+  return exprParser.parseSetOrTee<LocalSetOp>(builder);
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::localTee>(OpBuilder &builder) {
-  return parseSetOrTee<LocalTeeOp>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::localTee, builder,
+                                  exprParser) {
+  return exprParser.parseSetOrTee<LocalTeeOp>(builder);
 }
 
 template <typename T>
@@ -1251,28 +1309,24 @@ parsed_inst_t ExpressionParser::parseConstInst(
   return {{constOp.getResult()}};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI32>(OpBuilder &builder) {
-  return parseConstInst<int32_t>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI32, builder,
+                                  exprParser) {
+  return exprParser.parseConstInst<int32_t>(builder);
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constI64>(OpBuilder &builder) {
-  return parseConstInst<int64_t>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constI64, builder,
+                                  exprParser) {
+  return exprParser.parseConstInst<int64_t>(builder);
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP32>(OpBuilder &builder) {
-  return parseConstInst<float>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP32,
+                                  builder, exprParser) {
+  return exprParser.parseConstInst<float>(builder);
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::constFP64>(OpBuilder &builder) {
-  return parseConstInst<double>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::constFP64,
+                                  builder, exprParser) {
+  return exprParser.parseConstInst<double>(builder);
 }
 
 template <typename opcode, typename valueType, unsigned int numOperands>
@@ -1295,9 +1349,13 @@ inline parsed_inst_t ExpressionParser::buildNumericOp(
 // Convenience macro for generating numerical operations.
 #define BUILD_NUMERIC_OP(OP_NAME, N_ARGS, PREFIX, SUFFIX, TYPE)                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(OpBuilder & builder) {       \
-    return buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);                     \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX> = true;   \
+  template <>                                                                  \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::PREFIX##SUFFIX>(        \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildNumericOp<OP_NAME, TYPE, N_ARGS>(builder);          \
   }
 
 // Macro to define binops that only support integer types.
@@ -1398,24 +1456,25 @@ inline parsed_inst_t ExpressionParser::buildConvertOp(OpBuilder &builder,
   return {{op.getResult()}};
 }
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::demoteF64ToF32>(OpBuilder &builder) {
-  return buildConvertOp<DemoteOp, double, float>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::demoteF64ToF32,
+                                  builder, exprParser) {
+  return exprParser.buildConvertOp<DemoteOp, double, float>(builder);
 }
 
-template <>
-inline parsed_inst_t
-ExpressionParser::parseSpecificInstruction<WasmBinaryEncoding::OpCode::wrap>(
-    OpBuilder &builder) {
-  return buildConvertOp<WrapOp, int64_t, int32_t>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::wrap, builder,
+                                  exprParser) {
+  return exprParser.buildConvertOp<WrapOp, int64_t, int32_t>(builder);
 }
 
 #define BUILD_CONVERSION_OP(IN_T, OUT_T, SOURCE_OP, TARGET_OP)                 \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::SOURCE_OP>(OpBuilder & builder) {            \
-    return buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);                    \
+  constexpr bool ExpressionParser::TopLevelInstParserRegistry::                \
+      hasParserForOpcode<WasmBinaryEncoding::OpCode::SOURCE_OP> = true;        \
+  template <>                                                                  \
+  inline parsed_inst_t ExpressionParser::TopLevelInstParserRegistry::          \
+      parseInstrWithOpCode<WasmBinaryEncoding::OpCode::SOURCE_OP>(             \
+          OpBuilder & builder, ExpressionParser & exprParser) {                \
+    return exprParser.buildConvertOp<TARGET_OP, IN_T, OUT_T>(builder);         \
   }
 
 #define BUILD_CONVERT_OP_FOR(DEST_T, WIDTH)                                    \
@@ -1447,12 +1506,19 @@ BUILD_CONVERSION_OP(int32_t, int64_t, extendU, ExtendUI32Op)
 
 #define BUILD_SLICE_EXTEND_PARSER(IT_WIDTH, EXTRACT_WIDTH)                     \
   template <>                                                                  \
-  parsed_inst_t ExpressionParser::parseSpecificInstruction<                    \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S> =   \
+          true;                                                                \
+  template <>                                                                  \
+  parsed_inst_t                                                                \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
       WasmBinaryEncoding::OpCode::extendI##IT_WIDTH##EXTRACT_WIDTH##S>(        \
-      OpBuilder & builder) {                                                   \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
     using inout_t = int##IT_WIDTH##_t;                                         \
     auto attr = builder.getUI32IntegerAttr(EXTRACT_WIDTH);                     \
-    return buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(builder, attr);  \
+    return exprParser.buildConvertOp<ExtendLowBitsSOp, inout_t, inout_t>(      \
+        builder, attr);                                                        \
   }
 
 BUILD_SLICE_EXTEND_PARSER(32, 8)
@@ -1463,25 +1529,36 @@ BUILD_SLICE_EXTEND_PARSER(64, 32)
 
 #undef BUILD_SLICE_EXTEND_PARSER
 
-template <>
-inline parsed_inst_t ExpressionParser::parseSpecificInstruction<
-    WasmBinaryEncoding::OpCode::promoteF32ToF64>(OpBuilder &builder) {
-  return buildConvertOp<PromoteOp, float, double>(builder);
+REGISTER_PRIMARY_WASM_INST_PARSER(WasmBinaryEncoding::OpCode::promoteF32ToF64,
+                                  builder, exprParser) {
+  return exprParser.buildConvertOp<PromoteOp, float, double>(builder);
 }
 
 #define BUILD_REINTERPRET_PARSER(WIDTH, FP_TYPE)                               \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(builder);    \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH> = true; \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretF##WIDTH##AsI##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, FP_TYPE, int##WIDTH##_t>(  \
+        builder);                                                              \
   }                                                                            \
                                                                                \
   template <>                                                                  \
-  inline parsed_inst_t ExpressionParser::parseSpecificInstruction<             \
-      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(OpBuilder & \
-                                                                   builder) {  \
-    return buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(builder);    \
+  constexpr bool                                                               \
+      ExpressionParser::TopLevelInstParserRegistry::hasParserForOpcode<        \
+          WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH> = true; \
+  template <>                                                                  \
+  inline parsed_inst_t                                                         \
+  ExpressionParser::TopLevelInstParserRegistry::parseInstrWithOpCode<          \
+      WasmBinaryEncoding::OpCode::reinterpretI##WIDTH##AsF##WIDTH>(            \
+      OpBuilder & builder, ExpressionParser & exprParser) {                    \
+    return exprParser.buildConvertOp<ReinterpretOp, int##WIDTH##_t, FP_TYPE>(  \
+        builder);                                                              \
   }
 
 BUILD_REINTERPRET_PARSER(32, float)
@@ -1489,6 +1566,11 @@ BUILD_REINTERPRET_PARSER(64, double)
 
 #undef BUILD_REINTERPRET_PARSER
 
+inline parsed_inst_t
+ExpressionParser::dispatchToInstParser(std::byte opCode, OpBuilder &builder) {
+  return InstDispatcher<ExpressionParser::TopLevelInstParserRegistry>::dispatch(
+      opCode, builder, *this);
+}
 class WasmBinaryParser {
 private:
   struct SectionRegistry {


        

From llvmlistbot at llvm.org  Fri Jul 17 19:59:27 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Fri, 17 Jul 2026 19:59:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Instruction parser
 refactoring of WasmSSA importer (PR #195500)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195500@github.com>
Message-ID: <6a5aec0f.1728eaf3.2a2f6.5038@mx.google.com>


https://github.com/flemairen6 closed https://github.com/llvm/llvm-project/pull/195500

From llvmlistbot at llvm.org  Fri Jul 17 22:16:26 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Fri, 17 Jul 2026 22:16:26 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][LLVMIR] Avoid duplicate
 retained local variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5b0c2a.f06f4994.2f0620.11d4@mx.google.com>


gysit wrote:

How does the debug info look if we export it back to LLVM IR?

It may be ok if we have a duplication in MLIR as long as the exported debug information is correct. 


https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Sat Jul 18 02:09:07 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Sat, 18 Jul 2026 02:09:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add support for Static Single
	Information types (PR #210435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210435@github.com>
Message-ID: <6a5b42b3.e5907290.281687.e962@mx.google.com>


================
@@ -252,6 +252,16 @@ template <typename ConcreteType, template <typename> class TraitType>
 using TraitBase = detail::StorageUserTraitBase<ConcreteType, TraitType>;
 } // namespace TypeTrait
 
+namespace TypeTrait {
+/// Types with this trait opt out of redundant block argument simplification
+/// during region simplification. Block arguments of these types are not
+/// eliminated when all predecessors pass the same value. Use this to preserve
+/// SSI (Static Single Information) form where block arguments carry semantic
+/// meaning beyond their SSA def-use structure.
----------------
joker-eph wrote:

An SSA value carrying "meaning beyond their SSA def-use structure" does not seem to me like something MLIR is modeling. I'm not totally comfortable introducing such a notion here: I suspect this may have further ramifications in the system beyond the one instance in `dropRedundantArguments`.

https://github.com/llvm/llvm-project/pull/210435

From llvmlistbot at llvm.org  Sat Jul 18 02:14:44 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 02:14:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Add support for Static Single
	Information types (PR #210435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210435@github.com>
Message-ID: <6a5b4404.1e1ac41d.3969c3.49e3@mx.google.com>


matthias-springer wrote:

Can you share some more details about your use case? Note that "entry block arguments" are special (semantics defined by the operation) and not blindly eliminated. Is that not sufficient?


https://github.com/llvm/llvm-project/pull/210435

From llvmlistbot at llvm.org  Sat Jul 18 02:32:09 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b4819.d1496011.199a59.40db@mx.google.com>


https://github.com/FedericoBruzzone commented:

Hi Andrzej! Great catch on this PR. Thanks so much.
Overall, LGTM :D

I'll have some minor comments, and here's a concern of mine.
Currently, the verifiers are checking the equality between shapes:
https://github.com/llvm/llvm-project/blob/ccd0e982a02fa3be39a98dd98ea6cc1ccaf24d13/mlir/lib/Dialect/Vector/IR/VectorOps.cpp#L6710-L6711
https://github.com/llvm/llvm-project/blob/ccd0e982a02fa3be39a98dd98ea6cc1ccaf24d13/mlir/lib/Dialect/Vector/IR/VectorOps.cpp#L6656-L6657
Should we take into account the scalable dims via `getScalableDims()` or something?

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:32:09 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b4819.743400b0.23a7b2.6af9@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:32:09 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b4819.bd93e731.30f155.4c95@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Would it be right to update the documentation? :D

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:32:09 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b4819.c56fb102.2d07b8.ee94@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Would it be right to update the documentation? :D

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:32:10 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b481a.eb165e5c.4506c.47f9@mx.google.com>


================

----------------
FedericoBruzzone wrote:

I’m not sure if this is appropriate for the scope of this PR. However, it might be worth adding an e2e test file for `expandload`, mirroring `compress.mlir` within ArmSVE (this one)?

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:32:10 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:32:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b481a.7d702ed2.13d102.afc0@mx.google.com>


================
@@ -0,0 +1,214 @@
+// REQUIRES: arm-emulator
+
+/// End-to-end test for vector.compressstore for SVE
+
+// In order to demonstrate the impact of using scalable vectors, vscale is set
+// to 2 so that vector<[16]xi32> constains 32 rather than 16 elements at
----------------
FedericoBruzzone wrote:

```suggestion
// to 2 so that vector<[16]xi32> contains 32 rather than 16 elements at
```

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:34:43 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:34:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b48b3.160324b0.1f653e.f70f@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Would it be right to update the documentation? :D

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:35:37 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:35:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b48e9.481afadf.9e047.2d75@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Would it be right to update the documentation? :D

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:37:26 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:37:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b4956.4a342cbc.3a2f36.bc49@mx.google.com>


================

----------------
FedericoBruzzone wrote:


Sorry, wrong place. It was a careless mistake. See below :(

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:37:31 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 02:37:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b495b.824e11a6.9c376.9468@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Sorry, wrong place. It was a careless mistake. See below :(

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 02:50:02 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Sat, 18 Jul 2026 02:50:02 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][IR] Add SymbolUserTypeInterface
	(PR #198435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198435@github.com>
Message-ID: <6a5b4c4a.4987c784.2c60a9.9d9d@mx.google.com>


https://github.com/joker-eph closed https://github.com/llvm/llvm-project/pull/198435

From llvmlistbot at llvm.org  Sat Jul 18 03:41:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:41:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Replace dead
 function argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a5b5849.481afadf.9e047.2913@mx.google.com>


https://github.com/jjppp edited https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 18 03:46:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:46:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Replace dead
 function argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a5b599a.6c5720b3.2da3b9.9aae@mx.google.com>


jjppp wrote:

Ping. @joker-eph  @jpienaar , could you please take a look when you have a chance?

This is a follow-up to #153973 and the discussion in #206920.
In an unreachable private function, liveness analysis marks the function arguments as dead, while some side-effecting operations may still survive the cleanup.
Calling `dropAllUses()` on such an argument leaves null operands in those surviving operations, and the final canonicalization then crashes while folding invalid IR.

This patch replaces the remaining uses with `ub.poison` before erasing the argument. This follows the existing invariant in RemoveDeadValues, where live uses of erased operation results are also replaced with poison to keep the IR valid.

I would especially appreciate feedback on whether using poison here is the intended solution, as opposed to skipping dead-value cleanup inside unreachable functions ;-)

https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 18 03:51:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:51:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
Message-ID: <llvm.org/llvm/llvm-project/pull/210501@github.com>


https://github.com/PragmaTwice created https://github.com/llvm/llvm-project/pull/210501

Python-defined dialect loading currently relies on `Dialect._mlir_module` to infer whether a dialect has already been loaded. This state belongs to the Python dialect class rather than an MLIR context.

Consequently, loading the same dialect after switching contexts requires `reload=True`, while reloading it in a context where it is already present can hit the operation registration assertion reported in #210053.

This patch adds `mlirContextGetLoadedDialect` (following https://github.com/llvm/lighthouse/pull/228#discussion_r3589792891) to the C API and exposes it as `Context.is_dialect_loaded`. `Dialect.load()` now queries the active context:
- loading a dialect more than once in the same context raises a `RuntimeError`;
- loading the same Python-defined dialect in another context succeeds without a `reload` flag.

The `reload` parameter is semantically wrong (mentioned in https://github.com/llvm/lighthouse/pull/228#discussion_r3586672116) thus removed from the API as a small breaking change. Usually, users who are using `reload` can just remove it and everything will work fine.

Fixes #210053.
Assisted by GPT 5.6 Sol (for writing test cases).

>From 5da91e24140fbccb052b8ceeef422b3d9f7191bc Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 18 Jul 2026 18:37:27 +0800
Subject: [PATCH] [MLIR][Python] Make Python-defined dialect loading
 context-aware

---
 mlir/include/mlir-c/IR.h                      |  7 ++
 mlir/lib/Bindings/Python/IRCore.cpp           |  8 +++
 mlir/lib/CAPI/IR/IR.cpp                       |  5 ++
 mlir/python/mlir/dialects/ext.py              | 29 ++++----
 mlir/test/python/dialects/ext.py              | 66 +++++++++++++++++++
 .../python/dialects/transform_op_interface.py |  2 +-
 ...ansform_pattern_descriptor_op_interface.py |  2 +-
 7 files changed, 100 insertions(+), 19 deletions(-)

diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..3ea214da40531 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                          MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.


From llvmlistbot at llvm.org  Sat Jul 18 03:51:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:51:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5b5ac4.8e75e0cb.f69c7.c8b6@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Twice (PragmaTwice)

<details>
<summary>Changes</summary>

Python-defined dialect loading currently relies on `Dialect._mlir_module` to infer whether a dialect has already been loaded. This state belongs to the Python dialect class rather than an MLIR context.

Consequently, loading the same dialect after switching contexts requires `reload=True`, while reloading it in a context where it is already present can hit the operation registration assertion reported in #<!-- -->210053.

This patch adds `mlirContextGetLoadedDialect` (following https://github.com/llvm/lighthouse/pull/228#discussion_r3589792891) to the C API and exposes it as `Context.is_dialect_loaded`. `Dialect.load()` now queries the active context:
- loading a dialect more than once in the same context raises a `RuntimeError`;
- loading the same Python-defined dialect in another context succeeds without a `reload` flag.

The `reload` parameter is semantically wrong (mentioned in https://github.com/llvm/lighthouse/pull/228#discussion_r3586672116) thus removed from the API as a small breaking change. Usually, users who are using `reload` can just remove it and everything will work fine.

Fixes #<!-- -->210053.
Assisted by GPT 5.6 Sol (for writing test cases).

---
Full diff: https://github.com/llvm/llvm-project/pull/210501.diff


7 Files Affected:

- (modified) mlir/include/mlir-c/IR.h (+7) 
- (modified) mlir/lib/Bindings/Python/IRCore.cpp (+8) 
- (modified) mlir/lib/CAPI/IR/IR.cpp (+5) 
- (modified) mlir/python/mlir/dialects/ext.py (+12-17) 
- (modified) mlir/test/python/dialects/ext.py (+66) 
- (modified) mlir/test/python/dialects/transform_op_interface.py (+1-1) 
- (modified) mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py (+1-1) 


``````````diff
diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..3ea214da40531 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                          MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.

``````````

</details>


https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 03:52:12 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Sat, 18 Jul 2026 03:52:12 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5b5adc.13be6df8.22eac4.d189@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 024877c8be7853d993f97e932619bcd4be67779e Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH 1/3] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 clang/lib/CodeGen/TargetInfo.cpp              |  3 +
 .../cl20-device-side-enqueue-attributes.cl    |  4 +-
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 50 +++++++-------
 22 files changed, 146 insertions(+), 136 deletions(-)

diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..af90be4b1a47b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26587,9 +26589,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..5da8fac2fc914 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)

>From 52c14150dc12d14815aafb489f0ccd5c93e5be74 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 18:42:48 +0100
Subject: [PATCH 2/3] [LangRef/Verfier] Drop bad propagation terminology

---
 llvm/docs/LangRef.md     | 5 ++---
 llvm/lib/IR/Verifier.cpp | 2 +-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index af90be4b1a47b..9526cd020d724 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,9 +2794,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap. The
-    attribute propagates from callee to caller, and all function definitions
-    that contain strictfp calls must be marked strictfp.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5da8fac2fc914..d315372c1371c 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3891,7 +3891,7 @@ void Verifier::visitCallBase(CallBase &Call) {
 
   Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
             Call.getFunction()->isStrictFP(),
-        "strictfp attribute must propagate from the call site to the caller",
+        "call site marked strictfp without caller function marked strictfp",
         Call);
 
   // Verify call attributes.

>From 6ed37c5dcb9f8b0a43d0ecf19371fddcdd73f931 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Fri, 17 Jul 2026 11:01:45 +0100
Subject: [PATCH 3/3] [IR] Add Verifier test

---
 llvm/unittests/IR/VerifierTest.cpp | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/llvm/unittests/IR/VerifierTest.cpp b/llvm/unittests/IR/VerifierTest.cpp
index 48d461cab330a..f6a89cde0901b 100644
--- a/llvm/unittests/IR/VerifierTest.cpp
+++ b/llvm/unittests/IR/VerifierTest.cpp
@@ -699,4 +699,28 @@ TEST(VerifierTest, IntrinsicRetInvalidStruct) {
   }
 }
 
+TEST(VerifierTest, InvalidStrictFPAttribute) {
+  LLVMContext Ctx;
+  Module M("M", Ctx);
+  FunctionType *FuncTy =
+      FunctionType::get(Type::getVoidTy(Ctx), /*isVarArg=*/false);
+  Function *F =
+      Function::Create(FuncTy, Function::ExternalLinkage, "strictfp_test", M);
+  BasicBlock *Entry = BasicBlock::Create(Ctx, "entry", F);
+  Type *FloatTy = Type::getFloatTy(Ctx);
+
+  Function *CosF =
+      Intrinsic::getOrInsertDeclaration(&M, Intrinsic::cos, FloatTy);
+  CallInst *CI = CallInst::Create(CosF, ConstantFP::getNullValue(FloatTy),
+                                  "strictfp_call", Entry);
+  CI->addFnAttr(Attribute::StrictFP);
+  ReturnInst::Create(Ctx, Entry);
+
+  std::string Error;
+  raw_string_ostream ErrorOS(Error);
+  EXPECT_TRUE(verifyModule(M, &ErrorOS));
+  EXPECT_TRUE(StringRef(Error).starts_with(
+      "call site marked strictfp without caller function marked strictfp"))
+      << Error;
+}
 } // end anonymous namespace


From llvmlistbot at llvm.org  Sat Jul 18 03:52:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:52:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSVE] Fix comment inconsistency for
 `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC) (PR #210502)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210502@github.com>
Message-ID: <6a5b5b00.d681cab9.169270.b161@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-linalg

@llvm/pr-subscribers-mlir

Author: Federico Bruzzone (FedericoBruzzone)

<details>
<summary>Changes</summary>

This PR simply fix a comment inconsistency for `pack_lhs` in [ArmSVE/pack-unpack-mmt4d.mlir](https://github.com/llvm/llvm-project/compare/main...FedericoBruzzone:nfc-armsve?expand=1#diff-7be071860c440806b8dad954ed61946e7e9f27e032ed6044a704f0c0f70e4407) as identified in #<!-- -->208226.


---
Full diff: https://github.com/llvm/llvm-project/pull/210502.diff


1 Files Affected:

- (modified) mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir (+3-2) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
index 6192ed345debf..a5cc70119f595 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
@@ -119,9 +119,10 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 }
 
 //===----------------------------------------------------------------------===//
-// @matmul_via_mmt4d
+// @pack_lhs
 //
-// Implements matrix-multiplication via linalg.mmt4d
+// Implements packing for the A matrix (LHS) in matrix multiplication. The inner
+// tile size is fixed: 8 * 1.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
   %pad = arith.constant 0 : i32

``````````

</details>


https://github.com/llvm/llvm-project/pull/210502

From llvmlistbot at llvm.org  Sat Jul 18 03:52:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:52:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5b5b03.1b0626d0.f869d.4b05@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- mlir/include/mlir-c/IR.h mlir/lib/Bindings/Python/IRCore.cpp mlir/lib/CAPI/IR/IR.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 3ea214da4..98ea0dfc0 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -145,7 +145,7 @@ MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
 /// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
 /// registered with the context.
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
-                                                          MlirStringRef name);
+                                                           MlirStringRef name);
 
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,

``````````

</details>


https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 03:53:05 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 03:53:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSVE] Fix comment inconsistency for
 `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC) (PR #210502)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210502@github.com>
Message-ID: <6a5b5b11.1b0626d0.f869d.51ef@mx.google.com>


FedericoBruzzone wrote:

@AGindinson @banach-space It should be an easy merge :'D

https://github.com/llvm/llvm-project/pull/210502

From llvmlistbot at llvm.org  Sat Jul 18 03:59:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 03:59:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5b5cad.bc28b987.2e0c9a.1a77@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210501

>From 17d00d79d53674da4c0d54188ad2afe70dc60bf0 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 18 Jul 2026 18:37:27 +0800
Subject: [PATCH] [MLIR][Python] Make Python-defined dialect loading
 context-aware

---
 mlir/include/mlir-c/IR.h                      |  7 ++
 mlir/lib/Bindings/Python/IRCore.cpp           |  8 +++
 mlir/lib/CAPI/IR/IR.cpp                       |  5 ++
 mlir/python/mlir/dialects/ext.py              | 29 ++++----
 mlir/test/python/dialects/ext.py              | 66 +++++++++++++++++++
 .../python/dialects/transform_op_interface.py |  2 +-
 ...ansform_pattern_descriptor_op_interface.py |  2 +-
 7 files changed, 100 insertions(+), 19 deletions(-)

diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..3ea214da40531 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                          MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.


From llvmlistbot at llvm.org  Sat Jul 18 04:05:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 04:05:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5b5de8.a1accbb4.5eca2.e0a8@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210501

>From cd6ecf25efeece948522106a74e83d711c973027 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 18 Jul 2026 18:37:27 +0800
Subject: [PATCH] [MLIR][Python] Make Python-defined dialect loading
 context-aware

---
 mlir/include/mlir-c/IR.h                      |  7 ++
 mlir/lib/Bindings/Python/IRCore.cpp           |  8 +++
 mlir/lib/CAPI/IR/IR.cpp                       |  5 ++
 mlir/python/mlir/dialects/ext.py              | 29 ++++----
 mlir/test/python/dialects/ext.py              | 66 +++++++++++++++++++
 .../python/dialects/transform_op_interface.py |  2 +-
 ...ansform_pattern_descriptor_op_interface.py |  2 +-
 7 files changed, 100 insertions(+), 19 deletions(-)

diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..98ea0dfc00a1b 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                           MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.


From llvmlistbot at llvm.org  Sat Jul 18 04:17:54 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:17:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Replace dead
 function argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a5b60e2.b4a331f0.d9742.e7df@mx.google.com>


https://github.com/matthias-springer approved this pull request.

I'm wondering why `-remove-dead-values` does not erase the remaining function body. Given that "private" functions that are not called are considered unreachable.

In any case, this PR is fixing a crash and is improving the situation a bit.


https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 18 04:18:07 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:18:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][RemoveDeadValues] Replace dead
 function argument uses with poison (PR #208881)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208881@github.com>
Message-ID: <6a5b60ef.1e1ac41d.3969c3.9d70@mx.google.com>


https://github.com/matthias-springer edited https://github.com/llvm/llvm-project/pull/208881

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.eb165e5c.4506c.ff2c@mx.google.com>


================

----------------
matthias-springer wrote:

This seems unnecessary. You are just copying one vector into another and then returning it.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.d76d81af.23b347.7348@mx.google.com>


================

----------------
matthias-springer wrote:

I think I said we would want to error out (emitError + signalPassFailure) if we can't find a corresponding dealloc op.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.f1c1923d.2d2b59.3b1b@mx.google.com>


================

----------------
matthias-springer wrote:

Add TODO comment that this will have to be generalized with an interface in the future.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.d681cab9.169270.12f2@mx.google.com>


================

----------------
matthias-springer wrote:

nit: Instead of `erasedDeallocs`, you can maintain a SmallPtrSet of `deallocsToErase`, then the loop below will be simpler. (Inserting an element that's already part of the set is a no-op.)


https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.492e23e4.d19bc.9b3e@mx.google.com>


================

----------------
matthias-springer wrote:

Comment is outdated. The group constraint no longer exists.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6435.e624eb8d.4a8a7.838b@mx.google.com>


================

----------------
matthias-springer wrote:

Also error out if we find a dealloc op in a different block.

The only case where we should actually skip is when we see an alloc with dynamic shape.



https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:32:06 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:32:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5b6436.e5907290.281687.53cd@mx.google.com>


================

----------------
matthias-springer wrote:

Let's remove all wording about "fixpoint".

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sat Jul 18 04:35:45 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:35:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5b6511.27b3b003.1f84cc.2a34@mx.google.com>


https://github.com/matthias-springer commented:

I think there's a simpler fix: deactivate folding the `GreedyRewriteConfig`. Can you give that a try? Recomputing an analysis in a rewrite pattern could be expensive.


https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sat Jul 18 04:39:56 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:39:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][func] Fix a crash in
 DuplicateFunctionEliminationPass (PR #209667)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209667@github.com>
Message-ID: <6a5b660c.92ea0e2a.4a9a5.57a2@mx.google.com>


================
@@ -101,12 +101,12 @@ struct DuplicateFunctionEliminationPass
 
     // Update all symbol uses to reference unique func op
     // representants and erase redundant func ops.
-    SymbolTableCollection symbolTable;
-    SymbolUserMap userMap(symbolTable, module);
     for (auto it : toBeErased) {
       StringAttr oldSymbol = it.getSymNameAttr();
       StringAttr newSymbol = getRepresentant[oldSymbol].getSymNameAttr();
-      userMap.replaceAllUsesWith(it, newSymbol);
----------------
matthias-springer wrote:

Why did this crash? Is it a bug in the implementation of `replaceAllUsesWith`?

https://github.com/llvm/llvm-project/pull/209667

From llvmlistbot at llvm.org  Sat Jul 18 04:44:56 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:44:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Skip escaping
	allocation users (PR #208979)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208979@github.com>
Message-ID: <6a5b6738.481afadf.9e047.77df@mx.google.com>


================

----------------
matthias-springer wrote:

How is this related to the crash on the bug report? The crash there was in `mlir::Operation::isProperAncestor`. Why does this fix the crash?

https://github.com/llvm/llvm-project/pull/208979

From llvmlistbot at llvm.org  Sat Jul 18 04:45:52 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:45:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a5b6770.af6ba3ed.3c8c03.67f8@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 18 04:46:04 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:46:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a5b677c.27b3b003.1f84cc.7d2c@mx.google.com>


https://github.com/matthias-springer auto_merge_enabled https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 18 04:50:51 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:50:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Reject invalid boundary types (PR
	#208787)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208787@github.com>
Message-ID: <6a5b689b.c56fb102.2d07b8.323b@mx.google.com>


================

----------------
matthias-springer wrote:

This is not the right place to verify the element type. `getBufferizedFunctionArgType` returns a `BufferLikeType` which may or may not be a memref.

Probably the default implementation `functionArgTypeConverterFn` (and I think we have multiple of these lambdas) should error out by returning a "null" type.

https://github.com/llvm/llvm-project/pull/208787

From llvmlistbot at llvm.org  Sat Jul 18 04:52:01 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:52:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][interface] Use populateAndCompare
 cleanup compare (NFC) (PR #200156)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200156@github.com>
Message-ID: <6a5b68e1.1905e68e.30d957.9f6e@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/200156

From llvmlistbot at llvm.org  Sat Jul 18 04:58:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 04:58:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1f4d6ac - [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (#208973)
Message-ID: <6a5b6a79.d2cb4adc.ccd80.d5e8@mx.google.com>


Author: pzoln
Date: 2026-07-18T11:58:44Z
New Revision: 1f4d6acc935f93eca55cfef67348c8abcc1415db

URL: https://github.com/llvm/llvm-project/commit/1f4d6acc935f93eca55cfef67348c8abcc1415db
DIFF: https://github.com/llvm/llvm-project/commit/1f4d6acc935f93eca55cfef67348c8abcc1415db.diff

LOG: [mlir][bufferization] Avoid invalid memref.cast in to_buffer folding (#208973)

Fixes #202782

When folding bufferization.to_buffer(bufferization.to_tensor(...)), only
create a memref.cast if the source and destination memref types are
cast-compatible.

This avoids asserting on unranked memrefs with different memory spaces
and leaves the original bufferization ops in place instead.

 Assisted-by: OpenAI Codex

Added: 
    

Modified: 
    mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
    mlir/test/Dialect/Bufferization/canonicalize.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
index 45f5b3eaa5aea..80db89aa1b4bc 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizationOps.cpp
@@ -132,10 +132,11 @@ LogicalResult mlir::bufferization::foldToBufferToTensorPair(
   if (unrankedSrcType && rankedDestType)
     return failure();
 
-  // Unranked memref -> unranked memref cast
-  // Ranked memref -> unranked memref cast: No copy needed.
-  assert(memref::CastOp::areCastCompatible(srcType, destType) &&
-         "expected that types are cast compatible");
+  // Unranked/ranked memref -> unranked memref cast: No copy needed if the types
+  // are cast-compatible.
+  if (!memref::CastOp::areCastCompatible(srcType, destType))
+    return failure();
+
   rewriter.replaceOpWithNewOp<memref::CastOp>(toBuffer, destType,
                                               bufferToTensor.getBuffer());
   return success();

diff  --git a/mlir/test/Dialect/Bufferization/canonicalize.mlir b/mlir/test/Dialect/Bufferization/canonicalize.mlir
index df07511798b91..ab3d8df892bc1 100644
--- a/mlir/test/Dialect/Bufferization/canonicalize.mlir
+++ b/mlir/test/Dialect/Bufferization/canonicalize.mlir
@@ -50,6 +50,24 @@ func.func @canonicalize_buffer_cast_of_tensor_load_
diff erent_address_space(%arg0
 
 // -----
 
+// If unranked memrefs are not cast-compatible, don't fold them.
+// CHECK-LABEL: func @canonicalize_unranked_buffer_cast_of_tensor_load_
diff erent_address_space(
+//  CHECK-SAME:   %[[MEMREF:.*]]: memref<*xi64>)
+//  CHECK-SAME:     -> memref<*xi64, 1> {
+//  CHECK-NOT: memref.cast
+//      CHECK: %[[TENSOR:.*]] = bufferization.to_tensor %[[MEMREF]] : memref<*xi64> to tensor<*xi64>
+//      CHECK: %[[BUFFER:.*]] = bufferization.to_buffer %[[TENSOR]] : tensor<*xi64> to memref<*xi64, 1>
+//  CHECK-NOT: memref.cast
+//      CHECK: return %[[BUFFER]] : memref<*xi64, 1>
+func.func @canonicalize_unranked_buffer_cast_of_tensor_load_
diff erent_address_space(%arg0: memref<*xi64>)
+    -> memref<*xi64, 1> {
+  %0 = bufferization.to_tensor %arg0 : memref<*xi64> to tensor<*xi64>
+  %1 = bufferization.to_buffer %0 : tensor<*xi64> to memref<*xi64, 1>
+  return %1 : memref<*xi64, 1>
+}
+
+// -----
+
 // If the memrefs are definitely cast-compatible, canonicalize to
 //            cast.
 // CHECK-LABEL: func @canonicalize_buffer_cast_of_tensor_load(


        

From llvmlistbot at llvm.org  Sat Jul 18 04:58:51 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 04:58:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a5b6a7b.92d2873c.aa1ff.44a2@mx.google.com>


https://github.com/matthias-springer closed https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 18 04:59:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 04:59:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Avoid invalid
 memref.cast in to_buffer folding (PR #208973)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208973@github.com>
Message-ID: <6a5b6a8a.e624eb8d.4a8a7.b654@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@pzoln Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/208973

From llvmlistbot at llvm.org  Sat Jul 18 05:05:56 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Sat, 18 Jul 2026 05:05:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSVE] Fix comment inconsistency for
 `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC) (PR #210502)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210502@github.com>
Message-ID: <6a5b6c24.5a0412d2.3daf32.05ef@mx.google.com>


https://github.com/AGindinson approved this pull request.


https://github.com/llvm/llvm-project/pull/210502

From llvmlistbot at llvm.org  Sat Jul 18 05:16:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 05:16:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] c69896a - [mlir][interface] Use
 populateAndCompare cleanup compare (NFC) (#200156)
Message-ID: <6a5b6e91.d2cb4adc.ccd80.d6bf@mx.google.com>


Author: lonely eagle
Date: 2026-07-18T20:16:12+08:00
New Revision: c69896aa8733ceed6e8cc982eec64b97f277f926

URL: https://github.com/llvm/llvm-project/commit/c69896aa8733ceed6e8cc982eec64b97f277f926
DIFF: https://github.com/llvm/llvm-project/commit/c69896aa8733ceed6e8cc982eec64b97f277f926.diff

LOG: [mlir][interface] Use populateAndCompare cleanup compare (NFC) (#200156)

Added: 
    

Modified: 
    mlir/lib/Interfaces/ValueBoundsOpInterface.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Interfaces/ValueBoundsOpInterface.cpp b/mlir/lib/Interfaces/ValueBoundsOpInterface.cpp
index 02780048f7556..2ac5c8ffe46cb 100644
--- a/mlir/lib/Interfaces/ValueBoundsOpInterface.cpp
+++ b/mlir/lib/Interfaces/ValueBoundsOpInterface.cpp
@@ -848,9 +848,7 @@ bool ValueBoundsConstraintSet::compare(const Variable &lhs,
     return cstr.comparePos(lhsPos, cmp, rhsPos);
   };
   ValueBoundsConstraintSet cstr(lhs.getContext(), stopCondition);
-  lhsPos = cstr.populateConstraints(lhs.map, lhs.mapOperands);
-  rhsPos = cstr.populateConstraints(rhs.map, rhs.mapOperands);
-  return cstr.comparePos(lhsPos, cmp, rhsPos);
+  return cstr.populateAndCompare(lhs, cmp, rhs);
 }
 
 FailureOr<bool> ValueBoundsConstraintSet::strongCompare(const Variable &lhs,


        

From llvmlistbot at llvm.org  Sat Jul 18 05:16:19 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Sat, 18 Jul 2026 05:16:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][interface] Use populateAndCompare
 cleanup compare (NFC) (PR #200156)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200156@github.com>
Message-ID: <6a5b6e93.1905e68e.30d957.6804@mx.google.com>


https://github.com/linuxlonelyeagle closed https://github.com/llvm/llvm-project/pull/200156

From llvmlistbot at llvm.org  Sat Jul 18 05:21:45 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Sat, 18 Jul 2026 05:21:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][interface] Use populateAndCompare
 cleanup compare (NFC) (PR #200156)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/200156@github.com>
Message-ID: <6a5b6fd9.1e1ac41d.3969c3.d65e@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `amdgpu-clang-flang` running on `AMD-bb-w-05` while building `mlir` at step 3 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/234/builds/129

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 3 (annotate) failure: 'python ../llvm.src/offload/ci/openmp-offload-amdgpu-clang-flang.py ...' (failure)
...
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_from.c (820 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: libc/malloc_parallel.c (821 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_alloc_tofrom.c (822 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_alloc_present_tofrom.c (823 of 1275)
PASS: libomptarget :: amdgpu-amd-amdhsa :: sanitizer/free_wrong_ptr_kind.c (824 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_tofrom.c (825 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_to_from.c (826 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_exit_data_delete_from.c (827 of 1275)
PASS: libomptarget :: amdgpu-amd-amdhsa :: sanitizer/free_wrong_ptr_kind.cpp (828 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_both_pointer_pointee.c (829 of 1275)
FAIL: libomptarget :: amdgpu-amd-amdhsa :: tools/omp-kernel-replay/record-replay-empty-memory.cpp (830 of 1275)
******************** TEST 'libomptarget :: amdgpu-amd-amdhsa :: tools/omp-kernel-replay/record-replay-empty-memory.cpp' FAILED ********************
Exit Code: 125

Command Output (stdout):
--
# RUN: at line 2
/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/clang++ -fopenmp    -I /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test -I /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src  -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/amdgpu-amd-amdhsa -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/clang/24/lib/amdgpu-amd-amdhsa  -fopenmp-targets=amdgpu-amd-amdhsa /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp -o /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp -Xoffload-linker -lompdevice
# executed command: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/clang++ -fopenmp -I /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test -I /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/amdgpu-amd-amdhsa -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/clang/24/lib/amdgpu-amd-amdhsa -fopenmp-targets=amdgpu-amd-amdhsa /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp -o /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp -Xoffload-linker -lompdevice
# note: command had no output on stdout or stderr
# RUN: at line 3
rm -rf /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# executed command: rm -rf /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# note: command had no output on stdout or stderr
# RUN: at line 4
mkdir -p /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# executed command: mkdir -p /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# note: command had no output on stdout or stderr
# RUN: at line 5
env LIBOMPTARGET_RECORD=1 LIBOMPTARGET_RECORD_MEMSIZE=536870912 LIBOMPTARGET_RECORD_DIR=/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp 2>&1 | /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/FileCheck /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp
# executed command: env LIBOMPTARGET_RECORD=1 LIBOMPTARGET_RECORD_MEMSIZE=536870912 LIBOMPTARGET_RECORD_DIR=/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp
# note: command had no output on stdout or stderr
# executed command: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/FileCheck /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp
# note: command had no output on stdout or stderr
# RUN: at line 6
ls -t /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir/*.json | sed -n '1p' | grep . | xargs -I {} /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay --verify {}
# executed command: ls -t '/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir/*.json'
# note: command had no output on stdout or stderr
# executed command: sed -n 1p
# note: command had no output on stdout or stderr
# executed command: grep .
# note: command had no output on stdout or stderr
# executed command: xargs -I '{}' /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay --verify '{}'
# .---command stdout------------
# | [llvm-omp-kernel-replay] Replay time (1): 59175606 ns
# | [llvm-omp-kernel-replay] Replay done, device memory verified
# `-----------------------------
# .---command stderr------------
# | xargs: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay: terminated by signal 11
Step 9 (Add check check-offload) failure: Add check check-offload (failure)
...
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_from.c (820 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: libc/malloc_parallel.c (821 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_alloc_tofrom.c (822 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_alloc_present_tofrom.c (823 of 1275)
PASS: libomptarget :: amdgpu-amd-amdhsa :: sanitizer/free_wrong_ptr_kind.c (824 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_tofrom.c (825 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_data_alloc_to_from.c (826 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_ordering_tgt_exit_data_delete_from.c (827 of 1275)
PASS: libomptarget :: amdgpu-amd-amdhsa :: sanitizer/free_wrong_ptr_kind.cpp (828 of 1275)
PASS: libomptarget :: x86_64-unknown-linux-gnu :: mapping/map_both_pointer_pointee.c (829 of 1275)
FAIL: libomptarget :: amdgpu-amd-amdhsa :: tools/omp-kernel-replay/record-replay-empty-memory.cpp (830 of 1275)
******************** TEST 'libomptarget :: amdgpu-amd-amdhsa :: tools/omp-kernel-replay/record-replay-empty-memory.cpp' FAILED ********************
Exit Code: 125

Command Output (stdout):
--
# RUN: at line 2
/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/clang++ -fopenmp    -I /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test -I /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src  -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/amdgpu-amd-amdhsa -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/clang/24/lib/amdgpu-amd-amdhsa  -fopenmp-targets=amdgpu-amd-amdhsa /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp -o /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp -Xoffload-linker -lompdevice
# executed command: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/clang++ -fopenmp -I /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test -I /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -L /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/openmp/runtime/src -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/x86_64-unknown-linux-gnu -Wl,-rpath,/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/amdgpu-amd-amdhsa -Xoffload-linker -L/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./lib/clang/24/lib/amdgpu-amd-amdhsa -fopenmp-targets=amdgpu-amd-amdhsa /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp -o /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp -Xoffload-linker -lompdevice
# note: command had no output on stdout or stderr
# RUN: at line 3
rm -rf /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# executed command: rm -rf /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# note: command had no output on stdout or stderr
# RUN: at line 4
mkdir -p /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# executed command: mkdir -p /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir
# note: command had no output on stdout or stderr
# RUN: at line 5
env LIBOMPTARGET_RECORD=1 LIBOMPTARGET_RECORD_MEMSIZE=536870912 LIBOMPTARGET_RECORD_DIR=/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp 2>&1 | /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/FileCheck /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp
# executed command: env LIBOMPTARGET_RECORD=1 LIBOMPTARGET_RECORD_MEMSIZE=536870912 LIBOMPTARGET_RECORD_DIR=/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp
# note: command had no output on stdout or stderr
# executed command: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/FileCheck /home/botworker/bbot/amdgpu-clang-flang/llvm.src/offload/test/tools/omp-kernel-replay/record-replay-empty-memory.cpp
# note: command had no output on stdout or stderr
# RUN: at line 6
ls -t /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir/*.json | sed -n '1p' | grep . | xargs -I {} /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay --verify {}
# executed command: ls -t '/home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/runtimes/runtimes-bins/offload/test/amdgpu-amd-amdhsa/tools/omp-kernel-replay/Output/record-replay-empty-memory.cpp.tmp.testdir/*.json'
# note: command had no output on stdout or stderr
# executed command: sed -n 1p
# note: command had no output on stdout or stderr
# executed command: grep .
# note: command had no output on stdout or stderr
# executed command: xargs -I '{}' /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay --verify '{}'
# .---command stdout------------
# | [llvm-omp-kernel-replay] Replay time (1): 59175606 ns
# | [llvm-omp-kernel-replay] Replay done, device memory verified
# `-----------------------------
# .---command stderr------------
# | xargs: /home/botworker/bbot/amdgpu-clang-flang/build/llvm.build/./bin/llvm-omp-kernel-replay: terminated by signal 11

```

</details>

https://github.com/llvm/llvm-project/pull/200156

From llvmlistbot at llvm.org  Sat Jul 18 05:48:45 2026
From: llvmlistbot at llvm.org (Yingwei Zheng)
Date: Sat, 18 Jul 2026 05:48:45 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5b762d.743400b0.23a7b2.428b@mx.google.com>


https://github.com/dtcxzyw approved this pull request.

LGTM.

https://github.com/llvm/llvm-project/pull/209465

From llvmlistbot at llvm.org  Sat Jul 18 05:55:48 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Sat, 18 Jul 2026 05:55:48 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5b77d4.d2cb4adc.ccd80.f519@mx.google.com>


https://github.com/artagnon updated https://github.com/llvm/llvm-project/pull/209465

>From 41663d18035b9ff1b524b47c2834f40403a491b3 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 12:31:58 +0100
Subject: [PATCH 1/3] [IR] Make semantics of strictfp consistent

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that strictfp propagates from
callee to caller, the general description of strictfp does not specify
this. Refine its semantics and make it so, eliminating the
inconsistency.
---
 clang/lib/CodeGen/TargetInfo.cpp              |  3 +
 .../cl20-device-side-enqueue-attributes.cl    |  4 +-
 llvm/docs/LangRef.md                          |  7 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 50 +++++++-------
 22 files changed, 146 insertions(+), 136 deletions(-)

diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..af90be4b1a47b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,9 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. The
+    attribute propagates from callee to caller, and all function definitions
+    that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26587,9 +26589,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..5da8fac2fc914 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "strictfp attribute must propagate from the call site to the caller",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)

>From 0e122c4a0d700fd30ae0ba51713b43021d928e53 Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Tue, 14 Jul 2026 18:42:48 +0100
Subject: [PATCH 2/3] [LangRef/Verfier] Drop bad propagation terminology

---
 llvm/docs/LangRef.md     | 5 ++---
 llvm/lib/IR/Verifier.cpp | 2 +-
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index af90be4b1a47b..9526cd020d724 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,9 +2794,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap. The
-    attribute propagates from callee to caller, and all function definitions
-    that contain strictfp calls must be marked strictfp.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 5da8fac2fc914..d315372c1371c 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3891,7 +3891,7 @@ void Verifier::visitCallBase(CallBase &Call) {
 
   Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
             Call.getFunction()->isStrictFP(),
-        "strictfp attribute must propagate from the call site to the caller",
+        "call site marked strictfp without caller function marked strictfp",
         Call);
 
   // Verify call attributes.

>From b90f1d4d7e7707ca9408196cbb8a9d2aafbbb0ec Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Fri, 17 Jul 2026 11:01:45 +0100
Subject: [PATCH 3/3] [IR] Add Verifier test

---
 llvm/unittests/IR/VerifierTest.cpp | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/llvm/unittests/IR/VerifierTest.cpp b/llvm/unittests/IR/VerifierTest.cpp
index 48d461cab330a..f6a89cde0901b 100644
--- a/llvm/unittests/IR/VerifierTest.cpp
+++ b/llvm/unittests/IR/VerifierTest.cpp
@@ -699,4 +699,28 @@ TEST(VerifierTest, IntrinsicRetInvalidStruct) {
   }
 }
 
+TEST(VerifierTest, InvalidStrictFPAttribute) {
+  LLVMContext Ctx;
+  Module M("M", Ctx);
+  FunctionType *FuncTy =
+      FunctionType::get(Type::getVoidTy(Ctx), /*isVarArg=*/false);
+  Function *F =
+      Function::Create(FuncTy, Function::ExternalLinkage, "strictfp_test", M);
+  BasicBlock *Entry = BasicBlock::Create(Ctx, "entry", F);
+  Type *FloatTy = Type::getFloatTy(Ctx);
+
+  Function *CosF =
+      Intrinsic::getOrInsertDeclaration(&M, Intrinsic::cos, FloatTy);
+  CallInst *CI = CallInst::Create(CosF, ConstantFP::getNullValue(FloatTy),
+                                  "strictfp_call", Entry);
+  CI->addFnAttr(Attribute::StrictFP);
+  ReturnInst::Create(Ctx, Entry);
+
+  std::string Error;
+  raw_string_ostream ErrorOS(Error);
+  EXPECT_TRUE(verifyModule(M, &ErrorOS));
+  EXPECT_TRUE(StringRef(Error).starts_with(
+      "call site marked strictfp without caller function marked strictfp"))
+      << Error;
+}
 } // end anonymous namespace


From llvmlistbot at llvm.org  Sat Jul 18 05:55:57 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sat, 18 Jul 2026 05:55:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5b77dd.1728eaf3.2a2f6.c577@mx.google.com>


secona wrote:

> I think there's a simpler fix: deactivate folding the `GreedyRewriteConfig`. Can you give that a try? Recomputing an analysis in a rewrite pattern could be expensive.

Disabling folding seems to fix the crash issue. Some tests are failing, though. I'll see if it introduces any regressions and push here if it looks good.

https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sat Jul 18 05:56:11 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Sat, 18 Jul 2026 05:56:11 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5b77eb.eb165e5c.4506c.0eee@mx.google.com>


https://github.com/artagnon auto_merge_enabled https://github.com/llvm/llvm-project/pull/209465

From llvmlistbot at llvm.org  Sat Jul 18 06:28:54 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 06:28:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b7f96.e55dc341.1c6a0.e461@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 06:29:09 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 06:29:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b7fa5.449bfb44.7b862.3171@mx.google.com>


https://github.com/FedericoBruzzone deleted https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 06:29:21 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 06:29:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b7fb1.743400b0.23a7b2.d852@mx.google.com>


https://github.com/FedericoBruzzone deleted https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 06:31:27 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 06:31:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b802f.743400b0.23a7b2.28a0@mx.google.com>


https://github.com/FedericoBruzzone deleted https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 06:31:35 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 18 Jul 2026 06:31:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5b8037.c5504292.78ef3.e585@mx.google.com>


https://github.com/FedericoBruzzone deleted https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Sat Jul 18 06:49:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 06:49:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] af0c933 - [IR] Make semantics of strictfp
	consistent (#209465)
Message-ID: <6a5b847a.3cce8983.3cf542.d49d@mx.google.com>


Author: Ramkumar Ramachandra
Date: 2026-07-18T13:49:40Z
New Revision: af0c933eda31b6beb436026a22121ec637f3cb53

URL: https://github.com/llvm/llvm-project/commit/af0c933eda31b6beb436026a22121ec637f3cb53
DIFF: https://github.com/llvm/llvm-project/commit/af0c933eda31b6beb436026a22121ec637f3cb53.diff

LOG: [IR] Make semantics of strictfp consistent (#209465)

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that a function with strictfp
calls must be marked with strictfp, the general description of strictfp
does not specify this. Refine its semantics and make it so, eliminating
the inconsistency.

Added: 
    

Modified: 
    clang/lib/CodeGen/TargetInfo.cpp
    clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
    llvm/docs/LangRef.md
    llvm/include/llvm/IR/Function.h
    llvm/lib/IR/Verifier.cpp
    llvm/test/CodeGen/AArch64/is_fpclass.ll
    llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
    llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
    llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
    llvm/test/CodeGen/PowerPC/milicode32.ll
    llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
    llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
    llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
    llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
    llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
    llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
    llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
    llvm/test/Transforms/InstCombine/erf.ll
    llvm/test/Transforms/InstCombine/fdim.ll
    llvm/test/Transforms/InstSimplify/disable_folding.ll
    llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
    llvm/unittests/IR/VerifierTest.cpp
    mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir

Removed: 
    


################################################################################
diff  --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;

diff  --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.

diff  --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..9526cd020d724 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2794,7 +2794,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26587,9 +26588,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:

diff  --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||

diff  --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..d315372c1371c 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3889,6 +3889,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "call site marked strictfp without caller function marked strictfp",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 

diff  --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0

diff  --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup

diff  --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0

diff  --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0

diff  --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0

diff  --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112

diff  --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16

diff  --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16

diff  --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16

diff  --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16

diff  --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,

diff  --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_
diff _attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_
diff _attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_
diff _attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_
diff _attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_
diff _attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_
diff _attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_
diff _attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_
diff _attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_
diff _attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_
diff _attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_
diff _attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_
diff _attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_
diff _attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_
diff _attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @
diff _parent_combine_
diff _attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @
diff _parent_combine_
diff _attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @
diff _parent_combine_
diff _attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @
diff _parent_combine_
diff _attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_
diff _attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_
diff _attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_
diff _attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_
diff _attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_
diff _attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_
diff _attrs_fail(i8 %x, i8 %y) {
 define i8 @
diff _parent_combine_
diff _attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @
diff _parent_combine_
diff _attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @
diff _parent_combine_
diff _attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @
diff _parent_combine_
diff _attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @
diff _parent_combine_
diff _attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.

diff  --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp

diff  --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp

diff  --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1

diff  --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.

diff  --git a/llvm/unittests/IR/VerifierTest.cpp b/llvm/unittests/IR/VerifierTest.cpp
index 48d461cab330a..f6a89cde0901b 100644
--- a/llvm/unittests/IR/VerifierTest.cpp
+++ b/llvm/unittests/IR/VerifierTest.cpp
@@ -699,4 +699,28 @@ TEST(VerifierTest, IntrinsicRetInvalidStruct) {
   }
 }
 
+TEST(VerifierTest, InvalidStrictFPAttribute) {
+  LLVMContext Ctx;
+  Module M("M", Ctx);
+  FunctionType *FuncTy =
+      FunctionType::get(Type::getVoidTy(Ctx), /*isVarArg=*/false);
+  Function *F =
+      Function::Create(FuncTy, Function::ExternalLinkage, "strictfp_test", M);
+  BasicBlock *Entry = BasicBlock::Create(Ctx, "entry", F);
+  Type *FloatTy = Type::getFloatTy(Ctx);
+
+  Function *CosF =
+      Intrinsic::getOrInsertDeclaration(&M, Intrinsic::cos, FloatTy);
+  CallInst *CI = CallInst::Create(CosF, ConstantFP::getNullValue(FloatTy),
+                                  "strictfp_call", Entry);
+  CI->addFnAttr(Attribute::StrictFP);
+  ReturnInst::Create(Ctx, Entry);
+
+  std::string Error;
+  raw_string_ostream ErrorOS(Error);
+  EXPECT_TRUE(verifyModule(M, &ErrorOS));
+  EXPECT_TRUE(StringRef(Error).starts_with(
+      "call site marked strictfp without caller function marked strictfp"))
+      << Error;
+}
 } // end anonymous namespace

diff  --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)


        

From llvmlistbot at llvm.org  Sat Jul 18 06:49:48 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Sat, 18 Jul 2026 06:49:48 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
	strictfp consistent (PR #209465)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209465@github.com>
Message-ID: <6a5b847c.1728eaf3.2a2f6.e6c7@mx.google.com>


https://github.com/artagnon closed https://github.com/llvm/llvm-project/pull/209465

From llvmlistbot at llvm.org  Sat Jul 18 10:45:28 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 10:45:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU (PR #210527)
Message-ID: <llvm.org/llvm/llvm-project/pull/210527@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/210527

Extend the vector.transfer_read/transfer_write lowerings so they can produce N-D xegpu.load_nd/store_nd, not just 1D/2D, and relax the out-of-bounds handling to match load_nd's implicit-zero padding. 
Restructure both patterns as "block first, then scatter as  fallback. 

>From 0d7ff5b87542e7ecee78d6d33e6696b9338212b9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 17:41:56 +0000
Subject: [PATCH] [mlir][xegpu] Support N-D block transfers in VectorToXeGPU

Extend the vector.transfer_read/transfer_write lowerings so they can
produce N-D xegpu.load_nd/store_nd, not just 1D/2D, and relax the
out-of-bounds handling to match load_nd's implicit-zero padding.

Changes:
- Allow high-dim (rank > 2) vectors on the nd block path. The rank
  restriction in storeLoadPreconditions is gated behind a new
  allowHighDimVector flag so plain vector.load/store keep their 1D/2D
  limit while transfers accept any non-0D rank.
- Accept an out-of-bounds read whose padding is a zero constant or a
  ub.poison value (isZeroOrPoisonPadding): load_nd zero-fills
  out-of-bounds elements, and poison means those elements are
  don't-care, so both match its semantics.
- Restructure both patterns to attempt the nd block op first via a
  single canLowerToLoadNd/canLowerToStoreNd predicate, then fall back
  to the scattered path. The block path only realizes a minor-identity
  map (reads also handle an innermost-two-dims transpose via a trailing
  vector.transpose); any other permutation is routed to the scattered
  path, which permutes strides explicitly. This fixes silently dropping
  the permutation on high-dim transposed transfers.

Tests updated for the new high-dim block lowering, plus new
load_high_dim_transposed / store_high_dim_transposed coverage.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 268 ++++++++++--------
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 169 +++++++----
 .../transfer-write-to-xegpu.mlir              | 225 +++++++--------
 3 files changed, 380 insertions(+), 282 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..306ce6f1e8008 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -15,6 +15,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/UB/IR/UBOps.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
@@ -49,14 +50,53 @@ static bool isZeroConstant(Value val) {
       .Default(false);
 }
 
+// Return true if the transfer padding value is compatible with the implicit
+// padding of an nd block load. LoadNdOp fills out-of-bounds elements with zero,
+// so a zero constant matches its semantics exactly. A poison padding means the
+// out-of-bounds elements are "don't care", so any implicit padding (including
+// zero) is also acceptable.
+static bool isZeroOrPoisonPadding(Value val) {
+  return isZeroConstant(val) || val.getDefiningOp<ub::PoisonOp>();
+}
+
+// Return true if the permutation map keeps every dimension in place except the
+// innermost two, which are swapped, e.g.:
+//   (d0, d1) -> (d1, d0)
+//   (d0, d1, d2) -> (d2, d1)
+//   (d0, d1, d2, d3) -> (d0, d1, d3, d2)
+// This is the only non-identity permutation an nd block load can realize (by
+// loading the untransposed block and applying a trailing vector.transpose).
+static bool isInnermostTwoDimsTransposed(AffineMap map) {
+  unsigned numResults = map.getNumResults();
+  if (numResults < 2)
+    return false;
+  MLIRContext *ctx = map.getContext();
+  unsigned numInputs = map.getNumInputs();
+  // All but the innermost two results must match the minor-identity map.
+  for (unsigned i = 0; i + 2 < numResults; ++i)
+    if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
+      return false;
+  // The innermost two results must be the last two input dims, swapped.
+  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+         map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
+}
+
 static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
                                             Operation *op, VectorType vecTy,
-                                            MemRefType memTy) {
+                                            MemRefType memTy,
+                                            bool allowHighDimVector = false) {
   // Validate only vector as the basic vector store and load ops guarantee
   // XeGPU-compatible memref source.
   unsigned vecRank = vecTy.getRank();
-  if (!(vecRank == 1 || vecRank == 2))
+  // nd block loads/stores support N-D block transfers, so callers that lower to
+  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
+  // vector.load/store lowering keeps the 1D/2D restriction.
+  if (allowHighDimVector) {
+    if (vecRank == 0)
+      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
+  } else if (!(vecRank == 1 || vecRank == 2)) {
     return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
+  }
 
   if (!vecTy.getElementType().isIntOrFloat())
     return rewriter.notifyMatchFailure(
@@ -594,92 +634,78 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(readOp);
-    // Lower to scattered load Op if the target HW doesn't have 2d block load
-    // support and the load is not from shared memory.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        readOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (isOutOfBounds)
-        return failure();
-      return lowerToScatteredLoadOp(readOp, rewriter);
-    }
-
-    // Handle the 1D non-SLM case using load.gather.
-    if (loadedVecTy.getRank() == 1 && !isOutOfBounds)
-      return lowerToScatteredLoadOp(readOp, rewriter);
-
-    // Perform common data transfer checks.
-    // TODO: Maybe too strict for SLM case.
-    if (failed(
-            storeLoadPreconditions(rewriter, readOp, loadedVecTy, readMemTy)))
-      return failure();
-
-    if (isOutOfBounds && !isZeroConstant(readOp.getPadding()))
-      return rewriter.notifyMatchFailure(
-          readOp, "Unsupported non-zero padded out-of-bounds read");
+    bool hasBlockLoadSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // An nd block load can realize a minor-identity map directly, or an
+    // innermost-two-dims transpose via a trailing vector.transpose. Any other
+    // permutation (e.g. a mid-vector transpose of a high-dim load) is left to
+    // the scattered path, which permutes strides explicitly.
     AffineMap readMap = readOp.getPermutationMap();
-    // Check if this is a transpose: the map must have exactly 2 results,
-    // and those 2 results must be the last 2 input dimensions interchanged.
-    // Examples:
-    //   (d0, d1) -> (d1, d0)      // transpose
-    //   (d0, d1) -> (d0, d1)      // not a transpose
-    //   (d0, d1, d2) -> (d2, d1)  // transpose (last 2 dims swapped)
-    bool isTransposeLoad = false;
-    if (readMap.getNumResults() == 2) {
-      auto results = readMap.getResults();
-      unsigned numInputs = readMap.getNumInputs();
-      if (numInputs >= 2) {
-        auto lastDim = getAffineDimExpr(numInputs - 1, readMap.getContext());
-        auto secondLastDim =
-            getAffineDimExpr(numInputs - 2, readMap.getContext());
-        isTransposeLoad =
-            (results[0] == lastDim && results[1] == secondLastDim);
+    bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
+
+    // Prefer an nd block load. It requires HW block-load support, a >1D vector
+    // of a scalar element type backed by a scalar-element memref, and a map the
+    // block load can realize. Out-of-bounds reads are allowed as long as the
+    // padding matches load_nd's implicit zero padding.
+    bool canLowerToLoadNd =
+        hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
+        (readMap.isMinorIdentity() || isTransposeLoad) &&
+        loadedVecTy.getElementType().isIntOrFloat() &&
+        readMemTy.getElementType().isIntOrFloat() &&
+        (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
+
+    if (canLowerToLoadNd) {
+      auto elementType = loadedVecTy.getElementType();
+
+      SmallVector<int64_t> descShape(loadedVecTy.getShape());
+      if (isTransposeLoad) {
+        // If load is transposed, simply swap the last two dimensions of the
+        // loaded vector type to get the descriptor shape.
+        size_t rank = descShape.size();
+        assert(rank >= 2 && "Transpose requires at least 2 dimensions");
+        std::swap(descShape[rank - 1], descShape[rank - 2]);
+        loadedVecTy = VectorType::get(descShape, elementType);
       }
+      auto descType = xegpu::TensorDescType::get(
+          descShape, elementType, /*array_length=*/1,
+          /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, readOp.getBase(),
+          getAsOpFoldResult(readOp.getIndices()), loadedVecTy.getRank());
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      Operation *loadedOp =
+          xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
+                                  /*packed=*/nullptr, /*transpose=*/nullptr,
+                                  /*l1_hint=*/hint,
+                                  /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                  /*layout=*/nullptr);
+      if (isTransposeLoad) {
+        // Undo the innermost-two-dims swap with a trailing vector.transpose:
+        // keep the leading dimensions in place and interchange only the last
+        // two.
+        int64_t rank = loadedVecTy.getRank();
+        SmallVector<int64_t> perm(llvm::to_vector(llvm::seq<int64_t>(0, rank)));
+        std::swap(perm[rank - 1], perm[rank - 2]);
+        loadedOp = vector::TransposeOp::create(rewriter, loc,
+                                               loadedOp->getResult(0), perm);
+      }
+      rewriter.replaceOp(readOp, loadedOp);
+      return success();
     }
-    auto elementType = loadedVecTy.getElementType();
-
-    SmallVector<int64_t> descShape(loadedVecTy.getShape());
-    if (isTransposeLoad) {
-      // If load is transposed, simply swap the last two dimensions of the
-      // loaded vector type to get the descriptor shape.
-      size_t rank = descShape.size();
-      assert(rank >= 2 && "Transpose requires at least 2 dimensions");
-      std::swap(descShape[rank - 1], descShape[rank - 2]);
-      loadedVecTy = VectorType::get(descShape, elementType);
-    }
-    auto descType = xegpu::TensorDescType::get(
-        descShape, elementType, /*array_length=*/1,
-        /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, readOp.getBase(), getAsOpFoldResult(readOp.getIndices()),
-        loadedVecTy.getRank());
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    Operation *loadedOp =
-        xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
-                                /*packed=*/nullptr, /*transpose=*/nullptr,
-                                /*l1_hint=*/hint,
-                                /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                /*layout=*/nullptr);
-    if (isTransposeLoad) {
-      // Transposing the loaded vector with a separate vector.transpose
-      // operation
-      auto range = llvm::seq<int64_t>(0, readMap.getResults().size());
-      SmallVector<int64_t> perm(
-          range.rbegin(), range.rend()); // reverse the range for transpose
-      loadedOp = vector::TransposeOp::create(rewriter, loc,
-                                             loadedOp->getResult(0), perm);
-    }
-    rewriter.replaceOp(readOp, loadedOp);
 
-    return success();
+    // Fall back to a scattered load. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (isOutOfBounds)
+      return failure();
+    return lowerToScatteredLoadOp(readOp, rewriter);
   }
 };
 
@@ -727,47 +753,51 @@ struct TransferWriteLowering
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(writeOp);
-    // Lower to scattered store Op if the target HW doesn't have 2d block
-    // store support and the memref is not SLM.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        writeOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (writeOp.hasOutOfBoundsDim())
-        return failure();
-      return lowerToScatteredStoreOp(writeOp, rewriter);
-    }
-
-    if (failed(storeLoadPreconditions(rewriter, writeOp, vecTy, writeMemTy)))
-      return failure();
+    bool hasBlockStoreSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // Prefer an nd block store. It requires HW block-store support, a >1D
+    // vector of a scalar element type backed by a scalar-element memref, and a
+    // minor-identity map (block stores have no transpose support). Out-of-bounds
+    // writes are handled by the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    if (!map.isMinorIdentity())
-      return rewriter.notifyMatchFailure(writeOp, "Expects identity map");
-
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, writeOp.getBase(),
-        getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
-
-    auto descType = xegpu::TensorDescType::get(
-        vecTy.getShape(), vecTy.getElementType(),
-        /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
-        xegpu::MemorySpace::Global);
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    auto storeOp = xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(),
-                                            ndDesc, indices,
-                                            /*l1_hint=*/hint,
-                                            /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                            /*layout=*/nullptr);
-    rewriter.replaceOp(writeOp, storeOp);
+    bool canLowerToStoreNd =
+        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
+        vecTy.getElementType().isIntOrFloat() &&
+        writeMemTy.getElementType().isIntOrFloat();
+
+    if (canLowerToStoreNd) {
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, writeOp.getBase(),
+          getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
+
+      auto descType = xegpu::TensorDescType::get(
+          vecTy.getShape(), vecTy.getElementType(),
+          /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
+          xegpu::MemorySpace::Global);
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      auto storeOp =
+          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
+                                   indices,
+                                   /*l1_hint=*/hint,
+                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                   /*layout=*/nullptr);
+      rewriter.replaceOp(writeOp, storeOp);
+      return success();
+    }
 
-    return success();
+    // Fall back to a scattered store. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (writeOp.hasOutOfBoundsDim())
+      return failure();
+    return lowerToScatteredStoreOp(writeOp, rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index 3354868cd6cb9..ed495729f9add 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -182,6 +182,33 @@ gpu.func @load_transpose_3d_memref(%source: memref<32x64x128xf32>,
 
 }
 
+// -----
+// A high-dim load whose innermost two dims are transposed lowers to an
+// (untransposed) nd block load followed by a vector.transpose of the last two
+// dims.
+gpu.module @xevm_module {
+gpu.func @load_high_dim_transposed(%source: memref<2x2x64x128xf16>,
+    %offset: index) -> vector<1x1x64x128xf16> {
+  %c0 = arith.constant 0.0 : f16
+  %0 = vector.transfer_read %source[%offset, %offset, %offset, %offset], %c0
+    {permutation_map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3, d2)>,
+    in_bounds = [true, true, true, true]}
+    : memref<2x2x64x128xf16>, vector<1x1x64x128xf16>
+  gpu.return %0 : vector<1x1x64x128xf16>
+}
+
+// LOAD-ND-LABEL:  @load_high_dim_transposed(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x64x128xf16>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x64x128xf16>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<1x1x128x64xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<1x1x128x64xf16>
+// LOAD-ND:        vector.transpose %[[VEC]], [0, 1, 3, 2] : vector<1x1x128x64xf16> to vector<1x1x64x128xf16>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_transposed(
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load {{.*}} : i64, vector<1x1x64x128xindex>, vector<1x1x64x128xi1> -> vector<1x1x64x128xf16>
+}
+
 // -----
 gpu.module @xevm_module {
 gpu.func @load_dynamic_source(%source: memref<?x?x?xf32>,
@@ -275,21 +302,31 @@ gpu.func @load_dynamic_source3(%source: memref<?x?x?x?x?xf32>,
   gpu.return %0 : vector<2x4x8x16xf32>
 }
 
-// CHECK-LABEL:  @load_dynamic_source3(
-// CHECK-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
-// CHECK:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
-// CHECK-COUNT4: vector.step
-// CHECK-COUNT3: vector.broadcast
-// CHECK-COUNT4: vector.shape_cast
-// CHECK-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
-// CHECK-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
-// CHECK:        return %[[VEC]]
+// LOAD-ND-LABEL:  @load_dynamic_source3(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFF1:.*]], %[[SIZES:.*]]:4, %[[STRIDES:.*]]:4 = memref.extract_strided_metadata %[[SUBVIEW]]
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x4x8x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x4x8x16xf32>
+// LOAD-ND:        return %[[VEC]]
+
+// LOAD-GATHER-LABEL:  @load_dynamic_source3(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
+// LOAD-GATHER:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
+// LOAD-GATHER-COUNT4: vector.step
+// LOAD-GATHER-COUNT3: vector.broadcast
+// LOAD-GATHER-COUNT4: vector.shape_cast
+// LOAD-GATHER-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
+// LOAD-GATHER:        return %[[VEC]]
 }
 
 // -----
@@ -302,21 +339,27 @@ gpu.func @load_high_dim_vector(%source: memref<16x32x64xf32>,
   gpu.return %0 : vector<8x16x32xf32>
 }
 
-// CHECK-LABEL:  @load_high_dim_vector(
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
+// LOAD-ND-LABEL:  @load_high_dim_vector(
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %arg0 : memref<16x32x64xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<8x16x32xf32>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_vector(
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// LOAD-GATHER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// LOAD-GATHER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// LOAD-GATHER:        %[[C2048:.+]] = arith.constant 2048 : index
+// LOAD-GATHER:        %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-COUNT3: vector.step
+// LOAD-GATHER-COUNT3: vector.shape_cast
+// LOAD-GATHER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
 
 }
 
@@ -330,18 +373,25 @@ gpu.func @load_8D_vector(%source: memref<2x2x2x2x2x2x2x2xf32>,
   gpu.return %0 : vector<2x2x2x2x2x2x2x2xf32>
 }
 
-// CHECK-LABEL:  @load_8D_vector(
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-LABEL:  @load_8D_vector(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x2x2x2x2x2x2x2xf32>
+
+// LOAD-GATHER-LABEL:  @load_8D_vector(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// LOAD-GATHER-COUNT8: vector.step
+// LOAD-GATHER-COUNT7: vector.shape_cast
+// LOAD-GATHER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
 
 }
 
@@ -728,23 +778,38 @@ gpu.func @transpose_1x1024x24x64(
   gpu.return
 }
 
-// CHECK-LABEL: @transpose_1x1024x24x64
-// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
-// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
-// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+// The vector.transpose is folded into the transfer_read via
+// CombineTransferReadOpTranspose, giving the read a mid-vector permutation map
+// (d0, d1, d2, d3) -> (d0, d2, d1, d3). An nd block load can only realize an
+// innermost-two-dims transpose, so the read falls back to the scattered path
+// while the identity-map write still lowers to store_nd.
+// LOAD-ND-LABEL: @transpose_1x1024x24x64
+// LOAD-ND-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-ND-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-ND:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-ND:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-ND:     %[[VEC:.+]] = xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-ND:     %[[WDESC:.+]] = xegpu.create_nd_tdesc %arg1 : memref<1x24x1024x64xf16>
+// LOAD-ND-SAME:  -> !xegpu.tensor_desc<1x1x16x8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:     xegpu.store_nd %[[VEC]], %[[WDESC]]
+
+// LOAD-GATHER-LABEL: @transpose_1x1024x24x64
+// LOAD-GATHER-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-GATHER-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-DAG: %[[C65536:.+]] = arith.constant 65536 : index
 
 // Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
 // Scalar base offset: seq_off * 1536 (original dim1 stride),
 //                    block_id_x * 64  (original dim2 stride).
-// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
-// CHECK:     arith.muli %block_id_x, %[[C64]] : index
-// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-GATHER:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
 
 // Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
 // Scalar base offset: block_id_x * 65536 (original dim1 stride),
 //                    seq_off * 64        (original dim2 stride).
-// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
-// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
-// CHECK:     xegpu.store {{.*}}
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C65536]] : index
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C64]] : index
+// LOAD-GATHER:     xegpu.store {{.*}}
 
 }
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index 45bda46f738b2..ad7332097cc8f 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -11,36 +11,20 @@ gpu.func @store_1D_vector(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector(
-// STORE-ND-SAME:  %[[VEC:.+]]: vector<8xf32>,
-// STORE-ND-SAME:  %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-ND-SAME:  %[[OFFSET:.+]]: index
-// STORE-ND:       %[[ELEM_BYTES:.+]] = arith.constant 4 : index
-// STORE-ND:       %[[COLLAPSED:.+]] = memref.subview %[[SRC]][%[[OFFSET]], %[[OFFSET]], 0]
-// STORE-ND:       %[[BASE_BUFFER:.+]], %[[OFFSET1:.+]], %[[SIZES:.+]], %[[STRIDES:.+]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND-SAME:    : memref<32xf32, strided<[1], offset: ?>> -> memref<f32>, index, index, index
-// STORE-ND:       %[[INTPTR:.+]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND-SAME:    : memref<f32> -> index
-// STORE-ND:       %[[MUL:.+]] = arith.muli %[[OFFSET1]], %[[ELEM_BYTES]] : index
-// STORE-ND:       %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:       %[[I64PTR:.+]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:       %[[DESC:.+]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [32],
-// STORE-ND-SAME:                   strides : [1] : i64  -> !xegpu.tensor_desc<8xf32,
-// STORE-ND-SAME:    boundary_check = false
-// STORE-ND:       xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFFSET]]] : vector<8xf32>
-
-// STORE-SCATTER-LABEL:  @store_1D_vector(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-SCATTER-DAG:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER-DAG:        %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: arith.muli {{.*}} : index
-// STORE-SCATTER-COUNT2: arith.addi {{.*}} : index
-// STORE-SCATTER-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:       xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_1D_vector(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
+// CHECK-DAG:    %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK-DAG:    %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: arith.muli {{.*}} : index
+// CHECK-COUNT2: arith.addi {{.*}} : index
+// CHECK-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----
@@ -150,7 +134,7 @@ gpu.func @store_out_of_bounds(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
+gpu.func @store_transposed(%vec: vector<8x16xf32>,
     %source: memref<32x64xf32>, %offset: index) {
   vector.transfer_write %vec, %source[%offset, %offset]
     {permutation_map = affine_map<(d0, d1) -> (d1, d0)>,
@@ -159,22 +143,38 @@ gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @no_store_transposed(
-// STORE-ND:       vector.transfer_write
+// An nd block store cannot transpose, so a transposed write lowers through the
+// scattered path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_transposed(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
+// CHECK-SAME:   %[[OFFSET:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
+// CHECK-COUNT2: %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: vector.shape_cast {{.*}}
+// CHECK-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+}
 
-// STORE-SCATTER-LABEL:  @no_store_transposed(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
-// STORE-SCATTER-SAME:   %[[OFFSET:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
-// STORE-SCATTER-COUNT2: %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: vector.shape_cast {{.*}}
-// STORE-SCATTER-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+// -----
+// A high-dim transposed write cannot use an nd block store (no transpose
+// support), so it lowers through the scattered path on both runs.
+gpu.module @xevm_module {
+gpu.func @store_high_dim_transposed(%vec: vector<8x16x4xf32>,
+    %source: memref<16x32x64xf32>, %offset: index) {
+  vector.transfer_write %vec, %source[%offset, %offset, %offset]
+    {permutation_map = affine_map<(d0, d1, d2) -> (d0, d2, d1)>,
+    in_bounds = [true, true, true]}
+    : vector<8x16x4xf32>, memref<16x32x64xf32>
+  gpu.return
+}
+
+// CHECK-LABEL:  @store_high_dim_transposed(
+// CHECK:        xegpu.store {{.*}} : vector<8x16x4xf32>, i64, vector<8x16x4xindex>, vector<8x16x4xi1>
 }
 
 // -----
@@ -187,23 +187,31 @@ gpu.func @store_high_dim_vector(%vec: vector<8x16x32xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_high_dim_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
+// STORE-ND-LABEL:  @store_high_dim_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<16x32x64xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<8x16x32xf32>, !xegpu.tensor_desc<8x16x32xf32
+
+// STORE-SCATTER-LABEL:  @store_high_dim_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// STORE-SCATTER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// STORE-SCATTER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// STORE-SCATTER:        %[[C2048:.+]] = arith.constant 2048 : index
+// STORE-SCATTER:        %[[C64:.+]] = arith.constant 64 : index
+// STORE-SCATTER-COUNT3: vector.step
+// STORE-SCATTER-COUNT3: vector.shape_cast
+// STORE-SCATTER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
 }
 
 // -----
@@ -216,19 +224,27 @@ gpu.func @store_8D_vector(%vec: vector<2x2x2x2x2x2x2x2xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_8D_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
+// STORE-ND-LABEL:  @store_8D_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<2x2x2x2x2x2x2x2xf32>, !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32
+
+// STORE-SCATTER-LABEL:  @store_8D_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// STORE-SCATTER-COUNT8: vector.step
+// STORE-SCATTER-COUNT7: vector.shape_cast
+// STORE-SCATTER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
 }
 
 // -----
@@ -330,41 +346,28 @@ gpu.func @store_to_subview(%vec: vector<8xf16>,
       : vector<8xf16>, memref<256x256xf16, strided<[4096, 1], offset: ?>>
   gpu.return
 }
-// STORE-ND-LABEL:  @store_to_subview(
-// STORE-ND-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-ND-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-ND-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-ND:        %[[ELEM_BYTES:.+]] = arith.constant 2 : index
-// STORE-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1] : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SUBVIEW]][%[[OFF2]], 0]
-// STORE-ND:        %[[BASE_BUFFER:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND:        %[[MUL:.+]] = arith.muli %[[OFFSET]], %[[ELEM_BYTES]] : index
-// STORE-ND:        %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:        %[[DESC:.*]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [256], strides : [1] : i64 ->
-// STORE-ND-SAME:                    !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
-// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFF2]]] : vector<8xf16>
-
-// STORE-SCATTER-LABEL:  @store_to_subview(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-SCATTER-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
-// STORE-SCATTER-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-SCATTER:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
-// STORE-SCATTER:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// STORE-SCATTER:        arith.muli {{.*}} : index
-// STORE-SCATTER:        arith.addi %[[OFFSET]]{{.*}} : index
-// STORE-SCATTER:        arith.addi {{.*}} : index
-// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
-// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both runs, even when the destination is a strided
+// subview.
+// CHECK-LABEL:  @store_to_subview(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf16>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
+// CHECK-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
+// CHECK-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
+// CHECK:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
+// CHECK:        %[[STEP:.+]] = vector.step : vector<8xindex>
+// CHECK:        arith.muli {{.*}} : index
+// CHECK:        arith.addi %[[OFFSET]]{{.*}} : index
+// CHECK:        arith.addi {{.*}} : index
+// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
+// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
+// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----


From llvmlistbot at llvm.org  Sat Jul 18 10:46:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 10:46:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU (PR #210527)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210527@github.com>
Message-ID: <6a5bbbd9.e5907290.281687.23c2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-gpu

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

Extend the vector.transfer_read/transfer_write lowerings so they can produce N-D xegpu.load_nd/store_nd, not just 1D/2D, and relax the out-of-bounds handling to match load_nd's implicit-zero padding. 
Restructure both patterns as "block first, then scatter as  fallback. 

---

Patch is 44.73 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210527.diff


3 Files Affected:

- (modified) mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp (+149-119) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir (+117-52) 
- (modified) mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir (+114-111) 


``````````diff
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..306ce6f1e8008 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -15,6 +15,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/UB/IR/UBOps.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
@@ -49,14 +50,53 @@ static bool isZeroConstant(Value val) {
       .Default(false);
 }
 
+// Return true if the transfer padding value is compatible with the implicit
+// padding of an nd block load. LoadNdOp fills out-of-bounds elements with zero,
+// so a zero constant matches its semantics exactly. A poison padding means the
+// out-of-bounds elements are "don't care", so any implicit padding (including
+// zero) is also acceptable.
+static bool isZeroOrPoisonPadding(Value val) {
+  return isZeroConstant(val) || val.getDefiningOp<ub::PoisonOp>();
+}
+
+// Return true if the permutation map keeps every dimension in place except the
+// innermost two, which are swapped, e.g.:
+//   (d0, d1) -> (d1, d0)
+//   (d0, d1, d2) -> (d2, d1)
+//   (d0, d1, d2, d3) -> (d0, d1, d3, d2)
+// This is the only non-identity permutation an nd block load can realize (by
+// loading the untransposed block and applying a trailing vector.transpose).
+static bool isInnermostTwoDimsTransposed(AffineMap map) {
+  unsigned numResults = map.getNumResults();
+  if (numResults < 2)
+    return false;
+  MLIRContext *ctx = map.getContext();
+  unsigned numInputs = map.getNumInputs();
+  // All but the innermost two results must match the minor-identity map.
+  for (unsigned i = 0; i + 2 < numResults; ++i)
+    if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
+      return false;
+  // The innermost two results must be the last two input dims, swapped.
+  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+         map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
+}
+
 static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
                                             Operation *op, VectorType vecTy,
-                                            MemRefType memTy) {
+                                            MemRefType memTy,
+                                            bool allowHighDimVector = false) {
   // Validate only vector as the basic vector store and load ops guarantee
   // XeGPU-compatible memref source.
   unsigned vecRank = vecTy.getRank();
-  if (!(vecRank == 1 || vecRank == 2))
+  // nd block loads/stores support N-D block transfers, so callers that lower to
+  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
+  // vector.load/store lowering keeps the 1D/2D restriction.
+  if (allowHighDimVector) {
+    if (vecRank == 0)
+      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
+  } else if (!(vecRank == 1 || vecRank == 2)) {
     return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
+  }
 
   if (!vecTy.getElementType().isIntOrFloat())
     return rewriter.notifyMatchFailure(
@@ -594,92 +634,78 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(readOp);
-    // Lower to scattered load Op if the target HW doesn't have 2d block load
-    // support and the load is not from shared memory.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        readOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (isOutOfBounds)
-        return failure();
-      return lowerToScatteredLoadOp(readOp, rewriter);
-    }
-
-    // Handle the 1D non-SLM case using load.gather.
-    if (loadedVecTy.getRank() == 1 && !isOutOfBounds)
-      return lowerToScatteredLoadOp(readOp, rewriter);
-
-    // Perform common data transfer checks.
-    // TODO: Maybe too strict for SLM case.
-    if (failed(
-            storeLoadPreconditions(rewriter, readOp, loadedVecTy, readMemTy)))
-      return failure();
-
-    if (isOutOfBounds && !isZeroConstant(readOp.getPadding()))
-      return rewriter.notifyMatchFailure(
-          readOp, "Unsupported non-zero padded out-of-bounds read");
+    bool hasBlockLoadSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // An nd block load can realize a minor-identity map directly, or an
+    // innermost-two-dims transpose via a trailing vector.transpose. Any other
+    // permutation (e.g. a mid-vector transpose of a high-dim load) is left to
+    // the scattered path, which permutes strides explicitly.
     AffineMap readMap = readOp.getPermutationMap();
-    // Check if this is a transpose: the map must have exactly 2 results,
-    // and those 2 results must be the last 2 input dimensions interchanged.
-    // Examples:
-    //   (d0, d1) -> (d1, d0)      // transpose
-    //   (d0, d1) -> (d0, d1)      // not a transpose
-    //   (d0, d1, d2) -> (d2, d1)  // transpose (last 2 dims swapped)
-    bool isTransposeLoad = false;
-    if (readMap.getNumResults() == 2) {
-      auto results = readMap.getResults();
-      unsigned numInputs = readMap.getNumInputs();
-      if (numInputs >= 2) {
-        auto lastDim = getAffineDimExpr(numInputs - 1, readMap.getContext());
-        auto secondLastDim =
-            getAffineDimExpr(numInputs - 2, readMap.getContext());
-        isTransposeLoad =
-            (results[0] == lastDim && results[1] == secondLastDim);
+    bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
+
+    // Prefer an nd block load. It requires HW block-load support, a >1D vector
+    // of a scalar element type backed by a scalar-element memref, and a map the
+    // block load can realize. Out-of-bounds reads are allowed as long as the
+    // padding matches load_nd's implicit zero padding.
+    bool canLowerToLoadNd =
+        hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
+        (readMap.isMinorIdentity() || isTransposeLoad) &&
+        loadedVecTy.getElementType().isIntOrFloat() &&
+        readMemTy.getElementType().isIntOrFloat() &&
+        (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
+
+    if (canLowerToLoadNd) {
+      auto elementType = loadedVecTy.getElementType();
+
+      SmallVector<int64_t> descShape(loadedVecTy.getShape());
+      if (isTransposeLoad) {
+        // If load is transposed, simply swap the last two dimensions of the
+        // loaded vector type to get the descriptor shape.
+        size_t rank = descShape.size();
+        assert(rank >= 2 && "Transpose requires at least 2 dimensions");
+        std::swap(descShape[rank - 1], descShape[rank - 2]);
+        loadedVecTy = VectorType::get(descShape, elementType);
       }
+      auto descType = xegpu::TensorDescType::get(
+          descShape, elementType, /*array_length=*/1,
+          /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, readOp.getBase(),
+          getAsOpFoldResult(readOp.getIndices()), loadedVecTy.getRank());
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      Operation *loadedOp =
+          xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
+                                  /*packed=*/nullptr, /*transpose=*/nullptr,
+                                  /*l1_hint=*/hint,
+                                  /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                  /*layout=*/nullptr);
+      if (isTransposeLoad) {
+        // Undo the innermost-two-dims swap with a trailing vector.transpose:
+        // keep the leading dimensions in place and interchange only the last
+        // two.
+        int64_t rank = loadedVecTy.getRank();
+        SmallVector<int64_t> perm(llvm::to_vector(llvm::seq<int64_t>(0, rank)));
+        std::swap(perm[rank - 1], perm[rank - 2]);
+        loadedOp = vector::TransposeOp::create(rewriter, loc,
+                                               loadedOp->getResult(0), perm);
+      }
+      rewriter.replaceOp(readOp, loadedOp);
+      return success();
     }
-    auto elementType = loadedVecTy.getElementType();
-
-    SmallVector<int64_t> descShape(loadedVecTy.getShape());
-    if (isTransposeLoad) {
-      // If load is transposed, simply swap the last two dimensions of the
-      // loaded vector type to get the descriptor shape.
-      size_t rank = descShape.size();
-      assert(rank >= 2 && "Transpose requires at least 2 dimensions");
-      std::swap(descShape[rank - 1], descShape[rank - 2]);
-      loadedVecTy = VectorType::get(descShape, elementType);
-    }
-    auto descType = xegpu::TensorDescType::get(
-        descShape, elementType, /*array_length=*/1,
-        /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, readOp.getBase(), getAsOpFoldResult(readOp.getIndices()),
-        loadedVecTy.getRank());
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    Operation *loadedOp =
-        xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
-                                /*packed=*/nullptr, /*transpose=*/nullptr,
-                                /*l1_hint=*/hint,
-                                /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                /*layout=*/nullptr);
-    if (isTransposeLoad) {
-      // Transposing the loaded vector with a separate vector.transpose
-      // operation
-      auto range = llvm::seq<int64_t>(0, readMap.getResults().size());
-      SmallVector<int64_t> perm(
-          range.rbegin(), range.rend()); // reverse the range for transpose
-      loadedOp = vector::TransposeOp::create(rewriter, loc,
-                                             loadedOp->getResult(0), perm);
-    }
-    rewriter.replaceOp(readOp, loadedOp);
 
-    return success();
+    // Fall back to a scattered load. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (isOutOfBounds)
+      return failure();
+    return lowerToScatteredLoadOp(readOp, rewriter);
   }
 };
 
@@ -727,47 +753,51 @@ struct TransferWriteLowering
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(writeOp);
-    // Lower to scattered store Op if the target HW doesn't have 2d block
-    // store support and the memref is not SLM.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        writeOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (writeOp.hasOutOfBoundsDim())
-        return failure();
-      return lowerToScatteredStoreOp(writeOp, rewriter);
-    }
-
-    if (failed(storeLoadPreconditions(rewriter, writeOp, vecTy, writeMemTy)))
-      return failure();
+    bool hasBlockStoreSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // Prefer an nd block store. It requires HW block-store support, a >1D
+    // vector of a scalar element type backed by a scalar-element memref, and a
+    // minor-identity map (block stores have no transpose support). Out-of-bounds
+    // writes are handled by the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    if (!map.isMinorIdentity())
-      return rewriter.notifyMatchFailure(writeOp, "Expects identity map");
-
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, writeOp.getBase(),
-        getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
-
-    auto descType = xegpu::TensorDescType::get(
-        vecTy.getShape(), vecTy.getElementType(),
-        /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
-        xegpu::MemorySpace::Global);
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    auto storeOp = xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(),
-                                            ndDesc, indices,
-                                            /*l1_hint=*/hint,
-                                            /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                            /*layout=*/nullptr);
-    rewriter.replaceOp(writeOp, storeOp);
+    bool canLowerToStoreNd =
+        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
+        vecTy.getElementType().isIntOrFloat() &&
+        writeMemTy.getElementType().isIntOrFloat();
+
+    if (canLowerToStoreNd) {
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, writeOp.getBase(),
+          getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
+
+      auto descType = xegpu::TensorDescType::get(
+          vecTy.getShape(), vecTy.getElementType(),
+          /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
+          xegpu::MemorySpace::Global);
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      auto storeOp =
+          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
+                                   indices,
+                                   /*l1_hint=*/hint,
+                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                   /*layout=*/nullptr);
+      rewriter.replaceOp(writeOp, storeOp);
+      return success();
+    }
 
-    return success();
+    // Fall back to a scattered store. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (writeOp.hasOutOfBoundsDim())
+      return failure();
+    return lowerToScatteredStoreOp(writeOp, rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index 3354868cd6cb9..ed495729f9add 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -182,6 +182,33 @@ gpu.func @load_transpose_3d_memref(%source: memref<32x64x128xf32>,
 
 }
 
+// -----
+// A high-dim load whose innermost two dims are transposed lowers to an
+// (untransposed) nd block load followed by a vector.transpose of the last two
+// dims.
+gpu.module @xevm_module {
+gpu.func @load_high_dim_transposed(%source: memref<2x2x64x128xf16>,
+    %offset: index) -> vector<1x1x64x128xf16> {
+  %c0 = arith.constant 0.0 : f16
+  %0 = vector.transfer_read %source[%offset, %offset, %offset, %offset], %c0
+    {permutation_map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3, d2)>,
+    in_bounds = [true, true, true, true]}
+    : memref<2x2x64x128xf16>, vector<1x1x64x128xf16>
+  gpu.return %0 : vector<1x1x64x128xf16>
+}
+
+// LOAD-ND-LABEL:  @load_high_dim_transposed(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x64x128xf16>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x64x128xf16>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<1x1x128x64xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<1x1x128x64xf16>
+// LOAD-ND:        vector.transpose %[[VEC]], [0, 1, 3, 2] : vector<1x1x128x64xf16> to vector<1x1x64x128xf16>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_transposed(
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load {{.*}} : i64, vector<1x1x64x128xindex>, vector<1x1x64x128xi1> -> vector<1x1x64x128xf16>
+}
+
 // -----
 gpu.module @xevm_module {
 gpu.func @load_dynamic_source(%source: memref<?x?x?xf32>,
@@ -275,21 +302,31 @@ gpu.func @load_dynamic_source3(%source: memref<?x?x?x?x?xf32>,
   gpu.return %0 : vector<2x4x8x16xf32>
 }
 
-// CHECK-LABEL:  @load_dynamic_source3(
-// CHECK-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
-// CHECK:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
-// CHECK-COUNT4: vector.step
-// CHECK-COUNT3: vector.broadcast
-// CHECK-COUNT4: vector.shape_cast
-// CHECK-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
-// CHECK-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
-// CHECK:        return %[[VEC]]
+// LOAD-ND-LABEL:  @load_dynamic_source3(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFF1:.*]], %[[SIZES:.*]]:4, %[[STRIDES:.*]]:4 = memref.extract_strided_metadata %[[SUBVIEW]]
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x4x8x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x4x8x16xf32>
+// LOAD-ND:        return %[[VEC]]
+
+// LOAD-GATHER-LABEL:  @load_dynamic_source3(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
+// LOAD-GATHER:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
+// LOAD-GATHER-COUNT4: vector.step
+// LOAD-GATHER-COUNT3: vector.broadcast
+// LOAD-GATHER-COUNT4: vector.shape_cast
+// LOAD-GATHER-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
+// LOAD-GATHER:        return %[[VEC]]
 }
 
 // -----
@@ -302,21 +339,27 @@ gpu.func @load_high_dim_vector(%source: memref<16x32x64xf32>,
   gpu.return %0 : vector<8x16x32xf32>
 }
 
-// CHECK-LABEL:  @load_high_dim_vector(
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210527

From llvmlistbot at llvm.org  Sat Jul 18 10:47:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 10:47:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU (PR #210527)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210527@github.com>
Message-ID: <6a5bbc1a.bc28b987.2e0c9a.b1f0@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 306ce6f1e..e7f7e9340 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -77,7 +77,8 @@ static bool isInnermostTwoDimsTransposed(AffineMap map) {
     if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
       return false;
   // The innermost two results must be the last two input dims, swapped.
-  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+  return map.getResult(numResults - 2) ==
+             getAffineDimExpr(numInputs - 1, ctx) &&
          map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
 }
 
@@ -760,13 +761,13 @@ struct TransferWriteLowering
 
     // Prefer an nd block store. It requires HW block-store support, a >1D
     // vector of a scalar element type backed by a scalar-element memref, and a
-    // minor-identity map (block stores have no transpose support). Out-of-bounds
-    // writes are handled by the descriptor's boundary check.
+    // minor-identity map (block stores have no transpose support).
+    // Out-of-bounds writes are handled by the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    bool canLowerToStoreNd =
-        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
-        vecTy.getElementType().isIntOrFloat() &&
-        writeMemTy.getElementType().isIntOrFloat();
+    bool canLowerToStoreNd = hasBlockStoreSupport && vecTy.getRank() > 1 &&
+                             map.isMinorIdentity() &&
+                             vecTy.getElementType().isIntOrFloat() &&
+                             writeMemTy.getElementType().isIntOrFloat();
 
     if (canLowerToStoreNd) {
       auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
@@ -782,18 +783,17 @@ struct TransferWriteLowering
       xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
           rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
 
-      auto storeOp =
-          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
-                                   indices,
-                                   /*l1_hint=*/hint,
-                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                   /*layout=*/nullptr);
+      auto storeOp = xegpu::StoreNdOp::create(
+          rewriter, loc, writeOp.getVector(), ndDesc, indices,
+          /*l1_hint=*/hint,
+          /*l2_hint=*/hint, /*l3_hint=*/hint,
+          /*layout=*/nullptr);
       rewriter.replaceOp(writeOp, storeOp);
       return success();
     }
 
-    // Fall back to a scattered store. It supports arbitrary permutations and any
-    // rank, but cannot express out-of-bounds accesses.
+    // Fall back to a scattered store. It supports arbitrary permutations and
+    // any rank, but cannot express out-of-bounds accesses.
     // TODO: add support for OutOfBound access.
     if (writeOp.hasOutOfBoundsDim())
       return failure();

``````````

</details>


https://github.com/llvm/llvm-project/pull/210527

From llvmlistbot at llvm.org  Sat Jul 18 11:02:25 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sat, 18 Jul 2026 11:02:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5bbfb1.92d2873c.aa1ff.1c9e@mx.google.com>


https://github.com/secona updated https://github.com/llvm/llvm-project/pull/210105

>From 7f78ea3f105503b8d107249726710007e6f18b91 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sun, 19 Jul 2026 00:35:39 +0700
Subject: [PATCH] [mlir][bufferization] Fix stale BufferOriginAnalysis in
 BufferDeallocationSimplification

---
 .../BufferDeallocationSimplification.cpp      | 16 ++++++++--
 .../buffer-deallocation-simplification.mlir   | 29 +++++++++++++++++++
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..9660786868498 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -469,12 +469,24 @@ struct BufferDeallocationSimplificationPass
                  RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
                                                                 analysis);
 
-    populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
     // `BufferOriginAnalysis` so we apply the rewrites with `Normal` level of
-    // region simplification
+    // region simplification and disable folding.
     if (failed(applyPatternsGreedily(
             getOperation(), std::move(patterns),
+            GreedyRewriteConfig()
+                .setRegionSimplificationLevel(GreedySimplifyRegionLevel::Normal)
+                .enableFolding(false))))
+      signalPassFailure();
+
+    // We run canonicalization separately so it can benefit from
+    // folding, which was disabled in the previous pass to avoid invalidating
+    // `BufferOriginAnalysis`.
+    RewritePatternSet canonicalizationPatterns(&getContext());
+    populateDeallocOpCanonicalizationPatterns(canonicalizationPatterns,
+                                              &getContext());
+    if (failed(applyPatternsGreedily(
+            getOperation(), std::move(canonicalizationPatterns),
             GreedyRewriteConfig().setRegionSimplificationLevel(
                 GreedySimplifyRegionLevel::Normal))))
       signalPassFailure();
diff --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)


From llvmlistbot at llvm.org  Sat Jul 18 11:04:57 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sat, 18 Jul 2026 11:04:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5bc049.1e1ac41d.3969c3.cf7a@mx.google.com>


secona wrote:

So some tests were failing because the pass resulted in additional foldable ops due to folding being disabled. I found that splitting the canonicalization patterns to run separately allows us to enable folding without affecting the tests. I've pushed the change that splits the canonicalization patterns. What do you think?

https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sat Jul 18 11:09:15 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Sat, 18 Jul 2026 11:09:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5bc14b.e55dc341.1c6a0.a0ea@mx.google.com>


================
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
----------------
rengolin wrote:

Is there no case where this is `False`? If so, then we can remove the option?

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 11:09:16 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Sat, 18 Jul 2026 11:09:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5bc14c.481afadf.9e047.8010@mx.google.com>


================
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
----------------
rengolin wrote:

Can you add another `with first_context` to make sure we catch the "already loaded" in that case?

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 11:29:20 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sat, 18 Jul 2026 11:29:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5bc600.449bfb44.7b862.1599@mx.google.com>


https://github.com/secona updated https://github.com/llvm/llvm-project/pull/210105

>From 7d58b4e935db92a50fb9eeb1c4ecce2965a314ee Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sun, 19 Jul 2026 00:35:39 +0700
Subject: [PATCH] [mlir][bufferization] Fix stale BufferOriginAnalysis in
 BufferDeallocationSimplification

---
 .../BufferDeallocationSimplification.cpp      | 16 ++++++++--
 .../buffer-deallocation-simplification.mlir   | 29 +++++++++++++++++++
 2 files changed, 43 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..06e90333fe469 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -469,12 +469,24 @@ struct BufferDeallocationSimplificationPass
                  RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
                                                                 analysis);
 
-    populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
     // `BufferOriginAnalysis` so we apply the rewrites with `Normal` level of
-    // region simplification
+    // region simplification and disable folding.
     if (failed(applyPatternsGreedily(
             getOperation(), std::move(patterns),
+            GreedyRewriteConfig()
+                .setRegionSimplificationLevel(GreedySimplifyRegionLevel::Normal)
+                .enableFolding(false))))
+      signalPassFailure();
+
+    // We run canonicalization separately so it can benefit from
+    // folding, which was disabled previously to avoid invalidating
+    // `BufferOriginAnalysis`.
+    RewritePatternSet canonicalizationPatterns(&getContext());
+    populateDeallocOpCanonicalizationPatterns(canonicalizationPatterns,
+                                              &getContext());
+    if (failed(applyPatternsGreedily(
+            getOperation(), std::move(canonicalizationPatterns),
             GreedyRewriteConfig().setRegionSimplificationLevel(
                 GreedySimplifyRegionLevel::Normal))))
       signalPassFailure();
diff --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)


From llvmlistbot at llvm.org  Sat Jul 18 11:41:35 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 18 Jul 2026 11:41:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bc8df.0c05e68e.3c2f5d.8677@mx.google.com>


https://github.com/makslevental updated https://github.com/llvm/llvm-project/pull/208934

>From 496a4a2d6b35cb97e7493d1d1612e6cb9e7b6781 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:57:46 -0700
Subject: [PATCH 1/5] [mlir-c] Add TypeConverter source materialization

---
 mlir/include/mlir-c/Rewrite.h        |  16 ++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  32 +++++++
 mlir/test/CAPI/rewrite.c             | 125 +++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..2d7ed3e4b7111 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,22 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..2fb8eff5ad79d 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,38 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..8e5b98abdb883 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,130 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +942,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 12609605057480049675bea96fc6de538c16141d Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:59:12 -0700
Subject: [PATCH 2/5] [mlir-c] Add TypeConverter target materialization

---
 mlir/include/mlir-c/Rewrite.h        |  7 +++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  9 +++
 mlir/test/CAPI/rewrite.c             | 90 ++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 2d7ed3e4b7111..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -621,6 +621,13 @@ MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
     MlirTypeConverter typeConverter,
     MlirTypeConverterMaterializationCallback callback, void *userData);
 
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 2fb8eff5ad79d..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -701,6 +701,15 @@ void mlirTypeConverterAddSourceMaterialization(
           wrapMaterializationCallback(callback, userData));
 }
 
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 8e5b98abdb883..c1624c54eb461 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -926,6 +926,95 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
   fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
 }
 
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -943,6 +1032,7 @@ int main(void) {
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
   testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 7fb9b326d964f82016e7553d911d4c479798e1f4 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 11 Jul 2026 11:55:10 -0700
Subject: [PATCH 3/5] [mlir-c] Tighten materialization test CHECKs to pin full
 module body

Use CHECK-NEXT to match the entire module body (module { ... }) rather
than loose CHECK lines, so the tests also assert the absence of any
stray ops -- e.g. a leftover builtin.unrealized_conversion_cast -- that
a loose CHECK would silently allow.
---
 mlir/test/CAPI/rewrite.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index c1624c54eb461..6d87d0236061b 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -911,9 +911,11 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
-  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK-NEXT:   "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);
@@ -1000,9 +1002,11 @@ void testTypeConverterTargetMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
-  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK-NEXT:   "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);

>From 0747c994b3badf2322f7af5362ceb9556e3f42ec Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Fri, 17 Jul 2026 12:22:08 -0700
Subject: [PATCH 4/5] address comment

---
 mlir/test/CAPI/rewrite.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 6d87d0236061b..b10afbd8b812c 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -627,7 +627,7 @@ static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
                                                                void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
 }
 
@@ -635,7 +635,7 @@ static MlirConversionTargetLegality
 dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
 }
 
@@ -643,7 +643,7 @@ static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
                                                              void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
 }
 
@@ -823,7 +823,7 @@ static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
                                           void *userData) {
   intptr_t *counter = (intptr_t *)userData;
   if (counter)
-    (*counter)++;
+    ++(*counter);
   MlirOperationState state =
       mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
   mlirOperationStateAddOperands(&state, nInputs, inputs);

>From d88556da98669fe01bdf7839a0adae890c43c615 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 18 Jul 2026 11:37:08 -0700
Subject: [PATCH 5/5] [mlir-c] Use a status enum for the type conversion
 callback

The 1:1 conversion callback returned MlirLogicalResult and encoded the
three C++ conversion states implicitly: returning failure meant "try
another conversion", while returning success with a null out-parameter
meant a hard failure. This dual encoding was easy to misuse and the doc
comment conflated the two.

Return a MlirTypeConverterConversionStatus enum with explicit Success,
Failure (do not try another), and Declined (try another) states, and add
a test covering the decline-fallback and hard-failure paths.
---
 mlir/include/mlir-c/Rewrite.h        | 25 +++++++--
 mlir/lib/CAPI/Transforms/Rewrite.cpp | 18 +++++--
 mlir/test/CAPI/rewrite.c             | 77 ++++++++++++++++++++++++++--
 3 files changed, 105 insertions(+), 15 deletions(-)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index e074fc96f2aeb..57cb979de012a 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -588,11 +588,26 @@ MLIR_CAPI_EXPORTED MlirTypeConverter mlirTypeConverterCreate(void);
 MLIR_CAPI_EXPORTED void
 mlirTypeConverterDestroy(MlirTypeConverter typeConverter);
 
-/// Callback type for type conversion functions.
-/// Returns failure or sets convertedType to MlirType{NULL} to indicate failure.
-/// If failure is returned, the converter is allowed to try another
-/// conversion function to perform the conversion.
-typedef MlirLogicalResult (*MlirTypeConverterConversionCallback)(
+/// Outcome of a type conversion callback (MlirTypeConverterConversionCallback).
+/// Mirrors the three states of the underlying C++ conversion function.
+typedef enum MlirTypeConverterConversionStatus {
+  /// The type was converted; the callback set `*convertedType` to the result.
+  MlirTypeConverterConversionStatusSuccess = 0,
+  /// The conversion failed hard; no further conversion function will be tried.
+  MlirTypeConverterConversionStatusFailure = 1,
+  /// The conversion was declined; another registered conversion function may be
+  /// tried.
+  MlirTypeConverterConversionStatusDeclined = 2,
+} MlirTypeConverterConversionStatus;
+
+/// Callback type for type conversion functions. On
+/// MlirTypeConverterConversionStatusSuccess the callback must set
+/// `*convertedType` to the converted type.
+/// MlirTypeConverterConversionStatusDeclined leaves the type unconverted and
+/// allows another conversion function to be tried, whereas
+/// MlirTypeConverterConversionStatusFailure fails the conversion outright
+/// without trying any further conversion function.
+typedef MlirTypeConverterConversionStatus (*MlirTypeConverterConversionCallback)(
     MlirType type, MlirType *convertedType, void *userData);
 
 /// Add a type conversion function to the given TypeConverter.
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 92456d6d8a435..e324a848691e9 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -654,13 +654,21 @@ void mlirTypeConverterAddConversion(
       ->addConversion(
           [convertType, userData](Type type) -> std::optional<Type> {
             MlirType converted{nullptr};
-            MlirLogicalResult result =
+            MlirTypeConverterConversionStatus status =
                 convertType(wrap(type), &converted, userData);
-            if (mlirLogicalResultIsFailure(result))
-              return std::nullopt; // allowed to try another conversion function
-            if (mlirTypeIsNull(converted))
+            switch (status) {
+            case MlirTypeConverterConversionStatusSuccess:
+              assert(!mlirTypeIsNull(converted) &&
+                     "a successful conversion must set convertedType");
+              return unwrap(converted);
+            case MlirTypeConverterConversionStatusFailure:
+              // A null result maps to success(false), i.e. a hard failure: the
+              // driver will not try another conversion function.
               return nullptr;
-            return unwrap(converted);
+            case MlirTypeConverterConversionStatusDeclined:
+              return std::nullopt; // allowed to try another conversion function
+            }
+            llvm_unreachable("unknown MlirTypeConverterConversionStatus");
           });
 }
 
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index b10afbd8b812c..6615364825e95 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -804,18 +804,45 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
 
 // Type conversion callback: maps i32 -> i64 and leaves every other type
 // unchanged (identity). Used by the materialization tests below.
-static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
-                                       void *userData) {
+static MlirTypeConverterConversionStatus
+widenI32ToI64(MlirType type, MlirType *result, void *userData) {
   (void)userData;
   if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
     *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
   else
     *result = type;
-  return mlirLogicalResultSuccess();
+  return MlirTypeConverterConversionStatusSuccess;
 }
 
-// Materialization callback: builds a `test.cast` op that produces a single
-// value of `outputType` from the given inputs, and records that it ran by
+// 1:1 conversion callback that declines i32 (returns Declined) so another
+// conversion function is tried, and is the identity on everything else. Bumps
+// the counter passed as userData when it declines.
+static MlirTypeConverterConversionStatus
+declineI32(MlirType type, MlirType *result, void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32) {
+    if (counter)
+      ++(*counter);
+    return MlirTypeConverterConversionStatusDeclined;
+  }
+  *result = type;
+  return MlirTypeConverterConversionStatusSuccess;
+}
+
+// 1:1 conversion callback that fails hard on i32 (returns Failure), which must
+// abort the conversion without trying another conversion function. Bumps the
+// counter passed as userData when it fails.
+static MlirTypeConverterConversionStatus
+failI32(MlirType type, MlirType *result, void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32) {
+    if (counter)
+      ++(*counter);
+    return MlirTypeConverterConversionStatusFailure;
+  }
+  *result = type;
+  return MlirTypeConverterConversionStatusSuccess;
+}
 // bumping the counter passed as userData.
 static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
                                           MlirType outputType, intptr_t nInputs,
@@ -862,6 +889,45 @@ static MlirLogicalResult convertSource(MlirConversionPattern pattern,
   return mlirLogicalResultSuccess();
 }
 
+void testTypeConverterConversionStates(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterConversionStates
+  fprintf(stderr, "@testTypeConverterConversionStates\n");
+
+  MlirType i32 = mlirIntegerTypeGet(ctx, 32);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  (void)i64;
+
+  // Declined: the i32 decliner is registered last (tried first) and defers to
+  // the i32 -> i64 conversion registered before it, so i32 still converts.
+  MlirTypeConverter declineConverter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(declineConverter, widenI32ToI64, NULL);
+  intptr_t declineCounter = 0;
+  mlirTypeConverterAddConversion(declineConverter, declineI32, &declineCounter);
+  MlirType declined = mlirTypeConverterConvertType(declineConverter, i32);
+  (void)declined;
+  assert(declineCounter == 1 && "declining conversion must be consulted");
+  assert(mlirTypeEqual(declined, i64) &&
+         "decline must fall back to the i32 -> i64 conversion");
+  mlirTypeConverterDestroy(declineConverter);
+
+  // Failure: the i32 hard-failure is registered last (tried first) and must
+  // abort the conversion without falling back to the i32 -> i64 conversion, so
+  // the type converts to null.
+  MlirTypeConverter failConverter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(failConverter, widenI32ToI64, NULL);
+  intptr_t failCounter = 0;
+  mlirTypeConverterAddConversion(failConverter, failI32, &failCounter);
+  MlirType failed = mlirTypeConverterConvertType(failConverter, i32);
+  (void)failed;
+  assert(failCounter == 1 && "failing conversion must be consulted");
+  assert(mlirTypeIsNull(failed) &&
+         "hard failure must not fall back to another conversion");
+  mlirTypeConverterDestroy(failConverter);
+
+  // CHECK: testTypeConverterConversionStates: PASSED
+  fprintf(stderr, "testTypeConverterConversionStates: PASSED\n");
+}
+
 void testTypeConverterSourceMaterialization(MlirContext ctx) {
   // CHECK-LABEL: @testTypeConverterSourceMaterialization
   fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
@@ -1035,6 +1101,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterConversionStates(ctx);
   testTypeConverterSourceMaterialization(ctx);
   testTypeConverterTargetMaterialization(ctx);
 


From llvmlistbot at llvm.org  Sat Jul 18 11:43:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:43:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bc944.492e23e4.d19bc.07e6@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions c,h,cpp -- mlir/include/mlir-c/Rewrite.h mlir/lib/CAPI/Transforms/Rewrite.cpp mlir/test/CAPI/rewrite.c --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 57cb979de..cc98bf7ba 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -607,8 +607,10 @@ typedef enum MlirTypeConverterConversionStatus {
 /// allows another conversion function to be tried, whereas
 /// MlirTypeConverterConversionStatusFailure fails the conversion outright
 /// without trying any further conversion function.
-typedef MlirTypeConverterConversionStatus (*MlirTypeConverterConversionCallback)(
-    MlirType type, MlirType *convertedType, void *userData);
+typedef MlirTypeConverterConversionStatus (
+    *MlirTypeConverterConversionCallback)(MlirType type,
+                                          MlirType *convertedType,
+                                          void *userData);
 
 /// Add a type conversion function to the given TypeConverter.
 MLIR_CAPI_EXPORTED void

``````````

</details>


https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 18 11:44:07 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 18 Jul 2026 11:44:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bc977.6c8668f2.1f24.3cc5@mx.google.com>


https://github.com/makslevental updated https://github.com/llvm/llvm-project/pull/208934

>From 496a4a2d6b35cb97e7493d1d1612e6cb9e7b6781 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:57:46 -0700
Subject: [PATCH 1/5] [mlir-c] Add TypeConverter source materialization

---
 mlir/include/mlir-c/Rewrite.h        |  16 ++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  32 +++++++
 mlir/test/CAPI/rewrite.c             | 125 +++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..2d7ed3e4b7111 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,22 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..2fb8eff5ad79d 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,38 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..8e5b98abdb883 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,130 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +942,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 12609605057480049675bea96fc6de538c16141d Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:59:12 -0700
Subject: [PATCH 2/5] [mlir-c] Add TypeConverter target materialization

---
 mlir/include/mlir-c/Rewrite.h        |  7 +++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  9 +++
 mlir/test/CAPI/rewrite.c             | 90 ++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 2d7ed3e4b7111..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -621,6 +621,13 @@ MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
     MlirTypeConverter typeConverter,
     MlirTypeConverterMaterializationCallback callback, void *userData);
 
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 2fb8eff5ad79d..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -701,6 +701,15 @@ void mlirTypeConverterAddSourceMaterialization(
           wrapMaterializationCallback(callback, userData));
 }
 
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 8e5b98abdb883..c1624c54eb461 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -926,6 +926,95 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
   fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
 }
 
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -943,6 +1032,7 @@ int main(void) {
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
   testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 7fb9b326d964f82016e7553d911d4c479798e1f4 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 11 Jul 2026 11:55:10 -0700
Subject: [PATCH 3/5] [mlir-c] Tighten materialization test CHECKs to pin full
 module body

Use CHECK-NEXT to match the entire module body (module { ... }) rather
than loose CHECK lines, so the tests also assert the absence of any
stray ops -- e.g. a leftover builtin.unrealized_conversion_cast -- that
a loose CHECK would silently allow.
---
 mlir/test/CAPI/rewrite.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index c1624c54eb461..6d87d0236061b 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -911,9 +911,11 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
-  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK-NEXT:   "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);
@@ -1000,9 +1002,11 @@ void testTypeConverterTargetMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
-  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK-NEXT:   "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);

>From 0747c994b3badf2322f7af5362ceb9556e3f42ec Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Fri, 17 Jul 2026 12:22:08 -0700
Subject: [PATCH 4/5] address comment

---
 mlir/test/CAPI/rewrite.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 6d87d0236061b..b10afbd8b812c 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -627,7 +627,7 @@ static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
                                                                void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
 }
 
@@ -635,7 +635,7 @@ static MlirConversionTargetLegality
 dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
 }
 
@@ -643,7 +643,7 @@ static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
                                                              void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
 }
 
@@ -823,7 +823,7 @@ static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
                                           void *userData) {
   intptr_t *counter = (intptr_t *)userData;
   if (counter)
-    (*counter)++;
+    ++(*counter);
   MlirOperationState state =
       mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
   mlirOperationStateAddOperands(&state, nInputs, inputs);

>From 0e10d1f261c6670fbcf1f45afce2f86c9fb29fe6 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 18 Jul 2026 11:37:08 -0700
Subject: [PATCH 5/5] [mlir-c] Use a status enum for the type conversion
 callback

The 1:1 conversion callback returned MlirLogicalResult and encoded the
three C++ conversion states implicitly: returning failure meant "try
another conversion", while returning success with a null out-parameter
meant a hard failure. This dual encoding was easy to misuse and the doc
comment conflated the two.

Return a MlirTypeConverterConversionStatus enum with explicit Success,
Failure (do not try another), and Declined (try another) states, and add
a test covering the decline-fallback and hard-failure paths.
---
 mlir/include/mlir-c/Rewrite.h        | 29 ++++++++---
 mlir/lib/CAPI/Transforms/Rewrite.cpp | 18 +++++--
 mlir/test/CAPI/rewrite.c             | 77 ++++++++++++++++++++++++++--
 3 files changed, 108 insertions(+), 16 deletions(-)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index e074fc96f2aeb..cc98bf7ba9a63 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -588,12 +588,29 @@ MLIR_CAPI_EXPORTED MlirTypeConverter mlirTypeConverterCreate(void);
 MLIR_CAPI_EXPORTED void
 mlirTypeConverterDestroy(MlirTypeConverter typeConverter);
 
-/// Callback type for type conversion functions.
-/// Returns failure or sets convertedType to MlirType{NULL} to indicate failure.
-/// If failure is returned, the converter is allowed to try another
-/// conversion function to perform the conversion.
-typedef MlirLogicalResult (*MlirTypeConverterConversionCallback)(
-    MlirType type, MlirType *convertedType, void *userData);
+/// Outcome of a type conversion callback (MlirTypeConverterConversionCallback).
+/// Mirrors the three states of the underlying C++ conversion function.
+typedef enum MlirTypeConverterConversionStatus {
+  /// The type was converted; the callback set `*convertedType` to the result.
+  MlirTypeConverterConversionStatusSuccess = 0,
+  /// The conversion failed hard; no further conversion function will be tried.
+  MlirTypeConverterConversionStatusFailure = 1,
+  /// The conversion was declined; another registered conversion function may be
+  /// tried.
+  MlirTypeConverterConversionStatusDeclined = 2,
+} MlirTypeConverterConversionStatus;
+
+/// Callback type for type conversion functions. On
+/// MlirTypeConverterConversionStatusSuccess the callback must set
+/// `*convertedType` to the converted type.
+/// MlirTypeConverterConversionStatusDeclined leaves the type unconverted and
+/// allows another conversion function to be tried, whereas
+/// MlirTypeConverterConversionStatusFailure fails the conversion outright
+/// without trying any further conversion function.
+typedef MlirTypeConverterConversionStatus (
+    *MlirTypeConverterConversionCallback)(MlirType type,
+                                          MlirType *convertedType,
+                                          void *userData);
 
 /// Add a type conversion function to the given TypeConverter.
 MLIR_CAPI_EXPORTED void
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 92456d6d8a435..e324a848691e9 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -654,13 +654,21 @@ void mlirTypeConverterAddConversion(
       ->addConversion(
           [convertType, userData](Type type) -> std::optional<Type> {
             MlirType converted{nullptr};
-            MlirLogicalResult result =
+            MlirTypeConverterConversionStatus status =
                 convertType(wrap(type), &converted, userData);
-            if (mlirLogicalResultIsFailure(result))
-              return std::nullopt; // allowed to try another conversion function
-            if (mlirTypeIsNull(converted))
+            switch (status) {
+            case MlirTypeConverterConversionStatusSuccess:
+              assert(!mlirTypeIsNull(converted) &&
+                     "a successful conversion must set convertedType");
+              return unwrap(converted);
+            case MlirTypeConverterConversionStatusFailure:
+              // A null result maps to success(false), i.e. a hard failure: the
+              // driver will not try another conversion function.
               return nullptr;
-            return unwrap(converted);
+            case MlirTypeConverterConversionStatusDeclined:
+              return std::nullopt; // allowed to try another conversion function
+            }
+            llvm_unreachable("unknown MlirTypeConverterConversionStatus");
           });
 }
 
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index b10afbd8b812c..6615364825e95 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -804,18 +804,45 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
 
 // Type conversion callback: maps i32 -> i64 and leaves every other type
 // unchanged (identity). Used by the materialization tests below.
-static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
-                                       void *userData) {
+static MlirTypeConverterConversionStatus
+widenI32ToI64(MlirType type, MlirType *result, void *userData) {
   (void)userData;
   if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
     *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
   else
     *result = type;
-  return mlirLogicalResultSuccess();
+  return MlirTypeConverterConversionStatusSuccess;
 }
 
-// Materialization callback: builds a `test.cast` op that produces a single
-// value of `outputType` from the given inputs, and records that it ran by
+// 1:1 conversion callback that declines i32 (returns Declined) so another
+// conversion function is tried, and is the identity on everything else. Bumps
+// the counter passed as userData when it declines.
+static MlirTypeConverterConversionStatus
+declineI32(MlirType type, MlirType *result, void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32) {
+    if (counter)
+      ++(*counter);
+    return MlirTypeConverterConversionStatusDeclined;
+  }
+  *result = type;
+  return MlirTypeConverterConversionStatusSuccess;
+}
+
+// 1:1 conversion callback that fails hard on i32 (returns Failure), which must
+// abort the conversion without trying another conversion function. Bumps the
+// counter passed as userData when it fails.
+static MlirTypeConverterConversionStatus
+failI32(MlirType type, MlirType *result, void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32) {
+    if (counter)
+      ++(*counter);
+    return MlirTypeConverterConversionStatusFailure;
+  }
+  *result = type;
+  return MlirTypeConverterConversionStatusSuccess;
+}
 // bumping the counter passed as userData.
 static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
                                           MlirType outputType, intptr_t nInputs,
@@ -862,6 +889,45 @@ static MlirLogicalResult convertSource(MlirConversionPattern pattern,
   return mlirLogicalResultSuccess();
 }
 
+void testTypeConverterConversionStates(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterConversionStates
+  fprintf(stderr, "@testTypeConverterConversionStates\n");
+
+  MlirType i32 = mlirIntegerTypeGet(ctx, 32);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  (void)i64;
+
+  // Declined: the i32 decliner is registered last (tried first) and defers to
+  // the i32 -> i64 conversion registered before it, so i32 still converts.
+  MlirTypeConverter declineConverter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(declineConverter, widenI32ToI64, NULL);
+  intptr_t declineCounter = 0;
+  mlirTypeConverterAddConversion(declineConverter, declineI32, &declineCounter);
+  MlirType declined = mlirTypeConverterConvertType(declineConverter, i32);
+  (void)declined;
+  assert(declineCounter == 1 && "declining conversion must be consulted");
+  assert(mlirTypeEqual(declined, i64) &&
+         "decline must fall back to the i32 -> i64 conversion");
+  mlirTypeConverterDestroy(declineConverter);
+
+  // Failure: the i32 hard-failure is registered last (tried first) and must
+  // abort the conversion without falling back to the i32 -> i64 conversion, so
+  // the type converts to null.
+  MlirTypeConverter failConverter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(failConverter, widenI32ToI64, NULL);
+  intptr_t failCounter = 0;
+  mlirTypeConverterAddConversion(failConverter, failI32, &failCounter);
+  MlirType failed = mlirTypeConverterConvertType(failConverter, i32);
+  (void)failed;
+  assert(failCounter == 1 && "failing conversion must be consulted");
+  assert(mlirTypeIsNull(failed) &&
+         "hard failure must not fall back to another conversion");
+  mlirTypeConverterDestroy(failConverter);
+
+  // CHECK: testTypeConverterConversionStates: PASSED
+  fprintf(stderr, "testTypeConverterConversionStates: PASSED\n");
+}
+
 void testTypeConverterSourceMaterialization(MlirContext ctx) {
   // CHECK-LABEL: @testTypeConverterSourceMaterialization
   fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
@@ -1035,6 +1101,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterConversionStates(ctx);
   testTypeConverterSourceMaterialization(ctx);
   testTypeConverterTargetMaterialization(ctx);
 


From llvmlistbot at llvm.org  Sat Jul 18 11:52:01 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Sat, 18 Jul 2026 11:52:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
Message-ID: <llvm.org/llvm/llvm-project/pull/210533@github.com>


https://github.com/hamzaqureshi5 created https://github.com/llvm/llvm-project/pull/210533

Affine producer-consumer fusion places the fused nest at an insertion point between the source and the destination nest. When the source nest is retained after fusion (e.g., non-maximal fusion with an escaping memref) and updates a memref in place (loads from and stores to it), this placement is unsound: the fused nest recomputes the source's values reading from that memref, but the retained source nest has already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest, where the in-place updated memrefs still hold their original values. The recomputed stores are redirected to a private memref, so the retained source nest also still reads the original values. An anti-dependence edge from the destination to the source node is added to the dependence graph to preserve this placement in subsequent fusion decisions. When no such placement can be shown to preserve all dependences, or the in-place updated memref can't be privatized, bail out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g., initialize-then-reuse patterns) don't observe the memref's incoming state and don't require the new placement.

Update @same_memref_load_store and @same_memref_load_multiple_stores, whose CHECK lines encoded the miscompiled output; their fused nests are unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490

>From 55734d809499916583aac0c2f465f244911245ed Mon Sep 17 00:00:00 2001
From: Hamza Qureshi <hamza7771.861 at gmail.com>
Date: Sat, 18 Jul 2026 23:17:43 +0500
Subject: [PATCH] [MLIR][Affine] Fix fusion of producers that update a memref
 in place

Affine producer-consumer fusion places the fused nest at an insertion
point between the source and the destination nest. When the source nest
is retained after fusion (e.g., non-maximal fusion with an escaping
memref) and updates a memref in place (loads from and stores to it),
this placement is unsound: the fused nest recomputes the source's
values reading from that memref, but the retained source nest has
already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest,
where the in-place updated memrefs still hold their original values.
The recomputed stores are redirected to a private memref, so the
retained source nest also still reads the original values. An
anti-dependence edge from the destination to the source node is added
to the dependence graph to preserve this placement in subsequent fusion
decisions. When no such placement can be shown to preserve all
dependences, or the in-place updated memref can't be privatized, bail
out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g.,
initialize-then-reuse patterns) don't observe the memref's incoming
state and don't require the new placement.

Update @same_memref_load_store and @same_memref_load_multiple_stores,
whose CHECK lines encoded the miscompiled output; their fused nests are
unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490
---
 .../Dialect/Affine/Transforms/LoopFusion.cpp  | 140 ++++++++++++++++++
 mlir/test/Dialect/Affine/loop-fusion-4.mlir   |  87 ++++++++++-
 2 files changed, 221 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 


From llvmlistbot at llvm.org  Sat Jul 18 11:52:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:52:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5bcb66.6c5720b3.2da3b9.e338@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @hamzaqureshi5 :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/210533

From llvmlistbot at llvm.org  Sat Jul 18 11:52:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:52:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bcb7e.9edb5c9d.280be3.5089@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 4577 tests passed
* 206 tests skipped

All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.

<details>
<summary>tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o</summary>

```
FAILED: tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o
sccache /opt/llvm/bin/clang++ -DMLIRPythonModules_extension__mlirDialectsTransform_dso_EXPORTS -DMLIR_BINDINGS_PYTHON_DOMAIN=mlir -DMLIR_INCLUDE_TESTS -DMLIR_PYTHON_PACKAGE_PREFIX=mlir. -DNB_DOMAIN=mlir -DNB_SHARED -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/python -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/python -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -isystem /usr/include/python3.12 -isystem /home/gha/.local/lib/python3.12/site-packages/nanobind/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O3 -DNDEBUG -std=c++17 -fPIC -fvisibility=hidden -UNDEBUG -fno-stack-protector -Os -Wno-c++98-compat-extra-semi -Wno-cast-qual -Wno-covered-switch-default -Wno-deprecated-literal-operator -Wno-nested-anon-types -Wno-unused-parameter -Wno-zero-length-array -Wno-missing-field-initializers -frtti -fexceptions -MD -MT tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o -MF tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o.d -o tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp:178:5: error: no matching function for call to 'mlirTypeConverterAddConversion'
178 |     mlirTypeConverterAddConversion(
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/include/mlir-c/Rewrite.h:617:1: note: candidate function not viable: no known conversion from '(lambda at /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp:180:9)' to 'MlirTypeConverterConversionCallback' (aka 'enum MlirTypeConverterConversionStatus (*)(struct MlirType, struct MlirType *, void *)') for 2nd argument
617 | mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
| ^
618 |                                MlirTypeConverterConversionCallback convertType,
|                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
</details>
<details>
<summary>tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o</summary>

```
FAILED: tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o
sccache /opt/llvm/bin/clang++ -DMLIRPythonModules_extension__mlir_dso_EXPORTS -DMLIR_BINDINGS_PYTHON_DOMAIN=mlir -DMLIR_INCLUDE_TESTS -DMLIR_PYTHON_PACKAGE_PREFIX=mlir. -DNB_DOMAIN=mlir -DNB_SHARED -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/python -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/python -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -isystem /usr/include/python3.12 -isystem /home/gha/.local/lib/python3.12/site-packages/nanobind/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O3 -DNDEBUG -std=c++17 -fPIC -fvisibility=hidden -UNDEBUG -fno-stack-protector -Os -Wno-c++98-compat-extra-semi -Wno-cast-qual -Wno-covered-switch-default -Wno-deprecated-literal-operator -Wno-nested-anon-types -Wno-unused-parameter -Wno-zero-length-array -Wno-missing-field-initializers -frtti -fexceptions -MD -MT tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o -MF tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o.d -o tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp:178:5: error: no matching function for call to 'mlirTypeConverterAddConversion'
178 |     mlirTypeConverterAddConversion(
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/include/mlir-c/Rewrite.h:617:1: note: candidate function not viable: no known conversion from '(lambda at /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp:180:9)' to 'MlirTypeConverterConversionCallback' (aka 'enum MlirTypeConverterConversionStatus (*)(struct MlirType, struct MlirType *, void *)') for 2nd argument
617 | mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
| ^
618 |                                MlirTypeConverterConversionCallback convertType,
|                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 18 11:52:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:52:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bcb7f.13be6df8.22eac4.ae94@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

The build failed before running any tests. Click on a failure below to see the details.

<details>
<summary>[code=1] tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/C_/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.obj</summary>

```
FAILED: [code=1] tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlir.dso.dir/C_/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.obj
sccache C:\clang\clang-msvc\bin\clang-cl.exe  /nologo -TP -DMLIRPythonModules_extension__mlir_dso_EXPORTS -DMLIR_BINDINGS_PYTHON_DOMAIN=mlir -DMLIR_INCLUDE_TESTS -DMLIR_PYTHON_PACKAGE_PREFIX=mlir. -DNB_DOMAIN=mlir -DNB_SHARED -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GLIBCXX_ASSERTIONS -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\_work\llvm-project\llvm-project\build\tools\mlir\python -IC:\_work\llvm-project\llvm-project\mlir\python -IC:\_work\llvm-project\llvm-project\build\tools\mlir\include -IC:\_work\llvm-project\llvm-project\mlir\include -IC:\_work\llvm-project\llvm-project\build\include -IC:\_work\llvm-project\llvm-project\llvm\include -imsvcC:\Python312\include -imsvcC:\Python312\Lib\site-packages\nanobind\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /Brepro /bigobj /permissive- -Werror=unguarded-availability-new /W4  -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported /Gw -Wundef -Werror=mismatched-tags /O2 /Ob2  -std:c++17 -MD -UNDEBUG /bigobj /MP /Os -Wno-c++98-compat-extra-semi -Wno-cast-qual -Wno-covered-switch-default -Wno-deprecated-literal-operator -Wno-nested-anon-types -Wno-unused-parameter -Wno-zero-length-array -Wno-missing-field-initializers /EHsc /GR /showIncludes /Fotools\mlir\python\CMakeFiles\MLIRPythonModules.extension._mlir.dso.dir\C_\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp.obj /Fdtools\mlir\python\CMakeFiles\MLIRPythonModules.extension._mlir.dso.dir\ -c -- C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp
C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp(178,5): error: no matching function for call to 'mlirTypeConverterAddConversion'
178 |     mlirTypeConverterAddConversion(
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\_work\llvm-project\llvm-project\mlir\include\mlir-c/Rewrite.h(617,1): note: candidate function not viable: no known conversion from '(lambda at C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp:180:9)' to 'MlirTypeConverterConversionCallback' (aka 'enum MlirTypeConverterConversionStatus (*)(struct MlirType, struct MlirType *, void *)') for 2nd argument
617 | mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
| ^
618 |                                MlirTypeConverterConversionCallback convertType,
|                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
</details>
<details>
<summary>[code=1] tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/C_/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.obj</summary>

```
FAILED: [code=1] tools/mlir/python/CMakeFiles/MLIRPythonModules.extension._mlirDialectsTransform.dso.dir/C_/_work/llvm-project/llvm-project/mlir/lib/Bindings/Python/Rewrite.cpp.obj
sccache C:\clang\clang-msvc\bin\clang-cl.exe  /nologo -TP -DMLIRPythonModules_extension__mlirDialectsTransform_dso_EXPORTS -DMLIR_BINDINGS_PYTHON_DOMAIN=mlir -DMLIR_INCLUDE_TESTS -DMLIR_PYTHON_PACKAGE_PREFIX=mlir. -DNB_DOMAIN=mlir -DNB_SHARED -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GLIBCXX_ASSERTIONS -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\_work\llvm-project\llvm-project\build\tools\mlir\python -IC:\_work\llvm-project\llvm-project\mlir\python -IC:\_work\llvm-project\llvm-project\build\tools\mlir\include -IC:\_work\llvm-project\llvm-project\mlir\include -IC:\_work\llvm-project\llvm-project\build\include -IC:\_work\llvm-project\llvm-project\llvm\include -imsvcC:\Python312\include -imsvcC:\Python312\Lib\site-packages\nanobind\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /Brepro /bigobj /permissive- -Werror=unguarded-availability-new /W4  -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported /Gw -Wundef -Werror=mismatched-tags /O2 /Ob2  -std:c++17 -MD -UNDEBUG /bigobj /MP /Os -Wno-c++98-compat-extra-semi -Wno-cast-qual -Wno-covered-switch-default -Wno-deprecated-literal-operator -Wno-nested-anon-types -Wno-unused-parameter -Wno-zero-length-array -Wno-missing-field-initializers /EHsc /GR /showIncludes /Fotools\mlir\python\CMakeFiles\MLIRPythonModules.extension._mlirDialectsTransform.dso.dir\C_\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp.obj /Fdtools\mlir\python\CMakeFiles\MLIRPythonModules.extension._mlirDialectsTransform.dso.dir\ -c -- C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp
C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp(178,5): error: no matching function for call to 'mlirTypeConverterAddConversion'
178 |     mlirTypeConverterAddConversion(
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\_work\llvm-project\llvm-project\mlir\include\mlir-c/Rewrite.h(617,1): note: candidate function not viable: no known conversion from '(lambda at C:\_work\llvm-project\llvm-project\mlir\lib\Bindings\Python\Rewrite.cpp:180:9)' to 'MlirTypeConverterConversionCallback' (aka 'enum MlirTypeConverterConversionStatus (*)(struct MlirType, struct MlirType *, void *)') for 2nd argument
617 | mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
| ^
618 |                                MlirTypeConverterConversionCallback convertType,
|                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 18 11:52:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:52:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5bcb87.f1c1923d.2d2b59.346e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-affine

Author: Hamza Qureshi (hamzaqureshi5)

<details>
<summary>Changes</summary>

Affine producer-consumer fusion places the fused nest at an insertion point between the source and the destination nest. When the source nest is retained after fusion (e.g., non-maximal fusion with an escaping memref) and updates a memref in place (loads from and stores to it), this placement is unsound: the fused nest recomputes the source's values reading from that memref, but the retained source nest has already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest, where the in-place updated memrefs still hold their original values. The recomputed stores are redirected to a private memref, so the retained source nest also still reads the original values. An anti-dependence edge from the destination to the source node is added to the dependence graph to preserve this placement in subsequent fusion decisions. When no such placement can be shown to preserve all dependences, or the in-place updated memref can't be privatized, bail out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g., initialize-then-reuse patterns) don't observe the memref's incoming state and don't require the new placement.

Update @<!-- -->same_memref_load_store and @<!-- -->same_memref_load_multiple_stores, whose CHECK lines encoded the miscompiled output; their fused nests are unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490

---
Full diff: https://github.com/llvm/llvm-project/pull/210533.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp (+140) 
- (modified) mlir/test/Dialect/Affine/loop-fusion-4.mlir (+81-6) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/210533

From llvmlistbot at llvm.org  Sat Jul 18 11:52:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 11:52:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5bcb88.0c05e68e.3c2f5d.5698@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Hamza Qureshi (hamzaqureshi5)

<details>
<summary>Changes</summary>

Affine producer-consumer fusion places the fused nest at an insertion point between the source and the destination nest. When the source nest is retained after fusion (e.g., non-maximal fusion with an escaping memref) and updates a memref in place (loads from and stores to it), this placement is unsound: the fused nest recomputes the source's values reading from that memref, but the retained source nest has already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest, where the in-place updated memrefs still hold their original values. The recomputed stores are redirected to a private memref, so the retained source nest also still reads the original values. An anti-dependence edge from the destination to the source node is added to the dependence graph to preserve this placement in subsequent fusion decisions. When no such placement can be shown to preserve all dependences, or the in-place updated memref can't be privatized, bail out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g., initialize-then-reuse patterns) don't observe the memref's incoming state and don't require the new placement.

Update @<!-- -->same_memref_load_store and @<!-- -->same_memref_load_multiple_stores, whose CHECK lines encoded the miscompiled output; their fused nests are unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490

---
Full diff: https://github.com/llvm/llvm-project/pull/210533.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp (+140) 
- (modified) mlir/test/Dialect/Affine/loop-fusion-4.mlir (+81-6) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/210533

From llvmlistbot at llvm.org  Sat Jul 18 11:56:16 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 18 Jul 2026 11:56:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bcc50.f06f4994.2f0620.e45e@mx.google.com>


https://github.com/makslevental updated https://github.com/llvm/llvm-project/pull/208934

>From 496a4a2d6b35cb97e7493d1d1612e6cb9e7b6781 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:57:46 -0700
Subject: [PATCH 1/4] [mlir-c] Add TypeConverter source materialization

---
 mlir/include/mlir-c/Rewrite.h        |  16 ++++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  32 +++++++
 mlir/test/CAPI/rewrite.c             | 125 +++++++++++++++++++++++++++
 3 files changed, 173 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 3356e6f445e47..2d7ed3e4b7111 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -605,6 +605,22 @@ mlirTypeConverterAddConversion(MlirTypeConverter typeConverter,
 MLIR_CAPI_EXPORTED MlirType
 mlirTypeConverterConvertType(MlirTypeConverter typeConverter, MlirType type);
 
+/// Callback type for type materializations. Given a builder (passed as a
+/// rewriter), the desired output type, the input values, and a location, the
+/// callback must build a cast-like operation that produces a single value of
+/// `outputType` and return it. Returning a null MlirValue indicates failure, in
+/// which case another registered materialization may be attempted.
+typedef MlirValue (*MlirTypeConverterMaterializationCallback)(
+    MlirRewriterBase rewriter, MlirType outputType, intptr_t nInputs,
+    MlirValue *inputs, MlirLocation loc, void *userData);
+
+/// Register a source materialization with the given TypeConverter. This is
+/// invoked when a replacement value must be converted back to its original
+/// source type because some uses persist beyond the main conversion.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 083ed6f999ae3..2fb8eff5ad79d 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -669,6 +669,38 @@ MlirType mlirTypeConverterConvertType(MlirTypeConverter typeConverter,
   return wrap(unwrap(typeConverter)->convertType(unwrap(type)));
 }
 
+namespace {
+/// Wraps a C materialization callback as a C++ materialization callback of the
+/// form `Value(OpBuilder &, Type, ValueRange, Location)`, shared by both source
+/// and target materializations. The builder is always a RewriterBase in the
+/// conversion driver, so it is safe to expose it as an MlirRewriterBase.
+std::function<Value(OpBuilder &, Type, ValueRange, Location)>
+wrapMaterializationCallback(MlirTypeConverterMaterializationCallback callback,
+                            void *userData) {
+  return [callback, userData](OpBuilder &builder, Type type, ValueRange inputs,
+                              Location loc) -> Value {
+    SmallVector<MlirValue> wrappedInputs;
+    wrappedInputs.reserve(inputs.size());
+    for (Value v : inputs)
+      wrappedInputs.push_back(wrap(v));
+    MlirValue result =
+        callback(wrap(static_cast<RewriterBase *>(&builder)), wrap(type),
+                 static_cast<intptr_t>(wrappedInputs.size()),
+                 wrappedInputs.data(), wrap(loc), userData);
+    return mlirValueIsNull(result) ? Value() : unwrap(result);
+  };
+}
+} // namespace
+
+void mlirTypeConverterAddSourceMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addSourceMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 439d1355af822..8e5b98abdb883 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -802,6 +802,130 @@ void testConversionTargetDynamicLegality(MlirContext ctx) {
   fprintf(stderr, "testConversionTargetDynamicLegality: PASSED\n");
 }
 
+// Type conversion callback: maps i32 -> i64 and leaves every other type
+// unchanged (identity). Used by the materialization tests below.
+static MlirLogicalResult widenI32ToI64(MlirType type, MlirType *result,
+                                       void *userData) {
+  (void)userData;
+  if (mlirTypeIsAInteger(type) && mlirIntegerTypeGetWidth(type) == 32)
+    *result = mlirIntegerTypeGet(mlirTypeGetContext(type), 64);
+  else
+    *result = type;
+  return mlirLogicalResultSuccess();
+}
+
+// Materialization callback: builds a `test.cast` op that produces a single
+// value of `outputType` from the given inputs, and records that it ran by
+// bumping the counter passed as userData.
+static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
+                                          MlirType outputType, intptr_t nInputs,
+                                          MlirValue *inputs, MlirLocation loc,
+                                          void *userData) {
+  intptr_t *counter = (intptr_t *)userData;
+  if (counter)
+    (*counter)++;
+  MlirOperationState state =
+      mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
+  mlirOperationStateAddOperands(&state, nInputs, inputs);
+  mlirOperationStateAddResults(&state, 1, &outputType);
+  MlirOperation castOp = mlirOperationCreate(&state);
+  mlirRewriterBaseInsert(rewriter, castOp);
+  return mlirOperationGetResult(castOp, 0);
+}
+
+// Conversion pattern for `test.source`: replaces it with a `test.source_i64`
+// op whose result has the widened (i64) type. Because the original result type
+// (i32) differs from the replacement type (i64), persisting uses force the
+// framework to insert a source materialization.
+static MlirLogicalResult convertSource(MlirConversionPattern pattern,
+                                       MlirOperation op, intptr_t nOperands,
+                                       MlirValue *operands,
+                                       MlirConversionPatternRewriter rewriter,
+                                       void *userData) {
+  (void)pattern;
+  (void)nOperands;
+  (void)operands;
+  (void)userData;
+  MlirContext ctx = mlirOperationGetContext(op);
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirType i64 = mlirIntegerTypeGet(ctx, 64);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.source_i64"), loc);
+  mlirOperationStateAddResults(&state, 1, &i64);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  MlirValue newVal = mlirOperationGetResult(newOp, 0);
+  mlirRewriterBaseReplaceOpWithValues(base, op, 1, &newVal);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterSourceMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterSourceMaterialization
+  fprintf(stderr, "@testTypeConverterSourceMaterialization\n");
+
+  // `test.source` produces an i32 that is consumed by the (legal) `test.user`.
+  // Converting `test.source` to an i64-producing op leaves `test.user` wanting
+  // the original i32, which triggers a source materialization back to i32.
+  const char *moduleString = "%0 = \"test.source\"() : () -> i32\n"
+                             "\"test.user\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddSourceMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertSource};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.source"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.source"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.source_i64"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.user"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "source materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterSourceMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -818,6 +942,7 @@ int main(void) {
   testGreedyRewriteDriverConfig(ctx);
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
+  testTypeConverterSourceMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 12609605057480049675bea96fc6de538c16141d Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Mon, 29 Jun 2026 09:59:12 -0700
Subject: [PATCH 2/4] [mlir-c] Add TypeConverter target materialization

---
 mlir/include/mlir-c/Rewrite.h        |  7 +++
 mlir/lib/CAPI/Transforms/Rewrite.cpp |  9 +++
 mlir/test/CAPI/rewrite.c             | 90 ++++++++++++++++++++++++++++
 3 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir-c/Rewrite.h b/mlir/include/mlir-c/Rewrite.h
index 2d7ed3e4b7111..e074fc96f2aeb 100644
--- a/mlir/include/mlir-c/Rewrite.h
+++ b/mlir/include/mlir-c/Rewrite.h
@@ -621,6 +621,13 @@ MLIR_CAPI_EXPORTED void mlirTypeConverterAddSourceMaterialization(
     MlirTypeConverter typeConverter,
     MlirTypeConverterMaterializationCallback callback, void *userData);
 
+/// Register a target materialization with the given TypeConverter. This is
+/// invoked when a value must be converted to a target type according to a
+/// pattern's type converter.
+MLIR_CAPI_EXPORTED void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData);
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/CAPI/Transforms/Rewrite.cpp b/mlir/lib/CAPI/Transforms/Rewrite.cpp
index 2fb8eff5ad79d..92456d6d8a435 100644
--- a/mlir/lib/CAPI/Transforms/Rewrite.cpp
+++ b/mlir/lib/CAPI/Transforms/Rewrite.cpp
@@ -701,6 +701,15 @@ void mlirTypeConverterAddSourceMaterialization(
           wrapMaterializationCallback(callback, userData));
 }
 
+void mlirTypeConverterAddTargetMaterialization(
+    MlirTypeConverter typeConverter,
+    MlirTypeConverterMaterializationCallback callback, void *userData) {
+  assert(callback && "expected non-null materialization callback");
+  unwrap(typeConverter)
+      ->addTargetMaterialization(
+          wrapMaterializationCallback(callback, userData));
+}
+
 //===----------------------------------------------------------------------===//
 /// ConversionPattern API
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 8e5b98abdb883..c1624c54eb461 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -926,6 +926,95 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
   fprintf(stderr, "testTypeConverterSourceMaterialization: PASSED\n");
 }
 
+// Conversion pattern for `test.consumer`: replaces it with a
+// `test.consumer_legal` op that consumes the (already remapped) operands. The
+// operand of the original op has type i32 but its producer is not converted, so
+// the framework inserts a target materialization to i64 before invoking this
+// pattern -- the remapped `operands` are therefore the i64 cast results.
+static MlirLogicalResult convertConsumer(MlirConversionPattern pattern,
+                                         MlirOperation op, intptr_t nOperands,
+                                         MlirValue *operands,
+                                         MlirConversionPatternRewriter rewriter,
+                                         void *userData) {
+  (void)pattern;
+  (void)userData;
+  MlirLocation loc = mlirOperationGetLocation(op);
+  MlirOperationState state = mlirOperationStateGet(
+      mlirStringRefCreateFromCString("test.consumer_legal"), loc);
+  mlirOperationStateAddOperands(&state, nOperands, operands);
+  MlirOperation newOp = mlirOperationCreate(&state);
+
+  MlirRewriterBase base = mlirPatternRewriterAsBase(
+      mlirConversionPatternRewriterAsPatternRewriter(rewriter));
+  mlirRewriterBaseInsert(base, newOp);
+  mlirRewriterBaseEraseOp(base, op);
+  return mlirLogicalResultSuccess();
+}
+
+void testTypeConverterTargetMaterialization(MlirContext ctx) {
+  // CHECK-LABEL: @testTypeConverterTargetMaterialization
+  fprintf(stderr, "@testTypeConverterTargetMaterialization\n");
+
+  // `test.consumer` takes an i32 from the (legal, unconverted) `test.producer`.
+  // Converting `test.consumer` requires its operand as i64, which triggers a
+  // target materialization from i32 to i64.
+  const char *moduleString = "%0 = \"test.producer\"() : () -> i32\n"
+                             "\"test.consumer\"(%0) : (i32) -> ()\n";
+  MlirModule module =
+      mlirModuleCreateParse(ctx, mlirStringRefCreateFromCString(moduleString));
+  MlirOperation moduleOp = mlirModuleGetOperation(module);
+
+  MlirTypeConverter converter = mlirTypeConverterCreate();
+  mlirTypeConverterAddConversion(converter, widenI32ToI64, NULL);
+  intptr_t materializationCounter = 0;
+  mlirTypeConverterAddTargetMaterialization(converter, buildCastMaterialization,
+                                            &materializationCounter);
+
+  MlirRewritePatternSet patterns = mlirRewritePatternSetCreate(ctx);
+  MlirConversionPatternCallbacks callbacks = {NULL, NULL, convertConsumer};
+  MlirConversionPattern pattern = mlirOpConversionPatternCreate(
+      mlirStringRefCreateFromCString("test.consumer"), 1, ctx, converter,
+      callbacks, NULL, 0, NULL);
+  mlirRewritePatternSetAdd(patterns,
+                           mlirConversionPatternAsRewritePattern(pattern));
+  MlirFrozenRewritePatternSet frozen = mlirFreezeRewritePattern(patterns);
+
+  MlirConversionTarget target = mlirConversionTargetCreate(ctx);
+  mlirConversionTargetAddIllegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.producer"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("test.consumer_legal"));
+  mlirConversionTargetAddLegalOp(target,
+                                 mlirStringRefCreateFromCString("test.cast"));
+  mlirConversionTargetAddLegalOp(
+      target, mlirStringRefCreateFromCString("builtin.module"));
+
+  MlirConversionConfig config = mlirConversionConfigCreate();
+  MlirLogicalResult result =
+      mlirApplyPartialConversion(moduleOp, target, frozen, config);
+  assert(mlirLogicalResultIsSuccess(result));
+  assert(materializationCounter > 0 &&
+         "target materialization callback must be invoked");
+
+  mlirOperationDump(moduleOp);
+  // clang-format off
+  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // clang-format on
+
+  mlirConversionConfigDestroy(config);
+  mlirConversionTargetDestroy(target);
+  mlirFrozenRewritePatternSetDestroy(frozen);
+  mlirTypeConverterDestroy(converter);
+  mlirModuleDestroy(module);
+
+  // CHECK: testTypeConverterTargetMaterialization: PASSED
+  fprintf(stderr, "testTypeConverterTargetMaterialization: PASSED\n");
+}
+
 int main(void) {
   MlirContext ctx = mlirContextCreate();
   mlirContextSetAllowUnregisteredDialects(ctx, true);
@@ -943,6 +1032,7 @@ int main(void) {
   testCloneWithMapping(ctx);
   testConversionTargetDynamicLegality(ctx);
   testTypeConverterSourceMaterialization(ctx);
+  testTypeConverterTargetMaterialization(ctx);
 
   mlirContextDestroy(ctx);
   return 0;

>From 7fb9b326d964f82016e7553d911d4c479798e1f4 Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Sat, 11 Jul 2026 11:55:10 -0700
Subject: [PATCH 3/4] [mlir-c] Tighten materialization test CHECKs to pin full
 module body

Use CHECK-NEXT to match the entire module body (module { ... }) rather
than loose CHECK lines, so the tests also assert the absence of any
stray ops -- e.g. a leftover builtin.unrealized_conversion_cast -- that
a loose CHECK would silently allow.
---
 mlir/test/CAPI/rewrite.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index c1624c54eb461..6d87d0236061b 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -911,9 +911,11 @@ void testTypeConverterSourceMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.source_i64"() : () -> i64
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
-  // CHECK: "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.source_i64"() : () -> i64
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i64) -> i32
+  // CHECK-NEXT:   "test.user"(%[[c]]) : (i32) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);
@@ -1000,9 +1002,11 @@ void testTypeConverterTargetMaterialization(MlirContext ctx) {
 
   mlirOperationDump(moduleOp);
   // clang-format off
-  // CHECK: %[[v:.*]] = "test.producer"() : () -> i32
-  // CHECK: %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
-  // CHECK: "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK:      module {
+  // CHECK-NEXT:   %[[v:.*]] = "test.producer"() : () -> i32
+  // CHECK-NEXT:   %[[c:.*]] = "test.cast"(%[[v]]) : (i32) -> i64
+  // CHECK-NEXT:   "test.consumer_legal"(%[[c]]) : (i64) -> ()
+  // CHECK-NEXT: }
   // clang-format on
 
   mlirConversionConfigDestroy(config);

>From 0747c994b3badf2322f7af5362ceb9556e3f42ec Mon Sep 17 00:00:00 2001
From: makslevental <maksim.levental at gmail.com>
Date: Fri, 17 Jul 2026 12:22:08 -0700
Subject: [PATCH 4/4] address comment

---
 mlir/test/CAPI/rewrite.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/CAPI/rewrite.c b/mlir/test/CAPI/rewrite.c
index 6d87d0236061b..b10afbd8b812c 100644
--- a/mlir/test/CAPI/rewrite.c
+++ b/mlir/test/CAPI/rewrite.c
@@ -627,7 +627,7 @@ static MlirConversionTargetLegality dynamicLegalityAlwaysLegal(MlirOperation op,
                                                                void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_LEGAL;
 }
 
@@ -635,7 +635,7 @@ static MlirConversionTargetLegality
 dynamicLegalityAlwaysIllegal(MlirOperation op, void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_ILLEGAL;
 }
 
@@ -643,7 +643,7 @@ static MlirConversionTargetLegality dynamicLegalityNoOpinion(MlirOperation op,
                                                              void *userData) {
   (void)op;
   intptr_t *counter = (intptr_t *)userData;
-  (*counter)++;
+  ++(*counter);
   return MLIR_CONVERSION_TARGET_LEGALITY_NO_OPINION;
 }
 
@@ -823,7 +823,7 @@ static MlirValue buildCastMaterialization(MlirRewriterBase rewriter,
                                           void *userData) {
   intptr_t *counter = (intptr_t *)userData;
   if (counter)
-    (*counter)++;
+    ++(*counter);
   MlirOperationState state =
       mlirOperationStateGet(mlirStringRefCreateFromCString("test.cast"), loc);
   mlirOperationStateAddOperands(&state, nInputs, inputs);


From llvmlistbot at llvm.org  Sat Jul 18 12:00:04 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sat, 18 Jul 2026 12:00:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter source and target
 materialization (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5bcd34.a1accbb4.5eca2.05b1@mx.google.com>


================

----------------
makslevental wrote:

done (including other instances of post increment)

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Sat Jul 18 12:36:28 2026
From: llvmlistbot at llvm.org (Jared Hoberock)
Date: Sat, 18 Jul 2026 12:36:28 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][IR] Add SymbolUserTypeInterface
	(PR #198435)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198435@github.com>
Message-ID: <6a5bd5bc.13be6df8.22eac4.4bbc@mx.google.com>


jaredhoberock wrote:

@joker-eph Thank you! Is this PR too late for the LLVM 23 release?

https://github.com/llvm/llvm-project/pull/198435

From llvmlistbot at llvm.org  Sat Jul 18 13:01:48 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 13:01:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu] Add VectorToXeGPU and adjust
 VectorToSCF placement in XeVM pipeline (PR #210539)
Message-ID: <llvm.org/llvm/llvm-project/pull/210539@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/210539

 Add convert-vector-to-xegpu (nested under gpu.module) to lower kernel-side vector ops to XeGPU.
    
Move convert-vector-to-scf from the pre-GPU common pipeline into the post-GPU pipeline, so host transfer ops (and any SCF the pass emits) are lowered to loops before the vector-to-LLVM conversion.


>From 2a734232f69e3588ec392626d09c8c0910c818ce Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 19:56:27 +0000
Subject: [PATCH] [mlir][gpu] Scope VectorToXeGPU to gpu.module and move
 VectorToSCF to host path

Nest convert-vector-to-xegpu under gpu::GPUModuleOp so it no longer rewrites
host-side vector.transfer ops into xegpu.load/store. Move convert-vector-to-scf
out of the pre-GPU common pipeline and into the post-GPU pipeline so remaining
host transfer ops (and any SCF the pass emits) are lowered before the
vector-to-LLVM conversion.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt        | 1 +
 mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
index 293166cbf8b83..3a0cd7ef5c39d 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
+++ b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
@@ -30,6 +30,7 @@ add_mlir_dialect_library(MLIRGPUPipelines
   MLIRSCFToControlFlow
   MLIRVectorToLLVMPass
   MLIRVectorToSCF
+  MLIRVectorToXeGPU
   MLIRXeGPUTransforms
   MLIRXeGPUToXeVM
   MLIRXeVMToLLVM
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
index 1cbf3bd633633..d0bab5cd74220 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
@@ -44,7 +44,6 @@ void buildPreGPUCommonPassPipeline(
     OpPassManager &pm, const mlir::gpu::GPUToXeVMPipelineOptions &options) {
   // builtin.module scope passes.
   pm.addPass(createCSEPass());
-  pm.addPass(createConvertVectorToSCFPass());
   {
     GpuXeVMAttachTargetOptions xevmTargetOptions;
     xevmTargetOptions.moduleMatcher = options.xevmModuleMatcher;
@@ -67,6 +66,7 @@ void buildGPUPassPipeline(OpPassManager &pm,
   laneLayoutOptions.indexBitWidth = options.use64bitIndex ? 64 : 32;
   laneLayoutOptions.layoutKind = "lane";
   pm.addNestedPass<ModuleOp>(createCSEPass());
+  pm.addNestedPass<gpu::GPUModuleOp>(createConvertVectorToXeGPU());
   if (options.xegpuOpLevel == "workgroup") {
     xegpu::XeGPUPropagateLayoutOptions sgLayoutOptions;
     sgLayoutOptions.layoutKind = "subgroup";
@@ -154,6 +154,7 @@ void buildGPUPassPipeline(OpPassManager &pm,
 void buildPostGPUCommonPassPipeline(
     OpPassManager &pm, const mlir::gpu::GPUToXeVMPipelineOptions &options) {
   // builtin.module scope passes.
+  pm.addPass(createConvertVectorToSCFPass());
   pm.addPass(createSCFToControlFlowPass());
   pm.addPass(memref::createExpandStridedMetadataPass());
   {


From llvmlistbot at llvm.org  Sat Jul 18 13:02:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 13:02:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu] Add VectorToXeGPU and adjust
 VectorToSCF placement in XeVM pipeline (PR #210539)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210539@github.com>
Message-ID: <6a5bdbcd.743400b0.23a7b2.8365@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

 Add convert-vector-to-xegpu (nested under gpu.module) to lower kernel-side vector ops to XeGPU.
    
Move convert-vector-to-scf from the pre-GPU common pipeline into the post-GPU pipeline, so host transfer ops (and any SCF the pass emits) are lowered to loops before the vector-to-LLVM conversion.


---
Full diff: https://github.com/llvm/llvm-project/pull/210539.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt (+1) 
- (modified) mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp (+2-1) 


``````````diff
diff --git a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
index 293166cbf8b83..3a0cd7ef5c39d 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
+++ b/mlir/lib/Dialect/GPU/Pipelines/CMakeLists.txt
@@ -30,6 +30,7 @@ add_mlir_dialect_library(MLIRGPUPipelines
   MLIRSCFToControlFlow
   MLIRVectorToLLVMPass
   MLIRVectorToSCF
+  MLIRVectorToXeGPU
   MLIRXeGPUTransforms
   MLIRXeGPUToXeVM
   MLIRXeVMToLLVM
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
index 1cbf3bd633633..d0bab5cd74220 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
@@ -44,7 +44,6 @@ void buildPreGPUCommonPassPipeline(
     OpPassManager &pm, const mlir::gpu::GPUToXeVMPipelineOptions &options) {
   // builtin.module scope passes.
   pm.addPass(createCSEPass());
-  pm.addPass(createConvertVectorToSCFPass());
   {
     GpuXeVMAttachTargetOptions xevmTargetOptions;
     xevmTargetOptions.moduleMatcher = options.xevmModuleMatcher;
@@ -67,6 +66,7 @@ void buildGPUPassPipeline(OpPassManager &pm,
   laneLayoutOptions.indexBitWidth = options.use64bitIndex ? 64 : 32;
   laneLayoutOptions.layoutKind = "lane";
   pm.addNestedPass<ModuleOp>(createCSEPass());
+  pm.addNestedPass<gpu::GPUModuleOp>(createConvertVectorToXeGPU());
   if (options.xegpuOpLevel == "workgroup") {
     xegpu::XeGPUPropagateLayoutOptions sgLayoutOptions;
     sgLayoutOptions.layoutKind = "subgroup";
@@ -154,6 +154,7 @@ void buildGPUPassPipeline(OpPassManager &pm,
 void buildPostGPUCommonPassPipeline(
     OpPassManager &pm, const mlir::gpu::GPUToXeVMPipelineOptions &options) {
   // builtin.module scope passes.
+  pm.addPass(createConvertVectorToSCFPass());
   pm.addPass(createSCFToControlFlowPass());
   pm.addPass(memref::createExpandStridedMetadataPass());
   {

``````````

</details>


https://github.com/llvm/llvm-project/pull/210539

From llvmlistbot at llvm.org  Sat Jul 18 13:29:03 2026
From: llvmlistbot at llvm.org (Ryan Thomas Lynch)
Date: Sat, 18 Jul 2026 13:29:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
Message-ID: <llvm.org/llvm/llvm-project/pull/210542@github.com>


https://github.com/emosy created https://github.com/llvm/llvm-project/pull/210542

similar to https://github.com/llvm/llvm-project/pull/146839

useful for cases such as being able to programmatically update the result segment sizes or clone an operation via `OperationState` while editing just a few results, programmatically

>From 6ee0b42d88567dad41cd3717dff39f931f59b227 Mon Sep 17 00:00:00 2001
From: Ryan Thomas Lynch <me at ryanlyn.ch>
Date: Sat, 18 Jul 2026 13:27:08 -0700
Subject: [PATCH] [mlir-tblgen] Emit named result indices

similar to https://github.com/llvm/llvm-project/pull/146839

useful for cases such as being able to programmatically update the
result segment sizes or clone an operation via `OperationState` while
editing just a few results, programmatically
---
 mlir/test/mlir-tblgen/op-result.td          | 11 +++++++++++
 mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp | 14 +++++++++++++-
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/mlir/test/mlir-tblgen/op-result.td b/mlir/test/mlir-tblgen/op-result.td
index a4f7af6dbcf1c..ddc1ef077d31c 100644
--- a/mlir/test/mlir-tblgen/op-result.td
+++ b/mlir/test/mlir-tblgen/op-result.td
@@ -14,6 +14,9 @@ def OpA : NS_Op<"one_normal_result_op", []> {
   let results = (outs I32:$result);
 }
 
+// DECL-LABEL: class OpA : {{.*}} {
+// DECL: static constexpr int odsIndex_result = 0;
+
 // CHECK-LABEL: void OpA::build
 // CHECK:         ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands
 // CHECK:         assert(resultTypes.size() == 1u && "mismatched number of return types");
@@ -24,6 +27,10 @@ def OpB : NS_Op<"same_input_output_type_op", [SameOperandsAndResultType]> {
   let results = (outs I32:$y);
 }
 
+// DECL-LABEL: class OpB : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_y = 0;
+
 // CHECK-LABEL: OpB definitions
 // CHECK: void OpB::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type y, ::mlir::Value x)
 // CHECK:   odsState.addTypes(y);
@@ -39,6 +46,10 @@ def OpC : NS_Op<"three_normal_result_op", []> {
   let results = (outs I32:$x, /*unnamed*/I32, I32:$z);
 }
 
+// DECL-LABEL: class OpC : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_z = 2;
+
 // CHECK-LABEL: OpC definitions
 // CHECK:       void OpC::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type x, ::mlir::Type resultType1, ::mlir::Type z)
 // CHECK-NEXT:   odsState.addTypes(x)
diff --git a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
index deef178e68dc1..4614459abb2b6 100644
--- a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
@@ -2175,7 +2175,8 @@ generateNamedOperandGetters(const Operator &op, Class &opClass,
                     "'SameVariadicOperandSize' traits");
   }
 
-  // Print the ods names so they don't need to be hardcoded in the source.
+  // Print the ODS indices of operands so they don't need to be hardcoded in the
+  // source.
   for (int i = 0; i != numOperands; ++i) {
     const auto &operand = op.getOperand(i);
     if (operand.name.empty())
@@ -2389,6 +2390,17 @@ void OpEmitter::genNamedResultGetters() {
                     "'SameVariadicResultSize' traits");
   }
 
+  // Print the ODS indices of results so they don't need to be hardcoded in the
+  // source.
+  for (int i = 0; i != numResults; ++i) {
+    const auto &result = op.getResult(i);
+    if (result.name.empty())
+      continue;
+
+    opClass.declare<Field>("static constexpr int",
+                           Twine("odsIndex_") + result.name + " = " + Twine(i));
+  }
+
   // Build the initializer string for the result segment size attribute.
   std::string attrSizeInitCode;
   if (attrSizedResults) {


From llvmlistbot at llvm.org  Sat Jul 18 13:29:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 13:29:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5be233.507fc713.239434.1397@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: Ryan Thomas Lynch (emosy)

<details>
<summary>Changes</summary>

similar to https://github.com/llvm/llvm-project/pull/146839

useful for cases such as being able to programmatically update the result segment sizes or clone an operation via `OperationState` while editing just a few results, programmatically

---
Full diff: https://github.com/llvm/llvm-project/pull/210542.diff


2 Files Affected:

- (modified) mlir/test/mlir-tblgen/op-result.td (+11) 
- (modified) mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp (+13-1) 


``````````diff
diff --git a/mlir/test/mlir-tblgen/op-result.td b/mlir/test/mlir-tblgen/op-result.td
index a4f7af6dbcf1c..ddc1ef077d31c 100644
--- a/mlir/test/mlir-tblgen/op-result.td
+++ b/mlir/test/mlir-tblgen/op-result.td
@@ -14,6 +14,9 @@ def OpA : NS_Op<"one_normal_result_op", []> {
   let results = (outs I32:$result);
 }
 
+// DECL-LABEL: class OpA : {{.*}} {
+// DECL: static constexpr int odsIndex_result = 0;
+
 // CHECK-LABEL: void OpA::build
 // CHECK:         ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands
 // CHECK:         assert(resultTypes.size() == 1u && "mismatched number of return types");
@@ -24,6 +27,10 @@ def OpB : NS_Op<"same_input_output_type_op", [SameOperandsAndResultType]> {
   let results = (outs I32:$y);
 }
 
+// DECL-LABEL: class OpB : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_y = 0;
+
 // CHECK-LABEL: OpB definitions
 // CHECK: void OpB::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type y, ::mlir::Value x)
 // CHECK:   odsState.addTypes(y);
@@ -39,6 +46,10 @@ def OpC : NS_Op<"three_normal_result_op", []> {
   let results = (outs I32:$x, /*unnamed*/I32, I32:$z);
 }
 
+// DECL-LABEL: class OpC : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_z = 2;
+
 // CHECK-LABEL: OpC definitions
 // CHECK:       void OpC::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type x, ::mlir::Type resultType1, ::mlir::Type z)
 // CHECK-NEXT:   odsState.addTypes(x)
diff --git a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
index deef178e68dc1..4614459abb2b6 100644
--- a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
@@ -2175,7 +2175,8 @@ generateNamedOperandGetters(const Operator &op, Class &opClass,
                     "'SameVariadicOperandSize' traits");
   }
 
-  // Print the ods names so they don't need to be hardcoded in the source.
+  // Print the ODS indices of operands so they don't need to be hardcoded in the
+  // source.
   for (int i = 0; i != numOperands; ++i) {
     const auto &operand = op.getOperand(i);
     if (operand.name.empty())
@@ -2389,6 +2390,17 @@ void OpEmitter::genNamedResultGetters() {
                     "'SameVariadicResultSize' traits");
   }
 
+  // Print the ODS indices of results so they don't need to be hardcoded in the
+  // source.
+  for (int i = 0; i != numResults; ++i) {
+    const auto &result = op.getResult(i);
+    if (result.name.empty())
+      continue;
+
+    opClass.declare<Field>("static constexpr int",
+                           Twine("odsIndex_") + result.name + " = " + Twine(i));
+  }
+
   // Build the initializer string for the result segment size attribute.
   std::string attrSizeInitCode;
   if (attrSizedResults) {

``````````

</details>


https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Sat Jul 18 14:55:25 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 14:55:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU (PR #210527)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210527@github.com>
Message-ID: <6a5bf64d.5a0412d2.3daf32.8d35@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/210527

>From 0d7ff5b87542e7ecee78d6d33e6696b9338212b9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 17:41:56 +0000
Subject: [PATCH 1/2] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU

Extend the vector.transfer_read/transfer_write lowerings so they can
produce N-D xegpu.load_nd/store_nd, not just 1D/2D, and relax the
out-of-bounds handling to match load_nd's implicit-zero padding.

Changes:
- Allow high-dim (rank > 2) vectors on the nd block path. The rank
  restriction in storeLoadPreconditions is gated behind a new
  allowHighDimVector flag so plain vector.load/store keep their 1D/2D
  limit while transfers accept any non-0D rank.
- Accept an out-of-bounds read whose padding is a zero constant or a
  ub.poison value (isZeroOrPoisonPadding): load_nd zero-fills
  out-of-bounds elements, and poison means those elements are
  don't-care, so both match its semantics.
- Restructure both patterns to attempt the nd block op first via a
  single canLowerToLoadNd/canLowerToStoreNd predicate, then fall back
  to the scattered path. The block path only realizes a minor-identity
  map (reads also handle an innermost-two-dims transpose via a trailing
  vector.transpose); any other permutation is routed to the scattered
  path, which permutes strides explicitly. This fixes silently dropping
  the permutation on high-dim transposed transfers.

Tests updated for the new high-dim block lowering, plus new
load_high_dim_transposed / store_high_dim_transposed coverage.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 268 ++++++++++--------
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 169 +++++++----
 .../transfer-write-to-xegpu.mlir              | 225 +++++++--------
 3 files changed, 380 insertions(+), 282 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..306ce6f1e8008 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -15,6 +15,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/UB/IR/UBOps.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
@@ -49,14 +50,53 @@ static bool isZeroConstant(Value val) {
       .Default(false);
 }
 
+// Return true if the transfer padding value is compatible with the implicit
+// padding of an nd block load. LoadNdOp fills out-of-bounds elements with zero,
+// so a zero constant matches its semantics exactly. A poison padding means the
+// out-of-bounds elements are "don't care", so any implicit padding (including
+// zero) is also acceptable.
+static bool isZeroOrPoisonPadding(Value val) {
+  return isZeroConstant(val) || val.getDefiningOp<ub::PoisonOp>();
+}
+
+// Return true if the permutation map keeps every dimension in place except the
+// innermost two, which are swapped, e.g.:
+//   (d0, d1) -> (d1, d0)
+//   (d0, d1, d2) -> (d2, d1)
+//   (d0, d1, d2, d3) -> (d0, d1, d3, d2)
+// This is the only non-identity permutation an nd block load can realize (by
+// loading the untransposed block and applying a trailing vector.transpose).
+static bool isInnermostTwoDimsTransposed(AffineMap map) {
+  unsigned numResults = map.getNumResults();
+  if (numResults < 2)
+    return false;
+  MLIRContext *ctx = map.getContext();
+  unsigned numInputs = map.getNumInputs();
+  // All but the innermost two results must match the minor-identity map.
+  for (unsigned i = 0; i + 2 < numResults; ++i)
+    if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
+      return false;
+  // The innermost two results must be the last two input dims, swapped.
+  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+         map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
+}
+
 static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
                                             Operation *op, VectorType vecTy,
-                                            MemRefType memTy) {
+                                            MemRefType memTy,
+                                            bool allowHighDimVector = false) {
   // Validate only vector as the basic vector store and load ops guarantee
   // XeGPU-compatible memref source.
   unsigned vecRank = vecTy.getRank();
-  if (!(vecRank == 1 || vecRank == 2))
+  // nd block loads/stores support N-D block transfers, so callers that lower to
+  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
+  // vector.load/store lowering keeps the 1D/2D restriction.
+  if (allowHighDimVector) {
+    if (vecRank == 0)
+      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
+  } else if (!(vecRank == 1 || vecRank == 2)) {
     return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
+  }
 
   if (!vecTy.getElementType().isIntOrFloat())
     return rewriter.notifyMatchFailure(
@@ -594,92 +634,78 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(readOp);
-    // Lower to scattered load Op if the target HW doesn't have 2d block load
-    // support and the load is not from shared memory.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        readOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (isOutOfBounds)
-        return failure();
-      return lowerToScatteredLoadOp(readOp, rewriter);
-    }
-
-    // Handle the 1D non-SLM case using load.gather.
-    if (loadedVecTy.getRank() == 1 && !isOutOfBounds)
-      return lowerToScatteredLoadOp(readOp, rewriter);
-
-    // Perform common data transfer checks.
-    // TODO: Maybe too strict for SLM case.
-    if (failed(
-            storeLoadPreconditions(rewriter, readOp, loadedVecTy, readMemTy)))
-      return failure();
-
-    if (isOutOfBounds && !isZeroConstant(readOp.getPadding()))
-      return rewriter.notifyMatchFailure(
-          readOp, "Unsupported non-zero padded out-of-bounds read");
+    bool hasBlockLoadSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // An nd block load can realize a minor-identity map directly, or an
+    // innermost-two-dims transpose via a trailing vector.transpose. Any other
+    // permutation (e.g. a mid-vector transpose of a high-dim load) is left to
+    // the scattered path, which permutes strides explicitly.
     AffineMap readMap = readOp.getPermutationMap();
-    // Check if this is a transpose: the map must have exactly 2 results,
-    // and those 2 results must be the last 2 input dimensions interchanged.
-    // Examples:
-    //   (d0, d1) -> (d1, d0)      // transpose
-    //   (d0, d1) -> (d0, d1)      // not a transpose
-    //   (d0, d1, d2) -> (d2, d1)  // transpose (last 2 dims swapped)
-    bool isTransposeLoad = false;
-    if (readMap.getNumResults() == 2) {
-      auto results = readMap.getResults();
-      unsigned numInputs = readMap.getNumInputs();
-      if (numInputs >= 2) {
-        auto lastDim = getAffineDimExpr(numInputs - 1, readMap.getContext());
-        auto secondLastDim =
-            getAffineDimExpr(numInputs - 2, readMap.getContext());
-        isTransposeLoad =
-            (results[0] == lastDim && results[1] == secondLastDim);
+    bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
+
+    // Prefer an nd block load. It requires HW block-load support, a >1D vector
+    // of a scalar element type backed by a scalar-element memref, and a map the
+    // block load can realize. Out-of-bounds reads are allowed as long as the
+    // padding matches load_nd's implicit zero padding.
+    bool canLowerToLoadNd =
+        hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
+        (readMap.isMinorIdentity() || isTransposeLoad) &&
+        loadedVecTy.getElementType().isIntOrFloat() &&
+        readMemTy.getElementType().isIntOrFloat() &&
+        (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
+
+    if (canLowerToLoadNd) {
+      auto elementType = loadedVecTy.getElementType();
+
+      SmallVector<int64_t> descShape(loadedVecTy.getShape());
+      if (isTransposeLoad) {
+        // If load is transposed, simply swap the last two dimensions of the
+        // loaded vector type to get the descriptor shape.
+        size_t rank = descShape.size();
+        assert(rank >= 2 && "Transpose requires at least 2 dimensions");
+        std::swap(descShape[rank - 1], descShape[rank - 2]);
+        loadedVecTy = VectorType::get(descShape, elementType);
       }
+      auto descType = xegpu::TensorDescType::get(
+          descShape, elementType, /*array_length=*/1,
+          /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, readOp.getBase(),
+          getAsOpFoldResult(readOp.getIndices()), loadedVecTy.getRank());
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      Operation *loadedOp =
+          xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
+                                  /*packed=*/nullptr, /*transpose=*/nullptr,
+                                  /*l1_hint=*/hint,
+                                  /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                  /*layout=*/nullptr);
+      if (isTransposeLoad) {
+        // Undo the innermost-two-dims swap with a trailing vector.transpose:
+        // keep the leading dimensions in place and interchange only the last
+        // two.
+        int64_t rank = loadedVecTy.getRank();
+        SmallVector<int64_t> perm(llvm::to_vector(llvm::seq<int64_t>(0, rank)));
+        std::swap(perm[rank - 1], perm[rank - 2]);
+        loadedOp = vector::TransposeOp::create(rewriter, loc,
+                                               loadedOp->getResult(0), perm);
+      }
+      rewriter.replaceOp(readOp, loadedOp);
+      return success();
     }
-    auto elementType = loadedVecTy.getElementType();
-
-    SmallVector<int64_t> descShape(loadedVecTy.getShape());
-    if (isTransposeLoad) {
-      // If load is transposed, simply swap the last two dimensions of the
-      // loaded vector type to get the descriptor shape.
-      size_t rank = descShape.size();
-      assert(rank >= 2 && "Transpose requires at least 2 dimensions");
-      std::swap(descShape[rank - 1], descShape[rank - 2]);
-      loadedVecTy = VectorType::get(descShape, elementType);
-    }
-    auto descType = xegpu::TensorDescType::get(
-        descShape, elementType, /*array_length=*/1,
-        /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, readOp.getBase(), getAsOpFoldResult(readOp.getIndices()),
-        loadedVecTy.getRank());
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    Operation *loadedOp =
-        xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
-                                /*packed=*/nullptr, /*transpose=*/nullptr,
-                                /*l1_hint=*/hint,
-                                /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                /*layout=*/nullptr);
-    if (isTransposeLoad) {
-      // Transposing the loaded vector with a separate vector.transpose
-      // operation
-      auto range = llvm::seq<int64_t>(0, readMap.getResults().size());
-      SmallVector<int64_t> perm(
-          range.rbegin(), range.rend()); // reverse the range for transpose
-      loadedOp = vector::TransposeOp::create(rewriter, loc,
-                                             loadedOp->getResult(0), perm);
-    }
-    rewriter.replaceOp(readOp, loadedOp);
 
-    return success();
+    // Fall back to a scattered load. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (isOutOfBounds)
+      return failure();
+    return lowerToScatteredLoadOp(readOp, rewriter);
   }
 };
 
@@ -727,47 +753,51 @@ struct TransferWriteLowering
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(writeOp);
-    // Lower to scattered store Op if the target HW doesn't have 2d block
-    // store support and the memref is not SLM.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        writeOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (writeOp.hasOutOfBoundsDim())
-        return failure();
-      return lowerToScatteredStoreOp(writeOp, rewriter);
-    }
-
-    if (failed(storeLoadPreconditions(rewriter, writeOp, vecTy, writeMemTy)))
-      return failure();
+    bool hasBlockStoreSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // Prefer an nd block store. It requires HW block-store support, a >1D
+    // vector of a scalar element type backed by a scalar-element memref, and a
+    // minor-identity map (block stores have no transpose support). Out-of-bounds
+    // writes are handled by the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    if (!map.isMinorIdentity())
-      return rewriter.notifyMatchFailure(writeOp, "Expects identity map");
-
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, writeOp.getBase(),
-        getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
-
-    auto descType = xegpu::TensorDescType::get(
-        vecTy.getShape(), vecTy.getElementType(),
-        /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
-        xegpu::MemorySpace::Global);
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    auto storeOp = xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(),
-                                            ndDesc, indices,
-                                            /*l1_hint=*/hint,
-                                            /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                            /*layout=*/nullptr);
-    rewriter.replaceOp(writeOp, storeOp);
+    bool canLowerToStoreNd =
+        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
+        vecTy.getElementType().isIntOrFloat() &&
+        writeMemTy.getElementType().isIntOrFloat();
+
+    if (canLowerToStoreNd) {
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, writeOp.getBase(),
+          getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
+
+      auto descType = xegpu::TensorDescType::get(
+          vecTy.getShape(), vecTy.getElementType(),
+          /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
+          xegpu::MemorySpace::Global);
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      auto storeOp =
+          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
+                                   indices,
+                                   /*l1_hint=*/hint,
+                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                   /*layout=*/nullptr);
+      rewriter.replaceOp(writeOp, storeOp);
+      return success();
+    }
 
-    return success();
+    // Fall back to a scattered store. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (writeOp.hasOutOfBoundsDim())
+      return failure();
+    return lowerToScatteredStoreOp(writeOp, rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index 3354868cd6cb9..ed495729f9add 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -182,6 +182,33 @@ gpu.func @load_transpose_3d_memref(%source: memref<32x64x128xf32>,
 
 }
 
+// -----
+// A high-dim load whose innermost two dims are transposed lowers to an
+// (untransposed) nd block load followed by a vector.transpose of the last two
+// dims.
+gpu.module @xevm_module {
+gpu.func @load_high_dim_transposed(%source: memref<2x2x64x128xf16>,
+    %offset: index) -> vector<1x1x64x128xf16> {
+  %c0 = arith.constant 0.0 : f16
+  %0 = vector.transfer_read %source[%offset, %offset, %offset, %offset], %c0
+    {permutation_map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3, d2)>,
+    in_bounds = [true, true, true, true]}
+    : memref<2x2x64x128xf16>, vector<1x1x64x128xf16>
+  gpu.return %0 : vector<1x1x64x128xf16>
+}
+
+// LOAD-ND-LABEL:  @load_high_dim_transposed(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x64x128xf16>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x64x128xf16>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<1x1x128x64xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<1x1x128x64xf16>
+// LOAD-ND:        vector.transpose %[[VEC]], [0, 1, 3, 2] : vector<1x1x128x64xf16> to vector<1x1x64x128xf16>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_transposed(
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load {{.*}} : i64, vector<1x1x64x128xindex>, vector<1x1x64x128xi1> -> vector<1x1x64x128xf16>
+}
+
 // -----
 gpu.module @xevm_module {
 gpu.func @load_dynamic_source(%source: memref<?x?x?xf32>,
@@ -275,21 +302,31 @@ gpu.func @load_dynamic_source3(%source: memref<?x?x?x?x?xf32>,
   gpu.return %0 : vector<2x4x8x16xf32>
 }
 
-// CHECK-LABEL:  @load_dynamic_source3(
-// CHECK-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
-// CHECK:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
-// CHECK-COUNT4: vector.step
-// CHECK-COUNT3: vector.broadcast
-// CHECK-COUNT4: vector.shape_cast
-// CHECK-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
-// CHECK-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
-// CHECK:        return %[[VEC]]
+// LOAD-ND-LABEL:  @load_dynamic_source3(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFF1:.*]], %[[SIZES:.*]]:4, %[[STRIDES:.*]]:4 = memref.extract_strided_metadata %[[SUBVIEW]]
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x4x8x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x4x8x16xf32>
+// LOAD-ND:        return %[[VEC]]
+
+// LOAD-GATHER-LABEL:  @load_dynamic_source3(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
+// LOAD-GATHER:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
+// LOAD-GATHER-COUNT4: vector.step
+// LOAD-GATHER-COUNT3: vector.broadcast
+// LOAD-GATHER-COUNT4: vector.shape_cast
+// LOAD-GATHER-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
+// LOAD-GATHER:        return %[[VEC]]
 }
 
 // -----
@@ -302,21 +339,27 @@ gpu.func @load_high_dim_vector(%source: memref<16x32x64xf32>,
   gpu.return %0 : vector<8x16x32xf32>
 }
 
-// CHECK-LABEL:  @load_high_dim_vector(
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
+// LOAD-ND-LABEL:  @load_high_dim_vector(
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %arg0 : memref<16x32x64xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<8x16x32xf32>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_vector(
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// LOAD-GATHER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// LOAD-GATHER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// LOAD-GATHER:        %[[C2048:.+]] = arith.constant 2048 : index
+// LOAD-GATHER:        %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-COUNT3: vector.step
+// LOAD-GATHER-COUNT3: vector.shape_cast
+// LOAD-GATHER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
 
 }
 
@@ -330,18 +373,25 @@ gpu.func @load_8D_vector(%source: memref<2x2x2x2x2x2x2x2xf32>,
   gpu.return %0 : vector<2x2x2x2x2x2x2x2xf32>
 }
 
-// CHECK-LABEL:  @load_8D_vector(
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-LABEL:  @load_8D_vector(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x2x2x2x2x2x2x2xf32>
+
+// LOAD-GATHER-LABEL:  @load_8D_vector(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// LOAD-GATHER-COUNT8: vector.step
+// LOAD-GATHER-COUNT7: vector.shape_cast
+// LOAD-GATHER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
 
 }
 
@@ -728,23 +778,38 @@ gpu.func @transpose_1x1024x24x64(
   gpu.return
 }
 
-// CHECK-LABEL: @transpose_1x1024x24x64
-// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
-// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
-// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+// The vector.transpose is folded into the transfer_read via
+// CombineTransferReadOpTranspose, giving the read a mid-vector permutation map
+// (d0, d1, d2, d3) -> (d0, d2, d1, d3). An nd block load can only realize an
+// innermost-two-dims transpose, so the read falls back to the scattered path
+// while the identity-map write still lowers to store_nd.
+// LOAD-ND-LABEL: @transpose_1x1024x24x64
+// LOAD-ND-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-ND-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-ND:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-ND:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-ND:     %[[VEC:.+]] = xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-ND:     %[[WDESC:.+]] = xegpu.create_nd_tdesc %arg1 : memref<1x24x1024x64xf16>
+// LOAD-ND-SAME:  -> !xegpu.tensor_desc<1x1x16x8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:     xegpu.store_nd %[[VEC]], %[[WDESC]]
+
+// LOAD-GATHER-LABEL: @transpose_1x1024x24x64
+// LOAD-GATHER-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-GATHER-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-DAG: %[[C65536:.+]] = arith.constant 65536 : index
 
 // Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
 // Scalar base offset: seq_off * 1536 (original dim1 stride),
 //                    block_id_x * 64  (original dim2 stride).
-// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
-// CHECK:     arith.muli %block_id_x, %[[C64]] : index
-// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-GATHER:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
 
 // Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
 // Scalar base offset: block_id_x * 65536 (original dim1 stride),
 //                    seq_off * 64        (original dim2 stride).
-// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
-// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
-// CHECK:     xegpu.store {{.*}}
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C65536]] : index
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C64]] : index
+// LOAD-GATHER:     xegpu.store {{.*}}
 
 }
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index 45bda46f738b2..ad7332097cc8f 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -11,36 +11,20 @@ gpu.func @store_1D_vector(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector(
-// STORE-ND-SAME:  %[[VEC:.+]]: vector<8xf32>,
-// STORE-ND-SAME:  %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-ND-SAME:  %[[OFFSET:.+]]: index
-// STORE-ND:       %[[ELEM_BYTES:.+]] = arith.constant 4 : index
-// STORE-ND:       %[[COLLAPSED:.+]] = memref.subview %[[SRC]][%[[OFFSET]], %[[OFFSET]], 0]
-// STORE-ND:       %[[BASE_BUFFER:.+]], %[[OFFSET1:.+]], %[[SIZES:.+]], %[[STRIDES:.+]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND-SAME:    : memref<32xf32, strided<[1], offset: ?>> -> memref<f32>, index, index, index
-// STORE-ND:       %[[INTPTR:.+]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND-SAME:    : memref<f32> -> index
-// STORE-ND:       %[[MUL:.+]] = arith.muli %[[OFFSET1]], %[[ELEM_BYTES]] : index
-// STORE-ND:       %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:       %[[I64PTR:.+]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:       %[[DESC:.+]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [32],
-// STORE-ND-SAME:                   strides : [1] : i64  -> !xegpu.tensor_desc<8xf32,
-// STORE-ND-SAME:    boundary_check = false
-// STORE-ND:       xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFFSET]]] : vector<8xf32>
-
-// STORE-SCATTER-LABEL:  @store_1D_vector(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-SCATTER-DAG:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER-DAG:        %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: arith.muli {{.*}} : index
-// STORE-SCATTER-COUNT2: arith.addi {{.*}} : index
-// STORE-SCATTER-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:       xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_1D_vector(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
+// CHECK-DAG:    %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK-DAG:    %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: arith.muli {{.*}} : index
+// CHECK-COUNT2: arith.addi {{.*}} : index
+// CHECK-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----
@@ -150,7 +134,7 @@ gpu.func @store_out_of_bounds(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
+gpu.func @store_transposed(%vec: vector<8x16xf32>,
     %source: memref<32x64xf32>, %offset: index) {
   vector.transfer_write %vec, %source[%offset, %offset]
     {permutation_map = affine_map<(d0, d1) -> (d1, d0)>,
@@ -159,22 +143,38 @@ gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @no_store_transposed(
-// STORE-ND:       vector.transfer_write
+// An nd block store cannot transpose, so a transposed write lowers through the
+// scattered path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_transposed(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
+// CHECK-SAME:   %[[OFFSET:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
+// CHECK-COUNT2: %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: vector.shape_cast {{.*}}
+// CHECK-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+}
 
-// STORE-SCATTER-LABEL:  @no_store_transposed(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
-// STORE-SCATTER-SAME:   %[[OFFSET:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
-// STORE-SCATTER-COUNT2: %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: vector.shape_cast {{.*}}
-// STORE-SCATTER-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+// -----
+// A high-dim transposed write cannot use an nd block store (no transpose
+// support), so it lowers through the scattered path on both runs.
+gpu.module @xevm_module {
+gpu.func @store_high_dim_transposed(%vec: vector<8x16x4xf32>,
+    %source: memref<16x32x64xf32>, %offset: index) {
+  vector.transfer_write %vec, %source[%offset, %offset, %offset]
+    {permutation_map = affine_map<(d0, d1, d2) -> (d0, d2, d1)>,
+    in_bounds = [true, true, true]}
+    : vector<8x16x4xf32>, memref<16x32x64xf32>
+  gpu.return
+}
+
+// CHECK-LABEL:  @store_high_dim_transposed(
+// CHECK:        xegpu.store {{.*}} : vector<8x16x4xf32>, i64, vector<8x16x4xindex>, vector<8x16x4xi1>
 }
 
 // -----
@@ -187,23 +187,31 @@ gpu.func @store_high_dim_vector(%vec: vector<8x16x32xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_high_dim_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
+// STORE-ND-LABEL:  @store_high_dim_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<16x32x64xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<8x16x32xf32>, !xegpu.tensor_desc<8x16x32xf32
+
+// STORE-SCATTER-LABEL:  @store_high_dim_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// STORE-SCATTER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// STORE-SCATTER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// STORE-SCATTER:        %[[C2048:.+]] = arith.constant 2048 : index
+// STORE-SCATTER:        %[[C64:.+]] = arith.constant 64 : index
+// STORE-SCATTER-COUNT3: vector.step
+// STORE-SCATTER-COUNT3: vector.shape_cast
+// STORE-SCATTER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
 }
 
 // -----
@@ -216,19 +224,27 @@ gpu.func @store_8D_vector(%vec: vector<2x2x2x2x2x2x2x2xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_8D_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
+// STORE-ND-LABEL:  @store_8D_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<2x2x2x2x2x2x2x2xf32>, !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32
+
+// STORE-SCATTER-LABEL:  @store_8D_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// STORE-SCATTER-COUNT8: vector.step
+// STORE-SCATTER-COUNT7: vector.shape_cast
+// STORE-SCATTER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
 }
 
 // -----
@@ -330,41 +346,28 @@ gpu.func @store_to_subview(%vec: vector<8xf16>,
       : vector<8xf16>, memref<256x256xf16, strided<[4096, 1], offset: ?>>
   gpu.return
 }
-// STORE-ND-LABEL:  @store_to_subview(
-// STORE-ND-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-ND-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-ND-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-ND:        %[[ELEM_BYTES:.+]] = arith.constant 2 : index
-// STORE-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1] : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SUBVIEW]][%[[OFF2]], 0]
-// STORE-ND:        %[[BASE_BUFFER:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND:        %[[MUL:.+]] = arith.muli %[[OFFSET]], %[[ELEM_BYTES]] : index
-// STORE-ND:        %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:        %[[DESC:.*]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [256], strides : [1] : i64 ->
-// STORE-ND-SAME:                    !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
-// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFF2]]] : vector<8xf16>
-
-// STORE-SCATTER-LABEL:  @store_to_subview(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-SCATTER-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
-// STORE-SCATTER-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-SCATTER:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
-// STORE-SCATTER:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// STORE-SCATTER:        arith.muli {{.*}} : index
-// STORE-SCATTER:        arith.addi %[[OFFSET]]{{.*}} : index
-// STORE-SCATTER:        arith.addi {{.*}} : index
-// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
-// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both runs, even when the destination is a strided
+// subview.
+// CHECK-LABEL:  @store_to_subview(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf16>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
+// CHECK-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
+// CHECK-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
+// CHECK:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
+// CHECK:        %[[STEP:.+]] = vector.step : vector<8xindex>
+// CHECK:        arith.muli {{.*}} : index
+// CHECK:        arith.addi %[[OFFSET]]{{.*}} : index
+// CHECK:        arith.addi {{.*}} : index
+// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
+// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
+// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----

>From 0c33ce78ecc38f5ab40ebc922a0ddc5306369df5 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 21:54:01 +0000
Subject: [PATCH 2/2] [mlir][xegpu] Refactor transfer lowering; inline
 storeLoadPreconditions

Restructure TransferReadLowering/TransferWriteLowering to try the nd block
op first via a single canLowerTo{Load,Store}Nd predicate, then fall back to
the scattered path (guarding out-of-bounds, which scatter cannot express).

Remove storeLoadPreconditions and inline its remaining checks into the plain
vector.load/store patterns. Drop the redundant vector element-type check since
it always matches the memref element type.

No functional change.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 86 +++++++------------
 1 file changed, 30 insertions(+), 56 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 306ce6f1e8008..80bd88beece6d 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -77,42 +77,11 @@ static bool isInnermostTwoDimsTransposed(AffineMap map) {
     if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
       return false;
   // The innermost two results must be the last two input dims, swapped.
-  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+  return map.getResult(numResults - 2) ==
+             getAffineDimExpr(numInputs - 1, ctx) &&
          map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
 }
 
-static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
-                                            Operation *op, VectorType vecTy,
-                                            MemRefType memTy,
-                                            bool allowHighDimVector = false) {
-  // Validate only vector as the basic vector store and load ops guarantee
-  // XeGPU-compatible memref source.
-  unsigned vecRank = vecTy.getRank();
-  // nd block loads/stores support N-D block transfers, so callers that lower to
-  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
-  // vector.load/store lowering keeps the 1D/2D restriction.
-  if (allowHighDimVector) {
-    if (vecRank == 0)
-      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
-  } else if (!(vecRank == 1 || vecRank == 2)) {
-    return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
-  }
-
-  if (!vecTy.getElementType().isIntOrFloat())
-    return rewriter.notifyMatchFailure(
-        op, "Expected scalar type with known bitwidth");
-
-  // XeGPU requires the memref to have a scalar integer or float element type.
-  // Memrefs with vector element types (e.g. memref<?xvector<4xf32>>) are not
-  // supported because createNdDescriptor computes byte offsets using
-  // getElementTypeBitWidth(), which asserts on non-integer/float types.
-  if (!memTy.getElementType().isIntOrFloat())
-    return rewriter.notifyMatchFailure(
-        op, "Unsupported memref element type: expected integer or float");
-
-  return success();
-}
-
 static LogicalResult transferPreconditions(PatternRewriter &rewriter,
                                            VectorTransferOpInterface xferOp) {
   if (xferOp.getMask())
@@ -647,13 +616,12 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
 
     // Prefer an nd block load. It requires HW block-load support, a >1D vector
-    // of a scalar element type backed by a scalar-element memref, and a map the
-    // block load can realize. Out-of-bounds reads are allowed as long as the
-    // padding matches load_nd's implicit zero padding.
+    // backed by a scalar-element memref, and a map the block load can realize.
+    // Out-of-bounds reads are allowed as long as the padding matches load_nd's
+    // implicit zero padding.
     bool canLowerToLoadNd =
         hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
         (readMap.isMinorIdentity() || isTransposeLoad) &&
-        loadedVecTy.getElementType().isIntOrFloat() &&
         readMemTy.getElementType().isIntOrFloat() &&
         (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
 
@@ -759,14 +727,13 @@ struct TransferWriteLowering
         (chip == "pvc" || chip == "bmg" || chip == "cri");
 
     // Prefer an nd block store. It requires HW block-store support, a >1D
-    // vector of a scalar element type backed by a scalar-element memref, and a
-    // minor-identity map (block stores have no transpose support). Out-of-bounds
-    // writes are handled by the descriptor's boundary check.
+    // vector backed by a scalar-element memref, and a minor-identity map (block
+    // stores have no transpose support). Out-of-bounds writes are handled by
+    // the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    bool canLowerToStoreNd =
-        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
-        vecTy.getElementType().isIntOrFloat() &&
-        writeMemTy.getElementType().isIntOrFloat();
+    bool canLowerToStoreNd = hasBlockStoreSupport && vecTy.getRank() > 1 &&
+                             map.isMinorIdentity() &&
+                             writeMemTy.getElementType().isIntOrFloat();
 
     if (canLowerToStoreNd) {
       auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
@@ -782,18 +749,17 @@ struct TransferWriteLowering
       xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
           rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
 
-      auto storeOp =
-          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
-                                   indices,
-                                   /*l1_hint=*/hint,
-                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                   /*layout=*/nullptr);
+      auto storeOp = xegpu::StoreNdOp::create(
+          rewriter, loc, writeOp.getVector(), ndDesc, indices,
+          /*l1_hint=*/hint,
+          /*l2_hint=*/hint, /*l3_hint=*/hint,
+          /*layout=*/nullptr);
       rewriter.replaceOp(writeOp, storeOp);
       return success();
     }
 
-    // Fall back to a scattered store. It supports arbitrary permutations and any
-    // rank, but cannot express out-of-bounds accesses.
+    // Fall back to a scattered store. It supports arbitrary permutations and
+    // any rank, but cannot express out-of-bounds accesses.
     // TODO: add support for OutOfBound access.
     if (writeOp.hasOutOfBoundsDim())
       return failure();
@@ -878,8 +844,12 @@ struct LoadLowering : public OpRewritePattern<vector::LoadOp> {
 
     VectorType vecTy = loadOp.getResult().getType();
     MemRefType memTy = loadOp.getBase().getType();
-    if (failed(storeLoadPreconditions(rewriter, loadOp, vecTy, memTy)))
-      return failure();
+    // The plain vector.load lowering only supports 1D/2D block loads.
+    if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
+      return rewriter.notifyMatchFailure(loadOp, "Expects 1D or 2D vector");
+    if (!memTy.getElementType().isIntOrFloat())
+      return rewriter.notifyMatchFailure(
+          loadOp, "Unsupported memref element type: expected integer or float");
 
     // Boundary check is available only for block instructions.
     bool boundaryCheck = vecTy.getRank() > 1;
@@ -918,8 +888,12 @@ struct StoreLowering : public OpRewritePattern<vector::StoreOp> {
     TypedValue<VectorType> vector = storeOp.getValueToStore();
     VectorType vecTy = vector.getType();
     MemRefType memTy = storeOp.getBase().getType();
-    if (failed(storeLoadPreconditions(rewriter, storeOp, vecTy, memTy)))
-      return failure();
+    // The plain vector.store lowering only supports 1D/2D block stores.
+    if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
+      return rewriter.notifyMatchFailure(storeOp, "Expects 1D or 2D vector");
+    if (!memTy.getElementType().isIntOrFloat())
+      return rewriter.notifyMatchFailure(
+          storeOp, "Unsupported memref element type: expected integer or float");
 
     // Boundary check is available only for block instructions.
     bool boundaryCheck = vecTy.getRank() > 1;


From llvmlistbot at llvm.org  Sat Jul 18 17:26:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 17:26:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5c199f.0c05e68e.3c2f5d.f065@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/209911

>From b3a3bafce7d55358a5f5872210ef6c97806418c1 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Wed, 15 Jul 2026 14:58:04 -0700
Subject: [PATCH] [mlir][LLVMIR] Test retained local variable export

---
 mlir/test/Target/LLVMIR/llvmir-debug.mlir | 47 +++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 72dadb2602c52..b93a38be6634d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -421,6 +421,53 @@ llvm.func @imp_fn() {
 
 // -----
 
+// Retained local variables may be represented separately from debug intrinsic
+// variables in MLIR. Ensure export reuses a single LLVM DILocalVariable node.
+
+#file = #llvm.di_file<"test.c" in "">
+#cu = #llvm.di_compile_unit<
+  id = distinct[0]<>, sourceLanguage = DW_LANG_C, file = #file
+>
+#void = #llvm.di_null_type
+#int = #llvm.di_basic_type<
+  tag = DW_TAG_base_type, name = "int",
+  sizeInBits = 32, encoding = DW_ATE_signed
+>
+#sp_self = #llvm.di_subprogram<recId = distinct[1]<>, isRecSelf = true>
+#retained_local = #llvm.di_local_variable<
+  scope = #sp_self, name = "retained_arg", file = #file,
+  line = 1, arg = 1, type = #int
+>
+#sp_type = #llvm.di_subroutine_type<types = #void, #int>
+#sp = #llvm.di_subprogram<
+  recId = distinct[1]<>, id = distinct[2]<>, compileUnit = #cu,
+  scope = #file, name = "fn_with_retained_local_export",
+  file = #file, line = 1, scopeLine = 1, subprogramFlags = Definition,
+  type = #sp_type, retainedNodes = [#retained_local]
+>
+#dbg_local = #llvm.di_local_variable<
+  scope = #sp, name = "retained_arg", file = #file,
+  line = 1, arg = 1, type = #int
+>
+
+// CHECK-LABEL: define void @fn_with_retained_local_export(
+// CHECK-SAME: i32 %[[ARG:.*]]) !dbg ![[SP:[0-9]+]]
+llvm.func @fn_with_retained_local_export(%arg0: i32) {
+  // RECORDS: #dbg_value(i32 %[[ARG]], ![[LOCAL:[0-9]+]],
+  // RECORDS-SAME: !DIExpression(), !{{.*}})
+  llvm.intr.dbg.value #dbg_local = %arg0 : i32 loc(fused<#sp>["test.c":1:1])
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])
+
+// CHECK-DAG: ![[SP]] = distinct !DISubprogram(
+// CHECK-SAME: name: "fn_with_retained_local_export"
+// CHECK-SAME: retainedNodes: ![[NODES:[0-9]+]]
+// CHECK-DAG: ![[NODES]] = !{![[LOCAL]]}
+// CHECK-DAG: ![[LOCAL]] = !DILocalVariable(name: "retained_arg", arg: 1,
+// CHECK-SAME: scope: ![[SP]]
+
+// -----
+
 // Nameless and scopeless global constant.
 
 // CHECK-LABEL: @.str.1 = external constant [10 x i8]


From llvmlistbot at llvm.org  Sat Jul 18 17:31:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 17:31:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5c1adc.f3151ff8.37ac76.c005@mx.google.com>


qyingwu wrote:

> How does the debug info look if we export it back to LLVM IR?
> 
> It may be ok if we have a duplication in MLIR as long as the exported debug information is correct.
> 
> Also looking at the IR above, I wonder why the di_local_variable1 is not recursive if we traverse it starting from the debug intrinsic?


> How does the debug info look if we export it back to LLVM IR?
I checked this. With the subprogram-recursive import path, the MLIR still has two local variable attrs, but exporting it back to LLVM IR produces a single `DILocalVariable` metadata node. Both the `dbg.value` and the subprogram `retainedNodes` reference the same node:

  ```llvm
  #dbg_value(i32 %0, !8, !DIExpression(), !9)
  !3 = distinct !DISubprogram(name: "fn_with_retained_local", ..., retainedNodes: !7)
  !7 = !{!8}
  !8 = !DILocalVariable(name: "n", arg: 1, scope: !3, file: !1, line: 1, type: !6)

> It may be ok if we have a duplication in MLIR as long as the exported debug information is correct.
That makes sense. 
I simplified the patch in that direction: it no longer extends the recursive debug-info machinery or tries to remove the MLIR-side duplication. The updated patch only adds export coverage showing that this duplicated MLIR shape is emitted as non-duplicated LLVM debug metadata.

> Also looking at the IR above, I wonder why the di_local_variable1 is not recursive if we traverse it starting from the debug intrinsic?

The repeated node during the retainedNodes traversal is the DISubprogram, so the cycle is broken there.
The local variable imported from retainedNodes is scoped by the self-recursive subprogram placeholder.

Later, when the debug intrinsic is imported, the subprogram has already been finalized. At that point, translating the same DILocalVariable no longer re-enters an active recursive traversal, so the local variable itself is not treated as recursive and is imported as a concrete local variable scoped by the finalized subprogram.

https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Sat Jul 18 17:38:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 17:38:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5c1c8c.6c5720b3.2da3b9.909f@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Sat Jul 18 18:43:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 18:43:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][AMDGPU] Diagnose unsupported packed
 fp8 conversions (PR #210565)
Message-ID: <llvm.org/llvm/llvm-project/pull/210565@github.com>


https://github.com/qyingwu created https://github.com/llvm/llvm-project/pull/210565

##Summary
Fixes #210266.

`amdgpu.packed_trunc_2xfp8` can crash during AMDGPU-to-ROCDL conversion when the result element type does not match the selected chipset. For example, `gfx942` expects FNUZ fp8 types, but the reproducer uses the OCP `f8E4M3FN` type. The lowering then failed to create an intrinsic result and later tried to bitcast a null value.

This patch diagnoses unsupported packed fp8 conversions instead of crashing. It also adds the same guard for `amdgpu.packed_stoch_round_fp8`, which had the same unchecked result path.

##Tests
  ```
ninja -C build mlir-opt FileCheck count not
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/8-bit-floats.mlir mlir/test/
Conversion/AMDGPUToROCDL/8-bit-floats-ocp.mlir mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL
```

>From e591eb01cd131469be632382f55cbc3df84d3a59 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 18:40:00 -0700
Subject: [PATCH] [mlir][AMDGPU] Diagnose unsupported packed fp8 conversions

---
 .../Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp |  7 +++++++
 .../AMDGPUToROCDL/packed-trunc-invalid.mlir    | 18 ++++++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir

diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 90e15b1680446..5c4195778be5f 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -3038,6 +3038,9 @@ LogicalResult PackedTrunc2xFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtPkFp8F32Op::create(rewriter, loc, i32, sourceA, sourceB,
                                           existing, op.getWordIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed trunc conversion on the given chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
@@ -3072,6 +3075,10 @@ LogicalResult PackedStochRoundFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtSrFp8F32Op::create(rewriter, loc, i32, source, stoch,
                                           existing, op.getStoreIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed stochastic round conversion on the given "
+        "chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
new file mode 100644
index 0000000000000..874fcae6555a6
--- /dev/null
+++ b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt %s --convert-amdgpu-to-rocdl=chipset=gfx942 --split-input-file --verify-diagnostics
+
+func.func @packed_trunc_ocp_type_requires_ocp_chipset(%arg0: f32) {
+  // expected-error at below {{'amdgpu.packed_trunc_2xfp8' op no intrinsic matching packed trunc conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_trunc_2xfp8'}}
+  %0 = amdgpu.packed_trunc_2xfp8 %arg0, undef into undef[word 0] : f32 to vector<4xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @packed_stoch_round_ocp_type_requires_ocp_chipset(%arg0: f32,
+                                                            %arg1: i32) {
+  // expected-error at below {{'amdgpu.packed_stoch_round_fp8' op no intrinsic matching packed stochastic round conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_stoch_round_fp8'}}
+  %0 = amdgpu.packed_stoch_round_fp8 %arg0 + %arg1 into undef[0] : f32 to vector<4xf8E4M3FN>
+  return
+}


From llvmlistbot at llvm.org  Sat Jul 18 18:43:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 18:43:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][AMDGPU] Diagnose unsupported packed
 fp8 conversions (PR #210565)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210565@github.com>
Message-ID: <6a5c2bd5.f06f4994.2f0620.678a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: qyingwu

<details>
<summary>Changes</summary>

##Summary
Fixes #<!-- -->210266.

`amdgpu.packed_trunc_2xfp8` can crash during AMDGPU-to-ROCDL conversion when the result element type does not match the selected chipset. For example, `gfx942` expects FNUZ fp8 types, but the reproducer uses the OCP `f8E4M3FN` type. The lowering then failed to create an intrinsic result and later tried to bitcast a null value.

This patch diagnoses unsupported packed fp8 conversions instead of crashing. It also adds the same guard for `amdgpu.packed_stoch_round_fp8`, which had the same unchecked result path.

##Tests
  ```
ninja -C build mlir-opt FileCheck count not
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/8-bit-floats.mlir mlir/test/
Conversion/AMDGPUToROCDL/8-bit-floats-ocp.mlir mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL
```

---
Full diff: https://github.com/llvm/llvm-project/pull/210565.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp (+7) 
- (added) mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir (+18) 


``````````diff
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 90e15b1680446..5c4195778be5f 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -3038,6 +3038,9 @@ LogicalResult PackedTrunc2xFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtPkFp8F32Op::create(rewriter, loc, i32, sourceA, sourceB,
                                           existing, op.getWordIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed trunc conversion on the given chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
@@ -3072,6 +3075,10 @@ LogicalResult PackedStochRoundFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtSrFp8F32Op::create(rewriter, loc, i32, source, stoch,
                                           existing, op.getStoreIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed stochastic round conversion on the given "
+        "chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
new file mode 100644
index 0000000000000..874fcae6555a6
--- /dev/null
+++ b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt %s --convert-amdgpu-to-rocdl=chipset=gfx942 --split-input-file --verify-diagnostics
+
+func.func @packed_trunc_ocp_type_requires_ocp_chipset(%arg0: f32) {
+  // expected-error at below {{'amdgpu.packed_trunc_2xfp8' op no intrinsic matching packed trunc conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_trunc_2xfp8'}}
+  %0 = amdgpu.packed_trunc_2xfp8 %arg0, undef into undef[word 0] : f32 to vector<4xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @packed_stoch_round_ocp_type_requires_ocp_chipset(%arg0: f32,
+                                                            %arg1: i32) {
+  // expected-error at below {{'amdgpu.packed_stoch_round_fp8' op no intrinsic matching packed stochastic round conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_stoch_round_fp8'}}
+  %0 = amdgpu.packed_stoch_round_fp8 %arg0 + %arg1 into undef[0] : f32 to vector<4xf8E4M3FN>
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210565

From llvmlistbot at llvm.org  Sat Jul 18 18:43:50 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 18:43:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][AMDGPU] Diagnose unsupported packed
 fp8 conversions (PR #210565)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210565@github.com>
Message-ID: <6a5c2bd6.1e1ac41d.3969c3.353a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: qyingwu

<details>
<summary>Changes</summary>

##Summary
Fixes #<!-- -->210266.

`amdgpu.packed_trunc_2xfp8` can crash during AMDGPU-to-ROCDL conversion when the result element type does not match the selected chipset. For example, `gfx942` expects FNUZ fp8 types, but the reproducer uses the OCP `f8E4M3FN` type. The lowering then failed to create an intrinsic result and later tried to bitcast a null value.

This patch diagnoses unsupported packed fp8 conversions instead of crashing. It also adds the same guard for `amdgpu.packed_stoch_round_fp8`, which had the same unchecked result path.

##Tests
  ```
ninja -C build mlir-opt FileCheck count not
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL/8-bit-floats.mlir mlir/test/
Conversion/AMDGPUToROCDL/8-bit-floats-ocp.mlir mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
build/bin/llvm-lit -v mlir/test/Conversion/AMDGPUToROCDL
```

---
Full diff: https://github.com/llvm/llvm-project/pull/210565.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp (+7) 
- (added) mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir (+18) 


``````````diff
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 90e15b1680446..5c4195778be5f 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -3038,6 +3038,9 @@ LogicalResult PackedTrunc2xFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtPkFp8F32Op::create(rewriter, loc, i32, sourceA, sourceB,
                                           existing, op.getWordIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed trunc conversion on the given chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
@@ -3072,6 +3075,10 @@ LogicalResult PackedStochRoundFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtSrFp8F32Op::create(rewriter, loc, i32, source, stoch,
                                           existing, op.getStoreIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed stochastic round conversion on the given "
+        "chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
new file mode 100644
index 0000000000000..874fcae6555a6
--- /dev/null
+++ b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt %s --convert-amdgpu-to-rocdl=chipset=gfx942 --split-input-file --verify-diagnostics
+
+func.func @packed_trunc_ocp_type_requires_ocp_chipset(%arg0: f32) {
+  // expected-error at below {{'amdgpu.packed_trunc_2xfp8' op no intrinsic matching packed trunc conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_trunc_2xfp8'}}
+  %0 = amdgpu.packed_trunc_2xfp8 %arg0, undef into undef[word 0] : f32 to vector<4xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @packed_stoch_round_ocp_type_requires_ocp_chipset(%arg0: f32,
+                                                            %arg1: i32) {
+  // expected-error at below {{'amdgpu.packed_stoch_round_fp8' op no intrinsic matching packed stochastic round conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_stoch_round_fp8'}}
+  %0 = amdgpu.packed_stoch_round_fp8 %arg0 + %arg1 into undef[0] : f32 to vector<4xf8E4M3FN>
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210565

From llvmlistbot at llvm.org  Sat Jul 18 18:43:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 18:43:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][AMDGPU] Diagnose unsupported packed
 fp8 conversions (PR #210565)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210565@github.com>
Message-ID: <6a5c2bdb.510f576d.2f694c.58f9@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210565

>From e591eb01cd131469be632382f55cbc3df84d3a59 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 18:40:00 -0700
Subject: [PATCH] [mlir][AMDGPU] Diagnose unsupported packed fp8 conversions

---
 .../Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp |  7 +++++++
 .../AMDGPUToROCDL/packed-trunc-invalid.mlir    | 18 ++++++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir

diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index 90e15b1680446..5c4195778be5f 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -3038,6 +3038,9 @@ LogicalResult PackedTrunc2xFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtPkFp8F32Op::create(rewriter, loc, i32, sourceA, sourceB,
                                           existing, op.getWordIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed trunc conversion on the given chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
@@ -3072,6 +3075,10 @@ LogicalResult PackedStochRoundFp8OpLowering::matchAndRewrite(
   else if (typeIsExpectedFp8ForChipset(chipset, resultElemType))
     result = ROCDL::CvtSrFp8F32Op::create(rewriter, loc, i32, source, stoch,
                                           existing, op.getStoreIndex());
+  else
+    return op.emitOpError(
+        "no intrinsic matching packed stochastic round conversion on the given "
+        "chipset");
 
   result = rewriter.replaceOpWithNewOp<LLVM::BitcastOp>(
       op, getTypeConverter()->convertType(resultType), result);
diff --git a/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
new file mode 100644
index 0000000000000..874fcae6555a6
--- /dev/null
+++ b/mlir/test/Conversion/AMDGPUToROCDL/packed-trunc-invalid.mlir
@@ -0,0 +1,18 @@
+// RUN: mlir-opt %s --convert-amdgpu-to-rocdl=chipset=gfx942 --split-input-file --verify-diagnostics
+
+func.func @packed_trunc_ocp_type_requires_ocp_chipset(%arg0: f32) {
+  // expected-error at below {{'amdgpu.packed_trunc_2xfp8' op no intrinsic matching packed trunc conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_trunc_2xfp8'}}
+  %0 = amdgpu.packed_trunc_2xfp8 %arg0, undef into undef[word 0] : f32 to vector<4xf8E4M3FN>
+  return
+}
+
+// -----
+
+func.func @packed_stoch_round_ocp_type_requires_ocp_chipset(%arg0: f32,
+                                                            %arg1: i32) {
+  // expected-error at below {{'amdgpu.packed_stoch_round_fp8' op no intrinsic matching packed stochastic round conversion on the given chipset}}
+  // expected-error at below {{failed to legalize operation 'amdgpu.packed_stoch_round_fp8'}}
+  %0 = amdgpu.packed_stoch_round_fp8 %arg0 + %arg1 into undef[0] : f32 to vector<4xf8E4M3FN>
+  return
+}


From llvmlistbot at llvm.org  Sat Jul 18 19:10:28 2026
From: llvmlistbot at llvm.org (Luc Forget)
Date: Sat, 18 Jul 2026 19:10:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a5c3214.c56fb102.2d07b8.f6c0@mx.google.com>


https://github.com/lforg37 approved this pull request.

LGTM, but could be useful at the occasion to get review from people with better knowledge of the memory dialect to ensure the raise is not overly complex there.

https://github.com/llvm/llvm-project/pull/205990

From llvmlistbot at llvm.org  Sat Jul 18 19:14:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 19:14:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
Message-ID: <llvm.org/llvm/llvm-project/pull/210566@github.com>


https://github.com/qyingwu created https://github.com/llvm/llvm-project/pull/210566

##Summary
Fixes #207350.

During dialect conversion, MLIR may emit an error for an unresolved materialization that remains live after conversion. The diagnostic also prints one existing live user.

That live user can be nested under IR that is temporarily not verifiable while conversion is in progress. In the issue reproducer, the live user is inside a `spirv.func`. Printing it without special flags causes the printer to verify the surrounding operation, which can enter SPIR-V verification on transient conversion IR and crash.

This patch prints the live user with `OpPrintingFlags().assumeVerified()` so the diagnostic path does not re-verify transient IR while reporting the conversion failure.

##Tests
```
ninja -C build mlir-opt FileCheck count not
build/bin/llvm-lit -v mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
build/bin/llvm-lit -v mlir/test/Transforms/test-legalize-type-conversion.mlir mlir/test/
Transforms/test-legalize-unknown-root.mlir mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
build/bin/llvm-lit -v mlir/test/Transforms
ninja -C build check-mlir
```

>From 4821f857074a82b620cc3a788e55652cbef25824 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir] Avoid verifying live user in conversion diagnostic

---
 .../Transforms/Utils/DialectConversion.cpp    |  6 ++++--
 .../test-legalize-erased-op-with-uses.mlir    | 21 ++++++++++++++++++-
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..9a0bffbc4e068 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3391,8 +3391,10 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
                             << inputOperands.getTypes() << ") to ("
                             << op.getResultTypes()
                             << ") that remained live after conversion";
-  diag.attachNote(op->getUsers().begin()->getLoc())
-      << "see existing live user here: " << *op->getUsers().begin();
+  Operation *liveUser = *op->getUsers().begin();
+  diag.attachNote(liveUser->getLoc())
+      << "see existing live user here: "
+      << OpWithFlags(liveUser, OpPrintingFlags().assumeVerified());
   return failure();
 }
 
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..829ff2f183e53 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Sat Jul 18 19:14:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 19:14:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a5c32f8.eb165e5c.4506c.c0fc@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210566

>From 4821f857074a82b620cc3a788e55652cbef25824 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir] Avoid verifying live user in conversion diagnostic

---
 .../Transforms/Utils/DialectConversion.cpp    |  6 ++++--
 .../test-legalize-erased-op-with-uses.mlir    | 21 ++++++++++++++++++-
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..9a0bffbc4e068 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3391,8 +3391,10 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
                             << inputOperands.getTypes() << ") to ("
                             << op.getResultTypes()
                             << ") that remained live after conversion";
-  diag.attachNote(op->getUsers().begin()->getLoc())
-      << "see existing live user here: " << *op->getUsers().begin();
+  Operation *liveUser = *op->getUsers().begin();
+  diag.attachNote(liveUser->getLoc())
+      << "see existing live user here: "
+      << OpWithFlags(liveUser, OpPrintingFlags().assumeVerified());
   return failure();
 }
 
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..829ff2f183e53 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Sat Jul 18 19:14:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 19:14:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a5c3308.492e23e4.d19bc.e60d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-core

Author: qyingwu

<details>
<summary>Changes</summary>

##Summary
Fixes #<!-- -->207350.

During dialect conversion, MLIR may emit an error for an unresolved materialization that remains live after conversion. The diagnostic also prints one existing live user.

That live user can be nested under IR that is temporarily not verifiable while conversion is in progress. In the issue reproducer, the live user is inside a `spirv.func`. Printing it without special flags causes the printer to verify the surrounding operation, which can enter SPIR-V verification on transient conversion IR and crash.

This patch prints the live user with `OpPrintingFlags().assumeVerified()` so the diagnostic path does not re-verify transient IR while reporting the conversion failure.

##Tests
```
ninja -C build mlir-opt FileCheck count not
build/bin/llvm-lit -v mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
build/bin/llvm-lit -v mlir/test/Transforms/test-legalize-type-conversion.mlir mlir/test/
Transforms/test-legalize-unknown-root.mlir mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
build/bin/llvm-lit -v mlir/test/Transforms
ninja -C build check-mlir
```

---
Full diff: https://github.com/llvm/llvm-project/pull/210566.diff


2 Files Affected:

- (modified) mlir/lib/Transforms/Utils/DialectConversion.cpp (+4-2) 
- (modified) mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir (+20-1) 


``````````diff
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..9a0bffbc4e068 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3391,8 +3391,10 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
                             << inputOperands.getTypes() << ") to ("
                             << op.getResultTypes()
                             << ") that remained live after conversion";
-  diag.attachNote(op->getUsers().begin()->getLoc())
-      << "see existing live user here: " << *op->getUsers().begin();
+  Operation *liveUser = *op->getUsers().begin();
+  diag.attachNote(liveUser->getLoc())
+      << "see existing live user here: "
+      << OpWithFlags(liveUser, OpPrintingFlags().assumeVerified());
   return failure();
 }
 
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..829ff2f183e53 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Sat Jul 18 19:24:41 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 19:24:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU (PR #210527)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210527@github.com>
Message-ID: <6a5c3569.9edb5c9d.280be3.ec07@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/210527

>From 0d7ff5b87542e7ecee78d6d33e6696b9338212b9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 17:41:56 +0000
Subject: [PATCH 1/3] [mlir][xegpu] Support N-D block transfers in
 VectorToXeGPU

Extend the vector.transfer_read/transfer_write lowerings so they can
produce N-D xegpu.load_nd/store_nd, not just 1D/2D, and relax the
out-of-bounds handling to match load_nd's implicit-zero padding.

Changes:
- Allow high-dim (rank > 2) vectors on the nd block path. The rank
  restriction in storeLoadPreconditions is gated behind a new
  allowHighDimVector flag so plain vector.load/store keep their 1D/2D
  limit while transfers accept any non-0D rank.
- Accept an out-of-bounds read whose padding is a zero constant or a
  ub.poison value (isZeroOrPoisonPadding): load_nd zero-fills
  out-of-bounds elements, and poison means those elements are
  don't-care, so both match its semantics.
- Restructure both patterns to attempt the nd block op first via a
  single canLowerToLoadNd/canLowerToStoreNd predicate, then fall back
  to the scattered path. The block path only realizes a minor-identity
  map (reads also handle an innermost-two-dims transpose via a trailing
  vector.transpose); any other permutation is routed to the scattered
  path, which permutes strides explicitly. This fixes silently dropping
  the permutation on high-dim transposed transfers.

Tests updated for the new high-dim block lowering, plus new
load_high_dim_transposed / store_high_dim_transposed coverage.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 268 ++++++++++--------
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir | 169 +++++++----
 .../transfer-write-to-xegpu.mlir              | 225 +++++++--------
 3 files changed, 380 insertions(+), 282 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..306ce6f1e8008 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -15,6 +15,7 @@
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/UB/IR/UBOps.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StructuredOpsUtils.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
@@ -49,14 +50,53 @@ static bool isZeroConstant(Value val) {
       .Default(false);
 }
 
+// Return true if the transfer padding value is compatible with the implicit
+// padding of an nd block load. LoadNdOp fills out-of-bounds elements with zero,
+// so a zero constant matches its semantics exactly. A poison padding means the
+// out-of-bounds elements are "don't care", so any implicit padding (including
+// zero) is also acceptable.
+static bool isZeroOrPoisonPadding(Value val) {
+  return isZeroConstant(val) || val.getDefiningOp<ub::PoisonOp>();
+}
+
+// Return true if the permutation map keeps every dimension in place except the
+// innermost two, which are swapped, e.g.:
+//   (d0, d1) -> (d1, d0)
+//   (d0, d1, d2) -> (d2, d1)
+//   (d0, d1, d2, d3) -> (d0, d1, d3, d2)
+// This is the only non-identity permutation an nd block load can realize (by
+// loading the untransposed block and applying a trailing vector.transpose).
+static bool isInnermostTwoDimsTransposed(AffineMap map) {
+  unsigned numResults = map.getNumResults();
+  if (numResults < 2)
+    return false;
+  MLIRContext *ctx = map.getContext();
+  unsigned numInputs = map.getNumInputs();
+  // All but the innermost two results must match the minor-identity map.
+  for (unsigned i = 0; i + 2 < numResults; ++i)
+    if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
+      return false;
+  // The innermost two results must be the last two input dims, swapped.
+  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+         map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
+}
+
 static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
                                             Operation *op, VectorType vecTy,
-                                            MemRefType memTy) {
+                                            MemRefType memTy,
+                                            bool allowHighDimVector = false) {
   // Validate only vector as the basic vector store and load ops guarantee
   // XeGPU-compatible memref source.
   unsigned vecRank = vecTy.getRank();
-  if (!(vecRank == 1 || vecRank == 2))
+  // nd block loads/stores support N-D block transfers, so callers that lower to
+  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
+  // vector.load/store lowering keeps the 1D/2D restriction.
+  if (allowHighDimVector) {
+    if (vecRank == 0)
+      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
+  } else if (!(vecRank == 1 || vecRank == 2)) {
     return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
+  }
 
   if (!vecTy.getElementType().isIntOrFloat())
     return rewriter.notifyMatchFailure(
@@ -594,92 +634,78 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(readOp);
-    // Lower to scattered load Op if the target HW doesn't have 2d block load
-    // support and the load is not from shared memory.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        readOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (isOutOfBounds)
-        return failure();
-      return lowerToScatteredLoadOp(readOp, rewriter);
-    }
-
-    // Handle the 1D non-SLM case using load.gather.
-    if (loadedVecTy.getRank() == 1 && !isOutOfBounds)
-      return lowerToScatteredLoadOp(readOp, rewriter);
-
-    // Perform common data transfer checks.
-    // TODO: Maybe too strict for SLM case.
-    if (failed(
-            storeLoadPreconditions(rewriter, readOp, loadedVecTy, readMemTy)))
-      return failure();
-
-    if (isOutOfBounds && !isZeroConstant(readOp.getPadding()))
-      return rewriter.notifyMatchFailure(
-          readOp, "Unsupported non-zero padded out-of-bounds read");
+    bool hasBlockLoadSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // An nd block load can realize a minor-identity map directly, or an
+    // innermost-two-dims transpose via a trailing vector.transpose. Any other
+    // permutation (e.g. a mid-vector transpose of a high-dim load) is left to
+    // the scattered path, which permutes strides explicitly.
     AffineMap readMap = readOp.getPermutationMap();
-    // Check if this is a transpose: the map must have exactly 2 results,
-    // and those 2 results must be the last 2 input dimensions interchanged.
-    // Examples:
-    //   (d0, d1) -> (d1, d0)      // transpose
-    //   (d0, d1) -> (d0, d1)      // not a transpose
-    //   (d0, d1, d2) -> (d2, d1)  // transpose (last 2 dims swapped)
-    bool isTransposeLoad = false;
-    if (readMap.getNumResults() == 2) {
-      auto results = readMap.getResults();
-      unsigned numInputs = readMap.getNumInputs();
-      if (numInputs >= 2) {
-        auto lastDim = getAffineDimExpr(numInputs - 1, readMap.getContext());
-        auto secondLastDim =
-            getAffineDimExpr(numInputs - 2, readMap.getContext());
-        isTransposeLoad =
-            (results[0] == lastDim && results[1] == secondLastDim);
+    bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
+
+    // Prefer an nd block load. It requires HW block-load support, a >1D vector
+    // of a scalar element type backed by a scalar-element memref, and a map the
+    // block load can realize. Out-of-bounds reads are allowed as long as the
+    // padding matches load_nd's implicit zero padding.
+    bool canLowerToLoadNd =
+        hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
+        (readMap.isMinorIdentity() || isTransposeLoad) &&
+        loadedVecTy.getElementType().isIntOrFloat() &&
+        readMemTy.getElementType().isIntOrFloat() &&
+        (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
+
+    if (canLowerToLoadNd) {
+      auto elementType = loadedVecTy.getElementType();
+
+      SmallVector<int64_t> descShape(loadedVecTy.getShape());
+      if (isTransposeLoad) {
+        // If load is transposed, simply swap the last two dimensions of the
+        // loaded vector type to get the descriptor shape.
+        size_t rank = descShape.size();
+        assert(rank >= 2 && "Transpose requires at least 2 dimensions");
+        std::swap(descShape[rank - 1], descShape[rank - 2]);
+        loadedVecTy = VectorType::get(descShape, elementType);
       }
+      auto descType = xegpu::TensorDescType::get(
+          descShape, elementType, /*array_length=*/1,
+          /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, readOp.getBase(),
+          getAsOpFoldResult(readOp.getIndices()), loadedVecTy.getRank());
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      Operation *loadedOp =
+          xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
+                                  /*packed=*/nullptr, /*transpose=*/nullptr,
+                                  /*l1_hint=*/hint,
+                                  /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                  /*layout=*/nullptr);
+      if (isTransposeLoad) {
+        // Undo the innermost-two-dims swap with a trailing vector.transpose:
+        // keep the leading dimensions in place and interchange only the last
+        // two.
+        int64_t rank = loadedVecTy.getRank();
+        SmallVector<int64_t> perm(llvm::to_vector(llvm::seq<int64_t>(0, rank)));
+        std::swap(perm[rank - 1], perm[rank - 2]);
+        loadedOp = vector::TransposeOp::create(rewriter, loc,
+                                               loadedOp->getResult(0), perm);
+      }
+      rewriter.replaceOp(readOp, loadedOp);
+      return success();
     }
-    auto elementType = loadedVecTy.getElementType();
-
-    SmallVector<int64_t> descShape(loadedVecTy.getShape());
-    if (isTransposeLoad) {
-      // If load is transposed, simply swap the last two dimensions of the
-      // loaded vector type to get the descriptor shape.
-      size_t rank = descShape.size();
-      assert(rank >= 2 && "Transpose requires at least 2 dimensions");
-      std::swap(descShape[rank - 1], descShape[rank - 2]);
-      loadedVecTy = VectorType::get(descShape, elementType);
-    }
-    auto descType = xegpu::TensorDescType::get(
-        descShape, elementType, /*array_length=*/1,
-        /*boundary_check=*/isOutOfBounds, xegpu::MemorySpace::Global);
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, readOp.getBase(), getAsOpFoldResult(readOp.getIndices()),
-        loadedVecTy.getRank());
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    Operation *loadedOp =
-        xegpu::LoadNdOp::create(rewriter, loc, loadedVecTy, ndDesc, indices,
-                                /*packed=*/nullptr, /*transpose=*/nullptr,
-                                /*l1_hint=*/hint,
-                                /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                /*layout=*/nullptr);
-    if (isTransposeLoad) {
-      // Transposing the loaded vector with a separate vector.transpose
-      // operation
-      auto range = llvm::seq<int64_t>(0, readMap.getResults().size());
-      SmallVector<int64_t> perm(
-          range.rbegin(), range.rend()); // reverse the range for transpose
-      loadedOp = vector::TransposeOp::create(rewriter, loc,
-                                             loadedOp->getResult(0), perm);
-    }
-    rewriter.replaceOp(readOp, loadedOp);
 
-    return success();
+    // Fall back to a scattered load. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (isOutOfBounds)
+      return failure();
+    return lowerToScatteredLoadOp(readOp, rewriter);
   }
 };
 
@@ -727,47 +753,51 @@ struct TransferWriteLowering
       return success();
     }
 
-    // TODO:This check needs to be replaced with proper uArch capability check
+    // TODO: This check needs to be replaced with proper uArch capability check.
     auto chip = xegpu::getChipStr(writeOp);
-    // Lower to scattered store Op if the target HW doesn't have 2d block
-    // store support and the memref is not SLM.
-    if ((chip != "pvc" && chip != "bmg" && chip != "cri") ||
-        writeOp.getVectorType().getRank() > 2) {
-
-      // TODO: add support for OutOfBound access
-      if (writeOp.hasOutOfBoundsDim())
-        return failure();
-      return lowerToScatteredStoreOp(writeOp, rewriter);
-    }
-
-    if (failed(storeLoadPreconditions(rewriter, writeOp, vecTy, writeMemTy)))
-      return failure();
+    bool hasBlockStoreSupport =
+        (chip == "pvc" || chip == "bmg" || chip == "cri");
 
+    // Prefer an nd block store. It requires HW block-store support, a >1D
+    // vector of a scalar element type backed by a scalar-element memref, and a
+    // minor-identity map (block stores have no transpose support). Out-of-bounds
+    // writes are handled by the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    if (!map.isMinorIdentity())
-      return rewriter.notifyMatchFailure(writeOp, "Expects identity map");
-
-    auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
-        rewriter, loc, writeOp.getBase(),
-        getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
-
-    auto descType = xegpu::TensorDescType::get(
-        vecTy.getShape(), vecTy.getElementType(),
-        /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
-        xegpu::MemorySpace::Global);
-    // By default, no specific caching policy is assigned.
-    xegpu::CachePolicyAttr hint = nullptr;
-    xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
-        rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
-
-    auto storeOp = xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(),
-                                            ndDesc, indices,
-                                            /*l1_hint=*/hint,
-                                            /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                            /*layout=*/nullptr);
-    rewriter.replaceOp(writeOp, storeOp);
+    bool canLowerToStoreNd =
+        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
+        vecTy.getElementType().isIntOrFloat() &&
+        writeMemTy.getElementType().isIntOrFloat();
+
+    if (canLowerToStoreNd) {
+      auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
+          rewriter, loc, writeOp.getBase(),
+          getAsOpFoldResult(writeOp.getIndices()), vecTy.getRank());
+
+      auto descType = xegpu::TensorDescType::get(
+          vecTy.getShape(), vecTy.getElementType(),
+          /*array_length=*/1, /*boundary_check=*/writeOp.hasOutOfBoundsDim(),
+          xegpu::MemorySpace::Global);
+      // By default, no specific caching policy is assigned.
+      xegpu::CachePolicyAttr hint = nullptr;
+      xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
+          rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
+
+      auto storeOp =
+          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
+                                   indices,
+                                   /*l1_hint=*/hint,
+                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
+                                   /*layout=*/nullptr);
+      rewriter.replaceOp(writeOp, storeOp);
+      return success();
+    }
 
-    return success();
+    // Fall back to a scattered store. It supports arbitrary permutations and any
+    // rank, but cannot express out-of-bounds accesses.
+    // TODO: add support for OutOfBound access.
+    if (writeOp.hasOutOfBoundsDim())
+      return failure();
+    return lowerToScatteredStoreOp(writeOp, rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index 3354868cd6cb9..ed495729f9add 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -182,6 +182,33 @@ gpu.func @load_transpose_3d_memref(%source: memref<32x64x128xf32>,
 
 }
 
+// -----
+// A high-dim load whose innermost two dims are transposed lowers to an
+// (untransposed) nd block load followed by a vector.transpose of the last two
+// dims.
+gpu.module @xevm_module {
+gpu.func @load_high_dim_transposed(%source: memref<2x2x64x128xf16>,
+    %offset: index) -> vector<1x1x64x128xf16> {
+  %c0 = arith.constant 0.0 : f16
+  %0 = vector.transfer_read %source[%offset, %offset, %offset, %offset], %c0
+    {permutation_map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3, d2)>,
+    in_bounds = [true, true, true, true]}
+    : memref<2x2x64x128xf16>, vector<1x1x64x128xf16>
+  gpu.return %0 : vector<1x1x64x128xf16>
+}
+
+// LOAD-ND-LABEL:  @load_high_dim_transposed(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x64x128xf16>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x64x128xf16>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<1x1x128x64xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<1x1x128x64xf16>
+// LOAD-ND:        vector.transpose %[[VEC]], [0, 1, 3, 2] : vector<1x1x128x64xf16> to vector<1x1x64x128xf16>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_transposed(
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load {{.*}} : i64, vector<1x1x64x128xindex>, vector<1x1x64x128xi1> -> vector<1x1x64x128xf16>
+}
+
 // -----
 gpu.module @xevm_module {
 gpu.func @load_dynamic_source(%source: memref<?x?x?xf32>,
@@ -275,21 +302,31 @@ gpu.func @load_dynamic_source3(%source: memref<?x?x?x?x?xf32>,
   gpu.return %0 : vector<2x4x8x16xf32>
 }
 
-// CHECK-LABEL:  @load_dynamic_source3(
-// CHECK-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
-// CHECK:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
-// CHECK-COUNT4: vector.step
-// CHECK-COUNT3: vector.broadcast
-// CHECK-COUNT4: vector.shape_cast
-// CHECK-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
-// CHECK-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
-// CHECK:        return %[[VEC]]
+// LOAD-ND-LABEL:  @load_dynamic_source3(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFF1:.*]], %[[SIZES:.*]]:4, %[[STRIDES:.*]]:4 = memref.extract_strided_metadata %[[SUBVIEW]]
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x4x8x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x4x8x16xf32>
+// LOAD-ND:        return %[[VEC]]
+
+// LOAD-GATHER-LABEL:  @load_dynamic_source3(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<?x?x?x?x?xf32>
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x4x8x16xi1>
+// LOAD-GATHER:        memref.extract_strided_metadata %[[SRC]] : memref<?x?x?x?x?xf32> -> memref<f32>, index, index, index, index, index, index, index, index, index, index, index
+// LOAD-GATHER-COUNT4: vector.step
+// LOAD-GATHER-COUNT3: vector.broadcast
+// LOAD-GATHER-COUNT4: vector.shape_cast
+// LOAD-GATHER-COUNT4: vector.broadcast {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER-COUNT3: arith.addi {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x4x8x16xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<?x?x?x?x?xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<2x4x8x16xindex>, vector<2x4x8x16xi1> -> vector<2x4x8x16xf32>
+// LOAD-GATHER:        return %[[VEC]]
 }
 
 // -----
@@ -302,21 +339,27 @@ gpu.func @load_high_dim_vector(%source: memref<16x32x64xf32>,
   gpu.return %0 : vector<8x16x32xf32>
 }
 
-// CHECK-LABEL:  @load_high_dim_vector(
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
+// LOAD-ND-LABEL:  @load_high_dim_vector(
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %arg0 : memref<16x32x64xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<8x16x32xf32>
+
+// LOAD-GATHER-LABEL:  @load_high_dim_vector(
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// LOAD-GATHER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// LOAD-GATHER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// LOAD-GATHER:        %[[C2048:.+]] = arith.constant 2048 : index
+// LOAD-GATHER:        %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-COUNT3: vector.step
+// LOAD-GATHER-COUNT3: vector.shape_cast
+// LOAD-GATHER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %arg0 : memref<16x32x64xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<8x16x32xindex>, vector<8x16x32xi1> -> vector<8x16x32xf32>
 
 }
 
@@ -330,18 +373,25 @@ gpu.func @load_8D_vector(%source: memref<2x2x2x2x2x2x2x2xf32>,
   gpu.return %0 : vector<2x2x2x2x2x2x2x2xf32>
 }
 
-// CHECK-LABEL:  @load_8D_vector(
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-LABEL:  @load_8D_vector(
+// LOAD-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// LOAD-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]]
+// LOAD-ND-SAME:     -> vector<2x2x2x2x2x2x2x2xf32>
+
+// LOAD-GATHER-LABEL:  @load_8D_vector(
+// LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>,
+// LOAD-GATHER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// LOAD-GATHER-COUNT8: vector.step
+// LOAD-GATHER-COUNT7: vector.shape_cast
+// LOAD-GATHER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// LOAD-GATHER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// LOAD-GATHER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// LOAD-GATHER:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1> -> vector<2x2x2x2x2x2x2x2xf32>
 
 }
 
@@ -728,23 +778,38 @@ gpu.func @transpose_1x1024x24x64(
   gpu.return
 }
 
-// CHECK-LABEL: @transpose_1x1024x24x64
-// CHECK-DAG: %[[C1536:.+]] = arith.constant 1536 : index
-// CHECK-DAG: %[[C64:.+]] = arith.constant 64 : index
-// CHECK-DAG: %[[C65536:.+]] = arith.constant 65536 : index
+// The vector.transpose is folded into the transfer_read via
+// CombineTransferReadOpTranspose, giving the read a mid-vector permutation map
+// (d0, d1, d2, d3) -> (d0, d2, d1, d3). An nd block load can only realize an
+// innermost-two-dims transpose, so the read falls back to the scattered path
+// while the identity-map write still lowers to store_nd.
+// LOAD-ND-LABEL: @transpose_1x1024x24x64
+// LOAD-ND-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-ND-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-ND:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-ND:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-ND:     %[[VEC:.+]] = xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-ND:     %[[WDESC:.+]] = xegpu.create_nd_tdesc %arg1 : memref<1x24x1024x64xf16>
+// LOAD-ND-SAME:  -> !xegpu.tensor_desc<1x1x16x8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:     xegpu.store_nd %[[VEC]], %[[WDESC]]
+
+// LOAD-GATHER-LABEL: @transpose_1x1024x24x64
+// LOAD-GATHER-DAG: %[[C1536:.+]] = arith.constant 1536 : index
+// LOAD-GATHER-DAG: %[[C64:.+]] = arith.constant 64 : index
+// LOAD-GATHER-DAG: %[[C65536:.+]] = arith.constant 65536 : index
 
 // Read from memref<1x1024x24x64xf16>, strides [1572864, 1536, 64, 1].
 // Scalar base offset: seq_off * 1536 (original dim1 stride),
 //                    block_id_x * 64  (original dim2 stride).
-// CHECK:     arith.muli %{{.+}}, %[[C1536]] : index
-// CHECK:     arith.muli %block_id_x, %[[C64]] : index
-// CHECK:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C1536]] : index
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C64]] : index
+// LOAD-GATHER:     xegpu.load {{.*}} -> vector<1x1x16x8xf16>
 
 // Write to memref<1x24x1024x64xf16>, strides [1572864, 65536, 64, 1].
 // Scalar base offset: block_id_x * 65536 (original dim1 stride),
 //                    seq_off * 64        (original dim2 stride).
-// CHECK:     arith.muli %block_id_x, %[[C65536]] : index
-// CHECK:     arith.muli %{{.+}}, %[[C64]] : index
-// CHECK:     xegpu.store {{.*}}
+// LOAD-GATHER:     arith.muli %block_id_x, %[[C65536]] : index
+// LOAD-GATHER:     arith.muli %{{.+}}, %[[C64]] : index
+// LOAD-GATHER:     xegpu.store {{.*}}
 
 }
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index 45bda46f738b2..ad7332097cc8f 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -11,36 +11,20 @@ gpu.func @store_1D_vector(%vec: vector<8xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @store_1D_vector(
-// STORE-ND-SAME:  %[[VEC:.+]]: vector<8xf32>,
-// STORE-ND-SAME:  %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-ND-SAME:  %[[OFFSET:.+]]: index
-// STORE-ND:       %[[ELEM_BYTES:.+]] = arith.constant 4 : index
-// STORE-ND:       %[[COLLAPSED:.+]] = memref.subview %[[SRC]][%[[OFFSET]], %[[OFFSET]], 0]
-// STORE-ND:       %[[BASE_BUFFER:.+]], %[[OFFSET1:.+]], %[[SIZES:.+]], %[[STRIDES:.+]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND-SAME:    : memref<32xf32, strided<[1], offset: ?>> -> memref<f32>, index, index, index
-// STORE-ND:       %[[INTPTR:.+]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND-SAME:    : memref<f32> -> index
-// STORE-ND:       %[[MUL:.+]] = arith.muli %[[OFFSET1]], %[[ELEM_BYTES]] : index
-// STORE-ND:       %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:       %[[I64PTR:.+]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:       %[[DESC:.+]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [32],
-// STORE-ND-SAME:                   strides : [1] : i64  -> !xegpu.tensor_desc<8xf32,
-// STORE-ND-SAME:    boundary_check = false
-// STORE-ND:       xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFFSET]]] : vector<8xf32>
-
-// STORE-SCATTER-LABEL:  @store_1D_vector(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
-// STORE-SCATTER-DAG:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER-DAG:        %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: arith.muli {{.*}} : index
-// STORE-SCATTER-COUNT2: arith.addi {{.*}} : index
-// STORE-SCATTER-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:       xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_1D_vector(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
+// CHECK-DAG:    %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK-DAG:    %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: arith.muli {{.*}} : index
+// CHECK-COUNT2: arith.addi {{.*}} : index
+// CHECK-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----
@@ -150,7 +134,7 @@ gpu.func @store_out_of_bounds(%vec: vector<8x16xf32>,
 
 // -----
 gpu.module @xevm_module {
-gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
+gpu.func @store_transposed(%vec: vector<8x16xf32>,
     %source: memref<32x64xf32>, %offset: index) {
   vector.transfer_write %vec, %source[%offset, %offset]
     {permutation_map = affine_map<(d0, d1) -> (d1, d0)>,
@@ -159,22 +143,38 @@ gpu.func @no_store_transposed(%vec: vector<8x16xf32>,
   gpu.return
 }
 
-// STORE-ND-LABEL: @no_store_transposed(
-// STORE-ND:       vector.transfer_write
+// An nd block store cannot transpose, so a transposed write lowers through the
+// scattered path identically on both the target and non-target runs.
+// CHECK-LABEL:  @store_transposed(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
+// CHECK-SAME:   %[[OFFSET:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
+// CHECK-COUNT2: %[[STEP:.+]] = vector.step
+// CHECK-COUNT2: vector.shape_cast {{.*}}
+// CHECK-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
+// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
+// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
+// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+}
 
-// STORE-SCATTER-LABEL:  @no_store_transposed(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16xf32>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<32x64xf32>,
-// STORE-SCATTER-SAME:   %[[OFFSET:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16xi1>
-// STORE-SCATTER-COUNT2: %[[STEP:.+]] = vector.step
-// STORE-SCATTER-COUNT2: vector.shape_cast {{.*}}
-// STORE-SCATTER-COUNT2: vector.broadcast {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[BCAST2:.+]] = vector.broadcast {{.*}} : index to vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST2]], {{.*}} : vector<8x16xindex>
-// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<32x64xf32> -> index
-// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8x16xf32>, i64, vector<8x16xindex>, vector<8x16xi1>
+// -----
+// A high-dim transposed write cannot use an nd block store (no transpose
+// support), so it lowers through the scattered path on both runs.
+gpu.module @xevm_module {
+gpu.func @store_high_dim_transposed(%vec: vector<8x16x4xf32>,
+    %source: memref<16x32x64xf32>, %offset: index) {
+  vector.transfer_write %vec, %source[%offset, %offset, %offset]
+    {permutation_map = affine_map<(d0, d1, d2) -> (d0, d2, d1)>,
+    in_bounds = [true, true, true]}
+    : vector<8x16x4xf32>, memref<16x32x64xf32>
+  gpu.return
+}
+
+// CHECK-LABEL:  @store_high_dim_transposed(
+// CHECK:        xegpu.store {{.*}} : vector<8x16x4xf32>, i64, vector<8x16x4xindex>, vector<8x16x4xi1>
 }
 
 // -----
@@ -187,23 +187,31 @@ gpu.func @store_high_dim_vector(%vec: vector<8x16x32xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_high_dim_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
-// CHECK:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
-// CHECK:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
-// CHECK:        %[[C2048:.+]] = arith.constant 2048 : index
-// CHECK:        %[[C64:.+]] = arith.constant 64 : index
-// CHECK-COUNT3: vector.step
-// CHECK-COUNT3: vector.shape_cast
-// CHECK-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
-// CHECK-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
+// STORE-ND-LABEL:  @store_high_dim_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<16x32x64xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<8x16x32xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<8x16x32xf32>, !xegpu.tensor_desc<8x16x32xf32
+
+// STORE-SCATTER-LABEL:  @store_high_dim_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8x16x32xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<16x32x64xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8x16x32xi1>
+// STORE-SCATTER:        %[[CST_0:.+]] = arith.constant dense<64> : vector<16xindex>
+// STORE-SCATTER:        %[[CST_1:.+]] = arith.constant dense<2048> : vector<8xindex>
+// STORE-SCATTER:        %[[C2048:.+]] = arith.constant 2048 : index
+// STORE-SCATTER:        %[[C64:.+]] = arith.constant 64 : index
+// STORE-SCATTER-COUNT3: vector.step
+// STORE-SCATTER-COUNT3: vector.shape_cast
+// STORE-SCATTER-COUNT3: vector.broadcast {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER-COUNT2: arith.addi {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[BCASTOFF:.+]] = vector.broadcast {{.*}} : index to vector<8x16x32xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[BCASTOFF]], {{.*}} : vector<8x16x32xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<16x32x64xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<8x16x32xf32>, i64, vector<8x16x32xindex>, vector<8x16x32xi1>
 }
 
 // -----
@@ -216,19 +224,27 @@ gpu.func @store_8D_vector(%vec: vector<2x2x2x2x2x2x2x2xf32>,
   gpu.return
 }
 
-// CHECK-LABEL:  @store_8D_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
-// CHECK-COUNT8: vector.step
-// CHECK-COUNT7: vector.shape_cast
-// CHECK-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
+// STORE-ND-LABEL:  @store_8D_vector(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32>
+// STORE-ND-SAME:     -> !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]]
+// STORE-ND-SAME:     : vector<2x2x2x2x2x2x2x2xf32>, !xegpu.tensor_desc<2x2x2x2x2x2x2x2xf32
+
+// STORE-SCATTER-LABEL:  @store_8D_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<2x2x2x2x2x2x2x2xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<2x2x2x2x2x2x2x2xf32>
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<2x2x2x2x2x2x2x2xi1>
+// STORE-SCATTER-COUNT8: vector.step
+// STORE-SCATTER-COUNT7: vector.shape_cast
+// STORE-SCATTER-COUNT8: vector.broadcast {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER-COUNT7: arith.addi {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], {{.*}} : vector<2x2x2x2x2x2x2x2xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<2x2x2x2x2x2x2x2xf32> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]][%[[IDX]]], %[[CST]] : vector<2x2x2x2x2x2x2x2xf32>, i64, vector<2x2x2x2x2x2x2x2xindex>, vector<2x2x2x2x2x2x2x2xi1>
 }
 
 // -----
@@ -330,41 +346,28 @@ gpu.func @store_to_subview(%vec: vector<8xf16>,
       : vector<8xf16>, memref<256x256xf16, strided<[4096, 1], offset: ?>>
   gpu.return
 }
-// STORE-ND-LABEL:  @store_to_subview(
-// STORE-ND-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-ND-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-ND-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-ND:        %[[ELEM_BYTES:.+]] = arith.constant 2 : index
-// STORE-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1] : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SUBVIEW]][%[[OFF2]], 0]
-// STORE-ND:        %[[BASE_BUFFER:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
-// STORE-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
-// STORE-ND:        %[[MUL:.+]] = arith.muli %[[OFFSET]], %[[ELEM_BYTES]] : index
-// STORE-ND:        %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
-// STORE-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
-// STORE-ND:        %[[DESC:.*]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [256], strides : [1] : i64 ->
-// STORE-ND-SAME:                    !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
-// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFF2]]] : vector<8xf16>
-
-// STORE-SCATTER-LABEL:  @store_to_subview(
-// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// STORE-SCATTER-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// STORE-SCATTER:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
-// STORE-SCATTER-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// STORE-SCATTER:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
-// STORE-SCATTER:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// STORE-SCATTER:        arith.muli {{.*}} : index
-// STORE-SCATTER:        arith.addi %[[OFFSET]]{{.*}} : index
-// STORE-SCATTER:        arith.addi {{.*}} : index
-// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
-// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
-// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
+// A 1D store cannot use an nd block store, so it lowers through the scattered
+// path identically on both runs, even when the destination is a strided
+// subview.
+// CHECK-LABEL:  @store_to_subview(
+// CHECK-SAME:   %[[VEC:.+]]: vector<8xf16>,
+// CHECK-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
+// CHECK-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
+// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// CHECK:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
+// CHECK-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
+// CHECK:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
+// CHECK:        %[[STEP:.+]] = vector.step : vector<8xindex>
+// CHECK:        arith.muli {{.*}} : index
+// CHECK:        arith.addi %[[OFFSET]]{{.*}} : index
+// CHECK:        arith.addi {{.*}} : index
+// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
+// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
+// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
+// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----

>From 0c33ce78ecc38f5ab40ebc922a0ddc5306369df5 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 18 Jul 2026 21:54:01 +0000
Subject: [PATCH 2/3] [mlir][xegpu] Refactor transfer lowering; inline
 storeLoadPreconditions

Restructure TransferReadLowering/TransferWriteLowering to try the nd block
op first via a single canLowerTo{Load,Store}Nd predicate, then fall back to
the scattered path (guarding out-of-bounds, which scatter cannot express).

Remove storeLoadPreconditions and inline its remaining checks into the plain
vector.load/store patterns. Drop the redundant vector element-type check since
it always matches the memref element type.

No functional change.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 86 +++++++------------
 1 file changed, 30 insertions(+), 56 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 306ce6f1e8008..80bd88beece6d 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -77,42 +77,11 @@ static bool isInnermostTwoDimsTransposed(AffineMap map) {
     if (map.getResult(i) != getAffineDimExpr(numInputs - numResults + i, ctx))
       return false;
   // The innermost two results must be the last two input dims, swapped.
-  return map.getResult(numResults - 2) == getAffineDimExpr(numInputs - 1, ctx) &&
+  return map.getResult(numResults - 2) ==
+             getAffineDimExpr(numInputs - 1, ctx) &&
          map.getResult(numResults - 1) == getAffineDimExpr(numInputs - 2, ctx);
 }
 
-static LogicalResult storeLoadPreconditions(PatternRewriter &rewriter,
-                                            Operation *op, VectorType vecTy,
-                                            MemRefType memTy,
-                                            bool allowHighDimVector = false) {
-  // Validate only vector as the basic vector store and load ops guarantee
-  // XeGPU-compatible memref source.
-  unsigned vecRank = vecTy.getRank();
-  // nd block loads/stores support N-D block transfers, so callers that lower to
-  // them (transfer_read/transfer_write) allow any non-zero rank. The plain
-  // vector.load/store lowering keeps the 1D/2D restriction.
-  if (allowHighDimVector) {
-    if (vecRank == 0)
-      return rewriter.notifyMatchFailure(op, "Expects non-0D vector");
-  } else if (!(vecRank == 1 || vecRank == 2)) {
-    return rewriter.notifyMatchFailure(op, "Expects 1D or 2D vector");
-  }
-
-  if (!vecTy.getElementType().isIntOrFloat())
-    return rewriter.notifyMatchFailure(
-        op, "Expected scalar type with known bitwidth");
-
-  // XeGPU requires the memref to have a scalar integer or float element type.
-  // Memrefs with vector element types (e.g. memref<?xvector<4xf32>>) are not
-  // supported because createNdDescriptor computes byte offsets using
-  // getElementTypeBitWidth(), which asserts on non-integer/float types.
-  if (!memTy.getElementType().isIntOrFloat())
-    return rewriter.notifyMatchFailure(
-        op, "Unsupported memref element type: expected integer or float");
-
-  return success();
-}
-
 static LogicalResult transferPreconditions(PatternRewriter &rewriter,
                                            VectorTransferOpInterface xferOp) {
   if (xferOp.getMask())
@@ -647,13 +616,12 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
 
     // Prefer an nd block load. It requires HW block-load support, a >1D vector
-    // of a scalar element type backed by a scalar-element memref, and a map the
-    // block load can realize. Out-of-bounds reads are allowed as long as the
-    // padding matches load_nd's implicit zero padding.
+    // backed by a scalar-element memref, and a map the block load can realize.
+    // Out-of-bounds reads are allowed as long as the padding matches load_nd's
+    // implicit zero padding.
     bool canLowerToLoadNd =
         hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
         (readMap.isMinorIdentity() || isTransposeLoad) &&
-        loadedVecTy.getElementType().isIntOrFloat() &&
         readMemTy.getElementType().isIntOrFloat() &&
         (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
 
@@ -759,14 +727,13 @@ struct TransferWriteLowering
         (chip == "pvc" || chip == "bmg" || chip == "cri");
 
     // Prefer an nd block store. It requires HW block-store support, a >1D
-    // vector of a scalar element type backed by a scalar-element memref, and a
-    // minor-identity map (block stores have no transpose support). Out-of-bounds
-    // writes are handled by the descriptor's boundary check.
+    // vector backed by a scalar-element memref, and a minor-identity map (block
+    // stores have no transpose support). Out-of-bounds writes are handled by
+    // the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    bool canLowerToStoreNd =
-        hasBlockStoreSupport && vecTy.getRank() > 1 && map.isMinorIdentity() &&
-        vecTy.getElementType().isIntOrFloat() &&
-        writeMemTy.getElementType().isIntOrFloat();
+    bool canLowerToStoreNd = hasBlockStoreSupport && vecTy.getRank() > 1 &&
+                             map.isMinorIdentity() &&
+                             writeMemTy.getElementType().isIntOrFloat();
 
     if (canLowerToStoreNd) {
       auto [src, indices] = convertMemrefAndOffsetsToTargetRank(
@@ -782,18 +749,17 @@ struct TransferWriteLowering
       xegpu::CreateNdDescOp ndDesc = createNdDescriptor(
           rewriter, loc, descType, dyn_cast<TypedValue<MemRefType>>(src));
 
-      auto storeOp =
-          xegpu::StoreNdOp::create(rewriter, loc, writeOp.getVector(), ndDesc,
-                                   indices,
-                                   /*l1_hint=*/hint,
-                                   /*l2_hint=*/hint, /*l3_hint=*/hint,
-                                   /*layout=*/nullptr);
+      auto storeOp = xegpu::StoreNdOp::create(
+          rewriter, loc, writeOp.getVector(), ndDesc, indices,
+          /*l1_hint=*/hint,
+          /*l2_hint=*/hint, /*l3_hint=*/hint,
+          /*layout=*/nullptr);
       rewriter.replaceOp(writeOp, storeOp);
       return success();
     }
 
-    // Fall back to a scattered store. It supports arbitrary permutations and any
-    // rank, but cannot express out-of-bounds accesses.
+    // Fall back to a scattered store. It supports arbitrary permutations and
+    // any rank, but cannot express out-of-bounds accesses.
     // TODO: add support for OutOfBound access.
     if (writeOp.hasOutOfBoundsDim())
       return failure();
@@ -878,8 +844,12 @@ struct LoadLowering : public OpRewritePattern<vector::LoadOp> {
 
     VectorType vecTy = loadOp.getResult().getType();
     MemRefType memTy = loadOp.getBase().getType();
-    if (failed(storeLoadPreconditions(rewriter, loadOp, vecTy, memTy)))
-      return failure();
+    // The plain vector.load lowering only supports 1D/2D block loads.
+    if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
+      return rewriter.notifyMatchFailure(loadOp, "Expects 1D or 2D vector");
+    if (!memTy.getElementType().isIntOrFloat())
+      return rewriter.notifyMatchFailure(
+          loadOp, "Unsupported memref element type: expected integer or float");
 
     // Boundary check is available only for block instructions.
     bool boundaryCheck = vecTy.getRank() > 1;
@@ -918,8 +888,12 @@ struct StoreLowering : public OpRewritePattern<vector::StoreOp> {
     TypedValue<VectorType> vector = storeOp.getValueToStore();
     VectorType vecTy = vector.getType();
     MemRefType memTy = storeOp.getBase().getType();
-    if (failed(storeLoadPreconditions(rewriter, storeOp, vecTy, memTy)))
-      return failure();
+    // The plain vector.store lowering only supports 1D/2D block stores.
+    if (vecTy.getRank() != 1 && vecTy.getRank() != 2)
+      return rewriter.notifyMatchFailure(storeOp, "Expects 1D or 2D vector");
+    if (!memTy.getElementType().isIntOrFloat())
+      return rewriter.notifyMatchFailure(
+          storeOp, "Unsupported memref element type: expected integer or float");
 
     // Boundary check is available only for block instructions.
     bool boundaryCheck = vecTy.getRank() > 1;

>From 09599a7d662d5d0fccfc266b259d6b3057a8d2ea Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sun, 19 Jul 2026 02:24:17 +0000
Subject: [PATCH 3/3] [mlir][xegpu] Keep 1D transfers on the nd block path

Route rank-1 transfer_read/transfer_write to load_nd/store_nd on
block-capable hardware (>0 rank), preserving the original per-op lowering
instead of diverting 1D transfers to the scattered path. Restore the 1D
store test expectations to the split block/scatter form.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           |  17 +--
 .../VectorToXeGPU/transfer-read-to-xegpu.mlir |  45 ++++----
 .../transfer-write-to-xegpu.mlir              | 101 +++++++++++-------
 3 files changed, 99 insertions(+), 64 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 80bd88beece6d..727c98aef6619 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -615,12 +615,12 @@ struct TransferReadLowering : public OpRewritePattern<vector::TransferReadOp> {
     AffineMap readMap = readOp.getPermutationMap();
     bool isTransposeLoad = isInnermostTwoDimsTransposed(readMap);
 
-    // Prefer an nd block load. It requires HW block-load support, a >1D vector
-    // backed by a scalar-element memref, and a map the block load can realize.
-    // Out-of-bounds reads are allowed as long as the padding matches load_nd's
-    // implicit zero padding.
+    // Prefer an nd block load. It requires HW block-load support, a non-0D
+    // vector backed by a scalar-element memref, and a map the block load can
+    // realize. Out-of-bounds reads are allowed as long as the padding matches
+    // load_nd's implicit zero padding.
     bool canLowerToLoadNd =
-        hasBlockLoadSupport && loadedVecTy.getRank() > 1 &&
+        hasBlockLoadSupport && loadedVecTy.getRank() > 0 &&
         (readMap.isMinorIdentity() || isTransposeLoad) &&
         readMemTy.getElementType().isIntOrFloat() &&
         (!isOutOfBounds || isZeroOrPoisonPadding(readOp.getPadding()));
@@ -726,12 +726,12 @@ struct TransferWriteLowering
     bool hasBlockStoreSupport =
         (chip == "pvc" || chip == "bmg" || chip == "cri");
 
-    // Prefer an nd block store. It requires HW block-store support, a >1D
+    // Prefer an nd block store. It requires HW block-store support, a non-0D
     // vector backed by a scalar-element memref, and a minor-identity map (block
     // stores have no transpose support). Out-of-bounds writes are handled by
     // the descriptor's boundary check.
     AffineMap map = writeOp.getPermutationMap();
-    bool canLowerToStoreNd = hasBlockStoreSupport && vecTy.getRank() > 1 &&
+    bool canLowerToStoreNd = hasBlockStoreSupport && vecTy.getRank() > 0 &&
                              map.isMinorIdentity() &&
                              writeMemTy.getElementType().isIntOrFloat();
 
@@ -893,7 +893,8 @@ struct StoreLowering : public OpRewritePattern<vector::StoreOp> {
       return rewriter.notifyMatchFailure(storeOp, "Expects 1D or 2D vector");
     if (!memTy.getElementType().isIntOrFloat())
       return rewriter.notifyMatchFailure(
-          storeOp, "Unsupported memref element type: expected integer or float");
+          storeOp,
+          "Unsupported memref element type: expected integer or float");
 
     // Boundary check is available only for block instructions.
     bool boundaryCheck = vecTy.getRank() > 1;
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
index ed495729f9add..e22e682053a2a 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-read-to-xegpu.mlir
@@ -11,15 +11,21 @@ gpu.func @load_1D_vector(%source: memref<8x16x32xf32>, %offset: index) -> vector
 
 // LOAD-ND-LABEL:  @load_1D_vector(
 // LOAD-ND-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
-// LOAD-ND:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// LOAD-ND:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// LOAD-ND-COUNT2: arith.muli {{.*}} : index
-// LOAD-ND-COUNT2: arith.addi {{.*}} : index
-// LOAD-ND:        %[[SPLAT:.+]] = vector.broadcast {{.*}}:  index to vector<8xindex>
-// LOAD-ND:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// LOAD-ND:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
-// LOAD-ND:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// LOAD-ND:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<8xindex>, vector<8xi1> -> vector<8xf32>
+// LOAD-ND-SAME:   %[[OFFSET:.+]]: index
+// LOAD-ND:        %[[ELEM_BYTES:.+]] = arith.constant 4 : index
+// LOAD-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SRC]][%[[OFFSET]], %[[OFFSET]], 0]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFF1:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
+// LOAD-ND-SAME:     : memref<32xf32, strided<[1], offset: ?>> -> memref<f32>, index, index, index
+// LOAD-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
+// LOAD-ND-SAME:     : memref<f32> -> index
+// LOAD-ND:        %[[MUL:.*]] = arith.muli %[[OFF1]], %[[ELEM_BYTES]] : index
+// LOAD-ND:        %[[ADD:.*]] = arith.addi %[[INTPTR]], %[[MUL]] : index
+// LOAD-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
+// LOAD-ND:        %[[DESC:.+]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [32],
+// LOAD-ND-SAME:                   strides : [1] : i64 -> !xegpu.tensor_desc<8xf32,
+// LOAD-ND-SAME:     #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]][%[[OFFSET]]]
+// LOAD-ND-SAME:     : !xegpu.tensor_desc<8xf32, #xegpu.block_tdesc_attr<boundary_check = false>> -> vector<8xf32>
 
 // LOAD-GATHER-LABEL:  @load_1D_vector(
 // LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
@@ -547,18 +553,17 @@ gpu.func @load_from_subview_1D(%source: memref<4096x4096xf16>, %off1: index, %of
 // LOAD-ND-LABEL:  @load_from_subview_1D(
 // LOAD-ND-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
 // LOAD-ND-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// LOAD-ND:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// LOAD-ND:        %[[ELEM_BYTES:.+]] = arith.constant 2 : index
 // LOAD-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1] : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// LOAD-ND:        %[[BB:.+]], %[[OFFSET:.+]],{{.*}},{{.*}} = memref.extract_strided_metadata %[[SUBVIEW]] : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
-// LOAD-ND:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// LOAD-ND:        arith.muli {{.*}} : index
-// LOAD-ND:        arith.addi %[[OFFSET]]{{.*}} : index
-// LOAD-ND:        arith.addi {{.*}} : index
-// LOAD-ND:        %[[SPLAT:.+]] = vector.broadcast {{.*}}:  index to vector<8xindex>
-// LOAD-ND:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// LOAD-ND:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]] : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
-// LOAD-ND:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// LOAD-ND:        %[[VEC:.+]] = xegpu.load %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : i64, vector<8xindex>, vector<8xi1> -> vector<8xf16>
+// LOAD-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SUBVIEW]][%[[OFF2]], 0]
+// LOAD-ND:        %[[BASE_BUFFER:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
+// LOAD-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
+// LOAD-ND:        %[[MUL:.+]] = arith.muli %[[OFFSET]], %[[ELEM_BYTES]] : index
+// LOAD-ND:        %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
+// LOAD-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
+// LOAD-ND:        %[[DESC:.*]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [256], strides : [1] : i64 ->
+// LOAD-ND-SAME:                    !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// LOAD-ND:        %[[VEC:.+]] = xegpu.load_nd %[[DESC]][%[[OFF2]]] : !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>> -> vector<8xf16>
 
 // LOAD-GATHER-LABEL:  @load_from_subview_1D(
 // LOAD-GATHER-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
diff --git a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
index ad7332097cc8f..2d55a785595f7 100644
--- a/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/transfer-write-to-xegpu.mlir
@@ -11,20 +11,36 @@ gpu.func @store_1D_vector(%vec: vector<8xf32>,
   gpu.return
 }
 
-// A 1D store cannot use an nd block store, so it lowers through the scattered
-// path identically on both the target and non-target runs.
-// CHECK-LABEL:  @store_1D_vector(
-// CHECK-SAME:   %[[VEC:.+]]: vector<8xf32>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
-// CHECK-DAG:    %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// CHECK-DAG:    %[[STEP:.+]] = vector.step
-// CHECK-COUNT2: arith.muli {{.*}} : index
-// CHECK-COUNT2: arith.addi {{.*}} : index
-// CHECK-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// CHECK-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
-// CHECK-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
-// CHECK-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
+// STORE-ND-LABEL: @store_1D_vector(
+// STORE-ND-SAME:  %[[VEC:.+]]: vector<8xf32>,
+// STORE-ND-SAME:  %[[SRC:.+]]: memref<8x16x32xf32>,
+// STORE-ND-SAME:  %[[OFFSET:.+]]: index
+// STORE-ND:       %[[ELEM_BYTES:.+]] = arith.constant 4 : index
+// STORE-ND:       %[[COLLAPSED:.+]] = memref.subview %[[SRC]][%[[OFFSET]], %[[OFFSET]], 0]
+// STORE-ND:       %[[BASE_BUFFER:.+]], %[[OFFSET1:.+]], %[[SIZES:.+]], %[[STRIDES:.+]] = memref.extract_strided_metadata %[[COLLAPSED]]
+// STORE-ND-SAME:    : memref<32xf32, strided<[1], offset: ?>> -> memref<f32>, index, index, index
+// STORE-ND:       %[[INTPTR:.+]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
+// STORE-ND-SAME:    : memref<f32> -> index
+// STORE-ND:       %[[MUL:.+]] = arith.muli %[[OFFSET1]], %[[ELEM_BYTES]] : index
+// STORE-ND:       %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
+// STORE-ND:       %[[I64PTR:.+]] = arith.index_cast %[[ADD]] : index to i64
+// STORE-ND:       %[[DESC:.+]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [32],
+// STORE-ND-SAME:                   strides : [1] : i64  -> !xegpu.tensor_desc<8xf32,
+// STORE-ND-SAME:    boundary_check = false
+// STORE-ND:       xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFFSET]]] : vector<8xf32>
+
+// STORE-SCATTER-LABEL:  @store_1D_vector(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf32>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<8x16x32xf32>,
+// STORE-SCATTER-DAG:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// STORE-SCATTER-DAG:        %[[STEP:.+]] = vector.step
+// STORE-SCATTER-COUNT2: arith.muli {{.*}} : index
+// STORE-SCATTER-COUNT2: arith.addi {{.*}} : index
+// STORE-SCATTER-DAG:    %[[BCAST:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// STORE-SCATTER-DAG:    %[[IDX:.+]] = arith.addi %[[BCAST]], %{{.*}} : vector<8xindex>
+// STORE-SCATTER-DAG:    %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SRC]] : memref<8x16x32xf32> -> index
+// STORE-SCATTER-DAG:    %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:       xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf32>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----
@@ -346,28 +362,41 @@ gpu.func @store_to_subview(%vec: vector<8xf16>,
       : vector<8xf16>, memref<256x256xf16, strided<[4096, 1], offset: ?>>
   gpu.return
 }
-// A 1D store cannot use an nd block store, so it lowers through the scattered
-// path identically on both runs, even when the destination is a strided
-// subview.
-// CHECK-LABEL:  @store_to_subview(
-// CHECK-SAME:   %[[VEC:.+]]: vector<8xf16>,
-// CHECK-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
-// CHECK-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
-// CHECK:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
-// CHECK:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
-// CHECK-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
-// CHECK:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
-// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
-// CHECK:        %[[STEP:.+]] = vector.step : vector<8xindex>
-// CHECK:        arith.muli {{.*}} : index
-// CHECK:        arith.addi %[[OFFSET]]{{.*}} : index
-// CHECK:        arith.addi {{.*}} : index
-// CHECK:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
-// CHECK:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
-// CHECK:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
-// CHECK-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
-// CHECK:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
-// CHECK:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
+// STORE-ND-LABEL:  @store_to_subview(
+// STORE-ND-SAME:   %[[VEC:.+]]: vector<8xf16>,
+// STORE-ND-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
+// STORE-ND-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
+// STORE-ND:        %[[ELEM_BYTES:.+]] = arith.constant 2 : index
+// STORE-ND:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1] : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
+// STORE-ND:        %[[COLLAPSED:.+]] = memref.subview %[[SUBVIEW]][%[[OFF2]], 0]
+// STORE-ND:        %[[BASE_BUFFER:.*]], %[[OFFSET:.*]], %[[SIZES:.*]], %[[STRIDES:.*]] = memref.extract_strided_metadata %[[COLLAPSED]]
+// STORE-ND:        %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %[[BASE_BUFFER]]
+// STORE-ND:        %[[MUL:.+]] = arith.muli %[[OFFSET]], %[[ELEM_BYTES]] : index
+// STORE-ND:        %[[ADD:.+]] = arith.addi %[[INTPTR]], %[[MUL]] : index
+// STORE-ND:        %[[I64PTR:.*]] = arith.index_cast %[[ADD]] : index to i64
+// STORE-ND:        %[[DESC:.*]] = xegpu.create_nd_tdesc %[[I64PTR]], shape : [256], strides : [1] : i64 ->
+// STORE-ND-SAME:                    !xegpu.tensor_desc<8xf16, #xegpu.block_tdesc_attr<boundary_check = false>>
+// STORE-ND:        xegpu.store_nd %[[VEC]], %[[DESC]][%[[OFF2]]] : vector<8xf16>
+
+// STORE-SCATTER-LABEL:  @store_to_subview(
+// STORE-SCATTER-SAME:   %[[VEC:.+]]: vector<8xf16>,
+// STORE-SCATTER-SAME:   %[[SRC:.+]]: memref<4096x4096xf16>,
+// STORE-SCATTER-SAME:   %[[OFF1:.+]]: index, %[[OFF2:.+]]: index
+// STORE-SCATTER:        %[[CST:.+]] = arith.constant dense<true> : vector<8xi1>
+// STORE-SCATTER:        %[[SUBVIEW:.+]] = memref.subview %[[SRC]][%[[OFF1]], %[[OFF2]]] [256, 256] [1, 1]
+// STORE-SCATTER-SAME:     : memref<4096x4096xf16> to memref<256x256xf16, strided<[4096, 1], offset: ?>>
+// STORE-SCATTER:        %[[BB:.+]], %[[OFFSET:.+]], {{.*}}, {{.*}} = memref.extract_strided_metadata %[[SUBVIEW]]
+// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> memref<f16>, index, index, index, index, index
+// STORE-SCATTER:        %[[STEP:.+]] = vector.step : vector<8xindex>
+// STORE-SCATTER:        arith.muli {{.*}} : index
+// STORE-SCATTER:        arith.addi %[[OFFSET]]{{.*}} : index
+// STORE-SCATTER:        arith.addi {{.*}} : index
+// STORE-SCATTER:        %[[SPLAT:.+]] = vector.broadcast {{.*}} : index to vector<8xindex>
+// STORE-SCATTER:        %[[IDX:.+]] = arith.addi %[[SPLAT]], %[[STEP]] : vector<8xindex>
+// STORE-SCATTER:        %[[COLLAPSE:.+]] = memref.extract_aligned_pointer_as_index %[[SUBVIEW]]
+// STORE-SCATTER-SAME:     : memref<256x256xf16, strided<[4096, 1], offset: ?>> -> index
+// STORE-SCATTER:        %[[COLLAPSE_I:.+]] = arith.index_cast %[[COLLAPSE]] : index to i64
+// STORE-SCATTER:        xegpu.store %[[VEC]], %[[COLLAPSE_I]]{{\[}}%[[IDX]]{{\]}}, %[[CST]] : vector<8xf16>, i64, vector<8xindex>, vector<8xi1>
 }
 
 // -----


From llvmlistbot at llvm.org  Sat Jul 18 20:19:24 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 20:19:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a5c423c.c5504292.78ef3.3bc7@mx.google.com>


https://github.com/Jianhui-Li edited https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Sat Jul 18 20:19:25 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 20:19:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a5c423d.13be6df8.22eac4.e465@mx.google.com>


https://github.com/Jianhui-Li approved this pull request.

LGTM. Thanks!

https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Sat Jul 18 20:19:25 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Sat, 18 Jul 2026 20:19:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a5c423d.481afadf.9e047.ce95@mx.google.com>


================

----------------
Jianhui-Li wrote:

please add a swapped-direction test: [8,64]→[8,16] to exercises a different code path

https://github.com/llvm/llvm-project/pull/209822

From llvmlistbot at llvm.org  Sat Jul 18 22:29:53 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Sat, 18 Jul 2026 22:29:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5c60d1.d76d81af.23b347.cdcd@mx.google.com>


https://github.com/hamzaqureshi5 updated https://github.com/llvm/llvm-project/pull/210533

>From 40a7e8c8b2158d4b695cf33c2581b5de67fab6f5 Mon Sep 17 00:00:00 2001
From: Hamza Qureshi <hamza7771.861 at gmail.com>
Date: Sat, 18 Jul 2026 23:17:43 +0500
Subject: [PATCH] [MLIR][Affine] Fix fusion of producers that update a memref
 in place

Affine producer-consumer fusion places the fused nest at an insertion
point between the source and the destination nest. When the source nest
is retained after fusion (e.g., non-maximal fusion with an escaping
memref) and updates a memref in place (loads from and stores to it),
this placement is unsound: the fused nest recomputes the source's
values reading from that memref, but the retained source nest has
already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest,
where the in-place updated memrefs still hold their original values.
The recomputed stores are redirected to a private memref, so the
retained source nest also still reads the original values. An
anti-dependence edge from the destination to the source node is added
to the dependence graph to preserve this placement in subsequent fusion
decisions. When no such placement can be shown to preserve all
dependences, or the in-place updated memref can't be privatized, bail
out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g.,
initialize-then-reuse patterns) don't observe the memref's incoming
state and don't require the new placement.

Update @same_memref_load_store and @same_memref_load_multiple_stores,
whose CHECK lines encoded the miscompiled output; their fused nests are
unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490
---
 .../Dialect/Affine/Transforms/LoopFusion.cpp  | 140 ++++++++++++++++++
 mlir/test/Dialect/Affine/loop-fusion-4.mlir   |  87 ++++++++++-
 2 files changed, 221 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 


From llvmlistbot at llvm.org  Sat Jul 18 22:32:09 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sat, 18 Jul 2026 22:32:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
Message-ID: <llvm.org/llvm/llvm-project/pull/210571@github.com>


https://github.com/takatodo created https://github.com/llvm/llvm-project/pull/210571

## Summary

- Preserve discardable attributes when
  `AffineForOp::replaceWithAdditionalYields` rebuilds an `affine.for`.
- Extend the existing unroll-and-jam regression to verify that an inner-loop
  attribute survives the `LoopLikeOpInterface` yield-extension path.

## Motivation

`replaceWithAdditionalYields` creates a replacement loop with additional
iteration arguments, but did not copy the original loop's discardable
attribute dictionary. Consequently, scheduling or transformation annotations
attached to the original loop were silently lost when a transform added loop
yields.

The replacement loop now receives the original discardable attributes. This
does not add an API, pass, or option; it only preserves existing IR metadata
across the replacement.

## Testing

- Built `mlir-opt` from an assertion-enabled Release configuration based on
  `origin/main`.
- Ran all three `RUN` variants in
  `mlir/test/Dialect/Affine/unroll-jam.mlir`.


>From c04f3e6990e446ca867a1594b3e9140a28ad8e51 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 12:04:15 +0900
Subject: [PATCH] [MLIR][Affine] Preserve discardable attrs when adding loop
 yields

AffineForOp::replaceWithAdditionalYields rebuilds the loop but previously dropped discardable attributes attached by scheduling and transformation pipelines.

Copy the original discardable attribute dictionary to the replacement loop and add an unroll-and-jam regression that exercises the LoopLikeOpInterface path.
---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 1 +
 mlir/test/Dialect/Affine/unroll-jam.mlir | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..d4c4dd6b48424 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2893,6 +2893,7 @@ FailureOr<LoopLikeOpInterface> AffineForOp::replaceWithAdditionalYields(
   AffineForOp newLoop = AffineForOp::create(
       rewriter, getLoc(), getLowerBoundOperands(), getLowerBoundMap(),
       getUpperBoundOperands(), getUpperBoundMap(), getStepAsInt(), inits);
+  newLoop->setDiscardableAttrs(getOperation()->getDiscardableAttrDictionary());
 
   // Generate the new yield values and append them to the scf.yield operation.
   auto yieldOp = cast<AffineYieldOp>(getBody()->getTerminator());
diff --git a/mlir/test/Dialect/Affine/unroll-jam.mlir b/mlir/test/Dialect/Affine/unroll-jam.mlir
index 8ed7fccf7d251..bba24844c3484 100644
--- a/mlir/test/Dialect/Affine/unroll-jam.mlir
+++ b/mlir/test/Dialect/Affine/unroll-jam.mlir
@@ -185,7 +185,7 @@ func.func @unroll_jam_one_iter_arg() {
     %red = affine.for %j = 0 to 17 iter_args(%acc = %cst) -> (i32) {
       %y = "bar"(%i, %j, %acc) : (index, index, i32) -> i32
       affine.yield %y : i32
-    }
+    } {test.keep = "inner"}
     %w = "foo"(%i, %x, %red) : (index, i32, i32) -> i32
   }
   return
@@ -201,7 +201,7 @@ func.func @unroll_jam_one_iter_arg() {
 // CHECK-NEXT:     [[INC1:%[0-9]+]] = affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:     [[RES5:%[0-9]+]] = "bar"([[INC1]], [[IV1]], [[ACC2]])
 // CHECK-NEXT:     affine.yield [[RES4]], [[RES5]]
-// CHECK-NEXT:   }
+// CHECK-NEXT:   } {test.keep = "inner"}
 // CHECK:        "foo"([[IV0]], [[RES1]], [[RES3]]#0)
 // CHECK-NEXT:   affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:   "foo"({{.*}}, [[RES2]], [[RES3]]#1)


From llvmlistbot at llvm.org  Sat Jul 18 22:32:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 22:32:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5c6171.1905e68e.30d957.a68f@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @takatodo :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sat Jul 18 22:37:30 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sat, 18 Jul 2026 22:37:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5c629a.b4a331f0.d9742.020b@mx.google.com>


https://github.com/takatodo edited https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sat Jul 18 23:21:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 23:21:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5c6ce7.4a342cbc.3a2f36.e9a8@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210501

>From cd6ecf25efeece948522106a74e83d711c973027 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 18 Jul 2026 18:37:27 +0800
Subject: [PATCH 1/2] [MLIR][Python] Make Python-defined dialect loading
 context-aware

---
 mlir/include/mlir-c/IR.h                      |  7 ++
 mlir/lib/Bindings/Python/IRCore.cpp           |  8 +++
 mlir/lib/CAPI/IR/IR.cpp                       |  5 ++
 mlir/python/mlir/dialects/ext.py              | 29 ++++----
 mlir/test/python/dialects/ext.py              | 66 +++++++++++++++++++
 .../python/dialects/transform_op_interface.py |  2 +-
 ...ansform_pattern_descriptor_op_interface.py |  2 +-
 7 files changed, 100 insertions(+), 19 deletions(-)

diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..98ea0dfc00a1b 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                           MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.

>From b96aa343d82a203ecec263e7b291aea0a792c24e Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 14:21:12 +0800
Subject: [PATCH 2/2] address comment

---
 mlir/test/python/dialects/ext.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 6791cbe7fde65..707f86aaac521 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -121,8 +121,6 @@ class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
         attr: ContextLoadAttr
         result: Result[ContextLoadType]
 
-    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
-
     def check_dialect(context_name, type_value):
         i32 = IntegerType.get_signless(32)
         result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
@@ -139,12 +137,6 @@ def check_dialect(context_name, type_value):
         assert isinstance(op.attr, ContextLoadAttr)
         assert isinstance(op.result.type, ContextLoadType)
 
-        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
-        # CHECK: "first context"
-        # CHECK: 1 : i32
-        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
-        # CHECK: "second context"
-        # CHECK: 2 : i32
         print(
             f"{context_name}: {type(op).__name__}, "
             f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
@@ -160,16 +152,28 @@ def check_dialect(context_name, type_value):
         try:
             ContextLoadDialect.load()
         except RuntimeError as e:
+            # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
             print("same context:", e)
-        else:
-            raise AssertionError("loading a dialect twice in one context must fail")
 
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
         check_dialect("first context", 1)
 
     with second_context, Location.unknown():
         ContextLoadDialect.load()
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
         check_dialect("second context", 2)
 
+    with first_context, Location.unknown():
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            # CHECK: same context again: Dialect 'ext_context_load' has already been loaded in the current context.
+            print("same context again:", e)
+
 
 # CHECK: TEST: testExtDialect
 @run


From llvmlistbot at llvm.org  Sat Jul 18 23:22:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 23:22:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5c6d17.1905e68e.30d957.d593@mx.google.com>


================
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
----------------
PragmaTwice wrote:

Done.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 23:23:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 18 Jul 2026 23:23:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a5c6d55.510f576d.2f694c.92cf@mx.google.com>


================
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
----------------
PragmaTwice wrote:

Ahh `_register_dialect_impl` is for all upstream dialect registering, not only for python-defined dialects. So I don't think we can change its signature here.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Sat Jul 18 23:41:32 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sat, 18 Jul 2026 23:41:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a5c719c.f3151ff8.37ac76.da30@mx.google.com>


================
@@ -3391,8 +3391,10 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
                             << inputOperands.getTypes() << ") to ("
                             << op.getResultTypes()
                             << ") that remained live after conversion";
-  diag.attachNote(op->getUsers().begin()->getLoc())
-      << "see existing live user here: " << *op->getUsers().begin();
+  Operation *liveUser = *op->getUsers().begin();
+  diag.attachNote(liveUser->getLoc())
+      << "see existing live user here: "
+      << OpWithFlags(liveUser, OpPrintingFlags().assumeVerified());
----------------
matthias-springer wrote:

This is still problematic. Potentially invalid IR should be printed in the generic form.

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Sun Jul 19 03:15:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 03:15:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
Message-ID: <llvm.org/llvm/llvm-project/pull/210586@github.com>


https://github.com/PragmaTwice created https://github.com/llvm/llvm-project/pull/210586

Python implementations of `MemoryEffectsOpInterface` receive a `MemoryEffectInstancesList` (added in #176920), but currently have no generic way to populate it.

This adds the C API for constructing and appending memory effect instances and exposes it in Python through `MemoryEffect`, `SideEffectResource`, and `MemoryEffectInstancesList.append`. The tests use CSE and DCE to verify that the declared effects are observed by MLIR passes.

Assisted-by: GPT 5.6 Sol

>From ee4b1d4aa4c3c0afaf896ace40531d702551cb77 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 18:07:37 +0800
Subject: [PATCH] [MLIR][CAPI][Python] Add support for constructing memory
 effect instances

---
 mlir/include/mlir-c/Interfaces.h              |  78 +++++
 .../mlir/Bindings/Python/IRInterfaces.h       |  53 +++-
 mlir/include/mlir/CAPI/Interfaces.h           |   4 +
 mlir/lib/Bindings/Python/DialectTransform.cpp |  20 +-
 mlir/lib/Bindings/Python/IRInterfaces.cpp     | 107 ++++++-
 mlir/lib/CAPI/Interfaces/Interfaces.cpp       |  72 +++++
 mlir/test/CAPI/ir.c                           |  43 +++
 .../dialects/memory_effects_op_interface.py   | 291 ++++++++++++++++++
 8 files changed, 654 insertions(+), 14 deletions(-)
 create mode 100644 mlir/test/python/dialects/memory_effects_op_interface.py

diff --git a/mlir/include/mlir-c/Interfaces.h b/mlir/include/mlir-c/Interfaces.h
index a5251dc78471f..a416a6ab76f87 100644
--- a/mlir/include/mlir-c/Interfaces.h
+++ b/mlir/include/mlir-c/Interfaces.h
@@ -28,7 +28,10 @@ extern "C" {
   };                                                                           \
   typedef struct name name
 
+DEFINE_C_API_STRUCT(MlirMemoryEffect, void);
+DEFINE_C_API_STRUCT(MlirMemoryEffectInstance, void);
 DEFINE_C_API_STRUCT(MlirMemoryEffectInstancesList, void);
+DEFINE_C_API_STRUCT(MlirSideEffectResource, void);
 
 #undef DEFINE_C_API_STRUCT
 
@@ -144,6 +147,81 @@ mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectsOpInterface
 //===---------------------------------------------------------------------===//
 
+/// Returns the borrowed singleton instance of the allocate memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsAllocateGet(void);
+
+/// Returns the borrowed singleton instance of the free memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsFreeGet(void);
+
+/// Returns the borrowed singleton instance of the read memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsReadGet(void);
+
+/// Returns the borrowed singleton instance of the write memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsWriteGet(void);
+
+/// Returns the borrowed singleton instance of the default side effect
+/// resource.
+MLIR_CAPI_EXPORTED MlirSideEffectResource
+mlirSideEffectsDefaultResourceGet(void);
+
+/// Creates a memory effect instance without an associated IR entity.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation operand.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpOperand(MlirMemoryEffect effect,
+                                           MlirOpOperand opOperand,
+                                           MlirAttribute parameters, int stage,
+                                           bool effectOnFullRegion,
+                                           MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation result.
+/// `result` must wrap an OpResult. `parameters` may be a null attribute. The
+/// caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpResult(MlirMemoryEffect effect,
+                                          MlirValue result,
+                                          MlirAttribute parameters, int stage,
+                                          bool effectOnFullRegion,
+                                          MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a block argument.
+/// `blockArgument` must wrap a BlockArgument. `parameters` may be a null
+/// attribute. The caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a symbol. `symbol` must be
+/// a SymbolRefAttr. `parameters` may be a null attribute. The caller owns the
+/// returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForSymbol(MlirMemoryEffect effect,
+                                        MlirAttribute symbol,
+                                        MlirAttribute parameters, int stage,
+                                        bool effectOnFullRegion,
+                                        MlirSideEffectResource resource);
+
+/// Destroys a memory effect instance created by one of the functions above.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance);
+
+/// Appends a copy of `instance` to the given list. This does not take ownership
+/// of `instance`; the caller remains responsible for destroying it.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                    MlirMemoryEffectInstance instance);
+
 /// Returns the interface TypeID of the MemoryEffectsOpInterface.
 MLIR_CAPI_EXPORTED MlirTypeID mlirMemoryEffectsOpInterfaceTypeID(void);
 
diff --git a/mlir/include/mlir/Bindings/Python/IRInterfaces.h b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
index fb30e030b6c32..1ba995aa82c5e 100644
--- a/mlir/include/mlir/Bindings/Python/IRInterfaces.h
+++ b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
+class PyMemoryEffect {
+public:
+  explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
+
+  MlirMemoryEffect get() const { return effect; }
+
+private:
+  MlirMemoryEffect effect;
+};
+
+/// A borrowed side effect resource.
+class PySideEffectResource {
+public:
+  explicit PySideEffectResource(MlirSideEffectResource resource)
+      : resource(resource) {}
+
+  MlirSideEffectResource get() const { return resource; }
+
+private:
+  MlirSideEffectResource resource;
+};
+
+/// An owning memory effect instance.
+class PyMemoryEffectInstance {
+public:
+  explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
+      : instance(instance) {}
+  PyMemoryEffectInstance(PyMemoryEffectInstance &&other) noexcept
+      : instance(other.instance) {
+    other.instance.ptr = nullptr;
+  }
+  ~PyMemoryEffectInstance() {
+    if (instance.ptr)
+      mlirMemoryEffectInstanceDestroy(instance);
+  }
+
+  MlirMemoryEffectInstance get() const { return instance; }
+
+private:
+  MlirMemoryEffectInstance instance;
+};
+
+/// A callback-scoped view of a list of memory effect instances.
+class PyMemoryEffectsInstanceList {
+public:
+  explicit PyMemoryEffectsInstanceList(MlirMemoryEffectInstancesList effects)
+      : effects(effects) {}
+
+  MlirMemoryEffectInstancesList get() const { return effects; }
+
+private:
   MlirMemoryEffectInstancesList effects;
 };
 
diff --git a/mlir/include/mlir/CAPI/Interfaces.h b/mlir/include/mlir/CAPI/Interfaces.h
index 15afc9fb0f18e..55d850ccd01eb 100644
--- a/mlir/include/mlir/CAPI/Interfaces.h
+++ b/mlir/include/mlir/CAPI/Interfaces.h
@@ -22,5 +22,9 @@
 DEFINE_C_API_PTR_METHODS(
     MlirMemoryEffectInstancesList,
     llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance>)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffect, mlir::MemoryEffects::Effect)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffectInstance,
+                         mlir::MemoryEffects::EffectInstance)
+DEFINE_C_API_PTR_METHODS(MlirSideEffectResource, mlir::SideEffects::Resource)
 
 #endif // MLIR_CAPI_INTERFACES_H
diff --git a/mlir/lib/Bindings/Python/DialectTransform.cpp b/mlir/lib/Bindings/Python/DialectTransform.cpp
index bd72082cea7e8..f84189e3a22ff 100644
--- a/mlir/lib/Bindings/Python/DialectTransform.cpp
+++ b/mlir/lib/Bindings/Python/DialectTransform.cpp
@@ -494,38 +494,38 @@ struct ParamType : PyConcreteType<ParamType> {
 
 namespace {
 void onlyReadsHandle(nb::iterable &operands,
-                     PyMemoryEffectsInstanceList effects) {
+                     const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(),
-                               effects.effects);
+                               effects.get());
 };
 
 void consumesHandle(nb::iterable &operands,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
 void producesHandle(nb::iterable &results,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirValue> resultsVec;
   for (auto result : results)
     resultsVec.push_back(nb::cast<PyOpResult>(result).get());
   mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
-void modifiesPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformModifiesPayload(effects.effects);
+void modifiesPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformModifiesPayload(effects.get());
 }
 
-void onlyReadsPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformOnlyReadsPayload(effects.effects);
+void onlyReadsPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformOnlyReadsPayload(effects.get());
 }
 } // namespace
 
diff --git a/mlir/lib/Bindings/Python/IRInterfaces.cpp b/mlir/lib/Bindings/Python/IRInterfaces.cpp
index c4e87246bd68c..4495cb6e8397d 100644
--- a/mlir/lib/Bindings/Python/IRInterfaces.cpp
+++ b/mlir/lib/Bindings/Python/IRInterfaces.cpp
@@ -34,6 +34,70 @@ its return shaped type components. Raises ValueError on failure.)";
 
 namespace {
 
+MlirAttribute unwrapOptionalAttribute(const nb::object &attribute) {
+  if (attribute.is_none())
+    return mlirAttributeGetNull();
+
+  PyAttribute *pyAttribute = nullptr;
+  if (!nb::try_cast<PyAttribute *>(attribute, pyAttribute) || !pyAttribute)
+    throw nb::type_error("parameters must be an Attribute or None");
+  return pyAttribute->get();
+}
+
+void appendMemoryEffectInstance(PyMemoryEffectsInstanceList &effects,
+                                const PyMemoryEffect &effect,
+                                const nb::object &target,
+                                const nb::object &parameters, int stage,
+                                bool effectOnFullRegion,
+                                const PySideEffectResource &resource) {
+  MlirMemoryEffectInstancesList list = effects.get();
+  MlirAttribute unwrappedParameters = unwrapOptionalAttribute(parameters);
+
+  MlirMemoryEffectInstance rawInstance{nullptr};
+  if (target.is_none()) {
+    rawInstance =
+        mlirMemoryEffectInstanceCreate(effect.get(), unwrappedParameters, stage,
+                                       effectOnFullRegion, resource.get());
+  } else {
+    PyOpOperand *opOperand = nullptr;
+    PyValue *value = nullptr;
+    PyAttribute *attribute = nullptr;
+    if (nb::try_cast<PyOpOperand *>(target, opOperand) && opOperand) {
+      rawInstance = mlirMemoryEffectInstanceCreateForOpOperand(
+          effect.get(), *opOperand, unwrappedParameters, stage,
+          effectOnFullRegion, resource.get());
+    } else if (nb::try_cast<PyValue *>(target, value) && value) {
+      MlirValue mlirValue = value->get();
+      if (mlirValueIsAOpResult(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForOpResult(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else if (mlirValueIsABlockArgument(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForBlockArgument(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else {
+        throw nb::type_error(
+            "target Value must be an OpResult or BlockArgument");
+      }
+    } else if (nb::try_cast<PyAttribute *>(target, attribute) && attribute) {
+      MlirAttribute symbol = attribute->get();
+      if (!mlirAttributeIsASymbolRef(symbol))
+        throw nb::type_error("target Attribute must be a SymbolRefAttr");
+      rawInstance = mlirMemoryEffectInstanceCreateForSymbol(
+          effect.get(), symbol, unwrappedParameters, stage, effectOnFullRegion,
+          resource.get());
+    } else {
+      throw nb::type_error(
+          "target must be an OpOperand, OpResult, BlockArgument, "
+          "SymbolRefAttr, or None");
+    }
+  }
+
+  PyMemoryEffectInstance instance(rawInstance);
+  mlirMemoryEffectInstancesListAppend(list, instance.get());
+}
+
 /// Takes in an optional ist of operands and converts them into a std::vector
 /// of MlirVlaues. Returns an empty std::vector if the list is empty.
 std::vector<MlirValue> wrapOperands(std::optional<nb::sequence> operandList) {
@@ -474,9 +538,46 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("Speculatable", MlirSpeculatabilitySpeculatable)
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
-  auto memoryEffectsInstanceListClass =
-      nb::class_<PyMemoryEffectsInstanceList>(m, "MemoryEffectInstancesList");
-  (void)memoryEffectsInstanceListClass;
+  nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
+      .def_prop_ro_static("allocate",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(
+                                mlirMemoryEffectsAllocateGet());
+                          })
+      .def_prop_ro_static("free",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
+                          })
+      .def_prop_ro_static("read",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
+                          })
+      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
+      });
+
+  nb::class_<PySideEffectResource>(m, "SideEffectResource",
+                                   "A side effect resource.")
+      .def_prop_ro_static("default", [](nb::object & /*class*/) {
+        return PySideEffectResource(mlirSideEffectsDefaultResourceGet());
+      });
+
+  nb::class_<PyMemoryEffectsInstanceList>(
+      m, "MemoryEffectInstancesList",
+      "A memory effect list that is valid only during get_effects.")
+      .def("append", &appendMemoryEffectInstance, nb::arg("effect"),
+           nb::arg("target").none() = nb::none(), nb::kw_only(),
+           nb::arg("parameters").none() = nb::none(), nb::arg("stage") = 0,
+           nb::arg("effect_on_full_region") = false,
+           nb::arg("resource") =
+               PySideEffectResource(mlirSideEffectsDefaultResourceGet()),
+           nb::sig("def append(self, effect: MemoryEffect, target: OpOperand | "
+                   "OpResult | BlockArgument | SymbolRefAttr | None = None, *, "
+                   "parameters: Attribute | None = None, stage: int = 0, "
+                   "effect_on_full_region: bool = False, resource: "
+                   "SideEffectResource = ...) -> None"),
+           "Append a memory effect instance. The target may be an OpOperand, "
+           "OpResult, BlockArgument, SymbolRefAttr, or None.");
 
   PyConditionallySpeculatableOpInterface::bind(m);
   PyInferShapedTypeOpInterface::bind(m);
diff --git a/mlir/lib/CAPI/Interfaces/Interfaces.cpp b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
index 35a2bd562a8a1..1bf8dffa9c431 100644
--- a/mlir/lib/CAPI/Interfaces/Interfaces.cpp
+++ b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
@@ -280,6 +280,78 @@ MlirSpeculatability mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectOpInterface
 //===---------------------------------------------------------------------===//
 
+MlirMemoryEffect mlirMemoryEffectsAllocateGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Allocate::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsFreeGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Free::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsReadGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Read::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsWriteGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Write::get()));
+}
+
+MlirSideEffectResource mlirSideEffectsDefaultResourceGet() {
+  return wrap(static_cast<SideEffects::Resource *>(
+      SideEffects::DefaultResource::get()));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(parameters), stage, effectOnFullRegion,
+      unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpOperand(
+    MlirMemoryEffect effect, MlirOpOperand opOperand, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(opOperand), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpResult(
+    MlirMemoryEffect effect, MlirValue result, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<OpResult>(unwrap(result)), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<BlockArgument>(unwrap(blockArgument)),
+      unwrap(parameters), stage, effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForSymbol(
+    MlirMemoryEffect effect, MlirAttribute symbol, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<SymbolRefAttr>(unwrap(symbol)), unwrap(parameters),
+      stage, effectOnFullRegion, unwrap(resource)));
+}
+
+void mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance) {
+  delete unwrap(instance);
+}
+
+void mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                         MlirMemoryEffectInstance instance) {
+  unwrap(list)->push_back(*unwrap(instance));
+}
+
 MlirTypeID mlirMemoryEffectsOpInterfaceTypeID() {
   return wrap(MemoryEffectOpInterface::getInterfaceID());
 }
diff --git a/mlir/test/CAPI/ir.c b/mlir/test/CAPI/ir.c
index 57ae8b9a2819b..3c72209ad95c3 100644
--- a/mlir/test/CAPI/ir.c
+++ b/mlir/test/CAPI/ir.c
@@ -2589,6 +2589,8 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: arith.constant speculatability: 1
 
   MlirOperationState storeState = mlirOperationStateGet(storeName, loc);
+  MlirValue constantResult = mlirOperationGetResult(constantOp, 0);
+  mlirOperationStateAddOperands(&storeState, 1, &constantResult);
   MlirOperation storeOp = mlirOperationCreate(&storeState);
   if (mlirOperationImplementsInterface(storeOp, condSpecTypeID)) {
     fprintf(stderr, "ERROR: Expected memref.store instance to not implement "
@@ -2619,6 +2621,47 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: memref.store speculatability: 2
   // CHECK: callback count: 1
 
+  MlirMemoryEffect allocate = mlirMemoryEffectsAllocateGet();
+  MlirMemoryEffect free = mlirMemoryEffectsFreeGet();
+  MlirMemoryEffect read = mlirMemoryEffectsReadGet();
+  MlirMemoryEffect write = mlirMemoryEffectsWriteGet();
+  MlirSideEffectResource defaultResource = mlirSideEffectsDefaultResourceGet();
+  if (!allocate.ptr || !free.ptr || !read.ptr || !write.ptr ||
+      !defaultResource.ptr) {
+    fprintf(stderr, "ERROR: Expected memory effect components\n");
+    return 6;
+  }
+
+  MlirAttribute nullParameters = {NULL};
+  MlirOpOperand opOperand = mlirOperationGetOpOperand(storeOp, 0);
+  MlirBlock block = mlirBlockCreate(1, &i32, &loc);
+  MlirValue blockArgument = mlirBlockGetArgument(block, 0);
+  MlirAttribute symbol = mlirFlatSymbolRefAttrGet(
+      ctx, mlirStringRefCreateFromCString("effect_target"));
+
+  MlirMemoryEffectInstance instances[] = {
+      mlirMemoryEffectInstanceCreate(allocate, nullParameters, 0, false,
+                                     defaultResource),
+      mlirMemoryEffectInstanceCreateForOpOperand(read, opOperand, zero, 1,
+                                                 false, defaultResource),
+      mlirMemoryEffectInstanceCreateForOpResult(
+          write, constantResult, nullParameters, 2, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForBlockArgument(free, blockArgument, zero,
+                                                     3, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForSymbol(read, symbol, zero, 4, true,
+                                              defaultResource),
+  };
+  for (intptr_t i = 0; i < 5; ++i) {
+    if (!instances[i].ptr) {
+      fprintf(stderr, "ERROR: Expected memory effect instance\n");
+      return 7;
+    }
+    mlirMemoryEffectInstanceDestroy(instances[i]);
+  }
+  mlirBlockDestroy(block);
+  fprintf(stderr, "memory effect instances constructed\n");
+  // CHECK: memory effect instances constructed
+
   mlirOperationDestroy(storeOp);
   mlirOperationDestroy(constantOp);
   return 0;
diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
new file mode 100644
index 0000000000000..5689130c584b5
--- /dev/null
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -0,0 +1,291 @@
+# RUN: env PYTHONUNBUFFERED=1 %PYTHON %s 2>&1 | FileCheck %s
+
+from typing import Any
+
+from mlir import ir
+from mlir.dialects import ext, func
+from mlir.passmanager import PassManager
+
+
+class MemoryEffectsTest(ext.Dialect, name="memory_effects_test"):
+    pass
+
+
+class NoEffectModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        pass
+
+
+class ReadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(
+            ir.MemoryEffect.read,
+            op.op_operands[0],
+            parameters=ir.StringAttr.get("read parameter"),
+            stage=1,
+            effect_on_full_region=True,
+            resource=ir.SideEffectResource.default,
+        )
+
+
+class ReadDeadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read)
+
+
+class WriteModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.write)
+
+
+class FreeModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.free)
+
+
+class AllocateModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate)
+
+
+class AllocateResultModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate, op.results[0])
+
+
+class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+
+
+class SymbolTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        try:
+            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+        except TypeError as error:
+            print("invalid symbol target:", error)
+        try:
+            effects.append(ir.MemoryEffect.read, parameters=42)
+        except TypeError as error:
+            print("invalid parameters:", error)
+        effects.append(
+            ir.MemoryEffect.read,
+            ir.FlatSymbolRefAttr.get("global"),
+            parameters=ir.StringAttr.get("symbol parameter"),
+            stage=2,
+            effect_on_full_region=True,
+        )
+
+
+class ReadOp(MemoryEffectsTest.Operation, name="read", traits=[ReadModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteOp(MemoryEffectsTest.Operation, name="write", traits=[WriteModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteBarrierOp(
+    MemoryEffectsTest.Operation, name="write_barrier", traits=[WriteModel]
+):
+    operand: ext.Operand[Any]
+
+
+class NoEffectOp(MemoryEffectsTest.Operation, name="no_effect", traits=[NoEffectModel]):
+    pass
+
+
+class ReadDeadOp(MemoryEffectsTest.Operation, name="read_dead", traits=[ReadDeadModel]):
+    pass
+
+
+class WriteDeadOp(MemoryEffectsTest.Operation, name="write_dead", traits=[WriteModel]):
+    pass
+
+
+class FreeDeadOp(MemoryEffectsTest.Operation, name="free_dead", traits=[FreeModel]):
+    pass
+
+
+class AllocateDeadOp(
+    MemoryEffectsTest.Operation, name="allocate_dead", traits=[AllocateModel]
+):
+    pass
+
+
+class AllocateResultOp(
+    MemoryEffectsTest.Operation,
+    name="allocate_result",
+    traits=[AllocateResultModel],
+):
+    result: ext.Result[Any]
+
+
+class BlockArgumentTargetOp(
+    MemoryEffectsTest.Operation,
+    name="block_argument_target",
+    traits=[ir.NoTerminatorTrait, BlockArgumentTargetModel],
+):
+    body: ext.Region
+
+
+class SymbolTargetOp(
+    MemoryEffectsTest.Operation, name="symbol_target", traits=[SymbolTargetModel]
+):
+    pass
+
+
+def run_pass(source, pipeline):
+    module = ir.Module.parse(source)
+    PassManager.parse(pipeline).run(module.operation)
+    return str(module)
+
+
+with ir.Context(), ir.Location.unknown():
+    MemoryEffectsTest.load()
+
+    # The static properties return wrappers around borrowed, statically-owned
+    # C++ singletons.
+    # CHECK: memory effect properties: True True True True
+    print(
+        "memory effect properties:",
+        isinstance(ir.MemoryEffect.allocate, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.free, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.read, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.write, ir.MemoryEffect),
+    )
+    # CHECK: default resource property: True
+    print(
+        "default resource property:",
+        isinstance(ir.SideEffectResource.default, ir.SideEffectResource),
+    )
+
+    read_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A single Read effect remains CSE-eligible.
+    # CHECK: CSE read count: 1
+    print("CSE read count:", read_cse.count('"memory_effects_test.read"'))
+
+    write_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # Writes cannot be CSE'd.
+    # CHECK: CSE write count: 2
+    print("CSE write count:", write_cse.count('"memory_effects_test.write"'))
+
+    read_across_write = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            "memory_effects_test.write_barrier"(%arg0) : (i32) -> ()
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A potentially-aliasing Write on the default resource blocks Read CSE.
+    # CHECK: CSE read across write count: 2
+    print(
+        "CSE read across write count:",
+        read_across_write.count('"memory_effects_test.read"'),
+    )
+
+    dead_code = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.no_effect"() : () -> ()
+            "memory_effects_test.read_dead"() : () -> ()
+            "memory_effects_test.write_dead"() : () -> ()
+            "memory_effects_test.free_dead"() : () -> ()
+            "memory_effects_test.allocate_dead"() : () -> ()
+            %0 = "memory_effects_test.allocate_result"() : () -> i32
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # Empty and Read-only effect lists are dead. Write, Free and untargeted
+    # Allocate effects are observable. An Allocate targeting its own unused
+    # result is dead.
+    # CHECK: DCE no effect count: 0
+    # CHECK: DCE read count: 0
+    # CHECK: DCE write count: 1
+    # CHECK: DCE free count: 1
+    # CHECK: DCE untargeted allocate count: 1
+    # CHECK: DCE result allocate count: 0
+    print("DCE no effect count:", dead_code.count('"memory_effects_test.no_effect"'))
+    print("DCE read count:", dead_code.count('"memory_effects_test.read_dead"'))
+    print("DCE write count:", dead_code.count('"memory_effects_test.write_dead"'))
+    print("DCE free count:", dead_code.count('"memory_effects_test.free_dead"'))
+    print(
+        "DCE untargeted allocate count:",
+        dead_code.count('"memory_effects_test.allocate_dead"'),
+    )
+    print(
+        "DCE result allocate count:",
+        dead_code.count('"memory_effects_test.allocate_result"'),
+    )
+
+    target_variants = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.block_argument_target"() ({
+            ^bb0(%arg0: i32):
+            }) : () -> ()
+            "memory_effects_test.symbol_target"() : () -> ()
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # These Read effects exercise BlockArgument and SymbolRefAttr targets and
+    # remain removable by trivial-dce.
+    # CHECK: invalid symbol target: target Attribute must be a SymbolRefAttr
+    # CHECK: invalid parameters: parameters must be an Attribute or None
+    # CHECK: DCE block argument target count: 0
+    # CHECK: DCE symbol target count: 0
+    print(
+        "DCE block argument target count:",
+        target_variants.count('"memory_effects_test.block_argument_target"'),
+    )
+    print(
+        "DCE symbol target count:",
+        target_variants.count('"memory_effects_test.symbol_target"'),
+    )


From llvmlistbot at llvm.org  Sun Jul 19 03:45:51 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sun, 19 Jul 2026 03:45:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5caadf.d1496011.199a59.4bdc@mx.google.com>


https://github.com/matthias-springer approved this pull request.


https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:01:47 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 04:01:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5cae9b.a1accbb4.5eca2.be3e@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:02:27 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 04:02:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5caec3.13be6df8.22eac4.428d@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:02:46 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 04:02:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5caed6.d2cb4adc.ccd80.66c3@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:05:10 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 04:05:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5caf66.8e75e0cb.f69c7.c4e1@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:06:38 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 04:06:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5cafbe.d2cb4adc.ccd80.cf68@mx.google.com>


secona wrote:

I've updated the description to reflect the latest change. I'll need help with the merge :)

https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 04:56:28 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 19 Jul 2026 04:56:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5cbb6c.160324b0.1f653e.97f4@mx.google.com>


takatodo wrote:

I have read the LLVM AI Tool Use Policy and the other linked policies.
AI assistance is disclosed in the PR description under the Assisted-by note.

https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sun Jul 19 04:56:35 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 19 Jul 2026 04:56:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5cbb73.3cce8983.3cf542.1e09@mx.google.com>


https://github.com/takatodo ready_for_review https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sun Jul 19 04:57:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 04:57:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5cbb9a.d681cab9.169270.f5f3@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Takayuki Todokoro (takatodo)

<details>
<summary>Changes</summary>

## Summary

- Preserve discardable attributes when
  `AffineForOp::replaceWithAdditionalYields` rebuilds an `affine.for`.
- Extend the existing unroll-and-jam regression to verify that an inner-loop
  attribute survives the `LoopLikeOpInterface` yield-extension path.

## Motivation

`replaceWithAdditionalYields` creates a replacement loop with additional
iteration arguments, but did not copy the original loop's discardable
attribute dictionary. Consequently, scheduling or transformation annotations
attached to the original loop were silently lost when a transform added loop
yields.

The replacement loop now receives the original discardable attributes. This
does not add an API, pass, or option; it only preserves existing IR metadata
across the replacement.

## Testing

- Built `mlir-opt` from an assertion-enabled Release configuration based on
  `origin/main`.
- Ran all three `RUN` variants in
  `mlir/test/Dialect/Affine/unroll-jam.mlir`.

Assisted-by: OpenAI Codex gpt-5.6-sol max

---
Full diff: https://github.com/llvm/llvm-project/pull/210571.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+1) 
- (modified) mlir/test/Dialect/Affine/unroll-jam.mlir (+2-2) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..d4c4dd6b48424 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2893,6 +2893,7 @@ FailureOr<LoopLikeOpInterface> AffineForOp::replaceWithAdditionalYields(
   AffineForOp newLoop = AffineForOp::create(
       rewriter, getLoc(), getLowerBoundOperands(), getLowerBoundMap(),
       getUpperBoundOperands(), getUpperBoundMap(), getStepAsInt(), inits);
+  newLoop->setDiscardableAttrs(getOperation()->getDiscardableAttrDictionary());
 
   // Generate the new yield values and append them to the scf.yield operation.
   auto yieldOp = cast<AffineYieldOp>(getBody()->getTerminator());
diff --git a/mlir/test/Dialect/Affine/unroll-jam.mlir b/mlir/test/Dialect/Affine/unroll-jam.mlir
index 8ed7fccf7d251..bba24844c3484 100644
--- a/mlir/test/Dialect/Affine/unroll-jam.mlir
+++ b/mlir/test/Dialect/Affine/unroll-jam.mlir
@@ -185,7 +185,7 @@ func.func @unroll_jam_one_iter_arg() {
     %red = affine.for %j = 0 to 17 iter_args(%acc = %cst) -> (i32) {
       %y = "bar"(%i, %j, %acc) : (index, index, i32) -> i32
       affine.yield %y : i32
-    }
+    } {test.keep = "inner"}
     %w = "foo"(%i, %x, %red) : (index, i32, i32) -> i32
   }
   return
@@ -201,7 +201,7 @@ func.func @unroll_jam_one_iter_arg() {
 // CHECK-NEXT:     [[INC1:%[0-9]+]] = affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:     [[RES5:%[0-9]+]] = "bar"([[INC1]], [[IV1]], [[ACC2]])
 // CHECK-NEXT:     affine.yield [[RES4]], [[RES5]]
-// CHECK-NEXT:   }
+// CHECK-NEXT:   } {test.keep = "inner"}
 // CHECK:        "foo"([[IV0]], [[RES1]], [[RES3]]#0)
 // CHECK-NEXT:   affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:   "foo"({{.*}}, [[RES2]], [[RES3]]#1)

``````````

</details>


https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sun Jul 19 04:57:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 04:57:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5cbb9c.d76d81af.23b347.1bdb@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-affine

Author: Takayuki Todokoro (takatodo)

<details>
<summary>Changes</summary>

## Summary

- Preserve discardable attributes when
  `AffineForOp::replaceWithAdditionalYields` rebuilds an `affine.for`.
- Extend the existing unroll-and-jam regression to verify that an inner-loop
  attribute survives the `LoopLikeOpInterface` yield-extension path.

## Motivation

`replaceWithAdditionalYields` creates a replacement loop with additional
iteration arguments, but did not copy the original loop's discardable
attribute dictionary. Consequently, scheduling or transformation annotations
attached to the original loop were silently lost when a transform added loop
yields.

The replacement loop now receives the original discardable attributes. This
does not add an API, pass, or option; it only preserves existing IR metadata
across the replacement.

## Testing

- Built `mlir-opt` from an assertion-enabled Release configuration based on
  `origin/main`.
- Ran all three `RUN` variants in
  `mlir/test/Dialect/Affine/unroll-jam.mlir`.

Assisted-by: OpenAI Codex gpt-5.6-sol max

---
Full diff: https://github.com/llvm/llvm-project/pull/210571.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+1) 
- (modified) mlir/test/Dialect/Affine/unroll-jam.mlir (+2-2) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..d4c4dd6b48424 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2893,6 +2893,7 @@ FailureOr<LoopLikeOpInterface> AffineForOp::replaceWithAdditionalYields(
   AffineForOp newLoop = AffineForOp::create(
       rewriter, getLoc(), getLowerBoundOperands(), getLowerBoundMap(),
       getUpperBoundOperands(), getUpperBoundMap(), getStepAsInt(), inits);
+  newLoop->setDiscardableAttrs(getOperation()->getDiscardableAttrDictionary());
 
   // Generate the new yield values and append them to the scf.yield operation.
   auto yieldOp = cast<AffineYieldOp>(getBody()->getTerminator());
diff --git a/mlir/test/Dialect/Affine/unroll-jam.mlir b/mlir/test/Dialect/Affine/unroll-jam.mlir
index 8ed7fccf7d251..bba24844c3484 100644
--- a/mlir/test/Dialect/Affine/unroll-jam.mlir
+++ b/mlir/test/Dialect/Affine/unroll-jam.mlir
@@ -185,7 +185,7 @@ func.func @unroll_jam_one_iter_arg() {
     %red = affine.for %j = 0 to 17 iter_args(%acc = %cst) -> (i32) {
       %y = "bar"(%i, %j, %acc) : (index, index, i32) -> i32
       affine.yield %y : i32
-    }
+    } {test.keep = "inner"}
     %w = "foo"(%i, %x, %red) : (index, i32, i32) -> i32
   }
   return
@@ -201,7 +201,7 @@ func.func @unroll_jam_one_iter_arg() {
 // CHECK-NEXT:     [[INC1:%[0-9]+]] = affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:     [[RES5:%[0-9]+]] = "bar"([[INC1]], [[IV1]], [[ACC2]])
 // CHECK-NEXT:     affine.yield [[RES4]], [[RES5]]
-// CHECK-NEXT:   }
+// CHECK-NEXT:   } {test.keep = "inner"}
 // CHECK:        "foo"([[IV0]], [[RES1]], [[RES3]]#0)
 // CHECK-NEXT:   affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:   "foo"({{.*}}, [[RES2]], [[RES3]]#1)

``````````

</details>


https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sun Jul 19 05:00:36 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 19 Jul 2026 05:00:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5cbc64.449bfb44.7b862.3108@mx.google.com>


takatodo wrote:

@ftynse ,could you please review this small Affine/LoopLike fix?

`AffineForOp::replaceWithAdditionalYields` rebuilds the loop but currently drops discardable attributes. This patch preserves the discardable attribute dictionary and adds a regression through the existing affine unroll-and-jam path.

The change is 2 files, +3/-2. Local assertion-enabled Affine tests and the currently reported CI checks pass.

The main review question is whether preserving the discardable attribute dictionary is the intended Affine contract. 
The SCF implementation already preserves its applicable attributes in the corresponding operation.

  CC @matthias-springer , since you introduced
  `LoopLikeOpInterface::replaceWithAdditionalYields`.

https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Sun Jul 19 05:08:37 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Sun, 19 Jul 2026 05:08:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5cbe45.0c05e68e.3c2f5d.1cae@mx.google.com>


https://github.com/hamzaqureshi5 updated https://github.com/llvm/llvm-project/pull/210533

>From 40a7e8c8b2158d4b695cf33c2581b5de67fab6f5 Mon Sep 17 00:00:00 2001
From: Hamza Qureshi <hamza7771.861 at gmail.com>
Date: Sat, 18 Jul 2026 23:17:43 +0500
Subject: [PATCH] [MLIR][Affine] Fix fusion of producers that update a memref
 in place

Affine producer-consumer fusion places the fused nest at an insertion
point between the source and the destination nest. When the source nest
is retained after fusion (e.g., non-maximal fusion with an escaping
memref) and updates a memref in place (loads from and stores to it),
this placement is unsound: the fused nest recomputes the source's
values reading from that memref, but the retained source nest has
already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest,
where the in-place updated memrefs still hold their original values.
The recomputed stores are redirected to a private memref, so the
retained source nest also still reads the original values. An
anti-dependence edge from the destination to the source node is added
to the dependence graph to preserve this placement in subsequent fusion
decisions. When no such placement can be shown to preserve all
dependences, or the in-place updated memref can't be privatized, bail
out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g.,
initialize-then-reuse patterns) don't observe the memref's incoming
state and don't require the new placement.

Update @same_memref_load_store and @same_memref_load_multiple_stores,
whose CHECK lines encoded the miscompiled output; their fused nests are
unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490
---
 .../Dialect/Affine/Transforms/LoopFusion.cpp  | 140 ++++++++++++++++++
 mlir/test/Dialect/Affine/loop-fusion-4.mlir   |  87 ++++++++++-
 2 files changed, 221 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 


From llvmlistbot at llvm.org  Sun Jul 19 05:47:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 05:47:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] cd99637 - [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (#210105)
Message-ID: <6a5cc747.f1c1923d.2d2b59.7937@mx.google.com>


Author: Vito Secona
Date: 2026-07-19T14:46:58+02:00
New Revision: cd99637821097c239b665793243d8e6a332fc719

URL: https://github.com/llvm/llvm-project/commit/cd99637821097c239b665793243d8e6a332fc719
DIFF: https://github.com/llvm/llvm-project/commit/cd99637821097c239b665793243d8e6a332fc719.diff

LOG: [mlir][bufferization] Fix stale BufferOriginAnalysis in BufferDeallocationSimplification (#210105)

Fixes #205228

This change fixes a use-after-free bug in the
BufferDeallocationSimplification pipeline caused by the greedy pattern
rewriter deleting operation tracked by the BufferOriginAnalysis.
BufferViewFlowAnalysis (internally used by BufferOriginAnalysis) creates
the dependencies map once during initialization. In this specific case,
the folder removed the `memref.cast` ops making later uses a
use-after-free bug.

Below is the snippet of running the pass using
`-debug-only=greedy-rewriter`. It erases a `memref.cast` and crashes in
RemoveDeallocMemrefsContainedInRetained.

```
...
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] Processing operation : 'memref.cast'(0x654d43d18670) {
[greedy-rewriter:1]   %4 = "memref.cast"(%3) : (memref<?x?xf32, 1>) -> memref<4x4xf32, 1>
[greedy-rewriter:1] 
[greedy-rewriter:1] } -> success : operation was folded
[greedy-rewriter:1] //===-------------------------------------------===//
[greedy-rewriter:1] ** Replace : 'memref.cast'(0x654d43d18670)
[greedy-rewriter:1] ** Modified: 'scf.yield'(0x654d43c97bb0)
[greedy-rewriter:1] ** Erase   : 'memref.cast'(0x654d43d18670)
...
[greedy-rewriter:1] Processing operation : 'bufferization.dealloc'(0x654d43cbd7d0) {
[greedy-rewriter:1]   %5 = "bufferization.dealloc"(%4#0, %1, %3) <{operandSegmentSizes = array<i32: 1, 1, 1>}> : (memref<f32, 1>, i1, memref<4x4xf32, 1>) -> i1
[greedy-rewriter:1] 
[greedy-rewriter:1] 
[greedy-rewriter:1]   * Pattern (anonymous namespace)::RemoveDeallocMemrefsContainedInRetained : 'bufferization.dealloc -> ()' {
```

The fix is to disable folding and split the canonicalization patterns to
run separately. This way, the patterns that rely on BufferOriginAnalysis
still work because folding was disabled and BufferOriginAnalysis never
invalidated. Then, the canonicalization patterns still benefit from
folding.

Added: 
    

Modified: 
    mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
    mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
index a465c957d063e..06e90333fe469 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/BufferDeallocationSimplification.cpp
@@ -469,12 +469,24 @@ struct BufferDeallocationSimplificationPass
                  RetainedMemrefAliasingAlwaysDeallocatedMemref>(&getContext(),
                                                                 analysis);
 
-    populateDeallocOpCanonicalizationPatterns(patterns, &getContext());
     // We don't want that the block structure changes invalidating the
     // `BufferOriginAnalysis` so we apply the rewrites with `Normal` level of
-    // region simplification
+    // region simplification and disable folding.
     if (failed(applyPatternsGreedily(
             getOperation(), std::move(patterns),
+            GreedyRewriteConfig()
+                .setRegionSimplificationLevel(GreedySimplifyRegionLevel::Normal)
+                .enableFolding(false))))
+      signalPassFailure();
+
+    // We run canonicalization separately so it can benefit from
+    // folding, which was disabled previously to avoid invalidating
+    // `BufferOriginAnalysis`.
+    RewritePatternSet canonicalizationPatterns(&getContext());
+    populateDeallocOpCanonicalizationPatterns(canonicalizationPatterns,
+                                              &getContext());
+    if (failed(applyPatternsGreedily(
+            getOperation(), std::move(canonicalizationPatterns),
             GreedyRewriteConfig().setRegionSimplificationLevel(
                 GreedySimplifyRegionLevel::Normal))))
       signalPassFailure();

diff  --git a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
index b40a17cf800bf..3acc09d23b59e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/buffer-deallocation-simplification.mlir
@@ -169,3 +169,32 @@ func.func @duplicate_memref(%arg0: memref<5xf32>, %arg1: memref<6xf32>, %c: i1)
 // CHECK-LABEL: func @duplicate_memref(
 //       CHECK:   %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<5xf32>) if (%{{.*}}) retain (%{{.*}} : memref<6xf32>)
 //       CHECK:   return %[[r]]
+
+// -----
+
+module {
+  func.func @memref_cast_folding(%arg0: memref<4x4xf32>) -> memref<4x4xf32, 1> {
+    %cfalse = arith.constant false
+    %ctrue = arith.constant true
+
+    %memspacecast = memref.memory_space_cast %arg0 : memref<4x4xf32> to memref<4x4xf32, 1>
+    %cast = memref.cast %memspacecast : memref<4x4xf32, 1> to memref<?x?xf32, 1>
+    %cast_1 = memref.cast %cast : memref<?x?xf32, 1> to memref<4x4xf32, 1>
+
+    %5 = scf.if %cfalse -> (memref<4x4xf32, 1>) {
+      scf.yield %cast_1 : memref<4x4xf32, 1>
+    } else {
+      %7 = bufferization.clone %cast_1 : memref<4x4xf32, 1> to memref<4x4xf32, 1>
+      scf.yield %7 : memref<4x4xf32, 1>
+    }
+
+    %base_buffer, %offset, %sizes:2, %strides:2 = memref.extract_strided_metadata %arg0 : memref<4x4xf32> -> memref<f32>, index, index, index, index, index
+    %base_buffer_5, %offset_6, %sizes_7:2, %strides_8:2 = memref.extract_strided_metadata %memspacecast : memref<4x4xf32, 1> -> memref<f32, 1>, index, index, index, index, index
+
+    %6 = bufferization.dealloc (%base_buffer, %base_buffer_5 : memref<f32>, memref<f32, 1>) if (%cfalse, %ctrue) retain (%5 : memref<4x4xf32, 1>)
+    return %memspacecast : memref<4x4xf32, 1>
+  }
+}
+
+// CHECK-LABEL: func @memref_cast_folding
+//       CHECK: %[[r:.*]] = bufferization.dealloc (%{{.*}} : memref<f32, 1>) if (%{{.*}}) retain (%{{.*}} : memref<4x4xf32, 1>)


        

From llvmlistbot at llvm.org  Sun Jul 19 05:47:05 2026
From: llvmlistbot at llvm.org (Matthias Springer)
Date: Sun, 19 Jul 2026 05:47:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Fix stale
 BufferOriginAnalysis in BufferDeallocationSimplification (PR #210105)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210105@github.com>
Message-ID: <6a5cc749.1e1ac41d.3969c3.23df@mx.google.com>


https://github.com/matthias-springer closed https://github.com/llvm/llvm-project/pull/210105

From llvmlistbot at llvm.org  Sun Jul 19 06:06:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:06:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] #208902: Added xegpu load/store flattening
	(PR #210592)
Message-ID: <llvm.org/llvm/llvm-project/pull/210592@github.com>


https://github.com/Naaz30 created https://github.com/llvm/llvm-project/pull/210592

None

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Sun Jul 19 06:07:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:07:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] #208902: Added xegpu load/store flattening
	(PR #210592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210592@github.com>
Message-ID: <6a5ccc0a.d1496011.199a59.8fb9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Naaz30

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/210592.diff


1 Files Affected:

- (modified) mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp (+11-7) 


``````````diff
diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

``````````

</details>


https://github.com/llvm/llvm-project/pull/210592

From llvmlistbot at llvm.org  Sun Jul 19 06:08:08 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:08:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] #208902: Added xegpu load/store flattening
	(PR #210592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210592@github.com>
Message-ID: <6a5ccc38.d2cb4adc.ccd80.77f8@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494..53a95149f 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,10 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts genuine multi-dim tiles
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting
+        // the whole tile as one flat, contiguous run. This is only valid when
+        // the underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }

``````````

</details>


https://github.com/llvm/llvm-project/pull/210592

From llvmlistbot at llvm.org  Sun Jul 19 06:08:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:08:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] #208902: Added xegpu load/store flattening
	(PR #210592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210592@github.com>
Message-ID: <6a5ccc43.c56fb102.2d07b8.17d3@mx.google.com>


https://github.com/Naaz30 closed https://github.com/llvm/llvm-project/pull/210592

From llvmlistbot at llvm.org  Sun Jul 19 06:41:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:41:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
Message-ID: <llvm.org/llvm/llvm-project/pull/210595@github.com>


https://github.com/Naaz30 created https://github.com/llvm/llvm-project/pull/210595

`LoadStoreMatrixToXeVMPattern` currently rejects `xegpu.load_matrix` and `xegpu.store_matrix` operations in two situations that are not required for the plain (non-subgroup_block_io) lowering path:

1. **Vector shape restriction** : The pass asserts that the vector type has at most one non-unit dimension, assuming that xegpu-sg-to-lane-distribute has       already reduced the operation to a per-lane fragment. This causes the pass to fail (or behave incorrectly in NDEBUG builds) for valid multi-dimensional tiles such as vector<4x8xf32>.

2. **Chip-specific target check** : The pass requires the operation to reside inside a gpu.module with an xevm.target attribute identifying a supported architecture (PVC, BMG, or CRI) via xegpu::getChipStr(). However, this requirement is unnecessary for the plain lowering path, which only generates chip-independent LLVM::LoadOp and LLVM::StoreOp operations.

**Changes**
Remove the unnecessary vector shape restriction for the plain lowering path.
Remove the chip target requirement for the plain lowering path.

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH 1/2] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

>From b8c1e1efbc222e0bb37782daa19ddb234e63eb01 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:00:05 +0530
Subject: [PATCH 2/2] MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp    | 10 ++---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 41 +++++++++++++++----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494e19c..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,8 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -789,9 +787,7 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      // Only reject an explicitly unsupported chip; a missing gpu.module/target
-      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
-      // path also works for standalone/non-GPU-kernel-context IR.
+      // reject an explicitly unsupported chip
       if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
           *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }


From llvmlistbot at llvm.org  Sun Jul 19 06:41:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:41:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd427.f3151ff8.37ac76.01c3@mx.google.com>


https://github.com/Naaz30 updated https://github.com/llvm/llvm-project/pull/210595

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH 1/2] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

>From b8c1e1efbc222e0bb37782daa19ddb234e63eb01 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:00:05 +0530
Subject: [PATCH 2/2] MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp    | 10 ++---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 41 +++++++++++++++----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494e19c..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,8 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -789,9 +787,7 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      // Only reject an explicitly unsupported chip; a missing gpu.module/target
-      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
-      // path also works for standalone/non-GPU-kernel-context IR.
+      // reject an explicitly unsupported chip
       if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
           *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }


From llvmlistbot at llvm.org  Sun Jul 19 06:42:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:42:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd438.b4a331f0.d9742.648b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir

@llvm/pr-subscribers-mlir-gpu

Author: Naaz30

<details>
<summary>Changes</summary>

`LoadStoreMatrixToXeVMPattern` currently rejects `xegpu.load_matrix` and `xegpu.store_matrix` operations in two situations that are not required for the plain (non-subgroup_block_io) lowering path:

1. **Vector shape restriction** : The pass asserts that the vector type has at most one non-unit dimension, assuming that xegpu-sg-to-lane-distribute has       already reduced the operation to a per-lane fragment. This causes the pass to fail (or behave incorrectly in NDEBUG builds) for valid multi-dimensional tiles such as vector&lt;4x8xf32&gt;.

2. **Chip-specific target check** : The pass requires the operation to reside inside a gpu.module with an xevm.target attribute identifying a supported architecture (PVC, BMG, or CRI) via xegpu::getChipStr(). However, this requirement is unnecessary for the plain lowering path, which only generates chip-independent LLVM::LoadOp and LLVM::StoreOp operations.

**Changes**
Remove the unnecessary vector shape restriction for the plain lowering path.
Remove the chip target requirement for the plain lowering path.

---
Full diff: https://github.com/llvm/llvm-project/pull/210595.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp (+7-7) 
- (modified) mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir (+32-9) 


``````````diff
diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +787,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // reject an explicitly unsupported chip
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1626,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/210595

From llvmlistbot at llvm.org  Sun Jul 19 06:43:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:43:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd485.f06f4994.2f0620.8acb@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 03808d2d0..fae578eef 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,8 +720,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts multi-dim tile as one flat, contiguous run. 
-        // This is only valid when the underlying mem_desc region is contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run.
+        // This is only valid when the underlying mem_desc region is contiguous
+        // in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }

``````````

</details>


https://github.com/llvm/llvm-project/pull/210595

From llvmlistbot at llvm.org  Sun Jul 19 06:51:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:51:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd648.92d2873c.aa1ff.1f65@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 8570 tests passed
* 630 tests skipped
* 1 test failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Conversion/XeGPUToXeVM/loadstore_matrix.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt  -split-input-file -convert-xegpu-to-xevm /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir | /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/mlir-opt -split-input-file -convert-xegpu-to-xevm /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir:328:5: error: invalid SSA name
# |     % c0 = arith.constant 0 : index % 0 =
# |     ^
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir:349:16: error: invalid SSA name
# |   func.func @m(% arg0 : index) {
# |                ^
# `-----------------------------
# error: command failed with exit status: 1
# executed command: /home/gha/actions-runner/_work/llvm-project/llvm-project/build/bin/FileCheck /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
# .---command stderr------------
# | /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir:8:18: error: CHECK-LABEL: expected string not found in input
# |  // CHECK-LABEL: load_store_matrix_plain
# |                  ^
# | <stdin>:1:1: note: scanning from here
# | // -----
# | ^
# | 
# | Input file: <stdin>
# | Check file: /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: // ----- 
# | label:8'0    {         } search range (exclusive bounds)
# | label:8'1                error: no match found in search range
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/210595

From llvmlistbot at llvm.org  Sun Jul 19 06:51:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 06:51:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd649.507fc713.239434.650f@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

* 3969 tests passed
* 440 tests skipped
* 1 test failed

## Failed Tests
(click on a test name to see its output)

### MLIR
<details>
<summary>MLIR.Conversion/XeGPUToXeVM/loadstore_matrix.mlir</summary>

```
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe  -split-input-file -convert-xegpu-to-xevm C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir | c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\mlir-opt.exe' -split-input-file -convert-xegpu-to-xevm 'C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir:328:5: error: invalid SSA name
# |     % c0 = arith.constant 0 : index % 0 =
# |     ^
# | C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir:349:16: error: invalid SSA name
# |   func.func @m(% arg0 : index) {
# |                ^
# `-----------------------------
# error: command failed with exit status: 1
# executed command: 'c:\_work\llvm-project\llvm-project\build\bin\filecheck.exe' 'C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir'
# .---command stderr------------
# | C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir:8:18: error: CHECK-LABEL: expected string not found in input
# |  // CHECK-LABEL: load_store_matrix_plain
# |                  ^
# | <stdin>:1:1: note: scanning from here
# | // -----
# | ^
# | 
# | Input file: <stdin>
# | Check file: C:\_work\llvm-project\llvm-project\mlir\test\Conversion\XeGPUToXeVM\loadstore_matrix.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: // ----- 
# | label:8'0    {         } search range (exclusive bounds)
# | label:8'1                error: no match found in search range
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/210595

From llvmlistbot at llvm.org  Sun Jul 19 07:04:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 07:04:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cd954.b1416da1.3242c8.f0f7@mx.google.com>


https://github.com/Naaz30 updated https://github.com/llvm/llvm-project/pull/210595

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH 1/3] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

>From b8c1e1efbc222e0bb37782daa19ddb234e63eb01 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:00:05 +0530
Subject: [PATCH 2/3] MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp    | 10 ++---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 41 +++++++++++++++----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494e19c..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,8 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -789,9 +787,7 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      // Only reject an explicitly unsupported chip; a missing gpu.module/target
-      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
-      // path also works for standalone/non-GPU-kernel-context IR.
+      // reject an explicitly unsupported chip
       if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
           *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }

>From 232df7d956d8cc04c4e8e0a28e4f4ae6a2feb3d6 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:31:36 +0530
Subject: [PATCH 3/3] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp        | 5 +++--
 mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir | 8 ++------
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 03808d2d0f8ff..fae578eefd107 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,8 +720,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts multi-dim tile as one flat, contiguous run. 
-        // This is only valid when the underlying mem_desc region is contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run.
+        // This is only valid when the underlying mem_desc region is contiguous
+        // in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 50f7263b65e05..040900d5f40f3 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -335,9 +335,6 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
         : !xegpu.mem_desc<32x32xf8E8M0FNU>,
       index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
-}
-
-// -----
 
 // CHECK-LABEL: func.func @m(
 // CHECK: memref.extract_aligned_pointer_as_index
@@ -345,8 +342,7 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 // CHECK: llvm.load
 // CHECK-SAME: vector<32xf32>
 
-module {
-  func.func @m(% arg0 : index) {
+func.func @m(%arg0: index) {
     % alloca =
         memref.alloca()
         : memref<4x8xf32, 3> %
@@ -355,4 +351,4 @@ module {
                 xegpu.load_matrix %
                 0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
   }
-}
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Sun Jul 19 07:51:51 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Sun, 19 Jul 2026 07:51:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
Message-ID: <llvm.org/llvm/llvm-project/pull/210600@github.com>


https://github.com/secona created https://github.com/llvm/llvm-project/pull/210600

The current SPIRVToLLVM conversion for SelectionOp does not handle merge blocks with yielding values. This change implements that by adding arguments to the continue block in the SelectionPattern.

Closes #204714

>From 8aa7d419ee5c0751b557f46e2881d94e902ea37e Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sun, 19 Jul 2026 20:57:19 +0700
Subject: [PATCH] [mlir][SPIRV] Add SPIRVToLLVM conversion for selection with
 yielding values

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    |  8 +++
 .../SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 56 +++++++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c5f9305a5c226..2982061c957e0 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1470,6 +1470,14 @@ class SelectionPattern : public SPIRVToLLVMConversion<spirv::SelectionOp> {
     auto position = rewriter.getInsertionPoint();
     auto *continueBlock = rewriter.splitBlock(currentBlock, position);
 
+    // Add arguments to the continue block for selections that yield values.
+    for (auto ty : op.getResultTypes()) {
+      Type dstTy = getTypeConverter()->convertType(ty);
+      if (!dstTy)
+        return rewriter.notifyMatchFailure(op, "failed to convert type");
+      continueBlock->addArgument(dstTy, loc);
+    }
+
     // Extract conditional branch information from the header block. By SPIR-V
     // dialect spec, it should contain `spirv.BranchConditional` or
     // `spirv.Switch` op. Note that `spirv.Switch op` is not supported at the
diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index ec47dd31cc637..389a21681bafb 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
 }
 
 // -----


From llvmlistbot at llvm.org  Sun Jul 19 07:52:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 07:52:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5ce4ab.7d702ed2.13d102.0bc8@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Vito Secona (secona)

<details>
<summary>Changes</summary>

The current SPIRVToLLVM conversion for SelectionOp does not handle merge blocks with yielding values. This change implements that by adding arguments to the continue block in the SelectionPattern.

Closes #<!-- -->204714

---
Full diff: https://github.com/llvm/llvm-project/pull/210600.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp (+8) 
- (modified) mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir (+56) 


``````````diff
diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c5f9305a5c226..2982061c957e0 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1470,6 +1470,14 @@ class SelectionPattern : public SPIRVToLLVMConversion<spirv::SelectionOp> {
     auto position = rewriter.getInsertionPoint();
     auto *continueBlock = rewriter.splitBlock(currentBlock, position);
 
+    // Add arguments to the continue block for selections that yield values.
+    for (auto ty : op.getResultTypes()) {
+      Type dstTy = getTypeConverter()->convertType(ty);
+      if (!dstTy)
+        return rewriter.notifyMatchFailure(op, "failed to convert type");
+      continueBlock->addArgument(dstTy, loc);
+    }
+
     // Extract conditional branch information from the header block. By SPIR-V
     // dialect spec, it should contain `spirv.BranchConditional` or
     // `spirv.Switch` op. Note that `spirv.Switch op` is not supported at the
diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index ec47dd31cc637..389a21681bafb 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
 }
 
 // -----

``````````

</details>


https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Sun Jul 19 07:56:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 07:56:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5ce5af.af6ba3ed.3c8c03.9c5e@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210586

>From ee4b1d4aa4c3c0afaf896ace40531d702551cb77 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 18:07:37 +0800
Subject: [PATCH 1/2] [MLIR][CAPI][Python] Add support for constructing memory
 effect instances

---
 mlir/include/mlir-c/Interfaces.h              |  78 +++++
 .../mlir/Bindings/Python/IRInterfaces.h       |  53 +++-
 mlir/include/mlir/CAPI/Interfaces.h           |   4 +
 mlir/lib/Bindings/Python/DialectTransform.cpp |  20 +-
 mlir/lib/Bindings/Python/IRInterfaces.cpp     | 107 ++++++-
 mlir/lib/CAPI/Interfaces/Interfaces.cpp       |  72 +++++
 mlir/test/CAPI/ir.c                           |  43 +++
 .../dialects/memory_effects_op_interface.py   | 291 ++++++++++++++++++
 8 files changed, 654 insertions(+), 14 deletions(-)
 create mode 100644 mlir/test/python/dialects/memory_effects_op_interface.py

diff --git a/mlir/include/mlir-c/Interfaces.h b/mlir/include/mlir-c/Interfaces.h
index a5251dc78471f..a416a6ab76f87 100644
--- a/mlir/include/mlir-c/Interfaces.h
+++ b/mlir/include/mlir-c/Interfaces.h
@@ -28,7 +28,10 @@ extern "C" {
   };                                                                           \
   typedef struct name name
 
+DEFINE_C_API_STRUCT(MlirMemoryEffect, void);
+DEFINE_C_API_STRUCT(MlirMemoryEffectInstance, void);
 DEFINE_C_API_STRUCT(MlirMemoryEffectInstancesList, void);
+DEFINE_C_API_STRUCT(MlirSideEffectResource, void);
 
 #undef DEFINE_C_API_STRUCT
 
@@ -144,6 +147,81 @@ mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectsOpInterface
 //===---------------------------------------------------------------------===//
 
+/// Returns the borrowed singleton instance of the allocate memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsAllocateGet(void);
+
+/// Returns the borrowed singleton instance of the free memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsFreeGet(void);
+
+/// Returns the borrowed singleton instance of the read memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsReadGet(void);
+
+/// Returns the borrowed singleton instance of the write memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsWriteGet(void);
+
+/// Returns the borrowed singleton instance of the default side effect
+/// resource.
+MLIR_CAPI_EXPORTED MlirSideEffectResource
+mlirSideEffectsDefaultResourceGet(void);
+
+/// Creates a memory effect instance without an associated IR entity.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation operand.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpOperand(MlirMemoryEffect effect,
+                                           MlirOpOperand opOperand,
+                                           MlirAttribute parameters, int stage,
+                                           bool effectOnFullRegion,
+                                           MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation result.
+/// `result` must wrap an OpResult. `parameters` may be a null attribute. The
+/// caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpResult(MlirMemoryEffect effect,
+                                          MlirValue result,
+                                          MlirAttribute parameters, int stage,
+                                          bool effectOnFullRegion,
+                                          MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a block argument.
+/// `blockArgument` must wrap a BlockArgument. `parameters` may be a null
+/// attribute. The caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a symbol. `symbol` must be
+/// a SymbolRefAttr. `parameters` may be a null attribute. The caller owns the
+/// returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForSymbol(MlirMemoryEffect effect,
+                                        MlirAttribute symbol,
+                                        MlirAttribute parameters, int stage,
+                                        bool effectOnFullRegion,
+                                        MlirSideEffectResource resource);
+
+/// Destroys a memory effect instance created by one of the functions above.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance);
+
+/// Appends a copy of `instance` to the given list. This does not take ownership
+/// of `instance`; the caller remains responsible for destroying it.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                    MlirMemoryEffectInstance instance);
+
 /// Returns the interface TypeID of the MemoryEffectsOpInterface.
 MLIR_CAPI_EXPORTED MlirTypeID mlirMemoryEffectsOpInterfaceTypeID(void);
 
diff --git a/mlir/include/mlir/Bindings/Python/IRInterfaces.h b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
index fb30e030b6c32..1ba995aa82c5e 100644
--- a/mlir/include/mlir/Bindings/Python/IRInterfaces.h
+++ b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
+class PyMemoryEffect {
+public:
+  explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
+
+  MlirMemoryEffect get() const { return effect; }
+
+private:
+  MlirMemoryEffect effect;
+};
+
+/// A borrowed side effect resource.
+class PySideEffectResource {
+public:
+  explicit PySideEffectResource(MlirSideEffectResource resource)
+      : resource(resource) {}
+
+  MlirSideEffectResource get() const { return resource; }
+
+private:
+  MlirSideEffectResource resource;
+};
+
+/// An owning memory effect instance.
+class PyMemoryEffectInstance {
+public:
+  explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
+      : instance(instance) {}
+  PyMemoryEffectInstance(PyMemoryEffectInstance &&other) noexcept
+      : instance(other.instance) {
+    other.instance.ptr = nullptr;
+  }
+  ~PyMemoryEffectInstance() {
+    if (instance.ptr)
+      mlirMemoryEffectInstanceDestroy(instance);
+  }
+
+  MlirMemoryEffectInstance get() const { return instance; }
+
+private:
+  MlirMemoryEffectInstance instance;
+};
+
+/// A callback-scoped view of a list of memory effect instances.
+class PyMemoryEffectsInstanceList {
+public:
+  explicit PyMemoryEffectsInstanceList(MlirMemoryEffectInstancesList effects)
+      : effects(effects) {}
+
+  MlirMemoryEffectInstancesList get() const { return effects; }
+
+private:
   MlirMemoryEffectInstancesList effects;
 };
 
diff --git a/mlir/include/mlir/CAPI/Interfaces.h b/mlir/include/mlir/CAPI/Interfaces.h
index 15afc9fb0f18e..55d850ccd01eb 100644
--- a/mlir/include/mlir/CAPI/Interfaces.h
+++ b/mlir/include/mlir/CAPI/Interfaces.h
@@ -22,5 +22,9 @@
 DEFINE_C_API_PTR_METHODS(
     MlirMemoryEffectInstancesList,
     llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance>)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffect, mlir::MemoryEffects::Effect)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffectInstance,
+                         mlir::MemoryEffects::EffectInstance)
+DEFINE_C_API_PTR_METHODS(MlirSideEffectResource, mlir::SideEffects::Resource)
 
 #endif // MLIR_CAPI_INTERFACES_H
diff --git a/mlir/lib/Bindings/Python/DialectTransform.cpp b/mlir/lib/Bindings/Python/DialectTransform.cpp
index bd72082cea7e8..f84189e3a22ff 100644
--- a/mlir/lib/Bindings/Python/DialectTransform.cpp
+++ b/mlir/lib/Bindings/Python/DialectTransform.cpp
@@ -494,38 +494,38 @@ struct ParamType : PyConcreteType<ParamType> {
 
 namespace {
 void onlyReadsHandle(nb::iterable &operands,
-                     PyMemoryEffectsInstanceList effects) {
+                     const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(),
-                               effects.effects);
+                               effects.get());
 };
 
 void consumesHandle(nb::iterable &operands,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
 void producesHandle(nb::iterable &results,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirValue> resultsVec;
   for (auto result : results)
     resultsVec.push_back(nb::cast<PyOpResult>(result).get());
   mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
-void modifiesPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformModifiesPayload(effects.effects);
+void modifiesPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformModifiesPayload(effects.get());
 }
 
-void onlyReadsPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformOnlyReadsPayload(effects.effects);
+void onlyReadsPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformOnlyReadsPayload(effects.get());
 }
 } // namespace
 
diff --git a/mlir/lib/Bindings/Python/IRInterfaces.cpp b/mlir/lib/Bindings/Python/IRInterfaces.cpp
index c4e87246bd68c..4495cb6e8397d 100644
--- a/mlir/lib/Bindings/Python/IRInterfaces.cpp
+++ b/mlir/lib/Bindings/Python/IRInterfaces.cpp
@@ -34,6 +34,70 @@ its return shaped type components. Raises ValueError on failure.)";
 
 namespace {
 
+MlirAttribute unwrapOptionalAttribute(const nb::object &attribute) {
+  if (attribute.is_none())
+    return mlirAttributeGetNull();
+
+  PyAttribute *pyAttribute = nullptr;
+  if (!nb::try_cast<PyAttribute *>(attribute, pyAttribute) || !pyAttribute)
+    throw nb::type_error("parameters must be an Attribute or None");
+  return pyAttribute->get();
+}
+
+void appendMemoryEffectInstance(PyMemoryEffectsInstanceList &effects,
+                                const PyMemoryEffect &effect,
+                                const nb::object &target,
+                                const nb::object &parameters, int stage,
+                                bool effectOnFullRegion,
+                                const PySideEffectResource &resource) {
+  MlirMemoryEffectInstancesList list = effects.get();
+  MlirAttribute unwrappedParameters = unwrapOptionalAttribute(parameters);
+
+  MlirMemoryEffectInstance rawInstance{nullptr};
+  if (target.is_none()) {
+    rawInstance =
+        mlirMemoryEffectInstanceCreate(effect.get(), unwrappedParameters, stage,
+                                       effectOnFullRegion, resource.get());
+  } else {
+    PyOpOperand *opOperand = nullptr;
+    PyValue *value = nullptr;
+    PyAttribute *attribute = nullptr;
+    if (nb::try_cast<PyOpOperand *>(target, opOperand) && opOperand) {
+      rawInstance = mlirMemoryEffectInstanceCreateForOpOperand(
+          effect.get(), *opOperand, unwrappedParameters, stage,
+          effectOnFullRegion, resource.get());
+    } else if (nb::try_cast<PyValue *>(target, value) && value) {
+      MlirValue mlirValue = value->get();
+      if (mlirValueIsAOpResult(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForOpResult(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else if (mlirValueIsABlockArgument(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForBlockArgument(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else {
+        throw nb::type_error(
+            "target Value must be an OpResult or BlockArgument");
+      }
+    } else if (nb::try_cast<PyAttribute *>(target, attribute) && attribute) {
+      MlirAttribute symbol = attribute->get();
+      if (!mlirAttributeIsASymbolRef(symbol))
+        throw nb::type_error("target Attribute must be a SymbolRefAttr");
+      rawInstance = mlirMemoryEffectInstanceCreateForSymbol(
+          effect.get(), symbol, unwrappedParameters, stage, effectOnFullRegion,
+          resource.get());
+    } else {
+      throw nb::type_error(
+          "target must be an OpOperand, OpResult, BlockArgument, "
+          "SymbolRefAttr, or None");
+    }
+  }
+
+  PyMemoryEffectInstance instance(rawInstance);
+  mlirMemoryEffectInstancesListAppend(list, instance.get());
+}
+
 /// Takes in an optional ist of operands and converts them into a std::vector
 /// of MlirVlaues. Returns an empty std::vector if the list is empty.
 std::vector<MlirValue> wrapOperands(std::optional<nb::sequence> operandList) {
@@ -474,9 +538,46 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("Speculatable", MlirSpeculatabilitySpeculatable)
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
-  auto memoryEffectsInstanceListClass =
-      nb::class_<PyMemoryEffectsInstanceList>(m, "MemoryEffectInstancesList");
-  (void)memoryEffectsInstanceListClass;
+  nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
+      .def_prop_ro_static("allocate",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(
+                                mlirMemoryEffectsAllocateGet());
+                          })
+      .def_prop_ro_static("free",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
+                          })
+      .def_prop_ro_static("read",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
+                          })
+      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
+      });
+
+  nb::class_<PySideEffectResource>(m, "SideEffectResource",
+                                   "A side effect resource.")
+      .def_prop_ro_static("default", [](nb::object & /*class*/) {
+        return PySideEffectResource(mlirSideEffectsDefaultResourceGet());
+      });
+
+  nb::class_<PyMemoryEffectsInstanceList>(
+      m, "MemoryEffectInstancesList",
+      "A memory effect list that is valid only during get_effects.")
+      .def("append", &appendMemoryEffectInstance, nb::arg("effect"),
+           nb::arg("target").none() = nb::none(), nb::kw_only(),
+           nb::arg("parameters").none() = nb::none(), nb::arg("stage") = 0,
+           nb::arg("effect_on_full_region") = false,
+           nb::arg("resource") =
+               PySideEffectResource(mlirSideEffectsDefaultResourceGet()),
+           nb::sig("def append(self, effect: MemoryEffect, target: OpOperand | "
+                   "OpResult | BlockArgument | SymbolRefAttr | None = None, *, "
+                   "parameters: Attribute | None = None, stage: int = 0, "
+                   "effect_on_full_region: bool = False, resource: "
+                   "SideEffectResource = ...) -> None"),
+           "Append a memory effect instance. The target may be an OpOperand, "
+           "OpResult, BlockArgument, SymbolRefAttr, or None.");
 
   PyConditionallySpeculatableOpInterface::bind(m);
   PyInferShapedTypeOpInterface::bind(m);
diff --git a/mlir/lib/CAPI/Interfaces/Interfaces.cpp b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
index 35a2bd562a8a1..1bf8dffa9c431 100644
--- a/mlir/lib/CAPI/Interfaces/Interfaces.cpp
+++ b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
@@ -280,6 +280,78 @@ MlirSpeculatability mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectOpInterface
 //===---------------------------------------------------------------------===//
 
+MlirMemoryEffect mlirMemoryEffectsAllocateGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Allocate::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsFreeGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Free::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsReadGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Read::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsWriteGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Write::get()));
+}
+
+MlirSideEffectResource mlirSideEffectsDefaultResourceGet() {
+  return wrap(static_cast<SideEffects::Resource *>(
+      SideEffects::DefaultResource::get()));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(parameters), stage, effectOnFullRegion,
+      unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpOperand(
+    MlirMemoryEffect effect, MlirOpOperand opOperand, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(opOperand), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpResult(
+    MlirMemoryEffect effect, MlirValue result, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<OpResult>(unwrap(result)), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<BlockArgument>(unwrap(blockArgument)),
+      unwrap(parameters), stage, effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForSymbol(
+    MlirMemoryEffect effect, MlirAttribute symbol, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<SymbolRefAttr>(unwrap(symbol)), unwrap(parameters),
+      stage, effectOnFullRegion, unwrap(resource)));
+}
+
+void mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance) {
+  delete unwrap(instance);
+}
+
+void mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                         MlirMemoryEffectInstance instance) {
+  unwrap(list)->push_back(*unwrap(instance));
+}
+
 MlirTypeID mlirMemoryEffectsOpInterfaceTypeID() {
   return wrap(MemoryEffectOpInterface::getInterfaceID());
 }
diff --git a/mlir/test/CAPI/ir.c b/mlir/test/CAPI/ir.c
index 57ae8b9a2819b..3c72209ad95c3 100644
--- a/mlir/test/CAPI/ir.c
+++ b/mlir/test/CAPI/ir.c
@@ -2589,6 +2589,8 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: arith.constant speculatability: 1
 
   MlirOperationState storeState = mlirOperationStateGet(storeName, loc);
+  MlirValue constantResult = mlirOperationGetResult(constantOp, 0);
+  mlirOperationStateAddOperands(&storeState, 1, &constantResult);
   MlirOperation storeOp = mlirOperationCreate(&storeState);
   if (mlirOperationImplementsInterface(storeOp, condSpecTypeID)) {
     fprintf(stderr, "ERROR: Expected memref.store instance to not implement "
@@ -2619,6 +2621,47 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: memref.store speculatability: 2
   // CHECK: callback count: 1
 
+  MlirMemoryEffect allocate = mlirMemoryEffectsAllocateGet();
+  MlirMemoryEffect free = mlirMemoryEffectsFreeGet();
+  MlirMemoryEffect read = mlirMemoryEffectsReadGet();
+  MlirMemoryEffect write = mlirMemoryEffectsWriteGet();
+  MlirSideEffectResource defaultResource = mlirSideEffectsDefaultResourceGet();
+  if (!allocate.ptr || !free.ptr || !read.ptr || !write.ptr ||
+      !defaultResource.ptr) {
+    fprintf(stderr, "ERROR: Expected memory effect components\n");
+    return 6;
+  }
+
+  MlirAttribute nullParameters = {NULL};
+  MlirOpOperand opOperand = mlirOperationGetOpOperand(storeOp, 0);
+  MlirBlock block = mlirBlockCreate(1, &i32, &loc);
+  MlirValue blockArgument = mlirBlockGetArgument(block, 0);
+  MlirAttribute symbol = mlirFlatSymbolRefAttrGet(
+      ctx, mlirStringRefCreateFromCString("effect_target"));
+
+  MlirMemoryEffectInstance instances[] = {
+      mlirMemoryEffectInstanceCreate(allocate, nullParameters, 0, false,
+                                     defaultResource),
+      mlirMemoryEffectInstanceCreateForOpOperand(read, opOperand, zero, 1,
+                                                 false, defaultResource),
+      mlirMemoryEffectInstanceCreateForOpResult(
+          write, constantResult, nullParameters, 2, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForBlockArgument(free, blockArgument, zero,
+                                                     3, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForSymbol(read, symbol, zero, 4, true,
+                                              defaultResource),
+  };
+  for (intptr_t i = 0; i < 5; ++i) {
+    if (!instances[i].ptr) {
+      fprintf(stderr, "ERROR: Expected memory effect instance\n");
+      return 7;
+    }
+    mlirMemoryEffectInstanceDestroy(instances[i]);
+  }
+  mlirBlockDestroy(block);
+  fprintf(stderr, "memory effect instances constructed\n");
+  // CHECK: memory effect instances constructed
+
   mlirOperationDestroy(storeOp);
   mlirOperationDestroy(constantOp);
   return 0;
diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
new file mode 100644
index 0000000000000..5689130c584b5
--- /dev/null
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -0,0 +1,291 @@
+# RUN: env PYTHONUNBUFFERED=1 %PYTHON %s 2>&1 | FileCheck %s
+
+from typing import Any
+
+from mlir import ir
+from mlir.dialects import ext, func
+from mlir.passmanager import PassManager
+
+
+class MemoryEffectsTest(ext.Dialect, name="memory_effects_test"):
+    pass
+
+
+class NoEffectModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        pass
+
+
+class ReadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(
+            ir.MemoryEffect.read,
+            op.op_operands[0],
+            parameters=ir.StringAttr.get("read parameter"),
+            stage=1,
+            effect_on_full_region=True,
+            resource=ir.SideEffectResource.default,
+        )
+
+
+class ReadDeadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read)
+
+
+class WriteModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.write)
+
+
+class FreeModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.free)
+
+
+class AllocateModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate)
+
+
+class AllocateResultModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate, op.results[0])
+
+
+class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+
+
+class SymbolTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        try:
+            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+        except TypeError as error:
+            print("invalid symbol target:", error)
+        try:
+            effects.append(ir.MemoryEffect.read, parameters=42)
+        except TypeError as error:
+            print("invalid parameters:", error)
+        effects.append(
+            ir.MemoryEffect.read,
+            ir.FlatSymbolRefAttr.get("global"),
+            parameters=ir.StringAttr.get("symbol parameter"),
+            stage=2,
+            effect_on_full_region=True,
+        )
+
+
+class ReadOp(MemoryEffectsTest.Operation, name="read", traits=[ReadModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteOp(MemoryEffectsTest.Operation, name="write", traits=[WriteModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteBarrierOp(
+    MemoryEffectsTest.Operation, name="write_barrier", traits=[WriteModel]
+):
+    operand: ext.Operand[Any]
+
+
+class NoEffectOp(MemoryEffectsTest.Operation, name="no_effect", traits=[NoEffectModel]):
+    pass
+
+
+class ReadDeadOp(MemoryEffectsTest.Operation, name="read_dead", traits=[ReadDeadModel]):
+    pass
+
+
+class WriteDeadOp(MemoryEffectsTest.Operation, name="write_dead", traits=[WriteModel]):
+    pass
+
+
+class FreeDeadOp(MemoryEffectsTest.Operation, name="free_dead", traits=[FreeModel]):
+    pass
+
+
+class AllocateDeadOp(
+    MemoryEffectsTest.Operation, name="allocate_dead", traits=[AllocateModel]
+):
+    pass
+
+
+class AllocateResultOp(
+    MemoryEffectsTest.Operation,
+    name="allocate_result",
+    traits=[AllocateResultModel],
+):
+    result: ext.Result[Any]
+
+
+class BlockArgumentTargetOp(
+    MemoryEffectsTest.Operation,
+    name="block_argument_target",
+    traits=[ir.NoTerminatorTrait, BlockArgumentTargetModel],
+):
+    body: ext.Region
+
+
+class SymbolTargetOp(
+    MemoryEffectsTest.Operation, name="symbol_target", traits=[SymbolTargetModel]
+):
+    pass
+
+
+def run_pass(source, pipeline):
+    module = ir.Module.parse(source)
+    PassManager.parse(pipeline).run(module.operation)
+    return str(module)
+
+
+with ir.Context(), ir.Location.unknown():
+    MemoryEffectsTest.load()
+
+    # The static properties return wrappers around borrowed, statically-owned
+    # C++ singletons.
+    # CHECK: memory effect properties: True True True True
+    print(
+        "memory effect properties:",
+        isinstance(ir.MemoryEffect.allocate, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.free, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.read, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.write, ir.MemoryEffect),
+    )
+    # CHECK: default resource property: True
+    print(
+        "default resource property:",
+        isinstance(ir.SideEffectResource.default, ir.SideEffectResource),
+    )
+
+    read_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A single Read effect remains CSE-eligible.
+    # CHECK: CSE read count: 1
+    print("CSE read count:", read_cse.count('"memory_effects_test.read"'))
+
+    write_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # Writes cannot be CSE'd.
+    # CHECK: CSE write count: 2
+    print("CSE write count:", write_cse.count('"memory_effects_test.write"'))
+
+    read_across_write = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            "memory_effects_test.write_barrier"(%arg0) : (i32) -> ()
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A potentially-aliasing Write on the default resource blocks Read CSE.
+    # CHECK: CSE read across write count: 2
+    print(
+        "CSE read across write count:",
+        read_across_write.count('"memory_effects_test.read"'),
+    )
+
+    dead_code = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.no_effect"() : () -> ()
+            "memory_effects_test.read_dead"() : () -> ()
+            "memory_effects_test.write_dead"() : () -> ()
+            "memory_effects_test.free_dead"() : () -> ()
+            "memory_effects_test.allocate_dead"() : () -> ()
+            %0 = "memory_effects_test.allocate_result"() : () -> i32
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # Empty and Read-only effect lists are dead. Write, Free and untargeted
+    # Allocate effects are observable. An Allocate targeting its own unused
+    # result is dead.
+    # CHECK: DCE no effect count: 0
+    # CHECK: DCE read count: 0
+    # CHECK: DCE write count: 1
+    # CHECK: DCE free count: 1
+    # CHECK: DCE untargeted allocate count: 1
+    # CHECK: DCE result allocate count: 0
+    print("DCE no effect count:", dead_code.count('"memory_effects_test.no_effect"'))
+    print("DCE read count:", dead_code.count('"memory_effects_test.read_dead"'))
+    print("DCE write count:", dead_code.count('"memory_effects_test.write_dead"'))
+    print("DCE free count:", dead_code.count('"memory_effects_test.free_dead"'))
+    print(
+        "DCE untargeted allocate count:",
+        dead_code.count('"memory_effects_test.allocate_dead"'),
+    )
+    print(
+        "DCE result allocate count:",
+        dead_code.count('"memory_effects_test.allocate_result"'),
+    )
+
+    target_variants = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.block_argument_target"() ({
+            ^bb0(%arg0: i32):
+            }) : () -> ()
+            "memory_effects_test.symbol_target"() : () -> ()
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # These Read effects exercise BlockArgument and SymbolRefAttr targets and
+    # remain removable by trivial-dce.
+    # CHECK: invalid symbol target: target Attribute must be a SymbolRefAttr
+    # CHECK: invalid parameters: parameters must be an Attribute or None
+    # CHECK: DCE block argument target count: 0
+    # CHECK: DCE symbol target count: 0
+    print(
+        "DCE block argument target count:",
+        target_variants.count('"memory_effects_test.block_argument_target"'),
+    )
+    print(
+        "DCE symbol target count:",
+        target_variants.count('"memory_effects_test.symbol_target"'),
+    )

>From ae347ca90903a7a23e8cd26d39019ad5cd75afa9 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 22:56:28 +0800
Subject: [PATCH 2/2] fix

---
 mlir/test/python/dialects/memory_effects_op_interface.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
index 5689130c584b5..cb043cb961bca 100644
--- a/mlir/test/python/dialects/memory_effects_op_interface.py
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -155,8 +155,6 @@ def run_pass(source, pipeline):
 with ir.Context(), ir.Location.unknown():
     MemoryEffectsTest.load()
 
-    # The static properties return wrappers around borrowed, statically-owned
-    # C++ singletons.
     # CHECK: memory effect properties: True True True True
     print(
         "memory effect properties:",


From llvmlistbot at llvm.org  Sun Jul 19 07:56:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 07:56:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5ce5b8.510f576d.2f694c.3b6b@mx.google.com>


https://github.com/PragmaTwice ready_for_review https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 07:57:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 07:57:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5ce5db.1728eaf3.2a2f6.4e50@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Twice (PragmaTwice)

<details>
<summary>Changes</summary>

Python implementations of `MemoryEffectsOpInterface` receive a `MemoryEffectInstancesList` (added in #<!-- -->176920), but currently have no generic way to populate it.

This adds the C API for constructing and appending memory effect instances and exposes it in Python through `MemoryEffect`, `SideEffectResource`, and `MemoryEffectInstancesList.append`. The tests use CSE and DCE to verify that the declared effects are observed by MLIR passes.

Assisted-by: GPT 5.6 Sol

---

Patch is 30.95 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210586.diff


8 Files Affected:

- (modified) mlir/include/mlir-c/Interfaces.h (+78) 
- (modified) mlir/include/mlir/Bindings/Python/IRInterfaces.h (+52-1) 
- (modified) mlir/include/mlir/CAPI/Interfaces.h (+4) 
- (modified) mlir/lib/Bindings/Python/DialectTransform.cpp (+10-10) 
- (modified) mlir/lib/Bindings/Python/IRInterfaces.cpp (+104-3) 
- (modified) mlir/lib/CAPI/Interfaces/Interfaces.cpp (+72) 
- (modified) mlir/test/CAPI/ir.c (+43) 
- (added) mlir/test/python/dialects/memory_effects_op_interface.py (+289) 


``````````diff
diff --git a/mlir/include/mlir-c/Interfaces.h b/mlir/include/mlir-c/Interfaces.h
index a5251dc78471f..a416a6ab76f87 100644
--- a/mlir/include/mlir-c/Interfaces.h
+++ b/mlir/include/mlir-c/Interfaces.h
@@ -28,7 +28,10 @@ extern "C" {
   };                                                                           \
   typedef struct name name
 
+DEFINE_C_API_STRUCT(MlirMemoryEffect, void);
+DEFINE_C_API_STRUCT(MlirMemoryEffectInstance, void);
 DEFINE_C_API_STRUCT(MlirMemoryEffectInstancesList, void);
+DEFINE_C_API_STRUCT(MlirSideEffectResource, void);
 
 #undef DEFINE_C_API_STRUCT
 
@@ -144,6 +147,81 @@ mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectsOpInterface
 //===---------------------------------------------------------------------===//
 
+/// Returns the borrowed singleton instance of the allocate memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsAllocateGet(void);
+
+/// Returns the borrowed singleton instance of the free memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsFreeGet(void);
+
+/// Returns the borrowed singleton instance of the read memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsReadGet(void);
+
+/// Returns the borrowed singleton instance of the write memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsWriteGet(void);
+
+/// Returns the borrowed singleton instance of the default side effect
+/// resource.
+MLIR_CAPI_EXPORTED MlirSideEffectResource
+mlirSideEffectsDefaultResourceGet(void);
+
+/// Creates a memory effect instance without an associated IR entity.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation operand.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpOperand(MlirMemoryEffect effect,
+                                           MlirOpOperand opOperand,
+                                           MlirAttribute parameters, int stage,
+                                           bool effectOnFullRegion,
+                                           MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation result.
+/// `result` must wrap an OpResult. `parameters` may be a null attribute. The
+/// caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpResult(MlirMemoryEffect effect,
+                                          MlirValue result,
+                                          MlirAttribute parameters, int stage,
+                                          bool effectOnFullRegion,
+                                          MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a block argument.
+/// `blockArgument` must wrap a BlockArgument. `parameters` may be a null
+/// attribute. The caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a symbol. `symbol` must be
+/// a SymbolRefAttr. `parameters` may be a null attribute. The caller owns the
+/// returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForSymbol(MlirMemoryEffect effect,
+                                        MlirAttribute symbol,
+                                        MlirAttribute parameters, int stage,
+                                        bool effectOnFullRegion,
+                                        MlirSideEffectResource resource);
+
+/// Destroys a memory effect instance created by one of the functions above.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance);
+
+/// Appends a copy of `instance` to the given list. This does not take ownership
+/// of `instance`; the caller remains responsible for destroying it.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                    MlirMemoryEffectInstance instance);
+
 /// Returns the interface TypeID of the MemoryEffectsOpInterface.
 MLIR_CAPI_EXPORTED MlirTypeID mlirMemoryEffectsOpInterfaceTypeID(void);
 
diff --git a/mlir/include/mlir/Bindings/Python/IRInterfaces.h b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
index fb30e030b6c32..1ba995aa82c5e 100644
--- a/mlir/include/mlir/Bindings/Python/IRInterfaces.h
+++ b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
+class PyMemoryEffect {
+public:
+  explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
+
+  MlirMemoryEffect get() const { return effect; }
+
+private:
+  MlirMemoryEffect effect;
+};
+
+/// A borrowed side effect resource.
+class PySideEffectResource {
+public:
+  explicit PySideEffectResource(MlirSideEffectResource resource)
+      : resource(resource) {}
+
+  MlirSideEffectResource get() const { return resource; }
+
+private:
+  MlirSideEffectResource resource;
+};
+
+/// An owning memory effect instance.
+class PyMemoryEffectInstance {
+public:
+  explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
+      : instance(instance) {}
+  PyMemoryEffectInstance(PyMemoryEffectInstance &&other) noexcept
+      : instance(other.instance) {
+    other.instance.ptr = nullptr;
+  }
+  ~PyMemoryEffectInstance() {
+    if (instance.ptr)
+      mlirMemoryEffectInstanceDestroy(instance);
+  }
+
+  MlirMemoryEffectInstance get() const { return instance; }
+
+private:
+  MlirMemoryEffectInstance instance;
+};
+
+/// A callback-scoped view of a list of memory effect instances.
+class PyMemoryEffectsInstanceList {
+public:
+  explicit PyMemoryEffectsInstanceList(MlirMemoryEffectInstancesList effects)
+      : effects(effects) {}
+
+  MlirMemoryEffectInstancesList get() const { return effects; }
+
+private:
   MlirMemoryEffectInstancesList effects;
 };
 
diff --git a/mlir/include/mlir/CAPI/Interfaces.h b/mlir/include/mlir/CAPI/Interfaces.h
index 15afc9fb0f18e..55d850ccd01eb 100644
--- a/mlir/include/mlir/CAPI/Interfaces.h
+++ b/mlir/include/mlir/CAPI/Interfaces.h
@@ -22,5 +22,9 @@
 DEFINE_C_API_PTR_METHODS(
     MlirMemoryEffectInstancesList,
     llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance>)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffect, mlir::MemoryEffects::Effect)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffectInstance,
+                         mlir::MemoryEffects::EffectInstance)
+DEFINE_C_API_PTR_METHODS(MlirSideEffectResource, mlir::SideEffects::Resource)
 
 #endif // MLIR_CAPI_INTERFACES_H
diff --git a/mlir/lib/Bindings/Python/DialectTransform.cpp b/mlir/lib/Bindings/Python/DialectTransform.cpp
index bd72082cea7e8..f84189e3a22ff 100644
--- a/mlir/lib/Bindings/Python/DialectTransform.cpp
+++ b/mlir/lib/Bindings/Python/DialectTransform.cpp
@@ -494,38 +494,38 @@ struct ParamType : PyConcreteType<ParamType> {
 
 namespace {
 void onlyReadsHandle(nb::iterable &operands,
-                     PyMemoryEffectsInstanceList effects) {
+                     const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(),
-                               effects.effects);
+                               effects.get());
 };
 
 void consumesHandle(nb::iterable &operands,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
 void producesHandle(nb::iterable &results,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirValue> resultsVec;
   for (auto result : results)
     resultsVec.push_back(nb::cast<PyOpResult>(result).get());
   mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
-void modifiesPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformModifiesPayload(effects.effects);
+void modifiesPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformModifiesPayload(effects.get());
 }
 
-void onlyReadsPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformOnlyReadsPayload(effects.effects);
+void onlyReadsPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformOnlyReadsPayload(effects.get());
 }
 } // namespace
 
diff --git a/mlir/lib/Bindings/Python/IRInterfaces.cpp b/mlir/lib/Bindings/Python/IRInterfaces.cpp
index c4e87246bd68c..4495cb6e8397d 100644
--- a/mlir/lib/Bindings/Python/IRInterfaces.cpp
+++ b/mlir/lib/Bindings/Python/IRInterfaces.cpp
@@ -34,6 +34,70 @@ its return shaped type components. Raises ValueError on failure.)";
 
 namespace {
 
+MlirAttribute unwrapOptionalAttribute(const nb::object &attribute) {
+  if (attribute.is_none())
+    return mlirAttributeGetNull();
+
+  PyAttribute *pyAttribute = nullptr;
+  if (!nb::try_cast<PyAttribute *>(attribute, pyAttribute) || !pyAttribute)
+    throw nb::type_error("parameters must be an Attribute or None");
+  return pyAttribute->get();
+}
+
+void appendMemoryEffectInstance(PyMemoryEffectsInstanceList &effects,
+                                const PyMemoryEffect &effect,
+                                const nb::object &target,
+                                const nb::object &parameters, int stage,
+                                bool effectOnFullRegion,
+                                const PySideEffectResource &resource) {
+  MlirMemoryEffectInstancesList list = effects.get();
+  MlirAttribute unwrappedParameters = unwrapOptionalAttribute(parameters);
+
+  MlirMemoryEffectInstance rawInstance{nullptr};
+  if (target.is_none()) {
+    rawInstance =
+        mlirMemoryEffectInstanceCreate(effect.get(), unwrappedParameters, stage,
+                                       effectOnFullRegion, resource.get());
+  } else {
+    PyOpOperand *opOperand = nullptr;
+    PyValue *value = nullptr;
+    PyAttribute *attribute = nullptr;
+    if (nb::try_cast<PyOpOperand *>(target, opOperand) && opOperand) {
+      rawInstance = mlirMemoryEffectInstanceCreateForOpOperand(
+          effect.get(), *opOperand, unwrappedParameters, stage,
+          effectOnFullRegion, resource.get());
+    } else if (nb::try_cast<PyValue *>(target, value) && value) {
+      MlirValue mlirValue = value->get();
+      if (mlirValueIsAOpResult(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForOpResult(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else if (mlirValueIsABlockArgument(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForBlockArgument(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else {
+        throw nb::type_error(
+            "target Value must be an OpResult or BlockArgument");
+      }
+    } else if (nb::try_cast<PyAttribute *>(target, attribute) && attribute) {
+      MlirAttribute symbol = attribute->get();
+      if (!mlirAttributeIsASymbolRef(symbol))
+        throw nb::type_error("target Attribute must be a SymbolRefAttr");
+      rawInstance = mlirMemoryEffectInstanceCreateForSymbol(
+          effect.get(), symbol, unwrappedParameters, stage, effectOnFullRegion,
+          resource.get());
+    } else {
+      throw nb::type_error(
+          "target must be an OpOperand, OpResult, BlockArgument, "
+          "SymbolRefAttr, or None");
+    }
+  }
+
+  PyMemoryEffectInstance instance(rawInstance);
+  mlirMemoryEffectInstancesListAppend(list, instance.get());
+}
+
 /// Takes in an optional ist of operands and converts them into a std::vector
 /// of MlirVlaues. Returns an empty std::vector if the list is empty.
 std::vector<MlirValue> wrapOperands(std::optional<nb::sequence> operandList) {
@@ -474,9 +538,46 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("Speculatable", MlirSpeculatabilitySpeculatable)
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
-  auto memoryEffectsInstanceListClass =
-      nb::class_<PyMemoryEffectsInstanceList>(m, "MemoryEffectInstancesList");
-  (void)memoryEffectsInstanceListClass;
+  nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
+      .def_prop_ro_static("allocate",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(
+                                mlirMemoryEffectsAllocateGet());
+                          })
+      .def_prop_ro_static("free",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
+                          })
+      .def_prop_ro_static("read",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
+                          })
+      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
+      });
+
+  nb::class_<PySideEffectResource>(m, "SideEffectResource",
+                                   "A side effect resource.")
+      .def_prop_ro_static("default", [](nb::object & /*class*/) {
+        return PySideEffectResource(mlirSideEffectsDefaultResourceGet());
+      });
+
+  nb::class_<PyMemoryEffectsInstanceList>(
+      m, "MemoryEffectInstancesList",
+      "A memory effect list that is valid only during get_effects.")
+      .def("append", &appendMemoryEffectInstance, nb::arg("effect"),
+           nb::arg("target").none() = nb::none(), nb::kw_only(),
+           nb::arg("parameters").none() = nb::none(), nb::arg("stage") = 0,
+           nb::arg("effect_on_full_region") = false,
+           nb::arg("resource") =
+               PySideEffectResource(mlirSideEffectsDefaultResourceGet()),
+           nb::sig("def append(self, effect: MemoryEffect, target: OpOperand | "
+                   "OpResult | BlockArgument | SymbolRefAttr | None = None, *, "
+                   "parameters: Attribute | None = None, stage: int = 0, "
+                   "effect_on_full_region: bool = False, resource: "
+                   "SideEffectResource = ...) -> None"),
+           "Append a memory effect instance. The target may be an OpOperand, "
+           "OpResult, BlockArgument, SymbolRefAttr, or None.");
 
   PyConditionallySpeculatableOpInterface::bind(m);
   PyInferShapedTypeOpInterface::bind(m);
diff --git a/mlir/lib/CAPI/Interfaces/Interfaces.cpp b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
index 35a2bd562a8a1..1bf8dffa9c431 100644
--- a/mlir/lib/CAPI/Interfaces/Interfaces.cpp
+++ b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
@@ -280,6 +280,78 @@ MlirSpeculatability mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectOpInterface
 //===---------------------------------------------------------------------===//
 
+MlirMemoryEffect mlirMemoryEffectsAllocateGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Allocate::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsFreeGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Free::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsReadGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Read::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsWriteGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Write::get()));
+}
+
+MlirSideEffectResource mlirSideEffectsDefaultResourceGet() {
+  return wrap(static_cast<SideEffects::Resource *>(
+      SideEffects::DefaultResource::get()));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(parameters), stage, effectOnFullRegion,
+      unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpOperand(
+    MlirMemoryEffect effect, MlirOpOperand opOperand, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(opOperand), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpResult(
+    MlirMemoryEffect effect, MlirValue result, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<OpResult>(unwrap(result)), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<BlockArgument>(unwrap(blockArgument)),
+      unwrap(parameters), stage, effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForSymbol(
+    MlirMemoryEffect effect, MlirAttribute symbol, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<SymbolRefAttr>(unwrap(symbol)), unwrap(parameters),
+      stage, effectOnFullRegion, unwrap(resource)));
+}
+
+void mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance) {
+  delete unwrap(instance);
+}
+
+void mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                         MlirMemoryEffectInstance instance) {
+  unwrap(list)->push_back(*unwrap(instance));
+}
+
 MlirTypeID mlirMemoryEffectsOpInterfaceTypeID() {
   return wrap(MemoryEffectOpInterface::getInterfaceID());
 }
diff --git a/mlir/test/CAPI/ir.c b/mlir/test/CAPI/ir.c
index 57ae8b9a2819b..3c72209ad95c3 100644
--- a/mlir/test/CAPI/ir.c
+++ b/mlir/test/CAPI/ir.c
@@ -2589,6 +2589,8 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: arith.constant speculatability: 1
 
   MlirOperationState storeState = mlirOperationStateGet(storeName, loc);
+  MlirValue constantResult = mlirOperationGetResult(constantOp, 0);
+  mlirOperationStateAddOperands(&storeState, 1, &constantResult);
   MlirOperation storeOp = mlirOperationCreate(&storeState);
   if (mlirOperationImplementsInterface(storeOp, condSpecTypeID)) {
     fprintf(stderr, "ERROR: Expected memref.store instance to not implement "
@@ -2619,6 +2621,47 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: memref.store speculatability: 2
   // CHECK: callback count: 1
 
+  MlirMemoryEffect allocate = mlirMemoryEffectsAllocateGet();
+  MlirMemoryEffect free = mlirMemoryEffectsFreeGet();
+  MlirMemoryEffect read...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 08:35:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 08:35:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5ceead.5a0412d2.3daf32.f1c9@mx.google.com>


https://github.com/Naaz30 updated https://github.com/llvm/llvm-project/pull/210595

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH 1/4] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

>From b8c1e1efbc222e0bb37782daa19ddb234e63eb01 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:00:05 +0530
Subject: [PATCH 2/4] MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp    | 10 ++---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 41 +++++++++++++++----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494e19c..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,8 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -789,9 +787,7 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      // Only reject an explicitly unsupported chip; a missing gpu.module/target
-      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
-      // path also works for standalone/non-GPU-kernel-context IR.
+      // reject an explicitly unsupported chip
       if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
           *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }

>From 232df7d956d8cc04c4e8e0a28e4f4ae6a2feb3d6 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:31:36 +0530
Subject: [PATCH 3/4] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp        | 5 +++--
 mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir | 8 ++------
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 03808d2d0f8ff..fae578eefd107 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,8 +720,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts multi-dim tile as one flat, contiguous run. 
-        // This is only valid when the underlying mem_desc region is contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run.
+        // This is only valid when the underlying mem_desc region is contiguous
+        // in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 50f7263b65e05..040900d5f40f3 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -335,9 +335,6 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
         : !xegpu.mem_desc<32x32xf8E8M0FNU>,
       index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
-}
-
-// -----
 
 // CHECK-LABEL: func.func @m(
 // CHECK: memref.extract_aligned_pointer_as_index
@@ -345,8 +342,7 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 // CHECK: llvm.load
 // CHECK-SAME: vector<32xf32>
 
-module {
-  func.func @m(% arg0 : index) {
+func.func @m(%arg0: index) {
     % alloca =
         memref.alloca()
         : memref<4x8xf32, 3> %
@@ -355,4 +351,4 @@ module {
                 xegpu.load_matrix %
                 0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
   }
-}
+}
\ No newline at end of file

>From 53f03a3dcc1adf30f3e4bf87324afd04759ccd78 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 21:04:52 +0530
Subject: [PATCH 4/4] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 .../Conversion/XeGPUToXeVM/loadstore_matrix.mlir | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 040900d5f40f3..1ea11f0628754 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -335,20 +335,4 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
         : !xegpu.mem_desc<32x32xf8E8M0FNU>,
       index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
-
-// CHECK-LABEL: func.func @m(
-// CHECK: memref.extract_aligned_pointer_as_index
-// CHECK: llvm.inttoptr
-// CHECK: llvm.load
-// CHECK-SAME: vector<32xf32>
-
-func.func @m(%arg0: index) {
-    % alloca =
-        memref.alloca()
-        : memref<4x8xf32, 3> %
-        0 = xegpu.create_mem_desc %
-            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
-                xegpu.load_matrix %
-                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
-  }
 }
\ No newline at end of file


From llvmlistbot at llvm.org  Sun Jul 19 09:26:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 09:26:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5cfaa5.d681cab9.169270.abfc@mx.google.com>


https://github.com/Naaz30 updated https://github.com/llvm/llvm-project/pull/210595

>From 54feb26eb7c8a206e3440d3eddbfddcb612ff92f Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sat, 18 Jul 2026 02:05:32 +0530
Subject: [PATCH 1/5] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 6144d7c0c1a15..645fb6494e19c 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -719,9 +719,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // Some transforms may leave unit dimension in the 2D vector, adaptors do
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
-        assert(llvm::count_if(vecType.getShape(),
-                              [](int64_t d) { return d != 1; }) <= 1 &&
-               "Expected either 1D vector or nD with unit dimensions");
+        // Flatten to 1D
+        // Accepts genuine multi-dim tiles 
+        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
+        // tile as one flat, contiguous run. This is only valid when the
+        // underlying `mem_desc` region is actually contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -787,9 +789,11 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      if (!chipOpt ||
-          (*chipOpt != "pvc" && *chipOpt != "bmg" && *chipOpt != "cri")) {
-        // the lowering for chunk load only works for pvc, bmg or cri
+      // Only reject an explicitly unsupported chip; a missing gpu.module/target
+      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
+      // path also works for standalone/non-GPU-kernel-context IR.
+      if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
+          *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
             op, "The lowering is specific to pvc, bmg or cri.");
       }
@@ -1626,4 +1630,4 @@ void mlir::populateXeGPUToXeVMConversionPatterns(
   patterns.add<DpasMxToXeVMPattern>(typeConverter, patterns.getContext());
   patterns.add<ExtfToXeVMPattern, TruncfToXeVMPattern>(typeConverter,
                                                        patterns.getContext());
-}
+}
\ No newline at end of file

>From b8c1e1efbc222e0bb37782daa19ddb234e63eb01 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:00:05 +0530
Subject: [PATCH 2/5] MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(#208902)

---
 .../Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp    | 10 ++---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 41 +++++++++++++++----
 2 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 645fb6494e19c..03808d2d0f8ff 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,10 +720,8 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts genuine multi-dim tiles 
-        // (e.g. a non-distributed `vector<4x8xf32>` result) by reinterpreting the whole
-        // tile as one flat, contiguous run. This is only valid when the
-        // underlying `mem_desc` region is actually contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run. 
+        // This is only valid when the underlying mem_desc region is contiguous in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
@@ -789,9 +787,7 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
 
     if (valOrResVecTy.getNumElements() >= 1) {
       auto chipOpt = xegpu::getChipStr(op);
-      // Only reject an explicitly unsupported chip; a missing gpu.module/target
-      // (chipOpt == nullopt) is no longer treated as a hard failure, so this
-      // path also works for standalone/non-GPU-kernel-context IR.
+      // reject an explicitly unsupported chip
       if (chipOpt && *chipOpt != "pvc" && *chipOpt != "bmg" &&
           *chipOpt != "cri") {
         return rewriter.notifyMatchFailure(
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 07fb09fa2c24b..50f7263b65e05 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,10 +1,11 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
-  //CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape =
+  // [1,1,32,32],strides=[1024,1024,32,1])
+  // CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -324,12 +325,34 @@ gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    %c0 = arith.constant 0 : index
-    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
-    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
-    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
-    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
-    gpu.return %1 : vector<8xf8E8M0FNU>
+    % c0 = arith.constant 0 : index % 0 =
+               xegpu.create_mem_desc %
+               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
+               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
+               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
+               // to vector<8xf8E8M0FNU>
+               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
+        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
+      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
+}
 
+// -----
+
+// CHECK-LABEL: func.func @m(
+// CHECK: memref.extract_aligned_pointer_as_index
+// CHECK: llvm.inttoptr
+// CHECK: llvm.load
+// CHECK-SAME: vector<32xf32>
+
+module {
+  func.func @m(% arg0 : index) {
+    % alloca =
+        memref.alloca()
+        : memref<4x8xf32, 3> %
+        0 = xegpu.create_mem_desc %
+            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
+                xegpu.load_matrix %
+                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
+  }
 }

>From 232df7d956d8cc04c4e8e0a28e4f4ae6a2feb3d6 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 19:31:36 +0530
Subject: [PATCH 3/5] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp        | 5 +++--
 mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir | 8 ++------
 2 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
index 03808d2d0f8ff..fae578eefd107 100644
--- a/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
+++ b/mlir/lib/Conversion/XeGPUToXeVM/XeGPUToXeVM.cpp
@@ -720,8 +720,9 @@ class LoadStoreMatrixToXeVMPattern : public OpConversionPattern<OpType> {
       // not catch it for results.
       if (auto vecType = dyn_cast<VectorType>(resType)) {
         // Flatten to 1D
-        // Accepts multi-dim tile as one flat, contiguous run. 
-        // This is only valid when the underlying mem_desc region is contiguous in memory
+        // Accepts multi-dim tile as one flat, contiguous run.
+        // This is only valid when the underlying mem_desc region is contiguous
+        // in memory
         resType = VectorType::get({vecType.getNumElements()},
                                   vecType.getElementType());
       }
diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 50f7263b65e05..040900d5f40f3 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -335,9 +335,6 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
         : !xegpu.mem_desc<32x32xf8E8M0FNU>,
       index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
-}
-
-// -----
 
 // CHECK-LABEL: func.func @m(
 // CHECK: memref.extract_aligned_pointer_as_index
@@ -345,8 +342,7 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 // CHECK: llvm.load
 // CHECK-SAME: vector<32xf32>
 
-module {
-  func.func @m(% arg0 : index) {
+func.func @m(%arg0: index) {
     % alloca =
         memref.alloca()
         : memref<4x8xf32, 3> %
@@ -355,4 +351,4 @@ module {
                 xegpu.load_matrix %
                 0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
   }
-}
+}
\ No newline at end of file

>From 53f03a3dcc1adf30f3e4bf87324afd04759ccd78 Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 21:04:52 +0530
Subject: [PATCH 4/5] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 .../Conversion/XeGPUToXeVM/loadstore_matrix.mlir | 16 ----------------
 1 file changed, 16 deletions(-)

diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 040900d5f40f3..1ea11f0628754 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -335,20 +335,4 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
         : !xegpu.mem_desc<32x32xf8E8M0FNU>,
       index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
   }
-
-// CHECK-LABEL: func.func @m(
-// CHECK: memref.extract_aligned_pointer_as_index
-// CHECK: llvm.inttoptr
-// CHECK: llvm.load
-// CHECK-SAME: vector<32xf32>
-
-func.func @m(%arg0: index) {
-    % alloca =
-        memref.alloca()
-        : memref<4x8xf32, 3> %
-        0 = xegpu.create_mem_desc %
-            alloca : memref<4x8xf32, 3>->!xegpu.mem_desc<4x8xf32> % 1 =
-                xegpu.load_matrix %
-                0 [0, 0] : !xegpu.mem_desc<4x8xf32>->vector<4x8xf32> return
-  }
 }
\ No newline at end of file

>From 94617e6e8d43b27fb89154c70b58fb2fb03f72bf Mon Sep 17 00:00:00 2001
From: Naazni Yahya <naazniyahya at Naaznis-MacBook-Air.local>
Date: Sun, 19 Jul 2026 21:55:58 +0530
Subject: [PATCH 5/5] [MLIR][Conversion][XeGPU][XeVM] Added xegpu load/store
 flattening(llvm#208902)

---
 .../XeGPUToXeVM/loadstore_matrix.mlir         | 23 ++++++++-----------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
index 1ea11f0628754..3307e10451008 100644
--- a/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
+++ b/mlir/test/Conversion/XeGPUToXeVM/loadstore_matrix.mlir
@@ -1,11 +1,10 @@
 // RUN: mlir-opt  -split-input-file -convert-xegpu-to-xevm %s | FileCheck %s
 
-gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
+gpu.module @test_kernel [#xevm.target<chip = "pvc">] {
 
   // e.g. for mem_desc<32x32xf16, @strides=[1, 16]>
-  // its memory layout tuple is (blocked shape =
-  // [1,1,32,32],strides=[1024,1024,32,1])
-  // CHECK-LABEL: load_store_matrix_plain
+  // its memory layout tuple is (blocked shape = [1,1,32,32],strides=[1024,1024,32,1])
+  //CHECK-LABEL: load_store_matrix_plain
   gpu.func @load_store_matrix_plain(%arg0: memref<4096xi8, 3>) -> f32 {
 
     //CHECK: %[[INTPTR:.*]] = memref.extract_aligned_pointer_as_index %arg0 : memref<4096xi8, 3> -> index
@@ -325,14 +324,12 @@ gpu.module @test_kernel[#xevm.target<chip = "pvc">] {
 
   //CHECK-LABEL: load_matrix_f8e8m0_vector
   gpu.func @load_matrix_f8e8m0_vector(%arg0: memref<1024xi8, 3>) -> vector<8xf8E8M0FNU> {
-    % c0 = arith.constant 0 : index % 0 =
-               xegpu.create_mem_desc %
-               arg0 : memref<1024xi8, 3>->!xegpu.mem_desc<32x32xf8E8M0FNU>
-               // CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> ->
-               // vector<8xi8> CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8>
-               // to vector<8xf8E8M0FNU>
-               % 1 = xegpu.load_matrix % 0 [% c0, % c0]
-        : !xegpu.mem_desc<32x32xf8E8M0FNU>,
-      index, index->vector<8xf8E8M0FNU> gpu.return % 1 : vector<8xf8E8M0FNU>
+    %c0 = arith.constant 0 : index
+    %0 = xegpu.create_mem_desc %arg0 : memref<1024xi8, 3> -> !xegpu.mem_desc<32x32xf8E8M0FNU>
+    //CHECK: %[[LOADEDV:.*]] = llvm.load %{{.*}} : !llvm.ptr<3> -> vector<8xi8>
+    //CHECK: vector.bitcast %[[LOADEDV]] : vector<8xi8> to vector<8xf8E8M0FNU>
+    %1 = xegpu.load_matrix %0[%c0, %c0]: !xegpu.mem_desc<32x32xf8E8M0FNU>, index, index -> vector<8xf8E8M0FNU>
+    gpu.return %1 : vector<8xf8E8M0FNU>
   }
+
 }
\ No newline at end of file


From llvmlistbot at llvm.org  Sun Jul 19 09:52:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 09:52:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Relax
 LoadStoreMatrixToXeVMPattern to accept non-distributed
 load_matrix/store_matrix (PR #210595)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210595@github.com>
Message-ID: <6a5d00e1.160324b0.1f653e.3ce2@mx.google.com>


https://github.com/Naaz30 edited https://github.com/llvm/llvm-project/pull/210595

From llvmlistbot at llvm.org  Sun Jul 19 11:07:37 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:07:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1269.a1accbb4.5eca2.fb6c@mx.google.com>


================
@@ -34,6 +34,70 @@ its return shaped type components. Raises ValueError on failure.)";
 
 namespace {
 
+MlirAttribute unwrapOptionalAttribute(const nb::object &attribute) {
+  if (attribute.is_none())
+    return mlirAttributeGetNull();
+
+  PyAttribute *pyAttribute = nullptr;
+  if (!nb::try_cast<PyAttribute *>(attribute, pyAttribute) || !pyAttribute)
+    throw nb::type_error("parameters must be an Attribute or None");
+  return pyAttribute->get();
+}
+
+void appendMemoryEffectInstance(PyMemoryEffectsInstanceList &effects,
+                                const PyMemoryEffect &effect,
+                                const nb::object &target,
+                                const nb::object &parameters, int stage,
+                                bool effectOnFullRegion,
+                                const PySideEffectResource &resource) {
+  MlirMemoryEffectInstancesList list = effects.get();
+  MlirAttribute unwrappedParameters = unwrapOptionalAttribute(parameters);
+
+  MlirMemoryEffectInstance rawInstance{nullptr};
+  if (target.is_none()) {
+    rawInstance =
+        mlirMemoryEffectInstanceCreate(effect.get(), unwrappedParameters, stage,
+                                       effectOnFullRegion, resource.get());
+  } else {
+    PyOpOperand *opOperand = nullptr;
+    PyValue *value = nullptr;
+    PyAttribute *attribute = nullptr;
+    if (nb::try_cast<PyOpOperand *>(target, opOperand) && opOperand) {
+      rawInstance = mlirMemoryEffectInstanceCreateForOpOperand(
+          effect.get(), *opOperand, unwrappedParameters, stage,
+          effectOnFullRegion, resource.get());
+    } else if (nb::try_cast<PyValue *>(target, value) && value) {
+      MlirValue mlirValue = value->get();
+      if (mlirValueIsAOpResult(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForOpResult(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else if (mlirValueIsABlockArgument(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForBlockArgument(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else {
+        throw nb::type_error(
+            "target Value must be an OpResult or BlockArgument");
+      }
+    } else if (nb::try_cast<PyAttribute *>(target, attribute) && attribute) {
+      MlirAttribute symbol = attribute->get();
+      if (!mlirAttributeIsASymbolRef(symbol))
+        throw nb::type_error("target Attribute must be a SymbolRefAttr");
+      rawInstance = mlirMemoryEffectInstanceCreateForSymbol(
+          effect.get(), symbol, unwrappedParameters, stage, effectOnFullRegion,
+          resource.get());
+    } else {
+      throw nb::type_error(
+          "target must be an OpOperand, OpResult, BlockArgument, "
+          "SymbolRefAttr, or None");
+    }
+  }
+
+  PyMemoryEffectInstance instance(rawInstance);
+  mlirMemoryEffectInstancesListAppend(list, instance.get());
----------------
makslevental wrote:

```suggestion
  mlirMemoryEffectInstancesListAppend(list, rawInstance);
```
no?

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:10:28 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:10:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1314.f1c1923d.2d2b59.cce7@mx.google.com>


================
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
----------------
makslevental wrote:

just curious what does `borrowed` mean here?

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:12:00 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:12:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1370.92ea0e2a.4a9a5.a641@mx.google.com>


================
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
----------------
makslevental wrote:

oh nm - that you don't need to destroy?

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:18:30 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:18:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d14f6.3cce8983.3cf542.84ee@mx.google.com>


================
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
----------------
makslevental wrote:

i feel like "borrowed" vs "owning" isn't the correct terminology? I think it's just "kind" vs "instance" ?

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:19:27 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:19:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d152f.3cce8983.3cf542.9650@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:20:32 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:20:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1570.9edb5c9d.280be3.85f6@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:21:25 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d15a5.3cce8983.3cf542.b9b3@mx.google.com>


================
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
+class PyMemoryEffect {
+public:
+  explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
+
+  MlirMemoryEffect get() const { return effect; }
+
+private:
+  MlirMemoryEffect effect;
+};
+
+/// A borrowed side effect resource.
+class PySideEffectResource {
+public:
+  explicit PySideEffectResource(MlirSideEffectResource resource)
+      : resource(resource) {}
+
+  MlirSideEffectResource get() const { return resource; }
+
+private:
+  MlirSideEffectResource resource;
+};
+
+/// An owning memory effect instance.
+class PyMemoryEffectInstance {
+public:
+  explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
+      : instance(instance) {}
+  PyMemoryEffectInstance(PyMemoryEffectInstance &&other) noexcept
+      : instance(other.instance) {
+    other.instance.ptr = nullptr;
----------------
makslevental wrote:

just curious: why do you need a move ctor?

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:25:10 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:25:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1686.4987c784.2c60a9.ec56@mx.google.com>


https://github.com/makslevental edited https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:27:19 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:27:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a5d1707.0c05e68e.3c2f5d.9f74@mx.google.com>


================
@@ -474,9 +538,46 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("Speculatable", MlirSpeculatabilitySpeculatable)
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
-  auto memoryEffectsInstanceListClass =
-      nb::class_<PyMemoryEffectsInstanceList>(m, "MemoryEffectInstancesList");
-  (void)memoryEffectsInstanceListClass;
+  nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
+      .def_prop_ro_static("allocate",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(
+                                mlirMemoryEffectsAllocateGet());
+                          })
+      .def_prop_ro_static("free",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
+                          })
+      .def_prop_ro_static("read",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
+                          })
+      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
----------------
makslevental wrote:

I feel like these should be capitalized to signal you're creating an object;


```suggestion
      .def_prop_ro_static("Allocate",
                          [](nb::object & /*class*/) {
                            return PyMemoryEffect(
                                mlirMemoryEffectsAllocateGet());
                          })
      .def_prop_ro_static("Free",
                          [](nb::object & /*class*/) {
                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
                          })
      .def_prop_ro_static("Read",
                          [](nb::object & /*class*/) {
                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
                          })
      .def_prop_ro_static("Write", [](nb::object & /*class*/) {
        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
```

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Sun Jul 19 11:29:46 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Sun, 19 Jul 2026 11:29:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5d179a.7270d2fd.a9bd6.39c5@mx.google.com>


https://github.com/makslevental approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Sun Jul 19 13:54:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Th=C3=A9o_Degioanni?=)
Date: Sun, 19 Jul 2026 13:54:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5d398a.f06f4994.2f0620.b529@mx.google.com>


Moxinilian wrote:

@gysit  I'm a bit out of the loop with respect to the LLVM implementation of MemorySlot, so I trust your judgement.

@iamaayushrivastava do you have merge rights into the LLVM repo?

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Sun Jul 19 14:27:45 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Sun, 19 Jul 2026 14:27:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5d4151.d681cab9.169270.9bf4@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/205464

>From ad1f5134c8a574c016d2dd2c0241afef08781a48 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 01/22] [mlir][EmitC] Create a pass to add a reflection map to
 a class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |   2 +
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  52 +++++
 .../Dialect/EmitC/Transforms/Transforms.h     |   7 +
 .../EmitC/Transforms/AddReflectionMap.cpp     | 181 ++++++++++++++++++
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |   1 +
 .../Dialect/EmitC/add-reflection-map.mlir     |  58 ++++++
 6 files changed, 301 insertions(+)
 create mode 100644 mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
 create mode 100644 mlir/test/Dialect/EmitC/add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 1af4aa06fa811..56596110a1e90 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,6 +16,8 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
+#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index c34c3303a6ab3..6da57c07a8047 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,4 +60,56 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
+def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+  let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
+  let description = [{
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
+    EmitC classes, enabling runtime lookup of class fields by name. 
+    This requires that the class has fields with attributes.
+    Each `emitc.field` is expected to have an attribute (configured by the
+    `field-attr-name` option) that is an array containing a single string
+    attribute.
+
+
+    Example:
+
+    ```mlir
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.func @execute() { ... }
+    }
+
+    Becomes:
+
+    emitc.class @MyClass {
+      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
+      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
+          -> !emitc.ptr<!emitc.opaque<"char">> {
+        %0 = get_field @reflectionMap
+          : !emitc.opaque<"const std::map<std::string, char*>">
+        %1 = member_call_opaque %0 "at"(%arg0)
+          : !emitc.opaque<"const std::map<std::string, char*>">,
+            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      }
+      emitc.func @execute() { ... }
+    }
+    ```
+  }];
+  let dependentDialects = ["emitc::EmitCDialect"];
+  let options = [
+    Option<"fieldAttrName", "field-attr-name", "std::string",
+      /*default=*/"",
+      "Attribute key used to extract field names from a field's "
+      "attribute dictionary">,
+    ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
+      "Attribute keys that, if present on a field, exclude it "
+      "from the reflection map">
+  ];
+}
+
 #endif // MLIR_DIALECT_EMITC_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 791e545a8edcf..8eadfe9a2460d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -12,6 +12,7 @@
 #include "mlir/Dialect/EmitC/IR/EmitC.h"
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -37,6 +38,12 @@ void populateWrapFuncInClass(
     RewritePatternSet &patterns, StringRef funcName,
     DenseMap<FuncOp, llvm::DenseSet<GlobalOp>> &globalsToMove);
 
+//===----------------------------------------------------------------------===//
+// The AddReflectionMap pass.
+//===----------------------------------------------------------------------===//
+
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+
 } // namespace emitc
 } // namespace mlir
 
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
new file mode 100644
index 0000000000000..c1588463ffb7a
--- /dev/null
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -0,0 +1,181 @@
+//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(
+      builder, module.getLoc(), includeAttr,
+      /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class AddReflectionMapPass
+    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
+  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+
+    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+        
+        hasMapHdr = include == mapLibraryHeader;
+        hasStringHdr = include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect all field names
+    std::vector<std::pair<std::string, std::string>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
+        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
+          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
+          fieldNames.emplace_back(stringAttr.getValue().str(),
+                                  fieldOp.getName().str());
+          return;
+        }
+      }
+
+      bool shouldIgnore = false;
+      for (const std::string &ignoreAttr : excludedFieldAttrs) {
+        if (fieldOp->hasAttr(ignoreAttr)) {
+          shouldIgnore = true;
+          break;
+        }
+      }
+
+      if (shouldIgnore)
+        return;
+
+      fieldOp.emitError()
+          << "FieldOp must have a dictionary attribute named '"
+          << fieldAttrName << "'"
+          << "with an array containing a string attribute";
+    });
+
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
+          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+    }
+    reflectionMapContents += " }";
+
+    if (FuncOp executeFunc =
+            classOp.lookupSymbol<FuncOp>("operator()"))
+      rewriter.setInsertionPoint(executeFunc);
+    else {
+      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
+                             "to add reflection map";
+      return failure();
+    }
+
+    FieldOp reflectionMapField = FieldOp::create(
+        rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
+        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+
+    // Create getBufferForName method
+    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    FuncOp getBufferForNameFunc = FuncOp::create(
+        rewriter, reflectionMapField->getLoc(), "getBufferForName",
+        FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
+
+    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    rewriter.setInsertionPointToStart(body);
+    GetFieldOp mapField = GetFieldOp::create(
+        rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
+    Value nameArg = body->getArgument(0);
+    MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
+        rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
+        "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+
+    return success();
+  }
+
+private:
+  /// The name of the attribute on FieldOps that contains the field name
+  /// metadata for the reflection map.
+  StringRef fieldAttrName;
+
+  /// Attributes that, if present on a field, exclude it from the
+  /// reflection map.
+  llvm::SmallVector<std::string> excludedFieldAttrs;
+};
+
+void mlir::emitc::populateAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    llvm::ArrayRef<std::string> excludedFieldAttrs) {
+  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                      excludedFieldAttrs);
+}
\ No newline at end of file
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index baf67afc30072..dd8f014dc4737 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,6 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
+  AddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
new file mode 100644
index 0000000000000..a1ec0356e19f4
--- /dev/null
+++ b/mlir/test/Dialect/EmitC/add-reflection-map.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+
+
+// Tests that a reflection map is created for fields with a certain attribute.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+// Test that a reflection map is created for fields with a certain named attribute
+// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @actionClass {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
\ No newline at end of file

>From fce6ac458f4eaeb0a701082864f0f9dda68fb422 Mon Sep 17 00:00:00 2001
From: Bhavesh M <85327930+beamandala at users.noreply.github.com>
Date: Tue, 23 Jun 2026 02:37:42 +0530
Subject: [PATCH 02/22] [mlir][EmitC] Make `GlobalOps` `FieldOps` in
 wrap-emitc-func-in-class pass (#203641)

Update the `WrapFuncInClassPass` pass so that `GlobalOp`s are moved into
the `ClassOp` as `FieldOps`. This respects MLIR's behavior of resolving
references to the closest parent operation that defines a symbol table
which is the `ClassOp` that we are creating in this pass.

Without this change, references to a `GlobalOp` in `GetGlobalOp` are
failing to resolve.

Details:
- Identify `GlobalOp`s
- Create a `FieldOp` within the `ClassOp` for each `GlobalOp`
- Delete the `GlobalOp`s after all functions have been wrapped in a
class. Doing this after every function can cause an error when multiple
functions refer to the same `GlobalOp`(s) which would be deleted after
the first function is wrapped in a class.

Also renamed `fName` parameter in `populateWrapFuncInClass` to
`funcName` to match naming in `WrapFuncInClass`.

Based on PR #153452. Key differences:
- No size is set for the `globalsToMove` `SmallVector` type because I'm
not sure if the number of global variables is consistent across
different models.
- `GlobalOp`s are deleted after all functions have been processed.
- Instead of directly cloning the `GlobalOp`, an equivalent `FieldOp` is
created
- `GetGlobalOp`s are translated to `GetFieldOp`s

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8eadfe9a2460d..4c8d3978af0f1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,6 +13,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/DenseMap.h"
 
 namespace mlir {
 namespace emitc {

>From 66bed5c47b37810c550c08f63a8f6dcf918c621d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 23 Jun 2026 17:32:22 -0700
Subject: [PATCH 03/22] [mlir][EmitC] Create a pass to add a reflection map to
 a class

This creates the `add-reflection-map` pass to add a reflection map and a
helper method to EmitC classes for runtime field lookup.

Details:
- Add headers to import `map` and `string` if not already present.
- For every `ClassOp`:
	- Collect all the `FieldOp`s that contain the attribute whose
	  value we want as the reflection map key. We ignore all
	  `FieldOp`s that contain an attribute present in
	  `excludedFieldAttrs` and if there's a `FieldOp` that neither
	  has the attribute we're looking for and isn't excluded by the
	  presence of one of the `excludedFieldAttrs`, emit and error.
	- Create the reflection map where the key is the value of the
	  attribute in each applicable `FieldOp`'s attribute dictionary
	  and the value is a pointer to the field's contents.
	- Create a `getBufferForName` method which serves as a getter
	  method for the reflection map.

Based on PR #150572. Key differences:
- Adds a `excluded-field-attrs` option which is a list of attributes
  that if present on a field exclude it from the reflection map.
- Adds a helper method called `getBufferForName` which given a string
  key returns a pointer to the field's buffer.

Co-authored-by: [Jaddyen](https://github.com/Jaddyen)
---
 mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 4c8d3978af0f1..5a4781dc31ea8 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {

>From 1b949e08c522bf1befdba3fc0612dcad5d04a02a Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 24 Jun 2026 10:31:17 -0700
Subject: [PATCH 04/22] Formatting

---
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +--
 .../EmitC/Transforms/AddReflectionMap.cpp     | 47 +++++++++++--------
 2 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 5a4781dc31ea8..8f1784f61f540 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -13,8 +13,6 @@
 #include "mlir/IR/PatternMatch.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/StringRef.h"
-#include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/StringRef.h"
 
 namespace mlir {
 namespace emitc {
@@ -44,7 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns, StringRef fieldAttrName, ArrayRef<std::string> excludedFieldAttrs);
+void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
+                                      StringRef fieldAttrName,
+                                      ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
index c1588463ffb7a..d444f3abbf4b1 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
@@ -31,9 +31,8 @@ constexpr const char *stringLibraryHeader = "string";
 
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
-  return IncludeOp::create(
-      builder, module.getLoc(), includeAttr,
-      /*is_standard_include=*/builder.getUnitAttr());
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
 }
 
 class AddReflectionMapPass
@@ -43,7 +42,8 @@ class AddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName, excludedFieldAttrs);
+    populateAddReflectionMapPatterns(patterns, fieldAttrName,
+                                     excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -55,7 +55,7 @@ class AddReflectionMapPass
 
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
-        
+
         hasMapHdr = include == mapLibraryHeader;
         hasStringHdr = include == stringLibraryHeader;
       }
@@ -82,7 +82,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   AddReflectionMapClass(MLIRContext *context, StringRef attrName,
                         llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
-        excludedFieldAttrs(excludedFieldAttrs.begin(), excludedFieldAttrs.end()) {}
+        excludedFieldAttrs(excludedFieldAttrs.begin(),
+                           excludedFieldAttrs.end()) {}
 
   LogicalResult matchAndRewrite(ClassOp classOp,
                                 PatternRewriter &rewriter) const override {
@@ -114,10 +115,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError()
-          << "FieldOp must have a dictionary attribute named '"
-          << fieldAttrName << "'"
-          << "with an array containing a string attribute";
+      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
+                          << fieldAttrName << "'"
+                          << "with an array containing a string attribute";
     });
 
     std::string reflectionMapContents;
@@ -129,28 +129,34 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc =
-            classOp.lookupSymbol<FuncOp>("operator()"))
+    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError() << "ClassOp must contain a function named 'operator()' "
-                             "to add reflection map";
+      classOp.emitError()
+          << "ClassOp must contain a function named 'operator()' "
+             "to add reflection map";
       return failure();
     }
 
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
-        TypeAttr::get(mapType), emitc::OpaqueAttr::get(context, reflectionMapContents));
+        TypeAttr::get(mapType),
+        emitc::OpaqueAttr::get(context, reflectionMapContents));
 
     // Create getBufferForName method
-    emitc::OpaqueType nameType = emitc::OpaqueType::get(rewriter.getContext(), "std::string");
-    emitc::OpaqueType charType = emitc::OpaqueType::get(rewriter.getContext(), "char");
-    emitc::PointerType valType = emitc::PointerType::get(rewriter.getContext(), charType);
+    emitc::OpaqueType nameType =
+        emitc::OpaqueType::get(rewriter.getContext(), "std::string");
+    emitc::OpaqueType charType =
+        emitc::OpaqueType::get(rewriter.getContext(), "char");
+    emitc::PointerType valType =
+        emitc::PointerType::get(rewriter.getContext(), charType);
     FuncOp getBufferForNameFunc = FuncOp::create(
         rewriter, reflectionMapField->getLoc(), "getBufferForName",
         FunctionType::get(rewriter.getContext(), {nameType}, {valType}));
 
-    Block *body = rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType}, {reflectionMapField->getLoc()});
+    Block *body =
+        rewriter.createBlock(&getBufferForNameFunc.getBody(), {}, {nameType},
+                             {reflectionMapField->getLoc()});
     rewriter.setInsertionPointToStart(body);
     GetFieldOp mapField = GetFieldOp::create(
         rewriter, reflectionMapField->getLoc(), mapType, "reflectionMap");
@@ -158,7 +164,8 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
     MemberCallOpaqueOp lookupCall = MemberCallOpaqueOp::create(
         rewriter, reflectionMapField->getLoc(), valType, mapField.getResult(),
         "at", ArrayAttr{}, ArrayAttr{}, ValueRange{nameArg});
-    ReturnOp::create(rewriter, reflectionMapField->getLoc(), lookupCall.getResult(0));
+    ReturnOp::create(rewriter, reflectionMapField->getLoc(),
+                     lookupCall.getResult(0));
 
     return success();
   }

>From ab36f41166fd803ff45b2be3ca958125d82e940d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Fri, 26 Jun 2026 16:04:13 -0700
Subject: [PATCH 05/22] Address review comments

---
 .../mlir/Dialect/EmitC/Transforms/Passes.h    |  2 +-
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 10 ++-
 .../Dialect/EmitC/Transforms/Transforms.h     |  6 +-
 .../Dialect/EmitC/Transforms/CMakeLists.txt   |  2 +-
 ...ectionMap.cpp => MLGOAddReflectionMap.cpp} | 76 +++++++++--------
 ...-map.mlir => mlgo-add-reflection-map.mlir} | 84 +++++++++++++++++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 57 +++++++++++++
 7 files changed, 188 insertions(+), 49 deletions(-)
 rename mlir/lib/Dialect/EmitC/Transforms/{AddReflectionMap.cpp => MLGOAddReflectionMap.cpp} (71%)
 rename mlir/test/Dialect/EmitC/{add-reflection-map.mlir => mlgo-add-reflection-map.mlir} (52%)
 create mode 100644 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
index 56596110a1e90..4f46d63774625 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.h
@@ -16,7 +16,7 @@ namespace emitc {
 
 #define GEN_PASS_DECL_FORMEXPRESSIONSPASS
 #define GEN_PASS_DECL_WRAPFUNCINCLASSPASS
-#define GEN_PASS_DECL_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DECL_MLGOADDREFLECTIONMAPPASS
 
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 6da57c07a8047..3d17f53032a7a 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -60,7 +60,7 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
   ];
 }
 
-def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
+def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
     This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
@@ -68,9 +68,11 @@ def AddReflectionMapPass : Pass<"add-reflection-map", "ModuleOp"> {
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
     `field-attr-name` option) that is an array containing a single string
-    attribute.
-
-
+    attribute. If a field possesses both the attribute specified by
+    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
+    the `field-attr-name` attribute takes precedence and the field will be
+    included in the reflection map.
+    
     Example:
 
     ```mlir
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 8f1784f61f540..6425614e1fbcd 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -42,9 +42,9 @@ void populateWrapFuncInClass(
 // The AddReflectionMap pass.
 //===----------------------------------------------------------------------===//
 
-void populateAddReflectionMapPatterns(RewritePatternSet &patterns,
-                                      StringRef fieldAttrName,
-                                      ArrayRef<std::string> excludedFieldAttrs);
+void populateMLGOAddReflectionMapPatterns(
+    RewritePatternSet &patterns, StringRef fieldAttrName,
+    ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
index dd8f014dc4737..ede2cb5e208f0 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/EmitC/Transforms/CMakeLists.txt
@@ -3,7 +3,7 @@ add_mlir_dialect_library(MLIREmitCTransforms
   FormExpressions.cpp
   TypeConversions.cpp
   WrapFuncInClass.cpp
-  AddReflectionMap.cpp
+  MLGOAddReflectionMap.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/EmitC/Transforms
diff --git a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
similarity index 71%
rename from mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
rename to mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index d444f3abbf4b1..9f506b10c722a 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/AddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -1,7 +1,7 @@
-//===- AddReflectionMap.cpp - Add a reflection map to a class --------===//
+//===----------------------------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt license information.
+// See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
@@ -22,7 +22,7 @@ using namespace emitc;
 
 namespace mlir {
 namespace emitc {
-#define GEN_PASS_DEF_ADDREFLECTIONMAPPASS
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
 #include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
 
 namespace {
@@ -35,15 +35,15 @@ IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
                            /*is_standard_include=*/builder.getUnitAttr());
 }
 
-class AddReflectionMapPass
-    : public impl::AddReflectionMapPassBase<AddReflectionMapPass> {
-  using AddReflectionMapPassBase::AddReflectionMapPassBase;
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
   void runOnOperation() override {
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateAddReflectionMapPatterns(patterns, fieldAttrName,
-                                     excludedFieldAttrs);
+    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+                                         excludedFieldAttrs);
 
     walkAndApplyPatterns(moduleOp, std::move(patterns));
     bool hasMapHdr = false;
@@ -56,8 +56,8 @@ class AddReflectionMapPass
       if (includeOp.getIsStandardInclude()) {
         auto include = includeOp.getInclude();
 
-        hasMapHdr = include == mapLibraryHeader;
-        hasStringHdr = include == stringLibraryHeader;
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
       }
 
       if (hasMapHdr && hasStringHdr)
@@ -77,10 +77,10 @@ class AddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
-class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  AddReflectionMapClass(MLIRContext *context, StringRef attrName,
-                        llvm::ArrayRef<std::string> excludedFieldAttrs)
+  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+                            llvm::ArrayRef<std::string> excludedFieldAttrs)
       : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
@@ -93,33 +93,35 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
         context, "const std::map<std::string, char*>");
 
     // Collect all field names
-    std::vector<std::pair<std::string, std::string>> fieldNames;
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (Attribute attr = fieldOp->getAttrDictionary().get(fieldAttrName)) {
-        if (ArrayAttr arrayAttr = dyn_cast<mlir::ArrayAttr>(attr)) {
-          StringAttr stringAttr = cast<mlir::StringAttr>(arrayAttr[0]);
-          fieldNames.emplace_back(stringAttr.getValue().str(),
-                                  fieldOp.getName().str());
-          return;
-        }
-      }
-
-      bool shouldIgnore = false;
-      for (const std::string &ignoreAttr : excludedFieldAttrs) {
-        if (fieldOp->hasAttr(ignoreAttr)) {
-          shouldIgnore = true;
-          break;
+      if (auto arrayAttr =
+              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
+        if (!arrayAttr.empty()) {
+          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
+            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
+            return;
+          }
         }
       }
 
+      bool shouldIgnore =
+          llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
+            return fieldOp->hasAttr(ignoreAttr);
+          });
       if (shouldIgnore)
         return;
 
       fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "'"
+                          << fieldAttrName << "' "
                           << "with an array containing a string attribute";
+      hasError = true;
     });
 
+    if (hasError || fieldNames.empty())
+      return failure();
+
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
@@ -138,6 +140,14 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
       return failure();
     }
 
+    // To generate the following C++ code
+    // const std::map<std::string, char*> reflectionMap = {
+    //  { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+    //  { "some_feature", reinterpret_cast<char*>(&fieldName1) },
+    //  ...
+    // };
+    // This can be used to retrieve a pointer to the field's contents given the
+    // attribute string identifying the field
     FieldOp reflectionMapField = FieldOp::create(
         rewriter, classOp.getLoc(), rewriter.getStringAttr("reflectionMap"),
         TypeAttr::get(mapType),
@@ -180,9 +190,9 @@ class AddReflectionMapClass : public OpRewritePattern<ClassOp> {
   llvm::SmallVector<std::string> excludedFieldAttrs;
 };
 
-void mlir::emitc::populateAddReflectionMapPatterns(
+void mlir::emitc::populateMLGOAddReflectionMapPatterns(
     RewritePatternSet &patterns, StringRef fieldAttrName,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<AddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                      excludedFieldAttrs);
-}
\ No newline at end of file
+  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
+                                          excludedFieldAttrs);
+}
diff --git a/mlir/test/Dialect/EmitC/add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
similarity index 52%
rename from mlir/test/Dialect/EmitC/add-reflection-map.mlir
rename to mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index a1ec0356e19f4..2bea6cfaf2677 100644
--- a/mlir/test/Dialect/EmitC/add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,7 +1,7 @@
-// RUN: mlir-opt -split-input-file --add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
 
-
-// Tests that a reflection map is created for fields with a certain attribute.
+/// Tests that a reflection map is created for fields with a certain attribute.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -29,10 +29,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:  }
 
 // -----
-// Test that a reflection map is created for fields with a certain named attribute
-// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClass {
+/// Test that a reflection map is created for fields with a certain named attribute
+/// but not ones with an attribute present in the ignore-attributes option.
+
+emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -41,7 +42,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @actionClassExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -55,4 +56,73 @@ emitc.class @actionClass {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if fields don't have any attributes
+
+emitc.class @actionClassNoAttrs {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+
+emitc.class @actionClassNoFields {
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if the ClassOp doesn't have
+/// a FunctionOp named operator()
+
+// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
+emitc.class @actionClassNoOperator {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+}
+
+// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass returns with a match failure if a FieldOp has the specified
+/// dictionary attribute with an array containing a type other than string
+
+emitc.class @actionClassNonStringAttr {
+  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
+// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
\ No newline at end of file
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
new file mode 100644
index 0000000000000..eb8a8e58c8b77
--- /dev/null
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -0,0 +1,57 @@
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+
+// Test that a reflection map and lookup function are generated in the class.
+
+emitc.class @actionClass {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };
+
+// -----
+
+// Test that fields with excluded attributes are ignored.
+
+emitc.class @actionClassExcluded {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
+  emitc.func @"operator()"() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       #include <map>
+// CHECK-NEXT:  #include <string>
+// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    float fieldName1[1];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
+// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    }
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 0b94b71ff7e2b25032b3203b9a58d9fee2780ae4 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Mon, 6 Jul 2026 12:50:33 -0700
Subject: [PATCH 06/22] update MLGOAddReflectionMap to use notifyMatchFailure,
 update example to be more minimal and clear about the transformation, use
 range based for loop in reflection map construction

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 24 ++++++------------
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 25 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        |  5 +---
 3 files changed, 24 insertions(+), 30 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 3d17f53032a7a..29b053f3469c7 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -77,28 +77,20 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
 
     ```mlir
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.func @execute() { ... }
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      emitc.func @operator()() { ... }
     }
 
     Becomes:
 
     emitc.class @MyClass {
-      emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-      emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
-      emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-        #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">
-      emitc.func @getBufferForName(%arg0: !emitc.opaque<"std::string">)
-          -> !emitc.ptr<!emitc.opaque<"char">> {
-        %0 = get_field @reflectionMap
-          : !emitc.opaque<"const std::map<std::string, char*>">
-        %1 = member_call_opaque %0 "at"(%arg0)
-          : !emitc.opaque<"const std::map<std::string, char*>">,
-            (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
-        return %1 : !emitc.ptr<!emitc.opaque<"char">>
+      emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
+      // reflectionMap maps the string "feature0" to the address of @field0
+      emitc.field @reflectionMap : const std::map<string, char*> = {{"feature0", &field0}}
+      emitc.func @getBufferForName(%name: string) -> char* {
+        return reflectionMap.at(%name)
       }
-      emitc.func @execute() { ... }
+      emitc.func @operator()() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 9f506b10c722a..05a850c0e7551 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -113,9 +113,11 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
       if (shouldIgnore)
         return;
 
-      fieldOp.emitError() << "FieldOp must have a dictionary attribute named '"
-                          << fieldAttrName << "' "
-                          << "with an array containing a string attribute";
+      (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
+        diag << "FieldOp must have a dictionary attribute named '"
+             << fieldAttrName << "' "
+             << "with an array containing a string attribute";
+      });
       hasError = true;
     });
 
@@ -124,20 +126,23 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
-    for (size_t i = 0, numFields = fieldNames.size(); i < numFields; ++i) {
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+
+      first = false;
       reflectionMapContents += llvm::formatv(
-          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }{2}", fieldNames[i].first,
-          fieldNames[i].second, (i < numFields - 1) ? ", " : "");
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
     }
     reflectionMapContents += " }";
 
     if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
       rewriter.setInsertionPoint(executeFunc);
     else {
-      classOp.emitError()
-          << "ClassOp must contain a function named 'operator()' "
-             "to add reflection map";
-      return failure();
+      return rewriter.notifyMatchFailure(
+          classOp, "ClassOp must contain a function named 'operator()' "
+                   "to add reflection map");
     }
 
     // To generate the following C++ code
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2bea6cfaf2677..1f8647910ada9 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" -verify-diagnostics %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -63,7 +63,6 @@ emitc.class @actionClassExcluded {
 /// Test that the pass leaves IR unchanged if fields don't have any attributes
 
 emitc.class @actionClassNoAttrs {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -98,7 +97,6 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-// expected-error @below {{ClassOp must contain a function named 'operator()' to add reflection map}}
 emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
@@ -113,7 +111,6 @@ emitc.class @actionClassNoOperator {
 /// dictionary attribute with an array containing a type other than string
 
 emitc.class @actionClassNonStringAttr {
-  // expected-error @below {{FieldOp must have a dictionary attribute named 'emitc.field_ref' with an array containing a string attribute}}
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return

>From d5e8301536841a90c281fd46a84414bb2507bd42 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 12:59:46 -0700
Subject: [PATCH 07/22] Update tests with formatting and wording changes,
 update pass description, only add headers if there was a match

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  4 +--
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 16 ++++++++-
 .../EmitC/mlgo-add-reflection-map.mlir        |  8 +++--
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 34 +++++++++++++++----
 4 files changed, 51 insertions(+), 11 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 29b053f3469c7..1e245e00c9b2d 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -78,7 +78,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
     ```mlir
     emitc.class @MyClass {
       emitc.field @field0 : !emitc.array<1xf32> {emitc.field_ref = ["feature0"]}
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
 
     Becomes:
@@ -90,7 +90,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
       emitc.func @getBufferForName(%name: string) -> char* {
         return reflectionMap.at(%name)
       }
-      emitc.func @operator()() { ... }
+      emitc.func @execute() { ... }
     }
     ```
   }];
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 05a850c0e7551..666d9bc01dfdf 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -29,6 +29,15 @@ namespace {
 constexpr const char *mapLibraryHeader = "map";
 constexpr const char *stringLibraryHeader = "string";
 
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
 IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
   StringAttr includeAttr = builder.getStringAttr(headerName);
   return IncludeOp::create(builder, module.getLoc(), includeAttr,
@@ -45,7 +54,12 @@ class MLGOAddReflectionMapPass
     populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
                                          excludedFieldAttrs);
 
-    walkAndApplyPatterns(moduleOp, std::move(patterns));
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    if (!listener.patternApplied)
+      return;
+
     bool hasMapHdr = false;
     bool hasStringHdr = false;
     for (auto &op : *moduleOp.getBody()) {
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 1f8647910ada9..030de796b308c 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -60,7 +60,7 @@ emitc.class @actionClassExcluded {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if fields don't have any attributes
+/// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
 
 emitc.class @actionClassNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
@@ -69,6 +69,7 @@ emitc.class @actionClassNoAttrs {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @"operator()"() {
@@ -78,7 +79,7 @@ emitc.class @actionClassNoAttrs {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields
+/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
 emitc.class @actionClassNoFields {
   emitc.func @"operator()"() {
@@ -86,6 +87,7 @@ emitc.class @actionClassNoFields {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoFields {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
@@ -101,6 +103,7 @@ emitc.class @actionClassNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:  }
@@ -117,6 +120,7 @@ emitc.class @actionClassNonStringAttr {
   }
 }
 
+// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @actionClassNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @"operator()"() {
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index eb8a8e58c8b77..e37b09bc6d611 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,6 +1,6 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
-// Test that a reflection map and lookup function are generated in the class.
+/// Test that a reflection map and lookup function are generated in the class.
 
 emitc.class @actionClass {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -17,10 +17,11 @@ emitc.class @actionClass {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
-// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) }, { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
-// CHECK-NEXT:    char* getBufferForName(std::string [[VAL_1:v[0-9]+]]) {
-// CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
-// CHECK-NEXT:      return [[VAL_2]];
+// CHECK-NEXT:    const std::map<std::string, char*> reflectionMap = { { "another_feature", reinterpret_cast<char*>(&fieldName0) },
+// CHECK-SAME:                                                         { "some_feature", reinterpret_cast<char*>(&fieldName1) } };
+// CHECK-NEXT:    char* getBufferForName(std::string [[ARG:v[0-9]+]]) {
+// CHECK-NEXT:      char* [[VAR:v[0-9]+]] = reflectionMap.at([[ARG]]);
+// CHECK-NEXT:      return [[VAR]];
 // CHECK-NEXT:    }
 // CHECK-NEXT:    void operator()() {
 // CHECK-NEXT:      return;
@@ -29,7 +30,7 @@ emitc.class @actionClass {
 
 // -----
 
-// Test that fields with excluded attributes are ignored.
+/// Test that fields with excluded attributes are ignored.
 
 emitc.class @actionClassExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -55,3 +56,24 @@ emitc.class @actionClassExcluded {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
+
+// -----
+
+/// Test that translation doesn't add headers if the class does not match the pass.
+
+emitc.class @actionClassNoAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32>
+  emitc.func @"operator()"() {
+    return
+  }
+}
+
+// CHECK-NOT:   #include <map>
+// CHECK-NOT:   #include <string>
+// CHECK:       class actionClassNoAttrs {
+// CHECK-NEXT:   public:
+// CHECK-NEXT:    float fieldName0[1];
+// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:      return;
+// CHECK-NEXT:    }
+// CHECK-NEXT:  };

>From 72d872a2e29df289955aedc5d85733a5f2d50b49 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 13:20:12 -0700
Subject: [PATCH 08/22] Update naming of classes in tests to be generic

---
 .../EmitC/mlgo-add-reflection-map.mlir        | 24 +++++++++----------
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 12 +++++-----
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 030de796b308c..bd6733f982fe0 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -3,7 +3,7 @@
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
-emitc.class @actionClass {
+emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -12,7 +12,7 @@ emitc.class @actionClass {
   }
 }
 
-// CHECK:       emitc.class @actionClass {
+// CHECK:       emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -33,7 +33,7 @@ emitc.class @actionClass {
 /// Test that a reflection map is created for fields with a certain named attribute
 /// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @actionClassExcluded {
+emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -42,7 +42,7 @@ emitc.class @actionClassExcluded {
   }
 }
 
-// CHECK:       emitc.class @actionClassExcluded {
+// CHECK:       emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -62,7 +62,7 @@ emitc.class @actionClassExcluded {
 
 /// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
 
-emitc.class @actionClassNoAttrs {
+emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -70,7 +70,7 @@ emitc.class @actionClassNoAttrs {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoAttrs {
+// CHECK-LABEL: emitc.class @fooNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
@@ -81,14 +81,14 @@ emitc.class @actionClassNoAttrs {
 
 /// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
-emitc.class @actionClassNoFields {
+emitc.class @fooNoFields {
   emitc.func @"operator()"() {
     return
   }
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoFields {
+// CHECK-LABEL: emitc.class @fooNoFields {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
@@ -99,12 +99,12 @@ emitc.class @actionClassNoFields {
 /// Test that the pass returns with a match failure if the ClassOp doesn't have
 /// a FunctionOp named operator()
 
-emitc.class @actionClassNoOperator {
+emitc.class @fooNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNoOperator {
+// CHECK-LABEL: emitc.class @fooNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:  }
 
@@ -113,7 +113,7 @@ emitc.class @actionClassNoOperator {
 /// Test that the pass returns with a match failure if a FieldOp has the specified
 /// dictionary attribute with an array containing a type other than string
 
-emitc.class @actionClassNonStringAttr {
+emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @"operator()"() {
     return
@@ -121,7 +121,7 @@ emitc.class @actionClassNonStringAttr {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @actionClassNonStringAttr {
+// CHECK-LABEL: emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index e37b09bc6d611..d0eab613672a7 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -2,7 +2,7 @@
 
 /// Test that a reflection map and lookup function are generated in the class.
 
-emitc.class @actionClass {
+emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -13,7 +13,7 @@ emitc.class @actionClass {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class actionClass {
+// CHECK-NEXT:  class foo {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
@@ -32,7 +32,7 @@ emitc.class @actionClass {
 
 /// Test that fields with excluded attributes are ignored.
 
-emitc.class @actionClassExcluded {
+emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @"operator()"() {
@@ -43,7 +43,7 @@ emitc.class @actionClassExcluded {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class actionClassExcluded {
+// CHECK-NEXT:  class fooExcluded {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
@@ -61,7 +61,7 @@ emitc.class @actionClassExcluded {
 
 /// Test that translation doesn't add headers if the class does not match the pass.
 
-emitc.class @actionClassNoAttrs {
+emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @"operator()"() {
     return
@@ -70,7 +70,7 @@ emitc.class @actionClassNoAttrs {
 
 // CHECK-NOT:   #include <map>
 // CHECK-NOT:   #include <string>
-// CHECK:       class actionClassNoAttrs {
+// CHECK:       class fooNoAttrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    void operator()() {

>From d3c18f1c82a794408eea80cf63e8ca4fbcdd2fae Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 13:29:53 -0700
Subject: [PATCH 09/22] Set a filecheck pattern variable for \22 escape
 character to make test more readable

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index bd6733f982fe0..2220d419b677c 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s -DQUOTE="\[[QUOTE]]"
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -16,7 +16,7 @@ emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) }, { \22some_feature\22, reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[QUOTE]]some_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -46,7 +46,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { \22another_feature\22, reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>

>From 076af93ded9db57ab7e0f3663ae7883ef6c6f05b Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:20:05 -0700
Subject: [PATCH 10/22] Relax the restriction where pass only matched classes
 with operator()() present

---
 mlir/include/mlir/Dialect/EmitC/IR/EmitC.td   |  2 +-
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 13 ++++++------
 .../EmitC/mlgo-add-reflection-map.mlir        | 20 ++++++++++++-------
 3 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index d42c74e963769..0636232906f1b 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -1801,7 +1801,7 @@ def EmitC_ClassOp
 
   let extraClassDeclaration = [{
     // Returns the body block containing class members and methods.
-    Block &getBlock();
+    Block &getBlock() { return getBody().front(); }
   }];
 
   let assemblyFormat =
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 666d9bc01dfdf..12b173074e5bb 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -151,13 +151,12 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
-    if (FuncOp executeFunc = classOp.lookupSymbol<FuncOp>("operator()"))
-      rewriter.setInsertionPoint(executeFunc);
-    else {
-      return rewriter.notifyMatchFailure(
-          classOp, "ClassOp must contain a function named 'operator()' "
-                   "to add reflection map");
-    }
+    auto funcs = classOp.getBlock().getOps<FuncOp>();
+    auto it = funcs.begin();
+    if (it != funcs.end())
+      rewriter.setInsertionPoint(*it);
+    else
+      rewriter.setInsertionPointToEnd(&classOp.getBlock());
 
     // To generate the following C++ code
     // const std::map<std::string, char*> reflectionMap = {
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 2220d419b677c..3374838aeac34 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,5 +1,5 @@
 // RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
-// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s -DQUOTE="\[[QUOTE]]"
+// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
 
 /// Tests that a reflection map is created for fields with a certain attribute.
 
@@ -16,7 +16,7 @@ emitc.class @foo {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[QUOTE]]some_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -46,7 +46,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-// CHECK-SAME:    #emitc.opaque<"{ { [[QUOTE]]another_feature[[QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">  
 // CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
 // CHECK-NEXT:      %[[MAP1:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
@@ -96,16 +96,22 @@ emitc.class @fooNoFields {
 
 // -----
 
-/// Test that the pass returns with a match failure if the ClassOp doesn't have
-/// a FunctionOp named operator()
+/// Test that a reflection map is still created in the case that there are no
+/// functions in the class
 
 emitc.class @fooNoOperator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
-// CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoOperator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP0:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
 // CHECK-NEXT:  }
 
 // -----
@@ -126,4 +132,4 @@ emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.func @"operator()"() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
-// CHECK-NEXT:  }
\ No newline at end of file
+// CHECK-NEXT:  }

>From ef2ab7ac726afd849958ce3f97fdb48a4e3f3f0f Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:23:11 -0700
Subject: [PATCH 11/22] Simplify test

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 3374838aeac34..5e7780dc23f78 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -7,7 +7,6 @@ emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
   emitc.func @"operator()"() {
-    %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
 }
@@ -23,7 +22,6 @@ emitc.class @foo {
 // CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
 // CHECK-NEXT:    emitc.func @"operator()"() {
-// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }

>From aeeb53b5ff87882f41a42a0f4b4eaca9db7d6fbe Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 16:25:12 -0700
Subject: [PATCH 12/22] Update function name in tests to be general

---
 .../EmitC/mlgo-add-reflection-map.mlir        | 20 +++++++++----------
 .../Target/Cpp/mlgo-add-reflection-map.mlir   | 12 +++++------
 2 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 5e7780dc23f78..d18c1583cfd38 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -6,7 +6,7 @@
 emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -21,7 +21,7 @@ emitc.class @foo {
 // CHECK-NEXT:      %[[VAL0:.*]] = member_call_opaque %[[MAP0]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:      return %[[VAL0]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -34,7 +34,7 @@ emitc.class @foo {
 emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -50,7 +50,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:      %[[VAL1:.*]] = member_call_opaque %[[MAP1]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:      return %[[VAL1]] : !emitc.ptr<!emitc.opaque<"char">>
 // CHECK-NEXT:    }
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
@@ -62,7 +62,7 @@ emitc.class @fooExcluded {
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -70,7 +70,7 @@ emitc.class @fooNoAttrs {
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoAttrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -80,14 +80,14 @@ emitc.class @fooNoAttrs {
 /// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
 
 emitc.class @fooNoFields {
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
 
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNoFields {
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
@@ -119,7 +119,7 @@ emitc.class @fooNoOperator {
 
 emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -127,7 +127,7 @@ emitc.class @fooNonStringAttr {
 // CHECK-NOT:     emitc.include
 // CHECK-LABEL: emitc.class @fooNonStringAttr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
-// CHECK-NEXT:    emitc.func @"operator()"() {
+// CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index d0eab613672a7..8aca1583f6313 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -5,7 +5,7 @@
 emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.field_ref = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -23,7 +23,7 @@ emitc.class @foo {
 // CHECK-NEXT:      char* [[VAR:v[0-9]+]] = reflectionMap.at([[ARG]]);
 // CHECK-NEXT:      return [[VAR]];
 // CHECK-NEXT:    }
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
@@ -35,7 +35,7 @@ emitc.class @foo {
 emitc.class @fooExcluded {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     %0 = get_field @fieldName0 : !emitc.array<1xf32>
     return
   }
@@ -52,7 +52,7 @@ emitc.class @fooExcluded {
 // CHECK-NEXT:      char* [[VAL_2:v[0-9]+]] = reflectionMap.at([[VAL_1]]);
 // CHECK-NEXT:      return [[VAL_2]];
 // CHECK-NEXT:    }
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };
@@ -63,7 +63,7 @@ emitc.class @fooExcluded {
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
-  emitc.func @"operator()"() {
+  emitc.func @bar() {
     return
   }
 }
@@ -73,7 +73,7 @@ emitc.class @fooNoAttrs {
 // CHECK:       class fooNoAttrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
-// CHECK-NEXT:    void operator()() {
+// CHECK-NEXT:    void bar() {
 // CHECK-NEXT:      return;
 // CHECK-NEXT:    }
 // CHECK-NEXT:  };

>From 835b1bebfe57f56923b339f64b7cb84e0c927066 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 14 Jul 2026 18:26:20 -0700
Subject: [PATCH 13/22] Switched pass input for field attr name to list to
 match the exclude list.

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   |  5 +--
 .../Dialect/EmitC/Transforms/Transforms.h     |  2 +-
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 41 +++++++++++--------
 .../EmitC/mlgo-add-reflection-map.mlir        | 29 ++++++++++++-
 .../Target/Cpp/mlgo-add-reflection-map.mlir   |  2 +-
 5 files changed, 56 insertions(+), 23 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 1e245e00c9b2d..328852a369e28 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -96,9 +96,8 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   }];
   let dependentDialects = ["emitc::EmitCDialect"];
   let options = [
-    Option<"fieldAttrName", "field-attr-name", "std::string",
-      /*default=*/"",
-      "Attribute key used to extract field names from a field's "
+    ListOption<"includedFieldAttrs", "included-field-attrs", "std::string",
+      "Attribute keys used to extract field names from a field's "
       "attribute dictionary">,
     ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
       "Attribute keys that, if present on a field, exclude it "
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index 6425614e1fbcd..b3899a2b070d9 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -43,7 +43,7 @@ void populateWrapFuncInClass(
 //===----------------------------------------------------------------------===//
 
 void populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, StringRef fieldAttrName,
+    RewritePatternSet &patterns, ArrayRef<std::string> includedFieldAttrs,
     ArrayRef<std::string> excludedFieldAttrs);
 
 } // namespace emitc
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 12b173074e5bb..afd1bf5b9b56b 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -15,6 +15,7 @@
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Support/FormatVariadic.h"
 
 using namespace mlir;
@@ -51,7 +52,7 @@ class MLGOAddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateMLGOAddReflectionMapPatterns(patterns, fieldAttrName,
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs,
                                          excludedFieldAttrs);
 
     PatternMatchListener listener;
@@ -93,9 +94,11 @@ class MLGOAddReflectionMapPass
 
 class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
-  MLGOAddReflectionMapClass(MLIRContext *context, StringRef attrName,
+  MLGOAddReflectionMapClass(MLIRContext *context,
+                            llvm::ArrayRef<std::string> includedFieldAttrs,
                             llvm::ArrayRef<std::string> excludedFieldAttrs)
-      : OpRewritePattern<ClassOp>(context), fieldAttrName(attrName),
+      : OpRewritePattern<ClassOp>(context),
+        includedFieldAttrs(includedFieldAttrs),
         excludedFieldAttrs(excludedFieldAttrs.begin(),
                            excludedFieldAttrs.end()) {}
 
@@ -110,13 +113,16 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     std::vector<std::pair<StringRef, StringRef>> fieldNames;
     bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
-      if (auto arrayAttr =
-              dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(fieldAttrName))) {
-        if (!arrayAttr.empty()) {
-          if (auto stringAttr = dyn_cast<StringAttr>(arrayAttr[0])) {
-            fieldNames.emplace_back(stringAttr.getValue(), fieldOp.getName());
-            return;
-          }
+      for (const auto &attr : includedFieldAttrs) {
+        auto arrayAttr = dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(attr));
+
+        if (!arrayAttr)
+          continue;
+
+        if (!arrayAttr.empty() && isa<StringAttr>(arrayAttr[0])) {
+          fieldNames.emplace_back(cast<StringAttr>(arrayAttr[0]).getValue(),
+                                  fieldOp.getName());
+          return;
         }
       }
 
@@ -129,7 +135,7 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 
       (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
         diag << "FieldOp must have a dictionary attribute named '"
-             << fieldAttrName << "' "
+             << includedFieldAttrs << "' "
              << "with an array containing a string attribute";
       });
       hasError = true;
@@ -199,9 +205,10 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
   }
 
 private:
-  /// The name of the attribute on FieldOps that contains the field name
-  /// metadata for the reflection map.
-  StringRef fieldAttrName;
+  /// The names of the attributes on FieldOps that contain the field name
+  /// metadata for the reflection map. The pass matches the first attribute
+  /// present in the order they are specified in this list.
+  llvm::SmallVector<std::string> includedFieldAttrs;
 
   /// Attributes that, if present on a field, exclude it from the
   /// reflection map.
@@ -209,8 +216,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 };
 
 void mlir::emitc::populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, StringRef fieldAttrName,
+    RewritePatternSet &patterns, llvm::ArrayRef<std::string> includedFieldAttrs,
     llvm::ArrayRef<std::string> excludedFieldAttrs) {
-  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(), fieldAttrName,
-                                          excludedFieldAttrs);
+  patterns.add<MLGOAddReflectionMapClass>(
+      patterns.getContext(), includedFieldAttrs, excludedFieldAttrs);
 }
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index d18c1583cfd38..ff5f44a1ac959 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref \
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
 // RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
 
 /// Tests that a reflection map is created for fields with a certain attribute.
@@ -131,3 +131,30 @@ emitc.class @fooNonStringAttr {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
 // CHECK-NEXT:  }
+
+// -----
+
+/// Test that the pass matches one of the multiple included attributes.
+
+emitc.class @fooMultipleAttrs {
+  emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
+  emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+  emitc.func @bar() {
+    return
+  }
+}
+
+// CHECK-LABEL: emitc.class @fooMultipleAttrs {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
+// CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
+// CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> =
+// CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) }, { [[$QUOTE]]some_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName1) } }">
+// CHECK-NEXT:    emitc.func @getBufferForName(%{{.*}}: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+// CHECK-NEXT:      %[[MAP:.*]] = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+// CHECK-NEXT:      %[[VAL:.*]] = member_call_opaque %[[MAP]] "at"({{.*}}) : !emitc.opaque<"const std::map<std::string, char*>">, (!emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:      return %[[VAL]] : !emitc.ptr<!emitc.opaque<"char">>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index 8aca1583f6313..41329771562da 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="field-attr-name=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
 /// Test that a reflection map and lookup function are generated in the class.
 

>From 35c5a45e4714fe5bba82a1cdac20a62078acd414 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:09:54 -0700
Subject: [PATCH 14/22] Add comments

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp        | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index afd1bf5b9b56b..881da3f4aee87 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -58,9 +58,12 @@ class MLGOAddReflectionMapPass
     PatternMatchListener listener;
     walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
 
+    // If nothing was matched, no reflection maps were added, removing the need
+    // to add include headers for map and string
     if (!listener.patternApplied)
       return;
 
+    // Check if the map and/or string headers are already present
     bool hasMapHdr = false;
     bool hasStringHdr = false;
     for (auto &op : *moduleOp.getBody()) {
@@ -109,7 +112,9 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
         context, "const std::map<std::string, char*>");
 
-    // Collect all field names
+    // Collect the names of all FieldOps that have one of the attributes in
+    // includedFieldAttrs to use the first element of the array attribute
+    // as the reflection map key
     std::vector<std::pair<StringRef, StringRef>> fieldNames;
     bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
@@ -126,6 +131,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
         }
       }
 
+      // If one of the attributes in excludedFieldAttrs is present,
+      // don't add this FieldOp
       bool shouldIgnore =
           llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
             return fieldOp->hasAttr(ignoreAttr);
@@ -144,6 +151,7 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     if (hasError || fieldNames.empty())
       return failure();
 
+    // Create reflection map contents
     std::string reflectionMapContents;
     reflectionMapContents += "{ ";
     bool first = true;
@@ -157,6 +165,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     }
     reflectionMapContents += " }";
 
+    // Set insertion point before the first function or after all fields
+    // if there are no functions within the class
     auto funcs = classOp.getBlock().getOps<FuncOp>();
     auto it = funcs.begin();
     if (it != funcs.end())

>From e8af11de937a1571b252c3d85b9bf8894eaa9209 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:28:08 -0700
Subject: [PATCH 15/22] Update test comments

---
 mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir | 8 ++++----
 mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir    | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index ff5f44a1ac959..f46bf4528048f 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -58,7 +58,7 @@ emitc.class @fooExcluded {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if fields don't have any attributes (match failure)
+/// Test that the pass bails out and leaves IR unchanged if fields don't have any attributes
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
@@ -77,7 +77,7 @@ emitc.class @fooNoAttrs {
 
 // -----
 
-/// Test that the pass leaves IR unchanged if the ClassOp doesn't have any fields (match failure)
+/// Test that the pass bails out and leaves IR unchanged if the ClassOp doesn't have any fields
 
 emitc.class @fooNoFields {
   emitc.func @bar() {
@@ -114,8 +114,8 @@ emitc.class @fooNoOperator {
 
 // -----
 
-/// Test that the pass returns with a match failure if a FieldOp has the specified
-/// dictionary attribute with an array containing a type other than string
+/// Test that the pass bails out if a FieldOp has the specified dictionary attribute
+/// with an array containing a type other than string
 
 emitc.class @fooNonStringAttr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index 41329771562da..e4c851b2889ff 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -59,7 +59,7 @@ emitc.class @fooExcluded {
 
 // -----
 
-/// Test that translation doesn't add headers if the class does not match the pass.
+/// Test that headers aren't added if the pass bails out.
 
 emitc.class @fooNoAttrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>

>From 1e7499d5aad7c38d328740c96b6b530129ca2823 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:37:17 -0700
Subject: [PATCH 16/22] Document OpRewritePattern

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 881da3f4aee87..23ade8ded7568 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -95,6 +95,36 @@ class MLGOAddReflectionMapPass
 } // namespace emitc
 } // namespace mlir
 
+/// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
+/// and a lookup method.
+///
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes (e.g.,
+/// `emitc.field_ref`). Fields containing `excludedFieldAttrs` are skipped. All
+/// other fields must match one of the inclusion/exclusion filters, otherwise the
+/// pattern fails.
+///
+/// Before:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+///   emitc.func @bar() { return }
+/// }
+/// ```
+///
+/// After:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
+///   emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
+///     #emitc.opaque<"{ { \"another_feature\", reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
+///     %ptr = member_call_opaque %map "at"(%name) : ...
+///     return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   }
+///   emitc.func @bar() { return }
+/// }
+/// ```
 class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
   MLGOAddReflectionMapClass(MLIRContext *context,

>From 654a2088ef8503ac1df77eb98381c688dac7818e Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:39:59 -0700
Subject: [PATCH 17/22] Formatting

---
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 29 ++++++++++---------
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 23ade8ded7568..adc6bcbe7f8be 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -98,29 +98,32 @@ class MLGOAddReflectionMapPass
 /// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
 /// and a lookup method.
 ///
-/// Fields to be mapped are identified via `includedFieldAttrs` attributes (e.g.,
-/// `emitc.field_ref`). Fields containing `excludedFieldAttrs` are skipped. All
-/// other fields must match one of the inclusion/exclusion filters, otherwise the
-/// pattern fails.
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes
+/// (e.g., `emitc.field_ref`). Fields containing `excludedFieldAttrs` are
+/// skipped. All other fields must match one of the inclusion/exclusion filters,
+/// otherwise the pattern fails.
 ///
 /// Before:
 /// ```mlir
 /// emitc.class @foo {
-///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-///   emitc.func @bar() { return }
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.func @bar() { return }
 /// }
 /// ```
 ///
 /// After:
 /// ```mlir
 /// emitc.class @foo {
-///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
-///   emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
-///     #emitc.opaque<"{ { \"another_feature\", reinterpret_cast<char*>(&fieldName0) } }">
-///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) -> !emitc.ptr<!emitc.opaque<"char">> {
-///     %map = get_field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>">
-///     %ptr = member_call_opaque %map "at"(%name) : ...
-///     return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.field @reflectionMap : !emitc.opaque<"const
+///   std::map<std::string, char*>"> =
+///     #emitc.opaque<"{ { \"another_feature\",
+///     reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) ->
+///   !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const
+///     std::map<std::string, char*>"> %ptr = member_call_opaque %map
+///     "at"(%name) : ... return %ptr : !emitc.ptr<!emitc.opaque<"char">>
 ///   }
 ///   emitc.func @bar() { return }
 /// }

>From 3c32f1e81cc3ea002d79d2a02c8c5a3e20f6f735 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 15 Jul 2026 12:43:52 -0700
Subject: [PATCH 18/22] Update pass documentation

---
 mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 328852a369e28..5c0dfe7bf69c1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -67,10 +67,10 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
     EmitC classes, enabling runtime lookup of class fields by name. 
     This requires that the class has fields with attributes.
     Each `emitc.field` is expected to have an attribute (configured by the
-    `field-attr-name` option) that is an array containing a single string
-    attribute. If a field possesses both the attribute specified by
-    `field-attr-name` and an attribute specified in `excluded-field-attrs`,
-    the `field-attr-name` attribute takes precedence and the field will be
+    `included-field-attrs` option) that is an array containing a single string
+    attribute. If a field possesses both an attribute specified by
+    `included-field-attrs` and an attribute specified in `excluded-field-attrs`,
+    the `included-field-attrs` attribute takes precedence and the field will be
     included in the reflection map.
     
     Example:

>From e63ba3ad0ee3b589f979e6fc06a5edb8e5f6e91c Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Thu, 16 Jul 2026 12:23:18 -0700
Subject: [PATCH 19/22] Update tests to follow naming guidelines

---
 .../EmitC/mlgo-add-reflection-map.mlir        | 24 +++++++++----------
 .../Target/Cpp/mlgo-add-reflection-map.mlir   |  8 +++----
 2 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index f46bf4528048f..1d5ecc61b118b 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -31,7 +31,7 @@ emitc.class @foo {
 /// Test that a reflection map is created for fields with a certain named attribute
 /// but not ones with an attribute present in the ignore-attributes option.
 
-emitc.class @fooExcluded {
+emitc.class @foo_excluded_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @bar() {
@@ -40,7 +40,7 @@ emitc.class @fooExcluded {
   }
 }
 
-// CHECK:       emitc.class @fooExcluded {
+// CHECK:       emitc.class @foo_excluded_attrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -60,7 +60,7 @@ emitc.class @fooExcluded {
 
 /// Test that the pass bails out and leaves IR unchanged if fields don't have any attributes
 
-emitc.class @fooNoAttrs {
+emitc.class @negative_foo_no_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @bar() {
     return
@@ -68,7 +68,7 @@ emitc.class @fooNoAttrs {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @fooNoAttrs {
+// CHECK-LABEL: emitc.class @negative_foo_no_attrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32>
 // CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
@@ -79,14 +79,14 @@ emitc.class @fooNoAttrs {
 
 /// Test that the pass bails out and leaves IR unchanged if the ClassOp doesn't have any fields
 
-emitc.class @fooNoFields {
+emitc.class @negative_foo_no_fields {
   emitc.func @bar() {
     return
   }
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @fooNoFields {
+// CHECK-LABEL: emitc.class @negative_foo_no_fields {
 // CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
 // CHECK-NEXT:    }
@@ -97,11 +97,11 @@ emitc.class @fooNoFields {
 /// Test that a reflection map is still created in the case that there are no
 /// functions in the class
 
-emitc.class @fooNoOperator {
+emitc.class @foo_no_operator {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 }
 
-// CHECK-LABEL: emitc.class @fooNoOperator {
+// CHECK-LABEL: emitc.class @foo_no_operator {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
 // CHECK-SAME:    #emitc.opaque<"{ { [[$QUOTE]]another_feature[[$QUOTE]], reinterpret_cast<char*>(&fieldName0) } }">
@@ -117,7 +117,7 @@ emitc.class @fooNoOperator {
 /// Test that the pass bails out if a FieldOp has the specified dictionary attribute
 /// with an array containing a type other than string
 
-emitc.class @fooNonStringAttr {
+emitc.class @negative_foo_non_string_attr {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
   emitc.func @bar() {
     return
@@ -125,7 +125,7 @@ emitc.class @fooNonStringAttr {
 }
 
 // CHECK-NOT:     emitc.include
-// CHECK-LABEL: emitc.class @fooNonStringAttr {
+// CHECK-LABEL: emitc.class @negative_foo_non_string_attr {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = [1]}
 // CHECK-NEXT:    emitc.func @bar() {
 // CHECK-NEXT:      return
@@ -136,7 +136,7 @@ emitc.class @fooNonStringAttr {
 
 /// Test that the pass matches one of the multiple included attributes.
 
-emitc.class @fooMultipleAttrs {
+emitc.class @foo_multiple_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
   emitc.func @bar() {
@@ -144,7 +144,7 @@ emitc.class @fooMultipleAttrs {
   }
 }
 
-// CHECK-LABEL: emitc.class @fooMultipleAttrs {
+// CHECK-LABEL: emitc.class @foo_multiple_attrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref_2 = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.field_ref = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> =
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index e4c851b2889ff..6a047851767d7 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -32,7 +32,7 @@ emitc.class @foo {
 
 /// Test that fields with excluded attributes are ignored.
 
-emitc.class @fooExcluded {
+emitc.class @foo_excluded_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @bar() {
@@ -43,7 +43,7 @@ emitc.class @fooExcluded {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class fooExcluded {
+// CHECK-NEXT:  class foo_excluded_attrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];
@@ -61,7 +61,7 @@ emitc.class @fooExcluded {
 
 /// Test that headers aren't added if the pass bails out.
 
-emitc.class @fooNoAttrs {
+emitc.class @negative_foo_no_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>
   emitc.func @bar() {
     return
@@ -70,7 +70,7 @@ emitc.class @fooNoAttrs {
 
 // CHECK-NOT:   #include <map>
 // CHECK-NOT:   #include <string>
-// CHECK:       class fooNoAttrs {
+// CHECK:       class negative_foo_no_attrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    void bar() {

>From 595ac3a6b1f9f47e4a53e286f8834fcdc27304c5 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Thu, 16 Jul 2026 14:28:43 -0700
Subject: [PATCH 20/22] Remove excluded-field-attrs option

---
 .../mlir/Dialect/EmitC/Transforms/Passes.td   | 17 +++-----
 .../Dialect/EmitC/Transforms/Transforms.h     |  3 +-
 .../EmitC/Transforms/MLGOAddReflectionMap.cpp | 43 ++++---------------
 .../EmitC/mlgo-add-reflection-map.mlir        | 35 ++++++++++++---
 .../Target/Cpp/mlgo-add-reflection-map.mlir   |  8 ++--
 5 files changed, 47 insertions(+), 59 deletions(-)

diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
index 5c0dfe7bf69c1..bf526db3a3af1 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Passes.td
@@ -63,15 +63,13 @@ def WrapFuncInClassPass : Pass<"wrap-emitc-func-in-class", "ModuleOp"> {
 def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let summary = "Add a reflection map and a helper method to EmitC classes for runtime field lookup.";
   let description = [{
-    This pass adds a `reflectionMap` field and an accompanying `getBufferForName` method to 
-    EmitC classes, enabling runtime lookup of class fields by name. 
-    This requires that the class has fields with attributes.
+    This pass adds a `reflectionMap` field and an accompanying `getBufferForName`
+    method to EmitC classes, enabling runtime lookup of class fields by name. 
     Each `emitc.field` is expected to have an attribute (configured by the
     `included-field-attrs` option) that is an array containing a single string
-    attribute. If a field possesses both an attribute specified by
-    `included-field-attrs` and an attribute specified in `excluded-field-attrs`,
-    the `included-field-attrs` attribute takes precedence and the field will be
-    included in the reflection map.
+    attribute which will be used as the reflection map key. If none of the specified
+    attributes are present on the field, the field will be excluded from the
+    reflection map.
     
     Example:
 
@@ -98,10 +96,7 @@ def MLGOAddReflectionMapPass : Pass<"mlgo-add-reflection-map", "ModuleOp"> {
   let options = [
     ListOption<"includedFieldAttrs", "included-field-attrs", "std::string",
       "Attribute keys used to extract field names from a field's "
-      "attribute dictionary">,
-    ListOption<"excludedFieldAttrs", "excluded-field-attrs", "std::string",
-      "Attribute keys that, if present on a field, exclude it "
-      "from the reflection map">
+      "attribute dictionary">
   ];
 }
 
diff --git a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
index b3899a2b070d9..795ea58b6b558 100644
--- a/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/EmitC/Transforms/Transforms.h
@@ -43,8 +43,7 @@ void populateWrapFuncInClass(
 //===----------------------------------------------------------------------===//
 
 void populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, ArrayRef<std::string> includedFieldAttrs,
-    ArrayRef<std::string> excludedFieldAttrs);
+    RewritePatternSet &patterns, ArrayRef<std::string> includedFieldAttrs);
 
 } // namespace emitc
 } // namespace mlir
diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index adc6bcbe7f8be..d176515018521 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -52,8 +52,7 @@ class MLGOAddReflectionMapPass
     mlir::ModuleOp moduleOp = getOperation();
 
     RewritePatternSet patterns(&getContext());
-    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs,
-                                         excludedFieldAttrs);
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs);
 
     PatternMatchListener listener;
     walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
@@ -99,9 +98,8 @@ class MLGOAddReflectionMapPass
 /// and a lookup method.
 ///
 /// Fields to be mapped are identified via `includedFieldAttrs` attributes
-/// (e.g., `emitc.field_ref`). Fields containing `excludedFieldAttrs` are
-/// skipped. All other fields must match one of the inclusion/exclusion filters,
-/// otherwise the pattern fails.
+/// (e.g., `emitc.field_ref`). Fields that do not have a matching attribute
+/// are omitted from the reflection map.
 ///
 /// Before:
 /// ```mlir
@@ -131,12 +129,9 @@ class MLGOAddReflectionMapPass
 class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 public:
   MLGOAddReflectionMapClass(MLIRContext *context,
-                            llvm::ArrayRef<std::string> includedFieldAttrs,
-                            llvm::ArrayRef<std::string> excludedFieldAttrs)
+                            llvm::ArrayRef<std::string> includedFieldAttrs)
       : OpRewritePattern<ClassOp>(context),
-        includedFieldAttrs(includedFieldAttrs),
-        excludedFieldAttrs(excludedFieldAttrs.begin(),
-                           excludedFieldAttrs.end()) {}
+        includedFieldAttrs(includedFieldAttrs) {}
 
   LogicalResult matchAndRewrite(ClassOp classOp,
                                 PatternRewriter &rewriter) const override {
@@ -149,7 +144,6 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
     // includedFieldAttrs to use the first element of the array attribute
     // as the reflection map key
     std::vector<std::pair<StringRef, StringRef>> fieldNames;
-    bool hasError = false;
     classOp.walk([&](FieldOp fieldOp) {
       for (const auto &attr : includedFieldAttrs) {
         auto arrayAttr = dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(attr));
@@ -163,25 +157,9 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
           return;
         }
       }
-
-      // If one of the attributes in excludedFieldAttrs is present,
-      // don't add this FieldOp
-      bool shouldIgnore =
-          llvm::any_of(excludedFieldAttrs, [&fieldOp](StringRef ignoreAttr) {
-            return fieldOp->hasAttr(ignoreAttr);
-          });
-      if (shouldIgnore)
-        return;
-
-      (void)rewriter.notifyMatchFailure(fieldOp, [&](Diagnostic &diag) {
-        diag << "FieldOp must have a dictionary attribute named '"
-             << includedFieldAttrs << "' "
-             << "with an array containing a string attribute";
-      });
-      hasError = true;
     });
 
-    if (hasError || fieldNames.empty())
+    if (fieldNames.empty())
       return failure();
 
     // Create reflection map contents
@@ -252,15 +230,10 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
   /// metadata for the reflection map. The pass matches the first attribute
   /// present in the order they are specified in this list.
   llvm::SmallVector<std::string> includedFieldAttrs;
-
-  /// Attributes that, if present on a field, exclude it from the
-  /// reflection map.
-  llvm::SmallVector<std::string> excludedFieldAttrs;
 };
 
 void mlir::emitc::populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, llvm::ArrayRef<std::string> includedFieldAttrs,
-    llvm::ArrayRef<std::string> excludedFieldAttrs) {
+    RewritePatternSet &patterns, llvm::ArrayRef<std::string> includedFieldAttrs) {
   patterns.add<MLGOAddReflectionMapClass>(
-      patterns.getContext(), includedFieldAttrs, excludedFieldAttrs);
+      patterns.getContext(), includedFieldAttrs);
 }
diff --git a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
index 1d5ecc61b118b..5fe768953f050 100644
--- a/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Dialect/EmitC/mlgo-add-reflection-map.mlir
@@ -1,7 +1,7 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2 \
-// RUN: excluded-field-attrs="emitc.other_field"" %s | FileCheck %s '-D$QUOTE=\22'
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref,emitc.field_ref_2" %s | FileCheck %s '-D$QUOTE=\22'
 
-/// Tests that a reflection map is created for fields with a certain attribute.
+/// Tests that a reflection map is created for fields with an attribute in the
+/// included-field-attrs option.
 
 emitc.class @foo {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
@@ -28,10 +28,10 @@ emitc.class @foo {
 
 // -----
 
-/// Test that a reflection map is created for fields with a certain named attribute
-/// but not ones with an attribute present in the ignore-attributes option.
+/// Tests that a reflection map is created for fields in the included-field-attrs
+/// option and skips fields without matching attributes.
 
-emitc.class @foo_excluded_attrs {
+emitc.class @foo_mixed_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @bar() {
@@ -40,7 +40,7 @@ emitc.class @foo_excluded_attrs {
   }
 }
 
-// CHECK:       emitc.class @foo_excluded_attrs {
+// CHECK:       emitc.class @foo_mixed_attrs {
 // CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref = ["another_feature"]}
 // CHECK-NEXT:    emitc.field @fieldName1 : !emitc.array<1xf32> {emitc.other_field = ["some_feature"]}
 // CHECK-NEXT:    emitc.field @reflectionMap : !emitc.opaque<"const std::map<std::string, char*>"> = 
@@ -58,6 +58,27 @@ emitc.class @foo_excluded_attrs {
 
 // -----
 
+/// Test that the pass bails out and leaves IR unchanged if no fields contain
+/// any of the attributes specified in included-field-attrs
+
+emitc.class @foo_unsupported_attr {
+  emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.other_field = ["another_feature"]}
+  emitc.func @bar() {
+    %0 = get_field @fieldName0 : !emitc.array<1xf32>
+    return
+  }
+}
+
+// CHECK:       emitc.class @foo_unsupported_attr {
+// CHECK-NEXT:    emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.other_field = ["another_feature"]}
+// CHECK-NEXT:    emitc.func @bar() {
+// CHECK-NEXT:      %{{.*}} = get_field @fieldName0 : !emitc.array<1xf32>
+// CHECK-NEXT:      return
+// CHECK-NEXT:    }
+// CHECK-NEXT:  }
+
+// -----
+
 /// Test that the pass bails out and leaves IR unchanged if fields don't have any attributes
 
 emitc.class @negative_foo_no_attrs {
diff --git a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
index 6a047851767d7..489b6af79fe2f 100644
--- a/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
+++ b/mlir/test/Target/Cpp/mlgo-add-reflection-map.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref excluded-field-attrs=emitc.other_field" %s | mlir-translate -mlir-to-cpp | FileCheck %s
+// RUN: mlir-opt -split-input-file --mlgo-add-reflection-map="included-field-attrs=emitc.field_ref" %s | mlir-translate -mlir-to-cpp | FileCheck %s
 
 /// Test that a reflection map and lookup function are generated in the class.
 
@@ -30,9 +30,9 @@ emitc.class @foo {
 
 // -----
 
-/// Test that fields with excluded attributes are ignored.
+/// Test that fields without included attributes are ignored.
 
-emitc.class @foo_excluded_attrs {
+emitc.class @foo_unsupported_attrs {
   emitc.field @fieldName0 : !emitc.array<1xf32>  {emitc.field_ref = ["another_feature"]}
   emitc.field @fieldName1 : !emitc.array<1xf32>  {emitc.other_field = ["some_feature"]}
   emitc.func @bar() {
@@ -43,7 +43,7 @@ emitc.class @foo_excluded_attrs {
 
 // CHECK:       #include <map>
 // CHECK-NEXT:  #include <string>
-// CHECK-NEXT:  class foo_excluded_attrs {
+// CHECK-NEXT:  class foo_unsupported_attrs {
 // CHECK-NEXT:   public:
 // CHECK-NEXT:    float fieldName0[1];
 // CHECK-NEXT:    float fieldName1[1];

>From ced3fee1253bfae065d04d545f50624c6a5fdda5 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Thu, 16 Jul 2026 14:44:59 -0700
Subject: [PATCH 21/22] Formatting

---
 mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index d176515018521..02a6d2d58255b 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -233,7 +233,8 @@ class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
 };
 
 void mlir::emitc::populateMLGOAddReflectionMapPatterns(
-    RewritePatternSet &patterns, llvm::ArrayRef<std::string> includedFieldAttrs) {
-  patterns.add<MLGOAddReflectionMapClass>(
-      patterns.getContext(), includedFieldAttrs);
+    RewritePatternSet &patterns,
+    llvm::ArrayRef<std::string> includedFieldAttrs) {
+  patterns.add<MLGOAddReflectionMapClass>(patterns.getContext(),
+                                          includedFieldAttrs);
 }

>From fcfa12876f117d186fbfe8ffe97b686741b765f6 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Sun, 19 Jul 2026 14:27:19 -0700
Subject: [PATCH 22/22] Fix comment formatting

---
 mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
index 02a6d2d58255b..6900e0d989b92 100644
--- a/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
+++ b/mlir/lib/Dialect/EmitC/Transforms/MLGOAddReflectionMap.cpp
@@ -105,7 +105,8 @@ class MLGOAddReflectionMapPass
 /// ```mlir
 /// emitc.class @foo {
 ///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
-///   ["another_feature"]} emitc.func @bar() { return }
+///   ["another_feature"]}
+/// emitc.func @bar() { return }
 /// }
 /// ```
 ///


From llvmlistbot at llvm.org  Sun Jul 19 14:30:35 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Sun, 19 Jul 2026 14:30:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Create a pass to add a
 reflection map to a class (PR #205464)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205464@github.com>
Message-ID: <6a5d41fb.c56fb102.2d07b8.ec84@mx.google.com>


================
@@ -0,0 +1,240 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/EmitC/IR/EmitC.h"
+#include "mlir/Dialect/EmitC/Transforms/Passes.h"
+#include "mlir/Dialect/EmitC/Transforms/Transforms.h"
+#include "mlir/IR/Attributes.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace mlir;
+using namespace emitc;
+
+namespace mlir {
+namespace emitc {
+#define GEN_PASS_DEF_MLGOADDREFLECTIONMAPPASS
+#include "mlir/Dialect/EmitC/Transforms/Passes.h.inc"
+
+namespace {
+constexpr const char *mapLibraryHeader = "map";
+constexpr const char *stringLibraryHeader = "string";
+
+struct PatternMatchListener : public RewriterBase::Listener {
+  bool patternApplied = false;
+
+  void notifyOperationInserted(Operation *op,
+                               OpBuilder::InsertPoint previous) override {
+    patternApplied = true;
+  }
+};
+
+IncludeOp addHeader(OpBuilder &builder, ModuleOp module, StringRef headerName) {
+  StringAttr includeAttr = builder.getStringAttr(headerName);
+  return IncludeOp::create(builder, module.getLoc(), includeAttr,
+                           /*is_standard_include=*/builder.getUnitAttr());
+}
+
+class MLGOAddReflectionMapPass
+    : public impl::MLGOAddReflectionMapPassBase<MLGOAddReflectionMapPass> {
+  using MLGOAddReflectionMapPassBase::MLGOAddReflectionMapPassBase;
+  void runOnOperation() override {
+    mlir::ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    populateMLGOAddReflectionMapPatterns(patterns, includedFieldAttrs);
+
+    PatternMatchListener listener;
+    walkAndApplyPatterns(moduleOp, std::move(patterns), &listener);
+
+    // If nothing was matched, no reflection maps were added, removing the need
+    // to add include headers for map and string
+    if (!listener.patternApplied)
+      return;
+
+    // Check if the map and/or string headers are already present
+    bool hasMapHdr = false;
+    bool hasStringHdr = false;
+    for (auto &op : *moduleOp.getBody()) {
+      IncludeOp includeOp = llvm::dyn_cast<IncludeOp>(op);
+      if (!includeOp)
+        continue;
+
+      if (includeOp.getIsStandardInclude()) {
+        auto include = includeOp.getInclude();
+
+        hasMapHdr |= include == mapLibraryHeader;
+        hasStringHdr |= include == stringLibraryHeader;
+      }
+
+      if (hasMapHdr && hasStringHdr)
+        return;
+    }
+
+    mlir::OpBuilder builder(moduleOp.getBody(), moduleOp.getBody()->begin());
+    if (!hasMapHdr)
+      addHeader(builder, moduleOp, mapLibraryHeader);
+
+    if (!hasStringHdr)
+      addHeader(builder, moduleOp, stringLibraryHeader);
+  }
+};
+
+} // namespace
+} // namespace emitc
+} // namespace mlir
+
+/// Rewrites a `emitc::ClassOp` to generate a reflection map of member fields
+/// and a lookup method.
+///
+/// Fields to be mapped are identified via `includedFieldAttrs` attributes
+/// (e.g., `emitc.field_ref`). Fields that do not have a matching attribute
+/// are omitted from the reflection map.
+///
+/// Before:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.func @bar() { return }
+/// }
+/// ```
+///
+/// After:
+/// ```mlir
+/// emitc.class @foo {
+///   emitc.field @fieldName0 : !emitc.array<1xf32> {emitc.field_ref =
+///   ["another_feature"]} emitc.field @reflectionMap : !emitc.opaque<"const
+///   std::map<std::string, char*>"> =
+///     #emitc.opaque<"{ { \"another_feature\",
+///     reinterpret_cast<char*>(&fieldName0) } }">
+///   emitc.func @getBufferForName(%name: !emitc.opaque<"std::string">) ->
+///   !emitc.ptr<!emitc.opaque<"char">> {
+///     %map = get_field @reflectionMap : !emitc.opaque<"const
+///     std::map<std::string, char*>"> %ptr = member_call_opaque %map
+///     "at"(%name) : ... return %ptr : !emitc.ptr<!emitc.opaque<"char">>
+///   }
+///   emitc.func @bar() { return }
+/// }
+/// ```
+class MLGOAddReflectionMapClass : public OpRewritePattern<ClassOp> {
+public:
+  MLGOAddReflectionMapClass(MLIRContext *context,
+                            llvm::ArrayRef<std::string> includedFieldAttrs)
+      : OpRewritePattern<ClassOp>(context),
+        includedFieldAttrs(includedFieldAttrs) {}
+
+  LogicalResult matchAndRewrite(ClassOp classOp,
+                                PatternRewriter &rewriter) const override {
+    MLIRContext *context = rewriter.getContext();
+
+    emitc::OpaqueType mapType = mlir::emitc::OpaqueType::get(
+        context, "const std::map<std::string, char*>");
+
+    // Collect the names of all FieldOps that have one of the attributes in
+    // includedFieldAttrs to use the first element of the array attribute
+    // as the reflection map key
+    std::vector<std::pair<StringRef, StringRef>> fieldNames;
+    classOp.walk([&](FieldOp fieldOp) {
+      for (const auto &attr : includedFieldAttrs) {
+        auto arrayAttr = dyn_cast_if_present<ArrayAttr>(fieldOp->getAttr(attr));
+
+        if (!arrayAttr)
+          continue;
+
+        if (!arrayAttr.empty() && isa<StringAttr>(arrayAttr[0])) {
+          fieldNames.emplace_back(cast<StringAttr>(arrayAttr[0]).getValue(),
+                                  fieldOp.getName());
+          return;
+        }
+      }
+    });
+
+    if (fieldNames.empty())
+      return failure();
+
+    // Create reflection map contents
+    std::string reflectionMapContents;
+    reflectionMapContents += "{ ";
+    bool first = true;
+    for (const auto &[name, value] : fieldNames) {
+      if (!first)
+        reflectionMapContents += ", ";
+
+      first = false;
+      reflectionMapContents += llvm::formatv(
+          "{ \"{0}\", reinterpret_cast<char*>(&{1}) }", name, value);
+    }
+    reflectionMapContents += " }";
+
+    // Set insertion point before the first function or after all fields
+    // if there are no functions within the class
+    auto funcs = classOp.getBlock().getOps<FuncOp>();
+    auto it = funcs.begin();
+    if (it != funcs.end())
+      rewriter.setInsertionPoint(*it);
+    else
+      rewriter.setInsertionPointToEnd(&classOp.getBlock());
----------------
beamandala wrote:

Since the reflection map is a field, I think it's cleaner to place it with the other fields.

https://github.com/llvm/llvm-project/pull/205464

From llvmlistbot at llvm.org  Sun Jul 19 14:57:51 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 14:57:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d485f.27b3b003.1f84cc.c300@mx.google.com>


https://github.com/KrxGu updated https://github.com/llvm/llvm-project/pull/209106

>From fbe630086b549c739d2c2686804a5ec9afd0b13c Mon Sep 17 00:00:00 2001
From: KrxGu <krishom70 at gmail.com>
Date: Mon, 13 Jul 2026 13:40:50 +0530
Subject: [PATCH 1/2] [mlir][bufferization] Handle arith.select-based deallocs
 in static memory planner
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Allocs freed indirectly via arith.select chains were previously skipped.
This adds forward select-chain traversal so patterns like:
  %2 = arith.select %c, %0, %1
  memref.dealloc %2
are now handled correctly.

A group constraint ensures that all allocs sharing a select-based
dealloc are either all placed in the arena or all skipped — putting
one alloc in while leaving its peer out would break the dealloc.

Also fixes the O(n*m) block scan in buildAllocInfos by doing a single
upfront pass with a DenseMap index.

Tests added for single-alloc select, shared select-dealloc, and the
two-select two-dealloc pattern.
---
 .../StaticMemoryPlannerAnalysis.cpp           | 120 ++++++++++++------
 .../static-memory-planner-analysis.mlir       |  61 +++++++++
 2 files changed, 143 insertions(+), 38 deletions(-)

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index c2ac40a8427e7..770a46ff3de14 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -18,6 +18,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/IR/Builders.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/Support/Debug.h"
 #include <numeric>
 
@@ -34,10 +35,12 @@ using namespace mlir;
 
 namespace {
 
-/// A candidate allocation with its matching deallocation and assigned offset.
+/// A candidate allocation with its matching deallocation(s) and assigned
+/// offset. An alloc may be freed indirectly through arith.select chains,
+/// yielding multiple potential deallocs — all must be in the same block.
 struct AllocationCandidate {
   memref::AllocOp alloc;
-  memref::DeallocOp dealloc;
+  SmallVector<memref::DeallocOp> deallocs;
   int64_t offset = 0; // Offset in bytes from arena start (assigned by planner)
   int64_t sizeInBytes = 0; // Size in bytes
   int64_t alignment = 1;   // Required alignment in bytes
@@ -47,18 +50,25 @@ struct AllocationCandidate {
 // Helper utilities
 //===----------------------------------------------------------------------===//
 
-/// Finds the unique dealloc operation for a given alloc value.
-/// Returns nullptr if there are zero or multiple deallocs.
-static memref::DeallocOp findUniqueDealloc(Value allocValue) {
-  memref::DeallocOp deallocOp = nullptr;
-  for (Operation *user : allocValue.getUsers()) {
+/// Collect all dealloc ops that might free the given value, following
+/// arith.select chains. For example:
+///   %0 = memref.alloc()
+///   %2 = arith.select %c, %0, %1
+///   memref.dealloc %2    <- this covers %0 conditionally
+/// `visited` prevents cycles in the use-def graph.
+static void findPotentialDeallocs(Value value,
+                                  SmallVectorImpl<memref::DeallocOp> &deallocs,
+                                  SmallPtrSetImpl<Value> &visited) {
+  if (!visited.insert(value).second)
+    return;
+  for (Operation *user : value.getUsers()) {
     if (auto dealloc = dyn_cast<memref::DeallocOp>(user)) {
-      if (deallocOp)
-        return nullptr; // Multiple deallocs found
-      deallocOp = dealloc;
+      deallocs.push_back(dealloc);
+    } else if (auto select = dyn_cast<arith::SelectOp>(user)) {
+      if (isa<MemRefType>(select.getType()))
+        findPotentialDeallocs(select.getResult(), deallocs, visited);
     }
   }
-  return deallocOp;
 }
 
 /// Compute the size in bytes for a memref type.
@@ -70,25 +80,31 @@ static int64_t computeSizeInBytes(MemRefType memrefType) {
 
 /// Build lifetime-annotated allocation descriptors from candidates.
 /// Returns the arena alignment (LCM of all individual alignments).
+/// Uses a single block scan (O(n+m)) instead of one scan per candidate.
 static int64_t buildAllocInfos(
     MutableArrayRef<AllocationCandidate> candidates,
     SmallVectorImpl<bufferization::MemoryPlannerAlloc> &allocInfos) {
+  // Build an op-index map with a single pass over the block.
+  DenseMap<Operation *, int64_t> opIndex;
+  if (!candidates.empty()) {
+    Block *block = candidates.front().alloc->getBlock();
+    int64_t idx = 0;
+    for (Operation &op : *block)
+      opIndex[&op] = idx++;
+  }
+
   int64_t arenaAlignment = 1;
   for (auto &candidate : candidates) {
     bufferization::MemoryPlannerAlloc info;
     info.sizeInBytes = candidate.sizeInBytes;
     info.alignment = candidate.alignment;
-
-    Block *block = candidate.alloc->getBlock();
-    int64_t opIdx = 0;
-    for (Operation &op : *block) {
-      if (&op == candidate.alloc.getOperation())
-        info.timeStart = opIdx;
-      if (&op == candidate.dealloc.getOperation())
-        info.timeEnd = opIdx;
-      ++opIdx;
-    }
-
+    info.timeStart = opIndex.lookup(candidate.alloc.getOperation());
+    // Conservative: timeEnd = latest dealloc index among all potential
+    // deallocs.
+    int64_t timeEnd = 0;
+    for (memref::DeallocOp d : candidate.deallocs)
+      timeEnd = std::max(timeEnd, opIndex.lookup(d.getOperation()));
+    info.timeEnd = timeEnd;
     allocInfos.push_back(info);
     arenaAlignment = std::lcm(arenaAlignment, candidate.alignment);
   }
@@ -96,45 +112,59 @@ static int64_t buildAllocInfos(
 }
 
 /// Collect alloc/dealloc pairs eligible for arena placement.
-/// An allocation is eligible if it has a static shape and a unique dealloc
-/// in the same block.
+/// An allocation is eligible if it has a static shape and all of its
+/// potential deallocs (including those reached via arith.select chains)
+/// are in the same block. A group constraint ensures that all allocs
+/// sharing a select-based dealloc are either all eligible or all skipped.
 static SmallVector<AllocationCandidate>
 collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
                   llvm::Statistic &numSkipNoDealloc,
                   llvm::Statistic &numEligible) {
-  SmallVector<AllocationCandidate> candidates;
+  // Walk allocs, find all potential deallocs via select chains.
+  SmallVector<AllocationCandidate> potentialCandidates;
 
   funcOp->walk([&](memref::AllocOp allocOp) {
     MemRefType memrefType = allocOp.getType();
-
-    // Skip dynamic shapes
     if (!memrefType.hasStaticShape()) {
       ++numSkipDynamic;
       return;
     }
 
-    // Find unique dealloc in the same block
-    memref::DeallocOp deallocOp = findUniqueDealloc(allocOp.getResult());
-    if (!deallocOp) {
+    SmallVector<memref::DeallocOp> deallocs;
+    SmallPtrSet<Value, 8> visited;
+    findPotentialDeallocs(allocOp.getResult(), deallocs, visited);
+
+    if (deallocs.empty()) {
       ++numSkipNoDealloc;
       return;
     }
 
-    if (deallocOp->getBlock() != allocOp->getBlock()) {
+    // All deallocs must be in the same block as the alloc.
+    bool allSameBlock = llvm::all_of(deallocs, [&](memref::DeallocOp d) {
+      return d->getBlock() == allocOp->getBlock();
+    });
+    if (!allSameBlock) {
       ++numSkipNoDealloc;
       return;
     }
 
-    // This allocation is eligible
-    ++numEligible;
     AllocationCandidate candidate;
     candidate.alloc = allocOp;
-    candidate.dealloc = deallocOp;
+    candidate.deallocs = deallocs;
     candidate.sizeInBytes = computeSizeInBytes(memrefType);
     candidate.alignment = allocOp.getAlignment().value_or(1);
-    candidates.push_back(candidate);
+    potentialCandidates.push_back(candidate);
   });
 
+  // Note: arith.select requires both operands to have the same type, so if
+  // one alloc in a select group has a static shape, all others must too.
+  // A mixed-eligibility group (some eligible, some not) is therefore
+  // impossible and no group-constraint fixpoint is needed.
+  SmallVector<AllocationCandidate> candidates;
+  for (auto &c : potentialCandidates) {
+    ++numEligible;
+    candidates.push_back(c);
+  }
   return candidates;
 }
 
@@ -180,8 +210,14 @@ static FailureOr<Value> createArena(OpBuilder &builder,
 }
 
 /// Replace each alloc/dealloc pair with a memref.view into the arena.
+/// Handles select-chained deallocs: a single dealloc may cover multiple allocs,
+/// so we track erased deallocs to avoid double-erase.
 static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
                                Value arenaValue) {
+  SmallPtrSet<Operation *, 8> erasedDeallocs;
+  SmallVector<Operation *> allocsToErase;
+
+  // First replace all alloc results (rewires selects too), collect for erase.
   for (auto &candidate : candidates) {
     OpBuilder builder(candidate.alloc);
     Location loc = candidate.alloc.getLoc();
@@ -191,11 +227,19 @@ static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
         arith::ConstantIndexOp::create(builder, loc, candidate.offset);
     auto view = memref::ViewOp::create(builder, loc, originalType, arenaValue,
                                        offsetIndex, SmallVector<Value>{});
-
     candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
-    candidate.alloc.erase();
-    candidate.dealloc.erase();
+    allocsToErase.push_back(candidate.alloc.getOperation());
   }
+
+  // Erase deallocs first (they may reference alloc results via selects).
+  for (auto &candidate : candidates)
+    for (memref::DeallocOp d : candidate.deallocs)
+      if (erasedDeallocs.insert(d.getOperation()).second)
+        d.erase();
+
+  // Erase allocs last (no users remain after replaceAllUsesWith).
+  for (Operation *allocOp : allocsToErase)
+    allocOp->erase();
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
index a80c0e13adc21..14c8dbf8fd0fc 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
@@ -190,3 +190,64 @@ func.func @lcm_alignment() {
   memref.dealloc %alloc1 : memref<3xi32>
   return
 }
+
+// -----
+
+// Test 10: Single alloc freed via arith.select-based dealloc.
+// CHECK-LABEL: func @select_single_alloc
+func.func @select_single_alloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<4096xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<4096xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %alloc = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %alloc, %alloc : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 11: Two allocs freed via a shared select-based dealloc.
+// Group constraint: both must be eligible together or neither is.
+// CHECK-LABEL: func @select_shared_dealloc
+func.func @select_shared_dealloc() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %[[V0:.*]] = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %[[V1:.*]] = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 12: Two allocs, two select-based deallocs (mentor's canonical example).
+// %a freed via dealloc(%sel1) or dealloc(%sel2), %b likewise.
+// CHECK-LABEL: func @select_two_deallocs
+func.func @select_two_deallocs() {
+  %c = arith.constant true
+  // CHECK: %[[ARENA:.*]] = memref.alloc() {alignment = 1 : i64} : memref<8192xi8>
+  // CHECK-NEXT: %[[C0:.*]] = arith.constant 0 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C0]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NEXT: %[[C4096:.*]] = arith.constant 4096 : index
+  // CHECK-NEXT: %{{.*}} = memref.view %[[ARENA]][%[[C4096]]][] : memref<8192xi8> to memref<1024xf32>
+  // CHECK-NOT: memref.alloc
+  // CHECK-NOT: memref.dealloc
+  %a = memref.alloc() : memref<1024xf32>
+  %b = memref.alloc() : memref<1024xf32>
+  %sel1 = arith.select %c, %a, %b : memref<1024xf32>
+  memref.dealloc %sel1 : memref<1024xf32>
+  %sel2 = arith.select %c, %b, %a : memref<1024xf32>
+  memref.dealloc %sel2 : memref<1024xf32>
+  return
+}

>From 30fd90d09690c96a63076c1df77d100448213f74 Mon Sep 17 00:00:00 2001
From: KrxGu <krishom70 at gmail.com>
Date: Mon, 20 Jul 2026 03:27:35 +0530
Subject: [PATCH 2/2] [mlir][bufferization] Address review feedback on
 arith.select dealloc pass

- Use BufferViewFlowOpInterface instead of hardcoded arith::SelectOp check,
  so any future op implementing the interface is handled automatically
- Error out (not skip) when no dealloc or cross-block dealloc is found;
  the only valid skip is dynamic shapes
- Remove redundant potentialCandidates vector copy in collectCandidates
- Replace erasedDeallocs with deallocsToErase set in rewriteAllocations
- Remove outdated comment about group constraint
- Add error tests for missing-dealloc and cross-block cases
---
 .../StaticMemoryPlannerAnalysis.cpp           | 95 ++++++++++---------
 .../static-memory-planner-analysis.mlir       | 40 ++------
 .../static-memory-planner-errors.mlir         | 24 +++++
 3 files changed, 78 insertions(+), 81 deletions(-)
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-errors.mlir

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
index 770a46ff3de14..f1db0d5ee55a4 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/StaticMemoryPlannerAnalysis.cpp
@@ -13,6 +13,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/IR/Arith.h"
+#include "mlir/Dialect/Bufferization/IR/BufferViewFlowOpInterface.h"
 #include "mlir/Dialect/Bufferization/Transforms/Passes.h"
 #include "mlir/Dialect/Bufferization/Transforms/StaticMemoryPlanning.h"
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
@@ -50,12 +51,16 @@ struct AllocationCandidate {
 // Helper utilities
 //===----------------------------------------------------------------------===//
 
-/// Collect all dealloc ops that might free the given value, following
-/// arith.select chains. For example:
+/// Collect all dealloc ops that might free the given value, following ops
+/// that implement BufferViewFlowOpInterface (e.g. arith.select). For example:
 ///   %0 = memref.alloc()
 ///   %2 = arith.select %c, %0, %1
 ///   memref.dealloc %2    <- this covers %0 conditionally
 /// `visited` prevents cycles in the use-def graph.
+///
+/// TODO: This relies on BufferViewFlowOpInterface external models being
+/// registered for the ops in the IR (e.g. via
+/// arith::registerBufferViewFlowOpInterfaceExternalModels).
 static void findPotentialDeallocs(Value value,
                                   SmallVectorImpl<memref::DeallocOp> &deallocs,
                                   SmallPtrSetImpl<Value> &visited) {
@@ -64,9 +69,11 @@ static void findPotentialDeallocs(Value value,
   for (Operation *user : value.getUsers()) {
     if (auto dealloc = dyn_cast<memref::DeallocOp>(user)) {
       deallocs.push_back(dealloc);
-    } else if (auto select = dyn_cast<arith::SelectOp>(user)) {
-      if (isa<MemRefType>(select.getType()))
-        findPotentialDeallocs(select.getResult(), deallocs, visited);
+    } else if (dyn_cast<bufferization::BufferViewFlowOpInterface>(user)) {
+      // Follow any op that propagates buffer values to its results.
+      for (Value result : user->getResults())
+        if (isa<MemRefType>(result.getType()))
+          findPotentialDeallocs(result, deallocs, visited);
     }
   }
 }
@@ -112,22 +119,20 @@ static int64_t buildAllocInfos(
 }
 
 /// Collect alloc/dealloc pairs eligible for arena placement.
-/// An allocation is eligible if it has a static shape and all of its
-/// potential deallocs (including those reached via arith.select chains)
-/// are in the same block. A group constraint ensures that all allocs
-/// sharing a select-based dealloc are either all eligible or all skipped.
-static SmallVector<AllocationCandidate>
+/// An allocation is eligible if it has a static shape and its deallocs
+/// (including those reached via BufferViewFlowOpInterface chains)
+/// are in the same block. Allocations with dynamic shapes are skipped.
+/// Missing deallocs or cross-block deallocs are reported as errors.
+static LogicalResult
 collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
-                  llvm::Statistic &numSkipNoDealloc,
-                  llvm::Statistic &numEligible) {
-  // Walk allocs, find all potential deallocs via select chains.
-  SmallVector<AllocationCandidate> potentialCandidates;
-
-  funcOp->walk([&](memref::AllocOp allocOp) {
+                  llvm::Statistic &numEligible,
+                  SmallVector<AllocationCandidate> &candidates) {
+  bool walkFailed = false;
+  funcOp->walk([&](memref::AllocOp allocOp) -> WalkResult {
     MemRefType memrefType = allocOp.getType();
     if (!memrefType.hasStaticShape()) {
       ++numSkipDynamic;
-      return;
+      return WalkResult::advance();
     }
 
     SmallVector<memref::DeallocOp> deallocs;
@@ -135,37 +140,32 @@ collectCandidates(FunctionOpInterface funcOp, llvm::Statistic &numSkipDynamic,
     findPotentialDeallocs(allocOp.getResult(), deallocs, visited);
 
     if (deallocs.empty()) {
-      ++numSkipNoDealloc;
-      return;
+      allocOp.emitError("no dealloc found; run the deallocation pipeline "
+                        "before this pass");
+      walkFailed = true;
+      return WalkResult::interrupt();
     }
 
-    // All deallocs must be in the same block as the alloc.
-    bool allSameBlock = llvm::all_of(deallocs, [&](memref::DeallocOp d) {
-      return d->getBlock() == allocOp->getBlock();
-    });
-    if (!allSameBlock) {
-      ++numSkipNoDealloc;
-      return;
+    for (memref::DeallocOp d : deallocs) {
+      if (d->getBlock() != allocOp->getBlock()) {
+        allocOp.emitError("dealloc is in a different block than the alloc; "
+                          "run the deallocation pipeline before this pass");
+        walkFailed = true;
+        return WalkResult::interrupt();
+      }
     }
 
+    ++numEligible;
     AllocationCandidate candidate;
     candidate.alloc = allocOp;
     candidate.deallocs = deallocs;
     candidate.sizeInBytes = computeSizeInBytes(memrefType);
     candidate.alignment = allocOp.getAlignment().value_or(1);
-    potentialCandidates.push_back(candidate);
+    candidates.push_back(candidate);
+    return WalkResult::advance();
   });
 
-  // Note: arith.select requires both operands to have the same type, so if
-  // one alloc in a select group has a static shape, all others must too.
-  // A mixed-eligibility group (some eligible, some not) is therefore
-  // impossible and no group-constraint fixpoint is needed.
-  SmallVector<AllocationCandidate> candidates;
-  for (auto &c : potentialCandidates) {
-    ++numEligible;
-    candidates.push_back(c);
-  }
-  return candidates;
+  return failure(walkFailed);
 }
 
 /// Create or obtain the arena buffer based on the arena mode.
@@ -210,14 +210,12 @@ static FailureOr<Value> createArena(OpBuilder &builder,
 }
 
 /// Replace each alloc/dealloc pair with a memref.view into the arena.
-/// Handles select-chained deallocs: a single dealloc may cover multiple allocs,
-/// so we track erased deallocs to avoid double-erase.
 static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
                                Value arenaValue) {
-  SmallPtrSet<Operation *, 8> erasedDeallocs;
+  SmallPtrSet<Operation *, 8> deallocsToErase;
   SmallVector<Operation *> allocsToErase;
 
-  // First replace all alloc results (rewires selects too), collect for erase.
+  // Replace all alloc results with views (rewires selects too).
   for (auto &candidate : candidates) {
     OpBuilder builder(candidate.alloc);
     Location loc = candidate.alloc.getLoc();
@@ -229,13 +227,14 @@ static void rewriteAllocations(MutableArrayRef<AllocationCandidate> candidates,
                                        offsetIndex, SmallVector<Value>{});
     candidate.alloc.getResult().replaceAllUsesWith(view.getResult());
     allocsToErase.push_back(candidate.alloc.getOperation());
+
+    for (memref::DeallocOp d : candidate.deallocs)
+      deallocsToErase.insert(d.getOperation());
   }
 
   // Erase deallocs first (they may reference alloc results via selects).
-  for (auto &candidate : candidates)
-    for (memref::DeallocOp d : candidate.deallocs)
-      if (erasedDeallocs.insert(d.getOperation()).second)
-        d.erase();
+  for (Operation *d : deallocsToErase)
+    d->erase();
 
   // Erase allocs last (no users remain after replaceAllUsesWith).
   for (Operation *allocOp : allocsToErase)
@@ -270,8 +269,10 @@ void StaticMemoryPlannerAnalysisPass::runOnOperation() {
   }
 
   // Step 1: Collect eligible allocation candidates.
-  SmallVector<AllocationCandidate> candidates =
-      collectCandidates(funcOp, numSkipDynamic, numSkipNoDealloc, numEligible);
+  SmallVector<AllocationCandidate> candidates;
+  if (failed(
+          collectCandidates(funcOp, numSkipDynamic, numEligible, candidates)))
+    return signalPassFailure();
 
   if (candidates.empty())
     return;
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
index 14c8dbf8fd0fc..af900be29a22e 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-analysis.mlir
@@ -85,35 +85,7 @@ func.func @dynamic_shape_skipped(%n: index) {
 
 // -----
 
-// Test 5: No dealloc - should be skipped
-// CHECK-LABEL: func @no_dealloc_skipped
-func.func @no_dealloc_skipped() {
-  // CHECK: %[[ALLOC:.*]] = memref.alloc() : memref<1024xf32>
-  // CHECK-NOT: memref.subview
-  %alloc = memref.alloc() : memref<1024xf32>
-  return
-}
-
-// -----
-
-// Test 6: Dealloc in different block - should be skipped
-// CHECK-LABEL: func @different_block_skipped
-func.func @different_block_skipped(%cond: i1) {
-  // CHECK: %[[ALLOC:.*]] = memref.alloc() : memref<1024xf32>
-  // CHECK: scf.if
-  // CHECK: memref.dealloc %[[ALLOC]]
-  // CHECK-NOT: memref.subview
-  %alloc = memref.alloc() : memref<1024xf32>
-  scf.if %cond {
-    memref.dealloc %alloc : memref<1024xf32>
-    scf.yield
-  }
-  return
-}
-
-// -----
-
-// Test 7: Multiple allocations with sequential offsets
+// Test 5: Multiple allocations with sequential offsets
 // CHECK-LABEL: func @multiple_sequential
 func.func @multiple_sequential() {
   // Arena: 1024*4 + 512*4 + 2048*4 = 14336 bytes
@@ -140,7 +112,7 @@ func.func @multiple_sequential() {
 
 // -----
 
-// Test 8: Alignment requirements with padding
+// Test 6: Alignment requirements with padding
 // CHECK-LABEL: func @alignment_padding
 func.func @alignment_padding() {
   // Arena: 256*4 + 128*4 + 64*4 = 1792 bytes, alignment = lcm(128,64,128) = 128
@@ -167,7 +139,7 @@ func.func @alignment_padding() {
 
 // -----
 
-// Test 9: LCM arena alignment (alignment=4, alignment=16 → lcm=16).
+// Test 7: LCM arena alignment (alignment=4, alignment=16 → lcm=16).
 // For power-of-2 alignments lcm equals max, but lcm is the correct
 // general formula. Arena must be aligned to 16 so that all views are
 // correctly aligned regardless of their individual requirements.
@@ -193,7 +165,7 @@ func.func @lcm_alignment() {
 
 // -----
 
-// Test 10: Single alloc freed via arith.select-based dealloc.
+// Test 8: Single alloc freed via arith.select-based dealloc.
 // CHECK-LABEL: func @select_single_alloc
 func.func @select_single_alloc() {
   %c = arith.constant true
@@ -210,7 +182,7 @@ func.func @select_single_alloc() {
 
 // -----
 
-// Test 11: Two allocs freed via a shared select-based dealloc.
+// Test 9: Two allocs freed via a shared select-based dealloc.
 // Group constraint: both must be eligible together or neither is.
 // CHECK-LABEL: func @select_shared_dealloc
 func.func @select_shared_dealloc() {
@@ -231,7 +203,7 @@ func.func @select_shared_dealloc() {
 
 // -----
 
-// Test 12: Two allocs, two select-based deallocs (mentor's canonical example).
+// Test 10: Two allocs, two select-based deallocs (mentor's canonical example).
 // %a freed via dealloc(%sel1) or dealloc(%sel2), %b likewise.
 // CHECK-LABEL: func @select_two_deallocs
 func.func @select_two_deallocs() {
diff --git a/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-errors.mlir b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-errors.mlir
new file mode 100644
index 0000000000000..86af0649c67ba
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/static-memory-planner-errors.mlir
@@ -0,0 +1,24 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(static-memory-planner-analysis))" \
+// RUN:     -split-input-file -verify-diagnostics
+
+// -----
+
+// Test 1: Alloc with no dealloc should be an error (not silently skipped).
+func.func @error_no_dealloc() {
+  // expected-error @+1 {{no dealloc found; run the deallocation pipeline before this pass}}
+  %alloc = memref.alloc() : memref<1024xf32>
+  return
+}
+
+// -----
+
+// Test 2: Alloc whose dealloc is in a different block should be an error.
+func.func @error_cross_block_dealloc(%cond: i1) {
+  // expected-error @+1 {{dealloc is in a different block than the alloc; run the deallocation pipeline before this pass}}
+  %alloc = memref.alloc() : memref<1024xf32>
+  scf.if %cond {
+    memref.dealloc %alloc : memref<1024xf32>
+    scf.yield
+  }
+  return
+}


From llvmlistbot at llvm.org  Sun Jul 19 15:10:39 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:10:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4b5f.7d702ed2.13d102.6092@mx.google.com>


================

----------------
KrxGu wrote:

made the required changes.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:11:45 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:11:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4ba1.c5504292.78ef3.d948@mx.google.com>


================

----------------
KrxGu wrote:

Done ,`collectCandidates` now calls `allocOp.emitError(...)` and returns `failure()` when no dealloc is found. `runOnOperation` checks the result and calls `signalPassFailure()`. Added a test in `static-memory-planner-errors.mlir`.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:12:21 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:12:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4bc5.d681cab9.169270.6632@mx.google.com>


================

----------------
KrxGu wrote:

Done, same pattern: `emitError` + `signalPassFailure`. Dynamic shapes remain the only case that's silently skipped. Added a test for this case too.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:12:34 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:12:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4bd2.eb165e5c.4506c.2f44@mx.google.com>


================

----------------
KrxGu wrote:

done

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:13:02 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:13:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4bee.66c0679a.24976c.199e@mx.google.com>


================

----------------
KrxGu wrote:

Fixed, removed `potentialCandidates` entirely and push directly into `candidates` inside the walk. Moved `++numEligible` inline. Also changed the return type to `LogicalResult` and moved the output candidates to an out-param since we now need to propagate failure.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:13:41 2026
From: llvmlistbot at llvm.org (Krish Gupta)
Date: Sun, 19 Jul 2026 15:13:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Handle
 arith.select-based deallocs in static memory planner (PR #209106)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209106@github.com>
Message-ID: <6a5d4c15.a1accbb4.5eca2.3d23@mx.google.com>


================

----------------
KrxGu wrote:

Done, collect all deallocs into a `SmallPtrSet<Operation*, 8> deallocsToErase` while processing each candidate (inserting duplicates is a no-op), then erase all of them in one loop. Simpler and clearer.

https://github.com/llvm/llvm-project/pull/209106

From llvmlistbot at llvm.org  Sun Jul 19 15:50:28 2026
From: llvmlistbot at llvm.org (Aayush Shrivastava)
Date: Sun, 19 Jul 2026 15:50:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5d54b4.b1416da1.3242c8.ecc9@mx.google.com>


iamaayushrivastava wrote:

> @gysit I'm a bit out of the loop with respect to the LLVM implementation of MemorySlot, so I trust your judgement.
> 
> @iamaayushrivastava do you have merge rights into the LLVM repo? If you are new to the LLVM project, please also keep in mind [per policy](https://llvm.org/docs/AIToolPolicy.html) any use of AI tools must be disclosed in PRs.

Thanks! I don't have merge rights to the LLVM repo, and yes, I'm aware of the [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html) regarding AI-assisted contributions.


https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Sun Jul 19 16:08:04 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 19 Jul 2026 16:08:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5d58d4.92d2873c.aa1ff.0bbd@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/210571

>From c04f3e6990e446ca867a1594b3e9140a28ad8e51 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 12:04:15 +0900
Subject: [PATCH] [MLIR][Affine] Preserve discardable attrs when adding loop
 yields

AffineForOp::replaceWithAdditionalYields rebuilds the loop but previously dropped discardable attributes attached by scheduling and transformation pipelines.

Copy the original discardable attribute dictionary to the replacement loop and add an unroll-and-jam regression that exercises the LoopLikeOpInterface path.
---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 1 +
 mlir/test/Dialect/Affine/unroll-jam.mlir | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..d4c4dd6b48424 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2893,6 +2893,7 @@ FailureOr<LoopLikeOpInterface> AffineForOp::replaceWithAdditionalYields(
   AffineForOp newLoop = AffineForOp::create(
       rewriter, getLoc(), getLowerBoundOperands(), getLowerBoundMap(),
       getUpperBoundOperands(), getUpperBoundMap(), getStepAsInt(), inits);
+  newLoop->setDiscardableAttrs(getOperation()->getDiscardableAttrDictionary());
 
   // Generate the new yield values and append them to the scf.yield operation.
   auto yieldOp = cast<AffineYieldOp>(getBody()->getTerminator());
diff --git a/mlir/test/Dialect/Affine/unroll-jam.mlir b/mlir/test/Dialect/Affine/unroll-jam.mlir
index 8ed7fccf7d251..bba24844c3484 100644
--- a/mlir/test/Dialect/Affine/unroll-jam.mlir
+++ b/mlir/test/Dialect/Affine/unroll-jam.mlir
@@ -185,7 +185,7 @@ func.func @unroll_jam_one_iter_arg() {
     %red = affine.for %j = 0 to 17 iter_args(%acc = %cst) -> (i32) {
       %y = "bar"(%i, %j, %acc) : (index, index, i32) -> i32
       affine.yield %y : i32
-    }
+    } {test.keep = "inner"}
     %w = "foo"(%i, %x, %red) : (index, i32, i32) -> i32
   }
   return
@@ -201,7 +201,7 @@ func.func @unroll_jam_one_iter_arg() {
 // CHECK-NEXT:     [[INC1:%[0-9]+]] = affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:     [[RES5:%[0-9]+]] = "bar"([[INC1]], [[IV1]], [[ACC2]])
 // CHECK-NEXT:     affine.yield [[RES4]], [[RES5]]
-// CHECK-NEXT:   }
+// CHECK-NEXT:   } {test.keep = "inner"}
 // CHECK:        "foo"([[IV0]], [[RES1]], [[RES3]]#0)
 // CHECK-NEXT:   affine.apply [[$MAP_PLUS_1]]([[IV0]])
 // CHECK-NEXT:   "foo"({{.*}}, [[RES2]], [[RES3]]#1)


From llvmlistbot at llvm.org  Sun Jul 19 16:51:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 16:51:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a5d62e5.bc28b987.2e0c9a.9032@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210566

>From c69db555ddc028d5aa1c49f94012ab24aac11435 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir] Avoid verifying live user in conversion diagnostic

---
 .../Transforms/Utils/DialectConversion.cpp    |  7 +++++--
 .../test-legalize-erased-op-with-uses.mlir    | 21 ++++++++++++++++++-
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..49986517d26b4 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -3391,8 +3391,11 @@ legalizeUnresolvedMaterialization(RewriterBase &rewriter,
                             << inputOperands.getTypes() << ") to ("
                             << op.getResultTypes()
                             << ") that remained live after conversion";
-  diag.attachNote(op->getUsers().begin()->getLoc())
-      << "see existing live user here: " << *op->getUsers().begin();
+  Operation *liveUser = *op->getUsers().begin();
+  diag.attachNote(liveUser->getLoc())
+      << "see existing live user here: "
+      << OpWithFlags(liveUser,
+                     OpPrintingFlags().assumeVerified().printGenericOpForm());
   return failure();
 }
 
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..829ff2f183e53 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Sun Jul 19 20:37:47 2026
From: llvmlistbot at llvm.org (Longsheng Mou)
Date: Sun, 19 Jul 2026 20:37:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][func] Fix a crash in
 DuplicateFunctionEliminationPass (PR #209667)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209667@github.com>
Message-ID: <6a5d980b.7d702ed2.13d102.5725@mx.google.com>


================
@@ -101,12 +101,12 @@ struct DuplicateFunctionEliminationPass
 
     // Update all symbol uses to reference unique func op
     // representants and erase redundant func ops.
-    SymbolTableCollection symbolTable;
-    SymbolUserMap userMap(symbolTable, module);
     for (auto it : toBeErased) {
       StringAttr oldSymbol = it.getSymNameAttr();
       StringAttr newSymbol = getRepresentant[oldSymbol].getSymNameAttr();
-      userMap.replaceAllUsesWith(it, newSymbol);
----------------
CoTinker wrote:

In this case , when the redundant caller1 is replaced first
```
func.func @callee0() {
  return
}

func.func @caller0() {
  call @callee1() : () -> ()
  return
}

func.func @caller1() {
  call @callee1() : () -> ()
  return
}

func.func @callee1() {
  return
}
```
--->
```
func.func @callee0() {
  return
}

func.func @caller0() {
  call @callee1() : () -> ()
  return
}

func.func @callee1() {
  return
}
```
And then the callee1 replace callee0, but the symbol table is not updated, when the callee1 try to replace callee0 in caller1 fucntion, it will crash, becase the caller1 op is actully erased.

https://github.com/llvm/llvm-project/pull/209667

From llvmlistbot at llvm.org  Sun Jul 19 21:47:06 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Sun, 19 Jul 2026 21:47:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a5da84a.eb165e5c.4506c.eb53@mx.google.com>


https://github.com/nbpatel updated https://github.com/llvm/llvm-project/pull/209822

>From e9401f094f7163276fd348056d310a895236c428 Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 14 Jul 2026 21:43:37 +0000
Subject: [PATCH 1/2] Enhance unrolling of convert layout

---
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 124 ++++++++++++++++--
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   |  32 ++++-
 2 files changed, 137 insertions(+), 19 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..bcb9946373298 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -957,8 +957,92 @@ struct UnrollStoreMatrixOp : public UnrollPattern<xegpu::StoreMatrixOp> {
 /// after inst_data stripped. If it does, it will unroll the vector into
 /// multiple smaller vectors according to the target shape, and create multiple
 /// ConvertLayoutOp with the unrolled vectors and the stripped layouts.
+///
+/// When the input and target layouts have different inst_data, the source is
+/// extracted at the input inst_data granularity and the result is inserted at
+/// the target inst_data granularity, enabling slice cancellation during
+/// canonicalization.
 struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
   using UnrollPattern<xegpu::ConvertLayoutOp>::UnrollPattern;
+
+  /// Extracts source in `inTile` slices, regroups into `convTile`-sized
+  /// ConvertLayoutOps, and inserts the result in `outTile` slices.
+  /// Returns failure if the tiles do not evenly divide.
+  LogicalResult
+  rewriteWithRegrouping(xegpu::ConvertLayoutOp op, VectorType valueTy,
+                        ArrayRef<int64_t> convTile, ArrayRef<int64_t> inTile,
+                        ArrayRef<int64_t> outTile,
+                        xegpu::DistributeLayoutAttr inputLayout,
+                        xegpu::DistributeLayoutAttr targetLayout, Location loc,
+                        PatternRewriter &rewriter) const {
+    ArrayRef<int64_t> vecShape = valueTy.getShape();
+    if (!computeShapeRatio(vecShape, convTile) ||
+        !computeShapeRatio(convTile, inTile) ||
+        !computeShapeRatio(convTile, outTile))
+      return failure();
+
+    Type elemTy = valueTy.getElementType();
+    int64_t rank = valueTy.getRank();
+    VectorType convTy = VectorType::get(convTile, elemTy);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value source = op.getSource();
+    auto zeroOf = [&](VectorType ty) -> Value {
+      return arith::ConstantOp::create(
+          rewriter, loc, ty,
+          DenseElementsAttr::get(ty, rewriter.getZeroAttr(elemTy)));
+    };
+    auto addOffsets = [](ArrayRef<int64_t> a,
+                         ArrayRef<int64_t> b) -> SmallVector<int64_t> {
+      SmallVector<int64_t> res(a);
+      for (auto [r, v] : llvm::zip_equal(res, b))
+        r += v;
+      return res;
+    };
+
+    Value result = zeroOf(valueTy);
+    for (SmallVector<int64_t> convOff :
+         StaticTileOffsetRange(vecShape, convTile)) {
+      // Build the convert tile from inTile-sized slices of the source.
+      Value conv;
+      if (convTile == inTile) {
+        conv = vector::ExtractStridedSliceOp::create(
+            rewriter, loc, source, convOff, convTile, strides);
+      } else {
+        conv = zeroOf(convTy);
+        for (SmallVector<int64_t> inLocal :
+             StaticTileOffsetRange(convTile, inTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, source, addOffsets(convOff, inLocal), inTile,
+              strides);
+          conv = vector::InsertStridedSliceOp::create(rewriter, loc, piece,
+                                                      conv, inLocal, strides);
+        }
+      }
+
+      conv = xegpu::ConvertLayoutOp::create(rewriter, loc, convTy, conv,
+                                            inputLayout, targetLayout);
+
+      // Write the converted tile into the result as outTile-sized slices.
+      if (convTile == outTile) {
+        result = vector::InsertStridedSliceOp::create(rewriter, loc, conv,
+                                                      result, convOff, strides);
+      } else {
+        for (SmallVector<int64_t> outLocal :
+             StaticTileOffsetRange(convTile, outTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, conv, outLocal, outTile, strides);
+          result = vector::InsertStridedSliceOp::create(
+              rewriter, loc, piece, result, addOffsets(convOff, outLocal),
+              strides);
+        }
+      }
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+
   LogicalResult matchAndRewrite(xegpu::ConvertLayoutOp op,
                                 PatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
@@ -974,8 +1058,10 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
       return success();
     }
 
-    if (inputLayout.getEffectiveInstDataAsInt().empty() ||
-        targetLayout.getEffectiveInstDataAsInt().empty())
+    // Capture inst_data granularities before stripping them.
+    SmallVector<int64_t> inTile = inputLayout.getEffectiveInstDataAsInt();
+    SmallVector<int64_t> outTile = targetLayout.getEffectiveInstDataAsInt();
+    if (inTile.empty() || outTile.empty())
       return rewriter.notifyMatchFailure(op, "Not a target ConvertLayoutOp.");
 
     inputLayout = inputLayout.dropInstData();
@@ -988,20 +1074,30 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     if (!targetShape || targetShape->size() != (size_t)valueTy.getRank())
       return failure();
 
-    Value newSource = op.getSource();
+    // Nothing to convert if layouts match after stripping inst_data.
+    if (!inputLayout || !targetLayout || inputLayout.isEqualTo(targetLayout)) {
+      rewriter.replaceOp(op, op.getSource());
+      return success();
+    }
+
+    // Try regrouping: extract at inTile, convert, insert at outTile.
+    if (succeeded(rewriteWithRegrouping(op, valueTy, *targetShape, inTile,
+                                        outTile, inputLayout, targetLayout, loc,
+                                        rewriter)))
+      return success();
+
+    // Fallback: pack/unpack at the convert tile granularity.
+    SmallVector<Type> convertedValTypes =
+        getUnrolledTypes(valueTy, *targetShape);
+    SmallVector<Value> convertedValues =
+        pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
     SmallVector<Value> newOps;
-    if (inputLayout && targetLayout && !inputLayout.isEqualTo(targetLayout)) {
-      SmallVector<Type> convertedValTypes =
-          getUnrolledTypes(valueTy, *targetShape);
-      SmallVector<Value> convertedValues =
-          pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
-      for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
-        auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
-                                                    inputLayout, targetLayout);
-        newOps.push_back(newOp);
-      }
-      newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
+    for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
+      auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
+                                                  inputLayout, targetLayout);
+      newOps.push_back(newOp);
     }
+    Value newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
 
     rewriter.replaceOp(op, newSource);
     return success();
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..27a88efcf443c 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -463,19 +463,19 @@ gpu.module @test_kernel {
   //CHECK: gpu.func @convert_layout([[arg0:%.+]]: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
   //CHECK: [[cst:%.+]] = arith.constant dense<0.000000e+00> : vector<8x32x2xf16>
   //CHECK: [[e0:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [0, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
-  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
   //CHECK: [[c0:%.+]] = xegpu.convert_layout [[e0]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
-  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
   //CHECK: [[e2:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
+  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
+  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m0:%.+]] = math.exp [[e2]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i0:%.+]] = vector.insert_strided_slice [[m0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m1:%.+]] = math.exp [[e3]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i1:%.+]] = vector.insert_strided_slice [[m1]], [[i0]] {offsets = [0, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m2:%.+]] = math.exp [[e4]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i2:%.+]] = vector.insert_strided_slice [[m2]], [[i1]] {offsets = [4, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m3:%.+]] = math.exp [[e5]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i3:%.+]] = vector.insert_strided_slice [[m3]], [[i2]] {offsets = [4, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
   //CHECK: gpu.return [[i3]] : vector<8x32x2xf16>
@@ -488,6 +488,28 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 16] vs [8, 64]).
+#in = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#tgt = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 16]
+  //CHECK: math.exp {{.*}} : vector<8x16xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 64]
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<8x64xbf16>
+  //CHECK: xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x64xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix

>From 41c298a78f70f3c2727bae352876565063aa5fde Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 20 Jul 2026 04:46:26 +0000
Subject: [PATCH 2/2] Add test

---
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index 27a88efcf443c..bc50bad2b2302 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -510,6 +510,27 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 64] vs [8, 16]).
+#in = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+#tgt = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup_swapped
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 64], strides = [1, 1]} : vector<16x64xbf16> to vector<8x64xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 16]
+  //CHECK: [[cvt:%.+]] = xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.extract_strided_slice [[cvt]] {{.*}}sizes = [8, 16]{{.*}} : vector<8x64xbf16> to vector<8x16xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup_swapped(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix


From llvmlistbot at llvm.org  Sun Jul 19 22:48:16 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Sun, 19 Jul 2026 22:48:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5db6a0.1728eaf3.2a2f6.5c04@mx.google.com>


https://github.com/gysit approved this pull request.

Thanks for the experiment.

The test LGTM.

I would keep https://github.com/llvm/llvm-project/issues/206713 open since the ticket also mentions global variables / global variable expressions. A comment on the ticket that the local variable roundtrip currently works as expected makes sense though.
  
Did you try to import the example with the self recursive local variable (without your extra change)? If yes, how did the imported IR look in this case?

https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Sun Jul 19 23:05:34 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Sun, 19 Jul 2026 23:05:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5dbaae.b400e095.8859c.eb4f@mx.google.com>


hamzaqureshi5 wrote:

@ftynse PR is ready for review

https://github.com/llvm/llvm-project/pull/210533

From llvmlistbot at llvm.org  Sun Jul 19 23:05:41 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Sun, 19 Jul 2026 23:05:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Fix fusion of producers that
 update a memref in place (PR #210533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210533@github.com>
Message-ID: <6a5dbab5.824e11a6.9c376.3656@mx.google.com>


https://github.com/hamzaqureshi5 updated https://github.com/llvm/llvm-project/pull/210533

>From 734a375ee85a0b8968131727ae1e7c1b02d7e032 Mon Sep 17 00:00:00 2001
From: Hamza Qureshi <hamza7771.861 at gmail.com>
Date: Sat, 18 Jul 2026 23:17:43 +0500
Subject: [PATCH] [MLIR][Affine] Fix fusion of producers that update a memref
 in place

Affine producer-consumer fusion places the fused nest at an insertion
point between the source and the destination nest. When the source nest
is retained after fusion (e.g., non-maximal fusion with an escaping
memref) and updates a memref in place (loads from and stores to it),
this placement is unsound: the fused nest recomputes the source's
values reading from that memref, but the retained source nest has
already overwritten it, so the source's computation is applied twice.

Fix this by placing the fused nest *before* the retained source nest,
where the in-place updated memrefs still hold their original values.
The recomputed stores are redirected to a private memref, so the
retained source nest also still reads the original values. An
anti-dependence edge from the destination to the source node is added
to the dependence graph to preserve this placement in subsequent fusion
decisions. When no such placement can be shown to preserve all
dependences, or the in-place updated memref can't be privatized, bail
out on the fusion candidate.

Loads that only read values stored earlier in the same iteration (e.g.,
initialize-then-reuse patterns) don't observe the memref's incoming
state and don't require the new placement.

Update @same_memref_load_store and @same_memref_load_multiple_stores,
whose CHECK lines encoded the miscompiled output; their fused nests are
unchanged but now precede the retained source nests.

Fixes: https://github.com/llvm/llvm-project/issues/210490
---
 .../Dialect/Affine/Transforms/LoopFusion.cpp  | 140 ++++++++++++++++++
 mlir/test/Dialect/Affine/loop-fusion-4.mlir   |  87 ++++++++++-
 2 files changed, 221 insertions(+), 6 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..5ec6ef7998431 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -124,6 +124,107 @@ static bool canRemoveSrcNodeAfterFusion(
   return true;
 }
 
+/// Returns true if `node`'s loads of `memref` may observe the memory state
+/// of `memref` from before the node's execution. This doesn't hold when
+/// every load of `memref` is dominated by a store to the same location
+/// within the same iteration (e.g., an initialize-then-reuse pattern): all
+/// such loads only read values produced by the node itself.
+static bool mayReadIncomingMemState(const MemRefDependenceGraph::Node &node,
+                                    Value memref,
+                                    const DominanceInfo &domInfo) {
+  for (Operation *loadOp : node.loads) {
+    if (cast<AffineReadOpInterface>(loadOp).getMemRef() != memref)
+      continue;
+    MemRefAccess loadAccess(loadOp);
+    bool coveredByStore = llvm::any_of(node.stores, [&](Operation *storeOp) {
+      return cast<AffineWriteOpInterface>(storeOp).getMemRef() == memref &&
+             domInfo.properlyDominates(storeOp, loadOp) &&
+             MemRefAccess(storeOp) == loadAccess;
+    });
+    if (!coveredByStore)
+      return true;
+  }
+  return false;
+}
+
+/// Returns true if the fused loop nest can be placed just before the source
+/// nest `srcId` while preserving all dependences. This placement is needed
+/// when the source nest is retained after fusion and updates the memrefs in
+/// `inPlaceUpdatedMemrefs` in place (loads from and stores to them): the
+/// fused nest recomputes the source's values reading from these memrefs,
+/// which is only sound while they still hold their original values, i.e.,
+/// before the retained source nest overwrites them. See
+/// https://github.com/llvm/llvm-project/issues/210490.
+static bool
+canPlaceFusedNestBeforeSrcNode(const MemRefDependenceGraph &mdg, unsigned srcId,
+                               unsigned dstId,
+                               ArrayRef<Value> inPlaceUpdatedMemrefs,
+                               const DenseSet<Value> &privateMemrefs) {
+  const MemRefDependenceGraph::Node *srcNode = mdg.getNode(srcId);
+  const MemRefDependenceGraph::Node *dstNode = mdg.getNode(dstId);
+  Operation *srcOp = srcNode->op;
+  Operation *dstOp = dstNode->op;
+
+  // The recomputed stores to an in-place updated memref must be redirected to
+  // a private memref; otherwise they would clobber values the retained source
+  // nest still has to read. Private memref creation happens only after
+  // fusion, so check its requirements here.
+  for (Value memref : inPlaceUpdatedMemrefs) {
+    // The memref must have been selected for privatization.
+    if (!privateMemrefs.contains(memref))
+      return false;
+    // Stores to the memref in the destination nest would in general prevent
+    // private memref creation as they don't share the source stores' access
+    // function.
+    if (dstNode->getStoreOpCount(memref) > 0)
+      return false;
+    // Private memref creation requires all stores to have the same access
+    // function.
+    SmallVector<Operation *, 4> storeOps;
+    srcNode->getStoreOpsForMemref(memref, &storeOps);
+    if (storeOps.size() > 1 &&
+        !std::equal(std::next(storeOps.begin()), storeOps.end(),
+                    storeOps.begin(), [](Operation *a, Operation *b) {
+                      return MemRefAccess(cast<AffineWriteOpInterface>(a)) ==
+                             MemRefAccess(cast<AffineWriteOpInterface>(b));
+                    }))
+      return false;
+  }
+
+  // All dependences from the source nest to the destination nest must be
+  // satisfied by the privatized recomputation; any other dependence would be
+  // violated by the new placement.
+  for (const auto &edge : mdg.outEdges.lookup(srcId))
+    if (edge.id == dstId && !privateMemrefs.contains(edge.value))
+      return false;
+
+  // The new placement makes the fused nest cross all graph nodes between the
+  // source and the destination nest; conservatively require that there are
+  // none.
+  for (const auto &idAndNode : mdg.nodes) {
+    Operation *op = idAndNode.second.op;
+    if (op->getBlock() == srcOp->getBlock() && srcOp->isBeforeInBlock(op) &&
+        op->isBeforeInBlock(dstOp))
+      return false;
+  }
+
+  // All SSA values used in the destination nest must dominate the source nest
+  // for the destination nest to be placed before it.
+  DominanceInfo domInfo;
+  WalkResult walkResult = dstOp->walk([&](Operation *op) {
+    for (Value operand : op->getOperands()) {
+      Operation *defOp = operand.getDefiningOp();
+      // Block arguments and values defined within the destination nest remain
+      // available; any other value must dominate the source nest.
+      if (defOp && !dstOp->isAncestor(defOp) &&
+          !domInfo.properlyDominates(defOp, srcOp))
+        return WalkResult::interrupt();
+    }
+    return WalkResult::advance();
+  });
+  return !walkResult.wasInterrupted();
+}
+
 /// Returns in 'srcIdCandidates' the producer fusion candidates for consumer
 /// 'dstId'. Candidates are sorted by node id order. This order corresponds to
 /// the program order when the 'mdg' is created. However, program order is not
@@ -1086,6 +1187,38 @@ struct GreedyFusion {
           }
         }
 
+        // Collect memrefs the source nest updates in place: memrefs it
+        // stores to while also reading their incoming values. If the source
+        // nest is retained after fusion, the fused nest has to be placed
+        // before it: the fused nest recomputes the source's values reading
+        // from these memrefs, which is only sound while they still hold
+        // their original values, i.e., before the retained source nest
+        // overwrites them (https://github.com/llvm/llvm-project/issues/210490).
+        // Bail out on this candidate if no legal such placement exists.
+        SmallVector<Value, 2> inPlaceUpdatedMemrefs;
+        if (!removeSrcNode) {
+          DominanceInfo domInfo;
+          for (Operation *store : srcNode->stores) {
+            Value memref = cast<AffineWriteOpInterface>(store).getMemRef();
+            if (srcNode->getLoadOpCount(memref) > 0 &&
+                !llvm::is_contained(inPlaceUpdatedMemrefs, memref) &&
+                mayReadIncomingMemState(*srcNode, memref, domInfo))
+              inPlaceUpdatedMemrefs.push_back(memref);
+          }
+        }
+        if (!inPlaceUpdatedMemrefs.empty()) {
+          if (!canPlaceFusedNestBeforeSrcNode(
+                  *mdg, srcId, dstId, inPlaceUpdatedMemrefs, privateMemrefs)) {
+            LDBG() << "Can't fuse: the source nest updates memref(s) in "
+                      "place, is retained after fusion, and the fused nest "
+                      "can't be placed before it";
+            continue;
+          }
+          // Place the fused nest right before the retained source nest so
+          // that it reads the in-place updated memrefs' original values.
+          fusedLoopInsPoint = srcNode->op;
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1102,6 +1235,13 @@ struct GreedyFusion {
         mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
                          removeSrcNode);
 
+        // If the fused nest was placed before the retained source nest, it
+        // reads the original values of the memrefs the source nest updates
+        // in place: record these anti-dependences to preserve the placement
+        // in subsequent fusion decisions.
+        for (Value memref : inPlaceUpdatedMemrefs)
+          mdg->addEdge(dstId, srcId, memref);
+
         // Create private memrefs.
         if (!privateMemrefs.empty()) {
           // Note the block into which fusion was performed. This can be used to
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..0b63b83d21d73 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -296,8 +296,6 @@ module {
 // PRODUCER-CONSUMER-LABEL: func @same_memref_load_store
 func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Source isn't removed.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -308,6 +306,9 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
     %2 = arith.addf %0, %cst : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed; the fused
+  // nest is placed before it so that its recomputation of the source's
+  // values reads %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -317,6 +318,12 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
   // PRODUCER-CONSUMER-NEXT:   arith.addf
   // PRODUCER-CONSUMER-NEXT:   affine.store
   // PRODUCER-CONSUMER-NEXT: }
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
   return
 }
 
@@ -326,10 +333,6 @@ func.func @same_memref_load_store(%producer : memref<32xf32>, %consumer: memref<
 // ALL-LABEL: func @same_memref_load_multiple_stores
 func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %producer_2 : memref<32xf32>, %consumer: memref<16xf32>){
   %cst = arith.constant 2.000000e+00 : f32
-  // Ensure that source isn't removed during both producer-consumer fusion and
-  // sibling fusion.
-  // PRODUCER-CONSUMER: affine.for %{{.*}} = 0 to 32
-  // ALL: affine.for %{{.*}} = 0 to 32
   affine.for %arg3 = 0 to 32 {
     %0 = affine.load %producer[%arg3] : memref<32xf32>
     %2 = arith.mulf %0, %cst : f32
@@ -342,6 +345,10 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
     %2 = arith.addf %0, %1 : f32
     affine.store %2, %consumer[%arg3] : memref<16xf32>
   }
+  // The source nest updates %producer in place and isn't removed during both
+  // producer-consumer fusion and sibling fusion; the fused nest is placed
+  // before it so that its recomputation of the source's values reads
+  // %producer's original values (issue #210490).
   // Fused nest.
   // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 16
   // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
@@ -356,6 +363,74 @@ func.func @same_memref_load_multiple_stores(%producer : memref<32xf32>, %produce
   // ALL:     affine.for %{{.*}} = 0 to 16
   // ALL:       mulf
   // ALL:       addf
+  // Source isn't removed, and runs after the fused nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 32
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<32xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // ALL:     affine.for %{{.*}} = 0 to 32
+  return
+}
+
+// -----
+
+// Reproducer for https://github.com/llvm/llvm-project/issues/210490: the
+// producer nest updates %p in place and is retained after fusion since the
+// fusion isn't maximal and %p escapes. The fused nest must be placed before
+// the retained producer nest so that its recomputation of the producer's
+// values reads %p's original values; placing it after would apply the
+// multiplication twice on the consumed prefix of %p.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_prefix_consumer
+func.func @in_place_producer_prefix_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+  }
+  // Fused nest, placed before the retained producer nest.
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[0] : memref<1xf32>
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<2xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER-NEXT:   affine.load %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT:   arith.mulf
+  // PRODUCER-CONSUMER-NEXT:   affine.store %{{.*}}, %{{.*}}[%{{.*}}] : memref<4xf32>
+  // PRODUCER-CONSUMER-NEXT: }
+  return
+}
+
+// -----
+
+// The consumer nest also stores to the in-place updated memref %p: %p can't
+// be privatized in the fused nest, so there is no legal placement for the
+// fused nest before the retained producer nest. Fusion must not happen.
+// PRODUCER-CONSUMER-LABEL: func @in_place_producer_storing_consumer
+func.func @in_place_producer_storing_consumer(%p: memref<4xf32>, %d: memref<2xf32>, %c: f32) {
+  affine.for %i = 0 to 4 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    %1 = arith.mulf %0, %c : f32
+    affine.store %1, %p[%i] : memref<4xf32>
+  }
+  affine.for %i = 0 to 2 {
+    %0 = affine.load %p[%i] : memref<4xf32>
+    affine.store %0, %d[%i] : memref<2xf32>
+    affine.store %c, %p[%i] : memref<4xf32>
+  }
+  // PRODUCER-CONSUMER-NOT:  memref.alloc
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 4
+  // PRODUCER-CONSUMER:        arith.mulf
+  // PRODUCER-CONSUMER:      affine.for %{{.*}} = 0 to 2
+  // PRODUCER-CONSUMER-NOT:    arith.mulf
   return
 }
 


From llvmlistbot at llvm.org  Sun Jul 19 23:52:54 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 19 Jul 2026 23:52:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] c34d2c3 - [mlir] Fix mem2reg crash on
 scalable vector store/load with matching type (#209426)
Message-ID: <6a5dc5c6.00bedb31.19c55.dec3@mx.google.com>


Author: Aayush Shrivastava
Date: 2026-07-20T08:52:49+02:00
New Revision: c34d2c30e86358bfe308558b5e0719e9f5305153

URL: https://github.com/llvm/llvm-project/commit/c34d2c30e86358bfe308558b5e0719e9f5305153
DIFF: https://github.com/llvm/llvm-project/commit/c34d2c30e86358bfe308558b5e0719e9f5305153.diff

LOG: [mlir] Fix mem2reg crash on scalable vector store/load with matching type (#209426)

Fixes #209065

`createInsertAndCast/createExtractAndCast` in `LLVMMemorySlot.cpp`
queried the bit size of the source/target types before checking whether
they were already identical. For scalable vector types (e.g.
`vector<[4]xi1>`), this size query implicitly converts a scalable
TypeSize to a scalar, which aborts. `mem2reg` hits this path for any
store or load whose value type exactly matches the slot's element type,
since `getStored/removeBlockingUses` run even when no cast is needed.

This fix adds an early return when the types are already equal, skipping
the size query, consistent with the existing short-circuit in
`areConversionCompatible/castSameSizedTypes`.

Added: 
    

Modified: 
    mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
    mlir/test/Dialect/LLVMIR/mem2reg.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
index 66d6a592a93df..2416e01a67955 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/LLVMMemorySlot.cpp
@@ -256,6 +256,11 @@ static Value createExtractAndCast(OpBuilder &builder, Location loc,
                                  /*narrowingConversion=*/true) &&
          "expected that the compatibility was checked before");
 
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of scalable vector types below.
+  if (srcType == targetType)
+    return srcValue;
+
   uint64_t srcTypeSize = dataLayout.getTypeSizeInBits(srcType);
   uint64_t targetTypeSize = dataLayout.getTypeSizeInBits(targetType);
   if (srcTypeSize == targetTypeSize)
@@ -291,6 +296,12 @@ static Value createInsertAndCast(OpBuilder &builder, Location loc,
                                  srcValue.getType(),
                                  /*narrowingConversion=*/false) &&
          "expected that the compatibility was checked before");
+
+  // Nothing has to be done if the types are already the same. This also
+  // avoids querying the bit size of scalable vector types below.
+  if (srcValue.getType() == reachingDef.getType())
+    return srcValue;
+
   uint64_t valueTypeSize = dataLayout.getTypeSizeInBits(srcValue.getType());
   uint64_t slotTypeSize = dataLayout.getTypeSizeInBits(reachingDef.getType());
   if (slotTypeSize == valueTypeSize)

diff  --git a/mlir/test/Dialect/LLVMIR/mem2reg.mlir b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
index cabbd1c1bc013..5f1029625b9b4 100644
--- a/mlir/test/Dialect/LLVMIR/mem2reg.mlir
+++ b/mlir/test/Dialect/LLVMIR/mem2reg.mlir
@@ -1035,6 +1035,20 @@ llvm.func @scalable_llvm_vector() -> i16 {
 
 // -----
 
+// CHECK-LABEL: @scalable_vector_matching_store_load
+// CHECK-SAME: %[[ARG:.+]]: vector<[4]xi1>
+llvm.func @scalable_vector_matching_store_load(%arg: vector<[4]xi1>) -> vector<[4]xi1> {
+  %0 = llvm.mlir.constant(1 : i32) : i32
+  // CHECK-NOT: llvm.alloca
+  %1 = llvm.alloca %0 x vector<[4]xi1> : (i32) -> !llvm.ptr
+  llvm.store %arg, %1 : vector<[4]xi1>, !llvm.ptr
+  %2 = llvm.load %1 : !llvm.ptr -> vector<[4]xi1>
+  // CHECK: llvm.return %[[ARG]] : vector<[4]xi1>
+  llvm.return %2 : vector<[4]xi1>
+}
+
+// -----
+
 // CHECK-LABEL: @smaller_store_forwarding
 // CHECK-SAME: %[[ARG:.+]]: i16
 llvm.func @smaller_store_forwarding(%arg : i16) {


        

From llvmlistbot at llvm.org  Sun Jul 19 23:52:57 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Sun, 19 Jul 2026 23:52:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5dc5c9.c20dae4b.209f3a.ad78@mx.google.com>


https://github.com/gysit closed https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Sun Jul 19 23:53:16 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Sun, 19 Jul 2026 23:53:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix mem2reg crash on scalable vector
 store/load with matching type (PR #209426)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209426@github.com>
Message-ID: <6a5dc5dc.e624eb8d.4a8a7.d9c7@mx.google.com>


gysit wrote:

Ok, merging this one then! 

https://github.com/llvm/llvm-project/pull/209426

From llvmlistbot at llvm.org  Mon Jul 20 00:28:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 00:28:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] QuantileType relax quantileType conditions
 and inheritance issue (PR #204793)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204793@github.com>
Message-ID: <6a5dce22.449bfb44.7b862.c982@mx.google.com>


vsimion26 wrote:

@javedabsar1 gentle ping on PR. Please review when you have time

https://github.com/llvm/llvm-project/pull/204793

From llvmlistbot at llvm.org  Mon Jul 20 01:34:08 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Mon, 20 Jul 2026 01:34:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5ddd80.27b3b003.1f84cc.0db4@mx.google.com>


lhutton1 wrote:

cc @sahas3 @udaya-ranga @Tai78641 

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Mon Jul 20 01:47:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 01:47:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][NVVM] Diagnose non-NVVM target ops
 before serialization (PR #210000)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210000@github.com>
Message-ID: <6a5de091.13be6df8.22eac4.4ec3@mx.google.com>


shubhamnarlawar wrote:

PING

https://github.com/llvm/llvm-project/pull/210000

From llvmlistbot at llvm.org  Mon Jul 20 01:50:19 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 20 Jul 2026 01:50:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5de14b.0c05e68e.3c2f5d.4264@mx.google.com>


https://github.com/IgWod edited https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 01:50:19 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 20 Jul 2026 01:50:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5de14b.1e1ac41d.3969c3.eb45@mx.google.com>


https://github.com/IgWod commented:

Looks good in general, just few comments on the test.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 01:50:19 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 20 Jul 2026 01:50:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5de14b.c4d3d80a.849c3.1902@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
----------------
IgWod wrote:

Would it make more sense to return `%0`, so that the yielded value is being used?

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 01:50:20 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 20 Jul 2026 01:50:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5de14c.4a342cbc.3a2f36.2516@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
----------------
IgWod wrote:

Same comment as in the other test.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 01:50:20 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Mon, 20 Jul 2026 01:50:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5de14c.0c05e68e.3c2f5d.432e@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
----------------
IgWod wrote:

Should it explicitly check for 2 values so it matches `// CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)`?

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 02:27:36 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Mon, 20 Jul 2026 02:27:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5dea08.6c5720b3.2da3b9.6818@mx.google.com>


hockyy wrote:

Logic looks good to me. Don't forget to update docs if there's any. It seems like this is changing the definition of a valid IR

https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Mon Jul 20 02:58:37 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 20 Jul 2026 02:58:37 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a5df14d.bd93e731.30f155.00da@mx.google.com>


https://github.com/skatrak approved this pull request.

Thank you Jan, LGTM.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Mon Jul 20 02:58:37 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 20 Jul 2026 02:58:37 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a5df14d.d76d81af.23b347.6b15@mx.google.com>


https://github.com/skatrak edited https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Mon Jul 20 02:58:38 2026
From: llvmlistbot at llvm.org (Sergio Afonso)
Date: Mon, 20 Jul 2026 02:58:38 -0700 (PDT)
Subject: [Mlir-commits] [clang] [mlir] [CIR][OpenMP] Add host op filtering
 pass to CIR pipeline (PR #209592)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209592@github.com>
Message-ID: <6a5df14e.824e11a6.9c376.c710@mx.google.com>


================
@@ -5130,22 +5130,25 @@ std::unique_ptr<mlir::Pass> createConvertCIRToLLVMPass() {
   return std::make_unique<ConvertCIRToLLVMPass>();
 }
 
-void populateCIRToLLVMPasses(mlir::OpPassManager &pm) {
+void populateCIRToLLVMPasses(mlir::OpPassManager &pm, bool enableOpenMP) {
   mlir::populateCIRPreLoweringPasses(pm);
-  pm.addPass(mlir::omp::createMarkDeclareTargetPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createMarkDeclareTargetPass());
   pm.addPass(createConvertCIRToLLVMPass());
+  if (enableOpenMP)
+    pm.addPass(mlir::omp::createHostOpFilteringPass());
 }
 
 std::unique_ptr<llvm::Module>
 lowerDirectlyFromCIRToLLVMIR(mlir::ModuleOp mlirModule, LLVMContext &llvmCtx,
----------------
skatrak wrote:

Nit: The `enableOpenMP` flag is obtained from the already present `mlirModule`. I'd suggest removing the new function argument, but it's not a blocker for me.

https://github.com/llvm/llvm-project/pull/209592

From llvmlistbot at llvm.org  Mon Jul 20 03:14:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 03:14:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 195df3d - [mlir][OpenACC] Privatize worker
 reduction accumulators per worker (#210456)
Message-ID: <6a5df4fe.d76d81af.23b347.7642@mx.google.com>


Author: Matsu
Date: 2026-07-20T03:14:16-07:00
New Revision: 195df3d4654b43ecf35927e574a8061bfaecbb78

URL: https://github.com/llvm/llvm-project/commit/195df3d4654b43ecf35927e574a8061bfaecbb78
DIFF: https://github.com/llvm/llvm-project/commit/195df3d4654b43ecf35927e574a8061bfaecbb78.diff

LOG: [mlir][OpenACC] Privatize worker reduction accumulators per worker (#210456)

Example:
```fortran
!$acc parallel
!$acc loop gang reduction(+:sum)
do k = 1, p
  !$acc loop worker reduction(+:sum)
  do j = 1, n
    !$acc loop vector reduction(+:sum)
    do i = 1, m
      sum = sum + a(i,j,k)
    end do
  end do
end do
```

In this code, the worker accumulator is both gang- and worker-scoped.
Shared
memory allocated only one copy, so workers raced while updating the same
value.

Fix: allocate one shared-memory accumulator per worker whenever worker
parallelism is present.

Added: 
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir

Modified: 
    mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
    mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index cd2e732dbc817..5f4af6a5bcc5b 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -319,8 +319,8 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
 
 static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
     PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
-    bool isWorkerPrivate, bool isGangPrivate, OpenACCSupport *support) {
-  if (!isWorkerPrivate || isGangPrivate)
+    bool isWorkerPrivate, OpenACCSupport *support) {
+  if (!isWorkerPrivate)
     return std::optional<int64_t>(1);
 
   GPUParallelDimAttr threadY =
@@ -394,8 +394,8 @@ FailureOr<bool> isPrivateLocalSharedMemoryCandidate(
     return false;
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(
-          privateLocal, computeRegion, isWorkerPrivate, isGangPrivate, support);
+      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
+                                            isWorkerPrivate, support);
   if (failed(numCopies))
     return failure();
   return numCopies->has_value();
@@ -411,15 +411,12 @@ std::optional<int64_t> getPrivateLocalSharedMemoryUpperBoundBytes(
 
   SmallVector<GPUParallelDimAttr> parDims =
       collectPrivateLocalParDims(privateLocal, computeRegion);
-  bool isGangPrivate =
-      llvm::any_of(parDims, [&](auto parDim) { return policy.isGang(parDim); });
   bool isWorkerPrivate = llvm::any_of(
       parDims, [&](auto parDim) { return policy.isWorker(parDim); });
 
   FailureOr<std::optional<int64_t>> numCopies =
-      getWorkerPrivateSharedMemoryNumCopies(privateLocal, computeRegion,
-                                            isWorkerPrivate, isGangPrivate,
-                                            /*support=*/nullptr);
+      getWorkerPrivateSharedMemoryNumCopies(
+          privateLocal, computeRegion, isWorkerPrivate, /*support=*/nullptr);
   if (failed(numCopies) || !numCopies->has_value())
     return std::nullopt;
 

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
new file mode 100644
index 0000000000000..3866c8057f129
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-private.mlir
@@ -0,0 +1,41 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// CHECK-LABEL: func.func @worker_reduction_private
+// CHECK: acc.gpu_shared_memory
+// CHECK-SAME: num_copies = 4
+
+func.func @worker_reduction_private() {
+  %c2 = arith.constant 2 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block = acc.par_width %c2 {par_dim = #acc.par_dim<block_x>}
+  %worker = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %vector = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>] : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%bx = %block, %wy = %worker, %vx = %vector)
+        ins(%private_arg = %private) : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1 = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%bx) step (%c1) {
+        scf.parallel (%k) = (%block_iv) to (%bx) step (%bx) {
+          %local = acc.private_local %private_arg
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+              : (!acc.private_type<memref<i32>>) -> memref<i32>
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.parallel (%worker_iv) = (%c0) to (%wy) step (%c1) {
+            scf.reduce
+          } {acc.par_dims = #acc<par_dims[thread_y]>}
+          %value = memref.load %local[] : memref<i32>
+          acc.reduction_accumulate %value to %local <add>
+              : i32 -> memref<i32> {par_dims = #acc<par_dims[thread_y]>}
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[sequential]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

diff  --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 66e500c4d2100..69f529abf6fa6 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -42,11 +42,13 @@ class OpenACCUtilsCGTest : public ::testing::Test {
   static ComputeRegionOp buildComputeRegionWithPrivateLocal(
       MLIRContext &context, OpBuilder &b, Location loc, ModuleOp module,
       GPUParallelDimsAttr privatizeParDims, ValueRange launchArgs,
-      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut) {
+      PrivateLocalOp &privateLocalOut, PrivatizeOp &privatizeOut,
+      MemRefType memTy = {}, bool addReductionAccumulator = false) {
     IRRewriter rewriter(&context);
     rewriter.setInsertionPointToStart(module.getBody());
 
-    MemRefType memTy = MemRefType::get({4}, b.getI32Type());
+    if (!memTy)
+      memTy = MemRefType::get({4}, b.getI32Type());
     Type privateTy = PrivateType::get(&context, memTy);
     privatizeOut = PrivatizeOp::create(rewriter, loc, privateTy, ValueRange{},
                                        privatizeParDims);
@@ -71,7 +73,20 @@ class OpenACCUtilsCGTest : public ::testing::Test {
       setParDimsAttr(par, GPUParallelDimsAttr::get(&context, {parDim}));
       srcBuilder.setInsertionPoint(par.getBody()->getTerminator());
     }
-    PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    PrivateLocalOp privateLocal =
+        PrivateLocalOp::create(srcBuilder, loc, memTy, privArg);
+    if (addReductionAccumulator) {
+      Value partial =
+          arith::ConstantIntOp::create(srcBuilder, loc, b.getI32Type(), 1);
+      SmallVector<GPUParallelDimAttr> reductionDims;
+      for (GPUParallelDimAttr parDim : privatizeParDims.getArray())
+        if (!parDim.isAnyBlock())
+          reductionDims.push_back(parDim);
+      ReductionAccumulateOp::create(
+          srcBuilder, loc, partial, privateLocal.getResult(),
+          ReductionOperator::AccAdd,
+          GPUParallelDimsAttr::get(&context, reductionDims));
+    }
 
     IRMapping mapping;
     auto cr = buildComputeRegion(
@@ -748,6 +763,35 @@ TEST_F(OpenACCUtilsCGTest,
   EXPECT_TRUE(*isCandidate);
 }
 
+TEST_F(OpenACCUtilsCGTest, getSharedMemoryBytesGangWorkerReductionAccumulator) {
+  OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
+  b.setInsertionPointToStart(module->getBody());
+  GPUParallelDimsAttr gangWorkerDims = GPUParallelDimsAttr::get(
+      &context, {GPUParallelDimAttr::blockXDim(&context),
+                 GPUParallelDimAttr::threadYDim(&context)});
+  auto c2 = arith::ConstantIndexOp::create(b, loc, 2);
+  auto c4 = arith::ConstantIndexOp::create(b, loc, 4);
+  auto bx =
+      ParWidthOp::create(b, loc, c2, GPUParallelDimAttr::blockXDim(&context));
+  auto ty =
+      ParWidthOp::create(b, loc, c4, GPUParallelDimAttr::threadYDim(&context));
+
+  PrivateLocalOp privateLocal;
+  PrivatizeOp privatize;
+  MemRefType scalarTy = MemRefType::get({}, b.getI32Type());
+  auto cr = buildComputeRegionWithPrivateLocal(
+      context, b, loc, *module, gangWorkerDims,
+      ValueRange{bx.getResult(), ty.getResult()}, privateLocal, privatize,
+      scalarTy, /*addReductionAccumulator=*/true);
+
+  DefaultACCToGPUMappingPolicy policy;
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 16);
+}
+
 TEST_F(OpenACCUtilsCGTest,
        isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);


        

From llvmlistbot at llvm.org  Mon Jul 20 03:14:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 03:14:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Privatize worker reduction
 accumulators per worker (PR #210456)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210456@github.com>
Message-ID: <6a5df501.9edb5c9d.280be3.cb8e@mx.google.com>


https://github.com/khaki3 closed https://github.com/llvm/llvm-project/pull/210456

From llvmlistbot at llvm.org  Mon Jul 20 03:18:08 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Mon, 20 Jul 2026 03:18:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208240@github.com>
Message-ID: <6a5df5e0.92ea0e2a.4a9a5.b3e5@mx.google.com>


================
@@ -448,6 +452,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   /// Output stream to write DOT file to.
   raw_indented_ostream os;
+  /// Re-usable assembly printer state for efficient printing. Initialized
+  /// on each pass run.
+  std::optional<AsmState> asmState;
----------------
jpienaar wrote:

I was wondering about just making this a non-optional class member. I don't think it changes anything wrt this pass: below we already have a map from Value to Node, also having the map (via AsmState) of Value to textual would be consistent with the existing. And if this actually needs to be reset, then those below probably also needs a reset.

https://github.com/llvm/llvm-project/pull/208240

From llvmlistbot at llvm.org  Mon Jul 20 03:42:52 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 20 Jul 2026 03:42:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [ViewOpGraph] Re-use `AsmState` to
 significantly speed-up printing. (PR #208240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208240@github.com>
Message-ID: <6a5dfbac.507fc713.239434.3e72@mx.google.com>


================
@@ -448,6 +452,9 @@ class PrintOpPass : public impl::ViewOpGraphPassBase<PrintOpPass> {
 
   /// Output stream to write DOT file to.
   raw_indented_ostream os;
+  /// Re-usable assembly printer state for efficient printing. Initialized
+  /// on each pass run.
+  std::optional<AsmState> asmState;
----------------
joker-eph wrote:

I would ask instead: why are these members instead of local variables inside the runOnOperation?

It looks to me this is done to pass state around between methods. However I would rather see this implemented as a separate class than the pass: a `class OpGraphPrinter` with all the state and the pass and `runOnOperation()` being just a thin wrapper instantiating `OpGraphPrinter` and invoking it.

https://github.com/llvm/llvm-project/pull/208240

From llvmlistbot at llvm.org  Mon Jul 20 04:06:27 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 20 Jul 2026 04:06:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5e0133.9edb5c9d.280be3.c076@mx.google.com>


joker-eph wrote:

> useful for cases such as being able to programmatically update the result segment sizes or clone an operation via OperationState while editing just a few results, programmatically

Can you elaborate with examples? It's not totally clear to me right now.

https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Mon Jul 20 04:07:19 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 20 Jul 2026 04:07:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5e0167.f3151ff8.37ac76.bb9b@mx.google.com>


joker-eph wrote:

Oh I looked at the linked PR, I get it now, LGTM then.

https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Mon Jul 20 04:07:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 04:07:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9428725 - [mlir-tblgen] Emit named result
	indices (#210542)
Message-ID: <6a5e016f.a1accbb4.5eca2.4373@mx.google.com>


Author: Ryan Thomas Lynch
Date: 2026-07-20T13:07:22+02:00
New Revision: 9428725fa09113905847decc48e8b5a731ad0ae6

URL: https://github.com/llvm/llvm-project/commit/9428725fa09113905847decc48e8b5a731ad0ae6
DIFF: https://github.com/llvm/llvm-project/commit/9428725fa09113905847decc48e8b5a731ad0ae6.diff

LOG: [mlir-tblgen] Emit named result indices (#210542)

similar to https://github.com/llvm/llvm-project/pull/146839

useful for cases such as being able to programmatically update the
result segment sizes or clone an operation via `OperationState` while
editing just a few results, programmatically

Added: 
    

Modified: 
    mlir/test/mlir-tblgen/op-result.td
    mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/test/mlir-tblgen/op-result.td b/mlir/test/mlir-tblgen/op-result.td
index a4f7af6dbcf1c..ddc1ef077d31c 100644
--- a/mlir/test/mlir-tblgen/op-result.td
+++ b/mlir/test/mlir-tblgen/op-result.td
@@ -14,6 +14,9 @@ def OpA : NS_Op<"one_normal_result_op", []> {
   let results = (outs I32:$result);
 }
 
+// DECL-LABEL: class OpA : {{.*}} {
+// DECL: static constexpr int odsIndex_result = 0;
+
 // CHECK-LABEL: void OpA::build
 // CHECK:         ::mlir::TypeRange resultTypes, ::mlir::ValueRange operands
 // CHECK:         assert(resultTypes.size() == 1u && "mismatched number of return types");
@@ -24,6 +27,10 @@ def OpB : NS_Op<"same_input_output_type_op", [SameOperandsAndResultType]> {
   let results = (outs I32:$y);
 }
 
+// DECL-LABEL: class OpB : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_y = 0;
+
 // CHECK-LABEL: OpB definitions
 // CHECK: void OpB::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type y, ::mlir::Value x)
 // CHECK:   odsState.addTypes(y);
@@ -39,6 +46,10 @@ def OpC : NS_Op<"three_normal_result_op", []> {
   let results = (outs I32:$x, /*unnamed*/I32, I32:$z);
 }
 
+// DECL-LABEL: class OpC : {{.*}} {
+// DECL: static constexpr int odsIndex_x = 0;
+// DECL: static constexpr int odsIndex_z = 2;
+
 // CHECK-LABEL: OpC definitions
 // CHECK:       void OpC::build(::mlir::OpBuilder &odsBuilder, ::mlir::OperationState &odsState, ::mlir::Type x, ::mlir::Type resultType1, ::mlir::Type z)
 // CHECK-NEXT:   odsState.addTypes(x)

diff  --git a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
index deef178e68dc1..4614459abb2b6 100644
--- a/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp
@@ -2175,7 +2175,8 @@ generateNamedOperandGetters(const Operator &op, Class &opClass,
                     "'SameVariadicOperandSize' traits");
   }
 
-  // Print the ods names so they don't need to be hardcoded in the source.
+  // Print the ODS indices of operands so they don't need to be hardcoded in the
+  // source.
   for (int i = 0; i != numOperands; ++i) {
     const auto &operand = op.getOperand(i);
     if (operand.name.empty())
@@ -2389,6 +2390,17 @@ void OpEmitter::genNamedResultGetters() {
                     "'SameVariadicResultSize' traits");
   }
 
+  // Print the ODS indices of results so they don't need to be hardcoded in the
+  // source.
+  for (int i = 0; i != numResults; ++i) {
+    const auto &result = op.getResult(i);
+    if (result.name.empty())
+      continue;
+
+    opClass.declare<Field>("static constexpr int",
+                           Twine("odsIndex_") + result.name + " = " + Twine(i));
+  }
+
   // Build the initializer string for the result segment size attribute.
   std::string attrSizeInitCode;
   if (attrSizedResults) {


        

From llvmlistbot at llvm.org  Mon Jul 20 04:07:29 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 20 Jul 2026 04:07:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5e0171.1b0626d0.f869d.f982@mx.google.com>


https://github.com/joker-eph closed https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Mon Jul 20 04:13:43 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Mon, 20 Jul 2026 04:13:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a5e02e7.d76d81af.23b347.1b4a@mx.google.com>


================
@@ -349,4 +349,31 @@ TEST(ExecutionContext, EnableDisableBreakpointOnCallback) {
   executionCtx(original, DebuggerAction{});
   EXPECT_EQ(counter, 4);
 }
+
+TEST(ExecutionContext, RerunRequestedFromPostActionCallback) {
+  // If rerun is requested from the post-action callback path,
+  // the same action must be rerun immediately.
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Next, ExecutionContext::Rerun, ExecutionContext::Apply};
+  int idx = 0;
+  int callbackCounter = 0;
+  int executionCounter = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++callbackCounter;
+    EXPECT_EQ(backtrace->getAction().getTag(), DebuggerAction::tag);
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  auto callback = [&]() { ++executionCounter; };
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(callbackCounter, 3);
+  EXPECT_EQ(executionCounter, 2);
+}
----------------
joker-eph wrote:

We're not testing the rerunControlStack here, can you add another test with some nested actions exercising it?

https://github.com/llvm/llvm-project/pull/209197

From llvmlistbot at llvm.org  Mon Jul 20 04:19:04 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Mon, 20 Jul 2026 04:19:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Privatize worker reduction
 accumulators per worker (PR #210456)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210456@github.com>
Message-ID: <6a5e0428.a1accbb4.5eca2.ccf4@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `sanitizer-aarch64-linux-bootstrap-hwasan` running on `sanitizer-buildbot11` while building `mlir` at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/55/builds/30633

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/wasm-ld
-- Testing: 96553 tests, 72 workers --
Testing:  0
FAIL: Clang :: ClangScanDeps/logging-two-threads.c (3749 of 96553)
******************** TEST 'Clang :: ClangScanDeps/logging-two-threads.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 5
rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# note: command had no output on stdout or stderr
# RUN: at line 6
split-file /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: split-file /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# note: command had no output on stdout or stderr
# RUN: at line 7
sed -e "s|DIR|/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template > /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json
# executed command: sed -e 's|DIR|/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g' /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template
# note: command had no output on stdout or stderr
# RUN: at line 9
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/clang-scan-deps -compilation-database /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json    -format experimental-full -log-path=/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/clang-scan-deps -compilation-database /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json -format experimental-full -log-path=/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# note: command had no output on stdout or stderr
# RUN: at line 13
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1 < /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1
# note: command had no output on stdout or stderr
# RUN: at line 14
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2 < /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2
# note: command had no output on stdout or stderr
# RUN: at line 24
grep -c "pcm_compile:" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# executed command: grep -c pcm_compile: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# note: command had no output on stdout or stderr
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# note: command had no output on stdout or stderr
# RUN: at line 25
grep -c "pcm_write:" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=PCMWRITES
# executed command: grep -c pcm_write: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
Step 14 (stage3/hwasan check) failure: stage3/hwasan check (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:569: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/wasm-ld
-- Testing: 96553 tests, 72 workers --
Testing:  0
FAIL: Clang :: ClangScanDeps/logging-two-threads.c (3749 of 96553)
******************** TEST 'Clang :: ClangScanDeps/logging-two-threads.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 5
rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: rm -rf /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# note: command had no output on stdout or stderr
# RUN: at line 6
split-file /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: split-file /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# note: command had no output on stdout or stderr
# RUN: at line 7
sed -e "s|DIR|/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template > /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json
# executed command: sed -e 's|DIR|/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g' /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template
# note: command had no output on stdout or stderr
# RUN: at line 9
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/clang-scan-deps -compilation-database /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json    -format experimental-full -log-path=/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/clang-scan-deps -compilation-database /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json -format experimental-full -log-path=/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# note: command had no output on stdout or stderr
# RUN: at line 13
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1 < /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1
# note: command had no output on stdout or stderr
# RUN: at line 14
/home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2 < /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2
# note: command had no output on stdout or stderr
# RUN: at line 24
grep -c "pcm_compile:" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# executed command: grep -c pcm_compile: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# note: command had no output on stdout or stderr
# executed command: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# note: command had no output on stdout or stderr
# RUN: at line 25
grep -c "pcm_write:" /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm-project/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=PCMWRITES
# executed command: grep -c pcm_write: /home/b/sanitizer-aarch64-linux-bootstrap-hwasan/build/llvm_build2_hwasan/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log

```

</details>

https://github.com/llvm/llvm-project/pull/210456

From llvmlistbot at llvm.org  Mon Jul 20 05:09:08 2026
From: llvmlistbot at llvm.org (Artem Gindinson)
Date: Mon, 20 Jul 2026 05:09:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ArmSVE] Fix comment inconsistency for
 `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC) (PR #210502)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210502@github.com>
Message-ID: <6a5e0fe4.94ba4530.177dd4.1454@mx.google.com>


https://github.com/AGindinson closed https://github.com/llvm/llvm-project/pull/210502

From llvmlistbot at llvm.org  Mon Jul 20 05:09:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 05:09:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1caf9e1 - [mlir][ArmSVE] Fix comment
 inconsistency for `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC)
 (#210502)
Message-ID: <6a5e0fe1.a2448446.d7f34.85d8@mx.google.com>


Author: Federico Bruzzone
Date: 2026-07-20T14:09:00+02:00
New Revision: 1caf9e1ef68ef85f1976e82bfc33befe1edc6eff

URL: https://github.com/llvm/llvm-project/commit/1caf9e1ef68ef85f1976e82bfc33befe1edc6eff
DIFF: https://github.com/llvm/llvm-project/commit/1caf9e1ef68ef85f1976e82bfc33befe1edc6eff.diff

LOG: [mlir][ArmSVE] Fix comment inconsistency for `pack_lhs` in `ArmSVE/pack-unpack-mmt4d.mlir` (NFC) (#210502)

This PR simply fix a comment inconsistency for `pack_lhs` in
[ArmSVE/pack-unpack-mmt4d.mlir](https://github.com/llvm/llvm-project/compare/main...FedericoBruzzone:nfc-armsve?expand=1#diff-7be071860c440806b8dad954ed61946e7e9f27e032ed6044a704f0c0f70e4407)
as identified in #208226.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>

Added: 
    

Modified: 
    mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
index 6192ed345debf..a5cc70119f595 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSVE/pack-unpack-mmt4d.mlir
@@ -119,9 +119,10 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 }
 
 //===----------------------------------------------------------------------===//
-// @matmul_via_mmt4d
+// @pack_lhs
 //
-// Implements matrix-multiplication via linalg.mmt4d
+// Implements packing for the A matrix (LHS) in matrix multiplication. The inner
+// tile size is fixed: 8 * 1.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
   %pad = arith.constant 0 : i32


        

From llvmlistbot at llvm.org  Mon Jul 20 06:05:55 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 06:05:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5e1d33.8e75e0cb.f69c7.419e@mx.google.com>


https://github.com/secona updated https://github.com/llvm/llvm-project/pull/210600

>From 8aa7d419ee5c0751b557f46e2881d94e902ea37e Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sun, 19 Jul 2026 20:57:19 +0700
Subject: [PATCH 1/4] [mlir][SPIRV] Add SPIRVToLLVM conversion for selection
 with yielding values

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    |  8 +++
 .../SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 56 +++++++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c5f9305a5c226..2982061c957e0 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1470,6 +1470,14 @@ class SelectionPattern : public SPIRVToLLVMConversion<spirv::SelectionOp> {
     auto position = rewriter.getInsertionPoint();
     auto *continueBlock = rewriter.splitBlock(currentBlock, position);
 
+    // Add arguments to the continue block for selections that yield values.
+    for (auto ty : op.getResultTypes()) {
+      Type dstTy = getTypeConverter()->convertType(ty);
+      if (!dstTy)
+        return rewriter.notifyMatchFailure(op, "failed to convert type");
+      continueBlock->addArgument(dstTy, loc);
+    }
+
     // Extract conditional branch information from the header block. By SPIR-V
     // dialect spec, it should contain `spirv.BranchConditional` or
     // `spirv.Switch` op. Note that `spirv.Switch op` is not supported at the
diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index ec47dd31cc637..389a21681bafb 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
 }
 
 // -----

>From edb194920ba746b10ee7f900cd75871e053801ab Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 19:52:25 +0700
Subject: [PATCH 2/4] test: clearer 1 value and 2 values

---
 .../test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index 389a21681bafb..769e9a0c61c20 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -236,7 +236,7 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
       spirv.mlir.merge %1 : i32
     }
-    // CHECK: ^bb4({{.*}}):
+    // CHECK: ^bb4({{.*}}: i32):
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
@@ -266,7 +266,7 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
       spirv.mlir.merge %1, %2 : i32, i32
     }
-    // CHECK: ^bb4({{.*}}):
+    // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }

>From 82837f0c3afd8b41035dd093319dfd59140d780f Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 20:03:19 +0700
Subject: [PATCH 3/4] test: return %0 in selection_with_yielding_value

---
 mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index 769e9a0c61c20..a043dafac6341 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -237,8 +237,7 @@ spirv.module Logical GLSL450 {
       spirv.mlir.merge %1 : i32
     }
     // CHECK: ^bb4({{.*}}: i32):
-    %one = spirv.Constant 1 : i32
-    spirv.ReturnValue %one : i32
+    spirv.ReturnValue %0 : i32
   }
 
   spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {

>From 26692812ded5658eea4eacc49d105cc0978e732a Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 20:05:28 +0700
Subject: [PATCH 4/4] test: use both values in
 selection_with_multiple_yielding_values

---
 .../Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir     | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index a043dafac6341..1a946dd5df9ac 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -266,8 +266,9 @@ spirv.module Logical GLSL450 {
       spirv.mlir.merge %1, %2 : i32, i32
     }
     // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
-    %one = spirv.Constant 1 : i32
-    spirv.ReturnValue %one : i32
+    // Makes sure both values are used.
+    %sum = spirv.IAdd %0#0, %0#1 : i32
+    spirv.ReturnValue %sum : i32
   }
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 06:06:24 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 06:06:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5e1d50.f06f4994.2f0620.b5b8@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
----------------
secona wrote:

I think it should. I've also edited the yield 1 value test, so its clearer that its 1 value. edb194920ba746b10ee7f900cd75871e053801ab

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 06:07:02 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 06:07:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5e1d76.492e23e4.d19bc.95fc@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
----------------
secona wrote:

I've made it return %0 in 82837f0c3afd8b41035dd093319dfd59140d780f

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 06:07:53 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 06:07:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a5e1da9.f1c1923d.2d2b59.c51a@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
----------------
secona wrote:

spirv.func does not return multiple value. What do you think about adding the two yielded selection value then returning? Its implemented in 26692812ded5658eea4eacc49d105cc0978e732a

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Mon Jul 20 06:49:24 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 06:49:24 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e2764.3d3bca2f.27f3c1.167e@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 72006f5ee8a29eb2feb34c387ee26ec12deaadf1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  22 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  75 +++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  10 +-
 mlir/lib/AsmParser/Token.cpp                  |  57 ++++++++-
 mlir/lib/AsmParser/Token.h                    |   9 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  39 +++++-
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  99 +++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 553 insertions(+), 163 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..6ada4081b54ea 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,19 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics());
+  if (!result) {
+    emitError(loc, "invalid floating point literal");
+    return nullptr;
+  }
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..c3abb1bd19f50 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,56 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal. `fromSign` starts at the
+/// (already consumed) leading sign and runs to the end of the buffer. On success
+/// returns a `floatliteral` (or `error`) token and advances `curPtr`; otherwise
+/// returns std::nullopt with `curPtr` unchanged so the caller can fall back to a
+/// plain `plus`/`minus` token. The sign is mandatory and folded into the token
+/// spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+
+  // Split off the alphabetic keyword (inf/qnan/nan/snan) that follows the sign.
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() &&
+        (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+         rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    // take_until returns all of `rest` when there is no ')'.
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1; // Consume the payload and the ')'.
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +379,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d5439bd192bc6 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,13 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
+    result = tok.getFloatingPointValue(isNegative, semantics);
+    if (!result)
+      return emitError(tok.getLoc()) << "invalid floating point literal";
     return success();
   }
 
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..7f6319292d8ad 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -13,6 +13,7 @@
 #include "Token.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +52,58 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. Returns std::nullopt if the
+/// literal cannot be parsed for the given semantics.
+std::optional<APFloat>
+Token::getFloatingPointValue(bool isNegative,
+                             const llvm::fltSemantics &semantics) const {
+  // A float literal carries its sign from exactly one place. Decimal and C-style
+  // hexadecimal literals have no sign in their spelling; a negative one is a
+  // separate '-' token, delivered here via `isNegative`. inf/NaN literals instead
+  // fold a mandatory sign into the spelling (`+inf`, `-nan(0x1)`, ...). A '-'
+  // token in front of an already-signed spelling would be a doubly-signed
+  // literal (e.g. `-+inf`), which is rejected. The leading '+' is dropped because
+  // APFloat::convertFromString accepts `+inf` but not a leading '+' on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-")))
     return std::nullopt;
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // A negative literal is only valid for a type with a sign representation;
+  // reject it for unsigned formats (e.g. f8E8M0FNU) rather than let APFloat
+  // abort later when the sign is materialized.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics))
+    return std::nullopt;
+
+  // A special value is only valid for a type that can represent it. Guard here
+  // rather than let APFloat abort (makeInf/makeNaN llvm_unreachable) on types
+  // with no Inf/NaN encoding (e.g. the FiniteOnly f4/f6/f8 formats).
+  bool wantsInf = str == "inf";
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if ((wantsInf && !APFloat::semanticsHasInf(semantics)) ||
+      (wantsNaN && !APFloat::semanticsHasNaN(semantics)))
+    return std::nullopt;
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics: this covers decimal,
+  // C-style hexadecimal (`0x1.fp13`), inf, and `nan(0x..)`/`snan(0x..)` forms,
+  // preserving NaN payloads. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..f27a9a6e0b85d 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,6 +10,7 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
@@ -84,9 +85,11 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. Returns std::nullopt if
+  /// the literal cannot be parsed for the given semantics.
+  std::optional<APFloat> getFloatingPointValue(
+      bool isNegative, const llvm::fltSemantics &semantics) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..6b55fe69f4f11 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,46 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
+  // Print special values in a human-readable, typeless form. The sign is always
+  // included, mirroring LLVM's textual float literals. Because these forms carry
+  // no type, signal the caller (via printedHex) to keep the trailing type.
+  if (apValue.isInfinity()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+  if (apValue.isNaN()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value. A type with no mantissa bits
+    // (e.g. f8E8M0FNU) has a zero-width payload with no quiet/payload bits to
+    // distinguish, so it prints as the canonical quiet NaN.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+    } else {
+      if (apValue.isSignaling())
+        os << 's';
+      os << "nan(";
+      // Clear the signaling/quiet bit and trim leading zeros for output.
+      payload.clearBit(payload.getBitWidth() - 1);
+      SmallVector<char, 16> str;
+      payload.trunc(std::max(payload.getActiveBits(), 1u))
+          .toString(str, /*Radix=*/16, /*Signed=*/false,
+                    /*formatAsCLiteral=*/true);
+      os << str << ')';
+    }
+    return;
+  }
+
   // We would like to output the FP constant value in exponential notation,
   // but we cannot do this if doing so will lose precision.  Check here to
   // make sure that we only output it in exponential format if we can parse
   // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..2255f79e2cb12 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,91 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // A '-' token in front of an already-signed inf/NaN literal is a doubly-signed
+  // literal and must be rejected.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN is a FiniteOnly format with no Inf/NaN encoding; requesting inf
+  // must be rejected, not crash APFloat.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN is a FiniteOnly format with no NaN encoding.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has a NaN encoding but no infinities; inf must be rejected.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is an unsigned format; a negative literal must be rejected, not
+  // materialized and crash the printer.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:13:39 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 07:13:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5e2d13.f1c1923d.2d2b59.8727@mx.google.com>


https://github.com/secona updated https://github.com/llvm/llvm-project/pull/209591

>From c3ecdc74839369e26f73f60e3733e6f070cc4429 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Wed, 15 Jul 2026 01:42:06 +0700
Subject: [PATCH 1/2] [mlir][SparseTensor] Reject sparse encoding on
 non-primary types

---
 .../SparseTensor/IR/SparseTensorDialect.cpp   |  14 ++
 .../Transforms/SparseTensorConversion.cpp     |   3 -
 .../Transforms/Utils/CodegenUtils.cpp         |  12 --
 .../Transforms/Utils/CodegenUtils.h           |   5 -
 .../SparseTensor/constant_index_map.mlir      |  44 ++---
 .../SparseTensor/conversion_invalid.mlir      |  99 ++++++++++-
 .../SparseTensor/vectorize_reduction.mlir     | 160 +++++++++---------
 7 files changed, 214 insertions(+), 123 deletions(-)

diff --git a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
index 7c9f552d62255..6c26a31bf88a0 100644
--- a/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
+++ b/mlir/lib/Dialect/SparseTensor/IR/SparseTensorDialect.cpp
@@ -919,6 +919,18 @@ LogicalResult SparseTensorEncodingAttr::verify(
   return success();
 }
 
+static bool isValidPrimaryType(Type elemTp) {
+  if (elemTp.isF64() || elemTp.isF32() || elemTp.isF16() || elemTp.isBF16() ||
+      elemTp.isInteger(64) || elemTp.isInteger(32) || elemTp.isInteger(16) ||
+      elemTp.isInteger(8))
+    return true;
+  if (auto complexTp = dyn_cast<ComplexType>(elemTp)) {
+    Type elt = complexTp.getElementType();
+    return elt.isF64() || elt.isF32();
+  }
+  return false;
+}
+
 LogicalResult SparseTensorEncodingAttr::verifyEncoding(
     ArrayRef<Size> dimShape, Type elementType,
     function_ref<InFlightDiagnostic()> emitError) const {
@@ -964,6 +976,8 @@ LogicalResult SparseTensorEncodingAttr::verifyEncoding(
       return emitError() << "implicit value must be zero";
     }
   }
+  if (!isValidPrimaryType(elementType))
+    return emitError() << "invalid primary type";
   return success();
 }
 
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
index c9bcefe92de79..0e88d31dae8e8 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorConversion.cpp
@@ -383,9 +383,6 @@ class SparseTensorNewConverter : public OpConversionPattern<NewOp> {
     const auto stt = getSparseTensorType(op);
     if (!stt.hasEncoding())
       return failure();
-    // Verify that the element type is supported by the runtime library.
-    if (!isValidPrimaryType(stt.getElementType()))
-      return rewriter.notifyMatchFailure(op, "unsupported element type");
     // Construct the `reader` opening method calls.
     SmallVector<Value> dimSizesValues;
     Value dimSizesBuffer;
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
index 614c29ba68481..3c6a905ebb696 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.cpp
@@ -99,18 +99,6 @@ StringRef mlir::sparse_tensor::overheadTypeFunctionSuffix(Type tp) {
   return overheadTypeFunctionSuffix(overheadTypeEncoding(tp));
 }
 
-bool mlir::sparse_tensor::isValidPrimaryType(Type elemTp) {
-  if (elemTp.isF64() || elemTp.isF32() || elemTp.isF16() || elemTp.isBF16() ||
-      elemTp.isInteger(64) || elemTp.isInteger(32) || elemTp.isInteger(16) ||
-      elemTp.isInteger(8))
-    return true;
-  if (auto complexTp = dyn_cast<ComplexType>(elemTp)) {
-    Type elt = complexTp.getElementType();
-    return elt.isF64() || elt.isF32();
-  }
-  return false;
-}
-
 PrimaryType mlir::sparse_tensor::primaryTypeEncoding(Type elemTp) {
   if (elemTp.isF64())
     return PrimaryType::kF64;
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
index 98e351a371ac1..1c10dd5566184 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/CodegenUtils.h
@@ -60,11 +60,6 @@ StringRef overheadTypeFunctionSuffix(OverheadType ot);
 /// Converts an overhead storage type to its function-name suffix.
 StringRef overheadTypeFunctionSuffix(Type overheadTp);
 
-/// Returns true if the given type is a valid sparse tensor element type
-/// supported by the runtime library (i.e., maps to a PrimaryType).
-/// Use this to guard calls to primaryTypeEncoding() with invalid types.
-bool isValidPrimaryType(Type elemTp);
-
 /// Converts a primary storage type to its internal type-encoding.
 PrimaryType primaryTypeEncoding(Type elemTp);
 
diff --git a/mlir/test/Dialect/SparseTensor/constant_index_map.mlir b/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
index cf1eb3e9e44f5..6c6e4579d4f11 100644
--- a/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
+++ b/mlir/test/Dialect/SparseTensor/constant_index_map.mlir
@@ -8,34 +8,34 @@
 #SpVec = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
 
 // CHECK-LABEL:   func.func @main(
-// CHECK-SAME:      %[[VAL_0:.*0]]: tensor<1x77xi1>,
-// CHECK-SAME:      %[[VAL_1:.*1]]: tensor<1x77xi1>) -> tensor<77xi1, #{{.*}}> {
+// CHECK-SAME:      %[[VAL_0:.*0]]: tensor<1x77xi32>,
+// CHECK-SAME:      %[[VAL_1:.*1]]: tensor<1x77xi32>) -> tensor<77xi32, #{{.*}}> {
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 77 : index
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant 0 : index
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1 : index
-// CHECK-DAG:       %[[VAL_5:.*]] = tensor.empty() : tensor<77xi1, #{{.*}}>
-// CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<1x77xi1>
-// CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_buffer %[[VAL_1]] : tensor<1x77xi1>
-// CHECK:           %[[VAL_8:.*]] = scf.for %[[VAL_9:.*]] = %[[VAL_3]] to %[[VAL_2]] step %[[VAL_4]] iter_args(%[[VAL_10:.*]] = %[[VAL_5]]) -> (tensor<77xi1, #{{.*}}>) {
-// CHECK:             %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi1>
-// CHECK:             %[[VAL_12:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi1>
-// CHECK:             %[[VAL_13:.*]] = arith.addi %[[VAL_11]], %[[VAL_12]] : i1
-// CHECK:             %[[VAL_14:.*]] = tensor.insert %[[VAL_13]] into %[[VAL_10]]{{\[}}%[[VAL_9]]] : tensor<77xi1, #{{.*}}>
-// CHECK:             scf.yield %[[VAL_14]] : tensor<77xi1, #{{.*}}>
+// CHECK-DAG:       %[[VAL_5:.*]] = tensor.empty() : tensor<77xi32, #{{.*}}>
+// CHECK-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<1x77xi32>
+// CHECK-DAG:       %[[VAL_7:.*]] = bufferization.to_buffer %[[VAL_1]] : tensor<1x77xi32>
+// CHECK:           %[[VAL_8:.*]] = scf.for %[[VAL_9:.*]] = %[[VAL_3]] to %[[VAL_2]] step %[[VAL_4]] iter_args(%[[VAL_10:.*]] = %[[VAL_5]]) -> (tensor<77xi32, #{{.*}}>) {
+// CHECK:             %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi32>
+// CHECK:             %[[VAL_12:.*]] = memref.load %[[VAL_7]]{{\[}}%[[VAL_3]], %[[VAL_9]]] : memref<1x77xi32>
+// CHECK:             %[[VAL_13:.*]] = arith.addi %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:             %[[VAL_14:.*]] = tensor.insert %[[VAL_13]] into %[[VAL_10]]{{\[}}%[[VAL_9]]] : tensor<77xi32, #{{.*}}>
+// CHECK:             scf.yield %[[VAL_14]] : tensor<77xi32, #{{.*}}>
 // CHECK:           }
-// CHECK:           %[[VAL_15:.*]] = sparse_tensor.load %[[VAL_16:.*]] hasInserts : tensor<77xi1, #{{.*}}>
-// CHECK:           return %[[VAL_15]] : tensor<77xi1, #{{.*}}>
+// CHECK:           %[[VAL_15:.*]] = sparse_tensor.load %[[VAL_16:.*]] hasInserts : tensor<77xi32, #{{.*}}>
+// CHECK:           return %[[VAL_15]] : tensor<77xi32, #{{.*}}>
 // CHECK:         }
-func.func @main(%arg0: tensor<1x77xi1>, %arg1: tensor<1x77xi1>) -> tensor<77xi1, #SpVec> {
-  %0 = tensor.empty() : tensor<77xi1, #SpVec>
+func.func @main(%arg0: tensor<1x77xi32>, %arg1: tensor<1x77xi32>) -> tensor<77xi32, #SpVec> {
+  %0 = tensor.empty() : tensor<77xi32, #SpVec>
   %1 = linalg.generic {
     indexing_maps = [#map1, #map1, #map2],
     iterator_types = ["parallel"]}
-    ins(%arg0, %arg1 : tensor<1x77xi1>, tensor<1x77xi1>)
-    outs(%0 : tensor<77xi1, #SpVec>) {
-  ^bb0(%in: i1, %in_0: i1, %out: i1):
-    %2 = arith.addi %in, %in_0 : i1
-    linalg.yield %2 : i1
-  } -> tensor<77xi1, #SpVec>
-  return %1 : tensor<77xi1, #SpVec>
+    ins(%arg0, %arg1 : tensor<1x77xi32>, tensor<1x77xi32>)
+    outs(%0 : tensor<77xi32, #SpVec>) {
+  ^bb0(%in: i32, %in_0: i32, %out: i32):
+    %2 = arith.addi %in, %in_0 : i32
+    linalg.yield %2 : i32
+  } -> tensor<77xi32, #SpVec>
+  return %1 : tensor<77xi32, #SpVec>
 }
diff --git a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
index 456c1898a9311..7e95be495b333 100644
--- a/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
+++ b/mlir/test/Dialect/SparseTensor/conversion_invalid.mlir
@@ -8,7 +8,104 @@
 #sparse = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
 
 func.func @new_index_elem_type(%arg0: index) {
-  // expected-error at +1 {{failed to legalize operation 'sparse_tensor.new'}}
+  // expected-error at +1 {{invalid primary type}}
   %0 = sparse_tensor.new %arg0 : index to tensor<?xindex, #sparse>
   return
 }
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @alloc_tensor_i1_elem_type(%sz: index) -> tensor<?xi1, #sparse1d> {
+  %0 = bufferization.alloc_tensor(%sz) : tensor<?xi1, #sparse1d>
+  return %0 : tensor<?xi1, #sparse1d>
+}
+
+// -----
+
+#unordered_coo = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : compressed(nonunique, nonordered), d1 : singleton(nonordered))}>
+#ordered_coo = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton)}>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @reorder_coo_i1_elem_type(%arg0: tensor<?x?xi1, #unordered_coo>)
+    -> tensor<?x?xi1, #ordered_coo> {
+  %0 = sparse_tensor.reorder_coo quick_sort %arg0
+    : tensor<?x?xi1, #unordered_coo> to tensor<?x?xi1, #ordered_coo>
+  return %0 : tensor<?x?xi1, #ordered_coo>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +2 {{invalid primary type}}
+func.func @assemble_i1_elem_type(%pos: tensor<2xindex>, %coords: tensor<4x1xindex>,
+                                 %vals: tensor<4xi1>) -> tensor<8xi1, #sparse1d> {
+  %0 = sparse_tensor.assemble (%pos, %coords), %vals
+    : (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1> to tensor<8xi1, #sparse1d>
+  return %0 : tensor<8xi1, #sparse1d>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @values_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> memref<?xi1> {
+  %0 = sparse_tensor.values %arg0 : tensor<8xi1, #sparse1d> to memref<?xi1>
+  return %0 : memref<?xi1>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @number_of_entries_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>) -> index {
+  %0 = sparse_tensor.number_of_entries %arg0 : tensor<8xi1, #sparse1d>
+  return %0 : index
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @disassemble_i1_elem_type(%sp: tensor<8xi1, #sparse1d>,
+                                      %op: tensor<2xindex>, %oi: tensor<4x1xindex>,
+                                      %od: tensor<4xi1>)
+    -> (tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>) {
+  %rp, %ri, %d, %rpl, %ril, %dl = sparse_tensor.disassemble %sp
+    : tensor<8xi1, #sparse1d>
+    out_lvls(%op, %oi : tensor<2xindex>, tensor<4x1xindex>)
+    out_vals(%od : tensor<4xi1>)
+    -> (tensor<2xindex>, tensor<4x1xindex>), tensor<4xi1>, (index, index), index
+  return %rp, %ri, %d : tensor<2xindex>, tensor<4x1xindex>, tensor<4xi1>
+}
+
+// -----
+
+#sparse1d = #sparse_tensor.encoding<{ map = (d0) -> (d0 : compressed) }>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @insert_i1_elem_type(%arg0: tensor<8xi1, #sparse1d>, %idx: index, %val: i1)
+    -> tensor<8xi1, #sparse1d> {
+  %0 = tensor.insert %val into %arg0[%idx] : tensor<8xi1, #sparse1d>
+  return %0 : tensor<8xi1, #sparse1d>
+}
+
+// -----
+
+#csr = #sparse_tensor.encoding<{map = (d0, d1) -> (d0 : dense, d1 : compressed)}>
+
+// expected-error at +1 {{invalid primary type}}
+func.func @compress_i1_elem_type(%tensor: tensor<8x8xi1, #csr>,
+                                 %values: memref<?xi1>, %filled: memref<?xi1>,
+                                 %added: memref<?xindex>, %count: index, %i: index)
+    -> tensor<8x8xi1, #csr> {
+  %0 = sparse_tensor.compress %values, %filled, %added, %count into %tensor[%i]
+    : memref<?xi1>, memref<?xi1>, memref<?xindex>, tensor<8x8xi1, #csr>
+  return %0 : tensor<8x8xi1, #csr>
+}
diff --git a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
index 6effbbf98abb7..5eb428f197a75 100644
--- a/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/SparseTensor/vectorize_reduction.mlir
@@ -8,52 +8,52 @@
 // Check that we vectorize reductions with ori.
 
 // CHECK-ON-LABEL:   func.func @sparse_reduction_ori(
-// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-ON-DAG:       %[[VAL_2:.*]] = arith.constant 8 : index
-// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
+// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
+// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
 // CHECK-ON:           %[[VAL_10:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i13 to vector<8xi13>
-// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi13>) {
+// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i32 to vector<8xi32>
+// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi32>) {
 // CHECK-ON:             %[[VAL_16:.*]] = affine.min #map(%[[VAL_11]], %[[VAL_14]]){{\[}}%[[VAL_2]]]
 // CHECK-ON:             %[[VAL_17:.*]] = vector.create_mask %[[VAL_16]] : vector<8xi1>
-// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi13>, vector<8xi1>, vector<8xi13> into vector<8xi13>
-// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_15]], %[[VAL_18]] : vector<8xi13>
-// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi13>
-// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi13>
+// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi32>, vector<8xi1>, vector<8xi32> into vector<8xi32>
+// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_15]], %[[VAL_18]] : vector<8xi32>
+// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi32>
+// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi32>
 // CHECK-ON:           } {"Emitted from" = "linalg.generic"}
-// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi13> into i13
-// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i13>
-// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i13>
-// CHECK-ON:           return %[[VAL_23]] : tensor<i13>
+// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi32> into i32
+// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i32>
+// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i32>
+// CHECK-ON:           return %[[VAL_23]] : tensor<i32>
 // CHECK-ON:         }
 //
 // CHECK-OFF-LABEL:   func.func @sparse_reduction_ori(
-// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
+// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK-OFF:           %[[VAL_9:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_3]]] : memref<?xindex>
-// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i13) {
-// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi13>
-// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_12]], %[[VAL_13]] : i13
-// CHECK-OFF:             scf.yield %[[VAL_14]] : i13
+// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i32) {
+// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi32>
+// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK-OFF:             scf.yield %[[VAL_14]] : i32
 // CHECK-OFF:           } {"Emitted from" = "linalg.generic"}
-// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i13>
-// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i13>
-// CHECK-OFF:           return %[[VAL_16]] : tensor<i13>
+// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i32>
+// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i32>
+// CHECK-OFF:           return %[[VAL_16]] : tensor<i32>
 // CHECK-OFF:         }
 #SparseVector = #sparse_tensor.encoding<{map = (d0) -> (d0 : compressed)}>
 
@@ -65,17 +65,17 @@
   iterator_types = ["reduction"]
 }
 
-func.func @sparse_reduction_ori(%argx: tensor<i13>,
-                                %arga: tensor<?xi13, #SparseVector>)
- -> tensor<i13> {
+func.func @sparse_reduction_ori(%argx: tensor<i32>,
+                                %arga: tensor<?xi32, #SparseVector>)
+ -> tensor<i32> {
   %0 = linalg.generic #trait
-     ins(%arga: tensor<?xi13, #SparseVector>)
-      outs(%argx: tensor<i13>) {
-      ^bb(%a: i13, %x: i13):
-        %t = arith.ori %x, %a: i13
-        linalg.yield %t : i13
-  } -> tensor<i13>
-  return %0 : tensor<i13>
+     ins(%arga: tensor<?xi32, #SparseVector>)
+      outs(%argx: tensor<i32>) {
+      ^bb(%a: i32, %x: i32):
+        %t = arith.ori %x, %a: i32
+        linalg.yield %t : i32
+  } -> tensor<i32>
+  return %0 : tensor<i32>
 }
 
 // -----
@@ -85,52 +85,52 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
 // irrespective to where the accumulator appears on commutative operations.
 
 // CHECK-ON-LABEL:   func.func @sparse_reduction_ori_accumulator_on_rhs(
-// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-ON-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-ON-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-ON-DAG:       %[[VAL_2:.*]] = arith.constant 8 : index
-// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi13>
+// CHECK-ON-DAG:       %[[VAL_3:.*]] = arith.constant dense<0> : vector<8xi32>
 // CHECK-ON-DAG:       %[[VAL_4:.*]] = arith.constant 0 : index
 // CHECK-ON-DAG:       %[[VAL_5:.*]] = arith.constant 1 : index
-// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i13>
+// CHECK-ON-DAG:       %[[VAL_6:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-ON-DAG:       %[[VAL_7:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-ON-DAG:       %[[VAL_8:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-ON:           %[[VAL_9:.*]] = memref.load %[[VAL_8]][] : memref<i32>
 // CHECK-ON:           %[[VAL_10:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_4]]] : memref<?xindex>
 // CHECK-ON:           %[[VAL_11:.*]] = memref.load %[[VAL_6]]{{\[}}%[[VAL_5]]] : memref<?xindex>
-// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i13 to vector<8xi13>
-// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi13>) {
+// CHECK-ON:           %[[VAL_12:.*]] = vector.broadcast %[[VAL_9]] : i32 to vector<8xi32>
+// CHECK-ON:           %[[VAL_13:.*]] = scf.for %[[VAL_14:.*]] = %[[VAL_10]] to %[[VAL_11]] step %[[VAL_2]] iter_args(%[[VAL_15:.*]] = %[[VAL_12]]) -> (vector<8xi32>) {
 // CHECK-ON:             %[[VAL_16:.*]] = affine.min #map(%[[VAL_11]], %[[VAL_14]]){{\[}}%[[VAL_2]]]
 // CHECK-ON:             %[[VAL_17:.*]] = vector.create_mask %[[VAL_16]] : vector<8xi1>
-// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi13>, vector<8xi1>, vector<8xi13> into vector<8xi13>
-// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_18]], %[[VAL_15]] : vector<8xi13>
-// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi13>
-// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi13>
+// CHECK-ON:             %[[VAL_18:.*]] = vector.maskedload %[[VAL_7]]{{\[}}%[[VAL_14]]], %[[VAL_17]], %[[VAL_3]] : memref<?xi32>, vector<8xi1>, vector<8xi32> into vector<8xi32>
+// CHECK-ON:             %[[VAL_19:.*]] = arith.ori %[[VAL_18]], %[[VAL_15]] : vector<8xi32>
+// CHECK-ON:             %[[VAL_20:.*]] = arith.select %[[VAL_17]], %[[VAL_19]], %[[VAL_15]] : vector<8xi1>, vector<8xi32>
+// CHECK-ON:             scf.yield %[[VAL_20]] : vector<8xi32>
 // CHECK-ON:           } {"Emitted from" = "linalg.generic"}
-// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi13> into i13
-// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i13>
-// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i13>
-// CHECK-ON:           return %[[VAL_23]] : tensor<i13>
+// CHECK-ON:           %[[VAL_21:.*]] = vector.reduction <or>, %[[VAL_22:.*]] : vector<8xi32> into i32
+// CHECK-ON:           memref.store %[[VAL_21]], %[[VAL_8]][] : memref<i32>
+// CHECK-ON:           %[[VAL_23:.*]] = bufferization.to_tensor %[[VAL_8]] : memref<i32>
+// CHECK-ON:           return %[[VAL_23]] : tensor<i32>
 // CHECK-ON:         }
 //
 // CHECK-OFF-LABEL:   func.func @sparse_reduction_ori_accumulator_on_rhs(
-// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i13>,
-// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi13, #sparse{{[0-9]*}}>) -> tensor<i13> {
+// CHECK-OFF-SAME:      %[[VAL_0:.*]]: tensor<i32>,
+// CHECK-OFF-SAME:      %[[VAL_1:.*]]: tensor<?xi32, #sparse{{[0-9]*}}>) -> tensor<i32> {
 // CHECK-OFF-DAG:       %[[VAL_2:.*]] = arith.constant 0 : index
 // CHECK-OFF-DAG:       %[[VAL_3:.*]] = arith.constant 1 : index
-// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xindex>
-// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi13, #sparse{{[0-9]*}}> to memref<?xi13>
-// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i13> to memref<i13>
-// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i13>
+// CHECK-OFF-DAG:       %[[VAL_4:.*]] = sparse_tensor.positions %[[VAL_1]] {level = 0 : index} : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xindex>
+// CHECK-OFF-DAG:       %[[VAL_5:.*]] = sparse_tensor.values %[[VAL_1]] : tensor<?xi32, #sparse{{[0-9]*}}> to memref<?xi32>
+// CHECK-OFF-DAG:       %[[VAL_6:.*]] = bufferization.to_buffer %[[VAL_0]] : tensor<i32> to memref<i32>
+// CHECK-OFF:           %[[VAL_7:.*]] = memref.load %[[VAL_6]][] : memref<i32>
 // CHECK-OFF:           %[[VAL_8:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_2]]] : memref<?xindex>
 // CHECK-OFF:           %[[VAL_9:.*]] = memref.load %[[VAL_4]]{{\[}}%[[VAL_3]]] : memref<?xindex>
-// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i13) {
-// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi13>
-// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_13]], %[[VAL_12]] : i13
-// CHECK-OFF:             scf.yield %[[VAL_14]] : i13
+// CHECK-OFF:           %[[VAL_10:.*]] = scf.for %[[VAL_11:.*]] = %[[VAL_8]] to %[[VAL_9]] step %[[VAL_3]] iter_args(%[[VAL_12:.*]] = %[[VAL_7]]) -> (i32) {
+// CHECK-OFF:             %[[VAL_13:.*]] = memref.load %[[VAL_5]]{{\[}}%[[VAL_11]]] : memref<?xi32>
+// CHECK-OFF:             %[[VAL_14:.*]] = arith.ori %[[VAL_13]], %[[VAL_12]] : i32
+// CHECK-OFF:             scf.yield %[[VAL_14]] : i32
 // CHECK-OFF:           } {"Emitted from" = "linalg.generic"}
-// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i13>
-// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i13>
-// CHECK-OFF:           return %[[VAL_16]] : tensor<i13>
+// CHECK-OFF:           memref.store %[[VAL_15:.*]], %[[VAL_6]][] : memref<i32>
+// CHECK-OFF:           %[[VAL_16:.*]] = bufferization.to_tensor %[[VAL_6]] : memref<i32>
+// CHECK-OFF:           return %[[VAL_16]] : tensor<i32>
 // CHECK-OFF:         }
 #SparseVector = #sparse_tensor.encoding<{map = (d0) -> (d0 : compressed)}>
 
@@ -142,17 +142,17 @@ func.func @sparse_reduction_ori(%argx: tensor<i13>,
   iterator_types = ["reduction"]
 }
 
-func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i13>,
-                                                   %arga: tensor<?xi13, #SparseVector>)
- -> tensor<i13> {
+func.func @sparse_reduction_ori_accumulator_on_rhs(%argx: tensor<i32>,
+                                                   %arga: tensor<?xi32, #SparseVector>)
+ -> tensor<i32> {
   %0 = linalg.generic #trait
-     ins(%arga: tensor<?xi13, #SparseVector>)
-      outs(%argx: tensor<i13>) {
-      ^bb(%a: i13, %x: i13):
-        %t = arith.ori %a, %x: i13
-        linalg.yield %t : i13
-  } -> tensor<i13>
-  return %0 : tensor<i13>
+     ins(%arga: tensor<?xi32, #SparseVector>)
+      outs(%argx: tensor<i32>) {
+      ^bb(%a: i32, %x: i32):
+        %t = arith.ori %a, %x: i32
+        linalg.yield %t : i32
+  } -> tensor<i32>
+  return %0 : tensor<i32>
 }
 
 // -----

>From e31a1ea800b60840e81ea3cdc51abcc8d6f1d6b0 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 21:11:23 +0700
Subject: [PATCH 2/2] docs: add docs regarding supported types

---
 .../mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td      | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
index 46fdff08466c1..b13a9928ae29d 100644
--- a/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
+++ b/mlir/include/mlir/Dialect/SparseTensor/IR/SparseTensorAttrDefs.td
@@ -126,6 +126,10 @@ def SparseTensorEncodingAttr : SparseTensor_Attr<"SparseTensorEncoding",
     encoding. Compiler passes that run before this sparsifier pass need to be aware
     of the semantics of tensor types with such a sparsity encoding.
 
+    This encoding can only be applied to tensors with supported primary element
+    types (e.g., `f16`, `f32`, `f64`, `bf16`, `i8`, `i16`, `i32`, `i64`, and
+    complex variants of `f32` and `f64`).
+
     In this encoding, we use **dimension** to refer to the axes of the semantic tensor,
     and **level** to refer to the axes of the actual storage format, i.e., the
     operational representation of the sparse tensor in memory. The number of


From llvmlistbot at llvm.org  Mon Jul 20 07:14:33 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 07:14:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5e2d49.510f576d.2f694c.a25b@mx.google.com>


secona wrote:

> Logic looks good to me. Don't forget to update docs if there's any. It seems like this is changing the definition of a valid IR

Good call! I've added docs in the SparseTensorEncodingAttr attribute to mention this restriction.

https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Mon Jul 20 07:15:28 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 07:15:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5e2d80.f3151ff8.37ac76.51b8@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Mon Jul 20 07:16:09 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 07:16:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5e2da9.92ea0e2a.4a9a5.a2e7@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Mon Jul 20 07:17:21 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Mon, 20 Jul 2026 07:17:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SparseTensor] Reject sparse encoding
 on non-primary types (PR #209591)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209591@github.com>
Message-ID: <6a5e2df1.92d2873c.aa1ff.191e@mx.google.com>


https://github.com/secona edited https://github.com/llvm/llvm-project/pull/209591

From llvmlistbot at llvm.org  Mon Jul 20 07:18:27 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 07:18:27 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e2e33.7d702ed2.13d102.bc62@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From ff683de993045530d78d4bb7ce51ee1bad0b6cb3 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  75 +++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  66 +++++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  39 +++++-
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  99 +++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 563 insertions(+), 164 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..7c0778b7c3f65 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result = tok.getFloatingPointValue(
+      isNegative, floatType.getFloatSemantics(),
+      [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..c3abb1bd19f50 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,56 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal. `fromSign` starts at the
+/// (already consumed) leading sign and runs to the end of the buffer. On success
+/// returns a `floatliteral` (or `error`) token and advances `curPtr`; otherwise
+/// returns std::nullopt with `curPtr` unchanged so the caller can fall back to a
+/// plain `plus`/`minus` token. The sign is mandatory and folded into the token
+/// spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+
+  // Split off the alphabetic keyword (inf/qnan/nan/snan) that follows the sign.
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() &&
+        (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+         rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    // take_until returns all of `rest` when there is no ')'.
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1; // Consume the payload and the ')'.
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +379,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..e4fc46b48107e 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(
+        isNegative, semantics, [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..f15b23aae3d0b 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,66 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat>
+Token::getFloatingPointValue(bool isNegative,
+                             const llvm::fltSemantics &semantics,
+                             function_ref<InFlightDiagnostic()> emitError) const {
+  // A float literal carries its sign from exactly one place. Decimal and C-style
+  // hexadecimal literals have no sign in their spelling; a negative one is a
+  // separate '-' token, delivered here via `isNegative`. inf/NaN literals instead
+  // fold a mandatory sign into the spelling (`+inf`, `-nan(0x1)`, ...). A '-'
+  // token in front of an already-signed spelling would be a doubly-signed
+  // literal (e.g. `-+inf`), which is rejected. The leading '+' is dropped because
+  // APFloat::convertFromString accepts `+inf` but not a leading '+' on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "invalid floating point literal";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // A negative literal is only valid for a type with a sign representation;
+  // reject it for unsigned formats (e.g. f8E8M0FNU) rather than let APFloat
+  // abort later when the sign is materialized.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point value not valid for specified type";
+    return std::nullopt;
+  }
+
+  // A special value is only valid for a type that can represent it. Guard here
+  // rather than let APFloat abort (makeInf/makeNaN llvm_unreachable) on types
+  // with no Inf/NaN encoding (e.g. the FiniteOnly f4/f6/f8 formats).
+  bool wantsInf = str == "inf";
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if ((wantsInf && !APFloat::semanticsHasInf(semantics)) ||
+      (wantsNaN && !APFloat::semanticsHasNaN(semantics))) {
+    emitError() << "floating point value not valid for specified type";
+    return std::nullopt;
+  }
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics: this covers decimal,
+  // C-style hexadecimal (`0x1.fp13`), inf, and `nan(0x..)`/`snan(0x..)` forms,
+  // preserving NaN payloads. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..6b55fe69f4f11 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,46 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
+  // Print special values in a human-readable, typeless form. The sign is always
+  // included, mirroring LLVM's textual float literals. Because these forms carry
+  // no type, signal the caller (via printedHex) to keep the trailing type.
+  if (apValue.isInfinity()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+  if (apValue.isNaN()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value. A type with no mantissa bits
+    // (e.g. f8E8M0FNU) has a zero-width payload with no quiet/payload bits to
+    // distinguish, so it prints as the canonical quiet NaN.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+    } else {
+      if (apValue.isSignaling())
+        os << 's';
+      os << "nan(";
+      // Clear the signaling/quiet bit and trim leading zeros for output.
+      payload.clearBit(payload.getBitWidth() - 1);
+      SmallVector<char, 16> str;
+      payload.trunc(std::max(payload.getActiveBits(), 1u))
+          .toString(str, /*Radix=*/16, /*Signed=*/false,
+                    /*formatAsCLiteral=*/true);
+      os << str << ')';
+    }
+    return;
+  }
+
   // We would like to output the FP constant value in exponential notation,
   // but we cannot do this if doing so will lose precision.  Check here to
   // make sure that we only output it in exponential format if we can parse
   // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..55610b10ac38e 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,91 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // A '-' token in front of an already-signed inf/NaN literal is a doubly-signed
+  // literal and must be rejected.
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN is a FiniteOnly format with no Inf/NaN encoding; requesting inf
+  // must be rejected, not crash APFloat.
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN is a FiniteOnly format with no NaN encoding.
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has a NaN encoding but no infinities; inf must be rejected.
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is an unsigned format; a negative literal must be rejected, not
+  // materialized and crash the printer.
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:24:40 2026
From: llvmlistbot at llvm.org (Ming Yan)
Date: Mon, 20 Jul 2026 07:24:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][LICM] Hoist loop-invariant load
 operations using alias analysis. (PR #193166)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/193166@github.com>
Message-ID: <6a5e2fa8.481afadf.9e047.f09d@mx.google.com>


NexMing wrote:

ping

https://github.com/llvm/llvm-project/pull/193166

From llvmlistbot at llvm.org  Mon Jul 20 07:25:32 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 07:25:32 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e2fdc.f1c1923d.2d2b59.0a5f@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 7101324470491ec717cc40f6231858c0ba2b659f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  75 +++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  68 +++++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  39 +++++-
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  99 +++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 565 insertions(+), 164 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..7c0778b7c3f65 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result = tok.getFloatingPointValue(
+      isNegative, floatType.getFloatSemantics(),
+      [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..c3abb1bd19f50 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,56 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal. `fromSign` starts at the
+/// (already consumed) leading sign and runs to the end of the buffer. On success
+/// returns a `floatliteral` (or `error`) token and advances `curPtr`; otherwise
+/// returns std::nullopt with `curPtr` unchanged so the caller can fall back to a
+/// plain `plus`/`minus` token. The sign is mandatory and folded into the token
+/// spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+
+  // Split off the alphabetic keyword (inf/qnan/nan/snan) that follows the sign.
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() &&
+        (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+         rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    // take_until returns all of `rest` when there is no ')'.
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1; // Consume the payload and the ')'.
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +379,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..e4fc46b48107e 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(
+        isNegative, semantics, [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..699812e92d0a6 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,68 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat>
+Token::getFloatingPointValue(bool isNegative,
+                             const llvm::fltSemantics &semantics,
+                             function_ref<InFlightDiagnostic()> emitError) const {
+  // A float literal carries its sign from exactly one place. Decimal and C-style
+  // hexadecimal literals have no sign in their spelling; a negative one is a
+  // separate '-' token, delivered here via `isNegative`. inf/NaN literals instead
+  // fold a mandatory sign into the spelling (`+inf`, `-nan(0x1)`, ...). A '-'
+  // token in front of an already-signed spelling would be a doubly-signed
+  // literal (e.g. `-+inf`), which is rejected. The leading '+' is dropped because
+  // APFloat::convertFromString accepts `+inf` but not a leading '+' on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // A negative literal is only valid for a type with a sign representation;
+  // reject it for unsigned formats (e.g. f8E8M0FNU) rather than let APFloat
+  // abort later when the sign is materialized.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+
+  // A special value is only valid for a type that can represent it. Guard here
+  // rather than let APFloat abort (makeInf/makeNaN llvm_unreachable) on types
+  // with no Inf/NaN encoding (e.g. the FiniteOnly f4/f6/f8 formats).
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics: this covers decimal,
+  // C-style hexadecimal (`0x1.fp13`), inf, and `nan(0x..)`/`snan(0x..)` forms,
+  // preserving NaN payloads. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..6b55fe69f4f11 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,46 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
+  // Print special values in a human-readable, typeless form. The sign is always
+  // included, mirroring LLVM's textual float literals. Because these forms carry
+  // no type, signal the caller (via printedHex) to keep the trailing type.
+  if (apValue.isInfinity()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+  if (apValue.isNaN()) {
+    if (printedHex)
+      *printedHex = true;
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value. A type with no mantissa bits
+    // (e.g. f8E8M0FNU) has a zero-width payload with no quiet/payload bits to
+    // distinguish, so it prints as the canonical quiet NaN.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+    } else {
+      if (apValue.isSignaling())
+        os << 's';
+      os << "nan(";
+      // Clear the signaling/quiet bit and trim leading zeros for output.
+      payload.clearBit(payload.getBitWidth() - 1);
+      SmallVector<char, 16> str;
+      payload.trunc(std::max(payload.getActiveBits(), 1u))
+          .toString(str, /*Radix=*/16, /*Signed=*/false,
+                    /*formatAsCLiteral=*/true);
+      os << str << ')';
+    }
+    return;
+  }
+
   // We would like to output the FP constant value in exponential notation,
   // but we cannot do this if doing so will lose precision.  Check here to
   // make sure that we only output it in exponential format if we can parse
   // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..2e3d315b45e7f 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,91 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // A '-' token in front of an already-signed inf/NaN literal is a doubly-signed
+  // literal and must be rejected.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN is a FiniteOnly format with no Inf/NaN encoding; requesting inf
+  // must be rejected, not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN is a FiniteOnly format with no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has a NaN encoding but no infinities; inf must be rejected.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is an unsigned format; a negative literal must be rejected, not
+  // materialized and crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:34:20 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 07:34:20 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e31ec.b1416da1.3242c8.f63f@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 615b9f30df35242796d7b84d6c330f94807a8543 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  75 +++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  68 +++++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  52 ++++++--
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  99 +++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 569 insertions(+), 173 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..7c0778b7c3f65 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result = tok.getFloatingPointValue(
+      isNegative, floatType.getFloatSemantics(),
+      [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..c3abb1bd19f50 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,56 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal. `fromSign` starts at the
+/// (already consumed) leading sign and runs to the end of the buffer. On success
+/// returns a `floatliteral` (or `error`) token and advances `curPtr`; otherwise
+/// returns std::nullopt with `curPtr` unchanged so the caller can fall back to a
+/// plain `plus`/`minus` token. The sign is mandatory and folded into the token
+/// spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+
+  // Split off the alphabetic keyword (inf/qnan/nan/snan) that follows the sign.
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() &&
+        (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+         rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    // take_until returns all of `rest` when there is no ')'.
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1; // Consume the payload and the ')'.
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +379,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..e4fc46b48107e 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(
+        isNegative, semantics, [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..699812e92d0a6 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,68 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat>
+Token::getFloatingPointValue(bool isNegative,
+                             const llvm::fltSemantics &semantics,
+                             function_ref<InFlightDiagnostic()> emitError) const {
+  // A float literal carries its sign from exactly one place. Decimal and C-style
+  // hexadecimal literals have no sign in their spelling; a negative one is a
+  // separate '-' token, delivered here via `isNegative`. inf/NaN literals instead
+  // fold a mandatory sign into the spelling (`+inf`, `-nan(0x1)`, ...). A '-'
+  // token in front of an already-signed spelling would be a doubly-signed
+  // literal (e.g. `-+inf`), which is rejected. The leading '+' is dropped because
+  // APFloat::convertFromString accepts `+inf` but not a leading '+' on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // A negative literal is only valid for a type with a sign representation;
+  // reject it for unsigned formats (e.g. f8E8M0FNU) rather than let APFloat
+  // abort later when the sign is materialized.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+
+  // A special value is only valid for a type that can represent it. Guard here
+  // rather than let APFloat abort (makeInf/makeNaN llvm_unreachable) on types
+  // with no Inf/NaN encoding (e.g. the FiniteOnly f4/f6/f8 formats).
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics: this covers decimal,
+  // C-style hexadecimal (`0x1.fp13`), inf, and `nan(0x..)`/`snan(0x..)` forms,
+  // preserving NaN payloads. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..39f80d73cc9b6 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,10 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  // A finite value is printed in exponential notation when that round-trips
+  // losslessly, otherwise in APFloat's default decimal form. This is the only
+  // form that carries the type implicitly, so the caller may elide it.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2292,45 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The remaining forms (inf, NaN, and the hexadecimal bit-pattern fallback for
+  // finite values that do not round-trip) are typeless, so signal the caller to
+  // keep the trailing type. The sign is always included, mirroring LLVM's
+  // textual float literals.
   if (printedHex)
     *printedHex = true;
+
+  if (apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value. A type with no mantissa bits
+    // (e.g. f8E8M0FNU) has a zero-width payload with no quiet/payload bits to
+    // distinguish, so it prints as the canonical quiet NaN.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Clear the signaling/quiet bit and trim leading zeros for output.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false, /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern (the sign bit is part of the bits).
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..2e3d315b45e7f 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,91 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // A '-' token in front of an already-signed inf/NaN literal is a doubly-signed
+  // literal and must be rejected.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN is a FiniteOnly format with no Inf/NaN encoding; requesting inf
+  // must be rejected, not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN is a FiniteOnly format with no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has a NaN encoding but no infinities; inf must be rejected.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is an unsigned format; a negative literal must be rejected, not
+  // materialized and crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:34:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 07:34:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix llvm-mlir-use-after-erase findings (PR
	#210733)
Message-ID: <llvm.org/llvm/llvm-project/pull/210733@github.com>


https://github.com/Harald-R created https://github.com/llvm/llvm-project/pull/210733

Fix the following warnings identified by the `llvm-mlir-use-after-erase` check from https://github.com/llvm/llvm-project/pull/210727:

```cpp
mlir/lib/Transforms/Utils/DialectConversion.cpp:2700:33: warning: operation 'op' is used after it was erased [llvm-mlir-use-after-erase]
 2700 |           curState, std::string(op->getName().getStringRef()) + " folder");
      |                                 ^
mlir/lib/Transforms/Utils/DialectConversion.cpp:2685:12: note: operation erased here
 2685 |   rewriter.replaceOp(op, replacementValues);
      |            ^
mlir/lib/Bytecode/Reader/BytecodeReader.cpp:1654:32: warning: operation 'op' is used after it was erased [llvm-mlir-use-after-erase]
 1654 |       lazyLoadableOpsMap.erase(op);
      |                                ^
mlir/lib/Bytecode/Reader/BytecodeReader.cpp:1652:11: note: operation erased here
 1652 |       op->erase();
      |           ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:313:32: warning: operation 'newCallOp' is used after it was erased [llvm-mlir-use-after-erase]
  313 |     userOp->replaceAllUsesWith(newCallOp);
      |                                ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:304:20: note: operation erased here 
  304 |         newCallOp->erase();
      |                    ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:313:32: note: the use happens in a later loop iteration than the erase
  313 |     userOp->replaceAllUsesWith(newCallOp);
      |                                ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:440:29: warning: operation 'newOp' is used after it was erased [llvm-mlir-use-after-erase]
  440 |           Value newMemRef = newOp->getResult(resIndex);
      |                             ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:459:20: note: operation erased here 
  459 |             newOp->erase();
      |                    ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:440:29: note: the use happens in a later loop iteration than the erase
  440 |           Value newMemRef = newOp->getResult(resIndex);
      |                             ^
```

The warning reported at `NormalizeMemRefs.cpp:313` is technically a false-positive, as the `replacingMemRefUsesFailed` flag is aligned with the lifetime of `newCallOp`. The other similar warning in the file, at `NormalizeMemRefs.cpp:440`, is however a true-positive, as the iteration continues to the next element in the loop after erasure instead of breaking the loop's execution; the erased operation could be accessed in this case. To avoid this potential issue, the operation is erased after the loop ends for both cases, if the flag is active.

>From e8b885a450c566cd9df23cd2e50b26ab50b7ce8f Mon Sep 17 00:00:00 2001
From: Harald-R <rotuna.razvan at gmail.com>
Date: Mon, 20 Jul 2026 12:52:09 +0300
Subject: [PATCH] Fix llvm-mlir-use-after-erase findings

---
 mlir/lib/Bytecode/Reader/BytecodeReader.cpp   |  4 ++--
 .../MemRef/Transforms/NormalizeMemRefs.cpp    | 20 ++++++++++---------
 .../Transforms/Utils/DialectConversion.cpp    |  3 +--
 3 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
index 96ead28d0aa40..5c54738dbed96 100644
--- a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
+++ b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
@@ -1648,10 +1648,10 @@ class mlir::BytecodeReader::Impl {
           return failure();
         continue;
       }
-      op->dropAllReferences();
-      op->erase();
       lazyLoadableOps.pop_front();
       lazyLoadableOpsMap.erase(op);
+      op->dropAllReferences();
+      op->erase();
     }
     return success();
   }
diff --git a/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp b/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
index fa7991e6c6a80..23591dd045f78 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
@@ -301,14 +301,15 @@ void NormalizeMemRefs::updateFunctionSignature(func::FuncOp funcOp,
         // If it failed (due to escapes for example), bail out.
         // It should never hit this part of the code because it is called by
         // only those functions which are normalizable.
-        newCallOp->erase();
         replacingMemRefUsesFailed = true;
         break;
       }
       returnTypeChanged = true;
     }
-    if (replacingMemRefUsesFailed)
+    if (replacingMemRefUsesFailed) {
+      newCallOp->erase();
       continue;
+    }
     // Replace all uses for other non-memref result types.
     userOp->replaceAllUsesWith(newCallOp);
     userOp->erase();
@@ -456,17 +457,18 @@ void NormalizeMemRefs::normalizeFuncOpMemRefs(func::FuncOp funcOp,
                                               /*userFilterFn=*/nullptr,
                                               /*allowNonDereferencingOps=*/true,
                                               /*replaceInDeallocOp=*/true))) {
-            newOp->erase();
             replacingMemRefUsesFailed = true;
-            continue;
+            break;
           }
         }
-        if (!replacingMemRefUsesFailed) {
-          // Replace other ops with new op and delete the old op when the
-          // replacement succeeded.
-          op->replaceAllUsesWith(newOp);
-          op->erase();
+        if (replacingMemRefUsesFailed) {
+          newOp->erase();
+          return;
         }
+        // Replace other ops with new op and delete the old op when the
+        // replacement succeeded.
+        op->replaceAllUsesWith(newOp);
+        op->erase();
       }
     }
   });
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..c8db49c226e43 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -2696,8 +2696,7 @@ LogicalResult OperationLegalizer::legalizeWithFold(Operation *op) {
             "op '" + opName +
             "' folder rollback of IR modifications requested");
       }
-      rewriterImpl.resetState(
-          curState, std::string(op->getName().getStringRef()) + " folder");
+      rewriterImpl.resetState(curState, std::string(opName) + " folder");
       return failure();
     }
   }


From llvmlistbot at llvm.org  Mon Jul 20 07:41:54 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 07:41:54 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e33b2.1905e68e.30d957.e21e@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 3bff145241c32355b33d8a6e9746cc8c7f4e7e13 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  74 +++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  68 +++++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  53 ++++++--
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  99 +++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 569 insertions(+), 173 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..9fe175540ba5c 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,55 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special floating point literal. `fromSign` starts at the
+/// (already consumed) leading sign and runs to the end of the buffer. On
+/// success returns a `floatliteral` (or `error`) token and advances `curPtr`;
+/// otherwise returns std::nullopt with `curPtr` unchanged so the caller can
+/// fall back to a plain `plus`/`minus` token. The sign is mandatory and folded
+/// into the token spelling, mirroring LLVM's textual float literals.
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan(0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+
+  // Split off the alphabetic keyword (inf/qnan/nan/snan) that follows the sign.
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Bare special values: `inf` and `qnan`. Reject when followed by more
+  // identifier characters so e.g. `+infty` stays a `plus` and an identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() && (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+                          rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // NaN with a hexadecimal payload: `nan(0x..)` / `snan(0x..)`. The payload is
+  // validated later by APFloat; here we only capture the token, spanning the
+  // parentheses.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    // take_until returns all of `rest` when there is no ')'.
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1; // Consume the payload and the ')'.
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +378,29 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // Handle C-style hexadecimal floating point literals, distinguished from
+    // the hexadecimal bit-pattern form by a fractional part and/or a binary
+    // exponent: `0x1.fp13`, `0x1p4`. The binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..8ca3971ffc931 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,68 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // A float literal carries its sign from exactly one place. Decimal and
+  // C-style hexadecimal literals have no sign in their spelling; a negative one
+  // is a separate '-' token, delivered here via `isNegative`. inf/NaN literals
+  // instead fold a mandatory sign into the spelling (`+inf`, `-nan(0x1)`, ...).
+  // A '-' token in front of an already-signed spelling would be a doubly-signed
+  // literal (e.g. `-+inf`), which is rejected. The leading '+' is dropped
+  // because APFloat::convertFromString accepts `+inf` but not a leading '+' on
+  // NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // A negative literal is only valid for a type with a sign representation;
+  // reject it for unsigned formats (e.g. f8E8M0FNU) rather than let APFloat
+  // abort later when the sign is materialized.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+
+  // A special value is only valid for a type that can represent it. Guard here
+  // rather than let APFloat abort (makeInf/makeNaN llvm_unreachable) on types
+  // with no Inf/NaN encoding (e.g. the FiniteOnly f4/f6/f8 formats).
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // APFloat::convertFromString does not understand "qnan"; build it directly.
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build the value directly in the target semantics: this covers decimal,
+  // C-style hexadecimal (`0x1.fp13`), inf, and `nan(0x..)`/`snan(0x..)` forms,
+  // preserving NaN payloads. Overflow/underflow are tolerated (yielding
+  // inf/zero), matching the historical behavior of decimal literals.
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..26e941daaebf2 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,10 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  // A finite value is printed in exponential notation when that round-trips
+  // losslessly, otherwise in APFloat's default decimal form. This is the only
+  // form that carries the type implicitly, so the caller may elide it.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2292,46 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The remaining forms (inf, NaN, and the hexadecimal bit-pattern fallback for
+  // finite values that do not round-trip) are typeless, so signal the caller to
+  // keep the trailing type. The sign is always included, mirroring LLVM's
+  // textual float literals.
   if (printedHex)
     *printedHex = true;
+
+  if (apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // The quiet bit is the highest bit of the payload, so the preferred quiet
+    // NaN payload is exactly the sign-mask value. A type with no mantissa bits
+    // (e.g. f8E8M0FNU) has a zero-width payload with no quiet/payload bits to
+    // distinguish, so it prints as the canonical quiet NaN.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Clear the signaling/quiet bit and trim leading zeros for output.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern (the sign bit is part of the bits).
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..2e3d315b45e7f 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,17 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The "not valid for specified type" diagnostic must point at the float literal
+// itself, not at whatever token happens to follow its type (here the `}` sits on
+// the next line, so a stale location would be reported on the wrong line).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +686,91 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // A '-' token in front of an already-signed inf/NaN literal is a doubly-signed
+  // literal and must be rejected.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN is a FiniteOnly format with no Inf/NaN encoding; requesting inf
+  // must be rejected, not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN is a FiniteOnly format with no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has a NaN encoding but no infinities; inf must be rejected.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is an unsigned format; a negative literal must be rejected, not
+  // materialized and crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:49:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 07:49:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix llvm-mlir-use-after-erase findings (PR
	#210733)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210733@github.com>
Message-ID: <6a5e355c.743400b0.23a7b2.e408@mx.google.com>


https://github.com/Harald-R ready_for_review https://github.com/llvm/llvm-project/pull/210733

From llvmlistbot at llvm.org  Mon Jul 20 07:49:16 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 07:49:16 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e356c.4987c784.2c60a9.5715@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 6580f220e3a31429ca63a67040f0b986b654edac Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  64 ++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  55 +++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  47 +++++--
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  95 ++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 536 insertions(+), 173 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..3191b9e2b1d92 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,46 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special float literal (inf/NaN) from `fromSign`, which
+/// starts at the already-consumed sign. On success returns a `floatliteral`
+/// (or `error`) token and advances `curPtr`, else std::nullopt (curPtr intact).
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan(0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Reject `inf`/`qnan` that is only a prefix of a longer identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() && (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+                          rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // `nan(0x..)` / `snan(0x..)`: capture through ')'; APFloat validates later.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1;
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +369,28 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // C-style hex float (`0x1.fp13`): a '.'/'p' distinguishes it from the
+    // bit-pattern form; the binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..835cd892c9a19 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,55 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // The sign is either a preceding '-' token (isNegative) or folded into an
+  // inf/NaN spelling; both at once (e.g. `-+inf`) is invalid. Drop a leading
+  // '+' since convertFromString rejects it on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // Reject values the type cannot represent; APFloat would otherwise abort.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // convertFromString does not accept "qnan".
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build in the target semantics to preserve NaN payloads; overflow/underflow
+  // are tolerated (inf/zero).
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..e14639c058e46 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,9 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  // Finite values print as decimal; this is the only form whose type may be
+  // elided.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2291,41 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The rest (inf, NaN, hex bit-pattern) are typeless; keep the type.
   if (printedHex)
     *printedHex = true;
+
+  if (apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // Preferred quiet NaN = only the quiet (sign-mask) bit set. A zero-width
+    // payload (no mantissa bits, e.g. f8E8M0FNU) has none, so print qnan.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Drop the quiet/signaling bit; trim leading zeros.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern.
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..03a25f43f09b7 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,16 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The diagnostic must point at the float literal, not the following token
+// (the `}` is on the next line, exposing a stale location).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +685,88 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // Doubly-signed literal: a '-' token before an already-signed inf/NaN.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN has no Inf encoding: reject, do not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN has no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has NaN but no Inf.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: reject negatives, do not crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 07:53:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 07:53:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix llvm-mlir-use-after-erase findings (PR
	#210733)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210733@github.com>
Message-ID: <6a5e3669.d76d81af.23b347.94df@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-memref

Author: Harald-R

<details>
<summary>Changes</summary>

Fix the following warnings identified by the `llvm-mlir-use-after-erase` check from https://github.com/llvm/llvm-project/pull/210727:

```cpp
mlir/lib/Transforms/Utils/DialectConversion.cpp:2700:33: warning: operation 'op' is used after it was erased [llvm-mlir-use-after-erase]
 2700 |           curState, std::string(op-&gt;getName().getStringRef()) + " folder");
      |                                 ^
mlir/lib/Transforms/Utils/DialectConversion.cpp:2685:12: note: operation erased here
 2685 |   rewriter.replaceOp(op, replacementValues);
      |            ^
mlir/lib/Bytecode/Reader/BytecodeReader.cpp:1654:32: warning: operation 'op' is used after it was erased [llvm-mlir-use-after-erase]
 1654 |       lazyLoadableOpsMap.erase(op);
      |                                ^
mlir/lib/Bytecode/Reader/BytecodeReader.cpp:1652:11: note: operation erased here
 1652 |       op-&gt;erase();
      |           ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:313:32: warning: operation 'newCallOp' is used after it was erased [llvm-mlir-use-after-erase]
  313 |     userOp-&gt;replaceAllUsesWith(newCallOp);
      |                                ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:304:20: note: operation erased here 
  304 |         newCallOp-&gt;erase();
      |                    ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:313:32: note: the use happens in a later loop iteration than the erase
  313 |     userOp-&gt;replaceAllUsesWith(newCallOp);
      |                                ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:440:29: warning: operation 'newOp' is used after it was erased [llvm-mlir-use-after-erase]
  440 |           Value newMemRef = newOp-&gt;getResult(resIndex);
      |                             ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:459:20: note: operation erased here 
  459 |             newOp-&gt;erase();
      |                    ^
mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp:440:29: note: the use happens in a later loop iteration than the erase
  440 |           Value newMemRef = newOp-&gt;getResult(resIndex);
      |                             ^
```

The warning reported at `NormalizeMemRefs.cpp:313` is technically a false-positive, as the `replacingMemRefUsesFailed` flag is aligned with the lifetime of `newCallOp`. The other similar warning in the file, at `NormalizeMemRefs.cpp:440`, is however a true-positive, as the iteration continues to the next element in the loop after erasure instead of breaking the loop's execution; the erased operation could be accessed in this case. To avoid this potential issue, the operation is erased after the loop ends for both cases, if the flag is active.

---
Full diff: https://github.com/llvm/llvm-project/pull/210733.diff


3 Files Affected:

- (modified) mlir/lib/Bytecode/Reader/BytecodeReader.cpp (+2-2) 
- (modified) mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp (+11-9) 
- (modified) mlir/lib/Transforms/Utils/DialectConversion.cpp (+1-2) 


``````````diff
diff --git a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
index 96ead28d0aa40..5c54738dbed96 100644
--- a/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
+++ b/mlir/lib/Bytecode/Reader/BytecodeReader.cpp
@@ -1648,10 +1648,10 @@ class mlir::BytecodeReader::Impl {
           return failure();
         continue;
       }
-      op->dropAllReferences();
-      op->erase();
       lazyLoadableOps.pop_front();
       lazyLoadableOpsMap.erase(op);
+      op->dropAllReferences();
+      op->erase();
     }
     return success();
   }
diff --git a/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp b/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
index fa7991e6c6a80..23591dd045f78 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/NormalizeMemRefs.cpp
@@ -301,14 +301,15 @@ void NormalizeMemRefs::updateFunctionSignature(func::FuncOp funcOp,
         // If it failed (due to escapes for example), bail out.
         // It should never hit this part of the code because it is called by
         // only those functions which are normalizable.
-        newCallOp->erase();
         replacingMemRefUsesFailed = true;
         break;
       }
       returnTypeChanged = true;
     }
-    if (replacingMemRefUsesFailed)
+    if (replacingMemRefUsesFailed) {
+      newCallOp->erase();
       continue;
+    }
     // Replace all uses for other non-memref result types.
     userOp->replaceAllUsesWith(newCallOp);
     userOp->erase();
@@ -456,17 +457,18 @@ void NormalizeMemRefs::normalizeFuncOpMemRefs(func::FuncOp funcOp,
                                               /*userFilterFn=*/nullptr,
                                               /*allowNonDereferencingOps=*/true,
                                               /*replaceInDeallocOp=*/true))) {
-            newOp->erase();
             replacingMemRefUsesFailed = true;
-            continue;
+            break;
           }
         }
-        if (!replacingMemRefUsesFailed) {
-          // Replace other ops with new op and delete the old op when the
-          // replacement succeeded.
-          op->replaceAllUsesWith(newOp);
-          op->erase();
+        if (replacingMemRefUsesFailed) {
+          newOp->erase();
+          return;
         }
+        // Replace other ops with new op and delete the old op when the
+        // replacement succeeded.
+        op->replaceAllUsesWith(newOp);
+        op->erase();
       }
     }
   });
diff --git a/mlir/lib/Transforms/Utils/DialectConversion.cpp b/mlir/lib/Transforms/Utils/DialectConversion.cpp
index c76e3808d3b37..c8db49c226e43 100644
--- a/mlir/lib/Transforms/Utils/DialectConversion.cpp
+++ b/mlir/lib/Transforms/Utils/DialectConversion.cpp
@@ -2696,8 +2696,7 @@ LogicalResult OperationLegalizer::legalizeWithFold(Operation *op) {
             "op '" + opName +
             "' folder rollback of IR modifications requested");
       }
-      rewriterImpl.resetState(
-          curState, std::string(op->getName().getStringRef()) + " folder");
+      rewriterImpl.resetState(curState, std::string(opName) + " folder");
       return failure();
     }
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/210733

From llvmlistbot at llvm.org  Mon Jul 20 08:52:07 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Mon, 20 Jul 2026 08:52:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Enhance unrolling of convert
	layout (PR #209822)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209822@github.com>
Message-ID: <6a5e4427.507fc713.239434.96b9@mx.google.com>


https://github.com/nbpatel updated https://github.com/llvm/llvm-project/pull/209822

>From e9401f094f7163276fd348056d310a895236c428 Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Tue, 14 Jul 2026 21:43:37 +0000
Subject: [PATCH 1/2] Enhance unrolling of convert layout

---
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 124 ++++++++++++++++--
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   |  32 ++++-
 2 files changed, 137 insertions(+), 19 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..bcb9946373298 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -957,8 +957,92 @@ struct UnrollStoreMatrixOp : public UnrollPattern<xegpu::StoreMatrixOp> {
 /// after inst_data stripped. If it does, it will unroll the vector into
 /// multiple smaller vectors according to the target shape, and create multiple
 /// ConvertLayoutOp with the unrolled vectors and the stripped layouts.
+///
+/// When the input and target layouts have different inst_data, the source is
+/// extracted at the input inst_data granularity and the result is inserted at
+/// the target inst_data granularity, enabling slice cancellation during
+/// canonicalization.
 struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
   using UnrollPattern<xegpu::ConvertLayoutOp>::UnrollPattern;
+
+  /// Extracts source in `inTile` slices, regroups into `convTile`-sized
+  /// ConvertLayoutOps, and inserts the result in `outTile` slices.
+  /// Returns failure if the tiles do not evenly divide.
+  LogicalResult
+  rewriteWithRegrouping(xegpu::ConvertLayoutOp op, VectorType valueTy,
+                        ArrayRef<int64_t> convTile, ArrayRef<int64_t> inTile,
+                        ArrayRef<int64_t> outTile,
+                        xegpu::DistributeLayoutAttr inputLayout,
+                        xegpu::DistributeLayoutAttr targetLayout, Location loc,
+                        PatternRewriter &rewriter) const {
+    ArrayRef<int64_t> vecShape = valueTy.getShape();
+    if (!computeShapeRatio(vecShape, convTile) ||
+        !computeShapeRatio(convTile, inTile) ||
+        !computeShapeRatio(convTile, outTile))
+      return failure();
+
+    Type elemTy = valueTy.getElementType();
+    int64_t rank = valueTy.getRank();
+    VectorType convTy = VectorType::get(convTile, elemTy);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value source = op.getSource();
+    auto zeroOf = [&](VectorType ty) -> Value {
+      return arith::ConstantOp::create(
+          rewriter, loc, ty,
+          DenseElementsAttr::get(ty, rewriter.getZeroAttr(elemTy)));
+    };
+    auto addOffsets = [](ArrayRef<int64_t> a,
+                         ArrayRef<int64_t> b) -> SmallVector<int64_t> {
+      SmallVector<int64_t> res(a);
+      for (auto [r, v] : llvm::zip_equal(res, b))
+        r += v;
+      return res;
+    };
+
+    Value result = zeroOf(valueTy);
+    for (SmallVector<int64_t> convOff :
+         StaticTileOffsetRange(vecShape, convTile)) {
+      // Build the convert tile from inTile-sized slices of the source.
+      Value conv;
+      if (convTile == inTile) {
+        conv = vector::ExtractStridedSliceOp::create(
+            rewriter, loc, source, convOff, convTile, strides);
+      } else {
+        conv = zeroOf(convTy);
+        for (SmallVector<int64_t> inLocal :
+             StaticTileOffsetRange(convTile, inTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, source, addOffsets(convOff, inLocal), inTile,
+              strides);
+          conv = vector::InsertStridedSliceOp::create(rewriter, loc, piece,
+                                                      conv, inLocal, strides);
+        }
+      }
+
+      conv = xegpu::ConvertLayoutOp::create(rewriter, loc, convTy, conv,
+                                            inputLayout, targetLayout);
+
+      // Write the converted tile into the result as outTile-sized slices.
+      if (convTile == outTile) {
+        result = vector::InsertStridedSliceOp::create(rewriter, loc, conv,
+                                                      result, convOff, strides);
+      } else {
+        for (SmallVector<int64_t> outLocal :
+             StaticTileOffsetRange(convTile, outTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, conv, outLocal, outTile, strides);
+          result = vector::InsertStridedSliceOp::create(
+              rewriter, loc, piece, result, addOffsets(convOff, outLocal),
+              strides);
+        }
+      }
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+
   LogicalResult matchAndRewrite(xegpu::ConvertLayoutOp op,
                                 PatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
@@ -974,8 +1058,10 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
       return success();
     }
 
-    if (inputLayout.getEffectiveInstDataAsInt().empty() ||
-        targetLayout.getEffectiveInstDataAsInt().empty())
+    // Capture inst_data granularities before stripping them.
+    SmallVector<int64_t> inTile = inputLayout.getEffectiveInstDataAsInt();
+    SmallVector<int64_t> outTile = targetLayout.getEffectiveInstDataAsInt();
+    if (inTile.empty() || outTile.empty())
       return rewriter.notifyMatchFailure(op, "Not a target ConvertLayoutOp.");
 
     inputLayout = inputLayout.dropInstData();
@@ -988,20 +1074,30 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     if (!targetShape || targetShape->size() != (size_t)valueTy.getRank())
       return failure();
 
-    Value newSource = op.getSource();
+    // Nothing to convert if layouts match after stripping inst_data.
+    if (!inputLayout || !targetLayout || inputLayout.isEqualTo(targetLayout)) {
+      rewriter.replaceOp(op, op.getSource());
+      return success();
+    }
+
+    // Try regrouping: extract at inTile, convert, insert at outTile.
+    if (succeeded(rewriteWithRegrouping(op, valueTy, *targetShape, inTile,
+                                        outTile, inputLayout, targetLayout, loc,
+                                        rewriter)))
+      return success();
+
+    // Fallback: pack/unpack at the convert tile granularity.
+    SmallVector<Type> convertedValTypes =
+        getUnrolledTypes(valueTy, *targetShape);
+    SmallVector<Value> convertedValues =
+        pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
     SmallVector<Value> newOps;
-    if (inputLayout && targetLayout && !inputLayout.isEqualTo(targetLayout)) {
-      SmallVector<Type> convertedValTypes =
-          getUnrolledTypes(valueTy, *targetShape);
-      SmallVector<Value> convertedValues =
-          pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
-      for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
-        auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
-                                                    inputLayout, targetLayout);
-        newOps.push_back(newOp);
-      }
-      newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
+    for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
+      auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
+                                                  inputLayout, targetLayout);
+      newOps.push_back(newOp);
     }
+    Value newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
 
     rewriter.replaceOp(op, newSource);
     return success();
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..27a88efcf443c 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -463,19 +463,19 @@ gpu.module @test_kernel {
   //CHECK: gpu.func @convert_layout([[arg0:%.+]]: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
   //CHECK: [[cst:%.+]] = arith.constant dense<0.000000e+00> : vector<8x32x2xf16>
   //CHECK: [[e0:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [0, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
-  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
   //CHECK: [[c0:%.+]] = xegpu.convert_layout [[e0]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
-  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
   //CHECK: [[e2:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
+  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
+  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m0:%.+]] = math.exp [[e2]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i0:%.+]] = vector.insert_strided_slice [[m0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m1:%.+]] = math.exp [[e3]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i1:%.+]] = vector.insert_strided_slice [[m1]], [[i0]] {offsets = [0, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m2:%.+]] = math.exp [[e4]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i2:%.+]] = vector.insert_strided_slice [[m2]], [[i1]] {offsets = [4, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m3:%.+]] = math.exp [[e5]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i3:%.+]] = vector.insert_strided_slice [[m3]], [[i2]] {offsets = [4, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
   //CHECK: gpu.return [[i3]] : vector<8x32x2xf16>
@@ -488,6 +488,28 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 16] vs [8, 64]).
+#in = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#tgt = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 16]
+  //CHECK: math.exp {{.*}} : vector<8x16xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 64]
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<8x64xbf16>
+  //CHECK: xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x64xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix

>From 41c298a78f70f3c2727bae352876565063aa5fde Mon Sep 17 00:00:00 2001
From: nbpatel <nishant.b.patel at intel.com>
Date: Mon, 20 Jul 2026 04:46:26 +0000
Subject: [PATCH 2/2] Add test

---
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index 27a88efcf443c..bc50bad2b2302 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -510,6 +510,27 @@ gpu.module @test_kernel {
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 64] vs [8, 16]).
+#in = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+#tgt = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup_swapped
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 64], strides = [1, 1]} : vector<16x64xbf16> to vector<8x64xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 16]
+  //CHECK: [[cvt:%.+]] = xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.extract_strided_slice [[cvt]] {{.*}}sizes = [8, 16]{{.*}} : vector<8x64xbf16> to vector<8x16xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup_swapped(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix


From llvmlistbot at llvm.org  Mon Jul 20 09:47:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 09:47:40 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Emit unified declare
 globals as extern. (PR #210757)
Message-ID: <llvm.org/llvm/llvm-project/pull/210757@github.com>


https://github.com/nvptm created https://github.com/llvm/llvm-project/pull/210757

Emit external device symbols for OpenACC declare create and copyin globals in -gpu=mem:unified; preserve device definitions for device_resident globals.

>From 3b8fb7e41b597e46bdf353e3e80d289a691829cc Mon Sep 17 00:00:00 2001
From: nvpm <pmathew at nvidia.com>
Date: Fri, 17 Jul 2026 17:05:49 -0700
Subject: [PATCH 1/3] [acc] Emit unified declare globals as extern

---
 .../acc-declare-gpu-module-insertion.fir      | 23 +++++++++++++++++++
 .../mlir/Dialect/OpenACC/Transforms/Passes.td |  4 ++++
 .../ACCDeclareGPUModuleInsertion.cpp          | 17 +++++++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)
 create mode 100644 flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir

diff --git a/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
new file mode 100644
index 0000000000000..07e9f0ea02da2
--- /dev/null
+++ b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
@@ -0,0 +1,23 @@
+// RUN: fir-opt %s --pass-pipeline="builtin.module(acc-declare-gpu-module-insertion{cuda-unified=true})" | FileCheck %s
+
+// Globals registered as host variables must be declarations in device code.
+// device_resident globals remain device definitions.
+module attributes {fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", gpu.container_module} {
+  fir.global internal @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+  fir.global @initialized_shared(dense<1.0> : vector<1xf32>) {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+  fir.global @resident {acc.declare = #acc.declare<dataClause = acc_declare_device_resident>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+}
+
+// CHECK-LABEL: gpu.module @acc_gpu_module {
+// CHECK: fir.global @shared {{.*}} : !fir.array<7xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @initialized_shared {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @resident {{.*}} : !fir.array<7xf32> {
+// CHECK: fir.has_value
diff --git a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
index 04723c5db2d12..fbc832a37f478 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
@@ -157,6 +157,10 @@ def ACCDeclareGPUModuleInsertion : Pass<"acc-declare-gpu-module-insertion", "mli
     that device code can reference them.
   }];
   let dependentDialects = ["mlir::acc::OpenACCDialect", "mlir::gpu::GPUDialect"];
+  let options = [
+    Option<"cudaUnified", "cuda-unified", "bool", "false",
+           "Emit declaration-only device globals for CUDA unified memory.">
+  ];
 }
 
 def ACCLegalizeSerial : Pass<"acc-legalize-serial", "mlir::func::FuncOp"> {
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index f815245882630..0b02c731e9b6f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -77,6 +77,15 @@ static bool hasAccDeclareGlobals(ModuleOp mod) {
   return false;
 }
 
+static void makeDeviceGlobalDeclaration(Operation &globalOp) {
+  globalOp.removeAttr("initVal");
+  globalOp.removeAttr("linkName");
+  for (Region &region : globalOp.getRegions()) {
+    region.dropAllReferences();
+    region.getBlocks().clear();
+  }
+}
+
 class ACCDeclareGPUModuleInsertion
     : public acc::impl::ACCDeclareGPUModuleInsertionBase<
           ACCDeclareGPUModuleInsertion> {
@@ -116,7 +125,13 @@ class ACCDeclareGPUModuleInsertion
         continue;
       }
 
-      gpuSymTable.insert(globalOp.clone());
+      Operation *deviceGlobal = globalOp.clone();
+      auto declareAttr =
+          globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
+      if (cudaUnified && declareAttr.getDataClause().getValue() !=
+          acc::DataClause::acc_declare_device_resident)
+        makeDeviceGlobalDeclaration(*deviceGlobal);
+      gpuSymTable.insert(deviceGlobal);
     }
     return success();
   }

>From 5567e2ba9e971d5874ce745788e89a88875194e4 Mon Sep 17 00:00:00 2001
From: nvpm <pmathew at nvidia.com>
Date: Mon, 20 Jul 2026 07:50:06 -0700
Subject: [PATCH 2/3] Make unified declare global insertion idempotent

---
 .../OpenACC/acc-declare-gpu-module-insertion.fir |  4 ++--
 .../Transforms/ACCDeclareGPUModuleInsertion.cpp  | 16 +++++++++-------
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
index 07e9f0ea02da2..b27094467f9cd 100644
--- a/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
+++ b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
@@ -1,4 +1,4 @@
-// RUN: fir-opt %s --pass-pipeline="builtin.module(acc-declare-gpu-module-insertion{cuda-unified=true})" | FileCheck %s
+// RUN: fir-opt %s --pass-pipeline="builtin.module(acc-declare-gpu-module-insertion{cuda-unified=true},acc-declare-gpu-module-insertion{cuda-unified=true})" | FileCheck %s
 
 // Globals registered as host variables must be declarations in device code.
 // device_resident globals remain device definitions.
@@ -15,7 +15,7 @@ module attributes {fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", gpu.conta
 }
 
 // CHECK-LABEL: gpu.module @acc_gpu_module {
-// CHECK: fir.global @shared {{.*}} : !fir.array<7xf32>
+// CHECK: fir.global @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32>
 // CHECK-NOT: fir.has_value
 // CHECK: fir.global @initialized_shared {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
 // CHECK-NOT: fir.has_value
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index 0b02c731e9b6f..e6ba250927a7c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -106,6 +106,12 @@ class ACCDeclareGPUModuleInsertion
         continue;
 
       StringAttr name = symOp.getNameAttr();
+      Operation *deviceGlobal = globalOp.clone();
+      auto declareAttr =
+          globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
+      if (cudaUnified && declareAttr.getDataClause().getValue() !=
+          acc::DataClause::acc_declare_device_resident)
+        makeDeviceGlobalDeclaration(*deviceGlobal);
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
         // Reuse only when the existing GPU symbol is structurally equivalent to
@@ -113,24 +119,20 @@ class ACCDeclareGPUModuleInsertion
         // op type or different definition).
         if (existing->getName() != globalOp.getName() ||
             !OperationEquivalence::isEquivalentTo(
-                existing, &globalOp,
+                existing, deviceGlobal,
                 OperationEquivalence::ignoreValueEquivalence,
                 /*markEquivalent=*/nullptr,
                 OperationEquivalence::IgnoreLocations)) {
+          deviceGlobal->destroy();
           accSupport.emitNYI(globalOp.getLoc(),
                              llvm::Twine("duplicate global symbol '") +
                                  name.getValue() + "' in gpu module");
           return failure();
         }
+        deviceGlobal->destroy();
         continue;
       }
 
-      Operation *deviceGlobal = globalOp.clone();
-      auto declareAttr =
-          globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
-      if (cudaUnified && declareAttr.getDataClause().getValue() !=
-          acc::DataClause::acc_declare_device_resident)
-        makeDeviceGlobalDeclaration(*deviceGlobal);
       gpuSymTable.insert(deviceGlobal);
     }
     return success();

>From 99b3e270c780c5f0c2c0c22cfaa92e4283280f2b Mon Sep 17 00:00:00 2001
From: nvpm <pmathew at nvidia.com>
Date: Mon, 20 Jul 2026 07:55:10 -0700
Subject: [PATCH 3/3] Format

---
 .../Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index e6ba250927a7c..463cd5a5b8043 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -110,7 +110,7 @@ class ACCDeclareGPUModuleInsertion
       auto declareAttr =
           globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
       if (cudaUnified && declareAttr.getDataClause().getValue() !=
-          acc::DataClause::acc_declare_device_resident)
+                             acc::DataClause::acc_declare_device_resident)
         makeDeviceGlobalDeclaration(*deviceGlobal);
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {


From llvmlistbot at llvm.org  Mon Jul 20 09:48:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 09:48:23 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Emit unified declare
 globals as extern. (PR #210757)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210757@github.com>
Message-ID: <6a5e5157.6c5720b3.2da3b9.f196@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: nvptm

<details>
<summary>Changes</summary>

Emit external device symbols for OpenACC declare create and copyin globals in -gpu=mem:unified; preserve device definitions for device_resident globals.

---
Full diff: https://github.com/llvm/llvm-project/pull/210757.diff


3 Files Affected:

- (added) flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir (+23) 
- (modified) mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td (+4) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp (+19-2) 


``````````diff
diff --git a/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
new file mode 100644
index 0000000000000..b27094467f9cd
--- /dev/null
+++ b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
@@ -0,0 +1,23 @@
+// RUN: fir-opt %s --pass-pipeline="builtin.module(acc-declare-gpu-module-insertion{cuda-unified=true},acc-declare-gpu-module-insertion{cuda-unified=true})" | FileCheck %s
+
+// Globals registered as host variables must be declarations in device code.
+// device_resident globals remain device definitions.
+module attributes {fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", gpu.container_module} {
+  fir.global internal @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+  fir.global @initialized_shared(dense<1.0> : vector<1xf32>) {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+  fir.global @resident {acc.declare = #acc.declare<dataClause = acc_declare_device_resident>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+}
+
+// CHECK-LABEL: gpu.module @acc_gpu_module {
+// CHECK: fir.global @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @initialized_shared {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @resident {{.*}} : !fir.array<7xf32> {
+// CHECK: fir.has_value
diff --git a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
index 04723c5db2d12..fbc832a37f478 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
@@ -157,6 +157,10 @@ def ACCDeclareGPUModuleInsertion : Pass<"acc-declare-gpu-module-insertion", "mli
     that device code can reference them.
   }];
   let dependentDialects = ["mlir::acc::OpenACCDialect", "mlir::gpu::GPUDialect"];
+  let options = [
+    Option<"cudaUnified", "cuda-unified", "bool", "false",
+           "Emit declaration-only device globals for CUDA unified memory.">
+  ];
 }
 
 def ACCLegalizeSerial : Pass<"acc-legalize-serial", "mlir::func::FuncOp"> {
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index f815245882630..463cd5a5b8043 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -77,6 +77,15 @@ static bool hasAccDeclareGlobals(ModuleOp mod) {
   return false;
 }
 
+static void makeDeviceGlobalDeclaration(Operation &globalOp) {
+  globalOp.removeAttr("initVal");
+  globalOp.removeAttr("linkName");
+  for (Region &region : globalOp.getRegions()) {
+    region.dropAllReferences();
+    region.getBlocks().clear();
+  }
+}
+
 class ACCDeclareGPUModuleInsertion
     : public acc::impl::ACCDeclareGPUModuleInsertionBase<
           ACCDeclareGPUModuleInsertion> {
@@ -97,6 +106,12 @@ class ACCDeclareGPUModuleInsertion
         continue;
 
       StringAttr name = symOp.getNameAttr();
+      Operation *deviceGlobal = globalOp.clone();
+      auto declareAttr =
+          globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
+      if (cudaUnified && declareAttr.getDataClause().getValue() !=
+                             acc::DataClause::acc_declare_device_resident)
+        makeDeviceGlobalDeclaration(*deviceGlobal);
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
         // Reuse only when the existing GPU symbol is structurally equivalent to
@@ -104,19 +119,21 @@ class ACCDeclareGPUModuleInsertion
         // op type or different definition).
         if (existing->getName() != globalOp.getName() ||
             !OperationEquivalence::isEquivalentTo(
-                existing, &globalOp,
+                existing, deviceGlobal,
                 OperationEquivalence::ignoreValueEquivalence,
                 /*markEquivalent=*/nullptr,
                 OperationEquivalence::IgnoreLocations)) {
+          deviceGlobal->destroy();
           accSupport.emitNYI(globalOp.getLoc(),
                              llvm::Twine("duplicate global symbol '") +
                                  name.getValue() + "' in gpu module");
           return failure();
         }
+        deviceGlobal->destroy();
         continue;
       }
 
-      gpuSymTable.insert(globalOp.clone());
+      gpuSymTable.insert(deviceGlobal);
     }
     return success();
   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/210757

From llvmlistbot at llvm.org  Mon Jul 20 10:46:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 10:46:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] Support dynamic worker-private
	shared memory (PR #210770)
Message-ID: <llvm.org/llvm/llvm-project/pull/210770@github.com>


https://github.com/khaki3 created https://github.com/llvm/llvm-project/pull/210770

Use the maximum ThreadY width when the runtime worker count prevents exact static sizing.

>From b72758c6ac4c5eff9bff3548ccf56ff648582cb8 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 10:45:35 -0700
Subject: [PATCH] [OpenACC] Support dynamic worker-private shared memory

Use the maximum ThreadY width when the runtime worker count prevents exact static sizing.
---
 .../Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp  | 20 ++++++++-----------
 ...c-cg-to-gpu-worker-private-dynamic-nw.mlir | 11 +++++-----
 .../Dialect/OpenACC/OpenACCUtilsCGTest.cpp    | 16 ++++++++++-----
 3 files changed, 25 insertions(+), 22 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 5f4af6a5bcc5b..745b6fa348533 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -317,9 +317,10 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
   return parDims;
 }
 
-static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
-    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
-    bool isWorkerPrivate, OpenACCSupport *support) {
+static FailureOr<std::optional<int64_t>>
+getWorkerPrivateSharedMemoryNumCopies(PrivateLocalOp privateLocal,
+                                      ComputeRegionOp computeRegion,
+                                      bool isWorkerPrivate, OpenACCSupport *) {
   if (!isWorkerPrivate)
     return std::optional<int64_t>(1);
 
@@ -330,15 +331,10 @@ static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
     return std::optional<int64_t>();
 
   auto workerArgConst = workerArg->getDefiningOp<arith::ConstantIndexOp>();
-  if (!workerArgConst) {
-    if (support) {
-      (void)support->emitNYI(privateLocal.getLoc(),
-                             "worker-private variables in shared memory "
-                             "require compile-time constant num_workers");
-      return failure();
-    }
-    return std::optional<int64_t>();
-  }
+  // ThreadY is bounded by 32 after subgroup alignment. Use that upper bound
+  // when the exact worker count is known only at runtime.
+  if (!workerArgConst)
+    return std::optional<int64_t>(32);
   return std::optional<int64_t>(workerArgConst.value());
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
index 5df3d986fa5b0..3120f8dc675e3 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
@@ -1,7 +1,10 @@
-// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" --verify-diagnostics
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
 
-// Verify that worker-private shared memory with dynamic num_workers emits
-// a diagnostic instead of silently miscompiling.
+// Dynamic num_workers uses the maximum ThreadY width for shared memory.
+
+// CHECK-LABEL: func.func @test_worker_private_dynamic_nw
+// CHECK: acc.gpu_shared_memory
+// CHECK-SAME: num_copies = 32
 
 func.func @test_worker_private_dynamic_nw(%nw: index) {
   %c32 = arith.constant 32 : index
@@ -11,13 +14,11 @@ func.func @test_worker_private_dynamic_nw(%nw: index) {
   %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
   acc.kernel_environment {
     %priv = acc.privatize : () -> !acc.private_type<memref<2xi32>>
-    // expected-error @below {{failed to legalize operation 'acc.compute_region'}}
     acc.compute_region launch(%arg0 = %block_x, %arg1 = %thread_y, %arg2 = %thread_x) ins(%arg10 = %priv) : (!acc.private_type<memref<2xi32>>) {
       %c0 = arith.constant 0 : index
       %c1 = arith.constant 1 : index
       %c0_i32 = arith.constant 0 : i32
       scf.parallel (%iv) = (%c0) to (%arg1) step (%c1) {
-        // expected-error @below {{worker-private variables in shared memory require compile-time constant num_workers}}
         %local = acc.private_local %arg10 : (!acc.private_type<memref<2xi32>>) -> memref<2xi32>
         memref.store %c0_i32, %local[%c0] : memref<2xi32>
         scf.reduce
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 69f529abf6fa6..9bf78556bf265 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -793,7 +793,7 @@ TEST_F(OpenACCUtilsCGTest, getSharedMemoryBytesGangWorkerReductionAccumulator) {
 }
 
 TEST_F(OpenACCUtilsCGTest,
-       isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
+       isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicUsesUpperBound) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
   b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
@@ -801,8 +801,7 @@ TEST_F(OpenACCUtilsCGTest,
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
   auto bx =
       ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
-  // A non-constant num_workers: the launch operand exists but is not an
-  // arith.constant, which is what triggers the diagnostic / failure path.
+  // A non-constant num_workers uses the maximum ThreadY width.
   auto dynNumWorkers = arith::AddIOp::create(b, loc, c1, c1);
   auto ty = ParWidthOp::create(b, loc, dynNumWorkers,
                                GPUParallelDimAttr::threadYDim(&context));
@@ -818,11 +817,18 @@ TEST_F(OpenACCUtilsCGTest,
   FailureOr<bool> silent =
       isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
   ASSERT_TRUE(succeeded(silent));
-  EXPECT_FALSE(*silent);
+  EXPECT_TRUE(*silent);
 
   FailureOr<bool> diagnosed = isPrivateLocalSharedMemoryCandidate(
       privateLocal, cr, *module, policy, &support);
-  EXPECT_TRUE(failed(diagnosed));
+  ASSERT_TRUE(succeeded(diagnosed));
+  EXPECT_TRUE(*diagnosed);
+
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 512);
 }
 
 TEST_F(OpenACCUtilsCGTest, getPrivateLocalSharedMemoryUpperBoundBytes) {


From llvmlistbot at llvm.org  Mon Jul 20 10:46:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 10:46:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] Support dynamic worker-private
	shared memory (PR #210770)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210770@github.com>
Message-ID: <6a5e5f13.6c8668f2.1f24.ad6b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-openacc

Author: Matsu (khaki3)

<details>
<summary>Changes</summary>

Use the maximum ThreadY width when the runtime worker count prevents exact static sizing.

---
Full diff: https://github.com/llvm/llvm-project/pull/210770.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp (+8-12) 
- (modified) mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir (+6-5) 
- (modified) mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp (+11-5) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
index 5f4af6a5bcc5b..745b6fa348533 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsCG.cpp
@@ -317,9 +317,10 @@ collectPrivateLocalParDims(PrivateLocalOp privateLocal,
   return parDims;
 }
 
-static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
-    PrivateLocalOp privateLocal, ComputeRegionOp computeRegion,
-    bool isWorkerPrivate, OpenACCSupport *support) {
+static FailureOr<std::optional<int64_t>>
+getWorkerPrivateSharedMemoryNumCopies(PrivateLocalOp privateLocal,
+                                      ComputeRegionOp computeRegion,
+                                      bool isWorkerPrivate, OpenACCSupport *) {
   if (!isWorkerPrivate)
     return std::optional<int64_t>(1);
 
@@ -330,15 +331,10 @@ static FailureOr<std::optional<int64_t>> getWorkerPrivateSharedMemoryNumCopies(
     return std::optional<int64_t>();
 
   auto workerArgConst = workerArg->getDefiningOp<arith::ConstantIndexOp>();
-  if (!workerArgConst) {
-    if (support) {
-      (void)support->emitNYI(privateLocal.getLoc(),
-                             "worker-private variables in shared memory "
-                             "require compile-time constant num_workers");
-      return failure();
-    }
-    return std::optional<int64_t>();
-  }
+  // ThreadY is bounded by 32 after subgroup alignment. Use that upper bound
+  // when the exact worker count is known only at runtime.
+  if (!workerArgConst)
+    return std::optional<int64_t>(32);
   return std::optional<int64_t>(workerArgConst.value());
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
index 5df3d986fa5b0..3120f8dc675e3 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-private-dynamic-nw.mlir
@@ -1,7 +1,10 @@
-// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" --verify-diagnostics
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
 
-// Verify that worker-private shared memory with dynamic num_workers emits
-// a diagnostic instead of silently miscompiling.
+// Dynamic num_workers uses the maximum ThreadY width for shared memory.
+
+// CHECK-LABEL: func.func @test_worker_private_dynamic_nw
+// CHECK: acc.gpu_shared_memory
+// CHECK-SAME: num_copies = 32
 
 func.func @test_worker_private_dynamic_nw(%nw: index) {
   %c32 = arith.constant 32 : index
@@ -11,13 +14,11 @@ func.func @test_worker_private_dynamic_nw(%nw: index) {
   %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
   acc.kernel_environment {
     %priv = acc.privatize : () -> !acc.private_type<memref<2xi32>>
-    // expected-error @below {{failed to legalize operation 'acc.compute_region'}}
     acc.compute_region launch(%arg0 = %block_x, %arg1 = %thread_y, %arg2 = %thread_x) ins(%arg10 = %priv) : (!acc.private_type<memref<2xi32>>) {
       %c0 = arith.constant 0 : index
       %c1 = arith.constant 1 : index
       %c0_i32 = arith.constant 0 : i32
       scf.parallel (%iv) = (%c0) to (%arg1) step (%c1) {
-        // expected-error @below {{worker-private variables in shared memory require compile-time constant num_workers}}
         %local = acc.private_local %arg10 : (!acc.private_type<memref<2xi32>>) -> memref<2xi32>
         memref.store %c0_i32, %local[%c0] : memref<2xi32>
         scf.reduce
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
index 69f529abf6fa6..9bf78556bf265 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsCGTest.cpp
@@ -793,7 +793,7 @@ TEST_F(OpenACCUtilsCGTest, getSharedMemoryBytesGangWorkerReductionAccumulator) {
 }
 
 TEST_F(OpenACCUtilsCGTest,
-       isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicFails) {
+       isPrivateLocalSharedMemoryCandidateWorkerPrivateDynamicUsesUpperBound) {
   OwningOpRef<ModuleOp> module = ModuleOp::create(b, loc);
   b.setInsertionPointToStart(module->getBody());
   GPUParallelDimsAttr workerDims = GPUParallelDimsAttr::get(
@@ -801,8 +801,7 @@ TEST_F(OpenACCUtilsCGTest,
   auto c1 = arith::ConstantIndexOp::create(b, loc, 1);
   auto bx =
       ParWidthOp::create(b, loc, c1, GPUParallelDimAttr::blockXDim(&context));
-  // A non-constant num_workers: the launch operand exists but is not an
-  // arith.constant, which is what triggers the diagnostic / failure path.
+  // A non-constant num_workers uses the maximum ThreadY width.
   auto dynNumWorkers = arith::AddIOp::create(b, loc, c1, c1);
   auto ty = ParWidthOp::create(b, loc, dynNumWorkers,
                                GPUParallelDimAttr::threadYDim(&context));
@@ -818,11 +817,18 @@ TEST_F(OpenACCUtilsCGTest,
   FailureOr<bool> silent =
       isPrivateLocalSharedMemoryCandidate(privateLocal, cr, *module, policy);
   ASSERT_TRUE(succeeded(silent));
-  EXPECT_FALSE(*silent);
+  EXPECT_TRUE(*silent);
 
   FailureOr<bool> diagnosed = isPrivateLocalSharedMemoryCandidate(
       privateLocal, cr, *module, policy, &support);
-  EXPECT_TRUE(failed(diagnosed));
+  ASSERT_TRUE(succeeded(diagnosed));
+  EXPECT_TRUE(*diagnosed);
+
+  std::optional<int64_t> upperBound =
+      getPrivateLocalSharedMemoryUpperBoundBytes(privateLocal, cr, *module,
+                                                 policy);
+  ASSERT_TRUE(upperBound.has_value());
+  EXPECT_EQ(*upperBound, 512);
 }
 
 TEST_F(OpenACCUtilsCGTest, getPrivateLocalSharedMemoryUpperBoundBytes) {

``````````

</details>


https://github.com/llvm/llvm-project/pull/210770

From llvmlistbot at llvm.org  Mon Jul 20 10:49:41 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Mon, 20 Jul 2026 10:49:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter to the chunked form (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5e5fb5.f06f4994.2f0620.d0d9@mx.google.com>


https://github.com/mshahneo updated https://github.com/llvm/llvm-project/pull/201688

>From 5f08474ba988d962b6853076f4caabe51020c2ca Mon Sep 17 00:00:00 2001
From: Claude <noreply at anthropic.com>
Date: Mon, 22 Jun 2026 15:32:10 +0000
Subject: [PATCH 1/2] [mlir][xegpu] Remove inferred chunk_size; rename
 contiguity attr to chunk_size

Two changes to the xegpu.load (LoadGatherOp) / xegpu.store (StoreScatterOp)
gather/scatter ops:

1. Remove the old optional `chunk_size` I64 attribute that duplicated
   information already carried by the operand/result types. The payload chunk
   size (contiguous elements per work item) is derived from the value and mask
   types via a shared helper `xegpu::getGatherScatterPayloadChunk` in
   XeGPUUtils; the op verifier keeps a file-local equivalent. Consumers in
   propagate-layout, sg-to-lane, unroll, and the operand-layout util now call
   the helper instead of an op accessor.

2. Rename the `contiguity` attribute (added by #201684) to `chunk_size`, so the
   name `chunk_size` now denotes the offsets-contiguity hint: an optional I64
   whose value must be >= 2 and must divide the innermost offsets dimension
   (verified by `isValidChunkSize`). The AxisInfo-based analysis
   (runContiguityAnalysis) stamps it via the generated setChunkSize accessor;
   the test driver and lit tests are updated accordingly.

Because the two uses no longer share the `getChunkSize()` name, there is no
collision: `getChunkSize()` is exclusively the generated accessor for the
renamed attribute.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../include/mlir/Dialect/XeGPU/IR/XeGPUOps.td | 37 ++++----
 .../Dialect/XeGPU/Transforms/Transforms.h     |  4 +-
 .../mlir/Dialect/XeGPU/Utils/XeGPUUtils.h     | 12 +++
 .../VectorToXeGPU/VectorToXeGPU.cpp           |  4 -
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp        | 86 ++++++++++++-------
 .../Transforms/XeGPUContiguityAnalysis.cpp    |  8 +-
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  6 +-
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 17 ++--
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 23 ++---
 .../Transforms/XeGPUWgToSgDistribute.cpp      | 16 ++--
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   | 20 ++++-
 .../Dialect/XeGPU/contiguity-analysis.mlir    | 38 ++++----
 mlir/test/Dialect/XeGPU/invalid.mlir          | 53 +++---------
 mlir/test/Dialect/XeGPU/ops.mlir              | 34 ++++----
 .../XeGPU/propagate-layout-subgroup.mlir      |  8 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |  4 +-
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 12 +--
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  | 16 ++--
 .../test-xegpu-coalesce-gather-scatter.mlir   |  2 +-
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   | 40 ++++-----
 .../Dialect/XeGPU/xegpu-unroll-patterns.mlir  | 16 ++--
 mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir   | 32 +++----
 .../lib/Dialect/XeGPU/TestXeGPUTransforms.cpp | 10 +--
 23 files changed, 254 insertions(+), 244 deletions(-)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..ee551e1065109 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -731,13 +731,17 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to load from per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the result
       of load. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`: the innermost `offsets` dimension is contiguous in runs of
+      `chunk_size` elements (so `chunk_size` must be >= 2 and must divide that
+      dimension). It is a hint used by layout propagation / coalescing; it does
+      not change the loaded data. Only valid on the vector-offsets form.
+
     Results:
     - `res`: represents loaded data
 
@@ -756,8 +760,8 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   ```
 
   Example 2 (lane level):
-  lane level only accepts the offsets variant. chunk_size can be inferred from result
-  type. In this example, chunk_size is 8.
+  lane level only accepts the offsets variant. The number of contiguous elements
+  loaded per work item is inferred from the result type; in this example it is 8.
   ```mlir
     %2 = xegpu.load %1[%2], %0 <{l1_hint = #xegpu.cache_hint<cached>,
                              l2_hint = #xegpu.cache_hint<uncached>,
@@ -769,12 +773,12 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
 
   let arguments = (ins XeGPU_GatherScatterBaseAddrType:$source,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
   let results = (outs XeGPU_ValueOrScalarType:$value);
 
   let extraClassDeclaration = extraBaseClassDeclaration # [{
@@ -815,13 +819,11 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   let builders = [
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
@@ -862,13 +864,18 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to store to per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the value
       to be stored. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`, analogously to `load`: the innermost `offsets` dimension is
+      contiguous in runs of `chunk_size` elements (so `chunk_size` must be >= 2
+      and must divide that dimension). It is a hint used by layout propagation /
+      coalescing; it does not change the stored data. Only valid on the
+      vector-offsets form.
+
 
   Example 1 (Subgroup level):
   A variant accepts memref as base pointer and an offset.
@@ -886,8 +893,8 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   ```
 
   Example 2 (Lane level):
-  Lane level IR only accepts the offsets variant. chunk_size can be inferred from value
-  type. In this example, chunk_size is 8.
+  Lane level IR only accepts the offsets variant. The number of contiguous elements
+  stored per work item is inferred from the value type; in this example it is 8.
   ```mlir
     xegpu.store %0, %1[%2], %3 <{l1_hint = #xegpu.cache_hint<uncached>,
                              l2_hint = #xegpu.cache_hint<write_back>,
@@ -900,12 +907,12 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let arguments = (ins XeGPU_ValueOrScalarType:$value,
       XeGPU_GatherScatterBaseAddrType:$dest,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
 
   let extraClassDeclaration = extraBaseClassDeclaration#[{
     Type getDestType() {
@@ -945,13 +952,11 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let builders = [
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
index 388bd6145df21..7f707d67ebd2a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
@@ -93,11 +93,11 @@ void populateXeGPUSgToLaneDistributeTypeConversionAndLegality(
 //===----------------------------------------------------------------------===//
 
 /// Run the AxisInfo-based contiguity analysis over `root` and stamp a
-/// `contiguity` attribute on every `xegpu.load` / `xegpu.store` whose
+/// `chunk_size` attribute on every `xegpu.load` / `xegpu.store` whose
 /// offsets are contiguous (in runs of >= 2) along the innermost dimension.
 /// The stamped value is the inner-dim contiguity; it is a target-independent
 /// property consumed downstream (e.g. to derive a `lane_data` split). Ops that
-/// already carry a `contiguity` attribute are left untouched.
+/// already carry a `chunk_size` attribute are left untouched.
 void runContiguityAnalysis(Operation *root);
 
 /// Collect a set of patterns to unroll xegpu operations to a smaller shapes.
diff --git a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
index 0125dfc44196b..33dfc1cea4ea0 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
@@ -41,6 +41,18 @@ namespace xegpu {
 /// Flatten a set of ValueRange into a single SmallVector<Value>
 SmallVector<Value> flattenValues(ArrayRef<ValueRange> values);
 
+/// Infer the payload chunk size (number of contiguous elements per work item)
+/// of a gather/scatter op (`xegpu.load` / `xegpu.store`) from its value/result
+/// and mask types. The mask carries one element per lane, so the payload is the
+/// value with the chunk dimension removed: the chunk size is the trailing value
+/// dimension whenever the value has more elements than the mask, and 1
+/// otherwise.
+///   - scalar value                               -> 1
+///   - value vector<LxC...>, mask vector<L...>    -> C (trailing dim)
+///   - 1D value vector<N...>, scalar/size-1 mask  -> N (lane-level chunk)
+///   - value and mask with matching element counts -> 1 (one elem per lane)
+int64_t getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy);
+
 /// If tensor descriptor has a layout attribute it is used in SIMT mode.
 /// In this mode, the distributed vector shape is determined as follows:
 /// Definitions:
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..feab0eb51ec27 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -502,7 +502,6 @@ static LogicalResult lowerToScatteredLoadOp(vector::TransferReadOp readOp,
       vectorShape);
   auto gatherOp = xegpu::LoadGatherOp::create(
       rewriter, loc, vectorType, flatMemref, localOffsets, mask,
-      /*chunk_size=*/IntegerAttr{},
       /*l1_hint=*/xegpu::CachePolicyAttr{},
       /*l2_hint=*/xegpu::CachePolicyAttr{},
       /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -537,7 +536,6 @@ static LogicalResult lowerToScatteredStoreOp(vector::TransferWriteOp writeOp,
       vectorShape);
   xegpu::StoreScatterOp::create(rewriter, loc, writeOp.getVector(), flatMemref,
                                 localOffsets, mask,
-                                /*chunk_size=*/IntegerAttr{},
                                 /*l1_hint=*/xegpu::CachePolicyAttr{},
                                 /*l2_hint=*/xegpu::CachePolicyAttr{},
                                 /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -793,7 +791,6 @@ struct GatherLowering : public OpRewritePattern<vector::GatherOp> {
 
     auto xeGatherOp = xegpu::LoadGatherOp::create(
         rewriter, loc, vectorType, flatMemref, localOffsets, gatherOp.getMask(),
-        /*chunk_size=*/IntegerAttr{},
         /*l1_hint=*/xegpu::CachePolicyAttr{},
         /*l2_hint=*/xegpu::CachePolicyAttr{},
         /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -828,7 +825,6 @@ struct ScatterLowering : public OpRewritePattern<vector::ScatterOp> {
 
     xegpu::StoreScatterOp::create(rewriter, loc, scatterOp.getValueToStore(),
                                   flatMemref, localOffsets, scatterOp.getMask(),
-                                  /*chunk_size=*/IntegerAttr{},
                                   /*l1_hint=*/xegpu::CachePolicyAttr{},
                                   /*l2_hint=*/xegpu::CachePolicyAttr{},
                                   /*l3_hint=*/xegpu::CachePolicyAttr{},
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..c0178035b59d5 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -61,6 +61,30 @@ static bool isWriteHintOrNone(const CachePolicyAttr &attr) {
          kind == CachePolicy::WRITE_BACK || kind == CachePolicy::WRITE_THROUGH;
 }
 
+// Infer the chunk size (number of contiguous elements per work item) of a
+// gather/scatter op from its value/result and mask types. The chunk size is no
+// longer carried as an attribute; it is fully determined by the types. The
+// mask carries one element per lane, so it is the value with the chunk
+// dimension removed. The chunk size is therefore the trailing value dimension
+// whenever the value has more elements than the mask, and 1 otherwise:
+//   - scalar value                                   -> 1
+//   - value vector<LxC...>, mask vector<L...>        -> C (trailing dim)
+//   - 1D value vector<N...>, scalar/size-1 mask      -> N (lane-level chunk)
+//   - value and mask with matching element counts    -> 1 (one elem per lane)
+static int64_t inferGatherScatterChunkSize(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 static LogicalResult
 isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
                                  VectorType valueTy, int64_t chunkSize,
@@ -113,23 +137,23 @@ isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
   return success();
 }
 
-// Validates the `contiguity` attribute against the op's offsets type: the
+// Validates the `chunk_size` attribute against the op's offsets type: the
 // innermost offsets dimension is contiguous in runs of `size`, so `size` must
 // be >= 2 and must divide that dimension.
 static LogicalResult
-isValidContiguity(std::optional<uint64_t> contiguity, Type offsetsTy,
-                  function_ref<InFlightDiagnostic()> emitError) {
-  if (!contiguity)
+isValidChunkSize(std::optional<uint64_t> chunkSize, Type offsetsTy,
+                 function_ref<InFlightDiagnostic()> emitError) {
+  if (!chunkSize)
     return success();
   auto offsetsVecTy = dyn_cast<VectorType>(offsetsTy);
   if (!offsetsVecTy)
-    return emitError() << "contiguity requires vector offsets (one per lane).";
-  int64_t size = static_cast<int64_t>(*contiguity);
+    return emitError() << "chunk_size requires vector offsets (one per lane).";
+  int64_t size = static_cast<int64_t>(*chunkSize);
   int64_t inner = offsetsVecTy.getShape().back();
   if (size < 2)
-    return emitError() << "contiguity = " << size << " (must be >= 2)";
+    return emitError() << "chunk_size = " << size << " (must be >= 2)";
   if (inner % size != 0)
-    return emitError() << "contiguity = " << size
+    return emitError() << "chunk_size = " << size
                        << " (must divide the innermost offsets dim " << inner
                        << ")";
   return success();
@@ -592,7 +616,7 @@ LogicalResult LoadGatherOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto srcTy = getSourceType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(srcTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -604,8 +628,8 @@ LogicalResult LoadGatherOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -614,7 +638,7 @@ LogicalResult LoadGatherOp::verify() {
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint) {
   auto loc = source.getLoc();
@@ -623,15 +647,14 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, /*anchor_layout=*/nullptr,
-        /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint,
                          DistributeLayoutAttr layout) {
@@ -641,8 +664,8 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
@@ -662,7 +685,7 @@ LogicalResult StoreScatterOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto destTy = getDestType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(destTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -674,8 +697,8 @@ LogicalResult StoreScatterOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -684,7 +707,6 @@ LogicalResult StoreScatterOp::verify() {
 void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
                            Value value, Value dest,
                            ArrayRef<OpFoldResult> offsets, Value mask,
-                           IntegerAttr chunk_size,
                            xegpu::CachePolicyAttr l1_hint,
                            xegpu::CachePolicyAttr l2_hint,
                            xegpu::CachePolicyAttr l3_hint) {
@@ -695,15 +717,17 @@ void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, /*anchor_layout=*/nullptr, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
-void StoreScatterOp::build(
-    OpBuilder &builder, OperationState &state, Value value, Value dest,
-    ArrayRef<OpFoldResult> offsets, Value mask, IntegerAttr chunk_size,
-    xegpu::CachePolicyAttr l1_hint, xegpu::CachePolicyAttr l2_hint,
-    xegpu::CachePolicyAttr l3_hint, DistributeLayoutAttr layout) {
+void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
+                           Value value, Value dest,
+                           ArrayRef<OpFoldResult> offsets, Value mask,
+                           xegpu::CachePolicyAttr l1_hint,
+                           xegpu::CachePolicyAttr l2_hint,
+                           xegpu::CachePolicyAttr l3_hint,
+                           DistributeLayoutAttr layout) {
   auto loc = dest.getLoc();
   int64_t size = static_cast<int64_t>(offsets.size());
   auto type = VectorType::get(size, builder.getIndexType());
@@ -711,8 +735,8 @@ void StoreScatterOp::build(
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
index 3cc8152561681..64f1d6b992e4f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
@@ -872,7 +872,7 @@ using ::mlir::xegpu::detail::axis_dataflow::AxisInfoLattice;
 // Analysis driver.
 //===----------------------------------------------------------------------===//
 
-/// Stamp a `contiguity` attribute on `op` recording the inner-dim contiguity
+/// Stamp a `chunk_size` attribute on `op` recording the inner-dim contiguity
 /// computed by the analysis. The contiguity is a target-independent property
 /// of the offsets.
 template <typename OpTy>
@@ -880,9 +880,9 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
   auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
   if (!offsetsTy || offsetsTy.getNumElements() <= 1)
     return;
-  // A pre-existing `contiguity` (user-authored, or stamped by an earlier run)
+  // A pre-existing `chunk_size` (user-authored, or stamped by an earlier run)
   // takes precedence; leave it untouched so the analysis is idempotent.
-  if (op.getContiguity())
+  if (op.getChunkSize())
     return;
   const auto *lat = solver.lookupState<AxisInfoLattice>(op.getOffsets());
   if (!lat || !lat->getValue().isInitialized())
@@ -898,7 +898,7 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
     --contiguity;
   if (contiguity < 2)
     return;
-  op.setContiguity(contiguity);
+  op.setChunkSize(contiguity);
 }
 
 } // namespace
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1681d295ae0ff..d1e27094564cb 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1218,7 +1218,8 @@ void LayoutInfoPropagation::visitLoadGatherOp(
   if (!uArch)
     return;
   VectorType resVecTy = load.getValueType();
-  int chunkSize = load.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(resVecTy, load.getMaskType());
 
   LayoutInfo resLayoutInfo = results[0]->getValue();
   if (!resLayoutInfo.isAssigned())
@@ -1283,7 +1284,8 @@ void LayoutInfoPropagation::visitStoreScatterOp(
   if (!uArch)
     return;
   VectorType srcVecTy = storeScatter.getValueType();
-  int chunkSize = storeScatter.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(srcVecTy, storeScatter.getMaskType());
 
   if (hasParamsOfLayoutKind(anchorLayoutAttr)) {
     requiredAnchorLayoutAttr = anchorLayoutAttr;
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..2ac3d1ff2bba5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -543,7 +543,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
       return failure();
 
     // Check that leading dimensions are unit.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origResultTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origResultTy.getShape();
     if (llvm::any_of(
@@ -581,8 +582,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
     Value distSource = adaptor.getSource();
     auto newOp = xegpu::LoadGatherOp::create(
         rewriter, op.getLoc(), distResultTy1D, distSource, distOffsets,
-        distMask, op.getChunkSizeAttr(), op.getL1HintAttr(), op.getL2HintAttr(),
-        op.getL3HintAttr(), /*layout=*/nullptr, /*contiguity=*/nullptr);
+        distMask, op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(),
+        /*layout=*/nullptr, /*contiguity=*/nullptr);
 
     Value result = newOp->getResult(0);
     if (distResultTy1D != distResultTy)
@@ -1071,7 +1072,8 @@ struct SgToLaneStoreScatter
       return failure();
 
     // Check that all leading dimensions are unit dimensions.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origValueTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origValueTy.getShape();
     if (llvm::any_of(shape.take_front(origValueTy.getRank() - effectiveVecRank),
@@ -1112,10 +1114,9 @@ struct SgToLaneStoreScatter
 
     Value distDest = adaptor.getDest();
     xegpu::StoreScatterOp::create(rewriter, op.getLoc(), distValue, distDest,
-                                  distOffsets, distMask, op.getChunkSizeAttr(),
-                                  op.getL1HintAttr(), op.getL2HintAttr(),
-                                  op.getL3HintAttr(), /*layout=*/nullptr,
-                                  /*contiguity=*/nullptr);
+                                  distOffsets, distMask, op.getL1HintAttr(),
+                                  op.getL2HintAttr(), op.getL3HintAttr(),
+                                  /*layout=*/nullptr, /*contiguity=*/nullptr);
     rewriter.eraseOp(op);
     return success();
   }
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..8e853e25558e7 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -699,11 +699,8 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       return failure();
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     // Unroll mask and offsets with correct shape
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -721,7 +718,6 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -763,10 +759,8 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
     SmallVector<Value> newOps;
     for (auto [o, m] : llvm::zip(convertedOffsets, convertedMasks)) {
       auto newOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newValueTy, op.getSource(), o, m,
-          rewriter.getI64IntegerAttr(chunkSize), op.getL1HintAttr(),
-          op.getL2HintAttr(), op.getL3HintAttr(), layout,
-          /*contiguity=*/nullptr);
+          rewriter, loc, newValueTy, op.getSource(), o, m, op.getL1HintAttr(),
+          op.getL2HintAttr(), op.getL3HintAttr(), layout, /*contiguity=*/nullptr);
       newOps.push_back(newOp);
     }
 
@@ -793,11 +787,8 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     if (!targetShape)
       return failure();
 
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -812,7 +803,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -859,7 +849,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     for (auto [v, o, m] :
          llvm::zip(convertedValues, convertedOffsets, convertedMasks)) {
       xegpu::StoreScatterOp::create(rewriter, loc, v, op.getDest(), o, m,
-                                    rewriter.getI64IntegerAttr(chunkSize),
                                     op.getL1HintAttr(), op.getL2HintAttr(),
                                     op.getL3HintAttr(), layout,
                                     /*contiguity=*/nullptr);
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..8cd1b1ad0d3f1 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -824,14 +824,12 @@ struct WgToSgLoadGatherOp : public OpConversionPattern<xegpu::LoadGatherOp> {
     }
 
     SmallVector<Value> newLoadOps;
-    auto chunkSizeAttr =
-        rewriter.getI64IntegerAttr(op.getChunkSize().value_or(1));
     VectorType newTy = VectorType::get(sgShape, resultType.getElementType());
     for (auto [offsets, mask] :
          llvm::zip(adaptor.getOffsets(), adaptor.getMask())) {
       auto newLayout = layout.dropSgLayoutAndData();
       auto newLoadOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newTy, op.getSource(), offsets, mask, chunkSizeAttr,
+          rewriter, loc, newTy, op.getSource(), offsets, mask,
           op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(), newLayout,
           /*contiguity=*/nullptr);
       newLoadOps.push_back(newLoadOp);
@@ -871,16 +869,12 @@ struct WgToSgStoreScatterOp
                                          "offsets have not been distributed");
     }
 
-    auto chunkSizeOpt = op.getChunkSize();
-    int64_t chunkSize = chunkSizeOpt ? static_cast<int64_t>(*chunkSizeOpt) : 1;
-    auto chunkSizeAttr = rewriter.getI64IntegerAttr(chunkSize);
     for (auto [val, offs, mask] : llvm::zip(
              adaptor.getValue(), adaptor.getOffsets(), adaptor.getMask())) {
-      xegpu::StoreScatterOp::create(rewriter, loc, val, op.getDest(), offs,
-                                    mask, chunkSizeAttr, op.getL1HintAttr(),
-                                    op.getL2HintAttr(), op.getL3HintAttr(),
-                                    layout.dropSgLayoutAndData(),
-                                    /*contiguity=*/nullptr);
+      xegpu::StoreScatterOp::create(
+          rewriter, loc, val, op.getDest(), offs, mask, op.getL1HintAttr(),
+          op.getL2HintAttr(), op.getL3HintAttr(), layout.dropSgLayoutAndData(),
+          /*contiguity=*/nullptr);
     }
     rewriter.eraseOp(op);
     return success();
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..92759fc7a69a6 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -40,6 +40,20 @@ SmallVector<Value> xegpu::flattenValues(ArrayRef<ValueRange> values) {
   return result;
 }
 
+int64_t xegpu::getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 FailureOr<VectorType>
 mlir::xegpu::getDistributedVectorType(xegpu::TensorDescType tdescTy) {
   auto layout = llvm::dyn_cast_if_present<LayoutAttr>(tdescTy.getLayout());
@@ -237,7 +251,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
 
     if (isa<xegpu::StoreScatterOp>(op)) {
       xegpu::StoreScatterOp store(op);
-      int chunkSize = store.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(store.getValueType(),
+                                                          store.getMaskType());
       if (layout && idx >= 2 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
@@ -245,7 +260,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
     }
     if (isa<xegpu::LoadGatherOp>(op)) {
       xegpu::LoadGatherOp load(op);
-      int chunkSize = load.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(load.getValueType(),
+                                                          load.getMaskType());
       if (layout && idx >= 1 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
diff --git a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
index f223b325c2f9e..df9f7fe69c2e2 100644
--- a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
+++ b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-opt -split-input-file \
 // RUN:   -test-xegpu-coalesce-gather-scatter="analyze-only=true" %s | FileCheck %s
 
-// Contiguity analysis: stamps the `contiguity` attribute on gather/scatter ops
+// Contiguity analysis: stamps the `chunk_size` attribute on gather/scatter ops
 // whose offsets are contiguous (runs of >= 2) along the innermost dimension.
 // The stamped value is the inner-dim contiguity, rounded down to a divisor of
 // the inner extent. The analysis is target-independent and mask-independent;
@@ -11,7 +11,7 @@
 // 1-D vector.step -> stride-1, fully contiguous over the 32-element inner dim.
 // CHECK-LABEL: func.func @load_step_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
@@ -25,7 +25,7 @@ func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
 // Dense stride-1 constant offsets -> contiguity 32.
 // CHECK-LABEL: func.func @load_dense_ap_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
   %offsets = arith.constant dense<[
     0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
@@ -41,7 +41,7 @@ func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
 // Stride-4 offsets: not contiguous, no attribute stamped.
 // CHECK-LABEL: func.func @load_stride4_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
   %c4 = arith.constant 4 : index
@@ -58,7 +58,7 @@ func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
 // All-equal offsets: inner dim is constant, not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_broadcast_offsets_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
   %offsets = arith.constant dense<0> : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -72,7 +72,7 @@ func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
 // (the mask check is a consumer concern).
 // CHECK-LABEL: func.func @load_partial_mask(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %v = xegpu.load %ptr[%offsets], %mask
@@ -84,7 +84,7 @@ func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32>
 // Store with vector.step offsets -> contiguity 32 on the store.
 // CHECK-LABEL: func.func @store_step_offsets(
 // CHECK: xegpu.store
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -97,7 +97,7 @@ func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
 // 2-D leading-unit dim: contiguity measured on the inner dim -> 32.
 // CHECK-LABEL: func.func @load_2d_leading_unit(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
   %step = vector.step : vector<32xindex>
   %offsets = vector.shape_cast %step : vector<32xindex> to vector<1x32xindex>
@@ -111,7 +111,7 @@ func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
 // True 2-D dense AP: each row stride-1 over 16 -> contiguity 16.
 // CHECK-LABEL: func.func @load_2d_dense_ap(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 16 : i64}>
+// CHECK-SAME: <{chunk_size = 16 : i64}>
 func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
@@ -127,7 +127,7 @@ func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
 // 2-D dense values whose inner row is not a stride-1 AP: no attribute.
 // CHECK-LABEL: func.func @load_2d_non_ap(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 100, 9, 10, 11, 12, 13, 14, 15],
@@ -146,7 +146,7 @@ func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
 // exercised even if the solver later folds all-constant arith ops.
 // CHECK-LABEL: func.func @load_divui_recovers(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %c2 = arith.constant dense<2> : vector<32xindex>
@@ -162,7 +162,7 @@ func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
 // `divui` by a constant that does not divide the inner stride: not recovered.
 // CHECK-LABEL: func.func @load_divui_non_divisor(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -180,7 +180,7 @@ func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
 // not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_remui_inner_uniform(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -196,7 +196,7 @@ func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
 // `shli` then `shrui` cancel: (step << 1) >> 1 -> stride 1 -> contiguity 32.
 // CHECK-LABEL: func.func @load_shli_then_shrui(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -212,7 +212,7 @@ func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
 // `shli` alone scales the stride to 2: not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_shli_unchanged(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -228,7 +228,7 @@ func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
 // contiguity 32.
 // CHECK-LABEL: func.func @load_select_two_aps(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %a = arith.constant dense<0>  : vector<32xindex>
@@ -243,14 +243,14 @@ func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
 }
 
 // -----
-// A pre-existing `contiguity` takes precedence: the analysis leaves it alone.
+// A pre-existing `chunk_size` takes precedence: the analysis leaves it alone.
 // CHECK-LABEL: func.func @user_attr_preserved(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 2 : i64}>
+// CHECK-SAME: <{chunk_size = 2 : i64}>
 func.func @user_attr_preserved(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
-  %v = xegpu.load %ptr[%offsets], %mask <{contiguity = 2 : i64}>
+  %v = xegpu.load %ptr[%offsets], %mask <{chunk_size = 2 : i64}>
       : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
   return %v : vector<32xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..c610ded4a5b50 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -233,7 +233,7 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
   %offsets = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<8xi1>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}>
+  %2 = xegpu.load %src[%offsets], %mask
       : memref<?xf32>, vector<4xindex>, vector<8xi1> -> vector<4x2xf32>
   return
 }
@@ -242,8 +242,8 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
 func.func @load_gather_simt_1(%src: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  %2 = xegpu.load %src[%0], %1
       : memref<?xf32>, vector<4xindex>, vector<4xi1> -> vector<6xf32>
   return
 }
@@ -265,7 +265,7 @@ func.func @store_scatter_vc_3(%dst: memref<?xf32>) {
   %1 = arith.constant dense<1>: vector<8xi1>
   %2 = arith.constant dense<2.9>: vector<4x2xf32>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  xegpu.store %2, %dst[%0], %1
       : vector<4x2xf32>, memref<?xf32>, vector<4xindex>, vector<8xi1>
   return
 }
@@ -275,8 +275,8 @@ func.func @store_scatter_simt_1(%dst: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
   %2 = arith.constant dense<2.9>: vector<6xf32>
-  // expected-error at +1 {{value elements must match chunk size}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  xegpu.store %2, %dst[%0], %1
       : vector<6xf32>, memref<?xf32>, vector<4xindex>, vector<4xi1>
   return
 }
@@ -316,26 +316,6 @@ func.func @load_gather_offset_sg(%src: memref<?xf16>) {
   return
 }
 
-// -----
-func.func @load_gather_offset_wi(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf32>
-  return
-}
-
-// -----
-func.func @store_scatter_offset_wi_1(%src: memref<?xf16>) {
-  %val = arith.constant dense<2.9>: vector<4xf16>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  %mask = arith.constant dense<1>: vector<1xi1>
-  // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %val, %src[%offsets], %mask
-        : vector<4xf16>, memref<?xf16>, vector<1xindex>, vector<1xi1>
-  return
-}
-
 // -----
 func.func @store_scatter_offset_wi_2(%src: memref<4x4xf16>) {
   %val = arith.constant dense<2.9>: vector<4xf16>
@@ -363,16 +343,7 @@ func.func @load_gather_offset_wi_4(%src: !xegpu.tensor_desc<1x2xf16>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
-  return
-}
-
-// -----
-func.func @load_gather_offset_wi_2(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf16>
+  %2 = xegpu.load %src[%offsets], %mask : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
   return
 }
 
@@ -381,7 +352,7 @@ func.func @load_gather_offset_wi_1(%src: memref<4x4xf32>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
+  %2 = xegpu.load %src[%offsets], %mask : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
   return
 }
 
@@ -792,16 +763,16 @@ func.func @dpas_mx_scale_b_layout_not_distributable(%a : vector<8x16xf8E5M2>, %b
 
 // -----
 func.func @contiguity_too_small(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // expected-error at +1 {{contiguity = 1 (must be >= 2)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 1 : i64}>
+  // expected-error at +1 {{chunk_size = 1 (must be >= 2)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 1 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   return
 }
 
 // -----
 func.func @contiguity_does_not_divide(%src: i64, %offset: vector<6xindex>, %mask: vector<6xi1>) {
-  // expected-error at +1 {{contiguity = 4 (must divide the innermost offsets dim 6)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // expected-error at +1 {{chunk_size = 4 (must divide the innermost offsets dim 6)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<6xindex>, vector<6xi1> -> vector<6xf32>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..2b82d426eba39 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -377,8 +377,8 @@ gpu.func @prefetch_nd_3d(%src: memref<4x8x16xf16>) {
 
 // CHECK: gpu.func @simt_load_4(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: vector<1xindex>, %[[arg2:.*]]: vector<1xi1>) {
 gpu.func @simt_load_4(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vector<1xi1>) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
   gpu.return
 }
 
@@ -391,8 +391,8 @@ gpu.func @simt_load_5(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vec
 
 // CHECK: gpu.func @simt_load_6(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: index, %[[arg2:.*]]: i1) {
 gpu.func @simt_load_6(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, index, i1 -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, index, i1 -> vector<8xf16>
   gpu.return
 }
 
@@ -407,23 +407,23 @@ gpu.func @simt_load_7(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
 gpu.func @subgroup_load_offset_1(%src: memref<?xf16>) {
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
-  %val = xegpu.load %src[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
+  %val = xegpu.load %src[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_4(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: vector<1xindex>, %[[arg3:.*]]: vector<1xi1>) {
 gpu.func @simt_store_4(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: vector<1xindex>, %arg3: vector<1xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_5(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: index, %[[arg3:.*]]: i1) {
 gpu.func @simt_store_5(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: index, %arg3: i1) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, index, i1
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, index, i1
   gpu.return
 }
 
@@ -446,25 +446,25 @@ gpu.func @subgroup_store_offset_1(%dest: memref<?xf16>) {
   %val = arith.constant dense<2.9>: vector<4x2xf16>
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
-  xegpu.store %val, %dest[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
+  xegpu.store %val, %dest[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @load_contiguity(%[[arg0:.*]]: i64, %[[arg1:.*]]: vector<16xindex>, %[[arg2:.*]]: vector<16xi1>) {
 gpu.func @load_contiguity(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // A user-provided `contiguity` round-trips through the optional op attribute.
-  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{contiguity = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // A user-provided `chunk_size` round-trips through the optional op attribute.
+  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.return
 }
 
 // CHECK: gpu.func @store_contiguity(%[[arg0:.*]]: vector<16xf32>, %[[arg1:.*]]: i64, %[[arg2:.*]]: vector<16xindex>, %[[arg3:.*]]: vector<16xi1>) {
 gpu.func @store_contiguity(%val: vector<16xf32>, %dest: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{contiguity = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
-  xegpu.store %val, %dest[%offset], %mask <{contiguity = 4 : i64}>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
+  xegpu.store %val, %dest[%offset], %mask <{chunk_size = 4 : i64}>
       : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..402700fd4c5e3 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -615,9 +615,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
@@ -646,9 +646,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..b8b8d86cf8b10 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -642,8 +642,8 @@ gpu.module @test{
     %5 = vector.broadcast %3 : index to vector<1xindex>
     %6 = arith.addi %4, %5 : vector<1xindex>
     %7 = vector.broadcast %6 : vector<1xindex> to vector<1x1x1x16xindex>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..3ced1d7e9c6f5 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -260,14 +260,14 @@ gpu.func @prefetch_nd() {
 // CHECK-LABEL: gpu.func @scatter_load_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[CAST:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   gpu.return
 }
@@ -275,20 +275,20 @@ gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
 // CHECK-LABEL: gpu.func @scatter_store_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[C1:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK: %[[C2:.*]] = vector.shape_cast %[[C1]] : vector<1x8xf16> to vector<8xf16>
-// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.func @scatter_store_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   xegpu.store %0, %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index fa9897770a08e..e8a41f6fc99f0 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -257,7 +257,7 @@ gpu.module @xevm_module{
 // CHECK-DAG:      %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK-DAG:      %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:          %[[IF:.*]] = scf.if %[[PREDICATE]] -> (vector<1x8xf16>) {
-// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:          : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK-NEXT:        %[[LD_CAST:.*]] = vector.shape_cast %[[LD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK-NEXT:        scf.yield %[[LD_CAST]] : vector<1x8xf16>
@@ -265,14 +265,14 @@ gpu.module @xevm_module{
 // CHECK-NEXT:        scf.yield %[[CST]] : vector<1x8xf16>
 // CHECK-NEXT:      }
 // CHECK-NEXT:      %[[IF_CAST:.*]] = vector.shape_cast %[[IF]] : vector<1x8xf16> to vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:        vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.module @xevm_module{
   gpu.func @scatter_ops_scf_yield(%src: memref<256xf16>, %pred : i1) {
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     %loaded = scf.if %pred -> (vector<16x8xf16>) {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
@@ -280,7 +280,7 @@ gpu.module @xevm_module{
       %3 = arith.constant dense<12.> : vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
     }
-    xegpu.store %loaded, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+    xegpu.store %loaded, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     gpu.return
   }
 }
@@ -291,9 +291,9 @@ gpu.module @xevm_module{
 // CHECK:         %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK:         %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:         scf.if %[[PREDICATE]] {
-// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 // CHECK-NEXT:    }
 gpu.module @xevm_module{
@@ -302,10 +302,10 @@ gpu.module @xevm_module{
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     scf.if %pred  {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
-      xegpu.store %3, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+      xegpu.store %3, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     }
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
index fb32e0012679b..885026f5c85ae 100644
--- a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
+++ b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
@@ -99,7 +99,7 @@ gpu.module @kernel_chunk4 [#xevm.target<chip = "pvc">] {
 // CHECK-LABEL: func.func @load_no_target_unchanged(
 // CHECK: xegpu.load
 // CHECK-NOT: lane_data
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_no_target_unchanged(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..3d769c86ee2cc 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -334,8 +334,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -353,8 +353,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -369,8 +369,8 @@ gpu.module @test_kernel {
     //CHECK: arith.addi [[step]], [[cst]] : vector<16xindex>
     %step = vector.step : vector<32xindex>
     %mask = vector.create_mask %c16 : vector<32xi1>
-    %ld = xegpu.load %src[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -514,7 +514,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: load_with_offsets
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -525,7 +525,7 @@ gpu.module @test_kernel {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17 : vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -534,7 +534,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: store_with_offsets
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -547,7 +547,7 @@ gpu.module @test_kernel {
       %mask = vector.create_mask %c17 : vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -561,7 +561,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -572,7 +572,7 @@ gpu.module @test_kernel {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17 : vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 }
@@ -585,7 +585,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -598,7 +598,7 @@ gpu.module @test_kernel {
     %mask = vector.create_mask %c17 : vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -611,8 +611,8 @@ gpu.module @test_kernel {
   // CHECK: [[cst_0:%.+]] = arith.constant dense<true> : vector<1x1x16xi1>
   // CHECK: [[cst_1:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
   // CHECK: [[cst_2:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
-  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
-  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
   // CHECK: [[ins_0:%.+]] = vector.insert_strided_slice [[ld_0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   // CHECK: [[ins_1:%.+]] = vector.insert_strided_slice [[ld_1]], [[ins_0]] {offsets = [0, 0, 16], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   gpu.func @preserve_unit_dim_of_load_inst_data(%src: ui64) -> vector<1x1x32xf32> {
@@ -624,7 +624,7 @@ gpu.module @test_kernel {
       ]]> : vector<1x1x32xindex>
 
       %mask = arith.constant dense<true> : vector<1x1x32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
 
       gpu.return %ld : vector<1x1x32xf32>
   }
@@ -692,10 +692,10 @@ gpu.module @test_kernel {
         128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]]
     ]> : vector<1x1x32xindex>
     %mask = arith.constant dense<true> : vector<1x1x32xi1>
-    %a = xegpu.load %A[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
-    %b = xegpu.load %B[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %a = xegpu.load %A[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %b = xegpu.load %B[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     %addf = arith.addf %a, %b : vector<1x1x32xf32>
-    xegpu.store %addf, %C[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
+    xegpu.store %addf, %C[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
index 49fdf1cbee174..75492d8e248d3 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
@@ -128,7 +128,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: load_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -139,7 +139,7 @@ gpu.module @test {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17: vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -147,7 +147,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: store_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -160,7 +160,7 @@ gpu.module @test {
       %mask = vector.create_mask %c17: vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -173,7 +173,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -184,7 +184,7 @@ gpu.module @test {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17: vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 
@@ -196,7 +196,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -209,7 +209,7 @@ gpu.module @test {
     %mask = vector.create_mask %c17: vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
index 8c5dd6c55b99d..2ca6e8d46ef16 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
@@ -317,11 +317,11 @@ gpu.module @test_distribution {
   gpu.func @load_gather(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<32x4xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<32x4xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<32x4xindex>, vector<32x4xi1> -> vector<32x4xf16>
     %offset =  arith.constant dense<0> : vector<256x16xindex>
     %mask = arith.constant dense<1> : vector<256x16xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256x16xindex>, vector<256x16xi1> -> vector<256x16xf16>
     gpu.return
   }
@@ -332,12 +332,12 @@ gpu.module @test_distribution {
     // CHECK: %[[VAL:.*]] = arith.constant dense<2.550000e+01> : vector<8xf16>
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
+    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
      // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<8xindex>, vector<8xi1>
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
+    xegpu.store %val, %dest[%offset], %mask {layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
                                              l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
@@ -348,11 +348,11 @@ gpu.module @test_distribution {
   gpu.func @load_with_non_unit_chunk_size(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 4 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<8xindex>, vector<8xi1> -> vector<8x4xf16>
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 4, layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256xindex>, vector<256xi1> -> vector<256x4xf16>
     gpu.return
   }
@@ -783,9 +783,9 @@ gpu.module @test_distribution {
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
 
-    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{chunk_size = 1 : i64, layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
     // CHECK-SAME: memref<4096xf32>, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    %3 = xegpu.load %2[%offset], %mask {chunk_size = 1, layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
+    %3 = xegpu.load %2[%offset], %mask {layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
 
     // CHECK: %[[BROADCAST:.*]] = vector.broadcast %[[LOAD]] : vector<32xf32> to vector<32x32xf32>
     %4 = vector.broadcast %3 : vector<256xf32> to vector<256x256xf32>
@@ -803,7 +803,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<1.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [1] : vector<1x1x32xf32> to vector<1x32xf32>
     // CHECK-DAG: %[[CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x32xf32> to vector<1x1x32xf32>
@@ -820,7 +820,7 @@ gpu.module @test_distribution {
     %cst_3 = arith.constant dense<1.0> : vector<1x32xf32>
     %offset = arith.constant dense<0> : vector<1x32x32xindex>
     %mask = arith.constant dense<true> : vector<1x32x32xi1>
-    %14 = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
+    %14 = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
     %15 = vector.multi_reduction <add>, %14, %cst_3 [1] : vector<1x32x32xf32> to vector<1x32xf32>
     %anchor = xegpu.convert_layout %15
       <{
@@ -878,7 +878,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<1x1x32x32xf32> to vector<1x1xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x1xf32> to vector<1x1x1x1xf32>
@@ -895,7 +895,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<2x2xf32>
     %offset = arith.constant dense<0> : vector<2x2x128x128xindex>
     %mask = arith.constant dense<true> : vector<2x2x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<2x2x128x128xf32> to vector<2x2xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -911,7 +911,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<16x16x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<16x16x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<16x16x32x32xf32> to vector<16x16xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<16x16xf32> to vector<16x16x1x1xf32>
@@ -928,7 +928,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<32x32xf32>
     %offset = arith.constant dense<0> : vector<32x32x128x128xindex>
     %mask = arith.constant dense<true> : vector<32x32x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<32x32x128x128xf32> to vector<32x32xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -1026,7 +1026,7 @@ gpu.module @test_distribution {
   gpu.func @convert_layout_3D(%arg0: memref<?xf32>) {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0> : vector<1x32x16xindex>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<true> : vector<1x32x16xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{chunk_size = 1 : i64, layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
     // CHECK-DAG: %[[ALLOCA:.*]] = memref.alloca() : memref<1048576xi8, 3>
     // CHECK-DAG: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<1048576xi8, 3> -> !xegpu.mem_desc<8x128x256xf32>
     // CHECK-DAG: %[[SGID:.*]] = gpu.subgroup_id : index
@@ -1055,7 +1055,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[LOAD_SLM:.*]] = xegpu.load_matrix %[[MDESC]][%[[LOAD_OFF_Z]], %[[LOAD_OFF_Y]], %[[LOAD_OFF_X]]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}>: !xegpu.mem_desc<8x128x256xf32>, index, index, index -> vector<1x16x32xf32>
     %offset = arith.constant dense<0> : vector<8x128x256xindex>
     %mask = arith.constant dense<true> : vector<8x128x256xi1>
-    %1 = xegpu.load %arg0[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
+    %1 = xegpu.load %arg0[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
     %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>,
                                    target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>}> : vector<8x128x256xf32>
     %anchor = xegpu.convert_layout %2
diff --git a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
index f6b0c50da91dd..a94d172b44801 100644
--- a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
+++ b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
@@ -456,7 +456,7 @@ struct TestXeGPUCoalesceGatherScatter
   }
 
   StringRef getDescription() const final {
-    return "Test driver that turns the contiguity attribute into a lane_data "
+    return "Test driver that turns the chunk_size attribute into a lane_data "
            "layout on gather/scatter ops.";
   }
 
@@ -477,7 +477,7 @@ struct TestXeGPUCoalesceGatherScatter
 
   Option<bool> analyzeOnly{
       *this, "analyze-only",
-      llvm::cl::desc("Only run the analysis (stamp the contiguity attribute); "
+      llvm::cl::desc("Only run the analysis (stamp the chunk_size attribute); "
                      "do not apply."),
       llvm::cl::init(false)};
 
@@ -509,16 +509,16 @@ struct TestXeGPUCoalesceGatherScatter
     return xegpu::LayoutAttr::get(ctx, instData, laneLayout, laneData);
   }
 
-  /// Minimal driver: read the `contiguity` attribute the analysis stamped and
+  /// Minimal driver: read the `chunk_size` attribute the analysis stamped and
   /// turn it into a `lane_data` layout. This is only a stand-in for the real
   /// consumer (layout propagation) so the analysis output can be checked
   /// end-to-end; it handles just the simple power-of-two case.
   template <typename OpTy>
   static void applyContiguity(OpTy op, unsigned maxChunkSize) {
-    std::optional<uint64_t> contiguity = op.getContiguity();
+    std::optional<uint64_t> contiguity = op.getChunkSize();
     if (!contiguity)
       return;
-    op.removeContiguityAttr();
+    op.removeChunkSizeAttr();
 
     auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
     auto valueTy = op.getValueType();

>From 4c5b053e132243228adb2b54a11de8e94d6b5ba9 Mon Sep 17 00:00:00 2001
From: Claude <noreply at anthropic.com>
Date: Mon, 20 Jul 2026 17:37:37 +0000
Subject: [PATCH 2/2] [mlir][XeGPU] Distribute coalesced gather/scatter via
 lane_data

Rework SgToLaneLoadGather / SgToLaneStoreScatter to lower a coalesced
gather/scatter to the chunked memory access purely from the layout's
lane_data, without relying on a chunk_size attribute.

When lane_data[FCD] = D > 1 covers the lane's entire per-lane fragment (one
round: lane_layout[FCD] * D == FCD extent, so the distributed per-lane vector
has exactly D elements), the lane owns a single contiguous run
{base, base+1, ..., base+D-1}. Distribution takes the first offset/mask of the
group (scalar base + scalar mask) and emits a value vector<D>; the chunk size
is implied by the value type, which the XeVM lowering already consumes. The
dropped per-lane offset computations become dead code removed during lowering.

The round-robin case (lane_data[FCD] == 1, multiple strided rounds) is excluded
by the lane_data > 1 guard.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 48 +++++++++++++++++++
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  | 26 ++++++++++
 2 files changed, 74 insertions(+)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 2ac3d1ff2bba5..9a84f172921b5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -580,6 +580,35 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
         castValueTo(rewriter, cast<TypedValue<VectorType>>(distMask), maskTy1D);
 
     Value distSource = adaptor.getSource();
+
+    // Coalesced case: the layout assigns a genuine contiguous chunk on the FCD
+    // via `lane_data[FCD] = D > 1`, and that chunk is the lane's *entire*
+    // per-lane fragment (one round: `lane_layout[FCD] * D == FCD extent`, so
+    // the distributed per-lane vector has exactly D elements). Only then does
+    // the lane own a single run of D contiguous elements
+    // `{base, base+1, ..., base+D-1}`, which the XeVM lowering expects as a
+    // chunked access: a scalar base offset + scalar mask + a `vector<D>` value
+    // (the chunk size is implied by the value type). Take the first offset /
+    // mask of the contiguous group; the remaining offset computations become
+    // dead code and are removed during lowering.
+    //
+    // This must NOT fire for the round-robin case (`lane_data[FCD] = 1`,
+    // multiple rounds, e.g. a reduction source where lane l owns
+    // `{l, l+SG, l+2*SG, ...}`): there the per-lane elements are strided, not
+    // contiguous, so a chunked access would read the wrong elements. That case
+    // has `lane_data[FCD] == 1`, so the guard below excludes it.
+    int64_t laneElems = distResultTy1D.getNumElements();
+    int64_t innerLaneData = 1;
+    if (auto laneDataArr = layout.getEffectiveLaneDataAsInt();
+        !laneDataArr.empty())
+      innerLaneData = laneDataArr.back();
+    if (innerLaneData > 1 && laneElems == innerLaneData) {
+      distOffsets = vector::ExtractOp::create(
+          rewriter, op.getLoc(), distOffsets, ArrayRef<int64_t>{0});
+      distMask = vector::ExtractOp::create(rewriter, op.getLoc(), distMask,
+                                           ArrayRef<int64_t>{0});
+    }
+
     auto newOp = xegpu::LoadGatherOp::create(
         rewriter, op.getLoc(), distResultTy1D, distSource, distOffsets,
         distMask, op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(),
@@ -1113,6 +1142,25 @@ struct SgToLaneStoreScatter
         castValueTo(rewriter, cast<TypedValue<VectorType>>(distMask), maskTy1D);
 
     Value distDest = adaptor.getDest();
+
+    // Coalesced case (mirrors SgToLaneLoadGather): a genuine contiguous chunk
+    // on the FCD, `lane_data[FCD] = D > 1` covering the lane's entire per-lane
+    // fragment (one round), stores as a chunked access: scalar base offset +
+    // scalar mask + a `vector<D>` value. Take the first offset / mask of the
+    // contiguous group; the dropped offset computations are DCE'd in lowering.
+    // Excludes the round-robin case (`lane_data[FCD] == 1`).
+    int64_t laneElems = distValueTy1D.getNumElements();
+    int64_t innerLaneData = 1;
+    if (auto laneDataArr = layout.getEffectiveLaneDataAsInt();
+        !laneDataArr.empty())
+      innerLaneData = laneDataArr.back();
+    if (innerLaneData > 1 && laneElems == innerLaneData) {
+      distOffsets = vector::ExtractOp::create(
+          rewriter, op.getLoc(), distOffsets, ArrayRef<int64_t>{0});
+      distMask = vector::ExtractOp::create(rewriter, op.getLoc(), distMask,
+                                           ArrayRef<int64_t>{0});
+    }
+
     xegpu::StoreScatterOp::create(rewriter, op.getLoc(), distValue, distDest,
                                   distOffsets, distMask, op.getL1HintAttr(),
                                   op.getL2HintAttr(), op.getL3HintAttr(),
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index e8a41f6fc99f0..19fc7fc87ff18 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -574,3 +574,29 @@ gpu.module @xevm_module {
     gpu.return
   }
 }
+
+// -----
+// Coalesced gather/scatter: the load/store layout has lane_data[FCD] = 2, i.e.
+// each lane owns 2 *contiguous* elements (one round: lane_layout[FCD] * 2 == 32
+// == FCD extent). Distribution must emit the chunked form the XeVM lowering
+// accepts: a scalar base offset + scalar mask + a value vector<2xf32> (the
+// chunk size is implied by the value type), taking element 0 of the per-lane
+// offsets/mask as the base -- NOT a 2-wide offsets/mask vector. This is driven
+// entirely by lane_data; the dropped offset lanes are DCE'd during lowering.
+gpu.module @xevm_module {
+    // CHECK-LABEL: gpu.func @coalesced_load_store
+    // CHECK: %[[LD:.*]] = xegpu.load %{{.*}}[%[[BASE:.*]]], %{{.*}}  : i64, index, i1 -> vector<2xf32>
+    // CHECK: %[[MUL:.*]] = arith.mulf %[[LD]], %{{.*}} : vector<2xf32>
+    // CHECK: xegpu.store %[[MUL]], %{{.*}}[%[[BASE]]], %{{.*}}  : vector<2xf32>, i64, index, i1
+  gpu.func @coalesced_load_store(%src: i64, %dst: i64) {
+    %step = vector.step : vector<32xindex>
+    %mask = arith.constant dense<true> : vector<32xi1>
+    %v = xegpu.load %src[%step], %mask <{layout = #xegpu.layout<lane_layout = [16], lane_data = [2]>}>
+        : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    %c = arith.constant dense<2.0> : vector<32xf32>
+    %p = arith.mulf %v, %c {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [2]>} : vector<32xf32>
+    xegpu.store %p, %dst[%step], %mask <{layout = #xegpu.layout<lane_layout = [16], lane_data = [2]>}>
+        : vector<32xf32>, i64, vector<32xindex>, vector<32xi1>
+    gpu.return
+  }
+}


From llvmlistbot at llvm.org  Mon Jul 20 10:51:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 10:51:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter to the chunked form (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5e601e.c5504292.78ef3.1f68@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 2226e20b0..4648a5827 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -760,7 +760,8 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
     for (auto [o, m] : llvm::zip(convertedOffsets, convertedMasks)) {
       auto newOp = xegpu::LoadGatherOp::create(
           rewriter, loc, newValueTy, op.getSource(), o, m, op.getL1HintAttr(),
-          op.getL2HintAttr(), op.getL3HintAttr(), layout, /*contiguity=*/nullptr);
+          op.getL2HintAttr(), op.getL3HintAttr(), layout,
+          /*contiguity=*/nullptr);
       newOps.push_back(newOp);
     }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/201688

From llvmlistbot at llvm.org  Mon Jul 20 10:53:03 2026
From: llvmlistbot at llvm.org (Slava Zakharin)
Date: Mon, 20 Jul 2026 10:53:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] Support dynamic worker-private
	shared memory (PR #210770)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210770@github.com>
Message-ID: <6a5e607f.e55dc341.1c6a0.9ee6@mx.google.com>


https://github.com/vzakhari approved this pull request.


https://github.com/llvm/llvm-project/pull/210770

From llvmlistbot at llvm.org  Mon Jul 20 11:55:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 11:55:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3d638ea - [mlir][xegpu] Change inst-data of
 load/load_matrix to fit lane layout (#209661)
Message-ID: <6a5e6f0b.d1496011.199a59.34d3@mx.google.com>


Author: Jianhui Li
Date: 2026-07-20T11:55:02-07:00
New Revision: 3d638ea31e8efee04e307e74e29dc80d5585dd27

URL: https://github.com/llvm/llvm-project/commit/3d638ea31e8efee04e307e74e29dc80d5585dd27
DIFF: https://github.com/llvm/llvm-project/commit/3d638ea31e8efee04e307e74e29dc80d5585dd27.diff

LOG: [mlir][xegpu] Change inst-data of load/load_matrix to fit lane layout (#209661)

The consumer layout may have multiple distribution of lane data, but
load_gather/load_matrix only allow one single distribution, change their
inst_data according to the single distribution of lane data.

Added: 
    

Modified: 
    mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
    mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index a7b5a3de73195..48889a8b8febc 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -1853,9 +1853,6 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   SmallVector<int64_t> consumerLaneData =
       consumerLayout.getEffectiveLaneDataAsInt();
 
-  // Pick lane_layout / lane_data: prefer consumer's, fall back to the
-  // scatter-store default (subgroupSize lanes on innermost dim, per-lane
-  // vector capped by maxChunkSize).
   SmallVector<int64_t> laneLayout;
   SmallVector<int64_t> laneData;
   assert(!consumerLaneLayout.empty() && !consumerLaneData.empty() &&
@@ -1864,16 +1861,10 @@ static xegpu::DistributeLayoutAttr setupGenericLoadAnchorLayout(
   laneData.assign(consumerLaneData.begin(), consumerLaneData.end());
 
   if (layoutKind == xegpu::LayoutKind::InstData) {
-    // Take consumer's inst_data as-is. If the consumer doesn't have one,
-    // fall back to lane_layout * lane_data per dim.
     SmallVector<int64_t> instData;
-    if (!consumerInstData.empty()) {
-      instData.assign(consumerInstData.begin(), consumerInstData.end());
-    } else {
-      instData.resize(resShape.size());
-      for (size_t i = 0; i < resShape.size(); ++i)
-        instData[i] = laneLayout[i] * laneData[i];
-    }
+    instData.resize(resShape.size());
+    for (size_t i = 0; i < resShape.size(); ++i)
+      instData[i] = laneLayout[i] * laneData[i];
     return buildInstDataLayoutWithLane(context, instData, laneLayout, laneData);
   }
   if (layoutKind == xegpu::LayoutKind::Lane)

diff  --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index e8cddb8fa1de3..a9182a01958ae 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -128,37 +128,65 @@ func.func @store_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
 }
 }
 
+// TODO: colaesce_chunksize tests should be adjusted when we enable colasce optimization
 // -----
+gpu.module @test {
+// CHECK-LABEL: func.func @load_matrix_with_coalesce_chunksize(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<2048x512xf32>, %[[ARG1:[0-9a-zA-Z]+]]: memref<2048x512xf32>)
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<128xf32, 3>
+// CHECK: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+// CHECK: %[[LOAD:.*]] = xegpu.load_matrix %[[MDESC]][%[[C0]]] <{layout = #xegpu.layout<inst_data = [1], lane_layout = [1], lane_data = [1]>}>: !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+// CHECK: %[[BCAST:.*]] = vector.broadcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [16, 8], lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>} : vector<16xf32> to vector<16x16xf32>
+// CHECK: %[[TRANS:.*]] = vector.transpose %[[BCAST]], [1, 0] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32> to vector<16x16xf32>
+// CHECK: %[[EXP:.*]] = math.exp %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[DIV:.*]] = arith.divf %[[EXP]], %[[TRANS]] {layout_result_0 = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>} : vector<16x16xf32>
+// CHECK: %[[TDESC:.*]] = xegpu.create_nd_tdesc %[[ARG1]] : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+// CHECK: xegpu.store_nd %[[DIV]], %[[TDESC]][%[[C0]], %[[C0]]] <{layout = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>, #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>>
+func.func @load_matrix_with_coalesce_chunksize(%arg0: memref<2048x512xf32>, %arg1: memref<2048x512xf32>) {
+  %c0 = arith.constant 0 : index
+  %alloca = memref.alloca() : memref<128xf32, 3>
+  %6 = xegpu.create_mem_desc %alloca : memref<128xf32, 3> -> !xegpu.mem_desc<128xf32>
+  %13 = xegpu.load_matrix %6[%c0] : !xegpu.mem_desc<128xf32>, index -> vector<16xf32>
+  %14 = vector.broadcast %13 : vector<16xf32> to vector<16x16xf32>
+  %15 = vector.transpose %14, [1, 0] : vector<16x16xf32> to vector<16x16xf32>
+  %17 = math.exp %15 : vector<16x16xf32>
+  %18 = arith.divf %17, %15 : vector<16x16xf32>
+  %22 = xegpu.create_nd_tdesc %arg1 : memref<2048x512xf32> -> !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  xegpu.store_nd %18, %22[%c0, %c0]  : vector<16x16xf32>, !xegpu.tensor_desc<16x16xf32, #xegpu.block_tdesc_attr<boundary_check = false>>
+  return
+}
+}
 
+// -----
 gpu.module @test {
 // CHECK-LABEL: func.func @scatter_ops_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<512xf32>) {
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<true> : vector<16x32xi1>
-// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>} dense<12> : vector<16x32xindex>
-// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<true> : vector<16x32xi1>
+// CHECK: %{{.*}} = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>} dense<12> : vector<16x32xindex>
+// CHECK: %{{.*}} = xegpu.load %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [1, 16], lane_data = [1, 1]>}> :
+// CHECK: xegpu.store %0, %[[ARG0]][%{{.*}}], %{{.*}} <{layout = #xegpu.layout<inst_data = [1, 32], lane_layout = [1, 16], lane_data = [1, 2]>}> :
 // CHECK-SAME: vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
 func.func @scatter_ops_coalesce_chunksize(%src: memref<512xf32>) {
   %1 = arith.constant dense<1>: vector<16x32xi1>
   %offset = arith.constant dense<12> : vector<16x32xindex>
   %3 = xegpu.load %src[%offset], %1
       : memref<512xf32>, vector<16x32xindex>, vector<16x32xi1> -> vector<16x32xf32>
-  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [16, 16]>}>
+  xegpu.store %3, %src[%offset], %1 <{layout = #xegpu.layout<inst_data = [1, 32]>}>
       : vector<16x32xf32>, memref<512xf32>, vector<16x32xindex>, vector<16x32xi1>
   return
 }
 }
 
 // -----
-
 gpu.module @test {
 // CHECK-LABEL: func.func @load_gather_with_coalesce_chunksize(
 // CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x16xf16>, %[[ARG1:[0-9a-zA-Z]+]]: memref<256xf16>, %[[ARG2:[0-9a-zA-Z]+]]: memref<8x16xf32>) {
-// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}
+// CHECK: %[[OFFSET:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}
 // CHECK-SAME:  dense<0> : vector<16x16xindex>
-// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
-// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 16], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
+// CHECK-NEXT: %[[MASK:.*]] = arith.constant {layout_result_0 = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>} dense<true> : vector<16x16xi1>
+// CHECK-NEXT: %{{.*}} = xegpu.load %arg1[%[[OFFSET]]], %[[MASK]] <{layout = #xegpu.layout<inst_data = [16, 2], lane_layout = [16, 1], lane_data = [1, 2]>}> : memref<256xf16>, vector<16x16xindex>, vector<16x16xi1> -> vector<16x16xf16>
 func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: memref<256xf16>, %arg2: memref<8x16xf32>) {
   %c0 = arith.constant 0 : index
   %0 = xegpu.create_nd_tdesc %arg0 : memref<8x16xf16> -> !xegpu.tensor_desc<8x16xf16>
@@ -176,6 +204,7 @@ func.func @load_gather_with_coalesce_chunksize(%arg0: memref<8x16xf16>, %arg1: m
 }
 
 // -----
+
 gpu.module @test {
 // CHECK-LABEL: func.func @vector_shape_cast_expand_non_unit_dims(
 // CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[STEP:.*]]], %[[CST:.*]] <{layout = #xegpu.layout<inst_data = [16], lane_layout = [16], lane_data = [1]>}> : memref<1024xf16>, vector<1024xindex>, vector<1024xi1> -> vector<1024xf16>


        

From llvmlistbot at llvm.org  Mon Jul 20 11:55:09 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 20 Jul 2026 11:55:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Change inst-data of
 load/load_matrix to fit lane layout (PR #209661)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209661@github.com>
Message-ID: <6a5e6f0d.c5504292.78ef3.6033@mx.google.com>


https://github.com/Jianhui-Li closed https://github.com/llvm/llvm-project/pull/209661

From llvmlistbot at llvm.org  Mon Jul 20 12:59:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 12:59:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5e7e37.d2cb4adc.ccd80.388d@mx.google.com>


qyingwu wrote:

> Thanks for the experiment.
> 
> The test LGTM.
> 
> I would keep #206713 open since the ticket also mentions global variables / global variable expressions. A comment on the ticket that the local variable roundtrip currently works as expected makes sense though.
> 
> Did you try to import the example with the self recursive local variable (without your extra change)? If yes, how did the imported IR look in this case?


Thanks for checking! Makes sense. I’ll keep #206713 open and add a comment there clarifying that the local variable roundtrip looks correct, while the global variable / global variable expression part still needs separate handling.

> Did you try to import the example with the self recursive local variable (without your extra change)? If yes, how did the imported IR look in this case?

Yes. With the retainedNodes path kept close to the existing machinery, i.e. translating the retained node directly and without the extra local-variable cycle-break special case, the import still produced two MLIR local variable attrs.

roughly like this:

```mlir
  #di_subprogram = #llvm.di_subprogram<
     recId = distinct[0]<>, isRecSelf = true
  >

 #di_local_variable = #llvm.di_local_variable<
    scope = #di_subprogram, name = "n", ...
  >

  #di_subprogram1 = #llvm.di_subprogram<
    recId = distinct[0]<>, id = distinct[2]<>, ...,
    retainedNodes = [#di_local_variable]
  >

  #di_local_variable1 = #llvm.di_local_variable<
    scope = #di_subprogram1, name = "n", ...
  >
```

So the retainedNodes traversal sees the subprogram cycle and creates a local variable scoped by the self-recursive subprogram placeholder. Later, when traversing from the debug intrinsic, the subprogram has already been finalized, so the local variable is imported with the concrete subprogram scope instead. That is why the second local variable is not recursive in that traversal.


https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Mon Jul 20 13:00:13 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Mon, 20 Jul 2026 13:00:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][acc] Lower per-thread array reductions
	correctly (PR #210453)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210453@github.com>
Message-ID: <6a5e7e4d.743400b0.23a7b2.2921@mx.google.com>


razvanlupusoru wrote:

> A large or dynamic `a` may use one temporary per gang even though `par_dims` includes `thread_x`. Explicit storage metadata could handle this case.

Thank you for the suggestion - I did initially add storage metadata but could not identify cases where this would a problem so I decided against it. Let's discuss in a follow-up if this is needed and create well defined semantics for such property.



https://github.com/llvm/llvm-project/pull/210453

From llvmlistbot at llvm.org  Mon Jul 20 13:00:45 2026
From: llvmlistbot at llvm.org (Ryan Thomas Lynch)
Date: Mon, 20 Jul 2026 13:00:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-tblgen] Emit named result indices (PR
	#210542)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210542@github.com>
Message-ID: <6a5e7e6d.f06f4994.2f0620.34bd@mx.google.com>


emosy wrote:

thank you @joker-eph @makslevental !! appreciate it :)

https://github.com/llvm/llvm-project/pull/210542

From llvmlistbot at llvm.org  Mon Jul 20 13:49:28 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Mon, 20 Jul 2026 13:49:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][openacc] Support already existing
 global in GPU module (PR #210806)
Message-ID: <llvm.org/llvm/llvm-project/pull/210806@github.com>


https://github.com/clementval created https://github.com/llvm/llvm-project/pull/210806

None

>From 18909c11e514588853789b0859a6f130785e0f03 Mon Sep 17 00:00:00 2001
From: Valentin Clement <clementval at gmail.com>
Date: Mon, 20 Jul 2026 13:45:03 -0700
Subject: [PATCH] [mlir][openacc] Support already existing global in GPU module

---
 .../ACCDeclareGPUModuleInsertion.cpp          | 21 ++++++++++---------
 .../acc-declare-gpu-module-insertion.mlir     | 19 ++++++++++++++++-
 2 files changed, 29 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index f815245882630..19460fd9daa31 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -54,7 +54,6 @@
 #include "mlir/Dialect/OpenACC/Transforms/Passes.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Operation.h"
-#include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/SymbolTable.h"
 
 namespace mlir {
@@ -99,20 +98,22 @@ class ACCDeclareGPUModuleInsertion
       StringAttr name = symOp.getNameAttr();
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
-        // Reuse only when the existing GPU symbol is structurally equivalent to
-        // the global we would insert. Otherwise treat as a conflict (different
-        // op type or different definition).
-        if (existing->getName() != globalOp.getName() ||
-            !OperationEquivalence::isEquivalentTo(
-                existing, &globalOp,
-                OperationEquivalence::ignoreValueEquivalence,
-                /*markEquivalent=*/nullptr,
-                OperationEquivalence::IgnoreLocations)) {
+        // A same-named symbol may already exist from an earlier pass (e.g.
+        // CUDA Fortran can clone device globals before ACCImplicitDeclare
+        // marks the host copy with acc.declare). Reuse it when the op type
+        // matches; only a different op type is a real conflict.
+        if (existing->getName() != globalOp.getName()) {
           accSupport.emitNYI(globalOp.getLoc(),
                              llvm::Twine("duplicate global symbol '") +
                                  name.getValue() + "' in gpu module");
           return failure();
         }
+        // Propagate acc.declare onto the GPU copy if it was cloned before the
+        // host global was marked.
+        if (!existing->getAttr(acc::getDeclareAttrName()))
+          if (Attribute declareAttr =
+                  globalOp.getAttr(acc::getDeclareAttrName()))
+            existing->setAttr(acc::getDeclareAttrName(), declareAttr);
         continue;
       }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir b/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
index 8d206673ac02d..2e5910d155641 100644
--- a/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -acc-declare-gpu-module-insertion | FileCheck %s
+// RUN: mlir-opt %s -split-input-file -acc-declare-gpu-module-insertion | FileCheck %s
 
 // Test that globals with acc.declare are copied into the GPU module.
 // The host global stays in the module; a copy is inserted into the GPU module.
@@ -12,3 +12,20 @@
 module {
   memref.global @arr : memref<7xf32> = dense<0.0> {acc.declare = #acc.declare<dataClause = acc_create>}
 }
+
+// -----
+
+// If the GPU module already has the global (e.g. from CUDA Fortran pass before
+// ACCImplicitDeclare marked the host), reuse it and propagate acc.declare.
+// CHECK-LABEL: module attributes {gpu.container_module}
+// CHECK: memref.global @precloned {{.*}} {acc.declare = #acc.declare<dataClause = acc_copyin>}
+// CHECK: gpu.module @acc_gpu_module {
+// CHECK-NEXT: memref.global @precloned {{.*}} {acc.declare = #acc.declare<dataClause = acc_copyin>}
+// CHECK-NEXT: }
+
+module attributes {gpu.container_module} {
+  memref.global @precloned : memref<4xf32> = dense<0.0> {acc.declare = #acc.declare<dataClause = acc_copyin>}
+  gpu.module @acc_gpu_module {
+    memref.global @precloned : memref<4xf32> = dense<0.0>
+  }
+}


From llvmlistbot at llvm.org  Mon Jul 20 13:50:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 13:50:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][openacc] Support already existing
 global in GPU module (PR #210806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210806@github.com>
Message-ID: <6a5e89ff.507fc713.239434.3f81@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-openacc

Author: Valentin Clement (バレンタイン クレメン) (clementval)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/210806.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp (+11-10) 
- (modified) mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir (+18-1) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index f815245882630..19460fd9daa31 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -54,7 +54,6 @@
 #include "mlir/Dialect/OpenACC/Transforms/Passes.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/Operation.h"
-#include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/SymbolTable.h"
 
 namespace mlir {
@@ -99,20 +98,22 @@ class ACCDeclareGPUModuleInsertion
       StringAttr name = symOp.getNameAttr();
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
-        // Reuse only when the existing GPU symbol is structurally equivalent to
-        // the global we would insert. Otherwise treat as a conflict (different
-        // op type or different definition).
-        if (existing->getName() != globalOp.getName() ||
-            !OperationEquivalence::isEquivalentTo(
-                existing, &globalOp,
-                OperationEquivalence::ignoreValueEquivalence,
-                /*markEquivalent=*/nullptr,
-                OperationEquivalence::IgnoreLocations)) {
+        // A same-named symbol may already exist from an earlier pass (e.g.
+        // CUDA Fortran can clone device globals before ACCImplicitDeclare
+        // marks the host copy with acc.declare). Reuse it when the op type
+        // matches; only a different op type is a real conflict.
+        if (existing->getName() != globalOp.getName()) {
           accSupport.emitNYI(globalOp.getLoc(),
                              llvm::Twine("duplicate global symbol '") +
                                  name.getValue() + "' in gpu module");
           return failure();
         }
+        // Propagate acc.declare onto the GPU copy if it was cloned before the
+        // host global was marked.
+        if (!existing->getAttr(acc::getDeclareAttrName()))
+          if (Attribute declareAttr =
+                  globalOp.getAttr(acc::getDeclareAttrName()))
+            existing->setAttr(acc::getDeclareAttrName(), declareAttr);
         continue;
       }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir b/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
index 8d206673ac02d..2e5910d155641 100644
--- a/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-declare-gpu-module-insertion.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -acc-declare-gpu-module-insertion | FileCheck %s
+// RUN: mlir-opt %s -split-input-file -acc-declare-gpu-module-insertion | FileCheck %s
 
 // Test that globals with acc.declare are copied into the GPU module.
 // The host global stays in the module; a copy is inserted into the GPU module.
@@ -12,3 +12,20 @@
 module {
   memref.global @arr : memref<7xf32> = dense<0.0> {acc.declare = #acc.declare<dataClause = acc_create>}
 }
+
+// -----
+
+// If the GPU module already has the global (e.g. from CUDA Fortran pass before
+// ACCImplicitDeclare marked the host), reuse it and propagate acc.declare.
+// CHECK-LABEL: module attributes {gpu.container_module}
+// CHECK: memref.global @precloned {{.*}} {acc.declare = #acc.declare<dataClause = acc_copyin>}
+// CHECK: gpu.module @acc_gpu_module {
+// CHECK-NEXT: memref.global @precloned {{.*}} {acc.declare = #acc.declare<dataClause = acc_copyin>}
+// CHECK-NEXT: }
+
+module attributes {gpu.container_module} {
+  memref.global @precloned : memref<4xf32> = dense<0.0> {acc.declare = #acc.declare<dataClause = acc_copyin>}
+  gpu.module @acc_gpu_module {
+    memref.global @precloned : memref<4xf32> = dense<0.0>
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210806

From llvmlistbot at llvm.org  Mon Jul 20 13:50:57 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Mon, 20 Jul 2026 13:50:57 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a5e8a31.d2cb4adc.ccd80.c15f@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 7c5c60e5d2a49f606c8d966b61d2b08c2f1c40bf Mon Sep 17 00:00:00 2001
From: Victor Perez <victor.pc.upm at gmail.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.

Signed-off-by: Victor Perez <victor.pc.upm at gmail.com>
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  64 ++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  55 +++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  47 +++++--
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  95 ++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 39 files changed, 536 insertions(+), 173 deletions(-)
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..d3d7964dacd59 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,20 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..3191b9e2b1d92 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,46 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special float literal (inf/NaN) from `fromSign`, which
+/// starts at the already-consumed sign. On success returns a `floatliteral`
+/// (or `error`) token and advances `curPtr`, else std::nullopt (curPtr intact).
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan(0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Reject `inf`/`qnan` that is only a prefix of a longer identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() && (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+                          rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // `nan(0x..)` / `snan(0x..)`: capture through ')'; APFloat validates later.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1;
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +369,28 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // C-style hex float (`0x1.fp13`): a '.'/'p' distinguishes it from the
+    // bit-pattern form; the binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..835cd892c9a19 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,55 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // The sign is either a preceding '-' token (isNegative) or folded into an
+  // inf/NaN spelling; both at once (e.g. `-+inf`) is invalid. Drop a leading
+  // '+' since convertFromString rejects it on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // Reject values the type cannot represent; APFloat would otherwise abort.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // convertFromString does not accept "qnan".
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build in the target semantics to preserve NaN payloads; overflow/underflow
+  // are tolerated (inf/zero).
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..e14639c058e46 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -2257,13 +2257,9 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
                             bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+  // Finite values print as decimal; this is the only form whose type may be
+  // elided.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2291,41 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The rest (inf, NaN, hex bit-pattern) are typeless; keep the type.
   if (printedHex)
     *printedHex = true;
+
+  if (apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // Preferred quiet NaN = only the quiet (sign-mask) bit set. A zero-width
+    // payload (no mantissa bits, e.g. f8E8M0FNU) has none, so print qnan.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Drop the quiet/signaling bit; trim leading zeros.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern.
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index d0336da15cee8..31ea455be65df 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2127,7 +2127,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2136,7 +2136,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..03a25f43f09b7 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,16 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The diagnostic must point at the float literal, not the following token
+// (the `}` is on the next line, exposing a stale location).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +685,88 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // Doubly-signed literal: a '-' token before an already-signed inf/NaN.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN has no Inf encoding: reject, do not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN has no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has NaN but no Inf.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: reject negatives, do not crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Mon Jul 20 15:49:10 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 20 Jul 2026 15:49:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Make convert_layout
	input_layout optional (PR #210833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210833@github.com>
Message-ID: <6a5ea5e6.1905e68e.30d957.54d2@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/210833

>From 5cb9b955fbaaf28ed3a72c050251a4faa9db4c93 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Mon, 20 Jul 2026 20:32:53 +0000
Subject: [PATCH] [mlir][xegpu] Make convert_layout input_layout optional

Make the `input_layout` attribute of `xegpu.convert_layout` optional. When
it is not set, the effective input layout defaults to `target_layout`, i.e.
the source is assumed to already be in the target distribution.

- Change `$input_layout` to `OptionalAttr<DistributeLayoutAttr>` and add a
  `getEffectiveInputLayout()` helper that falls back to `target_layout`.
- Update the verifier and all pass consumers (Blocking, WgToSg, SgToLane,
  Unroll, PropagateLayout, Utils) to use the effective input layout.
- In PropagateLayout, guard the input_layout access with
  dyn_cast_if_present since the attribute may now be null.
- In ResolveLayoutConflicts, pin the effective input layout before
  retargeting a producer convert's `target_layout`, so the omitted-input
  default does not silently turn a real conversion into a no-op.
- Update tests that used identical input_layout/target_layout to specify
  only `target_layout`.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../include/mlir/Dialect/XeGPU/IR/XeGPUOps.td |  16 ++-
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp        |   5 +-
 .../XeGPU/Transforms/XeGPUBlocking.cpp        |   2 +-
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  12 +-
 .../Transforms/XeGPUSgToLaneDistribute.cpp    |   2 +-
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 126 +++++++++++++++---
 .../Transforms/XeGPUWgToSgDistribute.cpp      |   5 +-
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   |   2 +-
 .../test/Dialect/XeGPU/peephole-optimize.mlir |   5 +-
 .../XeGPU/propagate-layout-inst-data.mlir     |  10 +-
 .../XeGPU/propagate-layout-subgroup.mlir      |   3 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   3 +-
 .../XeGPU/resolve-layout-conflicts.mlir       |   6 +-
 .../XeGPU/sg-to-lane-distribute-unit.mlir     |  58 +-------
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  |   1 -
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   |  36 ++++-
 .../XeGPU/xegpu-wg-to-sg-elemwise.mlir        |   7 -
 .../test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir |   9 --
 mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir   |  43 +-----
 19 files changed, 188 insertions(+), 163 deletions(-)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..849b038dc1851 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1227,8 +1227,9 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
       Arguments:
       - `source`: The input vector whose data is to be redistributed. The source and
       result types must match.
-      - `input_layout`: The layout attribute describing the current distribution of `source`
-      across subgroups and/or lanes.
+      - `input_layout`: [optional] The layout attribute describing the current distribution
+      of `source` across subgroups and/or lanes. When omitted, it defaults to `target_layout`,
+      i.e. the source is assumed to already be in the target distribution.
       - `target_layout`: The layout attribute describing the desired distribution of the result
       across subgroups and/or lanes.
 
@@ -1241,7 +1242,7 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
         ```
     }];
     let arguments = (ins XeGPU_VectorOrScalarType: $source,
-                         DistributeLayoutAttr: $input_layout,
+                         OptionalAttr<DistributeLayoutAttr>: $input_layout,
                          DistributeLayoutAttr: $target_layout);
     let results = (outs XeGPU_VectorOrScalarType: $result);
     let assemblyFormat = [{
@@ -1256,6 +1257,15 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
         setTargetLayoutAttr(anchorLayout);
       }
 
+      /// Returns the effective input layout: the explicit `input_layout` when
+      /// set, otherwise `target_layout` (the source is assumed to already be in
+      /// the target distribution).
+      xegpu::DistributeLayoutAttr getEffectiveInputLayout() {
+        if (auto layout = getInputLayoutAttr())
+          return layout;
+        return getTargetLayoutAttr();
+      }
+
     }];
 
     let hasVerifier = 1;
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..d6945b9c7fe3b 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -843,12 +843,11 @@ LogicalResult DpasOp::verify() {
 // XeGPU_ConvertLayoutOp
 //===----------------------------------------------------------------------===//
 LogicalResult ConvertLayoutOp::verify() {
-  auto srcLayout = getInputLayout();
   auto resLayout = getTargetLayout();
-  if (!srcLayout)
-    return emitOpError("expected input layout.");
   if (!resLayout)
     return emitOpError("expected target layout.");
+  // `input_layout` is optional; when omitted it defaults to `target_layout`.
+  auto srcLayout = getEffectiveInputLayout();
 
   // both input and target layouts should be WgLayout or SgLayout at the same
   // time.
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
index 57b8ac73f41ac..9d1686403a431 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
@@ -109,7 +109,7 @@ XeGPUBlockingPass::getTileShape(Operation *op) const {
 
   if (auto convertLayoutOp = dyn_cast<xegpu::ConvertLayoutOp>(op)) {
     auto inputInstData =
-        convertLayoutOp.getInputLayout().getEffectiveInstDataAsInt();
+        convertLayoutOp.getEffectiveInputLayout().getEffectiveInstDataAsInt();
     auto targetInstData =
         convertLayoutOp.getTargetLayout().getEffectiveInstDataAsInt();
     // return the one with larger size
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1681d295ae0ff..dc2083929e469 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1016,8 +1016,10 @@ void LayoutInfoPropagation::visitConvertLayoutOp(
   // TODO: fix if one of the layouts is a slice layout
   auto targetLayoutAttr =
       dyn_cast<xegpu::LayoutAttr>(convert.getTargetLayoutAttr());
+  // `input_layout` is optional; it may be null when omitted (defaults to
+  // `target_layout`), so guard against a null attribute here.
   auto inputLayoutAttr =
-      dyn_cast<xegpu::LayoutAttr>(convert.getInputLayoutAttr());
+      dyn_cast_if_present<xegpu::LayoutAttr>(convert.getInputLayoutAttr());
 
   // The result's propagated layout is authoritative for the converted value.
   // Fill the lane_layout / lane_data / order parameters the target_layout is
@@ -1053,7 +1055,7 @@ void LayoutInfoPropagation::visitConvertLayoutOp(
     }
   }
 
-  xegpu::DistributeLayoutAttr anchorLayout = convert.getInputLayoutAttr();
+  xegpu::DistributeLayoutAttr anchorLayout = convert.getEffectiveInputLayout();
   LayoutInfo convertLayout = makeLayoutInfo(anchorLayout);
   // Propagate the new layout to the tensor descriptor operand.
   propagateIfChanged(operands[0], operands[0]->meet(convertLayout));
@@ -1658,6 +1660,12 @@ ResolveLayoutConflicts::resolveVectorConsumer(OpOperand &operand) {
   if (auto producerConvert =
           vectorValue.getDefiningOp<xegpu::ConvertLayoutOp>();
       producerConvert && vectorValue.hasOneUse()) {
+    // Pin the effective input layout first: when input_layout is omitted it
+    // defaults to target_layout, so retargeting target_layout below would
+    // otherwise silently change the input as well and turn a real conversion
+    // into a no-op.
+    producerConvert.setInputLayoutAttr(
+        producerConvert.getEffectiveInputLayout());
     producerConvert.setTargetLayoutAttr(consumerLayout);
     return success();
   }
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..a5a998f9b4427 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -1673,7 +1673,7 @@ struct SgToLaneConvertLayout
   LogicalResult
   matchAndRewrite(xegpu::ConvertLayoutOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto inputLayout = op.getInputLayoutAttr();
+    auto inputLayout = op.getEffectiveInputLayout();
     auto targetLayout = op.getTargetLayoutAttr();
     Type valType = op.getResult().getType();
 
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..14061ab24ffe0 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -957,14 +957,98 @@ struct UnrollStoreMatrixOp : public UnrollPattern<xegpu::StoreMatrixOp> {
 /// after inst_data stripped. If it does, it will unroll the vector into
 /// multiple smaller vectors according to the target shape, and create multiple
 /// ConvertLayoutOp with the unrolled vectors and the stripped layouts.
+///
+/// When the input and target layouts have different inst_data, the source is
+/// extracted at the input inst_data granularity and the result is inserted at
+/// the target inst_data granularity, enabling slice cancellation during
+/// canonicalization.
 struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
   using UnrollPattern<xegpu::ConvertLayoutOp>::UnrollPattern;
+
+  /// Extracts source in `inTile` slices, regroups into `convTile`-sized
+  /// ConvertLayoutOps, and inserts the result in `outTile` slices.
+  /// Returns failure if the tiles do not evenly divide.
+  LogicalResult
+  rewriteWithRegrouping(xegpu::ConvertLayoutOp op, VectorType valueTy,
+                        ArrayRef<int64_t> convTile, ArrayRef<int64_t> inTile,
+                        ArrayRef<int64_t> outTile,
+                        xegpu::DistributeLayoutAttr inputLayout,
+                        xegpu::DistributeLayoutAttr targetLayout, Location loc,
+                        PatternRewriter &rewriter) const {
+    ArrayRef<int64_t> vecShape = valueTy.getShape();
+    if (!computeShapeRatio(vecShape, convTile) ||
+        !computeShapeRatio(convTile, inTile) ||
+        !computeShapeRatio(convTile, outTile))
+      return failure();
+
+    Type elemTy = valueTy.getElementType();
+    int64_t rank = valueTy.getRank();
+    VectorType convTy = VectorType::get(convTile, elemTy);
+    SmallVector<int64_t> strides(rank, 1);
+
+    Value source = op.getSource();
+    auto zeroOf = [&](VectorType ty) -> Value {
+      return arith::ConstantOp::create(
+          rewriter, loc, ty,
+          DenseElementsAttr::get(ty, rewriter.getZeroAttr(elemTy)));
+    };
+    auto addOffsets = [](ArrayRef<int64_t> a,
+                         ArrayRef<int64_t> b) -> SmallVector<int64_t> {
+      SmallVector<int64_t> res(a);
+      for (auto [r, v] : llvm::zip_equal(res, b))
+        r += v;
+      return res;
+    };
+
+    Value result = zeroOf(valueTy);
+    for (SmallVector<int64_t> convOff :
+         StaticTileOffsetRange(vecShape, convTile)) {
+      // Build the convert tile from inTile-sized slices of the source.
+      Value conv;
+      if (convTile == inTile) {
+        conv = vector::ExtractStridedSliceOp::create(
+            rewriter, loc, source, convOff, convTile, strides);
+      } else {
+        conv = zeroOf(convTy);
+        for (SmallVector<int64_t> inLocal :
+             StaticTileOffsetRange(convTile, inTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, source, addOffsets(convOff, inLocal), inTile,
+              strides);
+          conv = vector::InsertStridedSliceOp::create(rewriter, loc, piece,
+                                                      conv, inLocal, strides);
+        }
+      }
+
+      conv = xegpu::ConvertLayoutOp::create(rewriter, loc, convTy, conv,
+                                            inputLayout, targetLayout);
+
+      // Write the converted tile into the result as outTile-sized slices.
+      if (convTile == outTile) {
+        result = vector::InsertStridedSliceOp::create(rewriter, loc, conv,
+                                                      result, convOff, strides);
+      } else {
+        for (SmallVector<int64_t> outLocal :
+             StaticTileOffsetRange(convTile, outTile)) {
+          Value piece = vector::ExtractStridedSliceOp::create(
+              rewriter, loc, conv, outLocal, outTile, strides);
+          result = vector::InsertStridedSliceOp::create(
+              rewriter, loc, piece, result, addOffsets(convOff, outLocal),
+              strides);
+        }
+      }
+    }
+
+    rewriter.replaceOp(op, result);
+    return success();
+  }
+
   LogicalResult matchAndRewrite(xegpu::ConvertLayoutOp op,
                                 PatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
     Type valType = op.getType();
 
-    xegpu::DistributeLayoutAttr inputLayout = op.getInputLayoutAttr();
+    xegpu::DistributeLayoutAttr inputLayout = op.getEffectiveInputLayout();
     xegpu::DistributeLayoutAttr targetLayout = op.getTargetLayoutAttr();
     if (!inputLayout || !targetLayout)
       return rewriter.notifyMatchFailure(op, "missing layout attributes.");
@@ -974,8 +1058,10 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
       return success();
     }
 
-    if (inputLayout.getEffectiveInstDataAsInt().empty() ||
-        targetLayout.getEffectiveInstDataAsInt().empty())
+    // Capture inst_data granularities before stripping them.
+    SmallVector<int64_t> inTile = inputLayout.getEffectiveInstDataAsInt();
+    SmallVector<int64_t> outTile = targetLayout.getEffectiveInstDataAsInt();
+    if (inTile.empty() || outTile.empty())
       return rewriter.notifyMatchFailure(op, "Not a target ConvertLayoutOp.");
 
     inputLayout = inputLayout.dropInstData();
@@ -988,20 +1074,30 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     if (!targetShape || targetShape->size() != (size_t)valueTy.getRank())
       return failure();
 
-    Value newSource = op.getSource();
+    // Nothing to convert if layouts match after stripping inst_data.
+    if (!inputLayout || !targetLayout || inputLayout.isEqualTo(targetLayout)) {
+      rewriter.replaceOp(op, op.getSource());
+      return success();
+    }
+
+    // Try regrouping: extract at inTile, convert, insert at outTile.
+    if (succeeded(rewriteWithRegrouping(op, valueTy, *targetShape, inTile,
+                                        outTile, inputLayout, targetLayout, loc,
+                                        rewriter)))
+      return success();
+
+    // Fallback: pack/unpack at the convert tile granularity.
+    SmallVector<Type> convertedValTypes =
+        getUnrolledTypes(valueTy, *targetShape);
+    SmallVector<Value> convertedValues =
+        pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
     SmallVector<Value> newOps;
-    if (inputLayout && targetLayout && !inputLayout.isEqualTo(targetLayout)) {
-      SmallVector<Type> convertedValTypes =
-          getUnrolledTypes(valueTy, *targetShape);
-      SmallVector<Value> convertedValues =
-          pack(op.getOperand(), convertedValTypes, *targetShape, loc, rewriter);
-      for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
-        auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
-                                                    inputLayout, targetLayout);
-        newOps.push_back(newOp);
-      }
-      newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
+    for (auto [v, t] : llvm::zip(convertedValues, convertedValTypes)) {
+      auto newOp = xegpu::ConvertLayoutOp::create(rewriter, loc, t, v,
+                                                  inputLayout, targetLayout);
+      newOps.push_back(newOp);
     }
+    Value newSource = unpack(newOps, op.getType(), *targetShape, loc, rewriter);
 
     rewriter.replaceOp(op, newSource);
     return success();
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..afea358dafe29 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -530,7 +530,7 @@ struct WgToSgConvertLayoutOp
   matchAndRewrite(xegpu::ConvertLayoutOp op, OneToNOpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
-    auto inputLayout = op.getInputLayout();
+    auto inputLayout = op.getEffectiveInputLayout();
     auto targetLayout = op.getTargetLayout();
 
     if (!inputLayout || !targetLayout || !inputLayout.isForWorkgroup() ||
@@ -1661,7 +1661,8 @@ void XeGPUWgToSgDistributePass::runOnOperation() {
 
   target.addDynamicallyLegalOp<xegpu::ConvertLayoutOp>(
       [=](xegpu::ConvertLayoutOp op) -> bool {
-        return isLegal(op.getInputLayout()) && isLegal(op.getTargetLayout());
+        return isLegal(op.getEffectiveInputLayout()) &&
+               isLegal(op.getTargetLayout());
       });
 
   target.addDynamicallyLegalDialect<math::MathDialect, arith::ArithDialect>(
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..07dcb438ca553 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -223,7 +223,7 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
       return nullptr;
     }
     if (auto convertOp = dyn_cast<xegpu::ConvertLayoutOp>(op)) {
-      return convertOp.getInputLayoutAttr();
+      return convertOp.getEffectiveInputLayout();
     }
     auto layout = anchorOp.getAnchorLayout();
 
diff --git a/mlir/test/Dialect/XeGPU/peephole-optimize.mlir b/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
index 79a6e248b0d83..fa5ff9ca56d34 100644
--- a/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
+++ b/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
@@ -433,7 +433,7 @@ gpu.module @xevm_test {
 // CHECK:      %[[REDUCE_1:.*]] = vector.multi_reduction <add>, %[[LOADED]], %[[ACC_VEC]] [0] : vector<4x16xf32> to vector<16xf32>
 // CHECK:      %[[REDUCE_2:.*]] = vector.multi_reduction <add>, %[[REDUCE_1]], %[[ACC_SCALAR]] [0] : vector<16xf32> to f32
 // CHECK:      %[[BRIDGE:.*]] = xegpu.convert_layout %[[REDUCE_2]] <{input_layout = #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims = [0]>, target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
-// CHECK:      %[[CVT:.*]] = xegpu.convert_layout %[[BRIDGE]] <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>, target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
+// CHECK:      %[[CVT:.*]] = xegpu.convert_layout %[[BRIDGE]] <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
 // CHECK:      %[[BCAST:.*]] = vector.broadcast %[[CVT]] : f32 to vector<16xf32>
 // CHECK:      xegpu.store %[[BCAST]], %[[ARG1]]
 gpu.module @xevm_test {
@@ -448,8 +448,7 @@ gpu.module @xevm_test {
      {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}
      [0, 1] : vector<4x16xf32> to f32
     %cvt = xegpu.convert_layout %reduce
-     <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>,
-       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}>
+     <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}>
      : f32
     %reduce_bcast = vector.broadcast %cvt
      {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>}
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index e8cddb8fa1de3..f684241ce0ab3 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -330,15 +330,14 @@ gpu.module @test {
 // CHECK-SAME: !xegpu.tensor_desc<256x32xui8, #xegpu.layout<inst_data = [32, 16], lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<256x32xui8>
 // CHECK: %[[BC:.*]] = vector.bitcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>} : vector<256x32xui8> to vector<256x64xf4E2M1FN>
 // CHECK: xegpu.convert_layout %[[BC]]
-// CHECK-SAME: <{input_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>, target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
+// CHECK-SAME: <{target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
 // CHECK-SAME: : vector<256x64xf4E2M1FN>
 func.func @bitcast_ui8_to_f4(%arg0: memref<256x32xui8>) {
   %0 = xegpu.create_nd_tdesc %arg0 : memref<256x32xui8> -> !xegpu.tensor_desc<256x32xui8>
   %1 = xegpu.load_nd %0[0, 0] : !xegpu.tensor_desc<256x32xui8> -> vector<256x32xui8>
   %2 = vector.bitcast %1 : vector<256x32xui8> to vector<256x64xf4E2M1FN>
   %3 = xegpu.convert_layout %2
-     <{input_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>,
-      target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
+     <{target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
      : vector<256x64xf4E2M1FN>
   return
 }
@@ -353,15 +352,14 @@ gpu.module @test {
 // CHECK-SAME: !xegpu.tensor_desc<256x16xui16, #xegpu.layout<inst_data = [32, 16], lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<256x16xui16>
 // CHECK: %[[BC:.*]] = vector.bitcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>} : vector<256x16xui16> to vector<256x64xf4E2M1FN>
 // CHECK: xegpu.convert_layout %[[BC]]
-// CHECK-SAME: <{input_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>, target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
+// CHECK-SAME: <{target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
 // CHECK-SAME: : vector<256x64xf4E2M1FN>
 func.func @bitcast_ui16_to_f4(%arg0: memref<256x16xui16>) {
   %0 = xegpu.create_nd_tdesc %arg0 : memref<256x16xui16> -> !xegpu.tensor_desc<256x16xui16>
   %1 = xegpu.load_nd %0[0, 0] : !xegpu.tensor_desc<256x16xui16> -> vector<256x16xui16>
   %2 = vector.bitcast %1 : vector<256x16xui16> to vector<256x64xf4E2M1FN>
   %3 = xegpu.convert_layout %2
-     <{input_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>,
-      target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
+     <{target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
      : vector<256x64xf4E2M1FN>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..3b4bfe3e9951e 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -373,8 +373,7 @@ gpu.module @test {
     // CHECK-SAME: layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>
     %1 = xegpu.load_matrix %arg0[%c0, %c0] : !xegpu.mem_desc<32x128xf32>, index, index -> vector<32x128xf32>
     %2 = xegpu.convert_layout %1
-       <{input_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>,
-        target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
+       <{target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
        : vector<32x128xf32>
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..d48cb7993bfab 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -128,8 +128,7 @@ func.func @extf_truncf(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor
   %3 = arith.truncf %2 : vector<16x16xf32> to vector<16x16xf16>
   %4 = xegpu.dpas %0, %3 : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
     %5 = xegpu.convert_layout %4
-     <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
-      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}>
+     <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}>
      : vector<8x16xf32>
   return %4 : vector<8x16xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir b/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
index 55d0e64bb2c65..6a625ff515029 100644
--- a/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
+++ b/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
@@ -261,8 +261,7 @@ func.func @convert_layout() {
       dense<0.000000e+00>
      : vector<32x128xf32>
   %src0_cvt = xegpu.convert_layout %src0
-    <{input_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>,
-     target_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>}>
+    <{target_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>}>
     : vector<32x128xf32>
   %src1 = arith.constant
     {layout_result_0 = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>}
@@ -272,8 +271,7 @@ func.func @convert_layout() {
     {layout_result_0 = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>}
     : vector<32x128xf32>
   %desc_cvt = xegpu.convert_layout %dest
-     <{input_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>,
-      target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
+     <{target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
      : vector<32x128xf32>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..a338ed5de7efc 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -118,7 +118,6 @@ gpu.func @dpas() {
     : vector<8x16xf16>, vector<16x16xf16>, vector<8x16xf32>  -> vector<8x16xf32>
   %anchor = xegpu.convert_layout %4
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -141,7 +140,6 @@ gpu.func @elementwise() {
     : vector<16x16xf32>
   %cl3 = xegpu.convert_layout %3
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -154,7 +152,6 @@ gpu.func @arith_constant() {
   %0 = arith.constant dense<1.0> : vector<16x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -176,7 +173,6 @@ gpu.func @arith_constant_non_splat() {
     dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xindex>
   gpu.return
@@ -200,7 +196,6 @@ gpu.func @arith_constant_non_splat_lane_data() {
     dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>
     }> : vector<32xindex>
   gpu.return
@@ -239,7 +234,6 @@ gpu.func @arith_constant_non_splat_2d_vertical_lanedata() {
            [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127]]> : vector<4x32xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>
     }> : vector<4x32xindex>
   gpu.return
@@ -377,7 +371,6 @@ gpu.func @vector_reduction() {
   %2 = vector.reduction <add>, %0, %acc : vector<32xf32> into f32
   %anchor = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>
     }> : f32
   gpu.return
@@ -440,7 +433,6 @@ gpu.func @vector_multi_reduction_dim1_distributed_dim1_reduction(%laneid: index)
       [1] : vector<2x16xf32> to vector<2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>
     }> : vector<2xf32>
   gpu.return
@@ -503,7 +495,6 @@ gpu.func @vector_multi_reduction_dim0_distributed_dim0_reduction(%laneid: index)
       [0] : vector<16x2xf32> to vector<2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [0]>
     }> : vector<2xf32>
   gpu.return
@@ -526,7 +517,6 @@ gpu.func @vector_multi_reduction_dim1_distributed_dim0_reduction(%laneid: index)
       [0] : vector<4x16xf32> to vector<16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xf32>
   gpu.return
@@ -549,7 +539,6 @@ gpu.func @vector_multi_reduction_dim0_distributed_dim1_reduction(%laneid: index)
       [1] : vector<16x12xf32> to vector<16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -567,7 +556,6 @@ gpu.func @vector_transpose() {
     : vector<16x2xf32> to vector<2x16xf32>
   %transpose2 = xegpu.convert_layout %transpose
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<2x16xf32>
   gpu.return
@@ -583,7 +571,6 @@ gpu.func @vector_bitcast() {
   %bitcast = vector.bitcast %cst : vector<4x32xi8> to vector<4x16xi16>
   %anchor = xegpu.convert_layout %bitcast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<4x16xi16>
   gpu.return
@@ -602,7 +589,6 @@ gpu.func @create_mask_1d(%m0: index) {
     : vector<16xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xi1>
   gpu.return
@@ -621,7 +607,6 @@ gpu.func @constant_mask_1d() {
     : vector<16xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xi1>
   gpu.return
@@ -646,7 +631,6 @@ gpu.func @create_mask_2d(%m0: index, %m1: index) {
     : vector<8x4xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
     }> : vector<8x4xi1>
   gpu.return
@@ -672,7 +656,6 @@ gpu.func @constant_mask_2d() {
     : vector<8x4xi1>
       %mask_cl = xegpu.convert_layout %mask
         <{
-          input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
           target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
         }> : vector<8x4xi1>
       gpu.return
@@ -699,7 +682,6 @@ gpu.func @vector_multi_reduction_3d_leading_unit_dim_lane_local() {
       [1] : vector<1x16x32xf32> to vector<1x32xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>, dims = [1]>
     }> : vector<1x32xf32>
   gpu.return
@@ -730,7 +712,6 @@ gpu.func @vector_multi_reduction_3d_leading_unit_dim_cross_lane() {
       [1] : vector<1x16x2xf32> to vector<1x2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16, 1], lane_data = [1, 1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16, 1], lane_data = [1, 1, 1]>, dims = [1]>
     }> : vector<1x2xf32>
   gpu.return
@@ -744,7 +725,6 @@ gpu.func @vector_extract_from_2d() {
     : vector<16xf32> from vector<4x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -759,7 +739,6 @@ gpu.func @vector_extract_from_2d_offset2() {
     : vector<16xf32> from vector<8x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -776,7 +755,6 @@ gpu.func @vector_insert_into_2d() {
     : vector<16xf32> into vector<4x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<4x16xf32>
   gpu.return
@@ -793,7 +771,6 @@ gpu.func @vector_insert_into_2d_offset2() {
     : vector<16xf32> into vector<8x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -809,7 +786,6 @@ gpu.func @vector_extract_strided_slice_distributed_dim_fully_extracted() {
     : vector<24x16xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -825,7 +801,6 @@ gpu.func @vector_extract_strided_slice_inner_distributed() {
     : vector<24x64xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -841,7 +816,6 @@ gpu.func @vector_extract_strided_slice_outer_distributed() {
     : vector<32x16xf32> to vector<16x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -857,7 +831,6 @@ gpu.func @vector_extract_strided_slice_1d() {
     : vector<64xf32> to vector<32xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<32xf32>
   gpu.return
@@ -873,7 +846,6 @@ gpu.func @vector_extract_strided_slice_partial_offsets() {
     : vector<24x16xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -913,22 +885,18 @@ gpu.func @convert_layout_repack_lane_data() {
     : vector<32x16xi8> to vector<8x16xi8>
   %a0 = xegpu.convert_layout %s0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a1 = xegpu.convert_layout %s1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a2 = xegpu.convert_layout %s2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a3 = xegpu.convert_layout %s3
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   gpu.return
@@ -946,7 +914,6 @@ gpu.func @vector_insert_strided_slice_distributed_dim_fully_inserted() {
     : vector<16x16xf32> into vector<64x16xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x16xf32>
   gpu.return
@@ -964,7 +931,6 @@ gpu.func @vector_insert_strided_slice_inner_distributed() {
     : vector<16x16xf32> into vector<64x32xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x32xf32>
   gpu.return
@@ -982,7 +948,6 @@ gpu.func @vector_insert_strided_slice_outer_distributed() {
     : vector<16x16xf32> into vector<48x32xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>
     }> : vector<48x32xf32>
   gpu.return
@@ -1000,7 +965,6 @@ gpu.func @vector_insert_strided_slice_1d() {
     : vector<16xf32> into vector<48xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<48xf32>
   gpu.return
@@ -1018,7 +982,6 @@ gpu.func @vector_insert_strided_slice_different_ranks() {
     : vector<16xf32> into vector<64x16xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x16xf32>
   gpu.return
@@ -1054,8 +1017,7 @@ gpu.module @xevm_module {
 gpu.func @convert_layout_scalar() {
   %0 = "some_op"() : () -> f32
   %1 = xegpu.convert_layout %0
-    <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
-    target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>}>
+    <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>}>
     : f32
   "some_use"(%1) : (f32) -> ()
   gpu.return
@@ -1139,7 +1101,6 @@ gpu.func @elementwise_wrap_around_dim() {
     : vector<16x1xf16>
    %cl1 = xegpu.convert_layout %1
      <{
-       input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
        target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
      }> : vector<16x1xf16>
    gpu.return
@@ -1158,7 +1119,6 @@ gpu.func @vector_step_slice() {
   %0 = vector.step : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 2]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 2]>
     }> : vector<16xindex>
   gpu.return
@@ -1173,7 +1133,6 @@ gpu.func @vector_step_slice_unit() {
   %0 = vector.step : vector<1xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 3]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 3]>
     }> : vector<1xindex>
   gpu.return
@@ -1195,7 +1154,6 @@ gpu.func @vector_step_slice_multi_dist() {
   %0 = vector.step : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [2, 4, 2], lane_data = [1, 2, 1]>, dims = [0, 2]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [2, 4, 2], lane_data = [1, 2, 1]>, dims = [0, 2]>
     }> : vector<16xindex>
   gpu.return
@@ -1213,7 +1171,6 @@ gpu.func @vector_shapecast_rank_increasing() {
     : vector<16xf32> to vector<1x16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<1x16xf32>
   gpu.return
@@ -1231,7 +1188,6 @@ gpu.func @vector_shapecast_rank_reducing() {
     : vector<1x16xf32> to vector<16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xf32>
   gpu.return
@@ -1249,7 +1205,6 @@ gpu.func @vector_shapecast_rank_increasing_without_slicing_layout() {
     : vector<16xf32> to vector<1x16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<1x16xf32>
   gpu.return
@@ -1267,7 +1222,6 @@ gpu.func @vector_broadcast_1d_to_2d(%laneid: index) {
   %1 = vector.broadcast %0 : vector<16xf16> to vector<16x16xf16>
   %anchor = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf16>
   gpu.return
@@ -1283,7 +1237,6 @@ gpu.func @constant_wrap_around_dim() {
   %0 = arith.constant dense<1.0> : vector<16x1xf16>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x1xf16>
   gpu.return
@@ -1301,7 +1254,6 @@ gpu.func @vector_broadcast_2d_to_3d(%laneid: index) {
   %1 = vector.broadcast %0 : vector<16x16xf16> to vector<1x16x16xf16>
   %2 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>
     }> : vector<1x16x16xf16>
   "some_use"(%2) : (vector<1x16x16xf16>) -> ()
@@ -1317,7 +1269,7 @@ gpu.module @xevm_module {
 gpu.func @vector_broadcast_2d_to_2d_noop(%laneid: index) {
   %0 = "some_op"() : () -> vector<16x1xf16>
   %1 = vector.broadcast %0 : vector<16x1xf16> to vector<16x16xf16>
-  %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
+  %2 = xegpu.convert_layout %1 <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
   "some_use"(%2) : (vector<16x16xf16>) -> ()
   gpu.return
 }
@@ -1332,7 +1284,7 @@ gpu.module @xevm_module {
 gpu.func @vector_broadcast_scalar_to_vector(%laneid: index) {
   %0 = "some_op"() : () -> f16
   %1 = vector.broadcast %0 : f16 to vector<16x16xf16>
-  %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
+  %2 = xegpu.convert_layout %1 <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
   "some_use"(%2) : (vector<16x16xf16>) -> ()
   gpu.return
 }
@@ -1379,7 +1331,6 @@ gpu.func @vector_multi_reduction_1d_to_scalar() {
       [0] : vector<32xf32> to f32
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>
     }> : f32
   gpu.return
@@ -1400,7 +1351,6 @@ gpu.func @vector_interleave() {
 
   %cl1 = xegpu.convert_layout %2
   <{
-    input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 2]>,
     target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 2]>
   }> : vector<8x4xf32>
 
@@ -1419,7 +1369,6 @@ gpu.func @vector_deinterleave() {
 
   %cl1 = xegpu.convert_layout %even
   <{
-    input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
     target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
   }> : vector<8x2xf32>
 
@@ -1455,7 +1404,6 @@ gpu.func @xegpu_dpas_mx(%arg0: !xegpu.mem_desc<8x8xf16>, %arg1: !xegpu.mem_desc<
 
   %anchor = xegpu.convert_layout %4
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index fa9897770a08e..9eed7160c0baa 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -484,7 +484,6 @@ gpu.module @xevm_module {
       %4 = vector.reduction <add>, %3 : vector<16xf32> into f32
       %anchor = xegpu.convert_layout %4
         <{
-          input_layout = #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims=[0]>,
           target_layout =  #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims=[0]>
         }>
         : f32
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..ea6d3604ac0d3 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -424,7 +424,7 @@ gpu.module @test_kernel {
     %a_tdesc = xegpu.create_nd_tdesc %arg0 : memref<16x16xf16> -> !xegpu.tensor_desc<16x16xf16, #a>
     %a = xegpu.load_nd %a_tdesc[0, 0] {layout = #a}: !xegpu.tensor_desc<16x16xf16, #a> -> vector<16x16xf16>
     %a_reduce = vector.multi_reduction <add>, %a, %acc [0, 1] : vector<16x16xf16> to f16
-    %13 = xegpu.convert_layout %a_reduce <{input_layout = #xegpu.slice<#a, dims = [0, 1]>, target_layout = #xegpu.slice<#a, dims = [0, 1]>}> : f16
+    %13 = xegpu.convert_layout %a_reduce <{target_layout = #xegpu.slice<#a, dims = [0, 1]>}> : f16
     memref.store %13, %arg1[%c0] : memref<4xf16>
     gpu.return
   }
@@ -463,19 +463,19 @@ gpu.module @test_kernel {
   //CHECK: gpu.func @convert_layout([[arg0:%.+]]: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
   //CHECK: [[cst:%.+]] = arith.constant dense<0.000000e+00> : vector<8x32x2xf16>
   //CHECK: [[e0:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [0, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
-  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
   //CHECK: [[c0:%.+]] = xegpu.convert_layout [[e0]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
-  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
   //CHECK: [[e2:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e1:%.+]] = vector.extract_strided_slice [[arg0]] {offsets = [4, 0, 0], sizes = [4, 32, 2], strides = [1, 1, 1]} : vector<8x32x2xf16> to vector<4x32x2xf16>
+  //CHECK: [[c1:%.+]] = xegpu.convert_layout [[e1]] <{input_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 2]>, target_layout = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>}> : vector<4x32x2xf16>
+  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
+  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m0:%.+]] = math.exp [[e2]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i0:%.+]] = vector.insert_strided_slice [[m0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e3:%.+]] = vector.extract_strided_slice [[c0]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m1:%.+]] = math.exp [[e3]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i1:%.+]] = vector.insert_strided_slice [[m1]], [[i0]] {offsets = [0, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e4:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 0, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m2:%.+]] = math.exp [[e4]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i2:%.+]] = vector.insert_strided_slice [[m2]], [[i1]] {offsets = [4, 0, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
-  //CHECK: [[e5:%.+]] = vector.extract_strided_slice [[c1]] {offsets = [0, 16, 0], sizes = [4, 16, 2], strides = [1, 1, 1]} : vector<4x32x2xf16> to vector<4x16x2xf16>
   //CHECK: [[m3:%.+]] = math.exp [[e5]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16, 1], lane_data = [4, 1, 1]>} : vector<4x16x2xf16>
   //CHECK: [[i3:%.+]] = vector.insert_strided_slice [[m3]], [[i2]] {offsets = [4, 16, 0], strides = [1, 1, 1]} : vector<4x16x2xf16> into vector<8x32x2xf16>
   //CHECK: gpu.return [[i3]] : vector<8x32x2xf16>
@@ -483,11 +483,33 @@ gpu.module @test_kernel {
   gpu.func @convert_layout(%B: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
     %b = xegpu.convert_layout %B <{input_layout = #lb, target_layout = #b}> : vector<8x32x2xf16>
     %e = math.exp %b : vector<8x32x2xf16>
-    %anchor = xegpu.convert_layout %e <{input_layout = #b, target_layout = #b}> : vector<8x32x2xf16>
+    %anchor = xegpu.convert_layout %e <{target_layout = #b}> : vector<8x32x2xf16>
     gpu.return %e : vector<8x32x2xf16>
   }
 }
 
+// -----
+
+// Test regrouping when input and target inst_data differ ([8, 16] vs [8, 64]).
+#in = #xegpu.layout<inst_data = [8, 16], lane_layout = [1, 16], lane_data = [1, 1]>
+#tgt = #xegpu.layout<inst_data = [8, 64], lane_layout = [1, 16], lane_data = [1, 4]>
+
+gpu.module @test_kernel {
+  //CHECK-LABEL: gpu.func @convert_layout_regroup
+  //CHECK-SAME: ([[arg0:%.+]]: vector<16x64xbf16>)
+  //CHECK: vector.extract_strided_slice [[arg0]] {offsets = [0, 0], sizes = [8, 16]
+  //CHECK: math.exp {{.*}} : vector<8x16xbf16>
+  //CHECK-NOT: vector.extract_strided_slice [[arg0]] {{.*}}sizes = [8, 64]
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x16xbf16> into vector<8x64xbf16>
+  //CHECK: xegpu.convert_layout {{.*}} : vector<8x64xbf16>
+  //CHECK: vector.insert_strided_slice {{.*}} : vector<8x64xbf16> into vector<16x64xbf16>
+  gpu.func @convert_layout_regroup(%a: vector<16x64xbf16>) -> vector<16x64xbf16> {
+    %p = math.exp %a {layout_result_0 = #in} : vector<16x64xbf16>
+    %0 = xegpu.convert_layout %p <{input_layout = #in, target_layout = #tgt}> : vector<16x64xbf16>
+    gpu.return %0 : vector<16x64xbf16>
+  }
+}
+
 // -----
 gpu.module @test_kernel {
   //CHECK-LABEL: unroll_load_matrix
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
index 3e8d183242a91..918d587f12db5 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
@@ -15,7 +15,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %negf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8]>
       }> : vector<24x32xf32>
     gpu.return
@@ -34,7 +33,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %negf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -54,7 +52,6 @@ gpu.module @test_elementwise_ops {
     %powf = math.powf %addf, %load_b : vector<24x32xf32>
     %anchor = xegpu.convert_layout %powf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -79,7 +76,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %fma
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -101,7 +97,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf16> to vector<24x32xi16>
     %anchor = xegpu.convert_layout %bitcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xi16>
     gpu.return
@@ -130,7 +125,6 @@ gpu.module @test_elementwise_ops {
     %res = arith.select %cmpi, %cmpi, %cmpf : vector<24x32xi1>, vector<24x32xi1>
     %anchor = xegpu.convert_layout %res
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xi1>
     gpu.return
@@ -155,7 +149,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %powf
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2], lane_layout = [2, 2], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2], lane_layout = [2, 2], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
index 819937d3ec86c..2969e551d93cb 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
@@ -93,7 +93,6 @@ gpu.module @test_distribution {
       : vector<256x64xf32> to vector<256xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, dims = [1]>
       }> : vector<256xf32>
     gpu.return
@@ -123,7 +122,6 @@ gpu.module @test_distribution {
     %cst_2 = arith.constant dense<[[0], [16], [32], [48], [64], [80], [96], [112], [128], [144], [160], [176], [192], [208], [224], [240], [256], [272], [288], [304], [320], [336], [352], [368], [384], [400], [416], [432], [448], [464], [480], [496]]> : vector<32x1xindex>
     %anchor = xegpu.convert_layout %cst_2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [2, 1]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [2, 1]>
       }> : vector<32x1xindex>    
     gpu.return
@@ -142,7 +140,6 @@ gpu.module @test_distribution {
     : vector<256x128xf32> to vector<128x256xf32>
     %anchor = xegpu.convert_layout %trans
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [16, 32], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [16, 32], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<128x256xf32>
       gpu.return
@@ -155,7 +152,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [16, 16] : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>
       }> : vector<256x128xi1>
     gpu.return
@@ -168,7 +164,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst16, %cst16 : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>
       }> : vector<256x128xi1>
     gpu.return
@@ -241,7 +236,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %val, %acc [1] : vector<8x256xf32> to vector<8xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 16], sg_data = [4, 16]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 16], sg_data = [4, 16]>, dims = [1]>
       }> : vector<8xf32>
     gpu.return
@@ -253,7 +247,6 @@ gpu.module @test_distribution {
     %cst_2 = arith.constant dense<0> : vector<8xindex>
     %anchor = xegpu.convert_layout %cst_2
       <{
-        input_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4], order = [0, 1]>, dims = [0]>,
         target_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4], order = [0, 1]>, dims = [0]>
       }> : vector<8xindex>
     gpu.return
@@ -277,7 +270,6 @@ gpu.module @test_distribution {
     %bcast = vector.broadcast %2 : vector<8xindex> to vector<256x8xindex>
     %anchor = xegpu.convert_layout %bcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4]>,
         target_layout = #xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4]>
       }> : vector<256x8xindex>
     gpu.return
@@ -306,7 +298,6 @@ gpu.module @test_distribution {
       : vector<128x1xf32> to vector<128x64xf32>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 1], sg_data = [16, 32], lane_layout = [8, 1], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 1], sg_data = [16, 32], lane_layout = [8, 1], lane_data = [1, 1]>
       }> : vector<128x64xf32>
     gpu.return
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
index 8c5dd6c55b99d..cdc9392f02e04 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
@@ -164,8 +164,7 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %load
       : vector<256x1xf32> to vector<256x32xf32>
     %anchor = xegpu.convert_layout %broadcast
-      <{input_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>,
-        target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>}>
+      <{target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>}>
       : vector<256x32xf32>
     gpu.return
   }
@@ -183,7 +182,6 @@ gpu.module @test_distribution {
       : vector<1x128xf32> to vector<32x128xf32>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [1, 4], sg_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [1, 4], sg_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<32x128xf32>
     gpu.return
@@ -270,7 +268,6 @@ gpu.module @test_distribution {
       %exp = math.exp %load : vector<128x64xf32>
       %anchor = xegpu.convert_layout %exp
         <{
-          input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>,
           target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>
         }> : vector<128x64xf32>
     }{sg_id_range = #xegpu.range<[2, 18]>}
@@ -304,7 +301,6 @@ gpu.module @test_distribution {
         %exp = math.exp %ld : vector<128x64xf32>
         %anchor = xegpu.convert_layout %exp
           <{
-            input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>,
             target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>
           }> : vector<128x64xf32>
     }
@@ -419,8 +415,7 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [0]
       : vector<4x128xf32> to vector<128xf32>
     %anchor = xegpu.convert_layout %reduce
-      <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>,
-      target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>}>
+      <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>}>
       : vector<128xf32>
     gpu.return
   }
@@ -437,8 +432,7 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [1]
       : vector<256x64xf32> to vector<256xf32>
     %anchor = xegpu.convert_layout %reduce
-      <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>,
-      target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>}>
+      <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>}>
       : vector<256xf32>
     gpu.return
   }
@@ -454,7 +448,6 @@ gpu.module @test_distribution {
       : vector<4x2x6x32xf16> to vector<4x2x6xf16>
       %anchor = xegpu.convert_layout %reduce
         <{
-          input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>, dims = [3]>,
           target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>, dims = [3]>
         }> : vector<4x2x6xf16>
       gpu.return
@@ -483,7 +476,6 @@ gpu.module @test_distribution {
       : vector<32x32xf32> to f32
     %anchor = xegpu.convert_layout %reduce
         <{
-          input_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [4, 4], sg_data = [8, 8]>, dims = [0, 1]>,
           target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 4], sg_data = [8, 8]>, dims = [0, 1]>
         }> : f32
     gpu.return
@@ -507,7 +499,6 @@ gpu.module @test_distribution {
     %step = vector.step : vector<128xindex>
     %anchor = xegpu.convert_layout %step
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 8], sg_data = [32, 32]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 8], sg_data = [32, 32]>, dims = [1]>
       }> : vector<128xindex>
     gpu.return
@@ -527,7 +518,6 @@ gpu.module @test_distribution {
     %step = vector.step : vector<128xindex>
     %anchor = xegpu.convert_layout %step
       <{
-        input_layout = #xegpu.layout<sg_layout = [16], sg_data = [8]>,
         target_layout = #xegpu.layout<sg_layout = [16], sg_data = [8]>
       }> : vector<128xindex>
     gpu.return
@@ -539,7 +529,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<10> : vector<4xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 1]>, dims = [1, 2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 1]>, dims = [1, 2, 3]>
       }> : vector<4xindex>
     gpu.return
@@ -554,7 +543,6 @@ gpu.module @test_distribution {
     %shape_cast = vector.shape_cast %muli : vector<128xindex> to vector<1x1x1x128xindex>
     %anchor = xegpu.convert_layout %shape_cast
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1, 1, 4], sg_data = [1, 1, 1, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1, 1, 4], sg_data = [1, 1, 1, 32]>
       }> : vector<1x1x1x128xindex>
     gpu.return
@@ -567,7 +555,6 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %muli : index to vector<4x2x6x32xindex>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>,
         target_layout = #xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>
       }> : vector<4x2x6x32xindex>
     gpu.return
@@ -585,7 +572,6 @@ gpu.module @test_distribution {
       : vector<256x32xf32> to vector<32x256xf32>
     %anchor = xegpu.convert_layout %trans
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<32x256xf32>
       gpu.return
@@ -606,7 +592,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<[[0], [16], [32], [48], [64], [80], [96], [112], [128], [144], [160], [176], [192], [208], [224], [240], [256], [272], [288], [304], [320], [336], [352], [368], [384], [400], [416], [432], [448], [464], [480], [496]]> : vector<32x1xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 1]>
       }> : vector<32x1xindex>
     gpu.return
@@ -641,7 +626,6 @@ gpu.module @test_distribution {
       ]> : vector<8x8xindex>
     %anchor = xegpu.convert_layout %cst_8x8
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2]>,
         target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2]>
       }> : vector<8x8xindex>
       gpu.return
@@ -660,14 +644,12 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<[0, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496]> : vector<32xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [32], sg_data = [1]>,
         target_layout = #xegpu.layout<sg_layout = [32], sg_data = [1]>
       }> : vector<32xindex>
     // CHECK: arith.constant dense<{{\[}}{{\[}}0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15{{\]}}{{\]}}> : vector<1x16xindex>
     %cst_1 = arith.constant dense<[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]> : vector<1x16xindex>
     %anchor_1 = xegpu.convert_layout %cst_1
       <{
-        input_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 16]>,
         target_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 16]>
       }> : vector<1x16xindex>
     gpu.return
@@ -679,7 +661,6 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %arg0 : index to vector<4x1x1xindex>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 8, 1], sg_data = [1, 1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 8, 1], sg_data = [1, 1, 1]>
       }> : vector<4x1x1xindex>
     gpu.return
@@ -698,7 +679,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [8] : vector<32xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>,
         target_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>
       }> : vector<32xi1>
     gpu.return
@@ -724,7 +704,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [16, 16] : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi1>
     gpu.return
@@ -744,7 +723,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst8 : vector<32xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>,
         target_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>
       }> : vector<32xi1>
     gpu.return
@@ -771,7 +749,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst16, %cst16 : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi1>
     gpu.return
@@ -791,7 +768,6 @@ gpu.module @test_distribution {
     %4 = vector.broadcast %3 : vector<256xf32> to vector<256x256xf32>
     %anchor = xegpu.convert_layout %4
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>
       }> : vector<256x256xf32>
     gpu.return
@@ -824,7 +800,6 @@ gpu.module @test_distribution {
     %15 = vector.multi_reduction <add>, %14, %cst_3 [1] : vector<1x32x32xf32> to vector<1x32xf32>
     %anchor = xegpu.convert_layout %15
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>, dims = [1]>
       }> : vector<1x32xf32>
     gpu.return
@@ -866,7 +841,6 @@ gpu.module @test_distribution {
       : vector<256x128xf32> to vector<128xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>, dims = [0]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>, dims = [0]>
       }> : vector<128xf32>
     gpu.return
@@ -899,7 +873,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<2x2x128x128xf32> to vector<2x2xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>, dims = [2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>, dims = [2, 3]>
       }> : vector<2x2xf32>
     gpu.return
@@ -932,7 +905,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<32x32x128x128xf32> to vector<32x32xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>, dims = [2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>, dims = [2, 3]>
       }> : vector<32x32xf32>
     gpu.return
@@ -1017,7 +989,6 @@ gpu.module @test_distribution {
                                    target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>}> : vector<128x256xf32>
     %anchor = xegpu.convert_layout %2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>
       }> : vector<128x256xf32>
     gpu.return
@@ -1060,7 +1031,6 @@ gpu.module @test_distribution {
                                    target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>}> : vector<8x128x256xf32>
     %anchor = xegpu.convert_layout %2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>
       }> : vector<8x128x256xf32>
     gpu.return
@@ -1076,7 +1046,7 @@ gpu.module @test_distribution {
     %11 = xegpu.load %10[%offset], %mask <{layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>}> : i64, vector<32x32xindex>, vector<32x32xi1> -> vector<32x32xf32>
     %12 = vector.multi_reduction <add>, %11, %cst_0 [0, 1] : vector<32x32xf32> to f32
     // CHECK-NOT: xegpu.convert_layout
-    %13 = xegpu.convert_layout %12 <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>, target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>}> : f32
+    %13 = xegpu.convert_layout %12 <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>}> : f32
     gpu.return
   }
 
@@ -1111,7 +1081,6 @@ gpu.module @test_distribution {
     %bcast2 = vector.broadcast %scast2 : vector<256x16x1x256x16x16xf32> to vector<256x16x16x256x16x16xf32>
     %anchor = xegpu.convert_layout %bcast2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1, 1, 8, 1, 1], sg_data = [32, 16, 16, 32, 16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1, 1, 8, 1, 1], sg_data = [32, 16, 16, 32, 16, 16]>
       }> : vector<256x16x16x256x16x16xf32>
     gpu.return
@@ -1296,7 +1265,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<1.0> : vector<256x128xf32>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xf32>
     gpu.return
@@ -1340,7 +1308,6 @@ gpu.module @test_distribution {
     %bitcast2 = vector.bitcast %add : vector<256x256xi16> to vector<256x128xi32>
     %anchor = xegpu.convert_layout %bitcast2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi32>
     gpu.return
@@ -1361,7 +1328,6 @@ gpu.module @test_distribution {
       : vector<256x128xf32> -> vector<256x256xf32>
     %anchor = xegpu.convert_layout %interleave
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64]>
       }> : vector<256x256xf32>
     gpu.return
@@ -1375,7 +1341,6 @@ gpu.module @test_distribution {
     %deinterleave:2 = vector.deinterleave %load : vector<256x256xf32> -> vector<256x128xf32>
     %anchor = xegpu.convert_layout %deinterleave#0
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xf32>
     gpu.return


From llvmlistbot at llvm.org  Mon Jul 20 15:55:44 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 20 Jul 2026 15:55:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Lower lane_data repack
 convert_layout to bitcast_shuffle (PR #210837)
Message-ID: <llvm.org/llvm/llvm-project/pull/210837@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/210837

Extend the SgToLaneConvertLayout pattern to lower a convert_layout that only repacks lane_data between round-robin and contiguous form (keeping lane_layout and order unchanged) into xegpu.bitcast_shuffle. Each lane keeps the same elements and total bits along the repacked dimension, but their assignment to lanes changes, so the data is moved across lanes with
a bit-preserving bitcast_shuffle.

>From 90396243af4950884d9e936f4640720baf8dcd9a Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Mon, 20 Jul 2026 18:18:36 +0000
Subject: [PATCH 1/2] [mlir][xegpu] Add xegpu.bitcast_shuffle op

Add a new lane-level XeGPU operation that reinterprets a lane's fragment
as a different element type (possibly with a different number of elements
per lane) while re-distributing the data across the lanes of the subgroup.

The op takes a scalar or 1D vector of numerical type and produces a scalar
or 1D vector of a different type with the same total number of bits per
lane. A verifier enforces the different-type and equal-bit-size
constraints. Round-trip and invalid tests are included.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../include/mlir/Dialect/XeGPU/IR/XeGPUOps.td | 66 +++++++++++++++++++
 .../mlir/Dialect/XeGPU/IR/XeGPUTypes.td       |  5 ++
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp        | 29 ++++++++
 mlir/test/Dialect/XeGPU/invalid.mlir          | 14 ++++
 mlir/test/Dialect/XeGPU/ops.mlir              | 28 ++++++++
 5 files changed, 142 insertions(+)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..0a6dc0b0240c5 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1491,6 +1491,72 @@ def XeGPU_TruncfOp
   let hasVerifier = 1;
 }
 
+def XeGPU_BitcastShuffleOp
+    : XeGPU_Op<"bitcast_shuffle", [Pure]> {
+  let summary = "Bit-preserving type conversion and re-distribution of a "
+                "subgroup's fragments across its lanes.";
+
+  let description = [{
+    `xegpu.bitcast_shuffle` is a lane-level operation. The `source` operand is
+    the fragment (a scalar or 1D vector) held by one lane, and the `result` is
+    the fragment that lane holds afterwards. It reinterprets the fragments as a
+    different element type, possibly with a different number of elements per
+    lane, and at the same time re-distributes the data across the lanes of the
+    subgroup.
+
+    Both `source` and `result` must be a scalar or a 1D vector of numerical
+    type, and they must have a different type. When they have the same number of
+    elements the operation degenerates into a plain bitcast. Otherwise the total
+    number of bits per lane must be equal on both sides.
+
+    For example, consider `vector<2xi16> -> i32` with a subgroup size of 8. Each
+    lane's `source` is a `vector<2xi16>` of two 16-bit half-words, so the
+    subgroup holds 8 * 2 = 16 half-words in total. Number these half-words 0..15
+    and denote a lane's `source` as `<a,b>`, so lane `i` starts with half-words
+    `<2*i, 2*i+1>`. The `result` `i32` holds two packed half-words, denoted
+    `a|b` (`a` in the low bits, `b` in the high bits):
+
+    ```
+    lane:       0      1      2      3      4      5      6      7
+    source:   <0,1>  <2,3>  <4,5>  <6,7>  <8,9> <10,11> <12,13> <14,15>
+    result:    0|2    4|6   8|10   12|14   1|3    5|7    9|11   13|15
+    ```
+
+    The half-words are gathered one `source` element position at a time, in
+    lane order: first every lane's element 0 (half-words 0, 2, 4, ..., 14), then
+    every lane's element 1 (half-words 1, 3, 5, ..., 15). This sequence is then
+    handed back out two half-words per lane, in lane order, to form each lane's
+    `result`. In general each lane's two result half-words come from two
+    different lanes' `source` fragments -- that is the cross-lane shuffle.
+
+    The reverse pair `i32 -> vector<2xi16>` undoes this exactly, so bitcasting
+    to one type and back yields the original fragments.
+
+    The operation assumes a full subgroup (the subgroup size equals the maximum
+    subgroup size) and that all lanes execute the same dynamic instance of the
+    operation; otherwise the behavior is undefined.
+  }];
+
+  let arguments = (ins XeGPU_BitcastShuffleType:$source);
+  let results = (outs XeGPU_BitcastShuffleType:$result);
+
+  let extraClassDeclaration = [{
+    Type getSourceType() {
+      return getSource().getType();
+    }
+
+    Type getResultType() {
+      return getResult().getType();
+    }
+  }];
+
+  let assemblyFormat = [{
+    $source attr-dict `:` type($source) `->` type($result)
+  }];
+
+  let hasVerifier = 1;
+}
+
 def XeGPU_DpasMxOp : XeGPU_Op<"dpas_mx", [Pure, AttrSizedOperandSegments,
                                           AnchorLayoutInterface]> {
   let summary = "It performs scaled mma computation";
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
index 0423303c23493..dfaea842ba4cc 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
@@ -31,6 +31,11 @@ def XeGPU_VectorOrScalarType
     : AnyTypeOf<[VectorOfRankAndType<[1,2,3,4,5,6,7,8], [XeGPU_ScalarType, Index]>, XeGPU_ScalarType]>;
 def XeGPU_GatherScatterBaseAddrType
     : AnyTypeOf<[MemRefRankOf<[XeGPU_ScalarType], [1]>, XeGPU_PointerType]>;
+// Operand/result type of xegpu.bitcast_shuffle: a numerical scalar or a 1D
+// vector of numerical type.
+def XeGPU_BitcastShuffleType
+    : AnyTypeOf<[FixedVectorOfRankAndType<[1], [XeGPU_ScalarType]>,
+                 XeGPU_ScalarType]>;
 
 // common base class for types in XeGPU dialect
 class XeGPUTypeDef<string name, string typeMnemonic, list<Trait> traits = [],
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..24cd67720696f 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -937,6 +937,35 @@ LogicalResult TruncfOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// XeGPU_BitcastShuffleOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult BitcastShuffleOp::verify() {
+  Type srcType = getSourceType();
+  Type resType = getResultType();
+
+  // A bitcast_shuffle only makes sense between two different types; a
+  // same-type shuffle is a no-op.
+  if (srcType == resType)
+    return emitOpError("source and result must have different types.");
+
+  // Compute the total number of bits on both sides. The operation is
+  // bit-preserving, so the source and result must carry the same amount of
+  // data per lane.
+  auto sizeInBits = [](Type type) -> int64_t {
+    if (auto vecType = dyn_cast<VectorType>(type))
+      return vecType.getNumElements() * vecType.getElementTypeBitWidth();
+    return type.getIntOrFloatBitWidth();
+  };
+
+  if (sizeInBits(srcType) != sizeInBits(resType))
+    return emitOpError(
+        "source and result must have the same total size in bits.");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // XeGPU_DpasMxOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..2ae6cacd0bf50 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -693,6 +693,20 @@ func.func @truncf_invalid_result_size(%a: vector<8x16xf16>) {
   return
 }
 
+// -----
+func.func @bitcast_shuffle_same_type(%a: vector<2xi16>) {
+  // expected-error at +1 {{op source and result must have different types}}
+  %1 = xegpu.bitcast_shuffle %a : vector<2xi16> -> vector<2xi16>
+  return
+}
+
+// -----
+func.func @bitcast_shuffle_size_mismatch(%a: i32) {
+  // expected-error at +1 {{op source and result must have the same total size in bits}}
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<4xi16>
+  return
+}
+
 // -----
 func.func @dpas_mx_acc_result_type_mismatch(%a : vector<8x16xf8E5M2>, %b: vector<16x16xf8E5M2>, %acc: vector<8x16xbf16>) {
   // expected-error at +1 {{Accumulator type must match result type.}}
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..3dbe570b3b1ad 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -688,6 +688,34 @@ gpu.func @truncf(%a: vector<8x16xf16>) {
   gpu.return
 }
 
+// CHECK-LABEL: gpu.func @bitcast_shuffle_scalar_to_vector_16
+gpu.func @bitcast_shuffle_scalar_to_vector_16(%a: i32) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : i32 -> vector<2xi16>
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<2xi16>
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_vector_16_to_scalar
+gpu.func @bitcast_shuffle_vector_16_to_scalar(%a: vector<2xi16>) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : vector<2xi16> -> i32
+  %1 = xegpu.bitcast_shuffle %a : vector<2xi16> -> i32
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_scalar_to_vector_8
+gpu.func @bitcast_shuffle_scalar_to_vector_8(%a: i32) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : i32 -> vector<4xi8>
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<4xi8>
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_vector_8_to_scalar
+gpu.func @bitcast_shuffle_vector_8_to_scalar(%a: vector<4xi8>) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : vector<4xi8> -> i32
+  %1 = xegpu.bitcast_shuffle %a : vector<4xi8> -> i32
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @dpas_mx
 gpu.func @dpas_mx(%a : vector<8x32xf8E5M2>, %b: vector<32x16xf8E5M2>, %acc: vector<8x16xbf16>, %a_scale: vector<8x1xf8E8M0FNU>, %b_scale: vector<1x16xf8E8M0FNU>) {
   // CHECK: %{{.+}} = xegpu.dpas_mx %{{.+}}, %{{.+}}, %{{.+}} scale_a = %{{.+}} scale_b = %{{.+}} : (vector<8x32xf8E5M2>, vector<32x16xf8E5M2>, vector<8x16xbf16>, vector<8x1xf8E8M0FNU>, vector<1x16xf8E8M0FNU>) -> vector<8x16xbf16>

>From 2148ded2d83fd8bb86d4a824318907066a051ec8 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Mon, 20 Jul 2026 22:52:47 +0000
Subject: [PATCH 2/2] [mlir][xegpu] Lower lane_data repack convert_layout to
 bitcast_shuffle

Extend the SgToLaneConvertLayout pattern to lower a convert_layout that
only repacks lane_data between round-robin and contiguous form (keeping
lane_layout and order unchanged) into xegpu.bitcast_shuffle. Each lane
keeps the same elements and total bits along the repacked dimension, but
their assignment to lanes changes, so the data is moved across lanes with
a bit-preserving bitcast_shuffle.

The repacked dimension may be either of the two innermost dimensions
(sg-to-lane distribution is 2D); a change to a 3rd-innermost-or-higher
dimension, or a run wider than 64 bits, is not supported. Both repack
directions are handled since bitcast_shuffle is reversible.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 150 ++++++++++++++++++
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 102 ++++++++++++
 2 files changed, 252 insertions(+)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..509f9882decc8 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -1665,6 +1665,115 @@ shuffleDataAsLaneLayoutChange(ConversionPatternRewriter &rewriter, Location loc,
   return res;
 }
 
+/// Redistributes `src` for a `convert_layout` that repacks the `lane_data`
+/// along dimension `repackDim` between round-robin and contiguous, keeping
+/// `lane_layout` and `order` unchanged. Each lane keeps the same `k` elements
+/// and total bits along `repackDim`, but their assignment to lanes changes, so
+/// a `xegpu.bitcast_shuffle` moves the data across lanes: it converts each run
+/// of `k` elements between the round-robin form `vector<k x iElem>` (each lane
+/// owns `k` strided elements) and the contiguous form `vector<1 x iWide>` (each
+/// lane owns one packed `k * elemBits` value), with a `vector.bitcast` to and
+/// from the original element type on either side.
+///
+/// `inputData`/`targetData` are the `repackDim` `lane_data` of the input and
+/// target layouts; exactly one must be 1 (round-robin) and the other `k`
+/// (contiguous). Returns failure if that does not hold or the packed run would
+/// exceed the widest representable integer (64 bits).
+static FailureOr<Value>
+repackLaneData(ConversionPatternRewriter &rewriter, Location loc, Value src,
+               int64_t repackDim, int64_t inputData, int64_t targetData) {
+  auto srcTy = dyn_cast<VectorType>(src.getType());
+  if (!srcTy)
+    return failure();
+  int64_t rank = srcTy.getRank();
+  Type elemTy = srcTy.getElementType();
+  int64_t elemBits = elemTy.getIntOrFloatBitWidth();
+  int64_t k = srcTy.getShape()[repackDim];
+
+  bool roundRobinToContig = inputData == 1 && targetData == k;
+  bool contigToRoundRobin = inputData == k && targetData == 1;
+  if (!roundRobinToContig && !contigToRoundRobin)
+    return failure();
+
+  int64_t wideBits = k * elemBits;
+  if (wideBits > 64)
+    return failure();
+
+  MLIRContext *ctx = rewriter.getContext();
+  VectorType roundRobinIntTy = VectorType::get({k}, IntegerType::get(ctx, elemBits));
+  VectorType contigIntTy = VectorType::get({1}, IntegerType::get(ctx, wideBits));
+  VectorType runTy = VectorType::get({k}, elemTy);
+
+  // Repacks one run (vector<k x elemTy>) into the target form, returning a
+  // vector<k x elemTy>.
+  auto repackRun = [&](Value run) -> Value {
+    if (roundRobinToContig) {
+      Value asInt =
+          vector::BitCastOp::create(rewriter, loc, roundRobinIntTy, run);
+      Value shuffled =
+          xegpu::BitcastShuffleOp::create(rewriter, loc, contigIntTy, asInt);
+      return vector::BitCastOp::create(rewriter, loc, runTy, shuffled);
+    }
+    Value asWide = vector::BitCastOp::create(rewriter, loc, contigIntTy, run);
+    Value shuffled =
+        xegpu::BitcastShuffleOp::create(rewriter, loc, roundRobinIntTy, asWide);
+    return vector::BitCastOp::create(rewriter, loc, runTy, shuffled);
+  };
+
+  if (rank == 1)
+    return repackRun(src);
+
+  // When `repackDim` is innermost each run is a contiguous sub-vector and can
+  // be extracted and re-inserted as a whole.
+  if (repackDim == rank - 1) {
+    SmallVector<int64_t> outerShape(srcTy.getShape().drop_back());
+    int64_t numRuns = computeProduct(outerShape);
+    SmallVector<int64_t> outerStrides = computeStrides(outerShape);
+    Value result = arith::ConstantOp::create(rewriter, loc, srcTy,
+                                             rewriter.getZeroAttr(srcTy));
+    for (int64_t i = 0; i < numRuns; ++i) {
+      SmallVector<int64_t> pos = delinearize(i, outerStrides);
+      Value run = vector::ExtractOp::create(rewriter, loc, src, pos);
+      result = vector::InsertOp::create(rewriter, loc, repackRun(run), result,
+                                        pos);
+    }
+    return result;
+  }
+
+  // Otherwise each run is strided along `repackDim`, so it is gathered and
+  // scattered element by element for every combination of the kept indices.
+  SmallVector<int64_t> keptShape;
+  SmallVector<int64_t> keptDims;
+  for (int64_t d = 0; d < rank; ++d)
+    if (d != repackDim) {
+      keptShape.push_back(srcTy.getShape()[d]);
+      keptDims.push_back(d);
+    }
+  int64_t numRuns = computeProduct(keptShape);
+  SmallVector<int64_t> keptStrides = computeStrides(keptShape);
+  Value result = arith::ConstantOp::create(rewriter, loc, srcTy,
+                                           rewriter.getZeroAttr(srcTy));
+  for (int64_t i = 0; i < numRuns; ++i) {
+    SmallVector<int64_t> keptPos = delinearize(i, keptStrides);
+    SmallVector<int64_t> pos(rank, 0);
+    for (auto [dim, coord] : llvm::zip_equal(keptDims, keptPos))
+      pos[dim] = coord;
+    SmallVector<Value> runElems;
+    for (int64_t j = 0; j < k; ++j) {
+      pos[repackDim] = j;
+      runElems.push_back(vector::ExtractOp::create(rewriter, loc, src, pos));
+    }
+    Value repacked = repackRun(
+        vector::FromElementsOp::create(rewriter, loc, runTy, runElems));
+    for (int64_t j = 0; j < k; ++j) {
+      pos[repackDim] = j;
+      Value elem = vector::ExtractOp::create(rewriter, loc, repacked, j);
+      result = vector::InsertOp::create(rewriter, loc, elem, result, pos);
+    }
+  }
+  return result;
+}
+
 /// Folds a subgroup-level ConvertLayout op with compatible lane layouts.
 struct SgToLaneConvertLayout
     : public OpConversionPattern<xegpu::ConvertLayoutOp> {
@@ -1721,6 +1830,47 @@ struct SgToLaneConvertLayout
       }
     }
 
+    // Handle a pure `lane_data` repack: `lane_layout` and `order` are unchanged
+    // and exactly one dimension's `lane_data` differs, switching between
+    // round-robin (lane_data 1) and contiguous (lane_data == run length). The
+    // number of elements per lane is unchanged, but their assignment to lanes
+    // is not, so the data is moved across lanes with `xegpu.bitcast_shuffle`.
+    //
+    // The changed dimension must be one of the two innermost ones: sg-to-lane
+    // distribution is 2D, with any higher (leading) dimensions being unit, so a
+    // repack of a 3rd-innermost-or-higher dimension is not supported.
+    if (inputLayout.getEffectiveOrderAsInt() ==
+            targetLayout.getEffectiveOrderAsInt() &&
+        inputLayout.getEffectiveLaneLayoutAsInt() ==
+            targetLayout.getEffectiveLaneLayoutAsInt()) {
+      // lane_layout and lane_data always share the same rank (enforced by the
+      // LayoutAttr verifier), so the two lane_data arrays have equal size here
+      // since their lane_layouts are equal.
+      auto laneData = inputLayout.getEffectiveLaneDataAsInt();
+      auto targetLaneData = targetLayout.getEffectiveLaneDataAsInt();
+      // Find the single dimension whose lane_data changed.
+      SmallVector<int64_t> changedDims;
+      for (auto [dim, data] : llvm::enumerate(llvm::zip_equal(
+               ArrayRef<int64_t>(laneData), ArrayRef<int64_t>(targetLaneData))))
+        if (std::get<0>(data) != std::get<1>(data))
+          changedDims.push_back(dim);
+
+      int64_t rank = laneData.size();
+      // Only a single dimension may change, and it must be one of the two
+      // innermost dimensions (>= rank - 2).
+      if (changedDims.size() == 1 && changedDims[0] >= rank - 2) {
+        int64_t repackDim = changedDims[0];
+        FailureOr<Value> res =
+            repackLaneData(rewriter, op.getLoc(), adaptor.getSource(),
+                           repackDim, laneData[repackDim],
+                           targetLaneData[repackDim]);
+        if (succeeded(res)) {
+          rewriter.replaceOp(op, *res);
+          return success();
+        }
+      }
+    }
+
     return rewriter.notifyMatchFailure(
         op, "lowering incompatible convert_layout not yet supported");
   }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..584d48d3a3a5e 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -1062,6 +1062,108 @@ gpu.func @convert_layout_scalar() {
 }
 }
 
+// -----
+// A convert_layout that repacks the innermost lane_data from round-robin
+// ([1, 1]) to contiguous ([1, 4]) while keeping lane_layout unchanged lowers to
+// a per-row xegpu.bitcast_shuffle: each lane's 4 strided bf16 elements are
+// bitcast to i16, shuffled across lanes into one contiguous i64, and bitcast
+// back. The non-distributed outer dim (8 rows) is repacked independently.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<8x64xbf16> to vector<8x4xbf16>
+// CHECK:         %[[E0:.*]] = vector.extract %[[SRC]][0] : vector<4xbf16> from vector<8x4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %[[E0]] : vector<4xbf16> to vector<4xi16>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<4xi16> -> vector<1xi64>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<1xi64> to vector<4xbf16>
+// CHECK-COUNT-7: xegpu.bitcast_shuffle %{{.*}} : vector<4xi16> -> vector<1xi64>
+gpu.func @convert_layout_repack_innermost_lane_data() {
+  %src = "some_op"() : () -> vector<8x64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 4]>
+    }> : vector<8x64xbf16>
+  "some_use"(%cvt) : (vector<8x64xbf16>) -> ()
+  gpu.return
+}
+}
+
+// -----
+// The reverse repack (contiguous [1, 4] to round-robin [1, 1]) uses the
+// opposite bitcast_shuffle: each lane's contiguous i64 is shuffled back into 4
+// strided i16 elements.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data_reverse
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<8x64xbf16> to vector<8x4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %{{.*}} : vector<4xbf16> to vector<1xi64>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<1xi64> -> vector<4xi16>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<4xi16> to vector<4xbf16>
+gpu.func @convert_layout_repack_innermost_lane_data_reverse() {
+  %src = "some_op"() : () -> vector<8x64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 4]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
+    }> : vector<8x64xbf16>
+  "some_use"(%cvt) : (vector<8x64xbf16>) -> ()
+  gpu.return
+}
+}
+
+// -----
+// A 1D convert_layout repack is a single bitcast_shuffle with no outer loop.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data_1d
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<64xbf16> to vector<4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %[[SRC]] : vector<4xbf16> to vector<4xi16>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<4xi16> -> vector<1xi64>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<1xi64> to vector<4xbf16>
+gpu.func @convert_layout_repack_innermost_lane_data_1d() {
+  %src = "some_op"() : () -> vector<64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
+      target_layout = #xegpu.layout<lane_layout = [16], lane_data = [4]>
+    }> : vector<64xbf16>
+  "some_use"(%cvt) : (vector<64xbf16>) -> ()
+  gpu.return
+}
+}
+
+// -----
+// A repack of the 2nd-innermost (outer, distributed) dimension is also
+// supported: lane_layout = [16, 1] distributes dim 0, and its lane_data goes
+// from round-robin ([1, 1]) to contiguous ([4, 1]). Because the repacked
+// elements are strided along dim 0 they are gathered element by element into a
+// run, shuffled, then scattered back.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_second_innermost_lane_data
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<64x8xbf16> to vector<4x8xbf16>
+// CHECK:         %[[E0:.*]] = vector.extract %[[SRC]][0, 0] : bf16 from vector<4x8xbf16>
+// CHECK:         %[[E1:.*]] = vector.extract %[[SRC]][1, 0] : bf16 from vector<4x8xbf16>
+// CHECK:         %[[E2:.*]] = vector.extract %[[SRC]][2, 0] : bf16 from vector<4x8xbf16>
+// CHECK:         %[[E3:.*]] = vector.extract %[[SRC]][3, 0] : bf16 from vector<4x8xbf16>
+// CHECK:         %[[FE:.*]] = vector.from_elements %[[E0]], %[[E1]], %[[E2]], %[[E3]] : vector<4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %[[FE]] : vector<4xbf16> to vector<4xi16>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<4xi16> -> vector<1xi64>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<1xi64> to vector<4xbf16>
+// CHECK-COUNT-7: xegpu.bitcast_shuffle %{{.*}} : vector<4xi16> -> vector<1xi64>
+gpu.func @convert_layout_repack_second_innermost_lane_data() {
+  %src = "some_op"() : () -> vector<64x8xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [4, 1]>
+    }> : vector<64x8xbf16>
+  "some_use"(%cvt) : (vector<64x8xbf16>) -> ()
+  gpu.return
+}
+}
+
 // -----
 // load_matrix and store_matrix with coordinate computation (offsets [0,0])
 gpu.module @xevm_module {


From llvmlistbot at llvm.org  Mon Jul 20 15:56:17 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Mon, 20 Jul 2026 15:56:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Lower lane_data repack
 convert_layout to bitcast_shuffle (PR #210837)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210837@github.com>
Message-ID: <6a5ea791.9edb5c9d.280be3.680c@mx.google.com>


https://github.com/Jianhui-Li edited https://github.com/llvm/llvm-project/pull/210837

From llvmlistbot at llvm.org  Mon Jul 20 15:56:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 15:56:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Lower lane_data repack
 convert_layout to bitcast_shuffle (PR #210837)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210837@github.com>
Message-ID: <6a5ea793.510f576d.2f694c.8485@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

Extend the SgToLaneConvertLayout pattern to lower a convert_layout that only repacks lane_data between round-robin and contiguous form (keeping lane_layout and order unchanged) into xegpu.bitcast_shuffle. Each lane keeps the same elements and total bits along the repacked dimension, but their assignment to lanes changes, so the data is moved across lanes with
a bit-preserving bitcast_shuffle.

---

Patch is 22.10 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210837.diff


7 Files Affected:

- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td (+66) 
- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td (+5) 
- (modified) mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp (+29) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp (+150) 
- (modified) mlir/test/Dialect/XeGPU/invalid.mlir (+14) 
- (modified) mlir/test/Dialect/XeGPU/ops.mlir (+28) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir (+102) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..0a6dc0b0240c5 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1491,6 +1491,72 @@ def XeGPU_TruncfOp
   let hasVerifier = 1;
 }
 
+def XeGPU_BitcastShuffleOp
+    : XeGPU_Op<"bitcast_shuffle", [Pure]> {
+  let summary = "Bit-preserving type conversion and re-distribution of a "
+                "subgroup's fragments across its lanes.";
+
+  let description = [{
+    `xegpu.bitcast_shuffle` is a lane-level operation. The `source` operand is
+    the fragment (a scalar or 1D vector) held by one lane, and the `result` is
+    the fragment that lane holds afterwards. It reinterprets the fragments as a
+    different element type, possibly with a different number of elements per
+    lane, and at the same time re-distributes the data across the lanes of the
+    subgroup.
+
+    Both `source` and `result` must be a scalar or a 1D vector of numerical
+    type, and they must have a different type. When they have the same number of
+    elements the operation degenerates into a plain bitcast. Otherwise the total
+    number of bits per lane must be equal on both sides.
+
+    For example, consider `vector<2xi16> -> i32` with a subgroup size of 8. Each
+    lane's `source` is a `vector<2xi16>` of two 16-bit half-words, so the
+    subgroup holds 8 * 2 = 16 half-words in total. Number these half-words 0..15
+    and denote a lane's `source` as `<a,b>`, so lane `i` starts with half-words
+    `<2*i, 2*i+1>`. The `result` `i32` holds two packed half-words, denoted
+    `a|b` (`a` in the low bits, `b` in the high bits):
+
+    ```
+    lane:       0      1      2      3      4      5      6      7
+    source:   <0,1>  <2,3>  <4,5>  <6,7>  <8,9> <10,11> <12,13> <14,15>
+    result:    0|2    4|6   8|10   12|14   1|3    5|7    9|11   13|15
+    ```
+
+    The half-words are gathered one `source` element position at a time, in
+    lane order: first every lane's element 0 (half-words 0, 2, 4, ..., 14), then
+    every lane's element 1 (half-words 1, 3, 5, ..., 15). This sequence is then
+    handed back out two half-words per lane, in lane order, to form each lane's
+    `result`. In general each lane's two result half-words come from two
+    different lanes' `source` fragments -- that is the cross-lane shuffle.
+
+    The reverse pair `i32 -> vector<2xi16>` undoes this exactly, so bitcasting
+    to one type and back yields the original fragments.
+
+    The operation assumes a full subgroup (the subgroup size equals the maximum
+    subgroup size) and that all lanes execute the same dynamic instance of the
+    operation; otherwise the behavior is undefined.
+  }];
+
+  let arguments = (ins XeGPU_BitcastShuffleType:$source);
+  let results = (outs XeGPU_BitcastShuffleType:$result);
+
+  let extraClassDeclaration = [{
+    Type getSourceType() {
+      return getSource().getType();
+    }
+
+    Type getResultType() {
+      return getResult().getType();
+    }
+  }];
+
+  let assemblyFormat = [{
+    $source attr-dict `:` type($source) `->` type($result)
+  }];
+
+  let hasVerifier = 1;
+}
+
 def XeGPU_DpasMxOp : XeGPU_Op<"dpas_mx", [Pure, AttrSizedOperandSegments,
                                           AnchorLayoutInterface]> {
   let summary = "It performs scaled mma computation";
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
index 0423303c23493..dfaea842ba4cc 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
@@ -31,6 +31,11 @@ def XeGPU_VectorOrScalarType
     : AnyTypeOf<[VectorOfRankAndType<[1,2,3,4,5,6,7,8], [XeGPU_ScalarType, Index]>, XeGPU_ScalarType]>;
 def XeGPU_GatherScatterBaseAddrType
     : AnyTypeOf<[MemRefRankOf<[XeGPU_ScalarType], [1]>, XeGPU_PointerType]>;
+// Operand/result type of xegpu.bitcast_shuffle: a numerical scalar or a 1D
+// vector of numerical type.
+def XeGPU_BitcastShuffleType
+    : AnyTypeOf<[FixedVectorOfRankAndType<[1], [XeGPU_ScalarType]>,
+                 XeGPU_ScalarType]>;
 
 // common base class for types in XeGPU dialect
 class XeGPUTypeDef<string name, string typeMnemonic, list<Trait> traits = [],
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..24cd67720696f 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -937,6 +937,35 @@ LogicalResult TruncfOp::verify() {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// XeGPU_BitcastShuffleOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult BitcastShuffleOp::verify() {
+  Type srcType = getSourceType();
+  Type resType = getResultType();
+
+  // A bitcast_shuffle only makes sense between two different types; a
+  // same-type shuffle is a no-op.
+  if (srcType == resType)
+    return emitOpError("source and result must have different types.");
+
+  // Compute the total number of bits on both sides. The operation is
+  // bit-preserving, so the source and result must carry the same amount of
+  // data per lane.
+  auto sizeInBits = [](Type type) -> int64_t {
+    if (auto vecType = dyn_cast<VectorType>(type))
+      return vecType.getNumElements() * vecType.getElementTypeBitWidth();
+    return type.getIntOrFloatBitWidth();
+  };
+
+  if (sizeInBits(srcType) != sizeInBits(resType))
+    return emitOpError(
+        "source and result must have the same total size in bits.");
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // XeGPU_DpasMxOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..509f9882decc8 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -1665,6 +1665,115 @@ shuffleDataAsLaneLayoutChange(ConversionPatternRewriter &rewriter, Location loc,
   return res;
 }
 
+/// Redistributes `src` for a `convert_layout` that repacks the `lane_data`
+/// along dimension `repackDim` between round-robin and contiguous, keeping
+/// `lane_layout` and `order` unchanged. Each lane keeps the same `k` elements
+/// and total bits along `repackDim`, but their assignment to lanes changes, so
+/// a `xegpu.bitcast_shuffle` moves the data across lanes: it converts each run
+/// of `k` elements between the round-robin form `vector<k x iElem>` (each lane
+/// owns `k` strided elements) and the contiguous form `vector<1 x iWide>` (each
+/// lane owns one packed `k * elemBits` value), with a `vector.bitcast` to and
+/// from the original element type on either side.
+///
+/// `inputData`/`targetData` are the `repackDim` `lane_data` of the input and
+/// target layouts; exactly one must be 1 (round-robin) and the other `k`
+/// (contiguous). Returns failure if that does not hold or the packed run would
+/// exceed the widest representable integer (64 bits).
+static FailureOr<Value>
+repackLaneData(ConversionPatternRewriter &rewriter, Location loc, Value src,
+               int64_t repackDim, int64_t inputData, int64_t targetData) {
+  auto srcTy = dyn_cast<VectorType>(src.getType());
+  if (!srcTy)
+    return failure();
+  int64_t rank = srcTy.getRank();
+  Type elemTy = srcTy.getElementType();
+  int64_t elemBits = elemTy.getIntOrFloatBitWidth();
+  int64_t k = srcTy.getShape()[repackDim];
+
+  bool roundRobinToContig = inputData == 1 && targetData == k;
+  bool contigToRoundRobin = inputData == k && targetData == 1;
+  if (!roundRobinToContig && !contigToRoundRobin)
+    return failure();
+
+  int64_t wideBits = k * elemBits;
+  if (wideBits > 64)
+    return failure();
+
+  MLIRContext *ctx = rewriter.getContext();
+  VectorType roundRobinIntTy = VectorType::get({k}, IntegerType::get(ctx, elemBits));
+  VectorType contigIntTy = VectorType::get({1}, IntegerType::get(ctx, wideBits));
+  VectorType runTy = VectorType::get({k}, elemTy);
+
+  // Repacks one run (vector<k x elemTy>) into the target form, returning a
+  // vector<k x elemTy>.
+  auto repackRun = [&](Value run) -> Value {
+    if (roundRobinToContig) {
+      Value asInt =
+          vector::BitCastOp::create(rewriter, loc, roundRobinIntTy, run);
+      Value shuffled =
+          xegpu::BitcastShuffleOp::create(rewriter, loc, contigIntTy, asInt);
+      return vector::BitCastOp::create(rewriter, loc, runTy, shuffled);
+    }
+    Value asWide = vector::BitCastOp::create(rewriter, loc, contigIntTy, run);
+    Value shuffled =
+        xegpu::BitcastShuffleOp::create(rewriter, loc, roundRobinIntTy, asWide);
+    return vector::BitCastOp::create(rewriter, loc, runTy, shuffled);
+  };
+
+  if (rank == 1)
+    return repackRun(src);
+
+  // When `repackDim` is innermost each run is a contiguous sub-vector and can
+  // be extracted and re-inserted as a whole.
+  if (repackDim == rank - 1) {
+    SmallVector<int64_t> outerShape(srcTy.getShape().drop_back());
+    int64_t numRuns = computeProduct(outerShape);
+    SmallVector<int64_t> outerStrides = computeStrides(outerShape);
+    Value result = arith::ConstantOp::create(rewriter, loc, srcTy,
+                                             rewriter.getZeroAttr(srcTy));
+    for (int64_t i = 0; i < numRuns; ++i) {
+      SmallVector<int64_t> pos = delinearize(i, outerStrides);
+      Value run = vector::ExtractOp::create(rewriter, loc, src, pos);
+      result = vector::InsertOp::create(rewriter, loc, repackRun(run), result,
+                                        pos);
+    }
+    return result;
+  }
+
+  // Otherwise each run is strided along `repackDim`, so it is gathered and
+  // scattered element by element for every combination of the kept indices.
+  SmallVector<int64_t> keptShape;
+  SmallVector<int64_t> keptDims;
+  for (int64_t d = 0; d < rank; ++d)
+    if (d != repackDim) {
+      keptShape.push_back(srcTy.getShape()[d]);
+      keptDims.push_back(d);
+    }
+  int64_t numRuns = computeProduct(keptShape);
+  SmallVector<int64_t> keptStrides = computeStrides(keptShape);
+  Value result = arith::ConstantOp::create(rewriter, loc, srcTy,
+                                           rewriter.getZeroAttr(srcTy));
+  for (int64_t i = 0; i < numRuns; ++i) {
+    SmallVector<int64_t> keptPos = delinearize(i, keptStrides);
+    SmallVector<int64_t> pos(rank, 0);
+    for (auto [dim, coord] : llvm::zip_equal(keptDims, keptPos))
+      pos[dim] = coord;
+    SmallVector<Value> runElems;
+    for (int64_t j = 0; j < k; ++j) {
+      pos[repackDim] = j;
+      runElems.push_back(vector::ExtractOp::create(rewriter, loc, src, pos));
+    }
+    Value repacked = repackRun(
+        vector::FromElementsOp::create(rewriter, loc, runTy, runElems));
+    for (int64_t j = 0; j < k; ++j) {
+      pos[repackDim] = j;
+      Value elem = vector::ExtractOp::create(rewriter, loc, repacked, j);
+      result = vector::InsertOp::create(rewriter, loc, elem, result, pos);
+    }
+  }
+  return result;
+}
+
 /// Folds a subgroup-level ConvertLayout op with compatible lane layouts.
 struct SgToLaneConvertLayout
     : public OpConversionPattern<xegpu::ConvertLayoutOp> {
@@ -1721,6 +1830,47 @@ struct SgToLaneConvertLayout
       }
     }
 
+    // Handle a pure `lane_data` repack: `lane_layout` and `order` are unchanged
+    // and exactly one dimension's `lane_data` differs, switching between
+    // round-robin (lane_data 1) and contiguous (lane_data == run length). The
+    // number of elements per lane is unchanged, but their assignment to lanes
+    // is not, so the data is moved across lanes with `xegpu.bitcast_shuffle`.
+    //
+    // The changed dimension must be one of the two innermost ones: sg-to-lane
+    // distribution is 2D, with any higher (leading) dimensions being unit, so a
+    // repack of a 3rd-innermost-or-higher dimension is not supported.
+    if (inputLayout.getEffectiveOrderAsInt() ==
+            targetLayout.getEffectiveOrderAsInt() &&
+        inputLayout.getEffectiveLaneLayoutAsInt() ==
+            targetLayout.getEffectiveLaneLayoutAsInt()) {
+      // lane_layout and lane_data always share the same rank (enforced by the
+      // LayoutAttr verifier), so the two lane_data arrays have equal size here
+      // since their lane_layouts are equal.
+      auto laneData = inputLayout.getEffectiveLaneDataAsInt();
+      auto targetLaneData = targetLayout.getEffectiveLaneDataAsInt();
+      // Find the single dimension whose lane_data changed.
+      SmallVector<int64_t> changedDims;
+      for (auto [dim, data] : llvm::enumerate(llvm::zip_equal(
+               ArrayRef<int64_t>(laneData), ArrayRef<int64_t>(targetLaneData))))
+        if (std::get<0>(data) != std::get<1>(data))
+          changedDims.push_back(dim);
+
+      int64_t rank = laneData.size();
+      // Only a single dimension may change, and it must be one of the two
+      // innermost dimensions (>= rank - 2).
+      if (changedDims.size() == 1 && changedDims[0] >= rank - 2) {
+        int64_t repackDim = changedDims[0];
+        FailureOr<Value> res =
+            repackLaneData(rewriter, op.getLoc(), adaptor.getSource(),
+                           repackDim, laneData[repackDim],
+                           targetLaneData[repackDim]);
+        if (succeeded(res)) {
+          rewriter.replaceOp(op, *res);
+          return success();
+        }
+      }
+    }
+
     return rewriter.notifyMatchFailure(
         op, "lowering incompatible convert_layout not yet supported");
   }
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..2ae6cacd0bf50 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -693,6 +693,20 @@ func.func @truncf_invalid_result_size(%a: vector<8x16xf16>) {
   return
 }
 
+// -----
+func.func @bitcast_shuffle_same_type(%a: vector<2xi16>) {
+  // expected-error at +1 {{op source and result must have different types}}
+  %1 = xegpu.bitcast_shuffle %a : vector<2xi16> -> vector<2xi16>
+  return
+}
+
+// -----
+func.func @bitcast_shuffle_size_mismatch(%a: i32) {
+  // expected-error at +1 {{op source and result must have the same total size in bits}}
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<4xi16>
+  return
+}
+
 // -----
 func.func @dpas_mx_acc_result_type_mismatch(%a : vector<8x16xf8E5M2>, %b: vector<16x16xf8E5M2>, %acc: vector<8x16xbf16>) {
   // expected-error at +1 {{Accumulator type must match result type.}}
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..3dbe570b3b1ad 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -688,6 +688,34 @@ gpu.func @truncf(%a: vector<8x16xf16>) {
   gpu.return
 }
 
+// CHECK-LABEL: gpu.func @bitcast_shuffle_scalar_to_vector_16
+gpu.func @bitcast_shuffle_scalar_to_vector_16(%a: i32) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : i32 -> vector<2xi16>
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<2xi16>
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_vector_16_to_scalar
+gpu.func @bitcast_shuffle_vector_16_to_scalar(%a: vector<2xi16>) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : vector<2xi16> -> i32
+  %1 = xegpu.bitcast_shuffle %a : vector<2xi16> -> i32
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_scalar_to_vector_8
+gpu.func @bitcast_shuffle_scalar_to_vector_8(%a: i32) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : i32 -> vector<4xi8>
+  %1 = xegpu.bitcast_shuffle %a : i32 -> vector<4xi8>
+  gpu.return
+}
+
+// CHECK-LABEL: gpu.func @bitcast_shuffle_vector_8_to_scalar
+gpu.func @bitcast_shuffle_vector_8_to_scalar(%a: vector<4xi8>) {
+  // CHECK: %{{.+}} = xegpu.bitcast_shuffle %{{.+}} : vector<4xi8> -> i32
+  %1 = xegpu.bitcast_shuffle %a : vector<4xi8> -> i32
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @dpas_mx
 gpu.func @dpas_mx(%a : vector<8x32xf8E5M2>, %b: vector<32x16xf8E5M2>, %acc: vector<8x16xbf16>, %a_scale: vector<8x1xf8E8M0FNU>, %b_scale: vector<1x16xf8E8M0FNU>) {
   // CHECK: %{{.+}} = xegpu.dpas_mx %{{.+}}, %{{.+}}, %{{.+}} scale_a = %{{.+}} scale_b = %{{.+}} : (vector<8x32xf8E5M2>, vector<32x16xf8E5M2>, vector<8x16xbf16>, vector<8x1xf8E8M0FNU>, vector<1x16xf8E8M0FNU>) -> vector<8x16xbf16>
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..584d48d3a3a5e 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -1062,6 +1062,108 @@ gpu.func @convert_layout_scalar() {
 }
 }
 
+// -----
+// A convert_layout that repacks the innermost lane_data from round-robin
+// ([1, 1]) to contiguous ([1, 4]) while keeping lane_layout unchanged lowers to
+// a per-row xegpu.bitcast_shuffle: each lane's 4 strided bf16 elements are
+// bitcast to i16, shuffled across lanes into one contiguous i64, and bitcast
+// back. The non-distributed outer dim (8 rows) is repacked independently.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<8x64xbf16> to vector<8x4xbf16>
+// CHECK:         %[[E0:.*]] = vector.extract %[[SRC]][0] : vector<4xbf16> from vector<8x4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %[[E0]] : vector<4xbf16> to vector<4xi16>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<4xi16> -> vector<1xi64>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<1xi64> to vector<4xbf16>
+// CHECK-COUNT-7: xegpu.bitcast_shuffle %{{.*}} : vector<4xi16> -> vector<1xi64>
+gpu.func @convert_layout_repack_innermost_lane_data() {
+  %src = "some_op"() : () -> vector<8x64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 4]>
+    }> : vector<8x64xbf16>
+  "some_use"(%cvt) : (vector<8x64xbf16>) -> ()
+  gpu.return
+}
+}
+
+// -----
+// The reverse repack (contiguous [1, 4] to round-robin [1, 1]) uses the
+// opposite bitcast_shuffle: each lane's contiguous i64 is shuffled back into 4
+// strided i16 elements.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data_reverse
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<8x64xbf16> to vector<8x4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %{{.*}} : vector<4xbf16> to vector<1xi64>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<1xi64> -> vector<4xi16>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<4xi16> to vector<4xbf16>
+gpu.func @convert_layout_repack_innermost_lane_data_reverse() {
+  %src = "some_op"() : () -> vector<8x64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 4]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
+    }> : vector<8x64xbf16>
+  "some_use"(%cvt) : (vector<8x64xbf16>) -> ()
+  gpu.return
+}
+}
+
+// -----
+// A 1D convert_layout repack is a single bitcast_shuffle with no outer loop.
+gpu.module @xevm_module {
+// CHECK-LABEL: gpu.func @convert_layout_repack_innermost_lane_data_1d
+// CHECK-NOT:     xegpu.convert_layout
+// CHECK:         %[[SRC:.*]] = builtin.unrealized_conversion_cast %{{.*}} : vector<64xbf16> to vector<4xbf16>
+// CHECK:         %[[B0:.*]] = vector.bitcast %[[SRC]] : vector<4xbf16> to vector<4xi16>
+// CHECK:         %[[S0:.*]] = xegpu.bitcast_shuffle %[[B0]] : vector<4xi16> -> vector<1xi64>
+// CHECK:         %{{.*}} = vector.bitcast %[[S0]] : vector<1xi64> to vector<4xbf16>
+gpu.func @convert_layout_repack_innermost_lane_data_1d() {
+  %src = "some_op"() : () -> vector<64xbf16>
+  %cvt = xegpu.convert_layout %src
+    <{
+      input_layout = #xegpu.layout<lane_layout = [16], lan...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210837

From llvmlistbot at llvm.org  Mon Jul 20 18:50:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 18:50:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Preserve worker rows when
 combining reductions (PR #210804)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210804@github.com>
Message-ID: <6a5ed062.92d2873c.aa1ff.fe9e@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210804

>From 4c316621a473f21607795bafbd1ffa51a9dabe93 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 13:12:09 -0700
Subject: [PATCH 01/17] [OpenACC] Preserve worker rows when combining
 reductions

Keep ThreadY active for worker-private sources so every worker row contributes, and reject mixed scopes that require incompatible predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 53 +++++++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 43 +++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 95 +++++++++++++++++++
 3 files changed, 181 insertions(+), 10 deletions(-)
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 24aeed6e23bda..6be79b0dd40e4 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,7 +1246,30 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    block->walk([&](Operation *op) {
+    std::optional<bool> combineThreadYActive;
+    auto applyCombineParDims =
+        [&](Operation *combineOp, Value src,
+            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+          bool isWorkerPrivate =
+              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
+            if (parDim.isThreadY()) {
+              if (combineThreadYActive &&
+                  *combineThreadYActive != isWorkerPrivate) {
+                combineOp->emitError()
+                    << "mixed worker-private and non-worker-private reduction "
+                       "combines require incompatible ThreadY predication";
+                hasFailed = true;
+                return failure();
+              }
+              combineThreadYActive = isWorkerPrivate;
+            } else {
+              mlir::acc::removeParDim(ancestorParDims, parDim);
+            }
+          }
+          return success();
+        };
+    block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1277,17 +1300,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(reductionCombineOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                reductionCombineOp, reductionCombineOp.getSrcMemref(),
+                getReductionCombineParDims(reductionCombineOp))))
+          return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(combineRegionOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                combineRegionOp, combineRegionOp.getSrcVar(),
+                getReductionCombineParDims(combineRegionOp))))
+          return WalkResult::interrupt();
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1301,6 +1324,14 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
+    if (combineThreadYActive) {
+      mlir::acc::GPUParallelDimAttr threadY =
+          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+      if (*combineThreadYActive)
+        mlir::acc::insertParDim(ancestorParDims, threadY);
+      else
+        mlir::acc::removeParDim(ancestorParDims, threadY);
+    }
   }
 
   // Obtain launch dimensions
@@ -1747,7 +1778,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasBlock && hasThreadY)
+  if (hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;
@@ -1871,6 +1902,8 @@ void ACCCGToGPULowering::processPredicateRegion(
             SmallVector<mlir::acc::GPUParallelDimAttr>>
       parDimsPair = computeActiveAndInactiveParDims(
           interOp, &interOp.getRegion().front());
+  if (hasFailed)
+    return;
 
   // If ThreadY reduction exists, subgroup alignment is applied
   // (blockDim.x = subgroupSize), so ThreadX lanes exist even without explicit
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
new file mode 100644
index 0000000000000..a0d381740c034
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -0,0 +1,43 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" \
+// RUN:   -verify-diagnostics
+
+func.func @mixed_scope_worker_reduction_combine(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
new file mode 100644
index 0000000000000..b2fd246d33d1b
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -0,0 +1,95 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A worker-private reduction has one shared slot per ThreadY row. The combine
+// must keep ThreadY active and predicate only ThreadX.
+
+// CHECK-LABEL: func.func @worker_reduction_combine
+// CHECK: gpu.launch {{.*}} threads([[TID_X:%[^,]+]], [[TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[TID_Y]]
+// CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: acc.atomic.update
+
+func.func @worker_reduction_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @worker_reduction_combine_region
+// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
+// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[REGION_IS_X_ZERO]]
+// CHECK: arith.addi
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From d763e8065946687750b2c78fcf5393167b7de962 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 15:08:39 -0700
Subject: [PATCH 02/17] [OpenACC] Report mixed reduction scopes as NYI

Use the standard OpenACC unsupported-feature diagnostic for incompatible ThreadY predication.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6be79b0dd40e4..378fa17b24850 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1256,9 +1256,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             if (parDim.isThreadY()) {
               if (combineThreadYActive &&
                   *combineThreadYActive != isWorkerPrivate) {
-                combineOp->emitError()
-                    << "mixed worker-private and non-worker-private reduction "
-                       "combines require incompatible ThreadY predication";
+                (void)accSupport.emitNYI(
+                    combineOp->getLoc(),
+                    "mixed worker-private and non-worker-private reduction "
+                    "combines require incompatible ThreadY predication");
                 hasFailed = true;
                 return failure();
               }

>From 1fe76c2f1ea14bd036598286c2d9c0ea8d2291ef Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 16:50:59 -0700
Subject: [PATCH 03/17] [OpenACC] Isolate ThreadY predicate requirements

Propagate nested worker requirements without broadening unrelated side effects, and reject incompatible operations sharing one predicate region.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 126 +++++++++++++++---
 ...-worker-reduction-combine-mixed-scope.mlir |  40 +++++-
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  46 +++++++
 3 files changed, 189 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 378fa17b24850..34131dc3753a3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,27 +1246,111 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    std::optional<bool> combineThreadYActive;
+    struct ThreadYRequirement {
+      bool active = false;
+      bool inactive = false;
+    };
+    auto analyzeThreadYRequirements =
+        [&](auto &&self,
+            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
+      ThreadYRequirement requirement;
+      auto requireThreadY = [&](Operation *op, bool active) {
+        if ((active && requirement.inactive) ||
+            (!active && requirement.active)) {
+          (void)accSupport.emitNYI(
+              op->getLoc(),
+              "operations in the same predicate region require incompatible "
+              "ThreadY predication");
+          hasFailed = true;
+          return failure();
+        }
+        requirement.active |= active;
+        requirement.inactive |= !active;
+        return success();
+      };
+      auto applyCombineRequirement =
+          [&](Operation *combineOp, Value src,
+              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+            if (llvm::none_of(combineParDims,
+                              [](auto parDim) { return parDim.isThreadY(); }))
+              return success();
+            bool isWorkerPrivate =
+                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+            return requireThreadY(combineOp, isWorkerPrivate);
+          };
+
+      for (Operation &nestedOp : predicateBlock) {
+        if (acc::PredicateRegionOp nestedPredicate =
+                dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
+          FailureOr<ThreadYRequirement> nestedRequirement =
+              self(self, nestedPredicate.getRegion().front());
+          if (failed(nestedRequirement))
+            return failure();
+          // An active descendant must not be excluded by this region's
+          // predicate. Inactive descendants apply their own predicate.
+          if (nestedRequirement->active &&
+              failed(requireThreadY(&nestedOp, /*active=*/true)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineOp combineOp =
+                dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineOp, combineOp.getSrcMemref(),
+                  getReductionCombineParDims(combineOp))))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineRegionOp combineRegionOp =
+                dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineRegionOp, combineRegionOp.getSrcVar(),
+                  getReductionCombineParDims(combineRegionOp))))
+            return failure();
+          continue;
+        }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          bool threadYActive = false;
+          if (acc::PrivateLocalOp privateLocal =
+                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+            if (acc::GPUParallelDimsAttr parDims =
+                    getPrivatizeOp(privateLocal, computeRegion)
+                        .getParDimsAttr()) {
+              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
+                return parDim.isThreadY();
+              });
+            }
+          }
+          if (failed(requireThreadY(storeOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
+                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
+          bool threadYActive =
+              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
+                           [](auto parDim) { return parDim.isThreadY(); });
+          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (!isMemoryEffectFree(&nestedOp) &&
+            failed(requireThreadY(&nestedOp, /*active=*/false)))
+          return failure();
+      }
+      return requirement;
+    };
+
+    FailureOr<ThreadYRequirement> threadYRequirement =
+        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
+    if (failed(threadYRequirement))
+      return {};
+
     auto applyCombineParDims =
-        [&](Operation *combineOp, Value src,
-            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          bool isWorkerPrivate =
-              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
           for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (parDim.isThreadY()) {
-              if (combineThreadYActive &&
-                  *combineThreadYActive != isWorkerPrivate) {
-                (void)accSupport.emitNYI(
-                    combineOp->getLoc(),
-                    "mixed worker-private and non-worker-private reduction "
-                    "combines require incompatible ThreadY predication");
-                hasFailed = true;
-                return failure();
-              }
-              combineThreadYActive = isWorkerPrivate;
-            } else {
+            if (!parDim.isThreadY())
               mlir::acc::removeParDim(ancestorParDims, parDim);
-            }
           }
           return success();
         };
@@ -1302,14 +1386,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
         if (failed(applyCombineParDims(
-                reductionCombineOp, reductionCombineOp.getSrcMemref(),
                 getReductionCombineParDims(reductionCombineOp))))
           return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
         if (failed(applyCombineParDims(
-                combineRegionOp, combineRegionOp.getSrcVar(),
                 getReductionCombineParDims(combineRegionOp))))
           return WalkResult::interrupt();
       }
@@ -1325,10 +1407,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (combineThreadYActive) {
+    if (threadYRequirement->active || threadYRequirement->inactive) {
       mlir::acc::GPUParallelDimAttr threadY =
           mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (*combineThreadYActive)
+      if (threadYRequirement->active)
         mlir::acc::insertParDim(ancestorParDims, threadY);
       else
         mlir::acc::removeParDim(ancestorParDims, threadY);
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a0d381740c034..e80b5afa26d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -30,7 +30,7 @@ func.func @mixed_scope_worker_reduction_combine(
         acc.predicate_region {
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -41,3 +41,41 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_single_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c7_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          memref.store %c7_i32, %result_arg[] : memref<i32>
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index b2fd246d33d1b..afd6108d6e670 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -93,3 +93,49 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+// Nested predicate regions choose their ThreadY predicates independently. The
+// outer region must keep ThreadY active so it does not exclude worker rows
+// before the nested worker-private combine is reached.
+func.func @nested_worker_reduction_combines(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          acc.predicate_region {
+            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 477f10e0472195d6bb55598bf364a1655682ab6f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:04:51 -0700
Subject: [PATCH 04/17] [OpenACC] Propagate nested ThreadY requirements

Analyze control-flow regions recursively and recognize aliased worker-private stores so their worker rows remain active.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 32 ++++---
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 86 +++++++++++++++++++
 2 files changed, 107 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 34131dc3753a3..98c697f52668e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1310,17 +1310,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = false;
-          if (acc::PrivateLocalOp privateLocal =
-                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-            if (acc::GPUParallelDimsAttr parDims =
-                    getPrivatizeOp(privateLocal, computeRegion)
-                        .getParDimsAttr()) {
-              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
-                return parDim.isThreadY();
-              });
-            }
-          }
+          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
+                               PrivateMemScope::Worker;
           if (failed(requireThreadY(storeOp, threadYActive)))
             return failure();
           continue;
@@ -1334,6 +1325,25 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             return failure();
           continue;
         }
+        if (nestedOp.getNumRegions() != 0) {
+          for (Region &region : nestedOp.getRegions()) {
+            for (Block &nestedBlock : region) {
+              FailureOr<ThreadYRequirement> nestedRequirement =
+                  self(self, nestedBlock);
+              if (failed(nestedRequirement))
+                return failure();
+              // Other region-bearing operations do not introduce independent
+              // ACC predication, so both requirements apply to this region.
+              if (nestedRequirement->active &&
+                  failed(requireThreadY(&nestedOp, /*active=*/true)))
+                return failure();
+              if (nestedRequirement->inactive &&
+                  failed(requireThreadY(&nestedOp, /*active=*/false)))
+                return failure();
+            }
+          }
+          continue;
+        }
         if (!isMemoryEffectFree(&nestedOp) &&
             failed(requireThreadY(&nestedOp, /*active=*/false)))
           return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index afd6108d6e670..8a2dd6bd470f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -97,6 +97,12 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.
+// CHECK-LABEL: func.func @nested_worker_reduction_combines
+// CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
+// CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[NESTED_TX_ZERO]]
 func.func @nested_worker_reduction_combines(
     %other: memref<i32>, %result: memref<i32>) {
   %c1 = arith.constant 1 : index
@@ -139,3 +145,83 @@ func.func @nested_worker_reduction_combines(
   }
   return
 }
+
+// CHECK-LABEL: func.func @worker_combine_in_scf_if
+// CHECK: gpu.launch {{.*}} threads([[IF_TX:%[^,]+]], [[IF_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[IF_TY]]
+// CHECK: %[[IF_TX_ZERO:.*]] = arith.cmpi eq, [[IF_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IF_TX_ZERO]]
+func.func @worker_combine_in_scf_if(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          scf.if %true {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @aliased_worker_store
+// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
+// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[ALIAS_TX_ZERO]]
+// CHECK: memref.store
+// CHECK: acc.atomic.update
+func.func @aliased_worker_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2e234b5cb4dac8aa7cf1ab07b63984b7cb9f329f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:16:55 -0700
Subject: [PATCH 05/17] [OpenACC] Account for predicate operation effects

Preserve all privatized ThreadY dimensions through aliases and reject intrinsically effecting region operations that cannot share worker predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 33 +++++++++++-----
 ...-worker-reduction-combine-mixed-scope.mlir | 38 +++++++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 36 ++++++++++++++++++
 3 files changed, 97 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98c697f52668e..20e8977c907f3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1268,15 +1268,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         requirement.inactive |= !active;
         return success();
       };
+      auto hasThreadYPrivateStorage = [&](Value memref) {
+        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
+        if (!privatize)
+          return false;
+        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
+                 return parDim.isThreadY();
+               });
+      };
       auto applyCombineRequirement =
           [&](Operation *combineOp, Value src,
               ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
             if (llvm::none_of(combineParDims,
                               [](auto parDim) { return parDim.isThreadY(); }))
               return success();
-            bool isWorkerPrivate =
-                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
-            return requireThreadY(combineOp, isWorkerPrivate);
+            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
           };
 
       for (Operation &nestedOp : predicateBlock) {
@@ -1310,9 +1317,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
-                               PrivateMemScope::Worker;
-          if (failed(requireThreadY(storeOp, threadYActive)))
+          if (failed(requireThreadY(
+                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
             return failure();
           continue;
         }
@@ -1326,6 +1332,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
+          bool hasOwnEffects = true;
+          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
+            hasOwnEffects = !effectOp.hasNoEffect();
+          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
+            hasOwnEffects = false;
+          }
+          if (hasOwnEffects &&
+              failed(requireThreadY(&nestedOp, /*active=*/false)))
+            return failure();
           for (Region &region : nestedOp.getRegions()) {
             for (Block &nestedBlock : region) {
               FailureOr<ThreadYRequirement> nestedRequirement =
@@ -1368,10 +1383,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (auto privateLocalOp =
-                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-          acc::PrivatizeOp privatizeOp =
-              getPrivatizeOp(privateLocalOp, computeRegion);
+        if (acc::PrivatizeOp privatizeOp =
+                getPrivatizeForMemref(storeOp.getMemref())) {
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index e80b5afa26d20..19e1b3106d966 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -79,3 +79,41 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c1_i32 = arith.constant 1 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.atomic.update %result_arg : memref<i32> {
+          ^bb0(%current: i32):
+            %next = arith.addi %current, %c1_i32 : i32
+            acc.yield %next : i32
+          }
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 8a2dd6bd470f8..ba8f16d3966df 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -225,3 +225,39 @@ func.func @aliased_worker_store(%result: memref<i32>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @aliased_thread_store
+// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
+// CHECK: memref.store
+func.func @aliased_thread_store() {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private)
+        : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From ae0d329822ede9a6d80731ebd45abd7e520ef5ae Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:26:21 -0700
Subject: [PATCH 06/17] [OpenACC] Require proven private aliases

Follow only alias-preserving view operations and treat every non-private combine as ThreadY-inactive to avoid broadening ambiguous global effects.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 45 +++++++++----------
 ...-worker-reduction-combine-mixed-scope.mlir |  6 ++-
 2 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 20e8977c907f3..9e9114e9aa174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1277,14 +1277,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                  return parDim.isThreadY();
                });
       };
-      auto applyCombineRequirement =
-          [&](Operation *combineOp, Value src,
-              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-            if (llvm::none_of(combineParDims,
-                              [](auto parDim) { return parDim.isThreadY(); }))
-              return success();
-            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
-          };
+      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
+        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
+      };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
@@ -1302,17 +1297,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineOp, combineOp.getSrcMemref(),
-                  getReductionCombineParDims(combineOp))))
+          if (failed(
+                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
             return failure();
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineRegionOp, combineRegionOp.getSrcVar(),
-                  getReductionCombineParDims(combineRegionOp))))
+          if (failed(applyCombineRequirement(combineRegionOp,
+                                             combineRegionOp.getSrcVar())))
             return failure();
           continue;
         }
@@ -1893,18 +1886,22 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  llvm::SmallVector<Value, 8> worklist{memref};
-  llvm::SmallPtrSet<Value, 8> seen;
-  while (!worklist.empty()) {
-    Value v = worklist.pop_back_val();
-    if (!seen.insert(v).second)
-      continue;
-    Operation *def = v.getDefiningOp();
-    if (!def)
-      continue;
+  Value current = memref;
+  while (Operation *def = current.getDefiningOp()) {
     if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
       return privateLocal;
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+      current = viewLike.getViewSource();
+      continue;
+    }
+    if (acc::PartialEntityAccessOpInterface partialAccess =
+            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
+      current = partialAccess.getBaseEntity();
+      continue;
+    }
+    // Do not follow arbitrary operands: multi-source operations such as
+    // arith.select do not prove that their result aliases private storage.
+    return nullptr;
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 19e1b3106d966..1275e4c77890a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -32,7 +32,7 @@ func.func @mixed_scope_worker_reduction_combine(
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+              {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
@@ -59,15 +59,17 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
+      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %result_arg[] : memref<i32>
+          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}

>From 3ed1de6f3b95aee6516db811bbd62dc4c54f9335 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:38:25 -0700
Subject: [PATCH 07/17] [OpenACC] Restrict ThreadY predicate broadening

Preserve legacy predicates unless a proven worker combine requires ThreadY, and reject broadening around writes, unknown effects, or incompatible combines.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 223 +++++++++---------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  78 +-----
 2 files changed, 107 insertions(+), 194 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 9e9114e9aa174..50635dde561db 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,138 +1246,122 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    struct ThreadYRequirement {
-      bool active = false;
-      bool inactive = false;
+    struct ThreadYBroadeningInfo {
+      bool hasActiveWorkerCombine = false;
+      bool hasExplicitInactiveCombine = false;
+      bool hasBroadeningConflict = false;
+      Operation *diagnosticOp = nullptr;
     };
-    auto analyzeThreadYRequirements =
-        [&](auto &&self,
-            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
-      ThreadYRequirement requirement;
-      auto requireThreadY = [&](Operation *op, bool active) {
-        if ((active && requirement.inactive) ||
-            (!active && requirement.active)) {
-          (void)accSupport.emitNYI(
-              op->getLoc(),
-              "operations in the same predicate region require incompatible "
-              "ThreadY predication");
-          hasFailed = true;
-          return failure();
+    auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
+                                  const ThreadYBroadeningInfo &src) {
+      dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
+      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
+      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      if (!dst.diagnosticOp)
+        dst.diagnosticOp = src.diagnosticOp;
+    };
+    auto isProvenWorkerPrivate = [&](Value memref) {
+      Value current = memref;
+      while (Operation *def = current.getDefiningOp()) {
+        if (acc::PrivateLocalOp privateLocal =
+                dyn_cast<acc::PrivateLocalOp>(def)) {
+          return getPrivateMemScope(getPrivatizeOp(
+                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+        }
+        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+          current = viewLike.getViewSource();
+          continue;
+        }
+        break;
+      }
+      return false;
+    };
+    auto hasUnsafeOwnEffects = [](Operation *op) {
+      if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
+        SmallVector<MemoryEffects::EffectInstance> effects;
+        effectOp.getEffects(effects);
+        return llvm::any_of(effects, [](const auto &effect) {
+          return !isa<MemoryEffects::Read>(effect.getEffect());
+        });
+      }
+      return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
+    };
+    auto analyzeThreadYBroadening =
+        [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
+      ThreadYBroadeningInfo info;
+      auto classifyCombine = [&](Operation *combineOp, Value src,
+                                 ArrayRef<GPUParallelDimAttr> parDims) {
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        if (hasThreadY && isProvenWorkerPrivate(src)) {
+          info.hasActiveWorkerCombine = true;
+        } else {
+          info.hasExplicitInactiveCombine = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = combineOp;
         }
-        requirement.active |= active;
-        requirement.inactive |= !active;
-        return success();
-      };
-      auto hasThreadYPrivateStorage = [&](Value memref) {
-        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
-        if (!privatize)
-          return false;
-        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
-                 return parDim.isThreadY();
-               });
-      };
-      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
-        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
       };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
                 dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
-          FailureOr<ThreadYRequirement> nestedRequirement =
+          ThreadYBroadeningInfo nestedInfo =
               self(self, nestedPredicate.getRegion().front());
-          if (failed(nestedRequirement))
-            return failure();
-          // An active descendant must not be excluded by this region's
-          // predicate. Inactive descendants apply their own predicate.
-          if (nestedRequirement->active &&
-              failed(requireThreadY(&nestedOp, /*active=*/true)))
-            return failure();
+          // A worker-active descendant must reach its own predicate. Other
+          // requirements are enforced within the nested predicate region.
+          info.hasActiveWorkerCombine |= nestedInfo.hasActiveWorkerCombine;
           continue;
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(
-                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
-            return failure();
+          classifyCombine(combineOp, combineOp.getSrcMemref(),
+                          getReductionCombineParDims(combineOp));
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(combineRegionOp,
-                                             combineRegionOp.getSrcVar())))
-            return failure();
-          continue;
-        }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (failed(requireThreadY(
-                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
-            return failure();
-          continue;
-        }
-        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
-                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
-          bool threadYActive =
-              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
-                           [](auto parDim) { return parDim.isThreadY(); });
-          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
-            return failure();
+          classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
+                          getReductionCombineParDims(combineRegionOp));
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          bool hasOwnEffects = true;
-          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
-            hasOwnEffects = !effectOp.hasNoEffect();
-          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
-            hasOwnEffects = false;
+          if (hasUnsafeOwnEffects(&nestedOp)) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = &nestedOp;
           }
-          if (hasOwnEffects &&
-              failed(requireThreadY(&nestedOp, /*active=*/false)))
-            return failure();
           for (Region &region : nestedOp.getRegions()) {
-            for (Block &nestedBlock : region) {
-              FailureOr<ThreadYRequirement> nestedRequirement =
-                  self(self, nestedBlock);
-              if (failed(nestedRequirement))
-                return failure();
-              // Other region-bearing operations do not introduce independent
-              // ACC predication, so both requirements apply to this region.
-              if (nestedRequirement->active &&
-                  failed(requireThreadY(&nestedOp, /*active=*/true)))
-                return failure();
-              if (nestedRequirement->inactive &&
-                  failed(requireThreadY(&nestedOp, /*active=*/false)))
-                return failure();
-            }
+            for (Block &nestedBlock : region)
+              mergeBroadeningInfo(info, self(self, nestedBlock));
           }
           continue;
         }
-        if (!isMemoryEffectFree(&nestedOp) &&
-            failed(requireThreadY(&nestedOp, /*active=*/false)))
-          return failure();
+        if (hasUnsafeOwnEffects(&nestedOp)) {
+          info.hasBroadeningConflict = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = &nestedOp;
+        }
       }
-      return requirement;
+      return info;
     };
 
-    FailureOr<ThreadYRequirement> threadYRequirement =
-        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
-    if (failed(threadYRequirement))
-      return {};
+    ThreadYBroadeningInfo threadYInfo =
+        analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
     auto applyCombineParDims =
         [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (!parDim.isThreadY())
-              mlir::acc::removeParDim(ancestorParDims, parDim);
-          }
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
+            mlir::acc::removeParDim(ancestorParDims, parDim);
           return success();
         };
     block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (acc::PrivatizeOp privatizeOp =
-                getPrivatizeForMemref(storeOp.getMemref())) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
@@ -1423,13 +1407,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (threadYRequirement->active || threadYRequirement->inactive) {
-      mlir::acc::GPUParallelDimAttr threadY =
-          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (threadYRequirement->active)
-        mlir::acc::insertParDim(ancestorParDims, threadY);
-      else
-        mlir::acc::removeParDim(ancestorParDims, threadY);
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
+    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+      if (threadYInfo.hasExplicitInactiveCombine ||
+          threadYInfo.hasBroadeningConflict) {
+        Operation *diagnosticOp =
+            threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
+        (void)accSupport.emitNYI(
+            diagnosticOp->getLoc(),
+            "operations in the same predicate region require incompatible "
+            "ThreadY predication");
+        hasFailed = true;
+        return {};
+      }
+      mlir::acc::insertParDim(ancestorParDims, threadY);
     }
   }
 
@@ -1886,22 +1879,18 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  Value current = memref;
-  while (Operation *def = current.getDefiningOp()) {
-    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
-      return privateLocal;
-    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-      current = viewLike.getViewSource();
+  llvm::SmallVector<Value, 8> worklist{memref};
+  llvm::SmallPtrSet<Value, 8> seen;
+  while (!worklist.empty()) {
+    Value v = worklist.pop_back_val();
+    if (!seen.insert(v).second)
       continue;
-    }
-    if (acc::PartialEntityAccessOpInterface partialAccess =
-            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
-      current = partialAccess.getBaseEntity();
+    Operation *def = v.getDefiningOp();
+    if (!def)
       continue;
-    }
-    // Do not follow arbitrary operands: multi-source operations such as
-    // arith.select do not prove that their result aliases private storage.
-    return nullptr;
+    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
+      return privateLocal;
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index ba8f16d3966df..0de030eda1e59 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -184,80 +185,3 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
-
-// CHECK-LABEL: func.func @aliased_worker_store
-// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
-// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[ALIAS_TX_ZERO]]
-// CHECK: memref.store
-// CHECK: acc.atomic.update
-func.func @aliased_worker_store(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
-// CHECK-LABEL: func.func @aliased_thread_store
-// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
-// CHECK: memref.store
-func.func @aliased_thread_store() {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private)
-        : (!acc.private_type<memref<i32>>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}

>From 30926fc3c3e3a6d6b9fe8e3fcb8442319849edf1 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:35 -0700
Subject: [PATCH 08/17] [OpenACC][test] Match broadening conflict locations

Expect diagnostics on the side effect that makes worker broadening unsafe.
---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 1275e4c77890a..7f22fd20aa89e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -69,8 +69,8 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          memref.store %c7_i32, %selected[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -103,12 +103,12 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
         acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.atomic.update %result_arg : memref<i32> {
           ^bb0(%current: i32):
             %next = arith.addi %current, %c1_i32 : i32
             acc.yield %next : i32
           }
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 55293fcdfa6ce6f49e61e43293e80df04bfec368 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:51:30 -0700
Subject: [PATCH 09/17] [OpenACC] Require safe predicate speculation

Treat reads and non-speculatable operations as broadening conflicts so enabling worker rows cannot introduce invalid memory accesses.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp  | 13 +++++--------
 ...to-gpu-worker-reduction-combine-mixed-scope.mlir |  6 ++----
 .../acc-cg-to-gpu-worker-reduction-combine.mlir     |  1 -
 3 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 50635dde561db..250c2be09df85 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1276,13 +1276,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return false;
     };
-    auto hasUnsafeOwnEffects = [](Operation *op) {
+    auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
-        SmallVector<MemoryEffects::EffectInstance> effects;
-        effectOp.getEffects(effects);
-        return llvm::any_of(effects, [](const auto &effect) {
-          return !isa<MemoryEffects::Read>(effect.getEffect());
-        });
+        return !effectOp.hasNoEffect();
       }
       return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
     };
@@ -1325,7 +1321,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          if (hasUnsafeOwnEffects(&nestedOp)) {
+          if (hasOwnEffects(&nestedOp)) {
             info.hasBroadeningConflict = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
@@ -1336,7 +1332,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
           continue;
         }
-        if (hasUnsafeOwnEffects(&nestedOp)) {
+        if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
+            !isPure(&nestedOp)) {
           info.hasBroadeningConflict = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 7f22fd20aa89e..290786ab7c1ce 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,7 +42,7 @@ func.func @mixed_scope_worker_reduction_combine(
   return
 }
 
-func.func @worker_combine_with_single_store(%result: memref<i32>) {
+func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -59,18 +59,16 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
-      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          memref.store %c7_i32, %selected[] : memref<i32>
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 0de030eda1e59..fb1a0256d2a38 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 300b3dc6a9334c5bd8b47daf1c1371bb6f5db806 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:59:06 -0700
Subject: [PATCH 10/17] [OpenACC] Reject worker combine regions

Keep generic combine regions single-row because their non-atomic load/store lowering is unsafe across worker rows.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 14 ++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 37 ++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 48 -------------------
 3 files changed, 48 insertions(+), 51 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 250c2be09df85..3697ae7405c9f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1286,11 +1286,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
       auto classifyCombine = [&](Operation *combineOp, Value src,
-                                 ArrayRef<GPUParallelDimAttr> parDims) {
+                                 ArrayRef<GPUParallelDimAttr> parDims,
+                                 bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
         if (hasThreadY && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
+          if (!supportsWorkerBroadening) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = combineOp;
+          }
         } else {
           info.hasExplicitInactiveCombine = true;
           if (!info.diagnosticOp)
@@ -1311,13 +1317,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
           classifyCombine(combineOp, combineOp.getSrcMemref(),
-                          getReductionCombineParDims(combineOp));
+                          getReductionCombineParDims(combineOp),
+                          /*supportsWorkerBroadening=*/true);
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
           classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
-                          getReductionCombineParDims(combineRegionOp));
+                          getReductionCombineParDims(combineRegionOp),
+                          /*supportsWorkerBroadening=*/false);
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 290786ab7c1ce..fffbfcb461c56 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -117,3 +117,40 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fb1a0256d2a38..686e8fd233980 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,54 +46,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// CHECK-LABEL: func.func @worker_reduction_combine_region
-// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
-// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[REGION_IS_X_ZERO]]
-// CHECK: arith.addi
-
-func.func @worker_reduction_combine_region(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
-          memref.store %c0_i32, %local[] : memref<i32>
-          scf.reduce
-        } {acc.par_dims = #acc<par_dims[thread_y]>}
-        acc.predicate_region {
-          acc.reduction_combine_region %local into %result_arg : memref<i32> {
-            %lhs = memref.load %result_arg[] : memref<i32>
-            %rhs = memref.load %local[] : memref<i32>
-            %sum = arith.addi %lhs, %rhs : i32
-            memref.store %sum, %result_arg[] : memref<i32>
-            acc.yield
-          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.

>From cbc18adba26e38ff6bebd01bf252112da4412887 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:03:05 -0700
Subject: [PATCH 11/17] [OpenACC] Require atomic worker combines

Broaden ThreadY only when block dimensions select the atomic combine lowering; reject ThreadY-only shared updates.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  4 ++-
 ...-worker-reduction-combine-mixed-scope.mlir | 32 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 3697ae7405c9f..76f942a2730f9 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1290,7 +1290,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                                  bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
-        if (hasThreadY && isProvenWorkerPrivate(src)) {
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
             info.hasBroadeningConflict = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index fffbfcb461c56..5758702a3b92e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -154,3 +154,35 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2f6b6b2e74b62114ad20366b768685e6af8bc000 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:06:26 -0700
Subject: [PATCH 12/17] [OpenACC][test] Check non-atomic worker predication

Verify ThreadY-only combines retain single-row predication while unsafe combine regions remain rejected.
---
 ...-worker-reduction-combine-mixed-scope.mlir | 31 ---------------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 39 +++++++++++++++++++
 2 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 5758702a3b92e..3d15b4ca4a59f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -155,34 +155,3 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   return
 }
 
-func.func @thread_y_only_worker_combine(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        acc.predicate_region {
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 686e8fd233980..901b23b25316f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -136,3 +136,42 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
+
+// Without a block dimension the combine lowers to a non-atomic update, so
+// ThreadY must retain the legacy single-row predicate.
+// CHECK-LABEL: func.func @thread_y_only_worker_combine
+// CHECK: gpu.launch {{.*}} threads([[ONLY_TX:%[^,]+]], [[ONLY_TY:%[^,]+]],
+// CHECK: %[[ONLY_TY_ZERO:.*]] = arith.cmpi eq, [[ONLY_TY]],
+// CHECK: %[[ONLY_TX_ZERO:.*]] = arith.cmpi eq, [[ONLY_TX]],
+// CHECK: %[[ONLY_ROW_ZERO:.*]] = arith.andi %[[ONLY_TX_ZERO]], %[[ONLY_TY_ZERO]]
+// CHECK: scf.if %[[ONLY_ROW_ZERO]]
+// CHECK: memref.store
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 88026c11dfb0376be0ad438de4a530d99d821119 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:10:52 -0700
Subject: [PATCH 13/17] [OpenACC][test] Remove trailing blank line

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 3d15b4ca4a59f..dc634cff25b07 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -41,7 +41,6 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
-
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 984e8f2ce4366cb23a6104b6aa965783f2380a49 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:11:22 -0700
Subject: [PATCH 14/17] [OpenACC][test] Normalize test file ending

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dc634cff25b07..dba8057d48e95 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -78,7 +78,6 @@ func.func @worker_combine_with_read(%result: memref<i32>) {
   }
   return
 }
-
 func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 8f6c707f4e58523929b8482505946acdd9dc9991 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:12:19 -0700
Subject: [PATCH 15/17] [OpenACC][test] Finish file ending cleanup

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir    | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dba8057d48e95..a641fe74867e4 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -151,5 +151,4 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
     } {origin = "acc.parallel"}
   }
   return
-}
-
+}
\ No newline at end of file

>From f629ee527f4b56ac33d85867d488a793af7cfc09 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:43:52 -0700
Subject: [PATCH 16/17] [OpenACC] Localize worker combine classification

Keep private-memory scope behavior unchanged and remove redundant failure plumbing from combine dimension handling.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 37 ++++++++++---------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 76f942a2730f9..98258dc622e3c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1265,8 +1265,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       while (Operation *def = current.getDefiningOp()) {
         if (acc::PrivateLocalOp privateLocal =
                 dyn_cast<acc::PrivateLocalOp>(def)) {
-          return getPrivateMemScope(getPrivatizeOp(
-                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+          acc::PrivatizeOp privatize =
+              getPrivatizeOp(privateLocal, computeRegion);
+          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+          if (!parDims)
+            return false;
+          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadX();
+          });
+          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadY();
+          });
+          return hasThreadY && !hasThreadX;
         }
         if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
           current = viewLike.getViewSource();
@@ -1355,13 +1365,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     ThreadYBroadeningInfo threadYInfo =
         analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
-    auto applyCombineParDims =
-        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
-            mlir::acc::removeParDim(ancestorParDims, parDim);
-          return success();
-        };
-    block->walk([&](Operation *op) -> WalkResult {
+    block->walk([&](Operation *op) {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1392,15 +1396,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(reductionCombineOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(combineRegionOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1412,7 +1416,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           mlir::acc::insertParDim(ancestorParDims, parDim);
         }
       }
-      return WalkResult::advance();
     });
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
@@ -1877,7 +1880,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasThreadY)
+  if (hasBlock && hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;

>From edc8abd53cd69917a03f21c72c975e4a26a4ed24 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:50:05 -0700
Subject: [PATCH 17/17] [OpenACC] Merge ThreadY conflict state

Use one flag for all operations that require the legacy inactive ThreadY predicate.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98258dc622e3c..6a5aebc4db323 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,15 +1248,13 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
-      bool hasExplicitInactiveCombine = false;
-      bool hasBroadeningConflict = false;
+      bool requiresInactiveThreadY = false;
       Operation *diagnosticOp = nullptr;
     };
     auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
                                   const ThreadYBroadeningInfo &src) {
       dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
-      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
-      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      dst.requiresInactiveThreadY |= src.requiresInactiveThreadY;
       if (!dst.diagnosticOp)
         dst.diagnosticOp = src.diagnosticOp;
     };
@@ -1305,12 +1303,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = combineOp;
           }
         } else {
-          info.hasExplicitInactiveCombine = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1342,7 +1340,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
           }
@@ -1354,7 +1352,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
             !isPure(&nestedOp)) {
-          info.hasBroadeningConflict = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
         }
@@ -1421,8 +1419,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
     bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
-      if (threadYInfo.hasExplicitInactiveCombine ||
-          threadYInfo.hasBroadeningConflict) {
+      if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
         (void)accSupport.emitNYI(


From llvmlistbot at llvm.org  Mon Jul 20 18:54:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 18:54:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Preserve worker rows when
 combining reductions (PR #210804)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210804@github.com>
Message-ID: <6a5ed155.a1accbb4.5eca2.46c8@mx.google.com>


https://github.com/khaki3 edited https://github.com/llvm/llvm-project/pull/210804

From llvmlistbot at llvm.org  Mon Jul 20 20:34:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 20:34:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support multi-rank array
	reductions (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a5ee8b1.510f576d.2f694c.d0e0@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210853

>From 2d005d7cf11a32e49f59fafddd6ccc1d39991825 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:05:29 -0700
Subject: [PATCH 1/7] [mlir][OpenACC] Support multi-rank array reductions

Initialize static rank-N per-thread accumulators and delinearize flattened reduction indices before accessing them.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 61 +++++++++---
 .../acc-cg-to-gpu-reduction-array.mlir        | 94 +++++++++++++++++++
 2 files changed, 142 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 77fa82ba6bb4e..44ad1f3154174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,17 +736,28 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static rank-1");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
-  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
   Value step = arith::ConstantIndexOp::create(b, loc, 1);
-  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
-  OpBuilder::InsertionGuard g(b);
-  b.setInsertionPoint(forOp.getBody()->getTerminator());
-  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+  SmallVector<Value> indices;
+  auto buildLoopNest = [&](auto &&self, unsigned dim) -> void {
+    if (dim == baseTy.getRank()) {
+      memref::StoreOp::create(b, loc, ident, alloca, indices);
+      return;
+    }
+
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+    OpBuilder::InsertionGuard g(b);
+    b.setInsertionPoint(forOp.getBody()->getTerminator());
+    indices.push_back(forOp.getInductionVar());
+    self(self, dim + 1);
+    indices.pop_back();
+  };
+  buildLoopNest(buildLoopNest, 0);
 }
 
 std::optional<int64_t>
@@ -3244,10 +3255,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  if (!memref)
+  if (!memrefTy) {
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
-  if (memrefTy.getRank() != 1)
-    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() == 0) {
+    (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
+    return;
+  }
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
@@ -3355,9 +3375,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     OpBuilder::InsertionGuard guard(rewriter);
     rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
     Value iv = forOp.getInductionVar();
-    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
-    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
-                         ValueRange{iv},
+    SmallVector<Value> indices{iv};
+    if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
+      indices.resize(memrefTy.getRank());
+      Value linearIndex = iv;
+      for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
+        indices[dim] =
+            arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
+        if (dim != 0)
+          linearIndex =
+              arith::DivUIOp::create(rewriter, loc, linearIndex, dimSize);
+      }
+    }
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, indices);
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(), indices,
                          /*isPerThreadPrivateTarget=*/true);
   }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index bfb829c901af3..a008c84108f05 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -134,3 +134,97 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_array_reduction
+// CHECK: %[[R2_ALLOCA:.*]] = memref.alloca() : memref<2x3xi32>
+// CHECK: scf.for %[[R2_I:.*]] =
+// CHECK:   scf.for %[[R2_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[R2_ALLOCA]][%[[R2_I]], %[[R2_J]]] : memref<2x3xi32>
+// CHECK: scf.for %[[R2_LINEAR:.*]] =
+// CHECK: %[[R2_J_IDX:.*]] = arith.remui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+
+func.func @rank_two_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x3xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_three_array_reduction
+// CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_I:.*]] =
+// CHECK:   scf.for %[[R3_J:.*]] =
+// CHECK:     scf.for %[[R3_K:.*]] =
+// CHECK:       memref.store %{{.*}}, %[[R3_ALLOCA]][%[[R3_I]], %[[R3_J]], %[[R3_K]]] : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_LINEAR:.*]] =
+// CHECK: %[[R3_K_IDX:.*]] = arith.remui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_PLANE:.*]] = arith.divui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_J_IDX:.*]] = arith.remui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+
+func.func @rank_three_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x2x2xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x2x2xi32>>) {
+    %c8 = arith.constant 8 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x2x2xi32>>) -> memref<2x2x2xi32>
+    %bounds = acc.bounds extent(%c8 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x2x2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
+// invalid indexing operations.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_zero_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @rank_zero_array_reduction(%local: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
+    %extent = arith.constant 1 : index
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 1a70c1679ce1c49a2e54fc8ed501ba06fa3d7234 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:57 -0700
Subject: [PATCH 2/7] [mlir][acc] Use dynamic stack storage for thread-private
 arrays

Avoid grid-sized backing allocations for dynamic per-thread storage while preserving shared lowering for oversized static arrays.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 +++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 41 ++++++++++++----
 3 files changed, 90 insertions(+), 28 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 44ad1f3154174..7135ca9177153 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,8 +736,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static ranked");
+  assert(baseTy.getRank() > 0 &&
+         "per-thread array reduction accumulator must have positive rank");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
@@ -749,7 +749,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
       return;
     }
 
-    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    Value ub =
+        baseTy.isDynamicDim(dim)
+            ? memref::DimOp::create(b, loc, alloca, dim).getResult()
+            : arith::ConstantIndexOp::create(b, loc, baseTy.getDimSize(dim))
+                  .getResult();
     auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
     OpBuilder::InsertionGuard g(b);
     b.setInsertionPoint(forOp.getBody()->getTerminator());
@@ -2494,9 +2498,15 @@ void ACCCGToGPULowering::processPrivateLocal(
     // the accumulate can reduce each element across threads.
     acc::ReductionAccumulateArrayOp arrayAccum =
         perThreadArrayReductionAccum(privateLocal.getResult());
-    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+    bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
+    bool canUseDynamicAlloca =
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -2583,9 +2593,15 @@ void ACCCGToGPULowering::processPrivateLocal(
   acc::ReductionAccumulateArrayOp arrayAccum =
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    bool canUseDynamicAlloca =
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -3263,12 +3279,6 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
     return;
   }
-  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
-    (void)accSupport.emitNYI(loc,
-                             "reduction: dynamic multi-rank accumulate array");
-    return;
-  }
-
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3377,13 +3387,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
-      assert(memrefTy.hasStaticShape() &&
-             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize = arith::ConstantIndexOp::create(
-            rewriter, loc, memrefTy.getDimSize(dim));
+        Value dimSize =
+            memrefTy.isDynamicDim(dim)
+                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
+                : arith::ConstantIndexOp::create(rewriter, loc,
+                                                 memrefTy.getDimSize(dim))
+                      .getResult();
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index 281eae4b5b0f7..b186720512991 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,3 +23,32 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
+
+// Dynamic thread-private storage is allocated in the launch body. A
+// grid-wide backing allocation would scale with the logical block count.
+//
+// CHECK-LABEL: func.func @dynamic_threadprivate
+// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
+// CHECK:       gpu.launch
+// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
+// CHECK-NOT:     acc.unwrap_private
+
+func.func @dynamic_threadprivate(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
+      : (index) -> !acc.private_type<memref<?xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg = %private, %extent = %n)
+      : (!acc.private_type<memref<?xi32>>, index) {
+    %c0 = arith.constant 0 : index
+    %c0_i32 = arith.constant 0 : i32
+    %local = acc.private_local %arg
+        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
+    memref.store %c0_i32, %local[%c0] : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index a008c84108f05..f287ae054a846 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -192,25 +192,46 @@ func.func @rank_three_array_reduction() {
   return
 }
 
-// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
-// invalid indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
 // CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[DYN_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_I:.*]] =
+// CHECK:   scf.for %[[DYN_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_LINEAR:.*]] =
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: arith.remui %[[DYN_LINEAR]]
+// CHECK: arith.divui %[[DYN_LINEAR]]
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
   return
 }
 
+// Rank-zero accumulators must not generate invalid indexing operations.
 // CHECK-LABEL: func.func @rank_zero_array_reduction
 // CHECK: gpu.launch
 // CHECK-NOT: memref.load

>From 94cd48df3ac887a669b4f7c4cd871226d7014566 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:04:45 -0700
Subject: [PATCH 3/7] [mlir][OpenACC] Handle dynamic reductions on all thread
 dimensions

Classify dynamic accumulators using every GPU thread dimension so rank-N reductions consistently receive per-thread storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  7 +++--
 .../acc-cg-to-gpu-reduction-array.mlir        | 29 ++++---------------
 2 files changed, 9 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 7135ca9177153..e353f70939699 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -2500,7 +2500,8 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
@@ -2594,7 +2595,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
@@ -3334,7 +3335,7 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   } else {
     isPerThreadPrivate = llvm::any_of(
         op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isThreadX(); });
+        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index f287ae054a846..be91cc6762da6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -211,41 +211,22 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_x]>}
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
-
-// Rank-zero accumulators must not generate invalid indexing operations.
-// CHECK-LABEL: func.func @rank_zero_array_reduction
-// CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @rank_zero_array_reduction(%local: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
-    %extent = arith.constant 1 : index
-    %bounds = acc.bounds extent(%extent : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}

>From caf6ec726262faae429717edb8cadc1b6cd00abc Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:20:44 -0700
Subject: [PATCH 4/7] [mlir][OpenACC] Find array reductions through descriptor
 chains

Trace accumulate operands back to private storage so descriptor construction cannot bypass per-thread allocation and initialization.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 23 ++++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 35 +++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index e353f70939699..cc3b8910ad765 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -727,6 +727,29 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
+
+  // Descriptor construction may obscure the forward use chain. Fall back to
+  // tracing each accumulate operand back to this private local.
+  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
+  if (!privateLocal)
+    return nullptr;
+  acc::ComputeRegionOp computeRegion =
+      privateLocal->getParentOfType<acc::ComputeRegionOp>();
+  if (!computeRegion)
+    return nullptr;
+  acc::ReductionAccumulateArrayOp result;
+  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
+    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
+      return WalkResult::advance();
+    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
+                                  [](auto pd) { return pd.isAnyThread(); });
+    if (!hasThread || !reductionHasBlockContext(accArr))
+      return WalkResult::advance();
+    result = accArr;
+    return WalkResult::interrupt();
+  });
+  if (result)
+    return result;
   return nullptr;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index be91cc6762da6..ae9ea1e4c95bb 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -230,3 +230,38 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[INDIRECT_I:.*]] =
+// CHECK:   scf.for %[[INDIRECT_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor_value = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 3790839be0295d1bd7f4ca3bbd151aac1edd5a9a Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:34:24 -0700
Subject: [PATCH 5/7] [mlir][OpenACC] Allocate dynamic privates for any thread
 dimension

Use privatization parallel dimensions so descriptor indirection cannot force dynamic thread-private backing arrays into grid-sized storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 +++++++------------
 .../acc-cg-to-gpu-reduction-array.mlir        | 34 ++++++++++---
 2 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index cc3b8910ad765..99315cad34e1c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,6 +383,14 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
+/// True when \p privatize has any thread-level parallelism.
+static bool isThreadPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
+  return false;
+}
+
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -728,28 +736,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     }
   }
 
-  // Descriptor construction may obscure the forward use chain. Fall back to
-  // tracing each accumulate operand back to this private local.
-  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
-  if (!privateLocal)
-    return nullptr;
-  acc::ComputeRegionOp computeRegion =
-      privateLocal->getParentOfType<acc::ComputeRegionOp>();
-  if (!computeRegion)
-    return nullptr;
-  acc::ReductionAccumulateArrayOp result;
-  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
-    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
-      return WalkResult::advance();
-    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
-                                  [](auto pd) { return pd.isAnyThread(); });
-    if (!hasThread || !reductionHasBlockContext(accArr))
-      return WalkResult::advance();
-    result = accArr;
-    return WalkResult::interrupt();
-  });
-  if (result)
-    return result;
   return nullptr;
 }
 
@@ -2523,12 +2509,13 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((isThreadPrivate || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (isThreadPrivate || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2618,12 +2605,13 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((parDim.isThreadX() || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (parDim.isThreadX() || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index ae9ea1e4c95bb..b9fdf6ecabcb6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -245,17 +245,39 @@ func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %private_data = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
+        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
+            %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>,
+           !acc.private_type<memref<1xindex>>, index) {
+      %c0 = arith.constant 0 : index
+      %c1_k = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      %local_data = acc.private_local %arg_data
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %descriptor_value = builtin.unrealized_conversion_cast %local
-          : memref<?x?xi32> to memref<?x?xi32>
+      %local_descriptor = acc.private_local %arg_descriptor
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
+      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
+      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
+      scf.for %i = %c0 to %m_dim step %c1_k {
+        scf.for %j = %c0 to %n_dim step %c1_k {
+          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
+        }
+      }
+      %descriptor = builtin.unrealized_conversion_cast %local_data
+          : memref<?x?xi32> to index
+      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
+      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
+      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
+          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
           : memref<?x?xi32>

>From affc0c39172f713075ac1912eb631d85b4ce6b56 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:24 -0700
Subject: [PATCH 6/7] [mlir][OpenACC] Preserve worker-private array storage

Restrict generated stack reductions to thread_x and keep thread_y/thread_z arrays indexed or shared, including explicit dynamic shared memory.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  50 ++++----
 .../acc-cg-to-gpu-reduction-array.mlir        | 107 +++++++++++-------
 2 files changed, 86 insertions(+), 71 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 99315cad34e1c..48e7d13730eeb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,14 +383,6 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
-/// True when \p privatize has any thread-level parallelism.
-static bool isThreadPrivatize(PrivatizeOp privatize) {
-  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
-    return llvm::any_of(parDimsAttr.getArray(),
-                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
-  return false;
-}
-
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -709,7 +701,7 @@ static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
 }
 
 /// Returns the array reduction accumulate (through cast/view ops) that \p v
-/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// feeds if it needs per-thread storage: its par_dims include thread_x
 /// and it has block context so the cross-thread all_reduce is well defined.
 static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   SmallVector<Value> worklist{v};
@@ -721,10 +713,10 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     for (Operation *user : cur.getUsers()) {
       if (acc::ReductionAccumulateArrayOp accArr =
               dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
-        bool hasThread = false;
+        bool hasThreadX = false;
         for (auto pd : accArr.getParDims().getArray())
-          hasThread |= pd.isAnyThread();
-        if (hasThread && reductionHasBlockContext(accArr))
+          hasThreadX |= pd.isThreadX();
+        if (hasThreadX && reductionHasBlockContext(accArr))
           return accArr;
         continue;
       }
@@ -735,7 +727,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
-
   return nullptr;
 }
 
@@ -2509,13 +2500,11 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (isThreadPrivate || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2605,13 +2594,12 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((parDim.isThreadX() || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -3337,16 +3325,20 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   // it is too large. For a dynamically-shaped accumulator the type conveys no
   // size, so classify from par_dims (which the producer sets to the reduction's
   // actual parallel scope): a thread dimension means per-thread storage.
+  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
+  bool isExplicitlyShared =
+      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
   bool isPerThreadPrivate;
   if (memrefTy.hasStaticShape()) {
-    Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
     isPerThreadPrivate =
-        !isa_and_nonnull<memref::AllocOp>(rootOp) &&
+        !isExplicitlyShared &&
         canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
   } else {
-    isPerThreadPrivate = llvm::any_of(
-        op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
+    isPerThreadPrivate = !isExplicitlyShared &&
+                         llvm::any_of(op.getParDims().getArray(),
+                                      [](mlir::acc::GPUParallelDimAttr d) {
+                                        return d.isThreadX();
+                                      });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index b9fdf6ecabcb6..269e0a1287046 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,27 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// CHECK-LABEL: func.func @array_reduction_dynamic_shared
+// CHECK: acc.gpu_shared_memory
+// CHECK-NOT: gpu.all_reduce
+func.func @array_reduction_dynamic_shared(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%ext = %n) : (index) {
+    %shared = acc.gpu_shared_memory(%ext)
+        {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
+        : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
@@ -162,6 +183,33 @@ func.func @rank_two_array_reduction() {
   return
 }
 
+// CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @rank_two_array_reduction_thread_y() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %kty = %ty)
+      ins(%arg = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg
+        {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+        : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<2x3xi32>
+        {par_dims = #acc<par_dims[block_x, thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @rank_three_array_reduction
 // CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
 // CHECK: scf.for %[[R3_I:.*]] =
@@ -211,75 +259,50 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
 
-// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// Dynamic worker-level storage must remain indexed by thread_y instead of
+// becoming one stack allocation per physical CUDA thread.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
-// CHECK-NOT: acc.unwrap_private
-// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
-// CHECK: scf.for %[[INDIRECT_I:.*]] =
-// CHECK:   scf.for %[[INDIRECT_J:.*]] =
-// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
-// CHECK: gpu.all_reduce add
-func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private_data = acc.privatize(%m, %n)
-      [#acc<par_dims[block_x, thread_y]>]
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
-      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
-            %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>,
-           !acc.private_type<memref<1xindex>>, index) {
-      %c0 = arith.constant 0 : index
-      %c1_k = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      %local_data = acc.private_local %arg_data
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %local_descriptor = acc.private_local %arg_descriptor
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
-          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
-      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
-      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
-      scf.for %i = %c0 to %m_dim step %c1_k {
-        scf.for %j = %c0 to %n_dim step %c1_k {
-          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
-        }
-      }
-      %descriptor = builtin.unrealized_conversion_cast %local_data
-          : memref<?x?xi32> to index
-      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
-      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
-      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
-          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield

>From 3bba5f0a8722c8926daa522949668e420f5ea9aa Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:33:43 -0700
Subject: [PATCH 7/7] [mlir][OpenACC] Track array reduction storage provenance

Classify accumulators from their backing allocation through casts so shared storage is not reduced again and private storage is initialized correctly. Keep dynamic stack allocation reduction-specific and honor byte strides.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 71 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 --------
 .../acc-cg-to-gpu-reduction-array.mlir        | 61 ++++++++++------
 3 files changed, 82 insertions(+), 79 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 48e7d13730eeb..84243de10830a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -317,6 +317,17 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     return false;
   }
 
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
+    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
+        castOp->getOperand(0) == trackedOperand &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      passThroughResults.push_back(castOp->getResult(0));
+      return true;
+    }
+    return false;
+  }
+
   // Partial-entity accesses (e.g. array element or field access) forward the
   // base entity through to their results, so treat them as pass-through when
   // the base entity is the value being tracked.
@@ -341,6 +352,14 @@ static Value unwrapMemRefConversion(Value v) {
         continue;
       }
     }
+    if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+        castOp && castOp->getNumOperands() == 1 &&
+        castOp->getNumResults() == 1 &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      v = castOp->getOperand(0);
+      continue;
+    }
     break;
   }
   return v;
@@ -730,6 +749,13 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   return nullptr;
 }
 
+static bool
+canUseDynamicReductionAlloca(MemRefType baseTy, acc::PrivatizeOp privatizeOp,
+                             acc::ReductionAccumulateArrayOp arrayAccum) {
+  return arrayAccum && !baseTy.hasStaticShape() &&
+         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+}
+
 /// Store the reduction identity to every element of a freshly allocated
 /// per-thread array accumulator so all lanes start from identity (the original
 /// init loop may only run on one lane).
@@ -2500,8 +2526,7 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -2594,9 +2619,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((parDim.isThreadX() || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -3318,28 +3341,10 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   }
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
-  // accumulator copy. For a statically-shaped accumulator, classify from the
-  // operand: an explicit shared allocation is block-shared regardless of size,
-  // and anything else (a per-thread stack alloca, or a view over one) is
-  // per-thread when it fits the per-thread stack budget and block-shared when
-  // it is too large. For a dynamically-shaped accumulator the type conveys no
-  // size, so classify from par_dims (which the producer sets to the reduction's
-  // actual parallel scope): a thread dimension means per-thread storage.
+  // accumulator copy. Classify the operand from its underlying allocation
+  // rather than its shape or par_dims: only memref.alloca is thread-private.
   Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isExplicitlyShared =
-      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
-  bool isPerThreadPrivate;
-  if (memrefTy.hasStaticShape()) {
-    isPerThreadPrivate =
-        !isExplicitlyShared &&
-        canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
-  } else {
-    isPerThreadPrivate = !isExplicitlyShared &&
-                         llvm::any_of(op.getParDims().getArray(),
-                                      [](mlir::acc::GPUParallelDimAttr d) {
-                                        return d.isThreadX();
-                                      });
-  }
+  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
@@ -3367,7 +3372,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
   Value lb =
       boundsOp.getLowerbound() ? toIndex(boundsOp.getLowerbound()) : zero;
-  Value step = boundsOp.getStride() ? toIndex(boundsOp.getStride()) : one;
+  Value step = one;
+  if (Value stride = boundsOp.getStride()) {
+    step = toIndex(stride);
+    if (boundsOp.getStrideInBytes()) {
+      int64_t elementSize =
+          getElementSizeInBytes(loc, memrefTy.getElementType());
+      if (elementSize == 0)
+        return;
+      Value elementSizeValue =
+          arith::ConstantIndexOp::create(rewriter, loc, elementSize);
+      step = arith::DivUIOp::create(rewriter, loc, step, elementSizeValue);
+    }
+  }
   // Exclusive upper bound. `extent` counts elements, so the span is
   // `extent * step` (for the common unit-stride case step is 1); fall back to
   // the inclusive upperbound when no extent is given.
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index b186720512991..281eae4b5b0f7 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,32 +23,3 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
-
-// Dynamic thread-private storage is allocated in the launch body. A
-// grid-wide backing allocation would scale with the logical block count.
-//
-// CHECK-LABEL: func.func @dynamic_threadprivate
-// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
-// CHECK:       gpu.launch
-// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
-// CHECK-NOT:     acc.unwrap_private
-
-func.func @dynamic_threadprivate(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
-      : (index) -> !acc.private_type<memref<?xi32>>
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-      ins(%arg = %private, %extent = %n)
-      : (!acc.private_type<memref<?xi32>>, index) {
-    %c0 = arith.constant 0 : index
-    %c0_i32 = arith.constant 0 : i32
-    %local = acc.private_local %arg
-        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
-    memref.store %c0_i32, %local[%c0] : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 269e0a1287046..56469f801f026 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -85,7 +85,8 @@ func.func @array_reduction_small_shared() {
 }
 
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
-// CHECK: acc.gpu_shared_memory
+// CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
+// CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]
 // CHECK-NOT: gpu.all_reduce
 func.func @array_reduction_dynamic_shared(%n: index) {
   %c1 = arith.constant 1 : index
@@ -96,8 +97,11 @@ func.func @array_reduction_dynamic_shared(%n: index) {
     %shared = acc.gpu_shared_memory(%ext)
         {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
         : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %cast = builtin.unrealized_conversion_cast %shared
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        to memref<?xi32, #gpu.address_space<workgroup>>
     %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+    acc.reduction_accumulate_array %cast bounds(%bounds) <add>
         : memref<?xi32, #gpu.address_space<workgroup>>
         {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -108,11 +112,13 @@ func.func @array_reduction_dynamic_shared(%n: index) {
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
-// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[BYTE_STEP:.*]] = arith.constant 8 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[ELEMENT_SIZE:.*]] = arith.constant 4 : index
+// CHECK: %[[ELEMENT_STEP:.*]] = arith.divui %[[BYTE_STEP]], %[[ELEMENT_SIZE]] : index
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[ELEMENT_STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
-// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[ELEMENT_STEP]]
 func.func @array_reduction_strided_extent() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -120,11 +126,11 @@ func.func @array_reduction_strided_extent() {
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
   acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
     %c1_b = arith.constant 1 : index
-    %c2 = arith.constant 2 : index
+    %c8 = arith.constant 8 : index
     %c3 = arith.constant 3 : index
     %local = memref.alloca() : memref<8xi32>
     %bounds = acc.bounds lowerbound(%c1_b : index) extent(%c3 : index)
-        stride(%c2 : index)
+        stride(%c8 : index) {strideInBytes = true}
     acc.reduction_accumulate_array %local bounds(%bounds) <add>
         : memref<8xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -132,13 +138,14 @@ func.func @array_reduction_strided_extent() {
   return
 }
 
-// A dynamically-shaped accumulator (a strided view whose type conveys no size)
-// is classified per-thread from par_dims: a thread dimension means per-thread
-// storage, so lowering emits the per-element gpu.all_reduce.
+// A dynamically-shaped view over an external buffer is shared even when the
+// accumulate includes thread_x. Storage provenance, not par_dims, controls
+// whether gpu.all_reduce is valid.
 //
 // CHECK-LABEL: func.func @array_reduction_dynamic_par_dims
-// CHECK: scf.for
-// CHECK: gpu.all_reduce add
+// CHECK: memref.reinterpret_cast
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
 func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -186,6 +193,7 @@ func.func @rank_two_array_reduction() {
 // CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
+// CHECK: acc.gpu_shared_memory
 // CHECK-NOT: gpu.all_reduce
 // CHECK: gpu.terminator
 func.func @rank_two_array_reduction_thread_y() {
@@ -247,12 +255,13 @@ func.func @rank_three_array_reduction() {
 // CHECK: scf.for %[[DYN_I:.*]] =
 // CHECK:   scf.for %[[DYN_J:.*]] =
 // CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: %[[DYN_DESC:.*]] = builtin.unrealized_conversion_cast %[[DYN_ALLOCA]]
 // CHECK: scf.for %[[DYN_LINEAR:.*]] =
-// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.dim %[[DYN_DESC]]
 // CHECK: arith.remui %[[DYN_LINEAR]]
 // CHECK: arith.divui %[[DYN_LINEAR]]
-// CHECK: memref.dim %[[DYN_ALLOCA]]
-// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: memref.dim %[[DYN_DESC]]
+// CHECK: memref.load %[[DYN_DESC]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
 // CHECK: gpu.all_reduce add
 func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
@@ -269,8 +278,10 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
       %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+      acc.reduction_accumulate_array %descriptor bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
@@ -279,32 +290,36 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   return
 }
 
-// Dynamic worker-level storage must remain indexed by thread_y instead of
+// Dynamic worker-level storage must remain indexed by thread_y/thread_z instead
 // becoming one stack allocation per physical CUDA thread.
-// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_yz
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
 // CHECK-NOT: gpu.all_reduce
+// CHECK: %[[WORKER_STORAGE:.*]] = memref.view %{{.*}}[%{{.*}}][%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}] : memref<?xi8> to memref<?x?x?x?x?xi32>
+// CHECK: memref.subview %[[WORKER_STORAGE]][%{{.*}}, %{{.*}}, %{{.*}}, 0, 0] [1, 1, 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1, 1]
 // CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
+func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tz = acc.par_width %c128 {par_dim = #acc.par_dim<thread_z>}
+  %private = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y, thread_z]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty, %ktz = %tz)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
       acc.yield
     } {origin = "acc.parallel"}
   }


From llvmlistbot at llvm.org  Mon Jul 20 20:38:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 20:38:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5ee9bf.27b3b003.1f84cc.d493@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

* 145041 tests passed
* 3008 tests skipped

All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.

<details>
<summary>tools/lldb/source/Expression/CMakeFiles/lldbExpression.dir/DWARFExpression.cpp.o</summary>

```
FAILED: tools/lldb/source/Expression/CMakeFiles/lldbExpression.dir/DWARFExpression.cpp.o
sccache /opt/llvm/bin/clang++ -DHAVE_ROUND -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/lldb/source/Expression -I/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/source/Expression -I/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/lldb/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/../clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/lldb/../clang/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/source -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/lldb/source -isystem /usr/include/libxml2 -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wno-unknown-pragmas -Wno-strict-aliasing -Wno-vla-extension -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT tools/lldb/source/Expression/CMakeFiles/lldbExpression.dir/DWARFExpression.cpp.o -MF tools/lldb/source/Expression/CMakeFiles/lldbExpression.dir/DWARFExpression.cpp.o.d -o tools/lldb/source/Expression/CMakeFiles/lldbExpression.dir/DWARFExpression.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/source/Expression/DWARFExpression.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/lldb/source/Expression/DWARFExpression.cpp:198:11: error: enumeration values 'DW_OP_LLVM_label', 'DW_OP_LLVM_bra', and 'DW_OP_LLVM_skip' not handled in switch [-Werror,-Wswitch]
198 |   switch (op) {
|           ^~
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 20:44:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 20:44:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Preserve worker rows when
 combining reductions (PR #210804)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210804@github.com>
Message-ID: <6a5eeb15.e5907290.281687.567d@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210804

>From 4c316621a473f21607795bafbd1ffa51a9dabe93 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 13:12:09 -0700
Subject: [PATCH 01/20] [OpenACC] Preserve worker rows when combining
 reductions

Keep ThreadY active for worker-private sources so every worker row contributes, and reject mixed scopes that require incompatible predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 53 +++++++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 43 +++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 95 +++++++++++++++++++
 3 files changed, 181 insertions(+), 10 deletions(-)
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 24aeed6e23bda..6be79b0dd40e4 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,7 +1246,30 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    block->walk([&](Operation *op) {
+    std::optional<bool> combineThreadYActive;
+    auto applyCombineParDims =
+        [&](Operation *combineOp, Value src,
+            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+          bool isWorkerPrivate =
+              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
+            if (parDim.isThreadY()) {
+              if (combineThreadYActive &&
+                  *combineThreadYActive != isWorkerPrivate) {
+                combineOp->emitError()
+                    << "mixed worker-private and non-worker-private reduction "
+                       "combines require incompatible ThreadY predication";
+                hasFailed = true;
+                return failure();
+              }
+              combineThreadYActive = isWorkerPrivate;
+            } else {
+              mlir::acc::removeParDim(ancestorParDims, parDim);
+            }
+          }
+          return success();
+        };
+    block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1277,17 +1300,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(reductionCombineOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                reductionCombineOp, reductionCombineOp.getSrcMemref(),
+                getReductionCombineParDims(reductionCombineOp))))
+          return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(combineRegionOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                combineRegionOp, combineRegionOp.getSrcVar(),
+                getReductionCombineParDims(combineRegionOp))))
+          return WalkResult::interrupt();
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1301,6 +1324,14 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
+    if (combineThreadYActive) {
+      mlir::acc::GPUParallelDimAttr threadY =
+          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+      if (*combineThreadYActive)
+        mlir::acc::insertParDim(ancestorParDims, threadY);
+      else
+        mlir::acc::removeParDim(ancestorParDims, threadY);
+    }
   }
 
   // Obtain launch dimensions
@@ -1747,7 +1778,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasBlock && hasThreadY)
+  if (hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;
@@ -1871,6 +1902,8 @@ void ACCCGToGPULowering::processPredicateRegion(
             SmallVector<mlir::acc::GPUParallelDimAttr>>
       parDimsPair = computeActiveAndInactiveParDims(
           interOp, &interOp.getRegion().front());
+  if (hasFailed)
+    return;
 
   // If ThreadY reduction exists, subgroup alignment is applied
   // (blockDim.x = subgroupSize), so ThreadX lanes exist even without explicit
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
new file mode 100644
index 0000000000000..a0d381740c034
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -0,0 +1,43 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" \
+// RUN:   -verify-diagnostics
+
+func.func @mixed_scope_worker_reduction_combine(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
new file mode 100644
index 0000000000000..b2fd246d33d1b
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -0,0 +1,95 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A worker-private reduction has one shared slot per ThreadY row. The combine
+// must keep ThreadY active and predicate only ThreadX.
+
+// CHECK-LABEL: func.func @worker_reduction_combine
+// CHECK: gpu.launch {{.*}} threads([[TID_X:%[^,]+]], [[TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[TID_Y]]
+// CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: acc.atomic.update
+
+func.func @worker_reduction_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @worker_reduction_combine_region
+// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
+// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[REGION_IS_X_ZERO]]
+// CHECK: arith.addi
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From d763e8065946687750b2c78fcf5393167b7de962 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 15:08:39 -0700
Subject: [PATCH 02/20] [OpenACC] Report mixed reduction scopes as NYI

Use the standard OpenACC unsupported-feature diagnostic for incompatible ThreadY predication.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6be79b0dd40e4..378fa17b24850 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1256,9 +1256,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             if (parDim.isThreadY()) {
               if (combineThreadYActive &&
                   *combineThreadYActive != isWorkerPrivate) {
-                combineOp->emitError()
-                    << "mixed worker-private and non-worker-private reduction "
-                       "combines require incompatible ThreadY predication";
+                (void)accSupport.emitNYI(
+                    combineOp->getLoc(),
+                    "mixed worker-private and non-worker-private reduction "
+                    "combines require incompatible ThreadY predication");
                 hasFailed = true;
                 return failure();
               }

>From 1fe76c2f1ea14bd036598286c2d9c0ea8d2291ef Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 16:50:59 -0700
Subject: [PATCH 03/20] [OpenACC] Isolate ThreadY predicate requirements

Propagate nested worker requirements without broadening unrelated side effects, and reject incompatible operations sharing one predicate region.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 126 +++++++++++++++---
 ...-worker-reduction-combine-mixed-scope.mlir |  40 +++++-
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  46 +++++++
 3 files changed, 189 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 378fa17b24850..34131dc3753a3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,27 +1246,111 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    std::optional<bool> combineThreadYActive;
+    struct ThreadYRequirement {
+      bool active = false;
+      bool inactive = false;
+    };
+    auto analyzeThreadYRequirements =
+        [&](auto &&self,
+            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
+      ThreadYRequirement requirement;
+      auto requireThreadY = [&](Operation *op, bool active) {
+        if ((active && requirement.inactive) ||
+            (!active && requirement.active)) {
+          (void)accSupport.emitNYI(
+              op->getLoc(),
+              "operations in the same predicate region require incompatible "
+              "ThreadY predication");
+          hasFailed = true;
+          return failure();
+        }
+        requirement.active |= active;
+        requirement.inactive |= !active;
+        return success();
+      };
+      auto applyCombineRequirement =
+          [&](Operation *combineOp, Value src,
+              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+            if (llvm::none_of(combineParDims,
+                              [](auto parDim) { return parDim.isThreadY(); }))
+              return success();
+            bool isWorkerPrivate =
+                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+            return requireThreadY(combineOp, isWorkerPrivate);
+          };
+
+      for (Operation &nestedOp : predicateBlock) {
+        if (acc::PredicateRegionOp nestedPredicate =
+                dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
+          FailureOr<ThreadYRequirement> nestedRequirement =
+              self(self, nestedPredicate.getRegion().front());
+          if (failed(nestedRequirement))
+            return failure();
+          // An active descendant must not be excluded by this region's
+          // predicate. Inactive descendants apply their own predicate.
+          if (nestedRequirement->active &&
+              failed(requireThreadY(&nestedOp, /*active=*/true)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineOp combineOp =
+                dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineOp, combineOp.getSrcMemref(),
+                  getReductionCombineParDims(combineOp))))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineRegionOp combineRegionOp =
+                dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineRegionOp, combineRegionOp.getSrcVar(),
+                  getReductionCombineParDims(combineRegionOp))))
+            return failure();
+          continue;
+        }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          bool threadYActive = false;
+          if (acc::PrivateLocalOp privateLocal =
+                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+            if (acc::GPUParallelDimsAttr parDims =
+                    getPrivatizeOp(privateLocal, computeRegion)
+                        .getParDimsAttr()) {
+              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
+                return parDim.isThreadY();
+              });
+            }
+          }
+          if (failed(requireThreadY(storeOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
+                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
+          bool threadYActive =
+              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
+                           [](auto parDim) { return parDim.isThreadY(); });
+          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (!isMemoryEffectFree(&nestedOp) &&
+            failed(requireThreadY(&nestedOp, /*active=*/false)))
+          return failure();
+      }
+      return requirement;
+    };
+
+    FailureOr<ThreadYRequirement> threadYRequirement =
+        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
+    if (failed(threadYRequirement))
+      return {};
+
     auto applyCombineParDims =
-        [&](Operation *combineOp, Value src,
-            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          bool isWorkerPrivate =
-              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
           for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (parDim.isThreadY()) {
-              if (combineThreadYActive &&
-                  *combineThreadYActive != isWorkerPrivate) {
-                (void)accSupport.emitNYI(
-                    combineOp->getLoc(),
-                    "mixed worker-private and non-worker-private reduction "
-                    "combines require incompatible ThreadY predication");
-                hasFailed = true;
-                return failure();
-              }
-              combineThreadYActive = isWorkerPrivate;
-            } else {
+            if (!parDim.isThreadY())
               mlir::acc::removeParDim(ancestorParDims, parDim);
-            }
           }
           return success();
         };
@@ -1302,14 +1386,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
         if (failed(applyCombineParDims(
-                reductionCombineOp, reductionCombineOp.getSrcMemref(),
                 getReductionCombineParDims(reductionCombineOp))))
           return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
         if (failed(applyCombineParDims(
-                combineRegionOp, combineRegionOp.getSrcVar(),
                 getReductionCombineParDims(combineRegionOp))))
           return WalkResult::interrupt();
       }
@@ -1325,10 +1407,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (combineThreadYActive) {
+    if (threadYRequirement->active || threadYRequirement->inactive) {
       mlir::acc::GPUParallelDimAttr threadY =
           mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (*combineThreadYActive)
+      if (threadYRequirement->active)
         mlir::acc::insertParDim(ancestorParDims, threadY);
       else
         mlir::acc::removeParDim(ancestorParDims, threadY);
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a0d381740c034..e80b5afa26d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -30,7 +30,7 @@ func.func @mixed_scope_worker_reduction_combine(
         acc.predicate_region {
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -41,3 +41,41 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_single_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c7_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          memref.store %c7_i32, %result_arg[] : memref<i32>
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index b2fd246d33d1b..afd6108d6e670 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -93,3 +93,49 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+// Nested predicate regions choose their ThreadY predicates independently. The
+// outer region must keep ThreadY active so it does not exclude worker rows
+// before the nested worker-private combine is reached.
+func.func @nested_worker_reduction_combines(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          acc.predicate_region {
+            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 477f10e0472195d6bb55598bf364a1655682ab6f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:04:51 -0700
Subject: [PATCH 04/20] [OpenACC] Propagate nested ThreadY requirements

Analyze control-flow regions recursively and recognize aliased worker-private stores so their worker rows remain active.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 32 ++++---
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 86 +++++++++++++++++++
 2 files changed, 107 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 34131dc3753a3..98c697f52668e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1310,17 +1310,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = false;
-          if (acc::PrivateLocalOp privateLocal =
-                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-            if (acc::GPUParallelDimsAttr parDims =
-                    getPrivatizeOp(privateLocal, computeRegion)
-                        .getParDimsAttr()) {
-              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
-                return parDim.isThreadY();
-              });
-            }
-          }
+          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
+                               PrivateMemScope::Worker;
           if (failed(requireThreadY(storeOp, threadYActive)))
             return failure();
           continue;
@@ -1334,6 +1325,25 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             return failure();
           continue;
         }
+        if (nestedOp.getNumRegions() != 0) {
+          for (Region &region : nestedOp.getRegions()) {
+            for (Block &nestedBlock : region) {
+              FailureOr<ThreadYRequirement> nestedRequirement =
+                  self(self, nestedBlock);
+              if (failed(nestedRequirement))
+                return failure();
+              // Other region-bearing operations do not introduce independent
+              // ACC predication, so both requirements apply to this region.
+              if (nestedRequirement->active &&
+                  failed(requireThreadY(&nestedOp, /*active=*/true)))
+                return failure();
+              if (nestedRequirement->inactive &&
+                  failed(requireThreadY(&nestedOp, /*active=*/false)))
+                return failure();
+            }
+          }
+          continue;
+        }
         if (!isMemoryEffectFree(&nestedOp) &&
             failed(requireThreadY(&nestedOp, /*active=*/false)))
           return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index afd6108d6e670..8a2dd6bd470f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -97,6 +97,12 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.
+// CHECK-LABEL: func.func @nested_worker_reduction_combines
+// CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
+// CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[NESTED_TX_ZERO]]
 func.func @nested_worker_reduction_combines(
     %other: memref<i32>, %result: memref<i32>) {
   %c1 = arith.constant 1 : index
@@ -139,3 +145,83 @@ func.func @nested_worker_reduction_combines(
   }
   return
 }
+
+// CHECK-LABEL: func.func @worker_combine_in_scf_if
+// CHECK: gpu.launch {{.*}} threads([[IF_TX:%[^,]+]], [[IF_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[IF_TY]]
+// CHECK: %[[IF_TX_ZERO:.*]] = arith.cmpi eq, [[IF_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IF_TX_ZERO]]
+func.func @worker_combine_in_scf_if(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          scf.if %true {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @aliased_worker_store
+// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
+// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[ALIAS_TX_ZERO]]
+// CHECK: memref.store
+// CHECK: acc.atomic.update
+func.func @aliased_worker_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2e234b5cb4dac8aa7cf1ab07b63984b7cb9f329f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:16:55 -0700
Subject: [PATCH 05/20] [OpenACC] Account for predicate operation effects

Preserve all privatized ThreadY dimensions through aliases and reject intrinsically effecting region operations that cannot share worker predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 33 +++++++++++-----
 ...-worker-reduction-combine-mixed-scope.mlir | 38 +++++++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 36 ++++++++++++++++++
 3 files changed, 97 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98c697f52668e..20e8977c907f3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1268,15 +1268,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         requirement.inactive |= !active;
         return success();
       };
+      auto hasThreadYPrivateStorage = [&](Value memref) {
+        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
+        if (!privatize)
+          return false;
+        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
+                 return parDim.isThreadY();
+               });
+      };
       auto applyCombineRequirement =
           [&](Operation *combineOp, Value src,
               ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
             if (llvm::none_of(combineParDims,
                               [](auto parDim) { return parDim.isThreadY(); }))
               return success();
-            bool isWorkerPrivate =
-                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
-            return requireThreadY(combineOp, isWorkerPrivate);
+            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
           };
 
       for (Operation &nestedOp : predicateBlock) {
@@ -1310,9 +1317,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
-                               PrivateMemScope::Worker;
-          if (failed(requireThreadY(storeOp, threadYActive)))
+          if (failed(requireThreadY(
+                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
             return failure();
           continue;
         }
@@ -1326,6 +1332,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
+          bool hasOwnEffects = true;
+          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
+            hasOwnEffects = !effectOp.hasNoEffect();
+          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
+            hasOwnEffects = false;
+          }
+          if (hasOwnEffects &&
+              failed(requireThreadY(&nestedOp, /*active=*/false)))
+            return failure();
           for (Region &region : nestedOp.getRegions()) {
             for (Block &nestedBlock : region) {
               FailureOr<ThreadYRequirement> nestedRequirement =
@@ -1368,10 +1383,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (auto privateLocalOp =
-                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-          acc::PrivatizeOp privatizeOp =
-              getPrivatizeOp(privateLocalOp, computeRegion);
+        if (acc::PrivatizeOp privatizeOp =
+                getPrivatizeForMemref(storeOp.getMemref())) {
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index e80b5afa26d20..19e1b3106d966 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -79,3 +79,41 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c1_i32 = arith.constant 1 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.atomic.update %result_arg : memref<i32> {
+          ^bb0(%current: i32):
+            %next = arith.addi %current, %c1_i32 : i32
+            acc.yield %next : i32
+          }
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 8a2dd6bd470f8..ba8f16d3966df 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -225,3 +225,39 @@ func.func @aliased_worker_store(%result: memref<i32>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @aliased_thread_store
+// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
+// CHECK: memref.store
+func.func @aliased_thread_store() {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private)
+        : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From ae0d329822ede9a6d80731ebd45abd7e520ef5ae Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:26:21 -0700
Subject: [PATCH 06/20] [OpenACC] Require proven private aliases

Follow only alias-preserving view operations and treat every non-private combine as ThreadY-inactive to avoid broadening ambiguous global effects.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 45 +++++++++----------
 ...-worker-reduction-combine-mixed-scope.mlir |  6 ++-
 2 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 20e8977c907f3..9e9114e9aa174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1277,14 +1277,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                  return parDim.isThreadY();
                });
       };
-      auto applyCombineRequirement =
-          [&](Operation *combineOp, Value src,
-              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-            if (llvm::none_of(combineParDims,
-                              [](auto parDim) { return parDim.isThreadY(); }))
-              return success();
-            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
-          };
+      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
+        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
+      };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
@@ -1302,17 +1297,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineOp, combineOp.getSrcMemref(),
-                  getReductionCombineParDims(combineOp))))
+          if (failed(
+                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
             return failure();
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineRegionOp, combineRegionOp.getSrcVar(),
-                  getReductionCombineParDims(combineRegionOp))))
+          if (failed(applyCombineRequirement(combineRegionOp,
+                                             combineRegionOp.getSrcVar())))
             return failure();
           continue;
         }
@@ -1893,18 +1886,22 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  llvm::SmallVector<Value, 8> worklist{memref};
-  llvm::SmallPtrSet<Value, 8> seen;
-  while (!worklist.empty()) {
-    Value v = worklist.pop_back_val();
-    if (!seen.insert(v).second)
-      continue;
-    Operation *def = v.getDefiningOp();
-    if (!def)
-      continue;
+  Value current = memref;
+  while (Operation *def = current.getDefiningOp()) {
     if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
       return privateLocal;
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+      current = viewLike.getViewSource();
+      continue;
+    }
+    if (acc::PartialEntityAccessOpInterface partialAccess =
+            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
+      current = partialAccess.getBaseEntity();
+      continue;
+    }
+    // Do not follow arbitrary operands: multi-source operations such as
+    // arith.select do not prove that their result aliases private storage.
+    return nullptr;
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 19e1b3106d966..1275e4c77890a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -32,7 +32,7 @@ func.func @mixed_scope_worker_reduction_combine(
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+              {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
@@ -59,15 +59,17 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
+      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %result_arg[] : memref<i32>
+          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}

>From 3ed1de6f3b95aee6516db811bbd62dc4c54f9335 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:38:25 -0700
Subject: [PATCH 07/20] [OpenACC] Restrict ThreadY predicate broadening

Preserve legacy predicates unless a proven worker combine requires ThreadY, and reject broadening around writes, unknown effects, or incompatible combines.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 223 +++++++++---------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  78 +-----
 2 files changed, 107 insertions(+), 194 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 9e9114e9aa174..50635dde561db 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,138 +1246,122 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    struct ThreadYRequirement {
-      bool active = false;
-      bool inactive = false;
+    struct ThreadYBroadeningInfo {
+      bool hasActiveWorkerCombine = false;
+      bool hasExplicitInactiveCombine = false;
+      bool hasBroadeningConflict = false;
+      Operation *diagnosticOp = nullptr;
     };
-    auto analyzeThreadYRequirements =
-        [&](auto &&self,
-            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
-      ThreadYRequirement requirement;
-      auto requireThreadY = [&](Operation *op, bool active) {
-        if ((active && requirement.inactive) ||
-            (!active && requirement.active)) {
-          (void)accSupport.emitNYI(
-              op->getLoc(),
-              "operations in the same predicate region require incompatible "
-              "ThreadY predication");
-          hasFailed = true;
-          return failure();
+    auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
+                                  const ThreadYBroadeningInfo &src) {
+      dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
+      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
+      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      if (!dst.diagnosticOp)
+        dst.diagnosticOp = src.diagnosticOp;
+    };
+    auto isProvenWorkerPrivate = [&](Value memref) {
+      Value current = memref;
+      while (Operation *def = current.getDefiningOp()) {
+        if (acc::PrivateLocalOp privateLocal =
+                dyn_cast<acc::PrivateLocalOp>(def)) {
+          return getPrivateMemScope(getPrivatizeOp(
+                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+        }
+        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+          current = viewLike.getViewSource();
+          continue;
+        }
+        break;
+      }
+      return false;
+    };
+    auto hasUnsafeOwnEffects = [](Operation *op) {
+      if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
+        SmallVector<MemoryEffects::EffectInstance> effects;
+        effectOp.getEffects(effects);
+        return llvm::any_of(effects, [](const auto &effect) {
+          return !isa<MemoryEffects::Read>(effect.getEffect());
+        });
+      }
+      return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
+    };
+    auto analyzeThreadYBroadening =
+        [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
+      ThreadYBroadeningInfo info;
+      auto classifyCombine = [&](Operation *combineOp, Value src,
+                                 ArrayRef<GPUParallelDimAttr> parDims) {
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        if (hasThreadY && isProvenWorkerPrivate(src)) {
+          info.hasActiveWorkerCombine = true;
+        } else {
+          info.hasExplicitInactiveCombine = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = combineOp;
         }
-        requirement.active |= active;
-        requirement.inactive |= !active;
-        return success();
-      };
-      auto hasThreadYPrivateStorage = [&](Value memref) {
-        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
-        if (!privatize)
-          return false;
-        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
-                 return parDim.isThreadY();
-               });
-      };
-      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
-        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
       };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
                 dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
-          FailureOr<ThreadYRequirement> nestedRequirement =
+          ThreadYBroadeningInfo nestedInfo =
               self(self, nestedPredicate.getRegion().front());
-          if (failed(nestedRequirement))
-            return failure();
-          // An active descendant must not be excluded by this region's
-          // predicate. Inactive descendants apply their own predicate.
-          if (nestedRequirement->active &&
-              failed(requireThreadY(&nestedOp, /*active=*/true)))
-            return failure();
+          // A worker-active descendant must reach its own predicate. Other
+          // requirements are enforced within the nested predicate region.
+          info.hasActiveWorkerCombine |= nestedInfo.hasActiveWorkerCombine;
           continue;
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(
-                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
-            return failure();
+          classifyCombine(combineOp, combineOp.getSrcMemref(),
+                          getReductionCombineParDims(combineOp));
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(combineRegionOp,
-                                             combineRegionOp.getSrcVar())))
-            return failure();
-          continue;
-        }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (failed(requireThreadY(
-                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
-            return failure();
-          continue;
-        }
-        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
-                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
-          bool threadYActive =
-              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
-                           [](auto parDim) { return parDim.isThreadY(); });
-          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
-            return failure();
+          classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
+                          getReductionCombineParDims(combineRegionOp));
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          bool hasOwnEffects = true;
-          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
-            hasOwnEffects = !effectOp.hasNoEffect();
-          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
-            hasOwnEffects = false;
+          if (hasUnsafeOwnEffects(&nestedOp)) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = &nestedOp;
           }
-          if (hasOwnEffects &&
-              failed(requireThreadY(&nestedOp, /*active=*/false)))
-            return failure();
           for (Region &region : nestedOp.getRegions()) {
-            for (Block &nestedBlock : region) {
-              FailureOr<ThreadYRequirement> nestedRequirement =
-                  self(self, nestedBlock);
-              if (failed(nestedRequirement))
-                return failure();
-              // Other region-bearing operations do not introduce independent
-              // ACC predication, so both requirements apply to this region.
-              if (nestedRequirement->active &&
-                  failed(requireThreadY(&nestedOp, /*active=*/true)))
-                return failure();
-              if (nestedRequirement->inactive &&
-                  failed(requireThreadY(&nestedOp, /*active=*/false)))
-                return failure();
-            }
+            for (Block &nestedBlock : region)
+              mergeBroadeningInfo(info, self(self, nestedBlock));
           }
           continue;
         }
-        if (!isMemoryEffectFree(&nestedOp) &&
-            failed(requireThreadY(&nestedOp, /*active=*/false)))
-          return failure();
+        if (hasUnsafeOwnEffects(&nestedOp)) {
+          info.hasBroadeningConflict = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = &nestedOp;
+        }
       }
-      return requirement;
+      return info;
     };
 
-    FailureOr<ThreadYRequirement> threadYRequirement =
-        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
-    if (failed(threadYRequirement))
-      return {};
+    ThreadYBroadeningInfo threadYInfo =
+        analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
     auto applyCombineParDims =
         [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (!parDim.isThreadY())
-              mlir::acc::removeParDim(ancestorParDims, parDim);
-          }
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
+            mlir::acc::removeParDim(ancestorParDims, parDim);
           return success();
         };
     block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (acc::PrivatizeOp privatizeOp =
-                getPrivatizeForMemref(storeOp.getMemref())) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
@@ -1423,13 +1407,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (threadYRequirement->active || threadYRequirement->inactive) {
-      mlir::acc::GPUParallelDimAttr threadY =
-          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (threadYRequirement->active)
-        mlir::acc::insertParDim(ancestorParDims, threadY);
-      else
-        mlir::acc::removeParDim(ancestorParDims, threadY);
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
+    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+      if (threadYInfo.hasExplicitInactiveCombine ||
+          threadYInfo.hasBroadeningConflict) {
+        Operation *diagnosticOp =
+            threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
+        (void)accSupport.emitNYI(
+            diagnosticOp->getLoc(),
+            "operations in the same predicate region require incompatible "
+            "ThreadY predication");
+        hasFailed = true;
+        return {};
+      }
+      mlir::acc::insertParDim(ancestorParDims, threadY);
     }
   }
 
@@ -1886,22 +1879,18 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  Value current = memref;
-  while (Operation *def = current.getDefiningOp()) {
-    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
-      return privateLocal;
-    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-      current = viewLike.getViewSource();
+  llvm::SmallVector<Value, 8> worklist{memref};
+  llvm::SmallPtrSet<Value, 8> seen;
+  while (!worklist.empty()) {
+    Value v = worklist.pop_back_val();
+    if (!seen.insert(v).second)
       continue;
-    }
-    if (acc::PartialEntityAccessOpInterface partialAccess =
-            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
-      current = partialAccess.getBaseEntity();
+    Operation *def = v.getDefiningOp();
+    if (!def)
       continue;
-    }
-    // Do not follow arbitrary operands: multi-source operations such as
-    // arith.select do not prove that their result aliases private storage.
-    return nullptr;
+    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
+      return privateLocal;
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index ba8f16d3966df..0de030eda1e59 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -184,80 +185,3 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
-
-// CHECK-LABEL: func.func @aliased_worker_store
-// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
-// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[ALIAS_TX_ZERO]]
-// CHECK: memref.store
-// CHECK: acc.atomic.update
-func.func @aliased_worker_store(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
-// CHECK-LABEL: func.func @aliased_thread_store
-// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
-// CHECK: memref.store
-func.func @aliased_thread_store() {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private)
-        : (!acc.private_type<memref<i32>>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}

>From 30926fc3c3e3a6d6b9fe8e3fcb8442319849edf1 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:35 -0700
Subject: [PATCH 08/20] [OpenACC][test] Match broadening conflict locations

Expect diagnostics on the side effect that makes worker broadening unsafe.
---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 1275e4c77890a..7f22fd20aa89e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -69,8 +69,8 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          memref.store %c7_i32, %selected[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -103,12 +103,12 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
         acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.atomic.update %result_arg : memref<i32> {
           ^bb0(%current: i32):
             %next = arith.addi %current, %c1_i32 : i32
             acc.yield %next : i32
           }
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 55293fcdfa6ce6f49e61e43293e80df04bfec368 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:51:30 -0700
Subject: [PATCH 09/20] [OpenACC] Require safe predicate speculation

Treat reads and non-speculatable operations as broadening conflicts so enabling worker rows cannot introduce invalid memory accesses.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp  | 13 +++++--------
 ...to-gpu-worker-reduction-combine-mixed-scope.mlir |  6 ++----
 .../acc-cg-to-gpu-worker-reduction-combine.mlir     |  1 -
 3 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 50635dde561db..250c2be09df85 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1276,13 +1276,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return false;
     };
-    auto hasUnsafeOwnEffects = [](Operation *op) {
+    auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
-        SmallVector<MemoryEffects::EffectInstance> effects;
-        effectOp.getEffects(effects);
-        return llvm::any_of(effects, [](const auto &effect) {
-          return !isa<MemoryEffects::Read>(effect.getEffect());
-        });
+        return !effectOp.hasNoEffect();
       }
       return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
     };
@@ -1325,7 +1321,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          if (hasUnsafeOwnEffects(&nestedOp)) {
+          if (hasOwnEffects(&nestedOp)) {
             info.hasBroadeningConflict = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
@@ -1336,7 +1332,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
           continue;
         }
-        if (hasUnsafeOwnEffects(&nestedOp)) {
+        if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
+            !isPure(&nestedOp)) {
           info.hasBroadeningConflict = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 7f22fd20aa89e..290786ab7c1ce 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,7 +42,7 @@ func.func @mixed_scope_worker_reduction_combine(
   return
 }
 
-func.func @worker_combine_with_single_store(%result: memref<i32>) {
+func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -59,18 +59,16 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
-      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          memref.store %c7_i32, %selected[] : memref<i32>
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 0de030eda1e59..fb1a0256d2a38 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 300b3dc6a9334c5bd8b47daf1c1371bb6f5db806 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:59:06 -0700
Subject: [PATCH 10/20] [OpenACC] Reject worker combine regions

Keep generic combine regions single-row because their non-atomic load/store lowering is unsafe across worker rows.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 14 ++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 37 ++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 48 -------------------
 3 files changed, 48 insertions(+), 51 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 250c2be09df85..3697ae7405c9f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1286,11 +1286,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
       auto classifyCombine = [&](Operation *combineOp, Value src,
-                                 ArrayRef<GPUParallelDimAttr> parDims) {
+                                 ArrayRef<GPUParallelDimAttr> parDims,
+                                 bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
         if (hasThreadY && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
+          if (!supportsWorkerBroadening) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = combineOp;
+          }
         } else {
           info.hasExplicitInactiveCombine = true;
           if (!info.diagnosticOp)
@@ -1311,13 +1317,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
           classifyCombine(combineOp, combineOp.getSrcMemref(),
-                          getReductionCombineParDims(combineOp));
+                          getReductionCombineParDims(combineOp),
+                          /*supportsWorkerBroadening=*/true);
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
           classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
-                          getReductionCombineParDims(combineRegionOp));
+                          getReductionCombineParDims(combineRegionOp),
+                          /*supportsWorkerBroadening=*/false);
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 290786ab7c1ce..fffbfcb461c56 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -117,3 +117,40 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fb1a0256d2a38..686e8fd233980 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,54 +46,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// CHECK-LABEL: func.func @worker_reduction_combine_region
-// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
-// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[REGION_IS_X_ZERO]]
-// CHECK: arith.addi
-
-func.func @worker_reduction_combine_region(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
-          memref.store %c0_i32, %local[] : memref<i32>
-          scf.reduce
-        } {acc.par_dims = #acc<par_dims[thread_y]>}
-        acc.predicate_region {
-          acc.reduction_combine_region %local into %result_arg : memref<i32> {
-            %lhs = memref.load %result_arg[] : memref<i32>
-            %rhs = memref.load %local[] : memref<i32>
-            %sum = arith.addi %lhs, %rhs : i32
-            memref.store %sum, %result_arg[] : memref<i32>
-            acc.yield
-          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.

>From cbc18adba26e38ff6bebd01bf252112da4412887 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:03:05 -0700
Subject: [PATCH 11/20] [OpenACC] Require atomic worker combines

Broaden ThreadY only when block dimensions select the atomic combine lowering; reject ThreadY-only shared updates.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  4 ++-
 ...-worker-reduction-combine-mixed-scope.mlir | 32 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 3697ae7405c9f..76f942a2730f9 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1290,7 +1290,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                                  bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
-        if (hasThreadY && isProvenWorkerPrivate(src)) {
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
             info.hasBroadeningConflict = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index fffbfcb461c56..5758702a3b92e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -154,3 +154,35 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2f6b6b2e74b62114ad20366b768685e6af8bc000 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:06:26 -0700
Subject: [PATCH 12/20] [OpenACC][test] Check non-atomic worker predication

Verify ThreadY-only combines retain single-row predication while unsafe combine regions remain rejected.
---
 ...-worker-reduction-combine-mixed-scope.mlir | 31 ---------------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 39 +++++++++++++++++++
 2 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 5758702a3b92e..3d15b4ca4a59f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -155,34 +155,3 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   return
 }
 
-func.func @thread_y_only_worker_combine(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        acc.predicate_region {
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 686e8fd233980..901b23b25316f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -136,3 +136,42 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
+
+// Without a block dimension the combine lowers to a non-atomic update, so
+// ThreadY must retain the legacy single-row predicate.
+// CHECK-LABEL: func.func @thread_y_only_worker_combine
+// CHECK: gpu.launch {{.*}} threads([[ONLY_TX:%[^,]+]], [[ONLY_TY:%[^,]+]],
+// CHECK: %[[ONLY_TY_ZERO:.*]] = arith.cmpi eq, [[ONLY_TY]],
+// CHECK: %[[ONLY_TX_ZERO:.*]] = arith.cmpi eq, [[ONLY_TX]],
+// CHECK: %[[ONLY_ROW_ZERO:.*]] = arith.andi %[[ONLY_TX_ZERO]], %[[ONLY_TY_ZERO]]
+// CHECK: scf.if %[[ONLY_ROW_ZERO]]
+// CHECK: memref.store
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 88026c11dfb0376be0ad438de4a530d99d821119 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:10:52 -0700
Subject: [PATCH 13/20] [OpenACC][test] Remove trailing blank line

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 3d15b4ca4a59f..dc634cff25b07 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -41,7 +41,6 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
-
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 984e8f2ce4366cb23a6104b6aa965783f2380a49 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:11:22 -0700
Subject: [PATCH 14/20] [OpenACC][test] Normalize test file ending

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dc634cff25b07..dba8057d48e95 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -78,7 +78,6 @@ func.func @worker_combine_with_read(%result: memref<i32>) {
   }
   return
 }
-
 func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 8f6c707f4e58523929b8482505946acdd9dc9991 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:12:19 -0700
Subject: [PATCH 15/20] [OpenACC][test] Finish file ending cleanup

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir    | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dba8057d48e95..a641fe74867e4 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -151,5 +151,4 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
     } {origin = "acc.parallel"}
   }
   return
-}
-
+}
\ No newline at end of file

>From f629ee527f4b56ac33d85867d488a793af7cfc09 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:43:52 -0700
Subject: [PATCH 16/20] [OpenACC] Localize worker combine classification

Keep private-memory scope behavior unchanged and remove redundant failure plumbing from combine dimension handling.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 37 ++++++++++---------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 76f942a2730f9..98258dc622e3c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1265,8 +1265,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       while (Operation *def = current.getDefiningOp()) {
         if (acc::PrivateLocalOp privateLocal =
                 dyn_cast<acc::PrivateLocalOp>(def)) {
-          return getPrivateMemScope(getPrivatizeOp(
-                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+          acc::PrivatizeOp privatize =
+              getPrivatizeOp(privateLocal, computeRegion);
+          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+          if (!parDims)
+            return false;
+          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadX();
+          });
+          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadY();
+          });
+          return hasThreadY && !hasThreadX;
         }
         if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
           current = viewLike.getViewSource();
@@ -1355,13 +1365,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     ThreadYBroadeningInfo threadYInfo =
         analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
-    auto applyCombineParDims =
-        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
-            mlir::acc::removeParDim(ancestorParDims, parDim);
-          return success();
-        };
-    block->walk([&](Operation *op) -> WalkResult {
+    block->walk([&](Operation *op) {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1392,15 +1396,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(reductionCombineOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(combineRegionOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1412,7 +1416,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           mlir::acc::insertParDim(ancestorParDims, parDim);
         }
       }
-      return WalkResult::advance();
     });
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
@@ -1877,7 +1880,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasThreadY)
+  if (hasBlock && hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;

>From edc8abd53cd69917a03f21c72c975e4a26a4ed24 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:50:05 -0700
Subject: [PATCH 17/20] [OpenACC] Merge ThreadY conflict state

Use one flag for all operations that require the legacy inactive ThreadY predicate.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98258dc622e3c..6a5aebc4db323 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,15 +1248,13 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
-      bool hasExplicitInactiveCombine = false;
-      bool hasBroadeningConflict = false;
+      bool requiresInactiveThreadY = false;
       Operation *diagnosticOp = nullptr;
     };
     auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
                                   const ThreadYBroadeningInfo &src) {
       dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
-      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
-      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      dst.requiresInactiveThreadY |= src.requiresInactiveThreadY;
       if (!dst.diagnosticOp)
         dst.diagnosticOp = src.diagnosticOp;
     };
@@ -1305,12 +1303,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = combineOp;
           }
         } else {
-          info.hasExplicitInactiveCombine = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1342,7 +1340,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
           }
@@ -1354,7 +1352,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
             !isPure(&nestedOp)) {
-          info.hasBroadeningConflict = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
         }
@@ -1421,8 +1419,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
     bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
-      if (threadYInfo.hasExplicitInactiveCombine ||
-          threadYInfo.hasBroadeningConflict) {
+      if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
         (void)accSupport.emitNYI(

>From 90cbc61e638708c7e25e37e588e20329fbfb736e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:06:50 -0700
Subject: [PATCH 18/20] [OpenACC] Stabilize ThreadY conflict analysis

Capture structural ThreadY activity before operation-order mutations and cover a trailing worker-private store.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   |  6 +++---
 ...pu-worker-reduction-combine-mixed-scope.mlir |  1 +
 .../acc-cg-to-gpu-worker-reduction-combine.mlir | 17 +++++------------
 3 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6a5aebc4db323..09dbbf892ffbd 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,6 +1246,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1415,9 +1418,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    mlir::acc::GPUParallelDimAttr threadY =
-        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a641fe74867e4..a5ffb4bfb59f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -33,6 +33,7 @@ func.func @mixed_scope_worker_reduction_combine(
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
+          memref.store %c0_i32, %local[] : memref<i32>
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 901b23b25316f..4466ad74d1a91 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,17 +46,15 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// Nested predicate regions choose their ThreadY predicates independently. The
-// outer region must keep ThreadY active so it does not exclude worker rows
-// before the nested worker-private combine is reached.
+// The outer predicate must keep ThreadY active so it does not exclude worker
+// rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines
 // CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
 // CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
 // CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
 // CHECK-NOT: arith.andi
 // CHECK: scf.if %[[NESTED_TX_ZERO]]
-func.func @nested_worker_reduction_combines(
-    %other: memref<i32>, %result: memref<i32>) {
+func.func @nested_worker_reduction_combines(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -67,9 +65,8 @@ func.func @nested_worker_reduction_combines(
     %private = acc.privatize [#acc<par_dims[thread_y]>]
         : () -> !acc.private_type<memref<i32>>
     acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %other_arg = %other,
-            %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c0_i32 = arith.constant 0 : i32
@@ -85,10 +82,6 @@ func.func @nested_worker_reduction_combines(
             acc.reduction_combine %local into %result_arg <add> : memref<i32>
                 {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           }
-          acc.predicate_region {
-            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          }
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}

>From 9e6fcfb6dd7ff5d68e1e7e6a34b14a7a69763286 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:18 -0700
Subject: [PATCH 19/20] [OpenACC] Allow worker-private stores during combine

Treat proven worker-private stores as row-local so they can share the broadened ThreadY predicate safely.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp            | 4 ++++
 .../OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir       | 1 +
 2 files changed, 5 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 09dbbf892ffbd..f78a973da4e5a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1341,6 +1341,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          if (isProvenWorkerPrivate(storeOp.getMemref()))
+            continue;
+        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 4466ad74d1a91..cbfabf28ea891 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From fcc26cd10cf22d3fb30b46154828ecd7e4af7604 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:43:59 -0700
Subject: [PATCH 20/20] [OpenACC] Diagnose unproven worker aliases

Restrict ThreadY broadening to directly sized worker storage and conservatively reject aliases that may originate from it.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 73 +++++++++++--------
 ...-worker-reduction-combine-mixed-scope.mlir | 39 ++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 41 ++++++++++-
 3 files changed, 123 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f78a973da4e5a..f46a891f08652 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,7 +1248,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1262,30 +1261,46 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         dst.diagnosticOp = src.diagnosticOp;
     };
     auto isProvenWorkerPrivate = [&](Value memref) {
-      Value current = memref;
-      while (Operation *def = current.getDefiningOp()) {
-        if (acc::PrivateLocalOp privateLocal =
-                dyn_cast<acc::PrivateLocalOp>(def)) {
-          acc::PrivatizeOp privatize =
-              getPrivatizeOp(privateLocal, computeRegion);
-          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-          if (!parDims)
-            return false;
-          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadX();
-          });
-          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadY();
-          });
-          return hasThreadY && !hasThreadX;
-        }
-        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-          current = viewLike.getViewSource();
-          continue;
-        }
-        break;
+      acc::PrivateLocalOp privateLocal =
+          memref.getDefiningOp<acc::PrivateLocalOp>();
+      if (!privateLocal)
+        return false;
+      acc::PrivatizeOp privatize = getPrivatizeOp(privateLocal, computeRegion);
+      GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+      if (!parDims)
+        return false;
+      bool hasThreadX = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadX(); });
+      bool hasThreadY = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadY(); });
+      return hasThreadY && !hasThreadX;
+    };
+    auto hasWorkerPrivateOrigin = [&](auto &&self, Value value,
+                                      DenseSet<Value> &visited) -> bool {
+      if (!visited.insert(value).second)
+        return false;
+      if (isProvenWorkerPrivate(value))
+        return true;
+      Operation *def = value.getDefiningOp();
+      if (!def) {
+        auto blockArg = dyn_cast<BlockArgument>(value);
+        def = blockArg ? blockArg.getOwner()->getParentOp() : nullptr;
       }
-      return false;
+      if (!def)
+        return false;
+      if (llvm::any_of(def->getOperands(), [&](Value operand) {
+            return self(self, operand, visited);
+          }))
+        return true;
+      return llvm::any_of(def->getRegions(), [&](Region &region) {
+        return llvm::any_of(region, [&](Block &block) {
+          Operation *terminator = block.getTerminator();
+          return terminator &&
+                 llvm::any_of(terminator->getOperands(), [&](Value operand) {
+                   return self(self, operand, visited);
+                 });
+        });
+      });
     };
     auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
@@ -1312,6 +1327,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
         } else {
           info.requiresInactiveThreadY = true;
+          DenseSet<Value> visited;
+          if (hasThreadY && hasBlock &&
+              hasWorkerPrivateOrigin(hasWorkerPrivateOrigin, src, visited))
+            info.hasActiveWorkerCombine = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1341,10 +1360,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (isProvenWorkerPrivate(storeOp.getMemref()))
-            continue;
-        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
@@ -1422,7 +1437,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+    if (threadYInfo.hasActiveWorkerCombine) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a5ffb4bfb59f8..8724f1f57bc9b 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,6 +42,45 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_unproven_alias(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %alias = scf.if %true -> (memref<i32>) {
+          scf.yield %local : memref<i32>
+        } else {
+          scf.yield %local : memref<i32>
+        }
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %alias into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index cbfabf28ea891..fd4e12fd039a8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -47,6 +46,46 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
+// CHECK-LABEL: func.func @worker_combine_in_thread_y_scope
+// CHECK: gpu.launch {{.*}} threads([[SCOPE_TX:%[^,]+]], [[SCOPE_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[SCOPE_TY]]
+// CHECK: %[[SCOPE_TX_ZERO:.*]] = arith.cmpi eq, [[SCOPE_TX]],
+// CHECK: scf.if %[[SCOPE_TX_ZERO]]
+func.func @worker_combine_in_thread_y_scope(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 // The outer predicate must keep ThreadY active so it does not exclude worker
 // rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines


From llvmlistbot at llvm.org  Mon Jul 20 20:58:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 20:58:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Preserve worker rows when
 combining reductions (PR #210804)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210804@github.com>
Message-ID: <6a5eee4a.6c8668f2.1f24.3658@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210804

>From 4c316621a473f21607795bafbd1ffa51a9dabe93 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 13:12:09 -0700
Subject: [PATCH 01/21] [OpenACC] Preserve worker rows when combining
 reductions

Keep ThreadY active for worker-private sources so every worker row contributes, and reject mixed scopes that require incompatible predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 53 +++++++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 43 +++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 95 +++++++++++++++++++
 3 files changed, 181 insertions(+), 10 deletions(-)
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 24aeed6e23bda..6be79b0dd40e4 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,7 +1246,30 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    block->walk([&](Operation *op) {
+    std::optional<bool> combineThreadYActive;
+    auto applyCombineParDims =
+        [&](Operation *combineOp, Value src,
+            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+          bool isWorkerPrivate =
+              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
+            if (parDim.isThreadY()) {
+              if (combineThreadYActive &&
+                  *combineThreadYActive != isWorkerPrivate) {
+                combineOp->emitError()
+                    << "mixed worker-private and non-worker-private reduction "
+                       "combines require incompatible ThreadY predication";
+                hasFailed = true;
+                return failure();
+              }
+              combineThreadYActive = isWorkerPrivate;
+            } else {
+              mlir::acc::removeParDim(ancestorParDims, parDim);
+            }
+          }
+          return success();
+        };
+    block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1277,17 +1300,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(reductionCombineOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                reductionCombineOp, reductionCombineOp.getSrcMemref(),
+                getReductionCombineParDims(reductionCombineOp))))
+          return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(combineRegionOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                combineRegionOp, combineRegionOp.getSrcVar(),
+                getReductionCombineParDims(combineRegionOp))))
+          return WalkResult::interrupt();
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1301,6 +1324,14 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
+    if (combineThreadYActive) {
+      mlir::acc::GPUParallelDimAttr threadY =
+          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+      if (*combineThreadYActive)
+        mlir::acc::insertParDim(ancestorParDims, threadY);
+      else
+        mlir::acc::removeParDim(ancestorParDims, threadY);
+    }
   }
 
   // Obtain launch dimensions
@@ -1747,7 +1778,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasBlock && hasThreadY)
+  if (hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;
@@ -1871,6 +1902,8 @@ void ACCCGToGPULowering::processPredicateRegion(
             SmallVector<mlir::acc::GPUParallelDimAttr>>
       parDimsPair = computeActiveAndInactiveParDims(
           interOp, &interOp.getRegion().front());
+  if (hasFailed)
+    return;
 
   // If ThreadY reduction exists, subgroup alignment is applied
   // (blockDim.x = subgroupSize), so ThreadX lanes exist even without explicit
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
new file mode 100644
index 0000000000000..a0d381740c034
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -0,0 +1,43 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" \
+// RUN:   -verify-diagnostics
+
+func.func @mixed_scope_worker_reduction_combine(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
new file mode 100644
index 0000000000000..b2fd246d33d1b
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -0,0 +1,95 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A worker-private reduction has one shared slot per ThreadY row. The combine
+// must keep ThreadY active and predicate only ThreadX.
+
+// CHECK-LABEL: func.func @worker_reduction_combine
+// CHECK: gpu.launch {{.*}} threads([[TID_X:%[^,]+]], [[TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[TID_Y]]
+// CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: acc.atomic.update
+
+func.func @worker_reduction_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @worker_reduction_combine_region
+// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
+// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[REGION_IS_X_ZERO]]
+// CHECK: arith.addi
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From d763e8065946687750b2c78fcf5393167b7de962 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 15:08:39 -0700
Subject: [PATCH 02/21] [OpenACC] Report mixed reduction scopes as NYI

Use the standard OpenACC unsupported-feature diagnostic for incompatible ThreadY predication.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6be79b0dd40e4..378fa17b24850 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1256,9 +1256,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             if (parDim.isThreadY()) {
               if (combineThreadYActive &&
                   *combineThreadYActive != isWorkerPrivate) {
-                combineOp->emitError()
-                    << "mixed worker-private and non-worker-private reduction "
-                       "combines require incompatible ThreadY predication";
+                (void)accSupport.emitNYI(
+                    combineOp->getLoc(),
+                    "mixed worker-private and non-worker-private reduction "
+                    "combines require incompatible ThreadY predication");
                 hasFailed = true;
                 return failure();
               }

>From 1fe76c2f1ea14bd036598286c2d9c0ea8d2291ef Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 16:50:59 -0700
Subject: [PATCH 03/21] [OpenACC] Isolate ThreadY predicate requirements

Propagate nested worker requirements without broadening unrelated side effects, and reject incompatible operations sharing one predicate region.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 126 +++++++++++++++---
 ...-worker-reduction-combine-mixed-scope.mlir |  40 +++++-
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  46 +++++++
 3 files changed, 189 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 378fa17b24850..34131dc3753a3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,27 +1246,111 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    std::optional<bool> combineThreadYActive;
+    struct ThreadYRequirement {
+      bool active = false;
+      bool inactive = false;
+    };
+    auto analyzeThreadYRequirements =
+        [&](auto &&self,
+            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
+      ThreadYRequirement requirement;
+      auto requireThreadY = [&](Operation *op, bool active) {
+        if ((active && requirement.inactive) ||
+            (!active && requirement.active)) {
+          (void)accSupport.emitNYI(
+              op->getLoc(),
+              "operations in the same predicate region require incompatible "
+              "ThreadY predication");
+          hasFailed = true;
+          return failure();
+        }
+        requirement.active |= active;
+        requirement.inactive |= !active;
+        return success();
+      };
+      auto applyCombineRequirement =
+          [&](Operation *combineOp, Value src,
+              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+            if (llvm::none_of(combineParDims,
+                              [](auto parDim) { return parDim.isThreadY(); }))
+              return success();
+            bool isWorkerPrivate =
+                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+            return requireThreadY(combineOp, isWorkerPrivate);
+          };
+
+      for (Operation &nestedOp : predicateBlock) {
+        if (acc::PredicateRegionOp nestedPredicate =
+                dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
+          FailureOr<ThreadYRequirement> nestedRequirement =
+              self(self, nestedPredicate.getRegion().front());
+          if (failed(nestedRequirement))
+            return failure();
+          // An active descendant must not be excluded by this region's
+          // predicate. Inactive descendants apply their own predicate.
+          if (nestedRequirement->active &&
+              failed(requireThreadY(&nestedOp, /*active=*/true)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineOp combineOp =
+                dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineOp, combineOp.getSrcMemref(),
+                  getReductionCombineParDims(combineOp))))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineRegionOp combineRegionOp =
+                dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineRegionOp, combineRegionOp.getSrcVar(),
+                  getReductionCombineParDims(combineRegionOp))))
+            return failure();
+          continue;
+        }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          bool threadYActive = false;
+          if (acc::PrivateLocalOp privateLocal =
+                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+            if (acc::GPUParallelDimsAttr parDims =
+                    getPrivatizeOp(privateLocal, computeRegion)
+                        .getParDimsAttr()) {
+              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
+                return parDim.isThreadY();
+              });
+            }
+          }
+          if (failed(requireThreadY(storeOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
+                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
+          bool threadYActive =
+              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
+                           [](auto parDim) { return parDim.isThreadY(); });
+          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (!isMemoryEffectFree(&nestedOp) &&
+            failed(requireThreadY(&nestedOp, /*active=*/false)))
+          return failure();
+      }
+      return requirement;
+    };
+
+    FailureOr<ThreadYRequirement> threadYRequirement =
+        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
+    if (failed(threadYRequirement))
+      return {};
+
     auto applyCombineParDims =
-        [&](Operation *combineOp, Value src,
-            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          bool isWorkerPrivate =
-              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
           for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (parDim.isThreadY()) {
-              if (combineThreadYActive &&
-                  *combineThreadYActive != isWorkerPrivate) {
-                (void)accSupport.emitNYI(
-                    combineOp->getLoc(),
-                    "mixed worker-private and non-worker-private reduction "
-                    "combines require incompatible ThreadY predication");
-                hasFailed = true;
-                return failure();
-              }
-              combineThreadYActive = isWorkerPrivate;
-            } else {
+            if (!parDim.isThreadY())
               mlir::acc::removeParDim(ancestorParDims, parDim);
-            }
           }
           return success();
         };
@@ -1302,14 +1386,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
         if (failed(applyCombineParDims(
-                reductionCombineOp, reductionCombineOp.getSrcMemref(),
                 getReductionCombineParDims(reductionCombineOp))))
           return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
         if (failed(applyCombineParDims(
-                combineRegionOp, combineRegionOp.getSrcVar(),
                 getReductionCombineParDims(combineRegionOp))))
           return WalkResult::interrupt();
       }
@@ -1325,10 +1407,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (combineThreadYActive) {
+    if (threadYRequirement->active || threadYRequirement->inactive) {
       mlir::acc::GPUParallelDimAttr threadY =
           mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (*combineThreadYActive)
+      if (threadYRequirement->active)
         mlir::acc::insertParDim(ancestorParDims, threadY);
       else
         mlir::acc::removeParDim(ancestorParDims, threadY);
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a0d381740c034..e80b5afa26d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -30,7 +30,7 @@ func.func @mixed_scope_worker_reduction_combine(
         acc.predicate_region {
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -41,3 +41,41 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_single_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c7_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          memref.store %c7_i32, %result_arg[] : memref<i32>
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index b2fd246d33d1b..afd6108d6e670 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -93,3 +93,49 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+// Nested predicate regions choose their ThreadY predicates independently. The
+// outer region must keep ThreadY active so it does not exclude worker rows
+// before the nested worker-private combine is reached.
+func.func @nested_worker_reduction_combines(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          acc.predicate_region {
+            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 477f10e0472195d6bb55598bf364a1655682ab6f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:04:51 -0700
Subject: [PATCH 04/21] [OpenACC] Propagate nested ThreadY requirements

Analyze control-flow regions recursively and recognize aliased worker-private stores so their worker rows remain active.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 32 ++++---
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 86 +++++++++++++++++++
 2 files changed, 107 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 34131dc3753a3..98c697f52668e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1310,17 +1310,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = false;
-          if (acc::PrivateLocalOp privateLocal =
-                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-            if (acc::GPUParallelDimsAttr parDims =
-                    getPrivatizeOp(privateLocal, computeRegion)
-                        .getParDimsAttr()) {
-              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
-                return parDim.isThreadY();
-              });
-            }
-          }
+          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
+                               PrivateMemScope::Worker;
           if (failed(requireThreadY(storeOp, threadYActive)))
             return failure();
           continue;
@@ -1334,6 +1325,25 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             return failure();
           continue;
         }
+        if (nestedOp.getNumRegions() != 0) {
+          for (Region &region : nestedOp.getRegions()) {
+            for (Block &nestedBlock : region) {
+              FailureOr<ThreadYRequirement> nestedRequirement =
+                  self(self, nestedBlock);
+              if (failed(nestedRequirement))
+                return failure();
+              // Other region-bearing operations do not introduce independent
+              // ACC predication, so both requirements apply to this region.
+              if (nestedRequirement->active &&
+                  failed(requireThreadY(&nestedOp, /*active=*/true)))
+                return failure();
+              if (nestedRequirement->inactive &&
+                  failed(requireThreadY(&nestedOp, /*active=*/false)))
+                return failure();
+            }
+          }
+          continue;
+        }
         if (!isMemoryEffectFree(&nestedOp) &&
             failed(requireThreadY(&nestedOp, /*active=*/false)))
           return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index afd6108d6e670..8a2dd6bd470f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -97,6 +97,12 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.
+// CHECK-LABEL: func.func @nested_worker_reduction_combines
+// CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
+// CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[NESTED_TX_ZERO]]
 func.func @nested_worker_reduction_combines(
     %other: memref<i32>, %result: memref<i32>) {
   %c1 = arith.constant 1 : index
@@ -139,3 +145,83 @@ func.func @nested_worker_reduction_combines(
   }
   return
 }
+
+// CHECK-LABEL: func.func @worker_combine_in_scf_if
+// CHECK: gpu.launch {{.*}} threads([[IF_TX:%[^,]+]], [[IF_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[IF_TY]]
+// CHECK: %[[IF_TX_ZERO:.*]] = arith.cmpi eq, [[IF_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IF_TX_ZERO]]
+func.func @worker_combine_in_scf_if(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          scf.if %true {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @aliased_worker_store
+// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
+// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[ALIAS_TX_ZERO]]
+// CHECK: memref.store
+// CHECK: acc.atomic.update
+func.func @aliased_worker_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2e234b5cb4dac8aa7cf1ab07b63984b7cb9f329f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:16:55 -0700
Subject: [PATCH 05/21] [OpenACC] Account for predicate operation effects

Preserve all privatized ThreadY dimensions through aliases and reject intrinsically effecting region operations that cannot share worker predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 33 +++++++++++-----
 ...-worker-reduction-combine-mixed-scope.mlir | 38 +++++++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 36 ++++++++++++++++++
 3 files changed, 97 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98c697f52668e..20e8977c907f3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1268,15 +1268,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         requirement.inactive |= !active;
         return success();
       };
+      auto hasThreadYPrivateStorage = [&](Value memref) {
+        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
+        if (!privatize)
+          return false;
+        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
+                 return parDim.isThreadY();
+               });
+      };
       auto applyCombineRequirement =
           [&](Operation *combineOp, Value src,
               ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
             if (llvm::none_of(combineParDims,
                               [](auto parDim) { return parDim.isThreadY(); }))
               return success();
-            bool isWorkerPrivate =
-                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
-            return requireThreadY(combineOp, isWorkerPrivate);
+            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
           };
 
       for (Operation &nestedOp : predicateBlock) {
@@ -1310,9 +1317,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
-                               PrivateMemScope::Worker;
-          if (failed(requireThreadY(storeOp, threadYActive)))
+          if (failed(requireThreadY(
+                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
             return failure();
           continue;
         }
@@ -1326,6 +1332,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
+          bool hasOwnEffects = true;
+          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
+            hasOwnEffects = !effectOp.hasNoEffect();
+          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
+            hasOwnEffects = false;
+          }
+          if (hasOwnEffects &&
+              failed(requireThreadY(&nestedOp, /*active=*/false)))
+            return failure();
           for (Region &region : nestedOp.getRegions()) {
             for (Block &nestedBlock : region) {
               FailureOr<ThreadYRequirement> nestedRequirement =
@@ -1368,10 +1383,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (auto privateLocalOp =
-                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-          acc::PrivatizeOp privatizeOp =
-              getPrivatizeOp(privateLocalOp, computeRegion);
+        if (acc::PrivatizeOp privatizeOp =
+                getPrivatizeForMemref(storeOp.getMemref())) {
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index e80b5afa26d20..19e1b3106d966 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -79,3 +79,41 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c1_i32 = arith.constant 1 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.atomic.update %result_arg : memref<i32> {
+          ^bb0(%current: i32):
+            %next = arith.addi %current, %c1_i32 : i32
+            acc.yield %next : i32
+          }
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 8a2dd6bd470f8..ba8f16d3966df 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -225,3 +225,39 @@ func.func @aliased_worker_store(%result: memref<i32>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @aliased_thread_store
+// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
+// CHECK: memref.store
+func.func @aliased_thread_store() {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private)
+        : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From ae0d329822ede9a6d80731ebd45abd7e520ef5ae Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:26:21 -0700
Subject: [PATCH 06/21] [OpenACC] Require proven private aliases

Follow only alias-preserving view operations and treat every non-private combine as ThreadY-inactive to avoid broadening ambiguous global effects.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 45 +++++++++----------
 ...-worker-reduction-combine-mixed-scope.mlir |  6 ++-
 2 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 20e8977c907f3..9e9114e9aa174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1277,14 +1277,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                  return parDim.isThreadY();
                });
       };
-      auto applyCombineRequirement =
-          [&](Operation *combineOp, Value src,
-              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-            if (llvm::none_of(combineParDims,
-                              [](auto parDim) { return parDim.isThreadY(); }))
-              return success();
-            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
-          };
+      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
+        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
+      };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
@@ -1302,17 +1297,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineOp, combineOp.getSrcMemref(),
-                  getReductionCombineParDims(combineOp))))
+          if (failed(
+                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
             return failure();
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineRegionOp, combineRegionOp.getSrcVar(),
-                  getReductionCombineParDims(combineRegionOp))))
+          if (failed(applyCombineRequirement(combineRegionOp,
+                                             combineRegionOp.getSrcVar())))
             return failure();
           continue;
         }
@@ -1893,18 +1886,22 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  llvm::SmallVector<Value, 8> worklist{memref};
-  llvm::SmallPtrSet<Value, 8> seen;
-  while (!worklist.empty()) {
-    Value v = worklist.pop_back_val();
-    if (!seen.insert(v).second)
-      continue;
-    Operation *def = v.getDefiningOp();
-    if (!def)
-      continue;
+  Value current = memref;
+  while (Operation *def = current.getDefiningOp()) {
     if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
       return privateLocal;
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+      current = viewLike.getViewSource();
+      continue;
+    }
+    if (acc::PartialEntityAccessOpInterface partialAccess =
+            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
+      current = partialAccess.getBaseEntity();
+      continue;
+    }
+    // Do not follow arbitrary operands: multi-source operations such as
+    // arith.select do not prove that their result aliases private storage.
+    return nullptr;
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 19e1b3106d966..1275e4c77890a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -32,7 +32,7 @@ func.func @mixed_scope_worker_reduction_combine(
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+              {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
@@ -59,15 +59,17 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
+      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %result_arg[] : memref<i32>
+          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}

>From 3ed1de6f3b95aee6516db811bbd62dc4c54f9335 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:38:25 -0700
Subject: [PATCH 07/21] [OpenACC] Restrict ThreadY predicate broadening

Preserve legacy predicates unless a proven worker combine requires ThreadY, and reject broadening around writes, unknown effects, or incompatible combines.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 223 +++++++++---------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  78 +-----
 2 files changed, 107 insertions(+), 194 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 9e9114e9aa174..50635dde561db 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,138 +1246,122 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    struct ThreadYRequirement {
-      bool active = false;
-      bool inactive = false;
+    struct ThreadYBroadeningInfo {
+      bool hasActiveWorkerCombine = false;
+      bool hasExplicitInactiveCombine = false;
+      bool hasBroadeningConflict = false;
+      Operation *diagnosticOp = nullptr;
     };
-    auto analyzeThreadYRequirements =
-        [&](auto &&self,
-            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
-      ThreadYRequirement requirement;
-      auto requireThreadY = [&](Operation *op, bool active) {
-        if ((active && requirement.inactive) ||
-            (!active && requirement.active)) {
-          (void)accSupport.emitNYI(
-              op->getLoc(),
-              "operations in the same predicate region require incompatible "
-              "ThreadY predication");
-          hasFailed = true;
-          return failure();
+    auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
+                                  const ThreadYBroadeningInfo &src) {
+      dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
+      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
+      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      if (!dst.diagnosticOp)
+        dst.diagnosticOp = src.diagnosticOp;
+    };
+    auto isProvenWorkerPrivate = [&](Value memref) {
+      Value current = memref;
+      while (Operation *def = current.getDefiningOp()) {
+        if (acc::PrivateLocalOp privateLocal =
+                dyn_cast<acc::PrivateLocalOp>(def)) {
+          return getPrivateMemScope(getPrivatizeOp(
+                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+        }
+        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+          current = viewLike.getViewSource();
+          continue;
+        }
+        break;
+      }
+      return false;
+    };
+    auto hasUnsafeOwnEffects = [](Operation *op) {
+      if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
+        SmallVector<MemoryEffects::EffectInstance> effects;
+        effectOp.getEffects(effects);
+        return llvm::any_of(effects, [](const auto &effect) {
+          return !isa<MemoryEffects::Read>(effect.getEffect());
+        });
+      }
+      return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
+    };
+    auto analyzeThreadYBroadening =
+        [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
+      ThreadYBroadeningInfo info;
+      auto classifyCombine = [&](Operation *combineOp, Value src,
+                                 ArrayRef<GPUParallelDimAttr> parDims) {
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        if (hasThreadY && isProvenWorkerPrivate(src)) {
+          info.hasActiveWorkerCombine = true;
+        } else {
+          info.hasExplicitInactiveCombine = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = combineOp;
         }
-        requirement.active |= active;
-        requirement.inactive |= !active;
-        return success();
-      };
-      auto hasThreadYPrivateStorage = [&](Value memref) {
-        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
-        if (!privatize)
-          return false;
-        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
-                 return parDim.isThreadY();
-               });
-      };
-      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
-        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
       };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
                 dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
-          FailureOr<ThreadYRequirement> nestedRequirement =
+          ThreadYBroadeningInfo nestedInfo =
               self(self, nestedPredicate.getRegion().front());
-          if (failed(nestedRequirement))
-            return failure();
-          // An active descendant must not be excluded by this region's
-          // predicate. Inactive descendants apply their own predicate.
-          if (nestedRequirement->active &&
-              failed(requireThreadY(&nestedOp, /*active=*/true)))
-            return failure();
+          // A worker-active descendant must reach its own predicate. Other
+          // requirements are enforced within the nested predicate region.
+          info.hasActiveWorkerCombine |= nestedInfo.hasActiveWorkerCombine;
           continue;
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(
-                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
-            return failure();
+          classifyCombine(combineOp, combineOp.getSrcMemref(),
+                          getReductionCombineParDims(combineOp));
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(combineRegionOp,
-                                             combineRegionOp.getSrcVar())))
-            return failure();
-          continue;
-        }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (failed(requireThreadY(
-                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
-            return failure();
-          continue;
-        }
-        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
-                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
-          bool threadYActive =
-              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
-                           [](auto parDim) { return parDim.isThreadY(); });
-          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
-            return failure();
+          classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
+                          getReductionCombineParDims(combineRegionOp));
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          bool hasOwnEffects = true;
-          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
-            hasOwnEffects = !effectOp.hasNoEffect();
-          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
-            hasOwnEffects = false;
+          if (hasUnsafeOwnEffects(&nestedOp)) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = &nestedOp;
           }
-          if (hasOwnEffects &&
-              failed(requireThreadY(&nestedOp, /*active=*/false)))
-            return failure();
           for (Region &region : nestedOp.getRegions()) {
-            for (Block &nestedBlock : region) {
-              FailureOr<ThreadYRequirement> nestedRequirement =
-                  self(self, nestedBlock);
-              if (failed(nestedRequirement))
-                return failure();
-              // Other region-bearing operations do not introduce independent
-              // ACC predication, so both requirements apply to this region.
-              if (nestedRequirement->active &&
-                  failed(requireThreadY(&nestedOp, /*active=*/true)))
-                return failure();
-              if (nestedRequirement->inactive &&
-                  failed(requireThreadY(&nestedOp, /*active=*/false)))
-                return failure();
-            }
+            for (Block &nestedBlock : region)
+              mergeBroadeningInfo(info, self(self, nestedBlock));
           }
           continue;
         }
-        if (!isMemoryEffectFree(&nestedOp) &&
-            failed(requireThreadY(&nestedOp, /*active=*/false)))
-          return failure();
+        if (hasUnsafeOwnEffects(&nestedOp)) {
+          info.hasBroadeningConflict = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = &nestedOp;
+        }
       }
-      return requirement;
+      return info;
     };
 
-    FailureOr<ThreadYRequirement> threadYRequirement =
-        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
-    if (failed(threadYRequirement))
-      return {};
+    ThreadYBroadeningInfo threadYInfo =
+        analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
     auto applyCombineParDims =
         [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (!parDim.isThreadY())
-              mlir::acc::removeParDim(ancestorParDims, parDim);
-          }
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
+            mlir::acc::removeParDim(ancestorParDims, parDim);
           return success();
         };
     block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (acc::PrivatizeOp privatizeOp =
-                getPrivatizeForMemref(storeOp.getMemref())) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
@@ -1423,13 +1407,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (threadYRequirement->active || threadYRequirement->inactive) {
-      mlir::acc::GPUParallelDimAttr threadY =
-          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (threadYRequirement->active)
-        mlir::acc::insertParDim(ancestorParDims, threadY);
-      else
-        mlir::acc::removeParDim(ancestorParDims, threadY);
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
+    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+      if (threadYInfo.hasExplicitInactiveCombine ||
+          threadYInfo.hasBroadeningConflict) {
+        Operation *diagnosticOp =
+            threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
+        (void)accSupport.emitNYI(
+            diagnosticOp->getLoc(),
+            "operations in the same predicate region require incompatible "
+            "ThreadY predication");
+        hasFailed = true;
+        return {};
+      }
+      mlir::acc::insertParDim(ancestorParDims, threadY);
     }
   }
 
@@ -1886,22 +1879,18 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  Value current = memref;
-  while (Operation *def = current.getDefiningOp()) {
-    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
-      return privateLocal;
-    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-      current = viewLike.getViewSource();
+  llvm::SmallVector<Value, 8> worklist{memref};
+  llvm::SmallPtrSet<Value, 8> seen;
+  while (!worklist.empty()) {
+    Value v = worklist.pop_back_val();
+    if (!seen.insert(v).second)
       continue;
-    }
-    if (acc::PartialEntityAccessOpInterface partialAccess =
-            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
-      current = partialAccess.getBaseEntity();
+    Operation *def = v.getDefiningOp();
+    if (!def)
       continue;
-    }
-    // Do not follow arbitrary operands: multi-source operations such as
-    // arith.select do not prove that their result aliases private storage.
-    return nullptr;
+    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
+      return privateLocal;
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index ba8f16d3966df..0de030eda1e59 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -184,80 +185,3 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
-
-// CHECK-LABEL: func.func @aliased_worker_store
-// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
-// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[ALIAS_TX_ZERO]]
-// CHECK: memref.store
-// CHECK: acc.atomic.update
-func.func @aliased_worker_store(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
-// CHECK-LABEL: func.func @aliased_thread_store
-// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
-// CHECK: memref.store
-func.func @aliased_thread_store() {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private)
-        : (!acc.private_type<memref<i32>>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}

>From 30926fc3c3e3a6d6b9fe8e3fcb8442319849edf1 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:35 -0700
Subject: [PATCH 08/21] [OpenACC][test] Match broadening conflict locations

Expect diagnostics on the side effect that makes worker broadening unsafe.
---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 1275e4c77890a..7f22fd20aa89e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -69,8 +69,8 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          memref.store %c7_i32, %selected[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -103,12 +103,12 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
         acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.atomic.update %result_arg : memref<i32> {
           ^bb0(%current: i32):
             %next = arith.addi %current, %c1_i32 : i32
             acc.yield %next : i32
           }
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 55293fcdfa6ce6f49e61e43293e80df04bfec368 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:51:30 -0700
Subject: [PATCH 09/21] [OpenACC] Require safe predicate speculation

Treat reads and non-speculatable operations as broadening conflicts so enabling worker rows cannot introduce invalid memory accesses.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp  | 13 +++++--------
 ...to-gpu-worker-reduction-combine-mixed-scope.mlir |  6 ++----
 .../acc-cg-to-gpu-worker-reduction-combine.mlir     |  1 -
 3 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 50635dde561db..250c2be09df85 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1276,13 +1276,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return false;
     };
-    auto hasUnsafeOwnEffects = [](Operation *op) {
+    auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
-        SmallVector<MemoryEffects::EffectInstance> effects;
-        effectOp.getEffects(effects);
-        return llvm::any_of(effects, [](const auto &effect) {
-          return !isa<MemoryEffects::Read>(effect.getEffect());
-        });
+        return !effectOp.hasNoEffect();
       }
       return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
     };
@@ -1325,7 +1321,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          if (hasUnsafeOwnEffects(&nestedOp)) {
+          if (hasOwnEffects(&nestedOp)) {
             info.hasBroadeningConflict = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
@@ -1336,7 +1332,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
           continue;
         }
-        if (hasUnsafeOwnEffects(&nestedOp)) {
+        if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
+            !isPure(&nestedOp)) {
           info.hasBroadeningConflict = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 7f22fd20aa89e..290786ab7c1ce 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,7 +42,7 @@ func.func @mixed_scope_worker_reduction_combine(
   return
 }
 
-func.func @worker_combine_with_single_store(%result: memref<i32>) {
+func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -59,18 +59,16 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
-      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          memref.store %c7_i32, %selected[] : memref<i32>
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 0de030eda1e59..fb1a0256d2a38 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 300b3dc6a9334c5bd8b47daf1c1371bb6f5db806 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:59:06 -0700
Subject: [PATCH 10/21] [OpenACC] Reject worker combine regions

Keep generic combine regions single-row because their non-atomic load/store lowering is unsafe across worker rows.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 14 ++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 37 ++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 48 -------------------
 3 files changed, 48 insertions(+), 51 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 250c2be09df85..3697ae7405c9f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1286,11 +1286,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
       auto classifyCombine = [&](Operation *combineOp, Value src,
-                                 ArrayRef<GPUParallelDimAttr> parDims) {
+                                 ArrayRef<GPUParallelDimAttr> parDims,
+                                 bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
         if (hasThreadY && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
+          if (!supportsWorkerBroadening) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = combineOp;
+          }
         } else {
           info.hasExplicitInactiveCombine = true;
           if (!info.diagnosticOp)
@@ -1311,13 +1317,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
           classifyCombine(combineOp, combineOp.getSrcMemref(),
-                          getReductionCombineParDims(combineOp));
+                          getReductionCombineParDims(combineOp),
+                          /*supportsWorkerBroadening=*/true);
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
           classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
-                          getReductionCombineParDims(combineRegionOp));
+                          getReductionCombineParDims(combineRegionOp),
+                          /*supportsWorkerBroadening=*/false);
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 290786ab7c1ce..fffbfcb461c56 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -117,3 +117,40 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fb1a0256d2a38..686e8fd233980 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,54 +46,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// CHECK-LABEL: func.func @worker_reduction_combine_region
-// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
-// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[REGION_IS_X_ZERO]]
-// CHECK: arith.addi
-
-func.func @worker_reduction_combine_region(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
-          memref.store %c0_i32, %local[] : memref<i32>
-          scf.reduce
-        } {acc.par_dims = #acc<par_dims[thread_y]>}
-        acc.predicate_region {
-          acc.reduction_combine_region %local into %result_arg : memref<i32> {
-            %lhs = memref.load %result_arg[] : memref<i32>
-            %rhs = memref.load %local[] : memref<i32>
-            %sum = arith.addi %lhs, %rhs : i32
-            memref.store %sum, %result_arg[] : memref<i32>
-            acc.yield
-          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.

>From cbc18adba26e38ff6bebd01bf252112da4412887 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:03:05 -0700
Subject: [PATCH 11/21] [OpenACC] Require atomic worker combines

Broaden ThreadY only when block dimensions select the atomic combine lowering; reject ThreadY-only shared updates.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  4 ++-
 ...-worker-reduction-combine-mixed-scope.mlir | 32 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 3697ae7405c9f..76f942a2730f9 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1290,7 +1290,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                                  bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
-        if (hasThreadY && isProvenWorkerPrivate(src)) {
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
             info.hasBroadeningConflict = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index fffbfcb461c56..5758702a3b92e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -154,3 +154,35 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2f6b6b2e74b62114ad20366b768685e6af8bc000 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:06:26 -0700
Subject: [PATCH 12/21] [OpenACC][test] Check non-atomic worker predication

Verify ThreadY-only combines retain single-row predication while unsafe combine regions remain rejected.
---
 ...-worker-reduction-combine-mixed-scope.mlir | 31 ---------------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 39 +++++++++++++++++++
 2 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 5758702a3b92e..3d15b4ca4a59f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -155,34 +155,3 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   return
 }
 
-func.func @thread_y_only_worker_combine(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        acc.predicate_region {
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 686e8fd233980..901b23b25316f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -136,3 +136,42 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
+
+// Without a block dimension the combine lowers to a non-atomic update, so
+// ThreadY must retain the legacy single-row predicate.
+// CHECK-LABEL: func.func @thread_y_only_worker_combine
+// CHECK: gpu.launch {{.*}} threads([[ONLY_TX:%[^,]+]], [[ONLY_TY:%[^,]+]],
+// CHECK: %[[ONLY_TY_ZERO:.*]] = arith.cmpi eq, [[ONLY_TY]],
+// CHECK: %[[ONLY_TX_ZERO:.*]] = arith.cmpi eq, [[ONLY_TX]],
+// CHECK: %[[ONLY_ROW_ZERO:.*]] = arith.andi %[[ONLY_TX_ZERO]], %[[ONLY_TY_ZERO]]
+// CHECK: scf.if %[[ONLY_ROW_ZERO]]
+// CHECK: memref.store
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 88026c11dfb0376be0ad438de4a530d99d821119 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:10:52 -0700
Subject: [PATCH 13/21] [OpenACC][test] Remove trailing blank line

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 3d15b4ca4a59f..dc634cff25b07 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -41,7 +41,6 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
-
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 984e8f2ce4366cb23a6104b6aa965783f2380a49 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:11:22 -0700
Subject: [PATCH 14/21] [OpenACC][test] Normalize test file ending

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dc634cff25b07..dba8057d48e95 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -78,7 +78,6 @@ func.func @worker_combine_with_read(%result: memref<i32>) {
   }
   return
 }
-
 func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 8f6c707f4e58523929b8482505946acdd9dc9991 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:12:19 -0700
Subject: [PATCH 15/21] [OpenACC][test] Finish file ending cleanup

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir    | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dba8057d48e95..a641fe74867e4 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -151,5 +151,4 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
     } {origin = "acc.parallel"}
   }
   return
-}
-
+}
\ No newline at end of file

>From f629ee527f4b56ac33d85867d488a793af7cfc09 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:43:52 -0700
Subject: [PATCH 16/21] [OpenACC] Localize worker combine classification

Keep private-memory scope behavior unchanged and remove redundant failure plumbing from combine dimension handling.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 37 ++++++++++---------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 76f942a2730f9..98258dc622e3c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1265,8 +1265,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       while (Operation *def = current.getDefiningOp()) {
         if (acc::PrivateLocalOp privateLocal =
                 dyn_cast<acc::PrivateLocalOp>(def)) {
-          return getPrivateMemScope(getPrivatizeOp(
-                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+          acc::PrivatizeOp privatize =
+              getPrivatizeOp(privateLocal, computeRegion);
+          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+          if (!parDims)
+            return false;
+          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadX();
+          });
+          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadY();
+          });
+          return hasThreadY && !hasThreadX;
         }
         if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
           current = viewLike.getViewSource();
@@ -1355,13 +1365,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     ThreadYBroadeningInfo threadYInfo =
         analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
-    auto applyCombineParDims =
-        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
-            mlir::acc::removeParDim(ancestorParDims, parDim);
-          return success();
-        };
-    block->walk([&](Operation *op) -> WalkResult {
+    block->walk([&](Operation *op) {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1392,15 +1396,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(reductionCombineOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(combineRegionOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1412,7 +1416,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           mlir::acc::insertParDim(ancestorParDims, parDim);
         }
       }
-      return WalkResult::advance();
     });
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
@@ -1877,7 +1880,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasThreadY)
+  if (hasBlock && hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;

>From edc8abd53cd69917a03f21c72c975e4a26a4ed24 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:50:05 -0700
Subject: [PATCH 17/21] [OpenACC] Merge ThreadY conflict state

Use one flag for all operations that require the legacy inactive ThreadY predicate.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98258dc622e3c..6a5aebc4db323 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,15 +1248,13 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
-      bool hasExplicitInactiveCombine = false;
-      bool hasBroadeningConflict = false;
+      bool requiresInactiveThreadY = false;
       Operation *diagnosticOp = nullptr;
     };
     auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
                                   const ThreadYBroadeningInfo &src) {
       dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
-      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
-      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      dst.requiresInactiveThreadY |= src.requiresInactiveThreadY;
       if (!dst.diagnosticOp)
         dst.diagnosticOp = src.diagnosticOp;
     };
@@ -1305,12 +1303,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = combineOp;
           }
         } else {
-          info.hasExplicitInactiveCombine = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1342,7 +1340,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
           }
@@ -1354,7 +1352,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
             !isPure(&nestedOp)) {
-          info.hasBroadeningConflict = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
         }
@@ -1421,8 +1419,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
     bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
-      if (threadYInfo.hasExplicitInactiveCombine ||
-          threadYInfo.hasBroadeningConflict) {
+      if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
         (void)accSupport.emitNYI(

>From 90cbc61e638708c7e25e37e588e20329fbfb736e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:06:50 -0700
Subject: [PATCH 18/21] [OpenACC] Stabilize ThreadY conflict analysis

Capture structural ThreadY activity before operation-order mutations and cover a trailing worker-private store.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   |  6 +++---
 ...pu-worker-reduction-combine-mixed-scope.mlir |  1 +
 .../acc-cg-to-gpu-worker-reduction-combine.mlir | 17 +++++------------
 3 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6a5aebc4db323..09dbbf892ffbd 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,6 +1246,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1415,9 +1418,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    mlir::acc::GPUParallelDimAttr threadY =
-        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a641fe74867e4..a5ffb4bfb59f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -33,6 +33,7 @@ func.func @mixed_scope_worker_reduction_combine(
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
+          memref.store %c0_i32, %local[] : memref<i32>
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 901b23b25316f..4466ad74d1a91 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,17 +46,15 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// Nested predicate regions choose their ThreadY predicates independently. The
-// outer region must keep ThreadY active so it does not exclude worker rows
-// before the nested worker-private combine is reached.
+// The outer predicate must keep ThreadY active so it does not exclude worker
+// rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines
 // CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
 // CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
 // CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
 // CHECK-NOT: arith.andi
 // CHECK: scf.if %[[NESTED_TX_ZERO]]
-func.func @nested_worker_reduction_combines(
-    %other: memref<i32>, %result: memref<i32>) {
+func.func @nested_worker_reduction_combines(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -67,9 +65,8 @@ func.func @nested_worker_reduction_combines(
     %private = acc.privatize [#acc<par_dims[thread_y]>]
         : () -> !acc.private_type<memref<i32>>
     acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %other_arg = %other,
-            %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c0_i32 = arith.constant 0 : i32
@@ -85,10 +82,6 @@ func.func @nested_worker_reduction_combines(
             acc.reduction_combine %local into %result_arg <add> : memref<i32>
                 {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           }
-          acc.predicate_region {
-            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          }
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}

>From 9e6fcfb6dd7ff5d68e1e7e6a34b14a7a69763286 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:18 -0700
Subject: [PATCH 19/21] [OpenACC] Allow worker-private stores during combine

Treat proven worker-private stores as row-local so they can share the broadened ThreadY predicate safely.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp            | 4 ++++
 .../OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir       | 1 +
 2 files changed, 5 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 09dbbf892ffbd..f78a973da4e5a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1341,6 +1341,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          if (isProvenWorkerPrivate(storeOp.getMemref()))
+            continue;
+        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 4466ad74d1a91..cbfabf28ea891 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From fcc26cd10cf22d3fb30b46154828ecd7e4af7604 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:43:59 -0700
Subject: [PATCH 20/21] [OpenACC] Diagnose unproven worker aliases

Restrict ThreadY broadening to directly sized worker storage and conservatively reject aliases that may originate from it.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 73 +++++++++++--------
 ...-worker-reduction-combine-mixed-scope.mlir | 39 ++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 41 ++++++++++-
 3 files changed, 123 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f78a973da4e5a..f46a891f08652 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,7 +1248,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1262,30 +1261,46 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         dst.diagnosticOp = src.diagnosticOp;
     };
     auto isProvenWorkerPrivate = [&](Value memref) {
-      Value current = memref;
-      while (Operation *def = current.getDefiningOp()) {
-        if (acc::PrivateLocalOp privateLocal =
-                dyn_cast<acc::PrivateLocalOp>(def)) {
-          acc::PrivatizeOp privatize =
-              getPrivatizeOp(privateLocal, computeRegion);
-          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-          if (!parDims)
-            return false;
-          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadX();
-          });
-          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadY();
-          });
-          return hasThreadY && !hasThreadX;
-        }
-        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-          current = viewLike.getViewSource();
-          continue;
-        }
-        break;
+      acc::PrivateLocalOp privateLocal =
+          memref.getDefiningOp<acc::PrivateLocalOp>();
+      if (!privateLocal)
+        return false;
+      acc::PrivatizeOp privatize = getPrivatizeOp(privateLocal, computeRegion);
+      GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+      if (!parDims)
+        return false;
+      bool hasThreadX = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadX(); });
+      bool hasThreadY = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadY(); });
+      return hasThreadY && !hasThreadX;
+    };
+    auto hasWorkerPrivateOrigin = [&](auto &&self, Value value,
+                                      DenseSet<Value> &visited) -> bool {
+      if (!visited.insert(value).second)
+        return false;
+      if (isProvenWorkerPrivate(value))
+        return true;
+      Operation *def = value.getDefiningOp();
+      if (!def) {
+        auto blockArg = dyn_cast<BlockArgument>(value);
+        def = blockArg ? blockArg.getOwner()->getParentOp() : nullptr;
       }
-      return false;
+      if (!def)
+        return false;
+      if (llvm::any_of(def->getOperands(), [&](Value operand) {
+            return self(self, operand, visited);
+          }))
+        return true;
+      return llvm::any_of(def->getRegions(), [&](Region &region) {
+        return llvm::any_of(region, [&](Block &block) {
+          Operation *terminator = block.getTerminator();
+          return terminator &&
+                 llvm::any_of(terminator->getOperands(), [&](Value operand) {
+                   return self(self, operand, visited);
+                 });
+        });
+      });
     };
     auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
@@ -1312,6 +1327,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
         } else {
           info.requiresInactiveThreadY = true;
+          DenseSet<Value> visited;
+          if (hasThreadY && hasBlock &&
+              hasWorkerPrivateOrigin(hasWorkerPrivateOrigin, src, visited))
+            info.hasActiveWorkerCombine = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1341,10 +1360,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (isProvenWorkerPrivate(storeOp.getMemref()))
-            continue;
-        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
@@ -1422,7 +1437,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+    if (threadYInfo.hasActiveWorkerCombine) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a5ffb4bfb59f8..8724f1f57bc9b 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,6 +42,45 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_unproven_alias(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %alias = scf.if %true -> (memref<i32>) {
+          scf.yield %local : memref<i32>
+        } else {
+          scf.yield %local : memref<i32>
+        }
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %alias into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index cbfabf28ea891..fd4e12fd039a8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -47,6 +46,46 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
+// CHECK-LABEL: func.func @worker_combine_in_thread_y_scope
+// CHECK: gpu.launch {{.*}} threads([[SCOPE_TX:%[^,]+]], [[SCOPE_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[SCOPE_TY]]
+// CHECK: %[[SCOPE_TX_ZERO:.*]] = arith.cmpi eq, [[SCOPE_TX]],
+// CHECK: scf.if %[[SCOPE_TX_ZERO]]
+func.func @worker_combine_in_thread_y_scope(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 // The outer predicate must keep ThreadY active so it does not exclude worker
 // rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines

>From ca846e5a59a0e4ce0ec567a391cb4ab23f777640 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:57:39 -0700
Subject: [PATCH 21/21] [OpenACC] Allow stores to active worker sources

Permit row-local stores only when they target the exact private storage already sized for a supported worker combine.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp    | 16 ++++++++++++++++
 .../acc-cg-to-gpu-worker-reduction-combine.mlir  |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f46a891f08652..bf90b51a2d05b 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1311,6 +1311,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     auto analyzeThreadYBroadening =
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
+      DenseSet<Value> activeWorkerSources;
+      predicateBlock.walk([&](acc::ReductionCombineOp combineOp) {
+        ArrayRef<GPUParallelDimAttr> parDims =
+            getReductionCombineParDims(combineOp);
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock &&
+            isProvenWorkerPrivate(combineOp.getSrcMemref()))
+          activeWorkerSources.insert(combineOp.getSrcMemref());
+      });
       auto classifyCombine = [&](Operation *combineOp, Value src,
                                  ArrayRef<GPUParallelDimAttr> parDims,
                                  bool supportsWorkerBroadening) {
@@ -1360,6 +1372,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          if (activeWorkerSources.contains(storeOp.getMemref()))
+            continue;
+        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fd4e12fd039a8..7063239fbc09d 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -9,6 +9,7 @@
 // CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
 // CHECK-NOT: arith.andi
 // CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: memref.store
 // CHECK: acc.atomic.update
 
 func.func @worker_reduction_combine(%result: memref<i32>) {
@@ -35,6 +36,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }


From llvmlistbot at llvm.org  Mon Jul 20 20:58:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 20:58:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support multi-rank array
	reductions (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a5eee51.4a342cbc.3a2f36.ea5d@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210853

>From 2d005d7cf11a32e49f59fafddd6ccc1d39991825 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:05:29 -0700
Subject: [PATCH 1/8] [mlir][OpenACC] Support multi-rank array reductions

Initialize static rank-N per-thread accumulators and delinearize flattened reduction indices before accessing them.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 61 +++++++++---
 .../acc-cg-to-gpu-reduction-array.mlir        | 94 +++++++++++++++++++
 2 files changed, 142 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 77fa82ba6bb4e..44ad1f3154174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,17 +736,28 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static rank-1");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
-  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
   Value step = arith::ConstantIndexOp::create(b, loc, 1);
-  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
-  OpBuilder::InsertionGuard g(b);
-  b.setInsertionPoint(forOp.getBody()->getTerminator());
-  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+  SmallVector<Value> indices;
+  auto buildLoopNest = [&](auto &&self, unsigned dim) -> void {
+    if (dim == baseTy.getRank()) {
+      memref::StoreOp::create(b, loc, ident, alloca, indices);
+      return;
+    }
+
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+    OpBuilder::InsertionGuard g(b);
+    b.setInsertionPoint(forOp.getBody()->getTerminator());
+    indices.push_back(forOp.getInductionVar());
+    self(self, dim + 1);
+    indices.pop_back();
+  };
+  buildLoopNest(buildLoopNest, 0);
 }
 
 std::optional<int64_t>
@@ -3244,10 +3255,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  if (!memref)
+  if (!memrefTy) {
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
-  if (memrefTy.getRank() != 1)
-    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() == 0) {
+    (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
+    return;
+  }
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
@@ -3355,9 +3375,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     OpBuilder::InsertionGuard guard(rewriter);
     rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
     Value iv = forOp.getInductionVar();
-    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
-    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
-                         ValueRange{iv},
+    SmallVector<Value> indices{iv};
+    if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
+      indices.resize(memrefTy.getRank());
+      Value linearIndex = iv;
+      for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
+        indices[dim] =
+            arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
+        if (dim != 0)
+          linearIndex =
+              arith::DivUIOp::create(rewriter, loc, linearIndex, dimSize);
+      }
+    }
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, indices);
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(), indices,
                          /*isPerThreadPrivateTarget=*/true);
   }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index bfb829c901af3..a008c84108f05 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -134,3 +134,97 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_array_reduction
+// CHECK: %[[R2_ALLOCA:.*]] = memref.alloca() : memref<2x3xi32>
+// CHECK: scf.for %[[R2_I:.*]] =
+// CHECK:   scf.for %[[R2_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[R2_ALLOCA]][%[[R2_I]], %[[R2_J]]] : memref<2x3xi32>
+// CHECK: scf.for %[[R2_LINEAR:.*]] =
+// CHECK: %[[R2_J_IDX:.*]] = arith.remui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+
+func.func @rank_two_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x3xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_three_array_reduction
+// CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_I:.*]] =
+// CHECK:   scf.for %[[R3_J:.*]] =
+// CHECK:     scf.for %[[R3_K:.*]] =
+// CHECK:       memref.store %{{.*}}, %[[R3_ALLOCA]][%[[R3_I]], %[[R3_J]], %[[R3_K]]] : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_LINEAR:.*]] =
+// CHECK: %[[R3_K_IDX:.*]] = arith.remui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_PLANE:.*]] = arith.divui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_J_IDX:.*]] = arith.remui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+
+func.func @rank_three_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x2x2xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x2x2xi32>>) {
+    %c8 = arith.constant 8 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x2x2xi32>>) -> memref<2x2x2xi32>
+    %bounds = acc.bounds extent(%c8 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x2x2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
+// invalid indexing operations.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_zero_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @rank_zero_array_reduction(%local: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
+    %extent = arith.constant 1 : index
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 1a70c1679ce1c49a2e54fc8ed501ba06fa3d7234 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:57 -0700
Subject: [PATCH 2/8] [mlir][acc] Use dynamic stack storage for thread-private
 arrays

Avoid grid-sized backing allocations for dynamic per-thread storage while preserving shared lowering for oversized static arrays.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 +++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 41 ++++++++++++----
 3 files changed, 90 insertions(+), 28 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 44ad1f3154174..7135ca9177153 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,8 +736,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static ranked");
+  assert(baseTy.getRank() > 0 &&
+         "per-thread array reduction accumulator must have positive rank");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
@@ -749,7 +749,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
       return;
     }
 
-    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    Value ub =
+        baseTy.isDynamicDim(dim)
+            ? memref::DimOp::create(b, loc, alloca, dim).getResult()
+            : arith::ConstantIndexOp::create(b, loc, baseTy.getDimSize(dim))
+                  .getResult();
     auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
     OpBuilder::InsertionGuard g(b);
     b.setInsertionPoint(forOp.getBody()->getTerminator());
@@ -2494,9 +2498,15 @@ void ACCCGToGPULowering::processPrivateLocal(
     // the accumulate can reduce each element across threads.
     acc::ReductionAccumulateArrayOp arrayAccum =
         perThreadArrayReductionAccum(privateLocal.getResult());
-    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+    bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
+    bool canUseDynamicAlloca =
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -2583,9 +2593,15 @@ void ACCCGToGPULowering::processPrivateLocal(
   acc::ReductionAccumulateArrayOp arrayAccum =
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    bool canUseDynamicAlloca =
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -3263,12 +3279,6 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
     return;
   }
-  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
-    (void)accSupport.emitNYI(loc,
-                             "reduction: dynamic multi-rank accumulate array");
-    return;
-  }
-
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3377,13 +3387,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
-      assert(memrefTy.hasStaticShape() &&
-             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize = arith::ConstantIndexOp::create(
-            rewriter, loc, memrefTy.getDimSize(dim));
+        Value dimSize =
+            memrefTy.isDynamicDim(dim)
+                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
+                : arith::ConstantIndexOp::create(rewriter, loc,
+                                                 memrefTy.getDimSize(dim))
+                      .getResult();
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index 281eae4b5b0f7..b186720512991 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,3 +23,32 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
+
+// Dynamic thread-private storage is allocated in the launch body. A
+// grid-wide backing allocation would scale with the logical block count.
+//
+// CHECK-LABEL: func.func @dynamic_threadprivate
+// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
+// CHECK:       gpu.launch
+// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
+// CHECK-NOT:     acc.unwrap_private
+
+func.func @dynamic_threadprivate(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
+      : (index) -> !acc.private_type<memref<?xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg = %private, %extent = %n)
+      : (!acc.private_type<memref<?xi32>>, index) {
+    %c0 = arith.constant 0 : index
+    %c0_i32 = arith.constant 0 : i32
+    %local = acc.private_local %arg
+        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
+    memref.store %c0_i32, %local[%c0] : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index a008c84108f05..f287ae054a846 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -192,25 +192,46 @@ func.func @rank_three_array_reduction() {
   return
 }
 
-// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
-// invalid indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
 // CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[DYN_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_I:.*]] =
+// CHECK:   scf.for %[[DYN_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_LINEAR:.*]] =
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: arith.remui %[[DYN_LINEAR]]
+// CHECK: arith.divui %[[DYN_LINEAR]]
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
   return
 }
 
+// Rank-zero accumulators must not generate invalid indexing operations.
 // CHECK-LABEL: func.func @rank_zero_array_reduction
 // CHECK: gpu.launch
 // CHECK-NOT: memref.load

>From 94cd48df3ac887a669b4f7c4cd871226d7014566 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:04:45 -0700
Subject: [PATCH 3/8] [mlir][OpenACC] Handle dynamic reductions on all thread
 dimensions

Classify dynamic accumulators using every GPU thread dimension so rank-N reductions consistently receive per-thread storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  7 +++--
 .../acc-cg-to-gpu-reduction-array.mlir        | 29 ++++---------------
 2 files changed, 9 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 7135ca9177153..e353f70939699 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -2500,7 +2500,8 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
@@ -2594,7 +2595,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
@@ -3334,7 +3335,7 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   } else {
     isPerThreadPrivate = llvm::any_of(
         op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isThreadX(); });
+        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index f287ae054a846..be91cc6762da6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -211,41 +211,22 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_x]>}
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
-
-// Rank-zero accumulators must not generate invalid indexing operations.
-// CHECK-LABEL: func.func @rank_zero_array_reduction
-// CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @rank_zero_array_reduction(%local: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
-    %extent = arith.constant 1 : index
-    %bounds = acc.bounds extent(%extent : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}

>From caf6ec726262faae429717edb8cadc1b6cd00abc Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:20:44 -0700
Subject: [PATCH 4/8] [mlir][OpenACC] Find array reductions through descriptor
 chains

Trace accumulate operands back to private storage so descriptor construction cannot bypass per-thread allocation and initialization.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 23 ++++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 35 +++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index e353f70939699..cc3b8910ad765 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -727,6 +727,29 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
+
+  // Descriptor construction may obscure the forward use chain. Fall back to
+  // tracing each accumulate operand back to this private local.
+  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
+  if (!privateLocal)
+    return nullptr;
+  acc::ComputeRegionOp computeRegion =
+      privateLocal->getParentOfType<acc::ComputeRegionOp>();
+  if (!computeRegion)
+    return nullptr;
+  acc::ReductionAccumulateArrayOp result;
+  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
+    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
+      return WalkResult::advance();
+    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
+                                  [](auto pd) { return pd.isAnyThread(); });
+    if (!hasThread || !reductionHasBlockContext(accArr))
+      return WalkResult::advance();
+    result = accArr;
+    return WalkResult::interrupt();
+  });
+  if (result)
+    return result;
   return nullptr;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index be91cc6762da6..ae9ea1e4c95bb 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -230,3 +230,38 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[INDIRECT_I:.*]] =
+// CHECK:   scf.for %[[INDIRECT_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor_value = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 3790839be0295d1bd7f4ca3bbd151aac1edd5a9a Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:34:24 -0700
Subject: [PATCH 5/8] [mlir][OpenACC] Allocate dynamic privates for any thread
 dimension

Use privatization parallel dimensions so descriptor indirection cannot force dynamic thread-private backing arrays into grid-sized storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 +++++++------------
 .../acc-cg-to-gpu-reduction-array.mlir        | 34 ++++++++++---
 2 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index cc3b8910ad765..99315cad34e1c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,6 +383,14 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
+/// True when \p privatize has any thread-level parallelism.
+static bool isThreadPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
+  return false;
+}
+
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -728,28 +736,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     }
   }
 
-  // Descriptor construction may obscure the forward use chain. Fall back to
-  // tracing each accumulate operand back to this private local.
-  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
-  if (!privateLocal)
-    return nullptr;
-  acc::ComputeRegionOp computeRegion =
-      privateLocal->getParentOfType<acc::ComputeRegionOp>();
-  if (!computeRegion)
-    return nullptr;
-  acc::ReductionAccumulateArrayOp result;
-  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
-    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
-      return WalkResult::advance();
-    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
-                                  [](auto pd) { return pd.isAnyThread(); });
-    if (!hasThread || !reductionHasBlockContext(accArr))
-      return WalkResult::advance();
-    result = accArr;
-    return WalkResult::interrupt();
-  });
-  if (result)
-    return result;
   return nullptr;
 }
 
@@ -2523,12 +2509,13 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((isThreadPrivate || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (isThreadPrivate || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2618,12 +2605,13 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((parDim.isThreadX() || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (parDim.isThreadX() || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index ae9ea1e4c95bb..b9fdf6ecabcb6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -245,17 +245,39 @@ func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %private_data = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
+        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
+            %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>,
+           !acc.private_type<memref<1xindex>>, index) {
+      %c0 = arith.constant 0 : index
+      %c1_k = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      %local_data = acc.private_local %arg_data
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %descriptor_value = builtin.unrealized_conversion_cast %local
-          : memref<?x?xi32> to memref<?x?xi32>
+      %local_descriptor = acc.private_local %arg_descriptor
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
+      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
+      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
+      scf.for %i = %c0 to %m_dim step %c1_k {
+        scf.for %j = %c0 to %n_dim step %c1_k {
+          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
+        }
+      }
+      %descriptor = builtin.unrealized_conversion_cast %local_data
+          : memref<?x?xi32> to index
+      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
+      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
+      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
+          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
           : memref<?x?xi32>

>From affc0c39172f713075ac1912eb631d85b4ce6b56 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:24 -0700
Subject: [PATCH 6/8] [mlir][OpenACC] Preserve worker-private array storage

Restrict generated stack reductions to thread_x and keep thread_y/thread_z arrays indexed or shared, including explicit dynamic shared memory.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  50 ++++----
 .../acc-cg-to-gpu-reduction-array.mlir        | 107 +++++++++++-------
 2 files changed, 86 insertions(+), 71 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 99315cad34e1c..48e7d13730eeb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,14 +383,6 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
-/// True when \p privatize has any thread-level parallelism.
-static bool isThreadPrivatize(PrivatizeOp privatize) {
-  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
-    return llvm::any_of(parDimsAttr.getArray(),
-                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
-  return false;
-}
-
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -709,7 +701,7 @@ static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
 }
 
 /// Returns the array reduction accumulate (through cast/view ops) that \p v
-/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// feeds if it needs per-thread storage: its par_dims include thread_x
 /// and it has block context so the cross-thread all_reduce is well defined.
 static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   SmallVector<Value> worklist{v};
@@ -721,10 +713,10 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     for (Operation *user : cur.getUsers()) {
       if (acc::ReductionAccumulateArrayOp accArr =
               dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
-        bool hasThread = false;
+        bool hasThreadX = false;
         for (auto pd : accArr.getParDims().getArray())
-          hasThread |= pd.isAnyThread();
-        if (hasThread && reductionHasBlockContext(accArr))
+          hasThreadX |= pd.isThreadX();
+        if (hasThreadX && reductionHasBlockContext(accArr))
           return accArr;
         continue;
       }
@@ -735,7 +727,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
-
   return nullptr;
 }
 
@@ -2509,13 +2500,11 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (isThreadPrivate || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2605,13 +2594,12 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((parDim.isThreadX() || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -3337,16 +3325,20 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   // it is too large. For a dynamically-shaped accumulator the type conveys no
   // size, so classify from par_dims (which the producer sets to the reduction's
   // actual parallel scope): a thread dimension means per-thread storage.
+  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
+  bool isExplicitlyShared =
+      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
   bool isPerThreadPrivate;
   if (memrefTy.hasStaticShape()) {
-    Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
     isPerThreadPrivate =
-        !isa_and_nonnull<memref::AllocOp>(rootOp) &&
+        !isExplicitlyShared &&
         canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
   } else {
-    isPerThreadPrivate = llvm::any_of(
-        op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
+    isPerThreadPrivate = !isExplicitlyShared &&
+                         llvm::any_of(op.getParDims().getArray(),
+                                      [](mlir::acc::GPUParallelDimAttr d) {
+                                        return d.isThreadX();
+                                      });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index b9fdf6ecabcb6..269e0a1287046 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,27 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// CHECK-LABEL: func.func @array_reduction_dynamic_shared
+// CHECK: acc.gpu_shared_memory
+// CHECK-NOT: gpu.all_reduce
+func.func @array_reduction_dynamic_shared(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%ext = %n) : (index) {
+    %shared = acc.gpu_shared_memory(%ext)
+        {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
+        : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
@@ -162,6 +183,33 @@ func.func @rank_two_array_reduction() {
   return
 }
 
+// CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @rank_two_array_reduction_thread_y() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %kty = %ty)
+      ins(%arg = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg
+        {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+        : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<2x3xi32>
+        {par_dims = #acc<par_dims[block_x, thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @rank_three_array_reduction
 // CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
 // CHECK: scf.for %[[R3_I:.*]] =
@@ -211,75 +259,50 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
 
-// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// Dynamic worker-level storage must remain indexed by thread_y instead of
+// becoming one stack allocation per physical CUDA thread.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
-// CHECK-NOT: acc.unwrap_private
-// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
-// CHECK: scf.for %[[INDIRECT_I:.*]] =
-// CHECK:   scf.for %[[INDIRECT_J:.*]] =
-// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
-// CHECK: gpu.all_reduce add
-func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private_data = acc.privatize(%m, %n)
-      [#acc<par_dims[block_x, thread_y]>]
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
-      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
-            %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>,
-           !acc.private_type<memref<1xindex>>, index) {
-      %c0 = arith.constant 0 : index
-      %c1_k = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      %local_data = acc.private_local %arg_data
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %local_descriptor = acc.private_local %arg_descriptor
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
-          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
-      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
-      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
-      scf.for %i = %c0 to %m_dim step %c1_k {
-        scf.for %j = %c0 to %n_dim step %c1_k {
-          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
-        }
-      }
-      %descriptor = builtin.unrealized_conversion_cast %local_data
-          : memref<?x?xi32> to index
-      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
-      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
-      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
-          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield

>From 3bba5f0a8722c8926daa522949668e420f5ea9aa Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:33:43 -0700
Subject: [PATCH 7/8] [mlir][OpenACC] Track array reduction storage provenance

Classify accumulators from their backing allocation through casts so shared storage is not reduced again and private storage is initialized correctly. Keep dynamic stack allocation reduction-specific and honor byte strides.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 71 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 --------
 .../acc-cg-to-gpu-reduction-array.mlir        | 61 ++++++++++------
 3 files changed, 82 insertions(+), 79 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 48e7d13730eeb..84243de10830a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -317,6 +317,17 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     return false;
   }
 
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
+    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
+        castOp->getOperand(0) == trackedOperand &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      passThroughResults.push_back(castOp->getResult(0));
+      return true;
+    }
+    return false;
+  }
+
   // Partial-entity accesses (e.g. array element or field access) forward the
   // base entity through to their results, so treat them as pass-through when
   // the base entity is the value being tracked.
@@ -341,6 +352,14 @@ static Value unwrapMemRefConversion(Value v) {
         continue;
       }
     }
+    if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+        castOp && castOp->getNumOperands() == 1 &&
+        castOp->getNumResults() == 1 &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      v = castOp->getOperand(0);
+      continue;
+    }
     break;
   }
   return v;
@@ -730,6 +749,13 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   return nullptr;
 }
 
+static bool
+canUseDynamicReductionAlloca(MemRefType baseTy, acc::PrivatizeOp privatizeOp,
+                             acc::ReductionAccumulateArrayOp arrayAccum) {
+  return arrayAccum && !baseTy.hasStaticShape() &&
+         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+}
+
 /// Store the reduction identity to every element of a freshly allocated
 /// per-thread array accumulator so all lanes start from identity (the original
 /// init loop may only run on one lane).
@@ -2500,8 +2526,7 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -2594,9 +2619,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((parDim.isThreadX() || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -3318,28 +3341,10 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   }
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
-  // accumulator copy. For a statically-shaped accumulator, classify from the
-  // operand: an explicit shared allocation is block-shared regardless of size,
-  // and anything else (a per-thread stack alloca, or a view over one) is
-  // per-thread when it fits the per-thread stack budget and block-shared when
-  // it is too large. For a dynamically-shaped accumulator the type conveys no
-  // size, so classify from par_dims (which the producer sets to the reduction's
-  // actual parallel scope): a thread dimension means per-thread storage.
+  // accumulator copy. Classify the operand from its underlying allocation
+  // rather than its shape or par_dims: only memref.alloca is thread-private.
   Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isExplicitlyShared =
-      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
-  bool isPerThreadPrivate;
-  if (memrefTy.hasStaticShape()) {
-    isPerThreadPrivate =
-        !isExplicitlyShared &&
-        canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
-  } else {
-    isPerThreadPrivate = !isExplicitlyShared &&
-                         llvm::any_of(op.getParDims().getArray(),
-                                      [](mlir::acc::GPUParallelDimAttr d) {
-                                        return d.isThreadX();
-                                      });
-  }
+  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
@@ -3367,7 +3372,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
   Value lb =
       boundsOp.getLowerbound() ? toIndex(boundsOp.getLowerbound()) : zero;
-  Value step = boundsOp.getStride() ? toIndex(boundsOp.getStride()) : one;
+  Value step = one;
+  if (Value stride = boundsOp.getStride()) {
+    step = toIndex(stride);
+    if (boundsOp.getStrideInBytes()) {
+      int64_t elementSize =
+          getElementSizeInBytes(loc, memrefTy.getElementType());
+      if (elementSize == 0)
+        return;
+      Value elementSizeValue =
+          arith::ConstantIndexOp::create(rewriter, loc, elementSize);
+      step = arith::DivUIOp::create(rewriter, loc, step, elementSizeValue);
+    }
+  }
   // Exclusive upper bound. `extent` counts elements, so the span is
   // `extent * step` (for the common unit-stride case step is 1); fall back to
   // the inclusive upperbound when no extent is given.
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index b186720512991..281eae4b5b0f7 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,32 +23,3 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
-
-// Dynamic thread-private storage is allocated in the launch body. A
-// grid-wide backing allocation would scale with the logical block count.
-//
-// CHECK-LABEL: func.func @dynamic_threadprivate
-// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
-// CHECK:       gpu.launch
-// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
-// CHECK-NOT:     acc.unwrap_private
-
-func.func @dynamic_threadprivate(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
-      : (index) -> !acc.private_type<memref<?xi32>>
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-      ins(%arg = %private, %extent = %n)
-      : (!acc.private_type<memref<?xi32>>, index) {
-    %c0 = arith.constant 0 : index
-    %c0_i32 = arith.constant 0 : i32
-    %local = acc.private_local %arg
-        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
-    memref.store %c0_i32, %local[%c0] : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 269e0a1287046..56469f801f026 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -85,7 +85,8 @@ func.func @array_reduction_small_shared() {
 }
 
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
-// CHECK: acc.gpu_shared_memory
+// CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
+// CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]
 // CHECK-NOT: gpu.all_reduce
 func.func @array_reduction_dynamic_shared(%n: index) {
   %c1 = arith.constant 1 : index
@@ -96,8 +97,11 @@ func.func @array_reduction_dynamic_shared(%n: index) {
     %shared = acc.gpu_shared_memory(%ext)
         {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
         : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %cast = builtin.unrealized_conversion_cast %shared
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        to memref<?xi32, #gpu.address_space<workgroup>>
     %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+    acc.reduction_accumulate_array %cast bounds(%bounds) <add>
         : memref<?xi32, #gpu.address_space<workgroup>>
         {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -108,11 +112,13 @@ func.func @array_reduction_dynamic_shared(%n: index) {
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
-// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[BYTE_STEP:.*]] = arith.constant 8 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[ELEMENT_SIZE:.*]] = arith.constant 4 : index
+// CHECK: %[[ELEMENT_STEP:.*]] = arith.divui %[[BYTE_STEP]], %[[ELEMENT_SIZE]] : index
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[ELEMENT_STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
-// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[ELEMENT_STEP]]
 func.func @array_reduction_strided_extent() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -120,11 +126,11 @@ func.func @array_reduction_strided_extent() {
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
   acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
     %c1_b = arith.constant 1 : index
-    %c2 = arith.constant 2 : index
+    %c8 = arith.constant 8 : index
     %c3 = arith.constant 3 : index
     %local = memref.alloca() : memref<8xi32>
     %bounds = acc.bounds lowerbound(%c1_b : index) extent(%c3 : index)
-        stride(%c2 : index)
+        stride(%c8 : index) {strideInBytes = true}
     acc.reduction_accumulate_array %local bounds(%bounds) <add>
         : memref<8xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -132,13 +138,14 @@ func.func @array_reduction_strided_extent() {
   return
 }
 
-// A dynamically-shaped accumulator (a strided view whose type conveys no size)
-// is classified per-thread from par_dims: a thread dimension means per-thread
-// storage, so lowering emits the per-element gpu.all_reduce.
+// A dynamically-shaped view over an external buffer is shared even when the
+// accumulate includes thread_x. Storage provenance, not par_dims, controls
+// whether gpu.all_reduce is valid.
 //
 // CHECK-LABEL: func.func @array_reduction_dynamic_par_dims
-// CHECK: scf.for
-// CHECK: gpu.all_reduce add
+// CHECK: memref.reinterpret_cast
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
 func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -186,6 +193,7 @@ func.func @rank_two_array_reduction() {
 // CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
+// CHECK: acc.gpu_shared_memory
 // CHECK-NOT: gpu.all_reduce
 // CHECK: gpu.terminator
 func.func @rank_two_array_reduction_thread_y() {
@@ -247,12 +255,13 @@ func.func @rank_three_array_reduction() {
 // CHECK: scf.for %[[DYN_I:.*]] =
 // CHECK:   scf.for %[[DYN_J:.*]] =
 // CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: %[[DYN_DESC:.*]] = builtin.unrealized_conversion_cast %[[DYN_ALLOCA]]
 // CHECK: scf.for %[[DYN_LINEAR:.*]] =
-// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.dim %[[DYN_DESC]]
 // CHECK: arith.remui %[[DYN_LINEAR]]
 // CHECK: arith.divui %[[DYN_LINEAR]]
-// CHECK: memref.dim %[[DYN_ALLOCA]]
-// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: memref.dim %[[DYN_DESC]]
+// CHECK: memref.load %[[DYN_DESC]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
 // CHECK: gpu.all_reduce add
 func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
@@ -269,8 +278,10 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
       %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+      acc.reduction_accumulate_array %descriptor bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
@@ -279,32 +290,36 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   return
 }
 
-// Dynamic worker-level storage must remain indexed by thread_y instead of
+// Dynamic worker-level storage must remain indexed by thread_y/thread_z instead
 // becoming one stack allocation per physical CUDA thread.
-// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_yz
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
 // CHECK-NOT: gpu.all_reduce
+// CHECK: %[[WORKER_STORAGE:.*]] = memref.view %{{.*}}[%{{.*}}][%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}] : memref<?xi8> to memref<?x?x?x?x?xi32>
+// CHECK: memref.subview %[[WORKER_STORAGE]][%{{.*}}, %{{.*}}, %{{.*}}, 0, 0] [1, 1, 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1, 1]
 // CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
+func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tz = acc.par_width %c128 {par_dim = #acc.par_dim<thread_z>}
+  %private = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y, thread_z]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty, %ktz = %tz)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
       acc.yield
     } {origin = "acc.parallel"}
   }

>From 08888a2115abad8c630b5fab31617fcc3e48821e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:57:42 -0700
Subject: [PATCH 8/8] [mlir][OpenACC] Test partial multi-rank reduction bounds

Cover delinearization with partial bounds and a non-identity memref layout.
---
 .../acc-cg-to-gpu-reduction-array.mlir        | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 56469f801f026..357f1e14d7a49 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -325,3 +325,35 @@ func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_partial_bounds_strided_layout
+// CHECK: %[[LB:.*]] = arith.constant 5 : index
+// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
+// CHECK: %[[LAYOUT_ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
+// CHECK: scf.for %[[LINEAR:.*]] = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: %[[COL:.*]] = arith.remui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW_LINEAR:.*]] = arith.divui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW:.*]] = arith.remui %[[ROW_LINEAR]], %{{.*}} : index
+// CHECK: memref.load %[[LAYOUT_ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
+func.func @rank_two_partial_bounds_strided_layout() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
+    %lb = arith.constant 5 : index
+    %step = arith.constant 2 : index
+    %extent = arith.constant 3 : index
+    %local = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+    %bounds = acc.bounds lowerbound(%lb : index) extent(%extent : index)
+        stride(%step : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<3x4xi32, strided<[8, 2]>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}


From llvmlistbot at llvm.org  Mon Jul 20 21:00:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 21:00:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Preserve worker rows when
 combining reductions (PR #210804)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210804@github.com>
Message-ID: <6a5eeee9.af6ba3ed.3c8c03.4289@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210804

>From 4c316621a473f21607795bafbd1ffa51a9dabe93 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 13:12:09 -0700
Subject: [PATCH 01/22] [OpenACC] Preserve worker rows when combining
 reductions

Keep ThreadY active for worker-private sources so every worker row contributes, and reject mixed scopes that require incompatible predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 53 +++++++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 43 +++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 95 +++++++++++++++++++
 3 files changed, 181 insertions(+), 10 deletions(-)
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
 create mode 100644 mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 24aeed6e23bda..6be79b0dd40e4 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,7 +1246,30 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    block->walk([&](Operation *op) {
+    std::optional<bool> combineThreadYActive;
+    auto applyCombineParDims =
+        [&](Operation *combineOp, Value src,
+            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+          bool isWorkerPrivate =
+              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
+            if (parDim.isThreadY()) {
+              if (combineThreadYActive &&
+                  *combineThreadYActive != isWorkerPrivate) {
+                combineOp->emitError()
+                    << "mixed worker-private and non-worker-private reduction "
+                       "combines require incompatible ThreadY predication";
+                hasFailed = true;
+                return failure();
+              }
+              combineThreadYActive = isWorkerPrivate;
+            } else {
+              mlir::acc::removeParDim(ancestorParDims, parDim);
+            }
+          }
+          return success();
+        };
+    block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1277,17 +1300,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(reductionCombineOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                reductionCombineOp, reductionCombineOp.getSrcMemref(),
+                getReductionCombineParDims(reductionCombineOp))))
+          return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        for (mlir::acc::GPUParallelDimAttr parDim :
-             getReductionCombineParDims(combineRegionOp)) {
-          mlir::acc::removeParDim(ancestorParDims, parDim);
-        }
+        if (failed(applyCombineParDims(
+                combineRegionOp, combineRegionOp.getSrcVar(),
+                getReductionCombineParDims(combineRegionOp))))
+          return WalkResult::interrupt();
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1301,6 +1324,14 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
+    if (combineThreadYActive) {
+      mlir::acc::GPUParallelDimAttr threadY =
+          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+      if (*combineThreadYActive)
+        mlir::acc::insertParDim(ancestorParDims, threadY);
+      else
+        mlir::acc::removeParDim(ancestorParDims, threadY);
+    }
   }
 
   // Obtain launch dimensions
@@ -1747,7 +1778,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasBlock && hasThreadY)
+  if (hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;
@@ -1871,6 +1902,8 @@ void ACCCGToGPULowering::processPredicateRegion(
             SmallVector<mlir::acc::GPUParallelDimAttr>>
       parDimsPair = computeActiveAndInactiveParDims(
           interOp, &interOp.getRegion().front());
+  if (hasFailed)
+    return;
 
   // If ThreadY reduction exists, subgroup alignment is applied
   // (blockDim.x = subgroupSize), so ThreadX lanes exist even without explicit
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
new file mode 100644
index 0000000000000..a0d381740c034
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -0,0 +1,43 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" \
+// RUN:   -verify-diagnostics
+
+func.func @mixed_scope_worker_reduction_combine(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
new file mode 100644
index 0000000000000..b2fd246d33d1b
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -0,0 +1,95 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// A worker-private reduction has one shared slot per ThreadY row. The combine
+// must keep ThreadY active and predicate only ThreadX.
+
+// CHECK-LABEL: func.func @worker_reduction_combine
+// CHECK: gpu.launch {{.*}} threads([[TID_X:%[^,]+]], [[TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[TID_Y]]
+// CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: acc.atomic.update
+
+func.func @worker_reduction_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @worker_reduction_combine_region
+// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
+// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[REGION_IS_X_ZERO]]
+// CHECK: arith.addi
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From d763e8065946687750b2c78fcf5393167b7de962 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 15:08:39 -0700
Subject: [PATCH 02/22] [OpenACC] Report mixed reduction scopes as NYI

Use the standard OpenACC unsupported-feature diagnostic for incompatible ThreadY predication.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6be79b0dd40e4..378fa17b24850 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1256,9 +1256,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             if (parDim.isThreadY()) {
               if (combineThreadYActive &&
                   *combineThreadYActive != isWorkerPrivate) {
-                combineOp->emitError()
-                    << "mixed worker-private and non-worker-private reduction "
-                       "combines require incompatible ThreadY predication";
+                (void)accSupport.emitNYI(
+                    combineOp->getLoc(),
+                    "mixed worker-private and non-worker-private reduction "
+                    "combines require incompatible ThreadY predication");
                 hasFailed = true;
                 return failure();
               }

>From 1fe76c2f1ea14bd036598286c2d9c0ea8d2291ef Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 16:50:59 -0700
Subject: [PATCH 03/22] [OpenACC] Isolate ThreadY predicate requirements

Propagate nested worker requirements without broadening unrelated side effects, and reject incompatible operations sharing one predicate region.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 126 +++++++++++++++---
 ...-worker-reduction-combine-mixed-scope.mlir |  40 +++++-
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  46 +++++++
 3 files changed, 189 insertions(+), 23 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 378fa17b24850..34131dc3753a3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,27 +1246,111 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    std::optional<bool> combineThreadYActive;
+    struct ThreadYRequirement {
+      bool active = false;
+      bool inactive = false;
+    };
+    auto analyzeThreadYRequirements =
+        [&](auto &&self,
+            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
+      ThreadYRequirement requirement;
+      auto requireThreadY = [&](Operation *op, bool active) {
+        if ((active && requirement.inactive) ||
+            (!active && requirement.active)) {
+          (void)accSupport.emitNYI(
+              op->getLoc(),
+              "operations in the same predicate region require incompatible "
+              "ThreadY predication");
+          hasFailed = true;
+          return failure();
+        }
+        requirement.active |= active;
+        requirement.inactive |= !active;
+        return success();
+      };
+      auto applyCombineRequirement =
+          [&](Operation *combineOp, Value src,
+              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
+            if (llvm::none_of(combineParDims,
+                              [](auto parDim) { return parDim.isThreadY(); }))
+              return success();
+            bool isWorkerPrivate =
+                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+            return requireThreadY(combineOp, isWorkerPrivate);
+          };
+
+      for (Operation &nestedOp : predicateBlock) {
+        if (acc::PredicateRegionOp nestedPredicate =
+                dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
+          FailureOr<ThreadYRequirement> nestedRequirement =
+              self(self, nestedPredicate.getRegion().front());
+          if (failed(nestedRequirement))
+            return failure();
+          // An active descendant must not be excluded by this region's
+          // predicate. Inactive descendants apply their own predicate.
+          if (nestedRequirement->active &&
+              failed(requireThreadY(&nestedOp, /*active=*/true)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineOp combineOp =
+                dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineOp, combineOp.getSrcMemref(),
+                  getReductionCombineParDims(combineOp))))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionCombineRegionOp combineRegionOp =
+                dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
+          if (failed(applyCombineRequirement(
+                  combineRegionOp, combineRegionOp.getSrcVar(),
+                  getReductionCombineParDims(combineRegionOp))))
+            return failure();
+          continue;
+        }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          bool threadYActive = false;
+          if (acc::PrivateLocalOp privateLocal =
+                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+            if (acc::GPUParallelDimsAttr parDims =
+                    getPrivatizeOp(privateLocal, computeRegion)
+                        .getParDimsAttr()) {
+              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
+                return parDim.isThreadY();
+              });
+            }
+          }
+          if (failed(requireThreadY(storeOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
+                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
+          bool threadYActive =
+              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
+                           [](auto parDim) { return parDim.isThreadY(); });
+          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
+            return failure();
+          continue;
+        }
+        if (!isMemoryEffectFree(&nestedOp) &&
+            failed(requireThreadY(&nestedOp, /*active=*/false)))
+          return failure();
+      }
+      return requirement;
+    };
+
+    FailureOr<ThreadYRequirement> threadYRequirement =
+        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
+    if (failed(threadYRequirement))
+      return {};
+
     auto applyCombineParDims =
-        [&](Operation *combineOp, Value src,
-            ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          bool isWorkerPrivate =
-              getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
+        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
           for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (parDim.isThreadY()) {
-              if (combineThreadYActive &&
-                  *combineThreadYActive != isWorkerPrivate) {
-                (void)accSupport.emitNYI(
-                    combineOp->getLoc(),
-                    "mixed worker-private and non-worker-private reduction "
-                    "combines require incompatible ThreadY predication");
-                hasFailed = true;
-                return failure();
-              }
-              combineThreadYActive = isWorkerPrivate;
-            } else {
+            if (!parDim.isThreadY())
               mlir::acc::removeParDim(ancestorParDims, parDim);
-            }
           }
           return success();
         };
@@ -1302,14 +1386,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
         if (failed(applyCombineParDims(
-                reductionCombineOp, reductionCombineOp.getSrcMemref(),
                 getReductionCombineParDims(reductionCombineOp))))
           return WalkResult::interrupt();
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
         if (failed(applyCombineParDims(
-                combineRegionOp, combineRegionOp.getSrcVar(),
                 getReductionCombineParDims(combineRegionOp))))
           return WalkResult::interrupt();
       }
@@ -1325,10 +1407,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (combineThreadYActive) {
+    if (threadYRequirement->active || threadYRequirement->inactive) {
       mlir::acc::GPUParallelDimAttr threadY =
           mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (*combineThreadYActive)
+      if (threadYRequirement->active)
         mlir::acc::insertParDim(ancestorParDims, threadY);
       else
         mlir::acc::removeParDim(ancestorParDims, threadY);
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a0d381740c034..e80b5afa26d20 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -30,7 +30,7 @@ func.func @mixed_scope_worker_reduction_combine(
         acc.predicate_region {
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          // expected-error at +1 {{mixed worker-private and non-worker-private reduction combines require incompatible ThreadY predication}}
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -41,3 +41,41 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_single_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c7_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          memref.store %c7_i32, %result_arg[] : memref<i32>
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index b2fd246d33d1b..afd6108d6e670 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -93,3 +93,49 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+// Nested predicate regions choose their ThreadY predicates independently. The
+// outer region must keep ThreadY active so it does not exclude worker rows
+// before the nested worker-private combine is reached.
+func.func @nested_worker_reduction_combines(
+    %other: memref<i32>, %result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %other_arg = %other,
+            %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        acc.predicate_region {
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          acc.predicate_region {
+            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 477f10e0472195d6bb55598bf364a1655682ab6f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:04:51 -0700
Subject: [PATCH 04/22] [OpenACC] Propagate nested ThreadY requirements

Analyze control-flow regions recursively and recognize aliased worker-private stores so their worker rows remain active.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 32 ++++---
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 86 +++++++++++++++++++
 2 files changed, 107 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 34131dc3753a3..98c697f52668e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1310,17 +1310,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = false;
-          if (acc::PrivateLocalOp privateLocal =
-                  storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-            if (acc::GPUParallelDimsAttr parDims =
-                    getPrivatizeOp(privateLocal, computeRegion)
-                        .getParDimsAttr()) {
-              threadYActive = llvm::any_of(parDims.getArray(), [](auto parDim) {
-                return parDim.isThreadY();
-              });
-            }
-          }
+          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
+                               PrivateMemScope::Worker;
           if (failed(requireThreadY(storeOp, threadYActive)))
             return failure();
           continue;
@@ -1334,6 +1325,25 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
             return failure();
           continue;
         }
+        if (nestedOp.getNumRegions() != 0) {
+          for (Region &region : nestedOp.getRegions()) {
+            for (Block &nestedBlock : region) {
+              FailureOr<ThreadYRequirement> nestedRequirement =
+                  self(self, nestedBlock);
+              if (failed(nestedRequirement))
+                return failure();
+              // Other region-bearing operations do not introduce independent
+              // ACC predication, so both requirements apply to this region.
+              if (nestedRequirement->active &&
+                  failed(requireThreadY(&nestedOp, /*active=*/true)))
+                return failure();
+              if (nestedRequirement->inactive &&
+                  failed(requireThreadY(&nestedOp, /*active=*/false)))
+                return failure();
+            }
+          }
+          continue;
+        }
         if (!isMemoryEffectFree(&nestedOp) &&
             failed(requireThreadY(&nestedOp, /*active=*/false)))
           return failure();
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index afd6108d6e670..8a2dd6bd470f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -97,6 +97,12 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.
+// CHECK-LABEL: func.func @nested_worker_reduction_combines
+// CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
+// CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[NESTED_TX_ZERO]]
 func.func @nested_worker_reduction_combines(
     %other: memref<i32>, %result: memref<i32>) {
   %c1 = arith.constant 1 : index
@@ -139,3 +145,83 @@ func.func @nested_worker_reduction_combines(
   }
   return
 }
+
+// CHECK-LABEL: func.func @worker_combine_in_scf_if
+// CHECK: gpu.launch {{.*}} threads([[IF_TX:%[^,]+]], [[IF_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[IF_TY]]
+// CHECK: %[[IF_TX_ZERO:.*]] = arith.cmpi eq, [[IF_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[IF_TX_ZERO]]
+func.func @worker_combine_in_scf_if(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          scf.if %true {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @aliased_worker_store
+// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
+// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
+// CHECK-NOT: arith.andi
+// CHECK: scf.if %[[ALIAS_TX_ZERO]]
+// CHECK: memref.store
+// CHECK: acc.atomic.update
+func.func @aliased_worker_store(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2e234b5cb4dac8aa7cf1ab07b63984b7cb9f329f Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:16:55 -0700
Subject: [PATCH 05/22] [OpenACC] Account for predicate operation effects

Preserve all privatized ThreadY dimensions through aliases and reject intrinsically effecting region operations that cannot share worker predication.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 33 +++++++++++-----
 ...-worker-reduction-combine-mixed-scope.mlir | 38 +++++++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 36 ++++++++++++++++++
 3 files changed, 97 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98c697f52668e..20e8977c907f3 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1268,15 +1268,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         requirement.inactive |= !active;
         return success();
       };
+      auto hasThreadYPrivateStorage = [&](Value memref) {
+        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
+        if (!privatize)
+          return false;
+        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
+                 return parDim.isThreadY();
+               });
+      };
       auto applyCombineRequirement =
           [&](Operation *combineOp, Value src,
               ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
             if (llvm::none_of(combineParDims,
                               [](auto parDim) { return parDim.isThreadY(); }))
               return success();
-            bool isWorkerPrivate =
-                getPrivateScopeForMemref(src) == PrivateMemScope::Worker;
-            return requireThreadY(combineOp, isWorkerPrivate);
+            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
           };
 
       for (Operation &nestedOp : predicateBlock) {
@@ -1310,9 +1317,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          bool threadYActive = getPrivateScopeForMemref(storeOp.getMemref()) ==
-                               PrivateMemScope::Worker;
-          if (failed(requireThreadY(storeOp, threadYActive)))
+          if (failed(requireThreadY(
+                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
             return failure();
           continue;
         }
@@ -1326,6 +1332,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
+          bool hasOwnEffects = true;
+          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
+            hasOwnEffects = !effectOp.hasNoEffect();
+          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
+            hasOwnEffects = false;
+          }
+          if (hasOwnEffects &&
+              failed(requireThreadY(&nestedOp, /*active=*/false)))
+            return failure();
           for (Region &region : nestedOp.getRegions()) {
             for (Block &nestedBlock : region) {
               FailureOr<ThreadYRequirement> nestedRequirement =
@@ -1368,10 +1383,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (auto privateLocalOp =
-                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
-          acc::PrivatizeOp privatizeOp =
-              getPrivatizeOp(privateLocalOp, computeRegion);
+        if (acc::PrivatizeOp privatizeOp =
+                getPrivatizeForMemref(storeOp.getMemref())) {
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index e80b5afa26d20..19e1b3106d966 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -79,3 +79,41 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c1_i32 = arith.constant 1 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.atomic.update %result_arg : memref<i32> {
+          ^bb0(%current: i32):
+            %next = arith.addi %current, %c1_i32 : i32
+            acc.yield %next : i32
+          }
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 8a2dd6bd470f8..ba8f16d3966df 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -225,3 +225,39 @@ func.func @aliased_worker_store(%result: memref<i32>) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @aliased_thread_store
+// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
+// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
+// CHECK: memref.store
+func.func @aliased_thread_store() {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private)
+        : (!acc.private_type<memref<i32>>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c7_i32 = arith.constant 7 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %cast = memref.cast %local : memref<i32> to memref<i32>
+        acc.predicate_region {
+          memref.store %c7_i32, %cast[] : memref<i32>
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From ae0d329822ede9a6d80731ebd45abd7e520ef5ae Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:26:21 -0700
Subject: [PATCH 06/22] [OpenACC] Require proven private aliases

Follow only alias-preserving view operations and treat every non-private combine as ThreadY-inactive to avoid broadening ambiguous global effects.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 45 +++++++++----------
 ...-worker-reduction-combine-mixed-scope.mlir |  6 ++-
 2 files changed, 25 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 20e8977c907f3..9e9114e9aa174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1277,14 +1277,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                  return parDim.isThreadY();
                });
       };
-      auto applyCombineRequirement =
-          [&](Operation *combineOp, Value src,
-              ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-            if (llvm::none_of(combineParDims,
-                              [](auto parDim) { return parDim.isThreadY(); }))
-              return success();
-            return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
-          };
+      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
+        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
+      };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
@@ -1302,17 +1297,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineOp, combineOp.getSrcMemref(),
-                  getReductionCombineParDims(combineOp))))
+          if (failed(
+                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
             return failure();
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(
-                  combineRegionOp, combineRegionOp.getSrcVar(),
-                  getReductionCombineParDims(combineRegionOp))))
+          if (failed(applyCombineRequirement(combineRegionOp,
+                                             combineRegionOp.getSrcVar())))
             return failure();
           continue;
         }
@@ -1893,18 +1886,22 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  llvm::SmallVector<Value, 8> worklist{memref};
-  llvm::SmallPtrSet<Value, 8> seen;
-  while (!worklist.empty()) {
-    Value v = worklist.pop_back_val();
-    if (!seen.insert(v).second)
-      continue;
-    Operation *def = v.getDefiningOp();
-    if (!def)
-      continue;
+  Value current = memref;
+  while (Operation *def = current.getDefiningOp()) {
     if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
       return privateLocal;
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+      current = viewLike.getViewSource();
+      continue;
+    }
+    if (acc::PartialEntityAccessOpInterface partialAccess =
+            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
+      current = partialAccess.getBaseEntity();
+      continue;
+    }
+    // Do not follow arbitrary operands: multi-source operations such as
+    // arith.select do not prove that their result aliases private storage.
+    return nullptr;
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 19e1b3106d966..1275e4c77890a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -32,7 +32,7 @@ func.func @mixed_scope_worker_reduction_combine(
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+              {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
@@ -59,15 +59,17 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
+      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %result_arg[] : memref<i32>
+          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}

>From 3ed1de6f3b95aee6516db811bbd62dc4c54f9335 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:38:25 -0700
Subject: [PATCH 07/22] [OpenACC] Restrict ThreadY predicate broadening

Preserve legacy predicates unless a proven worker combine requires ThreadY, and reject broadening around writes, unknown effects, or incompatible combines.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 223 +++++++++---------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir |  78 +-----
 2 files changed, 107 insertions(+), 194 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 9e9114e9aa174..50635dde561db 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,138 +1246,122 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
-    struct ThreadYRequirement {
-      bool active = false;
-      bool inactive = false;
+    struct ThreadYBroadeningInfo {
+      bool hasActiveWorkerCombine = false;
+      bool hasExplicitInactiveCombine = false;
+      bool hasBroadeningConflict = false;
+      Operation *diagnosticOp = nullptr;
     };
-    auto analyzeThreadYRequirements =
-        [&](auto &&self,
-            Block &predicateBlock) -> FailureOr<ThreadYRequirement> {
-      ThreadYRequirement requirement;
-      auto requireThreadY = [&](Operation *op, bool active) {
-        if ((active && requirement.inactive) ||
-            (!active && requirement.active)) {
-          (void)accSupport.emitNYI(
-              op->getLoc(),
-              "operations in the same predicate region require incompatible "
-              "ThreadY predication");
-          hasFailed = true;
-          return failure();
+    auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
+                                  const ThreadYBroadeningInfo &src) {
+      dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
+      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
+      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      if (!dst.diagnosticOp)
+        dst.diagnosticOp = src.diagnosticOp;
+    };
+    auto isProvenWorkerPrivate = [&](Value memref) {
+      Value current = memref;
+      while (Operation *def = current.getDefiningOp()) {
+        if (acc::PrivateLocalOp privateLocal =
+                dyn_cast<acc::PrivateLocalOp>(def)) {
+          return getPrivateMemScope(getPrivatizeOp(
+                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+        }
+        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
+          current = viewLike.getViewSource();
+          continue;
+        }
+        break;
+      }
+      return false;
+    };
+    auto hasUnsafeOwnEffects = [](Operation *op) {
+      if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
+        SmallVector<MemoryEffects::EffectInstance> effects;
+        effectOp.getEffects(effects);
+        return llvm::any_of(effects, [](const auto &effect) {
+          return !isa<MemoryEffects::Read>(effect.getEffect());
+        });
+      }
+      return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
+    };
+    auto analyzeThreadYBroadening =
+        [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
+      ThreadYBroadeningInfo info;
+      auto classifyCombine = [&](Operation *combineOp, Value src,
+                                 ArrayRef<GPUParallelDimAttr> parDims) {
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        if (hasThreadY && isProvenWorkerPrivate(src)) {
+          info.hasActiveWorkerCombine = true;
+        } else {
+          info.hasExplicitInactiveCombine = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = combineOp;
         }
-        requirement.active |= active;
-        requirement.inactive |= !active;
-        return success();
-      };
-      auto hasThreadYPrivateStorage = [&](Value memref) {
-        acc::PrivatizeOp privatize = getPrivatizeForMemref(memref);
-        if (!privatize)
-          return false;
-        acc::GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-        return parDims && llvm::any_of(parDims.getArray(), [](auto parDim) {
-                 return parDim.isThreadY();
-               });
-      };
-      auto applyCombineRequirement = [&](Operation *combineOp, Value src) {
-        return requireThreadY(combineOp, hasThreadYPrivateStorage(src));
       };
 
       for (Operation &nestedOp : predicateBlock) {
         if (acc::PredicateRegionOp nestedPredicate =
                 dyn_cast<acc::PredicateRegionOp>(nestedOp)) {
-          FailureOr<ThreadYRequirement> nestedRequirement =
+          ThreadYBroadeningInfo nestedInfo =
               self(self, nestedPredicate.getRegion().front());
-          if (failed(nestedRequirement))
-            return failure();
-          // An active descendant must not be excluded by this region's
-          // predicate. Inactive descendants apply their own predicate.
-          if (nestedRequirement->active &&
-              failed(requireThreadY(&nestedOp, /*active=*/true)))
-            return failure();
+          // A worker-active descendant must reach its own predicate. Other
+          // requirements are enforced within the nested predicate region.
+          info.hasActiveWorkerCombine |= nestedInfo.hasActiveWorkerCombine;
           continue;
         }
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
-          if (failed(
-                  applyCombineRequirement(combineOp, combineOp.getSrcMemref())))
-            return failure();
+          classifyCombine(combineOp, combineOp.getSrcMemref(),
+                          getReductionCombineParDims(combineOp));
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
-          if (failed(applyCombineRequirement(combineRegionOp,
-                                             combineRegionOp.getSrcVar())))
-            return failure();
-          continue;
-        }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (failed(requireThreadY(
-                  storeOp, hasThreadYPrivateStorage(storeOp.getMemref()))))
-            return failure();
-          continue;
-        }
-        if (acc::ReductionAccumulateArrayOp accumulateArrayOp =
-                dyn_cast<acc::ReductionAccumulateArrayOp>(nestedOp)) {
-          bool threadYActive =
-              llvm::any_of(accumulateArrayOp.getParDims().getArray(),
-                           [](auto parDim) { return parDim.isThreadY(); });
-          if (failed(requireThreadY(accumulateArrayOp, threadYActive)))
-            return failure();
+          classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
+                          getReductionCombineParDims(combineRegionOp));
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          bool hasOwnEffects = true;
-          if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(&nestedOp)) {
-            hasOwnEffects = !effectOp.hasNoEffect();
-          } else if (nestedOp.hasTrait<OpTrait::HasRecursiveMemoryEffects>()) {
-            hasOwnEffects = false;
+          if (hasUnsafeOwnEffects(&nestedOp)) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = &nestedOp;
           }
-          if (hasOwnEffects &&
-              failed(requireThreadY(&nestedOp, /*active=*/false)))
-            return failure();
           for (Region &region : nestedOp.getRegions()) {
-            for (Block &nestedBlock : region) {
-              FailureOr<ThreadYRequirement> nestedRequirement =
-                  self(self, nestedBlock);
-              if (failed(nestedRequirement))
-                return failure();
-              // Other region-bearing operations do not introduce independent
-              // ACC predication, so both requirements apply to this region.
-              if (nestedRequirement->active &&
-                  failed(requireThreadY(&nestedOp, /*active=*/true)))
-                return failure();
-              if (nestedRequirement->inactive &&
-                  failed(requireThreadY(&nestedOp, /*active=*/false)))
-                return failure();
-            }
+            for (Block &nestedBlock : region)
+              mergeBroadeningInfo(info, self(self, nestedBlock));
           }
           continue;
         }
-        if (!isMemoryEffectFree(&nestedOp) &&
-            failed(requireThreadY(&nestedOp, /*active=*/false)))
-          return failure();
+        if (hasUnsafeOwnEffects(&nestedOp)) {
+          info.hasBroadeningConflict = true;
+          if (!info.diagnosticOp)
+            info.diagnosticOp = &nestedOp;
+        }
       }
-      return requirement;
+      return info;
     };
 
-    FailureOr<ThreadYRequirement> threadYRequirement =
-        analyzeThreadYRequirements(analyzeThreadYRequirements, *block);
-    if (failed(threadYRequirement))
-      return {};
+    ThreadYBroadeningInfo threadYInfo =
+        analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
     auto applyCombineParDims =
         [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims) {
-            if (!parDim.isThreadY())
-              mlir::acc::removeParDim(ancestorParDims, parDim);
-          }
+          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
+            mlir::acc::removeParDim(ancestorParDims, parDim);
           return success();
         };
     block->walk([&](Operation *op) -> WalkResult {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
-        if (acc::PrivatizeOp privatizeOp =
-                getPrivatizeForMemref(storeOp.getMemref())) {
+        if (auto privateLocalOp =
+                storeOp.getMemref().getDefiningOp<acc::PrivateLocalOp>()) {
+          acc::PrivatizeOp privatizeOp =
+              getPrivatizeOp(privateLocalOp, computeRegion);
           if (mlir::acc::GPUParallelDimsAttr parDimsAttr =
                   privatizeOp.getParDimsAttr()) {
             for (auto parDim : parDimsAttr.getArray())
@@ -1423,13 +1407,22 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return WalkResult::advance();
     });
-    if (threadYRequirement->active || threadYRequirement->inactive) {
-      mlir::acc::GPUParallelDimAttr threadY =
-          mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-      if (threadYRequirement->active)
-        mlir::acc::insertParDim(ancestorParDims, threadY);
-      else
-        mlir::acc::removeParDim(ancestorParDims, threadY);
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
+    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+      if (threadYInfo.hasExplicitInactiveCombine ||
+          threadYInfo.hasBroadeningConflict) {
+        Operation *diagnosticOp =
+            threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
+        (void)accSupport.emitNYI(
+            diagnosticOp->getLoc(),
+            "operations in the same predicate region require incompatible "
+            "ThreadY predication");
+        hasFailed = true;
+        return {};
+      }
+      mlir::acc::insertParDim(ancestorParDims, threadY);
     }
   }
 
@@ -1886,22 +1879,18 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
 
 /// Walks back from a memref use to its defining `acc.private_local`, if any.
 static acc::PrivateLocalOp getPrivateLocalForMemref(Value memref) {
-  Value current = memref;
-  while (Operation *def = current.getDefiningOp()) {
-    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
-      return privateLocal;
-    if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-      current = viewLike.getViewSource();
+  llvm::SmallVector<Value, 8> worklist{memref};
+  llvm::SmallPtrSet<Value, 8> seen;
+  while (!worklist.empty()) {
+    Value v = worklist.pop_back_val();
+    if (!seen.insert(v).second)
       continue;
-    }
-    if (acc::PartialEntityAccessOpInterface partialAccess =
-            dyn_cast<acc::PartialEntityAccessOpInterface>(def)) {
-      current = partialAccess.getBaseEntity();
+    Operation *def = v.getDefiningOp();
+    if (!def)
       continue;
-    }
-    // Do not follow arbitrary operands: multi-source operations such as
-    // arith.select do not prove that their result aliases private storage.
-    return nullptr;
+    if (acc::PrivateLocalOp privateLocal = dyn_cast<acc::PrivateLocalOp>(def))
+      return privateLocal;
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
   }
   return nullptr;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index ba8f16d3966df..0de030eda1e59 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -184,80 +185,3 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
-
-// CHECK-LABEL: func.func @aliased_worker_store
-// CHECK: gpu.launch {{.*}} threads([[ALIAS_TX:%[^,]+]], [[ALIAS_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[ALIAS_TY]]
-// CHECK: %[[ALIAS_TX_ZERO:.*]] = arith.cmpi eq, [[ALIAS_TX]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[ALIAS_TX_ZERO]]
-// CHECK: memref.store
-// CHECK: acc.atomic.update
-func.func @aliased_worker_store(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
-// CHECK-LABEL: func.func @aliased_thread_store
-// CHECK: gpu.launch {{.*}} threads([[THREAD_TX:%[^,]+]], [[THREAD_TY:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TX]]
-// CHECK-NOT: arith.cmpi eq, [[THREAD_TY]]
-// CHECK: memref.store
-func.func @aliased_thread_store() {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y, thread_x]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private)
-        : (!acc.private_type<memref<i32>>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c7_i32 = arith.constant 7 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %cast = memref.cast %local : memref<i32> to memref<i32>
-        acc.predicate_region {
-          memref.store %c7_i32, %cast[] : memref<i32>
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}

>From 30926fc3c3e3a6d6b9fe8e3fcb8442319849edf1 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:35 -0700
Subject: [PATCH 08/22] [OpenACC][test] Match broadening conflict locations

Expect diagnostics on the side effect that makes worker broadening unsafe.
---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 1275e4c77890a..7f22fd20aa89e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -69,8 +69,8 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c7_i32, %selected[] : memref<i32>
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          memref.store %c7_i32, %selected[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -103,12 +103,12 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
         acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.atomic.update %result_arg : memref<i32> {
           ^bb0(%current: i32):
             %next = arith.addi %current, %c1_i32 : i32
             acc.yield %next : i32
           }
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 55293fcdfa6ce6f49e61e43293e80df04bfec368 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:51:30 -0700
Subject: [PATCH 09/22] [OpenACC] Require safe predicate speculation

Treat reads and non-speculatable operations as broadening conflicts so enabling worker rows cannot introduce invalid memory accesses.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp  | 13 +++++--------
 ...to-gpu-worker-reduction-combine-mixed-scope.mlir |  6 ++----
 .../acc-cg-to-gpu-worker-reduction-combine.mlir     |  1 -
 3 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 50635dde561db..250c2be09df85 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1276,13 +1276,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       }
       return false;
     };
-    auto hasUnsafeOwnEffects = [](Operation *op) {
+    auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
-        SmallVector<MemoryEffects::EffectInstance> effects;
-        effectOp.getEffects(effects);
-        return llvm::any_of(effects, [](const auto &effect) {
-          return !isa<MemoryEffects::Read>(effect.getEffect());
-        });
+        return !effectOp.hasNoEffect();
       }
       return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
     };
@@ -1325,7 +1321,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
-          if (hasUnsafeOwnEffects(&nestedOp)) {
+          if (hasOwnEffects(&nestedOp)) {
             info.hasBroadeningConflict = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
@@ -1336,7 +1332,8 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
           continue;
         }
-        if (hasUnsafeOwnEffects(&nestedOp)) {
+        if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
+            !isPure(&nestedOp)) {
           info.hasBroadeningConflict = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 7f22fd20aa89e..290786ab7c1ce 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,7 +42,7 @@ func.func @mixed_scope_worker_reduction_combine(
   return
 }
 
-func.func @worker_combine_with_single_store(%result: memref<i32>) {
+func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -59,18 +59,16 @@ func.func @worker_combine_with_single_store(%result: memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c7_i32 = arith.constant 7 : i32
-      %false = arith.constant false
       scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
         %local = acc.private_local %private_arg
             : (!acc.private_type<memref<i32>>) -> memref<i32>
-        %selected = arith.select %false, %local, %result_arg : memref<i32>
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c7_i32, %local[] : memref<i32>
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          memref.store %c7_i32, %selected[] : memref<i32>
+          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 0de030eda1e59..fb1a0256d2a38 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          %unused = memref.load %result_arg[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From 300b3dc6a9334c5bd8b47daf1c1371bb6f5db806 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:59:06 -0700
Subject: [PATCH 10/22] [OpenACC] Reject worker combine regions

Keep generic combine regions single-row because their non-atomic load/store lowering is unsafe across worker rows.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 14 ++++--
 ...-worker-reduction-combine-mixed-scope.mlir | 37 ++++++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 48 -------------------
 3 files changed, 48 insertions(+), 51 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 250c2be09df85..3697ae7405c9f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1286,11 +1286,17 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
       auto classifyCombine = [&](Operation *combineOp, Value src,
-                                 ArrayRef<GPUParallelDimAttr> parDims) {
+                                 ArrayRef<GPUParallelDimAttr> parDims,
+                                 bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
         if (hasThreadY && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
+          if (!supportsWorkerBroadening) {
+            info.hasBroadeningConflict = true;
+            if (!info.diagnosticOp)
+              info.diagnosticOp = combineOp;
+          }
         } else {
           info.hasExplicitInactiveCombine = true;
           if (!info.diagnosticOp)
@@ -1311,13 +1317,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (acc::ReductionCombineOp combineOp =
                 dyn_cast<acc::ReductionCombineOp>(nestedOp)) {
           classifyCombine(combineOp, combineOp.getSrcMemref(),
-                          getReductionCombineParDims(combineOp));
+                          getReductionCombineParDims(combineOp),
+                          /*supportsWorkerBroadening=*/true);
           continue;
         }
         if (acc::ReductionCombineRegionOp combineRegionOp =
                 dyn_cast<acc::ReductionCombineRegionOp>(nestedOp)) {
           classifyCombine(combineRegionOp, combineRegionOp.getSrcVar(),
-                          getReductionCombineParDims(combineRegionOp));
+                          getReductionCombineParDims(combineRegionOp),
+                          /*supportsWorkerBroadening=*/false);
           continue;
         }
         if (nestedOp.getNumRegions() != 0) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 290786ab7c1ce..fffbfcb461c56 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -117,3 +117,40 @@ func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   }
   return
 }
+
+func.func @worker_reduction_combine_region(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine_region %local into %result_arg : memref<i32> {
+            %lhs = memref.load %result_arg[] : memref<i32>
+            %rhs = memref.load %local[] : memref<i32>
+            %sum = arith.addi %lhs, %rhs : i32
+            memref.store %sum, %result_arg[] : memref<i32>
+            acc.yield
+          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fb1a0256d2a38..686e8fd233980 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,54 +46,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// CHECK-LABEL: func.func @worker_reduction_combine_region
-// CHECK: gpu.launch {{.*}} threads([[REGION_TID_X:%[^,]+]], [[REGION_TID_Y:%[^,]+]],
-// CHECK-NOT: arith.cmpi eq, [[REGION_TID_Y]]
-// CHECK: %[[REGION_IS_X_ZERO:.*]] = arith.cmpi eq, [[REGION_TID_X]],
-// CHECK-NOT: arith.andi
-// CHECK: scf.if %[[REGION_IS_X_ZERO]]
-// CHECK: arith.addi
-
-func.func @worker_reduction_combine_region(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
-          memref.store %c0_i32, %local[] : memref<i32>
-          scf.reduce
-        } {acc.par_dims = #acc<par_dims[thread_y]>}
-        acc.predicate_region {
-          acc.reduction_combine_region %local into %result_arg : memref<i32> {
-            %lhs = memref.load %result_arg[] : memref<i32>
-            %rhs = memref.load %local[] : memref<i32>
-            %sum = arith.addi %lhs, %rhs : i32
-            memref.store %sum, %result_arg[] : memref<i32>
-            acc.yield
-          } {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
 // Nested predicate regions choose their ThreadY predicates independently. The
 // outer region must keep ThreadY active so it does not exclude worker rows
 // before the nested worker-private combine is reached.

>From cbc18adba26e38ff6bebd01bf252112da4412887 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:03:05 -0700
Subject: [PATCH 11/22] [OpenACC] Require atomic worker combines

Broaden ThreadY only when block dimensions select the atomic combine lowering; reject ThreadY-only shared updates.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  4 ++-
 ...-worker-reduction-combine-mixed-scope.mlir | 32 +++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 3697ae7405c9f..76f942a2730f9 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1290,7 +1290,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                                  bool supportsWorkerBroadening) {
         bool hasThreadY = llvm::any_of(
             parDims, [](auto parDim) { return parDim.isThreadY(); });
-        if (hasThreadY && isProvenWorkerPrivate(src)) {
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
             info.hasBroadeningConflict = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index fffbfcb461c56..5758702a3b92e 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -154,3 +154,35 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   }
   return
 }
+
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 2f6b6b2e74b62114ad20366b768685e6af8bc000 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:06:26 -0700
Subject: [PATCH 12/22] [OpenACC][test] Check non-atomic worker predication

Verify ThreadY-only combines retain single-row predication while unsafe combine regions remain rejected.
---
 ...-worker-reduction-combine-mixed-scope.mlir | 31 ---------------
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 39 +++++++++++++++++++
 2 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 5758702a3b92e..3d15b4ca4a59f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -155,34 +155,3 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
   return
 }
 
-func.func @thread_y_only_worker_combine(%result: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c4 = arith.constant 4 : index
-  %c32 = arith.constant 32 : index
-  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
-  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
-  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
-  acc.kernel_environment {
-    %private = acc.privatize [#acc<par_dims[thread_y]>]
-        : () -> !acc.private_type<memref<i32>>
-    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
-    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>) {
-      %c0 = arith.constant 0 : index
-      %c1_inner = arith.constant 1 : index
-      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
-        %local = acc.private_local %private_arg
-            : (!acc.private_type<memref<i32>>) -> memref<i32>
-        acc.predicate_region {
-          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
-          acc.reduction_combine %local into %result_arg <add> : memref<i32>
-              {acc.par_dims = #acc<par_dims[thread_y]>}
-        }
-        scf.reduce
-      } {acc.par_dims = #acc<par_dims[block_y]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 686e8fd233980..901b23b25316f 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -136,3 +136,42 @@ func.func @worker_combine_in_scf_if(%result: memref<i32>) {
   }
   return
 }
+
+// Without a block dimension the combine lowers to a non-atomic update, so
+// ThreadY must retain the legacy single-row predicate.
+// CHECK-LABEL: func.func @thread_y_only_worker_combine
+// CHECK: gpu.launch {{.*}} threads([[ONLY_TX:%[^,]+]], [[ONLY_TY:%[^,]+]],
+// CHECK: %[[ONLY_TY_ZERO:.*]] = arith.cmpi eq, [[ONLY_TY]],
+// CHECK: %[[ONLY_TX_ZERO:.*]] = arith.cmpi eq, [[ONLY_TX]],
+// CHECK: %[[ONLY_ROW_ZERO:.*]] = arith.andi %[[ONLY_TX_ZERO]], %[[ONLY_TY_ZERO]]
+// CHECK: scf.if %[[ONLY_ROW_ZERO]]
+// CHECK: memref.store
+func.func @thread_y_only_worker_combine(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        acc.predicate_region {
+          acc.reduction_combine %local into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 88026c11dfb0376be0ad438de4a530d99d821119 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:10:52 -0700
Subject: [PATCH 13/22] [OpenACC][test] Remove trailing blank line

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index 3d15b4ca4a59f..dc634cff25b07 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -41,7 +41,6 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
-
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 984e8f2ce4366cb23a6104b6aa965783f2380a49 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:11:22 -0700
Subject: [PATCH 14/22] [OpenACC][test] Normalize test file ending

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir      | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dc634cff25b07..dba8057d48e95 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -78,7 +78,6 @@ func.func @worker_combine_with_read(%result: memref<i32>) {
   }
   return
 }
-
 func.func @worker_combine_with_atomic_update(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index

>From 8f6c707f4e58523929b8482505946acdd9dc9991 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:12:19 -0700
Subject: [PATCH 15/22] [OpenACC][test] Finish file ending cleanup

---
 .../acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir    | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index dba8057d48e95..a641fe74867e4 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -151,5 +151,4 @@ func.func @worker_reduction_combine_region(%result: memref<i32>) {
     } {origin = "acc.parallel"}
   }
   return
-}
-
+}
\ No newline at end of file

>From f629ee527f4b56ac33d85867d488a793af7cfc09 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:43:52 -0700
Subject: [PATCH 16/22] [OpenACC] Localize worker combine classification

Keep private-memory scope behavior unchanged and remove redundant failure plumbing from combine dimension handling.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 37 ++++++++++---------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 76f942a2730f9..98258dc622e3c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1265,8 +1265,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       while (Operation *def = current.getDefiningOp()) {
         if (acc::PrivateLocalOp privateLocal =
                 dyn_cast<acc::PrivateLocalOp>(def)) {
-          return getPrivateMemScope(getPrivatizeOp(
-                     privateLocal, computeRegion)) == PrivateMemScope::Worker;
+          acc::PrivatizeOp privatize =
+              getPrivatizeOp(privateLocal, computeRegion);
+          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+          if (!parDims)
+            return false;
+          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadX();
+          });
+          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
+            return parDim.isThreadY();
+          });
+          return hasThreadY && !hasThreadX;
         }
         if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
           current = viewLike.getViewSource();
@@ -1355,13 +1365,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     ThreadYBroadeningInfo threadYInfo =
         analyzeThreadYBroadening(analyzeThreadYBroadening, *block);
 
-    auto applyCombineParDims =
-        [&](ArrayRef<mlir::acc::GPUParallelDimAttr> combineParDims) {
-          for (mlir::acc::GPUParallelDimAttr parDim : combineParDims)
-            mlir::acc::removeParDim(ancestorParDims, parDim);
-          return success();
-        };
-    block->walk([&](Operation *op) -> WalkResult {
+    block->walk([&](Operation *op) {
       // Check stores to acc.private_local - add the privatize's par_dims
       // as active dims so predication is correct for per-worker/gang memory.
       if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(op)) {
@@ -1392,15 +1396,15 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
       // kernel and loop in combined constructs.
       if (acc::ReductionCombineOp reductionCombineOp =
               dyn_cast<acc::ReductionCombineOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(reductionCombineOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(reductionCombineOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       if (acc::ReductionCombineRegionOp combineRegionOp =
               dyn_cast<acc::ReductionCombineRegionOp>(op)) {
-        if (failed(applyCombineParDims(
-                getReductionCombineParDims(combineRegionOp))))
-          return WalkResult::interrupt();
+        for (GPUParallelDimAttr parDim :
+             getReductionCombineParDims(combineRegionOp))
+          mlir::acc::removeParDim(ancestorParDims, parDim);
       }
       // An array accumulate reduces across its par_dims via gpu.all_reduce, so
       // all those threads must execute it - treat them as active (unlike the
@@ -1412,7 +1416,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           mlir::acc::insertParDim(ancestorParDims, parDim);
         }
       }
-      return WalkResult::advance();
     });
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
@@ -1877,7 +1880,7 @@ ACCCGToGPULowering::getPrivateMemScope(acc::PrivatizeOp privatizeOp) {
   }
   if (hasThreadX)
     return PrivateMemScope::Thread;
-  if (hasThreadY)
+  if (hasBlock && hasThreadY)
     return PrivateMemScope::Worker;
   if (hasBlock)
     return PrivateMemScope::Gang;

>From edc8abd53cd69917a03f21c72c975e4a26a4ed24 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:50:05 -0700
Subject: [PATCH 17/22] [OpenACC] Merge ThreadY conflict state

Use one flag for all operations that require the legacy inactive ThreadY predicate.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 98258dc622e3c..6a5aebc4db323 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,15 +1248,13 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
-      bool hasExplicitInactiveCombine = false;
-      bool hasBroadeningConflict = false;
+      bool requiresInactiveThreadY = false;
       Operation *diagnosticOp = nullptr;
     };
     auto mergeBroadeningInfo = [](ThreadYBroadeningInfo &dst,
                                   const ThreadYBroadeningInfo &src) {
       dst.hasActiveWorkerCombine |= src.hasActiveWorkerCombine;
-      dst.hasExplicitInactiveCombine |= src.hasExplicitInactiveCombine;
-      dst.hasBroadeningConflict |= src.hasBroadeningConflict;
+      dst.requiresInactiveThreadY |= src.requiresInactiveThreadY;
       if (!dst.diagnosticOp)
         dst.diagnosticOp = src.diagnosticOp;
     };
@@ -1305,12 +1303,12 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         if (hasThreadY && hasBlock && isProvenWorkerPrivate(src)) {
           info.hasActiveWorkerCombine = true;
           if (!supportsWorkerBroadening) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = combineOp;
           }
         } else {
-          info.hasExplicitInactiveCombine = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1342,7 +1340,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
-            info.hasBroadeningConflict = true;
+            info.requiresInactiveThreadY = true;
             if (!info.diagnosticOp)
               info.diagnosticOp = &nestedOp;
           }
@@ -1354,7 +1352,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
         if (!nestedOp.mightHaveTrait<OpTrait::IsTerminator>() &&
             !isPure(&nestedOp)) {
-          info.hasBroadeningConflict = true;
+          info.requiresInactiveThreadY = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = &nestedOp;
         }
@@ -1421,8 +1419,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
     bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
-      if (threadYInfo.hasExplicitInactiveCombine ||
-          threadYInfo.hasBroadeningConflict) {
+      if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
         (void)accSupport.emitNYI(

>From 90cbc61e638708c7e25e37e588e20329fbfb736e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:06:50 -0700
Subject: [PATCH 18/22] [OpenACC] Stabilize ThreadY conflict analysis

Capture structural ThreadY activity before operation-order mutations and cover a trailing worker-private store.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp   |  6 +++---
 ...pu-worker-reduction-combine-mixed-scope.mlir |  1 +
 .../acc-cg-to-gpu-worker-reduction-combine.mlir | 17 +++++------------
 3 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 6a5aebc4db323..09dbbf892ffbd 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1246,6 +1246,9 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   mlir::acc::GPUParallelDimAttr lowestParDim =
       mlir::acc::GPUParallelDimAttr::threadXDim(ctx);
   if (block) {
+    mlir::acc::GPUParallelDimAttr threadY =
+        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
+    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1415,9 +1418,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    mlir::acc::GPUParallelDimAttr threadY =
-        mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a641fe74867e4..a5ffb4bfb59f8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -33,6 +33,7 @@ func.func @mixed_scope_worker_reduction_combine(
           // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
           acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_x]>}
+          memref.store %c0_i32, %local[] : memref<i32>
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 901b23b25316f..4466ad74d1a91 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -46,17 +46,15 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
-// Nested predicate regions choose their ThreadY predicates independently. The
-// outer region must keep ThreadY active so it does not exclude worker rows
-// before the nested worker-private combine is reached.
+// The outer predicate must keep ThreadY active so it does not exclude worker
+// rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines
 // CHECK: gpu.launch {{.*}} threads([[NESTED_TX:%[^,]+]], [[NESTED_TY:%[^,]+]],
 // CHECK-NOT: arith.cmpi eq, [[NESTED_TY]]
 // CHECK: %[[NESTED_TX_ZERO:.*]] = arith.cmpi eq, [[NESTED_TX]],
 // CHECK-NOT: arith.andi
 // CHECK: scf.if %[[NESTED_TX_ZERO]]
-func.func @nested_worker_reduction_combines(
-    %other: memref<i32>, %result: memref<i32>) {
+func.func @nested_worker_reduction_combines(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
   %c32 = arith.constant 32 : index
@@ -67,9 +65,8 @@ func.func @nested_worker_reduction_combines(
     %private = acc.privatize [#acc<par_dims[thread_y]>]
         : () -> !acc.private_type<memref<i32>>
     acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
-        ins(%private_arg = %private, %other_arg = %other,
-            %result_arg = %result)
-        : (!acc.private_type<memref<i32>>, memref<i32>, memref<i32>) {
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
       %c0 = arith.constant 0 : index
       %c1_inner = arith.constant 1 : index
       %c0_i32 = arith.constant 0 : i32
@@ -85,10 +82,6 @@ func.func @nested_worker_reduction_combines(
             acc.reduction_combine %local into %result_arg <add> : memref<i32>
                 {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           }
-          acc.predicate_region {
-            acc.reduction_combine %other_arg into %result_arg <add> : memref<i32>
-                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
-          }
         }
         scf.reduce
       } {acc.par_dims = #acc<par_dims[block_y]>}

>From 9e6fcfb6dd7ff5d68e1e7e6a34b14a7a69763286 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:18 -0700
Subject: [PATCH 19/22] [OpenACC] Allow worker-private stores during combine

Treat proven worker-private stores as row-local so they can share the broadened ThreadY predicate safely.
---
 mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp            | 4 ++++
 .../OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir       | 1 +
 2 files changed, 5 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 09dbbf892ffbd..f78a973da4e5a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1341,6 +1341,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          if (isProvenWorkerPrivate(storeOp.getMemref()))
+            continue;
+        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 4466ad74d1a91..cbfabf28ea891 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,6 +35,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From fcc26cd10cf22d3fb30b46154828ecd7e4af7604 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:43:59 -0700
Subject: [PATCH 20/22] [OpenACC] Diagnose unproven worker aliases

Restrict ThreadY broadening to directly sized worker storage and conservatively reject aliases that may originate from it.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 73 +++++++++++--------
 ...-worker-reduction-combine-mixed-scope.mlir | 39 ++++++++++
 ...cc-cg-to-gpu-worker-reduction-combine.mlir | 41 ++++++++++-
 3 files changed, 123 insertions(+), 30 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f78a973da4e5a..f46a891f08652 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1248,7 +1248,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
   if (block) {
     mlir::acc::GPUParallelDimAttr threadY =
         mlir::acc::GPUParallelDimAttr::threadYDim(ctx);
-    bool baselineThreadYActive = llvm::is_contained(ancestorParDims, threadY);
     struct ThreadYBroadeningInfo {
       bool hasActiveWorkerCombine = false;
       bool requiresInactiveThreadY = false;
@@ -1262,30 +1261,46 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         dst.diagnosticOp = src.diagnosticOp;
     };
     auto isProvenWorkerPrivate = [&](Value memref) {
-      Value current = memref;
-      while (Operation *def = current.getDefiningOp()) {
-        if (acc::PrivateLocalOp privateLocal =
-                dyn_cast<acc::PrivateLocalOp>(def)) {
-          acc::PrivatizeOp privatize =
-              getPrivatizeOp(privateLocal, computeRegion);
-          GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
-          if (!parDims)
-            return false;
-          bool hasThreadX = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadX();
-          });
-          bool hasThreadY = llvm::any_of(parDims.getArray(), [](auto parDim) {
-            return parDim.isThreadY();
-          });
-          return hasThreadY && !hasThreadX;
-        }
-        if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(def)) {
-          current = viewLike.getViewSource();
-          continue;
-        }
-        break;
+      acc::PrivateLocalOp privateLocal =
+          memref.getDefiningOp<acc::PrivateLocalOp>();
+      if (!privateLocal)
+        return false;
+      acc::PrivatizeOp privatize = getPrivatizeOp(privateLocal, computeRegion);
+      GPUParallelDimsAttr parDims = privatize.getParDimsAttr();
+      if (!parDims)
+        return false;
+      bool hasThreadX = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadX(); });
+      bool hasThreadY = llvm::any_of(
+          parDims.getArray(), [](auto parDim) { return parDim.isThreadY(); });
+      return hasThreadY && !hasThreadX;
+    };
+    auto hasWorkerPrivateOrigin = [&](auto &&self, Value value,
+                                      DenseSet<Value> &visited) -> bool {
+      if (!visited.insert(value).second)
+        return false;
+      if (isProvenWorkerPrivate(value))
+        return true;
+      Operation *def = value.getDefiningOp();
+      if (!def) {
+        auto blockArg = dyn_cast<BlockArgument>(value);
+        def = blockArg ? blockArg.getOwner()->getParentOp() : nullptr;
       }
-      return false;
+      if (!def)
+        return false;
+      if (llvm::any_of(def->getOperands(), [&](Value operand) {
+            return self(self, operand, visited);
+          }))
+        return true;
+      return llvm::any_of(def->getRegions(), [&](Region &region) {
+        return llvm::any_of(region, [&](Block &block) {
+          Operation *terminator = block.getTerminator();
+          return terminator &&
+                 llvm::any_of(terminator->getOperands(), [&](Value operand) {
+                   return self(self, operand, visited);
+                 });
+        });
+      });
     };
     auto hasOwnEffects = [](Operation *op) {
       if (auto effectOp = dyn_cast<MemoryEffectOpInterface>(op)) {
@@ -1312,6 +1327,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
           }
         } else {
           info.requiresInactiveThreadY = true;
+          DenseSet<Value> visited;
+          if (hasThreadY && hasBlock &&
+              hasWorkerPrivateOrigin(hasWorkerPrivateOrigin, src, visited))
+            info.hasActiveWorkerCombine = true;
           if (!info.diagnosticOp)
             info.diagnosticOp = combineOp;
         }
@@ -1341,10 +1360,6 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
-        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
-          if (isProvenWorkerPrivate(storeOp.getMemref()))
-            continue;
-        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
@@ -1422,7 +1437,7 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
         }
       }
     });
-    if (threadYInfo.hasActiveWorkerCombine && !baselineThreadYActive) {
+    if (threadYInfo.hasActiveWorkerCombine) {
       if (threadYInfo.requiresInactiveThreadY) {
         Operation *diagnosticOp =
             threadYInfo.diagnosticOp ? threadYInfo.diagnosticOp : op;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
index a5ffb4bfb59f8..8724f1f57bc9b 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine-mixed-scope.mlir
@@ -42,6 +42,45 @@ func.func @mixed_scope_worker_reduction_combine(
   }
   return
 }
+
+func.func @worker_combine_with_unproven_alias(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    // expected-error at +1 {{failed to legalize operation 'acc.compute_region' that was explicitly marked illegal}}
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %true = arith.constant true
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        %alias = scf.if %true -> (memref<i32>) {
+          scf.yield %local : memref<i32>
+        } else {
+          scf.yield %local : memref<i32>
+        }
+        acc.predicate_region {
+          // expected-error at +1 {{operations in the same predicate region require incompatible ThreadY predication}}
+          acc.reduction_combine %alias into %result_arg <add> : memref<i32>
+              {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+        }
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 func.func @worker_combine_with_read(%result: memref<i32>) {
   %c1 = arith.constant 1 : index
   %c4 = arith.constant 4 : index
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index cbfabf28ea891..fd4e12fd039a8 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -35,7 +35,6 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
-          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }
@@ -47,6 +46,46 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
   return
 }
 
+// CHECK-LABEL: func.func @worker_combine_in_thread_y_scope
+// CHECK: gpu.launch {{.*}} threads([[SCOPE_TX:%[^,]+]], [[SCOPE_TY:%[^,]+]],
+// CHECK-NOT: arith.cmpi eq, [[SCOPE_TY]]
+// CHECK: %[[SCOPE_TX_ZERO:.*]] = arith.cmpi eq, [[SCOPE_TX]],
+// CHECK: scf.if %[[SCOPE_TX_ZERO]]
+func.func @worker_combine_in_thread_y_scope(%result: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c32 = arith.constant 32 : index
+  %block_y = acc.par_width %c1 {par_dim = #acc.par_dim<block_y>}
+  %thread_y = acc.par_width %c4 {par_dim = #acc.par_dim<thread_y>}
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.kernel_environment {
+    %private = acc.privatize [#acc<par_dims[thread_y]>]
+        : () -> !acc.private_type<memref<i32>>
+    acc.compute_region launch(%by = %block_y, %ty = %thread_y, %tx = %thread_x)
+        ins(%private_arg = %private, %result_arg = %result)
+        : (!acc.private_type<memref<i32>>, memref<i32>) {
+      %c0 = arith.constant 0 : index
+      %c1_inner = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      scf.parallel (%block_iv) = (%c0) to (%by) step (%c1_inner) {
+        %local = acc.private_local %private_arg
+            : (!acc.private_type<memref<i32>>) -> memref<i32>
+        scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
+          memref.store %c0_i32, %local[] : memref<i32>
+          acc.predicate_region {
+            acc.reduction_combine %local into %result_arg <add> : memref<i32>
+                {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
+          }
+          scf.reduce
+        } {acc.par_dims = #acc<par_dims[thread_y]>}
+        scf.reduce
+      } {acc.par_dims = #acc<par_dims[block_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}
+
 // The outer predicate must keep ThreadY active so it does not exclude worker
 // rows before the nested worker-private combine is reached.
 // CHECK-LABEL: func.func @nested_worker_reduction_combines

>From ca846e5a59a0e4ce0ec567a391cb4ab23f777640 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:57:39 -0700
Subject: [PATCH 21/22] [OpenACC] Allow stores to active worker sources

Permit row-local stores only when they target the exact private storage already sized for a supported worker combine.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp    | 16 ++++++++++++++++
 .../acc-cg-to-gpu-worker-reduction-combine.mlir  |  2 ++
 2 files changed, 18 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f46a891f08652..bf90b51a2d05b 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1311,6 +1311,18 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
     auto analyzeThreadYBroadening =
         [&](auto &&self, Block &predicateBlock) -> ThreadYBroadeningInfo {
       ThreadYBroadeningInfo info;
+      DenseSet<Value> activeWorkerSources;
+      predicateBlock.walk([&](acc::ReductionCombineOp combineOp) {
+        ArrayRef<GPUParallelDimAttr> parDims =
+            getReductionCombineParDims(combineOp);
+        bool hasThreadY = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isThreadY(); });
+        bool hasBlock = llvm::any_of(
+            parDims, [](auto parDim) { return parDim.isAnyBlock(); });
+        if (hasThreadY && hasBlock &&
+            isProvenWorkerPrivate(combineOp.getSrcMemref()))
+          activeWorkerSources.insert(combineOp.getSrcMemref());
+      });
       auto classifyCombine = [&](Operation *combineOp, Value src,
                                  ArrayRef<GPUParallelDimAttr> parDims,
                                  bool supportsWorkerBroadening) {
@@ -1360,6 +1372,10 @@ ACCCGToGPULowering::computeActiveAndInactiveParDims(Operation *op,
                           /*supportsWorkerBroadening=*/false);
           continue;
         }
+        if (memref::StoreOp storeOp = dyn_cast<memref::StoreOp>(nestedOp)) {
+          if (activeWorkerSources.contains(storeOp.getMemref()))
+            continue;
+        }
         if (nestedOp.getNumRegions() != 0) {
           if (hasOwnEffects(&nestedOp)) {
             info.requiresInactiveThreadY = true;
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index fd4e12fd039a8..7063239fbc09d 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -9,6 +9,7 @@
 // CHECK: %[[IS_X_ZERO:.*]] = arith.cmpi eq, [[TID_X]],
 // CHECK-NOT: arith.andi
 // CHECK: scf.if %[[IS_X_ZERO]]
+// CHECK: memref.store
 // CHECK: acc.atomic.update
 
 func.func @worker_reduction_combine(%result: memref<i32>) {
@@ -35,6 +36,7 @@ func.func @worker_reduction_combine(%result: memref<i32>) {
           scf.reduce
         } {acc.par_dims = #acc<par_dims[thread_y]>}
         acc.predicate_region {
+          memref.store %c0_i32, %local[] : memref<i32>
           acc.reduction_combine %local into %result_arg <add> : memref<i32>
               {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
         }

>From b67a8deeddbaaf77d0ef5677dc6f8a1ef274137a Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 21:00:18 -0700
Subject: [PATCH 22/22] [OpenACC][test] Cover active-scope worker store

Exercise a same-source store and worker combine inside an already active ThreadY scope.
---
 .../Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir  | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
index 7063239fbc09d..03da2186987a6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-worker-reduction-combine.mlir
@@ -75,6 +75,7 @@ func.func @worker_combine_in_thread_y_scope(%result: memref<i32>) {
         scf.parallel (%worker_iv) = (%c0) to (%ty) step (%c1_inner) {
           memref.store %c0_i32, %local[] : memref<i32>
           acc.predicate_region {
+            memref.store %c0_i32, %local[] : memref<i32>
             acc.reduction_combine %local into %result_arg <add> : memref<i32>
                 {acc.par_dims = #acc<par_dims[block_y, thread_y]>}
           }


From llvmlistbot at llvm.org  Mon Jul 20 22:03:13 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:13 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd91.c4d3d80a.849c3.85ad@mx.google.com>


https://github.com/bzcheeseman commented:

Thanks Eric :)

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:13 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:13 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd91.e55dc341.1c6a0.4236@mx.google.com>


https://github.com/bzcheeseman edited https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:13 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:13 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd91.4987c784.2c60a9.de0e@mx.google.com>


================
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
----------------
bzcheeseman wrote:

we don't have `isa` for these?

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:13 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:13 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd91.1905e68e.30d957.2e37@mx.google.com>


================
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
----------------
bzcheeseman wrote:

why are you setting the high bit here? Non-obvious to me, anyway

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:14 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:14 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd92.66c0679a.24976c.dcd9@mx.google.com>


================
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
----------------
bzcheeseman wrote:

could you clarify whose responsibility each of these restrictions are? It's not clear from the text as written.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:14 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:14 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd92.eb165e5c.4506c.f8e2@mx.google.com>


================
@@ -7181,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
----------------
bzcheeseman wrote:

Do these checks belong as methods on DbgVariableRecord? You have another example above (somewhere) of having to compute some properties like these that might be better suited to methods

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:14 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:14 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd92.f3151ff8.37ac76.110a@mx.google.com>


================
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
----------------
bzcheeseman wrote:

This control flow feels off - what is the expected behavior if one of the TmpDIE values  hits the `if (Offset < CurrentOffset + ValueSize)` branch before you've resolved all the fixups? Or alternatively, what if TmpDIE has entries that don't require fixups?

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:15 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:15 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efd93.c5504292.78ef3.e606@mx.google.com>


================
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
----------------
bzcheeseman wrote:

could you add a brief comment explaining why this works? It would help for the future not having to re-derive it

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:03:30 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Mon, 20 Jul 2026 22:03:30 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5efda2.7d702ed2.13d102.f25a@mx.google.com>


https://github.com/bzcheeseman edited https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Mon Jul 20 22:47:10 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Mon, 20 Jul 2026 22:47:10 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] Enforce single-operand form
 for llvm.loop.distribute metadata (PR #201077)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201077@github.com>
Message-ID: <6a5f07de.0c05e68e.3c2f5d.1555@mx.google.com>


https://github.com/madhur13490 updated https://github.com/llvm/llvm-project/pull/201077

>From 3cc53aee4b6f0cbbea5d53e6c5685e20b824e157 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Wed, 15 Apr 2026 01:00:40 -0700
Subject: [PATCH 1/3] Enforce single-operand form for llvm.loop.distribute
 metadata

This patch implements Phase 1 of the RFC "Enforce Single-Operand Format for All
.enable Metadata Nodes". The two-operand boolean form
!{!"llvm.loop.distribute.enable", i1 0/1} is replaced by a
single-operand enable/disable pair:

  !{!"llvm.loop.distribute.enable"}    ; force distribution
  !{!"llvm.loop.distribute.disable"}   ; suppress distribution
---
 clang/lib/CodeGen/CGLoopInfo.cpp              | 22 ++++-----
 .../test/CodeGenCXX/pragma-followup_inner.cpp |  2 +-
 .../test/CodeGenCXX/pragma-followup_outer.cpp |  2 +-
 .../CodeGenCXX/pragma-loop-distribute.cpp     |  2 +-
 clang/test/CodeGenCXX/pragma-loop.cpp         |  2 +-
 llvm/docs/LangRef.md                          |  8 ++-
 llvm/lib/Analysis/LoopAccessAnalysis.cpp      | 11 ++---
 llvm/lib/IR/Verifier.cpp                      |  8 +++
 llvm/lib/Transforms/Scalar/LoopDistribute.cpp | 13 ++---
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp |  3 +-
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |  3 ++
 .../Analysis/LoopAccessAnalysis/pr64637.ll    |  2 +-
 .../Other/optimization-remarks-lazy-bfi.ll    |  2 +-
 .../IRCE/add-metadata-pre-post-loops.ll       |  2 +-
 .../test/Transforms/IRCE/correct-loop-info.ll |  2 +-
 .../Transforms/IRCE/loop-guarded-bounds.ll    |  2 +-
 llvm/test/Transforms/IRCE/pre_post_loops.ll   |  2 +-
 .../IRCE/range_intersect_miscompile.ll        |  2 +-
 .../IRCE/unsigned_comparisons_ugt.ll          |  2 +-
 .../IRCE/unsigned_comparisons_ult.ll          |  2 +-
 .../Inline/dilocation-loop-metadata-update.ll |  4 +-
 .../LoopDistribute/basic-with-memchecks.ll    |  2 +-
 .../diagnostics-with-hotness.ll               |  2 +-
 .../Transforms/LoopDistribute/diagnostics.ll  |  2 +-
 .../disable_nonforced_enable.ll               |  2 +-
 .../Transforms/LoopDistribute/early-exit.ll   |  2 +-
 .../Transforms/LoopDistribute/followup.ll     |  2 +-
 .../Transforms/LoopDistribute/metadata.ll     |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 .../preserve-llvm-loop-metadata.ll            |  4 +-
 llvm/test/Verifier/llvm.loop.distribute.ll    | 49 +++++++++++++++++++
 llvm/unittests/Analysis/LoopInfoTest.cpp      |  8 +--
 .../Target/LLVMIR/LoopAnnotationImporter.cpp  |  6 ++-
 .../LLVMIR/LoopAnnotationTranslation.cpp      |  7 ++-
 .../Target/LLVMIR/Import/metadata-loop.ll     |  2 +-
 mlir/test/Target/LLVMIR/loop-metadata.mlir    |  2 +-
 36 files changed, 127 insertions(+), 69 deletions(-)
 create mode 100644 llvm/test/Verifier/llvm.loop.distribute.ll

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 6b0b1b55213e1..1ae6bbe0ff595 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -331,11 +331,12 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
-      NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.append(
+          LoopProperties.begin(), LoopProperties.end());
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx,
+          {MDString::get(
+              Ctx, "llvm.loop.distribute.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createLoopVectorizeMetadata(Attrs, LoopProperties,
@@ -344,16 +345,15 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
 
   bool FollowupHasTransforms = false;
   SmallVector<Metadata *, 4> Followup =
-      createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
+      createLoopVectorizeMetadata(
+          Attrs, LoopProperties, FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
-  Metadata *Vals[] = {MDString::get(Ctx, "llvm.loop.distribute.enable"),
-                      ConstantAsMetadata::get(ConstantInt::get(
-                          llvm::Type::getInt1Ty(Ctx),
-                          (Attrs.DistributeEnable == LoopAttributes::Enable)))};
-  Args.push_back(MDNode::get(Ctx, Vals));
+  Args.push_back(MDNode::get(
+      Ctx,
+      {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
 
   if (FollowupHasTransforms)
     Args.push_back(
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..2cf2dba5b913d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_inner(int n, int *x) {
 
 // CHECK-DAG: ![[INNERLOOP_3]] = distinct !{![[INNERLOOP_3]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..2e907fb814b3d 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -19,7 +19,7 @@ extern "C" void followup_outer(int n, int *x) {
 
 // CHECK-DAG: ![[LOOP_3:[0-9]+]] = distinct !{![[LOOP_3:[0-9]+]], [[MP:![0-9]+]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[DISTRIBUTE_5:[0-9]+]], ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]]}
 // CHECK-DAG: ![[PARALLEL_ACCESSES_4:[0-9]+]] = !{!"llvm.loop.parallel_accesses", ![[ACCESSGROUP_2]]}
-// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
 // CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
index 98aef21da2acd..66f0fb07b51fb 100644
--- a/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-distribute.cpp
@@ -20,5 +20,5 @@ void while_test(int *List, int Length, int *List2, int Length2) {
 }
 
 // CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], [[MP:![0-9]+]], ![[DISTRIBUTE_ENABLE:.*]]}
-// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable", i1 true}
+// CHECK: ![[DISTRIBUTE_ENABLE]] = !{!"llvm.loop.distribute.enable"}
 // CHECK: [[LOOP_2]] = distinct !{[[LOOP_2]], [[MP]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..f9106725ad99d 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -248,7 +248,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[UNROLL_32:[0-9]+]] = !{!"llvm.loop.unroll.count", i32 32}
 // CHECK-DAG: ![[UNROLL_FULL:[0-9]+]] = !{!"llvm.loop.unroll.full"}
 
-// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: ![[DISTRIBUTE_DISABLE:[0-9]+]] = !{!"llvm.loop.distribute.disable"}
 
 // CHECK-DAG: ![[INTERLEAVE_2:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 2}
 // CHECK-DAG: ![[INTERLEAVE_4:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 4}
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 192597944101b..81bf786a0dbba 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -8280,13 +8280,11 @@ memory dependencies.  The transformation will attempt to isolate the unsafe
 dependencies into their own loop.
 
 This metadata can be used to selectively enable or disable distribution of the
-loop.  The first operand is the string `llvm.loop.distribute.enable` and the
-second operand is a bit. If the bit operand value is 1 distribution is
-enabled. A value of 0 disables distribution:
+loop.  Each node has a single operand containing the name string:
 
 ```llvm
-!0 = !{!"llvm.loop.distribute.enable", i1 0}
-!1 = !{!"llvm.loop.distribute.enable", i1 1}
+!0 = !{!"llvm.loop.distribute.enable"}
+!1 = !{!"llvm.loop.distribute.disable"}
 ```
 
 This metadata should be used in conjunction with `llvm.loop` loop
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 8d3c66c8fe321..0b0da44bf44f7 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2940,14 +2940,9 @@ void LoopAccessInfo::emitUnsafeDependenceRemark() {
   LLVM_DEBUG(dbgs() << "LAA: unsafe dependent memory operations in loop\n");
 
   // Emit remark for first unsafe dependence
-  bool HasForcedDistribution = false;
-  std::optional<const MDOperand *> Value =
-      findStringMetadataForLoop(TheLoop, "llvm.loop.distribute.enable");
-  if (Value) {
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    HasForcedDistribution = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
-  }
+  bool HasForcedDistribution =
+      getBooleanLoopAttribute(
+          TheLoop, "llvm.loop.distribute.enable");
 
   const std::string Info =
       HasForcedDistribution
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 28459a39fedae..41c45be8de882 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1008,6 +1008,14 @@ void Verifier::visitMDNode(const MDNode &BaseMD,
             CurrentMD);
     }
 
+    // Enforce the single-operand form of llvm.loop.distribute metadata.
+    if (CurrentMD->getNumOperands() > 0 &&
+        (CurrentMD->getOperand(0).equalsStr("llvm.loop.distribute.enable") ||
+         CurrentMD->getOperand(0).equalsStr("llvm.loop.distribute.disable")))
+      Check(CurrentMD->getNumOperands() == 1,
+            "Expected one operand for llvm.loop.distribute metadata",
+            CurrentMD);
+
     // Check these last, so we diagnose problems in operands first.
     Check(!CurrentMD->isTemporary(), "Expected no forward declarations!",
           CurrentMD);
diff --git a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
index 2d0b08fed859b..07cfd8628a4f6 100644
--- a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -941,14 +941,11 @@ class LoopDistributeForLoop {
   /// Check whether the loop metadata is forcing distribution to be
   /// enabled/disabled.
   void setForced() {
-    std::optional<const MDOperand *> Value =
-        findStringMetadataForLoop(L, "llvm.loop.distribute.enable");
-    if (!Value)
-      return;
-
-    const MDOperand *Op = *Value;
-    assert(Op && mdconst::hasa<ConstantInt>(*Op) && "invalid metadata");
-    IsForced = mdconst::extract<ConstantInt>(*Op)->getZExtValue();
+    if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
+      IsForced = true;
+    else if (getBooleanLoopAttribute(
+                 L, "llvm.loop.distribute.disable"))
+      IsForced = false;
   }
 
   Loop *L;
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 72fae62e56e88..74cd2a729d285 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -453,7 +453,8 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
       Context,
-      {MDString::get(Context, "llvm.loop.distribute.enable"), FalseVal});
+      {MDString::get(
+          Context, "llvm.loop.distribute.disable")});
   MDNode *NewLoopID =
       MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
                             DisableLICMVersioning, DisableDistribution});
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 58b52519beaf4..598b864f720c5 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -441,6 +441,9 @@ TransformationMode llvm::hasVectorizeTransformation(const Loop *L) {
 }
 
 TransformationMode llvm::hasDistributeTransformation(const Loop *L) {
+  if (getBooleanLoopAttribute(L, "llvm.loop.distribute.disable"))
+    return TM_SuppressedByUser;
+
   if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
     return TM_ForcedByUser;
 
diff --git a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
index d3e589cf99cf3..7b25d1fffcbba 100644
--- a/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
+++ b/llvm/test/Analysis/LoopAccessAnalysis/pr64637.ll
@@ -76,4 +76,4 @@ for.body:                                         ; preds = %for.body, %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Other/optimization-remarks-lazy-bfi.ll b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
index c5099bb2179e4..5fc34cc14bdc8 100644
--- a/llvm/test/Other/optimization-remarks-lazy-bfi.ll
+++ b/llvm/test/Other/optimization-remarks-lazy-bfi.ll
@@ -75,7 +75,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 99, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
index eb72907730c6d..3a360814e361e 100644
--- a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
+++ b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
@@ -76,7 +76,7 @@ attributes #0 = { alwaysinline }
 !3 = !{!"llvm.loop.unroll.disable"}
 !4 = !{!"llvm.loop.vectorize.enable", i1 false}
 !5 = !{!"llvm.loop.licm_versioning.disable"}
-!6 = !{!"llvm.loop.distribute.enable", i1 false}
+!6 = !{!"llvm.loop.distribute.disable"}
 !7 = !{}
 !8 = distinct !{!8, !3, !4, !5}
 !9 = distinct !{!9, !3, !4, !5}
diff --git a/llvm/test/Transforms/IRCE/correct-loop-info.ll b/llvm/test/Transforms/IRCE/correct-loop-info.ll
index 7ea445718289a..bc5987ebba2e4 100644
--- a/llvm/test/Transforms/IRCE/correct-loop-info.ll
+++ b/llvm/test/Transforms/IRCE/correct-loop-info.ll
@@ -178,6 +178,6 @@ declare void @pluto()
 !1 = !{!"llvm.loop.unroll.disable"}
 !2 = !{!"llvm.loop.vectorize.enable", i1 false}
 !3 = !{!"llvm.loop.licm_versioning.disable"}
-!4 = !{!"llvm.loop.distribute.enable", i1 false}
+!4 = !{!"llvm.loop.distribute.disable"}
 !5 = !{}
 !6 = distinct !{!6, !1, !2, !3, !4}
diff --git a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
index ea169987cf2fc..61cfef6f1492a 100644
--- a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
+++ b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
@@ -130,7 +130,7 @@ exit:                                       ; preds = %guarded, %loop_, %prehead
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/pre_post_loops.ll b/llvm/test/Transforms/IRCE/pre_post_loops.ll
index da2faa34719af..2807adba35940 100644
--- a/llvm/test/Transforms/IRCE/pre_post_loops.ll
+++ b/llvm/test/Transforms/IRCE/pre_post_loops.ll
@@ -163,7 +163,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ;.
diff --git a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
index 68613feacba32..2e21bb5ad0bd0 100644
--- a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
+++ b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
@@ -490,7 +490,7 @@ deopt:                                          ; preds = %range_check_block
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
 ; CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]], [[META3]], [[META4]]}
 ; CHECK: [[RNG7]] = !{i32 0, i32 50}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
index 7236cf4817ade..cdd7235ac45b4 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
@@ -447,7 +447,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
index 546e1b0faa1c3..107a3554282ad 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
@@ -654,7 +654,7 @@ exit:
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
-; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
+; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
 ; CHECK: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META3]], [[META4]], [[META5]]}
 ; CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META2]], [[META3]], [[META4]], [[META5]]}
diff --git a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
index 1bc132663331b..0d8dcfbcf2a8b 100644
--- a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
+++ b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
@@ -47,7 +47,7 @@ entry:
 !7 = !DILocation(line: 6, column: 3, scope: !3)
 !8 = !DILocation(line: 7, column: 22, scope: !3)
 !9 = !{!"llvm.loop.mustprogress"}
-!10 = !{!"llvm.loop.distribute.enable", i1 true}
+!10 = !{!"llvm.loop.distribute.enable"}
 !11 = !{!"llvm.loop.distribute.followup_all", !7, !8, !9, !12, !13, !14}
 !12 = !{!"llvm.loop.vectorize.width", i32 8}
 !13 = !{!"llvm.loop.vectorize.enable", i1 true}
@@ -66,7 +66,7 @@ entry:
 ; CHECK: [[META7]] = !DILocation(line: 6, column: 3, scope: [[DBG3]])
 ; CHECK: [[META8]] = !DILocation(line: 7, column: 22, scope: [[DBG3]])
 ; CHECK: [[META9]] = !{!"llvm.loop.mustprogress"}
-; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable"}
 ; CHECK: [[META11]] = !{!"llvm.loop.distribute.followup_all", [[META7]], [[META8]], [[META9]], [[META12:![0-9]+]], [[META13:![0-9]+]], [[META14:![0-9]+]]}
 ; CHECK: [[META12]] = !{!"llvm.loop.vectorize.width", i32 8}
 ; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
index 2828882afe779..dcd231a900ee2 100644
--- a/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
+++ b/llvm/test/Transforms/LoopDistribute/basic-with-memchecks.ll
@@ -288,5 +288,5 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: ![[PROF1]] = !{!"unknown", !"loop-versioning"}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
index b5ef3578eadd9..4f2f76191e615 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics-with-hotness.ll
@@ -74,7 +74,7 @@ for.cond.cleanup:
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = !{!"function_entry_count", i64 3}
 !23 = !{!"branch_weights", i32 2000, i32 1}
 !24 = !{!"branch_weights", i32 1, i32 99}
diff --git a/llvm/test/Transforms/LoopDistribute/diagnostics.ll b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
index e6a0d83bd63d2..80b217d0abb2b 100644
--- a/llvm/test/Transforms/LoopDistribute/diagnostics.ll
+++ b/llvm/test/Transforms/LoopDistribute/diagnostics.ll
@@ -194,7 +194,7 @@ attributes #1 = { nounwind convergent }
 !18 = !DILocation(line: 4, column: 5, scope: !7)
 !19 = !DILocation(line: 4, column: 10, scope: !7)
 !20 = distinct !{!20, !21}
-!21 = !{!"llvm.loop.distribute.enable", i1 true}
+!21 = !{!"llvm.loop.distribute.enable"}
 !22 = distinct !DISubprogram(name: "not_forced", scope: !1, file: !1, line: 8, type: !8, isLocal: false, isDefinition: true, scopeLine: 8, flags: DIFlagPrototyped, isOptimized: true, unit: !0, retainedNodes: !2)
 !23 = !DILocation(line: 9, column: 20, scope: !22)
 !24 = !DILocation(line: 9, column: 3, scope: !22)
diff --git a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
index 45a2d31256a27..eff1621cd7ada 100644
--- a/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
+++ b/llvm/test/Transforms/LoopDistribute/disable_nonforced_enable.ll
@@ -43,4 +43,4 @@ for.end:
   ret void
 }
 
-!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable", i32 1}}
+!0 = distinct !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.distribute.enable"}}
diff --git a/llvm/test/Transforms/LoopDistribute/early-exit.ll b/llvm/test/Transforms/LoopDistribute/early-exit.ll
index 9353d842523f4..d2ad9b1bcd6c5 100644
--- a/llvm/test/Transforms/LoopDistribute/early-exit.ll
+++ b/llvm/test/Transforms/LoopDistribute/early-exit.ll
@@ -89,4 +89,4 @@ attributes #0 = { nounwind readnone convergent }
 attributes #1 = { nounwind convergent }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/LoopDistribute/followup.ll b/llvm/test/Transforms/LoopDistribute/followup.ll
index ae418ad9e6b42..2753bad5e1c1e 100644
--- a/llvm/test/Transforms/LoopDistribute/followup.ll
+++ b/llvm/test/Transforms/LoopDistribute/followup.ll
@@ -125,7 +125,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1, !2, !3, !4, !5}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = !{!"llvm.loop.distribute.followup_all", !{!"FollowupAll"}}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !{!"FollowupCoincident", i1 false}}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !{!"FollowupSequential", i32 8}}
diff --git a/llvm/test/Transforms/LoopDistribute/metadata.ll b/llvm/test/Transforms/LoopDistribute/metadata.ll
index b0e461fe0ea75..8b74ac4249550 100644
--- a/llvm/test/Transforms/LoopDistribute/metadata.ll
+++ b/llvm/test/Transforms/LoopDistribute/metadata.ll
@@ -131,6 +131,6 @@ for.end:                                          ; preds = %for.body
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.distribute.enable", i1 false}
+!3 = !{!"llvm.loop.distribute.disable"}
diff --git a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
index 2c1dff7ee5a5f..049880a97a211 100644
--- a/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/LoopSimplify/preserve-llvm-loop-metadata.ll
@@ -65,6 +65,6 @@ cleanup:                                          ; preds = %cleanup.loopexit, %
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!1 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.distribute.enable"}
diff --git a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
index 8b3bdbee2249c..d565b860a9ef6 100644
--- a/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
+++ b/llvm/test/Transforms/SimplifyCFG/preserve-llvm-loop-metadata.ll
@@ -147,7 +147,7 @@ while.end:                                        ; preds = %while.cond
 
 !0 = !{i32 2, !"Debug Info Version", i32 3}
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = !{!"llvm.loop.distribute.enable"}
 !3 = distinct !{!3, !4}
 !4 = !{!"llvm.loop.mustprogress"}
 !5 = distinct !{!5, !4, !6}
@@ -161,7 +161,7 @@ while.end:                                        ; preds = %while.cond
 !13 = !DILocation(line: 9, column: 23, scope: !12)
 
 ; CHECK: !1 = distinct !{!1, !2}
-; CHECK: !2 = !{!"llvm.loop.distribute.enable", i1 true}
+; CHECK: !2 = !{!"llvm.loop.distribute.enable"}
 ; CHECK: !3 = distinct !{!3, !4}
 ; CHECK: !4 = !{!"llvm.loop.mustprogress"}
 ; CHECK: !5 = distinct !{!5, !4, !6}
diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
new file mode 100644
index 0000000000000..9e1e8138e93dc
--- /dev/null
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -0,0 +1,49 @@
+; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
+; single-operand validation.
+
+; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
+; DEFINE:   FileCheck %s -allow-empty -check-prefix
+
+define void @test() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+!0 = distinct !{!0, !1}
+
+; GOOD-NOT: {{.}}
+
+;      BAD: Expected one operand for llvm.loop.distribute metadata
+
+; Single-operand enable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Single-operand disable.
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
+; RUN: %{RUN} GOOD
+
+; Two-operand enable with boolean false (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand enable with boolean true (legacy form, now rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
+; RUN: not %{RUN} BAD
+
+; Two-operand disable (rejected).
+; RUN: cp %s %t
+; RUN: chmod u+w %t
+; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
+; RUN: not %{RUN} BAD
diff --git a/llvm/unittests/Analysis/LoopInfoTest.cpp b/llvm/unittests/Analysis/LoopInfoTest.cpp
index 4828573c32c10..5321f72aa1784 100644
--- a/llvm/unittests/Analysis/LoopInfoTest.cpp
+++ b/llvm/unittests/Analysis/LoopInfoTest.cpp
@@ -72,7 +72,7 @@ TEST(LoopInfoTest, LoopWithSingleLatch) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -122,7 +122,7 @@ TEST(LoopInfoTest, LoopWithMultipleLatches) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1397,7 +1397,7 @@ TEST(LoopInfoTest, LoopUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
@@ -1440,7 +1440,7 @@ TEST(LoopInfoTest, LoopNonLatchUniqueExitBlocks) {
       "  ret void\n"
       "}\n"
       "!0 = distinct !{!0, !1}\n"
-      "!1 = !{!\"llvm.loop.distribute.enable\", i1 true}\n";
+      "!1 = !{!\"llvm.loop.distribute.enable\"}\n";
 
   // Parse the module.
   LLVMContext Context;
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index e4905423347a2..4022afc433a61 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -358,8 +358,10 @@ FailureOr<LoopLICMAttr> LoopMetadataConversion::convertLICMAttr() {
 }
 
 FailureOr<LoopDistributeAttr> LoopMetadataConversion::convertDistributeAttr() {
-  FailureOr<BoolAttr> disable =
-      lookupBoolNode("llvm.loop.distribute.enable", true);
+  FailureOr<BoolAttr> disable = lookupBooleanUnitNode(
+      "llvm.loop.distribute.enable",
+      "llvm.loop.distribute.disable",
+      /*negated=*/true);
   FailureOr<LoopAnnotationAttr> followupCoincident =
       lookupFollowupNode("llvm.loop.distribute.followup_coincident");
   FailureOr<LoopAnnotationAttr> followupSequential =
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
index f3ec18c9074e4..41ba31106f788 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
@@ -162,7 +162,12 @@ void LoopAnnotationConversion::convertLoopOptions(LoopLICMAttr options) {
 }
 
 void LoopAnnotationConversion::convertLoopOptions(LoopDistributeAttr options) {
-  convertBoolNode("llvm.loop.distribute.enable", options.getDisable(), true);
+  if (auto disable = options.getDisable()) {
+    if (disable.getValue())
+      addUnitNode("llvm.loop.distribute.disable");
+    else
+      addUnitNode("llvm.loop.distribute.enable");
+  }
   convertFollowupNode("llvm.loop.distribute.followup_coincident",
                       options.getFollowupCoincident());
   convertFollowupNode("llvm.loop.distribute.followup_sequential",
diff --git a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
index c09fe92dc9cdc..021818ded3d3c 100644
--- a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
+++ b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
@@ -219,7 +219,7 @@ end:
 }
 
 !1 = distinct !{!1, !2, !3, !4, !5, !6}
-!2 = !{!"llvm.loop.distribute.enable", i1 0}
+!2 = !{!"llvm.loop.distribute.disable"}
 !3 = !{!"llvm.loop.distribute.followup_coincident", !9}
 !4 = !{!"llvm.loop.distribute.followup_sequential", !9}
 !5 = !{!"llvm.loop.distribute.followup_fallback", !9}
diff --git a/mlir/test/Target/LLVMIR/loop-metadata.mlir b/mlir/test/Target/LLVMIR/loop-metadata.mlir
index 72f0cf06ed0fe..299bcb481f67f 100644
--- a/mlir/test/Target/LLVMIR/loop-metadata.mlir
+++ b/mlir/test/Target/LLVMIR/loop-metadata.mlir
@@ -185,7 +185,7 @@ llvm.func @distributeOptions() {
 // CHECK-DAG: ![[NON_FORCED:[0-9]+]] = !{!"llvm.loop.disable_nonforced"}
 // CHECK-DAG: ![[FOLLOWUP:[0-9]+]] = distinct !{![[FOLLOWUP]], ![[NON_FORCED]]}
 // CHECK-DAG: ![[LOOP_NODE]] = distinct !{![[LOOP_NODE]], !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}}
-// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.enable", i1 false}
+// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.disable"}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_coincident", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_sequential", ![[FOLLOWUP]]}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.distribute.followup_fallback", ![[FOLLOWUP]]}

>From 8387cd4e0b98f38fdd1212e52b2174c3090eb34a Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Tue, 7 Jul 2026 00:52:57 -0700
Subject: [PATCH 2/3] fixup! Enforce single-operand form for
 llvm.loop.distribute metadata

auto-upgrade old two-operand llvm.loop.distribute.enable metadata
---
 llvm/include/llvm/IR/AutoUpgrade.h            |   7 ++-
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    |   3 ++
 llvm/lib/IR/AutoUpgrade.cpp                   |  47 ++++++++++++++++--
 .../Bitcode/upgrade-loop-distribute-enable.ll |  37 ++++++++++++++
 .../upgrade-loop-distribute-enable.ll.bc      | Bin 0 -> 2080 bytes
 llvm/test/Verifier/llvm.loop.distribute.ll    |  30 ++++-------
 6 files changed, 101 insertions(+), 23 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
 create mode 100644 llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc

diff --git a/llvm/include/llvm/IR/AutoUpgrade.h b/llvm/include/llvm/IR/AutoUpgrade.h
index feb19e69389a9..423fea4205a80 100644
--- a/llvm/include/llvm/IR/AutoUpgrade.h
+++ b/llvm/include/llvm/IR/AutoUpgrade.h
@@ -112,7 +112,12 @@ namespace llvm {
 
   /// Check whether a string looks like an old loop attachment tag.
   inline bool mayBeOldLoopAttachmentTag(StringRef Name) {
-    return Name.starts_with("llvm.vectorizer.");
+    // "llvm.loop.distribute.enable" is intentionally included: the current
+    // single-operand form shares the tag with the removed two-operand form
+    // (!{!"llvm.loop.distribute.enable", i1 X}), so we can only decide by
+    // inspecting the operands, which happens in upgradeLoopArgument().
+    return Name.starts_with("llvm.vectorizer.") ||
+           Name == "llvm.loop.distribute.enable";
   }
 
   /// Upgrade the loop attachment metadata node.
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 40bc930d124f0..270ed23c5e6e2 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -2434,6 +2434,9 @@ Error MetadataLoader::MetadataLoaderImpl::parseOneMetadata(
   }
   case bitc::METADATA_STRINGS: {
     auto CreateNextMDString = [&](StringRef Str) {
+      // Modern bitcode encodes MDStrings via this bulk record, so mirror the
+      // METADATA_STRING check above to arm the loop-attachment upgrader.
+      HasSeenOldLoopTags |= mayBeOldLoopAttachmentTag(Str);
       ++NumMDStringLoaded;
       MetadataList.assignValue(MDString::get(Context, Str), NextMetadataNo);
       NextMetadataNo++;
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 880350a1b159b..926e18924956f 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6964,6 +6964,18 @@ void llvm::copyModuleAttrToFunctions(Module &M) {
   }
 }
 
+// Old two-operand form: !{!"llvm.loop.distribute.enable", i1 X}. The new
+// single-operand form uses "llvm.loop.distribute.enable" for X = true and
+// "llvm.loop.distribute.disable" for X = false.
+static bool isOldDistributeEnable(const MDTuple *T) {
+  if (T->getNumOperands() != 2)
+    return false;
+  auto *Tag = dyn_cast_or_null<MDString>(T->getOperand(0));
+  if (!Tag || Tag->getString() != "llvm.loop.distribute.enable")
+    return false;
+  return mdconst::hasa<ConstantInt>(T->getOperand(1));
+}
+
 static bool isOldLoopArgument(Metadata *MD) {
   auto *T = dyn_cast_or_null<MDTuple>(MD);
   if (!T)
@@ -6973,7 +6985,9 @@ static bool isOldLoopArgument(Metadata *MD) {
   auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!S)
     return false;
-  return S->getString().starts_with("llvm.vectorizer.");
+  if (S->getString().starts_with("llvm.vectorizer."))
+    return true;
+  return isOldDistributeEnable(T);
 }
 
 static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
@@ -6997,17 +7011,28 @@ static Metadata *upgradeLoopArgument(Metadata *MD) {
   auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!OldTag)
     return MD;
+
+  LLVMContext &C = T->getContext();
+
+  // Rewrite the old two-operand distribute form to the single-operand pair.
+  if (isOldDistributeEnable(T)) {
+    bool Enable = !mdconst::extract<ConstantInt>(T->getOperand(1))->isZero();
+    return MDTuple::get(
+        C, {MDString::get(C, Enable ? "llvm.loop.distribute.enable"
+                                    : "llvm.loop.distribute.disable")});
+  }
+
   if (!OldTag->getString().starts_with("llvm.vectorizer."))
     return MD;
 
   // This has an old tag.  Upgrade it.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
-  Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
+  Ops.push_back(upgradeLoopTag(C, OldTag->getString()));
   for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
     Ops.push_back(T->getOperand(I));
 
-  return MDTuple::get(T->getContext(), Ops);
+  return MDTuple::get(C, Ops);
 }
 
 MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
@@ -7018,6 +7043,22 @@ MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
   if (none_of(T->operands(), isOldLoopArgument))
     return &N;
 
+  // Fix the old two-operand llvm.loop.distribute.enable nodes in place: the
+  // Verifier rejects any MDNode carrying the distribute tag with more than one
+  // operand, so a leftover reference (from the distinct loop-ID) would still
+  // trigger a diagnostic. In-place mutation is safe on distinct MDNodes.
+  if (T->isDistinct()) {
+    for (unsigned I = 0, E = T->getNumOperands(); I < E; ++I) {
+      auto *OpT = dyn_cast_or_null<MDTuple>(T->getOperand(I));
+      if (OpT && isOldDistributeEnable(OpT))
+        T->replaceOperandWith(I, upgradeLoopArgument(OpT));
+    }
+    if (none_of(T->operands(), isOldLoopArgument))
+      return &N;
+  }
+
+  // Remaining old arguments (e.g. llvm.vectorizer.*) are handled via a wrapper
+  // attachment; the original distinct loop-ID is kept as the first operand.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
   for (Metadata *MD : T->operands())
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
new file mode 100644
index 0000000000000..8f1b5aec2d140
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll
@@ -0,0 +1,37 @@
+; Test that older bitcode carrying the two-operand form of
+; "llvm.loop.distribute.enable" is auto-upgraded to the single-operand
+; enable/disable pair on load.
+;
+; RUN: llvm-dis < %s.bc | FileCheck %s
+; RUN: verify-uselistorder < %s.bc
+
+define void @enable_true() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+
+define void @enable_false() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !2
+exit:
+  ret void
+}
+
+; i1 true  -> single-operand enable.
+; i1 false -> disable.
+; CHECK: !{!"llvm.loop.distribute.enable"}
+; CHECK: !{!"llvm.loop.distribute.disable"}
+; The old two-operand nodes must be gone from the module.
+; CHECK-NOT: llvm.loop.distribute.enable", i1
+; CHECK-NOT: llvm.loop.distribute.disable", i1
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.distribute.enable", i1 true}
+!2 = distinct !{!2, !3}
+!3 = !{!"llvm.loop.distribute.enable", i1 false}
diff --git a/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc b/llvm/test/Bitcode/upgrade-loop-distribute-enable.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..fdcbda4584522c06b5f5668e53c6c24fede99c65
GIT binary patch
literal 2080
zcmbtVZ%iBK8GntzXA*N~5;pAQ&btdVved*Jh87#!lI=^GD`lIT<^$cbg)_DTnFGcc
z0;J3A*yG4*b#CQlmX=JQPF1UZAuS(Jix$i|%t at _`rt5?n%Ixwd$wUH4*i>zqwC7B>
zec89Y(!F=D?!D)p-=F7so!sp5b2SK6AcQ<Bw*EV>{O;|a{A22S`*Zb2WX1}_s1W)@
zhS2UZ29?2{I=HZC-5ZWv`HKp*(fp<?u6m|IsnwOg=833YD0BY2E?!?7&(U6eP|aSd
z at p{#7blh-`s;VnrH+kc(FO$^=T*o3REpw^Nn^RWTo9%6*>W^hcV_a!G?mkYxHEnMT
zMO9kkC~@goqgSJS5jT3X?V%)7*Oq)nO|^z|W3MNfBcq9OV>3b*X~+$7I^Zu?zwf+o
zLwyXUWZ#_IwX5<VH(%j29#pFK$@C6I+1qGcDM3K{n4@@i@%s5+h#R$k|H5$XbKdx&
zfw;C9IQLP-udaQz*7(os{yTRXMuxxp)5=e`AN~5o`Ok`{hSMi{uKu}q=(Wqg`E;$j
z at 5+yQe)$F8_|b;v1EGEPfE4!M;}BxA-`4<V{}~-Jbg`=Jh4<tLy>H*)+NevbwADwG
ziOPfDo-gN4S0xa-szGQAK59}%$ZV1H*T`&RoGC6QoDpnz2<uPxsn!wkK#gxl at Fta0
zH<zJqO^|-JqO>ixKQFiCI9oz)%MT^OLF}9o8+Hl7AT|<|4-N at ITF~aJ at r@*2WQiV@
zSlPn2+<22ZOD!|xB16sbIzM~SG83~baONqwbtYzw%Psk>n!yh2To=~TC4^v(2*IQf
z?7#-~*x*+c5cX_DIH?r+G(spX45y=q9&X_e3;17W at eNk)$xV=}6J)+LFY)9}hRjB(
zo4n#rNzR<(%<0lREt*oCX-sZOi4E}6rxz at Llv^f6)$hB7V7m|s#=>3LP=^rK2;p%-
zd+TN5AtD~T at zo^0Vke3;z5ys`J++k4%{g?p8FDGATPl#NMd~jyonP~kX_9Nb9JA!z
z^<M+&5Y7%^L+O`}ZYK$e0o`fcO`g2RP|F$Wyi6x0%urjr;`q;Ero3pG<ZM95(>d!Z
zXBmT~n7F<RkP2rZA0;-T$3p4oi4`@zF^jLc at h5Cu=C(t(TC80xQp=ElOKX{o*`{Lq
z<Dx0Sne(D`I%Cd>fYF+FP4y`;$f^$rY!|e5=JAaV;z2>NMH6c*u|^YrbK^Z~e1*kf
z`J|w at U%*%0_*ZP*mm54e7o`@9wHpkz&QMu~0tN3e-}2K}EHj*KT5e9o%=z{iP#?gK
zK%Oq)Y*5gCd>7wjiDCi2Zzt}L!(T+1y#E~xu$)YTRV0SP3m|1Q?l2cbpw0$L<?Y#V
z>gEJhaOjpZq`$tC#fy@@1O!=QIVLus!TMdo$#LP;2z+}rC9 at _(Yf3aNaIIjcJTRtm
zG2d|VI!N_ahB_9l at gF_K;+r(FSx{`ciESwqzE)7I(?oFrz`kXV4R&F}N<n*U-?V4}
zAO4mz<$*)FIT^F&<yIgwC92x0=E1t)s0s3XRQFg0I?i8JfCC{8XwkIid5sW^U?cBf
z!+MY`GvzQ%$}KaZ>1mm5Mr=(%Ze2?>^|?$3?9Bs#nJP(re)MrE2qIQyrDA!)#-g#3
zQW8e+f4?3ZNk7UuTBjkgnB?1Wt`({e+`9 at jK1?ID7wY&;or~tiRN6|fH+O at MLc?ln
zYug)jjwc^wb4s&wG~uzo3j6lM6$lKTF$8*hg9dMRe>l|b83_9fz8>Ba at WESz-i6-K
z0{7j!LVBn<Y@|Jy2Aw5|%nRa$b#i{9>YhVt3Bd7=m%rJ((t+P~Lx~WpQlSuc-T3xo
zBr1_1Rq&7im?ZzU2QQ1JDH%YqrNpuyLo*<kisTYauJY1+FRGj4$%P3rw_`d!BO|s!
zD9|l}?;hI`zv2%jka`gcS_JD^!0FJ3a&pTdJ^1uCMB_6@@HOx<_^yK}!VG<4LGdI>
zJfQK-SzN;Si5ejV+_AhBSiKAlL2et5S)n;3MX8<g`W;^Dl%rcLtbuB at H|}g_R9OWB
z-x`LZ>dAt=be7fH%Wp*}yQBH#_!Z<hkt>J2G7l=9VLuqP?}24${O_2a|0`x6y_1^}
zS09L at _v9L6E&c0<aF#v`PkZDkn}laa-jm_bfbZ)^KOgA#8Ji5Ofj!k7rkgxwqshoN
zHk&-Wzp<tBdA7;#?euw at S&ymN+wvUSWNh^CEyjjGZzmsU=;}S=YdFJuy9PphgJh_~
x!BFpMUuU?qXbAiI!wt^vaA&XA*Dw$~72>_V!;<9=|JQPdcM>rK0;P-*`WG%g;B5c^

literal 0
HcmV?d00001

diff --git a/llvm/test/Verifier/llvm.loop.distribute.ll b/llvm/test/Verifier/llvm.loop.distribute.ll
index 9e1e8138e93dc..65ca27720542b 100644
--- a/llvm/test/Verifier/llvm.loop.distribute.ll
+++ b/llvm/test/Verifier/llvm.loop.distribute.ll
@@ -1,8 +1,7 @@
 ; Test "llvm.loop.distribute.enable" / "llvm.loop.distribute.disable"
 ; single-operand validation.
 
-; DEFINE: %{RUN} = opt -passes=verify %t -disable-output 2>&1 | \
-; DEFINE:   FileCheck %s -allow-empty -check-prefix
+; DEFINE: %{VERIFY} = llvm-as -disable-output %t 2>&1
 
 define void @test() {
 entry:
@@ -14,36 +13,29 @@ exit:
 }
 !0 = distinct !{!0, !1}
 
-; GOOD-NOT: {{.}}
-
 ;      BAD: Expected one operand for llvm.loop.distribute metadata
 
 ; Single-operand enable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Single-operand disable.
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable"}' >> %t
-; RUN: %{RUN} GOOD
+; RUN: %{VERIFY}
 
 ; Two-operand enable with boolean false (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand enable with boolean true (legacy form, now rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.enable", i1 1}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
 
 ; Two-operand disable (rejected).
-; RUN: cp %s %t
-; RUN: chmod u+w %t
+; RUN: cat %s > %t
 ; RUN: echo '!1 = !{!"llvm.loop.distribute.disable", i1 0}' >> %t
-; RUN: not %{RUN} BAD
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD

>From 617e5a8579474f6390c5ae72b04b8c7d86b4acfd Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Tue, 7 Jul 2026 10:00:19 -0700
Subject: [PATCH 3/3] fixup!clang-format

---
 clang/lib/CodeGen/CGLoopInfo.cpp                  | 15 +++++----------
 llvm/lib/Analysis/LoopAccessAnalysis.cpp          |  3 +--
 llvm/lib/Transforms/Scalar/LoopDistribute.cpp     |  3 +--
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp     |  4 +---
 mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp |  3 +--
 5 files changed, 9 insertions(+), 19 deletions(-)

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 1ae6bbe0ff595..499e2287ae84a 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -331,12 +331,9 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
   if (Enabled != true) {
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
-      NewLoopProperties.append(
-          LoopProperties.begin(), LoopProperties.end());
+      NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
       NewLoopProperties.push_back(MDNode::get(
-          Ctx,
-          {MDString::get(
-              Ctx, "llvm.loop.distribute.disable")}));
+          Ctx, {MDString::get(Ctx, "llvm.loop.distribute.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createLoopVectorizeMetadata(Attrs, LoopProperties,
@@ -345,15 +342,13 @@ clang::CodeGen::LoopInfo::createLoopDistributeMetadata(
 
   bool FollowupHasTransforms = false;
   SmallVector<Metadata *, 4> Followup =
-      createLoopVectorizeMetadata(
-          Attrs, LoopProperties, FollowupHasTransforms);
+      createLoopVectorizeMetadata(Attrs, LoopProperties, FollowupHasTransforms);
 
   SmallVector<Metadata *, 4> Args;
   Args.append(LoopProperties.begin(), LoopProperties.end());
 
-  Args.push_back(MDNode::get(
-      Ctx,
-      {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
+  Args.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.distribute.enable")}));
 
   if (FollowupHasTransforms)
     Args.push_back(
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 0b0da44bf44f7..ff1032e669265 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -2941,8 +2941,7 @@ void LoopAccessInfo::emitUnsafeDependenceRemark() {
 
   // Emit remark for first unsafe dependence
   bool HasForcedDistribution =
-      getBooleanLoopAttribute(
-          TheLoop, "llvm.loop.distribute.enable");
+      getBooleanLoopAttribute(TheLoop, "llvm.loop.distribute.enable");
 
   const std::string Info =
       HasForcedDistribution
diff --git a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
index 07cfd8628a4f6..19c6d2da62d37 100644
--- a/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
+++ b/llvm/lib/Transforms/Scalar/LoopDistribute.cpp
@@ -943,8 +943,7 @@ class LoopDistributeForLoop {
   void setForced() {
     if (getBooleanLoopAttribute(L, "llvm.loop.distribute.enable"))
       IsForced = true;
-    else if (getBooleanLoopAttribute(
-                 L, "llvm.loop.distribute.disable"))
+    else if (getBooleanLoopAttribute(L, "llvm.loop.distribute.disable"))
       IsForced = false;
   }
 
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 74cd2a729d285..58ebd0e5ee937 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -452,9 +452,7 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
   MDNode *DisableLICMVersioning = MDNode::get(
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
-      Context,
-      {MDString::get(
-          Context, "llvm.loop.distribute.disable")});
+      Context, {MDString::get(Context, "llvm.loop.distribute.disable")});
   MDNode *NewLoopID =
       MDNode::get(Context, {Dummy, DisableUnroll, DisableVectorize,
                             DisableLICMVersioning, DisableDistribution});
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index 4022afc433a61..fcb11ca4629fa 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -359,8 +359,7 @@ FailureOr<LoopLICMAttr> LoopMetadataConversion::convertLICMAttr() {
 
 FailureOr<LoopDistributeAttr> LoopMetadataConversion::convertDistributeAttr() {
   FailureOr<BoolAttr> disable = lookupBooleanUnitNode(
-      "llvm.loop.distribute.enable",
-      "llvm.loop.distribute.disable",
+      "llvm.loop.distribute.enable", "llvm.loop.distribute.disable",
       /*negated=*/true);
   FailureOr<LoopAnnotationAttr> followupCoincident =
       lookupFollowupNode("llvm.loop.distribute.followup_coincident");


From llvmlistbot at llvm.org  Mon Jul 20 22:58:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 22:58:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a5f0a7a.27b3b003.1f84cc.4f72@mx.google.com>


qyingwu wrote:

> Something does not seem right to me here: the verifier should be bullet-proof and never crash. The fix likely belongs to the verifier implementation instead.

Thanks, I checked this path.

A standalone malformed SPIR-V op does not crash the verifier; it reports a normal diagnostic. The crash happens when diagnostic printing invokes `verify` while dialect conversion is in a temporary inconsistent state. In the reproducer, the stack goes through `verifyOpAndAdjustFlags` into `spirv::FuncOp::verifyBody()`, where it reads a return operand type while that operand may refer to IR being erased/rolled back.

So I think this diagnostic path should not require verification of the live user. The patch now prints the live user in generic form and uses `assumeVerified()` only for this diagnostic note.

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Mon Jul 20 23:12:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 23:12:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8f48e19 - [mlir][LLVMIR] Add export coverage
 for retained local variables (#209911)
Message-ID: <6a5f0ddc.1e1ac41d.3969c3.fac5@mx.google.com>


Author: qyingwu
Date: 2026-07-21T08:12:38+02:00
New Revision: 8f48e1980f9da4e4d859c1c2713561195d043454

URL: https://github.com/llvm/llvm-project/commit/8f48e1980f9da4e4d859c1c2713561195d043454
DIFF: https://github.com/llvm/llvm-project/commit/8f48e1980f9da4e4d859c1c2713561195d043454.diff

LOG: [mlir][LLVMIR] Add export coverage for retained local variables (#209911)

Adds export coverage for a retained local variable case where MLIR may
contain separate `DILocalVariableAttr`s for the same source-level
variable.

A local variable can appear both in a `DISubprogram`'s `retainedNodes`
and in a debug intrinsic such as `dbg.value` or `dbg.declare`. During
import, the `retainedNodes` path may see the variable while the
containing subprogram is represented by a self-recursive placeholder,
while the debug intrinsic path later sees the finalized subprogram. This
can produce two MLIR local variable attrs with different subprogram
scopes.

The exported LLVM IR debug metadata is still canonical for this case:
both the debug intrinsic and the subprogram `retainedNodes` reference
the same `DILocalVariable` metadata node. This patch adds regression
coverage for that export behavior without extending the recursive
debug-info machinery.

Added: 
    

Modified: 
    mlir/test/Target/LLVMIR/llvmir-debug.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/test/Target/LLVMIR/llvmir-debug.mlir b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
index 72dadb2602c52..b93a38be6634d 100644
--- a/mlir/test/Target/LLVMIR/llvmir-debug.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-debug.mlir
@@ -421,6 +421,53 @@ llvm.func @imp_fn() {
 
 // -----
 
+// Retained local variables may be represented separately from debug intrinsic
+// variables in MLIR. Ensure export reuses a single LLVM DILocalVariable node.
+
+#file = #llvm.di_file<"test.c" in "">
+#cu = #llvm.di_compile_unit<
+  id = distinct[0]<>, sourceLanguage = DW_LANG_C, file = #file
+>
+#void = #llvm.di_null_type
+#int = #llvm.di_basic_type<
+  tag = DW_TAG_base_type, name = "int",
+  sizeInBits = 32, encoding = DW_ATE_signed
+>
+#sp_self = #llvm.di_subprogram<recId = distinct[1]<>, isRecSelf = true>
+#retained_local = #llvm.di_local_variable<
+  scope = #sp_self, name = "retained_arg", file = #file,
+  line = 1, arg = 1, type = #int
+>
+#sp_type = #llvm.di_subroutine_type<types = #void, #int>
+#sp = #llvm.di_subprogram<
+  recId = distinct[1]<>, id = distinct[2]<>, compileUnit = #cu,
+  scope = #file, name = "fn_with_retained_local_export",
+  file = #file, line = 1, scopeLine = 1, subprogramFlags = Definition,
+  type = #sp_type, retainedNodes = [#retained_local]
+>
+#dbg_local = #llvm.di_local_variable<
+  scope = #sp, name = "retained_arg", file = #file,
+  line = 1, arg = 1, type = #int
+>
+
+// CHECK-LABEL: define void @fn_with_retained_local_export(
+// CHECK-SAME: i32 %[[ARG:.*]]) !dbg ![[SP:[0-9]+]]
+llvm.func @fn_with_retained_local_export(%arg0: i32) {
+  // RECORDS: #dbg_value(i32 %[[ARG]], ![[LOCAL:[0-9]+]],
+  // RECORDS-SAME: !DIExpression(), !{{.*}})
+  llvm.intr.dbg.value #dbg_local = %arg0 : i32 loc(fused<#sp>["test.c":1:1])
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])
+
+// CHECK-DAG: ![[SP]] = distinct !DISubprogram(
+// CHECK-SAME: name: "fn_with_retained_local_export"
+// CHECK-SAME: retainedNodes: ![[NODES:[0-9]+]]
+// CHECK-DAG: ![[NODES]] = !{![[LOCAL]]}
+// CHECK-DAG: ![[LOCAL]] = !DILocalVariable(name: "retained_arg", arg: 1,
+// CHECK-SAME: scope: ![[SP]]
+
+// -----
+
 // Nameless and scopeless global constant.
 
 // CHECK-LABEL: @.str.1 = external constant [10 x i8]


        

From llvmlistbot at llvm.org  Mon Jul 20 23:12:46 2026
From: llvmlistbot at llvm.org (Tobias Gysi)
Date: Mon, 20 Jul 2026 23:12:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LLVMIR] Avoid duplicate retained local
	variables (PR #209911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209911@github.com>
Message-ID: <6a5f0dde.5a0412d2.3daf32.b884@mx.google.com>


https://github.com/gysit closed https://github.com/llvm/llvm-project/pull/209911

From llvmlistbot at llvm.org  Mon Jul 20 23:25:49 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 20 Jul 2026 23:25:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix ArrayType::getSizeInBytes
 double counting the stride (PR #203821)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203821@github.com>
Message-ID: <6a5f10ed.6be7f24f.27361e.4eb3@mx.google.com>


aobolensk wrote:

@IgWod could you please take a look?

https://github.com/llvm/llvm-project/pull/203821

From llvmlistbot at llvm.org  Mon Jul 20 23:25:52 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 20 Jul 2026 23:25:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add MathToSPIRV conversion for
 copysign on OpenCL targets (PR #205994)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205994@github.com>
Message-ID: <6a5f10f0.b7b4cce5.31b99a.4f66@mx.google.com>


aobolensk wrote:

@IgWod could you please take a look?

https://github.com/llvm/llvm-project/pull/205994

From llvmlistbot at llvm.org  Mon Jul 20 23:25:54 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Mon, 20 Jul 2026 23:25:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a5f10f2.87185949.414bb.5d82@mx.google.com>


aobolensk wrote:

@IgWod could you please take a look?

https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Mon Jul 20 23:36:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Mon, 20 Jul 2026 23:36:49 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a5f1381.a5745248.6091c.7cee@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/202315

>From 216cbd2e710ce3c725bb51c5606bea4ba84a33a3 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Sat, 6 Jun 2026 14:38:54 -0500
Subject: [PATCH 01/10] Support for "atomic compare capture".

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  88 ++++---
 .../Interfaces/AtomicInterfaces.td            |   9 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  10 +
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |   6 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 216 ++++++++++++++----
 5 files changed, 250 insertions(+), 79 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 9d711b92bb520..5e02d3fb10dbb 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -558,16 +558,45 @@ void Fortran::lower::omp::lowerAtomic(
   int action1 = analysis.op1.what & analysis.Action;
   memOrder = makeValidForAction(memOrder, action0, action1, version);
 
+  // --- Shared capture scaffolding ---
+  mlir::Operation *captureOp = nullptr;
+  fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
+  fir::FirOpBuilder::InsertPoint atomicAt, postAt;
+
+  if (construct.IsCapture()) {
+    assert(action0 != analysis.None && action1 != analysis.None &&
+           "Expexcing two actions");
+    (void)action0;
+    (void)action1;
+    captureOp = mlir::omp::AtomicCaptureOp::create(
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+    // Set the non-atomic insertion point to before the atomic.capture.
+    preAt = getInsertionPointBefore(captureOp);
+
+    mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
+    builder.setInsertionPointToEnd(block);
+    // Set the atomic insertion point to before the terminator inside
+    // atomic.capture.
+    mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
+    atomicAt = getInsertionPointBefore(term);
+    postAt = getInsertionPointAfter(captureOp);
+    hint = nullptr;
+    memOrder = std::nullopt;
+  }
+
   if (auto *cond = get(analysis.cond)) {
     // atomic compare: if (x == e) x = d
     // e : expecteVal
     // d : desiredVal
 
-    // Check for compound clauses (fail, capture, weak) that are not yet
+    // Restore insertion point so pre-processing code (e.g. computing
+    // expectedVal) is emitted before the capture op, not after the terminator.
+    builder.restoreInsertionPoint(preAt);
+
+    // Check for compound clauses (fail, weak) that are not yet
     // supported with atomic compare.
     if (llvm::any_of(clauses, [](const omp::Clause &clause) {
           return clause.id == llvm::omp::Clause::OMPC_fail ||
-                 clause.id == llvm::omp::Clause::OMPC_capture ||
                  clause.id == llvm::omp::Clause::OMPC_weak;
         })) {
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
@@ -616,6 +645,17 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // If this is a compare+capture, generate the read op first.
+    if (construct.IsCapture()) {
+      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
+             "Expected a read operation for compare capture");
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
+
     mlir::UnitAttr weakAttr = nullptr;
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
@@ -679,34 +719,9 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
-
     // END omp atomic compare
   } else {
-    mlir::Operation *captureOp = nullptr;
-    fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
-    fir::FirOpBuilder::InsertPoint atomicAt, postAt;
-
-    if (construct.IsCapture()) {
-      // Capturing operation.
-      assert(action0 != analysis.None && action1 != analysis.None &&
-             "Expexcing two actions");
-      (void)action0;
-      (void)action1;
-      captureOp = mlir::omp::AtomicCaptureOp::create(
-          builder, loc, hint, makeMemOrderAttr(converter, memOrder));
-      // Set the non-atomic insertion point to before the atomic.capture.
-      preAt = getInsertionPointBefore(captureOp);
-
-      mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
-      builder.setInsertionPointToEnd(block);
-      // Set the atomic insertion point to before the terminator inside
-      // atomic.capture.
-      mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
-      atomicAt = getInsertionPointBefore(term);
-      postAt = getInsertionPointAfter(captureOp);
-      hint = nullptr;
-      memOrder = std::nullopt;
-    } else {
+    if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
              "Expexcing single action");
@@ -729,16 +744,13 @@ void Fortran::lower::omp::lowerAtomic(
           *get(analysis.op1.assign), hint, memOrder, preAt, atomicAt, postAt);
     }
 
-    if (construct.IsCapture()) {
-      // If this is a capture operation, the first/second ops will be inside
-      // of it. Set the insertion point to past the capture op itself.
-      builder.restoreInsertionPoint(postAt);
-    } else {
-      if (secondOp) {
-        builder.setInsertionPointAfter(secondOp);
-      } else {
-        builder.setInsertionPointAfter(firstOp);
-      }
+    if (!construct.IsCapture()) {
+      builder.setInsertionPointAfter(secondOp ? secondOp : firstOp);
     }
   }
+
+  // Shared capture cleanup.
+  if (construct.IsCapture()) {
+    builder.restoreInsertionPoint(postAt);
+  }
 }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index abb21705b3c1c..8c9015f05bb72 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -289,10 +289,12 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
+        auto secondCompareStmt = dyn_cast<AtomicCompareOpInterface>(secondOp);
 
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
-              (firstReadStmt && secondWriteStmt)))
+              (firstReadStmt && secondWriteStmt) ||
+              (firstReadStmt && secondCompareStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -310,6 +312,11 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
           return firstReadStmt.emitError()
                 << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstReadStmt && secondCompareStmt &&
+            firstReadStmt.getX() != secondCompareStmt.getX())
+          return firstReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "second operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..1241abc10298f 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1928,6 +1928,12 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
         omp.atomic.write ...
         omp.terminator
       }
+
+      omp.atomic.capture {
+        omp.atomic.read ...
+        omp.atomic.compare ...
+        omp.terminator
+      }
     ```
   }] # clausesDescription;
 
@@ -1946,6 +1952,10 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     /// Returns the `atomic.update` operation inside the region, if any.
     /// Otherwise, it returns nullptr.
     AtomicUpdateOp getAtomicUpdateOp();
+
+    /// Returns the `atomic.compare` operation inside the region, if any.
+    /// Otherwise, it returns nullptr.
+    AtomicCompareOp getAtomicCompareOp();
   }] # clausesExtraClassDeclaration;
 
   let hasRegionVerifier = 1;
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..0eafd0a267b97 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4615,6 +4615,12 @@ AtomicUpdateOp AtomicCaptureOp::getAtomicUpdateOp() {
   return dyn_cast<AtomicUpdateOp>(getSecondOp());
 }
 
+AtomicCompareOp AtomicCaptureOp::getAtomicCompareOp() {
+  if (auto op = dyn_cast<AtomicCompareOp>(getFirstOp()))
+    return op;
+  return dyn_cast<AtomicCompareOp>(getSecondOp());
+}
+
 LogicalResult AtomicCaptureOp::verify() {
   return verifySynchronizationHint(*this, getHint());
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 07fa92bdabe50..4928b288adb65 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4795,6 +4795,43 @@ convertOmpAtomicUpdate(omp::AtomicUpdateOp &opInst,
   return success();
 }
 
+/// Helper to extract the OMPAtomicCompareOp from an integer comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::ICmpPredicate::eq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::ICmpPredicate::slt:
+  case LLVM::ICmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::ICmpPredicate::sgt:
+  case LLVM::ICmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
+/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::FCmpPredicate::oeq:
+  case LLVM::FCmpPredicate::ueq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::FCmpPredicate::olt:
+  case LLVM::FCmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::FCmpPredicate::ogt:
+  case LLVM::FCmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -4803,13 +4840,149 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   if (failed(checkImplementationStatus(*atomicCaptureOp)))
     return failure();
 
+  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
+  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
+  omp::AtomicCompareOp atomicCompareOp = atomicCaptureOp.getAtomicCompareOp();
+
+  // If the capture contains an atomic.compare, delegate to
+  // createAtomicCompare with the capture variable (V) set.
+  if (atomicCompareOp) {
+    omp::AtomicReadOp atomicReadOp = atomicCaptureOp.getAtomicReadOp();
+    assert(atomicReadOp && "expected atomic.read in capture+compare");
+
+    Region &region = atomicCompareOp.getRegion();
+    Block &block = region.front();
+
+    llvm::Type *llvmXElementType =
+        moduleTranslation.convertType(block.getArgument(0).getType());
+    llvm::Value *llvmX = moduleTranslation.lookupValue(atomicCompareOp.getX());
+    llvm::Value *llvmV = moduleTranslation.lookupValue(atomicReadOp.getV());
+
+    bool isSigned = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicX = {
+        llvmX, llvmXElementType, isSigned, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicV = {
+        llvmV, llvmXElementType, /*isSigned=*/false, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicR = {nullptr, nullptr, false,
+                                                        false};
+
+    llvm::AtomicOrdering atomicOrdering =
+        convertAtomicOrdering(atomicCaptureOp.getMemoryOrder());
+
+    // Pre-translate non-pattern operations inside the compare region.
+    auto isAtomicComparePatternOp = [](Operation &op) {
+      return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
+                       LLVM::OrOp>(op);
+    };
+    for (Operation &op : block.without_terminator()) {
+      if (isAtomicComparePatternOp(op))
+        continue;
+      bool allOperandsMapped =
+          llvm::all_of(op.getOperands(), [&](mlir::Value v) {
+            return moduleTranslation.lookupValue(v) != nullptr;
+          });
+      if (!allOperandsMapped)
+        continue;
+      if (failed(moduleTranslation.convertOperation(op, builder)))
+        return atomicCompareOp.emitError(
+            "failed to translate operation inside atomic compare region");
+    }
+
+    auto materializeValue = [&](mlir::Value val) -> llvm::Value * {
+      if (llvm::Value *existing = moduleTranslation.lookupValue(val))
+        return existing;
+      if (auto loadOp = val.getDefiningOp<LLVM::LoadOp>()) {
+        if (loadOp->getParentRegion() == &region) {
+          llvm::Value *loadAddr =
+              moduleTranslation.lookupValue(loadOp.getAddr());
+          if (!loadAddr)
+            return nullptr;
+          llvm::Type *loadType =
+              moduleTranslation.convertType(loadOp.getResult().getType());
+          return builder.CreateLoad(loadType, loadAddr);
+        }
+      }
+      return nullptr;
+    };
+
+    // Extract comparison predicate, eVal, and dVal from the region.
+    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    llvm::Value *eVal = nullptr;
+    llvm::Value *dVal = nullptr;
+    bool isXBinopExpr = false;
+
+    for (Operation &op : block.getOperations()) {
+      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
+        auto maybeOp =
+            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+        isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                    pred == LLVM::ICmpPredicate::sgt ||
+                    pred == LLVM::ICmpPredicate::sle ||
+                    pred == LLVM::ICmpPredicate::sge);
+
+        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
+        auto maybeOp =
+            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        if (!dVal)
+          dVal = materializeValue(selectOp.getTrueValue());
+      }
+    }
+
+    if (!eVal)
+      return atomicCompareOp.emitError(
+          "failed to extract expected value (e) from atomic compare region");
+    if (!dVal) {
+      auto yieldOp = cast<omp::YieldOp>(block.getTerminator());
+      if (yieldOp.getResults().empty())
+        return atomicCompareOp.emitError(
+            "failed to extract desired value (d) from atomic compare region");
+      dVal = materializeValue(yieldOp.getResults()[0]);
+    }
+
+    llvmAtomicX.IsSigned = isSigned;
+
+    llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    bool isPostfixUpdate = true;
+
+    bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, /*IsFailOnly=*/false);
+    ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
+
+    if (failed(handleError(afterIP, *atomicCaptureOp)))
+      return failure();
+
+    builder.restoreIP(*afterIP);
+    return success();
+  }
+
   mlir::Value mlirExpr;
   bool isXBinopExpr = false, isPostfixUpdate = false;
   llvm::AtomicRMWInst::BinOp binop = llvm::AtomicRMWInst::BinOp::BAD_BINOP;
 
-  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
-  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
-
   assert((atomicUpdateOp || atomicWriteOp) &&
          "internal op must be an atomic.update or atomic.write op");
 
@@ -4896,43 +5069,6 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   return success();
 }
 
-/// Helper to extract the OMPAtomicCompareOp from an integer comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::ICmpPredicate::eq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::ICmpPredicate::slt:
-  case LLVM::ICmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::ICmpPredicate::sgt:
-  case LLVM::ICmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
-/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::FCmpPredicate::oeq:
-  case LLVM::FCmpPredicate::ueq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::FCmpPredicate::olt:
-  case LLVM::FCmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::FCmpPredicate::ogt:
-  case LLVM::FCmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
 /// Converts an omp.atomic.compare operation to LLVM IR.
 ///
 ///      if (x == e) x = d

>From 7d3293944598e8090a64cdec2ac3c791a64b73f1 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Mon, 8 Jun 2026 05:12:22 -0500
Subject: [PATCH 02/10] Adding tests for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     | 32 +++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 56 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 29 ++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  9 +--
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 41 ++++++++++++++
 6 files changed, 211 insertions(+), 4 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 249fb0dd8fa64..650f64b80af12 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,3 +260,35 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
+! Integer equality compare+capture: cmpxchg + store old value
+!CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
+! Compare+capture with clause order reversed: capture compare
+!CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_capture_compare_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic capture compare
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ac70edbed4e60..752a221aa538d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -161,3 +161,59 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
   !$omp atomic compare weak
   if (x .eq. e) x = d
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_eq(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_gt(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           } {{.*}}weak{{.*}}
+! CHECK:         }
+subroutine atomic_compare_capture_int_gt(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+    v = x
+    if (x > e) x = d
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 4f3cf0eac0338..dc0cc1a62f6c2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -192,6 +192,26 @@ program main
       i = j
    end if
 
+!COMPARE CAPTURE
+!$omp atomic compare capture
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare weak
+   k = i
+   if (i < j) then
+      i = k
+   end if
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -296,6 +316,15 @@ end program main
 !CHECK: !$OMP ATOMIC WEAK COMPARE
 !CHECK: !$OMP ATOMIC COMPARE SEQ_CST WEAK
 
+!COMPARE CAPTURE
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 17f99a8c86d52..c5a07a7dc6cb2 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4963,13 +4963,14 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
     bool isPostfixUpdate = true;
+    bool isWeak = atomicCompareOp.getWeak();
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, /*IsFailOnly=*/false);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            /*IsFailOnly=*/false, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index acf6879906e74..289ec60551cb7 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2047,6 +2047,54 @@ func.func @omp_atomic_compare(%x: memref<i32>, %e: i32, %d: i32) {
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_weak
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: } {weak}
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi sgt, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 1890eaa6a3f0b..6ce57b304bfd9 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2795,6 +2795,47 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From f84987dea68a1dedc6b61bf0c9f5a2e24a0a1b40 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 9 Jun 2026 04:42:48 -0500
Subject: [PATCH 03/10] Minor changes incorporating feedback.

---
 flang/lib/Lower/OpenMP/Atomic.cpp                             | 4 ++--
 .../Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index c0f97aada637d..aa497b838e5b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -567,7 +567,7 @@ void Fortran::lower::omp::lowerAtomic(
 
   if (construct.IsCapture()) {
     assert(action0 != analysis.None && action1 != analysis.None &&
-           "Expexcing two actions");
+           "Expecting two actions");
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
@@ -730,7 +730,7 @@ void Fortran::lower::omp::lowerAtomic(
     if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
-             "Expexcing single action");
+             "Expecting single action");
       assert(!(analysis.op0.what & analysis.Condition));
       postAt = atomicAt = preAt;
     }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 8c9015f05bb72..dc5b04126b757 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -315,7 +315,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be updated in "
+                << "captured variable in atomic.read must be compared/updated in "
                     "second operation";
 
         return mlir::success();

>From ca24a147920d24c188127dbbe37ca62bcaeab792 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 14:18:17 -0500
Subject: [PATCH 04/10] Handling test cases mentionend in OpenMP spec.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 61 ++++++++++++++++---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 ++-
 .../Interfaces/AtomicInterfaces.td            | 11 +++-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  2 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 12 ++--
 5 files changed, 74 insertions(+), 19 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index aa497b838e5b6..209871abd52b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -571,7 +571,8 @@ void Fortran::lower::omp::lowerAtomic(
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
-        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder),
+        /*fail_only=*/nullptr);
     // Set the non-atomic insertion point to before the atomic.capture.
     preAt = getInsertionPointBefore(captureOp);
 
@@ -646,15 +647,45 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
-    // If this is a compare+capture, generate the read op first.
+    // If this is a compare+capture, determine the ordering of ops.
+    // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
+    // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
+    // Pattern 3 (fail-only): if (x == e) x = d; else v = x → read on failure
+    bool isPostfixCapture = false;
+    bool isFailOnly = false;
+    const evaluate::Assignment *readAssign = nullptr;
     if (construct.IsCapture()) {
-      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
-             "Expected a read operation for compare capture");
-      mlir::Operation *readOp = genAtomicRead(
-          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
-          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
-      assert(readOp && "Should have created an atomic read operation");
-      builder.setInsertionPointAfter(readOp);
+      // Determine which op is the read and check for fail-only (IfFalse).
+      int readWhat = 0;
+      if (analysis.op0.what & analysis.Read) {
+        readAssign = get(analysis.op0.assign);
+        readWhat = analysis.op0.what;
+      } else if (analysis.op1.what & analysis.Read) {
+        readAssign = get(analysis.op1.assign);
+        readWhat = analysis.op1.what;
+        isPostfixCapture = true;
+      }
+      assert(readAssign && "Expected a read assignment for compare capture");
+
+      // Check if the read is conditioned on comparison failure (else branch).
+      if (readWhat & analysis.IfFalse)
+        isFailOnly = true;
+
+      if (!isPostfixCapture && !isFailOnly) {
+        // Pattern 1 (prefix): read is first, generate it before compare.
+        mlir::Operation *readOp =
+            genAtomicRead(converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+                          *readAssign, hint, memOrder, preAt, atomicAt, postAt);
+        assert(readOp && "Should have created an atomic read operation");
+        builder.setInsertionPointAfter(readOp);
+      } else {
+        // Pattern 2 (postfix) or 3 (fail-only): compare first, read after.
+        builder.restoreInsertionPoint(atomicAt);
+      }
+
+      // Set the fail_only attribute on the capture op.
+      if (isFailOnly && captureOp)
+        mlir::cast<mlir::omp::AtomicCaptureOp>(captureOp).setFailOnly(true);
     }
 
     mlir::UnitAttr weakAttr = nullptr;
@@ -725,6 +756,18 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
+
+    // Pattern 2 (postfix) or 3 (fail-only): compare first, read second.
+    // Generate read after compare for postfix or fail-only patterns.
+    if (construct.IsCapture() && (isPostfixCapture || isFailOnly)) {
+      fir::FirOpBuilder::InsertPoint afterCompareAt =
+          builder.saveInsertionPoint();
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom, *readAssign, hint,
+          memOrder, preAt, afterCompareAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
     // END omp atomic compare
   } else {
     if (!construct.IsCapture()) {
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 92eb7de0d882f..b2ccc05ad6770 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11321,7 +11321,9 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11377,7 +11379,8 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index dc5b04126b757..5b1527bcb84ec 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -286,6 +286,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto &secondOp = *ops.getNextNode(firstOp);
         auto firstReadStmt = dyn_cast<AtomicReadOpInterface>(firstOp);
         auto firstUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(firstOp);
+        auto firstCompareStmt = dyn_cast<AtomicCompareOpInterface>(firstOp);
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
@@ -294,7 +295,8 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
               (firstReadStmt && secondWriteStmt) ||
-              (firstReadStmt && secondCompareStmt)))
+              (firstReadStmt && secondCompareStmt) ||
+              (firstCompareStmt && secondReadStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -315,8 +317,13 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be compared/updated in "
+                << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstCompareStmt && secondReadStmt &&
+            secondReadStmt.getX() != firstCompareStmt.getX())
+          return secondReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "first operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index ebee887f2afd1..69c22641df6a2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1938,6 +1938,8 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     ```
   }] # clausesDescription;
 
+  let arguments = !con(clausesArgs, (ins UnitAttr:$fail_only));
+
   // Override region definition.
   let regions = (region SizedRegion<1>:$region);
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index dff1ae30e20aa..76616c7fba27a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,15 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    bool isPostfixUpdate = true;
-    bool isWeak = atomicCompareOp.getWeak();
+    bool isFailOnly = atomicCaptureOp.getFailOnly();
+    bool isPostfixUpdate = !isFailOnly;
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(
-            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
-            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
-            /*IsFailOnly=*/false, isWeak);
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, isFailOnly);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))

>From afaf2025463e9e8cf15974747fa9e2d8b8e5fa95 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 18:02:52 -0500
Subject: [PATCH 05/10] Adding tests into flang and mlir for the grammar
 mentionend in the spec.

---
 .../Integration/OpenMP/atomic-compare.f90     | 56 ++++++++++++++++--
 flang/test/Lower/OpenMP/atomic-compare.f90    | 59 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 17 ++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 +++++-
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 58 ++++++++++++++++--
 6 files changed, 245 insertions(+), 10 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 650f64b80af12..da86f3ff6ef16 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,14 +260,17 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
-! Integer equality compare+capture: cmpxchg + store old value
+! Integer equality compare+capture (prefix): v=x; if(x==e) x=d
+! v captures old value of x
 !CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -276,14 +279,16 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
-! Compare+capture with clause order reversed: capture compare
+! Compare+capture with clause order reversed: capture compare (still prefix read)
 !CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
@@ -292,3 +297,46 @@ subroutine atomic_capture_compare_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
+! Postfix compare+capture: if (x == e) x = d; v = x
+! v captures new value (d if swapped, old x if not)
+!CHECK-LABEL: define void @atomic_compare_capture_postfix_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Fail-only compare+capture: if (x == e) x = d; else v = x
+! v is only written when the comparison fails
+!CHECK-LABEL: define void @atomic_compare_capture_fail_only_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+!CHECK: {{.*}}:
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: br label %[[EXIT2:.*]]
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 752a221aa538d..c770605954539 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -217,3 +217,62 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
     if (x > e) x = d
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_fail_only(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index dc0cc1a62f6c2..521c1bb2811e2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -211,6 +211,19 @@ program main
       i = k
    end if
 !$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   end if
+   k = i
+!$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   else
+      k = i
+   end if
+!$omp end atomic
 
 !ATOMIC
 !$omp atomic
@@ -324,6 +337,10 @@ end program main
 !CHECK: !$OMP END ATOMIC
 !CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
 !CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
 
 !ATOMIC
 !CHECK: !$OMP ATOMIC
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 76616c7fba27a..b6fc6d6dc3038 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,28 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    // In OMPIRBuilder::createAtomicCompare:
+    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
+    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
+    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
+    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
+    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
+    bool isPostfixUpdate =
+        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
     bool isFailOnly = atomicCaptureOp.getFailOnly();
-    bool isPostfixUpdate = !isFailOnly;
 
+    // For fail-only, the OMPIRBuilder's conditional store logic is under
+    // the !IsPostfixUpdate path, so force it to false.
+    if (isFailOnly)
+      isPostfixUpdate = false;
+
+    bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
         ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly);
+                                        isPostfixUpdate, isFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 87f9a6f8e4119..6b13be481e0e0 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2113,6 +2113,54 @@ func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e:
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_postfix
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_postfix(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: } {fail_only}
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  } {fail_only}
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 6ce57b304bfd9..978450d598b9b 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,10 +2798,12 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2819,10 +2821,12 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2836,6 +2840,52 @@ llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_postfix
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Postfix compare+capture: v captures new value (d if swapped, old x if not)
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[D]], i32 %[[OLD]]
+  // CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Fail-only compare+capture: v is only written when comparison fails
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+  // CHECK: [[CONT]]:
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: br label %[[EXIT2:.*]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From e8462a267b6d010ebc1e3ed881c1f38a9faca6be Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 24 Jun 2026 14:44:43 -0500
Subject: [PATCH 06/10] Simplifying the IR generator for one of the cases. This
 also fixes the failure in clang for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     |  8 +---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 43 +++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 12 ++----
 4 files changed, 38 insertions(+), 32 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index da86f3ff6ef16..4ae1b9a1f1e52 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -268,9 +268,7 @@ subroutine atomic_compare_weak(x, e, d)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -286,9 +284,7 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 51f210e189e22..f8c1999fe1b89 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11400,9 +11400,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11458,8 +11456,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 85389caa9ac42..e1fb08d780753 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5499,25 +5499,23 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    // In OMPIRBuilder::createAtomicCompare:
-    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
-    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
-    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
-    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
-    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
-    bool isPostfixUpdate =
-        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isReadFirst = isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
-    // For fail-only, the OMPIRBuilder's conditional store logic is under
-    // the !IsPostfixUpdate path, so force it to false.
-    if (isFailOnly)
-      isPostfixUpdate = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
+    // Postfix: bypass V in OMPIRBuilder, handled manually below.
+    if (isPostfixCapture && !isFailOnly)
+      llvmAtomicVForCall = {nullptr, nullptr, false, false};
+
+    // Prefix: IsPostfixUpdate=true → direct store of old value.
+    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
+    bool isPostfixUpdate = !isFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
                                         isPostfixUpdate, isFailOnly, isWeak);
@@ -5527,6 +5525,25 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
       return failure();
 
     builder.restoreIP(*afterIP);
+
+    // Postfix: v = select(success, D, old) — captures new value of x.
+    if (isPostfixCapture && !isFailOnly) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::Instruction *cmpxchgInst = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
+          cmpxchgInst = &inst;
+          break;
+        }
+      }
+      assert(cmpxchgInst && "expected cmpxchg instruction");
+      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
+      llvm::Value *successVal =
+          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+      llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
+      builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+    }
+
     return success();
   }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 978450d598b9b..377bd88894854 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,12 +2798,10 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2821,12 +2819,10 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality weak prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {

>From 101b9fa02882686a21d346ef12e541c717ca605b Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 14:00:46 -0500
Subject: [PATCH 07/10] Handling real and logical types.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 117 ++++++--
 .../Integration/OpenMP/atomic-compare.f90     |  58 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    |  83 +++++-
 flang/test/Parser/OpenMP/atomic-unparse.f90   |  12 +
 .../Interfaces/AtomicInterfaces.td            |  13 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 252 +++++++++++-------
 mlir/test/Dialect/OpenMP/ops.mlir             |  24 ++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++
 8 files changed, 470 insertions(+), 137 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 209871abd52b6..3811ea3d40bbd 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -375,6 +375,8 @@ genAtomicRead(lower::AbstractConverter &converter,
         fir::getBase(converter.genExprValue(assign.rhs, stmtCtx, &loc));
     converter.resetExprOverrides();
 
+    if (value.getType() != storeType)
+      value = builder.createConvert(loc, storeType, value);
     fir::StoreOp::create(builder, loc, value, storeAddr);
   }
   return op;
@@ -634,6 +636,45 @@ void Fortran::lower::omp::lowerAtomic(
           },
           rel->u);
     }
+
+    // Fortran uses .eqv./.neqv. for logical equality/inequality, which are
+    // LogicalOperation expressions rather than Relational expressions.
+    if (!expectedExprStorage) {
+      if (const auto *someLogical =
+              evaluate::UnwrapExpr<evaluate::Expr<evaluate::SomeLogical>>(
+                  *cond)) {
+        common::visit(
+            [&](const auto &kindLogical) {
+              using LogicalExpr = std::decay_t<decltype(kindLogical)>;
+              constexpr int K = LogicalExpr::Result::kind;
+              if (const auto *logOp =
+                      std::get_if<evaluate::LogicalOperation<K>>(
+                          &kindLogical.u)) {
+                if (logOp->logicalOperator == common::LogicalOperator::Eqv ||
+                    logOp->logicalOperator == common::LogicalOperator::Neqv) {
+                  relOpr =
+                      (logOp->logicalOperator == common::LogicalOperator::Eqv)
+                          ? common::RelationalOperator::EQ
+                          : common::RelationalOperator::NE;
+                  using Operand =
+                      typename evaluate::LogicalOperation<K>::Operand;
+                  auto leftExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->left()});
+                  auto rightExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->right()});
+                  if (evaluate::IsSameOrConvertOf(rightExpr, atom)) {
+                    expectedExprStorage = std::move(leftExpr);
+                    relOpr = reverseRelOp(relOpr);
+                  } else {
+                    expectedExprStorage = std::move(rightExpr);
+                  }
+                }
+              }
+            },
+            someLogical->u);
+      }
+    }
+
     if (!expectedExprStorage) {
       mlir::emitError(loc, "internal error: atomic compare condition is not a "
                            "recognized relational expression");
@@ -647,6 +688,20 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // For logical types, convert address and expected value to integer
+    // type here (above the atomic compare region) so that the region
+    // only contains arith.cmpi eq on integers. If done inside the region,
+    // fir.convert logical<4> -> i32 would lower to `icmp ne %val, 0`
+    // which violates the atomic compare verifier's expectations.
+    if (mlir::isa<fir::LogicalType>(elemTypeOfX)) {
+      unsigned kind = mlir::cast<fir::LogicalType>(elemTypeOfX).getFKind();
+      mlir::Type intTy = builder.getIntegerType(kind * 8);
+      mlir::Type intRefTy = builder.getRefType(intTy);
+      atomAddr = builder.createConvert(loc, intRefTy, atomAddr);
+      expectedVal = builder.createConvert(loc, intTy, expectedVal);
+      elemTypeOfX = intTy;
+    }
+
     // If this is a compare+capture, determine the ordering of ops.
     // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
     // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
@@ -694,6 +749,39 @@ void Fortran::lower::omp::lowerAtomic(
         })) {
       weakAttr = builder.getUnitAttr();
     }
+
+    // Extract write assignment (x = d) and generate desired value (d)
+    // before creating the compare region, so that d is defined outside
+    // the region and any intermediate conversions (e.g., logical-to-integer
+    // truthiness normalization) don't appear inside the atomic compare block.
+    [[maybe_unused]] int writeActionCond = 0;
+    const evaluate::Assignment *writeAssign = nullptr;
+    if (analysis.op0.what & analysis.Write) {
+      writeAssign = get(analysis.op0.assign);
+      writeActionCond = analysis.op0.what;
+    }
+    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
+      writeAssign = get(analysis.op1.assign);
+      writeActionCond = analysis.op1.what;
+    }
+    if (!writeAssign) {
+      mlir::emitError(loc,
+                      "internal error: atomic compare has no write assignment");
+      return;
+    }
+    assert((writeActionCond & analysis.IfTrue) &&
+           "atomic compare write should be conditioned on IfTrue");
+
+    // Generate desiredVal before the capture/compare region so any
+    // intermediate ops (loads, conversions) don't pollute the atomic blocks.
+    fir::FirOpBuilder::InsertPoint savedIP = builder.saveInsertionPoint();
+    builder.restoreInsertionPoint(preAt);
+    mlir::Value desiredVal =
+        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
+    if (desiredVal.getType() != elemTypeOfX)
+      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
+    builder.restoreInsertionPoint(savedIP);
+
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
         makeMemOrderAttr(converter, memOrder));
@@ -721,35 +809,6 @@ void Fortran::lower::omp::lowerAtomic(
       return;
     }
 
-    // Check for presence of Assignment (x = d) and wether it is being invoked
-    // only for IfTrue condition.
-
-    // writeActionCond is a bitmask combining the following flags:
-    //  1) the action type (Read/Write/Update)
-    //  2) condition (IfTrue/IfFalse)
-    [[maybe_unused]] int writeActionCond = 0;
-    const evaluate::Assignment *writeAssign = nullptr;
-    if (analysis.op0.what & analysis.Write) {
-      writeAssign = get(analysis.op0.assign);
-      writeActionCond = analysis.op0.what;
-    }
-    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
-      writeAssign = get(analysis.op1.assign);
-      writeActionCond = analysis.op1.what;
-    }
-    if (!writeAssign) {
-      mlir::emitError(loc,
-                      "internal error: atomic compare has no write assignment");
-      return;
-    }
-    assert((writeActionCond & analysis.IfTrue) &&
-           "atomic compare write should be conditioned on IfTrue");
-
-    // Generate new/desired value of x e.g. x = d
-    mlir::Value desiredVal =
-        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
-    if (desiredVal.getType() != elemTypeOfX)
-      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
     mlir::Value newVal = mlir::arith::SelectOp::create(builder, loc, cmpResult,
                                                        desiredVal, blockArg);
 
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 4ae1b9a1f1e52..f30349c42e33e 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -336,3 +336,61 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
   !$omp end atomic
 end
 
+! Real equality compare+capture (postfix): if(x==e) x=d; v=x
+! v captures new value of x (d if success, old if fail)
+!CHECK-LABEL: define void @atomic_compare_capture_real_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load float, ptr %[[E]]{{.*}}
+!CHECK: %[[DVAL:.*]] = load float, ptr %[[D]]{{.*}}
+!CHECK: %[[EBITS:.*]] = bitcast float %[[EVAL]] to i32
+!CHECK: %[[DBITS:.*]] = bitcast float %[[DVAL]] to i32
+!CHECK: %[[XLOAD:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+!CHECK: %[[XFP:.*]] = bitcast i32 %[[XLOAD]] to float
+! Part 1: NaN check - if either x or e is NaN, comparison fails
+!CHECK: %[[E_ISNAN:.*]] = fcmp uno float %[[EVAL]], %[[EVAL]]
+!CHECK: %[[X_ISNAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+!CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_ISNAN]], %[[X_ISNAN]]
+!CHECK: br i1 %[[EITHER_NAN]], label %[[NAN_BB:.*]], label %[[NOTNAN_BB:.*]]
+!CHECK: [[NAN_BB]]:
+!CHECK: br label %[[EXIT_BB:.*]]
+! Part 2: Both-zero check - handles +0.0 vs -0.0 (same value, different bits)
+!CHECK: [[NOTNAN_BB]]:
+!CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+!CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[EVAL]], 0.000000e+00
+!CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+!CHECK: br i1 %[[BOTHZERO]], label %[[ZERO_BB:.*]], label %[[NORMAL_BB:.*]]
+!CHECK: [[ZERO_BB]]:
+!CHECK: %[[ZERORES:.*]] = cmpxchg ptr %[[X]], i32 %[[XLOAD]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Part 3: Normal compare - standard cmpxchg with bitcasted expected value
+!CHECK: [[NORMAL_BB]]:
+!CHECK: %[[NORMRES:.*]] = cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Exit: select v = (success ? d : old_x)
+!CHECK: [[EXIT_BB]]:
+!CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+!CHECK: %[[OK:.*]] = phi i1 {{.*}}
+!CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+!CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[DVAL]], float %[[OLDFP]]
+!CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Logical .eqv. compare+capture (postfix): if(x.eqv.e) x=d; v=x
+! Logicals are compared as integers after truthiness normalization
+!CHECK-LABEL: define void @atomic_compare_capture_logical_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: cmpxchg ptr %[[X]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index c770605954539..1219d05296e4d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -10,10 +10,10 @@
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -31,10 +31,10 @@ subroutine atomic_compare_int_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -52,10 +52,10 @@ subroutine atomic_compare_float_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<complex<f32>>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<complex<f32>> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: complex<f32>):
 ! CHECK:           %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %[[EVAL]] {fastmath = #arith.fastmath<contract>} : complex<f32>
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : complex<f32>
 ! CHECK:           omp.yield(%[[SEL]] : complex<f32>)
 ! CHECK:         }
@@ -71,10 +71,10 @@ subroutine atomic_compare_complex_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi slt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -90,10 +90,10 @@ subroutine atomic_compare_int_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -109,10 +109,10 @@ subroutine atomic_compare_int_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -128,10 +128,10 @@ subroutine atomic_compare_float_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -149,10 +149,10 @@ subroutine atomic_compare_float_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         } {{.*}}weak{{.*}}
@@ -172,12 +172,12 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -200,12 +200,12 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           } {{.*}}weak{{.*}}
@@ -228,11 +228,11 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -256,11 +256,11 @@ subroutine atomic_compare_capture_postfix(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -276,3 +276,62 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
     end if
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<f32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<f32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<f32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<f32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:         %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         %[[DLOAD:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[DCONV:.*]] = fir.convert %[[DLOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DCONV]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 521c1bb2811e2..0d30cd48035ea 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -4,6 +4,7 @@ program main
    implicit none
    integer :: i, j = 10
    integer :: k
+   logical :: l1, l2, l3, l4
 !READ
 !$omp atomic read
    i = j
@@ -225,6 +226,12 @@ program main
    end if
 !$omp end atomic
 
+!COMPARE CAPTURE LOGICAL
+!$omp atomic compare capture
+   if (l1 .eqv. l2) l1 = l3
+   l4 = l1
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -342,6 +349,11 @@ end program main
 !CHECK: !$OMP ATOMIC COMPARE CAPTURE
 !CHECK: !$OMP END ATOMIC
 
+!COMPARE CAPTURE LOGICAL
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 5b1527bcb84ec..a61c43e2fd9cd 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -488,6 +488,19 @@ def AtomicCompareOpInterface : OpInterface<"AtomicCompareOpInterface"> {
               }
             }
             break;
+          } else if (llvm::is_contained(
+                         {"arith.maxsi", "arith.minsi", "arith.maxui",
+                          "arith.minui", "arith.maximumf", "arith.minimumf",
+                          "llvm.intr.smax", "llvm.intr.smin",
+                          "llvm.intr.umax", "llvm.intr.umin",
+                          "llvm.intr.maxnum", "llvm.intr.minnum"},
+                         opName)) {
+            // The canonicalizer may fold select(cmpi slt, e, x) into
+            // maxsi/minsi (or the float equivalents). After fir-to-llvm-ir
+            // these become llvm.intr.smax/smin etc. All are valid
+            // representations of atomic compare min/max operations.
+            foundComparison = true;
+            break;
           }
         }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e1fb08d780753..8e49571cd2461 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 #include "mlir/Analysis/TopologicalSortUtils.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMTypes.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
@@ -5369,6 +5370,102 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Holds the extracted comparison pattern information from an atomic compare
+/// region.
+struct AtomicComparePatternInfo {
+  llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+  llvm::Value *eVal = nullptr;
+  llvm::Value *dVal = nullptr;
+  bool isXBinopExpr = false;
+  bool isSigned = false;
+};
+
+/// Extract comparison predicate, expected value (e), desired value (d), and
+/// related flags from an atomic compare region block by scanning for
+/// icmp/fcmp/select/min/max operations.
+static LogicalResult extractAtomicComparePattern(
+    Block &block,
+    llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
+    omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  for (Operation &op : block.getOperations()) {
+    // Pre-filter: skip icmps that don't involve the block argument
+    // (e.g., truthiness extractions from logical-to-integer conversion).
+    if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op);
+        icmpOp && icmpOp.getOperand(0) != block.getArgument(0) &&
+        icmpOp.getOperand(1) != block.getArgument(0))
+      continue;
+
+    LogicalResult result =
+        llvm::TypeSwitch<Operation *, LogicalResult>(&op)
+            .Case<LLVM::ICmpOp>([&](LLVM::ICmpOp icmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+              info.isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                               pred == LLVM::ICmpPredicate::sgt ||
+                               pred == LLVM::ICmpPredicate::sle ||
+                               pred == LLVM::ICmpPredicate::sge);
+              info.isXBinopExpr =
+                  (icmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? icmpOp.getOperand(1)
+                                                       : icmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::FCmpOp>([&](LLVM::FCmpOp fcmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              info.isXBinopExpr =
+                  (fcmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? fcmpOp.getOperand(1)
+                                                       : fcmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::SelectOp>([&](LLVM::SelectOp selectOp) {
+              if (!info.dVal)
+                info.dVal = materializeValue(selectOp.getTrueValue());
+              return success();
+            })
+            .Case<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                  mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                  mlir::arith::MaximumFOp, mlir::arith::MinimumFOp,
+                  LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp, LLVM::UMinOp,
+                  LLVM::MaxNumOp, LLVM::MinNumOp>([&](Operation *) {
+              // Canonicalized min/max ops (arith or LLVM intrinsic form).
+              // max(x,e) came from slt/ult/olt -> OMPAtomicCompareOp::MIN
+              // min(x,e) came from sgt/ugt/ogt -> OMPAtomicCompareOp::MAX
+              // (OMPIRBuilder inverts: MIN->atomicrmw max, MAX->atomicrmw min)
+              bool isMax = isa<mlir::arith::MaxSIOp, mlir::arith::MaxUIOp,
+                               mlir::arith::MaximumFOp, LLVM::SMaxOp,
+                               LLVM::UMaxOp, LLVM::MaxNumOp>(op);
+              info.compareOp = isMax ? llvm::omp::OMPAtomicCompareOp::MIN
+                                     : llvm::omp::OMPAtomicCompareOp::MAX;
+              info.isSigned = isa<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                                  LLVM::SMaxOp, LLVM::SMinOp>(op);
+              info.isXBinopExpr = (op.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand =
+                  info.isXBinopExpr ? op.getOperand(1) : op.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              info.dVal = info.eVal;
+              return success();
+            })
+            .Default([](Operation *) { return success(); });
+
+    if (failed(result))
+      return result;
+  }
+  return success();
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -5409,7 +5506,11 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Pre-translate non-pattern operations inside the compare region.
     auto isAtomicComparePatternOp = [](Operation &op) {
       return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                       LLVM::OrOp>(op);
+                       LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                       LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                       mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                       mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                       mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
     };
     for (Operation &op : block.without_terminator()) {
       if (isAtomicComparePatternOp(op))
@@ -5443,47 +5544,16 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     };
 
     // Extract comparison predicate, eVal, and dVal from the region.
-    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    llvm::Value *eVal = nullptr;
-    llvm::Value *dVal = nullptr;
-    bool isXBinopExpr = false;
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
 
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
+    llvm::omp::OMPAtomicCompareOp compareOp = patternInfo.compareOp;
+    llvm::Value *eVal = patternInfo.eVal;
+    llvm::Value *dVal = patternInfo.dVal;
+    bool isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
 
     if (!eVal)
       return atomicCompareOp.emitError(
@@ -5529,17 +5599,42 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Postfix: v = select(success, D, old) — captures new value of x.
     if (isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
-      llvm::Instruction *cmpxchgInst = nullptr;
+      llvm::Value *oldVal = nullptr;
+      llvm::Value *successVal = nullptr;
+
+      // Integer path (and non-HandleFPNegZero FP path): a single cmpxchg
+      // lives in the current block.
       for (auto &inst : llvm::reverse(*curBB)) {
         if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
-          cmpxchgInst = &inst;
+          oldVal = builder.CreateExtractValue(&inst, /*Idxs=*/0);
+          successVal = builder.CreateExtractValue(&inst, /*Idxs=*/1);
           break;
         }
       }
-      assert(cmpxchgInst && "expected cmpxchg instruction");
-      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
-      llvm::Value *successVal =
-          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+
+      // FP HandleFPNegZero path: the OMPIRBuilder emits a multi-block
+      // structure (NaN / ±0.0 handling) with cmpxchg in predecessor
+      // blocks. Results are merged via PHI nodes in the current (exit)
+      // block: an i1 PHI for success and a bitcast of an integer PHI
+      // for the old FP value.
+      if (!oldVal) {
+        for (auto &inst : *curBB) {
+          auto *phi = dyn_cast<llvm::PHINode>(&inst);
+          if (!phi)
+            break;
+          if (phi->getType()->isIntegerTy(1))
+            successVal = phi;
+        }
+        for (auto &inst : *curBB) {
+          if (auto *bc = dyn_cast<llvm::BitCastInst>(&inst)) {
+            oldVal = bc;
+            break;
+          }
+        }
+      }
+
+      assert(oldVal && "expected cmpxchg or PHI+bitcast for compare capture");
+      assert(successVal && "expected success flag for compare capture");
       llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
       builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
     }
@@ -5686,7 +5781,11 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
 
   auto isAtomicComparePatternOp = [](Operation &op) {
     return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                     LLVM::OrOp>(op);
+                     LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                     LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                     mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                     mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                     mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
   };
 
   // Pre-translate operations inside the region that compute e and d (e.g.,
@@ -5859,54 +5958,15 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
     }
     return success();
   } else {
-
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        // Identify which operand is the block argument (x) and which is e.
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
-  }
-
-  // For non-complex patterns, also extract dVal from SelectOp.
-  if (!dVal) {
-    for (Operation &op : block.getOperations()) {
-      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        dVal = materializeValue(selectOp.getTrueValue());
-        break;
-      }
-    }
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
+    compareOp = patternInfo.compareOp;
+    eVal = patternInfo.eVal;
+    dVal = patternInfo.dVal;
+    isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
   }
 
   if (!eVal)
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 6b13be481e0e0..cc40506b909b1 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2161,6 +2161,30 @@ func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_real
+// CHECK-SAME: (%[[V:.*]]: memref<f32>, %[[X:.*]]: memref<f32>, %[[E:.*]]: f32, %[[D:.*]]: f32)
+func.func @omp_atomic_compare_capture_real(%v: memref<f32>, %x: memref<f32>, %e: f32, %d: f32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<f32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: f32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[E]] : f32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : f32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : f32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<f32>, memref<f32>, f32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<f32> {
+    ^bb0(%xval: f32):
+      %cmp = arith.cmpf oeq, %xval, %e : f32
+      %sel = arith.select %cmp, %d, %xval : f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : memref<f32>, memref<f32>, f32
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 377bd88894854..b779d20f42d0c 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2882,6 +2882,54 @@ llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr,
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_real
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], float %[[E:.*]], float %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_real(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32, %d : f32) {
+  // Real compare+capture: uses HandleFPNegZero multi-block path
+  // CHECK: %[[EBITS:.*]] = bitcast float %[[E]] to i32
+  // CHECK: %[[DBITS:.*]] = bitcast float %[[D]] to i32
+  // CHECK: %[[XI:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+  // CHECK: %[[XFP:.*]] = bitcast i32 %[[XI]] to float
+  // Part 1: NaN check
+  // CHECK: %[[E_NAN:.*]] = fcmp uno float %[[E]], %[[E]]
+  // CHECK: %[[X_NAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+  // CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_NAN]], %[[X_NAN]]
+  // CHECK: br i1 %[[EITHER_NAN]], label %[[NAN:.*]], label %[[NOTNAN:.*]]
+  // CHECK: [[NAN]]:
+  // CHECK: br label %[[EXIT:.*]]
+  // Part 2: Both-zero check
+  // CHECK: [[NOTNAN]]:
+  // CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+  // CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[E]], 0.000000e+00
+  // CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+  // CHECK: br i1 %[[BOTHZERO]], label %[[ZERO:.*]], label %[[NORMAL:.*]]
+  // CHECK: [[ZERO]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[XI]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Part 3: Normal cmpxchg
+  // CHECK: [[NORMAL]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Exit: select v = (success ? d : old_x)
+  // CHECK: [[EXIT]]:
+  // CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+  // CHECK: %[[OK:.*]] = phi i1 {{.*}}
+  // CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+  // CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[D]], float %[[OLDFP]]
+  // CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "oeq" %xval, %e : f32
+      %sel = llvm.select %cmp, %d, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From d42a6412d158c581c76cadf7d4626b4235e60725 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 07:25:53 -0500
Subject: [PATCH 08/10] Handling a crash in logical "omp atomic
 compare-capture" inside "omp parallel". Adding some extra tests.

---
 flang/lib/Semantics/check-omp-atomic.cpp      |  19 ++-
 .../Integration/OpenMP/atomic-compare.f90     |  27 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 117 ++++++++++++++++++
 .../test/Semantics/OpenMP/atomic-compare.f90  |  12 ++
 4 files changed, 174 insertions(+), 1 deletion(-)

diff --git a/flang/lib/Semantics/check-omp-atomic.cpp b/flang/lib/Semantics/check-omp-atomic.cpp
index ec307be469982..88997df26e4d0 100644
--- a/flang/lib/Semantics/check-omp-atomic.cpp
+++ b/flang/lib/Semantics/check-omp-atomic.cpp
@@ -419,6 +419,22 @@ static bool IsMaybeAtomicWrite(const evaluate::Assignment &assign) {
   return HasStorageOverlap(assign.lhs, assign.rhs) == nullptr;
 }
 
+// Determine whether two designators refer to the same atomic variable.
+// The comparison tolerates OpenMP host-association: a shared variable
+// referenced inside a parallel region may be designated through a
+// host-associated symbol in one statement and through its ultimate symbol
+// in another, even though both denote the same storage. Comparing the
+// (ultimate) symbol vectors together with the source form distinguishes
+// different variables and different constant subscripts (e.g. a(1) vs a(2))
+// while treating host-associated and ultimate references as equal.
+static bool IsSameAtomicVariable(const SomeExpr &x, const SomeExpr &y) {
+  if (x == y) {
+    return true;
+  }
+  return evaluate::GetSymbolVector(x) == evaluate::GetSymbolVector(y) &&
+      x.AsFortran() == y.AsFortran();
+}
+
 static void SetExpr(parser::TypedExpr &expr, MaybeExpr value) {
   if (value) {
     expr.Reset(new evaluate::GenericExprWrapper(std::move(value)),
@@ -822,7 +838,8 @@ void OmpStructureChecker::CheckAtomicCaptureAssignment(
     CheckAtomicVariable(
         atom, rsrc, /*checkTypeOnPointer=*/!IsPointerAssignment(capture));
     // This part should have been checked prior to calling this function.
-    assert(*GetConvertInput(capture.rhs) == atom &&
+    assert(GetConvertInput(capture.rhs) &&
+        IsSameAtomicVariable(*GetConvertInput(capture.rhs), atom) &&
         "This cannot be a capture assignment");
     CheckStorageOverlap(atom, {cap}, source);
   }
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index f30349c42e33e..e98b25e4b4539 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -394,3 +394,30 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
   !$omp end atomic
 end
 
+! Logical compare+capture inside a parallel region.
+!CHECK-LABEL: define void @atomic_compare_capture_logical_parallel_(
+!CHECK-SAME: ptr noalias %[[X:.*]])
+!CHECK: %[[CGEP:.*]] = getelementptr { ptr }, ptr %structArg, i32 0, i32 0
+!CHECK: store ptr %[[X]], ptr %[[CGEP]]
+!CHECK: call void {{.*}}@__kmpc_fork_call{{.*}}@atomic_compare_capture_logical_parallel_..omp_par
+!CHECK-LABEL: define internal void @atomic_compare_capture_logical_parallel_..omp_par(
+!CHECK-SAME: ptr noalias %{{.*}}, ptr noalias %{{.*}}, ptr %[[STRUCTARG:.*]])
+!CHECK: %[[GEP:.*]] = getelementptr { ptr }, ptr %[[STRUCTARG]], i32 0, i32 0
+!CHECK: %[[SHARED:.*]] = load ptr, ptr %[[GEP]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[SHARED]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %{{.*}}
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 1219d05296e4d..ccc44a370be79 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -305,6 +305,83 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Compare-capture with an explicit seq_cst memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
+! CHECK:         omp.atomic.capture memory_order(seq_cst) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_seq_cst(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture seq_cst
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acquire memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acquire(
+! CHECK:         omp.atomic.capture memory_order(acquire) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acquire(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acquire
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit release memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_release(
+! CHECK:         omp.atomic.capture memory_order(release) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_release(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture release
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acq_rel memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acq_rel(
+! CHECK:         omp.atomic.capture memory_order(acq_rel) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acq_rel(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acq_rel
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
@@ -335,3 +412,43 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
     v = x
   !$omp end atomic
 end
+
+! Logical compare-capture inside a parallel region. The shared atom 'x' is
+! host-associated, and the capture reads it while the conditional update writes
+! it. Verify the atom is converted to i32 once (above the atomic region) and
+! that the capture reads then compares the same converted reference.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical_parallel(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"})
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}Ex"
+! CHECK:         omp.parallel private(
+! CHECK:           %[[OLD_DECL:.*]]:2 = hlfir.declare %{{.*}}Eold_value"
+! CHECK:           %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Eexpected"
+! CHECK:           %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:           %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:           omp.atomic.capture memory_order(relaxed) {
+! CHECK:             omp.atomic.read %[[ATOMVAL:.*]] = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:             omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:             ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:               %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:               %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:               omp.yield(%[[SEL]] : i32)
+! CHECK:             }
+! CHECK:           }
+! CHECK:           %[[RES:.*]] = fir.load %[[ATOMVAL]] : !fir.ref<i32>
+! CHECK:           %[[RESCONV:.*]] = fir.convert %[[RES]] : (i32) -> !fir.logical<4>
+! CHECK:           fir.store %[[RESCONV]] to %[[OLD_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           omp.terminator
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
diff --git a/flang/test/Semantics/OpenMP/atomic-compare.f90 b/flang/test/Semantics/OpenMP/atomic-compare.f90
index 0e53729c2f02a..5cee2359f52be 100644
--- a/flang/test/Semantics/OpenMP/atomic-compare.f90
+++ b/flang/test/Semantics/OpenMP/atomic-compare.f90
@@ -9,6 +9,7 @@
 
   real a, b, c
   logical :: r, s
+  logical :: lx, le, ld, lo
   a = 1.0
   b = 2.0
   c = 3.0
@@ -58,6 +59,17 @@
   if (r) b = c
   !$omp end atomic
 
+  ! Compare-capture with a LOGICAL atom using .eqv. inside a parallel region.
+  ! Regression: the host-associated atom (from the update) must be recognized
+  ! as the same variable as the captured one, even though OpenMP resolves them
+  ! through different (but ultimately identical) symbols.
+  !$omp atomic compare capture
+  lo = lx
+  if (lx .eqv. le) then
+    lx = ld
+  end if
+  !$omp end atomic
+
   ! Check for error conditions:
   !ERROR: At most one SEQ_CST clause can appear on the ATOMIC directive
   !$omp atomic seq_cst seq_cst compare

>From b6f101ded4b5847a466a27c6a37185a3203b6ebc Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 10 Jul 2026 00:50:42 -0500
Subject: [PATCH 09/10] Added exhaustive tests covering all atomic compare
 capture combinations Handling changes for atomic compare capture were not
 covered earlier. Completed runs of llvm-test-suite and Fujitsu flang
 test-suite.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  14 +
 flang/test/Fir/atomic-compare-capture.fir     | 343 ++++++++++++++
 .../OpenMP/atomic-compare-capture.f90         | 443 ++++++++++++++++++
 .../Todo/atomic-compare-capture-result.f90    |  20 +
 .../Lower/OpenMP/Todo/atomic-compare-fail.f90 |   8 -
 .../Lower/OpenMP/atomic-compare-capture.f90   | 356 ++++++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 116 +++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 398 ++++++++++++----
 .../LLVMIR/openmp-atomic-compare-capture.mlir | 407 ++++++++++++++++
 9 files changed, 2007 insertions(+), 98 deletions(-)
 create mode 100644 flang/test/Fir/atomic-compare-capture.fir
 create mode 100644 flang/test/Integration/OpenMP/atomic-compare-capture.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
 create mode 100644 flang/test/Lower/OpenMP/atomic-compare-capture.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 3811ea3d40bbd..ca37eb4a73375 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -606,6 +606,20 @@ void Fortran::lower::omp::lowerAtomic(
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
     }
 
+    // The comparison-result forms of atomic compare
+    //     (e.g. `r = x == e;
+    //           if (r) x = d`)
+    // are accepted by semantics, but the assignment to `r` is
+    // not captured by the atomic analysis and would be silently dropped.
+    // Detect the extra assignment here and emit an explicit TODO instead of
+    // generating incorrect code.
+    if (!construct.IsCapture()) {
+      const parser::Block &body = std::get<parser::Block>(construct.t);
+      if (body.size() > 1)
+        TODO(loc, "atomic compare capturing the comparison result "
+                  "(e.g. 'r = x == e')");
+    }
+
     common::RelationalOperator relOpr = common::RelationalOperator::EQ;
     std::optional<semantics::SomeExpr> expectedExprStorage;
     bool isUnsigned = false;
diff --git a/flang/test/Fir/atomic-compare-capture.fir b/flang/test/Fir/atomic-compare-capture.fir
new file mode 100644
index 0000000000000..65264fbc688a8
--- /dev/null
+++ b/flang/test/Fir/atomic-compare-capture.fir
@@ -0,0 +1,343 @@
+// RUN: fir-opt --split-input-file --cfg-conversion --fir-to-llvm-ir="target=aarch64-unknown-linux-gnu" %s | FileCheck %s
+
+// Test FIR-to-LLVM-dialect conversion of OpenMP atomic compare-capture: the
+// !fir.ref operands become !llvm.ptr and the compare region's arith ops become
+// their llvm-dialect equivalents, while the omp.atomic.capture / .compare /
+// .read structure is preserved for the later MLIR-to-LLVM-IR translation.
+
+//
+// Integer postfix compare-capture: {compare, read}
+func.func @cc_int_postfix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> i32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> i32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: i32):
+// CHECK:        %[[CMP:.*]] = llvm.icmp "eq" %[[XVAL]], %[[EVAL]] : i32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, i32
+// CHECK:        omp.yield(%[[SEL]] : i32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Integer prefix compare-capture: {read, compare}
+func.func @cc_int_prefix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:        llvm.select
+// CHECK:        omp.yield
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Integer fail-only compare-capture: {compare, read} with fail_only attribute.
+func.func @cc_int_failonly(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Real postfix compare-capture: !fir.ref<f32> -> !llvm.ptr, arith.cmpf -> llvm.fcmp.
+func.func @cc_real_postfix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> f32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> f32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: f32):
+// CHECK:        %[[CMP:.*]] = llvm.fcmp "oeq" %[[XVAL]], %[[EVAL]] : f32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, f32
+// CHECK:        omp.yield(%[[SEL]] : f32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Real prefix compare-capture.
+func.func @cc_real_prefix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Real fail-only compare-capture.
+func.func @cc_real_failonly(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Complex postfix compare-capture: !fir.ref<complex<f32>> -> !llvm.ptr and the
+// fir.cmpc is decomposed into per-field llvm.fcmp + llvm.and over the struct.
+func.func @cc_complex_postfix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: !llvm.struct<(f32, f32)>):
+// CHECK:        llvm.extractvalue %[[XVAL]][0]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.extractvalue %[[XVAL]][1]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        %[[AND:.*]] = llvm.and
+// CHECK:        %[[SEL:.*]] = llvm.select %[[AND]], %[[DVAL]], %[[XVAL]] : i1, !llvm.struct<(f32, f32)>
+// CHECK:        omp.yield(%[[SEL]] : !llvm.struct<(f32, f32)>)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    }
+
+// -----
+
+// Complex prefix compare-capture.
+func.func @cc_complex_prefix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Complex fail-only compare-capture.
+func.func @cc_complex_failonly(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    } {fail_only}
+
+// -----
+
+// Integer min compare-capture (x > e): the compare region uses arith.cmpi sgt.
+func.func @cc_int_min(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi sgt, %xval, %eval : i32
+      %s = arith.select %c, %eval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "sgt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Real min compare-capture (x > e): the compare region uses arith.cmpf ogt.
+func.func @cc_real_min(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf ogt, %xval, %eval : f32
+      %s = arith.select %c, %eval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "ogt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Weak clause: preserved through FIR-to-LLVM conversion.
+func.func @cc_weak(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    } {weak}
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_weak(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      } {weak}
+// CHECK:    }
diff --git a/flang/test/Integration/OpenMP/atomic-compare-capture.f90 b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..fe97e7b12558b
--- /dev/null
+++ b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,443 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=FIR
+!RUN: %if x86-registered-target %{ %flang_fc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+!RUN: %if aarch64-registered-target %{ %flang_fc1 -triple aarch64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+
+! OpenMP atomic compare-capture tracked through every stage of the compiler
+! (HLFIR, FIR/omp dialect, LLVM IR) for every operand type: integer, real,
+! logical and complex under ==, plus integer and real under min/max (<, >).
+! Equality lowers to a cmpxchg; min/max lowers to an atomicrmw.
+
+! ===========================================================================
+! integer ==  (postfix): cmpxchg + select(new value) + store into v
+! HLFIR-LABEL: func.func @_QPcc_integer(
+! HLFIR:         %[[X:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           }
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real ==  (postfix): equality on floats uses the HandleFPNegZero cmpxchg path.
+! HLFIR-LABEL: func.func @_QPcc_real(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf oeq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf oeq
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! logical ==  (postfix): the logical atom is converted to i32, so the omp op
+! operates on !fir.ref<i32> and lowers to an i32 cmpxchg.
+! HLFIR-LABEL: func.func @_QPcc_logical(
+! HLFIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! HLFIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_logical(
+! FIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_(
+! LLVM:         cmpxchg ptr %{{.*}}, i32
+! LLVM:         store i32 %{{.*}}, ptr
+subroutine cc_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! complex ==  (postfix): fir.cmpc region; lowers to an i64 (bitcast) cmpxchg
+! and the captured new value is a { float, float } struct.
+! HLFIR-LABEL: func.func @_QPcc_complex(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:           fir.cmpc "oeq"
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_complex(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<complex<f32>> {
+! FIR:           fir.cmpc "oeq"
+! FIR:         }
+! LLVM-LABEL: define void @cc_complex_(
+! LLVM:         cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer min (x > e, postfix): atomicrmw min, v gets the new value smin(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi sgt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi sgt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer max (x < e, postfix): atomicrmw max, v gets the new value smax(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi slt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi slt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real min (x > e, postfix): atomicrmw fmin, v gets minnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf ogt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf ogt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real max (x < e, postfix): atomicrmw fmax, v gets maxnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf olt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf olt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! The remaining cases exercise the different *capture-statement* forms allowed
+! for atomic compare-capture (all with an integer == atom).
+
+! ===========================================================================
+! Prefix form  {v = x; cond-update-stmt}:  the read precedes the update, so v
+! captures the OLD value of x - the read is emitted first and v is a direct
+! store of the extracted old value (no select).
+! HLFIR-LABEL: func.func @_QPcc_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_prefix(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:         }
+! LLVM-LABEL: define void @cc_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Fail-only form  {if(x == e){x = d;} else {v = x;}}:  v is captured only when
+! the compare fails; the capture carries the fail_only attribute and lowers to
+! a conditional store of the old value.
+! HLFIR-LABEL: func.func @_QPcc_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_failonly(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %[[EXIT:.*]], label %[[CONT:.*]]
+! LLVM:         [[CONT]]:
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Reversed comparison operand order  {if(e == x){x = d;} ; v = x}:  the front
+! end normalises 'e == x' to the same compare-capture as 'x == e', so this
+! accepts the alternate spelling and lowers identically (cmpxchg + select).
+! HLFIR-LABEL: func.func @_QPcc_eq_reversed(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_eq_reversed(
+! LLVM-LABEL: define void @cc_eq_reversed_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! The prefix and fail-only forms for the remaining operand types (real,
+! logical, complex), to confirm each type honours every capture form.
+
+! real, prefix: v gets the old value of x (float).
+! HLFIR-LABEL: func.func @_QPcc_real_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! LLVM-LABEL: define void @cc_real_prefix_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! real, fail-only: conditional store of the old value (float).
+! HLFIR-LABEL: func.func @_QPcc_real_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_real_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_real_failonly_(
+! LLVM:         cmpxchg ptr
+! LLVM:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! logical, prefix: v gets the old value (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR-LABEL: func.func @_QPcc_logical_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end
+
+! logical, fail-only: conditional store (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_failonly(
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_logical_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_logical_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %{{.*}}, label %{{.*}}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! complex, prefix: v gets the old complex value (struct), no select.
+! HLFIR-LABEL: func.func @_QPcc_complex_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! FIR-LABEL: func.func @_QPcc_complex_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! LLVM-LABEL: define void @cc_complex_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! complex, fail-only: conditional store of the old complex value.
+! HLFIR-LABEL: func.func @_QPcc_complex_failonly(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_complex_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_complex_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         %[[OK:.*]] = extractvalue { i64, i1 } %[[RES]], 1
+! LLVM:         %[[FAILED:.*]] = xor i1 %[[OK]], true
+! LLVM:         br i1 %[[FAILED]], label %{{.*}}, label %{{.*}}
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! weak clause (postfix): cmpxchg gets the 'weak' modifier.
+! HLFIR-LABEL: func.func @_QPcc_weak(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           } {weak}
+! HLFIR:           omp.atomic.read %{{.*}} = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_weak(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           } {weak}
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_weak_(
+! LLVM:         %[[RES:.*]] = cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_weak(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
new file mode 100644
index 0000000000000..ce6f0a3d6454b
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+! The comparison-result form of atomic compare captures the boolean outcome of
+! the comparison into `r`:
+!   r = x == e
+!   if (r) x = d
+! This is accepted by semantics, but the atomic analysis does not record the
+! assignment to `r`, so lowering would silently drop it and never write `r`.
+! Ensure it is diagnosed as "not yet implemented" instead of producing code that
+! leaves `r` unwritten.
+
+! CHECK: not yet implemented: atomic compare capturing the comparison result
+subroutine f(x, e, d, r)
+  integer :: x, e, d
+  logical :: r
+  !$omp atomic compare
+  r = x == e
+  if (r) x = d
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
index 27c7b1fb1a432..7165bdb2fc122 100644
--- a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
@@ -5,20 +5,12 @@ program p
   integer :: x
   integer :: r
   integer :: d
-  integer :: v
   !$omp atomic compare fail(relaxed)
   if (x .eq. 0) then
      x = 2
   end if
   !$omp end atomic
 
-  !$omp atomic compare capture
-  v = x
-  if (x > r) then
-     x = d
-  end if
-  !$omp end atomic
-
   !$omp atomic compare fail(relaxed)
   if (x > r) then
      x = d
diff --git a/flang/test/Lower/OpenMP/atomic-compare-capture.f90 b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..93fbf2d4bd034
--- /dev/null
+++ b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,356 @@
+! This test checks lowering of atomic compare capture constructs to HLFIR.
+! RUN: bbc %openmp_flags -fopenmp-version=51 -emit-hlfir %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir %openmp_flags -fopenmp-version=51 %s -o - | FileCheck %s
+
+! ---------------------------------------------------------------------------
+! integer == postfix: { if (x==e) x=d; v=x }
+! CHECK-LABEL: func.func @_QPcc_int_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == prefix: { v=x; if (x==e) x=d }
+! CHECK-LABEL: func.func @_QPcc_int_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:         }
+subroutine cc_int_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == fail-only: { if (x==e) x=d; else v=x }
+! CHECK-LABEL: func.func @_QPcc_int_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_int_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == reversed (e==x): same lowering as x==e
+! CHECK-LABEL: func.func @_QPcc_int_eq_reversed(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == postfix
+! CHECK-LABEL: func.func @_QPcc_real_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_postfix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == prefix
+! CHECK-LABEL: func.func @_QPcc_real_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == fail-only
+! CHECK-LABEL: func.func @_QPcc_real_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         } {fail_only}
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. postfix
+! CHECK-LABEL: func.func @_QPcc_logical_postfix(
+! CHECK:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_logical_postfix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. prefix
+! CHECK-LABEL: func.func @_QPcc_logical_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. fail-only
+! CHECK-LABEL: func.func @_QPcc_logical_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == postfix
+! CHECK-LABEL: func.func @_QPcc_complex_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine cc_complex_postfix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == prefix
+! CHECK-LABEL: func.func @_QPcc_complex_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:         }
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == fail-only
+! CHECK-LABEL: func.func @_QPcc_complex_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         } {fail_only}
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi sgt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi slt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf ogt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf olt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (prefix form)
+! CHECK-LABEL: func.func @_QPcc_weak_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:         }
+subroutine cc_weak_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (postfix form)
+! CHECK-LABEL: func.func @_QPcc_weak_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_weak_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ccc44a370be79..83e6f81ca00e3 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -218,6 +218,101 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
   !$omp end atomic
 end
 
+! Min/max (postfix) compare-capture: update first, capture the new value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_min_postfix(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! Min/max fail-only compare-capture: v is captured in the else branch (when the
+! comparison, and hence the update, does not happen).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_min_failonly(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) then
+      x = e
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
+! REAL min (postfix): if (x > e) x = e ; v = x -> compare region uses arith.cmpf.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_min(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %{{.*}}Ev"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! REAL max (postfix): if (x < e) x = e ; v = x.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_max(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %{{.*}} fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x < e) x = e
+    v = x
+  !$omp end atomic
+end
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
@@ -305,6 +400,27 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Complex compare-capture (postfix form): the compare region decomposes into
+! per-field comparisons combined with fir.cmpc/arith and the desired value is
+! selected as a whole complex value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_complex(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<complex<f32>> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: complex<f32>):
+! CHECK:             %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %{{.*}} : complex<f32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : complex<f32>
+! CHECK:             omp.yield(%[[SEL]] : complex<f32>)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine atomic_compare_capture_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
 ! Compare-capture with an explicit seq_cst memory order (prefix read form).
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
 ! CHECK:         omp.atomic.capture memory_order(seq_cst) {
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 8e49571cd2461..060e563ecefa8 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5370,6 +5370,99 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Result of matching the decomposed complex equality pattern inside an atomic
+/// compare region.
+struct ComplexComparePattern {
+  bool isComplex = false;
+  bool isNE = false;         // `or` of the field compares => NE (unsupported).
+  mlir::Value eAggregate;    // The complex expected value (`e`).
+  bool isXBinopExpr = false; // True if x is the first fcmp operand.
+};
+
+/// Detect a decomposed complex equality comparison in an atomic compare region:
+///   %re_x = llvm.extractvalue %xval[0]
+///   %re_e = llvm.extractvalue %eStruct[0]
+///   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
+///   %im_x = llvm.extractvalue %xval[1]
+///   %im_e = llvm.extractvalue %eStruct[1]
+///   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
+///   %cmp = llvm.and %cmp_re, %cmp_im   (llvm.or would be NE)
+/// It is recognised by an and/or whose operands are both fcmps operating on
+/// extractvalues, one chain rooted at the block argument (x) and the other at
+/// the expected complex value (e).
+static ComplexComparePattern detectComplexCompareEq(Block &block) {
+  ComplexComparePattern result;
+  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
+    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
+      return extractOp.getContainer();
+    return nullptr;
+  };
+  for (Operation &op : block.getOperations()) {
+    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
+      continue;
+    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
+    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
+    if (!lhsFcmp || !rhsFcmp)
+      continue;
+    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
+    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
+    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
+    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
+    if (!lhsXIsOp0 && !lhsXIsOp1)
+      continue;
+    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
+    if (!eAggregate)
+      continue;
+    result.isComplex = true;
+    result.isNE = isa<LLVM::OrOp>(op);
+    result.eAggregate = eAggregate;
+    result.isXBinopExpr = lhsXIsOp0;
+    break;
+  }
+  return result;
+}
+
+/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
+/// compare. The expected/desired complex values are spilled to allocas and
+/// reloaded as an integer of the same width, then compared and swapped. Returns
+/// the cmpxchg (result type `{iN, i1}`); `maxAlign` receives the alignment used
+/// so callers can reinterpret the captured old value through memory.
+static llvm::AtomicCmpXchgInst *
+emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
+                         llvm::Type *complexTy, llvm::Value *eVal,
+                         llvm::Value *dVal, llvm::AtomicOrdering atomicOrdering,
+                         bool isWeak, llvm::Align &maxAlign) {
+  const llvm::DataLayout &DL =
+      builder.GetInsertBlock()->getModule()->getDataLayout();
+  unsigned totalBits = DL.getTypeStoreSizeInBits(complexTy).getFixedValue();
+  llvm::IntegerType *intTy =
+      llvm::IntegerType::get(builder.getContext(), totalBits);
+  llvm::Align complexAlign = DL.getABITypeAlign(complexTy);
+  llvm::Align intAlign = DL.getABITypeAlign(intTy);
+  maxAlign = std::max(complexAlign, intAlign);
+
+  llvm::AllocaInst *eAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.e");
+  eAlloca->setAlignment(maxAlign);
+  llvm::AllocaInst *dAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.d");
+  dAlloca->setAlignment(maxAlign);
+
+  builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
+  llvm::Value *eInt =
+      builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
+  builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
+  llvm::Value *dInt =
+      builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
+
+  llvm::AtomicOrdering failOrdering =
+      llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
+  auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
+                                              atomicOrdering, failOrdering);
+  cmpXchg->setWeak(isWeak);
+  return cmpXchg;
+}
+
 /// Holds the extracted comparison pattern information from an atomic compare
 /// region.
 struct AtomicComparePatternInfo {
@@ -5379,7 +5472,6 @@ struct AtomicComparePatternInfo {
   bool isXBinopExpr = false;
   bool isSigned = false;
 };
-
 /// Extract comparison predicate, expected value (e), desired value (d), and
 /// related flags from an atomic compare region block by scanning for
 /// icmp/fcmp/select/min/max operations.
@@ -5387,6 +5479,27 @@ static LogicalResult extractAtomicComparePattern(
     Block &block,
     llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
     omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  // Complex equality is a decomposed per-field pattern (extractvalue + fcmp +
+  // and) rather than a single scalar compare. Detect it first so the scalar
+  // icmp/fcmp handling below does not mistake a real/imaginary field for the
+  // whole expected value.
+  if (ComplexComparePattern cplx = detectComplexCompareEq(block);
+      cplx.isComplex) {
+    if (cplx.isNE)
+      return atomicCompareOp.emitError(
+          "unsupported comparison predicate (NE) for complex atomic compare");
+    info.compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    info.isXBinopExpr = cplx.isXBinopExpr;
+    info.eVal = materializeValue(cplx.eAggregate);
+    for (Operation &op : block.getOperations()) {
+      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        info.dVal = materializeValue(selectOp.getTrueValue());
+        break;
+      }
+    }
+    return success();
+  }
+
   for (Operation &op : block.getOperations()) {
     // Pre-filter: skip icmps that don't involve the block argument
     // (e.g., truthiness extractions from logical-to-integer conversion).
@@ -5573,22 +5686,101 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
+    // Complex equality capture: x is struct-typed, which the OMPIRBuilder
+    // cannot handle, so emit a bitcast-to-integer cmpxchg (as in the
+    // non-capture complex path) and reconstruct the captured value from its
+    // result through memory. Complex only supports the == comparison.
+    if (llvmXElementType->isStructTy()) {
+      llvm::Align maxAlign;
+      llvm::AtomicCmpXchgInst *cmpXchg = emitComplexAtomicCmpXchg(
+          builder, llvmX, llvmXElementType, eVal, dVal, atomicOrdering,
+          atomicCompareOp.getWeak(), maxAlign);
+
+      llvm::Value *oldInt = builder.CreateExtractValue(cmpXchg, 0);
+      llvm::Value *cmpOk = builder.CreateExtractValue(cmpXchg, 1);
+
+      // Reinterpret the old integer value as the complex struct via memory.
+      llvm::AllocaInst *oldAlloca =
+          builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.old");
+      oldAlloca->setAlignment(maxAlign);
+      builder.CreateAlignedStore(oldInt, oldAlloca, maxAlign);
+      llvm::Value *oldComplex = builder.CreateAlignedLoad(
+          llvmXElementType, oldAlloca, maxAlign, "cmplx.old.val");
+
+      if (isFailOnly) {
+        // v is written only when the compare fails (cmpOk == false).
+        llvm::Value *cmpFailed = builder.CreateNot(cmpOk);
+        llvm::BasicBlock *curBB = builder.GetInsertBlock();
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(cmpFailed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else if (isPostfixCapture) {
+        // v gets the new value of x: d on success, old x otherwise.
+        llvm::Value *newComplex = builder.CreateSelect(cmpOk, dVal, oldComplex);
+        builder.CreateStore(newComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      } else {
+        // Prefix: v gets the old value of x.
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      }
+
+      // Emit flush after atomic compare if needed (release/acq_rel/seq_cst).
+      if (atomicOrdering == llvm::AtomicOrdering::Release ||
+          atomicOrdering == llvm::AtomicOrdering::AcquireRelease ||
+          atomicOrdering == llvm::AtomicOrdering::SequentiallyConsistent) {
+        llvm::OpenMPIRBuilder::LocationDescription flushLoc(builder);
+        ompBuilder->createFlush(flushLoc);
+      }
+      return success();
+    }
+
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw, so the fail-only capture
+    // form (v written only when the compare fails) has no valid mapping and is
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw (it asserts on IsFailOnly),
+    // so for min/max the fail-only capture is reconstructed manually below.
+    bool isMinMax = compareOp != llvm::omp::OMPAtomicCompareOp::EQ;
+
     llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
-    // Postfix: bypass V in OMPIRBuilder, handled manually below.
-    if (isPostfixCapture && !isFailOnly)
+    // Capture into V is reconstructed manually below for:
+    //   * postfix capture (v gets the new value of x): equality from the
+    //     cmpxchg result, min/max from the atomicrmw result;
+    //   * min/max fail-only capture (the OMPIRBuilder cannot express it).
+    // Bypass V in the OMPIRBuilder for those cases so it does not also emit its
+    // own (for min/max, incorrect or unsupported) capture store.
+    bool minMaxManualCapture = isMinMax && (isPostfixCapture || isFailOnly);
+    bool eqPostfixManualCapture = !isMinMax && isPostfixCapture && !isFailOnly;
+    if (minMaxManualCapture || eqPostfixManualCapture)
       llvmAtomicVForCall = {nullptr, nullptr, false, false};
 
-    // Prefix: IsPostfixUpdate=true → direct store of old value.
-    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
-    bool isPostfixUpdate = !isFailOnly;
+    // The OMPIRBuilder only understands IsFailOnly for the equality (cmpxchg)
+    // path; for min/max it would assert. Min/max fail-only is handled here.
+    bool builderFailOnly = isFailOnly && !isMinMax;
+
+    // IsPostfixUpdate selects which value the OMPIRBuilder captures into V:
+    //   * min/max prefix and equality prefix: a direct store of the old value
+    //     (IsPostfixUpdate=true).
+    //   * equality fail-only: a conditional store (IsPostfixUpdate=false).
+    // Manually-reconstructed captures bypass V above.
+    bool isPostfixUpdate = !builderFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly, isWeak);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicVForCall, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            builderFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
@@ -5596,8 +5788,99 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     builder.restoreIP(*afterIP);
 
-    // Postfix: v = select(success, D, old) — captures new value of x.
-    if (isPostfixCapture && !isFailOnly) {
+    // Min/max capture is reconstructed from the atomicrmw the OMPIRBuilder
+    // emits (its result is the old value of x). V was bypassed above.
+    //   * postfix: v gets the new value min/max(old, e);
+    //   * fail-only: v gets the old value, but only when the compare failed
+    //     (i.e. the atomicrmw did not change x).
+    // (Prefix min/max captures the old value directly through V, so nothing
+    // extra is needed there.)
+    if (isMinMax && (isPostfixCapture || isFailOnly)) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::AtomicRMWInst *rmw = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (auto *r = dyn_cast<llvm::AtomicRMWInst>(&inst)) {
+          rmw = r;
+          break;
+        }
+      }
+      assert(rmw && "expected atomicrmw for min/max compare capture");
+      llvm::Value *oldVal = rmw;
+      llvm::Value *rhs = rmw->getValOperand();
+
+      if (isFailOnly) {
+        // The compare "failed" (the else branch runs) exactly when the
+        // atomicrmw did not change x. Recompute the original update condition
+        // on the old value and negate it. v is stored only in that case.
+        llvm::CmpInst::Predicate updatePred;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          updatePred = llvm::CmpInst::ICMP_SGT;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          updatePred = llvm::CmpInst::ICMP_SLT;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          updatePred = llvm::CmpInst::ICMP_UGT;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          updatePred = llvm::CmpInst::ICMP_ULT;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          updatePred = llvm::CmpInst::FCMP_OGT;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          updatePred = llvm::CmpInst::FCMP_OLT;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *updated = builder.CreateCmp(updatePred, oldVal, rhs);
+        llvm::Value *failed = builder.CreateNot(updated);
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(failed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else {
+        llvm::Intrinsic::ID id;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          id = llvm::Intrinsic::smin;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          id = llvm::Intrinsic::smax;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          id = llvm::Intrinsic::umin;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          id = llvm::Intrinsic::umax;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          id = llvm::Intrinsic::minnum;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          id = llvm::Intrinsic::maxnum;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *newVal = builder.CreateBinaryIntrinsic(id, oldVal, rhs);
+        builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+      }
+    }
+
+    // Equality postfix: v = select(success, D, old) — reconstructs the new
+    // value of x from the cmpxchg result.
+    if (!isMinMax && isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
       llvm::Value *oldVal = nullptr;
       llvm::Value *successVal = nullptr;
@@ -5848,56 +6131,18 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
   llvm::Value *dVal = nullptr;
   bool isXBinopExpr = false;
 
-  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
-    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
-      return extractOp.getContainer();
-    return nullptr;
-  };
-
-  // Check for a decomposed complex comparison pattern:
-  //   %re_x = llvm.extractvalue %xval[0]
-  //   %re_e = llvm.extractvalue %eStruct[0]
-  //   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-  //   %im_x = llvm.extractvalue %xval[1]
-  //   %im_e = llvm.extractvalue %eStruct[1]
-  //   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
-  //   %cmp = llvm.and %cmp_re, %cmp_im   (for EQ)
-  // Detect this by looking for AndOp/OrOp whose operands are both FCmpOps
-  // operating on ExtractValueOps from the block argument.
-  bool isComplexPattern = false;
-  for (Operation &op : block.getOperations()) {
-    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
-      continue;
-
-    // Using : %cmp = llvm.and %cmp_re, %cmp_im
-    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
-    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
-    if (!lhsFcmp || !rhsFcmp)
-      continue;
-
-    // Using : %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-    // Check presence of x (block argument) and get e.
-    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
-    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
-    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
-    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
-    if (!lhsXIsOp0 && !lhsXIsOp1)
-      continue;
-    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
-    if (!eAggregate)
-      continue;
-
-    if (isa<LLVM::AndOp>(op))
-      compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    else
+  // Check for a decomposed complex comparison pattern (extractvalue + fcmp +
+  // and/or of the real/imaginary fields).
+  ComplexComparePattern cplx = detectComplexCompareEq(block);
+  bool isComplexPattern = cplx.isComplex;
+  if (isComplexPattern) {
+    if (cplx.isNE)
       // OrOp corresponds to NE, which is not a valid atomic compare op.
       return atomicCompareOp.emitError(
           "unsupported comparison predicate (NE) for complex atomic compare");
-
-    isXBinopExpr = lhsXIsOp0;
-    eVal = materializeValue(eAggregate);
-    isComplexPattern = true;
-    break;
+    compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    isXBinopExpr = cplx.isXBinopExpr;
+    eVal = materializeValue(cplx.eAggregate);
   }
 
   if (isComplexPattern) {
@@ -5916,37 +6161,10 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
       dVal = materializeValue(yieldOp.getResults()[0]);
     }
 
-    const llvm::DataLayout &DL =
-        builder.GetInsertBlock()->getModule()->getDataLayout();
-    unsigned totalBits =
-        DL.getTypeStoreSizeInBits(llvmXElementType).getFixedValue();
-
-    llvm::IntegerType *intTy =
-        llvm::IntegerType::get(builder.getContext(), totalBits);
-
-    llvm::Align complexAlign = DL.getABITypeAlign(llvmXElementType);
-    llvm::Align intAlign = DL.getABITypeAlign(intTy);
-    llvm::Align maxAlign = std::max(complexAlign, intAlign);
-
-    llvm::AllocaInst *eAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.e");
-    eAlloca->setAlignment(maxAlign);
-    llvm::AllocaInst *dAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.d");
-    dAlloca->setAlignment(maxAlign);
-
-    builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
-    llvm::Value *eInt =
-        builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
-    builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
-    llvm::Value *dInt =
-        builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
-
-    llvm::AtomicOrdering failOrdering =
-        llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
-    auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
-                                                atomicOrdering, failOrdering);
-    cmpXchg->setWeak(atomicCompareOp.getWeak());
+    llvm::Align maxAlign;
+    emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
+                             atomicOrdering, atomicCompareOp.getWeak(),
+                             maxAlign);
 
     // Emit flush after atomic compare if needed (for release, acq_rel,
     // seq_cst orderings).
diff --git a/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
new file mode 100644
index 0000000000000..2a3cb6dc99133
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
@@ -0,0 +1,407 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Prefix compare+capture: {read, compare}
+// CHECK-LABEL: define void @compare_capture_prefix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Postfix compare+capture: {compare, read}
+// CHECK-LABEL: define void @compare_capture_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+// CHECK:         store i32 %[[NEWVAL]], ptr %[[V]], align 4
+llvm.func @compare_capture_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Fail-only compare+capture: {compare, read} with fail_only
+// CHECK-LABEL: define void @compare_capture_failonly(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         br i1 %[[SUCCESS]], label %{{.*}}.atomic.exit, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store i32 %[[OLD]], ptr %[[V]]
+// CHECK:         br label %{{.*}}.atomic.exit
+// CHECK:       {{.*}}.atomic.exit:
+llvm.func @compare_capture_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Weak compare+capture: {read, compare} with weak
+// CHECK-LABEL: define void @compare_capture_weak(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg weak ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Float (real) tests =====
+// Float uses the HandleFPNegZero path with NaN/zero checks and bitcast.
+
+// Float prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_float_prefix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         store float %{{.*}}, ptr %[[V:.*]], align 4
+llvm.func @compare_capture_float_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+  }
+  llvm.return
+}
+
+// Float postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_float_postfix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         %[[NEW:.*]] = select i1 %[[OK]], float %{{.*}}, float %[[OLD_FP]]
+// CHECK:         store float %[[NEW]], ptr %{{.*}}, align 4
+llvm.func @compare_capture_float_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_float_failonly(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         br i1 %[[OK]], label %{{.*}}.atomic.exit{{.*}}, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store float %[[OLD_FP]], ptr %{{.*}}
+llvm.func @compare_capture_float_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Complex (struct<(f32, f32)>) tests =====
+// Complex uses bitcast to i64 for cmpxchg, with FP neg-zero handling.
+
+// Complex prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_complex_prefix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+  }
+  llvm.return
+}
+
+// Complex postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_complex_postfix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         select i1 %{{.*}}, { float, float } %{{.*}}, { float, float } %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  }
+  llvm.return
+}
+
+// Complex fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_complex_failonly(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Weak clause with float =====
+
+// Float weak prefix: cmpxchg weak with FP neg-zero handling
+// CHECK-LABEL: define void @compare_capture_float_weak(
+// CHECK:         cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+// CHECK:         store float %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_float_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Min/Max compare+capture =====
+
+// Integer min, postfix: atomicrmw min, v gets min(old, e)
+// CHECK-LABEL: define void @compare_capture_min_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]])
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %[[X]], i32 %[[E]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %[[V]]
+llvm.func @compare_capture_min_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer max, postfix: atomicrmw max, v gets max(old, e)
+// CHECK-LABEL: define void @compare_capture_max_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_max_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer min, prefix: v gets old value
+// CHECK-LABEL: define void @compare_capture_min_prefix(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %{{.*}} monotonic
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_prefix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Float min, postfix: atomicrmw fmin, v gets minnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmin_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmin_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "ogt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float max, postfix: atomicrmw fmax, v gets maxnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmax_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmax_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "olt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// ===== Min/Max fail-only =====
+
+// Integer min, fail-only: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_min_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp sgt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Integer max, fail-only
+// CHECK-LABEL: define void @compare_capture_max_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp slt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_max_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}

>From ea6c6151e5dce95ca2b26cf7f1d0f7a2cb70c421 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 21 Jul 2026 01:34:26 -0500
Subject: [PATCH 10/10] Using load, fcmp oeq & cmpxchg for complex numbers.
 Avoiding CAS loops similar to atomic compare.

---
 .../OpenMP/atomic-compare-capture.f90         |  10 +-
 .../Integration/OpenMP/atomic-compare.f90     |  41 ++++--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 124 ++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  17 ++-
 4 files changed, 129 insertions(+), 63 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare-capture.f90 b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
index fe97e7b12558b..4f56720b75a0c 100644
--- a/flang/test/Integration/OpenMP/atomic-compare-capture.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
@@ -102,6 +102,8 @@ subroutine cc_logical(x, e, d, v)
 ! FIR:           fir.cmpc "oeq"
 ! FIR:         }
 ! LLVM-LABEL: define void @cc_complex_(
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
 ! LLVM:         cmpxchg ptr %{{.*}}, i64
 ! LLVM:         store { float, float } %{{.*}}, ptr
 subroutine cc_complex(x, e, d, v)
@@ -379,6 +381,8 @@ subroutine cc_logical_failonly(x, e, d, v)
 ! FIR-LABEL: func.func @_QPcc_complex_prefix(
 ! FIR:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
 ! LLVM-LABEL: define void @cc_complex_prefix_(
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
 ! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
 ! LLVM:         store { float, float } %{{.*}}, ptr
 subroutine cc_complex_prefix(x, e, d, v)
@@ -396,8 +400,10 @@ subroutine cc_complex_prefix(x, e, d, v)
 ! FIR-LABEL: func.func @_QPcc_complex_failonly(
 ! FIR:         } {fail_only}
 ! LLVM-LABEL: define void @cc_complex_failonly_(
-! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
-! LLVM:         %[[OK:.*]] = extractvalue { i64, i1 } %[[RES]], 1
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+! LLVM:         cmpxchg ptr %{{.*}}, i64
+! LLVM:         %[[OK:.*]] = phi i1
 ! LLVM:         %[[FAILED:.*]] = xor i1 %[[OK]], true
 ! LLVM:         br i1 %[[FAILED]], label %{{.*}}, label %{{.*}}
 ! LLVM:         store { float, float } %{{.*}}, ptr
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index e98b25e4b4539..88b4bca9e9968 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -150,32 +150,45 @@ subroutine atomic_compare_real(x, e, d)
   if (x == e) x = d
 end
 
-! Complex(4) equality → type-punned i64 cmpxchg with consistent alignment
+! Complex(4) equality -> component-wise IEEE-754 fcmp, then a cmpxchg that swaps
+! using X's loaded bit-pattern as the comparand (so -0.0/+0.0 and NaN follow the
+! scalar float semantics rather than a raw bitwise compare).
 !CHECK-LABEL: define void @atomic_compare_complex4_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]])
-!CHECK: %[[EALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
-!CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN]]
-!CHECK: store { float, float } %{{.*}}, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: %[[EINT:.*]] = load i64, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: store { float, float } %{{.*}}, ptr %[[DALLOCA]], align [[ALIGN]]
+!CHECK: %[[EVAL:.*]] = load { float, float }, ptr %[[E]], align 4
+!CHECK: %[[DVAL:.*]] = load { float, float }, ptr %[[D]], align 4
+!CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
+!CHECK: store { float, float } %[[DVAL]], ptr %[[DALLOCA]], align [[ALIGN]]
 !CHECK: %[[DINT:.*]] = load i64, ptr %[[DALLOCA]], align [[ALIGN]]
-!CHECK: cmpxchg ptr %[[X]], i64 %[[EINT]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+!CHECK: %[[XLOAD:.*]] = load atomic i64, ptr %[[X]] monotonic, align [[ALIGN]]
+!CHECK: %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+!CHECK: %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+!CHECK: %[[EQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+!CHECK: br i1 %[[EQ]], label %[[SWAP:[^,]+]], label %{{.*}}
+!CHECK: [[SWAP]]:
+!CHECK: cmpxchg ptr %[[X]], i64 %[[XLOAD]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
 subroutine atomic_compare_complex4(x, e, d)
   complex :: x, e, d
   !$omp atomic compare
   if (x == e) x = d
 end
 
-! Complex(8) equality → type-punned i128 cmpxchg with consistent alignment
+! Complex(8) equality -> component-wise IEEE-754 fcmp, then an i128 cmpxchg using
+! X's loaded bit-pattern as the comparand.
 !CHECK-LABEL: define void @atomic_compare_complex8_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]])
-!CHECK: %[[EALLOCA:.*]] = alloca { double, double }, align [[ALIGN:[0-9]+]]
-!CHECK: %[[DALLOCA:.*]] = alloca { double, double }, align [[ALIGN]]
-!CHECK: store { double, double } %{{.*}}, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: %[[EINT:.*]] = load i128, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: store { double, double } %{{.*}}, ptr %[[DALLOCA]], align [[ALIGN]]
+!CHECK: %[[EVAL:.*]] = load { double, double }, ptr %[[E]], align 8
+!CHECK: %[[DVAL:.*]] = load { double, double }, ptr %[[D]], align 8
+!CHECK: %[[DALLOCA:.*]] = alloca { double, double }, align [[ALIGN:[0-9]+]]
+!CHECK: store { double, double } %[[DVAL]], ptr %[[DALLOCA]], align [[ALIGN]]
 !CHECK: %[[DINT:.*]] = load i128, ptr %[[DALLOCA]], align [[ALIGN]]
-!CHECK: cmpxchg ptr %[[X]], i128 %[[EINT]], i128 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+!CHECK: %[[XLOAD:.*]] = load atomic i128, ptr %[[X]] monotonic, align [[ALIGN]]
+!CHECK: %[[REEQ:.*]] = fcmp oeq double %[[REX:.*]], %[[REE:.*]]
+!CHECK: %[[IMEQ:.*]] = fcmp oeq double %[[IMX:.*]], %[[IME:.*]]
+!CHECK: %[[EQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+!CHECK: br i1 %[[EQ]], label %[[SWAP:[^,]+]], label %{{.*}}
+!CHECK: [[SWAP]]:
+!CHECK: cmpxchg ptr %[[X]], i128 %[[XLOAD]], i128 %[[DINT]] monotonic monotonic, align [[ALIGN]]
 subroutine atomic_compare_complex8(x, e, d)
   complex(8) :: x, e, d
   !$omp atomic compare
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 060e563ecefa8..fdc8f09fe5a2d 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5422,16 +5422,15 @@ static ComplexComparePattern detectComplexCompareEq(Block &block) {
   return result;
 }
 
-/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
-/// compare. The expected/desired complex values are spilled to allocas and
-/// reloaded as an integer of the same width, then compared and swapped. Returns
-/// the cmpxchg (result type `{iN, i1}`); `maxAlign` receives the alignment used
-/// so callers can reinterpret the captured old value through memory.
-static llvm::AtomicCmpXchgInst *
-emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
-                         llvm::Type *complexTy, llvm::Value *eVal,
-                         llvm::Value *dVal, llvm::AtomicOrdering atomicOrdering,
-                         bool isWeak, llvm::Align &maxAlign) {
+/// Emit an IEEE-754-correct `cmpxchg` for a complex (struct-typed) atomic
+/// compare with `fcmp oeq`. The old value of X is returned (as the complex struct
+/// type) in \p oldComplex and the success flag (i1) in \p cmpOk.
+static void emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder,
+                                     llvm::Value *llvmX, llvm::Type *complexTy,
+                                     llvm::Value *eVal, llvm::Value *dVal,
+                                     llvm::AtomicOrdering atomicOrdering,
+                                     bool isWeak, llvm::Value *&oldComplex,
+                                     llvm::Value *&cmpOk) {
   const llvm::DataLayout &DL =
       builder.GetInsertBlock()->getModule()->getDataLayout();
   unsigned totalBits = DL.getTypeStoreSizeInBits(complexTy).getFixedValue();
@@ -5439,28 +5438,76 @@ emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
       llvm::IntegerType::get(builder.getContext(), totalBits);
   llvm::Align complexAlign = DL.getABITypeAlign(complexTy);
   llvm::Align intAlign = DL.getABITypeAlign(intTy);
-  maxAlign = std::max(complexAlign, intAlign);
+  llvm::Align maxAlign = std::max(complexAlign, intAlign);
 
-  llvm::AllocaInst *eAlloca =
-      builder.CreateAlloca(complexTy, nullptr, "cmplx.e");
-  eAlloca->setAlignment(maxAlign);
+  // Spill D to obtain its integer bit pattern for the swap value.
   llvm::AllocaInst *dAlloca =
       builder.CreateAlloca(complexTy, nullptr, "cmplx.d");
   dAlloca->setAlignment(maxAlign);
-
-  builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
-  llvm::Value *eInt =
-      builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
   builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
   llvm::Value *dInt =
       builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
 
+  // Load the current value of X atomically and reinterpret it as complex.
+  llvm::LoadInst *xCurr =
+      builder.CreateAlignedLoad(intTy, llvmX, maxAlign, "cmplx.x.load");
+  xCurr->setAtomic(llvm::AtomicOrdering::Monotonic);
+  llvm::AllocaInst *xAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.x");
+  xAlloca->setAlignment(maxAlign);
+  builder.CreateAlignedStore(xCurr, xAlloca, maxAlign);
+  llvm::Value *xStruct =
+      builder.CreateAlignedLoad(complexTy, xAlloca, maxAlign, "cmplx.x.val");
+
+  // Component-wise IEEE-754 equality: `fcmp oeq` yields false for NaN (so a
+  // NaN component correctly makes the compare fail) and true for +0.0 vs -0.0
+  // (so a zero-sign difference does not spuriously fail the compare).
+  llvm::Value *reX = builder.CreateExtractValue(xStruct, 0);
+  llvm::Value *imX = builder.CreateExtractValue(xStruct, 1);
+  llvm::Value *reE = builder.CreateExtractValue(eVal, 0);
+  llvm::Value *imE = builder.CreateExtractValue(eVal, 1);
+  llvm::Value *reEq = builder.CreateFCmpOEQ(reX, reE, "cmplx.re.eq");
+  llvm::Value *imEq = builder.CreateFCmpOEQ(imX, imE, "cmplx.im.eq");
+  llvm::Value *fpEqual = builder.CreateAnd(reEq, imEq, "cmplx.eq");
+
+  // When the components compare equal, attempt the swap using X's own loaded
+  // bit pattern as the comparand; otherwise the compare fails and X is left
+  // unchanged (the captured old value is the value just loaded).
+  llvm::BasicBlock *curBB = builder.GetInsertBlock();
+  llvm::Function *fn = curBB->getParent();
+  llvm::BasicBlock *swapBB =
+      llvm::BasicBlock::Create(builder.getContext(), "cmplx.atomic.swap", fn);
+  llvm::BasicBlock *exitBB =
+      llvm::BasicBlock::Create(builder.getContext(), "cmplx.atomic.exit", fn);
+  builder.CreateCondBr(fpEqual, swapBB, exitBB);
+
+  builder.SetInsertPoint(swapBB);
   llvm::AtomicOrdering failOrdering =
       llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
-  auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
-                                              atomicOrdering, failOrdering);
+  llvm::AtomicCmpXchgInst *cmpXchg = builder.CreateAtomicCmpXchg(
+      llvmX, xCurr, dInt, maxAlign, atomicOrdering, failOrdering);
   cmpXchg->setWeak(isWeak);
-  return cmpXchg;
+  llvm::Value *oldSwap = builder.CreateExtractValue(cmpXchg, 0);
+  llvm::Value *okSwap = builder.CreateExtractValue(cmpXchg, 1);
+  builder.CreateBr(exitBB);
+
+  // Merge the swap and no-swap paths.
+  builder.SetInsertPoint(exitBB);
+  llvm::PHINode *oldIntPHI = builder.CreatePHI(intTy, 2, "cmplx.old.int");
+  oldIntPHI->addIncoming(oldSwap, swapBB);
+  oldIntPHI->addIncoming(xCurr, curBB);
+  llvm::PHINode *okPHI = builder.CreatePHI(builder.getInt1Ty(), 2, "cmplx.ok");
+  okPHI->addIncoming(okSwap, swapBB);
+  okPHI->addIncoming(builder.getFalse(), curBB);
+
+  // Reinterpret the old integer value as the complex struct via memory.
+  llvm::AllocaInst *oldAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.old");
+  oldAlloca->setAlignment(maxAlign);
+  builder.CreateAlignedStore(oldIntPHI, oldAlloca, maxAlign);
+  oldComplex = builder.CreateAlignedLoad(complexTy, oldAlloca, maxAlign,
+                                         "cmplx.old.val");
+  cmpOk = okPHI;
 }
 
 /// Holds the extracted comparison pattern information from an atomic compare
@@ -5687,25 +5734,17 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
     // Complex equality capture: x is struct-typed, which the OMPIRBuilder
-    // cannot handle, so emit a bitcast-to-integer cmpxchg (as in the
-    // non-capture complex path) and reconstruct the captured value from its
-    // result through memory. Complex only supports the == comparison.
+    // cannot handle, so emit an IEEE-754-correct cmpxchg (as in the non-capture
+    // complex path) and reconstruct the captured value from its result. The
+    // helper compares components with `fcmp oeq` so `-0.0 == +0.0` and `NaN`
+    // are handled as in the scalar float path. Complex only supports the ==
+    // comparison.
     if (llvmXElementType->isStructTy()) {
-      llvm::Align maxAlign;
-      llvm::AtomicCmpXchgInst *cmpXchg = emitComplexAtomicCmpXchg(
-          builder, llvmX, llvmXElementType, eVal, dVal, atomicOrdering,
-          atomicCompareOp.getWeak(), maxAlign);
-
-      llvm::Value *oldInt = builder.CreateExtractValue(cmpXchg, 0);
-      llvm::Value *cmpOk = builder.CreateExtractValue(cmpXchg, 1);
-
-      // Reinterpret the old integer value as the complex struct via memory.
-      llvm::AllocaInst *oldAlloca =
-          builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.old");
-      oldAlloca->setAlignment(maxAlign);
-      builder.CreateAlignedStore(oldInt, oldAlloca, maxAlign);
-      llvm::Value *oldComplex = builder.CreateAlignedLoad(
-          llvmXElementType, oldAlloca, maxAlign, "cmplx.old.val");
+      llvm::Value *oldComplex = nullptr;
+      llvm::Value *cmpOk = nullptr;
+      emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
+                               atomicOrdering, atomicCompareOp.getWeak(),
+                               oldComplex, cmpOk);
 
       if (isFailOnly) {
         // v is written only when the compare fails (cmpOk == false).
@@ -6161,10 +6200,13 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
       dVal = materializeValue(yieldOp.getResults()[0]);
     }
 
-    llvm::Align maxAlign;
+    llvm::Value *oldComplex = nullptr;
+    llvm::Value *cmpOk = nullptr;
     emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
                              atomicOrdering, atomicCompareOp.getWeak(),
-                             maxAlign);
+                             oldComplex, cmpOk);
+    (void)oldComplex;
+    (void)cmpOk;
 
     // Emit flush after atomic compare if needed (for release, acq_rel,
     // seq_cst orderings).
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index b779d20f42d0c..63054056adadc 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2619,14 +2619,19 @@ llvm.func @omp_atomic_compare(
     omp.yield(%sel1 : f32)
   }
 
-  // Complex equality  →  bitcasted integer cmpxchg with consistent alignment
-  // CHECK: %[[EALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
-  // CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN]]
-  // CHECK: store { float, float } %[[EC]], ptr %[[EALLOCA]], align [[ALIGN]]
-  // CHECK: %[[EINT:.*]] = load i64, ptr %[[EALLOCA]], align [[ALIGN]]
+  // Complex equality  →  component-wise IEEE-754 fcmp, then a cmpxchg that
+  // swaps using x's loaded bit-pattern as the comparand (so -0.0/+0.0 and NaN
+  // follow the scalar float semantics rather than a raw bitwise compare).
+  // CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
   // CHECK: store { float, float } %[[DC]], ptr %[[DALLOCA]], align [[ALIGN]]
   // CHECK: %[[DINT:.*]] = load i64, ptr %[[DALLOCA]], align [[ALIGN]]
-  // CHECK: cmpxchg ptr %[[XC]], i64 %[[EINT]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+  // CHECK: %[[XLOAD:.*]] = load atomic i64, ptr %[[XC]] monotonic, align [[ALIGN]]
+  // CHECK: %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+  // CHECK: %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+  // CHECK: %[[CEQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+  // CHECK: br i1 %[[CEQ]], label %[[CSWAP:[^,]+]], label %{{.*}}
+  // CHECK: [[CSWAP]]:
+  // CHECK: cmpxchg ptr %[[XC]], i64 %[[XLOAD]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
   omp.atomic.compare %xc : !llvm.ptr {
   ^bb0(%xval : !llvm.struct<(f32, f32)>):
     %re_x = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>


From llvmlistbot at llvm.org  Tue Jul 21 00:06:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 00:06:47 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Support for
 "atomic compare capture" (PR #202315)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202315@github.com>
Message-ID: <6a5f1a87.dd74445a.13f06.e885@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/202315

>From 216cbd2e710ce3c725bb51c5606bea4ba84a33a3 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Sat, 6 Jun 2026 14:38:54 -0500
Subject: [PATCH 01/11] Support for "atomic compare capture".

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  88 ++++---
 .../Interfaces/AtomicInterfaces.td            |   9 +-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  10 +
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |   6 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 216 ++++++++++++++----
 5 files changed, 250 insertions(+), 79 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 9d711b92bb520..5e02d3fb10dbb 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -558,16 +558,45 @@ void Fortran::lower::omp::lowerAtomic(
   int action1 = analysis.op1.what & analysis.Action;
   memOrder = makeValidForAction(memOrder, action0, action1, version);
 
+  // --- Shared capture scaffolding ---
+  mlir::Operation *captureOp = nullptr;
+  fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
+  fir::FirOpBuilder::InsertPoint atomicAt, postAt;
+
+  if (construct.IsCapture()) {
+    assert(action0 != analysis.None && action1 != analysis.None &&
+           "Expexcing two actions");
+    (void)action0;
+    (void)action1;
+    captureOp = mlir::omp::AtomicCaptureOp::create(
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+    // Set the non-atomic insertion point to before the atomic.capture.
+    preAt = getInsertionPointBefore(captureOp);
+
+    mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
+    builder.setInsertionPointToEnd(block);
+    // Set the atomic insertion point to before the terminator inside
+    // atomic.capture.
+    mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
+    atomicAt = getInsertionPointBefore(term);
+    postAt = getInsertionPointAfter(captureOp);
+    hint = nullptr;
+    memOrder = std::nullopt;
+  }
+
   if (auto *cond = get(analysis.cond)) {
     // atomic compare: if (x == e) x = d
     // e : expecteVal
     // d : desiredVal
 
-    // Check for compound clauses (fail, capture, weak) that are not yet
+    // Restore insertion point so pre-processing code (e.g. computing
+    // expectedVal) is emitted before the capture op, not after the terminator.
+    builder.restoreInsertionPoint(preAt);
+
+    // Check for compound clauses (fail, weak) that are not yet
     // supported with atomic compare.
     if (llvm::any_of(clauses, [](const omp::Clause &clause) {
           return clause.id == llvm::omp::Clause::OMPC_fail ||
-                 clause.id == llvm::omp::Clause::OMPC_capture ||
                  clause.id == llvm::omp::Clause::OMPC_weak;
         })) {
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
@@ -616,6 +645,17 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // If this is a compare+capture, generate the read op first.
+    if (construct.IsCapture()) {
+      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
+             "Expected a read operation for compare capture");
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
+
     mlir::UnitAttr weakAttr = nullptr;
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
@@ -679,34 +719,9 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
-
     // END omp atomic compare
   } else {
-    mlir::Operation *captureOp = nullptr;
-    fir::FirOpBuilder::InsertPoint preAt = builder.saveInsertionPoint();
-    fir::FirOpBuilder::InsertPoint atomicAt, postAt;
-
-    if (construct.IsCapture()) {
-      // Capturing operation.
-      assert(action0 != analysis.None && action1 != analysis.None &&
-             "Expexcing two actions");
-      (void)action0;
-      (void)action1;
-      captureOp = mlir::omp::AtomicCaptureOp::create(
-          builder, loc, hint, makeMemOrderAttr(converter, memOrder));
-      // Set the non-atomic insertion point to before the atomic.capture.
-      preAt = getInsertionPointBefore(captureOp);
-
-      mlir::Block *block = builder.createBlock(&captureOp->getRegion(0));
-      builder.setInsertionPointToEnd(block);
-      // Set the atomic insertion point to before the terminator inside
-      // atomic.capture.
-      mlir::Operation *term = mlir::omp::TerminatorOp::create(builder, loc);
-      atomicAt = getInsertionPointBefore(term);
-      postAt = getInsertionPointAfter(captureOp);
-      hint = nullptr;
-      memOrder = std::nullopt;
-    } else {
+    if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
              "Expexcing single action");
@@ -729,16 +744,13 @@ void Fortran::lower::omp::lowerAtomic(
           *get(analysis.op1.assign), hint, memOrder, preAt, atomicAt, postAt);
     }
 
-    if (construct.IsCapture()) {
-      // If this is a capture operation, the first/second ops will be inside
-      // of it. Set the insertion point to past the capture op itself.
-      builder.restoreInsertionPoint(postAt);
-    } else {
-      if (secondOp) {
-        builder.setInsertionPointAfter(secondOp);
-      } else {
-        builder.setInsertionPointAfter(firstOp);
-      }
+    if (!construct.IsCapture()) {
+      builder.setInsertionPointAfter(secondOp ? secondOp : firstOp);
     }
   }
+
+  // Shared capture cleanup.
+  if (construct.IsCapture()) {
+    builder.restoreInsertionPoint(postAt);
+  }
 }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index abb21705b3c1c..8c9015f05bb72 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -289,10 +289,12 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
+        auto secondCompareStmt = dyn_cast<AtomicCompareOpInterface>(secondOp);
 
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
-              (firstReadStmt && secondWriteStmt)))
+              (firstReadStmt && secondWriteStmt) ||
+              (firstReadStmt && secondCompareStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -310,6 +312,11 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
           return firstReadStmt.emitError()
                 << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstReadStmt && secondCompareStmt &&
+            firstReadStmt.getX() != secondCompareStmt.getX())
+          return firstReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "second operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0962b330e2f23..1241abc10298f 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1928,6 +1928,12 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
         omp.atomic.write ...
         omp.terminator
       }
+
+      omp.atomic.capture {
+        omp.atomic.read ...
+        omp.atomic.compare ...
+        omp.terminator
+      }
     ```
   }] # clausesDescription;
 
@@ -1946,6 +1952,10 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     /// Returns the `atomic.update` operation inside the region, if any.
     /// Otherwise, it returns nullptr.
     AtomicUpdateOp getAtomicUpdateOp();
+
+    /// Returns the `atomic.compare` operation inside the region, if any.
+    /// Otherwise, it returns nullptr.
+    AtomicCompareOp getAtomicCompareOp();
   }] # clausesExtraClassDeclaration;
 
   let hasRegionVerifier = 1;
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..0eafd0a267b97 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4615,6 +4615,12 @@ AtomicUpdateOp AtomicCaptureOp::getAtomicUpdateOp() {
   return dyn_cast<AtomicUpdateOp>(getSecondOp());
 }
 
+AtomicCompareOp AtomicCaptureOp::getAtomicCompareOp() {
+  if (auto op = dyn_cast<AtomicCompareOp>(getFirstOp()))
+    return op;
+  return dyn_cast<AtomicCompareOp>(getSecondOp());
+}
+
 LogicalResult AtomicCaptureOp::verify() {
   return verifySynchronizationHint(*this, getHint());
 }
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 07fa92bdabe50..4928b288adb65 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4795,6 +4795,43 @@ convertOmpAtomicUpdate(omp::AtomicUpdateOp &opInst,
   return success();
 }
 
+/// Helper to extract the OMPAtomicCompareOp from an integer comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::ICmpPredicate::eq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::ICmpPredicate::slt:
+  case LLVM::ICmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::ICmpPredicate::sgt:
+  case LLVM::ICmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
+/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
+/// predicate. Returns std::nullopt for unsupported predicates.
+static std::optional<llvm::omp::OMPAtomicCompareOp>
+convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
+  switch (predicate) {
+  case LLVM::FCmpPredicate::oeq:
+  case LLVM::FCmpPredicate::ueq:
+    return llvm::omp::OMPAtomicCompareOp::EQ;
+  case LLVM::FCmpPredicate::olt:
+  case LLVM::FCmpPredicate::ult:
+    return llvm::omp::OMPAtomicCompareOp::MIN;
+  case LLVM::FCmpPredicate::ogt:
+  case LLVM::FCmpPredicate::ugt:
+    return llvm::omp::OMPAtomicCompareOp::MAX;
+  default:
+    return std::nullopt;
+  }
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -4803,13 +4840,149 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   if (failed(checkImplementationStatus(*atomicCaptureOp)))
     return failure();
 
+  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
+  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
+  omp::AtomicCompareOp atomicCompareOp = atomicCaptureOp.getAtomicCompareOp();
+
+  // If the capture contains an atomic.compare, delegate to
+  // createAtomicCompare with the capture variable (V) set.
+  if (atomicCompareOp) {
+    omp::AtomicReadOp atomicReadOp = atomicCaptureOp.getAtomicReadOp();
+    assert(atomicReadOp && "expected atomic.read in capture+compare");
+
+    Region &region = atomicCompareOp.getRegion();
+    Block &block = region.front();
+
+    llvm::Type *llvmXElementType =
+        moduleTranslation.convertType(block.getArgument(0).getType());
+    llvm::Value *llvmX = moduleTranslation.lookupValue(atomicCompareOp.getX());
+    llvm::Value *llvmV = moduleTranslation.lookupValue(atomicReadOp.getV());
+
+    bool isSigned = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicX = {
+        llvmX, llvmXElementType, isSigned, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicV = {
+        llvmV, llvmXElementType, /*isSigned=*/false, /*IsVolatile=*/false};
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicR = {nullptr, nullptr, false,
+                                                        false};
+
+    llvm::AtomicOrdering atomicOrdering =
+        convertAtomicOrdering(atomicCaptureOp.getMemoryOrder());
+
+    // Pre-translate non-pattern operations inside the compare region.
+    auto isAtomicComparePatternOp = [](Operation &op) {
+      return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
+                       LLVM::OrOp>(op);
+    };
+    for (Operation &op : block.without_terminator()) {
+      if (isAtomicComparePatternOp(op))
+        continue;
+      bool allOperandsMapped =
+          llvm::all_of(op.getOperands(), [&](mlir::Value v) {
+            return moduleTranslation.lookupValue(v) != nullptr;
+          });
+      if (!allOperandsMapped)
+        continue;
+      if (failed(moduleTranslation.convertOperation(op, builder)))
+        return atomicCompareOp.emitError(
+            "failed to translate operation inside atomic compare region");
+    }
+
+    auto materializeValue = [&](mlir::Value val) -> llvm::Value * {
+      if (llvm::Value *existing = moduleTranslation.lookupValue(val))
+        return existing;
+      if (auto loadOp = val.getDefiningOp<LLVM::LoadOp>()) {
+        if (loadOp->getParentRegion() == &region) {
+          llvm::Value *loadAddr =
+              moduleTranslation.lookupValue(loadOp.getAddr());
+          if (!loadAddr)
+            return nullptr;
+          llvm::Type *loadType =
+              moduleTranslation.convertType(loadOp.getResult().getType());
+          return builder.CreateLoad(loadType, loadAddr);
+        }
+      }
+      return nullptr;
+    };
+
+    // Extract comparison predicate, eVal, and dVal from the region.
+    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    llvm::Value *eVal = nullptr;
+    llvm::Value *dVal = nullptr;
+    bool isXBinopExpr = false;
+
+    for (Operation &op : block.getOperations()) {
+      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
+        auto maybeOp =
+            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+        isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                    pred == LLVM::ICmpPredicate::sgt ||
+                    pred == LLVM::ICmpPredicate::sle ||
+                    pred == LLVM::ICmpPredicate::sge);
+
+        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
+        auto maybeOp =
+            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+        if (!maybeOp)
+          return atomicCompareOp.emitError(
+              "unsupported comparison predicate in atomic compare");
+        compareOp = *maybeOp;
+
+        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
+        mlir::Value eOperand =
+            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
+        eVal = materializeValue(eOperand);
+      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        if (!dVal)
+          dVal = materializeValue(selectOp.getTrueValue());
+      }
+    }
+
+    if (!eVal)
+      return atomicCompareOp.emitError(
+          "failed to extract expected value (e) from atomic compare region");
+    if (!dVal) {
+      auto yieldOp = cast<omp::YieldOp>(block.getTerminator());
+      if (yieldOp.getResults().empty())
+        return atomicCompareOp.emitError(
+            "failed to extract desired value (d) from atomic compare region");
+      dVal = materializeValue(yieldOp.getResults()[0]);
+    }
+
+    llvmAtomicX.IsSigned = isSigned;
+
+    llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    bool isPostfixUpdate = true;
+
+    bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, /*IsFailOnly=*/false);
+    ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
+
+    if (failed(handleError(afterIP, *atomicCaptureOp)))
+      return failure();
+
+    builder.restoreIP(*afterIP);
+    return success();
+  }
+
   mlir::Value mlirExpr;
   bool isXBinopExpr = false, isPostfixUpdate = false;
   llvm::AtomicRMWInst::BinOp binop = llvm::AtomicRMWInst::BinOp::BAD_BINOP;
 
-  omp::AtomicUpdateOp atomicUpdateOp = atomicCaptureOp.getAtomicUpdateOp();
-  omp::AtomicWriteOp atomicWriteOp = atomicCaptureOp.getAtomicWriteOp();
-
   assert((atomicUpdateOp || atomicWriteOp) &&
          "internal op must be an atomic.update or atomic.write op");
 
@@ -4896,43 +5069,6 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
   return success();
 }
 
-/// Helper to extract the OMPAtomicCompareOp from an integer comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertICmpPredicateToAtomicCompareOp(LLVM::ICmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::ICmpPredicate::eq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::ICmpPredicate::slt:
-  case LLVM::ICmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::ICmpPredicate::sgt:
-  case LLVM::ICmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
-/// Helper to extract the OMPAtomicCompareOp from a floating-point comparison
-/// predicate. Returns std::nullopt for unsupported predicates.
-static std::optional<llvm::omp::OMPAtomicCompareOp>
-convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
-  switch (predicate) {
-  case LLVM::FCmpPredicate::oeq:
-  case LLVM::FCmpPredicate::ueq:
-    return llvm::omp::OMPAtomicCompareOp::EQ;
-  case LLVM::FCmpPredicate::olt:
-  case LLVM::FCmpPredicate::ult:
-    return llvm::omp::OMPAtomicCompareOp::MIN;
-  case LLVM::FCmpPredicate::ogt:
-  case LLVM::FCmpPredicate::ugt:
-    return llvm::omp::OMPAtomicCompareOp::MAX;
-  default:
-    return std::nullopt;
-  }
-}
-
 /// Converts an omp.atomic.compare operation to LLVM IR.
 ///
 ///      if (x == e) x = d

>From 7d3293944598e8090a64cdec2ac3c791a64b73f1 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Mon, 8 Jun 2026 05:12:22 -0500
Subject: [PATCH 02/11] Adding tests for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     | 32 +++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 56 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 29 ++++++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  9 +--
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 41 ++++++++++++++
 6 files changed, 211 insertions(+), 4 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 249fb0dd8fa64..650f64b80af12 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,3 +260,35 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
+! Integer equality compare+capture: cmpxchg + store old value
+!CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
+! Compare+capture with clause order reversed: capture compare
+!CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+subroutine atomic_capture_compare_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic capture compare
+    v = x
+    if (x == e) x = d
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ac70edbed4e60..752a221aa538d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -161,3 +161,59 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
   !$omp atomic compare weak
   if (x .eq. e) x = d
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_eq(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_int_eq(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_int_gt(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           } {{.*}}weak{{.*}}
+! CHECK:         }
+subroutine atomic_compare_capture_int_gt(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+    v = x
+    if (x > e) x = d
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 4f3cf0eac0338..dc0cc1a62f6c2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -192,6 +192,26 @@ program main
       i = j
    end if
 
+!COMPARE CAPTURE
+!$omp atomic compare capture
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare
+   k = i
+   if (i .eq. j) then
+      i = k
+   end if
+!$omp end atomic
+!$omp atomic capture compare weak
+   k = i
+   if (i < j) then
+      i = k
+   end if
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -296,6 +316,15 @@ end program main
 !CHECK: !$OMP ATOMIC WEAK COMPARE
 !CHECK: !$OMP ATOMIC COMPARE SEQ_CST WEAK
 
+!COMPARE CAPTURE
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 17f99a8c86d52..c5a07a7dc6cb2 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4963,13 +4963,14 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
     bool isPostfixUpdate = true;
+    bool isWeak = atomicCompareOp.getWeak();
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, /*IsFailOnly=*/false);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            /*IsFailOnly=*/false, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index acf6879906e74..289ec60551cb7 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2047,6 +2047,54 @@ func.func @omp_atomic_compare(%x: memref<i32>, %e: i32, %d: i32) {
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_weak
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: } {weak}
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi sgt, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 1890eaa6a3f0b..6ce57b304bfd9 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2795,6 +2795,47 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From f84987dea68a1dedc6b61bf0c9f5a2e24a0a1b40 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 9 Jun 2026 04:42:48 -0500
Subject: [PATCH 03/11] Minor changes incorporating feedback.

---
 flang/lib/Lower/OpenMP/Atomic.cpp                             | 4 ++--
 .../Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index c0f97aada637d..aa497b838e5b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -567,7 +567,7 @@ void Fortran::lower::omp::lowerAtomic(
 
   if (construct.IsCapture()) {
     assert(action0 != analysis.None && action1 != analysis.None &&
-           "Expexcing two actions");
+           "Expecting two actions");
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
@@ -730,7 +730,7 @@ void Fortran::lower::omp::lowerAtomic(
     if (!construct.IsCapture()) {
       // Non-capturing operation.
       assert(action0 != analysis.None && action1 == analysis.None &&
-             "Expexcing single action");
+             "Expecting single action");
       assert(!(analysis.op0.what & analysis.Condition));
       postAt = atomicAt = preAt;
     }
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 8c9015f05bb72..dc5b04126b757 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -315,7 +315,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be updated in "
+                << "captured variable in atomic.read must be compared/updated in "
                     "second operation";
 
         return mlir::success();

>From ca24a147920d24c188127dbbe37ca62bcaeab792 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 14:18:17 -0500
Subject: [PATCH 04/11] Handling test cases mentionend in OpenMP spec.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 61 ++++++++++++++++---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 ++-
 .../Interfaces/AtomicInterfaces.td            | 11 +++-
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  2 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 12 ++--
 5 files changed, 74 insertions(+), 19 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index aa497b838e5b6..209871abd52b6 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -571,7 +571,8 @@ void Fortran::lower::omp::lowerAtomic(
     (void)action0;
     (void)action1;
     captureOp = mlir::omp::AtomicCaptureOp::create(
-        builder, loc, hint, makeMemOrderAttr(converter, memOrder));
+        builder, loc, hint, makeMemOrderAttr(converter, memOrder),
+        /*fail_only=*/nullptr);
     // Set the non-atomic insertion point to before the atomic.capture.
     preAt = getInsertionPointBefore(captureOp);
 
@@ -646,15 +647,45 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
-    // If this is a compare+capture, generate the read op first.
+    // If this is a compare+capture, determine the ordering of ops.
+    // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
+    // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
+    // Pattern 3 (fail-only): if (x == e) x = d; else v = x → read on failure
+    bool isPostfixCapture = false;
+    bool isFailOnly = false;
+    const evaluate::Assignment *readAssign = nullptr;
     if (construct.IsCapture()) {
-      assert(get(analysis.op0.assign) && (analysis.op0.what & analysis.Read) &&
-             "Expected a read operation for compare capture");
-      mlir::Operation *readOp = genAtomicRead(
-          converter, semaCtx, loc, stmtCtx, atomAddr, atom,
-          *get(analysis.op0.assign), hint, memOrder, preAt, atomicAt, postAt);
-      assert(readOp && "Should have created an atomic read operation");
-      builder.setInsertionPointAfter(readOp);
+      // Determine which op is the read and check for fail-only (IfFalse).
+      int readWhat = 0;
+      if (analysis.op0.what & analysis.Read) {
+        readAssign = get(analysis.op0.assign);
+        readWhat = analysis.op0.what;
+      } else if (analysis.op1.what & analysis.Read) {
+        readAssign = get(analysis.op1.assign);
+        readWhat = analysis.op1.what;
+        isPostfixCapture = true;
+      }
+      assert(readAssign && "Expected a read assignment for compare capture");
+
+      // Check if the read is conditioned on comparison failure (else branch).
+      if (readWhat & analysis.IfFalse)
+        isFailOnly = true;
+
+      if (!isPostfixCapture && !isFailOnly) {
+        // Pattern 1 (prefix): read is first, generate it before compare.
+        mlir::Operation *readOp =
+            genAtomicRead(converter, semaCtx, loc, stmtCtx, atomAddr, atom,
+                          *readAssign, hint, memOrder, preAt, atomicAt, postAt);
+        assert(readOp && "Should have created an atomic read operation");
+        builder.setInsertionPointAfter(readOp);
+      } else {
+        // Pattern 2 (postfix) or 3 (fail-only): compare first, read after.
+        builder.restoreInsertionPoint(atomicAt);
+      }
+
+      // Set the fail_only attribute on the capture op.
+      if (isFailOnly && captureOp)
+        mlir::cast<mlir::omp::AtomicCaptureOp>(captureOp).setFailOnly(true);
     }
 
     mlir::UnitAttr weakAttr = nullptr;
@@ -725,6 +756,18 @@ void Fortran::lower::omp::lowerAtomic(
     // Generate omp.yield
     mlir::omp::YieldOp::create(builder, loc, newVal);
     builder.setInsertionPointAfter(atomicOp);
+
+    // Pattern 2 (postfix) or 3 (fail-only): compare first, read second.
+    // Generate read after compare for postfix or fail-only patterns.
+    if (construct.IsCapture() && (isPostfixCapture || isFailOnly)) {
+      fir::FirOpBuilder::InsertPoint afterCompareAt =
+          builder.saveInsertionPoint();
+      mlir::Operation *readOp = genAtomicRead(
+          converter, semaCtx, loc, stmtCtx, atomAddr, atom, *readAssign, hint,
+          memOrder, preAt, afterCompareAt, postAt);
+      assert(readOp && "Should have created an atomic read operation");
+      builder.setInsertionPointAfter(readOp);
+    }
     // END omp atomic compare
   } else {
     if (!construct.IsCapture()) {
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 92eb7de0d882f..b2ccc05ad6770 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11321,7 +11321,9 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11377,7 +11379,8 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
+          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
+          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index dc5b04126b757..5b1527bcb84ec 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -286,6 +286,7 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         auto &secondOp = *ops.getNextNode(firstOp);
         auto firstReadStmt = dyn_cast<AtomicReadOpInterface>(firstOp);
         auto firstUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(firstOp);
+        auto firstCompareStmt = dyn_cast<AtomicCompareOpInterface>(firstOp);
         auto secondReadStmt = dyn_cast<AtomicReadOpInterface>(secondOp);
         auto secondUpdateStmt = dyn_cast<AtomicUpdateOpInterface>(secondOp);
         auto secondWriteStmt = dyn_cast<AtomicWriteOpInterface>(secondOp);
@@ -294,7 +295,8 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (!((firstUpdateStmt && secondReadStmt) ||
               (firstReadStmt && secondUpdateStmt) ||
               (firstReadStmt && secondWriteStmt) ||
-              (firstReadStmt && secondCompareStmt)))
+              (firstReadStmt && secondCompareStmt) ||
+              (firstCompareStmt && secondReadStmt)))
           return ops.front().emitError()
                 << "invalid sequence of operations in the capture region";
         if (firstUpdateStmt && secondReadStmt &&
@@ -315,8 +317,13 @@ def AtomicCaptureOpInterface : OpInterface<"AtomicCaptureOpInterface"> {
         if (firstReadStmt && secondCompareStmt &&
             firstReadStmt.getX() != secondCompareStmt.getX())
           return firstReadStmt.emitError()
-                << "captured variable in atomic.read must be compared/updated in "
+                << "captured variable in atomic.read must be updated in "
                     "second operation";
+        if (firstCompareStmt && secondReadStmt &&
+            secondReadStmt.getX() != firstCompareStmt.getX())
+          return secondReadStmt.emitError()
+                << "captured variable in atomic.read must be updated in "
+                    "first operation";
 
         return mlir::success();
       }]
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index ebee887f2afd1..69c22641df6a2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1938,6 +1938,8 @@ def AtomicCaptureOp : OpenMP_Op<"atomic.capture", traits = [
     ```
   }] # clausesDescription;
 
+  let arguments = !con(clausesArgs, (ins UnitAttr:$fail_only));
+
   // Override region definition.
   let regions = (region SizedRegion<1>:$region);
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index dff1ae30e20aa..76616c7fba27a 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,15 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    bool isPostfixUpdate = true;
-    bool isWeak = atomicCompareOp.getWeak();
+    bool isFailOnly = atomicCaptureOp.getFailOnly();
+    bool isPostfixUpdate = !isFailOnly;
 
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(
-            ompLoc, llvmAtomicX, llvmAtomicV, llvmAtomicR, eVal, dVal,
-            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
-            /*IsFailOnly=*/false, isWeak);
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+                                        llvmAtomicR, eVal, dVal, atomicOrdering,
+                                        compareOp, isXBinopExpr,
+                                        isPostfixUpdate, isFailOnly);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))

>From afaf2025463e9e8cf15974747fa9e2d8b8e5fa95 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 23 Jun 2026 18:02:52 -0500
Subject: [PATCH 05/11] Adding tests into flang and mlir for the grammar
 mentionend in the spec.

---
 .../Integration/OpenMP/atomic-compare.f90     | 56 ++++++++++++++++--
 flang/test/Lower/OpenMP/atomic-compare.f90    | 59 +++++++++++++++++++
 flang/test/Parser/OpenMP/atomic-unparse.f90   | 17 ++++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 +++++-
 mlir/test/Dialect/OpenMP/ops.mlir             | 48 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 58 ++++++++++++++++--
 6 files changed, 245 insertions(+), 10 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 650f64b80af12..da86f3ff6ef16 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -260,14 +260,17 @@ subroutine atomic_compare_weak(x, e, d)
   if (x == e) x = d
 end 
 
-! Integer equality compare+capture: cmpxchg + store old value
+! Integer equality compare+capture (prefix): v=x; if(x==e) x=d
+! v captures old value of x
 !CHECK-LABEL: define void @atomic_compare_capture_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -276,14 +279,16 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
-! Compare+capture with clause order reversed: capture compare
+! Compare+capture with clause order reversed: capture compare (still prefix read)
 !CHECK-LABEL: define void @atomic_capture_compare_int_eq_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
 !CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
@@ -292,3 +297,46 @@ subroutine atomic_capture_compare_int_eq(x, e, d, v)
   !$omp end atomic
 end
 
+! Postfix compare+capture: if (x == e) x = d; v = x
+! v captures new value (d if swapped, old x if not)
+!CHECK-LABEL: define void @atomic_compare_capture_postfix_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+!CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Fail-only compare+capture: if (x == e) x = d; else v = x
+! v is only written when the comparison fails
+!CHECK-LABEL: define void @atomic_compare_capture_fail_only_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load i32, ptr %[[E]]
+!CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+!CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+!CHECK: {{.*}}:
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
+!CHECK: br label %[[EXIT2:.*]]
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 752a221aa538d..c770605954539 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -217,3 +217,62 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
     if (x > e) x = d
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_fail_only(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_fail_only(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eq. e) then
+      x = d
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index dc0cc1a62f6c2..521c1bb2811e2 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -211,6 +211,19 @@ program main
       i = k
    end if
 !$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   end if
+   k = i
+!$omp end atomic
+!$omp atomic compare capture
+   if (i .eq. j) then
+      i = k
+   else
+      k = i
+   end if
+!$omp end atomic
 
 !ATOMIC
 !$omp atomic
@@ -324,6 +337,10 @@ end program main
 !CHECK: !$OMP END ATOMIC
 !CHECK: !$OMP ATOMIC CAPTURE COMPARE WEAK
 !CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
 
 !ATOMIC
 !CHECK: !$OMP ATOMIC
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 76616c7fba27a..b6fc6d6dc3038 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4974,15 +4974,28 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+    // In OMPIRBuilder::createAtomicCompare:
+    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
+    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
+    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
+    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
+    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
+    bool isPostfixUpdate =
+        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
     bool isFailOnly = atomicCaptureOp.getFailOnly();
-    bool isPostfixUpdate = !isFailOnly;
 
+    // For fail-only, the OMPIRBuilder's conditional store logic is under
+    // the !IsPostfixUpdate path, so force it to false.
+    if (isFailOnly)
+      isPostfixUpdate = false;
+
+    bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
         ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly);
+                                        isPostfixUpdate, isFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 87f9a6f8e4119..6b13be481e0e0 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2113,6 +2113,54 @@ func.func @omp_atomic_compare_capture_weak(%v: memref<i32>, %x: memref<i32>, %e:
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_postfix
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_postfix(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (%[[V:.*]]: memref<i32>, %[[X:.*]]: memref<i32>, %[[E:.*]]: i32, %[[D:.*]]: i32)
+func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>, %e: i32, %d: i32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<i32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: i32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[E]] : i32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : i32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : i32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<i32>, memref<i32>, i32
+  // CHECK-NEXT: } {fail_only}
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<i32> {
+    ^bb0(%xval: i32):
+      %cmp = arith.cmpi eq, %xval, %e : i32
+      %sel = arith.select %cmp, %d, %xval : i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : memref<i32>, memref<i32>, i32
+  } {fail_only}
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 6ce57b304bfd9..978450d598b9b 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,10 +2798,12 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2819,10 +2821,12 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality compare+capture → cmpxchg + extractvalue + store
+  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
+  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2836,6 +2840,52 @@ llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_postfix
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Postfix compare+capture: v captures new value (d if swapped, old x if not)
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[D]], i32 %[[OLD]]
+  // CHECK: store i32 %[[NEWVAL]], ptr %[[V]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+// -----
+
+// CHECK-LABEL: @omp_atomic_compare_capture_fail_only
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
+  // Fail-only compare+capture: v is only written when comparison fails
+  // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
+  // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+  // CHECK: br i1 %[[SUCCESS]], label %[[EXIT:.*]], label %[[CONT:.*]]
+  // CHECK: [[CONT]]:
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
+  // CHECK: br label %[[EXIT2:.*]]
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "eq" %xval, %e : i32
+      %sel = llvm.select %cmp, %d, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From e8462a267b6d010ebc1e3ed881c1f38a9faca6be Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 24 Jun 2026 14:44:43 -0500
Subject: [PATCH 06/11] Simplifying the IR generator for one of the cases. This
 also fixes the failure in clang for "atomic compare capture".

---
 .../Integration/OpenMP/atomic-compare.f90     |  8 +---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  7 +--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 43 +++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      | 12 ++----
 4 files changed, 38 insertions(+), 32 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index da86f3ff6ef16..4ae1b9a1f1e52 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -268,9 +268,7 @@ subroutine atomic_compare_weak(x, e, d)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_compare_capture_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic compare capture
@@ -286,9 +284,7 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 !CHECK: %[[DVAL:.*]] = load i32, ptr %[[D]]
 !CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
 !CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-!CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-!CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[EVAL]], i32 %[[OLD]]
-!CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+!CHECK: store i32 %[[OLD]], ptr %[[V]]
 subroutine atomic_capture_compare_int_eq(x, e, d, v)
   integer :: x, e, d, v
   !$omp atomic capture compare
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 51f210e189e22..f8c1999fe1b89 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -11400,9 +11400,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           SuccessOrFail = Builder.CreateExtractValue(Result, /*Idxs=*/1);
           if (IsFailOnly) {
@@ -11458,8 +11456,7 @@ OpenMPIRBuilder::InsertPointTy OpenMPIRBuilder::createAtomicCompare(
         assert(OldValue->getType() == V.ElemTy &&
                "OldValue and V must be of same type");
         if (IsPostfixUpdate) {
-          Value *NewValue = Builder.CreateSelect(SuccessOrFail, D, OldValue);
-          Builder.CreateStore(NewValue, V.Var, V.IsVolatile);
+          Builder.CreateStore(OldValue, V.Var, V.IsVolatile);
         } else {
           if (IsFailOnly) {
             BasicBlock *CurBB = Builder.GetInsertBlock();
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 85389caa9ac42..e1fb08d780753 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5499,25 +5499,23 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     llvmAtomicX.IsSigned = isSigned;
 
     llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
-    // In OMPIRBuilder::createAtomicCompare:
-    //   IsPostfixUpdate=true  → select(success, D, OldValue) → new value
-    //   IsPostfixUpdate=false → select(success, E, OldValue) → old value
-    // Prefix read (v=x; if(x==e) x=d): v captures old → IsPostfixUpdate=false
-    // Postfix read (if(x==e) x=d; v=x): v captures new → IsPostfixUpdate=true
-    // Fail-only: conditional store only on failure → IsPostfixUpdate=false
-    bool isPostfixUpdate =
-        !isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isReadFirst = isa<omp::AtomicReadOp>(atomicCaptureOp.getFirstOp());
+    bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
-    // For fail-only, the OMPIRBuilder's conditional store logic is under
-    // the !IsPostfixUpdate path, so force it to false.
-    if (isFailOnly)
-      isPostfixUpdate = false;
+    llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
+    // Postfix: bypass V in OMPIRBuilder, handled manually below.
+    if (isPostfixCapture && !isFailOnly)
+      llvmAtomicVForCall = {nullptr, nullptr, false, false};
+
+    // Prefix: IsPostfixUpdate=true → direct store of old value.
+    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
+    bool isPostfixUpdate = !isFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicV,
+        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
                                         llvmAtomicR, eVal, dVal, atomicOrdering,
                                         compareOp, isXBinopExpr,
                                         isPostfixUpdate, isFailOnly, isWeak);
@@ -5527,6 +5525,25 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
       return failure();
 
     builder.restoreIP(*afterIP);
+
+    // Postfix: v = select(success, D, old) — captures new value of x.
+    if (isPostfixCapture && !isFailOnly) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::Instruction *cmpxchgInst = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
+          cmpxchgInst = &inst;
+          break;
+        }
+      }
+      assert(cmpxchgInst && "expected cmpxchg instruction");
+      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
+      llvm::Value *successVal =
+          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+      llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
+      builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+    }
+
     return success();
   }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 978450d598b9b..377bd88894854 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2798,12 +2798,10 @@ llvm.func @omp_atomic_compare_float_neg_zero(%xf : !llvm.ptr, %ef : f32, %df : f
 // CHECK-LABEL: @omp_atomic_compare_capture_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {
@@ -2821,12 +2819,10 @@ llvm.func @omp_atomic_compare_capture_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e
 // CHECK-LABEL: @omp_atomic_compare_capture_weak_int_eq
 // CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]], i32 %[[D:.*]])
 llvm.func @omp_atomic_compare_capture_weak_int_eq(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32, %d : i32) {
-  // Integer equality weak prefix compare+capture → cmpxchg + select(success, E, old) + store
+  // Integer equality weak prefix compare+capture → cmpxchg + store old value
   // CHECK: %[[RES:.*]] = cmpxchg weak ptr %[[X]], i32 %[[E]], i32 %[[D]] monotonic monotonic
   // CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
-  // CHECK: %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RES]], 1
-  // CHECK: %[[CAPTURED:.*]] = select i1 %[[SUCCESS]], i32 %[[E]], i32 %[[OLD]]
-  // CHECK: store i32 %[[CAPTURED]], ptr %[[V]]
+  // CHECK: store i32 %[[OLD]], ptr %[[V]]
   omp.atomic.capture {
     omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
     omp.atomic.compare %x : !llvm.ptr {

>From 101b9fa02882686a21d346ef12e541c717ca605b Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 14:00:46 -0500
Subject: [PATCH 07/11] Handling real and logical types.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             | 117 ++++++--
 .../Integration/OpenMP/atomic-compare.f90     |  58 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    |  83 +++++-
 flang/test/Parser/OpenMP/atomic-unparse.f90   |  12 +
 .../Interfaces/AtomicInterfaces.td            |  13 +
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 252 +++++++++++-------
 mlir/test/Dialect/OpenMP/ops.mlir             |  24 ++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++
 8 files changed, 470 insertions(+), 137 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 209871abd52b6..3811ea3d40bbd 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -375,6 +375,8 @@ genAtomicRead(lower::AbstractConverter &converter,
         fir::getBase(converter.genExprValue(assign.rhs, stmtCtx, &loc));
     converter.resetExprOverrides();
 
+    if (value.getType() != storeType)
+      value = builder.createConvert(loc, storeType, value);
     fir::StoreOp::create(builder, loc, value, storeAddr);
   }
   return op;
@@ -634,6 +636,45 @@ void Fortran::lower::omp::lowerAtomic(
           },
           rel->u);
     }
+
+    // Fortran uses .eqv./.neqv. for logical equality/inequality, which are
+    // LogicalOperation expressions rather than Relational expressions.
+    if (!expectedExprStorage) {
+      if (const auto *someLogical =
+              evaluate::UnwrapExpr<evaluate::Expr<evaluate::SomeLogical>>(
+                  *cond)) {
+        common::visit(
+            [&](const auto &kindLogical) {
+              using LogicalExpr = std::decay_t<decltype(kindLogical)>;
+              constexpr int K = LogicalExpr::Result::kind;
+              if (const auto *logOp =
+                      std::get_if<evaluate::LogicalOperation<K>>(
+                          &kindLogical.u)) {
+                if (logOp->logicalOperator == common::LogicalOperator::Eqv ||
+                    logOp->logicalOperator == common::LogicalOperator::Neqv) {
+                  relOpr =
+                      (logOp->logicalOperator == common::LogicalOperator::Eqv)
+                          ? common::RelationalOperator::EQ
+                          : common::RelationalOperator::NE;
+                  using Operand =
+                      typename evaluate::LogicalOperation<K>::Operand;
+                  auto leftExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->left()});
+                  auto rightExpr = evaluate::AsGenericExpr(
+                      evaluate::Expr<Operand>{logOp->right()});
+                  if (evaluate::IsSameOrConvertOf(rightExpr, atom)) {
+                    expectedExprStorage = std::move(leftExpr);
+                    relOpr = reverseRelOp(relOpr);
+                  } else {
+                    expectedExprStorage = std::move(rightExpr);
+                  }
+                }
+              }
+            },
+            someLogical->u);
+      }
+    }
+
     if (!expectedExprStorage) {
       mlir::emitError(loc, "internal error: atomic compare condition is not a "
                            "recognized relational expression");
@@ -647,6 +688,20 @@ void Fortran::lower::omp::lowerAtomic(
       expectedVal = builder.createConvert(loc, elemTypeOfX, expectedVal);
     }
 
+    // For logical types, convert address and expected value to integer
+    // type here (above the atomic compare region) so that the region
+    // only contains arith.cmpi eq on integers. If done inside the region,
+    // fir.convert logical<4> -> i32 would lower to `icmp ne %val, 0`
+    // which violates the atomic compare verifier's expectations.
+    if (mlir::isa<fir::LogicalType>(elemTypeOfX)) {
+      unsigned kind = mlir::cast<fir::LogicalType>(elemTypeOfX).getFKind();
+      mlir::Type intTy = builder.getIntegerType(kind * 8);
+      mlir::Type intRefTy = builder.getRefType(intTy);
+      atomAddr = builder.createConvert(loc, intRefTy, atomAddr);
+      expectedVal = builder.createConvert(loc, intTy, expectedVal);
+      elemTypeOfX = intTy;
+    }
+
     // If this is a compare+capture, determine the ordering of ops.
     // Pattern 1 (prefix): v = x; if (x == e) x = d  → read first
     // Pattern 2 (postfix): if (x == e) x = d; v = x → compare first
@@ -694,6 +749,39 @@ void Fortran::lower::omp::lowerAtomic(
         })) {
       weakAttr = builder.getUnitAttr();
     }
+
+    // Extract write assignment (x = d) and generate desired value (d)
+    // before creating the compare region, so that d is defined outside
+    // the region and any intermediate conversions (e.g., logical-to-integer
+    // truthiness normalization) don't appear inside the atomic compare block.
+    [[maybe_unused]] int writeActionCond = 0;
+    const evaluate::Assignment *writeAssign = nullptr;
+    if (analysis.op0.what & analysis.Write) {
+      writeAssign = get(analysis.op0.assign);
+      writeActionCond = analysis.op0.what;
+    }
+    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
+      writeAssign = get(analysis.op1.assign);
+      writeActionCond = analysis.op1.what;
+    }
+    if (!writeAssign) {
+      mlir::emitError(loc,
+                      "internal error: atomic compare has no write assignment");
+      return;
+    }
+    assert((writeActionCond & analysis.IfTrue) &&
+           "atomic compare write should be conditioned on IfTrue");
+
+    // Generate desiredVal before the capture/compare region so any
+    // intermediate ops (loads, conversions) don't pollute the atomic blocks.
+    fir::FirOpBuilder::InsertPoint savedIP = builder.saveInsertionPoint();
+    builder.restoreInsertionPoint(preAt);
+    mlir::Value desiredVal =
+        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
+    if (desiredVal.getType() != elemTypeOfX)
+      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
+    builder.restoreInsertionPoint(savedIP);
+
     mlir::Operation *atomicOp = mlir::omp::AtomicCompareOp::create(
         builder, loc, atomAddr, weakAttr, hint,
         makeMemOrderAttr(converter, memOrder));
@@ -721,35 +809,6 @@ void Fortran::lower::omp::lowerAtomic(
       return;
     }
 
-    // Check for presence of Assignment (x = d) and wether it is being invoked
-    // only for IfTrue condition.
-
-    // writeActionCond is a bitmask combining the following flags:
-    //  1) the action type (Read/Write/Update)
-    //  2) condition (IfTrue/IfFalse)
-    [[maybe_unused]] int writeActionCond = 0;
-    const evaluate::Assignment *writeAssign = nullptr;
-    if (analysis.op0.what & analysis.Write) {
-      writeAssign = get(analysis.op0.assign);
-      writeActionCond = analysis.op0.what;
-    }
-    if (!writeAssign && (analysis.op1.what & analysis.Write)) {
-      writeAssign = get(analysis.op1.assign);
-      writeActionCond = analysis.op1.what;
-    }
-    if (!writeAssign) {
-      mlir::emitError(loc,
-                      "internal error: atomic compare has no write assignment");
-      return;
-    }
-    assert((writeActionCond & analysis.IfTrue) &&
-           "atomic compare write should be conditioned on IfTrue");
-
-    // Generate new/desired value of x e.g. x = d
-    mlir::Value desiredVal =
-        fir::getBase(converter.genExprValue(writeAssign->rhs, stmtCtx, &loc));
-    if (desiredVal.getType() != elemTypeOfX)
-      desiredVal = builder.createConvert(loc, elemTypeOfX, desiredVal);
     mlir::Value newVal = mlir::arith::SelectOp::create(builder, loc, cmpResult,
                                                        desiredVal, blockArg);
 
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index 4ae1b9a1f1e52..f30349c42e33e 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -336,3 +336,61 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
   !$omp end atomic
 end
 
+! Real equality compare+capture (postfix): if(x==e) x=d; v=x
+! v captures new value of x (d if success, old if fail)
+!CHECK-LABEL: define void @atomic_compare_capture_real_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: %[[EVAL:.*]] = load float, ptr %[[E]]{{.*}}
+!CHECK: %[[DVAL:.*]] = load float, ptr %[[D]]{{.*}}
+!CHECK: %[[EBITS:.*]] = bitcast float %[[EVAL]] to i32
+!CHECK: %[[DBITS:.*]] = bitcast float %[[DVAL]] to i32
+!CHECK: %[[XLOAD:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+!CHECK: %[[XFP:.*]] = bitcast i32 %[[XLOAD]] to float
+! Part 1: NaN check - if either x or e is NaN, comparison fails
+!CHECK: %[[E_ISNAN:.*]] = fcmp uno float %[[EVAL]], %[[EVAL]]
+!CHECK: %[[X_ISNAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+!CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_ISNAN]], %[[X_ISNAN]]
+!CHECK: br i1 %[[EITHER_NAN]], label %[[NAN_BB:.*]], label %[[NOTNAN_BB:.*]]
+!CHECK: [[NAN_BB]]:
+!CHECK: br label %[[EXIT_BB:.*]]
+! Part 2: Both-zero check - handles +0.0 vs -0.0 (same value, different bits)
+!CHECK: [[NOTNAN_BB]]:
+!CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+!CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[EVAL]], 0.000000e+00
+!CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+!CHECK: br i1 %[[BOTHZERO]], label %[[ZERO_BB:.*]], label %[[NORMAL_BB:.*]]
+!CHECK: [[ZERO_BB]]:
+!CHECK: %[[ZERORES:.*]] = cmpxchg ptr %[[X]], i32 %[[XLOAD]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Part 3: Normal compare - standard cmpxchg with bitcasted expected value
+!CHECK: [[NORMAL_BB]]:
+!CHECK: %[[NORMRES:.*]] = cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+!CHECK: br label %[[EXIT_BB]]
+! Exit: select v = (success ? d : old_x)
+!CHECK: [[EXIT_BB]]:
+!CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+!CHECK: %[[OK:.*]] = phi i1 {{.*}}
+!CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+!CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[DVAL]], float %[[OLDFP]]
+!CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Logical .eqv. compare+capture (postfix): if(x.eqv.e) x=d; v=x
+! Logicals are compared as integers after truthiness normalization
+!CHECK-LABEL: define void @atomic_compare_capture_logical_(
+!CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]], ptr noalias %[[V:.*]])
+!CHECK: cmpxchg ptr %[[X]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index c770605954539..1219d05296e4d 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -10,10 +10,10 @@
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -31,10 +31,10 @@ subroutine atomic_compare_int_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -52,10 +52,10 @@ subroutine atomic_compare_float_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<complex<f32>>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<complex<f32>> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: complex<f32>):
 ! CHECK:           %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %[[EVAL]] {fastmath = #arith.fastmath<contract>} : complex<f32>
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<complex<f32>>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : complex<f32>
 ! CHECK:           omp.yield(%[[SEL]] : complex<f32>)
 ! CHECK:         }
@@ -71,10 +71,10 @@ subroutine atomic_compare_complex_eq(x, e, d)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi slt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -90,10 +90,10 @@ subroutine atomic_compare_int_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         }
@@ -109,10 +109,10 @@ subroutine atomic_compare_int_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -128,10 +128,10 @@ subroutine atomic_compare_float_lt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<f32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: f32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
-! CHECK:           %[[EVAL2:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[EVAL2]], %[[XVAL]] : f32
 ! CHECK:           omp.yield(%[[SEL]] : f32)
 ! CHECK:         }
@@ -149,10 +149,10 @@ subroutine atomic_compare_float_gt(x, e)
 ! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.compare memory_order(relaxed) %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:         ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:           %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:           %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:           %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:           omp.yield(%[[SEL]] : i32)
 ! CHECK:         } {{.*}}weak{{.*}}
@@ -172,12 +172,12 @@ subroutine atomic_compare_int_eq_weak(x, e, d)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -200,12 +200,12 @@ subroutine atomic_compare_capture_int_eq(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           } {{.*}}weak{{.*}}
@@ -228,11 +228,11 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -256,11 +256,11 @@ subroutine atomic_compare_capture_postfix(x, e, d, v)
 ! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
 ! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
 ! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:         omp.atomic.capture memory_order(relaxed) {
 ! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
 ! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
 ! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
-! CHECK:             %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
 ! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
 ! CHECK:             omp.yield(%[[SEL]] : i32)
 ! CHECK:           }
@@ -276,3 +276,62 @@ subroutine atomic_compare_capture_fail_only(x, e, d, v)
     end if
   !$omp end atomic
 end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<f32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<f32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<f32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<f32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:         %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         %[[DLOAD:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:         %[[DCONV:.*]] = fir.convert %[[DLOAD]] : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DCONV]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+    if (x .eqv. e) x = d
+    v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Parser/OpenMP/atomic-unparse.f90 b/flang/test/Parser/OpenMP/atomic-unparse.f90
index 521c1bb2811e2..0d30cd48035ea 100644
--- a/flang/test/Parser/OpenMP/atomic-unparse.f90
+++ b/flang/test/Parser/OpenMP/atomic-unparse.f90
@@ -4,6 +4,7 @@ program main
    implicit none
    integer :: i, j = 10
    integer :: k
+   logical :: l1, l2, l3, l4
 !READ
 !$omp atomic read
    i = j
@@ -225,6 +226,12 @@ program main
    end if
 !$omp end atomic
 
+!COMPARE CAPTURE LOGICAL
+!$omp atomic compare capture
+   if (l1 .eqv. l2) l1 = l3
+   l4 = l1
+!$omp end atomic
+
 !ATOMIC
 !$omp atomic
    i = j
@@ -342,6 +349,11 @@ end program main
 !CHECK: !$OMP ATOMIC COMPARE CAPTURE
 !CHECK: !$OMP END ATOMIC
 
+!COMPARE CAPTURE LOGICAL
+
+!CHECK: !$OMP ATOMIC COMPARE CAPTURE
+!CHECK: !$OMP END ATOMIC
+
 !ATOMIC
 !CHECK: !$OMP ATOMIC
 !CHECK: !$OMP ATOMIC SEQ_CST
diff --git a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
index 5b1527bcb84ec..a61c43e2fd9cd 100644
--- a/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACCMPCommon/Interfaces/AtomicInterfaces.td
@@ -488,6 +488,19 @@ def AtomicCompareOpInterface : OpInterface<"AtomicCompareOpInterface"> {
               }
             }
             break;
+          } else if (llvm::is_contained(
+                         {"arith.maxsi", "arith.minsi", "arith.maxui",
+                          "arith.minui", "arith.maximumf", "arith.minimumf",
+                          "llvm.intr.smax", "llvm.intr.smin",
+                          "llvm.intr.umax", "llvm.intr.umin",
+                          "llvm.intr.maxnum", "llvm.intr.minnum"},
+                         opName)) {
+            // The canonicalizer may fold select(cmpi slt, e, x) into
+            // maxsi/minsi (or the float equivalents). After fir-to-llvm-ir
+            // these become llvm.intr.smax/smin etc. All are valid
+            // representations of atomic compare min/max operations.
+            foundComparison = true;
+            break;
           }
         }
 
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e1fb08d780753..8e49571cd2461 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -12,6 +12,7 @@
 //===----------------------------------------------------------------------===//
 #include "mlir/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.h"
 #include "mlir/Analysis/TopologicalSortUtils.h"
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"
 #include "mlir/Dialect/LLVMIR/LLVMTypes.h"
 #include "mlir/Dialect/OpenMP/OpenMPDialect.h"
@@ -5369,6 +5370,102 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Holds the extracted comparison pattern information from an atomic compare
+/// region.
+struct AtomicComparePatternInfo {
+  llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+  llvm::Value *eVal = nullptr;
+  llvm::Value *dVal = nullptr;
+  bool isXBinopExpr = false;
+  bool isSigned = false;
+};
+
+/// Extract comparison predicate, expected value (e), desired value (d), and
+/// related flags from an atomic compare region block by scanning for
+/// icmp/fcmp/select/min/max operations.
+static LogicalResult extractAtomicComparePattern(
+    Block &block,
+    llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
+    omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  for (Operation &op : block.getOperations()) {
+    // Pre-filter: skip icmps that don't involve the block argument
+    // (e.g., truthiness extractions from logical-to-integer conversion).
+    if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op);
+        icmpOp && icmpOp.getOperand(0) != block.getArgument(0) &&
+        icmpOp.getOperand(1) != block.getArgument(0))
+      continue;
+
+    LogicalResult result =
+        llvm::TypeSwitch<Operation *, LogicalResult>(&op)
+            .Case<LLVM::ICmpOp>([&](LLVM::ICmpOp icmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              LLVM::ICmpPredicate pred = icmpOp.getPredicate();
+              info.isSigned = (pred == LLVM::ICmpPredicate::slt ||
+                               pred == LLVM::ICmpPredicate::sgt ||
+                               pred == LLVM::ICmpPredicate::sle ||
+                               pred == LLVM::ICmpPredicate::sge);
+              info.isXBinopExpr =
+                  (icmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? icmpOp.getOperand(1)
+                                                       : icmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::FCmpOp>([&](LLVM::FCmpOp fcmpOp) -> LogicalResult {
+              auto maybeOp =
+                  convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
+              if (!maybeOp)
+                return atomicCompareOp.emitError(
+                    "unsupported comparison predicate in atomic compare");
+              info.compareOp = *maybeOp;
+              info.isXBinopExpr =
+                  (fcmpOp.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand = info.isXBinopExpr ? fcmpOp.getOperand(1)
+                                                       : fcmpOp.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              return success();
+            })
+            .Case<LLVM::SelectOp>([&](LLVM::SelectOp selectOp) {
+              if (!info.dVal)
+                info.dVal = materializeValue(selectOp.getTrueValue());
+              return success();
+            })
+            .Case<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                  mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                  mlir::arith::MaximumFOp, mlir::arith::MinimumFOp,
+                  LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp, LLVM::UMinOp,
+                  LLVM::MaxNumOp, LLVM::MinNumOp>([&](Operation *) {
+              // Canonicalized min/max ops (arith or LLVM intrinsic form).
+              // max(x,e) came from slt/ult/olt -> OMPAtomicCompareOp::MIN
+              // min(x,e) came from sgt/ugt/ogt -> OMPAtomicCompareOp::MAX
+              // (OMPIRBuilder inverts: MIN->atomicrmw max, MAX->atomicrmw min)
+              bool isMax = isa<mlir::arith::MaxSIOp, mlir::arith::MaxUIOp,
+                               mlir::arith::MaximumFOp, LLVM::SMaxOp,
+                               LLVM::UMaxOp, LLVM::MaxNumOp>(op);
+              info.compareOp = isMax ? llvm::omp::OMPAtomicCompareOp::MIN
+                                     : llvm::omp::OMPAtomicCompareOp::MAX;
+              info.isSigned = isa<mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                                  LLVM::SMaxOp, LLVM::SMinOp>(op);
+              info.isXBinopExpr = (op.getOperand(0) == block.getArgument(0));
+              mlir::Value eOperand =
+                  info.isXBinopExpr ? op.getOperand(1) : op.getOperand(0);
+              info.eVal = materializeValue(eOperand);
+              info.dVal = info.eVal;
+              return success();
+            })
+            .Default([](Operation *) { return success(); });
+
+    if (failed(result))
+      return result;
+  }
+  return success();
+}
+
 static LogicalResult
 convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
                         llvm::IRBuilderBase &builder,
@@ -5409,7 +5506,11 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Pre-translate non-pattern operations inside the compare region.
     auto isAtomicComparePatternOp = [](Operation &op) {
       return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                       LLVM::OrOp>(op);
+                       LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                       LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                       mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                       mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                       mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
     };
     for (Operation &op : block.without_terminator()) {
       if (isAtomicComparePatternOp(op))
@@ -5443,47 +5544,16 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     };
 
     // Extract comparison predicate, eVal, and dVal from the region.
-    llvm::omp::OMPAtomicCompareOp compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    llvm::Value *eVal = nullptr;
-    llvm::Value *dVal = nullptr;
-    bool isXBinopExpr = false;
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
 
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
+    llvm::omp::OMPAtomicCompareOp compareOp = patternInfo.compareOp;
+    llvm::Value *eVal = patternInfo.eVal;
+    llvm::Value *dVal = patternInfo.dVal;
+    bool isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
 
     if (!eVal)
       return atomicCompareOp.emitError(
@@ -5529,17 +5599,42 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     // Postfix: v = select(success, D, old) — captures new value of x.
     if (isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
-      llvm::Instruction *cmpxchgInst = nullptr;
+      llvm::Value *oldVal = nullptr;
+      llvm::Value *successVal = nullptr;
+
+      // Integer path (and non-HandleFPNegZero FP path): a single cmpxchg
+      // lives in the current block.
       for (auto &inst : llvm::reverse(*curBB)) {
         if (isa<llvm::AtomicCmpXchgInst>(&inst)) {
-          cmpxchgInst = &inst;
+          oldVal = builder.CreateExtractValue(&inst, /*Idxs=*/0);
+          successVal = builder.CreateExtractValue(&inst, /*Idxs=*/1);
           break;
         }
       }
-      assert(cmpxchgInst && "expected cmpxchg instruction");
-      llvm::Value *oldVal = builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/0);
-      llvm::Value *successVal =
-          builder.CreateExtractValue(cmpxchgInst, /*Idxs=*/1);
+
+      // FP HandleFPNegZero path: the OMPIRBuilder emits a multi-block
+      // structure (NaN / ±0.0 handling) with cmpxchg in predecessor
+      // blocks. Results are merged via PHI nodes in the current (exit)
+      // block: an i1 PHI for success and a bitcast of an integer PHI
+      // for the old FP value.
+      if (!oldVal) {
+        for (auto &inst : *curBB) {
+          auto *phi = dyn_cast<llvm::PHINode>(&inst);
+          if (!phi)
+            break;
+          if (phi->getType()->isIntegerTy(1))
+            successVal = phi;
+        }
+        for (auto &inst : *curBB) {
+          if (auto *bc = dyn_cast<llvm::BitCastInst>(&inst)) {
+            oldVal = bc;
+            break;
+          }
+        }
+      }
+
+      assert(oldVal && "expected cmpxchg or PHI+bitcast for compare capture");
+      assert(successVal && "expected success flag for compare capture");
       llvm::Value *newVal = builder.CreateSelect(successVal, dVal, oldVal);
       builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
     }
@@ -5686,7 +5781,11 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
 
   auto isAtomicComparePatternOp = [](Operation &op) {
     return llvm::isa<LLVM::ICmpOp, LLVM::FCmpOp, LLVM::SelectOp, LLVM::AndOp,
-                     LLVM::OrOp>(op);
+                     LLVM::OrOp, LLVM::SMaxOp, LLVM::SMinOp, LLVM::UMaxOp,
+                     LLVM::UMinOp, LLVM::MaxNumOp, LLVM::MinNumOp,
+                     mlir::arith::MaxSIOp, mlir::arith::MinSIOp,
+                     mlir::arith::MaxUIOp, mlir::arith::MinUIOp,
+                     mlir::arith::MaximumFOp, mlir::arith::MinimumFOp>(op);
   };
 
   // Pre-translate operations inside the region that compute e and d (e.g.,
@@ -5859,54 +5958,15 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
     }
     return success();
   } else {
-
-    for (Operation &op : block.getOperations()) {
-      if (auto icmpOp = dyn_cast<LLVM::ICmpOp>(op)) {
-        auto maybeOp =
-            convertICmpPredicateToAtomicCompareOp(icmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        LLVM::ICmpPredicate pred = icmpOp.getPredicate();
-        isSigned = (pred == LLVM::ICmpPredicate::slt ||
-                    pred == LLVM::ICmpPredicate::sgt ||
-                    pred == LLVM::ICmpPredicate::sle ||
-                    pred == LLVM::ICmpPredicate::sge);
-
-        // Identify which operand is the block argument (x) and which is e.
-        isXBinopExpr = (icmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? icmpOp.getOperand(1) : icmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto fcmpOp = dyn_cast<LLVM::FCmpOp>(op)) {
-        auto maybeOp =
-            convertFCmpPredicateToAtomicCompareOp(fcmpOp.getPredicate());
-        if (!maybeOp)
-          return atomicCompareOp.emitError(
-              "unsupported comparison predicate in atomic compare");
-        compareOp = *maybeOp;
-
-        isXBinopExpr = (fcmpOp.getOperand(0) == block.getArgument(0));
-        mlir::Value eOperand =
-            isXBinopExpr ? fcmpOp.getOperand(1) : fcmpOp.getOperand(0);
-        eVal = materializeValue(eOperand);
-      } else if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        if (!dVal)
-          dVal = materializeValue(selectOp.getTrueValue());
-      }
-    }
-  }
-
-  // For non-complex patterns, also extract dVal from SelectOp.
-  if (!dVal) {
-    for (Operation &op : block.getOperations()) {
-      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
-        dVal = materializeValue(selectOp.getTrueValue());
-        break;
-      }
-    }
+    AtomicComparePatternInfo patternInfo;
+    if (failed(extractAtomicComparePattern(block, materializeValue,
+                                           atomicCompareOp, patternInfo)))
+      return failure();
+    compareOp = patternInfo.compareOp;
+    eVal = patternInfo.eVal;
+    dVal = patternInfo.dVal;
+    isXBinopExpr = patternInfo.isXBinopExpr;
+    isSigned = patternInfo.isSigned;
   }
 
   if (!eVal)
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 6b13be481e0e0..cc40506b909b1 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -2161,6 +2161,30 @@ func.func @omp_atomic_compare_capture_fail_only(%v: memref<i32>, %x: memref<i32>
   return
 }
 
+// CHECK-LABEL: omp_atomic_compare_capture_real
+// CHECK-SAME: (%[[V:.*]]: memref<f32>, %[[X:.*]]: memref<f32>, %[[E:.*]]: f32, %[[D:.*]]: f32)
+func.func @omp_atomic_compare_capture_real(%v: memref<f32>, %x: memref<f32>, %e: f32, %d: f32) {
+  // CHECK: omp.atomic.capture {
+  // CHECK-NEXT: omp.atomic.compare %[[X]] : memref<f32> {
+  // CHECK-NEXT: ^bb0(%[[XVAL:.*]]: f32):
+  // CHECK-NEXT:   %[[CMP:.*]] = arith.cmpf oeq, %[[XVAL]], %[[E]] : f32
+  // CHECK-NEXT:   %[[SEL:.*]] = arith.select %[[CMP]], %[[D]], %[[XVAL]] : f32
+  // CHECK-NEXT:   omp.yield(%[[SEL]] : f32)
+  // CHECK-NEXT: }
+  // CHECK-NEXT: omp.atomic.read %[[V]] = %[[X]] : memref<f32>, memref<f32>, f32
+  // CHECK-NEXT: }
+  omp.atomic.capture {
+    omp.atomic.compare %x : memref<f32> {
+    ^bb0(%xval: f32):
+      %cmp = arith.cmpf oeq, %xval, %e : f32
+      %sel = arith.select %cmp, %d, %xval : f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : memref<f32>, memref<f32>, f32
+  }
+  return
+}
+
 // CHECK-LABEL: omp_sectionsop
 func.func @omp_sectionsop(%data_var1 : memref<i32>, %data_var2 : memref<i32>,
                      %data_var3 : memref<i32>, %redn_var : !llvm.ptr) {
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 377bd88894854..b779d20f42d0c 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2882,6 +2882,54 @@ llvm.func @omp_atomic_compare_capture_fail_only(%x : !llvm.ptr, %v : !llvm.ptr,
 }
 // -----
 
+// CHECK-LABEL: @omp_atomic_compare_capture_real
+// CHECK-SAME: (ptr %[[X:.*]], ptr %[[V:.*]], float %[[E:.*]], float %[[D:.*]])
+llvm.func @omp_atomic_compare_capture_real(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32, %d : f32) {
+  // Real compare+capture: uses HandleFPNegZero multi-block path
+  // CHECK: %[[EBITS:.*]] = bitcast float %[[E]] to i32
+  // CHECK: %[[DBITS:.*]] = bitcast float %[[D]] to i32
+  // CHECK: %[[XI:.*]] = load atomic i32, ptr %[[X]] monotonic{{.*}}
+  // CHECK: %[[XFP:.*]] = bitcast i32 %[[XI]] to float
+  // Part 1: NaN check
+  // CHECK: %[[E_NAN:.*]] = fcmp uno float %[[E]], %[[E]]
+  // CHECK: %[[X_NAN:.*]] = fcmp uno float %[[XFP]], %[[XFP]]
+  // CHECK: %[[EITHER_NAN:.*]] = or i1 %[[E_NAN]], %[[X_NAN]]
+  // CHECK: br i1 %[[EITHER_NAN]], label %[[NAN:.*]], label %[[NOTNAN:.*]]
+  // CHECK: [[NAN]]:
+  // CHECK: br label %[[EXIT:.*]]
+  // Part 2: Both-zero check
+  // CHECK: [[NOTNAN]]:
+  // CHECK: %[[XISZERO:.*]] = fcmp oeq float %[[XFP]], 0.000000e+00
+  // CHECK: %[[EISZERO:.*]] = fcmp oeq float %[[E]], 0.000000e+00
+  // CHECK: %[[BOTHZERO:.*]] = and i1 %[[XISZERO]], %[[EISZERO]]
+  // CHECK: br i1 %[[BOTHZERO]], label %[[ZERO:.*]], label %[[NORMAL:.*]]
+  // CHECK: [[ZERO]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[XI]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Part 3: Normal cmpxchg
+  // CHECK: [[NORMAL]]:
+  // CHECK: cmpxchg ptr %[[X]], i32 %[[EBITS]], i32 %[[DBITS]] monotonic monotonic{{.*}}
+  // CHECK: br label %[[EXIT]]
+  // Exit: select v = (success ? d : old_x)
+  // CHECK: [[EXIT]]:
+  // CHECK: %[[OLD:.*]] = phi i32 {{.*}}
+  // CHECK: %[[OK:.*]] = phi i1 {{.*}}
+  // CHECK: %[[OLDFP:.*]] = bitcast i32 %[[OLD]] to float
+  // CHECK: %[[VVAL:.*]] = select i1 %[[OK]], float %[[D]], float %[[OLDFP]]
+  // CHECK: store float %[[VVAL]], ptr %[[V]]{{.*}}
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "oeq" %xval, %e : f32
+      %sel = llvm.select %cmp, %d, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+// -----
+
 // CHECK-LABEL: @omp_sections_empty
 llvm.func @omp_sections_empty() -> () {
   omp.sections {

>From d42a6412d158c581c76cadf7d4626b4235e60725 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 07:25:53 -0500
Subject: [PATCH 08/11] Handling a crash in logical "omp atomic
 compare-capture" inside "omp parallel". Adding some extra tests.

---
 flang/lib/Semantics/check-omp-atomic.cpp      |  19 ++-
 .../Integration/OpenMP/atomic-compare.f90     |  27 ++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 117 ++++++++++++++++++
 .../test/Semantics/OpenMP/atomic-compare.f90  |  12 ++
 4 files changed, 174 insertions(+), 1 deletion(-)

diff --git a/flang/lib/Semantics/check-omp-atomic.cpp b/flang/lib/Semantics/check-omp-atomic.cpp
index ec307be469982..88997df26e4d0 100644
--- a/flang/lib/Semantics/check-omp-atomic.cpp
+++ b/flang/lib/Semantics/check-omp-atomic.cpp
@@ -419,6 +419,22 @@ static bool IsMaybeAtomicWrite(const evaluate::Assignment &assign) {
   return HasStorageOverlap(assign.lhs, assign.rhs) == nullptr;
 }
 
+// Determine whether two designators refer to the same atomic variable.
+// The comparison tolerates OpenMP host-association: a shared variable
+// referenced inside a parallel region may be designated through a
+// host-associated symbol in one statement and through its ultimate symbol
+// in another, even though both denote the same storage. Comparing the
+// (ultimate) symbol vectors together with the source form distinguishes
+// different variables and different constant subscripts (e.g. a(1) vs a(2))
+// while treating host-associated and ultimate references as equal.
+static bool IsSameAtomicVariable(const SomeExpr &x, const SomeExpr &y) {
+  if (x == y) {
+    return true;
+  }
+  return evaluate::GetSymbolVector(x) == evaluate::GetSymbolVector(y) &&
+      x.AsFortran() == y.AsFortran();
+}
+
 static void SetExpr(parser::TypedExpr &expr, MaybeExpr value) {
   if (value) {
     expr.Reset(new evaluate::GenericExprWrapper(std::move(value)),
@@ -822,7 +838,8 @@ void OmpStructureChecker::CheckAtomicCaptureAssignment(
     CheckAtomicVariable(
         atom, rsrc, /*checkTypeOnPointer=*/!IsPointerAssignment(capture));
     // This part should have been checked prior to calling this function.
-    assert(*GetConvertInput(capture.rhs) == atom &&
+    assert(GetConvertInput(capture.rhs) &&
+        IsSameAtomicVariable(*GetConvertInput(capture.rhs), atom) &&
         "This cannot be a capture assignment");
     CheckStorageOverlap(atom, {cap}, source);
   }
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index f30349c42e33e..e98b25e4b4539 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -394,3 +394,30 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
   !$omp end atomic
 end
 
+! Logical compare+capture inside a parallel region.
+!CHECK-LABEL: define void @atomic_compare_capture_logical_parallel_(
+!CHECK-SAME: ptr noalias %[[X:.*]])
+!CHECK: %[[CGEP:.*]] = getelementptr { ptr }, ptr %structArg, i32 0, i32 0
+!CHECK: store ptr %[[X]], ptr %[[CGEP]]
+!CHECK: call void {{.*}}@__kmpc_fork_call{{.*}}@atomic_compare_capture_logical_parallel_..omp_par
+!CHECK-LABEL: define internal void @atomic_compare_capture_logical_parallel_..omp_par(
+!CHECK-SAME: ptr noalias %{{.*}}, ptr noalias %{{.*}}, ptr %[[STRUCTARG:.*]])
+!CHECK: %[[GEP:.*]] = getelementptr { ptr }, ptr %[[STRUCTARG]], i32 0, i32 0
+!CHECK: %[[SHARED:.*]] = load ptr, ptr %[[GEP]]
+!CHECK: %[[RES:.*]] = cmpxchg ptr %[[SHARED]], i32 %{{.*}}, i32 %{{.*}} monotonic monotonic{{.*}}
+!CHECK: %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+!CHECK: store i32 %[[OLD]], ptr %{{.*}}
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
+
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index 1219d05296e4d..ccc44a370be79 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -305,6 +305,83 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Compare-capture with an explicit seq_cst memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
+! CHECK:         omp.atomic.capture memory_order(seq_cst) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_seq_cst(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture seq_cst
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acquire memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acquire(
+! CHECK:         omp.atomic.capture memory_order(acquire) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acquire(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acquire
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit release memory order (prefix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_release(
+! CHECK:         omp.atomic.capture memory_order(release) {
+! CHECK:           omp.atomic.read %[[V_DECL:.*]]#0 = %[[X_DECL:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:         }
+subroutine atomic_compare_capture_release(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture release
+    v = x
+    if (x .eq. e) x = d
+  !$omp end atomic
+end
+
+! Compare-capture with an explicit acq_rel memory order (postfix read form).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_acq_rel(
+! CHECK:         omp.atomic.capture memory_order(acq_rel) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_acq_rel(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture acq_rel
+    if (x .eq. e) x = d
+    v = x
+  !$omp end atomic
+end
+
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "e"},
@@ -335,3 +412,43 @@ subroutine atomic_compare_capture_logical(x, e, d, v)
     v = x
   !$omp end atomic
 end
+
+! Logical compare-capture inside a parallel region. The shared atom 'x' is
+! host-associated, and the capture reads it while the conditional update writes
+! it. Verify the atom is converted to i32 once (above the atomic region) and
+! that the capture reads then compares the same converted reference.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_logical_parallel(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<!fir.logical<4>> {fir.bindc_name = "x"})
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}Ex"
+! CHECK:         omp.parallel private(
+! CHECK:           %[[OLD_DECL:.*]]:2 = hlfir.declare %{{.*}}Eold_value"
+! CHECK:           %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Eexpected"
+! CHECK:           %[[ELOAD:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           %[[XCONV:.*]] = fir.convert %[[X_DECL]]#0 : (!fir.ref<!fir.logical<4>>) -> !fir.ref<i32>
+! CHECK:           %[[ECONV:.*]] = fir.convert %[[ELOAD]] : (!fir.logical<4>) -> i32
+! CHECK:           omp.atomic.capture memory_order(relaxed) {
+! CHECK:             omp.atomic.read %[[ATOMVAL:.*]] = %[[XCONV]] : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:             omp.atomic.compare %[[XCONV]] : !fir.ref<i32> {
+! CHECK:             ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:               %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[ECONV]] : i32
+! CHECK:               %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:               omp.yield(%[[SEL]] : i32)
+! CHECK:             }
+! CHECK:           }
+! CHECK:           %[[RES:.*]] = fir.load %[[ATOMVAL]] : !fir.ref<i32>
+! CHECK:           %[[RESCONV:.*]] = fir.convert %[[RES]] : (i32) -> !fir.logical<4>
+! CHECK:           fir.store %[[RESCONV]] to %[[OLD_DECL]]#0 : !fir.ref<!fir.logical<4>>
+! CHECK:           omp.terminator
+subroutine atomic_compare_capture_logical_parallel(x)
+  logical :: x, expected, desired, old_value
+  !$omp parallel private(old_value, expected, desired)
+    expected = .true.
+    desired  = .false.
+    !$omp atomic compare capture
+      old_value = x
+      if (x .eqv. expected) then
+        x = desired
+      end if
+    !$omp end atomic
+  !$omp end parallel
+end
diff --git a/flang/test/Semantics/OpenMP/atomic-compare.f90 b/flang/test/Semantics/OpenMP/atomic-compare.f90
index 0e53729c2f02a..5cee2359f52be 100644
--- a/flang/test/Semantics/OpenMP/atomic-compare.f90
+++ b/flang/test/Semantics/OpenMP/atomic-compare.f90
@@ -9,6 +9,7 @@
 
   real a, b, c
   logical :: r, s
+  logical :: lx, le, ld, lo
   a = 1.0
   b = 2.0
   c = 3.0
@@ -58,6 +59,17 @@
   if (r) b = c
   !$omp end atomic
 
+  ! Compare-capture with a LOGICAL atom using .eqv. inside a parallel region.
+  ! Regression: the host-associated atom (from the update) must be recognized
+  ! as the same variable as the captured one, even though OpenMP resolves them
+  ! through different (but ultimately identical) symbols.
+  !$omp atomic compare capture
+  lo = lx
+  if (lx .eqv. le) then
+    lx = ld
+  end if
+  !$omp end atomic
+
   ! Check for error conditions:
   !ERROR: At most one SEQ_CST clause can appear on the ATOMIC directive
   !$omp atomic seq_cst seq_cst compare

>From b6f101ded4b5847a466a27c6a37185a3203b6ebc Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 10 Jul 2026 00:50:42 -0500
Subject: [PATCH 09/11] Added exhaustive tests covering all atomic compare
 capture combinations Handling changes for atomic compare capture were not
 covered earlier. Completed runs of llvm-test-suite and Fujitsu flang
 test-suite.

---
 flang/lib/Lower/OpenMP/Atomic.cpp             |  14 +
 flang/test/Fir/atomic-compare-capture.fir     | 343 ++++++++++++++
 .../OpenMP/atomic-compare-capture.f90         | 443 ++++++++++++++++++
 .../Todo/atomic-compare-capture-result.f90    |  20 +
 .../Lower/OpenMP/Todo/atomic-compare-fail.f90 |   8 -
 .../Lower/OpenMP/atomic-compare-capture.f90   | 356 ++++++++++++++
 flang/test/Lower/OpenMP/atomic-compare.f90    | 116 +++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 398 ++++++++++++----
 .../LLVMIR/openmp-atomic-compare-capture.mlir | 407 ++++++++++++++++
 9 files changed, 2007 insertions(+), 98 deletions(-)
 create mode 100644 flang/test/Fir/atomic-compare-capture.fir
 create mode 100644 flang/test/Integration/OpenMP/atomic-compare-capture.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
 create mode 100644 flang/test/Lower/OpenMP/atomic-compare-capture.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir

diff --git a/flang/lib/Lower/OpenMP/Atomic.cpp b/flang/lib/Lower/OpenMP/Atomic.cpp
index 3811ea3d40bbd..ca37eb4a73375 100644
--- a/flang/lib/Lower/OpenMP/Atomic.cpp
+++ b/flang/lib/Lower/OpenMP/Atomic.cpp
@@ -606,6 +606,20 @@ void Fortran::lower::omp::lowerAtomic(
       TODO(loc, "Compound clauses of OpenMP ATOMIC COMPARE");
     }
 
+    // The comparison-result forms of atomic compare
+    //     (e.g. `r = x == e;
+    //           if (r) x = d`)
+    // are accepted by semantics, but the assignment to `r` is
+    // not captured by the atomic analysis and would be silently dropped.
+    // Detect the extra assignment here and emit an explicit TODO instead of
+    // generating incorrect code.
+    if (!construct.IsCapture()) {
+      const parser::Block &body = std::get<parser::Block>(construct.t);
+      if (body.size() > 1)
+        TODO(loc, "atomic compare capturing the comparison result "
+                  "(e.g. 'r = x == e')");
+    }
+
     common::RelationalOperator relOpr = common::RelationalOperator::EQ;
     std::optional<semantics::SomeExpr> expectedExprStorage;
     bool isUnsigned = false;
diff --git a/flang/test/Fir/atomic-compare-capture.fir b/flang/test/Fir/atomic-compare-capture.fir
new file mode 100644
index 0000000000000..65264fbc688a8
--- /dev/null
+++ b/flang/test/Fir/atomic-compare-capture.fir
@@ -0,0 +1,343 @@
+// RUN: fir-opt --split-input-file --cfg-conversion --fir-to-llvm-ir="target=aarch64-unknown-linux-gnu" %s | FileCheck %s
+
+// Test FIR-to-LLVM-dialect conversion of OpenMP atomic compare-capture: the
+// !fir.ref operands become !llvm.ptr and the compare region's arith ops become
+// their llvm-dialect equivalents, while the omp.atomic.capture / .compare /
+// .read structure is preserved for the later MLIR-to-LLVM-IR translation.
+
+//
+// Integer postfix compare-capture: {compare, read}
+func.func @cc_int_postfix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> i32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> i32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: i32):
+// CHECK:        %[[CMP:.*]] = llvm.icmp "eq" %[[XVAL]], %[[EVAL]] : i32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, i32
+// CHECK:        omp.yield(%[[SEL]] : i32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Integer prefix compare-capture: {read, compare}
+func.func @cc_int_prefix(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:        llvm.select
+// CHECK:        omp.yield
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Integer fail-only compare-capture: {compare, read} with fail_only attribute.
+func.func @cc_int_failonly(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Real postfix compare-capture: !fir.ref<f32> -> !llvm.ptr, arith.cmpf -> llvm.fcmp.
+func.func @cc_real_postfix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> f32
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> f32
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: f32):
+// CHECK:        %[[CMP:.*]] = llvm.fcmp "oeq" %[[XVAL]], %[[EVAL]] : f32
+// CHECK:        %[[SEL:.*]] = llvm.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i1, f32
+// CHECK:        omp.yield(%[[SEL]] : f32)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Real prefix compare-capture.
+func.func @cc_real_prefix(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Real fail-only compare-capture.
+func.func @cc_real_failonly(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %d: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  %dval = fir.load %d : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf oeq, %xval, %eval : f32
+      %s = arith.select %c, %dval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    } {fail_only}
+
+// -----
+
+// Complex postfix compare-capture: !fir.ref<complex<f32>> -> !llvm.ptr and the
+// fir.cmpc is decomposed into per-field llvm.fcmp + llvm.and over the struct.
+func.func @cc_complex_postfix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_postfix(
+// CHECK-SAME:  %[[X:.*]]: !llvm.ptr, %[[E:.*]]: !llvm.ptr, %[[D:.*]]: !llvm.ptr, %[[V:.*]]: !llvm.ptr) {
+// CHECK:    %[[EVAL:.*]] = llvm.load %[[E]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    %[[DVAL:.*]] = llvm.load %[[D]] : !llvm.ptr -> !llvm.struct<(f32, f32)>
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %[[X]] : !llvm.ptr {
+// CHECK:      ^bb0(%[[XVAL:.*]]: !llvm.struct<(f32, f32)>):
+// CHECK:        llvm.extractvalue %[[XVAL]][0]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.extractvalue %[[XVAL]][1]
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        %[[AND:.*]] = llvm.and
+// CHECK:        %[[SEL:.*]] = llvm.select %[[AND]], %[[DVAL]], %[[XVAL]] : i1, !llvm.struct<(f32, f32)>
+// CHECK:        omp.yield(%[[SEL]] : !llvm.struct<(f32, f32)>)
+// CHECK:      }
+// CHECK:      omp.atomic.read %[[V]] = %[[X]] : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    }
+
+// -----
+
+// Complex prefix compare-capture.
+func.func @cc_complex_prefix(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_prefix(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:    }
+
+// -----
+
+// Complex fail-only compare-capture.
+func.func @cc_complex_failonly(%x: !fir.ref<complex<f32>>, %e: !fir.ref<complex<f32>>, %d: !fir.ref<complex<f32>>, %v: !fir.ref<complex<f32>>) {
+  %eval = fir.load %e : !fir.ref<complex<f32>>
+  %dval = fir.load %d : !fir.ref<complex<f32>>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<complex<f32>> {
+    ^bb0(%xval: complex<f32>):
+      %c = fir.cmpc "oeq", %xval, %eval : complex<f32>
+      %s = arith.select %c, %dval, %xval : complex<f32>
+      omp.yield(%s : complex<f32>)
+    }
+    omp.atomic.read %v = %x : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+  } {fail_only}
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_complex_failonly(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "oeq"
+// CHECK:        llvm.and
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+// CHECK:    } {fail_only}
+
+// -----
+
+// Integer min compare-capture (x > e): the compare region uses arith.cmpi sgt.
+func.func @cc_int_min(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi sgt, %xval, %eval : i32
+      %s = arith.select %c, %eval, %xval : i32
+      omp.yield(%s : i32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_int_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "sgt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:    }
+
+// -----
+
+// Real min compare-capture (x > e): the compare region uses arith.cmpf ogt.
+func.func @cc_real_min(%x: !fir.ref<f32>, %e: !fir.ref<f32>, %v: !fir.ref<f32>) {
+  %eval = fir.load %e : !fir.ref<f32>
+  omp.atomic.capture {
+    omp.atomic.compare %x : !fir.ref<f32> {
+    ^bb0(%xval: f32):
+      %c = arith.cmpf ogt, %xval, %eval : f32
+      %s = arith.select %c, %eval, %xval : f32
+      omp.yield(%s : f32)
+    }
+    omp.atomic.read %v = %x : !fir.ref<f32>, !fir.ref<f32>, f32
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_real_min(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.fcmp "ogt"
+// CHECK:      }
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, f32
+// CHECK:    }
+
+// -----
+
+// Weak clause: preserved through FIR-to-LLVM conversion.
+func.func @cc_weak(%x: !fir.ref<i32>, %e: !fir.ref<i32>, %d: !fir.ref<i32>, %v: !fir.ref<i32>) {
+  %eval = fir.load %e : !fir.ref<i32>
+  %dval = fir.load %d : !fir.ref<i32>
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !fir.ref<i32>, !fir.ref<i32>, i32
+    omp.atomic.compare %x : !fir.ref<i32> {
+    ^bb0(%xval: i32):
+      %c = arith.cmpi eq, %xval, %eval : i32
+      %s = arith.select %c, %dval, %xval : i32
+      omp.yield(%s : i32)
+    } {weak}
+  }
+  return
+}
+
+// CHECK-LABEL: llvm.func @cc_weak(
+// CHECK:    omp.atomic.capture {
+// CHECK:      omp.atomic.read %{{.*}} = %{{.*}} : !llvm.ptr, !llvm.ptr, i32
+// CHECK:      omp.atomic.compare %{{.*}} : !llvm.ptr {
+// CHECK:        llvm.icmp "eq"
+// CHECK:      } {weak}
+// CHECK:    }
diff --git a/flang/test/Integration/OpenMP/atomic-compare-capture.f90 b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..fe97e7b12558b
--- /dev/null
+++ b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,443 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=FIR
+!RUN: %if x86-registered-target %{ %flang_fc1 -triple x86_64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+!RUN: %if aarch64-registered-target %{ %flang_fc1 -triple aarch64-unknown-linux-gnu -emit-llvm -fopenmp -fopenmp-version=51 %s -o - | FileCheck %s --check-prefix=LLVM %}
+
+! OpenMP atomic compare-capture tracked through every stage of the compiler
+! (HLFIR, FIR/omp dialect, LLVM IR) for every operand type: integer, real,
+! logical and complex under ==, plus integer and real under min/max (<, >).
+! Equality lowers to a cmpxchg; min/max lowers to an atomicrmw.
+
+! ===========================================================================
+! integer ==  (postfix): cmpxchg + select(new value) + store into v
+! HLFIR-LABEL: func.func @_QPcc_integer(
+! HLFIR:         %[[X:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           }
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real ==  (postfix): equality on floats uses the HandleFPNegZero cmpxchg path.
+! HLFIR-LABEL: func.func @_QPcc_real(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf oeq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf oeq
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! logical ==  (postfix): the logical atom is converted to i32, so the omp op
+! operates on !fir.ref<i32> and lowers to an i32 cmpxchg.
+! HLFIR-LABEL: func.func @_QPcc_logical(
+! HLFIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! HLFIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_logical(
+! FIR:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_(
+! LLVM:         cmpxchg ptr %{{.*}}, i32
+! LLVM:         store i32 %{{.*}}, ptr
+subroutine cc_logical(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! complex ==  (postfix): fir.cmpc region; lowers to an i64 (bitcast) cmpxchg
+! and the captured new value is a { float, float } struct.
+! HLFIR-LABEL: func.func @_QPcc_complex(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:           fir.cmpc "oeq"
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_complex(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<complex<f32>> {
+! FIR:           fir.cmpc "oeq"
+! FIR:         }
+! LLVM-LABEL: define void @cc_complex_(
+! LLVM:         cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer min (x > e, postfix): atomicrmw min, v gets the new value smin(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi sgt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi sgt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! integer max (x < e, postfix): atomicrmw max, v gets the new value smax(old,e).
+! HLFIR-LABEL: func.func @_QPcc_integer_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi slt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_integer_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:           arith.cmpi slt
+! FIR:         }
+! LLVM-LABEL: define void @cc_integer_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_integer_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real min (x > e, postfix): atomicrmw fmin, v gets minnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_min(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf ogt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_min(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf ogt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_min_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! real max (x < e, postfix): atomicrmw fmax, v gets maxnum(old, e).
+! HLFIR-LABEL: func.func @_QPcc_real_max(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:           arith.cmpf olt
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_max(
+! FIR:         omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! FIR:           arith.cmpf olt
+! FIR:         }
+! LLVM-LABEL: define void @cc_real_max_(
+! LLVM:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+! LLVM:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+! LLVM:         store float %[[NEW]], ptr
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end
+
+! The remaining cases exercise the different *capture-statement* forms allowed
+! for atomic compare-capture (all with an integer == atom).
+
+! ===========================================================================
+! Prefix form  {v = x; cond-update-stmt}:  the read precedes the update, so v
+! captures the OLD value of x - the read is emitted first and v is a direct
+! store of the extracted old value (no select).
+! HLFIR-LABEL: func.func @_QPcc_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_prefix(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:         }
+! LLVM-LABEL: define void @cc_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Fail-only form  {if(x == e){x = d;} else {v = x;}}:  v is captured only when
+! the compare fails; the capture carries the fail_only attribute and lowers to
+! a conditional store of the old value.
+! HLFIR-LABEL: func.func @_QPcc_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           }
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_failonly(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %[[EXIT:.*]], label %[[CONT:.*]]
+! LLVM:         [[CONT]]:
+! LLVM:         store i32 %[[OLD]], ptr
+subroutine cc_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! Reversed comparison operand order  {if(e == x){x = d;} ; v = x}:  the front
+! end normalises 'e == x' to the same compare-capture as 'x == e', so this
+! accepts the alternate spelling and lowers identically (cmpxchg + select).
+! HLFIR-LABEL: func.func @_QPcc_eq_reversed(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:           arith.cmpi eq
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_eq_reversed(
+! LLVM-LABEL: define void @cc_eq_reversed_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end
+
+! ===========================================================================
+! The prefix and fail-only forms for the remaining operand types (real,
+! logical, complex), to confirm each type honours every capture form.
+
+! real, prefix: v gets the old value of x (float).
+! HLFIR-LABEL: func.func @_QPcc_real_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_real_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<f32> {
+! LLVM-LABEL: define void @cc_real_prefix_(
+! LLVM:         cmpxchg ptr
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! real, fail-only: conditional store of the old value (float).
+! HLFIR-LABEL: func.func @_QPcc_real_failonly(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_real_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_real_failonly_(
+! LLVM:         cmpxchg ptr
+! LLVM:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+! LLVM:         store float %{{.*}}, ptr
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! logical, prefix: v gets the old value (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR-LABEL: func.func @_QPcc_logical_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! LLVM-LABEL: define void @cc_logical_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end
+
+! logical, fail-only: conditional store (via the i32 atom).
+! HLFIR-LABEL: func.func @_QPcc_logical_failonly(
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_logical_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_logical_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i32
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         br i1 %[[OK]], label %{{.*}}, label %{{.*}}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! complex, prefix: v gets the old complex value (struct), no select.
+! HLFIR-LABEL: func.func @_QPcc_complex_prefix(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! HLFIR:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! FIR-LABEL: func.func @_QPcc_complex_prefix(
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! LLVM-LABEL: define void @cc_complex_prefix_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end
+
+! complex, fail-only: conditional store of the old complex value.
+! HLFIR-LABEL: func.func @_QPcc_complex_failonly(
+! HLFIR:         omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! HLFIR:         } {fail_only}
+! FIR-LABEL: func.func @_QPcc_complex_failonly(
+! FIR:         } {fail_only}
+! LLVM-LABEL: define void @cc_complex_failonly_(
+! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
+! LLVM:         %[[OK:.*]] = extractvalue { i64, i1 } %[[RES]], 1
+! LLVM:         %[[FAILED:.*]] = xor i1 %[[OK]], true
+! LLVM:         br i1 %[[FAILED]], label %{{.*}}, label %{{.*}}
+! LLVM:         store { float, float } %{{.*}}, ptr
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end
+
+! ===========================================================================
+! weak clause (postfix): cmpxchg gets the 'weak' modifier.
+! HLFIR-LABEL: func.func @_QPcc_weak(
+! HLFIR:         omp.atomic.capture memory_order(relaxed) {
+! HLFIR:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! HLFIR:             arith.cmpi eq
+! HLFIR:           } {weak}
+! HLFIR:           omp.atomic.read %{{.*}} = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! HLFIR:         }
+! FIR-LABEL: func.func @_QPcc_weak(
+! FIR:         omp.atomic.capture memory_order(relaxed) {
+! FIR:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! FIR:             arith.cmpi eq
+! FIR:           } {weak}
+! FIR:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! FIR:         }
+! LLVM-LABEL: define void @cc_weak_(
+! LLVM:         %[[RES:.*]] = cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %[[D:.*]] monotonic monotonic
+! LLVM:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RES]], 0
+! LLVM:         %[[OK:.*]] = extractvalue { i32, i1 } %[[RES]], 1
+! LLVM:         %[[NEW:.*]] = select i1 %[[OK]], i32 %[[D]], i32 %[[OLD]]
+! LLVM:         store i32 %[[NEW]], ptr
+subroutine cc_weak(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
new file mode 100644
index 0000000000000..ce6f0a3d6454b
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-capture-result.f90
@@ -0,0 +1,20 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+! The comparison-result form of atomic compare captures the boolean outcome of
+! the comparison into `r`:
+!   r = x == e
+!   if (r) x = d
+! This is accepted by semantics, but the atomic analysis does not record the
+! assignment to `r`, so lowering would silently drop it and never write `r`.
+! Ensure it is diagnosed as "not yet implemented" instead of producing code that
+! leaves `r` unwritten.
+
+! CHECK: not yet implemented: atomic compare capturing the comparison result
+subroutine f(x, e, d, r)
+  integer :: x, e, d
+  logical :: r
+  !$omp atomic compare
+  r = x == e
+  if (r) x = d
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90 b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
index 27c7b1fb1a432..7165bdb2fc122 100644
--- a/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
+++ b/flang/test/Lower/OpenMP/Todo/atomic-compare-fail.f90
@@ -5,20 +5,12 @@ program p
   integer :: x
   integer :: r
   integer :: d
-  integer :: v
   !$omp atomic compare fail(relaxed)
   if (x .eq. 0) then
      x = 2
   end if
   !$omp end atomic
 
-  !$omp atomic compare capture
-  v = x
-  if (x > r) then
-     x = d
-  end if
-  !$omp end atomic
-
   !$omp atomic compare fail(relaxed)
   if (x > r) then
      x = d
diff --git a/flang/test/Lower/OpenMP/atomic-compare-capture.f90 b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
new file mode 100644
index 0000000000000..93fbf2d4bd034
--- /dev/null
+++ b/flang/test/Lower/OpenMP/atomic-compare-capture.f90
@@ -0,0 +1,356 @@
+! This test checks lowering of atomic compare capture constructs to HLFIR.
+! RUN: bbc %openmp_flags -fopenmp-version=51 -emit-hlfir %s -o - | FileCheck %s
+! RUN: %flang_fc1 -emit-hlfir %openmp_flags -fopenmp-version=51 %s -o - | FileCheck %s
+
+! ---------------------------------------------------------------------------
+! integer == postfix: { if (x==e) x=d; v=x }
+! CHECK-LABEL: func.func @_QPcc_int_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[D:.*]]: !fir.ref<i32> {fir.bindc_name = "d"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[D_DECL:.*]]:2 = hlfir.declare %[[D]] {{.*}}
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[D_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi eq, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == prefix: { v=x; if (x==e) x=d }
+! CHECK-LABEL: func.func @_QPcc_int_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:         }
+subroutine cc_int_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == fail-only: { if (x==e) x=d; else v=x }
+! CHECK-LABEL: func.func @_QPcc_int_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             arith.cmpi eq, %[[XVAL]], %{{.*}} : i32
+! CHECK:             omp.yield
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %{{.*}}#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_int_failonly(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer == reversed (e==x): same lowering as x==e
+! CHECK-LABEL: func.func @_QPcc_int_eq_reversed(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_eq_reversed(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture
+  if (e == x) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == postfix
+! CHECK-LABEL: func.func @_QPcc_real_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_postfix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == prefix
+! CHECK-LABEL: func.func @_QPcc_real_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_real_prefix(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real == fail-only
+! CHECK-LABEL: func.func @_QPcc_real_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf oeq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         } {fail_only}
+subroutine cc_real_failonly(x, e, d, v)
+  real :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. postfix
+! CHECK-LABEL: func.func @_QPcc_logical_postfix(
+! CHECK:         fir.convert %{{.*}} : (!fir.logical<4>) -> i32
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_logical_postfix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. prefix
+! CHECK-LABEL: func.func @_QPcc_logical_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:         }
+subroutine cc_logical_prefix(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x .eqv. e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! logical .eqv. fail-only
+! CHECK-LABEL: func.func @_QPcc_logical_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}} : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine cc_logical_failonly(x, e, d, v)
+  logical :: x, e, d, v
+  !$omp atomic compare capture
+  if (x .eqv. e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == postfix
+! CHECK-LABEL: func.func @_QPcc_complex_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine cc_complex_postfix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == prefix
+! CHECK-LABEL: func.func @_QPcc_complex_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}} = %[[X:.*]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:         }
+subroutine cc_complex_prefix(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! complex == fail-only
+! CHECK-LABEL: func.func @_QPcc_complex_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<complex<f32>> {
+! CHECK:             fir.cmpc "oeq"
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         } {fail_only}
+subroutine cc_complex_failonly(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+  if (x == e) then
+    x = d
+  else
+    v = x
+  end if
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi sgt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_min(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! integer max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_int_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi slt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_int_max(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real min (x > e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_min(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf ogt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x > e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! real max (x < e) postfix
+! CHECK-LABEL: func.func @_QPcc_real_max(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<f32> {
+! CHECK:             arith.cmpf olt
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine cc_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+  if (x < e) x = e
+  v = x
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (prefix form)
+! CHECK-LABEL: func.func @_QPcc_weak_prefix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X:.*]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:           omp.atomic.compare %[[X]]#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:         }
+subroutine cc_weak_prefix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  v = x
+  if (x == e) x = d
+  !$omp end atomic
+end subroutine
+
+! ---------------------------------------------------------------------------
+! weak clause (postfix form)
+! CHECK-LABEL: func.func @_QPcc_weak_postfix(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %{{.*}}#0 : !fir.ref<i32> {
+! CHECK:             arith.cmpi eq
+! CHECK:           } {weak}
+! CHECK:           omp.atomic.read %{{.*}} : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine cc_weak_postfix(x, e, d, v)
+  integer :: x, e, d, v
+  !$omp atomic compare capture weak
+  if (x == e) x = d
+  v = x
+  !$omp end atomic
+end subroutine
diff --git a/flang/test/Lower/OpenMP/atomic-compare.f90 b/flang/test/Lower/OpenMP/atomic-compare.f90
index ccc44a370be79..83e6f81ca00e3 100644
--- a/flang/test/Lower/OpenMP/atomic-compare.f90
+++ b/flang/test/Lower/OpenMP/atomic-compare.f90
@@ -218,6 +218,101 @@ subroutine atomic_compare_capture_int_gt(x, e, d, v)
   !$omp end atomic
 end
 
+! Min/max (postfix) compare-capture: update first, capture the new value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_postfix(
+! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
+! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
+! CHECK-SAME:    %[[V:.*]]: !fir.ref<i32> {fir.bindc_name = "v"})
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %[[E]] {{.*}}
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %[[V]] {{.*}}
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %[[X]] {{.*}}
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<i32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %[[EVAL]] : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         }
+subroutine atomic_compare_capture_min_postfix(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! Min/max fail-only compare-capture: v is captured in the else branch (when the
+! comparison, and hence the update, does not happen).
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_min_failonly(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<i32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: i32):
+! CHECK:             %[[CMP:.*]] = arith.cmpi sgt, %[[XVAL]], %{{.*}} : i32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : i32
+! CHECK:             omp.yield(%[[SEL]] : i32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<i32>, !fir.ref<i32>, i32
+! CHECK:         } {fail_only}
+subroutine atomic_compare_capture_min_failonly(x, e, v)
+  integer :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) then
+      x = e
+    else
+      v = x
+    end if
+  !$omp end atomic
+end
+
+! REAL min (postfix): if (x > e) x = e ; v = x -> compare region uses arith.cmpf.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_min(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[V_DECL:.*]]:2 = hlfir.declare %{{.*}}Ev"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         %[[EVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         %[[DVAL:.*]] = fir.load %[[E_DECL]]#0 : !fir.ref<f32>
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf ogt, %[[XVAL]], %[[EVAL]] fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %[[DVAL]], %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %[[V_DECL]]#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_min(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x > e) x = e
+    v = x
+  !$omp end atomic
+end
+
+! REAL max (postfix): if (x < e) x = e ; v = x.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_real_max(
+! CHECK:         %[[E_DECL:.*]]:2 = hlfir.declare %{{.*}}Ee"
+! CHECK:         %[[X_DECL:.*]]:2 = hlfir.declare %{{.*}}Ex"
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL]]#0 : !fir.ref<f32> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: f32):
+! CHECK:             %[[CMP:.*]] = arith.cmpf olt, %[[XVAL]], %{{.*}} fastmath<contract> : f32
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : f32
+! CHECK:             omp.yield(%[[SEL]] : f32)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<f32>, !fir.ref<f32>, f32
+! CHECK:         }
+subroutine atomic_compare_capture_real_max(x, e, v)
+  real :: x, e, v
+  !$omp atomic compare capture
+    if (x < e) x = e
+    v = x
+  !$omp end atomic
+end
+
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_postfix(
 ! CHECK-SAME:    %[[X:.*]]: !fir.ref<i32> {fir.bindc_name = "x"},
 ! CHECK-SAME:    %[[E:.*]]: !fir.ref<i32> {fir.bindc_name = "e"},
@@ -305,6 +400,27 @@ subroutine atomic_compare_capture_real(x, e, d, v)
   !$omp end atomic
 end
 
+! Complex compare-capture (postfix form): the compare region decomposes into
+! per-field comparisons combined with fir.cmpc/arith and the desired value is
+! selected as a whole complex value.
+! CHECK-LABEL: func.func @_QPatomic_compare_capture_complex(
+! CHECK:         omp.atomic.capture memory_order(relaxed) {
+! CHECK:           omp.atomic.compare %[[X_DECL:.*]]#0 : !fir.ref<complex<f32>> {
+! CHECK:           ^bb0(%[[XVAL:.*]]: complex<f32>):
+! CHECK:             %[[CMP:.*]] = fir.cmpc "oeq", %[[XVAL]], %{{.*}} : complex<f32>
+! CHECK:             %[[SEL:.*]] = arith.select %[[CMP]], %{{.*}}, %[[XVAL]] : complex<f32>
+! CHECK:             omp.yield(%[[SEL]] : complex<f32>)
+! CHECK:           }
+! CHECK:           omp.atomic.read %{{.*}}#0 = %[[X_DECL]]#0 : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
+! CHECK:         }
+subroutine atomic_compare_capture_complex(x, e, d, v)
+  complex :: x, e, d, v
+  !$omp atomic compare capture
+    if (x == e) x = d
+    v = x
+  !$omp end atomic
+end
+
 ! Compare-capture with an explicit seq_cst memory order (prefix read form).
 ! CHECK-LABEL: func.func @_QPatomic_compare_capture_seq_cst(
 ! CHECK:         omp.atomic.capture memory_order(seq_cst) {
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 8e49571cd2461..060e563ecefa8 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5370,6 +5370,99 @@ convertFCmpPredicateToAtomicCompareOp(LLVM::FCmpPredicate predicate) {
   }
 }
 
+/// Result of matching the decomposed complex equality pattern inside an atomic
+/// compare region.
+struct ComplexComparePattern {
+  bool isComplex = false;
+  bool isNE = false;         // `or` of the field compares => NE (unsupported).
+  mlir::Value eAggregate;    // The complex expected value (`e`).
+  bool isXBinopExpr = false; // True if x is the first fcmp operand.
+};
+
+/// Detect a decomposed complex equality comparison in an atomic compare region:
+///   %re_x = llvm.extractvalue %xval[0]
+///   %re_e = llvm.extractvalue %eStruct[0]
+///   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
+///   %im_x = llvm.extractvalue %xval[1]
+///   %im_e = llvm.extractvalue %eStruct[1]
+///   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
+///   %cmp = llvm.and %cmp_re, %cmp_im   (llvm.or would be NE)
+/// It is recognised by an and/or whose operands are both fcmps operating on
+/// extractvalues, one chain rooted at the block argument (x) and the other at
+/// the expected complex value (e).
+static ComplexComparePattern detectComplexCompareEq(Block &block) {
+  ComplexComparePattern result;
+  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
+    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
+      return extractOp.getContainer();
+    return nullptr;
+  };
+  for (Operation &op : block.getOperations()) {
+    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
+      continue;
+    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
+    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
+    if (!lhsFcmp || !rhsFcmp)
+      continue;
+    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
+    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
+    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
+    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
+    if (!lhsXIsOp0 && !lhsXIsOp1)
+      continue;
+    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
+    if (!eAggregate)
+      continue;
+    result.isComplex = true;
+    result.isNE = isa<LLVM::OrOp>(op);
+    result.eAggregate = eAggregate;
+    result.isXBinopExpr = lhsXIsOp0;
+    break;
+  }
+  return result;
+}
+
+/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
+/// compare. The expected/desired complex values are spilled to allocas and
+/// reloaded as an integer of the same width, then compared and swapped. Returns
+/// the cmpxchg (result type `{iN, i1}`); `maxAlign` receives the alignment used
+/// so callers can reinterpret the captured old value through memory.
+static llvm::AtomicCmpXchgInst *
+emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
+                         llvm::Type *complexTy, llvm::Value *eVal,
+                         llvm::Value *dVal, llvm::AtomicOrdering atomicOrdering,
+                         bool isWeak, llvm::Align &maxAlign) {
+  const llvm::DataLayout &DL =
+      builder.GetInsertBlock()->getModule()->getDataLayout();
+  unsigned totalBits = DL.getTypeStoreSizeInBits(complexTy).getFixedValue();
+  llvm::IntegerType *intTy =
+      llvm::IntegerType::get(builder.getContext(), totalBits);
+  llvm::Align complexAlign = DL.getABITypeAlign(complexTy);
+  llvm::Align intAlign = DL.getABITypeAlign(intTy);
+  maxAlign = std::max(complexAlign, intAlign);
+
+  llvm::AllocaInst *eAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.e");
+  eAlloca->setAlignment(maxAlign);
+  llvm::AllocaInst *dAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.d");
+  dAlloca->setAlignment(maxAlign);
+
+  builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
+  llvm::Value *eInt =
+      builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
+  builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
+  llvm::Value *dInt =
+      builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
+
+  llvm::AtomicOrdering failOrdering =
+      llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
+  auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
+                                              atomicOrdering, failOrdering);
+  cmpXchg->setWeak(isWeak);
+  return cmpXchg;
+}
+
 /// Holds the extracted comparison pattern information from an atomic compare
 /// region.
 struct AtomicComparePatternInfo {
@@ -5379,7 +5472,6 @@ struct AtomicComparePatternInfo {
   bool isXBinopExpr = false;
   bool isSigned = false;
 };
-
 /// Extract comparison predicate, expected value (e), desired value (d), and
 /// related flags from an atomic compare region block by scanning for
 /// icmp/fcmp/select/min/max operations.
@@ -5387,6 +5479,27 @@ static LogicalResult extractAtomicComparePattern(
     Block &block,
     llvm::function_ref<llvm::Value *(mlir::Value)> materializeValue,
     omp::AtomicCompareOp atomicCompareOp, AtomicComparePatternInfo &info) {
+  // Complex equality is a decomposed per-field pattern (extractvalue + fcmp +
+  // and) rather than a single scalar compare. Detect it first so the scalar
+  // icmp/fcmp handling below does not mistake a real/imaginary field for the
+  // whole expected value.
+  if (ComplexComparePattern cplx = detectComplexCompareEq(block);
+      cplx.isComplex) {
+    if (cplx.isNE)
+      return atomicCompareOp.emitError(
+          "unsupported comparison predicate (NE) for complex atomic compare");
+    info.compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    info.isXBinopExpr = cplx.isXBinopExpr;
+    info.eVal = materializeValue(cplx.eAggregate);
+    for (Operation &op : block.getOperations()) {
+      if (auto selectOp = dyn_cast<LLVM::SelectOp>(op)) {
+        info.dVal = materializeValue(selectOp.getTrueValue());
+        break;
+      }
+    }
+    return success();
+  }
+
   for (Operation &op : block.getOperations()) {
     // Pre-filter: skip icmps that don't involve the block argument
     // (e.g., truthiness extractions from logical-to-integer conversion).
@@ -5573,22 +5686,101 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     bool isPostfixCapture = !isReadFirst;
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
+    // Complex equality capture: x is struct-typed, which the OMPIRBuilder
+    // cannot handle, so emit a bitcast-to-integer cmpxchg (as in the
+    // non-capture complex path) and reconstruct the captured value from its
+    // result through memory. Complex only supports the == comparison.
+    if (llvmXElementType->isStructTy()) {
+      llvm::Align maxAlign;
+      llvm::AtomicCmpXchgInst *cmpXchg = emitComplexAtomicCmpXchg(
+          builder, llvmX, llvmXElementType, eVal, dVal, atomicOrdering,
+          atomicCompareOp.getWeak(), maxAlign);
+
+      llvm::Value *oldInt = builder.CreateExtractValue(cmpXchg, 0);
+      llvm::Value *cmpOk = builder.CreateExtractValue(cmpXchg, 1);
+
+      // Reinterpret the old integer value as the complex struct via memory.
+      llvm::AllocaInst *oldAlloca =
+          builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.old");
+      oldAlloca->setAlignment(maxAlign);
+      builder.CreateAlignedStore(oldInt, oldAlloca, maxAlign);
+      llvm::Value *oldComplex = builder.CreateAlignedLoad(
+          llvmXElementType, oldAlloca, maxAlign, "cmplx.old.val");
+
+      if (isFailOnly) {
+        // v is written only when the compare fails (cmpOk == false).
+        llvm::Value *cmpFailed = builder.CreateNot(cmpOk);
+        llvm::BasicBlock *curBB = builder.GetInsertBlock();
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(cmpFailed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else if (isPostfixCapture) {
+        // v gets the new value of x: d on success, old x otherwise.
+        llvm::Value *newComplex = builder.CreateSelect(cmpOk, dVal, oldComplex);
+        builder.CreateStore(newComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      } else {
+        // Prefix: v gets the old value of x.
+        builder.CreateStore(oldComplex, llvmAtomicV.Var,
+                            llvmAtomicV.IsVolatile);
+      }
+
+      // Emit flush after atomic compare if needed (release/acq_rel/seq_cst).
+      if (atomicOrdering == llvm::AtomicOrdering::Release ||
+          atomicOrdering == llvm::AtomicOrdering::AcquireRelease ||
+          atomicOrdering == llvm::AtomicOrdering::SequentiallyConsistent) {
+        llvm::OpenMPIRBuilder::LocationDescription flushLoc(builder);
+        ompBuilder->createFlush(flushLoc);
+      }
+      return success();
+    }
+
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw, so the fail-only capture
+    // form (v written only when the compare fails) has no valid mapping and is
+    // Min/max (<, >) comparisons lower to an atomicrmw. The OMPIRBuilder has no
+    // notion of a failed compare for an atomicrmw (it asserts on IsFailOnly),
+    // so for min/max the fail-only capture is reconstructed manually below.
+    bool isMinMax = compareOp != llvm::omp::OMPAtomicCompareOp::EQ;
+
     llvm::OpenMPIRBuilder::AtomicOpValue llvmAtomicVForCall = llvmAtomicV;
-    // Postfix: bypass V in OMPIRBuilder, handled manually below.
-    if (isPostfixCapture && !isFailOnly)
+    // Capture into V is reconstructed manually below for:
+    //   * postfix capture (v gets the new value of x): equality from the
+    //     cmpxchg result, min/max from the atomicrmw result;
+    //   * min/max fail-only capture (the OMPIRBuilder cannot express it).
+    // Bypass V in the OMPIRBuilder for those cases so it does not also emit its
+    // own (for min/max, incorrect or unsupported) capture store.
+    bool minMaxManualCapture = isMinMax && (isPostfixCapture || isFailOnly);
+    bool eqPostfixManualCapture = !isMinMax && isPostfixCapture && !isFailOnly;
+    if (minMaxManualCapture || eqPostfixManualCapture)
       llvmAtomicVForCall = {nullptr, nullptr, false, false};
 
-    // Prefix: IsPostfixUpdate=true → direct store of old value.
-    // Fail-only: IsPostfixUpdate=false → conditional store on failure.
-    bool isPostfixUpdate = !isFailOnly;
+    // The OMPIRBuilder only understands IsFailOnly for the equality (cmpxchg)
+    // path; for min/max it would assert. Min/max fail-only is handled here.
+    bool builderFailOnly = isFailOnly && !isMinMax;
+
+    // IsPostfixUpdate selects which value the OMPIRBuilder captures into V:
+    //   * min/max prefix and equality prefix: a direct store of the old value
+    //     (IsPostfixUpdate=true).
+    //   * equality fail-only: a conditional store (IsPostfixUpdate=false).
+    // Manually-reconstructed captures bypass V above.
+    bool isPostfixUpdate = !builderFailOnly;
 
     bool isWeak = atomicCompareOp.getWeak();
     bool savedHandleFPNegZero = ompBuilder->setHandleFPNegZero(true);
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-        ompBuilder->createAtomicCompare(ompLoc, llvmAtomicX, llvmAtomicVForCall,
-                                        llvmAtomicR, eVal, dVal, atomicOrdering,
-                                        compareOp, isXBinopExpr,
-                                        isPostfixUpdate, isFailOnly, isWeak);
+        ompBuilder->createAtomicCompare(
+            ompLoc, llvmAtomicX, llvmAtomicVForCall, llvmAtomicR, eVal, dVal,
+            atomicOrdering, compareOp, isXBinopExpr, isPostfixUpdate,
+            builderFailOnly, isWeak);
     ompBuilder->setHandleFPNegZero(savedHandleFPNegZero);
 
     if (failed(handleError(afterIP, *atomicCaptureOp)))
@@ -5596,8 +5788,99 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
 
     builder.restoreIP(*afterIP);
 
-    // Postfix: v = select(success, D, old) — captures new value of x.
-    if (isPostfixCapture && !isFailOnly) {
+    // Min/max capture is reconstructed from the atomicrmw the OMPIRBuilder
+    // emits (its result is the old value of x). V was bypassed above.
+    //   * postfix: v gets the new value min/max(old, e);
+    //   * fail-only: v gets the old value, but only when the compare failed
+    //     (i.e. the atomicrmw did not change x).
+    // (Prefix min/max captures the old value directly through V, so nothing
+    // extra is needed there.)
+    if (isMinMax && (isPostfixCapture || isFailOnly)) {
+      llvm::BasicBlock *curBB = builder.GetInsertBlock();
+      llvm::AtomicRMWInst *rmw = nullptr;
+      for (auto &inst : llvm::reverse(*curBB)) {
+        if (auto *r = dyn_cast<llvm::AtomicRMWInst>(&inst)) {
+          rmw = r;
+          break;
+        }
+      }
+      assert(rmw && "expected atomicrmw for min/max compare capture");
+      llvm::Value *oldVal = rmw;
+      llvm::Value *rhs = rmw->getValOperand();
+
+      if (isFailOnly) {
+        // The compare "failed" (the else branch runs) exactly when the
+        // atomicrmw did not change x. Recompute the original update condition
+        // on the old value and negate it. v is stored only in that case.
+        llvm::CmpInst::Predicate updatePred;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          updatePred = llvm::CmpInst::ICMP_SGT;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          updatePred = llvm::CmpInst::ICMP_SLT;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          updatePred = llvm::CmpInst::ICMP_UGT;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          updatePred = llvm::CmpInst::ICMP_ULT;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          updatePred = llvm::CmpInst::FCMP_OGT;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          updatePred = llvm::CmpInst::FCMP_OLT;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *updated = builder.CreateCmp(updatePred, oldVal, rhs);
+        llvm::Value *failed = builder.CreateNot(updated);
+        llvm::Function *fn = curBB->getParent();
+        llvm::BasicBlock *contBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.cont", fn);
+        llvm::BasicBlock *exitBB = llvm::BasicBlock::Create(
+            builder.getContext(), "omp.atomic.exit", fn);
+        builder.CreateCondBr(failed, contBB, exitBB);
+        builder.SetInsertPoint(contBB);
+        builder.CreateStore(oldVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+        builder.CreateBr(exitBB);
+        builder.SetInsertPoint(exitBB);
+      } else {
+        llvm::Intrinsic::ID id;
+        switch (rmw->getOperation()) {
+        case llvm::AtomicRMWInst::Min:
+          id = llvm::Intrinsic::smin;
+          break;
+        case llvm::AtomicRMWInst::Max:
+          id = llvm::Intrinsic::smax;
+          break;
+        case llvm::AtomicRMWInst::UMin:
+          id = llvm::Intrinsic::umin;
+          break;
+        case llvm::AtomicRMWInst::UMax:
+          id = llvm::Intrinsic::umax;
+          break;
+        case llvm::AtomicRMWInst::FMin:
+          id = llvm::Intrinsic::minnum;
+          break;
+        case llvm::AtomicRMWInst::FMax:
+          id = llvm::Intrinsic::maxnum;
+          break;
+        default:
+          llvm_unreachable(
+              "unexpected atomicrmw op for min/max compare capture");
+        }
+        llvm::Value *newVal = builder.CreateBinaryIntrinsic(id, oldVal, rhs);
+        builder.CreateStore(newVal, llvmAtomicV.Var, llvmAtomicV.IsVolatile);
+      }
+    }
+
+    // Equality postfix: v = select(success, D, old) — reconstructs the new
+    // value of x from the cmpxchg result.
+    if (!isMinMax && isPostfixCapture && !isFailOnly) {
       llvm::BasicBlock *curBB = builder.GetInsertBlock();
       llvm::Value *oldVal = nullptr;
       llvm::Value *successVal = nullptr;
@@ -5848,56 +6131,18 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
   llvm::Value *dVal = nullptr;
   bool isXBinopExpr = false;
 
-  auto traceToAggregate = [](mlir::Value v) -> mlir::Value {
-    if (auto extractOp = v.getDefiningOp<LLVM::ExtractValueOp>())
-      return extractOp.getContainer();
-    return nullptr;
-  };
-
-  // Check for a decomposed complex comparison pattern:
-  //   %re_x = llvm.extractvalue %xval[0]
-  //   %re_e = llvm.extractvalue %eStruct[0]
-  //   %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-  //   %im_x = llvm.extractvalue %xval[1]
-  //   %im_e = llvm.extractvalue %eStruct[1]
-  //   %cmp_im = llvm.fcmp "oeq" %im_x, %im_e
-  //   %cmp = llvm.and %cmp_re, %cmp_im   (for EQ)
-  // Detect this by looking for AndOp/OrOp whose operands are both FCmpOps
-  // operating on ExtractValueOps from the block argument.
-  bool isComplexPattern = false;
-  for (Operation &op : block.getOperations()) {
-    if (!isa<LLVM::AndOp, LLVM::OrOp>(op))
-      continue;
-
-    // Using : %cmp = llvm.and %cmp_re, %cmp_im
-    auto lhsFcmp = op.getOperand(0).getDefiningOp<LLVM::FCmpOp>();
-    auto rhsFcmp = op.getOperand(1).getDefiningOp<LLVM::FCmpOp>();
-    if (!lhsFcmp || !rhsFcmp)
-      continue;
-
-    // Using : %cmp_re = llvm.fcmp "oeq" %re_x, %re_e
-    // Check presence of x (block argument) and get e.
-    mlir::Value lhsAgg0 = traceToAggregate(lhsFcmp.getOperand(0));
-    mlir::Value lhsAgg1 = traceToAggregate(lhsFcmp.getOperand(1));
-    bool lhsXIsOp0 = (lhsAgg0 == block.getArgument(0));
-    bool lhsXIsOp1 = (lhsAgg1 == block.getArgument(0));
-    if (!lhsXIsOp0 && !lhsXIsOp1)
-      continue;
-    mlir::Value eAggregate = lhsXIsOp0 ? lhsAgg1 : lhsAgg0;
-    if (!eAggregate)
-      continue;
-
-    if (isa<LLVM::AndOp>(op))
-      compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
-    else
+  // Check for a decomposed complex comparison pattern (extractvalue + fcmp +
+  // and/or of the real/imaginary fields).
+  ComplexComparePattern cplx = detectComplexCompareEq(block);
+  bool isComplexPattern = cplx.isComplex;
+  if (isComplexPattern) {
+    if (cplx.isNE)
       // OrOp corresponds to NE, which is not a valid atomic compare op.
       return atomicCompareOp.emitError(
           "unsupported comparison predicate (NE) for complex atomic compare");
-
-    isXBinopExpr = lhsXIsOp0;
-    eVal = materializeValue(eAggregate);
-    isComplexPattern = true;
-    break;
+    compareOp = llvm::omp::OMPAtomicCompareOp::EQ;
+    isXBinopExpr = cplx.isXBinopExpr;
+    eVal = materializeValue(cplx.eAggregate);
   }
 
   if (isComplexPattern) {
@@ -5916,37 +6161,10 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
       dVal = materializeValue(yieldOp.getResults()[0]);
     }
 
-    const llvm::DataLayout &DL =
-        builder.GetInsertBlock()->getModule()->getDataLayout();
-    unsigned totalBits =
-        DL.getTypeStoreSizeInBits(llvmXElementType).getFixedValue();
-
-    llvm::IntegerType *intTy =
-        llvm::IntegerType::get(builder.getContext(), totalBits);
-
-    llvm::Align complexAlign = DL.getABITypeAlign(llvmXElementType);
-    llvm::Align intAlign = DL.getABITypeAlign(intTy);
-    llvm::Align maxAlign = std::max(complexAlign, intAlign);
-
-    llvm::AllocaInst *eAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.e");
-    eAlloca->setAlignment(maxAlign);
-    llvm::AllocaInst *dAlloca =
-        builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.d");
-    dAlloca->setAlignment(maxAlign);
-
-    builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
-    llvm::Value *eInt =
-        builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
-    builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
-    llvm::Value *dInt =
-        builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
-
-    llvm::AtomicOrdering failOrdering =
-        llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
-    auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
-                                                atomicOrdering, failOrdering);
-    cmpXchg->setWeak(atomicCompareOp.getWeak());
+    llvm::Align maxAlign;
+    emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
+                             atomicOrdering, atomicCompareOp.getWeak(),
+                             maxAlign);
 
     // Emit flush after atomic compare if needed (for release, acq_rel,
     // seq_cst orderings).
diff --git a/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
new file mode 100644
index 0000000000000..2a3cb6dc99133
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-atomic-compare-capture.mlir
@@ -0,0 +1,407 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Prefix compare+capture: {read, compare}
+// CHECK-LABEL: define void @compare_capture_prefix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Postfix compare+capture: {compare, read}
+// CHECK-LABEL: define void @compare_capture_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         %[[NEWVAL:.*]] = select i1 %[[SUCCESS]], i32 %[[DVAL]], i32 %[[OLD]]
+// CHECK:         store i32 %[[NEWVAL]], ptr %[[V]], align 4
+llvm.func @compare_capture_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Fail-only compare+capture: {compare, read} with fail_only
+// CHECK-LABEL: define void @compare_capture_failonly(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         %[[SUCCESS:.*]] = extractvalue { i32, i1 } %[[RESULT]], 1
+// CHECK:         br i1 %[[SUCCESS]], label %{{.*}}.atomic.exit, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store i32 %[[OLD]], ptr %[[V]]
+// CHECK:         br label %{{.*}}.atomic.exit
+// CHECK:       {{.*}}.atomic.exit:
+llvm.func @compare_capture_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Weak compare+capture: {read, compare} with weak
+// CHECK-LABEL: define void @compare_capture_weak(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[E:.*]], ptr %[[D:.*]], ptr %[[V:.*]])
+// CHECK:         %[[EVAL:.*]] = load i32, ptr %[[E]], align 4
+// CHECK:         %[[DVAL:.*]] = load i32, ptr %[[D]], align 4
+// CHECK:         %[[RESULT:.*]] = cmpxchg weak ptr %[[X]], i32 %[[EVAL]], i32 %[[DVAL]] monotonic monotonic
+// CHECK:         %[[OLD:.*]] = extractvalue { i32, i1 } %[[RESULT]], 0
+// CHECK:         store i32 %[[OLD]], ptr %[[V]], align 4
+llvm.func @compare_capture_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> i32
+  %dval = llvm.load %d : !llvm.ptr -> i32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: i32):
+      %cmp = llvm.icmp "eq" %xval, %eval : i32
+      %sel = llvm.select %cmp, %dval, %xval : i1, i32
+      omp.yield(%sel : i32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Float (real) tests =====
+// Float uses the HandleFPNegZero path with NaN/zero checks and bitcast.
+
+// Float prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_float_prefix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         store float %{{.*}}, ptr %[[V:.*]], align 4
+llvm.func @compare_capture_float_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+  }
+  llvm.return
+}
+
+// Float postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_float_postfix(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         %[[NEW:.*]] = select i1 %[[OK]], float %{{.*}}, float %[[OLD_FP]]
+// CHECK:         store float %[[NEW]], ptr %{{.*}}, align 4
+llvm.func @compare_capture_float_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_float_failonly(
+// CHECK:         cmpxchg ptr
+// CHECK:       {{.*}}.atomic.exit:
+// CHECK:         %[[OK:.*]] = phi i1
+// CHECK:         %[[OLD_FP:.*]] = bitcast i32 %{{.*}} to float
+// CHECK:         br i1 %[[OK]], label %{{.*}}.atomic.exit{{.*}}, label %{{.*}}.atomic.cont
+// CHECK:       {{.*}}.atomic.cont:
+// CHECK:         store float %[[OLD_FP]], ptr %{{.*}}
+llvm.func @compare_capture_float_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Complex (struct<(f32, f32)>) tests =====
+// Complex uses bitcast to i64 for cmpxchg, with FP neg-zero handling.
+
+// Complex prefix compare+capture: v gets old value
+// CHECK-LABEL: define void @compare_capture_complex_prefix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_prefix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+  }
+  llvm.return
+}
+
+// Complex postfix compare+capture: v gets select(success, d, old)
+// CHECK-LABEL: define void @compare_capture_complex_postfix(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         select i1 %{{.*}}, { float, float } %{{.*}}, { float, float } %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_postfix(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  }
+  llvm.return
+}
+
+// Complex fail-only compare+capture: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_complex_failonly(
+// CHECK:         cmpxchg ptr %{{.*}}, i64 %{{.*}}, i64 %{{.*}} monotonic monotonic
+// CHECK:         br i1 %{{.*}}, label %{{.*}}, label %{{.*}}
+// CHECK:         store { float, float } %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_complex_failonly(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  %dval = llvm.load %d : !llvm.ptr -> !llvm.struct<(f32, f32)>
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: !llvm.struct<(f32, f32)>):
+      %xr = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>
+      %er = llvm.extractvalue %eval[0] : !llvm.struct<(f32, f32)>
+      %cmpr = llvm.fcmp "oeq" %xr, %er : f32
+      %xi = llvm.extractvalue %xval[1] : !llvm.struct<(f32, f32)>
+      %ei = llvm.extractvalue %eval[1] : !llvm.struct<(f32, f32)>
+      %cmpi = llvm.fcmp "oeq" %xi, %ei : f32
+      %cmp = llvm.and %cmpr, %cmpi : i1
+      %sel = llvm.select %cmp, %dval, %xval : i1, !llvm.struct<(f32, f32)>
+      omp.yield(%sel : !llvm.struct<(f32, f32)>)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, !llvm.struct<(f32, f32)>
+  } {fail_only}
+  llvm.return
+}
+
+// ===== Weak clause with float =====
+
+// Float weak prefix: cmpxchg weak with FP neg-zero handling
+// CHECK-LABEL: define void @compare_capture_float_weak(
+// CHECK:         cmpxchg weak ptr %{{.*}}, i32 %{{.*}}, i32 %{{.*}} monotonic monotonic
+// CHECK:         store float %{{.*}}, ptr %{{.*}}
+llvm.func @compare_capture_float_weak(%x : !llvm.ptr, %e : !llvm.ptr, %d : !llvm.ptr, %v : !llvm.ptr) {
+  %eval = llvm.load %e : !llvm.ptr -> f32
+  %dval = llvm.load %d : !llvm.ptr -> f32
+  omp.atomic.capture memory_order(relaxed) {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval: f32):
+      %cmp = llvm.fcmp "oeq" %xval, %eval : f32
+      %sel = llvm.select %cmp, %dval, %xval : i1, f32
+      omp.yield(%sel : f32)
+    } {weak}
+  }
+  llvm.return
+}
+
+// ===== Min/Max compare+capture =====
+
+// Integer min, postfix: atomicrmw min, v gets min(old, e)
+// CHECK-LABEL: define void @compare_capture_min_postfix(
+// CHECK-SAME:    ptr %[[X:.*]], ptr %[[V:.*]], i32 %[[E:.*]])
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %[[X]], i32 %[[E]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smin.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %[[V]]
+llvm.func @compare_capture_min_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer max, postfix: atomicrmw max, v gets max(old, e)
+// CHECK-LABEL: define void @compare_capture_max_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call i32 @llvm.smax.i32(i32 %[[OLD]], i32 %[[E]])
+// CHECK:         store i32 %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_max_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  }
+  llvm.return
+}
+
+// Integer min, prefix: v gets old value
+// CHECK-LABEL: define void @compare_capture_min_prefix(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %{{.*}} monotonic
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_prefix(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+  }
+  llvm.return
+}
+
+// Float min, postfix: atomicrmw fmin, v gets minnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmin_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmin ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.minnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmin_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "ogt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// Float max, postfix: atomicrmw fmax, v gets maxnum(old, e)
+// CHECK-LABEL: define void @compare_capture_fmax_postfix(
+// CHECK:         %[[OLD:.*]] = atomicrmw fmax ptr %{{.*}}, float %[[E:.*]] monotonic
+// CHECK:         %[[NEW:.*]] = call float @llvm.maxnum.f32(float %[[OLD]], float %[[E]])
+// CHECK:         store float %[[NEW]], ptr %{{.*}}
+llvm.func @compare_capture_fmax_postfix(%x : !llvm.ptr, %v : !llvm.ptr, %e : f32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : f32):
+      %cmp = llvm.fcmp "olt" %xval, %e : f32
+      %sel = llvm.select %cmp, %e, %xval : i1, f32
+      omp.yield(%sel : f32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, f32
+  }
+  llvm.return
+}
+
+// ===== Min/Max fail-only =====
+
+// Integer min, fail-only: conditional store on failure
+// CHECK-LABEL: define void @compare_capture_min_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw min ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp sgt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_min_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "sgt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}
+
+// Integer max, fail-only
+// CHECK-LABEL: define void @compare_capture_max_fail_only(
+// CHECK:         %[[OLD:.*]] = atomicrmw max ptr %{{.*}}, i32 %[[E:.*]] monotonic
+// CHECK:         %[[UPD:.*]] = icmp slt i32 %[[OLD]], %[[E]]
+// CHECK:         %[[FAILED:.*]] = xor i1 %[[UPD]], true
+// CHECK:         br i1 %[[FAILED]], label %[[CONT:.*]], label %[[EXIT:.*]]
+// CHECK:       [[CONT]]:
+// CHECK:         store i32 %[[OLD]], ptr %{{.*}}
+llvm.func @compare_capture_max_fail_only(%x : !llvm.ptr, %v : !llvm.ptr, %e : i32) {
+  omp.atomic.capture {
+    omp.atomic.compare %x : !llvm.ptr {
+    ^bb0(%xval : i32):
+      %cmp = llvm.icmp "slt" %xval, %e : i32
+      %sel = llvm.select %cmp, %e, %xval : i1, i32
+      omp.yield(%sel : i32)
+    }
+    omp.atomic.read %v = %x : !llvm.ptr, !llvm.ptr, i32
+  } {fail_only}
+  llvm.return
+}

>From ea6c6151e5dce95ca2b26cf7f1d0f7a2cb70c421 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 21 Jul 2026 01:34:26 -0500
Subject: [PATCH 10/11] Using load, fcmp oeq & cmpxchg for complex numbers.
 Avoiding CAS loops similar to atomic compare.

---
 .../OpenMP/atomic-compare-capture.f90         |  10 +-
 .../Integration/OpenMP/atomic-compare.f90     |  41 ++++--
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 124 ++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  17 ++-
 4 files changed, 129 insertions(+), 63 deletions(-)

diff --git a/flang/test/Integration/OpenMP/atomic-compare-capture.f90 b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
index fe97e7b12558b..4f56720b75a0c 100644
--- a/flang/test/Integration/OpenMP/atomic-compare-capture.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare-capture.f90
@@ -102,6 +102,8 @@ subroutine cc_logical(x, e, d, v)
 ! FIR:           fir.cmpc "oeq"
 ! FIR:         }
 ! LLVM-LABEL: define void @cc_complex_(
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
 ! LLVM:         cmpxchg ptr %{{.*}}, i64
 ! LLVM:         store { float, float } %{{.*}}, ptr
 subroutine cc_complex(x, e, d, v)
@@ -379,6 +381,8 @@ subroutine cc_logical_failonly(x, e, d, v)
 ! FIR-LABEL: func.func @_QPcc_complex_prefix(
 ! FIR:           omp.atomic.read %{{.*}} : !fir.ref<complex<f32>>, !fir.ref<complex<f32>>, complex<f32>
 ! LLVM-LABEL: define void @cc_complex_prefix_(
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
 ! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
 ! LLVM:         store { float, float } %{{.*}}, ptr
 subroutine cc_complex_prefix(x, e, d, v)
@@ -396,8 +400,10 @@ subroutine cc_complex_prefix(x, e, d, v)
 ! FIR-LABEL: func.func @_QPcc_complex_failonly(
 ! FIR:         } {fail_only}
 ! LLVM-LABEL: define void @cc_complex_failonly_(
-! LLVM:         %[[RES:.*]] = cmpxchg ptr %{{.*}}, i64
-! LLVM:         %[[OK:.*]] = extractvalue { i64, i1 } %[[RES]], 1
+! LLVM:         %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+! LLVM:         %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+! LLVM:         cmpxchg ptr %{{.*}}, i64
+! LLVM:         %[[OK:.*]] = phi i1
 ! LLVM:         %[[FAILED:.*]] = xor i1 %[[OK]], true
 ! LLVM:         br i1 %[[FAILED]], label %{{.*}}, label %{{.*}}
 ! LLVM:         store { float, float } %{{.*}}, ptr
diff --git a/flang/test/Integration/OpenMP/atomic-compare.f90 b/flang/test/Integration/OpenMP/atomic-compare.f90
index e98b25e4b4539..88b4bca9e9968 100644
--- a/flang/test/Integration/OpenMP/atomic-compare.f90
+++ b/flang/test/Integration/OpenMP/atomic-compare.f90
@@ -150,32 +150,45 @@ subroutine atomic_compare_real(x, e, d)
   if (x == e) x = d
 end
 
-! Complex(4) equality → type-punned i64 cmpxchg with consistent alignment
+! Complex(4) equality -> component-wise IEEE-754 fcmp, then a cmpxchg that swaps
+! using X's loaded bit-pattern as the comparand (so -0.0/+0.0 and NaN follow the
+! scalar float semantics rather than a raw bitwise compare).
 !CHECK-LABEL: define void @atomic_compare_complex4_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]])
-!CHECK: %[[EALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
-!CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN]]
-!CHECK: store { float, float } %{{.*}}, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: %[[EINT:.*]] = load i64, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: store { float, float } %{{.*}}, ptr %[[DALLOCA]], align [[ALIGN]]
+!CHECK: %[[EVAL:.*]] = load { float, float }, ptr %[[E]], align 4
+!CHECK: %[[DVAL:.*]] = load { float, float }, ptr %[[D]], align 4
+!CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
+!CHECK: store { float, float } %[[DVAL]], ptr %[[DALLOCA]], align [[ALIGN]]
 !CHECK: %[[DINT:.*]] = load i64, ptr %[[DALLOCA]], align [[ALIGN]]
-!CHECK: cmpxchg ptr %[[X]], i64 %[[EINT]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+!CHECK: %[[XLOAD:.*]] = load atomic i64, ptr %[[X]] monotonic, align [[ALIGN]]
+!CHECK: %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+!CHECK: %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+!CHECK: %[[EQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+!CHECK: br i1 %[[EQ]], label %[[SWAP:[^,]+]], label %{{.*}}
+!CHECK: [[SWAP]]:
+!CHECK: cmpxchg ptr %[[X]], i64 %[[XLOAD]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
 subroutine atomic_compare_complex4(x, e, d)
   complex :: x, e, d
   !$omp atomic compare
   if (x == e) x = d
 end
 
-! Complex(8) equality → type-punned i128 cmpxchg with consistent alignment
+! Complex(8) equality -> component-wise IEEE-754 fcmp, then an i128 cmpxchg using
+! X's loaded bit-pattern as the comparand.
 !CHECK-LABEL: define void @atomic_compare_complex8_(
 !CHECK-SAME: ptr noalias %[[X:.*]], ptr noalias %[[E:.*]], ptr noalias %[[D:.*]])
-!CHECK: %[[EALLOCA:.*]] = alloca { double, double }, align [[ALIGN:[0-9]+]]
-!CHECK: %[[DALLOCA:.*]] = alloca { double, double }, align [[ALIGN]]
-!CHECK: store { double, double } %{{.*}}, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: %[[EINT:.*]] = load i128, ptr %[[EALLOCA]], align [[ALIGN]]
-!CHECK: store { double, double } %{{.*}}, ptr %[[DALLOCA]], align [[ALIGN]]
+!CHECK: %[[EVAL:.*]] = load { double, double }, ptr %[[E]], align 8
+!CHECK: %[[DVAL:.*]] = load { double, double }, ptr %[[D]], align 8
+!CHECK: %[[DALLOCA:.*]] = alloca { double, double }, align [[ALIGN:[0-9]+]]
+!CHECK: store { double, double } %[[DVAL]], ptr %[[DALLOCA]], align [[ALIGN]]
 !CHECK: %[[DINT:.*]] = load i128, ptr %[[DALLOCA]], align [[ALIGN]]
-!CHECK: cmpxchg ptr %[[X]], i128 %[[EINT]], i128 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+!CHECK: %[[XLOAD:.*]] = load atomic i128, ptr %[[X]] monotonic, align [[ALIGN]]
+!CHECK: %[[REEQ:.*]] = fcmp oeq double %[[REX:.*]], %[[REE:.*]]
+!CHECK: %[[IMEQ:.*]] = fcmp oeq double %[[IMX:.*]], %[[IME:.*]]
+!CHECK: %[[EQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+!CHECK: br i1 %[[EQ]], label %[[SWAP:[^,]+]], label %{{.*}}
+!CHECK: [[SWAP]]:
+!CHECK: cmpxchg ptr %[[X]], i128 %[[XLOAD]], i128 %[[DINT]] monotonic monotonic, align [[ALIGN]]
 subroutine atomic_compare_complex8(x, e, d)
   complex(8) :: x, e, d
   !$omp atomic compare
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 060e563ecefa8..fdc8f09fe5a2d 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5422,16 +5422,15 @@ static ComplexComparePattern detectComplexCompareEq(Block &block) {
   return result;
 }
 
-/// Emit a bitcast-to-integer `cmpxchg` for a complex (struct-typed) atomic
-/// compare. The expected/desired complex values are spilled to allocas and
-/// reloaded as an integer of the same width, then compared and swapped. Returns
-/// the cmpxchg (result type `{iN, i1}`); `maxAlign` receives the alignment used
-/// so callers can reinterpret the captured old value through memory.
-static llvm::AtomicCmpXchgInst *
-emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
-                         llvm::Type *complexTy, llvm::Value *eVal,
-                         llvm::Value *dVal, llvm::AtomicOrdering atomicOrdering,
-                         bool isWeak, llvm::Align &maxAlign) {
+/// Emit an IEEE-754-correct `cmpxchg` for a complex (struct-typed) atomic
+/// compare with `fcmp oeq`. The old value of X is returned (as the complex struct
+/// type) in \p oldComplex and the success flag (i1) in \p cmpOk.
+static void emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder,
+                                     llvm::Value *llvmX, llvm::Type *complexTy,
+                                     llvm::Value *eVal, llvm::Value *dVal,
+                                     llvm::AtomicOrdering atomicOrdering,
+                                     bool isWeak, llvm::Value *&oldComplex,
+                                     llvm::Value *&cmpOk) {
   const llvm::DataLayout &DL =
       builder.GetInsertBlock()->getModule()->getDataLayout();
   unsigned totalBits = DL.getTypeStoreSizeInBits(complexTy).getFixedValue();
@@ -5439,28 +5438,76 @@ emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder, llvm::Value *llvmX,
       llvm::IntegerType::get(builder.getContext(), totalBits);
   llvm::Align complexAlign = DL.getABITypeAlign(complexTy);
   llvm::Align intAlign = DL.getABITypeAlign(intTy);
-  maxAlign = std::max(complexAlign, intAlign);
+  llvm::Align maxAlign = std::max(complexAlign, intAlign);
 
-  llvm::AllocaInst *eAlloca =
-      builder.CreateAlloca(complexTy, nullptr, "cmplx.e");
-  eAlloca->setAlignment(maxAlign);
+  // Spill D to obtain its integer bit pattern for the swap value.
   llvm::AllocaInst *dAlloca =
       builder.CreateAlloca(complexTy, nullptr, "cmplx.d");
   dAlloca->setAlignment(maxAlign);
-
-  builder.CreateAlignedStore(eVal, eAlloca, maxAlign);
-  llvm::Value *eInt =
-      builder.CreateAlignedLoad(intTy, eAlloca, maxAlign, "cmplx.e.int");
   builder.CreateAlignedStore(dVal, dAlloca, maxAlign);
   llvm::Value *dInt =
       builder.CreateAlignedLoad(intTy, dAlloca, maxAlign, "cmplx.d.int");
 
+  // Load the current value of X atomically and reinterpret it as complex.
+  llvm::LoadInst *xCurr =
+      builder.CreateAlignedLoad(intTy, llvmX, maxAlign, "cmplx.x.load");
+  xCurr->setAtomic(llvm::AtomicOrdering::Monotonic);
+  llvm::AllocaInst *xAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.x");
+  xAlloca->setAlignment(maxAlign);
+  builder.CreateAlignedStore(xCurr, xAlloca, maxAlign);
+  llvm::Value *xStruct =
+      builder.CreateAlignedLoad(complexTy, xAlloca, maxAlign, "cmplx.x.val");
+
+  // Component-wise IEEE-754 equality: `fcmp oeq` yields false for NaN (so a
+  // NaN component correctly makes the compare fail) and true for +0.0 vs -0.0
+  // (so a zero-sign difference does not spuriously fail the compare).
+  llvm::Value *reX = builder.CreateExtractValue(xStruct, 0);
+  llvm::Value *imX = builder.CreateExtractValue(xStruct, 1);
+  llvm::Value *reE = builder.CreateExtractValue(eVal, 0);
+  llvm::Value *imE = builder.CreateExtractValue(eVal, 1);
+  llvm::Value *reEq = builder.CreateFCmpOEQ(reX, reE, "cmplx.re.eq");
+  llvm::Value *imEq = builder.CreateFCmpOEQ(imX, imE, "cmplx.im.eq");
+  llvm::Value *fpEqual = builder.CreateAnd(reEq, imEq, "cmplx.eq");
+
+  // When the components compare equal, attempt the swap using X's own loaded
+  // bit pattern as the comparand; otherwise the compare fails and X is left
+  // unchanged (the captured old value is the value just loaded).
+  llvm::BasicBlock *curBB = builder.GetInsertBlock();
+  llvm::Function *fn = curBB->getParent();
+  llvm::BasicBlock *swapBB =
+      llvm::BasicBlock::Create(builder.getContext(), "cmplx.atomic.swap", fn);
+  llvm::BasicBlock *exitBB =
+      llvm::BasicBlock::Create(builder.getContext(), "cmplx.atomic.exit", fn);
+  builder.CreateCondBr(fpEqual, swapBB, exitBB);
+
+  builder.SetInsertPoint(swapBB);
   llvm::AtomicOrdering failOrdering =
       llvm::AtomicCmpXchgInst::getStrongestFailureOrdering(atomicOrdering);
-  auto *cmpXchg = builder.CreateAtomicCmpXchg(llvmX, eInt, dInt, maxAlign,
-                                              atomicOrdering, failOrdering);
+  llvm::AtomicCmpXchgInst *cmpXchg = builder.CreateAtomicCmpXchg(
+      llvmX, xCurr, dInt, maxAlign, atomicOrdering, failOrdering);
   cmpXchg->setWeak(isWeak);
-  return cmpXchg;
+  llvm::Value *oldSwap = builder.CreateExtractValue(cmpXchg, 0);
+  llvm::Value *okSwap = builder.CreateExtractValue(cmpXchg, 1);
+  builder.CreateBr(exitBB);
+
+  // Merge the swap and no-swap paths.
+  builder.SetInsertPoint(exitBB);
+  llvm::PHINode *oldIntPHI = builder.CreatePHI(intTy, 2, "cmplx.old.int");
+  oldIntPHI->addIncoming(oldSwap, swapBB);
+  oldIntPHI->addIncoming(xCurr, curBB);
+  llvm::PHINode *okPHI = builder.CreatePHI(builder.getInt1Ty(), 2, "cmplx.ok");
+  okPHI->addIncoming(okSwap, swapBB);
+  okPHI->addIncoming(builder.getFalse(), curBB);
+
+  // Reinterpret the old integer value as the complex struct via memory.
+  llvm::AllocaInst *oldAlloca =
+      builder.CreateAlloca(complexTy, nullptr, "cmplx.old");
+  oldAlloca->setAlignment(maxAlign);
+  builder.CreateAlignedStore(oldIntPHI, oldAlloca, maxAlign);
+  oldComplex = builder.CreateAlignedLoad(complexTy, oldAlloca, maxAlign,
+                                         "cmplx.old.val");
+  cmpOk = okPHI;
 }
 
 /// Holds the extracted comparison pattern information from an atomic compare
@@ -5687,25 +5734,17 @@ convertOmpAtomicCapture(omp::AtomicCaptureOp atomicCaptureOp,
     bool isFailOnly = atomicCaptureOp.getFailOnly();
 
     // Complex equality capture: x is struct-typed, which the OMPIRBuilder
-    // cannot handle, so emit a bitcast-to-integer cmpxchg (as in the
-    // non-capture complex path) and reconstruct the captured value from its
-    // result through memory. Complex only supports the == comparison.
+    // cannot handle, so emit an IEEE-754-correct cmpxchg (as in the non-capture
+    // complex path) and reconstruct the captured value from its result. The
+    // helper compares components with `fcmp oeq` so `-0.0 == +0.0` and `NaN`
+    // are handled as in the scalar float path. Complex only supports the ==
+    // comparison.
     if (llvmXElementType->isStructTy()) {
-      llvm::Align maxAlign;
-      llvm::AtomicCmpXchgInst *cmpXchg = emitComplexAtomicCmpXchg(
-          builder, llvmX, llvmXElementType, eVal, dVal, atomicOrdering,
-          atomicCompareOp.getWeak(), maxAlign);
-
-      llvm::Value *oldInt = builder.CreateExtractValue(cmpXchg, 0);
-      llvm::Value *cmpOk = builder.CreateExtractValue(cmpXchg, 1);
-
-      // Reinterpret the old integer value as the complex struct via memory.
-      llvm::AllocaInst *oldAlloca =
-          builder.CreateAlloca(llvmXElementType, nullptr, "cmplx.old");
-      oldAlloca->setAlignment(maxAlign);
-      builder.CreateAlignedStore(oldInt, oldAlloca, maxAlign);
-      llvm::Value *oldComplex = builder.CreateAlignedLoad(
-          llvmXElementType, oldAlloca, maxAlign, "cmplx.old.val");
+      llvm::Value *oldComplex = nullptr;
+      llvm::Value *cmpOk = nullptr;
+      emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
+                               atomicOrdering, atomicCompareOp.getWeak(),
+                               oldComplex, cmpOk);
 
       if (isFailOnly) {
         // v is written only when the compare fails (cmpOk == false).
@@ -6161,10 +6200,13 @@ convertOmpAtomicCompare(omp::AtomicCompareOp atomicCompareOp,
       dVal = materializeValue(yieldOp.getResults()[0]);
     }
 
-    llvm::Align maxAlign;
+    llvm::Value *oldComplex = nullptr;
+    llvm::Value *cmpOk = nullptr;
     emitComplexAtomicCmpXchg(builder, llvmX, llvmXElementType, eVal, dVal,
                              atomicOrdering, atomicCompareOp.getWeak(),
-                             maxAlign);
+                             oldComplex, cmpOk);
+    (void)oldComplex;
+    (void)cmpOk;
 
     // Emit flush after atomic compare if needed (for release, acq_rel,
     // seq_cst orderings).
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index b779d20f42d0c..63054056adadc 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -2619,14 +2619,19 @@ llvm.func @omp_atomic_compare(
     omp.yield(%sel1 : f32)
   }
 
-  // Complex equality  →  bitcasted integer cmpxchg with consistent alignment
-  // CHECK: %[[EALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
-  // CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN]]
-  // CHECK: store { float, float } %[[EC]], ptr %[[EALLOCA]], align [[ALIGN]]
-  // CHECK: %[[EINT:.*]] = load i64, ptr %[[EALLOCA]], align [[ALIGN]]
+  // Complex equality  →  component-wise IEEE-754 fcmp, then a cmpxchg that
+  // swaps using x's loaded bit-pattern as the comparand (so -0.0/+0.0 and NaN
+  // follow the scalar float semantics rather than a raw bitwise compare).
+  // CHECK: %[[DALLOCA:.*]] = alloca { float, float }, align [[ALIGN:[0-9]+]]
   // CHECK: store { float, float } %[[DC]], ptr %[[DALLOCA]], align [[ALIGN]]
   // CHECK: %[[DINT:.*]] = load i64, ptr %[[DALLOCA]], align [[ALIGN]]
-  // CHECK: cmpxchg ptr %[[XC]], i64 %[[EINT]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
+  // CHECK: %[[XLOAD:.*]] = load atomic i64, ptr %[[XC]] monotonic, align [[ALIGN]]
+  // CHECK: %[[REEQ:.*]] = fcmp oeq float %[[REX:.*]], %[[REE:.*]]
+  // CHECK: %[[IMEQ:.*]] = fcmp oeq float %[[IMX:.*]], %[[IME:.*]]
+  // CHECK: %[[CEQ:.*]] = and i1 %[[REEQ]], %[[IMEQ]]
+  // CHECK: br i1 %[[CEQ]], label %[[CSWAP:[^,]+]], label %{{.*}}
+  // CHECK: [[CSWAP]]:
+  // CHECK: cmpxchg ptr %[[XC]], i64 %[[XLOAD]], i64 %[[DINT]] monotonic monotonic, align [[ALIGN]]
   omp.atomic.compare %xc : !llvm.ptr {
   ^bb0(%xval : !llvm.struct<(f32, f32)>):
     %re_x = llvm.extractvalue %xval[0] : !llvm.struct<(f32, f32)>

>From a446fe321fd0023edecfbc3734cc761a61ff7fc0 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Tue, 21 Jul 2026 02:05:43 -0500
Subject: [PATCH 11/11] Correcting a code formatting error.

---
 .../LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp       | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index fdc8f09fe5a2d..69b722582d785 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -5423,8 +5423,8 @@ static ComplexComparePattern detectComplexCompareEq(Block &block) {
 }
 
 /// Emit an IEEE-754-correct `cmpxchg` for a complex (struct-typed) atomic
-/// compare with `fcmp oeq`. The old value of X is returned (as the complex struct
-/// type) in \p oldComplex and the success flag (i1) in \p cmpOk.
+/// compare with `fcmp oeq`. The old value of X is returned (as the complex
+/// struct type) in \p oldComplex and the success flag (i1) in \p cmpOk.
 static void emitComplexAtomicCmpXchg(llvm::IRBuilderBase &builder,
                                      llvm::Value *llvmX, llvm::Type *complexTy,
                                      llvm::Value *eVal, llvm::Value *dVal,


From llvmlistbot at llvm.org  Tue Jul 21 00:47:54 2026
From: llvmlistbot at llvm.org (Felipe de Azevedo Piovezan)
Date: Tue, 21 Jul 2026 00:47:54 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5f242a.05203383.10ce9e.fc90@mx.google.com>


felipepiovezan wrote:

What is the problem this PR is trying to solve? This seems like a pretty big extension to the DIExpression language, and it's not clear what the motivation is.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 00:55:03 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 00:55:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (PR #210902)
Message-ID: <llvm.org/llvm/llvm-project/pull/210902@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/210902

Update both:
  * CastAwayExtractStridedSliceLeadingOneDim
  * CastAwayInsertStridedSliceLeadingOneDim

to use vector.shape_cast, rather than vector.extract and
vector.broadcast, as the canonical form for stripping unit dimensions.

This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>


>From a56cf5f3e0ff2603c0df4227fb497cda3c18d4aa Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Tue, 21 Jul 2026 07:45:14 +0000
Subject: [PATCH] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim

Update both:
  * CastAwayExtractStridedSliceLeadingOneDim
  * CastAwayInsertStridedSliceLeadingOneDim

to use vector.shape_cast, rather than vector.extract and
vector.broadcast, as the canonical form for stripping unit dimensions.

This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>
---
 .../Transforms/VectorDropLeadUnitDim.cpp      | 16 +++++-----
 .../vector-dropleadunitdim-transforms.mlir    | 31 +++++++++----------
 2 files changed, 23 insertions(+), 24 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index 26a702ef0f512..d86e9f224bf9d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -78,8 +78,8 @@ struct CastAwayExtractStridedSliceLeadingOneDim
 
     Location loc = extractOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, extractOp.getSource(), splatZero(dropCount));
+    Value newSrcVector = rewriter.createOrFold<ShapeCastOp>(
+        loc, newSrcType, extractOp.getSource());
 
     // The offsets/sizes/strides attribute can have a less number of elements
     // than the input vector's rank: it is meant for the leading dimensions.
@@ -94,7 +94,7 @@ struct CastAwayExtractStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newOffsets, newSizes,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(extractOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(extractOp, oldDstType,
                                                      newExtractOp);
 
     return success();
@@ -122,10 +122,10 @@ struct CastAwayInsertStridedSliceLeadingOneDim
     // Trim leading one dimensions from both operands.
     Location loc = insertOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newSrcType, insertOp.getValueToStore());
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     auto newOffsets = rewriter.getArrayAttr(
         insertOp.getOffsets().getValue().take_back(newDstType.getRank()));
@@ -136,7 +136,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newDstVector, newOffsets,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..41cd2846fd752 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -270,44 +270,45 @@ func.func @cast_away_contraction_leading_one_dims_to_dot_product(%arg0: vector<6
 }
 
 // -----
+
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims
 func.func @cast_away_extract_strided_slice_leading_one_dims(%arg0: vector<1x8x8xf16>) -> vector<1x1x8xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x8xf16> to vector<1x8xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x8xf16> to vector<1x1x8xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims_scalable
 func.func @cast_away_extract_strided_slice_leading_one_dims_scalable(%arg0: vector<1x8x[8]xf16>) -> vector<1x1x[8]xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x[8]xf16> to vector<1x[8]xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x[8]xf16> to vector<1x1x[8]xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x[8]xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims
 func.func @cast_away_insert_strided_slice_leading_one_dims(%arg0: vector<1x8xf16>, %arg1: vector<1x8x8xf16>) -> vector<1x8x8xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8xf16> from vector<1x8xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8xf16> to vector<8xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<8xf16> into vector<8x8xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x8xf16> into vector<1x8x8xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_scalable
 func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vector<1x[8]xf16>, %arg1: vector<1x8x[8]xf16>) -> vector<1x8x[8]xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<[8]xf16> from vector<1x[8]xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x[8]xf16> to vector<[8]xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16> 
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<[8]xf16> into vector<8x[8]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[8]xf16> into vector<1x8x[8]xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x[8]xf16>
 }
@@ -315,20 +316,18 @@ func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vecto
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x1xf16>, %{{.+}}: vector<1x1x1xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element(%arg0: vector<1x1xf16>, %arg1: vector<1x1x1xf16>) -> vector<1x1x1xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<1xf16> from vector<1x1xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<1xf16> to vector<1x1x1xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x1xf16> to vector<1x1x1xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x1xf16> into vector<1x1x1xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x1xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x[1]xf16>, %{{.+}}: vector<1x1x[1]xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable(%arg0: vector<1x[1]xf16>, %arg1: vector<1x1x[1]xf16>) -> vector<1x1x[1]xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<[1]xf16> from vector<1x[1]xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<[1]xf16> to vector<1x1x[1]xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x[1]xf16> to vector<1x1x[1]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[1]xf16> into vector<1x1x[1]xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x[1]xf16>
 }
 


From llvmlistbot at llvm.org  Tue Jul 21 00:55:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 00:55:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (PR #210902)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210902@github.com>
Message-ID: <6a5f25fc.e27fddb3.ef63c.78dd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Update both:
  * CastAwayExtractStridedSliceLeadingOneDim
  * CastAwayInsertStridedSliceLeadingOneDim

to use vector.shape_cast, rather than vector.extract and
vector.broadcast, as the canonical form for stripping unit dimensions.

This change was originally implemented by @<!-- -->krzysz00 in #<!-- -->196206, but was
subsequently reverted in #<!-- -->199546. This PR intentionally restores only a
subset of #<!-- -->196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak &lt;Krzysztof.Drewniak@<!-- -->amd.com&gt;


---
Full diff: https://github.com/llvm/llvm-project/pull/210902.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp (+8-8) 
- (modified) mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir (+15-16) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index 26a702ef0f512..d86e9f224bf9d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -78,8 +78,8 @@ struct CastAwayExtractStridedSliceLeadingOneDim
 
     Location loc = extractOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, extractOp.getSource(), splatZero(dropCount));
+    Value newSrcVector = rewriter.createOrFold<ShapeCastOp>(
+        loc, newSrcType, extractOp.getSource());
 
     // The offsets/sizes/strides attribute can have a less number of elements
     // than the input vector's rank: it is meant for the leading dimensions.
@@ -94,7 +94,7 @@ struct CastAwayExtractStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newOffsets, newSizes,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(extractOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(extractOp, oldDstType,
                                                      newExtractOp);
 
     return success();
@@ -122,10 +122,10 @@ struct CastAwayInsertStridedSliceLeadingOneDim
     // Trim leading one dimensions from both operands.
     Location loc = insertOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newSrcType, insertOp.getValueToStore());
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     auto newOffsets = rewriter.getArrayAttr(
         insertOp.getOffsets().getValue().take_back(newDstType.getRank()));
@@ -136,7 +136,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newDstVector, newOffsets,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..41cd2846fd752 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -270,44 +270,45 @@ func.func @cast_away_contraction_leading_one_dims_to_dot_product(%arg0: vector<6
 }
 
 // -----
+
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims
 func.func @cast_away_extract_strided_slice_leading_one_dims(%arg0: vector<1x8x8xf16>) -> vector<1x1x8xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x8xf16> to vector<1x8xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x8xf16> to vector<1x1x8xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims_scalable
 func.func @cast_away_extract_strided_slice_leading_one_dims_scalable(%arg0: vector<1x8x[8]xf16>) -> vector<1x1x[8]xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x[8]xf16> to vector<1x[8]xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x[8]xf16> to vector<1x1x[8]xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x[8]xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims
 func.func @cast_away_insert_strided_slice_leading_one_dims(%arg0: vector<1x8xf16>, %arg1: vector<1x8x8xf16>) -> vector<1x8x8xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8xf16> from vector<1x8xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8xf16> to vector<8xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<8xf16> into vector<8x8xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x8xf16> into vector<1x8x8xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_scalable
 func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vector<1x[8]xf16>, %arg1: vector<1x8x[8]xf16>) -> vector<1x8x[8]xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<[8]xf16> from vector<1x[8]xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x[8]xf16> to vector<[8]xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16> 
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<[8]xf16> into vector<8x[8]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[8]xf16> into vector<1x8x[8]xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x[8]xf16>
 }
@@ -315,20 +316,18 @@ func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vecto
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x1xf16>, %{{.+}}: vector<1x1x1xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element(%arg0: vector<1x1xf16>, %arg1: vector<1x1x1xf16>) -> vector<1x1x1xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<1xf16> from vector<1x1xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<1xf16> to vector<1x1x1xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x1xf16> to vector<1x1x1xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x1xf16> into vector<1x1x1xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x1xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x[1]xf16>, %{{.+}}: vector<1x1x[1]xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable(%arg0: vector<1x[1]xf16>, %arg1: vector<1x1x[1]xf16>) -> vector<1x1x[1]xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<[1]xf16> from vector<1x[1]xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<[1]xf16> to vector<1x1x[1]xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x[1]xf16> to vector<1x1x[1]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[1]xf16> into vector<1x1x[1]xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x[1]xf16>
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/210902

From llvmlistbot at llvm.org  Tue Jul 21 00:55:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 00:55:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (PR #210902)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210902@github.com>
Message-ID: <6a5f25fc.32420449.289d.9246@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-vector

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Update both:
  * CastAwayExtractStridedSliceLeadingOneDim
  * CastAwayInsertStridedSliceLeadingOneDim

to use vector.shape_cast, rather than vector.extract and
vector.broadcast, as the canonical form for stripping unit dimensions.

This change was originally implemented by @<!-- -->krzysz00 in #<!-- -->196206, but was
subsequently reverted in #<!-- -->199546. This PR intentionally restores only a
subset of #<!-- -->196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak &lt;Krzysztof.Drewniak@<!-- -->amd.com&gt;


---
Full diff: https://github.com/llvm/llvm-project/pull/210902.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp (+8-8) 
- (modified) mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir (+15-16) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index 26a702ef0f512..d86e9f224bf9d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -78,8 +78,8 @@ struct CastAwayExtractStridedSliceLeadingOneDim
 
     Location loc = extractOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, extractOp.getSource(), splatZero(dropCount));
+    Value newSrcVector = rewriter.createOrFold<ShapeCastOp>(
+        loc, newSrcType, extractOp.getSource());
 
     // The offsets/sizes/strides attribute can have a less number of elements
     // than the input vector's rank: it is meant for the leading dimensions.
@@ -94,7 +94,7 @@ struct CastAwayExtractStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newOffsets, newSizes,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(extractOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(extractOp, oldDstType,
                                                      newExtractOp);
 
     return success();
@@ -122,10 +122,10 @@ struct CastAwayInsertStridedSliceLeadingOneDim
     // Trim leading one dimensions from both operands.
     Location loc = insertOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newSrcType, insertOp.getValueToStore());
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     auto newOffsets = rewriter.getArrayAttr(
         insertOp.getOffsets().getValue().take_back(newDstType.getRank()));
@@ -136,7 +136,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newDstVector, newOffsets,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..41cd2846fd752 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -270,44 +270,45 @@ func.func @cast_away_contraction_leading_one_dims_to_dot_product(%arg0: vector<6
 }
 
 // -----
+
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims
 func.func @cast_away_extract_strided_slice_leading_one_dims(%arg0: vector<1x8x8xf16>) -> vector<1x1x8xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x8xf16> to vector<1x8xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x8xf16> to vector<1x1x8xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims_scalable
 func.func @cast_away_extract_strided_slice_leading_one_dims_scalable(%arg0: vector<1x8x[8]xf16>) -> vector<1x1x[8]xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x[8]xf16> to vector<1x[8]xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x[8]xf16> to vector<1x1x[8]xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x[8]xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims
 func.func @cast_away_insert_strided_slice_leading_one_dims(%arg0: vector<1x8xf16>, %arg1: vector<1x8x8xf16>) -> vector<1x8x8xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8xf16> from vector<1x8xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8xf16> to vector<8xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<8xf16> into vector<8x8xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x8xf16> into vector<1x8x8xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_scalable
 func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vector<1x[8]xf16>, %arg1: vector<1x8x[8]xf16>) -> vector<1x8x[8]xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<[8]xf16> from vector<1x[8]xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x[8]xf16> to vector<[8]xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16> 
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<[8]xf16> into vector<8x[8]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[8]xf16> into vector<1x8x[8]xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x[8]xf16>
 }
@@ -315,20 +316,18 @@ func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vecto
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x1xf16>, %{{.+}}: vector<1x1x1xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element(%arg0: vector<1x1xf16>, %arg1: vector<1x1x1xf16>) -> vector<1x1x1xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<1xf16> from vector<1x1xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<1xf16> to vector<1x1x1xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x1xf16> to vector<1x1x1xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x1xf16> into vector<1x1x1xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x1xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x[1]xf16>, %{{.+}}: vector<1x1x[1]xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable(%arg0: vector<1x[1]xf16>, %arg1: vector<1x1x[1]xf16>) -> vector<1x1x[1]xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<[1]xf16> from vector<1x[1]xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<[1]xf16> to vector<1x1x[1]xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x[1]xf16> to vector<1x1x[1]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[1]xf16> into vector<1x1x[1]xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x[1]xf16>
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/210902

From llvmlistbot at llvm.org  Tue Jul 21 01:08:33 2026
From: llvmlistbot at llvm.org (Kigyosi Alexandru)
Date: Tue, 21 Jul 2026 01:08:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a5f2901.e55dc341.1c6a0.8b17@mx.google.com>


https://github.com/akigyosi updated https://github.com/llvm/llvm-project/pull/209197

>From 409df4d8fa956710f8d4c62a6964abefa6497829 Mon Sep 17 00:00:00 2001
From: Alexandru Kigyosi <alexandru.kigyosi at intel.com>
Date: Mon, 13 Jul 2026 14:33:57 +0000
Subject: [PATCH 1/3] rerun control for actions

---
 mlir/include/mlir/Debug/ExecutionContext.h    |  6 ++++-
 mlir/lib/Debug/ExecutionContext.cpp           | 22 ++++++++++++++-
 mlir/unittests/Debug/ExecutionContextTest.cpp | 27 +++++++++++++++++++
 3 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
index fbad04b9a2f1d..e42d61087f22d 100644
--- a/mlir/include/mlir/Debug/ExecutionContext.h
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -65,7 +65,8 @@ class ExecutionContext {
   /// - Finish: The action is executed and the execution is paused only when we
   ///           reach the parent/enclosing operation. If there are no enclosing
   ///           operation, the execution continues without stopping.
-  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5 };
+  /// - Rerun: The action gets executed again immediately after it finishes.
+  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5, Rerun = 6 };
 
   /// The type of the callback that is used to control the execution.
   /// The callback is passed the current action.
@@ -134,6 +135,9 @@ class ExecutionContext {
 
   /// The list of managers that are queried for breakpoints.
   SmallVector<BreakpointManager *> breakpoints;
+
+  /// Stack of depths of actions that should be rerun.
+  SmallVector<int> rerunControlStack;
 };
 
 } // namespace tracing
diff --git a/mlir/lib/Debug/ExecutionContext.cpp b/mlir/lib/Debug/ExecutionContext.cpp
index 7fc5c165d391e..cc72bac549ab0 100644
--- a/mlir/lib/Debug/ExecutionContext.cpp
+++ b/mlir/lib/Debug/ExecutionContext.cpp
@@ -84,10 +84,25 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
     case ExecutionContext::Finish:
       depthToBreak = depth - 1;
       return true;
+    case ExecutionContext::Rerun:
+      depthToBreak = std::nullopt;
+      rerunControlStack.push_back(depth);
+      return true;
     }
     llvm::report_fatal_error("Unknown control request");
   };
 
+  auto rerunCurrentActionIfRequested = [&]() -> bool {
+    if (!rerunControlStack.empty() && rerunControlStack.back() == depth) {
+      // If the user requested to rerun this action, we do it here.
+      rerunControlStack.pop_back();
+      actionStack = info.getParent();
+      (*this)(transform, action);
+      return true;
+    }
+    return false;
+  };
+
   // Try to find a breakpoint that would hit on this action.
   // Right now there is no way to collect them all, we stop at the first one.
   for (auto *breakpointManager : breakpoints) {
@@ -116,6 +131,11 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
       observer->afterExecute(actionStack);
   }
 
-  if (depthToBreak && depth <= depthToBreak)
+  if (rerunCurrentActionIfRequested())
+    return;
+
+  if (depthToBreak && depth <= depthToBreak) {
     handleUserInput();
+    rerunCurrentActionIfRequested();
+  }
 }
diff --git a/mlir/unittests/Debug/ExecutionContextTest.cpp b/mlir/unittests/Debug/ExecutionContextTest.cpp
index 642adff51002a..16c785b6f826e 100644
--- a/mlir/unittests/Debug/ExecutionContextTest.cpp
+++ b/mlir/unittests/Debug/ExecutionContextTest.cpp
@@ -349,4 +349,31 @@ TEST(ExecutionContext, EnableDisableBreakpointOnCallback) {
   executionCtx(original, DebuggerAction{});
   EXPECT_EQ(counter, 4);
 }
+
+TEST(ExecutionContext, RerunRequestedFromPostActionCallback) {
+  // If rerun is requested from the post-action callback path,
+  // the same action must be rerun immediately.
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Next, ExecutionContext::Rerun, ExecutionContext::Apply};
+  int idx = 0;
+  int callbackCounter = 0;
+  int executionCounter = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++callbackCounter;
+    EXPECT_EQ(backtrace->getAction().getTag(), DebuggerAction::tag);
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  auto callback = [&]() { ++executionCounter; };
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(callbackCounter, 3);
+  EXPECT_EQ(executionCounter, 2);
+}
 } // namespace

>From 47e9a4cf1478ae197aa12b4f96413e6a6af7b47d Mon Sep 17 00:00:00 2001
From: Alexandru Kigyosi <alexandru.kigyosi at intel.com>
Date: Mon, 13 Jul 2026 14:52:37 +0000
Subject: [PATCH 2/3] apply clang format

---
 mlir/include/mlir/Debug/ExecutionContext.h | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
index e42d61087f22d..93c26e0ce5f82 100644
--- a/mlir/include/mlir/Debug/ExecutionContext.h
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -66,7 +66,14 @@ class ExecutionContext {
   ///           reach the parent/enclosing operation. If there are no enclosing
   ///           operation, the execution continues without stopping.
   /// - Rerun: The action gets executed again immediately after it finishes.
-  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5, Rerun = 6 };
+  enum Control {
+    Apply = 1,
+    Skip = 2,
+    Step = 3,
+    Next = 4,
+    Finish = 5,
+    Rerun = 6
+  };
 
   /// The type of the callback that is used to control the execution.
   /// The callback is passed the current action.

>From 5c211435c096f16a6883f0f581feca09731c3ce4 Mon Sep 17 00:00:00 2001
From: Alexandru Kigyosi <alexandru.kigyosi at intel.com>
Date: Tue, 21 Jul 2026 08:07:15 +0000
Subject: [PATCH 3/3] add nested action test for rerun

---
 mlir/unittests/Debug/ExecutionContextTest.cpp | 40 +++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/mlir/unittests/Debug/ExecutionContextTest.cpp b/mlir/unittests/Debug/ExecutionContextTest.cpp
index 16c785b6f826e..1c442d4aa1acc 100644
--- a/mlir/unittests/Debug/ExecutionContextTest.cpp
+++ b/mlir/unittests/Debug/ExecutionContextTest.cpp
@@ -376,4 +376,44 @@ TEST(ExecutionContext, RerunRequestedFromPostActionCallback) {
   EXPECT_EQ(callbackCounter, 3);
   EXPECT_EQ(executionCounter, 2);
 }
+
+TEST(ExecutionContext, RerunStackWithNestedActions) {
+  // Request rerun at an outer depth, then request rerun again at an inner
+  // depth before the outer rerun is consumed.
+  int debuggerHits = 0;
+  int otherHits = 0;
+  int debuggerExecutions = 0;
+  int otherExecutions = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    StringRef tag = backtrace->getAction().getTag();
+    if (tag == DebuggerAction::tag)
+      return ++debuggerHits == 1 ? ExecutionContext::Rerun
+                                 : ExecutionContext::Apply;
+    if (tag == OtherAction::tag)
+      return ++otherHits == 1 ? ExecutionContext::Rerun
+                              : ExecutionContext::Apply;
+    ADD_FAILURE();
+    return ExecutionContext::Apply;
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+  simpleManager.addBreakpoint(OtherAction::tag);
+
+  auto nested = [&]() { ++otherExecutions; };
+  auto original = [&]() {
+    ++debuggerExecutions;
+    executionCtx(nested, OtherAction{});
+  };
+
+  executionCtx(original, DebuggerAction{});
+
+  EXPECT_EQ(debuggerHits, 2);
+  EXPECT_EQ(otherHits, 3);
+  EXPECT_EQ(debuggerExecutions, 2);
+  EXPECT_EQ(otherExecutions, 3);
+}
 } // namespace


From llvmlistbot at llvm.org  Tue Jul 21 01:13:27 2026
From: llvmlistbot at llvm.org (Felipe de Azevedo Piovezan)
Date: Tue, 21 Jul 2026 01:13:27 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5f2a27.1b0626d0.59100.8bae@mx.google.com>


felipepiovezan wrote:

Also note that if this PR were to go forward, it would need to be broken up into smaller components: maybe just the IR component, then the MIR component, etc.
It's also missing a producer for these (part of what I meant by "motivation").

I strongly recommend you write an RFC on the forums before going forward wit this

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 01:23:54 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 01:23:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f2c9a.eb165e5c.4506c.de15@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 02:04:08 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Tue, 21 Jul 2026 02:04:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ods] Fix string interpolation at end
 of description (PR #209744)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209744@github.com>
Message-ID: <6a5f3608.66c0679a.24976c.dd9d@mx.google.com>


https://github.com/jpienaar approved this pull request.

Makes sense to me, thanks!

https://github.com/llvm/llvm-project/pull/209744

From llvmlistbot at llvm.org  Tue Jul 21 02:05:34 2026
From: llvmlistbot at llvm.org (Jacques Pienaar)
Date: Tue, 21 Jul 2026 02:05:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa][WIP] More informative error
 messages on block scaled types (PR #209736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209736@github.com>
Message-ID: <6a5f365e.0c05e68e.3c2f5d.be7a@mx.google.com>


================
@@ -172,35 +172,42 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::"
+            "verifyBlockScaledTensorType($_self))">;
+
+// Analogous to ShapedContainerType, but with additional 'summary' information.
+// This is used to provide more informative error messages when a type predicate
+// is false.
+class TosaShapedContainerType<list<Type> allowedTypes, Pred containerPred, string descr, string cppType = "::mlir::Type"> :
+    Type<And<[containerPred,
+              Concat<"[](::mlir::Type elementType) { return ",
+                SubstLeaves<"$_self", "elementType",
+                AnyTypeOf<allowedTypes>.predicate>,
+                "; }(::llvm::cast<::mlir::ShapedType>($_self).getElementType())">]>,
----------------
jpienaar wrote:

You can use `getElementTypeOrSelf($self)` just to make this more concise.

https://github.com/llvm/llvm-project/pull/209736

From llvmlistbot at llvm.org  Tue Jul 21 02:08:28 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 02:08:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f370c.66c0679a.24976c.7bfb@mx.google.com>


================

----------------
banach-space wrote:

Great catch, thanks!

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 02:08:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 02:08:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f371a.a1accbb4.5eca2.0aeb@mx.google.com>


================

----------------
banach-space wrote:

Yes, thank you 🙏🏻 

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 02:52:58 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 21 Jul 2026 02:52:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a5f417a.7d702ed2.13d102.e1b8@mx.google.com>


https://github.com/joker-eph approved this pull request.


https://github.com/llvm/llvm-project/pull/209197

From llvmlistbot at llvm.org  Tue Jul 21 02:53:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 02:53:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] d8d1d44 - [MLIR] Rerun control for actions in
 execution context (#209197)
Message-ID: <6a5f4193.7270d2fd.a9bd6.4f8a@mx.google.com>


Author: Kigyosi Alexandru
Date: 2026-07-21T11:53:17+02:00
New Revision: d8d1d444a2a69293292f10f19691da05207630f7

URL: https://github.com/llvm/llvm-project/commit/d8d1d444a2a69293292f10f19691da05207630f7
DIFF: https://github.com/llvm/llvm-project/commit/d8d1d444a2a69293292f10f19691da05207630f7.diff

LOG: [MLIR] Rerun control for actions in execution context (#209197)

Adds a Rerun control value to ExecutionContext::Control that allows the
re-execution of the current action immediately after it completes,
without restarting the full compilation pipeline. This is analogous to
GDB's ability to restart execution from a breakpoint. When the callback
returns Rerun, the action is executed normally, then re-dispatched
through the full ExecutionContext::operator() pipeline, including
breakpoint matching, so the user gets a fresh opportunity to inspect or
control the re-execution. As a practical usage example, a breakpoint +
an observer can be added, to save and restore IR between runs to check
if each run produces the same IR or something different each time. A
depth-keyed structure is used, so rerun requests survive nested action
dispatch and are consumed by the correct stack frame.

Added: 
    

Modified: 
    mlir/include/mlir/Debug/ExecutionContext.h
    mlir/lib/Debug/ExecutionContext.cpp
    mlir/unittests/Debug/ExecutionContextTest.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Debug/ExecutionContext.h b/mlir/include/mlir/Debug/ExecutionContext.h
index fbad04b9a2f1d..93c26e0ce5f82 100644
--- a/mlir/include/mlir/Debug/ExecutionContext.h
+++ b/mlir/include/mlir/Debug/ExecutionContext.h
@@ -65,7 +65,15 @@ class ExecutionContext {
   /// - Finish: The action is executed and the execution is paused only when we
   ///           reach the parent/enclosing operation. If there are no enclosing
   ///           operation, the execution continues without stopping.
-  enum Control { Apply = 1, Skip = 2, Step = 3, Next = 4, Finish = 5 };
+  /// - Rerun: The action gets executed again immediately after it finishes.
+  enum Control {
+    Apply = 1,
+    Skip = 2,
+    Step = 3,
+    Next = 4,
+    Finish = 5,
+    Rerun = 6
+  };
 
   /// The type of the callback that is used to control the execution.
   /// The callback is passed the current action.
@@ -134,6 +142,9 @@ class ExecutionContext {
 
   /// The list of managers that are queried for breakpoints.
   SmallVector<BreakpointManager *> breakpoints;
+
+  /// Stack of depths of actions that should be rerun.
+  SmallVector<int> rerunControlStack;
 };
 
 } // namespace tracing

diff  --git a/mlir/lib/Debug/ExecutionContext.cpp b/mlir/lib/Debug/ExecutionContext.cpp
index 7fc5c165d391e..cc72bac549ab0 100644
--- a/mlir/lib/Debug/ExecutionContext.cpp
+++ b/mlir/lib/Debug/ExecutionContext.cpp
@@ -84,10 +84,25 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
     case ExecutionContext::Finish:
       depthToBreak = depth - 1;
       return true;
+    case ExecutionContext::Rerun:
+      depthToBreak = std::nullopt;
+      rerunControlStack.push_back(depth);
+      return true;
     }
     llvm::report_fatal_error("Unknown control request");
   };
 
+  auto rerunCurrentActionIfRequested = [&]() -> bool {
+    if (!rerunControlStack.empty() && rerunControlStack.back() == depth) {
+      // If the user requested to rerun this action, we do it here.
+      rerunControlStack.pop_back();
+      actionStack = info.getParent();
+      (*this)(transform, action);
+      return true;
+    }
+    return false;
+  };
+
   // Try to find a breakpoint that would hit on this action.
   // Right now there is no way to collect them all, we stop at the first one.
   for (auto *breakpointManager : breakpoints) {
@@ -116,6 +131,11 @@ void ExecutionContext::operator()(llvm::function_ref<void()> transform,
       observer->afterExecute(actionStack);
   }
 
-  if (depthToBreak && depth <= depthToBreak)
+  if (rerunCurrentActionIfRequested())
+    return;
+
+  if (depthToBreak && depth <= depthToBreak) {
     handleUserInput();
+    rerunCurrentActionIfRequested();
+  }
 }

diff  --git a/mlir/unittests/Debug/ExecutionContextTest.cpp b/mlir/unittests/Debug/ExecutionContextTest.cpp
index 642adff51002a..1c442d4aa1acc 100644
--- a/mlir/unittests/Debug/ExecutionContextTest.cpp
+++ b/mlir/unittests/Debug/ExecutionContextTest.cpp
@@ -349,4 +349,71 @@ TEST(ExecutionContext, EnableDisableBreakpointOnCallback) {
   executionCtx(original, DebuggerAction{});
   EXPECT_EQ(counter, 4);
 }
+
+TEST(ExecutionContext, RerunRequestedFromPostActionCallback) {
+  // If rerun is requested from the post-action callback path,
+  // the same action must be rerun immediately.
+  std::vector<ExecutionContext::Control> controlSequence = {
+      ExecutionContext::Next, ExecutionContext::Rerun, ExecutionContext::Apply};
+  int idx = 0;
+  int callbackCounter = 0;
+  int executionCounter = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    ++callbackCounter;
+    EXPECT_EQ(backtrace->getAction().getTag(), DebuggerAction::tag);
+    return controlSequence[idx++];
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+
+  auto callback = [&]() { ++executionCounter; };
+
+  executionCtx(callback, DebuggerAction{});
+  EXPECT_EQ(callbackCounter, 3);
+  EXPECT_EQ(executionCounter, 2);
+}
+
+TEST(ExecutionContext, RerunStackWithNestedActions) {
+  // Request rerun at an outer depth, then request rerun again at an inner
+  // depth before the outer rerun is consumed.
+  int debuggerHits = 0;
+  int otherHits = 0;
+  int debuggerExecutions = 0;
+  int otherExecutions = 0;
+
+  auto onBreakpoint = [&](const ActionActiveStack *backtrace) {
+    StringRef tag = backtrace->getAction().getTag();
+    if (tag == DebuggerAction::tag)
+      return ++debuggerHits == 1 ? ExecutionContext::Rerun
+                                 : ExecutionContext::Apply;
+    if (tag == OtherAction::tag)
+      return ++otherHits == 1 ? ExecutionContext::Rerun
+                              : ExecutionContext::Apply;
+    ADD_FAILURE();
+    return ExecutionContext::Apply;
+  };
+
+  TagBreakpointManager simpleManager;
+  ExecutionContext executionCtx(onBreakpoint);
+  executionCtx.addBreakpointManager(&simpleManager);
+  simpleManager.addBreakpoint(DebuggerAction::tag);
+  simpleManager.addBreakpoint(OtherAction::tag);
+
+  auto nested = [&]() { ++otherExecutions; };
+  auto original = [&]() {
+    ++debuggerExecutions;
+    executionCtx(nested, OtherAction{});
+  };
+
+  executionCtx(original, DebuggerAction{});
+
+  EXPECT_EQ(debuggerHits, 2);
+  EXPECT_EQ(otherHits, 3);
+  EXPECT_EQ(debuggerExecutions, 2);
+  EXPECT_EQ(otherExecutions, 3);
+}
 } // namespace


        

From llvmlistbot at llvm.org  Tue Jul 21 02:53:26 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 21 Jul 2026 02:53:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Rerun control for actions in execution
	context (PR #209197)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209197@github.com>
Message-ID: <6a5f4196.eb165e5c.4506c.2a78@mx.google.com>


https://github.com/joker-eph closed https://github.com/llvm/llvm-project/pull/209197

From llvmlistbot at llvm.org  Tue Jul 21 03:18:35 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Tue, 21 Jul 2026 03:18:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
Message-ID: <llvm.org/llvm/llvm-project/pull/210931@github.com>


https://github.com/Wolfram70 created https://github.com/llvm/llvm-project/pull/210931

This change adds the `store.async` op to the NVVM dialect to
perform asynchronous stores to global or shared-cluster address
spaces.

PTX Spec References:
1. [`st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-async)
2. [`multimem.st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-st-async)

>From 29bdc0e1de0625b05696735951ab58252028754b Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Fri, 17 Jul 2026 06:48:11 +0000
Subject: [PATCH 1/2] [MLIR][NVVM] Add asynchronous store Op

This change adds the `store.async` op to the NVVM dialect to
perform asynchronous stores to global or shared cluster address
spaces.

PTX Spec References:
1. `st.async`: https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-async
2. `multimem.st.async`: https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-st-async
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td | 14 ++++
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td   | 19 +++++
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp    | 78 +++++++++++++++++++
 mlir/test/Target/LLVMIR/nvvm/store_async.mlir | 58 ++++++++++++++
 .../LLVMIR/nvvm/store_async_invalid.mlir      | 63 +++++++++++++++
 5 files changed, 232 insertions(+)
 create mode 100644 mlir/test/Target/LLVMIR/nvvm/store_async.mlir
 create mode 100644 mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
index 42d196c5662d1..9d6a7f3bf1a6c 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
@@ -69,4 +69,18 @@ def SaturationModeAttr : EnumAttr<NVVM_Dialect, SaturationMode, "sat_mode"> {
  let assemblyFormat = "`<` $value `>`";
 }
 
+def AsyncStoreScopeNone : I32EnumCase<"NONE", 0, "none">;
+def AsyncStoreScopeSys : I32EnumCase<"SYS", 1, "sys">;
+def AsyncStoreScopeGpu : I32EnumCase<"GPU", 2, "gpu">;
+
+def AsyncStoreScope :
+  I32Enum<"AsyncStoreScope", "NVVM Asynchronous Store Scope",
+    [AsyncStoreScopeNone, AsyncStoreScopeSys, AsyncStoreScopeGpu]> {
+  let cppNamespace = "::mlir::NVVM";
+}
+
+def AsyncStoreScopeAttr : EnumAttr<NVVM_Dialect, AsyncStoreScope, 
+                                   "async_store_scope"> {
+  let assemblyFormat = "`<` $value `>`";
+}
 #endif // NVVMIR_ENUMS
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..d905a5b212450 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
+  let summary = "Asynchronous Store Op";
+  let arguments = (ins AnyTypeOf<[LLVM_PointerGlobal, 
+                                  LLVM_PointerSharedCluster]>:$addr,
+                       AnyTypeOf<[I8, I16, I32, I64, I128]>:$value, 
+                       Optional<LLVM_PointerSharedCluster>:$mbarrier,
+                       DefaultValuedAttr<AsyncStoreScopeAttr, 
+                                         "AsyncStoreScope::NONE">:$scope,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_multimem,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_mmio);
+  let results = (outs );
+  let assemblyFormat = "$addr `,` $value (`,` `mbarrier` `=` $mbarrier^)? attr-dict `:` type($addr) `,` type($value) (`,` type($mbarrier)^)?";
+  let hasVerifier = 1;
+}
+
 def NVVM_Exit : NVVM_Op<"exit"> {
   let summary = "Exit Op";
   let description = [{
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index b29cba96a410f..b18f9a96debc9 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
+
+    if (scope != AsyncStoreScope::NONE)
+      return emitOpError("scope is not supported for async store to shared "
+                         "cluster address space");
+  }
+
+  return success();
+}
+
 LogicalResult PMEventOp::verify() {
   auto eventId = getEventId();
   auto maskedEventId = getMaskedEventId();
@@ -3662,6 +3706,40 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
+
+  switch (addrSpace) {
+  case NVVMMemorySpace::Global:
+    if (scope == AsyncStoreScope::SYS) {
+      if (thisOp.getIsMmio())
+        return {llvm::Intrinsic::nvvm_st_async_mmio_sys, {addr, value}};
+      else
+        return {llvm::Intrinsic::nvvm_st_async_sys, {addr, value, isMultimem}};
+    } else if (scope == AsyncStoreScope::GPU) {
+      return {llvm::Intrinsic::nvvm_st_async_gpu, {addr, value, isMultimem}};
+    }
+  case NVVMMemorySpace::SharedCluster:
+    return {llvm::Intrinsic::nvvm_st_async, {addr, value, mbarrier}};
+  default:
+    llvm_unreachable("unsupported address space");
+    return {llvm::Intrinsic::not_intrinsic, {}};
+  }
+}
+
 mlir::NVVM::IDArgPair
 PMEventOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
                                  llvm::IRBuilderBase &builder) {
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
new file mode 100644
index 0000000000000..8e59f48c987ac
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// CHECK-LABEL: define void @st_async_global_sys
+llvm.func @st_async_global_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_gpu
+llvm.func @st_async_global_gpu(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_multimem
+llvm.func @st_async_global_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_multimem = true} : !llvm.ptr<1>, i32
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_mmio
+llvm.func @st_async_global_mmio(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.mmio.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}})
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_types
+llvm.func @st_async_global_types(%addr: !llvm.ptr<1>, %v8: i8, %v16: i16, %v64: i64) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i8(ptr addrspace(1) %{{.*}}, i8 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v8 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i8
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i16(ptr addrspace(1) %{{.*}}, i16 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v16 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i16
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i64(ptr addrspace(1) %{{.*}}, i64 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v64 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i64
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster
+llvm.func @st_async_shared_cluster(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i32(ptr addrspace(7) %{{.*}}, i32 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster_types
+llvm.func @st_async_shared_cluster_types(%addr: !llvm.ptr<7>, %v64: i64, %v128: i128, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i64(ptr addrspace(7) %{{.*}}, i64 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v64, mbarrier = %mbar : !llvm.ptr<7>, i64, !llvm.ptr<7>
+  // CHECK: call void @llvm.nvvm.st.async.i128(ptr addrspace(7) %{{.*}}, i128 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v128, mbarrier = %mbar : !llvm.ptr<7>, i128, !llvm.ptr<7>
+  llvm.return
+}
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
new file mode 100644
index 0000000000000..1fe4bbb85235f
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
@@ -0,0 +1,63 @@
+// RUN: mlir-translate -verify-diagnostics -split-input-file -mlir-to-llvmir %s
+
+llvm.func @st_async_global_mbarrier(%addr: !llvm.ptr<1>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{mbarrier is not supported for global address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_i128(%addr: !llvm.ptr<1>, %value: i128) {
+  // expected-error @below {{only 8, 16, 32, and 64 bit values are supported for global address space}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i128
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_no_scope(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{scope must be set for async store to global address space}}
+  nvvm.store.async %addr, %value : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_non_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{mmio is only supported for SYS scope}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{multimem is not supported for mmio}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_i8(%addr: !llvm.ptr<7>, %value: i8, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{only 32, 64, and 128 bit values are supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i8, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_multimem(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{multimem and mmio are not supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {is_multimem = true} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_scope(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{scope is not supported for async store to shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}

>From 6d12a2dd4357a93b9795a29994c0fff65cd74fa3 Mon Sep 17 00:00:00 2001
From: Srinivasa Ravi <srinivasar at nvidia.com>
Date: Tue, 21 Jul 2026 10:15:20 +0000
Subject: [PATCH 2/2] add missing llvm_unreachable

---
 mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index b18f9a96debc9..603d28957176f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -3732,12 +3732,14 @@ AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
     } else if (scope == AsyncStoreScope::GPU) {
       return {llvm::Intrinsic::nvvm_st_async_gpu, {addr, value, isMultimem}};
     }
+    llvm_unreachable("unsupported async store scope for global address space");
   case NVVMMemorySpace::SharedCluster:
     return {llvm::Intrinsic::nvvm_st_async, {addr, value, mbarrier}};
   default:
     llvm_unreachable("unsupported address space");
-    return {llvm::Intrinsic::not_intrinsic, {}};
   }
+
+  return {llvm::Intrinsic::not_intrinsic, {}};
 }
 
 mlir::NVVM::IDArgPair


From llvmlistbot at llvm.org  Tue Jul 21 03:19:09 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 21 Jul 2026 03:19:09 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
Message-ID: <llvm.org/llvm/llvm-project/pull/210932@github.com>


https://github.com/madhur13490 created https://github.com/llvm/llvm-project/pull/210932

This patch implements Phase 2 of the RFC "Enforce Single-Operand Format for All .enable Metadata Nodes". The two-operand boolean form !{!"llvm.loop.vectorize.enable", i1 0/1} is replaced by a single-operand enable/disable pair:

  !{!"llvm.loop.vectorize.enable"}    ; force vectorization
  !{!"llvm.loop.vectorize.disable"}   ; suppress vectorization

The Verifier rejects the two-operand form, AutoUpgrade rewrites old bitcode (including the legacy llvm.vectorizer.enable tag), and the readers and producers across LLVM, Clang, MLIR and Polly are updated.

Please refer to RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571/

>From 2ea017c18a7127eb3177bb0eefbca5e7ee7f2358 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Sun, 12 Jul 2026 23:37:19 -0700
Subject: [PATCH] Enforce single-operand form for llvm.loop.vectorize metadata

This patch implements Phase 2 of the RFC "Enforce Single-Operand Format
for All .enable Metadata Nodes". The two-operand boolean form
!{!"llvm.loop.vectorize.enable", i1 0/1} is replaced by a single-operand
enable/disable pair:

  !{!"llvm.loop.vectorize.enable"}    ; force vectorization
  !{!"llvm.loop.vectorize.disable"}   ; suppress vectorization

The Verifier rejects the two-operand form, AutoUpgrade rewrites old
bitcode (including the legacy llvm.vectorizer.enable tag), and the
readers and producers across LLVM, Clang, MLIR and Polly are updated.

Please refer to RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571/
---
 clang/lib/CodeGen/CGLoopInfo.cpp              |  14 ++--
 .../test/CodeGenCXX/pragma-followup_inner.cpp |   2 +-
 .../test/CodeGenCXX/pragma-followup_outer.cpp |   2 +-
 clang/test/CodeGenCXX/pragma-loop-pr27643.cpp |   2 +-
 .../test/CodeGenCXX/pragma-loop-predicate.cpp |   2 +-
 clang/test/CodeGenCXX/pragma-loop-safety.cpp  |   2 +-
 clang/test/CodeGenCXX/pragma-loop.cpp         |   2 +-
 clang/test/OpenMP/for_simd_codegen.cpp        |   8 +--
 clang/test/OpenMP/for_simd_scan_codegen.cpp   |   2 +-
 clang/test/OpenMP/irbuilder_safelen.cpp       |   2 +-
 .../irbuilder_safelen_order_concurrent.cpp    |   2 +-
 clang/test/OpenMP/irbuilder_simd.cpp          |   2 +-
 clang/test/OpenMP/irbuilder_simd_aligned.cpp  |   2 +-
 clang/test/OpenMP/irbuilder_simdlen.cpp       |   2 +-
 .../test/OpenMP/irbuilder_simdlen_safelen.cpp |   2 +-
 .../OpenMP/master_taskloop_simd_codegen.cpp   |   8 +--
 .../test/OpenMP/parallel_for_simd_codegen.cpp |  10 +--
 .../OpenMP/parallel_for_simd_scan_codegen.cpp |   2 +-
 clang/test/OpenMP/simd_codegen.cpp            |   2 +-
 clang/test/OpenMP/simd_metadata.c             |   2 +-
 clang/test/OpenMP/target_simd_codegen.cpp     |   8 +--
 ...s_distribute_simd_codegen_registration.cpp |   4 +-
 ...ibute_simd_codegen_registration_naming.cpp |   4 +-
 clang/test/OpenMP/taskloop_simd_codegen.cpp   |   2 +-
 ...ute_parallel_for_simd_collapse_codegen.cpp |   4 +-
 .../OpenMP/teams_distribute_simd_codegen.cpp  |   2 +-
 .../test/PCH/delayed-template-with-pragma.cpp |   2 +-
 .../proto-to-llvm/loop_proto_to_llvm.cpp      |  10 +--
 flang/test/Integration/ivdep.f90              |   2 +-
 flang/test/Integration/vector-always.f90      |   4 +-
 llvm/docs/LangRef.md                          |  10 ++-
 llvm/docs/TransformMetadata.rst               |   6 +-
 llvm/include/llvm/IR/AutoUpgrade.h            |   7 +-
 llvm/lib/Bitcode/Reader/MetadataLoader.cpp    |   3 +
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |   6 +-
 llvm/lib/IR/AutoUpgrade.cpp                   |  62 +++++++++++++++++-
 llvm/lib/IR/Verifier.cpp                      |   8 +++
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp |   3 +-
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |   6 +-
 .../Vectorize/LoopVectorizationLegality.cpp   |   8 +++
 llvm/test/Bitcode/upgrade-loop-metadata.ll    |   2 +-
 .../Bitcode/upgrade-loop-vectorize-enable.ll  |  37 +++++++++++
 .../upgrade-loop-vectorize-enable.ll.bc       | Bin 0 -> 2328 bytes
 .../IRCE/add-metadata-pre-post-loops.ll       |   2 +-
 .../test/Transforms/IRCE/correct-loop-info.ll |   2 +-
 .../Transforms/IRCE/loop-guarded-bounds.ll    |   2 +-
 llvm/test/Transforms/IRCE/pre_post_loops.ll   |   2 +-
 .../IRCE/range_intersect_miscompile.ll        |   2 +-
 .../IRCE/unsigned_comparisons_ugt.ll          |   2 +-
 .../IRCE/unsigned_comparisons_ult.ll          |   2 +-
 .../Inline/dilocation-loop-metadata-update.ll |   4 +-
 .../InstCombine/mem-par-metadata-memcpy.ll    |   2 +-
 .../LoopIdiom/AArch64/disable-vectorize.ll    |   4 +-
 .../LoopRotate/vect.omp.persistence.ll        |   4 +-
 .../enable_and_isvectorized.ll                |   2 +-
 .../LoopTransformWarning/optnone.ll           |   2 +-
 .../vectorization-remarks-missed.ll           |   2 +-
 .../vectorizer-loop-kind-unroll-warning.ll    |   2 +-
 .../LoopUnroll/unroll-pragmas-disabled.ll     |   4 +-
 .../vectorizer-loop-kind-remarks.ll           |   2 +-
 .../LoopVectorize/2016-07-27-loop-vec.ll      |   2 +-
 .../LoopVectorize/AArch64/call-costs.ll       |   2 +-
 .../AArch64/conditional-branches-cost.ll      |   2 +-
 .../AArch64/conditional-scalar-assignment.ll  |   2 +-
 .../AArch64/interleave-with-gaps.ll           |   2 +-
 .../LoopVectorize/AArch64/masked-op-cost.ll   |   2 +-
 .../LoopVectorize/AArch64/masked_ldst_sme.ll  |   2 +-
 .../LoopVectorize/AArch64/optsize_minsize.ll  |   2 +-
 .../AArch64/outer_loop_prefer_scalable.ll     |   2 +-
 ...outer_loop_test1_no_explicit_vect_width.ll |   2 +-
 .../partial-reduce-dot-product-epilogue.ll    |   2 +-
 .../partial-reduce-dot-product-mixed.ll       |   2 +-
 .../partial-reduce-dot-product-neon.ll        |   2 +-
 .../AArch64/partial-reduce-dot-product.ll     |   2 +-
 .../AArch64/partial-reduce-sub.ll             |   2 +-
 .../AArch64/pr60831-sve-inv-store-crash.ll    |   2 +-
 .../LoopVectorize/AArch64/predicated-costs.ll |   2 +-
 .../AArch64/scalable-predicate-instruction.ll |   2 +-
 .../AArch64/scalable-reductions-tf.ll         |   2 +-
 .../AArch64/scalable-reductions.ll            |   2 +-
 .../LoopVectorize/AArch64/strict-fadd.ll      |   2 +-
 .../LoopVectorize/AArch64/sve-basic-vec.ll    |   2 +-
 .../AArch64/sve-cond-inv-loads.ll             |   2 +-
 .../AArch64/sve-extract-last-veclane.ll       |   2 +-
 .../AArch64/sve-gather-scatter-cost.ll        |   2 +-
 .../AArch64/sve-gather-scatter.ll             |   2 +-
 .../LoopVectorize/AArch64/sve-inductions.ll   |   2 +-
 .../LoopVectorize/AArch64/sve-inv-store.ll    |   2 +-
 .../AArch64/sve-large-strides.ll              |   2 +-
 .../AArch64/sve-masked-loadstore.ll           |   2 +-
 .../AArch64/sve-tail-folding-option.ll        |   2 +-
 .../LoopVectorize/AArch64/sve-type-conv.ll    |   2 +-
 .../AArch64/sve-vector-reverse-mask4.ll       |   2 +-
 .../AArch64/sve-vector-reverse.ll             |   2 +-
 .../AArch64/sve-widen-extractvalue.ll         |   2 +-
 .../LoopVectorize/AArch64/sve-widen-phi.ll    |   2 +-
 .../AArch64/unsafe-vf-hint-remark.ll          |   2 +-
 .../AArch64/vector-reverse-mask4.ll           |   2 +-
 .../LoopVectorize/AArch64/vector-reverse.ll   |   2 +-
 .../vplan-native-outer-loop-wide-type.ll      |   2 +-
 .../widen-gep-all-indices-invariant.ll        |   2 +-
 .../LoopVectorize/ARM/optsize_minsize.ll      |   2 +-
 .../ARM/prefer-tail-loop-folding.ll           |   2 +-
 .../ARM/tail-folding-loop-hint.ll             |   2 +-
 .../ARM/tail-folding-prefer-flag.ll           |   2 +-
 .../RISCV/dbg-tail-folding-by-evl.ll          |   2 +-
 .../LoopVectorize/RISCV/force-vect-msg.ll     |   2 +-
 .../RISCV/riscv-vector-reverse.ll             |   2 +-
 .../RISCV/tail-folding-cond-reduction.ll      |   2 +-
 .../tail-folding-fixed-order-recurrence.ll    |   2 +-
 .../RISCV/tail-folding-inloop-reduction.ll    |   2 +-
 .../RISCV/tail-folding-interleave.ll          |   2 +-
 .../RISCV/tail-folding-intermediate-store.ll  |   2 +-
 .../RISCV/tail-folding-ordered-reduction.ll   |   2 +-
 .../RISCV/tail-folding-reduction.ll           |   2 +-
 .../RISCV/tail-folding-uniform-store.ll       |   2 +-
 .../LoopVectorize/RISCV/uniform-load-store.ll |   2 +-
 .../LoopVectorize/Sparc/no-vectorize.ll       |   6 +-
 .../Transforms/LoopVectorize/VE/disable_lv.ll |   2 +-
 ...an-vp-intrinsics-fixed-order-recurrence.ll |   2 +-
 .../RISCV/vplan-vp-intrinsics-reduction.ll    |   2 +-
 .../LoopVectorize/VPlan/constant-fold.ll      |   2 +-
 .../VPlan/vplan-printing-outer-loop.ll        |   2 +-
 .../VPlan/vplan-stress-test-no-explict-vf.ll  |   2 +-
 .../LoopVectorize/X86/amdlibm-calls-finite.ll |   6 +-
 .../X86/conditional-scalar-assignment.ll      |   2 +-
 .../LoopVectorize/X86/cost-model.ll           |   2 +-
 .../LoopVectorize/X86/induction-costs.ll      |   2 +-
 .../LoopVectorize/X86/iv-live-outs.ll         |   2 +-
 .../X86/libm-vector-calls-VF2-VF8.ll          |  34 +++++-----
 .../X86/libm-vector-calls-finite.ll           |  12 ++--
 .../LoopVectorize/X86/libm-vector-calls.ll    |  26 ++++----
 .../Transforms/LoopVectorize/X86/no_fpmath.ll |   2 +-
 .../X86/no_fpmath_with_hotness.ll             |   2 +-
 .../X86/outer-loop-non-power-of-2-type.ll     |   2 +-
 ...outer_loop_test1_no_explicit_vect_width.ll |   2 +-
 .../LoopVectorize/X86/redundant-vf2-cost.ll   |   2 +-
 ...licating-load-store-costs-max-bandwidth.ll |   2 +-
 .../X86/replicating-load-store-costs.ll       |   2 +-
 .../LoopVectorize/X86/svml-calls-finite.ll    |  28 ++++----
 .../X86/tail_folding_and_assume_safety.ll     |   2 +-
 .../LoopVectorize/X86/tail_loop_folding.ll    |   2 +-
 .../LoopVectorize/X86/vect.omp.force.ll       |   2 +-
 .../X86/vect.omp.force.small-tc.ll            |   2 +-
 .../X86/vectorization-remarks-missed.ll       |   2 +-
 .../Transforms/LoopVectorize/alias-mask.ll    |   2 +-
 .../LoopVectorize/dbg-outer-loop-vect.ll      |   2 +-
 .../diag-disabled-vectorization-msgs.ll       |   2 +-
 .../LoopVectorize/disable_nonforced_enable.ll |   2 +-
 .../LoopVectorize/explicit_outer_detection.ll |   2 +-
 .../explicit_outer_nonuniform_inner.ll        |   2 +-
 .../explicit_outer_uniform_diverg_branch.ll   |   2 +-
 .../LoopVectorize/make-followup-loop-id.ll    |   2 +-
 .../LoopVectorize/memdep-fold-tail.ll         |   2 +-
 .../LoopVectorize/nested-outer-loop-vect.ll   |   2 +-
 ...o-fold-tail-by-masking-iv-external-uses.ll |   2 +-
 .../LoopVectorize/no_array_bounds.ll          |   2 +-
 .../LoopVectorize/no_array_bounds_scalable.ll |   2 +-
 .../Transforms/LoopVectorize/no_switch.ll     |   2 +-
 .../no_switch_disable_vectorization.ll        |   2 +-
 .../noalias-scope-decl-outer-loop.ll          |   2 +-
 .../non-widenable-intrinsics-outer-loop.ll    |   2 +-
 llvm/test/Transforms/LoopVectorize/optsize.ll |   2 +-
 .../LoopVectorize/outer-loop-atomic.ll        |   2 +-
 .../outer-loop-inner-latch-successors.ll      |   2 +-
 .../outer-loop-vec-phi-predecessor-order.ll   |   2 +-
 .../LoopVectorize/outer-loop-wide-phis.ll     |   2 +-
 .../LoopVectorize/outer_loop_contiguous.ll    |   2 +-
 .../LoopVectorize/outer_loop_early_exit.ll    |   2 +-
 .../outer_loop_hcfg_construction.ll           |   2 +-
 .../LoopVectorize/outer_loop_scalable.ll      |   2 +-
 .../LoopVectorize/outer_loop_test1.ll         |   2 +-
 .../LoopVectorize/outer_loop_test2.ll         |   2 +-
 .../LoopVectorize/reduction-inloop-uf4.ll     |   2 +-
 .../LoopVectorize/reduction-inloop.ll         |   2 +-
 .../LoopVectorize/remove_metadata.ll          |   2 +-
 ...ime-check-threshold-with-force-metadata.ll |   4 +-
 .../Transforms/LoopVectorize/runtime-check.ll |   2 +-
 .../LoopVectorize/scalable-inductions.ll      |   2 +-
 .../LoopVectorize/unsafe-vf-hint-remark.ll    |   2 +-
 .../use-scalar-epilogue-if-tp-fails.ll        |   2 +-
 .../LoopVectorize/vect.omp.persistence.ll     |   4 +-
 ...kedge-elimination-predicated-early-exit.ll |   2 +-
 ...plan-outer-loop-uncomputable-trip-count.ll |   2 +-
 .../vplan-vectorize-inner-loop-reduction.ll   |   2 +-
 .../vplan-widen-call-instruction.ll           |   4 +-
 .../vplan-widen-select-instruction.ll         |   2 +-
 .../AArch64/Oz-and-forced-vectorize.ll        |   2 +-
 .../X86/loop-vectorize-metadata.ll            |   4 +-
 .../X86/preserve-access-group.ll              |   2 +-
 .../Transforms/SROA/mem-par-metadata-sroa.ll  |   2 +-
 .../X86/combine-parallel-mem-md.ll            |   2 +-
 .../Verifier/llvm.loop-cu-strip-followup.ll   |   4 +-
 llvm/test/Verifier/llvm.loop.vectorize.ll     |  41 ++++++++++++
 .../Target/LLVMIR/LoopAnnotationImporter.cpp  |   5 +-
 .../LLVMIR/LoopAnnotationTranslation.cpp      |   7 +-
 .../Target/LLVMIR/Import/import-failure.ll    |  16 -----
 .../Target/LLVMIR/Import/metadata-loop.ll     |   2 +-
 mlir/test/Target/LLVMIR/loop-metadata.mlir    |   2 +-
 .../LLVMIR/openmp-parallel-do-simd.mlir       |   2 +-
 .../Target/LLVMIR/openmp-simd-ordered.mlir    |   2 +-
 .../LLVMIR/openmp-simd-reduction-byref.mlir   |   2 +-
 .../LLVMIR/openmp-simd-reduction-simple.mlir  |   2 +-
 ...nmp-teams-distribute-parallel-do-simd.mlir |   2 +-
 .../LLVMIR/openmp-wsloop-simd-ordered.mlir    |   2 +-
 polly/lib/CodeGen/CodeGeneration.cpp          |  18 ++++-
 polly/lib/CodeGen/IRBuilder.cpp               |   9 ++-
 polly/test/CodeGen/MemAccess/create_arrays.ll |   2 +-
 .../CodeGen/Metadata/basic_vec_annotate.ll    |   4 +-
 .../CodeGen/Metadata/fallback_vec_annotate.ll |   2 +-
 polly/test/CodeGen/annotated_alias_scopes.ll  |   2 +-
 polly/test/CodeGen/scev-backedgetaken.ll      |   2 +-
 .../pattern-matching-based-opts_14.ll         |   2 +-
 213 files changed, 486 insertions(+), 330 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
 create mode 100644 llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc
 create mode 100644 llvm/test/Verifier/llvm.loop.vectorize.ll

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 6b0b1b55213e1..b46ae34e58a44 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -209,10 +209,8 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createUnrollAndJamMetadata(Attrs, LoopProperties, HasUserTransforms);
@@ -279,10 +277,10 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
       (Attrs.VectorizeScalable == LoopAttributes::Disable &&
        Attrs.VectorizeWidth != 1)) {
     VectorizeEnabled = Attrs.VectorizeEnable != LoopAttributes::Disable;
-    Args.push_back(
-        MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                          ConstantAsMetadata::get(ConstantInt::get(
-                              llvm::Type::getInt1Ty(Ctx), VectorizeEnabled))}));
+    Args.push_back(MDNode::get(
+        Ctx, {MDString::get(Ctx, VectorizeEnabled
+                                     ? "llvm.loop.vectorize.enable"
+                                     : "llvm.loop.vectorize.disable")}));
   }
 
   // Apply all loop properties to the vectorized loop.
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..20615d9744c6e 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -25,7 +25,7 @@ extern "C" void followup_inner(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[OUTERLOOP_9]] = distinct !{![[OUTERLOOP_9]], [[MP:![0-9]+]], ![[UNROLLANDJAM_COUNT_10:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPINNER_11:[0-9]+]]}
 // CHECK-DAG: ![[UNROLLANDJAM_COUNT_10]] = !{!"llvm.loop.unroll_and_jam.count", i32 4}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..70ce57c6fb6a7 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_outer(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[VECTORIZE_FOLLOWUP_9:[0-9]+]] = !{!"llvm.loop.vectorize.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[ISVECTORIZED_11:[0-9]+]], ![[UNROLLANDJAM_12:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPOUTER_13:[0-9]+]]}
 // CHECK-DAG: ![[ISVECTORIZED_11:[0-9]+]] = !{!"llvm.loop.isvectorized"}
diff --git a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
index 37e29776264b6..afab64f4c6867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
@@ -43,7 +43,7 @@ void loop4(int *List, int Length) {
 // CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE:.*]]}
 // CHECK: ![[VEC_WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
 // CHECK: ![[FIXED_WIDTH]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE]]}
 // CHECK: ![[VEC_WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
index 8a25ed8de6239..421110fd9f867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
@@ -103,7 +103,7 @@ void test9(int *List, int Length) {
 
 // CHECK:      ![[LOOP0]] = distinct !{![[LOOP0]], [[MP:![0-9]+]], [[GEN3:![0-9]+]]}
 // CHECK:      [[MP]] = !{!"llvm.loop.mustprogress"}
-// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-NEXT: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP]], [[GEN6:![0-9]+]], [[GEN3]]}
 // CHECK-NEXT: [[GEN6]] = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-safety.cpp b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
index db1b6769e0514..e2a7a124890a6 100644
--- a/clang/test/CodeGenCXX/pragma-loop-safety.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
@@ -51,7 +51,7 @@ void interleave_test(int *List, int Length) {
 // CHECK: ![[PARALLEL_ACCESSES_7]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_2]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[INTERLEAVE_1]] = !{!"llvm.loop.interleave.count", i32 1}
-// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: ![[ACCESS_GROUP_8]] = distinct !{}
 // CHECK: ![[LOOP2_HINTS]] = distinct !{![[LOOP2_HINTS]], [[MP]], ![[PARALLEL_ACCESSES_11:[0-9]+]], ![[UNROLL_DISABLE]], ![[WIDTH_1:[0-9]+]], ![[INTENABLE_1]]}
 // CHECK: ![[PARALLEL_ACCESSES_11]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_8]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..d4a9faaf2be6b 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -256,7 +256,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[INTERLEAVE_10:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 10}
 // CHECK-DAG: ![[INTERLEAVE_16:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 16}
 
-// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: ![[FIXED_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: ![[SCALABLE_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 // CHECK-DAG: ![[WIDTH_1:[0-9]+]] = !{!"llvm.loop.vectorize.width", i32 1}
diff --git a/clang/test/OpenMP/for_simd_codegen.cpp b/clang/test/OpenMP/for_simd_codegen.cpp
index 1ac95e8943640..ae6161d03978d 100644
--- a/clang/test/OpenMP/for_simd_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_codegen.cpp
@@ -802,10 +802,10 @@ void parallel_simd(float *a) {
 }
 // TERM_DEBUG: !{{[0-9]+}} = !DILocation(line: [[@LINE-11]],
 // TERM_DEBUG-NOT: line: 0,
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // CHECK-DAG: ![[SIMD_LOOP]] = distinct !{![[SIMD_LOOP]], {{.*}}![[VECT_LOOP:[^,]+]]}
-// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // OMP50-DAG: ![[NOSIMD_LOOP]] = distinct !{![[NOSIMD_LOOP]], {{.*}}![[NOVECT_LOOP:[^,]+]]}
-// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.disable"}
 #endif // HEADER
diff --git a/clang/test/OpenMP/for_simd_scan_codegen.cpp b/clang/test/OpenMP/for_simd_scan_codegen.cpp
index 829f2656042fb..9ab62f8a682b2 100644
--- a/clang/test/OpenMP/for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_scan_codegen.cpp
@@ -300,5 +300,5 @@ void baz(int n) {
 }
 
 #endif
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
diff --git a/clang/test/OpenMP/irbuilder_safelen.cpp b/clang/test/OpenMP/irbuilder_safelen.cpp
index 6a4266f9646a7..3090eafa6a508 100644
--- a/clang/test/OpenMP/irbuilder_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
index 210978fc6069c..215b37e4bbb5f 100644
--- a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simd.cpp b/clang/test/OpenMP/irbuilder_simd.cpp
index 9c36e4ab97182..03f92d8117d45 100644
--- a/clang/test/OpenMP/irbuilder_simd.cpp
+++ b/clang/test/OpenMP/irbuilder_simd.cpp
@@ -65,7 +65,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: ![[META3:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META4]]  = distinct !{![[META4]], ![[META5:[0-9]+]], ![[META6:[0-9]+]]}
 // CHECK-NEXT: ![[META5]]  = !{!"llvm.loop.parallel_accesses", ![[META3]]}
-// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable"}
 // CHECK-NEXT: ![[META7:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META8]]  = distinct !{![[META8]], ![[META9:[0-9]+]], ![[META6]]}
 // CHECK-NEXT: ![[META9]]  = !{!"llvm.loop.parallel_accesses", ![[META7]]}
diff --git a/clang/test/OpenMP/irbuilder_simd_aligned.cpp b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
index b5ce43c8ea6a9..bb121036b4f0c 100644
--- a/clang/test/OpenMP/irbuilder_simd_aligned.cpp
+++ b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
@@ -293,7 +293,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP4]] = distinct !{}
 // CHECK: [[LOOP5]] = distinct !{[[LOOP5]], [[META6:![0-9]+]], [[META7:![0-9]+]]}
 // CHECK: [[META6]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP4]]}
-// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[ACC_GRP8]] = distinct !{}
 // CHECK: [[LOOP9]] = distinct !{[[LOOP9]], [[META10:![0-9]+]], [[META7]]}
 // CHECK: [[META10]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP8]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen.cpp b/clang/test/OpenMP/irbuilder_simdlen.cpp
index f6c26923851d0..9a7e8d5332ae0 100644
--- a/clang/test/OpenMP/irbuilder_simdlen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
index e46d5dde841b3..bb35f9341ded4 100644
--- a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 2}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
index d78d193e2edd1..f1eae642b613c 100644
--- a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
@@ -238,10 +238,10 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// OMP50-DAG: !{!"llvm.loop.vectorize.disable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/parallel_for_simd_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
index 40af7069bbf7d..cc8cab706975a 100644
--- a/clang/test/OpenMP/parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
@@ -798,16 +798,16 @@ for (int i = 0; i < 10; ++i);
 // OMP50: omp.inner.for.body:
 // OMP50: %{{[0-9]+}} = atomicrmw add ptr %{{[0-9]+}}, i32 1 monotonic, align 4, !llvm.access.group !{{[0-9]+}}
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP45-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP45-DAG: ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP50-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP50-DAG  ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
 // OMP50-DAG: ![[NOVECT]] = distinct !{![[NOVECT]], ![[NOVM:.+]]}
-// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.disable"}
 
 // TERM_DEBUG-LABEL: bar
 int bar() { extern void mayThrow(); mayThrow(); return 0; };
diff --git a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
index cac997753d480..e3810c27163a4 100644
--- a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
@@ -299,7 +299,7 @@ void baz(int n) {
   }
 }
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
 #endif
 
diff --git a/clang/test/OpenMP/simd_codegen.cpp b/clang/test/OpenMP/simd_codegen.cpp
index b96e4213e8e0e..ee8d709ba47fe 100644
--- a/clang/test/OpenMP/simd_codegen.cpp
+++ b/clang/test/OpenMP/simd_codegen.cpp
@@ -861,7 +861,7 @@ S8 s8(0);
 
 // TERM_DEBUG-NOT: line: 0,
 // TERM_DEBUG: distinct !DISubprogram(linkageName: "_GLOBAL__sub_I_simd_codegen.cpp",
-// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.disable"}
 // OMP5-DAG: ![[DISABLE_VECT]] = distinct !{{.*}}![[NOVECT]]{{[,}]}}
 #endif // HEADER
 
diff --git a/clang/test/OpenMP/simd_metadata.c b/clang/test/OpenMP/simd_metadata.c
index 83bcbf6deefe2..6fc4c401135f2 100644
--- a/clang/test/OpenMP/simd_metadata.c
+++ b/clang/test/OpenMP/simd_metadata.c
@@ -109,7 +109,7 @@ void h3(float *c, float *a, float *b, int size)
 // Metadata for h1:
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_16:![0-9]+]], [[LOOP_VEC_ENABLE:![0-9]+]]}
 // CHECK: [[LOOP_WIDTH_16]] = !{!"llvm.loop.vectorize.width", i32 16}
-// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_8:![0-9]+]], [[LOOP_VEC_ENABLE]]}
 // CHECK: [[LOOP_WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 // CHECK: ![[ACCESS_GROUP_7]] = distinct !{}
diff --git a/clang/test/OpenMP/target_simd_codegen.cpp b/clang/test/OpenMP/target_simd_codegen.cpp
index 0c2dde23f6c46..59284c201bb30 100644
--- a/clang/test/OpenMP/target_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_simd_codegen.cpp
@@ -702,9 +702,9 @@ int bar(int n){
 // Store captures in the context.
 // CHECK-DAG:   [[REF_B:%.+]] = load ptr, ptr [[LOCAL_B]],
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP51: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP51: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// TOMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// OMP51: !{!"llvm.loop.vectorize.disable"}
+// TOMP51: !{!"llvm.loop.vectorize.disable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
index 42e8377bd22dd..fd2d0d0c18737 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
@@ -455,7 +455,7 @@ int bar(int a){
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2STILi100EE3fooEv", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2SCC1Ev", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
index b63f0b6caeee2..c169d4a556bf0 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
@@ -82,6 +82,6 @@ int nested(int a){
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T1L]], i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T2L]], i32 0, i32 {{[0-9]+}}}
 
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 #endif
diff --git a/clang/test/OpenMP/taskloop_simd_codegen.cpp b/clang/test/OpenMP/taskloop_simd_codegen.cpp
index ab34ff66e5c7a..df2df9f609a57 100644
--- a/clang/test/OpenMP/taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/taskloop_simd_codegen.cpp
@@ -208,7 +208,7 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
index 365f11cfac5c6..688849851aa4d 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
@@ -46,7 +46,7 @@ int teams_template_struct(void) {
 
 }
 
-// CK4: !{!"llvm.loop.vectorize.enable", i1 true}
+// CK4: !{!"llvm.loop.vectorize.enable"}
 
 #endif // CK1
 
@@ -103,7 +103,7 @@ int main (int argc, char **argv) {
 // discard loop variables not needed here
 
 
-// CK4: !{!"llvm.loop.vectorize.enable", i1 true}
+// CK4: !{!"llvm.loop.vectorize.enable"}
 
 #endif // CK2
 #endif // #ifndef HEADER
diff --git a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
index d5000a5b18a53..c2234296a059f 100644
--- a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
@@ -207,7 +207,7 @@ int main (int argc, char **argv) {
 
 
 
-// OMP5_50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5_50-DAG: !{!"llvm.loop.vectorize.disable"}
 #endif // CK4
 #endif
 // CHECK1-LABEL: define {{[^@]+}}@_Z21teams_argument_globali
diff --git a/clang/test/PCH/delayed-template-with-pragma.cpp b/clang/test/PCH/delayed-template-with-pragma.cpp
index e54388456245a..124d0e5db99e8 100644
--- a/clang/test/PCH/delayed-template-with-pragma.cpp
+++ b/clang/test/PCH/delayed-template-with-pragma.cpp
@@ -27,7 +27,7 @@ void a(T t) {
 // CHECK: [[LOOP1B]] = !{!"llvm.loop.unroll.count", i32 4}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[LOOP2A:!.*]], [[LOOP2B:!.*]]}
 // CHECK: [[LOOP2A]] = !{!"llvm.loop.parallel_accesses", [[LOOP2C:!.*]]}
-// CHECK: [[LOOP2B]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP2B]] = !{!"llvm.loop.vectorize.enable"}
 // expected-warning at 17 {{unused variable 'zz'}}
 void foo()
 {
diff --git a/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp b/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
index 409570c3f0fa0..d37726b74b48e 100644
--- a/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
+++ b/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
@@ -157,12 +157,13 @@ void NestedLoopToString(std::ostream &os, const LoopFunction &x) {
   }
   os << "%i_ct_new = add i64 %inner_ct, 1\n"
      << "%jmp_inner = icmp eq i64 %i_ct_new, %s\n"
-     << "br i1 %jmp_inner, label %outer_loop, label %inner_loop, !llvm.loop !0\n"
+     << "br i1 %jmp_inner, label %outer_loop, label %inner_loop, !llvm.loop "
+        "!0\n"
      << "end:\n"
      << "ret void\n"
      << "}\n"
      << "!0 = distinct !{!0, !1, !2}\n"
-     << "!1 = !{!\"llvm.loop.vectorize.enable\", i1 true}\n"
+     << "!1 = !{!\"llvm.loop.vectorize.enable\"}\n"
      << "!2 = !{!\"llvm.loop.vectorize.width\", i32 " << kArraySize << "}\n";
 }
 void SingleLoopToString(std::ostream &os, const LoopFunction &x) {
@@ -176,12 +177,11 @@ void SingleLoopToString(std::ostream &os, const LoopFunction &x) {
      << "ret void\n"
      << "loop:\n"
      << "%outer_ct = phi i64 [ %ctnew, %loop ], [ 0, %start ]\n"
-     << x.outer_statements()
-     << "%ctnew = add i64 %outer_ct, 1\n"
+     << x.outer_statements() << "%ctnew = add i64 %outer_ct, 1\n"
      << "%j = icmp eq i64 %ctnew, %s\n"
      << "br i1 %j, label %end, label %loop, !llvm.loop !0\n}\n"
      << "!0 = distinct !{!0, !1, !2}\n"
-     << "!1 = !{!\"llvm.loop.vectorize.enable\", i1 true}\n"
+     << "!1 = !{!\"llvm.loop.vectorize.enable\"}\n"
      << "!2 = !{!\"llvm.loop.vectorize.width\", i32 " << kArraySize << "}\n";
 }
 std::ostream &operator<<(std::ostream &os, const LoopFunction &x) {
diff --git a/flang/test/Integration/ivdep.f90 b/flang/test/Integration/ivdep.f90
index 9882fe148262f..a53d46ed446de 100644
--- a/flang/test/Integration/ivdep.f90
+++ b/flang/test/Integration/ivdep.f90
@@ -120,5 +120,5 @@ end subroutine ivdep_test4
 ! CHECK: ![[PARALLEL_ACCESSES2]] = !{!"llvm.loop.parallel_accesses", [[DISTRINCT2]]}
 ! CHECK: [[DISTRINCT3:.*]] = distinct !{}
 ! CHECK: ![[ANNOTATION3]] = distinct !{![[ANNOTATION3]], ![[VECTORIZE:.*]], ![[PARALLEL_ACCESSES3:.*]]}
-! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
 ! CHECK: ![[PARALLEL_ACCESSES3]] = !{!"llvm.loop.parallel_accesses", [[DISTRINCT3]]}
diff --git a/flang/test/Integration/vector-always.f90 b/flang/test/Integration/vector-always.f90
index 1d8aad97bde70..5634a6a13b63f 100644
--- a/flang/test/Integration/vector-always.f90
+++ b/flang/test/Integration/vector-always.f90
@@ -25,6 +25,6 @@ subroutine no_vector
 end subroutine no_vector
 
 ! CHECK: ![[ANNOTATION]] = distinct !{![[ANNOTATION]], ![[VECTORIZE:.*]]}
-! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
 ! CHECK: ![[ANNOTATION2]] = distinct !{![[ANNOTATION2]], ![[VECTORIZE2:.*]]}
-! CHECK: ![[VECTORIZE2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+! CHECK: ![[VECTORIZE2]] = !{!"llvm.loop.vectorize.disable"}
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 192597944101b..538dfab6c098c 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7993,14 +7993,12 @@ then the interleave count will be determined automatically.
 
 #### '`llvm.loop.vectorize.enable`' Metadata
 
-This metadata selectively enables or disables vectorization for the loop. The
-first operand is the string `llvm.loop.vectorize.enable` and the second operand
-is a bit. If the bit operand value is 1 vectorization is enabled. A value of
-0 disables vectorization:
+This metadata selectively enables or disables vectorization for the loop. Each
+node has a single operand containing the name string:
 
 ```llvm
-!0 = !{!"llvm.loop.vectorize.enable", i1 0}
-!1 = !{!"llvm.loop.vectorize.enable", i1 1}
+!0 = !{!"llvm.loop.vectorize.enable"}
+!1 = !{!"llvm.loop.vectorize.disable"}
 ```
 
 #### '`llvm.loop.vectorize.predicate.enable`' Metadata
diff --git a/llvm/docs/TransformMetadata.rst b/llvm/docs/TransformMetadata.rst
index 700761a383e9a..d92f25641f149 100644
--- a/llvm/docs/TransformMetadata.rst
+++ b/llvm/docs/TransformMetadata.rst
@@ -79,7 +79,7 @@ vectorized, for instance being unrolled.
       br i1 %exitcond, label %for.exit, label %for.header, !llvm.loop !0
     ...
     !0 = distinct !{!0, !1, !2}
-    !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+    !1 = !{!"llvm.loop.vectorize.enable"}
     !2 = !{!"llvm.loop.disable_nonforced"}
 
 After a transformation is applied, follow-up attributes are set on the
@@ -97,7 +97,7 @@ then unrolled.
 .. code-block:: llvm
 
     !0 = distinct !{!0, !1, !2, !3}
-    !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+    !1 = !{!"llvm.loop.vectorize.enable"}
     !2 = !{!"llvm.loop.disable_nonforced"}
     !3 = !{!"llvm.loop.vectorize.followup_vectorized", !{"llvm.loop.unroll.enable"}}
 
@@ -130,7 +130,7 @@ Loop Vectorization and Interleaving
 
 Loop vectorization and interleaving is interpreted as a single
 transformation. It is interpreted as forced if
-``!{"llvm.loop.vectorize.enable", i1 true}`` is set.
+``!{"llvm.loop.vectorize.enable"}`` is set.
 
 Assuming the pre-vectorization loop is
 
diff --git a/llvm/include/llvm/IR/AutoUpgrade.h b/llvm/include/llvm/IR/AutoUpgrade.h
index feb19e69389a9..c9a882a898553 100644
--- a/llvm/include/llvm/IR/AutoUpgrade.h
+++ b/llvm/include/llvm/IR/AutoUpgrade.h
@@ -112,7 +112,12 @@ namespace llvm {
 
   /// Check whether a string looks like an old loop attachment tag.
   inline bool mayBeOldLoopAttachmentTag(StringRef Name) {
-    return Name.starts_with("llvm.vectorizer.");
+    // "llvm.loop.vectorize.enable" is intentionally included: the current
+    // single-operand form shares the tag with the removed two-operand form
+    // (!{!"llvm.loop.vectorize.enable", i1 X}), so we can only decide by
+    // inspecting the operands, which happens in upgradeLoopArgument().
+    return Name.starts_with("llvm.vectorizer.") ||
+           Name == "llvm.loop.vectorize.enable";
   }
 
   /// Upgrade the loop attachment metadata node.
diff --git a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
index 40bc930d124f0..270ed23c5e6e2 100644
--- a/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
+++ b/llvm/lib/Bitcode/Reader/MetadataLoader.cpp
@@ -2434,6 +2434,9 @@ Error MetadataLoader::MetadataLoaderImpl::parseOneMetadata(
   }
   case bitc::METADATA_STRINGS: {
     auto CreateNextMDString = [&](StringRef Str) {
+      // Modern bitcode encodes MDStrings via this bulk record, so mirror the
+      // METADATA_STRING check above to arm the loop-attachment upgrader.
+      HasSeenOldLoopTags |= mayBeOldLoopAttachmentTag(Str);
       ++NumMDStringLoaded;
       MetadataList.assignValue(MDString::get(Context, Str), NextMetadataNo);
       NextMetadataNo++;
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index e19ed110adf1e..aa7af1318016c 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -7462,10 +7462,8 @@ void OpenMPIRBuilder::applySimd(CanonicalLoopInfo *CanonicalLoop,
 
   // Use the above access group metadata to create loop level
   // metadata, which should be distinct for each loop.
-  ConstantAsMetadata *BoolConst =
-      ConstantAsMetadata::get(ConstantInt::getTrue(Type::getInt1Ty(Ctx)));
-  LoopMDList.push_back(MDNode::get(
-      Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"), BoolConst}));
+  LoopMDList.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable")}));
 
   if (Simdlen || Safelen) {
     // If both simdlen and safelen clauses are specified, the value of the
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 880350a1b159b..0f46a299ba00d 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6964,6 +6964,27 @@ void llvm::copyModuleAttrToFunctions(Module &M) {
   }
 }
 
+// Old two-operand form: !{!"llvm.loop.vectorize.enable", i1 X}. The new
+// single-operand form uses "llvm.loop.vectorize.enable" for X = true and
+// "llvm.loop.vectorize.disable" for X = false.
+static bool isOldVectorizeEnable(const MDTuple *T) {
+  if (T->getNumOperands() != 2)
+    return false;
+  auto *Tag = dyn_cast_or_null<MDString>(T->getOperand(0));
+  if (!Tag || Tag->getString() != "llvm.loop.vectorize.enable")
+    return false;
+  return mdconst::hasa<ConstantInt>(T->getOperand(1));
+}
+
+// Build the single-operand vectorize enable/disable node that replaces a
+// boolean operand: nonzero -> enable, zero -> disable.
+static Metadata *makeVectorizeEnableNode(LLVMContext &C, const MDOperand &Op) {
+  bool Enable = !mdconst::extract<ConstantInt>(Op)->isZero();
+  return MDTuple::get(
+      C, {MDString::get(C, Enable ? "llvm.loop.vectorize.enable"
+                                  : "llvm.loop.vectorize.disable")});
+}
+
 static bool isOldLoopArgument(Metadata *MD) {
   auto *T = dyn_cast_or_null<MDTuple>(MD);
   if (!T)
@@ -6973,7 +6994,9 @@ static bool isOldLoopArgument(Metadata *MD) {
   auto *S = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!S)
     return false;
-  return S->getString().starts_with("llvm.vectorizer.");
+  if (S->getString().starts_with("llvm.vectorizer."))
+    return true;
+  return isOldVectorizeEnable(T);
 }
 
 static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
@@ -6997,17 +7020,33 @@ static Metadata *upgradeLoopArgument(Metadata *MD) {
   auto *OldTag = dyn_cast_or_null<MDString>(T->getOperand(0));
   if (!OldTag)
     return MD;
+
+  LLVMContext &C = T->getContext();
+
+  // Rewrite the modern two-operand vectorize.enable form to the single-operand
+  // enable/disable pair.
+  if (isOldVectorizeEnable(T))
+    return makeVectorizeEnableNode(C, T->getOperand(1));
+
   if (!OldTag->getString().starts_with("llvm.vectorizer."))
     return MD;
 
   // This has an old tag.  Upgrade it.
+  MDString *NewTag = upgradeLoopTag(C, OldTag->getString());
+
+  // The legacy !{!"llvm.vectorizer.enable", i1 X} maps onto the single-operand
+  // vectorize.enable/disable pair, not a two-operand enable node.
+  if (NewTag->getString() == "llvm.loop.vectorize.enable" &&
+      T->getNumOperands() == 2 && mdconst::hasa<ConstantInt>(T->getOperand(1)))
+    return makeVectorizeEnableNode(C, T->getOperand(1));
+
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
-  Ops.push_back(upgradeLoopTag(T->getContext(), OldTag->getString()));
+  Ops.push_back(NewTag);
   for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
     Ops.push_back(T->getOperand(I));
 
-  return MDTuple::get(T->getContext(), Ops);
+  return MDTuple::get(C, Ops);
 }
 
 MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
@@ -7018,6 +7057,23 @@ MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
   if (none_of(T->operands(), isOldLoopArgument))
     return &N;
 
+  // Fix the old two-operand llvm.loop.vectorize.enable nodes in place: the
+  // Verifier rejects any MDNode carrying the vectorize.enable tag with more
+  // than one operand, so a leftover reference (from the distinct loop-ID)
+  // would still trigger a diagnostic. In-place mutation is safe on distinct
+  // MDNodes.
+  if (T->isDistinct()) {
+    for (unsigned I = 0, E = T->getNumOperands(); I < E; ++I) {
+      auto *OpT = dyn_cast_or_null<MDTuple>(T->getOperand(I));
+      if (OpT && isOldVectorizeEnable(OpT))
+        T->replaceOperandWith(I, upgradeLoopArgument(OpT));
+    }
+    if (none_of(T->operands(), isOldLoopArgument))
+      return &N;
+  }
+
+  // Remaining old arguments (e.g. llvm.vectorizer.*) are handled via a wrapper
+  // attachment; the original distinct loop-ID is kept as the first operand.
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
   for (Metadata *MD : T->operands())
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 28459a39fedae..7ea4d5d87412e 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1008,6 +1008,14 @@ void Verifier::visitMDNode(const MDNode &BaseMD,
             CurrentMD);
     }
 
+    // Enforce the single-operand form of llvm.loop.vectorize.enable metadata.
+    if (CurrentMD->getNumOperands() > 0 &&
+        (CurrentMD->getOperand(0).equalsStr("llvm.loop.vectorize.enable") ||
+         CurrentMD->getOperand(0).equalsStr("llvm.loop.vectorize.disable")))
+      Check(CurrentMD->getNumOperands() == 1,
+            "Expected one operand for llvm.loop.vectorize.enable metadata",
+            CurrentMD);
+
     // Check these last, so we diagnose problems in operands first.
     Check(!CurrentMD->isTemporary(), "Expected no forward declarations!",
           CurrentMD);
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 72fae62e56e88..b5141bf8d5fb5 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -447,8 +447,7 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
   Metadata *FalseVal =
       ConstantAsMetadata::get(ConstantInt::get(Type::getInt1Ty(Context), 0));
   MDNode *DisableVectorize = MDNode::get(
-      Context,
-      {MDString::get(Context, "llvm.loop.vectorize.enable"), FalseVal});
+      Context, {MDString::get(Context, "llvm.loop.vectorize.disable")});
   MDNode *DisableLICMVersioning = MDNode::get(
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 58b52519beaf4..6b1a4a7d05129 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -405,12 +405,12 @@ TransformationMode llvm::hasUnrollAndJamTransformation(const Loop *L) {
 }
 
 TransformationMode llvm::hasVectorizeTransformation(const Loop *L) {
+  if (getBooleanLoopAttribute(L, "llvm.loop.vectorize.disable"))
+    return TM_SuppressedByUser;
+
   std::optional<bool> Enable =
       getOptionalBoolLoopAttribute(L, "llvm.loop.vectorize.enable");
 
-  if (Enable == false)
-    return TM_SuppressedByUser;
-
   std::optional<ElementCount> VectorizeWidth =
       getOptionalElementCountLoopAttribute(L);
   std::optional<int> InterleaveCount =
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index 8d875b2b6e492..9086880599231 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -287,6 +287,14 @@ void LoopVectorizeHints::getHintsFromMetadata() {
 
     // Check if the hint starts with the loop metadata prefix.
     StringRef Name = S->getString();
+    // The single-operand enable/disable pair carries no argument.
+    if (Args.empty()) {
+      if (Name == "llvm.loop.vectorize.enable")
+        Force.Value = FK_Enabled;
+      else if (Name == "llvm.loop.vectorize.disable")
+        Force.Value = FK_Disabled;
+      continue;
+    }
     if (Args.size() == 1)
       setHint(Name, Args[0]);
   }
diff --git a/llvm/test/Bitcode/upgrade-loop-metadata.ll b/llvm/test/Bitcode/upgrade-loop-metadata.ll
index 8dee907578760..9cd46acd26e84 100644
--- a/llvm/test/Bitcode/upgrade-loop-metadata.ll
+++ b/llvm/test/Bitcode/upgrade-loop-metadata.ll
@@ -29,7 +29,7 @@ for.end:                                          ; preds = %for.cond
 
 ; CHECK: !{!"llvm.loop.interleave.count", i32 4}
 ; CHECK: !{!"llvm.loop.vectorize.width", i32 8}
-; CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !{!"llvm.loop.vectorize.enable"}
 
 !0 = metadata !{metadata !"clang version 3.5.0 (trunk 211528)"}
 !1 = metadata !{metadata !1, metadata !2, metadata !3, metadata !4, metadata !4}
diff --git a/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
new file mode 100644
index 0000000000000..b757b787ce51c
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
@@ -0,0 +1,37 @@
+; Test that older bitcode carrying the two-operand form of
+; "llvm.loop.vectorize.enable" is auto-upgraded to the single-operand
+; enable/disable pair on load.
+;
+; RUN: llvm-dis < %s.bc | FileCheck %s
+; RUN: verify-uselistorder < %s.bc
+
+define void @enable_true() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+
+define void @enable_false() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !2
+exit:
+  ret void
+}
+
+; i1 true  -> single-operand enable.
+; i1 false -> disable.
+; CHECK: !{!"llvm.loop.vectorize.enable"}
+; CHECK: !{!"llvm.loop.vectorize.disable"}
+; The old two-operand nodes must be gone from the module.
+; CHECK-NOT: llvm.loop.vectorize.enable", i1
+; CHECK-NOT: llvm.loop.vectorize.disable", i1
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = distinct !{!2, !3}
+!3 = !{!"llvm.loop.vectorize.enable", i1 false}
diff --git a/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..e51d9ed57170e2593ed01c29f2ca45bc1fc90b2d
GIT binary patch
literal 2328
zcmbtWZ%iBK89!t2ImNtZ5*qAqcl8AtA&rom3BiZ$$@VqO)l$t(GmUDUz{Yl<Vu>-1
zF?6l$oW~`tRk;<4{2`S<ouaPV2d6?Ck=PP*lseR9EUKy~L(DFiB at +!UkSS85Y0sHz
z>W6J#cGA6f=ezfLpXc}IdF|kDt+v-;SQUn09u3>{!pncW_WC!|KW at l2J*O~MVT=aD
zbQKs at UBO^*!5ZPjV~y|Fvz5=Qv_|e-MO^c2m0C|%zU7H(PF6VHY>YS6$Fl}+b3n^p
zsq=cZ at ATewjB9H5y>0TwonIzv4?9msHG1Ysg*U6NZQ^WQ<JwOZMq}J*JmWfJcz?#$
z6^v>0#&P1x_l#bh{&{5d&UXisOk-E_SuNEW%3gXq$viins5BnOu*(L>4RSi{uhf3z
zxOiK88iwT1oXh#tz9alXmBV;MtvRG<wyP?xVH;|>1ndxV3KgTD{qFZYyW3a)x&PXO
zwT>H4K9v5n&E8n?-27|l${#G(ZWae#zwwi>`0?gPZ+-TK(_L}0R-XDljwo|5t^V%&
zYD{zeVApql>Bejx!RAH^EZU;o!`w)xH2T_8$;7 at RKU}EfU#&@C*mWI-J*~j7Hg%Mo
zFOdE^h4m6|ic1Mc6d#S?!>J+721b0PMY~bdrg6}78ER>Y^s`mv&xD4Y(wgP138gg`
zNrVFUYifMdDFy=gSU?$$hyjDB&()&MBr34PIhI%}p^^)=Y3Hd`hP=yAb8gztUN+AP
z=4GCnR$68SOI&HrmFmL1_-p-mN52?^F)9X<VxSifH{;>QRS<S0D)y+wA)OdZiKD4l
z%fk|Sm`9Ipp-ool$xe~$Q)I3 at uDHoXhMbR4i*D5yNuJB{T&g_INTvzibV+HRkdDEk
zPcE82SDMq3=92+2&@Bc7LZ}~)^ok*!7 at 8FIOD_=*F=E?=){|(>MidNa6RI#YQ!5#I
z&Q9N9$dwqqk|);-)ZZ1fU-zOZ&39fE%sE%nw}teIBN053dhyh5l8_OgU!@n_<b8%(
z%}{SBXgOhqD!El>ekGW4k~z&=0gxwimUZ5I2_B`y$4^0};t1rU#>blRU at G>?nig%|
zLPZzaVH-1d?DTq}{%(O<g#?^>b6T)Y3k`9}l;F9XWSPluSqW;i<ebw(Y8<i}0sy;3
z{a+T)W-swoUR5#>MV2TUh<~`yIW1aa5j^hXRS)uL-Gv^rjbCoM$+;MHw@|;yP#X+2
z&rrbNedZaz;XU&#Z=F$c69Sj(o(1-y*fGe{DUJk0{ilCJTP#t?qX#zP!6d9ADU`z>
z;806X3Zx<<99jk{V{to|mjF5|FqN~-Pg09hRNhXnW=MZiABzeyzXSv+vOFVP(BZ>Q
zv1d~JWfbOKpHNs5l4U|NE%TiqryL-raWYRk+%%-Rl%Y<?>inn9vS`adY~@v3E at D>>
zg^GFAhJh$7L$UAMgm6DTsuuO956wts(BZFnQw}gxa!J9GQ(6Gb2}#pcvjEZsMNN@!
z#OQ4W at VIbY1qy^Xz(w2s6FM;v#m7FtN1K7N%(UH<R+?ue(~}D8tkgLHx%ICYsCx<;
z&|3fiGc_{%?(yw%5R6#6C7Wf36=Fh}DY-__|8O%tmil_$-Z=w_39 at b{`A%>jsCQjT
z{0J?^s9d3bV~Sje(RT{eN}*nc4$`yJAhSg<THSB}JxZc=c5O-sH;dr_#21HbV(@WQ
z$cc}D&R?d$5s>O{g@(@sScQY&(|qSL|AQHY1 at x2RJJUdoYqNJx3FJCM<`tCRG!jKc
zEh?1-xy7QLB-(W$Y}9~Z2cRdtOFIqxC5?Wc*PFd<h(R;$>gqZWb4(_`p3kZ|$9TeH
zi@<k>;2azb|Dtv9{P{p@*w+_29~>C-wffGvJ%c`Y3CC}sU;PxW)o%vCo+f;x+m-->
zD7lu%e=&aMBmz|zd&^@Ut+~*ie{Yu1cB%TIivSZ<C*)8OnX)QTZbvh`6<BLAt$?F5
z6hzLs6~~Dp2HqzMpuke~j*S2<6#}T3o-~L-aQ7t4<p8$?bQPtwrGLpy-iy)z(TbbC
zvqy^C-i9`S1ufdPp<NhZx{avTMUrArcBO3az@!*7h*h6(3BCc^$X%XmNiW1Gx$$Od
zpxZ3K7z0kp1qHPjBlpyJkSy7VZ7q6aBkrf8-Qo}{hPq3D%#2{25G+ at DPVRW%4#AR^
zE?$<qqO22WUwes7)od`&+_RIbF$%-}4Eu!&FvfEhP}<dUIg^3~8fH?0tpam5 at A2BG
z9xt)*tc3GG+|kWw=2Z;LZH>gV>AbDH&)3^3m!g!-e*C5Qdzk%|Y$bfF at L=Wr{b%F0
z{qR^m{&P&v{}pq{u$P+=X%EM-ZKW>YEU)!*_ at 8kY?)EDuzL9^}(^pR at c)|DWt=~O3
z>@&8tvaRgdfsny#^7=VXyN@%r_qCf?KihGlql5Fc_j&w2qsQ;>JKp~NzBbPZS-i(W
nzTwa@$3Up>yw`W^Lf~xB?e!g%6 at B#IiaxrxOzYraIiG(4+2lin

literal 0
HcmV?d00001

diff --git a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
index eb72907730c6d..fdab35d493c48 100644
--- a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
+++ b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
@@ -74,7 +74,7 @@ attributes #0 = { alwaysinline }
 !1 = !{!"branch_weights", i32 64, i32 4}
 !2 = distinct !{!2, !3, !4, !5, !6}
 !3 = !{!"llvm.loop.unroll.disable"}
-!4 = !{!"llvm.loop.vectorize.enable", i1 false}
+!4 = !{!"llvm.loop.vectorize.disable"}
 !5 = !{!"llvm.loop.licm_versioning.disable"}
 !6 = !{!"llvm.loop.distribute.enable", i1 false}
 !7 = !{}
diff --git a/llvm/test/Transforms/IRCE/correct-loop-info.ll b/llvm/test/Transforms/IRCE/correct-loop-info.ll
index 7ea445718289a..a2115f18133e4 100644
--- a/llvm/test/Transforms/IRCE/correct-loop-info.ll
+++ b/llvm/test/Transforms/IRCE/correct-loop-info.ll
@@ -176,7 +176,7 @@ declare void @pluto()
 
 !0 = distinct !{!0, !1, !2, !3, !4}
 !1 = !{!"llvm.loop.unroll.disable"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 false}
+!2 = !{!"llvm.loop.vectorize.disable"}
 !3 = !{!"llvm.loop.licm_versioning.disable"}
 !4 = !{!"llvm.loop.distribute.enable", i1 false}
 !5 = !{}
diff --git a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
index ea169987cf2fc..04ed8aa56d8b4 100644
--- a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
+++ b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
@@ -128,7 +128,7 @@ exit:                                       ; preds = %guarded, %loop_, %prehead
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META2]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
 ; CHECK: [[META5]] = !{}
diff --git a/llvm/test/Transforms/IRCE/pre_post_loops.ll b/llvm/test/Transforms/IRCE/pre_post_loops.ll
index da2faa34719af..87f9a9a54e8bd 100644
--- a/llvm/test/Transforms/IRCE/pre_post_loops.ll
+++ b/llvm/test/Transforms/IRCE/pre_post_loops.ll
@@ -161,7 +161,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
index 68613feacba32..30ee5476acae0 100644
--- a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
+++ b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
@@ -488,7 +488,7 @@ deopt:                                          ; preds = %range_check_block
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META2]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.distribute.enable", i1 false}
 ; CHECK: [[META5]] = !{}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
index 7236cf4817ade..3d8834f1aa52d 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
@@ -445,7 +445,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
index 546e1b0faa1c3..b3adbad8f41a2 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
@@ -652,7 +652,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.enable", i1 false}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
index 1bc132663331b..81f2f8292c7f6 100644
--- a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
+++ b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
@@ -50,7 +50,7 @@ entry:
 !10 = !{!"llvm.loop.distribute.enable", i1 true}
 !11 = !{!"llvm.loop.distribute.followup_all", !7, !8, !9, !12, !13, !14}
 !12 = !{!"llvm.loop.vectorize.width", i32 8}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !{!"llvm.loop.vectorize.followup_all", !7, !8, !9, !15, !16}
 !15 = !{!"llvm.loop.isvectorized"}
 !16 = !{!"llvm.loop.unroll.count", i32 1}
@@ -69,7 +69,7 @@ entry:
 ; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable", i1 true}
 ; CHECK: [[META11]] = !{!"llvm.loop.distribute.followup_all", [[META7]], [[META8]], [[META9]], [[META12:![0-9]+]], [[META13:![0-9]+]], [[META14:![0-9]+]]}
 ; CHECK: [[META12]] = !{!"llvm.loop.vectorize.width", i32 8}
-; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: [[META14]] = !{!"llvm.loop.vectorize.followup_all", [[META7]], [[META8]], [[META9]], [[META15:![0-9]+]], [[META16:![0-9]+]]}
 ; CHECK: [[META15]] = !{!"llvm.loop.isvectorized"}
 ; CHECK: [[META16]] = !{!"llvm.loop.unroll.count", i32 1}
diff --git a/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll b/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
index 644795f08ac22..ae3d76f2dce21 100644
--- a/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
+++ b/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
@@ -61,5 +61,5 @@ declare void @llvm.memcpy.p0.p0.i64(ptr nocapture writeonly, ptr nocapture reado
 
 !1 = distinct !{!1, !2, !3, !{!"llvm.loop.parallel_accesses", !4}}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = distinct !{} ; access group
diff --git a/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll b/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
index 875f45ebd951c..04b2ec7f9e8b3 100644
--- a/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
+++ b/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
@@ -379,12 +379,12 @@ exit:
 !0 = distinct !{!0, !1}
 !1 = !{!"llvm.loop.vectorize.width", i32 1}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 false}
+!3 = !{!"llvm.loop.vectorize.disable"}
 
 attributes #0 = { "target-features"="+sve2" }
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.vectorize.width", i32 1}
 ; CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]]}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ;.
diff --git a/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll b/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
index 4465288e0f1b2..6e28973518552 100644
--- a/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
+++ b/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
@@ -12,7 +12,7 @@ target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f3
 ; CHECK-NOT: loop-inc
 ; CHECK: br i1 %cmp, label %return, label %loop_cond, !llvm.loop !0
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.vectorize.enable"}
 define i32 @foo(i32 %a) {
 entry:
   br label %loop_cond
@@ -31,4 +31,4 @@ return:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll b/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
index 6fcd552d9cffa..33dad5c3a68b0 100644
--- a/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
@@ -26,7 +26,7 @@ simd.if.end:
 }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.isvectorized"}
 
 
diff --git a/llvm/test/Transforms/LoopTransformWarning/optnone.ll b/llvm/test/Transforms/LoopTransformWarning/optnone.ll
index 469f3ce7ffc2e..caa75266d28ee 100644
--- a/llvm/test/Transforms/LoopTransformWarning/optnone.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/optnone.ll
@@ -40,7 +40,7 @@ attributes #0 = { noinline optnone }
 !1 = !{!"llvm.loop.unroll.enable"}
 !2 = !{!"llvm.loop.distribute.enable"}
 !3 = !{!"llvm.loop.unroll_and_jam.enable"}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 
 
 ; CHECK-NOT: warning
diff --git a/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll b/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
index fb33df0967db4..41bf5fe0dbe76 100644
--- a/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
@@ -73,7 +73,7 @@ for.end:
 !12 = !DILocation(line: 3, column: 8, scope: !13)
 !13 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !14 = !{!14, !15, !15}
-!15 = !{!"llvm.loop.vectorize.enable", i1 true}
+!15 = !{!"llvm.loop.vectorize.enable"}
 !16 = !DILocation(line: 4, column: 5, scope: !17)
 !17 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !13)
 !18 = !{!19, !19, i64 0}
diff --git a/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll b/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
index 0168bddacca51..4fc87ac4044be 100644
--- a/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
@@ -105,7 +105,7 @@ exit:
 ; (runtime unroll disabled to prevent the unroller from succeeding on this
 ; runtime-trip-count loop, so the hint survives to transform-warning)
 !10 = distinct !{!10, !11, !12, !13}
-!11 = !{!"llvm.loop.vectorize.enable", i1 false}
+!11 = !{!"llvm.loop.vectorize.disable"}
 !12 = !{!"llvm.loop.unroll.enable"}
 !13 = !{!"llvm.loop.unroll.runtime.disable"}
 
diff --git a/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll b/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
index 28202bb4adbd5..74f22685e6542 100644
--- a/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
+++ b/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
@@ -31,7 +31,7 @@ for.end:                                          ; preds = %for.body
   ret void
 }
 !1 = !{!1, !2, !3, !4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.unroll.count", i32 4}
 !4 = !{!"llvm.loop.vectorize.width", i32 8}
 
@@ -139,7 +139,7 @@ for.inc5.1:                                       ; preds = %for.body3.1
 
 
 ; CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[VEC_ENABLE:.*]], ![[WIDTH_8:.*]], ![[UNROLL_DISABLE:.*]]}
-; CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: ![[WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 ; CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2]], ![[UNROLL_FULL:.*]]}
diff --git a/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll b/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
index 723a0273e90a7..68913a90e8185 100644
--- a/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
+++ b/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
@@ -101,7 +101,7 @@ exit:
 !99 = !DISubroutineType(types: !{})
 
 !10 = distinct !{!10, !11}
-!11 = !{!"llvm.loop.vectorize.enable", i1 false}
+!11 = !{!"llvm.loop.vectorize.disable"}
 
 !100 = distinct !DISubprogram(name: "plain_loop", scope: !1, file: !1, line: 1, type: !99, isLocal: false, isDefinition: true, scopeLine: 1, unit: !0)
 !108 = !DILocation(line: 1, column: 1, scope: !100)
diff --git a/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll b/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
index 0941c2367dfb8..51ce91a0b4c94 100644
--- a/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
+++ b/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
@@ -16,4 +16,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll b/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
index 54dbddff5d9fd..47ec13b653a18 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
@@ -324,4 +324,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
index cc1eb9c9614f9..09852d7951c09 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
@@ -1353,4 +1353,4 @@ attributes #3 = { "target-cpu"="neoverse-v2" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 8}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
index f38d8897ca8ff..61ebc1e306ca5 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
@@ -2120,4 +2120,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.interleave.count", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
index d64504685fa5d..6a78f2b2bcf2b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
@@ -1625,4 +1625,4 @@ attributes #1 = { "target-cpu"="grace" }
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
index e007b706d3d44..f49b18c80f528 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
@@ -89,6 +89,6 @@ attributes #0 = { "target-features"="+neon,+sve" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = distinct !{!5, !6}
 !6 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll b/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
index 14d6ef1716bd6..127057c58db2d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
@@ -163,7 +163,7 @@ attributes #0 = { uwtable vscale_range(1,16) "aarch64_pstate_sm_body" "target-fe
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 16}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[META0]] = !{[[META1:![0-9]+]]}
 ; CHECK: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll b/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
index 11370a93b2ffa..a533b861067a6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
@@ -779,4 +779,4 @@ for.cond.cleanup:
 attributes #0 = { "target-features"="+sve" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
index b111eb26b286a..8ac02a0cd5961 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
@@ -102,4 +102,4 @@ exit:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
index 1ad20d7ed0655..2396cf4cf905e 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
@@ -150,4 +150,4 @@ exit:
 
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
index 2f27092622192..0282a9cac5c8d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
@@ -448,7 +448,7 @@ exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-cpu"="apple-m1" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
index 0985d7f052a90..339aea4551d6b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
@@ -394,6 +394,6 @@ for.exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-features"="+neon" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
index f0f9e8e74b3f5..3ba46e2fd46d6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
@@ -2395,4 +2395,4 @@ exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
index 09e5959f7083c..f19db4519c09a 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
@@ -2692,6 +2692,6 @@ for.body:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { vscale_range(1,16) "target-features"="+neon,+dotprod,+sve" "target-cpu"="neoverse-v2" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
index dd21d1994d95d..641d6db59147d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
@@ -524,7 +524,7 @@ for.exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-cpu"="apple-m1" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll b/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
index 6bc2c098576c7..980d501954124 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
@@ -661,6 +661,6 @@ attributes #0 = { "target-features"="+neon,+sve" vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
 
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll b/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
index 363223a66f398..3c47627e5f45f 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
@@ -371,7 +371,7 @@ attributes #1 = { "target-cpu"="neoverse-v2" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 ; BFI computes if is taken 20 times, and loop 32 times. Make sure we round the
 ; divisor up to 2 so that we don't vectorize the loop unprofitably.
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
index 13af29855be23..71e814bd4bdfd 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
@@ -88,4 +88,4 @@ attributes #0 = { "target-features"="+sve" }
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
index b68c7151c3a89..25d01a2295899 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
@@ -64,4 +64,4 @@ for.end:
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
index 75e30632c5cc8..cec1dd6c359af 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
@@ -456,4 +456,4 @@ for.end:
 !1 = !{!"llvm.loop.vectorize.width", i32 8}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 2}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll b/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
index 4b2e5d051730b..b79cc7cf2bcee 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
@@ -1416,6 +1416,6 @@ for.cond.cleanup:
 !8 = !{!"llvm.loop.vectorize.width", i32 1}
 !9 = !{!"llvm.loop.interleave.count", i32 1}
 !10 = !{!"llvm.loop.interleave.count", i32 4}
-!11 = !{!"llvm.loop.vectorize.enable", i1 true}
+!11 = !{!"llvm.loop.vectorize.enable"}
 !12 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 !13 = distinct !{!13, !6, !9, !11}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
index e98c726e82f8e..0d5d4a42f100c 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
@@ -95,4 +95,4 @@ for.end:
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
index b6c2abe3d805d..d51ae84401031 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
@@ -183,4 +183,4 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
index 834e915d4685e..687a16cefd65b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
@@ -67,5 +67,5 @@ attributes #0 = { "target-cpu"="generic" "target-features"="+neon,+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !2, !3, !4, !5}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
index 6fbab439e2035..eeae794f068cf 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
@@ -187,4 +187,4 @@ attributes #0 = { vscale_range(1, 16) "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
index ba1fd71659b59..b43e2c8359525 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
@@ -271,4 +271,4 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
index 4073b1055b504..eb12b7f5c1cd4 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
@@ -74,6 +74,6 @@ attributes #0 = { "target-features"="+sve" }
 !0 = distinct !{!0, !1, !2, !3, !4, !5}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
index 8b3fa9279768e..a01188ca95678 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
@@ -106,5 +106,5 @@ attributes #0 = { "target-features"="+neon,+sve" vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
index a7c164fa7de16..561cbce0f8738 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
@@ -97,6 +97,6 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !7, !3, !4, !5}
 !7 = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
index 826617da6f903..5ee66f493a9cd 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
@@ -76,4 +76,4 @@ exit:
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
index fb17b9a80d09b..e80df2a09824c 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
@@ -401,4 +401,4 @@ attributes #0 = { "target-features"="+sve" }
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
index 78858fe512fd4..0b7782c95c417 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
@@ -263,4 +263,4 @@ attributes #0 = { "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
index 6d16222f25f92..91df2229db6e6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
@@ -101,4 +101,4 @@ attributes #0 = {"target-cpu"="generic" "target-features"="+neon,+sve"}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
index 4f53d31fd98fa..82fe12a43f842 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
@@ -152,6 +152,6 @@ attributes #0 = { "target-cpu"="generic" "target-features"="+neon,+sve" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
 
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
index 1794a6ffc3ccd..0039941d2d47f 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
@@ -58,4 +58,4 @@ attributes #0 = { "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
index 017982a4f8a8a..f1085e21d30f1 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
@@ -347,7 +347,7 @@ attributes #0 = { vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
 !6 = distinct !{!6, !1, !7, !3, !4, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll b/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
index fc20277b1d858..a354f94ba832d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
@@ -39,4 +39,4 @@ exit:
 
 !0 = !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i64 32}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
index 90c4ed0fb8ded..d99ab30f5ebb2 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
@@ -119,5 +119,5 @@ attributes #0 = {"target-cpu"="generic" "target-features"="+neon,+sve" vscale_ra
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
index 7106be6ec78dd..cf06100f988ef 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
@@ -266,4 +266,4 @@ attributes #1 = { "target-cpu"="neoverse-v2" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
index 3d8863368e74c..a15b377d4ef3e 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
@@ -56,4 +56,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll b/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
index fbc313eddea8a..3b29335c36301 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
@@ -54,7 +54,7 @@ exit:
 attributes #0 = { "target-cpu"="neoverse-v2" }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll b/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
index 83be0708774f1..8d8766051a5b9 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
@@ -482,4 +482,4 @@ for.cond.cleanup:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll b/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
index 1b5db04061631..75862d9c72e00 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
@@ -422,7 +422,7 @@ for.body:
 attributes #0 = { nofree norecurse nounwind "target-features"="+armv8.1-m.main,+mve.fp" }
 
 !5 = distinct !{!5, !6}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
 
 !7 = distinct !{!7, !8}
 !8 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
index f205e21762317..3038b7fd935d3 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
@@ -81,4 +81,4 @@ for.body:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
index c7af4646ccdee..a477fee265d59 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
@@ -102,7 +102,7 @@ for.body:
 
 !10 = distinct !{!10, !11, !12}
 !11 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
-!12 = !{!"llvm.loop.vectorize.enable", i1 true}
+!12 = !{!"llvm.loop.vectorize.enable"}
 
 !14 = distinct !{!14, !15}
 !15 = !{!"llvm.loop.interleave.count", i32 4}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll b/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
index 3e3c6c09050b4..2cccae4de2fa5 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
@@ -72,7 +72,7 @@ exit:
 !9 = !DILocation(line: 2, column: 27, scope: !3)
 !10 = !DILocation(line: 2, column: 5, scope: !3)
 !11 = distinct !{!11, !12}
-!12 = !{!"llvm.loop.vectorize.enable", i1 true}
+!12 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[META0:![0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: [[META1:![0-9]+]], producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 ; CHECK: [[META1]] = !DIFile(filename: "{{.*}}dbg-tail-folding-by-evl.cpp", directory: {{.*}})
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll b/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
index 7adb8b6e1b2cf..22560793e4504 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
@@ -31,4 +31,4 @@ for.end:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
index 1c9e48fdf4aa6..100ecc172fab0 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
@@ -908,5 +908,5 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = distinct !{!4, !1, !3}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
index 4e18b73abc73f..b068c03e8729c 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
@@ -770,7 +770,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL-OUTLOOP: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; IF-EVL-OUTLOOP: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
index 062c052b95391..43c734471537b 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
@@ -623,7 +623,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; IF-EVL: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
index 1fad71f9f9e04..ba6af22e10718 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
@@ -1658,4 +1658,4 @@ for.end:
 
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
index 935e37e94c266..77808152757e8 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
@@ -757,4 +757,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
index 8b86d42fe4377..e7631a2ddf9b5 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
@@ -234,7 +234,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL-OUTLOOP: [[META0]] = !{[[META1:![0-9]+]]}
 ; IF-EVL-OUTLOOP: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
index 85dc6b407c877..19099b43557ad 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
@@ -91,4 +91,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
index a918e4c61425d..88a3c99f77b66 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
@@ -1682,4 +1682,4 @@ for.end:
 
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
index 7a7ae1edbc415..8313e911e2b40 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
@@ -59,4 +59,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
index 64d412666c74f..256528c25a781 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
@@ -895,4 +895,4 @@ for.end:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll b/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
index c4ceb53200cce..97a3eb830213d 100644
--- a/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
+++ b/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
@@ -59,15 +59,15 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.mustprogress"}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; SPARC32: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; SPARC32: [[META1]] = !{!"llvm.loop.mustprogress"}
-; SPARC32: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; SPARC32: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; SPARC64: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; SPARC64: [[META1]] = !{!"llvm.loop.mustprogress"}
-; SPARC64: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; SPARC64: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
 ; SPARC32: {{.*}}
diff --git a/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll b/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
index 5058b4c85d057..4e55fcc295fe1 100644
--- a/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
+++ b/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
@@ -35,4 +35,4 @@ simd.if.end:
 !6 = distinct !{}
 !7 = distinct !{!7, !8, !9}
 !8 = !{!"llvm.loop.parallel_accesses", !6}
-!9 = !{!"llvm.loop.vectorize.enable", i1 true}
+!9 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
index 7cdc33a44b3a6..c5f61f94e2c2f 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
@@ -72,4 +72,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
index 2d818db242689..51f3305a25893 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
@@ -222,4 +222,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll b/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
index 92bb9704c4fd2..4ec0ec66d4321 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
@@ -318,5 +318,5 @@ exit:
 
 !1 = distinct !{!1, !2, !3, !4}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
index 6eff6d1df3523..a86a211bf1417 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
@@ -70,4 +70,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
index 7f442f7d72e78..2edddc1079178 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
@@ -40,4 +40,4 @@ for.end10:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
index 7ad03ebaca7c2..985a2c693aded 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
@@ -484,15 +484,15 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 !4 = distinct !{!4, !5, !6}
 !5 = !{!"llvm.loop.vectorize.width", i32 4}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
 
 !7 = distinct !{!7, !8, !9}
 !8 = !{!"llvm.loop.vectorize.width", i32 8}
-!9 = !{!"llvm.loop.vectorize.enable", i1 true}
+!9 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__expf_finite(float) #0
 declare double @__exp_finite(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll b/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
index 9cb7887346d3e..283aeb2a5ba21 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
@@ -1016,4 +1016,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.interleave.count", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll b/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
index 1853b2648af7e..49f5afb3aa705 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
@@ -1117,4 +1117,4 @@ attributes #4 = { "target-cpu"="haswell" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll b/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
index 165d81b285a19..54a265110185a 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
@@ -642,7 +642,7 @@ exit:
 
 !1 = distinct !{!1, !2, !3, !4}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 
 define void @wide_iv_trunc_reuse(ptr %dst) {
diff --git a/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll b/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
index 9823d18da2905..d08c1a404e6c3 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
@@ -176,4 +176,4 @@ attributes #0 = { "target-cpu"="skylake-avx512" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
index 025e24bcbc5eb..5f7261240e3d4 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
@@ -28,7 +28,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @sin_f32(ptr nocapture %varray) {
@@ -56,7 +56,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 8}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f64_intrinsic(
@@ -83,7 +83,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 2}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f32_intrinsic(
@@ -110,7 +110,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 8}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64(
@@ -137,7 +137,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 2}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32(
@@ -164,7 +164,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 8}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64_intrinsic(
@@ -191,7 +191,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 2}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32_intrinsic(
@@ -218,7 +218,7 @@ for.end:
 
 !81 = distinct !{!81, !82, !83}
 !82 = !{!"llvm.loop.vectorize.width", i32 8}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @exp_f32(ptr nocapture %varray) {
@@ -245,7 +245,7 @@ for.end:
 
 !91 = distinct !{!91, !92, !93}
 !92 = !{!"llvm.loop.vectorize.width", i32 8}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f32_intrin(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f32_intrin
@@ -271,7 +271,7 @@ for.end:
 
 !101 = distinct !{!101, !102, !103}
 !102 = !{!"llvm.loop.vectorize.width", i32 8}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @log_f32(ptr nocapture %varray) {
@@ -298,7 +298,7 @@ for.end:
 
 !111 = distinct !{!111, !112, !113}
 !112 = !{!"llvm.loop.vectorize.width", i32 8}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -326,7 +326,7 @@ for.end:
 
 !121 = distinct !{!121, !122, !123}
 !122 = !{!"llvm.loop.vectorize.width", i32 8}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32_intrin(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32_intrin
@@ -354,7 +354,7 @@ for.end:
 
 !131 = distinct !{!131, !132, !133}
 !132 = !{!"llvm.loop.vectorize.width", i32 8}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f64(
@@ -381,7 +381,7 @@ for.end:
 
 !141 = distinct !{!141, !142, !143}
 !142 = !{!"llvm.loop.vectorize.width", i32 2}
-!143 = !{!"llvm.loop.vectorize.enable", i1 true}
+!143 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @tan_f32(ptr nocapture %varray) {
@@ -409,7 +409,7 @@ for.end:
 
 !151 = distinct !{!151, !152, !153}
 !152 = !{!"llvm.loop.vectorize.width", i32 8}
-!153 = !{!"llvm.loop.vectorize.enable", i1 true}
+!153 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f64_intrinsic(
@@ -436,7 +436,7 @@ for.end:
 
 !161 = distinct !{!161, !162, !163}
 !162 = !{!"llvm.loop.vectorize.width", i32 2}
-!163 = !{!"llvm.loop.vectorize.enable", i1 true}
+!163 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f32_intrinsic(
@@ -465,7 +465,7 @@ for.end:
 
 !171 = distinct !{!171, !172, !173}
 !172 = !{!"llvm.loop.vectorize.width", i32 8}
-!173 = !{!"llvm.loop.vectorize.enable", i1 true}
+!173 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { nounwind readnone }
 
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
index 95aec377738b8..176ce4e05299f 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
@@ -27,7 +27,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f64
@@ -54,7 +54,7 @@ for.end:
 
 !11 = distinct !{!11, !12, !13}
 !12 = !{!"llvm.loop.vectorize.width", i32 4}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 
 define void @log_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @log_f32
@@ -81,7 +81,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @log_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @log_f64
@@ -108,7 +108,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -137,7 +137,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f64(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f64
@@ -166,7 +166,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__expf_finite(float) #0
 declare double @__exp_finite(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
index 071f262ec903b..acbd8dbbd76a9 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
@@ -28,7 +28,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @sin_f32(ptr nocapture %varray) {
@@ -56,7 +56,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f64_intrinsic(
@@ -83,7 +83,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f32_intrinsic(
@@ -110,7 +110,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64(
@@ -137,7 +137,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32(
@@ -164,7 +164,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 4}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64_intrinsic(
@@ -191,7 +191,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 4}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32_intrinsic(
@@ -218,7 +218,7 @@ for.end:
 
 !81 = distinct !{!81, !82, !83}
 !82 = !{!"llvm.loop.vectorize.width", i32 4}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @exp_f32(ptr nocapture %varray) {
@@ -245,7 +245,7 @@ for.end:
 
 !91 = distinct !{!91, !92, !93}
 !92 = !{!"llvm.loop.vectorize.width", i32 4}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f32_intrin(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f32_intrin
@@ -271,7 +271,7 @@ for.end:
 
 !101 = distinct !{!101, !102, !103}
 !102 = !{!"llvm.loop.vectorize.width", i32 4}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @log_f32(ptr nocapture %varray) {
@@ -298,7 +298,7 @@ for.end:
 
 !111 = distinct !{!111, !112, !113}
 !112 = !{!"llvm.loop.vectorize.width", i32 4}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -326,7 +326,7 @@ for.end:
 
 !121 = distinct !{!121, !122, !123}
 !122 = !{!"llvm.loop.vectorize.width", i32 4}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32_intrin(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32_intrin
@@ -354,7 +354,7 @@ for.end:
 
 !131 = distinct !{!131, !132, !133}
 !132 = !{!"llvm.loop.vectorize.width", i32 4}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { nounwind readnone }
 
diff --git a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
index e61e22e50ff56..ae77b4270ab3e 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
@@ -90,7 +90,7 @@ for.body:
 !24 = !DILocation(line: 17, column: 19, scope: !20)
 !25 = !DILocation(line: 17, column: 11, scope: !20)
 !26 = distinct !{!26, !27}
-!27 = !{!"llvm.loop.vectorize.enable", i1 true}
+!27 = !{!"llvm.loop.vectorize.enable"}
 !28 = distinct !DICompileUnit(language: DW_LANG_C99, producer: "clang",
                              file: !5,
                              isOptimized: true, flags: "-O2",
diff --git a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
index a4bfbcb8ca4d2..cbedbf7fd10c0 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
@@ -92,7 +92,7 @@ for.body:
 !24 = !DILocation(line: 17, column: 19, scope: !20)
 !25 = !DILocation(line: 17, column: 11, scope: !20)
 !26 = distinct !{!26, !27, !18}
-!27 = !{!"llvm.loop.vectorize.enable", i1 true}
+!27 = !{!"llvm.loop.vectorize.enable"}
 !28 = distinct !DICompileUnit(language: DW_LANG_C99, producer: "clang",
                              file: !5,
                              isOptimized: true, flags: "-O2",
diff --git a/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll b/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
index d09f42ff558db..7c4f98670719f 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
@@ -92,4 +92,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll b/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
index d10449327d9c0..5b8020cc6630a 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
@@ -114,4 +114,4 @@ exit:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll b/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
index 21146e2e82216..b3f2048e81663 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
@@ -31,4 +31,4 @@ exit:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
index a33874a871443..a52e2c7a0dde8 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
@@ -369,7 +369,7 @@ exit:
 attributes #0 = { "target-cpu"="znver2" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
index ddb1905f3ffc4..edb115ce81b20 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
@@ -1263,4 +1263,4 @@ exit:
 attributes #0 = { "target-cpu"="znver2" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
index 6586b08e5c294..a768a38da5593 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
@@ -33,7 +33,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__exp_finite(double) #0
@@ -62,7 +62,7 @@ for.end:
 
 !11 = distinct !{!11, !12, !13}
 !12 = !{!"llvm.loop.vectorize.width", i32 4}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 
 
 
@@ -93,7 +93,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log_finite(double) #0
@@ -122,7 +122,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare float @__powf_finite(float, float) #0
@@ -153,7 +153,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__pow_finite(double, double) #0
@@ -184,7 +184,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__exp2f_finite(float) #0
 
@@ -213,7 +213,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 4}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 declare double @__exp2_finite(double) #0
 
@@ -242,7 +242,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 4}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__log2f_finite(float) #0
 
@@ -270,7 +270,7 @@ for.end:
 
 !81 = distinct !{!21, !22, !23}
 !82 = !{!"llvm.loop.vectorize.width", i32 4}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log2_finite(double) #0
@@ -299,7 +299,7 @@ for.end:
 
 !91 = distinct !{!31, !32, !33}
 !92 = !{!"llvm.loop.vectorize.width", i32 4}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__log10f_finite(float) #0
 
@@ -327,7 +327,7 @@ for.end:
 
 !101 = distinct !{!21, !22, !23}
 !102 = !{!"llvm.loop.vectorize.width", i32 4}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log10_finite(double) #0
@@ -356,7 +356,7 @@ for.end:
 
 !111 = distinct !{!31, !32, !33}
 !112 = !{!"llvm.loop.vectorize.width", i32 4}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__sqrtf_finite(float) #0
 
@@ -384,7 +384,7 @@ for.end:
 
 !121 = distinct !{!21, !22, !23}
 !122 = !{!"llvm.loop.vectorize.width", i32 4}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__sqrt_finite(double) #0
@@ -413,4 +413,4 @@ for.end:
 
 !131 = distinct !{!31, !32, !33}
 !132 = !{!"llvm.loop.vectorize.width", i32 4}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll b/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
index 52c504b090ae1..a1def726e10ed 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
@@ -143,7 +143,7 @@ for.inc:
 attributes #0 = { "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "use-soft-float"="false" }
 
 !6 = distinct !{!6, !7}
-!7 = !{!"llvm.loop.vectorize.enable", i1 true}
+!7 = !{!"llvm.loop.vectorize.enable"}
 
 !8 = distinct !{!8, !9}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll b/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
index 90866061c4188..e5b88dd5219db 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
@@ -172,7 +172,7 @@ attributes #0 = { optsize "target-cpu"="core-avx2" "target-features"="+avx,+avx2
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
 
 !10 = distinct !{!10, !11}
 !11 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
index dd30c045182e4..0333344025de1 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
@@ -49,7 +49,7 @@ for.end:
 }
 
 !1 = !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !11 = distinct !{}
 
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
index 78b14423578fb..98c0a3661fa1c 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
@@ -107,7 +107,7 @@ for.end:
 }
 
 !1 = !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !11 = distinct !{}
 
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll b/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
index 5f2a93b230995..c389765dff2d1 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
@@ -320,7 +320,7 @@ declare i32 @foo(...)
 !12 = !DILocation(line: 3, column: 8, scope: !13)
 !13 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !14 = !{!14, !15, !15}
-!15 = !{!"llvm.loop.vectorize.enable", i1 true}
+!15 = !{!"llvm.loop.vectorize.enable"}
 !16 = !DILocation(line: 4, column: 5, scope: !17)
 !17 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !13)
 !22 = !DILocation(line: 5, column: 9, scope: !23)
diff --git a/llvm/test/Transforms/LoopVectorize/alias-mask.ll b/llvm/test/Transforms/LoopVectorize/alias-mask.ll
index 7035332a1b763..2a811955e8bd3 100644
--- a/llvm/test/Transforms/LoopVectorize/alias-mask.ll
+++ b/llvm/test/Transforms/LoopVectorize/alias-mask.ll
@@ -650,4 +650,4 @@ exit:
 
 
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll b/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
index 45fa8d19952a4..ab61d2aa710ce 100644
--- a/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
+++ b/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
@@ -131,7 +131,7 @@ exit:
 !30 = !DILocation(line: 10, column: 24, scope: !16)
 !31 = distinct !{!31, !21, !32, !33}
 !32 = !DILocation(line: 13, column: 13, scope: !12)
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 !34 = !DILocation(line: 10, column: 5, scope: !12)
 ;.
 ; CHECK: [[META0:![0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C11, file: [[META1:![0-9]+]], producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
diff --git a/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll b/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
index 31454401876b6..da41fa7a9da93 100644
--- a/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
+++ b/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
@@ -63,7 +63,7 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 false}
+!1 = !{!"llvm.loop.vectorize.disable"}
 
 ; ALL-LABEL: 'disable_nonforced' from <stdin>
 ; ALL-NOT: LV: We can vectorize this loop
diff --git a/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll b/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
index 8c4e50fdd7a61..e7622d30f5e2c 100644
--- a/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
+++ b/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
@@ -26,4 +26,4 @@ for.end:
   ret void
 }
 
-!0 = !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.vectorize.enable", i32 1}}
+!0 = !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.vectorize.enable"}}
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
index a610f0669f483..7393ad8ea4dab 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
@@ -219,7 +219,7 @@ for.end15:
 ; Case 1
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 4}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
 ; Case 2
 !9 = distinct !{!9, !8}
 ; Case 3
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
index ef6fabb11694b..9c87b9cb1e818 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
@@ -161,4 +161,4 @@ for.end15:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 8}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
index 2152c48c61e81..75858e626b728 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
@@ -124,4 +124,4 @@ for.end19:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 8}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll b/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
index 1fe802f9e1093..6e10cb0181629 100644
--- a/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
+++ b/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
@@ -108,7 +108,7 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.followup_all", !3, !4}
 !3 = !{!"llvm.loop.isvectorized"}
 !4 = !{!"llvm.loop.unroll.count", i32 8}
diff --git a/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll b/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
index 69e97069ae498..fc6cfe5942254 100644
--- a/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
+++ b/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
@@ -89,4 +89,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll b/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
index d411783fa9df8..0d691a89a2241 100644
--- a/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
+++ b/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
@@ -86,4 +86,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll b/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
index 93c81c02d4e5d..6a80b02aaf50e 100644
--- a/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
+++ b/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
@@ -129,7 +129,7 @@ done:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.unroll.disable"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
diff --git a/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll b/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
index bfa9263bb5cf7..2f23747f44f9c 100644
--- a/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
@@ -79,7 +79,7 @@ for.end15:
 !10 = !DILocation(line: 4, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 4, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !DILocation(line: 5, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 4, column: 36, file: !1, scope: !11)
 !16 = !DILocation(line: 9, column: 8, scope: !17)
diff --git a/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll b/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
index 2c1f485aab080..c00086ea9496e 100644
--- a/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
@@ -63,7 +63,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.width", i32 1}
 !3 = distinct !{!3, !1, !2, !4}
 !4 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/no_switch.ll b/llvm/test/Transforms/LoopVectorize/no_switch.ll
index c3662f1b4be5d..531dfd0aed5f0 100644
--- a/llvm/test/Transforms/LoopVectorize/no_switch.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_switch.ll
@@ -72,7 +72,7 @@ for.end:
 !10 = !DILocation(line: 3, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !DILocation(line: 4, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !11)
 !16 = !{!17, !17, i64 0}
diff --git a/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll b/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
index d73264dc01836..4c989e0b3428a 100644
--- a/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
@@ -76,7 +76,7 @@ for.end:
 !10 = !DILocation(line: 3, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 false}
+!13 = !{!"llvm.loop.vectorize.disable"}
 !14 = !DILocation(line: 4, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !11)
 !16 = !{!17, !17, i64 0}
diff --git a/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
index 8020480f0dc65..efbfc49a21804 100644
--- a/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
@@ -66,7 +66,7 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!4}
 !4 = distinct !{!4, !5}
 !5 = distinct !{!5}
diff --git a/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
index 2c46af4ccbfe2..e59e5fbfb12fb 100644
--- a/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
@@ -454,4 +454,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/optsize.ll b/llvm/test/Transforms/LoopVectorize/optsize.ll
index 2fe02c62c7790..294dbbbb3a5e2 100644
--- a/llvm/test/Transforms/LoopVectorize/optsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/optsize.ll
@@ -1071,7 +1071,7 @@ exit:
 !13 = !{i32 999999, i64 1, i32 2}
 !14 = !{!"function_entry_count", i64 0}
 !15 = distinct !{!15, !16}
-!16 = !{!"llvm.loop.vectorize.enable", i1 true}
+!16 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[PROF14]] = !{!"function_entry_count", i64 0}
 ; CHECK: [[LOOP15]] = distinct !{[[LOOP15]], [[META16:![0-9]+]], [[META17:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
index 6e19f71736544..2618f15665b7d 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
@@ -165,4 +165,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
index 960a6e6445ee1..3ec0a6e624303 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
@@ -207,4 +207,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
index 4c0b5c618d7ac..045f0f58847c4 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
@@ -119,4 +119,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
index 6f47696b8008b..ac4b91f40541e 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
@@ -161,4 +161,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
index ebe1f07c402ea..358a2edb05ba8 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
@@ -285,4 +285,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
index e0ad90d9c2bc0..a5442f4cdb7f0 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
@@ -46,4 +46,4 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
index 778805965e57e..6cd433ee48246 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
@@ -317,4 +317,4 @@ innermost.loop.j1:
 !3 = distinct !{!3, !4, !5, !6}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
 !5 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
index 07293812a6ffb..30ba3770b172d 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
@@ -134,6 +134,6 @@ exit:
 }
 
 !1 = distinct !{!1, !2, !3, !4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
index cda0b24a69180..48ccd47f58f65 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
@@ -86,4 +86,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
index 1353445a7c29c..8a0cf20c0fd9c 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
@@ -154,4 +154,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
index e7d284726b1c2..035c80110ae98 100644
--- a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
+++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
@@ -611,4 +611,4 @@ for.end:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
index dc095e55edd15..c77e3cf806598 100644
--- a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
+++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
@@ -3136,4 +3136,4 @@ for.exit:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/remove_metadata.ll b/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
index 96de172a0ea7c..de9f2467a6d3e 100644
--- a/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
+++ b/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
@@ -25,7 +25,7 @@ for.end:
   ret void
 }
 
-!0 = !{!0, !{!"llvm.loop.vectorize.some_property"}, !{!"llvm.loop.vectorize.enable", i32 1}}
+!0 = !{!0, !{!"llvm.loop.vectorize.some_property"}, !{!"llvm.loop.vectorize.enable"}}
 
 ; CHECK-NOT: llvm.loop.vectorize.
 ; CHECK: {!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll b/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
index 5376eb86882b7..d4854fbcf9c75 100644
--- a/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
+++ b/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
@@ -79,12 +79,12 @@ exit:
 !0 = distinct !{!0, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 ;.
 ; LIMIT0: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; LIMIT0: [[META1]] = !{!"llvm.loop.vectorize.width", i32 2}
-; LIMIT0: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; LIMIT0: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; LIMIT1: [[META0]] = !{[[META1:![0-9]+]]}
 ; LIMIT1: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/runtime-check.ll b/llvm/test/Transforms/LoopVectorize/runtime-check.ll
index d7a747bfe6d66..b61c7d7140925 100644
--- a/llvm/test/Transforms/LoopVectorize/runtime-check.ll
+++ b/llvm/test/Transforms/LoopVectorize/runtime-check.ll
@@ -592,4 +592,4 @@ exit:
 !11 = !{i32 2, !"Debug Info Version", i32 3}
 !12 = distinct !{!12, !13, !14}
 !13 = !{!"llvm.loop.vectorize.width", i32 2}
-!14 = !{!"llvm.loop.vectorize.enable", i1 true}
+!14 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll b/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
index 1aaac9e6881ef..afd3ae3806fb1 100644
--- a/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
@@ -307,7 +307,7 @@ exit:
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 2}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !7, !3, !8, !5}
 !7 = !{!"llvm.loop.vectorize.width", i32 4}
 !8 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll b/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
index 4fab0c38dd0da..fe8410adb8bae 100644
--- a/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
+++ b/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
@@ -48,4 +48,4 @@ exit:
 
 !0 = !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll b/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
index 33a772285e423..5d65362df3cc2 100644
--- a/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
+++ b/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
@@ -245,4 +245,4 @@ end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll b/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
index 2c268664ef991..eb3ad35df5bc5 100644
--- a/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
+++ b/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
@@ -14,7 +14,7 @@ target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f3
 ; CHECK-LABEL: @foo
 ; CHECK: !llvm.loop !0
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.vectorize.enable"}
 define i32 @foo(i32 %a) {
 entry:
   br label %loop_cond
@@ -33,4 +33,4 @@ return:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll b/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
index cd68c81f94054..ce0da2efcab20 100644
--- a/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
+++ b/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
@@ -51,4 +51,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll b/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
index efc29b3c8a32d..358082968efe4 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
@@ -44,4 +44,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll b/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
index c61887e8e3386..76379c3935323 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
@@ -75,4 +75,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll b/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
index 3ad77bfd7787f..5467c0354168b 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
@@ -119,11 +119,11 @@ exit:
 declare void @use()
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
 ; CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]]}
-; CHECK: [[META4]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[META4]] = !{!"llvm.loop.vectorize.enable"}
 ;.
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll b/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
index 75f56a74986c4..83a3fd69b1e6c 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
@@ -256,4 +256,4 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll b/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
index 049d1baf5c615..0ab38d96bfe4c 100644
--- a/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
+++ b/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
@@ -78,4 +78,4 @@ for.cond.cleanup:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll b/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
index e27584bb421f1..15c4b10ec71b2 100644
--- a/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
+++ b/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
@@ -1163,6 +1163,6 @@ for.end:                                          ; preds = %for.body
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 1}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 0}
+!3 = !{!"llvm.loop.vectorize.disable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll b/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
index fe4f81a6f4749..a66528f1f12b4 100644
--- a/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
+++ b/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
@@ -187,7 +187,7 @@ attributes #1 = { nocallback nofree nosync nounwind willreturn memory(argmem: re
 !15 = distinct !{!15, !16, !17, !18}
 !16 = !{!"llvm.loop.mustprogress"}
 !17 = !{!"llvm.loop.parallel_accesses", !12}
-!18 = !{!"llvm.loop.vectorize.enable", i1 true}
+!18 = !{!"llvm.loop.vectorize.enable"}
 
 ;.
 ; CHECK: [[INT_TBAA0]] = !{[[META1:![0-9]+]], [[META1]], i64 0}
diff --git a/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll b/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
index 82ebdaa900fe1..dec130bd533d6 100644
--- a/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
+++ b/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
@@ -113,7 +113,7 @@ declare void @llvm.memcpy.p0.p0.i64(ptr nocapture writeonly, ptr nocapture reado
 
 !0 = !{!"clang version 4.0.0 (cfe/trunk 277751)"}
 !1 = distinct !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!4}
 !4 = distinct !{!4, !5, !"_ZNK7ComplexplERKS_: %agg.result"}
 !5 = distinct !{!5, !"_ZNK7ComplexplERKS_"}
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll b/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
index 8c5733e5babed..d334adc2ceb6e 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
@@ -52,5 +52,5 @@ for.end:                                          ; preds = %cond.end
 attributes #0 = { norecurse nounwind uwtable }
 
 !0 = distinct !{!0, !1, !{!"llvm.loop.parallel_accesses", !10}}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !10 = distinct !{}
diff --git a/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll b/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
index 25a09b15ca652..833c76276e458 100644
--- a/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
+++ b/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
@@ -23,7 +23,7 @@ define hidden void @f() {
 ; CHECK-NOT: DILocation
 ; CHECK: [[LID]] = distinct !{[[LID]], [[VE:![0-9]+]], [[VF:![0-9]+]]}
 ; CHECK-NOT: DILocation
-; CHECK: [[VE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[VE]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: [[VF]] = !{!"llvm.loop.vectorize.followup_all", [[FU:![0-9]+]]}
 ; CHECK: [[FU]] = distinct !{[[FU]], [[VD:![0-9]+]], [[UC:![0-9]+]]}
 ; CHECK: [[VD]] = !{!"llvm.loop.isvectorized"}
@@ -40,7 +40,7 @@ define hidden void @f() {
 !8 = !{}
 !9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !5, isOptimized: false, runtimeVersion: 0, emissionKind: NoDebug)
 !10 = !DILocation(line: 6, column: 1, scope: !4)
-!11 = !{!"llvm.loop.vectorize.enable", i1 true}
+!11 = !{!"llvm.loop.vectorize.enable"}
 !12 = !{!"llvm.loop.vectorize.followup_all", !13}
 !13 = distinct !{!13, !3, !10, !14, !15}
 !14 = !{!"llvm.loop.isvectorized"}
diff --git a/llvm/test/Verifier/llvm.loop.vectorize.ll b/llvm/test/Verifier/llvm.loop.vectorize.ll
new file mode 100644
index 0000000000000..6f3d354fc2bb1
--- /dev/null
+++ b/llvm/test/Verifier/llvm.loop.vectorize.ll
@@ -0,0 +1,41 @@
+; Test "llvm.loop.vectorize.enable" / "llvm.loop.vectorize.disable"
+; single-operand validation.
+
+; DEFINE: %{VERIFY} = llvm-as -disable-output %t 2>&1
+
+define void @test() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+!0 = distinct !{!0, !1}
+
+;      BAD: Expected one operand for llvm.loop.vectorize.enable metadata
+
+; Single-operand enable.
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable"}' >> %t
+; RUN: %{VERIFY}
+
+; Single-operand disable.
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.disable"}' >> %t
+; RUN: %{VERIFY}
+
+; Two-operand enable with boolean false (legacy form, now rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable", i1 0}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
+
+; Two-operand enable with boolean true (legacy form, now rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable", i1 1}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
+
+; Two-operand disable (rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.disable", i1 0}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index e4905423347a2..8dfe4d7f2b299 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -285,8 +285,9 @@ static T createIfNonNull(MLIRContext *ctx, const P &...args) {
 }
 
 FailureOr<LoopVectorizeAttr> LoopMetadataConversion::convertVectorizeAttr() {
-  FailureOr<BoolAttr> enable =
-      lookupBoolNode("llvm.loop.vectorize.enable", true);
+  FailureOr<BoolAttr> enable = lookupBooleanUnitNode(
+      "llvm.loop.vectorize.enable", "llvm.loop.vectorize.disable",
+      /*negated=*/true);
   FailureOr<BoolAttr> predicateEnable =
       lookupBoolNode("llvm.loop.vectorize.predicate.enable");
   FailureOr<BoolAttr> scalableEnable =
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
index f3ec18c9074e4..d9744601badbe 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
@@ -103,7 +103,12 @@ void LoopAnnotationConversion::convertFollowupNode(StringRef name,
 }
 
 void LoopAnnotationConversion::convertLoopOptions(LoopVectorizeAttr options) {
-  convertBoolNode("llvm.loop.vectorize.enable", options.getDisable(), true);
+  if (auto disable = options.getDisable()) {
+    if (disable.getValue())
+      addUnitNode("llvm.loop.vectorize.disable");
+    else
+      addUnitNode("llvm.loop.vectorize.enable");
+  }
   convertBoolNode("llvm.loop.vectorize.predicate.enable",
                   options.getPredicateEnable());
   convertBoolNode("llvm.loop.vectorize.scalable.enable",
diff --git a/mlir/test/Target/LLVMIR/Import/import-failure.ll b/mlir/test/Target/LLVMIR/Import/import-failure.ll
index 57c8438bfd362..1b3a666348eed 100644
--- a/mlir/test/Target/LLVMIR/Import/import-failure.ll
+++ b/mlir/test/Target/LLVMIR/Import/import-failure.ll
@@ -160,22 +160,6 @@ end:
 
 ; // -----
 
-; CHECK:      <unknown>
-; CHECK-SAME: warning: expected metadata node llvm.loop.vectorize.enable to hold a boolean value
-; CHECK:      <unknown>
-; CHECK-SAME: warning: unhandled metadata: !0 = distinct !{!0, !1}
-define void @unsupported_loop_annotation(i64 %n, ptr %A) {
-entry:
-  br label %end, !llvm.loop !0
-end:
-  ret void
-}
-
-!0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable"}
-
-; // -----
-
 ; CHECK:      <unknown>
 ; CHECK-SAME: warning: expected metadata node llvm.loop.vectorize.width to hold an i32 value
 ; CHECK:      <unknown>
diff --git a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
index c09fe92dc9cdc..e554f1b295b96 100644
--- a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
+++ b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
@@ -86,7 +86,7 @@ end:
 }
 
 !1 = distinct !{!1, !2, !3, !4, !5, !6, !7, !8}
-!2 = !{!"llvm.loop.vectorize.enable", i1 1}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.vectorize.predicate.enable", i1 1}
 !4 = !{!"llvm.loop.vectorize.scalable.enable", i1 0}
 !5 = !{!"llvm.loop.vectorize.width", i32 16}
diff --git a/mlir/test/Target/LLVMIR/loop-metadata.mlir b/mlir/test/Target/LLVMIR/loop-metadata.mlir
index 72f0cf06ed0fe..6c67182acb513 100644
--- a/mlir/test/Target/LLVMIR/loop-metadata.mlir
+++ b/mlir/test/Target/LLVMIR/loop-metadata.mlir
@@ -55,7 +55,7 @@ llvm.func @vectorizeOptions() {
 // CHECK-DAG: ![[NON_FORCED:[0-9]+]] = !{!"llvm.loop.disable_nonforced"}
 // CHECK-DAG: ![[FOLLOWUP:[0-9]+]] = distinct !{![[FOLLOWUP]], ![[NON_FORCED]]}
 // CHECK-DAG: ![[LOOP_NODE]] = distinct !{![[LOOP_NODE]], !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}}
-// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.width", i32 16}
diff --git a/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir b/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
index bfa57b85245ca..70b5b4b0af0a8 100644
--- a/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
@@ -12,7 +12,7 @@
 // CHECK-NEXT:   br label %omp_loop.header, !llvm.loop ![[LOOP_ATTR:.*]]
 // CHECK: ![[LOOP_ATTR]] = distinct !{![[LOOP_ATTR]], ![[LPAR:.*]], ![[LVEC:.*]]}
 // CHECK: ![[LPAR]] = !{!"llvm.loop.parallel_accesses", ![[PAR_ACC:.*]]}
-// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable"}
 
 llvm.func @test_parallel_do_simd() {
   %0 = llvm.mlir.constant(1 : i64) : i64
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
index 8da1fab1f48af..bb9aa246ba0fb 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
@@ -83,5 +83,5 @@ module {
     } {linear_var_types = [i32]}
     llvm.return
   }
-  // CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+  // CHECK: !{!"llvm.loop.vectorize.enable"}
 }
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir b/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
index 3e73fd22e69ca..0f2e20cc61aa7 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
@@ -95,4 +95,4 @@ llvm.func @_QPsimd_reduction(%arg0: !llvm.ptr {fir.bindc_name = "a", llvm.nocapt
 // CHECK: ![[ACCESS_GROUP]] = distinct !{}
 // CHECK: ![[LOOP]] = distinct !{![[LOOP]], ![[PARALLEL_ACCESS:.*]], ![[VECTORIZE:.*]]}
 // CHECK: ![[PARALLEL_ACCESS]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP]]}
-// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir b/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
index ab3c87cca56ea..a712b847e8557 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
@@ -93,4 +93,4 @@ llvm.func @_QPsimd_reduction(%arg0: !llvm.ptr {fir.bindc_name = "a", llvm.nocapt
 // CHECK: ![[ACCESS_GROUP]] = distinct !{}
 // CHECK: ![[LOOP]] = distinct !{![[LOOP]], ![[PARALLEL_ACCESS:.*]], ![[VECTORIZE:.*]]}
 // CHECK: ![[PARALLEL_ACCESS]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP]]}
-// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
diff --git a/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir b/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
index 69d5d225d0515..553d56034d2c1 100644
--- a/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
@@ -20,7 +20,7 @@
 
 // CHECK: ![[LOOP_ATTR]] = distinct !{![[LOOP_ATTR]], ![[LPAR:.*]], ![[LVEC:.*]]}
 // CHECK: ![[LPAR]] = !{!"llvm.loop.parallel_accesses", ![[PAR_ACC:.*]]}
-// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable"}
 
 omp.private {type = private} @_QFEi_private_i32 : i32
 llvm.func @test_teams_distribute_parallel_do_simd() {
diff --git a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
index beb300c23651f..3ae41b58f5d21 100644
--- a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
@@ -86,5 +86,5 @@ module {
     } {omp.composite}
     llvm.return
   }
-  // CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+  // CHECK: !{!"llvm.loop.vectorize.enable"}
 }
diff --git a/polly/lib/CodeGen/CodeGeneration.cpp b/polly/lib/CodeGen/CodeGeneration.cpp
index d9f4a7f78a322..9454108d29bf3 100644
--- a/polly/lib/CodeGen/CodeGeneration.cpp
+++ b/polly/lib/CodeGen/CodeGeneration.cpp
@@ -48,6 +48,22 @@ using namespace polly;
 #include "polly/Support/PollyDebug.h"
 #define DEBUG_TYPE "polly-codegen"
 
+// Append the single-operand "llvm.loop.vectorize.disable" node to \p L,
+// keeping any existing loop metadata intact.
+static void addVectorizeDisableMetadataToLoop(Loop *L) {
+  LLVMContext &Ctx = L->getHeader()->getContext();
+  SmallVector<Metadata *, 4> MDs(1);
+  if (MDNode *LoopID = L->getLoopID())
+    for (unsigned I = 1, E = LoopID->getNumOperands(); I < E; ++I)
+      MDs.push_back(LoopID->getOperand(I));
+  MDs.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
+  MDNode *NewLoopID = MDNode::get(Ctx, MDs);
+  // Set operand 0 to refer to the loop id itself.
+  NewLoopID->replaceOperandWith(0, NewLoopID);
+  L->setLoopID(NewLoopID);
+}
+
 static cl::opt<bool> Verify("polly-codegen-verify",
                             cl::desc("Verify the function generated by Polly"),
                             cl::Hidden, cl::cat(PollyCategory));
@@ -284,7 +300,7 @@ static bool generateCode(Scop &S, IslAstInfo &AI, LoopInfo &LI,
     if (!(CI && CI->isZero())) {
       for (Loop *L : LI.getLoopsInPreorder()) {
         if (S.contains(L))
-          addStringMetadataToLoop(L, "llvm.loop.vectorize.enable", 0);
+          addVectorizeDisableMetadataToLoop(L);
       }
     }
 
diff --git a/polly/lib/CodeGen/IRBuilder.cpp b/polly/lib/CodeGen/IRBuilder.cpp
index d326ffd8c6a2b..b461b72170c77 100644
--- a/polly/lib/CodeGen/IRBuilder.cpp
+++ b/polly/lib/CodeGen/IRBuilder.cpp
@@ -131,11 +131,10 @@ void ScopAnnotator::popLoop(bool IsParallel) {
 static void addVectorizeMetadata(LLVMContext &Ctx,
                                  SmallVector<Metadata *, 3> *Args,
                                  bool EnableLoopVectorizer) {
-  MDString *PropName = MDString::get(Ctx, "llvm.loop.vectorize.enable");
-  ConstantInt *Value =
-      ConstantInt::get(Type::getInt1Ty(Ctx), EnableLoopVectorizer);
-  ValueAsMetadata *PropValue = ValueAsMetadata::get(Value);
-  Args->push_back(MDNode::get(Ctx, {PropName, PropValue}));
+  MDString *PropName =
+      MDString::get(Ctx, EnableLoopVectorizer ? "llvm.loop.vectorize.enable"
+                                              : "llvm.loop.vectorize.disable");
+  Args->push_back(MDNode::get(Ctx, {PropName}));
 }
 
 void addParallelMetadata(LLVMContext &Ctx, SmallVector<Metadata *, 3> *Args,
diff --git a/polly/test/CodeGen/MemAccess/create_arrays.ll b/polly/test/CodeGen/MemAccess/create_arrays.ll
index 8443e0f7be327..29eaff8ad3fab 100644
--- a/polly/test/CodeGen/MemAccess/create_arrays.ll
+++ b/polly/test/CodeGen/MemAccess/create_arrays.ll
@@ -32,7 +32,7 @@
 ; CODEGEN:  store double {{%.*}}, ptr %scevgep34, align 8, !alias.scope !12, !noalias !13
 ;
 ; CODEGEN: !0 = distinct !{!0, !1}
-; CODEGEN: !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; CODEGEN: !1 = !{!"llvm.loop.vectorize.disable"}
 ; CODEGEN: !2 = distinct !{!2, !1}
 ; CODEGEN: !3 = distinct !{!3, !1}
 ; CODEGEN: !4 = !{!5}
diff --git a/polly/test/CodeGen/Metadata/basic_vec_annotate.ll b/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
index e059fea707321..7e0c1de322d16 100644
--- a/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
+++ b/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
@@ -13,9 +13,9 @@
 ; CHECK: polly.stmt.for.body:
 ; CHECK: br {{.*}} !llvm.loop [[POLLY_LOOP:![0-9]+]]
 ; CHECK: [[LOOP]] = distinct !{[[LOOP]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[POLLY_LOOP]] = distinct !{[[POLLY_LOOP]], {{.*}}}
-; CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32"
 target triple = "aarch64-unknown-linux-gnu"
diff --git a/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll b/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
index 8f3c4461f79aa..f232be9892781 100644
--- a/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
+++ b/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
@@ -4,7 +4,7 @@
 ; Verify vectorization is not disabled when RTC of Polly is false
 
 ; CHECK: attributes {{.*}} = { "polly-optimized" }
-; CHECK-NOT: {{.*}} = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK-NOT: {{.*}} = !{!"llvm.loop.vectorize.disable"}
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32"
 target triple = "aarch64-unknown-linux-android10000"
diff --git a/polly/test/CodeGen/annotated_alias_scopes.ll b/polly/test/CodeGen/annotated_alias_scopes.ll
index 7d2d9038270a9..1f3a07b5519be 100644
--- a/polly/test/CodeGen/annotated_alias_scopes.ll
+++ b/polly/test/CodeGen/annotated_alias_scopes.ll
@@ -11,7 +11,7 @@
 ; SCOPES:      store i32 %{{[._a-zA-Z0-9]*}}, ptr %[[AIdx]], align 4, !alias.scope !10, !noalias !11
 ;
 ; SCOPES: !0 = distinct !{!0, !1}
-; SCOPES: !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; SCOPES: !1 = !{!"llvm.loop.vectorize.disable"}
 ; SCOPES: !2 = !{!3}
 ; SCOPES: !3 = distinct !{!3, !4, !"polly.alias.scope.MemRef_B"}
 ; SCOPES: !4 = distinct !{!4, !"polly.alias.scope.domain"}
diff --git a/polly/test/CodeGen/scev-backedgetaken.ll b/polly/test/CodeGen/scev-backedgetaken.ll
index 09fcfe3e4a09c..df416f3d6f63e 100644
--- a/polly/test/CodeGen/scev-backedgetaken.ll
+++ b/polly/test/CodeGen/scev-backedgetaken.ll
@@ -44,7 +44,7 @@ for.cond.cleanup.loopexit:
 ; CHECK:         load i32, ptr %scevgep, align 4, !alias.scope !2, !noalias !5
 
 ; CHECK:       !0 = distinct !{!0, !1}
-; CHECK:       !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK:       !1 = !{!"llvm.loop.vectorize.disable"}
 ; CHECK:       !2 = !{!3}
 ; CHECK:       !3 = distinct !{!3, !4, !"polly.alias.scope.MemRef_d"}
 ; CHECK:       !4 = distinct !{!4, !"polly.alias.scope.domain"}
diff --git a/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll b/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
index 3705c3fd27ed9..977dcda9b2e9a 100644
--- a/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
+++ b/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
@@ -2,7 +2,7 @@
 ;
 ; Check that we disable the Loop Vectorizer.
 ;
-; CHECK: !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: !{!"llvm.loop.vectorize.disable"}
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-unknown"


From llvmlistbot at llvm.org  Tue Jul 21 03:19:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 03:19:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f47a2.3cce8983.3cf542.2dc4@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Srinivasa Ravi (Wolfram70)

<details>
<summary>Changes</summary>

This change adds the `store.async` op to the NVVM dialect to
perform asynchronous stores to global or shared-cluster address
spaces.

PTX Spec References:
1. [`st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-async)
2. [`multimem.st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-st-async)

---
Full diff: https://github.com/llvm/llvm-project/pull/210931.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td (+14) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+19) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+80) 
- (added) mlir/test/Target/LLVMIR/nvvm/store_async.mlir (+58) 
- (added) mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir (+63) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
index 42d196c5662d1..9d6a7f3bf1a6c 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
@@ -69,4 +69,18 @@ def SaturationModeAttr : EnumAttr<NVVM_Dialect, SaturationMode, "sat_mode"> {
  let assemblyFormat = "`<` $value `>`";
 }
 
+def AsyncStoreScopeNone : I32EnumCase<"NONE", 0, "none">;
+def AsyncStoreScopeSys : I32EnumCase<"SYS", 1, "sys">;
+def AsyncStoreScopeGpu : I32EnumCase<"GPU", 2, "gpu">;
+
+def AsyncStoreScope :
+  I32Enum<"AsyncStoreScope", "NVVM Asynchronous Store Scope",
+    [AsyncStoreScopeNone, AsyncStoreScopeSys, AsyncStoreScopeGpu]> {
+  let cppNamespace = "::mlir::NVVM";
+}
+
+def AsyncStoreScopeAttr : EnumAttr<NVVM_Dialect, AsyncStoreScope, 
+                                   "async_store_scope"> {
+  let assemblyFormat = "`<` $value `>`";
+}
 #endif // NVVMIR_ENUMS
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..d905a5b212450 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
+  let summary = "Asynchronous Store Op";
+  let arguments = (ins AnyTypeOf<[LLVM_PointerGlobal, 
+                                  LLVM_PointerSharedCluster]>:$addr,
+                       AnyTypeOf<[I8, I16, I32, I64, I128]>:$value, 
+                       Optional<LLVM_PointerSharedCluster>:$mbarrier,
+                       DefaultValuedAttr<AsyncStoreScopeAttr, 
+                                         "AsyncStoreScope::NONE">:$scope,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_multimem,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_mmio);
+  let results = (outs );
+  let assemblyFormat = "$addr `,` $value (`,` `mbarrier` `=` $mbarrier^)? attr-dict `:` type($addr) `,` type($value) (`,` type($mbarrier)^)?";
+  let hasVerifier = 1;
+}
+
 def NVVM_Exit : NVVM_Op<"exit"> {
   let summary = "Exit Op";
   let description = [{
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index b29cba96a410f..603d28957176f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
+
+    if (scope != AsyncStoreScope::NONE)
+      return emitOpError("scope is not supported for async store to shared "
+                         "cluster address space");
+  }
+
+  return success();
+}
+
 LogicalResult PMEventOp::verify() {
   auto eventId = getEventId();
   auto maskedEventId = getMaskedEventId();
@@ -3662,6 +3706,42 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
+
+  switch (addrSpace) {
+  case NVVMMemorySpace::Global:
+    if (scope == AsyncStoreScope::SYS) {
+      if (thisOp.getIsMmio())
+        return {llvm::Intrinsic::nvvm_st_async_mmio_sys, {addr, value}};
+      else
+        return {llvm::Intrinsic::nvvm_st_async_sys, {addr, value, isMultimem}};
+    } else if (scope == AsyncStoreScope::GPU) {
+      return {llvm::Intrinsic::nvvm_st_async_gpu, {addr, value, isMultimem}};
+    }
+    llvm_unreachable("unsupported async store scope for global address space");
+  case NVVMMemorySpace::SharedCluster:
+    return {llvm::Intrinsic::nvvm_st_async, {addr, value, mbarrier}};
+  default:
+    llvm_unreachable("unsupported address space");
+  }
+
+  return {llvm::Intrinsic::not_intrinsic, {}};
+}
+
 mlir::NVVM::IDArgPair
 PMEventOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
                                  llvm::IRBuilderBase &builder) {
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
new file mode 100644
index 0000000000000..8e59f48c987ac
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// CHECK-LABEL: define void @st_async_global_sys
+llvm.func @st_async_global_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_gpu
+llvm.func @st_async_global_gpu(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_multimem
+llvm.func @st_async_global_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_multimem = true} : !llvm.ptr<1>, i32
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_mmio
+llvm.func @st_async_global_mmio(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.mmio.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}})
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_types
+llvm.func @st_async_global_types(%addr: !llvm.ptr<1>, %v8: i8, %v16: i16, %v64: i64) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i8(ptr addrspace(1) %{{.*}}, i8 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v8 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i8
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i16(ptr addrspace(1) %{{.*}}, i16 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v16 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i16
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i64(ptr addrspace(1) %{{.*}}, i64 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v64 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i64
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster
+llvm.func @st_async_shared_cluster(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i32(ptr addrspace(7) %{{.*}}, i32 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster_types
+llvm.func @st_async_shared_cluster_types(%addr: !llvm.ptr<7>, %v64: i64, %v128: i128, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i64(ptr addrspace(7) %{{.*}}, i64 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v64, mbarrier = %mbar : !llvm.ptr<7>, i64, !llvm.ptr<7>
+  // CHECK: call void @llvm.nvvm.st.async.i128(ptr addrspace(7) %{{.*}}, i128 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v128, mbarrier = %mbar : !llvm.ptr<7>, i128, !llvm.ptr<7>
+  llvm.return
+}
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
new file mode 100644
index 0000000000000..1fe4bbb85235f
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
@@ -0,0 +1,63 @@
+// RUN: mlir-translate -verify-diagnostics -split-input-file -mlir-to-llvmir %s
+
+llvm.func @st_async_global_mbarrier(%addr: !llvm.ptr<1>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{mbarrier is not supported for global address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_i128(%addr: !llvm.ptr<1>, %value: i128) {
+  // expected-error @below {{only 8, 16, 32, and 64 bit values are supported for global address space}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i128
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_no_scope(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{scope must be set for async store to global address space}}
+  nvvm.store.async %addr, %value : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_non_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{mmio is only supported for SYS scope}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{multimem is not supported for mmio}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_i8(%addr: !llvm.ptr<7>, %value: i8, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{only 32, 64, and 128 bit values are supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i8, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_multimem(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{multimem and mmio are not supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {is_multimem = true} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_scope(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{scope is not supported for async store to shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 03:19:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 03:19:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f47a2.9edb5c9d.280be3.ce0b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Srinivasa Ravi (Wolfram70)

<details>
<summary>Changes</summary>

This change adds the `store.async` op to the NVVM dialect to
perform asynchronous stores to global or shared-cluster address
spaces.

PTX Spec References:
1. [`st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-st-async)
2. [`multimem.st.async`](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-multimem-st-async)

---
Full diff: https://github.com/llvm/llvm-project/pull/210931.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td (+14) 
- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+19) 
- (modified) mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp (+80) 
- (added) mlir/test/Target/LLVMIR/nvvm/store_async.mlir (+58) 
- (added) mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir (+63) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
index 42d196c5662d1..9d6a7f3bf1a6c 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMEnums.td
@@ -69,4 +69,18 @@ def SaturationModeAttr : EnumAttr<NVVM_Dialect, SaturationMode, "sat_mode"> {
  let assemblyFormat = "`<` $value `>`";
 }
 
+def AsyncStoreScopeNone : I32EnumCase<"NONE", 0, "none">;
+def AsyncStoreScopeSys : I32EnumCase<"SYS", 1, "sys">;
+def AsyncStoreScopeGpu : I32EnumCase<"GPU", 2, "gpu">;
+
+def AsyncStoreScope :
+  I32Enum<"AsyncStoreScope", "NVVM Asynchronous Store Scope",
+    [AsyncStoreScopeNone, AsyncStoreScopeSys, AsyncStoreScopeGpu]> {
+  let cppNamespace = "::mlir::NVVM";
+}
+
+def AsyncStoreScopeAttr : EnumAttr<NVVM_Dialect, AsyncStoreScope, 
+                                   "async_store_scope"> {
+  let assemblyFormat = "`<` $value `>`";
+}
 #endif // NVVMIR_ENUMS
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..d905a5b212450 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
+  let summary = "Asynchronous Store Op";
+  let arguments = (ins AnyTypeOf<[LLVM_PointerGlobal, 
+                                  LLVM_PointerSharedCluster]>:$addr,
+                       AnyTypeOf<[I8, I16, I32, I64, I128]>:$value, 
+                       Optional<LLVM_PointerSharedCluster>:$mbarrier,
+                       DefaultValuedAttr<AsyncStoreScopeAttr, 
+                                         "AsyncStoreScope::NONE">:$scope,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_multimem,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_mmio);
+  let results = (outs );
+  let assemblyFormat = "$addr `,` $value (`,` `mbarrier` `=` $mbarrier^)? attr-dict `:` type($addr) `,` type($value) (`,` type($mbarrier)^)?";
+  let hasVerifier = 1;
+}
+
 def NVVM_Exit : NVVM_Op<"exit"> {
   let summary = "Exit Op";
   let description = [{
diff --git a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
index b29cba96a410f..603d28957176f 100644
--- a/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
+++ b/mlir/lib/Dialect/LLVMIR/IR/NVVMDialect.cpp
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
+
+    if (scope != AsyncStoreScope::NONE)
+      return emitOpError("scope is not supported for async store to shared "
+                         "cluster address space");
+  }
+
+  return success();
+}
+
 LogicalResult PMEventOp::verify() {
   auto eventId = getEventId();
   auto maskedEventId = getMaskedEventId();
@@ -3662,6 +3706,42 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
+
+  switch (addrSpace) {
+  case NVVMMemorySpace::Global:
+    if (scope == AsyncStoreScope::SYS) {
+      if (thisOp.getIsMmio())
+        return {llvm::Intrinsic::nvvm_st_async_mmio_sys, {addr, value}};
+      else
+        return {llvm::Intrinsic::nvvm_st_async_sys, {addr, value, isMultimem}};
+    } else if (scope == AsyncStoreScope::GPU) {
+      return {llvm::Intrinsic::nvvm_st_async_gpu, {addr, value, isMultimem}};
+    }
+    llvm_unreachable("unsupported async store scope for global address space");
+  case NVVMMemorySpace::SharedCluster:
+    return {llvm::Intrinsic::nvvm_st_async, {addr, value, mbarrier}};
+  default:
+    llvm_unreachable("unsupported address space");
+  }
+
+  return {llvm::Intrinsic::not_intrinsic, {}};
+}
+
 mlir::NVVM::IDArgPair
 PMEventOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
                                  llvm::IRBuilderBase &builder) {
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
new file mode 100644
index 0000000000000..8e59f48c987ac
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async.mlir
@@ -0,0 +1,58 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// CHECK-LABEL: define void @st_async_global_sys
+llvm.func @st_async_global_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_gpu
+llvm.func @st_async_global_gpu(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_multimem
+llvm.func @st_async_global_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_multimem = true} : !llvm.ptr<1>, i32
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}}, /* isMultimem= */ i1 true)
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_mmio
+llvm.func @st_async_global_mmio(%addr: !llvm.ptr<1>, %value: i32) {
+  // CHECK: call void @llvm.nvvm.st.async.mmio.sys.i32(ptr addrspace(1) %{{.*}}, i32 %{{.*}})
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_global_types
+llvm.func @st_async_global_types(%addr: !llvm.ptr<1>, %v8: i8, %v16: i16, %v64: i64) {
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i8(ptr addrspace(1) %{{.*}}, i8 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v8 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i8
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i16(ptr addrspace(1) %{{.*}}, i16 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v16 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i16
+  // CHECK: call void @llvm.nvvm.st.async.gpu.i64(ptr addrspace(1) %{{.*}}, i64 %{{.*}}, /* isMultimem= */ i1 false)
+  nvvm.store.async %addr, %v64 {scope = #nvvm.async_store_scope<gpu>} : !llvm.ptr<1>, i64
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster
+llvm.func @st_async_shared_cluster(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i32(ptr addrspace(7) %{{.*}}, i32 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// CHECK-LABEL: define void @st_async_shared_cluster_types
+llvm.func @st_async_shared_cluster_types(%addr: !llvm.ptr<7>, %v64: i64, %v128: i128, %mbar: !llvm.ptr<7>) {
+  // CHECK: call void @llvm.nvvm.st.async.i64(ptr addrspace(7) %{{.*}}, i64 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v64, mbarrier = %mbar : !llvm.ptr<7>, i64, !llvm.ptr<7>
+  // CHECK: call void @llvm.nvvm.st.async.i128(ptr addrspace(7) %{{.*}}, i128 %{{.*}}, ptr addrspace(7) %{{.*}})
+  nvvm.store.async %addr, %v128, mbarrier = %mbar : !llvm.ptr<7>, i128, !llvm.ptr<7>
+  llvm.return
+}
diff --git a/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
new file mode 100644
index 0000000000000..1fe4bbb85235f
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/nvvm/store_async_invalid.mlir
@@ -0,0 +1,63 @@
+// RUN: mlir-translate -verify-diagnostics -split-input-file -mlir-to-llvmir %s
+
+llvm.func @st_async_global_mbarrier(%addr: !llvm.ptr<1>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{mbarrier is not supported for global address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_i128(%addr: !llvm.ptr<1>, %value: i128) {
+  // expected-error @below {{only 8, 16, 32, and 64 bit values are supported for global address space}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<1>, i128
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_no_scope(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{scope must be set for async store to global address space}}
+  nvvm.store.async %addr, %value : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_non_sys(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{mmio is only supported for SYS scope}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<gpu>, is_mmio = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_global_mmio_multimem(%addr: !llvm.ptr<1>, %value: i32) {
+  // expected-error @below {{multimem is not supported for mmio}}
+  nvvm.store.async %addr, %value {scope = #nvvm.async_store_scope<sys>, is_mmio = true, is_multimem = true} : !llvm.ptr<1>, i32
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_i8(%addr: !llvm.ptr<7>, %value: i8, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{only 32, 64, and 128 bit values are supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar : !llvm.ptr<7>, i8, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_multimem(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{multimem and mmio are not supported for shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {is_multimem = true} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}
+
+// -----
+
+llvm.func @st_async_shared_cluster_scope(%addr: !llvm.ptr<7>, %value: i32, %mbar: !llvm.ptr<7>) {
+  // expected-error @below {{scope is not supported for async store to shared cluster address space}}
+  nvvm.store.async %addr, %value, mbarrier = %mbar {scope = #nvvm.async_store_scope<sys>} : !llvm.ptr<7>, i32, !llvm.ptr<7>
+  llvm.return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 03:19:52 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 03:19:52 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210932@github.com>
Message-ID: <6a5f47c8.3cce8983.3cf542.4842@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-clang-codegen

@llvm/pr-subscribers-llvm-ir

Author: Madhur Amilkanthwar (madhur13490)

<details>
<summary>Changes</summary>

This patch implements Phase 2 of the RFC "Enforce Single-Operand Format for All .enable Metadata Nodes". The two-operand boolean form !{!"llvm.loop.vectorize.enable", i1 0/1} is replaced by a single-operand enable/disable pair:

  !{!"llvm.loop.vectorize.enable"}    ; force vectorization
  !{!"llvm.loop.vectorize.disable"}   ; suppress vectorization

The Verifier rejects the two-operand form, AutoUpgrade rewrites old bitcode (including the legacy llvm.vectorizer.enable tag), and the readers and producers across LLVM, Clang, MLIR and Polly are updated.

Please refer to RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571/

---

Patch is 161.08 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210932.diff


213 Files Affected:

- (modified) clang/lib/CodeGen/CGLoopInfo.cpp (+6-8) 
- (modified) clang/test/CodeGenCXX/pragma-followup_inner.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-followup_outer.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-pr27643.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-predicate.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-safety.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop.cpp (+1-1) 
- (modified) clang/test/OpenMP/for_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/for_simd_scan_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_safelen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simd.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simd_aligned.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simdlen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simdlen_safelen.cpp (+1-1) 
- (modified) clang/test/OpenMP/master_taskloop_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/parallel_for_simd_codegen.cpp (+5-5) 
- (modified) clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/simd_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/simd_metadata.c (+1-1) 
- (modified) clang/test/OpenMP/target_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp (+2-2) 
- (modified) clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp (+2-2) 
- (modified) clang/test/OpenMP/taskloop_simd_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp (+2-2) 
- (modified) clang/test/OpenMP/teams_distribute_simd_codegen.cpp (+1-1) 
- (modified) clang/test/PCH/delayed-template-with-pragma.cpp (+1-1) 
- (modified) clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp (+5-5) 
- (modified) flang/test/Integration/ivdep.f90 (+1-1) 
- (modified) flang/test/Integration/vector-always.f90 (+2-2) 
- (modified) llvm/docs/LangRef.md (+4-6) 
- (modified) llvm/docs/TransformMetadata.rst (+3-3) 
- (modified) llvm/include/llvm/IR/AutoUpgrade.h (+6-1) 
- (modified) llvm/lib/Bitcode/Reader/MetadataLoader.cpp (+3) 
- (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+2-4) 
- (modified) llvm/lib/IR/AutoUpgrade.cpp (+59-3) 
- (modified) llvm/lib/IR/Verifier.cpp (+8) 
- (modified) llvm/lib/Transforms/Utils/LoopConstrainer.cpp (+1-2) 
- (modified) llvm/lib/Transforms/Utils/LoopUtils.cpp (+3-3) 
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp (+8) 
- (modified) llvm/test/Bitcode/upgrade-loop-metadata.ll (+1-1) 
- (added) llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll (+37) 
- (added) llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc () 
- (modified) llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/correct-loop-info.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/loop-guarded-bounds.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/pre_post_loops.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/range_intersect_miscompile.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll (+1-1) 
- (modified) llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll (+2-2) 
- (modified) llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/optnone.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll (+3-3) 
- (modified) llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll (+3-3) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/cost-model.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll (+17-17) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll (+6-6) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll (+13-13) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll (+14-14) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/alias-mask.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_array_bounds.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_switch.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/optsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/reduction-inloop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/remove_metadata.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/scalable-inductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll (+1-1) 
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll (+1-1) 
- (modified) llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll (+2-2) 
- (modified) llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll (+1-1) 
- (modified) llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll (+1-1) 
- (modified) llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll (+1-1) 
- (modified) llvm/test/Verifier/llvm.loop-cu-strip-followup.ll (+2-2) 
- (added) llvm/test/Verifier/llvm.loop.vectorize.ll (+41) 
- (modified) mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp (+3-2) 
- (modified) mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp (+6-1) 
- (modified) mlir/test/Target/LLVMIR/Import/import-failure.ll (-16) 
- (modified) mlir/test/Target/LLVMIR/Import/metadata-loop.ll (+1-1) 
- (modified) mlir/test/Target/LLVMIR/loop-metadata.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir (+1-1) 
- (modified) polly/lib/CodeGen/CodeGeneration.cpp (+17-1) 
- (modified) polly/lib/CodeGen/IRBuilder.cpp (+4-5) 
- (modified) polly/test/CodeGen/MemAccess/create_arrays.ll (+1-1) 
- (modified) polly/test/CodeGen/Metadata/basic_vec_annotate.ll (+2-2) 
- (modified) polly/test/CodeGen/Metadata/fallback_vec_annotate.ll (+1-1) 
- (modified) polly/test/CodeGen/annotated_alias_scopes.ll (+1-1) 
- (modified) polly/test/CodeGen/scev-backedgetaken.ll (+1-1) 
- (modified) polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll (+1-1) 


``````````diff
diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 6b0b1b55213e1..b46ae34e58a44 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -209,10 +209,8 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createUnrollAndJamMetadata(Attrs, LoopProperties, HasUserTransforms);
@@ -279,10 +277,10 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
       (Attrs.VectorizeScalable == LoopAttributes::Disable &&
        Attrs.VectorizeWidth != 1)) {
     VectorizeEnabled = Attrs.VectorizeEnable != LoopAttributes::Disable;
-    Args.push_back(
-        MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                          ConstantAsMetadata::get(ConstantInt::get(
-                              llvm::Type::getInt1Ty(Ctx), VectorizeEnabled))}));
+    Args.push_back(MDNode::get(
+        Ctx, {MDString::get(Ctx, VectorizeEnabled
+                                     ? "llvm.loop.vectorize.enable"
+                                     : "llvm.loop.vectorize.disable")}));
   }
 
   // Apply all loop properties to the vectorized loop.
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..20615d9744c6e 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -25,7 +25,7 @@ extern "C" void followup_inner(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[OUTERLOOP_9]] = distinct !{![[OUTERLOOP_9]], [[MP:![0-9]+]], ![[UNROLLANDJAM_COUNT_10:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPINNER_11:[0-9]+]]}
 // CHECK-DAG: ![[UNROLLANDJAM_COUNT_10]] = !{!"llvm.loop.unroll_and_jam.count", i32 4}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..70ce57c6fb6a7 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_outer(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[VECTORIZE_FOLLOWUP_9:[0-9]+]] = !{!"llvm.loop.vectorize.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[ISVECTORIZED_11:[0-9]+]], ![[UNROLLANDJAM_12:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPOUTER_13:[0-9]+]]}
 // CHECK-DAG: ![[ISVECTORIZED_11:[0-9]+]] = !{!"llvm.loop.isvectorized"}
diff --git a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
index 37e29776264b6..afab64f4c6867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
@@ -43,7 +43,7 @@ void loop4(int *List, int Length) {
 // CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE:.*]]}
 // CHECK: ![[VEC_WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
 // CHECK: ![[FIXED_WIDTH]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE]]}
 // CHECK: ![[VEC_WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
index 8a25ed8de6239..421110fd9f867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
@@ -103,7 +103,7 @@ void test9(int *List, int Length) {
 
 // CHECK:      ![[LOOP0]] = distinct !{![[LOOP0]], [[MP:![0-9]+]], [[GEN3:![0-9]+]]}
 // CHECK:      [[MP]] = !{!"llvm.loop.mustprogress"}
-// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-NEXT: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP]], [[GEN6:![0-9]+]], [[GEN3]]}
 // CHECK-NEXT: [[GEN6]] = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-safety.cpp b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
index db1b6769e0514..e2a7a124890a6 100644
--- a/clang/test/CodeGenCXX/pragma-loop-safety.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
@@ -51,7 +51,7 @@ void interleave_test(int *List, int Length) {
 // CHECK: ![[PARALLEL_ACCESSES_7]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_2]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[INTERLEAVE_1]] = !{!"llvm.loop.interleave.count", i32 1}
-// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: ![[ACCESS_GROUP_8]] = distinct !{}
 // CHECK: ![[LOOP2_HINTS]] = distinct !{![[LOOP2_HINTS]], [[MP]], ![[PARALLEL_ACCESSES_11:[0-9]+]], ![[UNROLL_DISABLE]], ![[WIDTH_1:[0-9]+]], ![[INTENABLE_1]]}
 // CHECK: ![[PARALLEL_ACCESSES_11]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_8]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..d4a9faaf2be6b 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -256,7 +256,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[INTERLEAVE_10:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 10}
 // CHECK-DAG: ![[INTERLEAVE_16:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 16}
 
-// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: ![[FIXED_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: ![[SCALABLE_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 // CHECK-DAG: ![[WIDTH_1:[0-9]+]] = !{!"llvm.loop.vectorize.width", i32 1}
diff --git a/clang/test/OpenMP/for_simd_codegen.cpp b/clang/test/OpenMP/for_simd_codegen.cpp
index 1ac95e8943640..ae6161d03978d 100644
--- a/clang/test/OpenMP/for_simd_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_codegen.cpp
@@ -802,10 +802,10 @@ void parallel_simd(float *a) {
 }
 // TERM_DEBUG: !{{[0-9]+}} = !DILocation(line: [[@LINE-11]],
 // TERM_DEBUG-NOT: line: 0,
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // CHECK-DAG: ![[SIMD_LOOP]] = distinct !{![[SIMD_LOOP]], {{.*}}![[VECT_LOOP:[^,]+]]}
-// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // OMP50-DAG: ![[NOSIMD_LOOP]] = distinct !{![[NOSIMD_LOOP]], {{.*}}![[NOVECT_LOOP:[^,]+]]}
-// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.disable"}
 #endif // HEADER
diff --git a/clang/test/OpenMP/for_simd_scan_codegen.cpp b/clang/test/OpenMP/for_simd_scan_codegen.cpp
index 829f2656042fb..9ab62f8a682b2 100644
--- a/clang/test/OpenMP/for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_scan_codegen.cpp
@@ -300,5 +300,5 @@ void baz(int n) {
 }
 
 #endif
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
diff --git a/clang/test/OpenMP/irbuilder_safelen.cpp b/clang/test/OpenMP/irbuilder_safelen.cpp
index 6a4266f9646a7..3090eafa6a508 100644
--- a/clang/test/OpenMP/irbuilder_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
index 210978fc6069c..215b37e4bbb5f 100644
--- a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simd.cpp b/clang/test/OpenMP/irbuilder_simd.cpp
index 9c36e4ab97182..03f92d8117d45 100644
--- a/clang/test/OpenMP/irbuilder_simd.cpp
+++ b/clang/test/OpenMP/irbuilder_simd.cpp
@@ -65,7 +65,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: ![[META3:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META4]]  = distinct !{![[META4]], ![[META5:[0-9]+]], ![[META6:[0-9]+]]}
 // CHECK-NEXT: ![[META5]]  = !{!"llvm.loop.parallel_accesses", ![[META3]]}
-// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable"}
 // CHECK-NEXT: ![[META7:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META8]]  = distinct !{![[META8]], ![[META9:[0-9]+]], ![[META6]]}
 // CHECK-NEXT: ![[META9]]  = !{!"llvm.loop.parallel_accesses", ![[META7]]}
diff --git a/clang/test/OpenMP/irbuilder_simd_aligned.cpp b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
index b5ce43c8ea6a9..bb121036b4f0c 100644
--- a/clang/test/OpenMP/irbuilder_simd_aligned.cpp
+++ b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
@@ -293,7 +293,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP4]] = distinct !{}
 // CHECK: [[LOOP5]] = distinct !{[[LOOP5]], [[META6:![0-9]+]], [[META7:![0-9]+]]}
 // CHECK: [[META6]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP4]]}
-// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[ACC_GRP8]] = distinct !{}
 // CHECK: [[LOOP9]] = distinct !{[[LOOP9]], [[META10:![0-9]+]], [[META7]]}
 // CHECK: [[META10]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP8]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen.cpp b/clang/test/OpenMP/irbuilder_simdlen.cpp
index f6c26923851d0..9a7e8d5332ae0 100644
--- a/clang/test/OpenMP/irbuilder_simdlen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
index e46d5dde841b3..bb35f9341ded4 100644
--- a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 2}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
index d78d193e2edd1..f1eae642b613c 100644
--- a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
@@ -238,10 +238,10 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// OMP50-DAG: !{!"llvm.loop.vectorize.disable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/parallel_for_simd_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
index 40af7069bbf7d..cc8cab706975a 100644
--- a/clang/test/OpenMP/parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
@@ -798,16 +798,16 @@ for (int i = 0; i < 10; ++i);
 // OMP50: omp.inner.for.body:
 // OMP50: %{{[0-9]+}} = atomicrmw add ptr %{{[0-9]+}}, i32 1 monotonic, align 4, !llvm.access.group !{{[0-9]+}}
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP45-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP45-DAG: ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP50-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP50-DAG  ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
 // OMP50-DAG: ![[NOVECT]] = distinct !{![[NOVECT]], ![[NOVM:.+]]}
-// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.disable"}
 
 // TERM_DEBUG-LABEL: bar
 int bar() { extern void mayThrow(); mayThrow(); return 0; };
diff --git a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
index cac997753d480..e3810c27163a4 100644
--- a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
@@ -299,7 +299,7 @@ void baz(int n) {
   }
 }
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
 #endif
 
diff --git a/clang/test/OpenMP/simd_codegen.cpp b/clang/test/OpenMP/simd_codegen.cpp
index b96e4213e8e0e..ee8d709ba47fe 100644
--- a/clang/test/OpenMP/simd_codegen.cpp
+++ b/clang/test/OpenMP/simd_codegen.cpp
@@ -861,7 +861,7 @@ S8 s8(0);
 
 // TERM_DEBUG-NOT: line: 0,
 // TERM_DEBUG: distinct !DISubprogram(linkageName: "_GLOBAL__sub_I_simd_codegen.cpp",
-// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.disable"}
 // OMP5-DAG: ![[DISABLE_VECT]] = distinct !{{.*}}![[NOVECT]]{{[,}]}}
 #endif // HEADER
 
diff --git a/clang/test/OpenMP/simd_metadata.c b/clang/test/OpenMP/simd_metadata.c
index 83bcbf6deefe2..6fc4c401135f2 100644
--- a/clang/test/OpenMP/simd_metadata.c
+++ b/clang/test/OpenMP/simd_metadata.c
@@ -109,7 +109,7 @@ void h3(float *c, float *a, float *b, int size)
 // Metadata for h1:
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_16:![0-9]+]], [[LOOP_VEC_ENABLE:![0-9]+]]}
 // CHECK: [[LOOP_WIDTH_16]] = !{!"llvm.loop.vectorize.width", i32 16}
-// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_8:![0-9]+]], [[LOOP_VEC_ENABLE]]}
 // CHECK: [[LOOP_WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 // CHECK: ![[ACCESS_GROUP_7]] = distinct !{}
diff --git a/clang/test/OpenMP/target_simd_codegen.cpp b/clang/test/OpenMP/target_simd_codegen.cpp
index 0c2dde23f6c46..59284c201bb30 100644
--- a/clang/test/OpenMP/target_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_simd_codegen.cpp
@@ -702,9 +702,9 @@ int bar(int n){
 // Store captures in the context.
 // CHECK-DAG:   [[REF_B:%.+]] = load ptr, ptr [[LOCAL_B]],
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP51: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP51: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// TOMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// OMP51: !{!"llvm.loop.vectorize.disable"}
+// TOMP51: !{!"llvm.loop.vectorize.disable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
index 42e8377bd22dd..fd2d0d0c18737 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
@@ -455,7 +455,7 @@ int bar(int a){
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2STILi100EE3fooEv", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2SCC1Ev", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
index b63f0b6caeee2..c169d4a556bf0 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
@@ -82,6 +82,6 @@ int nested(int a){
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T1L]], i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T2L]], i32 0, i32 {{[0-9]+}}}
 
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// TCHECK-DAG: ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210932

From llvmlistbot at llvm.org  Tue Jul 21 03:19:52 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 03:19:52 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210932@github.com>
Message-ID: <6a5f47c8.5a0412d2.3daf32.3ae4@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-llvm-transforms

Author: Madhur Amilkanthwar (madhur13490)

<details>
<summary>Changes</summary>

This patch implements Phase 2 of the RFC "Enforce Single-Operand Format for All .enable Metadata Nodes". The two-operand boolean form !{!"llvm.loop.vectorize.enable", i1 0/1} is replaced by a single-operand enable/disable pair:

  !{!"llvm.loop.vectorize.enable"}    ; force vectorization
  !{!"llvm.loop.vectorize.disable"}   ; suppress vectorization

The Verifier rejects the two-operand form, AutoUpgrade rewrites old bitcode (including the legacy llvm.vectorizer.enable tag), and the readers and producers across LLVM, Clang, MLIR and Polly are updated.

Please refer to RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571/

---

Patch is 161.08 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210932.diff


213 Files Affected:

- (modified) clang/lib/CodeGen/CGLoopInfo.cpp (+6-8) 
- (modified) clang/test/CodeGenCXX/pragma-followup_inner.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-followup_outer.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-pr27643.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-predicate.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop-safety.cpp (+1-1) 
- (modified) clang/test/CodeGenCXX/pragma-loop.cpp (+1-1) 
- (modified) clang/test/OpenMP/for_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/for_simd_scan_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_safelen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simd.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simd_aligned.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simdlen.cpp (+1-1) 
- (modified) clang/test/OpenMP/irbuilder_simdlen_safelen.cpp (+1-1) 
- (modified) clang/test/OpenMP/master_taskloop_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/parallel_for_simd_codegen.cpp (+5-5) 
- (modified) clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/simd_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/simd_metadata.c (+1-1) 
- (modified) clang/test/OpenMP/target_simd_codegen.cpp (+4-4) 
- (modified) clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp (+2-2) 
- (modified) clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp (+2-2) 
- (modified) clang/test/OpenMP/taskloop_simd_codegen.cpp (+1-1) 
- (modified) clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp (+2-2) 
- (modified) clang/test/OpenMP/teams_distribute_simd_codegen.cpp (+1-1) 
- (modified) clang/test/PCH/delayed-template-with-pragma.cpp (+1-1) 
- (modified) clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp (+5-5) 
- (modified) flang/test/Integration/ivdep.f90 (+1-1) 
- (modified) flang/test/Integration/vector-always.f90 (+2-2) 
- (modified) llvm/docs/LangRef.md (+4-6) 
- (modified) llvm/docs/TransformMetadata.rst (+3-3) 
- (modified) llvm/include/llvm/IR/AutoUpgrade.h (+6-1) 
- (modified) llvm/lib/Bitcode/Reader/MetadataLoader.cpp (+3) 
- (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+2-4) 
- (modified) llvm/lib/IR/AutoUpgrade.cpp (+59-3) 
- (modified) llvm/lib/IR/Verifier.cpp (+8) 
- (modified) llvm/lib/Transforms/Utils/LoopConstrainer.cpp (+1-2) 
- (modified) llvm/lib/Transforms/Utils/LoopUtils.cpp (+3-3) 
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp (+8) 
- (modified) llvm/test/Bitcode/upgrade-loop-metadata.ll (+1-1) 
- (added) llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll (+37) 
- (added) llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc () 
- (modified) llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/correct-loop-info.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/loop-guarded-bounds.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/pre_post_loops.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/range_intersect_miscompile.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll (+1-1) 
- (modified) llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll (+1-1) 
- (modified) llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll (+2-2) 
- (modified) llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/optnone.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll (+3-3) 
- (modified) llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll (+3-3) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/cost-model.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll (+17-17) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll (+6-6) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll (+13-13) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll (+14-14) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/alias-mask.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_array_bounds.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_switch.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/optsize.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/reduction-inloop.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/remove_metadata.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/runtime-check.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/scalable-inductions.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll (+1-1) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll (+2-2) 
- (modified) llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll (+1-1) 
- (modified) llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll (+1-1) 
- (modified) llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll (+2-2) 
- (modified) llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll (+1-1) 
- (modified) llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll (+1-1) 
- (modified) llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll (+1-1) 
- (modified) llvm/test/Verifier/llvm.loop-cu-strip-followup.ll (+2-2) 
- (added) llvm/test/Verifier/llvm.loop.vectorize.ll (+41) 
- (modified) mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp (+3-2) 
- (modified) mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp (+6-1) 
- (modified) mlir/test/Target/LLVMIR/Import/import-failure.ll (-16) 
- (modified) mlir/test/Target/LLVMIR/Import/metadata-loop.ll (+1-1) 
- (modified) mlir/test/Target/LLVMIR/loop-metadata.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir (+1-1) 
- (modified) mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir (+1-1) 
- (modified) polly/lib/CodeGen/CodeGeneration.cpp (+17-1) 
- (modified) polly/lib/CodeGen/IRBuilder.cpp (+4-5) 
- (modified) polly/test/CodeGen/MemAccess/create_arrays.ll (+1-1) 
- (modified) polly/test/CodeGen/Metadata/basic_vec_annotate.ll (+2-2) 
- (modified) polly/test/CodeGen/Metadata/fallback_vec_annotate.ll (+1-1) 
- (modified) polly/test/CodeGen/annotated_alias_scopes.ll (+1-1) 
- (modified) polly/test/CodeGen/scev-backedgetaken.ll (+1-1) 
- (modified) polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll (+1-1) 


``````````diff
diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 6b0b1b55213e1..b46ae34e58a44 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -209,10 +209,8 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createUnrollAndJamMetadata(Attrs, LoopProperties, HasUserTransforms);
@@ -279,10 +277,10 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
       (Attrs.VectorizeScalable == LoopAttributes::Disable &&
        Attrs.VectorizeWidth != 1)) {
     VectorizeEnabled = Attrs.VectorizeEnable != LoopAttributes::Disable;
-    Args.push_back(
-        MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                          ConstantAsMetadata::get(ConstantInt::get(
-                              llvm::Type::getInt1Ty(Ctx), VectorizeEnabled))}));
+    Args.push_back(MDNode::get(
+        Ctx, {MDString::get(Ctx, VectorizeEnabled
+                                     ? "llvm.loop.vectorize.enable"
+                                     : "llvm.loop.vectorize.disable")}));
   }
 
   // Apply all loop properties to the vectorized loop.
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 04f5656dcd638..20615d9744c6e 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -25,7 +25,7 @@ extern "C" void followup_inner(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[OUTERLOOP_9]] = distinct !{![[OUTERLOOP_9]], [[MP:![0-9]+]], ![[UNROLLANDJAM_COUNT_10:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPINNER_11:[0-9]+]]}
 // CHECK-DAG: ![[UNROLLANDJAM_COUNT_10]] = !{!"llvm.loop.unroll_and_jam.count", i32 4}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index fdd5a07d00857..70ce57c6fb6a7 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_outer(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable", i1 true}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[VECTORIZE_FOLLOWUP_9:[0-9]+]] = !{!"llvm.loop.vectorize.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[ISVECTORIZED_11:[0-9]+]], ![[UNROLLANDJAM_12:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPOUTER_13:[0-9]+]]}
 // CHECK-DAG: ![[ISVECTORIZED_11:[0-9]+]] = !{!"llvm.loop.isvectorized"}
diff --git a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
index 37e29776264b6..afab64f4c6867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
@@ -43,7 +43,7 @@ void loop4(int *List, int Length) {
 // CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE:.*]]}
 // CHECK: ![[VEC_WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
 // CHECK: ![[FIXED_WIDTH]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE]]}
 // CHECK: ![[VEC_WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
index 8a25ed8de6239..421110fd9f867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
@@ -103,7 +103,7 @@ void test9(int *List, int Length) {
 
 // CHECK:      ![[LOOP0]] = distinct !{![[LOOP0]], [[MP:![0-9]+]], [[GEN3:![0-9]+]]}
 // CHECK:      [[MP]] = !{!"llvm.loop.mustprogress"}
-// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-NEXT: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP]], [[GEN6:![0-9]+]], [[GEN3]]}
 // CHECK-NEXT: [[GEN6]] = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-safety.cpp b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
index db1b6769e0514..e2a7a124890a6 100644
--- a/clang/test/CodeGenCXX/pragma-loop-safety.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
@@ -51,7 +51,7 @@ void interleave_test(int *List, int Length) {
 // CHECK: ![[PARALLEL_ACCESSES_7]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_2]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[INTERLEAVE_1]] = !{!"llvm.loop.interleave.count", i32 1}
-// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: ![[ACCESS_GROUP_8]] = distinct !{}
 // CHECK: ![[LOOP2_HINTS]] = distinct !{![[LOOP2_HINTS]], [[MP]], ![[PARALLEL_ACCESSES_11:[0-9]+]], ![[UNROLL_DISABLE]], ![[WIDTH_1:[0-9]+]], ![[INTENABLE_1]]}
 // CHECK: ![[PARALLEL_ACCESSES_11]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_8]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index 8cb3346247daf..d4a9faaf2be6b 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -256,7 +256,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[INTERLEAVE_10:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 10}
 // CHECK-DAG: ![[INTERLEAVE_16:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 16}
 
-// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: ![[FIXED_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: ![[SCALABLE_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 // CHECK-DAG: ![[WIDTH_1:[0-9]+]] = !{!"llvm.loop.vectorize.width", i32 1}
diff --git a/clang/test/OpenMP/for_simd_codegen.cpp b/clang/test/OpenMP/for_simd_codegen.cpp
index 1ac95e8943640..ae6161d03978d 100644
--- a/clang/test/OpenMP/for_simd_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_codegen.cpp
@@ -802,10 +802,10 @@ void parallel_simd(float *a) {
 }
 // TERM_DEBUG: !{{[0-9]+}} = !DILocation(line: [[@LINE-11]],
 // TERM_DEBUG-NOT: line: 0,
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // CHECK-DAG: ![[SIMD_LOOP]] = distinct !{![[SIMD_LOOP]], {{.*}}![[VECT_LOOP:[^,]+]]}
-// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // OMP50-DAG: ![[NOSIMD_LOOP]] = distinct !{![[NOSIMD_LOOP]], {{.*}}![[NOVECT_LOOP:[^,]+]]}
-// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.disable"}
 #endif // HEADER
diff --git a/clang/test/OpenMP/for_simd_scan_codegen.cpp b/clang/test/OpenMP/for_simd_scan_codegen.cpp
index 829f2656042fb..9ab62f8a682b2 100644
--- a/clang/test/OpenMP/for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_scan_codegen.cpp
@@ -300,5 +300,5 @@ void baz(int n) {
 }
 
 #endif
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
diff --git a/clang/test/OpenMP/irbuilder_safelen.cpp b/clang/test/OpenMP/irbuilder_safelen.cpp
index 6a4266f9646a7..3090eafa6a508 100644
--- a/clang/test/OpenMP/irbuilder_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
index 210978fc6069c..215b37e4bbb5f 100644
--- a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simd.cpp b/clang/test/OpenMP/irbuilder_simd.cpp
index 9c36e4ab97182..03f92d8117d45 100644
--- a/clang/test/OpenMP/irbuilder_simd.cpp
+++ b/clang/test/OpenMP/irbuilder_simd.cpp
@@ -65,7 +65,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: ![[META3:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META4]]  = distinct !{![[META4]], ![[META5:[0-9]+]], ![[META6:[0-9]+]]}
 // CHECK-NEXT: ![[META5]]  = !{!"llvm.loop.parallel_accesses", ![[META3]]}
-// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable"}
 // CHECK-NEXT: ![[META7:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META8]]  = distinct !{![[META8]], ![[META9:[0-9]+]], ![[META6]]}
 // CHECK-NEXT: ![[META9]]  = !{!"llvm.loop.parallel_accesses", ![[META7]]}
diff --git a/clang/test/OpenMP/irbuilder_simd_aligned.cpp b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
index b5ce43c8ea6a9..bb121036b4f0c 100644
--- a/clang/test/OpenMP/irbuilder_simd_aligned.cpp
+++ b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
@@ -293,7 +293,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP4]] = distinct !{}
 // CHECK: [[LOOP5]] = distinct !{[[LOOP5]], [[META6:![0-9]+]], [[META7:![0-9]+]]}
 // CHECK: [[META6]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP4]]}
-// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[ACC_GRP8]] = distinct !{}
 // CHECK: [[LOOP9]] = distinct !{[[LOOP9]], [[META10:![0-9]+]], [[META7]]}
 // CHECK: [[META10]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP8]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen.cpp b/clang/test/OpenMP/irbuilder_simdlen.cpp
index f6c26923851d0..9a7e8d5332ae0 100644
--- a/clang/test/OpenMP/irbuilder_simdlen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
index e46d5dde841b3..bb35f9341ded4 100644
--- a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 2}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
index d78d193e2edd1..f1eae642b613c 100644
--- a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
@@ -238,10 +238,10 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// OMP50-DAG: !{!"llvm.loop.vectorize.disable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/parallel_for_simd_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
index 40af7069bbf7d..cc8cab706975a 100644
--- a/clang/test/OpenMP/parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
@@ -798,16 +798,16 @@ for (int i = 0; i < 10; ++i);
 // OMP50: omp.inner.for.body:
 // OMP50: %{{[0-9]+}} = atomicrmw add ptr %{{[0-9]+}}, i32 1 monotonic, align 4, !llvm.access.group !{{[0-9]+}}
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP45-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP45-DAG: ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP50-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP50-DAG  ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
 // OMP50-DAG: ![[NOVECT]] = distinct !{![[NOVECT]], ![[NOVM:.+]]}
-// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.disable"}
 
 // TERM_DEBUG-LABEL: bar
 int bar() { extern void mayThrow(); mayThrow(); return 0; };
diff --git a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
index cac997753d480..e3810c27163a4 100644
--- a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
@@ -299,7 +299,7 @@ void baz(int n) {
   }
 }
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
 #endif
 
diff --git a/clang/test/OpenMP/simd_codegen.cpp b/clang/test/OpenMP/simd_codegen.cpp
index b96e4213e8e0e..ee8d709ba47fe 100644
--- a/clang/test/OpenMP/simd_codegen.cpp
+++ b/clang/test/OpenMP/simd_codegen.cpp
@@ -861,7 +861,7 @@ S8 s8(0);
 
 // TERM_DEBUG-NOT: line: 0,
 // TERM_DEBUG: distinct !DISubprogram(linkageName: "_GLOBAL__sub_I_simd_codegen.cpp",
-// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.disable"}
 // OMP5-DAG: ![[DISABLE_VECT]] = distinct !{{.*}}![[NOVECT]]{{[,}]}}
 #endif // HEADER
 
diff --git a/clang/test/OpenMP/simd_metadata.c b/clang/test/OpenMP/simd_metadata.c
index 83bcbf6deefe2..6fc4c401135f2 100644
--- a/clang/test/OpenMP/simd_metadata.c
+++ b/clang/test/OpenMP/simd_metadata.c
@@ -109,7 +109,7 @@ void h3(float *c, float *a, float *b, int size)
 // Metadata for h1:
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_16:![0-9]+]], [[LOOP_VEC_ENABLE:![0-9]+]]}
 // CHECK: [[LOOP_WIDTH_16]] = !{!"llvm.loop.vectorize.width", i32 16}
-// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_8:![0-9]+]], [[LOOP_VEC_ENABLE]]}
 // CHECK: [[LOOP_WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 // CHECK: ![[ACCESS_GROUP_7]] = distinct !{}
diff --git a/clang/test/OpenMP/target_simd_codegen.cpp b/clang/test/OpenMP/target_simd_codegen.cpp
index 0c2dde23f6c46..59284c201bb30 100644
--- a/clang/test/OpenMP/target_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_simd_codegen.cpp
@@ -702,9 +702,9 @@ int bar(int n){
 // Store captures in the context.
 // CHECK-DAG:   [[REF_B:%.+]] = load ptr, ptr [[LOCAL_B]],
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP51: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP51: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// TOMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// OMP51: !{!"llvm.loop.vectorize.disable"}
+// TOMP51: !{!"llvm.loop.vectorize.disable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
index 42e8377bd22dd..fd2d0d0c18737 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
@@ -455,7 +455,7 @@ int bar(int a){
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2STILi100EE3fooEv", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2SCC1Ev", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
index b63f0b6caeee2..c169d4a556bf0 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
@@ -82,6 +82,6 @@ int nested(int a){
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T1L]], i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T2L]], i32 0, i32 {{[0-9]+}}}
 
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// TCHECK-DAG: ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/210932

From llvmlistbot at llvm.org  Tue Jul 21 03:21:40 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 21 Jul 2026 03:21:40 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210932@github.com>
Message-ID: <6a5f4834.d681cab9.169270.3748@mx.google.com>


https://github.com/madhur13490 edited https://github.com/llvm/llvm-project/pull/210932

From llvmlistbot at llvm.org  Tue Jul 21 03:25:43 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 21 Jul 2026 03:25:43 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210932@github.com>
Message-ID: <6a5f4927.d2cb4adc.ccd80.357e@mx.google.com>


https://github.com/madhur13490 edited https://github.com/llvm/llvm-project/pull/210932

From llvmlistbot at llvm.org  Tue Jul 21 03:37:34 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 21 Jul 2026 03:37:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][scf] Expose kPeeledLoopLabel as a
 public marker attribute (PR #210922)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210922@github.com>
Message-ID: <6a5f4bee.92d2873c.aa1ff.62be@mx.google.com>


================
@@ -106,6 +106,9 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
 LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
                                         scf::ForOp &partialIteration);
 
+/// Marker attribute name for loops that have already been peeled.
+static constexpr StringLiteral kPeeledLoopLabel = "__peeled_loop__";
----------------
joker-eph wrote:

This is breaking any encapsulation, can we hide this behind method checks please?


https://github.com/llvm/llvm-project/pull/210922

From llvmlistbot at llvm.org  Tue Jul 21 03:37:45 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 03:37:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f4bf9.bd93e731.30f155.4f73@mx.google.com>


https://github.com/banach-space updated https://github.com/llvm/llvm-project/pull/210288

>From d9c4ebfc9363b1c9fd3af3f7dbb04815dbf9d6a1 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 17 Jul 2026 09:54:59 +0000
Subject: [PATCH 1/2] [mlir][vector] Make CompressstoreOp` + ExpandloadOp
 support scalable vectors

Extends `vector.compressstore` + `vector.expandload` to support scalable
vectors and updates relevant tests.
---
 .../mlir/Dialect/Vector/IR/VectorOps.td       |   4 +-
 .../vector-to-llvm-interface.mlir             |  53 ++++-
 mlir/test/Dialect/Vector/invalid.mlir         |  16 --
 mlir/test/Dialect/Vector/ops.mlir             |  20 ++
 .../vector-dropleadunitdim-transforms.mlir    |  24 ++
 .../Dialect/Vector/vector-mem-transforms.mlir |  50 ++++
 .../Dialect/Vector/CPU/ArmSVE/compress.mlir   | 214 ++++++++++++++++++
 7 files changed, 351 insertions(+), 30 deletions(-)
 create mode 100644 mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index 6e0134fc0cdc6..ebfb379684dd3 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2289,7 +2289,7 @@ def Vector_ExpandLoadOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$pass_thru,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)>,
     Results<(outs AnyVectorOfNonZeroRank:$result)> {
@@ -2381,7 +2381,7 @@ def Vector_CompressStoreOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$valueToStore,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)> {
 
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
index cc35c6600d12c..4cbea04e8076e 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
@@ -1944,13 +1944,13 @@ func.func @negative_scatter_on_strided_memref(%arg0: memref<?xf32, strided<[2],
 // vector.expandload
 //===----------------------------------------------------------------------===//
 
-func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
+func.func @expandload(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32> into vector<11xf32>
   return %0 : vector<11xf32>
 }
 
-// CHECK-LABEL: func @expand_load_op
+// CHECK-LABEL: func @expandload
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1959,21 +1959,36 @@ func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vec
 
 // -----
 
-func.func @expand_load_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
+func.func @expandload_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) -> vector<[11]xf32> {
+  %c0 = arith.constant 0: index
+  %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32> into vector<[11]xf32>
+  return %0 : vector<[11]xf32>
+}
+
+// CHECK-LABEL: func @expandload_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: %[[E:.*]] = "llvm.intr.masked.expandload"(%[[P]], %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<[11]xi1>, vector<[11]xf32>) -> vector<[11]xf32>
+// CHECK: return %[[E]] : vector<[11]xf32>
+
+// -----
+
+func.func @expandload_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_index
+// CHECK-LABEL: func @expandload_index
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
 
-func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
+func.func @expandload_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_with_alignment
+// CHECK-LABEL: func @expandload_with_alignment
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{llvm.align = 8 : i64}, {}, {}]}> : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
@@ -1982,13 +1997,13 @@ func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<1
 // vector.compressstore
 //===----------------------------------------------------------------------===//
 
-func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
+func.func @compressstore(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32>
   return
 }
 
-// CHECK-LABEL: func @compress_store_op
+// CHECK-LABEL: func @compressstore
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1996,21 +2011,35 @@ func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2:
 
 // -----
 
-func.func @compress_store_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
+func.func @compressstore_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32>
+  return
+}
+
+// CHECK-LABEL: func @compressstore_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %[[P]], %{{.*}}) : (vector<[11]xf32>, !llvm.ptr, vector<[11]xi1>) -> ()
+
+// -----
+
+func.func @compressstore_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_index
+// CHECK-LABEL: func @compressstore_index
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
 
-func.func @compress_store_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
+func.func @compressstore_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
   vector.compressstore %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_with_alignment
+// CHECK-LABEL: func @compressstore_with_alignment
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{}, {llvm.align = 8 : i64}, {}]}> : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..d4f305f20b595 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1688,14 +1688,6 @@ func.func @expand_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>,
 
 // -----
 
-func.func @expand_base_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.expandload' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
-}
-
-// -----
-
 func.func @expand_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.expandload' op expected result shape to match mask shape}}
@@ -1750,14 +1742,6 @@ func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1
 
 // -----
 
-func.func @compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %value: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.compressstore' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
-}
-
-// -----
-
 func.func @compress_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op expected valueToStore shape to match mask shape}}
diff --git a/mlir/test/Dialect/Vector/ops.mlir b/mlir/test/Dialect/Vector/ops.mlir
index e84bd3f1dce17..1b4109a3368ea 100644
--- a/mlir/test/Dialect/Vector/ops.mlir
+++ b/mlir/test/Dialect/Vector/ops.mlir
@@ -885,6 +885,16 @@ func.func @expand_and_compress(%base: memref<?xf32>, %mask: vector<16xi1>, %pass
   return
 }
 
+// CHECK-LABEL: @expand_and_compress_scalable
+func.func @expand_and_compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}], %{{.*}}, %{{.*}} : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}], %{{.*}}, %[[X]] : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0], %mask, %0 : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @expand_and_compress2d
 func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -895,6 +905,16 @@ func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %
   return
 }
 
+// CHECK-LABEL: @expand_and_compress2d_scalable
+func.func @expand_and_compress2d_scalable(%base: memref<?x?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %{{.*}} : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0, %c0], %mask, %pass_thru : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %[[X]] : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0, %c0], %mask, %0 : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @multi_reduction
 func.func @multi_reduction(%0: vector<4x8x16x32xf32>, %acc0: vector<4x16xf32>,
                            %acc1: f32) -> f32 {
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..ff978665125ed 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -733,6 +733,19 @@ func.func @cast_away_expandload_leading_one_dims(%base: memref<16xf32>, %i: inde
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_expandload_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[P:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         %[[L:.+]] = vector.expandload %{{.*}}[%{{.*}}], %[[M]], %[[P]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32> into vector<[4]xf32>
+// CHECK:         %[[B:.+]] = vector.broadcast %[[L]] : vector<[4]xf32> to vector<1x[4]xf32>
+// CHECK:         return %[[B]] : vector<1x[4]xf32>
+func.func @cast_away_expandload_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %pass: vector<1x[4]xf32>) -> vector<1x[4]xf32> {
+  %0 = vector.expandload %base[%i], %mask, %pass : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32> into vector<1x[4]xf32>
+  return %0 : vector<1x[4]xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_gather_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
@@ -779,6 +792,17 @@ func.func @cast_away_compressstore_leading_one_dims(%base: memref<16xf32>, %i: i
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_compressstore_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[V:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         vector.compressstore %{{.*}}[%{{.*}}], %[[M]], %[[V]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32>
+func.func @cast_away_compressstore_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %val: vector<1x[4]xf32>) {
+  vector.compressstore %base[%i], %mask, %val : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_scatter_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
diff --git a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
index 2004a47851e2e..cd7ea84cdb2eb 100644
--- a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
@@ -175,6 +175,20 @@ func.func @fold_expandload_all_true(%base: memref<16xf32>, %pass_thru: vector<16
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_true_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-DAG:       %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      %[[T:.*]] = vector.load %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return %[[T]] : vector<[16]xf32>
+func.func @fold_expandload_all_true_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 // CHECK-LABEL:   func @fold_expandload_all_false(
 // CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<16xf32>) -> vector<16xf32> {
@@ -187,6 +201,18 @@ func.func @fold_expandload_all_false(%base: memref<16xf32>, %pass_thru: vector<1
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_false_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-NEXT:      return %[[PASS_THRU]] : vector<[16]xf32>
+func.func @fold_expandload_all_false_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 //-----------------------------------------------------------------------------
 // [Pattern: CompressStoreFolder]
 //-----------------------------------------------------------------------------
@@ -204,6 +230,19 @@ func.func @fold_compressstore_all_true(%base: memref<16xf32>, %value: vector<16x
   return
 }
 
+// CHECK-LABEL:   func @fold_compressstore_all_true_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      vector.store %[[VALUE]], %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_true_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value  : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL:   func @fold_compressstore_all_false(
 // CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                    %[[VALUE:.*]]: vector<16xf32>) {
@@ -214,3 +253,14 @@ func.func @fold_compressstore_all_false(%base: memref<16xf32>, %value: vector<16
   vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<16xi1>, vector<16xf32>
   return
 }
+
+// CHECK-LABEL:   func @fold_compressstore_all_false_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_false_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
new file mode 100644
index 0000000000000..d8752cfb7fc92
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
@@ -0,0 +1,214 @@
+// REQUIRES: arm-emulator
+
+/// End-to-end test for vector.compressstore for SVE
+
+// In order to demonstrate the impact of using scalable vectors, vscale is set
+// to 2 so that vector<[16]xi32> constains 32 rather than 16 elements at
+// run-time
+//
+// Note that you can also tweak the size of vscale by passing this flag to
+// QEMU:
+//  * -cpu max,sve-max-vq=[1-16]
+// (select the value between 1 and 16).
+
+// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm
+// DEFINE: %{run} = %mcr_aarch64_cmd %t -e main -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils,%native_mlir_arm_runner_utils
+
+// RUN: rm -f %t && %{compile} &&  %{run} |  FileCheck %s
+
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// The number of inserted elements is 16 x vscale. Insertion index is
+// hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xi32>,
+                 %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %base[%c0], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xi32>,
+                     %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c8 = arith.constant 8: index
+  vector.compressstore %base[%c8], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xi32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xi32> to memref<*xi32>
+
+  call @printMemrefI32(%cast): (memref<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @reset_mem_i32
+//
+// Resets the input memory to 0.
+
+// TODO: Create a run-time utility funcion.
+//===----------------------------------------------------------------------===//
+func.func @reset_mem_i32(%ptr: memref<?xi32>, %size: index) {
+  %c0_idx = arith.constant 0: index
+  %c0 = arith.constant 0: i32
+  %step = arith.constant 1: index
+
+  scf.for %i = %c0_idx to %size step %step {
+    memref.store %c0, %ptr[%i] : memref<?xi32>
+  }
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point - sets the value of vscale.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Set vscale to 2 (vector width = 256). This will have identical effect to:
+  //  * qemu-aarch64 -cpu max,sve-max-vq=2 (...)
+  %c256 = arith.constant 256 : i32
+  func.call @setArmVLBits(%c256) : (i32) -> ()
+
+  // Run the tests.
+  func.call @test() : () -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @test
+//
+// Set-up and run tests.
+//===----------------------------------------------------------------------===//
+func.func @test() {
+  //
+  // Shared constants.
+  //
+  %vs = vector.vscale
+
+  //
+  // Set up memory.
+  //
+  %c16 = arith.constant 16: index
+  %vs_16 = arith.muli %vs, %c16 : index
+  %A = memref.alloc(%vs_16) : memref<?xi32>
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+
+  //
+  // Set the input vector.
+  //
+  %value = vector.step : vector<[16]xi32>
+  vector.print %value : vector<[16]xi32>
+
+  //
+  // Set up masks.
+  //
+
+  %f = arith.constant 0: i1
+  %t = arith.constant 1: i1
+
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %none = vector.constant_mask [0] : vector<[16]xi1>
+
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ..., 1]
+  %all = vector.constant_mask [16] : vector<[16]xi1>
+  vector.print %all : vector<[16]xi1>
+
+  // %first_vscale_4 = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %c4 = arith.constant 4 : index
+  %vs_4 = arith.muli %vs, %c4 : index
+  %first_vscale_4 = vector.create_mask %vs_4 : vector<[16]xi1>
+  vector.print %first_vscale_4 : vector<[16]xi1>
+
+  // %odd = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, ..., 0]
+  %0 = vector.insert %t, %none[1] : i1 into vector<[16]xi1>
+  %1 = vector.insert %t, %0[3] : i1 into vector<[16]xi1>
+  %2 = vector.insert %t, %1[5] : i1 into vector<[16]xi1>
+  %3 = vector.insert %t, %2[7] : i1 into vector<[16]xi1>
+  %4 = vector.insert %t, %3[9] : i1 into vector<[16]xi1>
+  %5 = vector.insert %t, %4[11] : i1 into vector<[16]xi1>
+  %odd = vector.insert %t, %5[13] : i1 into vector<[16]xi1>
+
+  // %even = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, ..., 0]
+  %6 = vector.insert %t, %none[0] : i1 into vector<[16]xi1>
+  %7 = vector.insert %t, %6[2] : i1 into vector<[16]xi1>
+  %8 = vector.insert %t, %7[4] : i1 into vector<[16]xi1>
+  %9 = vector.insert %t, %8[6] : i1 into vector<[16]xi1>
+  %10 = vector.insert %t, %9[8] : i1 into vector<[16]xi1>
+  %11 = vector.insert %t, %10[10] : i1 into vector<[16]xi1>
+  %even = vector.insert %t, %11[12] : i1 into vector<[16]xi1>
+
+
+  //
+  // Tests.
+  //
+
+  call @compress_16(%A, %none, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %all, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
+  /// (...)
+  // CHECK-SAME: 30, 31]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16_at_8(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %odd, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [1,  3,  5,  7,  9,  11,  13,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %even, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0,  2,  4,  6,  8,  10,  12,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  memref.dealloc %A : memref<?xi32>
+  return
+}
+
+func.func private @printMemrefI32(%ptr : memref<*xi32>)
+func.func private @setArmVLBits(%bits : i32)

>From ced689efcae970392f8c8b1da697dc8ae351d352 Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Tue, 21 Jul 2026 09:26:10 +0000
Subject: [PATCH 2/2] Address PR comments

---
 mlir/include/mlir/Dialect/Vector/IR/VectorOps.td  |  4 ----
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp          |  6 ++++++
 mlir/test/Dialect/Vector/invalid.mlir             | 15 +++++++++++++++
 .../Dialect/Vector/CPU/ArmSVE/compress.mlir       |  4 ++--
 4 files changed, 23 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index ebfb379684dd3..e2f8f1a8817a5 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2328,8 +2328,6 @@ def Vector_ExpandLoadOp :
     memory at an address aligned to this boundary. Violating this requirement
     triggers immediate undefined behavior.
 
-    Note, at the moment this Op is only available for fixed-width vectors.
-
     Examples:
 
     ```mlir
@@ -2419,8 +2417,6 @@ def Vector_CompressStoreOp :
     memory at an address aligned to this boundary. Violating this requirement
     triggers immediate undefined behavior.
 
-    Note, at the moment this Op is only available for fixed-width vectors.
-
     Examples:
 
     ```mlir
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 94f627d12752e..e2b77c51e9217 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6653,6 +6653,9 @@ LogicalResult ExpandLoadOp::verify() {
     return emitOpError("requires ") << memType.getRank() << " indices";
   if (resVType.getShape() != maskVType.getShape())
     return emitOpError("expected result shape to match mask shape");
+  if (resVType.getScalableDims() != maskVType.getScalableDims())
+    return emitOpError(
+        "expected result scalable dims to match mask scalable dims");
   if (resVType != passVType)
     return emitOpError("expected pass_thru of same type as result type");
   return success();
@@ -6707,6 +6710,9 @@ LogicalResult CompressStoreOp::verify() {
     return emitOpError("requires ") << memType.getRank() << " indices";
   if (valueVType.getShape() != maskVType.getShape())
     return emitOpError("expected valueToStore shape to match mask shape");
+  if (valueVType.getScalableDims() != maskVType.getScalableDims())
+    return emitOpError(
+        "expected valueToStore scalable dims to match mask scalable dims");
   return success();
 }
 
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index d4f305f20b595..0097db39f6ed9 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1734,6 +1734,13 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @expand_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16xi1>, %pass_thru: vector<[16]xf32>, %c0: index) {
+  // expected-error at +1 {{expected result scalable dims to match mask scalable dims}}
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<16xi1>, vector<[16]xf32> into vector<[16]xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1780,6 +1787,14 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compress_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16xi1>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error at +1 {{expected valueToStore scalable dims to match mask scalable dims}}
+  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<16xi1>, vector<[16]xf32>
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
diff --git a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
index d8752cfb7fc92..eddc7c664d3f1 100644
--- a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
+++ b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
@@ -3,7 +3,7 @@
 /// End-to-end test for vector.compressstore for SVE
 
 // In order to demonstrate the impact of using scalable vectors, vscale is set
-// to 2 so that vector<[16]xi32> constains 32 rather than 16 elements at
+// to 2 so that vector<[16]xi32> contains 32 rather than 16 elements at
 // run-time
 //
 // Note that you can also tweak the size of vscale by passing this flag to
@@ -11,7 +11,7 @@
 //  * -cpu max,sve-max-vq=[1-16]
 // (select the value between 1 and 16).
 
-// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm
+// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm -o %t
 // DEFINE: %{run} = %mcr_aarch64_cmd %t -e main -entry-point-result=void --march=aarch64 --mattr="+sve"\
 // DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils,%native_mlir_arm_runner_utils
 


From llvmlistbot at llvm.org  Tue Jul 21 03:38:11 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 03:38:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f4c13.d2cb4adc.ccd80.adc7@mx.google.com>


banach-space wrote:

> Great catch on this PR. Thanks so much. Overall, LGTM :D
> 
> I'll have some minor comments, and here's a concern of mine. Currently, the verifiers are checking the equality between shapes:
> 
> https://github.com/llvm/llvm-project/blob/ccd0e982a02fa3be39a98dd98ea6cc1ccaf24d13/mlir/lib/Dialect/Vector/IR/VectorOps.cpp#L6710-L6711
> 
> 
> https://github.com/llvm/llvm-project/blob/ccd0e982a02fa3be39a98dd98ea6cc1ccaf24d13/mlir/lib/Dialect/Vector/IR/VectorOps.cpp#L6656-L6657
> 
> 
> Should we take into account the scalable dims via `getScalableDims()` or something?

Updated, see https://github.com/llvm/llvm-project/pull/210288/changes/ced689efcae970392f8c8b1da697dc8ae351d352

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 03:41:31 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 03:41:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f4cdb.6c5720b3.2da3b9.de2f@mx.google.com>


================

----------------
banach-space wrote:

This is not possible at this point. `expand` is an SVE2p2 instruction, see:
* https://developer.arm.com/documentation/111108/2026-03/SVE-Instructions/EXPAND--Copy-lower-numbered-vector-elements-to-Active-elements-

However, there's no hardware that would implement it nor QEMU support:
* https://linaro.atlassian.net/browse/QEMU-975

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 03:50:51 2026
From: llvmlistbot at llvm.org (Madhur Amilkanthwar)
Date: Tue, 21 Jul 2026 03:50:51 -0700 (PDT)
Subject: [Mlir-commits] [clang] [flang] [llvm] [mlir] [polly] Enforce
 single-operand form for llvm.loop.vectorize metadata (PR #210932)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210932@github.com>
Message-ID: <6a5f4f0b.f06f4994.2f0620.3663@mx.google.com>


https://github.com/madhur13490 updated https://github.com/llvm/llvm-project/pull/210932

>From 42290bae11c4385c08d6b84d72dfc96896155754 Mon Sep 17 00:00:00 2001
From: Madhur Amilkanthwar <madhura at nvidia.com>
Date: Sun, 12 Jul 2026 23:37:19 -0700
Subject: [PATCH] Enforce single-operand form for llvm.loop.vectorize metadata

This patch implements Phase 2 of the RFC "Enforce Single-Operand Format
for All .enable Metadata Nodes". The two-operand boolean form
!{!"llvm.loop.vectorize.enable", i1 0/1} is replaced by a single-operand
enable/disable pair:

  !{!"llvm.loop.vectorize.enable"}    ; force vectorization
  !{!"llvm.loop.vectorize.disable"}   ; suppress vectorization

The Verifier rejects the two-operand form, AutoUpgrade rewrites old
bitcode (including the legacy llvm.vectorizer.enable tag), and the
readers and producers across LLVM, Clang, MLIR and Polly are updated.

Please refer to RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571/
---
 clang/lib/CodeGen/CGLoopInfo.cpp              |  14 +++--
 .../test/CodeGenCXX/pragma-followup_inner.cpp |   2 +-
 .../test/CodeGenCXX/pragma-followup_outer.cpp |   2 +-
 clang/test/CodeGenCXX/pragma-loop-pr27643.cpp |   2 +-
 .../test/CodeGenCXX/pragma-loop-predicate.cpp |   2 +-
 clang/test/CodeGenCXX/pragma-loop-safety.cpp  |   2 +-
 clang/test/CodeGenCXX/pragma-loop.cpp         |   2 +-
 clang/test/OpenMP/for_simd_codegen.cpp        |   8 +--
 clang/test/OpenMP/for_simd_scan_codegen.cpp   |   2 +-
 clang/test/OpenMP/irbuilder_safelen.cpp       |   2 +-
 .../irbuilder_safelen_order_concurrent.cpp    |   2 +-
 clang/test/OpenMP/irbuilder_simd.cpp          |   2 +-
 clang/test/OpenMP/irbuilder_simd_aligned.cpp  |   2 +-
 clang/test/OpenMP/irbuilder_simdlen.cpp       |   2 +-
 .../test/OpenMP/irbuilder_simdlen_safelen.cpp |   2 +-
 .../OpenMP/master_taskloop_simd_codegen.cpp   |   8 +--
 .../test/OpenMP/parallel_for_simd_codegen.cpp |  10 ++--
 .../OpenMP/parallel_for_simd_scan_codegen.cpp |   2 +-
 clang/test/OpenMP/simd_codegen.cpp            |   2 +-
 clang/test/OpenMP/simd_metadata.c             |   2 +-
 clang/test/OpenMP/target_simd_codegen.cpp     |   8 +--
 ...s_distribute_simd_codegen_registration.cpp |   4 +-
 ...ibute_simd_codegen_registration_naming.cpp |   4 +-
 clang/test/OpenMP/taskloop_simd_codegen.cpp   |   2 +-
 ...ute_parallel_for_simd_collapse_codegen.cpp |   4 +-
 .../OpenMP/teams_distribute_simd_codegen.cpp  |   2 +-
 .../test/PCH/delayed-template-with-pragma.cpp |   2 +-
 .../proto-to-llvm/loop_proto_to_llvm.cpp      |  10 ++--
 flang/test/Integration/ivdep.f90              |   2 +-
 flang/test/Integration/vector-always.f90      |   4 +-
 llvm/docs/LangRef.md                          |  10 ++--
 llvm/docs/TransformMetadata.rst               |   6 +--
 llvm/include/llvm/IR/AutoUpgrade.h            |  12 +++--
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |   6 +--
 llvm/lib/IR/AutoUpgrade.cpp                   |  48 +++++++++++++++---
 llvm/lib/IR/Verifier.cpp                      |   8 +++
 llvm/lib/Transforms/Utils/LoopConstrainer.cpp |   3 +-
 llvm/lib/Transforms/Utils/LoopUtils.cpp       |   6 +--
 .../Vectorize/LoopVectorizationLegality.cpp   |   8 +++
 llvm/test/Bitcode/upgrade-loop-metadata.ll    |   2 +-
 .../Bitcode/upgrade-loop-vectorize-enable.ll  |  37 ++++++++++++++
 .../upgrade-loop-vectorize-enable.ll.bc       | Bin 0 -> 2328 bytes
 .../IRCE/add-metadata-pre-post-loops.ll       |   2 +-
 .../test/Transforms/IRCE/correct-loop-info.ll |   2 +-
 .../Transforms/IRCE/loop-guarded-bounds.ll    |   2 +-
 llvm/test/Transforms/IRCE/pre_post_loops.ll   |   2 +-
 .../IRCE/range_intersect_miscompile.ll        |   2 +-
 .../IRCE/unsigned_comparisons_ugt.ll          |   2 +-
 .../IRCE/unsigned_comparisons_ult.ll          |   2 +-
 .../Inline/dilocation-loop-metadata-update.ll |   4 +-
 .../InstCombine/mem-par-metadata-memcpy.ll    |   2 +-
 .../LoopIdiom/AArch64/disable-vectorize.ll    |   4 +-
 .../LoopRotate/vect.omp.persistence.ll        |   4 +-
 .../enable_and_isvectorized.ll                |   2 +-
 .../LoopTransformWarning/optnone.ll           |   2 +-
 .../vectorization-remarks-missed.ll           |   2 +-
 .../vectorizer-loop-kind-unroll-warning.ll    |   2 +-
 .../LoopUnroll/unroll-pragmas-disabled.ll     |   4 +-
 .../vectorizer-loop-kind-remarks.ll           |   2 +-
 .../LoopVectorize/2016-07-27-loop-vec.ll      |   2 +-
 .../LoopVectorize/AArch64/call-costs.ll       |   2 +-
 .../AArch64/conditional-branches-cost.ll      |   2 +-
 .../AArch64/conditional-scalar-assignment.ll  |   2 +-
 .../AArch64/interleave-with-gaps.ll           |   2 +-
 .../LoopVectorize/AArch64/masked-op-cost.ll   |   2 +-
 .../LoopVectorize/AArch64/masked_ldst_sme.ll  |   2 +-
 .../LoopVectorize/AArch64/optsize_minsize.ll  |   2 +-
 .../AArch64/outer_loop_prefer_scalable.ll     |   2 +-
 ...outer_loop_test1_no_explicit_vect_width.ll |   2 +-
 .../partial-reduce-dot-product-epilogue.ll    |   2 +-
 .../partial-reduce-dot-product-mixed.ll       |   2 +-
 .../partial-reduce-dot-product-neon.ll        |   2 +-
 .../AArch64/partial-reduce-dot-product.ll     |   2 +-
 .../AArch64/partial-reduce-sub.ll             |   2 +-
 .../AArch64/pr60831-sve-inv-store-crash.ll    |   2 +-
 .../LoopVectorize/AArch64/predicated-costs.ll |   2 +-
 .../AArch64/scalable-predicate-instruction.ll |   2 +-
 .../AArch64/scalable-reductions-tf.ll         |   2 +-
 .../AArch64/scalable-reductions.ll            |   2 +-
 .../LoopVectorize/AArch64/strict-fadd.ll      |   2 +-
 .../LoopVectorize/AArch64/sve-basic-vec.ll    |   2 +-
 .../AArch64/sve-cond-inv-loads.ll             |   2 +-
 .../AArch64/sve-extract-last-veclane.ll       |   2 +-
 .../AArch64/sve-gather-scatter-cost.ll        |   2 +-
 .../AArch64/sve-gather-scatter.ll             |   2 +-
 .../LoopVectorize/AArch64/sve-inductions.ll   |   2 +-
 .../LoopVectorize/AArch64/sve-inv-store.ll    |   2 +-
 .../AArch64/sve-large-strides.ll              |   2 +-
 .../AArch64/sve-masked-loadstore.ll           |   2 +-
 .../AArch64/sve-tail-folding-option.ll        |   2 +-
 .../LoopVectorize/AArch64/sve-type-conv.ll    |   2 +-
 .../AArch64/sve-vector-reverse-mask4.ll       |   2 +-
 .../AArch64/sve-vector-reverse.ll             |   2 +-
 .../AArch64/sve-widen-extractvalue.ll         |   2 +-
 .../LoopVectorize/AArch64/sve-widen-phi.ll    |   2 +-
 .../AArch64/unsafe-vf-hint-remark.ll          |   2 +-
 .../AArch64/vector-reverse-mask4.ll           |   2 +-
 .../LoopVectorize/AArch64/vector-reverse.ll   |   2 +-
 .../vplan-native-outer-loop-wide-type.ll      |   2 +-
 .../widen-gep-all-indices-invariant.ll        |   2 +-
 .../LoopVectorize/ARM/optsize_minsize.ll      |   2 +-
 .../ARM/prefer-tail-loop-folding.ll           |   2 +-
 .../ARM/tail-folding-loop-hint.ll             |   2 +-
 .../ARM/tail-folding-prefer-flag.ll           |   2 +-
 .../RISCV/dbg-tail-folding-by-evl.ll          |   2 +-
 .../LoopVectorize/RISCV/force-vect-msg.ll     |   2 +-
 .../RISCV/riscv-vector-reverse.ll             |   2 +-
 .../RISCV/tail-folding-cond-reduction.ll      |   2 +-
 .../tail-folding-fixed-order-recurrence.ll    |   2 +-
 .../RISCV/tail-folding-inloop-reduction.ll    |   2 +-
 .../RISCV/tail-folding-interleave.ll          |   2 +-
 .../RISCV/tail-folding-intermediate-store.ll  |   2 +-
 .../RISCV/tail-folding-ordered-reduction.ll   |   2 +-
 .../RISCV/tail-folding-reduction.ll           |   2 +-
 .../RISCV/tail-folding-uniform-store.ll       |   2 +-
 .../LoopVectorize/RISCV/uniform-load-store.ll |   2 +-
 .../LoopVectorize/Sparc/no-vectorize.ll       |   6 +--
 .../Transforms/LoopVectorize/VE/disable_lv.ll |   2 +-
 ...an-vp-intrinsics-fixed-order-recurrence.ll |   2 +-
 .../RISCV/vplan-vp-intrinsics-reduction.ll    |   2 +-
 .../LoopVectorize/VPlan/constant-fold.ll      |   2 +-
 .../VPlan/vplan-printing-outer-loop.ll        |   2 +-
 .../VPlan/vplan-stress-test-no-explict-vf.ll  |   2 +-
 .../LoopVectorize/X86/amdlibm-calls-finite.ll |   6 +--
 .../X86/conditional-scalar-assignment.ll      |   2 +-
 .../LoopVectorize/X86/cost-model.ll           |   2 +-
 .../LoopVectorize/X86/induction-costs.ll      |   2 +-
 .../LoopVectorize/X86/iv-live-outs.ll         |   2 +-
 .../X86/libm-vector-calls-VF2-VF8.ll          |  34 ++++++-------
 .../X86/libm-vector-calls-finite.ll           |  12 ++---
 .../LoopVectorize/X86/libm-vector-calls.ll    |  26 +++++-----
 .../Transforms/LoopVectorize/X86/no_fpmath.ll |   2 +-
 .../X86/no_fpmath_with_hotness.ll             |   2 +-
 .../X86/outer-loop-non-power-of-2-type.ll     |   2 +-
 ...outer_loop_test1_no_explicit_vect_width.ll |   2 +-
 .../LoopVectorize/X86/redundant-vf2-cost.ll   |   2 +-
 ...licating-load-store-costs-max-bandwidth.ll |   2 +-
 .../X86/replicating-load-store-costs.ll       |   2 +-
 .../LoopVectorize/X86/svml-calls-finite.ll    |  28 +++++-----
 .../X86/tail_folding_and_assume_safety.ll     |   2 +-
 .../LoopVectorize/X86/tail_loop_folding.ll    |   2 +-
 .../LoopVectorize/X86/vect.omp.force.ll       |   2 +-
 .../X86/vect.omp.force.small-tc.ll            |   2 +-
 .../X86/vectorization-remarks-missed.ll       |   2 +-
 .../Transforms/LoopVectorize/alias-mask.ll    |   2 +-
 .../LoopVectorize/dbg-outer-loop-vect.ll      |   2 +-
 .../diag-disabled-vectorization-msgs.ll       |   2 +-
 .../LoopVectorize/disable_nonforced_enable.ll |   2 +-
 .../LoopVectorize/explicit_outer_detection.ll |   2 +-
 .../explicit_outer_nonuniform_inner.ll        |   2 +-
 .../explicit_outer_uniform_diverg_branch.ll   |   2 +-
 .../LoopVectorize/make-followup-loop-id.ll    |   2 +-
 .../LoopVectorize/memdep-fold-tail.ll         |   2 +-
 .../LoopVectorize/nested-outer-loop-vect.ll   |   2 +-
 ...o-fold-tail-by-masking-iv-external-uses.ll |   2 +-
 .../LoopVectorize/no_array_bounds.ll          |   2 +-
 .../LoopVectorize/no_array_bounds_scalable.ll |   2 +-
 .../Transforms/LoopVectorize/no_switch.ll     |   2 +-
 .../no_switch_disable_vectorization.ll        |   2 +-
 .../noalias-scope-decl-outer-loop.ll          |   2 +-
 .../non-widenable-intrinsics-outer-loop.ll    |   2 +-
 llvm/test/Transforms/LoopVectorize/optsize.ll |   2 +-
 .../LoopVectorize/outer-loop-atomic.ll        |   2 +-
 .../outer-loop-inner-latch-successors.ll      |   2 +-
 .../outer-loop-vec-phi-predecessor-order.ll   |   2 +-
 .../LoopVectorize/outer-loop-wide-phis.ll     |   2 +-
 .../LoopVectorize/outer_loop_contiguous.ll    |   2 +-
 .../LoopVectorize/outer_loop_early_exit.ll    |   2 +-
 .../outer_loop_hcfg_construction.ll           |   2 +-
 .../LoopVectorize/outer_loop_scalable.ll      |   2 +-
 .../LoopVectorize/outer_loop_test1.ll         |   2 +-
 .../LoopVectorize/outer_loop_test2.ll         |   2 +-
 .../LoopVectorize/reduction-inloop-uf4.ll     |   2 +-
 .../LoopVectorize/reduction-inloop.ll         |   2 +-
 .../LoopVectorize/remove_metadata.ll          |   2 +-
 ...ime-check-threshold-with-force-metadata.ll |   4 +-
 .../Transforms/LoopVectorize/runtime-check.ll |   2 +-
 .../LoopVectorize/scalable-inductions.ll      |   2 +-
 .../LoopVectorize/unsafe-vf-hint-remark.ll    |   2 +-
 .../use-scalar-epilogue-if-tp-fails.ll        |   2 +-
 .../LoopVectorize/vect.omp.persistence.ll     |   4 +-
 ...kedge-elimination-predicated-early-exit.ll |   2 +-
 ...plan-outer-loop-uncomputable-trip-count.ll |   2 +-
 .../vplan-vectorize-inner-loop-reduction.ll   |   2 +-
 .../vplan-widen-call-instruction.ll           |   4 +-
 .../vplan-widen-select-instruction.ll         |   2 +-
 .../AArch64/Oz-and-forced-vectorize.ll        |   2 +-
 .../X86/loop-vectorize-metadata.ll            |   4 +-
 .../X86/preserve-access-group.ll              |   2 +-
 .../Transforms/SROA/mem-par-metadata-sroa.ll  |   2 +-
 .../X86/combine-parallel-mem-md.ll            |   2 +-
 .../Verifier/llvm.loop-cu-strip-followup.ll   |   4 +-
 llvm/test/Verifier/llvm.loop.vectorize.ll     |  41 +++++++++++++++
 .../Target/LLVMIR/LoopAnnotationImporter.cpp  |   5 +-
 .../LLVMIR/LoopAnnotationTranslation.cpp      |   7 ++-
 .../Target/LLVMIR/Import/import-failure.ll    |  16 ------
 .../Target/LLVMIR/Import/metadata-loop.ll     |   2 +-
 mlir/test/Target/LLVMIR/loop-metadata.mlir    |   2 +-
 .../LLVMIR/openmp-parallel-do-simd.mlir       |   2 +-
 .../Target/LLVMIR/openmp-simd-ordered.mlir    |   2 +-
 .../LLVMIR/openmp-simd-reduction-byref.mlir   |   2 +-
 .../LLVMIR/openmp-simd-reduction-simple.mlir  |   2 +-
 ...nmp-teams-distribute-parallel-do-simd.mlir |   2 +-
 .../LLVMIR/openmp-wsloop-simd-ordered.mlir    |   2 +-
 polly/lib/CodeGen/CodeGeneration.cpp          |  18 ++++++-
 polly/lib/CodeGen/IRBuilder.cpp               |   9 ++--
 polly/test/CodeGen/MemAccess/create_arrays.ll |   2 +-
 .../CodeGen/Metadata/basic_vec_annotate.ll    |   4 +-
 .../CodeGen/Metadata/fallback_vec_annotate.ll |   2 +-
 polly/test/CodeGen/annotated_alias_scopes.ll  |   2 +-
 polly/test/CodeGen/scev-backedgetaken.ll      |   2 +-
 .../pattern-matching-based-opts_14.ll         |   2 +-
 212 files changed, 466 insertions(+), 338 deletions(-)
 create mode 100644 llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
 create mode 100644 llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc
 create mode 100644 llvm/test/Verifier/llvm.loop.vectorize.ll

diff --git a/clang/lib/CodeGen/CGLoopInfo.cpp b/clang/lib/CodeGen/CGLoopInfo.cpp
index 499e2287ae84a..93857f9aaab8d 100644
--- a/clang/lib/CodeGen/CGLoopInfo.cpp
+++ b/clang/lib/CodeGen/CGLoopInfo.cpp
@@ -209,10 +209,8 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
     SmallVector<Metadata *, 4> NewLoopProperties;
     if (Enabled == false) {
       NewLoopProperties.append(LoopProperties.begin(), LoopProperties.end());
-      NewLoopProperties.push_back(
-          MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                            ConstantAsMetadata::get(ConstantInt::get(
-                                llvm::Type::getInt1Ty(Ctx), 0))}));
+      NewLoopProperties.push_back(MDNode::get(
+          Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
       LoopProperties = NewLoopProperties;
     }
     return createUnrollAndJamMetadata(Attrs, LoopProperties, HasUserTransforms);
@@ -279,10 +277,10 @@ clang::CodeGen::LoopInfo::createLoopVectorizeMetadata(
       (Attrs.VectorizeScalable == LoopAttributes::Disable &&
        Attrs.VectorizeWidth != 1)) {
     VectorizeEnabled = Attrs.VectorizeEnable != LoopAttributes::Disable;
-    Args.push_back(
-        MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"),
-                          ConstantAsMetadata::get(ConstantInt::get(
-                              llvm::Type::getInt1Ty(Ctx), VectorizeEnabled))}));
+    Args.push_back(MDNode::get(
+        Ctx, {MDString::get(Ctx, VectorizeEnabled
+                                     ? "llvm.loop.vectorize.enable"
+                                     : "llvm.loop.vectorize.disable")}));
   }
 
   // Apply all loop properties to the vectorized loop.
diff --git a/clang/test/CodeGenCXX/pragma-followup_inner.cpp b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
index 2cf2dba5b913d..9ab8547338832 100644
--- a/clang/test/CodeGenCXX/pragma-followup_inner.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_inner.cpp
@@ -25,7 +25,7 @@ extern "C" void followup_inner(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6]] = !{!"llvm.loop.distribute.followup_all", ![[PARALLEL_ACCESSES_4]], ![[VECTORIZE_8:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[OUTERLOOP_9]] = distinct !{![[OUTERLOOP_9]], [[MP:![0-9]+]], ![[UNROLLANDJAM_COUNT_10:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPINNER_11:[0-9]+]]}
 // CHECK-DAG: ![[UNROLLANDJAM_COUNT_10]] = !{!"llvm.loop.unroll_and_jam.count", i32 4}
diff --git a/clang/test/CodeGenCXX/pragma-followup_outer.cpp b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
index 2e907fb814b3d..c4f8cde943be7 100644
--- a/clang/test/CodeGenCXX/pragma-followup_outer.cpp
+++ b/clang/test/CodeGenCXX/pragma-followup_outer.cpp
@@ -22,7 +22,7 @@ extern "C" void followup_outer(int n, int *x) {
 // CHECK-DAG: ![[DISTRIBUTE_5:[0-9]+]] = !{!"llvm.loop.distribute.enable"}
 
 // CHECK-DAG: ![[DISTRIBUTE_FOLLOWUP_6:[0-9]+]] = !{!"llvm.loop.distribute.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[VECTORIZE_8:[0-9]+]], ![[VECTORIZE_FOLLOWUP_9:[0-9]+]]}
-// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_8:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-DAG: ![[VECTORIZE_FOLLOWUP_9:[0-9]+]] = !{!"llvm.loop.vectorize.followup_all", [[MP]], ![[PARALLEL_ACCESSES_4:[0-9]+]], ![[ISVECTORIZED_11:[0-9]+]], ![[UNROLLANDJAM_12:[0-9]+]], ![[UNROLLANDJAM_FOLLOWUPOUTER_13:[0-9]+]]}
 // CHECK-DAG: ![[ISVECTORIZED_11:[0-9]+]] = !{!"llvm.loop.isvectorized"}
diff --git a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
index 37e29776264b6..afab64f4c6867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-pr27643.cpp
@@ -43,7 +43,7 @@ void loop4(int *List, int Length) {
 // CHECK: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP:![0-9]+]], ![[VEC_WIDTH_1:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE:.*]]}
 // CHECK: ![[VEC_WIDTH_1]] = !{!"llvm.loop.vectorize.width", i32 1}
 // CHECK: ![[FIXED_WIDTH]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK: ![[LOOP2]] = distinct !{![[LOOP2]], [[MP]], ![[VEC_WIDTH_2:.*]], ![[FIXED_WIDTH:.*]], ![[VEC_ENABLE]]}
 // CHECK: ![[VEC_WIDTH_2]] = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
index 8a25ed8de6239..421110fd9f867 100644
--- a/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-predicate.cpp
@@ -103,7 +103,7 @@ void test9(int *List, int Length) {
 
 // CHECK:      ![[LOOP0]] = distinct !{![[LOOP0]], [[MP:![0-9]+]], [[GEN3:![0-9]+]]}
 // CHECK:      [[MP]] = !{!"llvm.loop.mustprogress"}
-// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: [[GEN3]] = !{!"llvm.loop.vectorize.enable"}
 
 // CHECK-NEXT: ![[LOOP1]] = distinct !{![[LOOP1]], [[MP]], [[GEN6:![0-9]+]], [[GEN3]]}
 // CHECK-NEXT: [[GEN6]] = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/clang/test/CodeGenCXX/pragma-loop-safety.cpp b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
index db1b6769e0514..e2a7a124890a6 100644
--- a/clang/test/CodeGenCXX/pragma-loop-safety.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop-safety.cpp
@@ -51,7 +51,7 @@ void interleave_test(int *List, int Length) {
 // CHECK: ![[PARALLEL_ACCESSES_7]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_2]]}
 // CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 // CHECK: ![[INTERLEAVE_1]] = !{!"llvm.loop.interleave.count", i32 1}
-// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[INTENABLE_1]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: ![[ACCESS_GROUP_8]] = distinct !{}
 // CHECK: ![[LOOP2_HINTS]] = distinct !{![[LOOP2_HINTS]], [[MP]], ![[PARALLEL_ACCESSES_11:[0-9]+]], ![[UNROLL_DISABLE]], ![[WIDTH_1:[0-9]+]], ![[INTENABLE_1]]}
 // CHECK: ![[PARALLEL_ACCESSES_11]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP_8]]}
diff --git a/clang/test/CodeGenCXX/pragma-loop.cpp b/clang/test/CodeGenCXX/pragma-loop.cpp
index f9106725ad99d..68ab7b51ea833 100644
--- a/clang/test/CodeGenCXX/pragma-loop.cpp
+++ b/clang/test/CodeGenCXX/pragma-loop.cpp
@@ -256,7 +256,7 @@ void for_test_interleave_vectorize_disable_unroll(int *List, int Length) {
 // CHECK-DAG: ![[INTERLEAVE_10:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 10}
 // CHECK-DAG: ![[INTERLEAVE_16:[0-9]+]] = !{!"llvm.loop.interleave.count", i32 16}
 
-// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: ![[VECTORIZE_ENABLE:[0-9]+]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: ![[FIXED_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: ![[SCALABLE_VEC:[0-9]+]] = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 // CHECK-DAG: ![[WIDTH_1:[0-9]+]] = !{!"llvm.loop.vectorize.width", i32 1}
diff --git a/clang/test/OpenMP/for_simd_codegen.cpp b/clang/test/OpenMP/for_simd_codegen.cpp
index 1ac95e8943640..ae6161d03978d 100644
--- a/clang/test/OpenMP/for_simd_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_codegen.cpp
@@ -802,10 +802,10 @@ void parallel_simd(float *a) {
 }
 // TERM_DEBUG: !{{[0-9]+}} = !DILocation(line: [[@LINE-11]],
 // TERM_DEBUG-NOT: line: 0,
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // CHECK-DAG: ![[SIMD_LOOP]] = distinct !{![[SIMD_LOOP]], {{.*}}![[VECT_LOOP:[^,]+]]}
-// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !"llvm.loop.vectorize.enable", i1 false
+// CHECK-DAG: ![[VECT_LOOP]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !"llvm.loop.vectorize.disable"
 // OMP50-DAG: ![[NOSIMD_LOOP]] = distinct !{![[NOSIMD_LOOP]], {{.*}}![[NOVECT_LOOP:[^,]+]]}
-// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVECT_LOOP]] = !{!"llvm.loop.vectorize.disable"}
 #endif // HEADER
diff --git a/clang/test/OpenMP/for_simd_scan_codegen.cpp b/clang/test/OpenMP/for_simd_scan_codegen.cpp
index 829f2656042fb..9ab62f8a682b2 100644
--- a/clang/test/OpenMP/for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/for_simd_scan_codegen.cpp
@@ -300,5 +300,5 @@ void baz(int n) {
 }
 
 #endif
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
diff --git a/clang/test/OpenMP/irbuilder_safelen.cpp b/clang/test/OpenMP/irbuilder_safelen.cpp
index 6a4266f9646a7..3090eafa6a508 100644
--- a/clang/test/OpenMP/irbuilder_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
index 210978fc6069c..215b37e4bbb5f 100644
--- a/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
+++ b/clang/test/OpenMP/irbuilder_safelen_order_concurrent.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simd.cpp b/clang/test/OpenMP/irbuilder_simd.cpp
index 9c36e4ab97182..03f92d8117d45 100644
--- a/clang/test/OpenMP/irbuilder_simd.cpp
+++ b/clang/test/OpenMP/irbuilder_simd.cpp
@@ -65,7 +65,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: ![[META3:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META4]]  = distinct !{![[META4]], ![[META5:[0-9]+]], ![[META6:[0-9]+]]}
 // CHECK-NEXT: ![[META5]]  = !{!"llvm.loop.parallel_accesses", ![[META3]]}
-// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-NEXT: ![[META6]]  = !{!"llvm.loop.vectorize.enable"}
 // CHECK-NEXT: ![[META7:[0-9]+]] = distinct !{}
 // CHECK-NEXT: ![[META8]]  = distinct !{![[META8]], ![[META9:[0-9]+]], ![[META6]]}
 // CHECK-NEXT: ![[META9]]  = !{!"llvm.loop.parallel_accesses", ![[META7]]}
diff --git a/clang/test/OpenMP/irbuilder_simd_aligned.cpp b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
index b5ce43c8ea6a9..bb121036b4f0c 100644
--- a/clang/test/OpenMP/irbuilder_simd_aligned.cpp
+++ b/clang/test/OpenMP/irbuilder_simd_aligned.cpp
@@ -293,7 +293,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP4]] = distinct !{}
 // CHECK: [[LOOP5]] = distinct !{[[LOOP5]], [[META6:![0-9]+]], [[META7:![0-9]+]]}
 // CHECK: [[META6]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP4]]}
-// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META7]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[ACC_GRP8]] = distinct !{}
 // CHECK: [[LOOP9]] = distinct !{[[LOOP9]], [[META10:![0-9]+]], [[META7]]}
 // CHECK: [[META10]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP8]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen.cpp b/clang/test/OpenMP/irbuilder_simdlen.cpp
index f6c26923851d0..9a7e8d5332ae0 100644
--- a/clang/test/OpenMP/irbuilder_simdlen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen.cpp
@@ -252,7 +252,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[ACC_GRP2]] = distinct !{}
 // CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]], [[META5:![0-9]+]], [[META6:![0-9]+]]}
 // CHECK: [[META4]] = !{!"llvm.loop.parallel_accesses", [[ACC_GRP2]]}
-// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META5]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META6]] = !{!"llvm.loop.vectorize.width", i32 3}
 // CHECK: [[ACC_GRP7]] = distinct !{}
 // CHECK: [[LOOP8]] = distinct !{[[LOOP8]], [[META9:![0-9]+]], [[META5]]}
diff --git a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
index e46d5dde841b3..bb35f9341ded4 100644
--- a/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
+++ b/clang/test/OpenMP/irbuilder_simdlen_safelen.cpp
@@ -250,7 +250,7 @@ void simple(float *a, float *b, int *c) {
 // CHECK: [[META0:![0-9]+]] = !{i32 7, !"openmp", i32 45}
 // CHECK: [[META1:![0-9]+]] = !{!"{{.*}}clang version {{.*}}"}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
-// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[META4]] = !{!"llvm.loop.vectorize.width", i32 2}
 // CHECK: [[ACC_GRP5]] = distinct !{}
 // CHECK: [[LOOP6]] = distinct !{[[LOOP6]], [[META7:![0-9]+]], [[META3]]}
diff --git a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
index d78d193e2edd1..f1eae642b613c 100644
--- a/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/master_taskloop_simd_codegen.cpp
@@ -238,10 +238,10 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// OMP50-DAG: !{!"llvm.loop.vectorize.disable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK-DAG: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/parallel_for_simd_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
index 40af7069bbf7d..cc8cab706975a 100644
--- a/clang/test/OpenMP/parallel_for_simd_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_codegen.cpp
@@ -798,16 +798,16 @@ for (int i = 0; i < 10; ++i);
 // OMP50: omp.inner.for.body:
 // OMP50: %{{[0-9]+}} = atomicrmw add ptr %{{[0-9]+}}, i32 1 monotonic, align 4, !llvm.access.group !{{[0-9]+}}
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP45-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP45-DAG: ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
 // OMP50-DAG: ![[VECT]] = distinct !{![[VECT]], ![[PA:.+]], ![[VM:.+]]}
 // OMP50-DAG  ![[PA]] = !{!"llvm.loop.parallel_accesses", !{{.+}}}
-// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// OMP50-DAG: ![[VM]] = !{!"llvm.loop.vectorize.enable"}
 // OMP50-DAG: ![[NOVECT]] = distinct !{![[NOVECT]], ![[NOVM:.+]]}
-// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP50-DAG: ![[NOVM]] = !{!"llvm.loop.vectorize.disable"}
 
 // TERM_DEBUG-LABEL: bar
 int bar() { extern void mayThrow(); mayThrow(); return 0; };
diff --git a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
index cac997753d480..e3810c27163a4 100644
--- a/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
+++ b/clang/test/OpenMP/parallel_for_simd_scan_codegen.cpp
@@ -299,7 +299,7 @@ void baz(int n) {
   }
 }
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 
 #endif
 
diff --git a/clang/test/OpenMP/simd_codegen.cpp b/clang/test/OpenMP/simd_codegen.cpp
index b96e4213e8e0e..ee8d709ba47fe 100644
--- a/clang/test/OpenMP/simd_codegen.cpp
+++ b/clang/test/OpenMP/simd_codegen.cpp
@@ -861,7 +861,7 @@ S8 s8(0);
 
 // TERM_DEBUG-NOT: line: 0,
 // TERM_DEBUG: distinct !DISubprogram(linkageName: "_GLOBAL__sub_I_simd_codegen.cpp",
-// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5-DAG: ![[NOVECT:.+]] = !{!"llvm.loop.vectorize.disable"}
 // OMP5-DAG: ![[DISABLE_VECT]] = distinct !{{.*}}![[NOVECT]]{{[,}]}}
 #endif // HEADER
 
diff --git a/clang/test/OpenMP/simd_metadata.c b/clang/test/OpenMP/simd_metadata.c
index 83bcbf6deefe2..6fc4c401135f2 100644
--- a/clang/test/OpenMP/simd_metadata.c
+++ b/clang/test/OpenMP/simd_metadata.c
@@ -109,7 +109,7 @@ void h3(float *c, float *a, float *b, int size)
 // Metadata for h1:
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_16:![0-9]+]], [[LOOP_VEC_ENABLE:![0-9]+]]}
 // CHECK: [[LOOP_WIDTH_16]] = !{!"llvm.loop.vectorize.width", i32 16}
-// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP_VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 // CHECK: [[LOOP_H1_HEADER:![0-9]+]] = distinct !{[[LOOP_H1_HEADER]], [[LOOP_WIDTH_8:![0-9]+]], [[LOOP_VEC_ENABLE]]}
 // CHECK: [[LOOP_WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 // CHECK: ![[ACCESS_GROUP_7]] = distinct !{}
diff --git a/clang/test/OpenMP/target_simd_codegen.cpp b/clang/test/OpenMP/target_simd_codegen.cpp
index 0c2dde23f6c46..59284c201bb30 100644
--- a/clang/test/OpenMP/target_simd_codegen.cpp
+++ b/clang/test/OpenMP/target_simd_codegen.cpp
@@ -702,9 +702,9 @@ int bar(int n){
 // Store captures in the context.
 // CHECK-DAG:   [[REF_B:%.+]] = load ptr, ptr [[LOCAL_B]],
 
-// OMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP45-NOT: !{!"llvm.loop.vectorize.enable", i1 false}
-// OMP51: !{!"llvm.loop.vectorize.enable", i1 false}
-// TOMP51: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// TOMP45-NOT: !{!"llvm.loop.vectorize.disable"}
+// OMP51: !{!"llvm.loop.vectorize.disable"}
+// TOMP51: !{!"llvm.loop.vectorize.disable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
index 42e8377bd22dd..fd2d0d0c18737 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration.cpp
@@ -455,7 +455,7 @@ int bar(int a){
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2STILi100EE3fooEv", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 [[DEVID]], i32 [[FILEID]], !"_ZN2SCC1Ev", i32 {{[0-9]+}}, i32 0, i32 {{[0-9]+}}}
 
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 #endif
diff --git a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
index b63f0b6caeee2..c169d4a556bf0 100644
--- a/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
+++ b/clang/test/OpenMP/target_teams_distribute_simd_codegen_registration_naming.cpp
@@ -82,6 +82,6 @@ int nested(int a){
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T1L]], i32 0, i32 {{[0-9]+}}}
 // TCHECK-DAG: = !{i32 0, i32 {{-?[0-9]+}}, i32 {{-?[0-9]+}}, !"[[NNAME]]", i32 [[T2L]], i32 0, i32 {{[0-9]+}}}
 
-// CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
-// TCHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
+// TCHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 #endif
diff --git a/clang/test/OpenMP/taskloop_simd_codegen.cpp b/clang/test/OpenMP/taskloop_simd_codegen.cpp
index ab34ff66e5c7a..df2df9f609a57 100644
--- a/clang/test/OpenMP/taskloop_simd_codegen.cpp
+++ b/clang/test/OpenMP/taskloop_simd_codegen.cpp
@@ -208,7 +208,7 @@ struct S {
 // CHECK: br label %{{.*}}!llvm.loop
 // CHECK: ret i32 0
 
-// CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: !{!"llvm.loop.vectorize.enable"}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 4}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 32}
 // CHECK: !{!"llvm.loop.vectorize.width", i32 8}
diff --git a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
index 365f11cfac5c6..688849851aa4d 100644
--- a/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_parallel_for_simd_collapse_codegen.cpp
@@ -46,7 +46,7 @@ int teams_template_struct(void) {
 
 }
 
-// CK4: !{!"llvm.loop.vectorize.enable", i1 true}
+// CK4: !{!"llvm.loop.vectorize.enable"}
 
 #endif // CK1
 
@@ -103,7 +103,7 @@ int main (int argc, char **argv) {
 // discard loop variables not needed here
 
 
-// CK4: !{!"llvm.loop.vectorize.enable", i1 true}
+// CK4: !{!"llvm.loop.vectorize.enable"}
 
 #endif // CK2
 #endif // #ifndef HEADER
diff --git a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
index d5000a5b18a53..c2234296a059f 100644
--- a/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
+++ b/clang/test/OpenMP/teams_distribute_simd_codegen.cpp
@@ -207,7 +207,7 @@ int main (int argc, char **argv) {
 
 
 
-// OMP5_50-DAG: !{!"llvm.loop.vectorize.enable", i1 false}
+// OMP5_50-DAG: !{!"llvm.loop.vectorize.disable"}
 #endif // CK4
 #endif
 // CHECK1-LABEL: define {{[^@]+}}@_Z21teams_argument_globali
diff --git a/clang/test/PCH/delayed-template-with-pragma.cpp b/clang/test/PCH/delayed-template-with-pragma.cpp
index e54388456245a..124d0e5db99e8 100644
--- a/clang/test/PCH/delayed-template-with-pragma.cpp
+++ b/clang/test/PCH/delayed-template-with-pragma.cpp
@@ -27,7 +27,7 @@ void a(T t) {
 // CHECK: [[LOOP1B]] = !{!"llvm.loop.unroll.count", i32 4}
 // CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[LOOP2A:!.*]], [[LOOP2B:!.*]]}
 // CHECK: [[LOOP2A]] = !{!"llvm.loop.parallel_accesses", [[LOOP2C:!.*]]}
-// CHECK: [[LOOP2B]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: [[LOOP2B]] = !{!"llvm.loop.vectorize.enable"}
 // expected-warning at 17 {{unused variable 'zz'}}
 void foo()
 {
diff --git a/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp b/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
index 409570c3f0fa0..d37726b74b48e 100644
--- a/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
+++ b/clang/tools/clang-fuzzer/proto-to-llvm/loop_proto_to_llvm.cpp
@@ -157,12 +157,13 @@ void NestedLoopToString(std::ostream &os, const LoopFunction &x) {
   }
   os << "%i_ct_new = add i64 %inner_ct, 1\n"
      << "%jmp_inner = icmp eq i64 %i_ct_new, %s\n"
-     << "br i1 %jmp_inner, label %outer_loop, label %inner_loop, !llvm.loop !0\n"
+     << "br i1 %jmp_inner, label %outer_loop, label %inner_loop, !llvm.loop "
+        "!0\n"
      << "end:\n"
      << "ret void\n"
      << "}\n"
      << "!0 = distinct !{!0, !1, !2}\n"
-     << "!1 = !{!\"llvm.loop.vectorize.enable\", i1 true}\n"
+     << "!1 = !{!\"llvm.loop.vectorize.enable\"}\n"
      << "!2 = !{!\"llvm.loop.vectorize.width\", i32 " << kArraySize << "}\n";
 }
 void SingleLoopToString(std::ostream &os, const LoopFunction &x) {
@@ -176,12 +177,11 @@ void SingleLoopToString(std::ostream &os, const LoopFunction &x) {
      << "ret void\n"
      << "loop:\n"
      << "%outer_ct = phi i64 [ %ctnew, %loop ], [ 0, %start ]\n"
-     << x.outer_statements()
-     << "%ctnew = add i64 %outer_ct, 1\n"
+     << x.outer_statements() << "%ctnew = add i64 %outer_ct, 1\n"
      << "%j = icmp eq i64 %ctnew, %s\n"
      << "br i1 %j, label %end, label %loop, !llvm.loop !0\n}\n"
      << "!0 = distinct !{!0, !1, !2}\n"
-     << "!1 = !{!\"llvm.loop.vectorize.enable\", i1 true}\n"
+     << "!1 = !{!\"llvm.loop.vectorize.enable\"}\n"
      << "!2 = !{!\"llvm.loop.vectorize.width\", i32 " << kArraySize << "}\n";
 }
 std::ostream &operator<<(std::ostream &os, const LoopFunction &x) {
diff --git a/flang/test/Integration/ivdep.f90 b/flang/test/Integration/ivdep.f90
index 9882fe148262f..a53d46ed446de 100644
--- a/flang/test/Integration/ivdep.f90
+++ b/flang/test/Integration/ivdep.f90
@@ -120,5 +120,5 @@ end subroutine ivdep_test4
 ! CHECK: ![[PARALLEL_ACCESSES2]] = !{!"llvm.loop.parallel_accesses", [[DISTRINCT2]]}
 ! CHECK: [[DISTRINCT3:.*]] = distinct !{}
 ! CHECK: ![[ANNOTATION3]] = distinct !{![[ANNOTATION3]], ![[VECTORIZE:.*]], ![[PARALLEL_ACCESSES3:.*]]}
-! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
 ! CHECK: ![[PARALLEL_ACCESSES3]] = !{!"llvm.loop.parallel_accesses", [[DISTRINCT3]]}
diff --git a/flang/test/Integration/vector-always.f90 b/flang/test/Integration/vector-always.f90
index 1d8aad97bde70..5634a6a13b63f 100644
--- a/flang/test/Integration/vector-always.f90
+++ b/flang/test/Integration/vector-always.f90
@@ -25,6 +25,6 @@ subroutine no_vector
 end subroutine no_vector
 
 ! CHECK: ![[ANNOTATION]] = distinct !{![[ANNOTATION]], ![[VECTORIZE:.*]]}
-! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+! CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
 ! CHECK: ![[ANNOTATION2]] = distinct !{![[ANNOTATION2]], ![[VECTORIZE2:.*]]}
-! CHECK: ![[VECTORIZE2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+! CHECK: ![[VECTORIZE2]] = !{!"llvm.loop.vectorize.disable"}
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index 461177d24bfa5..f11e9a3417a4d 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -8008,14 +8008,12 @@ then the interleave count will be determined automatically.
 
 #### '`llvm.loop.vectorize.enable`' Metadata
 
-This metadata selectively enables or disables vectorization for the loop. The
-first operand is the string `llvm.loop.vectorize.enable` and the second operand
-is a bit. If the bit operand value is 1 vectorization is enabled. A value of
-0 disables vectorization:
+This metadata selectively enables or disables vectorization for the loop. Each
+node has a single operand containing the name string:
 
 ```llvm
-!0 = !{!"llvm.loop.vectorize.enable", i1 0}
-!1 = !{!"llvm.loop.vectorize.enable", i1 1}
+!0 = !{!"llvm.loop.vectorize.enable"}
+!1 = !{!"llvm.loop.vectorize.disable"}
 ```
 
 #### '`llvm.loop.vectorize.predicate.enable`' Metadata
diff --git a/llvm/docs/TransformMetadata.rst b/llvm/docs/TransformMetadata.rst
index 700761a383e9a..d92f25641f149 100644
--- a/llvm/docs/TransformMetadata.rst
+++ b/llvm/docs/TransformMetadata.rst
@@ -79,7 +79,7 @@ vectorized, for instance being unrolled.
       br i1 %exitcond, label %for.exit, label %for.header, !llvm.loop !0
     ...
     !0 = distinct !{!0, !1, !2}
-    !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+    !1 = !{!"llvm.loop.vectorize.enable"}
     !2 = !{!"llvm.loop.disable_nonforced"}
 
 After a transformation is applied, follow-up attributes are set on the
@@ -97,7 +97,7 @@ then unrolled.
 .. code-block:: llvm
 
     !0 = distinct !{!0, !1, !2, !3}
-    !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+    !1 = !{!"llvm.loop.vectorize.enable"}
     !2 = !{!"llvm.loop.disable_nonforced"}
     !3 = !{!"llvm.loop.vectorize.followup_vectorized", !{"llvm.loop.unroll.enable"}}
 
@@ -130,7 +130,7 @@ Loop Vectorization and Interleaving
 
 Loop vectorization and interleaving is interpreted as a single
 transformation. It is interpreted as forced if
-``!{"llvm.loop.vectorize.enable", i1 true}`` is set.
+``!{"llvm.loop.vectorize.enable"}`` is set.
 
 Assuming the pre-vectorization loop is
 
diff --git a/llvm/include/llvm/IR/AutoUpgrade.h b/llvm/include/llvm/IR/AutoUpgrade.h
index 423fea4205a80..12aeb2793f705 100644
--- a/llvm/include/llvm/IR/AutoUpgrade.h
+++ b/llvm/include/llvm/IR/AutoUpgrade.h
@@ -112,12 +112,14 @@ namespace llvm {
 
   /// Check whether a string looks like an old loop attachment tag.
   inline bool mayBeOldLoopAttachmentTag(StringRef Name) {
-    // "llvm.loop.distribute.enable" is intentionally included: the current
-    // single-operand form shares the tag with the removed two-operand form
-    // (!{!"llvm.loop.distribute.enable", i1 X}), so we can only decide by
-    // inspecting the operands, which happens in upgradeLoopArgument().
+    // "llvm.loop.distribute.enable" and "llvm.loop.vectorize.enable" are
+    // intentionally included: the current single-operand form shares the tag
+    // with the removed two-operand form (!{!"...", i1 X}), so we can only
+    // decide by inspecting the operands, which happens in
+    // upgradeLoopArgument().
     return Name.starts_with("llvm.vectorizer.") ||
-           Name == "llvm.loop.distribute.enable";
+           Name == "llvm.loop.distribute.enable" ||
+           Name == "llvm.loop.vectorize.enable";
   }
 
   /// Upgrade the loop attachment metadata node.
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index e19ed110adf1e..aa7af1318016c 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -7462,10 +7462,8 @@ void OpenMPIRBuilder::applySimd(CanonicalLoopInfo *CanonicalLoop,
 
   // Use the above access group metadata to create loop level
   // metadata, which should be distinct for each loop.
-  ConstantAsMetadata *BoolConst =
-      ConstantAsMetadata::get(ConstantInt::getTrue(Type::getInt1Ty(Ctx)));
-  LoopMDList.push_back(MDNode::get(
-      Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable"), BoolConst}));
+  LoopMDList.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.enable")}));
 
   if (Simdlen || Safelen) {
     // If both simdlen and safelen clauses are specified, the value of the
diff --git a/llvm/lib/IR/AutoUpgrade.cpp b/llvm/lib/IR/AutoUpgrade.cpp
index 926e18924956f..c889a19500d3b 100644
--- a/llvm/lib/IR/AutoUpgrade.cpp
+++ b/llvm/lib/IR/AutoUpgrade.cpp
@@ -6976,6 +6976,27 @@ static bool isOldDistributeEnable(const MDTuple *T) {
   return mdconst::hasa<ConstantInt>(T->getOperand(1));
 }
 
+// Old two-operand form: !{!"llvm.loop.vectorize.enable", i1 X}. The new
+// single-operand form uses "llvm.loop.vectorize.enable" for X = true and
+// "llvm.loop.vectorize.disable" for X = false.
+static bool isOldVectorizeEnable(const MDTuple *T) {
+  if (T->getNumOperands() != 2)
+    return false;
+  auto *Tag = dyn_cast_or_null<MDString>(T->getOperand(0));
+  if (!Tag || Tag->getString() != "llvm.loop.vectorize.enable")
+    return false;
+  return mdconst::hasa<ConstantInt>(T->getOperand(1));
+}
+
+// Build the single-operand vectorize enable/disable node that replaces a
+// boolean operand: nonzero -> enable, zero -> disable.
+static Metadata *makeVectorizeEnableNode(LLVMContext &C, const MDOperand &Op) {
+  bool Enable = !mdconst::extract<ConstantInt>(Op)->isZero();
+  return MDTuple::get(
+      C, {MDString::get(C, Enable ? "llvm.loop.vectorize.enable"
+                                  : "llvm.loop.vectorize.disable")});
+}
+
 static bool isOldLoopArgument(Metadata *MD) {
   auto *T = dyn_cast_or_null<MDTuple>(MD);
   if (!T)
@@ -6987,7 +7008,7 @@ static bool isOldLoopArgument(Metadata *MD) {
     return false;
   if (S->getString().starts_with("llvm.vectorizer."))
     return true;
-  return isOldDistributeEnable(T);
+  return isOldDistributeEnable(T) || isOldVectorizeEnable(T);
 }
 
 static MDString *upgradeLoopTag(LLVMContext &C, StringRef OldTag) {
@@ -7022,13 +7043,26 @@ static Metadata *upgradeLoopArgument(Metadata *MD) {
                                     : "llvm.loop.distribute.disable")});
   }
 
+  // Rewrite the modern two-operand vectorize.enable form to the single-operand
+  // enable/disable pair.
+  if (isOldVectorizeEnable(T))
+    return makeVectorizeEnableNode(C, T->getOperand(1));
+
   if (!OldTag->getString().starts_with("llvm.vectorizer."))
     return MD;
 
   // This has an old tag.  Upgrade it.
+  MDString *NewTag = upgradeLoopTag(C, OldTag->getString());
+
+  // The legacy !{!"llvm.vectorizer.enable", i1 X} maps onto the single-operand
+  // vectorize.enable/disable pair, not a two-operand enable node.
+  if (NewTag->getString() == "llvm.loop.vectorize.enable" &&
+      T->getNumOperands() == 2 && mdconst::hasa<ConstantInt>(T->getOperand(1)))
+    return makeVectorizeEnableNode(C, T->getOperand(1));
+
   SmallVector<Metadata *, 8> Ops;
   Ops.reserve(T->getNumOperands());
-  Ops.push_back(upgradeLoopTag(C, OldTag->getString()));
+  Ops.push_back(NewTag);
   for (unsigned I = 1, E = T->getNumOperands(); I != E; ++I)
     Ops.push_back(T->getOperand(I));
 
@@ -7043,14 +7077,14 @@ MDNode *llvm::upgradeInstructionLoopAttachment(MDNode &N) {
   if (none_of(T->operands(), isOldLoopArgument))
     return &N;
 
-  // Fix the old two-operand llvm.loop.distribute.enable nodes in place: the
-  // Verifier rejects any MDNode carrying the distribute tag with more than one
-  // operand, so a leftover reference (from the distinct loop-ID) would still
-  // trigger a diagnostic. In-place mutation is safe on distinct MDNodes.
+  // Fix the old two-operand distribute/vectorize.enable nodes in place: the
+  // Verifier rejects any MDNode carrying those tags with more than one operand,
+  // so a leftover reference (from the distinct loop-ID) would still trigger a
+  // diagnostic. In-place mutation is safe on distinct MDNodes.
   if (T->isDistinct()) {
     for (unsigned I = 0, E = T->getNumOperands(); I < E; ++I) {
       auto *OpT = dyn_cast_or_null<MDTuple>(T->getOperand(I));
-      if (OpT && isOldDistributeEnable(OpT))
+      if (OpT && (isOldDistributeEnable(OpT) || isOldVectorizeEnable(OpT)))
         T->replaceOperandWith(I, upgradeLoopArgument(OpT));
     }
     if (none_of(T->operands(), isOldLoopArgument))
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 41c45be8de882..0edb99e17a6c1 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -1016,6 +1016,14 @@ void Verifier::visitMDNode(const MDNode &BaseMD,
             "Expected one operand for llvm.loop.distribute metadata",
             CurrentMD);
 
+    // Enforce the single-operand form of llvm.loop.vectorize.enable metadata.
+    if (CurrentMD->getNumOperands() > 0 &&
+        (CurrentMD->getOperand(0).equalsStr("llvm.loop.vectorize.enable") ||
+         CurrentMD->getOperand(0).equalsStr("llvm.loop.vectorize.disable")))
+      Check(CurrentMD->getNumOperands() == 1,
+            "Expected one operand for llvm.loop.vectorize.enable metadata",
+            CurrentMD);
+
     // Check these last, so we diagnose problems in operands first.
     Check(!CurrentMD->isTemporary(), "Expected no forward declarations!",
           CurrentMD);
diff --git a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
index 58ebd0e5ee937..82bfd1873f293 100644
--- a/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
+++ b/llvm/lib/Transforms/Utils/LoopConstrainer.cpp
@@ -447,8 +447,7 @@ static void DisableAllLoopOptsOnLoop(Loop &L) {
   Metadata *FalseVal =
       ConstantAsMetadata::get(ConstantInt::get(Type::getInt1Ty(Context), 0));
   MDNode *DisableVectorize = MDNode::get(
-      Context,
-      {MDString::get(Context, "llvm.loop.vectorize.enable"), FalseVal});
+      Context, {MDString::get(Context, "llvm.loop.vectorize.disable")});
   MDNode *DisableLICMVersioning = MDNode::get(
       Context, {MDString::get(Context, "llvm.loop.licm_versioning.disable")});
   MDNode *DisableDistribution = MDNode::get(
diff --git a/llvm/lib/Transforms/Utils/LoopUtils.cpp b/llvm/lib/Transforms/Utils/LoopUtils.cpp
index 591407770d73a..5d16a85427857 100644
--- a/llvm/lib/Transforms/Utils/LoopUtils.cpp
+++ b/llvm/lib/Transforms/Utils/LoopUtils.cpp
@@ -405,12 +405,12 @@ TransformationMode llvm::hasUnrollAndJamTransformation(const Loop *L) {
 }
 
 TransformationMode llvm::hasVectorizeTransformation(const Loop *L) {
+  if (getBooleanLoopAttribute(L, "llvm.loop.vectorize.disable"))
+    return TM_SuppressedByUser;
+
   std::optional<bool> Enable =
       getOptionalBoolLoopAttribute(L, "llvm.loop.vectorize.enable");
 
-  if (Enable == false)
-    return TM_SuppressedByUser;
-
   std::optional<ElementCount> VectorizeWidth =
       getOptionalElementCountLoopAttribute(L);
   std::optional<int> InterleaveCount =
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index 8d875b2b6e492..9086880599231 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -287,6 +287,14 @@ void LoopVectorizeHints::getHintsFromMetadata() {
 
     // Check if the hint starts with the loop metadata prefix.
     StringRef Name = S->getString();
+    // The single-operand enable/disable pair carries no argument.
+    if (Args.empty()) {
+      if (Name == "llvm.loop.vectorize.enable")
+        Force.Value = FK_Enabled;
+      else if (Name == "llvm.loop.vectorize.disable")
+        Force.Value = FK_Disabled;
+      continue;
+    }
     if (Args.size() == 1)
       setHint(Name, Args[0]);
   }
diff --git a/llvm/test/Bitcode/upgrade-loop-metadata.ll b/llvm/test/Bitcode/upgrade-loop-metadata.ll
index 8dee907578760..9cd46acd26e84 100644
--- a/llvm/test/Bitcode/upgrade-loop-metadata.ll
+++ b/llvm/test/Bitcode/upgrade-loop-metadata.ll
@@ -29,7 +29,7 @@ for.end:                                          ; preds = %for.cond
 
 ; CHECK: !{!"llvm.loop.interleave.count", i32 4}
 ; CHECK: !{!"llvm.loop.vectorize.width", i32 8}
-; CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !{!"llvm.loop.vectorize.enable"}
 
 !0 = metadata !{metadata !"clang version 3.5.0 (trunk 211528)"}
 !1 = metadata !{metadata !1, metadata !2, metadata !3, metadata !4, metadata !4}
diff --git a/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
new file mode 100644
index 0000000000000..b757b787ce51c
--- /dev/null
+++ b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll
@@ -0,0 +1,37 @@
+; Test that older bitcode carrying the two-operand form of
+; "llvm.loop.vectorize.enable" is auto-upgraded to the single-operand
+; enable/disable pair on load.
+;
+; RUN: llvm-dis < %s.bc | FileCheck %s
+; RUN: verify-uselistorder < %s.bc
+
+define void @enable_true() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+
+define void @enable_false() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !2
+exit:
+  ret void
+}
+
+; i1 true  -> single-operand enable.
+; i1 false -> disable.
+; CHECK: !{!"llvm.loop.vectorize.enable"}
+; CHECK: !{!"llvm.loop.vectorize.disable"}
+; The old two-operand nodes must be gone from the module.
+; CHECK-NOT: llvm.loop.vectorize.enable", i1
+; CHECK-NOT: llvm.loop.vectorize.disable", i1
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = distinct !{!2, !3}
+!3 = !{!"llvm.loop.vectorize.enable", i1 false}
diff --git a/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc b/llvm/test/Bitcode/upgrade-loop-vectorize-enable.ll.bc
new file mode 100644
index 0000000000000000000000000000000000000000..e51d9ed57170e2593ed01c29f2ca45bc1fc90b2d
GIT binary patch
literal 2328
zcmbtWZ%iBK89!t2ImNtZ5*qAqcl8AtA&rom3BiZ$$@VqO)l$t(GmUDUz{Yl<Vu>-1
zF?6l$oW~`tRk;<4{2`S<ouaPV2d6?Ck=PP*lseR9EUKy~L(DFiB at +!UkSS85Y0sHz
z>W6J#cGA6f=ezfLpXc}IdF|kDt+v-;SQUn09u3>{!pncW_WC!|KW at l2J*O~MVT=aD
zbQKs at UBO^*!5ZPjV~y|Fvz5=Qv_|e-MO^c2m0C|%zU7H(PF6VHY>YS6$Fl}+b3n^p
zsq=cZ at ATewjB9H5y>0TwonIzv4?9msHG1Ysg*U6NZQ^WQ<JwOZMq}J*JmWfJcz?#$
z6^v>0#&P1x_l#bh{&{5d&UXisOk-E_SuNEW%3gXq$viins5BnOu*(L>4RSi{uhf3z
zxOiK88iwT1oXh#tz9alXmBV;MtvRG<wyP?xVH;|>1ndxV3KgTD{qFZYyW3a)x&PXO
zwT>H4K9v5n&E8n?-27|l${#G(ZWae#zwwi>`0?gPZ+-TK(_L}0R-XDljwo|5t^V%&
zYD{zeVApql>Bejx!RAH^EZU;o!`w)xH2T_8$;7 at RKU}EfU#&@C*mWI-J*~j7Hg%Mo
zFOdE^h4m6|ic1Mc6d#S?!>J+721b0PMY~bdrg6}78ER>Y^s`mv&xD4Y(wgP138gg`
zNrVFUYifMdDFy=gSU?$$hyjDB&()&MBr34PIhI%}p^^)=Y3Hd`hP=yAb8gztUN+AP
z=4GCnR$68SOI&HrmFmL1_-p-mN52?^F)9X<VxSifH{;>QRS<S0D)y+wA)OdZiKD4l
z%fk|Sm`9Ipp-ool$xe~$Q)I3 at uDHoXhMbR4i*D5yNuJB{T&g_INTvzibV+HRkdDEk
zPcE82SDMq3=92+2&@Bc7LZ}~)^ok*!7 at 8FIOD_=*F=E?=){|(>MidNa6RI#YQ!5#I
z&Q9N9$dwqqk|);-)ZZ1fU-zOZ&39fE%sE%nw}teIBN053dhyh5l8_OgU!@n_<b8%(
z%}{SBXgOhqD!El>ekGW4k~z&=0gxwimUZ5I2_B`y$4^0};t1rU#>blRU at G>?nig%|
zLPZzaVH-1d?DTq}{%(O<g#?^>b6T)Y3k`9}l;F9XWSPluSqW;i<ebw(Y8<i}0sy;3
z{a+T)W-swoUR5#>MV2TUh<~`yIW1aa5j^hXRS)uL-Gv^rjbCoM$+;MHw@|;yP#X+2
z&rrbNedZaz;XU&#Z=F$c69Sj(o(1-y*fGe{DUJk0{ilCJTP#t?qX#zP!6d9ADU`z>
z;806X3Zx<<99jk{V{to|mjF5|FqN~-Pg09hRNhXnW=MZiABzeyzXSv+vOFVP(BZ>Q
zv1d~JWfbOKpHNs5l4U|NE%TiqryL-raWYRk+%%-Rl%Y<?>inn9vS`adY~@v3E at D>>
zg^GFAhJh$7L$UAMgm6DTsuuO956wts(BZFnQw}gxa!J9GQ(6Gb2}#pcvjEZsMNN@!
z#OQ4W at VIbY1qy^Xz(w2s6FM;v#m7FtN1K7N%(UH<R+?ue(~}D8tkgLHx%ICYsCx<;
z&|3fiGc_{%?(yw%5R6#6C7Wf36=Fh}DY-__|8O%tmil_$-Z=w_39 at b{`A%>jsCQjT
z{0J?^s9d3bV~Sje(RT{eN}*nc4$`yJAhSg<THSB}JxZc=c5O-sH;dr_#21HbV(@WQ
z$cc}D&R?d$5s>O{g@(@sScQY&(|qSL|AQHY1 at x2RJJUdoYqNJx3FJCM<`tCRG!jKc
zEh?1-xy7QLB-(W$Y}9~Z2cRdtOFIqxC5?Wc*PFd<h(R;$>gqZWb4(_`p3kZ|$9TeH
zi@<k>;2azb|Dtv9{P{p@*w+_29~>C-wffGvJ%c`Y3CC}sU;PxW)o%vCo+f;x+m-->
zD7lu%e=&aMBmz|zd&^@Ut+~*ie{Yu1cB%TIivSZ<C*)8OnX)QTZbvh`6<BLAt$?F5
z6hzLs6~~Dp2HqzMpuke~j*S2<6#}T3o-~L-aQ7t4<p8$?bQPtwrGLpy-iy)z(TbbC
zvqy^C-i9`S1ufdPp<NhZx{avTMUrArcBO3az@!*7h*h6(3BCc^$X%XmNiW1Gx$$Od
zpxZ3K7z0kp1qHPjBlpyJkSy7VZ7q6aBkrf8-Qo}{hPq3D%#2{25G+ at DPVRW%4#AR^
zE?$<qqO22WUwes7)od`&+_RIbF$%-}4Eu!&FvfEhP}<dUIg^3~8fH?0tpam5 at A2BG
z9xt)*tc3GG+|kWw=2Z;LZH>gV>AbDH&)3^3m!g!-e*C5Qdzk%|Y$bfF at L=Wr{b%F0
z{qR^m{&P&v{}pq{u$P+=X%EM-ZKW>YEU)!*_ at 8kY?)EDuzL9^}(^pR at c)|DWt=~O3
z>@&8tvaRgdfsny#^7=VXyN@%r_qCf?KihGlql5Fc_j&w2qsQ;>JKp~NzBbPZS-i(W
nzTwa@$3Up>yw`W^Lf~xB?e!g%6 at B#IiaxrxOzYraIiG(4+2lin

literal 0
HcmV?d00001

diff --git a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
index 3a360814e361e..e6f2c53e8fa6e 100644
--- a/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
+++ b/llvm/test/Transforms/IRCE/add-metadata-pre-post-loops.ll
@@ -74,7 +74,7 @@ attributes #0 = { alwaysinline }
 !1 = !{!"branch_weights", i32 64, i32 4}
 !2 = distinct !{!2, !3, !4, !5, !6}
 !3 = !{!"llvm.loop.unroll.disable"}
-!4 = !{!"llvm.loop.vectorize.enable", i1 false}
+!4 = !{!"llvm.loop.vectorize.disable"}
 !5 = !{!"llvm.loop.licm_versioning.disable"}
 !6 = !{!"llvm.loop.distribute.disable"}
 !7 = !{}
diff --git a/llvm/test/Transforms/IRCE/correct-loop-info.ll b/llvm/test/Transforms/IRCE/correct-loop-info.ll
index bc5987ebba2e4..f1cfd0e518de9 100644
--- a/llvm/test/Transforms/IRCE/correct-loop-info.ll
+++ b/llvm/test/Transforms/IRCE/correct-loop-info.ll
@@ -176,7 +176,7 @@ declare void @pluto()
 
 !0 = distinct !{!0, !1, !2, !3, !4}
 !1 = !{!"llvm.loop.unroll.disable"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 false}
+!2 = !{!"llvm.loop.vectorize.disable"}
 !3 = !{!"llvm.loop.licm_versioning.disable"}
 !4 = !{!"llvm.loop.distribute.disable"}
 !5 = !{}
diff --git a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
index 61cfef6f1492a..027a84d546462 100644
--- a/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
+++ b/llvm/test/Transforms/IRCE/loop-guarded-bounds.ll
@@ -128,7 +128,7 @@ exit:                                       ; preds = %guarded, %loop_, %prehead
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META2]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
diff --git a/llvm/test/Transforms/IRCE/pre_post_loops.ll b/llvm/test/Transforms/IRCE/pre_post_loops.ll
index 2807adba35940..ec67fc7ceb7f9 100644
--- a/llvm/test/Transforms/IRCE/pre_post_loops.ll
+++ b/llvm/test/Transforms/IRCE/pre_post_loops.ll
@@ -161,7 +161,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
index 2e21bb5ad0bd0..c8b77e7ddcecd 100644
--- a/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
+++ b/llvm/test/Transforms/IRCE/range_intersect_miscompile.ll
@@ -488,7 +488,7 @@ deopt:                                          ; preds = %range_check_block
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META2]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META3]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META5]] = !{}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
index cdd7235ac45b4..e5caf440aca3a 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ugt.ll
@@ -445,7 +445,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
index 107a3554282ad..3fb7b46e69569 100644
--- a/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
+++ b/llvm/test/Transforms/IRCE/unsigned_comparisons_ult.ll
@@ -652,7 +652,7 @@ exit:
 ; CHECK: [[RNG0]] = !{i32 0, i32 50}
 ; CHECK: [[LOOP1]] = distinct !{[[LOOP1]], [[META2:![0-9]+]], [[META3:![0-9]+]], [[META4:![0-9]+]], [[META5:![0-9]+]]}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.disable"}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[META4]] = !{!"llvm.loop.licm_versioning.disable"}
 ; CHECK: [[META5]] = !{!"llvm.loop.distribute.disable"}
 ; CHECK: [[META6]] = !{}
diff --git a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
index 0d8dcfbcf2a8b..86a9d00a2b1e3 100644
--- a/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
+++ b/llvm/test/Transforms/Inline/dilocation-loop-metadata-update.ll
@@ -50,7 +50,7 @@ entry:
 !10 = !{!"llvm.loop.distribute.enable"}
 !11 = !{!"llvm.loop.distribute.followup_all", !7, !8, !9, !12, !13, !14}
 !12 = !{!"llvm.loop.vectorize.width", i32 8}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !{!"llvm.loop.vectorize.followup_all", !7, !8, !9, !15, !16}
 !15 = !{!"llvm.loop.isvectorized"}
 !16 = !{!"llvm.loop.unroll.count", i32 1}
@@ -69,7 +69,7 @@ entry:
 ; CHECK: [[META10]] = !{!"llvm.loop.distribute.enable"}
 ; CHECK: [[META11]] = !{!"llvm.loop.distribute.followup_all", [[META7]], [[META8]], [[META9]], [[META12:![0-9]+]], [[META13:![0-9]+]], [[META14:![0-9]+]]}
 ; CHECK: [[META12]] = !{!"llvm.loop.vectorize.width", i32 8}
-; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[META13]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: [[META14]] = !{!"llvm.loop.vectorize.followup_all", [[META7]], [[META8]], [[META9]], [[META15:![0-9]+]], [[META16:![0-9]+]]}
 ; CHECK: [[META15]] = !{!"llvm.loop.isvectorized"}
 ; CHECK: [[META16]] = !{!"llvm.loop.unroll.count", i32 1}
diff --git a/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll b/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
index 644795f08ac22..ae3d76f2dce21 100644
--- a/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
+++ b/llvm/test/Transforms/InstCombine/mem-par-metadata-memcpy.ll
@@ -61,5 +61,5 @@ declare void @llvm.memcpy.p0.p0.i64(ptr nocapture writeonly, ptr nocapture reado
 
 !1 = distinct !{!1, !2, !3, !{!"llvm.loop.parallel_accesses", !4}}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = distinct !{} ; access group
diff --git a/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll b/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
index 875f45ebd951c..04b2ec7f9e8b3 100644
--- a/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
+++ b/llvm/test/Transforms/LoopIdiom/AArch64/disable-vectorize.ll
@@ -379,12 +379,12 @@ exit:
 !0 = distinct !{!0, !1}
 !1 = !{!"llvm.loop.vectorize.width", i32 1}
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 false}
+!3 = !{!"llvm.loop.vectorize.disable"}
 
 attributes #0 = { "target-features"="+sve2" }
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.vectorize.width", i32 1}
 ; CHECK: [[LOOP2]] = distinct !{[[LOOP2]], [[META3:![0-9]+]]}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ;.
diff --git a/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll b/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
index 4465288e0f1b2..6e28973518552 100644
--- a/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
+++ b/llvm/test/Transforms/LoopRotate/vect.omp.persistence.ll
@@ -12,7 +12,7 @@ target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f3
 ; CHECK-NOT: loop-inc
 ; CHECK: br i1 %cmp, label %return, label %loop_cond, !llvm.loop !0
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.vectorize.enable"}
 define i32 @foo(i32 %a) {
 entry:
   br label %loop_cond
@@ -31,4 +31,4 @@ return:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll b/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
index 6fcd552d9cffa..33dad5c3a68b0 100644
--- a/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/enable_and_isvectorized.ll
@@ -26,7 +26,7 @@ simd.if.end:
 }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.isvectorized"}
 
 
diff --git a/llvm/test/Transforms/LoopTransformWarning/optnone.ll b/llvm/test/Transforms/LoopTransformWarning/optnone.ll
index 469f3ce7ffc2e..caa75266d28ee 100644
--- a/llvm/test/Transforms/LoopTransformWarning/optnone.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/optnone.ll
@@ -40,7 +40,7 @@ attributes #0 = { noinline optnone }
 !1 = !{!"llvm.loop.unroll.enable"}
 !2 = !{!"llvm.loop.distribute.enable"}
 !3 = !{!"llvm.loop.unroll_and_jam.enable"}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 
 
 ; CHECK-NOT: warning
diff --git a/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll b/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
index fb33df0967db4..41bf5fe0dbe76 100644
--- a/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/vectorization-remarks-missed.ll
@@ -73,7 +73,7 @@ for.end:
 !12 = !DILocation(line: 3, column: 8, scope: !13)
 !13 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !14 = !{!14, !15, !15}
-!15 = !{!"llvm.loop.vectorize.enable", i1 true}
+!15 = !{!"llvm.loop.vectorize.enable"}
 !16 = !DILocation(line: 4, column: 5, scope: !17)
 !17 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !13)
 !18 = !{!19, !19, i64 0}
diff --git a/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll b/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
index 0168bddacca51..4fc87ac4044be 100644
--- a/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
+++ b/llvm/test/Transforms/LoopTransformWarning/vectorizer-loop-kind-unroll-warning.ll
@@ -105,7 +105,7 @@ exit:
 ; (runtime unroll disabled to prevent the unroller from succeeding on this
 ; runtime-trip-count loop, so the hint survives to transform-warning)
 !10 = distinct !{!10, !11, !12, !13}
-!11 = !{!"llvm.loop.vectorize.enable", i1 false}
+!11 = !{!"llvm.loop.vectorize.disable"}
 !12 = !{!"llvm.loop.unroll.enable"}
 !13 = !{!"llvm.loop.unroll.runtime.disable"}
 
diff --git a/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll b/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
index 28202bb4adbd5..74f22685e6542 100644
--- a/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
+++ b/llvm/test/Transforms/LoopUnroll/unroll-pragmas-disabled.ll
@@ -31,7 +31,7 @@ for.end:                                          ; preds = %for.body
   ret void
 }
 !1 = !{!1, !2, !3, !4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.unroll.count", i32 4}
 !4 = !{!"llvm.loop.vectorize.width", i32 8}
 
@@ -139,7 +139,7 @@ for.inc5.1:                                       ; preds = %for.body3.1
 
 
 ; CHECK: ![[LOOP_1]] = distinct !{![[LOOP_1]], ![[VEC_ENABLE:.*]], ![[WIDTH_8:.*]], ![[UNROLL_DISABLE:.*]]}
-; CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: ![[VEC_ENABLE]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: ![[WIDTH_8]] = !{!"llvm.loop.vectorize.width", i32 8}
 ; CHECK: ![[UNROLL_DISABLE]] = !{!"llvm.loop.unroll.disable"}
 ; CHECK: ![[LOOP_2]] = distinct !{![[LOOP_2]], ![[UNROLL_FULL:.*]]}
diff --git a/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll b/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
index 723a0273e90a7..68913a90e8185 100644
--- a/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
+++ b/llvm/test/Transforms/LoopUnroll/vectorizer-loop-kind-remarks.ll
@@ -101,7 +101,7 @@ exit:
 !99 = !DISubroutineType(types: !{})
 
 !10 = distinct !{!10, !11}
-!11 = !{!"llvm.loop.vectorize.enable", i1 false}
+!11 = !{!"llvm.loop.vectorize.disable"}
 
 !100 = distinct !DISubprogram(name: "plain_loop", scope: !1, file: !1, line: 1, type: !99, isLocal: false, isDefinition: true, scopeLine: 1, unit: !0)
 !108 = !DILocation(line: 1, column: 1, scope: !100)
diff --git a/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll b/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
index 0941c2367dfb8..51ce91a0b4c94 100644
--- a/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
+++ b/llvm/test/Transforms/LoopVectorize/2016-07-27-loop-vec.ll
@@ -16,4 +16,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll b/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
index 54dbddff5d9fd..47ec13b653a18 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/call-costs.ll
@@ -324,4 +324,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
index cc1eb9c9614f9..09852d7951c09 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-branches-cost.ll
@@ -1353,4 +1353,4 @@ attributes #3 = { "target-cpu"="neoverse-v2" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 8}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
index f38d8897ca8ff..61ebc1e306ca5 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/conditional-scalar-assignment.ll
@@ -2120,4 +2120,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.interleave.count", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
index d64504685fa5d..6a78f2b2bcf2b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/interleave-with-gaps.ll
@@ -1625,4 +1625,4 @@ attributes #1 = { "target-cpu"="grace" }
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
index e007b706d3d44..f49b18c80f528 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/masked-op-cost.ll
@@ -89,6 +89,6 @@ attributes #0 = { "target-features"="+neon,+sve" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = distinct !{!5, !6}
 !6 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll b/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
index 14d6ef1716bd6..127057c58db2d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/masked_ldst_sme.ll
@@ -163,7 +163,7 @@ attributes #0 = { uwtable vscale_range(1,16) "aarch64_pstate_sm_body" "target-fe
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 16}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[META0]] = !{[[META1:![0-9]+]]}
 ; CHECK: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll b/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
index 11370a93b2ffa..a533b861067a6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/optsize_minsize.ll
@@ -779,4 +779,4 @@ for.cond.cleanup:
 attributes #0 = { "target-features"="+sve" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
index b111eb26b286a..8ac02a0cd5961 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_prefer_scalable.ll
@@ -102,4 +102,4 @@ exit:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
index 1ad20d7ed0655..2396cf4cf905e 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/outer_loop_test1_no_explicit_vect_width.ll
@@ -150,4 +150,4 @@ exit:
 
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
index 2f27092622192..0282a9cac5c8d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-epilogue.ll
@@ -448,7 +448,7 @@ exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-cpu"="apple-m1" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
index 0985d7f052a90..339aea4551d6b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-mixed.ll
@@ -394,6 +394,6 @@ for.exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-features"="+neon" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
index f0f9e8e74b3f5..3ba46e2fd46d6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product-neon.ll
@@ -2395,4 +2395,4 @@ exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
index 09e5959f7083c..f19db4519c09a 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-dot-product.ll
@@ -2692,6 +2692,6 @@ for.body:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { vscale_range(1,16) "target-features"="+neon,+dotprod,+sve" "target-cpu"="neoverse-v2" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
index dd21d1994d95d..641d6db59147d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/partial-reduce-sub.ll
@@ -524,7 +524,7 @@ for.exit:
 !7 = distinct !{!7, !8, !9, !10}
 !8 = !{!"llvm.loop.mustprogress"}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!10 = !{!"llvm.loop.vectorize.enable", i1 true}
+!10 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { vscale_range(1,16) "target-features"="+sve" }
 attributes #1 = { "target-cpu"="apple-m1" }
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll b/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
index 6bc2c098576c7..980d501954124 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/pr60831-sve-inv-store-crash.ll
@@ -661,6 +661,6 @@ attributes #0 = { "target-features"="+neon,+sve" vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
 
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll b/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
index 363223a66f398..3c47627e5f45f 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/predicated-costs.ll
@@ -371,7 +371,7 @@ attributes #1 = { "target-cpu"="neoverse-v2" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 ; BFI computes if is taken 20 times, and loop 32 times. Make sure we round the
 ; divisor up to 2 so that we don't vectorize the loop unprofitably.
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
index 13af29855be23..71e814bd4bdfd 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-predicate-instruction.ll
@@ -88,4 +88,4 @@ attributes #0 = { "target-features"="+sve" }
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
index b68c7151c3a89..25d01a2295899 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions-tf.ll
@@ -64,4 +64,4 @@ for.end:
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
index 75e30632c5cc8..cec1dd6c359af 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/scalable-reductions.ll
@@ -456,4 +456,4 @@ for.end:
 !1 = !{!"llvm.loop.vectorize.width", i32 8}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 2}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll b/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
index 4b2e5d051730b..b79cc7cf2bcee 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/strict-fadd.ll
@@ -1416,6 +1416,6 @@ for.cond.cleanup:
 !8 = !{!"llvm.loop.vectorize.width", i32 1}
 !9 = !{!"llvm.loop.interleave.count", i32 1}
 !10 = !{!"llvm.loop.interleave.count", i32 4}
-!11 = !{!"llvm.loop.vectorize.enable", i1 true}
+!11 = !{!"llvm.loop.vectorize.enable"}
 !12 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 !13 = distinct !{!13, !6, !9, !11}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
index e98c726e82f8e..0d5d4a42f100c 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-basic-vec.ll
@@ -95,4 +95,4 @@ for.end:
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
index b6c2abe3d805d..d51ae84401031 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-cond-inv-loads.ll
@@ -183,4 +183,4 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
index 834e915d4685e..687a16cefd65b 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-extract-last-veclane.ll
@@ -67,5 +67,5 @@ attributes #0 = { "target-cpu"="generic" "target-features"="+neon,+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !2, !3, !4, !5}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
index 6fbab439e2035..eeae794f068cf 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter-cost.ll
@@ -187,4 +187,4 @@ attributes #0 = { vscale_range(1, 16) "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
index ba1fd71659b59..b43e2c8359525 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-gather-scatter.ll
@@ -271,4 +271,4 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
index 4073b1055b504..eb12b7f5c1cd4 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inductions.ll
@@ -74,6 +74,6 @@ attributes #0 = { "target-features"="+sve" }
 !0 = distinct !{!0, !1, !2, !3, !4, !5}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
index 8b3fa9279768e..a01188ca95678 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-inv-store.ll
@@ -106,5 +106,5 @@ attributes #0 = { "target-features"="+neon,+sve" vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
index a7c164fa7de16..561cbce0f8738 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-large-strides.ll
@@ -97,6 +97,6 @@ attributes #0 = { vscale_range(1, 16) }
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !7, !3, !4, !5}
 !7 = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
index 826617da6f903..5ee66f493a9cd 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-masked-loadstore.ll
@@ -76,4 +76,4 @@ exit:
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
index fb17b9a80d09b..e80df2a09824c 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll
@@ -401,4 +401,4 @@ attributes #0 = { "target-features"="+sve" }
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !3 = !{!"llvm.loop.interleave.count", i32 1}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
index 78858fe512fd4..0b7782c95c417 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-type-conv.ll
@@ -263,4 +263,4 @@ attributes #0 = { "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
index 6d16222f25f92..91df2229db6e6 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse-mask4.ll
@@ -101,4 +101,4 @@ attributes #0 = {"target-cpu"="generic" "target-features"="+neon,+sve"}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
index 4f53d31fd98fa..82fe12a43f842 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-vector-reverse.ll
@@ -152,6 +152,6 @@ attributes #0 = { "target-cpu"="generic" "target-features"="+neon,+sve" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
 
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
index 1794a6ffc3ccd..0039941d2d47f 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-extractvalue.ll
@@ -58,4 +58,4 @@ attributes #0 = { "target-features"="+sve" }
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 1}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
index 017982a4f8a8a..f1085e21d30f1 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/sve-widen-phi.ll
@@ -347,7 +347,7 @@ attributes #0 = { vscale_range(1, 16) }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
 !6 = distinct !{!6, !1, !7, !3, !4, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll b/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
index fc20277b1d858..a354f94ba832d 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/unsafe-vf-hint-remark.ll
@@ -39,4 +39,4 @@ exit:
 
 !0 = !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i64 32}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
index 90c4ed0fb8ded..d99ab30f5ebb2 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse-mask4.ll
@@ -119,5 +119,5 @@ attributes #0 = {"target-cpu"="generic" "target-features"="+neon,+sve" vscale_ra
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
 !5 = !{!"llvm.loop.interleave.count", i32 2}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
index 7106be6ec78dd..cf06100f988ef 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vector-reverse.ll
@@ -266,4 +266,4 @@ attributes #1 = { "target-cpu"="neoverse-v2" }
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 8}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!4 = !{!"llvm.loop.vectorize.enable", i1 true}
+!4 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll b/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
index 3d8863368e74c..a15b377d4ef3e 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/vplan-native-outer-loop-wide-type.ll
@@ -56,4 +56,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll b/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
index fbc313eddea8a..3b29335c36301 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/widen-gep-all-indices-invariant.ll
@@ -54,7 +54,7 @@ exit:
 attributes #0 = { "target-cpu"="neoverse-v2" }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll b/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
index 83be0708774f1..8d8766051a5b9 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/optsize_minsize.ll
@@ -482,4 +482,4 @@ for.cond.cleanup:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll b/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
index 1b5db04061631..75862d9c72e00 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/prefer-tail-loop-folding.ll
@@ -422,7 +422,7 @@ for.body:
 attributes #0 = { nofree norecurse nounwind "target-features"="+armv8.1-m.main,+mve.fp" }
 
 !5 = distinct !{!5, !6}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
 
 !7 = distinct !{!7, !8}
 !8 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
index f205e21762317..3038b7fd935d3 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-loop-hint.ll
@@ -81,4 +81,4 @@ for.body:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
index c7af4646ccdee..a477fee265d59 100644
--- a/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
+++ b/llvm/test/Transforms/LoopVectorize/ARM/tail-folding-prefer-flag.ll
@@ -102,7 +102,7 @@ for.body:
 
 !10 = distinct !{!10, !11, !12}
 !11 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
-!12 = !{!"llvm.loop.vectorize.enable", i1 true}
+!12 = !{!"llvm.loop.vectorize.enable"}
 
 !14 = distinct !{!14, !15}
 !15 = !{!"llvm.loop.interleave.count", i32 4}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll b/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
index 3e3c6c09050b4..2cccae4de2fa5 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/dbg-tail-folding-by-evl.ll
@@ -72,7 +72,7 @@ exit:
 !9 = !DILocation(line: 2, column: 27, scope: !3)
 !10 = !DILocation(line: 2, column: 5, scope: !3)
 !11 = distinct !{!11, !12}
-!12 = !{!"llvm.loop.vectorize.enable", i1 true}
+!12 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[META0:![0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C_plus_plus_14, file: [[META1:![0-9]+]], producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
 ; CHECK: [[META1]] = !DIFile(filename: "{{.*}}dbg-tail-folding-by-evl.cpp", directory: {{.*}})
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll b/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
index 7adb8b6e1b2cf..22560793e4504 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/force-vect-msg.ll
@@ -31,4 +31,4 @@ for.end:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll b/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
index 1c9e48fdf4aa6..100ecc172fab0 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/riscv-vector-reverse.ll
@@ -908,5 +908,5 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = distinct !{!4, !1, !3}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
index 4e18b73abc73f..b068c03e8729c 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-cond-reduction.ll
@@ -770,7 +770,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL-OUTLOOP: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; IF-EVL-OUTLOOP: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
index 062c052b95391..43c734471537b 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-fixed-order-recurrence.ll
@@ -623,7 +623,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; IF-EVL: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
index 1fad71f9f9e04..ba6af22e10718 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-inloop-reduction.ll
@@ -1658,4 +1658,4 @@ for.end:
 
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
index 935e37e94c266..77808152757e8 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-interleave.ll
@@ -757,4 +757,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
index 8b86d42fe4377..e7631a2ddf9b5 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-intermediate-store.ll
@@ -234,7 +234,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; IF-EVL-OUTLOOP: [[META0]] = !{[[META1:![0-9]+]]}
 ; IF-EVL-OUTLOOP: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
index 85dc6b407c877..19099b43557ad 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-ordered-reduction.ll
@@ -91,4 +91,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
index a918e4c61425d..88a3c99f77b66 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-reduction.ll
@@ -1682,4 +1682,4 @@ for.end:
 
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
index 7a7ae1edbc415..8313e911e2b40 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/tail-folding-uniform-store.ll
@@ -59,4 +59,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll b/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
index 64d412666c74f..256528c25a781 100644
--- a/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
+++ b/llvm/test/Transforms/LoopVectorize/RISCV/uniform-load-store.ll
@@ -895,4 +895,4 @@ for.end:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll b/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
index c4ceb53200cce..97a3eb830213d 100644
--- a/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
+++ b/llvm/test/Transforms/LoopVectorize/Sparc/no-vectorize.ll
@@ -59,15 +59,15 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.mustprogress"}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; SPARC32: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; SPARC32: [[META1]] = !{!"llvm.loop.mustprogress"}
-; SPARC32: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; SPARC32: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; SPARC64: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; SPARC64: [[META1]] = !{!"llvm.loop.mustprogress"}
-; SPARC64: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; SPARC64: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
 ; SPARC32: {{.*}}
diff --git a/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll b/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
index 5058b4c85d057..4e55fcc295fe1 100644
--- a/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
+++ b/llvm/test/Transforms/LoopVectorize/VE/disable_lv.ll
@@ -35,4 +35,4 @@ simd.if.end:
 !6 = distinct !{}
 !7 = distinct !{!7, !8, !9}
 !8 = !{!"llvm.loop.parallel_accesses", !6}
-!9 = !{!"llvm.loop.vectorize.enable", i1 true}
+!9 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
index 7cdc33a44b3a6..c5f61f94e2c2f 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-fixed-order-recurrence.ll
@@ -72,4 +72,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
index 2d818db242689..51f3305a25893 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/RISCV/vplan-vp-intrinsics-reduction.ll
@@ -222,4 +222,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll b/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
index 92bb9704c4fd2..4ec0ec66d4321 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/constant-fold.ll
@@ -318,5 +318,5 @@ exit:
 
 !1 = distinct !{!1, !2, !3, !4}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
index 6eff6d1df3523..a86a211bf1417 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-printing-outer-loop.ll
@@ -70,4 +70,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
index 7f442f7d72e78..2edddc1079178 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-stress-test-no-explict-vf.ll
@@ -40,4 +40,4 @@ for.end10:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
index 7ad03ebaca7c2..985a2c693aded 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/amdlibm-calls-finite.ll
@@ -484,15 +484,15 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 !4 = distinct !{!4, !5, !6}
 !5 = !{!"llvm.loop.vectorize.width", i32 4}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
 
 !7 = distinct !{!7, !8, !9}
 !8 = !{!"llvm.loop.vectorize.width", i32 8}
-!9 = !{!"llvm.loop.vectorize.enable", i1 true}
+!9 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__expf_finite(float) #0
 declare double @__exp_finite(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll b/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
index 9cb7887346d3e..283aeb2a5ba21 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/conditional-scalar-assignment.ll
@@ -1016,4 +1016,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.interleave.count", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll b/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
index 1853b2648af7e..49f5afb3aa705 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/cost-model.ll
@@ -1117,4 +1117,4 @@ attributes #4 = { "target-cpu"="haswell" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll b/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
index 165d81b285a19..54a265110185a 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/induction-costs.ll
@@ -642,7 +642,7 @@ exit:
 
 !1 = distinct !{!1, !2, !3, !4}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 !4 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 
 define void @wide_iv_trunc_reuse(ptr %dst) {
diff --git a/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll b/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
index 9823d18da2905..d08c1a404e6c3 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/iv-live-outs.ll
@@ -176,4 +176,4 @@ attributes #0 = { "target-cpu"="skylake-avx512" }
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.vectorize.width", i32 2}
 !2 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
index 025e24bcbc5eb..5f7261240e3d4 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-VF2-VF8.ll
@@ -28,7 +28,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @sin_f32(ptr nocapture %varray) {
@@ -56,7 +56,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 8}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f64_intrinsic(
@@ -83,7 +83,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 2}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f32_intrinsic(
@@ -110,7 +110,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 8}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64(
@@ -137,7 +137,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 2}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32(
@@ -164,7 +164,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 8}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64_intrinsic(
@@ -191,7 +191,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 2}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32_intrinsic(
@@ -218,7 +218,7 @@ for.end:
 
 !81 = distinct !{!81, !82, !83}
 !82 = !{!"llvm.loop.vectorize.width", i32 8}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @exp_f32(ptr nocapture %varray) {
@@ -245,7 +245,7 @@ for.end:
 
 !91 = distinct !{!91, !92, !93}
 !92 = !{!"llvm.loop.vectorize.width", i32 8}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f32_intrin(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f32_intrin
@@ -271,7 +271,7 @@ for.end:
 
 !101 = distinct !{!101, !102, !103}
 !102 = !{!"llvm.loop.vectorize.width", i32 8}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @log_f32(ptr nocapture %varray) {
@@ -298,7 +298,7 @@ for.end:
 
 !111 = distinct !{!111, !112, !113}
 !112 = !{!"llvm.loop.vectorize.width", i32 8}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -326,7 +326,7 @@ for.end:
 
 !121 = distinct !{!121, !122, !123}
 !122 = !{!"llvm.loop.vectorize.width", i32 8}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32_intrin(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32_intrin
@@ -354,7 +354,7 @@ for.end:
 
 !131 = distinct !{!131, !132, !133}
 !132 = !{!"llvm.loop.vectorize.width", i32 8}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f64(
@@ -381,7 +381,7 @@ for.end:
 
 !141 = distinct !{!141, !142, !143}
 !142 = !{!"llvm.loop.vectorize.width", i32 2}
-!143 = !{!"llvm.loop.vectorize.enable", i1 true}
+!143 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @tan_f32(ptr nocapture %varray) {
@@ -409,7 +409,7 @@ for.end:
 
 !151 = distinct !{!151, !152, !153}
 !152 = !{!"llvm.loop.vectorize.width", i32 8}
-!153 = !{!"llvm.loop.vectorize.enable", i1 true}
+!153 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f64_intrinsic(
@@ -436,7 +436,7 @@ for.end:
 
 !161 = distinct !{!161, !162, !163}
 !162 = !{!"llvm.loop.vectorize.width", i32 2}
-!163 = !{!"llvm.loop.vectorize.enable", i1 true}
+!163 = !{!"llvm.loop.vectorize.enable"}
 
 define void @tan_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @tan_f32_intrinsic(
@@ -465,7 +465,7 @@ for.end:
 
 !171 = distinct !{!171, !172, !173}
 !172 = !{!"llvm.loop.vectorize.width", i32 8}
-!173 = !{!"llvm.loop.vectorize.enable", i1 true}
+!173 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { nounwind readnone }
 
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
index 95aec377738b8..176ce4e05299f 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls-finite.ll
@@ -27,7 +27,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f64
@@ -54,7 +54,7 @@ for.end:
 
 !11 = distinct !{!11, !12, !13}
 !12 = !{!"llvm.loop.vectorize.width", i32 4}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 
 define void @log_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @log_f32
@@ -81,7 +81,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @log_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @log_f64
@@ -108,7 +108,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -137,7 +137,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f64(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f64
@@ -166,7 +166,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__expf_finite(float) #0
 declare double @__exp_finite(double) #0
diff --git a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
index 071f262ec903b..acbd8dbbd76a9 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/libm-vector-calls.ll
@@ -28,7 +28,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @sin_f32(ptr nocapture %varray) {
@@ -56,7 +56,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f64_intrinsic(
@@ -83,7 +83,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 define void @sin_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @sin_f32_intrinsic(
@@ -110,7 +110,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64(
@@ -137,7 +137,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32(
@@ -164,7 +164,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 4}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f64_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f64_intrinsic(
@@ -191,7 +191,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 4}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 define void @cos_f32_intrinsic(ptr nocapture %varray) {
 ; CHECK-LABEL: @cos_f32_intrinsic(
@@ -218,7 +218,7 @@ for.end:
 
 !81 = distinct !{!81, !82, !83}
 !82 = !{!"llvm.loop.vectorize.width", i32 4}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @exp_f32(ptr nocapture %varray) {
@@ -245,7 +245,7 @@ for.end:
 
 !91 = distinct !{!91, !92, !93}
 !92 = !{!"llvm.loop.vectorize.width", i32 4}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 define void @exp_f32_intrin(ptr nocapture %varray) {
 ; CHECK-LABEL: @exp_f32_intrin
@@ -271,7 +271,7 @@ for.end:
 
 !101 = distinct !{!101, !102, !103}
 !102 = !{!"llvm.loop.vectorize.width", i32 4}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 define void @log_f32(ptr nocapture %varray) {
@@ -298,7 +298,7 @@ for.end:
 
 !111 = distinct !{!111, !112, !113}
 !112 = !{!"llvm.loop.vectorize.width", i32 4}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32
@@ -326,7 +326,7 @@ for.end:
 
 !121 = distinct !{!121, !122, !123}
 !122 = !{!"llvm.loop.vectorize.width", i32 4}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 define void @pow_f32_intrin(ptr nocapture %varray, ptr nocapture readonly %exp) {
 ; CHECK-LABEL: @pow_f32_intrin
@@ -354,7 +354,7 @@ for.end:
 
 !131 = distinct !{!131, !132, !133}
 !132 = !{!"llvm.loop.vectorize.width", i32 4}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
 
 attributes #0 = { nounwind readnone }
 
diff --git a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
index e61e22e50ff56..ae77b4270ab3e 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath.ll
@@ -90,7 +90,7 @@ for.body:
 !24 = !DILocation(line: 17, column: 19, scope: !20)
 !25 = !DILocation(line: 17, column: 11, scope: !20)
 !26 = distinct !{!26, !27}
-!27 = !{!"llvm.loop.vectorize.enable", i1 true}
+!27 = !{!"llvm.loop.vectorize.enable"}
 !28 = distinct !DICompileUnit(language: DW_LANG_C99, producer: "clang",
                              file: !5,
                              isOptimized: true, flags: "-O2",
diff --git a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
index a4bfbcb8ca4d2..cbedbf7fd10c0 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/no_fpmath_with_hotness.ll
@@ -92,7 +92,7 @@ for.body:
 !24 = !DILocation(line: 17, column: 19, scope: !20)
 !25 = !DILocation(line: 17, column: 11, scope: !20)
 !26 = distinct !{!26, !27, !18}
-!27 = !{!"llvm.loop.vectorize.enable", i1 true}
+!27 = !{!"llvm.loop.vectorize.enable"}
 !28 = distinct !DICompileUnit(language: DW_LANG_C99, producer: "clang",
                              file: !5,
                              isOptimized: true, flags: "-O2",
diff --git a/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll b/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
index d09f42ff558db..7c4f98670719f 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/outer-loop-non-power-of-2-type.ll
@@ -92,4 +92,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll b/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
index d10449327d9c0..5b8020cc6630a 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/outer_loop_test1_no_explicit_vect_width.ll
@@ -114,4 +114,4 @@ exit:
 }
 
 !1 = distinct !{!1, !2}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll b/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
index 21146e2e82216..b3f2048e81663 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/redundant-vf2-cost.ll
@@ -31,4 +31,4 @@ exit:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
index a33874a871443..a52e2c7a0dde8 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs-max-bandwidth.ll
@@ -369,7 +369,7 @@ exit:
 attributes #0 = { "target-cpu"="znver2" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
index ddb1905f3ffc4..edb115ce81b20 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/replicating-load-store-costs.ll
@@ -1263,4 +1263,4 @@ exit:
 attributes #0 = { "target-cpu"="znver2" }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll b/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
index 6586b08e5c294..a768a38da5593 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/svml-calls-finite.ll
@@ -33,7 +33,7 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__exp_finite(double) #0
@@ -62,7 +62,7 @@ for.end:
 
 !11 = distinct !{!11, !12, !13}
 !12 = !{!"llvm.loop.vectorize.width", i32 4}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 
 
 
@@ -93,7 +93,7 @@ for.end:
 
 !21 = distinct !{!21, !22, !23}
 !22 = !{!"llvm.loop.vectorize.width", i32 4}
-!23 = !{!"llvm.loop.vectorize.enable", i1 true}
+!23 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log_finite(double) #0
@@ -122,7 +122,7 @@ for.end:
 
 !31 = distinct !{!31, !32, !33}
 !32 = !{!"llvm.loop.vectorize.width", i32 4}
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare float @__powf_finite(float, float) #0
@@ -153,7 +153,7 @@ for.end:
 
 !41 = distinct !{!41, !42, !43}
 !42 = !{!"llvm.loop.vectorize.width", i32 4}
-!43 = !{!"llvm.loop.vectorize.enable", i1 true}
+!43 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__pow_finite(double, double) #0
@@ -184,7 +184,7 @@ for.end:
 
 !51 = distinct !{!51, !52, !53}
 !52 = !{!"llvm.loop.vectorize.width", i32 4}
-!53 = !{!"llvm.loop.vectorize.enable", i1 true}
+!53 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__exp2f_finite(float) #0
 
@@ -213,7 +213,7 @@ for.end:
 
 !61 = distinct !{!61, !62, !63}
 !62 = !{!"llvm.loop.vectorize.width", i32 4}
-!63 = !{!"llvm.loop.vectorize.enable", i1 true}
+!63 = !{!"llvm.loop.vectorize.enable"}
 
 declare double @__exp2_finite(double) #0
 
@@ -242,7 +242,7 @@ for.end:
 
 !71 = distinct !{!71, !72, !73}
 !72 = !{!"llvm.loop.vectorize.width", i32 4}
-!73 = !{!"llvm.loop.vectorize.enable", i1 true}
+!73 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__log2f_finite(float) #0
 
@@ -270,7 +270,7 @@ for.end:
 
 !81 = distinct !{!21, !22, !23}
 !82 = !{!"llvm.loop.vectorize.width", i32 4}
-!83 = !{!"llvm.loop.vectorize.enable", i1 true}
+!83 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log2_finite(double) #0
@@ -299,7 +299,7 @@ for.end:
 
 !91 = distinct !{!31, !32, !33}
 !92 = !{!"llvm.loop.vectorize.width", i32 4}
-!93 = !{!"llvm.loop.vectorize.enable", i1 true}
+!93 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__log10f_finite(float) #0
 
@@ -327,7 +327,7 @@ for.end:
 
 !101 = distinct !{!21, !22, !23}
 !102 = !{!"llvm.loop.vectorize.width", i32 4}
-!103 = !{!"llvm.loop.vectorize.enable", i1 true}
+!103 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__log10_finite(double) #0
@@ -356,7 +356,7 @@ for.end:
 
 !111 = distinct !{!31, !32, !33}
 !112 = !{!"llvm.loop.vectorize.width", i32 4}
-!113 = !{!"llvm.loop.vectorize.enable", i1 true}
+!113 = !{!"llvm.loop.vectorize.enable"}
 
 declare float @__sqrtf_finite(float) #0
 
@@ -384,7 +384,7 @@ for.end:
 
 !121 = distinct !{!21, !22, !23}
 !122 = !{!"llvm.loop.vectorize.width", i32 4}
-!123 = !{!"llvm.loop.vectorize.enable", i1 true}
+!123 = !{!"llvm.loop.vectorize.enable"}
 
 
 declare double @__sqrt_finite(double) #0
@@ -413,4 +413,4 @@ for.end:
 
 !131 = distinct !{!31, !32, !33}
 !132 = !{!"llvm.loop.vectorize.width", i32 4}
-!133 = !{!"llvm.loop.vectorize.enable", i1 true}
+!133 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll b/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
index 52c504b090ae1..a1def726e10ed 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/tail_folding_and_assume_safety.ll
@@ -143,7 +143,7 @@ for.inc:
 attributes #0 = { "target-features"="+fxsr,+mmx,+sse,+sse2,+x87" "use-soft-float"="false" }
 
 !6 = distinct !{!6, !7}
-!7 = !{!"llvm.loop.vectorize.enable", i1 true}
+!7 = !{!"llvm.loop.vectorize.enable"}
 
 !8 = distinct !{!8, !9}
 !9 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll b/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
index 90866061c4188..e5b88dd5219db 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/tail_loop_folding.ll
@@ -172,7 +172,7 @@ attributes #0 = { optsize "target-cpu"="core-avx2" "target-features"="+avx,+avx2
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
 
 !10 = distinct !{!10, !11}
 !11 = !{!"llvm.loop.vectorize.predicate.enable", i1 false}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
index dd30c045182e4..0333344025de1 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.ll
@@ -49,7 +49,7 @@ for.end:
 }
 
 !1 = !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !11 = distinct !{}
 
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
index 78b14423578fb..98c0a3661fa1c 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vect.omp.force.small-tc.ll
@@ -107,7 +107,7 @@ for.end:
 }
 
 !1 = !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !11 = distinct !{}
 
 ;
diff --git a/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll b/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
index 5f2a93b230995..c389765dff2d1 100644
--- a/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
+++ b/llvm/test/Transforms/LoopVectorize/X86/vectorization-remarks-missed.ll
@@ -320,7 +320,7 @@ declare i32 @foo(...)
 !12 = !DILocation(line: 3, column: 8, scope: !13)
 !13 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !14 = !{!14, !15, !15}
-!15 = !{!"llvm.loop.vectorize.enable", i1 true}
+!15 = !{!"llvm.loop.vectorize.enable"}
 !16 = !DILocation(line: 4, column: 5, scope: !17)
 !17 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !13)
 !22 = !DILocation(line: 5, column: 9, scope: !23)
diff --git a/llvm/test/Transforms/LoopVectorize/alias-mask.ll b/llvm/test/Transforms/LoopVectorize/alias-mask.ll
index 7035332a1b763..2a811955e8bd3 100644
--- a/llvm/test/Transforms/LoopVectorize/alias-mask.ll
+++ b/llvm/test/Transforms/LoopVectorize/alias-mask.ll
@@ -650,4 +650,4 @@ exit:
 
 
 !2 = distinct !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll b/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
index 45fa8d19952a4..ab61d2aa710ce 100644
--- a/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
+++ b/llvm/test/Transforms/LoopVectorize/dbg-outer-loop-vect.ll
@@ -131,7 +131,7 @@ exit:
 !30 = !DILocation(line: 10, column: 24, scope: !16)
 !31 = distinct !{!31, !21, !32, !33}
 !32 = !DILocation(line: 13, column: 13, scope: !12)
-!33 = !{!"llvm.loop.vectorize.enable", i1 true}
+!33 = !{!"llvm.loop.vectorize.enable"}
 !34 = !DILocation(line: 10, column: 5, scope: !12)
 ;.
 ; CHECK: [[META0:![0-9]+]] = distinct !DICompileUnit(language: DW_LANG_C11, file: [[META1:![0-9]+]], producer: "clang", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, splitDebugInlining: false, nameTableKind: None)
diff --git a/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll b/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
index 31454401876b6..da41fa7a9da93 100644
--- a/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
+++ b/llvm/test/Transforms/LoopVectorize/diag-disabled-vectorization-msgs.ll
@@ -63,7 +63,7 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 false}
+!1 = !{!"llvm.loop.vectorize.disable"}
 
 ; ALL-LABEL: 'disable_nonforced' from <stdin>
 ; ALL-NOT: LV: We can vectorize this loop
diff --git a/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll b/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
index 8c4e50fdd7a61..e7622d30f5e2c 100644
--- a/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
+++ b/llvm/test/Transforms/LoopVectorize/disable_nonforced_enable.ll
@@ -26,4 +26,4 @@ for.end:
   ret void
 }
 
-!0 = !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.vectorize.enable", i32 1}}
+!0 = !{!0, !{!"llvm.loop.disable_nonforced"}, !{!"llvm.loop.vectorize.enable"}}
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
index a610f0669f483..7393ad8ea4dab 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_detection.ll
@@ -219,7 +219,7 @@ for.end15:
 ; Case 1
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 4}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
 ; Case 2
 !9 = distinct !{!9, !8}
 ; Case 3
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
index ef6fabb11694b..9c87b9cb1e818 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_nonuniform_inner.ll
@@ -161,4 +161,4 @@ for.end15:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 8}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll b/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
index 2152c48c61e81..75858e626b728 100644
--- a/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
+++ b/llvm/test/Transforms/LoopVectorize/explicit_outer_uniform_diverg_branch.ll
@@ -124,4 +124,4 @@ for.end19:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.width", i32 8}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll b/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
index 1fe802f9e1093..6e10cb0181629 100644
--- a/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
+++ b/llvm/test/Transforms/LoopVectorize/make-followup-loop-id.ll
@@ -108,7 +108,7 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.followup_all", !3, !4}
 !3 = !{!"llvm.loop.isvectorized"}
 !4 = !{!"llvm.loop.unroll.count", i32 8}
diff --git a/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll b/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
index 69e97069ae498..fc6cfe5942254 100644
--- a/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
+++ b/llvm/test/Transforms/LoopVectorize/memdep-fold-tail.ll
@@ -89,4 +89,4 @@ for.end:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll b/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
index d411783fa9df8..0d691a89a2241 100644
--- a/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
+++ b/llvm/test/Transforms/LoopVectorize/nested-outer-loop-vect.ll
@@ -86,4 +86,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll b/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
index 93c81c02d4e5d..6a80b02aaf50e 100644
--- a/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
+++ b/llvm/test/Transforms/LoopVectorize/no-fold-tail-by-masking-iv-external-uses.ll
@@ -129,7 +129,7 @@ done:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.unroll.disable"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.unroll.disable"}
diff --git a/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll b/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
index bfa9263bb5cf7..2f23747f44f9c 100644
--- a/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_array_bounds.ll
@@ -79,7 +79,7 @@ for.end15:
 !10 = !DILocation(line: 4, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 4, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !DILocation(line: 5, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 4, column: 36, file: !1, scope: !11)
 !16 = !DILocation(line: 9, column: 8, scope: !17)
diff --git a/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll b/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
index 2c1f485aab080..c00086ea9496e 100644
--- a/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_array_bounds_scalable.ll
@@ -63,7 +63,7 @@ for.end:
 }
 
 !0 = distinct !{!0, !1, !2}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!"llvm.loop.vectorize.width", i32 1}
 !3 = distinct !{!3, !1, !2, !4}
 !4 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
diff --git a/llvm/test/Transforms/LoopVectorize/no_switch.ll b/llvm/test/Transforms/LoopVectorize/no_switch.ll
index c3662f1b4be5d..531dfd0aed5f0 100644
--- a/llvm/test/Transforms/LoopVectorize/no_switch.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_switch.ll
@@ -72,7 +72,7 @@ for.end:
 !10 = !DILocation(line: 3, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 true}
+!13 = !{!"llvm.loop.vectorize.enable"}
 !14 = !DILocation(line: 4, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !11)
 !16 = !{!17, !17, i64 0}
diff --git a/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll b/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
index d73264dc01836..4c989e0b3428a 100644
--- a/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
+++ b/llvm/test/Transforms/LoopVectorize/no_switch_disable_vectorization.ll
@@ -76,7 +76,7 @@ for.end:
 !10 = !DILocation(line: 3, column: 8, scope: !11)
 !11 = distinct !DILexicalBlock(line: 3, column: 3, file: !1, scope: !4)
 !12 = !{!12, !13, !13}
-!13 = !{!"llvm.loop.vectorize.enable", i1 false}
+!13 = !{!"llvm.loop.vectorize.disable"}
 !14 = !DILocation(line: 4, column: 5, scope: !15)
 !15 = distinct !DILexicalBlock(line: 3, column: 36, file: !1, scope: !11)
 !16 = !{!17, !17, i64 0}
diff --git a/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
index 8020480f0dc65..efbfc49a21804 100644
--- a/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/noalias-scope-decl-outer-loop.ll
@@ -66,7 +66,7 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!4}
 !4 = distinct !{!4, !5}
 !5 = distinct !{!5}
diff --git a/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll b/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
index 2c46af4ccbfe2..e59e5fbfb12fb 100644
--- a/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/non-widenable-intrinsics-outer-loop.ll
@@ -454,4 +454,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.mustprogress"}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/optsize.ll b/llvm/test/Transforms/LoopVectorize/optsize.ll
index 2fe02c62c7790..294dbbbb3a5e2 100644
--- a/llvm/test/Transforms/LoopVectorize/optsize.ll
+++ b/llvm/test/Transforms/LoopVectorize/optsize.ll
@@ -1071,7 +1071,7 @@ exit:
 !13 = !{i32 999999, i64 1, i32 2}
 !14 = !{!"function_entry_count", i64 0}
 !15 = distinct !{!15, !16}
-!16 = !{!"llvm.loop.vectorize.enable", i1 true}
+!16 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[PROF14]] = !{!"function_entry_count", i64 0}
 ; CHECK: [[LOOP15]] = distinct !{[[LOOP15]], [[META16:![0-9]+]], [[META17:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
index 6e19f71736544..2618f15665b7d 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-atomic.ll
@@ -165,4 +165,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
index 960a6e6445ee1..3ec0a6e624303 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-inner-latch-successors.ll
@@ -207,4 +207,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
index 4c0b5c618d7ac..045f0f58847c4 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-vec-phi-predecessor-order.ll
@@ -119,4 +119,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll b/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
index 6f47696b8008b..ac4b91f40541e 100644
--- a/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer-loop-wide-phis.ll
@@ -161,4 +161,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
index ebe1f07c402ea..358a2edb05ba8 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_contiguous.ll
@@ -285,4 +285,4 @@ exit:
 
 !0 = distinct !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
index e0ad90d9c2bc0..a5442f4cdb7f0 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_early_exit.ll
@@ -46,4 +46,4 @@ for.end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
index 778805965e57e..6cd433ee48246 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_hcfg_construction.ll
@@ -317,4 +317,4 @@ innermost.loop.j1:
 !3 = distinct !{!3, !4, !5, !6}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
 !5 = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
-!6 = !{!"llvm.loop.vectorize.enable", i1 true}
+!6 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
index 07293812a6ffb..30ba3770b172d 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_scalable.ll
@@ -134,6 +134,6 @@ exit:
 }
 
 !1 = distinct !{!1, !2, !3, !4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.vectorize.width", i32 4}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
index cda0b24a69180..48ccd47f58f65 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_test1.ll
@@ -86,4 +86,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll b/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
index 1353445a7c29c..8a0cf20c0fd9c 100644
--- a/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
+++ b/llvm/test/Transforms/LoopVectorize/outer_loop_test2.ll
@@ -154,4 +154,4 @@ exit:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.width", i32 4}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
index e7d284726b1c2..035c80110ae98 100644
--- a/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
+++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop-uf4.ll
@@ -611,4 +611,4 @@ for.end:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
index dc095e55edd15..c77e3cf806598 100644
--- a/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
+++ b/llvm/test/Transforms/LoopVectorize/reduction-inloop.ll
@@ -3136,4 +3136,4 @@ for.exit:
 
 !6 = distinct !{!6, !7, !8}
 !7 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!8 = !{!"llvm.loop.vectorize.enable", i1 true}
+!8 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/remove_metadata.ll b/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
index 96de172a0ea7c..de9f2467a6d3e 100644
--- a/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
+++ b/llvm/test/Transforms/LoopVectorize/remove_metadata.ll
@@ -25,7 +25,7 @@ for.end:
   ret void
 }
 
-!0 = !{!0, !{!"llvm.loop.vectorize.some_property"}, !{!"llvm.loop.vectorize.enable", i32 1}}
+!0 = !{!0, !{!"llvm.loop.vectorize.some_property"}, !{!"llvm.loop.vectorize.enable"}}
 
 ; CHECK-NOT: llvm.loop.vectorize.
 ; CHECK: {!"llvm.loop.isvectorized", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll b/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
index 5376eb86882b7..d4854fbcf9c75 100644
--- a/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
+++ b/llvm/test/Transforms/LoopVectorize/runtime-check-threshold-with-force-metadata.ll
@@ -79,12 +79,12 @@ exit:
 !0 = distinct !{!0, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
 
 ;.
 ; LIMIT0: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; LIMIT0: [[META1]] = !{!"llvm.loop.vectorize.width", i32 2}
-; LIMIT0: [[META2]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; LIMIT0: [[META2]] = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; LIMIT1: [[META0]] = !{[[META1:![0-9]+]]}
 ; LIMIT1: [[META1]] = distinct !{[[META1]], [[META2:![0-9]+]]}
diff --git a/llvm/test/Transforms/LoopVectorize/runtime-check.ll b/llvm/test/Transforms/LoopVectorize/runtime-check.ll
index d7a747bfe6d66..b61c7d7140925 100644
--- a/llvm/test/Transforms/LoopVectorize/runtime-check.ll
+++ b/llvm/test/Transforms/LoopVectorize/runtime-check.ll
@@ -592,4 +592,4 @@ exit:
 !11 = !{i32 2, !"Debug Info Version", i32 3}
 !12 = distinct !{!12, !13, !14}
 !13 = !{!"llvm.loop.vectorize.width", i32 2}
-!14 = !{!"llvm.loop.vectorize.enable", i1 true}
+!14 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll b/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
index 1aaac9e6881ef..afd3ae3806fb1 100644
--- a/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
+++ b/llvm/test/Transforms/LoopVectorize/scalable-inductions.ll
@@ -307,7 +307,7 @@ exit:
 !2 = !{!"llvm.loop.vectorize.width", i32 2}
 !3 = !{!"llvm.loop.vectorize.scalable.enable", i1 true}
 !4 = !{!"llvm.loop.interleave.count", i32 2}
-!5 = !{!"llvm.loop.vectorize.enable", i1 true}
+!5 = !{!"llvm.loop.vectorize.enable"}
 !6 = distinct !{!6, !1, !7, !3, !8, !5}
 !7 = !{!"llvm.loop.vectorize.width", i32 4}
 !8 = !{!"llvm.loop.interleave.count", i32 1}
diff --git a/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll b/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
index 4fab0c38dd0da..fe8410adb8bae 100644
--- a/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
+++ b/llvm/test/Transforms/LoopVectorize/unsafe-vf-hint-remark.ll
@@ -48,4 +48,4 @@ exit:
 
 !0 = !{!0, !1, !2}
 !1 = !{!"llvm.loop.vectorize.width", i32 4}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll b/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
index 33a772285e423..5d65362df3cc2 100644
--- a/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
+++ b/llvm/test/Transforms/LoopVectorize/use-scalar-epilogue-if-tp-fails.ll
@@ -245,4 +245,4 @@ end:
 
 !1 = distinct !{!1, !2, !3}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll b/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
index 2c268664ef991..eb3ad35df5bc5 100644
--- a/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
+++ b/llvm/test/Transforms/LoopVectorize/vect.omp.persistence.ll
@@ -14,7 +14,7 @@ target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f3
 ; CHECK-LABEL: @foo
 ; CHECK: !llvm.loop !0
 ; CHECK: !0 = distinct !{!0, !1}
-; CHECK: !1 = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: !1 = !{!"llvm.loop.vectorize.enable"}
 define i32 @foo(i32 %a) {
 entry:
   br label %loop_cond
@@ -33,4 +33,4 @@ return:
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll b/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
index cd68c81f94054..ce0da2efcab20 100644
--- a/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
+++ b/llvm/test/Transforms/LoopVectorize/vector-loop-backedge-elimination-predicated-early-exit.ll
@@ -51,4 +51,4 @@ exit:
 !0 = distinct !{!0, !1, !2, !3}
 !1 = !{!"llvm.loop.mustprogress"}
 !2 = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
-!3 = !{!"llvm.loop.vectorize.enable", i1 true}
+!3 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll b/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
index efc29b3c8a32d..358082968efe4 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-outer-loop-uncomputable-trip-count.ll
@@ -44,4 +44,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll b/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
index c61887e8e3386..76379c3935323 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-vectorize-inner-loop-reduction.ll
@@ -75,4 +75,4 @@ exit:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll b/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
index 3ad77bfd7787f..5467c0354168b 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-widen-call-instruction.ll
@@ -119,11 +119,11 @@ exit:
 declare void @use()
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 ;.
 ; CHECK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
 ; CHECK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
 ; CHECK: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
 ; CHECK: [[LOOP3]] = distinct !{[[LOOP3]], [[META4:![0-9]+]]}
-; CHECK: [[META4]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[META4]] = !{!"llvm.loop.vectorize.enable"}
 ;.
diff --git a/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll b/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
index 75f56a74986c4..83a3fd69b1e6c 100644
--- a/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
+++ b/llvm/test/Transforms/LoopVectorize/vplan-widen-select-instruction.ll
@@ -256,4 +256,4 @@ exit:
   ret void
 }
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll b/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
index 049d1baf5c615..0ab38d96bfe4c 100644
--- a/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
+++ b/llvm/test/Transforms/PhaseOrdering/AArch64/Oz-and-forced-vectorize.ll
@@ -78,4 +78,4 @@ for.cond.cleanup:
 }
 
 !0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll b/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
index e27584bb421f1..15c4b10ec71b2 100644
--- a/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
+++ b/llvm/test/Transforms/PhaseOrdering/X86/loop-vectorize-metadata.ll
@@ -1163,6 +1163,6 @@ for.end:                                          ; preds = %for.body
 }
 
 !0 = !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable", i1 1}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !2 = !{!2, !3}
-!3 = !{!"llvm.loop.vectorize.enable", i1 0}
+!3 = !{!"llvm.loop.vectorize.disable"}
diff --git a/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll b/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
index fe4f81a6f4749..a66528f1f12b4 100644
--- a/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
+++ b/llvm/test/Transforms/PhaseOrdering/X86/preserve-access-group.ll
@@ -187,7 +187,7 @@ attributes #1 = { nocallback nofree nosync nounwind willreturn memory(argmem: re
 !15 = distinct !{!15, !16, !17, !18}
 !16 = !{!"llvm.loop.mustprogress"}
 !17 = !{!"llvm.loop.parallel_accesses", !12}
-!18 = !{!"llvm.loop.vectorize.enable", i1 true}
+!18 = !{!"llvm.loop.vectorize.enable"}
 
 ;.
 ; CHECK: [[INT_TBAA0]] = !{[[META1:![0-9]+]], [[META1]], i64 0}
diff --git a/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll b/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
index 82ebdaa900fe1..dec130bd533d6 100644
--- a/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
+++ b/llvm/test/Transforms/SROA/mem-par-metadata-sroa.ll
@@ -113,7 +113,7 @@ declare void @llvm.memcpy.p0.p0.i64(ptr nocapture writeonly, ptr nocapture reado
 
 !0 = !{!"clang version 4.0.0 (cfe/trunk 277751)"}
 !1 = distinct !{!1, !2, !{!"llvm.loop.parallel_accesses", !11}}
-!2 = !{!"llvm.loop.vectorize.enable", i1 true}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!4}
 !4 = distinct !{!4, !5, !"_ZNK7ComplexplERKS_: %agg.result"}
 !5 = distinct !{!5, !"_ZNK7ComplexplERKS_"}
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll b/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
index 8c5733e5babed..d334adc2ceb6e 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/combine-parallel-mem-md.ll
@@ -52,5 +52,5 @@ for.end:                                          ; preds = %cond.end
 attributes #0 = { norecurse nounwind uwtable }
 
 !0 = distinct !{!0, !1, !{!"llvm.loop.parallel_accesses", !10}}
-!1 = !{!"llvm.loop.vectorize.enable", i1 true}
+!1 = !{!"llvm.loop.vectorize.enable"}
 !10 = distinct !{}
diff --git a/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll b/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
index 25a09b15ca652..833c76276e458 100644
--- a/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
+++ b/llvm/test/Verifier/llvm.loop-cu-strip-followup.ll
@@ -23,7 +23,7 @@ define hidden void @f() {
 ; CHECK-NOT: DILocation
 ; CHECK: [[LID]] = distinct !{[[LID]], [[VE:![0-9]+]], [[VF:![0-9]+]]}
 ; CHECK-NOT: DILocation
-; CHECK: [[VE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK: [[VE]] = !{!"llvm.loop.vectorize.enable"}
 ; CHECK: [[VF]] = !{!"llvm.loop.vectorize.followup_all", [[FU:![0-9]+]]}
 ; CHECK: [[FU]] = distinct !{[[FU]], [[VD:![0-9]+]], [[UC:![0-9]+]]}
 ; CHECK: [[VD]] = !{!"llvm.loop.isvectorized"}
@@ -40,7 +40,7 @@ define hidden void @f() {
 !8 = !{}
 !9 = distinct !DICompileUnit(language: DW_LANG_C_plus_plus, file: !5, isOptimized: false, runtimeVersion: 0, emissionKind: NoDebug)
 !10 = !DILocation(line: 6, column: 1, scope: !4)
-!11 = !{!"llvm.loop.vectorize.enable", i1 true}
+!11 = !{!"llvm.loop.vectorize.enable"}
 !12 = !{!"llvm.loop.vectorize.followup_all", !13}
 !13 = distinct !{!13, !3, !10, !14, !15}
 !14 = !{!"llvm.loop.isvectorized"}
diff --git a/llvm/test/Verifier/llvm.loop.vectorize.ll b/llvm/test/Verifier/llvm.loop.vectorize.ll
new file mode 100644
index 0000000000000..6f3d354fc2bb1
--- /dev/null
+++ b/llvm/test/Verifier/llvm.loop.vectorize.ll
@@ -0,0 +1,41 @@
+; Test "llvm.loop.vectorize.enable" / "llvm.loop.vectorize.disable"
+; single-operand validation.
+
+; DEFINE: %{VERIFY} = llvm-as -disable-output %t 2>&1
+
+define void @test() {
+entry:
+  br label %body
+body:
+  br i1 0, label %body, label %exit, !llvm.loop !0
+exit:
+  ret void
+}
+!0 = distinct !{!0, !1}
+
+;      BAD: Expected one operand for llvm.loop.vectorize.enable metadata
+
+; Single-operand enable.
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable"}' >> %t
+; RUN: %{VERIFY}
+
+; Single-operand disable.
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.disable"}' >> %t
+; RUN: %{VERIFY}
+
+; Two-operand enable with boolean false (legacy form, now rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable", i1 0}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
+
+; Two-operand enable with boolean true (legacy form, now rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.enable", i1 1}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
+
+; Two-operand disable (rejected).
+; RUN: cat %s > %t
+; RUN: echo '!1 = !{!"llvm.loop.vectorize.disable", i1 0}' >> %t
+; RUN: not %{VERIFY} | FileCheck %s -check-prefix=BAD
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
index fcb11ca4629fa..7f80b5be28b0b 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationImporter.cpp
@@ -285,8 +285,9 @@ static T createIfNonNull(MLIRContext *ctx, const P &...args) {
 }
 
 FailureOr<LoopVectorizeAttr> LoopMetadataConversion::convertVectorizeAttr() {
-  FailureOr<BoolAttr> enable =
-      lookupBoolNode("llvm.loop.vectorize.enable", true);
+  FailureOr<BoolAttr> enable = lookupBooleanUnitNode(
+      "llvm.loop.vectorize.enable", "llvm.loop.vectorize.disable",
+      /*negated=*/true);
   FailureOr<BoolAttr> predicateEnable =
       lookupBoolNode("llvm.loop.vectorize.predicate.enable");
   FailureOr<BoolAttr> scalableEnable =
diff --git a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
index 41ba31106f788..7420142957d4c 100644
--- a/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/LoopAnnotationTranslation.cpp
@@ -103,7 +103,12 @@ void LoopAnnotationConversion::convertFollowupNode(StringRef name,
 }
 
 void LoopAnnotationConversion::convertLoopOptions(LoopVectorizeAttr options) {
-  convertBoolNode("llvm.loop.vectorize.enable", options.getDisable(), true);
+  if (auto disable = options.getDisable()) {
+    if (disable.getValue())
+      addUnitNode("llvm.loop.vectorize.disable");
+    else
+      addUnitNode("llvm.loop.vectorize.enable");
+  }
   convertBoolNode("llvm.loop.vectorize.predicate.enable",
                   options.getPredicateEnable());
   convertBoolNode("llvm.loop.vectorize.scalable.enable",
diff --git a/mlir/test/Target/LLVMIR/Import/import-failure.ll b/mlir/test/Target/LLVMIR/Import/import-failure.ll
index 57c8438bfd362..1b3a666348eed 100644
--- a/mlir/test/Target/LLVMIR/Import/import-failure.ll
+++ b/mlir/test/Target/LLVMIR/Import/import-failure.ll
@@ -160,22 +160,6 @@ end:
 
 ; // -----
 
-; CHECK:      <unknown>
-; CHECK-SAME: warning: expected metadata node llvm.loop.vectorize.enable to hold a boolean value
-; CHECK:      <unknown>
-; CHECK-SAME: warning: unhandled metadata: !0 = distinct !{!0, !1}
-define void @unsupported_loop_annotation(i64 %n, ptr %A) {
-entry:
-  br label %end, !llvm.loop !0
-end:
-  ret void
-}
-
-!0 = distinct !{!0, !1}
-!1 = !{!"llvm.loop.vectorize.enable"}
-
-; // -----
-
 ; CHECK:      <unknown>
 ; CHECK-SAME: warning: expected metadata node llvm.loop.vectorize.width to hold an i32 value
 ; CHECK:      <unknown>
diff --git a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
index 021818ded3d3c..d64a96c1b1422 100644
--- a/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
+++ b/mlir/test/Target/LLVMIR/Import/metadata-loop.ll
@@ -86,7 +86,7 @@ end:
 }
 
 !1 = distinct !{!1, !2, !3, !4, !5, !6, !7, !8}
-!2 = !{!"llvm.loop.vectorize.enable", i1 1}
+!2 = !{!"llvm.loop.vectorize.enable"}
 !3 = !{!"llvm.loop.vectorize.predicate.enable", i1 1}
 !4 = !{!"llvm.loop.vectorize.scalable.enable", i1 0}
 !5 = !{!"llvm.loop.vectorize.width", i32 16}
diff --git a/mlir/test/Target/LLVMIR/loop-metadata.mlir b/mlir/test/Target/LLVMIR/loop-metadata.mlir
index 299bcb481f67f..ada830e103a2c 100644
--- a/mlir/test/Target/LLVMIR/loop-metadata.mlir
+++ b/mlir/test/Target/LLVMIR/loop-metadata.mlir
@@ -55,7 +55,7 @@ llvm.func @vectorizeOptions() {
 // CHECK-DAG: ![[NON_FORCED:[0-9]+]] = !{!"llvm.loop.disable_nonforced"}
 // CHECK-DAG: ![[FOLLOWUP:[0-9]+]] = distinct !{![[FOLLOWUP]], ![[NON_FORCED]]}
 // CHECK-DAG: ![[LOOP_NODE]] = distinct !{![[LOOP_NODE]], !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}, !{{[0-9]+}}}
-// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.enable"}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.predicate.enable", i1 true}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.scalable.enable", i1 false}
 // CHECK-DAG: !{{[0-9]+}} = !{!"llvm.loop.vectorize.width", i32 16}
diff --git a/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir b/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
index bfa57b85245ca..70b5b4b0af0a8 100644
--- a/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-parallel-do-simd.mlir
@@ -12,7 +12,7 @@
 // CHECK-NEXT:   br label %omp_loop.header, !llvm.loop ![[LOOP_ATTR:.*]]
 // CHECK: ![[LOOP_ATTR]] = distinct !{![[LOOP_ATTR]], ![[LPAR:.*]], ![[LVEC:.*]]}
 // CHECK: ![[LPAR]] = !{!"llvm.loop.parallel_accesses", ![[PAR_ACC:.*]]}
-// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable"}
 
 llvm.func @test_parallel_do_simd() {
   %0 = llvm.mlir.constant(1 : i64) : i64
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
index 8da1fab1f48af..bb9aa246ba0fb 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
@@ -83,5 +83,5 @@ module {
     } {linear_var_types = [i32]}
     llvm.return
   }
-  // CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+  // CHECK: !{!"llvm.loop.vectorize.enable"}
 }
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir b/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
index 3e73fd22e69ca..0f2e20cc61aa7 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-reduction-byref.mlir
@@ -95,4 +95,4 @@ llvm.func @_QPsimd_reduction(%arg0: !llvm.ptr {fir.bindc_name = "a", llvm.nocapt
 // CHECK: ![[ACCESS_GROUP]] = distinct !{}
 // CHECK: ![[LOOP]] = distinct !{![[LOOP]], ![[PARALLEL_ACCESS:.*]], ![[VECTORIZE:.*]]}
 // CHECK: ![[PARALLEL_ACCESS]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP]]}
-// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir b/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
index ab3c87cca56ea..a712b847e8557 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-reduction-simple.mlir
@@ -93,4 +93,4 @@ llvm.func @_QPsimd_reduction(%arg0: !llvm.ptr {fir.bindc_name = "a", llvm.nocapt
 // CHECK: ![[ACCESS_GROUP]] = distinct !{}
 // CHECK: ![[LOOP]] = distinct !{![[LOOP]], ![[PARALLEL_ACCESS:.*]], ![[VECTORIZE:.*]]}
 // CHECK: ![[PARALLEL_ACCESS]] = !{!"llvm.loop.parallel_accesses", ![[ACCESS_GROUP]]}
-// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[VECTORIZE]] = !{!"llvm.loop.vectorize.enable"}
diff --git a/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir b/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
index 69d5d225d0515..553d56034d2c1 100644
--- a/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-teams-distribute-parallel-do-simd.mlir
@@ -20,7 +20,7 @@
 
 // CHECK: ![[LOOP_ATTR]] = distinct !{![[LOOP_ATTR]], ![[LPAR:.*]], ![[LVEC:.*]]}
 // CHECK: ![[LPAR]] = !{!"llvm.loop.parallel_accesses", ![[PAR_ACC:.*]]}
-// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable", i1 true}
+// CHECK: ![[LVEC]] = !{!"llvm.loop.vectorize.enable"}
 
 omp.private {type = private} @_QFEi_private_i32 : i32
 llvm.func @test_teams_distribute_parallel_do_simd() {
diff --git a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
index beb300c23651f..3ae41b58f5d21 100644
--- a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
@@ -86,5 +86,5 @@ module {
     } {omp.composite}
     llvm.return
   }
-  // CHECK: !{!"llvm.loop.vectorize.enable", i1 true}
+  // CHECK: !{!"llvm.loop.vectorize.enable"}
 }
diff --git a/polly/lib/CodeGen/CodeGeneration.cpp b/polly/lib/CodeGen/CodeGeneration.cpp
index d9f4a7f78a322..9454108d29bf3 100644
--- a/polly/lib/CodeGen/CodeGeneration.cpp
+++ b/polly/lib/CodeGen/CodeGeneration.cpp
@@ -48,6 +48,22 @@ using namespace polly;
 #include "polly/Support/PollyDebug.h"
 #define DEBUG_TYPE "polly-codegen"
 
+// Append the single-operand "llvm.loop.vectorize.disable" node to \p L,
+// keeping any existing loop metadata intact.
+static void addVectorizeDisableMetadataToLoop(Loop *L) {
+  LLVMContext &Ctx = L->getHeader()->getContext();
+  SmallVector<Metadata *, 4> MDs(1);
+  if (MDNode *LoopID = L->getLoopID())
+    for (unsigned I = 1, E = LoopID->getNumOperands(); I < E; ++I)
+      MDs.push_back(LoopID->getOperand(I));
+  MDs.push_back(
+      MDNode::get(Ctx, {MDString::get(Ctx, "llvm.loop.vectorize.disable")}));
+  MDNode *NewLoopID = MDNode::get(Ctx, MDs);
+  // Set operand 0 to refer to the loop id itself.
+  NewLoopID->replaceOperandWith(0, NewLoopID);
+  L->setLoopID(NewLoopID);
+}
+
 static cl::opt<bool> Verify("polly-codegen-verify",
                             cl::desc("Verify the function generated by Polly"),
                             cl::Hidden, cl::cat(PollyCategory));
@@ -284,7 +300,7 @@ static bool generateCode(Scop &S, IslAstInfo &AI, LoopInfo &LI,
     if (!(CI && CI->isZero())) {
       for (Loop *L : LI.getLoopsInPreorder()) {
         if (S.contains(L))
-          addStringMetadataToLoop(L, "llvm.loop.vectorize.enable", 0);
+          addVectorizeDisableMetadataToLoop(L);
       }
     }
 
diff --git a/polly/lib/CodeGen/IRBuilder.cpp b/polly/lib/CodeGen/IRBuilder.cpp
index d326ffd8c6a2b..b461b72170c77 100644
--- a/polly/lib/CodeGen/IRBuilder.cpp
+++ b/polly/lib/CodeGen/IRBuilder.cpp
@@ -131,11 +131,10 @@ void ScopAnnotator::popLoop(bool IsParallel) {
 static void addVectorizeMetadata(LLVMContext &Ctx,
                                  SmallVector<Metadata *, 3> *Args,
                                  bool EnableLoopVectorizer) {
-  MDString *PropName = MDString::get(Ctx, "llvm.loop.vectorize.enable");
-  ConstantInt *Value =
-      ConstantInt::get(Type::getInt1Ty(Ctx), EnableLoopVectorizer);
-  ValueAsMetadata *PropValue = ValueAsMetadata::get(Value);
-  Args->push_back(MDNode::get(Ctx, {PropName, PropValue}));
+  MDString *PropName =
+      MDString::get(Ctx, EnableLoopVectorizer ? "llvm.loop.vectorize.enable"
+                                              : "llvm.loop.vectorize.disable");
+  Args->push_back(MDNode::get(Ctx, {PropName}));
 }
 
 void addParallelMetadata(LLVMContext &Ctx, SmallVector<Metadata *, 3> *Args,
diff --git a/polly/test/CodeGen/MemAccess/create_arrays.ll b/polly/test/CodeGen/MemAccess/create_arrays.ll
index 8443e0f7be327..29eaff8ad3fab 100644
--- a/polly/test/CodeGen/MemAccess/create_arrays.ll
+++ b/polly/test/CodeGen/MemAccess/create_arrays.ll
@@ -32,7 +32,7 @@
 ; CODEGEN:  store double {{%.*}}, ptr %scevgep34, align 8, !alias.scope !12, !noalias !13
 ;
 ; CODEGEN: !0 = distinct !{!0, !1}
-; CODEGEN: !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; CODEGEN: !1 = !{!"llvm.loop.vectorize.disable"}
 ; CODEGEN: !2 = distinct !{!2, !1}
 ; CODEGEN: !3 = distinct !{!3, !1}
 ; CODEGEN: !4 = !{!5}
diff --git a/polly/test/CodeGen/Metadata/basic_vec_annotate.ll b/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
index e059fea707321..7e0c1de322d16 100644
--- a/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
+++ b/polly/test/CodeGen/Metadata/basic_vec_annotate.ll
@@ -13,9 +13,9 @@
 ; CHECK: polly.stmt.for.body:
 ; CHECK: br {{.*}} !llvm.loop [[POLLY_LOOP:![0-9]+]]
 ; CHECK: [[LOOP]] = distinct !{[[LOOP]], [[META2:![0-9]+]], [[META3:![0-9]+]]}
-; CHECK: [[META3]] = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK: [[META3]] = !{!"llvm.loop.vectorize.disable"}
 ; CHECK: [[POLLY_LOOP]] = distinct !{[[POLLY_LOOP]], {{.*}}}
-; CHECK-DAG: !{!"llvm.loop.vectorize.enable", i1 true}
+; CHECK-DAG: !{!"llvm.loop.vectorize.enable"}
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32"
 target triple = "aarch64-unknown-linux-gnu"
diff --git a/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll b/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
index 8f3c4461f79aa..f232be9892781 100644
--- a/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
+++ b/polly/test/CodeGen/Metadata/fallback_vec_annotate.ll
@@ -4,7 +4,7 @@
 ; Verify vectorization is not disabled when RTC of Polly is false
 
 ; CHECK: attributes {{.*}} = { "polly-optimized" }
-; CHECK-NOT: {{.*}} = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK-NOT: {{.*}} = !{!"llvm.loop.vectorize.disable"}
 
 target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128-Fn32"
 target triple = "aarch64-unknown-linux-android10000"
diff --git a/polly/test/CodeGen/annotated_alias_scopes.ll b/polly/test/CodeGen/annotated_alias_scopes.ll
index 7d2d9038270a9..1f3a07b5519be 100644
--- a/polly/test/CodeGen/annotated_alias_scopes.ll
+++ b/polly/test/CodeGen/annotated_alias_scopes.ll
@@ -11,7 +11,7 @@
 ; SCOPES:      store i32 %{{[._a-zA-Z0-9]*}}, ptr %[[AIdx]], align 4, !alias.scope !10, !noalias !11
 ;
 ; SCOPES: !0 = distinct !{!0, !1}
-; SCOPES: !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; SCOPES: !1 = !{!"llvm.loop.vectorize.disable"}
 ; SCOPES: !2 = !{!3}
 ; SCOPES: !3 = distinct !{!3, !4, !"polly.alias.scope.MemRef_B"}
 ; SCOPES: !4 = distinct !{!4, !"polly.alias.scope.domain"}
diff --git a/polly/test/CodeGen/scev-backedgetaken.ll b/polly/test/CodeGen/scev-backedgetaken.ll
index 09fcfe3e4a09c..df416f3d6f63e 100644
--- a/polly/test/CodeGen/scev-backedgetaken.ll
+++ b/polly/test/CodeGen/scev-backedgetaken.ll
@@ -44,7 +44,7 @@ for.cond.cleanup.loopexit:
 ; CHECK:         load i32, ptr %scevgep, align 4, !alias.scope !2, !noalias !5
 
 ; CHECK:       !0 = distinct !{!0, !1}
-; CHECK:       !1 = !{!"llvm.loop.vectorize.enable", i32 0}
+; CHECK:       !1 = !{!"llvm.loop.vectorize.disable"}
 ; CHECK:       !2 = !{!3}
 ; CHECK:       !3 = distinct !{!3, !4, !"polly.alias.scope.MemRef_d"}
 ; CHECK:       !4 = distinct !{!4, !"polly.alias.scope.domain"}
diff --git a/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll b/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
index 3705c3fd27ed9..977dcda9b2e9a 100644
--- a/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
+++ b/polly/test/ScheduleOptimizer/pattern-matching-based-opts_14.ll
@@ -2,7 +2,7 @@
 ;
 ; Check that we disable the Loop Vectorizer.
 ;
-; CHECK: !{!"llvm.loop.vectorize.enable", i1 false}
+; CHECK: !{!"llvm.loop.vectorize.disable"}
 ;
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-unknown"


From llvmlistbot at llvm.org  Tue Jul 21 03:53:20 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 03:53:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f4fa0.c5504292.78ef3.7fda@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 03:55:25 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Tue, 21 Jul 2026 03:55:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f501d.7270d2fd.a9bd6.931d@mx.google.com>


================
@@ -69,4 +69,18 @@ def SaturationModeAttr : EnumAttr<NVVM_Dialect, SaturationMode, "sat_mode"> {
  let assemblyFormat = "`<` $value `>`";
 }
 
+def AsyncStoreScopeNone : I32EnumCase<"NONE", 0, "none">;
+def AsyncStoreScopeSys : I32EnumCase<"SYS", 1, "sys">;
+def AsyncStoreScopeGpu : I32EnumCase<"GPU", 2, "gpu">;
----------------
durga4github wrote:

We already seem to have a `MemScopeKindAttr`.
Can we not use the same?

>From the global variant below, I see that we need a None option here.
We could extend the existing attr with a None field too..

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 03:57:27 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Tue, 21 Jul 2026 03:57:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f5097.9edb5c9d.280be3.db84@mx.google.com>


================
@@ -3662,6 +3706,42 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
+
+  switch (addrSpace) {
+  case NVVMMemorySpace::Global:
+    if (scope == AsyncStoreScope::SYS) {
+      if (thisOp.getIsMmio())
+        return {llvm::Intrinsic::nvvm_st_async_mmio_sys, {addr, value}};
+      else
----------------
durga4github wrote:

else not required

+
Can a ternary simplify things here?

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 04:02:12 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Tue, 21 Jul 2026 04:02:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f51b4.e624eb8d.4a8a7.5d3d@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
durga4github wrote:

Looking at this Verifier logic, I wonder if we should split the Op into two i.e., st.async.global and st.async.shared 

MMIO/multimem are things that we do not usually associate with shared-memory.
Similarly, mbarrier is not generally used for global-memory stores.

So, the split will naturally simplify the Op(s) and we will not need `none` for the scope-attribute also

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 04:22:41 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Tue, 21 Jul 2026 04:22:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f5681.27b3b003.1f84cc.d7fe@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
Wolfram70 wrote:

We could, as `st.async.shared` itself would be a very small Op lowering to a single intrinsic. But since we call the op `store.async` and have verifier checks in place, should we keep them combined? As we have done for the `prefetch` Op. Please let me know what you think.

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 04:26:40 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:26:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
Message-ID: <llvm.org/llvm/llvm-project/pull/210952@github.com>


https://github.com/FedericoBruzzone created https://github.com/llvm/llvm-project/pull/210952

This closes the stride-verification gap left open by #204611 and #205869 for `vector.maskedload`, `vector.maskedstore`,    `vector.expandload`, and `vector.compressstore`, and uncovers and fixes a real, pre-existing silent miscompile in the SparseVectorizer.


### First Fix

`vector.maskedload`/`maskedstore` lower to [`llvm.masked.load`](https://llvm.org/docs/LangRef.html#llvm-masked-load-intrinsics) / [`llvm.masked.store`](https://llvm.org/docs/LangRef.html#llvm-masked-store-intrinsics), and `vector.expandload`/`compressstore` lower to [`llvm.masked.expandload`](https://llvm.org/docs/LangRef.html#llvm-masked-expandload-intrinsics) / [`llvm.masked.compressstore`](https://llvm.org/docs/LangRef.html#llvm-masked-compressstore-intrinsics).
All four LLVM intrinsics read or write N *consecutive* memory locations starting from a single computed pointer. None of these four ops verified that the memref's most minor dimension actually has unit stride, so a memref like `memref<4xi32, strided<[2], offset: ?>>` passed verification and produced a masked load/store that silently read/wrote the wrong memory.

Fixed by adding a new verifier helper distinct.. It only rejects a most-minor stride that is *statically known* and `!= 1`, while a *dynamic* stride (`?`) is accepted. 
**Note**: This is intentionally more permissive than `vector.load`/`vector.store`, the sparsifier legitimately produces masked loads on memrefs whose last-dim stride is contiguous by construction but not provable at the type level (see below).


### Related fix (SparseVectorization)


When SparseVectorization encounters a _direct_ access like `a[lo:hi]`, it emits a `vector.maskedload` assuming that reading N consecutive indices (`lo, lo+1, ..., lo+vl-1`) means reading N consecutive *bytes* in memory. That's only true if the underlying buffer has stride 1.
At vectorization time the memref type is still `memref<?xindex, strided<[?], offset: ?>>`, fully dynamic, even when the real stride will turn out to be 2 or 3.

The fix adds `hasKnownNonUnitStride()`, which special-cases `sparse_tensor.coordinates` results: instead of trusting the memref type, it reads the stride directly off the source tensor's encoding, which is fully known at compile time this pipeline stage. 

**While all tests pass, this section needs to be carefully reviewed, as I am not very familiar with it (I studied it during this development). I am not sure if it is correct to handle it at this level. 🙏**

AI was used to investigate in this PR.

>From b7efef5b2658bd25f9bd373f69f501cc021e0d0c Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 12:45:52 +0200
Subject: [PATCH] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../Transforms/SparseVectorization.cpp        | 50 +++++++++++++++++
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp      | 50 +++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir         | 54 +++++++++++++++++++
 3 files changed, 154 insertions(+)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index f37083803a2a1..155ae020bf368 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,32 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Unlike vector.load/store, ops such as vector.maskedload/maskedstore and
+/// vector.expandload/compressstore are also used on memrefs whose most minor
+/// dimension has a dynamic stride that is contiguous at runtime but not
+/// provable as such at the type level (e.g., buffers produced by the
+/// sparsifier).
+///
+/// Only reject strides that are statically known to be non-unit: a dynamic
+/// stride is assumed to be safe.
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6325,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6394,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6680,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6702,6 +6743,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..03b8f12d29cad 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1742,6 +1760,24 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1796,6 +1832,24 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :


From llvmlistbot at llvm.org  Tue Jul 21 04:27:16 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:27:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f5794.6c5720b3.2da3b9.d5c3@mx.google.com>


FedericoBruzzone wrote:

**The current state**

| Op | unit-stride check | negative-stride check |
|---|---|---|
| `vector.load` / `vector.store` | ✅ pre-existing (strict) | ✅ (#204611) |
| `vector.maskedload` / `maskedstore` | ✅ **this PR** (lenient: dynamic OK) | ✅ (#205869) |
| `vector.expandload` / `compressstore` | ✅ **this PR** (lenient: dynamic OK) | ✅ **this PR** |
| `vector.gather` / `scatter` | ❌ still permissive (we need to discuss it) | ✅ (#205869) |

Related prior work in this series
- #204309 
- #204611
- #205869

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 04:27:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 04:27:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f5795.1b0626d0.59100.8100@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Federico Bruzzone (FedericoBruzzone)

<details>
<summary>Changes</summary>

This closes the stride-verification gap left open by #<!-- -->204611 and #<!-- -->205869 for `vector.maskedload`, `vector.maskedstore`,    `vector.expandload`, and `vector.compressstore`, and uncovers and fixes a real, pre-existing silent miscompile in the SparseVectorizer.


### First Fix

`vector.maskedload`/`maskedstore` lower to [`llvm.masked.load`](https://llvm.org/docs/LangRef.html#llvm-masked-load-intrinsics) / [`llvm.masked.store`](https://llvm.org/docs/LangRef.html#llvm-masked-store-intrinsics), and `vector.expandload`/`compressstore` lower to [`llvm.masked.expandload`](https://llvm.org/docs/LangRef.html#llvm-masked-expandload-intrinsics) / [`llvm.masked.compressstore`](https://llvm.org/docs/LangRef.html#llvm-masked-compressstore-intrinsics).
All four LLVM intrinsics read or write N *consecutive* memory locations starting from a single computed pointer. None of these four ops verified that the memref's most minor dimension actually has unit stride, so a memref like `memref&lt;4xi32, strided&lt;[2], offset: ?&gt;&gt;` passed verification and produced a masked load/store that silently read/wrote the wrong memory.

Fixed by adding a new verifier helper distinct.. It only rejects a most-minor stride that is *statically known* and `!= 1`, while a *dynamic* stride (`?`) is accepted. 
**Note**: This is intentionally more permissive than `vector.load`/`vector.store`, the sparsifier legitimately produces masked loads on memrefs whose last-dim stride is contiguous by construction but not provable at the type level (see below).


### Related fix (SparseVectorization)


When SparseVectorization encounters a _direct_ access like `a[lo:hi]`, it emits a `vector.maskedload` assuming that reading N consecutive indices (`lo, lo+1, ..., lo+vl-1`) means reading N consecutive *bytes* in memory. That's only true if the underlying buffer has stride 1.
At vectorization time the memref type is still `memref&lt;?xindex, strided&lt;[?], offset: ?&gt;&gt;`, fully dynamic, even when the real stride will turn out to be 2 or 3.

The fix adds `hasKnownNonUnitStride()`, which special-cases `sparse_tensor.coordinates` results: instead of trusting the memref type, it reads the stride directly off the source tensor's encoding, which is fully known at compile time this pipeline stage. 

**While all tests pass, this section needs to be carefully reviewed, as I am not very familiar with it (I studied it during this development). I am not sure if it is correct to handle it at this level. 🙏**

AI was used to investigate in this PR.

---
Full diff: https://github.com/llvm/llvm-project/pull/210952.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp (+50) 
- (modified) mlir/lib/Dialect/Vector/IR/VectorOps.cpp (+50) 
- (modified) mlir/test/Dialect/Vector/invalid.mlir (+54) 


``````````diff
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();
diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index f37083803a2a1..155ae020bf368 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,32 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Unlike vector.load/store, ops such as vector.maskedload/maskedstore and
+/// vector.expandload/compressstore are also used on memrefs whose most minor
+/// dimension has a dynamic stride that is contiguous at runtime but not
+/// provable as such at the type level (e.g., buffers produced by the
+/// sparsifier).
+///
+/// Only reject strides that are statically known to be non-unit: a dynamic
+/// stride is assumed to be safe.
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6325,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6394,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6680,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6702,6 +6743,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..03b8f12d29cad 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1742,6 +1760,24 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1796,6 +1832,24 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :

``````````

</details>


https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 04:29:22 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:29:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f5812.449bfb44.7b862.c40f@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 04:30:38 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:30:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f585e.c56fb102.2d07b8.1b87@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 04:34:23 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:34:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f593f.f3151ff8.37ac76.9301@mx.google.com>


FedericoBruzzone wrote:

> Updated, see [ced689e](https://github.com/llvm/llvm-project/commit/ced689efcae970392f8c8b1da697dc8ae351d352)

Thanks for addressing to my comments. I really appreciate it!
For what it's worth, LTGM 🫶

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 04:36:05 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:36:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f59a5.1b0626d0.59100.95ee@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Wow! Thanks for sharing this valuable information. Unfortunately, I wasn't aware of these details :(
I've seen the issue, nice :D

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 04:36:11 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:36:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f59ab.d76d81af.23b347.0979@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 04:37:22 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:37:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f59f2.27b3b003.1f84cc.dfb4@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 04:43:27 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:43:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f5b5f.b1416da1.3242c8.9f7b@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM % nits, thank you!

https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 04:43:27 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:43:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f5b5f.507fc713.239434.99e7@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 04:43:28 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:43:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f5b60.b4a331f0.d9742.120f@mx.google.com>


================

----------------
banach-space wrote:

Any reason not to use a named Op here?

https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 04:43:28 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:43:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f5b60.7270d2fd.a9bd6.753f@mx.google.com>


================

----------------
banach-space wrote:

Could you update this comment?

https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 04:43:28 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:43:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f5b60.3cce8983.3cf542.fad9@mx.google.com>


================

----------------
banach-space wrote:

OK, so the intent is to remove that test and add a completely new. Makes sense. 

[nit] This would benefit from a note in the summary.

https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 04:44:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 04:44:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f5bb7.1e1ac41d.3969c3.d1d5@mx.google.com>


banach-space wrote:

Thank you for the review, @FedericoBruzzone ! If there are no further comments, I plan to land this tomorrow.

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 04:46:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 04:46:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f72f994 - [mlir][ods] Fix string
 interpolation at end of description (#209744)
Message-ID: <6a5f5c26.28a0c303.361634.0998@mx.google.com>


Author: Luke Hutton
Date: 2026-07-21T12:46:42+01:00
New Revision: f72f994ad894bf82ab8d12f848f583c5d0cf0048

URL: https://github.com/llvm/llvm-project/commit/f72f994ad894bf82ab8d12f848f583c5d0cf0048
DIFF: https://github.com/llvm/llvm-project/commit/f72f994ad894bf82ab8d12f848f583c5d0cf0048.diff

LOG: [mlir][ods] Fix string interpolation at end of description (#209744)

Updates the error streaming string logic to handle the case where string
interpolation used at the end of the description. Previously this could
generate malformed code that would not compile e.g.:
```
"' failed to satisfy constraint: another attribute " << reformat(attr)";
```
With this change the above example would now generate:
```
"' failed to satisfy constraint: another attribute " << reformat(attr) << "";
```

Added: 
    

Modified: 
    mlir/lib/TableGen/CodeGenHelpers.cpp
    mlir/test/mlir-tblgen/constraint-unique.td

Removed: 
    


################################################################################
diff  --git a/mlir/lib/TableGen/CodeGenHelpers.cpp b/mlir/lib/TableGen/CodeGenHelpers.cpp
index c8c419669f3d4..19439eddcb333 100644
--- a/mlir/lib/TableGen/CodeGenHelpers.cpp
+++ b/mlir/lib/TableGen/CodeGenHelpers.cpp
@@ -157,18 +157,16 @@ std::string mlir::tblgen::buildErrorStreamingString(
 
     os << " << " << tgfmt(var, &ctx);
 
-    if (rest.empty())
-      break;
-
-    split = rest.split("{{");
-    if (split.second.empty() &&
-        errorStreamType == ErrorStreamType::InsideOpError) {
+    if (errorStreamType == ErrorStreamType::InsideOpError) {
       // To enable having part of string post, this adds a parenthesis before
       // the last string segment to match the existing one.
-      os << " << (\"" << split.first;
+      os << " << (\"";
     } else {
-      os << " << \"" << split.first;
+      os << " << \"";
     }
+
+    split = rest.split("{{");
+    os << split.first;
     msg = split.second;
   }
 

diff  --git a/mlir/test/mlir-tblgen/constraint-unique.td b/mlir/test/mlir-tblgen/constraint-unique.td
index 5fdaf368c323a..55d6a56e3fc80 100644
--- a/mlir/test/mlir-tblgen/constraint-unique.td
+++ b/mlir/test/mlir-tblgen/constraint-unique.td
@@ -17,7 +17,7 @@ def OtherType : Type<ATypePred, "another type">;
 
 def AnAttrPred : CPred<"attrPred($_self, $_op)">;
 def AnAttr : Attr<AnAttrPred, "an attribute (got {{reformat($_self)}})">;
-def OtherAttr : Attr<AnAttrPred, "another attribute">;
+def OtherAttr : Attr<AnAttrPred, "another attribute {{reformat($_self)}}">;
 
 def ASuccessorPred : CPred<"successorPred($_self, $_op)">;
 def ASuccessor : Successor<ASuccessorPred, "a successor">;
@@ -81,7 +81,7 @@ def OpC : NS_Op<"op_c"> {
 // CHECK:    static ::llvm::LogicalResult [[$O_ATTR_CONSTRAINT:__mlir_ods_local_attr_constraint.*]](
 // CHECK:      if (attr && !((attrPred(attr, *op))))
 // CHECK-NEXT:   return emitError() << "attribute '" << attrName
-// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute";
+// CHECK-NEXT:       << "' failed to satisfy constraint: another attribute " << reformat(attr) << "";
 
 /// Test that a successor contraint was generated.
 // CHECK:    static ::llvm::LogicalResult [[$A_SUCCESSOR_CONSTRAINT:__mlir_ods_local_successor_constraint.*]](


        

From llvmlistbot at llvm.org  Tue Jul 21 04:46:49 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Tue, 21 Jul 2026 04:46:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][ods] Fix string interpolation at end
 of description (PR #209744)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209744@github.com>
Message-ID: <6a5f5c29.c530d079.2d61a6.e20f@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/209744

From llvmlistbot at llvm.org  Tue Jul 21 04:47:17 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:47:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f5c45.1728eaf3.2a2f6.23d8@mx.google.com>


FedericoBruzzone wrote:

> Thank you for the review, @FedericoBruzzone ! If there are no further comments, I plan to land this tomorrow.

It's a pleasure. Go ahead and merge whenever you like :D Everything is perfect on my end!

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 04:53:47 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Tue, 21 Jul 2026 04:53:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f5dcb.8d4c1b95.34a863.e200@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
durga4github wrote:

I would use the TMA Ops (where we have separate Ops for TMA-load and TMA-store) that perform async-data-copy as an analogy for this Op and not the `prefetch` family.

It seems we cannot implement a combined Op without updating the existing' MemAttr` or adding a new one. This seems unnecessary since we can solve it by cleanly splitting the Op.


https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 04:54:19 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Tue, 21 Jul 2026 04:54:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f5deb.b922cc60.312677.49e9@mx.google.com>


https://github.com/durga4github edited https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 04:56:51 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 04:56:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f5e83.7947377b.ab791.d040@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 05:02:55 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 05:02:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f5fef.94610230.10de11.a81f@mx.google.com>


banach-space wrote:

> Note: This is intentionally more permissive than vector.load/vector.store, the sparsifier legitimately produces masked loads on memrefs whose last-dim stride is contiguous by construction but not provable at the type level (see below).

This a bit confusing, TBH. If dynamic stride is considered as "unit" in some scenarios, then it should be considered as "unit" everywhere. Special-casing like this  can be quite problematic. For now, I would at least:
*  Add tests for `vector.load` + `vector.store` with dynamic strides (so that the difference in behaviour is documented through tests)
* Update your table from this [comment](https://github.com/llvm/llvm-project/pull/210952#issuecomment-5033367754) accordingly (it's very useful, thank you!).

Also, your summary is quite long and verbose for a relatively small change like this. All in all, the discussion is relevant, but please remember that the summary will end-up as a commit message. A commit message should provide a _brief_ overview of _what_ and _why_. Here are some additional tips https://chris.beams.io/git-commit (we [reccomend](https://mlir.llvm.org/getting_started/Contributing/) that all contributors to MLIR follow it source in MLIR).

I suggest that you open a GitHub issue for the wider discussion and to track all the relevant PRs. The summary for this change should be much shorter.`

Last, but not least, thank you so much for all the fixes, that is greatly appreciated 🙏🏻 

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 05:04:40 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 05:04:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f6058.dfa3d646.12b476.145f@mx.google.com>


================
@@ -6190,6 +6190,32 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Unlike vector.load/store, ops such as vector.maskedload/maskedstore and
+/// vector.expandload/compressstore are also used on memrefs whose most minor
+/// dimension has a dynamic stride that is contiguous at runtime but not
+/// provable as such at the type level (e.g., buffers produced by the
+/// sparsifier).
----------------
banach-space wrote:

This is the first paragraph documenting this method, yet it doesn't say what the method does. Instead, it discusses the context in which it might be used. Please update.

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 05:31:47 2026
From: llvmlistbot at llvm.org (Srinivasa Ravi)
Date: Tue, 21 Jul 2026 05:31:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a5f66b3.e55dc341.1c6a0.ac17@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
Wolfram70 wrote:

Makes sense! One alternative is to use `OptionalAttr`, but splitting the Ops seems like a more concise solution.

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Tue Jul 21 05:46:50 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Tue, 21 Jul 2026 05:46:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f6a3a.ec04e30f.2307a8.105c@mx.google.com>


================
@@ -2381,7 +2379,7 @@ def Vector_CompressStoreOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$valueToStore,
----------------
dcaballe wrote:

We may want to align the operand order with the rest of the store ops (i.e., valueToStore, mask, base, indices), but that's for another PR.

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 05:46:50 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Tue, 21 Jul 2026 05:46:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f6a3a.39bfcb41.28f5f.b903@mx.google.com>


https://github.com/dcaballe edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 05:46:50 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Tue, 21 Jul 2026 05:46:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f6a3a.8ecc52f4.125f5f.83de@mx.google.com>


https://github.com/dcaballe approved this pull request.

LGTM, thanks!

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 06:00:10 2026
From: llvmlistbot at llvm.org (Emil Pedersen)
Date: Tue, 21 Jul 2026 06:00:10 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5f6d5a.39f23b82.34592b.be01@mx.google.com>


Snowy1803 wrote:

Why exactly is this incompatible with DIArgList? I don't see a good use case for this feature without support for multiple arguments: how often is a branch operator needed, if all values except the condition are constant? I believe that in those cases, the expression can be rewritten in a branchless way anyway.


https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 06:00:45 2026
From: llvmlistbot at llvm.org (Alexey Bataev)
Date: Tue, 21 Jul 2026 06:00:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][math] Fold FPowIOp with
 square-and-multiply to match powi expansion (PR #210982)
Message-ID: <llvm.org/llvm/llvm-project/pull/210982@github.com>


https://github.com/alexey-bataev created https://github.com/llvm/llvm-project/pull/210982

Fold math.fpowi using the base's own floating-point semantics via iterative
square-and-multiply, matching the multiply sequence ExpandPowI builds in
SelectionDAGBuilder. The prior pow/powf-based fold computed in float/double
and rounded differently from the runtime expansion, disagreeing with x**n by ~1 ULP.


>From 3638f27aed6f70e12e2d9860f2941a9e506ec007 Mon Sep 17 00:00:00 2001
From: Alexey Bataev <a.bataev at outlook.com>
Date: Tue, 21 Jul 2026 06:00:31 -0700
Subject: [PATCH] =?UTF-8?q?[=F0=9D=98=80=F0=9D=97=BD=F0=9D=97=BF]=20initia?=
 =?UTF-8?q?l=20version?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Created using spr 1.3.7
---
 mlir/lib/Dialect/Math/IR/MathOps.cpp     | 30 ++++++++++++++----------
 mlir/test/Dialect/Math/canonicalize.mlir | 20 ++++++++++++----
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Dialect/Math/IR/MathOps.cpp b/mlir/lib/Dialect/Math/IR/MathOps.cpp
index b900cb1911759..2fd081282c2aa 100644
--- a/mlir/lib/Dialect/Math/IR/MathOps.cpp
+++ b/mlir/lib/Dialect/Math/IR/MathOps.cpp
@@ -831,23 +831,29 @@ OpFoldResult math::FPowIOp::fold(FoldAdaptor adaptor) {
   return constFoldBinaryOpConditional<FloatAttr, IntegerAttr>(
       adaptor.getOperands(),
       [](const APFloat &base, const APInt &exp) -> std::optional<APFloat> {
-        const llvm::fltSemantics &sem = base.getSemantics();
-        // Fold when the exponent is exactly representable in the
-        // floating-point type of the base.
-        APFloat fExp(sem);
-        if (fExp.convertFromAPInt(exp, /*isSigned=*/true,
-                                  APFloat::rmNearestTiesToEven) !=
-            APFloat::opOK)
-          return {};
-
-        switch (APFloat::SemanticsToEnum(sem)) {
+        switch (APFloat::SemanticsToEnum(base.getSemantics())) {
         case APFloat::Semantics::S_IEEEdouble:
-          return APFloat(pow(base.convertToDouble(), fExp.convertToDouble()));
         case APFloat::Semantics::S_IEEEsingle:
-          return APFloat(powf(base.convertToFloat(), fExp.convertToFloat()));
+          break;
         default:
           return {};
         }
+
+        // Square-and-multiply using the base's own semantics, matching the
+        // multiply sequence ExpandPowI builds in SelectionDAGBuilder.cpp.
+        const llvm::fltSemantics &sem = base.getSemantics();
+        APInt magnitude = exp.abs();
+        APFloat res = APFloat::getOne(sem);
+        APFloat curSquare = base;
+        while (!magnitude.isZero()) {
+          if (magnitude[0])
+            res = res * curSquare;
+          curSquare = curSquare * curSquare;
+          magnitude.lshrInPlace(1);
+        }
+        if (exp.isNegative())
+          res = APFloat::getOne(sem) / res;
+        return res;
       });
 }
 
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..a73a15f34e577 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -638,10 +638,22 @@ func.func @fpowi_fold_vec() -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
-// 16777217 is not exactly representable in f32.
-// CHECK-LABEL: @fpowi_fold_failed
-// CHECK:       math.fpowi
-func.func @fpowi_fold_failed() -> f32 {
+// CHECK-LABEL: @fpowi_fold_const
+// CHECK: %[[cst:.+]] = arith.constant 29.1229134 : f32
+// CHECK: return %[[cst]]
+func.func @fpowi_fold_const() -> f32 {
+  %cst = arith.constant 1.234567 : f32
+  %c16_i32 = arith.constant 16 : i32
+  %0 = math.fpowi %cst, %c16_i32 : f32, i32
+  return %0 : f32
+}
+
+// The fold no longer requires the exponent to be exactly representable as
+// f32: it is used as an integer, so this folds and overflows to +inf.
+// CHECK-LABEL: @fpowi_fold_overflow
+// CHECK: %[[cst:.+]] = arith.constant 0x7F800000 : f32
+// CHECK: return %[[cst]]
+func.func @fpowi_fold_overflow() -> f32 {
   %cst = arith.constant 2.000000e+00 : f32
   %c16777217_i32 = arith.constant 16777217 : i32
   %0 = math.fpowi %cst, %c16777217_i32 : f32, i32


From llvmlistbot at llvm.org  Tue Jul 21 06:05:49 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 21 Jul 2026 06:05:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5f6ead.28a0c303.361634.2ba3@mx.google.com>


================
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
----------------
sahas3 wrote:

IIUC, `TosaTensorRankOf` will now allow unranked tensors too since the base `TosaTensorOf` allows it -- is this change expected?

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Tue Jul 21 06:05:49 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Tue, 21 Jul 2026 06:05:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a5f6ead.3ff6034f.194075.09c4@mx.google.com>


================
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
----------------
sahas3 wrote:

This may cause a breaking change in downstream consumers -- I think before the type will be `RankedTensorType` but now it will be `TensorType`? I think we have some code that implicitly casts to `TypedValue<RankedTensorType>` which won't work anymore. Easily fixable but I don't know if there's some announcement to be made about this change.

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Tue Jul 21 06:12:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 06:12:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support multi-rank array
	reductions (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a5f7045.1b0626d0.59100.5d19@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210853

>From 2d005d7cf11a32e49f59fafddd6ccc1d39991825 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:05:29 -0700
Subject: [PATCH 01/10] [mlir][OpenACC] Support multi-rank array reductions

Initialize static rank-N per-thread accumulators and delinearize flattened reduction indices before accessing them.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 61 +++++++++---
 .../acc-cg-to-gpu-reduction-array.mlir        | 94 +++++++++++++++++++
 2 files changed, 142 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 77fa82ba6bb4e..44ad1f3154174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,17 +736,28 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static rank-1");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
-  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
   Value step = arith::ConstantIndexOp::create(b, loc, 1);
-  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
-  OpBuilder::InsertionGuard g(b);
-  b.setInsertionPoint(forOp.getBody()->getTerminator());
-  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+  SmallVector<Value> indices;
+  auto buildLoopNest = [&](auto &&self, unsigned dim) -> void {
+    if (dim == baseTy.getRank()) {
+      memref::StoreOp::create(b, loc, ident, alloca, indices);
+      return;
+    }
+
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+    OpBuilder::InsertionGuard g(b);
+    b.setInsertionPoint(forOp.getBody()->getTerminator());
+    indices.push_back(forOp.getInductionVar());
+    self(self, dim + 1);
+    indices.pop_back();
+  };
+  buildLoopNest(buildLoopNest, 0);
 }
 
 std::optional<int64_t>
@@ -3244,10 +3255,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  if (!memref)
+  if (!memrefTy) {
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
-  if (memrefTy.getRank() != 1)
-    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() == 0) {
+    (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
+    return;
+  }
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
@@ -3355,9 +3375,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     OpBuilder::InsertionGuard guard(rewriter);
     rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
     Value iv = forOp.getInductionVar();
-    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
-    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
-                         ValueRange{iv},
+    SmallVector<Value> indices{iv};
+    if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
+      indices.resize(memrefTy.getRank());
+      Value linearIndex = iv;
+      for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
+        indices[dim] =
+            arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
+        if (dim != 0)
+          linearIndex =
+              arith::DivUIOp::create(rewriter, loc, linearIndex, dimSize);
+      }
+    }
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, indices);
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(), indices,
                          /*isPerThreadPrivateTarget=*/true);
   }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index bfb829c901af3..a008c84108f05 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -134,3 +134,97 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_array_reduction
+// CHECK: %[[R2_ALLOCA:.*]] = memref.alloca() : memref<2x3xi32>
+// CHECK: scf.for %[[R2_I:.*]] =
+// CHECK:   scf.for %[[R2_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[R2_ALLOCA]][%[[R2_I]], %[[R2_J]]] : memref<2x3xi32>
+// CHECK: scf.for %[[R2_LINEAR:.*]] =
+// CHECK: %[[R2_J_IDX:.*]] = arith.remui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+
+func.func @rank_two_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x3xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_three_array_reduction
+// CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_I:.*]] =
+// CHECK:   scf.for %[[R3_J:.*]] =
+// CHECK:     scf.for %[[R3_K:.*]] =
+// CHECK:       memref.store %{{.*}}, %[[R3_ALLOCA]][%[[R3_I]], %[[R3_J]], %[[R3_K]]] : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_LINEAR:.*]] =
+// CHECK: %[[R3_K_IDX:.*]] = arith.remui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_PLANE:.*]] = arith.divui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_J_IDX:.*]] = arith.remui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+
+func.func @rank_three_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x2x2xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x2x2xi32>>) {
+    %c8 = arith.constant 8 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x2x2xi32>>) -> memref<2x2x2xi32>
+    %bounds = acc.bounds extent(%c8 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x2x2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
+// invalid indexing operations.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_zero_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @rank_zero_array_reduction(%local: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
+    %extent = arith.constant 1 : index
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 1a70c1679ce1c49a2e54fc8ed501ba06fa3d7234 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:57 -0700
Subject: [PATCH 02/10] [mlir][acc] Use dynamic stack storage for
 thread-private arrays

Avoid grid-sized backing allocations for dynamic per-thread storage while preserving shared lowering for oversized static arrays.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 +++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 41 ++++++++++++----
 3 files changed, 90 insertions(+), 28 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 44ad1f3154174..7135ca9177153 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,8 +736,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static ranked");
+  assert(baseTy.getRank() > 0 &&
+         "per-thread array reduction accumulator must have positive rank");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
@@ -749,7 +749,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
       return;
     }
 
-    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    Value ub =
+        baseTy.isDynamicDim(dim)
+            ? memref::DimOp::create(b, loc, alloca, dim).getResult()
+            : arith::ConstantIndexOp::create(b, loc, baseTy.getDimSize(dim))
+                  .getResult();
     auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
     OpBuilder::InsertionGuard g(b);
     b.setInsertionPoint(forOp.getBody()->getTerminator());
@@ -2494,9 +2498,15 @@ void ACCCGToGPULowering::processPrivateLocal(
     // the accumulate can reduce each element across threads.
     acc::ReductionAccumulateArrayOp arrayAccum =
         perThreadArrayReductionAccum(privateLocal.getResult());
-    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+    bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
+    bool canUseDynamicAlloca =
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -2583,9 +2593,15 @@ void ACCCGToGPULowering::processPrivateLocal(
   acc::ReductionAccumulateArrayOp arrayAccum =
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    bool canUseDynamicAlloca =
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -3263,12 +3279,6 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
     return;
   }
-  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
-    (void)accSupport.emitNYI(loc,
-                             "reduction: dynamic multi-rank accumulate array");
-    return;
-  }
-
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3377,13 +3387,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
-      assert(memrefTy.hasStaticShape() &&
-             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize = arith::ConstantIndexOp::create(
-            rewriter, loc, memrefTy.getDimSize(dim));
+        Value dimSize =
+            memrefTy.isDynamicDim(dim)
+                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
+                : arith::ConstantIndexOp::create(rewriter, loc,
+                                                 memrefTy.getDimSize(dim))
+                      .getResult();
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index 281eae4b5b0f7..b186720512991 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,3 +23,32 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
+
+// Dynamic thread-private storage is allocated in the launch body. A
+// grid-wide backing allocation would scale with the logical block count.
+//
+// CHECK-LABEL: func.func @dynamic_threadprivate
+// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
+// CHECK:       gpu.launch
+// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
+// CHECK-NOT:     acc.unwrap_private
+
+func.func @dynamic_threadprivate(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
+      : (index) -> !acc.private_type<memref<?xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg = %private, %extent = %n)
+      : (!acc.private_type<memref<?xi32>>, index) {
+    %c0 = arith.constant 0 : index
+    %c0_i32 = arith.constant 0 : i32
+    %local = acc.private_local %arg
+        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
+    memref.store %c0_i32, %local[%c0] : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index a008c84108f05..f287ae054a846 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -192,25 +192,46 @@ func.func @rank_three_array_reduction() {
   return
 }
 
-// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
-// invalid indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
 // CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[DYN_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_I:.*]] =
+// CHECK:   scf.for %[[DYN_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_LINEAR:.*]] =
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: arith.remui %[[DYN_LINEAR]]
+// CHECK: arith.divui %[[DYN_LINEAR]]
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
   return
 }
 
+// Rank-zero accumulators must not generate invalid indexing operations.
 // CHECK-LABEL: func.func @rank_zero_array_reduction
 // CHECK: gpu.launch
 // CHECK-NOT: memref.load

>From 94cd48df3ac887a669b4f7c4cd871226d7014566 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:04:45 -0700
Subject: [PATCH 03/10] [mlir][OpenACC] Handle dynamic reductions on all thread
 dimensions

Classify dynamic accumulators using every GPU thread dimension so rank-N reductions consistently receive per-thread storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  7 +++--
 .../acc-cg-to-gpu-reduction-array.mlir        | 29 ++++---------------
 2 files changed, 9 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 7135ca9177153..e353f70939699 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -2500,7 +2500,8 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
@@ -2594,7 +2595,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
@@ -3334,7 +3335,7 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   } else {
     isPerThreadPrivate = llvm::any_of(
         op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isThreadX(); });
+        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index f287ae054a846..be91cc6762da6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -211,41 +211,22 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_x]>}
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
-
-// Rank-zero accumulators must not generate invalid indexing operations.
-// CHECK-LABEL: func.func @rank_zero_array_reduction
-// CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @rank_zero_array_reduction(%local: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
-    %extent = arith.constant 1 : index
-    %bounds = acc.bounds extent(%extent : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}

>From caf6ec726262faae429717edb8cadc1b6cd00abc Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:20:44 -0700
Subject: [PATCH 04/10] [mlir][OpenACC] Find array reductions through
 descriptor chains

Trace accumulate operands back to private storage so descriptor construction cannot bypass per-thread allocation and initialization.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 23 ++++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 35 +++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index e353f70939699..cc3b8910ad765 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -727,6 +727,29 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
+
+  // Descriptor construction may obscure the forward use chain. Fall back to
+  // tracing each accumulate operand back to this private local.
+  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
+  if (!privateLocal)
+    return nullptr;
+  acc::ComputeRegionOp computeRegion =
+      privateLocal->getParentOfType<acc::ComputeRegionOp>();
+  if (!computeRegion)
+    return nullptr;
+  acc::ReductionAccumulateArrayOp result;
+  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
+    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
+      return WalkResult::advance();
+    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
+                                  [](auto pd) { return pd.isAnyThread(); });
+    if (!hasThread || !reductionHasBlockContext(accArr))
+      return WalkResult::advance();
+    result = accArr;
+    return WalkResult::interrupt();
+  });
+  if (result)
+    return result;
   return nullptr;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index be91cc6762da6..ae9ea1e4c95bb 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -230,3 +230,38 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[INDIRECT_I:.*]] =
+// CHECK:   scf.for %[[INDIRECT_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor_value = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 3790839be0295d1bd7f4ca3bbd151aac1edd5a9a Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:34:24 -0700
Subject: [PATCH 05/10] [mlir][OpenACC] Allocate dynamic privates for any
 thread dimension

Use privatization parallel dimensions so descriptor indirection cannot force dynamic thread-private backing arrays into grid-sized storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 +++++++------------
 .../acc-cg-to-gpu-reduction-array.mlir        | 34 ++++++++++---
 2 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index cc3b8910ad765..99315cad34e1c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,6 +383,14 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
+/// True when \p privatize has any thread-level parallelism.
+static bool isThreadPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
+  return false;
+}
+
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -728,28 +736,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     }
   }
 
-  // Descriptor construction may obscure the forward use chain. Fall back to
-  // tracing each accumulate operand back to this private local.
-  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
-  if (!privateLocal)
-    return nullptr;
-  acc::ComputeRegionOp computeRegion =
-      privateLocal->getParentOfType<acc::ComputeRegionOp>();
-  if (!computeRegion)
-    return nullptr;
-  acc::ReductionAccumulateArrayOp result;
-  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
-    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
-      return WalkResult::advance();
-    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
-                                  [](auto pd) { return pd.isAnyThread(); });
-    if (!hasThread || !reductionHasBlockContext(accArr))
-      return WalkResult::advance();
-    result = accArr;
-    return WalkResult::interrupt();
-  });
-  if (result)
-    return result;
   return nullptr;
 }
 
@@ -2523,12 +2509,13 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((isThreadPrivate || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (isThreadPrivate || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2618,12 +2605,13 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((parDim.isThreadX() || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (parDim.isThreadX() || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index ae9ea1e4c95bb..b9fdf6ecabcb6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -245,17 +245,39 @@ func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %private_data = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
+        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
+            %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>,
+           !acc.private_type<memref<1xindex>>, index) {
+      %c0 = arith.constant 0 : index
+      %c1_k = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      %local_data = acc.private_local %arg_data
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %descriptor_value = builtin.unrealized_conversion_cast %local
-          : memref<?x?xi32> to memref<?x?xi32>
+      %local_descriptor = acc.private_local %arg_descriptor
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
+      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
+      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
+      scf.for %i = %c0 to %m_dim step %c1_k {
+        scf.for %j = %c0 to %n_dim step %c1_k {
+          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
+        }
+      }
+      %descriptor = builtin.unrealized_conversion_cast %local_data
+          : memref<?x?xi32> to index
+      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
+      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
+      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
+          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
           : memref<?x?xi32>

>From affc0c39172f713075ac1912eb631d85b4ce6b56 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:24 -0700
Subject: [PATCH 06/10] [mlir][OpenACC] Preserve worker-private array storage

Restrict generated stack reductions to thread_x and keep thread_y/thread_z arrays indexed or shared, including explicit dynamic shared memory.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  50 ++++----
 .../acc-cg-to-gpu-reduction-array.mlir        | 107 +++++++++++-------
 2 files changed, 86 insertions(+), 71 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 99315cad34e1c..48e7d13730eeb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,14 +383,6 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
-/// True when \p privatize has any thread-level parallelism.
-static bool isThreadPrivatize(PrivatizeOp privatize) {
-  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
-    return llvm::any_of(parDimsAttr.getArray(),
-                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
-  return false;
-}
-
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -709,7 +701,7 @@ static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
 }
 
 /// Returns the array reduction accumulate (through cast/view ops) that \p v
-/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// feeds if it needs per-thread storage: its par_dims include thread_x
 /// and it has block context so the cross-thread all_reduce is well defined.
 static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   SmallVector<Value> worklist{v};
@@ -721,10 +713,10 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     for (Operation *user : cur.getUsers()) {
       if (acc::ReductionAccumulateArrayOp accArr =
               dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
-        bool hasThread = false;
+        bool hasThreadX = false;
         for (auto pd : accArr.getParDims().getArray())
-          hasThread |= pd.isAnyThread();
-        if (hasThread && reductionHasBlockContext(accArr))
+          hasThreadX |= pd.isThreadX();
+        if (hasThreadX && reductionHasBlockContext(accArr))
           return accArr;
         continue;
       }
@@ -735,7 +727,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
-
   return nullptr;
 }
 
@@ -2509,13 +2500,11 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (isThreadPrivate || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2605,13 +2594,12 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((parDim.isThreadX() || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -3337,16 +3325,20 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   // it is too large. For a dynamically-shaped accumulator the type conveys no
   // size, so classify from par_dims (which the producer sets to the reduction's
   // actual parallel scope): a thread dimension means per-thread storage.
+  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
+  bool isExplicitlyShared =
+      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
   bool isPerThreadPrivate;
   if (memrefTy.hasStaticShape()) {
-    Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
     isPerThreadPrivate =
-        !isa_and_nonnull<memref::AllocOp>(rootOp) &&
+        !isExplicitlyShared &&
         canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
   } else {
-    isPerThreadPrivate = llvm::any_of(
-        op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
+    isPerThreadPrivate = !isExplicitlyShared &&
+                         llvm::any_of(op.getParDims().getArray(),
+                                      [](mlir::acc::GPUParallelDimAttr d) {
+                                        return d.isThreadX();
+                                      });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index b9fdf6ecabcb6..269e0a1287046 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,27 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// CHECK-LABEL: func.func @array_reduction_dynamic_shared
+// CHECK: acc.gpu_shared_memory
+// CHECK-NOT: gpu.all_reduce
+func.func @array_reduction_dynamic_shared(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%ext = %n) : (index) {
+    %shared = acc.gpu_shared_memory(%ext)
+        {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
+        : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
@@ -162,6 +183,33 @@ func.func @rank_two_array_reduction() {
   return
 }
 
+// CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @rank_two_array_reduction_thread_y() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %kty = %ty)
+      ins(%arg = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg
+        {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+        : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<2x3xi32>
+        {par_dims = #acc<par_dims[block_x, thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @rank_three_array_reduction
 // CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
 // CHECK: scf.for %[[R3_I:.*]] =
@@ -211,75 +259,50 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
 
-// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// Dynamic worker-level storage must remain indexed by thread_y instead of
+// becoming one stack allocation per physical CUDA thread.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
-// CHECK-NOT: acc.unwrap_private
-// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
-// CHECK: scf.for %[[INDIRECT_I:.*]] =
-// CHECK:   scf.for %[[INDIRECT_J:.*]] =
-// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
-// CHECK: gpu.all_reduce add
-func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private_data = acc.privatize(%m, %n)
-      [#acc<par_dims[block_x, thread_y]>]
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
-      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
-            %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>,
-           !acc.private_type<memref<1xindex>>, index) {
-      %c0 = arith.constant 0 : index
-      %c1_k = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      %local_data = acc.private_local %arg_data
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %local_descriptor = acc.private_local %arg_descriptor
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
-          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
-      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
-      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
-      scf.for %i = %c0 to %m_dim step %c1_k {
-        scf.for %j = %c0 to %n_dim step %c1_k {
-          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
-        }
-      }
-      %descriptor = builtin.unrealized_conversion_cast %local_data
-          : memref<?x?xi32> to index
-      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
-      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
-      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
-          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield

>From 3bba5f0a8722c8926daa522949668e420f5ea9aa Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:33:43 -0700
Subject: [PATCH 07/10] [mlir][OpenACC] Track array reduction storage
 provenance

Classify accumulators from their backing allocation through casts so shared storage is not reduced again and private storage is initialized correctly. Keep dynamic stack allocation reduction-specific and honor byte strides.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 71 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 --------
 .../acc-cg-to-gpu-reduction-array.mlir        | 61 ++++++++++------
 3 files changed, 82 insertions(+), 79 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 48e7d13730eeb..84243de10830a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -317,6 +317,17 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     return false;
   }
 
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
+    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
+        castOp->getOperand(0) == trackedOperand &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      passThroughResults.push_back(castOp->getResult(0));
+      return true;
+    }
+    return false;
+  }
+
   // Partial-entity accesses (e.g. array element or field access) forward the
   // base entity through to their results, so treat them as pass-through when
   // the base entity is the value being tracked.
@@ -341,6 +352,14 @@ static Value unwrapMemRefConversion(Value v) {
         continue;
       }
     }
+    if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+        castOp && castOp->getNumOperands() == 1 &&
+        castOp->getNumResults() == 1 &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      v = castOp->getOperand(0);
+      continue;
+    }
     break;
   }
   return v;
@@ -730,6 +749,13 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   return nullptr;
 }
 
+static bool
+canUseDynamicReductionAlloca(MemRefType baseTy, acc::PrivatizeOp privatizeOp,
+                             acc::ReductionAccumulateArrayOp arrayAccum) {
+  return arrayAccum && !baseTy.hasStaticShape() &&
+         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+}
+
 /// Store the reduction identity to every element of a freshly allocated
 /// per-thread array accumulator so all lanes start from identity (the original
 /// init loop may only run on one lane).
@@ -2500,8 +2526,7 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -2594,9 +2619,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((parDim.isThreadX() || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -3318,28 +3341,10 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   }
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
-  // accumulator copy. For a statically-shaped accumulator, classify from the
-  // operand: an explicit shared allocation is block-shared regardless of size,
-  // and anything else (a per-thread stack alloca, or a view over one) is
-  // per-thread when it fits the per-thread stack budget and block-shared when
-  // it is too large. For a dynamically-shaped accumulator the type conveys no
-  // size, so classify from par_dims (which the producer sets to the reduction's
-  // actual parallel scope): a thread dimension means per-thread storage.
+  // accumulator copy. Classify the operand from its underlying allocation
+  // rather than its shape or par_dims: only memref.alloca is thread-private.
   Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isExplicitlyShared =
-      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
-  bool isPerThreadPrivate;
-  if (memrefTy.hasStaticShape()) {
-    isPerThreadPrivate =
-        !isExplicitlyShared &&
-        canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
-  } else {
-    isPerThreadPrivate = !isExplicitlyShared &&
-                         llvm::any_of(op.getParDims().getArray(),
-                                      [](mlir::acc::GPUParallelDimAttr d) {
-                                        return d.isThreadX();
-                                      });
-  }
+  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
@@ -3367,7 +3372,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
   Value lb =
       boundsOp.getLowerbound() ? toIndex(boundsOp.getLowerbound()) : zero;
-  Value step = boundsOp.getStride() ? toIndex(boundsOp.getStride()) : one;
+  Value step = one;
+  if (Value stride = boundsOp.getStride()) {
+    step = toIndex(stride);
+    if (boundsOp.getStrideInBytes()) {
+      int64_t elementSize =
+          getElementSizeInBytes(loc, memrefTy.getElementType());
+      if (elementSize == 0)
+        return;
+      Value elementSizeValue =
+          arith::ConstantIndexOp::create(rewriter, loc, elementSize);
+      step = arith::DivUIOp::create(rewriter, loc, step, elementSizeValue);
+    }
+  }
   // Exclusive upper bound. `extent` counts elements, so the span is
   // `extent * step` (for the common unit-stride case step is 1); fall back to
   // the inclusive upperbound when no extent is given.
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index b186720512991..281eae4b5b0f7 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,32 +23,3 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
-
-// Dynamic thread-private storage is allocated in the launch body. A
-// grid-wide backing allocation would scale with the logical block count.
-//
-// CHECK-LABEL: func.func @dynamic_threadprivate
-// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
-// CHECK:       gpu.launch
-// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
-// CHECK-NOT:     acc.unwrap_private
-
-func.func @dynamic_threadprivate(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
-      : (index) -> !acc.private_type<memref<?xi32>>
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-      ins(%arg = %private, %extent = %n)
-      : (!acc.private_type<memref<?xi32>>, index) {
-    %c0 = arith.constant 0 : index
-    %c0_i32 = arith.constant 0 : i32
-    %local = acc.private_local %arg
-        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
-    memref.store %c0_i32, %local[%c0] : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 269e0a1287046..56469f801f026 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -85,7 +85,8 @@ func.func @array_reduction_small_shared() {
 }
 
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
-// CHECK: acc.gpu_shared_memory
+// CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
+// CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]
 // CHECK-NOT: gpu.all_reduce
 func.func @array_reduction_dynamic_shared(%n: index) {
   %c1 = arith.constant 1 : index
@@ -96,8 +97,11 @@ func.func @array_reduction_dynamic_shared(%n: index) {
     %shared = acc.gpu_shared_memory(%ext)
         {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
         : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %cast = builtin.unrealized_conversion_cast %shared
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        to memref<?xi32, #gpu.address_space<workgroup>>
     %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+    acc.reduction_accumulate_array %cast bounds(%bounds) <add>
         : memref<?xi32, #gpu.address_space<workgroup>>
         {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -108,11 +112,13 @@ func.func @array_reduction_dynamic_shared(%n: index) {
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
-// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[BYTE_STEP:.*]] = arith.constant 8 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[ELEMENT_SIZE:.*]] = arith.constant 4 : index
+// CHECK: %[[ELEMENT_STEP:.*]] = arith.divui %[[BYTE_STEP]], %[[ELEMENT_SIZE]] : index
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[ELEMENT_STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
-// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[ELEMENT_STEP]]
 func.func @array_reduction_strided_extent() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -120,11 +126,11 @@ func.func @array_reduction_strided_extent() {
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
   acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
     %c1_b = arith.constant 1 : index
-    %c2 = arith.constant 2 : index
+    %c8 = arith.constant 8 : index
     %c3 = arith.constant 3 : index
     %local = memref.alloca() : memref<8xi32>
     %bounds = acc.bounds lowerbound(%c1_b : index) extent(%c3 : index)
-        stride(%c2 : index)
+        stride(%c8 : index) {strideInBytes = true}
     acc.reduction_accumulate_array %local bounds(%bounds) <add>
         : memref<8xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -132,13 +138,14 @@ func.func @array_reduction_strided_extent() {
   return
 }
 
-// A dynamically-shaped accumulator (a strided view whose type conveys no size)
-// is classified per-thread from par_dims: a thread dimension means per-thread
-// storage, so lowering emits the per-element gpu.all_reduce.
+// A dynamically-shaped view over an external buffer is shared even when the
+// accumulate includes thread_x. Storage provenance, not par_dims, controls
+// whether gpu.all_reduce is valid.
 //
 // CHECK-LABEL: func.func @array_reduction_dynamic_par_dims
-// CHECK: scf.for
-// CHECK: gpu.all_reduce add
+// CHECK: memref.reinterpret_cast
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
 func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -186,6 +193,7 @@ func.func @rank_two_array_reduction() {
 // CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
+// CHECK: acc.gpu_shared_memory
 // CHECK-NOT: gpu.all_reduce
 // CHECK: gpu.terminator
 func.func @rank_two_array_reduction_thread_y() {
@@ -247,12 +255,13 @@ func.func @rank_three_array_reduction() {
 // CHECK: scf.for %[[DYN_I:.*]] =
 // CHECK:   scf.for %[[DYN_J:.*]] =
 // CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: %[[DYN_DESC:.*]] = builtin.unrealized_conversion_cast %[[DYN_ALLOCA]]
 // CHECK: scf.for %[[DYN_LINEAR:.*]] =
-// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.dim %[[DYN_DESC]]
 // CHECK: arith.remui %[[DYN_LINEAR]]
 // CHECK: arith.divui %[[DYN_LINEAR]]
-// CHECK: memref.dim %[[DYN_ALLOCA]]
-// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: memref.dim %[[DYN_DESC]]
+// CHECK: memref.load %[[DYN_DESC]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
 // CHECK: gpu.all_reduce add
 func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
@@ -269,8 +278,10 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
       %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+      acc.reduction_accumulate_array %descriptor bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
@@ -279,32 +290,36 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   return
 }
 
-// Dynamic worker-level storage must remain indexed by thread_y instead of
+// Dynamic worker-level storage must remain indexed by thread_y/thread_z instead
 // becoming one stack allocation per physical CUDA thread.
-// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_yz
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
 // CHECK-NOT: gpu.all_reduce
+// CHECK: %[[WORKER_STORAGE:.*]] = memref.view %{{.*}}[%{{.*}}][%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}] : memref<?xi8> to memref<?x?x?x?x?xi32>
+// CHECK: memref.subview %[[WORKER_STORAGE]][%{{.*}}, %{{.*}}, %{{.*}}, 0, 0] [1, 1, 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1, 1]
 // CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
+func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tz = acc.par_width %c128 {par_dim = #acc.par_dim<thread_z>}
+  %private = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y, thread_z]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty, %ktz = %tz)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
       acc.yield
     } {origin = "acc.parallel"}
   }

>From 08888a2115abad8c630b5fab31617fcc3e48821e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:57:42 -0700
Subject: [PATCH 08/10] [mlir][OpenACC] Test partial multi-rank reduction
 bounds

Cover delinearization with partial bounds and a non-identity memref layout.
---
 .../acc-cg-to-gpu-reduction-array.mlir        | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 56469f801f026..357f1e14d7a49 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -325,3 +325,35 @@ func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_partial_bounds_strided_layout
+// CHECK: %[[LB:.*]] = arith.constant 5 : index
+// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
+// CHECK: %[[LAYOUT_ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
+// CHECK: scf.for %[[LINEAR:.*]] = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: %[[COL:.*]] = arith.remui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW_LINEAR:.*]] = arith.divui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW:.*]] = arith.remui %[[ROW_LINEAR]], %{{.*}} : index
+// CHECK: memref.load %[[LAYOUT_ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
+func.func @rank_two_partial_bounds_strided_layout() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
+    %lb = arith.constant 5 : index
+    %step = arith.constant 2 : index
+    %extent = arith.constant 3 : index
+    %local = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+    %bounds = acc.bounds lowerbound(%lb : index) extent(%extent : index)
+        stride(%step : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<3x4xi32, strided<[8, 2]>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 35c5c495bad0d3e902e6b32cdb1a3365bd378b72 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 22:44:28 -0700
Subject: [PATCH 09/10] [mlir][OpenACC] Preserve descriptor-backed reduction
 provenance

Trace unique stores and loads so descriptor-backed thread-private accumulators are initialized and reduced correctly without re-reducing indexed shared storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 155 +++++++++++++++++-
 .../acc-cg-to-gpu-reduction-array.mlir        |  29 ++++
 2 files changed, 181 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 84243de10830a..61ba40247d8b0 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -339,6 +339,119 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     }
     return false;
   }
+
+  bool hasMemoryEffects = false;
+  if (auto effects = dyn_cast<MemoryEffectOpInterface>(userOp)) {
+    SmallVector<MemoryEffects::EffectInstance> instances;
+    effects.getEffects(instances);
+    hasMemoryEffects = !instances.empty();
+  }
+  if (userOp->getNumResults() == 1 &&
+      llvm::is_contained(userOp->getOperands(), trackedOperand) &&
+      !hasMemoryEffects) {
+    passThroughResults.push_back(userOp->getResult(0));
+    return true;
+  }
+  return false;
+}
+
+template <typename Effect>
+static bool hasEffectOnValue(Operation *op, Value value) {
+  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
+  if (!effects)
+    return false;
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  return llvm::any_of(instances, [&](const auto &instance) {
+    return isa<Effect>(instance.getEffect()) && instance.getValue() == value;
+  });
+}
+
+static Value getReadAddress(Operation *op) {
+  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
+  if (!effects)
+    return {};
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  Value address;
+  for (const auto &instance : instances) {
+    Value value = instance.getValue();
+    if (!isa<MemoryEffects::Read>(instance.getEffect()) || !value)
+      continue;
+    if (address && address != value)
+      return {};
+    address = value;
+  }
+  return address;
+}
+
+/// Returns the unique pointer-like value stored to \p address before \p before.
+static Value getUniqueStoredPointerValue(Value address, Operation *before) {
+  Operation *storedBy = nullptr;
+  Value stored;
+  SmallVector<Value> worklist{address};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value alias = worklist.pop_back_val();
+    if (!seen.insert(alias).second)
+      continue;
+    for (Operation *user : alias.getUsers()) {
+      if (hasEffectOnValue<MemoryEffects::Write>(user, alias)) {
+        if (storedBy || user->getBlock() != before->getBlock() ||
+            !user->isBeforeInBlock(before))
+          return {};
+        Value candidate;
+        for (Value operand : user->getOperands()) {
+          if (operand == alias)
+            continue;
+          if (candidate)
+            return {};
+          candidate = operand;
+        }
+        if (!candidate)
+          return {};
+        storedBy = user;
+        stored = candidate;
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, alias, through))
+        worklist.append(through.begin(), through.end());
+    }
+  }
+  return stored;
+}
+
+/// Tracks a uniquely stored pointer-like value through its subsequent load.
+static bool
+getMemoryPassThroughResults(Operation *userOp, Value trackedOperand,
+                            SmallVectorImpl<Value> &passThroughResults) {
+  if (hasEffectOnValue<MemoryEffects::Read>(userOp, trackedOperand) &&
+      userOp->getNumResults() == 1) {
+    passThroughResults.push_back(userOp->getResult(0));
+    return true;
+  }
+
+  if (!llvm::is_contained(userOp->getOperands(), trackedOperand))
+    return false;
+  auto effects = dyn_cast<MemoryEffectOpInterface>(userOp);
+  if (!effects)
+    return false;
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  for (const auto &instance : instances) {
+    Value target = instance.getValue();
+    if (!isa<MemoryEffects::Write>(instance.getEffect()) || !target ||
+        target == trackedOperand)
+      continue;
+    unsigned writers = llvm::count_if(target.getUsers(), [&](Operation *user) {
+      return hasEffectOnValue<MemoryEffects::Write>(user, target);
+    });
+    if (writers != 1)
+      return false;
+    passThroughResults.push_back(target);
+    return true;
+  }
   return false;
 }
 
@@ -742,6 +855,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
       SmallVector<Value> through;
       if (getPassThroughResults(user, cur, through))
         worklist.append(through.begin(), through.end());
+      else if (getMemoryPassThroughResults(user, cur, through))
+        worklist.append(through.begin(), through.end());
       else if (isa<ViewLikeOpInterface>(user))
         worklist.append(user->result_begin(), user->result_end());
     }
@@ -790,6 +905,41 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
   buildLoopNest(buildLoopNest, 0);
 }
 
+/// True when \p memref resolves to storage owned by the current thread.
+static bool isPerThreadArrayReductionStorage(Value memref) {
+  DenseSet<Value> seen;
+  SmallVector<Value> worklist{memref};
+  while (!worklist.empty()) {
+    Value current = worklist.pop_back_val();
+    if (!current || !seen.insert(current).second)
+      continue;
+    Operation *def = current.getDefiningOp();
+    if (!def)
+      continue;
+    if (isa<memref::AllocaOp>(def))
+      return true;
+    if (isa<acc::UnwrapPrivateOp, acc::GPUSharedMemoryOp>(def))
+      return false;
+
+    if (Value address = getReadAddress(def)) {
+      Value stored = getUniqueStoredPointerValue(address, def);
+      if (!stored)
+        return false;
+      worklist.push_back(stored);
+      continue;
+    }
+
+    if (mlir::acc::GPUParallelDimsAttr parDims =
+            mlir::acc::getParDimsAttr(def)) {
+      return llvm::any_of(parDims.getArray(),
+                          [](auto dim) { return dim.isThreadX(); });
+    }
+
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
+  }
+  return false;
+}
+
 std::optional<int64_t>
 ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
                                               MemRefType baseTy) {
@@ -3342,9 +3492,8 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
   // accumulator copy. Classify the operand from its underlying allocation
-  // rather than its shape or par_dims: only memref.alloca is thread-private.
-  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
+  // rather than its shape or the accumulate's par_dims.
+  bool isPerThreadPrivate = isPerThreadArrayReductionStorage(memref);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 357f1e14d7a49..1573e0cb76f2a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,35 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// A reduction private that exceeds the per-thread stack budget uses indexed
+// backing storage shared by thread_x lanes. It must not be reduced again.
+// CHECK-LABEL: func.func @array_reduction_large_indexed_private
+// CHECK: acc.unwrap_private
+// CHECK: memref.view
+// CHECK: memref.subview
+// CHECK-NOT: gpu.all_reduce
+// CHECK-NOT: acc.reduction_accumulate_array
+func.func @array_reduction_large_indexed_private() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>]
+      : () -> !acc.private_type<memref<8192xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg0 = %private) : (!acc.private_type<memref<8192xi32>>) {
+    %c8192 = arith.constant 8192 : index
+    %local = acc.private_local %arg0
+        {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+        : (!acc.private_type<memref<8192xi32>>) -> memref<8192xi32>
+    %bounds = acc.bounds extent(%c8192 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<8192xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
 // CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
 // CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]

>From 46f3341a39dfe96de4e1865dca58ffd94086f890 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Tue, 21 Jul 2026 06:12:10 -0700
Subject: [PATCH 10/10] [mlir][OpenACC] Harden reduction storage provenance

Use explicit storage-preserving interfaces and reject ambiguous aliases so descriptor-backed reductions are classified conservatively.
---
 .../Dialect/OpenACC/OpenACCOpsInterfaces.td   |  15 +++
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 115 +++++++++---------
 .../acc-cg-to-gpu-reduction-array.mlir        |  52 ++++++++
 3 files changed, 126 insertions(+), 56 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index eddd21a7e917b..33bd833ef9737 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -45,6 +45,21 @@ def PartialEntityAccessOpInterface : OpInterface<"PartialEntityAccessOpInterface
   ];
 }
 
+def UnderlyingStorageOpInterface : OpInterface<"UnderlyingStorageOpInterface"> {
+  let cppNamespace = "::mlir::acc";
+
+  let description = [{
+    An interface for operations that derive a pointer-like result from one
+    underlying storage value. Other operands may describe metadata such as
+    bounds, offsets, or strides and are not storage sources.
+  }];
+
+  let methods = [
+    InterfaceMethod<"Get the underlying storage value", "::mlir::Value",
+      "getStorageSource", (ins)>,
+  ];
+}
+
 def AddressOfGlobalOpInterface : OpInterface<"AddressOfGlobalOpInterface"> {
   let cppNamespace = "::mlir::acc";
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 61ba40247d8b0..845b913e81e90 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -306,53 +306,29 @@ static bool sameEffectiveValue(Value x, int64_t y) {
   return *conX == y;
 }
 
-/// Continues tracking a memref through view-like and partial-access ops.
+/// Returns the operand that preserves the storage represented by \p op.
+static Value getStorageSource(Operation *op) {
+  if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op))
+    return viewLike.getViewSource();
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+      castOp && castOp->getNumOperands() == 1 && castOp->getNumResults() == 1)
+    return castOp->getOperand(0);
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(op))
+    return partialAccess.getBaseEntity();
+  if (acc::UnderlyingStorageOpInterface storage =
+          dyn_cast<acc::UnderlyingStorageOpInterface>(op))
+    return storage.getStorageSource();
+  return {};
+}
+
+/// Continues tracking a storage value through preserving operations.
 static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
                                   SmallVectorImpl<Value> &passThroughResults) {
-  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
-    if (viewLikeOp.getViewSource() == trackedOperand) {
-      passThroughResults.push_back(viewLikeOp.getViewDest());
-      return true;
-    }
+  if (getStorageSource(userOp) != trackedOperand)
     return false;
-  }
-
-  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
-    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
-        castOp->getOperand(0) == trackedOperand &&
-        isa<MemRefType>(castOp->getOperand(0).getType()) &&
-        isa<MemRefType>(castOp->getResult(0).getType())) {
-      passThroughResults.push_back(castOp->getResult(0));
-      return true;
-    }
-    return false;
-  }
-
-  // Partial-entity accesses (e.g. array element or field access) forward the
-  // base entity through to their results, so treat them as pass-through when
-  // the base entity is the value being tracked.
-  if (acc::PartialEntityAccessOpInterface partialAccess =
-          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
-    if (partialAccess.getBaseEntity() == trackedOperand) {
-      passThroughResults.append(userOp->result_begin(), userOp->result_end());
-      return true;
-    }
-    return false;
-  }
-
-  bool hasMemoryEffects = false;
-  if (auto effects = dyn_cast<MemoryEffectOpInterface>(userOp)) {
-    SmallVector<MemoryEffects::EffectInstance> instances;
-    effects.getEffects(instances);
-    hasMemoryEffects = !instances.empty();
-  }
-  if (userOp->getNumResults() == 1 &&
-      llvm::is_contained(userOp->getOperands(), trackedOperand) &&
-      !hasMemoryEffects) {
-    passThroughResults.push_back(userOp->getResult(0));
-    return true;
-  }
-  return false;
+  passThroughResults.append(userOp->result_begin(), userOp->result_end());
+  return true;
 }
 
 template <typename Effect>
@@ -385,8 +361,39 @@ static Value getReadAddress(Operation *op) {
   return address;
 }
 
+static bool hasAllocaRoot(Value value) {
+  DenseSet<Value> seen;
+  Value current = value;
+  while (current && seen.insert(current).second) {
+    Operation *def = current.getDefiningOp();
+    if (!def)
+      return false;
+    if (isa<memref::AllocaOp>(def))
+      return true;
+    current = getStorageSource(def);
+  }
+  return false;
+}
+
 /// Returns the unique pointer-like value stored to \p address before \p before.
 static Value getUniqueStoredPointerValue(Value address, Operation *before) {
+  Operation *scope = before->getParentOfType<acc::ComputeRegionOp>();
+  DominanceInfo dominance(scope ? scope : before->getParentOp());
+  auto safelyPrecedes = [&](Operation *store) {
+    if (dominance.dominates(store, before))
+      return true;
+    auto predicate = store->getParentOfType<acc::PredicateRegionOp>();
+    auto privateLocal = address.getDefiningOp<acc::PrivateLocalOp>();
+    GPUParallelDimsAttr parDims;
+    if (privateLocal)
+      parDims = mlir::acc::getParDimsAttr(privateLocal);
+    bool isThreadPrivate =
+        parDims && llvm::any_of(parDims.getArray(),
+                                [](auto dim) { return dim.isAnyThread(); });
+    return predicate && store->getParentOp() == predicate.getOperation() &&
+           dominance.dominates(predicate, before) &&
+           (isThreadPrivate || hasAllocaRoot(address));
+  };
   Operation *storedBy = nullptr;
   Value stored;
   SmallVector<Value> worklist{address};
@@ -397,8 +404,7 @@ static Value getUniqueStoredPointerValue(Value address, Operation *before) {
       continue;
     for (Operation *user : alias.getUsers()) {
       if (hasEffectOnValue<MemoryEffects::Write>(user, alias)) {
-        if (storedBy || user->getBlock() != before->getBlock() ||
-            !user->isBeforeInBlock(before))
+        if (storedBy || !safelyPrecedes(user))
           return {};
         Value candidate;
         for (Value operand : user->getOperands()) {
@@ -417,6 +423,8 @@ static Value getUniqueStoredPointerValue(Value address, Operation *before) {
       SmallVector<Value> through;
       if (getPassThroughResults(user, alias, through))
         worklist.append(through.begin(), through.end());
+      else if (!hasEffectOnValue<MemoryEffects::Read>(user, alias))
+        return {};
     }
   }
   return stored;
@@ -857,8 +865,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(through.begin(), through.end());
       else if (getMemoryPassThroughResults(user, cur, through))
         worklist.append(through.begin(), through.end());
-      else if (isa<ViewLikeOpInterface>(user))
-        worklist.append(user->result_begin(), user->result_end());
     }
   }
   return nullptr;
@@ -908,14 +914,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
 /// True when \p memref resolves to storage owned by the current thread.
 static bool isPerThreadArrayReductionStorage(Value memref) {
   DenseSet<Value> seen;
-  SmallVector<Value> worklist{memref};
-  while (!worklist.empty()) {
-    Value current = worklist.pop_back_val();
-    if (!current || !seen.insert(current).second)
-      continue;
+  Value current = memref;
+  while (current && seen.insert(current).second) {
     Operation *def = current.getDefiningOp();
     if (!def)
-      continue;
+      return false;
     if (isa<memref::AllocaOp>(def))
       return true;
     if (isa<acc::UnwrapPrivateOp, acc::GPUSharedMemoryOp>(def))
@@ -925,7 +928,7 @@ static bool isPerThreadArrayReductionStorage(Value memref) {
       Value stored = getUniqueStoredPointerValue(address, def);
       if (!stored)
         return false;
-      worklist.push_back(stored);
+      current = stored;
       continue;
     }
 
@@ -935,7 +938,7 @@ static bool isPerThreadArrayReductionStorage(Value memref) {
                           [](auto dim) { return dim.isThreadX(); });
     }
 
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    current = getStorageSource(def);
   }
   return false;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 1573e0cb76f2a..fd3bb1e5240f0 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,58 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// Dynamic allocation metadata must not be mistaken for storage provenance.
+// CHECK-LABEL: func.func @array_reduction_heap_alloca_extent
+// CHECK: memref.alloca() : memref<index>
+// CHECK: %[[EXTENT:.*]] = memref.load
+// CHECK: memref.alloc(%[[EXTENT]]) : memref<?xi32>
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @array_reduction_heap_alloca_extent(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
+    %slot = memref.alloca() : memref<index>
+    memref.store %extent, %slot[] : memref<index>
+    %loaded_extent = memref.load %slot[] : memref<index>
+    %heap = memref.alloc(%loaded_extent) : memref<?xi32>
+    %bounds = acc.bounds extent(%loaded_extent : index)
+    acc.reduction_accumulate_array %heap bounds(%bounds) <add>
+        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    memref.dealloc %heap : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Non-storage operands of pure operations must not affect classification.
+// CHECK-LABEL: func.func @array_reduction_shared_select
+// CHECK: arith.select
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @array_reduction_shared_select(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
+    %true = arith.constant true
+    %condition_slot = memref.alloca() : memref<i1>
+    memref.store %true, %condition_slot[] : memref<i1>
+    %condition = memref.load %condition_slot[] : memref<i1>
+    %heap = memref.alloc(%extent) : memref<?xi32>
+    %selected = arith.select %condition, %heap, %heap : memref<?xi32>
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %selected bounds(%bounds) <add>
+        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    memref.dealloc %heap : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // A reduction private that exceeds the per-thread stack budget uses indexed
 // backing storage shared by thread_x lanes. It must not be reduced again.
 // CHECK-LABEL: func.func @array_reduction_large_indexed_private


From llvmlistbot at llvm.org  Tue Jul 21 06:13:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 06:13:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][math] Fold FPowIOp with
 square-and-multiply to match powi expansion (PR #210982)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210982@github.com>
Message-ID: <6a5f707b.1728eaf3.2a2f6.22ef@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-math

Author: Alexey Bataev (alexey-bataev)

<details>
<summary>Changes</summary>

Fold math.fpowi using the base's own floating-point semantics via iterative
square-and-multiply, matching the multiply sequence ExpandPowI builds in
SelectionDAGBuilder. The prior pow/powf-based fold computed in float/double
and rounded differently from the runtime expansion, disagreeing with x**n by ~1 ULP.


---
Full diff: https://github.com/llvm/llvm-project/pull/210982.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Math/IR/MathOps.cpp (+18-12) 
- (modified) mlir/test/Dialect/Math/canonicalize.mlir (+16-4) 


``````````diff
diff --git a/mlir/lib/Dialect/Math/IR/MathOps.cpp b/mlir/lib/Dialect/Math/IR/MathOps.cpp
index b900cb1911759..2fd081282c2aa 100644
--- a/mlir/lib/Dialect/Math/IR/MathOps.cpp
+++ b/mlir/lib/Dialect/Math/IR/MathOps.cpp
@@ -831,23 +831,29 @@ OpFoldResult math::FPowIOp::fold(FoldAdaptor adaptor) {
   return constFoldBinaryOpConditional<FloatAttr, IntegerAttr>(
       adaptor.getOperands(),
       [](const APFloat &base, const APInt &exp) -> std::optional<APFloat> {
-        const llvm::fltSemantics &sem = base.getSemantics();
-        // Fold when the exponent is exactly representable in the
-        // floating-point type of the base.
-        APFloat fExp(sem);
-        if (fExp.convertFromAPInt(exp, /*isSigned=*/true,
-                                  APFloat::rmNearestTiesToEven) !=
-            APFloat::opOK)
-          return {};
-
-        switch (APFloat::SemanticsToEnum(sem)) {
+        switch (APFloat::SemanticsToEnum(base.getSemantics())) {
         case APFloat::Semantics::S_IEEEdouble:
-          return APFloat(pow(base.convertToDouble(), fExp.convertToDouble()));
         case APFloat::Semantics::S_IEEEsingle:
-          return APFloat(powf(base.convertToFloat(), fExp.convertToFloat()));
+          break;
         default:
           return {};
         }
+
+        // Square-and-multiply using the base's own semantics, matching the
+        // multiply sequence ExpandPowI builds in SelectionDAGBuilder.cpp.
+        const llvm::fltSemantics &sem = base.getSemantics();
+        APInt magnitude = exp.abs();
+        APFloat res = APFloat::getOne(sem);
+        APFloat curSquare = base;
+        while (!magnitude.isZero()) {
+          if (magnitude[0])
+            res = res * curSquare;
+          curSquare = curSquare * curSquare;
+          magnitude.lshrInPlace(1);
+        }
+        if (exp.isNegative())
+          res = APFloat::getOne(sem) / res;
+        return res;
       });
 }
 
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..a73a15f34e577 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -638,10 +638,22 @@ func.func @fpowi_fold_vec() -> vector<4xf32> {
   return %0 : vector<4xf32>
 }
 
-// 16777217 is not exactly representable in f32.
-// CHECK-LABEL: @fpowi_fold_failed
-// CHECK:       math.fpowi
-func.func @fpowi_fold_failed() -> f32 {
+// CHECK-LABEL: @fpowi_fold_const
+// CHECK: %[[cst:.+]] = arith.constant 29.1229134 : f32
+// CHECK: return %[[cst]]
+func.func @fpowi_fold_const() -> f32 {
+  %cst = arith.constant 1.234567 : f32
+  %c16_i32 = arith.constant 16 : i32
+  %0 = math.fpowi %cst, %c16_i32 : f32, i32
+  return %0 : f32
+}
+
+// The fold no longer requires the exponent to be exactly representable as
+// f32: it is used as an integer, so this folds and overflows to +inf.
+// CHECK-LABEL: @fpowi_fold_overflow
+// CHECK: %[[cst:.+]] = arith.constant 0x7F800000 : f32
+// CHECK: return %[[cst]]
+func.func @fpowi_fold_overflow() -> f32 {
   %cst = arith.constant 2.000000e+00 : f32
   %c16777217_i32 = arith.constant 16777217 : i32
   %0 = math.fpowi %cst, %c16777217_i32 : f32, i32

``````````

</details>


https://github.com/llvm/llvm-project/pull/210982

From llvmlistbot at llvm.org  Tue Jul 21 06:21:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 06:21:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support multi-rank array
	reductions (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a5f7249.e5907290.281687.e825@mx.google.com>


https://github.com/khaki3 edited https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Tue Jul 21 06:21:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 06:21:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static and dynamic
 multi-rank OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a5f7251.0c05e68e.3c2f5d.f500@mx.google.com>


https://github.com/khaki3 edited https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Tue Jul 21 06:28:30 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 21 Jul 2026 06:28:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f73fe.5a0412d2.3daf32.8eee@mx.google.com>


https://github.com/nujaa edited https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 06:30:34 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 21 Jul 2026 06:30:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f747a.b4a331f0.d9742.c9f0@mx.google.com>


================

----------------
nujaa wrote:

Description updated.

https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 06:39:50 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 21 Jul 2026 06:39:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Introduce support for early exits (PR
	#166688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/166688@github.com>
Message-ID: <6a5f76a6.510f576d.2f694c.137a@mx.google.com>


https://github.com/joker-eph updated https://github.com/llvm/llvm-project/pull/166688

>From 9cf4b25e3ae766c25f85455bef30f894de9bea54 Mon Sep 17 00:00:00 2001
From: Mehdi Amini <joker.eph at gmail.com>
Date: Sat, 26 Apr 2025 04:51:22 -0700
Subject: [PATCH 1/2] [mlir][scf] Add token-targeted early exits for scf.loop

Summary:
- Introduce a structured region-exit control-flow model for MLIR regions.
- Model region exits with RegionExitTerminatorOpInterface. Normal exits may
  target the immediate parent operation; breaking-control-flow exits request
  propagation to a specific ancestor receiver. Every intermediate parent must
  opt in with PropagateControlFlowBreak until the addressed receiver handles
  the request. In the SCF dialect, scf.loop requests are addressed with the
  control token defined by the loop body.

Core IR:
- Add RegionExitTerminatorOpInterface for terminators that identify possible
  receiver operations for a region exit.
- Add PropagateControlFlowBreak for intermediate operations that forward
  breaking-control-flow requests without consuming them.
- Add HasBreakingControlFlowOpInterface for operations that receive addressed
  breaking-control-flow requests, plus helpers to find the receiver, collect
  nested predecessors, and detect escaping breaking control flow.
- Represent propagated RegionBranchOpInterface successors with explicit target
  operations instead of a parent or propagation sentinel. Add a helper to
  determine which operation owns a successor input list.

SCF dialect:
- Add scf.loop, which defines a control token, accepts optional iter_args and
  results, and implements the breaking-control-flow receiver interface.
- Add scf.break and scf.continue as token-consuming region-exit terminators.
  scf.break requests exit from the target loop and supplies loop results;
  scf.continue requests re-entry into the target loop with next iteration
  values.
- Let scf.yield and scf.reduce implement RegionExitTerminatorOpInterface by
  targeting their immediate parent operation.
- Verify token receivers, propagation traits on forwarding operations, and
  region-branch operand/result type consistency.
- Let scf.if propagate breaks and continues through its regions. Its parser,
  printer, result inference, implicit terminator handling, and
  canonicalizations now account for scf.yield, scf.break, and scf.continue
  terminators.
- Guard yield-specific scf.if rewrites and add a canonicalization that removes
  dead code after an if whose branches both transfer breaking control flow.

Analyses and utilities:
- Teach dead code analysis, dense and sparse backward dataflow, region-branch
  graph traversal, post-dominance, and region inlining to follow concrete exit
  targets instead of treating the immediate parent as the final successor.

Lowering:
- Lower scf.loop in the SCF-to-CF conversion by legalizing nested regions,
  collecting direct break and continue predecessors, inlining the body, turning
  breaks into branches to the continuation block, turning continues into
  backedges, removing the loop token argument, and wiring results through
  continuation block arguments.
- Adjust scf.if lowering to branch only from scf.yield terminators, leaving
  early-exit terminators for the target loop lowering.

Tests and docs:
- Document the cooperative region-exit propagation model in the LangRef.
- Add tests for valid and invalid early-exit IR, region-branch interface
  printing, dominance, dead code analysis, SCCP, SCF-to-CF lowering,
  SCF canonicalization, and an end-to-end early-exit integration case.

Assisted-by: Claude Code
Assisted-by: Codex
---
 mlir/docs/LangRef.md                          | 122 +++-
 mlir/include/mlir/Dialect/SCF/IR/SCF.h        |  70 +++
 mlir/include/mlir/Dialect/SCF/IR/SCFOps.td    | 179 +++++-
 mlir/include/mlir/IR/Dominance.h              |   9 +
 mlir/include/mlir/IR/OpBase.td                |   7 +
 mlir/include/mlir/IR/OpDefinition.h           |  13 +
 mlir/include/mlir/IR/RegionKindInterface.h    | 121 ++++
 mlir/include/mlir/IR/RegionKindInterface.td   |  83 +++
 .../mlir/Interfaces/ControlFlowInterfaces.h   |   6 +
 .../Analysis/DataFlow/DeadCodeAnalysis.cpp    |  12 +-
 mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp  |   9 +-
 mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp |  23 +-
 .../SCFToControlFlow/SCFToControlFlow.cpp     | 237 +++++--
 mlir/lib/Dialect/SCF/IR/SCF.cpp               | 576 +++++++++++++++---
 .../SCF/IR/ValueBoundsOpInterfaceImpl.cpp     |   4 +-
 .../BufferizableOpInterfaceImpl.cpp           |   4 +-
 mlir/lib/IR/Dominance.cpp                     |  94 ++-
 mlir/lib/IR/RegionKindInterface.cpp           | 168 +++++
 mlir/lib/IR/Verifier.cpp                      |   1 +
 mlir/lib/Interfaces/ControlFlowInterfaces.cpp | 141 ++++-
 mlir/lib/Transforms/Utils/CMakeLists.txt      |   3 +
 mlir/lib/Transforms/Utils/InliningUtils.cpp   |   6 +
 .../test-dead-code-analysis-early-exit.mlir   | 100 +++
 mlir/test/Analysis/test-dominance.mlir        |  60 ++
 .../convert-early-exit-to-cfg.mlir            | 135 ++++
 mlir/test/Dialect/SCF/loop_canonicalize.mlir  | 300 +++++++++
 mlir/test/IR/early-exit-invalid.mlir          | 142 +++++
 mlir/test/IR/early-exit.mlir                  |  82 +++
 .../Integration/Dialect/SCF/early_exit.mlir   |  82 +++
 mlir/test/Transforms/sccp-early-exit.mlir     | 117 ++++
 mlir/test/lib/Dialect/Test/TestOps.td         |   6 +
 mlir/test/lib/IR/TestDominance.cpp            |  33 +
 mlir/test/lib/Interfaces/CMakeLists.txt       |   1 +
 .../RegionBranchOpInterface/CMakeLists.txt    |   9 +
 .../TestRegionBranchOpInterface.cpp           |  77 +++
 mlir/tools/mlir-opt/CMakeLists.txt            |   1 +
 mlir/tools/mlir-opt/mlir-opt.cpp              |   2 +
 37 files changed, 2839 insertions(+), 196 deletions(-)
 create mode 100644 mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
 create mode 100644 mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
 create mode 100644 mlir/test/Dialect/SCF/loop_canonicalize.mlir
 create mode 100644 mlir/test/IR/early-exit-invalid.mlir
 create mode 100644 mlir/test/IR/early-exit.mlir
 create mode 100644 mlir/test/Integration/Dialect/SCF/early_exit.mlir
 create mode 100644 mlir/test/Transforms/sccp-early-exit.mlir
 create mode 100644 mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
 create mode 100644 mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp

diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..6ba4277ffac9f 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -509,7 +509,18 @@ operation produces results and the operation returns, those results also have
 well-defined values. Execution then proceeds to the next operation in the block
 until the terminator operation at the end of the block is reached; the
 terminator determines the next continuation, if any. The determination of the
-next instruction to execute is the 'passing of control flow'.
+next instruction to execute is the 'passing of control flow'. A nested
+[region-exit terminator](#region-exit-terminators) can also terminate the
+current region with a request for a breaking-control-flow event addressed to a
+specific ancestor operation that implements
+`HasBreakingControlFlowOpInterface`. The terminator itself does not jump
+directly to that ancestor; it transfers control back to the containing
+operation with that request. If the containing operation is not the addressed
+receiver, it must define `PropagateControlFlowBreak` and propagate the same
+request to its own parent. This repeats until the addressed receiver handles
+the request, so every intermediate operation on the path from the terminator to
+the receiving `HasBreakingControlFlowOpInterface` operation must define
+`PropagateControlFlowBreak`.
 
 In general, when control flow is passed to an operation, MLIR does not restrict
 when control flow enters or exits the regions contained in that operation.
@@ -519,25 +530,26 @@ represent possible continuations. Successors explicitly specify destination
 blocks, so control flow within the region can only pass to one of the specified
 successor blocks, as in a `branch` operation. When a terminator has no
 successors, it may pass 1) control back to the containing operation, as in a
-`return` operation, or 2) define that control flow does not continue, as in
-`ub.unreachable`. Terminators without successors therefore do not necessarily
-imply a return to the containing operation, the specific dialect operation
-determines the terminator's semantics. Blocks (other than the entry block) that
-are not listed as a successor of any terminator operation are defined to be
-unreachable and can be removed without affecting the semantics of the
-containing operation.
+`return` operation, 2) define that control flow does not continue, as in
+`ub.unreachable`, or 3) request that the containing operation propagate breaking
+control flow outward toward a specific ancestor. This target designator is
+dialect-defined. Terminators without successors therefore do not
+necessarily imply a return to the containing operation. Blocks (other than the
+entry block) that are not listed as a successor of any terminator operation are
+defined to be unreachable and can be removed without affecting the semantics of
+the containing operation.
 
 Although control flow always enters a region through the entry block, control
 flow may exit a region through any block with an appropriate terminator. The
-standard dialect leverages this capability to define operations with
+SCF dialect for example leverages this capability to define operations with
 Single-Entry-Multiple-Exit (SEME) regions, possibly flowing through different
 blocks in the region and exiting through any block with a `return` operation.
-This behavior is similar to that of a function body in most programming
-languages. In addition, control flow may also not reach the end of a block or
-region, for example if a function call does not return. Such an operation
-prevents control flow from reaching later operations, but does not remove the
-structural requirement that the block must end with a terminator unless the
-enclosing operation opts out with `NoTerminator`.
+This behavior can model that of a function body in most programming languages.
+In addition, control flow may also not reach the end of a block or region, for
+example if a function call does not return. Such an operation prevents control
+flow from reaching later operations, but does not remove the structural
+requirement that the block must end with a terminator unless the enclosing
+operation opts out with `NoTerminator`.
 
 Example:
 
@@ -565,6 +577,86 @@ func.func @accelerator_compute(i64, i1) -> i64 { // An SSACFG region
 }
 ```
 
+#### Region-Exit Terminators
+
+A terminator that implements `RegionExitTerminatorOpInterface` terminates the
+current region and identifies the operation that may receive the region exit. A
+normal region exit can target the immediately containing operation. For
+example, `scf.yield` exits only its own immediately enclosing region and
+returns control to the parent operation.
+
+Some terminators instead request a breaking-control-flow event addressed to a
+specific ancestor. The terminator transfers control back only to its immediate
+parent operation; reaching the addressed ancestor requires each intermediate
+parent operation to collaborate by propagating the request outward. The way a
+terminator designates the ancestor is dialect-defined: it may be a builtin
+`token` operand, an integer count of region levels to exit, a named symbolic
+relationship to a parent operation, or another dialect-specific mechanism.
+
+Every intermediate operation between a breaking terminator and the addressed
+receiver must define `PropagateControlFlowBreak`. The receiver must implement
+`HasBreakingControlFlowOpInterface`. For example, a loop operation nested
+inside another loop operation body carries both traits simultaneously: it
+handles breaks whose token identifies itself, and propagates breaks whose token
+identifies an outer loop.
+
+Region-exit terminators may carry values, which are propagated to the
+target operation. For example, when breaking out of a loop that produces
+results, the terminator supplies those result values. The exact mapping between
+terminator operands and the receiving op's results is dialect-defined (the
+receiving op may also ignore operands entirely).
+
+Examples:
+
+```mlir
+// scf.yield is the standard immediate region-exit terminator.
+// It exits only its own immediately enclosing region.
+scf.if %cond {
+  scf.yield  // returns control to the immediate parent of scf.if
+}
+```
+
+```mlir
+// Trait legend:
+//   [H]    = HasBreakingControlFlowOpInterface (receives/catches the break)
+//   [P]    = PropagateControlFlowBreak (passes the break request upward)
+//   [H][P] = both: handles breaks whose token identifies it, and propagates
+//            breaks whose token identifies an outer loop
+scf.loop token(%outer) {                    // [H][P]
+  scf.loop token(%inner) {                  // [H][P]
+    scf.if %cond1 {                         // [P]
+      // Requests a break of the inner loop.
+      scf.break [%inner]
+    }
+    scf.if %cond2 {                         // [P]
+      // Requests a break of the outer loop. The scf.if and inner scf.loop
+      // both propagate the request upward.
+      scf.break [%outer]
+    }
+    scf.if %cond3 {                         // [P]
+      // Requests re-entry into the inner loop.
+      scf.continue [%inner]
+    }
+  }
+}
+return
+```
+
+```mlir
+// A loop that yields a result value on early exit.
+// scf.break carries operands that become the loop's results. (scf.continue
+// would instead carry operands that become the next iteration's iter_args,
+// but this loop has none.)
+%result = scf.loop token(%loop) -> f32 {    // [H]
+  scf.if %found {                           // [P]
+    // %value becomes the loop result.
+    scf.break [%loop] %value : f32
+  }
+  // Re-enter the loop for the next iteration (no iter_args here).
+  scf.continue [%loop]
+}
+```
+
 #### Operations with Multiple Regions
 
 An operation containing multiple regions also completely determines the
diff --git a/mlir/include/mlir/Dialect/SCF/IR/SCF.h b/mlir/include/mlir/Dialect/SCF/IR/SCF.h
index 44cbb458d94fe..8918a1e89efdf 100644
--- a/mlir/include/mlir/Dialect/SCF/IR/SCF.h
+++ b/mlir/include/mlir/Dialect/SCF/IR/SCF.h
@@ -30,6 +30,11 @@
 namespace mlir {
 namespace scf {
 void buildTerminatedBody(OpBuilder &builder, Location loc);
+
+namespace op_impl {
+struct IfOpImplicitTerminatorType;
+struct LoopOpImplicitTerminatorType;
+} // namespace op_impl
 } // namespace scf
 } // namespace mlir
 
@@ -112,6 +117,71 @@ SmallVector<Value> replaceAndCastForOpIterArg(RewriterBase &rewriter,
                                               OpOperand &operand,
                                               Value replacement,
                                               const ValueTypeCastFnTy &castFn);
+namespace op_impl {
+
+//===----------------------------------------------------------------------===//
+// ControlFlowImplicitTerminatorOperation
+//===----------------------------------------------------------------------===//
+
+/// This class provides an interface compatible with
+/// SingleBlockImplicitTerminator, but allows multiple types of potential
+/// terminators aside from just one. If a terminator isn't present, this will
+/// generate a `ImplicitOpT` operation.
+template <typename ImplicitOpT, typename... OtherTerminatorOpTs>
+struct ControlFlowImplicitTerminatorOpType {
+  /// Implementation of `classof` that supports all of the potential terminator
+  /// operations.
+  static bool classof(Operation *op) {
+    return isa<ImplicitOpT, OtherTerminatorOpTs...>(op);
+  }
+
+  //===--------------------------------------------------------------------===//
+  // Implicit Terminator Methods
+
+  /// The following methods are all used when interacting with the "implicit"
+  /// terminator.
+
+  template <typename... Args>
+  static void build(Args &&...args) {
+    ImplicitOpT::build(std::forward<Args>(args)...);
+  }
+  static constexpr StringLiteral getOperationName() {
+    return ImplicitOpT::getOperationName();
+  }
+};
+/// An implicit terminator type for `if` operations. Empty regions still receive
+/// an `scf.yield`, but explicitly terminated regions may use any terminator
+/// implementing RegionExitTerminatorOpInterface so dialect-defined exits can
+/// propagate through `scf.if`.
+struct IfOpImplicitTerminatorType {
+  static bool classof(Operation *op) {
+    return isa<RegionExitTerminatorOpInterface>(op);
+  }
+
+  template <typename... Args>
+  static void build(Args &&...args) {
+    YieldOp::build(std::forward<Args>(args)...);
+  }
+  static constexpr StringLiteral getOperationName() {
+    return YieldOp::getOperationName();
+  }
+};
+struct LoopOpImplicitTerminatorType
+    : public ControlFlowImplicitTerminatorOpType<ContinueOp, BreakOp> {
+  /// Build the implicit `scf.continue` terminator. The control token consumed
+  /// by the terminator is the loop body's entry block argument #0; the
+  /// insertion block is guaranteed to be that body when the implicit
+  /// terminator is materialized. This keeps `scf.continue`'s public builders
+  /// free of any hidden block-argument assumption.
+  static void build(OpBuilder &builder, OperationState &state) {
+    Block *block = builder.getInsertionBlock();
+    assert(block && block->getNumArguments() > 0 &&
+           isa<TokenType>(block->getArgument(0).getType()) &&
+           "expected insertion block with a token-typed loop control argument");
+    state.addOperands(block->getArgument(0));
+  }
+};
+} // namespace op_impl
 
 /// Helper function to compute the difference between two values. This is used
 /// by the loop implementations to compute the trip count.
diff --git a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
index c0d1ac501cc77..1eeeb184e0bbb 100644
--- a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
+++ b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
@@ -15,6 +15,7 @@
 
 include "mlir/Interfaces/ControlFlowInterfaces.td"
 include "mlir/Interfaces/LoopLikeInterface.td"
+include "mlir/IR/OpAsmInterface.td"
 include "mlir/IR/RegionKindInterface.td"
 include "mlir/Dialect/SCF/IR/DeviceMappingInterface.td"
 include "mlir/Interfaces/DestinationStyleOpInterface.td"
@@ -146,6 +147,168 @@ def ExecuteRegionOp : SCF_Op<"execute_region", [
 
   let hasVerifier = 1;
 }
+//===----------------------------------------------------------------------===//
+// LoopOp
+//===----------------------------------------------------------------------===//
+
+def LoopOp : SCF_Op<"loop",[
+    AutomaticAllocationScope,
+    OpAsmOpInterface,
+    RecursiveMemoryEffects,
+    PropagateControlFlowBreak,
+    TokenProducerTrait,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface,
+        ["getEntrySuccessorOperands", "getSuccessorInputs"]>,
+    SingleBlockImplicitTerminator<"op_impl::LoopOpImplicitTerminatorType">,
+    HasBreakingControlFlowOpInterface,
+    HasNestedTerminator<["ContinueOp", "BreakOp"]>
+  ]> {
+  let summary = "Loop until a break operation";
+  let description = [{
+    The `loop` operation represents a loop that executes until a `break` is reached.
+
+    The loop consists of (1) a set of loop-carried values which are initialized by
+    `initValues` and updated by each iteration of the loop, and
+    (2) a region which represents the loop body.
+
+    The loop will execute the body of the loop until a `break` is executed.
+
+    Each control path of the loop must be terminated by:
+
+    - a `continue` that yields the next iteration's value for each loop carried variable.
+    - a `break` that terminates the loop and yields the final loop results.
+    - a nested terminator requesting to propagate the control-flow to a parent.
+  }];
+
+  let arguments = (ins Variadic<AnyType>:$initValues);
+  let results = (outs Variadic<AnyType>:$resultValues);
+  let regions = (region SizedRegion<1>:$region);
+
+  let extraClassDeclaration = [{
+    /// Return the iteration values of the loop region.
+    Block::BlockArgListType getRegionIterValues() {
+      return getRegion().getArguments().drop_front();
+    }
+
+    /// Return the `index`-th region iteration value.
+    BlockArgument getRegionIterValue(unsigned index) {
+      return getRegionIterValues()[index];
+    }
+
+    /// Return the loop control token.
+    BlockArgument getControlToken() {
+      return getRegion().getArgument(0);
+    }
+
+    /// Returns the number of region arguments for loop-carried values.
+    unsigned getNumRegionIterValues() {
+      return getRegion().getNumArguments() - 1;
+    }
+
+    /// Returns the loop block body
+    Block *getBody() { return &getRegion().front(); }
+  }];
+
+  let hasCustomAssemblyFormat = 1;
+  let hasRegionVerifier = 1;
+  let hasCanonicalizer = 1;
+}
+
+
+//===----------------------------------------------------------------------===//
+// BreakOp
+//===----------------------------------------------------------------------===//
+
+def BreakOp : SCF_Op<"break", [
+    Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>,
+    ParentOneOf<["IfOp", "LoopOp"]>
+  ]> {
+  let summary = "Break from loop";
+  let description = [{
+    The `break` operation is a terminator that exits one or more nested
+    regions and terminates the `scf.loop` that defined its control token.
+
+    The `break` may yield any number of operands; their types must match the
+    result types of the target `scf.loop`.
+
+    Example — break out of the immediately enclosing loop:
+    ```mlir
+    scf.loop token(%loop) -> i32 {
+      scf.break [%loop] %result : i32
+    }
+    ```
+
+    Example — break out of a loop through an enclosing `scf.if`:
+    ```mlir
+    scf.loop token(%loop) {
+      scf.if %cond {
+        scf.break [%loop]
+      }
+      scf.continue [%loop]
+    }
+    ```
+  }];
+
+
+  let arguments = (ins Token:$targetToken, Variadic<AnyType>:$args);
+  let assemblyFormat = [{
+    ` ` `[` $targetToken `]` ($args^ `:` type($args))? attr-dict
+  }];
+  let hasVerifier = 1;
+}
+
+
+//===----------------------------------------------------------------------===//
+// ContinueOp
+//===----------------------------------------------------------------------===//
+
+def ContinueOp : SCF_Op<"continue", [
+    Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>,
+    ParentOneOf<["IfOp", "LoopOp"]>
+  ]> {
+  let summary = "Continue to next loop iteration";
+  let description = [{
+    The `continue` operation is a terminator that re-enters a `scf.loop`
+    for its next iteration. The target loop is the one that defined the control
+    token operand.
+
+    The operands of `continue` become the loop-carried values (iter_args) for
+    the next iteration; their types must match the loop's iter_arg types.
+
+    Example — continue the immediately enclosing loop:
+    ```mlir
+    scf.loop token(%loop) iter_args(%i = %init) : i32 {
+      %next = arith.addi %i, %one : i32
+      scf.continue [%loop] %next : i32
+    }
+    ```
+
+    Example — continue an outer loop from inside a nested `scf.if`:
+    ```mlir
+    scf.loop token(%outer) iter_args(%counter = %init) : i64 {
+      scf.loop token(%inner) iter_args(%inner_arg = %counter) : i64 {
+        scf.if %restart_outer {
+          scf.continue [%outer] %inner_arg : i64
+        }
+        scf.continue [%inner] %inner_arg : i64
+      }
+      scf.continue [%outer] %counter : i64
+    }
+    ```
+  }];
+
+  let arguments = (ins Token:$targetToken, Variadic<AnyType>:$args);
+  let assemblyFormat = [{
+    ` ` `[` $targetToken `]` ($args^ `:` type($args))? attr-dict
+  }];
+  let hasVerifier = 1;
+}
 
 //===----------------------------------------------------------------------===//
 // ForOp
@@ -714,8 +877,8 @@ def IfOp : SCF_Op<"if", [DeclareOpInterfaceMethods<RegionBranchOpInterface, [
     "getNumRegionInvocations", "getRegionInvocationBounds",
     "getEntrySuccessorRegions", "getSuccessorInputs"]>,
     DeclareOpInterfaceMethods<PromotableRegionOpInterface>,
-    InferTypeOpAdaptor, SingleBlockImplicitTerminator<"scf::YieldOp">,
-    RecursiveMemoryEffects, RecursivelySpeculatable, NoRegionArguments]> {
+    InferTypeOpAdaptor, SingleBlockImplicitTerminator<"op_impl::IfOpImplicitTerminatorType">,
+    RecursiveMemoryEffects, RecursivelySpeculatable, NoRegionArguments, PropagateControlFlowBreak]> {
   let summary = "if-then-else operation";
   let description = [{
     The `scf.if` operation represents an if-then-else construct for
@@ -798,8 +961,18 @@ def IfOp : SCF_Op<"if", [DeclareOpInterfaceMethods<RegionBranchOpInterface, [
                                   : OpBuilder::atBlockEnd(body, listener);
     }
     Block* thenBlock();
+    /// Returns the terminator of the then block. May be scf.break,
+    /// scf.continue, or scf.yield.
+    Operation *thenTerminator() {
+      return thenBlock()->getTerminator();
+    }
     YieldOp thenYield();
     Block* elseBlock();
+    /// Returns the terminator of the else block. May be scf.break,
+    /// scf.continue, or scf.yield.
+    Operation *elseTerminator() {
+      return elseBlock()->getTerminator();
+    }
     YieldOp elseYield();
   }];
   let hasFolder = 1;
@@ -918,6 +1091,7 @@ def ParallelOp : SCF_Op<"parallel",
 
 def ReduceOp : SCF_Op<"reduce", [
     Terminator, HasParent<"ParallelOp">, RecursiveMemoryEffects,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
     DeclareOpInterfaceMethods<PromotableRegionOpInterface>,
     DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
                               ["getMutableSuccessorOperands"]>]> {
@@ -1219,6 +1393,7 @@ def IndexSwitchOp : SCF_Op<"index_switch", [RecursiveMemoryEffects,
 //===----------------------------------------------------------------------===//
 
 def YieldOp : SCF_Op<"yield", [Pure, ReturnLike, Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
     ParentOneOf<["ExecuteRegionOp", "ForOp", "IfOp", "IndexSwitchOp",
                  "WhileOp"]>]> {
   let summary = "loop yield and termination operation";
diff --git a/mlir/include/mlir/IR/Dominance.h b/mlir/include/mlir/IR/Dominance.h
index 70924a2e9ae59..1585f1881136b 100644
--- a/mlir/include/mlir/IR/Dominance.h
+++ b/mlir/include/mlir/IR/Dominance.h
@@ -123,6 +123,11 @@ class DominanceInfoBase {
                              Block::iterator bIt,
                              bool enclosingOk = true) const;
 
+  /// Return true if `op` contains a nested breaking terminator that escapes
+  /// through it to an ancestor. This cache is invalidated together with
+  /// dominance info.
+  bool hasBreakingControlFlowOpsCached(Operation *op) const;
+
   /// A mapping of regions to their base dominator tree and a cached
   /// "hasSSADominance" bit. This map does not contain dominator trees for
   /// single block CFG regions, but we do want to cache the "hasSSADominance"
@@ -131,6 +136,10 @@ class DominanceInfoBase {
   ///
   mutable DenseMap<Region *, llvm::PointerIntPair<DomTree *, 1, bool>>
       dominanceInfos;
+
+  /// Cached escaping breaking-terminator query results. This is stable for the
+  /// lifetime of the analyzed IR and cleared on dominance invalidation.
+  mutable DenseMap<Operation *, bool> breakingControlFlowOpsCache;
 };
 
 extern template class DominanceInfoBase</*IsPostDom=*/true>;
diff --git a/mlir/include/mlir/IR/OpBase.td b/mlir/include/mlir/IR/OpBase.td
index 0d0669e90c3f7..ce7c891076420 100644
--- a/mlir/include/mlir/IR/OpBase.td
+++ b/mlir/include/mlir/IR/OpBase.td
@@ -134,6 +134,13 @@ class SingleBlockImplicitTerminatorImpl<string op>
 class SingleBlockImplicitTerminator<string op>
     : TraitList<[SingleBlock, SingleBlockImplicitTerminatorImpl<op>]>;
 
+// This operation has nested regions with the supplied list of terminator
+// operations.
+class HasNestedTerminator<list<string> ops>
+    : ParamNativeOpTrait<"HasNestedTerminators", !interleave(ops, ", ")>,
+      StructuralOpTrait;
+
+
 // Op's regions don't have terminator.
 def NoTerminator : NativeOpTrait<"NoTerminator">, StructuralOpTrait;
 
diff --git a/mlir/include/mlir/IR/OpDefinition.h b/mlir/include/mlir/IR/OpDefinition.h
index bd7fa1ffd4428..a146d17433ce3 100644
--- a/mlir/include/mlir/IR/OpDefinition.h
+++ b/mlir/include/mlir/IR/OpDefinition.h
@@ -1362,6 +1362,19 @@ struct HasAncestor {
   };
 };
 
+/// This class provides a verifier for ops that are expecting to have nested
+/// predecessors.
+template <typename... NestedPredecessorOpTypes>
+struct HasNestedTerminators {
+  template <typename ConcreteType>
+  class Impl : public TraitBase<ConcreteType, Impl> {
+  public:
+    static bool acceptsTerminator(Operation *predecessor) {
+      return llvm::isa_and_nonnull<NestedPredecessorOpTypes...>(predecessor);
+    }
+  };
+};
+
 /// A trait for operations that have an attribute specifying operand segments.
 ///
 /// Certain operations can have multiple variadic operands and their size
diff --git a/mlir/include/mlir/IR/RegionKindInterface.h b/mlir/include/mlir/IR/RegionKindInterface.h
index d6d3aeeb9bd05..2ada56ba6c193 100644
--- a/mlir/include/mlir/IR/RegionKindInterface.h
+++ b/mlir/include/mlir/IR/RegionKindInterface.h
@@ -36,6 +36,26 @@ class HasOnlyGraphRegion : public TraitBase<ConcreteType, HasOnlyGraphRegion> {
   static RegionKind getRegionKind(unsigned index) { return RegionKind::Graph; }
   static bool hasSSADominance(unsigned index) { return false; }
 };
+
+/// Indicates that this operation is transparent to breaking control flow:
+/// a terminator (e.g. scf.break / scf.continue) can propagate through
+/// this op on its way to the addressed HasBreakingControlFlowOpInterface
+/// ancestor. The op does NOT consume the break; it simply passes it upward.
+/// All intermediate ops that receive and forward the break request must carry
+/// this trait.
+template <typename ConcreteType>
+class PropagateControlFlowBreak
+    : public TraitBase<ConcreteType, PropagateControlFlowBreak> {
+public:
+  static LogicalResult verifyTrait(Operation *op) {
+    // Verify the operation has regions and can handle breaking control flow
+    if (op->getNumRegions() == 0)
+      return op->emitOpError(
+          "operation with PropagateControlFlowBreak trait must have regions");
+    return success();
+  }
+};
+
 } // namespace OpTrait
 
 /// Return "true" if the given region may have SSA dominance. This function also
@@ -49,8 +69,109 @@ bool mayHaveSSADominance(Region &region);
 /// implement the RegionKindInterface.
 bool mayBeGraphRegion(Region &region);
 
+/// Summary of breaking terminator operations nested under an op.
+struct NestedBreakingControlFlowInfo {
+  /// Breaking terminators that may target the queried op directly. These
+  /// transfer control to the queried op rather than only to an intermediate
+  /// PropagateControlFlowBreak op.
+  SmallVector<Operation *> predecessors;
+
+  /// True if any direct predecessor targets the queried op from below the
+  /// queried op's immediate region.
+  bool hasNestedPredecessors = false;
+
+  /// True if any breaking terminator under the queried op targets one of the
+  /// queried op's ancestors, escaping through the queried op.
+  bool hasBreakingControlFlowOps = false;
+};
+
+/// Collect nested breaking-control-flow information for `op` with a single walk
+/// of its nested region tree.
+NestedBreakingControlFlowInfo getNestedBreakingControlFlowInfo(Operation *op);
+
+/// Return true if `op` (which implements HasBreakingControlFlowOpInterface)
+/// contains at least one breaking terminator that directly targets it from a
+/// nested region. Such a terminator is a "nested predecessor" of `op` because
+/// control flow may re-enter or exit `op` through a request propagated from a
+/// deeply nested site rather than only through an immediately enclosing
+/// terminator.
+bool hasNestedPredecessors(Operation *op);
+
+/// Return true if `op` contains any breaking terminator that would "break
+/// through" `op` towards an outer HasBreakingControlFlowOpInterface ancestor.
+/// This is used to detect whether an op's post-dominance is disrupted by an
+/// early-exit request that propagates through it.
+bool hasBreakingControlFlowOps(Operation *op);
+
+/// Collect all breaking terminators nested inside `op` that potentially
+/// directly target `op`. These are the ops that may transfer control flow to
+/// `op` on an early exit.
+void collectAllNestedPredecessors(Operation *op,
+                                  SmallVector<Operation *> &predecessors);
 } // namespace mlir
 
 #include "mlir/IR/RegionKindInterface.h.inc"
 
+namespace mlir {
+namespace detail {
+/// Implementation helper for visitNestedBreakingControlFlowOps. Walks the
+/// regions of `op` and invokes `callback` for every breaking terminator that
+/// either targets `op` or propagates further upward through `op`.
+/// The `nestedLevel` argument passed to the callback is the 1-based depth of
+/// the terminator relative to `op`'s outermost region.
+void visitNestedBreakingControlFlowOpsImpl(
+    Operation *op,
+    function_ref<WalkResult(RegionExitTerminatorOpInterface, int nestedLevel)>
+        callback);
+} // namespace detail
+
+/// Walk all breaking terminators that are relevant to breaking control
+/// flow inside `op` (see visitNestedBreakingControlFlowOpsImpl). The callback
+/// receives the terminator op and its 1-based nesting level. Callbacks
+/// returning WalkResult support early termination via WalkResult::interrupt();
+/// void-returning callbacks always continue.
+template <typename CallbackT>
+void visitNestedBreakingControlFlowOps(Operation *op, CallbackT &&callback) {
+  using RetT =
+      decltype(callback(std::declval<Operation *>(), std::declval<int>()));
+  if constexpr (std::is_same_v<RetT, WalkResult>) {
+    detail::visitNestedBreakingControlFlowOpsImpl(op, callback);
+  } else {
+    detail::visitNestedBreakingControlFlowOpsImpl(
+        op, [&](RegionExitTerminatorOpInterface visitedOp, int nestedLevel) {
+          callback(visitedOp, nestedLevel);
+          return WalkResult::advance();
+        });
+  }
+}
+
+/// Walk all breaking terminators relevant to breaking control flow
+/// across all top-level ops in `region`.
+template <typename CallbackT>
+void visitNestedBreakingControlFlowOps(Region &region, CallbackT &&callback) {
+  // Pass `callback` as an lvalue: it is reused across iterations, so it must
+  // not be forwarded (which could move from it on the first iteration).
+  for (Operation &op : region.getOps())
+    visitNestedBreakingControlFlowOps(&op, callback);
+}
+
+/// Return true if the given region may contain breaking control flow — either
+/// because its parent op propagates breaks (PropagateControlFlowBreak) or
+/// because it is the body of a HasBreakingControlFlowOpInterface op. Used to
+/// decide whether post-dominance analysis must account for early-exit paths.
+inline bool mightHaveBreakingControlFlow(Region *region) {
+  Operation *parentOp = region->getParentOp();
+  return (!parentOp->isRegistered() ||
+          parentOp->hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+          isa<HasBreakingControlFlowOpInterface>(parentOp));
+}
+
+/// Return the HasBreakingControlFlowOpInterface operations potentially
+/// addressed by the given terminator. Returns an empty vector for non-breaking
+/// terminators or malformed target designators.
+SmallVector<HasBreakingControlFlowOpInterface>
+findPotentialBreakTargets(Operation *terminator);
+
+} // namespace mlir
+
 #endif // MLIR_IR_REGIONKINDINTERFACE_H_
diff --git a/mlir/include/mlir/IR/RegionKindInterface.td b/mlir/include/mlir/IR/RegionKindInterface.td
index 607001a89250e..25f88249e547d 100644
--- a/mlir/include/mlir/IR/RegionKindInterface.td
+++ b/mlir/include/mlir/IR/RegionKindInterface.td
@@ -61,4 +61,87 @@ def GraphRegionNoTerminator : TraitList<[
     HasOnlyGraphRegion
   ]>;
 
+// Indicates that this op may propagate a breaking control-flow event from a
+// nested region upward to the addressed HasBreakingControlFlowOpInterface
+// operation. The op does NOT consume the break itself; it is merely transparent
+// to it. All ops that sit between a breaking terminator and the
+// HasBreakingControlFlowOpInterface ancestor that will ultimately receive the
+// break must carry this trait.
+def PropagateControlFlowBreak : NativeOpTrait<"PropagateControlFlowBreak">;
+
+def HasBreakingControlFlowOpInterface : OpInterface<"HasBreakingControlFlowOpInterface"> {
+  let description = [{
+    Interface for operations that act as the target of a breaking control-flow
+    event (e.g. `scf.break` or `scf.continue`). Every intermediate op must
+    carry the `PropagateControlFlowBreak` trait. The terminator may identify the
+    receiver with any dialect-defined mechanism, such as a builtin token value,
+    an attribute, or a symbolic relationship to an enclosing operation.
+  }];
+  let cppNamespace = "::mlir";
+
+  let methods = [
+    StaticInterfaceMethod<
+      /*desc=*/[{
+        Return true if this operation accepts the given terminator operation
+        as a breaking-control-flow predecessor. This can be implemented through
+        the `HasNestedTerminator<[...]>` trait, otherwise an explicit implementation
+        must be provided (e.g. `return true;` to accept all, or a type check to
+        restrict).
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"acceptsTerminator",
+      /*args=*/(ins "Operation *":$op)
+    >,
+    InterfaceMethod<
+      /*desc=*/[{
+        Return true if this operation has at least one breaking terminator
+        nested inside it that targets this operation directly. Used to decide
+        whether post-dominance analysis must account for early-exit paths.
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"hasNestedPredecessors",
+      /*args=*/(ins),
+      /*methodBody=*/[{}],
+      /*defaultImplementation=*/[{
+        return ::mlir::hasNestedPredecessors(this->getOperation());
+      }]
+    >
+  ];
+}
+
+// OpInterface for terminators that leave their current region and identify
+// potential receiving operations using a dialect-defined mechanism. A normal
+// region return may simply target the immediately containing operation; a
+// breaking-control-flow region exit may target an ancestor operation that
+// implements HasBreakingControlFlowOpInterface.
+def RegionExitTerminatorOpInterface : OpInterface<"RegionExitTerminatorOpInterface"> {
+  let description = [{
+    Interface for terminators that leave their current region. Implemented by
+    ops such as `scf.yield`, `scf.break`, and `scf.continue`.
+  }];
+  let cppNamespace = "::mlir";
+
+  let methods = [
+    InterfaceMethod<
+      /*desc=*/[{
+        Return the operations that may receive this region-terminating control
+        flow event. Receivers for breaking control flow must implement
+        `HasBreakingControlFlowOpInterface`; normal region exits may return the
+        immediate parent operation.
+      }],
+      /*retTy=*/"::llvm::SmallVector<::mlir::Operation *>",
+      /*methodName=*/"getPotentialTargets",
+      /*args=*/(ins),
+      /*methodBody=*/[{}]
+    >
+  ];
+
+  let verify = [{
+    static_assert(ConcreteOp::template hasTrait<OpTrait::IsTerminator>(),
+                  "expected operation to be a terminator");
+    return success();
+  }];
+}
+
+
 #endif // MLIR_IR_REGIONKINDINTERFACE
diff --git a/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h b/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
index 48b80e2059e11..cd74de81a7b10 100644
--- a/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
+++ b/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
@@ -446,6 +446,12 @@ inline llvm::raw_ostream &operator<<(llvm::raw_ostream &os,
                            OpPrintingFlags().skipRegions())
             << ">";
 }
+
+/// Return the RegionBranchOpInterface operation that owns `successor` and
+/// defines its successor inputs.
+RegionBranchOpInterface
+getRegionBranchSuccessorOwner(RegionSuccessor successor);
+
 } // namespace mlir
 
 #endif // MLIR_INTERFACES_CONTROLFLOWINTERFACES_H
diff --git a/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp b/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
index 37aaf1f634a64..cf703785c7e08 100644
--- a/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
@@ -509,11 +509,15 @@ void DeadCodeAnalysis::visitRegionTerminator(Operation *op,
   if (!operands)
     return;
 
-  SmallVector<RegionSuccessor> successors;
   auto terminator = dyn_cast<RegionBranchTerminatorOpInterface>(op);
   if (!terminator)
     return;
+
+  SmallVector<RegionSuccessor> successors;
+  // Use operand-aware resolution to prune dead successors via constant folding
+  // (e.g. a scf.while condition known to be false).
   terminator.getSuccessorRegions(*operands, successors);
+
   visitRegionBranchEdges(branch, op, successors);
 }
 
@@ -540,8 +544,10 @@ void DeadCodeAnalysis::visitRegionBranchEdges(
     auto *predecessors = getOrCreate<PredecessorState>(point);
     propagateIfChanged(
         predecessors,
-        predecessors->join(predecessorOp,
-                           regionBranchOp.getSuccessorInputs(successor)));
+        predecessors->join(
+            predecessorOp,
+            getRegionBranchSuccessorOwner(successor).getSuccessorInputs(
+                successor)));
     LDBG() << "Added region branch as predecessor for successor: " << *point;
   }
 }
diff --git a/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp b/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
index a2a9018b851ea..dc30aae167ac7 100644
--- a/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
@@ -636,6 +636,9 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
   branch.getSuccessorRegions(branchPoint, successors);
   LDBG() << "  Processing " << successors.size() << " successor regions";
   for (const RegionSuccessor &successor : successors) {
+    RegionBranchOpInterface successorOwner =
+        getRegionBranchSuccessorOwner(successor);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
     const AbstractDenseLattice *after;
     if (successor.isOperation()) {
       LDBG() << "    Successor is operation";
@@ -643,7 +646,7 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
                             getProgramPointAfter(successor.getSuccessorOp()));
     } else if (successor.getSuccessor()->empty()) {
       LDBG() << "    Successor is empty region";
-      after = getLatticeFor(point, getProgramPointAfter(branch));
+      after = getLatticeFor(point, getProgramPointAfter(successorOwner));
     } else {
       Region *successorRegion = successor.getSuccessor();
       assert(!successorRegion->empty() && "unexpected empty successor region");
@@ -662,7 +665,7 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
     }
     LDBG() << "    After state: " << *after;
 
-    visitRegionBranchControlFlowTransfer(branch, branchPoint, successor, *after,
-                                         before);
+    visitRegionBranchControlFlowTransfer(successorOwner, branchPoint, successor,
+                                         *after, before);
   }
 }
diff --git a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
index 3d9a375bfb2b7..04f27c8bf622a 100644
--- a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
@@ -647,14 +647,23 @@ void AbstractSparseBackwardDataFlowAnalysis::
   // non-contiguous in the presence of multiple successors.
   BitVector unaccounted(terminator->getNumOperands(), true);
 
-  RegionBranchSuccessorMapping mapping;
-  branch.getSuccessorOperandInputMapping(mapping,
-                                         RegionBranchPoint(terminator));
-  for (const auto &[operand, inputs] : mapping) {
-    for (Value input : inputs) {
-      meet(getLatticeElement(operand->get()),
+  SmallVector<RegionSuccessor> successors;
+  branch.getSuccessorRegions(RegionBranchPoint(terminator), successors);
+
+  for (RegionSuccessor successor : successors) {
+    RegionBranchOpInterface successorOwner =
+        getRegionBranchSuccessorOwner(successor);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
+    OperandRange operands =
+        branch.getSuccessorOperands(RegionBranchPoint(terminator), successor);
+    ValueRange inputs = successorOwner.getSuccessorInputs(successor);
+    assert(operands.size() == inputs.size() &&
+           "expected the same number of operands and inputs");
+    MutableArrayRef<OpOperand> opOperands(operands.getBase(), operands.size());
+    for (const auto &[operand, input] : llvm::zip_equal(opOperands, inputs)) {
+      meet(getLatticeElement(operand.get()),
            *getLatticeElementFor(getProgramPointAfter(terminator), input));
-      unaccounted.reset(operand->getOperandNumber());
+      unaccounted.reset(operand.getOperandNumber());
     }
   }
 
diff --git a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
index 2972d79c4302f..9dc7a56152a3c 100644
--- a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
+++ b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
@@ -311,8 +311,81 @@ struct ForallLowering : public OpRewritePattern<mlir::scf::ForallOp> {
                                 PatternRewriter &rewriter) const override;
 };
 
+/// Lowers `scf.loop` to a CFG loop. The loop body is inlined; `scf.break` ops
+/// that target this loop become branches to the continuation block and
+/// `scf.continue` ops that target this loop become back-edges to the loop
+/// header.
+struct LoopOpLowering : public OpConversionPattern<LoopOp> {
+  using OpConversionPattern<LoopOp>::OpConversionPattern;
+  void initialize() { setHasBoundedRewriteRecursion(); }
+
+  LogicalResult
+  matchAndRewrite(LoopOp loopOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override;
+};
+
 } // namespace
 
+static LogicalResult lowerIfOpToCFG(IfOp ifOp, PatternRewriter &rewriter) {
+  auto loc = ifOp.getLoc();
+  // Start by splitting the block containing the 'scf.if' into two parts.
+  // The part before will contain the condition, the part after will be the
+  // continuation point.
+  auto *condBlock = rewriter.getInsertionBlock();
+  auto opPosition = rewriter.getInsertionPoint();
+  auto *remainingOpsBlock = rewriter.splitBlock(condBlock, opPosition);
+  Block *continueBlock;
+  if (ifOp.getNumResults() == 0) {
+    continueBlock = remainingOpsBlock;
+  } else {
+    continueBlock =
+        rewriter.createBlock(remainingOpsBlock, ifOp.getResultTypes(),
+                             SmallVector<Location>(ifOp.getNumResults(), loc));
+    cf::BranchOp::create(rewriter, loc, remainingOpsBlock);
+  }
+
+  // Move blocks from the "then" region to the region containing 'scf.if',
+  // place it before the continuation block, and branch to it.
+  auto &thenRegion = ifOp.getThenRegion();
+  if (thenRegion.empty() || thenRegion.back().empty())
+    return failure();
+  auto *thenBlock = &thenRegion.front();
+  Operation *thenTerminator = thenRegion.back().getTerminator();
+  ValueRange thenTerminatorOperands = thenTerminator->getOperands();
+  rewriter.setInsertionPointToEnd(&thenRegion.back());
+  if (isa<scf::YieldOp>(thenTerminator)) {
+    cf::BranchOp::create(rewriter, loc, continueBlock, thenTerminatorOperands);
+    rewriter.eraseOp(thenTerminator);
+  }
+  rewriter.inlineRegionBefore(thenRegion, continueBlock);
+
+  // Move blocks from the "else" region (if present) to the region containing
+  // 'scf.if', place it before the continuation block and branch to it.  It
+  // will be placed after the "then" regions.
+  auto *elseBlock = continueBlock;
+  auto &elseRegion = ifOp.getElseRegion();
+  if (!elseRegion.empty() && !elseRegion.back().empty()) {
+    elseBlock = &elseRegion.front();
+    Operation *elseTerminator = elseRegion.back().getTerminator();
+    ValueRange elseTerminatorOperands = elseTerminator->getOperands();
+    rewriter.setInsertionPointToEnd(&elseRegion.back());
+    if (isa<scf::YieldOp>(elseTerminator)) {
+      cf::BranchOp::create(rewriter, loc, continueBlock,
+                           elseTerminatorOperands);
+      rewriter.eraseOp(elseTerminator);
+    }
+    rewriter.inlineRegionBefore(elseRegion, continueBlock);
+  }
+
+  rewriter.setInsertionPointToEnd(condBlock);
+  cf::CondBranchOp::create(rewriter, loc, ifOp.getCondition(), thenBlock,
+                           /*trueArgs=*/ArrayRef<Value>(), elseBlock,
+                           /*falseArgs=*/ArrayRef<Value>());
+
+  rewriter.replaceOp(ifOp, continueBlock->getArguments());
+  return success();
+}
+
 static void propagateLoopAttrs(Operation *scfOp, Operation *brOp) {
   // Let the CondBranchOp carry the LLVM attributes from the ForOp, such as the
   // llvm.loop_annotation attribute.
@@ -400,58 +473,7 @@ LogicalResult ForLowering::matchAndRewrite(ForOp forOp,
 
 LogicalResult IfLowering::matchAndRewrite(IfOp ifOp,
                                           PatternRewriter &rewriter) const {
-  auto loc = ifOp.getLoc();
-
-  // Start by splitting the block containing the 'scf.if' into two parts.
-  // The part before will contain the condition, the part after will be the
-  // continuation point.
-  auto *condBlock = rewriter.getInsertionBlock();
-  auto opPosition = rewriter.getInsertionPoint();
-  auto *remainingOpsBlock = rewriter.splitBlock(condBlock, opPosition);
-  Block *continueBlock;
-  if (ifOp.getNumResults() == 0) {
-    continueBlock = remainingOpsBlock;
-  } else {
-    continueBlock =
-        rewriter.createBlock(remainingOpsBlock, ifOp.getResultTypes(),
-                             SmallVector<Location>(ifOp.getNumResults(), loc));
-    cf::BranchOp::create(rewriter, loc, remainingOpsBlock);
-  }
-
-  // Move blocks from the "then" region to the region containing 'scf.if',
-  // place it before the continuation block, and branch to it.
-  auto &thenRegion = ifOp.getThenRegion();
-  auto *thenBlock = &thenRegion.front();
-  Operation *thenTerminator = thenRegion.back().getTerminator();
-  ValueRange thenTerminatorOperands = thenTerminator->getOperands();
-  rewriter.setInsertionPointToEnd(&thenRegion.back());
-  cf::BranchOp::create(rewriter, loc, continueBlock, thenTerminatorOperands);
-  rewriter.eraseOp(thenTerminator);
-  rewriter.inlineRegionBefore(thenRegion, continueBlock);
-
-  // Move blocks from the "else" region (if present) to the region containing
-  // 'scf.if', place it before the continuation block and branch to it.  It
-  // will be placed after the "then" regions.
-  auto *elseBlock = continueBlock;
-  auto &elseRegion = ifOp.getElseRegion();
-  if (!elseRegion.empty()) {
-    elseBlock = &elseRegion.front();
-    Operation *elseTerminator = elseRegion.back().getTerminator();
-    ValueRange elseTerminatorOperands = elseTerminator->getOperands();
-    rewriter.setInsertionPointToEnd(&elseRegion.back());
-    cf::BranchOp::create(rewriter, loc, continueBlock, elseTerminatorOperands);
-    rewriter.eraseOp(elseTerminator);
-    rewriter.inlineRegionBefore(elseRegion, continueBlock);
-  }
-
-  rewriter.setInsertionPointToEnd(condBlock);
-  cf::CondBranchOp::create(rewriter, loc, ifOp.getCondition(), thenBlock,
-                           /*trueArgs=*/ArrayRef<Value>(), elseBlock,
-                           /*falseArgs=*/ArrayRef<Value>());
-
-  // Ok, we're done!
-  rewriter.replaceOp(ifOp, continueBlock->getArguments());
-  return success();
+  return lowerIfOpToCFG(ifOp, rewriter);
 }
 
 LogicalResult
@@ -719,10 +741,114 @@ LogicalResult ForallLowering::matchAndRewrite(ForallOp forallOp,
   return scf::forallToParallelLoop(rewriter, forallOp);
 }
 
+LogicalResult
+LoopOpLowering::matchAndRewrite(LoopOp loopOp, OpAdaptor adaptor,
+                                ConversionPatternRewriter &rewriter) const {
+  {
+    OpBuilder::InsertionGuard guard(rewriter);
+    if (failed(rewriter.legalize(&loopOp.getRegion())))
+      return rewriter.notifyMatchFailure(loopOp,
+                                         "failed to convert nested region");
+  }
+
+  // Handle degenerate case before modifying any IR.
+  Region &bodyRegion = loopOp.getRegion();
+  if (bodyRegion.empty() || bodyRegion.front().empty()) {
+    rewriter.eraseOp(loopOp);
+    return success();
+  }
+
+  NestedBreakingControlFlowInfo nestedBreakingControlFlow =
+      getNestedBreakingControlFlowInfo(loopOp);
+  if (nestedBreakingControlFlow.hasNestedPredecessors)
+    return rewriter.notifyMatchFailure(loopOp,
+                                       "loop op with nested predecessors");
+
+  // Collect direct predecessors (break/continue targeting this loop).
+  SmallVector<Operation *> predecessors;
+  llvm::append_range(predecessors, nestedBreakingControlFlow.predecessors);
+
+  // Lower `scf.loop` to CFG by converting breaks/continues to branches.
+  Location loc = loopOp.getLoc();
+  BlockArgument controlToken = loopOp.getControlToken();
+  // Split the block containing loopOp into the init block and continuation.
+  Block *initBlock = rewriter.getInsertionBlock();
+  auto initPos = rewriter.getInsertionPoint();
+  Block *continueBlock = rewriter.splitBlock(initBlock, initPos);
+  continueBlock->addArguments(
+      loopOp.getResultTypes(),
+      SmallVector<Location>(loopOp.getNumResults(), loc));
+  Block *loopBody = &bodyRegion.front();
+
+  // After the split above, `initBlock` holds the ops that preceded the loop and
+  // `continueBlock` holds the ops that followed it. Position the builder at the
+  // end of `initBlock`; the branch into the loop body is emitted there below.
+  rewriter.setInsertionPoint(initBlock, initBlock->end());
+
+  // Collect the blocks from the loop body region before inlining so we can
+  // restrict the scf.yield scan to only those blocks (not the whole function).
+  SmallVector<Block *> inlinedBlocks;
+  for (Block &b : bodyRegion)
+    inlinedBlocks.push_back(&b);
+
+  // Move all blocks from the scf.loop region before continueBlock.
+  rewriter.inlineRegionBefore(bodyRegion, continueBlock);
+  // We will remember all break/continue ops to fix up after.
+  SmallVector<Operation *> toErase;
+
+  for (auto predecessor : predecessors) {
+    if (auto breakOp = dyn_cast<scf::BreakOp>(predecessor)) {
+      rewriter.setInsertionPointAfter(breakOp);
+      cf::BranchOp::create(rewriter, breakOp->getLoc(), continueBlock,
+                           ValueRange{breakOp.getArgs()});
+    } else if (auto contOp = dyn_cast<scf::ContinueOp>(predecessor)) {
+      rewriter.setInsertionPointAfter(contOp);
+      cf::BranchOp::create(rewriter, contOp->getLoc(), loopBody,
+                           ValueRange{contOp.getArgs()});
+    }
+    toErase.push_back(predecessor);
+  }
+
+  // Erase the old scf.break/scf.continue ops. Drop their operands first so the
+  // control token (operand #0) loses its last uses here; this lets the
+  // `controlToken.use_empty()` check below confirm no stray consumer remains
+  // before we erase the token block argument.
+  for (Operation *op : toErase) {
+    op->setOperands({});
+    rewriter.eraseOp(op);
+  }
+
+  if (!controlToken.use_empty())
+    return rewriter.notifyMatchFailure(loopOp, "loop token still has uses");
+  loopBody->eraseArgument(0);
+
+  // The loop region is now a CFG. Jump from initBlock to the loop body.
+  rewriter.setInsertionPointToEnd(initBlock);
+  cf::BranchOp::create(rewriter, loc, loopBody,
+                       ValueRange{loopOp.getOperands()});
+
+  // Replace any remaining scf.yield terminators in the inlined loop blocks
+  // with branches back to the loop header (treat as implicit continue).
+  for (Block *block : inlinedBlocks) {
+    if (auto yield = dyn_cast<scf::YieldOp>(block->getTerminator())) {
+      rewriter.setInsertionPoint(yield);
+      cf::BranchOp::create(rewriter, yield.getLoc(), loopBody,
+                           yield.getOperands());
+      rewriter.eraseOp(yield);
+    }
+  }
+
+  // Replace the original scf.loop op with a branch to continueBlock assigning
+  // results.
+  rewriter.replaceOp(loopOp, continueBlock->getArguments());
+  return success();
+}
+
 void mlir::populateSCFToControlFlowConversionPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<ForallLowering, ForLowering, IfLowering, ParallelLowering,
-               WhileLowering, ExecuteRegionLowering, IndexSwitchLowering>(
+  patterns.add<IfLowering>(patterns.getContext(), /*benefit=*/2);
+  patterns.add<ForallLowering, ForLowering, ParallelLowering, WhileLowering,
+               ExecuteRegionLowering, IndexSwitchLowering, LoopOpLowering>(
       patterns.getContext());
   patterns.add<DoWhileLowering>(patterns.getContext(), /*benefit=*/2);
 }
@@ -734,7 +860,8 @@ void SCFToControlFlowPass::runOnOperation() {
   // Configure conversion to lower out SCF operations.
   ConversionTarget target(getContext());
   target.addIllegalOp<scf::ForallOp, scf::ForOp, scf::IfOp, scf::IndexSwitchOp,
-                      scf::ParallelOp, scf::WhileOp, scf::ExecuteRegionOp>();
+                      scf::LoopOp, scf::ParallelOp, scf::WhileOp,
+                      scf::ExecuteRegionOp>();
   target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
   ConversionConfig config;
   config.allowPatternRollback = allowPatternRollback;
diff --git a/mlir/lib/Dialect/SCF/IR/SCF.cpp b/mlir/lib/Dialect/SCF/IR/SCF.cpp
index 54d28783ddc4a..f2b1d827a47f7 100644
--- a/mlir/lib/Dialect/SCF/IR/SCF.cpp
+++ b/mlir/lib/Dialect/SCF/IR/SCF.cpp
@@ -132,6 +132,17 @@ std::optional<llvm::APSInt> mlir::scf::computeUbMinusLb(Value lb, Value ub,
 // ExecuteRegionOp
 //===----------------------------------------------------------------------===//
 
+/// Erase all operations after `afterOp` in the same block (not including
+/// afterOp itself). Walks backwards to avoid use-after-free.
+static void eraseOpsAfter(PatternRewriter &rewriter, Operation *afterOp) {
+  Operation *cur = &afterOp->getBlock()->back();
+  while (cur != afterOp) {
+    Operation *prev = cur->getPrevNode();
+    rewriter.eraseOp(cur);
+    cur = prev;
+  }
+}
+
 ///
 /// (ssa-id `=`)? `execute_region` `->` function-result-type `{`
 ///    block+
@@ -282,6 +293,10 @@ ValueRange ExecuteRegionOp::getSuccessorInputs(RegionSuccessor successor) {
                                  : ValueRange();
 }
 
+SmallVector<Operation *> YieldOp::getPotentialTargets() {
+  return {getOperation()->getParentOp()};
+}
+
 //===----------------------------------------------------------------------===//
 // ConditionOp
 //===----------------------------------------------------------------------===//
@@ -311,6 +326,316 @@ void ConditionOp::getSuccessorRegions(
     regions.push_back(RegionSuccessor(whileOp.getOperation()));
 }
 
+//===----------------------------------------------------------------------===//
+// LoopOp
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// Control Flow Op Utilities
+//===----------------------------------------------------------------------===//
+
+template <typename OpT>
+static ParseResult
+parseControlFlowRegion(OpAsmParser &p, Region &region,
+                       ArrayRef<OpAsmParser::Argument> arguments = {}) {
+  if (failed(p.parseRegion(region, arguments)))
+    return failure();
+  OpT::ensureTerminator(region, p.getBuilder(),
+                        p.getEncodedSourceLoc(p.getNameLoc()));
+  return success();
+}
+
+static LoopOp getLoopTargetFromToken(Value target) {
+  auto targetArg = cast<BlockArgument>(target);
+  auto loopOp = cast<LoopOp>(targetArg.getOwner()->getParentOp());
+  assert(targetArg == loopOp.getControlToken() &&
+         "expected target token to be the loop control token");
+  return loopOp;
+}
+
+static LogicalResult verifyLoopTerminatorTarget(Operation *terminator,
+                                                Value target) {
+  auto targetArg = dyn_cast<BlockArgument>(target);
+  if (!targetArg)
+    return terminator->emitOpError()
+           << "target token must be an entry block argument of an scf.loop";
+
+  Block *targetBlock = targetArg.getOwner();
+  auto loopOp = dyn_cast_or_null<LoopOp>(targetBlock->getParentOp());
+  if (!loopOp || targetArg != loopOp.getControlToken())
+    return terminator->emitOpError()
+           << "target token must be the control token of an scf.loop";
+
+  Operation *currentOp = terminator->getParentOp();
+  while (currentOp && currentOp != loopOp.getOperation()) {
+    if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      return terminator->emitOpError()
+             << "target token crosses an op that does not have the "
+                "PropagateControlFlowBreak trait: "
+             << OpWithFlags(currentOp, OpPrintingFlags().skipRegions());
+    currentOp = currentOp->getParentOp();
+  }
+
+  if (!currentOp)
+    return terminator->emitOpError()
+           << "target token must be defined by an enclosing scf.loop";
+
+  if (!loopOp.acceptsTerminator(terminator))
+    return loopOp.emitOpError("does not accept terminator: ")
+           << OpWithFlags(terminator, OpPrintingFlags().skipRegions());
+
+  return success();
+}
+
+static bool terminatorPropagatesThrough(Operation *terminator, Operation *op) {
+  for (HasBreakingControlFlowOpInterface target :
+       findPotentialBreakTargets(terminator)) {
+    if (target.getOperation() != op &&
+        target.getOperation()->isProperAncestor(op))
+      return true;
+  }
+  return false;
+}
+
+static void appendPropagatedTerminatorSuccessors(
+    RegionBranchTerminatorOpInterface terminator, Operation *op,
+    SmallVectorImpl<RegionSuccessor> &regions) {
+  for (HasBreakingControlFlowOpInterface target :
+       findPotentialBreakTargets(terminator)) {
+    Operation *targetOp = target.getOperation();
+    if (targetOp == op || !targetOp->isProperAncestor(op))
+      continue;
+    auto targetBranch = dyn_cast<RegionBranchOpInterface>(targetOp);
+    assert(targetBranch && "expected breaking target to model region branch");
+    targetBranch.getSuccessorRegions(RegionBranchPoint(terminator), regions);
+  }
+}
+
+LogicalResult BreakOp::verify() {
+  return verifyLoopTerminatorTarget(getOperation(), getTargetToken());
+}
+
+SmallVector<Operation *> BreakOp::getPotentialTargets() {
+  LoopOp loopOp = getLoopTargetFromToken(getTargetToken());
+  return {loopOp.getOperation()};
+}
+
+MutableOperandRange
+BreakOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult ContinueOp::verify() {
+  return verifyLoopTerminatorTarget(getOperation(), getTargetToken());
+}
+
+SmallVector<Operation *> ContinueOp::getPotentialTargets() {
+  LoopOp loopOp = getLoopTargetFromToken(getTargetToken());
+  return {loopOp.getOperation()};
+}
+
+MutableOperandRange
+ContinueOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult LoopOp::verifyRegions() {
+  // Check matching between the operands and the region arguments.
+  if (getRegion().empty())
+    return emitOpError("region cannot be empty");
+  if (getRegion().front().getNumArguments() != getNumOperands() + 1)
+    return emitOpError("expected the region to have one argument per "
+                       "loop-carried value plus the leading control token (")
+           << getNumOperands() + 1 << " expected, but got "
+           << getRegion().front().getNumArguments() << ")";
+  if (!isa<TokenType>(getControlToken().getType()))
+    return emitOpError("first region argument must be a control token");
+  for (auto [index, argAndOperand] :
+       llvm::enumerate(llvm::zip(getRegionIterValues(), getOperands()))) {
+    auto argType = std::get<0>(argAndOperand).getType();
+    auto operandType = std::get<1>(argAndOperand).getType();
+    if (argType != operandType)
+      return emitOpError() << "types mismatch between " << index
+                           << "th iter operand (" << operandType
+                           << ") and defined region argument (" << argType
+                           << ")";
+  }
+  return success();
+}
+
+void LoopOp::print(OpAsmPrinter &p) {
+  p << " token(" << getControlToken() << ") ";
+  bool hasIters = !getInitValues().empty();
+  bool hasReturn = !getResultTypes().empty();
+
+  if (hasIters) {
+    p << "iter_args(";
+    llvm::interleaveComma(
+        llvm::zip(getRegionIterValues(), getInitValues()), p,
+        [&](auto it) { p << std::get<0>(it) << " = " << std::get<1>(it); });
+    p << ") : ";
+    p << getInitValues().getTypes();
+    p << " ";
+  }
+  if (hasReturn) {
+    p << "-> ";
+    p << getResultTypes();
+    p << " ";
+  }
+
+  Operation *terminator = getRegion().front().getTerminator();
+  // Elide the terminator only when it is the trivial implicit terminator:
+  // a `scf.continue` that targets this loop's control token and carries no
+  // iter values. A single-operand `scf.continue` targeting an *outer* loop
+  // must be printed explicitly, otherwise the implicit terminator rebuilt on
+  // parse would silently retarget it to this loop.
+  auto continueOp = dyn_cast<ContinueOp>(terminator);
+  bool printBlockTerminators = !continueOp || !continueOp.getArgs().empty() ||
+                               continueOp.getTargetToken() != getControlToken();
+  p.printRegion(getRegion(), /*printEntryBlockArgs=*/false,
+                printBlockTerminators);
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+ParseResult LoopOp::parse(OpAsmParser &parser, OperationState &result) {
+  SmallVector<OpAsmParser::Argument, 4> regionArgs;
+  SmallVector<OpAsmParser::Argument, 4> iterRegionArgs;
+  SmallVector<OpAsmParser::UnresolvedOperand, 4> iterOperands;
+  SmallVector<Type, 4> iterTypes;
+
+  OpAsmParser::Argument controlToken;
+  if (parser.parseKeyword("token") || parser.parseLParen() ||
+      parser.parseArgument(controlToken) || parser.parseRParen())
+    return failure();
+  controlToken.type = parser.getBuilder().getType<TokenType>();
+  regionArgs.push_back(controlToken);
+
+  if (failed(parser.parseOptionalKeyword("iter_args"))) {
+    // no iter_args, but can still have a return type
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+  } else {
+    // iter_args are present and must have colon followed by types
+    if (parser.parseAssignmentList(iterRegionArgs, iterOperands) ||
+        parser.parseColon() || parser.parseTypeList(iterTypes))
+      return failure();
+    if (iterRegionArgs.size() != iterTypes.size())
+      return parser.emitError(parser.getCurrentLocation(),
+                              "found different number of iter_args and types");
+    // check for optional result type(s)
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+    // Set region argument types for loop body
+    for (auto [regionArg, type] : llvm::zip_equal(iterRegionArgs, iterTypes)) {
+      regionArg.type = type;
+    }
+    llvm::append_range(regionArgs, iterRegionArgs);
+  }
+
+  // Parse region and attr dict.
+  if (parseControlFlowRegion<LoopOp>(parser, *result.addRegion(), regionArgs) ||
+      parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  // Resolve operands.
+  if (parser.resolveOperands(iterOperands, iterTypes, parser.getNameLoc(),
+                             result.operands))
+    return failure();
+
+  return success();
+}
+
+void LoopOp::getSuccessorRegions(RegionBranchPoint point,
+                                 SmallVectorImpl<RegionSuccessor> &regions) {
+  if (point.isParent()) {
+    regions.push_back(RegionSuccessor(&getRegion()));
+    return;
+  }
+
+  // Otherwise, it depends on the terminator: a continue branches back to the
+  // body and a break to the parent.
+  RegionBranchTerminatorOpInterface terminator =
+      point.getTerminatorPredecessorOrNull();
+  if (terminator && terminatorPropagatesThrough(terminator, getOperation())) {
+    appendPropagatedTerminatorSuccessors(terminator, getOperation(), regions);
+    return;
+  }
+
+  if (isa<ContinueOp>(terminator)) {
+    regions.push_back(RegionSuccessor(&getRegion()));
+    return;
+  }
+  assert(isa<BreakOp>(terminator) && "expected continue or break terminator");
+
+  regions.push_back(RegionSuccessor(getOperation()));
+}
+
+OperandRange LoopOp::getEntrySuccessorOperands(RegionSuccessor successor) {
+  return getInitValues();
+}
+
+ValueRange LoopOp::getSuccessorInputs(RegionSuccessor successor) {
+  return successor.isOperation() ? ValueRange(getResults())
+                                 : ValueRange(getRegionIterValues());
+}
+
+namespace {
+
+/// Rewriting pattern that erases loops that have a single iteration.
+struct SimplifyTrivialLoops : public OpRewritePattern<LoopOp> {
+  using OpRewritePattern<LoopOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(LoopOp op,
+                                PatternRewriter &rewriter) const override {
+    // Terminator must be a break.
+    auto breakOp = dyn_cast<BreakOp>(op.getBody()->getTerminator());
+    if (!breakOp)
+      return rewriter.notifyMatchFailure(op, "loop terminator isn't a break");
+    SmallVector<HasBreakingControlFlowOpInterface> targets =
+        findPotentialBreakTargets(breakOp);
+    if (targets.size() != 1 ||
+        targets.front().getOperation() != op.getOperation())
+      return rewriter.notifyMatchFailure(
+          op, "loop terminator targets another loop");
+
+    // If it has nested predecessors, it can't be trivially simplified.
+    if (hasNestedPredecessors(op))
+      return rewriter.notifyMatchFailure(op, "has nested predecessors");
+
+    // Great: it is a single iteration loop, we can simplify it.
+    Block *body = op.getBody();
+    SmallVector<Value> replacements;
+    for (Value value : breakOp.getArgs()) {
+      if (auto blockArg = dyn_cast<BlockArgument>(value);
+          blockArg && blockArg.getOwner() == body) {
+        if (blockArg.getArgNumber() == 0)
+          return rewriter.notifyMatchFailure(
+              op, "loop terminator cannot yield the control token");
+        replacements.push_back(op.getInitValues()[blockArg.getArgNumber() - 1]);
+        continue;
+      }
+      replacements.push_back(value);
+    }
+    rewriter.eraseOp(breakOp);
+    assert(op.getControlToken().use_empty() && "expected token to be unused");
+    body->eraseArgument(0);
+    rewriter.inlineBlockBefore(body, op, op.getInitValues());
+    rewriter.replaceOp(op, replacements);
+
+    return success();
+  }
+};
+} // namespace
+
+void LoopOp::getCanonicalizationPatterns(RewritePatternSet &results,
+                                         MLIRContext *context) {
+  results.add<SimplifyTrivialLoops>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // ForOp
 //===----------------------------------------------------------------------===//
@@ -1940,13 +2265,21 @@ IfOp::inferReturnTypes(MLIRContext *ctx, std::optional<Location> loc,
   Region *r = &adaptor.getThenRegion();
   if (r->empty())
     return failure();
-  Block &b = r->front();
-  if (b.empty())
-    return failure();
-  auto yieldOp = llvm::dyn_cast<YieldOp>(b.back());
-  if (!yieldOp)
+  Block *b = &r->front();
+  if (b->empty())
     return failure();
-  TypeRange types = yieldOp.getOperandTypes();
+  Operation *terminator = &b->back();
+  if (terminatorPropagatesThrough(terminator, terminator->getParentOp())) {
+    if (adaptor.getElseRegion().empty())
+      return success();
+    b = &adaptor.getElseRegion().front();
+    if (b->empty())
+      return success();
+    terminator = &b->back();
+    if (terminatorPropagatesThrough(terminator, terminator->getParentOp()))
+      return success();
+  }
+  TypeRange types = terminator->getOperandTypes();
   llvm::append_range(inferredReturnTypes, types);
   return success();
 }
@@ -2071,7 +2404,9 @@ ParseResult IfOp::parse(OpAsmParser &parser, OperationState &result) {
 }
 
 void IfOp::print(OpAsmPrinter &p) {
-  bool printBlockTerminators = false;
+  bool printBlockTerminators =
+      !isa<YieldOp>(thenBlock()->back()) ||
+      (elseBlock() && !isa<YieldOp>(elseBlock()->back()));
 
   p << " " << getCondition();
   if (!getResults().empty()) {
@@ -2101,6 +2436,16 @@ void IfOp::getSuccessorRegions(RegionBranchPoint point,
   // The `then` and the `else` region branch back to the parent operation or one
   // of the recursive parent operations (early exit case).
   if (!point.isParent()) {
+    // Propagating breaks/continues pass through this if-op to reach an
+    // enclosing loop. Don't report parent() as a successor for them; they
+    // don't yield values to this if-op.
+    if (auto terminator = point.getTerminatorPredecessorOrNull()) {
+      if (terminatorPropagatesThrough(terminator, getOperation())) {
+        appendPropagatedTerminatorSuccessors(terminator, getOperation(),
+                                             regions);
+        return;
+      }
+    }
     regions.push_back(RegionSuccessor(getOperation()));
     return;
   }
@@ -2185,9 +2530,14 @@ struct ConvertTrivialIfToSelect : public OpRewritePattern<IfOp> {
     if (op->getNumResults() == 0)
       return failure();
 
+    YieldOp thenYield = dyn_cast<YieldOp>(op.thenTerminator());
+    YieldOp elseYield = dyn_cast<YieldOp>(op.elseTerminator());
+    if (!thenYield || !elseYield)
+      return failure();
+
     auto cond = op.getCondition();
-    auto thenYieldArgs = op.thenYield().getOperands();
-    auto elseYieldArgs = op.elseYield().getOperands();
+    auto thenYieldArgs = thenYield.getOperands();
+    auto elseYieldArgs = elseYield.getOperands();
 
     SmallVector<Type> nonHoistable;
     for (auto [trueVal, falseVal] : llvm::zip(thenYieldArgs, elseYieldArgs)) {
@@ -2231,10 +2581,12 @@ struct ConvertTrivialIfToSelect : public OpRewritePattern<IfOp> {
     }
 
     rewriter.setInsertionPointToEnd(replacement.thenBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(replacement.thenYield(), trueYields);
+    rewriter.replaceOpWithNewOp<YieldOp>(replacement.thenTerminator(),
+                                         trueYields);
 
     rewriter.setInsertionPointToEnd(replacement.elseBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(replacement.elseYield(), falseYields);
+    rewriter.replaceOpWithNewOp<YieldOp>(replacement.elseTerminator(),
+                                         falseYields);
 
     rewriter.replaceOp(op, results);
     return success();
@@ -2386,36 +2738,35 @@ struct ReplaceIfYieldWithConditionOrValue : public OpRewritePattern<IfOp> {
     if (op.getNumResults() == 0)
       return failure();
 
-    auto trueYield =
-        cast<scf::YieldOp>(op.getThenRegion().back().getTerminator());
-    auto falseYield =
-        cast<scf::YieldOp>(op.getElseRegion().back().getTerminator());
+    YieldOp thenYield = dyn_cast<YieldOp>(op.thenTerminator());
+    YieldOp elseYield = dyn_cast<YieldOp>(op.elseTerminator());
+    if (!thenYield || !elseYield)
+      return failure();
 
     rewriter.setInsertionPoint(op->getBlock(),
                                op.getOperation()->getIterator());
     bool changed = false;
     Type i1Ty = rewriter.getI1Type();
-    for (auto [trueResult, falseResult, opResult] :
-         llvm::zip(trueYield.getResults(), falseYield.getResults(),
-                   op.getResults())) {
-      if (trueResult == falseResult) {
+    for (auto [thenResult, elseResult, opResult] : llvm::zip(
+             thenYield.getResults(), elseYield.getResults(), op.getResults())) {
+      if (thenResult == elseResult) {
         if (!opResult.use_empty()) {
-          opResult.replaceAllUsesWith(trueResult);
+          opResult.replaceAllUsesWith(thenResult);
           changed = true;
         }
         continue;
       }
 
-      BoolAttr trueYield, falseYield;
-      if (!matchPattern(trueResult, m_Constant(&trueYield)) ||
-          !matchPattern(falseResult, m_Constant(&falseYield)))
+      BoolAttr thenYield, elseYield;
+      if (!matchPattern(thenResult, m_Constant(&thenYield)) ||
+          !matchPattern(elseResult, m_Constant(&elseYield)))
         continue;
 
-      bool trueVal = trueYield.getValue();
-      bool falseVal = falseYield.getValue();
-      if (!trueVal && falseVal) {
+      bool thenVal = thenYield.getValue();
+      bool elseVal = elseYield.getValue();
+      if (!thenVal && elseVal) {
         if (!opResult.use_empty()) {
-          Dialect *constDialect = trueResult.getDefiningOp()->getDialect();
+          Dialect *constDialect = thenResult.getDefiningOp()->getDialect();
           Value notCond = arith::XOrIOp::create(
               rewriter, op.getLoc(), op.getCondition(),
               constDialect
@@ -2427,7 +2778,7 @@ struct ReplaceIfYieldWithConditionOrValue : public OpRewritePattern<IfOp> {
           changed = true;
         }
       }
-      if (trueVal && !falseVal) {
+      if (thenVal && !elseVal) {
         if (!opResult.use_empty()) {
           opResult.replaceAllUsesWith(op.getCondition());
           changed = true;
@@ -2482,18 +2833,16 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
       nextThen = nextIf.thenBlock();
       if (!nextIf.getElseRegion().empty())
         nextElse = nextIf.elseBlock();
-    }
-    if (arith::XOrIOp notv =
-            nextIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
+    } else if (arith::XOrIOp notv =
+                   nextIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
       if (notv.getLhs() == prevIf.getCondition() &&
           matchPattern(notv.getRhs(), m_One())) {
         nextElse = nextIf.thenBlock();
         if (!nextIf.getElseRegion().empty())
           nextThen = nextIf.elseBlock();
       }
-    }
-    if (arith::XOrIOp notv =
-            prevIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
+    } else if (arith::XOrIOp notv =
+                   prevIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
       if (notv.getLhs() == nextIf.getCondition() &&
           matchPattern(notv.getRhs(), m_One())) {
         nextElse = nextIf.thenBlock();
@@ -2504,14 +2853,25 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
 
     if (!nextThen && !nextElse)
       return failure();
+    // Check that the terminators are all YieldOp
+    if (!isa<YieldOp>(prevIf.thenTerminator()) ||
+        (nextThen && !isa<YieldOp>(nextThen->getTerminator())))
+      return failure();
+    if (!prevIf.getElseRegion().empty() &&
+        !isa<YieldOp>(prevIf.elseTerminator()))
+      return failure();
+    if (nextElse && !nextElse->empty() &&
+        !isa<YieldOp>(nextElse->getTerminator()))
+      return failure();
 
     SmallVector<Value> prevElseYielded;
     if (!prevIf.getElseRegion().empty())
-      prevElseYielded = prevIf.elseYield().getOperands();
+      prevElseYielded = prevIf.elseTerminator()->getOperands();
     // Replace all uses of return values of op within nextIf with the
     // corresponding yields
-    for (auto it : llvm::zip(prevIf.getResults(),
-                             prevIf.thenYield().getOperands(), prevElseYielded))
+    for (auto it :
+         llvm::zip(prevIf.getResults(), prevIf.thenTerminator()->getOperands(),
+                   prevElseYielded))
       for (OpOperand &use :
            llvm::make_early_inc_range(std::get<0>(it).getUses())) {
         if (nextThen && nextThen->getParent()->isAncestor(
@@ -2539,16 +2899,16 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
                                 combinedIf.getThenRegion().begin());
 
     if (nextThen) {
-      YieldOp thenYield = combinedIf.thenYield();
-      YieldOp thenYield2 = cast<YieldOp>(nextThen->getTerminator());
+      Operation *thenTerminator = combinedIf.thenTerminator();
+      Operation *thenTerminator2 = nextThen->getTerminator();
       rewriter.mergeBlocks(nextThen, combinedIf.thenBlock());
       rewriter.setInsertionPointToEnd(combinedIf.thenBlock());
 
-      SmallVector<Value> mergedYields(thenYield.getOperands());
-      llvm::append_range(mergedYields, thenYield2.getOperands());
-      YieldOp::create(rewriter, thenYield2.getLoc(), mergedYields);
-      rewriter.eraseOp(thenYield);
-      rewriter.eraseOp(thenYield2);
+      SmallVector<Value> mergedYields(thenTerminator->getOperands());
+      llvm::append_range(mergedYields, thenTerminator2->getOperands());
+      YieldOp::create(rewriter, thenTerminator->getLoc(), mergedYields);
+      rewriter.eraseOp(thenTerminator);
+      rewriter.eraseOp(thenTerminator2);
     }
 
     rewriter.inlineRegionBefore(prevIf.getElseRegion(),
@@ -2561,18 +2921,17 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
                                     combinedIf.getElseRegion(),
                                     combinedIf.getElseRegion().begin());
       } else {
-        YieldOp elseYield = combinedIf.elseYield();
-        YieldOp elseYield2 = cast<YieldOp>(nextElse->getTerminator());
+        Operation *elseTerminator = combinedIf.elseTerminator();
+        Operation *elseTerminator2 = nextElse->getTerminator();
         rewriter.mergeBlocks(nextElse, combinedIf.elseBlock());
-
         rewriter.setInsertionPointToEnd(combinedIf.elseBlock());
 
-        SmallVector<Value> mergedElseYields(elseYield.getOperands());
-        llvm::append_range(mergedElseYields, elseYield2.getOperands());
+        SmallVector<Value> mergedElseYields(elseTerminator->getOperands());
+        llvm::append_range(mergedElseYields, elseTerminator2->getOperands());
 
-        YieldOp::create(rewriter, elseYield2.getLoc(), mergedElseYields);
-        rewriter.eraseOp(elseYield);
-        rewriter.eraseOp(elseYield2);
+        YieldOp::create(rewriter, elseTerminator->getLoc(), mergedElseYields);
+        rewriter.eraseOp(elseTerminator);
+        rewriter.eraseOp(elseTerminator2);
       }
     }
 
@@ -2600,7 +2959,8 @@ struct RemoveEmptyElseBranch : public OpRewritePattern<IfOp> {
     if (ifOp.getNumResults())
       return failure();
     Block *elseBlock = ifOp.elseBlock();
-    if (!elseBlock || !llvm::hasSingleElement(*elseBlock))
+    if (!elseBlock || (!llvm::hasSingleElement(*elseBlock) ||
+                       !isa<YieldOp>(elseBlock->getTerminator())))
       return failure();
     auto newIfOp = rewriter.cloneWithoutRegions(ifOp);
     rewriter.inlineRegionBefore(ifOp.getThenRegion(), newIfOp.getThenRegion(),
@@ -2636,21 +2996,32 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     if (!llvm::hasSingleElement(nestedOps))
       return failure();
 
+    auto nestedIf = dyn_cast<IfOp>(*nestedOps.begin());
+    if (!nestedIf)
+      return failure();
+
+    // Terminator must be a YieldOp
+    if (!isa<YieldOp>(op.thenTerminator()))
+      return failure();
+
     // If there is an else block, it can only yield
-    if (op.elseBlock() && !llvm::hasSingleElement(*op.elseBlock()))
+    if (op.elseBlock() && (!llvm::hasSingleElement(*op.elseBlock()) ||
+                           !isa<YieldOp>(op.elseTerminator())))
       return failure();
 
-    auto nestedIf = dyn_cast<IfOp>(*nestedOps.begin());
-    if (!nestedIf)
+    // Same for the nested if: the then and else blocks can only yield.
+    if (!isa<YieldOp>(nestedIf.thenTerminator()))
       return failure();
 
-    if (nestedIf.elseBlock() && !llvm::hasSingleElement(*nestedIf.elseBlock()))
+    if (nestedIf.elseBlock() &&
+        (!llvm::hasSingleElement(*nestedIf.elseBlock()) ||
+         !isa<YieldOp>(nestedIf.elseTerminator())))
       return failure();
 
-    SmallVector<Value> thenYield(op.thenYield().getOperands());
-    SmallVector<Value> elseYield;
+    SmallVector<Value> thenTerminator(op.thenTerminator()->getOperands());
+    SmallVector<Value> elseTerminator;
     if (op.elseBlock())
-      llvm::append_range(elseYield, op.elseYield().getOperands());
+      llvm::append_range(elseTerminator, op.elseTerminator()->getOperands());
 
     // A list of indices for which we should upgrade the value yielded
     // in the else to a select.
@@ -2660,19 +3031,20 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     // only permit combining if the value yielded when the condition
     // is false in the outer scf.if is the same value yielded when the
     // inner scf.if condition is false.
-    // Note that the array access to elseYield will not go out of bounds
-    // since it must have the same length as thenYield, since they both
+    // Note that the array access to elseTerminator will not go out of bounds
+    // since it must have the same length as thenTerminator, since they both
     // come from the same scf.if.
-    for (const auto &tup : llvm::enumerate(thenYield)) {
+    for (const auto &tup : llvm::enumerate(thenTerminator)) {
       if (tup.value().getDefiningOp() == nestedIf) {
         auto nestedIdx = llvm::cast<OpResult>(tup.value()).getResultNumber();
-        if (nestedIf.elseYield().getOperand(nestedIdx) !=
-            elseYield[tup.index()]) {
+        if (nestedIf.elseTerminator()->getOperand(nestedIdx) !=
+            elseTerminator[tup.index()]) {
           return failure();
         }
         // If the correctness test passes, we will yield
         // corresponding value from the inner scf.if
-        thenYield[tup.index()] = nestedIf.thenYield().getOperand(nestedIdx);
+        thenTerminator[tup.index()] =
+            nestedIf.thenTerminator()->getOperand(nestedIdx);
         continue;
       }
 
@@ -2704,28 +3076,72 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     for (auto idx : elseYieldsToUpgradeToSelect)
       results[idx] =
           arith::SelectOp::create(rewriter, op.getLoc(), op.getCondition(),
-                                  thenYield[idx], elseYield[idx]);
+                                  thenTerminator[idx], elseTerminator[idx]);
 
     rewriter.mergeBlocks(nestedIf.thenBlock(), newIfBlock);
     rewriter.setInsertionPointToEnd(newIf.thenBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(newIf.thenYield(), thenYield);
-    if (!elseYield.empty()) {
+    rewriter.replaceOpWithNewOp<YieldOp>(newIf.thenTerminator(),
+                                         thenTerminator);
+    if (!elseTerminator.empty()) {
       rewriter.createBlock(&newIf.getElseRegion());
       rewriter.setInsertionPointToEnd(newIf.elseBlock());
-      YieldOp::create(rewriter, loc, elseYield);
+      YieldOp::create(rewriter, loc, elseTerminator);
     }
     rewriter.replaceOp(op, results);
     return success();
   }
 };
 
+/// Simplify if with breaking control flow in both branches.
+/// For example:
+///    scf.if %cmp {
+///       scf.break [%loop] %arg1
+///    } else {
+///       scf.continue [%loop]
+///    }
+///    print(...) // This is dead code
+///  becomes
+///    scf.if %cmp {
+///       scf.break [%loop] %arg1
+///    }
+///    scf.continue [%loop]
+struct SimplifyIfWithBreakingControlFlowInBothBranches
+    : public OpRewritePattern<IfOp> {
+  using OpRewritePattern<IfOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(IfOp op,
+                                PatternRewriter &rewriter) const override {
+    if (op.getElseRegion().empty() || isa<YieldOp>(op.thenTerminator()) ||
+        isa<YieldOp>(op.elseTerminator()))
+      return failure();
+
+    // Inline the else block after the current op and erase everything after.
+    Block *block = op.elseBlock();
+
+    Operation *terminator = block->getTerminator();
+    // Inline the else block after the current op
+    rewriter.inlineBlockBefore(block, op->getNextNode());
+
+    // Erase everything that comes after the inlined terminator (dead code).
+    eraseOpsAfter(rewriter, terminator);
+
+    // The "else" region is now empty, let's clone the if op and inline the then
+    // region.
+    auto newIfOp = rewriter.cloneWithoutRegions(op);
+    rewriter.inlineRegionBefore(op.getThenRegion(), newIfOp.getThenRegion(),
+                                newIfOp.getThenRegion().begin());
+    rewriter.eraseOp(op);
+
+    return success();
+  }
+};
 } // namespace
 
 void IfOp::getCanonicalizationPatterns(RewritePatternSet &results,
                                        MLIRContext *context) {
   results.add<CombineIfs, CombineNestedIfs, ConditionPropagation,
               ConvertTrivialIfToSelect, RemoveEmptyElseBranch,
-              ReplaceIfYieldWithConditionOrValue>(context);
+              ReplaceIfYieldWithConditionOrValue,
+              SimplifyIfWithBreakingControlFlowInBothBranches>(context);
   populateRegionBranchOpInterfaceCanonicalizationPatterns(
       results, IfOp::getOperationName());
   populateRegionBranchOpInterfaceInliningPattern(results,
@@ -2733,14 +3149,18 @@ void IfOp::getCanonicalizationPatterns(RewritePatternSet &results,
 }
 
 Block *IfOp::thenBlock() { return &getThenRegion().back(); }
-YieldOp IfOp::thenYield() { return cast<YieldOp>(&thenBlock()->back()); }
+YieldOp IfOp::thenYield() { return dyn_cast<YieldOp>(thenTerminator()); }
+
 Block *IfOp::elseBlock() {
   Region &r = getElseRegion();
   if (r.empty())
     return nullptr;
   return &r.back();
 }
-YieldOp IfOp::elseYield() { return cast<YieldOp>(&elseBlock()->back()); }
+YieldOp IfOp::elseYield() {
+  Block *block = elseBlock();
+  return block ? dyn_cast<YieldOp>(block->getTerminator()) : nullptr;
+}
 
 //===----------------------------------------------------------------------===//
 // ParallelOp
@@ -3184,6 +3604,10 @@ ReduceOp::getMutableSuccessorOperands(RegionSuccessor point) {
   return MutableOperandRange(getOperation(), /*start=*/0, /*length=*/0);
 }
 
+SmallVector<Operation *> ReduceOp::getPotentialTargets() {
+  return {getOperation()->getParentOp()};
+}
+
 //===----------------------------------------------------------------------===//
 // ReduceReturnOp
 //===----------------------------------------------------------------------===//
@@ -3467,8 +3891,8 @@ struct WhileMoveIfDown : public OpRewritePattern<scf::WhileOp> {
       auto it = llvm::find(ifOp->getResults(), arg);
       if (it != ifOp->getResults().end()) {
         size_t ifOpIdx = it.getIndex();
-        Value thenValue = ifOp.thenYield()->getOperand(ifOpIdx);
-        Value elseValue = ifOp.elseYield()->getOperand(ifOpIdx);
+        Value thenValue = ifOp.thenTerminator()->getOperand(ifOpIdx);
+        Value elseValue = ifOp.elseTerminator()->getOperand(ifOpIdx);
 
         rewriter.replaceAllUsesWith(ifOp->getResults()[ifOpIdx], elseValue);
         rewriter.replaceAllUsesWith(op.getAfterArguments()[idx], thenValue);
@@ -3516,7 +3940,7 @@ struct WhileMoveIfDown : public OpRewritePattern<scf::WhileOp> {
         });
 
     // Inline ifOp then region into new whileOp after region.
-    rewriter.eraseOp(ifOp.thenYield());
+    rewriter.eraseOp(ifOp.thenTerminator());
     rewriter.inlineBlockBefore(ifOp.thenBlock(), newWhileOp.getAfterBody(),
                                newWhileOp.getAfterBody()->begin());
     rewriter.eraseOp(ifOp);
diff --git a/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
index 496a7b036e65d..8a0b7f955267e 100644
--- a/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -179,8 +179,8 @@ struct IfOpInterface
                              std::optional<int64_t> dim,
                              ValueBoundsConstraintSet &cstr) {
     unsigned int resultNum = cast<OpResult>(value).getResultNumber();
-    Value thenValue = ifOp.thenYield().getResults()[resultNum];
-    Value elseValue = ifOp.elseYield().getResults()[resultNum];
+    Value thenValue = ifOp.thenTerminator()->getOperand(resultNum);
+    Value elseValue = ifOp.elseTerminator()->getOperand(resultNum);
 
     auto boundsBuilder = cstr.bound(value);
     if (dim)
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index f8302126a4a4e..456797b8fbec3 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -227,8 +227,8 @@ struct IfOpInterface
     auto ifOp = cast<scf::IfOp>(op);
     size_t resultNum = std::distance(op->getOpResults().begin(),
                                      llvm::find(op->getOpResults(), value));
-    OpOperand *thenOperand = &ifOp.thenYield()->getOpOperand(resultNum);
-    OpOperand *elseOperand = &ifOp.elseYield()->getOpOperand(resultNum);
+    OpOperand *thenOperand = &ifOp.thenTerminator()->getOpOperand(resultNum);
+    OpOperand *elseOperand = &ifOp.elseTerminator()->getOpOperand(resultNum);
     return {{thenOperand, BufferRelation::Equivalent, /*isDefinite=*/false},
             {elseOperand, BufferRelation::Equivalent, /*isDefinite=*/false}};
   }
diff --git a/mlir/lib/IR/Dominance.cpp b/mlir/lib/IR/Dominance.cpp
index 79fb41f2e6b30..f8820822dc01f 100644
--- a/mlir/lib/IR/Dominance.cpp
+++ b/mlir/lib/IR/Dominance.cpp
@@ -14,8 +14,12 @@
 #include "mlir/IR/Dominance.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/RegionKindInterface.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/Support/DebugLog.h"
 #include "llvm/Support/GenericDomTreeConstruction.h"
 
+#define DEBUG_TYPE "dominance"
+
 using namespace mlir;
 using namespace mlir::detail;
 
@@ -38,6 +42,7 @@ void DominanceInfoBase<IsPostDom>::invalidate() {
   for (auto entry : dominanceInfos)
     delete entry.second.getPointer();
   dominanceInfos.clear();
+  breakingControlFlowOpsCache.clear();
 }
 
 template <bool IsPostDom>
@@ -53,6 +58,7 @@ void DominanceInfoBase<IsPostDom>::invalidate(Region *region, bool recursive) {
     region->walk([&](Region *r) { invalidate(r); });
   else
     invalidate(region);
+  breakingControlFlowOpsCache.clear();
 }
 
 /// Return the dom tree and "hasSSADominance" bit for the given region.  The
@@ -255,6 +261,15 @@ static bool isBeforeInBlock(Block *block, Block::iterator a,
   return a->isBeforeInBlock(&*b);
 }
 
+template <bool IsPostDom>
+bool DominanceInfoBase<IsPostDom>::hasBreakingControlFlowOpsCached(
+    Operation *op) const {
+  auto [it, inserted] = breakingControlFlowOpsCache.try_emplace(op, false);
+  if (inserted)
+    it->second = hasBreakingControlFlowOps(op);
+  return it->second;
+}
+
 template <bool IsPostDom>
 bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
     Block *aBlock, Block::iterator aIt, Block *bBlock, Block::iterator bIt,
@@ -288,6 +303,23 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
       return true;
   }
 
+  auto hasEscapingBreakingControlFlowInRange =
+      [&](Block *block, Block::iterator begin, Block::iterator end) {
+        assert(begin == block->end() || begin->getBlock() == block);
+        assert(end == block->end() || end->getBlock() == block);
+        for (auto it = begin; it != end; ++it) {
+          Operation *op = &*it;
+          if ((op->hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+               !op->isRegistered()) &&
+              hasBreakingControlFlowOpsCached(op)) {
+            LDBG() << "Breaking control flow: "
+                   << OpWithFlags(op, OpPrintingFlags().skipRegions());
+            return true;
+          }
+        }
+        return false;
+      };
+
   // Ok, they are in the same region now.
   if (aBlock == bBlock) {
     // Dominance changes based on the region type. In a region with SSA
@@ -295,6 +327,26 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
     // regions kinds, uses and defs can come in any order inside a block.
     if (!hasSSADominance(aBlock))
       return true;
+
+    if (IsPostDom && aIt != aBlock->end() && bIt != bBlock->end() &&
+        mightHaveBreakingControlFlow(aBlock->getParent())) {
+      bool inRange = false;
+      for (Operation &op : *aBlock) {
+        if (&op == &*bIt)
+          inRange = true;
+        if (inRange) {
+          if ((op.hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+               !op.isRegistered()) &&
+              hasBreakingControlFlowOpsCached(&op)) {
+            LDBG() << "Breaking control flow: "
+                   << OpWithFlags(&op, OpPrintingFlags().skipRegions());
+            return false;
+          }
+        }
+        if (&op == &*aIt)
+          break;
+      }
+    }
     if constexpr (IsPostDom) {
       return isBeforeInBlock(aBlock, bIt, aIt);
     } else {
@@ -303,7 +355,47 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
   }
 
   // If the blocks are different, use DomTree to resolve the query.
-  return getDomTree(aRegion).properlyDominates(aBlock, bBlock);
+  bool properlyDominates =
+      getDomTree(aRegion).properlyDominates(aBlock, bBlock);
+  if (!IsPostDom || !properlyDominates ||
+      !mightHaveBreakingControlFlow(aRegion))
+    return properlyDominates;
+
+  // A nested breaking terminator can bypass the containing block's ordinary
+  // CFG terminator, so the block post-dominator tree is not enough for
+  // operation post-dominance. Look for escaping breaks on paths from bBlock to
+  // aBlock.
+  if (bIt != bBlock->end() &&
+      hasEscapingBreakingControlFlowInRange(bBlock, bIt, bBlock->end()))
+    return false;
+
+  SmallPtrSet<Block *, 16> visited;
+  SmallVector<Block *> worklist;
+  auto enqueueSuccessor = [&](Block *successor) {
+    if (!successor || successor == aBlock || successor->getParent() != aRegion)
+      return;
+    if (visited.insert(successor).second)
+      worklist.push_back(successor);
+  };
+  for (Block *successor : bBlock->getSuccessors())
+    enqueueSuccessor(successor);
+
+  while (!worklist.empty()) {
+    Block *block = worklist.pop_back_val();
+    if (hasEscapingBreakingControlFlowInRange(block, block->begin(),
+                                              block->end()))
+      return false;
+    for (Block *successor : block->getSuccessors())
+      enqueueSuccessor(successor);
+  }
+
+  if (aIt != aBlock->end()) {
+    Block::iterator endIt = std::next(aIt);
+    if (hasEscapingBreakingControlFlowInRange(aBlock, aBlock->begin(), endIt))
+      return false;
+  }
+
+  return true;
 }
 
 /// Return true if the specified block is reachable from the entry block of
diff --git a/mlir/lib/IR/RegionKindInterface.cpp b/mlir/lib/IR/RegionKindInterface.cpp
index 007f4cf92dbc7..59c741ea7e11e 100644
--- a/mlir/lib/IR/RegionKindInterface.cpp
+++ b/mlir/lib/IR/RegionKindInterface.cpp
@@ -12,6 +12,12 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/IR/RegionKindInterface.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Support/WalkResult.h"
+
+#include "llvm/Support/DebugLog.h"
+
+#define DEBUG_TYPE "region-kind-interface"
 
 using namespace mlir;
 
@@ -32,3 +38,165 @@ bool mlir::mayBeGraphRegion(Region &region) {
     return false;
   return !regionKindOp.hasSSADominance(region.getRegionNumber());
 }
+
+namespace {
+// Worklist entry for walking block terminators in nested regions.
+// Tracks the current position within a region and the nesting depth.
+struct NestedOpIterator {
+  NestedOpIterator(Region *region, int nestedLevel)
+      : region(region), nestedLevel(nestedLevel) {
+    regionIt = region->begin();
+    blockIt = regionIt->end();
+    if (regionIt != region->end())
+      blockIt = regionIt->begin();
+  }
+  // Advance the iterator to the next reachable operation.
+  void advance() {
+    assert(regionIt != region->end());
+    if (blockIt == regionIt->end()) {
+      ++regionIt;
+      if (regionIt != region->end())
+        blockIt = regionIt->begin();
+      return;
+    }
+    ++blockIt;
+    if (blockIt != regionIt->end()) {
+      LDBG() << "Advancing to next op: "
+             << OpWithFlags(&*blockIt, OpPrintingFlags().skipRegions());
+    }
+  }
+
+  // The region we're iterating over.
+  Region *region;
+  // The Block currently being iterated over.
+  Region::iterator regionIt;
+  // The Operation currently being iterated over.
+  Block::iterator blockIt;
+  // The nested level of the current region relative to the starting region.
+  int nestedLevel = 0;
+};
+} // namespace
+
+/// Walk all block terminators (last operation in each block) nested within
+/// `rootOp`. The callback receives the terminator and its 1-based nesting
+/// level relative to `rootOp`. Callers filter based on breaking-control-flow
+/// properties as needed.
+static void walk(Operation *rootOp,
+                 function_ref<WalkResult(Operation *, int)> callback) {
+  // Worklist of regions to visit to drive the traversal.
+  SmallVector<NestedOpIterator> worklist;
+
+  // Perform a traversal of the regions, visiting each
+  // reachable operation.
+  for (Region &region : rootOp->getRegions()) {
+    if (region.empty())
+      continue;
+    worklist.push_back({&region, 1});
+  }
+  while (!worklist.empty()) {
+    NestedOpIterator &it = worklist.back();
+    if (it.regionIt == it.region->end()) {
+      // We're done with this region.
+      worklist.pop_back();
+      continue;
+    }
+    if (it.blockIt == it.regionIt->end()) {
+      // We're done with this block.
+      it.advance();
+      continue;
+    }
+    Operation *op = &*it.blockIt;
+
+    // Only call the callback if we're at the end of the block.
+    if (std::next(it.blockIt) == it.regionIt->end() &&
+        callback(op, it.nestedLevel).wasInterrupted())
+      return;
+
+    // Advance before pushing nested regions to avoid reference invalidation.
+    int currentNestedLevel = it.nestedLevel;
+    it.advance();
+
+    // Recursively visit the nested regions.
+    for (Region &nestedRegion : op->getRegions()) {
+      if (nestedRegion.empty())
+        continue;
+      worklist.push_back({&nestedRegion, currentNestedLevel + 1});
+    }
+  }
+}
+
+NestedBreakingControlFlowInfo
+mlir::getNestedBreakingControlFlowInfo(Operation *op) {
+  NestedBreakingControlFlowInfo info;
+  walk(op, [&](Operation *visitedOp, int nestedLevel) {
+    bool directlyTargetsOp = false;
+    bool escapesThroughOp = false;
+    for (HasBreakingControlFlowOpInterface target :
+         findPotentialBreakTargets(visitedOp)) {
+      if (!target)
+        continue;
+      Operation *targetOp = target.getOperation();
+      if (targetOp == op)
+        directlyTargetsOp = true;
+      else if (targetOp->isProperAncestor(op))
+        escapesThroughOp = true;
+    }
+
+    if (directlyTargetsOp) {
+      info.predecessors.push_back(visitedOp);
+      if (nestedLevel > 1)
+        info.hasNestedPredecessors = true;
+    }
+    if (escapesThroughOp)
+      info.hasBreakingControlFlowOps = true;
+
+    return WalkResult::advance();
+  });
+  return info;
+}
+
+bool mlir::hasNestedPredecessors(Operation *op) {
+  return getNestedBreakingControlFlowInfo(op).hasNestedPredecessors;
+}
+
+bool mlir::hasBreakingControlFlowOps(Operation *op) {
+  return getNestedBreakingControlFlowInfo(op).hasBreakingControlFlowOps;
+}
+
+void mlir::detail::visitNestedBreakingControlFlowOpsImpl(
+    Operation *op,
+    function_ref<WalkResult(RegionExitTerminatorOpInterface, int nestedLevel)>
+        callback) {
+  ::walk(op, [&](Operation *visitedOp, int nestedLevel) {
+    for (HasBreakingControlFlowOpInterface target :
+         findPotentialBreakTargets(visitedOp)) {
+      if (target && (target.getOperation() == op ||
+                     target.getOperation()->isProperAncestor(op)))
+        return callback(cast<RegionExitTerminatorOpInterface>(visitedOp),
+                        nestedLevel);
+    }
+    return WalkResult::advance();
+  });
+}
+
+void mlir::collectAllNestedPredecessors(
+    Operation *op, SmallVector<Operation *> &predecessors) {
+  llvm::append_range(predecessors,
+                     getNestedBreakingControlFlowInfo(op).predecessors);
+}
+
+SmallVector<HasBreakingControlFlowOpInterface>
+mlir::findPotentialBreakTargets(Operation *terminator) {
+  auto breakingTerminator =
+      dyn_cast<RegionExitTerminatorOpInterface>(terminator);
+  if (!breakingTerminator)
+    return {};
+  SmallVector<HasBreakingControlFlowOpInterface> targets;
+  for (Operation *target : breakingTerminator.getPotentialTargets()) {
+    if (!target)
+      continue;
+    if (auto breakTarget = dyn_cast<HasBreakingControlFlowOpInterface>(target))
+      targets.push_back(breakTarget);
+  }
+  return targets;
+}
diff --git a/mlir/lib/IR/Verifier.cpp b/mlir/lib/IR/Verifier.cpp
index 11771e78d5f20..4209a36b1c560 100644
--- a/mlir/lib/IR/Verifier.cpp
+++ b/mlir/lib/IR/Verifier.cpp
@@ -30,6 +30,7 @@
 #include "mlir/IR/Dialect.h"
 #include "mlir/IR/Dominance.h"
 #include "mlir/IR/Operation.h"
+#include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/RegionKindInterface.h"
 #include "mlir/IR/Threading.h"
 #include "llvm/ADT/PointerIntPair.h"
diff --git a/mlir/lib/Interfaces/ControlFlowInterfaces.cpp b/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
index 475969a7c0d09..241e447a3e7f1 100644
--- a/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
+++ b/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
@@ -13,6 +13,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/RegionKindInterface.h"
 #include "mlir/Interfaces/ControlFlowInterfaces.h"
 #include "llvm/ADT/EquivalenceClasses.h"
 #include "llvm/Support/DebugLog.h"
@@ -76,14 +77,14 @@ detail::getBranchSuccessorArgument(const SuccessorOperands &operands,
 LogicalResult
 detail::verifyBranchSuccessorOperands(Operation *op, unsigned succNo,
                                       const SuccessorOperands &operands) {
-  LDBG() << "Verifying branch successor operands for successor #" << succNo
-         << " in operation " << op->getName();
+  LDBG(3) << "Verifying branch successor operands for successor #" << succNo
+          << " in operation " << op->getName();
 
   // Check the count.
   unsigned operandCount = operands.size();
   Block *destBB = op->getSuccessor(succNo);
-  LDBG() << "Branch has " << operandCount << " operands, target block has "
-         << destBB->getNumArguments() << " arguments";
+  LDBG(3) << "Branch has " << operandCount << " operands, target block has "
+          << destBB->getNumArguments() << " arguments";
 
   if (operandCount != destBB->getNumArguments())
     return op->emitError() << "branch has " << operandCount
@@ -92,22 +93,22 @@ detail::verifyBranchSuccessorOperands(Operation *op, unsigned succNo,
                            << destBB->getNumArguments();
 
   // Check the types.
-  LDBG() << "Checking type compatibility for "
-         << (operandCount - operands.getProducedOperandCount())
-         << " forwarded operands";
+  LDBG(3) << "Checking type compatibility for "
+          << (operandCount - operands.getProducedOperandCount())
+          << " forwarded operands";
   for (unsigned i = operands.getProducedOperandCount(); i != operandCount;
        ++i) {
     Type operandType = operands[i].getType();
     Type argType = destBB->getArgument(i).getType();
-    LDBG() << "Checking type compatibility: operand type " << operandType
-           << " vs argument type " << argType;
+    LDBG(3) << "Checking type compatibility: operand type " << operandType
+            << " vs argument type " << argType;
 
     if (!cast<BranchOpInterface>(op).areTypesCompatible(operandType, argType))
       return op->emitError() << "type mismatch for bb argument #" << i
                              << " of successor #" << succNo;
   }
 
-  LDBG() << "Branch successor operand verification successful";
+  LDBG(3) << "Branch successor operand verification successful";
   return success();
 }
 
@@ -168,11 +169,17 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
     SmallVector<RegionSuccessor> successors;
     regionInterface.getSuccessorRegions(branchPoint, successors);
     for (const RegionSuccessor &successor : successors) {
+      RegionBranchOpInterface successorOwner =
+          getRegionBranchSuccessorOwner(successor);
+      if (!successorOwner)
+        return regionInterface->emitOpError()
+               << "has region successor not owned by a "
+                  "RegionBranchOpInterface";
       // Helper function that print the region branch point and the region
       // successor.
       auto emitRegionEdgeError = [&]() {
         InFlightDiagnostic diag =
-            regionInterface->emitOpError("along control flow edge from ");
+            successorOwner->emitOpError("along control flow edge from ");
         if (branchPoint.isParent()) {
           diag << "parent";
           diag.attachNote(op->getLoc()) << "region branch point";
@@ -195,7 +202,7 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
       // Verify number of successor operands and successor inputs.
       OperandRange succOperands =
           regionInterface.getSuccessorOperands(branchPoint, successor);
-      ValueRange succInputs = regionInterface.getSuccessorInputs(successor);
+      ValueRange succInputs = successorOwner.getSuccessorInputs(successor);
       if (succOperands.size() != succInputs.size()) {
         return emitRegionEdgeError()
                << ": region branch point has " << succOperands.size()
@@ -210,7 +217,7 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
            llvm::enumerate(llvm::zip(succOperandTypes, succInputTypes))) {
         Type succOperandType = std::get<0>(typesIdx.value());
         Type succInputType = std::get<1>(typesIdx.value());
-        if (!regionInterface.areTypesCompatible(succOperandType, succInputType))
+        if (!successorOwner.areTypesCompatible(succOperandType, succInputType))
           return emitRegionEdgeError()
                  << ": successor operand type #" << typesIdx.index() << " "
                  << succOperandType << " should match successor input type #"
@@ -227,6 +234,17 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
 /// regions.
 using StopConditionFn = function_ref<bool(Region *, ArrayRef<bool> visited)>;
 
+/// Given a range of values, return a vector of attributes of the same size,
+/// where the i-th attribute is the constant value of the i-th value. If a
+/// value is not constant, the corresponding attribute is null.
+static SmallVector<Attribute> extractConstants(ValueRange values) {
+  return llvm::map_to_vector(values, [](Value value) {
+    Attribute attr;
+    matchPattern(value, m_Constant(&attr));
+    return attr;
+  });
+}
+
 /// Traverse the region graph starting at `begin`. The traversal is interrupted
 /// if `stopCondition` evaluates to "true" for a successor region. In that case,
 /// this function returns "true". Otherwise, if the traversal was not
@@ -246,7 +264,6 @@ static bool traverseRegionGraph(Region *begin,
   SmallVector<Region *> worklist;
   auto enqueueAllSuccessors = [&](Region *region) {
     LDBG() << "Enqueuing successors for region #" << region->getRegionNumber();
-    SmallVector<Attribute> operandAttributes(op->getNumOperands());
     for (Block &block : *region) {
       if (block.empty())
         continue;
@@ -255,8 +272,8 @@ static bool traverseRegionGraph(Region *begin,
       if (!terminator)
         continue;
       SmallVector<RegionSuccessor> successors;
-      operandAttributes.resize(terminator->getNumOperands());
-      terminator.getSuccessorRegions(operandAttributes, successors);
+      terminator.getSuccessorRegions(
+          extractConstants(terminator->getOperands()), successors);
       LDBG() << "Found " << successors.size()
              << " successors from terminator in block";
       for (RegionSuccessor successor : successors) {
@@ -462,6 +479,14 @@ RegionBranchOpInterface::getNonSuccessorInputs(RegionSuccessor successor) {
   return results;
 }
 
+RegionBranchOpInterface
+mlir::getRegionBranchSuccessorOwner(RegionSuccessor successor) {
+  if (Operation *successorOp = successor.getSuccessorOp())
+    return dyn_cast<RegionBranchOpInterface>(successorOp);
+  return dyn_cast<RegionBranchOpInterface>(
+      successor.getSuccessor()->getParentOp());
+}
+
 static MutableArrayRef<OpOperand> operandsToOpOperands(OperandRange &operands) {
   return MutableArrayRef<OpOperand>(operands.getBase(), operands.size());
 }
@@ -473,11 +498,14 @@ getSuccessorOperandInputMapping(RegionBranchOpInterface branchOp,
   SmallVector<RegionSuccessor> successors;
   branchOp.getSuccessorRegions(src, successors);
   for (RegionSuccessor dst : successors) {
+    RegionBranchOpInterface successorOwner = getRegionBranchSuccessorOwner(dst);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
     OperandRange operands = branchOp.getSuccessorOperands(src, dst);
-    assert(operands.size() == branchOp.getSuccessorInputs(dst).size() &&
+    ValueRange inputs = successorOwner.getSuccessorInputs(dst);
+    assert(operands.size() == inputs.size() &&
            "expected the same number of operands and inputs");
-    for (const auto &[operand, input] : llvm::zip_equal(
-             operandsToOpOperands(operands), branchOp.getSuccessorInputs(dst)))
+    for (const auto &[operand, input] :
+         llvm::zip_equal(operandsToOpOperands(operands), inputs))
       mapping[&operand].push_back(input);
   }
 }
@@ -524,6 +552,62 @@ RegionBranchOpInterface::getAllRegionBranchPoints() {
         branchPoints.push_back(RegionBranchPoint(terminator));
     }
   }
+
+  Operation *op = getOperation();
+  if (!op->hasTrait<OpTrait::PropagateControlFlowBreak>() &&
+      !isa<HasBreakingControlFlowOpInterface>(op))
+    return branchPoints;
+
+  // The loop above only records terminators that are immediate branch points of
+  // this op: terminators ending blocks directly contained in one of this op's
+  // regions. Breaking control flow adds another class of branch points. A
+  // nested RegionBranchOpInterface that defines PropagateControlFlowBreak can
+  // contain a breaking terminator whose addressed receiver is this op or one of
+  // this op's ancestors. Even though the terminator is not directly contained
+  // in this op's region, it can still create a control-flow edge that leaves or
+  // propagates through this op.
+  //
+  // For example, consider:
+  //
+  //   scf.loop token(%outer) {
+  //     scf.loop token(%inner) {
+  //       scf.if %cond {
+  //         scf.break [%outer]
+  //       }
+  //       scf.continue [%inner]
+  //     }
+  //     scf.continue [%outer]
+  //   }
+  //
+  // When enumerating branch points for the inner loop, its direct body
+  // terminator is only `scf.continue [%inner]`. The `scf.break [%outer]` is
+  // hidden behind the immediately nested `scf.if`, but the inner loop must
+  // still expose a possible edge for that break: the break request propagates
+  // through `scf.if`, then through the inner loop, and is ultimately handled by
+  // the outer loop. Without adding the nested break as a branch point of the
+  // inner loop, generic RegionBranchOpInterface verification and data-flow
+  // consumers only see the continue edge and miss the propagated exit edge.
+  //
+  // This is intentionally broader than collectAllNestedPredecessors(op). That
+  // helper collects only terminators that directly target `op`, which is enough
+  // for a receiver to find incoming breaks but insufficient for an intermediate
+  // PropagateControlFlowBreak op: an escaping terminator targets an ancestor,
+  // not the intermediate op it propagates through.
+  // visitNestedBreakingControlFlowOps reports both nested terminators that
+  // target `op` and those that target an ancestor of `op`, which is exactly the
+  // set that may affect this op's RegionBranchOpInterface edges.
+  visitNestedBreakingControlFlowOps(
+      op, [&](Operation *nestedTerminator, int nestedLevel) {
+        // Immediate region terminators are already covered above. Deeper
+        // terminators may add a propagated control-flow edge through this op.
+        if (nestedLevel <= 1)
+          return;
+        auto terminator =
+            dyn_cast<RegionBranchTerminatorOpInterface>(nestedTerminator);
+        if (!terminator)
+          return;
+        branchPoints.push_back(RegionBranchPoint(terminator));
+      });
   return branchPoints;
 }
 
@@ -628,6 +712,12 @@ static bool isDefinedBefore(Operation *regionBranchOp, Value a, Value b) {
   return true;
 }
 
+static bool isSuccessorInputOwnedBy(Operation *regionBranchOp, Value value) {
+  if (Operation *definingOp = value.getDefiningOp())
+    return definingOp == regionBranchOp;
+  return value.getParentRegion()->getParentOp() == regionBranchOp;
+}
+
 /// Compute all non-successor-input values that a successor input could have
 /// based on the given successor input to successor operand mapping.
 ///
@@ -733,6 +823,8 @@ struct MakeRegionBranchOpSuccessorInputsDead : public RewritePattern {
     // Try to replace the uses of each successor input one-by-one.
     bool changed = false;
     for (Value value : inputToOperands.keys()) {
+      if (!isSuccessorInputOwnedBy(regionBranchOp, value))
+        continue;
       // Nothing to do for successor inputs that are already dead.
       if (value.use_empty())
         continue;
@@ -1070,17 +1162,6 @@ struct RemoveDuplicateSuccessorInputUses : public RewritePattern {
   }
 };
 
-/// Given a range of values, return a vector of attributes of the same size,
-/// where the i-th attribute is the constant value of the i-th value. If a
-/// value is not constant, the corresponding attribute is null.
-static SmallVector<Attribute> extractConstants(ValueRange values) {
-  return llvm::map_to_vector(values, [](Value value) {
-    Attribute attr;
-    matchPattern(value, m_Constant(&attr));
-    return attr;
-  });
-}
-
 /// Return all successor regions when branching from the given region branch
 /// point. This helper functions extracts all constant operand values and
 /// passes them to the `RegionBranchOpInterface`.
diff --git a/mlir/lib/Transforms/Utils/CMakeLists.txt b/mlir/lib/Transforms/Utils/CMakeLists.txt
index 3ecf3859a7f4c..cb84fb26b2408 100644
--- a/mlir/lib/Transforms/Utils/CMakeLists.txt
+++ b/mlir/lib/Transforms/Utils/CMakeLists.txt
@@ -15,6 +15,9 @@ add_mlir_library(MLIRTransformUtils
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Transforms
 
+  DEPENDS
+  MLIRRegionKindInterfaceIncGen
+
   LINK_LIBS PUBLIC
   MLIRAnalysis
   MLIRCallInterfaces
diff --git a/mlir/lib/Transforms/Utils/InliningUtils.cpp b/mlir/lib/Transforms/Utils/InliningUtils.cpp
index 73107cfc36ea9..4bfc20e7ed885 100644
--- a/mlir/lib/Transforms/Utils/InliningUtils.cpp
+++ b/mlir/lib/Transforms/Utils/InliningUtils.cpp
@@ -286,6 +286,12 @@ static LogicalResult inlineRegionImpl(
                    [&](BlockArgument arg) { return !mapper.contains(arg); }))
     return failure();
 
+  // Block inlining only if breaks escape the region (propagate through the
+  // parent op toward an ancestor). Self-contained breaks that target ops
+  // within the region are fine.
+  if (hasBreakingControlFlowOps(src->getParentOp()))
+    return failure();
+
   // Check that the operations within the source region are valid to inline.
   Region *insertRegion = inlineBlock->getParent();
   if (!interface.isLegalToInline(insertRegion, src, shouldCloneInlinedRegion,
diff --git a/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir b/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
new file mode 100644
index 0000000000000..49338ac634462
--- /dev/null
+++ b/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
@@ -0,0 +1,100 @@
+// RUN: mlir-opt -test-dead-code-analysis --split-input-file 2>&1 %s | FileCheck %s
+
+// Tests verifying that DeadCodeAnalysis correctly propagates breaking control
+// flow through PropagateControlFlowBreak ops (e.g. scf.if). The fix is in
+// visitRegionTerminator: when the immediate parent RegionBranchOpInterface
+// returns a propagating successor sentinel for a propagating break, we resolve
+// the actual HasBreakingControlFlowOp ancestor and re-dispatch through it, so
+// the correct predecessor edge is established at the loop's exit point.
+
+// -----
+
+// A loop whose only exit is scf.break propagated through scf.if.
+// The loop's exit point (op_preds) must show the break as a predecessor.
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK:   scf.continue
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_break_through_if(%cond: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond {
+      %c42 = arith.constant 42 : i32
+      scf.break [%loop] %c42 : i32
+    }
+    scf.continue [%loop]
+  } {tag = "loop"}
+  return %result : i32
+}
+
+// -----
+
+// For comparison: a loop with a direct scf.break.
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_direct_break(%cond: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond {
+    }
+    %c42 = arith.constant 42 : i32
+    scf.break [%loop] %c42 : i32
+  } {tag = "loop"}
+  return %result : i32
+}
+
+// -----
+
+// A loop whose only exit is scf.break propagated through two nested scf.if
+// ops. The loop's op_preds must show the break. The intermediate if ops do not
+// appear in op_preds for the break — the break bypasses them without yielding
+// values to them, which is correct.
+
+// CHECK-LABEL: inner_if:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "inner_if"}
+// CHECK:  region #1
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "inner_if"}
+
+// CHECK-LABEL: outer_if:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "outer_if"}
+// CHECK:  region #1
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "outer_if"}
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK:   scf.continue
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_break_through_nested_ifs(%cond1: i1, %cond2: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond1 {
+      scf.if %cond2 {
+        %c99 = arith.constant 99 : i32
+        scf.break [%loop] %c99 : i32
+      } {tag = "inner_if"}
+      scf.continue [%loop]
+    } {tag = "outer_if"}
+    scf.continue [%loop]
+  } {tag = "loop"}
+  return %result : i32
+}
diff --git a/mlir/test/Analysis/test-dominance.mlir b/mlir/test/Analysis/test-dominance.mlir
index a926a8271200a..5627c9994b0e8 100644
--- a/mlir/test/Analysis/test-dominance.mlir
+++ b/mlir/test/Analysis/test-dominance.mlir
@@ -680,3 +680,63 @@ func.func @func_loop_nested_region(
 // CHECK:   ^{{.*}}
 // CHECK:   }
 // CHECK: }
+
+
+// -----
+
+// CHECK-LABEL: Testing : func_loop_early_exit
+func.func @func_loop_early_exit(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      scf.if %cond {
+        scf.break [%outer] %arg0 : index
+      }
+      "test.foo"() : () -> ()
+      scf.break [%inner] {test.print_dominance = true}
+    }
+  }
+  return %0 : index
+}
+
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.break
+// CHECK-SAME: = 0
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.if
+// CHECK-SAME: = 0
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} "test.foo"
+// CHECK-SAME: = 1
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.break
+// CHECK-SAME: = 1
+
+// -----
+
+// CHECK-LABEL: Testing : func_loop_early_exit_cross_block
+func.func @func_loop_early_exit_cross_block(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      test.propagate_control_flow_break {
+      ^bb0:
+        cf.cond_br %cond, ^bb1, ^bb2
+      ^bb1:
+        scf.if %cond {
+          scf.break [%outer] %arg0 : index
+        }
+        cf.br ^bb2
+      ^bb2:
+        "test.foo"() {test.print_dominance = true} : () -> ()
+        "test.return"() : () -> ()
+      }
+      scf.break [%inner]
+    }
+  }
+  return %0 : index
+}
+
+// CHECK: --- PostDominanceInfo ---
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} cf.cond_br
+// CHECK-SAME: = 0
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} scf.if
+// CHECK-SAME: = 0
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} cf.br
+// CHECK-SAME: = 1
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} "test.foo"
+// CHECK-SAME: = 1
diff --git a/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir b/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
new file mode 100644
index 0000000000000..c5f5a05e47c46
--- /dev/null
+++ b/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
@@ -0,0 +1,135 @@
+// RUN: mlir-opt -convert-scf-to-cf -split-input-file %s | FileCheck %s
+
+
+func.func @loop_break(%cond : i1) {
+  // CHECK: test.op1
+   "test.op1"() : () -> ()
+  // CHECK-NEXT: cf.br [[LOOP1_ENTRY:.*]]
+  // CHECK-NEXT: [[LOOP1_ENTRY]]
+   scf.loop token(%loop) {
+    // CHECK-NEXT: test.op2
+     "test.op2"() : () -> ()
+    // CHECK-NEXT: cf.cond_br %arg0, [[IF_ENTRY:.*]], [[IF_CONTINUE:.*]]
+    // CHECK-NEXT: [[IF_ENTRY]]
+     scf.if %cond {
+       "test.op3"() : () -> ()
+       scf.break [%loop] loc("break1")
+     }
+     "test.op3"() : () -> ()
+   } loc("loop1")
+   "test.op4"() : () -> ()
+   return
+}
+
+// -----
+
+// Bug regression test: IfLowering was using thenTerminator on line 461 instead
+// of elseTerminator, causing the else block's scf.yield to not be replaced with
+// a cf.branch when the then-branch has a scf.break.
+// CHECK-LABEL: func @if_break_then_yield_else
+func.func @if_break_then_yield_else(%cond : i1) {
+  // CHECK: cf.br ^[[LOOP:.*]]
+  scf.loop token(%loop) {
+    // CHECK: ^[[LOOP]]:
+    // CHECK: cf.cond_br %arg0, ^[[THEN:.*]], ^[[ELSE:.*]]
+    scf.if %cond {
+      // CHECK: ^[[THEN]]:
+      // CHECK-NEXT: cf.br ^[[EXIT:.*]]
+      scf.break [%loop]
+    } else {
+      // CHECK: ^[[ELSE]]:
+      // CHECK-NEXT: cf.br ^[[AFTER_IF:.*]]
+    }
+    // CHECK: ^[[AFTER_IF]]:
+    // CHECK-NEXT: cf.br ^[[LOOP]]
+    scf.continue [%loop]
+  }
+  // CHECK: ^[[EXIT]]:
+  // CHECK-NEXT: return
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @nested_loops_and_ifs(
+// CHECK-SAME: %[[COND1:.*]]: i1,
+// CHECK-SAME: %[[COND2:.*]]: i1
+func.func @nested_loops_and_ifs(%cond1 : i1, %cond2 : i1) {
+  // CHECK: test.op1
+   "test.op1"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[OUTER_LOOP_ENTRY:.*]]
+   scf.loop token(%outer) {
+  // CHECK-NEXT: ^[[OUTER_LOOP_ENTRY]]:
+  // CHECK-NEXT: cf.cond_br %[[COND1]], ^[[IF1_THEN_BLOCK:.*]], ^[[IF1_EXIT:.*]]
+     scf.if %cond1 {
+  // CHECK-NEXT: ^[[IF1_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op2
+       "test.op2"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY:.*]]
+       scf.loop token(%inner) {
+  // CHECK-NEXT: ^[[INNER_LOOP_ENTRY]]:
+  // CHECK-NEXT: test.op3
+         "test.op3"() : () -> ()
+  // CHECK-NEXT: cf.cond_br %[[COND1]], ^[[IF2_THEN_BLOCK:.*]], ^[[IF2_EXIT:.*]]
+         scf.if %cond1 {
+  // CHECK-NEXT: ^[[IF2_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op4
+           "test.op4"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY]]
+           scf.continue [%inner] loc("continue1")
+         }
+  // CHECK-NEXT: ^[[IF2_EXIT]]:
+  // CHECK-NEXT: test.op5
+         "test.op5"() : () -> ()
+  // CHECK-NEXT: cf.cond_br %[[COND2]], ^[[IF3_THEN_BLOCK:.*]], ^[[IF3_EXIT:.*]]
+         scf.if %cond2 {
+  // CHECK-NEXT: ^[[IF3_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op6
+           "test.op6"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[FUNC_EXIT:.*]]
+           scf.break [%outer] loc("break2")
+         }
+  // CHECK-NEXT: ^[[IF3_EXIT]]:
+  // CHECK-NEXT: test.op7
+         "test.op7"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY]]
+         scf.continue [%inner] loc("continue2")
+       } loc("loop3")
+  // CHECK-NEXT: ^[[AFTER_INNER_LOOP:.*]]:
+  // CHECK-NEXT: test.op8
+       "test.op8"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[IF1_EXIT]]
+     } loc("if1")
+  // CHECK-NEXT: ^[[IF1_EXIT]]:
+  // CHECK-NEXT: cf.br ^[[OUTER_LOOP_ENTRY]]
+     scf.continue [%outer] loc("continue3")
+   } loc("loop2")
+  // CHECK-NEXT: ^[[FUNC_EXIT]]:
+  // CHECK-NEXT: test.op9
+   "test.op9"() : () -> ()
+  // CHECK-NEXT: return
+   return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_with_iter_args
+// CHECK-SAME: %[[INIT:.*]]: i32, %[[COND:.*]]: i1
+func.func @loop_with_iter_args(%init: i32, %cond: i1) -> i32 {
+  // CHECK: cf.br ^[[LOOP:.*]](%[[INIT]] : i32)
+  %result = scf.loop token(%loop) iter_args(%arg = %init) : i32 -> i32 {
+    // CHECK: ^[[LOOP]](%[[ARG:.*]]: i32):
+    // CHECK: cf.cond_br %[[COND]], ^[[THEN:.*]], ^[[ELSE:.*]]
+    scf.if %cond {
+      // CHECK: ^[[THEN]]:
+      // CHECK-NEXT: cf.br ^[[EXIT:.*]](%[[ARG]] : i32)
+      scf.break [%loop] %arg : i32
+    }
+    // CHECK: ^[[ELSE]]:
+    // CHECK-NEXT: cf.br ^[[LOOP]](%[[ARG]] : i32)
+    scf.continue [%loop] %arg : i32
+  }
+  // CHECK: ^[[EXIT]](%[[RES:.*]]: i32):
+  // CHECK-NEXT: return %[[RES]]
+  return %result : i32
+}
diff --git a/mlir/test/Dialect/SCF/loop_canonicalize.mlir b/mlir/test/Dialect/SCF/loop_canonicalize.mlir
new file mode 100644
index 0000000000000..90e2068290082
--- /dev/null
+++ b/mlir/test/Dialect/SCF/loop_canonicalize.mlir
@@ -0,0 +1,300 @@
+// RUN: mlir-opt %s -pass-pipeline='builtin.module(func.func(canonicalize{test-convergence}))' -split-input-file | FileCheck %s
+
+// CHECK-LABEL: func @fold_single_iteration_loop1
+func.func @fold_single_iteration_loop1(%arg0 : index) -> index {
+  // CHECK-NOT: loop
+  %0 = scf.loop token(%loop) -> index {
+    scf.break [%loop] %arg0 : index
+  }
+  return %0 : index
+}
+
+// -----
+
+// CHECK-LABEL: func @fold_single_iteration_loop_with_propagating_control_flow
+func.func @fold_single_iteration_loop_with_propagating_control_flow(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      scf.if %cond {
+        scf.break [%outer] %arg0 : index
+      }
+      scf.break [%inner]
+    }
+  }
+  return %0 : index
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_not_combine_ifs
+func.func @loop_not_combine_ifs(%arg0 : i1, %arg2: i64) {
+  // Verify that we don't combine ifs when terminator mismatches
+  scf.loop token(%loop) {
+    // CHECK: scf.if
+    %res = scf.if %arg0 -> i32 {
+      %v = "test.firstCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.firstCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+    // CHECK: scf.if
+    %res2 = scf.if %arg0 -> i32 {
+      %v = "test.secondCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.secondCodeFalse"() : () -> i32
+      scf.continue [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_combine_ifs
+func.func @loop_combine_ifs(%arg0 : i1, %arg2: i64) {
+  // CombineIfs is yield-only, so breaking terminators remain separate.
+  scf.loop token(%loop) {
+    // CHECK: scf.if
+    %res = scf.if %arg0 -> i32 {
+      %v = "test.firstCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.firstCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+    %res2 = scf.if %arg0 -> i32 {
+      %v = "test.secondCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.secondCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow1
+func.func @do_not_merge_nested_if_with_breaking_control_flow1(%arg0: i1, %arg1: i1) {
+// The outer if then terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.yield
+      }
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow2
+func.func @do_not_merge_nested_if_with_breaking_control_flow2(%arg0: i1, %arg1: i1) {
+// The outer if else block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: else
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.yield
+      }
+      scf.yield
+    } else {
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow3
+func.func @do_not_merge_nested_if_with_breaking_control_flow3(%arg0: i1, %arg1: i1) {
+// The nested if then block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: test.op
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow4
+func.func @do_not_merge_nested_if_with_breaking_control_flow4(%arg0: i1, %arg1: i1) {
+// The nested if else block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: else
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+      } else {
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @do_not_convert_if_to_select1
+func.func @do_not_convert_if_to_select1(%cond: i1, %arg0 : index, %arg1 : index) -> index {
+  %loop_res  = scf.loop token(%loop) -> index {
+    // Inner then terminator is not a yield, blocking transform to select.
+    // CHECK: scf.if
+    %0 = scf.if %cond -> index {
+      scf.break [%loop] %arg0 : index
+    } else {
+      scf.yield %arg1 : index
+    }
+    scf.break [%loop] %0 : index
+  }
+  return %loop_res : index
+}
+
+// -----
+
+// CHECK-LABEL: func @do_not_convert_if_to_select2
+func.func @do_not_convert_if_to_select2(%cond: i1, %arg0 : index, %arg1 : index) -> index {
+  %loop_res  = scf.loop token(%loop) -> index {
+    // Inner then terminator is not a yield, blocking transform to select.
+    // CHECK: scf.if
+    %0 = scf.if %cond -> index {
+      scf.yield %arg0 : index
+    } else {
+      scf.break [%loop] %arg1 : index
+    }
+    scf.break [%loop] %0 : index
+  }
+  return %loop_res : index
+}
+
+
+// -----
+
+// Verify that removing the unused results of an if with nested breaking control flow
+// operation works.
+// CHECK-LABEL: func @remove_unused_if_results1
+func.func @remove_unused_if_results1(%cond : i1, %arg0 : index) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    // CHECK: %[[FOO:.*]]:3 = "test.foo"
+    %foo:3 = "test.foo" () : () -> (i32, i64, index)
+    // CHECK-NOT: %[[RES:.*]] = scf.if
+    // CHECK: scf.if
+    %res:3 = scf.if %cond -> (i32, i64, index) {
+      // CHECK: scf.yield
+      scf.yield %foo#0, %foo#1, %foo#2 : i32, i64, index
+    } else {
+      // CHECK: scf.break {{.*}} %[[FOO]]#2 : index
+      scf.break [%loop] %foo#2 : index
+    }
+      // CHECK: "test.op"(%[[FOO]]#1)
+    "test.op"(%res#1) : (i64) -> ()
+  }
+  return %0 : index
+}
+
+// -----
+
+// Verify that removing the unused results of an if with nested breaking control flow
+// operation works.
+// CHECK-LABEL: func @simplify_if_with_breaking_controlflow_in_both_branches
+func.func @simplify_if_with_breaking_controlflow_in_both_branches(%cond : i1, %cond2 : i1, %arg0 : index) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    // CHECK: %[[FOO:.*]] = "test.foo"
+    %foo = "test.foo" () : () -> (index)
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.break {{.*}} %[[FOO]] : index
+      scf.break [%loop] %foo : index
+      // CHECK-NOT: else
+    } else {
+      // CHECK: %[[BAR:.*]] = "test.bar"
+      %bar = "test.bar" () : () -> (index)
+      // CHECK: scf.if
+      scf.if %cond2 {
+        // verify that this is correctly updated when inlining the parent region.
+        // CHECK: scf.break {{.*}} %[[BAR]] : index
+        scf.break [%loop] %bar : index
+      }
+      scf.continue [%loop]
+    }
+    "test.op"() : () -> ()
+  }
+  return %0 : index
+}
+
+// -----
+
+// Verify simplification when both branches have continue.
+// CHECK-LABEL: func @simplify_if_with_continue_in_both_branches
+func.func @simplify_if_with_continue_in_both_branches(%cond : i1, %init : i32) {
+  // CHECK: scf.loop
+  scf.loop token(%loop) iter_args(%arg = %init) : i32 {
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.continue {{.*}} %{{.*}} : i32
+      scf.continue [%loop] %arg : i32
+    } else {
+      scf.continue [%loop] %arg : i32
+    }
+    // The dead code after the if should be removed.
+    // CHECK-NOT: test.op
+    "test.op"() : () -> ()
+    scf.continue [%loop] %arg : i32
+  }
+  return
+}
+
+// -----
+
+// Verify simplification when both branches have break.
+// CHECK-LABEL: func @simplify_if_with_break_in_both_branches
+func.func @simplify_if_with_break_in_both_branches(%cond : i1) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.break
+      scf.break [%loop] %c0 : index
+    } else {
+      scf.break [%loop] %c1 : index
+    }
+    // Dead code after the if.
+    // CHECK-NOT: test.op
+    "test.op"() : () -> ()
+    scf.continue [%loop]
+  }
+  return %0 : index
+}
diff --git a/mlir/test/IR/early-exit-invalid.mlir b/mlir/test/IR/early-exit-invalid.mlir
new file mode 100644
index 0000000000000..509b39629ebc7
--- /dev/null
+++ b/mlir/test/IR/early-exit-invalid.mlir
@@ -0,0 +1,142 @@
+
+// RUN: mlir-opt %s --split-input-file --verify-diagnostics
+
+func.func @loop_result_mismatch(%value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ %result = scf.loop token(%loop) -> i32 {
+   scf.break [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+// -----
+
+// A PropagateControlFlowBreak op that is immediately nested in a
+// RegionBranchOpInterface can contain a region terminator that targets an
+// ancestor of that RegionBranchOpInterface. The intermediate RegionBranchOp
+// must expose this as a possible propagated edge so the addressed receiver can
+// verify the break operands against its results.
+func.func @outer_loop_result_mismatch_through_if_in_inner_loop(%cond : i1,
+                                                              %value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ %result = scf.loop token(%outer) -> i32 {
+   scf.loop token(%inner) {
+     scf.if %cond {
+       scf.break [%outer] %value : f32 // expected-note {{region branch point}}
+     }
+     scf.continue [%inner]
+   }
+   scf.continue [%outer]
+ }
+ return
+}
+
+// -----
+
+func.func @loop_result_number_mismatch(%value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: region branch point has 1 operands, but region successor needs 2 inputs}}
+ %result:2 = scf.loop token(%loop) -> f32, f32 {
+   scf.break [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+// -----
+
+func.func @loop_continue_mismatch(%init : i32, %value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.continue to Region #0: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ scf.loop token(%loop) iter_args(%next = %init) : i32 {
+   scf.continue [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+
+// -----
+
+func.func @loop_iterargs_mismatch(%init : i32, %value : f32) {
+ // expected-error @+2 {{'scf.loop' op along control flow edge from parent to Region #0: successor operand type #0 'i32' should match successor input type #0 'f32'}}
+ // expected-note @+1 {{region branch point}}
+ "scf.loop"(%init) ({
+    ^body(%token : token, %next : f32):
+   scf.continue [%token] %init : i32
+ })  : (i32) -> ()
+ return
+}
+
+// -----
+
+func.func @loop_iterargs_mismatch(%init : i32, %value : f32) {
+ // expected-error @+2 {{'scf.loop' op along control flow edge from parent to Region #0: region branch point has 1 operands, but region successor needs 2 inputs}}
+ // expected-note @+1 {{region branch point}}
+ "scf.loop"(%init) ({
+    ^body(%token : token, %next : i32, %next2 : f32):
+   scf.continue [%token] %init : i32
+ })  : (i32) -> ()
+ return
+}
+
+// -----
+
+// scf.for lacks PropagateControlFlowBreak, so it cannot be an intermediate
+// parent for a break targeting an enclosing loop token.
+func.func @break_through_for_missing_trait(%lb: index, %ub: index, %step: index, %cond: i1) {
+  scf.loop token(%loop) {
+    scf.for %i = %lb to %ub step %step {
+      scf.if %cond {
+        // expected-error @+1 {{target token crosses an op that does not have the PropagateControlFlowBreak trait}}
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// scf.while lacks PropagateControlFlowBreak, so break through it is rejected.
+// The scf.while verifier catches this first as the after region must terminate
+// with scf.yield.
+func.func @break_through_while(%cond: i1) {
+  %init = arith.constant true
+  scf.loop token(%loop) {
+    // expected-error @+1 {{'scf.while' op expects the 'after' region to terminate with 'scf.yield'}}
+    scf.while(%arg = %init) : (i1) -> i1 {
+      scf.condition(%arg) %arg : i1
+    } do {
+    ^bb0(%arg2: i1):
+      // expected-note @+1 {{terminator here}}
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// Verify that a break without a token target is rejected.
+func.func @break_without_token() {
+  scf.loop token(%loop) {
+    // expected-error @+1 {{'scf.break' op expected 1 or more operands}}
+    "scf.break"() : () -> ()
+  }
+  return
+}
+
+// -----
+
+// Verify that a loop whose first region argument is not a control token is
+// rejected. The break targets the outer loop so the inner loop does not need a
+// self-targeting terminator, isolating the control-token check.
+func.func @loop_non_token_control(%cond: i1) {
+  "scf.loop"() ({
+  ^bb0(%outer: token):
+    // expected-error @+1 {{'scf.loop' op first region argument must be a control token}}
+    "scf.loop"() ({
+    ^bb1(%bad: i32):
+      "scf.break"(%outer) : (token) -> ()
+    }) : () -> ()
+    "scf.continue"(%outer) : (token) -> ()
+  }) : () -> ()
+  return
+}
diff --git a/mlir/test/IR/early-exit.mlir b/mlir/test/IR/early-exit.mlir
new file mode 100644
index 0000000000000..82e7f92070e5b
--- /dev/null
+++ b/mlir/test/IR/early-exit.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt --print-region-branch-op-interface %s --split-input-file | FileCheck %s
+// RUN: mlir-opt %s --mlir-print-debuginfo --mlir-print-op-generic --split-input-file  | mlir-opt --print-region-branch-op-interface --split-input-file  | FileCheck %s
+
+
+func.func @loop_break(%cond : i1) {
+  // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop1")
+  // CHECK:  - Successor is region #0
+  // CHECK:  - Found 2 predecessor(s)
+  // CHECK:    - Predecessor is scf.break {{.*}} loc("break1")
+  // CHECK:    - Predecessor is scf.continue
+   scf.loop token(%loop) {
+     scf.if %cond {
+       scf.break [%loop] loc("break1")
+     }
+   } loc("loop1")
+   return
+}
+
+// -----
+
+func.func @loop_continue(%cond1 : i1, %cond2 : i1) {
+  // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop2")
+  // CHECK:  - Successor is region #0
+  // CHECK:  - Found 2 predecessor(s)
+  // CHECK:    - Predecessor is scf.break {{.*}} loc("break2")
+  // CHECK:    - Predecessor is scf.continue
+   scf.loop token(%outer) {
+    // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop3")
+    // CHECK:  - Successor is region #0
+    // CHECK:  - Found 2 predecessor(s)
+    // CHECK:    - Predecessor is scf.continue {{.*}} loc("continue1")
+    // CHECK:    - Predecessor is scf.continue
+     scf.loop token(%inner) {
+       scf.if %cond1 {
+         scf.continue [%inner] loc("continue1")
+       }
+       scf.if %cond2 {
+         scf.break [%outer] loc("break2")
+       }
+     } loc("loop3")
+   } loc("loop2")
+   return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_with_results(
+func.func @loop_with_results(%value : f32) -> f32 {
+ %result = scf.loop token(%loop) -> f32 {
+   scf.break [%loop] %value : f32
+ }
+ return %result : f32
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_continue_iterargs(
+func.func @loop_continue_iterargs(%init : i32) {
+ scf.loop token(%loop) iter_args(%next = %init) : i32 {
+   scf.continue [%loop] %next : i32
+ }
+ return
+}
+
+// -----
+
+// A single-operand continue that targets an *outer* loop must be printed
+// explicitly. If the printer elided it as the trivial implicit terminator, the
+// parser would rebuild it as a continue of the inner loop, silently changing
+// the target.
+// CHECK-LABEL: func @continue_outer_from_inner
+func.func @continue_outer_from_inner() {
+  // CHECK: scf.loop token(%[[OUTER:.*]]) {
+  scf.loop token(%outer) {
+    // CHECK: scf.loop token(%[[INNER:.*]]) {
+    scf.loop token(%inner) {
+      // CHECK: scf.continue [%[[OUTER]]]
+      scf.continue [%outer]
+    }
+  }
+  return
+}
diff --git a/mlir/test/Integration/Dialect/SCF/early_exit.mlir b/mlir/test/Integration/Dialect/SCF/early_exit.mlir
new file mode 100644
index 0000000000000..518b602f790aa
--- /dev/null
+++ b/mlir/test/Integration/Dialect/SCF/early_exit.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt %s  -convert-scf-to-cf --canonicalize --convert-cf-to-llvm --convert-to-llvm | \
+// RUN: mlir-runner -e entry -entry-point-result=void  \
+// RUN: -shared-libs=%mlir_c_runner_utils | \
+// RUN: FileCheck %s
+
+
+
+// End-to-end test of all fp reduction intrinsics (not exhaustive unit tests).
+module {
+  llvm.func @entry() {
+   // Constant for the iteration space and various conditions
+   %one = llvm.mlir.constant(1 : i64) : i64
+   %two = llvm.mlir.constant(2 : i64) : i64
+   %three = llvm.mlir.constant(3 : i64) : i64
+   %four = llvm.mlir.constant(4 : i64) : i64
+   %counter_init = llvm.mlir.constant(0 : i64) : i64
+
+
+// CHECK: Outer Loop Begin with counter: 0
+// CHECK-NEXT: Inner Loop Begin, counter: 1
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 2
+// CHECK-NEXT: Iteration 2, loop back to outer loop
+// CHECK-NEXT: Outer Loop Begin with counter: 2
+// CHECK-NEXT: Inner Loop Begin, counter: 3
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 4
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 5
+// CHECK-NEXT: Last iteration, break out of outer loop
+// CHECK-NEXT: Outer loop finished with result: 4
+
+
+   %result = scf.loop token(%outer) iter_args(%counter_out = %counter_init) : i64 -> i64 {
+     // Outer loop iteration
+     vector.print str "Outer Loop Begin with counter: "
+     vector.print %counter_out : i64
+
+     scf.loop token(%inner) iter_args(%counter = %counter_out) : i64 {
+       // %counter will go from 0 to 4
+       // %counter_update will go from 1 to 5
+       %counter_update = llvm.add %counter, %one : i64
+
+       // Inner loop iteration
+       // print from 1..5
+       vector.print str "Inner Loop Begin, counter: "
+       vector.print  %counter_update : i64
+
+       // On the second iteration, print 2.3 and loop back to the outer loop.
+       %cond1 = llvm.icmp "eq" %counter_update, %two : i64
+       scf.if %cond1 {
+         vector.print str "Iteration 2, loop back to outer loop\n"
+         scf.continue [%outer] %counter_update : i64
+       }
+
+       // Exit condition when counter>4
+       %cond2 = llvm.icmp "sge" %counter, %four : i64
+       scf.if %cond2 {
+         vector.print str "Last iteration, break out of outer loop\n"
+         // return the counter from the previous iteration here (pre-update)
+         scf.break [%outer] %counter : i64
+       }
+
+       %cond3 = llvm.icmp "eq" %counter_update, %three : i64
+       scf.if %cond2 {
+         vector.print str "Iteration 3, break out of inner loop"
+         scf.break [%inner]
+       }
+       vector.print str "continue inner loop\n"
+       scf.continue [%inner] %counter_update : i64
+     }
+      vector.print str "continue outer loop\n"
+      scf.continue [%outer] %counter_out : i64
+   }
+
+// After the loop nest finishes
+    vector.print str "Outer loop finished with result: "
+    vector.print  %result : i64
+
+    llvm.return
+  }
+}
diff --git a/mlir/test/Transforms/sccp-early-exit.mlir b/mlir/test/Transforms/sccp-early-exit.mlir
new file mode 100644
index 0000000000000..4a4136b94b891
--- /dev/null
+++ b/mlir/test/Transforms/sccp-early-exit.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt -allow-unregistered-dialect %s -pass-pipeline="builtin.module(func.func(sccp))" -split-input-file | FileCheck %s
+
+/// The inner loop's only normal exit (break 1) carries the constant 5.
+/// The early-exit path (break 3) carries -5 but bypasses the code after the
+/// inner loop entirely — it exits both loops at once.
+///
+/// Without early-exit support the analysis would conservatively join {5, -5}
+/// and mark %inner as overdefined, keeping the dead branch alive.
+/// With early-exit support SCCP sees that %inner is always 5, folds the
+/// comparison, and removes the dead scf.if.
+///
+/// Pseudocode:
+///   loop {
+///     a = loop {
+///       if (cond) { break_all -5; }  // early exit from both loops
+///       break 5;                      // normal exit from inner loop
+///     };
+///     // here a == 5 always
+///     if (a < 0) "dead"();            // dead code
+///     break a;
+///   }
+
+// CHECK-LABEL: func @early_exit_dead_code(
+// CHECK-DAG:     %[[FALSE:.*]] = arith.constant false
+// CHECK-DAG:     %[[C5:.*]] = arith.constant 5 : i32
+// CHECK:         scf.loop
+// CHECK:           scf.loop
+// CHECK:           scf.if %[[FALSE]]
+// CHECK:             "test.dead_op"
+// CHECK:           scf.break {{.*}} %[[C5]] : i32
+func.func @early_exit_dead_code(%cond: i1) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %c5 = arith.constant 5 : i32
+  %cm5 = arith.constant -5 : i32
+  %outer = scf.loop token(%outer_token) -> i32 {
+    %inner = scf.loop token(%inner_token) -> i32 {
+      scf.if %cond {
+        scf.break [%outer_token] %cm5 : i32
+      }
+      scf.break [%inner_token] %c5 : i32
+    }
+    %is_neg = arith.cmpi slt, %inner, %c0 : i32
+    scf.if %is_neg {
+      "test.dead_op"() : () -> ()
+    }
+    scf.break [%outer_token] %inner : i32
+  }
+  return %outer : i32
+}
+
+// -----
+
+/// For contrast: the same logic emulated with scf.while and a boolean flag
+/// instead of early exit.  Without break 3 to bypass the code after the inner
+/// loop, both values (-5 and 5) flow through %inner and SCCP sees it as
+/// overdefined — the comparison and dead branch survive.
+///
+/// Pseudocode:
+///   done = false
+///   while (!done) {
+///     a, should_break = while (!done) {
+///       if (cond) yield -5, true   // want to break all, but can't
+///       else      yield  5, false
+///     };
+///     if (!should_break) {
+///       // a ∈ {5, -5} — flag can't help the analysis narrow it
+///       if (a < 0) "not dead"();
+///     }
+///     done = true
+///   }
+
+// CHECK-LABEL: func @no_early_exit_not_foldable(
+// CHECK:         scf.while
+// CHECK:           scf.while
+// CHECK:           scf.if
+// CHECK:             arith.cmpi slt,
+// CHECK:             scf.if
+// CHECK:               "test.not_dead_op"
+func.func @no_early_exit_not_foldable(%cond: i1) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %c5 = arith.constant 5 : i32
+  %cm5 = arith.constant -5 : i32
+  %false = arith.constant false
+  %true = arith.constant true
+
+  // Outer while: (result, done)
+  %outer, %_ = scf.while(%o_res = %c0, %o_done = %false) : (i32, i1) -> (i32, i1) {
+    %o_go = arith.xori %o_done, %true : i1
+    scf.condition(%o_go) %o_res, %o_done : i32, i1
+  } do {
+  ^bb0(%o_res: i32, %o_done: i1):
+    // Inner while: (result, done, should_break_outer)
+    %inner, %_2, %break_flag = scf.while(%i_res = %c0, %i_done = %false, %i_brk = %false)
+        : (i32, i1, i1) -> (i32, i1, i1) {
+      %i_go = arith.xori %i_done, %true : i1
+      scf.condition(%i_go) %i_res, %i_done, %i_brk : i32, i1, i1
+    } do {
+    ^bb0(%i_res: i32, %i_done: i1, %i_brk: i1):
+      // if (cond) a = -5, flag = true; else a = 5, flag = false
+      %a = arith.select %cond, %cm5, %c5 : i32
+      %brk = arith.select %cond, %true, %false : i1
+      scf.yield %a, %true, %brk : i32, i1, i1
+    }
+    // %inner ∈ {5, -5}: both values reach here (no early exit to bypass)
+    // %break_flag correlates with %inner but SCCP can't exploit that
+    %not_break = arith.xori %break_flag, %true : i1
+    scf.if %not_break {
+      // %inner ∈ {5} in theory: but the analysis can't recover this here.
+      %is_neg = arith.cmpi slt, %inner, %c0 : i32
+      scf.if %is_neg {
+        "test.not_dead_op"() : () -> ()
+      }
+    }
+    scf.yield %inner, %true : i32, i1
+  }
+  return %outer : i32
+}
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index c12e2abb3ab75..9d7e21cabda6f 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -2385,6 +2385,12 @@ def TestMergeBlocksOp : TEST_Op<"merge_blocks"> {
   let results = (outs Variadic<AnyType>:$result);
 }
 
+def TestPropagateControlFlowBreakOp
+    : TEST_Op<"propagate_control_flow_break", [PropagateControlFlowBreak]> {
+  let regions = (region AnyRegion:$body);
+  let assemblyFormat = "attr-dict-with-keyword $body";
+}
+
 def TestRemappedValueRegionOp : TEST_Op<"remapped_value_region",
                                         [SingleBlock]> {
   let summary = "remapped_value_region operation";
diff --git a/mlir/test/lib/IR/TestDominance.cpp b/mlir/test/lib/IR/TestDominance.cpp
index b34149b3e2cbd..24696c8dd028c 100644
--- a/mlir/test/lib/IR/TestDominance.cpp
+++ b/mlir/test/lib/IR/TestDominance.cpp
@@ -14,6 +14,7 @@
 
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/Dominance.h"
+#include "mlir/IR/Operation.h"
 #include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
@@ -21,6 +22,14 @@ using namespace mlir;
 
 /// Overloaded helper to call the right function based on whether we are testing
 /// dominance or post-dominance.
+static bool dominatesOrPostDominates(DominanceInfo &dominanceInfo, Operation *a,
+                                     Operation *b) {
+  return dominanceInfo.dominates(a, b);
+}
+static bool dominatesOrPostDominates(PostDominanceInfo &dominanceInfo,
+                                     Operation *a, Operation *b) {
+  return dominanceInfo.postDominates(a, b);
+}
 static bool dominatesOrPostDominates(DominanceInfo &dominanceInfo, Block *a,
                                      Block *b) {
   return dominanceInfo.dominates(a, b);
@@ -72,6 +81,30 @@ class DominanceTest {
   template <typename DominanceT>
   void printDominance(DominanceT &dominanceInfo,
                       bool printCommonDominatorInfo) {
+    if (printCommonDominatorInfo) {
+      operation->walk([&](Operation *op) {
+        if (!op->getDiscardableAttr("test.print_dominance"))
+          return;
+        operation->walk([&](Operation *nested) {
+          if (std::is_same<DominanceInfo, DominanceT>::value)
+            llvm::outs() << "dominates(";
+          else
+            llvm::outs() << "postdominates(";
+          bool isDominated =
+              dominatesOrPostDominates(dominanceInfo, op, nested);
+          llvm::outs() << OpWithFlags(op, OpPrintingFlags()
+                                              .skipRegions()
+                                              .enableDebugInfo()
+                                              .assumeVerified())
+                       << ", "
+                       << OpWithFlags(nested, OpPrintingFlags()
+                                                  .skipRegions()
+                                                  .enableDebugInfo()
+                                                  .assumeVerified())
+                       << ") = " << std::to_string(isDominated) << "\n";
+        });
+      });
+    }
     DenseSet<Block *> parentVisited;
     operation->walk([&](Operation *op) {
       Block *block = op->getBlock();
diff --git a/mlir/test/lib/Interfaces/CMakeLists.txt b/mlir/test/lib/Interfaces/CMakeLists.txt
index 6a21ed10eec6f..3aa5097b7ed20 100644
--- a/mlir/test/lib/Interfaces/CMakeLists.txt
+++ b/mlir/test/lib/Interfaces/CMakeLists.txt
@@ -1,2 +1,3 @@
 add_subdirectory(LoopLikeInterface)
+add_subdirectory(RegionBranchOpInterface)
 add_subdirectory(TilingInterface)
diff --git a/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt b/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
new file mode 100644
index 0000000000000..8e003942e41c0
--- /dev/null
+++ b/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
@@ -0,0 +1,9 @@
+add_mlir_library(MLIRTestRegionBranchOpInterface
+  TestRegionBranchOpInterface.cpp
+
+  EXCLUDE_FROM_LIBMLIR
+  )
+mlir_target_link_libraries(MLIRTestRegionBranchOpInterface PUBLIC
+  MLIRControlFlowInterfaces
+  MLIRPass
+  )
diff --git a/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp b/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp
new file mode 100644
index 0000000000000..878de7d44da01
--- /dev/null
+++ b/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp
@@ -0,0 +1,77 @@
+//===- TestRegionBranchOpInterface.cpp - RegionBranch test pass -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/Interfaces/ControlFlowInterfaces.h"
+#include "mlir/Pass/Pass.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace mlir;
+
+namespace {
+/// Test pass that prints RegionBranchOpInterface successor information and
+/// nested breaking-control-flow predecessors.
+struct PrintRegionBranchOpInterfacePass
+    : public PassWrapper<PrintRegionBranchOpInterfacePass, OperationPass<>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(PrintRegionBranchOpInterfacePass)
+
+  StringRef getArgument() const final {
+    return "print-region-branch-op-interface";
+  }
+  StringRef getDescription() const final {
+    return "Print control-flow edges represented by "
+           "mlir::RegionBranchOpInterface";
+  }
+
+  void runOnOperation() override {
+    Operation *op = getOperation();
+    op->walk<WalkOrder::PreOrder>([&](RegionBranchOpInterface branchOp) {
+      llvm::outs() << "Found RegionBranchOpInterface operation: "
+                   << OpWithFlags(
+                          branchOp,
+                          OpPrintingFlags().skipRegions().enableDebugInfo())
+                   << "\n";
+      SmallVector<RegionSuccessor> regions;
+      branchOp.getSuccessorRegions(RegionBranchPoint::parent(), regions);
+      for (auto &successor : regions) {
+        if (successor.isOperation()) {
+          llvm::outs() << " - Successor is operation "
+                       << successor.getSuccessorOp()->getName() << "\n";
+        } else {
+          llvm::outs() << " - Successor is region #"
+                       << successor.getSuccessor()->getRegionNumber() << "\n";
+        }
+      }
+      if (auto breakingControlFlowOp =
+              dyn_cast<HasBreakingControlFlowOpInterface>(
+                  branchOp.getOperation())) {
+        SmallVector<Operation *> predecessors;
+        llvm::outs() << " - Collecting all nested predecessors\n";
+        collectAllNestedPredecessors(breakingControlFlowOp, predecessors);
+        llvm::outs() << " - Found " << predecessors.size()
+                     << " predecessor(s)\n";
+        for (auto &predecessor : predecessors) {
+          llvm::outs() << "  - Predecessor is "
+                       << OpWithFlags(
+                              predecessor,
+                              OpPrintingFlags().skipRegions().enableDebugInfo())
+                       << "\n";
+        }
+      }
+    });
+  }
+};
+
+} // namespace
+
+namespace mlir {
+void registerRegionBranchOpInterfaceTestPasses() {
+  PassRegistration<PrintRegionBranchOpInterfacePass>();
+}
+} // namespace mlir
diff --git a/mlir/tools/mlir-opt/CMakeLists.txt b/mlir/tools/mlir-opt/CMakeLists.txt
index c607ccfa80e3c..821bad3a4166a 100644
--- a/mlir/tools/mlir-opt/CMakeLists.txt
+++ b/mlir/tools/mlir-opt/CMakeLists.txt
@@ -22,6 +22,7 @@ if(MLIR_INCLUDE_TESTS)
     MLIRGPUTestPasses
     MLIRLinalgTestPasses
     MLIRLoopLikeInterfaceTestPasses
+    MLIRTestRegionBranchOpInterface
     MLIRMathTestPasses
     MLIRTestMathToVCIX
     MLIRMemRefTestPasses
diff --git a/mlir/tools/mlir-opt/mlir-opt.cpp b/mlir/tools/mlir-opt/mlir-opt.cpp
index 13c0934f34656..8c8813b480a99 100644
--- a/mlir/tools/mlir-opt/mlir-opt.cpp
+++ b/mlir/tools/mlir-opt/mlir-opt.cpp
@@ -38,6 +38,7 @@ void registerLazyLoadingTestPasses();
 void registerLoopLikeInterfaceTestPasses();
 void registerPassManagerTestPass();
 void registerPrintSpirvAvailabilityPass();
+void registerRegionBranchOpInterfaceTestPasses();
 void registerRegionTestPasses();
 void registerPrintTosaAvailabilityPass();
 void registerShapeFunctionTestPasses();
@@ -191,6 +192,7 @@ static void registerTestPasses() {
   registerPassManagerTestPass();
   registerPrintSpirvAvailabilityPass();
   registerRegionTestPasses();
+  registerRegionBranchOpInterfaceTestPasses();
   registerShapeFunctionTestPasses();
   registerSideEffectTestPasses();
   registerSliceAnalysisTestPass();

>From 5eaff8afdd45c90e7a999be5b47f42195e8ef0db Mon Sep 17 00:00:00 2001
From: Mehdi Amini <joker.eph at gmail.com>
Date: Thu, 11 Jun 2026 07:11:07 -0700
Subject: [PATCH 2/2] [mlir][test] Add dynamic-depth early-exit loop ops

Add Test dialect loop and terminators that model dynamic breaking control-flow
targets with an index depth. Constant depths resolve to one enclosing
test.breakable_loop; dynamic depths expose compatible reachable loop targets
through getPotentialTargets().

Filter dynamic potential targets by the terminator payload types. Runtime depths
that select an incompatible filtered-out target are undefined, while constant
depths still diagnose payload mismatches against the selected loop.

Add a test conversion pass that lowers test.breakable_loop to scf.loop and
rewrites dynamic break/continue terminators to token-targeted SCF terminators.
Dynamic depths lower through a dispatch ladder over compatible target loops
while preserving each target's original enclosing-loop depth.

Add verifier coverage for invalid depths, blocked propagation, and payload type
mismatches, RegionBranchOpInterface coverage for dynamic targets, FileCheck
coverage for the SCF conversion, and execution tests that lower through
SCF-to-CF.

Assisted-by: Codex
---
 .../IR/early-exit-test-dialect-invalid.mlir   |  93 +++++
 mlir/test/IR/early-exit-test-dialect.mlir     | 124 +++++++
 .../test-convert-breakable-loop-to-scf.mlir   | 139 +++++++
 mlir/test/lib/Dialect/Test/TestOpDefs.cpp     | 340 ++++++++++++++++++
 mlir/test/lib/Dialect/Test/TestOps.td         |  67 ++++
 mlir/test/lib/Dialect/Test/TestPatterns.cpp   | 213 +++++++++++
 .../test-convert-breakable-loop-to-scf.mlir   | 145 ++++++++
 7 files changed, 1121 insertions(+)
 create mode 100644 mlir/test/IR/early-exit-test-dialect-invalid.mlir
 create mode 100644 mlir/test/IR/early-exit-test-dialect.mlir
 create mode 100644 mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
 create mode 100644 mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir

diff --git a/mlir/test/IR/early-exit-test-dialect-invalid.mlir b/mlir/test/IR/early-exit-test-dialect-invalid.mlir
new file mode 100644
index 0000000000000..c5ca385e4aa6b
--- /dev/null
+++ b/mlir/test/IR/early-exit-test-dialect-invalid.mlir
@@ -0,0 +1,93 @@
+// RUN: mlir-opt %s --split-input-file --verify-diagnostics
+
+func.func @non_index_depth(%depth: i32) {
+  test.breakable_loop {
+    // expected-error @+1 {{operand #0 must be index}}
+    "test.dynamic_break"(%depth) : (i32) -> ()
+  }
+  return
+}
+
+// -----
+
+func.func @no_reachable_loop(%depth: index) {
+  // expected-error @+1 {{must be nested inside a reachable test.breakable_loop}}
+  test.dynamic_break %depth
+}
+
+// -----
+
+func.func @zero_depth() {
+  %c0 = arith.constant 0 : index
+  test.breakable_loop {
+    // expected-error @+1 {{depth must be positive}}
+    test.dynamic_continue %c0
+  }
+  return
+}
+
+// -----
+
+func.func @too_deep() {
+  %c2 = arith.constant 2 : index
+  test.breakable_loop {
+    // expected-error @+1 {{constant depth exceeds the number of reachable test.breakable_loop operations}}
+    test.dynamic_break %c2
+  }
+  return
+}
+
+// -----
+
+func.func @blocked_parent() {
+  %c1 = arith.constant 1 : index
+  test.breakable_loop {
+    "test.any_cond"() ({
+      // expected-error @+1 {{depth target crosses an op that does not have the PropagateControlFlowBreak trait}}
+      test.dynamic_break %c1
+    }) : () -> ()
+    test.dynamic_continue %c1
+  }
+  return
+}
+
+// -----
+
+func.func @dynamic_depth_blocked_outer(%depth: index) {
+  %c1 = arith.constant 1 : index
+  test.breakable_loop {
+    test.single_no_terminator_custom_asm_op {
+      test.breakable_loop {
+        // expected-error @+1 {{dynamic depth may target across an op that does not have the PropagateControlFlowBreak trait}}
+        test.dynamic_break %depth
+      }
+    }
+    test.dynamic_continue %c1
+  }
+  return
+}
+
+// -----
+
+// Constant depths select one concrete loop, so payload mismatches are still
+// diagnosed against that selected target.
+func.func @constant_break_payload_mismatch(%value: f32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %result = test.breakable_loop -> i32 {
+    // expected-error @+1 {{payload operand #0 has type 'f32', but target results}}
+    test.dynamic_break %c1 %value : f32
+  }
+  return %result : i32
+}
+
+// -----
+
+// Dynamic depths filter out incompatible targets. This is still invalid when
+// filtering leaves no target that can accept the terminator payload.
+func.func @dynamic_break_no_compatible_target(%depth: index, %value: f32) -> i32 {
+  %result = test.breakable_loop -> i32 {
+    // expected-error @+1 {{dynamic depth has no compatible test.breakable_loop target}}
+    test.dynamic_break %depth %value : f32
+  }
+  return %result : i32
+}
diff --git a/mlir/test/IR/early-exit-test-dialect.mlir b/mlir/test/IR/early-exit-test-dialect.mlir
new file mode 100644
index 0000000000000..3ea7f0e8154b8
--- /dev/null
+++ b/mlir/test/IR/early-exit-test-dialect.mlir
@@ -0,0 +1,124 @@
+// RUN: mlir-opt %s --split-input-file | FileCheck %s --check-prefix=PRINT
+// RUN: mlir-opt --print-region-branch-op-interface %s --split-input-file | FileCheck %s --check-prefix=BRANCH
+
+// PRINT-LABEL: func.func @immediate_continue(
+func.func @immediate_continue(%depth: index) {
+  // PRINT: test.breakable_loop
+  test.breakable_loop {
+    // PRINT: test.dynamic_continue %arg0
+    test.dynamic_continue %depth
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @break_with_result(
+func.func @break_with_result(%depth: index, %value: i32) -> i32 {
+  // PRINT: test.breakable_loop -> i32
+  %result = test.breakable_loop -> i32 {
+    // PRINT: test.dynamic_break %arg0 %arg1 : i32
+    test.dynamic_break %depth %value : i32
+  }
+  return %result : i32
+}
+
+// -----
+
+// PRINT-LABEL: func.func @continue_iter_args(
+func.func @continue_iter_args(%depth: index, %init: i32) {
+  // PRINT: test.breakable_loop iter_args(%{{.*}} = %arg1) : i32
+  test.breakable_loop iter_args(%iter = %init) : i32 {
+    // PRINT: test.dynamic_continue %arg0 %{{.*}} : i32
+    test.dynamic_continue %depth %iter : i32
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @constant_depth_selects_outer(
+func.func @constant_depth_selects_outer(%outer_init: i32, %inner_init: f32) {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  test.breakable_loop iter_args(%outer = %outer_init) : i32 {
+    test.breakable_loop iter_args(%inner = %inner_init) : f32 {
+      // PRINT: test.dynamic_continue %c2 %{{.*}} : i32
+      test.dynamic_continue %c2 %outer : i32
+    }
+    test.dynamic_continue %c1 %outer : i32
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @through_scf_if(
+func.func @through_scf_if(%cond: i1, %depth: index) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("loop")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 2 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("if_break")
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("after_if")
+  test.breakable_loop {
+    // PRINT: scf.if
+    scf.if %cond {
+      // PRINT: test.dynamic_break %arg1
+      test.dynamic_break %depth loc("if_break")
+    } loc("if")
+    test.dynamic_continue %c1 loc("after_if")
+  } loc("loop")
+  return
+}
+
+// -----
+
+// A dynamic depth can target either the immediately enclosing loop or an outer
+// compatible loop. The inner loop sees the dynamic break as a direct
+// predecessor, and the outer loop sees both the propagated dynamic break and
+// its own explicit continue.
+func.func @nested_dynamic(%depth: index) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("outer")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 2 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("dyn_break")
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("outer_continue")
+  test.breakable_loop {
+    // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("inner")
+    // BRANCH:  - Successor is region #0
+    // BRANCH:  - Found 1 predecessor(s)
+    // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("dyn_break")
+    test.breakable_loop {
+      test.dynamic_break %depth loc("dyn_break")
+    } loc("inner")
+    test.dynamic_continue %c1 loc("outer_continue")
+  } loc("outer")
+  return
+}
+
+// -----
+
+// The dynamic continue payload is f32. That makes the inner f32 loop a
+// potential target, but filters out the enclosing i32 loop. The outer loop
+// should only see its explicit continue predecessor.
+func.func @dynamic_continue_filters_incompatible_outer(%depth: index, %i: i32,
+                                                       %f: f32) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("filtered_outer")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 1 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("filtered_outer_continue")
+  test.breakable_loop iter_args(%outer = %i) : i32 {
+    // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("filtered_inner")
+    // BRANCH:  - Successor is region #0
+    // BRANCH:  - Found 1 predecessor(s)
+    // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("filtered_inner_continue")
+    test.breakable_loop iter_args(%inner = %f) : f32 {
+      test.dynamic_continue %depth %inner : f32 loc("filtered_inner_continue")
+    } loc("filtered_inner")
+    test.dynamic_continue %c1 %outer : i32 loc("filtered_outer_continue")
+  } loc("filtered_outer")
+  return
+}
diff --git a/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir b/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
new file mode 100644
index 0000000000000..f33530e6d91d1
--- /dev/null
+++ b/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
@@ -0,0 +1,139 @@
+// RUN: mlir-opt %s --test-convert-breakable-loop-to-scf --split-input-file | FileCheck %s
+
+// CHECK-LABEL: func.func @immediate_continue(
+// CHECK-SAME: %[[INIT:.*]]: i32
+func.func @immediate_continue(%init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[TOKEN:.*]]) iter_args(%[[ITER:.*]] = %[[INIT]]) : i32 -> i32
+  %result = test.breakable_loop iter_args(%iter = %init) : i32 -> i32 {
+    // CHECK: scf.continue [%[[TOKEN]]] %[[ITER]] : i32
+    test.dynamic_continue %c1 %iter : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// CHECK-LABEL: func.func @constant_outer_break(
+func.func @constant_outer_break(%init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) {
+    test.breakable_loop {
+      // CHECK: scf.break [%[[OUTER_TOKEN]]] %[[OUTER_ARG]] : i32
+      test.dynamic_break %c2 %outer : i32
+    }
+    test.dynamic_break %c1 %outer : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// Dynamic break over two compatible loops lowers to a dispatch ladder with one
+// arm for depth 1 (inner) and one for depth 2 (outer), plus a deterministic
+// fallback for UB depth values.
+// CHECK-LABEL: func.func @dynamic_break_dispatch(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32
+func.func @dynamic_break_dispatch(%choose_outer: i1, %init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c2, %c1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: %[[INNER_RESULT:.*]] = scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+    %inner_result = test.breakable_loop iter_args(%inner = %outer) : i32 -> i32 {
+      // CHECK: %[[DEPTH_ONE:.*]] = arith.constant 1 : index
+      // CHECK: %[[IS_ONE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_ONE]] : index
+      // CHECK: scf.if %[[IS_ONE]] {
+      // CHECK:   scf.break [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+      // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+      // CHECK: scf.if %[[IS_TWO]] {
+      // CHECK:   scf.break [%[[OUTER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: scf.break [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      test.dynamic_break %depth %inner : i32
+    }
+    test.dynamic_break %c1 %inner_result : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// The same dispatch shape is required for dynamic continue, except each arm
+// targets the next iteration of the selected loop.
+// CHECK-LABEL: func.func @dynamic_continue_dispatch(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32
+func.func @dynamic_continue_dispatch(%choose_outer: i1, %init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c2, %c1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+    test.breakable_loop iter_args(%inner = %outer) : i32 {
+      // CHECK: %[[DEPTH_ONE:.*]] = arith.constant 1 : index
+      // CHECK: %[[IS_ONE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_ONE]] : index
+      // CHECK: scf.if %[[IS_ONE]] {
+      // CHECK:   scf.continue [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+      // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+      // CHECK: scf.if %[[IS_TWO]] {
+      // CHECK:   scf.continue [%[[OUTER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: scf.continue [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      test.dynamic_continue %depth %inner : i32
+    }
+    test.dynamic_break %c1 %outer : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// The innermost loop carries f32, so it is not compatible with the i32 break
+// payload and must be filtered out. The remaining compatible targets are still
+// addressed by their original depths: 2 for the middle loop and 3 for the outer
+// loop.
+// CHECK-LABEL: func.func @dynamic_break_dispatch_filtered_depths(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32, %[[F_INIT:.*]]: f32
+func.func @dynamic_break_dispatch_filtered_depths(%choose_outer: i1,
+                                                  %init: i32, %inner_init: f32)
+    -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %c3 = arith.constant 3 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c3, %c2 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: %[[MIDDLE_RESULT:.*]] = scf.loop token(%[[MIDDLE_TOKEN:.*]]) iter_args(%[[MIDDLE_ARG:.*]] =
+    %middle_result = test.breakable_loop iter_args(%middle = %outer) : i32 -> i32 {
+      // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+      test.breakable_loop iter_args(%inner = %inner_init) : f32 {
+        // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+        // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+        // CHECK: scf.if %[[IS_TWO]] {
+        // CHECK:   scf.break [%[[MIDDLE_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        // CHECK: %[[DEPTH_THREE:.*]] = arith.constant 3 : index
+        // CHECK: %[[IS_THREE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_THREE]] : index
+        // CHECK: scf.if %[[IS_THREE]] {
+        // CHECK:   scf.break [%[[OUTER_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        // CHECK: scf.break [%[[MIDDLE_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        test.dynamic_break %depth %middle : i32
+      }
+      test.dynamic_break %c1 %middle : i32
+    }
+    test.dynamic_break %c1 %middle_result : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
diff --git a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
index c6470da1b7852..b38e6b1455c23 100644
--- a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
@@ -10,6 +10,7 @@
 #include "TestOps.h"
 #include "mlir/Dialect/Bufferization/IR/Bufferization.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/IR/Matchers.h"
 #include "mlir/IR/Verifier.h"
 #include "mlir/Interfaces/FunctionImplementation.h"
 #include "mlir/Interfaces/MemorySlotInterfaces.h"
@@ -827,6 +828,345 @@ void AnyCondOp::getRegionInvocationBounds(
   invocationBounds.emplace_back(1, 1);
 }
 
+//===----------------------------------------------------------------------===//
+// TestBreakableLoopOp / TestDynamicBreakOp / TestDynamicContinueOp
+//===----------------------------------------------------------------------===//
+
+static std::optional<int64_t> getStaticDepth(Value depth) {
+  APInt value;
+  if (!matchPattern(depth, m_ConstantInt(&value)))
+    return std::nullopt;
+  return value.getSExtValue();
+}
+
+static SmallVector<TestBreakableLoopOp>
+getReachableBreakableLoops(Operation *terminator) {
+  SmallVector<TestBreakableLoopOp> loops;
+  for (Operation *currentOp = terminator->getParentOp(); currentOp;
+       currentOp = currentOp->getParentOp()) {
+    if (auto loopOp = dyn_cast<TestBreakableLoopOp>(currentOp))
+      loops.push_back(loopOp);
+    if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      break;
+  }
+  return loops;
+}
+
+static Operation *getPropagationBlockingParent(Operation *terminator) {
+  for (Operation *currentOp = terminator->getParentOp(); currentOp;
+       currentOp = currentOp->getParentOp()) {
+    if (currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      continue;
+    for (Operation *parentOp = currentOp->getParentOp(); parentOp;
+         parentOp = parentOp->getParentOp())
+      if (isa<TestBreakableLoopOp>(parentOp))
+        return currentOp;
+    return nullptr;
+  }
+  return nullptr;
+}
+
+static SmallVector<TestBreakableLoopOp>
+getResolvedBreakableLoops(Operation *terminator, Value depth) {
+  SmallVector<TestBreakableLoopOp> loops =
+      getReachableBreakableLoops(terminator);
+  std::optional<int64_t> staticDepth = getStaticDepth(depth);
+  if (!staticDepth)
+    return loops;
+  if (*staticDepth <= 0 || static_cast<uint64_t>(*staticDepth) > loops.size())
+    return {};
+  return {loops[*staticDepth - 1]};
+}
+
+static OperandRange getTerminatorPayload(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getArgs();
+  return cast<TestDynamicContinueOp>(terminator).getArgs();
+}
+
+static bool isContinueTerminator(Operation *terminator) {
+  if (isa<TestDynamicContinueOp>(terminator))
+    return true;
+  assert(isa<TestDynamicBreakOp>(terminator) &&
+         "expected test.dynamic_break or test.dynamic_continue");
+  return false;
+}
+
+static ValueRange getTargetExpectedValues(TestBreakableLoopOp target,
+                                          bool isContinue) {
+  return isContinue ? ValueRange(target.getRegionIterArgs())
+                    : ValueRange(target.getResults());
+}
+
+static bool isTerminatorPayloadCompatible(OperandRange args,
+                                          ValueRange expectedValues) {
+  if (args.size() != expectedValues.size())
+    return false;
+  return llvm::all_of(llvm::zip(args, expectedValues), [](auto values) {
+    return std::get<0>(values).getType() == std::get<1>(values).getType();
+  });
+}
+
+static bool isPotentialTerminatorTarget(Operation *terminator,
+                                        OperandRange args,
+                                        TestBreakableLoopOp target,
+                                        bool isContinue) {
+  return target.acceptsTerminator(terminator) &&
+         isTerminatorPayloadCompatible(
+             args, getTargetExpectedValues(target, isContinue));
+}
+
+static SmallVector<Operation *>
+getPotentialBreakableLoopTargets(Operation *terminator, Value depth) {
+  SmallVector<Operation *> targets;
+  OperandRange args = getTerminatorPayload(terminator);
+  bool isContinue = isContinueTerminator(terminator);
+  for (TestBreakableLoopOp loopOp :
+       getResolvedBreakableLoops(terminator, depth)) {
+    if (!isPotentialTerminatorTarget(terminator, args, loopOp, isContinue))
+      continue;
+    targets.push_back(loopOp.getOperation());
+  }
+  return targets;
+}
+
+static void
+appendTestBreakableLoopSuccessor(TestBreakableLoopOp loopOp, bool isContinue,
+                                 SmallVectorImpl<RegionSuccessor> &regions) {
+  regions.push_back(isContinue ? RegionSuccessor(&loopOp.getBody())
+                               : RegionSuccessor(loopOp.getOperation()));
+}
+
+static LogicalResult verifyTerminatorPayload(Operation *terminator,
+                                             OperandRange args,
+                                             ValueRange expectedValues,
+                                             TestBreakableLoopOp target,
+                                             StringRef targetKind) {
+  if (args.size() != expectedValues.size())
+    return terminator->emitOpError()
+           << "has " << args.size() << " payload operands, but target "
+           << targetKind << " expects " << expectedValues.size();
+
+  for (auto [index, argAndExpected] :
+       llvm::enumerate(llvm::zip(args, expectedValues))) {
+    Value arg = std::get<0>(argAndExpected);
+    Value expected = std::get<1>(argAndExpected);
+    if (arg.getType() == expected.getType())
+      continue;
+    return terminator->emitOpError()
+           << "payload operand #" << index << " has type " << arg.getType()
+           << ", but target " << targetKind << " of "
+           << OpWithFlags(target, OpPrintingFlags().skipRegions())
+           << " expects " << expected.getType();
+  }
+  return success();
+}
+
+static LogicalResult verifyDynamicTerminator(Operation *terminator, Value depth,
+                                             OperandRange args,
+                                             bool isContinue) {
+  SmallVector<TestBreakableLoopOp> reachableLoops =
+      getReachableBreakableLoops(terminator);
+  Operation *blockingOp = getPropagationBlockingParent(terminator);
+  if (reachableLoops.empty()) {
+    if (blockingOp)
+      return terminator->emitOpError()
+             << "depth target crosses an op that does not have the "
+                "PropagateControlFlowBreak trait: "
+             << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+    return terminator->emitOpError()
+           << "must be nested inside a reachable test.breakable_loop";
+  }
+
+  SmallVector<TestBreakableLoopOp> targets = reachableLoops;
+  if (std::optional<int64_t> staticDepth = getStaticDepth(depth)) {
+    if (*staticDepth <= 0)
+      return terminator->emitOpError() << "depth must be positive";
+    if (static_cast<uint64_t>(*staticDepth) > reachableLoops.size()) {
+      if (blockingOp)
+        return terminator->emitOpError()
+               << "depth target crosses an op that does not have the "
+                  "PropagateControlFlowBreak trait: "
+               << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+      return terminator->emitOpError()
+             << "constant depth exceeds the number of reachable "
+                "test.breakable_loop operations";
+    }
+    targets.clear();
+    targets.push_back(reachableLoops[*staticDepth - 1]);
+  } else if (blockingOp) {
+    return terminator->emitOpError()
+           << "dynamic depth may target across an op that does not have the "
+              "PropagateControlFlowBreak trait: "
+           << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+  }
+
+  if (!getStaticDepth(depth)) {
+    for (TestBreakableLoopOp target : targets)
+      if (isPotentialTerminatorTarget(terminator, args, target, isContinue))
+        return success();
+    return terminator->emitOpError()
+           << "dynamic depth has no compatible test.breakable_loop target";
+  }
+
+  for (TestBreakableLoopOp target : targets) {
+    if (!target.acceptsTerminator(terminator))
+      return target.emitOpError("does not accept terminator: ")
+             << OpWithFlags(terminator, OpPrintingFlags().skipRegions());
+    ValueRange expectedValues = getTargetExpectedValues(target, isContinue);
+    if (failed(verifyTerminatorPayload(terminator, args, expectedValues, target,
+                                       isContinue ? "loop-carried values"
+                                                  : "results")))
+      return failure();
+  }
+  return success();
+}
+
+void TestBreakableLoopOp::print(OpAsmPrinter &p) {
+  p << " ";
+  if (!getInitArgs().empty()) {
+    p << "iter_args(";
+    llvm::interleaveComma(
+        llvm::zip(getRegionIterArgs(), getInitArgs()), p,
+        [&](auto it) { p << std::get<0>(it) << " = " << std::get<1>(it); });
+    p << ") : " << getInitArgs().getTypes() << " ";
+  }
+  if (!getResultTypes().empty())
+    p << "-> " << getResultTypes() << " ";
+  p.printRegion(getBody(), /*printEntryBlockArgs=*/false,
+                /*printBlockTerminators=*/true);
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+ParseResult TestBreakableLoopOp::parse(OpAsmParser &parser,
+                                       OperationState &result) {
+  SmallVector<OpAsmParser::Argument, 4> regionArgs;
+  SmallVector<OpAsmParser::Argument, 4> iterRegionArgs;
+  SmallVector<OpAsmParser::UnresolvedOperand, 4> iterOperands;
+  SmallVector<Type, 4> iterTypes;
+
+  if (failed(parser.parseOptionalKeyword("iter_args"))) {
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+  } else {
+    if (parser.parseAssignmentList(iterRegionArgs, iterOperands) ||
+        parser.parseColon() || parser.parseTypeList(iterTypes))
+      return failure();
+    if (iterRegionArgs.size() != iterTypes.size())
+      return parser.emitError(parser.getCurrentLocation(),
+                              "found different number of iter_args and types");
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+    for (auto [regionArg, type] : llvm::zip_equal(iterRegionArgs, iterTypes))
+      regionArg.type = type;
+    llvm::append_range(regionArgs, iterRegionArgs);
+  }
+
+  Region *body = result.addRegion();
+  if (parser.parseRegion(*body, regionArgs) ||
+      parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return parser.resolveOperands(iterOperands, iterTypes, parser.getNameLoc(),
+                                result.operands);
+}
+
+LogicalResult TestBreakableLoopOp::verifyRegions() {
+  if (getBody().empty())
+    return emitOpError("region cannot be empty");
+  if (getBody().front().getNumArguments() != getNumOperands())
+    return emitOpError("expected the region to have one argument per "
+                       "loop-carried value (")
+           << getNumOperands() << " expected, but got "
+           << getBody().front().getNumArguments() << ")";
+  for (auto [index, argAndOperand] :
+       llvm::enumerate(llvm::zip(getRegionIterArgs(), getOperands()))) {
+    Type argType = std::get<0>(argAndOperand).getType();
+    Type operandType = std::get<1>(argAndOperand).getType();
+    if (argType != operandType)
+      return emitOpError() << "types mismatch between " << index
+                           << "th iter operand (" << operandType
+                           << ") and defined region argument (" << argType
+                           << ")";
+  }
+  return success();
+}
+
+void TestBreakableLoopOp::getSuccessorRegions(
+    RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> &regions) {
+  if (point.isParent()) {
+    regions.push_back(RegionSuccessor(&getBody()));
+    return;
+  }
+
+  Operation *terminator = point.getTerminatorPredecessorOrNull();
+  Value depth;
+  bool isContinue = false;
+  if (auto continueOp = dyn_cast<TestDynamicContinueOp>(terminator)) {
+    depth = continueOp.getDepth();
+    isContinue = true;
+  } else if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator)) {
+    depth = breakOp.getDepth();
+  } else {
+    llvm_unreachable("expected test.dynamic_break or test.dynamic_continue");
+  }
+
+  bool isImmediateBranch = terminator->getParentOp() == getOperation();
+  for (Operation *target :
+       getPotentialBreakableLoopTargets(terminator, depth)) {
+    if (target == getOperation()) {
+      appendTestBreakableLoopSuccessor(*this, isContinue, regions);
+      continue;
+    }
+    if (!isImmediateBranch || !target->isProperAncestor(getOperation()))
+      continue;
+    appendTestBreakableLoopSuccessor(cast<TestBreakableLoopOp>(target),
+                                     isContinue, regions);
+  }
+}
+
+OperandRange
+TestBreakableLoopOp::getEntrySuccessorOperands(RegionSuccessor successor) {
+  return getInitArgs();
+}
+
+ValueRange TestBreakableLoopOp::getSuccessorInputs(RegionSuccessor successor) {
+  return successor.isOperation() ? ValueRange(getResults())
+                                 : ValueRange(getRegionIterArgs());
+}
+
+LogicalResult TestDynamicBreakOp::verify() {
+  return verifyDynamicTerminator(getOperation(), getDepth(), getArgs(),
+                                 /*isContinue=*/false);
+}
+
+SmallVector<Operation *> TestDynamicBreakOp::getPotentialTargets() {
+  return getPotentialBreakableLoopTargets(getOperation(), getDepth());
+}
+
+MutableOperandRange
+TestDynamicBreakOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult TestDynamicContinueOp::verify() {
+  return verifyDynamicTerminator(getOperation(), getDepth(), getArgs(),
+                                 /*isContinue=*/true);
+}
+
+SmallVector<Operation *> TestDynamicContinueOp::getPotentialTargets() {
+  return getPotentialBreakableLoopTargets(getOperation(), getDepth());
+}
+
+MutableOperandRange
+TestDynamicContinueOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
 //===----------------------------------------------------------------------===//
 // SingleBlockImplicitTerminatorOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index 9d7e21cabda6f..1439cf405d28a 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -2850,6 +2850,73 @@ def LoopBlockTerminatorOp : TEST_Op<"loop_block_term",
   }];
 }
 
+def TestBreakableLoopOp : TEST_Op<"breakable_loop", [
+    AutomaticAllocationScope,
+    RecursiveMemoryEffects,
+    PropagateControlFlowBreak,
+    SingleBlock,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface,
+        ["getEntrySuccessorOperands", "getSuccessorInputs"]>,
+    HasBreakingControlFlowOpInterface,
+    HasNestedTerminator<["TestDynamicContinueOp", "TestDynamicBreakOp"]>
+  ]> {
+  let summary = "test loop with dynamically addressed break/continue";
+  let description = [{
+    Test dialect loop operation that models breaking control flow with
+    dynamically selected targets. Its terminators use an SSA index value to
+    count enclosing `test.breakable_loop` operations. A depth of one targets the
+    immediately enclosing test loop, a depth of two targets the next enclosing
+    test loop, and so on. A constant depth must identify a reachable target with
+    compatible payload types. A dynamic depth may evaluate to any reachable depth
+    with compatible payload types; evaluating to zero, to an incompatible target,
+    or to a depth greater than the number of reachable loops is undefined
+    behavior.
+  }];
+
+  let arguments = (ins Variadic<AnyType>:$initArgs);
+  let results = (outs Variadic<AnyType>:$results);
+  let regions = (region SizedRegion<1>:$body);
+
+  let extraClassDeclaration = [{
+    ::mlir::Block::BlockArgListType getRegionIterArgs() {
+      return getBody().getArguments();
+    }
+  }];
+
+  let hasCustomAssemblyFormat = 1;
+  let hasRegionVerifier = 1;
+}
+
+def TestDynamicBreakOp : TEST_Op<"dynamic_break", [
+    Terminator, RegionExitTerminatorOpInterface,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>
+  ]> {
+  let summary = "dynamically targeted break from test.breakable_loop";
+  let arguments = (ins Index:$depth, Variadic<AnyType>:$args);
+  let assemblyFormat = "$depth ($args^ `:` type($args))? attr-dict";
+  let extraClassDeclaration = [{
+    ::llvm::SmallVector<::mlir::Operation *>
+    getPotentialTargets();
+  }];
+  let hasVerifier = 1;
+}
+
+def TestDynamicContinueOp : TEST_Op<"dynamic_continue", [
+    Terminator, RegionExitTerminatorOpInterface,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>
+  ]> {
+  let summary = "dynamically targeted continue of test.breakable_loop";
+  let arguments = (ins Index:$depth, Variadic<AnyType>:$args);
+  let assemblyFormat = "$depth ($args^ `:` type($args))? attr-dict";
+  let extraClassDeclaration = [{
+    ::llvm::SmallVector<::mlir::Operation *>
+    getPotentialTargets();
+  }];
+  let hasVerifier = 1;
+}
+
 def TestNoTerminatorOp : TEST_Op<"switch_with_no_break", [
     NoTerminator,
     DeclareOpInterfaceMethods<RegionBranchOpInterface>
diff --git a/mlir/test/lib/Dialect/Test/TestPatterns.cpp b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
index 552a1a473c9fd..1ace1d1101df0 100644
--- a/mlir/test/lib/Dialect/Test/TestPatterns.cpp
+++ b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
@@ -14,9 +14,11 @@
 #include "mlir/Dialect/ControlFlow/Transforms/StructuralTypeConversions.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Func/Transforms/FuncConversions.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/Visitors.h"
@@ -25,6 +27,7 @@
 #include "mlir/Transforms/FoldUtils.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/ScopeExit.h"
 #include <cstdint>
 
@@ -2490,6 +2493,214 @@ struct TestFoldTypeConvertingOp
       signalPassFailure();
   }
 };
+
+class TestConvertBreakableLoopToSCFPass
+    : public PassWrapper<TestConvertBreakableLoopToSCFPass, OperationPass<>> {
+public:
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(
+      TestConvertBreakableLoopToSCFPass)
+
+  StringRef getArgument() const final {
+    return "test-convert-breakable-loop-to-scf";
+  }
+  StringRef getDescription() const final {
+    return "Convert test.breakable_loop operations to scf.loop operations";
+  }
+
+  void runOnOperation() override;
+
+  struct TargetInfo {
+    Operation *target;
+    int64_t depth;
+  };
+  using TargetMap = llvm::DenseMap<Operation *, SmallVector<TargetInfo>>;
+  using TokenMap = llvm::DenseMap<Operation *, Value>;
+
+private:
+  LogicalResult rewriteLoop(TestBreakableLoopOp loopOp,
+                            const TargetMap &terminatorTargets,
+                            TokenMap &loopTokens);
+  LogicalResult rewriteTerminator(Operation *terminator,
+                                  const TargetMap &terminatorTargets,
+                                  const TokenMap &loopTokens,
+                                  PatternRewriter &rewriter);
+};
+
+static void appendTerminatorTargets(
+    Operation *terminator,
+    SmallVectorImpl<TestConvertBreakableLoopToSCFPass::TargetInfo> &targets,
+    SmallVector<Operation *> potentialTargets) {
+  for (Operation *targetOp : potentialTargets) {
+    int64_t depth = 0;
+    for (Operation *currentOp = terminator->getParentOp(); currentOp;
+         currentOp = currentOp->getParentOp()) {
+      if (isa<TestBreakableLoopOp>(currentOp)) {
+        ++depth;
+        if (currentOp == targetOp) {
+          targets.push_back({targetOp, depth});
+          break;
+        }
+      }
+      if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+        break;
+    }
+  }
+}
+
+static bool isBreakableLoopTerminator(Operation *op) {
+  return isa<TestDynamicBreakOp, TestDynamicContinueOp>(op);
+}
+
+static OperandRange getTerminatorPayload(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getArgs();
+  return cast<TestDynamicContinueOp>(terminator).getArgs();
+}
+
+static Value getTerminatorDepth(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getDepth();
+  return cast<TestDynamicContinueOp>(terminator).getDepth();
+}
+
+static void createSCFTerminator(PatternRewriter &rewriter, Location loc,
+                                Operation *terminator, Value targetToken,
+                                ValueRange args) {
+  if (isa<TestDynamicBreakOp>(terminator)) {
+    scf::BreakOp::create(rewriter, loc, targetToken, args);
+    return;
+  }
+  scf::ContinueOp::create(rewriter, loc, targetToken, args);
+}
+
+static std::optional<Value>
+lookupToken(Operation *terminator, Operation *target,
+            const TestConvertBreakableLoopToSCFPass::TokenMap &loopTokens) {
+  auto it = loopTokens.find(target);
+  if (it != loopTokens.end())
+    return it->second;
+  terminator->emitOpError()
+      << "cannot convert because target loop has not been converted";
+  return std::nullopt;
+}
+
+LogicalResult TestConvertBreakableLoopToSCFPass::rewriteTerminator(
+    Operation *terminator, const TargetMap &terminatorTargets,
+    const TokenMap &loopTokens, PatternRewriter &rewriter) {
+  auto targetIt = terminatorTargets.find(terminator);
+  if (targetIt == terminatorTargets.end() || targetIt->second.empty())
+    return terminator->emitOpError()
+           << "cannot convert terminator without a target loop";
+
+  ArrayRef<TargetInfo> targets = targetIt->second;
+  Location loc = terminator->getLoc();
+  SmallVector<Value> args(getTerminatorPayload(terminator));
+
+  if (targets.size() == 1) {
+    std::optional<Value> token =
+        lookupToken(terminator, targets.front().target, loopTokens);
+    if (!token)
+      return failure();
+    rewriter.setInsertionPoint(terminator);
+    createSCFTerminator(rewriter, loc, terminator, *token, args);
+    rewriter.eraseOp(terminator);
+    return success();
+  }
+
+  rewriter.setInsertionPoint(terminator);
+  Value depth = getTerminatorDepth(terminator);
+  for (TargetInfo target : targets) {
+    std::optional<Value> token =
+        lookupToken(terminator, target.target, loopTokens);
+    if (!token)
+      return failure();
+
+    Value depthValue =
+        arith::ConstantIndexOp::create(rewriter, loc, target.depth);
+    Value isTarget = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::eq, depth, depthValue);
+    auto ifOp = scf::IfOp::create(rewriter, loc, TypeRange{}, isTarget,
+                                  /*addThenBlock=*/true,
+                                  /*addElseBlock=*/false);
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      rewriter.setInsertionPointToStart(ifOp.thenBlock());
+      createSCFTerminator(rewriter, loc, terminator, *token, args);
+    }
+    rewriter.setInsertionPointAfter(ifOp);
+  }
+
+  // Dynamic depths are verified to be able to target any listed loop. If the
+  // value falls outside that set, selects an incompatible target, or otherwise
+  // has undefined behavior, the fallback may choose any valid target. Use the
+  // first compatible target to keep the generated IR deterministic.
+  std::optional<Value> fallbackToken =
+      lookupToken(terminator, targets.front().target, loopTokens);
+  if (!fallbackToken)
+    return failure();
+  createSCFTerminator(rewriter, loc, terminator, *fallbackToken, args);
+  rewriter.eraseOp(terminator);
+  return success();
+}
+
+LogicalResult TestConvertBreakableLoopToSCFPass::rewriteLoop(
+    TestBreakableLoopOp loopOp, const TargetMap &terminatorTargets,
+    TokenMap &loopTokens) {
+  PatternRewriter rewriter(loopOp.getContext());
+  rewriter.setInsertionPoint(loopOp);
+
+  auto scfLoop = scf::LoopOp::create(
+      rewriter, loopOp.getLoc(), loopOp.getResultTypes(), loopOp.getInitArgs());
+  scfLoop->setAttrs(loopOp->getAttrs());
+  scfLoop.getRegion().takeBody(loopOp.getBody());
+  scfLoop.getBody()->insertArgument(
+      /*index=*/0u, TokenType::get(loopOp.getContext()), loopOp.getLoc());
+  loopTokens[loopOp.getOperation()] = scfLoop.getControlToken();
+
+  SmallVector<Operation *> terminators;
+  scfLoop.getRegion().walk([&](Operation *op) {
+    if (!isBreakableLoopTerminator(op))
+      return;
+    if (op->getParentOfType<TestBreakableLoopOp>())
+      return;
+    terminators.push_back(op);
+  });
+
+  for (Operation *terminator : terminators)
+    if (failed(rewriteTerminator(terminator, terminatorTargets, loopTokens,
+                                 rewriter)))
+      return failure();
+
+  rewriter.replaceOp(loopOp, scfLoop.getResults());
+  return success();
+}
+
+void TestConvertBreakableLoopToSCFPass::runOnOperation() {
+  SmallVector<TestBreakableLoopOp> loops;
+  TargetMap terminatorTargets;
+  getOperation()->walk<WalkOrder::PreOrder>([&](Operation *op) {
+    if (auto loopOp = dyn_cast<TestBreakableLoopOp>(op)) {
+      loops.push_back(loopOp);
+      return;
+    }
+    if (auto breakOp = dyn_cast<TestDynamicBreakOp>(op)) {
+      appendTerminatorTargets(op, terminatorTargets[op],
+                              breakOp.getPotentialTargets());
+      return;
+    }
+    if (auto continueOp = dyn_cast<TestDynamicContinueOp>(op))
+      appendTerminatorTargets(op, terminatorTargets[op],
+                              continueOp.getPotentialTargets());
+  });
+
+  TokenMap loopTokens;
+  for (TestBreakableLoopOp loopOp : loops) {
+    if (failed(rewriteLoop(loopOp, terminatorTargets, loopTokens))) {
+      signalPassFailure();
+      return;
+    }
+  }
+}
 } // namespace
 
 //===----------------------------------------------------------------------===//
@@ -2522,6 +2733,8 @@ void registerPatternsTestPass() {
   PassRegistration<TestSelectiveReplacementPatternDriver>();
 
   PassRegistration<TestFoldTypeConvertingOp>();
+
+  PassRegistration<TestConvertBreakableLoopToSCFPass>();
 }
 } // namespace test
 } // namespace mlir
diff --git a/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir b/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir
new file mode 100644
index 0000000000000..d2a9ccfc84650
--- /dev/null
+++ b/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir
@@ -0,0 +1,145 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(test-convert-breakable-loop-to-scf,convert-scf-to-cf,canonicalize,convert-arith-to-llvm),convert-func-to-llvm,convert-cf-to-llvm,reconcile-unrealized-casts)" -o %t.mlir
+// RUN: mlir-runner %t.mlir -e immediate_continue -entry-point-result=i32 | FileCheck %s --check-prefix=IMMEDIATE
+// RUN: mlir-runner %t.mlir -e break_with_result -entry-point-result=i32 | FileCheck %s --check-prefix=BREAK
+// RUN: mlir-runner %t.mlir -e constant_outer_break -entry-point-result=i32 | FileCheck %s --check-prefix=OUTER-BREAK
+// RUN: mlir-runner %t.mlir -e constant_outer_continue -entry-point-result=i32 | FileCheck %s --check-prefix=OUTER-CONTINUE
+// RUN: mlir-runner %t.mlir -e dynamic_break_depth_1 -entry-point-result=i32 | FileCheck %s --check-prefix=DYNAMIC-ONE
+// RUN: mlir-runner %t.mlir -e dynamic_break_depth_2 -entry-point-result=i32 | FileCheck %s --check-prefix=DYNAMIC-TWO
+// RUN: mlir-runner %t.mlir -e dynamic_filtered_depth_2 -entry-point-result=i32 | FileCheck %s --check-prefix=FILTERED-TWO
+// RUN: mlir-runner %t.mlir -e dynamic_filtered_depth_3 -entry-point-result=i32 | FileCheck %s --check-prefix=FILTERED-THREE
+// XFAIL: system-aix
+
+func.func @immediate_continue() -> i32 {
+  %depth = arith.constant 1 : index
+  %c0 = arith.constant 0 : i32
+  %c1 = arith.constant 1 : i32
+  %c4 = arith.constant 4 : i32
+  %result = test.breakable_loop iter_args(%i = %c0) : i32 -> i32 {
+    %done = arith.cmpi eq, %i, %c4 : i32
+    scf.if %done {
+      test.dynamic_break %depth %i : i32
+    }
+    %next = arith.addi %i, %c1 : i32
+    test.dynamic_continue %depth %next : i32
+  }
+  return %result : i32
+}
+// IMMEDIATE: 4
+
+func.func @break_with_result() -> i32 {
+  %depth = arith.constant 1 : index
+  %value = arith.constant 17 : i32
+  %result = test.breakable_loop -> i32 {
+    test.dynamic_break %depth %value : i32
+  }
+  return %result : i32
+}
+// BREAK: 17
+
+func.func @constant_outer_break() -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %outer_value = arith.constant 42 : i32
+  %fallback = arith.constant 13 : i32
+  %result = test.breakable_loop -> i32 {
+    test.breakable_loop {
+      test.dynamic_break %outer_depth %outer_value : i32
+    }
+    test.dynamic_break %inner_depth %fallback : i32
+  }
+  return %result : i32
+}
+// OUTER-BREAK: 42
+
+func.func @constant_outer_continue() -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %c0 = arith.constant 0 : i32
+  %c1 = arith.constant 1 : i32
+  %c3 = arith.constant 3 : i32
+  %result = test.breakable_loop iter_args(%i = %c0) : i32 -> i32 {
+    %done = arith.cmpi eq, %i, %c3 : i32
+    scf.if %done {
+      test.dynamic_break %inner_depth %i : i32
+    }
+    test.breakable_loop {
+      %next = arith.addi %i, %c1 : i32
+      test.dynamic_continue %outer_depth %next : i32
+    }
+    test.dynamic_break %inner_depth %i : i32
+  }
+  return %result : i32
+}
+// OUTER-CONTINUE: 3
+
+// Distinguish the two compatible dynamic break targets at runtime. Depth 1
+// breaks the inner loop, then the outer loop adds one; depth 2 skips that outer
+// post-inner add by breaking directly to the outer loop result.
+func.func @dynamic_break(%choose_outer: i1) -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %depth = arith.select %choose_outer, %outer_depth, %inner_depth : index
+  %value = arith.constant 7 : i32
+  %one = arith.constant 1 : i32
+  %result = test.breakable_loop -> i32 {
+    %inner_result = test.breakable_loop -> i32 {
+      test.dynamic_break %depth %value : i32
+    }
+    %after_inner = arith.addi %inner_result, %one : i32
+    test.dynamic_break %inner_depth %after_inner : i32
+  }
+  return %result : i32
+}
+
+func.func @dynamic_break_depth_1() -> i32 {
+  %choose_outer = arith.constant false
+  %result = func.call @dynamic_break(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// DYNAMIC-ONE: 8
+
+func.func @dynamic_break_depth_2() -> i32 {
+  %choose_outer = arith.constant true
+  %result = func.call @dynamic_break(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// DYNAMIC-TWO: 7
+
+// The innermost f32 loop is incompatible with the i32 break payload and is
+// filtered out. Depth 2 therefore targets the middle loop and observes the
+// outer post-middle add, while depth 3 targets the outer loop directly.
+func.func @dynamic_break_filtered(%choose_outer: i1) -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %middle_depth = arith.constant 2 : index
+  %outer_depth = arith.constant 3 : index
+  %depth = arith.select %choose_outer, %outer_depth, %middle_depth : index
+  %value = arith.constant 5 : i32
+  %one = arith.constant 1 : i32
+  %f0 = arith.constant 0.000000e+00 : f32
+  %result = test.breakable_loop -> i32 {
+    %middle_result = test.breakable_loop -> i32 {
+      test.breakable_loop iter_args(%inner = %f0) : f32 {
+        test.dynamic_break %depth %value : i32
+      }
+      %after_inner = arith.addi %value, %one : i32
+      test.dynamic_break %inner_depth %after_inner : i32
+    }
+    %after_middle = arith.addi %middle_result, %one : i32
+    test.dynamic_break %inner_depth %after_middle : i32
+  }
+  return %result : i32
+}
+
+func.func @dynamic_filtered_depth_2() -> i32 {
+  %choose_outer = arith.constant false
+  %result = func.call @dynamic_break_filtered(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// FILTERED-TWO: 6
+
+func.func @dynamic_filtered_depth_3() -> i32 {
+  %choose_outer = arith.constant true
+  %result = func.call @dynamic_break_filtered(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// FILTERED-THREE: 5


From llvmlistbot at llvm.org  Tue Jul 21 06:39:58 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 06:39:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f76ae.4a342cbc.3a2f36.b3d3@mx.google.com>


https://github.com/egebeysel edited https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 06:39:59 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 06:39:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f76af.9edb5c9d.280be3.8a6b@mx.google.com>


https://github.com/egebeysel approved this pull request.

lgtm :)

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 06:40:00 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 06:40:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a5f76b0.d2cb4adc.ccd80.dee9@mx.google.com>


================

----------------
egebeysel wrote:

out of curiosity, does compressstore lower to `compact`?
https://developer.arm.com/documentation/111108/2026-03/SVE-Instructions/COMPACT--Copy-Active-vector-elements-to-lower-numbered-elements- ?

https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Tue Jul 21 06:45:35 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Tue, 21 Jul 2026 06:45:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5f77ff.13be6df8.22eac4.18cf@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/208226

>From 27327d2c28378c73a64c24621a90e176c17cc7eb Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:38:23 -0700
Subject: [PATCH 01/23] Copy pack-unpack-mmt4d.mlir from ArmSVE

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 403 ++++++++++++++++++
 1 file changed, 403 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..6192ed345debf
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,403 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimension that's scalable: N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimension that's scalable: N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Ideally, we should consider fusion opportunities by moving
+/// pack/unapack/mmt4d Ops into one function:
+///   * https://github.com/llvm/llvm-project/issues/159770
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xi32>
+  %B_empty = tensor.empty() : tensor<16x13xi32>
+
+  %c3 = arith.constant 3 : i32
+  %c4 = arith.constant 4 : i32
+  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
+  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %C = arith.constant dense<[
+    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
+    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
+    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
+    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
+    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
+    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
+    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
+  ]> : tensor<7x13xi32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
+                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+
+  return %C_matmul : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : i32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, 1]
+    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+
+  return %A_pack : tensor<1x16x8x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : i32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+
+  return %B_pack : tensor<?x16x?x1xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
+  %pad = arith.constant 0 : i32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : i32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+
+  return %C_pack : tensor<1x?x8x?xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile size is "scalable": 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [8, %vs_c8]
+    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+
+  return %C_out_unpack: tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+//  Helper methods for printing
+//===----------------------------------------------------------------------===//
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
+  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
+  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+
+  // Print the packed matrices (this is the only _visible_ part that changes
+  // when adjusting the SVE vector size).
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+
+  return %C_out_unpack : tensor<7x13xi32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
+    transform.structured.vectorize %tiled_mmt4d
+      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
+
+    // Step 3: Simplify
+    // vector.multi_reduction --> vector.contract
+    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
+    // and with the following split into parallel and reduction dims:
+    //    * parallel, parallel, reduction, parallel, parallel, reduction
+    transform.apply_patterns to %mmt4d_func {
+      transform.apply_patterns.vector.reduction_to_contract
+      // Reduce the rank of xfer ops. This transforms vector.contract to be
+      // more matmul-like and to enable the lowering to outer product Ops.
+      transform.apply_patterns.vector.transfer_permutation_patterns
+    } : !transform.op<"func.func">
+
+   // Hoisting and LICM - not strictly required
+   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
+     : (!transform.op<"func.func">) -> !transform.op<"func.func">
+   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
+     : (!transform.op<"func.func">) -> !transform.any_op
+   transform.apply_licm to %all_loops : !transform.any_op
+   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
+
+   // Simplification
+   transform.apply_patterns to %mmt4d_func_h {
+     transform.apply_patterns.vector.reduction_to_contract
+     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+     transform.apply_patterns.canonicalization
+   } : !transform.op<"func.func">
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2)
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+
+   //==========================================================================
+   // SIMPLIFY THE CONTRACT Op
+   //==========================================================================
+   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
+   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+
+   // Drop trailing unit dims (the correspondong pattern works only
+   // post-bufferization)
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // LOWER CONTRACT TO FMA
+   //==========================================================================
+   transform.apply_patterns to %contract_func {
+      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.lower_outerproduct
+   } : !transform.op<"func.func">
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+
+  transform.named_sequence @match_contract(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefI32(%ptr : tensor<*xi32>)

>From 888bff0512276f88c6c4d70768587e19790f1b58 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 09:40:44 -0700
Subject: [PATCH 02/23] Convert from I32 to F32

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 140 +++++++++---------
 1 file changed, 70 insertions(+), 70 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 6192ed345debf..1b15c964d7bd3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,9 +1,9 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -cse -canonicalize -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve"\
-// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s
 
@@ -30,7 +30,7 @@
 ///   * The matrix-multiplication dimension that's scalable: N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: N (to match data
+///   * The matrix-multiplication dimension that's scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
@@ -54,22 +54,22 @@
 //===----------------------------------------------------------------------===//
 func.func @main() {
   // Allocate and initialise the inputs
-  %A_empty = tensor.empty() : tensor<7x16xi32>
-  %B_empty = tensor.empty() : tensor<16x13xi32>
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
 
-  %c3 = arith.constant 3 : i32
-  %c4 = arith.constant 4 : i32
-  %A = linalg.fill ins(%c3 : i32) outs(%A_empty : tensor<7x16xi32>) -> tensor<7x16xi32>
-  %B = linalg.fill ins(%c4 : i32) outs(%B_empty : tensor<16x13xi32>) -> tensor<16x13xi32>
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
   %C = arith.constant dense<[
-    [ 1,  8, 15, 22, 29, 36, 43, 50, 57, 64, 71, 78, 85],
-    [ 2,  9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86],
-    [ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87],
-    [ 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88],
-    [ 5, 12, 19, 26, 33, 40, 47, 54, 61, 68, 75, 82, 89],
-    [ 6, 13, 20, 27, 34, 41, 48, 55, 62, 69, 76, 83, 90],
-    [ 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91]
-  ]> : tensor<7x13xi32>
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
 
   // VARIANT: Matrix multiplication via linalg.mmt4d
   // CHECK: Unranked Memref
@@ -80,12 +80,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xi32> to tensor<*xi32>
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "--------------------------\n"
   vector.print str "RESULT FROM linalg.mmt4d:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_mmt4d_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
 
   // VARIANT: Matrix multiplication via linalg.matmul
   // CHECK: Unranked Memref
@@ -96,12 +96,12 @@ func.func @main() {
   // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
   // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
   // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
-  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xi32>, tensor<16x13xi32>, tensor<7x13xi32>) -> tensor<7x13xi32>
-  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xi32> to tensor<*xi32>
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
   vector.print str "\n--------------------------\n"
   vector.print str "RESULT FROM linalg.matmul:\n"
   vector.print str "--------------------------\n"
-  call @printMemrefI32(%C_matmul_cast) : (tensor<*xi32>) -> ()
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -111,11 +111,11 @@ func.func @main() {
 //
 // Implements matrix-multiplication via linalg.matmul
 //===----------------------------------------------------------------------===//
-func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
-  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xi32>, tensor<16x13xi32>)
-                            outs(%C: tensor<7x13xi32>) -> tensor<7x13xi32>
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
 
-  return %C_matmul : tensor<7x13xi32>
+  return %C_matmul : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -123,17 +123,17 @@ func.func private @matmul(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xi32>
+  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
   %A_pack = linalg.pack %A
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
     inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xi32> -> tensor<1x16x8x1xi32>
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xi32>
+  return %A_pack : tensor<1x16x8x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -142,8 +142,8 @@ func.func private @pack_lhs(%A: tensor<7x16xi32>) -> tensor<1x16x8x1xi32> {
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
 // inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
-func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %vs = vector.vscale
@@ -152,15 +152,15 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
   %c13 = arith.constant 13 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xi32>
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
   %B_pack = linalg.pack %B
-     padding_value(%pad : i32)
+     padding_value(%pad : f32)
      outer_dims_perm = [1, 0]
      inner_dims_pos = [1, 0]
      inner_tiles = [%vs_c8, 1]
-     into %B_pack_empty : tensor<16x13xi32> -> tensor<?x16x?x1xi32>
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
 
-  return %B_pack : tensor<?x16x?x1xi32>
+  return %B_pack : tensor<?x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -169,8 +169,8 @@ func.func private @pack_rhs(%B: tensor<16x13xi32>) ->  tensor<?x16x?x1xi32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
-  %pad = arith.constant 0 : i32
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+  %pad = arith.constant 0.0 : f32
 
   // Compute the outer tile size.
   %c13 = arith.constant 13 : index
@@ -179,14 +179,14 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
   %vs_c8 = arith.muli %vs, %c8 : index
   %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xi32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
   %C_pack = linalg.pack %C
-    padding_value(%pad : i32)
+    padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xi32> -> tensor<1x?x8x?xi32>
+    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xi32>
+  return %C_pack : tensor<1x?x8x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -195,41 +195,41 @@ func.func private @pack_acc(%C: tensor<7x13xi32>) -> tensor<1x?x8x?xi32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xi32>) -> tensor<7x13xi32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
 
-  %C_out_empty = tensor.empty() : tensor<7x13xi32>
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xi32> -> tensor<7x13xi32>
+    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
 
-  return %C_out_unpack: tensor<7x13xi32>
+  return %C_out_unpack: tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
 //  Helper methods for printing
 //===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xi32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%A_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
+  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xi32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xi32> to tensor<*xi32>
-  call @printMemrefI32(%B_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
+  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
+  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
 
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xi32> to tensor<*xi32>
-  call @printMemrefI32(%C_pack_cast) : (tensor<*xi32>) -> ()
+func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
+  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
+  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
 
   return
 }
@@ -239,25 +239,25 @@ func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xi32>) -> () {
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @matmul_via_mmt4d(%A: tensor<7x16xi32>, %B: tensor<16x13xi32>, %C: tensor<7x13xi32>) -> tensor<7x13xi32> {
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xi32>) -> tensor<1x16x8x1xi32>
-  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xi32>) -> tensor<?x16x?x1xi32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xi32>) -> tensor<1x?x8x?xi32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // Print the packed matrices (this is the only _visible_ part that changes
   // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xi32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xi32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xi32>) -> ()
+  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
+  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
+  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xi32>, tensor<?x16x?x1xi32>) outs(%C_pack : tensor<1x?x8x?xi32>) -> tensor<1x?x8x?xi32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xi32>) -> tensor<7x13xi32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
 
-  return %C_out_unpack : tensor<7x13xi32>
+  return %C_out_unpack : tensor<7x13xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -400,4 +400,4 @@ module @transforms attributes { transform.with_named_sequence } {
 //===----------------------------------------------------------------------===//
 // Function signatures
 //===----------------------------------------------------------------------===//
-func.func private @printMemrefI32(%ptr : tensor<*xi32>)
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)

>From c7be7ca7cbc0c1687ac60556df78e7779fa80acc Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 12:25:38 -0700
Subject: [PATCH 03/23] Add lowering sequence that hoists

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 98 ++++++++-----------
 1 file changed, 39 insertions(+), 59 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 1b15c964d7bd3..f4a3650ed4ac9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -272,44 +272,44 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the N dim is scalable!)
-    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, 8, [8], 0]
+    // Tile parallel dims (note, the M, N dim is scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
-    %tiled_mmt4d, %_1:2 = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 1]
-      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
     // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
     transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, 8, [8], 1]  {assume_dynamic_dims_match_vec_sizes} : !transform.any_op
-
-    // Step 3: Simplify
-    // vector.multi_reduction --> vector.contract
-    // Generates a 6-dim vector.contract with the dim matching the original MMT4D Op
-    // and with the following split into parallel and reduction dims:
-    //    * parallel, parallel, reduction, parallel, parallel, reduction
-    transform.apply_patterns to %mmt4d_func {
+      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
       transform.apply_patterns.vector.reduction_to_contract
-      // Reduce the rank of xfer ops. This transforms vector.contract to be
-      // more matmul-like and to enable the lowering to outer product Ops.
       transform.apply_patterns.vector.transfer_permutation_patterns
-    } : !transform.op<"func.func">
-
-   // Hoisting and LICM - not strictly required
-   %mmt4d_func_h = transform.structured.hoist_redundant_vector_transfers %mmt4d_func
-     : (!transform.op<"func.func">) -> !transform.op<"func.func">
-   %all_loops = transform.structured.match interface{LoopLikeInterface} in %mmt4d_func_h
-     : (!transform.op<"func.func">) -> !transform.any_op
-   transform.apply_licm to %all_loops : !transform.any_op
-   transform.loop.hoist_loop_invariant_subsets %all_loops : !transform.any_op
-
-   // Simplification
-   transform.apply_patterns to %mmt4d_func_h {
-     transform.apply_patterns.vector.reduction_to_contract
-     transform.apply_patterns.vector.cast_away_vector_leading_one_dim
-     transform.apply_patterns.canonicalization
-   } : !transform.op<"func.func">
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
 
    //==========================================================================
    // HANDLE PACK + UNPACK
@@ -353,30 +353,16 @@ module @transforms attributes { transform.with_named_sequence } {
    //==========================================================================
    // BUFFERIZATION
    //==========================================================================
-   %bufferize = transform.bufferization.one_shot_bufferize %module
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
      {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
 
-   //==========================================================================
-   // SIMPLIFY THE CONTRACT Op
-   //==========================================================================
-   %contract = transform.collect_matching @match_contract in %bufferize : (!transform.any_op) -> (!transform.any_op)
-   %contract_func = transform.get_parent_op %contract {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
-
-   // Drop trailing unit dims (the correspondong pattern works only
-   // post-bufferization)
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.tensor.fold_tensor_subset_ops
-      transform.apply_patterns.vector.drop_inner_most_unit_dims_from_xfer_ops
-      transform.apply_patterns.canonicalization
-    } : !transform.op<"func.func">
-
-   //==========================================================================
-   // LOWER CONTRACT TO FMA
-   //==========================================================================
-   transform.apply_patterns to %contract_func {
-      transform.apply_patterns.vector.lower_contraction lowering_strategy = "outerproduct"
-      transform.apply_patterns.vector.lower_outerproduct
-   } : !transform.op<"func.func">
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
 
    transform.yield
    }
@@ -389,12 +375,6 @@ module @transforms attributes { transform.with_named_sequence } {
     transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
     transform.yield %entry : !transform.any_op
   }
-
-  transform.named_sequence @match_contract(
-      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
-    transform.match.operation_name %entry ["vector.contract"] : !transform.any_op
-    transform.yield %entry : !transform.any_op
-  }
 }
 
 //===----------------------------------------------------------------------===//

>From aef815fbba2f7b352b27ac579fc33cedc1c1d97d Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Tue, 7 Jul 2026 14:32:54 -0700
Subject: [PATCH 04/23] Removed print pack functions because they introduced
 memcpy calls that required bringing in __arm_sc_memcpy

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 ++-----------------
 1 file changed, 2 insertions(+), 31 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f4a3650ed4ac9..745c076c265f8 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -1,6 +1,7 @@
 // DEFINE: %{compile} =  mlir-opt %s \
 // DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
-// DEFINE:    -canonicalize -test-lower-to-arm-sme -test-lower-to-llvm
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
 // DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
@@ -210,30 +211,6 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   return %C_out_unpack: tensor<7x13xf32>
 }
 
-//===----------------------------------------------------------------------===//
-//  Helper methods for printing
-//===----------------------------------------------------------------------===//
-func.func private @print_pack_A(%A_pack : tensor<1x16x8x1xf32>) -> () {
-  %A_pack_cast = tensor.cast %A_pack : tensor<1x16x8x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%A_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_B(%B_pack : tensor<?x16x?x1xf32>) -> () {
-  %B_pack_cast = tensor.cast %B_pack : tensor<?x16x?x1xf32> to tensor<*xf32>
-  call @printMemrefF32(%B_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
-func.func private @print_pack_C(%C_pack : tensor<1x?x8x?xf32>) -> () {
-  %C_pack_cast = tensor.cast %C_pack : tensor<1x?x8x?xf32> to tensor<*xf32>
-  call @printMemrefF32(%C_pack_cast) : (tensor<*xf32>) -> ()
-
-  return
-}
-
 //===----------------------------------------------------------------------===//
 // @matmul_via_mmt4d
 //
@@ -245,12 +222,6 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
-  // Print the packed matrices (this is the only _visible_ part that changes
-  // when adjusting the SVE vector size).
-  func.call @print_pack_A(%A_pack) : (tensor<1x16x8x1xf32>) -> ()
-  func.call @print_pack_B(%B_pack) : (tensor<?x16x?x1xf32>) -> ()
-  func.call @print_pack_C(%C_pack) : (tensor<1x?x8x?xf32>) -> ()
-
   // MMT4D
   %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 

>From 9d77231066583b8085c8060a77009cb3f9b74bed Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:31 -0400
Subject: [PATCH 05/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 745c076c265f8..76cded79bcce7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -28,7 +28,7 @@
 ///
 /// NOTES ON IMPLEMENTATION
 /// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
-///   * The matrix-multiplication dimension that's scalable: N.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
 ///   * The matrix-multiplication dimension that's scalable: M, N (to match data

>From a15409a499e9b76eb6a9bcfe013bd546607e4f94 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 9 Jul 2026 10:13:47 -0400
Subject: [PATCH 06/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Co-authored-by: Ege Beysel <beyselege at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 76cded79bcce7..b71995f9b49a7 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,7 +40,7 @@
 ///    the corresponding lowering and lowering configs.
 /// 
 /// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unapack/mmt4d Ops into one function:
+/// pack/unpack/mmt4d ops into one function:
 ///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751

>From 8e4bca2ca57ce5c8ced18691a95f3468ceb23445 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:18:49 -0700
Subject: [PATCH 07/23] Remove +sve flag

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b71995f9b49a7..893710fd800d2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -3,7 +3,7 @@
 // DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
 // DEFINE:    -test-lower-to-llvm
 // DEFINE: %{entry_point} = main
-// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sve,+sme"\
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
 // DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
 
 // RUN: %{compile} | %{run} | FileCheck %s

>From 457f16cf5289905ee0b49c287480cd1f7ba3bd55 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:22:33 -0700
Subject: [PATCH 08/23] Add linalg.unpack comment

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 893710fd800d2..d6a6bedc75a10 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -19,6 +19,7 @@
 ///   B_pack = linalg.pack B
 ///   C_pack = linalg.pack C
 ///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
 ///
 /// is equivalent to:
 ///

>From d608b7d4f6291ff3f435d488f82f89da912f25bf Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:35:29 -0700
Subject: [PATCH 09/23] Added create_named_contractions. Removing 2nd
 reduction_to_contract causes test to fail

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir        | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index d6a6bedc75a10..32984a04f6a99 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -252,9 +252,8 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
-    // TODO: Lower directly to named contractions: https://github.com/llvm/llvm-project/issues/159749
-    transform.structured.vectorize %tiled_mmt4d
-      vector_sizes  [1, 1, 1, [8], [8], 1]  : !transform.any_op
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+      : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
     transform.apply_patterns to %loop_k {
@@ -270,7 +269,6 @@ module @transforms attributes { transform.with_named_sequence } {
     %func_pre = transform.structured.match ops{["func.func"]} in %module
       : (!transform.any_op) -> !transform.any_op
     transform.apply_patterns to %func_pre {
-      transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.transfer_permutation_patterns
       transform.apply_patterns.canonicalization
     } : !transform.any_op

>From c1290ec4e38aa1c53dd4285ace01665dce5ac65c Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 07:47:50 -0700
Subject: [PATCH 10/23] Fix some comments

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir          | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 32984a04f6a99..06dae3ee89042 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -244,14 +244,14 @@ module @transforms attributes { transform.with_named_sequence } {
     %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
-    // Tile parallel dims (note, the M, N dim is scalable!)
+    // Tile parallel dims (note, the M, N dims are scalable!)
     %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
     // Tile reduction dims
     %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
-    // Step 2: Vectorize linalg.mmt4d (note, the M, N dim is scalable!)
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
     transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
       : !transform.any_op
 

>From 5e79874fa5aceaa9a854ce19646a871213ae0af0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 09:35:52 -0700
Subject: [PATCH 11/23] Make A_pack inner_tiles [%vscale_c8, 1]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir   | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 06dae3ee89042..7b8b3b4c74ff1 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -125,17 +125,21 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 //
 // Implements matrix-multiplication via linalg.mmt4d
 //===----------------------------------------------------------------------===//
-func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x8x1xf32> {
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32
 
-  %A_pack_empty = tensor.empty() : tensor<1x16x8x1xf32>
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
   %A_pack = linalg.pack %A
     padding_value(%pad : f32)
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, 1]
-    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x8x1xf32>
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
 
-  return %A_pack : tensor<1x16x8x1xf32>
+  return %A_pack : tensor<1x16x?x1xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -219,12 +223,12 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
 //===----------------------------------------------------------------------===//
 func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
   // Pack input matrices
-  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x8x1xf32>
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
   %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x8x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
 
   // Unpack the output
   %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>

>From ff4a55c252b87045781e1a375221091b43c4780f Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Thu, 9 Jul 2026 10:12:37 -0700
Subject: [PATCH 12/23] Make C_pack and C_out_unpack's inner tiles [%vs_c8,
 %vs_c8]

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 33 +++++++++++--------
 1 file changed, 19 insertions(+), 14 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 7b8b3b4c74ff1..f0348c1679723 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -175,24 +175,26 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32
 
-  // Compute the outer tile size.
+  // Compute the outer tile sizes.
+  %c7 = arith.constant 7 : index
   %c13 = arith.constant 13 : index
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
-  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
+  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<1x?x8x?xf32>
+  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
   %C_pack = linalg.pack %C
     padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x8x?xf32>
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
 
-  return %C_pack : tensor<1x?x8x?xf32>
+  return %C_pack : tensor<?x?x?x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -201,7 +203,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x8x?xf32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile size is "scalable": 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32> {
+func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
@@ -210,8 +212,8 @@ func.func private @unpack_acc(%C_packed: tensor<1x?x8x?xf32>) -> tensor<7x13xf32
   %C_out_unpack = linalg.unpack %C_packed
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [8, %vs_c8]
-    into %C_out_empty : tensor<1x?x8x?xf32> -> tensor<7x13xf32>
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
 
   return %C_out_unpack: tensor<7x13xf32>
 }
@@ -225,13 +227,13 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   // Pack input matrices
   %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x8x?xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x8x?xf32>) -> tensor<1x?x8x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x8x?xf32>) -> tensor<7x13xf32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
 
   return %C_out_unpack : tensor<7x13xf32>
 }
@@ -297,8 +299,11 @@ module @transforms attributes { transform.with_named_sequence } {
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
-    //    and other lower-level Ops (see step 2)
-    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [8, 1]
+    //    and other lower-level Ops (see step 2). Tile by [8*vscale, 8*vscale]
+    //    (scalable, equal to the inner tile sizes) so the tiled outer dims are
+    //    statically 1 and DecomposeOuterUnitDimsUnPackOp can fire.
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [[8], [8]]
+        inner_tile_alignments = [Equal, Equal]
        : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
 
     // 2.1. Decompose tiled PackOp into lower-level Ops + simplify

>From d3cf3bb016b8edc571a7e56209c722f9efb619b0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:38:23 -0700
Subject: [PATCH 13/23] Remove dead %mmt4d_func

---
 .../Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index f0348c1679723..db15eff92cad3 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -247,7 +247,6 @@ module @transforms attributes { transform.with_named_sequence } {
     // HANDLE MMT4D
     //==========================================================================
     %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
-    %mmt4d_func = transform.get_parent_op %mmt4d {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
 
     // Step 1: Tile
     // Tile parallel dims (note, the M, N dims are scalable!)

>From bb0524298418dfdfe92a0a1b9a87a8735caa9b42 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 06:41:16 -0700
Subject: [PATCH 14/23] Fix comment for pack_lhs

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir         | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index db15eff92cad3..b599b95cd19fa 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -121,9 +121,10 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 }
 
 //===----------------------------------------------------------------------===//
-// @matmul_via_mmt4d
+// @pack_lhs
 //
-// Implements matrix-multiplication via linalg.mmt4d
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From 6bbc2112fef26cf55990ec419c54e97955ed7bc4 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 08:54:55 -0700
Subject: [PATCH 15/23] Add flag assume_dynamic_dims_match_vec_sizes when
 vectorizing

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index b599b95cd19fa..0db52af86c17e 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -258,7 +258,7 @@ module @transforms attributes { transform.with_named_sequence } {
       : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
 
     // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
-    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {create_named_contraction}
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
       : !transform.any_op
 
     // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask

>From bee3857e57916b02fe0693ddcc429b6985001734 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Mon, 13 Jul 2026 08:58:12 -0700
Subject: [PATCH 16/23] Remove TODO about fusion

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir           | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 0db52af86c17e..a520816bce2f0 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -40,9 +40,6 @@
 /// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
 ///    the corresponding lowering and lowering configs.
 /// 
-/// TODO: Ideally, we should consider fusion opportunities by moving
-/// pack/unpack/mmt4d ops into one function:
-///   * https://github.com/llvm/llvm-project/issues/159770
 /// TODO: Vectorize linalg.pack + linalg.unpack:
 ///   * https://github.com/llvm/llvm-project/issues/159751
 //===----------------------------------------------------------------------===//

>From 19b681bc5ccaf1ebb04aafa23089126556cf634d Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:19:44 -0400
Subject: [PATCH 17/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index a520816bce2f0..a516148c80a7e 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -32,7 +32,7 @@
 ///   * The matrix-multiplication dimensions that are scalable: M and N.
 ///
 /// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
-///   * The matrix-multiplication dimension that's scalable: M, N (to match data
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
 ///     tiling configuration).
 ///
 /// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.

>From 9a5f92cacfa68e7a5bc30e9022a1bd51086b34f1 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:20:26 -0400
Subject: [PATCH 18/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index a516148c80a7e..7923ff09aa9d9 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -171,7 +171,7 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // @pack_acc
 //
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
-// The inner tile size is "scalable": 8 * vscale
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
 func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32

>From 5a3b8546c149ec007a5aa673b477d0937b1a0ffe Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:20:44 -0400
Subject: [PATCH 19/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 7923ff09aa9d9..78bcd5cf63ce5 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -199,7 +199,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
 // @unpack_acc
 //
 // Implements unpacking for the C matrix (accumulator) in matrix
-// multiplication. The inner tile size is "scalable": 8 * vscale
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
 func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale

>From 70243043c22de9587bbf2f9e506f9ded83b71169 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:21:01 -0400
Subject: [PATCH 20/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 78bcd5cf63ce5..34abbdf31ab47 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -121,7 +121,7 @@ func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tenso
 // @pack_lhs
 //
 // Implements packing for the A matrix (LHS) in matrix multiplication. The
-// inner tile size is "scalable": 8 * vscale.
+// inner tile size for dim M is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From 45937bcf98c6ab252f023657ad2a2eadafbca90e Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Tue, 14 Jul 2026 16:21:19 -0400
Subject: [PATCH 21/23] Update
 mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>
---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir            | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 34abbdf31ab47..27322db143b30 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -144,7 +144,7 @@ func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
 // @pack_rhs
 //
 // Implements packing for the B matrix (RHS) in matrix multiplication. The
-// inner tile size is "scalable": 8 * vscale.
+// inner tile size for dim N is "scalable": 8 * vscale.
 //===----------------------------------------------------------------------===//
 func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
   %pad = arith.constant 0.0 : f32

>From 6466c104f84c5cf67df73361dce2ab1afc055cf8 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Fri, 17 Jul 2026 12:37:19 -0700
Subject: [PATCH 22/23] Add comment about create_named_contraction

---
 .../Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir           | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index 27322db143b30..ed8c12992b3a2 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -276,6 +276,9 @@ module @transforms attributes { transform.with_named_sequence } {
       transform.apply_patterns.canonicalization
     } : !transform.any_op
     transform.apply_patterns to %func_pre {
+      // TODO: `create_named_contraction` above lowers directly to a named contraction. If we want to remove
+      // `transform.apply_patterns.vector.reduction_to_contract` below, `CombineContractBroadcastMask`
+      // will have to be moved to a dedicated TD Op.
       transform.apply_patterns.vector.reduction_to_contract
       transform.apply_patterns.vector.cast_away_vector_leading_one_dim
       transform.apply_patterns.vector.lower_contraction

>From 55a72ad0312bab0f6f6caafad91fb19c1f38c3ef Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at quicinc.com>
Date: Fri, 17 Jul 2026 13:55:58 -0700
Subject: [PATCH 23/23] Fix up dims of pack_lhs, pack_rhs, pack_acc, unpack_acc

---
 .../Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir  | 24 +++++++++----------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
index ed8c12992b3a2..73e286f960008 100644
--- a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -173,26 +173,24 @@ func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
 // Implements packing for the C matrix (accumulator) in matrix multiplication.
 // The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
   %pad = arith.constant 0.0 : f32
 
-  // Compute the outer tile sizes.
-  %c7 = arith.constant 7 : index
+  // Compute the outer tile size.
   %c13 = arith.constant 13 : index
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
-  %outer_tile_size_0 = arith.ceildivui %c7, %vs_c8 : index
-  %outer_tile_size_1 = arith.ceildivui %c13, %vs_c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
 
-  %C_pack_empty = tensor.empty(%outer_tile_size_0, %outer_tile_size_1, %vs_c8, %vs_c8) : tensor<?x?x?x?xf32>
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
   %C_pack = linalg.pack %C
     padding_value(%pad : f32)
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
-    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<?x?x?x?xf32>
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
 
-  return %C_pack : tensor<?x?x?x?xf32>
+  return %C_pack : tensor<1x?x?x?xf32>
 }
 
 //===----------------------------------------------------------------------===//
@@ -201,7 +199,7 @@ func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<?x?x?x?xf32> {
 // Implements unpacking for the C matrix (accumulator) in matrix
 // multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
 //===----------------------------------------------------------------------===//
-func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32> {
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
   %vs = vector.vscale
   %c8 = arith.constant 8 : index
   %vs_c8 = arith.muli %vs, %c8 : index
@@ -211,7 +209,7 @@ func.func private @unpack_acc(%C_packed: tensor<?x?x?x?xf32>) -> tensor<7x13xf32
     outer_dims_perm = [0, 1]
     inner_dims_pos = [0, 1]
     inner_tiles = [%vs_c8, %vs_c8]
-    into %C_out_empty : tensor<?x?x?x?xf32> -> tensor<7x13xf32>
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
 
   return %C_out_unpack: tensor<7x13xf32>
 }
@@ -225,13 +223,13 @@ func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>,
   // Pack input matrices
   %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
   %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
-  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<?x?x?x?xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
 
   // MMT4D
-  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<?x?x?x?xf32>) -> tensor<?x?x?x?xf32>
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
 
   // Unpack the output
-  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<?x?x?x?xf32>) -> tensor<7x13xf32>
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
 
   return %C_out_unpack : tensor<7x13xf32>
 }


From llvmlistbot at llvm.org  Tue Jul 21 06:50:30 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 06:50:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5f7926.1728eaf3.2a2f6.a9bd@mx.google.com>


https://github.com/FedericoBruzzone commented:

LGTM @steplong, thanks a lot!

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 06:56:03 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 21 Jul 2026 06:56:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f7a73.1905e68e.30d957.0be4@mx.google.com>


================

----------------
nujaa wrote:

`linalg.broadcast` has a mandatory `dimension` attribute which must satisfy `inputRank + dimensions.size() == initRank` which suggests calling a scalar broadcast with something like `  %0 = linalg.broadcast ins(%arg0 : tensor<i32>) outs(%arg1 : tensor<32x2xi32>) dimensions = [0, 1]` but this is considered incorrect mlir input too. I tried   `%0 = linalg.broadcast ins(%arg0 : tensor<i32>) outs(%arg1 : tensor<32x2xi32>) dimensions = [0, 0]` (both dimensions as zeros) which segfaults. I will look into it and submit a patch to support this kind of linalg.broadcast.


https://github.com/llvm/llvm-project/pull/210738

From llvmlistbot at llvm.org  Tue Jul 21 07:01:15 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Tue, 21 Jul 2026 07:01:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast (PR #210738)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210738@github.com>
Message-ID: <6a5f7bab.e5907290.281687.7a0f@mx.google.com>


https://github.com/nujaa updated https://github.com/llvm/llvm-project/pull/210738

>From 73992bf4e13e40d5c0c33df2f32f2c246ba6bf28 Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Mon, 20 Jul 2026 23:07:09 +0800
Subject: [PATCH 1/2] [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast

---
 .../TransformOps/LinalgTransformOps.cpp       |  2 +-
 .../Dialect/Linalg/flatten-elementwise.mlir   | 28 +++++++++++++++++++
 .../Dialect/Linalg/flatten-unsupported.mlir   | 12 ++------
 3 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index 42ce13b6f214c..01957c91497c8 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4311,7 +4311,7 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
            << "only elementwise flattening is supported";
 
   if (!llvm::all_of(target.getIndexingMapsArray(), [](AffineMap m) {
-        return m.isProjectedPermutation(/*allowZeroInResults=*/false);
+        return m.isPermutation() || m.getNumResults() == 0;
       })) {
     results.push_back(target);
     return mlir::emitSilenceableFailure(target->getLoc())
diff --git a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
index f9f8f9e2b8b44..ca06062f61840 100644
--- a/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-elementwise.mlir
@@ -43,6 +43,34 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
+// CHECK-LABEL: func.func @broadcast_rank0_tensor(
+// CHECK-SAME:                         %[[ARG0:.*]]: tensor<i32>,
+// CHECK-SAME:                         %[[ARG1:.*]]: tensor<32x2xi32>
+// CHECK-NEXT:    %[[FLATTENED:.*]] = tensor.collapse_shape %[[ARG1]] {{\[}}[0, 1]]
+// CHECK-NEXT:    %[[FLATTENED_RESULT:.*]] = linalg.generic {{.*}} ins(%[[ARG0]] : tensor<i32>) outs(%[[FLATTENED]] : tensor<64xi32>)
+// CHECK:         %[[RESULT:.*]] = tensor.expand_shape %[[FLATTENED_RESULT]] {{\[}}[0, 1]] output_shape [32, 2] : tensor<64xi32> into tensor<32x2xi32>
+#map0 = affine_map<(d0, d1) -> ()>
+#map1 = affine_map<(d0, d1) -> (d0, d1)>
+
+func.func @broadcast_rank0_tensor(%arg0: tensor<i32>, %arg1: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %0 = linalg.generic {indexing_maps = [#map0, #map1], iterator_types = ["parallel", "parallel"]} ins(%arg0 : tensor<i32>) outs(%arg1 : tensor<32x2xi32>) {
+    ^bb0(%in: i32, %out: i32):
+      linalg.yield %in : i32
+  } -> tensor<32x2xi32>
+  return %0 : tensor<32x2xi32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %0 = transform.structured.match interface{LinalgOp} in %arg1 : (!transform.any_op) -> !transform.any_op
+    %flattened = transform.structured.flatten_elementwise %0
+      : (!transform.any_op) -> !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
 // CHECK-LABEL: func.func @map_memref(
 // CHECK-SAME:                 %[[ARG0:[a-zA-Z0-9_]*]]: memref<32x7xf32>
 // CHECK-SAME:                 %[[ARG1:[a-zA-Z0-9_]*]]: memref<32x7xf32>
diff --git a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
index f0b8b8b153bbb..7ecafcb6fdd9e 100644
--- a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
@@ -56,16 +56,10 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
-#map1 = affine_map<(d0, d1, d2) -> (d1, 0)>
-#map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
-
-func.func @unsupported_rank_expanding_broadcasting_elementwise(%arg0: tensor<2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
+func.func @unsupported_dim_expanding_broadcast(%arg0: tensor<64xi16>, %arg1: tensor<32x64xi16>) -> tensor<32x64xi16> {
   // expected-error @below {{operators with broadcasting semantics are not supported}}
-  %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
-    ^bb0(%in: i32, %out: i32):
-      linalg.yield %in : i32
-  } -> tensor<32x2x2xi32>
-  return %0 : tensor<32x2x2xi32>
+  %broadcasted = linalg.broadcast ins(%arg0 : tensor<64xi16>) outs(%arg1 : tensor<32x64xi16>) dimensions = [0] 
+  return %broadcasted : tensor<32x64xi16>
 }
 
 module attributes {transform.with_named_sequence} {

>From 851168bcdf1be59fc0bc1cab6075f7ef526d64fd Mon Sep 17 00:00:00 2001
From: Hugo <hugo.trachino at huawei.com>
Date: Tue, 21 Jul 2026 21:59:35 +0800
Subject: [PATCH 2/2] fixup! [MLIR][Linalg] Fix FlattenElementwiseOp for
 linalg.broadcast

---
 mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp | 2 +-
 mlir/test/Dialect/Linalg/flatten-unsupported.mlir           | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
index 01957c91497c8..8c5bd50bd6f8b 100644
--- a/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
+++ b/mlir/lib/Dialect/Linalg/TransformOps/LinalgTransformOps.cpp
@@ -4315,7 +4315,7 @@ DiagnosedSilenceableFailure transform::FlattenElementwiseLinalgOp::applyToOne(
       })) {
     results.push_back(target);
     return mlir::emitSilenceableFailure(target->getLoc())
-           << "operators with broadcasting semantics are not supported";
+           << "broadcasting of non scalar operands is not supported";
   }
 
   // If rank <= 1, do nothing
diff --git a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
index 7ecafcb6fdd9e..492211a5e5141 100644
--- a/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
+++ b/mlir/test/Dialect/Linalg/flatten-unsupported.mlir
@@ -37,7 +37,7 @@ module attributes {transform.with_named_sequence} {
 #map2 = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 
 func.func @unsupported_broadcasting_elementwise(%arg0: tensor<1x2x1xi32>, %arg1: tensor<32x2x2xi32>) -> tensor<32x2x2xi32> {
-  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  // expected-error @below {{broadcasting of non scalar operands is not supported}}
   %0 = linalg.generic {indexing_maps = [#map1, #map2], iterator_types = ["parallel", "parallel", "parallel"]} ins(%arg0 : tensor<1x2x1xi32>) outs(%arg1 : tensor<32x2x2xi32>) {
     ^bb0(%in: i32, %out: i32):
       linalg.yield %in : i32
@@ -57,7 +57,7 @@ module attributes {transform.with_named_sequence} {
 // -----
 
 func.func @unsupported_dim_expanding_broadcast(%arg0: tensor<64xi16>, %arg1: tensor<32x64xi16>) -> tensor<32x64xi16> {
-  // expected-error @below {{operators with broadcasting semantics are not supported}}
+  // expected-error @below {{broadcasting of non scalar operands is not supported}}
   %broadcasted = linalg.broadcast ins(%arg0 : tensor<64xi16>) outs(%arg1 : tensor<32x64xi16>) dimensions = [0] 
   return %broadcasted : tensor<32x64xi16>
 }


From llvmlistbot at llvm.org  Tue Jul 21 07:01:25 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:01:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
Message-ID: <llvm.org/llvm/llvm-project/pull/211004@github.com>


https://github.com/FedericoBruzzone created https://github.com/llvm/llvm-project/pull/211004

`SparseVectorization` assumes direct loop accesses (`a[lo:hi]`) are contiguous and vectorizes them with `vector.maskedload/maskedstore`. This is false for `sparse_tensor.coordinates` of a level inside a trailing AoS COO region, whose buffer is interleaved with other levels: a silent miscompile.

The true stride is already known from the tensor's encoding, even though the memref type is still dynamic at this point. Use it to fall back to a scalar loop when the stride is provably non-unit.

>From ee815c22e10991ea39025101b8bc71d00558f0a4 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:28 +0200
Subject: [PATCH] [mlir][sparse] Avoid vectorizing non-contiguous COO
 coordinate loads

SparseVectorization emits vector.maskedload/maskedstore for direct
loop accesses, assuming consecutive loop indices map to consecutive
memory. This is false for sparse_tensor.coordinates of a level inside
a trailing AoS COO region, whose buffer is interleaved with other
levels -- a real, silent miscompile.

Fall back to a scalar load/store when the stride is statically known
to be non-unit, including cases only derivable from the sparse
tensor's encoding before sparse-tensor-codegen materializes the
concrete stride.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../Transforms/SparseVectorization.cpp        | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();


From llvmlistbot at llvm.org  Tue Jul 21 07:03:49 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:03:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a5f7c45.c5504292.78ef3.b5ea@mx.google.com>


FedericoBruzzone wrote:

@banach-space I isolated the `SparseVectorization` miscompilation previously in #210952.

https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Tue Jul 21 07:04:36 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:04:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f7c74.1e1ac41d.3969c3.31ce@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/210952

>From ee815c22e10991ea39025101b8bc71d00558f0a4 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:28 +0200
Subject: [PATCH 1/2] [mlir][sparse] Avoid vectorizing non-contiguous COO
 coordinate loads

SparseVectorization emits vector.maskedload/maskedstore for direct
loop accesses, assuming consecutive loop indices map to consecutive
memory. This is false for sparse_tensor.coordinates of a level inside
a trailing AoS COO region, whose buffer is interleaved with other
levels -- a real, silent miscompile.

Fall back to a scalar load/store when the stride is statically known
to be non-unit, including cases only derivable from the sparse
tensor's encoding before sparse-tensor-codegen materializes the
concrete stride.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../Transforms/SparseVectorization.cpp        | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();

>From 982fc5dcf309734db2d09dc4bf16cb5f313f5630 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:54 +0200
Subject: [PATCH 2/2] [mlir][vector] Reject non-unit strides on
 masked/expand/compress ops

vector.maskedload/maskedstore/expandload/compressstore lower to LLVM
masked intrinsics that read/write N *consecutive* elements from a
single pointer, but none of them verified the memref's minor-dim
stride, so e.g. `strided<[2]>` verified successfully and silently
miscompiled.

Reject statically-known non-unit strides; a dynamic stride is still
accepted (see linked issue for discussion on why this differs from
vector.load/store). expandload/compressstore also gain the
negative-stride check vector.load/store already have.

Depends on the SparseVectorization fix in the preceding commit: the
sparsifier's vectorizer previously relied on maskedload having no
stride check to (unsafely) vectorize non-contiguous COO coordinate
buffers.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 51 ++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir    | 77 ++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index f37083803a2a1..8b594c378912e 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
+/// statically known non-unit stride; a dynamic (not provably unit) stride
+/// is accepted.
+///
+/// This is more permissive than vector.load/store's stride check: ops such
+/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
+/// also used on memrefs whose most minor dimension is contiguous at runtime
+/// but not provable as such at the type level (e.g., buffers produced by
+/// the sparsifier).
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6326,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6395,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6681,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6702,6 +6744,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..9ec17024bc1d3 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1742,6 +1760,24 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1796,6 +1832,24 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
@@ -2179,6 +2233,19 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
+// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
+// (unprovable) stride is rejected here too: vector.load/store require a
+// statically known unit stride, with no exception for strides that merely
+// aren't provably non-unit.
+func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
+  %0 = vector.load %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.store
 //===----------------------------------------------------------------------===//
@@ -2215,6 +2282,16 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
+// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
+// unlike vector.maskedstore.
+func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
+  // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
+  vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 func.func @store_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %val: vector<4xf32>) {
   // expected-error @+2 {{'vector.store' op memref strides must be non-negative}}
   %c0 = arith.constant 0 : index


From llvmlistbot at llvm.org  Tue Jul 21 07:07:23 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:07:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f7d1b.507fc713.239434.b6c3@mx.google.com>


================
@@ -6190,6 +6190,32 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Unlike vector.load/store, ops such as vector.maskedload/maskedstore and
+/// vector.expandload/compressstore are also used on memrefs whose most minor
+/// dimension has a dynamic stride that is contiguous at runtime but not
+/// provable as such at the type level (e.g., buffers produced by the
+/// sparsifier).
----------------
FedericoBruzzone wrote:

Thanks @banach-space. I moved it to the other PR.

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 07:13:39 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:13:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f7e93.e624eb8d.4a8a7.8063@mx.google.com>


FedericoBruzzone wrote:

> > Note: This is intentionally more permissive than vector.load/vector.store, the sparsifier legitimately produces masked loads on memrefs whose last-dim stride is contiguous by construction but not provable at the type level (see below).
> 
> This a bit confusing, TBH. If dynamic stride is considered as "unit" in some scenarios, then it should be considered as "unit" everywhere. Special-casing like this can be quite problematic. For now, I would at least:
> 
> * Add tests for `vector.load` + `vector.store` with dynamic strides (so that the difference in behaviour is documented through tests)
> * Update your table from this [comment](https://github.com/llvm/llvm-project/pull/210952#issuecomment-5033367754) accordingly (it's very useful, thank you!).
> 
> Also, your summary is quite long and verbose for a relatively small change like this. All in all, the discussion is relevant, but please remember that the summary will end-up as a commit message. A commit message should provide a _brief_ overview of _what_ and _why_. Here are some additional tips https://chris.beams.io/git-commit (we [reccomend](https://mlir.llvm.org/getting_started/Contributing/) that all contributors to MLIR follow it source in MLIR).
> 
> I suggest that you open a GitHub issue for the wider discussion and to track all the relevant PRs. The summary for this change should be much shorter.`
> 
> Last, but not least, thank you so much for all the fixes, that is greatly appreciated 🙏🏻
> 
> **EDIT** Btw, this feels like two independent changes. Could you split it?

Thanks @banach-space! 
I sincerely appreciate these suggestions and the time you are dedicating to guiding me (and the project in general) 🫶.

So,
- Split done: SparseVectorization fix is now standalone in #211004 (this PR is stacked on it).
- Added `load_dynamic_stride`/`store_dynamic_stride` to `invalid.mlir` documenting that `load/store` reject `[?]`, unlike the `masked/expand/compress` ops. (I hope this is what you meant. If not, let me know and I'll make the changes.)

Thanks for the thorough review!

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 07:16:55 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:16:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops and fix `SparseVectorization` miscompilation
 (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f7f57.481afadf.9e047.856c@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 07:20:03 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:20:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f8013.f1c1923d.2d2b59.cf42@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 07:31:23 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 07:31:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5f82bb.f1c1923d.2d2b59.fef2@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Tue Jul 21 07:41:39 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Tue, 21 Jul 2026 07:41:39 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
Message-ID: <llvm.org/llvm/llvm-project/pull/211011@github.com>


https://github.com/rscottmanley created https://github.com/llvm/llvm-project/pull/211011

The `acc.firstprivate_save` operation is intended to provide a temporary link from the compute region's private variable and the firstprivate map init op. This facilitates analysis in cases where the compiler may determine that a firstprivate should be converted to a copy and can be safely erased when that analysis is complete.

>From c2ae0042cab5115e802afcfb98f58ebaf37741c4 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Fri, 17 Jul 2026 12:35:40 -0700
Subject: [PATCH] [OpenACC] add acc::FirstprivateSaveOp

The `acc.firstprivate_save` operation is intended to provide a temporary
link from the compute region's private variable and the firstprivate map
init op. This facilitates analysis in cases where the compiler may
determine that a firstprivate should be converted to a copy and can be
safely erased when that analysis is complete.
---
 ...cc-recipe-materialization-firstprivate.fir | 34 ++++++++++++-------
 .../mlir/Dialect/OpenACC/OpenACCCGOps.td      | 22 ++++++++++++
 .../Transforms/ACCRecipeMaterialization.cpp   | 12 +++++++
 ...c-recipe-materialization-firstprivate.mlir | 20 ++++++-----
 4 files changed, 67 insertions(+), 21 deletions(-)

diff --git a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
index a2f1dbc6e6565..998740f0e8237 100644
--- a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
+++ b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
@@ -39,18 +39,26 @@ module {
 // and an alloca (as per the recipe) inside the region.
 // Then ensure that all uses are of the private alloca.
 // CHECK-LABEL: func.func @firstpriv
-// CHECK: acc.parallel
-// CHECK: %[[ALLOCA:.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
-// CHECK: %[[FIRSTPRIVLOAD:.*]] = fir.load %{{.*}} : !fir.ref<i32>
-// CHECK: fir.store %[[FIRSTPRIVLOAD]] to %[[ALLOCA]] : !fir.ref<i32>
-// CHECK: %[[ALLOCALOAD:.*]] = fir.load %[[ALLOCA]] : !fir.ref<i32>
-// CHECK: %[[ADDI:.*]] = arith.addi %[[ALLOCALOAD]], %c1{{.*}} : i32
-// CHECK: fir.store %[[ADDI]] to %[[ALLOCA]] : !fir.ref<i32>
+// CHECK:       [[DECL:%.*]] = fir.declare
+// CHECK:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[DECL]] : !fir.ref<i32>) -> !fir.ref<i32> {implicit = true, name = "t"}
+// CHECK:       acc.parallel {
+// CHECK:       [[FIRSTPRIV:%.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = fir.load [[MAP]] : !fir.ref<i32>
+// CHECK:       fir.store [[FIRSTPRIVLOAD]] to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = fir.load [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK:       [[ADDI:%.*]] = arith.addi [[FIRSTPRIVLOAD]], %c1{{.*}} : i32
+// CHECK:       fir.store [[ADDI]] to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK-NEXT:  [[VALUELOAD:%.*]] = fir.load
+// CHECK-NEXT:  acc.firstprivate_save [[VALUELOAD]] to [[MAP]] : i32 -> !fir.ref<i32>
 
+// With -acc-optimize-firstprivate-map, the map is retained because of the
+// firstprivate_save live-out.
 // CHECK-MAP-LABEL: func.func @firstpriv
-// CHECK-MAP: fir.load {{.*}} : !fir.ref<i32>
-// CHECK-MAP: acc.parallel {
-// CHECK-MAP-NOT: acc.firstprivate_map
-// CHECK-MAP: fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
-// CHECK-MAP: fir.store {{.*}} to {{.*}} : !fir.ref<i32>
-// CHECK-MAP: arith.addi {{.*}} %c1
+// CHECK-MAP:       [[DECL:%.*]] = fir.declare
+// CHECK-MAP:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[DECL]] : !fir.ref<i32>) -> !fir.ref<i32> {implicit = true, name = "t"}
+// CHECK-MAP:       acc.parallel {
+// CHECK-MAP:       [[FIRSTPRIV:%.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
+// CHECK-MAP:       fir.load [[MAP]] : !fir.ref<i32>
+// CHECK-MAP:       fir.store {{.*}} to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK-MAP:       arith.addi {{.*}} %c1
+// CHECK-MAP:       acc.firstprivate_save {{.*}} to [[MAP]] : i32 -> !fir.ref<i32>
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index 730ea1f73a3ac..4b1895d4adabc 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -302,6 +302,28 @@ def OpenACC_FirstprivateMapInitialOp
   let extraClassDeclaration = extraClassDeclarationBase;
 }
 
+//===----------------------------------------------------------------------===//
+// acc.firstprivate_save
+//===----------------------------------------------------------------------===//
+
+def OpenACC_FirstprivateSaveOp : OpenACC_Op<"firstprivate_save", []> {
+ let summary = "\"Saves\" a value to a firstprivate variable";
+ let description = [{
+    The `acc.firstprivate_save` operation is intended to provide a temporary
+    link from the compute region's private variable and the firstprivate map
+    init op. This facilitates analysis in cases where the compiler may
+    determine that a firstprivate should be converted to a copy and can be
+    safely erased when that analysis is complete.
+  }];
+  let arguments = (ins AnyType:$value,
+                       Arg<OpenACC_PointerLikeType,
+                           "FirstprivateMapInitOp",
+                           [MemWrite]>:$memref);
+  let assemblyFormat = [{
+    $value `to` $memref `:` type($value) `->` type($memref) attr-dict
+  }];
+}
+
 //===----------------------------------------------------------------------===//
 // acc.privatize
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index 70aacd251c329..712a7b82a02bb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -305,6 +305,7 @@ LogicalResult ACCRecipeMaterialization::materialize(
     auto [results, ip] = acc::cloneACCRegionInto(
         &initRegion, block, block->begin(), mapping, {accPtr});
     assert(results.size() == 1 && "expected single result from init region");
+    Value alloca = results[0];
     saveVarName(op.getAccVar(), results[0]);
     resolveVarNamePlaceholders(block, ip, acc::getVariableName(op.getAccVar()));
     // We want the copy to store the origPtr to private
@@ -318,6 +319,17 @@ LogicalResult ACCRecipeMaterialization::materialize(
     Region &copyRegion = recipe.getCopyRegion();
     setLocation(copyRegion, loc);
     acc::cloneACCRegionInto(&copyRegion, block, std::next(ip), mapping, {});
+
+    // Create a temporary acc.firstprivate_save op to keep the value live-out
+    // until after other passes have determined it is truly firstprivate.
+    b.setInsertionPoint(accOp.getBody().getTerminator());
+    auto ptrLikeType = dyn_cast<acc::PointerLikeType>(alloca.getType());
+    assert(ptrLikeType && "alloca must be a pointer-like type");
+    auto xTyped = cast<TypedValue<acc::PointerLikeType>>(alloca);
+    assert(xTyped && "alloca must be a typed value");
+    auto loadOp = ptrLikeType.genLoad(b, op.getLoc(), xTyped, {});
+    acc::FirstprivateSaveOp::create(b, op.getLoc(), loadOp, origPtr);
+
     if (!recipe.getDestroyRegion().empty()) {
       // origPtr was already pushed.
       cloneDestroy(loc, recipe, block, std::prev(block->end()), results);
diff --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
index e100ec2df139b..533c7e6ec22dd 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
@@ -24,14 +24,18 @@ acc.private.recipe @privatization_memref_i32 : memref<i32> init {
 // and an alloca (as per the recipe) inside the region.
 // Then ensure that all uses are of the private alloca.
 // CHECK-LABEL: func.func @firstpriv
-// CHECK: acc.parallel {
-// CHECK: %[[ALLOCA:.*]] = memref.alloca() {acc.var_name = #acc.var_name<"t">} : memref<i32>
-// CHECK: %[[FIRSTPRIVLOAD:.*]] = memref.load %{{.*}}[] : memref<i32>
-// CHECK: memref.store %[[FIRSTPRIVLOAD]], %[[ALLOCA]][] : memref<i32>
-// CHECK: %[[ALLOCALOAD:.*]] = memref.load %[[ALLOCA]][] : memref<i32>
-// CHECK: %[[ADDI:.*]] = arith.addi %[[ALLOCALOAD]], %c1{{.*}} : i32
-// CHECK: memref.store %[[ADDI]], %[[ALLOCA]][] : memref<i32>
-// CHECK: memref.dealloc %[[ALLOCA]] : memref<i32>
+// CHECK:       [[ALLOCA:%.*]] = memref.alloca() : memref<i32>
+// CHECK:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[ALLOCA]] : memref<i32>) -> memref<i32> {implicit = true, name = "t"}
+// CHECK:       acc.parallel {
+// CHECK:       [[FIRSTPRIV:%.*]] = memref.alloca() {acc.var_name = #acc.var_name<"t">} : memref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = memref.load [[MAP]][] : memref<i32>
+// CHECK:       memref.store [[FIRSTPRIVLOAD]], [[FIRSTPRIV]][] : memref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = memref.load [[FIRSTPRIV]][] : memref<i32>
+// CHECK:       [[ADDI:%.*]] = arith.addi [[FIRSTPRIVLOAD]], %c1{{.*}} : i32
+// CHECK:       memref.store [[ADDI]], [[FIRSTPRIV]][] : memref<i32>
+// CHECK-NEXT:  [[VALUELOAD:%.*]] = memref.load
+// CHECK-NEXT:  acc.firstprivate_save [[VALUELOAD]] to [[MAP]] : i32 -> memref<i32>
+// CHECK-NEXT:  memref.dealloc [[FIRSTPRIV]] : memref<i32>
 
 func.func @firstpriv() {
   %c1336 = arith.constant 1336 : i32


From llvmlistbot at llvm.org  Tue Jul 21 07:42:08 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Tue, 21 Jul 2026 07:42:08 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211011@github.com>
Message-ID: <6a5f8540.f3151ff8.37ac76.9bb4@mx.google.com>


rscottmanley wrote:

Happy to reword any of the descriptions if you have better ideas. Please lmk

https://github.com/llvm/llvm-project/pull/211011

From llvmlistbot at llvm.org  Tue Jul 21 07:49:48 2026
From: llvmlistbot at llvm.org (Evgeny Baskakov)
Date: Tue, 21 Jul 2026 07:49:48 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Emit unified declare
 globals as extern. (PR #210757)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210757@github.com>
Message-ID: <6a5f870c.d1496011.199a59.05f0@mx.google.com>


ebaskakov wrote:

LGTM

https://github.com/llvm/llvm-project/pull/210757

From llvmlistbot at llvm.org  Tue Jul 21 07:55:22 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 07:55:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter via lane_data (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5f885a.0c05e68e.3c2f5d.377d@mx.google.com>


https://github.com/mshahneo edited https://github.com/llvm/llvm-project/pull/201688

From llvmlistbot at llvm.org  Tue Jul 21 07:55:24 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Tue, 21 Jul 2026 07:55:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
Message-ID: <llvm.org/llvm/llvm-project/pull/211014@github.com>


https://github.com/takatodo created https://github.com/llvm/llvm-project/pull/211014

Affine dependence analysis can return `DependenceResult::Failure` when it
cannot construct an access relation. Some loop transformations currently
treat this result like `NoDependence`, which can allow an invalid
transformation and cause a miscompilation.

This change handles analysis failures conservatively:

- Reject loop tiling when a dependence check fails.
- Reject validity-checked loop permutation when the complete dependence
  graph cannot be constructed.
- Preserve the original loop nest when sequential-loop sinking encounters
  the same failure.
- Add regression tests using a semi-affine same-memref dependence.

Fixes #210585.

Assisted-by: OpenAI Codex

>From f60088f7074b108d6ad6596823fc028cfa980ac3 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Tue Jul 21 07:55:33 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Tue, 21 Jul 2026 07:55:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a5f8865.eb165e5c.4506c.e348@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From f60088f7074b108d6ad6596823fc028cfa980ac3 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Tue Jul 21 07:56:56 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 07:56:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5f88b8.6c8668f2.1f24.7a43@mx.google.com>


https://github.com/egebeysel edited https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 07:56:56 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 07:56:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5f88b8.4a342cbc.3a2f36.20fe@mx.google.com>


https://github.com/egebeysel approved this pull request.

LGTM, thanks! I have a few questions regarding the masks and lowering to contractions directly, but we can leave them to different PRs.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 07:56:56 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Tue, 21 Jul 2026 07:56:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5f88b8.c5504292.78ef3.9075@mx.google.com>


================
@@ -0,0 +1,360 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_lhs
+//
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size for dim M is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size for dim N is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
+
+  return %C_pack : tensor<1x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dims are scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
+      : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
----------------
egebeysel wrote:

do we still need this after `assume_dynamic_dims_match_vec_sizes`? I would expect to have no masks.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 07:58:50 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 07:58:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter via lane_data (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5f892a.f1c1923d.2d2b59.7c80@mx.google.com>


https://github.com/mshahneo edited https://github.com/llvm/llvm-project/pull/201688

From llvmlistbot at llvm.org  Tue Jul 21 07:59:18 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 07:59:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter via lane_data (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5f8946.13be6df8.22eac4.b045@mx.google.com>


https://github.com/mshahneo edited https://github.com/llvm/llvm-project/pull/201688

From llvmlistbot at llvm.org  Tue Jul 21 08:03:08 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 08:03:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a5f8a2c.92d2873c.aa1ff.3668@mx.google.com>


https://github.com/mshahneo updated https://github.com/llvm/llvm-project/pull/205122

>From 178ee5add97f929794f053dd56c5904e0b42ee49 Mon Sep 17 00:00:00 2001
From: Claude <noreply at anthropic.com>
Date: Mon, 22 Jun 2026 15:32:10 +0000
Subject: [PATCH] [mlir][xegpu] Remove inferred chunk_size; rename contiguity
 attr to chunk_size

Two changes to the xegpu.load (LoadGatherOp) / xegpu.store (StoreScatterOp)
gather/scatter ops:

1. Remove the old optional `chunk_size` I64 attribute that duplicated
   information already carried by the operand/result types. The payload chunk
   size (contiguous elements per work item) is derived from the value and mask
   types via a shared helper `xegpu::getGatherScatterPayloadChunk` in
   XeGPUUtils; the op verifier keeps a file-local equivalent. Consumers in
   propagate-layout, sg-to-lane, unroll, and the operand-layout util now call
   the helper instead of an op accessor.

2. Rename the `contiguity` attribute (added by #201684) to `chunk_size`, so the
   name `chunk_size` now denotes the offsets-contiguity hint: an optional I64
   whose value must be >= 2 and must divide the innermost offsets dimension
   (verified by `isValidChunkSize`). The AxisInfo-based analysis
   (runContiguityAnalysis) stamps it via the generated setChunkSize accessor;
   the test driver and lit tests are updated accordingly.

Because the two uses no longer share the `getChunkSize()` name, there is no
collision: `getChunkSize()` is exclusively the generated accessor for the
renamed attribute.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../include/mlir/Dialect/XeGPU/IR/XeGPUOps.td | 37 ++++----
 .../Dialect/XeGPU/Transforms/Transforms.h     |  4 +-
 .../mlir/Dialect/XeGPU/Utils/XeGPUUtils.h     | 12 +++
 .../VectorToXeGPU/VectorToXeGPU.cpp           |  4 -
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp        | 86 ++++++++++++-------
 .../Transforms/XeGPUContiguityAnalysis.cpp    |  8 +-
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  6 +-
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 17 ++--
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 20 ++---
 .../Transforms/XeGPUWgToSgDistribute.cpp      | 16 ++--
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   | 20 ++++-
 .../Dialect/XeGPU/contiguity-analysis.mlir    | 38 ++++----
 mlir/test/Dialect/XeGPU/invalid.mlir          | 53 +++---------
 mlir/test/Dialect/XeGPU/ops.mlir              | 34 ++++----
 .../XeGPU/propagate-layout-subgroup.mlir      |  8 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |  4 +-
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 12 +--
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  | 16 ++--
 .../test-xegpu-coalesce-gather-scatter.mlir   |  2 +-
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   | 40 ++++-----
 .../Dialect/XeGPU/xegpu-unroll-patterns.mlir  | 16 ++--
 mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir   | 32 +++----
 .../lib/Dialect/XeGPU/TestXeGPUTransforms.cpp | 10 +--
 23 files changed, 253 insertions(+), 242 deletions(-)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..ee551e1065109 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -731,13 +731,17 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to load from per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the result
       of load. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`: the innermost `offsets` dimension is contiguous in runs of
+      `chunk_size` elements (so `chunk_size` must be >= 2 and must divide that
+      dimension). It is a hint used by layout propagation / coalescing; it does
+      not change the loaded data. Only valid on the vector-offsets form.
+
     Results:
     - `res`: represents loaded data
 
@@ -756,8 +760,8 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   ```
 
   Example 2 (lane level):
-  lane level only accepts the offsets variant. chunk_size can be inferred from result
-  type. In this example, chunk_size is 8.
+  lane level only accepts the offsets variant. The number of contiguous elements
+  loaded per work item is inferred from the result type; in this example it is 8.
   ```mlir
     %2 = xegpu.load %1[%2], %0 <{l1_hint = #xegpu.cache_hint<cached>,
                              l2_hint = #xegpu.cache_hint<uncached>,
@@ -769,12 +773,12 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
 
   let arguments = (ins XeGPU_GatherScatterBaseAddrType:$source,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
   let results = (outs XeGPU_ValueOrScalarType:$value);
 
   let extraClassDeclaration = extraBaseClassDeclaration # [{
@@ -815,13 +819,11 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   let builders = [
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
@@ -862,13 +864,18 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to store to per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the value
       to be stored. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`, analogously to `load`: the innermost `offsets` dimension is
+      contiguous in runs of `chunk_size` elements (so `chunk_size` must be >= 2
+      and must divide that dimension). It is a hint used by layout propagation /
+      coalescing; it does not change the stored data. Only valid on the
+      vector-offsets form.
+
 
   Example 1 (Subgroup level):
   A variant accepts memref as base pointer and an offset.
@@ -886,8 +893,8 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   ```
 
   Example 2 (Lane level):
-  Lane level IR only accepts the offsets variant. chunk_size can be inferred from value
-  type. In this example, chunk_size is 8.
+  Lane level IR only accepts the offsets variant. The number of contiguous elements
+  stored per work item is inferred from the value type; in this example it is 8.
   ```mlir
     xegpu.store %0, %1[%2], %3 <{l1_hint = #xegpu.cache_hint<uncached>,
                              l2_hint = #xegpu.cache_hint<write_back>,
@@ -900,12 +907,12 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let arguments = (ins XeGPU_ValueOrScalarType:$value,
       XeGPU_GatherScatterBaseAddrType:$dest,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
 
   let extraClassDeclaration = extraBaseClassDeclaration#[{
     Type getDestType() {
@@ -945,13 +952,11 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let builders = [
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
index 388bd6145df21..7f707d67ebd2a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
@@ -93,11 +93,11 @@ void populateXeGPUSgToLaneDistributeTypeConversionAndLegality(
 //===----------------------------------------------------------------------===//
 
 /// Run the AxisInfo-based contiguity analysis over `root` and stamp a
-/// `contiguity` attribute on every `xegpu.load` / `xegpu.store` whose
+/// `chunk_size` attribute on every `xegpu.load` / `xegpu.store` whose
 /// offsets are contiguous (in runs of >= 2) along the innermost dimension.
 /// The stamped value is the inner-dim contiguity; it is a target-independent
 /// property consumed downstream (e.g. to derive a `lane_data` split). Ops that
-/// already carry a `contiguity` attribute are left untouched.
+/// already carry a `chunk_size` attribute are left untouched.
 void runContiguityAnalysis(Operation *root);
 
 /// Collect a set of patterns to unroll xegpu operations to a smaller shapes.
diff --git a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
index 0125dfc44196b..33dfc1cea4ea0 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
@@ -41,6 +41,18 @@ namespace xegpu {
 /// Flatten a set of ValueRange into a single SmallVector<Value>
 SmallVector<Value> flattenValues(ArrayRef<ValueRange> values);
 
+/// Infer the payload chunk size (number of contiguous elements per work item)
+/// of a gather/scatter op (`xegpu.load` / `xegpu.store`) from its value/result
+/// and mask types. The mask carries one element per lane, so the payload is the
+/// value with the chunk dimension removed: the chunk size is the trailing value
+/// dimension whenever the value has more elements than the mask, and 1
+/// otherwise.
+///   - scalar value                               -> 1
+///   - value vector<LxC...>, mask vector<L...>    -> C (trailing dim)
+///   - 1D value vector<N...>, scalar/size-1 mask  -> N (lane-level chunk)
+///   - value and mask with matching element counts -> 1 (one elem per lane)
+int64_t getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy);
+
 /// If tensor descriptor has a layout attribute it is used in SIMT mode.
 /// In this mode, the distributed vector shape is determined as follows:
 /// Definitions:
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..feab0eb51ec27 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -502,7 +502,6 @@ static LogicalResult lowerToScatteredLoadOp(vector::TransferReadOp readOp,
       vectorShape);
   auto gatherOp = xegpu::LoadGatherOp::create(
       rewriter, loc, vectorType, flatMemref, localOffsets, mask,
-      /*chunk_size=*/IntegerAttr{},
       /*l1_hint=*/xegpu::CachePolicyAttr{},
       /*l2_hint=*/xegpu::CachePolicyAttr{},
       /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -537,7 +536,6 @@ static LogicalResult lowerToScatteredStoreOp(vector::TransferWriteOp writeOp,
       vectorShape);
   xegpu::StoreScatterOp::create(rewriter, loc, writeOp.getVector(), flatMemref,
                                 localOffsets, mask,
-                                /*chunk_size=*/IntegerAttr{},
                                 /*l1_hint=*/xegpu::CachePolicyAttr{},
                                 /*l2_hint=*/xegpu::CachePolicyAttr{},
                                 /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -793,7 +791,6 @@ struct GatherLowering : public OpRewritePattern<vector::GatherOp> {
 
     auto xeGatherOp = xegpu::LoadGatherOp::create(
         rewriter, loc, vectorType, flatMemref, localOffsets, gatherOp.getMask(),
-        /*chunk_size=*/IntegerAttr{},
         /*l1_hint=*/xegpu::CachePolicyAttr{},
         /*l2_hint=*/xegpu::CachePolicyAttr{},
         /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -828,7 +825,6 @@ struct ScatterLowering : public OpRewritePattern<vector::ScatterOp> {
 
     xegpu::StoreScatterOp::create(rewriter, loc, scatterOp.getValueToStore(),
                                   flatMemref, localOffsets, scatterOp.getMask(),
-                                  /*chunk_size=*/IntegerAttr{},
                                   /*l1_hint=*/xegpu::CachePolicyAttr{},
                                   /*l2_hint=*/xegpu::CachePolicyAttr{},
                                   /*l3_hint=*/xegpu::CachePolicyAttr{},
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..c0178035b59d5 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -61,6 +61,30 @@ static bool isWriteHintOrNone(const CachePolicyAttr &attr) {
          kind == CachePolicy::WRITE_BACK || kind == CachePolicy::WRITE_THROUGH;
 }
 
+// Infer the chunk size (number of contiguous elements per work item) of a
+// gather/scatter op from its value/result and mask types. The chunk size is no
+// longer carried as an attribute; it is fully determined by the types. The
+// mask carries one element per lane, so it is the value with the chunk
+// dimension removed. The chunk size is therefore the trailing value dimension
+// whenever the value has more elements than the mask, and 1 otherwise:
+//   - scalar value                                   -> 1
+//   - value vector<LxC...>, mask vector<L...>        -> C (trailing dim)
+//   - 1D value vector<N...>, scalar/size-1 mask      -> N (lane-level chunk)
+//   - value and mask with matching element counts    -> 1 (one elem per lane)
+static int64_t inferGatherScatterChunkSize(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 static LogicalResult
 isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
                                  VectorType valueTy, int64_t chunkSize,
@@ -113,23 +137,23 @@ isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
   return success();
 }
 
-// Validates the `contiguity` attribute against the op's offsets type: the
+// Validates the `chunk_size` attribute against the op's offsets type: the
 // innermost offsets dimension is contiguous in runs of `size`, so `size` must
 // be >= 2 and must divide that dimension.
 static LogicalResult
-isValidContiguity(std::optional<uint64_t> contiguity, Type offsetsTy,
-                  function_ref<InFlightDiagnostic()> emitError) {
-  if (!contiguity)
+isValidChunkSize(std::optional<uint64_t> chunkSize, Type offsetsTy,
+                 function_ref<InFlightDiagnostic()> emitError) {
+  if (!chunkSize)
     return success();
   auto offsetsVecTy = dyn_cast<VectorType>(offsetsTy);
   if (!offsetsVecTy)
-    return emitError() << "contiguity requires vector offsets (one per lane).";
-  int64_t size = static_cast<int64_t>(*contiguity);
+    return emitError() << "chunk_size requires vector offsets (one per lane).";
+  int64_t size = static_cast<int64_t>(*chunkSize);
   int64_t inner = offsetsVecTy.getShape().back();
   if (size < 2)
-    return emitError() << "contiguity = " << size << " (must be >= 2)";
+    return emitError() << "chunk_size = " << size << " (must be >= 2)";
   if (inner % size != 0)
-    return emitError() << "contiguity = " << size
+    return emitError() << "chunk_size = " << size
                        << " (must divide the innermost offsets dim " << inner
                        << ")";
   return success();
@@ -592,7 +616,7 @@ LogicalResult LoadGatherOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto srcTy = getSourceType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(srcTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -604,8 +628,8 @@ LogicalResult LoadGatherOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -614,7 +638,7 @@ LogicalResult LoadGatherOp::verify() {
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint) {
   auto loc = source.getLoc();
@@ -623,15 +647,14 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, /*anchor_layout=*/nullptr,
-        /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint,
                          DistributeLayoutAttr layout) {
@@ -641,8 +664,8 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
@@ -662,7 +685,7 @@ LogicalResult StoreScatterOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto destTy = getDestType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(destTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -674,8 +697,8 @@ LogicalResult StoreScatterOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -684,7 +707,6 @@ LogicalResult StoreScatterOp::verify() {
 void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
                            Value value, Value dest,
                            ArrayRef<OpFoldResult> offsets, Value mask,
-                           IntegerAttr chunk_size,
                            xegpu::CachePolicyAttr l1_hint,
                            xegpu::CachePolicyAttr l2_hint,
                            xegpu::CachePolicyAttr l3_hint) {
@@ -695,15 +717,17 @@ void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, /*anchor_layout=*/nullptr, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
-void StoreScatterOp::build(
-    OpBuilder &builder, OperationState &state, Value value, Value dest,
-    ArrayRef<OpFoldResult> offsets, Value mask, IntegerAttr chunk_size,
-    xegpu::CachePolicyAttr l1_hint, xegpu::CachePolicyAttr l2_hint,
-    xegpu::CachePolicyAttr l3_hint, DistributeLayoutAttr layout) {
+void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
+                           Value value, Value dest,
+                           ArrayRef<OpFoldResult> offsets, Value mask,
+                           xegpu::CachePolicyAttr l1_hint,
+                           xegpu::CachePolicyAttr l2_hint,
+                           xegpu::CachePolicyAttr l3_hint,
+                           DistributeLayoutAttr layout) {
   auto loc = dest.getLoc();
   int64_t size = static_cast<int64_t>(offsets.size());
   auto type = VectorType::get(size, builder.getIndexType());
@@ -711,8 +735,8 @@ void StoreScatterOp::build(
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
index 3cc8152561681..64f1d6b992e4f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
@@ -872,7 +872,7 @@ using ::mlir::xegpu::detail::axis_dataflow::AxisInfoLattice;
 // Analysis driver.
 //===----------------------------------------------------------------------===//
 
-/// Stamp a `contiguity` attribute on `op` recording the inner-dim contiguity
+/// Stamp a `chunk_size` attribute on `op` recording the inner-dim contiguity
 /// computed by the analysis. The contiguity is a target-independent property
 /// of the offsets.
 template <typename OpTy>
@@ -880,9 +880,9 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
   auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
   if (!offsetsTy || offsetsTy.getNumElements() <= 1)
     return;
-  // A pre-existing `contiguity` (user-authored, or stamped by an earlier run)
+  // A pre-existing `chunk_size` (user-authored, or stamped by an earlier run)
   // takes precedence; leave it untouched so the analysis is idempotent.
-  if (op.getContiguity())
+  if (op.getChunkSize())
     return;
   const auto *lat = solver.lookupState<AxisInfoLattice>(op.getOffsets());
   if (!lat || !lat->getValue().isInitialized())
@@ -898,7 +898,7 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
     --contiguity;
   if (contiguity < 2)
     return;
-  op.setContiguity(contiguity);
+  op.setChunkSize(contiguity);
 }
 
 } // namespace
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1681d295ae0ff..d1e27094564cb 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1218,7 +1218,8 @@ void LayoutInfoPropagation::visitLoadGatherOp(
   if (!uArch)
     return;
   VectorType resVecTy = load.getValueType();
-  int chunkSize = load.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(resVecTy, load.getMaskType());
 
   LayoutInfo resLayoutInfo = results[0]->getValue();
   if (!resLayoutInfo.isAssigned())
@@ -1283,7 +1284,8 @@ void LayoutInfoPropagation::visitStoreScatterOp(
   if (!uArch)
     return;
   VectorType srcVecTy = storeScatter.getValueType();
-  int chunkSize = storeScatter.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(srcVecTy, storeScatter.getMaskType());
 
   if (hasParamsOfLayoutKind(anchorLayoutAttr)) {
     requiredAnchorLayoutAttr = anchorLayoutAttr;
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..2ac3d1ff2bba5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -543,7 +543,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
       return failure();
 
     // Check that leading dimensions are unit.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origResultTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origResultTy.getShape();
     if (llvm::any_of(
@@ -581,8 +582,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
     Value distSource = adaptor.getSource();
     auto newOp = xegpu::LoadGatherOp::create(
         rewriter, op.getLoc(), distResultTy1D, distSource, distOffsets,
-        distMask, op.getChunkSizeAttr(), op.getL1HintAttr(), op.getL2HintAttr(),
-        op.getL3HintAttr(), /*layout=*/nullptr, /*contiguity=*/nullptr);
+        distMask, op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(),
+        /*layout=*/nullptr, /*contiguity=*/nullptr);
 
     Value result = newOp->getResult(0);
     if (distResultTy1D != distResultTy)
@@ -1071,7 +1072,8 @@ struct SgToLaneStoreScatter
       return failure();
 
     // Check that all leading dimensions are unit dimensions.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origValueTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origValueTy.getShape();
     if (llvm::any_of(shape.take_front(origValueTy.getRank() - effectiveVecRank),
@@ -1112,10 +1114,9 @@ struct SgToLaneStoreScatter
 
     Value distDest = adaptor.getDest();
     xegpu::StoreScatterOp::create(rewriter, op.getLoc(), distValue, distDest,
-                                  distOffsets, distMask, op.getChunkSizeAttr(),
-                                  op.getL1HintAttr(), op.getL2HintAttr(),
-                                  op.getL3HintAttr(), /*layout=*/nullptr,
-                                  /*contiguity=*/nullptr);
+                                  distOffsets, distMask, op.getL1HintAttr(),
+                                  op.getL2HintAttr(), op.getL3HintAttr(),
+                                  /*layout=*/nullptr, /*contiguity=*/nullptr);
     rewriter.eraseOp(op);
     return success();
   }
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..3841ae3070490 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -699,11 +699,8 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       return failure();
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     // Unroll mask and offsets with correct shape
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -721,7 +718,6 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -763,8 +759,7 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
     SmallVector<Value> newOps;
     for (auto [o, m] : llvm::zip(convertedOffsets, convertedMasks)) {
       auto newOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newValueTy, op.getSource(), o, m,
-          rewriter.getI64IntegerAttr(chunkSize), op.getL1HintAttr(),
+          rewriter, loc, newValueTy, op.getSource(), o, m, op.getL1HintAttr(),
           op.getL2HintAttr(), op.getL3HintAttr(), layout,
           /*contiguity=*/nullptr);
       newOps.push_back(newOp);
@@ -793,11 +788,8 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     if (!targetShape)
       return failure();
 
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -812,7 +804,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -859,7 +850,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     for (auto [v, o, m] :
          llvm::zip(convertedValues, convertedOffsets, convertedMasks)) {
       xegpu::StoreScatterOp::create(rewriter, loc, v, op.getDest(), o, m,
-                                    rewriter.getI64IntegerAttr(chunkSize),
                                     op.getL1HintAttr(), op.getL2HintAttr(),
                                     op.getL3HintAttr(), layout,
                                     /*contiguity=*/nullptr);
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..8cd1b1ad0d3f1 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -824,14 +824,12 @@ struct WgToSgLoadGatherOp : public OpConversionPattern<xegpu::LoadGatherOp> {
     }
 
     SmallVector<Value> newLoadOps;
-    auto chunkSizeAttr =
-        rewriter.getI64IntegerAttr(op.getChunkSize().value_or(1));
     VectorType newTy = VectorType::get(sgShape, resultType.getElementType());
     for (auto [offsets, mask] :
          llvm::zip(adaptor.getOffsets(), adaptor.getMask())) {
       auto newLayout = layout.dropSgLayoutAndData();
       auto newLoadOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newTy, op.getSource(), offsets, mask, chunkSizeAttr,
+          rewriter, loc, newTy, op.getSource(), offsets, mask,
           op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(), newLayout,
           /*contiguity=*/nullptr);
       newLoadOps.push_back(newLoadOp);
@@ -871,16 +869,12 @@ struct WgToSgStoreScatterOp
                                          "offsets have not been distributed");
     }
 
-    auto chunkSizeOpt = op.getChunkSize();
-    int64_t chunkSize = chunkSizeOpt ? static_cast<int64_t>(*chunkSizeOpt) : 1;
-    auto chunkSizeAttr = rewriter.getI64IntegerAttr(chunkSize);
     for (auto [val, offs, mask] : llvm::zip(
              adaptor.getValue(), adaptor.getOffsets(), adaptor.getMask())) {
-      xegpu::StoreScatterOp::create(rewriter, loc, val, op.getDest(), offs,
-                                    mask, chunkSizeAttr, op.getL1HintAttr(),
-                                    op.getL2HintAttr(), op.getL3HintAttr(),
-                                    layout.dropSgLayoutAndData(),
-                                    /*contiguity=*/nullptr);
+      xegpu::StoreScatterOp::create(
+          rewriter, loc, val, op.getDest(), offs, mask, op.getL1HintAttr(),
+          op.getL2HintAttr(), op.getL3HintAttr(), layout.dropSgLayoutAndData(),
+          /*contiguity=*/nullptr);
     }
     rewriter.eraseOp(op);
     return success();
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..92759fc7a69a6 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -40,6 +40,20 @@ SmallVector<Value> xegpu::flattenValues(ArrayRef<ValueRange> values) {
   return result;
 }
 
+int64_t xegpu::getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 FailureOr<VectorType>
 mlir::xegpu::getDistributedVectorType(xegpu::TensorDescType tdescTy) {
   auto layout = llvm::dyn_cast_if_present<LayoutAttr>(tdescTy.getLayout());
@@ -237,7 +251,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
 
     if (isa<xegpu::StoreScatterOp>(op)) {
       xegpu::StoreScatterOp store(op);
-      int chunkSize = store.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(store.getValueType(),
+                                                          store.getMaskType());
       if (layout && idx >= 2 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
@@ -245,7 +260,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
     }
     if (isa<xegpu::LoadGatherOp>(op)) {
       xegpu::LoadGatherOp load(op);
-      int chunkSize = load.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(load.getValueType(),
+                                                          load.getMaskType());
       if (layout && idx >= 1 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
diff --git a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
index f223b325c2f9e..df9f7fe69c2e2 100644
--- a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
+++ b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-opt -split-input-file \
 // RUN:   -test-xegpu-coalesce-gather-scatter="analyze-only=true" %s | FileCheck %s
 
-// Contiguity analysis: stamps the `contiguity` attribute on gather/scatter ops
+// Contiguity analysis: stamps the `chunk_size` attribute on gather/scatter ops
 // whose offsets are contiguous (runs of >= 2) along the innermost dimension.
 // The stamped value is the inner-dim contiguity, rounded down to a divisor of
 // the inner extent. The analysis is target-independent and mask-independent;
@@ -11,7 +11,7 @@
 // 1-D vector.step -> stride-1, fully contiguous over the 32-element inner dim.
 // CHECK-LABEL: func.func @load_step_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
@@ -25,7 +25,7 @@ func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
 // Dense stride-1 constant offsets -> contiguity 32.
 // CHECK-LABEL: func.func @load_dense_ap_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
   %offsets = arith.constant dense<[
     0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
@@ -41,7 +41,7 @@ func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
 // Stride-4 offsets: not contiguous, no attribute stamped.
 // CHECK-LABEL: func.func @load_stride4_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
   %c4 = arith.constant 4 : index
@@ -58,7 +58,7 @@ func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
 // All-equal offsets: inner dim is constant, not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_broadcast_offsets_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
   %offsets = arith.constant dense<0> : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -72,7 +72,7 @@ func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
 // (the mask check is a consumer concern).
 // CHECK-LABEL: func.func @load_partial_mask(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %v = xegpu.load %ptr[%offsets], %mask
@@ -84,7 +84,7 @@ func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32>
 // Store with vector.step offsets -> contiguity 32 on the store.
 // CHECK-LABEL: func.func @store_step_offsets(
 // CHECK: xegpu.store
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -97,7 +97,7 @@ func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
 // 2-D leading-unit dim: contiguity measured on the inner dim -> 32.
 // CHECK-LABEL: func.func @load_2d_leading_unit(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
   %step = vector.step : vector<32xindex>
   %offsets = vector.shape_cast %step : vector<32xindex> to vector<1x32xindex>
@@ -111,7 +111,7 @@ func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
 // True 2-D dense AP: each row stride-1 over 16 -> contiguity 16.
 // CHECK-LABEL: func.func @load_2d_dense_ap(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 16 : i64}>
+// CHECK-SAME: <{chunk_size = 16 : i64}>
 func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
@@ -127,7 +127,7 @@ func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
 // 2-D dense values whose inner row is not a stride-1 AP: no attribute.
 // CHECK-LABEL: func.func @load_2d_non_ap(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 100, 9, 10, 11, 12, 13, 14, 15],
@@ -146,7 +146,7 @@ func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
 // exercised even if the solver later folds all-constant arith ops.
 // CHECK-LABEL: func.func @load_divui_recovers(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %c2 = arith.constant dense<2> : vector<32xindex>
@@ -162,7 +162,7 @@ func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
 // `divui` by a constant that does not divide the inner stride: not recovered.
 // CHECK-LABEL: func.func @load_divui_non_divisor(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -180,7 +180,7 @@ func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
 // not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_remui_inner_uniform(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -196,7 +196,7 @@ func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
 // `shli` then `shrui` cancel: (step << 1) >> 1 -> stride 1 -> contiguity 32.
 // CHECK-LABEL: func.func @load_shli_then_shrui(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -212,7 +212,7 @@ func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
 // `shli` alone scales the stride to 2: not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_shli_unchanged(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -228,7 +228,7 @@ func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
 // contiguity 32.
 // CHECK-LABEL: func.func @load_select_two_aps(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %a = arith.constant dense<0>  : vector<32xindex>
@@ -243,14 +243,14 @@ func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
 }
 
 // -----
-// A pre-existing `contiguity` takes precedence: the analysis leaves it alone.
+// A pre-existing `chunk_size` takes precedence: the analysis leaves it alone.
 // CHECK-LABEL: func.func @user_attr_preserved(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 2 : i64}>
+// CHECK-SAME: <{chunk_size = 2 : i64}>
 func.func @user_attr_preserved(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
-  %v = xegpu.load %ptr[%offsets], %mask <{contiguity = 2 : i64}>
+  %v = xegpu.load %ptr[%offsets], %mask <{chunk_size = 2 : i64}>
       : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
   return %v : vector<32xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..c610ded4a5b50 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -233,7 +233,7 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
   %offsets = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<8xi1>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}>
+  %2 = xegpu.load %src[%offsets], %mask
       : memref<?xf32>, vector<4xindex>, vector<8xi1> -> vector<4x2xf32>
   return
 }
@@ -242,8 +242,8 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
 func.func @load_gather_simt_1(%src: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  %2 = xegpu.load %src[%0], %1
       : memref<?xf32>, vector<4xindex>, vector<4xi1> -> vector<6xf32>
   return
 }
@@ -265,7 +265,7 @@ func.func @store_scatter_vc_3(%dst: memref<?xf32>) {
   %1 = arith.constant dense<1>: vector<8xi1>
   %2 = arith.constant dense<2.9>: vector<4x2xf32>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  xegpu.store %2, %dst[%0], %1
       : vector<4x2xf32>, memref<?xf32>, vector<4xindex>, vector<8xi1>
   return
 }
@@ -275,8 +275,8 @@ func.func @store_scatter_simt_1(%dst: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
   %2 = arith.constant dense<2.9>: vector<6xf32>
-  // expected-error at +1 {{value elements must match chunk size}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  xegpu.store %2, %dst[%0], %1
       : vector<6xf32>, memref<?xf32>, vector<4xindex>, vector<4xi1>
   return
 }
@@ -316,26 +316,6 @@ func.func @load_gather_offset_sg(%src: memref<?xf16>) {
   return
 }
 
-// -----
-func.func @load_gather_offset_wi(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf32>
-  return
-}
-
-// -----
-func.func @store_scatter_offset_wi_1(%src: memref<?xf16>) {
-  %val = arith.constant dense<2.9>: vector<4xf16>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  %mask = arith.constant dense<1>: vector<1xi1>
-  // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %val, %src[%offsets], %mask
-        : vector<4xf16>, memref<?xf16>, vector<1xindex>, vector<1xi1>
-  return
-}
-
 // -----
 func.func @store_scatter_offset_wi_2(%src: memref<4x4xf16>) {
   %val = arith.constant dense<2.9>: vector<4xf16>
@@ -363,16 +343,7 @@ func.func @load_gather_offset_wi_4(%src: !xegpu.tensor_desc<1x2xf16>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
-  return
-}
-
-// -----
-func.func @load_gather_offset_wi_2(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf16>
+  %2 = xegpu.load %src[%offsets], %mask : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
   return
 }
 
@@ -381,7 +352,7 @@ func.func @load_gather_offset_wi_1(%src: memref<4x4xf32>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
+  %2 = xegpu.load %src[%offsets], %mask : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
   return
 }
 
@@ -792,16 +763,16 @@ func.func @dpas_mx_scale_b_layout_not_distributable(%a : vector<8x16xf8E5M2>, %b
 
 // -----
 func.func @contiguity_too_small(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // expected-error at +1 {{contiguity = 1 (must be >= 2)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 1 : i64}>
+  // expected-error at +1 {{chunk_size = 1 (must be >= 2)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 1 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   return
 }
 
 // -----
 func.func @contiguity_does_not_divide(%src: i64, %offset: vector<6xindex>, %mask: vector<6xi1>) {
-  // expected-error at +1 {{contiguity = 4 (must divide the innermost offsets dim 6)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // expected-error at +1 {{chunk_size = 4 (must divide the innermost offsets dim 6)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<6xindex>, vector<6xi1> -> vector<6xf32>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..2b82d426eba39 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -377,8 +377,8 @@ gpu.func @prefetch_nd_3d(%src: memref<4x8x16xf16>) {
 
 // CHECK: gpu.func @simt_load_4(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: vector<1xindex>, %[[arg2:.*]]: vector<1xi1>) {
 gpu.func @simt_load_4(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vector<1xi1>) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
   gpu.return
 }
 
@@ -391,8 +391,8 @@ gpu.func @simt_load_5(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vec
 
 // CHECK: gpu.func @simt_load_6(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: index, %[[arg2:.*]]: i1) {
 gpu.func @simt_load_6(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, index, i1 -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, index, i1 -> vector<8xf16>
   gpu.return
 }
 
@@ -407,23 +407,23 @@ gpu.func @simt_load_7(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
 gpu.func @subgroup_load_offset_1(%src: memref<?xf16>) {
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
-  %val = xegpu.load %src[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
+  %val = xegpu.load %src[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_4(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: vector<1xindex>, %[[arg3:.*]]: vector<1xi1>) {
 gpu.func @simt_store_4(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: vector<1xindex>, %arg3: vector<1xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_5(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: index, %[[arg3:.*]]: i1) {
 gpu.func @simt_store_5(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: index, %arg3: i1) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, index, i1
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, index, i1
   gpu.return
 }
 
@@ -446,25 +446,25 @@ gpu.func @subgroup_store_offset_1(%dest: memref<?xf16>) {
   %val = arith.constant dense<2.9>: vector<4x2xf16>
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
-  xegpu.store %val, %dest[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
+  xegpu.store %val, %dest[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @load_contiguity(%[[arg0:.*]]: i64, %[[arg1:.*]]: vector<16xindex>, %[[arg2:.*]]: vector<16xi1>) {
 gpu.func @load_contiguity(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // A user-provided `contiguity` round-trips through the optional op attribute.
-  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{contiguity = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // A user-provided `chunk_size` round-trips through the optional op attribute.
+  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.return
 }
 
 // CHECK: gpu.func @store_contiguity(%[[arg0:.*]]: vector<16xf32>, %[[arg1:.*]]: i64, %[[arg2:.*]]: vector<16xindex>, %[[arg3:.*]]: vector<16xi1>) {
 gpu.func @store_contiguity(%val: vector<16xf32>, %dest: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{contiguity = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
-  xegpu.store %val, %dest[%offset], %mask <{contiguity = 4 : i64}>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
+  xegpu.store %val, %dest[%offset], %mask <{chunk_size = 4 : i64}>
       : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..402700fd4c5e3 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -615,9 +615,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
@@ -646,9 +646,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..b8b8d86cf8b10 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -642,8 +642,8 @@ gpu.module @test{
     %5 = vector.broadcast %3 : index to vector<1xindex>
     %6 = arith.addi %4, %5 : vector<1xindex>
     %7 = vector.broadcast %6 : vector<1xindex> to vector<1x1x1x16xindex>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..3ced1d7e9c6f5 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -260,14 +260,14 @@ gpu.func @prefetch_nd() {
 // CHECK-LABEL: gpu.func @scatter_load_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[CAST:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   gpu.return
 }
@@ -275,20 +275,20 @@ gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
 // CHECK-LABEL: gpu.func @scatter_store_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[C1:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK: %[[C2:.*]] = vector.shape_cast %[[C1]] : vector<1x8xf16> to vector<8xf16>
-// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.func @scatter_store_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   xegpu.store %0, %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index fa9897770a08e..e8a41f6fc99f0 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -257,7 +257,7 @@ gpu.module @xevm_module{
 // CHECK-DAG:      %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK-DAG:      %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:          %[[IF:.*]] = scf.if %[[PREDICATE]] -> (vector<1x8xf16>) {
-// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:          : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK-NEXT:        %[[LD_CAST:.*]] = vector.shape_cast %[[LD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK-NEXT:        scf.yield %[[LD_CAST]] : vector<1x8xf16>
@@ -265,14 +265,14 @@ gpu.module @xevm_module{
 // CHECK-NEXT:        scf.yield %[[CST]] : vector<1x8xf16>
 // CHECK-NEXT:      }
 // CHECK-NEXT:      %[[IF_CAST:.*]] = vector.shape_cast %[[IF]] : vector<1x8xf16> to vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:        vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.module @xevm_module{
   gpu.func @scatter_ops_scf_yield(%src: memref<256xf16>, %pred : i1) {
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     %loaded = scf.if %pred -> (vector<16x8xf16>) {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
@@ -280,7 +280,7 @@ gpu.module @xevm_module{
       %3 = arith.constant dense<12.> : vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
     }
-    xegpu.store %loaded, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+    xegpu.store %loaded, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     gpu.return
   }
 }
@@ -291,9 +291,9 @@ gpu.module @xevm_module{
 // CHECK:         %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK:         %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:         scf.if %[[PREDICATE]] {
-// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 // CHECK-NEXT:    }
 gpu.module @xevm_module{
@@ -302,10 +302,10 @@ gpu.module @xevm_module{
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     scf.if %pred  {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
-      xegpu.store %3, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+      xegpu.store %3, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     }
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
index fb32e0012679b..885026f5c85ae 100644
--- a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
+++ b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
@@ -99,7 +99,7 @@ gpu.module @kernel_chunk4 [#xevm.target<chip = "pvc">] {
 // CHECK-LABEL: func.func @load_no_target_unchanged(
 // CHECK: xegpu.load
 // CHECK-NOT: lane_data
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_no_target_unchanged(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..3d769c86ee2cc 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -334,8 +334,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -353,8 +353,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -369,8 +369,8 @@ gpu.module @test_kernel {
     //CHECK: arith.addi [[step]], [[cst]] : vector<16xindex>
     %step = vector.step : vector<32xindex>
     %mask = vector.create_mask %c16 : vector<32xi1>
-    %ld = xegpu.load %src[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -514,7 +514,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: load_with_offsets
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -525,7 +525,7 @@ gpu.module @test_kernel {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17 : vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -534,7 +534,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: store_with_offsets
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -547,7 +547,7 @@ gpu.module @test_kernel {
       %mask = vector.create_mask %c17 : vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -561,7 +561,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -572,7 +572,7 @@ gpu.module @test_kernel {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17 : vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 }
@@ -585,7 +585,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -598,7 +598,7 @@ gpu.module @test_kernel {
     %mask = vector.create_mask %c17 : vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -611,8 +611,8 @@ gpu.module @test_kernel {
   // CHECK: [[cst_0:%.+]] = arith.constant dense<true> : vector<1x1x16xi1>
   // CHECK: [[cst_1:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
   // CHECK: [[cst_2:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
-  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
-  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
   // CHECK: [[ins_0:%.+]] = vector.insert_strided_slice [[ld_0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   // CHECK: [[ins_1:%.+]] = vector.insert_strided_slice [[ld_1]], [[ins_0]] {offsets = [0, 0, 16], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   gpu.func @preserve_unit_dim_of_load_inst_data(%src: ui64) -> vector<1x1x32xf32> {
@@ -624,7 +624,7 @@ gpu.module @test_kernel {
       ]]> : vector<1x1x32xindex>
 
       %mask = arith.constant dense<true> : vector<1x1x32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
 
       gpu.return %ld : vector<1x1x32xf32>
   }
@@ -692,10 +692,10 @@ gpu.module @test_kernel {
         128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]]
     ]> : vector<1x1x32xindex>
     %mask = arith.constant dense<true> : vector<1x1x32xi1>
-    %a = xegpu.load %A[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
-    %b = xegpu.load %B[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %a = xegpu.load %A[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %b = xegpu.load %B[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     %addf = arith.addf %a, %b : vector<1x1x32xf32>
-    xegpu.store %addf, %C[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
+    xegpu.store %addf, %C[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
index 49fdf1cbee174..75492d8e248d3 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
@@ -128,7 +128,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: load_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -139,7 +139,7 @@ gpu.module @test {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17: vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -147,7 +147,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: store_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -160,7 +160,7 @@ gpu.module @test {
       %mask = vector.create_mask %c17: vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -173,7 +173,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -184,7 +184,7 @@ gpu.module @test {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17: vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 
@@ -196,7 +196,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -209,7 +209,7 @@ gpu.module @test {
     %mask = vector.create_mask %c17: vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
index 8c5dd6c55b99d..2ca6e8d46ef16 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
@@ -317,11 +317,11 @@ gpu.module @test_distribution {
   gpu.func @load_gather(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<32x4xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<32x4xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<32x4xindex>, vector<32x4xi1> -> vector<32x4xf16>
     %offset =  arith.constant dense<0> : vector<256x16xindex>
     %mask = arith.constant dense<1> : vector<256x16xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256x16xindex>, vector<256x16xi1> -> vector<256x16xf16>
     gpu.return
   }
@@ -332,12 +332,12 @@ gpu.module @test_distribution {
     // CHECK: %[[VAL:.*]] = arith.constant dense<2.550000e+01> : vector<8xf16>
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
+    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
      // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<8xindex>, vector<8xi1>
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
+    xegpu.store %val, %dest[%offset], %mask {layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
                                              l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
@@ -348,11 +348,11 @@ gpu.module @test_distribution {
   gpu.func @load_with_non_unit_chunk_size(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 4 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<8xindex>, vector<8xi1> -> vector<8x4xf16>
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 4, layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256xindex>, vector<256xi1> -> vector<256x4xf16>
     gpu.return
   }
@@ -783,9 +783,9 @@ gpu.module @test_distribution {
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
 
-    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{chunk_size = 1 : i64, layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
     // CHECK-SAME: memref<4096xf32>, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    %3 = xegpu.load %2[%offset], %mask {chunk_size = 1, layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
+    %3 = xegpu.load %2[%offset], %mask {layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
 
     // CHECK: %[[BROADCAST:.*]] = vector.broadcast %[[LOAD]] : vector<32xf32> to vector<32x32xf32>
     %4 = vector.broadcast %3 : vector<256xf32> to vector<256x256xf32>
@@ -803,7 +803,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<1.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [1] : vector<1x1x32xf32> to vector<1x32xf32>
     // CHECK-DAG: %[[CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x32xf32> to vector<1x1x32xf32>
@@ -820,7 +820,7 @@ gpu.module @test_distribution {
     %cst_3 = arith.constant dense<1.0> : vector<1x32xf32>
     %offset = arith.constant dense<0> : vector<1x32x32xindex>
     %mask = arith.constant dense<true> : vector<1x32x32xi1>
-    %14 = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
+    %14 = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
     %15 = vector.multi_reduction <add>, %14, %cst_3 [1] : vector<1x32x32xf32> to vector<1x32xf32>
     %anchor = xegpu.convert_layout %15
       <{
@@ -878,7 +878,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<1x1x32x32xf32> to vector<1x1xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x1xf32> to vector<1x1x1x1xf32>
@@ -895,7 +895,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<2x2xf32>
     %offset = arith.constant dense<0> : vector<2x2x128x128xindex>
     %mask = arith.constant dense<true> : vector<2x2x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<2x2x128x128xf32> to vector<2x2xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -911,7 +911,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<16x16x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<16x16x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<16x16x32x32xf32> to vector<16x16xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<16x16xf32> to vector<16x16x1x1xf32>
@@ -928,7 +928,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<32x32xf32>
     %offset = arith.constant dense<0> : vector<32x32x128x128xindex>
     %mask = arith.constant dense<true> : vector<32x32x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<32x32x128x128xf32> to vector<32x32xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -1026,7 +1026,7 @@ gpu.module @test_distribution {
   gpu.func @convert_layout_3D(%arg0: memref<?xf32>) {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0> : vector<1x32x16xindex>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<true> : vector<1x32x16xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{chunk_size = 1 : i64, layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
     // CHECK-DAG: %[[ALLOCA:.*]] = memref.alloca() : memref<1048576xi8, 3>
     // CHECK-DAG: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<1048576xi8, 3> -> !xegpu.mem_desc<8x128x256xf32>
     // CHECK-DAG: %[[SGID:.*]] = gpu.subgroup_id : index
@@ -1055,7 +1055,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[LOAD_SLM:.*]] = xegpu.load_matrix %[[MDESC]][%[[LOAD_OFF_Z]], %[[LOAD_OFF_Y]], %[[LOAD_OFF_X]]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}>: !xegpu.mem_desc<8x128x256xf32>, index, index, index -> vector<1x16x32xf32>
     %offset = arith.constant dense<0> : vector<8x128x256xindex>
     %mask = arith.constant dense<true> : vector<8x128x256xi1>
-    %1 = xegpu.load %arg0[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
+    %1 = xegpu.load %arg0[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
     %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>,
                                    target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>}> : vector<8x128x256xf32>
     %anchor = xegpu.convert_layout %2
diff --git a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
index f6b0c50da91dd..a94d172b44801 100644
--- a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
+++ b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
@@ -456,7 +456,7 @@ struct TestXeGPUCoalesceGatherScatter
   }
 
   StringRef getDescription() const final {
-    return "Test driver that turns the contiguity attribute into a lane_data "
+    return "Test driver that turns the chunk_size attribute into a lane_data "
            "layout on gather/scatter ops.";
   }
 
@@ -477,7 +477,7 @@ struct TestXeGPUCoalesceGatherScatter
 
   Option<bool> analyzeOnly{
       *this, "analyze-only",
-      llvm::cl::desc("Only run the analysis (stamp the contiguity attribute); "
+      llvm::cl::desc("Only run the analysis (stamp the chunk_size attribute); "
                      "do not apply."),
       llvm::cl::init(false)};
 
@@ -509,16 +509,16 @@ struct TestXeGPUCoalesceGatherScatter
     return xegpu::LayoutAttr::get(ctx, instData, laneLayout, laneData);
   }
 
-  /// Minimal driver: read the `contiguity` attribute the analysis stamped and
+  /// Minimal driver: read the `chunk_size` attribute the analysis stamped and
   /// turn it into a `lane_data` layout. This is only a stand-in for the real
   /// consumer (layout propagation) so the analysis output can be checked
   /// end-to-end; it handles just the simple power-of-two case.
   template <typename OpTy>
   static void applyContiguity(OpTy op, unsigned maxChunkSize) {
-    std::optional<uint64_t> contiguity = op.getContiguity();
+    std::optional<uint64_t> contiguity = op.getChunkSize();
     if (!contiguity)
       return;
-    op.removeContiguityAttr();
+    op.removeChunkSizeAttr();
 
     auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
     auto valueTy = op.getValueType();


From llvmlistbot at llvm.org  Tue Jul 21 08:03:29 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 08:03:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][XeGPU] Distribute coalesced
 gather/scatter via lane_data (PR #201688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/201688@github.com>
Message-ID: <6a5f8a41.1b0626d0.59100.a59f@mx.google.com>


https://github.com/mshahneo updated https://github.com/llvm/llvm-project/pull/201688

>From 178ee5add97f929794f053dd56c5904e0b42ee49 Mon Sep 17 00:00:00 2001
From: Claude <noreply at anthropic.com>
Date: Mon, 22 Jun 2026 15:32:10 +0000
Subject: [PATCH 1/2] [mlir][xegpu] Remove inferred chunk_size; rename
 contiguity attr to chunk_size

Two changes to the xegpu.load (LoadGatherOp) / xegpu.store (StoreScatterOp)
gather/scatter ops:

1. Remove the old optional `chunk_size` I64 attribute that duplicated
   information already carried by the operand/result types. The payload chunk
   size (contiguous elements per work item) is derived from the value and mask
   types via a shared helper `xegpu::getGatherScatterPayloadChunk` in
   XeGPUUtils; the op verifier keeps a file-local equivalent. Consumers in
   propagate-layout, sg-to-lane, unroll, and the operand-layout util now call
   the helper instead of an op accessor.

2. Rename the `contiguity` attribute (added by #201684) to `chunk_size`, so the
   name `chunk_size` now denotes the offsets-contiguity hint: an optional I64
   whose value must be >= 2 and must divide the innermost offsets dimension
   (verified by `isValidChunkSize`). The AxisInfo-based analysis
   (runContiguityAnalysis) stamps it via the generated setChunkSize accessor;
   the test driver and lit tests are updated accordingly.

Because the two uses no longer share the `getChunkSize()` name, there is no
collision: `getChunkSize()` is exclusively the generated accessor for the
renamed attribute.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../include/mlir/Dialect/XeGPU/IR/XeGPUOps.td | 37 ++++----
 .../Dialect/XeGPU/Transforms/Transforms.h     |  4 +-
 .../mlir/Dialect/XeGPU/Utils/XeGPUUtils.h     | 12 +++
 .../VectorToXeGPU/VectorToXeGPU.cpp           |  4 -
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp        | 86 ++++++++++++-------
 .../Transforms/XeGPUContiguityAnalysis.cpp    |  8 +-
 .../XeGPU/Transforms/XeGPUPropagateLayout.cpp |  6 +-
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 17 ++--
 .../Dialect/XeGPU/Transforms/XeGPUUnroll.cpp  | 20 ++---
 .../Transforms/XeGPUWgToSgDistribute.cpp      | 16 ++--
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   | 20 ++++-
 .../Dialect/XeGPU/contiguity-analysis.mlir    | 38 ++++----
 mlir/test/Dialect/XeGPU/invalid.mlir          | 53 +++---------
 mlir/test/Dialect/XeGPU/ops.mlir              | 34 ++++----
 .../XeGPU/propagate-layout-subgroup.mlir      |  8 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |  4 +-
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 12 +--
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  | 16 ++--
 .../test-xegpu-coalesce-gather-scatter.mlir   |  2 +-
 mlir/test/Dialect/XeGPU/xegpu-blocking.mlir   | 40 ++++-----
 .../Dialect/XeGPU/xegpu-unroll-patterns.mlir  | 16 ++--
 mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir   | 32 +++----
 .../lib/Dialect/XeGPU/TestXeGPUTransforms.cpp | 10 +--
 23 files changed, 253 insertions(+), 242 deletions(-)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..ee551e1065109 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -731,13 +731,17 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to load from per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the result
       of load. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`: the innermost `offsets` dimension is contiguous in runs of
+      `chunk_size` elements (so `chunk_size` must be >= 2 and must divide that
+      dimension). It is a hint used by layout propagation / coalescing; it does
+      not change the loaded data. Only valid on the vector-offsets form.
+
     Results:
     - `res`: represents loaded data
 
@@ -756,8 +760,8 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   ```
 
   Example 2 (lane level):
-  lane level only accepts the offsets variant. chunk_size can be inferred from result
-  type. In this example, chunk_size is 8.
+  lane level only accepts the offsets variant. The number of contiguous elements
+  loaded per work item is inferred from the result type; in this example it is 8.
   ```mlir
     %2 = xegpu.load %1[%2], %0 <{l1_hint = #xegpu.cache_hint<cached>,
                              l2_hint = #xegpu.cache_hint<uncached>,
@@ -769,12 +773,12 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
 
   let arguments = (ins XeGPU_GatherScatterBaseAddrType:$source,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
   let results = (outs XeGPU_ValueOrScalarType:$value);
 
   let extraClassDeclaration = extraBaseClassDeclaration # [{
@@ -815,13 +819,11 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   let builders = [
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
@@ -862,13 +864,18 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to store to per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the value
       to be stored. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`, analogously to `load`: the innermost `offsets` dimension is
+      contiguous in runs of `chunk_size` elements (so `chunk_size` must be >= 2
+      and must divide that dimension). It is a hint used by layout propagation /
+      coalescing; it does not change the stored data. Only valid on the
+      vector-offsets form.
+
 
   Example 1 (Subgroup level):
   A variant accepts memref as base pointer and an offset.
@@ -886,8 +893,8 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   ```
 
   Example 2 (Lane level):
-  Lane level IR only accepts the offsets variant. chunk_size can be inferred from value
-  type. In this example, chunk_size is 8.
+  Lane level IR only accepts the offsets variant. The number of contiguous elements
+  stored per work item is inferred from the value type; in this example it is 8.
   ```mlir
     xegpu.store %0, %1[%2], %3 <{l1_hint = #xegpu.cache_hint<uncached>,
                              l2_hint = #xegpu.cache_hint<write_back>,
@@ -900,12 +907,12 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let arguments = (ins XeGPU_ValueOrScalarType:$value,
       XeGPU_GatherScatterBaseAddrType:$dest,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
 
   let extraClassDeclaration = extraBaseClassDeclaration#[{
     Type getDestType() {
@@ -945,13 +952,11 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let builders = [
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
index 388bd6145df21..7f707d67ebd2a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
@@ -93,11 +93,11 @@ void populateXeGPUSgToLaneDistributeTypeConversionAndLegality(
 //===----------------------------------------------------------------------===//
 
 /// Run the AxisInfo-based contiguity analysis over `root` and stamp a
-/// `contiguity` attribute on every `xegpu.load` / `xegpu.store` whose
+/// `chunk_size` attribute on every `xegpu.load` / `xegpu.store` whose
 /// offsets are contiguous (in runs of >= 2) along the innermost dimension.
 /// The stamped value is the inner-dim contiguity; it is a target-independent
 /// property consumed downstream (e.g. to derive a `lane_data` split). Ops that
-/// already carry a `contiguity` attribute are left untouched.
+/// already carry a `chunk_size` attribute are left untouched.
 void runContiguityAnalysis(Operation *root);
 
 /// Collect a set of patterns to unroll xegpu operations to a smaller shapes.
diff --git a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
index 0125dfc44196b..33dfc1cea4ea0 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
@@ -41,6 +41,18 @@ namespace xegpu {
 /// Flatten a set of ValueRange into a single SmallVector<Value>
 SmallVector<Value> flattenValues(ArrayRef<ValueRange> values);
 
+/// Infer the payload chunk size (number of contiguous elements per work item)
+/// of a gather/scatter op (`xegpu.load` / `xegpu.store`) from its value/result
+/// and mask types. The mask carries one element per lane, so the payload is the
+/// value with the chunk dimension removed: the chunk size is the trailing value
+/// dimension whenever the value has more elements than the mask, and 1
+/// otherwise.
+///   - scalar value                               -> 1
+///   - value vector<LxC...>, mask vector<L...>    -> C (trailing dim)
+///   - 1D value vector<N...>, scalar/size-1 mask  -> N (lane-level chunk)
+///   - value and mask with matching element counts -> 1 (one elem per lane)
+int64_t getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy);
+
 /// If tensor descriptor has a layout attribute it is used in SIMT mode.
 /// In this mode, the distributed vector shape is determined as follows:
 /// Definitions:
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..feab0eb51ec27 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -502,7 +502,6 @@ static LogicalResult lowerToScatteredLoadOp(vector::TransferReadOp readOp,
       vectorShape);
   auto gatherOp = xegpu::LoadGatherOp::create(
       rewriter, loc, vectorType, flatMemref, localOffsets, mask,
-      /*chunk_size=*/IntegerAttr{},
       /*l1_hint=*/xegpu::CachePolicyAttr{},
       /*l2_hint=*/xegpu::CachePolicyAttr{},
       /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -537,7 +536,6 @@ static LogicalResult lowerToScatteredStoreOp(vector::TransferWriteOp writeOp,
       vectorShape);
   xegpu::StoreScatterOp::create(rewriter, loc, writeOp.getVector(), flatMemref,
                                 localOffsets, mask,
-                                /*chunk_size=*/IntegerAttr{},
                                 /*l1_hint=*/xegpu::CachePolicyAttr{},
                                 /*l2_hint=*/xegpu::CachePolicyAttr{},
                                 /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -793,7 +791,6 @@ struct GatherLowering : public OpRewritePattern<vector::GatherOp> {
 
     auto xeGatherOp = xegpu::LoadGatherOp::create(
         rewriter, loc, vectorType, flatMemref, localOffsets, gatherOp.getMask(),
-        /*chunk_size=*/IntegerAttr{},
         /*l1_hint=*/xegpu::CachePolicyAttr{},
         /*l2_hint=*/xegpu::CachePolicyAttr{},
         /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -828,7 +825,6 @@ struct ScatterLowering : public OpRewritePattern<vector::ScatterOp> {
 
     xegpu::StoreScatterOp::create(rewriter, loc, scatterOp.getValueToStore(),
                                   flatMemref, localOffsets, scatterOp.getMask(),
-                                  /*chunk_size=*/IntegerAttr{},
                                   /*l1_hint=*/xegpu::CachePolicyAttr{},
                                   /*l2_hint=*/xegpu::CachePolicyAttr{},
                                   /*l3_hint=*/xegpu::CachePolicyAttr{},
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..c0178035b59d5 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -61,6 +61,30 @@ static bool isWriteHintOrNone(const CachePolicyAttr &attr) {
          kind == CachePolicy::WRITE_BACK || kind == CachePolicy::WRITE_THROUGH;
 }
 
+// Infer the chunk size (number of contiguous elements per work item) of a
+// gather/scatter op from its value/result and mask types. The chunk size is no
+// longer carried as an attribute; it is fully determined by the types. The
+// mask carries one element per lane, so it is the value with the chunk
+// dimension removed. The chunk size is therefore the trailing value dimension
+// whenever the value has more elements than the mask, and 1 otherwise:
+//   - scalar value                                   -> 1
+//   - value vector<LxC...>, mask vector<L...>        -> C (trailing dim)
+//   - 1D value vector<N...>, scalar/size-1 mask      -> N (lane-level chunk)
+//   - value and mask with matching element counts    -> 1 (one elem per lane)
+static int64_t inferGatherScatterChunkSize(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 static LogicalResult
 isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
                                  VectorType valueTy, int64_t chunkSize,
@@ -113,23 +137,23 @@ isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
   return success();
 }
 
-// Validates the `contiguity` attribute against the op's offsets type: the
+// Validates the `chunk_size` attribute against the op's offsets type: the
 // innermost offsets dimension is contiguous in runs of `size`, so `size` must
 // be >= 2 and must divide that dimension.
 static LogicalResult
-isValidContiguity(std::optional<uint64_t> contiguity, Type offsetsTy,
-                  function_ref<InFlightDiagnostic()> emitError) {
-  if (!contiguity)
+isValidChunkSize(std::optional<uint64_t> chunkSize, Type offsetsTy,
+                 function_ref<InFlightDiagnostic()> emitError) {
+  if (!chunkSize)
     return success();
   auto offsetsVecTy = dyn_cast<VectorType>(offsetsTy);
   if (!offsetsVecTy)
-    return emitError() << "contiguity requires vector offsets (one per lane).";
-  int64_t size = static_cast<int64_t>(*contiguity);
+    return emitError() << "chunk_size requires vector offsets (one per lane).";
+  int64_t size = static_cast<int64_t>(*chunkSize);
   int64_t inner = offsetsVecTy.getShape().back();
   if (size < 2)
-    return emitError() << "contiguity = " << size << " (must be >= 2)";
+    return emitError() << "chunk_size = " << size << " (must be >= 2)";
   if (inner % size != 0)
-    return emitError() << "contiguity = " << size
+    return emitError() << "chunk_size = " << size
                        << " (must divide the innermost offsets dim " << inner
                        << ")";
   return success();
@@ -592,7 +616,7 @@ LogicalResult LoadGatherOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto srcTy = getSourceType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(srcTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -604,8 +628,8 @@ LogicalResult LoadGatherOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -614,7 +638,7 @@ LogicalResult LoadGatherOp::verify() {
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint) {
   auto loc = source.getLoc();
@@ -623,15 +647,14 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, /*anchor_layout=*/nullptr,
-        /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint,
                          DistributeLayoutAttr layout) {
@@ -641,8 +664,8 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
@@ -662,7 +685,7 @@ LogicalResult StoreScatterOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto destTy = getDestType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(destTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -674,8 +697,8 @@ LogicalResult StoreScatterOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -684,7 +707,6 @@ LogicalResult StoreScatterOp::verify() {
 void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
                            Value value, Value dest,
                            ArrayRef<OpFoldResult> offsets, Value mask,
-                           IntegerAttr chunk_size,
                            xegpu::CachePolicyAttr l1_hint,
                            xegpu::CachePolicyAttr l2_hint,
                            xegpu::CachePolicyAttr l3_hint) {
@@ -695,15 +717,17 @@ void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, /*anchor_layout=*/nullptr, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
-void StoreScatterOp::build(
-    OpBuilder &builder, OperationState &state, Value value, Value dest,
-    ArrayRef<OpFoldResult> offsets, Value mask, IntegerAttr chunk_size,
-    xegpu::CachePolicyAttr l1_hint, xegpu::CachePolicyAttr l2_hint,
-    xegpu::CachePolicyAttr l3_hint, DistributeLayoutAttr layout) {
+void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
+                           Value value, Value dest,
+                           ArrayRef<OpFoldResult> offsets, Value mask,
+                           xegpu::CachePolicyAttr l1_hint,
+                           xegpu::CachePolicyAttr l2_hint,
+                           xegpu::CachePolicyAttr l3_hint,
+                           DistributeLayoutAttr layout) {
   auto loc = dest.getLoc();
   int64_t size = static_cast<int64_t>(offsets.size());
   auto type = VectorType::get(size, builder.getIndexType());
@@ -711,8 +735,8 @@ void StoreScatterOp::build(
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
index 3cc8152561681..64f1d6b992e4f 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp
@@ -872,7 +872,7 @@ using ::mlir::xegpu::detail::axis_dataflow::AxisInfoLattice;
 // Analysis driver.
 //===----------------------------------------------------------------------===//
 
-/// Stamp a `contiguity` attribute on `op` recording the inner-dim contiguity
+/// Stamp a `chunk_size` attribute on `op` recording the inner-dim contiguity
 /// computed by the analysis. The contiguity is a target-independent property
 /// of the offsets.
 template <typename OpTy>
@@ -880,9 +880,9 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
   auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
   if (!offsetsTy || offsetsTy.getNumElements() <= 1)
     return;
-  // A pre-existing `contiguity` (user-authored, or stamped by an earlier run)
+  // A pre-existing `chunk_size` (user-authored, or stamped by an earlier run)
   // takes precedence; leave it untouched so the analysis is idempotent.
-  if (op.getContiguity())
+  if (op.getChunkSize())
     return;
   const auto *lat = solver.lookupState<AxisInfoLattice>(op.getOffsets());
   if (!lat || !lat->getValue().isInitialized())
@@ -898,7 +898,7 @@ static void analyzeAndStampContiguity(OpTy op, DataFlowSolver &solver) {
     --contiguity;
   if (contiguity < 2)
     return;
-  op.setContiguity(contiguity);
+  op.setChunkSize(contiguity);
 }
 
 } // namespace
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 1681d295ae0ff..d1e27094564cb 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1218,7 +1218,8 @@ void LayoutInfoPropagation::visitLoadGatherOp(
   if (!uArch)
     return;
   VectorType resVecTy = load.getValueType();
-  int chunkSize = load.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(resVecTy, load.getMaskType());
 
   LayoutInfo resLayoutInfo = results[0]->getValue();
   if (!resLayoutInfo.isAssigned())
@@ -1283,7 +1284,8 @@ void LayoutInfoPropagation::visitStoreScatterOp(
   if (!uArch)
     return;
   VectorType srcVecTy = storeScatter.getValueType();
-  int chunkSize = storeScatter.getChunkSize().value_or(1);
+  int chunkSize =
+      xegpu::getGatherScatterPayloadChunk(srcVecTy, storeScatter.getMaskType());
 
   if (hasParamsOfLayoutKind(anchorLayoutAttr)) {
     requiredAnchorLayoutAttr = anchorLayoutAttr;
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..2ac3d1ff2bba5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -543,7 +543,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
       return failure();
 
     // Check that leading dimensions are unit.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origResultTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origResultTy.getShape();
     if (llvm::any_of(
@@ -581,8 +582,8 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
     Value distSource = adaptor.getSource();
     auto newOp = xegpu::LoadGatherOp::create(
         rewriter, op.getLoc(), distResultTy1D, distSource, distOffsets,
-        distMask, op.getChunkSizeAttr(), op.getL1HintAttr(), op.getL2HintAttr(),
-        op.getL3HintAttr(), /*layout=*/nullptr, /*contiguity=*/nullptr);
+        distMask, op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(),
+        /*layout=*/nullptr, /*contiguity=*/nullptr);
 
     Value result = newOp->getResult(0);
     if (distResultTy1D != distResultTy)
@@ -1071,7 +1072,8 @@ struct SgToLaneStoreScatter
       return failure();
 
     // Check that all leading dimensions are unit dimensions.
-    int chunkSize = op.getChunkSize().value_or(1);
+    int chunkSize =
+        xegpu::getGatherScatterPayloadChunk(origValueTy, op.getMaskType());
     int effectiveVecRank = (chunkSize == 1) ? 1 : 2;
     ArrayRef<int64_t> shape = origValueTy.getShape();
     if (llvm::any_of(shape.take_front(origValueTy.getRank() - effectiveVecRank),
@@ -1112,10 +1114,9 @@ struct SgToLaneStoreScatter
 
     Value distDest = adaptor.getDest();
     xegpu::StoreScatterOp::create(rewriter, op.getLoc(), distValue, distDest,
-                                  distOffsets, distMask, op.getChunkSizeAttr(),
-                                  op.getL1HintAttr(), op.getL2HintAttr(),
-                                  op.getL3HintAttr(), /*layout=*/nullptr,
-                                  /*contiguity=*/nullptr);
+                                  distOffsets, distMask, op.getL1HintAttr(),
+                                  op.getL2HintAttr(), op.getL3HintAttr(),
+                                  /*layout=*/nullptr, /*contiguity=*/nullptr);
     rewriter.eraseOp(op);
     return success();
   }
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index 74c358cef90df..3841ae3070490 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -699,11 +699,8 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       return failure();
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     // Unroll mask and offsets with correct shape
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -721,7 +718,6 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -763,8 +759,7 @@ struct UnrollLoadGatherOp : public UnrollPattern<xegpu::LoadGatherOp> {
     SmallVector<Value> newOps;
     for (auto [o, m] : llvm::zip(convertedOffsets, convertedMasks)) {
       auto newOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newValueTy, op.getSource(), o, m,
-          rewriter.getI64IntegerAttr(chunkSize), op.getL1HintAttr(),
+          rewriter, loc, newValueTy, op.getSource(), o, m, op.getL1HintAttr(),
           op.getL2HintAttr(), op.getL3HintAttr(), layout,
           /*contiguity=*/nullptr);
       newOps.push_back(newOp);
@@ -793,11 +788,8 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     if (!targetShape)
       return failure();
 
-    int64_t chunkSize = 1;
-    if (auto chunkSizeAttr = op->getAttr("chunk_size")) {
-      if (auto intAttr = llvm::dyn_cast<IntegerAttr>(chunkSizeAttr))
-        chunkSize = intAttr.getInt();
-    }
+    int64_t chunkSize =
+        xegpu::getGatherScatterPayloadChunk(valueTy, mask.getType());
 
     SmallVector<int64_t> targetMaskShape(*targetShape);
     VectorType maskTy = llvm::dyn_cast<VectorType>(mask.getType());
@@ -812,7 +804,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
       targetMaskShape.pop_back();
       int64_t blockedChunkSize = targetShape->back();
       int64_t numNewChunks = chunkSize / blockedChunkSize;
-      chunkSize = blockedChunkSize;
 
       convertedMaskTypes = getUnrolledTypes(maskTy, targetMaskShape);
       convertedOffsetTypes = getUnrolledTypes(offsetsTy, targetMaskShape);
@@ -859,7 +850,6 @@ struct UnrollStoreScatterOp : public UnrollPattern<xegpu::StoreScatterOp> {
     for (auto [v, o, m] :
          llvm::zip(convertedValues, convertedOffsets, convertedMasks)) {
       xegpu::StoreScatterOp::create(rewriter, loc, v, op.getDest(), o, m,
-                                    rewriter.getI64IntegerAttr(chunkSize),
                                     op.getL1HintAttr(), op.getL2HintAttr(),
                                     op.getL3HintAttr(), layout,
                                     /*contiguity=*/nullptr);
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..8cd1b1ad0d3f1 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -824,14 +824,12 @@ struct WgToSgLoadGatherOp : public OpConversionPattern<xegpu::LoadGatherOp> {
     }
 
     SmallVector<Value> newLoadOps;
-    auto chunkSizeAttr =
-        rewriter.getI64IntegerAttr(op.getChunkSize().value_or(1));
     VectorType newTy = VectorType::get(sgShape, resultType.getElementType());
     for (auto [offsets, mask] :
          llvm::zip(adaptor.getOffsets(), adaptor.getMask())) {
       auto newLayout = layout.dropSgLayoutAndData();
       auto newLoadOp = xegpu::LoadGatherOp::create(
-          rewriter, loc, newTy, op.getSource(), offsets, mask, chunkSizeAttr,
+          rewriter, loc, newTy, op.getSource(), offsets, mask,
           op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(), newLayout,
           /*contiguity=*/nullptr);
       newLoadOps.push_back(newLoadOp);
@@ -871,16 +869,12 @@ struct WgToSgStoreScatterOp
                                          "offsets have not been distributed");
     }
 
-    auto chunkSizeOpt = op.getChunkSize();
-    int64_t chunkSize = chunkSizeOpt ? static_cast<int64_t>(*chunkSizeOpt) : 1;
-    auto chunkSizeAttr = rewriter.getI64IntegerAttr(chunkSize);
     for (auto [val, offs, mask] : llvm::zip(
              adaptor.getValue(), adaptor.getOffsets(), adaptor.getMask())) {
-      xegpu::StoreScatterOp::create(rewriter, loc, val, op.getDest(), offs,
-                                    mask, chunkSizeAttr, op.getL1HintAttr(),
-                                    op.getL2HintAttr(), op.getL3HintAttr(),
-                                    layout.dropSgLayoutAndData(),
-                                    /*contiguity=*/nullptr);
+      xegpu::StoreScatterOp::create(
+          rewriter, loc, val, op.getDest(), offs, mask, op.getL1HintAttr(),
+          op.getL2HintAttr(), op.getL3HintAttr(), layout.dropSgLayoutAndData(),
+          /*contiguity=*/nullptr);
     }
     rewriter.eraseOp(op);
     return success();
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..92759fc7a69a6 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -40,6 +40,20 @@ SmallVector<Value> xegpu::flattenValues(ArrayRef<ValueRange> values) {
   return result;
 }
 
+int64_t xegpu::getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 FailureOr<VectorType>
 mlir::xegpu::getDistributedVectorType(xegpu::TensorDescType tdescTy) {
   auto layout = llvm::dyn_cast_if_present<LayoutAttr>(tdescTy.getLayout());
@@ -237,7 +251,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
 
     if (isa<xegpu::StoreScatterOp>(op)) {
       xegpu::StoreScatterOp store(op);
-      int chunkSize = store.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(store.getValueType(),
+                                                          store.getMaskType());
       if (layout && idx >= 2 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
@@ -245,7 +260,8 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
     }
     if (isa<xegpu::LoadGatherOp>(op)) {
       xegpu::LoadGatherOp load(op);
-      int chunkSize = load.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(load.getValueType(),
+                                                          load.getMaskType());
       if (layout && idx >= 1 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
             llvm::seq<int64_t>(layout.getRank() - 1, layout.getRank())));
diff --git a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
index f223b325c2f9e..df9f7fe69c2e2 100644
--- a/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
+++ b/mlir/test/Dialect/XeGPU/contiguity-analysis.mlir
@@ -1,7 +1,7 @@
 // RUN: mlir-opt -split-input-file \
 // RUN:   -test-xegpu-coalesce-gather-scatter="analyze-only=true" %s | FileCheck %s
 
-// Contiguity analysis: stamps the `contiguity` attribute on gather/scatter ops
+// Contiguity analysis: stamps the `chunk_size` attribute on gather/scatter ops
 // whose offsets are contiguous (runs of >= 2) along the innermost dimension.
 // The stamped value is the inner-dim contiguity, rounded down to a divisor of
 // the inner extent. The analysis is target-independent and mask-independent;
@@ -11,7 +11,7 @@
 // 1-D vector.step -> stride-1, fully contiguous over the 32-element inner dim.
 // CHECK-LABEL: func.func @load_step_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
@@ -25,7 +25,7 @@ func.func @load_step_offsets(%ptr: i64) -> vector<32xf32> {
 // Dense stride-1 constant offsets -> contiguity 32.
 // CHECK-LABEL: func.func @load_dense_ap_offsets(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
   %offsets = arith.constant dense<[
     0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
@@ -41,7 +41,7 @@ func.func @load_dense_ap_offsets(%ptr: i64) -> vector<32xi32> {
 // Stride-4 offsets: not contiguous, no attribute stamped.
 // CHECK-LABEL: func.func @load_stride4_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
   %c4 = arith.constant 4 : index
@@ -58,7 +58,7 @@ func.func @load_stride4_no_attr(%ptr: i64) -> vector<32xf32> {
 // All-equal offsets: inner dim is constant, not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_broadcast_offsets_no_attr(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
   %offsets = arith.constant dense<0> : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -72,7 +72,7 @@ func.func @load_broadcast_offsets_no_attr(%ptr: i64) -> vector<32xf32> {
 // (the mask check is a consumer concern).
 // CHECK-LABEL: func.func @load_partial_mask(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %v = xegpu.load %ptr[%offsets], %mask
@@ -84,7 +84,7 @@ func.func @load_partial_mask(%ptr: i64, %mask: vector<32xi1>) -> vector<32xf32>
 // Store with vector.step offsets -> contiguity 32 on the store.
 // CHECK-LABEL: func.func @store_step_offsets(
 // CHECK: xegpu.store
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
@@ -97,7 +97,7 @@ func.func @store_step_offsets(%ptr: i64, %v: vector<32xf32>) {
 // 2-D leading-unit dim: contiguity measured on the inner dim -> 32.
 // CHECK-LABEL: func.func @load_2d_leading_unit(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
   %step = vector.step : vector<32xindex>
   %offsets = vector.shape_cast %step : vector<32xindex> to vector<1x32xindex>
@@ -111,7 +111,7 @@ func.func @load_2d_leading_unit(%ptr: i64) -> vector<1x32xf32> {
 // True 2-D dense AP: each row stride-1 over 16 -> contiguity 16.
 // CHECK-LABEL: func.func @load_2d_dense_ap(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 16 : i64}>
+// CHECK-SAME: <{chunk_size = 16 : i64}>
 func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
@@ -127,7 +127,7 @@ func.func @load_2d_dense_ap(%ptr: i64) -> vector<2x16xf32> {
 // 2-D dense values whose inner row is not a stride-1 AP: no attribute.
 // CHECK-LABEL: func.func @load_2d_non_ap(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
   %offsets = arith.constant dense<[
     [0, 1, 2, 3, 4, 5, 6, 7, 100, 9, 10, 11, 12, 13, 14, 15],
@@ -146,7 +146,7 @@ func.func @load_2d_non_ap(%ptr: i64) -> vector<2x16xf32> {
 // exercised even if the solver later folds all-constant arith ops.
 // CHECK-LABEL: func.func @load_divui_recovers(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %c2 = arith.constant dense<2> : vector<32xindex>
@@ -162,7 +162,7 @@ func.func @load_divui_recovers(%ptr: i64) -> vector<32xf32> {
 // `divui` by a constant that does not divide the inner stride: not recovered.
 // CHECK-LABEL: func.func @load_divui_non_divisor(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -180,7 +180,7 @@ func.func @load_divui_non_divisor(%ptr: i64) -> vector<16xf32> {
 // not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_remui_inner_uniform(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
   %step = vector.step : vector<16xindex>
   %c2 = arith.constant dense<2> : vector<16xindex>
@@ -196,7 +196,7 @@ func.func @load_remui_inner_uniform(%ptr: i64) -> vector<16xf32> {
 // `shli` then `shrui` cancel: (step << 1) >> 1 -> stride 1 -> contiguity 32.
 // CHECK-LABEL: func.func @load_shli_then_shrui(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -212,7 +212,7 @@ func.func @load_shli_then_shrui(%ptr: i64) -> vector<32xf32> {
 // `shli` alone scales the stride to 2: not contiguous -> no attribute.
 // CHECK-LABEL: func.func @load_shli_unchanged(
 // CHECK: xegpu.load
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %k = arith.constant dense<1> : vector<32xindex>
@@ -228,7 +228,7 @@ func.func @load_shli_unchanged(%ptr: i64) -> vector<32xf32> {
 // contiguity 32.
 // CHECK-LABEL: func.func @load_select_two_aps(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 32 : i64}>
+// CHECK-SAME: <{chunk_size = 32 : i64}>
 func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
   %step = vector.step : vector<32xindex>
   %a = arith.constant dense<0>  : vector<32xindex>
@@ -243,14 +243,14 @@ func.func @load_select_two_aps(%ptr: i64, %cond: i1) -> vector<32xf32> {
 }
 
 // -----
-// A pre-existing `contiguity` takes precedence: the analysis leaves it alone.
+// A pre-existing `chunk_size` takes precedence: the analysis leaves it alone.
 // CHECK-LABEL: func.func @user_attr_preserved(
 // CHECK: xegpu.load
-// CHECK-SAME: <{contiguity = 2 : i64}>
+// CHECK-SAME: <{chunk_size = 2 : i64}>
 func.func @user_attr_preserved(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
   %mask = arith.constant dense<true> : vector<32xi1>
-  %v = xegpu.load %ptr[%offsets], %mask <{contiguity = 2 : i64}>
+  %v = xegpu.load %ptr[%offsets], %mask <{chunk_size = 2 : i64}>
       : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
   return %v : vector<32xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..c610ded4a5b50 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -233,7 +233,7 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
   %offsets = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<8xi1>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}>
+  %2 = xegpu.load %src[%offsets], %mask
       : memref<?xf32>, vector<4xindex>, vector<8xi1> -> vector<4x2xf32>
   return
 }
@@ -242,8 +242,8 @@ func.func @load_gather_vc_3(%src: memref<?xf32>) {
 func.func @load_gather_simt_1(%src: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  %2 = xegpu.load %src[%0], %1
       : memref<?xf32>, vector<4xindex>, vector<4xi1> -> vector<6xf32>
   return
 }
@@ -265,7 +265,7 @@ func.func @store_scatter_vc_3(%dst: memref<?xf32>) {
   %1 = arith.constant dense<1>: vector<8xi1>
   %2 = arith.constant dense<2.9>: vector<4x2xf32>
   // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  xegpu.store %2, %dst[%0], %1
       : vector<4x2xf32>, memref<?xf32>, vector<4xindex>, vector<8xi1>
   return
 }
@@ -275,8 +275,8 @@ func.func @store_scatter_simt_1(%dst: memref<?xf32>) {
   %0 = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %1 = arith.constant dense<1>: vector<4xi1>
   %2 = arith.constant dense<2.9>: vector<6xf32>
-  // expected-error at +1 {{value elements must match chunk size}}
-  xegpu.store %2, %dst[%0], %1 <{chunk_size = 2}>
+  // expected-error at +1 {{Mask should match value except the chunk size dim}}
+  xegpu.store %2, %dst[%0], %1
       : vector<6xf32>, memref<?xf32>, vector<4xindex>, vector<4xi1>
   return
 }
@@ -316,26 +316,6 @@ func.func @load_gather_offset_sg(%src: memref<?xf16>) {
   return
 }
 
-// -----
-func.func @load_gather_offset_wi(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf32>
-  return
-}
-
-// -----
-func.func @store_scatter_offset_wi_1(%src: memref<?xf16>) {
-  %val = arith.constant dense<2.9>: vector<4xf16>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  %mask = arith.constant dense<1>: vector<1xi1>
-  // expected-error at +1 {{Mask should match value except the chunk size dim}}
-  xegpu.store %val, %src[%offsets], %mask
-        : vector<4xf16>, memref<?xf16>, vector<1xindex>, vector<1xi1>
-  return
-}
-
 // -----
 func.func @store_scatter_offset_wi_2(%src: memref<4x4xf16>) {
   %val = arith.constant dense<2.9>: vector<4xf16>
@@ -363,16 +343,7 @@ func.func @load_gather_offset_wi_4(%src: !xegpu.tensor_desc<1x2xf16>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
-  return
-}
-
-// -----
-func.func @load_gather_offset_wi_2(%src: ui64) {
-  %mask = arith.constant dense<1>: vector<1xi1>
-  %offsets = arith.constant dense<[0]> : vector<1xindex>
-  // expected-error at +1 {{value elements must match chunk size}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : ui64,  vector<1xindex>, vector<1xi1> -> vector<3xf16>
+  %2 = xegpu.load %src[%offsets], %mask : !xegpu.tensor_desc<1x2xf16>, vector<1xindex>, vector<1xi1> -> vector<2xf16>
   return
 }
 
@@ -381,7 +352,7 @@ func.func @load_gather_offset_wi_1(%src: memref<4x4xf32>) {
   %mask = arith.constant dense<1>: vector<1xi1>
   %offsets = arith.constant dense<[0]> : vector<1xindex>
   // expected-error at +1 {{op operand #0 must be 1D memref}}
-  %2 = xegpu.load %src[%offsets], %mask <{chunk_size = 2}> : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
+  %2 = xegpu.load %src[%offsets], %mask : memref<4x4xf32>,  vector<1xindex>, vector<1xi1> -> vector<2xf32>
   return
 }
 
@@ -792,16 +763,16 @@ func.func @dpas_mx_scale_b_layout_not_distributable(%a : vector<8x16xf8E5M2>, %b
 
 // -----
 func.func @contiguity_too_small(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // expected-error at +1 {{contiguity = 1 (must be >= 2)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 1 : i64}>
+  // expected-error at +1 {{chunk_size = 1 (must be >= 2)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 1 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   return
 }
 
 // -----
 func.func @contiguity_does_not_divide(%src: i64, %offset: vector<6xindex>, %mask: vector<6xi1>) {
-  // expected-error at +1 {{contiguity = 4 (must divide the innermost offsets dim 6)}}
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // expected-error at +1 {{chunk_size = 4 (must divide the innermost offsets dim 6)}}
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<6xindex>, vector<6xi1> -> vector<6xf32>
   return
 }
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..2b82d426eba39 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -377,8 +377,8 @@ gpu.func @prefetch_nd_3d(%src: memref<4x8x16xf16>) {
 
 // CHECK: gpu.func @simt_load_4(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: vector<1xindex>, %[[arg2:.*]]: vector<1xi1>) {
 gpu.func @simt_load_4(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vector<1xi1>) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
   gpu.return
 }
 
@@ -391,8 +391,8 @@ gpu.func @simt_load_5(%arg0: memref<256xf16>, %arg1: vector<1xindex>, %arg2: vec
 
 // CHECK: gpu.func @simt_load_6(%[[arg0:.*]]: memref<256xf16>, %[[arg1:.*]]: index, %[[arg2:.*]]: i1) {
 gpu.func @simt_load_6(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
-  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
-  %0 = xegpu.load %arg0[%arg1], %arg2 <{chunk_size = 8 : i64}> : memref<256xf16>, index, i1 -> vector<8xf16>
+  // CHECK: %0 = xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] : memref<256xf16>, index, i1 -> vector<8xf16>
+  %0 = xegpu.load %arg0[%arg1], %arg2 : memref<256xf16>, index, i1 -> vector<8xf16>
   gpu.return
 }
 
@@ -407,23 +407,23 @@ gpu.func @simt_load_7(%arg0: memref<256xf16>, %arg1: index, %arg2: i1) {
 gpu.func @subgroup_load_offset_1(%src: memref<?xf16>) {
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
-  %val = xegpu.load %src[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: %[[R1:.*]] = xegpu.load %arg0[%cst], %cst_0 <{l1_hint = #xegpu.cache_hint<cached>}> : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
+  %val = xegpu.load %src[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : memref<?xf16>, vector<4xindex>, vector<4xi1> -> vector<4x2xf16>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_4(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: vector<1xindex>, %[[arg3:.*]]: vector<1xi1>) {
 gpu.func @simt_store_4(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: vector<1xindex>, %arg3: vector<1xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @simt_store_5(%[[arg0:.*]]: vector<8xf16>, %[[arg1:.*]]: memref<256xf16>, %[[arg2:.*]]: index, %[[arg3:.*]]: i1) {
 gpu.func @simt_store_5(%arg0: vector<8xf16>, %arg1: memref<256xf16>, %arg2: index, %arg3: i1) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
-  xegpu.store %arg0, %arg1[%arg2], %arg3 <{chunk_size = 8 : i64}> : vector<8xf16>, memref<256xf16>, index, i1
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] : vector<8xf16>, memref<256xf16>, index, i1
+  xegpu.store %arg0, %arg1[%arg2], %arg3 : vector<8xf16>, memref<256xf16>, index, i1
   gpu.return
 }
 
@@ -446,25 +446,25 @@ gpu.func @subgroup_store_offset_1(%dest: memref<?xf16>) {
   %val = arith.constant dense<2.9>: vector<4x2xf16>
   %offset = arith.constant dense<[0, 8, 16, 24]> : vector<4xindex>
   %mask = arith.constant dense<1>: vector<4xi1>
-  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
-  xegpu.store %val, %dest[%offset], %mask <{chunk_size=2, l1_hint = #xegpu.cache_hint<cached>}>
+  //CHECK: xegpu.store %[[R0:.*]], %arg0[%cst_0], %cst_1 <{l1_hint = #xegpu.cache_hint<cached>}> : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
+  xegpu.store %val, %dest[%offset], %mask <{l1_hint = #xegpu.cache_hint<cached>}>
       : vector<4x2xf16>, memref<?xf16>, vector<4xindex>, vector<4xi1>
   gpu.return
 }
 
 // CHECK: gpu.func @load_contiguity(%[[arg0:.*]]: i64, %[[arg1:.*]]: vector<16xindex>, %[[arg2:.*]]: vector<16xi1>) {
 gpu.func @load_contiguity(%src: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // A user-provided `contiguity` round-trips through the optional op attribute.
-  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{contiguity = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
-  %val = xegpu.load %src[%offset], %mask <{contiguity = 4 : i64}>
+  // A user-provided `chunk_size` round-trips through the optional op attribute.
+  // CHECK: xegpu.load %[[arg0]][%[[arg1]]], %[[arg2]] <{chunk_size = 4 : i64}> : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  %val = xegpu.load %src[%offset], %mask <{chunk_size = 4 : i64}>
       : i64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.return
 }
 
 // CHECK: gpu.func @store_contiguity(%[[arg0:.*]]: vector<16xf32>, %[[arg1:.*]]: i64, %[[arg2:.*]]: vector<16xindex>, %[[arg3:.*]]: vector<16xi1>) {
 gpu.func @store_contiguity(%val: vector<16xf32>, %dest: i64, %offset: vector<16xindex>, %mask: vector<16xi1>) {
-  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{contiguity = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
-  xegpu.store %val, %dest[%offset], %mask <{contiguity = 4 : i64}>
+  // CHECK: xegpu.store %[[arg0]], %[[arg1]][%[[arg2]]], %[[arg3]] <{chunk_size = 4 : i64}> : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
+  xegpu.store %val, %dest[%offset], %mask <{chunk_size = 4 : i64}>
       : vector<16xf32>, i64, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index fa00c1d894d8f..402700fd4c5e3 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -615,9 +615,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<sg_layout = [16], sg_data = [16]>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
@@ -646,9 +646,9 @@ gpu.module @test {
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: xegpu.store %{{.*}}, %{{.*}}[%{{.*}}], %{{.*}} <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, l1_hint = #xegpu.cache_hint<cached>}
+    xegpu.store %val, %dest[%offset], %mask {l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..b8b8d86cf8b10 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -642,8 +642,8 @@ gpu.module @test{
     %5 = vector.broadcast %3 : index to vector<1xindex>
     %6 = arith.addi %4, %5 : vector<1xindex>
     %7 = vector.broadcast %6 : vector<1xindex> to vector<1x1x1x16xindex>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
-    xegpu.store %cst, %0[%7], %cst_0 <{chunk_size = 1 : i64}> : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
+    xegpu.store %cst, %0[%7], %cst_0 : vector<1x1x1x16xf32>, i64, vector<1x1x1x16xindex>, vector<1x1x1x16xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..3ced1d7e9c6f5 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -260,14 +260,14 @@ gpu.func @prefetch_nd() {
 // CHECK-LABEL: gpu.func @scatter_load_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[CAST:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   gpu.return
 }
@@ -275,20 +275,20 @@ gpu.func @scatter_load_chunksize(%src: memref<256xf16>) {
 // CHECK-LABEL: gpu.func @scatter_store_chunksize
 // CHECK: %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
-// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: %[[LOAD:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK: %[[C1:.*]] = vector.shape_cast %[[LOAD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK: %[[C2:.*]] = vector.shape_cast %[[C1]] : vector<1x8xf16> to vector<8xf16>
-// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK: xegpu.store %[[C2]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.func @scatter_store_chunksize(%src: memref<256xf16>) {
   %offset = arith.constant dense<12> : vector<16xindex>
   %mask = arith.constant dense<true> : vector<16xi1>
   %0 = xegpu.load %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
   xegpu.store %0, %src[%offset], %mask
-    <{chunk_size = 8, layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
+    <{layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>}>
     : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
   gpu.return
 }
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index fa9897770a08e..e8a41f6fc99f0 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -257,7 +257,7 @@ gpu.module @xevm_module{
 // CHECK-DAG:      %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK-DAG:      %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:          %[[IF:.*]] = scf.if %[[PREDICATE]] -> (vector<1x8xf16>) {
-// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:        %[[LD:.*]] = xegpu.load %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:          : memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
 // CHECK-NEXT:        %[[LD_CAST:.*]] = vector.shape_cast %[[LD]] : vector<8xf16> to vector<1x8xf16>
 // CHECK-NEXT:        scf.yield %[[LD_CAST]] : vector<1x8xf16>
@@ -265,14 +265,14 @@ gpu.module @xevm_module{
 // CHECK-NEXT:        scf.yield %[[CST]] : vector<1x8xf16>
 // CHECK-NEXT:      }
 // CHECK-NEXT:      %[[IF_CAST:.*]] = vector.shape_cast %[[IF]] : vector<1x8xf16> to vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[IF_CAST]], %{{.*}}[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:        vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 gpu.module @xevm_module{
   gpu.func @scatter_ops_scf_yield(%src: memref<256xf16>, %pred : i1) {
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     %loaded = scf.if %pred -> (vector<16x8xf16>) {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
@@ -280,7 +280,7 @@ gpu.module @xevm_module{
       %3 = arith.constant dense<12.> : vector<16x8xf16>
       scf.yield %3 : vector<16x8xf16>
     }
-    xegpu.store %loaded, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+    xegpu.store %loaded, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     gpu.return
   }
 }
@@ -291,9 +291,9 @@ gpu.module @xevm_module{
 // CHECK:         %[[MASK:.*]] = arith.constant dense<true> : vector<1xi1>
 // CHECK:         %[[OFFSET:.*]] = arith.constant dense<12> : vector<1xindex>
 // CHECK:         scf.if %[[PREDICATE]] {
-// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      %[[LOADED:.*]] = xegpu.load %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         memref<256xf16>, vector<1xindex>, vector<1xi1> -> vector<8xf16>
-// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]] <{chunk_size = 8 : i64}>
+// CHECK-NEXT:      xegpu.store %[[LOADED]], %arg0[%[[OFFSET]]], %[[MASK]]
 // CHECK-SAME:         vector<8xf16>, memref<256xf16>, vector<1xindex>, vector<1xi1>
 // CHECK-NEXT:    }
 gpu.module @xevm_module{
@@ -302,10 +302,10 @@ gpu.module @xevm_module{
     %1 = arith.constant dense<1>: vector<16xi1>
     %offset = arith.constant dense<12> : vector<16xindex>
     scf.if %pred  {
-      %3 = xegpu.load %src[%offset], %1 <{chunk_size=8}> {
+      %3 = xegpu.load %src[%offset], %1 {
         layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>
       } : memref<256xf16>, vector<16xindex>, vector<16xi1> -> vector<16x8xf16>
-      xegpu.store %3, %src[%offset], %1 <{chunk_size=8}> {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
+      xegpu.store %3, %src[%offset], %1 {layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 2]>} : vector<16x8xf16>, memref<256xf16>, vector<16xindex>, vector<16xi1>
     }
     gpu.return
   }
diff --git a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
index fb32e0012679b..885026f5c85ae 100644
--- a/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
+++ b/mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir
@@ -99,7 +99,7 @@ gpu.module @kernel_chunk4 [#xevm.target<chip = "pvc">] {
 // CHECK-LABEL: func.func @load_no_target_unchanged(
 // CHECK: xegpu.load
 // CHECK-NOT: lane_data
-// CHECK-NOT: contiguity
+// CHECK-NOT: chunk_size
 // CHECK-SAME: : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 func.func @load_no_target_unchanged(%ptr: i64) -> vector<32xf32> {
   %offsets = vector.step : vector<32xindex>
diff --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index b0b16c2adba6e..3d769c86ee2cc 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -334,8 +334,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -353,8 +353,8 @@ gpu.module @test_kernel {
       128, 136, 144, 152, 160, 168, 176, 184,
       192, 200, 208, 216, 224, 232, 240, 248
     ]> : vector<32xindex>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%cst], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%cst], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -369,8 +369,8 @@ gpu.module @test_kernel {
     //CHECK: arith.addi [[step]], [[cst]] : vector<16xindex>
     %step = vector.step : vector<32xindex>
     %mask = vector.create_mask %c16 : vector<32xi1>
-    %ld = xegpu.load %src[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    xegpu.store %ld, %dst[%step], %mask {chunk_size = 1, layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+    %ld = xegpu.load %src[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    xegpu.store %ld, %dst[%step], %mask {layout = #l, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -514,7 +514,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: load_with_offsets
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -525,7 +525,7 @@ gpu.module @test_kernel {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17 : vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -534,7 +534,7 @@ gpu.module @test_kernel {
 // -----
 gpu.module @test_kernel {
   // CHECK-LABEL: store_with_offsets
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -547,7 +547,7 @@ gpu.module @test_kernel {
       %mask = vector.create_mask %c17 : vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -561,7 +561,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -572,7 +572,7 @@ gpu.module @test_kernel {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17 : vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 }
@@ -585,7 +585,7 @@ gpu.module @test_kernel {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -598,7 +598,7 @@ gpu.module @test_kernel {
     %mask = vector.create_mask %c17 : vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 }
@@ -611,8 +611,8 @@ gpu.module @test_kernel {
   // CHECK: [[cst_0:%.+]] = arith.constant dense<true> : vector<1x1x16xi1>
   // CHECK: [[cst_1:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
   // CHECK: [[cst_2:%.+]] = arith.constant dense<{{.*}}> : vector<1x1x16xindex>
-  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
-  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_0:%.+]] = xegpu.load [[arg0]][[[cst_1]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
+  // CHECK: [[ld_1:%.+]] = xegpu.load [[arg0]][[[cst_2]]], [[cst_0]] <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<1x1x16xindex>, vector<1x1x16xi1> -> vector<1x1x16xf32>
   // CHECK: [[ins_0:%.+]] = vector.insert_strided_slice [[ld_0]], [[cst]] {offsets = [0, 0, 0], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   // CHECK: [[ins_1:%.+]] = vector.insert_strided_slice [[ld_1]], [[ins_0]] {offsets = [0, 0, 16], strides = [1, 1, 1]} : vector<1x1x16xf32> into vector<1x1x32xf32>
   gpu.func @preserve_unit_dim_of_load_inst_data(%src: ui64) -> vector<1x1x32xf32> {
@@ -624,7 +624,7 @@ gpu.module @test_kernel {
       ]]> : vector<1x1x32xindex>
 
       %mask = arith.constant dense<true> : vector<1x1x32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [1, 1, 16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
 
       gpu.return %ld : vector<1x1x32xf32>
   }
@@ -692,10 +692,10 @@ gpu.module @test_kernel {
         128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]]
     ]> : vector<1x1x32xindex>
     %mask = arith.constant dense<true> : vector<1x1x32xi1>
-    %a = xegpu.load %A[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
-    %b = xegpu.load %B[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %a = xegpu.load %A[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    %b = xegpu.load %B[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     %addf = arith.addf %a, %b : vector<1x1x32xf32>
-    xegpu.store %addf, %C[%cst], %mask {chunk_size = 1, layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
+    xegpu.store %addf, %C[%cst], %mask {layout = #inst_data, l1_hint = #xegpu.cache_hint<cached>} : vector<1x1x32xf32>, ui64, vector<1x1x32xindex>, vector<1x1x32xi1>
     gpu.return
   }
 }
diff --git a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
index 49fdf1cbee174..75492d8e248d3 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir
@@ -128,7 +128,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: load_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
+  // CHECK-COUNT-2: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16xf32>
   gpu.func @load_with_offsets(%src: ui64) -> vector<32xf32> {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -139,7 +139,7 @@ gpu.module @test {
 
       %c17 = arith.constant 17: index
       %mask = vector.create_mask %c17: vector<32xi1>
-      %ld = xegpu.load %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+      %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
 
       gpu.return %ld : vector<32xf32>
   }
@@ -147,7 +147,7 @@ gpu.module @test {
 //-----
   // CHECK-LABEL: store_with_offsets
   // CHECK-SAME: [[arg0:%.+]]: ui64
-  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-2: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets(%src: ui64) {
       %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -160,7 +160,7 @@ gpu.module @test {
       %mask = vector.create_mask %c17: vector<32xi1>
 
       %st_vec = arith.constant dense<1023.0>: vector<32xf32>
-      xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 1, layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
+      xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32xf32>, ui64, vector<32xindex>, vector<32xi1>
 
       gpu.return
   }
@@ -173,7 +173,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
+  // CHECK-COUNT-4: xegpu.load  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : ui64, vector<16xindex>, vector<16xi1> -> vector<16x2xf32>
    gpu.func @load_with_offsets_chunk(%src: ui64) -> vector<32x4xf32> {
     %cst = arith.constant dense<[
         0,   8,  16,  24,  32,  40,  48,  56,
@@ -184,7 +184,7 @@ gpu.module @test {
 
     %c17 = arith.constant 17: index
     %mask = vector.create_mask %c17: vector<32xi1>
-    %ld = xegpu.load %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
+    %ld = xegpu.load %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : ui64, vector<32xindex>, vector<32xi1> -> vector<32x4xf32>
     gpu.return %ld : vector<32x4xf32>
    }
 
@@ -196,7 +196,7 @@ gpu.module @test {
   // CHECK: [[cst1:%.+]] = arith.constant dense<[2, 10, 18, 26, 34, 42, 50, 58, 66, 74, 82, 90, 98, 106, 114, 122]> : vector<16xindex>
   // CHECK: [[cst2:%.+]] = arith.constant dense<[128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248]> : vector<16xindex>
   // CHECK: [[cst3:%.+]] = arith.constant dense<[0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120]> : vector<16xindex>
-  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{chunk_size = 2 : i64, l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
+  // CHECK-COUNT-4: xegpu.store  {{.*}}[{{.*}}], {{.*}} <{l1_hint = #xegpu.cache_hint<cached>}> : vector<16x2xf32>, ui64, vector<16xindex>, vector<16xi1>
   gpu.func @store_with_offsets_chunk(%src: ui64) {
     %cst = arith.constant dense<[
       0,   8,  16,  24,  32,  40,  48,  56,
@@ -209,7 +209,7 @@ gpu.module @test {
     %mask = vector.create_mask %c17: vector<32xi1>
 
     %st_vec = arith.constant dense<1023.>: vector<32x4xf32>
-    xegpu.store %st_vec, %src[%cst], %mask {chunk_size = 4, layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
+    xegpu.store %st_vec, %src[%cst], %mask {layout = #xegpu.layout<inst_data = [16, 2]>, l1_hint = #xegpu.cache_hint<cached>} : vector<32x4xf32>, ui64, vector<32xindex>, vector<32xi1>
     gpu.return
   }
 
diff --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
index 8c5dd6c55b99d..2ca6e8d46ef16 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
@@ -317,11 +317,11 @@ gpu.module @test_distribution {
   gpu.func @load_gather(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<32x4xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<32x4xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<32x4xindex>, vector<32x4xi1> -> vector<32x4xf16>
     %offset =  arith.constant dense<0> : vector<256x16xindex>
     %mask = arith.constant dense<1> : vector<256x16xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256x16xindex>, vector<256x16xi1> -> vector<256x16xf16>
     gpu.return
   }
@@ -332,12 +332,12 @@ gpu.module @test_distribution {
     // CHECK: %[[VAL:.*]] = arith.constant dense<2.550000e+01> : vector<8xf16>
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 1 : i64, l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
+    // CHECK: xegpu.store %[[VAL]], %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>, layout = #xegpu.layout<inst_data = [8]>}>
      // CHECK-SAME: : vector<8xf16>, memref<256xf16>, vector<8xindex>, vector<8xi1>
     %val = arith.constant dense<25.5> : vector<256xf16>
     %offset = arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    xegpu.store %val, %dest[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
+    xegpu.store %val, %dest[%offset], %mask {layout = #xegpu.layout<sg_layout = [32], sg_data = [8], inst_data = [8]>,
                                              l1_hint = #xegpu.cache_hint<cached>}
       : vector<256xf16>, memref<256xf16>, vector<256xindex>, vector<256xi1>
     gpu.return
@@ -348,11 +348,11 @@ gpu.module @test_distribution {
   gpu.func @load_with_non_unit_chunk_size(%src : memref<?xf16>) {
     // CHECK: %[[CST:.*]] = arith.constant dense<0> : vector<8xindex>
     // CHECK: %[[MASK:.*]] = arith.constant dense<true> : vector<8xi1>
-    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{chunk_size = 4 : i64, l1_hint = #xegpu.cache_hint<cached>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST]]], %[[MASK]] <{l1_hint = #xegpu.cache_hint<cached>}>
     // CHECK-SAME: : memref<?xf16>, vector<8xindex>, vector<8xi1> -> vector<8x4xf16>
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 4, layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [8, 4]>, l1_hint = #xegpu.cache_hint<cached>}
       : memref<?xf16>, vector<256xindex>, vector<256xi1> -> vector<256x4xf16>
     gpu.return
   }
@@ -783,9 +783,9 @@ gpu.module @test_distribution {
     %offset =  arith.constant dense<0> : vector<256xindex>
     %mask = arith.constant dense<1> : vector<256xi1>
 
-    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{chunk_size = 1 : i64, layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
+    // CHECK: %[[LOAD:.*]] = xegpu.load {{.*}} <{layout = #xegpu.slice<#xegpu.layout<inst_data = [8, 16]>, dims = [0]>}>
     // CHECK-SAME: memref<4096xf32>, vector<32xindex>, vector<32xi1> -> vector<32xf32>
-    %3 = xegpu.load %2[%offset], %mask {chunk_size = 1, layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
+    %3 = xegpu.load %2[%offset], %mask {layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>, dims = [0]> } : memref<4096xf32>, vector<256xindex>, vector<256xi1> -> vector<256xf32>
 
     // CHECK: %[[BROADCAST:.*]] = vector.broadcast %[[LOAD]] : vector<32xf32> to vector<32x32xf32>
     %4 = vector.broadcast %3 : vector<256xf32> to vector<256x256xf32>
@@ -803,7 +803,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<1.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0:.*]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32xindex>, vector<1x1x32xi1> -> vector<1x1x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x32xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [1] : vector<1x1x32xf32> to vector<1x32xf32>
     // CHECK-DAG: %[[CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x32xf32> to vector<1x1x32xf32>
@@ -820,7 +820,7 @@ gpu.module @test_distribution {
     %cst_3 = arith.constant dense<1.0> : vector<1x32xf32>
     %offset = arith.constant dense<0> : vector<1x32x32xindex>
     %mask = arith.constant dense<true> : vector<1x32x32xi1>
-    %14 = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
+    %14 = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>} : memref<?xf32>, vector<1x32x32xindex>, vector<1x32x32xi1> -> vector<1x32x32xf32>
     %15 = vector.multi_reduction <add>, %14, %cst_3 [1] : vector<1x32x32xf32> to vector<1x32xf32>
     %anchor = xegpu.convert_layout %15
       <{
@@ -878,7 +878,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<1x1x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<1x1x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<1x1x32x32xindex>, vector<1x1x32x32xi1> -> vector<1x1x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<1x1xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<1x1x32x32xf32> to vector<1x1xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<1x1xf32> to vector<1x1x1x1xf32>
@@ -895,7 +895,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<2x2xf32>
     %offset = arith.constant dense<0> : vector<2x2x128x128xindex>
     %mask = arith.constant dense<true> : vector<2x2x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>} : memref<?xf32>, vector<2x2x128x128xindex>, vector<2x2x128x128xi1> -> vector<2x2x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<2x2x128x128xf32> to vector<2x2xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -911,7 +911,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<0> : vector<16x16x32x32xindex>
     // CHECK-DAG: %[[CST_1:.*]] = arith.constant dense<true> : vector<16x16x32x32xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] <{chunk_size = 1 : i64}> : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %[[ARG0]][%[[CST_0]]], %[[CST_1]] : memref<?xf32>, vector<16x16x32x32xindex>, vector<16x16x32x32xi1> -> vector<16x16x32x32xf32>
     // CHECK-DAG: %[[CST_2:.*]] = arith.constant dense<0.000000e+00> : vector<16x16xf32>
     // CHECK-DAG: %[[LOCAL_REDUCE:.*]] = vector.multi_reduction <add>, %[[LOAD]], %[[CST_2]] [2, 3] : vector<16x16x32x32xf32> to vector<16x16xf32>
     // CHECK-DAG: %[[SHAPE_CAST:.*]] = vector.shape_cast %[[LOCAL_REDUCE]] : vector<16x16xf32> to vector<16x16x1x1xf32>
@@ -928,7 +928,7 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<0.0> : vector<32x32xf32>
     %offset = arith.constant dense<0> : vector<32x32x128x128xindex>
     %mask = arith.constant dense<true> : vector<32x32x128x128xi1>
-    %load = xegpu.load %src[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
+    %load = xegpu.load %src[%offset], %mask {layout = #xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>} : memref<?xf32>, vector<32x32x128x128xindex>, vector<32x32x128x128xi1> -> vector<32x32x128x128xf32>
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<32x32x128x128xf32> to vector<32x32xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
@@ -1026,7 +1026,7 @@ gpu.module @test_distribution {
   gpu.func @convert_layout_3D(%arg0: memref<?xf32>) {
     // CHECK-DAG: %[[CST:.*]] = arith.constant dense<0> : vector<1x32x16xindex>
     // CHECK-DAG: %[[CST_0:.*]] = arith.constant dense<true> : vector<1x32x16xi1>
-    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{chunk_size = 1 : i64, layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
+    // CHECK-DAG: %[[LOAD:.*]] = xegpu.load %{{.*}}[%[[CST]]], %[[CST_0]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}> : memref<?xf32>, vector<1x32x16xindex>, vector<1x32x16xi1> -> vector<1x32x16xf32>
     // CHECK-DAG: %[[ALLOCA:.*]] = memref.alloca() : memref<1048576xi8, 3>
     // CHECK-DAG: %[[MDESC:.*]] = xegpu.create_mem_desc %[[ALLOCA]] : memref<1048576xi8, 3> -> !xegpu.mem_desc<8x128x256xf32>
     // CHECK-DAG: %[[SGID:.*]] = gpu.subgroup_id : index
@@ -1055,7 +1055,7 @@ gpu.module @test_distribution {
     // CHECK-DAG: %[[LOAD_SLM:.*]] = xegpu.load_matrix %[[MDESC]][%[[LOAD_OFF_Z]], %[[LOAD_OFF_Y]], %[[LOAD_OFF_X]]] <{layout = #xegpu.layout<inst_data = [1, 16, 16]>}>: !xegpu.mem_desc<8x128x256xf32>, index, index, index -> vector<1x16x32xf32>
     %offset = arith.constant dense<0> : vector<8x128x256xindex>
     %mask = arith.constant dense<true> : vector<8x128x256xi1>
-    %1 = xegpu.load %arg0[%offset], %mask {chunk_size = 1, layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
+    %1 = xegpu.load %arg0[%offset], %mask {layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>} : memref<?xf32>, vector<8x128x256xindex>, vector<8x128x256xi1> -> vector<8x128x256xf32>
     %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<sg_layout = [8, 4, 16], sg_data = [1, 32, 16], inst_data = [1, 16, 16]>,
                                    target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>}> : vector<8x128x256xf32>
     %anchor = xegpu.convert_layout %2
diff --git a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
index f6b0c50da91dd..a94d172b44801 100644
--- a/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
+++ b/mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp
@@ -456,7 +456,7 @@ struct TestXeGPUCoalesceGatherScatter
   }
 
   StringRef getDescription() const final {
-    return "Test driver that turns the contiguity attribute into a lane_data "
+    return "Test driver that turns the chunk_size attribute into a lane_data "
            "layout on gather/scatter ops.";
   }
 
@@ -477,7 +477,7 @@ struct TestXeGPUCoalesceGatherScatter
 
   Option<bool> analyzeOnly{
       *this, "analyze-only",
-      llvm::cl::desc("Only run the analysis (stamp the contiguity attribute); "
+      llvm::cl::desc("Only run the analysis (stamp the chunk_size attribute); "
                      "do not apply."),
       llvm::cl::init(false)};
 
@@ -509,16 +509,16 @@ struct TestXeGPUCoalesceGatherScatter
     return xegpu::LayoutAttr::get(ctx, instData, laneLayout, laneData);
   }
 
-  /// Minimal driver: read the `contiguity` attribute the analysis stamped and
+  /// Minimal driver: read the `chunk_size` attribute the analysis stamped and
   /// turn it into a `lane_data` layout. This is only a stand-in for the real
   /// consumer (layout propagation) so the analysis output can be checked
   /// end-to-end; it handles just the simple power-of-two case.
   template <typename OpTy>
   static void applyContiguity(OpTy op, unsigned maxChunkSize) {
-    std::optional<uint64_t> contiguity = op.getContiguity();
+    std::optional<uint64_t> contiguity = op.getChunkSize();
     if (!contiguity)
       return;
-    op.removeContiguityAttr();
+    op.removeChunkSizeAttr();
 
     auto offsetsTy = dyn_cast<VectorType>(op.getOffsets().getType());
     auto valueTy = op.getValueType();

>From 3049773883c222ed540b6fd7cbb539654d12fecb Mon Sep 17 00:00:00 2001
From: Claude <noreply at anthropic.com>
Date: Mon, 20 Jul 2026 17:37:37 +0000
Subject: [PATCH 2/2] [mlir][XeGPU] Distribute coalesced gather/scatter via
 lane_data

Rework SgToLaneLoadGather / SgToLaneStoreScatter to lower a coalesced
gather/scatter to the chunked memory access purely from the layout's
lane_data, without relying on a chunk_size attribute.

When lane_data[FCD] = D > 1 covers the lane's entire per-lane fragment (one
round: lane_layout[FCD] * D == FCD extent, so the distributed per-lane vector
has exactly D elements), the lane owns a single contiguous run
{base, base+1, ..., base+D-1}. Distribution takes the first offset/mask of the
group (scalar base + scalar mask) and emits a value vector<D>; the chunk size
is implied by the value type, which the XeVM lowering already consumes. The
dropped per-lane offset computations become dead code removed during lowering.

The round-robin case (lane_data[FCD] == 1, multiple strided rounds) is excluded
by the lane_data > 1 guard.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Transforms/XeGPUSgToLaneDistribute.cpp    | 48 +++++++++++++++++++
 .../Dialect/XeGPU/sg-to-lane-distribute.mlir  | 26 ++++++++++
 2 files changed, 74 insertions(+)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 2ac3d1ff2bba5..9a84f172921b5 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -580,6 +580,35 @@ struct SgToLaneLoadGather : public OpConversionPattern<xegpu::LoadGatherOp> {
         castValueTo(rewriter, cast<TypedValue<VectorType>>(distMask), maskTy1D);
 
     Value distSource = adaptor.getSource();
+
+    // Coalesced case: the layout assigns a genuine contiguous chunk on the FCD
+    // via `lane_data[FCD] = D > 1`, and that chunk is the lane's *entire*
+    // per-lane fragment (one round: `lane_layout[FCD] * D == FCD extent`, so
+    // the distributed per-lane vector has exactly D elements). Only then does
+    // the lane own a single run of D contiguous elements
+    // `{base, base+1, ..., base+D-1}`, which the XeVM lowering expects as a
+    // chunked access: a scalar base offset + scalar mask + a `vector<D>` value
+    // (the chunk size is implied by the value type). Take the first offset /
+    // mask of the contiguous group; the remaining offset computations become
+    // dead code and are removed during lowering.
+    //
+    // This must NOT fire for the round-robin case (`lane_data[FCD] = 1`,
+    // multiple rounds, e.g. a reduction source where lane l owns
+    // `{l, l+SG, l+2*SG, ...}`): there the per-lane elements are strided, not
+    // contiguous, so a chunked access would read the wrong elements. That case
+    // has `lane_data[FCD] == 1`, so the guard below excludes it.
+    int64_t laneElems = distResultTy1D.getNumElements();
+    int64_t innerLaneData = 1;
+    if (auto laneDataArr = layout.getEffectiveLaneDataAsInt();
+        !laneDataArr.empty())
+      innerLaneData = laneDataArr.back();
+    if (innerLaneData > 1 && laneElems == innerLaneData) {
+      distOffsets = vector::ExtractOp::create(
+          rewriter, op.getLoc(), distOffsets, ArrayRef<int64_t>{0});
+      distMask = vector::ExtractOp::create(rewriter, op.getLoc(), distMask,
+                                           ArrayRef<int64_t>{0});
+    }
+
     auto newOp = xegpu::LoadGatherOp::create(
         rewriter, op.getLoc(), distResultTy1D, distSource, distOffsets,
         distMask, op.getL1HintAttr(), op.getL2HintAttr(), op.getL3HintAttr(),
@@ -1113,6 +1142,25 @@ struct SgToLaneStoreScatter
         castValueTo(rewriter, cast<TypedValue<VectorType>>(distMask), maskTy1D);
 
     Value distDest = adaptor.getDest();
+
+    // Coalesced case (mirrors SgToLaneLoadGather): a genuine contiguous chunk
+    // on the FCD, `lane_data[FCD] = D > 1` covering the lane's entire per-lane
+    // fragment (one round), stores as a chunked access: scalar base offset +
+    // scalar mask + a `vector<D>` value. Take the first offset / mask of the
+    // contiguous group; the dropped offset computations are DCE'd in lowering.
+    // Excludes the round-robin case (`lane_data[FCD] == 1`).
+    int64_t laneElems = distValueTy1D.getNumElements();
+    int64_t innerLaneData = 1;
+    if (auto laneDataArr = layout.getEffectiveLaneDataAsInt();
+        !laneDataArr.empty())
+      innerLaneData = laneDataArr.back();
+    if (innerLaneData > 1 && laneElems == innerLaneData) {
+      distOffsets = vector::ExtractOp::create(
+          rewriter, op.getLoc(), distOffsets, ArrayRef<int64_t>{0});
+      distMask = vector::ExtractOp::create(rewriter, op.getLoc(), distMask,
+                                           ArrayRef<int64_t>{0});
+    }
+
     xegpu::StoreScatterOp::create(rewriter, op.getLoc(), distValue, distDest,
                                   distOffsets, distMask, op.getL1HintAttr(),
                                   op.getL2HintAttr(), op.getL3HintAttr(),
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index e8a41f6fc99f0..19fc7fc87ff18 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -574,3 +574,29 @@ gpu.module @xevm_module {
     gpu.return
   }
 }
+
+// -----
+// Coalesced gather/scatter: the load/store layout has lane_data[FCD] = 2, i.e.
+// each lane owns 2 *contiguous* elements (one round: lane_layout[FCD] * 2 == 32
+// == FCD extent). Distribution must emit the chunked form the XeVM lowering
+// accepts: a scalar base offset + scalar mask + a value vector<2xf32> (the
+// chunk size is implied by the value type), taking element 0 of the per-lane
+// offsets/mask as the base -- NOT a 2-wide offsets/mask vector. This is driven
+// entirely by lane_data; the dropped offset lanes are DCE'd during lowering.
+gpu.module @xevm_module {
+    // CHECK-LABEL: gpu.func @coalesced_load_store
+    // CHECK: %[[LD:.*]] = xegpu.load %{{.*}}[%[[BASE:.*]]], %{{.*}}  : i64, index, i1 -> vector<2xf32>
+    // CHECK: %[[MUL:.*]] = arith.mulf %[[LD]], %{{.*}} : vector<2xf32>
+    // CHECK: xegpu.store %[[MUL]], %{{.*}}[%[[BASE]]], %{{.*}}  : vector<2xf32>, i64, index, i1
+  gpu.func @coalesced_load_store(%src: i64, %dst: i64) {
+    %step = vector.step : vector<32xindex>
+    %mask = arith.constant dense<true> : vector<32xi1>
+    %v = xegpu.load %src[%step], %mask <{layout = #xegpu.layout<lane_layout = [16], lane_data = [2]>}>
+        : i64, vector<32xindex>, vector<32xi1> -> vector<32xf32>
+    %c = arith.constant dense<2.0> : vector<32xf32>
+    %p = arith.mulf %v, %c {layout_result_0 = #xegpu.layout<lane_layout = [16], lane_data = [2]>} : vector<32xf32>
+    xegpu.store %p, %dst[%step], %mask <{layout = #xegpu.layout<lane_layout = [16], lane_data = [2]>}>
+        : vector<32xf32>, i64, vector<32xindex>, vector<32xi1>
+    gpu.return
+  }
+}


From llvmlistbot at llvm.org  Tue Jul 21 08:05:15 2026
From: llvmlistbot at llvm.org (Md Abdullah Shahneous Bari)
Date: Tue, 21 Jul 2026 08:05:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a5f8aab.5a0412d2.3daf32.6c4c@mx.google.com>


https://github.com/mshahneo ready_for_review https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Tue Jul 21 08:06:52 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Tue, 21 Jul 2026 08:06:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a5f8b0c.160324b0.1f653e.1ef3@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Tue Jul 21 08:48:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:48:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a5f94c8.a1accbb4.1ace1c.213e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Federico Bruzzone (FedericoBruzzone)

<details>
<summary>Changes</summary>

`SparseVectorization` assumes direct loop accesses (`a[lo:hi]`) are contiguous and vectorizes them with `vector.maskedload/maskedstore`. This is false for `sparse_tensor.coordinates` of a level inside a trailing AoS COO region, whose buffer is interleaved with other levels: a silent miscompile.

The true stride is already known from the tensor's encoding, even though the memref type is still dynamic at this point. Use it to fall back to a scalar loop when the stride is provably non-unit.

---
Full diff: https://github.com/llvm/llvm-project/pull/211004.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp (+50) 


``````````diff
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();

``````````

</details>


https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Tue Jul 21 08:50:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:50:32 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211011@github.com>
Message-ID: <6a5f9548.4a342cbc.1aeb63.1bd1@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: Scott Manley (rscottmanley)

<details>
<summary>Changes</summary>

The `acc.firstprivate_save` operation is intended to provide a temporary link from the compute region's private variable and the firstprivate map init op. This facilitates analysis in cases where the compiler may determine that a firstprivate should be converted to a copy and can be safely erased when that analysis is complete.

---
Full diff: https://github.com/llvm/llvm-project/pull/211011.diff


4 Files Affected:

- (modified) flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir (+21-13) 
- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td (+22) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp (+12) 
- (modified) mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir (+12-8) 


``````````diff
diff --git a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
index a2f1dbc6e6565..998740f0e8237 100644
--- a/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
+++ b/flang/test/Transforms/OpenACC/acc-recipe-materialization-firstprivate.fir
@@ -39,18 +39,26 @@ module {
 // and an alloca (as per the recipe) inside the region.
 // Then ensure that all uses are of the private alloca.
 // CHECK-LABEL: func.func @firstpriv
-// CHECK: acc.parallel
-// CHECK: %[[ALLOCA:.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
-// CHECK: %[[FIRSTPRIVLOAD:.*]] = fir.load %{{.*}} : !fir.ref<i32>
-// CHECK: fir.store %[[FIRSTPRIVLOAD]] to %[[ALLOCA]] : !fir.ref<i32>
-// CHECK: %[[ALLOCALOAD:.*]] = fir.load %[[ALLOCA]] : !fir.ref<i32>
-// CHECK: %[[ADDI:.*]] = arith.addi %[[ALLOCALOAD]], %c1{{.*}} : i32
-// CHECK: fir.store %[[ADDI]] to %[[ALLOCA]] : !fir.ref<i32>
+// CHECK:       [[DECL:%.*]] = fir.declare
+// CHECK:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[DECL]] : !fir.ref<i32>) -> !fir.ref<i32> {implicit = true, name = "t"}
+// CHECK:       acc.parallel {
+// CHECK:       [[FIRSTPRIV:%.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = fir.load [[MAP]] : !fir.ref<i32>
+// CHECK:       fir.store [[FIRSTPRIVLOAD]] to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = fir.load [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK:       [[ADDI:%.*]] = arith.addi [[FIRSTPRIVLOAD]], %c1{{.*}} : i32
+// CHECK:       fir.store [[ADDI]] to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK-NEXT:  [[VALUELOAD:%.*]] = fir.load
+// CHECK-NEXT:  acc.firstprivate_save [[VALUELOAD]] to [[MAP]] : i32 -> !fir.ref<i32>
 
+// With -acc-optimize-firstprivate-map, the map is retained because of the
+// firstprivate_save live-out.
 // CHECK-MAP-LABEL: func.func @firstpriv
-// CHECK-MAP: fir.load {{.*}} : !fir.ref<i32>
-// CHECK-MAP: acc.parallel {
-// CHECK-MAP-NOT: acc.firstprivate_map
-// CHECK-MAP: fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
-// CHECK-MAP: fir.store {{.*}} to {{.*}} : !fir.ref<i32>
-// CHECK-MAP: arith.addi {{.*}} %c1
+// CHECK-MAP:       [[DECL:%.*]] = fir.declare
+// CHECK-MAP:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[DECL]] : !fir.ref<i32>) -> !fir.ref<i32> {implicit = true, name = "t"}
+// CHECK-MAP:       acc.parallel {
+// CHECK-MAP:       [[FIRSTPRIV:%.*]] = fir.alloca i32 {{.*}}acc.var_name = #acc.var_name<"t">
+// CHECK-MAP:       fir.load [[MAP]] : !fir.ref<i32>
+// CHECK-MAP:       fir.store {{.*}} to [[FIRSTPRIV]] : !fir.ref<i32>
+// CHECK-MAP:       arith.addi {{.*}} %c1
+// CHECK-MAP:       acc.firstprivate_save {{.*}} to [[MAP]] : i32 -> !fir.ref<i32>
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
index 730ea1f73a3ac..4b1895d4adabc 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCCGOps.td
@@ -302,6 +302,28 @@ def OpenACC_FirstprivateMapInitialOp
   let extraClassDeclaration = extraClassDeclarationBase;
 }
 
+//===----------------------------------------------------------------------===//
+// acc.firstprivate_save
+//===----------------------------------------------------------------------===//
+
+def OpenACC_FirstprivateSaveOp : OpenACC_Op<"firstprivate_save", []> {
+ let summary = "\"Saves\" a value to a firstprivate variable";
+ let description = [{
+    The `acc.firstprivate_save` operation is intended to provide a temporary
+    link from the compute region's private variable and the firstprivate map
+    init op. This facilitates analysis in cases where the compiler may
+    determine that a firstprivate should be converted to a copy and can be
+    safely erased when that analysis is complete.
+  }];
+  let arguments = (ins AnyType:$value,
+                       Arg<OpenACC_PointerLikeType,
+                           "FirstprivateMapInitOp",
+                           [MemWrite]>:$memref);
+  let assemblyFormat = [{
+    $value `to` $memref `:` type($value) `->` type($memref) attr-dict
+  }];
+}
+
 //===----------------------------------------------------------------------===//
 // acc.privatize
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
index 70aacd251c329..712a7b82a02bb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRecipeMaterialization.cpp
@@ -305,6 +305,7 @@ LogicalResult ACCRecipeMaterialization::materialize(
     auto [results, ip] = acc::cloneACCRegionInto(
         &initRegion, block, block->begin(), mapping, {accPtr});
     assert(results.size() == 1 && "expected single result from init region");
+    Value alloca = results[0];
     saveVarName(op.getAccVar(), results[0]);
     resolveVarNamePlaceholders(block, ip, acc::getVariableName(op.getAccVar()));
     // We want the copy to store the origPtr to private
@@ -318,6 +319,17 @@ LogicalResult ACCRecipeMaterialization::materialize(
     Region &copyRegion = recipe.getCopyRegion();
     setLocation(copyRegion, loc);
     acc::cloneACCRegionInto(&copyRegion, block, std::next(ip), mapping, {});
+
+    // Create a temporary acc.firstprivate_save op to keep the value live-out
+    // until after other passes have determined it is truly firstprivate.
+    b.setInsertionPoint(accOp.getBody().getTerminator());
+    auto ptrLikeType = dyn_cast<acc::PointerLikeType>(alloca.getType());
+    assert(ptrLikeType && "alloca must be a pointer-like type");
+    auto xTyped = cast<TypedValue<acc::PointerLikeType>>(alloca);
+    assert(xTyped && "alloca must be a typed value");
+    auto loadOp = ptrLikeType.genLoad(b, op.getLoc(), xTyped, {});
+    acc::FirstprivateSaveOp::create(b, op.getLoc(), loadOp, origPtr);
+
     if (!recipe.getDestroyRegion().empty()) {
       // origPtr was already pushed.
       cloneDestroy(loc, recipe, block, std::prev(block->end()), results);
diff --git a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
index e100ec2df139b..533c7e6ec22dd 100644
--- a/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-recipe-materialization-firstprivate.mlir
@@ -24,14 +24,18 @@ acc.private.recipe @privatization_memref_i32 : memref<i32> init {
 // and an alloca (as per the recipe) inside the region.
 // Then ensure that all uses are of the private alloca.
 // CHECK-LABEL: func.func @firstpriv
-// CHECK: acc.parallel {
-// CHECK: %[[ALLOCA:.*]] = memref.alloca() {acc.var_name = #acc.var_name<"t">} : memref<i32>
-// CHECK: %[[FIRSTPRIVLOAD:.*]] = memref.load %{{.*}}[] : memref<i32>
-// CHECK: memref.store %[[FIRSTPRIVLOAD]], %[[ALLOCA]][] : memref<i32>
-// CHECK: %[[ALLOCALOAD:.*]] = memref.load %[[ALLOCA]][] : memref<i32>
-// CHECK: %[[ADDI:.*]] = arith.addi %[[ALLOCALOAD]], %c1{{.*}} : i32
-// CHECK: memref.store %[[ADDI]], %[[ALLOCA]][] : memref<i32>
-// CHECK: memref.dealloc %[[ALLOCA]] : memref<i32>
+// CHECK:       [[ALLOCA:%.*]] = memref.alloca() : memref<i32>
+// CHECK:       [[MAP:%.*]] = acc.firstprivate_map varPtr([[ALLOCA]] : memref<i32>) -> memref<i32> {implicit = true, name = "t"}
+// CHECK:       acc.parallel {
+// CHECK:       [[FIRSTPRIV:%.*]] = memref.alloca() {acc.var_name = #acc.var_name<"t">} : memref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = memref.load [[MAP]][] : memref<i32>
+// CHECK:       memref.store [[FIRSTPRIVLOAD]], [[FIRSTPRIV]][] : memref<i32>
+// CHECK:       [[FIRSTPRIVLOAD:%.*]] = memref.load [[FIRSTPRIV]][] : memref<i32>
+// CHECK:       [[ADDI:%.*]] = arith.addi [[FIRSTPRIVLOAD]], %c1{{.*}} : i32
+// CHECK:       memref.store [[ADDI]], [[FIRSTPRIV]][] : memref<i32>
+// CHECK-NEXT:  [[VALUELOAD:%.*]] = memref.load
+// CHECK-NEXT:  acc.firstprivate_save [[VALUELOAD]] to [[MAP]] : i32 -> memref<i32>
+// CHECK-NEXT:  memref.dealloc [[FIRSTPRIV]] : memref<i32>
 
 func.func @firstpriv() {
   %c1336 = arith.constant 1336 : i32

``````````

</details>


https://github.com/llvm/llvm-project/pull/211011

From llvmlistbot at llvm.org  Tue Jul 21 08:50:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:50:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a5f9558.c4d3d80a.16dae5.4bef@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Takayuki Todokoro (takatodo)

<details>
<summary>Changes</summary>

Affine dependence analysis can return `DependenceResult::Failure` when it
cannot construct an access relation. Some loop transformations currently
treat this result like `NoDependence`, which can allow an invalid
transformation and cause a miscompilation.

This change handles analysis failures conservatively:

- Reject loop tiling when a dependence check fails.
- Reject validity-checked loop permutation when the complete dependence
  graph cannot be constructed.
- Preserve the original loop nest when sequential-loop sinking encounters
  the same failure.
- Add regression tests using a semi-affine same-memref dependence.

Fixes #<!-- -->210585.

Assisted-by: OpenAI Codex

---
Full diff: https://github.com/llvm/llvm-project/pull/211014.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h (+2-2) 
- (modified) mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp (+4-1) 
- (modified) mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp (+4-1) 
- (modified) mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp (+4-2) 
- (added) mlir/test/Dialect/Affine/loop-transformation-validity.mlir (+25) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Tue Jul 21 08:51:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:51:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a5f9588.d2cb4adc.ccd80.42ce@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Md Abdullah Shahneous Bari (mshahneo)

<details>
<summary>Changes</summary>

The chunk_size attribute on xegpu.load (LoadGatherOp) and xegpu.store (StoreScatterOp) was redundant with the operand/result types: the XeVM lowering never read it, VectorToXeGPU already built these ops with an empty chunk_size, and the op docs noted it could be inferred from the type.

Remove the attribute and infer the chunk size from the value/result and mask types via a computed getChunkSize() op method. The mask carries one element per lane, so the chunk size is the trailing value dimension when the value has more elements than the mask, and 1 otherwise. This keeps the existing op.getChunkSize() call sites working while respecting library layering (no XeGPUUtils dependency from the IR library).

Update the gather/scatter builders, transforms (propagate-layout, sg-to-lane, wg-to-sg, unroll), VectorToXeGPU lowerings, and lit tests. Two invalid.mlir cases that were only invalid because a stated chunk_size contradicted the type are removed; two others are reclassified to the type/shape diagnostic.

---

Patch is 95.88 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/205122.diff


23 Files Affected:

- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td (+21-16) 
- (modified) mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h (+2-2) 
- (modified) mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h (+12) 
- (modified) mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp (-4) 
- (modified) mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp (+55-31) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp (+4-4) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp (+4-2) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp (+9-8) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp (+5-15) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp (+5-11) 
- (modified) mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp (+18-2) 
- (modified) mlir/test/Dialect/XeGPU/contiguity-analysis.mlir (+19-19) 
- (modified) mlir/test/Dialect/XeGPU/invalid.mlir (+12-41) 
- (modified) mlir/test/Dialect/XeGPU/ops.mlir (+17-17) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir (+4-4) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout.mlir (+2-2) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir (+6-6) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir (+8-8) 
- (modified) mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir (+1-1) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-blocking.mlir (+20-20) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir (+8-8) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir (+16-16) 
- (modified) mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..ee551e1065109 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -731,13 +731,17 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to load from per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the result
       of load. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`: the innermost `offsets` dimension is contiguous in runs of
+      `chunk_size` elements (so `chunk_size` must be >= 2 and must divide that
+      dimension). It is a hint used by layout propagation / coalescing; it does
+      not change the loaded data. Only valid on the vector-offsets form.
+
     Results:
     - `res`: represents loaded data
 
@@ -756,8 +760,8 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   ```
 
   Example 2 (lane level):
-  lane level only accepts the offsets variant. chunk_size can be inferred from result
-  type. In this example, chunk_size is 8.
+  lane level only accepts the offsets variant. The number of contiguous elements
+  loaded per work item is inferred from the result type; in this example it is 8.
   ```mlir
     %2 = xegpu.load %1[%2], %0 <{l1_hint = #xegpu.cache_hint<cached>,
                              l2_hint = #xegpu.cache_hint<uncached>,
@@ -769,12 +773,12 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
 
   let arguments = (ins XeGPU_GatherScatterBaseAddrType:$source,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
   let results = (outs XeGPU_ValueOrScalarType:$value);
 
   let extraClassDeclaration = extraBaseClassDeclaration # [{
@@ -815,13 +819,11 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   let builders = [
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
@@ -862,13 +864,18 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to store to per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the value
       to be stored. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`, analogously to `load`: the innermost `offsets` dimension is
+      contiguous in runs of `chunk_size` elements (so `chunk_size` must be >= 2
+      and must divide that dimension). It is a hint used by layout propagation /
+      coalescing; it does not change the stored data. Only valid on the
+      vector-offsets form.
+
 
   Example 1 (Subgroup level):
   A variant accepts memref as base pointer and an offset.
@@ -886,8 +893,8 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   ```
 
   Example 2 (Lane level):
-  Lane level IR only accepts the offsets variant. chunk_size can be inferred from value
-  type. In this example, chunk_size is 8.
+  Lane level IR only accepts the offsets variant. The number of contiguous elements
+  stored per work item is inferred from the value type; in this example it is 8.
   ```mlir
     xegpu.store %0, %1[%2], %3 <{l1_hint = #xegpu.cache_hint<uncached>,
                              l2_hint = #xegpu.cache_hint<write_back>,
@@ -900,12 +907,12 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let arguments = (ins XeGPU_ValueOrScalarType:$value,
       XeGPU_GatherScatterBaseAddrType:$dest,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
 
   let extraClassDeclaration = extraBaseClassDeclaration#[{
     Type getDestType() {
@@ -945,13 +952,11 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let builders = [
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
index 388bd6145df21..7f707d67ebd2a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
@@ -93,11 +93,11 @@ void populateXeGPUSgToLaneDistributeTypeConversionAndLegality(
 //===----------------------------------------------------------------------===//
 
 /// Run the AxisInfo-based contiguity analysis over `root` and stamp a
-/// `contiguity` attribute on every `xegpu.load` / `xegpu.store` whose
+/// `chunk_size` attribute on every `xegpu.load` / `xegpu.store` whose
 /// offsets are contiguous (in runs of >= 2) along the innermost dimension.
 /// The stamped value is the inner-dim contiguity; it is a target-independent
 /// property consumed downstream (e.g. to derive a `lane_data` split). Ops that
-/// already carry a `contiguity` attribute are left untouched.
+/// already carry a `chunk_size` attribute are left untouched.
 void runContiguityAnalysis(Operation *root);
 
 /// Collect a set of patterns to unroll xegpu operations to a smaller shapes.
diff --git a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
index 0125dfc44196b..33dfc1cea4ea0 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
@@ -41,6 +41,18 @@ namespace xegpu {
 /// Flatten a set of ValueRange into a single SmallVector<Value>
 SmallVector<Value> flattenValues(ArrayRef<ValueRange> values);
 
+/// Infer the payload chunk size (number of contiguous elements per work item)
+/// of a gather/scatter op (`xegpu.load` / `xegpu.store`) from its value/result
+/// and mask types. The mask carries one element per lane, so the payload is the
+/// value with the chunk dimension removed: the chunk size is the trailing value
+/// dimension whenever the value has more elements than the mask, and 1
+/// otherwise.
+///   - scalar value                               -> 1
+///   - value vector<LxC...>, mask vector<L...>    -> C (trailing dim)
+///   - 1D value vector<N...>, scalar/size-1 mask  -> N (lane-level chunk)
+///   - value and mask with matching element counts -> 1 (one elem per lane)
+int64_t getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy);
+
 /// If tensor descriptor has a layout attribute it is used in SIMT mode.
 /// In this mode, the distributed vector shape is determined as follows:
 /// Definitions:
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..feab0eb51ec27 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -502,7 +502,6 @@ static LogicalResult lowerToScatteredLoadOp(vector::TransferReadOp readOp,
       vectorShape);
   auto gatherOp = xegpu::LoadGatherOp::create(
       rewriter, loc, vectorType, flatMemref, localOffsets, mask,
-      /*chunk_size=*/IntegerAttr{},
       /*l1_hint=*/xegpu::CachePolicyAttr{},
       /*l2_hint=*/xegpu::CachePolicyAttr{},
       /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -537,7 +536,6 @@ static LogicalResult lowerToScatteredStoreOp(vector::TransferWriteOp writeOp,
       vectorShape);
   xegpu::StoreScatterOp::create(rewriter, loc, writeOp.getVector(), flatMemref,
                                 localOffsets, mask,
-                                /*chunk_size=*/IntegerAttr{},
                                 /*l1_hint=*/xegpu::CachePolicyAttr{},
                                 /*l2_hint=*/xegpu::CachePolicyAttr{},
                                 /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -793,7 +791,6 @@ struct GatherLowering : public OpRewritePattern<vector::GatherOp> {
 
     auto xeGatherOp = xegpu::LoadGatherOp::create(
         rewriter, loc, vectorType, flatMemref, localOffsets, gatherOp.getMask(),
-        /*chunk_size=*/IntegerAttr{},
         /*l1_hint=*/xegpu::CachePolicyAttr{},
         /*l2_hint=*/xegpu::CachePolicyAttr{},
         /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -828,7 +825,6 @@ struct ScatterLowering : public OpRewritePattern<vector::ScatterOp> {
 
     xegpu::StoreScatterOp::create(rewriter, loc, scatterOp.getValueToStore(),
                                   flatMemref, localOffsets, scatterOp.getMask(),
-                                  /*chunk_size=*/IntegerAttr{},
                                   /*l1_hint=*/xegpu::CachePolicyAttr{},
                                   /*l2_hint=*/xegpu::CachePolicyAttr{},
                                   /*l3_hint=*/xegpu::CachePolicyAttr{},
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..c0178035b59d5 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -61,6 +61,30 @@ static bool isWriteHintOrNone(const CachePolicyAttr &attr) {
          kind == CachePolicy::WRITE_BACK || kind == CachePolicy::WRITE_THROUGH;
 }
 
+// Infer the chunk size (number of contiguous elements per work item) of a
+// gather/scatter op from its value/result and mask types. The chunk size is no
+// longer carried as an attribute; it is fully determined by the types. The
+// mask carries one element per lane, so it is the value with the chunk
+// dimension removed. The chunk size is therefore the trailing value dimension
+// whenever the value has more elements than the mask, and 1 otherwise:
+//   - scalar value                                   -> 1
+//   - value vector<LxC...>, mask vector<L...>        -> C (trailing dim)
+//   - 1D value vector<N...>, scalar/size-1 mask      -> N (lane-level chunk)
+//   - value and mask with matching element counts    -> 1 (one elem per lane)
+static int64_t inferGatherScatterChunkSize(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 static LogicalResult
 isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
                                  VectorType valueTy, int64_t chunkSize,
@@ -113,23 +137,23 @@ isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
   return success();
 }
 
-// Validates the `contiguity` attribute against the op's offsets type: the
+// Validates the `chunk_size` attribute against the op's offsets type: the
 // innermost offsets dimension is contiguous in runs of `size`, so `size` must
 // be >= 2 and must divide that dimension.
 static LogicalResult
-isValidContiguity(std::optional<uint64_t> contiguity, Type offsetsTy,
-                  function_ref<InFlightDiagnostic()> emitError) {
-  if (!contiguity)
+isValidChunkSize(std::optional<uint64_t> chunkSize, Type offsetsTy,
+                 function_ref<InFlightDiagnostic()> emitError) {
+  if (!chunkSize)
     return success();
   auto offsetsVecTy = dyn_cast<VectorType>(offsetsTy);
   if (!offsetsVecTy)
-    return emitError() << "contiguity requires vector offsets (one per lane).";
-  int64_t size = static_cast<int64_t>(*contiguity);
+    return emitError() << "chunk_size requires vector offsets (one per lane).";
+  int64_t size = static_cast<int64_t>(*chunkSize);
   int64_t inner = offsetsVecTy.getShape().back();
   if (size < 2)
-    return emitError() << "contiguity = " << size << " (must be >= 2)";
+    return emitError() << "chunk_size = " << size << " (must be >= 2)";
   if (inner % size != 0)
-    return emitError() << "contiguity = " << size
+    return emitError() << "chunk_size = " << size
                        << " (must divide the innermost offsets dim " << inner
                        << ")";
   return success();
@@ -592,7 +616,7 @@ LogicalResult LoadGatherOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto srcTy = getSourceType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(srcTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -604,8 +628,8 @@ LogicalResult LoadGatherOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -614,7 +638,7 @@ LogicalResult LoadGatherOp::verify() {
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint) {
   auto loc = source.getLoc();
@@ -623,15 +647,14 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, /*anchor_layout=*/nullptr,
-        /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint,
                          DistributeLayoutAttr layout) {
@@ -641,8 +664,8 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
@@ -662,7 +685,7 @@ LogicalResult StoreScatterOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto destTy = getDestType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(destTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -674,8 +697,8 @@ LogicalResult StoreScatterOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -684,7 +707,6 @@ LogicalResult StoreScatterOp::verify() {
 void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
                            Value value, Value dest,
                            ArrayRef<OpFoldResult> offsets, Value mask,
-                           IntegerAttr chunk_size,
                            xegpu::CachePolicyAttr l1_hint,
                            xegpu::CachePolicyAttr l2_hint,
                            xegpu::CachePolicyAttr l3_hint) {
@@ -695,15 +717,17 @@ void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, /*anchor_layout=*/nullptr, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
-void StoreScatterOp::build(
-    OpBuilder &builder, OperationState &state, Value value, Value dest,
-    ArrayRef<OpFoldResult> offsets, Value mask, IntegerAttr chunk_size,
-    xegpu::CachePolicyAttr l1_hint, xegpu::CachePolicyAttr l2_hint,
-    xegpu::CachePolicyAttr l3_hint, DistributeLayoutAttr layout) {
+void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
+       ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Tue Jul 21 08:51:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:51:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a5f9589.8e75e0cb.19a983.4b3a@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Md Abdullah Shahneous Bari (mshahneo)

<details>
<summary>Changes</summary>

The chunk_size attribute on xegpu.load (LoadGatherOp) and xegpu.store (StoreScatterOp) was redundant with the operand/result types: the XeVM lowering never read it, VectorToXeGPU already built these ops with an empty chunk_size, and the op docs noted it could be inferred from the type.

Remove the attribute and infer the chunk size from the value/result and mask types via a computed getChunkSize() op method. The mask carries one element per lane, so the chunk size is the trailing value dimension when the value has more elements than the mask, and 1 otherwise. This keeps the existing op.getChunkSize() call sites working while respecting library layering (no XeGPUUtils dependency from the IR library).

Update the gather/scatter builders, transforms (propagate-layout, sg-to-lane, wg-to-sg, unroll), VectorToXeGPU lowerings, and lit tests. Two invalid.mlir cases that were only invalid because a stated chunk_size contradicted the type are removed; two others are reclassified to the type/shape diagnostic.

---

Patch is 95.88 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/205122.diff


23 Files Affected:

- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td (+21-16) 
- (modified) mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h (+2-2) 
- (modified) mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h (+12) 
- (modified) mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp (-4) 
- (modified) mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp (+55-31) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUContiguityAnalysis.cpp (+4-4) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp (+4-2) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp (+9-8) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp (+5-15) 
- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp (+5-11) 
- (modified) mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp (+18-2) 
- (modified) mlir/test/Dialect/XeGPU/contiguity-analysis.mlir (+19-19) 
- (modified) mlir/test/Dialect/XeGPU/invalid.mlir (+12-41) 
- (modified) mlir/test/Dialect/XeGPU/ops.mlir (+17-17) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir (+4-4) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout.mlir (+2-2) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir (+6-6) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir (+8-8) 
- (modified) mlir/test/Dialect/XeGPU/test-xegpu-coalesce-gather-scatter.mlir (+1-1) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-blocking.mlir (+20-20) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-unroll-patterns.mlir (+8-8) 
- (modified) mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir (+16-16) 
- (modified) mlir/test/lib/Dialect/XeGPU/TestXeGPUTransforms.cpp (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..ee551e1065109 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -731,13 +731,17 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to load from per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the result
       of load. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`: the innermost `offsets` dimension is contiguous in runs of
+      `chunk_size` elements (so `chunk_size` must be >= 2 and must divide that
+      dimension). It is a hint used by layout propagation / coalescing; it does
+      not change the loaded data. Only valid on the vector-offsets form.
+
     Results:
     - `res`: represents loaded data
 
@@ -756,8 +760,8 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   ```
 
   Example 2 (lane level):
-  lane level only accepts the offsets variant. chunk_size can be inferred from result
-  type. In this example, chunk_size is 8.
+  lane level only accepts the offsets variant. The number of contiguous elements
+  loaded per work item is inferred from the result type; in this example it is 8.
   ```mlir
     %2 = xegpu.load %1[%2], %0 <{l1_hint = #xegpu.cache_hint<cached>,
                              l2_hint = #xegpu.cache_hint<uncached>,
@@ -769,12 +773,12 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
 
   let arguments = (ins XeGPU_GatherScatterBaseAddrType:$source,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
   let results = (outs XeGPU_ValueOrScalarType:$value);
 
   let extraClassDeclaration = extraBaseClassDeclaration # [{
@@ -815,13 +819,11 @@ def XeGPU_LoadGatherOp : XeGPU_Op<"load", [MemoryEffects<[MemRead]>, AnchorLayou
   let builders = [
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Type": $value, "Value": $source,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
@@ -862,13 +864,18 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
         mask is a vector of size equal to the subgroup size, or 1 at lane level.
         scalar mask is also valid for lane level.
 
-    - `chunk_size`: [optional] represents contiguous number of elements to store to per work item.
-
     - `l1_hint`, `l2_hint`, `l3_hint`: [optional] cache hints for each level of cache.
 
     - `layout`: [optional] Describes the expected layout of the `tensor_desc` operand or the value
       to be stored. Only valid at workgroup and subgroup levels.
 
+    - `chunk_size`: [optional] An I64 attribute describing the contiguity of the
+      `offsets`, analogously to `load`: the innermost `offsets` dimension is
+      contiguous in runs of `chunk_size` elements (so `chunk_size` must be >= 2
+      and must divide that dimension). It is a hint used by layout propagation /
+      coalescing; it does not change the stored data. Only valid on the
+      vector-offsets form.
+
 
   Example 1 (Subgroup level):
   A variant accepts memref as base pointer and an offset.
@@ -886,8 +893,8 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   ```
 
   Example 2 (Lane level):
-  Lane level IR only accepts the offsets variant. chunk_size can be inferred from value
-  type. In this example, chunk_size is 8.
+  Lane level IR only accepts the offsets variant. The number of contiguous elements
+  stored per work item is inferred from the value type; in this example it is 8.
   ```mlir
     xegpu.store %0, %1[%2], %3 <{l1_hint = #xegpu.cache_hint<uncached>,
                              l2_hint = #xegpu.cache_hint<write_back>,
@@ -900,12 +907,12 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let arguments = (ins XeGPU_ValueOrScalarType:$value,
       XeGPU_GatherScatterBaseAddrType:$dest,
       AnyTypeOf<[XeGPU_OffsetType, Index]>:$offsets,
-      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask, OptionalAttr<I64Attr>:$chunk_size,
+      AnyTypeOf<[XeGPU_MaskType, I1]>:$mask,
       OptionalAttr<XeGPU_CacheHintAttr>:$l1_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l2_hint,
       OptionalAttr<XeGPU_CacheHintAttr>:$l3_hint,
       OptionalAttr<DistributeLayoutAttr>:$layout,
-      OptionalAttr<I64Attr>:$contiguity);
+      OptionalAttr<I64Attr>:$chunk_size);
 
   let extraClassDeclaration = extraBaseClassDeclaration#[{
     Type getDestType() {
@@ -945,13 +952,11 @@ def XeGPU_StoreScatterOp : XeGPU_Op<"store", [MemoryEffects<[MemWrite]>, AnchorL
   let builders = [
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint)>,
     OpBuilder<(ins "Value": $value, "Value": $dest,
                     "ArrayRef<OpFoldResult>": $offsets, "Value": $mask,
-                    "IntegerAttr": $chunk_size,
                     "xegpu::CachePolicyAttr": $l1_hint,
                     "xegpu::CachePolicyAttr": $l2_hint,
                     "xegpu::CachePolicyAttr": $l3_hint,
diff --git a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
index 388bd6145df21..7f707d67ebd2a 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Transforms/Transforms.h
@@ -93,11 +93,11 @@ void populateXeGPUSgToLaneDistributeTypeConversionAndLegality(
 //===----------------------------------------------------------------------===//
 
 /// Run the AxisInfo-based contiguity analysis over `root` and stamp a
-/// `contiguity` attribute on every `xegpu.load` / `xegpu.store` whose
+/// `chunk_size` attribute on every `xegpu.load` / `xegpu.store` whose
 /// offsets are contiguous (in runs of >= 2) along the innermost dimension.
 /// The stamped value is the inner-dim contiguity; it is a target-independent
 /// property consumed downstream (e.g. to derive a `lane_data` split). Ops that
-/// already carry a `contiguity` attribute are left untouched.
+/// already carry a `chunk_size` attribute are left untouched.
 void runContiguityAnalysis(Operation *root);
 
 /// Collect a set of patterns to unroll xegpu operations to a smaller shapes.
diff --git a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
index 0125dfc44196b..33dfc1cea4ea0 100644
--- a/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
+++ b/mlir/include/mlir/Dialect/XeGPU/Utils/XeGPUUtils.h
@@ -41,6 +41,18 @@ namespace xegpu {
 /// Flatten a set of ValueRange into a single SmallVector<Value>
 SmallVector<Value> flattenValues(ArrayRef<ValueRange> values);
 
+/// Infer the payload chunk size (number of contiguous elements per work item)
+/// of a gather/scatter op (`xegpu.load` / `xegpu.store`) from its value/result
+/// and mask types. The mask carries one element per lane, so the payload is the
+/// value with the chunk dimension removed: the chunk size is the trailing value
+/// dimension whenever the value has more elements than the mask, and 1
+/// otherwise.
+///   - scalar value                               -> 1
+///   - value vector<LxC...>, mask vector<L...>    -> C (trailing dim)
+///   - 1D value vector<N...>, scalar/size-1 mask  -> N (lane-level chunk)
+///   - value and mask with matching element counts -> 1 (one elem per lane)
+int64_t getGatherScatterPayloadChunk(VectorType valueTy, Type maskTy);
+
 /// If tensor descriptor has a layout attribute it is used in SIMT mode.
 /// In this mode, the distributed vector shape is determined as follows:
 /// Definitions:
diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 9a994e87697f6..feab0eb51ec27 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -502,7 +502,6 @@ static LogicalResult lowerToScatteredLoadOp(vector::TransferReadOp readOp,
       vectorShape);
   auto gatherOp = xegpu::LoadGatherOp::create(
       rewriter, loc, vectorType, flatMemref, localOffsets, mask,
-      /*chunk_size=*/IntegerAttr{},
       /*l1_hint=*/xegpu::CachePolicyAttr{},
       /*l2_hint=*/xegpu::CachePolicyAttr{},
       /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -537,7 +536,6 @@ static LogicalResult lowerToScatteredStoreOp(vector::TransferWriteOp writeOp,
       vectorShape);
   xegpu::StoreScatterOp::create(rewriter, loc, writeOp.getVector(), flatMemref,
                                 localOffsets, mask,
-                                /*chunk_size=*/IntegerAttr{},
                                 /*l1_hint=*/xegpu::CachePolicyAttr{},
                                 /*l2_hint=*/xegpu::CachePolicyAttr{},
                                 /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -793,7 +791,6 @@ struct GatherLowering : public OpRewritePattern<vector::GatherOp> {
 
     auto xeGatherOp = xegpu::LoadGatherOp::create(
         rewriter, loc, vectorType, flatMemref, localOffsets, gatherOp.getMask(),
-        /*chunk_size=*/IntegerAttr{},
         /*l1_hint=*/xegpu::CachePolicyAttr{},
         /*l2_hint=*/xegpu::CachePolicyAttr{},
         /*l3_hint=*/xegpu::CachePolicyAttr{},
@@ -828,7 +825,6 @@ struct ScatterLowering : public OpRewritePattern<vector::ScatterOp> {
 
     xegpu::StoreScatterOp::create(rewriter, loc, scatterOp.getValueToStore(),
                                   flatMemref, localOffsets, scatterOp.getMask(),
-                                  /*chunk_size=*/IntegerAttr{},
                                   /*l1_hint=*/xegpu::CachePolicyAttr{},
                                   /*l2_hint=*/xegpu::CachePolicyAttr{},
                                   /*l3_hint=*/xegpu::CachePolicyAttr{},
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..c0178035b59d5 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -61,6 +61,30 @@ static bool isWriteHintOrNone(const CachePolicyAttr &attr) {
          kind == CachePolicy::WRITE_BACK || kind == CachePolicy::WRITE_THROUGH;
 }
 
+// Infer the chunk size (number of contiguous elements per work item) of a
+// gather/scatter op from its value/result and mask types. The chunk size is no
+// longer carried as an attribute; it is fully determined by the types. The
+// mask carries one element per lane, so it is the value with the chunk
+// dimension removed. The chunk size is therefore the trailing value dimension
+// whenever the value has more elements than the mask, and 1 otherwise:
+//   - scalar value                                   -> 1
+//   - value vector<LxC...>, mask vector<L...>        -> C (trailing dim)
+//   - 1D value vector<N...>, scalar/size-1 mask      -> N (lane-level chunk)
+//   - value and mask with matching element counts    -> 1 (one elem per lane)
+static int64_t inferGatherScatterChunkSize(VectorType valueTy, Type maskTy) {
+  if (!valueTy)
+    return 1;
+  auto maskVecTy = dyn_cast<VectorType>(maskTy);
+  int64_t maskSize = maskVecTy ? maskVecTy.getNumElements() : 1;
+  int64_t valueSize = valueTy.getNumElements();
+  if (valueTy.getRank() >= 2)
+    return maskSize == valueSize ? 1 : valueTy.getShape().back();
+  // 1D value: a size-1 (or scalar) mask denotes a single work item performing a
+  // chunked load/store, so the whole vector is the chunk; a wider mask denotes
+  // one element per lane (chunk size 1).
+  return maskSize == 1 ? valueSize : 1;
+}
+
 static LogicalResult
 isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
                                  VectorType valueTy, int64_t chunkSize,
@@ -113,23 +137,23 @@ isValidGatherScatterBufferParams(Type offsetsTy, Type maskTy,
   return success();
 }
 
-// Validates the `contiguity` attribute against the op's offsets type: the
+// Validates the `chunk_size` attribute against the op's offsets type: the
 // innermost offsets dimension is contiguous in runs of `size`, so `size` must
 // be >= 2 and must divide that dimension.
 static LogicalResult
-isValidContiguity(std::optional<uint64_t> contiguity, Type offsetsTy,
-                  function_ref<InFlightDiagnostic()> emitError) {
-  if (!contiguity)
+isValidChunkSize(std::optional<uint64_t> chunkSize, Type offsetsTy,
+                 function_ref<InFlightDiagnostic()> emitError) {
+  if (!chunkSize)
     return success();
   auto offsetsVecTy = dyn_cast<VectorType>(offsetsTy);
   if (!offsetsVecTy)
-    return emitError() << "contiguity requires vector offsets (one per lane).";
-  int64_t size = static_cast<int64_t>(*contiguity);
+    return emitError() << "chunk_size requires vector offsets (one per lane).";
+  int64_t size = static_cast<int64_t>(*chunkSize);
   int64_t inner = offsetsVecTy.getShape().back();
   if (size < 2)
-    return emitError() << "contiguity = " << size << " (must be >= 2)";
+    return emitError() << "chunk_size = " << size << " (must be >= 2)";
   if (inner % size != 0)
-    return emitError() << "contiguity = " << size
+    return emitError() << "chunk_size = " << size
                        << " (must divide the innermost offsets dim " << inner
                        << ")";
   return success();
@@ -592,7 +616,7 @@ LogicalResult LoadGatherOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto srcTy = getSourceType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(srcTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -604,8 +628,8 @@ LogicalResult LoadGatherOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -614,7 +638,7 @@ LogicalResult LoadGatherOp::verify() {
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint) {
   auto loc = source.getLoc();
@@ -623,15 +647,14 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, /*anchor_layout=*/nullptr,
-        /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
 void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
                          Type valueType, Value source,
                          ArrayRef<OpFoldResult> offsets, Value mask,
-                         IntegerAttr chunk_size, xegpu::CachePolicyAttr l1_hint,
+                         xegpu::CachePolicyAttr l1_hint,
                          xegpu::CachePolicyAttr l2_hint,
                          xegpu::CachePolicyAttr l3_hint,
                          DistributeLayoutAttr layout) {
@@ -641,8 +664,8 @@ void LoadGatherOp::build(OpBuilder &builder, OperationState &state,
   auto values = getValueOrCreateConstantIndexOp(builder, loc, offsets);
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
-  build(builder, state, valueType, source, offset, mask, chunk_size, l1_hint,
-        l2_hint, l3_hint, layout, /*contiguity=*/nullptr);
+  build(builder, state, valueType, source, offset, mask, l1_hint, l2_hint,
+        l3_hint, layout, /*chunk_size=*/nullptr);
 }
 
 //===----------------------------------------------------------------------===//
@@ -662,7 +685,7 @@ LogicalResult StoreScatterOp::verify() {
     return emitOpError("invalid l3_hint: ") << getL3HintAttr();
 
   auto destTy = getDestType();
-  uint64_t chunkSize = static_cast<int64_t>(getChunkSize().value_or(1));
+  int64_t chunkSize = inferGatherScatterChunkSize(valueTy, maskTy);
   auto memTy = dyn_cast<MemRefType>(destTy);
 
   if (memTy && (getElementType() != memTy.getElementType()))
@@ -674,8 +697,8 @@ LogicalResult StoreScatterOp::verify() {
   }
 
   auto offsetsTy = getOffsets().getType();
-  if (failed(isValidContiguity(getContiguity(), offsetsTy,
-                               [&]() { return emitOpError(); })))
+  if (failed(isValidChunkSize(getChunkSize(), offsetsTy,
+                              [&]() { return emitOpError(); })))
     return failure();
   return isValidGatherScatterBufferParams(offsetsTy, maskTy, valueTy, chunkSize,
                                           [&]() { return emitOpError(); });
@@ -684,7 +707,6 @@ LogicalResult StoreScatterOp::verify() {
 void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
                            Value value, Value dest,
                            ArrayRef<OpFoldResult> offsets, Value mask,
-                           IntegerAttr chunk_size,
                            xegpu::CachePolicyAttr l1_hint,
                            xegpu::CachePolicyAttr l2_hint,
                            xegpu::CachePolicyAttr l3_hint) {
@@ -695,15 +717,17 @@ void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
   auto offset = vector::FromElementsOp::create(builder, loc, type, values);
 
   // Call the correct builder overload that does not expect result types.
-  build(builder, state, value, dest, offset, mask, chunk_size, l1_hint, l2_hint,
-        l3_hint, /*anchor_layout=*/nullptr, /*contiguity=*/nullptr);
+  build(builder, state, value, dest, offset, mask, l1_hint, l2_hint, l3_hint,
+        /*anchor_layout=*/nullptr, /*chunk_size=*/nullptr);
 }
 
-void StoreScatterOp::build(
-    OpBuilder &builder, OperationState &state, Value value, Value dest,
-    ArrayRef<OpFoldResult> offsets, Value mask, IntegerAttr chunk_size,
-    xegpu::CachePolicyAttr l1_hint, xegpu::CachePolicyAttr l2_hint,
-    xegpu::CachePolicyAttr l3_hint, DistributeLayoutAttr layout) {
+void StoreScatterOp::build(OpBuilder &builder, OperationState &state,
+       ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Tue Jul 21 08:52:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:52:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Analysis][Presburger] Remove redundant
 local vars after composition (PR #211023)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211023@github.com>
Message-ID: <6a5f95d1.6c5720b3.1f1229.4b20@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: asraa

<details>
<summary>Changes</summary>

Call `removeRedundantLocalVars()` at the end of `IntegerRelation::compose` to simplify existential constraints generated when projecting out intermediate dimensions.

Removing redundant local vars also happens in `IntegerRelation::mergeAndCompose` and `FlatAffineRelation::compose`.

When composing A-&gt;B with B-&gt;C, intermediate vars B are converted to local existential variables. When performing multiple compositions, this causes bloat of existential variables that impact later performance, bounds checks, and emptiness checks.


---
Full diff: https://github.com/llvm/llvm-project/pull/211023.diff


2 Files Affected:

- (modified) mlir/lib/Analysis/Presburger/IntegerRelation.cpp (+3) 
- (modified) mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp (+19) 


``````````diff
diff --git a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
index 93a725571078e..719aa5962c8fa 100644
--- a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
+++ b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
@@ -2573,6 +2573,9 @@ void IntegerRelation::compose(const IntegerRelation &rel) {
 
   // Project out B in R1.
   convertVarKind(VarKind::Range, 0, numBVars, VarKind::Local);
+
+  // Eliminate local variables  
+  removeRedundantLocalVars();
 }
 
 void IntegerRelation::applyDomain(const IntegerRelation &rel) {
diff --git a/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp b/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
index 90753d502d12c..c3da1c1556d26 100644
--- a/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
+++ b/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
@@ -119,6 +119,25 @@ TEST(IntegerRelationTest, applyDomainAndRange) {
   }
 }
 
+TEST(IntegerRelationTest, composeRemoveRedundantLocalVars) {
+  // map x to  y = x + 10.
+  IntegerRelation map1 =
+      parseRelationFromSet("(x, y) : (y - x - 10 == 0)", /*numDomain=*/1);
+
+  // map y to z = y + 20.
+  IntegerRelation map2 =
+      parseRelationFromSet("(y, z) : (z - y - 20 == 0)", /*numDomain=*/1);
+
+  // composing projects out y (converting it to a local variable)
+  map1.compose(map2);
+
+  // y is fully determined by x so should be eliminated after composition
+  EXPECT_EQ(map1.getNumLocalVars(), 0u);
+  IntegerRelation expectedMap =
+      parseRelationFromSet("(x, z) : (z - x - 30 == 0)", /*numDomain=*/1);
+  EXPECT_TRUE(map1.isEqual(expectedMap));
+}
+
 TEST(IntegerRelationTest, symbolicLexmin) {
   SymbolicLexOpt lexmin =
       parseRelationFromSet("(a, x)[b] : (x - a >= 0, x - b >= 0)", 1)

``````````

</details>


https://github.com/llvm/llvm-project/pull/211023

From llvmlistbot at llvm.org  Tue Jul 21 08:53:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:53:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Analysis][Presburger] Remove redundant
 local vars after composition (PR #211023)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211023@github.com>
Message-ID: <6a5f95e1.d681cab9.33af1e.3e55@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Analysis/Presburger/IntegerRelation.cpp mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
index 719aa5962..94a15dc3f 100644
--- a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
+++ b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
@@ -2574,7 +2574,7 @@ void IntegerRelation::compose(const IntegerRelation &rel) {
   // Project out B in R1.
   convertVarKind(VarKind::Range, 0, numBVars, VarKind::Local);
 
-  // Eliminate local variables  
+  // Eliminate local variables
   removeRedundantLocalVars();
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/211023

From llvmlistbot at llvm.org  Tue Jul 21 08:53:45 2026
From: llvmlistbot at llvm.org (Shay Kleiman)
Date: Tue, 21 Jul 2026 08:53:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][scf] Expose kPeeledLoopLabel as a
 public marker attribute (PR #210922)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210922@github.com>
Message-ID: <6a5f9609.481afadf.1fc9f0.17d1@mx.google.com>


https://github.com/shay-kl updated https://github.com/llvm/llvm-project/pull/210922

>From 5c7a50eb59808b89c343e38fbb1f4ba98027dc05 Mon Sep 17 00:00:00 2001
From: Shay Kleiman <shay.kleiman at mobileye.com>
Date: Tue, 21 Jul 2026 12:15:15 +0300
Subject: [PATCH 1/2] [mlir][scf] Expose kPeeledLoopLabel as a public marker
 attribute

Change-Id: I08fa8bcab6cf76bc36894fb79aa32635c1aebe15
---
 mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h  | 3 +++
 mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp | 9 ++++-----
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 00e8572307151..3b96bd77f6928 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -106,6 +106,9 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
 LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
                                         scf::ForOp &partialIteration);
 
+/// Marker attribute name for loops that have already been peeled.
+static constexpr StringLiteral kPeeledLoopLabel = "__peeled_loop__";
+
 /// Tile a parallel loop of the form
 ///   scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
 ///                                             step (%arg4, %arg5)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index a39e5520a144b..a8609b2c9eafb 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -256,7 +256,6 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
   return success();
 }
 
-static constexpr char kPeeledLoopLabel[] = "__peeled_loop__";
 static constexpr char kPartialIterationLabel[] = "__partial_iteration__";
 
 namespace {
@@ -272,7 +271,7 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
                                          "unsigned loops are not supported");
 
     // Do not peel already peeled loops.
-    if (forOp->hasAttr(kPeeledLoopLabel))
+    if (forOp->hasAttr(scf::kPeeledLoopLabel))
       return failure();
 
     scf::ForOp partialIteration;
@@ -300,11 +299,11 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
 
     // Apply label, so that the same loop is not rewritten a second time.
     rewriter.modifyOpInPlace(partialIteration, [&]() {
-      partialIteration->setAttr(kPeeledLoopLabel, rewriter.getUnitAttr());
+      partialIteration->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
       partialIteration->setAttr(kPartialIterationLabel, rewriter.getUnitAttr());
     });
     rewriter.modifyOpInPlace(forOp, [&]() {
-      forOp->setAttr(kPeeledLoopLabel, rewriter.getUnitAttr());
+      forOp->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
     });
     return success();
   }
@@ -351,7 +350,7 @@ struct ForLoopPeeling : public impl::SCFForLoopPeelingBase<ForLoopPeeling> {
 
     // Drop the markers.
     parentOp->walk([](Operation *op) {
-      op->removeAttr(kPeeledLoopLabel);
+      op->removeAttr(scf::kPeeledLoopLabel);
       op->removeAttr(kPartialIterationLabel);
     });
   }

>From 01f25f7b0844c05cca5e70d9314e3ae8ac5f1de7 Mon Sep 17 00:00:00 2001
From: Shay Kleiman <shay.kleiman at mobileye.com>
Date: Tue, 21 Jul 2026 16:14:04 +0300
Subject: [PATCH 2/2] cr

Change-Id: I57556c92a6889719537955f5e92932c7faeef678
---
 mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h |  4 ++--
 .../lib/Dialect/SCF/Transforms/LoopSpecialization.cpp | 11 +++++++----
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 3b96bd77f6928..a11a60629b844 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -106,8 +106,8 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
 LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
                                         scf::ForOp &partialIteration);
 
-/// Marker attribute name for loops that have already been peeled.
-static constexpr StringLiteral kPeeledLoopLabel = "__peeled_loop__";
+/// Attribute name used to mark loops that have already been peeled.
+StringRef getPeeledLoopAttrName();
 
 /// Tile a parallel loop of the form
 ///   scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index a8609b2c9eafb..53359b46bde57 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -258,6 +258,8 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
 
 static constexpr char kPartialIterationLabel[] = "__partial_iteration__";
 
+StringRef mlir::scf::getPeeledLoopAttrName() { return "__peeled_loop__"; }
+
 namespace {
 struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
   ForLoopPeelingPattern(MLIRContext *ctx, bool peelFront, bool skipPartial)
@@ -271,7 +273,7 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
                                          "unsigned loops are not supported");
 
     // Do not peel already peeled loops.
-    if (forOp->hasAttr(scf::kPeeledLoopLabel))
+    if (forOp->hasAttr(scf::getPeeledLoopAttrName()))
       return failure();
 
     scf::ForOp partialIteration;
@@ -299,11 +301,12 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
 
     // Apply label, so that the same loop is not rewritten a second time.
     rewriter.modifyOpInPlace(partialIteration, [&]() {
-      partialIteration->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
+      partialIteration->setAttr(scf::getPeeledLoopAttrName(),
+                                rewriter.getUnitAttr());
       partialIteration->setAttr(kPartialIterationLabel, rewriter.getUnitAttr());
     });
     rewriter.modifyOpInPlace(forOp, [&]() {
-      forOp->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
+      forOp->setAttr(scf::getPeeledLoopAttrName(), rewriter.getUnitAttr());
     });
     return success();
   }
@@ -350,7 +353,7 @@ struct ForLoopPeeling : public impl::SCFForLoopPeelingBase<ForLoopPeeling> {
 
     // Drop the markers.
     parentOp->walk([](Operation *op) {
-      op->removeAttr(scf::kPeeledLoopLabel);
+      op->removeAttr(scf::getPeeledLoopAttrName());
       op->removeAttr(kPartialIterationLabel);
     });
   }


From llvmlistbot at llvm.org  Tue Jul 21 08:55:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 08:55:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Analysis][Presburger] Remove redundant
 local vars after composition (PR #211023)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211023@github.com>
Message-ID: <6a5f968f.e624eb8d.4a8a7.e767@mx.google.com>


https://github.com/asraa updated https://github.com/llvm/llvm-project/pull/211023

>From 7c9dd6361f8459ae18c996396b1a4319dbe55333 Mon Sep 17 00:00:00 2001
From: Asra Ali <asraa at google.com>
Date: Tue, 21 Jul 2026 15:29:53 +0000
Subject: [PATCH] [mlir] integer relation: eliminate local vars when composing

Signed-off-by: Asra Ali <asraa at google.com>
---
 .../Analysis/Presburger/IntegerRelation.cpp   |  3 +++
 .../Presburger/IntegerRelationTest.cpp        | 19 +++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
index 93a725571078e..94a15dc3f8663 100644
--- a/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
+++ b/mlir/lib/Analysis/Presburger/IntegerRelation.cpp
@@ -2573,6 +2573,9 @@ void IntegerRelation::compose(const IntegerRelation &rel) {
 
   // Project out B in R1.
   convertVarKind(VarKind::Range, 0, numBVars, VarKind::Local);
+
+  // Eliminate local variables
+  removeRedundantLocalVars();
 }
 
 void IntegerRelation::applyDomain(const IntegerRelation &rel) {
diff --git a/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp b/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
index 90753d502d12c..c3da1c1556d26 100644
--- a/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
+++ b/mlir/unittests/Analysis/Presburger/IntegerRelationTest.cpp
@@ -119,6 +119,25 @@ TEST(IntegerRelationTest, applyDomainAndRange) {
   }
 }
 
+TEST(IntegerRelationTest, composeRemoveRedundantLocalVars) {
+  // map x to  y = x + 10.
+  IntegerRelation map1 =
+      parseRelationFromSet("(x, y) : (y - x - 10 == 0)", /*numDomain=*/1);
+
+  // map y to z = y + 20.
+  IntegerRelation map2 =
+      parseRelationFromSet("(y, z) : (z - y - 20 == 0)", /*numDomain=*/1);
+
+  // composing projects out y (converting it to a local variable)
+  map1.compose(map2);
+
+  // y is fully determined by x so should be eliminated after composition
+  EXPECT_EQ(map1.getNumLocalVars(), 0u);
+  IntegerRelation expectedMap =
+      parseRelationFromSet("(x, z) : (z - x - 30 == 0)", /*numDomain=*/1);
+  EXPECT_TRUE(map1.isEqual(expectedMap));
+}
+
 TEST(IntegerRelationTest, symbolicLexmin) {
   SymbolicLexOpt lexmin =
       parseRelationFromSet("(a, x)[b] : (x - a >= 0, x - b >= 0)", 1)


From llvmlistbot at llvm.org  Tue Jul 21 08:56:25 2026
From: llvmlistbot at llvm.org (Shay Kleiman)
Date: Tue, 21 Jul 2026 08:56:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][scf] Expose kPeeledLoopLabel as a
 public marker attribute (PR #210922)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210922@github.com>
Message-ID: <6a5f96a9.6c8668f2.1842d3.6e64@mx.google.com>


https://github.com/shay-kl updated https://github.com/llvm/llvm-project/pull/210922

>From 5c7a50eb59808b89c343e38fbb1f4ba98027dc05 Mon Sep 17 00:00:00 2001
From: Shay Kleiman <shay.kleiman at mobileye.com>
Date: Tue, 21 Jul 2026 12:15:15 +0300
Subject: [PATCH 1/2] [mlir][scf] Expose kPeeledLoopLabel as a public marker
 attribute

Change-Id: I08fa8bcab6cf76bc36894fb79aa32635c1aebe15
---
 mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h  | 3 +++
 mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp | 9 ++++-----
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 00e8572307151..3b96bd77f6928 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -106,6 +106,9 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
 LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
                                         scf::ForOp &partialIteration);
 
+/// Marker attribute name for loops that have already been peeled.
+static constexpr StringLiteral kPeeledLoopLabel = "__peeled_loop__";
+
 /// Tile a parallel loop of the form
 ///   scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
 ///                                             step (%arg4, %arg5)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index a39e5520a144b..a8609b2c9eafb 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -256,7 +256,6 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
   return success();
 }
 
-static constexpr char kPeeledLoopLabel[] = "__peeled_loop__";
 static constexpr char kPartialIterationLabel[] = "__partial_iteration__";
 
 namespace {
@@ -272,7 +271,7 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
                                          "unsigned loops are not supported");
 
     // Do not peel already peeled loops.
-    if (forOp->hasAttr(kPeeledLoopLabel))
+    if (forOp->hasAttr(scf::kPeeledLoopLabel))
       return failure();
 
     scf::ForOp partialIteration;
@@ -300,11 +299,11 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
 
     // Apply label, so that the same loop is not rewritten a second time.
     rewriter.modifyOpInPlace(partialIteration, [&]() {
-      partialIteration->setAttr(kPeeledLoopLabel, rewriter.getUnitAttr());
+      partialIteration->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
       partialIteration->setAttr(kPartialIterationLabel, rewriter.getUnitAttr());
     });
     rewriter.modifyOpInPlace(forOp, [&]() {
-      forOp->setAttr(kPeeledLoopLabel, rewriter.getUnitAttr());
+      forOp->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
     });
     return success();
   }
@@ -351,7 +350,7 @@ struct ForLoopPeeling : public impl::SCFForLoopPeelingBase<ForLoopPeeling> {
 
     // Drop the markers.
     parentOp->walk([](Operation *op) {
-      op->removeAttr(kPeeledLoopLabel);
+      op->removeAttr(scf::kPeeledLoopLabel);
       op->removeAttr(kPartialIterationLabel);
     });
   }

>From fa87261bba539d496a93d0e6003113e252cf619b Mon Sep 17 00:00:00 2001
From: Shay Kleiman <shay.kleiman at mobileye.com>
Date: Tue, 21 Jul 2026 16:14:04 +0300
Subject: [PATCH 2/2] cr

Change-Id: I57556c92a6889719537955f5e92932c7faeef678
---
 mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h |  4 ++--
 .../lib/Dialect/SCF/Transforms/LoopSpecialization.cpp | 11 +++++++----
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
index 3b96bd77f6928..d95a4e31a5b5b 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/Transforms.h
@@ -106,8 +106,8 @@ LogicalResult peelForLoopAndSimplifyBounds(RewriterBase &rewriter, ForOp forOp,
 LogicalResult peelForLoopFirstIteration(RewriterBase &rewriter, ForOp forOp,
                                         scf::ForOp &partialIteration);
 
-/// Marker attribute name for loops that have already been peeled.
-static constexpr StringLiteral kPeeledLoopLabel = "__peeled_loop__";
+/// Returns the attribute name used to mark loops that have already been peeled.
+StringRef getPeeledLoopAttrName();
 
 /// Tile a parallel loop of the form
 ///   scf.parallel (%i0, %i1) = (%arg0, %arg1) to (%arg2, %arg3)
diff --git a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
index a8609b2c9eafb..53359b46bde57 100644
--- a/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/LoopSpecialization.cpp
@@ -258,6 +258,8 @@ LogicalResult mlir::scf::peelForLoopFirstIteration(RewriterBase &b, ForOp forOp,
 
 static constexpr char kPartialIterationLabel[] = "__partial_iteration__";
 
+StringRef mlir::scf::getPeeledLoopAttrName() { return "__peeled_loop__"; }
+
 namespace {
 struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
   ForLoopPeelingPattern(MLIRContext *ctx, bool peelFront, bool skipPartial)
@@ -271,7 +273,7 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
                                          "unsigned loops are not supported");
 
     // Do not peel already peeled loops.
-    if (forOp->hasAttr(scf::kPeeledLoopLabel))
+    if (forOp->hasAttr(scf::getPeeledLoopAttrName()))
       return failure();
 
     scf::ForOp partialIteration;
@@ -299,11 +301,12 @@ struct ForLoopPeelingPattern : public OpRewritePattern<ForOp> {
 
     // Apply label, so that the same loop is not rewritten a second time.
     rewriter.modifyOpInPlace(partialIteration, [&]() {
-      partialIteration->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
+      partialIteration->setAttr(scf::getPeeledLoopAttrName(),
+                                rewriter.getUnitAttr());
       partialIteration->setAttr(kPartialIterationLabel, rewriter.getUnitAttr());
     });
     rewriter.modifyOpInPlace(forOp, [&]() {
-      forOp->setAttr(scf::kPeeledLoopLabel, rewriter.getUnitAttr());
+      forOp->setAttr(scf::getPeeledLoopAttrName(), rewriter.getUnitAttr());
     });
     return success();
   }
@@ -350,7 +353,7 @@ struct ForLoopPeeling : public impl::SCFForLoopPeelingBase<ForLoopPeeling> {
 
     // Drop the markers.
     parentOp->walk([](Operation *op) {
-      op->removeAttr(scf::kPeeledLoopLabel);
+      op->removeAttr(scf::getPeeledLoopAttrName());
       op->removeAttr(kPartialIterationLabel);
     });
   }


From llvmlistbot at llvm.org  Tue Jul 21 09:47:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 09:47:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] e959a28 - [mlir][openacc] Do not erase acc
 routine information early (#210813)
Message-ID: <6a5fa2bc.bc28b987.2e0c9a.d7ef@mx.google.com>


Author: Valentin Clement (バレンタイン クレメン)
Date: 2026-07-21T09:47:51-07:00
New Revision: e959a280d5ac91d219abb802913c4e5cadb45f2a

URL: https://github.com/llvm/llvm-project/commit/e959a280d5ac91d219abb802913c4e5cadb45f2a
DIFF: https://github.com/llvm/llvm-project/commit/e959a280d5ac91d219abb802913c4e5cadb45f2a.diff

LOG: [mlir][openacc] Do not erase acc routine information early (#210813)

acc routine information is used in ACCCGToGPU to compute parallelism. If
the info is missing, predicated region might be wrong and lead to
deadlock.

Just keep them in ACCRountineToGPUFunc and they will be cleaned up later
in the pipeline.

Added: 
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-bound-routine-call.mlir

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCRoutineToGPUFunc.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCRoutineToGPUFunc.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCRoutineToGPUFunc.cpp
index 27a34695decb2..6bbb4b53cb5bb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCRoutineToGPUFunc.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCRoutineToGPUFunc.cpp
@@ -320,8 +320,6 @@ class ACCRoutineToGPUFunc
       return signalPassFailure();
 
     cleanupHostModule(funcsToClone);
-    for (RoutineOp bindOp : bindAccRoutines)
-      bindOp.erase();
   }
 };
 

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-bound-routine-call.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-bound-routine-call.mlir
new file mode 100644
index 0000000000000..c55f87b73d181
--- /dev/null
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-bound-routine-call.mlir
@@ -0,0 +1,28 @@
+// RUN: mlir-opt %s --pass-pipeline="builtin.module(func.func(acc-bind-routine),acc-routine-to-gpu-func,func.func(acc-cg-to-gpu))" | FileCheck %s
+
+// Bound acc.routine metadata must remain available after ACCBindRoutine and
+// ACCRoutineToGPUFunc so ACCCGToGPU can classify the bound callee as vector.
+// Otherwise the call inside acc.predicate_region is incorrectly predicated to
+// thread_id.x == 0.
+
+// CHECK-LABEL: func.func @bound_vector_call
+// CHECK: gpu.launch
+// CHECK-NOT: scf.if
+// CHECK: func.call @bound_vector
+func.func @bound_vector_call(%arg0: memref<4xf32>) {
+  %c32 = arith.constant 32 : index
+  %thread_x = acc.par_width %c32 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%tx = %thread_x) ins(%arg10 = %arg0) : (memref<4xf32>) {
+    acc.predicate_region {
+      func.call @wrapped_vector(%arg10) : (memref<4xf32>) -> ()
+    }
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+func.func private @wrapped_vector(memref<4xf32>)
+    attributes {acc.routine_info = #acc.routine_info<[@acc_routine_bound_vector]>}
+acc.routine @acc_routine_bound_vector func(@wrapped_vector) vector bind("bound_vector")
+func.func private @bound_vector(memref<4xf32>)
+    attributes {acc.routine_info = #acc.routine_info<[@acc_routine_bound_vector]>}


        

From llvmlistbot at llvm.org  Tue Jul 21 09:47:59 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 21 Jul 2026 09:47:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][openacc] Do not erase acc routine
 information early (PR #210813)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210813@github.com>
Message-ID: <6a5fa2bf.160324b0.17e275.b3d7@mx.google.com>


https://github.com/clementval closed https://github.com/llvm/llvm-project/pull/210813

From llvmlistbot at llvm.org  Tue Jul 21 09:49:10 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 09:49:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a5fa306.e624eb8d.4a8a7.e4fb@mx.google.com>


FedericoBruzzone wrote:

> thanks for looking into this; can you please add a test case (fail-before, pass-after)

Thanks for the suggestion 🙏 
I hope this meets your expectations.

https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Tue Jul 21 09:51:00 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Tue, 21 Jul 2026 09:51:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a5fa374.92ea0e2a.4a9a5.5017@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/210952

>From ee815c22e10991ea39025101b8bc71d00558f0a4 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:28 +0200
Subject: [PATCH 1/3] [mlir][sparse] Avoid vectorizing non-contiguous COO
 coordinate loads

SparseVectorization emits vector.maskedload/maskedstore for direct
loop accesses, assuming consecutive loop indices map to consecutive
memory. This is false for sparse_tensor.coordinates of a level inside
a trailing AoS COO region, whose buffer is interleaved with other
levels -- a real, silent miscompile.

Fall back to a scalar load/store when the stride is statically known
to be non-unit, including cases only derivable from the sparse
tensor's encoding before sparse-tensor-codegen materializes the
concrete stride.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../Transforms/SparseVectorization.cpp        | 50 +++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();

>From 01b65145ac56c914001612fc6a6ed72f138fa976 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 18:31:37 +0200
Subject: [PATCH 2/3] [mlir][sparse] Add fail-before/pass-after test for COO
 vectorization fix

Adds a dedicated pass-level test: vectorizing X(i,j) = A(i,j) * j over
a compressed(nonunique)+singleton (COO) tensor must not vectorize the
level-1 coordinate load, since that level's buffer is interleaved
(AoS) and not contiguous.

Verified this test fails on main (pre-fix): the loop gets vectorized
with vl=2 and vector.maskedload reads the interleaved buffer as if
contiguous. It passes on this branch: the loop stays scalar (step 1).

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../SparseTensor/sparse_vector_coo.mlir       | 105 ++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir

diff --git a/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir
new file mode 100644
index 0000000000000..9b13b15a60e87
--- /dev/null
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir
@@ -0,0 +1,105 @@
+// RUN: mlir-opt %s --sparse-reinterpret-map -sparsification -cse -sparse-vectorization="vl=2" -cse | FileCheck %s
+
+// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
+
+#SortedCOO = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton)
+}>
+
+#trait_index = {
+  indexing_maps = [
+    affine_map<(i,j) -> (i,j)>,  // A
+    affine_map<(i,j) -> (i,j)>   // X (out)
+  ],
+  iterator_types = ["parallel", "parallel"],
+  doc = "X(i,j) = A(i,j) * j"
+}
+
+// CHECK: #[[$ATTR_0:.+]] = #sparse_tensor.encoding<{ map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton) }>
+// CHECK-LABEL:   func.func @sparse_index_2d_coo(
+// CHECK-SAME:      %[[ARG0:.*]]: tensor<8x8xi64, #[[$ATTR_0]]>) -> tensor<8x8xi64> {
+// CHECK:           %[[CONSTANT_0:.*]] = arith.constant true
+// CHECK:           %[[CONSTANT_1:.*]] = arith.constant false
+// CHECK:           %[[CONSTANT_2:.*]] = arith.constant 1 : index
+// CHECK:           %[[CONSTANT_3:.*]] = arith.constant 0 : index
+// CHECK:           %[[CONSTANT_4:.*]] = arith.constant 0 : i64
+// CHECK:           %[[EMPTY_0:.*]] = tensor.empty() : tensor<8x8xi64>
+// CHECK:           %[[VALUES_0:.*]] = sparse_tensor.values %[[ARG0]] : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xi64>
+// CHECK:           %[[TO_BUFFER_0:.*]] = bufferization.to_buffer %[[EMPTY_0]] : tensor<8x8xi64> to memref<8x8xi64>
+// CHECK:           linalg.fill ins(%[[CONSTANT_4]] : i64) outs(%[[TO_BUFFER_0]] : memref<8x8xi64>)
+// CHECK:           %[[POSITIONS_0:.*]] = sparse_tensor.positions %[[ARG0]] {level = 0 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex>
+// CHECK:           %[[COORDINATES_0:.*]] = sparse_tensor.coordinates %[[ARG0]] {level = 0 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK:           %[[COORDINATES_1:.*]] = sparse_tensor.coordinates %[[ARG0]] {level = 1 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK:           %[[LOAD_0:.*]] = memref.load %[[POSITIONS_0]]{{\[}}%[[CONSTANT_3]]] : memref<?xindex>
+// CHECK:           %[[LOAD_1:.*]] = memref.load %[[POSITIONS_0]]{{\[}}%[[CONSTANT_2]]] : memref<?xindex>
+// CHECK:           %[[WHILE_0:.*]] = scf.while (%[[VAL_0:.*]] = %[[LOAD_0]]) : (index) -> index {
+// CHECK:             %[[CMPI_0:.*]] = arith.cmpi ult, %[[VAL_0]], %[[LOAD_1]] : index
+// CHECK:             %[[IF_0:.*]] = scf.if %[[CMPI_0]] -> (i1) {
+// CHECK:               %[[LOAD_2:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[LOAD_0]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:               %[[LOAD_3:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_0]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:               %[[CMPI_1:.*]] = arith.cmpi eq, %[[LOAD_2]], %[[LOAD_3]] : index
+// CHECK:               scf.yield %[[CMPI_1]] : i1
+// CHECK:             } else {
+// CHECK:               scf.yield %[[CONSTANT_1]] : i1
+// CHECK:             }
+// CHECK:             scf.condition(%[[IF_0]]) %[[VAL_0]] : index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_1:.*]]: index):
+// CHECK:             %[[ADDI_0:.*]] = arith.addi %[[VAL_1]], %[[CONSTANT_2]] : index
+// CHECK:             scf.yield %[[ADDI_0]] : index
+// CHECK:           }
+// CHECK:           %[[WHILE_1:.*]]:2 = scf.while (%[[VAL_2:.*]] = %[[LOAD_0]], %[[VAL_3:.*]] = %[[WHILE_0]]) : (index, index) -> (index, index) {
+// CHECK:             %[[CMPI_2:.*]] = arith.cmpi ult, %[[VAL_2]], %[[LOAD_1]] : index
+// CHECK:             scf.condition(%[[CMPI_2]]) %[[VAL_2]], %[[VAL_3]] : index, index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_4:.*]]: index, %[[VAL_5:.*]]: index):
+// CHECK:             %[[LOAD_4:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_4]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             scf.if %[[CONSTANT_0]] {
+// CHECK:               scf.for %[[VAL_6:.*]] = %[[VAL_4]] to %[[VAL_5]] step %[[CONSTANT_2]] {
+// CHECK:                 %[[LOAD_5:.*]] = memref.load %[[COORDINATES_1]]{{\[}}%[[VAL_6]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[LOAD_6:.*]] = memref.load %[[VALUES_0]]{{\[}}%[[VAL_6]]] : memref<?xi64>
+// CHECK:                 %[[INDEX_CAST_0:.*]] = arith.index_cast %[[LOAD_5]] : index to i64
+// CHECK:                 %[[MULI_0:.*]] = arith.muli %[[LOAD_6]], %[[INDEX_CAST_0]] : i64
+// CHECK:                 memref.store %[[MULI_0]], %[[TO_BUFFER_0]]{{\[}}%[[LOAD_4]], %[[LOAD_5]]] : memref<8x8xi64>
+// CHECK:               } {"Emitted from" = "linalg.generic"}
+// CHECK:             } else {
+// CHECK:             }
+// CHECK:             %[[IF_1:.*]]:2 = scf.if %[[CONSTANT_0]] -> (index, index) {
+// CHECK:               %[[WHILE_2:.*]] = scf.while (%[[VAL_7:.*]] = %[[VAL_5]]) : (index) -> index {
+// CHECK:                 %[[CMPI_3:.*]] = arith.cmpi ult, %[[VAL_7]], %[[LOAD_1]] : index
+// CHECK:                 %[[IF_2:.*]] = scf.if %[[CMPI_3]] -> (i1) {
+// CHECK:                   %[[LOAD_7:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_5]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                   %[[LOAD_8:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_7]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                   %[[CMPI_4:.*]] = arith.cmpi eq, %[[LOAD_7]], %[[LOAD_8]] : index
+// CHECK:                   scf.yield %[[CMPI_4]] : i1
+// CHECK:                 } else {
+// CHECK:                   scf.yield %[[CONSTANT_1]] : i1
+// CHECK:                 }
+// CHECK:                 scf.condition(%[[IF_2]]) %[[VAL_7]] : index
+// CHECK:               } do {
+// CHECK:               ^bb0(%[[VAL_8:.*]]: index):
+// CHECK:                 %[[ADDI_1:.*]] = arith.addi %[[VAL_8]], %[[CONSTANT_2]] : index
+// CHECK:                 scf.yield %[[ADDI_1]] : index
+// CHECK:               }
+// CHECK:               scf.yield %[[VAL_5]], %[[WHILE_2]] : index, index
+// CHECK:             } else {
+// CHECK:               scf.yield %[[VAL_4]], %[[VAL_5]] : index, index
+// CHECK:             }
+// CHECK:             scf.yield %[[VAL_9:.*]]#0, %[[VAL_9]]#1 : index, index
+// CHECK:           } attributes {"Emitted from" = "linalg.generic"}
+// CHECK:           %[[TO_TENSOR_0:.*]] = bufferization.to_tensor %[[TO_BUFFER_0]] : memref<8x8xi64> to tensor<8x8xi64>
+// CHECK:           return %[[TO_TENSOR_0]] : tensor<8x8xi64>
+// CHECK:         }
+func.func @sparse_index_2d_coo(%arga: tensor<8x8xi64, #SortedCOO>) -> tensor<8x8xi64> {
+  %init = tensor.empty() : tensor<8x8xi64>
+  %r = linalg.generic #trait_index
+      ins(%arga: tensor<8x8xi64, #SortedCOO>)
+     outs(%init: tensor<8x8xi64>) {
+      ^bb(%a: i64, %x: i64):
+        %j = linalg.index 1 : index
+        %jj = arith.index_cast %j : index to i64
+        %m1 = arith.muli %a, %jj : i64
+        linalg.yield %m1 : i64
+  } -> tensor<8x8xi64>
+  return %r : tensor<8x8xi64>
+}

>From 2f328a1688b0e8ecf5ec7f88543da1003e4d8bfe Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:54 +0200
Subject: [PATCH 3/3] [mlir][vector] Reject non-unit strides on
 masked/expand/compress ops

vector.maskedload/maskedstore/expandload/compressstore lower to LLVM
masked intrinsics that read/write N *consecutive* elements from a
single pointer, but none of them verified the memref's minor-dim
stride, so e.g. `strided<[2]>` verified successfully and silently
miscompiled.

Reject statically-known non-unit strides; a dynamic stride is still
accepted (see linked issue for discussion on why this differs from
vector.load/store). expandload/compressstore also gain the
negative-stride check vector.load/store already have.

Depends on the SparseVectorization fix in the preceding commit: the
sparsifier's vectorizer previously relied on maskedload having no
stride check to (unsafely) vectorize non-contiguous COO coordinate
buffers.

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 51 ++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir    | 77 ++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index f37083803a2a1..8b594c378912e 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
+/// statically known non-unit stride; a dynamic (not provably unit) stride
+/// is accepted.
+///
+/// This is more permissive than vector.load/store's stride check: ops such
+/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
+/// also used on memrefs whose most minor dimension is contiguous at runtime
+/// but not provable as such at the type level (e.g., buffers produced by
+/// the sparsifier).
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6326,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6395,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6681,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6702,6 +6744,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..9ec17024bc1d3 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1742,6 +1760,24 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1796,6 +1832,24 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
@@ -2179,6 +2233,19 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
+// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
+// (unprovable) stride is rejected here too: vector.load/store require a
+// statically known unit stride, with no exception for strides that merely
+// aren't provably non-unit.
+func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
+  %0 = vector.load %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.store
 //===----------------------------------------------------------------------===//
@@ -2215,6 +2282,16 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
+// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
+// unlike vector.maskedstore.
+func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
+  // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
+  vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 func.func @store_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %val: vector<4xf32>) {
   // expected-error @+2 {{'vector.store' op memref strides must be non-negative}}
   %c0 = arith.constant 0 : index


From llvmlistbot at llvm.org  Tue Jul 21 09:51:21 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Tue, 21 Jul 2026 09:51:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a5fa389.c56fb102.6921f.c4a1@mx.google.com>


================
@@ -189,6 +190,26 @@ class LinearClauseProcessor {
     }
   }
 
+  // Find linear iteration variable and save it for later updates
+  void initLinearIV(omp::SimdOp simdOp) {
+    auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
+    // NOTE iteration variables can only be linear in non-nested loops.
+    if (loopOp.getIVs().size() != 1)
+      return;
+    // The linear IV is the loop IV's store address.
+    BlockArgument arg = loopOp.getIVs().front();
+    for (const Operation *user : arg.getUsers()) {
+      if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
+        for (Value linearVar : simdOp.getLinearVars()) {
+          if (linearVar == storeOp.getAddr()) {
+            linearLoopIV = linearVar;
+            break;
----------------
tblah wrote:

I think it would be okay to stick with this patch and just add an error for this case. Maybe I was too strong in suggesting an MLIR verifier error - this could perhaps be a translation error (ideally all valid MLIR can be translated, but that could make things much harder than they need to be...).

So then if some frontend generates this IR we can't understand, we at least crash rather than generate potentially invalid code.

(2) is the most correct solution IMO but I can understand why you don't want to do all of that.

If you would prefer to do (3) rather than my suggestion then that is okay by me.

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Tue Jul 21 10:47:19 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Tue, 21 Jul 2026 10:47:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5fb0a7.6c8668f2.1842d3.b195@mx.google.com>


================
@@ -0,0 +1,360 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_lhs
+//
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size for dim M is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size for dim N is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
+
+  return %C_pack : tensor<1x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dims are scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
+      : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
----------------
banach-space wrote:

IIUC, this is still related to https://github.com/llvm/llvm-project/pull/208226#discussion_r3581787011 ?

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 10:51:54 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Tue, 21 Jul 2026 10:51:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a5fb1ba.824e11a6.322560.a284@mx.google.com>


================
@@ -0,0 +1,360 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_lhs
+//
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size for dim M is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size for dim N is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
+
+  return %C_pack : tensor<1x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dims are scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
+      : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
----------------
steplong wrote:

Nope, this is about why there are still masks even though we are using `assume_dynamic_dims_match_vec_sizes`. I think it's because of the odd dimension 13. It's not generated from the vectorizer.

https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Tue Jul 21 10:53:31 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 21 Jul 2026 10:53:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][openacc] Support already existing
 global in GPU module (PR #210806)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210806@github.com>
Message-ID: <6a5fb21b.13be6df8.323b61.b697@mx.google.com>


================
@@ -99,20 +98,22 @@ class ACCDeclareGPUModuleInsertion
       StringAttr name = symOp.getNameAttr();
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
-        // Reuse only when the existing GPU symbol is structurally equivalent to
-        // the global we would insert. Otherwise treat as a conflict (different
-        // op type or different definition).
-        if (existing->getName() != globalOp.getName() ||
-            !OperationEquivalence::isEquivalentTo(
----------------
clementval wrote:

The difference is that one global has the acc declare attribute already. 

```
fir.global linkonce_odr @g1 constant : !fir.char<1> {
  %0 = fir.string_lit "p"(1) : !fir.char<1>
  fir.has_value %0 : !fir.char<1>
}
fir.global linkonce_odr @g1 {acc.declare = #acc.declare<dataClause = acc_copyin>} constant : !fir.char<1> {
  %0 = fir.string_lit "p"(1) : !fir.char<1>
  fir.has_value %0 : !fir.char<1>
}
```

https://github.com/llvm/llvm-project/pull/210806

From llvmlistbot at llvm.org  Tue Jul 21 11:47:18 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Tue, 21 Jul 2026 11:47:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter materialization;
 use a status enum for the type conversion callback (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a5fbeb6.3cce8983.247e47.3421@mx.google.com>


makslevental wrote:

@PragmaTwice can you take a quick scan again since I moved the ternary into this PR.

https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Tue Jul 21 12:48:58 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Tue, 21 Jul 2026 12:48:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve discardable attrs
 when adding loop yields (PR #210571)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210571@github.com>
Message-ID: <6a5fcd2a.f1c1923d.d9163.4f8e@mx.google.com>


joker-eph wrote:

Please trim the commit description (streamline, remove section), this has just too much AI-generated noise right now.

This should also describe why it is safe / reasonable to do: in general discardable attributes **should** be discarded by transformation that don't understand them, because their semantic may not be preserved (for example they could refer to existing operands/results that are removed/swapped by a transformation).


https://github.com/llvm/llvm-project/pull/210571

From llvmlistbot at llvm.org  Tue Jul 21 12:51:03 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 12:51:03 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [DebugInfo] Add symbolic branches to
	DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5fcda7.d681cab9.33af1e.2cdc@mx.google.com>


echristo wrote:

> I too would like to see the rationale / use-case for this -- a new form of optimisation (or opportunity for one) that can only be recovered with conditional branching perhaps?
> 

GPU debugging, and Tile DSLs in particular, are coming up with more and more complicated expressions and so we're needing to use more of the dwarf standard - heck, these are from dwarf2. :) I also have a procedures (dwarf3) patch and a change for calls in expressions coming (in order to call those procedures) and newer extensions (e.g. dwarf6) are going to be used in order to describe what's going on. There's a little bit of chicken and egg here with producer(s)/consumers so starting here seemed like a good place so that we had something for folks to target as well as emission to work from.

> As it stands, `DIExpression`s have to be decompiled by various parts of LLVM to edit them, and taking this direction is only going to make that process more complicated. If we're going to support these opcodes we should probably have a plan for making the whole situation managable (although that might be a lot of future work).

Agreed. I tried to keep the changes fairly minimal here so that the support existed, but didn't require a lot of changes to expression rewriters, etc. About the only other patch split I could come up with was boilerplate and emission, but the emission seemed to be straightforward enough with some small initial limitations in use.

Thoughts? and thanks. I'll add you to the review, I'd meant to do it before :)


https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 12:55:50 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 12:55:50 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5fcec6.e5907290.281687.5a6b@mx.google.com>


https://github.com/echristo updated https://github.com/llvm/llvm-project/pull/210850

>From ab4cecf7ddb25e85c1a7a8bbfc38743dc2125ac2 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Wed, 15 Jul 2026 08:51:28 -0700
Subject: [PATCH 1/2] [DebugInfo] Add symbolic branches to DIExpression

Added three pseudo-ops to DIExpression:

- DW_OP_LLVM_label declares a label and emits no bytes.
- DW_OP_LLVM_bra branches to a label when the top of the stack is non-zero.
- DW_OP_LLVM_skip always branches to a label.

DW_OP_bra and DW_OP_skip store a two-byte offset in [-32768, 32767], but
we don't know those offsets until we emit the expression. DIExpression keeps
label IDs, and we fix them up late in CodeGen.

CodeGen records each label's byte offset and leaves a zero placeholder for each
branch. Once it has seen the whole expression, it patches the placeholders in
the existing temporary buffers used for DIE and location-list expressions.
Both paths keep their current encoding and target byte order.

The verifier makes sure every branch has a label in the same expression and
each label ID is declared once. Labels don't need a branch, and branches can go
forward, backward, to themselves, or form cycles. Raw DW_OP_bra and DW_OP_skip
remain invalid in IR because their offsets depend on the final encoding.

Symbolic control flow only works with one location operand. We reject expressions
that combine it with DW_OP_LLVM_arg or DW_OP_LLVM_implicit_pointer, and diagnose
DIArgList records. Existing expression rewrites keep the pseudo-ops and their
IDs in place.

CodeGen reports an error if the final offset is outside [-32768, 32767], or if a
label, branch, or skip splits a deferred conversion. DWARF 5 emits DW_OP_convert
directly, but older versions may defer one conversion until they see the next.

Tests cover IR and MLIR round trips, verifier failures, forward and backward
branches, self-branches and cycles, both byte orders, conversion boundaries,
and out-of-range offsets.
---
 llvm/docs/LangRef.md                          |  16 ++
 llvm/docs/ReleaseNotes.md                     |   6 +
 llvm/docs/SourceLevelDebugging.md             |  61 ++++++-
 llvm/include/llvm/BinaryFormat/Dwarf.h        |   5 +
 llvm/lib/BinaryFormat/Dwarf.cpp               |   9 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  27 ++-
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  83 +++++++++-
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h |  27 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     |  52 +++++-
 llvm/lib/IR/DebugInfoMetadata.cpp             | 103 ++++++++++--
 llvm/lib/IR/Verifier.cpp                      |  13 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../X86/di-expression-symbolic-branches.mir   | 102 ++++++++++++
 .../di-expression-symbolic-branches.ll        |  44 +++++
 ...expression-symbolic-branch-out-of-range.py |  33 ++++
 ...ession-symbolic-branch-convert-boundary.ll |  96 +++++++++++
 ...expression-symbolic-branch-out-of-range.ll |   7 +
 .../X86/di-expression-symbolic-branches.ll    |  84 ++++++++++
 .../di-expression-symbolic-branches.ll        |  93 +++++++++++
 llvm/unittests/BinaryFormat/DwarfTest.cpp     |   7 +
 llvm/unittests/IR/MetadataTest.cpp            | 155 ++++++++++++++++++
 .../Import/debug-info-symbolic-branches.ll    |  23 +++
 .../llvmir-debug-symbolic-branches.mlir       |  19 +++
 24 files changed, 1054 insertions(+), 35 deletions(-)
 create mode 100644 llvm/test/Assembler/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Bitcode/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
 create mode 100644 llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Verifier/di-expression-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..05162290f330a 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7159,6 +7159,22 @@ Some examples of expressions:
 !DIExpression(DW_OP_constu, 42, DW_OP_stack_value)
 ```
 
+`DIExpression` uses three pseudo-ops for local control flow:
+
+```text
+DW_OP_LLVM_label, <label-id>
+DW_OP_LLVM_bra,   <label-id>
+DW_OP_LLVM_skip,  <label-id>
+```
+
+`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
+branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
+branches. Label IDs are local to the expression. Every branch needs a matching
+label, labels without branches are valid, and each ID can only be declared
+once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
+
+See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+
 ##### DIAssignID
 
 `DIAssignID` nodes have no operands and are always distinct. They are used to
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 94208f8e36d19..988740a7e5e39 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -109,6 +109,12 @@ Makes programs 10x faster by doing Special New Thing.
 
 ### Changes to the Debug Info
 
+* Added `DW_OP_LLVM_label`, `DW_OP_LLVM_bra`, and `DW_OP_LLVM_skip` for symbolic
+  branches in `DIExpression`. These operations use label IDs that CodeGen
+  resolves when it emits the expression. `DW_OP_LLVM_convert` can appear before
+  or after them, but when CodeGen can't emit `DW_OP_convert`, it reports an
+  error if a deferred conversion reaches a label, branch, or skip.
+
 ### Changes to the LLVM tools
 
 ### Changes to LLDB
diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 74bb8840a56e4..645e441e6fe2a 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -386,10 +386,9 @@ call void @llvm.dbg.assign(
 
 Debug expressions are represented as {ref}`specialized-metadata`.
 
-Debug expressions are interpreted left-to-right: start by pushing the
-value/address operand of the record onto a stack, then repeatedly push and
-evaluate opcodes from the `DIExpression` until the final variable description
-is produced.
+A debug expression starts with the record's value or address operand on the
+stack, then evaluates operations from left to right unless a symbolic branch
+jumps to a label in the same `DIExpression`.
 
 The opcodes available in these expressions are described in
 {ref}`dwarf-opcodes` and {ref}`internal-opcodes`.
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
+
+- Put labels, branches, and skips before `DW_OP_stack_value` and
+  `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
+- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
+  separately, so they can't be used with symbolic control flow.
+- Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
+  applies to every path.
+
+We don't check reachability, termination, or stack state where paths meet.
+
+`DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
+and we don't match conversions on different paths. When CodeGen can't emit
+`DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
+branch, or skip would split the pair, CodeGen reports an error.
+
+Local expression rewrites stop at labels, branches, and skips; they can still
+add operations to either end, but they don't move, remove, or copy labels.
+
+##### Other Internal Opcodes
+
 - `DW_OP_LLVM_fragment, <offset>, <size>` may appear at most once in an
   expression, and must be the last opcode. It specifies the bit offset and bit
   size of the variable fragment being described by the record or intrinsic
@@ -482,11 +525,15 @@ them.
 - `DW_OP_LLVM_convert, 16, DW_ATE_signed` specifies a bit size and encoding
   (`16` and `DW_ATE_signed` here, respectively) to which the top of the
   expression stack is to be converted. Maps into a `DW_OP_convert` operation
-  that references a base type constructed from the supplied values.
+  that references a base type constructed from the supplied values. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for how conversions work
+  with labels, branches, and skips.
 - `DW_OP_LLVM_tag_offset, tag_offset` specifies that a memory tag should be
   optionally applied to the pointer. The memory tag is derived from the given
-  tag offset in an implementation-defined manner. (This does not affect the
-  semantics of the expression containing it.)
+  tag offset in an implementation-defined manner. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for its placement with
+  labels, branches, and skips.
+  (This does not affect the semantics of the expression containing it.)
 - `DW_OP_LLVM_entry_value, N` evaluates a sub-expression as-if it were
   evaluated upon entry to the current call frame.
 
diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 75eb3a163e06f..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -149,6 +149,11 @@ enum LocationAtom {
   DW_OP_LLVM_arg = 0x1005,               ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_sext = 0x1006, ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_zext = 0x1007, ///< Only used in LLVM metadata.
+  // Labels declare IDs local to each expression, and branches use them as
+  // targets.
+  DW_OP_LLVM_label = 0x1008, ///< Only used in LLVM metadata.
+  DW_OP_LLVM_bra = 0x1009,   ///< Only used in LLVM metadata.
+  DW_OP_LLVM_skip = 0x100a,  ///< Only used in LLVM metadata.
 };
 
 enum LlvmUserLocationAtom {
diff --git a/llvm/lib/BinaryFormat/Dwarf.cpp b/llvm/lib/BinaryFormat/Dwarf.cpp
index 9ede7ebbe20ad..431d01c953768 100644
--- a/llvm/lib/BinaryFormat/Dwarf.cpp
+++ b/llvm/lib/BinaryFormat/Dwarf.cpp
@@ -159,6 +159,12 @@ StringRef llvm::dwarf::OperationEncodingString(unsigned Encoding) {
     return "DW_OP_LLVM_extract_bits_sext";
   case DW_OP_LLVM_extract_bits_zext:
     return "DW_OP_LLVM_extract_bits_zext";
+  case DW_OP_LLVM_label:
+    return "DW_OP_LLVM_label";
+  case DW_OP_LLVM_bra:
+    return "DW_OP_LLVM_bra";
+  case DW_OP_LLVM_skip:
+    return "DW_OP_LLVM_skip";
   }
 }
 
@@ -175,6 +181,9 @@ unsigned llvm::dwarf::getOperationEncoding(StringRef OperationEncodingString) {
       .Case("DW_OP_LLVM_arg", DW_OP_LLVM_arg)
       .Case("DW_OP_LLVM_extract_bits_sext", DW_OP_LLVM_extract_bits_sext)
       .Case("DW_OP_LLVM_extract_bits_zext", DW_OP_LLVM_extract_bits_zext)
+      .Case("DW_OP_LLVM_label", DW_OP_LLVM_label)
+      .Case("DW_OP_LLVM_bra", DW_OP_LLVM_bra)
+      .Case("DW_OP_LLVM_skip", DW_OP_LLVM_skip)
       .Default(0);
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfbb35d516534..45f1f13e999fc 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -191,8 +191,14 @@ void DebugLocDwarfExpression::emitUnsigned(uint64_t Value) {
   getActiveStreamer().emitULEB128(Value, Twine(Value));
 }
 
-void DebugLocDwarfExpression::emitData1(uint8_t Value) {
-  getActiveStreamer().emitInt8(Value, Twine(Value));
+void DebugLocDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    getActiveStreamer().emitInt8(Value >> (Byte * 8), Twine(Value));
+  }
 }
 
 void DebugLocDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -232,6 +238,23 @@ void DebugLocDwarfExpression::commitTemporaryBuffer() {
   TmpBuf->Comments.clear();
 }
 
+void DebugLocDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                         uint64_t Value,
+                                                         unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  assert(TmpBuf && Offset < TmpBuf->Bytes.size() &&
+         Size <= TmpBuf->Bytes.size() - Offset &&
+         "invalid temporary buffer offset");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    TmpBuf->Bytes[Offset + I] = Value >> (Byte * 8);
+    if (Offset + I < TmpBuf->Comments.size())
+      TmpBuf->Comments[Offset + I].clear();
+  }
+}
+
 const DIType *DbgVariable::getType() const {
   return getVariable()->getType();
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 9b676f88a0cff..4054817abb95d 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -13,13 +13,16 @@
 #include "DwarfExpression.h"
 #include "DwarfCompileUnit.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallBitVector.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/CodeGen/Register.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
 #include <algorithm>
 
 using namespace llvm;
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+      });
+
+  SmallVector<LabelOffset, 4> Labels;
+  SmallVector<BranchFixup, 4> Fixups;
+  // Buffer the expression until every label has a byte offset, then patch the
+  // branches.
+  if (HasSymbolicBranches)
+    enableTemporaryBuffer();
 
+  std::optional<DIExpression::ExprOperand> PrevConvertOp;
   while (ExprCursor) {
     auto Op = ExprCursor.take();
     uint64_t OpNum = Op->getOp();
@@ -545,9 +571,27 @@ bool DwarfExpression::addExpression(
     }
 
     switch (OpNum) {
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      if (PrevConvertOp)
+        report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
+                           dwarf::OperationEncodingString(OpNum) +
+                           " without DW_OP_convert support");
+      if (OpNum == dwarf::DW_OP_LLVM_label) {
+        Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
+        break;
+      }
+      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
+                                            : dwarf::DW_OP_skip);
+      Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
+      emitData2(0);
+      break;
     case dwarf::DW_OP_LLVM_arg:
       if (!InsertArg(Op->getArg(0), ExprCursor)) {
         LocationKind = Unknown;
+        if (HasSymbolicBranches)
+          disableTemporaryBuffer();
         return false;
       }
       break;
@@ -579,7 +623,10 @@ bool DwarfExpression::addExpression(
       setSubRegisterPiece(0, 0);
       // Reset the location description kind.
       LocationKind = Unknown;
-      return true;
+      if (!HasSymbolicBranches)
+        return true;
+      // Keep going so we apply the branch fixups before returning.
+      break;
     }
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext: {
@@ -744,9 +791,12 @@ bool DwarfExpression::addExpression(
       break;
     case dwarf::DW_OP_LLVM_implicit_pointer:
       // Handled in DwarfCompileUnit::emitImplicitPointerLocation for
-      // Loc::Single variables. If we reach here, the variable has a
-      // location list or other unsupported path. Drop the
-      // location rather than crashing.
+      // Loc::Single variables. If we reach here, the variable has a location
+      // list or another unsupported path, so stop emitting the expression.
+      // We buffer expressions with symbolic branches, so disable the buffer
+      // before returning.
+      if (HasSymbolicBranches)
+        disableTemporaryBuffer();
       return false;
     default:
       llvm_unreachable("unhandled opcode found in expression");
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
+          static_cast<int64_t>(Label->Offset) -
+          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+      if (!isInt<16>(Displacement))
+        report_fatal_error(Twine("DWARF expression branch offset ") +
+                           Twine(Displacement) + " is outside [-32768, 32767]");
+
+      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                  static_cast<uint16_t>(Displacement));
+    }
+
+    disableTemporaryBuffer();
+    commitTemporaryBuffer();
+  }
   return true;
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
index 2525938c78d86..f5c37140a1ef9 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
@@ -130,7 +130,11 @@ class DwarfExpression {
   /// Emit a raw unsigned value.
   virtual void emitUnsigned(uint64_t Value) = 0;
 
-  virtual void emitData1(uint8_t Value) = 0;
+  /// Emit the low 1, 2, 4, or 8 bytes of a value in the target byte order.
+  virtual void emitData(uint64_t Value, unsigned Size) = 0;
+
+  void emitData1(uint8_t Value) { emitData(Value, 1); }
+  void emitData2(uint16_t Value) { emitData(Value, 2); }
 
   virtual void emitBaseTypeRef(uint64_t Idx) = 0;
 
@@ -150,6 +154,15 @@ class DwarfExpression {
   /// Commit the data stored in the temporary buffer to the main output.
   virtual void commitTemporaryBuffer() = 0;
 
+  /// Replace a 1-, 2-, 4-, or 8-byte zero placeholder at Offset with the low
+  /// bytes of Value in the target byte order.
+  virtual void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                          unsigned Size) = 0;
+
+  void replaceTemporaryBufferData2(unsigned Offset, uint16_t Value) {
+    replaceTemporaryBufferData(Offset, Value, 2);
+  }
+
   /// Emit a normalized unsigned constant.
   void emitConstu(uint64_t Value);
 
@@ -285,10 +298,14 @@ class DwarfExpression {
 
   /// Emit all remaining operations in the DIExpressionCursor. The
   /// cursor must not contain any DW_OP_LLVM_arg operations.
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   void addExpression(DIExpressionCursor &&Expr);
 
   /// Emit all remaining operations in the DIExpressionCursor.
   /// DW_OP_LLVM_arg operations are resolved by calling (\p InsertArg).
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   //
   /// \return false if any call to (\p InsertArg) returns false.
   bool addExpression(
@@ -328,13 +345,15 @@ class DebugLocDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
@@ -358,13 +377,15 @@ class DIEDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index 78c0769e49161..c1c1218a87974 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+}
+
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
                                        DwarfCompileUnit &CU, DIELoc &DIE)
     : DwarfExpression(AP.getDwarfVersion(), CU), AP(AP), OutDIE(DIE) {}
@@ -53,8 +74,8 @@ void DIEDwarfExpression::emitUnsigned(uint64_t Value) {
   CU.addUInt(getActiveDIE(), dwarf::DW_FORM_udata, Value);
 }
 
-void DIEDwarfExpression::emitData1(uint8_t Value) {
-  CU.addUInt(getActiveDIE(), dwarf::DW_FORM_data1, Value);
+void DIEDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  CU.addUInt(getActiveDIE(), getDataForm(Size), getDataValue(Value, Size));
 }
 
 void DIEDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
+}
+
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
                                          llvm::Register MachineReg) {
   return MachineReg == TRI.getFrameRegister(*AP.MF);
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 88f7f2f6240e0..518e4c130e34c 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -1750,6 +1751,9 @@ unsigned DIExpression::ExprOperand::getSize() const {
   case dwarf::DW_OP_LLVM_tag_offset:
   case dwarf::DW_OP_LLVM_entry_value:
   case dwarf::DW_OP_LLVM_arg:
+  case dwarf::DW_OP_LLVM_label:
+  case dwarf::DW_OP_LLVM_bra:
+  case dwarf::DW_OP_LLVM_skip:
   case dwarf::DW_OP_regx:
     return 2;
   default:
@@ -1758,12 +1762,81 @@ unsigned DIExpression::ExprOperand::getSize() const {
 }
 
 bool DIExpression::isValid() const {
+  auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
+    auto FirstOp = expr_op_begin();
+    if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
+      ++FirstOp;
+    return EntryValue.get() == FirstOp->get() && EntryValue.getArg(0) == 1;
+  };
+
+  SmallDenseSet<uint64_t, 4> Labels;
+  SmallVector<uint64_t, 4> LabelReferences;
+  bool HasControlFlow = false;
+  bool HasControlFlowConflict = false;
+  bool HasStackValue = false;
+  bool HasFragment = false;
+  bool HasInvalidControlFlowSuffix = false;
+
+  // Collect labels and branch targets before running the checks below, which
+  // may return early.
   for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
     // Check that there's space for the operand.
     if (I->get() + I->getSize() > E->get())
       return false;
 
     uint64_t Op = I->getOp();
+
+    // Only DW_OP_LLVM_fragment may follow DW_OP_stack_value, and nothing may
+    // follow the fragment.
+    HasInvalidControlFlowSuffix |=
+        HasFragment || (HasStackValue && Op != dwarf::DW_OP_LLVM_fragment);
+
+    switch (Op) {
+    case dwarf::DW_OP_LLVM_label:
+      if (!Labels.insert(I->getArg(0)).second)
+        return false;
+      HasControlFlow = true;
+      break;
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      LabelReferences.push_back(I->getArg(0));
+      HasControlFlow = true;
+      break;
+    // We don't know the DW_OP_bra and DW_OP_skip offsets until CodeGen, so
+    // DIExpression uses the symbolic ops.
+    case dwarf::DW_OP_bra:
+    case dwarf::DW_OP_skip:
+      return false;
+    case dwarf::DW_OP_LLVM_tag_offset:
+      if (HasControlFlow)
+        return false;
+      break;
+    // These ops use lowering paths that don't support symbolic branches.
+    case dwarf::DW_OP_LLVM_arg:
+    case dwarf::DW_OP_LLVM_implicit_pointer:
+      HasControlFlowConflict = true;
+      break;
+    case dwarf::DW_OP_LLVM_entry_value:
+      HasControlFlowConflict |= !IsEntryValueValid(*I);
+      break;
+    default:
+      break;
+    }
+
+    if (Op == dwarf::DW_OP_stack_value)
+      HasStackValue = true;
+    else if (Op == dwarf::DW_OP_LLVM_fragment)
+      HasFragment = true;
+  }
+
+  if (HasControlFlow && (HasControlFlowConflict || HasInvalidControlFlowSuffix))
+    return false;
+  for (uint64_t Label : LabelReferences)
+    if (!Labels.contains(Label))
+      return false;
+
+  for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
+    uint64_t Op = I->getOp();
     if ((Op >= dwarf::DW_OP_reg0 && Op <= dwarf::DW_OP_reg31) ||
         (Op >= dwarf::DW_OP_breg0 && Op <= dwarf::DW_OP_breg31))
       return true;
@@ -1784,7 +1857,7 @@ bool DIExpression::isValid() const {
         return false;
       break;
     }
-    case dwarf::DW_OP_swap: {
+    case dwarf::DW_OP_swap:
       // Must be more than one implicit element on the stack.
 
       // FIXME: A better way to implement this would be to add a local variable
@@ -1795,22 +1868,20 @@ bool DIExpression::isValid() const {
       if (getNumElements() == 1)
         return false;
       break;
-    }
-    case dwarf::DW_OP_LLVM_entry_value: {
+    case dwarf::DW_OP_LLVM_entry_value:
       // An entry value operator must appear at the beginning or immediately
       // following `DW_OP_LLVM_arg 0`, and the number of operations it cover can
       // currently only be 1, because we support only entry values of a simple
       // register location. One reason for this is that we currently can't
       // calculate the size of the resulting DWARF block for other expressions.
-      auto FirstOp = expr_op_begin();
-      if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
-        ++FirstOp;
-      return I->get() == FirstOp->get() && I->getArg(0) == 1;
-    }
+      return IsEntryValueValid(*I);
     case dwarf::DW_OP_LLVM_implicit_pointer:
     case dwarf::DW_OP_LLVM_convert:
     case dwarf::DW_OP_LLVM_arg:
     case dwarf::DW_OP_LLVM_tag_offset:
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext:
     case dwarf::DW_OP_constu:
@@ -2327,13 +2398,15 @@ DIExpression *DIExpression::appendToStack(const DIExpression *Expr,
   // has no DW_OP_stack_value.
   //
   // Match .* DW_OP_stack_value (DW_OP_LLVM_fragment A B)?.
-  std::optional<FragmentInfo> FI = Expr->getFragmentInfo();
-  unsigned DropUntilStackValue = FI ? 3 : 0;
-  ArrayRef<uint64_t> ExprOpsBeforeFragment =
-      Expr->getElements().drop_back(DropUntilStackValue);
-  bool NeedsDeref = (Expr->getNumElements() > DropUntilStackValue) &&
-                    (ExprOpsBeforeFragment.back() != dwarf::DW_OP_stack_value);
-  bool NeedsStackValue = NeedsDeref || ExprOpsBeforeFragment.empty();
+  std::optional<uint64_t> LastOp;
+  // A raw operand can have the same value as an opcode, so use expr_ops().
+  // Labels and branches can't follow DW_OP_stack_value, so LastOp is still
+  // DW_OP_stack_value when the expression has a value suffix.
+  for (auto Op : Expr->expr_ops())
+    if (Op.getOp() != dwarf::DW_OP_LLVM_fragment)
+      LastOp = Op.getOp();
+  bool NeedsDeref = LastOp && *LastOp != dwarf::DW_OP_stack_value;
+  bool NeedsStackValue = NeedsDeref || !LastOp;
 
   // Append a DW_OP_deref after Expr's current op list if needed, then append
   // the new ops, and finally ensure that a single DW_OP_stack_value is present.
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..be14be6a43656 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7182,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        });
+    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
+            &DVR, MD, DVR.getExpression(), BB, F);
+  }
+
   if (DVR.isDbgAssign()) {
     CheckDI(isa_and_nonnull<DIAssignID>(DVR.getRawAssignID()),
             "invalid #dbg_assign DIAssignID", &DVR, DVR.getRawAssignID(), BB,
diff --git a/llvm/test/Assembler/di-expression-symbolic-branches.ll b/llvm/test/Assembler/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..475594c62d444
--- /dev/null
+++ b/llvm/test/Assembler/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Labels use IDs in a DIExpression, so make sure the assembler keeps both
+; forward and backward references.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42)
+!0 = !DIExpression(DW_OP_LLVM_label, 0,
+                   DW_OP_LLVM_bra, 42,
+                   DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 42)
diff --git a/llvm/test/Bitcode/di-expression-symbolic-branches.ll b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..82b3c630360fe
--- /dev/null
+++ b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | llvm-as | llvm-dis | FileCheck %s
+
+; Label IDs are uint64_t values, so use the largest one while checking that the
+; opcodes and IDs survive two bitcode round-trips.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 18446744073709551615, DW_OP_LLVM_bra, 7, DW_OP_LLVM_skip, 18446744073709551615, DW_OP_LLVM_label, 7)
+!0 = !DIExpression(DW_OP_LLVM_label, 18446744073709551615,
+                   DW_OP_LLVM_bra, 7,
+                   DW_OP_LLVM_skip, 18446744073709551615,
+                   DW_OP_LLVM_label, 7)
diff --git a/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
new file mode 100644
index 0000000000000..eda9f210cd378
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
@@ -0,0 +1,102 @@
+# RUN: split-file %s %t
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=asm -o - %t/valid.mir | FileCheck %s
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=ENTRY
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=NATIVE
+# RUN: not --crash llc -disable-verify -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t/missing-label.mir 2>&1 | FileCheck %s --check-prefix=MISSING
+
+# This checks the machine debug-expression path:
+#
+# - DWARF 4 expands conversions before we patch a location-list branch, while
+#   DWARF 5 emits DW_OP_convert.
+# - Entry values keep their branches when we emit an inline expression.
+# - With verification disabled, CodeGen still reports a missing label.
+
+#--- valid.mir
+--- |
+  define void @f() !dbg !5 {
+  entry:
+    ret void, !dbg !12
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9, !10}
+  !9 = !DILocalVariable(name: "before", scope: !5, type: !11)
+  !10 = !DILocalVariable(name: "entry", arg: 1, scope: !5, type: !11)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+  !12 = !DILocation(line: 1, scope: !5)
+...
+---
+name: f
+body: |
+  bb.0.entry:
+    $rcx = MOV64ri 1
+    DBG_VALUE $rdi, $noreg, !10, !DIExpression(DW_OP_LLVM_entry_value, 1, DW_OP_lit0, DW_OP_plus, DW_OP_LLVM_bra, 3, DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 3), debug-location !12
+    DBG_VALUE $rcx, 0, !9, !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_LLVM_convert, 64, DW_ATE_signed, DW_OP_plus_uconst, 1, DW_OP_LLVM_skip, 1), debug-location !12
+    $rcx = MOV64ri 2
+    RET64 debug-location !12
+...
+
+# CHECK: .Ldebug_loc0:
+# CHECK: .byte 114{{.*}}# DW_OP_breg2
+# CHECK-NEXT: .byte 0
+# CHECK-NEXT: .byte 18{{.*}}# DW_OP_dup
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 31
+# CHECK-NEXT: .byte 37{{.*}}# DW_OP_shr
+# CHECK-NEXT: .byte 48{{.*}}# DW_OP_lit0
+# CHECK-NEXT: .byte 32{{.*}}# DW_OP_not
+# CHECK-NEXT: .byte 30{{.*}}# DW_OP_mul
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 32
+# CHECK-NEXT: .byte 36{{.*}}# DW_OP_shl
+# CHECK-NEXT: .byte 33{{.*}}# DW_OP_or
+# CHECK-NEXT: .byte 35{{.*}}# DW_OP_plus_uconst
+# CHECK-NEXT: .byte 1
+# CHECK-NEXT: .byte 47{{.*}}# DW_OP_skip
+# CHECK-NEXT: .byte 240
+# CHECK-NEXT: .byte 255
+
+# ENTRY: DW_AT_location (DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_lit0, DW_OP_plus, DW_OP_bra +2, DW_OP_plus_uconst 0x1)
+
+# NATIVE: DW_OP_breg2 RCX+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_plus_uconst 0x1, DW_OP_skip -15
+
+# MISSING: LLVM ERROR: DWARF expression branch to label 2 has no matching label
+
+#--- missing-label.mir
+--- |
+  define void @missing_label() !dbg !5 {
+  entry:
+    ret void, !dbg !10
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "missing_label", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+  !10 = !DILocation(line: 1, scope: !5)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+...
+---
+name: missing_label
+body: |
+  bb.0.entry:
+    DBG_VALUE 0, $noreg, !9, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_bra, 2), debug-location !10
+    RET64 debug-location !10
+...
diff --git a/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..44159954127be
--- /dev/null
+++ b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
@@ -0,0 +1,44 @@
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -filetype=asm -o - %s | FileCheck %s
+
+; We patch branches in two temporary buffers, so make sure both use the PowerPC
+; byte order: a forward skip in a location list and a backward branch in an
+; inline expression.
+
+define void @f(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_bra, 2), !11)
+  #dbg_value(i64 %x, !10,
+             !DIExpression(DW_OP_LLVM_skip, 3, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_label, 3), !11)
+  call void @clobber(), !dbg !11
+  ret void, !dbg !11
+}
+
+declare void @clobber()
+
+; CHECK: .section .debug_loclists
+; CHECK: .byte 47{{.*}}# DW_OP_skip
+; CHECK-NEXT: .byte 0
+; CHECK-NEXT: .byte 2
+; CHECK: .byte 5{{.*}}# DW_AT_location
+; CHECK-NEXT: .byte 48
+; CHECK-NEXT: .byte 40
+; CHECK-NEXT: .short 65533
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "backward", scope: !5, type: !12)
+!10 = !DILocalVariable(name: "value", arg: 1, scope: !5, type: !12)
+!11 = !DILocation(line: 1, column: 1, scope: !5)
+!12 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
new file mode 100644
index 0000000000000..fc7e0246a6ae2
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
@@ -0,0 +1,33 @@
+import sys
+
+# Put 32768 one-byte DW_OP_dup operations between the branch and its label,
+# which gives us the first offset that won't fit.
+DUPLICATE_OPS = "DW_OP_dup, " * 32768
+
+sys.stdout.write(
+    f"""\
+define void @f() !dbg !5 {{
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, {DUPLICATE_OPS}
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}}
+
+!llvm.dbg.cu = !{{!0}}
+!llvm.module.flags = !{{!3}}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{{i32 2, !"Debug Info Version", i32 3}}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{{null}}
+!9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+!10 = !DILocation(line: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+"""
+)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
new file mode 100644
index 0000000000000..5cb9861502585
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
@@ -0,0 +1,96 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/skip.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=SKIP-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/skip.ll 2>&1 | FileCheck %s --check-prefix=SKIP-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/bra.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=BRA-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/bra.ll 2>&1 | FileCheck %s --check-prefix=BRA-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/label.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=LABEL-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/label.ll 2>&1 | FileCheck %s --check-prefix=LABEL-LEGACY
+
+; DWARF 5 emits DW_OP_convert directly, so labels and branches can appear
+; between conversions. With DWARF 4 we may defer one conversion until the next;
+; if a label, branch, or skip splits the pair, CodeGen reports an error.
+
+; SKIP-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_skip +5, DW_OP_convert {{.*}} "DW_ATE_signed_64")
+; SKIP-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_skip without DW_OP_convert support
+
+; BRA-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_dup, DW_OP_bra +5, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; BRA-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_bra without DW_OP_convert support
+
+; LABEL-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; LABEL-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_label without DW_OP_convert support
+
+;--- skip.ll
+define void @skip(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_skip, 1,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "skip", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "skip", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- label.ll
+define void @label(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 3), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "label", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "label", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- bra.ll
+define void @bra(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_dup,
+                           DW_OP_LLVM_bra, 2,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 2), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "bra", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "bra", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
new file mode 100644
index 0000000000000..677032dc9c7a7
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
@@ -0,0 +1,7 @@
+; RUN: %python %S/Inputs/generate-di-expression-symbolic-branch-out-of-range.py > %t.ll
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t.ll 2>&1 | FileCheck %s
+
+; The first positive offset that doesn't fit is 32768, so make sure CodeGen
+; reports it instead of truncating it.
+
+; CHECK: LLVM ERROR: DWARF expression branch offset 32768 is outside [-32768, 32767]
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..163c3c24c6927
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
@@ -0,0 +1,84 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump -v - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s --check-prefix=LEGACY
+
+; Branch offsets come from the emitted bytes, so check:
+;
+; - zero, forward, backward, and cyclic branches;
+; - ops which grow during lowering;
+; - labels on either side of an emitted op; and
+; - the fragment return path and DWARF 4 convert expansion.
+
+define void @f() !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, DW_OP_LLVM_label, 1), !18)
+  #dbg_value(i64 0, !10,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_skip, 2), !18)
+  #dbg_value(i64 0, !11,
+             !DIExpression(DW_OP_LLVM_label, 3, DW_OP_LLVM_bra, 4,
+                           DW_OP_LLVM_skip, 3, DW_OP_LLVM_label, 4), !18)
+  #dbg_value(i64 0, !12,
+             !DIExpression(DW_OP_LLVM_bra, 5, DW_OP_deref_size, 1,
+                           DW_OP_plus_uconst, 128,
+                           DW_OP_LLVM_extract_bits_sext, 4, 4,
+                           DW_OP_LLVM_label, 5), !18)
+  #dbg_value(i64 0, !13,
+             !DIExpression(DW_OP_LLVM_skip, 6,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 6), !18)
+  #dbg_value(i64 0, !14,
+             !DIExpression(DW_OP_LLVM_bra, 7, DW_OP_LLVM_label, 7,
+                           DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 32), !18)
+  #dbg_value(i64 0, !15,
+             !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 8,
+                           DW_OP_LLVM_skip, 8), !18)
+  #dbg_value(i64 0, !16,
+             !DIExpression(DW_OP_LLVM_label, 9, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_skip, 9), !18)
+  ret void, !dbg !18
+}
+
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"zero"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"backward"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +3, DW_OP_skip -6, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"cycle"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +11, DW_OP_deref_size 0x1, DW_OP_plus_uconst 0x80, DW_OP_constu 0x38, DW_OP_shl, DW_OP_constu 0x3c, DW_OP_shra, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"expanded"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip +10, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"convert"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value, DW_OP_piece 0x4)
+; CHECK: DW_AT_name{{.*}}"fragment"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_after_offset"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -5, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_before_offset"
+
+; LEGACY: DW_AT_location (DW_OP_lit0, DW_OP_skip +11, DW_OP_dup, DW_OP_constu 0x1f, DW_OP_shr, DW_OP_lit0, DW_OP_not, DW_OP_mul, DW_OP_constu 0x20, DW_OP_shl, DW_OP_or, DW_OP_stack_value)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !10, !11, !12, !13, !14, !15, !16}
+!9 = !DILocalVariable(name: "zero", scope: !5, type: !19)
+!10 = !DILocalVariable(name: "backward", scope: !5, type: !19)
+!11 = !DILocalVariable(name: "cycle", scope: !5, type: !19)
+!12 = !DILocalVariable(name: "expanded", scope: !5, type: !19)
+!13 = !DILocalVariable(name: "convert", scope: !5, type: !19)
+!14 = !DILocalVariable(name: "fragment", scope: !5, type: !19)
+!15 = !DILocalVariable(name: "label_after_offset", scope: !5, type: !19)
+!16 = !DILocalVariable(name: "label_before_offset", scope: !5, type: !19)
+!18 = !DILocation(line: 1, scope: !5)
+!19 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/Verifier/di-expression-symbolic-branches.ll b/llvm/test/Verifier/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..db0e0dacdf32f
--- /dev/null
+++ b/llvm/test/Verifier/di-expression-symbolic-branches.ll
@@ -0,0 +1,93 @@
+; RUN: split-file %s %t
+; RUN: opt -passes=verify -disable-output %t/valid.ll
+; RUN: not opt -passes=verify -disable-output %t/arity.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/duplicate.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/missing.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw-after-register.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/incompatible.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/location-arg.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/tag-ordering.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/terminal.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+
+; Check the verifier rules separately:
+;
+; - a label needs an ID;
+; - labels are unique and each branch target exists;
+; - raw branches and incompatible ops are rejected; and
+; - tag_offset stays before control flow, which stays before stack_value.
+
+; DIArgList doesn't support symbolic branches, but normal IR loading drops the
+; bad debug info, so opt still succeeds.
+; RUN: opt -passes=verify -disable-output %t/arg-list.ll 2>&1 | FileCheck %s --check-prefix=ARG-LIST
+
+; INVALID: invalid expression
+; ARG-LIST: DIArgList doesn't support symbolic branches
+; ARG-LIST: warning: ignoring invalid debug info
+
+;--- valid.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 0, DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 0)
+
+;--- arity.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label)
+
+;--- duplicate.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_label, 1)
+
+;--- missing.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 1)
+
+;--- raw.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_bra, 0)
+
+;--- raw-after-register.ll
+; A register normally ends validation, but it must not hide a raw branch later
+; in the expression.
+!named = !{!0}
+!0 = !DIExpression(DW_OP_reg0, DW_OP_skip, 0)
+
+;--- incompatible.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_implicit_pointer, DW_OP_LLVM_label, 1)
+
+;--- location-arg.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_skip, 1,
+                   DW_OP_LLVM_label, 1)
+
+;--- tag-ordering.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_tag_offset, 0)
+
+;--- terminal.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_stack_value, DW_OP_LLVM_label, 1)
+
+;--- arg-list.ll
+; The expression is valid by itself, so this diagnostic comes from DIArgList.
+define void @f(i32 %x) !dbg !4 {
+entry:
+  #dbg_value(!DIArgList(i32 %x), !7,
+             !DIExpression(DW_OP_LLVM_skip, 1, DW_OP_LLVM_label, 1), !8)
+  ret void, !dbg !8
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1, type: !5,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!5 = !DISubroutineType(types: !6)
+!6 = !{null}
+!7 = !DILocalVariable(name: "x", scope: !4)
+!8 = !DILocation(line: 1, column: 1, scope: !4)
diff --git a/llvm/unittests/BinaryFormat/DwarfTest.cpp b/llvm/unittests/BinaryFormat/DwarfTest.cpp
index ba7d59182ea53..6ffb5a921510c 100644
--- a/llvm/unittests/BinaryFormat/DwarfTest.cpp
+++ b/llvm/unittests/BinaryFormat/DwarfTest.cpp
@@ -47,6 +47,13 @@ TEST(DwarfTest, getOperationEncoding) {
   // Some valid ops.
   EXPECT_EQ(DW_OP_deref, getOperationEncoding("DW_OP_deref"));
   EXPECT_EQ(DW_OP_bit_piece, getOperationEncoding("DW_OP_bit_piece"));
+  // These are metadata-only ops, but they still need names in both directions.
+  EXPECT_EQ(DW_OP_LLVM_label, getOperationEncoding("DW_OP_LLVM_label"));
+  EXPECT_EQ(DW_OP_LLVM_bra, getOperationEncoding("DW_OP_LLVM_bra"));
+  EXPECT_EQ(DW_OP_LLVM_skip, getOperationEncoding("DW_OP_LLVM_skip"));
+  EXPECT_EQ("DW_OP_LLVM_label", OperationEncodingString(DW_OP_LLVM_label));
+  EXPECT_EQ("DW_OP_LLVM_bra", OperationEncodingString(DW_OP_LLVM_bra));
+  EXPECT_EQ("DW_OP_LLVM_skip", OperationEncodingString(DW_OP_LLVM_skip));
 
   // Invalid ops.
   EXPECT_EQ(0u, getOperationEncoding("DW_OP_otherthings"));
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 65398b5ca2b1f..9be5384f9647b 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -4251,6 +4251,86 @@ TEST_F(DIExpressionTest, Append) {
   EXPECT_EQ(ResExpr, AppendExpr);
 }
 
+TEST_F(DIExpressionTest, SymbolicBranchRewrites) {
+  // A symbolic op has an ID operand, so fold the math on both sides without
+  // consuming it.
+  for (uint64_t Barrier : {dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                           dwarf::DW_OP_LLVM_skip}) {
+    SmallVector<uint64_t> Ops = {dwarf::DW_OP_constu,
+                                 1,
+                                 dwarf::DW_OP_constu,
+                                 2,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 4,
+                                 Barrier,
+                                 7,
+                                 dwarf::DW_OP_constu,
+                                 5,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 6,
+                                 dwarf::DW_OP_constu,
+                                 7,
+                                 dwarf::DW_OP_plus};
+    SmallVector<uint64_t> Expected = {
+        dwarf::DW_OP_constu,      3, dwarf::DW_OP_constu, 4, Barrier, 7,
+        dwarf::DW_OP_plus_uconst, 5, dwarf::DW_OP_constu, 13};
+    if (Barrier != dwarf::DW_OP_LLVM_label) {
+      Ops.append({dwarf::DW_OP_LLVM_label, 7});
+      Expected.append({dwarf::DW_OP_LLVM_label, 7});
+    }
+    EXPECT_EQ(DIExpression::get(Context, Expected),
+              DIExpression::get(Context, Ops)->foldConstantMath());
+  }
+
+  // Adding ops before or after an expression leaves its labels in place.
+  SmallVector<uint64_t> Ops = {dwarf::DW_OP_LLVM_label, 7,
+                               dwarf::DW_OP_plus_uconst, 1};
+  auto *Expr = DIExpression::get(Context, Ops);
+  auto *Prepended = DIExpression::prepend(Expr, DIExpression::DerefBefore, 0);
+  SmallVector<uint64_t> Expected = {dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_label,
+                                    7, dwarf::DW_OP_plus_uconst, 1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+
+  SmallVector<uint64_t> Prefix = {dwarf::DW_OP_LLVM_tag_offset, 3};
+  Prepended = DIExpression::prependOpcodes(Expr, Prefix);
+  Expected = {dwarf::DW_OP_LLVM_tag_offset, 3, dwarf::DW_OP_LLVM_label, 7,
+              dwarf::DW_OP_plus_uconst,     1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+  EXPECT_TRUE(Prepended->isValid());
+
+  SmallVector<uint64_t> AppendOps = {
+      dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+      dwarf::DW_OP_LLVM_convert, 64, dwarf::DW_ATE_signed};
+  auto *Appended = DIExpression::append(Expr, AppendOps);
+  Expected = {dwarf::DW_OP_LLVM_label,
+              7,
+              dwarf::DW_OP_plus_uconst,
+              1,
+              dwarf::DW_OP_LLVM_convert,
+              32,
+              dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert,
+              64,
+              dwarf::DW_ATE_signed};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+
+  // Label IDs are arbitrary uint64_t values, so appendExt must not mistake one
+  // for DW_OP_stack_value.
+  uint64_t LabelID = dwarf::DW_OP_stack_value;
+  Ops = {dwarf::DW_OP_LLVM_label, LabelID};
+  Appended =
+      DIExpression::appendExt(DIExpression::get(Context, Ops), 32, 64, true);
+  Expected = {dwarf::DW_OP_LLVM_label,   LabelID, dwarf::DW_OP_deref,
+              dwarf::DW_OP_LLVM_convert, 32,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert, 64,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_stack_value};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+}
+
 TEST_F(DIExpressionTest, isValid) {
 #define EXPECT_VALID(...)                                                      \
   do {                                                                         \
@@ -4271,6 +4351,7 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_VALID(dwarf::DW_OP_constu, 6, dwarf::DW_OP_plus);
   EXPECT_VALID(dwarf::DW_OP_constu, 5, dwarf::DW_OP_swap);
   EXPECT_VALID(dwarf::DW_OP_deref);
+  EXPECT_VALID(dwarf::DW_OP_stack_value);
   EXPECT_VALID(dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_plus_uconst, 6, dwarf::DW_OP_deref);
   EXPECT_VALID(dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst, 6);
@@ -4279,12 +4360,31 @@ TEST_F(DIExpressionTest, isValid) {
                dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_VALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_entry_value, 1);
+  // A label may be unused, and branches may refer forward or backward.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 2, dwarf::DW_OP_LLVM_label, 2);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 3, dwarf::DW_OP_LLVM_label, 4);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 4, dwarf::DW_OP_LLVM_skip, 5,
+               dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_LLVM_skip, 4);
+  // Label IDs can use the full uint64_t range.
+  EXPECT_VALID(
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max() - 1,
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max(),
+      dwarf::DW_OP_LLVM_skip, std::numeric_limits<uint64_t>::max() - 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_stack_value);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 6, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_tag_offset, 1, dwarf::DW_OP_LLVM_label, 6,
+               dwarf::DW_OP_LLVM_skip, 6, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
 
   // Invalid constructions.
   EXPECT_INVALID(~0u);
   EXPECT_INVALID(dwarf::DW_OP_plus, 0);
   EXPECT_INVALID(dwarf::DW_OP_plus_uconst);
   EXPECT_INVALID(dwarf::DW_OP_swap);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_deref);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3, 7, dwarf::DW_OP_plus_uconst, 3);
@@ -4294,6 +4394,48 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_plus_uconst, 5,
                  dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_LLVM_entry_value, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip, 1);
+  EXPECT_INVALID(dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_implicit_pointer, dwarf::DW_OP_LLVM_label,
+                 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_tag_offset, 0);
+  // Converts and other stack ops can appear around labels and branches.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_skip, 7, dwarf::DW_OP_LLVM_label, 7);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 1, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_convert, 64,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_dup, dwarf::DW_OP_LLVM_bra, 1,
+               dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 0, 32, dwarf::DW_OP_LLVM_label, 1);
+  // reg0 and entry_value normally end validation early, but they must not hide
+  // invalid control flow later in the expression.
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_LLVM_bra, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_label, 1,
+                 dwarf::DW_OP_LLVM_entry_value, 1);
 
 #undef EXPECT_VALID
 #undef EXPECT_INVALID
@@ -4371,6 +4513,19 @@ TEST_F(DIExpressionTest, createFragmentExpression) {
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
+  // Creating a fragment leaves the branches alone and adds the fragment at the
+  // end.
+  SmallVector<uint64_t> ControlFlowOps = {dwarf::DW_OP_LLVM_label, 1,
+                                          dwarf::DW_OP_LLVM_bra,   1,
+                                          dwarf::DW_OP_LLVM_skip,  1};
+  DIExpression *ControlFlowExpr = DIExpression::get(Context, ControlFlowOps);
+  auto Fragment =
+      DIExpression::createFragmentExpression(ControlFlowExpr, 0, 32);
+  ASSERT_TRUE(Fragment.has_value());
+  ControlFlowOps.append({dwarf::DW_OP_LLVM_fragment, 0, 32});
+  EXPECT_EQ(DIExpression::get(Context, ControlFlowOps), *Fragment);
+  EXPECT_TRUE((*Fragment)->isValid());
+
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
new file mode 100644
index 0000000000000..7ba2b0802fa3c
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
@@ -0,0 +1,23 @@
+; RUN: mlir-translate -import-llvm -mlir-print-debuginfo -emit-expensive-warnings %s 2>&1 | FileCheck %s
+
+; Import a forward branch and a backward skip, and make sure MLIR keeps the op
+; order and label IDs.
+
+; CHECK: llvm.intr.dbg.value {{.*}} #llvm.di_expression<[DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0), DW_OP_LLVM_label(42)]> = {{.*}} : i64
+define void @f(i64 %x) !dbg !4 {
+  #dbg_value(i64 %x, !DILocalVariable(scope: !4),
+             !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42,
+                           DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42),
+             !DILocation(scope: !4))
+  ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1,
+                            type: !DISubroutineType(types: !{null}),
+                            spFlags: DISPFlagDefinition, unit: !0)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
new file mode 100644
index 0000000000000..74a549b952327
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Export a forward branch and a backward skip, and make sure LLVM IR keeps the
+// op order and label IDs.
+
+#file = #llvm.di_file<"test.c" in "/">
+#cu = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_C,
+                            file = #file>
+#sp = #llvm.di_subprogram<compileUnit = #cu, scope = #file, name = "f",
+                          subprogramFlags = "Definition",
+                          type = #llvm.di_subroutine_type<types = #llvm.di_null_type>>
+
+// CHECK: #dbg_value(i64 %{{.*}}, !{{.*}}, !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42), !{{.*}})
+llvm.func @f(%arg: i64) {
+  llvm.intr.dbg.value #llvm.di_local_variable<scope = #sp> #llvm.di_expression<[
+    DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0),
+    DW_OP_LLVM_label(42)]> = %arg : i64
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])

>From 57cf90ad67b0e05c411e79518212f90e3faa39bc Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:53:45 -0700
Subject: [PATCH 2/2] [LLDB] Handle symbolic DIExpression opcodes

These opcodes only appear in LLVM metadata, but adding them to the shared DWARF enum makes LLDB's switch incomplete. List them with the other metadata-only opcodes so a -Wswitch build stays clean.
---
 lldb/source/Expression/DWARFExpression.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lldb/source/Expression/DWARFExpression.cpp b/lldb/source/Expression/DWARFExpression.cpp
index 299dbe97d6176..73bdd18d82cae 100644
--- a/lldb/source/Expression/DWARFExpression.cpp
+++ b/lldb/source/Expression/DWARFExpression.cpp
@@ -205,6 +205,9 @@ GetOpcodeDataSize(const DataExtractor &data, const lldb::offset_t data_offset,
   case DW_OP_LLVM_arg:
   case DW_OP_LLVM_extract_bits_sext:
   case DW_OP_LLVM_extract_bits_zext:
+  case DW_OP_LLVM_label:
+  case DW_OP_LLVM_bra:
+  case DW_OP_LLVM_skip:
     break;
   // Vendor extensions:
   case DW_OP_HP_is_value:


From llvmlistbot at llvm.org  Tue Jul 21 12:59:01 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 12:59:01 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5fcf85.bd93e731.2ac69c.9526@mx.google.com>


https://github.com/echristo updated https://github.com/llvm/llvm-project/pull/210850

>From ab4cecf7ddb25e85c1a7a8bbfc38743dc2125ac2 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Wed, 15 Jul 2026 08:51:28 -0700
Subject: [PATCH 1/3] [DebugInfo] Add symbolic branches to DIExpression

Added three pseudo-ops to DIExpression:

- DW_OP_LLVM_label declares a label and emits no bytes.
- DW_OP_LLVM_bra branches to a label when the top of the stack is non-zero.
- DW_OP_LLVM_skip always branches to a label.

DW_OP_bra and DW_OP_skip store a two-byte offset in [-32768, 32767], but
we don't know those offsets until we emit the expression. DIExpression keeps
label IDs, and we fix them up late in CodeGen.

CodeGen records each label's byte offset and leaves a zero placeholder for each
branch. Once it has seen the whole expression, it patches the placeholders in
the existing temporary buffers used for DIE and location-list expressions.
Both paths keep their current encoding and target byte order.

The verifier makes sure every branch has a label in the same expression and
each label ID is declared once. Labels don't need a branch, and branches can go
forward, backward, to themselves, or form cycles. Raw DW_OP_bra and DW_OP_skip
remain invalid in IR because their offsets depend on the final encoding.

Symbolic control flow only works with one location operand. We reject expressions
that combine it with DW_OP_LLVM_arg or DW_OP_LLVM_implicit_pointer, and diagnose
DIArgList records. Existing expression rewrites keep the pseudo-ops and their
IDs in place.

CodeGen reports an error if the final offset is outside [-32768, 32767], or if a
label, branch, or skip splits a deferred conversion. DWARF 5 emits DW_OP_convert
directly, but older versions may defer one conversion until they see the next.

Tests cover IR and MLIR round trips, verifier failures, forward and backward
branches, self-branches and cycles, both byte orders, conversion boundaries,
and out-of-range offsets.
---
 llvm/docs/LangRef.md                          |  16 ++
 llvm/docs/ReleaseNotes.md                     |   6 +
 llvm/docs/SourceLevelDebugging.md             |  61 ++++++-
 llvm/include/llvm/BinaryFormat/Dwarf.h        |   5 +
 llvm/lib/BinaryFormat/Dwarf.cpp               |   9 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  27 ++-
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  83 +++++++++-
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h |  27 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     |  52 +++++-
 llvm/lib/IR/DebugInfoMetadata.cpp             | 103 ++++++++++--
 llvm/lib/IR/Verifier.cpp                      |  13 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../X86/di-expression-symbolic-branches.mir   | 102 ++++++++++++
 .../di-expression-symbolic-branches.ll        |  44 +++++
 ...expression-symbolic-branch-out-of-range.py |  33 ++++
 ...ession-symbolic-branch-convert-boundary.ll |  96 +++++++++++
 ...expression-symbolic-branch-out-of-range.ll |   7 +
 .../X86/di-expression-symbolic-branches.ll    |  84 ++++++++++
 .../di-expression-symbolic-branches.ll        |  93 +++++++++++
 llvm/unittests/BinaryFormat/DwarfTest.cpp     |   7 +
 llvm/unittests/IR/MetadataTest.cpp            | 155 ++++++++++++++++++
 .../Import/debug-info-symbolic-branches.ll    |  23 +++
 .../llvmir-debug-symbolic-branches.mlir       |  19 +++
 24 files changed, 1054 insertions(+), 35 deletions(-)
 create mode 100644 llvm/test/Assembler/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Bitcode/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
 create mode 100644 llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Verifier/di-expression-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..05162290f330a 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7159,6 +7159,22 @@ Some examples of expressions:
 !DIExpression(DW_OP_constu, 42, DW_OP_stack_value)
 ```
 
+`DIExpression` uses three pseudo-ops for local control flow:
+
+```text
+DW_OP_LLVM_label, <label-id>
+DW_OP_LLVM_bra,   <label-id>
+DW_OP_LLVM_skip,  <label-id>
+```
+
+`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
+branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
+branches. Label IDs are local to the expression. Every branch needs a matching
+label, labels without branches are valid, and each ID can only be declared
+once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
+
+See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+
 ##### DIAssignID
 
 `DIAssignID` nodes have no operands and are always distinct. They are used to
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 94208f8e36d19..988740a7e5e39 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -109,6 +109,12 @@ Makes programs 10x faster by doing Special New Thing.
 
 ### Changes to the Debug Info
 
+* Added `DW_OP_LLVM_label`, `DW_OP_LLVM_bra`, and `DW_OP_LLVM_skip` for symbolic
+  branches in `DIExpression`. These operations use label IDs that CodeGen
+  resolves when it emits the expression. `DW_OP_LLVM_convert` can appear before
+  or after them, but when CodeGen can't emit `DW_OP_convert`, it reports an
+  error if a deferred conversion reaches a label, branch, or skip.
+
 ### Changes to the LLVM tools
 
 ### Changes to LLDB
diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 74bb8840a56e4..645e441e6fe2a 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -386,10 +386,9 @@ call void @llvm.dbg.assign(
 
 Debug expressions are represented as {ref}`specialized-metadata`.
 
-Debug expressions are interpreted left-to-right: start by pushing the
-value/address operand of the record onto a stack, then repeatedly push and
-evaluate opcodes from the `DIExpression` until the final variable description
-is produced.
+A debug expression starts with the record's value or address operand on the
+stack, then evaluates operations from left to right unless a symbolic branch
+jumps to a label in the same `DIExpression`.
 
 The opcodes available in these expressions are described in
 {ref}`dwarf-opcodes` and {ref}`internal-opcodes`.
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
+
+- Put labels, branches, and skips before `DW_OP_stack_value` and
+  `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
+- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
+  separately, so they can't be used with symbolic control flow.
+- Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
+  applies to every path.
+
+We don't check reachability, termination, or stack state where paths meet.
+
+`DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
+and we don't match conversions on different paths. When CodeGen can't emit
+`DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
+branch, or skip would split the pair, CodeGen reports an error.
+
+Local expression rewrites stop at labels, branches, and skips; they can still
+add operations to either end, but they don't move, remove, or copy labels.
+
+##### Other Internal Opcodes
+
 - `DW_OP_LLVM_fragment, <offset>, <size>` may appear at most once in an
   expression, and must be the last opcode. It specifies the bit offset and bit
   size of the variable fragment being described by the record or intrinsic
@@ -482,11 +525,15 @@ them.
 - `DW_OP_LLVM_convert, 16, DW_ATE_signed` specifies a bit size and encoding
   (`16` and `DW_ATE_signed` here, respectively) to which the top of the
   expression stack is to be converted. Maps into a `DW_OP_convert` operation
-  that references a base type constructed from the supplied values.
+  that references a base type constructed from the supplied values. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for how conversions work
+  with labels, branches, and skips.
 - `DW_OP_LLVM_tag_offset, tag_offset` specifies that a memory tag should be
   optionally applied to the pointer. The memory tag is derived from the given
-  tag offset in an implementation-defined manner. (This does not affect the
-  semantics of the expression containing it.)
+  tag offset in an implementation-defined manner. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for its placement with
+  labels, branches, and skips.
+  (This does not affect the semantics of the expression containing it.)
 - `DW_OP_LLVM_entry_value, N` evaluates a sub-expression as-if it were
   evaluated upon entry to the current call frame.
 
diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 75eb3a163e06f..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -149,6 +149,11 @@ enum LocationAtom {
   DW_OP_LLVM_arg = 0x1005,               ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_sext = 0x1006, ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_zext = 0x1007, ///< Only used in LLVM metadata.
+  // Labels declare IDs local to each expression, and branches use them as
+  // targets.
+  DW_OP_LLVM_label = 0x1008, ///< Only used in LLVM metadata.
+  DW_OP_LLVM_bra = 0x1009,   ///< Only used in LLVM metadata.
+  DW_OP_LLVM_skip = 0x100a,  ///< Only used in LLVM metadata.
 };
 
 enum LlvmUserLocationAtom {
diff --git a/llvm/lib/BinaryFormat/Dwarf.cpp b/llvm/lib/BinaryFormat/Dwarf.cpp
index 9ede7ebbe20ad..431d01c953768 100644
--- a/llvm/lib/BinaryFormat/Dwarf.cpp
+++ b/llvm/lib/BinaryFormat/Dwarf.cpp
@@ -159,6 +159,12 @@ StringRef llvm::dwarf::OperationEncodingString(unsigned Encoding) {
     return "DW_OP_LLVM_extract_bits_sext";
   case DW_OP_LLVM_extract_bits_zext:
     return "DW_OP_LLVM_extract_bits_zext";
+  case DW_OP_LLVM_label:
+    return "DW_OP_LLVM_label";
+  case DW_OP_LLVM_bra:
+    return "DW_OP_LLVM_bra";
+  case DW_OP_LLVM_skip:
+    return "DW_OP_LLVM_skip";
   }
 }
 
@@ -175,6 +181,9 @@ unsigned llvm::dwarf::getOperationEncoding(StringRef OperationEncodingString) {
       .Case("DW_OP_LLVM_arg", DW_OP_LLVM_arg)
       .Case("DW_OP_LLVM_extract_bits_sext", DW_OP_LLVM_extract_bits_sext)
       .Case("DW_OP_LLVM_extract_bits_zext", DW_OP_LLVM_extract_bits_zext)
+      .Case("DW_OP_LLVM_label", DW_OP_LLVM_label)
+      .Case("DW_OP_LLVM_bra", DW_OP_LLVM_bra)
+      .Case("DW_OP_LLVM_skip", DW_OP_LLVM_skip)
       .Default(0);
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfbb35d516534..45f1f13e999fc 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -191,8 +191,14 @@ void DebugLocDwarfExpression::emitUnsigned(uint64_t Value) {
   getActiveStreamer().emitULEB128(Value, Twine(Value));
 }
 
-void DebugLocDwarfExpression::emitData1(uint8_t Value) {
-  getActiveStreamer().emitInt8(Value, Twine(Value));
+void DebugLocDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    getActiveStreamer().emitInt8(Value >> (Byte * 8), Twine(Value));
+  }
 }
 
 void DebugLocDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -232,6 +238,23 @@ void DebugLocDwarfExpression::commitTemporaryBuffer() {
   TmpBuf->Comments.clear();
 }
 
+void DebugLocDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                         uint64_t Value,
+                                                         unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  assert(TmpBuf && Offset < TmpBuf->Bytes.size() &&
+         Size <= TmpBuf->Bytes.size() - Offset &&
+         "invalid temporary buffer offset");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    TmpBuf->Bytes[Offset + I] = Value >> (Byte * 8);
+    if (Offset + I < TmpBuf->Comments.size())
+      TmpBuf->Comments[Offset + I].clear();
+  }
+}
+
 const DIType *DbgVariable::getType() const {
   return getVariable()->getType();
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 9b676f88a0cff..4054817abb95d 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -13,13 +13,16 @@
 #include "DwarfExpression.h"
 #include "DwarfCompileUnit.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallBitVector.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/CodeGen/Register.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
 #include <algorithm>
 
 using namespace llvm;
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+      });
+
+  SmallVector<LabelOffset, 4> Labels;
+  SmallVector<BranchFixup, 4> Fixups;
+  // Buffer the expression until every label has a byte offset, then patch the
+  // branches.
+  if (HasSymbolicBranches)
+    enableTemporaryBuffer();
 
+  std::optional<DIExpression::ExprOperand> PrevConvertOp;
   while (ExprCursor) {
     auto Op = ExprCursor.take();
     uint64_t OpNum = Op->getOp();
@@ -545,9 +571,27 @@ bool DwarfExpression::addExpression(
     }
 
     switch (OpNum) {
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      if (PrevConvertOp)
+        report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
+                           dwarf::OperationEncodingString(OpNum) +
+                           " without DW_OP_convert support");
+      if (OpNum == dwarf::DW_OP_LLVM_label) {
+        Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
+        break;
+      }
+      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
+                                            : dwarf::DW_OP_skip);
+      Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
+      emitData2(0);
+      break;
     case dwarf::DW_OP_LLVM_arg:
       if (!InsertArg(Op->getArg(0), ExprCursor)) {
         LocationKind = Unknown;
+        if (HasSymbolicBranches)
+          disableTemporaryBuffer();
         return false;
       }
       break;
@@ -579,7 +623,10 @@ bool DwarfExpression::addExpression(
       setSubRegisterPiece(0, 0);
       // Reset the location description kind.
       LocationKind = Unknown;
-      return true;
+      if (!HasSymbolicBranches)
+        return true;
+      // Keep going so we apply the branch fixups before returning.
+      break;
     }
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext: {
@@ -744,9 +791,12 @@ bool DwarfExpression::addExpression(
       break;
     case dwarf::DW_OP_LLVM_implicit_pointer:
       // Handled in DwarfCompileUnit::emitImplicitPointerLocation for
-      // Loc::Single variables. If we reach here, the variable has a
-      // location list or other unsupported path. Drop the
-      // location rather than crashing.
+      // Loc::Single variables. If we reach here, the variable has a location
+      // list or another unsupported path, so stop emitting the expression.
+      // We buffer expressions with symbolic branches, so disable the buffer
+      // before returning.
+      if (HasSymbolicBranches)
+        disableTemporaryBuffer();
       return false;
     default:
       llvm_unreachable("unhandled opcode found in expression");
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
+          static_cast<int64_t>(Label->Offset) -
+          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+      if (!isInt<16>(Displacement))
+        report_fatal_error(Twine("DWARF expression branch offset ") +
+                           Twine(Displacement) + " is outside [-32768, 32767]");
+
+      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                  static_cast<uint16_t>(Displacement));
+    }
+
+    disableTemporaryBuffer();
+    commitTemporaryBuffer();
+  }
   return true;
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
index 2525938c78d86..f5c37140a1ef9 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
@@ -130,7 +130,11 @@ class DwarfExpression {
   /// Emit a raw unsigned value.
   virtual void emitUnsigned(uint64_t Value) = 0;
 
-  virtual void emitData1(uint8_t Value) = 0;
+  /// Emit the low 1, 2, 4, or 8 bytes of a value in the target byte order.
+  virtual void emitData(uint64_t Value, unsigned Size) = 0;
+
+  void emitData1(uint8_t Value) { emitData(Value, 1); }
+  void emitData2(uint16_t Value) { emitData(Value, 2); }
 
   virtual void emitBaseTypeRef(uint64_t Idx) = 0;
 
@@ -150,6 +154,15 @@ class DwarfExpression {
   /// Commit the data stored in the temporary buffer to the main output.
   virtual void commitTemporaryBuffer() = 0;
 
+  /// Replace a 1-, 2-, 4-, or 8-byte zero placeholder at Offset with the low
+  /// bytes of Value in the target byte order.
+  virtual void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                          unsigned Size) = 0;
+
+  void replaceTemporaryBufferData2(unsigned Offset, uint16_t Value) {
+    replaceTemporaryBufferData(Offset, Value, 2);
+  }
+
   /// Emit a normalized unsigned constant.
   void emitConstu(uint64_t Value);
 
@@ -285,10 +298,14 @@ class DwarfExpression {
 
   /// Emit all remaining operations in the DIExpressionCursor. The
   /// cursor must not contain any DW_OP_LLVM_arg operations.
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   void addExpression(DIExpressionCursor &&Expr);
 
   /// Emit all remaining operations in the DIExpressionCursor.
   /// DW_OP_LLVM_arg operations are resolved by calling (\p InsertArg).
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   //
   /// \return false if any call to (\p InsertArg) returns false.
   bool addExpression(
@@ -328,13 +345,15 @@ class DebugLocDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
@@ -358,13 +377,15 @@ class DIEDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index 78c0769e49161..c1c1218a87974 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+}
+
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
                                        DwarfCompileUnit &CU, DIELoc &DIE)
     : DwarfExpression(AP.getDwarfVersion(), CU), AP(AP), OutDIE(DIE) {}
@@ -53,8 +74,8 @@ void DIEDwarfExpression::emitUnsigned(uint64_t Value) {
   CU.addUInt(getActiveDIE(), dwarf::DW_FORM_udata, Value);
 }
 
-void DIEDwarfExpression::emitData1(uint8_t Value) {
-  CU.addUInt(getActiveDIE(), dwarf::DW_FORM_data1, Value);
+void DIEDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  CU.addUInt(getActiveDIE(), getDataForm(Size), getDataValue(Value, Size));
 }
 
 void DIEDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
+}
+
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
                                          llvm::Register MachineReg) {
   return MachineReg == TRI.getFrameRegister(*AP.MF);
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 88f7f2f6240e0..518e4c130e34c 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -1750,6 +1751,9 @@ unsigned DIExpression::ExprOperand::getSize() const {
   case dwarf::DW_OP_LLVM_tag_offset:
   case dwarf::DW_OP_LLVM_entry_value:
   case dwarf::DW_OP_LLVM_arg:
+  case dwarf::DW_OP_LLVM_label:
+  case dwarf::DW_OP_LLVM_bra:
+  case dwarf::DW_OP_LLVM_skip:
   case dwarf::DW_OP_regx:
     return 2;
   default:
@@ -1758,12 +1762,81 @@ unsigned DIExpression::ExprOperand::getSize() const {
 }
 
 bool DIExpression::isValid() const {
+  auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
+    auto FirstOp = expr_op_begin();
+    if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
+      ++FirstOp;
+    return EntryValue.get() == FirstOp->get() && EntryValue.getArg(0) == 1;
+  };
+
+  SmallDenseSet<uint64_t, 4> Labels;
+  SmallVector<uint64_t, 4> LabelReferences;
+  bool HasControlFlow = false;
+  bool HasControlFlowConflict = false;
+  bool HasStackValue = false;
+  bool HasFragment = false;
+  bool HasInvalidControlFlowSuffix = false;
+
+  // Collect labels and branch targets before running the checks below, which
+  // may return early.
   for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
     // Check that there's space for the operand.
     if (I->get() + I->getSize() > E->get())
       return false;
 
     uint64_t Op = I->getOp();
+
+    // Only DW_OP_LLVM_fragment may follow DW_OP_stack_value, and nothing may
+    // follow the fragment.
+    HasInvalidControlFlowSuffix |=
+        HasFragment || (HasStackValue && Op != dwarf::DW_OP_LLVM_fragment);
+
+    switch (Op) {
+    case dwarf::DW_OP_LLVM_label:
+      if (!Labels.insert(I->getArg(0)).second)
+        return false;
+      HasControlFlow = true;
+      break;
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      LabelReferences.push_back(I->getArg(0));
+      HasControlFlow = true;
+      break;
+    // We don't know the DW_OP_bra and DW_OP_skip offsets until CodeGen, so
+    // DIExpression uses the symbolic ops.
+    case dwarf::DW_OP_bra:
+    case dwarf::DW_OP_skip:
+      return false;
+    case dwarf::DW_OP_LLVM_tag_offset:
+      if (HasControlFlow)
+        return false;
+      break;
+    // These ops use lowering paths that don't support symbolic branches.
+    case dwarf::DW_OP_LLVM_arg:
+    case dwarf::DW_OP_LLVM_implicit_pointer:
+      HasControlFlowConflict = true;
+      break;
+    case dwarf::DW_OP_LLVM_entry_value:
+      HasControlFlowConflict |= !IsEntryValueValid(*I);
+      break;
+    default:
+      break;
+    }
+
+    if (Op == dwarf::DW_OP_stack_value)
+      HasStackValue = true;
+    else if (Op == dwarf::DW_OP_LLVM_fragment)
+      HasFragment = true;
+  }
+
+  if (HasControlFlow && (HasControlFlowConflict || HasInvalidControlFlowSuffix))
+    return false;
+  for (uint64_t Label : LabelReferences)
+    if (!Labels.contains(Label))
+      return false;
+
+  for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
+    uint64_t Op = I->getOp();
     if ((Op >= dwarf::DW_OP_reg0 && Op <= dwarf::DW_OP_reg31) ||
         (Op >= dwarf::DW_OP_breg0 && Op <= dwarf::DW_OP_breg31))
       return true;
@@ -1784,7 +1857,7 @@ bool DIExpression::isValid() const {
         return false;
       break;
     }
-    case dwarf::DW_OP_swap: {
+    case dwarf::DW_OP_swap:
       // Must be more than one implicit element on the stack.
 
       // FIXME: A better way to implement this would be to add a local variable
@@ -1795,22 +1868,20 @@ bool DIExpression::isValid() const {
       if (getNumElements() == 1)
         return false;
       break;
-    }
-    case dwarf::DW_OP_LLVM_entry_value: {
+    case dwarf::DW_OP_LLVM_entry_value:
       // An entry value operator must appear at the beginning or immediately
       // following `DW_OP_LLVM_arg 0`, and the number of operations it cover can
       // currently only be 1, because we support only entry values of a simple
       // register location. One reason for this is that we currently can't
       // calculate the size of the resulting DWARF block for other expressions.
-      auto FirstOp = expr_op_begin();
-      if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
-        ++FirstOp;
-      return I->get() == FirstOp->get() && I->getArg(0) == 1;
-    }
+      return IsEntryValueValid(*I);
     case dwarf::DW_OP_LLVM_implicit_pointer:
     case dwarf::DW_OP_LLVM_convert:
     case dwarf::DW_OP_LLVM_arg:
     case dwarf::DW_OP_LLVM_tag_offset:
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext:
     case dwarf::DW_OP_constu:
@@ -2327,13 +2398,15 @@ DIExpression *DIExpression::appendToStack(const DIExpression *Expr,
   // has no DW_OP_stack_value.
   //
   // Match .* DW_OP_stack_value (DW_OP_LLVM_fragment A B)?.
-  std::optional<FragmentInfo> FI = Expr->getFragmentInfo();
-  unsigned DropUntilStackValue = FI ? 3 : 0;
-  ArrayRef<uint64_t> ExprOpsBeforeFragment =
-      Expr->getElements().drop_back(DropUntilStackValue);
-  bool NeedsDeref = (Expr->getNumElements() > DropUntilStackValue) &&
-                    (ExprOpsBeforeFragment.back() != dwarf::DW_OP_stack_value);
-  bool NeedsStackValue = NeedsDeref || ExprOpsBeforeFragment.empty();
+  std::optional<uint64_t> LastOp;
+  // A raw operand can have the same value as an opcode, so use expr_ops().
+  // Labels and branches can't follow DW_OP_stack_value, so LastOp is still
+  // DW_OP_stack_value when the expression has a value suffix.
+  for (auto Op : Expr->expr_ops())
+    if (Op.getOp() != dwarf::DW_OP_LLVM_fragment)
+      LastOp = Op.getOp();
+  bool NeedsDeref = LastOp && *LastOp != dwarf::DW_OP_stack_value;
+  bool NeedsStackValue = NeedsDeref || !LastOp;
 
   // Append a DW_OP_deref after Expr's current op list if needed, then append
   // the new ops, and finally ensure that a single DW_OP_stack_value is present.
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..be14be6a43656 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7182,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        });
+    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
+            &DVR, MD, DVR.getExpression(), BB, F);
+  }
+
   if (DVR.isDbgAssign()) {
     CheckDI(isa_and_nonnull<DIAssignID>(DVR.getRawAssignID()),
             "invalid #dbg_assign DIAssignID", &DVR, DVR.getRawAssignID(), BB,
diff --git a/llvm/test/Assembler/di-expression-symbolic-branches.ll b/llvm/test/Assembler/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..475594c62d444
--- /dev/null
+++ b/llvm/test/Assembler/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Labels use IDs in a DIExpression, so make sure the assembler keeps both
+; forward and backward references.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42)
+!0 = !DIExpression(DW_OP_LLVM_label, 0,
+                   DW_OP_LLVM_bra, 42,
+                   DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 42)
diff --git a/llvm/test/Bitcode/di-expression-symbolic-branches.ll b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..82b3c630360fe
--- /dev/null
+++ b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | llvm-as | llvm-dis | FileCheck %s
+
+; Label IDs are uint64_t values, so use the largest one while checking that the
+; opcodes and IDs survive two bitcode round-trips.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 18446744073709551615, DW_OP_LLVM_bra, 7, DW_OP_LLVM_skip, 18446744073709551615, DW_OP_LLVM_label, 7)
+!0 = !DIExpression(DW_OP_LLVM_label, 18446744073709551615,
+                   DW_OP_LLVM_bra, 7,
+                   DW_OP_LLVM_skip, 18446744073709551615,
+                   DW_OP_LLVM_label, 7)
diff --git a/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
new file mode 100644
index 0000000000000..eda9f210cd378
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
@@ -0,0 +1,102 @@
+# RUN: split-file %s %t
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=asm -o - %t/valid.mir | FileCheck %s
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=ENTRY
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=NATIVE
+# RUN: not --crash llc -disable-verify -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t/missing-label.mir 2>&1 | FileCheck %s --check-prefix=MISSING
+
+# This checks the machine debug-expression path:
+#
+# - DWARF 4 expands conversions before we patch a location-list branch, while
+#   DWARF 5 emits DW_OP_convert.
+# - Entry values keep their branches when we emit an inline expression.
+# - With verification disabled, CodeGen still reports a missing label.
+
+#--- valid.mir
+--- |
+  define void @f() !dbg !5 {
+  entry:
+    ret void, !dbg !12
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9, !10}
+  !9 = !DILocalVariable(name: "before", scope: !5, type: !11)
+  !10 = !DILocalVariable(name: "entry", arg: 1, scope: !5, type: !11)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+  !12 = !DILocation(line: 1, scope: !5)
+...
+---
+name: f
+body: |
+  bb.0.entry:
+    $rcx = MOV64ri 1
+    DBG_VALUE $rdi, $noreg, !10, !DIExpression(DW_OP_LLVM_entry_value, 1, DW_OP_lit0, DW_OP_plus, DW_OP_LLVM_bra, 3, DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 3), debug-location !12
+    DBG_VALUE $rcx, 0, !9, !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_LLVM_convert, 64, DW_ATE_signed, DW_OP_plus_uconst, 1, DW_OP_LLVM_skip, 1), debug-location !12
+    $rcx = MOV64ri 2
+    RET64 debug-location !12
+...
+
+# CHECK: .Ldebug_loc0:
+# CHECK: .byte 114{{.*}}# DW_OP_breg2
+# CHECK-NEXT: .byte 0
+# CHECK-NEXT: .byte 18{{.*}}# DW_OP_dup
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 31
+# CHECK-NEXT: .byte 37{{.*}}# DW_OP_shr
+# CHECK-NEXT: .byte 48{{.*}}# DW_OP_lit0
+# CHECK-NEXT: .byte 32{{.*}}# DW_OP_not
+# CHECK-NEXT: .byte 30{{.*}}# DW_OP_mul
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 32
+# CHECK-NEXT: .byte 36{{.*}}# DW_OP_shl
+# CHECK-NEXT: .byte 33{{.*}}# DW_OP_or
+# CHECK-NEXT: .byte 35{{.*}}# DW_OP_plus_uconst
+# CHECK-NEXT: .byte 1
+# CHECK-NEXT: .byte 47{{.*}}# DW_OP_skip
+# CHECK-NEXT: .byte 240
+# CHECK-NEXT: .byte 255
+
+# ENTRY: DW_AT_location (DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_lit0, DW_OP_plus, DW_OP_bra +2, DW_OP_plus_uconst 0x1)
+
+# NATIVE: DW_OP_breg2 RCX+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_plus_uconst 0x1, DW_OP_skip -15
+
+# MISSING: LLVM ERROR: DWARF expression branch to label 2 has no matching label
+
+#--- missing-label.mir
+--- |
+  define void @missing_label() !dbg !5 {
+  entry:
+    ret void, !dbg !10
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "missing_label", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+  !10 = !DILocation(line: 1, scope: !5)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+...
+---
+name: missing_label
+body: |
+  bb.0.entry:
+    DBG_VALUE 0, $noreg, !9, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_bra, 2), debug-location !10
+    RET64 debug-location !10
+...
diff --git a/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..44159954127be
--- /dev/null
+++ b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
@@ -0,0 +1,44 @@
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -filetype=asm -o - %s | FileCheck %s
+
+; We patch branches in two temporary buffers, so make sure both use the PowerPC
+; byte order: a forward skip in a location list and a backward branch in an
+; inline expression.
+
+define void @f(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_bra, 2), !11)
+  #dbg_value(i64 %x, !10,
+             !DIExpression(DW_OP_LLVM_skip, 3, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_label, 3), !11)
+  call void @clobber(), !dbg !11
+  ret void, !dbg !11
+}
+
+declare void @clobber()
+
+; CHECK: .section .debug_loclists
+; CHECK: .byte 47{{.*}}# DW_OP_skip
+; CHECK-NEXT: .byte 0
+; CHECK-NEXT: .byte 2
+; CHECK: .byte 5{{.*}}# DW_AT_location
+; CHECK-NEXT: .byte 48
+; CHECK-NEXT: .byte 40
+; CHECK-NEXT: .short 65533
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "backward", scope: !5, type: !12)
+!10 = !DILocalVariable(name: "value", arg: 1, scope: !5, type: !12)
+!11 = !DILocation(line: 1, column: 1, scope: !5)
+!12 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
new file mode 100644
index 0000000000000..fc7e0246a6ae2
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
@@ -0,0 +1,33 @@
+import sys
+
+# Put 32768 one-byte DW_OP_dup operations between the branch and its label,
+# which gives us the first offset that won't fit.
+DUPLICATE_OPS = "DW_OP_dup, " * 32768
+
+sys.stdout.write(
+    f"""\
+define void @f() !dbg !5 {{
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, {DUPLICATE_OPS}
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}}
+
+!llvm.dbg.cu = !{{!0}}
+!llvm.module.flags = !{{!3}}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{{i32 2, !"Debug Info Version", i32 3}}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{{null}}
+!9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+!10 = !DILocation(line: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+"""
+)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
new file mode 100644
index 0000000000000..5cb9861502585
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
@@ -0,0 +1,96 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/skip.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=SKIP-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/skip.ll 2>&1 | FileCheck %s --check-prefix=SKIP-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/bra.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=BRA-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/bra.ll 2>&1 | FileCheck %s --check-prefix=BRA-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/label.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=LABEL-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/label.ll 2>&1 | FileCheck %s --check-prefix=LABEL-LEGACY
+
+; DWARF 5 emits DW_OP_convert directly, so labels and branches can appear
+; between conversions. With DWARF 4 we may defer one conversion until the next;
+; if a label, branch, or skip splits the pair, CodeGen reports an error.
+
+; SKIP-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_skip +5, DW_OP_convert {{.*}} "DW_ATE_signed_64")
+; SKIP-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_skip without DW_OP_convert support
+
+; BRA-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_dup, DW_OP_bra +5, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; BRA-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_bra without DW_OP_convert support
+
+; LABEL-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; LABEL-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_label without DW_OP_convert support
+
+;--- skip.ll
+define void @skip(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_skip, 1,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "skip", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "skip", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- label.ll
+define void @label(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 3), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "label", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "label", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- bra.ll
+define void @bra(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_dup,
+                           DW_OP_LLVM_bra, 2,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 2), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "bra", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "bra", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
new file mode 100644
index 0000000000000..677032dc9c7a7
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
@@ -0,0 +1,7 @@
+; RUN: %python %S/Inputs/generate-di-expression-symbolic-branch-out-of-range.py > %t.ll
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t.ll 2>&1 | FileCheck %s
+
+; The first positive offset that doesn't fit is 32768, so make sure CodeGen
+; reports it instead of truncating it.
+
+; CHECK: LLVM ERROR: DWARF expression branch offset 32768 is outside [-32768, 32767]
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..163c3c24c6927
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
@@ -0,0 +1,84 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump -v - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s --check-prefix=LEGACY
+
+; Branch offsets come from the emitted bytes, so check:
+;
+; - zero, forward, backward, and cyclic branches;
+; - ops which grow during lowering;
+; - labels on either side of an emitted op; and
+; - the fragment return path and DWARF 4 convert expansion.
+
+define void @f() !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, DW_OP_LLVM_label, 1), !18)
+  #dbg_value(i64 0, !10,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_skip, 2), !18)
+  #dbg_value(i64 0, !11,
+             !DIExpression(DW_OP_LLVM_label, 3, DW_OP_LLVM_bra, 4,
+                           DW_OP_LLVM_skip, 3, DW_OP_LLVM_label, 4), !18)
+  #dbg_value(i64 0, !12,
+             !DIExpression(DW_OP_LLVM_bra, 5, DW_OP_deref_size, 1,
+                           DW_OP_plus_uconst, 128,
+                           DW_OP_LLVM_extract_bits_sext, 4, 4,
+                           DW_OP_LLVM_label, 5), !18)
+  #dbg_value(i64 0, !13,
+             !DIExpression(DW_OP_LLVM_skip, 6,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 6), !18)
+  #dbg_value(i64 0, !14,
+             !DIExpression(DW_OP_LLVM_bra, 7, DW_OP_LLVM_label, 7,
+                           DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 32), !18)
+  #dbg_value(i64 0, !15,
+             !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 8,
+                           DW_OP_LLVM_skip, 8), !18)
+  #dbg_value(i64 0, !16,
+             !DIExpression(DW_OP_LLVM_label, 9, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_skip, 9), !18)
+  ret void, !dbg !18
+}
+
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"zero"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"backward"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +3, DW_OP_skip -6, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"cycle"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +11, DW_OP_deref_size 0x1, DW_OP_plus_uconst 0x80, DW_OP_constu 0x38, DW_OP_shl, DW_OP_constu 0x3c, DW_OP_shra, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"expanded"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip +10, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"convert"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value, DW_OP_piece 0x4)
+; CHECK: DW_AT_name{{.*}}"fragment"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_after_offset"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -5, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_before_offset"
+
+; LEGACY: DW_AT_location (DW_OP_lit0, DW_OP_skip +11, DW_OP_dup, DW_OP_constu 0x1f, DW_OP_shr, DW_OP_lit0, DW_OP_not, DW_OP_mul, DW_OP_constu 0x20, DW_OP_shl, DW_OP_or, DW_OP_stack_value)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !10, !11, !12, !13, !14, !15, !16}
+!9 = !DILocalVariable(name: "zero", scope: !5, type: !19)
+!10 = !DILocalVariable(name: "backward", scope: !5, type: !19)
+!11 = !DILocalVariable(name: "cycle", scope: !5, type: !19)
+!12 = !DILocalVariable(name: "expanded", scope: !5, type: !19)
+!13 = !DILocalVariable(name: "convert", scope: !5, type: !19)
+!14 = !DILocalVariable(name: "fragment", scope: !5, type: !19)
+!15 = !DILocalVariable(name: "label_after_offset", scope: !5, type: !19)
+!16 = !DILocalVariable(name: "label_before_offset", scope: !5, type: !19)
+!18 = !DILocation(line: 1, scope: !5)
+!19 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/Verifier/di-expression-symbolic-branches.ll b/llvm/test/Verifier/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..db0e0dacdf32f
--- /dev/null
+++ b/llvm/test/Verifier/di-expression-symbolic-branches.ll
@@ -0,0 +1,93 @@
+; RUN: split-file %s %t
+; RUN: opt -passes=verify -disable-output %t/valid.ll
+; RUN: not opt -passes=verify -disable-output %t/arity.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/duplicate.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/missing.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw-after-register.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/incompatible.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/location-arg.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/tag-ordering.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/terminal.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+
+; Check the verifier rules separately:
+;
+; - a label needs an ID;
+; - labels are unique and each branch target exists;
+; - raw branches and incompatible ops are rejected; and
+; - tag_offset stays before control flow, which stays before stack_value.
+
+; DIArgList doesn't support symbolic branches, but normal IR loading drops the
+; bad debug info, so opt still succeeds.
+; RUN: opt -passes=verify -disable-output %t/arg-list.ll 2>&1 | FileCheck %s --check-prefix=ARG-LIST
+
+; INVALID: invalid expression
+; ARG-LIST: DIArgList doesn't support symbolic branches
+; ARG-LIST: warning: ignoring invalid debug info
+
+;--- valid.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 0, DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 0)
+
+;--- arity.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label)
+
+;--- duplicate.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_label, 1)
+
+;--- missing.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 1)
+
+;--- raw.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_bra, 0)
+
+;--- raw-after-register.ll
+; A register normally ends validation, but it must not hide a raw branch later
+; in the expression.
+!named = !{!0}
+!0 = !DIExpression(DW_OP_reg0, DW_OP_skip, 0)
+
+;--- incompatible.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_implicit_pointer, DW_OP_LLVM_label, 1)
+
+;--- location-arg.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_skip, 1,
+                   DW_OP_LLVM_label, 1)
+
+;--- tag-ordering.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_tag_offset, 0)
+
+;--- terminal.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_stack_value, DW_OP_LLVM_label, 1)
+
+;--- arg-list.ll
+; The expression is valid by itself, so this diagnostic comes from DIArgList.
+define void @f(i32 %x) !dbg !4 {
+entry:
+  #dbg_value(!DIArgList(i32 %x), !7,
+             !DIExpression(DW_OP_LLVM_skip, 1, DW_OP_LLVM_label, 1), !8)
+  ret void, !dbg !8
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1, type: !5,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!5 = !DISubroutineType(types: !6)
+!6 = !{null}
+!7 = !DILocalVariable(name: "x", scope: !4)
+!8 = !DILocation(line: 1, column: 1, scope: !4)
diff --git a/llvm/unittests/BinaryFormat/DwarfTest.cpp b/llvm/unittests/BinaryFormat/DwarfTest.cpp
index ba7d59182ea53..6ffb5a921510c 100644
--- a/llvm/unittests/BinaryFormat/DwarfTest.cpp
+++ b/llvm/unittests/BinaryFormat/DwarfTest.cpp
@@ -47,6 +47,13 @@ TEST(DwarfTest, getOperationEncoding) {
   // Some valid ops.
   EXPECT_EQ(DW_OP_deref, getOperationEncoding("DW_OP_deref"));
   EXPECT_EQ(DW_OP_bit_piece, getOperationEncoding("DW_OP_bit_piece"));
+  // These are metadata-only ops, but they still need names in both directions.
+  EXPECT_EQ(DW_OP_LLVM_label, getOperationEncoding("DW_OP_LLVM_label"));
+  EXPECT_EQ(DW_OP_LLVM_bra, getOperationEncoding("DW_OP_LLVM_bra"));
+  EXPECT_EQ(DW_OP_LLVM_skip, getOperationEncoding("DW_OP_LLVM_skip"));
+  EXPECT_EQ("DW_OP_LLVM_label", OperationEncodingString(DW_OP_LLVM_label));
+  EXPECT_EQ("DW_OP_LLVM_bra", OperationEncodingString(DW_OP_LLVM_bra));
+  EXPECT_EQ("DW_OP_LLVM_skip", OperationEncodingString(DW_OP_LLVM_skip));
 
   // Invalid ops.
   EXPECT_EQ(0u, getOperationEncoding("DW_OP_otherthings"));
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 65398b5ca2b1f..9be5384f9647b 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -4251,6 +4251,86 @@ TEST_F(DIExpressionTest, Append) {
   EXPECT_EQ(ResExpr, AppendExpr);
 }
 
+TEST_F(DIExpressionTest, SymbolicBranchRewrites) {
+  // A symbolic op has an ID operand, so fold the math on both sides without
+  // consuming it.
+  for (uint64_t Barrier : {dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                           dwarf::DW_OP_LLVM_skip}) {
+    SmallVector<uint64_t> Ops = {dwarf::DW_OP_constu,
+                                 1,
+                                 dwarf::DW_OP_constu,
+                                 2,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 4,
+                                 Barrier,
+                                 7,
+                                 dwarf::DW_OP_constu,
+                                 5,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 6,
+                                 dwarf::DW_OP_constu,
+                                 7,
+                                 dwarf::DW_OP_plus};
+    SmallVector<uint64_t> Expected = {
+        dwarf::DW_OP_constu,      3, dwarf::DW_OP_constu, 4, Barrier, 7,
+        dwarf::DW_OP_plus_uconst, 5, dwarf::DW_OP_constu, 13};
+    if (Barrier != dwarf::DW_OP_LLVM_label) {
+      Ops.append({dwarf::DW_OP_LLVM_label, 7});
+      Expected.append({dwarf::DW_OP_LLVM_label, 7});
+    }
+    EXPECT_EQ(DIExpression::get(Context, Expected),
+              DIExpression::get(Context, Ops)->foldConstantMath());
+  }
+
+  // Adding ops before or after an expression leaves its labels in place.
+  SmallVector<uint64_t> Ops = {dwarf::DW_OP_LLVM_label, 7,
+                               dwarf::DW_OP_plus_uconst, 1};
+  auto *Expr = DIExpression::get(Context, Ops);
+  auto *Prepended = DIExpression::prepend(Expr, DIExpression::DerefBefore, 0);
+  SmallVector<uint64_t> Expected = {dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_label,
+                                    7, dwarf::DW_OP_plus_uconst, 1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+
+  SmallVector<uint64_t> Prefix = {dwarf::DW_OP_LLVM_tag_offset, 3};
+  Prepended = DIExpression::prependOpcodes(Expr, Prefix);
+  Expected = {dwarf::DW_OP_LLVM_tag_offset, 3, dwarf::DW_OP_LLVM_label, 7,
+              dwarf::DW_OP_plus_uconst,     1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+  EXPECT_TRUE(Prepended->isValid());
+
+  SmallVector<uint64_t> AppendOps = {
+      dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+      dwarf::DW_OP_LLVM_convert, 64, dwarf::DW_ATE_signed};
+  auto *Appended = DIExpression::append(Expr, AppendOps);
+  Expected = {dwarf::DW_OP_LLVM_label,
+              7,
+              dwarf::DW_OP_plus_uconst,
+              1,
+              dwarf::DW_OP_LLVM_convert,
+              32,
+              dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert,
+              64,
+              dwarf::DW_ATE_signed};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+
+  // Label IDs are arbitrary uint64_t values, so appendExt must not mistake one
+  // for DW_OP_stack_value.
+  uint64_t LabelID = dwarf::DW_OP_stack_value;
+  Ops = {dwarf::DW_OP_LLVM_label, LabelID};
+  Appended =
+      DIExpression::appendExt(DIExpression::get(Context, Ops), 32, 64, true);
+  Expected = {dwarf::DW_OP_LLVM_label,   LabelID, dwarf::DW_OP_deref,
+              dwarf::DW_OP_LLVM_convert, 32,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert, 64,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_stack_value};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+}
+
 TEST_F(DIExpressionTest, isValid) {
 #define EXPECT_VALID(...)                                                      \
   do {                                                                         \
@@ -4271,6 +4351,7 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_VALID(dwarf::DW_OP_constu, 6, dwarf::DW_OP_plus);
   EXPECT_VALID(dwarf::DW_OP_constu, 5, dwarf::DW_OP_swap);
   EXPECT_VALID(dwarf::DW_OP_deref);
+  EXPECT_VALID(dwarf::DW_OP_stack_value);
   EXPECT_VALID(dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_plus_uconst, 6, dwarf::DW_OP_deref);
   EXPECT_VALID(dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst, 6);
@@ -4279,12 +4360,31 @@ TEST_F(DIExpressionTest, isValid) {
                dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_VALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_entry_value, 1);
+  // A label may be unused, and branches may refer forward or backward.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 2, dwarf::DW_OP_LLVM_label, 2);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 3, dwarf::DW_OP_LLVM_label, 4);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 4, dwarf::DW_OP_LLVM_skip, 5,
+               dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_LLVM_skip, 4);
+  // Label IDs can use the full uint64_t range.
+  EXPECT_VALID(
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max() - 1,
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max(),
+      dwarf::DW_OP_LLVM_skip, std::numeric_limits<uint64_t>::max() - 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_stack_value);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 6, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_tag_offset, 1, dwarf::DW_OP_LLVM_label, 6,
+               dwarf::DW_OP_LLVM_skip, 6, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
 
   // Invalid constructions.
   EXPECT_INVALID(~0u);
   EXPECT_INVALID(dwarf::DW_OP_plus, 0);
   EXPECT_INVALID(dwarf::DW_OP_plus_uconst);
   EXPECT_INVALID(dwarf::DW_OP_swap);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_deref);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3, 7, dwarf::DW_OP_plus_uconst, 3);
@@ -4294,6 +4394,48 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_plus_uconst, 5,
                  dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_LLVM_entry_value, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip, 1);
+  EXPECT_INVALID(dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_implicit_pointer, dwarf::DW_OP_LLVM_label,
+                 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_tag_offset, 0);
+  // Converts and other stack ops can appear around labels and branches.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_skip, 7, dwarf::DW_OP_LLVM_label, 7);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 1, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_convert, 64,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_dup, dwarf::DW_OP_LLVM_bra, 1,
+               dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 0, 32, dwarf::DW_OP_LLVM_label, 1);
+  // reg0 and entry_value normally end validation early, but they must not hide
+  // invalid control flow later in the expression.
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_LLVM_bra, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_label, 1,
+                 dwarf::DW_OP_LLVM_entry_value, 1);
 
 #undef EXPECT_VALID
 #undef EXPECT_INVALID
@@ -4371,6 +4513,19 @@ TEST_F(DIExpressionTest, createFragmentExpression) {
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
+  // Creating a fragment leaves the branches alone and adds the fragment at the
+  // end.
+  SmallVector<uint64_t> ControlFlowOps = {dwarf::DW_OP_LLVM_label, 1,
+                                          dwarf::DW_OP_LLVM_bra,   1,
+                                          dwarf::DW_OP_LLVM_skip,  1};
+  DIExpression *ControlFlowExpr = DIExpression::get(Context, ControlFlowOps);
+  auto Fragment =
+      DIExpression::createFragmentExpression(ControlFlowExpr, 0, 32);
+  ASSERT_TRUE(Fragment.has_value());
+  ControlFlowOps.append({dwarf::DW_OP_LLVM_fragment, 0, 32});
+  EXPECT_EQ(DIExpression::get(Context, ControlFlowOps), *Fragment);
+  EXPECT_TRUE((*Fragment)->isValid());
+
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
new file mode 100644
index 0000000000000..7ba2b0802fa3c
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
@@ -0,0 +1,23 @@
+; RUN: mlir-translate -import-llvm -mlir-print-debuginfo -emit-expensive-warnings %s 2>&1 | FileCheck %s
+
+; Import a forward branch and a backward skip, and make sure MLIR keeps the op
+; order and label IDs.
+
+; CHECK: llvm.intr.dbg.value {{.*}} #llvm.di_expression<[DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0), DW_OP_LLVM_label(42)]> = {{.*}} : i64
+define void @f(i64 %x) !dbg !4 {
+  #dbg_value(i64 %x, !DILocalVariable(scope: !4),
+             !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42,
+                           DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42),
+             !DILocation(scope: !4))
+  ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1,
+                            type: !DISubroutineType(types: !{null}),
+                            spFlags: DISPFlagDefinition, unit: !0)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
new file mode 100644
index 0000000000000..74a549b952327
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Export a forward branch and a backward skip, and make sure LLVM IR keeps the
+// op order and label IDs.
+
+#file = #llvm.di_file<"test.c" in "/">
+#cu = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_C,
+                            file = #file>
+#sp = #llvm.di_subprogram<compileUnit = #cu, scope = #file, name = "f",
+                          subprogramFlags = "Definition",
+                          type = #llvm.di_subroutine_type<types = #llvm.di_null_type>>
+
+// CHECK: #dbg_value(i64 %{{.*}}, !{{.*}}, !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42), !{{.*}})
+llvm.func @f(%arg: i64) {
+  llvm.intr.dbg.value #llvm.di_local_variable<scope = #sp> #llvm.di_expression<[
+    DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0),
+    DW_OP_LLVM_label(42)]> = %arg : i64
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])

>From 57cf90ad67b0e05c411e79518212f90e3faa39bc Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:53:45 -0700
Subject: [PATCH 2/3] [LLDB] Handle symbolic DIExpression opcodes

These opcodes only appear in LLVM metadata, but adding them to the shared DWARF enum makes LLDB's switch incomplete. List them with the other metadata-only opcodes so a -Wswitch build stays clean.
---
 lldb/source/Expression/DWARFExpression.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lldb/source/Expression/DWARFExpression.cpp b/lldb/source/Expression/DWARFExpression.cpp
index 299dbe97d6176..73bdd18d82cae 100644
--- a/lldb/source/Expression/DWARFExpression.cpp
+++ b/lldb/source/Expression/DWARFExpression.cpp
@@ -205,6 +205,9 @@ GetOpcodeDataSize(const DataExtractor &data, const lldb::offset_t data_offset,
   case DW_OP_LLVM_arg:
   case DW_OP_LLVM_extract_bits_sext:
   case DW_OP_LLVM_extract_bits_zext:
+  case DW_OP_LLVM_label:
+  case DW_OP_LLVM_bra:
+  case DW_OP_LLVM_skip:
     break;
   // Vendor extensions:
   case DW_OP_HP_is_value:

>From e1e1ba8492cb9a32cb924caadb78ff4129cc9581 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:58:04 -0700
Subject: [PATCH 3/3] [DebugInfo] Clarify symbolic branch limitations

Separate the DIExpression validity rules from the combinations we don't handle yet. DIArgList and DW_OP_LLVM_arg need more work in expressions and expression writers, while DW_OP_LLVM_implicit_pointer takes a different emission path. Also document the final branch offset range check.
---
 llvm/docs/SourceLevelDebugging.md | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 645e441e6fe2a..834781444c607 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -492,15 +492,23 @@ Label IDs are local to an expression:
   offset.
 - Branches can go forward, backward, to themselves, or form cycles.
 
-There are a few other restrictions:
+`DIExpression` validation also checks the following:
 
 - Put labels, branches, and skips before `DW_OP_stack_value` and
   `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
-- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
-  separately, so they can't be used with symbolic control flow.
 - Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
   applies to every path.
 
+There are also a couple of cases we don't handle yet:
+
+- `DIArgList` and `DW_OP_LLVM_arg` are currently rejected. CodeGen expands each
+  argument before it resolves the label offsets, so there isn't a representation
+  problem here; it mostly needs work handling it in expressions and expression
+  writers.
+- `DW_OP_LLVM_implicit_pointer` bypasses normal expression emission and only
+  handles a single location today. Supporting branches there is a bit more
+  work, since we'll need to work it back into our normal emission order.
+
 We don't check reachability, termination, or stack state where paths meet.
 
 `DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
@@ -508,6 +516,9 @@ and we don't match conversions on different paths. When CodeGen can't emit
 `DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
 branch, or skip would split the pair, CodeGen reports an error.
 
+CodeGen also reports an error if the final branch offset is outside
+`[-32768, 32767]`.
+
 Local expression rewrites stop at labels, branches, and skips; they can still
 add operations to either end, but they don't move, remove, or copy labels.
 


From llvmlistbot at llvm.org  Tue Jul 21 12:59:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 12:59:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 5df4aa1 - [flang][acc] Emit unified declare
 globals as extern. (#210757)
Message-ID: <6a5fcf8a.5a0412d2.3daf32.b2e5@mx.google.com>


Author: nvptm
Date: 2026-07-21T12:59:01-07:00
New Revision: 5df4aa19aa1e6308509356db19b5e46884d47361

URL: https://github.com/llvm/llvm-project/commit/5df4aa19aa1e6308509356db19b5e46884d47361
DIFF: https://github.com/llvm/llvm-project/commit/5df4aa19aa1e6308509356db19b5e46884d47361.diff

LOG: [flang][acc] Emit unified declare globals as extern. (#210757)

Emit external device symbols for OpenACC declare create and copyin
globals in -gpu=mem:unified; preserve device definitions for
device_resident globals.

Added: 
    flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir

Modified: 
    mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
    mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp

Removed: 
    


################################################################################
diff  --git a/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
new file mode 100644
index 0000000000000..b27094467f9cd
--- /dev/null
+++ b/flang/test/Fir/OpenACC/acc-declare-gpu-module-insertion.fir
@@ -0,0 +1,23 @@
+// RUN: fir-opt %s --pass-pipeline="builtin.module(acc-declare-gpu-module-insertion{cuda-unified=true},acc-declare-gpu-module-insertion{cuda-unified=true})" | FileCheck %s
+
+// Globals registered as host variables must be declarations in device code.
+// device_resident globals remain device definitions.
+module attributes {fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", gpu.container_module} {
+  fir.global internal @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+  fir.global @initialized_shared(dense<1.0> : vector<1xf32>) {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+  fir.global @resident {acc.declare = #acc.declare<dataClause = acc_declare_device_resident>} : !fir.array<7xf32> {
+    %0 = fir.zero_bits !fir.array<7xf32>
+    fir.has_value %0 : !fir.array<7xf32>
+  }
+}
+
+// CHECK-LABEL: gpu.module @acc_gpu_module {
+// CHECK: fir.global @shared {acc.declare = #acc.declare<dataClause = acc_create>} : !fir.array<7xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @initialized_shared {acc.declare = #acc.declare<dataClause = acc_copyin>} : !fir.array<1xf32>
+// CHECK-NOT: fir.has_value
+// CHECK: fir.global @resident {{.*}} : !fir.array<7xf32> {
+// CHECK: fir.has_value

diff  --git a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
index 04723c5db2d12..fbc832a37f478 100644
--- a/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/OpenACC/Transforms/Passes.td
@@ -157,6 +157,10 @@ def ACCDeclareGPUModuleInsertion : Pass<"acc-declare-gpu-module-insertion", "mli
     that device code can reference them.
   }];
   let dependentDialects = ["mlir::acc::OpenACCDialect", "mlir::gpu::GPUDialect"];
+  let options = [
+    Option<"cudaUnified", "cuda-unified", "bool", "false",
+           "Emit declaration-only device globals for CUDA unified memory.">
+  ];
 }
 
 def ACCLegalizeSerial : Pass<"acc-legalize-serial", "mlir::func::FuncOp"> {

diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
index 705b4f9df7770..51e2222208d0d 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCDeclareGPUModuleInsertion.cpp
@@ -77,6 +77,15 @@ static bool hasAccDeclareGlobals(ModuleOp mod) {
   return false;
 }
 
+static void makeDeviceGlobalDeclaration(Operation &globalOp) {
+  globalOp.removeAttr("initVal");
+  globalOp.removeAttr("linkName");
+  for (Region &region : globalOp.getRegions()) {
+    region.dropAllReferences();
+    region.getBlocks().clear();
+  }
+}
+
 class ACCDeclareGPUModuleInsertion
     : public acc::impl::ACCDeclareGPUModuleInsertionBase<
           ACCDeclareGPUModuleInsertion> {
@@ -97,6 +106,12 @@ class ACCDeclareGPUModuleInsertion
         continue;
 
       StringAttr name = symOp.getNameAttr();
+      Operation *deviceGlobal = globalOp.clone();
+      auto declareAttr =
+          globalOp.getAttrOfType<acc::DeclareAttr>(acc::getDeclareAttrName());
+      if (cudaUnified && declareAttr.getDataClause().getValue() !=
+                             acc::DataClause::acc_declare_device_resident)
+        makeDeviceGlobalDeclaration(*deviceGlobal);
 
       if (Operation *existing = gpuSymTable.lookup(name.getValue())) {
         // Reuse when structurally equivalent ignoring locations and discardable
@@ -104,11 +119,12 @@ class ACCDeclareGPUModuleInsertion
         // true definition mismatch is a conflict.
         if (existing->getName() != globalOp.getName() ||
             !OperationEquivalence::isEquivalentTo(
-                existing, &globalOp,
+                existing, deviceGlobal,
                 OperationEquivalence::ignoreValueEquivalence,
                 /*markEquivalent=*/nullptr,
                 OperationEquivalence::IgnoreLocations |
                     OperationEquivalence::IgnoreDiscardableAttrs)) {
+          deviceGlobal->destroy();
           accSupport.emitNYI(globalOp.getLoc(),
                              llvm::Twine("duplicate global symbol '") +
                                  name.getValue() + "' in gpu module");
@@ -120,10 +136,11 @@ class ACCDeclareGPUModuleInsertion
           if (Attribute declareAttr =
                   globalOp.getAttr(acc::getDeclareAttrName()))
             existing->setAttr(acc::getDeclareAttrName(), declareAttr);
+        deviceGlobal->destroy();
         continue;
       }
 
-      gpuSymTable.insert(globalOp.clone());
+      gpuSymTable.insert(deviceGlobal);
     }
     return success();
   }


        

From llvmlistbot at llvm.org  Tue Jul 21 12:59:09 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Tue, 21 Jul 2026 12:59:09 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][acc] Emit unified declare
 globals as extern. (PR #210757)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210757@github.com>
Message-ID: <6a5fcf8d.d681cab9.33af1e.ca00@mx.google.com>


https://github.com/clementval closed https://github.com/llvm/llvm-project/pull/210757

From llvmlistbot at llvm.org  Tue Jul 21 12:59:36 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 12:59:36 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5fcfa8.d76d81af.3b68ca.f201@mx.google.com>


================
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
----------------
echristo wrote:

Good point, they're also not nearly as dire as written :)

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 15:46:15 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 15:46:15 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a5ff6b7.13be6df8.323b61.7f28@mx.google.com>


================
@@ -7181,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
----------------
echristo wrote:

Let me see what that patch would look like :)

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 17:49:04 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Tue, 21 Jul 2026 17:49:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a601380.c5504292.1bbfc8.2086@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/211141

>From 829c25c037707f17d8c98fcb15c041914a26a46d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 16:07:42 -0700
Subject: [PATCH 1/2] Elevate allocs to globals pass

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |  10 ++
 .../Dialect/MemRef/Transforms/Transforms.h    |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElevateAllocsToGlobals.cpp     | 112 ++++++++++++++++++
 .../MemRef/elevate-allocs-to-globals.mlir     |  56 +++++++++
 5 files changed, 182 insertions(+)
 create mode 100644 mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
 create mode 100644 mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index c6be600247696..4e6ead48cbff2 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -340,4 +340,14 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
   ];
 }
 
+def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
+  let summary = "Elevate allocs to globals";
+  let description = [{
+
+  }];
+  let dependentDialects = [
+      "memref::MemRefDialect"
+  ];
+}
+
 #endif // MLIR_DIALECT_MEMREF_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
index 720677455ae5d..6960b8ef00f06 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #ifndef MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 #define MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
 
@@ -165,6 +166,8 @@ void populateExtractAddressComputationsPatterns(RewritePatternSet &patterns);
 /// into one-dimensional memref operations.
 void populateFlattenMemrefsPatterns(RewritePatternSet &patterns);
 
+void populateElevateAllocsToGlobalsPatterns(RewritePatternSet &patterns);
+
 /// Build a new memref::AllocaOp whose dynamic sizes are independent of all
 /// given independencies. If the op is already independent of all
 /// independencies, the same AllocaOp result is returned.
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1a8b03dabcfb7..7d43e66989bc4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -17,6 +17,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   ReifyResultShapes.cpp
   ResolveShapedTypeResultDims.cpp
   RuntimeOpVerification.cpp
+  ElevateAllocsToGlobals.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/MemRef
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
new file mode 100644
index 0000000000000..017b23b810dd3
--- /dev/null
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -0,0 +1,112 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Interfaces/LoopLikeInterface.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/LogicalResult.h"
+
+namespace mlir {
+namespace memref {
+#define GEN_PASS_DEF_ELEVATEALLOCSTOGLOBALSPASS
+#include "mlir/Dialect/MemRef/Transforms/Passes.h.inc"
+} // namespace memref
+} // namespace mlir
+
+using namespace mlir;
+
+namespace {
+
+// Checks if 'op' is contained inside any branching or looping structure
+static bool isInsideControlFlow(mlir::Operation *op) {
+  if (mlir::getEnclosingRepetitiveRegion(op) != nullptr)
+    return true;
+
+  if (op->getParentOfType<mlir::LoopLikeOpInterface>())
+    return true;
+
+  if (auto regionParent = op->getParentOfType<mlir::RegionBranchOpInterface>())
+    return true;
+
+  return false;
+}
+
+struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
+public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(memref::AllocOp allocOp,
+                                PatternRewriter &rewriter) const final {
+
+    auto memrefType = allocOp.getType();
+    // memref.global requires statically shaped memrefs
+    if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
+      return failure();
+
+    auto loopParent = allocOp->getParentOfType<mlir::LoopLikeOpInterface>();
+    if (loopParent != nullptr || isInsideControlFlow(allocOp))
+      return failure();
+
+    memref::GlobalOp globalOp;
+    {
+      Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
+
+      SymbolTable symbolTable(symbolTableOp);
+
+      OpBuilder builder(rewriter.getContext());
+      StringAttr globalName = rewriter.getStringAttr("global_alloc");
+      globalOp = memref::GlobalOp::create(builder, allocOp.getLoc(), globalName,
+                                          rewriter.getStringAttr("private"),
+                                          memrefType, rewriter.getUnitAttr(),
+                                          false, allocOp.getAlignmentAttr());
+
+      symbolTable.insert(globalOp);
+    }
+
+    SmallVector<Operation *> deallocsToDelete;
+    for (OpOperand &use : allocOp.getResult().getUses()) {
+      Operation *user = use.getOwner();
+      if (isa<memref::DeallocOp>(user))
+        deallocsToDelete.push_back(user);
+    }
+    for (Operation *dealloc : deallocsToDelete)
+      rewriter.eraseOp(dealloc);
+
+    rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
+                                                     globalOp.getName());
+
+    return success();
+  }
+};
+
+struct ElevateAllocsToGlobalsPass
+    : public mlir::memref::impl::ElevateAllocsToGlobalsPassBase<
+          ElevateAllocsToGlobalsPass> {
+  using Base::Base;
+
+  void runOnOperation() override {
+    ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    memref::populateElevateAllocsToGlobalsPatterns(patterns);
+
+    (void)applyPatternsGreedily(moduleOp, std::move(patterns));
+  }
+};
+} // namespace
+
+void mlir::memref::populateElevateAllocsToGlobalsPatterns(
+    RewritePatternSet &patterns) {
+  patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
new file mode 100644
index 0000000000000..dd65cd12989cc
--- /dev/null
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -0,0 +1,56 @@
+// RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
+
+/// Test that a single static memref.alloc is elevated to a memref.global,
+/// replaced with memref.get_global, alignment attribute is preserved, and
+/// associated memref.dealloc is removed.
+
+func.func @single_alloc(%val: f32, %idx: index) {
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
+  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
+  memref.dealloc %0 : memref<10x20xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @single_alloc(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: return
+// CHECK-NOT: memref.dealloc
+// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+
+// -----
+
+/// Test that multiple static memref.alloc ops in the same function are elevated
+/// to global memrefs without symbol name collisions.
+
+func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+  %0 = memref.alloc() : memref<10xf32>
+  %1 = memref.alloc() : memref<20xi32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  return
+}
+
+// CHECK-LABEL: func.func @multiple_allocs(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
+
+// -----
+
+/// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<?xf32>
+  memref.store %val, %0[%sz] : memref<?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>

>From 39d4e08842d4dafd659020418859af23193db91c Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 17:48:50 -0700
Subject: [PATCH 2/2] Simplify and add tests

---
 .../Transforms/ElevateAllocsToGlobals.cpp     |  2 +-
 .../MemRef/elevate-allocs-to-globals.mlir     | 88 +++++++++++++++----
 2 files changed, 70 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 017b23b810dd3..2dbdeb00c5354 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -109,4 +109,4 @@ struct ElevateAllocsToGlobalsPass
 void mlir::memref::populateElevateAllocsToGlobalsPatterns(
     RewritePatternSet &patterns) {
   patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
index dd65cd12989cc..43477960cf7be 100644
--- a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -1,49 +1,49 @@
 // RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
 
-/// Test that a single static memref.alloc is elevated to a memref.global,
-/// replaced with memref.get_global, alignment attribute is preserved, and
-/// associated memref.dealloc is removed.
+/// Test that a single static memref.alloc is elevated to a memref.global, references
+/// are replaced with memref.get_global, and the associated memref.dealloc is removed.
 
 func.func @single_alloc(%val: f32, %idx: index) {
-  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
-  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
-  memref.dealloc %0 : memref<10x20xf32>
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.dealloc %0 : memref<10xf32>
   return
 }
 
 // CHECK-LABEL: func.func @single_alloc(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<10xf32>
 // CHECK-NEXT: return
 // CHECK-NOT: memref.dealloc
-// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
 
 // -----
 
 /// Test that multiple static memref.alloc ops in the same function are elevated
 /// to global memrefs without symbol name collisions.
 
-func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+func.func @multiple_allocs(%val: f32, %idx: index) {
   %0 = memref.alloc() : memref<10xf32>
-  %1 = memref.alloc() : memref<20xi32>
+  %1 = memref.alloc() : memref<20xf32>
   memref.store %val, %0[%idx] : memref<10xf32>
-  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  memref.store %val, %1[%idx] : memref<20xf32>
   return
 }
 
 // CHECK-LABEL: func.func @multiple_allocs(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
 // CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
-// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
-// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
-// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
-// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG1]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM1]][%[[ARG1]]] : memref<20xf32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xf32> = uninitialized
 // CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
 
 // -----
 
 /// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+
 func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
   %0 = memref.alloc(%sz) : memref<?xf32>
   memref.store %val, %0[%sz] : memref<?xf32>
@@ -52,5 +52,55 @@ func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
 
 // CHECK-LABEL: func.func @dynamic_alloc_ignored(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
-// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+
+// -----
+
+/// Test that a partially dynamic memref.alloc is ignored and not elevated to a global.
+
+func.func @partially_dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<10x?xf32>
+  memref.store %val, %0[%sz, %sz] : memref<10x?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @partially_dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<10x?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x?xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside a loop is ignored and not elevated to a global.
+
+func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index, %val: f32, %idx: index) {
+  scf.for %i = %lb to %ub step %step {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_loop_ignored(
+// CHECK: scf.for
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+
+// -----
+
+/// Test that a static memref.alloc inside control flow (scf.if) is ignored and not
+/// elevated to a global.
+
+func.func @alloc_in_control_flow_ignored(%cond: i1, %val: f32, %idx: index) {
+  scf.if %cond {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_control_flow_ignored(
+// CHECK: scf.if
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]


From llvmlistbot at llvm.org  Tue Jul 21 20:23:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 20:23:05 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a603799.7d702ed2.2f6c05.9341@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

* 168229 tests passed
* 4039 tests skipped

All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.

<details>
<summary>[code=4294967295] bin/mlir-lsp-server.exe</summary>

```
FAILED: [code=4294967295] bin/mlir-lsp-server.exe
C:\Windows\system32\cmd.exe /C "cd . && C:\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe -E vs_link_exe --msvc-ver=1944 --intdir=tools\mlir\tools\mlir-lsp-server\CMakeFiles\mlir-lsp-server.dir --rc="C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\rc.exe" --mt="C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\x64\mt.exe" --manifests  -- C:\clang\clang-msvc\bin\lld-link.exe /nologo @CMakeFiles\mlir-lsp-server.rsp  /out:bin\mlir-lsp-server.exe /implib:lib\mlir-lsp-server.lib /pdb:bin\mlir-lsp-server.pdb /version:0.0 /MANIFEST:NO /Brepro /INCREMENTAL:NO /subsystem:console  /STACK:10000000 && cd ."
LINK: command "C:\clang\clang-msvc\bin\lld-link.exe /nologo @CMakeFiles\mlir-lsp-server.rsp /out:bin\mlir-lsp-server.exe /implib:lib\mlir-lsp-server.lib /pdb:bin\mlir-lsp-server.pdb /version:0.0 /MANIFEST:NO /Brepro /INCREMENTAL:NO /subsystem:console /STACK:10000000" failed (exit code 1) with the following output:
lld-link: error: undefined symbol: void __cdecl mlir::vector::registerValueBoundsOpInterfaceExternalModels(class mlir::DialectRegistry &)
>>> referenced by MLIRRegisterAllDialects.lib(RegisterAllDialects.cpp.obj):(void __cdecl mlir::registerAllDialects(class mlir::DialectRegistry &))

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::VectorDialect, void>::id
>>> referenced by MLIRRegisterAllDialects.lib(RegisterAllDialects.cpp.obj):(public: void __cdecl mlir::DialectRegistry::insert<class mlir::vector::VectorDialect, class mlir::wasmssa::WasmSSADialect, class mlir::x86::X86Dialect, class mlir::xegpu::XeGPUDialect, class mlir::xevm::XeVMDialect>(void))
>>> referenced by MLIRRegisterAllDialects.lib(RegisterAllDialects.cpp.obj):(private: virtual class mlir::Dialect * __cdecl std::_Func_impl_no_alloc<class `public: void __cdecl mlir::DialectRegistry::insert<class mlir::vector::VectorDialect>(void)'::`1'::<lambda_1>, class mlir::Dialect *, class mlir::MLIRContext *>::_Do_call(class mlir::MLIRContext *&&))
>>> referenced by MLIRArmSMEDialect.lib(ArmSME.cpp.obj):(private: __cdecl mlir::arm_sme::ArmSMEDialect::ArmSMEDialect(class mlir::MLIRContext *))
>>> referenced 15 more times

lld-link: error: undefined symbol: private: __cdecl mlir::vector::VectorDialect::VectorDialect(class mlir::MLIRContext *)
>>> referenced by MLIRRegisterAllDialects.lib(RegisterAllDialects.cpp.obj):(private: static class std::unique_ptr<class mlir::Dialect, struct std::default_delete<class mlir::Dialect>> __cdecl llvm::function_ref<class std::unique_ptr<class mlir::Dialect, struct std::default_delete<class mlir::Dialect>> __cdecl(void)>::callback_fn<class `public: class mlir::vector::VectorDialect * __cdecl mlir::MLIRContext::getOrLoadDialect<class mlir::vector::VectorDialect>(void)'::`1'::<lambda_1>>(__int64))
>>> referenced by MLIRArmSMEDialect.lib(ArmSME.cpp.obj)
>>> referenced by MLIRArmSVEDialect.lib(ArmSVEDialect.cpp.obj)

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::TransferReadOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::TransferReadOp, class mlir::OpTrait::ZeroRegions, class mlir::OpTrait::OneResult, class mlir::OpTrait::OneTypedResult<class mlir::VectorType>::Impl, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::AtLeastNOperands<2>::Impl, class mlir::OpTrait::AttrSizedOperandSegments, class mlir::OpTrait::OpInvariants, struct mlir::BytecodeOpInterface::Trait, struct mlir::VectorTransferOpInterface::Trait, struct mlir::VectorUnrollOpInterface::Trait, struct mlir::vector::MaskableOpInterface::Trait, struct mlir::MemoryEffectOpInterface::Trait, struct mlir::ConditionallySpeculatable::Trait, struct mlir::MemorySpaceCastConsumerOpInterface::Trait, struct mlir::DestinationStyleOpInterface::Trait>::classof(class mlir::Operation *))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerSubsetOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced 4 more times

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::TransferWriteOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::TransferWriteOp, class mlir::OpTrait::ZeroRegions, class mlir::OpTrait::VariadicResults, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::AtLeastNOperands<2>::Impl, class mlir::OpTrait::AttrSizedOperandSegments, class mlir::OpTrait::OpInvariants, struct mlir::BytecodeOpInterface::Trait, struct mlir::VectorTransferOpInterface::Trait, struct mlir::VectorUnrollOpInterface::Trait, struct mlir::vector::MaskableOpInterface::Trait, struct mlir::MemoryEffectOpInterface::Trait, struct mlir::ConditionallySpeculatable::Trait, struct mlir::MemorySpaceCastConsumerOpInterface::Trait, struct mlir::DestinationStyleOpInterface::Trait>::classof(class mlir::Operation *))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerSubsetOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced 1 more times

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::GatherOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::GatherOp, class mlir::OpTrait::ZeroRegions, class mlir::OpTrait::OneResult, class mlir::OpTrait::OneTypedResult<class mlir::VectorType>::Impl, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::AtLeastNOperands<4>::Impl, class mlir::OpTrait::OpInvariants, struct mlir::BytecodeOpInterface::Trait, struct mlir::vector::MaskableOpInterface::Trait, struct mlir::MemorySpaceCastConsumerOpInterface::Trait, struct mlir::VectorUnrollOpInterface::Trait, struct mlir::AlignmentAttrOpInterface::Trait, struct mlir::MemoryEffectOpInterface::Trait>::classof(class mlir::Operation *))
>>> referenced by MLIRVectorToLLVM.lib(ConvertVectorToLLVM.cpp.obj)

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::MaskOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::MaskOp, class mlir::OpTrait::OneRegion, class mlir::OpTrait::VariadicResults, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::AtLeastNOperands<1>::Impl, struct mlir::OpTrait::SingleBlock, class mlir::OpTrait::SingleBlockImplicitTerminator<class mlir::vector::YieldOp>::Impl, struct mlir::OpTrait::NoRegionArguments, class mlir::OpTrait::OpInvariants, struct mlir::vector::MaskingOpInterface::Trait, class mlir::OpTrait::HasRecursiveMemoryEffects>::classof(class mlir::Operation *))
>>> referenced by MLIRVectorToLLVM.lib(ConvertVectorToLLVM.cpp.obj)

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::YieldOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::YieldOp, class mlir::OpTrait::ZeroRegions, class mlir::OpTrait::ZeroResults, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::VariadicOperands, class mlir::OpTrait::OpInvariants, struct mlir::ConditionallySpeculatable::Trait, struct mlir::OpTrait::AlwaysSpeculatableImplTrait, struct mlir::MemoryEffectOpInterface::Trait, struct mlir::RegionBranchTerminatorOpInterface::Trait, struct mlir::OpTrait::ReturnLike, class mlir::OpTrait::IsTerminator>::classof(class mlir::Operation *))
>>> referenced by MLIRLinalgTransforms.lib(ConvertToDestinationStyle.cpp.obj)

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<class mlir::vector::ScatterOp, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(private: static <auto> __cdecl `void __cdecl mlir::vector::registerBufferizableOpInterfaceExternalModels(class mlir::DialectRegistry &)'::`1'::<lambda_0>::__invoke(class mlir::MLIRContext *, class mlir::vector::VectorDialect *))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::Op<class mlir::vector::ScatterOp, class mlir::OpTrait::ZeroRegions, class mlir::OpTrait::VariadicResults, class mlir::OpTrait::ZeroSuccessors, class mlir::OpTrait::AtLeastNOperands<4>::Impl, class mlir::OpTrait::OpInvariants, struct mlir::BytecodeOpInterface::Trait, struct mlir::MemorySpaceCastConsumerOpInterface::Trait, struct mlir::AlignmentAttrOpInterface::Trait, struct mlir::MemoryEffectOpInterface::Trait>::classof(class mlir::Operation *))
>>> referenced by MLIRVectorToLLVM.lib(ConvertVectorToLLVM.cpp.obj)

lld-link: error: undefined symbol: public: struct std::pair<unsigned int, unsigned int> __cdecl mlir::vector::TransferReadOp::getODSOperandIndexAndLength(unsigned int)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced 122 more times

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<struct mlir::vector::detail::TransferReadOpGenericAdaptorBase::Properties, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRTensorTransforms.lib(FoldTensorSubsetOps.cpp.obj):(public: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferReadOfExtractSliceOpFolder::matchAndRewriteMaskableOp(class mlir::vector::TransferReadOp, class vector::TransferReadOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced by MLIRVectorTransforms.lib(LowerVectorTransfer.cpp.obj):(public: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferOpReduceRank::matchAndRewriteMaskableOp(class mlir::vector::TransferReadOp, class vector::TransferReadOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced 11 more times

lld-link: error: undefined symbol: public: class mlir::AffineMap __cdecl mlir::vector::TransferReadOp::getPermutationMap(void)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(public: class llvm::SmallVector<__int64, 6> __cdecl mlir::detail::VectorTransferOpInterfaceTrait<class mlir::vector::TransferReadOp>::getTransferChunkAccessed(void))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(public: class llvm::SmallVector<__int64, 6> __cdecl mlir::detail::VectorTransferOpInterfaceTrait<class mlir::vector::TransferReadOp>::getTransferChunkAccessed(void))
>>> referenced 43 more times

lld-link: error: undefined symbol: public: class mlir::VectorType __cdecl mlir::vector::TransferReadOp::getVectorType(void)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(public: class llvm::SmallVector<__int64, 6> __cdecl mlir::detail::VectorTransferOpInterfaceTrait<class mlir::vector::TransferReadOp>::getTransferChunkAccessed(void))
>>> referenced by MLIRTensorTransforms.lib(FoldTensorSubsetOps.cpp.obj):(public: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferReadOfExtractSliceOpFolder::matchAndRewriteMaskableOp(class mlir::vector::TransferReadOp, class vector::TransferReadOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced 37 more times

lld-link: error: undefined symbol: public: static class mlir::vector::TransferReadOp __cdecl mlir::vector::TransferReadOp::create(class mlir::OpBuilder &, class mlir::Location, class mlir::Type, class mlir::Value, class mlir::ValueRange, class mlir::AffineMap, class mlir::Value, class mlir::Value, class mlir::ArrayAttr)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferReadOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(VectorTransferOpTransforms.cpp.obj):(private: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferReadDropUnitDimsPattern::matchAndRewriteMaskableOp(class mlir::vector::TransferReadOp, class vector::TransferReadOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced by MLIRVectorTransforms.lib(LowerVectorMask.cpp.obj):(public: virtual struct llvm::LogicalResult __cdecl `anonymous namespace'::MaskedTransferReadOpPattern::matchAndRewriteMaskableOp(class mlir::vector::TransferReadOp, class vector::TransferReadOp::MaskingOpInterface, class vector::PatternRewriter &) const)

lld-link: error: undefined symbol: public: struct std::pair<unsigned int, unsigned int> __cdecl mlir::vector::TransferWriteOp::getODSOperandIndexAndLength(unsigned int)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferizesToMemoryRead(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::OpOperand &, class mlir::bufferization::AnalysisState const &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced 91 more times

lld-link: error: undefined symbol: public: class mlir::VectorType __cdecl mlir::vector::TransferWriteOp::getVectorType(void)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static bool __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferizesToMemoryRead(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::OpOperand &, class mlir::bufferization::AnalysisState const &))
>>> referenced by MLIRVectorTransforms.lib(SubsetOpInterfaceImpl.cpp.obj):(public: class llvm::SmallVector<__int64, 6> __cdecl mlir::detail::VectorTransferOpInterfaceTrait<class mlir::vector::TransferWriteOp>::getTransferChunkAccessed(void))
>>> referenced by MLIRGPUTransformOps.lib(GPUTransformOps.cpp.obj):(private: virtual class std::optional<class llvm::SmallVector<__int64, 6>> __cdecl std::_Func_impl_no_alloc<class `public: void __cdecl mlir::transform::ApplyUnrollVectorsSubgroupMmaOp::populatePatterns(class mlir::RewritePatternSet &)'::`1'::<lambda_3>, class std::optional<class llvm::SmallVector<__int64, 6>>, class mlir::Operation *>::_Do_call(class mlir::Operation *&&))
>>> referenced 25 more times

lld-link: error: undefined symbol: private: static class mlir::SelfOwningTypeID mlir::detail::TypeIDResolver<struct mlir::vector::detail::TransferWriteOpGenericAdaptorBase::Properties, void>::id
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRTensorTransforms.lib(FoldTensorSubsetOps.cpp.obj):(public: virtual struct llvm::LogicalResult __cdecl `anonymous namespace'::InsertSliceOfTransferWriteOpFolder::matchAndRewrite(class mlir::tensor::InsertSliceOp, class tensor::PatternRewriter &) const)
>>> referenced 12 more times

lld-link: error: undefined symbol: public: class mlir::Value __cdecl mlir::vector::TransferWriteOp::getVector(void)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRNVGPUTransformOps.lib(NVGPUTransformOps.cpp.obj):(public: class mlir::DiagnosedSilenceableFailure __cdecl mlir::transform::PipelineSharedMemoryCopiesOp::applyToOne(class mlir::transform::TransformRewriter &, class mlir::scf::ForOp, class mlir::transform::ApplyToEachResultList &, class mlir::transform::TransformState &))
>>> referenced by MLIRTensorTransforms.lib(FoldTensorSubsetOps.cpp.obj):(public: virtual struct llvm::LogicalResult __cdecl `anonymous namespace'::InsertSliceOfTransferWriteOpFolder::matchAndRewrite(class mlir::tensor::InsertSliceOp, class tensor::PatternRewriter &) const)
>>> referenced 29 more times

lld-link: error: undefined symbol: public: static class mlir::vector::TransferWriteOp __cdecl mlir::vector::TransferWriteOp::create(class mlir::OpBuilder &, class mlir::Location, class mlir::Value, class mlir::Value, class mlir::ValueRange, class mlir::AffineMapAttr, class mlir::Value, class mlir::ArrayAttr)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::TransferWriteOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(LowerVectorTransfer.cpp.obj):(public: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferWritePermutationLowering::matchAndRewriteMaskableOp(class mlir::vector::TransferWriteOp, class vector::TransferWriteOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced by MLIRVectorTransforms.lib(LowerVectorTransfer.cpp.obj):(public: virtual class llvm::FailureOr<class mlir::Value> __cdecl `anonymous namespace'::TransferWriteNonPermutationLowering::matchAndRewriteMaskableOp(class mlir::vector::TransferWriteOp, class vector::TransferWriteOp::MaskingOpInterface, class vector::PatternRewriter &) const)
>>> referenced 3 more times

lld-link: error: undefined symbol: public: struct std::pair<unsigned int, unsigned int> __cdecl mlir::vector::GatherOp::getODSOperandIndexAndLength(unsigned int)
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::GatherOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::GatherOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced by MLIRVectorTransforms.lib(BufferizableOpInterfaceImpl.cpp.obj):(public: static struct llvm::LogicalResult __cdecl mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::FallbackModel<struct mlir::vector::`anonymous namespace'::GatherOpInterface>::bufferize(struct mlir::bufferization::detail::BufferizableOpInterfaceInterfaceTraits::Concept const *, class mlir::Operation *, class mlir::RewriterBase &, struct mlir::bufferization::BufferizationOptions const &, class mlir::bufferization::BufferizationState &))
>>> referenced 27 more times

lld-link: error: too many errors emitted, stopping now (use /errorlimit:0 to see all errors)
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 20:37:06 2026
From: llvmlistbot at llvm.org (Aman LaChapelle)
Date: Tue, 21 Jul 2026 20:37:06 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a603ae2.b1416da1.20b4cf.f1e5@mx.google.com>


================
@@ -7181,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
----------------
bzcheeseman wrote:

Nice, thank you!

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Tue Jul 21 22:22:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:22:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] e8b66ff - [OpenACC] Accept funcs with
 `acc.specialized_routine` in `acc::isValidSymbolUse` (#211111)
Message-ID: <6a6053a4.9edb5c9d.3c157a.9c97@mx.google.com>


Author: Moazin K.
Date: 2026-07-21T22:22:40-07:00
New Revision: e8b66ff170f99c6cf275444f22a3aded65ef0bc1

URL: https://github.com/llvm/llvm-project/commit/e8b66ff170f99c6cf275444f22a3aded65ef0bc1
DIFF: https://github.com/llvm/llvm-project/commit/e8b66ff170f99c6cf275444f22a3aded65ef0bc1.diff

LOG: [OpenACC] Accept funcs with `acc.specialized_routine` in `acc::isValidSymbolUse` (#211111)

`ACCRoutineLowering` generates device side specialized version of
routines that have the `acc.specialized_routine` attribute. Those are
expected to be offloaded and therefore are valid symbol uses.

Assisted-by: Claude Code

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Utils/OpenACCUtils.cpp
    mlir/test/Dialect/OpenACC/offload-target-verifier.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtils.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtils.cpp
index a84898b7678fc..5e04a4154adf3 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtils.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtils.cpp
@@ -220,9 +220,10 @@ bool mlir::acc::isValidSymbolUse(mlir::Operation *user,
   // Check if the defining op is a function
   if (auto func =
           mlir::dyn_cast_if_present<mlir::FunctionOpInterface>(definingOp)) {
-    // If this symbol is actually an acc routine - then it is expected for it
-    // to be offloaded - therefore it is valid.
-    if (func->hasAttr(mlir::acc::getRoutineInfoAttrName()))
+    // If this symbol is actually an acc routine or a specialized acc routine -
+    // then it is expected for it to be offloaded - therefore it is valid.
+    if (func->hasAttr(mlir::acc::getRoutineInfoAttrName()) ||
+        func->hasAttr(mlir::acc::getSpecializedRoutineAttrName()))
       return true;
 
     // If this symbol is a call to an LLVM intrinsic, then it is likely valid.

diff  --git a/mlir/test/Dialect/OpenACC/offload-target-verifier.mlir b/mlir/test/Dialect/OpenACC/offload-target-verifier.mlir
index f004f150cdd37..78b0018237d68 100644
--- a/mlir/test/Dialect/OpenACC/offload-target-verifier.mlir
+++ b/mlir/test/Dialect/OpenACC/offload-target-verifier.mlir
@@ -245,3 +245,37 @@ func.func @test_f64_scalar() {
   }
   return
 }
+
+// -----
+
+// Test call to a function marked as an acc routine - should pass
+acc.routine @acc_routine_0 func(@routine_callee) gang
+func.func @routine_callee() attributes {acc.routine_info = #acc.routine_info<[@acc_routine_0]>} {
+  return
+}
+
+func.func @test_acc_routine() {
+  // expected-remark @below {{passed validity check}}
+  acc.parallel {
+    func.call @routine_callee() : () -> ()
+    acc.yield
+  }
+  return
+}
+
+// -----
+
+// Test call to a function marked as a specialized acc routine - should pass
+acc.routine @acc_routine_1 func(@specialized_callee) gang
+func.func @specialized_callee() attributes {acc.specialized_routine = #acc.specialized_routine<@acc_routine_1, <gang_dim1>, "specialized_callee">} {
+  return
+}
+
+func.func @test_acc_specialized_routine() {
+  // expected-remark @below {{passed validity check}}
+  acc.parallel {
+    func.call @specialized_callee() : () -> ()
+    acc.yield
+  }
+  return
+}


        

From llvmlistbot at llvm.org  Tue Jul 21 22:22:47 2026
From: llvmlistbot at llvm.org (Moazin K.)
Date: Tue, 21 Jul 2026 22:22:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] Accept funcs with
 `acc.specialized_routine` in `acc::isValidSymbolUse` (PR #211111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211111@github.com>
Message-ID: <6a6053a7.e5907290.33b622.6c64@mx.google.com>


https://github.com/moazin closed https://github.com/llvm/llvm-project/pull/211111

From llvmlistbot at llvm.org  Tue Jul 21 22:27:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:27:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][NVVM] Diagnose non-NVVM target ops
 before serialization (PR #210000)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210000@github.com>
Message-ID: <6a6054c6.7270d2fd.1ec500.417e@mx.google.com>


https://github.com/shubhamnarlawar closed https://github.com/llvm/llvm-project/pull/210000

From llvmlistbot at llvm.org  Tue Jul 21 22:29:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:29:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
Message-ID: <llvm.org/llvm/llvm-project/pull/211173@github.com>


https://github.com/1sgtpepper created https://github.com/llvm/llvm-project/pull/211173

# [mlir][arith] Preserve fast-math flags when lowering floating casts

Fixes #156692.

Convert Arith fast-math flags to LLVM fast-math flags when lowering
`arith.extf` and unconstrained `arith.truncf`. Previously these two patterns
passed the source-dialect attribute through unchanged, producing an Arith
attribute on an LLVM operation.

The constrained `arith.truncf` path remains unchanged.

Tests cover fast-math flag conversion for both floating-point casts.

Checks:
- Fork CI: `mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir`
  (https://github.com/1sgtpepper/llvm-project/actions/runs/29861025327)


>From f23adcfb3a26bd0b1a1eb898587bb5954603efc5 Mon Sep 17 00:00:00 2001
From: 1sgtpepper <165563006+1sgtpepper at users.noreply.github.com>
Date: Wed, 22 Jul 2026 00:05:30 +0800
Subject: [PATCH] [mlir][arith] Preserve fast-math flags when lowering floating
 casts

---
 mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp |  4 ++--
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir   | 17 ++++++++++-------
 2 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..254a8b9a4d296 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -114,7 +114,7 @@ using DivSIOpLowering =
 using DivUIOpLowering =
     VectorConvertToLLVMPattern<arith::DivUIOp, LLVM::UDivOp>;
 using ExtFOpLowering = VectorConvertToLLVMPattern<arith::ExtFOp, LLVM::FPExtOp,
-                                                  AttrConvertPassThrough,
+                                                  arith::AttrConvertFastMathToLLVM,
                                                   /*FailOnUnsupportedFP=*/true>;
 using ExtSIOpLowering =
     VectorConvertToLLVMPattern<arith::ExtSIOp, LLVM::SExtOp>;
@@ -202,7 +202,7 @@ using SubIOpLowering =
 using TruncFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::TruncFOp, LLVM::FPTruncOp,
                                           /*HasRoundingMode=*/false,
-                                          AttrConvertPassThrough,
+                                          arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedTruncFOpLowering = ConstrainedVectorConvertToLLVMPattern<
     arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*HasRoundingMode=*/true,
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d8f7cfcaded17 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -866,18 +866,22 @@ func.func @ops_supporting_fastmath(%arg0: f32, %arg1: f32, %arg2: i32) {
   %0 = arith.addf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fdiv %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
   %1 = arith.divf %arg0, %arg1 fastmath<fast> : f32
+// CHECK: llvm.fpext %arg0 fastmath<fast> : f32 to f64
+  %2 = arith.extf %arg0 fastmath<fast> : f32 to f64
 // CHECK: llvm.intr.maximum(%arg0, %arg1) {fastmathFlags = #llvm.fastmath<fast>} : (f32, f32) -> f32
-  %2 = arith.maximumf %arg0, %arg1 fastmath<fast> : f32
+  %3 = arith.maximumf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.intr.minimum(%arg0, %arg1) {fastmathFlags = #llvm.fastmath<fast>} : (f32, f32) -> f32
-  %3 = arith.minimumf %arg0, %arg1 fastmath<fast> : f32
+  %4 = arith.minimumf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fmul %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %4 = arith.mulf %arg0, %arg1 fastmath<fast> : f32
+  %5 = arith.mulf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fneg %arg0  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %5 = arith.negf %arg0 fastmath<fast> : f32
+  %6 = arith.negf %arg0 fastmath<fast> : f32
 // CHECK: llvm.frem %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %6 = arith.remf %arg0, %arg1 fastmath<fast> : f32
+  %7 = arith.remf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fsub %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %7 = arith.subf %arg0, %arg1 fastmath<fast> : f32
+  %8 = arith.subf %arg0, %arg1 fastmath<fast> : f32
+// CHECK: llvm.fptrunc %arg0 fastmath<fast> : f32 to f16
+  %9 = arith.truncf %arg0 fastmath<fast> : f32 to f16
   return
 }
 
@@ -981,4 +985,3 @@ func.func @supported_fp_type(%arg0: f32, %arg1: vector<4xf32>, %arg2: vector<4x8
   %3 = arith.cmpf oeq, %arg0, %arg3 : f32
   return
 }
-


From llvmlistbot at llvm.org  Tue Jul 21 22:29:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:29:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a60554b.d1496011.3cfe19.e2a1@mx.google.com>


https://github.com/1sgtpepper edited https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Tue Jul 21 22:30:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:30:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a605571.92ea0e2a.2ed880.e750@mx.google.com>


github-actions[bot] wrote:

⚠️ We detected that you are using a GitHub private e-mail address to contribute to the repo.<br/> Please turn off [Keep my email addresses private](https://github.com/settings/emails) setting in your account.<br/> See [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) and [LLVM Discourse](https://discourse.llvm.org/t/hidden-emails-on-github-should-we-do-something-about-it) for more information.

https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Tue Jul 21 22:30:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:30:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a605591.92d2873c.2cdd42.9255@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 254a8b9a4..619cb0b74 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -113,9 +113,10 @@ using DivSIOpLowering =
     VectorConvertToLLVMPattern<arith::DivSIOp, LLVM::SDivOp>;
 using DivUIOpLowering =
     VectorConvertToLLVMPattern<arith::DivUIOp, LLVM::UDivOp>;
-using ExtFOpLowering = VectorConvertToLLVMPattern<arith::ExtFOp, LLVM::FPExtOp,
-                                                  arith::AttrConvertFastMathToLLVM,
-                                                  /*FailOnUnsupportedFP=*/true>;
+using ExtFOpLowering =
+    VectorConvertToLLVMPattern<arith::ExtFOp, LLVM::FPExtOp,
+                               arith::AttrConvertFastMathToLLVM,
+                               /*FailOnUnsupportedFP=*/true>;
 using ExtSIOpLowering =
     VectorConvertToLLVMPattern<arith::ExtSIOp, LLVM::SExtOp>;
 using ExtUIOpLowering =

``````````

</details>


https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Tue Jul 21 22:53:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:53:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Reject invalid boundary types (PR
	#208787)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208787@github.com>
Message-ID: <6a605abd.510f576d.33aa7d.9296@mx.google.com>


https://github.com/mygitljf updated https://github.com/llvm/llvm-project/pull/208787

>From a66e3ae961cdcc17f6645dbf575f08964e7782a2 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Sat, 11 Jul 2026 01:29:55 +0000
Subject: [PATCH 1/2] [mlir][bufferization] Reject invalid boundary types

---
 .../FuncBufferizableOpInterfaceImpl.cpp       | 29 +++++++++++++++++--
 ...est-one-shot-module-bufferize-invalid.mlir |  8 +++++
 2 files changed, 34 insertions(+), 3 deletions(-)
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir

diff --git a/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
index 8ca968367b026..f38de559353a8 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
@@ -60,16 +60,32 @@ getDefaultMemorySpace(const BufferizationOptions &options,
   return nullptr;
 }
 
+static LogicalResult verifyMemRefElementType(FuncOp funcOp,
+                                             TensorLikeType type) {
+  auto tensorType = dyn_cast<TensorType>(type);
+  if (!tensorType)
+    return success();
+  Type elementType = tensorType.getElementType();
+  if (BaseMemRefType::isValidElementType(elementType))
+    return success();
+  return funcOp.emitError() << "cannot bufferize function boundary type "
+                            << tensorType << ": element type " << elementType
+                            << " is not a valid memref element type";
+}
+
 /// Return the index-th bufferized function argument type. This assumes that the
 /// specified argument is a tensor. If the tensor is ranked, a layout map may be
 /// specified by the user (as per `options.functionArgTypeConverterFn`).
-static BufferLikeType
+static FailureOr<BufferLikeType>
 getBufferizedFunctionArgType(FuncOp funcOp, int64_t index,
                              const BufferizationOptions &options) {
   auto type =
       dyn_cast<TensorLikeType>(funcOp.getFunctionType().getInput(index));
   assert(type && "expected TensorLikeType");
 
+  if (failed(verifyMemRefElementType(funcOp, type)))
+    return failure();
+
   // Note: For builtin tensors there is additional logic related to layout.
   if (auto tensorType = dyn_cast<TensorType>(type)) {
     BufferLikeType memrefType = options.functionArgTypeConverterFn(
@@ -245,6 +261,8 @@ struct CallOpInterface
 
     // Otherwise, call the type converter to compute the bufferized type.
     auto tensorType = cast<TensorLikeType>(resultType);
+    if (failed(verifyMemRefElementType(funcOp, tensorType)))
+      return failure();
     return cast<BufferLikeType>(options.functionArgTypeConverterFn(
         tensorType, getDefaultMemorySpace(options, tensorType), funcOp,
         options));
@@ -444,8 +462,11 @@ struct FuncOpInterface
     for (const auto &it : llvm::enumerate(funcType.getInputs())) {
       Type argType = it.value();
       if (isa<TensorLikeType>(argType)) {
-        argTypes.push_back(
-            getBufferizedFunctionArgType(funcOp, it.index(), options));
+        FailureOr<BufferLikeType> bufferType =
+            getBufferizedFunctionArgType(funcOp, it.index(), options);
+        if (failed(bufferType))
+          return failure();
+        argTypes.push_back(*bufferType);
         continue;
       }
       argTypes.push_back(argType);
@@ -455,6 +476,8 @@ struct FuncOpInterface
     SmallVector<Type> retTypes;
     for (Type resultType : funcType.getResults()) {
       if (auto tensorType = dyn_cast<TensorLikeType>(resultType)) {
+        if (failed(verifyMemRefElementType(funcOp, tensorType)))
+          return failure();
         BufferLikeType resultType = options.functionArgTypeConverterFn(
             tensorType, getDefaultMemorySpace(options, tensorType), funcOp,
             options);
diff --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir
new file mode 100644
index 0000000000000..4a2de50db1de9
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir
@@ -0,0 +1,8 @@
+// RUN: mlir-opt %s -test-one-shot-module-bufferize -verify-diagnostics
+// RUN: mlir-opt %s -one-shot-bufferize="bufferize-function-boundaries=1" -verify-diagnostics
+
+module {
+  // expected-error @below {{cannot bufferize function boundary type 'tensor<!llvm.array<1 x i32>>': element type '!llvm.array<1 x i32>' is not a valid memref element type}}
+  // expected-error @below {{failed to bufferize op}}
+  func.func private @sparse_csr(tensor<f64>) -> tensor<!llvm.array<1 x i32>>
+}

>From cad752a93f49ac53d2a421032428de11b1aff565 Mon Sep 17 00:00:00 2001
From: mygitljf <2410316423 at qq.com>
Date: Wed, 22 Jul 2026 13:52:12 +0800
Subject: [PATCH 2/2] [mlir][bufferization] Validate function boundary
 conversions

---
 .../IR/BufferizableOpInterface.h              |  1 +
 .../IR/BufferizableOpInterface.cpp            | 14 +++++
 .../FuncBufferizableOpInterfaceImpl.cpp       | 61 +++++++++----------
 ...-shot-module-bufferize-invalid-layout.mlir | 29 +++++++++
 ...est-one-shot-module-bufferize-invalid.mlir | 12 +++-
 .../test-one-shot-module-bufferize.mlir       | 34 +++++++++++
 .../TestOneShotModuleBufferize.cpp            | 22 ++++++-
 7 files changed, 138 insertions(+), 35 deletions(-)
 create mode 100644 mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid-layout.mlir

diff --git a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
index 9f7a6a638f307..6dcadee7f0fee 100644
--- a/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
+++ b/mlir/include/mlir/Dialect/Bufferization/IR/BufferizableOpInterface.h
@@ -262,6 +262,7 @@ struct BufferizationOptions {
   using AnalysisStateInitFn = std::function<void(AnalysisState &)>;
   /// Tensor-like -> Buffer-like type conversion.
   /// Parameters: tensor-like type, memory space, func op, bufferization options
+  /// Returning a null type signals conversion failure.
   using FunctionArgTypeConverterFn =
       std::function<BufferLikeType(TensorLikeType, Attribute memorySpace,
                                    func::FuncOp, const BufferizationOptions &)>;
diff --git a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
index e0a35e1932f3d..ff407bfa35251 100644
--- a/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
+++ b/mlir/lib/Dialect/Bufferization/IR/BufferizableOpInterface.cpp
@@ -358,12 +358,24 @@ BaseMemRefType getMemRefTypeWithFullyDynamicLayout(ArrayRef<int64_t> shape,
   return MemRefType::get(shape, elementType, stridedLayout, memorySpace);
 }
 
+LogicalResult verifyMemRefElementType(func::FuncOp funcOp,
+                                      TensorType tensorType) {
+  Type elementType = tensorType.getElementType();
+  if (BaseMemRefType::isValidElementType(elementType))
+    return success();
+  return funcOp.emitError() << "cannot bufferize function boundary type "
+                            << tensorType << ": element type " << elementType
+                            << " is not a valid memref element type";
+}
+
 /// Default function arg type converter: Use a fully dynamic layout map.
 BufferLikeType
 defaultFunctionArgTypeConverter(TensorLikeType type, Attribute memorySpace,
                                 func::FuncOp funcOp,
                                 const BufferizationOptions &options) {
   if (auto tensorType = mlir::dyn_cast<TensorType>(type)) {
+    if (failed(verifyMemRefElementType(funcOp, tensorType)))
+      return {};
     return cast<BufferLikeType>(
         bufferization::getMemRefTypeWithFullyDynamicLayout(tensorType,
                                                            memorySpace));
@@ -440,6 +452,8 @@ void BufferizationOptions::setFunctionBoundaryTypeConversion(
                                    func::FuncOp funcOp,
                                    const BufferizationOptions &options) {
     if (auto tensorType = mlir::dyn_cast<TensorType>(type)) {
+      if (failed(verifyMemRefElementType(funcOp, tensorType)))
+        return BufferLikeType{};
       if (layoutMapOption == LayoutMapOption::IdentityLayoutMap)
         return cast<BufferLikeType>(
             bufferization::getMemRefTypeWithStaticIdentityLayout(tensorType,
diff --git a/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
index f38de559353a8..90023bebda1bd 100644
--- a/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Bufferization/Transforms/FuncBufferizableOpInterfaceImpl.cpp
@@ -60,19 +60,6 @@ getDefaultMemorySpace(const BufferizationOptions &options,
   return nullptr;
 }
 
-static LogicalResult verifyMemRefElementType(FuncOp funcOp,
-                                             TensorLikeType type) {
-  auto tensorType = dyn_cast<TensorType>(type);
-  if (!tensorType)
-    return success();
-  Type elementType = tensorType.getElementType();
-  if (BaseMemRefType::isValidElementType(elementType))
-    return success();
-  return funcOp.emitError() << "cannot bufferize function boundary type "
-                            << tensorType << ": element type " << elementType
-                            << " is not a valid memref element type";
-}
-
 /// Return the index-th bufferized function argument type. This assumes that the
 /// specified argument is a tensor. If the tensor is ranked, a layout map may be
 /// specified by the user (as per `options.functionArgTypeConverterFn`).
@@ -83,29 +70,34 @@ getBufferizedFunctionArgType(FuncOp funcOp, int64_t index,
       dyn_cast<TensorLikeType>(funcOp.getFunctionType().getInput(index));
   assert(type && "expected TensorLikeType");
 
-  if (failed(verifyMemRefElementType(funcOp, type)))
-    return failure();
-
   // Note: For builtin tensors there is additional logic related to layout.
   if (auto tensorType = dyn_cast<TensorType>(type)) {
-    BufferLikeType memrefType = options.functionArgTypeConverterFn(
+    BufferLikeType bufferType = options.functionArgTypeConverterFn(
         type, *options.defaultMemorySpaceFn(type), funcOp, options);
+    if (!bufferType)
+      return failure();
 
     auto layoutAttr = funcOp.getArgAttrOfType<MemRefLayoutAttrInterface>(
         index, BufferizationDialect::kBufferLayoutAttrName);
     if (!layoutAttr)
-      return memrefType;
+      return bufferType;
 
-    auto rankedMemrefType = dyn_cast<MemRefType>(memrefType);
-    assert(rankedMemrefType &&
-           "buffer layout not supported on unranked tensors");
+    auto rankedMemrefType = dyn_cast<MemRefType>(bufferType);
+    if (!rankedMemrefType) {
+      funcOp.emitError() << "cannot apply buffer layout to buffer type "
+                         << bufferType;
+      return failure();
+    }
     return cast<BufferLikeType>(MemRefType::get(
         rankedMemrefType.getShape(), rankedMemrefType.getElementType(),
         layoutAttr, rankedMemrefType.getMemorySpace()));
   }
 
-  return options.functionArgTypeConverterFn(type, /*memSpace=*/nullptr, funcOp,
-                                            options);
+  BufferLikeType bufferType = options.functionArgTypeConverterFn(
+      type, /*memSpace=*/nullptr, funcOp, options);
+  if (!bufferType)
+    return failure();
+  return bufferType;
 }
 
 /// Return the FuncOp called by `callOp`.
@@ -261,11 +253,12 @@ struct CallOpInterface
 
     // Otherwise, call the type converter to compute the bufferized type.
     auto tensorType = cast<TensorLikeType>(resultType);
-    if (failed(verifyMemRefElementType(funcOp, tensorType)))
-      return failure();
-    return cast<BufferLikeType>(options.functionArgTypeConverterFn(
+    BufferLikeType bufferType = options.functionArgTypeConverterFn(
         tensorType, getDefaultMemorySpace(options, tensorType), funcOp,
-        options));
+        options);
+    if (!bufferType)
+      return failure();
+    return bufferType;
   }
 
   /// All function arguments are writable. It is the responsibility of the
@@ -337,9 +330,13 @@ struct CallOpInterface
       // something better. Insert a reallocation + copy if it cannot be
       // statically guaranteed that a direct cast would be valid.
       if (buffer.getType() != bufferType) {
-        auto memrefDstType = dyn_cast<MemRefType>(bufferType);
-        assert(memrefDstType &&
-               "buffer layout not supported on unranked tensors");
+        if (!isa<MemRefType>(buffer.getType()) ||
+            !isa<MemRefType>(bufferType)) {
+          callOp.emitError() << "cannot reconcile buffer types "
+                             << buffer.getType() << " and " << bufferType;
+          return failure();
+        }
+        auto memrefDstType = cast<MemRefType>(bufferType);
         FailureOr<Value> replacement = bufferization::castOrReallocMemRefValue(
             rewriter, buffer, memrefDstType, options);
         if (failed(replacement))
@@ -476,11 +473,11 @@ struct FuncOpInterface
     SmallVector<Type> retTypes;
     for (Type resultType : funcType.getResults()) {
       if (auto tensorType = dyn_cast<TensorLikeType>(resultType)) {
-        if (failed(verifyMemRefElementType(funcOp, tensorType)))
-          return failure();
         BufferLikeType resultType = options.functionArgTypeConverterFn(
             tensorType, getDefaultMemorySpace(options, tensorType), funcOp,
             options);
+        if (!resultType)
+          return failure();
         retTypes.push_back(resultType);
         continue;
       }
diff --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid-layout.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid-layout.mlir
new file mode 100644
index 0000000000000..b1fdb44c4193f
--- /dev/null
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid-layout.mlir
@@ -0,0 +1,29 @@
+// RUN: mlir-opt %s -test-one-shot-module-bufferize -verify-diagnostics -split-input-file
+
+#custom_buffer = #test.tensor_encoding<"custom_buffer">
+
+module {
+  // expected-error @below {{cannot apply buffer layout to buffer type '!test.test_memref<[1], !llvm.array<1 x i32>>'}}
+  // expected-error @below {{failed to bufferize op}}
+  func.func private @custom_buffer_with_layout(
+      tensor<1x!llvm.array<1 x i32>, #custom_buffer>
+        {bufferization.buffer_layout = affine_map<(d0) -> (d0)>})
+}
+
+// -----
+
+#custom_buffer = #test.tensor_encoding<"custom_buffer">
+
+module {
+  func.func private @custom_buffer_callee(
+      tensor<1xf32, #custom_buffer>)
+
+  func.func @custom_buffer_caller() {
+    %tensor = "test.create_tensor_op"() : () -> tensor<1xf32, #custom_buffer>
+    // expected-error @below {{cannot reconcile buffer types 'memref<1xf32, #test.memref_layout<"custom_buffer">>' and '!test.test_memref<[1], f32>'}}
+    // expected-error @below {{failed to bufferize op}}
+    func.call @custom_buffer_callee(%tensor)
+        : (tensor<1xf32, #custom_buffer>) -> ()
+    return
+  }
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir
index 4a2de50db1de9..410d74c2a44eb 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize-invalid.mlir
@@ -1,8 +1,16 @@
-// RUN: mlir-opt %s -test-one-shot-module-bufferize -verify-diagnostics
-// RUN: mlir-opt %s -one-shot-bufferize="bufferize-function-boundaries=1" -verify-diagnostics
+// RUN: mlir-opt %s -test-one-shot-module-bufferize -verify-diagnostics -split-input-file
+// RUN: mlir-opt %s -one-shot-bufferize="bufferize-function-boundaries=1" -verify-diagnostics -split-input-file
 
 module {
   // expected-error @below {{cannot bufferize function boundary type 'tensor<!llvm.array<1 x i32>>': element type '!llvm.array<1 x i32>' is not a valid memref element type}}
   // expected-error @below {{failed to bufferize op}}
   func.func private @sparse_csr(tensor<f64>) -> tensor<!llvm.array<1 x i32>>
 }
+
+// -----
+
+module {
+  // expected-error @below {{cannot bufferize function boundary type 'tensor<!llvm.array<1 x i32>>': element type '!llvm.array<1 x i32>' is not a valid memref element type}}
+  // expected-error @below {{failed to bufferize op}}
+  func.func private @invalid_arg(tensor<!llvm.array<1 x i32>>)
+}
diff --git a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
index b2674e9497927..a3701e45fa14c 100644
--- a/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
+++ b/mlir/test/Dialect/Bufferization/Transforms/test-one-shot-module-bufferize.mlir
@@ -92,6 +92,40 @@ func.func @custom_types_bar(%arg: !test.test_tensor<[4, 4], f64>)
   return %out : !test.test_tensor<[4, 8], f64>
 }
 
+// -----
+
+#custom_buffer = #test.tensor_encoding<"custom_buffer">
+
+// CHECK-LABEL: func.func @custom_buffer_like_callee(
+// CHECK-SAME:      %[[ARG:.*]]: !test.test_memref<[1], !llvm.array<1 x i32>>
+// CHECK-SAME:  ) -> !test.test_memref<[1], !llvm.array<1 x i32>>
+func.func @custom_buffer_like_callee(
+    %arg: tensor<1x!llvm.array<1 x i32>, #custom_buffer>)
+    -> tensor<1x!llvm.array<1 x i32>, #custom_buffer> {
+  // CHECK: %[[CALL:.*]] = call @custom_buffer_like_caller(%[[ARG]])
+  // CHECK-SAME: (!test.test_memref<[1], !llvm.array<1 x i32>>)
+  // CHECK-SAME: -> !test.test_memref<[1], !llvm.array<1 x i32>>
+  %call = func.call @custom_buffer_like_caller(%arg)
+      : (tensor<1x!llvm.array<1 x i32>, #custom_buffer>)
+      -> tensor<1x!llvm.array<1 x i32>, #custom_buffer>
+  return %call : tensor<1x!llvm.array<1 x i32>, #custom_buffer>
+}
+
+// CHECK-LABEL: func.func @custom_buffer_like_caller(
+// CHECK-SAME:      %[[ARG:.*]]: !test.test_memref<[1], !llvm.array<1 x i32>>
+// CHECK-SAME:  ) -> !test.test_memref<[1], !llvm.array<1 x i32>>
+func.func @custom_buffer_like_caller(
+    %arg: tensor<1x!llvm.array<1 x i32>, #custom_buffer>)
+    -> tensor<1x!llvm.array<1 x i32>, #custom_buffer> {
+  // CHECK: %[[CALL:.*]] = call @custom_buffer_like_callee(%[[ARG]])
+  // CHECK-SAME: (!test.test_memref<[1], !llvm.array<1 x i32>>)
+  // CHECK-SAME: -> !test.test_memref<[1], !llvm.array<1 x i32>>
+  %call = func.call @custom_buffer_like_callee(%arg)
+      : (tensor<1x!llvm.array<1 x i32>, #custom_buffer>)
+      -> tensor<1x!llvm.array<1 x i32>, #custom_buffer>
+  return %call : tensor<1x!llvm.array<1 x i32>, #custom_buffer>
+}
+
 
 // -----
 
diff --git a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
index 00eb2dc92e14f..4360352c6ea06 100644
--- a/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
+++ b/mlir/test/lib/Dialect/Bufferization/TestOneShotModuleBufferize.cpp
@@ -58,7 +58,27 @@ struct TestOneShotModuleBufferizePass
     opt.bufferizeFunctionBoundaries = true;
     opt.functionArgTypeConverterFn =
         [&](bufferization::TensorLikeType tensor, Attribute memSpace,
-            func::FuncOp, const bufferization::BufferizationOptions &options) {
+            func::FuncOp funcOp,
+            const bufferization::BufferizationOptions &options) {
+          if (auto rankedTensor = dyn_cast<RankedTensorType>(tensor)) {
+            auto encoding = dyn_cast_if_present<test::TestTensorEncodingAttr>(
+                rankedTensor.getEncoding());
+            if (encoding && encoding.getDummy().getValue() == "custom_buffer")
+              return cast<bufferization::BufferLikeType>(
+                  test::TestMemrefType::get(
+                      rankedTensor.getContext(), rankedTensor.getShape(),
+                      rankedTensor.getElementType(), memSpace));
+          }
+          if (auto tensorType = dyn_cast<TensorType>(tensor)) {
+            Type elementType = tensorType.getElementType();
+            if (!BaseMemRefType::isValidElementType(elementType)) {
+              funcOp.emitError()
+                  << "cannot bufferize function boundary type " << tensorType
+                  << ": element type " << elementType
+                  << " is not a valid memref element type";
+              return bufferization::BufferLikeType{};
+            }
+          }
           return options.unknownTypeConverterFn(tensor, memSpace, options);
         };
     opt.unknownTypeConverterFn =


From llvmlistbot at llvm.org  Tue Jul 21 22:56:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 22:56:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Reject invalid boundary types (PR
	#208787)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208787@github.com>
Message-ID: <6a605b88.d681cab9.33af1e.7cb5@mx.google.com>


================

----------------
mygitljf wrote:

Thanks! I moved the element type check into the default `functionArgTypeConverterFn` implementations and made them return a `null` type on failure. The callers now handle null results, while custom converters can still return non-memref `BufferLikeTypes`.

https://github.com/llvm/llvm-project/pull/208787

From llvmlistbot at llvm.org  Tue Jul 21 23:08:10 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Tue, 21 Jul 2026 23:08:10 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a605e4a.e624eb8d.12f676.4386@mx.google.com>


https://github.com/echristo updated https://github.com/llvm/llvm-project/pull/210850

>From ab4cecf7ddb25e85c1a7a8bbfc38743dc2125ac2 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Wed, 15 Jul 2026 08:51:28 -0700
Subject: [PATCH 1/5] [DebugInfo] Add symbolic branches to DIExpression

Added three pseudo-ops to DIExpression:

- DW_OP_LLVM_label declares a label and emits no bytes.
- DW_OP_LLVM_bra branches to a label when the top of the stack is non-zero.
- DW_OP_LLVM_skip always branches to a label.

DW_OP_bra and DW_OP_skip store a two-byte offset in [-32768, 32767], but
we don't know those offsets until we emit the expression. DIExpression keeps
label IDs, and we fix them up late in CodeGen.

CodeGen records each label's byte offset and leaves a zero placeholder for each
branch. Once it has seen the whole expression, it patches the placeholders in
the existing temporary buffers used for DIE and location-list expressions.
Both paths keep their current encoding and target byte order.

The verifier makes sure every branch has a label in the same expression and
each label ID is declared once. Labels don't need a branch, and branches can go
forward, backward, to themselves, or form cycles. Raw DW_OP_bra and DW_OP_skip
remain invalid in IR because their offsets depend on the final encoding.

Symbolic control flow only works with one location operand. We reject expressions
that combine it with DW_OP_LLVM_arg or DW_OP_LLVM_implicit_pointer, and diagnose
DIArgList records. Existing expression rewrites keep the pseudo-ops and their
IDs in place.

CodeGen reports an error if the final offset is outside [-32768, 32767], or if a
label, branch, or skip splits a deferred conversion. DWARF 5 emits DW_OP_convert
directly, but older versions may defer one conversion until they see the next.

Tests cover IR and MLIR round trips, verifier failures, forward and backward
branches, self-branches and cycles, both byte orders, conversion boundaries,
and out-of-range offsets.
---
 llvm/docs/LangRef.md                          |  16 ++
 llvm/docs/ReleaseNotes.md                     |   6 +
 llvm/docs/SourceLevelDebugging.md             |  61 ++++++-
 llvm/include/llvm/BinaryFormat/Dwarf.h        |   5 +
 llvm/lib/BinaryFormat/Dwarf.cpp               |   9 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  27 ++-
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  83 +++++++++-
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h |  27 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     |  52 +++++-
 llvm/lib/IR/DebugInfoMetadata.cpp             | 103 ++++++++++--
 llvm/lib/IR/Verifier.cpp                      |  13 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../X86/di-expression-symbolic-branches.mir   | 102 ++++++++++++
 .../di-expression-symbolic-branches.ll        |  44 +++++
 ...expression-symbolic-branch-out-of-range.py |  33 ++++
 ...ession-symbolic-branch-convert-boundary.ll |  96 +++++++++++
 ...expression-symbolic-branch-out-of-range.ll |   7 +
 .../X86/di-expression-symbolic-branches.ll    |  84 ++++++++++
 .../di-expression-symbolic-branches.ll        |  93 +++++++++++
 llvm/unittests/BinaryFormat/DwarfTest.cpp     |   7 +
 llvm/unittests/IR/MetadataTest.cpp            | 155 ++++++++++++++++++
 .../Import/debug-info-symbolic-branches.ll    |  23 +++
 .../llvmir-debug-symbolic-branches.mlir       |  19 +++
 24 files changed, 1054 insertions(+), 35 deletions(-)
 create mode 100644 llvm/test/Assembler/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Bitcode/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
 create mode 100644 llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Verifier/di-expression-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..05162290f330a 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7159,6 +7159,22 @@ Some examples of expressions:
 !DIExpression(DW_OP_constu, 42, DW_OP_stack_value)
 ```
 
+`DIExpression` uses three pseudo-ops for local control flow:
+
+```text
+DW_OP_LLVM_label, <label-id>
+DW_OP_LLVM_bra,   <label-id>
+DW_OP_LLVM_skip,  <label-id>
+```
+
+`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
+branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
+branches. Label IDs are local to the expression. Every branch needs a matching
+label, labels without branches are valid, and each ID can only be declared
+once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
+
+See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+
 ##### DIAssignID
 
 `DIAssignID` nodes have no operands and are always distinct. They are used to
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 94208f8e36d19..988740a7e5e39 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -109,6 +109,12 @@ Makes programs 10x faster by doing Special New Thing.
 
 ### Changes to the Debug Info
 
+* Added `DW_OP_LLVM_label`, `DW_OP_LLVM_bra`, and `DW_OP_LLVM_skip` for symbolic
+  branches in `DIExpression`. These operations use label IDs that CodeGen
+  resolves when it emits the expression. `DW_OP_LLVM_convert` can appear before
+  or after them, but when CodeGen can't emit `DW_OP_convert`, it reports an
+  error if a deferred conversion reaches a label, branch, or skip.
+
 ### Changes to the LLVM tools
 
 ### Changes to LLDB
diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 74bb8840a56e4..645e441e6fe2a 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -386,10 +386,9 @@ call void @llvm.dbg.assign(
 
 Debug expressions are represented as {ref}`specialized-metadata`.
 
-Debug expressions are interpreted left-to-right: start by pushing the
-value/address operand of the record onto a stack, then repeatedly push and
-evaluate opcodes from the `DIExpression` until the final variable description
-is produced.
+A debug expression starts with the record's value or address operand on the
+stack, then evaluates operations from left to right unless a symbolic branch
+jumps to a label in the same `DIExpression`.
 
 The opcodes available in these expressions are described in
 {ref}`dwarf-opcodes` and {ref}`internal-opcodes`.
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
+
+- Put labels, branches, and skips before `DW_OP_stack_value` and
+  `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
+- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
+  separately, so they can't be used with symbolic control flow.
+- Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
+  applies to every path.
+
+We don't check reachability, termination, or stack state where paths meet.
+
+`DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
+and we don't match conversions on different paths. When CodeGen can't emit
+`DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
+branch, or skip would split the pair, CodeGen reports an error.
+
+Local expression rewrites stop at labels, branches, and skips; they can still
+add operations to either end, but they don't move, remove, or copy labels.
+
+##### Other Internal Opcodes
+
 - `DW_OP_LLVM_fragment, <offset>, <size>` may appear at most once in an
   expression, and must be the last opcode. It specifies the bit offset and bit
   size of the variable fragment being described by the record or intrinsic
@@ -482,11 +525,15 @@ them.
 - `DW_OP_LLVM_convert, 16, DW_ATE_signed` specifies a bit size and encoding
   (`16` and `DW_ATE_signed` here, respectively) to which the top of the
   expression stack is to be converted. Maps into a `DW_OP_convert` operation
-  that references a base type constructed from the supplied values.
+  that references a base type constructed from the supplied values. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for how conversions work
+  with labels, branches, and skips.
 - `DW_OP_LLVM_tag_offset, tag_offset` specifies that a memory tag should be
   optionally applied to the pointer. The memory tag is derived from the given
-  tag offset in an implementation-defined manner. (This does not affect the
-  semantics of the expression containing it.)
+  tag offset in an implementation-defined manner. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for its placement with
+  labels, branches, and skips.
+  (This does not affect the semantics of the expression containing it.)
 - `DW_OP_LLVM_entry_value, N` evaluates a sub-expression as-if it were
   evaluated upon entry to the current call frame.
 
diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 75eb3a163e06f..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -149,6 +149,11 @@ enum LocationAtom {
   DW_OP_LLVM_arg = 0x1005,               ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_sext = 0x1006, ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_zext = 0x1007, ///< Only used in LLVM metadata.
+  // Labels declare IDs local to each expression, and branches use them as
+  // targets.
+  DW_OP_LLVM_label = 0x1008, ///< Only used in LLVM metadata.
+  DW_OP_LLVM_bra = 0x1009,   ///< Only used in LLVM metadata.
+  DW_OP_LLVM_skip = 0x100a,  ///< Only used in LLVM metadata.
 };
 
 enum LlvmUserLocationAtom {
diff --git a/llvm/lib/BinaryFormat/Dwarf.cpp b/llvm/lib/BinaryFormat/Dwarf.cpp
index 9ede7ebbe20ad..431d01c953768 100644
--- a/llvm/lib/BinaryFormat/Dwarf.cpp
+++ b/llvm/lib/BinaryFormat/Dwarf.cpp
@@ -159,6 +159,12 @@ StringRef llvm::dwarf::OperationEncodingString(unsigned Encoding) {
     return "DW_OP_LLVM_extract_bits_sext";
   case DW_OP_LLVM_extract_bits_zext:
     return "DW_OP_LLVM_extract_bits_zext";
+  case DW_OP_LLVM_label:
+    return "DW_OP_LLVM_label";
+  case DW_OP_LLVM_bra:
+    return "DW_OP_LLVM_bra";
+  case DW_OP_LLVM_skip:
+    return "DW_OP_LLVM_skip";
   }
 }
 
@@ -175,6 +181,9 @@ unsigned llvm::dwarf::getOperationEncoding(StringRef OperationEncodingString) {
       .Case("DW_OP_LLVM_arg", DW_OP_LLVM_arg)
       .Case("DW_OP_LLVM_extract_bits_sext", DW_OP_LLVM_extract_bits_sext)
       .Case("DW_OP_LLVM_extract_bits_zext", DW_OP_LLVM_extract_bits_zext)
+      .Case("DW_OP_LLVM_label", DW_OP_LLVM_label)
+      .Case("DW_OP_LLVM_bra", DW_OP_LLVM_bra)
+      .Case("DW_OP_LLVM_skip", DW_OP_LLVM_skip)
       .Default(0);
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfbb35d516534..45f1f13e999fc 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -191,8 +191,14 @@ void DebugLocDwarfExpression::emitUnsigned(uint64_t Value) {
   getActiveStreamer().emitULEB128(Value, Twine(Value));
 }
 
-void DebugLocDwarfExpression::emitData1(uint8_t Value) {
-  getActiveStreamer().emitInt8(Value, Twine(Value));
+void DebugLocDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    getActiveStreamer().emitInt8(Value >> (Byte * 8), Twine(Value));
+  }
 }
 
 void DebugLocDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -232,6 +238,23 @@ void DebugLocDwarfExpression::commitTemporaryBuffer() {
   TmpBuf->Comments.clear();
 }
 
+void DebugLocDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                         uint64_t Value,
+                                                         unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  assert(TmpBuf && Offset < TmpBuf->Bytes.size() &&
+         Size <= TmpBuf->Bytes.size() - Offset &&
+         "invalid temporary buffer offset");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    TmpBuf->Bytes[Offset + I] = Value >> (Byte * 8);
+    if (Offset + I < TmpBuf->Comments.size())
+      TmpBuf->Comments[Offset + I].clear();
+  }
+}
+
 const DIType *DbgVariable::getType() const {
   return getVariable()->getType();
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 9b676f88a0cff..4054817abb95d 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -13,13 +13,16 @@
 #include "DwarfExpression.h"
 #include "DwarfCompileUnit.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallBitVector.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/CodeGen/Register.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
 #include <algorithm>
 
 using namespace llvm;
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+      });
+
+  SmallVector<LabelOffset, 4> Labels;
+  SmallVector<BranchFixup, 4> Fixups;
+  // Buffer the expression until every label has a byte offset, then patch the
+  // branches.
+  if (HasSymbolicBranches)
+    enableTemporaryBuffer();
 
+  std::optional<DIExpression::ExprOperand> PrevConvertOp;
   while (ExprCursor) {
     auto Op = ExprCursor.take();
     uint64_t OpNum = Op->getOp();
@@ -545,9 +571,27 @@ bool DwarfExpression::addExpression(
     }
 
     switch (OpNum) {
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      if (PrevConvertOp)
+        report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
+                           dwarf::OperationEncodingString(OpNum) +
+                           " without DW_OP_convert support");
+      if (OpNum == dwarf::DW_OP_LLVM_label) {
+        Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
+        break;
+      }
+      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
+                                            : dwarf::DW_OP_skip);
+      Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
+      emitData2(0);
+      break;
     case dwarf::DW_OP_LLVM_arg:
       if (!InsertArg(Op->getArg(0), ExprCursor)) {
         LocationKind = Unknown;
+        if (HasSymbolicBranches)
+          disableTemporaryBuffer();
         return false;
       }
       break;
@@ -579,7 +623,10 @@ bool DwarfExpression::addExpression(
       setSubRegisterPiece(0, 0);
       // Reset the location description kind.
       LocationKind = Unknown;
-      return true;
+      if (!HasSymbolicBranches)
+        return true;
+      // Keep going so we apply the branch fixups before returning.
+      break;
     }
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext: {
@@ -744,9 +791,12 @@ bool DwarfExpression::addExpression(
       break;
     case dwarf::DW_OP_LLVM_implicit_pointer:
       // Handled in DwarfCompileUnit::emitImplicitPointerLocation for
-      // Loc::Single variables. If we reach here, the variable has a
-      // location list or other unsupported path. Drop the
-      // location rather than crashing.
+      // Loc::Single variables. If we reach here, the variable has a location
+      // list or another unsupported path, so stop emitting the expression.
+      // We buffer expressions with symbolic branches, so disable the buffer
+      // before returning.
+      if (HasSymbolicBranches)
+        disableTemporaryBuffer();
       return false;
     default:
       llvm_unreachable("unhandled opcode found in expression");
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
+          static_cast<int64_t>(Label->Offset) -
+          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+      if (!isInt<16>(Displacement))
+        report_fatal_error(Twine("DWARF expression branch offset ") +
+                           Twine(Displacement) + " is outside [-32768, 32767]");
+
+      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                  static_cast<uint16_t>(Displacement));
+    }
+
+    disableTemporaryBuffer();
+    commitTemporaryBuffer();
+  }
   return true;
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
index 2525938c78d86..f5c37140a1ef9 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
@@ -130,7 +130,11 @@ class DwarfExpression {
   /// Emit a raw unsigned value.
   virtual void emitUnsigned(uint64_t Value) = 0;
 
-  virtual void emitData1(uint8_t Value) = 0;
+  /// Emit the low 1, 2, 4, or 8 bytes of a value in the target byte order.
+  virtual void emitData(uint64_t Value, unsigned Size) = 0;
+
+  void emitData1(uint8_t Value) { emitData(Value, 1); }
+  void emitData2(uint16_t Value) { emitData(Value, 2); }
 
   virtual void emitBaseTypeRef(uint64_t Idx) = 0;
 
@@ -150,6 +154,15 @@ class DwarfExpression {
   /// Commit the data stored in the temporary buffer to the main output.
   virtual void commitTemporaryBuffer() = 0;
 
+  /// Replace a 1-, 2-, 4-, or 8-byte zero placeholder at Offset with the low
+  /// bytes of Value in the target byte order.
+  virtual void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                          unsigned Size) = 0;
+
+  void replaceTemporaryBufferData2(unsigned Offset, uint16_t Value) {
+    replaceTemporaryBufferData(Offset, Value, 2);
+  }
+
   /// Emit a normalized unsigned constant.
   void emitConstu(uint64_t Value);
 
@@ -285,10 +298,14 @@ class DwarfExpression {
 
   /// Emit all remaining operations in the DIExpressionCursor. The
   /// cursor must not contain any DW_OP_LLVM_arg operations.
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   void addExpression(DIExpressionCursor &&Expr);
 
   /// Emit all remaining operations in the DIExpressionCursor.
   /// DW_OP_LLVM_arg operations are resolved by calling (\p InsertArg).
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   //
   /// \return false if any call to (\p InsertArg) returns false.
   bool addExpression(
@@ -328,13 +345,15 @@ class DebugLocDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
@@ -358,13 +377,15 @@ class DIEDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index 78c0769e49161..c1c1218a87974 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+}
+
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
                                        DwarfCompileUnit &CU, DIELoc &DIE)
     : DwarfExpression(AP.getDwarfVersion(), CU), AP(AP), OutDIE(DIE) {}
@@ -53,8 +74,8 @@ void DIEDwarfExpression::emitUnsigned(uint64_t Value) {
   CU.addUInt(getActiveDIE(), dwarf::DW_FORM_udata, Value);
 }
 
-void DIEDwarfExpression::emitData1(uint8_t Value) {
-  CU.addUInt(getActiveDIE(), dwarf::DW_FORM_data1, Value);
+void DIEDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  CU.addUInt(getActiveDIE(), getDataForm(Size), getDataValue(Value, Size));
 }
 
 void DIEDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
+}
+
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
                                          llvm::Register MachineReg) {
   return MachineReg == TRI.getFrameRegister(*AP.MF);
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 88f7f2f6240e0..518e4c130e34c 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -1750,6 +1751,9 @@ unsigned DIExpression::ExprOperand::getSize() const {
   case dwarf::DW_OP_LLVM_tag_offset:
   case dwarf::DW_OP_LLVM_entry_value:
   case dwarf::DW_OP_LLVM_arg:
+  case dwarf::DW_OP_LLVM_label:
+  case dwarf::DW_OP_LLVM_bra:
+  case dwarf::DW_OP_LLVM_skip:
   case dwarf::DW_OP_regx:
     return 2;
   default:
@@ -1758,12 +1762,81 @@ unsigned DIExpression::ExprOperand::getSize() const {
 }
 
 bool DIExpression::isValid() const {
+  auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
+    auto FirstOp = expr_op_begin();
+    if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
+      ++FirstOp;
+    return EntryValue.get() == FirstOp->get() && EntryValue.getArg(0) == 1;
+  };
+
+  SmallDenseSet<uint64_t, 4> Labels;
+  SmallVector<uint64_t, 4> LabelReferences;
+  bool HasControlFlow = false;
+  bool HasControlFlowConflict = false;
+  bool HasStackValue = false;
+  bool HasFragment = false;
+  bool HasInvalidControlFlowSuffix = false;
+
+  // Collect labels and branch targets before running the checks below, which
+  // may return early.
   for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
     // Check that there's space for the operand.
     if (I->get() + I->getSize() > E->get())
       return false;
 
     uint64_t Op = I->getOp();
+
+    // Only DW_OP_LLVM_fragment may follow DW_OP_stack_value, and nothing may
+    // follow the fragment.
+    HasInvalidControlFlowSuffix |=
+        HasFragment || (HasStackValue && Op != dwarf::DW_OP_LLVM_fragment);
+
+    switch (Op) {
+    case dwarf::DW_OP_LLVM_label:
+      if (!Labels.insert(I->getArg(0)).second)
+        return false;
+      HasControlFlow = true;
+      break;
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      LabelReferences.push_back(I->getArg(0));
+      HasControlFlow = true;
+      break;
+    // We don't know the DW_OP_bra and DW_OP_skip offsets until CodeGen, so
+    // DIExpression uses the symbolic ops.
+    case dwarf::DW_OP_bra:
+    case dwarf::DW_OP_skip:
+      return false;
+    case dwarf::DW_OP_LLVM_tag_offset:
+      if (HasControlFlow)
+        return false;
+      break;
+    // These ops use lowering paths that don't support symbolic branches.
+    case dwarf::DW_OP_LLVM_arg:
+    case dwarf::DW_OP_LLVM_implicit_pointer:
+      HasControlFlowConflict = true;
+      break;
+    case dwarf::DW_OP_LLVM_entry_value:
+      HasControlFlowConflict |= !IsEntryValueValid(*I);
+      break;
+    default:
+      break;
+    }
+
+    if (Op == dwarf::DW_OP_stack_value)
+      HasStackValue = true;
+    else if (Op == dwarf::DW_OP_LLVM_fragment)
+      HasFragment = true;
+  }
+
+  if (HasControlFlow && (HasControlFlowConflict || HasInvalidControlFlowSuffix))
+    return false;
+  for (uint64_t Label : LabelReferences)
+    if (!Labels.contains(Label))
+      return false;
+
+  for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
+    uint64_t Op = I->getOp();
     if ((Op >= dwarf::DW_OP_reg0 && Op <= dwarf::DW_OP_reg31) ||
         (Op >= dwarf::DW_OP_breg0 && Op <= dwarf::DW_OP_breg31))
       return true;
@@ -1784,7 +1857,7 @@ bool DIExpression::isValid() const {
         return false;
       break;
     }
-    case dwarf::DW_OP_swap: {
+    case dwarf::DW_OP_swap:
       // Must be more than one implicit element on the stack.
 
       // FIXME: A better way to implement this would be to add a local variable
@@ -1795,22 +1868,20 @@ bool DIExpression::isValid() const {
       if (getNumElements() == 1)
         return false;
       break;
-    }
-    case dwarf::DW_OP_LLVM_entry_value: {
+    case dwarf::DW_OP_LLVM_entry_value:
       // An entry value operator must appear at the beginning or immediately
       // following `DW_OP_LLVM_arg 0`, and the number of operations it cover can
       // currently only be 1, because we support only entry values of a simple
       // register location. One reason for this is that we currently can't
       // calculate the size of the resulting DWARF block for other expressions.
-      auto FirstOp = expr_op_begin();
-      if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
-        ++FirstOp;
-      return I->get() == FirstOp->get() && I->getArg(0) == 1;
-    }
+      return IsEntryValueValid(*I);
     case dwarf::DW_OP_LLVM_implicit_pointer:
     case dwarf::DW_OP_LLVM_convert:
     case dwarf::DW_OP_LLVM_arg:
     case dwarf::DW_OP_LLVM_tag_offset:
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext:
     case dwarf::DW_OP_constu:
@@ -2327,13 +2398,15 @@ DIExpression *DIExpression::appendToStack(const DIExpression *Expr,
   // has no DW_OP_stack_value.
   //
   // Match .* DW_OP_stack_value (DW_OP_LLVM_fragment A B)?.
-  std::optional<FragmentInfo> FI = Expr->getFragmentInfo();
-  unsigned DropUntilStackValue = FI ? 3 : 0;
-  ArrayRef<uint64_t> ExprOpsBeforeFragment =
-      Expr->getElements().drop_back(DropUntilStackValue);
-  bool NeedsDeref = (Expr->getNumElements() > DropUntilStackValue) &&
-                    (ExprOpsBeforeFragment.back() != dwarf::DW_OP_stack_value);
-  bool NeedsStackValue = NeedsDeref || ExprOpsBeforeFragment.empty();
+  std::optional<uint64_t> LastOp;
+  // A raw operand can have the same value as an opcode, so use expr_ops().
+  // Labels and branches can't follow DW_OP_stack_value, so LastOp is still
+  // DW_OP_stack_value when the expression has a value suffix.
+  for (auto Op : Expr->expr_ops())
+    if (Op.getOp() != dwarf::DW_OP_LLVM_fragment)
+      LastOp = Op.getOp();
+  bool NeedsDeref = LastOp && *LastOp != dwarf::DW_OP_stack_value;
+  bool NeedsStackValue = NeedsDeref || !LastOp;
 
   // Append a DW_OP_deref after Expr's current op list if needed, then append
   // the new ops, and finally ensure that a single DW_OP_stack_value is present.
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..be14be6a43656 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7182,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        });
+    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
+            &DVR, MD, DVR.getExpression(), BB, F);
+  }
+
   if (DVR.isDbgAssign()) {
     CheckDI(isa_and_nonnull<DIAssignID>(DVR.getRawAssignID()),
             "invalid #dbg_assign DIAssignID", &DVR, DVR.getRawAssignID(), BB,
diff --git a/llvm/test/Assembler/di-expression-symbolic-branches.ll b/llvm/test/Assembler/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..475594c62d444
--- /dev/null
+++ b/llvm/test/Assembler/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Labels use IDs in a DIExpression, so make sure the assembler keeps both
+; forward and backward references.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42)
+!0 = !DIExpression(DW_OP_LLVM_label, 0,
+                   DW_OP_LLVM_bra, 42,
+                   DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 42)
diff --git a/llvm/test/Bitcode/di-expression-symbolic-branches.ll b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..82b3c630360fe
--- /dev/null
+++ b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | llvm-as | llvm-dis | FileCheck %s
+
+; Label IDs are uint64_t values, so use the largest one while checking that the
+; opcodes and IDs survive two bitcode round-trips.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 18446744073709551615, DW_OP_LLVM_bra, 7, DW_OP_LLVM_skip, 18446744073709551615, DW_OP_LLVM_label, 7)
+!0 = !DIExpression(DW_OP_LLVM_label, 18446744073709551615,
+                   DW_OP_LLVM_bra, 7,
+                   DW_OP_LLVM_skip, 18446744073709551615,
+                   DW_OP_LLVM_label, 7)
diff --git a/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
new file mode 100644
index 0000000000000..eda9f210cd378
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
@@ -0,0 +1,102 @@
+# RUN: split-file %s %t
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=asm -o - %t/valid.mir | FileCheck %s
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=ENTRY
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=NATIVE
+# RUN: not --crash llc -disable-verify -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t/missing-label.mir 2>&1 | FileCheck %s --check-prefix=MISSING
+
+# This checks the machine debug-expression path:
+#
+# - DWARF 4 expands conversions before we patch a location-list branch, while
+#   DWARF 5 emits DW_OP_convert.
+# - Entry values keep their branches when we emit an inline expression.
+# - With verification disabled, CodeGen still reports a missing label.
+
+#--- valid.mir
+--- |
+  define void @f() !dbg !5 {
+  entry:
+    ret void, !dbg !12
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9, !10}
+  !9 = !DILocalVariable(name: "before", scope: !5, type: !11)
+  !10 = !DILocalVariable(name: "entry", arg: 1, scope: !5, type: !11)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+  !12 = !DILocation(line: 1, scope: !5)
+...
+---
+name: f
+body: |
+  bb.0.entry:
+    $rcx = MOV64ri 1
+    DBG_VALUE $rdi, $noreg, !10, !DIExpression(DW_OP_LLVM_entry_value, 1, DW_OP_lit0, DW_OP_plus, DW_OP_LLVM_bra, 3, DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 3), debug-location !12
+    DBG_VALUE $rcx, 0, !9, !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_LLVM_convert, 64, DW_ATE_signed, DW_OP_plus_uconst, 1, DW_OP_LLVM_skip, 1), debug-location !12
+    $rcx = MOV64ri 2
+    RET64 debug-location !12
+...
+
+# CHECK: .Ldebug_loc0:
+# CHECK: .byte 114{{.*}}# DW_OP_breg2
+# CHECK-NEXT: .byte 0
+# CHECK-NEXT: .byte 18{{.*}}# DW_OP_dup
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 31
+# CHECK-NEXT: .byte 37{{.*}}# DW_OP_shr
+# CHECK-NEXT: .byte 48{{.*}}# DW_OP_lit0
+# CHECK-NEXT: .byte 32{{.*}}# DW_OP_not
+# CHECK-NEXT: .byte 30{{.*}}# DW_OP_mul
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 32
+# CHECK-NEXT: .byte 36{{.*}}# DW_OP_shl
+# CHECK-NEXT: .byte 33{{.*}}# DW_OP_or
+# CHECK-NEXT: .byte 35{{.*}}# DW_OP_plus_uconst
+# CHECK-NEXT: .byte 1
+# CHECK-NEXT: .byte 47{{.*}}# DW_OP_skip
+# CHECK-NEXT: .byte 240
+# CHECK-NEXT: .byte 255
+
+# ENTRY: DW_AT_location (DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_lit0, DW_OP_plus, DW_OP_bra +2, DW_OP_plus_uconst 0x1)
+
+# NATIVE: DW_OP_breg2 RCX+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_plus_uconst 0x1, DW_OP_skip -15
+
+# MISSING: LLVM ERROR: DWARF expression branch to label 2 has no matching label
+
+#--- missing-label.mir
+--- |
+  define void @missing_label() !dbg !5 {
+  entry:
+    ret void, !dbg !10
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "missing_label", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+  !10 = !DILocation(line: 1, scope: !5)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+...
+---
+name: missing_label
+body: |
+  bb.0.entry:
+    DBG_VALUE 0, $noreg, !9, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_bra, 2), debug-location !10
+    RET64 debug-location !10
+...
diff --git a/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..44159954127be
--- /dev/null
+++ b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
@@ -0,0 +1,44 @@
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -filetype=asm -o - %s | FileCheck %s
+
+; We patch branches in two temporary buffers, so make sure both use the PowerPC
+; byte order: a forward skip in a location list and a backward branch in an
+; inline expression.
+
+define void @f(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_bra, 2), !11)
+  #dbg_value(i64 %x, !10,
+             !DIExpression(DW_OP_LLVM_skip, 3, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_label, 3), !11)
+  call void @clobber(), !dbg !11
+  ret void, !dbg !11
+}
+
+declare void @clobber()
+
+; CHECK: .section .debug_loclists
+; CHECK: .byte 47{{.*}}# DW_OP_skip
+; CHECK-NEXT: .byte 0
+; CHECK-NEXT: .byte 2
+; CHECK: .byte 5{{.*}}# DW_AT_location
+; CHECK-NEXT: .byte 48
+; CHECK-NEXT: .byte 40
+; CHECK-NEXT: .short 65533
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "backward", scope: !5, type: !12)
+!10 = !DILocalVariable(name: "value", arg: 1, scope: !5, type: !12)
+!11 = !DILocation(line: 1, column: 1, scope: !5)
+!12 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
new file mode 100644
index 0000000000000..fc7e0246a6ae2
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
@@ -0,0 +1,33 @@
+import sys
+
+# Put 32768 one-byte DW_OP_dup operations between the branch and its label,
+# which gives us the first offset that won't fit.
+DUPLICATE_OPS = "DW_OP_dup, " * 32768
+
+sys.stdout.write(
+    f"""\
+define void @f() !dbg !5 {{
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, {DUPLICATE_OPS}
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}}
+
+!llvm.dbg.cu = !{{!0}}
+!llvm.module.flags = !{{!3}}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{{i32 2, !"Debug Info Version", i32 3}}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{{null}}
+!9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+!10 = !DILocation(line: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+"""
+)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
new file mode 100644
index 0000000000000..5cb9861502585
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
@@ -0,0 +1,96 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/skip.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=SKIP-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/skip.ll 2>&1 | FileCheck %s --check-prefix=SKIP-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/bra.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=BRA-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/bra.ll 2>&1 | FileCheck %s --check-prefix=BRA-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/label.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=LABEL-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/label.ll 2>&1 | FileCheck %s --check-prefix=LABEL-LEGACY
+
+; DWARF 5 emits DW_OP_convert directly, so labels and branches can appear
+; between conversions. With DWARF 4 we may defer one conversion until the next;
+; if a label, branch, or skip splits the pair, CodeGen reports an error.
+
+; SKIP-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_skip +5, DW_OP_convert {{.*}} "DW_ATE_signed_64")
+; SKIP-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_skip without DW_OP_convert support
+
+; BRA-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_dup, DW_OP_bra +5, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; BRA-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_bra without DW_OP_convert support
+
+; LABEL-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; LABEL-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_label without DW_OP_convert support
+
+;--- skip.ll
+define void @skip(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_skip, 1,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "skip", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "skip", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- label.ll
+define void @label(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 3), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "label", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "label", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- bra.ll
+define void @bra(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_dup,
+                           DW_OP_LLVM_bra, 2,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 2), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "bra", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "bra", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
new file mode 100644
index 0000000000000..677032dc9c7a7
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
@@ -0,0 +1,7 @@
+; RUN: %python %S/Inputs/generate-di-expression-symbolic-branch-out-of-range.py > %t.ll
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t.ll 2>&1 | FileCheck %s
+
+; The first positive offset that doesn't fit is 32768, so make sure CodeGen
+; reports it instead of truncating it.
+
+; CHECK: LLVM ERROR: DWARF expression branch offset 32768 is outside [-32768, 32767]
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..163c3c24c6927
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
@@ -0,0 +1,84 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump -v - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s --check-prefix=LEGACY
+
+; Branch offsets come from the emitted bytes, so check:
+;
+; - zero, forward, backward, and cyclic branches;
+; - ops which grow during lowering;
+; - labels on either side of an emitted op; and
+; - the fragment return path and DWARF 4 convert expansion.
+
+define void @f() !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, DW_OP_LLVM_label, 1), !18)
+  #dbg_value(i64 0, !10,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_skip, 2), !18)
+  #dbg_value(i64 0, !11,
+             !DIExpression(DW_OP_LLVM_label, 3, DW_OP_LLVM_bra, 4,
+                           DW_OP_LLVM_skip, 3, DW_OP_LLVM_label, 4), !18)
+  #dbg_value(i64 0, !12,
+             !DIExpression(DW_OP_LLVM_bra, 5, DW_OP_deref_size, 1,
+                           DW_OP_plus_uconst, 128,
+                           DW_OP_LLVM_extract_bits_sext, 4, 4,
+                           DW_OP_LLVM_label, 5), !18)
+  #dbg_value(i64 0, !13,
+             !DIExpression(DW_OP_LLVM_skip, 6,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 6), !18)
+  #dbg_value(i64 0, !14,
+             !DIExpression(DW_OP_LLVM_bra, 7, DW_OP_LLVM_label, 7,
+                           DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 32), !18)
+  #dbg_value(i64 0, !15,
+             !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 8,
+                           DW_OP_LLVM_skip, 8), !18)
+  #dbg_value(i64 0, !16,
+             !DIExpression(DW_OP_LLVM_label, 9, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_skip, 9), !18)
+  ret void, !dbg !18
+}
+
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"zero"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"backward"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +3, DW_OP_skip -6, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"cycle"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +11, DW_OP_deref_size 0x1, DW_OP_plus_uconst 0x80, DW_OP_constu 0x38, DW_OP_shl, DW_OP_constu 0x3c, DW_OP_shra, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"expanded"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip +10, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"convert"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value, DW_OP_piece 0x4)
+; CHECK: DW_AT_name{{.*}}"fragment"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_after_offset"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -5, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_before_offset"
+
+; LEGACY: DW_AT_location (DW_OP_lit0, DW_OP_skip +11, DW_OP_dup, DW_OP_constu 0x1f, DW_OP_shr, DW_OP_lit0, DW_OP_not, DW_OP_mul, DW_OP_constu 0x20, DW_OP_shl, DW_OP_or, DW_OP_stack_value)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !10, !11, !12, !13, !14, !15, !16}
+!9 = !DILocalVariable(name: "zero", scope: !5, type: !19)
+!10 = !DILocalVariable(name: "backward", scope: !5, type: !19)
+!11 = !DILocalVariable(name: "cycle", scope: !5, type: !19)
+!12 = !DILocalVariable(name: "expanded", scope: !5, type: !19)
+!13 = !DILocalVariable(name: "convert", scope: !5, type: !19)
+!14 = !DILocalVariable(name: "fragment", scope: !5, type: !19)
+!15 = !DILocalVariable(name: "label_after_offset", scope: !5, type: !19)
+!16 = !DILocalVariable(name: "label_before_offset", scope: !5, type: !19)
+!18 = !DILocation(line: 1, scope: !5)
+!19 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/Verifier/di-expression-symbolic-branches.ll b/llvm/test/Verifier/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..db0e0dacdf32f
--- /dev/null
+++ b/llvm/test/Verifier/di-expression-symbolic-branches.ll
@@ -0,0 +1,93 @@
+; RUN: split-file %s %t
+; RUN: opt -passes=verify -disable-output %t/valid.ll
+; RUN: not opt -passes=verify -disable-output %t/arity.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/duplicate.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/missing.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw-after-register.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/incompatible.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/location-arg.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/tag-ordering.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/terminal.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+
+; Check the verifier rules separately:
+;
+; - a label needs an ID;
+; - labels are unique and each branch target exists;
+; - raw branches and incompatible ops are rejected; and
+; - tag_offset stays before control flow, which stays before stack_value.
+
+; DIArgList doesn't support symbolic branches, but normal IR loading drops the
+; bad debug info, so opt still succeeds.
+; RUN: opt -passes=verify -disable-output %t/arg-list.ll 2>&1 | FileCheck %s --check-prefix=ARG-LIST
+
+; INVALID: invalid expression
+; ARG-LIST: DIArgList doesn't support symbolic branches
+; ARG-LIST: warning: ignoring invalid debug info
+
+;--- valid.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 0, DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 0)
+
+;--- arity.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label)
+
+;--- duplicate.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_label, 1)
+
+;--- missing.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 1)
+
+;--- raw.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_bra, 0)
+
+;--- raw-after-register.ll
+; A register normally ends validation, but it must not hide a raw branch later
+; in the expression.
+!named = !{!0}
+!0 = !DIExpression(DW_OP_reg0, DW_OP_skip, 0)
+
+;--- incompatible.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_implicit_pointer, DW_OP_LLVM_label, 1)
+
+;--- location-arg.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_skip, 1,
+                   DW_OP_LLVM_label, 1)
+
+;--- tag-ordering.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_tag_offset, 0)
+
+;--- terminal.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_stack_value, DW_OP_LLVM_label, 1)
+
+;--- arg-list.ll
+; The expression is valid by itself, so this diagnostic comes from DIArgList.
+define void @f(i32 %x) !dbg !4 {
+entry:
+  #dbg_value(!DIArgList(i32 %x), !7,
+             !DIExpression(DW_OP_LLVM_skip, 1, DW_OP_LLVM_label, 1), !8)
+  ret void, !dbg !8
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1, type: !5,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!5 = !DISubroutineType(types: !6)
+!6 = !{null}
+!7 = !DILocalVariable(name: "x", scope: !4)
+!8 = !DILocation(line: 1, column: 1, scope: !4)
diff --git a/llvm/unittests/BinaryFormat/DwarfTest.cpp b/llvm/unittests/BinaryFormat/DwarfTest.cpp
index ba7d59182ea53..6ffb5a921510c 100644
--- a/llvm/unittests/BinaryFormat/DwarfTest.cpp
+++ b/llvm/unittests/BinaryFormat/DwarfTest.cpp
@@ -47,6 +47,13 @@ TEST(DwarfTest, getOperationEncoding) {
   // Some valid ops.
   EXPECT_EQ(DW_OP_deref, getOperationEncoding("DW_OP_deref"));
   EXPECT_EQ(DW_OP_bit_piece, getOperationEncoding("DW_OP_bit_piece"));
+  // These are metadata-only ops, but they still need names in both directions.
+  EXPECT_EQ(DW_OP_LLVM_label, getOperationEncoding("DW_OP_LLVM_label"));
+  EXPECT_EQ(DW_OP_LLVM_bra, getOperationEncoding("DW_OP_LLVM_bra"));
+  EXPECT_EQ(DW_OP_LLVM_skip, getOperationEncoding("DW_OP_LLVM_skip"));
+  EXPECT_EQ("DW_OP_LLVM_label", OperationEncodingString(DW_OP_LLVM_label));
+  EXPECT_EQ("DW_OP_LLVM_bra", OperationEncodingString(DW_OP_LLVM_bra));
+  EXPECT_EQ("DW_OP_LLVM_skip", OperationEncodingString(DW_OP_LLVM_skip));
 
   // Invalid ops.
   EXPECT_EQ(0u, getOperationEncoding("DW_OP_otherthings"));
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 65398b5ca2b1f..9be5384f9647b 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -4251,6 +4251,86 @@ TEST_F(DIExpressionTest, Append) {
   EXPECT_EQ(ResExpr, AppendExpr);
 }
 
+TEST_F(DIExpressionTest, SymbolicBranchRewrites) {
+  // A symbolic op has an ID operand, so fold the math on both sides without
+  // consuming it.
+  for (uint64_t Barrier : {dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                           dwarf::DW_OP_LLVM_skip}) {
+    SmallVector<uint64_t> Ops = {dwarf::DW_OP_constu,
+                                 1,
+                                 dwarf::DW_OP_constu,
+                                 2,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 4,
+                                 Barrier,
+                                 7,
+                                 dwarf::DW_OP_constu,
+                                 5,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 6,
+                                 dwarf::DW_OP_constu,
+                                 7,
+                                 dwarf::DW_OP_plus};
+    SmallVector<uint64_t> Expected = {
+        dwarf::DW_OP_constu,      3, dwarf::DW_OP_constu, 4, Barrier, 7,
+        dwarf::DW_OP_plus_uconst, 5, dwarf::DW_OP_constu, 13};
+    if (Barrier != dwarf::DW_OP_LLVM_label) {
+      Ops.append({dwarf::DW_OP_LLVM_label, 7});
+      Expected.append({dwarf::DW_OP_LLVM_label, 7});
+    }
+    EXPECT_EQ(DIExpression::get(Context, Expected),
+              DIExpression::get(Context, Ops)->foldConstantMath());
+  }
+
+  // Adding ops before or after an expression leaves its labels in place.
+  SmallVector<uint64_t> Ops = {dwarf::DW_OP_LLVM_label, 7,
+                               dwarf::DW_OP_plus_uconst, 1};
+  auto *Expr = DIExpression::get(Context, Ops);
+  auto *Prepended = DIExpression::prepend(Expr, DIExpression::DerefBefore, 0);
+  SmallVector<uint64_t> Expected = {dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_label,
+                                    7, dwarf::DW_OP_plus_uconst, 1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+
+  SmallVector<uint64_t> Prefix = {dwarf::DW_OP_LLVM_tag_offset, 3};
+  Prepended = DIExpression::prependOpcodes(Expr, Prefix);
+  Expected = {dwarf::DW_OP_LLVM_tag_offset, 3, dwarf::DW_OP_LLVM_label, 7,
+              dwarf::DW_OP_plus_uconst,     1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+  EXPECT_TRUE(Prepended->isValid());
+
+  SmallVector<uint64_t> AppendOps = {
+      dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+      dwarf::DW_OP_LLVM_convert, 64, dwarf::DW_ATE_signed};
+  auto *Appended = DIExpression::append(Expr, AppendOps);
+  Expected = {dwarf::DW_OP_LLVM_label,
+              7,
+              dwarf::DW_OP_plus_uconst,
+              1,
+              dwarf::DW_OP_LLVM_convert,
+              32,
+              dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert,
+              64,
+              dwarf::DW_ATE_signed};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+
+  // Label IDs are arbitrary uint64_t values, so appendExt must not mistake one
+  // for DW_OP_stack_value.
+  uint64_t LabelID = dwarf::DW_OP_stack_value;
+  Ops = {dwarf::DW_OP_LLVM_label, LabelID};
+  Appended =
+      DIExpression::appendExt(DIExpression::get(Context, Ops), 32, 64, true);
+  Expected = {dwarf::DW_OP_LLVM_label,   LabelID, dwarf::DW_OP_deref,
+              dwarf::DW_OP_LLVM_convert, 32,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert, 64,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_stack_value};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+}
+
 TEST_F(DIExpressionTest, isValid) {
 #define EXPECT_VALID(...)                                                      \
   do {                                                                         \
@@ -4271,6 +4351,7 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_VALID(dwarf::DW_OP_constu, 6, dwarf::DW_OP_plus);
   EXPECT_VALID(dwarf::DW_OP_constu, 5, dwarf::DW_OP_swap);
   EXPECT_VALID(dwarf::DW_OP_deref);
+  EXPECT_VALID(dwarf::DW_OP_stack_value);
   EXPECT_VALID(dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_plus_uconst, 6, dwarf::DW_OP_deref);
   EXPECT_VALID(dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst, 6);
@@ -4279,12 +4360,31 @@ TEST_F(DIExpressionTest, isValid) {
                dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_VALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_entry_value, 1);
+  // A label may be unused, and branches may refer forward or backward.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 2, dwarf::DW_OP_LLVM_label, 2);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 3, dwarf::DW_OP_LLVM_label, 4);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 4, dwarf::DW_OP_LLVM_skip, 5,
+               dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_LLVM_skip, 4);
+  // Label IDs can use the full uint64_t range.
+  EXPECT_VALID(
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max() - 1,
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max(),
+      dwarf::DW_OP_LLVM_skip, std::numeric_limits<uint64_t>::max() - 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_stack_value);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 6, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_tag_offset, 1, dwarf::DW_OP_LLVM_label, 6,
+               dwarf::DW_OP_LLVM_skip, 6, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
 
   // Invalid constructions.
   EXPECT_INVALID(~0u);
   EXPECT_INVALID(dwarf::DW_OP_plus, 0);
   EXPECT_INVALID(dwarf::DW_OP_plus_uconst);
   EXPECT_INVALID(dwarf::DW_OP_swap);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_deref);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3, 7, dwarf::DW_OP_plus_uconst, 3);
@@ -4294,6 +4394,48 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_plus_uconst, 5,
                  dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_LLVM_entry_value, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip, 1);
+  EXPECT_INVALID(dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_implicit_pointer, dwarf::DW_OP_LLVM_label,
+                 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_tag_offset, 0);
+  // Converts and other stack ops can appear around labels and branches.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_skip, 7, dwarf::DW_OP_LLVM_label, 7);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 1, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_convert, 64,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_dup, dwarf::DW_OP_LLVM_bra, 1,
+               dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 0, 32, dwarf::DW_OP_LLVM_label, 1);
+  // reg0 and entry_value normally end validation early, but they must not hide
+  // invalid control flow later in the expression.
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_LLVM_bra, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_label, 1,
+                 dwarf::DW_OP_LLVM_entry_value, 1);
 
 #undef EXPECT_VALID
 #undef EXPECT_INVALID
@@ -4371,6 +4513,19 @@ TEST_F(DIExpressionTest, createFragmentExpression) {
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
+  // Creating a fragment leaves the branches alone and adds the fragment at the
+  // end.
+  SmallVector<uint64_t> ControlFlowOps = {dwarf::DW_OP_LLVM_label, 1,
+                                          dwarf::DW_OP_LLVM_bra,   1,
+                                          dwarf::DW_OP_LLVM_skip,  1};
+  DIExpression *ControlFlowExpr = DIExpression::get(Context, ControlFlowOps);
+  auto Fragment =
+      DIExpression::createFragmentExpression(ControlFlowExpr, 0, 32);
+  ASSERT_TRUE(Fragment.has_value());
+  ControlFlowOps.append({dwarf::DW_OP_LLVM_fragment, 0, 32});
+  EXPECT_EQ(DIExpression::get(Context, ControlFlowOps), *Fragment);
+  EXPECT_TRUE((*Fragment)->isValid());
+
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
new file mode 100644
index 0000000000000..7ba2b0802fa3c
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
@@ -0,0 +1,23 @@
+; RUN: mlir-translate -import-llvm -mlir-print-debuginfo -emit-expensive-warnings %s 2>&1 | FileCheck %s
+
+; Import a forward branch and a backward skip, and make sure MLIR keeps the op
+; order and label IDs.
+
+; CHECK: llvm.intr.dbg.value {{.*}} #llvm.di_expression<[DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0), DW_OP_LLVM_label(42)]> = {{.*}} : i64
+define void @f(i64 %x) !dbg !4 {
+  #dbg_value(i64 %x, !DILocalVariable(scope: !4),
+             !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42,
+                           DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42),
+             !DILocation(scope: !4))
+  ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1,
+                            type: !DISubroutineType(types: !{null}),
+                            spFlags: DISPFlagDefinition, unit: !0)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
new file mode 100644
index 0000000000000..74a549b952327
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Export a forward branch and a backward skip, and make sure LLVM IR keeps the
+// op order and label IDs.
+
+#file = #llvm.di_file<"test.c" in "/">
+#cu = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_C,
+                            file = #file>
+#sp = #llvm.di_subprogram<compileUnit = #cu, scope = #file, name = "f",
+                          subprogramFlags = "Definition",
+                          type = #llvm.di_subroutine_type<types = #llvm.di_null_type>>
+
+// CHECK: #dbg_value(i64 %{{.*}}, !{{.*}}, !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42), !{{.*}})
+llvm.func @f(%arg: i64) {
+  llvm.intr.dbg.value #llvm.di_local_variable<scope = #sp> #llvm.di_expression<[
+    DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0),
+    DW_OP_LLVM_label(42)]> = %arg : i64
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])

>From 57cf90ad67b0e05c411e79518212f90e3faa39bc Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:53:45 -0700
Subject: [PATCH 2/5] [LLDB] Handle symbolic DIExpression opcodes

These opcodes only appear in LLVM metadata, but adding them to the shared DWARF enum makes LLDB's switch incomplete. List them with the other metadata-only opcodes so a -Wswitch build stays clean.
---
 lldb/source/Expression/DWARFExpression.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lldb/source/Expression/DWARFExpression.cpp b/lldb/source/Expression/DWARFExpression.cpp
index 299dbe97d6176..73bdd18d82cae 100644
--- a/lldb/source/Expression/DWARFExpression.cpp
+++ b/lldb/source/Expression/DWARFExpression.cpp
@@ -205,6 +205,9 @@ GetOpcodeDataSize(const DataExtractor &data, const lldb::offset_t data_offset,
   case DW_OP_LLVM_arg:
   case DW_OP_LLVM_extract_bits_sext:
   case DW_OP_LLVM_extract_bits_zext:
+  case DW_OP_LLVM_label:
+  case DW_OP_LLVM_bra:
+  case DW_OP_LLVM_skip:
     break;
   // Vendor extensions:
   case DW_OP_HP_is_value:

>From e1e1ba8492cb9a32cb924caadb78ff4129cc9581 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:58:04 -0700
Subject: [PATCH 3/5] [DebugInfo] Clarify symbolic branch limitations

Separate the DIExpression validity rules from the combinations we don't handle yet. DIArgList and DW_OP_LLVM_arg need more work in expressions and expression writers, while DW_OP_LLVM_implicit_pointer takes a different emission path. Also document the final branch offset range check.
---
 llvm/docs/SourceLevelDebugging.md | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 645e441e6fe2a..834781444c607 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -492,15 +492,23 @@ Label IDs are local to an expression:
   offset.
 - Branches can go forward, backward, to themselves, or form cycles.
 
-There are a few other restrictions:
+`DIExpression` validation also checks the following:
 
 - Put labels, branches, and skips before `DW_OP_stack_value` and
   `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
-- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
-  separately, so they can't be used with symbolic control flow.
 - Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
   applies to every path.
 
+There are also a couple of cases we don't handle yet:
+
+- `DIArgList` and `DW_OP_LLVM_arg` are currently rejected. CodeGen expands each
+  argument before it resolves the label offsets, so there isn't a representation
+  problem here; it mostly needs work handling it in expressions and expression
+  writers.
+- `DW_OP_LLVM_implicit_pointer` bypasses normal expression emission and only
+  handles a single location today. Supporting branches there is a bit more
+  work, since we'll need to work it back into our normal emission order.
+
 We don't check reachability, termination, or stack state where paths meet.
 
 `DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
@@ -508,6 +516,9 @@ and we don't match conversions on different paths. When CodeGen can't emit
 `DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
 branch, or skip would split the pair, CodeGen reports an error.
 
+CodeGen also reports an error if the final branch offset is outside
+`[-32768, 32767]`.
+
 Local expression rewrites stop at labels, branches, and skips; they can still
 add operations to either end, but they don't move, remove, or copy labels.
 

>From 7d3fcc07c09bcf1d8150f79394720988d8810aae Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 14:14:38 -0700
Subject: [PATCH 4/5] [DebugInfo] Clean up symbolic DIExpression branch
 handling

Pull the symbolic branch checks into two small helpers, since CodeGen only needs branches while the verifier also needs labels.

Use maskTrailingOnes when storing fixed-width DIE values, and make the placeholder walk use a byte cursor with names that describe what we're patching. We still keep the existing form and size so later fixup offsets don't move.

Also make the displacement calculation and DIArgList verifier check a little clearer. There should be no behavior change here.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h        | 10 +++++
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  5 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     | 42 ++++++++++---------
 llvm/lib/IR/Verifier.cpp                      | 16 +++----
 4 files changed, 44 insertions(+), 29 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index c3cff08eccb5c..177c50be7b374 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,6 +1108,16 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
+/// Return true if Op is a symbolic branch to a label.
+inline bool isSymbolicBranchOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
+}
+
+/// Return true if Op is a symbolic label or branch.
+inline bool isSymbolicControlFlowOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
+}
+
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 4054817abb95d..f90edb1e68157 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,8 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        return dwarf::isSymbolicBranchOp(Op.getOp());
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -816,6 +815,8 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("DWARF expression branch to label ") +
                            Twine(Fixup.LabelID) + " has no matching label");
 
+      // DW_OP_bra and DW_OP_skip apply the displacement after reading their
+      // two-byte operand, so use the byte after the placeholder as the base.
       int64_t Displacement =
           static_cast<int64_t>(Label->Offset) -
           static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index c1c1218a87974..ea2a6ddca3c61 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -27,6 +27,7 @@
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/Support/Casting.h"
+#include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include <cassert>
 #include <cstdint>
@@ -53,9 +54,7 @@ static dwarf::Form getDataForm(unsigned Size) {
 }
 
 static uint64_t getDataValue(uint64_t Value, unsigned Size) {
-  if (Size == sizeof(Value))
-    return Value;
-  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+  return Value & maskTrailingOnes<uint64_t>(Size * 8);
 }
 
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
@@ -98,26 +97,31 @@ unsigned DIEDwarfExpression::getTemporaryBufferSize() {
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
-void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
-                                                    uint64_t Value,
-                                                    unsigned Size) {
-  dwarf::Form Form = getDataForm(Size);
-  // Keep the form so replacing the value doesn't move later labels.
-  unsigned CurrentOffset = 0;
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned PlaceholderOffset,
+                                                    uint64_t Replacement,
+                                                    unsigned PlaceholderSize) {
+  // Walk the encoded values until the cursor reaches the placeholder.
+  unsigned ByteCursor = 0;
   for (DIEValue &V : TmpDIE.values()) {
     unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
-    if (Offset < CurrentOffset + ValueSize) {
-      assert(Offset == CurrentOffset && ValueSize == Size &&
-             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
-             V.getDIEInteger().getValue() == 0 &&
-             "symbolic branch fixup does not match its placeholder");
-      V = DIEValue(V.getAttribute(), V.getForm(),
-                   DIEInteger(getDataValue(Value, Size)));
-      return;
+    unsigned ValueEnd = ByteCursor + ValueSize;
+    if (ValueEnd <= PlaceholderOffset) {
+      ByteCursor = ValueEnd;
+      continue;
     }
-    CurrentOffset += ValueSize;
+
+    // Replace the whole zero placeholder and keep its form, since changing its
+    // encoded size would move every later fixup.
+    assert(PlaceholderOffset == ByteCursor && ValueSize == PlaceholderSize &&
+           V.getType() == DIEValue::isInteger &&
+           V.getForm() == getDataForm(PlaceholderSize) &&
+           V.getDIEInteger().getValue() == 0 &&
+           "symbolic branch fixup does not match its placeholder");
+    V = DIEValue(V.getAttribute(), V.getForm(),
+                 DIEInteger(getDataValue(Replacement, PlaceholderSize)));
+    return;
   }
-  llvm_unreachable("invalid temporary DIE offset");
+  llvm_unreachable("temporary DIE placeholder not found");
 }
 
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index be14be6a43656..0dc59163fa864 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7184,15 +7184,15 @@ void Verifier::visit(DbgVariableRecord &DVR) {
 
   // A DIArgList can have a valid branch expression which doesn't use
   // DW_OP_LLVM_arg, so check the record as well.
-  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
-    bool HasControlFlow = llvm::any_of(
-        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+  const DIExpression *Expr = DVR.getExpression();
+  if (DVR.hasArgList() && Expr->isValid()) {
+    bool HasSymbolicControlFlow =
+        llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return dwarf::isSymbolicControlFlowOp(Op.getOp());
         });
-    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
-            &DVR, MD, DVR.getExpression(), BB, F);
+    CheckDI(!HasSymbolicControlFlow,
+            "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,
+            F);
   }
 
   if (DVR.isDbgAssign()) {

>From c173ccf8a312d19de4b574813c9df6f2cda66671 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 18:17:17 -0700
Subject: [PATCH 5/5] [DebugInfo] Add opcode queries to DIExpression operands

Add is and isOneOf to DIExpression::ExprOperand and use them for the
symbolic branch checks. This keeps the checks on the parsed operation and
lets us remove the branch-specific helpers from Dwarf.h.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h          | 10 ----------
 llvm/include/llvm/IR/DebugInfoMetadata.h        |  9 +++++++++
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp |  8 ++++----
 llvm/lib/IR/Verifier.cpp                        |  3 ++-
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 177c50be7b374..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,16 +1108,6 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
-/// Return true if Op is a symbolic branch to a label.
-inline bool isSymbolicBranchOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
-}
-
-/// Return true if Op is a symbolic label or branch.
-inline bool isSymbolicControlFlowOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
-}
-
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 846342bee6071..2d0d6a1e228cb 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -3539,6 +3539,15 @@ class DIExpression : public MDNode {
     /// Get the operand code.
     uint64_t getOp() const { return *Op; }
 
+    /// Return true if this is \p Opcode.
+    bool is(uint64_t Opcode) const { return getOp() == Opcode; }
+
+    /// Return true if this is one of \p Opcodes.
+    template <typename... Ts> bool isOneOf(Ts... Opcodes) const {
+      static_assert(sizeof...(Ts) > 0, "requires at least one opcode");
+      return (is(Opcodes) || ...);
+    }
+
     /// Get an argument to the operand.
     ///
     /// Never returns the operand itself.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index f90edb1e68157..6bf604aae814c 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,7 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return dwarf::isSymbolicBranchOp(Op.getOp());
+        return Op.isOneOf(dwarf::DW_OP_LLVM_bra, dwarf::DW_OP_LLVM_skip);
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -577,12 +577,12 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
                            dwarf::OperationEncodingString(OpNum) +
                            " without DW_OP_convert support");
-      if (OpNum == dwarf::DW_OP_LLVM_label) {
+      if (Op->is(dwarf::DW_OP_LLVM_label)) {
         Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
         break;
       }
-      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
-                                            : dwarf::DW_OP_skip);
+      emitOp(Op->is(dwarf::DW_OP_LLVM_bra) ? dwarf::DW_OP_bra
+                                           : dwarf::DW_OP_skip);
       Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
       emitData2(0);
       break;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 0dc59163fa864..48e7f88d9c07a 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7188,7 +7188,8 @@ void Verifier::visit(DbgVariableRecord &DVR) {
   if (DVR.hasArgList() && Expr->isValid()) {
     bool HasSymbolicControlFlow =
         llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return dwarf::isSymbolicControlFlowOp(Op.getOp());
+          return Op.isOneOf(dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                            dwarf::DW_OP_LLVM_skip);
         });
     CheckDI(!HasSymbolicControlFlow,
             "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,


From llvmlistbot at llvm.org  Tue Jul 21 23:22:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 23:22:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a6061a1.743400b0.25573c.640d@mx.google.com>


https://github.com/1sgtpepper edited https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Tue Jul 21 23:22:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 23:22:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a6061a3.d681cab9.33af1e.cb47@mx.google.com>


https://github.com/1sgtpepper ready_for_review https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Tue Jul 21 23:23:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Tue, 21 Jul 2026 23:23:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Preserve fast-math flags when
 lowering floating casts (PR #211173)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211173@github.com>
Message-ID: <6a6061c6.bc28b987.1c0bcd.514c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: 1sgtpepper

<details>
<summary>Changes</summary>

Fixes #<!-- -->156692.

Convert Arith fast-math flags to LLVM fast-math flags when lowering
`arith.extf` and unconstrained `arith.truncf`. Previously these two patterns
passed the source-dialect attribute through unchanged, producing an Arith
attribute on an LLVM operation.

The constrained `arith.truncf` path remains unchanged.

Tests cover fast-math flag conversion for both floating-point casts.

Checks:
- Fork CI: `mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir`
  (https://github.com/1sgtpepper/llvm-project/actions/runs/29861025327)


---
Full diff: https://github.com/llvm/llvm-project/pull/211173.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp (+5-4) 
- (modified) mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir (+10-7) 


``````````diff
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..619cb0b741812 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -113,9 +113,10 @@ using DivSIOpLowering =
     VectorConvertToLLVMPattern<arith::DivSIOp, LLVM::SDivOp>;
 using DivUIOpLowering =
     VectorConvertToLLVMPattern<arith::DivUIOp, LLVM::UDivOp>;
-using ExtFOpLowering = VectorConvertToLLVMPattern<arith::ExtFOp, LLVM::FPExtOp,
-                                                  AttrConvertPassThrough,
-                                                  /*FailOnUnsupportedFP=*/true>;
+using ExtFOpLowering =
+    VectorConvertToLLVMPattern<arith::ExtFOp, LLVM::FPExtOp,
+                               arith::AttrConvertFastMathToLLVM,
+                               /*FailOnUnsupportedFP=*/true>;
 using ExtSIOpLowering =
     VectorConvertToLLVMPattern<arith::ExtSIOp, LLVM::SExtOp>;
 using ExtUIOpLowering =
@@ -202,7 +203,7 @@ using SubIOpLowering =
 using TruncFOpLowering =
     ConstrainedVectorConvertToLLVMPattern<arith::TruncFOp, LLVM::FPTruncOp,
                                           /*HasRoundingMode=*/false,
-                                          AttrConvertPassThrough,
+                                          arith::AttrConvertFastMathToLLVM,
                                           /*FailOnUnsupportedFP=*/true>;
 using ConstrainedTruncFOpLowering = ConstrainedVectorConvertToLLVMPattern<
     arith::TruncFOp, LLVM::ConstrainedFPTruncIntr, /*HasRoundingMode=*/true,
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d8f7cfcaded17 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -866,18 +866,22 @@ func.func @ops_supporting_fastmath(%arg0: f32, %arg1: f32, %arg2: i32) {
   %0 = arith.addf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fdiv %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
   %1 = arith.divf %arg0, %arg1 fastmath<fast> : f32
+// CHECK: llvm.fpext %arg0 fastmath<fast> : f32 to f64
+  %2 = arith.extf %arg0 fastmath<fast> : f32 to f64
 // CHECK: llvm.intr.maximum(%arg0, %arg1) {fastmathFlags = #llvm.fastmath<fast>} : (f32, f32) -> f32
-  %2 = arith.maximumf %arg0, %arg1 fastmath<fast> : f32
+  %3 = arith.maximumf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.intr.minimum(%arg0, %arg1) {fastmathFlags = #llvm.fastmath<fast>} : (f32, f32) -> f32
-  %3 = arith.minimumf %arg0, %arg1 fastmath<fast> : f32
+  %4 = arith.minimumf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fmul %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %4 = arith.mulf %arg0, %arg1 fastmath<fast> : f32
+  %5 = arith.mulf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fneg %arg0  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %5 = arith.negf %arg0 fastmath<fast> : f32
+  %6 = arith.negf %arg0 fastmath<fast> : f32
 // CHECK: llvm.frem %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %6 = arith.remf %arg0, %arg1 fastmath<fast> : f32
+  %7 = arith.remf %arg0, %arg1 fastmath<fast> : f32
 // CHECK: llvm.fsub %arg0, %arg1  {fastmathFlags = #llvm.fastmath<fast>} : f32
-  %7 = arith.subf %arg0, %arg1 fastmath<fast> : f32
+  %8 = arith.subf %arg0, %arg1 fastmath<fast> : f32
+// CHECK: llvm.fptrunc %arg0 fastmath<fast> : f32 to f16
+  %9 = arith.truncf %arg0 fastmath<fast> : f32 to f16
   return
 }
 
@@ -981,4 +985,3 @@ func.func @supported_fp_type(%arg0: f32, %arg1: vector<4xf32>, %arg2: vector<4x8
   %3 = arith.cmpf oeq, %arg0, %arg3 : f32
   return
 }
-

``````````

</details>


https://github.com/llvm/llvm-project/pull/211173

From llvmlistbot at llvm.org  Wed Jul 22 00:17:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 00:17:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add TypeConverter materialization;
 use a status enum for the type conversion callback (PR #208934)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208934@github.com>
Message-ID: <6a606e91.e55dc341.dbc09.ac00@mx.google.com>


https://github.com/PragmaTwice approved this pull request.


https://github.com/llvm/llvm-project/pull/208934

From llvmlistbot at llvm.org  Wed Jul 22 01:02:22 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:02:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
Message-ID: <llvm.org/llvm/llvm-project/pull/211196@github.com>


https://github.com/Brythzz created https://github.com/llvm/llvm-project/pull/211196

When updating operators from undef (deprecated) to poison, the associated comment wasn't updated accordingly. This PR fixes it.

>From c61349bfc954d5d47c67790c2a1e8fcbf53e0079 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Matt=C3=A9o=20Rizza=20Murgier?=
 <matteo.rizza-murgier at sipearl.com>
Date: Mon, 20 Jul 2026 11:31:01 +0200
Subject: [PATCH] [mlir][VectorToLLVM] Fix comment still referring to undef
 instead of poison (NFC)

---
 mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
index 1720fd955a4e7..54c117bae476b 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
@@ -1821,7 +1821,7 @@ struct VectorBroadcastScalarToNdLowering
       return rewriter.notifyMatchFailure(
           broadcast, "broadcast to 1-d or 0-d handled elsewhere");
 
-    // First insert it into an undef vector so we can shuffle it.
+    // First insert it into a poison vector so we can shuffle it.
     auto loc = broadcast.getLoc();
     auto vectorTypeInfo =
         LLVM::detail::extractNDVectorTypeInfo(resultType, *getTypeConverter());


From llvmlistbot at llvm.org  Wed Jul 22 01:02:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:02:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211196@github.com>
Message-ID: <6a60791e.824e11a6.322560.db71@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @Brythzz :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/211196

From llvmlistbot at llvm.org  Wed Jul 22 01:03:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:03:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211196@github.com>
Message-ID: <6a607948.e55dc341.dbc09.4897@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Bryth (Brythzz)

<details>
<summary>Changes</summary>

When updating operators from undef (deprecated) to poison, the associated comment wasn't updated accordingly. This PR fixes it.

---
Full diff: https://github.com/llvm/llvm-project/pull/211196.diff


1 Files Affected:

- (modified) mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp (+1-1) 


``````````diff
diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
index 1720fd955a4e7..54c117bae476b 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
@@ -1821,7 +1821,7 @@ struct VectorBroadcastScalarToNdLowering
       return rewriter.notifyMatchFailure(
           broadcast, "broadcast to 1-d or 0-d handled elsewhere");
 
-    // First insert it into an undef vector so we can shuffle it.
+    // First insert it into a poison vector so we can shuffle it.
     auto loc = broadcast.getLoc();
     auto vectorTypeInfo =
         LLVM::detail::extractNDVectorTypeInfo(resultType, *getTypeConverter());

``````````

</details>


https://github.com/llvm/llvm-project/pull/211196

From llvmlistbot at llvm.org  Wed Jul 22 01:49:22 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Wed, 22 Jul 2026 01:49:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
Message-ID: <llvm.org/llvm/llvm-project/pull/211203@github.com>


https://github.com/bruteforceboy created https://github.com/llvm/llvm-project/pull/211203

per [#210738](https://github.com/llvm/llvm-project/pull/210738#discussion_r3622744153)

Currently, the following crashes on verification with a segfault:
```
func.func @broadcast_rank0_tensor(%arg0: tensor<i32>, %arg1: tensor<32x2xi32>) -> tensor<32x2xi32> {
  %0 = linalg.broadcast ins(%arg0 : tensor<i32>) outs(%arg1 : tensor<32x2xi32>) dimensions = [0, 0]
  return %0 : tensor<32x2xi32>
}
```

the bug was found on 0D inputs but extends to other ranks. The verifier should reject cases like this where we have a duplicate dimension. I've also added a couple of tests covering `linalg.broadcast` on various 0D input scenarios to confirm they are being handled properly.


>From 19475b38b64fc264dca3ea3dac589affafe74bbb Mon Sep 17 00:00:00 2001
From: workwilson <ogbonnachibuoyim12 at gmail.com>
Date: Wed, 22 Jul 2026 16:37:17 +0800
Subject: [PATCH] [MLIR][Linalg] Fix crash on duplicate dimensions in
 linalg.broadcast

---
 .../Dialect/Linalg/IR/LinalgStructuredOps.td  |  5 ++
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      |  4 ++
 mlir/test/Dialect/Linalg/invalid.mlir         | 37 ++++++++++++
 mlir/test/Dialect/Linalg/roundtrip.mlir       | 60 +++++++++++++++++++
 4 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
index fc5c9770d969b..ddb4f6675103e 100644
--- a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
+++ b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
@@ -478,6 +478,11 @@ def BroadcastOp : LinalgStructuredBase_Op<"broadcast", [
   let description = [{
     Broadcast the input into the given shape by adding `dimensions`.
 
+    Each index in `dimensions` attribute refers to a dimension of the init
+    tensor that is added by the operation. The indices must be unique and
+    within the init rank; the sizes of the remaining (non-added) init
+    dimensions must match the input shape.
+
     Example:
     ```mlir
       %bcast = linalg.broadcast
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..170e1edf8a55d 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -2356,6 +2356,10 @@ LogicalResult BroadcastOp::verify() {
                            << initRank - 1 << "], got: " << dim;
   }
 
+  DenseSet<int64_t> uniquedDims(llvm::from_range, dimensionsRef);
+  if (uniquedDims.size() != dimensionsRef.size())
+    return emitOpError() << "dimensions should not contain duplicates";
+
   // Mapping from input dims to init dims.
   SmallVector<int64_t> dimMap;
   for (auto dim : llvm::seq<int64_t>(0, initRank)) {
diff --git a/mlir/test/Dialect/Linalg/invalid.mlir b/mlir/test/Dialect/Linalg/invalid.mlir
index c52163e244ea2..45bd8b87c0970 100644
--- a/mlir/test/Dialect/Linalg/invalid.mlir
+++ b/mlir/test/Dialect/Linalg/invalid.mlir
@@ -1196,6 +1196,43 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank1(
+    %input: tensor<4xf32>, %init: tensor<4x8x16xf32>) -> tensor<4x8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4xf32>)
+      outs(%init:tensor<4x8x16xf32>)
+      dimensions = [1, 1]
+  func.return %bcast : tensor<4x8x16xf32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank2(
+    %input: tensor<4x8xf32>, %init: tensor<4x8x16x32xf32>)
+    -> tensor<4x8x16x32xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4x8xf32>)
+      outs(%init:tensor<4x8x16x32xf32>)
+      dimensions = [3, 3]
+  func.return %bcast : tensor<4x8x16x32xf32>
+}
+
+// -----
+
 func.func @broadcast_no_operands1() {
   // expected-error @+1 {{'linalg.broadcast' op expected 2 operands, but found 0}}
   linalg.broadcast dimensions = [1]
diff --git a/mlir/test/Dialect/Linalg/roundtrip.mlir b/mlir/test/Dialect/Linalg/roundtrip.mlir
index bfb92c3289a49..f2584f7b2ecec 100644
--- a/mlir/test/Dialect/Linalg/roundtrip.mlir
+++ b/mlir/test/Dialect/Linalg/roundtrip.mlir
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_2d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_to_1d(%input: tensor<f32>,
+                              %init: tensor<64xf32>) -> tensor<64xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<64xf32>)
+      dimensions = [0]
+  func.return %bcast : tensor<64xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_1d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<64xf32>)
+// CHECK-SAME:    dimensions = [0]
+
+// -----
+
+func.func @broadcast_0d_with_dynamic_sizes(
+              %input: tensor<f32>, %init: tensor<8x?xf32>) -> tensor<8x?xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<8x?xf32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<8x?xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_with_dynamic_sizes
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<8x?xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_memref(%input: memref<f32>,
+                               %init: memref<8x16xf32>) {
+  linalg.broadcast
+      ins(%input:memref<f32>)
+      outs(%init:memref<8x16xf32>)
+      dimensions = [0, 1]
+  func.return
+}
+// CHECK-LABEL: func @broadcast_0d_memref
+//      CHECK:    linalg.broadcast ins(%{{.*}} : memref<f32>)
+// CHECK-SAME:    outs(%{{.*}} : memref<8x16xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
 func.func @map_arith_with_attr(%lhs: tensor<64xf32>, %rhs: tensor<64xf32>,
                       %init: tensor<64xf32>) -> tensor<64xf32> {
   %add = linalg.map


From llvmlistbot at llvm.org  Wed Jul 22 01:50:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:50:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608439.13be6df8.323b61.4cb7@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-linalg

@llvm/pr-subscribers-mlir

Author: Chibuoyim (Wilson) Ogbonna (bruteforceboy)

<details>
<summary>Changes</summary>

per [#<!-- -->210738](https://github.com/llvm/llvm-project/pull/210738#discussion_r3622744153)

Currently, the following crashes on verification with a segfault:
```
func.func @<!-- -->broadcast_rank0_tensor(%arg0: tensor&lt;i32&gt;, %arg1: tensor&lt;32x2xi32&gt;) -&gt; tensor&lt;32x2xi32&gt; {
  %0 = linalg.broadcast ins(%arg0 : tensor&lt;i32&gt;) outs(%arg1 : tensor&lt;32x2xi32&gt;) dimensions = [0, 0]
  return %0 : tensor&lt;32x2xi32&gt;
}
```

the bug was found on 0D inputs but extends to other ranks. The verifier should reject cases like this where we have a duplicate dimension. I've also added a couple of tests covering `linalg.broadcast` on various 0D input scenarios to confirm they are being handled properly.


---
Full diff: https://github.com/llvm/llvm-project/pull/211203.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td (+5) 
- (modified) mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp (+4) 
- (modified) mlir/test/Dialect/Linalg/invalid.mlir (+37) 
- (modified) mlir/test/Dialect/Linalg/roundtrip.mlir (+60) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
index fc5c9770d969b..ddb4f6675103e 100644
--- a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
+++ b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
@@ -478,6 +478,11 @@ def BroadcastOp : LinalgStructuredBase_Op<"broadcast", [
   let description = [{
     Broadcast the input into the given shape by adding `dimensions`.
 
+    Each index in `dimensions` attribute refers to a dimension of the init
+    tensor that is added by the operation. The indices must be unique and
+    within the init rank; the sizes of the remaining (non-added) init
+    dimensions must match the input shape.
+
     Example:
     ```mlir
       %bcast = linalg.broadcast
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..170e1edf8a55d 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -2356,6 +2356,10 @@ LogicalResult BroadcastOp::verify() {
                            << initRank - 1 << "], got: " << dim;
   }
 
+  DenseSet<int64_t> uniquedDims(llvm::from_range, dimensionsRef);
+  if (uniquedDims.size() != dimensionsRef.size())
+    return emitOpError() << "dimensions should not contain duplicates";
+
   // Mapping from input dims to init dims.
   SmallVector<int64_t> dimMap;
   for (auto dim : llvm::seq<int64_t>(0, initRank)) {
diff --git a/mlir/test/Dialect/Linalg/invalid.mlir b/mlir/test/Dialect/Linalg/invalid.mlir
index c52163e244ea2..45bd8b87c0970 100644
--- a/mlir/test/Dialect/Linalg/invalid.mlir
+++ b/mlir/test/Dialect/Linalg/invalid.mlir
@@ -1196,6 +1196,43 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank1(
+    %input: tensor<4xf32>, %init: tensor<4x8x16xf32>) -> tensor<4x8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4xf32>)
+      outs(%init:tensor<4x8x16xf32>)
+      dimensions = [1, 1]
+  func.return %bcast : tensor<4x8x16xf32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank2(
+    %input: tensor<4x8xf32>, %init: tensor<4x8x16x32xf32>)
+    -> tensor<4x8x16x32xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4x8xf32>)
+      outs(%init:tensor<4x8x16x32xf32>)
+      dimensions = [3, 3]
+  func.return %bcast : tensor<4x8x16x32xf32>
+}
+
+// -----
+
 func.func @broadcast_no_operands1() {
   // expected-error @+1 {{'linalg.broadcast' op expected 2 operands, but found 0}}
   linalg.broadcast dimensions = [1]
diff --git a/mlir/test/Dialect/Linalg/roundtrip.mlir b/mlir/test/Dialect/Linalg/roundtrip.mlir
index bfb92c3289a49..f2584f7b2ecec 100644
--- a/mlir/test/Dialect/Linalg/roundtrip.mlir
+++ b/mlir/test/Dialect/Linalg/roundtrip.mlir
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_2d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_to_1d(%input: tensor<f32>,
+                              %init: tensor<64xf32>) -> tensor<64xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<64xf32>)
+      dimensions = [0]
+  func.return %bcast : tensor<64xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_1d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<64xf32>)
+// CHECK-SAME:    dimensions = [0]
+
+// -----
+
+func.func @broadcast_0d_with_dynamic_sizes(
+              %input: tensor<f32>, %init: tensor<8x?xf32>) -> tensor<8x?xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<8x?xf32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<8x?xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_with_dynamic_sizes
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<8x?xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_memref(%input: memref<f32>,
+                               %init: memref<8x16xf32>) {
+  linalg.broadcast
+      ins(%input:memref<f32>)
+      outs(%init:memref<8x16xf32>)
+      dimensions = [0, 1]
+  func.return
+}
+// CHECK-LABEL: func @broadcast_0d_memref
+//      CHECK:    linalg.broadcast ins(%{{.*}} : memref<f32>)
+// CHECK-SAME:    outs(%{{.*}} : memref<8x16xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
 func.func @map_arith_with_attr(%lhs: tensor<64xf32>, %rhs: tensor<64xf32>,
                       %init: tensor<64xf32>) -> tensor<64xf32> {
   %add = linalg.map

``````````

</details>


https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 01:56:19 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Wed, 22 Jul 2026 01:56:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a6085b3.c56fb102.6921f.1f6b@mx.google.com>


https://github.com/arun-thmn ready_for_review https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 01:56:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:56:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a6085db.e5907290.33b622.1ba9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Arun Thangamani (arun-thmn)

<details>
<summary>Changes</summary>

This path transforms `vector.contract(A, B, Acc)` into `vector.contract(A, B, 0) + Acc`  to decouple the contraction computation from the accumulator update for contractions over the loop.

---

Patch is 23.89 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/204327.diff


10 Files Affected:

- (modified) mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td (+10) 
- (modified) mlir/include/mlir/Dialect/X86/Transforms.h (+5) 
- (modified) mlir/include/mlir/Dialect/X86/Utils/X86Utils.h (+7-3) 
- (modified) mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp (+5) 
- (modified) mlir/lib/Dialect/X86/Transforms/CMakeLists.txt (+1) 
- (added) mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp (+114) 
- (modified) mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp (+12-16) 
- (modified) mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp (+17-13) 
- (modified) mlir/lib/Dialect/X86/Utils/X86Utils.cpp (+44-19) 
- (added) mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir (+136) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
index c474cfb47d003..48aabf90914e2 100644
--- a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
+++ b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
@@ -82,5 +82,15 @@ def ApplyVectorContractToAMXDotProductPatternsOp : Op<Transform_Dialect,
   let assemblyFormat = "attr-dict";
 }
 
+def ApplyMoveAccumulatorForContractLoopPatternsOp : Op<Transform_Dialect,
+    "apply_patterns.x86.move_accumulator_for_contract_loop",
+    [DeclareOpInterfaceMethods<PatternDescriptorOpInterface>]> {
+  let description = [{
+         Rewrites an additive vector contraction by zeroing the accumulator input
+         and materializing the accumulation as a separate add after the contraction.
+  }];
+
+  let assemblyFormat = "attr-dict";
+}
 
 #endif // X86_TRANSFORM_OPS
diff --git a/mlir/include/mlir/Dialect/X86/Transforms.h b/mlir/include/mlir/Dialect/X86/Transforms.h
index 6ebba5e94ec7c..8bcaf9a67feb0 100644
--- a/mlir/include/mlir/Dialect/X86/Transforms.h
+++ b/mlir/include/mlir/Dialect/X86/Transforms.h
@@ -110,6 +110,11 @@ void populateShuffleVectorFMAOpsPatterns(RewritePatternSet &patterns);
 // Int8).
 void populateVectorContractToAMXDotProductPatterns(RewritePatternSet &patterns);
 
+// Rewrites an additive vector contraction by zeroing the accumulator input
+// and materializing the accumulation as a separate add after the contraction.
+void populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns);
+
 //===----------------------------------------------------------------------===//
 /// Helpers extracted from:
 ///   - clang/lib/Headers/avxintrin.h
diff --git a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
index 2ff8f4f4283a2..f067ce3bb9b65 100644
--- a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
+++ b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
@@ -29,6 +29,10 @@ namespace x86 {
 bool isInVnniLayout(Operation *op, llvm::ArrayRef<AffineMap> indexingMaps,
                     std::optional<unsigned> blockingFactor = std::nullopt);
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v);
+
 // Returns true if two contraction ops form a valid pair for VNNI packing.
 // It verifies that both contractions share the appropriate operand, read from
 // the same source buffer, and use constant indices that differ by 8 or 16.
@@ -59,9 +63,9 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc, VectorType accTy);
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
+                                       VectorType accTy);
 
 } // namespace x86
 } // namespace mlir
diff --git a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
index 390b21e12b0ed..a52f08a5cd061 100644
--- a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
+++ b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
@@ -52,6 +52,11 @@ void mlir::transform::ApplyVectorContractToAMXDotProductPatternsOp::
   x86::populateVectorContractToAMXDotProductPatterns(patterns);
 }
 
+void mlir::transform::ApplyMoveAccumulatorForContractLoopPatternsOp::
+    populatePatterns(RewritePatternSet &patterns) {
+  x86::populateMoveAccumulatorForContractLoopPatterns(patterns);
+}
+
 //===----------------------------------------------------------------------===//
 // Transform op registration
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
index 9c3695536cda9..5118563c03f16 100644
--- a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
@@ -7,6 +7,7 @@ add_mlir_dialect_library(MLIRX86Transforms
   SinkVectorProducerOps.cpp
   ShuffleVectorFMAOps.cpp
   VectorContractToAMXDotProduct.cpp
+  MoveAccumulatorForContractLoop.cpp
 
   LINK_LIBS PUBLIC
   MLIRArithDialect
diff --git a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
new file mode 100644
index 0000000000000..6f87ebfa441dc
--- /dev/null
+++ b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
+    Value accValue = accReadOp->getResult(0);
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with acontract results before storing to acc
+    // matrix.
+    rewriter.setInsertionPoint(resultWriteOp);
+    Location locUser = resultWriteOp->getLoc();
+
+    Value addition;
+
+    if (llvm::isa<FloatType>(elemTy)) {
+      addition =
+          arith::AddFOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    if (llvm::isa<IntegerType>(elemTy)) {
+      addition =
+          arith::AddIOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    resultWriteOp->replaceUsesOfWith(contractValue, addition);
+    return success();
+  }
+};
+
+} // namespace
+
+void x86::populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns) {
+  patterns.add<MoveAccumulatorForContractLoop>(patterns.getContext());
+}
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
index 287892c3a660f..c70ad221c772d 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
@@ -415,25 +415,21 @@ struct VectorContractBF16ToFMA
       Operation *accReadOp1 =
           traceToVectorReadLikeParentOperation(pairContractOp.getAcc());
 
-      // Iterate down to find the users of contact operations until it is store
-      // or transfer_write.
-      Operation *resultWriteOp0 =
-          traceToVectorWriteLikeUserOperation(contractOp.getResult());
-      Operation *resultWriteOp1 =
-          traceToVectorWriteLikeUserOperation(pairContractOp.getResult());
-
-      // Shuffle the accumulators of the contract operations.
-      LogicalResult readShuffle =
-          shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                 pairContractOp, nonUnitDim, accTy);
-
-      if (failed(readShuffle))
-        return rewriter.notifyMatchFailure(
-            contractOp, "Accumulator read is not by transfer_read or load");
+      if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+        // Shuffle the accumulators of the contract operations.
+        LogicalResult readShuffle =
+            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
+                                   pairContractOp, nonUnitDim, accTy);
+
+        if (failed(readShuffle))
+          return rewriter.notifyMatchFailure(
+              contractOp, "Accumulator read is not by transfer_read or load");
+      }
 
       // Shuffle the output of contract operations before its use.
       LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-          rewriter, resultWriteOp0, resultWriteOp1, nonUnitDim, accTy);
+          rewriter, contractOp.getResult(), pairContractOp.getResult(),
+          nonUnitDim, accTy);
 
       if (failed(writeShuffle))
         return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
index d3487369355f0..394c950189ead 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
@@ -325,17 +325,17 @@ struct VectorContractToPackedTypeDotProduct
       }
 
       // If the accumulators are shuffled we get nullptr else the
-      // transfer_read or load operations.
-      Operation *accRead =
-          traceToVectorReadLikeParentOperation(contractOp.getAcc());
+      // transfer_load or store operations.
+      Operation *accWrite =
+          traceToVectorWriteLikeUserOperation(contractOp.getResult());
 
       if (!pairContractOp &&
-          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accRead))
+          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accWrite))
         return rewriter.notifyMatchFailure(contractOp,
                                            "Could not find a contract pair");
 
       // Validate and shuffle the accumulator
-      if (accRead) {
+      if (accWrite) {
         // Trace back to the load or transfer_read operations of the contract
         // accumulators.
         Operation *accReadOp0 =
@@ -373,18 +373,22 @@ struct VectorContractToPackedTypeDotProduct
           return rewriter.notifyMatchFailure(
               contractOp, "The store/write operation of contract operation is "
                           "before the pair contract operation");
-        // Shuffle the accumulators of the contract operations.
-        LogicalResult readShuffle =
-            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                   pairContractOp, nonUnitDimValue, accTy);
 
-        if (failed(readShuffle))
-          return rewriter.notifyMatchFailure(
-              contractOp, "Accumulator read is not by transfer_read or load");
+        if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+          // Shuffle the accumulators of the contract operations.
+          LogicalResult readShuffle = shuffleAfterReadLikeOp(
+              rewriter, accReadOp0, accReadOp1, contractOp, pairContractOp,
+              nonUnitDimValue, accTy);
+
+          if (failed(readShuffle))
+            return rewriter.notifyMatchFailure(
+                contractOp, "Accumulator read is not by transfer_read or load");
+        }
 
         // Shuffle the output of contract operations before it's use.
         LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-            rewriter, resultWriteOp0, resultWriteOp1, nonUnitDimValue, accTy);
+            rewriter, contractOp.getResult(), pairContractOp.getResult(),
+            nonUnitDimValue, accTy);
 
         if (failed(writeShuffle))
           return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
index a04a026f35ae6..232a912a84661 100644
--- a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
+++ b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
@@ -144,6 +144,25 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
   return {maskLo8, maskHi8};
 }
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v) {
+  if (v.getNumUses() != 1)
+    return nullptr;
+
+  OpOperand &use = *v.use_begin();
+  Operation *user = use.getOwner();
+
+  if (!isa<scf::YieldOp>(user))
+    return v;
+
+  auto yield = cast<scf::YieldOp>(user);
+  Operation *parent = yield->getParentOp();
+  unsigned idx = use.getOperandNumber();
+
+  return contractionUsersAfterYield(parent->getResult(idx));
+}
+
 // This function walks backward from a value to locate its originating
 // vector read-like operation (`vector.transfer_read` or `vector.load`).
 // It follows simple forwarding through unary ops and across `scf.for`
@@ -155,9 +174,21 @@ Operation *traceToVectorReadLikeParentOperation(Value v) {
   while (true) {
     // Case 1: Value defined by an operation
     if (Operation *defOp = v.getDefiningOp()) {
-      if (isa<vector::TransferReadOp, vector::LoadOp>(defOp))
+      if (isa<vector::TransferReadOp, vector::LoadOp, arith::ConstantOp>(defOp))
         return defOp;
 
+      if (isa<arith::ConstantOp>(defOp)) {
+        Attribute value = (dyn_cast<arith::ConstantOp>(defOp)).getValue();
+
+        if ((dyn_cast<IntegerAttr>(value)).getValue().isZero())
+          return defOp;
+
+        if ((dyn_cast<FloatAttr>(value)).getValue().isZero())
+          return defOp;
+
+        return nullptr;
+      }
+
       return nullptr;
     }
 
@@ -289,27 +320,23 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
-  // Helper to extract vector operand from write-like ops
-  auto getWrittenVector = [](Operation *op) -> Value {
-    if (auto write = dyn_cast<vector::TransferWriteOp>(op))
-      return write.getVector();
-    if (auto store = dyn_cast<vector::StoreOp>(op))
-      return store.getValueToStore();
-    return nullptr;
-  };
 
-  Value vecA = getWrittenVector(opA);
-  Value vecB = getWrittenVector(opB);
+  Value vecA = contractionUsersAfterYield(opA);
+  Value vecB = contractionUsersAfterYield(opB);
 
   if (!vecA || !vecB)
     return failure();
 
+  Operation *resultWriteOpA = *vecA.getUsers().begin();
+  Operation *resultWriteOpB = *vecB.getUsers().begin();
+
   // Decide insertion point and location
-  Operation *insertBefore = opA->isBeforeInBlock(opB) ? opA : opB;
+  Operation *insertBefore = resultWriteOpA->isBeforeInBlock(resultWriteOpB)
+                                ? resultWriteOpA
+                                : resultWriteOpB;
 
   rewriter.setInsertionPoint(insertBefore);
   Location loc = insertBefore->getLoc();
@@ -335,10 +362,8 @@ LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
   auto newVecB = vector::ShapeCastOp::create(rewriter, loc, accTy, shuffledHi);
 
   // Update write operands in place via the rewriter to notify it of changes.
-  rewriter.modifyOpInPlace(opA,
-                           [&]() { opA->setOperand(0, newVecA.getResult()); });
-  rewriter.modifyOpInPlace(opB,
-                           [&]() { opB->setOperand(0, newVecB.getResult()); });
+  resultWriteOpA->replaceUsesOfWith(vecA, newVecA);
+  resultWriteOpB->replaceUsesOfWith(vecB, newVecB);
 
   return success();
 }
diff --git a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
new file mode 100644
index 0000000000000..8ddec33b086df
--- /dev/null
+++ b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
@@ -0,0 +1,136 @@
+// RUN: mlir-opt %s -transform-interpreter -cse -split-input-file | FileCheck %s
+
+!vecA = vector<1x1x2xbf16>
+!vecB = vector<1x2x16xbf16>
+!vecC = vector<1x16xf32>
+!memrefA = memref<1x1x2xbf16, strided<[2048, 32, 1], offset: ?>>
+!memrefB = memref<1x2x32xbf16, strided<[2048, 64, 1], offset: ?>>
+!memrefC = memref<1x32xf32, strided<[64, 1], offset: ?>>
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64xbf16>,
+                             %arg2: memref<64x64xf32>) -> memref<64x64xf32> {
+  %0 = ub.poison : f32
+  %1 = ub.poison : bf16
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  scf.for %arg3 = %c0 to %c64 step %c1 {
+    scf.for %arg4 = %c0 to %c64 step %c32 {
+      %subview = memref.subview %arg2[%arg3, %arg4] [1, 32] [1, 1]
+                        : memref<64x64xf32> to !memrefC
+      %2 = vector.transfer_read %subview[%c0, %c0], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+      %3 = vector.transfer_read %subview[%c0, %c16], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+
+      %4:2 = scf.for %arg5 = %c0 to %c16 step %c1 iter_args(%arg6 = %2, %arg7 = %3) -> (!vecC, !vecC) {
+        %5:2 = scf.for %arg8 = %c0 to %c32 step %c2 iter_args(%arg9 = %arg6, %arg10 = %arg7) -> (!vecC, !vecC) {
+
+          %subview_0 = memref.subview %arg0[%arg5, %arg3, %arg8] [1, 1, 2] [1, 1, 1]
+                        : memref<16x64x32xbf16> to !memrefA
+          %subview_1 = memref.subview %arg1[%arg5, %arg8, %arg4] [1, 2, 32] [1, 1, 1]
+                   ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 01:57:09 2026
From: llvmlistbot at llvm.org (Jianjian Guan)
Date: Wed, 22 Jul 2026 01:57:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix warning, NFC (PR #211204)
Message-ID: <llvm.org/llvm/llvm-project/pull/211204@github.com>


https://github.com/jacquesguan created https://github.com/llvm/llvm-project/pull/211204

None

>From 85cf06a11b00645b7084bdaa798280b3de2880cd Mon Sep 17 00:00:00 2001
From: Jianjian GUAN <jacquesguan at me.com>
Date: Wed, 22 Jul 2026 16:51:40 +0800
Subject: [PATCH] [mlir] Fix warning, NFC

---
 mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..4e8ef252a10c9 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -314,9 +314,9 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
     // pair of FP types that are valid LLVM types.
     [[maybe_unused]] auto srcType = getElementTypeOrSelf(op.getIn().getType());
     [[maybe_unused]] auto dstType = getElementTypeOrSelf(op.getType());
-    assert((srcType.isBF16() && dstType.isF16()) ||
-           (srcType.isF16() && dstType.isBF16()) &&
-               "only bf16 <-> f16 conversions are supported");
+    assert(((srcType.isBF16() && dstType.isF16()) ||
+            (srcType.isF16() && dstType.isBF16())) &&
+           "only bf16 <-> f16 conversions are supported");
 
     Type convertedType = getTypeConverter()->convertType(op.getType());
     if (!convertedType)


From llvmlistbot at llvm.org  Wed Jul 22 01:57:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 01:57:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix warning, NFC (PR #211204)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211204@github.com>
Message-ID: <6a60860a.e55dc341.dbc09.2d47@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianjian Guan (jacquesguan)

<details>
<summary>Changes</summary>



---
Full diff: https://github.com/llvm/llvm-project/pull/211204.diff


1 Files Affected:

- (modified) mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp (+3-3) 


``````````diff
diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..4e8ef252a10c9 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -314,9 +314,9 @@ struct ConvertFOpLowering : public ConvertOpToLLVMPattern<arith::ConvertFOp> {
     // pair of FP types that are valid LLVM types.
     [[maybe_unused]] auto srcType = getElementTypeOrSelf(op.getIn().getType());
     [[maybe_unused]] auto dstType = getElementTypeOrSelf(op.getType());
-    assert((srcType.isBF16() && dstType.isF16()) ||
-           (srcType.isF16() && dstType.isBF16()) &&
-               "only bf16 <-> f16 conversions are supported");
+    assert(((srcType.isBF16() && dstType.isF16()) ||
+            (srcType.isF16() && dstType.isBF16())) &&
+           "only bf16 <-> f16 conversions are supported");
 
     Type convertedType = getTypeConverter()->convertType(op.getType());
     if (!convertedType)

``````````

</details>


https://github.com/llvm/llvm-project/pull/211204

From llvmlistbot at llvm.org  Wed Jul 22 01:58:18 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Wed, 22 Jul 2026 01:58:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a60862a.d76d81af.3b68ca.c98d@mx.google.com>


bruteforceboy wrote:

cc: @nujaa @banach-space

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:16:49 2026
From: llvmlistbot at llvm.org (Chuanqi Xu)
Date: Wed, 22 Jul 2026 02:16:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Reorder element/broadcast for
 FMA if the source is not scalar (PR #211208)
Message-ID: <llvm.org/llvm/llvm-project/pull/211208@github.com>


https://github.com/ChuanqiXu9 created https://github.com/llvm/llvm-project/pull/211208

The diagnostic says the transform is not performed as the source may be a scalar. But we can make it if we can check the source is not a scalar.

AI assisted.

>From ca4a8c504c93d837cfa053a1b86b1cbd0167a728 Mon Sep 17 00:00:00 2001
From: "yedeng.yd" <yedeng.yd at alibaba-inc.com>
Date: Wed, 22 Jul 2026 16:56:46 +0800
Subject: [PATCH] [mlir] [vector] Reorder element/broadcast for FMA if the
 source is not scalar

The diagnostic says the transform is not performed as the source may be
a scalar. But we can make it if we can check the source is not a scalar.

AI assisted.
---
 .../Vector/Transforms/VectorTransforms.cpp    | 17 ++--
 mlir/test/Dialect/Vector/vector-sink.mlir     | 81 +++++++++++++++++--
 2 files changed, 85 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 17e6ad640cada..e1d9bc8d0f4a0 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -1043,12 +1043,6 @@ struct ReorderElementwiseOpsOnBroadcast final
           op, "Op doesn't have ElementwiseMappableTraits");
     if (op->getNumOperands() == 0)
       return failure();
-    if (isa<vector::FMAOp>(op)) {
-      return rewriter.notifyMatchFailure(
-          op,
-          "Op only accepts vector types - not supported as broadcast source "
-          "might be a scalar");
-    }
 
     Type resultElemType = resultType.getElementType();
 
@@ -1068,6 +1062,17 @@ struct ReorderElementwiseOpsOnBroadcast final
     Type unbroadcastResultType =
         cloneOrReplace(broadcastSource.getType(), resultElemType);
 
+    // Some ops, e.g. `vector.fma`, only accept vector types. For such ops the
+    // reordering is only possible when the broadcast source is a vector as
+    // well; sinking past a broadcast from a scalar would create an invalid op.
+    // TODO: Support scalar sources for `vector.fma` by rewriting to
+    // `math.fma`.
+    if (isa<vector::FMAOp>(op) && !isa<VectorType>(unbroadcastResultType)) {
+      return rewriter.notifyMatchFailure(
+          op, "Op only accepts vector types, but the broadcast source is a "
+              "scalar");
+    }
+
     // Make sure that all operands are broadcast from identically-shaped types:
     //  * scalar (`vector.broadcast`), or
     //  * vector (`vector.broadcast`).
diff --git a/mlir/test/Dialect/Vector/vector-sink.mlir b/mlir/test/Dialect/Vector/vector-sink.mlir
index 69fba88a14048..b52828419b10a 100644
--- a/mlir/test/Dialect/Vector/vector-sink.mlir
+++ b/mlir/test/Dialect/Vector/vector-sink.mlir
@@ -195,17 +195,18 @@ func.func @source_and_result_mismatch(%arg0 : f32) -> vector<1xi1> {
 
 // -----
 
-// vector.fma only supports vectors - currently it's not possible to replace this with e.g.:
+// vector.fma only supports vectors - when the broadcast source is a scalar,
+// currently it's not possible to replace this with e.g.:
 //    %scalar_res = vector.fma %scalar_1, %scalar2
 //    %vec_res = vector.broadcast %scalar_res
 //
-// TODO: It should be possible to support this case
+// TODO: It should be possible to support this case by rewriting to math.fma
 
-// CHECK-LABEL: func.func @negative_op_only_supports_vectors
-  //     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
-  //     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
-  //     CHECK:   return %[[RESULT]]
-func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
+// CHECK-LABEL: func.func @negative_fma_scalar_broadcast_source
+//     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
+//     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
+//     CHECK:   return %[[RESULT]]
+func.func @negative_fma_scalar_broadcast_source(%arg0 : f32) -> vector<1xf32> {
   %0 = vector.broadcast %arg0 : f32 to vector<1xf32>
   %1 = vector.fma %0, %0, %0 : vector<1xf32>
   return %1 : vector<1xf32>
@@ -213,6 +214,72 @@ func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
 
 // -----
 
+// CHECK-LABEL: func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: f32, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[BCAST0:.*]] = vector.broadcast %[[ARG0]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[BCAST1:.*]] = vector.broadcast %[[ARG1]] : f32 to vector<3x4xf32>
+//       CHECK:   %[[BCAST2:.*]] = vector.broadcast %[[ARG2]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[BCAST0]], %[[BCAST1]], %[[BCAST2]] : vector<3x4xf32>
+//       CHECK:   return %[[FMA]] : vector<3x4xf32>
+func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: f32, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// vector.fma only supports vector operands, hence the broadcast can only be
+// sunk when the broadcast source is a vector as well.
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: vector<4xf32>, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_scalable(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<[4]xf32>, %[[ARG1:.*]]: vector<[4]xf32>, %[[ARG2:.*]]: vector<[4]xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<[4]xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<[4]xf32> to vector<3x[4]xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x[4]xf32>
+func.func @fma_vector_broadcast_source_scalable(%arg0: vector<[4]xf32>, %arg1: vector<[4]xf32>, %arg2: vector<[4]xf32>) -> vector<3x[4]xf32> {
+  %0 = vector.broadcast %arg0 : vector<[4]xf32> to vector<3x[4]xf32>
+  %1 = vector.broadcast %arg1 : vector<[4]xf32> to vector<3x[4]xf32>
+  %2 = vector.broadcast %arg2 : vector<[4]xf32> to vector<3x[4]xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x[4]xf32>
+  return %3 : vector<3x[4]xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_and_splat_const(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>)
+//       CHECK:   %[[NEW_CST:.*]] = arith.constant dense<2.000000e+00> : vector<4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[NEW_CST]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source_and_splat_const(%arg0: vector<4xf32>, %arg1: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %cst = arith.constant dense<2.0> : vector<3x4xf32>
+  %2 = vector.fma %0, %1, %cst : vector<3x4xf32>
+  return %2 : vector<3x4xf32>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @broadcast_scalar_and_splat_const(
 // CHECK-SAME:     %[[ARG_0:.*]]: index) -> vector<1x4xindex> {
 // CHECK:           %[[NEW_CST:.*]] = arith.constant 2 : index


From llvmlistbot at llvm.org  Wed Jul 22 02:17:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 02:17:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Reorder element/broadcast for
 FMA if the source is not scalar (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a608aaa.3cce8983.247e47.3e9d@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Chuanqi Xu (ChuanqiXu9)

<details>
<summary>Changes</summary>

The diagnostic says the transform is not performed as the source may be a scalar. But we can make it if we can check the source is not a scalar.

AI assisted.

---
Full diff: https://github.com/llvm/llvm-project/pull/211208.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp (+11-6) 
- (modified) mlir/test/Dialect/Vector/vector-sink.mlir (+74-7) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 17e6ad640cada..e1d9bc8d0f4a0 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -1043,12 +1043,6 @@ struct ReorderElementwiseOpsOnBroadcast final
           op, "Op doesn't have ElementwiseMappableTraits");
     if (op->getNumOperands() == 0)
       return failure();
-    if (isa<vector::FMAOp>(op)) {
-      return rewriter.notifyMatchFailure(
-          op,
-          "Op only accepts vector types - not supported as broadcast source "
-          "might be a scalar");
-    }
 
     Type resultElemType = resultType.getElementType();
 
@@ -1068,6 +1062,17 @@ struct ReorderElementwiseOpsOnBroadcast final
     Type unbroadcastResultType =
         cloneOrReplace(broadcastSource.getType(), resultElemType);
 
+    // Some ops, e.g. `vector.fma`, only accept vector types. For such ops the
+    // reordering is only possible when the broadcast source is a vector as
+    // well; sinking past a broadcast from a scalar would create an invalid op.
+    // TODO: Support scalar sources for `vector.fma` by rewriting to
+    // `math.fma`.
+    if (isa<vector::FMAOp>(op) && !isa<VectorType>(unbroadcastResultType)) {
+      return rewriter.notifyMatchFailure(
+          op, "Op only accepts vector types, but the broadcast source is a "
+              "scalar");
+    }
+
     // Make sure that all operands are broadcast from identically-shaped types:
     //  * scalar (`vector.broadcast`), or
     //  * vector (`vector.broadcast`).
diff --git a/mlir/test/Dialect/Vector/vector-sink.mlir b/mlir/test/Dialect/Vector/vector-sink.mlir
index 69fba88a14048..b52828419b10a 100644
--- a/mlir/test/Dialect/Vector/vector-sink.mlir
+++ b/mlir/test/Dialect/Vector/vector-sink.mlir
@@ -195,17 +195,18 @@ func.func @source_and_result_mismatch(%arg0 : f32) -> vector<1xi1> {
 
 // -----
 
-// vector.fma only supports vectors - currently it's not possible to replace this with e.g.:
+// vector.fma only supports vectors - when the broadcast source is a scalar,
+// currently it's not possible to replace this with e.g.:
 //    %scalar_res = vector.fma %scalar_1, %scalar2
 //    %vec_res = vector.broadcast %scalar_res
 //
-// TODO: It should be possible to support this case
+// TODO: It should be possible to support this case by rewriting to math.fma
 
-// CHECK-LABEL: func.func @negative_op_only_supports_vectors
-  //     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
-  //     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
-  //     CHECK:   return %[[RESULT]]
-func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
+// CHECK-LABEL: func.func @negative_fma_scalar_broadcast_source
+//     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
+//     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
+//     CHECK:   return %[[RESULT]]
+func.func @negative_fma_scalar_broadcast_source(%arg0 : f32) -> vector<1xf32> {
   %0 = vector.broadcast %arg0 : f32 to vector<1xf32>
   %1 = vector.fma %0, %0, %0 : vector<1xf32>
   return %1 : vector<1xf32>
@@ -213,6 +214,72 @@ func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
 
 // -----
 
+// CHECK-LABEL: func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: f32, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[BCAST0:.*]] = vector.broadcast %[[ARG0]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[BCAST1:.*]] = vector.broadcast %[[ARG1]] : f32 to vector<3x4xf32>
+//       CHECK:   %[[BCAST2:.*]] = vector.broadcast %[[ARG2]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[BCAST0]], %[[BCAST1]], %[[BCAST2]] : vector<3x4xf32>
+//       CHECK:   return %[[FMA]] : vector<3x4xf32>
+func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: f32, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// vector.fma only supports vector operands, hence the broadcast can only be
+// sunk when the broadcast source is a vector as well.
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: vector<4xf32>, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_scalable(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<[4]xf32>, %[[ARG1:.*]]: vector<[4]xf32>, %[[ARG2:.*]]: vector<[4]xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<[4]xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<[4]xf32> to vector<3x[4]xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x[4]xf32>
+func.func @fma_vector_broadcast_source_scalable(%arg0: vector<[4]xf32>, %arg1: vector<[4]xf32>, %arg2: vector<[4]xf32>) -> vector<3x[4]xf32> {
+  %0 = vector.broadcast %arg0 : vector<[4]xf32> to vector<3x[4]xf32>
+  %1 = vector.broadcast %arg1 : vector<[4]xf32> to vector<3x[4]xf32>
+  %2 = vector.broadcast %arg2 : vector<[4]xf32> to vector<3x[4]xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x[4]xf32>
+  return %3 : vector<3x[4]xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_and_splat_const(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>)
+//       CHECK:   %[[NEW_CST:.*]] = arith.constant dense<2.000000e+00> : vector<4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[NEW_CST]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source_and_splat_const(%arg0: vector<4xf32>, %arg1: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %cst = arith.constant dense<2.0> : vector<3x4xf32>
+  %2 = vector.fma %0, %1, %cst : vector<3x4xf32>
+  return %2 : vector<3x4xf32>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @broadcast_scalar_and_splat_const(
 // CHECK-SAME:     %[[ARG_0:.*]]: index) -> vector<1x4xindex> {
 // CHECK:           %[[NEW_CST:.*]] = arith.constant 2 : index

``````````

</details>


https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Wed Jul 22 02:21:37 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba1.510f576d.33aa7d.3fed@mx.google.com>


https://github.com/nujaa commented:

Thanks for targeting this.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:37 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba1.eb165e5c.2e0c9a.2452@mx.google.com>


https://github.com/nujaa edited https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:38 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba2.4a342cbc.1aeb63.0055@mx.google.com>


================
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_2d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_to_1d(%input: tensor<f32>,
----------------
nujaa wrote:

could we have an example with `%input: f32` (not a tensor)

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:38 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba2.492e23e4.292f20.a353@mx.google.com>


================
@@ -1196,6 +1196,43 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank1(
+    %input: tensor<4xf32>, %init: tensor<4x8x16xf32>) -> tensor<4x8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4xf32>)
+      outs(%init:tensor<4x8x16xf32>)
+      dimensions = [1, 1]
+  func.return %bcast : tensor<4x8x16xf32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank2(
+    %input: tensor<4x8xf32>, %init: tensor<4x8x16x32xf32>)
+    -> tensor<4x8x16x32xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4x8xf32>)
+      outs(%init:tensor<4x8x16x32xf32>)
+      dimensions = [3, 3]
+  func.return %bcast : tensor<4x8x16x32xf32>
+}
+
+// -----
+
----------------
nujaa wrote:

All those tests fall in the same condition, I don't think it is relevant to add all 3 of them.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:38 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba2.160324b0.17e275.8d49@mx.google.com>


================
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_2d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_to_1d(%input: tensor<f32>,
----------------
nujaa wrote:

Additionnally, I think it wouldn't hurt skimming the test coverage a bit as mentioned above.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:38 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba2.510f576d.33aa7d.40b7@mx.google.com>


================
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
----------------
nujaa wrote:

Thanks for adding this **working** tests. A bit of context @banach-space , I messed up in the previous MR. It is not the 0-D input but that was not tolerated by broadcast but flattening which generated improper code. It would create a new the LinalgOp with all attributes (among others the dimensions attribute) which is incorrect in that case and failing at `verify`.  @bruteforceboy will come submit a solution soon.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:21:39 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Wed, 22 Jul 2026 02:21:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a608ba3.1e1ac41d.317291.7752@mx.google.com>


================
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
----------------
nujaa wrote:

banach-space is trying to get some sort of consistency in the tests naming, do you think we could keep the `_{memref|tensor}` suffix as above ?

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.507fc713.3702e2.65da@mx.google.com>


================
@@ -144,6 +144,25 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
   return {maskLo8, maskHi8};
 }
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v) {
+  if (v.getNumUses() != 1)
+    return nullptr;
+
+  OpOperand &use = *v.use_begin();
+  Operation *user = use.getOwner();
+
+  if (!isa<scf::YieldOp>(user))
+    return v;
----------------
adam-smnk wrote:

Shouldn't this return `nullptr` as well?
Otherwise, not sure how I should interpret getting the same value back.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.f1c1923d.d9163.3b29@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
----------------
adam-smnk wrote:

nit: stray char

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.4987c784.13f2de.f649@mx.google.com>


================
@@ -0,0 +1,136 @@
+// RUN: mlir-opt %s -transform-interpreter -cse -split-input-file | FileCheck %s
+
+!vecA = vector<1x1x2xbf16>
+!vecB = vector<1x2x16xbf16>
+!vecC = vector<1x16xf32>
+!memrefA = memref<1x1x2xbf16, strided<[2048, 32, 1], offset: ?>>
+!memrefB = memref<1x2x32xbf16, strided<[2048, 64, 1], offset: ?>>
+!memrefC = memref<1x32xf32, strided<[64, 1], offset: ?>>
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64xbf16>,
+                             %arg2: memref<64x64xf32>) -> memref<64x64xf32> {
+  %0 = ub.poison : f32
+  %1 = ub.poison : bf16
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  scf.for %arg3 = %c0 to %c64 step %c1 {
+    scf.for %arg4 = %c0 to %c64 step %c32 {
+      %subview = memref.subview %arg2[%arg3, %arg4] [1, 32] [1, 1]
+                        : memref<64x64xf32> to !memrefC
+      %2 = vector.transfer_read %subview[%c0, %c0], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+      %3 = vector.transfer_read %subview[%c0, %c16], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+
+      %4:2 = scf.for %arg5 = %c0 to %c16 step %c1 iter_args(%arg6 = %2, %arg7 = %3) -> (!vecC, !vecC) {
+        %5:2 = scf.for %arg8 = %c0 to %c32 step %c2 iter_args(%arg9 = %arg6, %arg10 = %arg7) -> (!vecC, !vecC) {
+
+          %subview_0 = memref.subview %arg0[%arg5, %arg3, %arg8] [1, 1, 2] [1, 1, 1]
+                        : memref<16x64x32xbf16> to !memrefA
+          %subview_1 = memref.subview %arg1[%arg5, %arg8, %arg4] [1, 2, 32] [1, 1, 1]
+                        : memref<16x32x64xbf16> to !memrefB
+
+          %6 = vector.transfer_read %subview_0[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefA, !vecA
+          %7 = vector.transfer_read %subview_1[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+          %8 = vector.transfer_read %subview_1[%c0, %c0, %c16], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+
+          %9 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %7, %arg9
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+          %10 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %8, %arg10
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+
+          scf.yield %9, %10 : !vecC, !vecC
+        }
+        scf.yield %5#0, %5#1 : !vecC, !vecC
+      }
+
+      vector.transfer_write %4#1, %subview[%c0, %c16] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+      vector.transfer_write %4#0, %subview[%c0, %c0] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+    }
+  }
+
+  return %arg2 : memref<64x64xf32>
+}
+
+// CHECK-LABEL: @brmatmul_acc_mv
----------------
adam-smnk wrote:

nit: Could you expand checks to capture the new chain with contract etc.?
Currently I don't clearly see what rewrite is happening.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.e624eb8d.12f676.51a4@mx.google.com>


================
@@ -289,27 +320,23 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
-  // Helper to extract vector operand from write-like ops
-  auto getWrittenVector = [](Operation *op) -> Value {
-    if (auto write = dyn_cast<vector::TransferWriteOp>(op))
-      return write.getVector();
-    if (auto store = dyn_cast<vector::StoreOp>(op))
-      return store.getValueToStore();
-    return nullptr;
-  };
 
-  Value vecA = getWrittenVector(opA);
-  Value vecB = getWrittenVector(opB);
+  Value vecA = contractionUsersAfterYield(opA);
----------------
adam-smnk wrote:

nit: this is a bit of util regression by adding implicit constrain that ops have to be inside a loop.
Could the old and new behavior be both supported?

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.743400b0.25573c.1def@mx.google.com>


================
@@ -59,9 +63,9 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc, VectorType accTy);
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
----------------
adam-smnk wrote:

nit: the args are no longer operations, I'd rename `opX` to sth else

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.eb165e5c.2e0c9a.8f00@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
----------------
adam-smnk wrote:

nit: I'd move insertion point change just before creation of the relevant ops

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:39 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c57.eb165e5c.2e0c9a.8ef9@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
+    Value accValue = accReadOp->getResult(0);
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with acontract results before storing to acc
----------------
adam-smnk wrote:

nit: typo

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:24:41 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Wed, 22 Jul 2026 02:24:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a608c59.0c05e68e.81651.7b2b@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
----------------
adam-smnk wrote:

nit: it doesn't know it's a *zero* vector, just a constant

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Wed Jul 22 02:28:57 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 22 Jul 2026 02:28:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a608d59.e55dc341.dbc09.b088@mx.google.com>


joker-eph wrote:

@qyingwu can you provide a backtrace produced with a debug build so that I can understand the crash condition a bit better?

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Wed Jul 22 02:59:16 2026
From: llvmlistbot at llvm.org (Natanael Cintean)
Date: Wed, 22 Jul 2026 02:59:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add resultSegmentSizes/operandSegmentSizes to
	prop-dict. (PR #211222)
Message-ID: <llvm.org/llvm/llvm-project/pull/211222@github.com>


https://github.com/natanael-cintean created https://github.com/llvm/llvm-project/pull/211222

Fixes #211220

## Summary

`setPropertiesFromParsedAttr` (generated by `OpFormatGen.cpp` for ops using a
custom `assemblyFormat`) rejects the trait-injected `operandSegmentSizes` /
`resultSegmentSizes` properties when they appear in a `prop-dict`, even though
the printer emits them there for ops whose format uses a bulk operand/result
type directive (`type(operands)`, `type(results)`,
`functional-type(operands, results)`).

This breaks round-tripping: any op combining `AttrSizedOperandSegments` /
`AttrSizedResultSegments` with such a bulk directive fails to re-parse its own
printed output, because printer-side elision of these keys
(introduced in #115930) is intentionally skipped in that case — the sizes
can't be reconstructed from individually-typed operand/result groups, so they
must survive in the text, but the custom parser was never taught to read them
back.

`setPropertiesFromAttr` (used by the generic-form parser, bytecode, and C++
construction) already special-cases these two keys, so this change makes
`setPropertiesFromParsedAttr` mirror that behavior: keys are accepted from the
dictionary attr when the corresponding trait is present, and required exactly
when the format can't reconstruct the sizes itself (i.e. when
`fmt.allOperands` / `fmt.allResultTypes` is true). Otherwise the key remains
optional, since `genParserVariadicSegmentResolution` overwrites it with the
sizes inferred from the parsed operand/result groups.

## Changes

- `mlir/tools/mlir-tblgen/OpFormatGen.cpp`: `genParsedAttrPropertiesSetter` now
  emits a handler for `operandSegmentSizes` / `resultSegmentSizes` right after
  `usedKeys` is declared, for ops with `AttrSizedOperandSegments` /
  `AttrSizedResultSegments`.
- `mlir/test/lib/Dialect/Test/TestOps.td`: added
  `test.variadic_segment_prop_bulk_type`
  (`TestOpWithBulkTypesAndSegmentProperties`), which combines both
  `AttrSized*Segments` traits with a bulk `functional-type(operands, results)`
  directive — the pattern that triggers the bug.
- `mlir/test/IR/properties.mlir`: round-trip test (custom form and generic
  form) for the new op, exercising both the previously-broken parse path and
  confirming the printer output stays stable.

>From 41835df006b9b5146dda44d7e2824b5e0a2baaa9 Mon Sep 17 00:00:00 2001
From: Natanael Cintean <natanael.cintean at intel.com>
Date: Tue, 21 Jul 2026 14:56:22 +0000
Subject: [PATCH] Add resultSegmentSizes/operandSegmentSizes to prop-dict. Add
 test to check for these in prop-dict.

---
 mlir/test/IR/properties.mlir           | 11 +++++++++
 mlir/test/lib/Dialect/Test/TestOps.td  | 16 +++++++++++++
 mlir/tools/mlir-tblgen/OpFormatGen.cpp | 32 ++++++++++++++++++++++++++
 3 files changed, 59 insertions(+)

diff --git a/mlir/test/IR/properties.mlir b/mlir/test/IR/properties.mlir
index 4dbb8ebfa5db3..4d83038f31cdd 100644
--- a/mlir/test/IR/properties.mlir
+++ b/mlir/test/IR/properties.mlir
@@ -46,6 +46,17 @@ test.using_property_ref_in_custom 1 + 4 = 5
 %ci64 = arith.constant 0 : i64
 test.variadic_segment_prop %ci64, %ci64 : %ci64 : i64, i64 : i64 end
 
+// Tests that the variadic segment size properties survive a round-trip
+// through the *custom* (non-generic) parser/printer when the assembly format
+// uses a bulk `functional-type(operands, results)` directive, which prevents
+// the printer from eliding `operandSegmentSizes` / `resultSegmentSizes` from
+// `<{...}>`. Without the parser-side fix, re-parsing the CHECK line below
+// (which is exactly what the printer emits) fails with "duplicate or unknown
+// key 'operandSegmentSizes' in dictionary attribute".
+// CHECK: test.variadic_segment_prop_bulk_type(%[[CI64]], %[[CI64]], %[[CI64]]) : (i64, i64, i64) -> (i64, i64, i64) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}>
+// GENERIC: "test.variadic_segment_prop_bulk_type"(%[[CI64]], %[[CI64]], %[[CI64]]) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}> : (i64, i64, i64) -> (i64, i64, i64)
+test.variadic_segment_prop_bulk_type(%ci64, %ci64, %ci64) : (i64, i64, i64) -> (i64, i64, i64) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}>
+
 // CHECK:   test.with_default_valued_properties na{{$}}
 // GENERIC: "test.with_default_valued_properties"()
 // GENERIC-SAME: <{a = 0 : i32, b = "", c = -1 : i32, unit = false}> : () -> ()
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index c12e2abb3ab75..db55b90c294e1 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -3686,6 +3686,22 @@ def TestOpWithVariadicSegmentProperties : TEST_Op<"variadic_segment_prop",
   }];
 }
 
+// Same as `TestOpWithVariadicSegmentProperties`, but the assembly format uses
+// the bulk `type(operands)` / `type(results)` directives instead of spelling
+// out the type of each individual variadic group. This prevents the printer
+// from eliding `operandSegmentSizes` / `resultSegmentSizes` from the
+// `<{...}>` properties dictionary (since there is no other way to recover the
+// segment sizes on re-parse), so this op exercises the parser's ability to
+// read those trait-injected properties back out of `prop-dict`.
+def TestOpWithBulkTypesAndSegmentProperties : TEST_Op<"variadic_segment_prop_bulk_type",
+    [AttrSizedOperandSegments, AttrSizedResultSegments]> {
+  let arguments = (ins Variadic<I64>:$a1, Variadic<I64>:$a2);
+  let results = (outs Variadic<I64>:$b1, Variadic<I64>:$b2);
+  let assemblyFormat = [{
+    `(` operands `)` `:` functional-type(operands, results) prop-dict attr-dict
+  }];
+}
+
 def TestOpUsingPropertyRefInCustom : TEST_Op<"using_property_ref_in_custom"> {
   let assemblyFormat = "custom<IntProperty>($first) `+` custom<SumProperty>($second, ref($first)) attr-dict";
   let arguments = (ins IntProp<"int64_t">:$first, IntProp<"int64_t">:$second);
diff --git a/mlir/tools/mlir-tblgen/OpFormatGen.cpp b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
index cbcbc8e9bc102..f9d07f6b3ba0f 100644
--- a/mlir/tools/mlir-tblgen/OpFormatGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
@@ -1313,6 +1313,38 @@ ::mlir::MLIRContext *ctx = dict.getContext();
 (void)ctx;
 )decl";
 
+  // `operandSegmentSizes`/`resultSegmentSizes` are trait-injected properties
+  // not enumerated by `op.getProperties()`, so they need to be special-cased
+  // here, mirroring the handling in `setPropertiesFromAttr` in
+  // OpDefinitionsGen.cpp.
+  //
+  // {0}: segment sizes property name
+  // {1}: isRequired
+  const char *segmentSizesFromAttrFmt = R"decl(
+auto {0}AttrName = ::mlir::StringAttr::get(ctx, "{0}");
+usedKeys.insert({0}AttrName);
+auto attr = dict.get({0}AttrName);
+if (!attr && {1}) {{
+  emitError() << "expected key entry for {0} in DictionaryAttr to set "
+             "Properties.";
+  return ::mlir::failure();
+}
+if (attr && ::mlir::failed(::mlir::convertFromAttribute(prop.{0}, attr, [&]() {{
+      return emitError() << "for `{0}`: ";
+    })))
+  return ::mlir::failure();
+)decl";
+  if (op.getTrait("::mlir::OpTrait::AttrSizedOperandSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "operandSegmentSizes",
+                    fmt.allOperands);
+  }
+  if (op.getTrait("::mlir::OpTrait::AttrSizedResultSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "resultSegmentSizes",
+                    fmt.allResultTypes);
+  }
+
   // {0}: fromAttribute call
   // {1}: property name
   // {2}: isRequired


From llvmlistbot at llvm.org  Wed Jul 22 02:59:35 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 02:59:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add resultSegmentSizes/operandSegmentSizes to
	prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a609487.4987c784.13f2de.30f5@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @natanael-cintean :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 02:59:46 2026
From: llvmlistbot at llvm.org (Natanael Cintean)
Date: Wed, 22 Jul 2026 02:59:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a609492.66c0679a.9a15c.728a@mx.google.com>


https://github.com/natanael-cintean edited https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 03:00:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 03:00:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a6094a6.bd93e731.2ac69c.44e9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Natanael Cintean (natanael-cintean)

<details>
<summary>Changes</summary>

Fixes #<!-- -->211220

## Summary

`setPropertiesFromParsedAttr` (generated by `OpFormatGen.cpp` for ops using a
custom `assemblyFormat`) rejects the trait-injected `operandSegmentSizes` /
`resultSegmentSizes` properties when they appear in a `prop-dict`, even though
the printer emits them there for ops whose format uses a bulk operand/result
type directive (`type(operands)`, `type(results)`,
`functional-type(operands, results)`).

This breaks round-tripping: any op combining `AttrSizedOperandSegments` /
`AttrSizedResultSegments` with such a bulk directive fails to re-parse its own
printed output, because printer-side elision of these keys
(introduced in #<!-- -->115930) is intentionally skipped in that case — the sizes
can't be reconstructed from individually-typed operand/result groups, so they
must survive in the text, but the custom parser was never taught to read them
back.

`setPropertiesFromAttr` (used by the generic-form parser, bytecode, and C++
construction) already special-cases these two keys, so this change makes
`setPropertiesFromParsedAttr` mirror that behavior: keys are accepted from the
dictionary attr when the corresponding trait is present, and required exactly
when the format can't reconstruct the sizes itself (i.e. when
`fmt.allOperands` / `fmt.allResultTypes` is true). Otherwise the key remains
optional, since `genParserVariadicSegmentResolution` overwrites it with the
sizes inferred from the parsed operand/result groups.

## Changes

- `mlir/tools/mlir-tblgen/OpFormatGen.cpp`: `genParsedAttrPropertiesSetter` now
  emits a handler for `operandSegmentSizes` / `resultSegmentSizes` right after
  `usedKeys` is declared, for ops with `AttrSizedOperandSegments` /
  `AttrSizedResultSegments`.
- `mlir/test/lib/Dialect/Test/TestOps.td`: added
  `test.variadic_segment_prop_bulk_type`
  (`TestOpWithBulkTypesAndSegmentProperties`), which combines both
  `AttrSized*Segments` traits with a bulk `functional-type(operands, results)`
  directive — the pattern that triggers the bug.
- `mlir/test/IR/properties.mlir`: round-trip test (custom form and generic
  form) for the new op, exercising both the previously-broken parse path and
  confirming the printer output stays stable.

---
Full diff: https://github.com/llvm/llvm-project/pull/211222.diff


3 Files Affected:

- (modified) mlir/test/IR/properties.mlir (+11) 
- (modified) mlir/test/lib/Dialect/Test/TestOps.td (+16) 
- (modified) mlir/tools/mlir-tblgen/OpFormatGen.cpp (+32) 


``````````diff
diff --git a/mlir/test/IR/properties.mlir b/mlir/test/IR/properties.mlir
index 4dbb8ebfa5db3..4d83038f31cdd 100644
--- a/mlir/test/IR/properties.mlir
+++ b/mlir/test/IR/properties.mlir
@@ -46,6 +46,17 @@ test.using_property_ref_in_custom 1 + 4 = 5
 %ci64 = arith.constant 0 : i64
 test.variadic_segment_prop %ci64, %ci64 : %ci64 : i64, i64 : i64 end
 
+// Tests that the variadic segment size properties survive a round-trip
+// through the *custom* (non-generic) parser/printer when the assembly format
+// uses a bulk `functional-type(operands, results)` directive, which prevents
+// the printer from eliding `operandSegmentSizes` / `resultSegmentSizes` from
+// `<{...}>`. Without the parser-side fix, re-parsing the CHECK line below
+// (which is exactly what the printer emits) fails with "duplicate or unknown
+// key 'operandSegmentSizes' in dictionary attribute".
+// CHECK: test.variadic_segment_prop_bulk_type(%[[CI64]], %[[CI64]], %[[CI64]]) : (i64, i64, i64) -> (i64, i64, i64) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}>
+// GENERIC: "test.variadic_segment_prop_bulk_type"(%[[CI64]], %[[CI64]], %[[CI64]]) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}> : (i64, i64, i64) -> (i64, i64, i64)
+test.variadic_segment_prop_bulk_type(%ci64, %ci64, %ci64) : (i64, i64, i64) -> (i64, i64, i64) <{operandSegmentSizes = array<i32: 2, 1>, resultSegmentSizes = array<i32: 2, 1>}>
+
 // CHECK:   test.with_default_valued_properties na{{$}}
 // GENERIC: "test.with_default_valued_properties"()
 // GENERIC-SAME: <{a = 0 : i32, b = "", c = -1 : i32, unit = false}> : () -> ()
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index c12e2abb3ab75..db55b90c294e1 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -3686,6 +3686,22 @@ def TestOpWithVariadicSegmentProperties : TEST_Op<"variadic_segment_prop",
   }];
 }
 
+// Same as `TestOpWithVariadicSegmentProperties`, but the assembly format uses
+// the bulk `type(operands)` / `type(results)` directives instead of spelling
+// out the type of each individual variadic group. This prevents the printer
+// from eliding `operandSegmentSizes` / `resultSegmentSizes` from the
+// `<{...}>` properties dictionary (since there is no other way to recover the
+// segment sizes on re-parse), so this op exercises the parser's ability to
+// read those trait-injected properties back out of `prop-dict`.
+def TestOpWithBulkTypesAndSegmentProperties : TEST_Op<"variadic_segment_prop_bulk_type",
+    [AttrSizedOperandSegments, AttrSizedResultSegments]> {
+  let arguments = (ins Variadic<I64>:$a1, Variadic<I64>:$a2);
+  let results = (outs Variadic<I64>:$b1, Variadic<I64>:$b2);
+  let assemblyFormat = [{
+    `(` operands `)` `:` functional-type(operands, results) prop-dict attr-dict
+  }];
+}
+
 def TestOpUsingPropertyRefInCustom : TEST_Op<"using_property_ref_in_custom"> {
   let assemblyFormat = "custom<IntProperty>($first) `+` custom<SumProperty>($second, ref($first)) attr-dict";
   let arguments = (ins IntProp<"int64_t">:$first, IntProp<"int64_t">:$second);
diff --git a/mlir/tools/mlir-tblgen/OpFormatGen.cpp b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
index cbcbc8e9bc102..f9d07f6b3ba0f 100644
--- a/mlir/tools/mlir-tblgen/OpFormatGen.cpp
+++ b/mlir/tools/mlir-tblgen/OpFormatGen.cpp
@@ -1313,6 +1313,38 @@ ::mlir::MLIRContext *ctx = dict.getContext();
 (void)ctx;
 )decl";
 
+  // `operandSegmentSizes`/`resultSegmentSizes` are trait-injected properties
+  // not enumerated by `op.getProperties()`, so they need to be special-cased
+  // here, mirroring the handling in `setPropertiesFromAttr` in
+  // OpDefinitionsGen.cpp.
+  //
+  // {0}: segment sizes property name
+  // {1}: isRequired
+  const char *segmentSizesFromAttrFmt = R"decl(
+auto {0}AttrName = ::mlir::StringAttr::get(ctx, "{0}");
+usedKeys.insert({0}AttrName);
+auto attr = dict.get({0}AttrName);
+if (!attr && {1}) {{
+  emitError() << "expected key entry for {0} in DictionaryAttr to set "
+             "Properties.";
+  return ::mlir::failure();
+}
+if (attr && ::mlir::failed(::mlir::convertFromAttribute(prop.{0}, attr, [&]() {{
+      return emitError() << "for `{0}`: ";
+    })))
+  return ::mlir::failure();
+)decl";
+  if (op.getTrait("::mlir::OpTrait::AttrSizedOperandSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "operandSegmentSizes",
+                    fmt.allOperands);
+  }
+  if (op.getTrait("::mlir::OpTrait::AttrSizedResultSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "resultSegmentSizes",
+                    fmt.allResultTypes);
+  }
+
   // {0}: fromAttribute call
   // {1}: property name
   // {2}: isRequired

``````````

</details>


https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 03:10:10 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 22 Jul 2026 03:10:10 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a609702.824e11a6.322560.a5a3@mx.google.com>


https://github.com/chandraghale updated https://github.com/llvm/llvm-project/pull/206747

>From 82416bc775c4befdbcce1f999a5f5b97ef954cd8 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 10:00:56 -0500
Subject: [PATCH 1/3] Lower scan directive and inscan reduction modifier

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             |  57 ++-
 .../Lower/OpenMP/Todo/nested-wsloop-scan.f90  |  34 ++
 .../OpenMP/Todo/wsloop-scan-collapse.f90      |  29 ++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 476 +++++++++++++++---
 .../Target/LLVMIR/openmp-reduction-scan.mlir  | 123 +++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  41 +-
 openmp/runtime/test/scan/scan.f90             |  38 ++
 7 files changed, 726 insertions(+), 72 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
 create mode 100644 openmp/runtime/test/scan/scan.f90

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..e77584a142eea 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2972,12 +2972,59 @@ genParallelOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
 static mlir::omp::ScanOp
 genScanOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-          semantics::SemanticsContext &semaCtx, mlir::Location loc,
-          const ConstructQueue &queue, ConstructQueue::const_iterator item) {
+          semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
+          mlir::Location loc, const ConstructQueue &queue,
+          ConstructQueue::const_iterator item) {
   mlir::omp::ScanOperands clauseOps;
   genScanClauses(converter, semaCtx, item->clauses, loc, clauseOps);
-  return mlir::omp::ScanOp::create(converter.getFirOpBuilder(),
-                                   converter.getCurrentLocation(), clauseOps);
+  mlir::omp::ScanOp scanOp = mlir::omp::ScanOp::create(
+      converter.getFirOpBuilder(), converter.getCurrentLocation(), clauseOps);
+
+  /// Scan reduction is not implemented with nested workshare loops, linear
+  /// clause, tiling
+  mlir::omp::LoopNestOp loopNestOp =
+      scanOp->getParentOfType<mlir::omp::LoopNestOp>();
+  llvm::SmallVector<mlir::omp::LoopWrapperInterface> loopWrappers;
+  loopNestOp.gatherWrappers(loopWrappers);
+  mlir::Operation *loopWrapperOp = loopWrappers.front().getOperation();
+  if (llvm::isa<mlir::omp::SimdOp>(loopWrapperOp))
+    TODO(loc, "unsupported simd");
+  if (loopWrappers.size() > 1)
+    TODO(loc, "unsupported composite");
+  mlir::omp::WsloopOp wsLoopOp = llvm::cast<mlir::omp::WsloopOp>(loopWrapperOp);
+  bool isNested =
+      (loopNestOp.getNumLoops() > 1) ||
+      (wsLoopOp && (wsLoopOp->getParentOfType<mlir::omp::WsloopOp>()));
+  if (isNested)
+    TODO(loc, "Scan directive inside nested workshare loops");
+  if (wsLoopOp && !wsLoopOp.getLinearVars().empty())
+    TODO(loc, "Scan directive with linear clause");
+  if (loopNestOp.getTileSizes())
+    TODO(loc, "Scan directive with loop tiling");
+
+  // All loop indices should be loaded after the scan construct as otherwise,
+  // it would result in using the index variable across scan directive.
+  // (`Intra-iteration dependences from a statement in the structured
+  // block sequence that precede a scan directive to a statement in the
+  // structured block sequence that follows a scan directive must not exist,
+  // except for dependences for the list items specified in an inclusive or
+  // exclusive clause.`).
+  // TODO: Nested loops are not handled.
+  mlir::Region &region = loopNestOp->getRegion(0);
+  mlir::Value indexVal = fir::getBase(region.getArgument(0));
+  lower::pft::Evaluation *doConstructEval = eval.parentConstruct;
+  fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
+  lower::pft::Evaluation *doLoop = &doConstructEval->getFirstNestedEvaluation();
+  auto *doStmt = doLoop->getIf<parser::NonLabelDoStmt>();
+  assert(doStmt && "Expected do loop to be in the nested evaluation");
+  const auto &loopControl =
+      std::get<std::optional<parser::LoopControl>>(doStmt->t);
+  const parser::LoopControl::Bounds *bounds =
+      std::get_if<parser::LoopControl::Bounds>(&loopControl->u);
+  mlir::Operation *storeOp =
+      setLoopVar(converter, loc, indexVal, bounds->Name().thing.symbol);
+  firOpBuilder.setInsertionPointAfter(storeOp);
+  return scanOp;
 }
 
 static mlir::omp::SectionsOp
@@ -4305,7 +4352,7 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
                                   loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_scan:
-    newOp = genScanOp(converter, symTable, semaCtx, loc, queue, item);
+    newOp = genScanOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_section:
     llvm_unreachable("genOMPDispatch: OMPD_section");
diff --git a/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90 b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
new file mode 100644
index 0000000000000..414e2ef2d5aed
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
@@ -0,0 +1,34 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_scan_example
+  implicit none
+  integer, parameter :: n = 4, m = 5
+  integer :: a(n, m), b(n, m)
+  integer :: i, j
+  integer :: row_sum, col_sum
+
+  do i = 1, n
+     do j = 1, m
+        a(i, j) = i + j
+     end do
+  end do
+
+  !$omp parallel do reduction(inscan, +: row_sum) private(col_sum, j)
+  do i = 1, n
+     row_sum = row_sum + i
+     !$omp scan inclusive(row_sum)
+
+     col_sum = 0
+     !$omp parallel do reduction(inscan, +: col_sum)
+     do j = 1, m
+        col_sum = col_sum + a(i, j)
+        !CHECK: not yet implemented: Scan directive inside nested workshare loops
+        !$omp scan inclusive(col_sum)
+        b(i, j) = col_sum + row_sum 
+     end do
+     !$omp end parallel do
+  end do
+  !$omp end parallel do
+end program nested_scan_example
diff --git a/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90 b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
new file mode 100644
index 0000000000000..b8e6e831884ab
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
@@ -0,0 +1,29 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_loop_example
+  implicit none
+  integer :: i, j, x
+  integer, parameter :: N = 100, M = 200
+  real :: A(N, M), B(N, M)
+  x = 0
+
+  do i = 1, N
+    do j = 1, M
+      A(i, j) = i * j
+    end do
+  end do
+  
+  !$omp parallel do collapse(2) reduction(inscan, +:x)
+  do i = 1, N
+    do j = 1, M
+      x = x + A(i,j)
+      !CHECK: not yet implemented: Scan directive inside nested workshare loops
+      !$omp scan inclusive(x)
+      B(i, j) = x
+    end do
+  end do
+  !$omp end parallel do
+
+end program nested_loop_example
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d860c408e0fd2..297ebac51de88 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -41,6 +41,8 @@
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 
+#include <cassert>
+#include <cstddef>
 #include <cstdint>
 #include <iterator>
 #include <numeric>
@@ -84,6 +86,10 @@ class OpenMPAllocStackFrame
       : allocInsertPoint(allocaIP), deallocBlocks(deallocBlocks) {}
   llvm::OpenMPIRBuilder::InsertPointTy allocInsertPoint;
   llvm::SmallVector<llvm::BasicBlock *> deallocBlocks;
+  /// Set to true when this alloca frame encloses an omp.parallel operation.
+  /// The alloca insertion point of a function in which a parallel op is
+  /// defined may be used to allocate the temporary buffer for scan reductions.
+  bool containsParallelOp = false;
 };
 
 /// Stack frame to hold a \see llvm::CanonicalLoopInfo representing the
@@ -93,7 +99,17 @@ class OpenMPLoopInfoStackFrame
     : public StateStackFrameBase<OpenMPLoopInfoStackFrame> {
 public:
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(OpenMPLoopInfoStackFrame)
+  /// For constructs like `scan`, a single `omp.loop_nest` is split into an
+  /// input loop and a scan loop. In that case `loopInfo` holds the input loop
+  /// info and `scanloopInfo` holds the scan loop info.
   llvm::CanonicalLoopInfo *loopInfo = nullptr;
+  llvm::CanonicalLoopInfo *scanloopInfo = nullptr;
+  llvm::ScanInfo *scanInfo = nullptr;
+  /// Map reduction variables to their LLVM types. Populated when the reduction
+  /// clause is processed and used when a `scan` directive is encountered in the
+  /// loop body.
+  std::unique_ptr<llvm::DenseMap<llvm::Value *, llvm::Type *>>
+      reductionVarToType = nullptr;
 };
 
 /// Custom error class to signal translation errors that don't need reporting,
@@ -359,6 +375,10 @@ static LogicalResult checkImplementationStatus(Operation &op) {
     if (!op.getDependVars().empty() || op.getDependKinds())
       result = todo("depend");
   };
+  auto checkExclusive = [&todo](auto op, LogicalResult &result) {
+    if (!op.getExclusiveVars().empty())
+      result = todo("exclusive");
+  };
   auto checkHint = [](auto op, LogicalResult &) {
     if (op.getHint())
       op.emitWarning("hint clause discarded");
@@ -389,21 +409,27 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         op.getReductionMod().value() != omp::ReductionModifier::defaultmod) {
       omp::ReductionModifier mod = op.getReductionMod().value();
       // The `task` reduction modifier is supported on the parallel and
-      // worksharing (do/for and sections) constructs. Other modifiers, and the
-      // `task` modifier on other constructs, are not yet implemented.
+      // worksharing (do/for and sections) constructs. The `inscan` modifier is
+      // supported on the worksharing-loop construct (it is translated as a scan
+      // reduction). Other modifiers, and these modifiers on other constructs,
+      // are not yet implemented.
       bool taskModifierSupported =
           mod == omp::ReductionModifier::task &&
           isa<omp::ParallelOp, omp::WsloopOp, omp::SectionsOp>(op);
-      if (!taskModifierSupported) {
+      bool inscanModifierSupported =
+          mod == omp::ReductionModifier::inscan && isa<omp::WsloopOp>(op);
+      if (!taskModifierSupported && !inscanModifierSupported) {
         result = todo("reduction with modifier");
-      } else if (auto byref = op.getReductionByref()) {
-        // The task reduction modifier lowering only handles non-byref
-        // reductions for now.
-        for (bool isByRef : *byref)
-          if (isByRef) {
-            result = todo("task reduction modifier with by-ref reduction");
-            break;
-          }
+      } else if (taskModifierSupported) {
+        if (auto byref = op.getReductionByref()) {
+          // The task reduction modifier lowering only handles non-byref
+          // reductions for now.
+          for (bool isByRef : *byref)
+            if (isByRef) {
+              result = todo("task reduction modifier with by-ref reduction");
+              break;
+            }
+        }
       }
     }
   };
@@ -460,6 +486,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkAllocate(op, result);
         checkOrder(op, result);
       })
+      .Case([&](omp::ScanOp op) { checkExclusive(op, result); })
       .Case([&](omp::SectionsOp op) {
         checkAllocate(op, result);
         checkPrivate(op, result);
@@ -650,6 +677,97 @@ findCurrentLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
   return loopInfo;
 }
 
+/// Find the scan loop information structure for the scan loop nest being
+/// translated. It will return a `null` value unless called from the
+/// translation function for a loop wrapper operation after successfully
+/// translating its body.
+static llvm::CanonicalLoopInfo *
+findCurrentScanLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::CanonicalLoopInfo *scanLoopInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanLoopInfo = frame.scanloopInfo;
+        return WalkResult::interrupt();
+      });
+  return scanLoopInfo;
+}
+
+/// Find the `ScanInfo` stored on the loop stack frame. Upon encountering an
+/// `inscan` reduction modifier, `scanInfoInitialize` initializes the
+/// `ScanInfo`, which is then used when a `scan` directive is encountered in the
+/// body of the loop nest.
+static llvm::ScanInfo *
+findScanInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::ScanInfo *scanInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanInfo = frame.scanInfo;
+        return WalkResult::interrupt();
+      });
+  return scanInfo;
+}
+
+/// The types of reduction variables are used for lowering a `scan` directive
+/// that appears in the body of the loop. The types are stored in the loop frame
+/// when the reduction clause is encountered and used when the `scan` directive
+/// is encountered.
+static llvm::DenseMap<llvm::Value *, llvm::Type *> *
+findReductionVarTypes(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        if (!frame.reductionVarToType)
+          frame.reductionVarToType =
+              std::make_unique<llvm::DenseMap<llvm::Value *, llvm::Type *>>();
+        reductionVarToType = frame.reductionVarToType.get();
+        return WalkResult::interrupt();
+      });
+  return reductionVarToType;
+}
+
+/// Scan reduction requires a shared buffer to be allocated to perform the
+/// reduction. The allocation needs to be done outside the parallel region in
+/// which the scan operation is used.
+static llvm::OpenMPIRBuilder::InsertPointTy
+findParallelAllocaIP(llvm::IRBuilderBase &builder,
+                     LLVM::ModuleTranslation &moduleTranslation) {
+  // If there is an alloca insertion point on the stack belonging to a frame
+  // that encloses a parallel op, use it.
+  llvm::OpenMPIRBuilder::InsertPointTy allocaInsertPoint;
+  WalkResult walkResult = moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (frame.containsParallelOp) {
+          allocaInsertPoint = frame.allocInsertPoint;
+          return WalkResult::interrupt();
+        }
+        return WalkResult::skip();
+      });
+  if (walkResult.wasInterrupted())
+    return allocaInsertPoint;
+  // Otherwise, insert into the entry block of the surrounding function.
+  // If the current IRBuilder InsertPoint is the function's entry, it cannot
+  // also be used for alloca insertion which would result in insertion order
+  // confusion. Create a new BasicBlock for the Builder and use the entry block
+  // for the allocs.
+  // TODO: Create a dedicated alloca BasicBlock at function creation such that
+  // we do not need to move the current InsertPoint here.
+  if (builder.GetInsertBlock() ==
+      &builder.GetInsertBlock()->getParent()->getEntryBlock()) {
+    assert(builder.GetInsertPoint() == builder.GetInsertBlock()->end() &&
+           "Assuming end of basic block");
+    llvm::BasicBlock *entryBB = llvm::BasicBlock::Create(
+        builder.getContext(), "entry", builder.GetInsertBlock()->getParent(),
+        builder.GetInsertBlock()->getNextNode());
+    builder.CreateBr(entryBB);
+    builder.SetInsertPoint(entryBB);
+  }
+
+  llvm::BasicBlock &funcEntryBlock =
+      builder.GetInsertBlock()->getParent()->getEntryBlock();
+  return llvm::OpenMPIRBuilder::InsertPointTy(
+      &funcEntryBlock, funcEntryBlock.getFirstInsertionPt());
+}
+
 /// Converts the given region that appears within an OpenMP dialect operation to
 /// LLVM IR, creating a branch from the `sourceBlock` to the entry block of the
 /// region, and a branch from any block with an successor-less OpenMP terminator
@@ -1414,7 +1532,8 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
                   SmallVectorImpl<llvm::Value *> &privateReductionVariables,
                   DenseMap<Value, llvm::Value *> &reductionVariableMap,
                   llvm::ArrayRef<bool> isByRef,
-                  SmallVectorImpl<DeferredStore> &deferredStores) {
+                  SmallVectorImpl<DeferredStore> &deferredStores,
+                  bool isInScanRegion = false) {
   if (op.getNumReductionVars() == 0)
     return success();
 
@@ -1451,11 +1570,17 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
   for (auto [data, addr] : deferredStores)
     builder.CreateStore(data, addr);
 
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
   // Before the loop, store the initial values of reductions into reduction
   // variables. Although this could be done after allocas, we don't want to mess
   // up with the alloca insertion point.
   for (unsigned i = 0; i < op.getNumReductionVars(); ++i) {
     SmallVector<llvm::Value *, 1> phis;
+    llvm::Type *reductionType =
+        moduleTranslation.convertType(reductionDecls[i].getType());
+    if (isInScanRegion && reductionVarToType != nullptr)
+      (*reductionVarToType)[privateReductionVariables[i]] = reductionType;
 
     // map block argument to initializer region
     mapInitializationArgs(op, moduleTranslation, builder, reductionDecls,
@@ -4406,11 +4531,15 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     return failure();
 
   assert(afterAllocas.get()->getSinglePredecessor());
+  bool isInScanRegion =
+      wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                     mlir::omp::ReductionModifier::inscan);
   if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
                                moduleTranslation,
                                afterAllocas.get()->getSinglePredecessor(),
                                reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores)))
+                               reductionVariableMap, isByRef, deferredStores,
+                               isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -4432,6 +4561,10 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   std::optional<omp::ScheduleModifier> scheduleMod = wsloopOp.getScheduleMod();
   bool isSimd = wsloopOp.getScheduleSimd();
   bool loopNeedsBarrier = !wsloopOp.getNowait();
+  // TODO: Linear clause support needs to be enabled for scan reduction.
+  if (isInScanRegion)
+    assert(wsloopOp.getLinearVars().empty() &&
+           "Linear clause support is not enabled with scan reduction");
 
   // The only legal way for the direct parent to be omp.distribute is that this
   // represents 'distribute parallel do'. Otherwise, this is a regular
@@ -4469,21 +4602,93 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   if (failed(handleError(regionBlock, opInst)))
     return failure();
 
+  // Generates the loop body for a worksharing loop, including linear-variable
+  // handling and the call into the OpenMPIRBuilder's worksharing-loop helper.
+  // For scan reductions this lambda is invoked twice: once for the input loop
+  // and once for the scan loop.
+  const auto &&wsloopCodeGen = [&](llvm::CanonicalLoopInfo *loopInfo,
+                                   bool noLoopMode,
+                                   bool inputScanLoop) -> LogicalResult {
+    // Emit Initialization and Update IR for linear variables
+    if (!wsloopOp.getLinearVars().empty()) {
+      linearClauseProcessor.initLinearVar(builder, moduleTranslation,
+                                          loopInfo->getPreheader());
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          moduleTranslation.getOpenMPBuilder()->createBarrier(
+              builder.saveIP(), llvm::omp::OMPD_barrier);
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+      builder.restoreIP(*afterBarrierIP);
+      linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                            loopInfo->getIndVar());
+      linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+    }
+
+    builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
+
+    for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
+      linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
+                                           loopInfo->getLatch(), index);
+
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
+        ompBuilder->applyWorkshareLoop(
+            ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
+            convertToScheduleKind(schedule), chunk, isSimd,
+            scheduleMod == omp::ScheduleModifier::monotonic,
+            scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
+            workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
+
+    if (failed(handleError(wsloopIP, opInst)))
+      return failure();
+
+    // Emit finalization for linear vars.
+    if (!wsloopOp.getLinearVars().empty()) {
+      llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
+      assert(loopInfo->getLastIter() &&
+             "`lastiter` in CanonicalLoopInfo is nullptr");
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
+                                                  loopInfo->getLastIter());
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+
+      builder.restoreIP(oldIP);
+    }
+
+    // For scan reductions, the cancellation finalization callback is popped
+    // only after the scan (second) loop has been generated.
+    if (!inputScanLoop || !isInScanRegion)
+      popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
+
+    return success();
+  };
+
   llvm::CanonicalLoopInfo *loopInfo = findCurrentLoopInfo(moduleTranslation);
 
-  // Emit Initialization and Update IR for linear variables
-  if (!wsloopOp.getLinearVars().empty()) {
-    linearClauseProcessor.initLinearVar(builder, moduleTranslation,
-                                        loopInfo->getPreheader());
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    // Emit the scan reduction combiner between the input loop and the scan
+    // loop, operating on the per-iteration partial values stored in the shared
+    // buffer.
+    auto inputLoopFinishIp = loopInfo->getAfterIP();
+    builder.restoreIP(inputLoopFinishIp);
+    SmallVector<OwningReductionGen> owningReductionGens;
+    SmallVector<OwningAtomicReductionGen> owningAtomicReductionGens;
+    SmallVector<llvm::OpenMPIRBuilder::ReductionInfo, 2> reductionInfos;
+    SmallVector<OwningDataPtrPtrReductionGen> owningReductionGenRefDataPtrGens;
+    collectReductionInfo(wsloopOp, builder, moduleTranslation, reductionDecls,
+                         owningReductionGens, owningAtomicReductionGens,
+                         owningReductionGenRefDataPtrGens,
+                         privateReductionVariables, reductionInfos, isByRef);
+    llvm::BasicBlock *cont = splitBB(builder, false, "omp.scan.loop.cont");
+    llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy redIP =
+        ompBuilder->emitScanReduction(builder.saveIP(), reductionInfos,
+                                      scanInfo);
+    if (failed(handleError(redIP, opInst)))
       return failure();
-    builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
-    linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+
+    builder.restoreIP(*redIP);
+    builder.CreateBr(cont);
   }
 
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
@@ -4502,50 +4707,41 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
       noLoopMode = true;
   }
 
-  for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
-    linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
-                                         loopInfo->getLatch(), index);
-
-  llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
-      ompBuilder->applyWorkshareLoop(
-          ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
-          convertToScheduleKind(schedule), chunk, isSimd,
-          scheduleMod == omp::ScheduleModifier::monotonic,
-          scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
-          workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
-
-  if (failed(handleError(wsloopIP, opInst)))
+  // For scan loops, the input loop does not pop the cancellation finalization
+  // callback; that happens after the scan loop is generated below.
+  bool inputScanLoop = isInScanRegion;
+  if (failed(wsloopCodeGen(loopInfo, noLoopMode, inputScanLoop)))
     return failure();
+  inputScanLoop = false;
 
-  // Emit finalization and in-place rewrites for linear vars.
-  if (!wsloopOp.getLinearVars().empty()) {
-    llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
-    assert(loopInfo->getLastIter() &&
-           "`lastiter` in CanonicalLoopInfo is nullptr");
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
-                                                loopInfo->getLastIter());
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    llvm::CanonicalLoopInfo *scanLoopInfo =
+        findCurrentScanLoopInfo(moduleTranslation);
+    if (failed(wsloopCodeGen(scanLoopInfo, noLoopMode, inputScanLoop)))
+      return failure();
+    SmallVector<Region *> reductionRegions;
+    llvm::transform(reductionDecls, std::back_inserter(reductionRegions),
+                    [](omp::DeclareReductionOp reductionDecl) {
+                      return &reductionDecl.getCleanupRegion();
+                    });
+    if (failed(inlineOmpRegionCleanup(
+            reductionRegions, privateReductionVariables, moduleTranslation,
+            builder, "omp.reduction.cleanup")))
       return failure();
+  } else {
+    // Close the task-reduction scope before the worksharing reduction combine.
+    if (isTaskReductionMod)
+      emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
+                                    moduleTranslation);
 
-    builder.restoreIP(oldIP);
+    // Process the reductions if required.
+    if (failed(createReductionsAndCleanup(
+            wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
+            privateReductionVariables, isByRef, wsloopOp.getNowait(),
+            /*isTeamsReduction=*/false)))
+      return failure();
   }
 
-  // Set the correct branch target for task cancellation
-  popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
-
-  // Close the task-reduction scope before the worksharing reduction combine.
-  if (isTaskReductionMod)
-    emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
-                                  moduleTranslation);
-
-  // Process the reductions if required.
-  if (failed(createReductionsAndCleanup(
-          wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
-          privateReductionVariables, isByRef, wsloopOp.getNowait(),
-          /*isTeamsReduction=*/false)))
-    return failure();
-
   return cleanupPrivateVars(wsloopOp, builder, moduleTranslation,
                             wsloopOp.getLoc(), privateVarsInfo);
 }
@@ -4579,6 +4775,20 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       opInst.getReductionMod() == omp::ReductionModifier::task &&
       opInst.getNumReductionVars() > 0;
 
+  // Mark the enclosing alloca stack frame as containing a parallel op. Scan
+  // reductions use the alloca insertion point of the function enclosing the
+  // parallel region to allocate their shared temporary buffer.
+  bool foundParallelOp = false;
+  moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (foundParallelOp) {
+          frame.containsParallelOp = true;
+          return WalkResult::interrupt();
+        }
+        foundParallelOp = true;
+        return WalkResult::skip();
+      });
+
   auto bodyGenCB =
       [&](InsertPointTy allocaIP, InsertPointTy codeGenIP,
           llvm::ArrayRef<llvm::BasicBlock *> deallocBlocks) -> llvm::Error {
@@ -4972,6 +5182,99 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             privateVarsInfo);
 }
 
+static LogicalResult
+convertOmpScan(Operation &opInst, llvm::IRBuilderBase &builder,
+               LLVM::ModuleTranslation &moduleTranslation) {
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+  auto scanOp = cast<omp::ScanOp>(opInst);
+  bool isInclusive = scanOp.hasInclusiveVars();
+  SmallVector<llvm::Value *> llvmScanVars;
+  SmallVector<llvm::Type *> llvmScanVarsType;
+  mlir::OperandRange mlirScanVars = scanOp.getInclusiveVars();
+  if (!isInclusive)
+    mlirScanVars = scanOp.getExclusiveVars();
+
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
+  for (auto val : mlirScanVars) {
+    llvm::Value *llvmVal = moduleTranslation.lookupValue(val);
+    llvmScanVars.push_back(llvmVal);
+    llvmScanVarsType.push_back((*reductionVarToType)[llvmVal]);
+  }
+  llvm::OpenMPIRBuilder::InsertPointTy allocaIP =
+      findParallelAllocaIP(builder, moduleTranslation);
+  llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+      moduleTranslation.getOpenMPBuilder()->createScan(
+          ompLoc, allocaIP, llvmScanVars, llvmScanVarsType, isInclusive,
+          scanInfo);
+  if (failed(handleError(afterIP, opInst)))
+    return failure();
+  builder.restoreIP(*afterIP);
+  return success();
+}
+
+/// Re-initialize the scan reduction variables to their identity at the start of
+/// each input-loop iteration of a scan reduction. This is required because the
+/// OpenMPIRBuilder stores the per-iteration reduction value into the scan
+/// buffer (`buffer[i] = red`) and `emitScanReduction` then computes the prefix
+/// sum across the buffer. Without resetting `red` to the reduction identity
+/// each iteration, the buffer would hold a running sum and the prefix sum would
+/// be computed twice. This reset only applies to the input loop; in the scan
+/// loop the reduction variable is loaded back from the buffer.
+static LogicalResult
+initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation) {
+  auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>();
+  if (!wsloopOp || !wsloopOp.getReductionMod() ||
+      wsloopOp.getReductionMod().value() != omp::ReductionModifier::inscan)
+    return success();
+
+  // Only reset in the input loop; in the scan loop the reduction variable holds
+  // the value loaded from the buffer and must not be overwritten.
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  if (!scanInfo || !scanInfo->OMPFirstScanLoop)
+    return success();
+
+  unsigned numReductions = wsloopOp.getNumReductionVars();
+  if (numReductions == 0)
+    return success();
+
+  SmallVector<omp::DeclareReductionOp> reductionDecls;
+  collectReductionDecls(wsloopOp, reductionDecls);
+  ArrayRef<bool> isByRef = getIsByRef(wsloopOp.getReductionByref());
+  MutableArrayRef<BlockArgument> reductionArgs =
+      cast<omp::BlockArgOpenMPOpInterface>(wsloopOp.getOperation())
+          .getReductionBlockArgs();
+
+  DenseMap<Value, llvm::Value *> reductionVariableMap;
+  for (unsigned i = 0; i < numReductions; ++i)
+    reductionVariableMap.try_emplace(
+        wsloopOp.getReductionVars()[i],
+        moduleTranslation.lookupValue(reductionArgs[i]));
+
+  for (unsigned i = 0; i < numReductions; ++i) {
+    // Scan reductions only support by-value (scalar) reductions.
+    if (isByRef[i])
+      continue;
+    SmallVector<llvm::Value *, 1> phis;
+    mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
+                          reductionVariableMap, i);
+    if (failed(inlineConvertOmpRegions(
+            reductionDecls[i].getInitializerRegion(),
+            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+      return failure();
+    assert(phis.size() == 1 &&
+           "expected one value to be yielded from the reduction init region");
+    setInsertPointForPossiblyEmptyBlock(builder);
+    builder.CreateStore(phis[0],
+                        moduleTranslation.lookupValue(reductionArgs[i]));
+  }
+  return success();
+}
+
 /// Converts an OpenMP loop nest into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -5004,6 +5307,10 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
 
     // Convert the body of the loop.
     builder.restoreIP(ip);
+    // For scan reductions, reset the reduction variables to their identity at
+    // the start of each input-loop iteration, before the input phase runs.
+    if (failed(initScanReductionVars(loopOp, builder, moduleTranslation)))
+      return llvm::make_error<PreviouslyReportedError>();
     llvm::Expected<llvm::BasicBlock *> regionBlock = convertOmpOpRegions(
         loopOp.getRegion(), "omp.loop_nest.region", builder, moduleTranslation);
     if (!regionBlock)
@@ -5036,6 +5343,46 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
       computeIP = loopInfos.front()->getPreheaderIP();
     }
 
+    // If this loop is the worksharing loop of a scan reduction, generate a
+    // pair of canonical loops (an input loop and a scan loop) instead of a
+    // single loop. The `scan` directive in the body is translated against the
+    // `ScanInfo` recorded here.
+    bool isInScanRegion = false;
+    if (auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>())
+      isInScanRegion =
+          wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                         mlir::omp::ReductionModifier::inscan);
+    if (isInScanRegion) {
+      llvm::Expected<llvm::ScanInfo *> res = ompBuilder->scanInfoInitialize();
+      if (failed(handleError(res, *loopOp)))
+        return failure();
+      llvm::ScanInfo *scanInfo = res.get();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.scanInfo = scanInfo;
+            return WalkResult::interrupt();
+          });
+      llvm::Expected<llvm::SmallVector<llvm::CanonicalLoopInfo *>> loopResults =
+          ompBuilder->createCanonicalScanLoops(
+              loc, bodyGen, lowerBound, upperBound, step,
+              /*IsSigned=*/true, loopOp.getLoopInclusive(), computeIP, "loop",
+              scanInfo);
+
+      if (failed(handleError(loopResults, *loopOp)))
+        return failure();
+      llvm::CanonicalLoopInfo *inputLoop = loopResults.get().front();
+      llvm::CanonicalLoopInfo *scanLoop = loopResults.get().back();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.loopInfo = inputLoop;
+            frame.scanloopInfo = scanLoop;
+            return WalkResult::interrupt();
+          });
+      builder.restoreIP(scanLoop->getAfterIP());
+      // TODO: tiling and collapse are not yet implemented for scan reduction.
+      return success();
+    }
+
     llvm::Expected<llvm::CanonicalLoopInfo *> loopResult =
         ompBuilder->createCanonicalLoop(
             loc, bodyGen, lowerBound, upperBound, step,
@@ -9454,6 +9801,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::WsloopOp) {
             return convertOmpWsloop(*op, builder, moduleTranslation);
           })
+          .Case([&](omp::ScanOp) {
+            return convertOmpScan(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::SimdOp) {
             return convertOmpSimd(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
new file mode 100644
index 0000000000000..c6b8daccad313
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
@@ -0,0 +1,123 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+omp.declare_reduction @add_reduction_i32 : i32 init {
+^bb0(%arg0: i32):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%0 : i32)
+} combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %0 = llvm.add %arg0, %arg1 : i32
+  omp.yield(%0 : i32)
+}
+// CHECK-LABEL: @scan_reduction
+llvm.func @scan_reduction() {
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 {bindc_name = "z"} : (i64) -> !llvm.ptr
+  %3 = llvm.alloca %0 x i32 {bindc_name = "y"} : (i64) -> !llvm.ptr
+  %5 = llvm.alloca %0 x i32 {bindc_name = "x"} : (i64) -> !llvm.ptr
+  %7 = llvm.alloca %0 x i32 {bindc_name = "k"} : (i64) -> !llvm.ptr
+  %10 = llvm.mlir.constant(100 : i32) : i32
+  %11 = llvm.mlir.constant(1 : i32) : i32
+  %12 = llvm.mlir.constant(0 : i32) : i32
+  %13 = llvm.mlir.constant(100 : index) : i64
+  %14 = llvm.mlir.addressof @_QFEa : !llvm.ptr
+  %15 = llvm.mlir.addressof @_QFEb : !llvm.ptr
+  omp.parallel {
+    %37 = llvm.mlir.constant(1 : i64) : i64
+    %38 = llvm.alloca %37 x i32 {bindc_name = "k", pinned} : (i64) -> !llvm.ptr
+    %39 = llvm.mlir.constant(1 : i64) : i64
+    omp.wsloop reduction(mod: inscan, @add_reduction_i32 %5 -> %arg0 : !llvm.ptr) {
+      omp.loop_nest (%arg1) : i32 = (%11) to (%10) inclusive step (%11) {
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %40 = llvm.load %arg0 : !llvm.ptr -> i32
+        %41 = llvm.load %38 : !llvm.ptr -> i32
+        %42 = llvm.sext %41 : i32 to i64
+        %50 = llvm.getelementptr %14[%42] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        %51 = llvm.load %50 : !llvm.ptr -> i32
+        %52 = llvm.add %40, %51 : i32
+        llvm.store %52, %arg0 : i32, !llvm.ptr
+        omp.scan inclusive(%arg0 : !llvm.ptr)
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %53 = llvm.load %arg0 : !llvm.ptr -> i32
+        %54 = llvm.load %38 : !llvm.ptr -> i32
+        %55 = llvm.sext %54 : i32 to i64
+        %63 = llvm.getelementptr %15[%55] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        llvm.store %53, %63 : i32, !llvm.ptr
+        omp.yield
+      }
+    }
+    omp.terminator
+  }
+  llvm.return
+}
+llvm.mlir.global internal @_QFEa() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+//CHECK: %vla = alloca ptr, align 8
+//CHECK: omp_parallel
+//CHECK: store ptr %vla, ptr %gep_vla, align 8
+//CHECK: @__kmpc_fork_call
+//CHECK: void @scan_reduction..omp_par
+//CHECK: %[[BUFF_PTR:.+]] = load ptr, ptr %gep_vla
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: omp.scan.loop.cont:
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: %[[FREE_VAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:  %[[ARRLAST:.+]] = getelementptr inbounds i32, ptr %[[FREE_VAR]], i32 100
+//CHECK:  %[[RES:.+]] = load i32, ptr %[[ARRLAST]], align 4
+//CHECK:  store i32 %[[RES]], ptr %loadgep{{.*}}, align 4
+//CHECK: tail call void @free(ptr %[[FREE_VAR]])
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inscan.dispatch{{.*}}:                            ; preds = %omp_loop.body{{.*}}
+//CHECK:   %[[BUFFVAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:   %[[arrayOffset1:.+]] = getelementptr inbounds i32, ptr %[[BUFFVAR]], i32 %{{.*}}
+//CHECK:   %[[BUFFVAL1:.+]] = load i32, ptr %[[arrayOffset1]], align 4
+//CHECK:   store i32 %[[BUFFVAL1]], ptr %{{.*}}, align 4
+//CHECK:   %[[LOG:.+]] = call double @llvm.log2.f64(double 1.000000e+02) #0
+//CHECK:   %[[CEIL:.+]] = call double @llvm.ceil.f64(double %[[LOG]]) #0
+//CHECK:   %[[UB:.+]] = fptoui double %[[CEIL]] to i32
+//CHECK:   br label %omp.outer.log.scan.body
+//CHECK: omp.outer.log.scan.body:
+//CHECK:   %[[K:.+]] = phi i32 [ 0, %{{.*}} ], [ %[[NEXTK:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[I:.+]] = phi i32 [ 1, %{{.*}} ], [ %[[NEXTI:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[CMP1:.+]] = icmp uge i32 99, %[[I]]
+//CHECK:   br i1 %[[CMP1]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inner.log.scan.exit:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[NEXTK]] = add nuw i32 %[[K]], 1
+//CHECK:   %[[NEXTI]] = shl nuw i32 %[[I]], 1
+//CHECK:   %[[CMP2:.+]] = icmp ne i32 %[[NEXTK]], %[[UB]]
+//CHECK:   br i1 %[[CMP2]], label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
+//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inner.log.scan.body:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[CNT:.+]] = phi i32 [ 99, %omp.outer.log.scan.body ], [ %[[CNTNXT:.+]], %omp.inner.log.scan.body ]
+//CHECK:   %[[BUFF:.+]] = load ptr, ptr %[[BUFF_PTR]]
+//CHECK:   %[[IND1:.+]] = add i32 %[[CNT]], 1
+//CHECK:   %[[IND1PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND1]]
+//CHECK:   %[[IND2:.+]] = sub nuw i32 %[[IND1]], %[[I]]
+//CHECK:   %[[IND2PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND2]]
+//CHECK:   %[[IND1VAL:.+]] = load i32, ptr %[[IND1PTR]], align 4
+//CHECK:   %[[IND2VAL:.+]] = load i32, ptr %[[IND2PTR]], align 4
+//CHECK:   %[[REDVAL:.+]] = add i32 %[[IND1VAL]], %[[IND2VAL]]
+//CHECK:   store i32 %[[REDVAL]], ptr %[[IND1PTR]], align 4
+//CHECK:   %[[CNTNXT]] = sub nuw i32 %[[CNT]], 1
+//CHECK:   %[[CMP3:.+]] = icmp uge i32 %[[CNTNXT]], %[[I]]
+//CHECK:   br i1 %[[CMP3]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inscan.dispatch:                              ; preds = %omp_loop.body
+//CHECK:   br i1 true, label %omp.before.scan.bb, label %omp.after.scan.bb
+//CHECK: omp.before.scan.bb:
+//CHECK:   store i32 0, ptr %[[REDPRIV:.+]], align 4
+//CHECK: omp.loop_nest.region:                             ; preds = %omp.before.scan.bb
+//CHECK:   %[[BUFFER:.+]] = load ptr, ptr %loadgep_vla, align 8
+//CHECK:   %[[ARRAYOFFSET2:.+]] = getelementptr inbounds i32, ptr %[[BUFFER]], i32 %{{.*}}
+//CHECK-NEXT:   %[[REDPRIVVAL:.+]] = load i32, ptr %{{.*}}, align 4
+//CHECK:   store i32 %[[REDPRIVVAL]], ptr %[[ARRAYOFFSET2]], align 4
+//CHECK:   br label %omp.scan.loop.exit
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..7edf88b5ea559 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -120,10 +120,10 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.wsloop operation}}
-  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
-  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+llvm.func @simd_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.simd operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       omp.scan inclusive(%prv : !llvm.ptr)
       omp.yield
@@ -162,6 +162,39 @@ llvm.func @parallel_task_reduction_modifier_byref(%x : !llvm.ptr) {
 
 // -----
 
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> f32
+  llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
+  omp.yield
+}
+llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
+  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+    // expected-error at below {{LLVM Translation failed for operation: omp.loop_nest}}
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      // expected-error at below {{not yet implemented: Unhandled clause exclusive in omp.scan operation}}
+      // expected-error at below {{LLVM Translation failed for operation: omp.scan}}
+      omp.scan exclusive(%prv : !llvm.ptr)
+      omp.yield
+    }
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @single_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.single operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.single}}
diff --git a/openmp/runtime/test/scan/scan.f90 b/openmp/runtime/test/scan/scan.f90
new file mode 100644
index 0000000000000..76263508105d0
--- /dev/null
+++ b/openmp/runtime/test/scan/scan.f90
@@ -0,0 +1,38 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+program inclusive_scan
+ implicit none
+ integer, parameter :: n = 100
+ integer a(n), b(n)
+ integer x, k, y, z
+
+ ! initialization
+ x = 0
+ do k = 1, n
+  a(k) = k
+ end do
+
+ ! a(k) is included in the computation of producing results in b(k)
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, n
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+
+ print *,'x =', x
+ do k = 1, 10
+  print *, 'b(', k, ') =', b(k)
+ end do
+end program
+!CHECK: x = 5050
+!CHECK: b( 1 ) = 1
+!CHECK: b( 2 ) = 3
+!CHECK: b( 3 ) = 6
+!CHECK: b( 4 ) = 10
+!CHECK: b( 5 ) = 15
+!CHECK: b( 6 ) = 21
+!CHECK: b( 7 ) = 28
+!CHECK: b( 8 ) = 36
+!CHECK: b( 9 ) = 45
+!CHECK: b( 10 ) = 55

>From 4011dd537d9830ee4a0379cba80413d1e4cdf9c3 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 02:44:02 -0500
Subject: [PATCH 2/3] formatting correction

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp        | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 297ebac51de88..e18cc0df79860 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4534,12 +4534,11 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isInScanRegion =
       wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
                                      mlir::omp::ReductionModifier::inscan);
-  if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
-                               moduleTranslation,
-                               afterAllocas.get()->getSinglePredecessor(),
-                               reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores,
-                               isInScanRegion)))
+  if (failed(initReductionVars(
+          wsloopOp, reductionArgs, builder, moduleTranslation,
+          afterAllocas.get()->getSinglePredecessor(), reductionDecls,
+          privateReductionVariables, reductionVariableMap, isByRef,
+          deferredStores, isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -5262,9 +5261,9 @@ initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
     SmallVector<llvm::Value *, 1> phis;
     mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
                           reductionVariableMap, i);
-    if (failed(inlineConvertOmpRegions(
-            reductionDecls[i].getInitializerRegion(),
-            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+    if (failed(inlineConvertOmpRegions(reductionDecls[i].getInitializerRegion(),
+                                       "omp.scan.reduction.init", builder,
+                                       moduleTranslation, &phis)))
       return failure();
     assert(phis.size() == 1 &&
            "expected one value to be yielded from the reduction init region");

>From 64b530d3f37e0cd08ca622b538d9bec1eb348b50 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Wed, 22 Jul 2026 05:09:34 -0500
Subject: [PATCH 3/3] Address review comments

---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 46 +++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 26 +++++++-
 .../Target/LLVMIR/openmp-reduction-scan.mlir  | 14 +++--
 mlir/test/Target/LLVMIR/openmp-todo.mlir      | 31 ++++++++++
 openmp/runtime/test/scan/scan-bounds.f90      | 61 +++++++++++++++++++
 openmp/runtime/test/scan/scan-edge.f90        | 61 +++++++++++++++++++
 openmp/runtime/test/scan/scan-nonidentity.f90 | 28 +++++++++
 7 files changed, 251 insertions(+), 16 deletions(-)
 create mode 100644 openmp/runtime/test/scan/scan-bounds.f90
 create mode 100644 openmp/runtime/test/scan/scan-edge.f90
 create mode 100644 openmp/runtime/test/scan/scan-nonidentity.f90

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..f62ea4cf2f9da 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -5307,7 +5307,17 @@ Error OpenMPIRBuilder::emitScanBasedDirectiveFinalsIR(
                                              "arrayOffset");
       Value *Src = Builder.CreateLoad(SrcTy, Val);
 
-      Builder.CreateStore(Src, OrigVar);
+      // For a zero-trip loop the buffer slot is uninitialized; keep the
+      // original variable's incoming value instead of storing garbage. The
+      // load above stays in bounds because the buffer was allocated with
+      // `Span + 1` elements, so index `Span` is always valid.
+      Value *Cur = Builder.CreateLoad(SrcTy, OrigVar);
+      Value *HasIters = Builder.CreateICmpUGT(
+          ScanRedInfo->Span,
+          llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 0));
+      Value *Final = Builder.CreateSelect(HasIters, Src, Cur);
+
+      Builder.CreateStore(Final, OrigVar);
       Builder.CreateFree(Buff);
     }
     return Error::success();
@@ -5370,7 +5380,15 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::emitScanReduction(
         ScanRedInfo->Span,
         llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 1));
     Builder.SetInsertPoint(InputBB);
-    Builder.CreateBr(LoopBB);
+    // The log-scan prefix computation is only well-defined when there are at
+    // least two elements: `log2(Span)` is invalid for Span == 0 and the loop
+    // trip computation underflows/loops forever for Span <= 1. For Span <= 1
+    // the buffer already holds the final value(s), so skip straight to the
+    // exit.
+    llvm::Value *SpanGuard = Builder.CreateICmpUGT(
+        ScanRedInfo->Span,
+        llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 1));
+    Builder.CreateCondBr(SpanGuard, LoopBB, ExitBB);
     emitBlock(LoopBB, CurFn);
     Builder.SetInsertPoint(LoopBB);
 
@@ -5617,7 +5635,6 @@ OpenMPIRBuilder::createCanonicalScanLoops(
 
   auto BodyGen = [=](InsertPointTy CodeGenIP, Value *IV) {
     Builder.restoreIP(CodeGenIP);
-    ScanRedInfo->IV = IV;
     createScanBBs(ScanRedInfo);
     BasicBlock *InputBlock = Builder.GetInsertBlock();
     Instruction *Terminator = InputBlock->getTerminator();
@@ -5636,9 +5653,13 @@ OpenMPIRBuilder::createCanonicalScanLoops(
   };
 
   const auto &&InputLoopGen = [&]() -> Error {
+    // Pass an empty ComputeIP: the trip count must be (re)computed at the
+    // loop's own location. Reusing the outer ComputeIP is invalid here because
+    // it was invalidated by the `scan.init` split above and would emit the
+    // recomputed trip count after a block terminator for runtime bounds.
     Expected<CanonicalLoopInfo *> LoopInfo = createCanonicalLoop(
         Builder.saveIP(), BodyGen, Start, Stop, Step, IsSigned, InclusiveStop,
-        ComputeIP, Name, true, ScanRedInfo);
+        /*ComputeIP=*/InsertPointTy(), Name, true, ScanRedInfo);
     if (!LoopInfo)
       return LoopInfo.takeError();
     Result.push_back(*LoopInfo);
@@ -5646,9 +5667,9 @@ OpenMPIRBuilder::createCanonicalScanLoops(
     return Error::success();
   };
   const auto &&ScanLoopGen = [&](LocationDescription Loc) -> Error {
-    Expected<CanonicalLoopInfo *> LoopInfo =
-        createCanonicalLoop(Loc, BodyGen, Start, Stop, Step, IsSigned,
-                            InclusiveStop, ComputeIP, Name, true, ScanRedInfo);
+    Expected<CanonicalLoopInfo *> LoopInfo = createCanonicalLoop(
+        Loc, BodyGen, Start, Stop, Step, IsSigned, InclusiveStop,
+        /*ComputeIP=*/InsertPointTy(), Name, true, ScanRedInfo);
     if (!LoopInfo)
       return LoopInfo.takeError();
     Result.push_back(*LoopInfo);
@@ -5737,8 +5758,15 @@ Expected<CanonicalLoopInfo *> OpenMPIRBuilder::createCanonicalLoop(
     Builder.restoreIP(CodeGenIP);
     Value *Span = Builder.CreateMul(IV, Step);
     Value *IndVar = Builder.CreateAdd(Span, Start);
-    if (InScan)
-      ScanRedInfo->IV = IndVar;
+    if (InScan) {
+      // The scan buffer is 1-indexed by iteration number (buffer[1..TripCount])
+      // rather than by the source induction variable. Using the source value
+      // would index out of bounds for loops that do not start at 1 or that use
+      // a non-unit stride. `IV` here is the 0-based canonical loop counter, so
+      // the 1-based buffer index is `IV + 1`.
+      ScanRedInfo->IV =
+          Builder.CreateAdd(IV, ConstantInt::get(IV->getType(), 1));
+    }
     return BodyGenCB(Builder.saveIP(), IndVar);
   };
   LocationDescription LoopLoc =
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e18cc0df79860..5af1a384d3edb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -430,6 +430,16 @@ static LogicalResult checkImplementationStatus(Operation &op) {
               break;
             }
         }
+      } else if (inscanModifierSupported) {
+        if (auto byref = op.getReductionByref()) {
+          // Scan reductions (inscan modifier) only support by-value (scalar)
+          // reductions for now.
+          for (bool isByRef : *byref)
+            if (isByRef) {
+              result = todo("inscan reduction modifier with by-ref reduction");
+              break;
+            }
+        }
       }
     }
   };
@@ -5268,8 +5278,20 @@ initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
     assert(phis.size() == 1 &&
            "expected one value to be yielded from the reduction init region");
     setInsertPointForPossiblyEmptyBlock(builder);
-    builder.CreateStore(phis[0],
-                        moduleTranslation.lookupValue(reductionArgs[i]));
+
+    // On the first iteration, seed the reduction variable with the original
+    // variable's incoming value so that a non-identity initial value
+    // participates in the scan (e.g. `x = 10` before an inclusive-scan loop of
+    // `+1` must yield 11, 12, ...). On subsequent iterations, reset to the
+    // reduction identity. `scanInfo->IV` is the 1-based iteration index.
+    llvm::Value *identity = phis[0];
+    llvm::Value *origVar =
+        moduleTranslation.lookupValue(wsloopOp.getReductionVars()[i]);
+    llvm::Value *origVal = builder.CreateLoad(identity->getType(), origVar);
+    llvm::Value *isFirstIter = builder.CreateICmpEQ(
+        scanInfo->IV, llvm::ConstantInt::get(scanInfo->IV->getType(), 1));
+    llvm::Value *seed = builder.CreateSelect(isFirstIter, origVal, identity);
+    builder.CreateStore(seed, moduleTranslation.lookupValue(reductionArgs[i]));
   }
   return success();
 }
diff --git a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
index c6b8daccad313..6ef58b5013e08 100644
--- a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
@@ -73,7 +73,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK: %[[FREE_VAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
 //CHECK:  %[[ARRLAST:.+]] = getelementptr inbounds i32, ptr %[[FREE_VAR]], i32 100
 //CHECK:  %[[RES:.+]] = load i32, ptr %[[ARRLAST]], align 4
-//CHECK:  store i32 %[[RES]], ptr %loadgep{{.*}}, align 4
+//CHECK:  %[[CUR:.+]] = load i32, ptr %loadgep{{.*}}, align 4
+//CHECK:  %[[FIN:.+]] = select i1 {{.*}}, i32 %[[RES]], i32 %[[CUR]]
+//CHECK:  store i32 %[[FIN]], ptr %loadgep{{.*}}, align 4
 //CHECK: tail call void @free(ptr %[[FREE_VAR]])
 //CHECK: @__kmpc_end_masked
 //CHECK: omp.inscan.dispatch{{.*}}:                            ; preds = %omp_loop.body{{.*}}
@@ -84,7 +86,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK:   %[[LOG:.+]] = call double @llvm.log2.f64(double 1.000000e+02) #0
 //CHECK:   %[[CEIL:.+]] = call double @llvm.ceil.f64(double %[[LOG]]) #0
 //CHECK:   %[[UB:.+]] = fptoui double %[[CEIL]] to i32
-//CHECK:   br label %omp.outer.log.scan.body
+//CHECK:   br i1 {{.*}}, label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
+//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit, %omp_region.body{{.*}}
+//CHECK: @__kmpc_end_masked
 //CHECK: omp.outer.log.scan.body:
 //CHECK:   %[[K:.+]] = phi i32 [ 0, %{{.*}} ], [ %[[NEXTK:.+]], %omp.inner.log.scan.exit ]
 //CHECK:   %[[I:.+]] = phi i32 [ 1, %{{.*}} ], [ %[[NEXTI:.+]], %omp.inner.log.scan.exit ]
@@ -95,8 +99,6 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK:   %[[NEXTI]] = shl nuw i32 %[[I]], 1
 //CHECK:   %[[CMP2:.+]] = icmp ne i32 %[[NEXTK]], %[[UB]]
 //CHECK:   br i1 %[[CMP2]], label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
-//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit
-//CHECK: @__kmpc_end_masked
 //CHECK: omp.inner.log.scan.body:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
 //CHECK:   %[[CNT:.+]] = phi i32 [ 99, %omp.outer.log.scan.body ], [ %[[CNTNXT:.+]], %omp.inner.log.scan.body ]
 //CHECK:   %[[BUFF:.+]] = load ptr, ptr %[[BUFF_PTR]]
@@ -114,7 +116,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK: omp.inscan.dispatch:                              ; preds = %omp_loop.body
 //CHECK:   br i1 true, label %omp.before.scan.bb, label %omp.after.scan.bb
 //CHECK: omp.before.scan.bb:
-//CHECK:   store i32 0, ptr %[[REDPRIV:.+]], align 4
+//CHECK:   %[[ISFIRST:.+]] = icmp eq i32 %{{.*}}, 1
+//CHECK:   %[[SEED:.+]] = select i1 %[[ISFIRST]], i32 %{{.*}}, i32 0
+//CHECK:   store i32 %[[SEED]], ptr %[[REDPRIV:.+]], align 4
 //CHECK: omp.loop_nest.region:                             ; preds = %omp.before.scan.bb
 //CHECK:   %[[BUFFER:.+]] = load ptr, ptr %loadgep_vla, align 8
 //CHECK:   %[[ARRAYOFFSET2:.+]] = getelementptr inbounds i32, ptr %[[BUFFER]], i32 %{{.*}}
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 7edf88b5ea559..fe3c4259e75e0 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -195,6 +195,37 @@ llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // -----
 
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> f32
+  llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
+  omp.yield
+}
+llvm.func @scan_reduction_byref(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{not yet implemented: Unhandled clause inscan reduction modifier with by-ref reduction in omp.wsloop operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
+  omp.wsloop reduction(mod:inscan, byref @add_f32 %x -> %prv : !llvm.ptr) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      omp.scan inclusive(%prv : !llvm.ptr)
+      omp.yield
+    }
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @single_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.single operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.single}}
diff --git a/openmp/runtime/test/scan/scan-bounds.f90 b/openmp/runtime/test/scan/scan-bounds.f90
new file mode 100644
index 0000000000000..9d624954568ff
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-bounds.f90
@@ -0,0 +1,61 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! Arbitrary loop bounds and a non-unit stride must index the scan buffer by the
+! logical iteration number, not by the induction-variable value.
+program scan_bounds
+ implicit none
+ integer, parameter :: n = 30
+ integer :: a(n), b(n)
+ integer :: x, k
+
+ do k = 1, n
+   a(k) = k
+ end do
+
+ ! Arbitrary bounds: iterate from 10 to 20.
+ b = -1
+ x = 0
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 10, 20
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+ print *, 'bounds x =', x
+ do k = 10, 20
+   print *, 'bb(', k, ') =', b(k)
+ end do
+
+ ! Non-unit stride: iterate from 1 to 10 step 2.
+ b = -1
+ x = 0
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, 10, 2
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+ print *, 'stride x =', x
+ do k = 1, 10, 2
+   print *, 'sb(', k, ') =', b(k)
+ end do
+end program
+!CHECK: bounds x = 165
+!CHECK: bb( 10 ) = 10
+!CHECK: bb( 11 ) = 21
+!CHECK: bb( 12 ) = 33
+!CHECK: bb( 13 ) = 46
+!CHECK: bb( 14 ) = 60
+!CHECK: bb( 15 ) = 75
+!CHECK: bb( 16 ) = 91
+!CHECK: bb( 17 ) = 108
+!CHECK: bb( 18 ) = 126
+!CHECK: bb( 19 ) = 145
+!CHECK: bb( 20 ) = 165
+!CHECK: stride x = 25
+!CHECK: sb( 1 ) = 1
+!CHECK: sb( 3 ) = 4
+!CHECK: sb( 5 ) = 9
+!CHECK: sb( 7 ) = 16
+!CHECK: sb( 9 ) = 25
diff --git a/openmp/runtime/test/scan/scan-edge.f90 b/openmp/runtime/test/scan/scan-edge.f90
new file mode 100644
index 0000000000000..7cc00da958e6e
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-edge.f90
@@ -0,0 +1,61 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! Zero-trip loops must not crash and must leave the reduction variable
+! unchanged, and runtime (non-constant) trip counts must work correctly.
+module scan_edge_mod
+contains
+ subroutine run_scan(n, a, b)
+   implicit none
+   integer :: n, k
+   integer :: a(n), b(n)
+   integer :: x
+   x = 0
+   !$omp parallel do reduction(inscan, +: x)
+   do k = 1, n
+     x = x + a(k)
+     !$omp scan inclusive(x)
+     b(k) = x
+   end do
+ end subroutine
+end module
+
+program scan_edge
+ use scan_edge_mod
+ implicit none
+ integer :: a(6), b(6), k
+ integer :: z, zk
+ integer :: zb(1)
+
+ ! Runtime bounds: the trip count of run_scan is not a compile-time constant.
+ do k = 1, 6
+   a(k) = k
+ end do
+ b = -1
+ call run_scan(6, a, b)
+ do k = 1, 6
+   print *, 'rb(', k, ') =', b(k)
+ end do
+
+ ! Runtime zero-trip: n = 0 must not crash.
+ call run_scan(0, a, b)
+ print *, 'runtime zero-trip ok'
+
+ ! Constant zero-trip with a non-identity start: z must be unchanged.
+ z = 42
+ !$omp parallel do reduction(inscan, +: z)
+ do zk = 1, 0
+   z = z + 1
+   !$omp scan inclusive(z)
+   zb(zk) = z
+ end do
+ print *, 'z =', z
+end program
+!CHECK: rb( 1 ) = 1
+!CHECK: rb( 2 ) = 3
+!CHECK: rb( 3 ) = 6
+!CHECK: rb( 4 ) = 10
+!CHECK: rb( 5 ) = 15
+!CHECK: rb( 6 ) = 21
+!CHECK: runtime zero-trip ok
+!CHECK: z = 42
diff --git a/openmp/runtime/test/scan/scan-nonidentity.f90 b/openmp/runtime/test/scan/scan-nonidentity.f90
new file mode 100644
index 0000000000000..064b0d757af6e
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-nonidentity.f90
@@ -0,0 +1,28 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! The reduction variable's non-identity incoming value must be included in the
+! first partial result of an inclusive scan.
+program scan_nonidentity
+ implicit none
+ integer :: x, k
+ integer :: b(4)
+
+ x = 10
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, 4
+   x = x + 1
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+
+ print *, 'x =', x
+ do k = 1, 4
+   print *, 'b(', k, ') =', b(k)
+ end do
+end program
+!CHECK: x = 14
+!CHECK: b( 1 ) = 11
+!CHECK: b( 2 ) = 12
+!CHECK: b( 3 ) = 13
+!CHECK: b( 4 ) = 14


From llvmlistbot at llvm.org  Wed Jul 22 03:11:26 2026
From: llvmlistbot at llvm.org (Natanael Cintean)
Date: Wed, 22 Jul 2026 03:11:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a60974e.0c05e68e.81651.a2a4@mx.google.com>


https://github.com/natanael-cintean edited https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 03:12:46 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 03:12:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Fix ArrayType::getSizeInBytes
 double counting the stride (PR #203821)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203821@github.com>
Message-ID: <6a60979e.743400b0.25573c.e170@mx.google.com>


https://github.com/IgWod approved this pull request.

LGTM, but maybe wait for @kuhar since he wrote the code originally?

https://github.com/llvm/llvm-project/pull/203821

From llvmlistbot at llvm.org  Wed Jul 22 03:21:24 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a4.c5504292.1bbfc8.cba6@mx.google.com>


================
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
+  let summary = "Asynchronous Store Op";
+  let arguments = (ins AnyTypeOf<[LLVM_PointerGlobal, 
+                                  LLVM_PointerSharedCluster]>:$addr,
+                       AnyTypeOf<[I8, I16, I32, I64, I128]>:$value, 
+                       Optional<LLVM_PointerSharedCluster>:$mbarrier,
+                       DefaultValuedAttr<AsyncStoreScopeAttr, 
+                                         "AsyncStoreScope::NONE">:$scope,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_multimem,
+                       DefaultValuedAttr<BoolAttr, "false">:$is_mmio);
----------------
grypp wrote:

nit

```suggestion
                       DefaultValuedAttr<BoolAttr, "false">:$multimem,
                       DefaultValuedAttr<BoolAttr, "false">:$mmio);
```

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:24 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a4.7d702ed2.2f6c05.c30b@mx.google.com>


================
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
----------------
grypp wrote:

should we also add syncronized store op?

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:24 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a4.d76d81af.3b68ca.0847@mx.google.com>


================
@@ -5113,6 +5113,25 @@ def NVVM_BulkStoreOp: NVVM_Op<"st.bulk"> {
   let hasVerifier = 1;
 }
 
+//===----------------------------------------------------------------------===//
+// NVVM Asynchronous Store Op
+//===----------------------------------------------------------------------===//
+
+def NVVM_AsyncStoreOp: NVVM_VoidIntrinsicOp<"store.async", [NVVMRequiresSM<90>]> {
+  let summary = "Asynchronous Store Op";
+  let arguments = (ins AnyTypeOf<[LLVM_PointerGlobal, 
----------------
grypp wrote:

description is missing

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.bc28b987.1c0bcd.719f@mx.google.com>


================
@@ -3662,6 +3706,42 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
----------------
grypp wrote:

```suggestion
  llvm::Value *isMultimem = builder.getInt1(thisOp.getIsMultimem());
```

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.f06f4994.50b1b.da08@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
grypp wrote:

PTX itself splits into 2 insturciotn 
1. st.async 
2. multimem.st.async

Does this work? WDYT?

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.c4d3d80a.16dae5.21ca@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
grypp wrote:

>From the codegen side, splitting by address space doesn't buy us much. Every generator now has to decide which op to emit based on the address space, which just moves the complexity from one central verifier into all the generators. A slightly larger verifier is fine by me — that's exactly where this kind of constraint checking belongs.



https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.13be6df8.323b61.f7f1@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
grypp wrote:

I'd slightly prefer keeping this as a single `st.async` op. From the codegen side, splitting doesn't buy us anything, the generator now has to decide which op to emit based on the address space, which just pushes the complexity from the verifier into every producer of the op. 

A slightly larger verifier is a non-issue to me; that's exactly where this kind of constraint checking belongs.

Plus it's confusing to see multiple OPs while PTX has only one. It's hard to find. 

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.f06f4994.50b1b.da09@mx.google.com>


================
@@ -668,6 +668,50 @@ LogicalResult BulkStoreOp::verify() {
   return success();
 }
 
+LogicalResult AsyncStoreOp::verify() {
+  unsigned addrSpace =
+      llvm::cast<LLVM::LLVMPointerType>(getAddr().getType()).getAddressSpace();
+  NVVM::AsyncStoreScope scope = getScope();
+  mlir::Type valueType = getValue().getType();
+  bool isMmio = getIsMmio();
+  bool isMultimem = getIsMultimem();
+
+  if (addrSpace == NVVMMemorySpace::Global) {
+    if (getMbarrier())
+      return emitOpError("mbarrier is not supported for global address space");
+
+    if (valueType.isInteger(128))
+      return emitOpError("only 8, 16, 32, and 64 bit values are supported for "
+                         "global address space");
+
+    if (scope == AsyncStoreScope::NONE)
+      return emitOpError(
+          "scope must be set for async store to global address space");
+
+    if (isMmio && scope != AsyncStoreScope::SYS)
+      return emitOpError("mmio is only supported for SYS scope");
+
+    if (isMmio && isMultimem)
+      return emitOpError("multimem is not supported for mmio");
+  }
+
+  if (addrSpace == NVVMMemorySpace::SharedCluster) {
+    if (valueType.isInteger(8) || valueType.isInteger(16))
+      return emitOpError("only 32, 64, and 128 bit values are supported for "
+                         "shared cluster address space");
+
+    if (isMultimem || isMmio)
+      return emitOpError("multimem and mmio are not supported for shared "
+                         "cluster address space");
----------------
grypp wrote:

We also hearing complaints about discoverability. For example, PTX has a single `st.async` instruction, but NVVM has multipe OPs with different names. It makes hard to find the op if you don't know nvvm.




https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:21:25 2026
From: llvmlistbot at llvm.org (Guray Ozen)
Date: Wed, 22 Jul 2026 03:21:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][NVVM] Add asynchronous store Op (PR
	#210931)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210931@github.com>
Message-ID: <6a6099a5.7270d2fd.1ec500.5feb@mx.google.com>


================
@@ -3662,6 +3706,42 @@ DivFOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
           {mt.lookupValue(thisOp.getLhs()), mt.lookupValue(thisOp.getRhs())}};
 }
 
+mlir::NVVM::IDArgPair
+AsyncStoreOp::getIntrinsicIDAndArgs(Operation &op, LLVM::ModuleTranslation &mt,
+                                    llvm::IRBuilderBase &builder) {
+  auto thisOp = cast<NVVM::AsyncStoreOp>(op);
+  NVVM::NVVMMemorySpace addrSpace = static_cast<NVVM::NVVMMemorySpace>(
+      llvm::cast<LLVM::LLVMPointerType>(thisOp.getAddr().getType())
+          .getAddressSpace());
+  mlir::NVVM::AsyncStoreScope scope = thisOp.getScope();
+
+  llvm::Value *addr = mt.lookupValue(thisOp.getAddr());
+  llvm::Value *value = mt.lookupValue(thisOp.getValue());
+  llvm::Value *mbarrier =
+      thisOp.getMbarrier() ? mt.lookupValue(thisOp.getMbarrier()) : nullptr;
+  llvm::Value *isMultimem =
+      thisOp.getIsMultimem() ? builder.getInt1(true) : builder.getInt1(false);
+
+  switch (addrSpace) {
+  case NVVMMemorySpace::Global:
+    if (scope == AsyncStoreScope::SYS) {
+      if (thisOp.getIsMmio())
+        return {llvm::Intrinsic::nvvm_st_async_mmio_sys, {addr, value}};
+      else
+        return {llvm::Intrinsic::nvvm_st_async_sys, {addr, value, isMultimem}};
+    } else if (scope == AsyncStoreScope::GPU) {
+      return {llvm::Intrinsic::nvvm_st_async_gpu, {addr, value, isMultimem}};
+    }
+    llvm_unreachable("unsupported async store scope for global address space");
+  case NVVMMemorySpace::SharedCluster:
+    return {llvm::Intrinsic::nvvm_st_async, {addr, value, mbarrier}};
+  default:
+    llvm_unreachable("unsupported address space");
+  }
+
+  return {llvm::Intrinsic::not_intrinsic, {}};
----------------
grypp wrote:

use `llvm_unreachables`

https://github.com/llvm/llvm-project/pull/210931

From llvmlistbot at llvm.org  Wed Jul 22 03:22:54 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 03:22:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add MathToSPIRV conversion for
 copysign on OpenCL targets (PR #205994)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205994@github.com>
Message-ID: <6a6099fe.a1accbb4.1ace1c.87d3@mx.google.com>


https://github.com/IgWod approved this pull request.


https://github.com/llvm/llvm-project/pull/205994

From llvmlistbot at llvm.org  Wed Jul 22 03:33:01 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 22 Jul 2026 03:33:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][ODS] Add strict property assembly
	format mode (PR #196269)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196269@github.com>
Message-ID: <6a609c5d.92d2873c.2cdd42.b096@mx.google.com>


================
@@ -55,6 +55,12 @@ class Dialect {
   // dialect declaration.
   code extraClassDeclaration = "";
 
+  // If this dialect should require declarative parsers for property-backed
+  // operations to bind every inherent attribute and property directly in the
+  // custom assembly format, or otherwise cover them with `prop-dict`. This
+  // stricter mode is disabled by default for now.
+  bit useStrictPropertiesInAssemblyFormat = 0;
----------------
joker-eph wrote:

Ping again @jpienaar ?

https://github.com/llvm/llvm-project/pull/196269

From llvmlistbot at llvm.org  Wed Jul 22 03:33:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 03:33:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 16042c9 - [mlir][SPIR-V] Add MathToSPIRV
 conversion for copysign on OpenCL targets (#205994)
Message-ID: <6a609c8a.481afadf.1fc9f0.3220@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-22T12:33:42+02:00
New Revision: 16042c9480b4c9147d35f4e56b609c4189eb4f6f

URL: https://github.com/llvm/llvm-project/commit/16042c9480b4c9147d35f4e56b609c4189eb4f6f
DIFF: https://github.com/llvm/llvm-project/commit/16042c9480b4c9147d35f4e56b609c4189eb4f6f.diff

LOG: [mlir][SPIR-V] Add MathToSPIRV conversion for copysign on OpenCL targets (#205994)

Added: 
    

Modified: 
    mlir/lib/Conversion/MathToSPIRV/MathToSPIRV.cpp
    mlir/test/Conversion/MathToSPIRV/math-to-opencl-spirv.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/MathToSPIRV/MathToSPIRV.cpp b/mlir/lib/Conversion/MathToSPIRV/MathToSPIRV.cpp
index 61dddaf5155d3..e451f4316d4e3 100644
--- a/mlir/lib/Conversion/MathToSPIRV/MathToSPIRV.cpp
+++ b/mlir/lib/Conversion/MathToSPIRV/MathToSPIRV.cpp
@@ -128,6 +128,12 @@ struct CopySignPattern final : public OpConversionPattern<math::CopySignOp> {
         failed(res))
       return res;
 
+    // Defer to the CL copysign op on Kernel targets.
+    auto &typeConverter = *getTypeConverter<SPIRVTypeConverter>();
+    if (typeConverter.getTargetEnv().allows(spirv::Capability::Kernel))
+      return rewriter.notifyMatchFailure(copySignOp,
+                                         "Kernel target has native CL op");
+
     Type type = getTypeConverter()->convertType(copySignOp.getType());
     if (!type)
       return failure();
@@ -684,6 +690,7 @@ void populateMathToSPIRVPatterns(const SPIRVTypeConverter &typeConverter,
       CheckedElementwiseOpPattern<math::Atan2Op, spirv::CLAtan2Op>,
       CheckedElementwiseOpPattern<math::CbrtOp, spirv::CLCbrtOp>,
       CheckedElementwiseOpPattern<math::CeilOp, spirv::CLCeilOp>,
+      CheckedElementwiseOpPattern<math::CopySignOp, spirv::CLCopysignOp>,
       CheckedElementwiseOpPattern<math::CosOp, spirv::CLCosOp>,
       CheckedElementwiseOpPattern<math::ErfOp, spirv::CLErfOp>,
       CheckedElementwiseOpPattern<math::ErfcOp, spirv::CLErfcOp>,

diff  --git a/mlir/test/Conversion/MathToSPIRV/math-to-opencl-spirv.mlir b/mlir/test/Conversion/MathToSPIRV/math-to-opencl-spirv.mlir
index 9830f3a5d055e..faa509851a2e2 100644
--- a/mlir/test/Conversion/MathToSPIRV/math-to-opencl-spirv.mlir
+++ b/mlir/test/Conversion/MathToSPIRV/math-to-opencl-spirv.mlir
@@ -132,6 +132,8 @@ func.func @float32_binary_scalar(%lhs: f32, %rhs: f32) {
   %0 = math.atan2 %lhs, %rhs : f32
   // CHECK: spirv.CL.pow %{{.*}}: f32
   %1 = math.powf %lhs, %rhs : f32
+  // CHECK: spirv.CL.copysign %{{.*}}, %{{.*}}: f32
+  %copysign = math.copysign %lhs, %rhs : f32
   return
 }
 
@@ -141,6 +143,8 @@ func.func @float32_binary_vector(%lhs: vector<4xf32>, %rhs: vector<4xf32>) {
   %0 = math.atan2 %lhs, %rhs : vector<4xf32>
   // CHECK: spirv.CL.pow %{{.*}}: vector<4xf32>
   %1 = math.powf %lhs, %rhs : vector<4xf32>
+  // CHECK: spirv.CL.copysign %{{.*}}, %{{.*}}: vector<4xf32>
+  %copysign = math.copysign %lhs, %rhs : vector<4xf32>
   return
 }
 


        

From llvmlistbot at llvm.org  Wed Jul 22 03:33:49 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 22 Jul 2026 03:33:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Add MathToSPIRV conversion for
 copysign on OpenCL targets (PR #205994)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205994@github.com>
Message-ID: <6a609c8d.eb165e5c.2e0c9a.7b37@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/205994

From llvmlistbot at llvm.org  Wed Jul 22 03:48:33 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 22 Jul 2026 03:48:33 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a60a001.b1416da1.20b4cf.1388@mx.google.com>


https://github.com/chandraghale updated https://github.com/llvm/llvm-project/pull/206747

>From 82416bc775c4befdbcce1f999a5f5b97ef954cd8 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Tue, 30 Jun 2026 10:00:56 -0500
Subject: [PATCH 1/3] Lower scan directive and inscan reduction modifier

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             |  57 ++-
 .../Lower/OpenMP/Todo/nested-wsloop-scan.f90  |  34 ++
 .../OpenMP/Todo/wsloop-scan-collapse.f90      |  29 ++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 476 +++++++++++++++---
 .../Target/LLVMIR/openmp-reduction-scan.mlir  | 123 +++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir      |  41 +-
 openmp/runtime/test/scan/scan.f90             |  38 ++
 7 files changed, 726 insertions(+), 72 deletions(-)
 create mode 100644 flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
 create mode 100644 flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
 create mode 100644 mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
 create mode 100644 openmp/runtime/test/scan/scan.f90

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index ea8c279962508..e77584a142eea 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2972,12 +2972,59 @@ genParallelOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
 
 static mlir::omp::ScanOp
 genScanOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-          semantics::SemanticsContext &semaCtx, mlir::Location loc,
-          const ConstructQueue &queue, ConstructQueue::const_iterator item) {
+          semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
+          mlir::Location loc, const ConstructQueue &queue,
+          ConstructQueue::const_iterator item) {
   mlir::omp::ScanOperands clauseOps;
   genScanClauses(converter, semaCtx, item->clauses, loc, clauseOps);
-  return mlir::omp::ScanOp::create(converter.getFirOpBuilder(),
-                                   converter.getCurrentLocation(), clauseOps);
+  mlir::omp::ScanOp scanOp = mlir::omp::ScanOp::create(
+      converter.getFirOpBuilder(), converter.getCurrentLocation(), clauseOps);
+
+  /// Scan reduction is not implemented with nested workshare loops, linear
+  /// clause, tiling
+  mlir::omp::LoopNestOp loopNestOp =
+      scanOp->getParentOfType<mlir::omp::LoopNestOp>();
+  llvm::SmallVector<mlir::omp::LoopWrapperInterface> loopWrappers;
+  loopNestOp.gatherWrappers(loopWrappers);
+  mlir::Operation *loopWrapperOp = loopWrappers.front().getOperation();
+  if (llvm::isa<mlir::omp::SimdOp>(loopWrapperOp))
+    TODO(loc, "unsupported simd");
+  if (loopWrappers.size() > 1)
+    TODO(loc, "unsupported composite");
+  mlir::omp::WsloopOp wsLoopOp = llvm::cast<mlir::omp::WsloopOp>(loopWrapperOp);
+  bool isNested =
+      (loopNestOp.getNumLoops() > 1) ||
+      (wsLoopOp && (wsLoopOp->getParentOfType<mlir::omp::WsloopOp>()));
+  if (isNested)
+    TODO(loc, "Scan directive inside nested workshare loops");
+  if (wsLoopOp && !wsLoopOp.getLinearVars().empty())
+    TODO(loc, "Scan directive with linear clause");
+  if (loopNestOp.getTileSizes())
+    TODO(loc, "Scan directive with loop tiling");
+
+  // All loop indices should be loaded after the scan construct as otherwise,
+  // it would result in using the index variable across scan directive.
+  // (`Intra-iteration dependences from a statement in the structured
+  // block sequence that precede a scan directive to a statement in the
+  // structured block sequence that follows a scan directive must not exist,
+  // except for dependences for the list items specified in an inclusive or
+  // exclusive clause.`).
+  // TODO: Nested loops are not handled.
+  mlir::Region &region = loopNestOp->getRegion(0);
+  mlir::Value indexVal = fir::getBase(region.getArgument(0));
+  lower::pft::Evaluation *doConstructEval = eval.parentConstruct;
+  fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
+  lower::pft::Evaluation *doLoop = &doConstructEval->getFirstNestedEvaluation();
+  auto *doStmt = doLoop->getIf<parser::NonLabelDoStmt>();
+  assert(doStmt && "Expected do loop to be in the nested evaluation");
+  const auto &loopControl =
+      std::get<std::optional<parser::LoopControl>>(doStmt->t);
+  const parser::LoopControl::Bounds *bounds =
+      std::get_if<parser::LoopControl::Bounds>(&loopControl->u);
+  mlir::Operation *storeOp =
+      setLoopVar(converter, loc, indexVal, bounds->Name().thing.symbol);
+  firOpBuilder.setInsertionPointAfter(storeOp);
+  return scanOp;
 }
 
 static mlir::omp::SectionsOp
@@ -4305,7 +4352,7 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
                                   loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_scan:
-    newOp = genScanOp(converter, symTable, semaCtx, loc, queue, item);
+    newOp = genScanOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
   case llvm::omp::Directive::OMPD_section:
     llvm_unreachable("genOMPDispatch: OMPD_section");
diff --git a/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90 b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
new file mode 100644
index 0000000000000..414e2ef2d5aed
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/nested-wsloop-scan.f90
@@ -0,0 +1,34 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_scan_example
+  implicit none
+  integer, parameter :: n = 4, m = 5
+  integer :: a(n, m), b(n, m)
+  integer :: i, j
+  integer :: row_sum, col_sum
+
+  do i = 1, n
+     do j = 1, m
+        a(i, j) = i + j
+     end do
+  end do
+
+  !$omp parallel do reduction(inscan, +: row_sum) private(col_sum, j)
+  do i = 1, n
+     row_sum = row_sum + i
+     !$omp scan inclusive(row_sum)
+
+     col_sum = 0
+     !$omp parallel do reduction(inscan, +: col_sum)
+     do j = 1, m
+        col_sum = col_sum + a(i, j)
+        !CHECK: not yet implemented: Scan directive inside nested workshare loops
+        !$omp scan inclusive(col_sum)
+        b(i, j) = col_sum + row_sum 
+     end do
+     !$omp end parallel do
+  end do
+  !$omp end parallel do
+end program nested_scan_example
diff --git a/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90 b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
new file mode 100644
index 0000000000000..b8e6e831884ab
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/wsloop-scan-collapse.f90
@@ -0,0 +1,29 @@
+! Tests scan reduction behavior when used in nested workshare loops
+
+! RUN: %not_todo_cmd %flang_fc1 -emit-hlfir -fopenmp -o - %s 2>&1 | FileCheck %s
+
+program nested_loop_example
+  implicit none
+  integer :: i, j, x
+  integer, parameter :: N = 100, M = 200
+  real :: A(N, M), B(N, M)
+  x = 0
+
+  do i = 1, N
+    do j = 1, M
+      A(i, j) = i * j
+    end do
+  end do
+  
+  !$omp parallel do collapse(2) reduction(inscan, +:x)
+  do i = 1, N
+    do j = 1, M
+      x = x + A(i,j)
+      !CHECK: not yet implemented: Scan directive inside nested workshare loops
+      !$omp scan inclusive(x)
+      B(i, j) = x
+    end do
+  end do
+  !$omp end parallel do
+
+end program nested_loop_example
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d860c408e0fd2..297ebac51de88 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -41,6 +41,8 @@
 #include "llvm/TargetParser/Triple.h"
 #include "llvm/Transforms/Utils/ModuleUtils.h"
 
+#include <cassert>
+#include <cstddef>
 #include <cstdint>
 #include <iterator>
 #include <numeric>
@@ -84,6 +86,10 @@ class OpenMPAllocStackFrame
       : allocInsertPoint(allocaIP), deallocBlocks(deallocBlocks) {}
   llvm::OpenMPIRBuilder::InsertPointTy allocInsertPoint;
   llvm::SmallVector<llvm::BasicBlock *> deallocBlocks;
+  /// Set to true when this alloca frame encloses an omp.parallel operation.
+  /// The alloca insertion point of a function in which a parallel op is
+  /// defined may be used to allocate the temporary buffer for scan reductions.
+  bool containsParallelOp = false;
 };
 
 /// Stack frame to hold a \see llvm::CanonicalLoopInfo representing the
@@ -93,7 +99,17 @@ class OpenMPLoopInfoStackFrame
     : public StateStackFrameBase<OpenMPLoopInfoStackFrame> {
 public:
   MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(OpenMPLoopInfoStackFrame)
+  /// For constructs like `scan`, a single `omp.loop_nest` is split into an
+  /// input loop and a scan loop. In that case `loopInfo` holds the input loop
+  /// info and `scanloopInfo` holds the scan loop info.
   llvm::CanonicalLoopInfo *loopInfo = nullptr;
+  llvm::CanonicalLoopInfo *scanloopInfo = nullptr;
+  llvm::ScanInfo *scanInfo = nullptr;
+  /// Map reduction variables to their LLVM types. Populated when the reduction
+  /// clause is processed and used when a `scan` directive is encountered in the
+  /// loop body.
+  std::unique_ptr<llvm::DenseMap<llvm::Value *, llvm::Type *>>
+      reductionVarToType = nullptr;
 };
 
 /// Custom error class to signal translation errors that don't need reporting,
@@ -359,6 +375,10 @@ static LogicalResult checkImplementationStatus(Operation &op) {
     if (!op.getDependVars().empty() || op.getDependKinds())
       result = todo("depend");
   };
+  auto checkExclusive = [&todo](auto op, LogicalResult &result) {
+    if (!op.getExclusiveVars().empty())
+      result = todo("exclusive");
+  };
   auto checkHint = [](auto op, LogicalResult &) {
     if (op.getHint())
       op.emitWarning("hint clause discarded");
@@ -389,21 +409,27 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         op.getReductionMod().value() != omp::ReductionModifier::defaultmod) {
       omp::ReductionModifier mod = op.getReductionMod().value();
       // The `task` reduction modifier is supported on the parallel and
-      // worksharing (do/for and sections) constructs. Other modifiers, and the
-      // `task` modifier on other constructs, are not yet implemented.
+      // worksharing (do/for and sections) constructs. The `inscan` modifier is
+      // supported on the worksharing-loop construct (it is translated as a scan
+      // reduction). Other modifiers, and these modifiers on other constructs,
+      // are not yet implemented.
       bool taskModifierSupported =
           mod == omp::ReductionModifier::task &&
           isa<omp::ParallelOp, omp::WsloopOp, omp::SectionsOp>(op);
-      if (!taskModifierSupported) {
+      bool inscanModifierSupported =
+          mod == omp::ReductionModifier::inscan && isa<omp::WsloopOp>(op);
+      if (!taskModifierSupported && !inscanModifierSupported) {
         result = todo("reduction with modifier");
-      } else if (auto byref = op.getReductionByref()) {
-        // The task reduction modifier lowering only handles non-byref
-        // reductions for now.
-        for (bool isByRef : *byref)
-          if (isByRef) {
-            result = todo("task reduction modifier with by-ref reduction");
-            break;
-          }
+      } else if (taskModifierSupported) {
+        if (auto byref = op.getReductionByref()) {
+          // The task reduction modifier lowering only handles non-byref
+          // reductions for now.
+          for (bool isByRef : *byref)
+            if (isByRef) {
+              result = todo("task reduction modifier with by-ref reduction");
+              break;
+            }
+        }
       }
     }
   };
@@ -460,6 +486,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkAllocate(op, result);
         checkOrder(op, result);
       })
+      .Case([&](omp::ScanOp op) { checkExclusive(op, result); })
       .Case([&](omp::SectionsOp op) {
         checkAllocate(op, result);
         checkPrivate(op, result);
@@ -650,6 +677,97 @@ findCurrentLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
   return loopInfo;
 }
 
+/// Find the scan loop information structure for the scan loop nest being
+/// translated. It will return a `null` value unless called from the
+/// translation function for a loop wrapper operation after successfully
+/// translating its body.
+static llvm::CanonicalLoopInfo *
+findCurrentScanLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::CanonicalLoopInfo *scanLoopInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanLoopInfo = frame.scanloopInfo;
+        return WalkResult::interrupt();
+      });
+  return scanLoopInfo;
+}
+
+/// Find the `ScanInfo` stored on the loop stack frame. Upon encountering an
+/// `inscan` reduction modifier, `scanInfoInitialize` initializes the
+/// `ScanInfo`, which is then used when a `scan` directive is encountered in the
+/// body of the loop nest.
+static llvm::ScanInfo *
+findScanInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::ScanInfo *scanInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanInfo = frame.scanInfo;
+        return WalkResult::interrupt();
+      });
+  return scanInfo;
+}
+
+/// The types of reduction variables are used for lowering a `scan` directive
+/// that appears in the body of the loop. The types are stored in the loop frame
+/// when the reduction clause is encountered and used when the `scan` directive
+/// is encountered.
+static llvm::DenseMap<llvm::Value *, llvm::Type *> *
+findReductionVarTypes(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        if (!frame.reductionVarToType)
+          frame.reductionVarToType =
+              std::make_unique<llvm::DenseMap<llvm::Value *, llvm::Type *>>();
+        reductionVarToType = frame.reductionVarToType.get();
+        return WalkResult::interrupt();
+      });
+  return reductionVarToType;
+}
+
+/// Scan reduction requires a shared buffer to be allocated to perform the
+/// reduction. The allocation needs to be done outside the parallel region in
+/// which the scan operation is used.
+static llvm::OpenMPIRBuilder::InsertPointTy
+findParallelAllocaIP(llvm::IRBuilderBase &builder,
+                     LLVM::ModuleTranslation &moduleTranslation) {
+  // If there is an alloca insertion point on the stack belonging to a frame
+  // that encloses a parallel op, use it.
+  llvm::OpenMPIRBuilder::InsertPointTy allocaInsertPoint;
+  WalkResult walkResult = moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (frame.containsParallelOp) {
+          allocaInsertPoint = frame.allocInsertPoint;
+          return WalkResult::interrupt();
+        }
+        return WalkResult::skip();
+      });
+  if (walkResult.wasInterrupted())
+    return allocaInsertPoint;
+  // Otherwise, insert into the entry block of the surrounding function.
+  // If the current IRBuilder InsertPoint is the function's entry, it cannot
+  // also be used for alloca insertion which would result in insertion order
+  // confusion. Create a new BasicBlock for the Builder and use the entry block
+  // for the allocs.
+  // TODO: Create a dedicated alloca BasicBlock at function creation such that
+  // we do not need to move the current InsertPoint here.
+  if (builder.GetInsertBlock() ==
+      &builder.GetInsertBlock()->getParent()->getEntryBlock()) {
+    assert(builder.GetInsertPoint() == builder.GetInsertBlock()->end() &&
+           "Assuming end of basic block");
+    llvm::BasicBlock *entryBB = llvm::BasicBlock::Create(
+        builder.getContext(), "entry", builder.GetInsertBlock()->getParent(),
+        builder.GetInsertBlock()->getNextNode());
+    builder.CreateBr(entryBB);
+    builder.SetInsertPoint(entryBB);
+  }
+
+  llvm::BasicBlock &funcEntryBlock =
+      builder.GetInsertBlock()->getParent()->getEntryBlock();
+  return llvm::OpenMPIRBuilder::InsertPointTy(
+      &funcEntryBlock, funcEntryBlock.getFirstInsertionPt());
+}
+
 /// Converts the given region that appears within an OpenMP dialect operation to
 /// LLVM IR, creating a branch from the `sourceBlock` to the entry block of the
 /// region, and a branch from any block with an successor-less OpenMP terminator
@@ -1414,7 +1532,8 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
                   SmallVectorImpl<llvm::Value *> &privateReductionVariables,
                   DenseMap<Value, llvm::Value *> &reductionVariableMap,
                   llvm::ArrayRef<bool> isByRef,
-                  SmallVectorImpl<DeferredStore> &deferredStores) {
+                  SmallVectorImpl<DeferredStore> &deferredStores,
+                  bool isInScanRegion = false) {
   if (op.getNumReductionVars() == 0)
     return success();
 
@@ -1451,11 +1570,17 @@ initReductionVars(OP op, ArrayRef<BlockArgument> reductionArgs,
   for (auto [data, addr] : deferredStores)
     builder.CreateStore(data, addr);
 
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
   // Before the loop, store the initial values of reductions into reduction
   // variables. Although this could be done after allocas, we don't want to mess
   // up with the alloca insertion point.
   for (unsigned i = 0; i < op.getNumReductionVars(); ++i) {
     SmallVector<llvm::Value *, 1> phis;
+    llvm::Type *reductionType =
+        moduleTranslation.convertType(reductionDecls[i].getType());
+    if (isInScanRegion && reductionVarToType != nullptr)
+      (*reductionVarToType)[privateReductionVariables[i]] = reductionType;
 
     // map block argument to initializer region
     mapInitializationArgs(op, moduleTranslation, builder, reductionDecls,
@@ -4406,11 +4531,15 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     return failure();
 
   assert(afterAllocas.get()->getSinglePredecessor());
+  bool isInScanRegion =
+      wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                     mlir::omp::ReductionModifier::inscan);
   if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
                                moduleTranslation,
                                afterAllocas.get()->getSinglePredecessor(),
                                reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores)))
+                               reductionVariableMap, isByRef, deferredStores,
+                               isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -4432,6 +4561,10 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   std::optional<omp::ScheduleModifier> scheduleMod = wsloopOp.getScheduleMod();
   bool isSimd = wsloopOp.getScheduleSimd();
   bool loopNeedsBarrier = !wsloopOp.getNowait();
+  // TODO: Linear clause support needs to be enabled for scan reduction.
+  if (isInScanRegion)
+    assert(wsloopOp.getLinearVars().empty() &&
+           "Linear clause support is not enabled with scan reduction");
 
   // The only legal way for the direct parent to be omp.distribute is that this
   // represents 'distribute parallel do'. Otherwise, this is a regular
@@ -4469,21 +4602,93 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   if (failed(handleError(regionBlock, opInst)))
     return failure();
 
+  // Generates the loop body for a worksharing loop, including linear-variable
+  // handling and the call into the OpenMPIRBuilder's worksharing-loop helper.
+  // For scan reductions this lambda is invoked twice: once for the input loop
+  // and once for the scan loop.
+  const auto &&wsloopCodeGen = [&](llvm::CanonicalLoopInfo *loopInfo,
+                                   bool noLoopMode,
+                                   bool inputScanLoop) -> LogicalResult {
+    // Emit Initialization and Update IR for linear variables
+    if (!wsloopOp.getLinearVars().empty()) {
+      linearClauseProcessor.initLinearVar(builder, moduleTranslation,
+                                          loopInfo->getPreheader());
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          moduleTranslation.getOpenMPBuilder()->createBarrier(
+              builder.saveIP(), llvm::omp::OMPD_barrier);
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+      builder.restoreIP(*afterBarrierIP);
+      linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                            loopInfo->getIndVar());
+      linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+    }
+
+    builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
+
+    for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
+      linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
+                                           loopInfo->getLatch(), index);
+
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
+        ompBuilder->applyWorkshareLoop(
+            ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
+            convertToScheduleKind(schedule), chunk, isSimd,
+            scheduleMod == omp::ScheduleModifier::monotonic,
+            scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
+            workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
+
+    if (failed(handleError(wsloopIP, opInst)))
+      return failure();
+
+    // Emit finalization for linear vars.
+    if (!wsloopOp.getLinearVars().empty()) {
+      llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
+      assert(loopInfo->getLastIter() &&
+             "`lastiter` in CanonicalLoopInfo is nullptr");
+      llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
+          linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
+                                                  loopInfo->getLastIter());
+      if (failed(handleError(afterBarrierIP, *loopOp)))
+        return failure();
+
+      builder.restoreIP(oldIP);
+    }
+
+    // For scan reductions, the cancellation finalization callback is popped
+    // only after the scan (second) loop has been generated.
+    if (!inputScanLoop || !isInScanRegion)
+      popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
+
+    return success();
+  };
+
   llvm::CanonicalLoopInfo *loopInfo = findCurrentLoopInfo(moduleTranslation);
 
-  // Emit Initialization and Update IR for linear variables
-  if (!wsloopOp.getLinearVars().empty()) {
-    linearClauseProcessor.initLinearVar(builder, moduleTranslation,
-                                        loopInfo->getPreheader());
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    // Emit the scan reduction combiner between the input loop and the scan
+    // loop, operating on the per-iteration partial values stored in the shared
+    // buffer.
+    auto inputLoopFinishIp = loopInfo->getAfterIP();
+    builder.restoreIP(inputLoopFinishIp);
+    SmallVector<OwningReductionGen> owningReductionGens;
+    SmallVector<OwningAtomicReductionGen> owningAtomicReductionGens;
+    SmallVector<llvm::OpenMPIRBuilder::ReductionInfo, 2> reductionInfos;
+    SmallVector<OwningDataPtrPtrReductionGen> owningReductionGenRefDataPtrGens;
+    collectReductionInfo(wsloopOp, builder, moduleTranslation, reductionDecls,
+                         owningReductionGens, owningAtomicReductionGens,
+                         owningReductionGenRefDataPtrGens,
+                         privateReductionVariables, reductionInfos, isByRef);
+    llvm::BasicBlock *cont = splitBB(builder, false, "omp.scan.loop.cont");
+    llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+    llvm::OpenMPIRBuilder::InsertPointOrErrorTy redIP =
+        ompBuilder->emitScanReduction(builder.saveIP(), reductionInfos,
+                                      scanInfo);
+    if (failed(handleError(redIP, opInst)))
       return failure();
-    builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
-    linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
+
+    builder.restoreIP(*redIP);
+    builder.CreateBr(cont);
   }
 
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
@@ -4502,50 +4707,41 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
       noLoopMode = true;
   }
 
-  for (size_t index = 0; index < wsloopOp.getLinearVars().size(); index++)
-    linearClauseProcessor.rewriteInPlace(builder, loopInfo->getBody(),
-                                         loopInfo->getLatch(), index);
-
-  llvm::OpenMPIRBuilder::InsertPointOrErrorTy wsloopIP =
-      ompBuilder->applyWorkshareLoop(
-          ompLoc.DL, loopInfo, allocaIP, loopNeedsBarrier,
-          convertToScheduleKind(schedule), chunk, isSimd,
-          scheduleMod == omp::ScheduleModifier::monotonic,
-          scheduleMod == omp::ScheduleModifier::nonmonotonic, isOrdered,
-          workshareLoopType, noLoopMode, hasDistSchedule, distScheduleChunk);
-
-  if (failed(handleError(wsloopIP, opInst)))
+  // For scan loops, the input loop does not pop the cancellation finalization
+  // callback; that happens after the scan loop is generated below.
+  bool inputScanLoop = isInScanRegion;
+  if (failed(wsloopCodeGen(loopInfo, noLoopMode, inputScanLoop)))
     return failure();
+  inputScanLoop = false;
 
-  // Emit finalization and in-place rewrites for linear vars.
-  if (!wsloopOp.getLinearVars().empty()) {
-    llvm::OpenMPIRBuilder::InsertPointTy oldIP = builder.saveIP();
-    assert(loopInfo->getLastIter() &&
-           "`lastiter` in CanonicalLoopInfo is nullptr");
-    llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
-        linearClauseProcessor.finalizeLinearVar(builder, moduleTranslation,
-                                                loopInfo->getLastIter());
-    if (failed(handleError(afterBarrierIP, *loopOp)))
+  if (isInScanRegion) {
+    llvm::CanonicalLoopInfo *scanLoopInfo =
+        findCurrentScanLoopInfo(moduleTranslation);
+    if (failed(wsloopCodeGen(scanLoopInfo, noLoopMode, inputScanLoop)))
+      return failure();
+    SmallVector<Region *> reductionRegions;
+    llvm::transform(reductionDecls, std::back_inserter(reductionRegions),
+                    [](omp::DeclareReductionOp reductionDecl) {
+                      return &reductionDecl.getCleanupRegion();
+                    });
+    if (failed(inlineOmpRegionCleanup(
+            reductionRegions, privateReductionVariables, moduleTranslation,
+            builder, "omp.reduction.cleanup")))
       return failure();
+  } else {
+    // Close the task-reduction scope before the worksharing reduction combine.
+    if (isTaskReductionMod)
+      emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
+                                    moduleTranslation);
 
-    builder.restoreIP(oldIP);
+    // Process the reductions if required.
+    if (failed(createReductionsAndCleanup(
+            wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
+            privateReductionVariables, isByRef, wsloopOp.getNowait(),
+            /*isTeamsReduction=*/false)))
+      return failure();
   }
 
-  // Set the correct branch target for task cancellation
-  popCancelFinalizationCB(cancelTerminators, *ompBuilder, wsloopIP.get());
-
-  // Close the task-reduction scope before the worksharing reduction combine.
-  if (isTaskReductionMod)
-    emitTaskReductionModifierFini(/*isWorksharing=*/true, builder,
-                                  moduleTranslation);
-
-  // Process the reductions if required.
-  if (failed(createReductionsAndCleanup(
-          wsloopOp, builder, moduleTranslation, allocaIP, reductionDecls,
-          privateReductionVariables, isByRef, wsloopOp.getNowait(),
-          /*isTeamsReduction=*/false)))
-    return failure();
-
   return cleanupPrivateVars(wsloopOp, builder, moduleTranslation,
                             wsloopOp.getLoc(), privateVarsInfo);
 }
@@ -4579,6 +4775,20 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       opInst.getReductionMod() == omp::ReductionModifier::task &&
       opInst.getNumReductionVars() > 0;
 
+  // Mark the enclosing alloca stack frame as containing a parallel op. Scan
+  // reductions use the alloca insertion point of the function enclosing the
+  // parallel region to allocate their shared temporary buffer.
+  bool foundParallelOp = false;
+  moduleTranslation.stackWalk<OpenMPAllocStackFrame>(
+      [&](OpenMPAllocStackFrame &frame) {
+        if (foundParallelOp) {
+          frame.containsParallelOp = true;
+          return WalkResult::interrupt();
+        }
+        foundParallelOp = true;
+        return WalkResult::skip();
+      });
+
   auto bodyGenCB =
       [&](InsertPointTy allocaIP, InsertPointTy codeGenIP,
           llvm::ArrayRef<llvm::BasicBlock *> deallocBlocks) -> llvm::Error {
@@ -4972,6 +5182,99 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             privateVarsInfo);
 }
 
+static LogicalResult
+convertOmpScan(Operation &opInst, llvm::IRBuilderBase &builder,
+               LLVM::ModuleTranslation &moduleTranslation) {
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+  auto scanOp = cast<omp::ScanOp>(opInst);
+  bool isInclusive = scanOp.hasInclusiveVars();
+  SmallVector<llvm::Value *> llvmScanVars;
+  SmallVector<llvm::Type *> llvmScanVarsType;
+  mlir::OperandRange mlirScanVars = scanOp.getInclusiveVars();
+  if (!isInclusive)
+    mlirScanVars = scanOp.getExclusiveVars();
+
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType =
+      findReductionVarTypes(moduleTranslation);
+  for (auto val : mlirScanVars) {
+    llvm::Value *llvmVal = moduleTranslation.lookupValue(val);
+    llvmScanVars.push_back(llvmVal);
+    llvmScanVarsType.push_back((*reductionVarToType)[llvmVal]);
+  }
+  llvm::OpenMPIRBuilder::InsertPointTy allocaIP =
+      findParallelAllocaIP(builder, moduleTranslation);
+  llvm::OpenMPIRBuilder::LocationDescription ompLoc(builder);
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
+      moduleTranslation.getOpenMPBuilder()->createScan(
+          ompLoc, allocaIP, llvmScanVars, llvmScanVarsType, isInclusive,
+          scanInfo);
+  if (failed(handleError(afterIP, opInst)))
+    return failure();
+  builder.restoreIP(*afterIP);
+  return success();
+}
+
+/// Re-initialize the scan reduction variables to their identity at the start of
+/// each input-loop iteration of a scan reduction. This is required because the
+/// OpenMPIRBuilder stores the per-iteration reduction value into the scan
+/// buffer (`buffer[i] = red`) and `emitScanReduction` then computes the prefix
+/// sum across the buffer. Without resetting `red` to the reduction identity
+/// each iteration, the buffer would hold a running sum and the prefix sum would
+/// be computed twice. This reset only applies to the input loop; in the scan
+/// loop the reduction variable is loaded back from the buffer.
+static LogicalResult
+initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation) {
+  auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>();
+  if (!wsloopOp || !wsloopOp.getReductionMod() ||
+      wsloopOp.getReductionMod().value() != omp::ReductionModifier::inscan)
+    return success();
+
+  // Only reset in the input loop; in the scan loop the reduction variable holds
+  // the value loaded from the buffer and must not be overwritten.
+  llvm::ScanInfo *scanInfo = findScanInfo(moduleTranslation);
+  if (!scanInfo || !scanInfo->OMPFirstScanLoop)
+    return success();
+
+  unsigned numReductions = wsloopOp.getNumReductionVars();
+  if (numReductions == 0)
+    return success();
+
+  SmallVector<omp::DeclareReductionOp> reductionDecls;
+  collectReductionDecls(wsloopOp, reductionDecls);
+  ArrayRef<bool> isByRef = getIsByRef(wsloopOp.getReductionByref());
+  MutableArrayRef<BlockArgument> reductionArgs =
+      cast<omp::BlockArgOpenMPOpInterface>(wsloopOp.getOperation())
+          .getReductionBlockArgs();
+
+  DenseMap<Value, llvm::Value *> reductionVariableMap;
+  for (unsigned i = 0; i < numReductions; ++i)
+    reductionVariableMap.try_emplace(
+        wsloopOp.getReductionVars()[i],
+        moduleTranslation.lookupValue(reductionArgs[i]));
+
+  for (unsigned i = 0; i < numReductions; ++i) {
+    // Scan reductions only support by-value (scalar) reductions.
+    if (isByRef[i])
+      continue;
+    SmallVector<llvm::Value *, 1> phis;
+    mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
+                          reductionVariableMap, i);
+    if (failed(inlineConvertOmpRegions(
+            reductionDecls[i].getInitializerRegion(),
+            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+      return failure();
+    assert(phis.size() == 1 &&
+           "expected one value to be yielded from the reduction init region");
+    setInsertPointForPossiblyEmptyBlock(builder);
+    builder.CreateStore(phis[0],
+                        moduleTranslation.lookupValue(reductionArgs[i]));
+  }
+  return success();
+}
+
 /// Converts an OpenMP loop nest into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -5004,6 +5307,10 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
 
     // Convert the body of the loop.
     builder.restoreIP(ip);
+    // For scan reductions, reset the reduction variables to their identity at
+    // the start of each input-loop iteration, before the input phase runs.
+    if (failed(initScanReductionVars(loopOp, builder, moduleTranslation)))
+      return llvm::make_error<PreviouslyReportedError>();
     llvm::Expected<llvm::BasicBlock *> regionBlock = convertOmpOpRegions(
         loopOp.getRegion(), "omp.loop_nest.region", builder, moduleTranslation);
     if (!regionBlock)
@@ -5036,6 +5343,46 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
       computeIP = loopInfos.front()->getPreheaderIP();
     }
 
+    // If this loop is the worksharing loop of a scan reduction, generate a
+    // pair of canonical loops (an input loop and a scan loop) instead of a
+    // single loop. The `scan` directive in the body is translated against the
+    // `ScanInfo` recorded here.
+    bool isInScanRegion = false;
+    if (auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>())
+      isInScanRegion =
+          wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                         mlir::omp::ReductionModifier::inscan);
+    if (isInScanRegion) {
+      llvm::Expected<llvm::ScanInfo *> res = ompBuilder->scanInfoInitialize();
+      if (failed(handleError(res, *loopOp)))
+        return failure();
+      llvm::ScanInfo *scanInfo = res.get();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.scanInfo = scanInfo;
+            return WalkResult::interrupt();
+          });
+      llvm::Expected<llvm::SmallVector<llvm::CanonicalLoopInfo *>> loopResults =
+          ompBuilder->createCanonicalScanLoops(
+              loc, bodyGen, lowerBound, upperBound, step,
+              /*IsSigned=*/true, loopOp.getLoopInclusive(), computeIP, "loop",
+              scanInfo);
+
+      if (failed(handleError(loopResults, *loopOp)))
+        return failure();
+      llvm::CanonicalLoopInfo *inputLoop = loopResults.get().front();
+      llvm::CanonicalLoopInfo *scanLoop = loopResults.get().back();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.loopInfo = inputLoop;
+            frame.scanloopInfo = scanLoop;
+            return WalkResult::interrupt();
+          });
+      builder.restoreIP(scanLoop->getAfterIP());
+      // TODO: tiling and collapse are not yet implemented for scan reduction.
+      return success();
+    }
+
     llvm::Expected<llvm::CanonicalLoopInfo *> loopResult =
         ompBuilder->createCanonicalLoop(
             loc, bodyGen, lowerBound, upperBound, step,
@@ -9454,6 +9801,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::WsloopOp) {
             return convertOmpWsloop(*op, builder, moduleTranslation);
           })
+          .Case([&](omp::ScanOp) {
+            return convertOmpScan(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::SimdOp) {
             return convertOmpSimd(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
new file mode 100644
index 0000000000000..c6b8daccad313
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
@@ -0,0 +1,123 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+omp.declare_reduction @add_reduction_i32 : i32 init {
+^bb0(%arg0: i32):
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  omp.yield(%0 : i32)
+} combiner {
+^bb0(%arg0: i32, %arg1: i32):
+  %0 = llvm.add %arg0, %arg1 : i32
+  omp.yield(%0 : i32)
+}
+// CHECK-LABEL: @scan_reduction
+llvm.func @scan_reduction() {
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  %1 = llvm.alloca %0 x i32 {bindc_name = "z"} : (i64) -> !llvm.ptr
+  %3 = llvm.alloca %0 x i32 {bindc_name = "y"} : (i64) -> !llvm.ptr
+  %5 = llvm.alloca %0 x i32 {bindc_name = "x"} : (i64) -> !llvm.ptr
+  %7 = llvm.alloca %0 x i32 {bindc_name = "k"} : (i64) -> !llvm.ptr
+  %10 = llvm.mlir.constant(100 : i32) : i32
+  %11 = llvm.mlir.constant(1 : i32) : i32
+  %12 = llvm.mlir.constant(0 : i32) : i32
+  %13 = llvm.mlir.constant(100 : index) : i64
+  %14 = llvm.mlir.addressof @_QFEa : !llvm.ptr
+  %15 = llvm.mlir.addressof @_QFEb : !llvm.ptr
+  omp.parallel {
+    %37 = llvm.mlir.constant(1 : i64) : i64
+    %38 = llvm.alloca %37 x i32 {bindc_name = "k", pinned} : (i64) -> !llvm.ptr
+    %39 = llvm.mlir.constant(1 : i64) : i64
+    omp.wsloop reduction(mod: inscan, @add_reduction_i32 %5 -> %arg0 : !llvm.ptr) {
+      omp.loop_nest (%arg1) : i32 = (%11) to (%10) inclusive step (%11) {
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %40 = llvm.load %arg0 : !llvm.ptr -> i32
+        %41 = llvm.load %38 : !llvm.ptr -> i32
+        %42 = llvm.sext %41 : i32 to i64
+        %50 = llvm.getelementptr %14[%42] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        %51 = llvm.load %50 : !llvm.ptr -> i32
+        %52 = llvm.add %40, %51 : i32
+        llvm.store %52, %arg0 : i32, !llvm.ptr
+        omp.scan inclusive(%arg0 : !llvm.ptr)
+        llvm.store %arg1, %38 : i32, !llvm.ptr
+        %53 = llvm.load %arg0 : !llvm.ptr -> i32
+        %54 = llvm.load %38 : !llvm.ptr -> i32
+        %55 = llvm.sext %54 : i32 to i64
+        %63 = llvm.getelementptr %15[%55] : (!llvm.ptr, i64) -> !llvm.ptr, i32
+        llvm.store %53, %63 : i32, !llvm.ptr
+        omp.yield
+      }
+    }
+    omp.terminator
+  }
+  llvm.return
+}
+llvm.mlir.global internal @_QFEa() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i32> {
+  %0 = llvm.mlir.zero : !llvm.array<100 x i32>
+  llvm.return %0 : !llvm.array<100 x i32>
+}
+//CHECK: %vla = alloca ptr, align 8
+//CHECK: omp_parallel
+//CHECK: store ptr %vla, ptr %gep_vla, align 8
+//CHECK: @__kmpc_fork_call
+//CHECK: void @scan_reduction..omp_par
+//CHECK: %[[BUFF_PTR:.+]] = load ptr, ptr %gep_vla
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: omp.scan.loop.cont:
+//CHECK: @__kmpc_masked
+//CHECK: @__kmpc_barrier
+//CHECK: %[[FREE_VAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:  %[[ARRLAST:.+]] = getelementptr inbounds i32, ptr %[[FREE_VAR]], i32 100
+//CHECK:  %[[RES:.+]] = load i32, ptr %[[ARRLAST]], align 4
+//CHECK:  store i32 %[[RES]], ptr %loadgep{{.*}}, align 4
+//CHECK: tail call void @free(ptr %[[FREE_VAR]])
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inscan.dispatch{{.*}}:                            ; preds = %omp_loop.body{{.*}}
+//CHECK:   %[[BUFFVAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
+//CHECK:   %[[arrayOffset1:.+]] = getelementptr inbounds i32, ptr %[[BUFFVAR]], i32 %{{.*}}
+//CHECK:   %[[BUFFVAL1:.+]] = load i32, ptr %[[arrayOffset1]], align 4
+//CHECK:   store i32 %[[BUFFVAL1]], ptr %{{.*}}, align 4
+//CHECK:   %[[LOG:.+]] = call double @llvm.log2.f64(double 1.000000e+02) #0
+//CHECK:   %[[CEIL:.+]] = call double @llvm.ceil.f64(double %[[LOG]]) #0
+//CHECK:   %[[UB:.+]] = fptoui double %[[CEIL]] to i32
+//CHECK:   br label %omp.outer.log.scan.body
+//CHECK: omp.outer.log.scan.body:
+//CHECK:   %[[K:.+]] = phi i32 [ 0, %{{.*}} ], [ %[[NEXTK:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[I:.+]] = phi i32 [ 1, %{{.*}} ], [ %[[NEXTI:.+]], %omp.inner.log.scan.exit ]
+//CHECK:   %[[CMP1:.+]] = icmp uge i32 99, %[[I]]
+//CHECK:   br i1 %[[CMP1]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inner.log.scan.exit:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[NEXTK]] = add nuw i32 %[[K]], 1
+//CHECK:   %[[NEXTI]] = shl nuw i32 %[[I]], 1
+//CHECK:   %[[CMP2:.+]] = icmp ne i32 %[[NEXTK]], %[[UB]]
+//CHECK:   br i1 %[[CMP2]], label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
+//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit
+//CHECK: @__kmpc_end_masked
+//CHECK: omp.inner.log.scan.body:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
+//CHECK:   %[[CNT:.+]] = phi i32 [ 99, %omp.outer.log.scan.body ], [ %[[CNTNXT:.+]], %omp.inner.log.scan.body ]
+//CHECK:   %[[BUFF:.+]] = load ptr, ptr %[[BUFF_PTR]]
+//CHECK:   %[[IND1:.+]] = add i32 %[[CNT]], 1
+//CHECK:   %[[IND1PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND1]]
+//CHECK:   %[[IND2:.+]] = sub nuw i32 %[[IND1]], %[[I]]
+//CHECK:   %[[IND2PTR:.+]] = getelementptr inbounds i32, ptr %[[BUFF]], i32 %[[IND2]]
+//CHECK:   %[[IND1VAL:.+]] = load i32, ptr %[[IND1PTR]], align 4
+//CHECK:   %[[IND2VAL:.+]] = load i32, ptr %[[IND2PTR]], align 4
+//CHECK:   %[[REDVAL:.+]] = add i32 %[[IND1VAL]], %[[IND2VAL]]
+//CHECK:   store i32 %[[REDVAL]], ptr %[[IND1PTR]], align 4
+//CHECK:   %[[CNTNXT]] = sub nuw i32 %[[CNT]], 1
+//CHECK:   %[[CMP3:.+]] = icmp uge i32 %[[CNTNXT]], %[[I]]
+//CHECK:   br i1 %[[CMP3]], label %omp.inner.log.scan.body, label %omp.inner.log.scan.exit
+//CHECK: omp.inscan.dispatch:                              ; preds = %omp_loop.body
+//CHECK:   br i1 true, label %omp.before.scan.bb, label %omp.after.scan.bb
+//CHECK: omp.before.scan.bb:
+//CHECK:   store i32 0, ptr %[[REDPRIV:.+]], align 4
+//CHECK: omp.loop_nest.region:                             ; preds = %omp.before.scan.bb
+//CHECK:   %[[BUFFER:.+]] = load ptr, ptr %loadgep_vla, align 8
+//CHECK:   %[[ARRAYOFFSET2:.+]] = getelementptr inbounds i32, ptr %[[BUFFER]], i32 %{{.*}}
+//CHECK-NEXT:   %[[REDPRIVVAL:.+]] = load i32, ptr %{{.*}}, align 4
+//CHECK:   store i32 %[[REDPRIVVAL]], ptr %[[ARRAYOFFSET2]], align 4
+//CHECK:   br label %omp.scan.loop.exit
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 71e8628595aab..7edf88b5ea559 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -120,10 +120,10 @@ atomic {
   llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
   omp.yield
 }
-llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
-  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.wsloop operation}}
-  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
-  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+llvm.func @simd_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{not yet implemented: Unhandled clause reduction with modifier in omp.simd operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       omp.scan inclusive(%prv : !llvm.ptr)
       omp.yield
@@ -162,6 +162,39 @@ llvm.func @parallel_task_reduction_modifier_byref(%x : !llvm.ptr) {
 
 // -----
 
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> f32
+  llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
+  omp.yield
+}
+llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
+  omp.wsloop reduction(mod:inscan, @add_f32 %x -> %prv : !llvm.ptr) {
+    // expected-error at below {{LLVM Translation failed for operation: omp.loop_nest}}
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      // expected-error at below {{not yet implemented: Unhandled clause exclusive in omp.scan operation}}
+      // expected-error at below {{LLVM Translation failed for operation: omp.scan}}
+      omp.scan exclusive(%prv : !llvm.ptr)
+      omp.yield
+    }
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @single_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.single operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.single}}
diff --git a/openmp/runtime/test/scan/scan.f90 b/openmp/runtime/test/scan/scan.f90
new file mode 100644
index 0000000000000..76263508105d0
--- /dev/null
+++ b/openmp/runtime/test/scan/scan.f90
@@ -0,0 +1,38 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+program inclusive_scan
+ implicit none
+ integer, parameter :: n = 100
+ integer a(n), b(n)
+ integer x, k, y, z
+
+ ! initialization
+ x = 0
+ do k = 1, n
+  a(k) = k
+ end do
+
+ ! a(k) is included in the computation of producing results in b(k)
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, n
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+
+ print *,'x =', x
+ do k = 1, 10
+  print *, 'b(', k, ') =', b(k)
+ end do
+end program
+!CHECK: x = 5050
+!CHECK: b( 1 ) = 1
+!CHECK: b( 2 ) = 3
+!CHECK: b( 3 ) = 6
+!CHECK: b( 4 ) = 10
+!CHECK: b( 5 ) = 15
+!CHECK: b( 6 ) = 21
+!CHECK: b( 7 ) = 28
+!CHECK: b( 8 ) = 36
+!CHECK: b( 9 ) = 45
+!CHECK: b( 10 ) = 55

>From 4011dd537d9830ee4a0379cba80413d1e4cdf9c3 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Wed, 1 Jul 2026 02:44:02 -0500
Subject: [PATCH 2/3] formatting correction

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp        | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 297ebac51de88..e18cc0df79860 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4534,12 +4534,11 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
   bool isInScanRegion =
       wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
                                      mlir::omp::ReductionModifier::inscan);
-  if (failed(initReductionVars(wsloopOp, reductionArgs, builder,
-                               moduleTranslation,
-                               afterAllocas.get()->getSinglePredecessor(),
-                               reductionDecls, privateReductionVariables,
-                               reductionVariableMap, isByRef, deferredStores,
-                               isInScanRegion)))
+  if (failed(initReductionVars(
+          wsloopOp, reductionArgs, builder, moduleTranslation,
+          afterAllocas.get()->getSinglePredecessor(), reductionDecls,
+          privateReductionVariables, reductionVariableMap, isByRef,
+          deferredStores, isInScanRegion)))
     return failure();
 
   // For `reduction(task, ...)` open a task-reduction scope for the worksharing
@@ -5262,9 +5261,9 @@ initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
     SmallVector<llvm::Value *, 1> phis;
     mapInitializationArgs(wsloopOp, moduleTranslation, builder, reductionDecls,
                           reductionVariableMap, i);
-    if (failed(inlineConvertOmpRegions(
-            reductionDecls[i].getInitializerRegion(),
-            "omp.scan.reduction.init", builder, moduleTranslation, &phis)))
+    if (failed(inlineConvertOmpRegions(reductionDecls[i].getInitializerRegion(),
+                                       "omp.scan.reduction.init", builder,
+                                       moduleTranslation, &phis)))
       return failure();
     assert(phis.size() == 1 &&
            "expected one value to be yielded from the reduction init region");

>From 64b530d3f37e0cd08ca622b538d9bec1eb348b50 Mon Sep 17 00:00:00 2001
From: Chandra Ghale <ghale at pe34genoa.hpc.amslabs.hpecorp.net>
Date: Wed, 22 Jul 2026 05:09:34 -0500
Subject: [PATCH 3/3] Address review comments

---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 46 +++++++++++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 26 +++++++-
 .../Target/LLVMIR/openmp-reduction-scan.mlir  | 14 +++--
 mlir/test/Target/LLVMIR/openmp-todo.mlir      | 31 ++++++++++
 openmp/runtime/test/scan/scan-bounds.f90      | 61 +++++++++++++++++++
 openmp/runtime/test/scan/scan-edge.f90        | 61 +++++++++++++++++++
 openmp/runtime/test/scan/scan-nonidentity.f90 | 28 +++++++++
 7 files changed, 251 insertions(+), 16 deletions(-)
 create mode 100644 openmp/runtime/test/scan/scan-bounds.f90
 create mode 100644 openmp/runtime/test/scan/scan-edge.f90
 create mode 100644 openmp/runtime/test/scan/scan-nonidentity.f90

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index f8c1999fe1b89..f62ea4cf2f9da 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -5307,7 +5307,17 @@ Error OpenMPIRBuilder::emitScanBasedDirectiveFinalsIR(
                                              "arrayOffset");
       Value *Src = Builder.CreateLoad(SrcTy, Val);
 
-      Builder.CreateStore(Src, OrigVar);
+      // For a zero-trip loop the buffer slot is uninitialized; keep the
+      // original variable's incoming value instead of storing garbage. The
+      // load above stays in bounds because the buffer was allocated with
+      // `Span + 1` elements, so index `Span` is always valid.
+      Value *Cur = Builder.CreateLoad(SrcTy, OrigVar);
+      Value *HasIters = Builder.CreateICmpUGT(
+          ScanRedInfo->Span,
+          llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 0));
+      Value *Final = Builder.CreateSelect(HasIters, Src, Cur);
+
+      Builder.CreateStore(Final, OrigVar);
       Builder.CreateFree(Buff);
     }
     return Error::success();
@@ -5370,7 +5380,15 @@ OpenMPIRBuilder::InsertPointOrErrorTy OpenMPIRBuilder::emitScanReduction(
         ScanRedInfo->Span,
         llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 1));
     Builder.SetInsertPoint(InputBB);
-    Builder.CreateBr(LoopBB);
+    // The log-scan prefix computation is only well-defined when there are at
+    // least two elements: `log2(Span)` is invalid for Span == 0 and the loop
+    // trip computation underflows/loops forever for Span <= 1. For Span <= 1
+    // the buffer already holds the final value(s), so skip straight to the
+    // exit.
+    llvm::Value *SpanGuard = Builder.CreateICmpUGT(
+        ScanRedInfo->Span,
+        llvm::ConstantInt::get(ScanRedInfo->Span->getType(), 1));
+    Builder.CreateCondBr(SpanGuard, LoopBB, ExitBB);
     emitBlock(LoopBB, CurFn);
     Builder.SetInsertPoint(LoopBB);
 
@@ -5617,7 +5635,6 @@ OpenMPIRBuilder::createCanonicalScanLoops(
 
   auto BodyGen = [=](InsertPointTy CodeGenIP, Value *IV) {
     Builder.restoreIP(CodeGenIP);
-    ScanRedInfo->IV = IV;
     createScanBBs(ScanRedInfo);
     BasicBlock *InputBlock = Builder.GetInsertBlock();
     Instruction *Terminator = InputBlock->getTerminator();
@@ -5636,9 +5653,13 @@ OpenMPIRBuilder::createCanonicalScanLoops(
   };
 
   const auto &&InputLoopGen = [&]() -> Error {
+    // Pass an empty ComputeIP: the trip count must be (re)computed at the
+    // loop's own location. Reusing the outer ComputeIP is invalid here because
+    // it was invalidated by the `scan.init` split above and would emit the
+    // recomputed trip count after a block terminator for runtime bounds.
     Expected<CanonicalLoopInfo *> LoopInfo = createCanonicalLoop(
         Builder.saveIP(), BodyGen, Start, Stop, Step, IsSigned, InclusiveStop,
-        ComputeIP, Name, true, ScanRedInfo);
+        /*ComputeIP=*/InsertPointTy(), Name, true, ScanRedInfo);
     if (!LoopInfo)
       return LoopInfo.takeError();
     Result.push_back(*LoopInfo);
@@ -5646,9 +5667,9 @@ OpenMPIRBuilder::createCanonicalScanLoops(
     return Error::success();
   };
   const auto &&ScanLoopGen = [&](LocationDescription Loc) -> Error {
-    Expected<CanonicalLoopInfo *> LoopInfo =
-        createCanonicalLoop(Loc, BodyGen, Start, Stop, Step, IsSigned,
-                            InclusiveStop, ComputeIP, Name, true, ScanRedInfo);
+    Expected<CanonicalLoopInfo *> LoopInfo = createCanonicalLoop(
+        Loc, BodyGen, Start, Stop, Step, IsSigned, InclusiveStop,
+        /*ComputeIP=*/InsertPointTy(), Name, true, ScanRedInfo);
     if (!LoopInfo)
       return LoopInfo.takeError();
     Result.push_back(*LoopInfo);
@@ -5737,8 +5758,15 @@ Expected<CanonicalLoopInfo *> OpenMPIRBuilder::createCanonicalLoop(
     Builder.restoreIP(CodeGenIP);
     Value *Span = Builder.CreateMul(IV, Step);
     Value *IndVar = Builder.CreateAdd(Span, Start);
-    if (InScan)
-      ScanRedInfo->IV = IndVar;
+    if (InScan) {
+      // The scan buffer is 1-indexed by iteration number (buffer[1..TripCount])
+      // rather than by the source induction variable. Using the source value
+      // would index out of bounds for loops that do not start at 1 or that use
+      // a non-unit stride. `IV` here is the 0-based canonical loop counter, so
+      // the 1-based buffer index is `IV + 1`.
+      ScanRedInfo->IV =
+          Builder.CreateAdd(IV, ConstantInt::get(IV->getType(), 1));
+    }
     return BodyGenCB(Builder.saveIP(), IndVar);
   };
   LocationDescription LoopLoc =
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e18cc0df79860..5af1a384d3edb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -430,6 +430,16 @@ static LogicalResult checkImplementationStatus(Operation &op) {
               break;
             }
         }
+      } else if (inscanModifierSupported) {
+        if (auto byref = op.getReductionByref()) {
+          // Scan reductions (inscan modifier) only support by-value (scalar)
+          // reductions for now.
+          for (bool isByRef : *byref)
+            if (isByRef) {
+              result = todo("inscan reduction modifier with by-ref reduction");
+              break;
+            }
+        }
       }
     }
   };
@@ -5268,8 +5278,20 @@ initScanReductionVars(omp::LoopNestOp loopOp, llvm::IRBuilderBase &builder,
     assert(phis.size() == 1 &&
            "expected one value to be yielded from the reduction init region");
     setInsertPointForPossiblyEmptyBlock(builder);
-    builder.CreateStore(phis[0],
-                        moduleTranslation.lookupValue(reductionArgs[i]));
+
+    // On the first iteration, seed the reduction variable with the original
+    // variable's incoming value so that a non-identity initial value
+    // participates in the scan (e.g. `x = 10` before an inclusive-scan loop of
+    // `+1` must yield 11, 12, ...). On subsequent iterations, reset to the
+    // reduction identity. `scanInfo->IV` is the 1-based iteration index.
+    llvm::Value *identity = phis[0];
+    llvm::Value *origVar =
+        moduleTranslation.lookupValue(wsloopOp.getReductionVars()[i]);
+    llvm::Value *origVal = builder.CreateLoad(identity->getType(), origVar);
+    llvm::Value *isFirstIter = builder.CreateICmpEQ(
+        scanInfo->IV, llvm::ConstantInt::get(scanInfo->IV->getType(), 1));
+    llvm::Value *seed = builder.CreateSelect(isFirstIter, origVal, identity);
+    builder.CreateStore(seed, moduleTranslation.lookupValue(reductionArgs[i]));
   }
   return success();
 }
diff --git a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
index c6b8daccad313..6ef58b5013e08 100644
--- a/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-reduction-scan.mlir
@@ -73,7 +73,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK: %[[FREE_VAR:.+]] = load ptr, ptr %[[BUFF_PTR]], align 8
 //CHECK:  %[[ARRLAST:.+]] = getelementptr inbounds i32, ptr %[[FREE_VAR]], i32 100
 //CHECK:  %[[RES:.+]] = load i32, ptr %[[ARRLAST]], align 4
-//CHECK:  store i32 %[[RES]], ptr %loadgep{{.*}}, align 4
+//CHECK:  %[[CUR:.+]] = load i32, ptr %loadgep{{.*}}, align 4
+//CHECK:  %[[FIN:.+]] = select i1 {{.*}}, i32 %[[RES]], i32 %[[CUR]]
+//CHECK:  store i32 %[[FIN]], ptr %loadgep{{.*}}, align 4
 //CHECK: tail call void @free(ptr %[[FREE_VAR]])
 //CHECK: @__kmpc_end_masked
 //CHECK: omp.inscan.dispatch{{.*}}:                            ; preds = %omp_loop.body{{.*}}
@@ -84,7 +86,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK:   %[[LOG:.+]] = call double @llvm.log2.f64(double 1.000000e+02) #0
 //CHECK:   %[[CEIL:.+]] = call double @llvm.ceil.f64(double %[[LOG]]) #0
 //CHECK:   %[[UB:.+]] = fptoui double %[[CEIL]] to i32
-//CHECK:   br label %omp.outer.log.scan.body
+//CHECK:   br i1 {{.*}}, label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
+//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit, %omp_region.body{{.*}}
+//CHECK: @__kmpc_end_masked
 //CHECK: omp.outer.log.scan.body:
 //CHECK:   %[[K:.+]] = phi i32 [ 0, %{{.*}} ], [ %[[NEXTK:.+]], %omp.inner.log.scan.exit ]
 //CHECK:   %[[I:.+]] = phi i32 [ 1, %{{.*}} ], [ %[[NEXTI:.+]], %omp.inner.log.scan.exit ]
@@ -95,8 +99,6 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK:   %[[NEXTI]] = shl nuw i32 %[[I]], 1
 //CHECK:   %[[CMP2:.+]] = icmp ne i32 %[[NEXTK]], %[[UB]]
 //CHECK:   br i1 %[[CMP2]], label %omp.outer.log.scan.body, label %omp.outer.log.scan.exit
-//CHECK: omp.outer.log.scan.exit:                          ; preds = %omp.inner.log.scan.exit
-//CHECK: @__kmpc_end_masked
 //CHECK: omp.inner.log.scan.body:                          ; preds = %omp.inner.log.scan.body, %omp.outer.log.scan.body
 //CHECK:   %[[CNT:.+]] = phi i32 [ 99, %omp.outer.log.scan.body ], [ %[[CNTNXT:.+]], %omp.inner.log.scan.body ]
 //CHECK:   %[[BUFF:.+]] = load ptr, ptr %[[BUFF_PTR]]
@@ -114,7 +116,9 @@ llvm.mlir.global internal @_QFEb() {addr_space = 0 : i32} : !llvm.array<100 x i3
 //CHECK: omp.inscan.dispatch:                              ; preds = %omp_loop.body
 //CHECK:   br i1 true, label %omp.before.scan.bb, label %omp.after.scan.bb
 //CHECK: omp.before.scan.bb:
-//CHECK:   store i32 0, ptr %[[REDPRIV:.+]], align 4
+//CHECK:   %[[ISFIRST:.+]] = icmp eq i32 %{{.*}}, 1
+//CHECK:   %[[SEED:.+]] = select i1 %[[ISFIRST]], i32 %{{.*}}, i32 0
+//CHECK:   store i32 %[[SEED]], ptr %[[REDPRIV:.+]], align 4
 //CHECK: omp.loop_nest.region:                             ; preds = %omp.before.scan.bb
 //CHECK:   %[[BUFFER:.+]] = load ptr, ptr %loadgep_vla, align 8
 //CHECK:   %[[ARRAYOFFSET2:.+]] = getelementptr inbounds i32, ptr %[[BUFFER]], i32 %{{.*}}
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 7edf88b5ea559..fe3c4259e75e0 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -195,6 +195,37 @@ llvm.func @scan_reduction(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // -----
 
+omp.declare_reduction @add_f32 : f32
+init {
+^bb0(%arg: f32):
+  %0 = llvm.mlir.constant(0.0 : f32) : f32
+  omp.yield (%0 : f32)
+}
+combiner {
+^bb1(%arg0: f32, %arg1: f32):
+  %1 = llvm.fadd %arg0, %arg1 : f32
+  omp.yield (%1 : f32)
+}
+atomic {
+^bb2(%arg2: !llvm.ptr, %arg3: !llvm.ptr):
+  %2 = llvm.load %arg3 : !llvm.ptr -> f32
+  llvm.atomicrmw fadd %arg2, %2 monotonic : !llvm.ptr, f32
+  omp.yield
+}
+llvm.func @scan_reduction_byref(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
+  // expected-error at below {{not yet implemented: Unhandled clause inscan reduction modifier with by-ref reduction in omp.wsloop operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.wsloop}}
+  omp.wsloop reduction(mod:inscan, byref @add_f32 %x -> %prv : !llvm.ptr) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      omp.scan inclusive(%prv : !llvm.ptr)
+      omp.yield
+    }
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @single_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.single operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.single}}
diff --git a/openmp/runtime/test/scan/scan-bounds.f90 b/openmp/runtime/test/scan/scan-bounds.f90
new file mode 100644
index 0000000000000..9d624954568ff
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-bounds.f90
@@ -0,0 +1,61 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! Arbitrary loop bounds and a non-unit stride must index the scan buffer by the
+! logical iteration number, not by the induction-variable value.
+program scan_bounds
+ implicit none
+ integer, parameter :: n = 30
+ integer :: a(n), b(n)
+ integer :: x, k
+
+ do k = 1, n
+   a(k) = k
+ end do
+
+ ! Arbitrary bounds: iterate from 10 to 20.
+ b = -1
+ x = 0
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 10, 20
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+ print *, 'bounds x =', x
+ do k = 10, 20
+   print *, 'bb(', k, ') =', b(k)
+ end do
+
+ ! Non-unit stride: iterate from 1 to 10 step 2.
+ b = -1
+ x = 0
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, 10, 2
+   x = x + a(k)
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+ print *, 'stride x =', x
+ do k = 1, 10, 2
+   print *, 'sb(', k, ') =', b(k)
+ end do
+end program
+!CHECK: bounds x = 165
+!CHECK: bb( 10 ) = 10
+!CHECK: bb( 11 ) = 21
+!CHECK: bb( 12 ) = 33
+!CHECK: bb( 13 ) = 46
+!CHECK: bb( 14 ) = 60
+!CHECK: bb( 15 ) = 75
+!CHECK: bb( 16 ) = 91
+!CHECK: bb( 17 ) = 108
+!CHECK: bb( 18 ) = 126
+!CHECK: bb( 19 ) = 145
+!CHECK: bb( 20 ) = 165
+!CHECK: stride x = 25
+!CHECK: sb( 1 ) = 1
+!CHECK: sb( 3 ) = 4
+!CHECK: sb( 5 ) = 9
+!CHECK: sb( 7 ) = 16
+!CHECK: sb( 9 ) = 25
diff --git a/openmp/runtime/test/scan/scan-edge.f90 b/openmp/runtime/test/scan/scan-edge.f90
new file mode 100644
index 0000000000000..7cc00da958e6e
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-edge.f90
@@ -0,0 +1,61 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! Zero-trip loops must not crash and must leave the reduction variable
+! unchanged, and runtime (non-constant) trip counts must work correctly.
+module scan_edge_mod
+contains
+ subroutine run_scan(n, a, b)
+   implicit none
+   integer :: n, k
+   integer :: a(n), b(n)
+   integer :: x
+   x = 0
+   !$omp parallel do reduction(inscan, +: x)
+   do k = 1, n
+     x = x + a(k)
+     !$omp scan inclusive(x)
+     b(k) = x
+   end do
+ end subroutine
+end module
+
+program scan_edge
+ use scan_edge_mod
+ implicit none
+ integer :: a(6), b(6), k
+ integer :: z, zk
+ integer :: zb(1)
+
+ ! Runtime bounds: the trip count of run_scan is not a compile-time constant.
+ do k = 1, 6
+   a(k) = k
+ end do
+ b = -1
+ call run_scan(6, a, b)
+ do k = 1, 6
+   print *, 'rb(', k, ') =', b(k)
+ end do
+
+ ! Runtime zero-trip: n = 0 must not crash.
+ call run_scan(0, a, b)
+ print *, 'runtime zero-trip ok'
+
+ ! Constant zero-trip with a non-identity start: z must be unchanged.
+ z = 42
+ !$omp parallel do reduction(inscan, +: z)
+ do zk = 1, 0
+   z = z + 1
+   !$omp scan inclusive(z)
+   zb(zk) = z
+ end do
+ print *, 'z =', z
+end program
+!CHECK: rb( 1 ) = 1
+!CHECK: rb( 2 ) = 3
+!CHECK: rb( 3 ) = 6
+!CHECK: rb( 4 ) = 10
+!CHECK: rb( 5 ) = 15
+!CHECK: rb( 6 ) = 21
+!CHECK: runtime zero-trip ok
+!CHECK: z = 42
diff --git a/openmp/runtime/test/scan/scan-nonidentity.f90 b/openmp/runtime/test/scan/scan-nonidentity.f90
new file mode 100644
index 0000000000000..064b0d757af6e
--- /dev/null
+++ b/openmp/runtime/test/scan/scan-nonidentity.f90
@@ -0,0 +1,28 @@
+! RUN: %flang %flags %openmp_flags -fopenmp-version=51 %s -o %t.exe
+! RUN: %t.exe | FileCheck %s --match-full-lines
+
+! The reduction variable's non-identity incoming value must be included in the
+! first partial result of an inclusive scan.
+program scan_nonidentity
+ implicit none
+ integer :: x, k
+ integer :: b(4)
+
+ x = 10
+ !$omp parallel do reduction(inscan, +: x)
+ do k = 1, 4
+   x = x + 1
+   !$omp scan inclusive(x)
+   b(k) = x
+ end do
+
+ print *, 'x =', x
+ do k = 1, 4
+   print *, 'b(', k, ') =', b(k)
+ end do
+end program
+!CHECK: x = 14
+!CHECK: b( 1 ) = 11
+!CHECK: b( 2 ) = 12
+!CHECK: b( 3 ) = 13
+!CHECK: b( 4 ) = 14


From llvmlistbot at llvm.org  Wed Jul 22 03:58:06 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 22 Jul 2026 03:58:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR] Introduce support for early exits (PR
	#166688)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/166688@github.com>
Message-ID: <6a60a23e.510f576d.33aa7d.f175@mx.google.com>


https://github.com/joker-eph updated https://github.com/llvm/llvm-project/pull/166688

>From 4a20d82d2d0e4b7361c68642e0f695233570650a Mon Sep 17 00:00:00 2001
From: Mehdi Amini <joker.eph at gmail.com>
Date: Sat, 26 Apr 2025 04:51:22 -0700
Subject: [PATCH 1/2] [mlir][scf] Add token-targeted early exits for scf.loop

Summary:
- Introduce a structured region-exit control-flow model for MLIR regions.
- Model region exits with RegionExitTerminatorOpInterface. Normal exits may
  target the immediate parent operation; breaking-control-flow exits request
  propagation to a specific ancestor receiver. Every intermediate parent must
  opt in with PropagateControlFlowBreak until the addressed receiver handles
  the request. In the SCF dialect, scf.loop requests are addressed with the
  control token defined by the loop body.

Core IR:
- Add RegionExitTerminatorOpInterface for terminators that identify possible
  receiver operations for a region exit.
- Add PropagateControlFlowBreak for intermediate operations that forward
  breaking-control-flow requests without consuming them.
- Add HasBreakingControlFlowOpInterface for operations that receive addressed
  breaking-control-flow requests, plus helpers to find the receiver, collect
  nested predecessors, and detect escaping breaking control flow.
- Represent propagated RegionBranchOpInterface successors with explicit target
  operations instead of a parent or propagation sentinel. Add a helper to
  determine which operation owns a successor input list.

SCF dialect:
- Add scf.loop, which defines a control token, accepts optional iter_args and
  results, and implements the breaking-control-flow receiver interface.
- Add scf.break and scf.continue as token-consuming region-exit terminators.
  scf.break requests exit from the target loop and supplies loop results;
  scf.continue requests re-entry into the target loop with next iteration
  values.
- Let scf.yield and scf.reduce implement RegionExitTerminatorOpInterface by
  targeting their immediate parent operation.
- Verify token receivers, propagation traits on forwarding operations, and
  region-branch operand/result type consistency.
- Let scf.if propagate breaks and continues through its regions. Its parser,
  printer, result inference, implicit terminator handling, and
  canonicalizations now account for scf.yield, scf.break, and scf.continue
  terminators.
- Guard yield-specific scf.if rewrites and add a canonicalization that removes
  dead code after an if whose branches both transfer breaking control flow.

Analyses and utilities:
- Teach dead code analysis, dense and sparse backward dataflow, region-branch
  graph traversal, post-dominance, and region inlining to follow concrete exit
  targets instead of treating the immediate parent as the final successor.

Lowering:
- Lower scf.loop in the SCF-to-CF conversion by legalizing nested regions,
  collecting direct break and continue predecessors, inlining the body, turning
  breaks into branches to the continuation block, turning continues into
  backedges, removing the loop token argument, and wiring results through
  continuation block arguments.
- Adjust scf.if lowering to branch only from scf.yield terminators, leaving
  early-exit terminators for the target loop lowering.

Tests and docs:
- Document the cooperative region-exit propagation model in the LangRef.
- Add tests for valid and invalid early-exit IR, region-branch interface
  printing, dominance, dead code analysis, SCCP, SCF-to-CF lowering,
  SCF canonicalization, and an end-to-end early-exit integration case.

Assisted-by: Claude Code
Assisted-by: Codex
---
 mlir/docs/LangRef.md                          | 122 +++-
 mlir/include/mlir/Dialect/SCF/IR/SCF.h        |  70 +++
 mlir/include/mlir/Dialect/SCF/IR/SCFOps.td    | 181 +++++-
 mlir/include/mlir/IR/Dominance.h              |   9 +
 mlir/include/mlir/IR/OpBase.td                |   7 +
 mlir/include/mlir/IR/OpDefinition.h           |  13 +
 mlir/include/mlir/IR/RegionKindInterface.h    | 121 ++++
 mlir/include/mlir/IR/RegionKindInterface.td   |  83 +++
 .../mlir/Interfaces/ControlFlowInterfaces.h   |   6 +
 .../Analysis/DataFlow/DeadCodeAnalysis.cpp    |  12 +-
 mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp  |   9 +-
 mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp |  23 +-
 .../SCFToControlFlow/SCFToControlFlow.cpp     | 237 ++++++--
 mlir/lib/Dialect/SCF/IR/SCF.cpp               | 571 +++++++++++++++---
 .../SCF/IR/ValueBoundsOpInterfaceImpl.cpp     |   4 +-
 .../BufferizableOpInterfaceImpl.cpp           |   4 +-
 mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp   |  14 +-
 mlir/lib/IR/Dominance.cpp                     |  94 ++-
 mlir/lib/IR/RegionKindInterface.cpp           | 168 ++++++
 mlir/lib/IR/Verifier.cpp                      |   1 +
 mlir/lib/Interfaces/ControlFlowInterfaces.cpp | 141 ++++-
 mlir/lib/Transforms/Utils/CMakeLists.txt      |   3 +
 mlir/lib/Transforms/Utils/InliningUtils.cpp   |   6 +
 .../test-dead-code-analysis-early-exit.mlir   | 100 +++
 mlir/test/Analysis/test-dominance.mlir        |  60 ++
 .../convert-early-exit-to-cfg.mlir            | 135 +++++
 mlir/test/Dialect/SCF/loop_canonicalize.mlir  | 300 +++++++++
 mlir/test/IR/early-exit-invalid.mlir          | 142 +++++
 mlir/test/IR/early-exit.mlir                  |  82 +++
 .../Integration/Dialect/SCF/early_exit.mlir   |  82 +++
 mlir/test/Transforms/sccp-early-exit.mlir     | 117 ++++
 mlir/test/lib/Dialect/Test/TestOps.td         |   6 +
 mlir/test/lib/IR/TestDominance.cpp            |  33 +
 mlir/test/lib/Interfaces/CMakeLists.txt       |   1 +
 .../RegionBranchOpInterface/CMakeLists.txt    |   9 +
 .../TestRegionBranchOpInterface.cpp           |  77 +++
 mlir/tools/mlir-opt/CMakeLists.txt            |   1 +
 mlir/tools/mlir-opt/mlir-opt.cpp              |   2 +
 38 files changed, 2844 insertions(+), 202 deletions(-)
 create mode 100644 mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
 create mode 100644 mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
 create mode 100644 mlir/test/Dialect/SCF/loop_canonicalize.mlir
 create mode 100644 mlir/test/IR/early-exit-invalid.mlir
 create mode 100644 mlir/test/IR/early-exit.mlir
 create mode 100644 mlir/test/Integration/Dialect/SCF/early_exit.mlir
 create mode 100644 mlir/test/Transforms/sccp-early-exit.mlir
 create mode 100644 mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
 create mode 100644 mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp

diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..6ba4277ffac9f 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -509,7 +509,18 @@ operation produces results and the operation returns, those results also have
 well-defined values. Execution then proceeds to the next operation in the block
 until the terminator operation at the end of the block is reached; the
 terminator determines the next continuation, if any. The determination of the
-next instruction to execute is the 'passing of control flow'.
+next instruction to execute is the 'passing of control flow'. A nested
+[region-exit terminator](#region-exit-terminators) can also terminate the
+current region with a request for a breaking-control-flow event addressed to a
+specific ancestor operation that implements
+`HasBreakingControlFlowOpInterface`. The terminator itself does not jump
+directly to that ancestor; it transfers control back to the containing
+operation with that request. If the containing operation is not the addressed
+receiver, it must define `PropagateControlFlowBreak` and propagate the same
+request to its own parent. This repeats until the addressed receiver handles
+the request, so every intermediate operation on the path from the terminator to
+the receiving `HasBreakingControlFlowOpInterface` operation must define
+`PropagateControlFlowBreak`.
 
 In general, when control flow is passed to an operation, MLIR does not restrict
 when control flow enters or exits the regions contained in that operation.
@@ -519,25 +530,26 @@ represent possible continuations. Successors explicitly specify destination
 blocks, so control flow within the region can only pass to one of the specified
 successor blocks, as in a `branch` operation. When a terminator has no
 successors, it may pass 1) control back to the containing operation, as in a
-`return` operation, or 2) define that control flow does not continue, as in
-`ub.unreachable`. Terminators without successors therefore do not necessarily
-imply a return to the containing operation, the specific dialect operation
-determines the terminator's semantics. Blocks (other than the entry block) that
-are not listed as a successor of any terminator operation are defined to be
-unreachable and can be removed without affecting the semantics of the
-containing operation.
+`return` operation, 2) define that control flow does not continue, as in
+`ub.unreachable`, or 3) request that the containing operation propagate breaking
+control flow outward toward a specific ancestor. This target designator is
+dialect-defined. Terminators without successors therefore do not
+necessarily imply a return to the containing operation. Blocks (other than the
+entry block) that are not listed as a successor of any terminator operation are
+defined to be unreachable and can be removed without affecting the semantics of
+the containing operation.
 
 Although control flow always enters a region through the entry block, control
 flow may exit a region through any block with an appropriate terminator. The
-standard dialect leverages this capability to define operations with
+SCF dialect for example leverages this capability to define operations with
 Single-Entry-Multiple-Exit (SEME) regions, possibly flowing through different
 blocks in the region and exiting through any block with a `return` operation.
-This behavior is similar to that of a function body in most programming
-languages. In addition, control flow may also not reach the end of a block or
-region, for example if a function call does not return. Such an operation
-prevents control flow from reaching later operations, but does not remove the
-structural requirement that the block must end with a terminator unless the
-enclosing operation opts out with `NoTerminator`.
+This behavior can model that of a function body in most programming languages.
+In addition, control flow may also not reach the end of a block or region, for
+example if a function call does not return. Such an operation prevents control
+flow from reaching later operations, but does not remove the structural
+requirement that the block must end with a terminator unless the enclosing
+operation opts out with `NoTerminator`.
 
 Example:
 
@@ -565,6 +577,86 @@ func.func @accelerator_compute(i64, i1) -> i64 { // An SSACFG region
 }
 ```
 
+#### Region-Exit Terminators
+
+A terminator that implements `RegionExitTerminatorOpInterface` terminates the
+current region and identifies the operation that may receive the region exit. A
+normal region exit can target the immediately containing operation. For
+example, `scf.yield` exits only its own immediately enclosing region and
+returns control to the parent operation.
+
+Some terminators instead request a breaking-control-flow event addressed to a
+specific ancestor. The terminator transfers control back only to its immediate
+parent operation; reaching the addressed ancestor requires each intermediate
+parent operation to collaborate by propagating the request outward. The way a
+terminator designates the ancestor is dialect-defined: it may be a builtin
+`token` operand, an integer count of region levels to exit, a named symbolic
+relationship to a parent operation, or another dialect-specific mechanism.
+
+Every intermediate operation between a breaking terminator and the addressed
+receiver must define `PropagateControlFlowBreak`. The receiver must implement
+`HasBreakingControlFlowOpInterface`. For example, a loop operation nested
+inside another loop operation body carries both traits simultaneously: it
+handles breaks whose token identifies itself, and propagates breaks whose token
+identifies an outer loop.
+
+Region-exit terminators may carry values, which are propagated to the
+target operation. For example, when breaking out of a loop that produces
+results, the terminator supplies those result values. The exact mapping between
+terminator operands and the receiving op's results is dialect-defined (the
+receiving op may also ignore operands entirely).
+
+Examples:
+
+```mlir
+// scf.yield is the standard immediate region-exit terminator.
+// It exits only its own immediately enclosing region.
+scf.if %cond {
+  scf.yield  // returns control to the immediate parent of scf.if
+}
+```
+
+```mlir
+// Trait legend:
+//   [H]    = HasBreakingControlFlowOpInterface (receives/catches the break)
+//   [P]    = PropagateControlFlowBreak (passes the break request upward)
+//   [H][P] = both: handles breaks whose token identifies it, and propagates
+//            breaks whose token identifies an outer loop
+scf.loop token(%outer) {                    // [H][P]
+  scf.loop token(%inner) {                  // [H][P]
+    scf.if %cond1 {                         // [P]
+      // Requests a break of the inner loop.
+      scf.break [%inner]
+    }
+    scf.if %cond2 {                         // [P]
+      // Requests a break of the outer loop. The scf.if and inner scf.loop
+      // both propagate the request upward.
+      scf.break [%outer]
+    }
+    scf.if %cond3 {                         // [P]
+      // Requests re-entry into the inner loop.
+      scf.continue [%inner]
+    }
+  }
+}
+return
+```
+
+```mlir
+// A loop that yields a result value on early exit.
+// scf.break carries operands that become the loop's results. (scf.continue
+// would instead carry operands that become the next iteration's iter_args,
+// but this loop has none.)
+%result = scf.loop token(%loop) -> f32 {    // [H]
+  scf.if %found {                           // [P]
+    // %value becomes the loop result.
+    scf.break [%loop] %value : f32
+  }
+  // Re-enter the loop for the next iteration (no iter_args here).
+  scf.continue [%loop]
+}
+```
+
 #### Operations with Multiple Regions
 
 An operation containing multiple regions also completely determines the
diff --git a/mlir/include/mlir/Dialect/SCF/IR/SCF.h b/mlir/include/mlir/Dialect/SCF/IR/SCF.h
index 44cbb458d94fe..8918a1e89efdf 100644
--- a/mlir/include/mlir/Dialect/SCF/IR/SCF.h
+++ b/mlir/include/mlir/Dialect/SCF/IR/SCF.h
@@ -30,6 +30,11 @@
 namespace mlir {
 namespace scf {
 void buildTerminatedBody(OpBuilder &builder, Location loc);
+
+namespace op_impl {
+struct IfOpImplicitTerminatorType;
+struct LoopOpImplicitTerminatorType;
+} // namespace op_impl
 } // namespace scf
 } // namespace mlir
 
@@ -112,6 +117,71 @@ SmallVector<Value> replaceAndCastForOpIterArg(RewriterBase &rewriter,
                                               OpOperand &operand,
                                               Value replacement,
                                               const ValueTypeCastFnTy &castFn);
+namespace op_impl {
+
+//===----------------------------------------------------------------------===//
+// ControlFlowImplicitTerminatorOperation
+//===----------------------------------------------------------------------===//
+
+/// This class provides an interface compatible with
+/// SingleBlockImplicitTerminator, but allows multiple types of potential
+/// terminators aside from just one. If a terminator isn't present, this will
+/// generate a `ImplicitOpT` operation.
+template <typename ImplicitOpT, typename... OtherTerminatorOpTs>
+struct ControlFlowImplicitTerminatorOpType {
+  /// Implementation of `classof` that supports all of the potential terminator
+  /// operations.
+  static bool classof(Operation *op) {
+    return isa<ImplicitOpT, OtherTerminatorOpTs...>(op);
+  }
+
+  //===--------------------------------------------------------------------===//
+  // Implicit Terminator Methods
+
+  /// The following methods are all used when interacting with the "implicit"
+  /// terminator.
+
+  template <typename... Args>
+  static void build(Args &&...args) {
+    ImplicitOpT::build(std::forward<Args>(args)...);
+  }
+  static constexpr StringLiteral getOperationName() {
+    return ImplicitOpT::getOperationName();
+  }
+};
+/// An implicit terminator type for `if` operations. Empty regions still receive
+/// an `scf.yield`, but explicitly terminated regions may use any terminator
+/// implementing RegionExitTerminatorOpInterface so dialect-defined exits can
+/// propagate through `scf.if`.
+struct IfOpImplicitTerminatorType {
+  static bool classof(Operation *op) {
+    return isa<RegionExitTerminatorOpInterface>(op);
+  }
+
+  template <typename... Args>
+  static void build(Args &&...args) {
+    YieldOp::build(std::forward<Args>(args)...);
+  }
+  static constexpr StringLiteral getOperationName() {
+    return YieldOp::getOperationName();
+  }
+};
+struct LoopOpImplicitTerminatorType
+    : public ControlFlowImplicitTerminatorOpType<ContinueOp, BreakOp> {
+  /// Build the implicit `scf.continue` terminator. The control token consumed
+  /// by the terminator is the loop body's entry block argument #0; the
+  /// insertion block is guaranteed to be that body when the implicit
+  /// terminator is materialized. This keeps `scf.continue`'s public builders
+  /// free of any hidden block-argument assumption.
+  static void build(OpBuilder &builder, OperationState &state) {
+    Block *block = builder.getInsertionBlock();
+    assert(block && block->getNumArguments() > 0 &&
+           isa<TokenType>(block->getArgument(0).getType()) &&
+           "expected insertion block with a token-typed loop control argument");
+    state.addOperands(block->getArgument(0));
+  }
+};
+} // namespace op_impl
 
 /// Helper function to compute the difference between two values. This is used
 /// by the loop implementations to compute the trip count.
diff --git a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
index c0d1ac501cc77..05a3e37dc8676 100644
--- a/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
+++ b/mlir/include/mlir/Dialect/SCF/IR/SCFOps.td
@@ -15,6 +15,7 @@
 
 include "mlir/Interfaces/ControlFlowInterfaces.td"
 include "mlir/Interfaces/LoopLikeInterface.td"
+include "mlir/IR/OpAsmInterface.td"
 include "mlir/IR/RegionKindInterface.td"
 include "mlir/Dialect/SCF/IR/DeviceMappingInterface.td"
 include "mlir/Interfaces/DestinationStyleOpInterface.td"
@@ -146,6 +147,168 @@ def ExecuteRegionOp : SCF_Op<"execute_region", [
 
   let hasVerifier = 1;
 }
+//===----------------------------------------------------------------------===//
+// LoopOp
+//===----------------------------------------------------------------------===//
+
+def LoopOp : SCF_Op<"loop",[
+    AutomaticAllocationScope,
+    OpAsmOpInterface,
+    RecursiveMemoryEffects,
+    PropagateControlFlowBreak,
+    TokenProducerTrait,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface,
+        ["getEntrySuccessorOperands", "getSuccessorInputs"]>,
+    SingleBlockImplicitTerminator<"op_impl::LoopOpImplicitTerminatorType">,
+    HasBreakingControlFlowOpInterface,
+    HasNestedTerminator<["ContinueOp", "BreakOp"]>
+  ]> {
+  let summary = "Loop until a break operation";
+  let description = [{
+    The `loop` operation represents a loop that executes until a `break` is reached.
+
+    The loop consists of (1) a set of loop-carried values which are initialized by
+    `initValues` and updated by each iteration of the loop, and
+    (2) a region which represents the loop body.
+
+    The loop will execute the body of the loop until a `break` is executed.
+
+    Each control path of the loop must be terminated by:
+
+    - a `continue` that yields the next iteration's value for each loop carried variable.
+    - a `break` that terminates the loop and yields the final loop results.
+    - a nested terminator requesting to propagate the control-flow to a parent.
+  }];
+
+  let arguments = (ins Variadic<AnyType>:$initValues);
+  let results = (outs Variadic<AnyType>:$resultValues);
+  let regions = (region SizedRegion<1>:$region);
+
+  let extraClassDeclaration = [{
+    /// Return the iteration values of the loop region.
+    Block::BlockArgListType getRegionIterValues() {
+      return getRegion().getArguments().drop_front();
+    }
+
+    /// Return the `index`-th region iteration value.
+    BlockArgument getRegionIterValue(unsigned index) {
+      return getRegionIterValues()[index];
+    }
+
+    /// Return the loop control token.
+    BlockArgument getControlToken() {
+      return getRegion().getArgument(0);
+    }
+
+    /// Returns the number of region arguments for loop-carried values.
+    unsigned getNumRegionIterValues() {
+      return getRegion().getNumArguments() - 1;
+    }
+
+    /// Returns the loop block body
+    Block *getBody() { return &getRegion().front(); }
+  }];
+
+  let hasCustomAssemblyFormat = 1;
+  let hasRegionVerifier = 1;
+  let hasCanonicalizer = 1;
+}
+
+
+//===----------------------------------------------------------------------===//
+// BreakOp
+//===----------------------------------------------------------------------===//
+
+def BreakOp : SCF_Op<"break", [
+    Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>,
+    ParentOneOf<["IfOp", "LoopOp"]>
+  ]> {
+  let summary = "Break from loop";
+  let description = [{
+    The `break` operation is a terminator that exits one or more nested
+    regions and terminates the `scf.loop` that defined its control token.
+
+    The `break` may yield any number of operands; their types must match the
+    result types of the target `scf.loop`.
+
+    Example — break out of the immediately enclosing loop:
+    ```mlir
+    scf.loop token(%loop) -> i32 {
+      scf.break [%loop] %result : i32
+    }
+    ```
+
+    Example — break out of a loop through an enclosing `scf.if`:
+    ```mlir
+    scf.loop token(%loop) {
+      scf.if %cond {
+        scf.break [%loop]
+      }
+      scf.continue [%loop]
+    }
+    ```
+  }];
+
+
+  let arguments = (ins Token:$targetToken, Variadic<AnyType>:$args);
+  let assemblyFormat = [{
+    ` ` `[` $targetToken `]` ($args^ `:` type($args))? attr-dict
+  }];
+  let hasVerifier = 1;
+}
+
+
+//===----------------------------------------------------------------------===//
+// ContinueOp
+//===----------------------------------------------------------------------===//
+
+def ContinueOp : SCF_Op<"continue", [
+    Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>,
+    ParentOneOf<["IfOp", "LoopOp"]>
+  ]> {
+  let summary = "Continue to next loop iteration";
+  let description = [{
+    The `continue` operation is a terminator that re-enters a `scf.loop`
+    for its next iteration. The target loop is the one that defined the control
+    token operand.
+
+    The operands of `continue` become the loop-carried values (iter_args) for
+    the next iteration; their types must match the loop's iter_arg types.
+
+    Example — continue the immediately enclosing loop:
+    ```mlir
+    scf.loop token(%loop) iter_args(%i = %init) : i32 {
+      %next = arith.addi %i, %one : i32
+      scf.continue [%loop] %next : i32
+    }
+    ```
+
+    Example — continue an outer loop from inside a nested `scf.if`:
+    ```mlir
+    scf.loop token(%outer) iter_args(%counter = %init) : i64 {
+      scf.loop token(%inner) iter_args(%inner_arg = %counter) : i64 {
+        scf.if %restart_outer {
+          scf.continue [%outer] %inner_arg : i64
+        }
+        scf.continue [%inner] %inner_arg : i64
+      }
+      scf.continue [%outer] %counter : i64
+    }
+    ```
+  }];
+
+  let arguments = (ins Token:$targetToken, Variadic<AnyType>:$args);
+  let assemblyFormat = [{
+    ` ` `[` $targetToken `]` ($args^ `:` type($args))? attr-dict
+  }];
+  let hasVerifier = 1;
+}
 
 //===----------------------------------------------------------------------===//
 // ForOp
@@ -714,8 +877,8 @@ def IfOp : SCF_Op<"if", [DeclareOpInterfaceMethods<RegionBranchOpInterface, [
     "getNumRegionInvocations", "getRegionInvocationBounds",
     "getEntrySuccessorRegions", "getSuccessorInputs"]>,
     DeclareOpInterfaceMethods<PromotableRegionOpInterface>,
-    InferTypeOpAdaptor, SingleBlockImplicitTerminator<"scf::YieldOp">,
-    RecursiveMemoryEffects, RecursivelySpeculatable, NoRegionArguments]> {
+    InferTypeOpAdaptor, SingleBlockImplicitTerminator<"op_impl::IfOpImplicitTerminatorType">,
+    RecursiveMemoryEffects, RecursivelySpeculatable, NoRegionArguments, PropagateControlFlowBreak]> {
   let summary = "if-then-else operation";
   let description = [{
     The `scf.if` operation represents an if-then-else construct for
@@ -798,9 +961,17 @@ def IfOp : SCF_Op<"if", [DeclareOpInterfaceMethods<RegionBranchOpInterface, [
                                   : OpBuilder::atBlockEnd(body, listener);
     }
     Block* thenBlock();
-    YieldOp thenYield();
+    /// Returns the terminator of the then block. May be scf.break,
+    /// scf.continue, or scf.yield.
+    Operation *thenTerminator() {
+      return thenBlock()->getTerminator();
+    }
     Block* elseBlock();
-    YieldOp elseYield();
+    /// Returns the terminator of the else block. May be scf.break,
+    /// scf.continue, or scf.yield.
+    Operation *elseTerminator() {
+      return elseBlock()->getTerminator();
+    }
   }];
   let hasFolder = 1;
   let hasCanonicalizer = 1;
@@ -918,6 +1089,7 @@ def ParallelOp : SCF_Op<"parallel",
 
 def ReduceOp : SCF_Op<"reduce", [
     Terminator, HasParent<"ParallelOp">, RecursiveMemoryEffects,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
     DeclareOpInterfaceMethods<PromotableRegionOpInterface>,
     DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
                               ["getMutableSuccessorOperands"]>]> {
@@ -1219,6 +1391,7 @@ def IndexSwitchOp : SCF_Op<"index_switch", [RecursiveMemoryEffects,
 //===----------------------------------------------------------------------===//
 
 def YieldOp : SCF_Op<"yield", [Pure, ReturnLike, Terminator,
+    DeclareOpInterfaceMethods<RegionExitTerminatorOpInterface>,
     ParentOneOf<["ExecuteRegionOp", "ForOp", "IfOp", "IndexSwitchOp",
                  "WhileOp"]>]> {
   let summary = "loop yield and termination operation";
diff --git a/mlir/include/mlir/IR/Dominance.h b/mlir/include/mlir/IR/Dominance.h
index 70924a2e9ae59..1585f1881136b 100644
--- a/mlir/include/mlir/IR/Dominance.h
+++ b/mlir/include/mlir/IR/Dominance.h
@@ -123,6 +123,11 @@ class DominanceInfoBase {
                              Block::iterator bIt,
                              bool enclosingOk = true) const;
 
+  /// Return true if `op` contains a nested breaking terminator that escapes
+  /// through it to an ancestor. This cache is invalidated together with
+  /// dominance info.
+  bool hasBreakingControlFlowOpsCached(Operation *op) const;
+
   /// A mapping of regions to their base dominator tree and a cached
   /// "hasSSADominance" bit. This map does not contain dominator trees for
   /// single block CFG regions, but we do want to cache the "hasSSADominance"
@@ -131,6 +136,10 @@ class DominanceInfoBase {
   ///
   mutable DenseMap<Region *, llvm::PointerIntPair<DomTree *, 1, bool>>
       dominanceInfos;
+
+  /// Cached escaping breaking-terminator query results. This is stable for the
+  /// lifetime of the analyzed IR and cleared on dominance invalidation.
+  mutable DenseMap<Operation *, bool> breakingControlFlowOpsCache;
 };
 
 extern template class DominanceInfoBase</*IsPostDom=*/true>;
diff --git a/mlir/include/mlir/IR/OpBase.td b/mlir/include/mlir/IR/OpBase.td
index 0d0669e90c3f7..ce7c891076420 100644
--- a/mlir/include/mlir/IR/OpBase.td
+++ b/mlir/include/mlir/IR/OpBase.td
@@ -134,6 +134,13 @@ class SingleBlockImplicitTerminatorImpl<string op>
 class SingleBlockImplicitTerminator<string op>
     : TraitList<[SingleBlock, SingleBlockImplicitTerminatorImpl<op>]>;
 
+// This operation has nested regions with the supplied list of terminator
+// operations.
+class HasNestedTerminator<list<string> ops>
+    : ParamNativeOpTrait<"HasNestedTerminators", !interleave(ops, ", ")>,
+      StructuralOpTrait;
+
+
 // Op's regions don't have terminator.
 def NoTerminator : NativeOpTrait<"NoTerminator">, StructuralOpTrait;
 
diff --git a/mlir/include/mlir/IR/OpDefinition.h b/mlir/include/mlir/IR/OpDefinition.h
index bd7fa1ffd4428..a146d17433ce3 100644
--- a/mlir/include/mlir/IR/OpDefinition.h
+++ b/mlir/include/mlir/IR/OpDefinition.h
@@ -1362,6 +1362,19 @@ struct HasAncestor {
   };
 };
 
+/// This class provides a verifier for ops that are expecting to have nested
+/// predecessors.
+template <typename... NestedPredecessorOpTypes>
+struct HasNestedTerminators {
+  template <typename ConcreteType>
+  class Impl : public TraitBase<ConcreteType, Impl> {
+  public:
+    static bool acceptsTerminator(Operation *predecessor) {
+      return llvm::isa_and_nonnull<NestedPredecessorOpTypes...>(predecessor);
+    }
+  };
+};
+
 /// A trait for operations that have an attribute specifying operand segments.
 ///
 /// Certain operations can have multiple variadic operands and their size
diff --git a/mlir/include/mlir/IR/RegionKindInterface.h b/mlir/include/mlir/IR/RegionKindInterface.h
index d6d3aeeb9bd05..2ada56ba6c193 100644
--- a/mlir/include/mlir/IR/RegionKindInterface.h
+++ b/mlir/include/mlir/IR/RegionKindInterface.h
@@ -36,6 +36,26 @@ class HasOnlyGraphRegion : public TraitBase<ConcreteType, HasOnlyGraphRegion> {
   static RegionKind getRegionKind(unsigned index) { return RegionKind::Graph; }
   static bool hasSSADominance(unsigned index) { return false; }
 };
+
+/// Indicates that this operation is transparent to breaking control flow:
+/// a terminator (e.g. scf.break / scf.continue) can propagate through
+/// this op on its way to the addressed HasBreakingControlFlowOpInterface
+/// ancestor. The op does NOT consume the break; it simply passes it upward.
+/// All intermediate ops that receive and forward the break request must carry
+/// this trait.
+template <typename ConcreteType>
+class PropagateControlFlowBreak
+    : public TraitBase<ConcreteType, PropagateControlFlowBreak> {
+public:
+  static LogicalResult verifyTrait(Operation *op) {
+    // Verify the operation has regions and can handle breaking control flow
+    if (op->getNumRegions() == 0)
+      return op->emitOpError(
+          "operation with PropagateControlFlowBreak trait must have regions");
+    return success();
+  }
+};
+
 } // namespace OpTrait
 
 /// Return "true" if the given region may have SSA dominance. This function also
@@ -49,8 +69,109 @@ bool mayHaveSSADominance(Region &region);
 /// implement the RegionKindInterface.
 bool mayBeGraphRegion(Region &region);
 
+/// Summary of breaking terminator operations nested under an op.
+struct NestedBreakingControlFlowInfo {
+  /// Breaking terminators that may target the queried op directly. These
+  /// transfer control to the queried op rather than only to an intermediate
+  /// PropagateControlFlowBreak op.
+  SmallVector<Operation *> predecessors;
+
+  /// True if any direct predecessor targets the queried op from below the
+  /// queried op's immediate region.
+  bool hasNestedPredecessors = false;
+
+  /// True if any breaking terminator under the queried op targets one of the
+  /// queried op's ancestors, escaping through the queried op.
+  bool hasBreakingControlFlowOps = false;
+};
+
+/// Collect nested breaking-control-flow information for `op` with a single walk
+/// of its nested region tree.
+NestedBreakingControlFlowInfo getNestedBreakingControlFlowInfo(Operation *op);
+
+/// Return true if `op` (which implements HasBreakingControlFlowOpInterface)
+/// contains at least one breaking terminator that directly targets it from a
+/// nested region. Such a terminator is a "nested predecessor" of `op` because
+/// control flow may re-enter or exit `op` through a request propagated from a
+/// deeply nested site rather than only through an immediately enclosing
+/// terminator.
+bool hasNestedPredecessors(Operation *op);
+
+/// Return true if `op` contains any breaking terminator that would "break
+/// through" `op` towards an outer HasBreakingControlFlowOpInterface ancestor.
+/// This is used to detect whether an op's post-dominance is disrupted by an
+/// early-exit request that propagates through it.
+bool hasBreakingControlFlowOps(Operation *op);
+
+/// Collect all breaking terminators nested inside `op` that potentially
+/// directly target `op`. These are the ops that may transfer control flow to
+/// `op` on an early exit.
+void collectAllNestedPredecessors(Operation *op,
+                                  SmallVector<Operation *> &predecessors);
 } // namespace mlir
 
 #include "mlir/IR/RegionKindInterface.h.inc"
 
+namespace mlir {
+namespace detail {
+/// Implementation helper for visitNestedBreakingControlFlowOps. Walks the
+/// regions of `op` and invokes `callback` for every breaking terminator that
+/// either targets `op` or propagates further upward through `op`.
+/// The `nestedLevel` argument passed to the callback is the 1-based depth of
+/// the terminator relative to `op`'s outermost region.
+void visitNestedBreakingControlFlowOpsImpl(
+    Operation *op,
+    function_ref<WalkResult(RegionExitTerminatorOpInterface, int nestedLevel)>
+        callback);
+} // namespace detail
+
+/// Walk all breaking terminators that are relevant to breaking control
+/// flow inside `op` (see visitNestedBreakingControlFlowOpsImpl). The callback
+/// receives the terminator op and its 1-based nesting level. Callbacks
+/// returning WalkResult support early termination via WalkResult::interrupt();
+/// void-returning callbacks always continue.
+template <typename CallbackT>
+void visitNestedBreakingControlFlowOps(Operation *op, CallbackT &&callback) {
+  using RetT =
+      decltype(callback(std::declval<Operation *>(), std::declval<int>()));
+  if constexpr (std::is_same_v<RetT, WalkResult>) {
+    detail::visitNestedBreakingControlFlowOpsImpl(op, callback);
+  } else {
+    detail::visitNestedBreakingControlFlowOpsImpl(
+        op, [&](RegionExitTerminatorOpInterface visitedOp, int nestedLevel) {
+          callback(visitedOp, nestedLevel);
+          return WalkResult::advance();
+        });
+  }
+}
+
+/// Walk all breaking terminators relevant to breaking control flow
+/// across all top-level ops in `region`.
+template <typename CallbackT>
+void visitNestedBreakingControlFlowOps(Region &region, CallbackT &&callback) {
+  // Pass `callback` as an lvalue: it is reused across iterations, so it must
+  // not be forwarded (which could move from it on the first iteration).
+  for (Operation &op : region.getOps())
+    visitNestedBreakingControlFlowOps(&op, callback);
+}
+
+/// Return true if the given region may contain breaking control flow — either
+/// because its parent op propagates breaks (PropagateControlFlowBreak) or
+/// because it is the body of a HasBreakingControlFlowOpInterface op. Used to
+/// decide whether post-dominance analysis must account for early-exit paths.
+inline bool mightHaveBreakingControlFlow(Region *region) {
+  Operation *parentOp = region->getParentOp();
+  return (!parentOp->isRegistered() ||
+          parentOp->hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+          isa<HasBreakingControlFlowOpInterface>(parentOp));
+}
+
+/// Return the HasBreakingControlFlowOpInterface operations potentially
+/// addressed by the given terminator. Returns an empty vector for non-breaking
+/// terminators or malformed target designators.
+SmallVector<HasBreakingControlFlowOpInterface>
+findPotentialBreakTargets(Operation *terminator);
+
+} // namespace mlir
+
 #endif // MLIR_IR_REGIONKINDINTERFACE_H_
diff --git a/mlir/include/mlir/IR/RegionKindInterface.td b/mlir/include/mlir/IR/RegionKindInterface.td
index 607001a89250e..25f88249e547d 100644
--- a/mlir/include/mlir/IR/RegionKindInterface.td
+++ b/mlir/include/mlir/IR/RegionKindInterface.td
@@ -61,4 +61,87 @@ def GraphRegionNoTerminator : TraitList<[
     HasOnlyGraphRegion
   ]>;
 
+// Indicates that this op may propagate a breaking control-flow event from a
+// nested region upward to the addressed HasBreakingControlFlowOpInterface
+// operation. The op does NOT consume the break itself; it is merely transparent
+// to it. All ops that sit between a breaking terminator and the
+// HasBreakingControlFlowOpInterface ancestor that will ultimately receive the
+// break must carry this trait.
+def PropagateControlFlowBreak : NativeOpTrait<"PropagateControlFlowBreak">;
+
+def HasBreakingControlFlowOpInterface : OpInterface<"HasBreakingControlFlowOpInterface"> {
+  let description = [{
+    Interface for operations that act as the target of a breaking control-flow
+    event (e.g. `scf.break` or `scf.continue`). Every intermediate op must
+    carry the `PropagateControlFlowBreak` trait. The terminator may identify the
+    receiver with any dialect-defined mechanism, such as a builtin token value,
+    an attribute, or a symbolic relationship to an enclosing operation.
+  }];
+  let cppNamespace = "::mlir";
+
+  let methods = [
+    StaticInterfaceMethod<
+      /*desc=*/[{
+        Return true if this operation accepts the given terminator operation
+        as a breaking-control-flow predecessor. This can be implemented through
+        the `HasNestedTerminator<[...]>` trait, otherwise an explicit implementation
+        must be provided (e.g. `return true;` to accept all, or a type check to
+        restrict).
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"acceptsTerminator",
+      /*args=*/(ins "Operation *":$op)
+    >,
+    InterfaceMethod<
+      /*desc=*/[{
+        Return true if this operation has at least one breaking terminator
+        nested inside it that targets this operation directly. Used to decide
+        whether post-dominance analysis must account for early-exit paths.
+      }],
+      /*retTy=*/"bool",
+      /*methodName=*/"hasNestedPredecessors",
+      /*args=*/(ins),
+      /*methodBody=*/[{}],
+      /*defaultImplementation=*/[{
+        return ::mlir::hasNestedPredecessors(this->getOperation());
+      }]
+    >
+  ];
+}
+
+// OpInterface for terminators that leave their current region and identify
+// potential receiving operations using a dialect-defined mechanism. A normal
+// region return may simply target the immediately containing operation; a
+// breaking-control-flow region exit may target an ancestor operation that
+// implements HasBreakingControlFlowOpInterface.
+def RegionExitTerminatorOpInterface : OpInterface<"RegionExitTerminatorOpInterface"> {
+  let description = [{
+    Interface for terminators that leave their current region. Implemented by
+    ops such as `scf.yield`, `scf.break`, and `scf.continue`.
+  }];
+  let cppNamespace = "::mlir";
+
+  let methods = [
+    InterfaceMethod<
+      /*desc=*/[{
+        Return the operations that may receive this region-terminating control
+        flow event. Receivers for breaking control flow must implement
+        `HasBreakingControlFlowOpInterface`; normal region exits may return the
+        immediate parent operation.
+      }],
+      /*retTy=*/"::llvm::SmallVector<::mlir::Operation *>",
+      /*methodName=*/"getPotentialTargets",
+      /*args=*/(ins),
+      /*methodBody=*/[{}]
+    >
+  ];
+
+  let verify = [{
+    static_assert(ConcreteOp::template hasTrait<OpTrait::IsTerminator>(),
+                  "expected operation to be a terminator");
+    return success();
+  }];
+}
+
+
 #endif // MLIR_IR_REGIONKINDINTERFACE
diff --git a/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h b/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
index 48b80e2059e11..cd74de81a7b10 100644
--- a/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
+++ b/mlir/include/mlir/Interfaces/ControlFlowInterfaces.h
@@ -446,6 +446,12 @@ inline llvm::raw_ostream &operator<<(llvm::raw_ostream &os,
                            OpPrintingFlags().skipRegions())
             << ">";
 }
+
+/// Return the RegionBranchOpInterface operation that owns `successor` and
+/// defines its successor inputs.
+RegionBranchOpInterface
+getRegionBranchSuccessorOwner(RegionSuccessor successor);
+
 } // namespace mlir
 
 #endif // MLIR_INTERFACES_CONTROLFLOWINTERFACES_H
diff --git a/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp b/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
index 37aaf1f634a64..cf703785c7e08 100644
--- a/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/DeadCodeAnalysis.cpp
@@ -509,11 +509,15 @@ void DeadCodeAnalysis::visitRegionTerminator(Operation *op,
   if (!operands)
     return;
 
-  SmallVector<RegionSuccessor> successors;
   auto terminator = dyn_cast<RegionBranchTerminatorOpInterface>(op);
   if (!terminator)
     return;
+
+  SmallVector<RegionSuccessor> successors;
+  // Use operand-aware resolution to prune dead successors via constant folding
+  // (e.g. a scf.while condition known to be false).
   terminator.getSuccessorRegions(*operands, successors);
+
   visitRegionBranchEdges(branch, op, successors);
 }
 
@@ -540,8 +544,10 @@ void DeadCodeAnalysis::visitRegionBranchEdges(
     auto *predecessors = getOrCreate<PredecessorState>(point);
     propagateIfChanged(
         predecessors,
-        predecessors->join(predecessorOp,
-                           regionBranchOp.getSuccessorInputs(successor)));
+        predecessors->join(
+            predecessorOp,
+            getRegionBranchSuccessorOwner(successor).getSuccessorInputs(
+                successor)));
     LDBG() << "Added region branch as predecessor for successor: " << *point;
   }
 }
diff --git a/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp b/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
index a2a9018b851ea..dc30aae167ac7 100644
--- a/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/DenseAnalysis.cpp
@@ -636,6 +636,9 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
   branch.getSuccessorRegions(branchPoint, successors);
   LDBG() << "  Processing " << successors.size() << " successor regions";
   for (const RegionSuccessor &successor : successors) {
+    RegionBranchOpInterface successorOwner =
+        getRegionBranchSuccessorOwner(successor);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
     const AbstractDenseLattice *after;
     if (successor.isOperation()) {
       LDBG() << "    Successor is operation";
@@ -643,7 +646,7 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
                             getProgramPointAfter(successor.getSuccessorOp()));
     } else if (successor.getSuccessor()->empty()) {
       LDBG() << "    Successor is empty region";
-      after = getLatticeFor(point, getProgramPointAfter(branch));
+      after = getLatticeFor(point, getProgramPointAfter(successorOwner));
     } else {
       Region *successorRegion = successor.getSuccessor();
       assert(!successorRegion->empty() && "unexpected empty successor region");
@@ -662,7 +665,7 @@ void AbstractDenseBackwardDataFlowAnalysis::visitRegionBranchOperation(
     }
     LDBG() << "    After state: " << *after;
 
-    visitRegionBranchControlFlowTransfer(branch, branchPoint, successor, *after,
-                                         before);
+    visitRegionBranchControlFlowTransfer(successorOwner, branchPoint, successor,
+                                         *after, before);
   }
 }
diff --git a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
index 3d9a375bfb2b7..04f27c8bf622a 100644
--- a/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
+++ b/mlir/lib/Analysis/DataFlow/SparseAnalysis.cpp
@@ -647,14 +647,23 @@ void AbstractSparseBackwardDataFlowAnalysis::
   // non-contiguous in the presence of multiple successors.
   BitVector unaccounted(terminator->getNumOperands(), true);
 
-  RegionBranchSuccessorMapping mapping;
-  branch.getSuccessorOperandInputMapping(mapping,
-                                         RegionBranchPoint(terminator));
-  for (const auto &[operand, inputs] : mapping) {
-    for (Value input : inputs) {
-      meet(getLatticeElement(operand->get()),
+  SmallVector<RegionSuccessor> successors;
+  branch.getSuccessorRegions(RegionBranchPoint(terminator), successors);
+
+  for (RegionSuccessor successor : successors) {
+    RegionBranchOpInterface successorOwner =
+        getRegionBranchSuccessorOwner(successor);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
+    OperandRange operands =
+        branch.getSuccessorOperands(RegionBranchPoint(terminator), successor);
+    ValueRange inputs = successorOwner.getSuccessorInputs(successor);
+    assert(operands.size() == inputs.size() &&
+           "expected the same number of operands and inputs");
+    MutableArrayRef<OpOperand> opOperands(operands.getBase(), operands.size());
+    for (const auto &[operand, input] : llvm::zip_equal(opOperands, inputs)) {
+      meet(getLatticeElement(operand.get()),
            *getLatticeElementFor(getProgramPointAfter(terminator), input));
-      unaccounted.reset(operand->getOperandNumber());
+      unaccounted.reset(operand.getOperandNumber());
     }
   }
 
diff --git a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
index 2972d79c4302f..9dc7a56152a3c 100644
--- a/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
+++ b/mlir/lib/Conversion/SCFToControlFlow/SCFToControlFlow.cpp
@@ -311,8 +311,81 @@ struct ForallLowering : public OpRewritePattern<mlir::scf::ForallOp> {
                                 PatternRewriter &rewriter) const override;
 };
 
+/// Lowers `scf.loop` to a CFG loop. The loop body is inlined; `scf.break` ops
+/// that target this loop become branches to the continuation block and
+/// `scf.continue` ops that target this loop become back-edges to the loop
+/// header.
+struct LoopOpLowering : public OpConversionPattern<LoopOp> {
+  using OpConversionPattern<LoopOp>::OpConversionPattern;
+  void initialize() { setHasBoundedRewriteRecursion(); }
+
+  LogicalResult
+  matchAndRewrite(LoopOp loopOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override;
+};
+
 } // namespace
 
+static LogicalResult lowerIfOpToCFG(IfOp ifOp, PatternRewriter &rewriter) {
+  auto loc = ifOp.getLoc();
+  // Start by splitting the block containing the 'scf.if' into two parts.
+  // The part before will contain the condition, the part after will be the
+  // continuation point.
+  auto *condBlock = rewriter.getInsertionBlock();
+  auto opPosition = rewriter.getInsertionPoint();
+  auto *remainingOpsBlock = rewriter.splitBlock(condBlock, opPosition);
+  Block *continueBlock;
+  if (ifOp.getNumResults() == 0) {
+    continueBlock = remainingOpsBlock;
+  } else {
+    continueBlock =
+        rewriter.createBlock(remainingOpsBlock, ifOp.getResultTypes(),
+                             SmallVector<Location>(ifOp.getNumResults(), loc));
+    cf::BranchOp::create(rewriter, loc, remainingOpsBlock);
+  }
+
+  // Move blocks from the "then" region to the region containing 'scf.if',
+  // place it before the continuation block, and branch to it.
+  auto &thenRegion = ifOp.getThenRegion();
+  if (thenRegion.empty() || thenRegion.back().empty())
+    return failure();
+  auto *thenBlock = &thenRegion.front();
+  Operation *thenTerminator = thenRegion.back().getTerminator();
+  ValueRange thenTerminatorOperands = thenTerminator->getOperands();
+  rewriter.setInsertionPointToEnd(&thenRegion.back());
+  if (isa<scf::YieldOp>(thenTerminator)) {
+    cf::BranchOp::create(rewriter, loc, continueBlock, thenTerminatorOperands);
+    rewriter.eraseOp(thenTerminator);
+  }
+  rewriter.inlineRegionBefore(thenRegion, continueBlock);
+
+  // Move blocks from the "else" region (if present) to the region containing
+  // 'scf.if', place it before the continuation block and branch to it.  It
+  // will be placed after the "then" regions.
+  auto *elseBlock = continueBlock;
+  auto &elseRegion = ifOp.getElseRegion();
+  if (!elseRegion.empty() && !elseRegion.back().empty()) {
+    elseBlock = &elseRegion.front();
+    Operation *elseTerminator = elseRegion.back().getTerminator();
+    ValueRange elseTerminatorOperands = elseTerminator->getOperands();
+    rewriter.setInsertionPointToEnd(&elseRegion.back());
+    if (isa<scf::YieldOp>(elseTerminator)) {
+      cf::BranchOp::create(rewriter, loc, continueBlock,
+                           elseTerminatorOperands);
+      rewriter.eraseOp(elseTerminator);
+    }
+    rewriter.inlineRegionBefore(elseRegion, continueBlock);
+  }
+
+  rewriter.setInsertionPointToEnd(condBlock);
+  cf::CondBranchOp::create(rewriter, loc, ifOp.getCondition(), thenBlock,
+                           /*trueArgs=*/ArrayRef<Value>(), elseBlock,
+                           /*falseArgs=*/ArrayRef<Value>());
+
+  rewriter.replaceOp(ifOp, continueBlock->getArguments());
+  return success();
+}
+
 static void propagateLoopAttrs(Operation *scfOp, Operation *brOp) {
   // Let the CondBranchOp carry the LLVM attributes from the ForOp, such as the
   // llvm.loop_annotation attribute.
@@ -400,58 +473,7 @@ LogicalResult ForLowering::matchAndRewrite(ForOp forOp,
 
 LogicalResult IfLowering::matchAndRewrite(IfOp ifOp,
                                           PatternRewriter &rewriter) const {
-  auto loc = ifOp.getLoc();
-
-  // Start by splitting the block containing the 'scf.if' into two parts.
-  // The part before will contain the condition, the part after will be the
-  // continuation point.
-  auto *condBlock = rewriter.getInsertionBlock();
-  auto opPosition = rewriter.getInsertionPoint();
-  auto *remainingOpsBlock = rewriter.splitBlock(condBlock, opPosition);
-  Block *continueBlock;
-  if (ifOp.getNumResults() == 0) {
-    continueBlock = remainingOpsBlock;
-  } else {
-    continueBlock =
-        rewriter.createBlock(remainingOpsBlock, ifOp.getResultTypes(),
-                             SmallVector<Location>(ifOp.getNumResults(), loc));
-    cf::BranchOp::create(rewriter, loc, remainingOpsBlock);
-  }
-
-  // Move blocks from the "then" region to the region containing 'scf.if',
-  // place it before the continuation block, and branch to it.
-  auto &thenRegion = ifOp.getThenRegion();
-  auto *thenBlock = &thenRegion.front();
-  Operation *thenTerminator = thenRegion.back().getTerminator();
-  ValueRange thenTerminatorOperands = thenTerminator->getOperands();
-  rewriter.setInsertionPointToEnd(&thenRegion.back());
-  cf::BranchOp::create(rewriter, loc, continueBlock, thenTerminatorOperands);
-  rewriter.eraseOp(thenTerminator);
-  rewriter.inlineRegionBefore(thenRegion, continueBlock);
-
-  // Move blocks from the "else" region (if present) to the region containing
-  // 'scf.if', place it before the continuation block and branch to it.  It
-  // will be placed after the "then" regions.
-  auto *elseBlock = continueBlock;
-  auto &elseRegion = ifOp.getElseRegion();
-  if (!elseRegion.empty()) {
-    elseBlock = &elseRegion.front();
-    Operation *elseTerminator = elseRegion.back().getTerminator();
-    ValueRange elseTerminatorOperands = elseTerminator->getOperands();
-    rewriter.setInsertionPointToEnd(&elseRegion.back());
-    cf::BranchOp::create(rewriter, loc, continueBlock, elseTerminatorOperands);
-    rewriter.eraseOp(elseTerminator);
-    rewriter.inlineRegionBefore(elseRegion, continueBlock);
-  }
-
-  rewriter.setInsertionPointToEnd(condBlock);
-  cf::CondBranchOp::create(rewriter, loc, ifOp.getCondition(), thenBlock,
-                           /*trueArgs=*/ArrayRef<Value>(), elseBlock,
-                           /*falseArgs=*/ArrayRef<Value>());
-
-  // Ok, we're done!
-  rewriter.replaceOp(ifOp, continueBlock->getArguments());
-  return success();
+  return lowerIfOpToCFG(ifOp, rewriter);
 }
 
 LogicalResult
@@ -719,10 +741,114 @@ LogicalResult ForallLowering::matchAndRewrite(ForallOp forallOp,
   return scf::forallToParallelLoop(rewriter, forallOp);
 }
 
+LogicalResult
+LoopOpLowering::matchAndRewrite(LoopOp loopOp, OpAdaptor adaptor,
+                                ConversionPatternRewriter &rewriter) const {
+  {
+    OpBuilder::InsertionGuard guard(rewriter);
+    if (failed(rewriter.legalize(&loopOp.getRegion())))
+      return rewriter.notifyMatchFailure(loopOp,
+                                         "failed to convert nested region");
+  }
+
+  // Handle degenerate case before modifying any IR.
+  Region &bodyRegion = loopOp.getRegion();
+  if (bodyRegion.empty() || bodyRegion.front().empty()) {
+    rewriter.eraseOp(loopOp);
+    return success();
+  }
+
+  NestedBreakingControlFlowInfo nestedBreakingControlFlow =
+      getNestedBreakingControlFlowInfo(loopOp);
+  if (nestedBreakingControlFlow.hasNestedPredecessors)
+    return rewriter.notifyMatchFailure(loopOp,
+                                       "loop op with nested predecessors");
+
+  // Collect direct predecessors (break/continue targeting this loop).
+  SmallVector<Operation *> predecessors;
+  llvm::append_range(predecessors, nestedBreakingControlFlow.predecessors);
+
+  // Lower `scf.loop` to CFG by converting breaks/continues to branches.
+  Location loc = loopOp.getLoc();
+  BlockArgument controlToken = loopOp.getControlToken();
+  // Split the block containing loopOp into the init block and continuation.
+  Block *initBlock = rewriter.getInsertionBlock();
+  auto initPos = rewriter.getInsertionPoint();
+  Block *continueBlock = rewriter.splitBlock(initBlock, initPos);
+  continueBlock->addArguments(
+      loopOp.getResultTypes(),
+      SmallVector<Location>(loopOp.getNumResults(), loc));
+  Block *loopBody = &bodyRegion.front();
+
+  // After the split above, `initBlock` holds the ops that preceded the loop and
+  // `continueBlock` holds the ops that followed it. Position the builder at the
+  // end of `initBlock`; the branch into the loop body is emitted there below.
+  rewriter.setInsertionPoint(initBlock, initBlock->end());
+
+  // Collect the blocks from the loop body region before inlining so we can
+  // restrict the scf.yield scan to only those blocks (not the whole function).
+  SmallVector<Block *> inlinedBlocks;
+  for (Block &b : bodyRegion)
+    inlinedBlocks.push_back(&b);
+
+  // Move all blocks from the scf.loop region before continueBlock.
+  rewriter.inlineRegionBefore(bodyRegion, continueBlock);
+  // We will remember all break/continue ops to fix up after.
+  SmallVector<Operation *> toErase;
+
+  for (auto predecessor : predecessors) {
+    if (auto breakOp = dyn_cast<scf::BreakOp>(predecessor)) {
+      rewriter.setInsertionPointAfter(breakOp);
+      cf::BranchOp::create(rewriter, breakOp->getLoc(), continueBlock,
+                           ValueRange{breakOp.getArgs()});
+    } else if (auto contOp = dyn_cast<scf::ContinueOp>(predecessor)) {
+      rewriter.setInsertionPointAfter(contOp);
+      cf::BranchOp::create(rewriter, contOp->getLoc(), loopBody,
+                           ValueRange{contOp.getArgs()});
+    }
+    toErase.push_back(predecessor);
+  }
+
+  // Erase the old scf.break/scf.continue ops. Drop their operands first so the
+  // control token (operand #0) loses its last uses here; this lets the
+  // `controlToken.use_empty()` check below confirm no stray consumer remains
+  // before we erase the token block argument.
+  for (Operation *op : toErase) {
+    op->setOperands({});
+    rewriter.eraseOp(op);
+  }
+
+  if (!controlToken.use_empty())
+    return rewriter.notifyMatchFailure(loopOp, "loop token still has uses");
+  loopBody->eraseArgument(0);
+
+  // The loop region is now a CFG. Jump from initBlock to the loop body.
+  rewriter.setInsertionPointToEnd(initBlock);
+  cf::BranchOp::create(rewriter, loc, loopBody,
+                       ValueRange{loopOp.getOperands()});
+
+  // Replace any remaining scf.yield terminators in the inlined loop blocks
+  // with branches back to the loop header (treat as implicit continue).
+  for (Block *block : inlinedBlocks) {
+    if (auto yield = dyn_cast<scf::YieldOp>(block->getTerminator())) {
+      rewriter.setInsertionPoint(yield);
+      cf::BranchOp::create(rewriter, yield.getLoc(), loopBody,
+                           yield.getOperands());
+      rewriter.eraseOp(yield);
+    }
+  }
+
+  // Replace the original scf.loop op with a branch to continueBlock assigning
+  // results.
+  rewriter.replaceOp(loopOp, continueBlock->getArguments());
+  return success();
+}
+
 void mlir::populateSCFToControlFlowConversionPatterns(
     RewritePatternSet &patterns) {
-  patterns.add<ForallLowering, ForLowering, IfLowering, ParallelLowering,
-               WhileLowering, ExecuteRegionLowering, IndexSwitchLowering>(
+  patterns.add<IfLowering>(patterns.getContext(), /*benefit=*/2);
+  patterns.add<ForallLowering, ForLowering, ParallelLowering, WhileLowering,
+               ExecuteRegionLowering, IndexSwitchLowering, LoopOpLowering>(
       patterns.getContext());
   patterns.add<DoWhileLowering>(patterns.getContext(), /*benefit=*/2);
 }
@@ -734,7 +860,8 @@ void SCFToControlFlowPass::runOnOperation() {
   // Configure conversion to lower out SCF operations.
   ConversionTarget target(getContext());
   target.addIllegalOp<scf::ForallOp, scf::ForOp, scf::IfOp, scf::IndexSwitchOp,
-                      scf::ParallelOp, scf::WhileOp, scf::ExecuteRegionOp>();
+                      scf::LoopOp, scf::ParallelOp, scf::WhileOp,
+                      scf::ExecuteRegionOp>();
   target.markUnknownOpDynamicallyLegal([](Operation *) { return true; });
   ConversionConfig config;
   config.allowPatternRollback = allowPatternRollback;
diff --git a/mlir/lib/Dialect/SCF/IR/SCF.cpp b/mlir/lib/Dialect/SCF/IR/SCF.cpp
index 54d28783ddc4a..aa09d59c94acc 100644
--- a/mlir/lib/Dialect/SCF/IR/SCF.cpp
+++ b/mlir/lib/Dialect/SCF/IR/SCF.cpp
@@ -132,6 +132,17 @@ std::optional<llvm::APSInt> mlir::scf::computeUbMinusLb(Value lb, Value ub,
 // ExecuteRegionOp
 //===----------------------------------------------------------------------===//
 
+/// Erase all operations after `afterOp` in the same block (not including
+/// afterOp itself). Walks backwards to avoid use-after-free.
+static void eraseOpsAfter(PatternRewriter &rewriter, Operation *afterOp) {
+  Operation *cur = &afterOp->getBlock()->back();
+  while (cur != afterOp) {
+    Operation *prev = cur->getPrevNode();
+    rewriter.eraseOp(cur);
+    cur = prev;
+  }
+}
+
 ///
 /// (ssa-id `=`)? `execute_region` `->` function-result-type `{`
 ///    block+
@@ -282,6 +293,10 @@ ValueRange ExecuteRegionOp::getSuccessorInputs(RegionSuccessor successor) {
                                  : ValueRange();
 }
 
+SmallVector<Operation *> YieldOp::getPotentialTargets() {
+  return {getOperation()->getParentOp()};
+}
+
 //===----------------------------------------------------------------------===//
 // ConditionOp
 //===----------------------------------------------------------------------===//
@@ -311,6 +326,316 @@ void ConditionOp::getSuccessorRegions(
     regions.push_back(RegionSuccessor(whileOp.getOperation()));
 }
 
+//===----------------------------------------------------------------------===//
+// LoopOp
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// Control Flow Op Utilities
+//===----------------------------------------------------------------------===//
+
+template <typename OpT>
+static ParseResult
+parseControlFlowRegion(OpAsmParser &p, Region &region,
+                       ArrayRef<OpAsmParser::Argument> arguments = {}) {
+  if (failed(p.parseRegion(region, arguments)))
+    return failure();
+  OpT::ensureTerminator(region, p.getBuilder(),
+                        p.getEncodedSourceLoc(p.getNameLoc()));
+  return success();
+}
+
+static LoopOp getLoopTargetFromToken(Value target) {
+  auto targetArg = cast<BlockArgument>(target);
+  auto loopOp = cast<LoopOp>(targetArg.getOwner()->getParentOp());
+  assert(targetArg == loopOp.getControlToken() &&
+         "expected target token to be the loop control token");
+  return loopOp;
+}
+
+static LogicalResult verifyLoopTerminatorTarget(Operation *terminator,
+                                                Value target) {
+  auto targetArg = dyn_cast<BlockArgument>(target);
+  if (!targetArg)
+    return terminator->emitOpError()
+           << "target token must be an entry block argument of an scf.loop";
+
+  Block *targetBlock = targetArg.getOwner();
+  auto loopOp = dyn_cast_or_null<LoopOp>(targetBlock->getParentOp());
+  if (!loopOp || targetArg != loopOp.getControlToken())
+    return terminator->emitOpError()
+           << "target token must be the control token of an scf.loop";
+
+  Operation *currentOp = terminator->getParentOp();
+  while (currentOp && currentOp != loopOp.getOperation()) {
+    if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      return terminator->emitOpError()
+             << "target token crosses an op that does not have the "
+                "PropagateControlFlowBreak trait: "
+             << OpWithFlags(currentOp, OpPrintingFlags().skipRegions());
+    currentOp = currentOp->getParentOp();
+  }
+
+  if (!currentOp)
+    return terminator->emitOpError()
+           << "target token must be defined by an enclosing scf.loop";
+
+  if (!loopOp.acceptsTerminator(terminator))
+    return loopOp.emitOpError("does not accept terminator: ")
+           << OpWithFlags(terminator, OpPrintingFlags().skipRegions());
+
+  return success();
+}
+
+static bool terminatorPropagatesThrough(Operation *terminator, Operation *op) {
+  for (HasBreakingControlFlowOpInterface target :
+       findPotentialBreakTargets(terminator)) {
+    if (target.getOperation() != op &&
+        target.getOperation()->isProperAncestor(op))
+      return true;
+  }
+  return false;
+}
+
+static void appendPropagatedTerminatorSuccessors(
+    RegionBranchTerminatorOpInterface terminator, Operation *op,
+    SmallVectorImpl<RegionSuccessor> &regions) {
+  for (HasBreakingControlFlowOpInterface target :
+       findPotentialBreakTargets(terminator)) {
+    Operation *targetOp = target.getOperation();
+    if (targetOp == op || !targetOp->isProperAncestor(op))
+      continue;
+    auto targetBranch = dyn_cast<RegionBranchOpInterface>(targetOp);
+    assert(targetBranch && "expected breaking target to model region branch");
+    targetBranch.getSuccessorRegions(RegionBranchPoint(terminator), regions);
+  }
+}
+
+LogicalResult BreakOp::verify() {
+  return verifyLoopTerminatorTarget(getOperation(), getTargetToken());
+}
+
+SmallVector<Operation *> BreakOp::getPotentialTargets() {
+  LoopOp loopOp = getLoopTargetFromToken(getTargetToken());
+  return {loopOp.getOperation()};
+}
+
+MutableOperandRange
+BreakOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult ContinueOp::verify() {
+  return verifyLoopTerminatorTarget(getOperation(), getTargetToken());
+}
+
+SmallVector<Operation *> ContinueOp::getPotentialTargets() {
+  LoopOp loopOp = getLoopTargetFromToken(getTargetToken());
+  return {loopOp.getOperation()};
+}
+
+MutableOperandRange
+ContinueOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult LoopOp::verifyRegions() {
+  // Check matching between the operands and the region arguments.
+  if (getRegion().empty())
+    return emitOpError("region cannot be empty");
+  if (getRegion().front().getNumArguments() != getNumOperands() + 1)
+    return emitOpError("expected the region to have one argument per "
+                       "loop-carried value plus the leading control token (")
+           << getNumOperands() + 1 << " expected, but got "
+           << getRegion().front().getNumArguments() << ")";
+  if (!isa<TokenType>(getControlToken().getType()))
+    return emitOpError("first region argument must be a control token");
+  for (auto [index, argAndOperand] :
+       llvm::enumerate(llvm::zip(getRegionIterValues(), getOperands()))) {
+    auto argType = std::get<0>(argAndOperand).getType();
+    auto operandType = std::get<1>(argAndOperand).getType();
+    if (argType != operandType)
+      return emitOpError() << "types mismatch between " << index
+                           << "th iter operand (" << operandType
+                           << ") and defined region argument (" << argType
+                           << ")";
+  }
+  return success();
+}
+
+void LoopOp::print(OpAsmPrinter &p) {
+  p << " token(" << getControlToken() << ") ";
+  bool hasIters = !getInitValues().empty();
+  bool hasReturn = !getResultTypes().empty();
+
+  if (hasIters) {
+    p << "iter_args(";
+    llvm::interleaveComma(
+        llvm::zip(getRegionIterValues(), getInitValues()), p,
+        [&](auto it) { p << std::get<0>(it) << " = " << std::get<1>(it); });
+    p << ") : ";
+    p << getInitValues().getTypes();
+    p << " ";
+  }
+  if (hasReturn) {
+    p << "-> ";
+    p << getResultTypes();
+    p << " ";
+  }
+
+  Operation *terminator = getRegion().front().getTerminator();
+  // Elide the terminator only when it is the trivial implicit terminator:
+  // a `scf.continue` that targets this loop's control token and carries no
+  // iter values. A single-operand `scf.continue` targeting an *outer* loop
+  // must be printed explicitly, otherwise the implicit terminator rebuilt on
+  // parse would silently retarget it to this loop.
+  auto continueOp = dyn_cast<ContinueOp>(terminator);
+  bool printBlockTerminators = !continueOp || !continueOp.getArgs().empty() ||
+                               continueOp.getTargetToken() != getControlToken();
+  p.printRegion(getRegion(), /*printEntryBlockArgs=*/false,
+                printBlockTerminators);
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+ParseResult LoopOp::parse(OpAsmParser &parser, OperationState &result) {
+  SmallVector<OpAsmParser::Argument, 4> regionArgs;
+  SmallVector<OpAsmParser::Argument, 4> iterRegionArgs;
+  SmallVector<OpAsmParser::UnresolvedOperand, 4> iterOperands;
+  SmallVector<Type, 4> iterTypes;
+
+  OpAsmParser::Argument controlToken;
+  if (parser.parseKeyword("token") || parser.parseLParen() ||
+      parser.parseArgument(controlToken) || parser.parseRParen())
+    return failure();
+  controlToken.type = parser.getBuilder().getType<TokenType>();
+  regionArgs.push_back(controlToken);
+
+  if (failed(parser.parseOptionalKeyword("iter_args"))) {
+    // no iter_args, but can still have a return type
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+  } else {
+    // iter_args are present and must have colon followed by types
+    if (parser.parseAssignmentList(iterRegionArgs, iterOperands) ||
+        parser.parseColon() || parser.parseTypeList(iterTypes))
+      return failure();
+    if (iterRegionArgs.size() != iterTypes.size())
+      return parser.emitError(parser.getCurrentLocation(),
+                              "found different number of iter_args and types");
+    // check for optional result type(s)
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+    // Set region argument types for loop body
+    for (auto [regionArg, type] : llvm::zip_equal(iterRegionArgs, iterTypes)) {
+      regionArg.type = type;
+    }
+    llvm::append_range(regionArgs, iterRegionArgs);
+  }
+
+  // Parse region and attr dict.
+  if (parseControlFlowRegion<LoopOp>(parser, *result.addRegion(), regionArgs) ||
+      parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  // Resolve operands.
+  if (parser.resolveOperands(iterOperands, iterTypes, parser.getNameLoc(),
+                             result.operands))
+    return failure();
+
+  return success();
+}
+
+void LoopOp::getSuccessorRegions(RegionBranchPoint point,
+                                 SmallVectorImpl<RegionSuccessor> &regions) {
+  if (point.isParent()) {
+    regions.push_back(RegionSuccessor(&getRegion()));
+    return;
+  }
+
+  // Otherwise, it depends on the terminator: a continue branches back to the
+  // body and a break to the parent.
+  RegionBranchTerminatorOpInterface terminator =
+      point.getTerminatorPredecessorOrNull();
+  if (terminator && terminatorPropagatesThrough(terminator, getOperation())) {
+    appendPropagatedTerminatorSuccessors(terminator, getOperation(), regions);
+    return;
+  }
+
+  if (isa<ContinueOp>(terminator)) {
+    regions.push_back(RegionSuccessor(&getRegion()));
+    return;
+  }
+  assert(isa<BreakOp>(terminator) && "expected continue or break terminator");
+
+  regions.push_back(RegionSuccessor(getOperation()));
+}
+
+OperandRange LoopOp::getEntrySuccessorOperands(RegionSuccessor successor) {
+  return getInitValues();
+}
+
+ValueRange LoopOp::getSuccessorInputs(RegionSuccessor successor) {
+  return successor.isOperation() ? ValueRange(getResults())
+                                 : ValueRange(getRegionIterValues());
+}
+
+namespace {
+
+/// Rewriting pattern that erases loops that have a single iteration.
+struct SimplifyTrivialLoops : public OpRewritePattern<LoopOp> {
+  using OpRewritePattern<LoopOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(LoopOp op,
+                                PatternRewriter &rewriter) const override {
+    // Terminator must be a break.
+    auto breakOp = dyn_cast<BreakOp>(op.getBody()->getTerminator());
+    if (!breakOp)
+      return rewriter.notifyMatchFailure(op, "loop terminator isn't a break");
+    SmallVector<HasBreakingControlFlowOpInterface> targets =
+        findPotentialBreakTargets(breakOp);
+    if (targets.size() != 1 ||
+        targets.front().getOperation() != op.getOperation())
+      return rewriter.notifyMatchFailure(
+          op, "loop terminator targets another loop");
+
+    // If it has nested predecessors, it can't be trivially simplified.
+    if (hasNestedPredecessors(op))
+      return rewriter.notifyMatchFailure(op, "has nested predecessors");
+
+    // Great: it is a single iteration loop, we can simplify it.
+    Block *body = op.getBody();
+    SmallVector<Value> replacements;
+    for (Value value : breakOp.getArgs()) {
+      if (auto blockArg = dyn_cast<BlockArgument>(value);
+          blockArg && blockArg.getOwner() == body) {
+        if (blockArg.getArgNumber() == 0)
+          return rewriter.notifyMatchFailure(
+              op, "loop terminator cannot yield the control token");
+        replacements.push_back(op.getInitValues()[blockArg.getArgNumber() - 1]);
+        continue;
+      }
+      replacements.push_back(value);
+    }
+    rewriter.eraseOp(breakOp);
+    assert(op.getControlToken().use_empty() && "expected token to be unused");
+    body->eraseArgument(0);
+    rewriter.inlineBlockBefore(body, op, op.getInitValues());
+    rewriter.replaceOp(op, replacements);
+
+    return success();
+  }
+};
+} // namespace
+
+void LoopOp::getCanonicalizationPatterns(RewritePatternSet &results,
+                                         MLIRContext *context) {
+  results.add<SimplifyTrivialLoops>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // ForOp
 //===----------------------------------------------------------------------===//
@@ -1940,13 +2265,21 @@ IfOp::inferReturnTypes(MLIRContext *ctx, std::optional<Location> loc,
   Region *r = &adaptor.getThenRegion();
   if (r->empty())
     return failure();
-  Block &b = r->front();
-  if (b.empty())
+  Block *b = &r->front();
+  if (b->empty())
     return failure();
-  auto yieldOp = llvm::dyn_cast<YieldOp>(b.back());
-  if (!yieldOp)
-    return failure();
-  TypeRange types = yieldOp.getOperandTypes();
+  Operation *terminator = &b->back();
+  if (terminatorPropagatesThrough(terminator, terminator->getParentOp())) {
+    if (adaptor.getElseRegion().empty())
+      return success();
+    b = &adaptor.getElseRegion().front();
+    if (b->empty())
+      return success();
+    terminator = &b->back();
+    if (terminatorPropagatesThrough(terminator, terminator->getParentOp()))
+      return success();
+  }
+  TypeRange types = terminator->getOperandTypes();
   llvm::append_range(inferredReturnTypes, types);
   return success();
 }
@@ -2071,7 +2404,9 @@ ParseResult IfOp::parse(OpAsmParser &parser, OperationState &result) {
 }
 
 void IfOp::print(OpAsmPrinter &p) {
-  bool printBlockTerminators = false;
+  bool printBlockTerminators =
+      !isa<YieldOp>(thenBlock()->back()) ||
+      (elseBlock() && !isa<YieldOp>(elseBlock()->back()));
 
   p << " " << getCondition();
   if (!getResults().empty()) {
@@ -2101,6 +2436,16 @@ void IfOp::getSuccessorRegions(RegionBranchPoint point,
   // The `then` and the `else` region branch back to the parent operation or one
   // of the recursive parent operations (early exit case).
   if (!point.isParent()) {
+    // Propagating breaks/continues pass through this if-op to reach an
+    // enclosing loop. Don't report parent() as a successor for them; they
+    // don't yield values to this if-op.
+    if (auto terminator = point.getTerminatorPredecessorOrNull()) {
+      if (terminatorPropagatesThrough(terminator, getOperation())) {
+        appendPropagatedTerminatorSuccessors(terminator, getOperation(),
+                                             regions);
+        return;
+      }
+    }
     regions.push_back(RegionSuccessor(getOperation()));
     return;
   }
@@ -2185,9 +2530,14 @@ struct ConvertTrivialIfToSelect : public OpRewritePattern<IfOp> {
     if (op->getNumResults() == 0)
       return failure();
 
+    YieldOp thenYield = dyn_cast<YieldOp>(op.thenTerminator());
+    YieldOp elseYield = dyn_cast<YieldOp>(op.elseTerminator());
+    if (!thenYield || !elseYield)
+      return failure();
+
     auto cond = op.getCondition();
-    auto thenYieldArgs = op.thenYield().getOperands();
-    auto elseYieldArgs = op.elseYield().getOperands();
+    auto thenYieldArgs = thenYield.getOperands();
+    auto elseYieldArgs = elseYield.getOperands();
 
     SmallVector<Type> nonHoistable;
     for (auto [trueVal, falseVal] : llvm::zip(thenYieldArgs, elseYieldArgs)) {
@@ -2231,10 +2581,12 @@ struct ConvertTrivialIfToSelect : public OpRewritePattern<IfOp> {
     }
 
     rewriter.setInsertionPointToEnd(replacement.thenBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(replacement.thenYield(), trueYields);
+    rewriter.replaceOpWithNewOp<YieldOp>(replacement.thenTerminator(),
+                                         trueYields);
 
     rewriter.setInsertionPointToEnd(replacement.elseBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(replacement.elseYield(), falseYields);
+    rewriter.replaceOpWithNewOp<YieldOp>(replacement.elseTerminator(),
+                                         falseYields);
 
     rewriter.replaceOp(op, results);
     return success();
@@ -2386,36 +2738,35 @@ struct ReplaceIfYieldWithConditionOrValue : public OpRewritePattern<IfOp> {
     if (op.getNumResults() == 0)
       return failure();
 
-    auto trueYield =
-        cast<scf::YieldOp>(op.getThenRegion().back().getTerminator());
-    auto falseYield =
-        cast<scf::YieldOp>(op.getElseRegion().back().getTerminator());
+    YieldOp thenYield = dyn_cast<YieldOp>(op.thenTerminator());
+    YieldOp elseYield = dyn_cast<YieldOp>(op.elseTerminator());
+    if (!thenYield || !elseYield)
+      return failure();
 
     rewriter.setInsertionPoint(op->getBlock(),
                                op.getOperation()->getIterator());
     bool changed = false;
     Type i1Ty = rewriter.getI1Type();
-    for (auto [trueResult, falseResult, opResult] :
-         llvm::zip(trueYield.getResults(), falseYield.getResults(),
-                   op.getResults())) {
-      if (trueResult == falseResult) {
+    for (auto [thenResult, elseResult, opResult] : llvm::zip(
+             thenYield.getResults(), elseYield.getResults(), op.getResults())) {
+      if (thenResult == elseResult) {
         if (!opResult.use_empty()) {
-          opResult.replaceAllUsesWith(trueResult);
+          opResult.replaceAllUsesWith(thenResult);
           changed = true;
         }
         continue;
       }
 
-      BoolAttr trueYield, falseYield;
-      if (!matchPattern(trueResult, m_Constant(&trueYield)) ||
-          !matchPattern(falseResult, m_Constant(&falseYield)))
+      BoolAttr thenYield, elseYield;
+      if (!matchPattern(thenResult, m_Constant(&thenYield)) ||
+          !matchPattern(elseResult, m_Constant(&elseYield)))
         continue;
 
-      bool trueVal = trueYield.getValue();
-      bool falseVal = falseYield.getValue();
-      if (!trueVal && falseVal) {
+      bool thenVal = thenYield.getValue();
+      bool elseVal = elseYield.getValue();
+      if (!thenVal && elseVal) {
         if (!opResult.use_empty()) {
-          Dialect *constDialect = trueResult.getDefiningOp()->getDialect();
+          Dialect *constDialect = thenResult.getDefiningOp()->getDialect();
           Value notCond = arith::XOrIOp::create(
               rewriter, op.getLoc(), op.getCondition(),
               constDialect
@@ -2427,7 +2778,7 @@ struct ReplaceIfYieldWithConditionOrValue : public OpRewritePattern<IfOp> {
           changed = true;
         }
       }
-      if (trueVal && !falseVal) {
+      if (thenVal && !elseVal) {
         if (!opResult.use_empty()) {
           opResult.replaceAllUsesWith(op.getCondition());
           changed = true;
@@ -2482,18 +2833,16 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
       nextThen = nextIf.thenBlock();
       if (!nextIf.getElseRegion().empty())
         nextElse = nextIf.elseBlock();
-    }
-    if (arith::XOrIOp notv =
-            nextIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
+    } else if (arith::XOrIOp notv =
+                   nextIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
       if (notv.getLhs() == prevIf.getCondition() &&
           matchPattern(notv.getRhs(), m_One())) {
         nextElse = nextIf.thenBlock();
         if (!nextIf.getElseRegion().empty())
           nextThen = nextIf.elseBlock();
       }
-    }
-    if (arith::XOrIOp notv =
-            prevIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
+    } else if (arith::XOrIOp notv =
+                   prevIf.getCondition().getDefiningOp<arith::XOrIOp>()) {
       if (notv.getLhs() == nextIf.getCondition() &&
           matchPattern(notv.getRhs(), m_One())) {
         nextElse = nextIf.thenBlock();
@@ -2504,14 +2853,25 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
 
     if (!nextThen && !nextElse)
       return failure();
+    // Check that the terminators are all YieldOp
+    if (!isa<YieldOp>(prevIf.thenTerminator()) ||
+        (nextThen && !isa<YieldOp>(nextThen->getTerminator())))
+      return failure();
+    if (!prevIf.getElseRegion().empty() &&
+        !isa<YieldOp>(prevIf.elseTerminator()))
+      return failure();
+    if (nextElse && !nextElse->empty() &&
+        !isa<YieldOp>(nextElse->getTerminator()))
+      return failure();
 
     SmallVector<Value> prevElseYielded;
     if (!prevIf.getElseRegion().empty())
-      prevElseYielded = prevIf.elseYield().getOperands();
+      prevElseYielded = prevIf.elseTerminator()->getOperands();
     // Replace all uses of return values of op within nextIf with the
     // corresponding yields
-    for (auto it : llvm::zip(prevIf.getResults(),
-                             prevIf.thenYield().getOperands(), prevElseYielded))
+    for (auto it :
+         llvm::zip(prevIf.getResults(), prevIf.thenTerminator()->getOperands(),
+                   prevElseYielded))
       for (OpOperand &use :
            llvm::make_early_inc_range(std::get<0>(it).getUses())) {
         if (nextThen && nextThen->getParent()->isAncestor(
@@ -2539,16 +2899,16 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
                                 combinedIf.getThenRegion().begin());
 
     if (nextThen) {
-      YieldOp thenYield = combinedIf.thenYield();
-      YieldOp thenYield2 = cast<YieldOp>(nextThen->getTerminator());
+      Operation *thenTerminator = combinedIf.thenTerminator();
+      Operation *thenTerminator2 = nextThen->getTerminator();
       rewriter.mergeBlocks(nextThen, combinedIf.thenBlock());
       rewriter.setInsertionPointToEnd(combinedIf.thenBlock());
 
-      SmallVector<Value> mergedYields(thenYield.getOperands());
-      llvm::append_range(mergedYields, thenYield2.getOperands());
-      YieldOp::create(rewriter, thenYield2.getLoc(), mergedYields);
-      rewriter.eraseOp(thenYield);
-      rewriter.eraseOp(thenYield2);
+      SmallVector<Value> mergedYields(thenTerminator->getOperands());
+      llvm::append_range(mergedYields, thenTerminator2->getOperands());
+      YieldOp::create(rewriter, thenTerminator->getLoc(), mergedYields);
+      rewriter.eraseOp(thenTerminator);
+      rewriter.eraseOp(thenTerminator2);
     }
 
     rewriter.inlineRegionBefore(prevIf.getElseRegion(),
@@ -2561,18 +2921,17 @@ struct CombineIfs : public OpRewritePattern<IfOp> {
                                     combinedIf.getElseRegion(),
                                     combinedIf.getElseRegion().begin());
       } else {
-        YieldOp elseYield = combinedIf.elseYield();
-        YieldOp elseYield2 = cast<YieldOp>(nextElse->getTerminator());
+        Operation *elseTerminator = combinedIf.elseTerminator();
+        Operation *elseTerminator2 = nextElse->getTerminator();
         rewriter.mergeBlocks(nextElse, combinedIf.elseBlock());
-
         rewriter.setInsertionPointToEnd(combinedIf.elseBlock());
 
-        SmallVector<Value> mergedElseYields(elseYield.getOperands());
-        llvm::append_range(mergedElseYields, elseYield2.getOperands());
+        SmallVector<Value> mergedElseYields(elseTerminator->getOperands());
+        llvm::append_range(mergedElseYields, elseTerminator2->getOperands());
 
-        YieldOp::create(rewriter, elseYield2.getLoc(), mergedElseYields);
-        rewriter.eraseOp(elseYield);
-        rewriter.eraseOp(elseYield2);
+        YieldOp::create(rewriter, elseTerminator->getLoc(), mergedElseYields);
+        rewriter.eraseOp(elseTerminator);
+        rewriter.eraseOp(elseTerminator2);
       }
     }
 
@@ -2600,7 +2959,8 @@ struct RemoveEmptyElseBranch : public OpRewritePattern<IfOp> {
     if (ifOp.getNumResults())
       return failure();
     Block *elseBlock = ifOp.elseBlock();
-    if (!elseBlock || !llvm::hasSingleElement(*elseBlock))
+    if (!elseBlock || (!llvm::hasSingleElement(*elseBlock) ||
+                       !isa<YieldOp>(elseBlock->getTerminator())))
       return failure();
     auto newIfOp = rewriter.cloneWithoutRegions(ifOp);
     rewriter.inlineRegionBefore(ifOp.getThenRegion(), newIfOp.getThenRegion(),
@@ -2636,21 +2996,32 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     if (!llvm::hasSingleElement(nestedOps))
       return failure();
 
+    auto nestedIf = dyn_cast<IfOp>(*nestedOps.begin());
+    if (!nestedIf)
+      return failure();
+
+    // Terminator must be a YieldOp
+    if (!isa<YieldOp>(op.thenTerminator()))
+      return failure();
+
     // If there is an else block, it can only yield
-    if (op.elseBlock() && !llvm::hasSingleElement(*op.elseBlock()))
+    if (op.elseBlock() && (!llvm::hasSingleElement(*op.elseBlock()) ||
+                           !isa<YieldOp>(op.elseTerminator())))
       return failure();
 
-    auto nestedIf = dyn_cast<IfOp>(*nestedOps.begin());
-    if (!nestedIf)
+    // Same for the nested if: the then and else blocks can only yield.
+    if (!isa<YieldOp>(nestedIf.thenTerminator()))
       return failure();
 
-    if (nestedIf.elseBlock() && !llvm::hasSingleElement(*nestedIf.elseBlock()))
+    if (nestedIf.elseBlock() &&
+        (!llvm::hasSingleElement(*nestedIf.elseBlock()) ||
+         !isa<YieldOp>(nestedIf.elseTerminator())))
       return failure();
 
-    SmallVector<Value> thenYield(op.thenYield().getOperands());
-    SmallVector<Value> elseYield;
+    SmallVector<Value> thenTerminator(op.thenTerminator()->getOperands());
+    SmallVector<Value> elseTerminator;
     if (op.elseBlock())
-      llvm::append_range(elseYield, op.elseYield().getOperands());
+      llvm::append_range(elseTerminator, op.elseTerminator()->getOperands());
 
     // A list of indices for which we should upgrade the value yielded
     // in the else to a select.
@@ -2660,19 +3031,20 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     // only permit combining if the value yielded when the condition
     // is false in the outer scf.if is the same value yielded when the
     // inner scf.if condition is false.
-    // Note that the array access to elseYield will not go out of bounds
-    // since it must have the same length as thenYield, since they both
+    // Note that the array access to elseTerminator will not go out of bounds
+    // since it must have the same length as thenTerminator, since they both
     // come from the same scf.if.
-    for (const auto &tup : llvm::enumerate(thenYield)) {
+    for (const auto &tup : llvm::enumerate(thenTerminator)) {
       if (tup.value().getDefiningOp() == nestedIf) {
         auto nestedIdx = llvm::cast<OpResult>(tup.value()).getResultNumber();
-        if (nestedIf.elseYield().getOperand(nestedIdx) !=
-            elseYield[tup.index()]) {
+        if (nestedIf.elseTerminator()->getOperand(nestedIdx) !=
+            elseTerminator[tup.index()]) {
           return failure();
         }
         // If the correctness test passes, we will yield
         // corresponding value from the inner scf.if
-        thenYield[tup.index()] = nestedIf.thenYield().getOperand(nestedIdx);
+        thenTerminator[tup.index()] =
+            nestedIf.thenTerminator()->getOperand(nestedIdx);
         continue;
       }
 
@@ -2704,28 +3076,72 @@ struct CombineNestedIfs : public OpRewritePattern<IfOp> {
     for (auto idx : elseYieldsToUpgradeToSelect)
       results[idx] =
           arith::SelectOp::create(rewriter, op.getLoc(), op.getCondition(),
-                                  thenYield[idx], elseYield[idx]);
+                                  thenTerminator[idx], elseTerminator[idx]);
 
     rewriter.mergeBlocks(nestedIf.thenBlock(), newIfBlock);
     rewriter.setInsertionPointToEnd(newIf.thenBlock());
-    rewriter.replaceOpWithNewOp<YieldOp>(newIf.thenYield(), thenYield);
-    if (!elseYield.empty()) {
+    rewriter.replaceOpWithNewOp<YieldOp>(newIf.thenTerminator(),
+                                         thenTerminator);
+    if (!elseTerminator.empty()) {
       rewriter.createBlock(&newIf.getElseRegion());
       rewriter.setInsertionPointToEnd(newIf.elseBlock());
-      YieldOp::create(rewriter, loc, elseYield);
+      YieldOp::create(rewriter, loc, elseTerminator);
     }
     rewriter.replaceOp(op, results);
     return success();
   }
 };
 
+/// Simplify if with breaking control flow in both branches.
+/// For example:
+///    scf.if %cmp {
+///       scf.break [%loop] %arg1
+///    } else {
+///       scf.continue [%loop]
+///    }
+///    print(...) // This is dead code
+///  becomes
+///    scf.if %cmp {
+///       scf.break [%loop] %arg1
+///    }
+///    scf.continue [%loop]
+struct SimplifyIfWithBreakingControlFlowInBothBranches
+    : public OpRewritePattern<IfOp> {
+  using OpRewritePattern<IfOp>::OpRewritePattern;
+  LogicalResult matchAndRewrite(IfOp op,
+                                PatternRewriter &rewriter) const override {
+    if (op.getElseRegion().empty() || isa<YieldOp>(op.thenTerminator()) ||
+        isa<YieldOp>(op.elseTerminator()))
+      return failure();
+
+    // Inline the else block after the current op and erase everything after.
+    Block *block = op.elseBlock();
+
+    Operation *terminator = block->getTerminator();
+    // Inline the else block after the current op
+    rewriter.inlineBlockBefore(block, op->getNextNode());
+
+    // Erase everything that comes after the inlined terminator (dead code).
+    eraseOpsAfter(rewriter, terminator);
+
+    // The "else" region is now empty, let's clone the if op and inline the then
+    // region.
+    auto newIfOp = rewriter.cloneWithoutRegions(op);
+    rewriter.inlineRegionBefore(op.getThenRegion(), newIfOp.getThenRegion(),
+                                newIfOp.getThenRegion().begin());
+    rewriter.eraseOp(op);
+
+    return success();
+  }
+};
 } // namespace
 
 void IfOp::getCanonicalizationPatterns(RewritePatternSet &results,
                                        MLIRContext *context) {
   results.add<CombineIfs, CombineNestedIfs, ConditionPropagation,
               ConvertTrivialIfToSelect, RemoveEmptyElseBranch,
-              ReplaceIfYieldWithConditionOrValue>(context);
+              ReplaceIfYieldWithConditionOrValue,
+              SimplifyIfWithBreakingControlFlowInBothBranches>(context);
   populateRegionBranchOpInterfaceCanonicalizationPatterns(
       results, IfOp::getOperationName());
   populateRegionBranchOpInterfaceInliningPattern(results,
@@ -2733,14 +3149,13 @@ void IfOp::getCanonicalizationPatterns(RewritePatternSet &results,
 }
 
 Block *IfOp::thenBlock() { return &getThenRegion().back(); }
-YieldOp IfOp::thenYield() { return cast<YieldOp>(&thenBlock()->back()); }
+
 Block *IfOp::elseBlock() {
   Region &r = getElseRegion();
   if (r.empty())
     return nullptr;
   return &r.back();
 }
-YieldOp IfOp::elseYield() { return cast<YieldOp>(&elseBlock()->back()); }
 
 //===----------------------------------------------------------------------===//
 // ParallelOp
@@ -3184,6 +3599,10 @@ ReduceOp::getMutableSuccessorOperands(RegionSuccessor point) {
   return MutableOperandRange(getOperation(), /*start=*/0, /*length=*/0);
 }
 
+SmallVector<Operation *> ReduceOp::getPotentialTargets() {
+  return {getOperation()->getParentOp()};
+}
+
 //===----------------------------------------------------------------------===//
 // ReduceReturnOp
 //===----------------------------------------------------------------------===//
@@ -3467,8 +3886,8 @@ struct WhileMoveIfDown : public OpRewritePattern<scf::WhileOp> {
       auto it = llvm::find(ifOp->getResults(), arg);
       if (it != ifOp->getResults().end()) {
         size_t ifOpIdx = it.getIndex();
-        Value thenValue = ifOp.thenYield()->getOperand(ifOpIdx);
-        Value elseValue = ifOp.elseYield()->getOperand(ifOpIdx);
+        Value thenValue = ifOp.thenTerminator()->getOperand(ifOpIdx);
+        Value elseValue = ifOp.elseTerminator()->getOperand(ifOpIdx);
 
         rewriter.replaceAllUsesWith(ifOp->getResults()[ifOpIdx], elseValue);
         rewriter.replaceAllUsesWith(op.getAfterArguments()[idx], thenValue);
@@ -3516,7 +3935,7 @@ struct WhileMoveIfDown : public OpRewritePattern<scf::WhileOp> {
         });
 
     // Inline ifOp then region into new whileOp after region.
-    rewriter.eraseOp(ifOp.thenYield());
+    rewriter.eraseOp(ifOp.thenTerminator());
     rewriter.inlineBlockBefore(ifOp.thenBlock(), newWhileOp.getAfterBody(),
                                newWhileOp.getAfterBody()->begin());
     rewriter.eraseOp(ifOp);
diff --git a/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
index 496a7b036e65d..8a0b7f955267e 100644
--- a/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/IR/ValueBoundsOpInterfaceImpl.cpp
@@ -179,8 +179,8 @@ struct IfOpInterface
                              std::optional<int64_t> dim,
                              ValueBoundsConstraintSet &cstr) {
     unsigned int resultNum = cast<OpResult>(value).getResultNumber();
-    Value thenValue = ifOp.thenYield().getResults()[resultNum];
-    Value elseValue = ifOp.elseYield().getResults()[resultNum];
+    Value thenValue = ifOp.thenTerminator()->getOperand(resultNum);
+    Value elseValue = ifOp.elseTerminator()->getOperand(resultNum);
 
     auto boundsBuilder = cstr.bound(value);
     if (dim)
diff --git a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
index f8302126a4a4e..456797b8fbec3 100644
--- a/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/BufferizableOpInterfaceImpl.cpp
@@ -227,8 +227,8 @@ struct IfOpInterface
     auto ifOp = cast<scf::IfOp>(op);
     size_t resultNum = std::distance(op->getOpResults().begin(),
                                      llvm::find(op->getOpResults(), value));
-    OpOperand *thenOperand = &ifOp.thenYield()->getOpOperand(resultNum);
-    OpOperand *elseOperand = &ifOp.elseYield()->getOpOperand(resultNum);
+    OpOperand *thenOperand = &ifOp.thenTerminator()->getOpOperand(resultNum);
+    OpOperand *elseOperand = &ifOp.elseTerminator()->getOpOperand(resultNum);
     return {{thenOperand, BufferRelation::Equivalent, /*isDefinite=*/false},
             {elseOperand, BufferRelation::Equivalent, /*isDefinite=*/false}};
   }
diff --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..306a4cbc2dcef 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -927,11 +927,17 @@ xegpu::precomputeLoopBlockArgTypes(Operation *topLevelOp,
     }
     if (auto ifOp = dyn_cast<scf::IfOp>(op)) {
       // Each result and its then/else yield operands share one position and
-      // must convert identically; derive all from the result's layout.
-      scf::YieldOp thenYield = ifOp.thenYield();
-      scf::YieldOp elseYield = ifOp.elseBlock() ? ifOp.elseYield() : nullptr;
+      // must convert identically; derive all from the result's layout. A
+      // branch that exits early does not yield a value to this `scf.if`, so
+      // only record operands from branches that actually end with scf.yield.
+      scf::YieldOp thenYield = dyn_cast<scf::YieldOp>(ifOp.thenTerminator());
+      scf::YieldOp elseYield;
+      if (ifOp.elseBlock())
+        elseYield = dyn_cast<scf::YieldOp>(ifOp.elseTerminator());
       for (auto [idx, res] : llvm::enumerate(ifOp.getResults())) {
-        SmallVector<Value> dests{res, thenYield.getOperand(idx)};
+        SmallVector<Value> dests{res};
+        if (thenYield)
+          dests.push_back(thenYield.getOperand(idx));
         if (elseYield)
           dests.push_back(elseYield.getOperand(idx));
         recordTypes(res, dests);
diff --git a/mlir/lib/IR/Dominance.cpp b/mlir/lib/IR/Dominance.cpp
index 79fb41f2e6b30..f8820822dc01f 100644
--- a/mlir/lib/IR/Dominance.cpp
+++ b/mlir/lib/IR/Dominance.cpp
@@ -14,8 +14,12 @@
 #include "mlir/IR/Dominance.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/RegionKindInterface.h"
+#include "llvm/ADT/SmallPtrSet.h"
+#include "llvm/Support/DebugLog.h"
 #include "llvm/Support/GenericDomTreeConstruction.h"
 
+#define DEBUG_TYPE "dominance"
+
 using namespace mlir;
 using namespace mlir::detail;
 
@@ -38,6 +42,7 @@ void DominanceInfoBase<IsPostDom>::invalidate() {
   for (auto entry : dominanceInfos)
     delete entry.second.getPointer();
   dominanceInfos.clear();
+  breakingControlFlowOpsCache.clear();
 }
 
 template <bool IsPostDom>
@@ -53,6 +58,7 @@ void DominanceInfoBase<IsPostDom>::invalidate(Region *region, bool recursive) {
     region->walk([&](Region *r) { invalidate(r); });
   else
     invalidate(region);
+  breakingControlFlowOpsCache.clear();
 }
 
 /// Return the dom tree and "hasSSADominance" bit for the given region.  The
@@ -255,6 +261,15 @@ static bool isBeforeInBlock(Block *block, Block::iterator a,
   return a->isBeforeInBlock(&*b);
 }
 
+template <bool IsPostDom>
+bool DominanceInfoBase<IsPostDom>::hasBreakingControlFlowOpsCached(
+    Operation *op) const {
+  auto [it, inserted] = breakingControlFlowOpsCache.try_emplace(op, false);
+  if (inserted)
+    it->second = hasBreakingControlFlowOps(op);
+  return it->second;
+}
+
 template <bool IsPostDom>
 bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
     Block *aBlock, Block::iterator aIt, Block *bBlock, Block::iterator bIt,
@@ -288,6 +303,23 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
       return true;
   }
 
+  auto hasEscapingBreakingControlFlowInRange =
+      [&](Block *block, Block::iterator begin, Block::iterator end) {
+        assert(begin == block->end() || begin->getBlock() == block);
+        assert(end == block->end() || end->getBlock() == block);
+        for (auto it = begin; it != end; ++it) {
+          Operation *op = &*it;
+          if ((op->hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+               !op->isRegistered()) &&
+              hasBreakingControlFlowOpsCached(op)) {
+            LDBG() << "Breaking control flow: "
+                   << OpWithFlags(op, OpPrintingFlags().skipRegions());
+            return true;
+          }
+        }
+        return false;
+      };
+
   // Ok, they are in the same region now.
   if (aBlock == bBlock) {
     // Dominance changes based on the region type. In a region with SSA
@@ -295,6 +327,26 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
     // regions kinds, uses and defs can come in any order inside a block.
     if (!hasSSADominance(aBlock))
       return true;
+
+    if (IsPostDom && aIt != aBlock->end() && bIt != bBlock->end() &&
+        mightHaveBreakingControlFlow(aBlock->getParent())) {
+      bool inRange = false;
+      for (Operation &op : *aBlock) {
+        if (&op == &*bIt)
+          inRange = true;
+        if (inRange) {
+          if ((op.hasTrait<OpTrait::PropagateControlFlowBreak>() ||
+               !op.isRegistered()) &&
+              hasBreakingControlFlowOpsCached(&op)) {
+            LDBG() << "Breaking control flow: "
+                   << OpWithFlags(&op, OpPrintingFlags().skipRegions());
+            return false;
+          }
+        }
+        if (&op == &*aIt)
+          break;
+      }
+    }
     if constexpr (IsPostDom) {
       return isBeforeInBlock(aBlock, bIt, aIt);
     } else {
@@ -303,7 +355,47 @@ bool DominanceInfoBase<IsPostDom>::properlyDominatesImpl(
   }
 
   // If the blocks are different, use DomTree to resolve the query.
-  return getDomTree(aRegion).properlyDominates(aBlock, bBlock);
+  bool properlyDominates =
+      getDomTree(aRegion).properlyDominates(aBlock, bBlock);
+  if (!IsPostDom || !properlyDominates ||
+      !mightHaveBreakingControlFlow(aRegion))
+    return properlyDominates;
+
+  // A nested breaking terminator can bypass the containing block's ordinary
+  // CFG terminator, so the block post-dominator tree is not enough for
+  // operation post-dominance. Look for escaping breaks on paths from bBlock to
+  // aBlock.
+  if (bIt != bBlock->end() &&
+      hasEscapingBreakingControlFlowInRange(bBlock, bIt, bBlock->end()))
+    return false;
+
+  SmallPtrSet<Block *, 16> visited;
+  SmallVector<Block *> worklist;
+  auto enqueueSuccessor = [&](Block *successor) {
+    if (!successor || successor == aBlock || successor->getParent() != aRegion)
+      return;
+    if (visited.insert(successor).second)
+      worklist.push_back(successor);
+  };
+  for (Block *successor : bBlock->getSuccessors())
+    enqueueSuccessor(successor);
+
+  while (!worklist.empty()) {
+    Block *block = worklist.pop_back_val();
+    if (hasEscapingBreakingControlFlowInRange(block, block->begin(),
+                                              block->end()))
+      return false;
+    for (Block *successor : block->getSuccessors())
+      enqueueSuccessor(successor);
+  }
+
+  if (aIt != aBlock->end()) {
+    Block::iterator endIt = std::next(aIt);
+    if (hasEscapingBreakingControlFlowInRange(aBlock, aBlock->begin(), endIt))
+      return false;
+  }
+
+  return true;
 }
 
 /// Return true if the specified block is reachable from the entry block of
diff --git a/mlir/lib/IR/RegionKindInterface.cpp b/mlir/lib/IR/RegionKindInterface.cpp
index 007f4cf92dbc7..59c741ea7e11e 100644
--- a/mlir/lib/IR/RegionKindInterface.cpp
+++ b/mlir/lib/IR/RegionKindInterface.cpp
@@ -12,6 +12,12 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/IR/RegionKindInterface.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Support/WalkResult.h"
+
+#include "llvm/Support/DebugLog.h"
+
+#define DEBUG_TYPE "region-kind-interface"
 
 using namespace mlir;
 
@@ -32,3 +38,165 @@ bool mlir::mayBeGraphRegion(Region &region) {
     return false;
   return !regionKindOp.hasSSADominance(region.getRegionNumber());
 }
+
+namespace {
+// Worklist entry for walking block terminators in nested regions.
+// Tracks the current position within a region and the nesting depth.
+struct NestedOpIterator {
+  NestedOpIterator(Region *region, int nestedLevel)
+      : region(region), nestedLevel(nestedLevel) {
+    regionIt = region->begin();
+    blockIt = regionIt->end();
+    if (regionIt != region->end())
+      blockIt = regionIt->begin();
+  }
+  // Advance the iterator to the next reachable operation.
+  void advance() {
+    assert(regionIt != region->end());
+    if (blockIt == regionIt->end()) {
+      ++regionIt;
+      if (regionIt != region->end())
+        blockIt = regionIt->begin();
+      return;
+    }
+    ++blockIt;
+    if (blockIt != regionIt->end()) {
+      LDBG() << "Advancing to next op: "
+             << OpWithFlags(&*blockIt, OpPrintingFlags().skipRegions());
+    }
+  }
+
+  // The region we're iterating over.
+  Region *region;
+  // The Block currently being iterated over.
+  Region::iterator regionIt;
+  // The Operation currently being iterated over.
+  Block::iterator blockIt;
+  // The nested level of the current region relative to the starting region.
+  int nestedLevel = 0;
+};
+} // namespace
+
+/// Walk all block terminators (last operation in each block) nested within
+/// `rootOp`. The callback receives the terminator and its 1-based nesting
+/// level relative to `rootOp`. Callers filter based on breaking-control-flow
+/// properties as needed.
+static void walk(Operation *rootOp,
+                 function_ref<WalkResult(Operation *, int)> callback) {
+  // Worklist of regions to visit to drive the traversal.
+  SmallVector<NestedOpIterator> worklist;
+
+  // Perform a traversal of the regions, visiting each
+  // reachable operation.
+  for (Region &region : rootOp->getRegions()) {
+    if (region.empty())
+      continue;
+    worklist.push_back({&region, 1});
+  }
+  while (!worklist.empty()) {
+    NestedOpIterator &it = worklist.back();
+    if (it.regionIt == it.region->end()) {
+      // We're done with this region.
+      worklist.pop_back();
+      continue;
+    }
+    if (it.blockIt == it.regionIt->end()) {
+      // We're done with this block.
+      it.advance();
+      continue;
+    }
+    Operation *op = &*it.blockIt;
+
+    // Only call the callback if we're at the end of the block.
+    if (std::next(it.blockIt) == it.regionIt->end() &&
+        callback(op, it.nestedLevel).wasInterrupted())
+      return;
+
+    // Advance before pushing nested regions to avoid reference invalidation.
+    int currentNestedLevel = it.nestedLevel;
+    it.advance();
+
+    // Recursively visit the nested regions.
+    for (Region &nestedRegion : op->getRegions()) {
+      if (nestedRegion.empty())
+        continue;
+      worklist.push_back({&nestedRegion, currentNestedLevel + 1});
+    }
+  }
+}
+
+NestedBreakingControlFlowInfo
+mlir::getNestedBreakingControlFlowInfo(Operation *op) {
+  NestedBreakingControlFlowInfo info;
+  walk(op, [&](Operation *visitedOp, int nestedLevel) {
+    bool directlyTargetsOp = false;
+    bool escapesThroughOp = false;
+    for (HasBreakingControlFlowOpInterface target :
+         findPotentialBreakTargets(visitedOp)) {
+      if (!target)
+        continue;
+      Operation *targetOp = target.getOperation();
+      if (targetOp == op)
+        directlyTargetsOp = true;
+      else if (targetOp->isProperAncestor(op))
+        escapesThroughOp = true;
+    }
+
+    if (directlyTargetsOp) {
+      info.predecessors.push_back(visitedOp);
+      if (nestedLevel > 1)
+        info.hasNestedPredecessors = true;
+    }
+    if (escapesThroughOp)
+      info.hasBreakingControlFlowOps = true;
+
+    return WalkResult::advance();
+  });
+  return info;
+}
+
+bool mlir::hasNestedPredecessors(Operation *op) {
+  return getNestedBreakingControlFlowInfo(op).hasNestedPredecessors;
+}
+
+bool mlir::hasBreakingControlFlowOps(Operation *op) {
+  return getNestedBreakingControlFlowInfo(op).hasBreakingControlFlowOps;
+}
+
+void mlir::detail::visitNestedBreakingControlFlowOpsImpl(
+    Operation *op,
+    function_ref<WalkResult(RegionExitTerminatorOpInterface, int nestedLevel)>
+        callback) {
+  ::walk(op, [&](Operation *visitedOp, int nestedLevel) {
+    for (HasBreakingControlFlowOpInterface target :
+         findPotentialBreakTargets(visitedOp)) {
+      if (target && (target.getOperation() == op ||
+                     target.getOperation()->isProperAncestor(op)))
+        return callback(cast<RegionExitTerminatorOpInterface>(visitedOp),
+                        nestedLevel);
+    }
+    return WalkResult::advance();
+  });
+}
+
+void mlir::collectAllNestedPredecessors(
+    Operation *op, SmallVector<Operation *> &predecessors) {
+  llvm::append_range(predecessors,
+                     getNestedBreakingControlFlowInfo(op).predecessors);
+}
+
+SmallVector<HasBreakingControlFlowOpInterface>
+mlir::findPotentialBreakTargets(Operation *terminator) {
+  auto breakingTerminator =
+      dyn_cast<RegionExitTerminatorOpInterface>(terminator);
+  if (!breakingTerminator)
+    return {};
+  SmallVector<HasBreakingControlFlowOpInterface> targets;
+  for (Operation *target : breakingTerminator.getPotentialTargets()) {
+    if (!target)
+      continue;
+    if (auto breakTarget = dyn_cast<HasBreakingControlFlowOpInterface>(target))
+      targets.push_back(breakTarget);
+  }
+  return targets;
+}
diff --git a/mlir/lib/IR/Verifier.cpp b/mlir/lib/IR/Verifier.cpp
index 11771e78d5f20..4209a36b1c560 100644
--- a/mlir/lib/IR/Verifier.cpp
+++ b/mlir/lib/IR/Verifier.cpp
@@ -30,6 +30,7 @@
 #include "mlir/IR/Dialect.h"
 #include "mlir/IR/Dominance.h"
 #include "mlir/IR/Operation.h"
+#include "mlir/IR/OperationSupport.h"
 #include "mlir/IR/RegionKindInterface.h"
 #include "mlir/IR/Threading.h"
 #include "llvm/ADT/PointerIntPair.h"
diff --git a/mlir/lib/Interfaces/ControlFlowInterfaces.cpp b/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
index 475969a7c0d09..241e447a3e7f1 100644
--- a/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
+++ b/mlir/lib/Interfaces/ControlFlowInterfaces.cpp
@@ -13,6 +13,7 @@
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/Operation.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/RegionKindInterface.h"
 #include "mlir/Interfaces/ControlFlowInterfaces.h"
 #include "llvm/ADT/EquivalenceClasses.h"
 #include "llvm/Support/DebugLog.h"
@@ -76,14 +77,14 @@ detail::getBranchSuccessorArgument(const SuccessorOperands &operands,
 LogicalResult
 detail::verifyBranchSuccessorOperands(Operation *op, unsigned succNo,
                                       const SuccessorOperands &operands) {
-  LDBG() << "Verifying branch successor operands for successor #" << succNo
-         << " in operation " << op->getName();
+  LDBG(3) << "Verifying branch successor operands for successor #" << succNo
+          << " in operation " << op->getName();
 
   // Check the count.
   unsigned operandCount = operands.size();
   Block *destBB = op->getSuccessor(succNo);
-  LDBG() << "Branch has " << operandCount << " operands, target block has "
-         << destBB->getNumArguments() << " arguments";
+  LDBG(3) << "Branch has " << operandCount << " operands, target block has "
+          << destBB->getNumArguments() << " arguments";
 
   if (operandCount != destBB->getNumArguments())
     return op->emitError() << "branch has " << operandCount
@@ -92,22 +93,22 @@ detail::verifyBranchSuccessorOperands(Operation *op, unsigned succNo,
                            << destBB->getNumArguments();
 
   // Check the types.
-  LDBG() << "Checking type compatibility for "
-         << (operandCount - operands.getProducedOperandCount())
-         << " forwarded operands";
+  LDBG(3) << "Checking type compatibility for "
+          << (operandCount - operands.getProducedOperandCount())
+          << " forwarded operands";
   for (unsigned i = operands.getProducedOperandCount(); i != operandCount;
        ++i) {
     Type operandType = operands[i].getType();
     Type argType = destBB->getArgument(i).getType();
-    LDBG() << "Checking type compatibility: operand type " << operandType
-           << " vs argument type " << argType;
+    LDBG(3) << "Checking type compatibility: operand type " << operandType
+            << " vs argument type " << argType;
 
     if (!cast<BranchOpInterface>(op).areTypesCompatible(operandType, argType))
       return op->emitError() << "type mismatch for bb argument #" << i
                              << " of successor #" << succNo;
   }
 
-  LDBG() << "Branch successor operand verification successful";
+  LDBG(3) << "Branch successor operand verification successful";
   return success();
 }
 
@@ -168,11 +169,17 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
     SmallVector<RegionSuccessor> successors;
     regionInterface.getSuccessorRegions(branchPoint, successors);
     for (const RegionSuccessor &successor : successors) {
+      RegionBranchOpInterface successorOwner =
+          getRegionBranchSuccessorOwner(successor);
+      if (!successorOwner)
+        return regionInterface->emitOpError()
+               << "has region successor not owned by a "
+                  "RegionBranchOpInterface";
       // Helper function that print the region branch point and the region
       // successor.
       auto emitRegionEdgeError = [&]() {
         InFlightDiagnostic diag =
-            regionInterface->emitOpError("along control flow edge from ");
+            successorOwner->emitOpError("along control flow edge from ");
         if (branchPoint.isParent()) {
           diag << "parent";
           diag.attachNote(op->getLoc()) << "region branch point";
@@ -195,7 +202,7 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
       // Verify number of successor operands and successor inputs.
       OperandRange succOperands =
           regionInterface.getSuccessorOperands(branchPoint, successor);
-      ValueRange succInputs = regionInterface.getSuccessorInputs(successor);
+      ValueRange succInputs = successorOwner.getSuccessorInputs(successor);
       if (succOperands.size() != succInputs.size()) {
         return emitRegionEdgeError()
                << ": region branch point has " << succOperands.size()
@@ -210,7 +217,7 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
            llvm::enumerate(llvm::zip(succOperandTypes, succInputTypes))) {
         Type succOperandType = std::get<0>(typesIdx.value());
         Type succInputType = std::get<1>(typesIdx.value());
-        if (!regionInterface.areTypesCompatible(succOperandType, succInputType))
+        if (!successorOwner.areTypesCompatible(succOperandType, succInputType))
           return emitRegionEdgeError()
                  << ": successor operand type #" << typesIdx.index() << " "
                  << succOperandType << " should match successor input type #"
@@ -227,6 +234,17 @@ LogicalResult detail::verifyRegionBranchOpInterface(Operation *op) {
 /// regions.
 using StopConditionFn = function_ref<bool(Region *, ArrayRef<bool> visited)>;
 
+/// Given a range of values, return a vector of attributes of the same size,
+/// where the i-th attribute is the constant value of the i-th value. If a
+/// value is not constant, the corresponding attribute is null.
+static SmallVector<Attribute> extractConstants(ValueRange values) {
+  return llvm::map_to_vector(values, [](Value value) {
+    Attribute attr;
+    matchPattern(value, m_Constant(&attr));
+    return attr;
+  });
+}
+
 /// Traverse the region graph starting at `begin`. The traversal is interrupted
 /// if `stopCondition` evaluates to "true" for a successor region. In that case,
 /// this function returns "true". Otherwise, if the traversal was not
@@ -246,7 +264,6 @@ static bool traverseRegionGraph(Region *begin,
   SmallVector<Region *> worklist;
   auto enqueueAllSuccessors = [&](Region *region) {
     LDBG() << "Enqueuing successors for region #" << region->getRegionNumber();
-    SmallVector<Attribute> operandAttributes(op->getNumOperands());
     for (Block &block : *region) {
       if (block.empty())
         continue;
@@ -255,8 +272,8 @@ static bool traverseRegionGraph(Region *begin,
       if (!terminator)
         continue;
       SmallVector<RegionSuccessor> successors;
-      operandAttributes.resize(terminator->getNumOperands());
-      terminator.getSuccessorRegions(operandAttributes, successors);
+      terminator.getSuccessorRegions(
+          extractConstants(terminator->getOperands()), successors);
       LDBG() << "Found " << successors.size()
              << " successors from terminator in block";
       for (RegionSuccessor successor : successors) {
@@ -462,6 +479,14 @@ RegionBranchOpInterface::getNonSuccessorInputs(RegionSuccessor successor) {
   return results;
 }
 
+RegionBranchOpInterface
+mlir::getRegionBranchSuccessorOwner(RegionSuccessor successor) {
+  if (Operation *successorOp = successor.getSuccessorOp())
+    return dyn_cast<RegionBranchOpInterface>(successorOp);
+  return dyn_cast<RegionBranchOpInterface>(
+      successor.getSuccessor()->getParentOp());
+}
+
 static MutableArrayRef<OpOperand> operandsToOpOperands(OperandRange &operands) {
   return MutableArrayRef<OpOperand>(operands.getBase(), operands.size());
 }
@@ -473,11 +498,14 @@ getSuccessorOperandInputMapping(RegionBranchOpInterface branchOp,
   SmallVector<RegionSuccessor> successors;
   branchOp.getSuccessorRegions(src, successors);
   for (RegionSuccessor dst : successors) {
+    RegionBranchOpInterface successorOwner = getRegionBranchSuccessorOwner(dst);
+    assert(successorOwner && "expected RegionBranchOpInterface owner");
     OperandRange operands = branchOp.getSuccessorOperands(src, dst);
-    assert(operands.size() == branchOp.getSuccessorInputs(dst).size() &&
+    ValueRange inputs = successorOwner.getSuccessorInputs(dst);
+    assert(operands.size() == inputs.size() &&
            "expected the same number of operands and inputs");
-    for (const auto &[operand, input] : llvm::zip_equal(
-             operandsToOpOperands(operands), branchOp.getSuccessorInputs(dst)))
+    for (const auto &[operand, input] :
+         llvm::zip_equal(operandsToOpOperands(operands), inputs))
       mapping[&operand].push_back(input);
   }
 }
@@ -524,6 +552,62 @@ RegionBranchOpInterface::getAllRegionBranchPoints() {
         branchPoints.push_back(RegionBranchPoint(terminator));
     }
   }
+
+  Operation *op = getOperation();
+  if (!op->hasTrait<OpTrait::PropagateControlFlowBreak>() &&
+      !isa<HasBreakingControlFlowOpInterface>(op))
+    return branchPoints;
+
+  // The loop above only records terminators that are immediate branch points of
+  // this op: terminators ending blocks directly contained in one of this op's
+  // regions. Breaking control flow adds another class of branch points. A
+  // nested RegionBranchOpInterface that defines PropagateControlFlowBreak can
+  // contain a breaking terminator whose addressed receiver is this op or one of
+  // this op's ancestors. Even though the terminator is not directly contained
+  // in this op's region, it can still create a control-flow edge that leaves or
+  // propagates through this op.
+  //
+  // For example, consider:
+  //
+  //   scf.loop token(%outer) {
+  //     scf.loop token(%inner) {
+  //       scf.if %cond {
+  //         scf.break [%outer]
+  //       }
+  //       scf.continue [%inner]
+  //     }
+  //     scf.continue [%outer]
+  //   }
+  //
+  // When enumerating branch points for the inner loop, its direct body
+  // terminator is only `scf.continue [%inner]`. The `scf.break [%outer]` is
+  // hidden behind the immediately nested `scf.if`, but the inner loop must
+  // still expose a possible edge for that break: the break request propagates
+  // through `scf.if`, then through the inner loop, and is ultimately handled by
+  // the outer loop. Without adding the nested break as a branch point of the
+  // inner loop, generic RegionBranchOpInterface verification and data-flow
+  // consumers only see the continue edge and miss the propagated exit edge.
+  //
+  // This is intentionally broader than collectAllNestedPredecessors(op). That
+  // helper collects only terminators that directly target `op`, which is enough
+  // for a receiver to find incoming breaks but insufficient for an intermediate
+  // PropagateControlFlowBreak op: an escaping terminator targets an ancestor,
+  // not the intermediate op it propagates through.
+  // visitNestedBreakingControlFlowOps reports both nested terminators that
+  // target `op` and those that target an ancestor of `op`, which is exactly the
+  // set that may affect this op's RegionBranchOpInterface edges.
+  visitNestedBreakingControlFlowOps(
+      op, [&](Operation *nestedTerminator, int nestedLevel) {
+        // Immediate region terminators are already covered above. Deeper
+        // terminators may add a propagated control-flow edge through this op.
+        if (nestedLevel <= 1)
+          return;
+        auto terminator =
+            dyn_cast<RegionBranchTerminatorOpInterface>(nestedTerminator);
+        if (!terminator)
+          return;
+        branchPoints.push_back(RegionBranchPoint(terminator));
+      });
   return branchPoints;
 }
 
@@ -628,6 +712,12 @@ static bool isDefinedBefore(Operation *regionBranchOp, Value a, Value b) {
   return true;
 }
 
+static bool isSuccessorInputOwnedBy(Operation *regionBranchOp, Value value) {
+  if (Operation *definingOp = value.getDefiningOp())
+    return definingOp == regionBranchOp;
+  return value.getParentRegion()->getParentOp() == regionBranchOp;
+}
+
 /// Compute all non-successor-input values that a successor input could have
 /// based on the given successor input to successor operand mapping.
 ///
@@ -733,6 +823,8 @@ struct MakeRegionBranchOpSuccessorInputsDead : public RewritePattern {
     // Try to replace the uses of each successor input one-by-one.
     bool changed = false;
     for (Value value : inputToOperands.keys()) {
+      if (!isSuccessorInputOwnedBy(regionBranchOp, value))
+        continue;
       // Nothing to do for successor inputs that are already dead.
       if (value.use_empty())
         continue;
@@ -1070,17 +1162,6 @@ struct RemoveDuplicateSuccessorInputUses : public RewritePattern {
   }
 };
 
-/// Given a range of values, return a vector of attributes of the same size,
-/// where the i-th attribute is the constant value of the i-th value. If a
-/// value is not constant, the corresponding attribute is null.
-static SmallVector<Attribute> extractConstants(ValueRange values) {
-  return llvm::map_to_vector(values, [](Value value) {
-    Attribute attr;
-    matchPattern(value, m_Constant(&attr));
-    return attr;
-  });
-}
-
 /// Return all successor regions when branching from the given region branch
 /// point. This helper functions extracts all constant operand values and
 /// passes them to the `RegionBranchOpInterface`.
diff --git a/mlir/lib/Transforms/Utils/CMakeLists.txt b/mlir/lib/Transforms/Utils/CMakeLists.txt
index 3ecf3859a7f4c..cb84fb26b2408 100644
--- a/mlir/lib/Transforms/Utils/CMakeLists.txt
+++ b/mlir/lib/Transforms/Utils/CMakeLists.txt
@@ -15,6 +15,9 @@ add_mlir_library(MLIRTransformUtils
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Transforms
 
+  DEPENDS
+  MLIRRegionKindInterfaceIncGen
+
   LINK_LIBS PUBLIC
   MLIRAnalysis
   MLIRCallInterfaces
diff --git a/mlir/lib/Transforms/Utils/InliningUtils.cpp b/mlir/lib/Transforms/Utils/InliningUtils.cpp
index 73107cfc36ea9..4bfc20e7ed885 100644
--- a/mlir/lib/Transforms/Utils/InliningUtils.cpp
+++ b/mlir/lib/Transforms/Utils/InliningUtils.cpp
@@ -286,6 +286,12 @@ static LogicalResult inlineRegionImpl(
                    [&](BlockArgument arg) { return !mapper.contains(arg); }))
     return failure();
 
+  // Block inlining only if breaks escape the region (propagate through the
+  // parent op toward an ancestor). Self-contained breaks that target ops
+  // within the region are fine.
+  if (hasBreakingControlFlowOps(src->getParentOp()))
+    return failure();
+
   // Check that the operations within the source region are valid to inline.
   Region *insertRegion = inlineBlock->getParent();
   if (!interface.isLegalToInline(insertRegion, src, shouldCloneInlinedRegion,
diff --git a/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir b/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
new file mode 100644
index 0000000000000..49338ac634462
--- /dev/null
+++ b/mlir/test/Analysis/DataFlow/test-dead-code-analysis-early-exit.mlir
@@ -0,0 +1,100 @@
+// RUN: mlir-opt -test-dead-code-analysis --split-input-file 2>&1 %s | FileCheck %s
+
+// Tests verifying that DeadCodeAnalysis correctly propagates breaking control
+// flow through PropagateControlFlowBreak ops (e.g. scf.if). The fix is in
+// visitRegionTerminator: when the immediate parent RegionBranchOpInterface
+// returns a propagating successor sentinel for a propagating break, we resolve
+// the actual HasBreakingControlFlowOp ancestor and re-dispatch through it, so
+// the correct predecessor edge is established at the loop's exit point.
+
+// -----
+
+// A loop whose only exit is scf.break propagated through scf.if.
+// The loop's exit point (op_preds) must show the break as a predecessor.
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK:   scf.continue
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_break_through_if(%cond: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond {
+      %c42 = arith.constant 42 : i32
+      scf.break [%loop] %c42 : i32
+    }
+    scf.continue [%loop]
+  } {tag = "loop"}
+  return %result : i32
+}
+
+// -----
+
+// For comparison: a loop with a direct scf.break.
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_direct_break(%cond: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond {
+    }
+    %c42 = arith.constant 42 : i32
+    scf.break [%loop] %c42 : i32
+  } {tag = "loop"}
+  return %result : i32
+}
+
+// -----
+
+// A loop whose only exit is scf.break propagated through two nested scf.if
+// ops. The loop's op_preds must show the break. The intermediate if ops do not
+// appear in op_preds for the break — the break bypasses them without yielding
+// values to them, which is correct.
+
+// CHECK-LABEL: inner_if:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "inner_if"}
+// CHECK:  region #1
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "inner_if"}
+
+// CHECK-LABEL: outer_if:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "outer_if"}
+// CHECK:  region #1
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.if {{.*}} {tag = "outer_if"}
+
+// CHECK-LABEL: loop:
+// CHECK:  region #0
+// CHECK:   ^bb0 = live
+// CHECK: region_preds: (all) predecessors:
+// CHECK:   %0 = scf.loop
+// CHECK:   scf.continue
+// CHECK: op_preds: (all) predecessors:
+// CHECK:   scf.break
+func.func @test_break_through_nested_ifs(%cond1: i1, %cond2: i1) -> i32 {
+  %result = scf.loop token(%loop) -> i32 {
+    scf.if %cond1 {
+      scf.if %cond2 {
+        %c99 = arith.constant 99 : i32
+        scf.break [%loop] %c99 : i32
+      } {tag = "inner_if"}
+      scf.continue [%loop]
+    } {tag = "outer_if"}
+    scf.continue [%loop]
+  } {tag = "loop"}
+  return %result : i32
+}
diff --git a/mlir/test/Analysis/test-dominance.mlir b/mlir/test/Analysis/test-dominance.mlir
index a926a8271200a..5627c9994b0e8 100644
--- a/mlir/test/Analysis/test-dominance.mlir
+++ b/mlir/test/Analysis/test-dominance.mlir
@@ -680,3 +680,63 @@ func.func @func_loop_nested_region(
 // CHECK:   ^{{.*}}
 // CHECK:   }
 // CHECK: }
+
+
+// -----
+
+// CHECK-LABEL: Testing : func_loop_early_exit
+func.func @func_loop_early_exit(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      scf.if %cond {
+        scf.break [%outer] %arg0 : index
+      }
+      "test.foo"() : () -> ()
+      scf.break [%inner] {test.print_dominance = true}
+    }
+  }
+  return %0 : index
+}
+
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.break
+// CHECK-SAME: = 0
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.if
+// CHECK-SAME: = 0
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} "test.foo"
+// CHECK-SAME: = 1
+// CHECK: postdominates(scf.break {{.*}} {test.print_dominance = true} {{.*}} scf.break
+// CHECK-SAME: = 1
+
+// -----
+
+// CHECK-LABEL: Testing : func_loop_early_exit_cross_block
+func.func @func_loop_early_exit_cross_block(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      test.propagate_control_flow_break {
+      ^bb0:
+        cf.cond_br %cond, ^bb1, ^bb2
+      ^bb1:
+        scf.if %cond {
+          scf.break [%outer] %arg0 : index
+        }
+        cf.br ^bb2
+      ^bb2:
+        "test.foo"() {test.print_dominance = true} : () -> ()
+        "test.return"() : () -> ()
+      }
+      scf.break [%inner]
+    }
+  }
+  return %0 : index
+}
+
+// CHECK: --- PostDominanceInfo ---
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} cf.cond_br
+// CHECK-SAME: = 0
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} scf.if
+// CHECK-SAME: = 0
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} cf.br
+// CHECK-SAME: = 1
+// CHECK: postdominates("test.foo"() {test.print_dominance = true} {{.*}} "test.foo"
+// CHECK-SAME: = 1
diff --git a/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir b/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
new file mode 100644
index 0000000000000..c5f5a05e47c46
--- /dev/null
+++ b/mlir/test/Conversion/SCFToControlFlow/convert-early-exit-to-cfg.mlir
@@ -0,0 +1,135 @@
+// RUN: mlir-opt -convert-scf-to-cf -split-input-file %s | FileCheck %s
+
+
+func.func @loop_break(%cond : i1) {
+  // CHECK: test.op1
+   "test.op1"() : () -> ()
+  // CHECK-NEXT: cf.br [[LOOP1_ENTRY:.*]]
+  // CHECK-NEXT: [[LOOP1_ENTRY]]
+   scf.loop token(%loop) {
+    // CHECK-NEXT: test.op2
+     "test.op2"() : () -> ()
+    // CHECK-NEXT: cf.cond_br %arg0, [[IF_ENTRY:.*]], [[IF_CONTINUE:.*]]
+    // CHECK-NEXT: [[IF_ENTRY]]
+     scf.if %cond {
+       "test.op3"() : () -> ()
+       scf.break [%loop] loc("break1")
+     }
+     "test.op3"() : () -> ()
+   } loc("loop1")
+   "test.op4"() : () -> ()
+   return
+}
+
+// -----
+
+// Bug regression test: IfLowering was using thenTerminator on line 461 instead
+// of elseTerminator, causing the else block's scf.yield to not be replaced with
+// a cf.branch when the then-branch has a scf.break.
+// CHECK-LABEL: func @if_break_then_yield_else
+func.func @if_break_then_yield_else(%cond : i1) {
+  // CHECK: cf.br ^[[LOOP:.*]]
+  scf.loop token(%loop) {
+    // CHECK: ^[[LOOP]]:
+    // CHECK: cf.cond_br %arg0, ^[[THEN:.*]], ^[[ELSE:.*]]
+    scf.if %cond {
+      // CHECK: ^[[THEN]]:
+      // CHECK-NEXT: cf.br ^[[EXIT:.*]]
+      scf.break [%loop]
+    } else {
+      // CHECK: ^[[ELSE]]:
+      // CHECK-NEXT: cf.br ^[[AFTER_IF:.*]]
+    }
+    // CHECK: ^[[AFTER_IF]]:
+    // CHECK-NEXT: cf.br ^[[LOOP]]
+    scf.continue [%loop]
+  }
+  // CHECK: ^[[EXIT]]:
+  // CHECK-NEXT: return
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @nested_loops_and_ifs(
+// CHECK-SAME: %[[COND1:.*]]: i1,
+// CHECK-SAME: %[[COND2:.*]]: i1
+func.func @nested_loops_and_ifs(%cond1 : i1, %cond2 : i1) {
+  // CHECK: test.op1
+   "test.op1"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[OUTER_LOOP_ENTRY:.*]]
+   scf.loop token(%outer) {
+  // CHECK-NEXT: ^[[OUTER_LOOP_ENTRY]]:
+  // CHECK-NEXT: cf.cond_br %[[COND1]], ^[[IF1_THEN_BLOCK:.*]], ^[[IF1_EXIT:.*]]
+     scf.if %cond1 {
+  // CHECK-NEXT: ^[[IF1_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op2
+       "test.op2"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY:.*]]
+       scf.loop token(%inner) {
+  // CHECK-NEXT: ^[[INNER_LOOP_ENTRY]]:
+  // CHECK-NEXT: test.op3
+         "test.op3"() : () -> ()
+  // CHECK-NEXT: cf.cond_br %[[COND1]], ^[[IF2_THEN_BLOCK:.*]], ^[[IF2_EXIT:.*]]
+         scf.if %cond1 {
+  // CHECK-NEXT: ^[[IF2_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op4
+           "test.op4"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY]]
+           scf.continue [%inner] loc("continue1")
+         }
+  // CHECK-NEXT: ^[[IF2_EXIT]]:
+  // CHECK-NEXT: test.op5
+         "test.op5"() : () -> ()
+  // CHECK-NEXT: cf.cond_br %[[COND2]], ^[[IF3_THEN_BLOCK:.*]], ^[[IF3_EXIT:.*]]
+         scf.if %cond2 {
+  // CHECK-NEXT: ^[[IF3_THEN_BLOCK]]:
+  // CHECK-NEXT: test.op6
+           "test.op6"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[FUNC_EXIT:.*]]
+           scf.break [%outer] loc("break2")
+         }
+  // CHECK-NEXT: ^[[IF3_EXIT]]:
+  // CHECK-NEXT: test.op7
+         "test.op7"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[INNER_LOOP_ENTRY]]
+         scf.continue [%inner] loc("continue2")
+       } loc("loop3")
+  // CHECK-NEXT: ^[[AFTER_INNER_LOOP:.*]]:
+  // CHECK-NEXT: test.op8
+       "test.op8"() : () -> ()
+  // CHECK-NEXT: cf.br ^[[IF1_EXIT]]
+     } loc("if1")
+  // CHECK-NEXT: ^[[IF1_EXIT]]:
+  // CHECK-NEXT: cf.br ^[[OUTER_LOOP_ENTRY]]
+     scf.continue [%outer] loc("continue3")
+   } loc("loop2")
+  // CHECK-NEXT: ^[[FUNC_EXIT]]:
+  // CHECK-NEXT: test.op9
+   "test.op9"() : () -> ()
+  // CHECK-NEXT: return
+   return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_with_iter_args
+// CHECK-SAME: %[[INIT:.*]]: i32, %[[COND:.*]]: i1
+func.func @loop_with_iter_args(%init: i32, %cond: i1) -> i32 {
+  // CHECK: cf.br ^[[LOOP:.*]](%[[INIT]] : i32)
+  %result = scf.loop token(%loop) iter_args(%arg = %init) : i32 -> i32 {
+    // CHECK: ^[[LOOP]](%[[ARG:.*]]: i32):
+    // CHECK: cf.cond_br %[[COND]], ^[[THEN:.*]], ^[[ELSE:.*]]
+    scf.if %cond {
+      // CHECK: ^[[THEN]]:
+      // CHECK-NEXT: cf.br ^[[EXIT:.*]](%[[ARG]] : i32)
+      scf.break [%loop] %arg : i32
+    }
+    // CHECK: ^[[ELSE]]:
+    // CHECK-NEXT: cf.br ^[[LOOP]](%[[ARG]] : i32)
+    scf.continue [%loop] %arg : i32
+  }
+  // CHECK: ^[[EXIT]](%[[RES:.*]]: i32):
+  // CHECK-NEXT: return %[[RES]]
+  return %result : i32
+}
diff --git a/mlir/test/Dialect/SCF/loop_canonicalize.mlir b/mlir/test/Dialect/SCF/loop_canonicalize.mlir
new file mode 100644
index 0000000000000..90e2068290082
--- /dev/null
+++ b/mlir/test/Dialect/SCF/loop_canonicalize.mlir
@@ -0,0 +1,300 @@
+// RUN: mlir-opt %s -pass-pipeline='builtin.module(func.func(canonicalize{test-convergence}))' -split-input-file | FileCheck %s
+
+// CHECK-LABEL: func @fold_single_iteration_loop1
+func.func @fold_single_iteration_loop1(%arg0 : index) -> index {
+  // CHECK-NOT: loop
+  %0 = scf.loop token(%loop) -> index {
+    scf.break [%loop] %arg0 : index
+  }
+  return %0 : index
+}
+
+// -----
+
+// CHECK-LABEL: func @fold_single_iteration_loop_with_propagating_control_flow
+func.func @fold_single_iteration_loop_with_propagating_control_flow(%cond : i1, %arg0 : index) -> index {
+  %0 = scf.loop token(%outer) -> index {
+    scf.loop token(%inner) {
+      scf.if %cond {
+        scf.break [%outer] %arg0 : index
+      }
+      scf.break [%inner]
+    }
+  }
+  return %0 : index
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_not_combine_ifs
+func.func @loop_not_combine_ifs(%arg0 : i1, %arg2: i64) {
+  // Verify that we don't combine ifs when terminator mismatches
+  scf.loop token(%loop) {
+    // CHECK: scf.if
+    %res = scf.if %arg0 -> i32 {
+      %v = "test.firstCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.firstCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+    // CHECK: scf.if
+    %res2 = scf.if %arg0 -> i32 {
+      %v = "test.secondCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.secondCodeFalse"() : () -> i32
+      scf.continue [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_combine_ifs
+func.func @loop_combine_ifs(%arg0 : i1, %arg2: i64) {
+  // CombineIfs is yield-only, so breaking terminators remain separate.
+  scf.loop token(%loop) {
+    // CHECK: scf.if
+    %res = scf.if %arg0 -> i32 {
+      %v = "test.firstCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.firstCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+    %res2 = scf.if %arg0 -> i32 {
+      %v = "test.secondCodeTrue"() : () -> i32
+      scf.yield %v : i32
+    } else {
+      %v2 = "test.secondCodeFalse"() : () -> i32
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow1
+func.func @do_not_merge_nested_if_with_breaking_control_flow1(%arg0: i1, %arg1: i1) {
+// The outer if then terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.yield
+      }
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow2
+func.func @do_not_merge_nested_if_with_breaking_control_flow2(%arg0: i1, %arg1: i1) {
+// The outer if else block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: else
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.yield
+      }
+      scf.yield
+    } else {
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow3
+func.func @do_not_merge_nested_if_with_breaking_control_flow3(%arg0: i1, %arg1: i1) {
+// The nested if then block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: test.op
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: @do_not_merge_nested_if_with_breaking_control_flow4
+func.func @do_not_merge_nested_if_with_breaking_control_flow4(%arg0: i1, %arg1: i1) {
+// The nested if else block terminator isn't a yield, blocking the merge.
+// CHECK: scf.loop
+// CHECK: scf.if
+// CHECK: scf.if
+// CHECK: else
+// CHECK-NEXT: scf.break
+  scf.loop token(%loop) {
+    scf.if %arg0 {
+      scf.if %arg1 {
+        "test.op"() : () -> ()
+      } else {
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func @do_not_convert_if_to_select1
+func.func @do_not_convert_if_to_select1(%cond: i1, %arg0 : index, %arg1 : index) -> index {
+  %loop_res  = scf.loop token(%loop) -> index {
+    // Inner then terminator is not a yield, blocking transform to select.
+    // CHECK: scf.if
+    %0 = scf.if %cond -> index {
+      scf.break [%loop] %arg0 : index
+    } else {
+      scf.yield %arg1 : index
+    }
+    scf.break [%loop] %0 : index
+  }
+  return %loop_res : index
+}
+
+// -----
+
+// CHECK-LABEL: func @do_not_convert_if_to_select2
+func.func @do_not_convert_if_to_select2(%cond: i1, %arg0 : index, %arg1 : index) -> index {
+  %loop_res  = scf.loop token(%loop) -> index {
+    // Inner then terminator is not a yield, blocking transform to select.
+    // CHECK: scf.if
+    %0 = scf.if %cond -> index {
+      scf.yield %arg0 : index
+    } else {
+      scf.break [%loop] %arg1 : index
+    }
+    scf.break [%loop] %0 : index
+  }
+  return %loop_res : index
+}
+
+
+// -----
+
+// Verify that removing the unused results of an if with nested breaking control flow
+// operation works.
+// CHECK-LABEL: func @remove_unused_if_results1
+func.func @remove_unused_if_results1(%cond : i1, %arg0 : index) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    // CHECK: %[[FOO:.*]]:3 = "test.foo"
+    %foo:3 = "test.foo" () : () -> (i32, i64, index)
+    // CHECK-NOT: %[[RES:.*]] = scf.if
+    // CHECK: scf.if
+    %res:3 = scf.if %cond -> (i32, i64, index) {
+      // CHECK: scf.yield
+      scf.yield %foo#0, %foo#1, %foo#2 : i32, i64, index
+    } else {
+      // CHECK: scf.break {{.*}} %[[FOO]]#2 : index
+      scf.break [%loop] %foo#2 : index
+    }
+      // CHECK: "test.op"(%[[FOO]]#1)
+    "test.op"(%res#1) : (i64) -> ()
+  }
+  return %0 : index
+}
+
+// -----
+
+// Verify that removing the unused results of an if with nested breaking control flow
+// operation works.
+// CHECK-LABEL: func @simplify_if_with_breaking_controlflow_in_both_branches
+func.func @simplify_if_with_breaking_controlflow_in_both_branches(%cond : i1, %cond2 : i1, %arg0 : index) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    // CHECK: %[[FOO:.*]] = "test.foo"
+    %foo = "test.foo" () : () -> (index)
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.break {{.*}} %[[FOO]] : index
+      scf.break [%loop] %foo : index
+      // CHECK-NOT: else
+    } else {
+      // CHECK: %[[BAR:.*]] = "test.bar"
+      %bar = "test.bar" () : () -> (index)
+      // CHECK: scf.if
+      scf.if %cond2 {
+        // verify that this is correctly updated when inlining the parent region.
+        // CHECK: scf.break {{.*}} %[[BAR]] : index
+        scf.break [%loop] %bar : index
+      }
+      scf.continue [%loop]
+    }
+    "test.op"() : () -> ()
+  }
+  return %0 : index
+}
+
+// -----
+
+// Verify simplification when both branches have continue.
+// CHECK-LABEL: func @simplify_if_with_continue_in_both_branches
+func.func @simplify_if_with_continue_in_both_branches(%cond : i1, %init : i32) {
+  // CHECK: scf.loop
+  scf.loop token(%loop) iter_args(%arg = %init) : i32 {
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.continue {{.*}} %{{.*}} : i32
+      scf.continue [%loop] %arg : i32
+    } else {
+      scf.continue [%loop] %arg : i32
+    }
+    // The dead code after the if should be removed.
+    // CHECK-NOT: test.op
+    "test.op"() : () -> ()
+    scf.continue [%loop] %arg : i32
+  }
+  return
+}
+
+// -----
+
+// Verify simplification when both branches have break.
+// CHECK-LABEL: func @simplify_if_with_break_in_both_branches
+func.func @simplify_if_with_break_in_both_branches(%cond : i1) -> index {
+  // CHECK: scf.loop
+  %0 = scf.loop token(%loop) -> index {
+    %c0 = arith.constant 0 : index
+    %c1 = arith.constant 1 : index
+    // CHECK: scf.if
+    scf.if %cond {
+      // CHECK: scf.break
+      scf.break [%loop] %c0 : index
+    } else {
+      scf.break [%loop] %c1 : index
+    }
+    // Dead code after the if.
+    // CHECK-NOT: test.op
+    "test.op"() : () -> ()
+    scf.continue [%loop]
+  }
+  return %0 : index
+}
diff --git a/mlir/test/IR/early-exit-invalid.mlir b/mlir/test/IR/early-exit-invalid.mlir
new file mode 100644
index 0000000000000..509b39629ebc7
--- /dev/null
+++ b/mlir/test/IR/early-exit-invalid.mlir
@@ -0,0 +1,142 @@
+
+// RUN: mlir-opt %s --split-input-file --verify-diagnostics
+
+func.func @loop_result_mismatch(%value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ %result = scf.loop token(%loop) -> i32 {
+   scf.break [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+// -----
+
+// A PropagateControlFlowBreak op that is immediately nested in a
+// RegionBranchOpInterface can contain a region terminator that targets an
+// ancestor of that RegionBranchOpInterface. The intermediate RegionBranchOp
+// must expose this as a possible propagated edge so the addressed receiver can
+// verify the break operands against its results.
+func.func @outer_loop_result_mismatch_through_if_in_inner_loop(%cond : i1,
+                                                              %value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ %result = scf.loop token(%outer) -> i32 {
+   scf.loop token(%inner) {
+     scf.if %cond {
+       scf.break [%outer] %value : f32 // expected-note {{region branch point}}
+     }
+     scf.continue [%inner]
+   }
+   scf.continue [%outer]
+ }
+ return
+}
+
+// -----
+
+func.func @loop_result_number_mismatch(%value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.break to Operation scf.loop: region branch point has 1 operands, but region successor needs 2 inputs}}
+ %result:2 = scf.loop token(%loop) -> f32, f32 {
+   scf.break [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+// -----
+
+func.func @loop_continue_mismatch(%init : i32, %value : f32) {
+ // expected-error @+1 {{'scf.loop' op along control flow edge from Operation scf.continue to Region #0: successor operand type #0 'f32' should match successor input type #0 'i32'}}
+ scf.loop token(%loop) iter_args(%next = %init) : i32 {
+   scf.continue [%loop] %value : f32 // expected-note {{region branch point}}
+ }
+ return
+}
+
+
+// -----
+
+func.func @loop_iterargs_mismatch(%init : i32, %value : f32) {
+ // expected-error @+2 {{'scf.loop' op along control flow edge from parent to Region #0: successor operand type #0 'i32' should match successor input type #0 'f32'}}
+ // expected-note @+1 {{region branch point}}
+ "scf.loop"(%init) ({
+    ^body(%token : token, %next : f32):
+   scf.continue [%token] %init : i32
+ })  : (i32) -> ()
+ return
+}
+
+// -----
+
+func.func @loop_iterargs_mismatch(%init : i32, %value : f32) {
+ // expected-error @+2 {{'scf.loop' op along control flow edge from parent to Region #0: region branch point has 1 operands, but region successor needs 2 inputs}}
+ // expected-note @+1 {{region branch point}}
+ "scf.loop"(%init) ({
+    ^body(%token : token, %next : i32, %next2 : f32):
+   scf.continue [%token] %init : i32
+ })  : (i32) -> ()
+ return
+}
+
+// -----
+
+// scf.for lacks PropagateControlFlowBreak, so it cannot be an intermediate
+// parent for a break targeting an enclosing loop token.
+func.func @break_through_for_missing_trait(%lb: index, %ub: index, %step: index, %cond: i1) {
+  scf.loop token(%loop) {
+    scf.for %i = %lb to %ub step %step {
+      scf.if %cond {
+        // expected-error @+1 {{target token crosses an op that does not have the PropagateControlFlowBreak trait}}
+        scf.break [%loop]
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// scf.while lacks PropagateControlFlowBreak, so break through it is rejected.
+// The scf.while verifier catches this first as the after region must terminate
+// with scf.yield.
+func.func @break_through_while(%cond: i1) {
+  %init = arith.constant true
+  scf.loop token(%loop) {
+    // expected-error @+1 {{'scf.while' op expects the 'after' region to terminate with 'scf.yield'}}
+    scf.while(%arg = %init) : (i1) -> i1 {
+      scf.condition(%arg) %arg : i1
+    } do {
+    ^bb0(%arg2: i1):
+      // expected-note @+1 {{terminator here}}
+      scf.break [%loop]
+    }
+  }
+  return
+}
+
+// -----
+
+// Verify that a break without a token target is rejected.
+func.func @break_without_token() {
+  scf.loop token(%loop) {
+    // expected-error @+1 {{'scf.break' op expected 1 or more operands}}
+    "scf.break"() : () -> ()
+  }
+  return
+}
+
+// -----
+
+// Verify that a loop whose first region argument is not a control token is
+// rejected. The break targets the outer loop so the inner loop does not need a
+// self-targeting terminator, isolating the control-token check.
+func.func @loop_non_token_control(%cond: i1) {
+  "scf.loop"() ({
+  ^bb0(%outer: token):
+    // expected-error @+1 {{'scf.loop' op first region argument must be a control token}}
+    "scf.loop"() ({
+    ^bb1(%bad: i32):
+      "scf.break"(%outer) : (token) -> ()
+    }) : () -> ()
+    "scf.continue"(%outer) : (token) -> ()
+  }) : () -> ()
+  return
+}
diff --git a/mlir/test/IR/early-exit.mlir b/mlir/test/IR/early-exit.mlir
new file mode 100644
index 0000000000000..82e7f92070e5b
--- /dev/null
+++ b/mlir/test/IR/early-exit.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt --print-region-branch-op-interface %s --split-input-file | FileCheck %s
+// RUN: mlir-opt %s --mlir-print-debuginfo --mlir-print-op-generic --split-input-file  | mlir-opt --print-region-branch-op-interface --split-input-file  | FileCheck %s
+
+
+func.func @loop_break(%cond : i1) {
+  // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop1")
+  // CHECK:  - Successor is region #0
+  // CHECK:  - Found 2 predecessor(s)
+  // CHECK:    - Predecessor is scf.break {{.*}} loc("break1")
+  // CHECK:    - Predecessor is scf.continue
+   scf.loop token(%loop) {
+     scf.if %cond {
+       scf.break [%loop] loc("break1")
+     }
+   } loc("loop1")
+   return
+}
+
+// -----
+
+func.func @loop_continue(%cond1 : i1, %cond2 : i1) {
+  // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop2")
+  // CHECK:  - Successor is region #0
+  // CHECK:  - Found 2 predecessor(s)
+  // CHECK:    - Predecessor is scf.break {{.*}} loc("break2")
+  // CHECK:    - Predecessor is scf.continue
+   scf.loop token(%outer) {
+    // CHECK: Found RegionBranchOpInterface operation: scf.loop {{.*}} {...} loc("loop3")
+    // CHECK:  - Successor is region #0
+    // CHECK:  - Found 2 predecessor(s)
+    // CHECK:    - Predecessor is scf.continue {{.*}} loc("continue1")
+    // CHECK:    - Predecessor is scf.continue
+     scf.loop token(%inner) {
+       scf.if %cond1 {
+         scf.continue [%inner] loc("continue1")
+       }
+       scf.if %cond2 {
+         scf.break [%outer] loc("break2")
+       }
+     } loc("loop3")
+   } loc("loop2")
+   return
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_with_results(
+func.func @loop_with_results(%value : f32) -> f32 {
+ %result = scf.loop token(%loop) -> f32 {
+   scf.break [%loop] %value : f32
+ }
+ return %result : f32
+}
+
+// -----
+
+// CHECK-LABEL: func @loop_continue_iterargs(
+func.func @loop_continue_iterargs(%init : i32) {
+ scf.loop token(%loop) iter_args(%next = %init) : i32 {
+   scf.continue [%loop] %next : i32
+ }
+ return
+}
+
+// -----
+
+// A single-operand continue that targets an *outer* loop must be printed
+// explicitly. If the printer elided it as the trivial implicit terminator, the
+// parser would rebuild it as a continue of the inner loop, silently changing
+// the target.
+// CHECK-LABEL: func @continue_outer_from_inner
+func.func @continue_outer_from_inner() {
+  // CHECK: scf.loop token(%[[OUTER:.*]]) {
+  scf.loop token(%outer) {
+    // CHECK: scf.loop token(%[[INNER:.*]]) {
+    scf.loop token(%inner) {
+      // CHECK: scf.continue [%[[OUTER]]]
+      scf.continue [%outer]
+    }
+  }
+  return
+}
diff --git a/mlir/test/Integration/Dialect/SCF/early_exit.mlir b/mlir/test/Integration/Dialect/SCF/early_exit.mlir
new file mode 100644
index 0000000000000..518b602f790aa
--- /dev/null
+++ b/mlir/test/Integration/Dialect/SCF/early_exit.mlir
@@ -0,0 +1,82 @@
+// RUN: mlir-opt %s  -convert-scf-to-cf --canonicalize --convert-cf-to-llvm --convert-to-llvm | \
+// RUN: mlir-runner -e entry -entry-point-result=void  \
+// RUN: -shared-libs=%mlir_c_runner_utils | \
+// RUN: FileCheck %s
+
+
+
+// End-to-end test of all fp reduction intrinsics (not exhaustive unit tests).
+module {
+  llvm.func @entry() {
+   // Constant for the iteration space and various conditions
+   %one = llvm.mlir.constant(1 : i64) : i64
+   %two = llvm.mlir.constant(2 : i64) : i64
+   %three = llvm.mlir.constant(3 : i64) : i64
+   %four = llvm.mlir.constant(4 : i64) : i64
+   %counter_init = llvm.mlir.constant(0 : i64) : i64
+
+
+// CHECK: Outer Loop Begin with counter: 0
+// CHECK-NEXT: Inner Loop Begin, counter: 1
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 2
+// CHECK-NEXT: Iteration 2, loop back to outer loop
+// CHECK-NEXT: Outer Loop Begin with counter: 2
+// CHECK-NEXT: Inner Loop Begin, counter: 3
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 4
+// CHECK-NEXT: continue inner loop
+// CHECK-NEXT: Inner Loop Begin, counter: 5
+// CHECK-NEXT: Last iteration, break out of outer loop
+// CHECK-NEXT: Outer loop finished with result: 4
+
+
+   %result = scf.loop token(%outer) iter_args(%counter_out = %counter_init) : i64 -> i64 {
+     // Outer loop iteration
+     vector.print str "Outer Loop Begin with counter: "
+     vector.print %counter_out : i64
+
+     scf.loop token(%inner) iter_args(%counter = %counter_out) : i64 {
+       // %counter will go from 0 to 4
+       // %counter_update will go from 1 to 5
+       %counter_update = llvm.add %counter, %one : i64
+
+       // Inner loop iteration
+       // print from 1..5
+       vector.print str "Inner Loop Begin, counter: "
+       vector.print  %counter_update : i64
+
+       // On the second iteration, print 2.3 and loop back to the outer loop.
+       %cond1 = llvm.icmp "eq" %counter_update, %two : i64
+       scf.if %cond1 {
+         vector.print str "Iteration 2, loop back to outer loop\n"
+         scf.continue [%outer] %counter_update : i64
+       }
+
+       // Exit condition when counter>4
+       %cond2 = llvm.icmp "sge" %counter, %four : i64
+       scf.if %cond2 {
+         vector.print str "Last iteration, break out of outer loop\n"
+         // return the counter from the previous iteration here (pre-update)
+         scf.break [%outer] %counter : i64
+       }
+
+       %cond3 = llvm.icmp "eq" %counter_update, %three : i64
+       scf.if %cond2 {
+         vector.print str "Iteration 3, break out of inner loop"
+         scf.break [%inner]
+       }
+       vector.print str "continue inner loop\n"
+       scf.continue [%inner] %counter_update : i64
+     }
+      vector.print str "continue outer loop\n"
+      scf.continue [%outer] %counter_out : i64
+   }
+
+// After the loop nest finishes
+    vector.print str "Outer loop finished with result: "
+    vector.print  %result : i64
+
+    llvm.return
+  }
+}
diff --git a/mlir/test/Transforms/sccp-early-exit.mlir b/mlir/test/Transforms/sccp-early-exit.mlir
new file mode 100644
index 0000000000000..4a4136b94b891
--- /dev/null
+++ b/mlir/test/Transforms/sccp-early-exit.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt -allow-unregistered-dialect %s -pass-pipeline="builtin.module(func.func(sccp))" -split-input-file | FileCheck %s
+
+/// The inner loop's only normal exit (break 1) carries the constant 5.
+/// The early-exit path (break 3) carries -5 but bypasses the code after the
+/// inner loop entirely — it exits both loops at once.
+///
+/// Without early-exit support the analysis would conservatively join {5, -5}
+/// and mark %inner as overdefined, keeping the dead branch alive.
+/// With early-exit support SCCP sees that %inner is always 5, folds the
+/// comparison, and removes the dead scf.if.
+///
+/// Pseudocode:
+///   loop {
+///     a = loop {
+///       if (cond) { break_all -5; }  // early exit from both loops
+///       break 5;                      // normal exit from inner loop
+///     };
+///     // here a == 5 always
+///     if (a < 0) "dead"();            // dead code
+///     break a;
+///   }
+
+// CHECK-LABEL: func @early_exit_dead_code(
+// CHECK-DAG:     %[[FALSE:.*]] = arith.constant false
+// CHECK-DAG:     %[[C5:.*]] = arith.constant 5 : i32
+// CHECK:         scf.loop
+// CHECK:           scf.loop
+// CHECK:           scf.if %[[FALSE]]
+// CHECK:             "test.dead_op"
+// CHECK:           scf.break {{.*}} %[[C5]] : i32
+func.func @early_exit_dead_code(%cond: i1) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %c5 = arith.constant 5 : i32
+  %cm5 = arith.constant -5 : i32
+  %outer = scf.loop token(%outer_token) -> i32 {
+    %inner = scf.loop token(%inner_token) -> i32 {
+      scf.if %cond {
+        scf.break [%outer_token] %cm5 : i32
+      }
+      scf.break [%inner_token] %c5 : i32
+    }
+    %is_neg = arith.cmpi slt, %inner, %c0 : i32
+    scf.if %is_neg {
+      "test.dead_op"() : () -> ()
+    }
+    scf.break [%outer_token] %inner : i32
+  }
+  return %outer : i32
+}
+
+// -----
+
+/// For contrast: the same logic emulated with scf.while and a boolean flag
+/// instead of early exit.  Without break 3 to bypass the code after the inner
+/// loop, both values (-5 and 5) flow through %inner and SCCP sees it as
+/// overdefined — the comparison and dead branch survive.
+///
+/// Pseudocode:
+///   done = false
+///   while (!done) {
+///     a, should_break = while (!done) {
+///       if (cond) yield -5, true   // want to break all, but can't
+///       else      yield  5, false
+///     };
+///     if (!should_break) {
+///       // a ∈ {5, -5} — flag can't help the analysis narrow it
+///       if (a < 0) "not dead"();
+///     }
+///     done = true
+///   }
+
+// CHECK-LABEL: func @no_early_exit_not_foldable(
+// CHECK:         scf.while
+// CHECK:           scf.while
+// CHECK:           scf.if
+// CHECK:             arith.cmpi slt,
+// CHECK:             scf.if
+// CHECK:               "test.not_dead_op"
+func.func @no_early_exit_not_foldable(%cond: i1) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %c5 = arith.constant 5 : i32
+  %cm5 = arith.constant -5 : i32
+  %false = arith.constant false
+  %true = arith.constant true
+
+  // Outer while: (result, done)
+  %outer, %_ = scf.while(%o_res = %c0, %o_done = %false) : (i32, i1) -> (i32, i1) {
+    %o_go = arith.xori %o_done, %true : i1
+    scf.condition(%o_go) %o_res, %o_done : i32, i1
+  } do {
+  ^bb0(%o_res: i32, %o_done: i1):
+    // Inner while: (result, done, should_break_outer)
+    %inner, %_2, %break_flag = scf.while(%i_res = %c0, %i_done = %false, %i_brk = %false)
+        : (i32, i1, i1) -> (i32, i1, i1) {
+      %i_go = arith.xori %i_done, %true : i1
+      scf.condition(%i_go) %i_res, %i_done, %i_brk : i32, i1, i1
+    } do {
+    ^bb0(%i_res: i32, %i_done: i1, %i_brk: i1):
+      // if (cond) a = -5, flag = true; else a = 5, flag = false
+      %a = arith.select %cond, %cm5, %c5 : i32
+      %brk = arith.select %cond, %true, %false : i1
+      scf.yield %a, %true, %brk : i32, i1, i1
+    }
+    // %inner ∈ {5, -5}: both values reach here (no early exit to bypass)
+    // %break_flag correlates with %inner but SCCP can't exploit that
+    %not_break = arith.xori %break_flag, %true : i1
+    scf.if %not_break {
+      // %inner ∈ {5} in theory: but the analysis can't recover this here.
+      %is_neg = arith.cmpi slt, %inner, %c0 : i32
+      scf.if %is_neg {
+        "test.not_dead_op"() : () -> ()
+      }
+    }
+    scf.yield %inner, %true : i32, i1
+  }
+  return %outer : i32
+}
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index c12e2abb3ab75..9d7e21cabda6f 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -2385,6 +2385,12 @@ def TestMergeBlocksOp : TEST_Op<"merge_blocks"> {
   let results = (outs Variadic<AnyType>:$result);
 }
 
+def TestPropagateControlFlowBreakOp
+    : TEST_Op<"propagate_control_flow_break", [PropagateControlFlowBreak]> {
+  let regions = (region AnyRegion:$body);
+  let assemblyFormat = "attr-dict-with-keyword $body";
+}
+
 def TestRemappedValueRegionOp : TEST_Op<"remapped_value_region",
                                         [SingleBlock]> {
   let summary = "remapped_value_region operation";
diff --git a/mlir/test/lib/IR/TestDominance.cpp b/mlir/test/lib/IR/TestDominance.cpp
index b34149b3e2cbd..24696c8dd028c 100644
--- a/mlir/test/lib/IR/TestDominance.cpp
+++ b/mlir/test/lib/IR/TestDominance.cpp
@@ -14,6 +14,7 @@
 
 #include "mlir/IR/Builders.h"
 #include "mlir/IR/Dominance.h"
+#include "mlir/IR/Operation.h"
 #include "mlir/IR/SymbolTable.h"
 #include "mlir/Pass/Pass.h"
 
@@ -21,6 +22,14 @@ using namespace mlir;
 
 /// Overloaded helper to call the right function based on whether we are testing
 /// dominance or post-dominance.
+static bool dominatesOrPostDominates(DominanceInfo &dominanceInfo, Operation *a,
+                                     Operation *b) {
+  return dominanceInfo.dominates(a, b);
+}
+static bool dominatesOrPostDominates(PostDominanceInfo &dominanceInfo,
+                                     Operation *a, Operation *b) {
+  return dominanceInfo.postDominates(a, b);
+}
 static bool dominatesOrPostDominates(DominanceInfo &dominanceInfo, Block *a,
                                      Block *b) {
   return dominanceInfo.dominates(a, b);
@@ -72,6 +81,30 @@ class DominanceTest {
   template <typename DominanceT>
   void printDominance(DominanceT &dominanceInfo,
                       bool printCommonDominatorInfo) {
+    if (printCommonDominatorInfo) {
+      operation->walk([&](Operation *op) {
+        if (!op->getDiscardableAttr("test.print_dominance"))
+          return;
+        operation->walk([&](Operation *nested) {
+          if (std::is_same<DominanceInfo, DominanceT>::value)
+            llvm::outs() << "dominates(";
+          else
+            llvm::outs() << "postdominates(";
+          bool isDominated =
+              dominatesOrPostDominates(dominanceInfo, op, nested);
+          llvm::outs() << OpWithFlags(op, OpPrintingFlags()
+                                              .skipRegions()
+                                              .enableDebugInfo()
+                                              .assumeVerified())
+                       << ", "
+                       << OpWithFlags(nested, OpPrintingFlags()
+                                                  .skipRegions()
+                                                  .enableDebugInfo()
+                                                  .assumeVerified())
+                       << ") = " << std::to_string(isDominated) << "\n";
+        });
+      });
+    }
     DenseSet<Block *> parentVisited;
     operation->walk([&](Operation *op) {
       Block *block = op->getBlock();
diff --git a/mlir/test/lib/Interfaces/CMakeLists.txt b/mlir/test/lib/Interfaces/CMakeLists.txt
index 6a21ed10eec6f..3aa5097b7ed20 100644
--- a/mlir/test/lib/Interfaces/CMakeLists.txt
+++ b/mlir/test/lib/Interfaces/CMakeLists.txt
@@ -1,2 +1,3 @@
 add_subdirectory(LoopLikeInterface)
+add_subdirectory(RegionBranchOpInterface)
 add_subdirectory(TilingInterface)
diff --git a/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt b/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
new file mode 100644
index 0000000000000..8e003942e41c0
--- /dev/null
+++ b/mlir/test/lib/Interfaces/RegionBranchOpInterface/CMakeLists.txt
@@ -0,0 +1,9 @@
+add_mlir_library(MLIRTestRegionBranchOpInterface
+  TestRegionBranchOpInterface.cpp
+
+  EXCLUDE_FROM_LIBMLIR
+  )
+mlir_target_link_libraries(MLIRTestRegionBranchOpInterface PUBLIC
+  MLIRControlFlowInterfaces
+  MLIRPass
+  )
diff --git a/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp b/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp
new file mode 100644
index 0000000000000..878de7d44da01
--- /dev/null
+++ b/mlir/test/lib/Interfaces/RegionBranchOpInterface/TestRegionBranchOpInterface.cpp
@@ -0,0 +1,77 @@
+//===- TestRegionBranchOpInterface.cpp - RegionBranch test pass -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Func/IR/FuncOps.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/Interfaces/ControlFlowInterfaces.h"
+#include "mlir/Pass/Pass.h"
+#include "llvm/Support/raw_ostream.h"
+
+using namespace mlir;
+
+namespace {
+/// Test pass that prints RegionBranchOpInterface successor information and
+/// nested breaking-control-flow predecessors.
+struct PrintRegionBranchOpInterfacePass
+    : public PassWrapper<PrintRegionBranchOpInterfacePass, OperationPass<>> {
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(PrintRegionBranchOpInterfacePass)
+
+  StringRef getArgument() const final {
+    return "print-region-branch-op-interface";
+  }
+  StringRef getDescription() const final {
+    return "Print control-flow edges represented by "
+           "mlir::RegionBranchOpInterface";
+  }
+
+  void runOnOperation() override {
+    Operation *op = getOperation();
+    op->walk<WalkOrder::PreOrder>([&](RegionBranchOpInterface branchOp) {
+      llvm::outs() << "Found RegionBranchOpInterface operation: "
+                   << OpWithFlags(
+                          branchOp,
+                          OpPrintingFlags().skipRegions().enableDebugInfo())
+                   << "\n";
+      SmallVector<RegionSuccessor> regions;
+      branchOp.getSuccessorRegions(RegionBranchPoint::parent(), regions);
+      for (auto &successor : regions) {
+        if (successor.isOperation()) {
+          llvm::outs() << " - Successor is operation "
+                       << successor.getSuccessorOp()->getName() << "\n";
+        } else {
+          llvm::outs() << " - Successor is region #"
+                       << successor.getSuccessor()->getRegionNumber() << "\n";
+        }
+      }
+      if (auto breakingControlFlowOp =
+              dyn_cast<HasBreakingControlFlowOpInterface>(
+                  branchOp.getOperation())) {
+        SmallVector<Operation *> predecessors;
+        llvm::outs() << " - Collecting all nested predecessors\n";
+        collectAllNestedPredecessors(breakingControlFlowOp, predecessors);
+        llvm::outs() << " - Found " << predecessors.size()
+                     << " predecessor(s)\n";
+        for (auto &predecessor : predecessors) {
+          llvm::outs() << "  - Predecessor is "
+                       << OpWithFlags(
+                              predecessor,
+                              OpPrintingFlags().skipRegions().enableDebugInfo())
+                       << "\n";
+        }
+      }
+    });
+  }
+};
+
+} // namespace
+
+namespace mlir {
+void registerRegionBranchOpInterfaceTestPasses() {
+  PassRegistration<PrintRegionBranchOpInterfacePass>();
+}
+} // namespace mlir
diff --git a/mlir/tools/mlir-opt/CMakeLists.txt b/mlir/tools/mlir-opt/CMakeLists.txt
index c607ccfa80e3c..821bad3a4166a 100644
--- a/mlir/tools/mlir-opt/CMakeLists.txt
+++ b/mlir/tools/mlir-opt/CMakeLists.txt
@@ -22,6 +22,7 @@ if(MLIR_INCLUDE_TESTS)
     MLIRGPUTestPasses
     MLIRLinalgTestPasses
     MLIRLoopLikeInterfaceTestPasses
+    MLIRTestRegionBranchOpInterface
     MLIRMathTestPasses
     MLIRTestMathToVCIX
     MLIRMemRefTestPasses
diff --git a/mlir/tools/mlir-opt/mlir-opt.cpp b/mlir/tools/mlir-opt/mlir-opt.cpp
index 13c0934f34656..8c8813b480a99 100644
--- a/mlir/tools/mlir-opt/mlir-opt.cpp
+++ b/mlir/tools/mlir-opt/mlir-opt.cpp
@@ -38,6 +38,7 @@ void registerLazyLoadingTestPasses();
 void registerLoopLikeInterfaceTestPasses();
 void registerPassManagerTestPass();
 void registerPrintSpirvAvailabilityPass();
+void registerRegionBranchOpInterfaceTestPasses();
 void registerRegionTestPasses();
 void registerPrintTosaAvailabilityPass();
 void registerShapeFunctionTestPasses();
@@ -191,6 +192,7 @@ static void registerTestPasses() {
   registerPassManagerTestPass();
   registerPrintSpirvAvailabilityPass();
   registerRegionTestPasses();
+  registerRegionBranchOpInterfaceTestPasses();
   registerShapeFunctionTestPasses();
   registerSideEffectTestPasses();
   registerSliceAnalysisTestPass();

>From 968e49b11932bc78f5870972edb987bec35fd8ff Mon Sep 17 00:00:00 2001
From: Mehdi Amini <joker.eph at gmail.com>
Date: Thu, 11 Jun 2026 07:11:07 -0700
Subject: [PATCH 2/2] [mlir][test] Add dynamic-depth early-exit loop ops

Add Test dialect loop and terminators that model dynamic breaking control-flow
targets with an index depth. Constant depths resolve to one enclosing
test.breakable_loop; dynamic depths expose compatible reachable loop targets
through getPotentialTargets().

Filter dynamic potential targets by the terminator payload types. Runtime depths
that select an incompatible filtered-out target are undefined, while constant
depths still diagnose payload mismatches against the selected loop.

Add a test conversion pass that lowers test.breakable_loop to scf.loop and
rewrites dynamic break/continue terminators to token-targeted SCF terminators.
Dynamic depths lower through a dispatch ladder over compatible target loops
while preserving each target's original enclosing-loop depth.

Add verifier coverage for invalid depths, blocked propagation, and payload type
mismatches, RegionBranchOpInterface coverage for dynamic targets, FileCheck
coverage for the SCF conversion, and execution tests that lower through
SCF-to-CF.

Assisted-by: Codex
---
 .../IR/early-exit-test-dialect-invalid.mlir   |  93 +++++
 mlir/test/IR/early-exit-test-dialect.mlir     | 124 +++++++
 .../test-convert-breakable-loop-to-scf.mlir   | 139 +++++++
 mlir/test/lib/Dialect/Test/TestOpDefs.cpp     | 340 ++++++++++++++++++
 mlir/test/lib/Dialect/Test/TestOps.td         |  67 ++++
 mlir/test/lib/Dialect/Test/TestPatterns.cpp   | 213 +++++++++++
 .../test-convert-breakable-loop-to-scf.mlir   | 145 ++++++++
 7 files changed, 1121 insertions(+)
 create mode 100644 mlir/test/IR/early-exit-test-dialect-invalid.mlir
 create mode 100644 mlir/test/IR/early-exit-test-dialect.mlir
 create mode 100644 mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
 create mode 100644 mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir

diff --git a/mlir/test/IR/early-exit-test-dialect-invalid.mlir b/mlir/test/IR/early-exit-test-dialect-invalid.mlir
new file mode 100644
index 0000000000000..c5ca385e4aa6b
--- /dev/null
+++ b/mlir/test/IR/early-exit-test-dialect-invalid.mlir
@@ -0,0 +1,93 @@
+// RUN: mlir-opt %s --split-input-file --verify-diagnostics
+
+func.func @non_index_depth(%depth: i32) {
+  test.breakable_loop {
+    // expected-error @+1 {{operand #0 must be index}}
+    "test.dynamic_break"(%depth) : (i32) -> ()
+  }
+  return
+}
+
+// -----
+
+func.func @no_reachable_loop(%depth: index) {
+  // expected-error @+1 {{must be nested inside a reachable test.breakable_loop}}
+  test.dynamic_break %depth
+}
+
+// -----
+
+func.func @zero_depth() {
+  %c0 = arith.constant 0 : index
+  test.breakable_loop {
+    // expected-error @+1 {{depth must be positive}}
+    test.dynamic_continue %c0
+  }
+  return
+}
+
+// -----
+
+func.func @too_deep() {
+  %c2 = arith.constant 2 : index
+  test.breakable_loop {
+    // expected-error @+1 {{constant depth exceeds the number of reachable test.breakable_loop operations}}
+    test.dynamic_break %c2
+  }
+  return
+}
+
+// -----
+
+func.func @blocked_parent() {
+  %c1 = arith.constant 1 : index
+  test.breakable_loop {
+    "test.any_cond"() ({
+      // expected-error @+1 {{depth target crosses an op that does not have the PropagateControlFlowBreak trait}}
+      test.dynamic_break %c1
+    }) : () -> ()
+    test.dynamic_continue %c1
+  }
+  return
+}
+
+// -----
+
+func.func @dynamic_depth_blocked_outer(%depth: index) {
+  %c1 = arith.constant 1 : index
+  test.breakable_loop {
+    test.single_no_terminator_custom_asm_op {
+      test.breakable_loop {
+        // expected-error @+1 {{dynamic depth may target across an op that does not have the PropagateControlFlowBreak trait}}
+        test.dynamic_break %depth
+      }
+    }
+    test.dynamic_continue %c1
+  }
+  return
+}
+
+// -----
+
+// Constant depths select one concrete loop, so payload mismatches are still
+// diagnosed against that selected target.
+func.func @constant_break_payload_mismatch(%value: f32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %result = test.breakable_loop -> i32 {
+    // expected-error @+1 {{payload operand #0 has type 'f32', but target results}}
+    test.dynamic_break %c1 %value : f32
+  }
+  return %result : i32
+}
+
+// -----
+
+// Dynamic depths filter out incompatible targets. This is still invalid when
+// filtering leaves no target that can accept the terminator payload.
+func.func @dynamic_break_no_compatible_target(%depth: index, %value: f32) -> i32 {
+  %result = test.breakable_loop -> i32 {
+    // expected-error @+1 {{dynamic depth has no compatible test.breakable_loop target}}
+    test.dynamic_break %depth %value : f32
+  }
+  return %result : i32
+}
diff --git a/mlir/test/IR/early-exit-test-dialect.mlir b/mlir/test/IR/early-exit-test-dialect.mlir
new file mode 100644
index 0000000000000..3ea7f0e8154b8
--- /dev/null
+++ b/mlir/test/IR/early-exit-test-dialect.mlir
@@ -0,0 +1,124 @@
+// RUN: mlir-opt %s --split-input-file | FileCheck %s --check-prefix=PRINT
+// RUN: mlir-opt --print-region-branch-op-interface %s --split-input-file | FileCheck %s --check-prefix=BRANCH
+
+// PRINT-LABEL: func.func @immediate_continue(
+func.func @immediate_continue(%depth: index) {
+  // PRINT: test.breakable_loop
+  test.breakable_loop {
+    // PRINT: test.dynamic_continue %arg0
+    test.dynamic_continue %depth
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @break_with_result(
+func.func @break_with_result(%depth: index, %value: i32) -> i32 {
+  // PRINT: test.breakable_loop -> i32
+  %result = test.breakable_loop -> i32 {
+    // PRINT: test.dynamic_break %arg0 %arg1 : i32
+    test.dynamic_break %depth %value : i32
+  }
+  return %result : i32
+}
+
+// -----
+
+// PRINT-LABEL: func.func @continue_iter_args(
+func.func @continue_iter_args(%depth: index, %init: i32) {
+  // PRINT: test.breakable_loop iter_args(%{{.*}} = %arg1) : i32
+  test.breakable_loop iter_args(%iter = %init) : i32 {
+    // PRINT: test.dynamic_continue %arg0 %{{.*}} : i32
+    test.dynamic_continue %depth %iter : i32
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @constant_depth_selects_outer(
+func.func @constant_depth_selects_outer(%outer_init: i32, %inner_init: f32) {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  test.breakable_loop iter_args(%outer = %outer_init) : i32 {
+    test.breakable_loop iter_args(%inner = %inner_init) : f32 {
+      // PRINT: test.dynamic_continue %c2 %{{.*}} : i32
+      test.dynamic_continue %c2 %outer : i32
+    }
+    test.dynamic_continue %c1 %outer : i32
+  }
+  return
+}
+
+// -----
+
+// PRINT-LABEL: func.func @through_scf_if(
+func.func @through_scf_if(%cond: i1, %depth: index) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("loop")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 2 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("if_break")
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("after_if")
+  test.breakable_loop {
+    // PRINT: scf.if
+    scf.if %cond {
+      // PRINT: test.dynamic_break %arg1
+      test.dynamic_break %depth loc("if_break")
+    } loc("if")
+    test.dynamic_continue %c1 loc("after_if")
+  } loc("loop")
+  return
+}
+
+// -----
+
+// A dynamic depth can target either the immediately enclosing loop or an outer
+// compatible loop. The inner loop sees the dynamic break as a direct
+// predecessor, and the outer loop sees both the propagated dynamic break and
+// its own explicit continue.
+func.func @nested_dynamic(%depth: index) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("outer")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 2 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("dyn_break")
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("outer_continue")
+  test.breakable_loop {
+    // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("inner")
+    // BRANCH:  - Successor is region #0
+    // BRANCH:  - Found 1 predecessor(s)
+    // BRANCH:  - Predecessor is test.dynamic_break {{.*}} loc("dyn_break")
+    test.breakable_loop {
+      test.dynamic_break %depth loc("dyn_break")
+    } loc("inner")
+    test.dynamic_continue %c1 loc("outer_continue")
+  } loc("outer")
+  return
+}
+
+// -----
+
+// The dynamic continue payload is f32. That makes the inner f32 loop a
+// potential target, but filters out the enclosing i32 loop. The outer loop
+// should only see its explicit continue predecessor.
+func.func @dynamic_continue_filters_incompatible_outer(%depth: index, %i: i32,
+                                                       %f: f32) {
+  %c1 = arith.constant 1 : index
+  // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("filtered_outer")
+  // BRANCH:  - Successor is region #0
+  // BRANCH:  - Found 1 predecessor(s)
+  // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("filtered_outer_continue")
+  test.breakable_loop iter_args(%outer = %i) : i32 {
+    // BRANCH-LABEL: Found RegionBranchOpInterface operation: test.breakable_loop {{.*}} loc("filtered_inner")
+    // BRANCH:  - Successor is region #0
+    // BRANCH:  - Found 1 predecessor(s)
+    // BRANCH:  - Predecessor is test.dynamic_continue {{.*}} loc("filtered_inner_continue")
+    test.breakable_loop iter_args(%inner = %f) : f32 {
+      test.dynamic_continue %depth %inner : f32 loc("filtered_inner_continue")
+    } loc("filtered_inner")
+    test.dynamic_continue %c1 %outer : i32 loc("filtered_outer_continue")
+  } loc("filtered_outer")
+  return
+}
diff --git a/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir b/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
new file mode 100644
index 0000000000000..f33530e6d91d1
--- /dev/null
+++ b/mlir/test/Transforms/test-convert-breakable-loop-to-scf.mlir
@@ -0,0 +1,139 @@
+// RUN: mlir-opt %s --test-convert-breakable-loop-to-scf --split-input-file | FileCheck %s
+
+// CHECK-LABEL: func.func @immediate_continue(
+// CHECK-SAME: %[[INIT:.*]]: i32
+func.func @immediate_continue(%init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[TOKEN:.*]]) iter_args(%[[ITER:.*]] = %[[INIT]]) : i32 -> i32
+  %result = test.breakable_loop iter_args(%iter = %init) : i32 -> i32 {
+    // CHECK: scf.continue [%[[TOKEN]]] %[[ITER]] : i32
+    test.dynamic_continue %c1 %iter : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// CHECK-LABEL: func.func @constant_outer_break(
+func.func @constant_outer_break(%init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) {
+    test.breakable_loop {
+      // CHECK: scf.break [%[[OUTER_TOKEN]]] %[[OUTER_ARG]] : i32
+      test.dynamic_break %c2 %outer : i32
+    }
+    test.dynamic_break %c1 %outer : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// Dynamic break over two compatible loops lowers to a dispatch ladder with one
+// arm for depth 1 (inner) and one for depth 2 (outer), plus a deterministic
+// fallback for UB depth values.
+// CHECK-LABEL: func.func @dynamic_break_dispatch(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32
+func.func @dynamic_break_dispatch(%choose_outer: i1, %init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c2, %c1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: %[[INNER_RESULT:.*]] = scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+    %inner_result = test.breakable_loop iter_args(%inner = %outer) : i32 -> i32 {
+      // CHECK: %[[DEPTH_ONE:.*]] = arith.constant 1 : index
+      // CHECK: %[[IS_ONE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_ONE]] : index
+      // CHECK: scf.if %[[IS_ONE]] {
+      // CHECK:   scf.break [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+      // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+      // CHECK: scf.if %[[IS_TWO]] {
+      // CHECK:   scf.break [%[[OUTER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: scf.break [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      test.dynamic_break %depth %inner : i32
+    }
+    test.dynamic_break %c1 %inner_result : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// The same dispatch shape is required for dynamic continue, except each arm
+// targets the next iteration of the selected loop.
+// CHECK-LABEL: func.func @dynamic_continue_dispatch(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32
+func.func @dynamic_continue_dispatch(%choose_outer: i1, %init: i32) -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c2, %c1 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+    test.breakable_loop iter_args(%inner = %outer) : i32 {
+      // CHECK: %[[DEPTH_ONE:.*]] = arith.constant 1 : index
+      // CHECK: %[[IS_ONE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_ONE]] : index
+      // CHECK: scf.if %[[IS_ONE]] {
+      // CHECK:   scf.continue [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+      // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+      // CHECK: scf.if %[[IS_TWO]] {
+      // CHECK:   scf.continue [%[[OUTER_TOKEN]]] %[[INNER_ARG]] : i32
+      // CHECK: scf.continue [%[[INNER_TOKEN]]] %[[INNER_ARG]] : i32
+      test.dynamic_continue %depth %inner : i32
+    }
+    test.dynamic_break %c1 %outer : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
+
+// -----
+
+// The innermost loop carries f32, so it is not compatible with the i32 break
+// payload and must be filtered out. The remaining compatible targets are still
+// addressed by their original depths: 2 for the middle loop and 3 for the outer
+// loop.
+// CHECK-LABEL: func.func @dynamic_break_dispatch_filtered_depths(
+// CHECK-SAME: %[[CHOOSE:.*]]: i1, %[[INIT:.*]]: i32, %[[F_INIT:.*]]: f32
+func.func @dynamic_break_dispatch_filtered_depths(%choose_outer: i1,
+                                                  %init: i32, %inner_init: f32)
+    -> i32 {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %c3 = arith.constant 3 : index
+  // CHECK: %[[DEPTH:.*]] = arith.select %[[CHOOSE]],
+  %depth = arith.select %choose_outer, %c3, %c2 : index
+  // CHECK: %[[RESULT:.*]] = scf.loop token(%[[OUTER_TOKEN:.*]]) iter_args(%[[OUTER_ARG:.*]] =
+  %result = test.breakable_loop iter_args(%outer = %init) : i32 -> i32 {
+    // CHECK: %[[MIDDLE_RESULT:.*]] = scf.loop token(%[[MIDDLE_TOKEN:.*]]) iter_args(%[[MIDDLE_ARG:.*]] =
+    %middle_result = test.breakable_loop iter_args(%middle = %outer) : i32 -> i32 {
+      // CHECK: scf.loop token(%[[INNER_TOKEN:.*]]) iter_args(%[[INNER_ARG:.*]] =
+      test.breakable_loop iter_args(%inner = %inner_init) : f32 {
+        // CHECK: %[[DEPTH_TWO:.*]] = arith.constant 2 : index
+        // CHECK: %[[IS_TWO:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_TWO]] : index
+        // CHECK: scf.if %[[IS_TWO]] {
+        // CHECK:   scf.break [%[[MIDDLE_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        // CHECK: %[[DEPTH_THREE:.*]] = arith.constant 3 : index
+        // CHECK: %[[IS_THREE:.*]] = arith.cmpi eq, %[[DEPTH]], %[[DEPTH_THREE]] : index
+        // CHECK: scf.if %[[IS_THREE]] {
+        // CHECK:   scf.break [%[[OUTER_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        // CHECK: scf.break [%[[MIDDLE_TOKEN]]] %[[MIDDLE_ARG]] : i32
+        test.dynamic_break %depth %middle : i32
+      }
+      test.dynamic_break %c1 %middle : i32
+    }
+    test.dynamic_break %c1 %middle_result : i32
+  }
+  // CHECK: return %[[RESULT]] : i32
+  return %result : i32
+}
diff --git a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
index c6470da1b7852..b38e6b1455c23 100644
--- a/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
+++ b/mlir/test/lib/Dialect/Test/TestOpDefs.cpp
@@ -10,6 +10,7 @@
 #include "TestOps.h"
 #include "mlir/Dialect/Bufferization/IR/Bufferization.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
+#include "mlir/IR/Matchers.h"
 #include "mlir/IR/Verifier.h"
 #include "mlir/Interfaces/FunctionImplementation.h"
 #include "mlir/Interfaces/MemorySlotInterfaces.h"
@@ -827,6 +828,345 @@ void AnyCondOp::getRegionInvocationBounds(
   invocationBounds.emplace_back(1, 1);
 }
 
+//===----------------------------------------------------------------------===//
+// TestBreakableLoopOp / TestDynamicBreakOp / TestDynamicContinueOp
+//===----------------------------------------------------------------------===//
+
+static std::optional<int64_t> getStaticDepth(Value depth) {
+  APInt value;
+  if (!matchPattern(depth, m_ConstantInt(&value)))
+    return std::nullopt;
+  return value.getSExtValue();
+}
+
+static SmallVector<TestBreakableLoopOp>
+getReachableBreakableLoops(Operation *terminator) {
+  SmallVector<TestBreakableLoopOp> loops;
+  for (Operation *currentOp = terminator->getParentOp(); currentOp;
+       currentOp = currentOp->getParentOp()) {
+    if (auto loopOp = dyn_cast<TestBreakableLoopOp>(currentOp))
+      loops.push_back(loopOp);
+    if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      break;
+  }
+  return loops;
+}
+
+static Operation *getPropagationBlockingParent(Operation *terminator) {
+  for (Operation *currentOp = terminator->getParentOp(); currentOp;
+       currentOp = currentOp->getParentOp()) {
+    if (currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+      continue;
+    for (Operation *parentOp = currentOp->getParentOp(); parentOp;
+         parentOp = parentOp->getParentOp())
+      if (isa<TestBreakableLoopOp>(parentOp))
+        return currentOp;
+    return nullptr;
+  }
+  return nullptr;
+}
+
+static SmallVector<TestBreakableLoopOp>
+getResolvedBreakableLoops(Operation *terminator, Value depth) {
+  SmallVector<TestBreakableLoopOp> loops =
+      getReachableBreakableLoops(terminator);
+  std::optional<int64_t> staticDepth = getStaticDepth(depth);
+  if (!staticDepth)
+    return loops;
+  if (*staticDepth <= 0 || static_cast<uint64_t>(*staticDepth) > loops.size())
+    return {};
+  return {loops[*staticDepth - 1]};
+}
+
+static OperandRange getTerminatorPayload(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getArgs();
+  return cast<TestDynamicContinueOp>(terminator).getArgs();
+}
+
+static bool isContinueTerminator(Operation *terminator) {
+  if (isa<TestDynamicContinueOp>(terminator))
+    return true;
+  assert(isa<TestDynamicBreakOp>(terminator) &&
+         "expected test.dynamic_break or test.dynamic_continue");
+  return false;
+}
+
+static ValueRange getTargetExpectedValues(TestBreakableLoopOp target,
+                                          bool isContinue) {
+  return isContinue ? ValueRange(target.getRegionIterArgs())
+                    : ValueRange(target.getResults());
+}
+
+static bool isTerminatorPayloadCompatible(OperandRange args,
+                                          ValueRange expectedValues) {
+  if (args.size() != expectedValues.size())
+    return false;
+  return llvm::all_of(llvm::zip(args, expectedValues), [](auto values) {
+    return std::get<0>(values).getType() == std::get<1>(values).getType();
+  });
+}
+
+static bool isPotentialTerminatorTarget(Operation *terminator,
+                                        OperandRange args,
+                                        TestBreakableLoopOp target,
+                                        bool isContinue) {
+  return target.acceptsTerminator(terminator) &&
+         isTerminatorPayloadCompatible(
+             args, getTargetExpectedValues(target, isContinue));
+}
+
+static SmallVector<Operation *>
+getPotentialBreakableLoopTargets(Operation *terminator, Value depth) {
+  SmallVector<Operation *> targets;
+  OperandRange args = getTerminatorPayload(terminator);
+  bool isContinue = isContinueTerminator(terminator);
+  for (TestBreakableLoopOp loopOp :
+       getResolvedBreakableLoops(terminator, depth)) {
+    if (!isPotentialTerminatorTarget(terminator, args, loopOp, isContinue))
+      continue;
+    targets.push_back(loopOp.getOperation());
+  }
+  return targets;
+}
+
+static void
+appendTestBreakableLoopSuccessor(TestBreakableLoopOp loopOp, bool isContinue,
+                                 SmallVectorImpl<RegionSuccessor> &regions) {
+  regions.push_back(isContinue ? RegionSuccessor(&loopOp.getBody())
+                               : RegionSuccessor(loopOp.getOperation()));
+}
+
+static LogicalResult verifyTerminatorPayload(Operation *terminator,
+                                             OperandRange args,
+                                             ValueRange expectedValues,
+                                             TestBreakableLoopOp target,
+                                             StringRef targetKind) {
+  if (args.size() != expectedValues.size())
+    return terminator->emitOpError()
+           << "has " << args.size() << " payload operands, but target "
+           << targetKind << " expects " << expectedValues.size();
+
+  for (auto [index, argAndExpected] :
+       llvm::enumerate(llvm::zip(args, expectedValues))) {
+    Value arg = std::get<0>(argAndExpected);
+    Value expected = std::get<1>(argAndExpected);
+    if (arg.getType() == expected.getType())
+      continue;
+    return terminator->emitOpError()
+           << "payload operand #" << index << " has type " << arg.getType()
+           << ", but target " << targetKind << " of "
+           << OpWithFlags(target, OpPrintingFlags().skipRegions())
+           << " expects " << expected.getType();
+  }
+  return success();
+}
+
+static LogicalResult verifyDynamicTerminator(Operation *terminator, Value depth,
+                                             OperandRange args,
+                                             bool isContinue) {
+  SmallVector<TestBreakableLoopOp> reachableLoops =
+      getReachableBreakableLoops(terminator);
+  Operation *blockingOp = getPropagationBlockingParent(terminator);
+  if (reachableLoops.empty()) {
+    if (blockingOp)
+      return terminator->emitOpError()
+             << "depth target crosses an op that does not have the "
+                "PropagateControlFlowBreak trait: "
+             << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+    return terminator->emitOpError()
+           << "must be nested inside a reachable test.breakable_loop";
+  }
+
+  SmallVector<TestBreakableLoopOp> targets = reachableLoops;
+  if (std::optional<int64_t> staticDepth = getStaticDepth(depth)) {
+    if (*staticDepth <= 0)
+      return terminator->emitOpError() << "depth must be positive";
+    if (static_cast<uint64_t>(*staticDepth) > reachableLoops.size()) {
+      if (blockingOp)
+        return terminator->emitOpError()
+               << "depth target crosses an op that does not have the "
+                  "PropagateControlFlowBreak trait: "
+               << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+      return terminator->emitOpError()
+             << "constant depth exceeds the number of reachable "
+                "test.breakable_loop operations";
+    }
+    targets.clear();
+    targets.push_back(reachableLoops[*staticDepth - 1]);
+  } else if (blockingOp) {
+    return terminator->emitOpError()
+           << "dynamic depth may target across an op that does not have the "
+              "PropagateControlFlowBreak trait: "
+           << OpWithFlags(blockingOp, OpPrintingFlags().skipRegions());
+  }
+
+  if (!getStaticDepth(depth)) {
+    for (TestBreakableLoopOp target : targets)
+      if (isPotentialTerminatorTarget(terminator, args, target, isContinue))
+        return success();
+    return terminator->emitOpError()
+           << "dynamic depth has no compatible test.breakable_loop target";
+  }
+
+  for (TestBreakableLoopOp target : targets) {
+    if (!target.acceptsTerminator(terminator))
+      return target.emitOpError("does not accept terminator: ")
+             << OpWithFlags(terminator, OpPrintingFlags().skipRegions());
+    ValueRange expectedValues = getTargetExpectedValues(target, isContinue);
+    if (failed(verifyTerminatorPayload(terminator, args, expectedValues, target,
+                                       isContinue ? "loop-carried values"
+                                                  : "results")))
+      return failure();
+  }
+  return success();
+}
+
+void TestBreakableLoopOp::print(OpAsmPrinter &p) {
+  p << " ";
+  if (!getInitArgs().empty()) {
+    p << "iter_args(";
+    llvm::interleaveComma(
+        llvm::zip(getRegionIterArgs(), getInitArgs()), p,
+        [&](auto it) { p << std::get<0>(it) << " = " << std::get<1>(it); });
+    p << ") : " << getInitArgs().getTypes() << " ";
+  }
+  if (!getResultTypes().empty())
+    p << "-> " << getResultTypes() << " ";
+  p.printRegion(getBody(), /*printEntryBlockArgs=*/false,
+                /*printBlockTerminators=*/true);
+  p.printOptionalAttrDict((*this)->getAttrs());
+}
+
+ParseResult TestBreakableLoopOp::parse(OpAsmParser &parser,
+                                       OperationState &result) {
+  SmallVector<OpAsmParser::Argument, 4> regionArgs;
+  SmallVector<OpAsmParser::Argument, 4> iterRegionArgs;
+  SmallVector<OpAsmParser::UnresolvedOperand, 4> iterOperands;
+  SmallVector<Type, 4> iterTypes;
+
+  if (failed(parser.parseOptionalKeyword("iter_args"))) {
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+  } else {
+    if (parser.parseAssignmentList(iterRegionArgs, iterOperands) ||
+        parser.parseColon() || parser.parseTypeList(iterTypes))
+      return failure();
+    if (iterRegionArgs.size() != iterTypes.size())
+      return parser.emitError(parser.getCurrentLocation(),
+                              "found different number of iter_args and types");
+    if (succeeded(parser.parseOptionalArrow()))
+      if (parser.parseTypeList(result.types))
+        return failure();
+    for (auto [regionArg, type] : llvm::zip_equal(iterRegionArgs, iterTypes))
+      regionArg.type = type;
+    llvm::append_range(regionArgs, iterRegionArgs);
+  }
+
+  Region *body = result.addRegion();
+  if (parser.parseRegion(*body, regionArgs) ||
+      parser.parseOptionalAttrDict(result.attributes))
+    return failure();
+
+  return parser.resolveOperands(iterOperands, iterTypes, parser.getNameLoc(),
+                                result.operands);
+}
+
+LogicalResult TestBreakableLoopOp::verifyRegions() {
+  if (getBody().empty())
+    return emitOpError("region cannot be empty");
+  if (getBody().front().getNumArguments() != getNumOperands())
+    return emitOpError("expected the region to have one argument per "
+                       "loop-carried value (")
+           << getNumOperands() << " expected, but got "
+           << getBody().front().getNumArguments() << ")";
+  for (auto [index, argAndOperand] :
+       llvm::enumerate(llvm::zip(getRegionIterArgs(), getOperands()))) {
+    Type argType = std::get<0>(argAndOperand).getType();
+    Type operandType = std::get<1>(argAndOperand).getType();
+    if (argType != operandType)
+      return emitOpError() << "types mismatch between " << index
+                           << "th iter operand (" << operandType
+                           << ") and defined region argument (" << argType
+                           << ")";
+  }
+  return success();
+}
+
+void TestBreakableLoopOp::getSuccessorRegions(
+    RegionBranchPoint point, SmallVectorImpl<RegionSuccessor> &regions) {
+  if (point.isParent()) {
+    regions.push_back(RegionSuccessor(&getBody()));
+    return;
+  }
+
+  Operation *terminator = point.getTerminatorPredecessorOrNull();
+  Value depth;
+  bool isContinue = false;
+  if (auto continueOp = dyn_cast<TestDynamicContinueOp>(terminator)) {
+    depth = continueOp.getDepth();
+    isContinue = true;
+  } else if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator)) {
+    depth = breakOp.getDepth();
+  } else {
+    llvm_unreachable("expected test.dynamic_break or test.dynamic_continue");
+  }
+
+  bool isImmediateBranch = terminator->getParentOp() == getOperation();
+  for (Operation *target :
+       getPotentialBreakableLoopTargets(terminator, depth)) {
+    if (target == getOperation()) {
+      appendTestBreakableLoopSuccessor(*this, isContinue, regions);
+      continue;
+    }
+    if (!isImmediateBranch || !target->isProperAncestor(getOperation()))
+      continue;
+    appendTestBreakableLoopSuccessor(cast<TestBreakableLoopOp>(target),
+                                     isContinue, regions);
+  }
+}
+
+OperandRange
+TestBreakableLoopOp::getEntrySuccessorOperands(RegionSuccessor successor) {
+  return getInitArgs();
+}
+
+ValueRange TestBreakableLoopOp::getSuccessorInputs(RegionSuccessor successor) {
+  return successor.isOperation() ? ValueRange(getResults())
+                                 : ValueRange(getRegionIterArgs());
+}
+
+LogicalResult TestDynamicBreakOp::verify() {
+  return verifyDynamicTerminator(getOperation(), getDepth(), getArgs(),
+                                 /*isContinue=*/false);
+}
+
+SmallVector<Operation *> TestDynamicBreakOp::getPotentialTargets() {
+  return getPotentialBreakableLoopTargets(getOperation(), getDepth());
+}
+
+MutableOperandRange
+TestDynamicBreakOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
+LogicalResult TestDynamicContinueOp::verify() {
+  return verifyDynamicTerminator(getOperation(), getDepth(), getArgs(),
+                                 /*isContinue=*/true);
+}
+
+SmallVector<Operation *> TestDynamicContinueOp::getPotentialTargets() {
+  return getPotentialBreakableLoopTargets(getOperation(), getDepth());
+}
+
+MutableOperandRange
+TestDynamicContinueOp::getMutableSuccessorOperands(RegionSuccessor point) {
+  return MutableOperandRange(getOperation(), /*start=*/1,
+                             /*length=*/getOperation()->getNumOperands() - 1);
+}
+
 //===----------------------------------------------------------------------===//
 // SingleBlockImplicitTerminatorOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/lib/Dialect/Test/TestOps.td b/mlir/test/lib/Dialect/Test/TestOps.td
index 9d7e21cabda6f..1439cf405d28a 100644
--- a/mlir/test/lib/Dialect/Test/TestOps.td
+++ b/mlir/test/lib/Dialect/Test/TestOps.td
@@ -2850,6 +2850,73 @@ def LoopBlockTerminatorOp : TEST_Op<"loop_block_term",
   }];
 }
 
+def TestBreakableLoopOp : TEST_Op<"breakable_loop", [
+    AutomaticAllocationScope,
+    RecursiveMemoryEffects,
+    PropagateControlFlowBreak,
+    SingleBlock,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface,
+        ["getEntrySuccessorOperands", "getSuccessorInputs"]>,
+    HasBreakingControlFlowOpInterface,
+    HasNestedTerminator<["TestDynamicContinueOp", "TestDynamicBreakOp"]>
+  ]> {
+  let summary = "test loop with dynamically addressed break/continue";
+  let description = [{
+    Test dialect loop operation that models breaking control flow with
+    dynamically selected targets. Its terminators use an SSA index value to
+    count enclosing `test.breakable_loop` operations. A depth of one targets the
+    immediately enclosing test loop, a depth of two targets the next enclosing
+    test loop, and so on. A constant depth must identify a reachable target with
+    compatible payload types. A dynamic depth may evaluate to any reachable depth
+    with compatible payload types; evaluating to zero, to an incompatible target,
+    or to a depth greater than the number of reachable loops is undefined
+    behavior.
+  }];
+
+  let arguments = (ins Variadic<AnyType>:$initArgs);
+  let results = (outs Variadic<AnyType>:$results);
+  let regions = (region SizedRegion<1>:$body);
+
+  let extraClassDeclaration = [{
+    ::mlir::Block::BlockArgListType getRegionIterArgs() {
+      return getBody().getArguments();
+    }
+  }];
+
+  let hasCustomAssemblyFormat = 1;
+  let hasRegionVerifier = 1;
+}
+
+def TestDynamicBreakOp : TEST_Op<"dynamic_break", [
+    Terminator, RegionExitTerminatorOpInterface,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>
+  ]> {
+  let summary = "dynamically targeted break from test.breakable_loop";
+  let arguments = (ins Index:$depth, Variadic<AnyType>:$args);
+  let assemblyFormat = "$depth ($args^ `:` type($args))? attr-dict";
+  let extraClassDeclaration = [{
+    ::llvm::SmallVector<::mlir::Operation *>
+    getPotentialTargets();
+  }];
+  let hasVerifier = 1;
+}
+
+def TestDynamicContinueOp : TEST_Op<"dynamic_continue", [
+    Terminator, RegionExitTerminatorOpInterface,
+    DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface,
+      ["getMutableSuccessorOperands"]>
+  ]> {
+  let summary = "dynamically targeted continue of test.breakable_loop";
+  let arguments = (ins Index:$depth, Variadic<AnyType>:$args);
+  let assemblyFormat = "$depth ($args^ `:` type($args))? attr-dict";
+  let extraClassDeclaration = [{
+    ::llvm::SmallVector<::mlir::Operation *>
+    getPotentialTargets();
+  }];
+  let hasVerifier = 1;
+}
+
 def TestNoTerminatorOp : TEST_Op<"switch_with_no_break", [
     NoTerminator,
     DeclareOpInterfaceMethods<RegionBranchOpInterface>
diff --git a/mlir/test/lib/Dialect/Test/TestPatterns.cpp b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
index 552a1a473c9fd..1ace1d1101df0 100644
--- a/mlir/test/lib/Dialect/Test/TestPatterns.cpp
+++ b/mlir/test/lib/Dialect/Test/TestPatterns.cpp
@@ -14,9 +14,11 @@
 #include "mlir/Dialect/ControlFlow/Transforms/StructuralTypeConversions.h"
 #include "mlir/Dialect/Func/IR/FuncOps.h"
 #include "mlir/Dialect/Func/Transforms/FuncConversions.h"
+#include "mlir/Dialect/SCF/IR/SCF.h"
 #include "mlir/Dialect/SCF/Transforms/Patterns.h"
 #include "mlir/Dialect/Tensor/IR/Tensor.h"
 #include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinTypes.h"
 #include "mlir/IR/Matchers.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/IR/Visitors.h"
@@ -25,6 +27,7 @@
 #include "mlir/Transforms/FoldUtils.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "mlir/Transforms/WalkPatternRewriteDriver.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/ScopeExit.h"
 #include <cstdint>
 
@@ -2490,6 +2493,214 @@ struct TestFoldTypeConvertingOp
       signalPassFailure();
   }
 };
+
+class TestConvertBreakableLoopToSCFPass
+    : public PassWrapper<TestConvertBreakableLoopToSCFPass, OperationPass<>> {
+public:
+  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(
+      TestConvertBreakableLoopToSCFPass)
+
+  StringRef getArgument() const final {
+    return "test-convert-breakable-loop-to-scf";
+  }
+  StringRef getDescription() const final {
+    return "Convert test.breakable_loop operations to scf.loop operations";
+  }
+
+  void runOnOperation() override;
+
+  struct TargetInfo {
+    Operation *target;
+    int64_t depth;
+  };
+  using TargetMap = llvm::DenseMap<Operation *, SmallVector<TargetInfo>>;
+  using TokenMap = llvm::DenseMap<Operation *, Value>;
+
+private:
+  LogicalResult rewriteLoop(TestBreakableLoopOp loopOp,
+                            const TargetMap &terminatorTargets,
+                            TokenMap &loopTokens);
+  LogicalResult rewriteTerminator(Operation *terminator,
+                                  const TargetMap &terminatorTargets,
+                                  const TokenMap &loopTokens,
+                                  PatternRewriter &rewriter);
+};
+
+static void appendTerminatorTargets(
+    Operation *terminator,
+    SmallVectorImpl<TestConvertBreakableLoopToSCFPass::TargetInfo> &targets,
+    SmallVector<Operation *> potentialTargets) {
+  for (Operation *targetOp : potentialTargets) {
+    int64_t depth = 0;
+    for (Operation *currentOp = terminator->getParentOp(); currentOp;
+         currentOp = currentOp->getParentOp()) {
+      if (isa<TestBreakableLoopOp>(currentOp)) {
+        ++depth;
+        if (currentOp == targetOp) {
+          targets.push_back({targetOp, depth});
+          break;
+        }
+      }
+      if (!currentOp->mightHaveTrait<OpTrait::PropagateControlFlowBreak>())
+        break;
+    }
+  }
+}
+
+static bool isBreakableLoopTerminator(Operation *op) {
+  return isa<TestDynamicBreakOp, TestDynamicContinueOp>(op);
+}
+
+static OperandRange getTerminatorPayload(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getArgs();
+  return cast<TestDynamicContinueOp>(terminator).getArgs();
+}
+
+static Value getTerminatorDepth(Operation *terminator) {
+  if (auto breakOp = dyn_cast<TestDynamicBreakOp>(terminator))
+    return breakOp.getDepth();
+  return cast<TestDynamicContinueOp>(terminator).getDepth();
+}
+
+static void createSCFTerminator(PatternRewriter &rewriter, Location loc,
+                                Operation *terminator, Value targetToken,
+                                ValueRange args) {
+  if (isa<TestDynamicBreakOp>(terminator)) {
+    scf::BreakOp::create(rewriter, loc, targetToken, args);
+    return;
+  }
+  scf::ContinueOp::create(rewriter, loc, targetToken, args);
+}
+
+static std::optional<Value>
+lookupToken(Operation *terminator, Operation *target,
+            const TestConvertBreakableLoopToSCFPass::TokenMap &loopTokens) {
+  auto it = loopTokens.find(target);
+  if (it != loopTokens.end())
+    return it->second;
+  terminator->emitOpError()
+      << "cannot convert because target loop has not been converted";
+  return std::nullopt;
+}
+
+LogicalResult TestConvertBreakableLoopToSCFPass::rewriteTerminator(
+    Operation *terminator, const TargetMap &terminatorTargets,
+    const TokenMap &loopTokens, PatternRewriter &rewriter) {
+  auto targetIt = terminatorTargets.find(terminator);
+  if (targetIt == terminatorTargets.end() || targetIt->second.empty())
+    return terminator->emitOpError()
+           << "cannot convert terminator without a target loop";
+
+  ArrayRef<TargetInfo> targets = targetIt->second;
+  Location loc = terminator->getLoc();
+  SmallVector<Value> args(getTerminatorPayload(terminator));
+
+  if (targets.size() == 1) {
+    std::optional<Value> token =
+        lookupToken(terminator, targets.front().target, loopTokens);
+    if (!token)
+      return failure();
+    rewriter.setInsertionPoint(terminator);
+    createSCFTerminator(rewriter, loc, terminator, *token, args);
+    rewriter.eraseOp(terminator);
+    return success();
+  }
+
+  rewriter.setInsertionPoint(terminator);
+  Value depth = getTerminatorDepth(terminator);
+  for (TargetInfo target : targets) {
+    std::optional<Value> token =
+        lookupToken(terminator, target.target, loopTokens);
+    if (!token)
+      return failure();
+
+    Value depthValue =
+        arith::ConstantIndexOp::create(rewriter, loc, target.depth);
+    Value isTarget = arith::CmpIOp::create(
+        rewriter, loc, arith::CmpIPredicate::eq, depth, depthValue);
+    auto ifOp = scf::IfOp::create(rewriter, loc, TypeRange{}, isTarget,
+                                  /*addThenBlock=*/true,
+                                  /*addElseBlock=*/false);
+    {
+      OpBuilder::InsertionGuard guard(rewriter);
+      rewriter.setInsertionPointToStart(ifOp.thenBlock());
+      createSCFTerminator(rewriter, loc, terminator, *token, args);
+    }
+    rewriter.setInsertionPointAfter(ifOp);
+  }
+
+  // Dynamic depths are verified to be able to target any listed loop. If the
+  // value falls outside that set, selects an incompatible target, or otherwise
+  // has undefined behavior, the fallback may choose any valid target. Use the
+  // first compatible target to keep the generated IR deterministic.
+  std::optional<Value> fallbackToken =
+      lookupToken(terminator, targets.front().target, loopTokens);
+  if (!fallbackToken)
+    return failure();
+  createSCFTerminator(rewriter, loc, terminator, *fallbackToken, args);
+  rewriter.eraseOp(terminator);
+  return success();
+}
+
+LogicalResult TestConvertBreakableLoopToSCFPass::rewriteLoop(
+    TestBreakableLoopOp loopOp, const TargetMap &terminatorTargets,
+    TokenMap &loopTokens) {
+  PatternRewriter rewriter(loopOp.getContext());
+  rewriter.setInsertionPoint(loopOp);
+
+  auto scfLoop = scf::LoopOp::create(
+      rewriter, loopOp.getLoc(), loopOp.getResultTypes(), loopOp.getInitArgs());
+  scfLoop->setAttrs(loopOp->getAttrs());
+  scfLoop.getRegion().takeBody(loopOp.getBody());
+  scfLoop.getBody()->insertArgument(
+      /*index=*/0u, TokenType::get(loopOp.getContext()), loopOp.getLoc());
+  loopTokens[loopOp.getOperation()] = scfLoop.getControlToken();
+
+  SmallVector<Operation *> terminators;
+  scfLoop.getRegion().walk([&](Operation *op) {
+    if (!isBreakableLoopTerminator(op))
+      return;
+    if (op->getParentOfType<TestBreakableLoopOp>())
+      return;
+    terminators.push_back(op);
+  });
+
+  for (Operation *terminator : terminators)
+    if (failed(rewriteTerminator(terminator, terminatorTargets, loopTokens,
+                                 rewriter)))
+      return failure();
+
+  rewriter.replaceOp(loopOp, scfLoop.getResults());
+  return success();
+}
+
+void TestConvertBreakableLoopToSCFPass::runOnOperation() {
+  SmallVector<TestBreakableLoopOp> loops;
+  TargetMap terminatorTargets;
+  getOperation()->walk<WalkOrder::PreOrder>([&](Operation *op) {
+    if (auto loopOp = dyn_cast<TestBreakableLoopOp>(op)) {
+      loops.push_back(loopOp);
+      return;
+    }
+    if (auto breakOp = dyn_cast<TestDynamicBreakOp>(op)) {
+      appendTerminatorTargets(op, terminatorTargets[op],
+                              breakOp.getPotentialTargets());
+      return;
+    }
+    if (auto continueOp = dyn_cast<TestDynamicContinueOp>(op))
+      appendTerminatorTargets(op, terminatorTargets[op],
+                              continueOp.getPotentialTargets());
+  });
+
+  TokenMap loopTokens;
+  for (TestBreakableLoopOp loopOp : loops) {
+    if (failed(rewriteLoop(loopOp, terminatorTargets, loopTokens))) {
+      signalPassFailure();
+      return;
+    }
+  }
+}
 } // namespace
 
 //===----------------------------------------------------------------------===//
@@ -2522,6 +2733,8 @@ void registerPatternsTestPass() {
   PassRegistration<TestSelectiveReplacementPatternDriver>();
 
   PassRegistration<TestFoldTypeConvertingOp>();
+
+  PassRegistration<TestConvertBreakableLoopToSCFPass>();
 }
 } // namespace test
 } // namespace mlir
diff --git a/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir b/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir
new file mode 100644
index 0000000000000..d2a9ccfc84650
--- /dev/null
+++ b/mlir/test/mlir-runner/test-convert-breakable-loop-to-scf.mlir
@@ -0,0 +1,145 @@
+// RUN: mlir-opt %s -pass-pipeline="builtin.module(func.func(test-convert-breakable-loop-to-scf,convert-scf-to-cf,canonicalize,convert-arith-to-llvm),convert-func-to-llvm,convert-cf-to-llvm,reconcile-unrealized-casts)" -o %t.mlir
+// RUN: mlir-runner %t.mlir -e immediate_continue -entry-point-result=i32 | FileCheck %s --check-prefix=IMMEDIATE
+// RUN: mlir-runner %t.mlir -e break_with_result -entry-point-result=i32 | FileCheck %s --check-prefix=BREAK
+// RUN: mlir-runner %t.mlir -e constant_outer_break -entry-point-result=i32 | FileCheck %s --check-prefix=OUTER-BREAK
+// RUN: mlir-runner %t.mlir -e constant_outer_continue -entry-point-result=i32 | FileCheck %s --check-prefix=OUTER-CONTINUE
+// RUN: mlir-runner %t.mlir -e dynamic_break_depth_1 -entry-point-result=i32 | FileCheck %s --check-prefix=DYNAMIC-ONE
+// RUN: mlir-runner %t.mlir -e dynamic_break_depth_2 -entry-point-result=i32 | FileCheck %s --check-prefix=DYNAMIC-TWO
+// RUN: mlir-runner %t.mlir -e dynamic_filtered_depth_2 -entry-point-result=i32 | FileCheck %s --check-prefix=FILTERED-TWO
+// RUN: mlir-runner %t.mlir -e dynamic_filtered_depth_3 -entry-point-result=i32 | FileCheck %s --check-prefix=FILTERED-THREE
+// XFAIL: system-aix
+
+func.func @immediate_continue() -> i32 {
+  %depth = arith.constant 1 : index
+  %c0 = arith.constant 0 : i32
+  %c1 = arith.constant 1 : i32
+  %c4 = arith.constant 4 : i32
+  %result = test.breakable_loop iter_args(%i = %c0) : i32 -> i32 {
+    %done = arith.cmpi eq, %i, %c4 : i32
+    scf.if %done {
+      test.dynamic_break %depth %i : i32
+    }
+    %next = arith.addi %i, %c1 : i32
+    test.dynamic_continue %depth %next : i32
+  }
+  return %result : i32
+}
+// IMMEDIATE: 4
+
+func.func @break_with_result() -> i32 {
+  %depth = arith.constant 1 : index
+  %value = arith.constant 17 : i32
+  %result = test.breakable_loop -> i32 {
+    test.dynamic_break %depth %value : i32
+  }
+  return %result : i32
+}
+// BREAK: 17
+
+func.func @constant_outer_break() -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %outer_value = arith.constant 42 : i32
+  %fallback = arith.constant 13 : i32
+  %result = test.breakable_loop -> i32 {
+    test.breakable_loop {
+      test.dynamic_break %outer_depth %outer_value : i32
+    }
+    test.dynamic_break %inner_depth %fallback : i32
+  }
+  return %result : i32
+}
+// OUTER-BREAK: 42
+
+func.func @constant_outer_continue() -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %c0 = arith.constant 0 : i32
+  %c1 = arith.constant 1 : i32
+  %c3 = arith.constant 3 : i32
+  %result = test.breakable_loop iter_args(%i = %c0) : i32 -> i32 {
+    %done = arith.cmpi eq, %i, %c3 : i32
+    scf.if %done {
+      test.dynamic_break %inner_depth %i : i32
+    }
+    test.breakable_loop {
+      %next = arith.addi %i, %c1 : i32
+      test.dynamic_continue %outer_depth %next : i32
+    }
+    test.dynamic_break %inner_depth %i : i32
+  }
+  return %result : i32
+}
+// OUTER-CONTINUE: 3
+
+// Distinguish the two compatible dynamic break targets at runtime. Depth 1
+// breaks the inner loop, then the outer loop adds one; depth 2 skips that outer
+// post-inner add by breaking directly to the outer loop result.
+func.func @dynamic_break(%choose_outer: i1) -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %outer_depth = arith.constant 2 : index
+  %depth = arith.select %choose_outer, %outer_depth, %inner_depth : index
+  %value = arith.constant 7 : i32
+  %one = arith.constant 1 : i32
+  %result = test.breakable_loop -> i32 {
+    %inner_result = test.breakable_loop -> i32 {
+      test.dynamic_break %depth %value : i32
+    }
+    %after_inner = arith.addi %inner_result, %one : i32
+    test.dynamic_break %inner_depth %after_inner : i32
+  }
+  return %result : i32
+}
+
+func.func @dynamic_break_depth_1() -> i32 {
+  %choose_outer = arith.constant false
+  %result = func.call @dynamic_break(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// DYNAMIC-ONE: 8
+
+func.func @dynamic_break_depth_2() -> i32 {
+  %choose_outer = arith.constant true
+  %result = func.call @dynamic_break(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// DYNAMIC-TWO: 7
+
+// The innermost f32 loop is incompatible with the i32 break payload and is
+// filtered out. Depth 2 therefore targets the middle loop and observes the
+// outer post-middle add, while depth 3 targets the outer loop directly.
+func.func @dynamic_break_filtered(%choose_outer: i1) -> i32 {
+  %inner_depth = arith.constant 1 : index
+  %middle_depth = arith.constant 2 : index
+  %outer_depth = arith.constant 3 : index
+  %depth = arith.select %choose_outer, %outer_depth, %middle_depth : index
+  %value = arith.constant 5 : i32
+  %one = arith.constant 1 : i32
+  %f0 = arith.constant 0.000000e+00 : f32
+  %result = test.breakable_loop -> i32 {
+    %middle_result = test.breakable_loop -> i32 {
+      test.breakable_loop iter_args(%inner = %f0) : f32 {
+        test.dynamic_break %depth %value : i32
+      }
+      %after_inner = arith.addi %value, %one : i32
+      test.dynamic_break %inner_depth %after_inner : i32
+    }
+    %after_middle = arith.addi %middle_result, %one : i32
+    test.dynamic_break %inner_depth %after_middle : i32
+  }
+  return %result : i32
+}
+
+func.func @dynamic_filtered_depth_2() -> i32 {
+  %choose_outer = arith.constant false
+  %result = func.call @dynamic_break_filtered(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// FILTERED-TWO: 6
+
+func.func @dynamic_filtered_depth_3() -> i32 {
+  %choose_outer = arith.constant true
+  %result = func.call @dynamic_break_filtered(%choose_outer) : (i1) -> i32
+  return %result : i32
+}
+// FILTERED-THREE: 5


From llvmlistbot at llvm.org  Wed Jul 22 03:58:48 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Wed, 22 Jul 2026 03:58:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a60a268.7d702ed2.2f6c05.fc82@mx.google.com>


https://github.com/flemairen6 updated https://github.com/llvm/llvm-project/pull/205990

>From f294988d80afcf50d52102adf91aa52dd2e9199e Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 12:48:24 +0900
Subject: [PATCH 1/7] [mlir][wasm] RaiseWasmMLIRPass: Add suport for comparison
 operators

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 139 ++++++
 .../wasm-comparisons-to-arith-cmp.mlir        | 421 ++++++++++++++++++
 .../RaiseWasm/wasm-eqz-to-arith-cmp.mlir      |  28 ++
 3 files changed, 588 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 83bfde7032ef8..1fb4564032421 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,104 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+template <typename SourceOp, typename TargetOp, typename AttrType,
+          typename ValType, ValType flag>
+struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto cmpRes =
+        rewriter
+            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                              AttrType::get(rewriter.getContext(), flag),
+                              adaptor.getLhs(), adaptor.getRhs())
+            .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
+template <typename SourceOp, arith::CmpFPredicate compFlag>
+using FPComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpFOp, arith::CmpFPredicateAttr,
+                           arith::CmpFPredicate, compFlag>;
+
+template <typename SourceOp, arith::CmpIPredicate compFlag>
+using IntComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpIOp, arith::CmpIPredicateAttr,
+                           arith::CmpIPredicate, compFlag>;
+
+using WasmLtSIOpConversion =
+    IntComparisonConversion<LtSIOp, arith::CmpIPredicate::slt>;
+using WasmLeSIOpConversion =
+    IntComparisonConversion<LeSIOp, arith::CmpIPredicate::sle>;
+using WasmGtSIOpConversion =
+    IntComparisonConversion<GtSIOp, arith::CmpIPredicate::sgt>;
+using WasmGeSIOpConversion =
+    IntComparisonConversion<GeSIOp, arith::CmpIPredicate::sge>;
+using WasmLtUIOpConversion =
+    IntComparisonConversion<LtUIOp, arith::CmpIPredicate::ult>;
+using WasmLeUIOpConversion =
+    IntComparisonConversion<LeUIOp, arith::CmpIPredicate::ule>;
+using WasmGtUIOpConversion =
+    IntComparisonConversion<GtUIOp, arith::CmpIPredicate::ugt>;
+using WasmGeUIOpConversion =
+    IntComparisonConversion<GeUIOp, arith::CmpIPredicate::uge>;
+using WasmLtOpConversion =
+    FPComparisonConversion<LtOp, arith::CmpFPredicate::OLT>;
+using WasmLeOpConversion =
+    FPComparisonConversion<LeOp, arith::CmpFPredicate::OLE>;
+using WasmGtOpConversion =
+    FPComparisonConversion<GtOp, arith::CmpFPredicate::OGT>;
+using WasmGeOpConversion =
+    FPComparisonConversion<GeOp, arith::CmpFPredicate::OGE>;
+
+template <typename SourceOp, arith::CmpIPredicate IntFlag,
+          arith::CmpFPredicate FloatFlag>
+struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Value comparisonResult;
+    if (srcOp.getLhs().getType().isInteger())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpIOp>(
+                  srcOp.getLoc(), rewriter.getI1Type(),
+                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                  adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else if (srcOp.getLhs().getType().isFloat())
+      comparisonResult =
+          rewriter
+              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
+                                     arith::CmpFPredicateAttr::get(
+                                         rewriter.getContext(), FloatFlag),
+                                     adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else
+      return rewriter.notifyMatchFailure(
+          srcOp.getLoc(), "Unsupported datatype for comparison OP.");
+
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                comparisonResult);
+    return success();
+  }
+};
+
+using WasmEqOpConversion =
+    IntFpComparisonOpConversion<EqOp, arith::CmpIPredicate::eq,
+                                arith::CmpFPredicate::OEQ>;
+using WasmNeOpConversion =
+    IntFpComparisonOpConversion<NeOp, arith::CmpIPredicate::ne,
+                                arith::CmpFPredicate::ONE>;
+
 struct WasmCallOpConversion : OpConversionPattern<FuncCallOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -148,6 +246,32 @@ struct WasmConstOpConversion : OpConversionPattern<ConstOp> {
   }
 };
 
+struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = eqzOp->getLoc();
+    auto zero =
+        rewriter
+            .create<arith::ConstantOp>(
+                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+            .getResult();
+    auto cmpRes = rewriter
+                      .create<arith::CmpIOp>(
+                          loc, rewriter.getI1Type(),
+                          arith::CmpIPredicateAttr::get(
+                              rewriter.getContext(), arith::CmpIPredicate::eq),
+                          adaptor.getInput(), zero)
+                      .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -429,21 +553,36 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivFPOpConversion,
            WasmDivSIOpConversion,
            WasmDivUIOpConversion,
+           WasmEqOpConversion,
+           WasmEqzOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
            WasmFuncImportOpConversion,
            WasmFuncOpConversion,
+           WasmGeOpConversion,
+           WasmGeSIOpConversion,
+           WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
+           WasmGtOpConversion,
+           WasmGtSIOpConversion,
+           WasmGtUIOpConversion,
+           WasmLeOpConversion,
+           WasmLeSIOpConversion,
+           WasmLeUIOpConversion,
            WasmLocalConversion,
            WasmLocalGetConversion,
            WasmLocalSetConversion,
            WasmLocalTeeConversion,
+           WasmLtOpConversion,
+           WasmLtSIOpConversion,
+           WasmLtUIOpConversion,
            WasmMaxOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
+           WasmNeOpConversion,
            WasmNegOpConversion,
            WasmOrOpConversion,
            WasmPopCntOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..bc952b47520ca
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -0,0 +1,421 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+
+module {
+  wasmssa.func nested @func_lt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+
+  wasmssa.func nested @func_le_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.lt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.le %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.gt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ge %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_lt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.lt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_le_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.le %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_gt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.gt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ge_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.ge %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.eq %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.eq %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.eq %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_eq_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.eq %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+    wasmssa.func nested @func_ne_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ne %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.ne %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ne %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func nested @func_ne_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.ne %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_lt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..083a24c5f36a8
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
@@ -0,0 +1,28 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+module {
+  wasmssa.func @eqz_i32() -> i32 {
+    %0 = wasmssa.const 13 : i32
+    %1 = wasmssa.eqz %0 : i32 -> i32
+    wasmssa.return %1 : i32
+  }
+  wasmssa.func @eqz_i64() -> i32 {
+    %0 = wasmssa.const 13 : i64
+    %1 = wasmssa.eqz %0 : i64 -> i32
+    wasmssa.return %1 : i32
+  }
+}
+
+// CHECK-LABEL:   func.func @eqz_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @eqz_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32

>From 49944e5c6ec9e356338dfaabbd248fb969ce49b8 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:06:50 +0900
Subject: [PATCH 2/7] [mlir][wasm] RaiseWasmMLIRPass: Add rotl and extend
 support

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 87 +++++++++++++++++++
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   | 76 ++++++++++++++++
 .../RaiseWasm/wasm-rotl-to-arith.mlir         | 75 ++++++++++++++++
 .../RaiseWasm/wasm-rotr-to-arith.mlir         | 77 ++++++++++++++++
 4 files changed, 315 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 1fb4564032421..bf958aad5012b 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -124,6 +124,72 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+/// Lower a rotate to a series of bitwise operations. Intended for us
+/// in dialects that do not natively support rotate operations.
+///
+/// Result stays in the wasm dialect. It will then subsequently be lowered to
+/// the target dialect.
+///
+/// The rotate will be lowered to a pattern like so:
+///
+/// (val LHSShiftOp (bits & (width-1))) | (val RHSShiftOp (-bits & (width-1)))
+///
+/// Where LHSShiftOp and RHSShiftOp are shift operations. Concretely,
+///
+/// rotr = (val >> (bits & (width - 1))) | (val << (-bits & (width - 1)))
+/// rotl = (val << (bits & (width - 1))) | (val >> (-bits & (width - 1)))
+///
+/// Using this variant ensures that our rotate is defined in the target dialect.
+///
+/// \p SourceOp - Rotate operation to replace.
+/// \p LHSShiftOp - Shift operation to use on the left-hand side of the OR.
+/// \p RHSShiftOp - Shift operation to use on the right-hand side of the OR.
+template <typename SourceOp, typename LHSShiftOp, typename RHSShiftOp>
+struct RotateOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    const Type ty = srcOp->getResultTypes()[0];
+    const Location loc = srcOp->getLoc();
+    const Value val = adaptor.getVal();
+    const Value bits = adaptor.getBits();
+    const unsigned width = ty.getIntOrFloatBitWidth();
+
+    // Materialize (width - 1) for use in both sides of the expression.
+    auto cstWidthMinusOne =
+        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+
+    // Form the left-hand side of the OR:
+    // (val (lhs shift op) (bits & (width - 1)))
+    auto orLHS = rewriter.create<LHSShiftOp>(
+        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+
+    // Form the right-hand side of the OR:
+    // (val (rhs shift op) (-bits & (width - 1)))
+    auto orRHS = rewriter.create<RHSShiftOp>(
+        loc, val,
+        // (-bits & (width - 1))
+        rewriter.create<AndOp>(
+            loc,
+            // 0 - bits == -bits
+            rewriter.create<SubOp>(
+                loc,
+                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
+                bits),
+            cstWidthMinusOne));
+
+    // OR together the two shifts and replace the rotate with the new
+    // expression.
+    rewriter.replaceOpWithNewOp<OrOp>(srcOp, orLHS, orRHS);
+    return success();
+  }
+};
+
+using WasmRotrOpConversion = RotateOpConversion<RotrOp, ShRUOp, ShLOp>;
+using WasmRotlOpConversion = RotateOpConversion<RotlOp, ShLOp, ShRUOp>;
+
 template <typename SourceOp, typename TargetOp, typename AttrType,
           typename ValType, ValType flag>
 struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
@@ -272,6 +338,24 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   }
 };
 
+struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(ExtendLowBitsSOp extendLowBytesSOp,
+                  ExtendLowBitsSOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
+    auto truncation = rewriter.create<arith::TruncIOp>(
+        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
+        adaptor.getInput());
+    rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
+        extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
+        truncation.getResult());
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -555,6 +639,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivUIOpConversion,
            WasmEqOpConversion,
            WasmEqzOpConversion,
+           WasmExtendLowBitsOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
@@ -591,6 +676,8 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmRemSIOpConversion,
            WasmRemUIOpConversion,
            WasmReturnOpConversion,
+           WasmRotlOpConversion,
+           WasmRotrOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
new file mode 100644
index 0000000000000..90c92001a78d6
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @func_0() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_0() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_s %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_1() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func nested @func_1() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_u %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 8 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 16 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_4() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_4() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 8 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_5() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_5() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 16 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_6() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func nested @func_6() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 32 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
new file mode 100644
index 0000000000000..91e929bd5ff35
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -0,0 +1,75 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotl %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotl_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val << (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i32
+
+// (val >> (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotl_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val << (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i64
+
+// (val >> (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
new file mode 100644
index 0000000000000..77e301c3221e3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotr %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotr_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val >> (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i32
+
+// (val << (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+// CHECK:         }
+wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotr_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val >> (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i64
+
+// (val << (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+// CHECK:         }
+wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}

>From 6431a7494a5edc89de997962d92b1803d6aed43f Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:24:21 +0900
Subject: [PATCH 3/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for control
 flow operations in functions

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 184 ++++++++-
 .../RaiseWasm/wasm-blocks-to-cf.mlir          | 366 ++++++++++++++++++
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 206 ++++++++++
 3 files changed, 755 insertions(+), 1 deletion(-)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index bf958aad5012b..d9dcb756790f8 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -371,6 +371,187 @@ struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
 
 struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   using OpConversionPattern::OpConversionPattern;
+  ///
+  /// Control flow conversion needs shared state for tracking which block
+  /// corresponds to which operation at which level.
+  ///
+  /// This class handles such tracking and performs the conversion of control
+  /// flow related ops contained in a function.
+  class CFRewriterVisitor {
+  private:
+    using branch_to_dest_t =
+        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+    Value getCompResultAsI1(Value compResult,
+                            ConversionPatternRewriter &rewriter) {
+      auto testValue = rewriter.create<arith::ConstantOp>(
+          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = rewriter
+                      .create<arith::CmpIOp>(
+                          compResult.getLoc(), rewriter.getIntegerType(1),
+                          arith::CmpIPredicate::ne, compResult, testValue)
+                      .getResult();
+      return flag;
+    }
+
+    void replaceNestLevelWithBranch(BlockOp blockOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(blockOp, regionsToEntry[0],
+                                                blockOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(LoopOp loopOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(loopOp, regionsToEntry[0],
+                                                loopOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(IfOp ifOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      Block *falseDest =
+          regionsToEntry.size() == 2 ? regionsToEntry[1] : ifOp.getTarget();
+      auto flag = getCompResultAsI1(ifOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          ifOp, flag, regionsToEntry[0], ifOp.getInputs(), falseDest,
+          ifOp.getInputs());
+    }
+
+    template <typename LevelType>
+    LogicalResult
+    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+                                      llvm::ArrayRef<Block *> regionsToEntry,
+                                      ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
+      if (!cast)
+        return failure();
+      replaceNestLevelWithBranch(cast, regionsToEntry, rewriter);
+      return success();
+    }
+
+    template <typename... LevelTypes>
+    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+                                       ConversionPatternRewriter &rewriter) {
+      auto sip = rewriter.saveInsertionPoint();
+      Block *blockSuccessor = nestingOp->getSuccessor(0);
+      llvm::SmallVector<Block *, 2> regionEntries;
+      LLVM_DEBUG(llvm::dbgs()
+                     << "Starting inlining blocks for " << nestingOp << "\n";);
+      for (auto &region : nestingOp->getRegions()) {
+        if (region.empty())
+          continue;
+        regionEntries.push_back(&region.front());
+        /// Inline blocks of nested ops
+        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
+            region.getOps<WasmSSALabelLevelInterface>()};
+        for (auto nestedOp : nestedOps) {
+          LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
+          if (failed(inlineBlocks(nestedOp, rewriter)))
+            return failure();
+        }
+        rewriter.inlineRegionBefore(region, blockSuccessor);
+      }
+      LLVM_DEBUG(llvm::dbgs() << "End of region inlining\n");
+      LLVM_DEBUG(llvm::dbgs() << "Replacing initial op with branching\n");
+      rewriter.setInsertionPoint(nestingOp);
+      auto res = success(
+          (... || succeeded(replaceNestLevelWithBranchWrapper<LevelTypes>(
+                      nestingOp, regionEntries, rewriter))));
+      rewriter.restoreInsertionPoint(sip);
+      if (failed(res))
+        return emitError(nestingOp->getLoc(),
+                         "Unable to inline the operation regions.");
+      return success();
+    }
+
+    /// Take a nesting level defining op and inline it in the parent region.
+    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+                               ConversionPatternRewriter &rewriter) {
+      return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
+    }
+
+    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+      auto destIter = branchToDest.find(branchOp);
+      if (destIter == branchToDest.end())
+        return branchOp->emitError("No indexed label op for this operation: ")
+               << branchOp;
+      return destIter->second;
+    }
+
+    inline void convertBranch(BranchIfOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      auto flag = getCompResultAsI1(brOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          brOp, flag, dest, brOp.getInputs(), brOp.getElseSuccessor(),
+          ValueRange{});
+    }
+
+    inline void convertBranch(BlockReturnOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(brOp, dest, brOp.getInputs());
+    }
+
+    template <typename LevelInterfaceT>
+    inline LogicalResult
+    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+                         ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
+      if (!cast)
+        return failure();
+      auto sip = rewriter.saveInsertionPoint();
+      rewriter.setInsertionPoint(branchOp);
+      convertBranch(cast, dest, rewriter);
+      rewriter.restoreInsertionPoint(sip);
+      return success();
+    }
+
+    template <typename... BranchInterfaceT>
+    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+                                        ConversionPatternRewriter &rewriter) {
+      auto dest = getBlockFor(branchOp);
+      if (failed(dest))
+        return failure();
+      auto res =
+          success((... || succeeded(convertBranchWrapper<BranchInterfaceT>(
+                              branchOp, *dest, rewriter))));
+      if (failed(res))
+        return emitError(branchOp->getLoc(), "No known converter for op ")
+               << branchOp;
+      return res;
+    }
+
+    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+                                ConversionPatternRewriter &rewriter) {
+      return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
+                                                              rewriter);
+    }
+
+    func::FuncOp func;
+    branch_to_dest_t branchToDest;
+
+  public:
+    CFRewriterVisitor(func::FuncOp func) : func{func} {
+      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+        branchToDest.insert({branchOp, branchOp.getTarget()});
+      });
+    }
+    LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
+      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
+          func.getOps<WasmSSALabelLevelInterface>()};
+      for (auto nestingOp : nestingOps)
+        if (failed(inlineBlocks(nestingOp, rewriter)))
+          return failure();
+
+      auto res =
+          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+            if (failed(convertBranch(branchOp, rewriter)))
+              return WalkResult::interrupt();
+            return WalkResult::advance();
+          });
+      return failure(res.wasInterrupted());
+    }
+  };
 
   LogicalResult
   matchAndRewrite(FuncOp funcOp, FuncOp::Adaptor adaptor,
@@ -393,7 +574,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     rewriter.applySignatureConversion(oldEntryBlock, sC, getTypeConverter());
     rewriter.replaceOp(funcOp, newFunc);
-    return success();
+    CFRewriterVisitor cfRewriter{newFunc};
+    return cfRewriter.rewrite(rewriter);
   }
 };
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
new file mode 100644
index 0000000000000..3b275573bcbd3
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -0,0 +1,366 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK_CANONICALIZED %s
+
+// CHECK-LABEL:   func.func @i_am_a_block(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 17 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 42 : i32
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_3:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @i_am_a_block(
+// CHECK_CANONICALIZED-SAME:      %[[VAL_0:.*]]: i32) -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_1:.*]] = arith.constant 42 : i32
+// CHECK_CANONICALIZED:           %[[VAL_3:.*]] = memref.alloca() : memref<i32>
+// CHECK_CANONICALIZED:           memref.store %[[VAL_1]], %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED-NOT:       memref.store
+// CHECK_CANONICALIZED:           %[[VAL_4:.*]] = memref.load %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED:           return %[[VAL_4]] : i32
+wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.const 17 : i32
+  wasmssa.local_set %arg0 : ref to i32 to %1 : i32
+  wasmssa.block : {
+    %2 = wasmssa.const 42 : i32
+    wasmssa.local_set %arg0 : ref to i32 to %2 : i32
+    wasmssa.block_return
+  }> ^bb1
+  ^bb1:
+  %res = wasmssa.local_get %arg0 : ref to i32
+  wasmssa.return %res : i32
+}
+
+
+// CHECK-LABEL:   func.func @func_0(
+// CHECK-SAME:                      %[[ARG0:.*]]: i32) {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block : {
+      %3 = wasmssa.const 3: i32
+      wasmssa.block : {
+        %4 = wasmssa.const 4: i32
+        wasmssa.block_return
+      }> ^bb1
+    ^bb1:  // pred: ^bb0
+      %5 = wasmssa.const 5: i32
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_1() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func nested @func_1() {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %3 = wasmssa.const 3: i32
+  wasmssa.block : {
+    %4 = wasmssa.const 4: i32
+    wasmssa.block_return
+  }> ^bb2
+^bb2:  // pred: ^bb1
+  %5 = wasmssa.const 5: i32
+  wasmssa.block : {
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb3
+^bb3:  // pred: ^bb2
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 14 : i32
+// CHECK:           cf.br ^bb1(%[[VAL_0]] : i32)
+// CHECK:         ^bb1(%[[VAL_1:.*]]: i32):
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.addi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
+// CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
+// CHECK:           return %[[VAL_4]] : i32
+wasmssa.func nested @func_2() -> i32 {
+  %0 = wasmssa.const 14 : i32
+  wasmssa.block(%0) : i32 : {
+  ^bb0(%arg0: i32):
+    %2 = wasmssa.const 1 : i32
+    %3 = wasmssa.add %arg0 %2 : i32
+    wasmssa.block_return %3 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
+// CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
+// CHECK:           return %[[VAL_1]] : i32
+wasmssa.func nested @func_3() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 17 : i32
+    wasmssa.block_return %1 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+//// ============= Branch instructions etc ==========
+
+// CHECK-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+
+wasmssa.func nested @branch_if_taken() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 2 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+// CHECK_CANONICALIZED:         }
+wasmssa.func nested @branch_if_continue() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 0 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @if(
+// CHECK-SAME:                  %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.andi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.cmpi ne, %[[VAL_3]], %[[VAL_4]] : i32
+// CHECK:           cf.cond_br %[[VAL_5]], ^bb1, ^bb2
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 3 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.muli %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_10:.*]] = arith.addi %[[VAL_8]], %[[VAL_9]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_10]] : i32)
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_13:.*]] = arith.shrui %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
+// CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
+// CHECK:           return %[[VAL_14]] : i32
+wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.const 1 : i32
+  %3 = wasmssa.and %1 %2 : i32
+  "wasmssa.if"(%3)[^bb1] ({
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 3 : i32
+    %7 = wasmssa.mul %5 %6 : i32
+    %8 = wasmssa.const 1 : i32
+    %9 = wasmssa.add %7 %8 : i32
+    wasmssa.block_return %9 : i32
+  }, {
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    wasmssa.block_return %7 : i32
+  }) : (i32) -> ()
+^bb1(%4: i32):  // pred: ^bb0
+  wasmssa.return %4 : i32
+}
+
+// CHECK-LABEL:   func.func @if_else(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.andi %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.cmpi ne, %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           cf.cond_br %[[VAL_6]], ^bb1(%[[VAL_1]] : i32), ^bb2(%[[VAL_1]] : i32)
+// CHECK:         ^bb1(%[[VAL_7:.*]]: i32):
+// CHECK:           %[[VAL_8:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
+// CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
+// CHECK:           return %[[VAL_10]] : i32
+wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.local_get %arg0 : ref to i32
+  %3 = wasmssa.const 1 : i32
+  %4 = wasmssa.and %2 %3 : i32
+  "wasmssa.if"(%4, %1)[^bb1] ({
+  ^bb0(%arg1: i32):
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.add %arg1 %6 : i32
+    wasmssa.block_return %7 : i32
+  }, {
+  }) : (i32, i32) -> ()
+^bb1(%5: i32):  // pred: ^bb0
+  wasmssa.return %5 : i32
+}
+
+// CHECK-LABEL:   func.func @if_if(
+// CHECK-SAME:                     %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = math.cttz %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi ne, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           cf.cond_br %[[VAL_4]], ^bb1, ^bb4
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.shrui %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = math.cttz %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_11:.*]] = arith.cmpi ne, %[[VAL_9]], %[[VAL_10]] : i32
+// CHECK:           cf.cond_br %[[VAL_11]], ^bb2(%[[VAL_5]] : i32), ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_12:.*]]: i32):
+// CHECK:           %[[VAL_13:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_14:.*]] = arith.addi %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_14]] : i32)
+// CHECK:         ^bb3(%[[VAL_15:.*]]: i32):
+// CHECK:           cf.br ^bb5(%[[VAL_15]] : i32)
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_16:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
+// CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
+// CHECK:           return %[[VAL_17]] : i32
+wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.ctz %1 : i32
+  "wasmssa.if"(%2)[^bb1] ({
+    %4 = wasmssa.const 2 : i32
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    %8 = wasmssa.ctz %7 : i32
+    "wasmssa.if"(%8, %4)[^bb1] ({
+    ^bb0(%arg1: i32):
+      %10 = wasmssa.const 2 : i32
+      %11 = wasmssa.add %arg1 %10 : i32
+      wasmssa.block_return %11 : i32
+    }, {
+    }) : (i32, i32) -> ()
+  ^bb1(%9: i32):  // pred: ^bb0
+    wasmssa.block_return %9 : i32
+  }, {
+    %4 = wasmssa.const 1 : i32
+    wasmssa.block_return %4 : i32
+  }) : (i32) -> ()
+^bb1(%3: i32):  // pred: ^bb0
+  wasmssa.return %3 : i32
+}
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
new file mode 100644
index 0000000000000..e49909cc9417e
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -0,0 +1,206 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
+
+module {
+  wasmssa.func nested @func_0() {
+    wasmssa.loop : {
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   module {
+// CHECK:           func.func @func_0() {
+// CHECK:             cf.br ^bb1
+// CHECK:           ^bb1:
+// CHECK:             cf.br ^bb2
+// CHECK:           ^bb2:
+// CHECK:             return
+// CHECK:           }
+
+// CHECK-CANONICAL-LABEL:  func.func @func_0() {
+// CHECK-CANONICAL:             return
+// CHECK-CANONICAL:           }
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() -> i32 {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.block_return %3 : i32
+    }> ^bb1
+  ^bb1(%1: i32):  // pred: ^bb0
+    wasmssa.return %1 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_0() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           cf.br ^bb2(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() -> i32 {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK-CANONICAL:           return %[[VAL_5]] : i32
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.branch_if %3 to level 0 else ^bb1
+    ^bb1:  // pred: ^bb0
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_7:.*]] = arith.cmpi ne, %[[VAL_5]], %[[VAL_6]] : i32
+// CHECK:           cf.cond_br %[[VAL_7]], ^bb1, ^bb2
+// CHECK:         ^bb2:
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_4]], ^bb1, ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           return
+
+// -----
+
+module {
+  wasmssa.func nested @func_0() {
+    %0 = wasmssa.local of type i32
+    %1 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %2 = wasmssa.local_get %0 : ref to i32
+      %3 = wasmssa.const 1 : i32
+      %4 = wasmssa.add %2 %3 : i32
+      wasmssa.local_set %0 : ref to i32 to %4 : i32
+      wasmssa.loop : {
+        %8 = wasmssa.const 12 : i32
+        %9 = wasmssa.local_get %0 : ref to i32
+        %10 = wasmssa.gt_si %8 %9 : i32 -> i32
+        wasmssa.branch_if %10 to level 0 else ^bb1
+      ^bb1:  // pred: ^bb0
+        wasmssa.block_return %8 : i32
+      }> ^bb1
+    ^bb1(%5: i32):  // pred: ^bb0
+      %6 = wasmssa.const 10 : i32
+      %7 = wasmssa.lt_si %5 %6 : i32 -> i32
+      wasmssa.branch_if %7 to level 0 else ^bb2
+    ^bb2:  // pred: ^bb1
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_3]], %[[VAL_2]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_4:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_5:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.addi %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           memref.store %[[VAL_6]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_7:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_9:.*]] = arith.cmpi sgt, %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.extui %[[VAL_9]] : i1 to i32
+// CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_12:.*]] = arith.cmpi ne, %[[VAL_10]], %[[VAL_11]] : i32
+// CHECK:           cf.cond_br %[[VAL_12]], ^bb2, ^bb3
+// CHECK:         ^bb3:
+// CHECK:           cf.br ^bb4(%[[VAL_7]] : i32)
+// CHECK:         ^bb4(%[[VAL_13:.*]]: i32):
+// CHECK:           %[[VAL_14:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_15:.*]] = arith.cmpi slt, %[[VAL_13]], %[[VAL_14]] : i32
+// CHECK:           %[[VAL_16:.*]] = arith.extui %[[VAL_15]] : i1 to i32
+// CHECK:           %[[VAL_17:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_18:.*]] = arith.cmpi ne, %[[VAL_16]], %[[VAL_17]] : i32
+// CHECK:           cf.cond_br %[[VAL_18]], ^bb1, ^bb5
+// CHECK:         ^bb5:
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
+// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
+// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
+// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
+// CHECK-CANONICAL:         ^bb4:
+// CHECK-CANONICAL:           return

>From 2af7f4e6300c16db66abf0ad83efa9e338ca21bd Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 3 Jul 2025 13:36:04 +0900
Subject: [PATCH 4/7] [mlir][wasm] RaiseWasmMLIRPass: Add support for memory

--
 Co-authored-by: Luc Forget <luc.forget at woven-planet.global>
 Co-authored-by: Jessica Paquette <jessica.paquette at woven-planet.global>
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 41 +++++++++++++++++++
 .../RaiseWasm/wasm-memory-to-memref.mlir      | 14 +++++++
 2 files changed, 55 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index d9dcb756790f8..53a8900b965a3 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -677,6 +677,46 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(MemOp memOp, MemOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = memOp.getLoc();
+    auto bufferType =
+        MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
+    auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
+        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
+    auto initializerName = (memPtr.getSymName() + "::initializer").str();
+    auto memInitializer = rewriter.create<func::FuncOp>(
+        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    memInitializer->setAttr(rewriter.getStringAttr("initializer"),
+                            rewriter.getUnitAttr());
+    auto *initializerBody = memInitializer.addEntryBlock();
+    auto sip = rewriter.saveInsertionPoint();
+    rewriter.setInsertionPointToStart(initializerBody);
+    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
+        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = rewriter.create<memref::AllocOp>(
+        loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto castOp =
+        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
+    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
+    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
+                                     memRefPtr.getResult(),
+                                     ValueRange{idx.getResult()});
+    rewriter.create<func::ReturnOp>(loc);
+    rewriter.restoreInsertionPoint(sip);
+    rewriter.create<func::CallOp>(loc, memInitializer);
+    return success();
+  }
+};
+
 inline TypedAttr getInitializerAttr(Type t) {
   assert(t.isIntOrFloat() &&
          "This helper is intended to use with int and float types");
@@ -847,6 +887,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmLtSIOpConversion,
            WasmLtUIOpConversion,
            WasmMaxOpConversion,
+           WasmMemoryOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
            WasmNeOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
new file mode 100644
index 0000000000000..09b07184d97fa
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
@@ -0,0 +1,14 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+"wasmssa.memory"() <{limits = !wasmssa<limit[17:]>, sym_name = "mem_0", sym_visibility = "nested"}> : () -> ()
+
+// CHECK-LABEL:   memref.global "nested" @mem_0 : memref<1xmemref<?xi8>>
+
+// CHECK-LABEL:   func.func @"mem_0::initializer"() attributes {initializer} {
+// CHECK:           %[[VAL_0:.*]] = memref.get_global @mem_0 : memref<1xmemref<?xi8>>
+// CHECK:           %[[VAL_1:.*]] = memref.alloc() : memref<17xi8>
+// CHECK:           %[[VAL_2:.*]] = memref.cast %[[VAL_1]] : memref<17xi8> to memref<?xi8>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]]{{\[}}%[[VAL_3]]] : memref<1xmemref<?xi8>>
+// CHECK:           return
+// CHECK:         func.call @"mem_0::initializer"() : () -> ()

>From ec4763983eff03797ce2a4f1d60e296b250450a1 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Thu, 30 Apr 2026 21:00:48 +0900
Subject: [PATCH 5/7] [mlir][wasm] Deprecate rewriter.create in favor of
 <op>::create and cleanup the tests

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 123 ++++++++----------
 .../RaiseWasm/wasm-blocks-to-cf.mlir          |  18 +--
 .../wasm-comparisons-to-arith-cmp.mlir        |  64 ++++-----
 .../RaiseWasm/wasm-extend-to-arith-ext.mlir   |  14 +-
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir |   8 +-
 .../RaiseWasm/wasm-rotl-to-arith.mlir         |   4 +-
 .../RaiseWasm/wasm-rotr-to-arith.mlir         |   4 +-
 7 files changed, 112 insertions(+), 123 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 53a8900b965a3..0e6e0c05b79b6 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
+#include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinDialect.h"
 #include "mlir/IR/ValueRange.h"
@@ -159,25 +160,22 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
 
     // Materialize (width - 1) for use in both sides of the expression.
     auto cstWidthMinusOne =
-        rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, width - 1));
+        ConstOp::create(rewriter, loc, IntegerAttr::get(ty, width - 1));
 
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
-    auto orLHS = rewriter.create<LHSShiftOp>(
-        loc, val, rewriter.create<AndOp>(loc, bits, cstWidthMinusOne));
+    auto orLHS = LHSShiftOp::create(
+        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
-    auto orRHS = rewriter.create<RHSShiftOp>(
-        loc, val,
+    auto orRHS = RHSShiftOp::create(
+        rewriter, loc, val,
         // (-bits & (width - 1))
-        rewriter.create<AndOp>(
-            loc,
+        AndOp::create(
+            rewriter, loc,
             // 0 - bits == -bits
-            rewriter.create<SubOp>(
-                loc,
-                rewriter.create<ConstOp>(loc, IntegerAttr::get(ty, 0)),
-                bits),
+            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
             cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
@@ -199,8 +197,7 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
   matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
-        rewriter
-            .create<TargetOp>(srcOp.getLoc(), rewriter.getI1Type(),
+        TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
                               AttrType::get(rewriter.getContext(), flag),
                               adaptor.getLhs(), adaptor.getRhs())
             .getResult();
@@ -257,19 +254,15 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          rewriter
-              .create<arith::CmpIOp>(
-                  srcOp.getLoc(), rewriter.getI1Type(),
-                  arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                  adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          rewriter
-              .create<arith::CmpFOp>(srcOp.getLoc(), rewriter.getI1Type(),
-                                     arith::CmpFPredicateAttr::get(
-                                         rewriter.getContext(), FloatFlag),
-                                     adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+                                adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -320,13 +313,9 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
     auto zero =
-        rewriter
-            .create<arith::ConstantOp>(
-                loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
             .getResult();
-    auto cmpRes = rewriter
-                      .create<arith::CmpIOp>(
-                          loc, rewriter.getI1Type(),
+    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
                           arith::CmpIPredicateAttr::get(
                               rewriter.getContext(), arith::CmpIPredicate::eq),
                           adaptor.getInput(), zero)
@@ -346,9 +335,7 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = rewriter.create<arith::TruncIOp>(
-        extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth),
-        adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -380,15 +367,11 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   class CFRewriterVisitor {
   private:
     using branch_to_dest_t =
-        llvm::DenseMap<WasmSSALabelBranchingInterface, Block *>;
+        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = rewriter.create<arith::ConstantOp>(
-          compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = rewriter
-                      .create<arith::CmpIOp>(
-                          compResult.getLoc(), rewriter.getIntegerType(1),
-                          arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -420,7 +403,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelType>
     LogicalResult
-    replaceNestLevelWithBranchWrapper(WasmSSALabelLevelInterface nestingOp,
+    replaceNestLevelWithBranchWrapper(LabelLevelOpInterface nestingOp,
                                       llvm::ArrayRef<Block *> regionsToEntry,
                                       ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
@@ -431,7 +414,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... LevelTypes>
-    LogicalResult inlineNestDispatcher(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineNestDispatcher(LabelLevelOpInterface nestingOp,
                                        ConversionPatternRewriter &rewriter) {
       auto sip = rewriter.saveInsertionPoint();
       Block *blockSuccessor = nestingOp->getSuccessor(0);
@@ -443,8 +426,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
           continue;
         regionEntries.push_back(&region.front());
         /// Inline blocks of nested ops
-        llvm::SmallVector<WasmSSALabelLevelInterface> nestedOps{
-            region.getOps<WasmSSALabelLevelInterface>()};
+        llvm::SmallVector<LabelLevelOpInterface> nestedOps{
+            region.getOps<LabelLevelOpInterface>()};
         for (auto nestedOp : nestedOps) {
           LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
           if (failed(inlineBlocks(nestedOp, rewriter)))
@@ -466,12 +449,12 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     /// Take a nesting level defining op and inline it in the parent region.
-    LogicalResult inlineBlocks(WasmSSALabelLevelInterface nestingOp,
+    LogicalResult inlineBlocks(LabelLevelOpInterface nestingOp,
                                ConversionPatternRewriter &rewriter) {
       return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
     }
 
-    llvm::FailureOr<Block *> getBlockFor(WasmSSALabelBranchingInterface branchOp) {
+    llvm::FailureOr<Block *> getBlockFor(LabelBranchingOpInterface branchOp) {
       auto destIter = branchToDest.find(branchOp);
       if (destIter == branchToDest.end())
         return branchOp->emitError("No indexed label op for this operation: ")
@@ -494,7 +477,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     template <typename LevelInterfaceT>
     inline LogicalResult
-    convertBranchWrapper(WasmSSALabelBranchingInterface branchOp, Block *dest,
+    convertBranchWrapper(LabelBranchingOpInterface branchOp, Block *dest,
                          ConversionPatternRewriter &rewriter) {
       auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
       if (!cast)
@@ -507,7 +490,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
     }
 
     template <typename... BranchInterfaceT>
-    LogicalResult convertBranchDispatch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranchDispatch(LabelBranchingOpInterface branchOp,
                                         ConversionPatternRewriter &rewriter) {
       auto dest = getBlockFor(branchOp);
       if (failed(dest))
@@ -521,7 +504,7 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
       return res;
     }
 
-    LogicalResult convertBranch(WasmSSALabelBranchingInterface branchOp,
+    LogicalResult convertBranch(LabelBranchingOpInterface branchOp,
                                 ConversionPatternRewriter &rewriter) {
       return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
                                                               rewriter);
@@ -532,19 +515,19 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
   public:
     CFRewriterVisitor(func::FuncOp func) : func{func} {
-      func.walk([this](WasmSSALabelBranchingInterface branchOp) {
+      func.walk([this](LabelBranchingOpInterface branchOp) {
         branchToDest.insert({branchOp, branchOp.getTarget()});
       });
     }
     LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
-      llvm::SmallVector<WasmSSALabelLevelInterface> nestingOps{
-          func.getOps<WasmSSALabelLevelInterface>()};
+      llvm::SmallVector<LabelLevelOpInterface> nestingOps{
+          func.getOps<LabelLevelOpInterface>()};
       for (auto nestingOp : nestingOps)
         if (failed(inlineBlocks(nestingOp, rewriter)))
           return failure();
 
       auto res =
-          func->walk([this, &rewriter](WasmSSALabelBranchingInterface branchOp) {
+          func->walk([this, &rewriter](LabelBranchingOpInterface branchOp) {
             if (failed(convertBranch(branchOp, rewriter)))
               return WalkResult::interrupt();
             return WalkResult::advance();
@@ -687,32 +670,38 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
     auto bufferType =
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memVisibility = memOp.getVisibility();
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    mlir::StringAttr visAttr;
+    if (memVisibility == mlir::SymbolTable::Visibility::Public)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+    else if (memVisibility == mlir::SymbolTable::Visibility::Private)
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+    else
+        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), memOp.getSymVisibilityAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr,
         TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = rewriter.create<func::FuncOp>(
-        loc, initializerName, FunctionType::get(getContext(), {}, {}));
+    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
+                                               FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = rewriter.create<memref::GetGlobalOp>(
-        loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = rewriter.create<memref::AllocOp>(
-        loc,
-        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
-        rewriter.create<memref::CastOp>(loc, bufferType, alloc.getResult());
-    auto idx = rewriter.create<arith::ConstantIndexOp>(loc, 0);
-    rewriter.create<memref::StoreOp>(loc, castOp.getResult(),
-                                     memRefPtr.getResult(),
-                                     ValueRange{idx.getResult()});
-    rewriter.create<func::ReturnOp>(loc);
+        memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    memref::StoreOp::create(rewriter, loc, castOp.getResult(),
+                            memRefPtr.getResult(), ValueRange{idx.getResult()});
+    func::ReturnOp::create(rewriter, loc);
     rewriter.restoreInsertionPoint(sip);
-    rewriter.create<func::CallOp>(loc, memInitializer);
+    func::CallOp::create(rewriter, loc, memInitializer);
     return success();
   }
 };
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
index 3b275573bcbd3..fdce71cab096b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -60,7 +60,7 @@ wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
+wasmssa.func @func_0(%arg0: !wasmssa<local ref to i32>) {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -104,7 +104,7 @@ wasmssa.func nested @func_0(%arg0: !wasmssa<local ref to i32>) {
 // CHECK:         ^bb6:
 // CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
 // CHECK:           return
-wasmssa.func nested @func_1() {
+wasmssa.func @func_1() {
   %1 = wasmssa.const 1: i32
   wasmssa.block : {
     %2 = wasmssa.const 2: i32
@@ -136,7 +136,7 @@ wasmssa.func nested @func_1() {
 // CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
 // CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
 // CHECK:           return %[[VAL_4]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 14 : i32
   wasmssa.block(%0) : i32 : {
   ^bb0(%arg0: i32):
@@ -155,7 +155,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
 // CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
 // CHECK:           return %[[VAL_1]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 17 : i32
     wasmssa.block_return %1 : i32
@@ -185,7 +185,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 
-wasmssa.func nested @branch_if_taken() -> i32 {
+wasmssa.func @branch_if_taken() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 2 : i32
@@ -218,7 +218,7 @@ wasmssa.func nested @branch_if_taken() -> i32 {
 // CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
 // CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
 // CHECK_CANONICALIZED:         }
-wasmssa.func nested @branch_if_continue() -> i32 {
+wasmssa.func @branch_if_continue() -> i32 {
   wasmssa.block : {
     %1 = wasmssa.const 1 : i32
     %2 = wasmssa.const 0 : i32
@@ -256,7 +256,7 @@ wasmssa.func nested @branch_if_continue() -> i32 {
 // CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
 // CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
 // CHECK:           return %[[VAL_14]] : i32
-wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.const 1 : i32
   %3 = wasmssa.and %1 %2 : i32
@@ -294,7 +294,7 @@ wasmssa.func nested @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
 // CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
 // CHECK:           return %[[VAL_10]] : i32
-wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.local_get %arg0 : ref to i32
   %3 = wasmssa.const 1 : i32
@@ -339,7 +339,7 @@ wasmssa.func nested @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
 // CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
 // CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
 // CHECK:           return %[[VAL_17]] : i32
-wasmssa.func nested @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
   %1 = wasmssa.local_get %arg0 : ref to i32
   %2 = wasmssa.ctz %1 : i32
   "wasmssa.if"(%2)[^bb1] ({
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
index bc952b47520ca..6643b0389f405 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -2,194 +2,194 @@
 
 
 module {
-  wasmssa.func nested @func_lt_si32() -> i32 {
+  wasmssa.func @func_lt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
 
-  wasmssa.func nested @func_le_si32() -> i32 {
+  wasmssa.func @func_le_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui32() -> i32 {
+  wasmssa.func @func_lt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui32() -> i32 {
+  wasmssa.func @func_le_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.le_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si32() -> i32 {
+  wasmssa.func @func_gt_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui32() -> i32 {
+  wasmssa.func @func_gt_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si32() -> i32 {
+  wasmssa.func @func_ge_si32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_si %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui32() -> i32 {
+  wasmssa.func @func_ge_ui32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_si64() -> i32 {
+  wasmssa.func @func_lt_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_si64() -> i32 {
+  wasmssa.func @func_le_si64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_ui_i64() -> i32 {
+  wasmssa.func @func_lt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_ui_i64() -> i32 {
+  wasmssa.func @func_le_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.le_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_si_i64() -> i32 {
+  wasmssa.func @func_gt_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_ui_i64() -> i32 {
+  wasmssa.func @func_gt_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_si_i64() -> i32 {
+  wasmssa.func @func_ge_si_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_si %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_ui_i64() -> i32 {
+  wasmssa.func @func_ge_ui_i64() -> i32 {
     %0 = wasmssa.const 12 : i64
     %1 = wasmssa.const 50 : i64
     %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f32() -> i32 {
+  wasmssa.func @func_lt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.lt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f32() -> i32 {
+  wasmssa.func @func_le_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.le %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f32() -> i32 {
+  wasmssa.func @func_gt_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.gt %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f32() -> i32 {
+  wasmssa.func @func_ge_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ge %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_lt_f64() -> i32 {
+  wasmssa.func @func_lt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.lt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_le_f64() -> i32 {
+  wasmssa.func @func_le_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.le %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_gt_f64() -> i32 {
+  wasmssa.func @func_gt_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.gt %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ge_f64() -> i32 {
+  wasmssa.func @func_ge_f64() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f64
     %1 = wasmssa.const 1.400000e+01 : f64
     %2 = wasmssa.ge %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i32() -> i32 {
+  wasmssa.func @func_eq_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.eq %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_i64() -> i32 {
+  wasmssa.func @func_eq_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.eq %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f32() -> i32 {
+  wasmssa.func @func_eq_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.eq %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_eq_f64() -> i32 {
+  wasmssa.func @func_eq_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.eq %0 %1 : f64 -> i32
     wasmssa.return %2 : i32
   }
-    wasmssa.func nested @func_ne_i32() -> i32 {
+    wasmssa.func @func_ne_i32() -> i32 {
     %0 = wasmssa.const 12 : i32
     %1 = wasmssa.const 50 : i32
     %2 = wasmssa.ne %0 %1 : i32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_i64() -> i32 {
+  wasmssa.func @func_ne_i64() -> i32 {
     %0 = wasmssa.const 20 : i64
     %1 = wasmssa.const 5 : i64
     %2 = wasmssa.ne %0 %1 : i64 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f32() -> i32 {
+  wasmssa.func @func_ne_f32() -> i32 {
     %0 = wasmssa.const 5.000000e+00 : f32
     %1 = wasmssa.const 1.400000e+01 : f32
     %2 = wasmssa.ne %0 %1 : f32 -> i32
     wasmssa.return %2 : i32
   }
-  wasmssa.func nested @func_ne_f64() -> i32 {
+  wasmssa.func @func_ne_f64() -> i32 {
     %0 = wasmssa.const 1.700000e+01 : f64
     %1 = wasmssa.const 0.000000e+00 : f64
     %2 = wasmssa.ne %0 %1 : f64 -> i32
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
index 90c92001a78d6..2b8bb651bbe5b 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -4,7 +4,7 @@
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_0() -> i64 {
+wasmssa.func @func_0() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_s %0 to i64
   wasmssa.return %1 : i64
@@ -14,7 +14,7 @@ wasmssa.func nested @func_0() -> i64 {
 // CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
 // CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
 // CHECK:           return %[[VAL_1]] : i64
-wasmssa.func nested @func_1() -> i64 {
+wasmssa.func @func_1() -> i64 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend_i32_u %0 to i64
   wasmssa.return %1 : i64
@@ -25,7 +25,7 @@ wasmssa.func nested @func_1() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_2() -> i32 {
+wasmssa.func @func_2() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 8 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -36,7 +36,7 @@ wasmssa.func nested @func_2() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
 // CHECK:           return %[[VAL_2]] : i32
-wasmssa.func nested @func_3() -> i32 {
+wasmssa.func @func_3() -> i32 {
   %0 = wasmssa.const 10 : i32
   %1 = wasmssa.extend 16 low bits from %0: i32
   wasmssa.return %1 : i32
@@ -47,7 +47,7 @@ wasmssa.func nested @func_3() -> i32 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_4() -> i64 {
+wasmssa.func @func_4() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 8 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -58,7 +58,7 @@ wasmssa.func nested @func_4() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_5() -> i64 {
+wasmssa.func @func_5() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 16 low bits from %0: i64
   wasmssa.return %1 : i64
@@ -69,7 +69,7 @@ wasmssa.func nested @func_5() -> i64 {
 // CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
 // CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
 // CHECK:           return %[[VAL_2]] : i64
-wasmssa.func nested @func_6() -> i64 {
+wasmssa.func @func_6() -> i64 {
   %0 = wasmssa.const 10 : i64
   %1 = wasmssa.extend 32 low bits from %0: i64
   wasmssa.return %1 : i64
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index e49909cc9417e..6ab812008a2f8 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -2,7 +2,7 @@
 // RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     wasmssa.loop : {
       wasmssa.block_return
     }> ^bb1
@@ -27,7 +27,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() -> i32 {
+  wasmssa.func @func_0() -> i32 {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -67,7 +67,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     wasmssa.loop : {
       %1 = wasmssa.local_get %0 : ref to i32
@@ -117,7 +117,7 @@ module {
 // -----
 
 module {
-  wasmssa.func nested @func_0() {
+  wasmssa.func @func_0() {
     %0 = wasmssa.local of type i32
     %1 = wasmssa.local of type i32
     wasmssa.loop : {
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
index 91e929bd5ff35..b69f88afd4d7c 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -31,7 +31,7 @@
 
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
-wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -66,7 +66,7 @@ wasmssa.func nested @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // Form final result.
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
-wasmssa.func nested @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
index 77e301c3221e3..f0b77271c86dd 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -32,7 +32,7 @@
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
 // CHECK:           return %[[RES]] : i32
 // CHECK:         }
-wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+wasmssa.func @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
     %v0 = wasmssa.local_get %arg0 : ref to i32
     %v1 = wasmssa.local_get %arg1 : ref to i32
 
@@ -68,7 +68,7 @@ wasmssa.func nested @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa
 // CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
 // CHECK:           return %[[RES]] : i64
 // CHECK:         }
-wasmssa.func nested @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+wasmssa.func @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
     %v0 = wasmssa.local_get %arg0 : ref to i64
     %v1 = wasmssa.local_get %arg1 : ref to i64
 

>From a249e73f27672c4d5dfb8552cd0057da55e9e3c0 Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 16:03:07 +0900
Subject: [PATCH 6/7] [mlir][wasm] Fix format

---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 88 +++++++++++--------
 1 file changed, 52 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 0e6e0c05b79b6..b4b5006352c11 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -165,18 +165,21 @@ struct RotateOpConversion : OpConversionPattern<SourceOp> {
     // Form the left-hand side of the OR:
     // (val (lhs shift op) (bits & (width - 1)))
     auto orLHS = LHSShiftOp::create(
-        rewriter, loc, val, AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
+        rewriter, loc, val,
+        AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
 
     // Form the right-hand side of the OR:
     // (val (rhs shift op) (-bits & (width - 1)))
     auto orRHS = RHSShiftOp::create(
         rewriter, loc, val,
         // (-bits & (width - 1))
-        AndOp::create(
-            rewriter, loc,
-            // 0 - bits == -bits
-            SubOp::create(rewriter, loc, ConstOp::create(rewriter, loc, IntegerAttr::get(ty, 0)), bits),
-            cstWidthMinusOne));
+        AndOp::create(rewriter, loc,
+                      // 0 - bits == -bits
+                      SubOp::create(rewriter, loc,
+                                    ConstOp::create(rewriter, loc,
+                                                    IntegerAttr::get(ty, 0)),
+                                    bits),
+                      cstWidthMinusOne));
 
     // OR together the two shifts and replace the rotate with the new
     // expression.
@@ -198,8 +201,8 @@ struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
                   ConversionPatternRewriter &rewriter) const override {
     auto cmpRes =
         TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                              AttrType::get(rewriter.getContext(), flag),
-                              adaptor.getLhs(), adaptor.getRhs())
+                         AttrType::get(rewriter.getContext(), flag),
+                         adaptor.getLhs(), adaptor.getRhs())
             .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -254,15 +257,17 @@ struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
     Value comparisonResult;
     if (srcOp.getLhs().getType().isInteger())
       comparisonResult =
-          arith::CmpIOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpIOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else if (srcOp.getLhs().getType().isFloat())
       comparisonResult =
-          arith::CmpFOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
-                                arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
-                                adaptor.getLhs(), adaptor.getRhs())
+          arith::CmpFOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+              adaptor.getLhs(), adaptor.getRhs())
               .getResult();
     else
       return rewriter.notifyMatchFailure(
@@ -312,13 +317,15 @@ struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
   matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto loc = eqzOp->getLoc();
-    auto zero =
-        arith::ConstantOp::create(rewriter, loc, rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
-            .getResult();
-    auto cmpRes = arith::CmpIOp::create(rewriter, loc, rewriter.getI1Type(),
-                          arith::CmpIPredicateAttr::get(
-                              rewriter.getContext(), arith::CmpIPredicate::eq),
-                          adaptor.getInput(), zero)
+    auto zero = arith::ConstantOp::create(
+                    rewriter, loc,
+                    rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+                    .getResult();
+    auto cmpRes = arith::CmpIOp::create(
+                      rewriter, loc, rewriter.getI1Type(),
+                      arith::CmpIPredicateAttr::get(rewriter.getContext(),
+                                                    arith::CmpIPredicate::eq),
+                      adaptor.getInput(), zero)
                       .getResult();
     rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
                                                 cmpRes);
@@ -335,7 +342,9 @@ struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
                   ExtendLowBitsSOp::Adaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
-    auto truncation = arith::TruncIOp::create(rewriter, extendLowBytesSOp->getLoc(), rewriter.getIntegerType(truncWidth), adaptor.getInput());
+    auto truncation = arith::TruncIOp::create(
+        rewriter, extendLowBytesSOp->getLoc(),
+        rewriter.getIntegerType(truncWidth), adaptor.getInput());
     rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
         extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
         truncation.getResult());
@@ -366,12 +375,14 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   /// flow related ops contained in a function.
   class CFRewriterVisitor {
   private:
-    using branch_to_dest_t =
-        llvm::DenseMap<LabelBranchingOpInterface, Block *>;
+    using branch_to_dest_t = llvm::DenseMap<LabelBranchingOpInterface, Block *>;
     Value getCompResultAsI1(Value compResult,
                             ConversionPatternRewriter &rewriter) {
-      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(), rewriter.getI32IntegerAttr(0));
-      auto flag = arith::CmpIOp::create(rewriter, compResult.getLoc(), rewriter.getIntegerType(1), arith::CmpIPredicate::ne, compResult, testValue)
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(),
+                                                 rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(
+                      rewriter, compResult.getLoc(), rewriter.getIntegerType(1),
+                      arith::CmpIPredicate::ne, compResult, testValue)
                       .getResult();
       return flag;
     }
@@ -671,29 +682,34 @@ struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
         MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
     auto bufferPtrType = MemRefType::get({1}, bufferType);
     auto memVisibility = memOp.getVisibility();
-    // Convert to StringAttr since memref::GlobalOp expects visibility as a string attribute.
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a
+    // string attribute.
     mlir::StringAttr visAttr;
     if (memVisibility == mlir::SymbolTable::Visibility::Public)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
     else if (memVisibility == mlir::SymbolTable::Visibility::Private)
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
     else
-        visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
 
     auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
-        memOp, memOp.getSymNameAttr(), visAttr,
-        TypeAttr::get(bufferPtrType), /*initialValue*/ rewriter.getUnitAttr(),
+        memOp, memOp.getSymNameAttr(), visAttr, TypeAttr::get(bufferPtrType),
+        /*initialValue*/ rewriter.getUnitAttr(),
         /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
     auto initializerName = (memPtr.getSymName() + "::initializer").str();
-    auto memInitializer = func::FuncOp::create(rewriter, loc, initializerName,
-                                               FunctionType::get(getContext(), {}, {}));
+    auto memInitializer =
+        func::FuncOp::create(rewriter, loc, initializerName,
+                             FunctionType::get(getContext(), {}, {}));
     memInitializer->setAttr(rewriter.getStringAttr("initializer"),
                             rewriter.getUnitAttr());
     auto *initializerBody = memInitializer.addEntryBlock();
     auto sip = rewriter.saveInsertionPoint();
     rewriter.setInsertionPointToStart(initializerBody);
-    auto memRefPtr = memref::GetGlobalOp::create(rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
-    auto alloc = memref::AllocOp::create(rewriter, loc, MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto memRefPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(
+        rewriter, loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
     auto castOp =
         memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
     auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);

>From 528e3fc46245cf2e62904b6d60508b8aa301b5ca Mon Sep 17 00:00:00 2001
From: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>
Date: Fri, 26 Jun 2026 21:26:54 +0900
Subject: [PATCH 7/7] [mlir][wasm] Update tests references

---
 .../Conversion/RaiseWasm/wasm-loop-to-cf.mlir | 29 ++++++++-----------
 1 file changed, 12 insertions(+), 17 deletions(-)

diff --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
index 6ab812008a2f8..cdaf249d9ccf6 100644
--- a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -183,24 +183,19 @@ module {
 // CHECK:         }
 
 // CHECK-CANONICAL-LABEL:   func.func @func_0() {
-// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
-// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 12 : i32
-// CHECK-CANONICAL:           %[[VAL_2:.*]] = arith.constant 1 : i32
-// CHECK-CANONICAL:           %[[VAL_3:.*]] = arith.constant 0 : i32
-// CHECK-CANONICAL:           %[[VAL_4:.*]] = memref.alloca() : memref<i32>
-// CHECK-CANONICAL:           memref.store %[[VAL_3]], %[[VAL_4]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CONSTANT_0:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_1:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_2:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[ALLOCA_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[CONSTANT_2]], %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[LOAD_0:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[ADDI_0:.*]] = arith.addi %[[LOAD_0]], %[[CONSTANT_1]] : i32
+// CHECK-CANONICAL:           memref.store %[[ADDI_0]], %[[ALLOCA_0]][] : memref<i32>
 // CHECK-CANONICAL:           cf.br ^bb1
 // CHECK-CANONICAL:         ^bb1:
-// CHECK-CANONICAL:           %[[VAL_5:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_6:.*]] = arith.addi %[[VAL_5]], %[[VAL_2]] : i32
-// CHECK-CANONICAL:           memref.store %[[VAL_6]], %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           cf.br ^bb2
+// CHECK-CANONICAL:           %[[LOAD_1:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CMPI_0:.*]] = arith.cmpi slt, %[[LOAD_1]], %[[CONSTANT_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[CMPI_0]], ^bb1, ^bb2
 // CHECK-CANONICAL:         ^bb2:
-// CHECK-CANONICAL:           %[[VAL_7:.*]] = memref.load %[[VAL_4]][] : memref<i32>
-// CHECK-CANONICAL:           %[[VAL_8:.*]] = arith.cmpi slt, %[[VAL_7]], %[[VAL_1]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_8]], ^bb2, ^bb3(%[[VAL_1]] : i32)
-// CHECK-CANONICAL:         ^bb3(%[[VAL_9:.*]]: i32):
-// CHECK-CANONICAL:           %[[VAL_10:.*]] = arith.cmpi slt, %[[VAL_9]], %[[VAL_0]] : i32
-// CHECK-CANONICAL:           cf.cond_br %[[VAL_10]], ^bb1, ^bb4
-// CHECK-CANONICAL:         ^bb4:
 // CHECK-CANONICAL:           return
+// CHECK-CANONICAL:         }


From llvmlistbot at llvm.org  Wed Jul 22 03:58:54 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Wed, 22 Jul 2026 03:58:54 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a60a26e.f3151ff8.f584a.aa10@mx.google.com>


https://github.com/flemairen6 auto_merge_enabled https://github.com/llvm/llvm-project/pull/205990

From llvmlistbot at llvm.org  Wed Jul 22 04:07:39 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 04:07:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f61499b - [MLIR][WASM] Introduce full support
 for raising WASM MLIR to other dialects (#205990)
Message-ID: <6a60a47b.f3151ff8.f584a.0842@mx.google.com>


Author: Ferdinand Lemaire
Date: 2026-07-22T11:07:34Z
New Revision: f61499bff6880928ed0a3f11f79086943be16b8f

URL: https://github.com/llvm/llvm-project/commit/f61499bff6880928ed0a3f11f79086943be16b8f
DIFF: https://github.com/llvm/llvm-project/commit/f61499bff6880928ed0a3f11f79086943be16b8f.diff

LOG: [MLIR][WASM] Introduce full support for raising WASM MLIR to other dialects (#205990)

Following https://github.com/llvm/llvm-project/pull/164562 where
RaiseWasm was introduced.
This PR completes the support for rewriting the currently supported Wasm
MLIR operators to arith, math, cf and memref.

---------

Co-authored-by: Ferdinand Lemaire <ferdinand.lemaire at woven-planet.global>

Added: 
    mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
    mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
    mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
    mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
    mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
    mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
    mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
    mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir

Modified: 
    mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index 83bfde7032ef8..b4b5006352c11 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -20,6 +20,7 @@
 #include "mlir/Dialect/MemRef/IR/MemRef.h"
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
+#include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.h"
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinDialect.h"
 #include "mlir/IR/ValueRange.h"
@@ -124,6 +125,167 @@ using WasmTruncOpConversion = OpMappingConversion<TruncOp, math::TruncOp>;
 using WasmSqrtOpConversion = OpMappingConversion<SqrtOp, math::SqrtOp>;
 using WasmWrapOpConversion = OpMappingConversion<WrapOp, arith::TruncIOp>;
 
+/// Lower a rotate to a series of bitwise operations. Intended for us
+/// in dialects that do not natively support rotate operations.
+///
+/// Result stays in the wasm dialect. It will then subsequently be lowered to
+/// the target dialect.
+///
+/// The rotate will be lowered to a pattern like so:
+///
+/// (val LHSShiftOp (bits & (width-1))) | (val RHSShiftOp (-bits & (width-1)))
+///
+/// Where LHSShiftOp and RHSShiftOp are shift operations. Concretely,
+///
+/// rotr = (val >> (bits & (width - 1))) | (val << (-bits & (width - 1)))
+/// rotl = (val << (bits & (width - 1))) | (val >> (-bits & (width - 1)))
+///
+/// Using this variant ensures that our rotate is defined in the target dialect.
+///
+/// \p SourceOp - Rotate operation to replace.
+/// \p LHSShiftOp - Shift operation to use on the left-hand side of the OR.
+/// \p RHSShiftOp - Shift operation to use on the right-hand side of the OR.
+template <typename SourceOp, typename LHSShiftOp, typename RHSShiftOp>
+struct RotateOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    const Type ty = srcOp->getResultTypes()[0];
+    const Location loc = srcOp->getLoc();
+    const Value val = adaptor.getVal();
+    const Value bits = adaptor.getBits();
+    const unsigned width = ty.getIntOrFloatBitWidth();
+
+    // Materialize (width - 1) for use in both sides of the expression.
+    auto cstWidthMinusOne =
+        ConstOp::create(rewriter, loc, IntegerAttr::get(ty, width - 1));
+
+    // Form the left-hand side of the OR:
+    // (val (lhs shift op) (bits & (width - 1)))
+    auto orLHS = LHSShiftOp::create(
+        rewriter, loc, val,
+        AndOp::create(rewriter, loc, bits, cstWidthMinusOne));
+
+    // Form the right-hand side of the OR:
+    // (val (rhs shift op) (-bits & (width - 1)))
+    auto orRHS = RHSShiftOp::create(
+        rewriter, loc, val,
+        // (-bits & (width - 1))
+        AndOp::create(rewriter, loc,
+                      // 0 - bits == -bits
+                      SubOp::create(rewriter, loc,
+                                    ConstOp::create(rewriter, loc,
+                                                    IntegerAttr::get(ty, 0)),
+                                    bits),
+                      cstWidthMinusOne));
+
+    // OR together the two shifts and replace the rotate with the new
+    // expression.
+    rewriter.replaceOpWithNewOp<OrOp>(srcOp, orLHS, orRHS);
+    return success();
+  }
+};
+
+using WasmRotrOpConversion = RotateOpConversion<RotrOp, ShRUOp, ShLOp>;
+using WasmRotlOpConversion = RotateOpConversion<RotlOp, ShLOp, ShRUOp>;
+
+template <typename SourceOp, typename TargetOp, typename AttrType,
+          typename ValType, ValType flag>
+struct ComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto cmpRes =
+        TargetOp::create(rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+                         AttrType::get(rewriter.getContext(), flag),
+                         adaptor.getLhs(), adaptor.getRhs())
+            .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
+template <typename SourceOp, arith::CmpFPredicate compFlag>
+using FPComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpFOp, arith::CmpFPredicateAttr,
+                           arith::CmpFPredicate, compFlag>;
+
+template <typename SourceOp, arith::CmpIPredicate compFlag>
+using IntComparisonConversion =
+    ComparisonOpConversion<SourceOp, arith::CmpIOp, arith::CmpIPredicateAttr,
+                           arith::CmpIPredicate, compFlag>;
+
+using WasmLtSIOpConversion =
+    IntComparisonConversion<LtSIOp, arith::CmpIPredicate::slt>;
+using WasmLeSIOpConversion =
+    IntComparisonConversion<LeSIOp, arith::CmpIPredicate::sle>;
+using WasmGtSIOpConversion =
+    IntComparisonConversion<GtSIOp, arith::CmpIPredicate::sgt>;
+using WasmGeSIOpConversion =
+    IntComparisonConversion<GeSIOp, arith::CmpIPredicate::sge>;
+using WasmLtUIOpConversion =
+    IntComparisonConversion<LtUIOp, arith::CmpIPredicate::ult>;
+using WasmLeUIOpConversion =
+    IntComparisonConversion<LeUIOp, arith::CmpIPredicate::ule>;
+using WasmGtUIOpConversion =
+    IntComparisonConversion<GtUIOp, arith::CmpIPredicate::ugt>;
+using WasmGeUIOpConversion =
+    IntComparisonConversion<GeUIOp, arith::CmpIPredicate::uge>;
+using WasmLtOpConversion =
+    FPComparisonConversion<LtOp, arith::CmpFPredicate::OLT>;
+using WasmLeOpConversion =
+    FPComparisonConversion<LeOp, arith::CmpFPredicate::OLE>;
+using WasmGtOpConversion =
+    FPComparisonConversion<GtOp, arith::CmpFPredicate::OGT>;
+using WasmGeOpConversion =
+    FPComparisonConversion<GeOp, arith::CmpFPredicate::OGE>;
+
+template <typename SourceOp, arith::CmpIPredicate IntFlag,
+          arith::CmpFPredicate FloatFlag>
+struct IntFpComparisonOpConversion : OpConversionPattern<SourceOp> {
+  using OpConversionPattern<SourceOp>::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SourceOp srcOp, typename SourceOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    Value comparisonResult;
+    if (srcOp.getLhs().getType().isInteger())
+      comparisonResult =
+          arith::CmpIOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpIPredicateAttr::get(rewriter.getContext(), IntFlag),
+              adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else if (srcOp.getLhs().getType().isFloat())
+      comparisonResult =
+          arith::CmpFOp::create(
+              rewriter, srcOp.getLoc(), rewriter.getI1Type(),
+              arith::CmpFPredicateAttr::get(rewriter.getContext(), FloatFlag),
+              adaptor.getLhs(), adaptor.getRhs())
+              .getResult();
+    else
+      return rewriter.notifyMatchFailure(
+          srcOp.getLoc(), "Unsupported datatype for comparison OP.");
+
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(srcOp, rewriter.getI32Type(),
+                                                comparisonResult);
+    return success();
+  }
+};
+
+using WasmEqOpConversion =
+    IntFpComparisonOpConversion<EqOp, arith::CmpIPredicate::eq,
+                                arith::CmpFPredicate::OEQ>;
+using WasmNeOpConversion =
+    IntFpComparisonOpConversion<NeOp, arith::CmpIPredicate::ne,
+                                arith::CmpFPredicate::ONE>;
+
 struct WasmCallOpConversion : OpConversionPattern<FuncCallOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -148,6 +310,48 @@ struct WasmConstOpConversion : OpConversionPattern<ConstOp> {
   }
 };
 
+struct WasmEqzOpConversion : OpConversionPattern<EqzOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(EqzOp eqzOp, EqzOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = eqzOp->getLoc();
+    auto zero = arith::ConstantOp::create(
+                    rewriter, loc,
+                    rewriter.getIntegerAttr(adaptor.getInput().getType(), 0))
+                    .getResult();
+    auto cmpRes = arith::CmpIOp::create(
+                      rewriter, loc, rewriter.getI1Type(),
+                      arith::CmpIPredicateAttr::get(rewriter.getContext(),
+                                                    arith::CmpIPredicate::eq),
+                      adaptor.getInput(), zero)
+                      .getResult();
+    rewriter.replaceOpWithNewOp<arith::ExtUIOp>(eqzOp, rewriter.getI32Type(),
+                                                cmpRes);
+
+    return success();
+  }
+};
+
+struct WasmExtendLowBitsOpConversion : OpConversionPattern<ExtendLowBitsSOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(ExtendLowBitsSOp extendLowBytesSOp,
+                  ExtendLowBitsSOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto truncWidth = extendLowBytesSOp.getBitsToTake().getInt();
+    auto truncation = arith::TruncIOp::create(
+        rewriter, extendLowBytesSOp->getLoc(),
+        rewriter.getIntegerType(truncWidth), adaptor.getInput());
+    rewriter.replaceOpWithNewOp<arith::ExtSIOp>(
+        extendLowBytesSOp, extendLowBytesSOp.getResult().getType(),
+        truncation.getResult());
+    return success();
+  }
+};
+
 struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -163,6 +367,185 @@ struct WasmFuncImportOpConversion : OpConversionPattern<FuncImportOp> {
 
 struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
   using OpConversionPattern::OpConversionPattern;
+  ///
+  /// Control flow conversion needs shared state for tracking which block
+  /// corresponds to which operation at which level.
+  ///
+  /// This class handles such tracking and performs the conversion of control
+  /// flow related ops contained in a function.
+  class CFRewriterVisitor {
+  private:
+    using branch_to_dest_t = llvm::DenseMap<LabelBranchingOpInterface, Block *>;
+    Value getCompResultAsI1(Value compResult,
+                            ConversionPatternRewriter &rewriter) {
+      auto testValue = arith::ConstantOp::create(rewriter, compResult.getLoc(),
+                                                 rewriter.getI32IntegerAttr(0));
+      auto flag = arith::CmpIOp::create(
+                      rewriter, compResult.getLoc(), rewriter.getIntegerType(1),
+                      arith::CmpIPredicate::ne, compResult, testValue)
+                      .getResult();
+      return flag;
+    }
+
+    void replaceNestLevelWithBranch(BlockOp blockOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(blockOp, regionsToEntry[0],
+                                                blockOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(LoopOp loopOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(loopOp, regionsToEntry[0],
+                                                loopOp->getOperands());
+    }
+
+    void replaceNestLevelWithBranch(IfOp ifOp,
+                                    llvm::ArrayRef<Block *> regionsToEntry,
+                                    ConversionPatternRewriter &rewriter) {
+      Block *falseDest =
+          regionsToEntry.size() == 2 ? regionsToEntry[1] : ifOp.getTarget();
+      auto flag = getCompResultAsI1(ifOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          ifOp, flag, regionsToEntry[0], ifOp.getInputs(), falseDest,
+          ifOp.getInputs());
+    }
+
+    template <typename LevelType>
+    LogicalResult
+    replaceNestLevelWithBranchWrapper(LabelLevelOpInterface nestingOp,
+                                      llvm::ArrayRef<Block *> regionsToEntry,
+                                      ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelType>(nestingOp.getOperation());
+      if (!cast)
+        return failure();
+      replaceNestLevelWithBranch(cast, regionsToEntry, rewriter);
+      return success();
+    }
+
+    template <typename... LevelTypes>
+    LogicalResult inlineNestDispatcher(LabelLevelOpInterface nestingOp,
+                                       ConversionPatternRewriter &rewriter) {
+      auto sip = rewriter.saveInsertionPoint();
+      Block *blockSuccessor = nestingOp->getSuccessor(0);
+      llvm::SmallVector<Block *, 2> regionEntries;
+      LLVM_DEBUG(llvm::dbgs()
+                     << "Starting inlining blocks for " << nestingOp << "\n";);
+      for (auto &region : nestingOp->getRegions()) {
+        if (region.empty())
+          continue;
+        regionEntries.push_back(&region.front());
+        /// Inline blocks of nested ops
+        llvm::SmallVector<LabelLevelOpInterface> nestedOps{
+            region.getOps<LabelLevelOpInterface>()};
+        for (auto nestedOp : nestedOps) {
+          LLVM_DEBUG(llvm::dbgs() << " Found nested op: " << nestedOp);
+          if (failed(inlineBlocks(nestedOp, rewriter)))
+            return failure();
+        }
+        rewriter.inlineRegionBefore(region, blockSuccessor);
+      }
+      LLVM_DEBUG(llvm::dbgs() << "End of region inlining\n");
+      LLVM_DEBUG(llvm::dbgs() << "Replacing initial op with branching\n");
+      rewriter.setInsertionPoint(nestingOp);
+      auto res = success(
+          (... || succeeded(replaceNestLevelWithBranchWrapper<LevelTypes>(
+                      nestingOp, regionEntries, rewriter))));
+      rewriter.restoreInsertionPoint(sip);
+      if (failed(res))
+        return emitError(nestingOp->getLoc(),
+                         "Unable to inline the operation regions.");
+      return success();
+    }
+
+    /// Take a nesting level defining op and inline it in the parent region.
+    LogicalResult inlineBlocks(LabelLevelOpInterface nestingOp,
+                               ConversionPatternRewriter &rewriter) {
+      return inlineNestDispatcher<BlockOp, IfOp, LoopOp>(nestingOp, rewriter);
+    }
+
+    llvm::FailureOr<Block *> getBlockFor(LabelBranchingOpInterface branchOp) {
+      auto destIter = branchToDest.find(branchOp);
+      if (destIter == branchToDest.end())
+        return branchOp->emitError("No indexed label op for this operation: ")
+               << branchOp;
+      return destIter->second;
+    }
+
+    inline void convertBranch(BranchIfOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      auto flag = getCompResultAsI1(brOp.getCondition(), rewriter);
+      rewriter.replaceOpWithNewOp<cf::CondBranchOp>(
+          brOp, flag, dest, brOp.getInputs(), brOp.getElseSuccessor(),
+          ValueRange{});
+    }
+
+    inline void convertBranch(BlockReturnOp brOp, Block *dest,
+                              ConversionPatternRewriter &rewriter) {
+      rewriter.replaceOpWithNewOp<cf::BranchOp>(brOp, dest, brOp.getInputs());
+    }
+
+    template <typename LevelInterfaceT>
+    inline LogicalResult
+    convertBranchWrapper(LabelBranchingOpInterface branchOp, Block *dest,
+                         ConversionPatternRewriter &rewriter) {
+      auto cast = dyn_cast<LevelInterfaceT>(branchOp.getOperation());
+      if (!cast)
+        return failure();
+      auto sip = rewriter.saveInsertionPoint();
+      rewriter.setInsertionPoint(branchOp);
+      convertBranch(cast, dest, rewriter);
+      rewriter.restoreInsertionPoint(sip);
+      return success();
+    }
+
+    template <typename... BranchInterfaceT>
+    LogicalResult convertBranchDispatch(LabelBranchingOpInterface branchOp,
+                                        ConversionPatternRewriter &rewriter) {
+      auto dest = getBlockFor(branchOp);
+      if (failed(dest))
+        return failure();
+      auto res =
+          success((... || succeeded(convertBranchWrapper<BranchInterfaceT>(
+                              branchOp, *dest, rewriter))));
+      if (failed(res))
+        return emitError(branchOp->getLoc(), "No known converter for op ")
+               << branchOp;
+      return res;
+    }
+
+    LogicalResult convertBranch(LabelBranchingOpInterface branchOp,
+                                ConversionPatternRewriter &rewriter) {
+      return convertBranchDispatch<BlockReturnOp, BranchIfOp>(branchOp,
+                                                              rewriter);
+    }
+
+    func::FuncOp func;
+    branch_to_dest_t branchToDest;
+
+  public:
+    CFRewriterVisitor(func::FuncOp func) : func{func} {
+      func.walk([this](LabelBranchingOpInterface branchOp) {
+        branchToDest.insert({branchOp, branchOp.getTarget()});
+      });
+    }
+    LogicalResult rewrite(ConversionPatternRewriter &rewriter) {
+      llvm::SmallVector<LabelLevelOpInterface> nestingOps{
+          func.getOps<LabelLevelOpInterface>()};
+      for (auto nestingOp : nestingOps)
+        if (failed(inlineBlocks(nestingOp, rewriter)))
+          return failure();
+
+      auto res =
+          func->walk([this, &rewriter](LabelBranchingOpInterface branchOp) {
+            if (failed(convertBranch(branchOp, rewriter)))
+              return WalkResult::interrupt();
+            return WalkResult::advance();
+          });
+      return failure(res.wasInterrupted());
+    }
+  };
 
   LogicalResult
   matchAndRewrite(FuncOp funcOp, FuncOp::Adaptor adaptor,
@@ -185,7 +568,8 @@ struct WasmFuncOpConversion : OpConversionPattern<FuncOp> {
 
     rewriter.applySignatureConversion(oldEntryBlock, sC, getTypeConverter());
     rewriter.replaceOp(funcOp, newFunc);
-    return success();
+    CFRewriterVisitor cfRewriter{newFunc};
+    return cfRewriter.rewrite(rewriter);
   }
 };
 
@@ -287,6 +671,57 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(MemOp memOp, MemOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = memOp.getLoc();
+    auto bufferType =
+        MemRefType::get({ShapedType::kDynamic}, rewriter.getI8Type());
+    auto bufferPtrType = MemRefType::get({1}, bufferType);
+    auto memVisibility = memOp.getVisibility();
+    // Convert to StringAttr since memref::GlobalOp expects visibility as a
+    // string attribute.
+    mlir::StringAttr visAttr;
+    if (memVisibility == mlir::SymbolTable::Visibility::Public)
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "public");
+    else if (memVisibility == mlir::SymbolTable::Visibility::Private)
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "private");
+    else
+      visAttr = mlir::StringAttr::get(memOp->getContext(), "nested");
+
+    auto memPtr = rewriter.replaceOpWithNewOp<memref::GlobalOp>(
+        memOp, memOp.getSymNameAttr(), visAttr, TypeAttr::get(bufferPtrType),
+        /*initialValue*/ rewriter.getUnitAttr(),
+        /*constant*/ UnitAttr{}, /*alignment*/ IntegerAttr{});
+    auto initializerName = (memPtr.getSymName() + "::initializer").str();
+    auto memInitializer =
+        func::FuncOp::create(rewriter, loc, initializerName,
+                             FunctionType::get(getContext(), {}, {}));
+    memInitializer->setAttr(rewriter.getStringAttr("initializer"),
+                            rewriter.getUnitAttr());
+    auto *initializerBody = memInitializer.addEntryBlock();
+    auto sip = rewriter.saveInsertionPoint();
+    rewriter.setInsertionPointToStart(initializerBody);
+    auto memRefPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, bufferType), memPtr.getSymName());
+    auto alloc = memref::AllocOp::create(
+        rewriter, loc,
+        MemRefType::get({memOp.getLimits().getMin()}, rewriter.getI8Type()));
+    auto castOp =
+        memref::CastOp::create(rewriter, loc, bufferType, alloc.getResult());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    memref::StoreOp::create(rewriter, loc, castOp.getResult(),
+                            memRefPtr.getResult(), ValueRange{idx.getResult()});
+    func::ReturnOp::create(rewriter, loc);
+    rewriter.restoreInsertionPoint(sip);
+    func::CallOp::create(rewriter, loc, memInitializer);
+    return success();
+  }
+};
+
 inline TypedAttr getInitializerAttr(Type t) {
   assert(t.isIntOrFloat() &&
          "This helper is intended to use with int and float types");
@@ -429,21 +864,38 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmDivFPOpConversion,
            WasmDivSIOpConversion,
            WasmDivUIOpConversion,
+           WasmEqOpConversion,
+           WasmEqzOpConversion,
+           WasmExtendLowBitsOpConversion,
            WasmExtendSOpConversion,
            WasmExtendUOpConversion,
            WasmFloorOpConversion,
            WasmFuncImportOpConversion,
            WasmFuncOpConversion,
+           WasmGeOpConversion,
+           WasmGeSIOpConversion,
+           WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
+           WasmGtOpConversion,
+           WasmGtSIOpConversion,
+           WasmGtUIOpConversion,
+           WasmLeOpConversion,
+           WasmLeSIOpConversion,
+           WasmLeUIOpConversion,
            WasmLocalConversion,
            WasmLocalGetConversion,
            WasmLocalSetConversion,
            WasmLocalTeeConversion,
+           WasmLtOpConversion,
+           WasmLtSIOpConversion,
+           WasmLtUIOpConversion,
            WasmMaxOpConversion,
+           WasmMemoryOpConversion,
            WasmMinOpConversion,
            WasmMulOpConversion,
+           WasmNeOpConversion,
            WasmNegOpConversion,
            WasmOrOpConversion,
            WasmPopCntOpConversion,
@@ -452,6 +904,8 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmRemSIOpConversion,
            WasmRemUIOpConversion,
            WasmReturnOpConversion,
+           WasmRotlOpConversion,
+           WasmRotrOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
new file mode 100644
index 0000000000000..fdce71cab096b
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-blocks-to-cf.mlir
@@ -0,0 +1,366 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK_CANONICALIZED %s
+
+// CHECK-LABEL:   func.func @i_am_a_block(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 17 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 42 : i32
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_3:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @i_am_a_block(
+// CHECK_CANONICALIZED-SAME:      %[[VAL_0:.*]]: i32) -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_1:.*]] = arith.constant 42 : i32
+// CHECK_CANONICALIZED:           %[[VAL_3:.*]] = memref.alloca() : memref<i32>
+// CHECK_CANONICALIZED:           memref.store %[[VAL_1]], %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED-NOT:       memref.store
+// CHECK_CANONICALIZED:           %[[VAL_4:.*]] = memref.load %[[VAL_3]][] : memref<i32>
+// CHECK_CANONICALIZED:           return %[[VAL_4]] : i32
+wasmssa.func @i_am_a_block(%arg0 : !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.const 17 : i32
+  wasmssa.local_set %arg0 : ref to i32 to %1 : i32
+  wasmssa.block : {
+    %2 = wasmssa.const 42 : i32
+    wasmssa.local_set %arg0 : ref to i32 to %2 : i32
+    wasmssa.block_return
+  }> ^bb1
+  ^bb1:
+  %res = wasmssa.local_get %arg0 : ref to i32
+  wasmssa.return %res : i32
+}
+
+
+// CHECK-LABEL:   func.func @func_0(
+// CHECK-SAME:                      %[[ARG0:.*]]: i32) {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func @func_0(%arg0: !wasmssa<local ref to i32>) {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block : {
+      %3 = wasmssa.const 3: i32
+      wasmssa.block : {
+        %4 = wasmssa.const 4: i32
+        wasmssa.block_return
+      }> ^bb1
+    ^bb1:  // pred: ^bb0
+      %5 = wasmssa.const 5: i32
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_1() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_2:.*]] = arith.constant 3 : i32
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           %[[VAL_3:.*]] = arith.constant 4 : i32
+// CHECK:           cf.br ^bb4
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 5 : i32
+// CHECK:           cf.br ^bb5
+// CHECK:         ^bb5:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 6 : i32
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           %[[VAL_6:.*]] = arith.constant 7 : i32
+// CHECK:           return
+wasmssa.func @func_1() {
+  %1 = wasmssa.const 1: i32
+  wasmssa.block : {
+    %2 = wasmssa.const 2: i32
+    wasmssa.block_return
+  }> ^bb1
+^bb1:  // pred: ^bb0
+  %3 = wasmssa.const 3: i32
+  wasmssa.block : {
+    %4 = wasmssa.const 4: i32
+    wasmssa.block_return
+  }> ^bb2
+^bb2:  // pred: ^bb1
+  %5 = wasmssa.const 5: i32
+  wasmssa.block : {
+    %6 = wasmssa.const 6: i32
+    wasmssa.block_return
+  }> ^bb3
+^bb3:  // pred: ^bb2
+  %7 = wasmssa.const 7: i32
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 14 : i32
+// CHECK:           cf.br ^bb1(%[[VAL_0]] : i32)
+// CHECK:         ^bb1(%[[VAL_1:.*]]: i32):
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.addi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_3]] : i32)
+// CHECK:         ^bb2(%[[VAL_4:.*]]: i32):
+// CHECK:           return %[[VAL_4]] : i32
+wasmssa.func @func_2() -> i32 {
+  %0 = wasmssa.const 14 : i32
+  wasmssa.block(%0) : i32 : {
+  ^bb0(%arg0: i32):
+    %2 = wasmssa.const 1 : i32
+    %3 = wasmssa.add %arg0 %2 : i32
+    wasmssa.block_return %3 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK:           cf.br ^bb2(%[[VAL_0]] : i32)
+// CHECK:         ^bb2(%[[VAL_1:.*]]: i32):
+// CHECK:           return %[[VAL_1]] : i32
+wasmssa.func @func_3() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 17 : i32
+    wasmssa.block_return %1 : i32
+  }> ^bb1
+^bb1(%arg0: i32):
+  wasmssa.return %arg0 : i32
+}
+
+//// ============= Branch instructions etc ==========
+
+// CHECK-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_taken() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+
+wasmssa.func @branch_if_taken() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 2 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.cmpi ne, %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           cf.cond_br %[[VAL_3]], ^bb3(%[[VAL_0]] : i32), ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_4:.*]] = arith.constant 16 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.addi %[[VAL_0]], %[[VAL_4]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb3(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+
+// CHECK_CANONICALIZED-LABEL:   func.func @branch_if_continue() -> i32 {
+// CHECK_CANONICALIZED:           %[[VAL_0:.*]] = arith.constant 17 : i32
+// CHECK_CANONICALIZED:           return %[[VAL_0]] : i32
+// CHECK_CANONICALIZED:         }
+wasmssa.func @branch_if_continue() -> i32 {
+  wasmssa.block : {
+    %1 = wasmssa.const 1 : i32
+    %2 = wasmssa.const 0 : i32
+    wasmssa.branch_if %2 to level 0 with args(%1 : i32) else ^bb1
+  ^bb1:  // pred: ^bb0
+    %3 = wasmssa.const 16 : i32
+    %4 = wasmssa.add %1 %3 : i32
+    wasmssa.block_return %4 : i32
+  }> ^bb1
+^bb1(%0: i32):  // pred: ^bb0
+  wasmssa.return %0 : i32
+}
+
+// CHECK-LABEL:   func.func @if(
+// CHECK-SAME:                  %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_3:.*]] = arith.andi %[[VAL_1]], %[[VAL_2]] : i32
+// CHECK:           %[[VAL_4:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_5:.*]] = arith.cmpi ne, %[[VAL_3]], %[[VAL_4]] : i32
+// CHECK:           cf.cond_br %[[VAL_5]], ^bb1, ^bb2
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 3 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.muli %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_10:.*]] = arith.addi %[[VAL_8]], %[[VAL_9]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_10]] : i32)
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_11:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_12:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_13:.*]] = arith.shrui %[[VAL_11]], %[[VAL_12]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_13]] : i32)
+// CHECK:         ^bb3(%[[VAL_14:.*]]: i32):
+// CHECK:           return %[[VAL_14]] : i32
+wasmssa.func @if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.const 1 : i32
+  %3 = wasmssa.and %1 %2 : i32
+  "wasmssa.if"(%3)[^bb1] ({
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 3 : i32
+    %7 = wasmssa.mul %5 %6 : i32
+    %8 = wasmssa.const 1 : i32
+    %9 = wasmssa.add %7 %8 : i32
+    wasmssa.block_return %9 : i32
+  }, {
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    wasmssa.block_return %7 : i32
+  }) : (i32) -> ()
+^bb1(%4: i32):  // pred: ^bb0
+  wasmssa.return %4 : i32
+}
+
+// CHECK-LABEL:   func.func @if_else(
+// CHECK-SAME:      %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.andi %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.cmpi ne, %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           cf.cond_br %[[VAL_6]], ^bb1(%[[VAL_1]] : i32), ^bb2(%[[VAL_1]] : i32)
+// CHECK:         ^bb1(%[[VAL_7:.*]]: i32):
+// CHECK:           %[[VAL_8:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_9:.*]] = arith.addi %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           cf.br ^bb2(%[[VAL_9]] : i32)
+// CHECK:         ^bb2(%[[VAL_10:.*]]: i32):
+// CHECK:           return %[[VAL_10]] : i32
+wasmssa.func @if_else(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.local_get %arg0 : ref to i32
+  %3 = wasmssa.const 1 : i32
+  %4 = wasmssa.and %2 %3 : i32
+  "wasmssa.if"(%4, %1)[^bb1] ({
+  ^bb0(%arg1: i32):
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.add %arg1 %6 : i32
+    wasmssa.block_return %7 : i32
+  }, {
+  }) : (i32, i32) -> ()
+^bb1(%5: i32):  // pred: ^bb0
+  wasmssa.return %5 : i32
+}
+
+// CHECK-LABEL:   func.func @if_if(
+// CHECK-SAME:                     %[[ARG0:.*]]: i32) -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           memref.store %[[ARG0]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = math.cttz %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi ne, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           cf.cond_br %[[VAL_4]], ^bb1, ^bb4
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_5:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_6:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_7:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_8:.*]] = arith.shrui %[[VAL_6]], %[[VAL_7]] : i32
+// CHECK:           %[[VAL_9:.*]] = math.cttz %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_11:.*]] = arith.cmpi ne, %[[VAL_9]], %[[VAL_10]] : i32
+// CHECK:           cf.cond_br %[[VAL_11]], ^bb2(%[[VAL_5]] : i32), ^bb3(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_12:.*]]: i32):
+// CHECK:           %[[VAL_13:.*]] = arith.constant 2 : i32
+// CHECK:           %[[VAL_14:.*]] = arith.addi %[[VAL_12]], %[[VAL_13]] : i32
+// CHECK:           cf.br ^bb3(%[[VAL_14]] : i32)
+// CHECK:         ^bb3(%[[VAL_15:.*]]: i32):
+// CHECK:           cf.br ^bb5(%[[VAL_15]] : i32)
+// CHECK:         ^bb4:
+// CHECK:           %[[VAL_16:.*]] = arith.constant 1 : i32
+// CHECK:           cf.br ^bb5(%[[VAL_16]] : i32)
+// CHECK:         ^bb5(%[[VAL_17:.*]]: i32):
+// CHECK:           return %[[VAL_17]] : i32
+wasmssa.func @if_if(%arg0: !wasmssa<local ref to i32>) -> i32 {
+  %1 = wasmssa.local_get %arg0 : ref to i32
+  %2 = wasmssa.ctz %1 : i32
+  "wasmssa.if"(%2)[^bb1] ({
+    %4 = wasmssa.const 2 : i32
+    %5 = wasmssa.local_get %arg0 : ref to i32
+    %6 = wasmssa.const 1 : i32
+    %7 = wasmssa.shr_u %5 by %6 bits : i32
+    %8 = wasmssa.ctz %7 : i32
+    "wasmssa.if"(%8, %4)[^bb1] ({
+    ^bb0(%arg1: i32):
+      %10 = wasmssa.const 2 : i32
+      %11 = wasmssa.add %arg1 %10 : i32
+      wasmssa.block_return %11 : i32
+    }, {
+    }) : (i32, i32) -> ()
+  ^bb1(%9: i32):  // pred: ^bb0
+    wasmssa.block_return %9 : i32
+  }, {
+    %4 = wasmssa.const 1 : i32
+    wasmssa.block_return %4 : i32
+  }) : (i32) -> ()
+^bb1(%3: i32):  // pred: ^bb0
+  wasmssa.return %3 : i32
+}

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..6643b0389f405
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-comparisons-to-arith-cmp.mlir
@@ -0,0 +1,421 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+
+module {
+  wasmssa.func @func_lt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+
+  wasmssa.func @func_le_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_lt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.lt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_le_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.le_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.gt_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_si32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_si %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_ui32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ge_ui %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_lt_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_le_si64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_lt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.lt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_le_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.le_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.gt_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_si_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_si %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_ui_i64() -> i32 {
+    %0 = wasmssa.const 12 : i64
+    %1 = wasmssa.const 50 : i64
+    %2 = wasmssa.ge_ui %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_lt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.lt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_le_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.le %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.gt %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ge %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_lt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.lt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_le_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.le %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_gt_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.gt %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ge_f64() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f64
+    %1 = wasmssa.const 1.400000e+01 : f64
+    %2 = wasmssa.ge %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_eq_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.eq %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_eq_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.eq %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_eq_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.eq %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_eq_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.eq %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+    wasmssa.func @func_ne_i32() -> i32 {
+    %0 = wasmssa.const 12 : i32
+    %1 = wasmssa.const 50 : i32
+    %2 = wasmssa.ne %0 %1 : i32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ne_i64() -> i32 {
+    %0 = wasmssa.const 20 : i64
+    %1 = wasmssa.const 5 : i64
+    %2 = wasmssa.ne %0 %1 : i64 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ne_f32() -> i32 {
+    %0 = wasmssa.const 5.000000e+00 : f32
+    %1 = wasmssa.const 1.400000e+01 : f32
+    %2 = wasmssa.ne %0 %1 : f32 -> i32
+    wasmssa.return %2 : i32
+  }
+  wasmssa.func @func_ne_f64() -> i32 {
+    %0 = wasmssa.const 1.700000e+01 : f64
+    %1 = wasmssa.const 0.000000e+00 : f64
+    %2 = wasmssa.ne %0 %1 : f64 -> i32
+    wasmssa.return %2 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_lt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi slt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_si64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sle, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ult, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ule, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sgt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ugt, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_si_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi sge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_ui_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi uge, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_lt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf olt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_le_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ole, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_gt_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf ogt, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ge_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oge, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_eq_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf oeq, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 50 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 20 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 5 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi ne, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 5.000000e+00 : f32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 1.400000e+01 : f32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @func_ne_f64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 1.700000e+01 : f64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0.000000e+00 : f64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpf one, %[[VAL_0]], %[[VAL_1]] : f64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
new file mode 100644
index 0000000000000..083a24c5f36a8
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-eqz-to-arith-cmp.mlir
@@ -0,0 +1,28 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+module {
+  wasmssa.func @eqz_i32() -> i32 {
+    %0 = wasmssa.const 13 : i32
+    %1 = wasmssa.eqz %0 : i32 -> i32
+    wasmssa.return %1 : i32
+  }
+  wasmssa.func @eqz_i64() -> i32 {
+    %0 = wasmssa.const 13 : i64
+    %1 = wasmssa.eqz %0 : i64 -> i32
+    wasmssa.return %1 : i32
+  }
+}
+
+// CHECK-LABEL:   func.func @eqz_i32() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i32
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32
+
+// CHECK-LABEL:   func.func @eqz_i64() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 13 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i64
+// CHECK:           %[[VAL_2:.*]] = arith.cmpi eq, %[[VAL_0]], %[[VAL_1]] : i64
+// CHECK:           %[[VAL_3:.*]] = arith.extui %[[VAL_2]] : i1 to i32
+// CHECK:           return %[[VAL_3]] : i32

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
new file mode 100644
index 0000000000000..2b8bb651bbe5b
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-extend-to-arith-ext.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @func_0() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extsi %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func @func_0() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_s %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_1() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.extui %[[VAL_0]] : i32 to i64
+// CHECK:           return %[[VAL_1]] : i64
+wasmssa.func @func_1() -> i64 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend_i32_u %0 to i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_2() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func @func_2() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 8 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_3() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i32 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i32
+// CHECK:           return %[[VAL_2]] : i32
+wasmssa.func @func_3() -> i32 {
+  %0 = wasmssa.const 10 : i32
+  %1 = wasmssa.extend 16 low bits from %0: i32
+  wasmssa.return %1 : i32
+}
+
+// CHECK-LABEL:   func.func @func_4() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i8
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i8 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func @func_4() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 8 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_5() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i16
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i16 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func @func_5() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 16 low bits from %0: i64
+  wasmssa.return %1 : i64
+}
+
+// CHECK-LABEL:   func.func @func_6() -> i64 {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 10 : i64
+// CHECK:           %[[VAL_1:.*]] = arith.trunci %[[VAL_0]] : i64 to i32
+// CHECK:           %[[VAL_2:.*]] = arith.extsi %[[VAL_1]] : i32 to i64
+// CHECK:           return %[[VAL_2]] : i64
+wasmssa.func @func_6() -> i64 {
+  %0 = wasmssa.const 10 : i64
+  %1 = wasmssa.extend 32 low bits from %0: i64
+  wasmssa.return %1 : i64
+}

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
new file mode 100644
index 0000000000000..cdaf249d9ccf6
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-loop-to-cf.mlir
@@ -0,0 +1,201 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir --canonicalize -o - | FileCheck --check-prefix=CHECK-CANONICAL %s
+
+module {
+  wasmssa.func @func_0() {
+    wasmssa.loop : {
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   module {
+// CHECK:           func.func @func_0() {
+// CHECK:             cf.br ^bb1
+// CHECK:           ^bb1:
+// CHECK:             cf.br ^bb2
+// CHECK:           ^bb2:
+// CHECK:             return
+// CHECK:           }
+
+// CHECK-CANONICAL-LABEL:  func.func @func_0() {
+// CHECK-CANONICAL:             return
+// CHECK-CANONICAL:           }
+
+// -----
+
+module {
+  wasmssa.func @func_0() -> i32 {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.block_return %3 : i32
+    }> ^bb1
+  ^bb1(%1: i32):  // pred: ^bb0
+    wasmssa.return %1 : i32
+  }
+}
+// CHECK-LABEL:   func.func @func_0() -> i32 {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           cf.br ^bb2(%[[VAL_5]] : i32)
+// CHECK:         ^bb2(%[[VAL_6:.*]]: i32):
+// CHECK:           return %[[VAL_6]] : i32
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() -> i32 {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK-CANONICAL:           return %[[VAL_5]] : i32
+
+// -----
+
+module {
+  wasmssa.func @func_0() {
+    %0 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %1 = wasmssa.local_get %0 : ref to i32
+      %2 = wasmssa.const 10 : i32
+      %3 = wasmssa.lt_si %1 %2 : i32 -> i32
+      wasmssa.branch_if %3 to level 0 else ^bb1
+    ^bb1:  // pred: ^bb0
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_2:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_2]], %[[VAL_3]] : i32
+// CHECK:           %[[VAL_5:.*]] = arith.extui %[[VAL_4]] : i1 to i32
+// CHECK:           %[[VAL_6:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_7:.*]] = arith.cmpi ne, %[[VAL_5]], %[[VAL_6]] : i32
+// CHECK:           cf.cond_br %[[VAL_7]], ^bb1, ^bb2
+// CHECK:         ^bb2:
+// CHECK:           cf.br ^bb3
+// CHECK:         ^bb3:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[VAL_0:.*]] = arith.constant 10 : i32
+// CHECK-CANONICAL:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[VAL_1]], %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[VAL_3:.*]] = memref.load %[[VAL_2]][] : memref<i32>
+// CHECK-CANONICAL:           %[[VAL_4:.*]] = arith.cmpi slt, %[[VAL_3]], %[[VAL_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[VAL_4]], ^bb1, ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           return
+
+// -----
+
+module {
+  wasmssa.func @func_0() {
+    %0 = wasmssa.local of type i32
+    %1 = wasmssa.local of type i32
+    wasmssa.loop : {
+      %2 = wasmssa.local_get %0 : ref to i32
+      %3 = wasmssa.const 1 : i32
+      %4 = wasmssa.add %2 %3 : i32
+      wasmssa.local_set %0 : ref to i32 to %4 : i32
+      wasmssa.loop : {
+        %8 = wasmssa.const 12 : i32
+        %9 = wasmssa.local_get %0 : ref to i32
+        %10 = wasmssa.gt_si %8 %9 : i32 -> i32
+        wasmssa.branch_if %10 to level 0 else ^bb1
+      ^bb1:  // pred: ^bb0
+        wasmssa.block_return %8 : i32
+      }> ^bb1
+    ^bb1(%5: i32):  // pred: ^bb0
+      %6 = wasmssa.const 10 : i32
+      %7 = wasmssa.lt_si %5 %6 : i32 -> i32
+      wasmssa.branch_if %7 to level 0 else ^bb2
+    ^bb2:  // pred: ^bb1
+      wasmssa.block_return
+    }> ^bb1
+  ^bb1:  // pred: ^bb0
+    wasmssa.return
+  }
+}
+
+// CHECK-LABEL:   func.func @func_0() {
+// CHECK:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_1:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_1]], %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_2:.*]] = memref.alloca() : memref<i32>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : i32
+// CHECK:           memref.store %[[VAL_3]], %[[VAL_2]][] : memref<i32>
+// CHECK:           cf.br ^bb1
+// CHECK:         ^bb1:
+// CHECK:           %[[VAL_4:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_5:.*]] = arith.constant 1 : i32
+// CHECK:           %[[VAL_6:.*]] = arith.addi %[[VAL_4]], %[[VAL_5]] : i32
+// CHECK:           memref.store %[[VAL_6]], %[[VAL_0]][] : memref<i32>
+// CHECK:           cf.br ^bb2
+// CHECK:         ^bb2:
+// CHECK:           %[[VAL_7:.*]] = arith.constant 12 : i32
+// CHECK:           %[[VAL_8:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK:           %[[VAL_9:.*]] = arith.cmpi sgt, %[[VAL_7]], %[[VAL_8]] : i32
+// CHECK:           %[[VAL_10:.*]] = arith.extui %[[VAL_9]] : i1 to i32
+// CHECK:           %[[VAL_11:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_12:.*]] = arith.cmpi ne, %[[VAL_10]], %[[VAL_11]] : i32
+// CHECK:           cf.cond_br %[[VAL_12]], ^bb2, ^bb3
+// CHECK:         ^bb3:
+// CHECK:           cf.br ^bb4(%[[VAL_7]] : i32)
+// CHECK:         ^bb4(%[[VAL_13:.*]]: i32):
+// CHECK:           %[[VAL_14:.*]] = arith.constant 10 : i32
+// CHECK:           %[[VAL_15:.*]] = arith.cmpi slt, %[[VAL_13]], %[[VAL_14]] : i32
+// CHECK:           %[[VAL_16:.*]] = arith.extui %[[VAL_15]] : i1 to i32
+// CHECK:           %[[VAL_17:.*]] = arith.constant 0 : i32
+// CHECK:           %[[VAL_18:.*]] = arith.cmpi ne, %[[VAL_16]], %[[VAL_17]] : i32
+// CHECK:           cf.cond_br %[[VAL_18]], ^bb1, ^bb5
+// CHECK:         ^bb5:
+// CHECK:           cf.br ^bb6
+// CHECK:         ^bb6:
+// CHECK:           return
+// CHECK:         }
+
+// CHECK-CANONICAL-LABEL:   func.func @func_0() {
+// CHECK-CANONICAL:           %[[CONSTANT_0:.*]] = arith.constant 12 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_1:.*]] = arith.constant 1 : i32
+// CHECK-CANONICAL:           %[[CONSTANT_2:.*]] = arith.constant 0 : i32
+// CHECK-CANONICAL:           %[[ALLOCA_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-CANONICAL:           memref.store %[[CONSTANT_2]], %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[LOAD_0:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[ADDI_0:.*]] = arith.addi %[[LOAD_0]], %[[CONSTANT_1]] : i32
+// CHECK-CANONICAL:           memref.store %[[ADDI_0]], %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           cf.br ^bb1
+// CHECK-CANONICAL:         ^bb1:
+// CHECK-CANONICAL:           %[[LOAD_1:.*]] = memref.load %[[ALLOCA_0]][] : memref<i32>
+// CHECK-CANONICAL:           %[[CMPI_0:.*]] = arith.cmpi slt, %[[LOAD_1]], %[[CONSTANT_0]] : i32
+// CHECK-CANONICAL:           cf.cond_br %[[CMPI_0]], ^bb1, ^bb2
+// CHECK-CANONICAL:         ^bb2:
+// CHECK-CANONICAL:           return
+// CHECK-CANONICAL:         }

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
new file mode 100644
index 0000000000000..09b07184d97fa
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-memory-to-memref.mlir
@@ -0,0 +1,14 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+"wasmssa.memory"() <{limits = !wasmssa<limit[17:]>, sym_name = "mem_0", sym_visibility = "nested"}> : () -> ()
+
+// CHECK-LABEL:   memref.global "nested" @mem_0 : memref<1xmemref<?xi8>>
+
+// CHECK-LABEL:   func.func @"mem_0::initializer"() attributes {initializer} {
+// CHECK:           %[[VAL_0:.*]] = memref.get_global @mem_0 : memref<1xmemref<?xi8>>
+// CHECK:           %[[VAL_1:.*]] = memref.alloc() : memref<17xi8>
+// CHECK:           %[[VAL_2:.*]] = memref.cast %[[VAL_1]] : memref<17xi8> to memref<?xi8>
+// CHECK:           %[[VAL_3:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_2]], %[[VAL_0]]{{\[}}%[[VAL_3]]] : memref<1xmemref<?xi8>>
+// CHECK:           return
+// CHECK:         func.call @"mem_0::initializer"() : () -> ()

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
new file mode 100644
index 0000000000000..b69f88afd4d7c
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotl-to-arith.mlir
@@ -0,0 +1,75 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotl %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotl_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val << (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i32
+
+// (val >> (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+wasmssa.func @rotl_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotl_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val << (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shli %[[VAL]], %[[LHS_AND]] : i64
+
+// (val >> (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shrui %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+wasmssa.func @rotl_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotl %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
new file mode 100644
index 0000000000000..f0b77271c86dd
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-rotr-to-arith.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt --split-input-file %s --raise-wasm-mlir -o - | FileCheck %s
+
+// Given:
+// %res = wasmssa.rotr %val by %bits bits : i32
+//
+// Produce:
+// res = (val >> (bits & 31)) | (val << (-bits & 31))
+
+// CHECK-LABEL:   func.func @rotr_i32(
+// CHECK-SAME:      %[[VALREF:.*]]: i32,
+// CHECK-SAME:      %[[BITSREF:.*]]: i32) -> i32 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i32>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i32>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i32>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i32>
+
+// (val >> (bits & 31))
+// CHECK:           %[[THIRTY_ONE:.*]] = arith.constant 31 : i32
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i32
+
+// (val << (-bits & 31))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i32
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[THIRTY_ONE]] : i32
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i32
+
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i32
+// CHECK:           return %[[RES]] : i32
+// CHECK:         }
+wasmssa.func @rotr_i32(%arg0: !wasmssa<local ref to i32>, %arg1: !wasmssa<local ref to i32>) -> i32 {
+    %v0 = wasmssa.local_get %arg0 : ref to i32
+    %v1 = wasmssa.local_get %arg1 : ref to i32
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i32
+    wasmssa.return %op : i32
+}
+
+// Same as above, but with 64 bits.
+// CHECK-LABEL:   func.func @rotr_i64(
+// CHECK-SAME:      %[[VALREF:.*]]: i64,
+// CHECK-SAME:      %[[BITSREF:.*]]: i64) -> i64 {
+
+// Storage etc
+// CHECK-DAG:           %[[VAL_1:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           %[[VAL_0:.*]] = memref.alloca() : memref<i64>
+// CHECK-DAG:           memref.store %[[VALREF]], %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           memref.store %[[BITSREF]], %[[VAL_1]][] : memref<i64>
+// CHECK-DAG:           %[[VAL:.*]] = memref.load %[[VAL_0]][] : memref<i64>
+// CHECK-DAG:           %[[BITS:.*]] = memref.load %[[VAL_1]][] : memref<i64>
+
+// (val >> (bits & 63))
+// CHECK:           %[[SIXTY_THREE:.*]] = arith.constant 63 : i64
+// CHECK:           %[[LHS_AND:.*]] = arith.andi %[[BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHRU:.*]] = arith.shrui %[[VAL]], %[[LHS_AND]] : i64
+
+// (val << (-bits & 63))
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i64
+// CHECK:           %[[NEG_BITS:.*]] = arith.subi %[[ZERO]], %[[BITS]] : i64
+// CHECK:           %[[RHS_AND:.*]] = arith.andi %[[NEG_BITS]], %[[SIXTY_THREE]] : i64
+// CHECK:           %[[SHL:.*]] = arith.shli %[[VAL]], %[[RHS_AND]] : i64
+
+// Form final result.
+// CHECK:           %[[RES:.*]] = arith.ori %[[SHRU]], %[[SHL]] : i64
+// CHECK:           return %[[RES]] : i64
+// CHECK:         }
+wasmssa.func @rotr_i64(%arg0: !wasmssa<local ref to i64>, %arg1: !wasmssa<local ref to i64>) -> i64 {
+    %v0 = wasmssa.local_get %arg0 : ref to i64
+    %v1 = wasmssa.local_get %arg1 : ref to i64
+
+    %op = wasmssa.rotr %v0 by %v1 bits : i64
+    wasmssa.return %op : i64
+}


        

From llvmlistbot at llvm.org  Wed Jul 22 04:07:45 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Wed, 22 Jul 2026 04:07:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WASM] Introduce full support for
 raising WASM MLIR to other dialects (PR #205990)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205990@github.com>
Message-ID: <6a60a481.7270d2fd.1ec500.45ad@mx.google.com>


https://github.com/flemairen6 closed https://github.com/llvm/llvm-project/pull/205990

From llvmlistbot at llvm.org  Wed Jul 22 04:15:05 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 22 Jul 2026 04:15:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a60a639.a1accbb4.1ace1c.de33@mx.google.com>


================
@@ -1313,6 +1313,38 @@ ::mlir::MLIRContext *ctx = dict.getContext();
 (void)ctx;
 )decl";
 
+  // `operandSegmentSizes`/`resultSegmentSizes` are trait-injected properties
+  // not enumerated by `op.getProperties()`, so they need to be special-cased
+  // here, mirroring the handling in `setPropertiesFromAttr` in
+  // OpDefinitionsGen.cpp.
+  //
+  // {0}: segment sizes property name
+  // {1}: isRequired
+  const char *segmentSizesFromAttrFmt = R"decl(
+auto {0}AttrName = ::mlir::StringAttr::get(ctx, "{0}");
+usedKeys.insert({0}AttrName);
+auto attr = dict.get({0}AttrName);
+if (!attr && {1}) {{
+  emitError() << "expected key entry for {0} in DictionaryAttr to set "
+             "Properties.";
+  return ::mlir::failure();
+}
+if (attr && ::mlir::failed(::mlir::convertFromAttribute(prop.{0}, attr, [&]() {{
+      return emitError() << "for `{0}`: ";
+    })))
+  return ::mlir::failure();
+)decl";
+  if (op.getTrait("::mlir::OpTrait::AttrSizedOperandSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "operandSegmentSizes",
+                    fmt.allOperands);
+  }
+  if (op.getTrait("::mlir::OpTrait::AttrSizedResultSegments")) {
+    auto scope = body.scope("{\n", "}\n", /*indent=*/true);
+    body << formatv(segmentSizesFromAttrFmt, "resultSegmentSizes",
+                    fmt.allResultTypes);
+  }
----------------
joker-eph wrote:

I think this should only be done when these aren't already inferred. 

https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 04:15:07 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 04:15:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale NumPy detection
	message (PR #211235)
Message-ID: <llvm.org/llvm/llvm-project/pull/211235@github.com>


https://github.com/anutosh491 created https://github.com/llvm/llvm-project/pull/211235

NumPy stopped being requested as a CMake Python component when its unused build-time dependency was removed in #108465  . Remove the remaining status message, which now prints empty version and include directory values.

>From a6199bce8c2be3593892db42c3062ae3c806ab69 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 16:43:22 +0530
Subject: [PATCH] [MLIR][Python] Remove stale NumPy detection message

---
 mlir/cmake/modules/MLIRDetectPythonEnv.cmake | 1 -
 1 file changed, 1 deletion(-)

diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..325144bd82dd5 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -52,7 +52,6 @@ macro(mlir_configure_python_dev_packages)
     endif()
     message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
     message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
-    message(STATUS "Found numpy v${Python3_NumPy_VERSION}: ${Python3_NumPy_INCLUDE_DIRS}")
     message(STATUS "Python extension suffix for modules: '${Python3_SOABI}'")
     if(nanobind_DIR)
       message(STATUS "Using explicit nanobind cmake directory: ${nanobind_DIR} (-Dnanobind_DIR to change)")


From llvmlistbot at llvm.org  Wed Jul 22 04:15:31 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Wed, 22 Jul 2026 04:15:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a60a653.27b3b003.207f1a.a54b@mx.google.com>


joker-eph wrote:

Thanks for the fix.

Please at least remove the "Summary" title (this is just AI noise).



https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 04:15:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 04:15:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale NumPy detection
	message (PR #211235)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211235@github.com>
Message-ID: <6a60a65f.743400b0.25573c.00c2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Anutosh Bhat (anutosh491)

<details>
<summary>Changes</summary>

NumPy stopped being requested as a CMake Python component when its unused build-time dependency was removed in #<!-- -->108465  . Remove the remaining status message, which now prints empty version and include directory values.

---
Full diff: https://github.com/llvm/llvm-project/pull/211235.diff


1 Files Affected:

- (modified) mlir/cmake/modules/MLIRDetectPythonEnv.cmake (-1) 


``````````diff
diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..325144bd82dd5 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -52,7 +52,6 @@ macro(mlir_configure_python_dev_packages)
     endif()
     message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
     message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
-    message(STATUS "Found numpy v${Python3_NumPy_VERSION}: ${Python3_NumPy_INCLUDE_DIRS}")
     message(STATUS "Python extension suffix for modules: '${Python3_SOABI}'")
     if(nanobind_DIR)
       message(STATUS "Using explicit nanobind cmake directory: ${nanobind_DIR} (-Dnanobind_DIR to change)")

``````````

</details>


https://github.com/llvm/llvm-project/pull/211235

From llvmlistbot at llvm.org  Wed Jul 22 04:18:55 2026
From: llvmlistbot at llvm.org (Zmicier Prybysh)
Date: Wed, 22 Jul 2026 04:18:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Allow extra DPS inputs in
 convolution dim inference (PR #198462)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198462@github.com>
Message-ID: <6a60a71f.0c05e68e.81651.e6e2@mx.google.com>


dimp-pl wrote:

Hi guys, checking in as a person interested in the downstream PR (iree-org/iree#24016) that's blocked by this PR.
@Tameem-10xE could you rebase this PR to resolve the conflict? I'll be happy to help if you don't have the time right now.
@hanhanW is there anything else you'd like to see done before you can approve this PR?

https://github.com/llvm/llvm-project/pull/198462

From llvmlistbot at llvm.org  Wed Jul 22 04:44:28 2026
From: llvmlistbot at llvm.org (Natanael Cintean)
Date: Wed, 22 Jul 2026 04:44:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir]Add
 resultSegmentSizes/operandSegmentSizes to prop-dict. (PR #211222)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211222@github.com>
Message-ID: <6a60ad1c.5a0412d2.325e39.d342@mx.google.com>


https://github.com/natanael-cintean edited https://github.com/llvm/llvm-project/pull/211222

From llvmlistbot at llvm.org  Wed Jul 22 04:50:44 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 04:50:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Avoid printing an unset
 nanobind include directory (PR #211240)
Message-ID: <llvm.org/llvm/llvm-project/pull/211240@github.com>


https://github.com/anutosh491 created https://github.com/llvm/llvm-project/pull/211240

`nanobind_INCLUDE_DIR` is only populated when MLIR discovers nanobind through the Python package. When `nanobind_DIR` is supplied explicitly, the variable remains unset and the configuration output contains an empty path:
```
  Found nanobind v2.13.0:
```
The nanobind CMake package location is already reported by both discovery paths. 

We can only print the detected nanobind version in the final status message. Or maybe if we want we can find the include dir for the explicit path too and merge them to `nanobind_INCLUDE_DIR` but I think this is better.

>From e7a5080584e736bfaae74b233b81ee913063d499 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 17:18:18 +0530
Subject: [PATCH] [MLIR][Python] Avoid printing an unset nanobind include
 directory

---
 mlir/cmake/modules/MLIRDetectPythonEnv.cmake | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..4d99c8be0de7f 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -83,7 +83,7 @@ macro(mlir_configure_python_dev_packages)
       set(nanobind_INCLUDE_DIR "${PACKAGE_DIR}")
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
-    message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
+    message(STATUS "Found nanobind v${nanobind_VERSION}")
     message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
             "suffix = '${PYTHON_MODULE_SUFFIX}', "
             "extension = '${PYTHON_MODULE_EXTENSION}")


From llvmlistbot at llvm.org  Wed Jul 22 04:51:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 04:51:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Avoid printing an unset
 nanobind include directory (PR #211240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211240@github.com>
Message-ID: <6a60aec4.66c0679a.9a15c.d7f9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Anutosh Bhat (anutosh491)

<details>
<summary>Changes</summary>

`nanobind_INCLUDE_DIR` is only populated when MLIR discovers nanobind through the Python package. When `nanobind_DIR` is supplied explicitly, the variable remains unset and the configuration output contains an empty path:
```
  Found nanobind v2.13.0:
```
The nanobind CMake package location is already reported by both discovery paths. 

We can only print the detected nanobind version in the final status message. Or maybe if we want we can find the include dir for the explicit path too and merge them to `nanobind_INCLUDE_DIR` but I think this is better.

---
Full diff: https://github.com/llvm/llvm-project/pull/211240.diff


1 Files Affected:

- (modified) mlir/cmake/modules/MLIRDetectPythonEnv.cmake (+1-1) 


``````````diff
diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..4d99c8be0de7f 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -83,7 +83,7 @@ macro(mlir_configure_python_dev_packages)
       set(nanobind_INCLUDE_DIR "${PACKAGE_DIR}")
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
-    message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
+    message(STATUS "Found nanobind v${nanobind_VERSION}")
     message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
             "suffix = '${PYTHON_MODULE_SUFFIX}', "
             "extension = '${PYTHON_MODULE_EXTENSION}")

``````````

</details>


https://github.com/llvm/llvm-project/pull/211240

From llvmlistbot at llvm.org  Wed Jul 22 04:57:18 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Wed, 22 Jul 2026 04:57:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a60b01e.0c05e68e.81651.221e@mx.google.com>


flemairen6 wrote:

Sorry this PR had completely slipped my mind, it looks good to me - can I just ask you to add the lowering of wasmssa.global_set and wasmssa.select to `mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp`  ?

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Wed Jul 22 05:10:50 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Wed, 22 Jul 2026 05:10:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a60b34a.507fc713.3702e2.2f72@mx.google.com>


https://github.com/bruteforceboy updated https://github.com/llvm/llvm-project/pull/211203

>From 19475b38b64fc264dca3ea3dac589affafe74bbb Mon Sep 17 00:00:00 2001
From: workwilson <ogbonnachibuoyim12 at gmail.com>
Date: Wed, 22 Jul 2026 16:37:17 +0800
Subject: [PATCH 1/2] [MLIR][Linalg] Fix crash on duplicate dimensions in
 linalg.broadcast

---
 .../Dialect/Linalg/IR/LinalgStructuredOps.td  |  5 ++
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      |  4 ++
 mlir/test/Dialect/Linalg/invalid.mlir         | 37 ++++++++++++
 mlir/test/Dialect/Linalg/roundtrip.mlir       | 60 +++++++++++++++++++
 4 files changed, 106 insertions(+)

diff --git a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
index fc5c9770d969b..ddb4f6675103e 100644
--- a/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
+++ b/mlir/include/mlir/Dialect/Linalg/IR/LinalgStructuredOps.td
@@ -478,6 +478,11 @@ def BroadcastOp : LinalgStructuredBase_Op<"broadcast", [
   let description = [{
     Broadcast the input into the given shape by adding `dimensions`.
 
+    Each index in `dimensions` attribute refers to a dimension of the init
+    tensor that is added by the operation. The indices must be unique and
+    within the init rank; the sizes of the remaining (non-added) init
+    dimensions must match the input shape.
+
     Example:
     ```mlir
       %bcast = linalg.broadcast
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..170e1edf8a55d 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -2356,6 +2356,10 @@ LogicalResult BroadcastOp::verify() {
                            << initRank - 1 << "], got: " << dim;
   }
 
+  DenseSet<int64_t> uniquedDims(llvm::from_range, dimensionsRef);
+  if (uniquedDims.size() != dimensionsRef.size())
+    return emitOpError() << "dimensions should not contain duplicates";
+
   // Mapping from input dims to init dims.
   SmallVector<int64_t> dimMap;
   for (auto dim : llvm::seq<int64_t>(0, initRank)) {
diff --git a/mlir/test/Dialect/Linalg/invalid.mlir b/mlir/test/Dialect/Linalg/invalid.mlir
index c52163e244ea2..45bd8b87c0970 100644
--- a/mlir/test/Dialect/Linalg/invalid.mlir
+++ b/mlir/test/Dialect/Linalg/invalid.mlir
@@ -1196,6 +1196,43 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank1(
+    %input: tensor<4xf32>, %init: tensor<4x8x16xf32>) -> tensor<4x8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4xf32>)
+      outs(%init:tensor<4x8x16xf32>)
+      dimensions = [1, 1]
+  func.return %bcast : tensor<4x8x16xf32>
+}
+
+// -----
+
+func.func @broadcast_duplicate_dims_rank2(
+    %input: tensor<4x8xf32>, %init: tensor<4x8x16x32xf32>)
+    -> tensor<4x8x16x32xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<4x8xf32>)
+      outs(%init:tensor<4x8x16x32xf32>)
+      dimensions = [3, 3]
+  func.return %bcast : tensor<4x8x16x32xf32>
+}
+
+// -----
+
 func.func @broadcast_no_operands1() {
   // expected-error @+1 {{'linalg.broadcast' op expected 2 operands, but found 0}}
   linalg.broadcast dimensions = [1]
diff --git a/mlir/test/Dialect/Linalg/roundtrip.mlir b/mlir/test/Dialect/Linalg/roundtrip.mlir
index bfb92c3289a49..f2584f7b2ecec 100644
--- a/mlir/test/Dialect/Linalg/roundtrip.mlir
+++ b/mlir/test/Dialect/Linalg/roundtrip.mlir
@@ -599,6 +599,66 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_to_2d(%input: tensor<i32>,
+                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_2d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_to_1d(%input: tensor<f32>,
+                              %init: tensor<64xf32>) -> tensor<64xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<64xf32>)
+      dimensions = [0]
+  func.return %bcast : tensor<64xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_to_1d
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<64xf32>)
+// CHECK-SAME:    dimensions = [0]
+
+// -----
+
+func.func @broadcast_0d_with_dynamic_sizes(
+              %input: tensor<f32>, %init: tensor<8x?xf32>) -> tensor<8x?xf32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<f32>)
+      outs(%init:tensor<8x?xf32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<8x?xf32>
+}
+// CHECK-LABEL: func @broadcast_0d_with_dynamic_sizes
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<8x?xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_memref(%input: memref<f32>,
+                               %init: memref<8x16xf32>) {
+  linalg.broadcast
+      ins(%input:memref<f32>)
+      outs(%init:memref<8x16xf32>)
+      dimensions = [0, 1]
+  func.return
+}
+// CHECK-LABEL: func @broadcast_0d_memref
+//      CHECK:    linalg.broadcast ins(%{{.*}} : memref<f32>)
+// CHECK-SAME:    outs(%{{.*}} : memref<8x16xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
 func.func @map_arith_with_attr(%lhs: tensor<64xf32>, %rhs: tensor<64xf32>,
                       %init: tensor<64xf32>) -> tensor<64xf32> {
   %add = linalg.map

>From 49929ad199e0a16f4be874624a6ec1a19f80ba1e Mon Sep 17 00:00:00 2001
From: workwilson <ogbonnachibuoyim12 at gmail.com>
Date: Wed, 22 Jul 2026 20:10:14 +0800
Subject: [PATCH 2/2] Apply review comments: trim tests, add scalar input test,
 follow test naming convention

---
 mlir/test/Dialect/Linalg/invalid.mlir   | 27 +++++--------------
 mlir/test/Dialect/Linalg/roundtrip.mlir | 36 +++----------------------
 2 files changed, 10 insertions(+), 53 deletions(-)

diff --git a/mlir/test/Dialect/Linalg/invalid.mlir b/mlir/test/Dialect/Linalg/invalid.mlir
index 45bd8b87c0970..8c3f27efcd58a 100644
--- a/mlir/test/Dialect/Linalg/invalid.mlir
+++ b/mlir/test/Dialect/Linalg/invalid.mlir
@@ -1208,27 +1208,14 @@ func.func @broadcast_duplicate_dims(
 
 // -----
 
-func.func @broadcast_duplicate_dims_rank1(
-    %input: tensor<4xf32>, %init: tensor<4x8x16xf32>) -> tensor<4x8x16xf32> {
-  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
-  %bcast = linalg.broadcast
-      ins(%input:tensor<4xf32>)
-      outs(%init:tensor<4x8x16xf32>)
-      dimensions = [1, 1]
-  func.return %bcast : tensor<4x8x16xf32>
-}
-
-// -----
-
-func.func @broadcast_duplicate_dims_rank2(
-    %input: tensor<4x8xf32>, %init: tensor<4x8x16x32xf32>)
-    -> tensor<4x8x16x32xf32> {
-  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+func.func @broadcast_scalar_input(
+    %input: f32, %init: tensor<8x16xf32>) -> tensor<8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op operand #0 must be memref of any non-token type values or ranked tensor of any non-token type values, but got 'f32'}}
   %bcast = linalg.broadcast
-      ins(%input:tensor<4x8xf32>)
-      outs(%init:tensor<4x8x16x32xf32>)
-      dimensions = [3, 3]
-  func.return %bcast : tensor<4x8x16x32xf32>
+      ins(%input:f32)
+      outs(%init:tensor<8x16xf32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<8x16xf32>
 }
 
 // -----
diff --git a/mlir/test/Dialect/Linalg/roundtrip.mlir b/mlir/test/Dialect/Linalg/roundtrip.mlir
index f2584f7b2ecec..5f9e98bdadb5d 100644
--- a/mlir/test/Dialect/Linalg/roundtrip.mlir
+++ b/mlir/test/Dialect/Linalg/roundtrip.mlir
@@ -599,51 +599,21 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
-func.func @broadcast_0d_to_2d(%input: tensor<i32>,
-                              %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+func.func @broadcast_0d_tensor(%input: tensor<i32>,
+                               %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
   %bcast = linalg.broadcast
       ins(%input:tensor<i32>)
       outs(%init:tensor<32x2xi32>)
       dimensions = [0, 1]
   func.return %bcast : tensor<32x2xi32>
 }
-// CHECK-LABEL: func @broadcast_0d_to_2d
+// CHECK-LABEL: func @broadcast_0d_tensor
 //      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
 // CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
 // CHECK-SAME:    dimensions = [0, 1]
 
 // -----
 
-func.func @broadcast_0d_to_1d(%input: tensor<f32>,
-                              %init: tensor<64xf32>) -> tensor<64xf32> {
-  %bcast = linalg.broadcast
-      ins(%input:tensor<f32>)
-      outs(%init:tensor<64xf32>)
-      dimensions = [0]
-  func.return %bcast : tensor<64xf32>
-}
-// CHECK-LABEL: func @broadcast_0d_to_1d
-//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
-// CHECK-SAME:    outs(%{{.*}} : tensor<64xf32>)
-// CHECK-SAME:    dimensions = [0]
-
-// -----
-
-func.func @broadcast_0d_with_dynamic_sizes(
-              %input: tensor<f32>, %init: tensor<8x?xf32>) -> tensor<8x?xf32> {
-  %bcast = linalg.broadcast
-      ins(%input:tensor<f32>)
-      outs(%init:tensor<8x?xf32>)
-      dimensions = [0, 1]
-  func.return %bcast : tensor<8x?xf32>
-}
-// CHECK-LABEL: func @broadcast_0d_with_dynamic_sizes
-//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<f32>)
-// CHECK-SAME:    outs(%{{.*}} : tensor<8x?xf32>)
-// CHECK-SAME:    dimensions = [0, 1]
-
-// -----
-
 func.func @broadcast_0d_memref(%input: memref<f32>,
                                %init: memref<8x16xf32>) {
   linalg.broadcast


From llvmlistbot at llvm.org  Wed Jul 22 05:12:09 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Wed, 22 Jul 2026 05:12:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a60b399.1b0626d0.1ff4c3.ea82@mx.google.com>


bruteforceboy wrote:

@nujaa thanks for the review. I've applied the initial comments.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Wed Jul 22 05:15:04 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 05:15:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale Python module
 naming status message (PR #211243)
Message-ID: <llvm.org/llvm/llvm-project/pull/211243@github.com>


https://github.com/anutosh491 created https://github.com/llvm/llvm-project/pull/211243

`PYTHON_MODULE_PREFIX`, `PYTHON_MODULE_SUFFIX`, and `PYTHON_MODULE_EXTENSION` are remnants of the previous pybind11-based Python discovery path and are no longer populated or used by the nanobind build.

The relevant Python SOABI is already reported through `Python3_SOABI`, so we can remove the redundant status message, which currently prints only empty values.

>From 888aade84499d8a4bdf4f32de12860a2f7fd36e4 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 17:42:37 +0530
Subject: [PATCH] [MLIR][Python] Remove stale Python module naming status
 message

---
 mlir/cmake/modules/MLIRDetectPythonEnv.cmake | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..ad176af6ef580 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -84,8 +84,5 @@ macro(mlir_configure_python_dev_packages)
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
     message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
-    message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
-            "suffix = '${PYTHON_MODULE_SUFFIX}', "
-            "extension = '${PYTHON_MODULE_EXTENSION}")
   endif()
 endmacro()


From llvmlistbot at llvm.org  Wed Jul 22 05:15:20 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 05:15:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale Python module
 naming status message (PR #211243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211243@github.com>
Message-ID: <6a60b458.507fc713.3702e2.d4c3@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/211243

>From e27e7eb00fea2e2e73ae733aeb015eaa2ee66f0b Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 17:42:37 +0530
Subject: [PATCH] [MLIR][Python] Remove stale Python module naming status
 message

---
 mlir/cmake/modules/MLIRDetectPythonEnv.cmake | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..ad176af6ef580 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -84,8 +84,5 @@ macro(mlir_configure_python_dev_packages)
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
     message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
-    message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
-            "suffix = '${PYTHON_MODULE_SUFFIX}', "
-            "extension = '${PYTHON_MODULE_EXTENSION}")
   endif()
 endmacro()


From llvmlistbot at llvm.org  Wed Jul 22 05:15:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 05:15:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale Python module
 naming status message (PR #211243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211243@github.com>
Message-ID: <6a60b46d.3cce8983.247e47.954c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Anutosh Bhat (anutosh491)

<details>
<summary>Changes</summary>

`PYTHON_MODULE_PREFIX`, `PYTHON_MODULE_SUFFIX`, and `PYTHON_MODULE_EXTENSION` are remnants of the previous pybind11-based Python discovery path and are no longer populated or used by the nanobind build.

The relevant Python SOABI is already reported through `Python3_SOABI`, so we can remove the redundant status message, which currently prints only empty values.

---
Full diff: https://github.com/llvm/llvm-project/pull/211243.diff


1 Files Affected:

- (modified) mlir/cmake/modules/MLIRDetectPythonEnv.cmake (-3) 


``````````diff
diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..ad176af6ef580 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -84,8 +84,5 @@ macro(mlir_configure_python_dev_packages)
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
     message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
-    message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
-            "suffix = '${PYTHON_MODULE_SUFFIX}', "
-            "extension = '${PYTHON_MODULE_EXTENSION}")
   endif()
 endmacro()

``````````

</details>


https://github.com/llvm/llvm-project/pull/211243

From llvmlistbot at llvm.org  Wed Jul 22 05:22:26 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 05:22:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60b602.eb165e5c.2e0c9a.9d39@mx.google.com>


https://github.com/IgWod commented:

I take OpCopy is roughly an equivalent to OpLoad and OpStore? If so does this pattern fire for the same memrefs? I think there was some work on supporting image fetches so I think we should against such memref in Copy lowering.

https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Wed Jul 22 05:28:23 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 05:28:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60b767.f06f4994.50b1b.5b52@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}: i32):
+    spirv.ReturnValue %0 : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
+    // Makes sure both values are used.
+    %sum = spirv.IAdd %0#0, %0#1 : i32
+    spirv.ReturnValue %sum : i32
----------------
IgWod wrote:

Please add check for those two lines.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 05:28:23 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 05:28:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60b767.1905e68e.25404.7a63@mx.google.com>


================
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}: i32):
+    spirv.ReturnValue %0 : i32
----------------
IgWod wrote:

Probably worth adding check for return here as well.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 05:28:23 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 05:28:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60b767.824e11a6.322560.033b@mx.google.com>


https://github.com/IgWod approved this pull request.

LGTM, just two final comments.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 05:28:23 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 05:28:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60b767.1b0626d0.1ff4c3.1fb9@mx.google.com>


https://github.com/IgWod edited https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 05:40:07 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 22 Jul 2026 05:40:07 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211011@github.com>
Message-ID: <6a60ba27.9a11f723.3d57d8.217b@mx.google.com>


================
@@ -302,6 +302,28 @@ def OpenACC_FirstprivateMapInitialOp
   let extraClassDeclaration = extraClassDeclarationBase;
 }
 
+//===----------------------------------------------------------------------===//
+// acc.firstprivate_save
+//===----------------------------------------------------------------------===//
+
+def OpenACC_FirstprivateSaveOp : OpenACC_Op<"firstprivate_save", []> {
+ let summary = "\"Saves\" a value to a firstprivate variable";
+ let description = [{
+    The `acc.firstprivate_save` operation is intended to provide a temporary
+    link from the compute region's private variable and the firstprivate map
+    init op. This facilitates analysis in cases where the compiler may
+    determine that a firstprivate should be converted to a copy and can be
+    safely erased when that analysis is complete.
+  }];
+  let arguments = (ins AnyType:$value,
----------------
rscottmanley wrote:

Yes, the issue isnt losing the firstprivate - we need to connect the alloca that gets materialized to the firstprivate. The alloca is what gets optimized away - so the value that is tracked through that (or perhaps promoted) needs be to saved somehow.

https://github.com/llvm/llvm-project/pull/211011

From llvmlistbot at llvm.org  Wed Jul 22 05:41:48 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 05:41:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
Message-ID: <llvm.org/llvm/llvm-project/pull/211250@github.com>


https://github.com/anutosh491 created https://github.com/llvm/llvm-project/pull/211250

The `_mlirExecutionEngine` extension is declared only when `MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However, `execution_engine.py` and its type stub are currently added to the Python package unconditionally.

This leaves an installed `mlir.execution_engine` module that fails immediately when imported because its native extension is absent. It also exposes a type stub for an unavailable API.

Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is enabled.

>From d4920ce022d49d737a8157802141eeecc53a6942 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 18:10:06 +0530
Subject: [PATCH] [MLIR][Python] Do not install ExecutionEngine wrapper when
 disabled

---
 mlir/python/CMakeLists.txt | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..693701ef50c13 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"


From llvmlistbot at llvm.org  Wed Jul 22 05:42:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 05:42:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a60bab5.00bedb31.e961c.7ae9@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Anutosh Bhat (anutosh491)

<details>
<summary>Changes</summary>

The `_mlirExecutionEngine` extension is declared only when `MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However, `execution_engine.py` and its type stub are currently added to the Python package unconditionally.

This leaves an installed `mlir.execution_engine` module that fails immediately when imported because its native extension is absent. It also exposes a type stub for an unavailable API.

Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is enabled.

---
Full diff: https://github.com/llvm/llvm-project/pull/211250.diff


1 Files Affected:

- (modified) mlir/python/CMakeLists.txt (+10-2) 


``````````diff
diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..693701ef50c13 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"

``````````

</details>


https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Wed Jul 22 05:54:14 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Wed, 22 Jul 2026 05:54:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix XeGPUToXeVM crash on
 non-integer memref memory spaces (PR #211053)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211053@github.com>
Message-ID: <6a60bd76.af6ba3ed.1989e7.5bca@mx.google.com>


hamzaqureshi5 wrote:

@silee2 @akroviakov 

https://github.com/llvm/llvm-project/pull/211053

From llvmlistbot at llvm.org  Wed Jul 22 06:03:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 06:03:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 9d3fe6c - [mlir][sme] Add e2e test for
	lowering mmt4d to sme (#208226)
Message-ID: <6a60bf9b.e5907290.33b622.343c@mx.google.com>


Author: Stephen Long
Date: 2026-07-22T09:03:19-04:00
New Revision: 9d3fe6cf06f1c20a337ce546115523bf1db925de

URL: https://github.com/llvm/llvm-project/commit/9d3fe6cf06f1c20a337ce546115523bf1db925de
DIFF: https://github.com/llvm/llvm-project/commit/9d3fe6cf06f1c20a337ce546115523bf1db925de.diff

LOG: [mlir][sme] Add e2e test for lowering mmt4d to sme (#208226)

Lowers mmt4d to arm sme with hoisting the load/store of the accumulator
out of the K-loop. Similar to ArmSVE/pack-unpack-mmt4d.mlir with some
additional changes.

Appreciate @banach-space for helping me out with this (#201562)

Assisted-by: Claude

---------

Co-authored-by: Stephen Long <steplong at quicinc.com>
Co-authored-by: Ege Beysel <beyselege at gmail.com>
Co-authored-by: Andrzej Warzyński <andrzej.warzynski at gmail.com>

Added: 
    mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir

Modified: 
    

Removed: 
    


################################################################################
diff  --git a/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
new file mode 100644
index 0000000000000..73e286f960008
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Linalg/CPU/ArmSME/pack-unpack-mmt4d.mlir
@@ -0,0 +1,360 @@
+// DEFINE: %{compile} =  mlir-opt %s \
+// DEFINE:    -transform-interpreter -test-transform-dialect-erase-schedule \
+// DEFINE:    -canonicalize -test-lower-to-arm-sme -convert-vector-to-llvm="enable-arm-sve" \
+// DEFINE:    -test-lower-to-llvm
+// DEFINE: %{entry_point} = main
+// DEFINE: %{run} = %mcr_aarch64_cmd -e %{entry_point} -entry-point-result=void --march=aarch64 --mattr="+sme"\
+// DEFINE:    -shared-libs=%native_mlir_runner_utils,%native_mlir_c_runner_utils,%native_arm_sme_abi_shlib
+
+// RUN: %{compile} | %{run} | FileCheck %s
+
+//===----------------------------------------------------------------------===//
+/// HIGH-LEVEL OVERVIEW
+///
+/// End-to-end test for computing matrix-multiplication using linalg.mmt4d. In
+/// particular, demonstrates how the following MLIR sequence (implemented in
+/// @matmul_via_mmt4d):
+///
+///   A_pack = linalg.pack A
+///   B_pack = linalg.pack B
+///   C_pack = linalg.pack C
+///   out_pack = linalg.mmt4d(A_pack, B_pack, C_pack)
+///   out = linalg.unpack out_pack
+///
+/// is equivalent to:
+///
+///  linalg.matmul(A, B, C)
+///
+/// (implemented in @matmul_via_matmul).
+///
+/// NOTES ON IMPLEMENTATION
+/// 1. The MMT4D example uses _scalable_ tile sizes for data tiling.
+///   * The matrix-multiplication dimensions that are scalable: M and N.
+///
+/// 2. The lowering of linalg.mmt4d leverages scalable vectorisation.
+///   * The matrix-multiplication dimensions that are scalable: M, N (to match data
+///     tiling configuration).
+///
+/// 3. Neither `linalg.pack` nor `linalg.unpack` are vectorised ATM.
+///
+/// 4. The MMT4D and Pack/Unpack Ops are kept in seperate functions to isolate
+///    the corresponding lowering and lowering configs.
+/// 
+/// TODO: Vectorize linalg.pack + linalg.unpack:
+///   * https://github.com/llvm/llvm-project/issues/159751
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point that computes matrix multiplication via linalg.mmt4d
+// and linalg.matmul. Note, the output should be independent of the underlying
+// Linalg Op used, as well as SVE vector length.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Allocate and initialise the inputs
+  %A_empty = tensor.empty() : tensor<7x16xf32>
+  %B_empty = tensor.empty() : tensor<16x13xf32>
+
+  %c3 = arith.constant 3.0 : f32
+  %c4 = arith.constant 4.0 : f32
+  %A = linalg.fill ins(%c3 : f32) outs(%A_empty : tensor<7x16xf32>) -> tensor<7x16xf32>
+  %B = linalg.fill ins(%c4 : f32) outs(%B_empty : tensor<16x13xf32>) -> tensor<16x13xf32>
+  %C = arith.constant dense<[
+    [ 1.,  8., 15., 22., 29., 36., 43., 50., 57., 64., 71., 78., 85.],
+    [ 2.,  9., 16., 23., 30., 37., 44., 51., 58., 65., 72., 79., 86.],
+    [ 3., 10., 17., 24., 31., 38., 45., 52., 59., 66., 73., 80., 87.],
+    [ 4., 11., 18., 25., 32., 39., 46., 53., 60., 67., 74., 81., 88.],
+    [ 5., 12., 19., 26., 33., 40., 47., 54., 61., 68., 75., 82., 89.],
+    [ 6., 13., 20., 27., 34., 41., 48., 55., 62., 69., 76., 83., 90.],
+    [ 7., 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91.]
+  ]> : tensor<7x13xf32>
+
+  // VARIANT: Matrix multiplication via linalg.mmt4d
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_mmt4d = func.call @matmul_via_mmt4d(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_mmt4d_cast = tensor.cast %C_mmt4d : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "--------------------------\n"
+  vector.print str "RESULT FROM linalg.mmt4d:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_mmt4d_cast) : (tensor<*xf32>) -> ()
+
+  // VARIANT: Matrix multiplication via linalg.matmul
+  // CHECK: Unranked Memref
+  // CHECK:  [193,   200,   207,   214,   221,   228,   235,   242,   249,   256,   263,   270,   277]
+  // CHECK:  [194,   201,   208,   215,   222,   229,   236,   243,   250,   257,   264,   271,   278]
+  // CHECK:  [195,   202,   209,   216,   223,   230,   237,   244,   251,   258,   265,   272,   279]
+  // CHECK:  [196,   203,   210,   217,   224,   231,   238,   245,   252,   259,   266,   273,   280]
+  // CHECK:  [197,   204,   211,   218,   225,   232,   239,   246,   253,   260,   267,   274,   281]
+  // CHECK:  [198,   205,   212,   219,   226,   233,   240,   247,   254,   261,   268,   275,   282]
+  // CHECK:  [199,   206,   213,   220,   227,   234,   241,   248,   255,   262,   269,   276,   283]
+  %C_matmul = func.call @matmul(%A, %B, %C) : (tensor<7x16xf32>, tensor<16x13xf32>, tensor<7x13xf32>) -> tensor<7x13xf32>
+  %C_matmul_cast = tensor.cast %C_matmul : tensor<7x13xf32> to tensor<*xf32>
+  vector.print str "\n--------------------------\n"
+  vector.print str "RESULT FROM linalg.matmul:\n"
+  vector.print str "--------------------------\n"
+  call @printMemrefF32(%C_matmul_cast) : (tensor<*xf32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_matmul
+//
+// Implements matrix-multiplication via linalg.matmul
+//===----------------------------------------------------------------------===//
+func.func private @matmul(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  %C_matmul = linalg.matmul ins(%A, %B: tensor<7x16xf32>, tensor<16x13xf32>)
+                            outs(%C: tensor<7x13xf32>) -> tensor<7x13xf32>
+
+  return %C_matmul : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_lhs
+//
+// Implements packing for the A matrix (LHS) in matrix multiplication. The
+// inner tile size for dim M is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_lhs(%A: tensor<7x16xf32>) -> tensor<1x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %A_pack_empty = tensor.empty(%vs_c8) : tensor<1x16x?x1xf32>
+  %A_pack = linalg.pack %A
+    padding_value(%pad : f32)
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, 1]
+    into %A_pack_empty : tensor<7x16xf32> -> tensor<1x16x?x1xf32>
+
+  return %A_pack : tensor<1x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_rhs
+//
+// Implements packing for the B matrix (RHS) in matrix multiplication. The
+// inner tile size for dim N is "scalable": 8 * vscale.
+//===----------------------------------------------------------------------===//
+func.func private @pack_rhs(%B: tensor<16x13xf32>) ->  tensor<?x16x?x1xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %c13 = arith.constant 13 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %B_pack_empty = tensor.empty(%outer_tile_size, %vs_c8) : tensor<?x16x?x1xf32>
+  %B_pack = linalg.pack %B
+     padding_value(%pad : f32)
+     outer_dims_perm = [1, 0]
+     inner_dims_pos = [1, 0]
+     inner_tiles = [%vs_c8, 1]
+     into %B_pack_empty : tensor<16x13xf32> -> tensor<?x16x?x1xf32>
+
+  return %B_pack : tensor<?x16x?x1xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @pack_acc
+//
+// Implements packing for the C matrix (accumulator) in matrix multiplication.
+// The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @pack_acc(%C: tensor<7x13xf32>) -> tensor<1x?x?x?xf32> {
+  %pad = arith.constant 0.0 : f32
+
+  // Compute the outer tile size.
+  %c13 = arith.constant 13 : index
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+  %outer_tile_size = arith.ceildivui %c13, %vs_c8 : index
+
+  %C_pack_empty = tensor.empty(%outer_tile_size, %vs_c8, %vs_c8) : tensor<1x?x?x?xf32>
+  %C_pack = linalg.pack %C
+    padding_value(%pad : f32)
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8] into %C_pack_empty : tensor<7x13xf32> -> tensor<1x?x?x?xf32>
+
+  return %C_pack : tensor<1x?x?x?xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @unpack_acc
+//
+// Implements unpacking for the C matrix (accumulator) in matrix
+// multiplication. The inner tile sizes are "scalable": 8 * vscale, 8 * vscale
+//===----------------------------------------------------------------------===//
+func.func private @unpack_acc(%C_packed: tensor<1x?x?x?xf32>) -> tensor<7x13xf32> {
+  %vs = vector.vscale
+  %c8 = arith.constant 8 : index
+  %vs_c8 = arith.muli %vs, %c8 : index
+
+  %C_out_empty = tensor.empty() : tensor<7x13xf32>
+  %C_out_unpack = linalg.unpack %C_packed
+    outer_dims_perm = [0, 1]
+    inner_dims_pos = [0, 1]
+    inner_tiles = [%vs_c8, %vs_c8]
+    into %C_out_empty : tensor<1x?x?x?xf32> -> tensor<7x13xf32>
+
+  return %C_out_unpack: tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// @matmul_via_mmt4d
+//
+// Implements matrix-multiplication via linalg.mmt4d
+//===----------------------------------------------------------------------===//
+func.func private @matmul_via_mmt4d(%A: tensor<7x16xf32>, %B: tensor<16x13xf32>, %C: tensor<7x13xf32>) -> tensor<7x13xf32> {
+  // Pack input matrices
+  %A_pack = func.call @pack_lhs(%A): (tensor<7x16xf32>) -> tensor<1x16x?x1xf32>
+  %B_pack = func.call @pack_rhs(%B): (tensor<16x13xf32>) -> tensor<?x16x?x1xf32>
+  %C_pack = func.call @pack_acc(%C): (tensor<7x13xf32>) -> tensor<1x?x?x?xf32>
+
+  // MMT4D
+  %mmt4d = linalg.mmt4d ins(%A_pack, %B_pack : tensor<1x16x?x1xf32>, tensor<?x16x?x1xf32>) outs(%C_pack : tensor<1x?x?x?xf32>) -> tensor<1x?x?x?xf32>
+
+  // Unpack the output
+  %C_out_unpack = func.call @unpack_acc(%mmt4d) : (tensor<1x?x?x?xf32>) -> tensor<7x13xf32>
+
+  return %C_out_unpack : tensor<7x13xf32>
+}
+
+//===----------------------------------------------------------------------===//
+// TD Sequence
+//===----------------------------------------------------------------------===//
+module @transforms attributes { transform.with_named_sequence } {
+  transform.named_sequence @__transform_main(%module: !transform.any_op {transform.consumed}) {
+    //==========================================================================
+    // HANDLE MMT4D
+    //==========================================================================
+    %mmt4d = transform.collect_matching @match_mmt4d in %module : (!transform.any_op) -> (!transform.any_op)
+
+    // Step 1: Tile
+    // Tile parallel dims (note, the M, N dims are scalable!)
+    %tiled_mmt4d_parallel, %_:4 = transform.structured.tile_using_for %mmt4d tile_sizes [1, 1, 0, [8], [8], 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
+    // Tile reduction dims
+    %tiled_mmt4d, %loop_k = transform.structured.tile_using_for %tiled_mmt4d_parallel tile_sizes [0, 0, 1, 0, 0, 0]
+      : (!transform.any_op) -> (!transform.any_op, !transform.op<"scf.for">)
+
+    // Step 2: Vectorize linalg.mmt4d (note, the M, N dims are scalable!)
+    transform.structured.vectorize %tiled_mmt4d vector_sizes  [1, 1, 1, [8], [8], 1] {assume_dynamic_dims_match_vec_sizes, create_named_contraction}
+      : !transform.any_op
+
+    // Step 3: Lower vector.mask %mask { vector.transfer_* } to vector.transfer_* %mask
+    transform.apply_patterns to %loop_k {
+      transform.apply_patterns.vector.lower_masked_transfers
+    } : !transform.op<"scf.for">
+
+    // Step 4: Hoist the C accumulator load/store out of the k-loop while still
+    // in tensor form, so transfer_write has a result value the loop can yield.
+    transform.apply_licm to %loop_k : !transform.op<"scf.for">
+    transform.loop.hoist_loop_invariant_subsets %loop_k : !transform.op<"scf.for">
+
+    // Lower to outerproduct
+    %func_pre = transform.structured.match ops{["func.func"]} in %module
+      : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func_pre {
+      transform.apply_patterns.vector.transfer_permutation_patterns
+      transform.apply_patterns.canonicalization
+    } : !transform.any_op
+    transform.apply_patterns to %func_pre {
+      // TODO: `create_named_contraction` above lowers directly to a named contraction. If we want to remove
+      // `transform.apply_patterns.vector.reduction_to_contract` below, `CombineContractBroadcastMask`
+      // will have to be moved to a dedicated TD Op.
+      transform.apply_patterns.vector.reduction_to_contract
+      transform.apply_patterns.vector.cast_away_vector_leading_one_dim
+      transform.apply_patterns.vector.lower_contraction
+          lowering_strategy = "outerproduct"
+      transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+      transform.apply_patterns.canonicalization
+    } {apply_cse} : !transform.any_op
+
+   //==========================================================================
+   // HANDLE PACK + UNPACK
+   //==========================================================================
+    %pack = transform.structured.match ops{["linalg.pack"]} in %module : (!transform.any_op) -> !transform.any_op
+    %unpack = transform.structured.match ops{["linalg.unpack"]} in %module : (!transform.any_op) -> !transform.any_op
+
+    // 1.1 Tile the linalg.pack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2.1)
+    %tiled_pack_op_p, %loops_pack:2 = transform.structured.tile_using_for %pack tile_sizes [1, 1]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 1.2 Tile the linalg.unpack Op so that we can decompose it into e.g. tensor.pad
+    //    and other lower-level Ops (see step 2). Tile by [8*vscale, 8*vscale]
+    //    (scalable, equal to the inner tile sizes) so the tiled outer dims are
+    //    statically 1 and DecomposeOuterUnitDimsUnPackOp can fire.
+    %tiled_unpack_op_p, %loops_unpack:2 = transform.structured.tile_using_for %unpack tile_sizes [[8], [8]]
+        inner_tile_alignments = [Equal, Equal]
+       : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op)
+
+    // 2.1. Decompose tiled PackOp into lower-level Ops + simplify
+    %func_op_pack = transform.get_parent_op %tiled_pack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+      transform.apply_patterns.linalg.decompose_pad
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_pack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+    // 2.2. Decompose tiled UnpackOp into lower-level Ops + simplify
+    %func_op_unpack = transform.get_parent_op %tiled_unpack_op_p {isolated_from_above} : (!transform.any_op) -> !transform.op<"func.func">
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.linalg.decompose_pack_unpack
+    } : !transform.op<"func.func">
+
+    transform.apply_patterns to %func_op_unpack {
+      transform.apply_patterns.tensor.fold_tensor_subset_ops
+      transform.apply_patterns.canonicalization
+    } : !transform.op<"func.func">
+
+   //==========================================================================
+   // BUFFERIZATION
+   //==========================================================================
+   %bufferize = transform.bufferization.one_shot_bufferize layout{IdentityLayoutMap} %module
+     {bufferize_function_boundaries=true} : (!transform.any_op) -> !transform.any_op
+   %func = transform.structured.match ops{["func.func"]} in %bufferize
+     : (!transform.any_op) -> !transform.any_op
+
+   transform.apply_patterns to %func {
+     transform.apply_patterns.vector.rank_reducing_subview_patterns
+     transform.apply_patterns.vector.drop_unit_dims_with_shape_cast
+     transform.apply_patterns.canonicalization
+   } {apply_cse} : !transform.any_op
+
+   transform.yield
+   }
+
+   //==========================================================================
+   // TD MATCHERS (helper hooks)
+   //==========================================================================
+  transform.named_sequence @match_mmt4d(
+      %entry: !transform.any_op {transform.readonly}) -> !transform.any_op {
+    transform.match.operation_name %entry ["linalg.mmt4d"] : !transform.any_op
+    transform.yield %entry : !transform.any_op
+  }
+}
+
+//===----------------------------------------------------------------------===//
+// Function signatures
+//===----------------------------------------------------------------------===//
+func.func private @printMemrefF32(%ptr : tensor<*xf32>)


        

From llvmlistbot at llvm.org  Wed Jul 22 06:03:41 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Wed, 22 Jul 2026 06:03:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sme] Add e2e test for lowering mmt4d
	to sme (PR #208226)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208226@github.com>
Message-ID: <6a60bfad.d681cab9.33af1e.7fd1@mx.google.com>


https://github.com/steplong closed https://github.com/llvm/llvm-project/pull/208226

From llvmlistbot at llvm.org  Wed Jul 22 06:07:28 2026
From: llvmlistbot at llvm.org (Abid Qadeer)
Date: Wed, 22 Jul 2026 06:07:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Preserve debug location in
 OpenMPIRBuilder calls. (PR #211254)
Message-ID: <llvm.org/llvm/llvm-project/pull/211254@github.com>


https://github.com/abidh created https://github.com/llvm/llvm-project/pull/211254

Many `OpenMPIRBuilder` entry points take an `OpenMPIRBuilder::LocationDescription`. It has two relevant constructors:

`LocationDescription(const IRBuilderBase &IRB)`  // captures IP and debug loc
`LocationDescription(const InsertPointTy &IP)`   // captures IP only; DL is empty

The OpenMP MLIR-to-LLVM-IR translation constructs the location from `builder` in almost all places (~45 call sites), which selects the first constructor and propagates both the insertion point and the current debug location. A few call sites instead passed `builder.saveIP()`, which selects the second constructor and silently drops the debug location.

Change the 9 offending call sites to pass `builder` instead of `builder.saveIP()`, so the debug location is preserved.

>From c4c1728e3f9af8c60768d41f571a555e76818925 Mon Sep 17 00:00:00 2001
From: Abid Qadeer <haqadeer at amd.com>
Date: Wed, 22 Jul 2026 10:26:43 +0100
Subject: [PATCH] [MLIR][OpenMP] Preserve debug location in OpenMPIRBuilder
 calls.

Many OpenMPIRBuilder entry points take an
OpenMPIRBuilder::LocationDescription. It has two relevant constructors:

LocationDescription(const IRBuilderBase &IRB)  // captures IP *and* debug loc
LocationDescription(const InsertPointTy &IP)   // captures IP only; DL is empty

The OpenMP MLIR-to-LLVM-IR translation constructs the location from
`builder` in almost all places (~45 call sites), which selects the first
constructor and propagates both the insertion point and the current
debug location. A few call sites instead passed `builder.saveIP()`,
which selects the second constructor and silently drops the debug
location.

Change the 9 offending call sites to pass `builder` instead of
`builder.saveIP()`, so the debug location is preserved.
---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 26 +++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f2fefa1f5a53f..1e3cbeecdb8c7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -271,7 +271,7 @@ class LinearClauseProcessor {
     // Emit barrier
     builder.SetInsertPoint(linearExitBB->getTerminator());
     return moduleTranslation.getOpenMPBuilder()->createBarrier(
-        builder.saveIP(), llvm::omp::OMPD_barrier);
+        builder, llvm::omp::OMPD_barrier);
   }
 
   // Emit stores for linear variables. Useful in case of SIMD
@@ -1654,8 +1654,8 @@ static LogicalResult createReductionsAndCleanup(
   llvm::UnreachableInst *tempTerminator = builder.CreateUnreachable();
   builder.SetInsertPoint(tempTerminator);
   llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-      ompBuilder->createReductions(builder.saveIP(), allocaIP, reductionInfos,
-                                   isByRef, isNowait, isTeamsReduction);
+      ompBuilder->createReductions(builder, allocaIP, reductionInfos, isByRef,
+                                   isNowait, isTeamsReduction);
 
   if (failed(handleError(contInsertPoint, *op)))
     return failure();
@@ -2003,7 +2003,7 @@ static LogicalResult copyFirstPrivateVars(
   if (insertBarrier && !opIsInSingleThread(op)) {
     llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy res =
-        ompBuilder->createBarrier(builder.saveIP(), llvm::omp::OMPD_barrier);
+        ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(res, *op)))
       return failure();
   }
@@ -4377,7 +4377,7 @@ convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
     return failure();
   }
 
-  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder.saveIP(), dds);
+  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder, dds);
   if (dds.DepArray) {
     builder.CreateFree(dds.DepArray);
   }
@@ -4537,7 +4537,7 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
                                         loopInfo->getPreheader());
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
         moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
+            builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
@@ -4734,9 +4734,10 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       builder.SetInsertPoint(tempTerminator);
 
       llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-          ompBuilder->createReductions(
-              builder.saveIP(), allocaIP, reductionInfos, isByRef,
-              /*IsNoWait=*/false, /*IsTeamsReduction=*/false);
+          ompBuilder->createReductions(builder, allocaIP, reductionInfos,
+                                       isByRef,
+                                       /*IsNoWait=*/false,
+                                       /*IsTeamsReduction=*/false);
       if (!contInsertPoint)
         return contInsertPoint.takeError();
 
@@ -9698,8 +9699,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
               return failure();
 
             llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-                ompBuilder->createBarrier(builder.saveIP(),
-                                          llvm::omp::OMPD_barrier);
+                ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
             LogicalResult res = handleError(afterIP, *op);
             if (res.succeeded()) {
               // If the barrier generated a cancellation check, the insertion
@@ -9712,7 +9712,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             if (failed(checkImplementationStatus(*op)))
               return failure();
 
-            ompBuilder->createTaskyield(builder.saveIP());
+            ompBuilder->createTaskyield(builder);
             return success();
           })
           .Case([&](omp::FlushOp op) {
@@ -9727,7 +9727,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             //
             // The argument list is discarded so that, flush with a list is
             // treated same as a flush without a list.
-            ompBuilder->createFlush(builder.saveIP());
+            ompBuilder->createFlush(builder);
             return success();
           })
           .Case([&](omp::ParallelOp op) {


From llvmlistbot at llvm.org  Wed Jul 22 06:21:13 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 22 Jul 2026 06:21:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60c3c9.bd93e731.2ac69c.4af0@mx.google.com>


https://github.com/aobolensk updated https://github.com/llvm/llvm-project/pull/206016

>From b3172e577a92b663badf1508cc0baa194b789724 Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Fri, 26 Jun 2026 11:43:33 +0200
Subject: [PATCH 1/2] [mlir][SPIR-V] Lower memref.copy to spirv.CopyMemory

---
 .../MemRefToSPIRV/MemRefToSPIRV.cpp           | 48 +++++++++++++++++--
 .../MemRefToSPIRV/memref-to-spirv.mlir        | 22 +++++++++
 2 files changed, 65 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
index 4674aa351315f..de0356fe7b59e 100644
--- a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
+++ b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
@@ -339,6 +339,16 @@ class StoreOpPattern final : public OpConversionPattern<memref::StoreOp> {
                   ConversionPatternRewriter &rewriter) const override;
 };
 
+/// Converts memref.copy to spirv.CopyMemory.
+class CopyOpPattern final : public OpConversionPattern<memref::CopyOp> {
+public:
+  using Base::Base;
+
+  LogicalResult
+  matchAndRewrite(memref::CopyOp copyOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override;
+};
+
 class ReinterpretCastPattern final
     : public OpConversionPattern<memref::ReinterpretCastOp> {
 public:
@@ -1171,6 +1181,34 @@ StoreOpPattern::matchAndRewrite(memref::StoreOp storeOp, OpAdaptor adaptor,
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// CopyOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+CopyOpPattern::matchAndRewrite(memref::CopyOp copyOp, OpAdaptor adaptor,
+                               ConversionPatternRewriter &rewriter) const {
+  // The converted operands are SPIR-V pointers to the source and target
+  // storage. spirv.CopyMemory copies the whole pointed-to object, so it only
+  // applies when both pointers point to the same fixed-size element type.
+  Value source = adaptor.getSource();
+  Value target = adaptor.getTarget();
+  auto sourcePtrType = dyn_cast<spirv::PointerType>(source.getType());
+  auto targetPtrType = dyn_cast<spirv::PointerType>(target.getType());
+  if (!sourcePtrType || !targetPtrType)
+    return rewriter.notifyMatchFailure(copyOp, "failed to convert memref type");
+
+  if (sourcePtrType.getPointeeType() != targetPtrType.getPointeeType())
+    return rewriter.notifyMatchFailure(
+        copyOp, "source and target pointee types do not match");
+
+  rewriter.replaceOpWithNewOp<spirv::CopyMemoryOp>(
+      copyOp, target, source, /*memory_access=*/spirv::MemoryAccessAttr{},
+      /*alignment=*/IntegerAttr{}, /*source_memory_access=*/
+      spirv::MemoryAccessAttr{}, /*source_alignment=*/IntegerAttr{});
+  return success();
+}
+
 LogicalResult ReinterpretCastPattern::matchAndRewrite(
     memref::ReinterpretCastOp op, OpAdaptor adaptor,
     ConversionPatternRewriter &rewriter) const {
@@ -1239,10 +1277,10 @@ namespace mlir {
 void populateMemRefToSPIRVPatterns(const SPIRVTypeConverter &typeConverter,
                                    RewritePatternSet &patterns) {
   patterns.add<AllocaOpPattern, AllocOpPattern, AtomicRMWOpPattern,
-               DeallocOpPattern, IntLoadOpPattern, ImageLoadOpPattern,
-               IntStoreOpPattern, LoadOpPattern, MemorySpaceCastOpPattern,
-               StoreOpPattern, ReinterpretCastPattern, CastPattern,
-               ExtractAlignedPointerAsIndexOpPattern>(typeConverter,
-                                                      patterns.getContext());
+               CopyOpPattern, DeallocOpPattern, IntLoadOpPattern,
+               ImageLoadOpPattern, IntStoreOpPattern, LoadOpPattern,
+               MemorySpaceCastOpPattern, StoreOpPattern, ReinterpretCastPattern,
+               CastPattern, ExtractAlignedPointerAsIndexOpPattern>(
+      typeConverter, patterns.getContext());
 }
 } // namespace mlir
diff --git a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
index 931dd43be33c3..8702e82154664 100644
--- a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
+++ b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
@@ -544,6 +544,28 @@ module attributes {
 
 // -----
 
+// Check memref.copy lowering to spirv.CopyMemory.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Kernel, Addresses], []>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy
+func.func @copy() {
+  // CHECK: %[[SRC:.+]] = spirv.Variable : !spirv.ptr<!spirv.array<4 x f32>, Function>
+  // CHECK: %[[DST:.+]] = spirv.Variable : !spirv.ptr<!spirv.array<4 x f32>, Function>
+  // CHECK: spirv.CopyMemory "Function" %[[DST]], "Function" %[[SRC]]{{.*}} : !spirv.array<4 x f32>
+  %0 = memref.alloca() : memref<4xf32, #spirv.storage_class<Function>>
+  %1 = memref.alloca() : memref<4xf32, #spirv.storage_class<Function>>
+  memref.copy %0, %1 : memref<4xf32, #spirv.storage_class<Function>> to memref<4xf32, #spirv.storage_class<Function>>
+  return
+}
+
+} // end module
+
+// -----
+
 // Check Image Support.
 
 // CHECK: #[[$COLMAJMAP:.*]] = affine_map<(d0, d1) -> (d1, d0)>

>From 0f324623758c3f3c962a560b0034b00630735aee Mon Sep 17 00:00:00 2001
From: Arseniy Obolenskiy <arseniy.obolenskiy at amd.com>
Date: Wed, 22 Jul 2026 15:21:00 +0200
Subject: [PATCH 2/2] Address comment

---
 .../MemRefToSPIRV/MemRefToSPIRV.cpp           | 14 +++++
 .../MemRefToSPIRV/memref-to-spirv.mlir        | 63 +++++++++++++++++++
 2 files changed, 77 insertions(+)

diff --git a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
index de0356fe7b59e..7eca3d65052a3 100644
--- a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
+++ b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
@@ -1188,6 +1188,20 @@ StoreOpPattern::matchAndRewrite(memref::StoreOp storeOp, OpAdaptor adaptor,
 LogicalResult
 CopyOpPattern::matchAndRewrite(memref::CopyOp copyOp, OpAdaptor adaptor,
                                ConversionPatternRewriter &rewriter) const {
+  auto memrefType = cast<MemRefType>(copyOp.getSource().getType());
+  if (!memrefType.hasStaticShape())
+    return rewriter.notifyMatchFailure(copyOp, "unsupported dynamic shape");
+
+  for (MemRefType type :
+       {memrefType, cast<MemRefType>(copyOp.getTarget().getType())}) {
+    auto memorySpaceAttr =
+        dyn_cast_if_present<spirv::StorageClassAttr>(type.getMemorySpace());
+    if (memorySpaceAttr &&
+        memorySpaceAttr.getValue() == spirv::StorageClass::Image)
+      return rewriter.notifyMatchFailure(
+          copyOp, "cannot lower memref.copy in image storage class");
+  }
+
   // The converted operands are SPIR-V pointers to the source and target
   // storage. spirv.CopyMemory copies the whole pointed-to object, so it only
   // applies when both pointers point to the same fixed-size element type.
diff --git a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
index 8702e82154664..5163120a8339e 100644
--- a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
+++ b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
@@ -566,6 +566,69 @@ func.func @copy() {
 
 // -----
 
+// Check memref.copy is not lowered when either operand is in Image storage
+// class, since spirv.CopyMemory does not apply to image memory.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<#spirv.vce<v1.0, [
+    Shader,
+    Image1D,
+    StorageImageExtendedFormats
+  ], [
+    SPV_KHR_storage_buffer_storage_class
+  ]>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_image
+// CHECK: memref.copy
+func.func @copy_image(%arg0: memref<4xf32, #spirv.storage_class<Image>>, %arg1: memref<4xf32, #spirv.storage_class<Image>>) {
+  memref.copy %arg0, %arg1 : memref<4xf32, #spirv.storage_class<Image>> to memref<4xf32, #spirv.storage_class<Image>>
+  return
+}
+
+} // end module
+
+// -----
+
+// Check memref.copy is not lowered for dynamically shaped memrefs, since
+// spirv.CopyMemory requires the copied type to have a fixed size.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Shader], [SPV_KHR_storage_buffer_storage_class]>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_dynamic_shape
+// CHECK: memref.copy
+func.func @copy_dynamic_shape(%arg0: memref<?xf32, #spirv.storage_class<StorageBuffer>>, %arg1: memref<?xf32, #spirv.storage_class<StorageBuffer>>) {
+  memref.copy %arg0, %arg1 : memref<?xf32, #spirv.storage_class<StorageBuffer>> to memref<?xf32, #spirv.storage_class<StorageBuffer>>
+  return
+}
+
+} // end module
+
+// -----
+
+// Check memref.copy is not lowered for dynamically shaped memrefs under the
+// Kernel addressing model either, since the converted pointer only refers to
+// a single element and does not carry the dynamic extent.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Kernel, Addresses], []>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_dynamic_shape_kernel
+// CHECK: memref.copy
+func.func @copy_dynamic_shape_kernel(%arg0: memref<?xf32, #spirv.storage_class<CrossWorkgroup>>, %arg1: memref<?xf32, #spirv.storage_class<CrossWorkgroup>>) {
+  memref.copy %arg0, %arg1 : memref<?xf32, #spirv.storage_class<CrossWorkgroup>> to memref<?xf32, #spirv.storage_class<CrossWorkgroup>>
+  return
+}
+
+} // end module
+
+// -----
+
 // Check Image Support.
 
 // CHECK: #[[$COLMAJMAP:.*]] = affine_map<(d0, d1) -> (d1, d0)>


From llvmlistbot at llvm.org  Wed Jul 22 06:21:43 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 22 Jul 2026 06:21:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60c3e7.1905e68e.25404.1881@mx.google.com>


aobolensk wrote:

> I take OpCopy is roughly an equivalent to OpLoad and OpStore? If so does this pattern fire for the same memrefs? I think there was some work on supporting image fetches so I think we should against such memref in Copy lowering.

Yes, you're right. I have limited that now. Also, I have limited dyn shape case lowering

https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Wed Jul 22 06:45:05 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Wed, 22 Jul 2026 06:45:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60c961.5c68b3fc.12265b.750e@mx.google.com>


https://github.com/secona updated https://github.com/llvm/llvm-project/pull/210600

>From 8aa7d419ee5c0751b557f46e2881d94e902ea37e Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Sun, 19 Jul 2026 20:57:19 +0700
Subject: [PATCH 1/5] [mlir][SPIRV] Add SPIRVToLLVM conversion for selection
 with yielding values

---
 .../Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp    |  8 +++
 .../SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 56 +++++++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c5f9305a5c226..2982061c957e0 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1470,6 +1470,14 @@ class SelectionPattern : public SPIRVToLLVMConversion<spirv::SelectionOp> {
     auto position = rewriter.getInsertionPoint();
     auto *continueBlock = rewriter.splitBlock(currentBlock, position);
 
+    // Add arguments to the continue block for selections that yield values.
+    for (auto ty : op.getResultTypes()) {
+      Type dstTy = getTypeConverter()->convertType(ty);
+      if (!dstTy)
+        return rewriter.notifyMatchFailure(op, "failed to convert type");
+      continueBlock->addArgument(dstTy, loc);
+    }
+
     // Extract conditional branch information from the header block. By SPIR-V
     // dialect spec, it should contain `spirv.BranchConditional` or
     // `spirv.Switch` op. Note that `spirv.Switch op` is not supported at the
diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index ec47dd31cc637..389a21681bafb 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -214,6 +214,62 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4({{.*}}):
+    %one = spirv.Constant 1 : i32
+    spirv.ReturnValue %one : i32
+  }
 }
 
 // -----

>From edb194920ba746b10ee7f900cd75871e053801ab Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 19:52:25 +0700
Subject: [PATCH 2/5] test: clearer 1 value and 2 values

---
 .../test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index 389a21681bafb..769e9a0c61c20 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -236,7 +236,7 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
       spirv.mlir.merge %1 : i32
     }
-    // CHECK: ^bb4({{.*}}):
+    // CHECK: ^bb4({{.*}}: i32):
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
@@ -266,7 +266,7 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
       spirv.mlir.merge %1, %2 : i32, i32
     }
-    // CHECK: ^bb4({{.*}}):
+    // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }

>From 82837f0c3afd8b41035dd093319dfd59140d780f Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 20:03:19 +0700
Subject: [PATCH 3/5] test: return %0 in selection_with_yielding_value

---
 mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index 769e9a0c61c20..a043dafac6341 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -237,8 +237,7 @@ spirv.module Logical GLSL450 {
       spirv.mlir.merge %1 : i32
     }
     // CHECK: ^bb4({{.*}}: i32):
-    %one = spirv.Constant 1 : i32
-    spirv.ReturnValue %one : i32
+    spirv.ReturnValue %0 : i32
   }
 
   spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {

>From 26692812ded5658eea4eacc49d105cc0978e732a Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Mon, 20 Jul 2026 20:05:28 +0700
Subject: [PATCH 4/5] test: use both values in
 selection_with_multiple_yielding_values

---
 .../Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir     | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index a043dafac6341..1a946dd5df9ac 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -266,8 +266,9 @@ spirv.module Logical GLSL450 {
       spirv.mlir.merge %1, %2 : i32, i32
     }
     // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
-    %one = spirv.Constant 1 : i32
-    spirv.ReturnValue %one : i32
+    // Makes sure both values are used.
+    %sum = spirv.IAdd %0#0, %0#1 : i32
+    spirv.ReturnValue %sum : i32
   }
 }
 

>From 92b379cf911b96c0983a91343266dbb5c9163856 Mon Sep 17 00:00:00 2001
From: Vito Secona <secona00 at gmail.com>
Date: Wed, 22 Jul 2026 20:44:41 +0700
Subject: [PATCH 5/5] test: add CHECK for remaining lines

---
 .../Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir   | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index 1a946dd5df9ac..d9a721be7f240 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -236,7 +236,8 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
       spirv.mlir.merge %1 : i32
     }
-    // CHECK: ^bb4({{.*}}: i32):
+    // CHECK: ^bb4(%[[RES:.*]]: i32):
+    // CHECK: llvm.return %[[RES]] : i32
     spirv.ReturnValue %0 : i32
   }
 
@@ -265,9 +266,11 @@ spirv.module Logical GLSL450 {
       // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
       spirv.mlir.merge %1, %2 : i32, i32
     }
-    // CHECK: ^bb4({{.*}}: i32, {{.*}}: i32):
+    // CHECK: ^bb4(%[[ARG2:.*]]: i32, %[[ARG3:.*]]: i32):
     // Makes sure both values are used.
+    // CHECK: %[[RES:.*]] = llvm.add %[[ARG2]], %[[ARG3]] : i32
     %sum = spirv.IAdd %0#0, %0#1 : i32
+    // CHECK: llvm.return %[[RES]] : i32
     spirv.ReturnValue %sum : i32
   }
 }


From llvmlistbot at llvm.org  Wed Jul 22 06:46:52 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Wed, 22 Jul 2026 06:46:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60c9cc.2c570c9e.2ad8fb.5d8d@mx.google.com>


secona wrote:

This should be good now. Feel free to merge once you've approved; I don't have merge access yet.

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 06:48:46 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 06:48:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60ca3e.34fdf6f1.a29e6.38a0@mx.google.com>


https://github.com/IgWod approved this pull request.

LGTM, sorry for being slow, I've been juggling multiple tasks recently.

https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Wed Jul 22 06:50:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 06:50:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static and dynamic
 multi-rank OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60cab4.dcf3bad1.29a5a5.f0c3@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/210853

>From 2d005d7cf11a32e49f59fafddd6ccc1d39991825 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:05:29 -0700
Subject: [PATCH 01/11] [mlir][OpenACC] Support multi-rank array reductions

Initialize static rank-N per-thread accumulators and delinearize flattened reduction indices before accessing them.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 61 +++++++++---
 .../acc-cg-to-gpu-reduction-array.mlir        | 94 +++++++++++++++++++
 2 files changed, 142 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 77fa82ba6bb4e..44ad1f3154174 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,17 +736,28 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static rank-1");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
-  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
   Value step = arith::ConstantIndexOp::create(b, loc, 1);
-  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
-  OpBuilder::InsertionGuard g(b);
-  b.setInsertionPoint(forOp.getBody()->getTerminator());
-  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+  SmallVector<Value> indices;
+  auto buildLoopNest = [&](auto &&self, unsigned dim) -> void {
+    if (dim == baseTy.getRank()) {
+      memref::StoreOp::create(b, loc, ident, alloca, indices);
+      return;
+    }
+
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+    OpBuilder::InsertionGuard g(b);
+    b.setInsertionPoint(forOp.getBody()->getTerminator());
+    indices.push_back(forOp.getInductionVar());
+    self(self, dim + 1);
+    indices.pop_back();
+  };
+  buildLoopNest(buildLoopNest, 0);
 }
 
 std::optional<int64_t>
@@ -3244,10 +3255,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  if (!memref)
+  if (!memrefTy) {
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
-  if (memrefTy.getRank() != 1)
-    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() == 0) {
+    (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
+    return;
+  }
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
@@ -3355,9 +3375,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     OpBuilder::InsertionGuard guard(rewriter);
     rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
     Value iv = forOp.getInductionVar();
-    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
-    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
-                         ValueRange{iv},
+    SmallVector<Value> indices{iv};
+    if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
+      indices.resize(memrefTy.getRank());
+      Value linearIndex = iv;
+      for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
+        indices[dim] =
+            arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
+        if (dim != 0)
+          linearIndex =
+              arith::DivUIOp::create(rewriter, loc, linearIndex, dimSize);
+      }
+    }
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, indices);
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(), indices,
                          /*isPerThreadPrivateTarget=*/true);
   }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index bfb829c901af3..a008c84108f05 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -134,3 +134,97 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_array_reduction
+// CHECK: %[[R2_ALLOCA:.*]] = memref.alloca() : memref<2x3xi32>
+// CHECK: scf.for %[[R2_I:.*]] =
+// CHECK:   scf.for %[[R2_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[R2_ALLOCA]][%[[R2_I]], %[[R2_J]]] : memref<2x3xi32>
+// CHECK: scf.for %[[R2_LINEAR:.*]] =
+// CHECK: %[[R2_J_IDX:.*]] = arith.remui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+
+func.func @rank_two_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x3xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_three_array_reduction
+// CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_I:.*]] =
+// CHECK:   scf.for %[[R3_J:.*]] =
+// CHECK:     scf.for %[[R3_K:.*]] =
+// CHECK:       memref.store %{{.*}}, %[[R3_ALLOCA]][%[[R3_I]], %[[R3_J]], %[[R3_K]]] : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_LINEAR:.*]] =
+// CHECK: %[[R3_K_IDX:.*]] = arith.remui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_PLANE:.*]] = arith.divui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_J_IDX:.*]] = arith.remui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+
+func.func @rank_three_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x2x2xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x2x2xi32>>) {
+    %c8 = arith.constant 8 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x2x2xi32>>) -> memref<2x2x2xi32>
+    %bounds = acc.bounds extent(%c8 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x2x2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
+// invalid indexing operations.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_zero_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @rank_zero_array_reduction(%local: memref<i32>) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
+    %extent = arith.constant 1 : index
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 1a70c1679ce1c49a2e54fc8ed501ba06fa3d7234 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 17:43:57 -0700
Subject: [PATCH 02/11] [mlir][acc] Use dynamic stack storage for
 thread-private arrays

Avoid grid-sized backing allocations for dynamic per-thread storage while preserving shared lowering for oversized static arrays.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 +++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 41 ++++++++++++----
 3 files changed, 90 insertions(+), 28 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 44ad1f3154174..7135ca9177153 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,8 +736,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static ranked");
+  assert(baseTy.getRank() > 0 &&
+         "per-thread array reduction accumulator must have positive rank");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
@@ -749,7 +749,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
       return;
     }
 
-    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    Value ub =
+        baseTy.isDynamicDim(dim)
+            ? memref::DimOp::create(b, loc, alloca, dim).getResult()
+            : arith::ConstantIndexOp::create(b, loc, baseTy.getDimSize(dim))
+                  .getResult();
     auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
     OpBuilder::InsertionGuard g(b);
     b.setInsertionPoint(forOp.getBody()->getTerminator());
@@ -2494,9 +2498,15 @@ void ACCCGToGPULowering::processPrivateLocal(
     // the accumulate can reduce each element across threads.
     acc::ReductionAccumulateArrayOp arrayAccum =
         perThreadArrayReductionAccum(privateLocal.getResult());
-    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+    bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
+    bool canUseDynamicAlloca =
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -2583,9 +2593,15 @@ void ACCCGToGPULowering::processPrivateLocal(
   acc::ReductionAccumulateArrayOp arrayAccum =
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
+    bool canUseDynamicAlloca =
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
+        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
+                                              privatizeOp.getDynamicSizes());
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -3263,12 +3279,6 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
     return;
   }
-  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
-    (void)accSupport.emitNYI(loc,
-                             "reduction: dynamic multi-rank accumulate array");
-    return;
-  }
-
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3377,13 +3387,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
-      assert(memrefTy.hasStaticShape() &&
-             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize = arith::ConstantIndexOp::create(
-            rewriter, loc, memrefTy.getDimSize(dim));
+        Value dimSize =
+            memrefTy.isDynamicDim(dim)
+                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
+                : arith::ConstantIndexOp::create(rewriter, loc,
+                                                 memrefTy.getDimSize(dim))
+                      .getResult();
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index 281eae4b5b0f7..b186720512991 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,3 +23,32 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
+
+// Dynamic thread-private storage is allocated in the launch body. A
+// grid-wide backing allocation would scale with the logical block count.
+//
+// CHECK-LABEL: func.func @dynamic_threadprivate
+// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
+// CHECK:       gpu.launch
+// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
+// CHECK-NOT:     acc.unwrap_private
+
+func.func @dynamic_threadprivate(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
+      : (index) -> !acc.private_type<memref<?xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg = %private, %extent = %n)
+      : (!acc.private_type<memref<?xi32>>, index) {
+    %c0 = arith.constant 0 : index
+    %c0_i32 = arith.constant 0 : i32
+    %local = acc.private_local %arg
+        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
+    memref.store %c0_i32, %local[%c0] : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index a008c84108f05..f287ae054a846 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -192,25 +192,46 @@ func.func @rank_three_array_reduction() {
   return
 }
 
-// Unsupported dynamic multi-rank and rank-zero accumulators must not generate
-// invalid indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
 // CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[DYN_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_I:.*]] =
+// CHECK:   scf.for %[[DYN_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: scf.for %[[DYN_LINEAR:.*]] =
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: arith.remui %[[DYN_LINEAR]]
+// CHECK: arith.divui %[[DYN_LINEAR]]
+// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
   return
 }
 
+// Rank-zero accumulators must not generate invalid indexing operations.
 // CHECK-LABEL: func.func @rank_zero_array_reduction
 // CHECK: gpu.launch
 // CHECK-NOT: memref.load

>From 94cd48df3ac887a669b4f7c4cd871226d7014566 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:04:45 -0700
Subject: [PATCH 03/11] [mlir][OpenACC] Handle dynamic reductions on all thread
 dimensions

Classify dynamic accumulators using every GPU thread dimension so rank-N reductions consistently receive per-thread storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  7 +++--
 .../acc-cg-to-gpu-reduction-array.mlir        | 29 ++++---------------
 2 files changed, 9 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 7135ca9177153..e353f70939699 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -2500,7 +2500,8 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
+        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
@@ -2594,7 +2595,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
+        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
     if ((parDim.isThreadX() || arrayAccum) &&
@@ -3334,7 +3335,7 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   } else {
     isPerThreadPrivate = llvm::any_of(
         op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isThreadX(); });
+        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index f287ae054a846..be91cc6762da6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -211,41 +211,22 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_x]>}
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
-
-// Rank-zero accumulators must not generate invalid indexing operations.
-// CHECK-LABEL: func.func @rank_zero_array_reduction
-// CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @rank_zero_array_reduction(%local: memref<i32>) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local) : (memref<i32>) {
-    %extent = arith.constant 1 : index
-    %bounds = acc.bounds extent(%extent : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<i32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}

>From caf6ec726262faae429717edb8cadc1b6cd00abc Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:20:44 -0700
Subject: [PATCH 04/11] [mlir][OpenACC] Find array reductions through
 descriptor chains

Trace accumulate operands back to private storage so descriptor construction cannot bypass per-thread allocation and initialization.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 23 ++++++++++++
 .../acc-cg-to-gpu-reduction-array.mlir        | 35 +++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index e353f70939699..cc3b8910ad765 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -727,6 +727,29 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
+
+  // Descriptor construction may obscure the forward use chain. Fall back to
+  // tracing each accumulate operand back to this private local.
+  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
+  if (!privateLocal)
+    return nullptr;
+  acc::ComputeRegionOp computeRegion =
+      privateLocal->getParentOfType<acc::ComputeRegionOp>();
+  if (!computeRegion)
+    return nullptr;
+  acc::ReductionAccumulateArrayOp result;
+  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
+    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
+      return WalkResult::advance();
+    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
+                                  [](auto pd) { return pd.isAnyThread(); });
+    if (!hasThread || !reductionHasBlockContext(accArr))
+      return WalkResult::advance();
+    result = accArr;
+    return WalkResult::interrupt();
+  });
+  if (result)
+    return result;
   return nullptr;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index be91cc6762da6..ae9ea1e4c95bb 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -230,3 +230,38 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: acc.unwrap_private
+// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
+// CHECK: scf.for %[[INDIRECT_I:.*]] =
+// CHECK:   scf.for %[[INDIRECT_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
+// CHECK: gpu.all_reduce add
+func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %extent = arith.muli %m, %n : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+      : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  acc.kernel_environment {
+    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor_value = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
+      %bounds = acc.bounds extent(%ext : index)
+      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+          : memref<?x?xi32>
+          {par_dims = #acc<par_dims[block_x, thread_y]>}
+      acc.yield
+    } {origin = "acc.parallel"}
+  }
+  return
+}

>From 3790839be0295d1bd7f4ca3bbd151aac1edd5a9a Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 18:34:24 -0700
Subject: [PATCH 05/11] [mlir][OpenACC] Allocate dynamic privates for any
 thread dimension

Use privatization parallel dimensions so descriptor indirection cannot force dynamic thread-private backing arrays into grid-sized storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 48 +++++++------------
 .../acc-cg-to-gpu-reduction-array.mlir        | 34 ++++++++++---
 2 files changed, 46 insertions(+), 36 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index cc3b8910ad765..99315cad34e1c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,6 +383,14 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
+/// True when \p privatize has any thread-level parallelism.
+static bool isThreadPrivatize(PrivatizeOp privatize) {
+  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
+    return llvm::any_of(parDimsAttr.getArray(),
+                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
+  return false;
+}
+
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -728,28 +736,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     }
   }
 
-  // Descriptor construction may obscure the forward use chain. Fall back to
-  // tracing each accumulate operand back to this private local.
-  acc::PrivateLocalOp privateLocal = v.getDefiningOp<acc::PrivateLocalOp>();
-  if (!privateLocal)
-    return nullptr;
-  acc::ComputeRegionOp computeRegion =
-      privateLocal->getParentOfType<acc::ComputeRegionOp>();
-  if (!computeRegion)
-    return nullptr;
-  acc::ReductionAccumulateArrayOp result;
-  computeRegion.walk([&](acc::ReductionAccumulateArrayOp accArr) {
-    if (getPrivateLocalForMemref(accArr.getMemref()) != privateLocal)
-      return WalkResult::advance();
-    bool hasThread = llvm::any_of(accArr.getParDims().getArray(),
-                                  [](auto pd) { return pd.isAnyThread(); });
-    if (!hasThread || !reductionHasBlockContext(accArr))
-      return WalkResult::advance();
-    result = accArr;
-    return WalkResult::interrupt();
-  });
-  if (result)
-    return result;
   return nullptr;
 }
 
@@ -2523,12 +2509,13 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        (isThreadPrivate || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((isThreadPrivate || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (isThreadPrivate || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2618,12 +2605,13 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        (parDim.isThreadX() || arrayAccum) && baseTy.getRank() > 0 &&
+        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    if ((parDim.isThreadX() || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
+    bool canUseStaticAlloca =
+        (parDim.isThreadX() || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
+    if (canUseStaticAlloca || canUseDynamicAlloca) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index ae9ea1e4c95bb..b9fdf6ecabcb6 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -245,17 +245,39 @@ func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %private_data = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
+  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
+        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
+            %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>,
+           !acc.private_type<memref<1xindex>>, index) {
+      %c0 = arith.constant 0 : index
+      %c1_k = arith.constant 1 : index
+      %c0_i32 = arith.constant 0 : i32
+      %local_data = acc.private_local %arg_data
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %descriptor_value = builtin.unrealized_conversion_cast %local
-          : memref<?x?xi32> to memref<?x?xi32>
+      %local_descriptor = acc.private_local %arg_descriptor
+          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
+      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
+      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
+      scf.for %i = %c0 to %m_dim step %c1_k {
+        scf.for %j = %c0 to %n_dim step %c1_k {
+          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
+        }
+      }
+      %descriptor = builtin.unrealized_conversion_cast %local_data
+          : memref<?x?xi32> to index
+      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
+      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
+      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
+          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
           : memref<?x?xi32>

>From affc0c39172f713075ac1912eb631d85b4ce6b56 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 19:10:24 -0700
Subject: [PATCH 06/11] [mlir][OpenACC] Preserve worker-private array storage

Restrict generated stack reductions to thread_x and keep thread_y/thread_z arrays indexed or shared, including explicit dynamic shared memory.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp |  50 ++++----
 .../acc-cg-to-gpu-reduction-array.mlir        | 107 +++++++++++-------
 2 files changed, 86 insertions(+), 71 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 99315cad34e1c..48e7d13730eeb 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -383,14 +383,6 @@ static bool isThreadXPrivatize(PrivatizeOp privatize) {
   return false;
 }
 
-/// True when \p privatize has any thread-level parallelism.
-static bool isThreadPrivatize(PrivatizeOp privatize) {
-  if (GPUParallelDimsAttr parDimsAttr = privatize.getParDimsAttr())
-    return llvm::any_of(parDimsAttr.getArray(),
-                        [](GPUParallelDimAttr d) { return d.isAnyThread(); });
-  return false;
-}
-
 /// Emits a workgroup-wide GPU barrier.
 static void emitGPUBarrierWorkgroup(OpBuilder &builder, Location loc) {
   gpu::BarrierOp::create(builder, loc);
@@ -709,7 +701,7 @@ static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
 }
 
 /// Returns the array reduction accumulate (through cast/view ops) that \p v
-/// feeds if it needs per-thread storage: its par_dims include a thread dim
+/// feeds if it needs per-thread storage: its par_dims include thread_x
 /// and it has block context so the cross-thread all_reduce is well defined.
 static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   SmallVector<Value> worklist{v};
@@ -721,10 +713,10 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     for (Operation *user : cur.getUsers()) {
       if (acc::ReductionAccumulateArrayOp accArr =
               dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
-        bool hasThread = false;
+        bool hasThreadX = false;
         for (auto pd : accArr.getParDims().getArray())
-          hasThread |= pd.isAnyThread();
-        if (hasThread && reductionHasBlockContext(accArr))
+          hasThreadX |= pd.isThreadX();
+        if (hasThreadX && reductionHasBlockContext(accArr))
           return accArr;
         continue;
       }
@@ -735,7 +727,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(user->result_begin(), user->result_end());
     }
   }
-
   return nullptr;
 }
 
@@ -2509,13 +2500,11 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
+        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (isThreadPrivate || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((isThreadPrivate || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -2605,13 +2594,12 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        isThreadPrivatize(privatizeOp) && baseTy.getRank() > 0 &&
+        parDim.isThreadX() && baseTy.getRank() > 0 &&
         !baseTy.hasStaticShape() &&
         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-    bool canUseStaticAlloca =
-        (parDim.isThreadX() || arrayAccum) &&
-        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack);
-    if (canUseStaticAlloca || canUseDynamicAlloca) {
+    if ((parDim.isThreadX() || arrayAccum) &&
+        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
+         canUseDynamicAlloca)) {
       Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
                                               privatizeOp.getDynamicSizes());
       if (arrayAccum) {
@@ -3337,16 +3325,20 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   // it is too large. For a dynamically-shaped accumulator the type conveys no
   // size, so classify from par_dims (which the producer sets to the reduction's
   // actual parallel scope): a thread dimension means per-thread storage.
+  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
+  bool isExplicitlyShared =
+      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
   bool isPerThreadPrivate;
   if (memrefTy.hasStaticShape()) {
-    Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
     isPerThreadPrivate =
-        !isa_and_nonnull<memref::AllocOp>(rootOp) &&
+        !isExplicitlyShared &&
         canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
   } else {
-    isPerThreadPrivate = llvm::any_of(
-        op.getParDims().getArray(),
-        [](mlir::acc::GPUParallelDimAttr d) { return d.isAnyThread(); });
+    isPerThreadPrivate = !isExplicitlyShared &&
+                         llvm::any_of(op.getParDims().getArray(),
+                                      [](mlir::acc::GPUParallelDimAttr d) {
+                                        return d.isThreadX();
+                                      });
   }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index b9fdf6ecabcb6..269e0a1287046 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,27 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// CHECK-LABEL: func.func @array_reduction_dynamic_shared
+// CHECK: acc.gpu_shared_memory
+// CHECK-NOT: gpu.all_reduce
+func.func @array_reduction_dynamic_shared(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%ext = %n) : (index) {
+    %shared = acc.gpu_shared_memory(%ext)
+        {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
+        : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
@@ -162,6 +183,33 @@ func.func @rank_two_array_reduction() {
   return
 }
 
+// CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
+// CHECK: gpu.launch
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @rank_two_array_reduction_thread_y() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_y]>]
+      : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %kty = %ty)
+      ins(%arg = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg
+        {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+        : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<2x3xi32>
+        {par_dims = #acc<par_dims[block_x, thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @rank_three_array_reduction
 // CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
 // CHECK: scf.for %[[R3_I:.*]] =
@@ -211,75 +259,50 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
     } {origin = "acc.parallel"}
   }
   return
 }
 
-// CHECK-LABEL: func.func @descriptor_indirect_dynamic_thread_y
+// Dynamic worker-level storage must remain indexed by thread_y instead of
+// becoming one stack allocation per physical CUDA thread.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
-// CHECK-NOT: acc.unwrap_private
-// CHECK: %[[INDIRECT_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
-// CHECK: scf.for %[[INDIRECT_I:.*]] =
-// CHECK:   scf.for %[[INDIRECT_J:.*]] =
-// CHECK:     memref.store %{{.*}}, %[[INDIRECT_ALLOCA]][%[[INDIRECT_I]], %[[INDIRECT_J]]] : memref<?x?xi32>
-// CHECK: gpu.all_reduce add
-func.func @descriptor_indirect_dynamic_thread_y(%m: index, %n: index) {
+// CHECK-NOT: memref.alloca
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private_data = acc.privatize(%m, %n)
-      [#acc<par_dims[block_x, thread_y]>]
+  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  %private_descriptor = acc.privatize [#acc<par_dims[block_x, thread_y]>]
-      : () -> !acc.private_type<memref<1xindex>>
   acc.kernel_environment {
     acc.compute_region launch(%kbx = %bx, %kty = %ty)
-        ins(%arg_data = %private_data, %arg_descriptor = %private_descriptor,
-            %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>,
-           !acc.private_type<memref<1xindex>>, index) {
-      %c0 = arith.constant 0 : index
-      %c1_k = arith.constant 1 : index
-      %c0_i32 = arith.constant 0 : i32
-      %local_data = acc.private_local %arg_data
+        ins(%arg = %private, %ext = %extent)
+        : (!acc.private_type<memref<?x?xi32>>, index) {
+      %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %local_descriptor = acc.private_local %arg_descriptor
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
-          : (!acc.private_type<memref<1xindex>>) -> memref<1xindex>
-      %m_dim = memref.dim %local_data, %c0 : memref<?x?xi32>
-      %n_dim = memref.dim %local_data, %c1_k : memref<?x?xi32>
-      scf.for %i = %c0 to %m_dim step %c1_k {
-        scf.for %j = %c0 to %n_dim step %c1_k {
-          memref.store %c0_i32, %local_data[%i, %j] : memref<?x?xi32>
-        }
-      }
-      %descriptor = builtin.unrealized_conversion_cast %local_data
-          : memref<?x?xi32> to index
-      memref.store %descriptor, %local_descriptor[%c0] : memref<1xindex>
-      %loaded_descriptor = memref.load %local_descriptor[%c0] : memref<1xindex>
-      %descriptor_value = builtin.unrealized_conversion_cast %loaded_descriptor
-          : index to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %descriptor_value bounds(%bounds) <add>
+      acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_y]>}
       acc.yield

>From 3bba5f0a8722c8926daa522949668e420f5ea9aa Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:33:43 -0700
Subject: [PATCH 07/11] [mlir][OpenACC] Track array reduction storage
 provenance

Classify accumulators from their backing allocation through casts so shared storage is not reduced again and private storage is initialized correctly. Keep dynamic stack allocation reduction-specific and honor byte strides.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 71 ++++++++++++-------
 ...acc-cg-to-gpu-privatize-threadprivate.mlir | 29 --------
 .../acc-cg-to-gpu-reduction-array.mlir        | 61 ++++++++++------
 3 files changed, 82 insertions(+), 79 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 48e7d13730eeb..84243de10830a 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -317,6 +317,17 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     return false;
   }
 
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
+    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
+        castOp->getOperand(0) == trackedOperand &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      passThroughResults.push_back(castOp->getResult(0));
+      return true;
+    }
+    return false;
+  }
+
   // Partial-entity accesses (e.g. array element or field access) forward the
   // base entity through to their results, so treat them as pass-through when
   // the base entity is the value being tracked.
@@ -341,6 +352,14 @@ static Value unwrapMemRefConversion(Value v) {
         continue;
       }
     }
+    if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+        castOp && castOp->getNumOperands() == 1 &&
+        castOp->getNumResults() == 1 &&
+        isa<MemRefType>(castOp->getOperand(0).getType()) &&
+        isa<MemRefType>(castOp->getResult(0).getType())) {
+      v = castOp->getOperand(0);
+      continue;
+    }
     break;
   }
   return v;
@@ -730,6 +749,13 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   return nullptr;
 }
 
+static bool
+canUseDynamicReductionAlloca(MemRefType baseTy, acc::PrivatizeOp privatizeOp,
+                             acc::ReductionAccumulateArrayOp arrayAccum) {
+  return arrayAccum && !baseTy.hasStaticShape() &&
+         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+}
+
 /// Store the reduction identity to every element of a freshly allocated
 /// per-thread array accumulator so all lanes start from identity (the original
 /// init loop may only run on one lane).
@@ -2500,8 +2526,7 @@ void ACCCGToGPULowering::processPrivateLocal(
         perThreadArrayReductionAccum(privateLocal.getResult());
     bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
     bool canUseDynamicAlloca =
-        isThreadPrivate && baseTy.getRank() > 0 && !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((isThreadPrivate || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -2594,9 +2619,7 @@ void ACCCGToGPULowering::processPrivateLocal(
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
     bool canUseDynamicAlloca =
-        parDim.isThreadX() && baseTy.getRank() > 0 &&
-        !baseTy.hasStaticShape() &&
-        baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
+        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((parDim.isThreadX() || arrayAccum) &&
         (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
          canUseDynamicAlloca)) {
@@ -3318,28 +3341,10 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   }
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
-  // accumulator copy. For a statically-shaped accumulator, classify from the
-  // operand: an explicit shared allocation is block-shared regardless of size,
-  // and anything else (a per-thread stack alloca, or a view over one) is
-  // per-thread when it fits the per-thread stack budget and block-shared when
-  // it is too large. For a dynamically-shaped accumulator the type conveys no
-  // size, so classify from par_dims (which the producer sets to the reduction's
-  // actual parallel scope): a thread dimension means per-thread storage.
+  // accumulator copy. Classify the operand from its underlying allocation
+  // rather than its shape or par_dims: only memref.alloca is thread-private.
   Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isExplicitlyShared =
-      isa_and_nonnull<memref::AllocOp, acc::GPUSharedMemoryOp>(rootOp);
-  bool isPerThreadPrivate;
-  if (memrefTy.hasStaticShape()) {
-    isPerThreadPrivate =
-        !isExplicitlyShared &&
-        canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
-  } else {
-    isPerThreadPrivate = !isExplicitlyShared &&
-                         llvm::any_of(op.getParDims().getArray(),
-                                      [](mlir::acc::GPUParallelDimAttr d) {
-                                        return d.isThreadX();
-                                      });
-  }
+  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
@@ -3367,7 +3372,19 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
   Value lb =
       boundsOp.getLowerbound() ? toIndex(boundsOp.getLowerbound()) : zero;
-  Value step = boundsOp.getStride() ? toIndex(boundsOp.getStride()) : one;
+  Value step = one;
+  if (Value stride = boundsOp.getStride()) {
+    step = toIndex(stride);
+    if (boundsOp.getStrideInBytes()) {
+      int64_t elementSize =
+          getElementSizeInBytes(loc, memrefTy.getElementType());
+      if (elementSize == 0)
+        return;
+      Value elementSizeValue =
+          arith::ConstantIndexOp::create(rewriter, loc, elementSize);
+      step = arith::DivUIOp::create(rewriter, loc, step, elementSizeValue);
+    }
+  }
   // Exclusive upper bound. `extent` counts elements, so the span is
   // `extent * step` (for the common unit-stride case step is 1); fall back to
   // the inclusive upperbound when no extent is given.
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
index b186720512991..281eae4b5b0f7 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-privatize-threadprivate.mlir
@@ -23,32 +23,3 @@ func.func @threadprivate(%host: memref<i32>) {
   } {origin = "acc.parallel"}
   return
 }
-
-// Dynamic thread-private storage is allocated in the launch body. A
-// grid-wide backing allocation would scale with the logical block count.
-//
-// CHECK-LABEL: func.func @dynamic_threadprivate
-// CHECK:       acc.privatize(%{{.*}}) [#acc<par_dims[thread_x]>] : (index) -> !acc.private_type<memref<?xi32>>
-// CHECK:       gpu.launch
-// CHECK:         memref.alloca(%{{.*}}) : memref<?xi32>
-// CHECK-NOT:     acc.unwrap_private
-
-func.func @dynamic_threadprivate(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%n) [#acc<par_dims[thread_x]>]
-      : (index) -> !acc.private_type<memref<?xi32>>
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-      ins(%arg = %private, %extent = %n)
-      : (!acc.private_type<memref<?xi32>>, index) {
-    %c0 = arith.constant 0 : index
-    %c0_i32 = arith.constant 0 : i32
-    %local = acc.private_local %arg
-        : (!acc.private_type<memref<?xi32>>) -> memref<?xi32>
-    memref.store %c0_i32, %local[%c0] : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 269e0a1287046..56469f801f026 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -85,7 +85,8 @@ func.func @array_reduction_small_shared() {
 }
 
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
-// CHECK: acc.gpu_shared_memory
+// CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
+// CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]
 // CHECK-NOT: gpu.all_reduce
 func.func @array_reduction_dynamic_shared(%n: index) {
   %c1 = arith.constant 1 : index
@@ -96,8 +97,11 @@ func.func @array_reduction_dynamic_shared(%n: index) {
     %shared = acc.gpu_shared_memory(%ext)
         {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
         : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
+    %cast = builtin.unrealized_conversion_cast %shared
+        : memref<?xi32, #gpu.address_space<workgroup>>
+        to memref<?xi32, #gpu.address_space<workgroup>>
     %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %shared bounds(%bounds) <add>
+    acc.reduction_accumulate_array %cast bounds(%bounds) <add>
         : memref<?xi32, #gpu.address_space<workgroup>>
         {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -108,11 +112,13 @@ func.func @array_reduction_dynamic_shared(%n: index) {
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
-// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[BYTE_STEP:.*]] = arith.constant 8 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[ELEMENT_SIZE:.*]] = arith.constant 4 : index
+// CHECK: %[[ELEMENT_STEP:.*]] = arith.divui %[[BYTE_STEP]], %[[ELEMENT_SIZE]] : index
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[ELEMENT_STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
-// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[ELEMENT_STEP]]
 func.func @array_reduction_strided_extent() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -120,11 +126,11 @@ func.func @array_reduction_strided_extent() {
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
   acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
     %c1_b = arith.constant 1 : index
-    %c2 = arith.constant 2 : index
+    %c8 = arith.constant 8 : index
     %c3 = arith.constant 3 : index
     %local = memref.alloca() : memref<8xi32>
     %bounds = acc.bounds lowerbound(%c1_b : index) extent(%c3 : index)
-        stride(%c2 : index)
+        stride(%c8 : index) {strideInBytes = true}
     acc.reduction_accumulate_array %local bounds(%bounds) <add>
         : memref<8xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -132,13 +138,14 @@ func.func @array_reduction_strided_extent() {
   return
 }
 
-// A dynamically-shaped accumulator (a strided view whose type conveys no size)
-// is classified per-thread from par_dims: a thread dimension means per-thread
-// storage, so lowering emits the per-element gpu.all_reduce.
+// A dynamically-shaped view over an external buffer is shared even when the
+// accumulate includes thread_x. Storage provenance, not par_dims, controls
+// whether gpu.all_reduce is valid.
 //
 // CHECK-LABEL: func.func @array_reduction_dynamic_par_dims
-// CHECK: scf.for
-// CHECK: gpu.all_reduce add
+// CHECK: memref.reinterpret_cast
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
 func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -186,6 +193,7 @@ func.func @rank_two_array_reduction() {
 // CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
+// CHECK: acc.gpu_shared_memory
 // CHECK-NOT: gpu.all_reduce
 // CHECK: gpu.terminator
 func.func @rank_two_array_reduction_thread_y() {
@@ -247,12 +255,13 @@ func.func @rank_three_array_reduction() {
 // CHECK: scf.for %[[DYN_I:.*]] =
 // CHECK:   scf.for %[[DYN_J:.*]] =
 // CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
+// CHECK: %[[DYN_DESC:.*]] = builtin.unrealized_conversion_cast %[[DYN_ALLOCA]]
 // CHECK: scf.for %[[DYN_LINEAR:.*]] =
-// CHECK: memref.dim %[[DYN_ALLOCA]]
+// CHECK: memref.dim %[[DYN_DESC]]
 // CHECK: arith.remui %[[DYN_LINEAR]]
 // CHECK: arith.divui %[[DYN_LINEAR]]
-// CHECK: memref.dim %[[DYN_ALLOCA]]
-// CHECK: memref.load %[[DYN_ALLOCA]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
+// CHECK: memref.dim %[[DYN_DESC]]
+// CHECK: memref.load %[[DYN_DESC]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
 // CHECK: gpu.all_reduce add
 func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
@@ -269,8 +278,10 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
       %local = acc.private_local %arg
           {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
+      %descriptor = builtin.unrealized_conversion_cast %local
+          : memref<?x?xi32> to memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %local bounds(%bounds) <add>
+      acc.reduction_accumulate_array %descriptor bounds(%bounds) <add>
           : memref<?x?xi32>
           {par_dims = #acc<par_dims[block_x, thread_x]>}
       acc.yield
@@ -279,32 +290,36 @@ func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
   return
 }
 
-// Dynamic worker-level storage must remain indexed by thread_y instead of
+// Dynamic worker-level storage must remain indexed by thread_y/thread_z instead
 // becoming one stack allocation per physical CUDA thread.
-// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_y
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_yz
 // CHECK: gpu.launch
 // CHECK-NOT: memref.alloca
 // CHECK-NOT: gpu.all_reduce
+// CHECK: %[[WORKER_STORAGE:.*]] = memref.view %{{.*}}[%{{.*}}][%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}] : memref<?xi8> to memref<?x?x?x?x?xi32>
+// CHECK: memref.subview %[[WORKER_STORAGE]][%{{.*}}, %{{.*}}, %{{.*}}, 0, 0] [1, 1, 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1, 1]
 // CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction_thread_y(%m: index, %n: index) {
+func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_y]>]
+  %tz = acc.par_width %c128 {par_dim = #acc.par_dim<thread_z>}
+  %private = acc.privatize(%m, %n)
+      [#acc<par_dims[block_x, thread_y, thread_z]>]
       : (index, index) -> !acc.private_type<memref<?x?xi32>>
   acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty)
+    acc.compute_region launch(%kbx = %bx, %kty = %ty, %ktz = %tz)
         ins(%arg = %private, %ext = %extent)
         : (!acc.private_type<memref<?x?xi32>>, index) {
       %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
+          {acc.par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
           : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
       %bounds = acc.bounds extent(%ext : index)
       acc.reduction_accumulate_array %local bounds(%bounds) <add>
           : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y]>}
+          {par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
       acc.yield
     } {origin = "acc.parallel"}
   }

>From 08888a2115abad8c630b5fab31617fcc3e48821e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 20:57:42 -0700
Subject: [PATCH 08/11] [mlir][OpenACC] Test partial multi-rank reduction
 bounds

Cover delinearization with partial bounds and a non-identity memref layout.
---
 .../acc-cg-to-gpu-reduction-array.mlir        | 32 +++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 56469f801f026..357f1e14d7a49 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -325,3 +325,35 @@ func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
   }
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_partial_bounds_strided_layout
+// CHECK: %[[LB:.*]] = arith.constant 5 : index
+// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
+// CHECK: %[[LAYOUT_ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
+// CHECK: scf.for %[[LINEAR:.*]] = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: %[[COL:.*]] = arith.remui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW_LINEAR:.*]] = arith.divui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW:.*]] = arith.remui %[[ROW_LINEAR]], %{{.*}} : index
+// CHECK: memref.load %[[LAYOUT_ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
+func.func @rank_two_partial_bounds_strided_layout() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
+    %lb = arith.constant 5 : index
+    %step = arith.constant 2 : index
+    %extent = arith.constant 3 : index
+    %local = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+    %bounds = acc.bounds lowerbound(%lb : index) extent(%extent : index)
+        stride(%step : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<3x4xi32, strided<[8, 2]>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}

>From 35c5c495bad0d3e902e6b32cdb1a3365bd378b72 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Mon, 20 Jul 2026 22:44:28 -0700
Subject: [PATCH 09/11] [mlir][OpenACC] Preserve descriptor-backed reduction
 provenance

Trace unique stores and loads so descriptor-backed thread-private accumulators are initialized and reduced correctly without re-reducing indexed shared storage.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 155 +++++++++++++++++-
 .../acc-cg-to-gpu-reduction-array.mlir        |  29 ++++
 2 files changed, 181 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 84243de10830a..61ba40247d8b0 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -339,6 +339,119 @@ static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
     }
     return false;
   }
+
+  bool hasMemoryEffects = false;
+  if (auto effects = dyn_cast<MemoryEffectOpInterface>(userOp)) {
+    SmallVector<MemoryEffects::EffectInstance> instances;
+    effects.getEffects(instances);
+    hasMemoryEffects = !instances.empty();
+  }
+  if (userOp->getNumResults() == 1 &&
+      llvm::is_contained(userOp->getOperands(), trackedOperand) &&
+      !hasMemoryEffects) {
+    passThroughResults.push_back(userOp->getResult(0));
+    return true;
+  }
+  return false;
+}
+
+template <typename Effect>
+static bool hasEffectOnValue(Operation *op, Value value) {
+  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
+  if (!effects)
+    return false;
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  return llvm::any_of(instances, [&](const auto &instance) {
+    return isa<Effect>(instance.getEffect()) && instance.getValue() == value;
+  });
+}
+
+static Value getReadAddress(Operation *op) {
+  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
+  if (!effects)
+    return {};
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  Value address;
+  for (const auto &instance : instances) {
+    Value value = instance.getValue();
+    if (!isa<MemoryEffects::Read>(instance.getEffect()) || !value)
+      continue;
+    if (address && address != value)
+      return {};
+    address = value;
+  }
+  return address;
+}
+
+/// Returns the unique pointer-like value stored to \p address before \p before.
+static Value getUniqueStoredPointerValue(Value address, Operation *before) {
+  Operation *storedBy = nullptr;
+  Value stored;
+  SmallVector<Value> worklist{address};
+  DenseSet<Value> seen;
+  while (!worklist.empty()) {
+    Value alias = worklist.pop_back_val();
+    if (!seen.insert(alias).second)
+      continue;
+    for (Operation *user : alias.getUsers()) {
+      if (hasEffectOnValue<MemoryEffects::Write>(user, alias)) {
+        if (storedBy || user->getBlock() != before->getBlock() ||
+            !user->isBeforeInBlock(before))
+          return {};
+        Value candidate;
+        for (Value operand : user->getOperands()) {
+          if (operand == alias)
+            continue;
+          if (candidate)
+            return {};
+          candidate = operand;
+        }
+        if (!candidate)
+          return {};
+        storedBy = user;
+        stored = candidate;
+        continue;
+      }
+      SmallVector<Value> through;
+      if (getPassThroughResults(user, alias, through))
+        worklist.append(through.begin(), through.end());
+    }
+  }
+  return stored;
+}
+
+/// Tracks a uniquely stored pointer-like value through its subsequent load.
+static bool
+getMemoryPassThroughResults(Operation *userOp, Value trackedOperand,
+                            SmallVectorImpl<Value> &passThroughResults) {
+  if (hasEffectOnValue<MemoryEffects::Read>(userOp, trackedOperand) &&
+      userOp->getNumResults() == 1) {
+    passThroughResults.push_back(userOp->getResult(0));
+    return true;
+  }
+
+  if (!llvm::is_contained(userOp->getOperands(), trackedOperand))
+    return false;
+  auto effects = dyn_cast<MemoryEffectOpInterface>(userOp);
+  if (!effects)
+    return false;
+  SmallVector<MemoryEffects::EffectInstance> instances;
+  effects.getEffects(instances);
+  for (const auto &instance : instances) {
+    Value target = instance.getValue();
+    if (!isa<MemoryEffects::Write>(instance.getEffect()) || !target ||
+        target == trackedOperand)
+      continue;
+    unsigned writers = llvm::count_if(target.getUsers(), [&](Operation *user) {
+      return hasEffectOnValue<MemoryEffects::Write>(user, target);
+    });
+    if (writers != 1)
+      return false;
+    passThroughResults.push_back(target);
+    return true;
+  }
   return false;
 }
 
@@ -742,6 +855,8 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
       SmallVector<Value> through;
       if (getPassThroughResults(user, cur, through))
         worklist.append(through.begin(), through.end());
+      else if (getMemoryPassThroughResults(user, cur, through))
+        worklist.append(through.begin(), through.end());
       else if (isa<ViewLikeOpInterface>(user))
         worklist.append(user->result_begin(), user->result_end());
     }
@@ -790,6 +905,41 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
   buildLoopNest(buildLoopNest, 0);
 }
 
+/// True when \p memref resolves to storage owned by the current thread.
+static bool isPerThreadArrayReductionStorage(Value memref) {
+  DenseSet<Value> seen;
+  SmallVector<Value> worklist{memref};
+  while (!worklist.empty()) {
+    Value current = worklist.pop_back_val();
+    if (!current || !seen.insert(current).second)
+      continue;
+    Operation *def = current.getDefiningOp();
+    if (!def)
+      continue;
+    if (isa<memref::AllocaOp>(def))
+      return true;
+    if (isa<acc::UnwrapPrivateOp, acc::GPUSharedMemoryOp>(def))
+      return false;
+
+    if (Value address = getReadAddress(def)) {
+      Value stored = getUniqueStoredPointerValue(address, def);
+      if (!stored)
+        return false;
+      worklist.push_back(stored);
+      continue;
+    }
+
+    if (mlir::acc::GPUParallelDimsAttr parDims =
+            mlir::acc::getParDimsAttr(def)) {
+      return llvm::any_of(parDims.getArray(),
+                          [](auto dim) { return dim.isThreadX(); });
+    }
+
+    worklist.append(def->getOperands().begin(), def->getOperands().end());
+  }
+  return false;
+}
+
 std::optional<int64_t>
 ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
                                               MemRefType baseTy) {
@@ -3342,9 +3492,8 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
   // accumulator copy. Classify the operand from its underlying allocation
-  // rather than its shape or par_dims: only memref.alloca is thread-private.
-  Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
-  bool isPerThreadPrivate = isa_and_nonnull<memref::AllocaOp>(rootOp);
+  // rather than its shape or the accumulate's par_dims.
+  bool isPerThreadPrivate = isPerThreadArrayReductionStorage(memref);
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 357f1e14d7a49..1573e0cb76f2a 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,35 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// A reduction private that exceeds the per-thread stack budget uses indexed
+// backing storage shared by thread_x lanes. It must not be reduced again.
+// CHECK-LABEL: func.func @array_reduction_large_indexed_private
+// CHECK: acc.unwrap_private
+// CHECK: memref.view
+// CHECK: memref.subview
+// CHECK-NOT: gpu.all_reduce
+// CHECK-NOT: acc.reduction_accumulate_array
+func.func @array_reduction_large_indexed_private() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>]
+      : () -> !acc.private_type<memref<8192xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg0 = %private) : (!acc.private_type<memref<8192xi32>>) {
+    %c8192 = arith.constant 8192 : index
+    %local = acc.private_local %arg0
+        {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
+        : (!acc.private_type<memref<8192xi32>>) -> memref<8192xi32>
+    %bounds = acc.bounds extent(%c8192 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<8192xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // CHECK-LABEL: func.func @array_reduction_dynamic_shared
 // CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
 // CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]

>From 46f3341a39dfe96de4e1865dca58ffd94086f890 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Tue, 21 Jul 2026 06:12:10 -0700
Subject: [PATCH 10/11] [mlir][OpenACC] Harden reduction storage provenance

Use explicit storage-preserving interfaces and reject ambiguous aliases so descriptor-backed reductions are classified conservatively.
---
 .../Dialect/OpenACC/OpenACCOpsInterfaces.td   |  15 +++
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 115 +++++++++---------
 .../acc-cg-to-gpu-reduction-array.mlir        |  52 ++++++++
 3 files changed, 126 insertions(+), 56 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index eddd21a7e917b..33bd833ef9737 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -45,6 +45,21 @@ def PartialEntityAccessOpInterface : OpInterface<"PartialEntityAccessOpInterface
   ];
 }
 
+def UnderlyingStorageOpInterface : OpInterface<"UnderlyingStorageOpInterface"> {
+  let cppNamespace = "::mlir::acc";
+
+  let description = [{
+    An interface for operations that derive a pointer-like result from one
+    underlying storage value. Other operands may describe metadata such as
+    bounds, offsets, or strides and are not storage sources.
+  }];
+
+  let methods = [
+    InterfaceMethod<"Get the underlying storage value", "::mlir::Value",
+      "getStorageSource", (ins)>,
+  ];
+}
+
 def AddressOfGlobalOpInterface : OpInterface<"AddressOfGlobalOpInterface"> {
   let cppNamespace = "::mlir::acc";
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 61ba40247d8b0..845b913e81e90 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -306,53 +306,29 @@ static bool sameEffectiveValue(Value x, int64_t y) {
   return *conX == y;
 }
 
-/// Continues tracking a memref through view-like and partial-access ops.
+/// Returns the operand that preserves the storage represented by \p op.
+static Value getStorageSource(Operation *op) {
+  if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op))
+    return viewLike.getViewSource();
+  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
+      castOp && castOp->getNumOperands() == 1 && castOp->getNumResults() == 1)
+    return castOp->getOperand(0);
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(op))
+    return partialAccess.getBaseEntity();
+  if (acc::UnderlyingStorageOpInterface storage =
+          dyn_cast<acc::UnderlyingStorageOpInterface>(op))
+    return storage.getStorageSource();
+  return {};
+}
+
+/// Continues tracking a storage value through preserving operations.
 static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
                                   SmallVectorImpl<Value> &passThroughResults) {
-  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
-    if (viewLikeOp.getViewSource() == trackedOperand) {
-      passThroughResults.push_back(viewLikeOp.getViewDest());
-      return true;
-    }
+  if (getStorageSource(userOp) != trackedOperand)
     return false;
-  }
-
-  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(userOp)) {
-    if (castOp->getNumOperands() == 1 && castOp->getNumResults() == 1 &&
-        castOp->getOperand(0) == trackedOperand &&
-        isa<MemRefType>(castOp->getOperand(0).getType()) &&
-        isa<MemRefType>(castOp->getResult(0).getType())) {
-      passThroughResults.push_back(castOp->getResult(0));
-      return true;
-    }
-    return false;
-  }
-
-  // Partial-entity accesses (e.g. array element or field access) forward the
-  // base entity through to their results, so treat them as pass-through when
-  // the base entity is the value being tracked.
-  if (acc::PartialEntityAccessOpInterface partialAccess =
-          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
-    if (partialAccess.getBaseEntity() == trackedOperand) {
-      passThroughResults.append(userOp->result_begin(), userOp->result_end());
-      return true;
-    }
-    return false;
-  }
-
-  bool hasMemoryEffects = false;
-  if (auto effects = dyn_cast<MemoryEffectOpInterface>(userOp)) {
-    SmallVector<MemoryEffects::EffectInstance> instances;
-    effects.getEffects(instances);
-    hasMemoryEffects = !instances.empty();
-  }
-  if (userOp->getNumResults() == 1 &&
-      llvm::is_contained(userOp->getOperands(), trackedOperand) &&
-      !hasMemoryEffects) {
-    passThroughResults.push_back(userOp->getResult(0));
-    return true;
-  }
-  return false;
+  passThroughResults.append(userOp->result_begin(), userOp->result_end());
+  return true;
 }
 
 template <typename Effect>
@@ -385,8 +361,39 @@ static Value getReadAddress(Operation *op) {
   return address;
 }
 
+static bool hasAllocaRoot(Value value) {
+  DenseSet<Value> seen;
+  Value current = value;
+  while (current && seen.insert(current).second) {
+    Operation *def = current.getDefiningOp();
+    if (!def)
+      return false;
+    if (isa<memref::AllocaOp>(def))
+      return true;
+    current = getStorageSource(def);
+  }
+  return false;
+}
+
 /// Returns the unique pointer-like value stored to \p address before \p before.
 static Value getUniqueStoredPointerValue(Value address, Operation *before) {
+  Operation *scope = before->getParentOfType<acc::ComputeRegionOp>();
+  DominanceInfo dominance(scope ? scope : before->getParentOp());
+  auto safelyPrecedes = [&](Operation *store) {
+    if (dominance.dominates(store, before))
+      return true;
+    auto predicate = store->getParentOfType<acc::PredicateRegionOp>();
+    auto privateLocal = address.getDefiningOp<acc::PrivateLocalOp>();
+    GPUParallelDimsAttr parDims;
+    if (privateLocal)
+      parDims = mlir::acc::getParDimsAttr(privateLocal);
+    bool isThreadPrivate =
+        parDims && llvm::any_of(parDims.getArray(),
+                                [](auto dim) { return dim.isAnyThread(); });
+    return predicate && store->getParentOp() == predicate.getOperation() &&
+           dominance.dominates(predicate, before) &&
+           (isThreadPrivate || hasAllocaRoot(address));
+  };
   Operation *storedBy = nullptr;
   Value stored;
   SmallVector<Value> worklist{address};
@@ -397,8 +404,7 @@ static Value getUniqueStoredPointerValue(Value address, Operation *before) {
       continue;
     for (Operation *user : alias.getUsers()) {
       if (hasEffectOnValue<MemoryEffects::Write>(user, alias)) {
-        if (storedBy || user->getBlock() != before->getBlock() ||
-            !user->isBeforeInBlock(before))
+        if (storedBy || !safelyPrecedes(user))
           return {};
         Value candidate;
         for (Value operand : user->getOperands()) {
@@ -417,6 +423,8 @@ static Value getUniqueStoredPointerValue(Value address, Operation *before) {
       SmallVector<Value> through;
       if (getPassThroughResults(user, alias, through))
         worklist.append(through.begin(), through.end());
+      else if (!hasEffectOnValue<MemoryEffects::Read>(user, alias))
+        return {};
     }
   }
   return stored;
@@ -857,8 +865,6 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
         worklist.append(through.begin(), through.end());
       else if (getMemoryPassThroughResults(user, cur, through))
         worklist.append(through.begin(), through.end());
-      else if (isa<ViewLikeOpInterface>(user))
-        worklist.append(user->result_begin(), user->result_end());
     }
   }
   return nullptr;
@@ -908,14 +914,11 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
 /// True when \p memref resolves to storage owned by the current thread.
 static bool isPerThreadArrayReductionStorage(Value memref) {
   DenseSet<Value> seen;
-  SmallVector<Value> worklist{memref};
-  while (!worklist.empty()) {
-    Value current = worklist.pop_back_val();
-    if (!current || !seen.insert(current).second)
-      continue;
+  Value current = memref;
+  while (current && seen.insert(current).second) {
     Operation *def = current.getDefiningOp();
     if (!def)
-      continue;
+      return false;
     if (isa<memref::AllocaOp>(def))
       return true;
     if (isa<acc::UnwrapPrivateOp, acc::GPUSharedMemoryOp>(def))
@@ -925,7 +928,7 @@ static bool isPerThreadArrayReductionStorage(Value memref) {
       Value stored = getUniqueStoredPointerValue(address, def);
       if (!stored)
         return false;
-      worklist.push_back(stored);
+      current = stored;
       continue;
     }
 
@@ -935,7 +938,7 @@ static bool isPerThreadArrayReductionStorage(Value memref) {
                           [](auto dim) { return dim.isThreadX(); });
     }
 
-    worklist.append(def->getOperands().begin(), def->getOperands().end());
+    current = getStorageSource(def);
   }
   return false;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 1573e0cb76f2a..fd3bb1e5240f0 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,6 +84,58 @@ func.func @array_reduction_small_shared() {
   return
 }
 
+// Dynamic allocation metadata must not be mistaken for storage provenance.
+// CHECK-LABEL: func.func @array_reduction_heap_alloca_extent
+// CHECK: memref.alloca() : memref<index>
+// CHECK: %[[EXTENT:.*]] = memref.load
+// CHECK: memref.alloc(%[[EXTENT]]) : memref<?xi32>
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @array_reduction_heap_alloca_extent(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
+    %slot = memref.alloca() : memref<index>
+    memref.store %extent, %slot[] : memref<index>
+    %loaded_extent = memref.load %slot[] : memref<index>
+    %heap = memref.alloc(%loaded_extent) : memref<?xi32>
+    %bounds = acc.bounds extent(%loaded_extent : index)
+    acc.reduction_accumulate_array %heap bounds(%bounds) <add>
+        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    memref.dealloc %heap : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Non-storage operands of pure operations must not affect classification.
+// CHECK-LABEL: func.func @array_reduction_shared_select
+// CHECK: arith.select
+// CHECK-NOT: gpu.all_reduce
+// CHECK: gpu.terminator
+func.func @array_reduction_shared_select(%n: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
+    %true = arith.constant true
+    %condition_slot = memref.alloca() : memref<i1>
+    memref.store %true, %condition_slot[] : memref<i1>
+    %condition = memref.load %condition_slot[] : memref<i1>
+    %heap = memref.alloc(%extent) : memref<?xi32>
+    %selected = arith.select %condition, %heap, %heap : memref<?xi32>
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %selected bounds(%bounds) <add>
+        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    memref.dealloc %heap : memref<?xi32>
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
 // A reduction private that exceeds the per-thread stack budget uses indexed
 // backing storage shared by thread_x lanes. It must not be reduced again.
 // CHECK-LABEL: func.func @array_reduction_large_indexed_private

>From 63348cf99dcdd04d151fd312cb6e6c42d61273fe Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Wed, 22 Jul 2026 06:49:56 -0700
Subject: [PATCH 11/11] [mlir][OpenACC] Keep array reduction lowering
 mechanical

---
 .../Dialect/OpenACC/OpenACCOpsInterfaces.td   |  15 -
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 297 ++++--------------
 .../acc-cg-to-gpu-reduction-array.mlir        | 244 ++------------
 3 files changed, 80 insertions(+), 476 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
index 33bd833ef9737..eddd21a7e917b 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCOpsInterfaces.td
@@ -45,21 +45,6 @@ def PartialEntityAccessOpInterface : OpInterface<"PartialEntityAccessOpInterface
   ];
 }
 
-def UnderlyingStorageOpInterface : OpInterface<"UnderlyingStorageOpInterface"> {
-  let cppNamespace = "::mlir::acc";
-
-  let description = [{
-    An interface for operations that derive a pointer-like result from one
-    underlying storage value. Other operands may describe metadata such as
-    bounds, offsets, or strides and are not storage sources.
-  }];
-
-  let methods = [
-    InterfaceMethod<"Get the underlying storage value", "::mlir::Value",
-      "getStorageSource", (ins)>,
-  ];
-}
-
 def AddressOfGlobalOpInterface : OpInterface<"AddressOfGlobalOpInterface"> {
   let cppNamespace = "::mlir::acc";
 
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 845b913e81e90..f0fd7ae96ad4f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -306,159 +306,27 @@ static bool sameEffectiveValue(Value x, int64_t y) {
   return *conX == y;
 }
 
-/// Returns the operand that preserves the storage represented by \p op.
-static Value getStorageSource(Operation *op) {
-  if (ViewLikeOpInterface viewLike = dyn_cast<ViewLikeOpInterface>(op))
-    return viewLike.getViewSource();
-  if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
-      castOp && castOp->getNumOperands() == 1 && castOp->getNumResults() == 1)
-    return castOp->getOperand(0);
-  if (acc::PartialEntityAccessOpInterface partialAccess =
-          dyn_cast<acc::PartialEntityAccessOpInterface>(op))
-    return partialAccess.getBaseEntity();
-  if (acc::UnderlyingStorageOpInterface storage =
-          dyn_cast<acc::UnderlyingStorageOpInterface>(op))
-    return storage.getStorageSource();
-  return {};
-}
-
-/// Continues tracking a storage value through preserving operations.
+/// Continues tracking a memref through view-like and partial-access ops.
 static bool getPassThroughResults(Operation *userOp, Value trackedOperand,
                                   SmallVectorImpl<Value> &passThroughResults) {
-  if (getStorageSource(userOp) != trackedOperand)
-    return false;
-  passThroughResults.append(userOp->result_begin(), userOp->result_end());
-  return true;
-}
-
-template <typename Effect>
-static bool hasEffectOnValue(Operation *op, Value value) {
-  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
-  if (!effects)
-    return false;
-  SmallVector<MemoryEffects::EffectInstance> instances;
-  effects.getEffects(instances);
-  return llvm::any_of(instances, [&](const auto &instance) {
-    return isa<Effect>(instance.getEffect()) && instance.getValue() == value;
-  });
-}
-
-static Value getReadAddress(Operation *op) {
-  auto effects = dyn_cast<MemoryEffectOpInterface>(op);
-  if (!effects)
-    return {};
-  SmallVector<MemoryEffects::EffectInstance> instances;
-  effects.getEffects(instances);
-  Value address;
-  for (const auto &instance : instances) {
-    Value value = instance.getValue();
-    if (!isa<MemoryEffects::Read>(instance.getEffect()) || !value)
-      continue;
-    if (address && address != value)
-      return {};
-    address = value;
-  }
-  return address;
-}
-
-static bool hasAllocaRoot(Value value) {
-  DenseSet<Value> seen;
-  Value current = value;
-  while (current && seen.insert(current).second) {
-    Operation *def = current.getDefiningOp();
-    if (!def)
-      return false;
-    if (isa<memref::AllocaOp>(def))
+  if (ViewLikeOpInterface viewLikeOp = dyn_cast<ViewLikeOpInterface>(userOp)) {
+    if (viewLikeOp.getViewSource() == trackedOperand) {
+      passThroughResults.push_back(viewLikeOp.getViewDest());
       return true;
-    current = getStorageSource(def);
+    }
+    return false;
   }
-  return false;
-}
 
-/// Returns the unique pointer-like value stored to \p address before \p before.
-static Value getUniqueStoredPointerValue(Value address, Operation *before) {
-  Operation *scope = before->getParentOfType<acc::ComputeRegionOp>();
-  DominanceInfo dominance(scope ? scope : before->getParentOp());
-  auto safelyPrecedes = [&](Operation *store) {
-    if (dominance.dominates(store, before))
+  // Partial-entity accesses (e.g. array element or field access) forward the
+  // base entity through to their results, so treat them as pass-through when
+  // the base entity is the value being tracked.
+  if (acc::PartialEntityAccessOpInterface partialAccess =
+          dyn_cast<acc::PartialEntityAccessOpInterface>(userOp)) {
+    if (partialAccess.getBaseEntity() == trackedOperand) {
+      passThroughResults.append(userOp->result_begin(), userOp->result_end());
       return true;
-    auto predicate = store->getParentOfType<acc::PredicateRegionOp>();
-    auto privateLocal = address.getDefiningOp<acc::PrivateLocalOp>();
-    GPUParallelDimsAttr parDims;
-    if (privateLocal)
-      parDims = mlir::acc::getParDimsAttr(privateLocal);
-    bool isThreadPrivate =
-        parDims && llvm::any_of(parDims.getArray(),
-                                [](auto dim) { return dim.isAnyThread(); });
-    return predicate && store->getParentOp() == predicate.getOperation() &&
-           dominance.dominates(predicate, before) &&
-           (isThreadPrivate || hasAllocaRoot(address));
-  };
-  Operation *storedBy = nullptr;
-  Value stored;
-  SmallVector<Value> worklist{address};
-  DenseSet<Value> seen;
-  while (!worklist.empty()) {
-    Value alias = worklist.pop_back_val();
-    if (!seen.insert(alias).second)
-      continue;
-    for (Operation *user : alias.getUsers()) {
-      if (hasEffectOnValue<MemoryEffects::Write>(user, alias)) {
-        if (storedBy || !safelyPrecedes(user))
-          return {};
-        Value candidate;
-        for (Value operand : user->getOperands()) {
-          if (operand == alias)
-            continue;
-          if (candidate)
-            return {};
-          candidate = operand;
-        }
-        if (!candidate)
-          return {};
-        storedBy = user;
-        stored = candidate;
-        continue;
-      }
-      SmallVector<Value> through;
-      if (getPassThroughResults(user, alias, through))
-        worklist.append(through.begin(), through.end());
-      else if (!hasEffectOnValue<MemoryEffects::Read>(user, alias))
-        return {};
     }
-  }
-  return stored;
-}
-
-/// Tracks a uniquely stored pointer-like value through its subsequent load.
-static bool
-getMemoryPassThroughResults(Operation *userOp, Value trackedOperand,
-                            SmallVectorImpl<Value> &passThroughResults) {
-  if (hasEffectOnValue<MemoryEffects::Read>(userOp, trackedOperand) &&
-      userOp->getNumResults() == 1) {
-    passThroughResults.push_back(userOp->getResult(0));
-    return true;
-  }
-
-  if (!llvm::is_contained(userOp->getOperands(), trackedOperand))
-    return false;
-  auto effects = dyn_cast<MemoryEffectOpInterface>(userOp);
-  if (!effects)
     return false;
-  SmallVector<MemoryEffects::EffectInstance> instances;
-  effects.getEffects(instances);
-  for (const auto &instance : instances) {
-    Value target = instance.getValue();
-    if (!isa<MemoryEffects::Write>(instance.getEffect()) || !target ||
-        target == trackedOperand)
-      continue;
-    unsigned writers = llvm::count_if(target.getUsers(), [&](Operation *user) {
-      return hasEffectOnValue<MemoryEffects::Write>(user, target);
-    });
-    if (writers != 1)
-      return false;
-    passThroughResults.push_back(target);
-    return true;
   }
   return false;
 }
@@ -473,14 +341,6 @@ static Value unwrapMemRefConversion(Value v) {
         continue;
       }
     }
-    if (auto castOp = dyn_cast<UnrealizedConversionCastOp>(op);
-        castOp && castOp->getNumOperands() == 1 &&
-        castOp->getNumResults() == 1 &&
-        isa<MemRefType>(castOp->getOperand(0).getType()) &&
-        isa<MemRefType>(castOp->getResult(0).getType())) {
-      v = castOp->getOperand(0);
-      continue;
-    }
     break;
   }
   return v;
@@ -841,7 +701,7 @@ static bool reductionHasBlockContext(acc::ReductionAccumulateArrayOp accArr) {
 }
 
 /// Returns the array reduction accumulate (through cast/view ops) that \p v
-/// feeds if it needs per-thread storage: its par_dims include thread_x
+/// feeds if it needs per-thread storage: its par_dims include a thread dim
 /// and it has block context so the cross-thread all_reduce is well defined.
 static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
   SmallVector<Value> worklist{v};
@@ -853,38 +713,31 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
     for (Operation *user : cur.getUsers()) {
       if (acc::ReductionAccumulateArrayOp accArr =
               dyn_cast<acc::ReductionAccumulateArrayOp>(user)) {
-        bool hasThreadX = false;
+        bool hasThread = false;
         for (auto pd : accArr.getParDims().getArray())
-          hasThreadX |= pd.isThreadX();
-        if (hasThreadX && reductionHasBlockContext(accArr))
+          hasThread |= pd.isAnyThread();
+        if (hasThread && reductionHasBlockContext(accArr))
           return accArr;
         continue;
       }
       SmallVector<Value> through;
       if (getPassThroughResults(user, cur, through))
         worklist.append(through.begin(), through.end());
-      else if (getMemoryPassThroughResults(user, cur, through))
-        worklist.append(through.begin(), through.end());
+      else if (isa<ViewLikeOpInterface>(user))
+        worklist.append(user->result_begin(), user->result_end());
     }
   }
   return nullptr;
 }
 
-static bool
-canUseDynamicReductionAlloca(MemRefType baseTy, acc::PrivatizeOp privatizeOp,
-                             acc::ReductionAccumulateArrayOp arrayAccum) {
-  return arrayAccum && !baseTy.hasStaticShape() &&
-         baseTy.getNumDynamicDims() == privatizeOp.getDynamicSizes().size();
-}
-
 /// Store the reduction identity to every element of a freshly allocated
 /// per-thread array accumulator so all lanes start from identity (the original
 /// init loop may only run on one lane).
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() > 0 &&
-         "per-thread array reduction accumulator must have positive rank");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
@@ -896,11 +749,7 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
       return;
     }
 
-    Value ub =
-        baseTy.isDynamicDim(dim)
-            ? memref::DimOp::create(b, loc, alloca, dim).getResult()
-            : arith::ConstantIndexOp::create(b, loc, baseTy.getDimSize(dim))
-                  .getResult();
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
     auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
     OpBuilder::InsertionGuard g(b);
     b.setInsertionPoint(forOp.getBody()->getTerminator());
@@ -911,38 +760,6 @@ static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
   buildLoopNest(buildLoopNest, 0);
 }
 
-/// True when \p memref resolves to storage owned by the current thread.
-static bool isPerThreadArrayReductionStorage(Value memref) {
-  DenseSet<Value> seen;
-  Value current = memref;
-  while (current && seen.insert(current).second) {
-    Operation *def = current.getDefiningOp();
-    if (!def)
-      return false;
-    if (isa<memref::AllocaOp>(def))
-      return true;
-    if (isa<acc::UnwrapPrivateOp, acc::GPUSharedMemoryOp>(def))
-      return false;
-
-    if (Value address = getReadAddress(def)) {
-      Value stored = getUniqueStoredPointerValue(address, def);
-      if (!stored)
-        return false;
-      current = stored;
-      continue;
-    }
-
-    if (mlir::acc::GPUParallelDimsAttr parDims =
-            mlir::acc::getParDimsAttr(def)) {
-      return llvm::any_of(parDims.getArray(),
-                          [](auto dim) { return dim.isThreadX(); });
-    }
-
-    current = getStorageSource(def);
-  }
-  return false;
-}
-
 std::optional<int64_t>
 ACCCGToGPULowering::isEligibleForSharedMemory(acc::PrivateLocalOp privateLocal,
                                               MemRefType baseTy) {
@@ -2677,14 +2494,9 @@ void ACCCGToGPULowering::processPrivateLocal(
     // the accumulate can reduce each element across threads.
     acc::ReductionAccumulateArrayOp arrayAccum =
         perThreadArrayReductionAccum(privateLocal.getResult());
-    bool isThreadPrivate = isThreadXPrivatize(privatizeOp);
-    bool canUseDynamicAlloca =
-        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
-    if ((isThreadPrivate || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
-                                              privatizeOp.getDynamicSizes());
+    if ((isThreadXPrivatize(privatizeOp) || arrayAccum) &&
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -2771,13 +2583,9 @@ void ACCCGToGPULowering::processPrivateLocal(
   acc::ReductionAccumulateArrayOp arrayAccum =
       perThreadArrayReductionAccum(privateLocal.getResult());
   for (mlir::acc::GPUParallelDimAttr parDim : parDimsPair.first) {
-    bool canUseDynamicAlloca =
-        canUseDynamicReductionAlloca(baseTy, privatizeOp, arrayAccum);
     if ((parDim.isThreadX() || arrayAccum) &&
-        (canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack) ||
-         canUseDynamicAlloca)) {
-      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy,
-                                              privatizeOp.getDynamicSizes());
+        canUseStackAlloca(baseTy, loc, options.maxThreadPrivateStack)) {
+      Value alloca = memref::AllocaOp::create(rewriter, loc, baseTy);
       if (arrayAccum) {
         FailureOr<arith::AtomicRMWKind> kind = getReductionKind(
             arrayAccum.getReductionOperator(), baseTy.getElementType(), loc);
@@ -3451,10 +3259,12 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
     return;
   }
-  if (memrefTy.getRank() == 0) {
-    (void)accSupport.emitNYI(loc, "reduction: rank-zero accumulate array");
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
     return;
   }
+
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3494,9 +3304,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   }
 
   // Per-element gpu.all_reduce is only correct when each thread owns its own
-  // accumulator copy. Classify the operand from its underlying allocation
-  // rather than its shape or the accumulate's par_dims.
-  bool isPerThreadPrivate = isPerThreadArrayReductionStorage(memref);
+  // accumulator copy. For a statically-shaped accumulator, classify from the
+  // operand: an explicit shared allocation is block-shared regardless of size,
+  // and anything else (a per-thread stack alloca, or a view over one) is
+  // per-thread when it fits the per-thread stack budget and block-shared when
+  // it is too large. For a dynamically-shaped accumulator the type conveys no
+  // size, so classify from par_dims (which the producer sets to the reduction's
+  // actual parallel scope): a thread dimension means per-thread storage.
+  bool isPerThreadPrivate;
+  if (memrefTy.hasStaticShape()) {
+    Operation *rootOp = unwrapMemRefConversion(memref).getDefiningOp();
+    isPerThreadPrivate =
+        !isa_and_nonnull<memref::AllocOp>(rootOp) &&
+        canUseStackAlloca(memrefTy, loc, options.maxThreadPrivateStack);
+  } else {
+    isPerThreadPrivate = llvm::any_of(
+        op.getParDims().getArray(),
+        [](mlir::acc::GPUParallelDimAttr d) { return d.isThreadX(); });
+  }
   if (!isPerThreadPrivate) {
     // Block-shared accumulator: no-op only when the accumulate spans a block
     // dim (threads distribute distinct elements, so the block partial is in
@@ -3524,19 +3349,7 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
   Value one = arith::ConstantIndexOp::create(rewriter, loc, 1);
   Value lb =
       boundsOp.getLowerbound() ? toIndex(boundsOp.getLowerbound()) : zero;
-  Value step = one;
-  if (Value stride = boundsOp.getStride()) {
-    step = toIndex(stride);
-    if (boundsOp.getStrideInBytes()) {
-      int64_t elementSize =
-          getElementSizeInBytes(loc, memrefTy.getElementType());
-      if (elementSize == 0)
-        return;
-      Value elementSizeValue =
-          arith::ConstantIndexOp::create(rewriter, loc, elementSize);
-      step = arith::DivUIOp::create(rewriter, loc, step, elementSizeValue);
-    }
-  }
+  Value step = boundsOp.getStride() ? toIndex(boundsOp.getStride()) : one;
   // Exclusive upper bound. `extent` counts elements, so the span is
   // `extent * step` (for the common unit-stride case step is 1); fall back to
   // the inclusive upperbound when no extent is given.
@@ -3560,15 +3373,13 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize =
-            memrefTy.isDynamicDim(dim)
-                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
-                : arith::ConstantIndexOp::create(rewriter, loc,
-                                                 memrefTy.getDimSize(dim))
-                      .getResult();
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index fd3bb1e5240f0..ace560d869792 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -84,122 +84,14 @@ func.func @array_reduction_small_shared() {
   return
 }
 
-// Dynamic allocation metadata must not be mistaken for storage provenance.
-// CHECK-LABEL: func.func @array_reduction_heap_alloca_extent
-// CHECK: memref.alloca() : memref<index>
-// CHECK: %[[EXTENT:.*]] = memref.load
-// CHECK: memref.alloc(%[[EXTENT]]) : memref<?xi32>
-// CHECK-NOT: gpu.all_reduce
-// CHECK: gpu.terminator
-func.func @array_reduction_heap_alloca_extent(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
-    %slot = memref.alloca() : memref<index>
-    memref.store %extent, %slot[] : memref<index>
-    %loaded_extent = memref.load %slot[] : memref<index>
-    %heap = memref.alloc(%loaded_extent) : memref<?xi32>
-    %bounds = acc.bounds extent(%loaded_extent : index)
-    acc.reduction_accumulate_array %heap bounds(%bounds) <add>
-        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    memref.dealloc %heap : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
-
-// Non-storage operands of pure operations must not affect classification.
-// CHECK-LABEL: func.func @array_reduction_shared_select
-// CHECK: arith.select
-// CHECK-NOT: gpu.all_reduce
-// CHECK: gpu.terminator
-func.func @array_reduction_shared_select(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%extent = %n) : (index) {
-    %true = arith.constant true
-    %condition_slot = memref.alloca() : memref<i1>
-    memref.store %true, %condition_slot[] : memref<i1>
-    %condition = memref.load %condition_slot[] : memref<i1>
-    %heap = memref.alloc(%extent) : memref<?xi32>
-    %selected = arith.select %condition, %heap, %heap : memref<?xi32>
-    %bounds = acc.bounds extent(%extent : index)
-    acc.reduction_accumulate_array %selected bounds(%bounds) <add>
-        : memref<?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    memref.dealloc %heap : memref<?xi32>
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
-
-// A reduction private that exceeds the per-thread stack budget uses indexed
-// backing storage shared by thread_x lanes. It must not be reduced again.
-// CHECK-LABEL: func.func @array_reduction_large_indexed_private
-// CHECK: acc.unwrap_private
-// CHECK: memref.view
-// CHECK: memref.subview
-// CHECK-NOT: gpu.all_reduce
-// CHECK-NOT: acc.reduction_accumulate_array
-func.func @array_reduction_large_indexed_private() {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>]
-      : () -> !acc.private_type<memref<8192xi32>>
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-      ins(%arg0 = %private) : (!acc.private_type<memref<8192xi32>>) {
-    %c8192 = arith.constant 8192 : index
-    %local = acc.private_local %arg0
-        {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
-        : (!acc.private_type<memref<8192xi32>>) -> memref<8192xi32>
-    %bounds = acc.bounds extent(%c8192 : index)
-    acc.reduction_accumulate_array %local bounds(%bounds) <add>
-        : memref<8192xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
-
-// CHECK-LABEL: func.func @array_reduction_dynamic_shared
-// CHECK: %[[DYN_SHARED:.*]] = acc.gpu_shared_memory
-// CHECK: builtin.unrealized_conversion_cast %[[DYN_SHARED]]
-// CHECK-NOT: gpu.all_reduce
-func.func @array_reduction_dynamic_shared(%n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%ext = %n) : (index) {
-    %shared = acc.gpu_shared_memory(%ext)
-        {num_copies = 1 : i64, static_upper_bound_bytes = 4096 : i64}
-        : (index) -> memref<?xi32, #gpu.address_space<workgroup>>
-    %cast = builtin.unrealized_conversion_cast %shared
-        : memref<?xi32, #gpu.address_space<workgroup>>
-        to memref<?xi32, #gpu.address_space<workgroup>>
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %cast bounds(%bounds) <add>
-        : memref<?xi32, #gpu.address_space<workgroup>>
-        {par_dims = #acc<par_dims[block_x, thread_x]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
-
 // CHECK-LABEL: func.func @array_reduction_strided_extent
 // CHECK: gpu.launch
 // CHECK: %[[LB:.*]] = arith.constant 1 : index
-// CHECK: %[[BYTE_STEP:.*]] = arith.constant 8 : index
+// CHECK: %[[STEP:.*]] = arith.constant 2 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[ELEMENT_SIZE:.*]] = arith.constant 4 : index
-// CHECK: %[[ELEMENT_STEP:.*]] = arith.divui %[[BYTE_STEP]], %[[ELEMENT_SIZE]] : index
-// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[ELEMENT_STEP]] : index
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
-// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[ELEMENT_STEP]]
+// CHECK: scf.for %{{.*}} = %[[LB]] to %[[UB]] step %[[STEP]]
 func.func @array_reduction_strided_extent() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -207,11 +99,11 @@ func.func @array_reduction_strided_extent() {
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
   acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
     %c1_b = arith.constant 1 : index
-    %c8 = arith.constant 8 : index
+    %c2 = arith.constant 2 : index
     %c3 = arith.constant 3 : index
     %local = memref.alloca() : memref<8xi32>
     %bounds = acc.bounds lowerbound(%c1_b : index) extent(%c3 : index)
-        stride(%c8 : index) {strideInBytes = true}
+        stride(%c2 : index)
     acc.reduction_accumulate_array %local bounds(%bounds) <add>
         : memref<8xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
@@ -219,14 +111,13 @@ func.func @array_reduction_strided_extent() {
   return
 }
 
-// A dynamically-shaped view over an external buffer is shared even when the
-// accumulate includes thread_x. Storage provenance, not par_dims, controls
-// whether gpu.all_reduce is valid.
+// A dynamically-shaped accumulator (a strided view whose type conveys no size)
+// is classified per-thread from par_dims: a thread dimension means per-thread
+// storage, so lowering emits the per-element gpu.all_reduce.
 //
 // CHECK-LABEL: func.func @array_reduction_dynamic_par_dims
-// CHECK: memref.reinterpret_cast
-// CHECK-NOT: gpu.all_reduce
-// CHECK: gpu.terminator
+// CHECK: scf.for
+// CHECK: gpu.all_reduce add
 func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
@@ -254,6 +145,8 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
 // CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
 // CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
 // CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+// CHECK: %[[R2_RESULT:.*]] = gpu.all_reduce add
+// CHECK: memref.store %[[R2_RESULT]], %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]]
 
 func.func @rank_two_array_reduction() {
   %c1 = arith.constant 1 : index
@@ -271,34 +164,6 @@ func.func @rank_two_array_reduction() {
   return
 }
 
-// CHECK-LABEL: func.func @rank_two_array_reduction_thread_y
-// CHECK: gpu.launch
-// CHECK-NOT: memref.alloca
-// CHECK: acc.gpu_shared_memory
-// CHECK-NOT: gpu.all_reduce
-// CHECK: gpu.terminator
-func.func @rank_two_array_reduction_thread_y() {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %private = acc.privatize [#acc<par_dims[block_x, thread_y]>]
-      : () -> !acc.private_type<memref<2x3xi32>>
-  acc.compute_region launch(%kbx = %bx, %kty = %ty)
-      ins(%arg = %private) : (!acc.private_type<memref<2x3xi32>>) {
-    %c6 = arith.constant 6 : index
-    %local = acc.private_local %arg
-        {acc.par_dims = #acc<par_dims[block_x, thread_y]>}
-        : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
-    %bounds = acc.bounds extent(%c6 : index)
-    acc.reduction_accumulate_array %local bounds(%bounds) <add>
-        : memref<2x3xi32>
-        {par_dims = #acc<par_dims[block_x, thread_y]>}
-    acc.yield
-  } {origin = "acc.parallel"}
-  return
-}
-
 // CHECK-LABEL: func.func @rank_three_array_reduction
 // CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
 // CHECK: scf.for %[[R3_I:.*]] =
@@ -312,6 +177,8 @@ func.func @rank_two_array_reduction_thread_y() {
 // CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
 // CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
 // CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+// CHECK: %[[R3_RESULT:.*]] = gpu.all_reduce add
+// CHECK: memref.store %[[R3_RESULT]], %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]]
 
 func.func @rank_three_array_reduction() {
   %c1 = arith.constant 1 : index
@@ -329,81 +196,22 @@ func.func @rank_three_array_reduction() {
   return
 }
 
+// Unsupported dynamic multi-rank accumulators must not generate invalid
+// indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
 // CHECK: gpu.launch
-// CHECK-NOT: acc.unwrap_private
-// CHECK: %[[DYN_ALLOCA:.*]] = memref.alloca(%{{.*}}, %{{.*}}) : memref<?x?xi32>
-// CHECK: scf.for %[[DYN_I:.*]] =
-// CHECK:   scf.for %[[DYN_J:.*]] =
-// CHECK:     memref.store %{{.*}}, %[[DYN_ALLOCA]][%[[DYN_I]], %[[DYN_J]]] : memref<?x?xi32>
-// CHECK: %[[DYN_DESC:.*]] = builtin.unrealized_conversion_cast %[[DYN_ALLOCA]]
-// CHECK: scf.for %[[DYN_LINEAR:.*]] =
-// CHECK: memref.dim %[[DYN_DESC]]
-// CHECK: arith.remui %[[DYN_LINEAR]]
-// CHECK: arith.divui %[[DYN_LINEAR]]
-// CHECK: memref.dim %[[DYN_DESC]]
-// CHECK: memref.load %[[DYN_DESC]][%{{.*}}, %{{.*}}] : memref<?x?xi32>
-// CHECK: gpu.all_reduce add
-func.func @dynamic_rank_two_array_reduction(%m: index, %n: index) {
-  %c1 = arith.constant 1 : index
-  %c128 = arith.constant 128 : index
-  %extent = arith.muli %m, %n : index
-  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  %private = acc.privatize(%m, %n) [#acc<par_dims[block_x, thread_x]>]
-      : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %ktx = %tx)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_x]>}
-          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %descriptor = builtin.unrealized_conversion_cast %local
-          : memref<?x?xi32> to memref<?x?xi32>
-      %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %descriptor bounds(%bounds) <add>
-          : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_x]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
-  return
-}
-
-// Dynamic worker-level storage must remain indexed by thread_y/thread_z instead
-// becoming one stack allocation per physical CUDA thread.
-// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction_thread_yz
-// CHECK: gpu.launch
-// CHECK-NOT: memref.alloca
-// CHECK-NOT: gpu.all_reduce
-// CHECK: %[[WORKER_STORAGE:.*]] = memref.view %{{.*}}[%{{.*}}][%{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}, %{{.*}}] : memref<?xi8> to memref<?x?x?x?x?xi32>
-// CHECK: memref.subview %[[WORKER_STORAGE]][%{{.*}}, %{{.*}}, %{{.*}}, 0, 0] [1, 1, 1, %{{.*}}, %{{.*}}] [1, 1, 1, 1, 1]
+// CHECK-NOT: memref.load
 // CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
-  %extent = arith.muli %m, %n : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
-  %ty = acc.par_width %c128 {par_dim = #acc.par_dim<thread_y>}
-  %tz = acc.par_width %c128 {par_dim = #acc.par_dim<thread_z>}
-  %private = acc.privatize(%m, %n)
-      [#acc<par_dims[block_x, thread_y, thread_z]>]
-      : (index, index) -> !acc.private_type<memref<?x?xi32>>
-  acc.kernel_environment {
-    acc.compute_region launch(%kbx = %bx, %kty = %ty, %ktz = %tz)
-        ins(%arg = %private, %ext = %extent)
-        : (!acc.private_type<memref<?x?xi32>>, index) {
-      %local = acc.private_local %arg
-          {acc.par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
-          : (!acc.private_type<memref<?x?xi32>>) -> memref<?x?xi32>
-      %bounds = acc.bounds extent(%ext : index)
-      acc.reduction_accumulate_array %local bounds(%bounds) <add>
-          : memref<?x?xi32>
-          {par_dims = #acc<par_dims[block_x, thread_y, thread_z]>}
-      acc.yield
-    } {origin = "acc.parallel"}
-  }
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
   return
 }
 
@@ -411,14 +219,14 @@ func.func @dynamic_rank_two_array_reduction_thread_yz(%m: index, %n: index) {
 // CHECK: %[[LB:.*]] = arith.constant 5 : index
 // CHECK: %[[STEP:.*]] = arith.constant 2 : index
 // CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
-// CHECK: %[[LAYOUT_ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
 // CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
 // CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
 // CHECK: scf.for %[[LINEAR:.*]] = %[[LB]] to %[[UB]] step %[[STEP]]
 // CHECK: %[[COL:.*]] = arith.remui %[[LINEAR]], %{{.*}} : index
 // CHECK: %[[ROW_LINEAR:.*]] = arith.divui %[[LINEAR]], %{{.*}} : index
 // CHECK: %[[ROW:.*]] = arith.remui %[[ROW_LINEAR]], %{{.*}} : index
-// CHECK: memref.load %[[LAYOUT_ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: memref.load %[[ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
 func.func @rank_two_partial_bounds_strided_layout() {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index


From llvmlistbot at llvm.org  Wed Jul 22 06:52:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 06:52:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static and dynamic
 multi-rank OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60cb09.35334a41.b898e.2d7e@mx.google.com>


https://github.com/khaki3 edited https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Wed Jul 22 06:54:26 2026
From: llvmlistbot at llvm.org (Ahmad Tameem)
Date: Wed, 22 Jul 2026 06:54:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Allow extra DPS inputs in
 convolution dim inference (PR #198462)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198462@github.com>
Message-ID: <6a60cb92.8e75e0cb.2de827.991b@mx.google.com>


https://github.com/Tameem-10xE updated https://github.com/llvm/llvm-project/pull/198462

>From 57a844c35d3bc82c5c159c5dd827df6b1255565b Mon Sep 17 00:00:00 2001
From: Tameem-10xE <ahmad.tameem at 10xengineers.ai>
Date: Wed, 22 Jul 2026 18:51:37 +0500
Subject: [PATCH] [mlir][linalg] Allow extra DPS inputs in convolution dim
 inference

Signed-off-by: Tameem-10xE <ahmad.tameem at 10xengineers.ai>
---
 .../Dialect/Linalg/IR/LinalgInterfaces.cpp    |   2 +-
 .../Linalg/InferConvolutionDimsTest.cpp       | 130 +++++++++++++++++-
 2 files changed, 130 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
index e3f4988b3ab76..2f04ce66fa326 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgInterfaces.cpp
@@ -884,7 +884,7 @@ static FailureOr<ConvolutionDimensions> inferConvolutionDimsImpl(
 /// Returns a failure if `output_image` (and implicitly `filter_loop`) is empty.
 FailureOr<ConvolutionDimensions>
 mlir::linalg::inferConvolutionDims(LinalgOp linalgOp) {
-  if (linalgOp.getNumDpsInits() != 1 || linalgOp.getNumDpsInputs() != 2)
+  if (linalgOp.getNumDpsInits() != 1 || linalgOp.getNumDpsInputs() < 2)
     return failure();
 
   auto indexingMaps = linalgOp.getIndexingMapsArray();
diff --git a/mlir/unittests/Dialect/Linalg/InferConvolutionDimsTest.cpp b/mlir/unittests/Dialect/Linalg/InferConvolutionDimsTest.cpp
index cab17f2a6e9c1..1dd4e7cb94755 100644
--- a/mlir/unittests/Dialect/Linalg/InferConvolutionDimsTest.cpp
+++ b/mlir/unittests/Dialect/Linalg/InferConvolutionDimsTest.cpp
@@ -130,6 +130,88 @@ createConv2DWithSwappedFilterLoops(OpBuilder &builder,
       });
 }
 
+/// Creates a quantized 2D convolution as a linalg.generic with the same
+/// input/filter access pattern as linalg.conv_2d, plus extra scalar
+/// input/filter zero-point operands (mirroring the `*_q` named ops).
+/// The zero points are hard-coded constants since their values do not
+/// affect dimension inference.
+///
+/// Loop order:
+///   d0 = output height (oh), parallel
+///   d1 = output width (ow), parallel
+///   d2 = kernel height (kh), reduction
+///   d3 = kernel width (kw), reduction
+///
+/// Indexing maps:
+///   input:     (d0 + d2, d1 + d3)
+///   filter:    (d2, d3)
+///   input zp:  scalar
+///   filter zp: scalar
+///   output:    (d0, d1)
+///
+/// Semantic pairing: d0 <-> d2, d1 <-> d3
+static linalg::GenericOp createQConv2DOp(OpBuilder &builder, int64_t oh,
+                                         int64_t ow, int64_t kh, int64_t kw) {
+  Location loc = builder.getUnknownLoc();
+  MLIRContext *ctx = builder.getContext();
+
+  auto i8Type = builder.getI8Type();
+  auto i32Type = builder.getI32Type();
+
+  int64_t ih = oh + kh - 1;
+  int64_t iw = ow + kw - 1;
+
+  auto inputType = RankedTensorType::get({ih, iw}, i8Type);
+  auto filterType = RankedTensorType::get({kh, kw}, i8Type);
+  auto outputType = RankedTensorType::get({oh, ow}, i32Type);
+
+  Value input = tensor::EmptyOp::create(builder, loc, inputType.getShape(),
+                                        inputType.getElementType());
+  Value filter = tensor::EmptyOp::create(builder, loc, filterType.getShape(),
+                                         filterType.getElementType());
+
+  // Non-zero input and filter zero points (their values are irrelevant to
+  // dimension inference).
+  Value inputZeroPoint = arith::ConstantIntOp::create(builder, loc, 7, 32);
+  Value filterZeroPoint = arith::ConstantIntOp::create(builder, loc, 9, 32);
+
+  Value output = tensor::EmptyOp::create(builder, loc, outputType.getShape(),
+                                         outputType.getElementType());
+
+  AffineExpr d0, d1, d2, d3;
+  bindDims(ctx, d0, d1, d2, d3);
+
+  auto inputMap = AffineMap::get(4, 0, {d0 + d2, d1 + d3}, ctx);
+  auto filterMap = AffineMap::get(4, 0, {d2, d3}, ctx);
+  auto scalarMap = AffineMap::get(4, 0, ArrayRef<AffineExpr>{}, ctx);
+  auto outputMap = AffineMap::get(4, 0, {d0, d1}, ctx);
+
+  SmallVector<AffineMap> indexingMaps = {inputMap, filterMap, scalarMap,
+                                         scalarMap, outputMap};
+
+  SmallVector<utils::IteratorType> iterTypes = {
+      utils::IteratorType::parallel, utils::IteratorType::parallel,
+      utils::IteratorType::reduction, utils::IteratorType::reduction};
+
+  return linalg::GenericOp::create(
+      builder, loc, outputType,
+      ValueRange{input, filter, inputZeroPoint, filterZeroPoint},
+      ValueRange{output}, indexingMaps, iterTypes,
+      [&](OpBuilder &b, Location loc, ValueRange args) {
+        Value inputI32 =
+            arith::ExtSIOp::create(b, loc, b.getI32Type(), args[0]);
+        Value filterI32 =
+            arith::ExtSIOp::create(b, loc, b.getI32Type(), args[1]);
+
+        inputI32 = arith::SubIOp::create(b, loc, inputI32, args[2]);
+        filterI32 = arith::SubIOp::create(b, loc, filterI32, args[3]);
+
+        Value mul = arith::MulIOp::create(b, loc, inputI32, filterI32);
+        Value add = arith::AddIOp::create(b, loc, args[4], mul);
+        linalg::YieldOp::create(b, loc, add);
+      });
+}
+
 TEST_F(InferConvolutionDimsTest, Conv2DPairing) {
   // Use non-square kernel to ensure dimension swapping is tested properly.
   const int64_t oh = 6, ow = 12, kh = 3, kw = 5;
@@ -191,7 +273,7 @@ static void expectConvDimsEq(const ConvolutionDimensions &lhs,
 }
 
 /// Verify that inferring convolution dimensions from indexing maps produces
-/// same result as inferring thrm directly from the convolution op.
+/// same result as inferring them directly from the convolution op.
 TEST_F(InferConvolutionDimsTest, MapsOverloadMatchesOpOverload) {
   OpBuilder builder(ctx.get());
   OwningOpRef<ModuleOp> module = ModuleOp::create(builder.getUnknownLoc());
@@ -312,4 +394,50 @@ TEST_F(InferConvolutionDimsTest, MapsOverloadRejectsInvalidInput) {
   }
 }
 
+/// Extra scalar zero-point operands must not affect convolution dim inference.
+TEST_F(InferConvolutionDimsTest, QConv2DWithZeroPoints) {
+  // Use non-square kernel to ensure dimension swapping is tested properly.
+  const int64_t oh = 6, ow = 12, kh = 3, kw = 5;
+
+  // Create a module to own all test operations and ensure proper cleanup.
+  OpBuilder builder(ctx.get());
+  OwningOpRef<ModuleOp> module = ModuleOp::create(builder.getUnknownLoc());
+  builder.setInsertionPointToStart(module->getBody());
+
+  // Create Quantize ConvOp with two extra scalar zero-point inputs.
+  linalg::GenericOp qConvOp = createQConv2DOp(builder, oh, ow, kh, kw);
+
+  // The qconv op should have:
+  //   input, filter, input zero point, filter zero point
+  ASSERT_EQ(qConvOp.getNumDpsInputs(), 4u);
+  ASSERT_EQ(qConvOp.getNumDpsInits(), 1u);
+
+  auto indexingMaps = qConvOp.getIndexingMapsArray();
+  ASSERT_EQ(indexingMaps.size(), static_cast<size_t>(qConvOp.getNumDpsInputs() +
+                                                     qConvOp.getNumDpsInits()));
+
+  // The two extra quantized conv operands must be scalar inputs.
+  EXPECT_EQ(indexingMaps[2].getNumResults(), 0u);
+  EXPECT_EQ(indexingMaps[3].getNumResults(), 0u);
+  EXPECT_EQ(indexingMaps[2].getNumDims(), 4u);
+  EXPECT_EQ(indexingMaps[3].getNumDims(), 4u);
+
+  FailureOr<ConvolutionDimensions> qConvDims = inferConvolutionDims(qConvOp);
+  ASSERT_TRUE(succeeded(qConvDims));
+
+  // The scalar zero-point operands must not affect dimension inference: the
+  // result must match a plain conv_2d, with standard pairing d0 <-> d2
+  // (oh <-> kh) and d1 <-> d3 (ow <-> kw), unit strides/dilations, and no
+  // batch/channel/depth dims.
+  ConvolutionDimensions expected{/*batch=*/{},
+                                 /*outputImage=*/{0, 1},
+                                 /*outputChannel=*/{},
+                                 /*filterLoop=*/{2, 3},
+                                 /*inputChannel=*/{},
+                                 /*depth=*/{},
+                                 /*strides=*/{1, 1},
+                                 /*dilations=*/{1, 1}};
+  expectConvDimsEq(expected, *qConvDims);
+}
+
 } // namespace


From llvmlistbot at llvm.org  Wed Jul 22 06:55:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 06:55:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Preserve debug location in
 OpenMPIRBuilder calls. (PR #211254)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211254@github.com>
Message-ID: <6a60cbda.4a342cbc.1aeb63.8823@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Abid Qadeer (abidh)

<details>
<summary>Changes</summary>

Many `OpenMPIRBuilder` entry points take an `OpenMPIRBuilder::LocationDescription`. It has two relevant constructors:

`LocationDescription(const IRBuilderBase &amp;IRB)`  // captures IP and debug loc
`LocationDescription(const InsertPointTy &amp;IP)`   // captures IP only; DL is empty

The OpenMP MLIR-to-LLVM-IR translation constructs the location from `builder` in almost all places (~45 call sites), which selects the first constructor and propagates both the insertion point and the current debug location. A few call sites instead passed `builder.saveIP()`, which selects the second constructor and silently drops the debug location.

Change the 9 offending call sites to pass `builder` instead of `builder.saveIP()`, so the debug location is preserved.

---
Full diff: https://github.com/llvm/llvm-project/pull/211254.diff


1 Files Affected:

- (modified) mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp (+13-13) 


``````````diff
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f2fefa1f5a53f..1e3cbeecdb8c7 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -271,7 +271,7 @@ class LinearClauseProcessor {
     // Emit barrier
     builder.SetInsertPoint(linearExitBB->getTerminator());
     return moduleTranslation.getOpenMPBuilder()->createBarrier(
-        builder.saveIP(), llvm::omp::OMPD_barrier);
+        builder, llvm::omp::OMPD_barrier);
   }
 
   // Emit stores for linear variables. Useful in case of SIMD
@@ -1654,8 +1654,8 @@ static LogicalResult createReductionsAndCleanup(
   llvm::UnreachableInst *tempTerminator = builder.CreateUnreachable();
   builder.SetInsertPoint(tempTerminator);
   llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-      ompBuilder->createReductions(builder.saveIP(), allocaIP, reductionInfos,
-                                   isByRef, isNowait, isTeamsReduction);
+      ompBuilder->createReductions(builder, allocaIP, reductionInfos, isByRef,
+                                   isNowait, isTeamsReduction);
 
   if (failed(handleError(contInsertPoint, *op)))
     return failure();
@@ -2003,7 +2003,7 @@ static LogicalResult copyFirstPrivateVars(
   if (insertBarrier && !opIsInSingleThread(op)) {
     llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy res =
-        ompBuilder->createBarrier(builder.saveIP(), llvm::omp::OMPD_barrier);
+        ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(res, *op)))
       return failure();
   }
@@ -4377,7 +4377,7 @@ convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
     return failure();
   }
 
-  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder.saveIP(), dds);
+  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder, dds);
   if (dds.DepArray) {
     builder.CreateFree(dds.DepArray);
   }
@@ -4537,7 +4537,7 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
                                         loopInfo->getPreheader());
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
         moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
+            builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
@@ -4734,9 +4734,10 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       builder.SetInsertPoint(tempTerminator);
 
       llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-          ompBuilder->createReductions(
-              builder.saveIP(), allocaIP, reductionInfos, isByRef,
-              /*IsNoWait=*/false, /*IsTeamsReduction=*/false);
+          ompBuilder->createReductions(builder, allocaIP, reductionInfos,
+                                       isByRef,
+                                       /*IsNoWait=*/false,
+                                       /*IsTeamsReduction=*/false);
       if (!contInsertPoint)
         return contInsertPoint.takeError();
 
@@ -9698,8 +9699,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
               return failure();
 
             llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-                ompBuilder->createBarrier(builder.saveIP(),
-                                          llvm::omp::OMPD_barrier);
+                ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
             LogicalResult res = handleError(afterIP, *op);
             if (res.succeeded()) {
               // If the barrier generated a cancellation check, the insertion
@@ -9712,7 +9712,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             if (failed(checkImplementationStatus(*op)))
               return failure();
 
-            ompBuilder->createTaskyield(builder.saveIP());
+            ompBuilder->createTaskyield(builder);
             return success();
           })
           .Case([&](omp::FlushOp op) {
@@ -9727,7 +9727,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             //
             // The argument list is discarded so that, flush with a list is
             // treated same as a flush without a list.
-            ompBuilder->createFlush(builder.saveIP());
+            ompBuilder->createFlush(builder);
             return success();
           })
           .Case([&](omp::ParallelOp op) {

``````````

</details>


https://github.com/llvm/llvm-project/pull/211254

From llvmlistbot at llvm.org  Wed Jul 22 06:57:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 06:57:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static and dynamic
 multi-rank OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60cc59.492e23e4.292f20.168f@mx.google.com>


khaki3 wrote:

>would it be worth carrying that decision forward explicitly (a small side-table keyed on the storage value, or an attribute on acc.reduction_accumulate_array) rather than reconstructing it through casts/stores/loads? 

Yes, you are right. This PR does not touch the classification logic anymore. It only adds rank-N initialization and indexing. I agree that we should carry the storage decision explicitly, but that should be separate work.

This PR is ready.

https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Wed Jul 22 07:03:46 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 22 Jul 2026 07:03:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static and dynamic
 multi-rank OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60cdc2.507fc713.3702e2.fbf8@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Thank you. This latest version looks much more simplified for statically sized arrays.

https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Wed Jul 22 07:03:59 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 22 Jul 2026 07:03:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60cdcf.3cce8983.247e47.7d54@mx.google.com>


aobolensk wrote:

> LGTM, sorry for being slow, I've been juggling multiple tasks recently.

No problem, thanks for the reviews!

https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Wed Jul 22 07:04:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 07:04:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 75e6377 - [mlir][SPIR-V] Lower memref.copy to
 spirv.CopyMemory (#206016)
Message-ID: <6a60cdd1.c56fb102.6921f.fe74@mx.google.com>


Author: Arseniy Obolenskiy
Date: 2026-07-22T16:03:55+02:00
New Revision: 75e637791bff7b6c0c11754fb29991b6b140ee2c

URL: https://github.com/llvm/llvm-project/commit/75e637791bff7b6c0c11754fb29991b6b140ee2c
DIFF: https://github.com/llvm/llvm-project/commit/75e637791bff7b6c0c11754fb29991b6b140ee2c.diff

LOG: [mlir][SPIR-V] Lower memref.copy to spirv.CopyMemory (#206016)

Added: 
    

Modified: 
    mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
    mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
index 4674aa351315f..7eca3d65052a3 100644
--- a/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
+++ b/mlir/lib/Conversion/MemRefToSPIRV/MemRefToSPIRV.cpp
@@ -339,6 +339,16 @@ class StoreOpPattern final : public OpConversionPattern<memref::StoreOp> {
                   ConversionPatternRewriter &rewriter) const override;
 };
 
+/// Converts memref.copy to spirv.CopyMemory.
+class CopyOpPattern final : public OpConversionPattern<memref::CopyOp> {
+public:
+  using Base::Base;
+
+  LogicalResult
+  matchAndRewrite(memref::CopyOp copyOp, OpAdaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override;
+};
+
 class ReinterpretCastPattern final
     : public OpConversionPattern<memref::ReinterpretCastOp> {
 public:
@@ -1171,6 +1181,48 @@ StoreOpPattern::matchAndRewrite(memref::StoreOp storeOp, OpAdaptor adaptor,
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// CopyOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+CopyOpPattern::matchAndRewrite(memref::CopyOp copyOp, OpAdaptor adaptor,
+                               ConversionPatternRewriter &rewriter) const {
+  auto memrefType = cast<MemRefType>(copyOp.getSource().getType());
+  if (!memrefType.hasStaticShape())
+    return rewriter.notifyMatchFailure(copyOp, "unsupported dynamic shape");
+
+  for (MemRefType type :
+       {memrefType, cast<MemRefType>(copyOp.getTarget().getType())}) {
+    auto memorySpaceAttr =
+        dyn_cast_if_present<spirv::StorageClassAttr>(type.getMemorySpace());
+    if (memorySpaceAttr &&
+        memorySpaceAttr.getValue() == spirv::StorageClass::Image)
+      return rewriter.notifyMatchFailure(
+          copyOp, "cannot lower memref.copy in image storage class");
+  }
+
+  // The converted operands are SPIR-V pointers to the source and target
+  // storage. spirv.CopyMemory copies the whole pointed-to object, so it only
+  // applies when both pointers point to the same fixed-size element type.
+  Value source = adaptor.getSource();
+  Value target = adaptor.getTarget();
+  auto sourcePtrType = dyn_cast<spirv::PointerType>(source.getType());
+  auto targetPtrType = dyn_cast<spirv::PointerType>(target.getType());
+  if (!sourcePtrType || !targetPtrType)
+    return rewriter.notifyMatchFailure(copyOp, "failed to convert memref type");
+
+  if (sourcePtrType.getPointeeType() != targetPtrType.getPointeeType())
+    return rewriter.notifyMatchFailure(
+        copyOp, "source and target pointee types do not match");
+
+  rewriter.replaceOpWithNewOp<spirv::CopyMemoryOp>(
+      copyOp, target, source, /*memory_access=*/spirv::MemoryAccessAttr{},
+      /*alignment=*/IntegerAttr{}, /*source_memory_access=*/
+      spirv::MemoryAccessAttr{}, /*source_alignment=*/IntegerAttr{});
+  return success();
+}
+
 LogicalResult ReinterpretCastPattern::matchAndRewrite(
     memref::ReinterpretCastOp op, OpAdaptor adaptor,
     ConversionPatternRewriter &rewriter) const {
@@ -1239,10 +1291,10 @@ namespace mlir {
 void populateMemRefToSPIRVPatterns(const SPIRVTypeConverter &typeConverter,
                                    RewritePatternSet &patterns) {
   patterns.add<AllocaOpPattern, AllocOpPattern, AtomicRMWOpPattern,
-               DeallocOpPattern, IntLoadOpPattern, ImageLoadOpPattern,
-               IntStoreOpPattern, LoadOpPattern, MemorySpaceCastOpPattern,
-               StoreOpPattern, ReinterpretCastPattern, CastPattern,
-               ExtractAlignedPointerAsIndexOpPattern>(typeConverter,
-                                                      patterns.getContext());
+               CopyOpPattern, DeallocOpPattern, IntLoadOpPattern,
+               ImageLoadOpPattern, IntStoreOpPattern, LoadOpPattern,
+               MemorySpaceCastOpPattern, StoreOpPattern, ReinterpretCastPattern,
+               CastPattern, ExtractAlignedPointerAsIndexOpPattern>(
+      typeConverter, patterns.getContext());
 }
 } // namespace mlir

diff  --git a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
index 931dd43be33c3..5163120a8339e 100644
--- a/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
+++ b/mlir/test/Conversion/MemRefToSPIRV/memref-to-spirv.mlir
@@ -544,6 +544,91 @@ module attributes {
 
 // -----
 
+// Check memref.copy lowering to spirv.CopyMemory.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Kernel, Addresses], []>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy
+func.func @copy() {
+  // CHECK: %[[SRC:.+]] = spirv.Variable : !spirv.ptr<!spirv.array<4 x f32>, Function>
+  // CHECK: %[[DST:.+]] = spirv.Variable : !spirv.ptr<!spirv.array<4 x f32>, Function>
+  // CHECK: spirv.CopyMemory "Function" %[[DST]], "Function" %[[SRC]]{{.*}} : !spirv.array<4 x f32>
+  %0 = memref.alloca() : memref<4xf32, #spirv.storage_class<Function>>
+  %1 = memref.alloca() : memref<4xf32, #spirv.storage_class<Function>>
+  memref.copy %0, %1 : memref<4xf32, #spirv.storage_class<Function>> to memref<4xf32, #spirv.storage_class<Function>>
+  return
+}
+
+} // end module
+
+// -----
+
+// Check memref.copy is not lowered when either operand is in Image storage
+// class, since spirv.CopyMemory does not apply to image memory.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<#spirv.vce<v1.0, [
+    Shader,
+    Image1D,
+    StorageImageExtendedFormats
+  ], [
+    SPV_KHR_storage_buffer_storage_class
+  ]>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_image
+// CHECK: memref.copy
+func.func @copy_image(%arg0: memref<4xf32, #spirv.storage_class<Image>>, %arg1: memref<4xf32, #spirv.storage_class<Image>>) {
+  memref.copy %arg0, %arg1 : memref<4xf32, #spirv.storage_class<Image>> to memref<4xf32, #spirv.storage_class<Image>>
+  return
+}
+
+} // end module
+
+// -----
+
+// Check memref.copy is not lowered for dynamically shaped memrefs, since
+// spirv.CopyMemory requires the copied type to have a fixed size.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Shader], [SPV_KHR_storage_buffer_storage_class]>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_dynamic_shape
+// CHECK: memref.copy
+func.func @copy_dynamic_shape(%arg0: memref<?xf32, #spirv.storage_class<StorageBuffer>>, %arg1: memref<?xf32, #spirv.storage_class<StorageBuffer>>) {
+  memref.copy %arg0, %arg1 : memref<?xf32, #spirv.storage_class<StorageBuffer>> to memref<?xf32, #spirv.storage_class<StorageBuffer>>
+  return
+}
+
+} // end module
+
+// -----
+
+// Check memref.copy is not lowered for dynamically shaped memrefs under the
+// Kernel addressing model either, since the converted pointer only refers to
+// a single element and does not carry the dynamic extent.
+
+module attributes {
+  spirv.target_env = #spirv.target_env<
+    #spirv.vce<v1.0, [Kernel, Addresses], []>, #spirv.resource_limits<>>
+} {
+
+// CHECK-LABEL: func @copy_dynamic_shape_kernel
+// CHECK: memref.copy
+func.func @copy_dynamic_shape_kernel(%arg0: memref<?xf32, #spirv.storage_class<CrossWorkgroup>>, %arg1: memref<?xf32, #spirv.storage_class<CrossWorkgroup>>) {
+  memref.copy %arg0, %arg1 : memref<?xf32, #spirv.storage_class<CrossWorkgroup>> to memref<?xf32, #spirv.storage_class<CrossWorkgroup>>
+  return
+}
+
+} // end module
+
+// -----
+
 // Check Image Support.
 
 // CHECK: #[[$COLMAJMAP:.*]] = affine_map<(d0, d1) -> (d1, d0)>


        

From llvmlistbot at llvm.org  Wed Jul 22 07:04:04 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Wed, 22 Jul 2026 07:04:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Lower memref.copy to
	spirv.CopyMemory (PR #206016)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206016@github.com>
Message-ID: <6a60cdd4.b1416da1.20b4cf.e948@mx.google.com>


https://github.com/aobolensk closed https://github.com/llvm/llvm-project/pull/206016

From llvmlistbot at llvm.org  Wed Jul 22 07:04:08 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 07:04:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60cdd8.f06f4994.50b1b.bdb1@mx.google.com>


IgWod wrote:

Could you please make your email address public before I merge it: https://llvm.org/docs/DeveloperPolicy.html#email-addresses?

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 07:04:17 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Wed, 22 Jul 2026 07:04:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static multi-rank
 OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60cde1.3cce8983.247e47.8610@mx.google.com>


https://github.com/razvanlupusoru edited https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Wed Jul 22 07:05:08 2026
From: llvmlistbot at llvm.org (Ahmad Tameem)
Date: Wed, 22 Jul 2026 07:05:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Allow extra DPS inputs in
 convolution dim inference (PR #198462)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198462@github.com>
Message-ID: <6a60ce14.481afadf.1fc9f0.b470@mx.google.com>


Tameem-10xE wrote:

Hi @dimp-pl, thanks for the ping. The conflict (from #203323 adding tests to the same file) is resolved; no functional change to the fix.

While rebasing, since the testing approach in #203323 is better, I also updated my quantized-conv test to reuse its expectConvDimsEq function (this change is assisted by Claude Code).


<img width="729" height="419" alt="image" src="https://github.com/user-attachments/assets/21e60a8e-1f91-41b7-9e73-a44b080029bf" />


https://github.com/llvm/llvm-project/pull/198462

From llvmlistbot at llvm.org  Wed Jul 22 07:13:07 2026
From: llvmlistbot at llvm.org (Vito Secona)
Date: Wed, 22 Jul 2026 07:13:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60cff3.481afadf.1fc9f0.d62c@mx.google.com>


secona wrote:

I believe its now public

https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 07:22:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 07:22:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 155689a - [mlir][SPIRV] Add SPIRVToLLVM
 conversion for selection with yielding values (#210600)
Message-ID: <6a60d20e.1e1ac41d.317291.d356@mx.google.com>


Author: Vito Secona
Date: 2026-07-22T15:22:01+01:00
New Revision: 155689a48e43878d2252cd39037909e87b018103

URL: https://github.com/llvm/llvm-project/commit/155689a48e43878d2252cd39037909e87b018103
DIFF: https://github.com/llvm/llvm-project/commit/155689a48e43878d2252cd39037909e87b018103.diff

LOG: [mlir][SPIRV] Add SPIRVToLLVM conversion for selection with yielding values (#210600)

The current SPIRVToLLVM conversion for SelectionOp does not handle merge
blocks with yielding values. This change implements that by adding
arguments to the continue block in the SelectionPattern.

Closes #204714

Added: 
    

Modified: 
    mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
    mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
index c5f9305a5c226..2982061c957e0 100644
--- a/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
+++ b/mlir/lib/Conversion/SPIRVToLLVM/SPIRVToLLVM.cpp
@@ -1470,6 +1470,14 @@ class SelectionPattern : public SPIRVToLLVMConversion<spirv::SelectionOp> {
     auto position = rewriter.getInsertionPoint();
     auto *continueBlock = rewriter.splitBlock(currentBlock, position);
 
+    // Add arguments to the continue block for selections that yield values.
+    for (auto ty : op.getResultTypes()) {
+      Type dstTy = getTypeConverter()->convertType(ty);
+      if (!dstTy)
+        return rewriter.notifyMatchFailure(op, "failed to convert type");
+      continueBlock->addArgument(dstTy, loc);
+    }
+
     // Extract conditional branch information from the header block. By SPIR-V
     // dialect spec, it should contain `spirv.BranchConditional` or
     // `spirv.Switch` op. Note that `spirv.Switch op` is not supported at the

diff  --git a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
index ec47dd31cc637..d9a721be7f240 100644
--- a/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
+++ b/mlir/test/Conversion/SPIRVToLLVM/control-flow-ops-to-llvm.mlir
@@ -214,6 +214,65 @@ spirv.module Logical GLSL450 {
     %one = spirv.Constant 1 : i32
     spirv.ReturnValue %one : i32
   }
+
+  spirv.func @selection_with_yielding_value(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0 = spirv.mlir.selection -> i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]] : i32)
+      spirv.Branch ^merge(%cst1 : i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]] : i32)
+      spirv.Branch ^merge(%cst2 : i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32):
+    ^merge(%1: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]] : i32)
+      spirv.mlir.merge %1 : i32
+    }
+    // CHECK: ^bb4(%[[RES:.*]]: i32):
+    // CHECK: llvm.return %[[RES]] : i32
+    spirv.ReturnValue %0 : i32
+  }
+
+  spirv.func @selection_with_multiple_yielding_values(%cond: i1) -> i32 "None" {
+    // CHECK: llvm.cond_br %{{.*}}, ^bb1, ^bb2
+    %0:2 = spirv.mlir.selection -> i32, i32 {
+      spirv.BranchConditional %cond, ^true, ^false
+    // CHECK: ^bb1:
+    ^true:
+      // CHECK: %[[C1:.*]] = llvm.mlir.constant(1 : i32) : i32
+      %cst1 = spirv.Constant 1 : i32
+      // CHECK: %[[C3:.*]] = llvm.mlir.constant(3 : i32) : i32
+      %cst3 = spirv.Constant 3 : i32
+      // CHECK: llvm.br ^bb3(%[[C1]], %[[C3]] : i32, i32)
+      spirv.Branch ^merge(%cst1, %cst3 : i32, i32)
+    // CHECK: ^bb2:
+    ^false:
+      // CHECK: %[[C2:.*]] = llvm.mlir.constant(2 : i32) : i32
+      %cst2 = spirv.Constant 2 : i32
+      // CHECK: %[[C4:.*]] = llvm.mlir.constant(4 : i32) : i32
+      %cst4 = spirv.Constant 4 : i32
+      // CHECK: llvm.br ^bb3(%[[C2]], %[[C4]] : i32, i32)
+      spirv.Branch ^merge(%cst2, %cst4 : i32, i32)
+    // CHECK: ^bb3(%[[ARG:.*]]: i32, %[[ARG1:.*]]: i32):
+    ^merge(%1: i32, %2: i32):
+      // CHECK: llvm.br ^bb4(%[[ARG]], %[[ARG1]] : i32, i32)
+      spirv.mlir.merge %1, %2 : i32, i32
+    }
+    // CHECK: ^bb4(%[[ARG2:.*]]: i32, %[[ARG3:.*]]: i32):
+    // Makes sure both values are used.
+    // CHECK: %[[RES:.*]] = llvm.add %[[ARG2]], %[[ARG3]] : i32
+    %sum = spirv.IAdd %0#0, %0#1 : i32
+    // CHECK: llvm.return %[[RES]] : i32
+    spirv.ReturnValue %sum : i32
+  }
 }
 
 // -----


        

From llvmlistbot at llvm.org  Wed Jul 22 07:22:11 2026
From: llvmlistbot at llvm.org (Igor Wodiany)
Date: Wed, 22 Jul 2026 07:22:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Add SPIRVToLLVM conversion for
 selection with yielding values (PR #210600)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210600@github.com>
Message-ID: <6a60d213.4a342cbc.1aeb63.c7ef@mx.google.com>


https://github.com/IgWod closed https://github.com/llvm/llvm-project/pull/210600

From llvmlistbot at llvm.org  Wed Jul 22 07:35:49 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 22 Jul 2026 07:35:49 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a60d545.1905e68e.25404.6bc6@mx.google.com>


chandraghale wrote:




> Thanks for working on this. I tested the lowering with the PR compiler, with substantial assistance from Codex in identifying edge cases and constructing reproducers. In particular:
> 
> * An inclusive scan with a nonidentity incoming value omits that value. With x = 10 and four +1 iterations, Flang produces 1, 2, 3, 4 instead of 11, 12, 13, 14, as required by OpenMP’s orig-val semantics.
> * Legal loop shapes outside 1..N with unit stride corrupt the scan buffer. A 10..20 loop and a stride-two loop both produce incorrect results; the generated IR indexes a tripCount + 1 allocation using the source induction value.
> * Zero-trip loops crash at runtime. Runtime bounds, by-reference reductions, and multidimensional MLIR loop nests can also crash during compilation.
> 
> The general legal cases—nonidentity initial values, arbitrary bounds and strides, zero-trip loops, and runtime bounds—need to work before scan lowering is enabled. Cases intentionally outside this patch’s scope, such as by-reference reductions and multidimensional/collapsed loops, should instead produce clear “not yet implemented” diagnostics.
> 
> Given the number and severity of the confirmed failures, I don’t think the patch is ready for detailed review yet. Please test the supported cases more broadly and ping me once the implementation either handles or explicitly rejects them; I’ll be happy to review the revised patch.

Thanks for the review!  

Fixed
- The reduction variable's incoming value is now seeded on the first iteration
  instead of the identity, so non-identity initial values are handled correctly.
- The scan buffer is now indexed by the 1-based logical iteration number rather
  than the induction-variable value, fixing arbitrary bounds and non-unit strides.
- Zero-trip loops are now guarded so they no longer crash and correctly leave the
  reduction variable unchanged.
- Inner scan loops now use an empty ComputeIP, fixing the compile-time crash for
  runtime (non-constant) trip counts.

Diagnosed (not yet implemented)
- By-ref inscan reductions now emit a clean "not yet implemented" error instead of crashing.
- Collapsed/nested scan loops remain guarded in flang lowering.

Tests added
- Runtime: scan-nonidentity, scan-bounds (bounds + stride), scan-edge (zero-trip + runtime bounds).
- MLIR: by-ref inscan TODO test in openmp-todo.mlir; updated openmp-reduction-scan.mlir CHECKs.

https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Wed Jul 22 07:36:48 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 22 Jul 2026 07:36:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a60d580.743400b0.25573c.ad7b@mx.google.com>


================
@@ -174,33 +174,28 @@ def AtLeastRankOne : And<[
 def IsValidBlockScaledTensorType
     : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
 
+// We include unranked tensors as a supported type for all possible tosa
+// Tensors as unranked does not guarantee invalid. If unranked tensors exist
+// they should be shape propagated using Tosa's shape inference pass and verified
+// to not include any remaining unranked tensors.
 class TosaTensorOf<list<Type> allowedTypes,
+                   list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
     : TensorOf<allowedTypes,
-               [Or<[HasNo0Dimensions, IsUnrankedTensorTypePred]>,
+               [Or<[
+                    IsUnrankedTensorTypePred, 
+                    And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
+                ]>,
                 IsValidBlockScaledTensorType],
                summary>;
 
-class TosaRankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                         string summary = "tosa-conformant ranked tensor">
-    : RankedTensorOf<
-          allowedTypes,
-          !listconcat([HasNo0Dimensions, IsValidBlockScaledTensorType], preds),
-          summary>;
-
-class TosaUnrankedTensorOf<list<Type> allowedTypes, list<Pred> preds = [],
-                           string summary = "tosa-conformant unranked tensor">
-    : UnrankedTensorOf<allowedTypes,
-                       !listconcat([IsValidBlockScaledTensorType], preds),
-                       summary>;
-
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
-    : TosaRankedTensorOf<allowedTypes,
+    : TosaTensorOf<allowedTypes,
----------------
lhutton1 wrote:

Yes, that's correct, most types already accept both unranked and ranked tensors, but there are some exceptions as discussed below

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 22 07:36:48 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 22 Jul 2026 07:36:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a60d580.1728eaf3.5e665.b303@mx.google.com>


================
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
----------------
lhutton1 wrote:

That's true, but just to clarify, existing functionality does not change. I believe types that previously only accepted ranked tensors but now accept both ranked/unranked are:
- Tosa_1DInt8Tensor
- Tosa_1DInt16Or32Tensor

These types currently only seem to be used by the rescale operator for "shift" and "multiply" operands. 

I checked all usages in MLIR and could only find: https://github.com/llvm/llvm-project/blob/ee4d927dfc01f124cbb3f0302d6f273e0894c637/mlir/lib/Conversion/TosaToLinalg/TosaToLinalg.cpp#L1463, which seems okay, since the default tosa to linalg pipeline is expected to reject unranked tensors via the tosa-validation pass.

Good point regarding communication/announcement. I'm wondering if [discourse](https://discourse.llvm.org/) is a good place for this?

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 22 07:37:12 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Wed, 22 Jul 2026 07:37:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a60d598.b1416da1.20b4cf.2622@mx.google.com>


https://github.com/lhutton1 edited https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Wed Jul 22 07:37:15 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 22 Jul 2026 07:37:15 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a60d59b.510f576d.33aa7d.fac9@mx.google.com>


chandraghale wrote:

> This builds on top of #167031 to complete scan reduction lowering for OpenMP workshare loops. On top of that PR, it resets each reduction variable to its identity at the start of every input-loop iteration; otherwise the scan buffer holds a running sum and emitScanReduction prefix-sums it twice, producing incorrect results. SIMD, composite/nested loops, collapse, tiling, the linear clause and the exclusive clause are not enabled for scan reductions in this PR.

Assisted-by: Claude Opus 4.8 

https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Wed Jul 22 07:37:26 2026
From: llvmlistbot at llvm.org (Dmitrii Makarenko)
Date: Wed, 22 Jul 2026 07:37:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tensor] Preserve encoding in more
 canonicalizers (concat, reshape, pad) (PR #207241)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207241@github.com>
Message-ID: <6a60d5a6.1905e68e.25404.9ab0@mx.google.com>


Devjiu wrote:

@joker-eph gentle ping. 

https://github.com/llvm/llvm-project/pull/207241

From llvmlistbot at llvm.org  Wed Jul 22 07:37:52 2026
From: llvmlistbot at llvm.org (CHANDRA GHALE)
Date: Wed, 22 Jul 2026 07:37:52 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a60d5c0.743400b0.25573c.c28e@mx.google.com>


https://github.com/chandraghale edited https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Wed Jul 22 07:44:14 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Wed, 22 Jul 2026 07:44:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Allow extra DPS inputs in
 convolution dim inference (PR #198462)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/198462@github.com>
Message-ID: <6a60d73e.1b0626d0.1ff4c3.ce43@mx.google.com>


egebeysel wrote:

cc @phemashekar 

https://github.com/llvm/llvm-project/pull/198462

From llvmlistbot at llvm.org  Wed Jul 22 07:59:36 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 22 Jul 2026 07:59:36 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211011@github.com>
Message-ID: <6a60dad8.e5907290.33b622.6b1a@mx.google.com>


================
@@ -318,6 +319,17 @@ LogicalResult ACCRecipeMaterialization::materialize(
     Region &copyRegion = recipe.getCopyRegion();
     setLocation(copyRegion, loc);
     acc::cloneACCRegionInto(&copyRegion, block, std::next(ip), mapping, {});
+
+    // Create a temporary acc.firstprivate_save op to keep the value live-out
+    // until after other passes have determined it is truly firstprivate.
+    b.setInsertionPoint(accOp.getBody().getTerminator());
+    auto ptrLikeType = dyn_cast<acc::PointerLikeType>(alloca.getType());
+    assert(ptrLikeType && "alloca must be a pointer-like type");
+    auto xTyped = cast<TypedValue<acc::PointerLikeType>>(alloca);
+    assert(xTyped && "alloca must be a typed value");
+    auto loadOp = ptrLikeType.genLoad(b, op.getLoc(), xTyped, {});
----------------
rscottmanley wrote:

This is a good point. Ugh, this sort of suggests we need a "save" recipe.

https://github.com/llvm/llvm-project/pull/211011

From llvmlistbot at llvm.org  Wed Jul 22 08:05:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 08:05:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 930cd75 - [mlir][OpenACC] Support static
 multi-rank OpenACC array reduction accumulators (#210853)
Message-ID: <6a60dc44.af6ba3ed.1989e7.47bc@mx.google.com>


Author: Matsu
Date: 2026-07-22T08:05:35-07:00
New Revision: 930cd75aa341776565af08cc098c9270e9a16623

URL: https://github.com/llvm/llvm-project/commit/930cd75aa341776565af08cc098c9270e9a16623
DIFF: https://github.com/llvm/llvm-project/commit/930cd75aa341776565af08cc098c9270e9a16623.diff

LOG: [mlir][OpenACC] Support static multi-rank OpenACC array reduction accumulators (#210853)

Example:
```fortran
!$acc parallel loop reduction(+:a)
do i = 1, n
  a(i,:) = a(i,:) + input(i,:)
end do
```

In this code, the reduction accumulator can have rank greater than one,
but GPU lowering assumes rank one.

Fix: initialize rank-N accumulators with nested loops and delinearize
flattened indices before loading, reducing, and storing each element.

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 77fa82ba6bb4e..f0fd7ae96ad4f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -736,17 +736,28 @@ static acc::ReductionAccumulateArrayOp perThreadArrayReductionAccum(Value v) {
 static void initPerThreadArrayAccum(OpBuilder &b, Location loc, Value alloca,
                                     MemRefType baseTy,
                                     arith::AtomicRMWKind kind) {
-  assert(baseTy.getRank() == 1 && baseTy.hasStaticShape() &&
-         "per-thread array reduction accumulator must be static rank-1");
+  assert(baseTy.getRank() > 0 && baseTy.hasStaticShape() &&
+         "per-thread array reduction accumulator must be static ranked");
   Value ident = createIdentityValue(b, loc, baseTy.getElementType(), kind,
                                     /*useOnlyFiniteValue=*/true);
   Value lb = arith::ConstantIndexOp::create(b, loc, 0);
-  Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[0]);
   Value step = arith::ConstantIndexOp::create(b, loc, 1);
-  auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
-  OpBuilder::InsertionGuard g(b);
-  b.setInsertionPoint(forOp.getBody()->getTerminator());
-  memref::StoreOp::create(b, loc, ident, alloca, forOp.getInductionVar());
+  SmallVector<Value> indices;
+  auto buildLoopNest = [&](auto &&self, unsigned dim) -> void {
+    if (dim == baseTy.getRank()) {
+      memref::StoreOp::create(b, loc, ident, alloca, indices);
+      return;
+    }
+
+    Value ub = arith::ConstantIndexOp::create(b, loc, baseTy.getShape()[dim]);
+    auto forOp = scf::ForOp::create(b, loc, lb, ub, step);
+    OpBuilder::InsertionGuard g(b);
+    b.setInsertionPoint(forOp.getBody()->getTerminator());
+    indices.push_back(forOp.getInductionVar());
+    self(self, dim + 1);
+    indices.pop_back();
+  };
+  buildLoopNest(buildLoopNest, 0);
 }
 
 std::optional<int64_t>
@@ -3244,10 +3255,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
 
   Value memref = mapping.lookupOrDefault(op.getMemref());
   MemRefType memrefTy = dyn_cast<MemRefType>(memref.getType());
-  if (!memref)
+  if (!memrefTy) {
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
-  if (memrefTy.getRank() != 1)
-    (void)accSupport.emitNYI(loc, "reduction: multi-rank accumulate array");
+    return;
+  }
+  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
+    (void)accSupport.emitNYI(loc,
+                             "reduction: dynamic multi-rank accumulate array");
+    return;
+  }
 
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
@@ -3355,9 +3371,24 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     OpBuilder::InsertionGuard guard(rewriter);
     rewriter.setInsertionPoint(forOp.getBody()->getTerminator());
     Value iv = forOp.getInductionVar();
-    Value elem = memref::LoadOp::create(rewriter, loc, memref, ValueRange{iv});
-    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(),
-                         ValueRange{iv},
+    SmallVector<Value> indices{iv};
+    if (memrefTy.getRank() > 1) {
+      assert(memrefTy.hasStaticShape() &&
+             "multi-rank array reduction accumulator must be static");
+      indices.resize(memrefTy.getRank());
+      Value linearIndex = iv;
+      for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
+        Value dimSize = arith::ConstantIndexOp::create(
+            rewriter, loc, memrefTy.getDimSize(dim));
+        indices[dim] =
+            arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
+        if (dim != 0)
+          linearIndex =
+              arith::DivUIOp::create(rewriter, loc, linearIndex, dimSize);
+      }
+    }
+    Value elem = memref::LoadOp::create(rewriter, loc, memref, indices);
+    createGPUAllReduceOp(loc, elem, memref, kind, op.getParDims(), indices,
                          /*isPerThreadPrivateTarget=*/true);
   }
 

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index bfb829c901af3..ace560d869792 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -134,3 +134,115 @@ func.func @array_reduction_dynamic_par_dims(%buf: memref<?xi32>, %n: index) {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @rank_two_array_reduction
+// CHECK: %[[R2_ALLOCA:.*]] = memref.alloca() : memref<2x3xi32>
+// CHECK: scf.for %[[R2_I:.*]] =
+// CHECK:   scf.for %[[R2_J:.*]] =
+// CHECK:     memref.store %{{.*}}, %[[R2_ALLOCA]][%[[R2_I]], %[[R2_J]]] : memref<2x3xi32>
+// CHECK: scf.for %[[R2_LINEAR:.*]] =
+// CHECK: %[[R2_J_IDX:.*]] = arith.remui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_ROW:.*]] = arith.divui %[[R2_LINEAR]], %{{.*}} : index
+// CHECK: %[[R2_I_IDX:.*]] = arith.remui %[[R2_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]] : memref<2x3xi32>
+// CHECK: %[[R2_RESULT:.*]] = gpu.all_reduce add
+// CHECK: memref.store %[[R2_RESULT]], %[[R2_ALLOCA]][%[[R2_I_IDX]], %[[R2_J_IDX]]]
+
+func.func @rank_two_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x3xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x3xi32>>) {
+    %c6 = arith.constant 6 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x3xi32>>) -> memref<2x3xi32>
+    %bounds = acc.bounds extent(%c6 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x3xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_three_array_reduction
+// CHECK: %[[R3_ALLOCA:.*]] = memref.alloca() : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_I:.*]] =
+// CHECK:   scf.for %[[R3_J:.*]] =
+// CHECK:     scf.for %[[R3_K:.*]] =
+// CHECK:       memref.store %{{.*}}, %[[R3_ALLOCA]][%[[R3_I]], %[[R3_J]], %[[R3_K]]] : memref<2x2x2xi32>
+// CHECK: scf.for %[[R3_LINEAR:.*]] =
+// CHECK: %[[R3_K_IDX:.*]] = arith.remui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_PLANE:.*]] = arith.divui %[[R3_LINEAR]], %{{.*}} : index
+// CHECK: %[[R3_J_IDX:.*]] = arith.remui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_ROW:.*]] = arith.divui %[[R3_PLANE]], %{{.*}} : index
+// CHECK: %[[R3_I_IDX:.*]] = arith.remui %[[R3_ROW]], %{{.*}} : index
+// CHECK: memref.load %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]] : memref<2x2x2xi32>
+// CHECK: %[[R3_RESULT:.*]] = gpu.all_reduce add
+// CHECK: memref.store %[[R3_RESULT]], %[[R3_ALLOCA]][%[[R3_I_IDX]], %[[R3_J_IDX]], %[[R3_K_IDX]]]
+
+func.func @rank_three_array_reduction() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  %private = acc.privatize [#acc<par_dims[block_x, thread_x]>] : () -> !acc.private_type<memref<2x2x2xi32>>
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %private) : (!acc.private_type<memref<2x2x2xi32>>) {
+    %c8 = arith.constant 8 : index
+    %local = acc.private_local %arg0 {acc.par_dims = #acc<par_dims[block_x, thread_x]>} : (!acc.private_type<memref<2x2x2xi32>>) -> memref<2x2x2xi32>
+    %bounds = acc.bounds extent(%c8 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add> : memref<2x2x2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// Unsupported dynamic multi-rank accumulators must not generate invalid
+// indexing operations.
+// CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
+// CHECK: gpu.launch
+// CHECK-NOT: memref.load
+// CHECK: gpu.terminator
+func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
+    %bounds = acc.bounds extent(%ext : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @rank_two_partial_bounds_strided_layout
+// CHECK: %[[LB:.*]] = arith.constant 5 : index
+// CHECK: %[[STEP:.*]] = arith.constant 2 : index
+// CHECK: %[[EXTENT:.*]] = arith.constant 3 : index
+// CHECK: %[[ALLOCA:.*]] = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+// CHECK: %[[SPAN:.*]] = arith.muli %[[EXTENT]], %[[STEP]] : index
+// CHECK: %[[UB:.*]] = arith.addi %[[LB]], %[[SPAN]] : index
+// CHECK: scf.for %[[LINEAR:.*]] = %[[LB]] to %[[UB]] step %[[STEP]]
+// CHECK: %[[COL:.*]] = arith.remui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW_LINEAR:.*]] = arith.divui %[[LINEAR]], %{{.*}} : index
+// CHECK: %[[ROW:.*]] = arith.remui %[[ROW_LINEAR]], %{{.*}} : index
+// CHECK: memref.load %[[ALLOCA]][%[[ROW]], %[[COL]]] : memref<3x4xi32, strided<[8, 2]>>
+func.func @rank_two_partial_bounds_strided_layout() {
+  %c1 = arith.constant 1 : index
+  %c128 = arith.constant 128 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
+    %lb = arith.constant 5 : index
+    %step = arith.constant 2 : index
+    %extent = arith.constant 3 : index
+    %local = memref.alloca() : memref<3x4xi32, strided<[8, 2]>>
+    %bounds = acc.bounds lowerbound(%lb : index) extent(%extent : index)
+        stride(%step : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<3x4xi32, strided<[8, 2]>>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}


        

From llvmlistbot at llvm.org  Wed Jul 22 08:05:49 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 08:05:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support static multi-rank
 OpenACC array reduction accumulators (PR #210853)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210853@github.com>
Message-ID: <6a60dc4d.1b0626d0.1ff4c3.1df7@mx.google.com>


https://github.com/khaki3 closed https://github.com/llvm/llvm-project/pull/210853

From llvmlistbot at llvm.org  Wed Jul 22 08:08:26 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Wed, 22 Jul 2026 08:08:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211196@github.com>
Message-ID: <6a60dcea.507fc713.3702e2.b502@mx.google.com>


https://github.com/dcaballe approved this pull request.

Thanks!

https://github.com/llvm/llvm-project/pull/211196

From llvmlistbot at llvm.org  Wed Jul 22 08:57:12 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:57:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale Python module
 naming status message (PR #211243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211243@github.com>
Message-ID: <6a60e858.bc28b987.1c0bcd.0cdc@mx.google.com>


anutosh491 wrote:

Thanks for the review !

https://github.com/llvm/llvm-project/pull/211243

From llvmlistbot at llvm.org  Wed Jul 22 08:57:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 08:57:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] e205257 - [MLIR][Python] Remove stale Python
 module naming status message (#211243)
Message-ID: <6a60e866.3cce8983.247e47.d428@mx.google.com>


Author: Anutosh Bhat
Date: 2026-07-22T21:27:22+05:30
New Revision: e2052579d43e2813d2e3429a9aa092023cbf0c5f

URL: https://github.com/llvm/llvm-project/commit/e2052579d43e2813d2e3429a9aa092023cbf0c5f
DIFF: https://github.com/llvm/llvm-project/commit/e2052579d43e2813d2e3429a9aa092023cbf0c5f.diff

LOG: [MLIR][Python] Remove stale Python module naming status message (#211243)

`PYTHON_MODULE_PREFIX`, `PYTHON_MODULE_SUFFIX`, and
`PYTHON_MODULE_EXTENSION` are remnants of the previous pybind11-based
Python discovery path and are no longer populated or used by the
nanobind build.

The relevant Python SOABI is already reported through `Python3_SOABI`,
so we can remove the redundant status message, which currently prints
only empty values.

Added: 
    

Modified: 
    mlir/cmake/modules/MLIRDetectPythonEnv.cmake

Removed: 
    


################################################################################
diff  --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..ad176af6ef580 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -84,8 +84,5 @@ macro(mlir_configure_python_dev_packages)
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
     message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
-    message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
-            "suffix = '${PYTHON_MODULE_SUFFIX}', "
-            "extension = '${PYTHON_MODULE_EXTENSION}")
   endif()
 endmacro()


        

From llvmlistbot at llvm.org  Wed Jul 22 08:57:28 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:57:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale Python module
 naming status message (PR #211243)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211243@github.com>
Message-ID: <6a60e868.6c8668f2.1842d3.f66a@mx.google.com>


https://github.com/anutosh491 closed https://github.com/llvm/llvm-project/pull/211243

From llvmlistbot at llvm.org  Wed Jul 22 08:57:52 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:57:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale NumPy detection
	message (PR #211235)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211235@github.com>
Message-ID: <6a60e880.492e23e4.292f20.564d@mx.google.com>


anutosh491 wrote:

Thanks for the review. Merging !

https://github.com/llvm/llvm-project/pull/211235

From llvmlistbot at llvm.org  Wed Jul 22 08:58:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 08:58:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] e0ec539 - [MLIR][Python] Remove stale NumPy
 detection message (#211235)
Message-ID: <6a60e88e.4a342cbc.1aeb63.2790@mx.google.com>


Author: Anutosh Bhat
Date: 2026-07-22T21:28:01+05:30
New Revision: e0ec5391088426b55a5fe61137e1b76dd92e8c5e

URL: https://github.com/llvm/llvm-project/commit/e0ec5391088426b55a5fe61137e1b76dd92e8c5e
DIFF: https://github.com/llvm/llvm-project/commit/e0ec5391088426b55a5fe61137e1b76dd92e8c5e.diff

LOG: [MLIR][Python] Remove stale NumPy detection message (#211235)

NumPy stopped being requested as a CMake Python component when its
unused build-time dependency was removed in #108465 . Remove the
remaining status message, which now prints empty version and include
directory values.

Added: 
    

Modified: 
    mlir/cmake/modules/MLIRDetectPythonEnv.cmake

Removed: 
    


################################################################################
diff  --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index ad176af6ef580..b66adbcc17adf 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -52,7 +52,6 @@ macro(mlir_configure_python_dev_packages)
     endif()
     message(STATUS "Found python include dirs: ${Python3_INCLUDE_DIRS}")
     message(STATUS "Found python libraries: ${Python3_LIBRARIES}")
-    message(STATUS "Found numpy v${Python3_NumPy_VERSION}: ${Python3_NumPy_INCLUDE_DIRS}")
     message(STATUS "Python extension suffix for modules: '${Python3_SOABI}'")
     if(nanobind_DIR)
       message(STATUS "Using explicit nanobind cmake directory: ${nanobind_DIR} (-Dnanobind_DIR to change)")


        

From llvmlistbot at llvm.org  Wed Jul 22 08:58:08 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:58:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Remove stale NumPy detection
	message (PR #211235)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211235@github.com>
Message-ID: <6a60e890.507fc713.3702e2.6eaa@mx.google.com>


https://github.com/anutosh491 closed https://github.com/llvm/llvm-project/pull/211235

From llvmlistbot at llvm.org  Wed Jul 22 08:59:11 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:59:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Avoid printing an unset
 nanobind include directory (PR #211240)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211240@github.com>
Message-ID: <6a60e8cf.e624eb8d.12f676.bcbc@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/211240

>From e7a5080584e736bfaae74b233b81ee913063d499 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 17:18:18 +0530
Subject: [PATCH] [MLIR][Python] Avoid printing an unset nanobind include
 directory

---
 mlir/cmake/modules/MLIRDetectPythonEnv.cmake | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
index 068d6712393c2..4d99c8be0de7f 100644
--- a/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
+++ b/mlir/cmake/modules/MLIRDetectPythonEnv.cmake
@@ -83,7 +83,7 @@ macro(mlir_configure_python_dev_packages)
       set(nanobind_INCLUDE_DIR "${PACKAGE_DIR}")
     endif()
     find_package(nanobind 2.9 CONFIG REQUIRED)
-    message(STATUS "Found nanobind v${nanobind_VERSION}: ${nanobind_INCLUDE_DIR}")
+    message(STATUS "Found nanobind v${nanobind_VERSION}")
     message(STATUS "Python prefix = '${PYTHON_MODULE_PREFIX}', "
             "suffix = '${PYTHON_MODULE_SUFFIX}', "
             "extension = '${PYTHON_MODULE_EXTENSION}")


From llvmlistbot at llvm.org  Wed Jul 22 08:59:40 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Wed, 22 Jul 2026 08:59:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a60e8ec.c56fb102.6921f.6a69@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/211250

>From a082772bf4871087e18f0b2d78887397b082e8fc Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 18:10:06 +0530
Subject: [PATCH] [MLIR][Python] Do not install ExecutionEngine wrapper when
 disabled

---
 mlir/python/CMakeLists.txt | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..693701ef50c13 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"


From llvmlistbot at llvm.org  Wed Jul 22 09:01:20 2026
From: llvmlistbot at llvm.org (Nishant Patel)
Date: Wed, 22 Jul 2026 09:01:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][gpu] Add VectorToXeGPU and adjust
 VectorToSCF placement in XeVM pipeline (PR #210539)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210539@github.com>
Message-ID: <6a60e950.160324b0.17e275.0dfe@mx.google.com>


https://github.com/nbpatel approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/210539

From llvmlistbot at llvm.org  Wed Jul 22 09:03:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 09:03:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 22351c7 - [Flang][OpenMP] Add support for
	interop construct (#203959)
Message-ID: <6a60e9e3.3cce8983.247e47.6447@mx.google.com>


Author: Sunil Shrestha
Date: 2026-07-22T12:03:34-04:00
New Revision: 22351c74e21ff19633cc101c6d92c7c179e5a4a3

URL: https://github.com/llvm/llvm-project/commit/22351c74e21ff19633cc101c6d92c7c179e5a4a3
DIFF: https://github.com/llvm/llvm-project/commit/22351c74e21ff19633cc101c6d92c7c179e5a4a3.diff

LOG: [Flang][OpenMP] Add support for interop construct (#203959)

This adds support for the OpenMP interop construct, which enables
interoperability with foreign runtime environments (e.g., CUDA, HIP).
The init, use, and destroy action clauses are lowered from the Fortran
parse tree to MLIR interop ops and translated to the appropriate runtime
calls. The omp_lib module is extended with interop query API interfaces,
property constants, and return code constants. A semantics fix relaxes
the depend/targetsync constraint for use/destroy directives where the
interop type was established at a separate init site. Unsupported depend
clauses and prefer_type selector-list forms emit explicit diagnostics
instead of being silently dropped.

Assisted-by: Claude Opus 4.6

Added: 
    flang/test/Lower/OpenMP/Todo/interop-prefer-type-selector-list.f90
    flang/test/Lower/OpenMP/Todo/interop-prefer-type-unknown-fr.f90
    flang/test/Lower/OpenMP/interop.f90
    flang/test/Semantics/OpenMP/interop-construct-v60.f90
    flang/test/Semantics/OpenMP/interop-prefer-type.f90
    mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
    mlir/test/Target/LLVMIR/openmp-interop.mlir

Modified: 
    flang/lib/Lower/OpenMP/Clauses.cpp
    flang/lib/Lower/OpenMP/OpenMP.cpp
    flang/lib/Semantics/check-omp-structure.cpp
    llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
    mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
    mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
    mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Dialect/OpenMP/ops.mlir
    openmp/module/omp_lib.F90.var
    openmp/module/omp_lib.h.var

Removed: 
    flang/test/Lower/OpenMP/Todo/interop-construct.f90


################################################################################
diff  --git a/flang/lib/Lower/OpenMP/Clauses.cpp b/flang/lib/Lower/OpenMP/Clauses.cpp
index df0a129ff9c6f..96dcfa9ee9fbe 100644
--- a/flang/lib/Lower/OpenMP/Clauses.cpp
+++ b/flang/lib/Lower/OpenMP/Clauses.cpp
@@ -10,6 +10,7 @@
 
 #include "flang/Common/idioms.h"
 #include "flang/Evaluate/expression.h"
+#include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Parser/parse-tree.h"
 #include "flang/Semantics/expression.h"
 #include "flang/Semantics/openmp-modifiers.h"
@@ -1061,8 +1062,53 @@ Indirect make(const parser::OmpClause::Indirect &inp,
 
 Init make(const parser::OmpClause::Init &inp,
           semantics::SemanticsContext &semaCtx) {
-  // inp -> empty
-  llvm_unreachable("Empty: init");
+  // OmpInitClause has modifiers: OmpPreferType, OmpInteropType,
+  //   OmpDepinfoModifier, and an OmpObject (the interop variable).
+  auto &mods = semantics::OmpGetModifiers(inp.v);
+
+  // Extract interop types (repeatable modifier).
+  Init::InteropTypes interopTypes;
+  for (auto *interopType :
+       semantics::OmpGetRepeatableModifier<parser::OmpInteropType>(mods)) {
+    switch (interopType->v) {
+    case parser::OmpInteropType::Value::Target:
+      interopTypes.push_back(Init::InteropType::Target);
+      break;
+    case parser::OmpInteropType::Value::Targetsync:
+      interopTypes.push_back(Init::InteropType::Targetsync);
+      break;
+    }
+  }
+
+  // Extract prefer_type (optional modifier).
+  std::optional<Init::InteropPreference> interopPreference;
+  if (auto *preferType =
+          semantics::OmpGetUniqueModifier<parser::OmpPreferType>(mods)) {
+    Init::InteropPreference prefs;
+    for (auto &prefSpec : preferType->v) {
+      common::visit(
+          common::visitors{
+              [&](const parser::OmpPreferenceSpecification::
+                      ForeignRuntimeIdentifier &fri) {
+                prefs.push_back(makeExpr(fri.value(), semaCtx));
+              },
+              [&](const std::list<parser::OmpPreferenceSelector> &selectors) {
+                // TODO: Handle preference-selector lists.
+                if (!selectors.empty())
+                  TODO_NOLOC("prefer_type with preference-selector lists in "
+                             "interop init clause");
+              },
+          },
+          prefSpec.u);
+    }
+    if (!prefs.empty())
+      interopPreference = std::move(prefs);
+  }
+
+  auto &interopVar = std::get<parser::OmpObject>(inp.v.t);
+  return Init{{/*InteropPreference=*/std::move(interopPreference),
+               /*InteropTypes=*/std::move(interopTypes),
+               /*InteropVar=*/makeObject(interopVar, semaCtx)}};
 }
 
 Initializer make(const parser::OmpClause::Initializer &inp,

diff  --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index a3acc2e991307..2d5ab995559af 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -19,6 +19,7 @@
 #include "Utils.h"
 #include "flang/Common/idioms.h"
 #include "flang/Evaluate/expression.h"
+#include "flang/Evaluate/fold.h"
 #include "flang/Evaluate/tools.h"
 #include "flang/Evaluate/type.h"
 #include "flang/Lower/Bridge.h"
@@ -56,6 +57,7 @@
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallSet.h"
+#include "llvm/ADT/StringSwitch.h"
 
 using namespace Fortran::lower::omp;
 using namespace Fortran::common::openmp;
@@ -6279,8 +6281,149 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
                    const parser::OpenMPInteropConstruct &interopConstruct) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPInteropConstruct");
+  if (semaCtx.langOptions().OpenMPSimd)
+    return;
+
+  fir::FirOpBuilder &firOpBuilder = converter.getFirOpBuilder();
+  mlir::Location loc = converter.genLocation(interopConstruct.source);
+  mlir::MLIRContext *context = firOpBuilder.getContext();
+
+  List<Clause> clauses = makeClauses(interopConstruct.v.Clauses(), semaCtx);
+
+  // Process shared clauses (depend, nowait, device) that apply to all
+  // action clauses.
+  lower::StatementContext stmtCtx;
+  mlir::omp::DependClauseOps dependOps;
+  mlir::omp::NowaitClauseOps nowaitOps;
+  mlir::Value deviceValue;
+
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processDepend(symTable, stmtCtx, dependOps);
+  cp.processNowait(nowaitOps);
+
+  // Process device clause manually.
+  for (auto &clause : clauses) {
+    if (auto *deviceClause = std::get_if<clause::Device>(&clause.u)) {
+      const auto &deviceExpr = std::get<SomeExpr>(deviceClause->t);
+      deviceValue = fir::getBase(converter.genExprValue(deviceExpr, stmtCtx));
+    }
+  }
+
+  // Helper to get the address of an interop variable from an Object.
+  auto getInteropVarAddr = [&](const Object &object) -> mlir::Value {
+    const semantics::Symbol *sym = object.sym();
+    assert(sym && "interop variable must have a symbol");
+    mlir::Value addr = converter.getSymbolAddress(*sym);
+    assert(addr && "interop variable must have a valid address");
+    return addr;
+  };
+
+  // Process action clauses: init, destroy, use.
+  for (auto &clause : clauses) {
+    if (auto *initClause = std::get_if<clause::Init>(&clause.u)) {
+      auto &interopVar = std::get<clause::Init::InteropVar>(initClause->t);
+      auto &interopTypes = std::get<clause::Init::InteropTypes>(initClause->t);
+      auto &interopPref =
+          std::get<std::optional<clause::Init::InteropPreference>>(
+              initClause->t);
+
+      mlir::Value interopVarAddr = getInteropVarAddr(interopVar);
+
+      // Convert interop types to MLIR attributes. A single init clause may
+      // list both `target` and `targetsync`; keep them on one op and let the
+      // LLVM-IR translation collapse them into a single runtime init call,
+      // matching Clang.
+      llvm::SmallVector<mlir::Attribute> interopTypeAttrs;
+      for (auto itype : interopTypes) {
+        switch (itype) {
+        case clause::Init::InteropType::Target:
+          interopTypeAttrs.push_back(mlir::omp::InteropTypeAttr::get(
+              context, mlir::omp::InteropType::target));
+          break;
+        case clause::Init::InteropType::Targetsync:
+          interopTypeAttrs.push_back(mlir::omp::InteropTypeAttr::get(
+              context, mlir::omp::InteropType::targetsync));
+          break;
+        }
+      }
+      auto interopTypesAttr = mlir::ArrayAttr::get(context, interopTypeAttrs);
+
+      // Convert prefer_type to I64ArrayAttr if present.
+      mlir::ArrayAttr preferTypeAttr;
+      if (interopPref) {
+        llvm::SmallVector<int64_t> prefValues;
+        for (auto &pref : *interopPref) {
+          // Try to evaluate as a constant integer (e.g., omp_ifr_cuda).
+          if (auto constVal = evaluate::ToInt64(pref)) {
+            prefValues.push_back(*constVal);
+          } else if (auto str =
+                         evaluate::GetScalarConstantValue<evaluate::Ascii>(
+                             pref)) {
+            // Map standard OpenMP foreign-runtime identifier strings to
+            // their well-known integer values (OpenMP 5.1, Table 22.2).
+            auto frId = llvm::StringSwitch<std::optional<int64_t>>(
+                            llvm::StringRef(*str).lower())
+                            .Case("cuda", 1)
+                            .Case("cuda_driver", 2)
+                            .Case("opencl", 3)
+                            .Case("sycl", 4)
+                            .Case("hip", 5)
+                            .Case("level_zero", 6)
+                            .Case("hsa", 7)
+                            .Default(std::nullopt);
+            if (frId)
+              prefValues.push_back(*frId);
+            else
+              TODO(loc, "unknown foreign-runtime identifier in "
+                        "prefer_type");
+          } else {
+            TODO(loc, "non-constant prefer_type value in interop init");
+          }
+        }
+        if (!prefValues.empty())
+          preferTypeAttr = firOpBuilder.getI64ArrayAttr(prefValues);
+      }
+
+      // Emit a single InteropInitOp carrying all interop-types from this init
+      // clause. The translation to LLVM IR collapses them into one runtime
+      // call, matching Clang (one init clause -> one __tgt_interop_init).
+      mlir::omp::InteropInitOp::create(
+          firOpBuilder, loc, interopVarAddr, interopTypesAttr, preferTypeAttr,
+          deviceValue,
+          dependOps.dependKinds.empty()
+              ? nullptr
+              : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+          dependOps.dependVars,
+          /*depend_iterated_kinds=*/nullptr,
+          /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+
+    } else if (auto *destroyClause = std::get_if<clause::Destroy>(&clause.u)) {
+      assert(destroyClause->v &&
+             "destroy clause must have an interop variable");
+      if (destroyClause->v) {
+        mlir::Value interopVarAddr = getInteropVarAddr(*destroyClause->v);
+        mlir::omp::InteropDestroyOp::create(
+            firOpBuilder, loc, interopVarAddr, deviceValue,
+            dependOps.dependKinds.empty()
+                ? nullptr
+                : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+            dependOps.dependVars,
+            /*depend_iterated_kinds=*/nullptr,
+            /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+      }
+
+    } else if (auto *useClause = std::get_if<clause::Use>(&clause.u)) {
+      mlir::Value interopVarAddr = getInteropVarAddr(useClause->v);
+      mlir::omp::InteropUseOp::create(
+          firOpBuilder, loc, interopVarAddr, deviceValue,
+          dependOps.dependKinds.empty()
+              ? nullptr
+              : firOpBuilder.getArrayAttr(dependOps.dependKinds),
+          dependOps.dependVars,
+          /*depend_iterated_kinds=*/nullptr,
+          /*depend_iterated=*/mlir::ValueRange{}, nowaitOps.nowait);
+    }
+  }
 }
 
 static void

diff  --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index bf38dcb823c5c..4e29a1c387ec2 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -5975,6 +5975,7 @@ void OmpStructureChecker::Enter(const parser::OmpClause::ThreadLimit &x) {
 
 void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
   bool isDependClauseOccurred{false};
+  bool hasInitClause{false};
   int targetCount{0}, targetSyncCount{0};
   std::set<const Symbol *> objectSymbolList;
   const auto &clauseList{std::get<std::optional<parser::OmpClauseList>>(x.v.t)};
@@ -5982,6 +5983,7 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     common::visit(
         common::visitors{
             [&](const parser::OmpClause::Init &initClause) {
+              hasInitClause = true;
               if (OmpVerifyModifiers(initClause.v, llvm::omp::OMPC_init,
                       GetContext().directiveSource, context_)) {
 
@@ -6004,6 +6006,38 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
                       "The '%s' is not allowed on INTEROP construct"_err_en_US,
                       desc.name.str());
                 }
+                // A prefer_type foreign-runtime-identifier must be a
+                // constant expression of integer OpenMP type or a base
+                // language string literal. This is enforced for the flat form
+                // (prefer_type(fr-id, ...)); the OpenMP 6.0
+                // brace form (prefer_type({fr(...), attr(...)}, ...)) is not
+                // yet supported in lowering and is diagnosed there, so its
+                // operands are not validated here.
+                if (auto *preferType{
+                        OmpGetUniqueModifier<parser::OmpPreferType>(
+                            modifiers)}) {
+                  for (const auto &prefSpec : preferType->v) {
+                    const auto *fri{
+                        std::get_if<parser::OmpPreferenceSpecification::
+                                ForeignRuntimeIdentifier>(&prefSpec.u)};
+                    if (!fri) {
+                      continue;
+                    }
+                    if (const auto *expr{GetExpr(context_, fri->value())}) {
+                      std::optional<evaluate::DynamicType> type{
+                          expr->GetType()};
+                      bool isIntOrChar{type &&
+                          (type->category() ==
+                                  evaluate::TypeCategory::Integer ||
+                              type->category() ==
+                                  evaluate::TypeCategory::Character)};
+                      if (!evaluate::IsConstantExpr(*expr) || !isIntOrChar) {
+                        context_.Say(fri->value().source,
+                            "The foreign-runtime-identifier in a `prefer_type` modifier must be a constant expression of integer OpenMP type or a base language string literal"_err_en_US);
+                      }
+                    }
+                  }
+                }
               }
               const auto *interopVar{parser::Unwrap<parser::OmpObject>(
                   std::get<parser::OmpObject>(initClause.v.t))};
@@ -6058,7 +6092,11 @@ void OmpStructureChecker::Enter(const parser::OpenMPInteropConstruct &x) {
     context_.Say(GetContext().directiveSource,
         "Each interop-type may be specified at most once."_err_en_US);
   }
-  if (isDependClauseOccurred && !targetSyncCount) {
+  // Only enforce the depend/targetsync constraint when an init clause is
+  // present on this directive. For use/destroy-only directives, the interop
+  // object's type was established at its init site, which may be in a
+  // 
diff erent statement — we cannot verify it statically here.
+  if (isDependClauseOccurred && hasInitClause && !targetSyncCount) {
     context_.Say(GetContext().directiveSource,
         "A DEPEND clause can only appear on the directive if the interop-type includes TARGETSYNC"_err_en_US);
   }

diff  --git a/flang/test/Lower/OpenMP/Todo/interop-construct.f90 b/flang/test/Lower/OpenMP/Todo/interop-construct.f90
deleted file mode 100644
index 4f69667863990..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/interop-construct.f90
+++ /dev/null
@@ -1,9 +0,0 @@
-! REQUIRES: openmp_runtime
-! RUN: %not_todo_cmd %flang_fc1 -emit-llvm %openmp_flags -fopenmp-version=52 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPInteropConstruct
-program interop_test
-  use omp_lib
-  integer(omp_interop_kind) :: obj
-  !$omp interop init(targetsync,target: obj)
-end program interop_test

diff  --git a/flang/test/Lower/OpenMP/Todo/interop-prefer-type-selector-list.f90 b/flang/test/Lower/OpenMP/Todo/interop-prefer-type-selector-list.f90
new file mode 100644
index 0000000000000..be0f63eb64bc2
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/interop-prefer-type-selector-list.f90
@@ -0,0 +1,11 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=60 -o - %s 2>&1 | FileCheck %s
+
+! A prefer_type preference-selector list (the OpenMP 6.0 brace form) is valid
+! syntax but not yet lowered.
+
+! CHECK: not yet implemented: prefer_type with preference-selector lists in interop init clause
+
+subroutine interop_prefer_type_selector_list(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type({fr("cuda")}), target: obj)
+end subroutine

diff  --git a/flang/test/Lower/OpenMP/Todo/interop-prefer-type-unknown-fr.f90 b/flang/test/Lower/OpenMP/Todo/interop-prefer-type-unknown-fr.f90
new file mode 100644
index 0000000000000..848fff122ddd8
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/interop-prefer-type-unknown-fr.f90
@@ -0,0 +1,11 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=60 -o - %s 2>&1 | FileCheck %s
+
+! A prefer_type foreign-runtime identifier string that is not one of the
+! standard OpenMP names cannot be mapped to a runtime id.
+
+! CHECK: not yet implemented: unknown foreign-runtime identifier in prefer_type
+
+subroutine interop_prefer_type_unknown_fr(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type("no_such_runtime"), target: obj)
+end subroutine

diff  --git a/flang/test/Lower/OpenMP/interop.f90 b/flang/test/Lower/OpenMP/interop.f90
new file mode 100644
index 0000000000000..b20332bef6893
--- /dev/null
+++ b/flang/test/Lower/OpenMP/interop.f90
@@ -0,0 +1,175 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=60 %s -o - | FileCheck %s
+
+!===============================================================================
+! Interop Init — target
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init(
+!CHECK-SAME:    %[[ARG:.*]]: !fir.ref<i64> {fir.bindc_name = "obj"})
+!CHECK:         %[[DECL:.*]]:2 = hlfir.declare %[[ARG]]
+!CHECK:         omp.interop.init %[[DECL]]#0 : !fir.ref<i64> interop_types([#omp<interop_type(target)>])
+subroutine test_interop_init(obj)
+  integer(8) :: obj
+  !$omp interop init(target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — targetsync
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_targetsync(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>])
+subroutine test_interop_init_targetsync(obj)
+  integer(8) :: obj
+  !$omp interop init(targetsync: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — targetsync, target (both)
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_both(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>])
+subroutine test_interop_init_both(obj)
+  integer(8) :: obj
+  !$omp interop init(targetsync, target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — untyped (OpenMP 6.0: interop-type modifier is optional)
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_untyped(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([])
+subroutine test_interop_init_untyped(obj)
+  integer(8) :: obj
+  !$omp interop init(obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — nowait
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_init_nowait(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) nowait
+subroutine test_interop_init_nowait(obj)
+  integer(8) :: obj
+  !$omp interop init(target: obj) nowait
+end subroutine
+
+!===============================================================================
+! Interop Init — device clause
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_device(
+!CHECK:         %[[DEV:.*]] = fir.load %{{.*}} : !fir.ref<i32>
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) device(%[[DEV]] : i32)
+subroutine test_interop_device(obj, dev)
+  integer(8) :: obj
+  integer :: dev
+  !$omp interop device(dev) init(target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — prefer_type with string FR identifiers
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_prefer_str(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>]) prefer_type([1, 6])
+subroutine test_interop_prefer_str(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type("cuda", "level_zero"), targetsync: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — prefer_type with string FR identifier (hip)
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_prefer_hip(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(target)>]) prefer_type([5])
+subroutine test_interop_prefer_hip(obj)
+  integer(8) :: obj
+  !$omp interop init(prefer_type("hip"), target: obj)
+end subroutine
+
+!===============================================================================
+! Interop Init — prefer_type with integer constant FR identifiers
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_prefer_int(
+!CHECK:         omp.interop.init %{{.*}} : !fir.ref<i64> interop_types([#omp<interop_type(targetsync)>]) prefer_type([4])
+subroutine test_interop_prefer_int(obj)
+  integer(8) :: obj
+  integer, parameter :: omp_ifr_sycl = 4
+  !$omp interop init(prefer_type(omp_ifr_sycl), targetsync: obj)
+end subroutine
+
+!===============================================================================
+! Interop Use
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_use(
+!CHECK:         omp.interop.use %{{.*}} : !fir.ref<i64>
+subroutine test_interop_use(obj)
+  integer(8) :: obj
+  !$omp interop use(obj)
+end subroutine
+
+!===============================================================================
+! Interop Use — nowait
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_use_nowait(
+!CHECK:         omp.interop.use %{{.*}} : !fir.ref<i64> nowait
+subroutine test_interop_use_nowait(obj)
+  integer(8) :: obj
+  !$omp interop use(obj) nowait
+end subroutine
+
+!===============================================================================
+! Interop Use — device clause
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_use_device(
+!CHECK:         %[[DEV:.*]] = fir.load %{{.*}} : !fir.ref<i32>
+!CHECK:         omp.interop.use %{{.*}} : !fir.ref<i64> device(%[[DEV]] : i32)
+subroutine test_interop_use_device(obj, dev)
+  integer(8) :: obj
+  integer :: dev
+  !$omp interop use(obj) device(dev)
+end subroutine
+
+!===============================================================================
+! Interop Destroy
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_destroy(
+!CHECK:         omp.interop.destroy %{{.*}} : !fir.ref<i64>
+subroutine test_interop_destroy(obj)
+  integer(8) :: obj
+  !$omp interop destroy(obj)
+end subroutine
+
+!===============================================================================
+! Interop Destroy — nowait
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_destroy_nowait(
+!CHECK:         omp.interop.destroy %{{.*}} : !fir.ref<i64> nowait
+subroutine test_interop_destroy_nowait(obj)
+  integer(8) :: obj
+  !$omp interop destroy(obj) nowait
+end subroutine
+
+!===============================================================================
+! Interop Destroy — device clause
+!===============================================================================
+
+!CHECK-LABEL: func.func @_QPtest_interop_destroy_device(
+!CHECK:         %[[DEV:.*]] = fir.load %{{.*}} : !fir.ref<i32>
+!CHECK:         omp.interop.destroy %{{.*}} : !fir.ref<i64> device(%[[DEV]] : i32)
+subroutine test_interop_destroy_device(obj, dev)
+  integer(8) :: obj
+  integer :: dev
+  !$omp interop destroy(obj) device(dev)
+end subroutine

diff  --git a/flang/test/Semantics/OpenMP/interop-construct-v60.f90 b/flang/test/Semantics/OpenMP/interop-construct-v60.f90
new file mode 100644
index 0000000000000..78a226f8a6b1f
--- /dev/null
+++ b/flang/test/Semantics/OpenMP/interop-construct-v60.f90
@@ -0,0 +1,24 @@
+! REQUIRES: openmp_runtime
+
+! RUN: %python %S/../test_errors.py %s %flang %openmp_flags -fopenmp-version=60
+! OpenMP Version 6.0
+! 14.1 Interop construct
+! In OpenMP 6.0 the interop-type modifier on an INIT clause is optional, so
+! INIT(obj) alone is accepted. A DEPEND clause, however, requires the
+! interop-type to include TARGETSYNC, which an untyped INIT does not provide,
+! so INIT(obj) combined with DEPEND is diagnosed.
+
+SUBROUTINE test_interop_untyped()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !$OMP INTEROP INIT(obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_interop_untyped
+
+SUBROUTINE test_interop_untyped_depend()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !ERROR: A DEPEND clause can only appear on the directive if the interop-type includes TARGETSYNC
+  !$OMP INTEROP INIT(obj) DEPEND(INOUT: obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_interop_untyped_depend

diff  --git a/flang/test/Semantics/OpenMP/interop-prefer-type.f90 b/flang/test/Semantics/OpenMP/interop-prefer-type.f90
new file mode 100644
index 0000000000000..3af9ae2b3f00c
--- /dev/null
+++ b/flang/test/Semantics/OpenMP/interop-prefer-type.f90
@@ -0,0 +1,30 @@
+! REQUIRES: openmp_runtime
+
+! RUN: %python %S/../test_errors.py %s %flang %openmp_flags -fopenmp-version=60
+! OpenMP Version 6.0
+! 14.1 Interop construct
+! The foreign-runtime-identifier in a `prefer_type` modifier must be a constant
+! expression of integer OpenMP type or a base language string literal.
+
+SUBROUTINE test_prefer_type_nonconstant()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  INTEGER :: n
+  !ERROR: The foreign-runtime-identifier in a `prefer_type` modifier must be a constant expression of integer OpenMP type or a base language string literal
+  !$OMP INTEROP INIT(PREFER_TYPE(n), TARGET: obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_prefer_type_nonconstant
+
+SUBROUTINE test_prefer_type_constant_int()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !$OMP INTEROP INIT(PREFER_TYPE(omp_ifr_cuda), TARGET: obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_prefer_type_constant_int
+
+SUBROUTINE test_prefer_type_string()
+  USE omp_lib
+  INTEGER(OMP_INTEROP_KIND) :: obj
+  !$OMP INTEROP INIT(PREFER_TYPE("cuda"), TARGET: obj)
+  PRINT *, 'pass'
+END SUBROUTINE test_prefer_type_string

diff  --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index e19ed110adf1e..fb1cebf28b95b 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -8227,6 +8227,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropInit(
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   Constant *InteropTypeVal = ConstantInt::get(Int32, (int)InteropType);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
@@ -8255,6 +8257,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropDestroy(
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
     PointerType *PointerTypeVar = PointerType::getUnqual(M.getContext());
@@ -8283,6 +8287,8 @@ CallInst *OpenMPIRBuilder::createOMPInteropUse(const LocationDescription &Loc,
   Value *ThreadId = getOrCreateThreadID(Ident);
   if (Device == nullptr)
     Device = Constant::getAllOnesValue(Int32);
+  else if (Device->getType() != Int32)
+    Device = Builder.CreateIntCast(Device, Int32, /*isSigned=*/true);
   if (NumDependences == nullptr) {
     NumDependences = ConstantInt::get(Int32, 0);
     PointerType *PointerTypeVar = PointerType::getUnqual(M.getContext());

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
index c9e6764e7d634..f6bc8f11ccccb 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPAttrDefs.td
@@ -79,6 +79,16 @@ def TaskDependArrayAttr
   let constBuilderCall = ?;
 }
 
+//===----------------------------------------------------------------------===//
+// InteropTypeArrayAttr
+//===----------------------------------------------------------------------===//
+
+def InteropTypeArrayAttr
+    : TypedArrayAttrBase<InteropTypeAttr,
+                         InteropTypeAttr.summary # " array"> {
+  let constBuilderCall = ?;
+}
+
 //===----------------------------------------------------------------------===//
 // VersionAttr
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
index deda86b26db61..e373d1a52a3fc 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
@@ -377,4 +377,22 @@ def FallbackModifierAttr : OpenMP_EnumAttr<FallbackModifier,
   let assemblyFormat = "`(` $value `)`";
 }
 
+//===----------------------------------------------------------------------===//
+// interop_type for interop construct
+//===----------------------------------------------------------------------===//
+
+def InteropTypeTarget : I32EnumAttrCase<"target", 0>;
+def InteropTypeTargetSync : I32EnumAttrCase<"targetsync", 1>;
+
+def InteropType : OpenMP_I32EnumAttr<
+    "InteropType",
+    "interop type", [
+      InteropTypeTarget,
+      InteropTypeTargetSync
+    ]>;
+
+def InteropTypeAttr : OpenMP_EnumAttr<InteropType, "interop_type"> {
+  let assemblyFormat = "`(` $value `)`";
+}
+
 #endif // OPENMP_ENUMS

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 98242a66e411f..0a5736f46b088 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2655,4 +2655,104 @@ def GroupprivateOp : OpenMP_Op<"groupprivate",
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// 14.1 interop construct
+//===----------------------------------------------------------------------===//
+
+def InteropInitOp
+    : OpenMP_Op<"interop.init", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop init operation";
+  let description = [{
+    The interop init operation initializes an interop object for
+    foreign runtime interoperability. The `interop_type` attribute specifies
+    whether the interop object provides access to the target execution
+    environment (`target`), a synchronization object (`targetsync`), or both.
+
+    The optional `prefer_type` attribute provides a list of preferred foreign
+    runtime types as integer identifiers.
+
+    The optional `device` operand specifies the device number.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      InteropTypeArrayAttr:$interop_types,
+      OptionalAttr<I64ArrayAttr>:$prefer_type,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    `interop_types` `(` $interop_types `)`
+    (`prefer_type` `(` $prefer_type^ `)`)?
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+}
+
+def InteropUseOp
+    : OpenMP_Op<"interop.use", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop use operation";
+  let description = [{
+    The interop use operation indicates use of an interop object. This is
+    used in combination with depend clauses to synchronize tasks with
+    foreign runtime operations.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+}
+
+def InteropDestroyOp
+    : OpenMP_Op<"interop.destroy", traits = [AttrSizedOperandSegments],
+                clauses = [OpenMP_DependClause, OpenMP_NowaitClause]> {
+  let summary = "interop destroy operation";
+  let description = [{
+    The interop destroy operation releases an interop object and frees
+    any resources associated with it.
+  }] # clausesDescription;
+
+  let arguments = !con((ins
+      OpenMP_PointerLikeType:$interop_var,
+      Optional<AnyInteger>:$device),
+      clausesArgs);
+
+  let assemblyFormat = [{
+    $interop_var `:` type($interop_var)
+    (`device` `(` $device^ `:` type($device) `)`)?
+    oilist(
+      `depend` `(`
+      custom<DependVarList>($depend_vars, type($depend_vars), $depend_kinds,
+                            $depend_iterated, type($depend_iterated),
+                            $depend_iterated_kinds) `)`
+      | `nowait` $nowait
+    )
+    attr-dict
+  }];
+}
+
 #endif // OPENMP_OPS

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index f2fefa1f5a53f..85963c402bc87 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -4364,6 +4364,104 @@ convertOmpTaskgroupOp(omp::TaskgroupOp tgOp, llvm::IRBuilderBase &builder,
   return success();
 }
 
+static LogicalResult
+convertOmpInteropInitOp(omp::InteropInitOp initOp, llvm::IRBuilderBase &builder,
+                        LLVM::ModuleTranslation &moduleTranslation) {
+  if (!initOp.getDependVars().empty() || initOp.getDependKinds() ||
+      !initOp.getDependIterated().empty() || initOp.getDependIteratedKinds())
+    return initOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropInitOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(initOp.getInteropVar());
+  llvm::Value *device = initOp.getDevice()
+                            ? moduleTranslation.lookupValue(initOp.getDevice())
+                            : nullptr;
+
+  // TODO: Handle depend clauses when supported.
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = initOp.getNowait();
+
+  // A single `init` clause may list both `target` and `targetsync`, but the
+  // runtime init call takes a single interop-type. Collapse the set to one
+  // value, matching Clang: if `target` is present use Target, otherwise
+  // TargetSync. The offload runtime object model supports only one type per
+  // object; representing both would require a runtime change.
+  bool hasTarget = false, hasTargetSync = false;
+  for (mlir::Attribute typeAttr : initOp.getInteropTypes()) {
+    switch (cast<omp::InteropTypeAttr>(typeAttr).getValue()) {
+    case omp::InteropType::target:
+      hasTarget = true;
+      break;
+    case omp::InteropType::targetsync:
+      hasTargetSync = true;
+      break;
+    }
+  }
+  llvm::omp::OMPInteropType interopType =
+      (!hasTarget && hasTargetSync) ? llvm::omp::OMPInteropType::TargetSync
+                                    : llvm::omp::OMPInteropType::Target;
+  ompBuilder->createOMPInteropInit(builder, interopVar, interopType, device,
+                                   numDeps, depArray, hasNowait);
+  return success();
+}
+
+static LogicalResult
+convertOmpInteropDestroyOp(omp::InteropDestroyOp destroyOp,
+                           llvm::IRBuilderBase &builder,
+                           LLVM::ModuleTranslation &moduleTranslation) {
+  if (!destroyOp.getDependVars().empty() || destroyOp.getDependKinds() ||
+      !destroyOp.getDependIterated().empty() ||
+      destroyOp.getDependIteratedKinds())
+    return destroyOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropDestroyOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(destroyOp.getInteropVar());
+  llvm::Value *device =
+      destroyOp.getDevice()
+          ? moduleTranslation.lookupValue(destroyOp.getDevice())
+          : nullptr;
+
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = destroyOp.getNowait();
+
+  ompBuilder->createOMPInteropDestroy(builder, interopVar, device, numDeps,
+                                      depArray, hasNowait);
+  return success();
+}
+
+static LogicalResult
+convertOmpInteropUseOp(omp::InteropUseOp useOp, llvm::IRBuilderBase &builder,
+                       LLVM::ModuleTranslation &moduleTranslation) {
+  if (!useOp.getDependVars().empty() || useOp.getDependKinds() ||
+      !useOp.getDependIterated().empty() || useOp.getDependIteratedKinds())
+    return useOp.emitError()
+           << "not yet implemented: Unhandled clause depend in "
+           << omp::InteropUseOp::getOperationName() << " operation";
+
+  llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
+  llvm::Value *interopVar =
+      moduleTranslation.lookupValue(useOp.getInteropVar());
+  llvm::Value *device = useOp.getDevice()
+                            ? moduleTranslation.lookupValue(useOp.getDevice())
+                            : nullptr;
+
+  llvm::Value *numDeps = llvm::ConstantInt::get(builder.getInt32Ty(), 0);
+  llvm::Value *depArray = llvm::ConstantPointerNull::get(builder.getPtrTy());
+  bool hasNowait = useOp.getNowait();
+
+  ompBuilder->createOMPInteropUse(builder, interopVar, device, numDeps,
+                                  depArray, hasNowait);
+  return success();
+}
+
 static LogicalResult
 convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
                      LLVM::ModuleTranslation &moduleTranslation) {
@@ -9802,6 +9900,15 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::TaskwaitOp op) {
             return convertOmpTaskwaitOp(op, builder, moduleTranslation);
           })
+          .Case([&](omp::InteropInitOp op) {
+            return convertOmpInteropInitOp(op, builder, moduleTranslation);
+          })
+          .Case([&](omp::InteropDestroyOp op) {
+            return convertOmpInteropDestroyOp(op, builder, moduleTranslation);
+          })
+          .Case([&](omp::InteropUseOp op) {
+            return convertOmpInteropUseOp(op, builder, moduleTranslation);
+          })
           .Case<omp::YieldOp, omp::TerminatorOp, omp::DeclareMapperOp,
                 omp::DeclareMapperInfoOp, omp::DeclareReductionOp,
                 omp::CriticalDeclareOp>([](auto op) {

diff  --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index 9a1874379d2fb..d08ca5664efe4 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -4234,3 +4234,57 @@ func.func @omp_target_map_iterated(%lb : index, %ub : index, %step : index,
   }
   return
 }
+
+// -----
+
+// CHECK-LABEL: func.func @omp_interop_init
+func.func @omp_interop_init(%obj : !llvm.ptr, %device : i32) -> () {
+  // CHECK: omp.interop.init %{{.*}} : !llvm.ptr interop_types([#omp<interop_type(target)>])
+  omp.interop.init %obj : !llvm.ptr interop_types([#omp<interop_type(target)>])
+
+  // CHECK: omp.interop.init %{{.*}} : !llvm.ptr interop_types([#omp<interop_type(targetsync)>])
+  omp.interop.init %obj : !llvm.ptr interop_types([#omp<interop_type(targetsync)>])
+
+  // CHECK: omp.interop.init %{{.*}} : !llvm.ptr interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>]) prefer_type([1, 6]) device(%{{.*}} : i32) nowait
+  omp.interop.init %obj : !llvm.ptr interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>]) prefer_type([1, 6]) device(%device : i32) nowait
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @omp_interop_use
+func.func @omp_interop_use(%obj : !llvm.ptr, %device : i32) -> () {
+  // CHECK: omp.interop.use %{{.*}} : !llvm.ptr
+  omp.interop.use %obj : !llvm.ptr
+
+  // CHECK: omp.interop.use %{{.*}} : !llvm.ptr device(%{{.*}} : i32) nowait
+  omp.interop.use %obj : !llvm.ptr device(%device : i32) nowait
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @omp_interop_destroy
+func.func @omp_interop_destroy(%obj : !llvm.ptr, %device : i32) -> () {
+  // CHECK: omp.interop.destroy %{{.*}} : !llvm.ptr
+  omp.interop.destroy %obj : !llvm.ptr
+
+  // CHECK: omp.interop.destroy %{{.*}} : !llvm.ptr device(%{{.*}} : i32) nowait
+  omp.interop.destroy %obj : !llvm.ptr device(%device : i32) nowait
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @omp_interop_depend
+func.func @omp_interop_depend(%obj : !llvm.ptr, %dep : !llvm.ptr) -> () {
+  // CHECK: omp.interop.init %{{.*}} : !llvm.ptr interop_types([#omp<interop_type(targetsync)>]) depend(taskdependinout -> %{{.*}} : !llvm.ptr)
+  omp.interop.init %obj : !llvm.ptr interop_types([#omp<interop_type(targetsync)>]) depend(taskdependinout -> %dep : !llvm.ptr)
+
+  // CHECK: omp.interop.use %{{.*}} : !llvm.ptr depend(taskdependin -> %{{.*}} : !llvm.ptr)
+  omp.interop.use %obj : !llvm.ptr depend(taskdependin -> %dep : !llvm.ptr)
+
+  // CHECK: omp.interop.destroy %{{.*}} : !llvm.ptr depend(taskdependout -> %{{.*}} : !llvm.ptr)
+  omp.interop.destroy %obj : !llvm.ptr depend(taskdependout -> %dep : !llvm.ptr)
+  return
+}

diff  --git a/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir b/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
new file mode 100644
index 0000000000000..d362714add856
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-interop-depend.mlir
@@ -0,0 +1,28 @@
+// RUN: rm -rf %t && split-file %s %t
+// RUN: not mlir-translate --mlir-to-llvmir %t/init.mlir 2>&1 | FileCheck %s --check-prefix=INIT
+// RUN: not mlir-translate --mlir-to-llvmir %t/use.mlir 2>&1 | FileCheck %s --check-prefix=USE
+// RUN: not mlir-translate --mlir-to-llvmir %t/destroy.mlir 2>&1 | FileCheck %s --check-prefix=DESTROY
+
+// The depend clause on interop operations is not yet handled by the
+// translation to LLVM IR; each action op must emit a clean diagnostic.
+
+//--- init.mlir
+// INIT: error: not yet implemented: Unhandled clause depend in omp.interop.init operation
+llvm.func @test_interop_init_depend(%interop: !llvm.ptr, %dep: !llvm.ptr) {
+  omp.interop.init %interop : !llvm.ptr interop_types([#omp<interop_type(targetsync)>]) depend(taskdependinout -> %dep : !llvm.ptr)
+  llvm.return
+}
+
+//--- use.mlir
+// USE: error: not yet implemented: Unhandled clause depend in omp.interop.use operation
+llvm.func @test_interop_use_depend(%interop: !llvm.ptr, %dep: !llvm.ptr) {
+  omp.interop.use %interop : !llvm.ptr depend(taskdependinout -> %dep : !llvm.ptr)
+  llvm.return
+}
+
+//--- destroy.mlir
+// DESTROY: error: not yet implemented: Unhandled clause depend in omp.interop.destroy operation
+llvm.func @test_interop_destroy_depend(%interop: !llvm.ptr, %dep: !llvm.ptr) {
+  omp.interop.destroy %interop : !llvm.ptr depend(taskdependinout -> %dep : !llvm.ptr)
+  llvm.return
+}

diff  --git a/mlir/test/Target/LLVMIR/openmp-interop.mlir b/mlir/test/Target/LLVMIR/openmp-interop.mlir
new file mode 100644
index 0000000000000..85cadd241e23d
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-interop.mlir
@@ -0,0 +1,95 @@
+// RUN: mlir-translate --mlir-to-llvmir %s | FileCheck %s
+
+// CHECK-LABEL: define void @test_interop_init(
+// CHECK-SAME:    ptr %[[OBJ:.*]])
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %[[GTID]], ptr %[[OBJ]], i32 1, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_init(%arg0: !llvm.ptr) {
+  %0 = llvm.mlir.constant(1 : i64) : i64
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_targetsync(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 2, i32 -1, i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_targetsync(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(targetsync)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_both(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 -1, i32 0, ptr null, i32 0)
+// CHECK-NOT:     call void @__tgt_interop_init
+llvm.func @test_interop_init_both(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(targetsync)>, #omp<interop_type(target)>])
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_use(
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_use(ptr @{{.*}}, i32 %[[GTID]], ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_use(%arg0: !llvm.ptr) {
+  omp.interop.use %arg0 : !llvm.ptr
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy(
+// CHECK:         %[[GTID:.*]] = call i32 @__kmpc_global_thread_num(ptr @{{.*}})
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %[[GTID]], ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 0)
+// CHECK:         ret void
+llvm.func @test_interop_destroy(%arg0: !llvm.ptr) {
+  omp.interop.destroy %arg0 : !llvm.ptr
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_nowait(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 -1, i32 0, ptr null, i32 1)
+llvm.func @test_interop_init_nowait(%arg0: !llvm.ptr) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) nowait
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy_nowait(
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 -1, i32 0, ptr null, i32 1)
+llvm.func @test_interop_destroy_nowait(%arg0: !llvm.ptr) {
+  omp.interop.destroy %arg0 : !llvm.ptr nowait
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_device(
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 %{{.*}}, i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_device(%arg0: !llvm.ptr, %arg1: i32) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) device(%arg1 : i32)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_init_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_init(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 1, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_init_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.init %arg0 : !llvm.ptr interop_types([#omp<interop_type(target)>]) device(%arg1 : i64)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_destroy_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_destroy(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_destroy_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.destroy %arg0 : !llvm.ptr device(%arg1 : i64)
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_interop_use_device_i64(
+// CHECK:         %[[DEVICE:.*]] = trunc i64 %{{.*}} to i32
+// CHECK:         call void @__tgt_interop_use(ptr @{{.*}}, i32 %{{.*}}, ptr %{{.*}}, i32 %[[DEVICE]], i32 0, ptr null, i32 0)
+llvm.func @test_interop_use_device_i64(%arg0: !llvm.ptr, %arg1: i64) {
+  omp.interop.use %arg0 : !llvm.ptr device(%arg1 : i64)
+  llvm.return
+}
+
+// CHECK: declare i32 @__kmpc_global_thread_num(ptr)
+// CHECK: declare void @__tgt_interop_init(ptr, i32, ptr, i32, i32, i32, ptr, i32)
+// CHECK: declare void @__tgt_interop_use(ptr, i32, ptr, i32, i32, ptr, i32)
+// CHECK: declare void @__tgt_interop_destroy(ptr, i32, ptr, i32, i32, ptr, i32)

diff  --git a/openmp/module/omp_lib.F90.var b/openmp/module/omp_lib.F90.var
index 2db2a33e90127..a41b2faad9d3d 100644
--- a/openmp/module/omp_lib.F90.var
+++ b/openmp/module/omp_lib.F90.var
@@ -40,6 +40,8 @@
         integer, parameter, public :: omp_alloctrait_val_kind = c_intptr_t
         integer, parameter, public :: omp_interop_kind = c_intptr_t
         integer, parameter, public :: omp_interop_fr_kind = omp_integer_kind
+        integer, parameter, public :: omp_interop_property_kind = omp_integer_kind
+        integer, parameter, public :: omp_interop_rc_kind = omp_integer_kind
 
         type omp_alloctrait
           integer(kind=omp_alloctrait_key_kind) key
@@ -84,6 +86,8 @@
         public :: omp_alloctrait_val_kind
         public :: omp_interop_kind
         public :: omp_interop_fr_kind
+        public :: omp_interop_property_kind
+        public :: omp_interop_rc_kind
         public :: omp_alloctrait
         public :: omp_pause_resource_kind
         public :: omp_depend_kind
@@ -218,6 +222,28 @@
 
         integer (kind=omp_interop_kind), parameter, public :: omp_interop_none = 0
 
+!       interop property constants
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_id = -1
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_fr_name = -2
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_vendor = -3
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_vendor_name = -4
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device_num = -5
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_platform = -6
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device = -7
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_device_context = -8
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_targetsync = -9
+        integer (kind=omp_interop_property_kind), parameter, public :: omp_ipr_first = -9
+
+!       interop return code constants
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_no_value = 1
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_success = 0
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_empty = -1
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_out_of_range = -2
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_int = -3
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_ptr = -4
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_type_str = -5
+        integer (kind=omp_interop_rc_kind), parameter, public :: omp_irc_other = -6
+
         interface
 
 !         ***
@@ -1061,6 +1087,67 @@
             logical (kind=omp_logical_kind) kmp_get_cancellation_status
           end function kmp_get_cancellation_status
 
+!         ***
+!         *** interop query routines
+!         ***
+
+          function omp_get_num_interop_properties(interop) bind(c)
+            use omp_lib_kinds
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_integer_kind) omp_get_num_interop_properties
+          end function omp_get_num_interop_properties
+
+          function omp_get_interop_int(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_intptr_t, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            integer (kind=c_intptr_t) omp_get_interop_int
+          end function omp_get_interop_int
+
+          function omp_get_interop_ptr(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            type(c_ptr) omp_get_interop_ptr
+          end function omp_get_interop_ptr
+
+          function omp_get_interop_str(interop, property_id, err) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            integer (kind=c_int) :: err
+            type(c_ptr) omp_get_interop_str
+          end function omp_get_interop_str
+
+          function omp_get_interop_name(interop, property_id) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            type(c_ptr) omp_get_interop_name
+          end function omp_get_interop_name
+
+          function omp_get_interop_type_desc(interop, property_id) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_property_kind), value :: property_id
+            type(c_ptr) omp_get_interop_type_desc
+          end function omp_get_interop_type_desc
+
+          function omp_get_interop_rc_desc(interop, ret_code) bind(c)
+            use omp_lib_kinds
+            use, intrinsic :: iso_c_binding, only: c_ptr
+            integer (kind=omp_interop_kind), value :: interop
+            integer (kind=omp_interop_rc_kind), value :: ret_code
+            type(c_ptr) omp_get_interop_rc_desc
+          end function omp_get_interop_rc_desc
+
         end interface
 
         ! make the above routine definitions public
@@ -1182,5 +1269,12 @@
         public :: kmp_set_warnings_on
         public :: kmp_set_warnings_off
         public :: kmp_get_cancellation_status
+        public :: omp_get_num_interop_properties
+        public :: omp_get_interop_int
+        public :: omp_get_interop_ptr
+        public :: omp_get_interop_str
+        public :: omp_get_interop_name
+        public :: omp_get_interop_type_desc
+        public :: omp_get_interop_rc_desc
 
       end module omp_lib

diff  --git a/openmp/module/omp_lib.h.var b/openmp/module/omp_lib.h.var
index e515c9434f120..5e1996e31fe0a 100644
--- a/openmp/module/omp_lib.h.var
+++ b/openmp/module/omp_lib.h.var
@@ -56,6 +56,10 @@
       parameter(omp_interop_kind=int_ptr_kind())
       integer omp_interop_fr_kind
       parameter(omp_interop_fr_kind=omp_integer_kind)
+      integer omp_interop_property_kind
+      parameter(omp_interop_property_kind=omp_integer_kind)
+      integer omp_interop_rc_kind
+      parameter(omp_interop_rc_kind=omp_integer_kind)
 
       integer(kind=omp_integer_kind)openmp_version
       parameter(openmp_version=@LIBOMP_OMP_YEAR_MONTH@)
@@ -296,6 +300,46 @@
       integer(kind=omp_interop_kind)omp_interop_none
       parameter(omp_interop_none=0)
 
+!     interop property constants
+      integer(kind=omp_interop_property_kind)omp_ipr_fr_id
+      parameter(omp_ipr_fr_id=-1)
+      integer(kind=omp_interop_property_kind)omp_ipr_fr_name
+      parameter(omp_ipr_fr_name=-2)
+      integer(kind=omp_interop_property_kind)omp_ipr_vendor
+      parameter(omp_ipr_vendor=-3)
+      integer(kind=omp_interop_property_kind)omp_ipr_vendor_name
+      parameter(omp_ipr_vendor_name=-4)
+      integer(kind=omp_interop_property_kind)omp_ipr_device_num
+      parameter(omp_ipr_device_num=-5)
+      integer(kind=omp_interop_property_kind)omp_ipr_platform
+      parameter(omp_ipr_platform=-6)
+      integer(kind=omp_interop_property_kind)omp_ipr_device
+      parameter(omp_ipr_device=-7)
+      integer(kind=omp_interop_property_kind)omp_ipr_device_context
+      parameter(omp_ipr_device_context=-8)
+      integer(kind=omp_interop_property_kind)omp_ipr_targetsync
+      parameter(omp_ipr_targetsync=-9)
+      integer(kind=omp_interop_property_kind)omp_ipr_first
+      parameter(omp_ipr_first=-9)
+
+!     interop return code constants
+      integer(kind=omp_interop_rc_kind)omp_irc_no_value
+      parameter(omp_irc_no_value=1)
+      integer(kind=omp_interop_rc_kind)omp_irc_success
+      parameter(omp_irc_success=0)
+      integer(kind=omp_interop_rc_kind)omp_irc_empty
+      parameter(omp_irc_empty=-1)
+      integer(kind=omp_interop_rc_kind)omp_irc_out_of_range
+      parameter(omp_irc_out_of_range=-2)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_int
+      parameter(omp_irc_type_int=-3)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_ptr
+      parameter(omp_irc_type_ptr=-4)
+      integer(kind=omp_interop_rc_kind)omp_irc_type_str
+      parameter(omp_irc_type_str=-5)
+      integer(kind=omp_interop_rc_kind)omp_irc_other
+      parameter(omp_irc_other=-6)
+
       interface
 
 !       ***
@@ -1129,6 +1173,67 @@
 
         subroutine kmp_set_warnings_off() bind(c)
         end subroutine kmp_set_warnings_off
+
+!       ***
+!       *** interop query routines
+!       ***
+
+        function omp_get_num_interop_properties(interop) bind(c)
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_integer_kind) omp_get_num_interop_properties
+        end function omp_get_num_interop_properties
+
+        function omp_get_interop_int(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_intptr_t, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          integer (kind=c_intptr_t) omp_get_interop_int
+        end function omp_get_interop_int
+
+        function omp_get_interop_ptr(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          type(c_ptr) omp_get_interop_ptr
+        end function omp_get_interop_ptr
+
+        function omp_get_interop_str(interop, property_id, err) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr, c_int
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          integer (kind=c_int) :: err
+          type(c_ptr) omp_get_interop_str
+        end function omp_get_interop_str
+
+        function omp_get_interop_name(interop, property_id) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          type(c_ptr) omp_get_interop_name
+        end function omp_get_interop_name
+
+        function omp_get_interop_type_desc(interop, property_id) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_property_kind), value :: property_id
+          type(c_ptr) omp_get_interop_type_desc
+        end function omp_get_interop_type_desc
+
+        function omp_get_interop_rc_desc(interop, ret_code) bind(c)
+          use, intrinsic :: iso_c_binding, only: c_ptr
+          import
+          integer (kind=omp_interop_kind), value :: interop
+          integer (kind=omp_interop_rc_kind), value :: ret_code
+          type(c_ptr) omp_get_interop_rc_desc
+        end function omp_get_interop_rc_desc
       end interface
 
 #ifdef __INTEL_COMPILER


        

From llvmlistbot at llvm.org  Wed Jul 22 09:03:44 2026
From: llvmlistbot at llvm.org (Caroline Newcombe)
Date: Wed, 22 Jul 2026 09:03:44 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Add
 support for interop construct (PR #203959)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203959@github.com>
Message-ID: <6a60e9e0.d2cb4adc.23815e.3221@mx.google.com>


https://github.com/cenewcombe closed https://github.com/llvm/llvm-project/pull/203959

From llvmlistbot at llvm.org  Wed Jul 22 09:04:44 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Wed, 22 Jul 2026 09:04:44 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [OpenACC] add acc::FirstprivateSaveOp
	(PR #211011)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211011@github.com>
Message-ID: <6a60ea1c.7270d2fd.1ec500.4d02@mx.google.com>


================
@@ -318,6 +319,17 @@ LogicalResult ACCRecipeMaterialization::materialize(
     Region &copyRegion = recipe.getCopyRegion();
     setLocation(copyRegion, loc);
     acc::cloneACCRegionInto(&copyRegion, block, std::next(ip), mapping, {});
+
+    // Create a temporary acc.firstprivate_save op to keep the value live-out
+    // until after other passes have determined it is truly firstprivate.
+    b.setInsertionPoint(accOp.getBody().getTerminator());
+    auto ptrLikeType = dyn_cast<acc::PointerLikeType>(alloca.getType());
+    assert(ptrLikeType && "alloca must be a pointer-like type");
+    auto xTyped = cast<TypedValue<acc::PointerLikeType>>(alloca);
+    assert(xTyped && "alloca must be a typed value");
+    auto loadOp = ptrLikeType.genLoad(b, op.getLoc(), xTyped, {});
----------------
rscottmanley wrote:

Maybe this is okay, I was looking at the spec. This would end up temporary, but still, not elegant.

```
The appearance of a Fortran optional argument arg in the following situations may result in undefined behavior if PRESENT(arg) is .false. when the associated construct is executed:
• as a var in private, firstprivate, and reduction clauses;
```

https://github.com/llvm/llvm-project/pull/211011

From llvmlistbot at llvm.org  Wed Jul 22 09:51:02 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 22 Jul 2026 09:51:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a60f4f6.a1accbb4.1ace1c.9dd8@mx.google.com>


================
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
----------------
makslevental wrote:

TIL `unset`

https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Wed Jul 22 09:52:49 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 22 Jul 2026 09:52:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a60f561.f3151ff8.f584a.910f@mx.google.com>


================
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
----------------
makslevental wrote:

since this look weird (exclude the `execution_engine` sources but not the target) can you add a comment about the specific thing (basically what you wrote in the PR description but shorter).

https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Wed Jul 22 09:52:53 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Wed, 22 Jul 2026 09:52:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a60f565.c4d3d80a.16dae5.c5ca@mx.google.com>


https://github.com/makslevental approved this pull request.


https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Wed Jul 22 10:59:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 10:59:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support dynamic multi-rank
 array reductions (PR #211336)
Message-ID: <llvm.org/llvm/llvm-project/pull/211336@github.com>


https://github.com/khaki3 created https://github.com/llvm/llvm-project/pull/211336

Example:
```fortran
integer :: r(:,:)
!$acc parallel loop reduction(+:r)
```

In this code, the reduction accumulator has dynamic dimensions and rank greater than one.

Fix: use runtime dimension sizes to delinearize flattened array-reduction indices.

>From 7e8a8a0c197a2a15a4c7d7997bedbca889b84769 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Wed, 22 Jul 2026 10:09:13 -0700
Subject: [PATCH] [mlir][OpenACC] Support dynamic multi-rank array reductions

---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 16 +++++------
 .../acc-cg-to-gpu-reduction-array.mlir        | 27 ++++++++++++-------
 2 files changed, 24 insertions(+), 19 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index f0fd7ae96ad4f..92f9cc90dc73f 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -3259,12 +3259,6 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     (void)accSupport.emitNYI(loc, "reduction: non-MemRefTy accumulate array");
     return;
   }
-  if (memrefTy.getRank() > 1 && !memrefTy.hasStaticShape()) {
-    (void)accSupport.emitNYI(loc,
-                             "reduction: dynamic multi-rank accumulate array");
-    return;
-  }
-
   FailureOr<arith::AtomicRMWKind> kindOr = getReductionKind(
       op.getReductionOperator(), memrefTy.getElementType(), loc);
   if (failed(kindOr))
@@ -3373,13 +3367,15 @@ void ACCCGToGPULowering::processAccumulateArrayOp(
     Value iv = forOp.getInductionVar();
     SmallVector<Value> indices{iv};
     if (memrefTy.getRank() > 1) {
-      assert(memrefTy.hasStaticShape() &&
-             "multi-rank array reduction accumulator must be static");
       indices.resize(memrefTy.getRank());
       Value linearIndex = iv;
       for (int64_t dim = memrefTy.getRank() - 1; dim >= 0; --dim) {
-        Value dimSize = arith::ConstantIndexOp::create(
-            rewriter, loc, memrefTy.getDimSize(dim));
+        Value dimSize =
+            memrefTy.isDynamicDim(dim)
+                ? memref::DimOp::create(rewriter, loc, memref, dim).getResult()
+                : arith::ConstantIndexOp::create(rewriter, loc,
+                                                 memrefTy.getDimSize(dim))
+                      .getResult();
         indices[dim] =
             arith::RemUIOp::create(rewriter, loc, linearIndex, dimSize);
         if (dim != 0)
diff --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index ace560d869792..422e172e1ec5b 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -196,20 +196,29 @@ func.func @rank_three_array_reduction() {
   return
 }
 
-// Unsupported dynamic multi-rank accumulators must not generate invalid
-// indexing operations.
 // CHECK-LABEL: func.func @dynamic_rank_two_array_reduction
-// CHECK: gpu.launch
-// CHECK-NOT: memref.load
-// CHECK: gpu.terminator
-func.func @dynamic_rank_two_array_reduction(%local: memref<?x?xi32>, %extent: index) {
+// CHECK: scf.for %[[LINEAR:.*]] =
+// CHECK: %[[DIM1:.*]] = memref.dim %{{.*}}, %{{.*}} : memref<2x?xi32>
+// CHECK: %[[IDX1:.*]] = arith.remui %[[LINEAR]], %[[DIM1]] : index
+// CHECK: %[[ROW:.*]] = arith.divui %[[LINEAR]], %[[DIM1]] : index
+// CHECK: %[[C2:.*]] = arith.constant 2 : index
+// CHECK: %[[IDX0:.*]] = arith.remui %[[ROW]], %[[C2]] : index
+// CHECK: %[[ELEM:.*]] = memref.load %{{.*}}[%[[IDX0]], %[[IDX1]]] : memref<2x?xi32>
+// CHECK: %[[RESULT:.*]] = gpu.all_reduce add %[[ELEM]]
+// CHECK: memref.store %[[RESULT]], %{{.*}}[%[[IDX0]], %[[IDX1]]] : memref<2x?xi32>
+func.func @dynamic_rank_two_array_reduction(
+    %local: memref<2x?xi32>, %n: index) {
   %c1 = arith.constant 1 : index
   %c128 = arith.constant 128 : index
   %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
   %tx = acc.par_width %c128 {par_dim = #acc.par_dim<thread_x>}
-  acc.compute_region launch(%kbx = %bx, %ktx = %tx) ins(%arg0 = %local, %ext = %extent) : (memref<?x?xi32>, index) {
-    %bounds = acc.bounds extent(%ext : index)
-    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<?x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx)
+      ins(%arg0 = %local, %arg1 = %n)
+      : (memref<2x?xi32>, index) {
+    %c2 = arith.constant 2 : index
+    %extent = arith.muli %c2, %arg1 : index
+    %bounds = acc.bounds extent(%extent : index)
+    acc.reduction_accumulate_array %arg0 bounds(%bounds) <add> : memref<2x?xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
     acc.yield
   } {origin = "acc.parallel"}
   return


From llvmlistbot at llvm.org  Wed Jul 22 11:52:24 2026
From: llvmlistbot at llvm.org (Jeremy Kun)
Date: Wed, 22 Jul 2026 11:52:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add compound assignment ops (PR
	#210233)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210233@github.com>
Message-ID: <6a611168.4a342cbc.1aeb63.392e@mx.google.com>


https://github.com/j2kun approved this pull request.


https://github.com/llvm/llvm-project/pull/210233

From llvmlistbot at llvm.org  Wed Jul 22 11:52:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 11:52:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8ae7803 - [mlir][emitc] Add compound
	assignment ops (#210233)
Message-ID: <6a61117d.743400b0.25573c.d2e4@mx.google.com>


Author: Jianjian Guan
Date: 2026-07-22T11:52:41-07:00
New Revision: 8ae7803bede98cca73e6fd6a8fabbf95da9d0b46

URL: https://github.com/llvm/llvm-project/commit/8ae7803bede98cca73e6fd6a8fabbf95da9d0b46
DIFF: https://github.com/llvm/llvm-project/commit/8ae7803bede98cca73e6fd6a8fabbf95da9d0b46.diff

LOG: [mlir][emitc] Add compound assignment ops (#210233)

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
    mlir/lib/Dialect/EmitC/IR/EmitC.cpp
    mlir/lib/Target/Cpp/TranslateToCpp.cpp
    mlir/test/Dialect/EmitC/invalid_ops.mlir
    mlir/test/Dialect/EmitC/ops.mlir
    mlir/test/Target/Cpp/common-cpp.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
index b6604cf45139f..49412d1dfb01c 100644
--- a/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
+++ b/mlir/include/mlir/Dialect/EmitC/IR/EmitC.td
@@ -88,6 +88,20 @@ class EmitC_IncDecOp<string mnemonic, string summaryStr>
   }];
 }
 
+// Base class for compound assignment operations.
+class EmitC_CompoundAssignOp<string mnemonic, string summaryStr>
+    : EmitC_Op<mnemonic, []> {
+  let summary = summaryStr;
+  let arguments = (ins
+    Res<EmitC_LValueType, "",
+        [MemRead<DefaultResource, 0, FullEffect>,
+         MemWrite<DefaultResource, 0, FullEffect>]>:$var,
+    EmitCType:$value);
+  let results = (outs);
+  let hasVerifier = 1;
+  let assemblyFormat = "$value `:` type($value) `to` $var `:` type($var) attr-dict";
+}
+
 // Types only used in binary arithmetic operations.
 def IntegerIndexOrOpaqueType : Type<CPred<"emitc::isIntegerIndexOrOpaqueType($_self)">,
 "integer, index or opaque type supported by EmitC">;
@@ -1591,6 +1605,76 @@ def EmitC_AssignOp : EmitC_Op<"assign", []> {
   let assemblyFormat = "$value `:` type($value) `to` $var `:` type($var) attr-dict";
 }
 
+def EmitC_AddAssignOp
+    : EmitC_CompoundAssignOp<"add_assign", "Addition assignment operation"> {
+  let description = [{
+    The `emitc.add_assign` operation applies the C/C++ `+=` operator to an
+    lvalue.
+
+    Example:
+
+    ```mlir
+    emitc.add_assign %value : i32 to %var : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_SubAssignOp
+    : EmitC_CompoundAssignOp<"sub_assign", "Subtraction assignment operation"> {
+  let description = [{
+    The `emitc.sub_assign` operation applies the C/C++ `-=` operator to an
+    lvalue.
+
+    Example:
+
+    ```mlir
+    emitc.sub_assign %value : i32 to %var : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_MulAssignOp
+    : EmitC_CompoundAssignOp<"mul_assign", "Multiplication assignment operation"> {
+  let description = [{
+    The `emitc.mul_assign` operation applies the C/C++ `*=` operator to an
+    lvalue.
+
+    Example:
+
+    ```mlir
+    emitc.mul_assign %value : i32 to %var : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_DivAssignOp
+    : EmitC_CompoundAssignOp<"div_assign", "Division assignment operation"> {
+  let description = [{
+    The `emitc.div_assign` operation applies the C/C++ `/=` operator to an
+    lvalue.
+
+    Example:
+
+    ```mlir
+    emitc.div_assign %value : i32 to %var : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
+def EmitC_RemAssignOp
+    : EmitC_CompoundAssignOp<"rem_assign", "Remainder assignment operation"> {
+  let description = [{
+    The `emitc.rem_assign` operation applies the C/C++ `%=` operator to an
+    lvalue.
+
+    Example:
+
+    ```mlir
+    emitc.rem_assign %value : i32 to %var : !emitc.lvalue<i32>
+    ```
+  }];
+}
+
 def EmitC_YieldOp : EmitC_Op<"yield",
       [Pure, Terminator, ParentOneOf<["DoOp", "ExpressionOp", "ForOp", "IfOp", "SwitchOp"]>]> {
   let summary = "Block termination operation";

diff  --git a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
index 6e68a3d0c5a41..820c4ad578a9e 100644
--- a/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
+++ b/mlir/lib/Dialect/EmitC/IR/EmitC.cpp
@@ -263,27 +263,38 @@ LogicalResult AddOp::verify() {
 }
 
 //===----------------------------------------------------------------------===//
-// AssignOp
+// Assignment operations
 //===----------------------------------------------------------------------===//
 
-/// The assign op requires that the assigned value's type matches the
-/// assigned-to variable type.
-LogicalResult emitc::AssignOp::verify() {
-  TypedValue<emitc::LValueType> variable = getVar();
+template <typename AssignmentOp>
+static LogicalResult verifyAssignmentOp(AssignmentOp op) {
+  TypedValue<emitc::LValueType> variable = op.getVar();
 
   if (!variable.getDefiningOp())
-    return emitOpError() << "cannot assign to block argument";
+    return op.emitOpError() << "cannot assign to block argument";
 
-  Type valueType = getValue().getType();
+  Type valueType = op.getValue().getType();
   Type variableType = variable.getType().getValueType();
   if (variableType != valueType)
-    return emitOpError() << "requires value's type (" << valueType
-                         << ") to match variable's type (" << variableType
-                         << ")\n  variable: " << variable
-                         << "\n  value: " << getValue() << "\n";
+    return op.emitOpError() << "requires value's type (" << valueType
+                            << ") to match variable's type (" << variableType
+                            << ")\n  variable: " << variable
+                            << "\n  value: " << op.getValue() << "\n";
   return success();
 }
 
+LogicalResult emitc::AssignOp::verify() { return verifyAssignmentOp(*this); }
+
+LogicalResult emitc::AddAssignOp::verify() { return verifyAssignmentOp(*this); }
+
+LogicalResult emitc::SubAssignOp::verify() { return verifyAssignmentOp(*this); }
+
+LogicalResult emitc::MulAssignOp::verify() { return verifyAssignmentOp(*this); }
+
+LogicalResult emitc::DivAssignOp::verify() { return verifyAssignmentOp(*this); }
+
+LogicalResult emitc::RemAssignOp::verify() { return verifyAssignmentOp(*this); }
+
 //===----------------------------------------------------------------------===//
 // CastOp
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Target/Cpp/TranslateToCpp.cpp b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
index 5ae51bd580d31..d5815fafcd6a7 100644
--- a/mlir/lib/Target/Cpp/TranslateToCpp.cpp
+++ b/mlir/lib/Target/Cpp/TranslateToCpp.cpp
@@ -608,6 +608,42 @@ static LogicalResult printOperation(CppEmitter &emitter,
   return emitter.emitOperand(assignOp.getValue());
 }
 
+static LogicalResult
+printCompoundAssignmentOperation(CppEmitter &emitter, Operation *operation,
+                                 StringRef compoundAssignmentOperator) {
+  if (failed(emitter.emitOperand(operation->getOperand(0))))
+    return failure();
+
+  emitter.ostream() << " " << compoundAssignmentOperator << " ";
+
+  return emitter.emitOperand(operation->getOperand(1));
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::AddAssignOp addAssignOp) {
+  return printCompoundAssignmentOperation(emitter, addAssignOp, "+=");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::SubAssignOp subAssignOp) {
+  return printCompoundAssignmentOperation(emitter, subAssignOp, "-=");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::MulAssignOp mulAssignOp) {
+  return printCompoundAssignmentOperation(emitter, mulAssignOp, "*=");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::DivAssignOp divAssignOp) {
+  return printCompoundAssignmentOperation(emitter, divAssignOp, "/=");
+}
+
+static LogicalResult printOperation(CppEmitter &emitter,
+                                    emitc::RemAssignOp remAssignOp) {
+  return printCompoundAssignmentOperation(emitter, remAssignOp, "%=");
+}
+
 static LogicalResult printOperation(CppEmitter &emitter, emitc::LoadOp loadOp) {
   if (failed(emitter.emitAssignPrefix(*loadOp)))
     return failure();
@@ -1927,25 +1963,26 @@ LogicalResult CppEmitter::emitOperation(Operation &op, bool trailingSemicolon) {
           .Case<cf::BranchOp, cf::CondBranchOp>(
               [&](auto op) { return printOperation(*this, op); })
           // EmitC ops.
-          .Case<emitc::AddressOfOp, emitc::AddOp, emitc::AssignOp,
-                emitc::BitwiseAndOp, emitc::BitwiseLeftShiftOp,
+          .Case<emitc::AddAssignOp, emitc::AddressOfOp, emitc::AddOp,
+                emitc::AssignOp, emitc::BitwiseAndOp, emitc::BitwiseLeftShiftOp,
                 emitc::BitwiseNotOp, emitc::BitwiseOrOp,
                 emitc::BitwiseRightShiftOp, emitc::BitwiseXorOp, emitc::CallOp,
                 emitc::CallOpaqueOp, emitc::CastOp, emitc::ClassOp,
                 emitc::CmpOp, emitc::ConditionalOp, emitc::ConstantOp,
-                emitc::DeclareFuncOp, emitc::DereferenceOp, emitc::DivOp,
-                emitc::DoOp, emitc::ExpressionOp, emitc::FieldOp, emitc::FileOp,
-                emitc::ForOp, emitc::FuncOp, emitc::GetFieldOp,
+                emitc::DeclareFuncOp, emitc::DereferenceOp, emitc::DivAssignOp,
+                emitc::DivOp, emitc::DoOp, emitc::ExpressionOp, emitc::FieldOp,
+                emitc::FileOp, emitc::ForOp, emitc::FuncOp, emitc::GetFieldOp,
                 emitc::GetGlobalOp, emitc::GlobalOp, emitc::IfOp,
                 emitc::IncludeOp, emitc::LiteralOp, emitc::LoadOp,
                 emitc::LogicalAndOp, emitc::LogicalNotOp, emitc::LogicalOrOp,
                 emitc::MemberCallOpaqueOp, emitc::MemberOfPtrOp,
-                emitc::MemberOp, emitc::MulOp, emitc::PostDecrementOp,
-                emitc::PostIncrementOp, emitc::PreDecrementOp,
-                emitc::PreIncrementOp, emitc::RemOp, emitc::ReturnOp,
-                emitc::SubscriptOp, emitc::SubOp, emitc::SwitchOp,
-                emitc::UnaryMinusOp, emitc::UnaryPlusOp, emitc::VariableOp,
-                emitc::VerbatimOp>(
+                emitc::MemberOp, emitc::MulAssignOp, emitc::MulOp,
+                emitc::PostDecrementOp, emitc::PostIncrementOp,
+                emitc::PreDecrementOp, emitc::PreIncrementOp,
+                emitc::RemAssignOp, emitc::RemOp, emitc::ReturnOp,
+                emitc::SubAssignOp, emitc::SubscriptOp, emitc::SubOp,
+                emitc::SwitchOp, emitc::UnaryMinusOp, emitc::UnaryPlusOp,
+                emitc::VariableOp, emitc::VerbatimOp>(
 
               [&](auto op) { return printOperation(*this, op); })
           // Func ops.

diff  --git a/mlir/test/Dialect/EmitC/invalid_ops.mlir b/mlir/test/Dialect/EmitC/invalid_ops.mlir
index 3048160d1590c..889138a22ede3 100644
--- a/mlir/test/Dialect/EmitC/invalid_ops.mlir
+++ b/mlir/test/Dialect/EmitC/invalid_ops.mlir
@@ -997,3 +997,20 @@ func.func @post_decrement_unmatch_type(%arg0: !emitc.lvalue<i32>) {
   %1 = "emitc.post_decrement"(%arg0) : (!emitc.lvalue<i32>) -> i8
   return
 }
+
+// -----
+
+func.func @add_assign_to_block_argument(%arg0: i32, %arg1: !emitc.lvalue<i32>) {
+  // expected-error @+1 {{'emitc.add_assign' op cannot assign to block argument}}
+  emitc.add_assign %arg0 : i32 to %arg1 : !emitc.lvalue<i32>
+  return
+}
+
+// -----
+
+func.func @sub_assign_type_mismatch(%arg0: f32) {
+  %v = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<i32>
+  // expected-error @+1 {{'emitc.sub_assign' op requires value's type ('f32') to match variable's type ('i32')}}
+  emitc.sub_assign %arg0 : f32 to %v : !emitc.lvalue<i32>
+  return
+}

diff  --git a/mlir/test/Dialect/EmitC/ops.mlir b/mlir/test/Dialect/EmitC/ops.mlir
index e5c4af67c4896..3261b2fa467da 100644
--- a/mlir/test/Dialect/EmitC/ops.mlir
+++ b/mlir/test/Dialect/EmitC/ops.mlir
@@ -187,6 +187,23 @@ func.func @test_assign(%arg1: f32) {
   return
 }
 
+// CHECK-LABEL: func.func @compound_assign
+// CHECK: emitc.add_assign
+// CHECK: emitc.sub_assign
+// CHECK: emitc.mul_assign
+// CHECK: emitc.div_assign
+// CHECK: emitc.rem_assign
+func.func @compound_assign(%arg0: i32, %arg1: !emitc.opaque<"number">) {
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  %opaque = "emitc.variable"() <{value = #emitc.opaque<"">}> : () -> !emitc.lvalue<!emitc.opaque<"number">>
+  emitc.add_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  emitc.sub_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  emitc.mul_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  emitc.div_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  emitc.rem_assign %arg1 : !emitc.opaque<"number"> to %opaque : !emitc.lvalue<!emitc.opaque<"number">>
+  return
+}
+
 func.func @test_expression(%arg0: i32, %arg1: i32, %arg2: i32, %arg3: f32, %arg4: f32) -> i32 {
   %c7 = "emitc.constant"() {value = 7 : i32} : () -> i32
   %q = emitc.expression %arg1, %c7 : (i32, i32) -> i32 {

diff  --git a/mlir/test/Target/Cpp/common-cpp.mlir b/mlir/test/Target/Cpp/common-cpp.mlir
index 1bffd59d4d44c..041321db46290 100644
--- a/mlir/test/Target/Cpp/common-cpp.mlir
+++ b/mlir/test/Target/Cpp/common-cpp.mlir
@@ -122,6 +122,24 @@ func.func @inc_dec() {
   return
 }
 
+// CHECK-LABEL: void compound_assign(
+// CHECK-SAME: int32_t [[ARG0:[^ ]*]]) {
+func.func @compound_assign(%arg0: i32) {
+  // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;
+  %v = "emitc.variable"() <{value = 0 : i32}> : () -> !emitc.lvalue<i32>
+  // CHECK-NEXT: [[V1]] += [[ARG0]];
+  emitc.add_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: [[V1]] -= [[ARG0]];
+  emitc.sub_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: [[V1]] *= [[ARG0]];
+  emitc.mul_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: [[V1]] /= [[ARG0]];
+  emitc.div_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  // CHECK-NEXT: [[V1]] %= [[ARG0]];
+  emitc.rem_assign %arg0 : i32 to %v : !emitc.lvalue<i32>
+  return
+}
+
 // CHECK-LABEL: int32_t inc_dec_expression() {
 func.func @inc_dec_expression() -> i32 {
   // CHECK-NEXT: int32_t [[V1:[^ ]*]] = 0;


        

From llvmlistbot at llvm.org  Wed Jul 22 11:52:48 2026
From: llvmlistbot at llvm.org (Jeremy Kun)
Date: Wed, 22 Jul 2026 11:52:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][emitc] Add compound assignment ops (PR
	#210233)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210233@github.com>
Message-ID: <6a611180.1ba19891.1d5373.5f64@mx.google.com>


https://github.com/j2kun closed https://github.com/llvm/llvm-project/pull/210233

From llvmlistbot at llvm.org  Wed Jul 22 13:46:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 13:46:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a612c36.92ea0e2a.2ed880.8e05@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210566

>From 73c99ea962f697de30027864648b5e9bd536a66d Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir][SPIRV] Diagnose missing return value

---
 mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp        |  6 ++++-
 .../test-legalize-erased-op-with-uses.mlir    | 22 ++++++++++++++++++-
 2 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index 99bc3913c56c6..6bd91e7dfd8df 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -1226,7 +1226,11 @@ LogicalResult spirv::FuncOp::verifyBody() {
                    "returns 1 value but enclosing function requires ")
                << fnType.getNumResults() << " results";
 
-      auto retOperandType = retOp.getValue().getType();
+      Value retOperand = retOp.getValue();
+      if (!retOperand)
+        return retOp.emitOpError("return value is missing");
+
+      auto retOperandType = retOperand.getType();
       auto fnResultType = fnType.getResult(0);
       if (retOperandType != fnResultType)
         return retOp.emitOpError(" return value's type (")
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..67a17ecc36ab1 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,5 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics -mlir-print-ir-after-all
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +9,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Wed Jul 22 13:54:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 13:54:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a612e11.d681cab9.33af1e.752f@mx.google.com>


qyingwu wrote:

@matthias-springer @joker-eph 
Thanks! I dug further and I think you were right that the fix belongs in the verifier.
The crash happens while reporting the unresolved materialization diagnostic, but the actual bad dereference is in the SPIR-V verifier:
  ```text
  legalizeUnresolvedMaterialization
    -> Diagnostic::operator<<(Operation *)
    -> Operation::print
    -> AsmState::AsmState
    -> verifyOpAndAdjustFlags
    -> mlir::verify
    -> spirv::FuncOp::verifyBody
    -> retOp.getValue().getType()
  ```

The relevant backtrace frames are:
frame #3  mlir::detail::ValueImpl::getType(this=0x0)
frame #4  mlir::Value::getType()
frame #5  mlir::spirv::FuncOp::verifyBody() at SPIRVOps.cpp:1229
frame #30 verifyOpAndAdjustFlags at AsmPrinter.cpp:2113
frame #34 mlir::Diagnostic::appendOp
frame #37 legalizeUnresolvedMaterialization at DialectConversion.cpp:3395
frame #38 mlir::OperationConverter::applyConversion

So the failed conversion leaves a malformed spirv.ReturnValue with a missing/null operand, and diagnostic printing triggers verification of that IR. The patch now makes spirv::FuncOp::verifyBody() diagnose the missing return value before reading its type, and removes the dialect-conversion printing workaround.

I also reran:
  `build/bin/llvm-lit -v mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir`

It passes after the current fix.

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Wed Jul 22 13:57:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Wed, 22 Jul 2026 13:57:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][inliner] Bound recursive expansion
 across SCC iterations (PR #211377)
Message-ID: <llvm.org/llvm/llvm-project/pull/211377@github.com>


https://github.com/khaki3 created https://github.com/llvm/llvm-project/pull/211377

The MLIR inliner tracks inline history only within one invocation of `inlineCallsInSCC`. This correctly detects recursion while processing the current call worklist.

However, `inlineSCC` repeatedly runs optimization and inlining. On every iteration, the inline history was recreated. A recursive call left from the previous iteration was therefore treated as a new root call and expanded again.

For example:

```mlir
func.func @caller(%arg: i32) -> i32 {
  %0 = call @a(%arg) : (i32) -> i32
  return %0 : i32
}

func.func @a(%arg: i32) -> i32 {
  %0 = arith.addi %arg, %c1 : i32
  %1 = call @b(%0) : (i32) -> i32
  return %1 : i32
}

func.func @b(%arg: i32) -> i32 {
  %0 = arith.addi %arg, %c2 : i32
  %1 = call @a(%0) : (i32) -> i32
  return %1 : i32
}
```

### Before

The first inlining iteration produced one bounded expansion:

```text
caller -> a -> b -> a
```

The local history stopped the final `a`, but the next SCC iteration forgot that history and expanded it again:

```text
caller -> a -> b -> a -> b -> a -> b -> a ...
```

With four inliner iterations, the caller contained eight cloned additions before the remaining recursive call.

Conceptually:

```mlir
%0 = arith.addi %arg, %c1
%1 = arith.addi %0, %c2
%2 = arith.addi %1, %c1
%3 = arith.addi %2, %c2
%4 = arith.addi %3, %c1
%5 = arith.addi %4, %c2
%6 = arith.addi %5, %c1
%7 = arith.addi %6, %c2
%8 = call @a(%7)
```

This causes recursive IR growth on every SCC fixed-point iteration.

## Algorithm change

The existing per-worklist inline history remains unchanged. It still detects whether a target already appears in the current expansion lineage.

When that history detects a recursive expansion, the inliner now records the corresponding call-graph edge:

```text
(sourceNode, targetNode)
```

Two edge sets are used:

```cpp
BlockedEdges blockedEdges;
BlockedEdges newlyBlockedEdges;
```

The algorithm is:

```text
for each SCC iteration:
  optimize the SCC

  for each call in the current worklist:
    if target occurs in the call's local inline history:
      add (source, target) to newlyBlockedEdges
      do not inline this recursive call
    else if (source, target) is in blockedEdges:
      do not inline it
    else:
      apply the normal legality and profitability checks
      inline when permitted

  after the complete worklist is processed:
    merge newlyBlockedEdges into blockedEdges
```

Newly detected edges are not blocked immediately. Activation is delayed until the next SCC iteration so that independent calls on the same edge already present in the current worklist retain their normal inlining opportunities.

The edge state is scoped to `inlineSCC`. Other call-graph edges continue through the normal optimization and inlining fixed-point process, including opportunities exposed by canonicalization.

Call-graph edges are used instead of call-operation pointers because the configurable optimization pipeline may replace call operations between iterations. Exact call-site provenance cannot safely be retained across arbitrary transformations, so a proven recursive graph edge is conservatively blocked for the remaining iterations of that SCC.

### After

The first expansion remains unchanged, but the recursive edge is blocked in later SCC iterations:

```mlir
%0 = arith.addi %arg, %c1
%1 = arith.addi %0, %c2
%2 = call @a(%1)
```

With canonicalization, the additions may be folded:

```mlir
%c3 = arith.constant 3 : i32
%0 = arith.addi %arg, %c3
%1 = call @a(%0)
```

The recursive expansion is bounded while unrelated inlining and optimization continue.

>From 33c8581519e2db6e148bff87e19e66c6bbfb3747 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Wed, 22 Jul 2026 13:50:09 -0700
Subject: [PATCH] [MLIR][Inliner] Bound recursive expansion across iterations

---
 mlir/lib/Transforms/Utils/Inliner.cpp        | 36 +++++---
 mlir/test/Transforms/inlining-recursive.mlir | 94 ++++++++++++++++++++
 2 files changed, 120 insertions(+), 10 deletions(-)

diff --git a/mlir/lib/Transforms/Utils/Inliner.cpp b/mlir/lib/Transforms/Utils/Inliner.cpp
index 40950312d566f..293789cf5873a 100644
--- a/mlir/lib/Transforms/Utils/Inliner.cpp
+++ b/mlir/lib/Transforms/Utils/Inliner.cpp
@@ -19,6 +19,7 @@
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Support/DebugStringHelper.h"
 #include "mlir/Transforms/InliningUtils.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SCCIterator.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Support/DebugLog.h"
@@ -354,7 +355,7 @@ static std::string getNodeName(CallOpInterface op) {
   return "_unnamed_callee_";
 }
 
-/// Return true if the specified `inlineHistoryID`  indicates an inline history
+/// Return true if the specified `inlineHistoryID` indicates an inline history
 /// that already includes `node`.
 static bool inlineHistoryIncludes(
     CallGraphNode *node, std::optional<size_t> inlineHistoryID,
@@ -419,6 +420,9 @@ struct InlinerInterfaceImpl : public InlinerInterface {
 
 namespace mlir {
 
+using CallGraphEdge = std::pair<CallGraphNode *, CallGraphNode *>;
+using BlockedEdges = DenseSet<CallGraphEdge>;
+
 class Inliner::Impl {
 public:
   Impl(Inliner &inliner) : inliner(inliner) {}
@@ -453,7 +457,8 @@ class Inliner::Impl {
   /// Attempt to inline calls within the given scc. This function returns
   /// success if any calls were inlined, failure otherwise.
   LogicalResult inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
-                                 CGUseList &useList, CallGraphSCC &currentSCC);
+                                 CGUseList &useList, CallGraphSCC &currentSCC,
+                                 BlockedEdges &blockedEdges);
 
   /// Returns true if the given call should be inlined.
   bool shouldInline(ResolvedCall &resolvedCall);
@@ -471,10 +476,14 @@ LogicalResult Inliner::Impl::inlineSCC(InlinerInterfaceImpl &inlinerIface,
   // hit the maximum iteration count. Simplifying early helps to refine the cost
   // model, and in future iterations may devirtualize new calls.
   unsigned iterationCount = 0;
+  // Optimization may replace calls, preventing exact provenance tracking.
+  // Conservatively retain proven recursive graph edges across iterations.
+  BlockedEdges blockedEdges;
   do {
     if (failed(optimizeSCC(inlinerIface.cg, useList, currentSCC, context)))
       return failure();
-    if (failed(inlineCallsInSCC(inlinerIface, useList, currentSCC)))
+    if (failed(
+            inlineCallsInSCC(inlinerIface, useList, currentSCC, blockedEdges)))
       break;
   } while (++iterationCount < inliner.config.getMaxInliningIterations());
   return success();
@@ -580,7 +589,8 @@ Inliner::Impl::optimizeCallable(CallGraphNode *node,
 /// success if any calls were inlined, failure otherwise.
 LogicalResult
 Inliner::Impl::inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
-                                CGUseList &useList, CallGraphSCC &currentSCC) {
+                                CGUseList &useList, CallGraphSCC &currentSCC,
+                                BlockedEdges &blockedEdges) {
   CallGraph &cg = inlinerIface.cg;
   auto &calls = inlinerIface.calls;
 
@@ -604,12 +614,12 @@ Inliner::Impl::inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
     }
   }
 
-  // When inlining a callee produces new call sites, we want to keep track of
-  // the fact that they were inlined from the callee. This allows us to avoid
-  // infinite inlining.
+  // When inlining a callee produces new call sites, remember that they came
+  // from the callee to avoid recursively inlining through a cycle.
   using InlineHistoryT = std::optional<size_t>;
   SmallVector<std::pair<CallGraphNode *, InlineHistoryT>, 8> inlineHistory;
   std::vector<InlineHistoryT> callHistory(calls.size(), InlineHistoryT{});
+  BlockedEdges newlyBlockedEdges;
 
   LLVM_DEBUG({
     LDBG() << "* Inliner: Initial calls in SCC are: {";
@@ -629,7 +639,11 @@ Inliner::Impl::inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
     InlineHistoryT inlineHistoryID = callHistory[i];
     bool inHistory =
         inlineHistoryIncludes(it.targetNode, inlineHistoryID, inlineHistory);
-    bool doInline = !inHistory && shouldInline(it);
+    auto edge = std::make_pair(it.sourceNode, it.targetNode);
+    if (inHistory)
+      newlyBlockedEdges.insert(edge);
+    bool doInline =
+        !inHistory && !blockedEdges.contains(edge) && shouldInline(it);
     CallOpInterface call = it.call;
     LLVM_DEBUG({
       if (doInline)
@@ -657,8 +671,7 @@ Inliner::Impl::inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
     }
     inlinedAnyCalls = true;
 
-    // Create a inline history entry for this inlined call, so that we remember
-    // that new callsites came about due to inlining Callee.
+    // Record that the new callsites came from inlining the callee.
     InlineHistoryT newInlineHistoryID{inlineHistory.size()};
     inlineHistory.push_back(std::make_pair(it.targetNode, inlineHistoryID));
 
@@ -695,6 +708,9 @@ Inliner::Impl::inlineCallsInSCC(InlinerInterfaceImpl &inlinerIface,
     currentSCC.remove(node);
     inlinerIface.markForDeletion(node);
   }
+  // Delay blocking until the next iteration so independent calls on the same
+  // edge in the current worklist retain their inlining opportunities.
+  blockedEdges.insert(newlyBlockedEdges.begin(), newlyBlockedEdges.end());
   calls.clear();
   return success(inlinedAnyCalls);
 }
diff --git a/mlir/test/Transforms/inlining-recursive.mlir b/mlir/test/Transforms/inlining-recursive.mlir
index f953935475e1a..4ea72884a4d59 100644
--- a/mlir/test/Transforms/inlining-recursive.mlir
+++ b/mlir/test/Transforms/inlining-recursive.mlir
@@ -1,5 +1,6 @@
 // RUN: mlir-opt %s -inline='default-pipeline=' | FileCheck %s
 // RUN: mlir-opt %s --mlir-disable-threading -inline='default-pipeline=' | FileCheck %s
+// RUN: mlir-opt %s -inline | FileCheck %s --check-prefix=DEFAULT
 
 // CHECK-LABEL: func.func @foo0
 func.func @foo0(%arg0 : i32) -> i32 {
@@ -23,3 +24,96 @@ func.func @foo1(%arg0 : i32) -> i32 {
   %2 = call @foo0(%1) : (i32) -> i32
   return %2 : i32
 }
+
+// Verify that recursive expansion is bounded across inliner iterations.
+// CHECK-LABEL: func.func @caller
+// CHECK-NEXT: %{{.*}} = arith.constant 1 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = arith.constant 2 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = call @cycle_a
+// CHECK-NEXT: %{{.*}} = constant @leaf
+// CHECK-NEXT: %{{.*}} = call_indirect
+// CHECK-NEXT: %{{.*}} = constant @cycle_a
+// CHECK-NEXT: %{{.*}} = call_indirect
+// CHECK-NEXT: return
+// CHECK-NEXT: }
+func.func @caller(%arg0 : i32) -> i32 {
+  %0 = call @cycle_a(%arg0) : (i32) -> i32
+  %1 = call @wrapper(%0) : (i32) -> i32
+  %2 = call @same_edge_wrapper(%1) : (i32) -> i32
+  return %2 : i32
+}
+
+// DEFAULT-LABEL: func.func @caller
+// DEFAULT-NEXT: %{{.*}} = arith.constant 6 : i32
+// DEFAULT-NEXT: %{{.*}} = arith.constant 4 : i32
+// DEFAULT-NEXT: %{{.*}} = arith.constant 3 : i32
+// DEFAULT-NEXT: %{{.*}} = arith.addi
+// DEFAULT-NEXT: %{{.*}} = call @cycle_a
+// DEFAULT-NEXT: %{{.*}} = arith.muli
+// DEFAULT-NEXT: %{{.*}} = arith.addi
+// DEFAULT-NEXT: %{{.*}} = call @cycle_a
+// DEFAULT-NEXT: return
+// DEFAULT-NEXT: }
+
+// CHECK-LABEL: func.func @cycle_a
+// DEFAULT-LABEL: func.func @cycle_a
+func.func @cycle_a(%arg0 : i32) -> i32 {
+  %c1 = arith.constant 1 : i32
+  %0 = arith.addi %arg0, %c1 : i32
+  %1 = call @cycle_b(%0) : (i32) -> i32
+  return %1 : i32
+}
+
+func.func @cycle_b(%arg0 : i32) -> i32 {
+  %c2 = arith.constant 2 : i32
+  %0 = arith.addi %arg0, %c2 : i32
+  %1 = call @cycle_a(%0) : (i32) -> i32
+  return %1 : i32
+}
+
+func.func @wrapper(%arg0 : i32) -> i32 {
+  %fn = constant @leaf : (i32) -> i32
+  %0 = call_indirect %fn(%arg0) : (i32) -> i32
+  return %0 : i32
+}
+
+func.func @same_edge_wrapper(%arg0 : i32) -> i32 {
+  %fn = constant @cycle_a : (i32) -> i32
+  %0 = call_indirect %fn(%arg0) : (i32) -> i32
+  return %0 : i32
+}
+
+func.func @leaf(%arg0 : i32) -> i32 {
+  %c4 = arith.constant 4 : i32
+  %0 = arith.muli %arg0, %c4 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func.func @two_calls
+// CHECK-NEXT: %{{.*}} = arith.constant 1 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = arith.constant 2 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = call @cycle_a
+// CHECK-NEXT: %{{.*}} = arith.constant 1 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = arith.constant 2 : i32
+// CHECK-NEXT: %{{.*}} = arith.addi
+// CHECK-NEXT: %{{.*}} = call @cycle_a
+// CHECK-NEXT: return
+// CHECK-NEXT: }
+// DEFAULT-LABEL: func.func @two_calls
+// DEFAULT-NEXT: %{{.*}} = arith.constant 3 : i32
+// DEFAULT-NEXT: %{{.*}} = arith.addi
+// DEFAULT-NEXT: %{{.*}} = call @cycle_a
+// DEFAULT-NEXT: %{{.*}} = arith.addi
+// DEFAULT-NEXT: %{{.*}} = call @cycle_a
+// DEFAULT-NEXT: return
+// DEFAULT-NEXT: }
+func.func @two_calls(%arg0 : i32) -> i32 {
+  %0 = call @cycle_a(%arg0) : (i32) -> i32
+  %1 = call @cycle_a(%0) : (i32) -> i32
+  return %1 : i32
+}


From llvmlistbot at llvm.org  Wed Jul 22 15:48:39 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Wed, 22 Jul 2026 15:48:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a6148c7.e624eb8d.12f676.df8a@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/211141

>From 829c25c037707f17d8c98fcb15c041914a26a46d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 16:07:42 -0700
Subject: [PATCH 1/5] Elevate allocs to globals pass

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |  10 ++
 .../Dialect/MemRef/Transforms/Transforms.h    |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElevateAllocsToGlobals.cpp     | 112 ++++++++++++++++++
 .../MemRef/elevate-allocs-to-globals.mlir     |  56 +++++++++
 5 files changed, 182 insertions(+)
 create mode 100644 mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
 create mode 100644 mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index c6be600247696..4e6ead48cbff2 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -340,4 +340,14 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
   ];
 }
 
+def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
+  let summary = "Elevate allocs to globals";
+  let description = [{
+
+  }];
+  let dependentDialects = [
+      "memref::MemRefDialect"
+  ];
+}
+
 #endif // MLIR_DIALECT_MEMREF_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
index 720677455ae5d..6960b8ef00f06 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #ifndef MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 #define MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
 
@@ -165,6 +166,8 @@ void populateExtractAddressComputationsPatterns(RewritePatternSet &patterns);
 /// into one-dimensional memref operations.
 void populateFlattenMemrefsPatterns(RewritePatternSet &patterns);
 
+void populateElevateAllocsToGlobalsPatterns(RewritePatternSet &patterns);
+
 /// Build a new memref::AllocaOp whose dynamic sizes are independent of all
 /// given independencies. If the op is already independent of all
 /// independencies, the same AllocaOp result is returned.
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1a8b03dabcfb7..7d43e66989bc4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -17,6 +17,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   ReifyResultShapes.cpp
   ResolveShapedTypeResultDims.cpp
   RuntimeOpVerification.cpp
+  ElevateAllocsToGlobals.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/MemRef
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
new file mode 100644
index 0000000000000..017b23b810dd3
--- /dev/null
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -0,0 +1,112 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Interfaces/LoopLikeInterface.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/LogicalResult.h"
+
+namespace mlir {
+namespace memref {
+#define GEN_PASS_DEF_ELEVATEALLOCSTOGLOBALSPASS
+#include "mlir/Dialect/MemRef/Transforms/Passes.h.inc"
+} // namespace memref
+} // namespace mlir
+
+using namespace mlir;
+
+namespace {
+
+// Checks if 'op' is contained inside any branching or looping structure
+static bool isInsideControlFlow(mlir::Operation *op) {
+  if (mlir::getEnclosingRepetitiveRegion(op) != nullptr)
+    return true;
+
+  if (op->getParentOfType<mlir::LoopLikeOpInterface>())
+    return true;
+
+  if (auto regionParent = op->getParentOfType<mlir::RegionBranchOpInterface>())
+    return true;
+
+  return false;
+}
+
+struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
+public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(memref::AllocOp allocOp,
+                                PatternRewriter &rewriter) const final {
+
+    auto memrefType = allocOp.getType();
+    // memref.global requires statically shaped memrefs
+    if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
+      return failure();
+
+    auto loopParent = allocOp->getParentOfType<mlir::LoopLikeOpInterface>();
+    if (loopParent != nullptr || isInsideControlFlow(allocOp))
+      return failure();
+
+    memref::GlobalOp globalOp;
+    {
+      Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
+
+      SymbolTable symbolTable(symbolTableOp);
+
+      OpBuilder builder(rewriter.getContext());
+      StringAttr globalName = rewriter.getStringAttr("global_alloc");
+      globalOp = memref::GlobalOp::create(builder, allocOp.getLoc(), globalName,
+                                          rewriter.getStringAttr("private"),
+                                          memrefType, rewriter.getUnitAttr(),
+                                          false, allocOp.getAlignmentAttr());
+
+      symbolTable.insert(globalOp);
+    }
+
+    SmallVector<Operation *> deallocsToDelete;
+    for (OpOperand &use : allocOp.getResult().getUses()) {
+      Operation *user = use.getOwner();
+      if (isa<memref::DeallocOp>(user))
+        deallocsToDelete.push_back(user);
+    }
+    for (Operation *dealloc : deallocsToDelete)
+      rewriter.eraseOp(dealloc);
+
+    rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
+                                                     globalOp.getName());
+
+    return success();
+  }
+};
+
+struct ElevateAllocsToGlobalsPass
+    : public mlir::memref::impl::ElevateAllocsToGlobalsPassBase<
+          ElevateAllocsToGlobalsPass> {
+  using Base::Base;
+
+  void runOnOperation() override {
+    ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    memref::populateElevateAllocsToGlobalsPatterns(patterns);
+
+    (void)applyPatternsGreedily(moduleOp, std::move(patterns));
+  }
+};
+} // namespace
+
+void mlir::memref::populateElevateAllocsToGlobalsPatterns(
+    RewritePatternSet &patterns) {
+  patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
new file mode 100644
index 0000000000000..dd65cd12989cc
--- /dev/null
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -0,0 +1,56 @@
+// RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
+
+/// Test that a single static memref.alloc is elevated to a memref.global,
+/// replaced with memref.get_global, alignment attribute is preserved, and
+/// associated memref.dealloc is removed.
+
+func.func @single_alloc(%val: f32, %idx: index) {
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
+  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
+  memref.dealloc %0 : memref<10x20xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @single_alloc(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: return
+// CHECK-NOT: memref.dealloc
+// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+
+// -----
+
+/// Test that multiple static memref.alloc ops in the same function are elevated
+/// to global memrefs without symbol name collisions.
+
+func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+  %0 = memref.alloc() : memref<10xf32>
+  %1 = memref.alloc() : memref<20xi32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  return
+}
+
+// CHECK-LABEL: func.func @multiple_allocs(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
+
+// -----
+
+/// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<?xf32>
+  memref.store %val, %0[%sz] : memref<?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>

>From 39d4e08842d4dafd659020418859af23193db91c Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 17:48:50 -0700
Subject: [PATCH 2/5] Simplify and add tests

---
 .../Transforms/ElevateAllocsToGlobals.cpp     |  2 +-
 .../MemRef/elevate-allocs-to-globals.mlir     | 88 +++++++++++++++----
 2 files changed, 70 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 017b23b810dd3..2dbdeb00c5354 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -109,4 +109,4 @@ struct ElevateAllocsToGlobalsPass
 void mlir::memref::populateElevateAllocsToGlobalsPatterns(
     RewritePatternSet &patterns) {
   patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
index dd65cd12989cc..43477960cf7be 100644
--- a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -1,49 +1,49 @@
 // RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
 
-/// Test that a single static memref.alloc is elevated to a memref.global,
-/// replaced with memref.get_global, alignment attribute is preserved, and
-/// associated memref.dealloc is removed.
+/// Test that a single static memref.alloc is elevated to a memref.global, references
+/// are replaced with memref.get_global, and the associated memref.dealloc is removed.
 
 func.func @single_alloc(%val: f32, %idx: index) {
-  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
-  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
-  memref.dealloc %0 : memref<10x20xf32>
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.dealloc %0 : memref<10xf32>
   return
 }
 
 // CHECK-LABEL: func.func @single_alloc(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<10xf32>
 // CHECK-NEXT: return
 // CHECK-NOT: memref.dealloc
-// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
 
 // -----
 
 /// Test that multiple static memref.alloc ops in the same function are elevated
 /// to global memrefs without symbol name collisions.
 
-func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+func.func @multiple_allocs(%val: f32, %idx: index) {
   %0 = memref.alloc() : memref<10xf32>
-  %1 = memref.alloc() : memref<20xi32>
+  %1 = memref.alloc() : memref<20xf32>
   memref.store %val, %0[%idx] : memref<10xf32>
-  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  memref.store %val, %1[%idx] : memref<20xf32>
   return
 }
 
 // CHECK-LABEL: func.func @multiple_allocs(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
 // CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
-// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
-// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
-// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
-// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG1]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM1]][%[[ARG1]]] : memref<20xf32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xf32> = uninitialized
 // CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
 
 // -----
 
 /// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+
 func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
   %0 = memref.alloc(%sz) : memref<?xf32>
   memref.store %val, %0[%sz] : memref<?xf32>
@@ -52,5 +52,55 @@ func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
 
 // CHECK-LABEL: func.func @dynamic_alloc_ignored(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
-// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+
+// -----
+
+/// Test that a partially dynamic memref.alloc is ignored and not elevated to a global.
+
+func.func @partially_dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<10x?xf32>
+  memref.store %val, %0[%sz, %sz] : memref<10x?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @partially_dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<10x?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x?xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside a loop is ignored and not elevated to a global.
+
+func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index, %val: f32, %idx: index) {
+  scf.for %i = %lb to %ub step %step {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_loop_ignored(
+// CHECK: scf.for
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+
+// -----
+
+/// Test that a static memref.alloc inside control flow (scf.if) is ignored and not
+/// elevated to a global.
+
+func.func @alloc_in_control_flow_ignored(%cond: i1, %val: f32, %idx: index) {
+  scf.if %cond {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_control_flow_ignored(
+// CHECK: scf.if
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]

>From 01a313bd5730d92c90ec57328ff27adc7040c38b Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:29:11 -0700
Subject: [PATCH 3/5] Pass description

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 4e6ead48cbff2..a0a4be7318889 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -343,7 +343,31 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
 def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
   let summary = "Elevate allocs to globals";
   let description = [{
+    This pass converts statically-shaped `memref.alloc` operations that are not
+    enclosed within loops or control flow into private `memref.global` operations.
+    `memref.alloc` operations are replaced with `memref.get_global` references and
+    any associated `memref.dealloc` operations are removed.
 
+    Example:
+
+    ```mlir
+    func.func @example() {
+      %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+      memref.dealloc %0 : memref<10xf32>
+      return
+    }
+    ```
+
+    is transformed to
+
+    ```mlir
+    memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
+
+    func.func @example() {
+      %0 = memref.get_global @global_alloc : memref<10xf32>
+      return
+    }
+    ```
   }];
   let dependentDialects = [
       "memref::MemRefDialect"

>From 8c050bcf5144257840ada1230f51a4ad249b39f8 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:36:18 -0700
Subject: [PATCH 4/5] Clean up helper func

---
 .../Transforms/ElevateAllocsToGlobals.cpp     | 20 ++++++-------------
 1 file changed, 6 insertions(+), 14 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 2dbdeb00c5354..5a079ac8c2b63 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -28,18 +28,11 @@ using namespace mlir;
 
 namespace {
 
-// Checks if 'op' is contained inside any branching or looping structure
-static bool isInsideControlFlow(mlir::Operation *op) {
-  if (mlir::getEnclosingRepetitiveRegion(op) != nullptr)
-    return true;
-
-  if (op->getParentOfType<mlir::LoopLikeOpInterface>())
-    return true;
-
-  if (auto regionParent = op->getParentOfType<mlir::RegionBranchOpInterface>())
-    return true;
-
-  return false;
+// Checks if 'op' is contained inside any branching or looping structure.
+static bool isInsideControlFlow(Operation *op) {
+  return getEnclosingRepetitiveRegion(op) ||
+         op->getParentOfType<LoopLikeOpInterface>() ||
+         op->getParentOfType<RegionBranchOpInterface>();
 }
 
 struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
@@ -54,8 +47,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
     if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
       return failure();
 
-    auto loopParent = allocOp->getParentOfType<mlir::LoopLikeOpInterface>();
-    if (loopParent != nullptr || isInsideControlFlow(allocOp))
+    if (isInsideControlFlow(allocOp))
       return failure();
 
     memref::GlobalOp globalOp;

>From 029c272c2e156ed6562aa4ed57902fb33b3db89c Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:48:22 -0700
Subject: [PATCH 5/5] Comments

---
 .../Transforms/ElevateAllocsToGlobals.cpp     | 27 ++++++++++++++++---
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 5a079ac8c2b63..4533986a2547e 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -28,32 +28,49 @@ using namespace mlir;
 
 namespace {
 
-// Checks if 'op' is contained inside any branching or looping structure.
+/// Returns true if `op` is contained inside any branching, region, or looping
+/// structure (such as scf.for, scf.if, or repetitive regions)
 static bool isInsideControlFlow(Operation *op) {
   return getEnclosingRepetitiveRegion(op) ||
          op->getParentOfType<LoopLikeOpInterface>() ||
          op->getParentOfType<RegionBranchOpInterface>();
 }
 
+/// Elevates a static `memref.alloc` operation to a top-level `memref.global` op
+/// if the allocation is not enclosed within any control flow constructs.
+///
+/// Converts:
+/// ```mlir
+/// %0 = memref.alloc() : memref<4x4xf32>
+/// memref.dealloc %0 : memref<4x4xf32>
+/// ```
+/// to:
+/// ```mlir
+/// memref.global "private" @global_alloc : memref<4x4xf32>
+/// ...
+/// %0 = memref.get_global @global_alloc : memref<4x4xf32>
+/// ```
 struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
   LogicalResult matchAndRewrite(memref::AllocOp allocOp,
                                 PatternRewriter &rewriter) const final {
-
     auto memrefType = allocOp.getType();
-    // memref.global requires statically shaped memrefs
+    // `memref.global` requires statically shaped memrefs with no dynamic sizes.
     if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
       return failure();
 
+    // Avoid elevating allocations inside control flow (loops or conditionals),
+    // as converting them to a single static global would make multiple executions
+    // share the same buffer, changing semantics or causing race conditions.
     if (isInsideControlFlow(allocOp))
       return failure();
 
+    // Create the global variable at the nearest enclosing symbol table (e.g. module).
     memref::GlobalOp globalOp;
     {
       Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
-
       SymbolTable symbolTable(symbolTableOp);
 
       OpBuilder builder(rewriter.getContext());
@@ -66,6 +83,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
       symbolTable.insert(globalOp);
     }
 
+    // Remove any `memref.dealloc` operations using this allocation
     SmallVector<Operation *> deallocsToDelete;
     for (OpOperand &use : allocOp.getResult().getUses()) {
       Operation *user = use.getOwner();
@@ -75,6 +93,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
     for (Operation *dealloc : deallocsToDelete)
       rewriter.eraseOp(dealloc);
 
+    // Replace the original `memref.alloc` with `memref.get_global`.
     rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
                                                      globalOp.getName());
 


From llvmlistbot at llvm.org  Thu Jul 23 00:02:09 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:02:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bc71.146286d8.31cf25.5696@mx.google.com>


================
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
----------------
anutosh491 wrote:

Added a comment explaining that the source group remains unconditional because runtime/*.py contains generic memref helpers, while only the wrapper and type stub depend on _mlirExecutionEngine.

https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Thu Jul 23 00:03:30 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:03:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bcc2.8f0cbdd2.ba895.3a4f@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/211250

>From d4920ce022d49d737a8157802141eeecc53a6942 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 18:10:06 +0530
Subject: [PATCH 1/2] [MLIR][Python] Do not install ExecutionEngine wrapper
 when disabled

---
 mlir/python/CMakeLists.txt | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..693701ef50c13 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"

>From a5e88cb89932a70a1d11f332e56e2798e114973c Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Thu, 23 Jul 2026 12:32:44 +0530
Subject: [PATCH 2/2] Add comment as per review

---
 mlir/python/CMakeLists.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 693701ef50c13..54d915fac3d48 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,9 +41,12 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+# Keep this source group unconditional because runtime/*.py contains generic
+# memref helpers that do not depend on the ExecutionEngine extension. Only the
+# wrapper and type stub require _mlirExecutionEngine.
 set(_execution_engine_python_sources)
 if(MLIR_ENABLE_EXECUTION_ENGINE)
-  list(APPEND _execution_engine_python_sources
+  set(_execution_engine_python_sources
     execution_engine.py
     _mlir_libs/_mlirExecutionEngine.pyi
   )


From llvmlistbot at llvm.org  Thu Jul 23 00:05:15 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:05:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bd2b.b3997315.1c21a.125d@mx.google.com>


https://github.com/anutosh491 updated https://github.com/llvm/llvm-project/pull/211250

>From 1a66068b298563a32912c916d7edb02bc0cba515 Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Wed, 22 Jul 2026 18:10:06 +0530
Subject: [PATCH 1/2] [MLIR][Python] Do not install ExecutionEngine wrapper
 when disabled

---
 mlir/python/CMakeLists.txt | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..693701ef50c13 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,23 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  list(APPEND _execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"

>From 2609c1cf039292a6c55dcd1f9dc7f77fddd489fe Mon Sep 17 00:00:00 2001
From: anutosh491 <andersonbhat491 at gmail.com>
Date: Thu, 23 Jul 2026 12:32:44 +0530
Subject: [PATCH 2/2] Add comment as per review

---
 mlir/python/CMakeLists.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 693701ef50c13..54d915fac3d48 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,9 +41,12 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+# Keep this source group unconditional because runtime/*.py contains generic
+# memref helpers that do not depend on the ExecutionEngine extension. Only the
+# wrapper and type stub require _mlirExecutionEngine.
 set(_execution_engine_python_sources)
 if(MLIR_ENABLE_EXECUTION_ENGINE)
-  list(APPEND _execution_engine_python_sources
+  set(_execution_engine_python_sources
     execution_engine.py
     _mlir_libs/_mlirExecutionEngine.pyi
   )


From llvmlistbot at llvm.org  Thu Jul 23 00:06:53 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:06:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bd8d.62a8d9ec.74724.dbdb@mx.google.com>


anutosh491 wrote:

Thanks for the approval !

Enabling auto merge.

https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Thu Jul 23 00:07:07 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:07:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bd9b.f1c1923d.d9163.0e30@mx.google.com>


https://github.com/anutosh491 auto_merge_enabled https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Thu Jul 23 00:15:17 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 00:15:17 -0700 (PDT)
Subject: [Mlir-commits] [mlir] c151d33 - [MLIR][Python] Do not install
 ExecutionEngine wrapper when disabled (#211250)
Message-ID: <6a61bf85.c8f5739a.2e110d.4f0a@mx.google.com>


Author: Anutosh Bhat
Date: 2026-07-23T07:15:12Z
New Revision: c151d33acea677cee0c29ec15f9d5f6c7d31d0fd

URL: https://github.com/llvm/llvm-project/commit/c151d33acea677cee0c29ec15f9d5f6c7d31d0fd
DIFF: https://github.com/llvm/llvm-project/commit/c151d33acea677cee0c29ec15f9d5f6c7d31d0fd.diff

LOG: [MLIR][Python] Do not install ExecutionEngine wrapper when disabled (#211250)

The `_mlirExecutionEngine` extension is declared only when
`MLIR_ENABLE_EXECUTION_ENGINE` is enabled. However,
`execution_engine.py` and its type stub are currently added to the
Python package unconditionally.

This leaves an installed `mlir.execution_engine` module that fails
immediately when imported because its native extension is absent. It
also exposes a type stub for an unavailable API.

Making sure they're added only if `MLIR_ENABLE_EXECUTION_ENGINE` is
enabled.

Added: 
    

Modified: 
    mlir/python/CMakeLists.txt

Removed: 
    


################################################################################
diff  --git a/mlir/python/CMakeLists.txt b/mlir/python/CMakeLists.txt
index 802f8c126512e..54d915fac3d48 100644
--- a/mlir/python/CMakeLists.txt
+++ b/mlir/python/CMakeLists.txt
@@ -41,15 +41,26 @@ declare_mlir_python_sources(MLIRPythonSources.Core.Python.Extras
     extras/meta.py
 )
 
+# Keep this source group unconditional because runtime/*.py contains generic
+# memref helpers that do not depend on the ExecutionEngine extension. Only the
+# wrapper and type stub require _mlirExecutionEngine.
+set(_execution_engine_python_sources)
+if(MLIR_ENABLE_EXECUTION_ENGINE)
+  set(_execution_engine_python_sources
+    execution_engine.py
+    _mlir_libs/_mlirExecutionEngine.pyi
+  )
+endif()
+
 declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
   ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
   ADD_TO_PARENT MLIRPythonSources
   SOURCES
-    execution_engine.py
-    _mlir_libs/_mlirExecutionEngine.pyi
+    ${_execution_engine_python_sources}
   SOURCES_GLOB
     runtime/*.py
 )
+unset(_execution_engine_python_sources)
 
 declare_mlir_python_sources(MLIRPythonCAPI.HeaderSources
   ROOT_DIR "${MLIR_SOURCE_DIR}/include"


        

From llvmlistbot at llvm.org  Thu Jul 23 00:15:21 2026
From: llvmlistbot at llvm.org (Anutosh Bhat)
Date: Thu, 23 Jul 2026 00:15:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Do not install ExecutionEngine
 wrapper when disabled (PR #211250)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211250@github.com>
Message-ID: <6a61bf89.bd23cc60.26c84a.6446@mx.google.com>


https://github.com/anutosh491 closed https://github.com/llvm/llvm-project/pull/211250

From llvmlistbot at llvm.org  Thu Jul 23 00:17:47 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:17:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c01b.b1823a4b.3d677e.447f@mx.google.com>


https://github.com/arun-thmn updated https://github.com/llvm/llvm-project/pull/204327

>From 23c66ee4340af485894825ac68c082ea2c28b07e Mon Sep 17 00:00:00 2001
From: Arun Thangamani <arun.thangamani at intel.com>
Date: Wed, 17 Jun 2026 02:48:59 -0700
Subject: [PATCH 1/2] does vector.contract(A, B, Acc) into vector.contract(A,
 B, 0) + Acc

---
 .../X86/TransformOps/X86TransformOps.td       |  10 ++
 mlir/include/mlir/Dialect/X86/Transforms.h    |   5 +
 .../include/mlir/Dialect/X86/Utils/X86Utils.h |  10 +-
 .../X86/TransformOps/X86TransformOps.cpp      |   5 +
 .../lib/Dialect/X86/Transforms/CMakeLists.txt |   1 +
 .../MoveAccumulatorForContractLoop.cpp        | 114 +++++++++++++++
 .../Transforms/VectorContractBF16ToFMA.cpp    |  28 ++--
 .../VectorContractToPackedTypeDotProduct.cpp  |  30 ++--
 mlir/lib/Dialect/X86/Utils/X86Utils.cpp       |  63 +++++---
 .../X86/move-acc-for-contract-loop.mlir       | 136 ++++++++++++++++++
 10 files changed, 351 insertions(+), 51 deletions(-)
 create mode 100644 mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
 create mode 100644 mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir

diff --git a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
index c474cfb47d003..48aabf90914e2 100644
--- a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
+++ b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
@@ -82,5 +82,15 @@ def ApplyVectorContractToAMXDotProductPatternsOp : Op<Transform_Dialect,
   let assemblyFormat = "attr-dict";
 }
 
+def ApplyMoveAccumulatorForContractLoopPatternsOp : Op<Transform_Dialect,
+    "apply_patterns.x86.move_accumulator_for_contract_loop",
+    [DeclareOpInterfaceMethods<PatternDescriptorOpInterface>]> {
+  let description = [{
+         Rewrites an additive vector contraction by zeroing the accumulator input
+         and materializing the accumulation as a separate add after the contraction.
+  }];
+
+  let assemblyFormat = "attr-dict";
+}
 
 #endif // X86_TRANSFORM_OPS
diff --git a/mlir/include/mlir/Dialect/X86/Transforms.h b/mlir/include/mlir/Dialect/X86/Transforms.h
index 6ebba5e94ec7c..8bcaf9a67feb0 100644
--- a/mlir/include/mlir/Dialect/X86/Transforms.h
+++ b/mlir/include/mlir/Dialect/X86/Transforms.h
@@ -110,6 +110,11 @@ void populateShuffleVectorFMAOpsPatterns(RewritePatternSet &patterns);
 // Int8).
 void populateVectorContractToAMXDotProductPatterns(RewritePatternSet &patterns);
 
+// Rewrites an additive vector contraction by zeroing the accumulator input
+// and materializing the accumulation as a separate add after the contraction.
+void populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns);
+
 //===----------------------------------------------------------------------===//
 /// Helpers extracted from:
 ///   - clang/lib/Headers/avxintrin.h
diff --git a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
index 2ff8f4f4283a2..f067ce3bb9b65 100644
--- a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
+++ b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
@@ -29,6 +29,10 @@ namespace x86 {
 bool isInVnniLayout(Operation *op, llvm::ArrayRef<AffineMap> indexingMaps,
                     std::optional<unsigned> blockingFactor = std::nullopt);
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v);
+
 // Returns true if two contraction ops form a valid pair for VNNI packing.
 // It verifies that both contractions share the appropriate operand, read from
 // the same source buffer, and use constant indices that differ by 8 or 16.
@@ -59,9 +63,9 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc, VectorType accTy);
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
+                                       VectorType accTy);
 
 } // namespace x86
 } // namespace mlir
diff --git a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
index 390b21e12b0ed..a52f08a5cd061 100644
--- a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
+++ b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
@@ -52,6 +52,11 @@ void mlir::transform::ApplyVectorContractToAMXDotProductPatternsOp::
   x86::populateVectorContractToAMXDotProductPatterns(patterns);
 }
 
+void mlir::transform::ApplyMoveAccumulatorForContractLoopPatternsOp::
+    populatePatterns(RewritePatternSet &patterns) {
+  x86::populateMoveAccumulatorForContractLoopPatterns(patterns);
+}
+
 //===----------------------------------------------------------------------===//
 // Transform op registration
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
index 9c3695536cda9..5118563c03f16 100644
--- a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
@@ -7,6 +7,7 @@ add_mlir_dialect_library(MLIRX86Transforms
   SinkVectorProducerOps.cpp
   ShuffleVectorFMAOps.cpp
   VectorContractToAMXDotProduct.cpp
+  MoveAccumulatorForContractLoop.cpp
 
   LINK_LIBS PUBLIC
   MLIRArithDialect
diff --git a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
new file mode 100644
index 0000000000000..6f87ebfa441dc
--- /dev/null
+++ b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
+    Value accValue = accReadOp->getResult(0);
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with acontract results before storing to acc
+    // matrix.
+    rewriter.setInsertionPoint(resultWriteOp);
+    Location locUser = resultWriteOp->getLoc();
+
+    Value addition;
+
+    if (llvm::isa<FloatType>(elemTy)) {
+      addition =
+          arith::AddFOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    if (llvm::isa<IntegerType>(elemTy)) {
+      addition =
+          arith::AddIOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    resultWriteOp->replaceUsesOfWith(contractValue, addition);
+    return success();
+  }
+};
+
+} // namespace
+
+void x86::populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns) {
+  patterns.add<MoveAccumulatorForContractLoop>(patterns.getContext());
+}
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
index 287892c3a660f..c70ad221c772d 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
@@ -415,25 +415,21 @@ struct VectorContractBF16ToFMA
       Operation *accReadOp1 =
           traceToVectorReadLikeParentOperation(pairContractOp.getAcc());
 
-      // Iterate down to find the users of contact operations until it is store
-      // or transfer_write.
-      Operation *resultWriteOp0 =
-          traceToVectorWriteLikeUserOperation(contractOp.getResult());
-      Operation *resultWriteOp1 =
-          traceToVectorWriteLikeUserOperation(pairContractOp.getResult());
-
-      // Shuffle the accumulators of the contract operations.
-      LogicalResult readShuffle =
-          shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                 pairContractOp, nonUnitDim, accTy);
-
-      if (failed(readShuffle))
-        return rewriter.notifyMatchFailure(
-            contractOp, "Accumulator read is not by transfer_read or load");
+      if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+        // Shuffle the accumulators of the contract operations.
+        LogicalResult readShuffle =
+            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
+                                   pairContractOp, nonUnitDim, accTy);
+
+        if (failed(readShuffle))
+          return rewriter.notifyMatchFailure(
+              contractOp, "Accumulator read is not by transfer_read or load");
+      }
 
       // Shuffle the output of contract operations before its use.
       LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-          rewriter, resultWriteOp0, resultWriteOp1, nonUnitDim, accTy);
+          rewriter, contractOp.getResult(), pairContractOp.getResult(),
+          nonUnitDim, accTy);
 
       if (failed(writeShuffle))
         return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
index d3487369355f0..394c950189ead 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
@@ -325,17 +325,17 @@ struct VectorContractToPackedTypeDotProduct
       }
 
       // If the accumulators are shuffled we get nullptr else the
-      // transfer_read or load operations.
-      Operation *accRead =
-          traceToVectorReadLikeParentOperation(contractOp.getAcc());
+      // transfer_load or store operations.
+      Operation *accWrite =
+          traceToVectorWriteLikeUserOperation(contractOp.getResult());
 
       if (!pairContractOp &&
-          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accRead))
+          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accWrite))
         return rewriter.notifyMatchFailure(contractOp,
                                            "Could not find a contract pair");
 
       // Validate and shuffle the accumulator
-      if (accRead) {
+      if (accWrite) {
         // Trace back to the load or transfer_read operations of the contract
         // accumulators.
         Operation *accReadOp0 =
@@ -373,18 +373,22 @@ struct VectorContractToPackedTypeDotProduct
           return rewriter.notifyMatchFailure(
               contractOp, "The store/write operation of contract operation is "
                           "before the pair contract operation");
-        // Shuffle the accumulators of the contract operations.
-        LogicalResult readShuffle =
-            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                   pairContractOp, nonUnitDimValue, accTy);
 
-        if (failed(readShuffle))
-          return rewriter.notifyMatchFailure(
-              contractOp, "Accumulator read is not by transfer_read or load");
+        if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+          // Shuffle the accumulators of the contract operations.
+          LogicalResult readShuffle = shuffleAfterReadLikeOp(
+              rewriter, accReadOp0, accReadOp1, contractOp, pairContractOp,
+              nonUnitDimValue, accTy);
+
+          if (failed(readShuffle))
+            return rewriter.notifyMatchFailure(
+                contractOp, "Accumulator read is not by transfer_read or load");
+        }
 
         // Shuffle the output of contract operations before it's use.
         LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-            rewriter, resultWriteOp0, resultWriteOp1, nonUnitDimValue, accTy);
+            rewriter, contractOp.getResult(), pairContractOp.getResult(),
+            nonUnitDimValue, accTy);
 
         if (failed(writeShuffle))
           return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
index a04a026f35ae6..232a912a84661 100644
--- a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
+++ b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
@@ -144,6 +144,25 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
   return {maskLo8, maskHi8};
 }
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v) {
+  if (v.getNumUses() != 1)
+    return nullptr;
+
+  OpOperand &use = *v.use_begin();
+  Operation *user = use.getOwner();
+
+  if (!isa<scf::YieldOp>(user))
+    return v;
+
+  auto yield = cast<scf::YieldOp>(user);
+  Operation *parent = yield->getParentOp();
+  unsigned idx = use.getOperandNumber();
+
+  return contractionUsersAfterYield(parent->getResult(idx));
+}
+
 // This function walks backward from a value to locate its originating
 // vector read-like operation (`vector.transfer_read` or `vector.load`).
 // It follows simple forwarding through unary ops and across `scf.for`
@@ -155,9 +174,21 @@ Operation *traceToVectorReadLikeParentOperation(Value v) {
   while (true) {
     // Case 1: Value defined by an operation
     if (Operation *defOp = v.getDefiningOp()) {
-      if (isa<vector::TransferReadOp, vector::LoadOp>(defOp))
+      if (isa<vector::TransferReadOp, vector::LoadOp, arith::ConstantOp>(defOp))
         return defOp;
 
+      if (isa<arith::ConstantOp>(defOp)) {
+        Attribute value = (dyn_cast<arith::ConstantOp>(defOp)).getValue();
+
+        if ((dyn_cast<IntegerAttr>(value)).getValue().isZero())
+          return defOp;
+
+        if ((dyn_cast<FloatAttr>(value)).getValue().isZero())
+          return defOp;
+
+        return nullptr;
+      }
+
       return nullptr;
     }
 
@@ -289,27 +320,23 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
-  // Helper to extract vector operand from write-like ops
-  auto getWrittenVector = [](Operation *op) -> Value {
-    if (auto write = dyn_cast<vector::TransferWriteOp>(op))
-      return write.getVector();
-    if (auto store = dyn_cast<vector::StoreOp>(op))
-      return store.getValueToStore();
-    return nullptr;
-  };
 
-  Value vecA = getWrittenVector(opA);
-  Value vecB = getWrittenVector(opB);
+  Value vecA = contractionUsersAfterYield(opA);
+  Value vecB = contractionUsersAfterYield(opB);
 
   if (!vecA || !vecB)
     return failure();
 
+  Operation *resultWriteOpA = *vecA.getUsers().begin();
+  Operation *resultWriteOpB = *vecB.getUsers().begin();
+
   // Decide insertion point and location
-  Operation *insertBefore = opA->isBeforeInBlock(opB) ? opA : opB;
+  Operation *insertBefore = resultWriteOpA->isBeforeInBlock(resultWriteOpB)
+                                ? resultWriteOpA
+                                : resultWriteOpB;
 
   rewriter.setInsertionPoint(insertBefore);
   Location loc = insertBefore->getLoc();
@@ -335,10 +362,8 @@ LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
   auto newVecB = vector::ShapeCastOp::create(rewriter, loc, accTy, shuffledHi);
 
   // Update write operands in place via the rewriter to notify it of changes.
-  rewriter.modifyOpInPlace(opA,
-                           [&]() { opA->setOperand(0, newVecA.getResult()); });
-  rewriter.modifyOpInPlace(opB,
-                           [&]() { opB->setOperand(0, newVecB.getResult()); });
+  resultWriteOpA->replaceUsesOfWith(vecA, newVecA);
+  resultWriteOpB->replaceUsesOfWith(vecB, newVecB);
 
   return success();
 }
diff --git a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
new file mode 100644
index 0000000000000..8ddec33b086df
--- /dev/null
+++ b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
@@ -0,0 +1,136 @@
+// RUN: mlir-opt %s -transform-interpreter -cse -split-input-file | FileCheck %s
+
+!vecA = vector<1x1x2xbf16>
+!vecB = vector<1x2x16xbf16>
+!vecC = vector<1x16xf32>
+!memrefA = memref<1x1x2xbf16, strided<[2048, 32, 1], offset: ?>>
+!memrefB = memref<1x2x32xbf16, strided<[2048, 64, 1], offset: ?>>
+!memrefC = memref<1x32xf32, strided<[64, 1], offset: ?>>
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64xbf16>,
+                             %arg2: memref<64x64xf32>) -> memref<64x64xf32> {
+  %0 = ub.poison : f32
+  %1 = ub.poison : bf16
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  scf.for %arg3 = %c0 to %c64 step %c1 {
+    scf.for %arg4 = %c0 to %c64 step %c32 {
+      %subview = memref.subview %arg2[%arg3, %arg4] [1, 32] [1, 1]
+                        : memref<64x64xf32> to !memrefC
+      %2 = vector.transfer_read %subview[%c0, %c0], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+      %3 = vector.transfer_read %subview[%c0, %c16], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+
+      %4:2 = scf.for %arg5 = %c0 to %c16 step %c1 iter_args(%arg6 = %2, %arg7 = %3) -> (!vecC, !vecC) {
+        %5:2 = scf.for %arg8 = %c0 to %c32 step %c2 iter_args(%arg9 = %arg6, %arg10 = %arg7) -> (!vecC, !vecC) {
+
+          %subview_0 = memref.subview %arg0[%arg5, %arg3, %arg8] [1, 1, 2] [1, 1, 1]
+                        : memref<16x64x32xbf16> to !memrefA
+          %subview_1 = memref.subview %arg1[%arg5, %arg8, %arg4] [1, 2, 32] [1, 1, 1]
+                        : memref<16x32x64xbf16> to !memrefB
+
+          %6 = vector.transfer_read %subview_0[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefA, !vecA
+          %7 = vector.transfer_read %subview_1[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+          %8 = vector.transfer_read %subview_1[%c0, %c0, %c16], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+
+          %9 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %7, %arg9
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+          %10 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %8, %arg10
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+
+          scf.yield %9, %10 : !vecC, !vecC
+        }
+        scf.yield %5#0, %5#1 : !vecC, !vecC
+      }
+
+      vector.transfer_write %4#1, %subview[%c0, %c16] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+      vector.transfer_write %4#0, %subview[%c0, %c0] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+    }
+  }
+
+  return %arg2 : memref<64x64xf32>
+}
+
+// CHECK-LABEL: @brmatmul_acc_mv
+// CHECK: arith.constant dense<0.000000e+00> : vector<1x16xf32>
+// CHECK: arith.addf
+// CHECK-NEXT: vector.transfer_write
+// CHECK: arith.addf
+// CHECK-NEXT: vector.transfer_write
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.move_accumulator_for_contract_loop
+    } : !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+!vecA = vector<1x16x16x2xbf16>
+!vecB = vector<1x16x16x2xbf16>
+!vecC = vector<16x16xf32>
+!memrefA = memref<1x32x16x2xbf16>
+!memrefB = memref<1x16x32x2xbf16>
+!memrefC = memref<32x32xf32>
+#map = affine_map<(d0, d4, d1, d2, d3) -> (d0, d1, d3, d4)>
+#map1 = affine_map<(d0, d4, d1, d2, d3) -> (d0, d3, d2, d4)>
+#map2 = affine_map<(d0, d4, d1, d2, d3) -> (d1, d2)>
+func.func @negative_no_loop(
+  %arg0: !memrefA, %arg1: !memrefB, %arg2: !memrefC) -> !memrefC
+{
+  %c0 = arith.constant 0 : index
+  %0 = ub.poison : bf16
+  %32 = ub.poison : f32
+
+  %1 = vector.transfer_read %arg0[%c0, %c0, %c0, %c0], %0 {in_bounds = [true, true, true, true]} :
+        !memrefA, !vecA
+  %2 = vector.transfer_read %arg1[%c0, %c0, %c0, %c0], %0 {in_bounds = [true, true, true, true]} :
+        !memrefB, !vecB
+
+  %3 = vector.transfer_read %arg2[%c0, %c0], %32 {in_bounds = [true, true]} : !memrefC, !vecC
+
+  %4 = vector.contract {
+    indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["reduction", "reduction", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>}
+    %1, %2, %3 : !vecA, !vecB into !vecC
+
+  vector.transfer_write %4, %arg2[%c0, %c0] {in_bounds = [true, true]} : !vecC, !memrefC
+
+  return %arg2 : !memrefC
+}
+
+// CHECK-LABEL: @negative_no_loop
+// CHECK-NOT: arith.constant dense<0.000000e+00> {{.*}}
+// CHECK-NOT: arith.addf
+// CHECK-NOT: arith.addf
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.move_accumulator_for_contract_loop
+    } : !transform.any_op
+    transform.yield
+  }
+}
+

>From 63ea2855e6bb37d179f13598c973cd5e83375d71 Mon Sep 17 00:00:00 2001
From: Arun Thangamani <arun.thangamani at intel.com>
Date: Thu, 23 Jul 2026 00:13:56 -0700
Subject: [PATCH 2/2] fix typos, expand test-case

---
 .../include/mlir/Dialect/X86/Utils/X86Utils.h |  4 ++--
 .../MoveAccumulatorForContractLoop.cpp        | 11 ++++++-----
 mlir/lib/Dialect/X86/Utils/X86Utils.cpp       | 10 +++++-----
 .../X86/move-acc-for-contract-loop.mlir       | 19 ++++++++++++++-----
 4 files changed, 27 insertions(+), 17 deletions(-)

diff --git a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
index f067ce3bb9b65..edc17e08cde7c 100644
--- a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
+++ b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
@@ -63,8 +63,8 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
-                                       Value opB, int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value valA,
+                                       Value valB, int64_t nonUnitDimAcc,
                                        VectorType accTy);
 
 } // namespace x86
diff --git a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
index 6f87ebfa441dc..3c15c680e65e6 100644
--- a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
@@ -58,8 +58,7 @@ struct MoveAccumulatorForContractLoop
     if (dyn_cast<arith::ConstantOp>(accReadOp))
       return rewriter.notifyMatchFailure(
           contractOp,
-          "The input acc to contract is already a constant zero vector.");
-    ;
+          "The input acc to contract is already a constant vector.");
 
     if ((accReadOp->getBlock() == contractOp->getBlock()) ||
         (resultWriteOp->getBlock() == contractOp->getBlock()))
@@ -67,11 +66,13 @@ struct MoveAccumulatorForContractLoop
           contractOp, "Acc read/write should be in a separate block.");
 
     // Replace acc of a contraction operation with vector constant.
-    rewriter.setInsertionPointAfter(accReadOp);
     Value accValue = accReadOp->getResult(0);
+    Operation *firstUser = *accValue.getUsers().begin();
+    rewriter.setInsertionPoint(firstUser);
+
     auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
     if (!vecTy)
-      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+      return rewriter.notifyMatchFailure(contractOp, "Expects vector type.");
 
     Location loc = accReadOp->getLoc();
     Type elemTy = vecTy.getElementType();
@@ -84,7 +85,7 @@ struct MoveAccumulatorForContractLoop
 
     accValue.replaceAllUsesWith(zeroVec);
 
-    // Adds the initial acc value with acontract results before storing to acc
+    // Adds the initial acc value with contract results before storing to acc
     // matrix.
     rewriter.setInsertionPoint(resultWriteOp);
     Location locUser = resultWriteOp->getLoc();
diff --git a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
index 232a912a84661..bd3f0d5244f2e 100644
--- a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
+++ b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
@@ -147,7 +147,7 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
 // Recursively follows single-use values through scf.yield operations
 // and returns the first non-yield user result in the contraction chain.
 Value contractionUsersAfterYield(Value v) {
-  if (v.getNumUses() != 1)
+  if (!v || v.getNumUses() != 1)
     return nullptr;
 
   OpOperand &use = *v.use_begin();
@@ -320,12 +320,12 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
-                                       Value opB, int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value valA,
+                                       Value valB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
 
-  Value vecA = contractionUsersAfterYield(opA);
-  Value vecB = contractionUsersAfterYield(opB);
+  Value vecA = contractionUsersAfterYield(valA);
+  Value vecB = contractionUsersAfterYield(valB);
 
   if (!vecA || !vecB)
     return failure();
diff --git a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
index 8ddec33b086df..547b6a591c1d6 100644
--- a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
+++ b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
@@ -67,11 +67,20 @@ func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64x
 }
 
 // CHECK-LABEL: @brmatmul_acc_mv
-// CHECK: arith.constant dense<0.000000e+00> : vector<1x16xf32>
-// CHECK: arith.addf
-// CHECK-NEXT: vector.transfer_write
-// CHECK: arith.addf
-// CHECK-NEXT: vector.transfer_write
+// CHECK: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1x16xf32>
+// CHECK: scf.for
+// CHECK-NEXT: scf.for
+// CHECK: memref.subview
+// CHECK: %[[TRANSFER_READ_0:.*]] = vector.transfer_read
+// CHECK: %[[TRANSFER_READ_1:.*]] = vector.transfer_read
+// CHECK: scf.for {{.*}} iter_args(%[[VAL_3:.*]] = %[[CST]], %[[VAL_4:.*]] = %[[CST]]) -> (vector<1x16xf32>, vector<1x16xf32>) {
+// CHECK-NEXT: scf.for
+// CHECK: scf.yield
+// CHECK: scf.yield
+// CHECK: %[[ADDF_0:.*]] = arith.addf {{.*}}, %[[TRANSFER_READ_1]] : vector<1x16xf32>
+// CHECK-NEXT: vector.transfer_write %[[ADDF_0:.*]], %subview[%c0, %c16] {{.*}}
+// CHECK: %[[ADDF_1:.*]] = arith.addf {{.*}}, %[[TRANSFER_READ_0]] : vector<1x16xf32>
+// CHECK-NEXT: vector.transfer_write %[[ADDF_1:.*]], %subview[%c0, %c0] {{.*}}
 
 module attributes {transform.with_named_sequence} {
   transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {


From llvmlistbot at llvm.org  Thu Jul 23 00:46:05 2026
From: llvmlistbot at llvm.org (Felipe de Azevedo Piovezan)
Date: Thu, 23 Jul 2026 00:46:05 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a61c6bd.6c8668f2.1842d3.5af3@mx.google.com>


felipepiovezan wrote:

> Also note that if this PR were to go forward, it would need to be broken up into smaller components: maybe just the IR component, then the MIR component, etc. It's also missing a producer for these (part of what I meant by "motivation").
> 
> I strongly recommend you write an RFC on the forums before going forward with this (if there is one already, please link it on the PR). For example, one question that comes to mind: why not re-use the existing OPs for this? If they are unused in their current form because it's impractical, can't we change them to use labels in the IR layer?

Since there seems to have been a stream of accepts here, and my post was missed, I wanted to make sure these questions were not lost:

1. Should we have had an RFC for this? It seems you have even more upcoming work, which would be nice to discuss beforehand? I suspect some of these were talked about in a round table last dev meeting, but still. An RFC would make clear the answer to 2 below, what the upcoming work is, whether we should expect an in-tree producer for these, etc. These answers would help evaluate whether this belongs in-tree.
2. Why not change the implementation of the existing attributes -- they seem largely unusable as they are -- instead of adding new ones? The end result of "offsets that are integers" are still compliant with the spec, the intermediate representation is irrelevant for this purpose.
3. You said that "About the only other patch split I could come up with was boilerplate and emission", what do you think of the split I suggested above? Separate IR and CodeGen patches are a very natural split.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Thu Jul 23 00:46:39 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:46:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c6df.492e23e4.292f20.46bb@mx.google.com>


================
@@ -144,6 +144,25 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
   return {maskLo8, maskHi8};
 }
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v) {
+  if (v.getNumUses() != 1)
+    return nullptr;
+
+  OpOperand &use = *v.use_begin();
+  Operation *user = use.getOwner();
+
+  if (!isa<scf::YieldOp>(user))
+    return v;
----------------
arun-thmn wrote:

Okay, Added an extra check to return `nullptr`, if `v` is `null`.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 00:47:02 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:47:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c6f6.507fc713.3702e2.a5a3@mx.google.com>


================
@@ -59,9 +63,9 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc, VectorType accTy);
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
----------------
arun-thmn wrote:

True, renamed those arg values.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 00:47:08 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:47:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c6fc.0c05e68e.81651.3c32@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
----------------
arun-thmn wrote:

Removed.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 00:47:29 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:47:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c711.83ce629a.2277d7.fff2@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
----------------
arun-thmn wrote:

Okay, did that.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 00:48:01 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 00:48:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61c731.1b0626d0.1ff4c3.7fbe@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
----------------
arun-thmn wrote:

True. Corrected the error msg.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 00:50:53 2026
From: llvmlistbot at llvm.org (Felipe de Azevedo Piovezan)
Date: Thu, 23 Jul 2026 00:50:53 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a61c7dd.0c05e68e.81651.bf0a@mx.google.com>


felipepiovezan wrote:

> hey seem largely unusable as they are

Also worth pointing out that _any_ existing use today can be rewritten in terms of labels. This indicates we don't need any new DW_OPs

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Thu Jul 23 01:06:29 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Thu, 23 Jul 2026 01:06:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a61cb85.0c05e68e.81651.07d1@mx.google.com>


================
@@ -1,4 +1,5 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics -mlir-print-ir-after-all
----------------
joker-eph wrote:

Leftover debug here

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Thu Jul 23 01:06:49 2026
From: llvmlistbot at llvm.org (Mehdi Amini)
Date: Thu, 23 Jul 2026 01:06:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a61cb99.8e75e0cb.2de827.f055@mx.google.com>


joker-eph wrote:

LG but you need to update the title and description

https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Thu Jul 23 01:14:59 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:14:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd83.c5504292.1bbfc8.0e5f@mx.google.com>


https://github.com/adam-smnk edited https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:14:59 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:14:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd83.507fc713.3702e2.9442@mx.google.com>


https://github.com/adam-smnk commented:

Not expert on scalable logic but overall approach seems reasonable.
Mostly a bit more polish is needed.

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:00 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd84.d76d81af.3b68ca.00ca@mx.google.com>


================
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
----------------
adam-smnk wrote:

```suggestion
        linalgOp, "scalable block factors require padding");
```

nit: more generic phrasing

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:00 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd84.6c8668f2.1842d3.39a2@mx.google.com>


================
@@ -0,0 +1,91 @@
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=0" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE-NOPAD
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=32,16,[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=MIXED
+
+// -----
+
+// All-scalable block factors, static input shapes, allow-padding=1.
+// Inner tile sizes and outer dimensions are all dynamic (vscale-relative).
+
+func.func @block_matmul_scalable_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
----------------
adam-smnk wrote:

I'd expect to see at least one `vscale` somewhere.
You could add `-cse` to make checks more readable as well.

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:00 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd84.92d2873c.2cdd42.5ff0@mx.google.com>


================

----------------
adam-smnk wrote:

nit: I think it'd make sense to add all three check variants to all test cases here

Also, maybe one more op variant like `batch_matmul` for completness

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:01 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd85.13be6df8.323b61.56ec@mx.google.com>


================
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
+
+  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
+  // arith.constant N : index multiplied by vector.vscale.
----------------
adam-smnk wrote:

nit: I can see that from the code, I'd generalize or skip the comment

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:03 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd87.13be6df8.323b61.5a3b@mx.google.com>


================
@@ -1423,6 +1423,11 @@ struct BlockPackMatmulOptions {
   /// the parallel dimensions and kb is the reduction dimension.
   SmallVector<int64_t, 3> blockFactors;
 
+  /// Scalable flags for block factors. When true, the corresponding block
+  /// factor is multiplied by vector.vscale at runtime (SVE-style scalable).
----------------
adam-smnk wrote:

nit: I think calling out `vector.vscale` explicitly is too implementation specific, I'd generalize the comment

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:06 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 01:15:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61cd8a.d76d81af.3b68ca.0597@mx.google.com>


================
@@ -289,27 +320,23 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
-  // Helper to extract vector operand from write-like ops
-  auto getWrittenVector = [](Operation *op) -> Value {
-    if (auto write = dyn_cast<vector::TransferWriteOp>(op))
-      return write.getVector();
-    if (auto store = dyn_cast<vector::StoreOp>(op))
-      return store.getValueToStore();
-    return nullptr;
-  };
 
-  Value vecA = getWrittenVector(opA);
-  Value vecB = getWrittenVector(opB);
+  Value vecA = contractionUsersAfterYield(opA);
----------------
arun-thmn wrote:

The old behaviour should also be supported by nano-kernels, if this rewrite is not happening.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 01:15:13 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd91.13be6df8.323b61.686c@mx.google.com>


================
@@ -0,0 +1,91 @@
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=0" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE-NOPAD
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=32,16,[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=MIXED
+
+// -----
----------------
adam-smnk wrote:

nit: not needed

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:15:26 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:15:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61cd9e.510f576d.33aa7d.a10c@mx.google.com>


================
@@ -300,7 +329,28 @@ struct LinalgBlockPackMatmul
     ControlBlockPackMatmulFn controlFn =
         [&](linalg::LinalgOp op) -> BlockPackMatmulOptions {
       BlockPackMatmulOptions options;
-      options.blockFactors = SmallVector<int64_t>{*blockFactors};
+
+      // Parse block-factors strings. Each element is either "N" (static) or
+      // "[N]" (scalable, i.e. N * vscale at runtime).
+      for (const std::string &f : *blockFactors) {
+        StringRef s(f);
----------------
adam-smnk wrote:

nit: I'd suggest more expressive variable names

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:20:22 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 01:20:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61cec6.c4d3d80a.16dae5.13b3@mx.google.com>


https://github.com/arun-thmn updated https://github.com/llvm/llvm-project/pull/204327

>From 23c66ee4340af485894825ac68c082ea2c28b07e Mon Sep 17 00:00:00 2001
From: Arun Thangamani <arun.thangamani at intel.com>
Date: Wed, 17 Jun 2026 02:48:59 -0700
Subject: [PATCH 1/2] does vector.contract(A, B, Acc) into vector.contract(A,
 B, 0) + Acc

---
 .../X86/TransformOps/X86TransformOps.td       |  10 ++
 mlir/include/mlir/Dialect/X86/Transforms.h    |   5 +
 .../include/mlir/Dialect/X86/Utils/X86Utils.h |  10 +-
 .../X86/TransformOps/X86TransformOps.cpp      |   5 +
 .../lib/Dialect/X86/Transforms/CMakeLists.txt |   1 +
 .../MoveAccumulatorForContractLoop.cpp        | 114 +++++++++++++++
 .../Transforms/VectorContractBF16ToFMA.cpp    |  28 ++--
 .../VectorContractToPackedTypeDotProduct.cpp  |  30 ++--
 mlir/lib/Dialect/X86/Utils/X86Utils.cpp       |  63 +++++---
 .../X86/move-acc-for-contract-loop.mlir       | 136 ++++++++++++++++++
 10 files changed, 351 insertions(+), 51 deletions(-)
 create mode 100644 mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
 create mode 100644 mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir

diff --git a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
index c474cfb47d003..48aabf90914e2 100644
--- a/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
+++ b/mlir/include/mlir/Dialect/X86/TransformOps/X86TransformOps.td
@@ -82,5 +82,15 @@ def ApplyVectorContractToAMXDotProductPatternsOp : Op<Transform_Dialect,
   let assemblyFormat = "attr-dict";
 }
 
+def ApplyMoveAccumulatorForContractLoopPatternsOp : Op<Transform_Dialect,
+    "apply_patterns.x86.move_accumulator_for_contract_loop",
+    [DeclareOpInterfaceMethods<PatternDescriptorOpInterface>]> {
+  let description = [{
+         Rewrites an additive vector contraction by zeroing the accumulator input
+         and materializing the accumulation as a separate add after the contraction.
+  }];
+
+  let assemblyFormat = "attr-dict";
+}
 
 #endif // X86_TRANSFORM_OPS
diff --git a/mlir/include/mlir/Dialect/X86/Transforms.h b/mlir/include/mlir/Dialect/X86/Transforms.h
index 6ebba5e94ec7c..8bcaf9a67feb0 100644
--- a/mlir/include/mlir/Dialect/X86/Transforms.h
+++ b/mlir/include/mlir/Dialect/X86/Transforms.h
@@ -110,6 +110,11 @@ void populateShuffleVectorFMAOpsPatterns(RewritePatternSet &patterns);
 // Int8).
 void populateVectorContractToAMXDotProductPatterns(RewritePatternSet &patterns);
 
+// Rewrites an additive vector contraction by zeroing the accumulator input
+// and materializing the accumulation as a separate add after the contraction.
+void populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns);
+
 //===----------------------------------------------------------------------===//
 /// Helpers extracted from:
 ///   - clang/lib/Headers/avxintrin.h
diff --git a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
index 2ff8f4f4283a2..f067ce3bb9b65 100644
--- a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
+++ b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
@@ -29,6 +29,10 @@ namespace x86 {
 bool isInVnniLayout(Operation *op, llvm::ArrayRef<AffineMap> indexingMaps,
                     std::optional<unsigned> blockingFactor = std::nullopt);
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v);
+
 // Returns true if two contraction ops form a valid pair for VNNI packing.
 // It verifies that both contractions share the appropriate operand, read from
 // the same source buffer, and use constant indices that differ by 8 or 16.
@@ -59,9 +63,9 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc, VectorType accTy);
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
+                                       VectorType accTy);
 
 } // namespace x86
 } // namespace mlir
diff --git a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
index 390b21e12b0ed..a52f08a5cd061 100644
--- a/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
+++ b/mlir/lib/Dialect/X86/TransformOps/X86TransformOps.cpp
@@ -52,6 +52,11 @@ void mlir::transform::ApplyVectorContractToAMXDotProductPatternsOp::
   x86::populateVectorContractToAMXDotProductPatterns(patterns);
 }
 
+void mlir::transform::ApplyMoveAccumulatorForContractLoopPatternsOp::
+    populatePatterns(RewritePatternSet &patterns) {
+  x86::populateMoveAccumulatorForContractLoopPatterns(patterns);
+}
+
 //===----------------------------------------------------------------------===//
 // Transform op registration
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
index 9c3695536cda9..5118563c03f16 100644
--- a/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/X86/Transforms/CMakeLists.txt
@@ -7,6 +7,7 @@ add_mlir_dialect_library(MLIRX86Transforms
   SinkVectorProducerOps.cpp
   ShuffleVectorFMAOps.cpp
   VectorContractToAMXDotProduct.cpp
+  MoveAccumulatorForContractLoop.cpp
 
   LINK_LIBS PUBLIC
   MLIRArithDialect
diff --git a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
new file mode 100644
index 0000000000000..6f87ebfa441dc
--- /dev/null
+++ b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
+    Value accValue = accReadOp->getResult(0);
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with acontract results before storing to acc
+    // matrix.
+    rewriter.setInsertionPoint(resultWriteOp);
+    Location locUser = resultWriteOp->getLoc();
+
+    Value addition;
+
+    if (llvm::isa<FloatType>(elemTy)) {
+      addition =
+          arith::AddFOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    if (llvm::isa<IntegerType>(elemTy)) {
+      addition =
+          arith::AddIOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    resultWriteOp->replaceUsesOfWith(contractValue, addition);
+    return success();
+  }
+};
+
+} // namespace
+
+void x86::populateMoveAccumulatorForContractLoopPatterns(
+    RewritePatternSet &patterns) {
+  patterns.add<MoveAccumulatorForContractLoop>(patterns.getContext());
+}
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
index 287892c3a660f..c70ad221c772d 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractBF16ToFMA.cpp
@@ -415,25 +415,21 @@ struct VectorContractBF16ToFMA
       Operation *accReadOp1 =
           traceToVectorReadLikeParentOperation(pairContractOp.getAcc());
 
-      // Iterate down to find the users of contact operations until it is store
-      // or transfer_write.
-      Operation *resultWriteOp0 =
-          traceToVectorWriteLikeUserOperation(contractOp.getResult());
-      Operation *resultWriteOp1 =
-          traceToVectorWriteLikeUserOperation(pairContractOp.getResult());
-
-      // Shuffle the accumulators of the contract operations.
-      LogicalResult readShuffle =
-          shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                 pairContractOp, nonUnitDim, accTy);
-
-      if (failed(readShuffle))
-        return rewriter.notifyMatchFailure(
-            contractOp, "Accumulator read is not by transfer_read or load");
+      if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+        // Shuffle the accumulators of the contract operations.
+        LogicalResult readShuffle =
+            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
+                                   pairContractOp, nonUnitDim, accTy);
+
+        if (failed(readShuffle))
+          return rewriter.notifyMatchFailure(
+              contractOp, "Accumulator read is not by transfer_read or load");
+      }
 
       // Shuffle the output of contract operations before its use.
       LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-          rewriter, resultWriteOp0, resultWriteOp1, nonUnitDim, accTy);
+          rewriter, contractOp.getResult(), pairContractOp.getResult(),
+          nonUnitDim, accTy);
 
       if (failed(writeShuffle))
         return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
index d3487369355f0..394c950189ead 100644
--- a/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/VectorContractToPackedTypeDotProduct.cpp
@@ -325,17 +325,17 @@ struct VectorContractToPackedTypeDotProduct
       }
 
       // If the accumulators are shuffled we get nullptr else the
-      // transfer_read or load operations.
-      Operation *accRead =
-          traceToVectorReadLikeParentOperation(contractOp.getAcc());
+      // transfer_load or store operations.
+      Operation *accWrite =
+          traceToVectorWriteLikeUserOperation(contractOp.getResult());
 
       if (!pairContractOp &&
-          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accRead))
+          (!isNonUnitDimOperandShuffled(nonUnitDimOperand) || accWrite))
         return rewriter.notifyMatchFailure(contractOp,
                                            "Could not find a contract pair");
 
       // Validate and shuffle the accumulator
-      if (accRead) {
+      if (accWrite) {
         // Trace back to the load or transfer_read operations of the contract
         // accumulators.
         Operation *accReadOp0 =
@@ -373,18 +373,22 @@ struct VectorContractToPackedTypeDotProduct
           return rewriter.notifyMatchFailure(
               contractOp, "The store/write operation of contract operation is "
                           "before the pair contract operation");
-        // Shuffle the accumulators of the contract operations.
-        LogicalResult readShuffle =
-            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                   pairContractOp, nonUnitDimValue, accTy);
 
-        if (failed(readShuffle))
-          return rewriter.notifyMatchFailure(
-              contractOp, "Accumulator read is not by transfer_read or load");
+        if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
+          // Shuffle the accumulators of the contract operations.
+          LogicalResult readShuffle = shuffleAfterReadLikeOp(
+              rewriter, accReadOp0, accReadOp1, contractOp, pairContractOp,
+              nonUnitDimValue, accTy);
+
+          if (failed(readShuffle))
+            return rewriter.notifyMatchFailure(
+                contractOp, "Accumulator read is not by transfer_read or load");
+        }
 
         // Shuffle the output of contract operations before it's use.
         LogicalResult writeShuffle = shuffleBeforeWriteLikeOp(
-            rewriter, resultWriteOp0, resultWriteOp1, nonUnitDimValue, accTy);
+            rewriter, contractOp.getResult(), pairContractOp.getResult(),
+            nonUnitDimValue, accTy);
 
         if (failed(writeShuffle))
           return rewriter.notifyMatchFailure(
diff --git a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
index a04a026f35ae6..232a912a84661 100644
--- a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
+++ b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
@@ -144,6 +144,25 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
   return {maskLo8, maskHi8};
 }
 
+// Recursively follows single-use values through scf.yield operations
+// and returns the first non-yield user result in the contraction chain.
+Value contractionUsersAfterYield(Value v) {
+  if (v.getNumUses() != 1)
+    return nullptr;
+
+  OpOperand &use = *v.use_begin();
+  Operation *user = use.getOwner();
+
+  if (!isa<scf::YieldOp>(user))
+    return v;
+
+  auto yield = cast<scf::YieldOp>(user);
+  Operation *parent = yield->getParentOp();
+  unsigned idx = use.getOperandNumber();
+
+  return contractionUsersAfterYield(parent->getResult(idx));
+}
+
 // This function walks backward from a value to locate its originating
 // vector read-like operation (`vector.transfer_read` or `vector.load`).
 // It follows simple forwarding through unary ops and across `scf.for`
@@ -155,9 +174,21 @@ Operation *traceToVectorReadLikeParentOperation(Value v) {
   while (true) {
     // Case 1: Value defined by an operation
     if (Operation *defOp = v.getDefiningOp()) {
-      if (isa<vector::TransferReadOp, vector::LoadOp>(defOp))
+      if (isa<vector::TransferReadOp, vector::LoadOp, arith::ConstantOp>(defOp))
         return defOp;
 
+      if (isa<arith::ConstantOp>(defOp)) {
+        Attribute value = (dyn_cast<arith::ConstantOp>(defOp)).getValue();
+
+        if ((dyn_cast<IntegerAttr>(value)).getValue().isZero())
+          return defOp;
+
+        if ((dyn_cast<FloatAttr>(value)).getValue().isZero())
+          return defOp;
+
+        return nullptr;
+      }
+
       return nullptr;
     }
 
@@ -289,27 +320,23 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
-                                       Operation *opA, Operation *opB,
-                                       int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
+                                       Value opB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
-  // Helper to extract vector operand from write-like ops
-  auto getWrittenVector = [](Operation *op) -> Value {
-    if (auto write = dyn_cast<vector::TransferWriteOp>(op))
-      return write.getVector();
-    if (auto store = dyn_cast<vector::StoreOp>(op))
-      return store.getValueToStore();
-    return nullptr;
-  };
 
-  Value vecA = getWrittenVector(opA);
-  Value vecB = getWrittenVector(opB);
+  Value vecA = contractionUsersAfterYield(opA);
+  Value vecB = contractionUsersAfterYield(opB);
 
   if (!vecA || !vecB)
     return failure();
 
+  Operation *resultWriteOpA = *vecA.getUsers().begin();
+  Operation *resultWriteOpB = *vecB.getUsers().begin();
+
   // Decide insertion point and location
-  Operation *insertBefore = opA->isBeforeInBlock(opB) ? opA : opB;
+  Operation *insertBefore = resultWriteOpA->isBeforeInBlock(resultWriteOpB)
+                                ? resultWriteOpA
+                                : resultWriteOpB;
 
   rewriter.setInsertionPoint(insertBefore);
   Location loc = insertBefore->getLoc();
@@ -335,10 +362,8 @@ LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter,
   auto newVecB = vector::ShapeCastOp::create(rewriter, loc, accTy, shuffledHi);
 
   // Update write operands in place via the rewriter to notify it of changes.
-  rewriter.modifyOpInPlace(opA,
-                           [&]() { opA->setOperand(0, newVecA.getResult()); });
-  rewriter.modifyOpInPlace(opB,
-                           [&]() { opB->setOperand(0, newVecB.getResult()); });
+  resultWriteOpA->replaceUsesOfWith(vecA, newVecA);
+  resultWriteOpB->replaceUsesOfWith(vecB, newVecB);
 
   return success();
 }
diff --git a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
new file mode 100644
index 0000000000000..8ddec33b086df
--- /dev/null
+++ b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
@@ -0,0 +1,136 @@
+// RUN: mlir-opt %s -transform-interpreter -cse -split-input-file | FileCheck %s
+
+!vecA = vector<1x1x2xbf16>
+!vecB = vector<1x2x16xbf16>
+!vecC = vector<1x16xf32>
+!memrefA = memref<1x1x2xbf16, strided<[2048, 32, 1], offset: ?>>
+!memrefB = memref<1x2x32xbf16, strided<[2048, 64, 1], offset: ?>>
+!memrefC = memref<1x32xf32, strided<[64, 1], offset: ?>>
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+
+func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64xbf16>,
+                             %arg2: memref<64x64xf32>) -> memref<64x64xf32> {
+  %0 = ub.poison : f32
+  %1 = ub.poison : bf16
+  %c0 = arith.constant 0 : index
+  %c64 = arith.constant 64 : index
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  scf.for %arg3 = %c0 to %c64 step %c1 {
+    scf.for %arg4 = %c0 to %c64 step %c32 {
+      %subview = memref.subview %arg2[%arg3, %arg4] [1, 32] [1, 1]
+                        : memref<64x64xf32> to !memrefC
+      %2 = vector.transfer_read %subview[%c0, %c0], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+      %3 = vector.transfer_read %subview[%c0, %c16], %0 {in_bounds = [true, true]}
+                        : !memrefC, !vecC
+
+      %4:2 = scf.for %arg5 = %c0 to %c16 step %c1 iter_args(%arg6 = %2, %arg7 = %3) -> (!vecC, !vecC) {
+        %5:2 = scf.for %arg8 = %c0 to %c32 step %c2 iter_args(%arg9 = %arg6, %arg10 = %arg7) -> (!vecC, !vecC) {
+
+          %subview_0 = memref.subview %arg0[%arg5, %arg3, %arg8] [1, 1, 2] [1, 1, 1]
+                        : memref<16x64x32xbf16> to !memrefA
+          %subview_1 = memref.subview %arg1[%arg5, %arg8, %arg4] [1, 2, 32] [1, 1, 1]
+                        : memref<16x32x64xbf16> to !memrefB
+
+          %6 = vector.transfer_read %subview_0[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefA, !vecA
+          %7 = vector.transfer_read %subview_1[%c0, %c0, %c0], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+          %8 = vector.transfer_read %subview_1[%c0, %c0, %c16], %1 {in_bounds = [true, true, true]}
+                        : !memrefB, !vecB
+
+          %9 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %7, %arg9
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+          %10 = vector.contract {indexing_maps = [#map, #map1, #map2], iterator_types =
+                        ["reduction", "parallel", "parallel", "reduction"], kind = #vector.kind<add>} %6, %8, %arg10
+                        {unroll_shape = array<i64: 1, 1, 16, 2>} : !vecA, !vecB into !vecC
+
+          scf.yield %9, %10 : !vecC, !vecC
+        }
+        scf.yield %5#0, %5#1 : !vecC, !vecC
+      }
+
+      vector.transfer_write %4#1, %subview[%c0, %c16] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+      vector.transfer_write %4#0, %subview[%c0, %c0] {in_bounds = [true, true]}
+                        : !vecC, !memrefC
+    }
+  }
+
+  return %arg2 : memref<64x64xf32>
+}
+
+// CHECK-LABEL: @brmatmul_acc_mv
+// CHECK: arith.constant dense<0.000000e+00> : vector<1x16xf32>
+// CHECK: arith.addf
+// CHECK-NEXT: vector.transfer_write
+// CHECK: arith.addf
+// CHECK-NEXT: vector.transfer_write
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.move_accumulator_for_contract_loop
+    } : !transform.any_op
+    transform.yield
+  }
+}
+
+// -----
+
+!vecA = vector<1x16x16x2xbf16>
+!vecB = vector<1x16x16x2xbf16>
+!vecC = vector<16x16xf32>
+!memrefA = memref<1x32x16x2xbf16>
+!memrefB = memref<1x16x32x2xbf16>
+!memrefC = memref<32x32xf32>
+#map = affine_map<(d0, d4, d1, d2, d3) -> (d0, d1, d3, d4)>
+#map1 = affine_map<(d0, d4, d1, d2, d3) -> (d0, d3, d2, d4)>
+#map2 = affine_map<(d0, d4, d1, d2, d3) -> (d1, d2)>
+func.func @negative_no_loop(
+  %arg0: !memrefA, %arg1: !memrefB, %arg2: !memrefC) -> !memrefC
+{
+  %c0 = arith.constant 0 : index
+  %0 = ub.poison : bf16
+  %32 = ub.poison : f32
+
+  %1 = vector.transfer_read %arg0[%c0, %c0, %c0, %c0], %0 {in_bounds = [true, true, true, true]} :
+        !memrefA, !vecA
+  %2 = vector.transfer_read %arg1[%c0, %c0, %c0, %c0], %0 {in_bounds = [true, true, true, true]} :
+        !memrefB, !vecB
+
+  %3 = vector.transfer_read %arg2[%c0, %c0], %32 {in_bounds = [true, true]} : !memrefC, !vecC
+
+  %4 = vector.contract {
+    indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["reduction", "reduction", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>}
+    %1, %2, %3 : !vecA, !vecB into !vecC
+
+  vector.transfer_write %4, %arg2[%c0, %c0] {in_bounds = [true, true]} : !vecC, !memrefC
+
+  return %arg2 : !memrefC
+}
+
+// CHECK-LABEL: @negative_no_loop
+// CHECK-NOT: arith.constant dense<0.000000e+00> {{.*}}
+// CHECK-NOT: arith.addf
+// CHECK-NOT: arith.addf
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {
+    %func = transform.structured.match ops{["func.func"]} in %arg1 : (!transform.any_op) -> !transform.any_op
+    transform.apply_patterns to %func {
+      transform.apply_patterns.x86.move_accumulator_for_contract_loop
+    } : !transform.any_op
+    transform.yield
+  }
+}
+

>From 63ea2855e6bb37d179f13598c973cd5e83375d71 Mon Sep 17 00:00:00 2001
From: Arun Thangamani <arun.thangamani at intel.com>
Date: Thu, 23 Jul 2026 00:13:56 -0700
Subject: [PATCH 2/2] fix typos, expand test-case

---
 .../include/mlir/Dialect/X86/Utils/X86Utils.h |  4 ++--
 .../MoveAccumulatorForContractLoop.cpp        | 11 ++++++-----
 mlir/lib/Dialect/X86/Utils/X86Utils.cpp       | 10 +++++-----
 .../X86/move-acc-for-contract-loop.mlir       | 19 ++++++++++++++-----
 4 files changed, 27 insertions(+), 17 deletions(-)

diff --git a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
index f067ce3bb9b65..edc17e08cde7c 100644
--- a/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
+++ b/mlir/include/mlir/Dialect/X86/Utils/X86Utils.h
@@ -63,8 +63,8 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // Shuffles vectors produced by vector.contraction ops into a flat layout
 // before they are written to memory.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
-                                       Value opB, int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value valA,
+                                       Value valB, int64_t nonUnitDimAcc,
                                        VectorType accTy);
 
 } // namespace x86
diff --git a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
index 6f87ebfa441dc..3c15c680e65e6 100644
--- a/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
+++ b/mlir/lib/Dialect/X86/Transforms/MoveAccumulatorForContractLoop.cpp
@@ -58,8 +58,7 @@ struct MoveAccumulatorForContractLoop
     if (dyn_cast<arith::ConstantOp>(accReadOp))
       return rewriter.notifyMatchFailure(
           contractOp,
-          "The input acc to contract is already a constant zero vector.");
-    ;
+          "The input acc to contract is already a constant vector.");
 
     if ((accReadOp->getBlock() == contractOp->getBlock()) ||
         (resultWriteOp->getBlock() == contractOp->getBlock()))
@@ -67,11 +66,13 @@ struct MoveAccumulatorForContractLoop
           contractOp, "Acc read/write should be in a separate block.");
 
     // Replace acc of a contraction operation with vector constant.
-    rewriter.setInsertionPointAfter(accReadOp);
     Value accValue = accReadOp->getResult(0);
+    Operation *firstUser = *accValue.getUsers().begin();
+    rewriter.setInsertionPoint(firstUser);
+
     auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
     if (!vecTy)
-      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+      return rewriter.notifyMatchFailure(contractOp, "Expects vector type.");
 
     Location loc = accReadOp->getLoc();
     Type elemTy = vecTy.getElementType();
@@ -84,7 +85,7 @@ struct MoveAccumulatorForContractLoop
 
     accValue.replaceAllUsesWith(zeroVec);
 
-    // Adds the initial acc value with acontract results before storing to acc
+    // Adds the initial acc value with contract results before storing to acc
     // matrix.
     rewriter.setInsertionPoint(resultWriteOp);
     Location locUser = resultWriteOp->getLoc();
diff --git a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
index 232a912a84661..bd3f0d5244f2e 100644
--- a/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
+++ b/mlir/lib/Dialect/X86/Utils/X86Utils.cpp
@@ -147,7 +147,7 @@ inline ShuffleMasks getShuffleMasks(int64_t nonUnitDimAcc, bool isInt8Avx2) {
 // Recursively follows single-use values through scf.yield operations
 // and returns the first non-yield user result in the contraction chain.
 Value contractionUsersAfterYield(Value v) {
-  if (v.getNumUses() != 1)
+  if (!v || v.getNumUses() != 1)
     return nullptr;
 
   OpOperand &use = *v.use_begin();
@@ -320,12 +320,12 @@ LogicalResult shuffleAfterReadLikeOp(PatternRewriter &rewriter, Operation *opA,
 
 // This function shuffles the vectors written by vector.contract operation
 // as a flat layout structure before they are stored.
-LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value opA,
-                                       Value opB, int64_t nonUnitDimAcc,
+LogicalResult shuffleBeforeWriteLikeOp(PatternRewriter &rewriter, Value valA,
+                                       Value valB, int64_t nonUnitDimAcc,
                                        VectorType accTy) {
 
-  Value vecA = contractionUsersAfterYield(opA);
-  Value vecB = contractionUsersAfterYield(opB);
+  Value vecA = contractionUsersAfterYield(valA);
+  Value vecB = contractionUsersAfterYield(valB);
 
   if (!vecA || !vecB)
     return failure();
diff --git a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
index 8ddec33b086df..547b6a591c1d6 100644
--- a/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
+++ b/mlir/test/Dialect/X86/move-acc-for-contract-loop.mlir
@@ -67,11 +67,20 @@ func.func @brmatmul_acc_mv(%arg0: memref<16x64x32xbf16>, %arg1: memref<16x32x64x
 }
 
 // CHECK-LABEL: @brmatmul_acc_mv
-// CHECK: arith.constant dense<0.000000e+00> : vector<1x16xf32>
-// CHECK: arith.addf
-// CHECK-NEXT: vector.transfer_write
-// CHECK: arith.addf
-// CHECK-NEXT: vector.transfer_write
+// CHECK: %[[CST:.*]] = arith.constant dense<0.000000e+00> : vector<1x16xf32>
+// CHECK: scf.for
+// CHECK-NEXT: scf.for
+// CHECK: memref.subview
+// CHECK: %[[TRANSFER_READ_0:.*]] = vector.transfer_read
+// CHECK: %[[TRANSFER_READ_1:.*]] = vector.transfer_read
+// CHECK: scf.for {{.*}} iter_args(%[[VAL_3:.*]] = %[[CST]], %[[VAL_4:.*]] = %[[CST]]) -> (vector<1x16xf32>, vector<1x16xf32>) {
+// CHECK-NEXT: scf.for
+// CHECK: scf.yield
+// CHECK: scf.yield
+// CHECK: %[[ADDF_0:.*]] = arith.addf {{.*}}, %[[TRANSFER_READ_1]] : vector<1x16xf32>
+// CHECK-NEXT: vector.transfer_write %[[ADDF_0:.*]], %subview[%c0, %c16] {{.*}}
+// CHECK: %[[ADDF_1:.*]] = arith.addf {{.*}}, %[[TRANSFER_READ_0]] : vector<1x16xf32>
+// CHECK-NEXT: vector.transfer_write %[[ADDF_1:.*]], %subview[%c0, %c0] {{.*}}
 
 module attributes {transform.with_named_sequence} {
   transform.named_sequence @__transform_main(%arg1: !transform.any_op {transform.readonly}) {


From llvmlistbot at llvm.org  Thu Jul 23 01:23:49 2026
From: llvmlistbot at llvm.org (Hamza Qureshi)
Date: Thu, 23 Jul 2026 01:23:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][XeGPU] Fix XeGPUToXeVM crash on
 non-integer memref memory spaces (PR #211053)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211053@github.com>
Message-ID: <6a61cf95.d2cb4adc.23815e.43ed@mx.google.com>


hamzaqureshi5 wrote:

@charithaintc  @Jianhui-Li 

https://github.com/llvm/llvm-project/pull/211053

From llvmlistbot at llvm.org  Thu Jul 23 01:29:15 2026
From: llvmlistbot at llvm.org (Arun Thangamani)
Date: Thu, 23 Jul 2026 01:29:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d0db.bd93e731.2ac69c.2f81@mx.google.com>


================
@@ -0,0 +1,114 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant zero vector.");
+    ;
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    rewriter.setInsertionPointAfter(accReadOp);
+    Value accValue = accReadOp->getResult(0);
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Excepts vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with acontract results before storing to acc
----------------
arun-thmn wrote:

Typo corrected.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 01:47:49 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:47:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a61d535.3cce8983.247e47.b3be@mx.google.com>


================
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
+
+  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
+  // arith.constant N : index multiplied by vector.vscale.
+  SmallVector<OpFoldResult> mnkTiles;
+  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
----------------
adam-smnk wrote:

```suggestion
  for (auto [idx, factor] : llvm::enumerate(options->blockFactors)) {
```

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 01:49:36 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:49:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61d5a0.f06f4994.50b1b.18b7@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
----------------
adam-smnk wrote:

Does it need to be a class attribute?

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 01:50:20 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:50:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61d5cc.d2cb4adc.23815e.a144@mx.google.com>


https://github.com/adam-smnk edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 01:50:35 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:50:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61d5db.1b0626d0.1ff4c3.b3e9@mx.google.com>


https://github.com/adam-smnk commented:

Thanks a lot for the fix 😎

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 01:51:02 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 01:51:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61d5f6.507fc713.3702e2.6ec4@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
adam-smnk wrote:

A general design question.

>From context perspective, loading twice is an error.
However, from end user perspective, do I care about it?
Could the load just do nothing and return?

Alternatively, if you want to keep current approach, could it raise more specific exception like `DialectReloadingError` or `DialectAlreadyLoadedError`?
Then the error can be handled more gracefully on the caller's side.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 01:52:23 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Thu, 23 Jul 2026 01:52:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
Message-ID: <llvm.org/llvm/llvm-project/pull/211485@github.com>


https://github.com/MaskRay created https://github.com/llvm/llvm-project/pull/211485

Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.

Switch to an Euler tour representation: one BlockLayout array per
LoopInfo, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's.

Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test.

Previously a loop's list was the function's reverse postorder restricted
to its members, so a subloop's blocks were interleaved with the parent's
own blocks at their RPO positions. Now a subloop's blocks are contiguous
after its header within every ancestor's list. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated.

AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.

Aided by Claude Fable 5


>From fff8f4d18de1b8d5d6d7d4234b7f8f405a7e9cc8 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Wed, 22 Jul 2026 21:51:02 -0700
Subject: [PATCH] [LoopInfo] Store blocks using Euler tour representation

Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.

Switch to an Euler tour representation: one BlockLayout array per
LoopInfo, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's.

Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test.

Previously a loop's list was the function's reverse postorder restricted
to its members, so a subloop's blocks were interleaved with the parent's
own blocks at their RPO positions. Now a subloop's blocks are contiguous
after its header within every ancestor's list. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated.

AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.

Aided by Claude Fable 5
---
 bolt/include/bolt/Core/BinaryLoop.h           |   5 -
 bolt/test/X86/loop-nest.test                  |   2 +-
 llvm/include/llvm/Analysis/LoopInfo.h         |   1 -
 llvm/include/llvm/CodeGen/MachineLoopInfo.h   |   3 -
 llvm/include/llvm/Support/GenericLoopInfo.h   | 126 +++++++++++----
 .../llvm/Support/GenericLoopInfoImpl.h        | 153 +++++++++---------
 .../early_exit_with_outer_loop.ll             |   4 +-
 .../nontrivial-unswitch-freeze.ll             |  26 +--
 .../SimpleLoopUnswitch/nontrivial-unswitch.ll |   6 +-
 mlir/include/mlir/Analysis/CFGLoopInfo.h      |   2 +-
 mlir/lib/Analysis/CFGLoopInfo.cpp             |   3 -
 11 files changed, 186 insertions(+), 145 deletions(-)

diff --git a/bolt/include/bolt/Core/BinaryLoop.h b/bolt/include/bolt/Core/BinaryLoop.h
index b425c75715d8b..1c52a051a84cc 100644
--- a/bolt/include/bolt/Core/BinaryLoop.h
+++ b/bolt/include/bolt/Core/BinaryLoop.h
@@ -36,11 +36,6 @@ class BinaryLoop : public LoopBase<BinaryBasicBlock, BinaryLoop> {
   uint64_t ExitCount{0};
 
   // Most of the public interface is provided by LoopBase.
-
-protected:
-  friend class LoopInfoBase<BinaryBasicBlock, BinaryLoop>;
-  explicit BinaryLoop(BinaryBasicBlock *BB)
-      : LoopBase<BinaryBasicBlock, BinaryLoop>(BB) {}
 };
 
 class BinaryLoopInfo : public LoopInfoBase<BinaryBasicBlock, BinaryLoop> {
diff --git a/bolt/test/X86/loop-nest.test b/bolt/test/X86/loop-nest.test
index 51c8fcdb32eaa..208cc9ebdfaaf 100644
--- a/bolt/test/X86/loop-nest.test
+++ b/bolt/test/X86/loop-nest.test
@@ -18,6 +18,6 @@ CHECK-NEXT: Loop basic blocks: .Ltmp[[#MAIN_OUTER_HDR]], .Ltmp[[#]]
 CHECK:      Binary Function "foo" after building cfg
 CHECK:      Loop Info for Function "foo"
 CHECK:      Outer loop header: .Ltmp[[#FOO_OUTER_HDR:]]
-CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_BB:]]
+CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#FOO_INNER_BB:]], .Ltmp[[#]]
 CHECK:      Nested loop header: .Ltmp[[#FOO_INNER_HDR]]
 CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_INNER_HDR]], .Ltmp[[#FOO_INNER_BB]]
diff --git a/llvm/include/llvm/Analysis/LoopInfo.h b/llvm/include/llvm/Analysis/LoopInfo.h
index 13b404505b5e2..15c22e43c44bf 100644
--- a/llvm/include/llvm/Analysis/LoopInfo.h
+++ b/llvm/include/llvm/Analysis/LoopInfo.h
@@ -417,7 +417,6 @@ class LLVM_ABI Loop : public LoopBase<BasicBlock, Loop> {
 
   friend class LoopInfoBase<BasicBlock, Loop>;
   friend class LoopBase<BasicBlock, Loop>;
-  explicit Loop(BasicBlock *BB) : LoopBase<BasicBlock, Loop>(BB) {}
   ~Loop() = default;
 };
 
diff --git a/llvm/include/llvm/CodeGen/MachineLoopInfo.h b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
index 60b3f6c6a3ba5..7893f70cd353c 100644
--- a/llvm/include/llvm/CodeGen/MachineLoopInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
@@ -96,9 +96,6 @@ class MachineLoop : public LoopBase<MachineBasicBlock, MachineLoop> {
   /// Returns true if the given physreg has no defs inside the loop.
   bool isLoopInvariantImplicitPhysReg(Register Reg) const;
 
-  explicit MachineLoop(MachineBasicBlock *MBB)
-    : LoopBase<MachineBasicBlock, MachineLoop>(MBB) {}
-
   MachineLoop() = default;
 };
 
diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 0bdfce7dc889e..cda6577deb31a 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -51,7 +51,6 @@ namespace llvm {
 
 template <class N, class M> class LoopInfoBase;
 template <class N, class M> class LoopBase;
-template <class N, class M> class PopulateLoopsDFS;
 
 //===----------------------------------------------------------------------===//
 /// Instances of this class are used to represent loops that are detected in the
@@ -62,8 +61,12 @@ template <class BlockT, class LoopT> class LoopBase {
   // Loops contained entirely within this one.
   std::vector<LoopT *> SubLoops;
 
-  // The list of blocks in this loop. First entry is the header node.
-  std::vector<BlockT *> Blocks;
+  // The list of blocks in this loop; first entry is the header. Either borrows
+  // from the owning LoopInfo's BlockLayout with BlockCapacity == BlockLen, or
+  // is a private allocation of BlockCapacity slots from its allocator.
+  BlockT **BlockData = nullptr;
+  unsigned BlockLen = 0;
+  unsigned BlockCapacity = 0;
 
   // The LoopInfo that owns this loop. Used to answer contains(BlockT *) from
   // the central block-to-loop map.
@@ -186,7 +189,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Get a list of the basic blocks which make up this loop.
   ArrayRef<BlockT *> getBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks;
+    return ArrayRef<BlockT *>(BlockData, BlockLen);
   }
   using block_iterator = typename ArrayRef<BlockT *>::const_iterator;
   block_iterator block_begin() const { return getBlocks().begin(); }
@@ -200,7 +203,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Invalidate the loop, indicating that it is no longer a loop.
   unsigned getNumBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks.size();
+    return BlockLen;
   }
 
   /// Return true if this loop is no longer valid.  The only valid use of this
@@ -400,19 +403,17 @@ template <class BlockT, class LoopT> class LoopBase {
   /// transformations should use addBasicBlockToLoop.
   void addBlockEntry(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.push_back(BB);
-  }
-
-  /// interface to reverse Blocks[from, end of loop] in this loop
-  void reverseBlock(unsigned from) {
-    assert(!isInvalid() && "Loop not in a valid state!");
-    std::reverse(Blocks.begin() + from, Blocks.end());
+    if (BlockLen == BlockCapacity)
+      LI->reallocBlocks(*static_cast<LoopT *>(this),
+                        std::max(2 * BlockLen, 4u));
+    BlockData[BlockLen++] = BB;
   }
 
   /// interface to do reserve() for Blocks
-  void reserveBlocks(unsigned size) {
+  void reserveBlocks(unsigned Size) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.reserve(size);
+    if (BlockCapacity < Size)
+      LI->reallocBlocks(*static_cast<LoopT *>(this), Size);
   }
 
   /// interface to do reserve() for SubLoops
@@ -421,21 +422,18 @@ template <class BlockT, class LoopT> class LoopBase {
     SubLoops.reserve(Size);
   }
 
-  /// Capacity of the block list; input to an enclosing loop's reserveBlocks()
-  /// during construction, when this loop's list is not yet fully populated.
-  unsigned getBlocksCapacity() const { return Blocks.capacity(); }
-
   /// This method is used to move BB (which must be part of this loop) to be the
   /// loop header of the loop (the block that dominates all others).
   void moveToHeader(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    if (Blocks[0] == BB)
+    if (BlockData[0] == BB)
       return;
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
     for (unsigned i = 0;; ++i) {
-      assert(i != Blocks.size() && "Loop does not contain BB!");
-      if (Blocks[i] == BB) {
-        Blocks[i] = Blocks[0];
-        Blocks[0] = BB;
+      assert(i != BlockLen && "Loop does not contain BB!");
+      if (BlockData[i] == BB) {
+        BlockData[i] = BlockData[0];
+        BlockData[0] = BB;
         return;
       }
     }
@@ -446,9 +444,12 @@ template <class BlockT, class LoopT> class LoopBase {
   /// class.
   void removeBlockFromLoop(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    auto I = find(Blocks, BB);
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
+    MutableArrayRef<BlockT *> Blocks(BlockData, BlockLen);
+    auto *I = llvm::find(Blocks, BB);
     assert(I != Blocks.end() && "N is not in this list!");
-    Blocks.erase(I);
+    std::move(I + 1, Blocks.end(), I);
+    --BlockLen;
   }
 
   /// Verify loop structure
@@ -469,15 +470,10 @@ template <class BlockT, class LoopT> class LoopBase {
 
 protected:
   friend class LoopInfoBase<BlockT, LoopT>;
-  friend class PopulateLoopsDFS<BlockT, LoopT>;
 
   /// This creates an empty loop.
   LoopBase() : ParentLoop(nullptr) {}
 
-  explicit LoopBase(BlockT *BB) : ParentLoop(nullptr) {
-    Blocks.push_back(BB);
-  }
-
   // Since loop passes like SCEV are allowed to key analysis results off of
   // `Loop` pointers, we cannot re-use pointers within a loop pass manager.
   // This means loop passes should not be `delete` ing `Loop` objects directly
@@ -495,7 +491,10 @@ template <class BlockT, class LoopT> class LoopBase {
     IsInvalid = true;
 #endif
     SubLoops.clear();
-    Blocks.clear();
+    // The block storage is reclaimed by the owning LoopInfo.
+    BlockData = nullptr;
+    BlockLen = 0;
+    BlockCapacity = 0;
     ParentLoop = nullptr;
   }
 };
@@ -525,6 +524,12 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   unsigned BlockNumberEpoch;
 
   std::vector<LoopT *> TopLevelLoops;
+
+  // Shared reverse postorder layout of all blocks. Each initial loop is a slice
+  // of this array, subloop slices nested inside their parent's.
+  std::unique_ptr<BlockT *[]> BlockLayout;
+  unsigned BlockLayoutLen = 0;
+
   BumpPtrAllocator LoopAllocator;
 
   friend class LoopBase<BlockT, LoopT>;
@@ -540,9 +545,11 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   LoopInfoBase(LoopInfoBase &&Arg)
       : BBMap(std::move(Arg.BBMap)),
         TopLevelLoops(std::move(Arg.TopLevelLoops)),
+        BlockLayout(std::move(Arg.BlockLayout)),
         LoopAllocator(std::move(Arg.LoopAllocator)) {
     ParentPtr = Arg.ParentPtr;
     BlockNumberEpoch = Arg.BlockNumberEpoch;
+    BlockLayoutLen = Arg.BlockLayoutLen;
     resetLoopInfoOwners();
     // We have to clear the arguments top level loops as we've taken ownership.
     Arg.TopLevelLoops.clear();
@@ -556,9 +563,12 @@ template <class BlockT, class LoopT> class LoopInfoBase {
       L->~LoopT();
 
     TopLevelLoops = std::move(RHS.TopLevelLoops);
+    BlockLayout = std::move(RHS.BlockLayout);
+    BlockLayoutLen = RHS.BlockLayoutLen;
     LoopAllocator = std::move(RHS.LoopAllocator);
     resetLoopInfoOwners();
     RHS.TopLevelLoops.clear();
+    RHS.BlockLayoutLen = 0;
     return *this;
   }
 
@@ -568,12 +578,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     for (auto *L : TopLevelLoops)
       L->~LoopT();
     TopLevelLoops.clear();
+    BlockLayout.reset();
+    BlockLayoutLen = 0;
     LoopAllocator.Reset();
   }
 
-  template <typename... ArgsTy> LoopT *AllocateLoop(ArgsTy &&...Args) {
+  LoopT *AllocateLoop() {
     LoopT *Storage = LoopAllocator.Allocate<LoopT>();
-    LoopT *L = new (Storage) LoopT(std::forward<ArgsTy>(Args)...);
+    LoopT *L = new (Storage) LoopT();
     L->LI = this;
     return L;
   }
@@ -636,6 +648,49 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
+  /// AllocateLoop for analyze(): stash \p Header (see pendingHeader).
+  /// getHeader() only works once the layout carve has replaced the stash with
+  /// the loop's block list.
+  LoopT *allocateLoop(BlockT *Header) {
+    LoopT *L = AllocateLoop();
+    L->BlockData = reinterpret_cast<BlockT **>(Header);
+    return L;
+  }
+
+  /// The header of a loop under construction, stashed in BlockData until the
+  /// layout carve builds the block list.
+  static BlockT *pendingHeader(const LoopT *L) {
+    return reinterpret_cast<BlockT *>(L->BlockData);
+  }
+
+  void discoverAndMapSubloop(LoopT *L, BlockT *Header,
+                             ArrayRef<BlockT *> Backedges,
+                             const DominatorTreeBase<BlockT, false> &DomTree);
+
+  /// True if \p L's block list is a borrowed slice of BlockLayout rather than
+  /// a private allocation.
+  bool hasBorrowedBlocks(const LoopT &L) const {
+    return L.BlockData >= BlockLayout.get() &&
+           L.BlockData < BlockLayout.get() + BlockLayoutLen;
+  }
+
+  /// Replace \p L's block list with a private allocation of NewCapacity
+  /// slots. The old storage is abandoned in place so slices sharing it stay
+  /// intact; it is reclaimed when this LoopInfo is cleared.
+  void reallocBlocks(LoopT &L, unsigned NewCapacity) {
+    assert(NewCapacity >= L.BlockLen && "capacity below size");
+    BlockT **New = LoopAllocator.Allocate<BlockT *>(NewCapacity);
+    llvm::copy(L.getBlocks(), New);
+    L.BlockData = New;
+    L.BlockCapacity = NewCapacity;
+  }
+
+  /// Copy \p L's borrowed block list into private storage before a mutation.
+  void materializeBlocks(LoopT &L) {
+    if (hasBorrowedBlocks(L))
+      reallocBlocks(L, L.BlockLen);
+  }
+
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
@@ -672,7 +727,10 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// ancestors or descendants, and not the block-to-loop mapping.
   template <typename PredicateT>
   void removeBlocksIf(LoopT &L, PredicateT Pred) {
-    llvm::erase_if(L.Blocks, Pred);
+    materializeBlocks(L);
+    L.BlockLen = llvm::remove_if(
+                     MutableArrayRef<BlockT *>(L.BlockData, L.BlockLen), Pred) -
+                 L.BlockData;
   }
 
   /// Remove every block satisfying \p Pred from \p Start and each of its
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 170f05e9d3a49..ad36d3343c8a8 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -282,7 +282,7 @@ void LoopBase<BlockT, LoopT>::addBasicBlockToLoop(
     BlockT *NewBB, LoopInfoBase<BlockT, LoopT> &LIB) {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  if (!Blocks.empty()) {
+  if (!getBlocks().empty()) {
     auto SameHeader = LIB[getHeader()];
     assert(contains(SameHeader) && getHeader() == SameHeader->getHeader() &&
            "Incorrect LI specified for this loop!");
@@ -325,7 +325,7 @@ template <class BlockT, class LoopT>
 void LoopBase<BlockT, LoopT>::verifyLoop() const {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  assert(!Blocks.empty() && "Loop header is missing");
+  assert(!getBlocks().empty() && "Loop header is missing");
 
   // Setup for using a depth-first iterator to visit every block in the loop.
   SmallVector<BlockT *, 8> ExitBBs;
@@ -371,7 +371,7 @@ void LoopBase<BlockT, LoopT>::verifyLoop() const {
 
   if (VisitedBBs.size() != getNumBlocks()) {
     dbgs() << "The following blocks are unreachable in the loop: ";
-    for (auto *BB : Blocks) {
+    for (auto *BB : getBlocks()) {
       if (!VisitedBBs.count(BB)) {
         dbgs() << *BB << "\n";
       }
@@ -455,12 +455,11 @@ void LoopBase<BlockT, LoopT>::print(raw_ostream &OS, bool Verbose,
 /// this loop are mapped to this loop or a subloop. And all subloops within this
 /// loop have their parent loop set to this loop or a subloop.
 template <class BlockT, class LoopT>
-static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
-                                  LoopInfoBase<BlockT, LoopT> *LI,
-                                  const DomTreeBase<BlockT> &DomTree) {
+void LoopInfoBase<BlockT, LoopT>::discoverAndMapSubloop(
+    LoopT *L, BlockT *Header, ArrayRef<BlockT *> Backedges,
+    const DominatorTreeBase<BlockT, false> &DomTree) {
   using InvBlockTraits = GraphTraits<Inverse<BlockT *>>;
 
-  unsigned NumBlocks = 0;
   unsigned NumSubloops = 0;
 
   // Perform a backward CFG traversal using a worklist.
@@ -469,15 +468,14 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
     BlockT *PredBB = ReverseCFGWorklist.back();
     ReverseCFGWorklist.pop_back();
 
-    LoopT *Subloop = LI->getLoopFor(PredBB);
+    LoopT *Subloop = getLoopFor(PredBB);
     if (!Subloop) {
       if (!DomTree.isReachableFromEntry(PredBB))
         continue;
 
       // This is an undiscovered block. Map it to the current loop.
-      LI->changeLoopFor(PredBB, L);
-      ++NumBlocks;
-      if (PredBB == L->getHeader())
+      changeLoopFor(PredBB, L);
+      if (PredBB == Header)
         continue;
       // Push all block predecessors on the worklist.
       ReverseCFGWorklist.insert(ReverseCFGWorklist.end(),
@@ -494,84 +492,31 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
       // Discover a subloop of this loop.
       Subloop->setParentLoop(L);
       ++NumSubloops;
-      NumBlocks += Subloop->getBlocksCapacity();
-      PredBB = Subloop->getHeader();
+      PredBB = pendingHeader(Subloop);
       // Continue traversal along predecessors that are not loop-back edges from
       // within this subloop tree itself. Note that a predecessor may directly
       // reach another subloop that is not yet discovered to be a subloop of
       // this loop, which we must traverse.
       for (const auto Pred : inverse_children<BlockT *>(PredBB)) {
-        if (LI->getLoopFor(Pred) != Subloop)
+        if (getLoopFor(Pred) != Subloop)
           ReverseCFGWorklist.push_back(Pred);
       }
     }
   }
   L->reserveSubLoops(NumSubloops);
-  L->reserveBlocks(NumBlocks);
-}
-
-/// Populate all loop data in a stable order during a single forward DFS.
-template <class BlockT, class LoopT> class PopulateLoopsDFS {
-  using BlockTraits = GraphTraits<BlockT *>;
-  using SuccIterTy = typename BlockTraits::ChildIteratorType;
-
-  LoopInfoBase<BlockT, LoopT> *LI;
-
-public:
-  PopulateLoopsDFS(LoopInfoBase<BlockT, LoopT> *li) : LI(li) {}
-
-  void traverse(BlockT *EntryBlock);
-
-protected:
-  void insertIntoLoop(BlockT *Block);
-};
-
-/// Top-level driver for the forward DFS within the loop.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::traverse(BlockT *EntryBlock) {
-  for (BlockT *BB : post_order(EntryBlock))
-    insertIntoLoop(BB);
-}
-
-/// Add a single Block to its ancestor loops in PostOrder. If the block is a
-/// subloop header, add the subloop to its parent in PostOrder, then reverse the
-/// Block and Subloop vectors of the now complete subloop to achieve RPO.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::insertIntoLoop(BlockT *Block) {
-  LoopT *Subloop = LI->getLoopFor(Block);
-  if (Subloop && Block == Subloop->getHeader()) {
-    // We reach this point once per subloop after processing all the blocks in
-    // the subloop.
-    if (!Subloop->isOutermost())
-      Subloop->getParentLoop()->SubLoops.push_back(Subloop);
-    else
-      LI->addTopLevelLoop(Subloop);
-
-    // For convenience, Blocks and Subloops are inserted in postorder. Reverse
-    // the lists, except for the loop header, which is always at the beginning.
-    Subloop->reverseBlock(1);
-    std::reverse(Subloop->SubLoops.begin(), Subloop->SubLoops.end());
-
-    Subloop = Subloop->getParentLoop();
-  }
-  for (; Subloop; Subloop = Subloop->getParentLoop())
-    Subloop->addBlockEntry(Block);
 }
 
 /// Analyze LoopInfo discovers loops during a reverse preorder DominatorTree
 /// traversal interleaved with backward CFG traversals within each subloop
 /// (discoverAndMapSubloop). The backward traversal skips inner subloops, so
-/// this part of the algorithm is linear in the number of CFG edges. Subloop and
-/// Block vectors are then populated during a single forward CFG traversal
-/// (PopulateLoopDFS).
+/// this part of the algorithm is linear in the number of CFG edges.
 ///
-/// During the two CFG traversals each block is seen three times:
-/// 1) Discovered and mapped by a reverse CFG traversal.
-/// 2) Visited during a forward DFS CFG traversal.
-/// 3) Reverse-inserted in the loop in postorder following forward DFS.
-///
-/// The Block vectors are inclusive, so step 3 requires loop-depth number of
-/// insertions per block.
+/// The block lists are then built in one shared layout: a forward CFG postorder
+/// records the in-loop blocks, and a reverse walk (RPO) carves each loop a
+/// contiguous slice of its parent's, writing each block once to its innermost
+/// loop. Lists are header-first with each subloop's blocks contiguous, ordered
+/// by first appearance in RPO; SubLoops keep program order, TopLevelLoops
+/// reverse program order.
 template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
@@ -587,6 +532,7 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   for (const DomTreeNodeBase<BlockT> *Node : DomTree.nodes())
     PreorderNodes[Node->getDFSNumIn()] = Node;
 
+  bool HasLoops = false;
   for (const DomTreeNodeBase<BlockT> *DomNode : llvm::reverse(PreorderNodes)) {
     BlockT *Header = DomNode->getBlock();
     SmallVector<BlockT *, 4> Backedges;
@@ -600,14 +546,63 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
     }
     // Perform a backward CFG traversal to discover and map blocks in this loop.
     if (!Backedges.empty()) {
-      LoopT *L = AllocateLoop(Header);
-      discoverAndMapSubloop(L, ArrayRef<BlockT *>(Backedges), this, DomTree);
+      HasLoops = true;
+      LoopT *L = allocateLoop(Header);
+      discoverAndMapSubloop(L, Header, Backedges, DomTree);
+    }
+  }
+  // Most functions have no loops; skip the layout construction.
+  if (!HasLoops)
+    return;
+
+  // Record each in-loop block with its innermost loop in forward CFG postorder,
+  // and build the loop list in PO.
+  SmallVector<std::pair<BlockT *, LoopT *>, 32> PO;
+  SmallVector<LoopT *, 4> LoopsPO;
+  PO.reserve(BBMap.size());
+  for (BlockT *BB : post_order(DomRoot->getBlock())) {
+    LoopT *L = lookupLoopFor(BB);
+    if (!L)
+      continue;
+    PO.emplace_back(BB, L);
+    ++L->BlockLen;
+    if (BB != pendingHeader(L))
+      continue;
+    LoopsPO.push_back(L);
+    if (LoopT *Parent = L->getParentLoop())
+      Parent->BlockLen += L->BlockLen;
+    else
+      TopLevelLoops.push_back(L);
+  }
+  // Headers are dominator-tree nodes, hence reachable and in the postorder.
+  assert(!LoopsPO.empty() && "discovered loops but found no header");
+
+  BlockLayoutLen = PO.size();
+  BlockLayout.reset(new BlockT *[BlockLayoutLen]);
+  BlockT **RootCursor = BlockLayout.get();
+  for (auto &[BB, L] : llvm::reverse(PO)) {
+    if (L->BlockCapacity == 0) {
+      // The first block of a L is its the header. Carve its slice from the
+      // parent (already visited)'s cursor.
+      if (LoopT *Parent = L->getParentLoop()) {
+        assert(Parent->BlockCapacity != 0 &&
+               "parent slice not carved before child");
+        L->BlockData = Parent->BlockData + Parent->BlockCapacity;
+        Parent->BlockCapacity += L->BlockLen;
+        Parent->SubLoops.push_back(L);
+      } else {
+        L->BlockData = RootCursor;
+        RootCursor += L->BlockLen;
+      }
     }
+    // Each block lands once, at its innermost loop's cursor.
+    L->BlockData[L->BlockCapacity++] = BB;
   }
-  // Perform a single forward CFG traversal to populate block and subloop
-  // vectors for all loops.
-  PopulateLoopsDFS<BlockT, LoopT> DFS(this);
-  DFS.traverse(DomRoot->getBlock());
+
+#ifndef NDEBUG
+  for (LoopT *L : LoopsPO)
+    assert(L->BlockCapacity == L->BlockLen && "layout slice not fully used");
+#endif
 }
 
 template <class BlockT, class LoopT>
diff --git a/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll b/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
index 69ef9f3e530ad..aec401ece0b3a 100644
--- a/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
@@ -46,7 +46,7 @@ loop.inner.end:
 ; loops at depths 1 and 2, respectively.
 define void @early_exit_in_outer_loop2() {
 ; CHECK-LABEL: Loop info for function 'early_exit_in_outer_loop2':
-; CHECK: Loop at depth 1 containing: %loop.outer<header>,%loop.middle,%loop.inner.found,%loop.inner.end,%loop.middle.end,%loop.outer.latch<latch>,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit
+; CHECK: Loop at depth 1 containing: %loop.outer<header>,%loop.middle,%loop.inner.end,%loop.inner.found,%loop.middle.end,%loop.outer.latch<latch>,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit
 ; CHECK:    Loop at depth 2 containing: %loop.middle<header>,%loop.inner.end<latch><exiting>,%vector.ph,%vector.body<exiting>,%vector.body.interim,%middle.block
 ; CHECK:        Loop at depth 3 containing: %vector.body<header><exiting>,%vector.body.interim<latch><exiting>
 entry:
@@ -268,7 +268,7 @@ early.exit.leave:
 ; (one going to a middle loop, one going to the outer loop).
 define i32 @multi_early_exit_two_different_loops(i1 %c, ptr dereferenceable(1024) %src) nofree {
 ; CHECK-LABEL: Loop info for function 'multi_early_exit_two_different_loops':
-; CHECK-NEXT: Loop at depth 1 containing: %outer.header<header>,%middle.header,%early.exit.outer,%early.exit.middle,%middle.latch,%outer.latch<latch>,%middle.latch.loopexit,%outer.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check,%vector.early.exit.1,%vector.early.exit.0
+; CHECK-NEXT: Loop at depth 1 containing: %outer.header<header>,%middle.header,%early.exit.middle,%middle.latch,%early.exit.outer,%outer.latch<latch>,%middle.latch.loopexit,%outer.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check,%vector.early.exit.1,%vector.early.exit.0
 ; CHECK-NEXT:     Loop at depth 2 containing: %middle.header<header>,%early.exit.middle,%middle.latch<latch><exiting>,%middle.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check<exiting>,%vector.early.exit.0
 ; CHECK-NEXT:         Loop at depth 3 containing: %vector.body<header><exiting>,%vector.body.interim<latch><exiting>
 entry:
diff --git a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
index f55ec1bedc0b8..925b919c5a397 100644
--- a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
+++ b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
@@ -1134,10 +1134,10 @@ define i32 @test13a(ptr %ptr, i1 %cond, ptr %a.ptr, ptr %b.ptr) {
 ; CHECK:       loop_b_inner_body.us:
 ; CHECK-NEXT:    [[V4_US:%.*]] = load i1, ptr [[PTR]], align 1
 ; CHECK-NEXT:    br i1 [[V4_US]], label [[LOOP_B_INNER_LATCH_US]], label [[LOOP_B_INNER_EXIT_US:%.*]]
-; CHECK:       loop_b_inner_exit.us:
-; CHECK-NEXT:    br label [[LOOP_LATCH_US:%.*]]
 ; CHECK:       loop_b_inner_latch.us:
 ; CHECK-NEXT:    br label [[LOOP_B_INNER_HEADER_US]]
+; CHECK:       loop_b_inner_exit.us:
+; CHECK-NEXT:    br label [[LOOP_LATCH_US:%.*]]
 ; CHECK:       loop_a.us:
 ; CHECK-NEXT:    [[V2_US:%.*]] = load i1, ptr [[PTR]], align 1
 ; CHECK-NEXT:    br i1 [[V2_US]], label [[LOOP_EXIT_SPLIT_US:%.*]], label [[LOOP_LATCH_US]]
@@ -1426,10 +1426,10 @@ define i32 @test29(i32 %arg) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARG_FR:%.*]] = freeze i32 [[ARG:%.*]]
 ; CHECK-NEXT:    switch i32 [[ARG_FR]], label [[ENTRY_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 0, label [[ENTRY_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 1, label [[ENTRY_SPLIT_US]]
-; CHECK-NEXT:    i32 2, label [[ENTRY_SPLIT_US1:%.*]]
-; CHECK-NEXT:    i32 3, label [[ENTRY_SPLIT]]
+; CHECK-NEXT:      i32 0, label [[ENTRY_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 1, label [[ENTRY_SPLIT_US]]
+; CHECK-NEXT:      i32 2, label [[ENTRY_SPLIT_US1:%.*]]
+; CHECK-NEXT:      i32 3, label [[ENTRY_SPLIT]]
 ; CHECK-NEXT:    ]
 ; CHECK:       entry.split.us:
 ; CHECK-NEXT:    br label [[HEADER_US:%.*]]
@@ -1587,10 +1587,10 @@ define i32 @test30(i32 %arg) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARG_FR:%.*]] = freeze i32 [[ARG:%.*]]
 ; CHECK-NEXT:    switch i32 [[ARG_FR]], label [[ENTRY_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 -1, label [[ENTRY_SPLIT]]
-; CHECK-NEXT:    i32 0, label [[ENTRY_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 1, label [[ENTRY_SPLIT_US1:%.*]]
-; CHECK-NEXT:    i32 2, label [[ENTRY_SPLIT_US1]]
+; CHECK-NEXT:      i32 -1, label [[ENTRY_SPLIT]]
+; CHECK-NEXT:      i32 0, label [[ENTRY_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 1, label [[ENTRY_SPLIT_US1:%.*]]
+; CHECK-NEXT:      i32 2, label [[ENTRY_SPLIT_US1]]
 ; CHECK-NEXT:    ]
 ; CHECK:       entry.split.us:
 ; CHECK-NEXT:    br label [[HEADER_US:%.*]]
@@ -2259,9 +2259,9 @@ define void @hoist_inner_loop_switch(ptr %ptr) {
 ; CHECK-NEXT:    [[V1:%.*]] = call i32 @cond.i32()
 ; CHECK-NEXT:    [[V1_FR:%.*]] = freeze i32 [[V1]]
 ; CHECK-NEXT:    switch i32 [[V1_FR]], label [[B_HEADER_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 1, label [[B_HEADER_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 2, label [[B_HEADER_SPLIT_US]]
-; CHECK-NEXT:    i32 3, label [[B_HEADER_SPLIT_US]]
+; CHECK-NEXT:      i32 1, label [[B_HEADER_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 2, label [[B_HEADER_SPLIT_US]]
+; CHECK-NEXT:      i32 3, label [[B_HEADER_SPLIT_US]]
 ; CHECK-NEXT:    ]
 ; CHECK:       b.header.split.us:
 ; CHECK-NEXT:    br label [[C_HEADER_US:%.*]]
diff --git a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
index 0cb0e0d1cca2b..8fddaa1839576 100644
--- a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
+++ b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
@@ -2116,12 +2116,12 @@ loop_latch:
 ; CHECK-NEXT:    %[[V:.*]] = load i1, ptr %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_b_inner_latch.us, label %loop_b_inner_exit.us
 ;
-; CHECK:       loop_b_inner_exit.us:
-; CHECK-NEXT:    br label %loop_latch.us
-;
 ; CHECK:       loop_b_inner_latch.us:
 ; CHECK-NEXT:    br label %loop_b_inner_header.us
 ;
+; CHECK:       loop_b_inner_exit.us:
+; CHECK-NEXT:    br label %loop_latch.us
+;
 ; CHECK:       loop_a.us:
 ; CHECK-NEXT:    %[[V:.*]] = load i1, ptr %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_exit.split.us, label %loop_latch.us
diff --git a/mlir/include/mlir/Analysis/CFGLoopInfo.h b/mlir/include/mlir/Analysis/CFGLoopInfo.h
index 15ac0d71cb94d..7060a9df7de2d 100644
--- a/mlir/include/mlir/Analysis/CFGLoopInfo.h
+++ b/mlir/include/mlir/Analysis/CFGLoopInfo.h
@@ -36,7 +36,7 @@ namespace mlir {
 /// class provides accessors to the loop analysis.
 class CFGLoop : public llvm::LoopBase<mlir::Block, mlir::CFGLoop> {
 private:
-  explicit CFGLoop(mlir::Block *block);
+  CFGLoop() = default;
 
   friend class llvm::LoopBase<mlir::Block, CFGLoop>;
   friend class llvm::LoopInfoBase<mlir::Block, CFGLoop>;
diff --git a/mlir/lib/Analysis/CFGLoopInfo.cpp b/mlir/lib/Analysis/CFGLoopInfo.cpp
index 9fe220344dcea..e03b664f9c72e 100644
--- a/mlir/lib/Analysis/CFGLoopInfo.cpp
+++ b/mlir/lib/Analysis/CFGLoopInfo.cpp
@@ -16,9 +16,6 @@ template class llvm::LoopInfoBase<mlir::Block, mlir::CFGLoop>;
 
 using namespace mlir;
 
-CFGLoop::CFGLoop(mlir::Block *block)
-    : llvm::LoopBase<mlir::Block, CFGLoop>(block) {}
-
 CFGLoopInfo::CFGLoopInfo(
     const llvm::DominatorTreeBase<mlir::Block, false> &domTree) {
   analyze(domTree);


From llvmlistbot at llvm.org  Thu Jul 23 01:57:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 01:57:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4138dc4 - [MLIR][OpenMP] Preserve debug
 location in OpenMPIRBuilder calls. (#211254)
Message-ID: <6a61d789.f1c1923d.d9163.3c78@mx.google.com>


Author: Abid Qadeer
Date: 2026-07-23T09:57:40+01:00
New Revision: 4138dc431ad849bd8c0e76f907345c6d0507890b

URL: https://github.com/llvm/llvm-project/commit/4138dc431ad849bd8c0e76f907345c6d0507890b
DIFF: https://github.com/llvm/llvm-project/commit/4138dc431ad849bd8c0e76f907345c6d0507890b.diff

LOG: [MLIR][OpenMP] Preserve debug location in OpenMPIRBuilder calls. (#211254)

Many `OpenMPIRBuilder` entry points take an
`OpenMPIRBuilder::LocationDescription`. It has two relevant
constructors:

`LocationDescription(const IRBuilderBase &IRB)` // captures IP and debug
loc
`LocationDescription(const InsertPointTy &IP)` // captures IP only; DL
is empty

The OpenMP MLIR-to-LLVM-IR translation constructs the location from
`builder` in almost all places (~45 call sites), which selects the first
constructor and propagates both the insertion point and the current
debug location. A few call sites instead passed `builder.saveIP()`,
which selects the second constructor and silently drops the debug
location.

Change the 9 offending call sites to pass `builder` instead of
`builder.saveIP()`, so the debug location is preserved.

Added: 
    

Modified: 
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 85963c402bc87..1c38fd27daae1 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -271,7 +271,7 @@ class LinearClauseProcessor {
     // Emit barrier
     builder.SetInsertPoint(linearExitBB->getTerminator());
     return moduleTranslation.getOpenMPBuilder()->createBarrier(
-        builder.saveIP(), llvm::omp::OMPD_barrier);
+        builder, llvm::omp::OMPD_barrier);
   }
 
   // Emit stores for linear variables. Useful in case of SIMD
@@ -1654,8 +1654,8 @@ static LogicalResult createReductionsAndCleanup(
   llvm::UnreachableInst *tempTerminator = builder.CreateUnreachable();
   builder.SetInsertPoint(tempTerminator);
   llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-      ompBuilder->createReductions(builder.saveIP(), allocaIP, reductionInfos,
-                                   isByRef, isNowait, isTeamsReduction);
+      ompBuilder->createReductions(builder, allocaIP, reductionInfos, isByRef,
+                                   isNowait, isTeamsReduction);
 
   if (failed(handleError(contInsertPoint, *op)))
     return failure();
@@ -2003,7 +2003,7 @@ static LogicalResult copyFirstPrivateVars(
   if (insertBarrier && !opIsInSingleThread(op)) {
     llvm::OpenMPIRBuilder *ompBuilder = moduleTranslation.getOpenMPBuilder();
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy res =
-        ompBuilder->createBarrier(builder.saveIP(), llvm::omp::OMPD_barrier);
+        ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(res, *op)))
       return failure();
   }
@@ -4475,7 +4475,7 @@ convertOmpTaskwaitOp(omp::TaskwaitOp twOp, llvm::IRBuilderBase &builder,
     return failure();
   }
 
-  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder.saveIP(), dds);
+  moduleTranslation.getOpenMPBuilder()->createTaskwait(builder, dds);
   if (dds.DepArray) {
     builder.CreateFree(dds.DepArray);
   }
@@ -4635,7 +4635,7 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
                                         loopInfo->getPreheader());
     llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterBarrierIP =
         moduleTranslation.getOpenMPBuilder()->createBarrier(
-            builder.saveIP(), llvm::omp::OMPD_barrier);
+            builder, llvm::omp::OMPD_barrier);
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
@@ -4832,9 +4832,10 @@ convertOmpParallel(omp::ParallelOp opInst, llvm::IRBuilderBase &builder,
       builder.SetInsertPoint(tempTerminator);
 
       llvm::OpenMPIRBuilder::InsertPointOrErrorTy contInsertPoint =
-          ompBuilder->createReductions(
-              builder.saveIP(), allocaIP, reductionInfos, isByRef,
-              /*IsNoWait=*/false, /*IsTeamsReduction=*/false);
+          ompBuilder->createReductions(builder, allocaIP, reductionInfos,
+                                       isByRef,
+                                       /*IsNoWait=*/false,
+                                       /*IsTeamsReduction=*/false);
       if (!contInsertPoint)
         return contInsertPoint.takeError();
 
@@ -9796,8 +9797,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
               return failure();
 
             llvm::OpenMPIRBuilder::InsertPointOrErrorTy afterIP =
-                ompBuilder->createBarrier(builder.saveIP(),
-                                          llvm::omp::OMPD_barrier);
+                ompBuilder->createBarrier(builder, llvm::omp::OMPD_barrier);
             LogicalResult res = handleError(afterIP, *op);
             if (res.succeeded()) {
               // If the barrier generated a cancellation check, the insertion
@@ -9810,7 +9810,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             if (failed(checkImplementationStatus(*op)))
               return failure();
 
-            ompBuilder->createTaskyield(builder.saveIP());
+            ompBuilder->createTaskyield(builder);
             return success();
           })
           .Case([&](omp::FlushOp op) {
@@ -9825,7 +9825,7 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             //
             // The argument list is discarded so that, flush with a list is
             // treated same as a flush without a list.
-            ompBuilder->createFlush(builder.saveIP());
+            ompBuilder->createFlush(builder);
             return success();
           })
           .Case([&](omp::ParallelOp op) {


        

From llvmlistbot at llvm.org  Thu Jul 23 01:59:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 01:59:27 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a61d7ee.7d702ed2.2f6c05.4d56@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-support
@llvm/pr-subscribers-mlir
@llvm/pr-subscribers-llvm-transforms
@llvm/pr-subscribers-bolt

@llvm/pr-subscribers-llvm-analysis

Author: Fangrui Song (MaskRay)

<details>
<summary>Changes</summary>

Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.

Switch to an Euler tour representation: one BlockLayout array per
LoopInfo, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's.

Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test.

Previously a loop's list was the function's reverse postorder restricted
to its members, so a subloop's blocks were interleaved with the parent's
own blocks at their RPO positions. Now a subloop's blocks are contiguous
after its header within every ancestor's list. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated.

AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.

Aided by Claude Fable 5


---

Patch is 30.04 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211485.diff


11 Files Affected:

- (modified) bolt/include/bolt/Core/BinaryLoop.h (-5) 
- (modified) bolt/test/X86/loop-nest.test (+1-1) 
- (modified) llvm/include/llvm/Analysis/LoopInfo.h (-1) 
- (modified) llvm/include/llvm/CodeGen/MachineLoopInfo.h (-3) 
- (modified) llvm/include/llvm/Support/GenericLoopInfo.h (+92-34) 
- (modified) llvm/include/llvm/Support/GenericLoopInfoImpl.h (+74-79) 
- (modified) llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll (+2-2) 
- (modified) llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll (+13-13) 
- (modified) llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll (+3-3) 
- (modified) mlir/include/mlir/Analysis/CFGLoopInfo.h (+1-1) 
- (modified) mlir/lib/Analysis/CFGLoopInfo.cpp (-3) 


``````````diff
diff --git a/bolt/include/bolt/Core/BinaryLoop.h b/bolt/include/bolt/Core/BinaryLoop.h
index b425c75715d8b..1c52a051a84cc 100644
--- a/bolt/include/bolt/Core/BinaryLoop.h
+++ b/bolt/include/bolt/Core/BinaryLoop.h
@@ -36,11 +36,6 @@ class BinaryLoop : public LoopBase<BinaryBasicBlock, BinaryLoop> {
   uint64_t ExitCount{0};
 
   // Most of the public interface is provided by LoopBase.
-
-protected:
-  friend class LoopInfoBase<BinaryBasicBlock, BinaryLoop>;
-  explicit BinaryLoop(BinaryBasicBlock *BB)
-      : LoopBase<BinaryBasicBlock, BinaryLoop>(BB) {}
 };
 
 class BinaryLoopInfo : public LoopInfoBase<BinaryBasicBlock, BinaryLoop> {
diff --git a/bolt/test/X86/loop-nest.test b/bolt/test/X86/loop-nest.test
index 51c8fcdb32eaa..208cc9ebdfaaf 100644
--- a/bolt/test/X86/loop-nest.test
+++ b/bolt/test/X86/loop-nest.test
@@ -18,6 +18,6 @@ CHECK-NEXT: Loop basic blocks: .Ltmp[[#MAIN_OUTER_HDR]], .Ltmp[[#]]
 CHECK:      Binary Function "foo" after building cfg
 CHECK:      Loop Info for Function "foo"
 CHECK:      Outer loop header: .Ltmp[[#FOO_OUTER_HDR:]]
-CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_BB:]]
+CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#FOO_INNER_BB:]], .Ltmp[[#]]
 CHECK:      Nested loop header: .Ltmp[[#FOO_INNER_HDR]]
 CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_INNER_HDR]], .Ltmp[[#FOO_INNER_BB]]
diff --git a/llvm/include/llvm/Analysis/LoopInfo.h b/llvm/include/llvm/Analysis/LoopInfo.h
index 13b404505b5e2..15c22e43c44bf 100644
--- a/llvm/include/llvm/Analysis/LoopInfo.h
+++ b/llvm/include/llvm/Analysis/LoopInfo.h
@@ -417,7 +417,6 @@ class LLVM_ABI Loop : public LoopBase<BasicBlock, Loop> {
 
   friend class LoopInfoBase<BasicBlock, Loop>;
   friend class LoopBase<BasicBlock, Loop>;
-  explicit Loop(BasicBlock *BB) : LoopBase<BasicBlock, Loop>(BB) {}
   ~Loop() = default;
 };
 
diff --git a/llvm/include/llvm/CodeGen/MachineLoopInfo.h b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
index 60b3f6c6a3ba5..7893f70cd353c 100644
--- a/llvm/include/llvm/CodeGen/MachineLoopInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
@@ -96,9 +96,6 @@ class MachineLoop : public LoopBase<MachineBasicBlock, MachineLoop> {
   /// Returns true if the given physreg has no defs inside the loop.
   bool isLoopInvariantImplicitPhysReg(Register Reg) const;
 
-  explicit MachineLoop(MachineBasicBlock *MBB)
-    : LoopBase<MachineBasicBlock, MachineLoop>(MBB) {}
-
   MachineLoop() = default;
 };
 
diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 0bdfce7dc889e..cda6577deb31a 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -51,7 +51,6 @@ namespace llvm {
 
 template <class N, class M> class LoopInfoBase;
 template <class N, class M> class LoopBase;
-template <class N, class M> class PopulateLoopsDFS;
 
 //===----------------------------------------------------------------------===//
 /// Instances of this class are used to represent loops that are detected in the
@@ -62,8 +61,12 @@ template <class BlockT, class LoopT> class LoopBase {
   // Loops contained entirely within this one.
   std::vector<LoopT *> SubLoops;
 
-  // The list of blocks in this loop. First entry is the header node.
-  std::vector<BlockT *> Blocks;
+  // The list of blocks in this loop; first entry is the header. Either borrows
+  // from the owning LoopInfo's BlockLayout with BlockCapacity == BlockLen, or
+  // is a private allocation of BlockCapacity slots from its allocator.
+  BlockT **BlockData = nullptr;
+  unsigned BlockLen = 0;
+  unsigned BlockCapacity = 0;
 
   // The LoopInfo that owns this loop. Used to answer contains(BlockT *) from
   // the central block-to-loop map.
@@ -186,7 +189,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Get a list of the basic blocks which make up this loop.
   ArrayRef<BlockT *> getBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks;
+    return ArrayRef<BlockT *>(BlockData, BlockLen);
   }
   using block_iterator = typename ArrayRef<BlockT *>::const_iterator;
   block_iterator block_begin() const { return getBlocks().begin(); }
@@ -200,7 +203,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Invalidate the loop, indicating that it is no longer a loop.
   unsigned getNumBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks.size();
+    return BlockLen;
   }
 
   /// Return true if this loop is no longer valid.  The only valid use of this
@@ -400,19 +403,17 @@ template <class BlockT, class LoopT> class LoopBase {
   /// transformations should use addBasicBlockToLoop.
   void addBlockEntry(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.push_back(BB);
-  }
-
-  /// interface to reverse Blocks[from, end of loop] in this loop
-  void reverseBlock(unsigned from) {
-    assert(!isInvalid() && "Loop not in a valid state!");
-    std::reverse(Blocks.begin() + from, Blocks.end());
+    if (BlockLen == BlockCapacity)
+      LI->reallocBlocks(*static_cast<LoopT *>(this),
+                        std::max(2 * BlockLen, 4u));
+    BlockData[BlockLen++] = BB;
   }
 
   /// interface to do reserve() for Blocks
-  void reserveBlocks(unsigned size) {
+  void reserveBlocks(unsigned Size) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.reserve(size);
+    if (BlockCapacity < Size)
+      LI->reallocBlocks(*static_cast<LoopT *>(this), Size);
   }
 
   /// interface to do reserve() for SubLoops
@@ -421,21 +422,18 @@ template <class BlockT, class LoopT> class LoopBase {
     SubLoops.reserve(Size);
   }
 
-  /// Capacity of the block list; input to an enclosing loop's reserveBlocks()
-  /// during construction, when this loop's list is not yet fully populated.
-  unsigned getBlocksCapacity() const { return Blocks.capacity(); }
-
   /// This method is used to move BB (which must be part of this loop) to be the
   /// loop header of the loop (the block that dominates all others).
   void moveToHeader(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    if (Blocks[0] == BB)
+    if (BlockData[0] == BB)
       return;
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
     for (unsigned i = 0;; ++i) {
-      assert(i != Blocks.size() && "Loop does not contain BB!");
-      if (Blocks[i] == BB) {
-        Blocks[i] = Blocks[0];
-        Blocks[0] = BB;
+      assert(i != BlockLen && "Loop does not contain BB!");
+      if (BlockData[i] == BB) {
+        BlockData[i] = BlockData[0];
+        BlockData[0] = BB;
         return;
       }
     }
@@ -446,9 +444,12 @@ template <class BlockT, class LoopT> class LoopBase {
   /// class.
   void removeBlockFromLoop(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    auto I = find(Blocks, BB);
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
+    MutableArrayRef<BlockT *> Blocks(BlockData, BlockLen);
+    auto *I = llvm::find(Blocks, BB);
     assert(I != Blocks.end() && "N is not in this list!");
-    Blocks.erase(I);
+    std::move(I + 1, Blocks.end(), I);
+    --BlockLen;
   }
 
   /// Verify loop structure
@@ -469,15 +470,10 @@ template <class BlockT, class LoopT> class LoopBase {
 
 protected:
   friend class LoopInfoBase<BlockT, LoopT>;
-  friend class PopulateLoopsDFS<BlockT, LoopT>;
 
   /// This creates an empty loop.
   LoopBase() : ParentLoop(nullptr) {}
 
-  explicit LoopBase(BlockT *BB) : ParentLoop(nullptr) {
-    Blocks.push_back(BB);
-  }
-
   // Since loop passes like SCEV are allowed to key analysis results off of
   // `Loop` pointers, we cannot re-use pointers within a loop pass manager.
   // This means loop passes should not be `delete` ing `Loop` objects directly
@@ -495,7 +491,10 @@ template <class BlockT, class LoopT> class LoopBase {
     IsInvalid = true;
 #endif
     SubLoops.clear();
-    Blocks.clear();
+    // The block storage is reclaimed by the owning LoopInfo.
+    BlockData = nullptr;
+    BlockLen = 0;
+    BlockCapacity = 0;
     ParentLoop = nullptr;
   }
 };
@@ -525,6 +524,12 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   unsigned BlockNumberEpoch;
 
   std::vector<LoopT *> TopLevelLoops;
+
+  // Shared reverse postorder layout of all blocks. Each initial loop is a slice
+  // of this array, subloop slices nested inside their parent's.
+  std::unique_ptr<BlockT *[]> BlockLayout;
+  unsigned BlockLayoutLen = 0;
+
   BumpPtrAllocator LoopAllocator;
 
   friend class LoopBase<BlockT, LoopT>;
@@ -540,9 +545,11 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   LoopInfoBase(LoopInfoBase &&Arg)
       : BBMap(std::move(Arg.BBMap)),
         TopLevelLoops(std::move(Arg.TopLevelLoops)),
+        BlockLayout(std::move(Arg.BlockLayout)),
         LoopAllocator(std::move(Arg.LoopAllocator)) {
     ParentPtr = Arg.ParentPtr;
     BlockNumberEpoch = Arg.BlockNumberEpoch;
+    BlockLayoutLen = Arg.BlockLayoutLen;
     resetLoopInfoOwners();
     // We have to clear the arguments top level loops as we've taken ownership.
     Arg.TopLevelLoops.clear();
@@ -556,9 +563,12 @@ template <class BlockT, class LoopT> class LoopInfoBase {
       L->~LoopT();
 
     TopLevelLoops = std::move(RHS.TopLevelLoops);
+    BlockLayout = std::move(RHS.BlockLayout);
+    BlockLayoutLen = RHS.BlockLayoutLen;
     LoopAllocator = std::move(RHS.LoopAllocator);
     resetLoopInfoOwners();
     RHS.TopLevelLoops.clear();
+    RHS.BlockLayoutLen = 0;
     return *this;
   }
 
@@ -568,12 +578,14 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     for (auto *L : TopLevelLoops)
       L->~LoopT();
     TopLevelLoops.clear();
+    BlockLayout.reset();
+    BlockLayoutLen = 0;
     LoopAllocator.Reset();
   }
 
-  template <typename... ArgsTy> LoopT *AllocateLoop(ArgsTy &&...Args) {
+  LoopT *AllocateLoop() {
     LoopT *Storage = LoopAllocator.Allocate<LoopT>();
-    LoopT *L = new (Storage) LoopT(std::forward<ArgsTy>(Args)...);
+    LoopT *L = new (Storage) LoopT();
     L->LI = this;
     return L;
   }
@@ -636,6 +648,49 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
+  /// AllocateLoop for analyze(): stash \p Header (see pendingHeader).
+  /// getHeader() only works once the layout carve has replaced the stash with
+  /// the loop's block list.
+  LoopT *allocateLoop(BlockT *Header) {
+    LoopT *L = AllocateLoop();
+    L->BlockData = reinterpret_cast<BlockT **>(Header);
+    return L;
+  }
+
+  /// The header of a loop under construction, stashed in BlockData until the
+  /// layout carve builds the block list.
+  static BlockT *pendingHeader(const LoopT *L) {
+    return reinterpret_cast<BlockT *>(L->BlockData);
+  }
+
+  void discoverAndMapSubloop(LoopT *L, BlockT *Header,
+                             ArrayRef<BlockT *> Backedges,
+                             const DominatorTreeBase<BlockT, false> &DomTree);
+
+  /// True if \p L's block list is a borrowed slice of BlockLayout rather than
+  /// a private allocation.
+  bool hasBorrowedBlocks(const LoopT &L) const {
+    return L.BlockData >= BlockLayout.get() &&
+           L.BlockData < BlockLayout.get() + BlockLayoutLen;
+  }
+
+  /// Replace \p L's block list with a private allocation of NewCapacity
+  /// slots. The old storage is abandoned in place so slices sharing it stay
+  /// intact; it is reclaimed when this LoopInfo is cleared.
+  void reallocBlocks(LoopT &L, unsigned NewCapacity) {
+    assert(NewCapacity >= L.BlockLen && "capacity below size");
+    BlockT **New = LoopAllocator.Allocate<BlockT *>(NewCapacity);
+    llvm::copy(L.getBlocks(), New);
+    L.BlockData = New;
+    L.BlockCapacity = NewCapacity;
+  }
+
+  /// Copy \p L's borrowed block list into private storage before a mutation.
+  void materializeBlocks(LoopT &L) {
+    if (hasBorrowedBlocks(L))
+      reallocBlocks(L, L.BlockLen);
+  }
+
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
@@ -672,7 +727,10 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// ancestors or descendants, and not the block-to-loop mapping.
   template <typename PredicateT>
   void removeBlocksIf(LoopT &L, PredicateT Pred) {
-    llvm::erase_if(L.Blocks, Pred);
+    materializeBlocks(L);
+    L.BlockLen = llvm::remove_if(
+                     MutableArrayRef<BlockT *>(L.BlockData, L.BlockLen), Pred) -
+                 L.BlockData;
   }
 
   /// Remove every block satisfying \p Pred from \p Start and each of its
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 170f05e9d3a49..ad36d3343c8a8 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -282,7 +282,7 @@ void LoopBase<BlockT, LoopT>::addBasicBlockToLoop(
     BlockT *NewBB, LoopInfoBase<BlockT, LoopT> &LIB) {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  if (!Blocks.empty()) {
+  if (!getBlocks().empty()) {
     auto SameHeader = LIB[getHeader()];
     assert(contains(SameHeader) && getHeader() == SameHeader->getHeader() &&
            "Incorrect LI specified for this loop!");
@@ -325,7 +325,7 @@ template <class BlockT, class LoopT>
 void LoopBase<BlockT, LoopT>::verifyLoop() const {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  assert(!Blocks.empty() && "Loop header is missing");
+  assert(!getBlocks().empty() && "Loop header is missing");
 
   // Setup for using a depth-first iterator to visit every block in the loop.
   SmallVector<BlockT *, 8> ExitBBs;
@@ -371,7 +371,7 @@ void LoopBase<BlockT, LoopT>::verifyLoop() const {
 
   if (VisitedBBs.size() != getNumBlocks()) {
     dbgs() << "The following blocks are unreachable in the loop: ";
-    for (auto *BB : Blocks) {
+    for (auto *BB : getBlocks()) {
       if (!VisitedBBs.count(BB)) {
         dbgs() << *BB << "\n";
       }
@@ -455,12 +455,11 @@ void LoopBase<BlockT, LoopT>::print(raw_ostream &OS, bool Verbose,
 /// this loop are mapped to this loop or a subloop. And all subloops within this
 /// loop have their parent loop set to this loop or a subloop.
 template <class BlockT, class LoopT>
-static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
-                                  LoopInfoBase<BlockT, LoopT> *LI,
-                                  const DomTreeBase<BlockT> &DomTree) {
+void LoopInfoBase<BlockT, LoopT>::discoverAndMapSubloop(
+    LoopT *L, BlockT *Header, ArrayRef<BlockT *> Backedges,
+    const DominatorTreeBase<BlockT, false> &DomTree) {
   using InvBlockTraits = GraphTraits<Inverse<BlockT *>>;
 
-  unsigned NumBlocks = 0;
   unsigned NumSubloops = 0;
 
   // Perform a backward CFG traversal using a worklist.
@@ -469,15 +468,14 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
     BlockT *PredBB = ReverseCFGWorklist.back();
     ReverseCFGWorklist.pop_back();
 
-    LoopT *Subloop = LI->getLoopFor(PredBB);
+    LoopT *Subloop = getLoopFor(PredBB);
     if (!Subloop) {
       if (!DomTree.isReachableFromEntry(PredBB))
         continue;
 
       // This is an undiscovered block. Map it to the current loop.
-      LI->changeLoopFor(PredBB, L);
-      ++NumBlocks;
-      if (PredBB == L->getHeader())
+      changeLoopFor(PredBB, L);
+      if (PredBB == Header)
         continue;
       // Push all block predecessors on the worklist.
       ReverseCFGWorklist.insert(ReverseCFGWorklist.end(),
@@ -494,84 +492,31 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
       // Discover a subloop of this loop.
       Subloop->setParentLoop(L);
       ++NumSubloops;
-      NumBlocks += Subloop->getBlocksCapacity();
-      PredBB = Subloop->getHeader();
+      PredBB = pendingHeader(Subloop);
       // Continue traversal along predecessors that are not loop-back edges from
       // within this subloop tree itself. Note that a predecessor may directly
       // reach another subloop that is not yet discovered to be a subloop of
       // this loop, which we must traverse.
       for (const auto Pred : inverse_children<BlockT *>(PredBB)) {
-        if (LI->getLoopFor(Pred) != Subloop)
+        if (getLoopFor(Pred) != Subloop)
           ReverseCFGWorklist.push_back(Pred);
       }
     }
   }
   L->reserveSubLoops(NumSubloops);
-  L->reserveBlocks(NumBlocks);
-}
-
-/// Populate all loop data in a stable order during a single forward DFS.
-template <class BlockT, class LoopT> class PopulateLoopsDFS {
-  using BlockTraits = GraphTraits<BlockT *>;
-  using SuccIterTy = typename BlockTraits::ChildIteratorType;
-
-  LoopInfoBase<BlockT, LoopT> *LI;
-
-public:
-  PopulateLoopsDFS(LoopInfoBase<BlockT, LoopT> *li) : LI(li) {}
-
-  void traverse(BlockT *EntryBlock);
-
-protected:
-  void insertIntoLoop(BlockT *Block);
-};
-
-/// Top-level driver for the forward DFS within the loop.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::traverse(BlockT *EntryBlock) {
-  for (BlockT *BB : post_order(EntryBlock))
-    insertIntoLoop(BB);
-}
-
-/// Add a single Block to its ancestor loops in PostOrder. If the block is a
-/// subloop header, add the subloop to its parent in PostOrder, then reverse the
-/// Block and Subloop vectors of the now complete subloop to achieve RPO.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::insertIntoLoop(BlockT *Block) {
-  LoopT *Subloop = LI->getLoopFor(Block);
-  if (Subloop && Block == Subloop->getHeader()) {
-    // We reach this point once per subloop after processing all the blocks in
-    // the subloop.
-    if (!Subloop->isOutermost())
-      Subloop->getParentLoop()->SubLoops.push_back(Subloop);
-    else
-      LI->addTopLevelLoop(Subloop);
-
-    // For convenience, Blocks and Subloops are inserted in postorder. Reverse
-    // the lists, except for the loop header, which is always at the beginning.
-    Subloop->reverseBlock(1);
-    std::reverse(Subloop->SubLoops.begin(), Subloop->SubLoops.end());
-
-    Subloop = Subloop->getParentLoop();
-  }
-  for (; Subloop; Subloop = Subloop->getParentLoop())
-    Subloop->addBlockEntry(Block);
 }
 
 /// Analyze LoopInfo discovers loops during a reverse preorder DominatorTree
 /// traversal interleaved with backward CFG traversals within each subloop
 /// (discoverAndMapSubloop). The backward traversal skips inner subloops, so
-/// this part of the algorithm is linear in the number of CFG edges. Subloop and
-/// Block vectors are then populated during a single forward CFG traversal
-/// (PopulateLoopDFS).
+/// this part of the algorithm is linear in the number of CFG edges.
 ///
-/// During the two CFG traversals each block is seen three times:
-/// 1) Discovered and mapped by a reverse CFG traversal.
-/// 2) Visited during a forward DFS CFG traversal.
-/// 3) Reverse-inserted in the loop in postorder following forward DFS.
-///
-/// The Block vectors are inclusive, so step 3 requires loop-depth number of
-/// insertions per block.
+/// The block lists are then built in one shared layout: a forward CFG postorder
+/// records the in-loop blocks, and a reverse walk (RPO) carves each loop a
+/// contiguous slice of its parent's, writing each block once to its innermost
+/// loop. Lists are header-first with each subloop's blocks contiguous, ordered
+/// by first appearance in RPO; SubLoops keep program order, TopLevelLoops
+/// reverse program order.
 template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
@@ -587,6 +532,7 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   for (const DomTreeNodeBase<BlockT> *Node : DomTree.nodes())
     PreorderNodes[Node->getDFSNumIn()] = Node;
 
+  bool HasLoops = false;
   for (const DomTreeNodeBase<BlockT> *DomNode : llvm::reverse(PreorderNodes)) {
     BlockT *Header = DomNode->getBlock();
     SmallVector<BlockT *, 4> Backedges;
@@ -600,14 +546,63 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const D...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Thu Jul 23 02:01:45 2026
From: llvmlistbot at llvm.org (Abid Qadeer)
Date: Thu, 23 Jul 2026 02:01:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][OpenMP] Preserve debug location in
 OpenMPIRBuilder calls. (PR #211254)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211254@github.com>
Message-ID: <6a61d879.5a0412d2.325e39.f7a4@mx.google.com>


https://github.com/abidh closed https://github.com/llvm/llvm-project/pull/211254

From llvmlistbot at llvm.org  Thu Jul 23 02:01:48 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:01:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d87c.1b0626d0.1ff4c3.5dad@mx.google.com>


https://github.com/jopperm requested changes to this pull request.

Overall approach LGTM, but I flagged some issues that need to be addressed before this can land.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:01:48 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:01:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d87c.a1accbb4.1ace1c.d78d@mx.google.com>


================
@@ -415,25 +415,21 @@ struct VectorContractBF16ToFMA
       Operation *accReadOp1 =
           traceToVectorReadLikeParentOperation(pairContractOp.getAcc());
 
-      // Iterate down to find the users of contact operations until it is store
-      // or transfer_write.
-      Operation *resultWriteOp0 =
-          traceToVectorWriteLikeUserOperation(contractOp.getResult());
-      Operation *resultWriteOp1 =
-          traceToVectorWriteLikeUserOperation(pairContractOp.getResult());
-
-      // Shuffle the accumulators of the contract operations.
-      LogicalResult readShuffle =
-          shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                 pairContractOp, nonUnitDim, accTy);
-
-      if (failed(readShuffle))
-        return rewriter.notifyMatchFailure(
-            contractOp, "Accumulator read is not by transfer_read or load");
+      if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
----------------
jopperm wrote:

Nit: `isa`.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:01:49 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:01:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d87d.1ba19891.1d5373.6b87@mx.google.com>


================
@@ -0,0 +1,115 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant vector.");
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    Value accValue = accReadOp->getResult(0);
+    Operation *firstUser = *accValue.getUsers().begin();
+    rewriter.setInsertionPoint(firstUser);
+
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Expects vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
+
+    accValue.replaceAllUsesWith(zeroVec);
+
+    // Adds the initial acc value with contract results before storing to acc
+    // matrix.
+    rewriter.setInsertionPoint(resultWriteOp);
+    Location locUser = resultWriteOp->getLoc();
+
+    Value addition;
+
+    if (llvm::isa<FloatType>(elemTy)) {
+      addition =
+          arith::AddFOp::create(rewriter, locUser, contractValue, accValue);
+    }
+
+    if (llvm::isa<IntegerType>(elemTy)) {
+      addition =
+          arith::AddIOp::create(rewriter, locUser, contractValue, accValue);
+    }
----------------
jopperm wrote:

Nits: No braces. Use if-else, and llvm_unreachable for any unsupported type.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:01:53 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:01:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d881.e624eb8d.12f676.c650@mx.google.com>


================
@@ -155,9 +174,21 @@ Operation *traceToVectorReadLikeParentOperation(Value v) {
   while (true) {
     // Case 1: Value defined by an operation
     if (Operation *defOp = v.getDefiningOp()) {
-      if (isa<vector::TransferReadOp, vector::LoadOp>(defOp))
+      if (isa<vector::TransferReadOp, vector::LoadOp, arith::ConstantOp>(defOp))
         return defOp;
 
+      if (isa<arith::ConstantOp>(defOp)) {
+        Attribute value = (dyn_cast<arith::ConstantOp>(defOp)).getValue();
----------------
jopperm wrote:

Nit: Use the `dyn_cast` already in the if-condition.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:03:02 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:03:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d8c6.0c05e68e.81651.21be@mx.google.com>


================
@@ -373,18 +373,22 @@ struct VectorContractToPackedTypeDotProduct
           return rewriter.notifyMatchFailure(
               contractOp, "The store/write operation of contract operation is "
                           "before the pair contract operation");
-        // Shuffle the accumulators of the contract operations.
-        LogicalResult readShuffle =
-            shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                   pairContractOp, nonUnitDimValue, accTy);
 
-        if (failed(readShuffle))
-          return rewriter.notifyMatchFailure(
-              contractOp, "Accumulator read is not by transfer_read or load");
+        if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
----------------
jopperm wrote:

Same as above.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:03:40 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:03:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d8ec.f06f4994.50b1b.3685@mx.google.com>


https://github.com/jopperm edited https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:03:53 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:03:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d8f9.4987c784.13f2de.896c@mx.google.com>


================
@@ -155,9 +174,21 @@ Operation *traceToVectorReadLikeParentOperation(Value v) {
   while (true) {
     // Case 1: Value defined by an operation
     if (Operation *defOp = v.getDefiningOp()) {
-      if (isa<vector::TransferReadOp, vector::LoadOp>(defOp))
+      if (isa<vector::TransferReadOp, vector::LoadOp, arith::ConstantOp>(defOp))
         return defOp;
 
+      if (isa<arith::ConstantOp>(defOp)) {
+        Attribute value = (dyn_cast<arith::ConstantOp>(defOp)).getValue();
+
+        if ((dyn_cast<IntegerAttr>(value)).getValue().isZero())
+          return defOp;
+
+        if ((dyn_cast<FloatAttr>(value)).getValue().isZero())
+          return defOp;
----------------
jopperm wrote:

As Copilot also detected, this will segfault.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:04:25 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:04:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d919.01e910a1.16b677.268f@mx.google.com>


================
@@ -0,0 +1,115 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant vector.");
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    Value accValue = accReadOp->getResult(0);
+    Operation *firstUser = *accValue.getUsers().begin();
+    rewriter.setInsertionPoint(firstUser);
+
+    auto vecTy = llvm::dyn_cast<VectorType>(accValue.getType());
+    if (!vecTy)
+      return rewriter.notifyMatchFailure(contractOp, "Expects vector type.");
+
+    Location loc = accReadOp->getLoc();
+    Type elemTy = vecTy.getElementType();
+
+    Value zeroScalar = arith::ConstantOp::create(rewriter, loc, elemTy,
+                                                 rewriter.getZeroAttr(elemTy));
+
+    Value zeroVec =
+        vector::BroadcastOp::create(rewriter, loc, vecTy, zeroScalar);
----------------
jopperm wrote:

Nit: It's probably folded right away, but why not construct a vector-typed constant directly?

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:04:45 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:04:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d92d.ca5545e3.326e2a.4fc9@mx.google.com>


================
@@ -0,0 +1,115 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant vector.");
+
+    if ((accReadOp->getBlock() == contractOp->getBlock()) ||
+        (resultWriteOp->getBlock() == contractOp->getBlock()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Acc read/write should be in a separate block.");
+
+    // Replace acc of a contraction operation with vector constant.
+    Value accValue = accReadOp->getResult(0);
+    Operation *firstUser = *accValue.getUsers().begin();
+    rewriter.setInsertionPoint(firstUser);
----------------
jopperm wrote:

Just curious, why here and not in front of the contract op?

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:05:09 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:05:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d945.6ddb6d48.32552.dab3@mx.google.com>


================
@@ -0,0 +1,115 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
+
+    if (dyn_cast<arith::ConstantOp>(accReadOp))
+      return rewriter.notifyMatchFailure(
+          contractOp,
+          "The input acc to contract is already a constant vector.");
----------------
jopperm wrote:

But what if it's not all zeros?

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:05:41 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:05:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d965.7270d2fd.1ec500.bffa@mx.google.com>


================
@@ -0,0 +1,115 @@
+//===- MoveAccumulatorForContractLoop.cpp ---------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/SCF/IR/SCF.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/Dialect/Vector/Utils/VectorUtils.h"
+#include "mlir/Dialect/X86/Transforms.h"
+#include "mlir/Dialect/X86/Utils/X86Utils.h"
+#include "mlir/Dialect/X86/X86Dialect.h"
+
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/Dominance.h"
+#include "mlir/IR/PatternMatch.h"
+#include "llvm/Support/Casting.h"
+
+#include "mlir/Pass/Pass.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+using namespace mlir::x86;
+
+namespace {
+// Transforms vector.contract(A, B, Acc) into vector.contract(A, B, 0) + Acc
+// to decouple the contraction computation from the accumulator update.
+struct MoveAccumulatorForContractLoop
+    : public OpRewritePattern<vector::ContractionOp> {
+  using OpRewritePattern<vector::ContractionOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(vector::ContractionOp contractOp,
+                                PatternRewriter &rewriter) const override {
+
+    if (contractOp.getKind() != vector::CombiningKind::ADD)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects add combining kind.");
+
+    Operation *accReadOp =
+        traceToVectorReadLikeParentOperation(contractOp.getAcc());
+
+    Value contractValue = contractionUsersAfterYield(contractOp.getResult());
+
+    if (!contractValue)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Final acc write might have multiple users.");
+
+    Operation *resultWriteOp = *contractValue.getUsers().begin();
+
+    if (!accReadOp || !resultWriteOp)
+      return rewriter.notifyMatchFailure(
+          contractOp, "Read/write from/to acc matrix is not by "
+                      "transfer_read/load/transfer_write/store ops.");
----------------
jopperm wrote:

Potential misnomer: `contractionUsersAfterYield` doesn't guarantee that the user is a `transfer_write` or `store` op, and the pattern doesn't require it IIUC.

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:05:52 2026
From: llvmlistbot at llvm.org (Julian Oppermann)
Date: Thu, 23 Jul 2026 02:05:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][x86] Decouple Accumulator from
 ADD-Based Vector Contractions (over loops) (PR #204327)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204327@github.com>
Message-ID: <6a61d970.d76d81af.3b68ca.b69f@mx.google.com>


================
@@ -415,25 +415,21 @@ struct VectorContractBF16ToFMA
       Operation *accReadOp1 =
           traceToVectorReadLikeParentOperation(pairContractOp.getAcc());
 
-      // Iterate down to find the users of contact operations until it is store
-      // or transfer_write.
-      Operation *resultWriteOp0 =
-          traceToVectorWriteLikeUserOperation(contractOp.getResult());
-      Operation *resultWriteOp1 =
-          traceToVectorWriteLikeUserOperation(pairContractOp.getResult());
-
-      // Shuffle the accumulators of the contract operations.
-      LogicalResult readShuffle =
-          shuffleAfterReadLikeOp(rewriter, accReadOp0, accReadOp1, contractOp,
-                                 pairContractOp, nonUnitDim, accTy);
-
-      if (failed(readShuffle))
-        return rewriter.notifyMatchFailure(
-            contractOp, "Accumulator read is not by transfer_read or load");
+      if (!(dyn_cast<arith::ConstantOp>(accReadOp0))) {
----------------
jopperm wrote:

Again, what if the accumulator was a non-zero constant?

https://github.com/llvm/llvm-project/pull/204327

From llvmlistbot at llvm.org  Thu Jul 23 02:26:04 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Thu, 23 Jul 2026 02:26:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][presburger] Avoid redundant
 zero-initialization in insertColumns (PR #199911)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/199911@github.com>
Message-ID: <6a61de2c.21a19891.105e40.9105@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `clang-aarch64-sve2-vla-2stage` running on `linaro-g4-02` while building `mlir` at step 19 "test-suite".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/199/builds/10220

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 19 (test-suite) failure: test (failure)
******************** TEST 'test-suite :: Fortran/gfortran/regression/gomp/gfortran-regression-compile-regression__gomp__imperfect2_f90.test' FAILED ********************

/home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/tools/timeit-target --timeout 7200 --limit-core 0 --limit-cpu 7200 --limit-file-size 209715200 --limit-rss-size 838860800 --redirect-input /dev/null --summary /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/Output/gfortran-regression-compile-regression__gomp__imperfect2_f90.test.time /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/../..//dummy
/usr/bin/diff /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/../..//gfortran-compile-empty.reference.out /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/gfortran-regression-compile-regression__gomp__imperfect2_f90.out
0a1,13
> error: Semantic errors in /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/test-suite/Fortran/gfortran/regression/gomp/imperfect2.f90
> /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/test-suite/Fortran/gfortran/regression/gomp/imperfect2.f90:38:9: error: This construct requires a perfect nest of depth 3, but the associated nest is a perfect nest of depth 1
>     !$omp do collapse(3) 
>           ^^
> /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/test-suite/Fortran/gfortran/regression/gomp/imperfect2.f90:43:7: because: This code prevents perfect nesting
>         call f1 (2, j)
>         ^^^^^^^^^^^^^^
> /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/test-suite/Fortran/gfortran/regression/gomp/imperfect2.f90:40:5: because: This code prevents perfect nesting
>       call f1 (1, i)
>       ^^^^^^^^^^^^^^
> /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/test-suite/Fortran/gfortran/regression/gomp/imperfect2.f90:38:12: because: COLLAPSE clause was specified with argument 3
>     !$omp do collapse(3) 
>              ^^^^^^^^^^^
\ No newline at end of file

+ /usr/bin/diff /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/../..//gfortran-compile-empty.reference.out /home/tcwg-buildbot/worker/clang-aarch64-sve2-vla-2stage/test/sandbox/build/Fortran/gfortran/regression/gomp/gfortran-regression-compile-regression__gomp__imperfect2_f90.out

********************


```

</details>

https://github.com/llvm/llvm-project/pull/199911

From llvmlistbot at llvm.org  Thu Jul 23 03:01:35 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 23 Jul 2026 03:01:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a61e67f.d2a41782.36e32c.d8fe@mx.google.com>


https://github.com/nujaa edited https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 03:01:35 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 23 Jul 2026 03:01:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a61e67f.ecf2568a.2a6521.5631@mx.google.com>


https://github.com/nujaa approved this pull request.

LGTM. I guess we have an additional task in case you have spare cycles. 

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 03:01:36 2026
From: llvmlistbot at llvm.org (Hugo Trachino)
Date: Thu, 23 Jul 2026 03:01:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a61e680.f0e68a03.2239ea.61f3@mx.google.com>


================
@@ -1196,6 +1196,30 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_scalar_input(
+    %input: f32, %init: tensor<8x16xf32>) -> tensor<8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op operand #0 must be memref of any non-token type values or ranked tensor of any non-token type values, but got 'f32'}}
----------------
nujaa wrote:

Would it make sense to add support ? 

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 03:07:34 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Thu, 23 Jul 2026 03:07:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a61e7e6.6edd682e.2b3f36.d737@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Thu Jul 23 03:10:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:10:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e8a6.e8e5c93d.19ef3f.83e0@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
PragmaTwice wrote:

> Could the load just do nothing and return?

That's quite tricky.

Let me share an example about why "do nothing" may actually not be what users want:
```python
class  MyDialect(Dialect):
  ...

class Op1(MyDialect.Operation):
  ...

c = Context()
with c:
  MyDialect.load() # first time to load

class Op2(MyDialect.Operation):
  ...

with c:
  MyDialect.load() # second time to load: should Op2 be loaded?
```

Note that in the second time, user may expect that `Op2` is loaded into the dialect!

And if the context is changed in the second time, then everything works fine (we actually have some test cases in such pattern).

```
class  MyDialect(Dialect):
  ...

class Op1(MyDialect.Operation):
  ...

with Context():
  MyDialect.load() # first time to load: just Op1

class Op2(MyDialect.Operation):
  ...

with Context():
  MyDialect.load() # second time to load: Op1 and Op2
```

> if you want to keep current approach, could it raise more specific exception like DialectReloadingError or DialectAlreadyLoadedError?

Sure.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:10:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:10:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e8b1.f804b674.76685.3235@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:11:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:11:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e8d8.dd7c1686.181a5.67c9@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:12:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:12:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e8f1.8f0cbdd2.ba895.2de1@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:12:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:12:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e91a.098bdcf5.2f0341.bc47@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:14:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:14:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e972.6b2462aa.2c0d42.6960@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
----------------
PragmaTwice wrote:

Ahh currently lots of test cases relies on `._mlir_module` to do some check. I think we can leave it as is for this PR to avoid many changes?

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:14:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:14:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61e983.e69a274b.295452.dc25@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:25:59 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 03:25:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61ec37.bd93e731.2ac69c.2f6e@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
----------------
adam-smnk wrote:

Alright 👍

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:28:42 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:28:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61ecda.21a19891.105e40.ae9e@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:29:40 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 03:29:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61ed14.f1c1923d.d9163.08f8@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
adam-smnk wrote:

First case is a good example indeed.
Just to clarify, is there any way to reload dialect as in:
```python
class Op2(MyDialect.Operation):
  ...

with c:
  MyDialect.load() # second time to load: should Op2 be loaded?
```

I guess no?

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:33:36 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Thu, 23 Jul 2026 03:33:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a61ee00.1ba19891.1d5373.dc18@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Thu Jul 23 03:37:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:37:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61eed8.f0e68a03.2239ea.3b9b@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
PragmaTwice wrote:

You mean how to make the `Op2` loaded in the second time? I think it requires some changes to some MLIR API like `irdl::loadDialects`. Theoretically it can work but just that nobody tries and tunes.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:37:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:37:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61eee9.797536ea.2e6c5.b757@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:39:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 03:39:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61ef4a.4a342cbc.1aeb63.83ac@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 03:57:25 2026
From: llvmlistbot at llvm.org (Renato Golin)
Date: Thu, 23 Jul 2026 03:57:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61f395.824e11a6.322560.9e0b@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
rengolin wrote:

I think the main problem here is that in C++ dialects are static at runtime, so doing this would not make sense (or would always be a noop). If there's a way to detect if the dialect has changed since (some boolean flag that sets to true when loaded and false when changed), then we can reserve the assertion above to only when the flag is false (ie. dialect changed and tried to reload).

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 04:06:13 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Thu, 23 Jul 2026 04:06:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61f5a5.83ce629a.2277d7.03b5@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
adam-smnk wrote:

Handling proper reloading probably needs some core C++ changes given that forced reloads assert without this PR.

For our current needs, the fix is fine and raising exception makes sense.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 04:08:46 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Thu, 23 Jul 2026 04:08:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add worklist comparator to tile and fuse
	options (PR #211523)
Message-ID: <llvm.org/llvm/llvm-project/pull/211523@github.com>


https://github.com/HagaiLevHacohen created https://github.com/llvm/llvm-project/pull/211523

## Summary

`tileConsumerAndFuseProducersUsingSCF` currently processes its `tensor.extract_slice` worklist in FIFO order, resulting in a breadth-first traversal.

This MR adds an optional comparator to `SCFTileAndFuseOptions`, allowing callers to prioritize worklist entries. The default remains FIFO. Our intended use is a topological consumer-to-producer traversal.

## Motivation

Consider this dependency graph, where arrows point from a consumer to its producer \[For a pseudo MLIR example - see below\]:

```text
1 → 2 → 3 → 4 → 5
└────────→ 4 → 5
```

Node 1 consumes nodes 2 and 4, forming a skip connection. Suppose we tile node 1 and fuse the entire producer graph, and nodes 1 and 3 request the same slice from node 4.

With FIFO processing, the fusion order is:

```text
2, 4, 3, 5
```

Node 4 is processed before node 3. Consequently:

1. Processing node 1 creates an extract slice from node 4.
2. That slice is processed and removed before node 3 is fused.
3. Fusing node 3 creates an identical extract slice from node 4.

Because the identical slices never coexist in the worklist, cleanup patterns cannot deduplicate them.

When this skip-connection pattern is repeated, redundant slices and fusion work grow exponentially. In our workloads, this causes compilation time to explode and may prevent compilation from completing.

With topological prioritization, the order becomes:

```text
2, 3, 4, 5
```

Node 3 is processed before node 4. Therefore, both identical slices from node 4 coexist before either is processed, allowing cleanup patterns to eliminate the duplicate.

This change lets callers select such an ordering while preserving FIFO behavior by default.

## Pseudo MLIR Example

```mlir
// Pseudo-MLIR; operation details and types are omitted.

// Node 5
%node5 = linalg.fill ...                     

// Node 4
%node4 = linalg.generic ins(%node5) ...

// Node 3
%node3 = linalg.generic ins(%node4) ...

// Node 2
%node2 = linalg.generic ins(%node3) ...

// Node 1: contains the skip connection to node 4.
%node1 = linalg.add ins(%node2, %node4) ...
```

When node 1 is tiled, it creates a slice from node 4:

```mlir
%slice_from_node1 = tensor.extract_slice %node4[%offset] [%size] [1]
```

Later, fusing node 3 creates the same slice:

```mlir
%slice_from_node3 = tensor.extract_slice %node4[%offset] [%size] [1]
```

With FIFO ordering, `%slice_from_node1` is processed before `%slice_from_node3` is created. With topological ordering, both slices coexist, allowing cleanup patterns to deduplicate them.

>From 389b3b7cd4326e8f3f076ac160c8d99693b5b59b Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Wed, 22 Jul 2026 15:23:09 +0300
Subject: [PATCH] Adding to tile and fuse options a worklist comparator

---
 .../SCF/Transforms/TileUsingInterface.h       | 15 +++
 .../SCF/Transforms/TileUsingInterface.cpp     | 30 +++++-
 .../tile-fuse-and-yield-using-interface.mlir  | 98 +++++++++++++++++++
 .../TestTilingInterfaceTransformOps.cpp       | 15 ++-
 .../TestTilingInterfaceTransformOps.td        | 10 +-
 5 files changed, 162 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 0bd5d19b136d0..125ff0edb7a89 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -312,6 +312,21 @@ struct SCFTileAndFuseOptions {
     return *this;
   }
 
+    /// Comparator used to select the next `tensor.extract_slice` to process from
+    /// the fusion worklist. Returns true if `lhs` should be processed before
+    /// `rhs`. This allows callers to enforce a desired tiling order. For example,
+    /// to process producers in topological order, a producer that is an ancestor
+    /// of another producer in the defining-op chain can be ordered before it.
+    /// By default, the worklist is processed in FIFO order.
+    using TileOrderControlFnTy =
+        std::function<bool(tensor::ExtractSliceOp lhs,
+                          tensor::ExtractSliceOp rhs)>;
+    TileOrderControlFnTy tileOrderControlFn = nullptr;
+    SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
+      tileOrderControlFn = std::move(controlFn);
+      return *this;
+    }
+
   /// An optional set of rewrite patterns to apply to the results of tiling
   /// before fusion. This will track deleted and newly inserted
   /// `tensor.extract_slice` ops and update the worklist.
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index ad4aff893a03a..00a27b4dc66c5 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -1641,6 +1641,14 @@ class SliceTrackingListener : public RewriterBase::Listener {
   /// Remove the operation from the worklist.
   void notifyOperationReplaced(Operation *op, ValueRange replacement) override;
 
+  /// Pop the next slice to process from the worklist. When
+  /// `tileOrderControlFn` is set, returns the preferred slice according to the
+  /// callback. Otherwise pops from the front (FIFO). The worklist must be
+  /// non-empty.
+  tensor::ExtractSliceOp popNext(
+    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+        &tileOrderControlFn);
+
   /// The worklist for this transformation keeps track of the slices to visit
   /// next for fusion.
   std::deque<tensor::ExtractSliceOp> worklist;
@@ -1707,6 +1715,24 @@ void SliceTrackingListener::notifyOperationReplaced(Operation *op,
   removeOp(op);
 }
 
+tensor::ExtractSliceOp SliceTrackingListener::popNext(
+  const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+      &tileOrderControlFn) {
+assert(!worklist.empty() && "expected non-empty worklist");
+if (!tileOrderControlFn) {
+  auto slice = worklist.front();
+  worklist.pop_front();
+  return slice;
+}
+auto it = llvm::min_element(worklist, [&](tensor::ExtractSliceOp lhs,
+                                          tensor::ExtractSliceOp rhs) {
+  return tileOrderControlFn(lhs, rhs);
+});
+auto slice = *it;
+worklist.erase(it);
+return slice;
+}
+
 //===----------------------------------------------------------------------===//
 // ReplacementListener
 //===----------------------------------------------------------------------===//
@@ -1814,8 +1840,8 @@ mlir::scf::tileConsumerAndFuseProducersUsingSCF(
   }
   OpBuilder::InsertionGuard g(rewriter);
   while (!sliceTracker.worklist.empty()) {
-    auto candidateSlice = sliceTracker.worklist.front();
-    sliceTracker.worklist.pop_front();
+    tensor::ExtractSliceOp candidateSlice =
+        sliceTracker.popNext(options.tileOrderControlFn);
 
     auto [fusableProducer, destinationInitArg] =
         getUntiledProducerFromSliceSource(&candidateSlice.getSourceMutable(),
diff --git a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
index 3c0ada9d2cabc..c11c7c4a5fc50 100644
--- a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
+++ b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
@@ -120,3 +120,101 @@ module attributes {transform.with_named_sequence} {
 //      CHECK:     %[[INSERT2:.+]] = tensor.insert_slice %[[GENERIC_TILE]]#1 into %[[ITERARG2]][0, %[[IV]]]
 //      CHECK:     scf.yield %[[INSERT0]], %[[INSERT1]], %[[INSERT2]]
 //      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#2, %[[RESULT]]#0
+
+// -----
+
+// Verify that the default FIFO worklist reconstructs the earlier `linalg.fill`
+// before the later `linalg.copy`. Their loop results are therefore passed to
+// the downstream `linalg.mul` as result #1 and result #2, respectively.
+
+// CHECK-LABEL: func.func @worklist_fifo_order(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#1, %[[RESULT]]#2
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+func.func @worklist_fifo_order(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Verify that preferring later producers reconstructs the `linalg.copy` before
+// the `linalg.fill`. Their loop result numbers are reversed, but the downstream
+// `linalg.mul` still receives the fill and copy values in its original order.
+
+// CHECK-LABEL: func.func @worklist_prefer_later_producers(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#2, %[[RESULT]]#1
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+
+func.func @worklist_prefer_later_producers(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+        worklist_prefer_later_producers true
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index 9467c925e543c..d891e576b7dd4 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -64,7 +64,9 @@ static LogicalResult
 applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
                       Range &&payloadOps, unsigned numLoops,
                       scf::SCFTilingOptions tilingOptions,
-                      TransformResults &transformResults) {
+                      TransformResults &transformResults,
+                      scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+                          tileOrderControlFn = nullptr) {
   SmallVector<Operation *> tiledOps;
   SmallVector<SmallVector<Operation *>> loopOps(numLoops);
 
@@ -98,6 +100,7 @@ applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
           yieldProducerReplacement};
     };
     tileAndFuseOptions.setFusionControlFn(controlFn);
+    tileAndFuseOptions.setTileOrderControlFn(tileOrderControlFn);
 
     rewriter.setInsertionPoint(target);
     FailureOr<scf::SCFTileAndFuseResult> tiledResults =
@@ -159,10 +162,18 @@ transform::TestFuseAndYieldOp::apply(TransformRewriter &rewriter,
     tilingOptions.setLoopType(scf::SCFTilingOptions::LoopType::ForallOp);
   }
 
+  scf::SCFTileAndFuseOptions::TileOrderControlFnTy tileOrderControlFn;
+  if (getWorklistPreferLaterProducers()) {
+    tileOrderControlFn = [](tensor::ExtractSliceOp lhs,
+                            tensor::ExtractSliceOp rhs) {
+      return rhs.getOperation()->isBeforeInBlock(lhs.getOperation());
+    };
+  }
+
   LogicalResult result = applyTileAndFuseToAll(
       rewriter, getOperation(), state.getPayloadOps(getTarget()),
       tileSizes.size() - llvm::count(tileSizes, 0), tilingOptions,
-      transformResults);
+      transformResults, tileOrderControlFn);
   return failed(result) ? DiagnosedSilenceableFailure::definiteFailure()
                         : DiagnosedSilenceableFailure::success();
 }
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
index efa16212f0fef..03032e7efb7ca 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
@@ -30,6 +30,9 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     producers greedily using the options provided as attributes.
     It also yields some of the fused producers for testing.
 
+    The `worklist_prefer_later_producers` attribute processes slices that occur
+    later in the tiled IR before earlier slices.
+
     On success returns the tiled operations as well as generated loops. Emits
     a definite failure if tiling fails.
   }];
@@ -38,13 +41,16 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     (ins TransformHandleTypeInterface:$target,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_sizes,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_interchange,
-        DefaultValuedAttr<BoolAttr, "false">:$use_forall);
+        DefaultValuedAttr<BoolAttr, "false">:$use_forall,
+        DefaultValuedAttr<BoolAttr, "false">:$worklist_prefer_later_producers);
   let results = (outs TransformHandleTypeInterface:$transfomed,
       Variadic<TransformHandleTypeInterface>:$loops);
 
   let assemblyFormat = [{
     $target ($tile_sizes^)? (`interchange` $tile_interchange^)?
-    (`use_forall` $use_forall^)? attr-dict 
+    (`use_forall` $use_forall^)?
+    (`worklist_prefer_later_producers` $worklist_prefer_later_producers^)?
+    attr-dict
     `:` functional-type(operands, results)
   }];
 }


From llvmlistbot at llvm.org  Thu Jul 23 04:09:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 04:09:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add worklist comparator to tile and fuse
	options (PR #211523)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211523@github.com>
Message-ID: <6a61f660.7d702ed2.2f6c05.9f52@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Hagai Lev Hacohen (HagaiLevHacohen)

<details>
<summary>Changes</summary>

## Summary

`tileConsumerAndFuseProducersUsingSCF` currently processes its `tensor.extract_slice` worklist in FIFO order, resulting in a breadth-first traversal.

This MR adds an optional comparator to `SCFTileAndFuseOptions`, allowing callers to prioritize worklist entries. The default remains FIFO. Our intended use is a topological consumer-to-producer traversal.

## Motivation

Consider this dependency graph, where arrows point from a consumer to its producer \[For a pseudo MLIR example - see below\]:

```text
1 → 2 → 3 → 4 → 5
└────────→ 4 → 5
```

Node 1 consumes nodes 2 and 4, forming a skip connection. Suppose we tile node 1 and fuse the entire producer graph, and nodes 1 and 3 request the same slice from node 4.

With FIFO processing, the fusion order is:

```text
2, 4, 3, 5
```

Node 4 is processed before node 3. Consequently:

1. Processing node 1 creates an extract slice from node 4.
2. That slice is processed and removed before node 3 is fused.
3. Fusing node 3 creates an identical extract slice from node 4.

Because the identical slices never coexist in the worklist, cleanup patterns cannot deduplicate them.

When this skip-connection pattern is repeated, redundant slices and fusion work grow exponentially. In our workloads, this causes compilation time to explode and may prevent compilation from completing.

With topological prioritization, the order becomes:

```text
2, 3, 4, 5
```

Node 3 is processed before node 4. Therefore, both identical slices from node 4 coexist before either is processed, allowing cleanup patterns to eliminate the duplicate.

This change lets callers select such an ordering while preserving FIFO behavior by default.

## Pseudo MLIR Example

```mlir
// Pseudo-MLIR; operation details and types are omitted.

// Node 5
%node5 = linalg.fill ...                     

// Node 4
%node4 = linalg.generic ins(%node5) ...

// Node 3
%node3 = linalg.generic ins(%node4) ...

// Node 2
%node2 = linalg.generic ins(%node3) ...

// Node 1: contains the skip connection to node 4.
%node1 = linalg.add ins(%node2, %node4) ...
```

When node 1 is tiled, it creates a slice from node 4:

```mlir
%slice_from_node1 = tensor.extract_slice %node4[%offset] [%size] [1]
```

Later, fusing node 3 creates the same slice:

```mlir
%slice_from_node3 = tensor.extract_slice %node4[%offset] [%size] [1]
```

With FIFO ordering, `%slice_from_node1` is processed before `%slice_from_node3` is created. With topological ordering, both slices coexist, allowing cleanup patterns to deduplicate them.

---
Full diff: https://github.com/llvm/llvm-project/pull/211523.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h (+15) 
- (modified) mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp (+28-2) 
- (modified) mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir (+98) 
- (modified) mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp (+13-2) 
- (modified) mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td (+8-2) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 0bd5d19b136d0..125ff0edb7a89 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -312,6 +312,21 @@ struct SCFTileAndFuseOptions {
     return *this;
   }
 
+    /// Comparator used to select the next `tensor.extract_slice` to process from
+    /// the fusion worklist. Returns true if `lhs` should be processed before
+    /// `rhs`. This allows callers to enforce a desired tiling order. For example,
+    /// to process producers in topological order, a producer that is an ancestor
+    /// of another producer in the defining-op chain can be ordered before it.
+    /// By default, the worklist is processed in FIFO order.
+    using TileOrderControlFnTy =
+        std::function<bool(tensor::ExtractSliceOp lhs,
+                          tensor::ExtractSliceOp rhs)>;
+    TileOrderControlFnTy tileOrderControlFn = nullptr;
+    SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
+      tileOrderControlFn = std::move(controlFn);
+      return *this;
+    }
+
   /// An optional set of rewrite patterns to apply to the results of tiling
   /// before fusion. This will track deleted and newly inserted
   /// `tensor.extract_slice` ops and update the worklist.
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index ad4aff893a03a..00a27b4dc66c5 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -1641,6 +1641,14 @@ class SliceTrackingListener : public RewriterBase::Listener {
   /// Remove the operation from the worklist.
   void notifyOperationReplaced(Operation *op, ValueRange replacement) override;
 
+  /// Pop the next slice to process from the worklist. When
+  /// `tileOrderControlFn` is set, returns the preferred slice according to the
+  /// callback. Otherwise pops from the front (FIFO). The worklist must be
+  /// non-empty.
+  tensor::ExtractSliceOp popNext(
+    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+        &tileOrderControlFn);
+
   /// The worklist for this transformation keeps track of the slices to visit
   /// next for fusion.
   std::deque<tensor::ExtractSliceOp> worklist;
@@ -1707,6 +1715,24 @@ void SliceTrackingListener::notifyOperationReplaced(Operation *op,
   removeOp(op);
 }
 
+tensor::ExtractSliceOp SliceTrackingListener::popNext(
+  const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+      &tileOrderControlFn) {
+assert(!worklist.empty() && "expected non-empty worklist");
+if (!tileOrderControlFn) {
+  auto slice = worklist.front();
+  worklist.pop_front();
+  return slice;
+}
+auto it = llvm::min_element(worklist, [&](tensor::ExtractSliceOp lhs,
+                                          tensor::ExtractSliceOp rhs) {
+  return tileOrderControlFn(lhs, rhs);
+});
+auto slice = *it;
+worklist.erase(it);
+return slice;
+}
+
 //===----------------------------------------------------------------------===//
 // ReplacementListener
 //===----------------------------------------------------------------------===//
@@ -1814,8 +1840,8 @@ mlir::scf::tileConsumerAndFuseProducersUsingSCF(
   }
   OpBuilder::InsertionGuard g(rewriter);
   while (!sliceTracker.worklist.empty()) {
-    auto candidateSlice = sliceTracker.worklist.front();
-    sliceTracker.worklist.pop_front();
+    tensor::ExtractSliceOp candidateSlice =
+        sliceTracker.popNext(options.tileOrderControlFn);
 
     auto [fusableProducer, destinationInitArg] =
         getUntiledProducerFromSliceSource(&candidateSlice.getSourceMutable(),
diff --git a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
index 3c0ada9d2cabc..c11c7c4a5fc50 100644
--- a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
+++ b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
@@ -120,3 +120,101 @@ module attributes {transform.with_named_sequence} {
 //      CHECK:     %[[INSERT2:.+]] = tensor.insert_slice %[[GENERIC_TILE]]#1 into %[[ITERARG2]][0, %[[IV]]]
 //      CHECK:     scf.yield %[[INSERT0]], %[[INSERT1]], %[[INSERT2]]
 //      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#2, %[[RESULT]]#0
+
+// -----
+
+// Verify that the default FIFO worklist reconstructs the earlier `linalg.fill`
+// before the later `linalg.copy`. Their loop results are therefore passed to
+// the downstream `linalg.mul` as result #1 and result #2, respectively.
+
+// CHECK-LABEL: func.func @worklist_fifo_order(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#1, %[[RESULT]]#2
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+func.func @worklist_fifo_order(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Verify that preferring later producers reconstructs the `linalg.copy` before
+// the `linalg.fill`. Their loop result numbers are reversed, but the downstream
+// `linalg.mul` still receives the fill and copy values in its original order.
+
+// CHECK-LABEL: func.func @worklist_prefer_later_producers(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#2, %[[RESULT]]#1
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+
+func.func @worklist_prefer_later_producers(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+        worklist_prefer_later_producers true
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index 9467c925e543c..d891e576b7dd4 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -64,7 +64,9 @@ static LogicalResult
 applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
                       Range &&payloadOps, unsigned numLoops,
                       scf::SCFTilingOptions tilingOptions,
-                      TransformResults &transformResults) {
+                      TransformResults &transformResults,
+                      scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+                          tileOrderControlFn = nullptr) {
   SmallVector<Operation *> tiledOps;
   SmallVector<SmallVector<Operation *>> loopOps(numLoops);
 
@@ -98,6 +100,7 @@ applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
           yieldProducerReplacement};
     };
     tileAndFuseOptions.setFusionControlFn(controlFn);
+    tileAndFuseOptions.setTileOrderControlFn(tileOrderControlFn);
 
     rewriter.setInsertionPoint(target);
     FailureOr<scf::SCFTileAndFuseResult> tiledResults =
@@ -159,10 +162,18 @@ transform::TestFuseAndYieldOp::apply(TransformRewriter &rewriter,
     tilingOptions.setLoopType(scf::SCFTilingOptions::LoopType::ForallOp);
   }
 
+  scf::SCFTileAndFuseOptions::TileOrderControlFnTy tileOrderControlFn;
+  if (getWorklistPreferLaterProducers()) {
+    tileOrderControlFn = [](tensor::ExtractSliceOp lhs,
+                            tensor::ExtractSliceOp rhs) {
+      return rhs.getOperation()->isBeforeInBlock(lhs.getOperation());
+    };
+  }
+
   LogicalResult result = applyTileAndFuseToAll(
       rewriter, getOperation(), state.getPayloadOps(getTarget()),
       tileSizes.size() - llvm::count(tileSizes, 0), tilingOptions,
-      transformResults);
+      transformResults, tileOrderControlFn);
   return failed(result) ? DiagnosedSilenceableFailure::definiteFailure()
                         : DiagnosedSilenceableFailure::success();
 }
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
index efa16212f0fef..03032e7efb7ca 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
@@ -30,6 +30,9 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     producers greedily using the options provided as attributes.
     It also yields some of the fused producers for testing.
 
+    The `worklist_prefer_later_producers` attribute processes slices that occur
+    later in the tiled IR before earlier slices.
+
     On success returns the tiled operations as well as generated loops. Emits
     a definite failure if tiling fails.
   }];
@@ -38,13 +41,16 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     (ins TransformHandleTypeInterface:$target,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_sizes,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_interchange,
-        DefaultValuedAttr<BoolAttr, "false">:$use_forall);
+        DefaultValuedAttr<BoolAttr, "false">:$use_forall,
+        DefaultValuedAttr<BoolAttr, "false">:$worklist_prefer_later_producers);
   let results = (outs TransformHandleTypeInterface:$transfomed,
       Variadic<TransformHandleTypeInterface>:$loops);
 
   let assemblyFormat = [{
     $target ($tile_sizes^)? (`interchange` $tile_interchange^)?
-    (`use_forall` $use_forall^)? attr-dict 
+    (`use_forall` $use_forall^)?
+    (`worklist_prefer_later_producers` $worklist_prefer_later_producers^)?
+    attr-dict
     `:` functional-type(operands, results)
   }];
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/211523

From llvmlistbot at llvm.org  Thu Jul 23 04:10:25 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 04:10:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add worklist comparator to tile and fuse
	options (PR #211523)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211523@github.com>
Message-ID: <6a61f6a1.bcf7152f.18738b.c8a4@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions h,cpp -- mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 125ff0edb..034bfbec5 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -312,20 +312,19 @@ struct SCFTileAndFuseOptions {
     return *this;
   }
 
-    /// Comparator used to select the next `tensor.extract_slice` to process from
-    /// the fusion worklist. Returns true if `lhs` should be processed before
-    /// `rhs`. This allows callers to enforce a desired tiling order. For example,
-    /// to process producers in topological order, a producer that is an ancestor
-    /// of another producer in the defining-op chain can be ordered before it.
-    /// By default, the worklist is processed in FIFO order.
-    using TileOrderControlFnTy =
-        std::function<bool(tensor::ExtractSliceOp lhs,
-                          tensor::ExtractSliceOp rhs)>;
-    TileOrderControlFnTy tileOrderControlFn = nullptr;
-    SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
-      tileOrderControlFn = std::move(controlFn);
-      return *this;
-    }
+  /// Comparator used to select the next `tensor.extract_slice` to process from
+  /// the fusion worklist. Returns true if `lhs` should be processed before
+  /// `rhs`. This allows callers to enforce a desired tiling order. For example,
+  /// to process producers in topological order, a producer that is an ancestor
+  /// of another producer in the defining-op chain can be ordered before it.
+  /// By default, the worklist is processed in FIFO order.
+  using TileOrderControlFnTy = std::function<bool(tensor::ExtractSliceOp lhs,
+                                                  tensor::ExtractSliceOp rhs)>;
+  TileOrderControlFnTy tileOrderControlFn = nullptr;
+  SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
+    tileOrderControlFn = std::move(controlFn);
+    return *this;
+  }
 
   /// An optional set of rewrite patterns to apply to the results of tiling
   /// before fusion. This will track deleted and newly inserted
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index 00a27b4dc..b856174b7 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -1645,9 +1645,9 @@ public:
   /// `tileOrderControlFn` is set, returns the preferred slice according to the
   /// callback. Otherwise pops from the front (FIFO). The worklist must be
   /// non-empty.
-  tensor::ExtractSliceOp popNext(
-    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-        &tileOrderControlFn);
+  tensor::ExtractSliceOp
+  popNext(const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+              &tileOrderControlFn);
 
   /// The worklist for this transformation keeps track of the slices to visit
   /// next for fusion.
@@ -1716,22 +1716,22 @@ void SliceTrackingListener::notifyOperationReplaced(Operation *op,
 }
 
 tensor::ExtractSliceOp SliceTrackingListener::popNext(
-  const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-      &tileOrderControlFn) {
-assert(!worklist.empty() && "expected non-empty worklist");
-if (!tileOrderControlFn) {
-  auto slice = worklist.front();
-  worklist.pop_front();
+    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+        &tileOrderControlFn) {
+  assert(!worklist.empty() && "expected non-empty worklist");
+  if (!tileOrderControlFn) {
+    auto slice = worklist.front();
+    worklist.pop_front();
+    return slice;
+  }
+  auto it = llvm::min_element(
+      worklist, [&](tensor::ExtractSliceOp lhs, tensor::ExtractSliceOp rhs) {
+        return tileOrderControlFn(lhs, rhs);
+      });
+  auto slice = *it;
+  worklist.erase(it);
   return slice;
 }
-auto it = llvm::min_element(worklist, [&](tensor::ExtractSliceOp lhs,
-                                          tensor::ExtractSliceOp rhs) {
-  return tileOrderControlFn(lhs, rhs);
-});
-auto slice = *it;
-worklist.erase(it);
-return slice;
-}
 
 //===----------------------------------------------------------------------===//
 // ReplacementListener
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index d891e576b..42139c110 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -60,13 +60,12 @@ static llvm::SmallDenseSet<Operation *> collectTiledAndFusedOps(Operation *op) {
 /// Apply a tile and fuse transformation to all payload ops and store both the
 /// tiled operation as well as the created tile loops.
 template <typename Range>
-static LogicalResult
-applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
-                      Range &&payloadOps, unsigned numLoops,
-                      scf::SCFTilingOptions tilingOptions,
-                      TransformResults &transformResults,
-                      scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-                          tileOrderControlFn = nullptr) {
+static LogicalResult applyTileAndFuseToAll(
+    RewriterBase &rewriter, Operation *transformOp, Range &&payloadOps,
+    unsigned numLoops, scf::SCFTilingOptions tilingOptions,
+    TransformResults &transformResults,
+    scf::SCFTileAndFuseOptions::TileOrderControlFnTy tileOrderControlFn =
+        nullptr) {
   SmallVector<Operation *> tiledOps;
   SmallVector<SmallVector<Operation *>> loopOps(numLoops);
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/211523

From llvmlistbot at llvm.org  Thu Jul 23 04:10:47 2026
From: llvmlistbot at llvm.org (Hagai Lev Hacohen)
Date: Thu, 23 Jul 2026 04:10:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add worklist comparator to tile and fuse
	options (PR #211523)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211523@github.com>
Message-ID: <6a61f6b7.6d3ea74f.17f70b.7b03@mx.google.com>


https://github.com/HagaiLevHacohen updated https://github.com/llvm/llvm-project/pull/211523

>From 389b3b7cd4326e8f3f076ac160c8d99693b5b59b Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Wed, 22 Jul 2026 15:23:09 +0300
Subject: [PATCH 1/2] Adding to tile and fuse options a worklist comparator

---
 .../SCF/Transforms/TileUsingInterface.h       | 15 +++
 .../SCF/Transforms/TileUsingInterface.cpp     | 30 +++++-
 .../tile-fuse-and-yield-using-interface.mlir  | 98 +++++++++++++++++++
 .../TestTilingInterfaceTransformOps.cpp       | 15 ++-
 .../TestTilingInterfaceTransformOps.td        | 10 +-
 5 files changed, 162 insertions(+), 6 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 0bd5d19b136d0..125ff0edb7a89 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -312,6 +312,21 @@ struct SCFTileAndFuseOptions {
     return *this;
   }
 
+    /// Comparator used to select the next `tensor.extract_slice` to process from
+    /// the fusion worklist. Returns true if `lhs` should be processed before
+    /// `rhs`. This allows callers to enforce a desired tiling order. For example,
+    /// to process producers in topological order, a producer that is an ancestor
+    /// of another producer in the defining-op chain can be ordered before it.
+    /// By default, the worklist is processed in FIFO order.
+    using TileOrderControlFnTy =
+        std::function<bool(tensor::ExtractSliceOp lhs,
+                          tensor::ExtractSliceOp rhs)>;
+    TileOrderControlFnTy tileOrderControlFn = nullptr;
+    SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
+      tileOrderControlFn = std::move(controlFn);
+      return *this;
+    }
+
   /// An optional set of rewrite patterns to apply to the results of tiling
   /// before fusion. This will track deleted and newly inserted
   /// `tensor.extract_slice` ops and update the worklist.
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index ad4aff893a03a..00a27b4dc66c5 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -1641,6 +1641,14 @@ class SliceTrackingListener : public RewriterBase::Listener {
   /// Remove the operation from the worklist.
   void notifyOperationReplaced(Operation *op, ValueRange replacement) override;
 
+  /// Pop the next slice to process from the worklist. When
+  /// `tileOrderControlFn` is set, returns the preferred slice according to the
+  /// callback. Otherwise pops from the front (FIFO). The worklist must be
+  /// non-empty.
+  tensor::ExtractSliceOp popNext(
+    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+        &tileOrderControlFn);
+
   /// The worklist for this transformation keeps track of the slices to visit
   /// next for fusion.
   std::deque<tensor::ExtractSliceOp> worklist;
@@ -1707,6 +1715,24 @@ void SliceTrackingListener::notifyOperationReplaced(Operation *op,
   removeOp(op);
 }
 
+tensor::ExtractSliceOp SliceTrackingListener::popNext(
+  const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+      &tileOrderControlFn) {
+assert(!worklist.empty() && "expected non-empty worklist");
+if (!tileOrderControlFn) {
+  auto slice = worklist.front();
+  worklist.pop_front();
+  return slice;
+}
+auto it = llvm::min_element(worklist, [&](tensor::ExtractSliceOp lhs,
+                                          tensor::ExtractSliceOp rhs) {
+  return tileOrderControlFn(lhs, rhs);
+});
+auto slice = *it;
+worklist.erase(it);
+return slice;
+}
+
 //===----------------------------------------------------------------------===//
 // ReplacementListener
 //===----------------------------------------------------------------------===//
@@ -1814,8 +1840,8 @@ mlir::scf::tileConsumerAndFuseProducersUsingSCF(
   }
   OpBuilder::InsertionGuard g(rewriter);
   while (!sliceTracker.worklist.empty()) {
-    auto candidateSlice = sliceTracker.worklist.front();
-    sliceTracker.worklist.pop_front();
+    tensor::ExtractSliceOp candidateSlice =
+        sliceTracker.popNext(options.tileOrderControlFn);
 
     auto [fusableProducer, destinationInitArg] =
         getUntiledProducerFromSliceSource(&candidateSlice.getSourceMutable(),
diff --git a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
index 3c0ada9d2cabc..c11c7c4a5fc50 100644
--- a/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
+++ b/mlir/test/Interfaces/TilingInterface/tile-fuse-and-yield-using-interface.mlir
@@ -120,3 +120,101 @@ module attributes {transform.with_named_sequence} {
 //      CHECK:     %[[INSERT2:.+]] = tensor.insert_slice %[[GENERIC_TILE]]#1 into %[[ITERARG2]][0, %[[IV]]]
 //      CHECK:     scf.yield %[[INSERT0]], %[[INSERT1]], %[[INSERT2]]
 //      CHECK:   return %[[RESULT]]#1, %[[RESULT]]#2, %[[RESULT]]#0
+
+// -----
+
+// Verify that the default FIFO worklist reconstructs the earlier `linalg.fill`
+// before the later `linalg.copy`. Their loop results are therefore passed to
+// the downstream `linalg.mul` as result #1 and result #2, respectively.
+
+// CHECK-LABEL: func.func @worklist_fifo_order(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#1, %[[RESULT]]#2
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+func.func @worklist_fifo_order(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
+// Verify that preferring later producers reconstructs the `linalg.copy` before
+// the `linalg.fill`. Their loop result numbers are reversed, but the downstream
+// `linalg.mul` still receives the fill and copy values in its original order.
+
+// CHECK-LABEL: func.func @worklist_prefer_later_producers(
+// CHECK:         %[[RESULT:.+]]:3 = scf.for
+// CHECK:           %[[FILL_TILE:.+]] = linalg.fill
+// CHECK:           %[[COPY_TILE:.+]] = linalg.copy
+// CHECK:           %[[CONSUMER_TILE:.+]] = linalg.add
+// CHECK:           tensor.insert_slice %[[CONSUMER_TILE]]
+// CHECK:           tensor.insert_slice %[[COPY_TILE]]
+// CHECK:           tensor.insert_slice %[[FILL_TILE]]
+// CHECK:           scf.yield
+// CHECK:         %[[FINAL:.+]] = linalg.mul
+// CHECK-SAME:        ins(%[[RESULT]]#2, %[[RESULT]]#1
+// CHECK-SAME:        outs(%[[RESULT]]#0
+// CHECK:         return %[[FINAL]]
+
+
+func.func @worklist_prefer_later_producers(
+    %input: tensor<32x32xf32>, %out: tensor<32x32xf32>)
+    -> tensor<32x32xf32> {
+  %c0 = arith.constant 0.0 : f32
+  %fill = linalg.fill ins(%c0 : f32)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %copy = linalg.copy ins(%input : tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %add = linalg.add
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%out : tensor<32x32xf32>) -> tensor<32x32xf32>
+  %result = linalg.mul
+      ins(%fill, %copy : tensor<32x32xf32>, tensor<32x32xf32>)
+      outs(%add : tensor<32x32xf32>) -> tensor<32x32xf32>
+  return %result : tensor<32x32xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %arg0 : !transform.any_op {transform.readonly}) {
+    %add = transform.structured.match ops{["linalg.add"]} in %arg0
+      : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.test.fuse_and_yield %add [16]
+        worklist_prefer_later_producers true
+      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index 9467c925e543c..d891e576b7dd4 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -64,7 +64,9 @@ static LogicalResult
 applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
                       Range &&payloadOps, unsigned numLoops,
                       scf::SCFTilingOptions tilingOptions,
-                      TransformResults &transformResults) {
+                      TransformResults &transformResults,
+                      scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+                          tileOrderControlFn = nullptr) {
   SmallVector<Operation *> tiledOps;
   SmallVector<SmallVector<Operation *>> loopOps(numLoops);
 
@@ -98,6 +100,7 @@ applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
           yieldProducerReplacement};
     };
     tileAndFuseOptions.setFusionControlFn(controlFn);
+    tileAndFuseOptions.setTileOrderControlFn(tileOrderControlFn);
 
     rewriter.setInsertionPoint(target);
     FailureOr<scf::SCFTileAndFuseResult> tiledResults =
@@ -159,10 +162,18 @@ transform::TestFuseAndYieldOp::apply(TransformRewriter &rewriter,
     tilingOptions.setLoopType(scf::SCFTilingOptions::LoopType::ForallOp);
   }
 
+  scf::SCFTileAndFuseOptions::TileOrderControlFnTy tileOrderControlFn;
+  if (getWorklistPreferLaterProducers()) {
+    tileOrderControlFn = [](tensor::ExtractSliceOp lhs,
+                            tensor::ExtractSliceOp rhs) {
+      return rhs.getOperation()->isBeforeInBlock(lhs.getOperation());
+    };
+  }
+
   LogicalResult result = applyTileAndFuseToAll(
       rewriter, getOperation(), state.getPayloadOps(getTarget()),
       tileSizes.size() - llvm::count(tileSizes, 0), tilingOptions,
-      transformResults);
+      transformResults, tileOrderControlFn);
   return failed(result) ? DiagnosedSilenceableFailure::definiteFailure()
                         : DiagnosedSilenceableFailure::success();
 }
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
index efa16212f0fef..03032e7efb7ca 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.td
@@ -30,6 +30,9 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     producers greedily using the options provided as attributes.
     It also yields some of the fused producers for testing.
 
+    The `worklist_prefer_later_producers` attribute processes slices that occur
+    later in the tiled IR before earlier slices.
+
     On success returns the tiled operations as well as generated loops. Emits
     a definite failure if tiling fails.
   }];
@@ -38,13 +41,16 @@ def TestFuseAndYieldOp : Op<Transform_Dialect, "test.fuse_and_yield",
     (ins TransformHandleTypeInterface:$target,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_sizes,
         DefaultValuedAttr<I64ArrayAttr, "{}">:$tile_interchange,
-        DefaultValuedAttr<BoolAttr, "false">:$use_forall);
+        DefaultValuedAttr<BoolAttr, "false">:$use_forall,
+        DefaultValuedAttr<BoolAttr, "false">:$worklist_prefer_later_producers);
   let results = (outs TransformHandleTypeInterface:$transfomed,
       Variadic<TransformHandleTypeInterface>:$loops);
 
   let assemblyFormat = [{
     $target ($tile_sizes^)? (`interchange` $tile_interchange^)?
-    (`use_forall` $use_forall^)? attr-dict 
+    (`use_forall` $use_forall^)?
+    (`worklist_prefer_later_producers` $worklist_prefer_later_producers^)?
+    attr-dict
     `:` functional-type(operands, results)
   }];
 }

>From 0d4d897399253a3b31be6c8c4c4dbcd1bd98e910 Mon Sep 17 00:00:00 2001
From: Hagai Lev Hacohen <hagai4000 at gmail.com>
Date: Thu, 23 Jul 2026 14:10:21 +0300
Subject: [PATCH 2/2] lint

---
 .../SCF/Transforms/TileUsingInterface.h       | 27 +++++++--------
 .../SCF/Transforms/TileUsingInterface.cpp     | 34 +++++++++----------
 .../TestTilingInterfaceTransformOps.cpp       | 13 ++++---
 3 files changed, 36 insertions(+), 38 deletions(-)

diff --git a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
index 125ff0edb7a89..034bfbec5b77b 100644
--- a/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
+++ b/mlir/include/mlir/Dialect/SCF/Transforms/TileUsingInterface.h
@@ -312,20 +312,19 @@ struct SCFTileAndFuseOptions {
     return *this;
   }
 
-    /// Comparator used to select the next `tensor.extract_slice` to process from
-    /// the fusion worklist. Returns true if `lhs` should be processed before
-    /// `rhs`. This allows callers to enforce a desired tiling order. For example,
-    /// to process producers in topological order, a producer that is an ancestor
-    /// of another producer in the defining-op chain can be ordered before it.
-    /// By default, the worklist is processed in FIFO order.
-    using TileOrderControlFnTy =
-        std::function<bool(tensor::ExtractSliceOp lhs,
-                          tensor::ExtractSliceOp rhs)>;
-    TileOrderControlFnTy tileOrderControlFn = nullptr;
-    SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
-      tileOrderControlFn = std::move(controlFn);
-      return *this;
-    }
+  /// Comparator used to select the next `tensor.extract_slice` to process from
+  /// the fusion worklist. Returns true if `lhs` should be processed before
+  /// `rhs`. This allows callers to enforce a desired tiling order. For example,
+  /// to process producers in topological order, a producer that is an ancestor
+  /// of another producer in the defining-op chain can be ordered before it.
+  /// By default, the worklist is processed in FIFO order.
+  using TileOrderControlFnTy = std::function<bool(tensor::ExtractSliceOp lhs,
+                                                  tensor::ExtractSliceOp rhs)>;
+  TileOrderControlFnTy tileOrderControlFn = nullptr;
+  SCFTileAndFuseOptions &setTileOrderControlFn(TileOrderControlFnTy controlFn) {
+    tileOrderControlFn = std::move(controlFn);
+    return *this;
+  }
 
   /// An optional set of rewrite patterns to apply to the results of tiling
   /// before fusion. This will track deleted and newly inserted
diff --git a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
index 00a27b4dc66c5..b856174b787cf 100644
--- a/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
+++ b/mlir/lib/Dialect/SCF/Transforms/TileUsingInterface.cpp
@@ -1645,9 +1645,9 @@ class SliceTrackingListener : public RewriterBase::Listener {
   /// `tileOrderControlFn` is set, returns the preferred slice according to the
   /// callback. Otherwise pops from the front (FIFO). The worklist must be
   /// non-empty.
-  tensor::ExtractSliceOp popNext(
-    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-        &tileOrderControlFn);
+  tensor::ExtractSliceOp
+  popNext(const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+              &tileOrderControlFn);
 
   /// The worklist for this transformation keeps track of the slices to visit
   /// next for fusion.
@@ -1716,22 +1716,22 @@ void SliceTrackingListener::notifyOperationReplaced(Operation *op,
 }
 
 tensor::ExtractSliceOp SliceTrackingListener::popNext(
-  const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-      &tileOrderControlFn) {
-assert(!worklist.empty() && "expected non-empty worklist");
-if (!tileOrderControlFn) {
-  auto slice = worklist.front();
-  worklist.pop_front();
+    const scf::SCFTileAndFuseOptions::TileOrderControlFnTy
+        &tileOrderControlFn) {
+  assert(!worklist.empty() && "expected non-empty worklist");
+  if (!tileOrderControlFn) {
+    auto slice = worklist.front();
+    worklist.pop_front();
+    return slice;
+  }
+  auto it = llvm::min_element(
+      worklist, [&](tensor::ExtractSliceOp lhs, tensor::ExtractSliceOp rhs) {
+        return tileOrderControlFn(lhs, rhs);
+      });
+  auto slice = *it;
+  worklist.erase(it);
   return slice;
 }
-auto it = llvm::min_element(worklist, [&](tensor::ExtractSliceOp lhs,
-                                          tensor::ExtractSliceOp rhs) {
-  return tileOrderControlFn(lhs, rhs);
-});
-auto slice = *it;
-worklist.erase(it);
-return slice;
-}
 
 //===----------------------------------------------------------------------===//
 // ReplacementListener
diff --git a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
index d891e576b7dd4..42139c110707d 100644
--- a/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
+++ b/mlir/test/lib/Interfaces/TilingInterface/TestTilingInterfaceTransformOps.cpp
@@ -60,13 +60,12 @@ static llvm::SmallDenseSet<Operation *> collectTiledAndFusedOps(Operation *op) {
 /// Apply a tile and fuse transformation to all payload ops and store both the
 /// tiled operation as well as the created tile loops.
 template <typename Range>
-static LogicalResult
-applyTileAndFuseToAll(RewriterBase &rewriter, Operation *transformOp,
-                      Range &&payloadOps, unsigned numLoops,
-                      scf::SCFTilingOptions tilingOptions,
-                      TransformResults &transformResults,
-                      scf::SCFTileAndFuseOptions::TileOrderControlFnTy
-                          tileOrderControlFn = nullptr) {
+static LogicalResult applyTileAndFuseToAll(
+    RewriterBase &rewriter, Operation *transformOp, Range &&payloadOps,
+    unsigned numLoops, scf::SCFTilingOptions tilingOptions,
+    TransformResults &transformResults,
+    scf::SCFTileAndFuseOptions::TileOrderControlFnTy tileOrderControlFn =
+        nullptr) {
   SmallVector<Operation *> tiledOps;
   SmallVector<SmallVector<Operation *>> loopOps(numLoops);
 


From llvmlistbot at llvm.org  Thu Jul 23 04:12:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 04:12:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a61f72e.bd93e731.2ac69c.bba0@mx.google.com>


================
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
----------------
PragmaTwice wrote:

Yeah we can do some detection but it may make the implementation more complicated. Dialect changes may be not only about new ops, but also about changing/added/deleting an operand/result/.. in existing ops (Python class is so dynamic so it's possible). Currently I think the simplest way is just to raising an exception.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 05:21:20 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Thu, 23 Jul 2026 05:21:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a620740.af6ba3ed.1989e7.d137@mx.google.com>


================
@@ -231,68 +223,48 @@ class Tosa_TensorOfOrNone<list<Type> allowedTypes, string description = ""> :
 // Tensor types with constrained ranks.
 //===----------------------------------------------------------------------===//
 
-def Tosa_Rank0Tensor : TosaTensorRankOf<[Tosa_AnyNumber], [0]>;
-
-def Tosa_ScalarTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_AnyNumber], [1]>]>;
-def Tosa_ScalarInt8Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int8]>, TosaScalarTensorOf<[Tosa_Int8], [1]>]>;
-def Tosa_ScalarIntOrFloatTensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int, AnyFloat]>, TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>]>;
-def Tosa_ScalarInt32Tensor : AnyTypeOf<[TosaUnrankedTensorOf<[Tosa_Int32]>, TosaScalarTensorOf<[Tosa_Int32], [1]>]>;
-
-// We include unranked tensors as a supported type for all possible tosa
-// Tensors as unranked does not guarantee invalid. If unranked tensors exist
-// they should be shape propagate used Tosa's shape inference pass and verified
-// to not include any remaining unranked tensors.
-def Tosa_UnrankedTensor : TosaUnrankedTensorOf<[Tosa_AnyNumber]>;
+def Tosa_ScalarTensor : TosaScalarTensorOf<[Tosa_AnyNumber], [1]>;
+def Tosa_ScalarInt8Tensor : TosaScalarTensorOf<[Tosa_Int8], [1]>;
+def Tosa_ScalarIntOrFloatTensor : TosaScalarTensorOf<[Tosa_Int, AnyFloat], [1]>;
+def Tosa_ScalarInt32Tensor : TosaScalarTensorOf<[Tosa_Int32], [1]>;
 
-def Tosa_Tensor1D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [1]>], "1-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor2D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [2]>], "2-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor3D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [3]>], "3-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor4D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [4]>], "4-d tosa-conformant tensor", "::mlir::TensorType">;
-def Tosa_Tensor5D : AnyTypeOf<[Tosa_UnrankedTensor, TosaTensorRankOf<[Tosa_AnyNumber], [5]>], "5-d tosa-conformant tensor", "::mlir::TensorType">;
+def Tosa_Tensor1D : TosaTensorRankOf<[Tosa_AnyNumber], [1]>;
+def Tosa_Tensor2D : TosaTensorRankOf<[Tosa_AnyNumber], [2]>;
+def Tosa_Tensor3D : TosaTensorRankOf<[Tosa_AnyNumber], [3]>;
+def Tosa_Tensor4D : TosaTensorRankOf<[Tosa_AnyNumber], [4]>;
+def Tosa_Tensor5D : TosaTensorRankOf<[Tosa_AnyNumber], [5]>;
 
 // 1D tensor of specific types
-def Tosa_1DInt8Tensor : 1DTensorOf<[Tosa_Int8]>;
-def Tosa_1DInt16Or32Tensor : 1DTensorOf<[Tosa_Int16Or32]>;
+def Tosa_1DInt8Tensor : TosaTensorRankOf<[Tosa_Int8], [1]>;
----------------
sahas3 wrote:

I think mentioning this in discourse if fine.

https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Thu Jul 23 05:21:38 2026
From: llvmlistbot at llvm.org (Sayan Saha)
Date: Thu, 23 Jul 2026 05:21:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Combine unranked/ranked tensor
 types into single type (PR #209737)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209737@github.com>
Message-ID: <6a620752.824e11a6.322560.da60@mx.google.com>


https://github.com/sahas3 approved this pull request.


https://github.com/llvm/llvm-project/pull/209737

From llvmlistbot at llvm.org  Thu Jul 23 05:56:43 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 23 Jul 2026 05:56:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SCFToSPIRV] Fix iter_args returning
 undef on zero trip scf.for (PR #206280)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206280@github.com>
Message-ID: <6a620f8b.c56fb102.6921f.2f6a@mx.google.com>


aobolensk wrote:

@IgWod please take a look

https://github.com/llvm/llvm-project/pull/206280

From llvmlistbot at llvm.org  Thu Jul 23 05:56:45 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 23 Jul 2026 05:56:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][MathToSPIRV] Allow math.cttz lowering
 for non-i32 integer widths (PR #206400)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206400@github.com>
Message-ID: <6a620f8d.d2cb4adc.23815e.06ba@mx.google.com>


aobolensk wrote:

@IgWod please take a look

https://github.com/llvm/llvm-project/pull/206400

From llvmlistbot at llvm.org  Thu Jul 23 05:56:48 2026
From: llvmlistbot at llvm.org (Arseniy Obolenskiy)
Date: Thu, 23 Jul 2026 05:56:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIR-V] Support bitwise and/or/xor in
 gpu reduction lowering (PR #202351)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/202351@github.com>
Message-ID: <6a620f90.1728eaf3.5e665.ce32@mx.google.com>


aobolensk wrote:

@IgWod please take a look

https://github.com/llvm/llvm-project/pull/202351

From llvmlistbot at llvm.org  Thu Jul 23 06:03:41 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Thu, 23 Jul 2026 06:03:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (PR #210902)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210902@github.com>
Message-ID: <6a62112d.32420449.e9655.9e16@mx.google.com>


https://github.com/dcaballe approved this pull request.

Thanks!

https://github.com/llvm/llvm-project/pull/210902

From llvmlistbot at llvm.org  Thu Jul 23 06:08:07 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Thu, 23 Jul 2026 06:08:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a621237.77a0a7d7.17808.3f72@mx.google.com>


================
@@ -1196,6 +1196,30 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_scalar_input(
+    %input: f32, %init: tensor<8x16xf32>) -> tensor<8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op operand #0 must be memref of any non-token type values or ranked tensor of any non-token type values, but got 'f32'}}
----------------
bruteforceboy wrote:

I think it does make sense to be able to broadcast a scalar just like we do for the OD case.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 06:13:55 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Thu, 23 Jul 2026 06:13:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Reorder element/broadcast for
 FMA if the source is not scalar (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a621393.96152a85.426cd.fb59@mx.google.com>


dcaballe wrote:

Thanks for the contribution!

I'm not sure what you are trying to achieve. The intent of this pass is to minimize redundant computation on the same data. For your example:
```
  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
  %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
```

I would expect that broadcasts on `%arg0` and `%arg2` are removed but `%arg1` is broadcasted to `<4xf32>`. Then FMA would happen on `<4xf32>`, followed by the broadcast to `<3x4xf32>`.

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Thu Jul 23 06:37:19 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 23 Jul 2026 06:37:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a62190f.8c92e5db.162b7b.503c@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/211354

>From 3c8ac21e769cdb7712ca7317daa492651220bdb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 11:04:40 -0700
Subject: [PATCH 1/3] [LinalgBlockPackMatmul] Add support for scalable block
 factors

Assisted by: Claude Sonnet 4.6
---
 mlir/include/mlir/Dialect/Linalg/Passes.td    |  8 +-
 .../Dialect/Linalg/Transforms/Transforms.h    |  5 +
 .../Linalg/Transforms/BlockPackMatmul.cpp     | 57 +++++++++++-
 .../Linalg/block-pack-matmul-scalable.mlir    | 91 +++++++++++++++++++
 4 files changed, 155 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir

diff --git a/mlir/include/mlir/Dialect/Linalg/Passes.td b/mlir/include/mlir/Dialect/Linalg/Passes.td
index 3a43af9ca1855..5573cf8b9feb7 100644
--- a/mlir/include/mlir/Dialect/Linalg/Passes.td
+++ b/mlir/include/mlir/Dialect/Linalg/Passes.td
@@ -215,10 +215,12 @@ def LinalgBlockPackMatmul : Pass<"linalg-block-pack-matmul"> {
       %res = unpack %res_packed : 4D <MBxNBxmbxnb> -> 2D <MxN>
     ```
   }];
-  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect",
+                           "arith::ArithDialect",
+                           "vector::VectorDialect"];
   let options = [
-    ListOption<"blockFactors", "block-factors", "int64_t",
-               "Block factors (mb, nb, kb) for relayout">,
+    ListOption<"blockFactors", "block-factors", "std::string",
+               "Block factors (mb, nb, kb) for relayout; use [N] for scalable">,
     Option<"allowPadding", "allow-padding", "bool",
            /*default=*/"true",
            "Allow packing padding">,
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 836682de4c404..9a59fb5ea7931 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -1423,6 +1423,11 @@ struct BlockPackMatmulOptions {
   /// the parallel dimensions and kb is the reduction dimension.
   SmallVector<int64_t, 3> blockFactors;
 
+  /// Scalable flags for block factors. When true, the corresponding block
+  /// factor is multiplied by vector.vscale at runtime (SVE-style scalable).
+  /// Must be empty (all static) or have the same size as blockFactors.
+  SmallVector<bool, 3> scalableBlockFactors;
+
   /// If true, allows packing of dimensions that only partially fit into the
   /// block factors.
   bool allowPadding = true;
diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index 6ea1eb50b13ce..cfc91fe537ccd 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -8,9 +8,11 @@
 
 #include "mlir/Dialect/Linalg/Passes.h"
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/Linalg/Transforms/Transforms.h"
 #include "mlir/Dialect/Linalg/Utils/Utils.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/SmallVector.h"
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
+
+  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
+  // arith.constant N : index multiplied by vector.vscale.
+  SmallVector<OpFoldResult> mnkTiles;
+  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
+    bool isScalable = hasScalable && options->scalableBlockFactors[i];
+    if (!isScalable) {
+      mnkTiles.push_back(rewriter.getIndexAttr(factor));
+      continue;
+    }
+    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
+                                               factor);
+    Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
+                                                 rewriter.getIndexType());
+    mnkTiles.push_back(
+        arith::MulIOp::create(rewriter, linalgOp.getLoc(), cst, vscale)
+            .getResult());
+  }
 
   // If padding is disabled, make sure that dimensions can be packed cleanly.
   if (!options->allowPadding &&
@@ -300,7 +329,29 @@ struct LinalgBlockPackMatmul
     ControlBlockPackMatmulFn controlFn =
         [&](linalg::LinalgOp op) -> BlockPackMatmulOptions {
       BlockPackMatmulOptions options;
-      options.blockFactors = SmallVector<int64_t>{*blockFactors};
+
+      // Parse block-factors strings. Each element is either "N" (static) or
+      // "[N]" (scalable, i.e. N * vscale at runtime).
+      for (const std::string &f : *blockFactors) {
+        StringRef s(f);
+        if (s.starts_with("[") && s.ends_with("]")) {
+          int64_t val = 0;
+          s.drop_front().drop_back().getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(true);
+        } else {
+          int64_t val = 0;
+          s.getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(false);
+        }
+      }
+      // If all flags are false, clear the vector so blockPackMatmul can take
+      // the cheaper static path.
+      if (llvm::none_of(options.scalableBlockFactors,
+                        [](bool b) { return b; }))
+        options.scalableBlockFactors.clear();
+
       options.allowPadding = allowPadding;
       options.mnkPaddedSizesNextMultipleOf =
           SmallVector<int64_t>{*mnkPaddedSizesNextMultipleOf};
diff --git a/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
new file mode 100644
index 0000000000000..a5829c210d1a8
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
@@ -0,0 +1,91 @@
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=0" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE-NOPAD
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=32,16,[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=MIXED
+
+// -----
+
+// All-scalable block factors, static input shapes, allow-padding=1.
+// Inner tile sizes and outer dimensions are all dynamic (vscale-relative).
+
+func.func @block_matmul_scalable_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// Scalable factors with allow-padding=0: transform does not apply.
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// All-scalable block factors, dynamic input shapes, allow-padding=1.
+// Both outer tile counts and inner tile sizes are fully dynamic.
+
+func.func @block_matmul_scalable_dynamic(
+    %A: tensor<?x?xf32>, %B: tensor<?x?xf32>, %C: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<?x?xf32>, tensor<?x?xf32>)
+                     outs(%C : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// Mixed block factors (mb=32, nb=16 static; kb=[64] scalable), static input.
+// Only the K-dimension inner tile is scalable; mb and nb remain static.
+
+func.func @block_matmul_mixed_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// MIXED-LABEL: func @block_matmul_mixed_static(
+// MIXED-DAG: %[[C64:.+]] = arith.constant 64 : index
+// MIXED-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [32, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [16, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+// MIXED: linalg.generic
+// MIXED: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+

>From 46852e32748c6ffab6973f12de9c06e5c6d6d060 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 14:34:41 -0700
Subject: [PATCH 2/3] clang-format PR

---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index cfc91fe537ccd..ebc287463bec2 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -184,8 +184,8 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
       mnkTiles.push_back(rewriter.getIndexAttr(factor));
       continue;
     }
-    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
-                                               factor);
+    Value cst =
+        arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(), factor);
     Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
                                                  rewriter.getIndexType());
     mnkTiles.push_back(
@@ -348,8 +348,7 @@ struct LinalgBlockPackMatmul
       }
       // If all flags are false, clear the vector so blockPackMatmul can take
       // the cheaper static path.
-      if (llvm::none_of(options.scalableBlockFactors,
-                        [](bool b) { return b; }))
+      if (llvm::none_of(options.scalableBlockFactors, [](bool b) { return b; }))
         options.scalableBlockFactors.clear();
 
       options.allowPadding = allowPadding;

>From c80811912e002c40c448ab690c13a6b97a5f5b4d Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 23 Jul 2026 09:37:08 -0400
Subject: [PATCH 3/3] Update
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp

Co-authored-by: Adam Siemieniuk <adam.siemieniuk at intel.com>
---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index ebc287463bec2..e6147c8814b9d 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -173,7 +173,7 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   // any ops to avoid modifying IR before returning notifyMatchFailure.
   if (!options->allowPadding && hasScalable)
     return rewriter.notifyMatchFailure(
-        linalgOp, "scalable block factors require allow-padding=true");
+        linalgOp, "scalable block factors require padding");
 
   // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
   // arith.constant N : index multiplied by vector.vscale.


From llvmlistbot at llvm.org  Thu Jul 23 06:37:31 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 23 Jul 2026 06:37:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a62191b.86686cdc.39703c.95b5@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/211354

>From 3c8ac21e769cdb7712ca7317daa492651220bdb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 11:04:40 -0700
Subject: [PATCH 1/4] [LinalgBlockPackMatmul] Add support for scalable block
 factors

Assisted by: Claude Sonnet 4.6
---
 mlir/include/mlir/Dialect/Linalg/Passes.td    |  8 +-
 .../Dialect/Linalg/Transforms/Transforms.h    |  5 +
 .../Linalg/Transforms/BlockPackMatmul.cpp     | 57 +++++++++++-
 .../Linalg/block-pack-matmul-scalable.mlir    | 91 +++++++++++++++++++
 4 files changed, 155 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir

diff --git a/mlir/include/mlir/Dialect/Linalg/Passes.td b/mlir/include/mlir/Dialect/Linalg/Passes.td
index 3a43af9ca1855..5573cf8b9feb7 100644
--- a/mlir/include/mlir/Dialect/Linalg/Passes.td
+++ b/mlir/include/mlir/Dialect/Linalg/Passes.td
@@ -215,10 +215,12 @@ def LinalgBlockPackMatmul : Pass<"linalg-block-pack-matmul"> {
       %res = unpack %res_packed : 4D <MBxNBxmbxnb> -> 2D <MxN>
     ```
   }];
-  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect",
+                           "arith::ArithDialect",
+                           "vector::VectorDialect"];
   let options = [
-    ListOption<"blockFactors", "block-factors", "int64_t",
-               "Block factors (mb, nb, kb) for relayout">,
+    ListOption<"blockFactors", "block-factors", "std::string",
+               "Block factors (mb, nb, kb) for relayout; use [N] for scalable">,
     Option<"allowPadding", "allow-padding", "bool",
            /*default=*/"true",
            "Allow packing padding">,
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 836682de4c404..9a59fb5ea7931 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -1423,6 +1423,11 @@ struct BlockPackMatmulOptions {
   /// the parallel dimensions and kb is the reduction dimension.
   SmallVector<int64_t, 3> blockFactors;
 
+  /// Scalable flags for block factors. When true, the corresponding block
+  /// factor is multiplied by vector.vscale at runtime (SVE-style scalable).
+  /// Must be empty (all static) or have the same size as blockFactors.
+  SmallVector<bool, 3> scalableBlockFactors;
+
   /// If true, allows packing of dimensions that only partially fit into the
   /// block factors.
   bool allowPadding = true;
diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index 6ea1eb50b13ce..cfc91fe537ccd 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -8,9 +8,11 @@
 
 #include "mlir/Dialect/Linalg/Passes.h"
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/Linalg/Transforms/Transforms.h"
 #include "mlir/Dialect/Linalg/Utils/Utils.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/SmallVector.h"
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
+
+  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
+  // arith.constant N : index multiplied by vector.vscale.
+  SmallVector<OpFoldResult> mnkTiles;
+  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
+    bool isScalable = hasScalable && options->scalableBlockFactors[i];
+    if (!isScalable) {
+      mnkTiles.push_back(rewriter.getIndexAttr(factor));
+      continue;
+    }
+    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
+                                               factor);
+    Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
+                                                 rewriter.getIndexType());
+    mnkTiles.push_back(
+        arith::MulIOp::create(rewriter, linalgOp.getLoc(), cst, vscale)
+            .getResult());
+  }
 
   // If padding is disabled, make sure that dimensions can be packed cleanly.
   if (!options->allowPadding &&
@@ -300,7 +329,29 @@ struct LinalgBlockPackMatmul
     ControlBlockPackMatmulFn controlFn =
         [&](linalg::LinalgOp op) -> BlockPackMatmulOptions {
       BlockPackMatmulOptions options;
-      options.blockFactors = SmallVector<int64_t>{*blockFactors};
+
+      // Parse block-factors strings. Each element is either "N" (static) or
+      // "[N]" (scalable, i.e. N * vscale at runtime).
+      for (const std::string &f : *blockFactors) {
+        StringRef s(f);
+        if (s.starts_with("[") && s.ends_with("]")) {
+          int64_t val = 0;
+          s.drop_front().drop_back().getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(true);
+        } else {
+          int64_t val = 0;
+          s.getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(false);
+        }
+      }
+      // If all flags are false, clear the vector so blockPackMatmul can take
+      // the cheaper static path.
+      if (llvm::none_of(options.scalableBlockFactors,
+                        [](bool b) { return b; }))
+        options.scalableBlockFactors.clear();
+
       options.allowPadding = allowPadding;
       options.mnkPaddedSizesNextMultipleOf =
           SmallVector<int64_t>{*mnkPaddedSizesNextMultipleOf};
diff --git a/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
new file mode 100644
index 0000000000000..a5829c210d1a8
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
@@ -0,0 +1,91 @@
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=0" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE-NOPAD
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=32,16,[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=MIXED
+
+// -----
+
+// All-scalable block factors, static input shapes, allow-padding=1.
+// Inner tile sizes and outer dimensions are all dynamic (vscale-relative).
+
+func.func @block_matmul_scalable_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// Scalable factors with allow-padding=0: transform does not apply.
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// All-scalable block factors, dynamic input shapes, allow-padding=1.
+// Both outer tile counts and inner tile sizes are fully dynamic.
+
+func.func @block_matmul_scalable_dynamic(
+    %A: tensor<?x?xf32>, %B: tensor<?x?xf32>, %C: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<?x?xf32>, tensor<?x?xf32>)
+                     outs(%C : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// Mixed block factors (mb=32, nb=16 static; kb=[64] scalable), static input.
+// Only the K-dimension inner tile is scalable; mb and nb remain static.
+
+func.func @block_matmul_mixed_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// MIXED-LABEL: func @block_matmul_mixed_static(
+// MIXED-DAG: %[[C64:.+]] = arith.constant 64 : index
+// MIXED-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [32, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [16, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+// MIXED: linalg.generic
+// MIXED: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+

>From 46852e32748c6ffab6973f12de9c06e5c6d6d060 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 14:34:41 -0700
Subject: [PATCH 2/4] clang-format PR

---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index cfc91fe537ccd..ebc287463bec2 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -184,8 +184,8 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
       mnkTiles.push_back(rewriter.getIndexAttr(factor));
       continue;
     }
-    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
-                                               factor);
+    Value cst =
+        arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(), factor);
     Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
                                                  rewriter.getIndexType());
     mnkTiles.push_back(
@@ -348,8 +348,7 @@ struct LinalgBlockPackMatmul
       }
       // If all flags are false, clear the vector so blockPackMatmul can take
       // the cheaper static path.
-      if (llvm::none_of(options.scalableBlockFactors,
-                        [](bool b) { return b; }))
+      if (llvm::none_of(options.scalableBlockFactors, [](bool b) { return b; }))
         options.scalableBlockFactors.clear();
 
       options.allowPadding = allowPadding;

>From c80811912e002c40c448ab690c13a6b97a5f5b4d Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 23 Jul 2026 09:37:08 -0400
Subject: [PATCH 3/4] Update
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp

Co-authored-by: Adam Siemieniuk <adam.siemieniuk at intel.com>
---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index ebc287463bec2..e6147c8814b9d 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -173,7 +173,7 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   // any ops to avoid modifying IR before returning notifyMatchFailure.
   if (!options->allowPadding && hasScalable)
     return rewriter.notifyMatchFailure(
-        linalgOp, "scalable block factors require allow-padding=true");
+        linalgOp, "scalable block factors require padding");
 
   // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
   // arith.constant N : index multiplied by vector.vscale.

>From 3f966c039d99578847d20ae6d06318c6f4ef3bb7 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 23 Jul 2026 09:37:21 -0400
Subject: [PATCH 4/4] Update
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp

Co-authored-by: Adam Siemieniuk <adam.siemieniuk at intel.com>
---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index e6147c8814b9d..ba64b6b2e4447 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -178,7 +178,7 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
   // arith.constant N : index multiplied by vector.vscale.
   SmallVector<OpFoldResult> mnkTiles;
-  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
+  for (auto [idx, factor] : llvm::enumerate(options->blockFactors)) {
     bool isScalable = hasScalable && options->scalableBlockFactors[i];
     if (!isScalable) {
       mnkTiles.push_back(rewriter.getIndexAttr(factor));


From llvmlistbot at llvm.org  Thu Jul 23 07:09:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:09:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a6220a6.0e54f35d.1a0615.0303@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 07:09:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:09:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a6220a6.0ce3539a.338cb7.0acf@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM % nits

@rengolin , as mentioned in the summary, rank-0 just happened to provide the initial example, but this is change is valid _in general_. @bruteforceboy , I would not refer to rank-0 in the summary, unless this change only applies to rank-0.

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 07:09:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:09:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a6220a6.36eaaffb.4dad5.0cc3@mx.google.com>


================
@@ -599,6 +599,36 @@ func.func @broadcast_memref(%input: memref<8x32xf32>,
 
 // -----
 
+func.func @broadcast_0d_tensor(%input: tensor<i32>,
+                               %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 1]
+  func.return %bcast : tensor<32x2xi32>
+}
+// CHECK-LABEL: func @broadcast_0d_tensor
+//      CHECK:    linalg.broadcast ins(%{{.*}} : tensor<i32>)
+// CHECK-SAME:    outs(%{{.*}} : tensor<32x2xi32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
+// -----
+
+func.func @broadcast_0d_memref(%input: memref<f32>,
+                               %init: memref<8x16xf32>) {
+  linalg.broadcast
+      ins(%input:memref<f32>)
+      outs(%init:memref<8x16xf32>)
+      dimensions = [0, 1]
+  func.return
+}
+// CHECK-LABEL: func @broadcast_0d_memref
+//      CHECK:    linalg.broadcast ins(%{{.*}} : memref<f32>)
+// CHECK-SAME:    outs(%{{.*}} : memref<8x16xf32>)
+// CHECK-SAME:    dimensions = [0, 1]
+
----------------
banach-space wrote:

No need to test both tensor and memref. IIUC, the code path does not depend on the type of the shaped type (i.e. `tensor` vs `memref`).

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 07:09:43 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:09:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a6220a7.9e67a41a.d5c6d.2a04@mx.google.com>


================
@@ -478,6 +478,11 @@ def BroadcastOp : LinalgStructuredBase_Op<"broadcast", [
   let description = [{
     Broadcast the input into the given shape by adding `dimensions`.
 
+    Each index in `dimensions` attribute refers to a dimension of the init
+    tensor that is added by the operation. The indices must be unique and
+    within the init rank; the sizes of the remaining (non-added) init
+    dimensions must match the input shape.
----------------
banach-space wrote:

1. "init" could be either a tensor or a memref
2. init -> `init` (to make it clear that you are referring to `$init`)
3. input -> `input` (to make it clear that you are referring to `$input`)

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 07:09:43 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:09:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a6220a7.b47a2ac0.2218e3.7aa1@mx.google.com>


================
@@ -1196,6 +1196,30 @@ func.func @broadcast_size_1_extension_not_supported(
 
 // -----
 
+func.func @broadcast_duplicate_dims(
+    %input: tensor<i32>, %init: tensor<32x2xi32>) -> tensor<32x2xi32> {
+  // expected-error @+1 {{'linalg.broadcast' op dimensions should not contain duplicates}}
+  %bcast = linalg.broadcast
+      ins(%input:tensor<i32>)
+      outs(%init:tensor<32x2xi32>)
+      dimensions = [0, 0]
+  func.return %bcast : tensor<32x2xi32>
+}
+
+// -----
+
+func.func @broadcast_scalar_input(
+    %input: f32, %init: tensor<8x16xf32>) -> tensor<8x16xf32> {
+  // expected-error @+1 {{'linalg.broadcast' op operand #0 must be memref of any non-token type values or ranked tensor of any non-token type values, but got 'f32'}}
----------------
banach-space wrote:

+1

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 07:10:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 07:10:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 3cbb24b - [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (#210902)
Message-ID: <6a6220d8.c1cb16b0.221393.c4d0@mx.google.com>


Author: Andrzej Warzyński
Date: 2026-07-23T15:10:27+01:00
New Revision: 3cbb24bd5412a5ecea5a0e4d1603e0a38676432d

URL: https://github.com/llvm/llvm-project/commit/3cbb24bd5412a5ecea5a0e4d1603e0a38676432d
DIFF: https://github.com/llvm/llvm-project/commit/3cbb24bd5412a5ecea5a0e4d1603e0a38676432d.diff

LOG: [mlir][vector] Update CastAway{Extract|Insert}StridedSliceLeadingOneDim (#210902)

Update both:
  * CastAwayExtractStridedSliceLeadingOneDim
  * CastAwayInsertStridedSliceLeadingOneDim

to use vector.shape_cast, rather than vector.extract and
vector.broadcast, as the canonical form for stripping unit dimensions.

This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>

Added: 
    

Modified: 
    mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
    mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index 26a702ef0f512..d86e9f224bf9d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -78,8 +78,8 @@ struct CastAwayExtractStridedSliceLeadingOneDim
 
     Location loc = extractOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, extractOp.getSource(), splatZero(dropCount));
+    Value newSrcVector = rewriter.createOrFold<ShapeCastOp>(
+        loc, newSrcType, extractOp.getSource());
 
     // The offsets/sizes/strides attribute can have a less number of elements
     // than the input vector's rank: it is meant for the leading dimensions.
@@ -94,7 +94,7 @@ struct CastAwayExtractStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newOffsets, newSizes,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(extractOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(extractOp, oldDstType,
                                                      newExtractOp);
 
     return success();
@@ -122,10 +122,10 @@ struct CastAwayInsertStridedSliceLeadingOneDim
     // Trim leading one dimensions from both operands.
     Location loc = insertOp.getLoc();
 
-    Value newSrcVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newSrcType, insertOp.getValueToStore());
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     auto newOffsets = rewriter.getArrayAttr(
         insertOp.getOffsets().getValue().take_back(newDstType.getRank()));
@@ -136,7 +136,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
         rewriter, loc, newDstType, newSrcVector, newDstVector, newOffsets,
         newStrides);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();

diff  --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index bf01c8a8589d9..41cd2846fd752 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -270,44 +270,45 @@ func.func @cast_away_contraction_leading_one_dims_to_dot_product(%arg0: vector<6
 }
 
 // -----
+
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims
 func.func @cast_away_extract_strided_slice_leading_one_dims(%arg0: vector<1x8x8xf16>) -> vector<1x1x8xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x8xf16> to vector<1x8xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x8xf16> to vector<1x1x8xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x8xf16> to vector<1x1x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_extract_strided_slice_leading_one_dims_scalable
 func.func @cast_away_extract_strided_slice_leading_one_dims_scalable(%arg0: vector<1x8x[8]xf16>) -> vector<1x1x[8]xf16> {
-  // CHECK:     %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:     %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16>
   // CHECK: %[[EXTRACT:.+]] = vector.extract_strided_slice %[[SRC]] {offsets = [4], sizes = [1], strides = [1]} : vector<8x[8]xf16> to vector<1x[8]xf16>
   %0 = vector.extract_strided_slice %arg0 {offsets = [0, 4], sizes = [1, 1], strides = [1, 1]} : vector<1x8x[8]xf16> to vector<1x1x[8]xf16>
-  // CHECK:     %[[RET:.+]] = vector.broadcast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
+  // CHECK:     %[[RET:.+]] = vector.shape_cast %[[EXTRACT]] : vector<1x[8]xf16> to vector<1x1x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x1x[8]xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims
 func.func @cast_away_insert_strided_slice_leading_one_dims(%arg0: vector<1x8xf16>, %arg1: vector<1x8x8xf16>) -> vector<1x8x8xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<8xf16> from vector<1x8xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x8xf16> from vector<1x8x8xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x8xf16> to vector<8xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x8xf16> to vector<8x8xf16>
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<8xf16> into vector<8x8xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x8xf16> into vector<1x8x8xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x8xf16> to vector<1x8x8xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x8xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_scalable
 func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vector<1x[8]xf16>, %arg1: vector<1x8x[8]xf16>) -> vector<1x8x[8]xf16> {
-  // CHECK:    %[[SRC:.+]] = vector.extract %{{.*}}[0] : vector<[8]xf16> from vector<1x[8]xf16>
-  // CHECK:    %[[DST:.+]] = vector.extract %{{.*}}[0] : vector<8x[8]xf16> from vector<1x8x[8]xf16>
+  // CHECK:    %[[SRC:.+]] = vector.shape_cast %{{.*}} : vector<1x[8]xf16> to vector<[8]xf16>
+  // CHECK:    %[[DST:.+]] = vector.shape_cast %{{.*}} : vector<1x8x[8]xf16> to vector<8x[8]xf16> 
   // CHECK: %[[INSERT:.+]] = vector.insert_strided_slice %[[SRC]], %[[DST]] {offsets = [0, 0], strides = [1]} : vector<[8]xf16> into vector<8x[8]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[8]xf16> into vector<1x8x[8]xf16>
-  // CHECK:    %[[RET:.+]] = vector.broadcast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
+  // CHECK:    %[[RET:.+]] = vector.shape_cast %[[INSERT]] : vector<8x[8]xf16> to vector<1x8x[8]xf16>
   // CHECK: return %[[RET]]
   return %0: vector<1x8x[8]xf16>
 }
@@ -315,20 +316,18 @@ func.func @cast_away_insert_strided_slice_leading_one_dims_scalable(%arg0: vecto
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x1xf16>, %{{.+}}: vector<1x1x1xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element(%arg0: vector<1x1xf16>, %arg1: vector<1x1x1xf16>) -> vector<1x1x1xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<1xf16> from vector<1x1xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<1xf16> to vector<1x1x1xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x1xf16> to vector<1x1x1xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x1xf16> into vector<1x1x1xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x1xf16>
 }
 
 // CHECK-LABEL: func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable
 //  CHECK-SAME: %[[ARG0:.+]]: vector<1x[1]xf16>, %{{.+}}: vector<1x1x[1]xf16>
 func.func @cast_away_insert_strided_slice_leading_one_dims_one_element_scalable(%arg0: vector<1x[1]xf16>, %arg1: vector<1x1x[1]xf16>) -> vector<1x1x[1]xf16> {
-  // CHECK: %[[EXT:.+]] = vector.extract %{{.*}}[0] : vector<[1]xf16> from vector<1x[1]xf16>
-  // CHECK: %[[B:.+]] = vector.broadcast %[[EXT]] : vector<[1]xf16> to vector<1x1x[1]xf16>
+  // CHECK: %[[RET:.+]] = vector.shape_cast %{{.*}} : vector<1x[1]xf16> to vector<1x1x[1]xf16>
   %0 = vector.insert_strided_slice %arg0, %arg1 {offsets = [0, 0, 0], strides = [1, 1]} : vector<1x[1]xf16> into vector<1x1x[1]xf16>
-  // CHECK: return %[[B]]
+  // CHECK: return %[[RET]]
   return %0: vector<1x1x[1]xf16>
 }
 


        

From llvmlistbot at llvm.org  Thu Jul 23 07:10:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:10:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
 CastAway{Extract|Insert}StridedSliceLeadingOneDim (PR #210902)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210902@github.com>
Message-ID: <6a6220da.797536ea.2e6c5.24ed@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/210902

From llvmlistbot at llvm.org  Thu Jul 23 07:11:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 07:11:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a622124.d9789fc0.4aea6.ba5e@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Windows-->
# :window: Windows x64 Test Results

The build failed before running any tests. Click on a failure below to see the details.

<details>
<summary>[code=1] tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.obj</summary>

```
FAILED: [code=1] tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.obj
sccache C:\clang\clang-msvc\bin\clang-cl.exe  /nologo -TP -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_GLIBCXX_ASSERTIONS -D_HAS_EXCEPTIONS=0 -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:\_work\llvm-project\llvm-project\build\tools\mlir\lib\Dialect\Linalg\Transforms -IC:\_work\llvm-project\llvm-project\mlir\lib\Dialect\Linalg\Transforms -IC:\_work\llvm-project\llvm-project\build\tools\mlir\include -IC:\_work\llvm-project\llvm-project\mlir\include -IC:\_work\llvm-project\llvm-project\build\include -IC:\_work\llvm-project\llvm-project\llvm\include /DWIN32 /D_WINDOWS   /Zc:inline /Zc:__cplusplus /Oi /Brepro /bigobj /permissive- -Werror=unguarded-availability-new /W4  -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported /Gw -Wundef -Werror=mismatched-tags -Werror=global-constructors /O2 /Ob2  -std:c++17 -MD -UNDEBUG /EHs-c- /GR- /showIncludes /Fotools\mlir\lib\Dialect\Linalg\Transforms\CMakeFiles\obj.MLIRLinalgTransforms.dir\BlockPackMatmul.cpp.obj /Fdtools\mlir\lib\Dialect\Linalg\Transforms\CMakeFiles\obj.MLIRLinalgTransforms.dir\ -c -- C:\_work\llvm-project\llvm-project\mlir\lib\Dialect\Linalg\Transforms\BlockPackMatmul.cpp
C:\_work\llvm-project\llvm-project\mlir\lib\Dialect\Linalg\Transforms\BlockPackMatmul.cpp(182,68): error: use of undeclared identifier 'i'
182 |     bool isScalable = hasScalable && options->scalableBlockFactors[i];
|                                                                    ^
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 07:11:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 07:11:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a622124.7947377b.2a014e.90bd@mx.google.com>


github-actions[bot] wrote:

<!--PREMERGE ADVISOR COMMENT: Linux-->
# :penguin: Linux x64 Test Results

The build failed before running any tests. Click on a failure below to see the details.

<details>
<summary>tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.o</summary>

```
FAILED: tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.o
sccache /opt/llvm/bin/clang++ -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GLIBCXX_USE_CXX11_ABI=1 -D_GNU_SOURCE -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_EXTENSIVE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/lib/Dialect/Linalg/Transforms -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Dialect/Linalg/Transforms -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/tools/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/build/include -I/home/gha/actions-runner/_work/llvm-project/llvm-project/llvm/include -gmlt -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -Werror=global-constructors -O3 -DNDEBUG -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.o -MF tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.o.d -o tools/mlir/lib/Dialect/Linalg/Transforms/CMakeFiles/obj.MLIRLinalgTransforms.dir/BlockPackMatmul.cpp.o -c /home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
/home/gha/actions-runner/_work/llvm-project/llvm-project/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp:182:68: error: use of undeclared identifier 'i'
182 |     bool isScalable = hasScalable && options->scalableBlockFactors[i];
|                                                                    ^
1 error generated.
```
</details>

If these failures are unrelated to your changes (for example tests are broken or flaky at HEAD), please open an issue at https://github.com/llvm/llvm-project/issues and add the `infrastructure` label.

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Thu Jul 23 07:24:26 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:24:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a62241a.88c8bd60.7502f.14bc@mx.google.com>


banach-space wrote:

@FedericoBruzzone , this is not my area of expertise, so I will refrain from reviewing. But I am happy to land this for you (given that you already got +1 from the sparsification expert).

https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Thu Jul 23 07:24:32 2026
From: llvmlistbot at llvm.org (Abid Qadeer)
Date: Thu, 23 Jul 2026 07:24:32 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
Message-ID: <llvm.org/llvm/llvm-project/pull/211566@github.com>


https://github.com/abidh created https://github.com/llvm/llvm-project/pull/211566

This fixes https://github.com/llvm/llvm-project/issues/211385. This was initially landed in https://github.com/llvm/llvm-project/pull/148284.

We have this pattern of code in OMPIRBuilder for many functions that are used in reduction operations.

 ```
 Function *LtGRFunc = Function::Create
  BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
  Builder.SetInsertPoint(EntryBlock);
```

The insertion point is moved to the new function but the debug location
is not updated. This means that reduction function will use the debug
location that points to another function. This problem gets hidden
because these functions gets inlined but the potential for failure
exists.

This patch resets the debug location when insertion point is moved to new
function. Some `InsertPointGuard` have been added to make sure we restore
the debug location correctly when we are done with the reduction function.

It also clears the debug location for the helper-internal barriers in
emitInterWarpCopyFunction.

>From 5c789ea6a6969bb82e41694624bfbbc90542bcba Mon Sep 17 00:00:00 2001
From: Abid Qadeer <haqadeer at amd.com>
Date: Thu, 23 Jul 2026 14:33:11 +0100
Subject: [PATCH] Reland [OMPIRBuilder] Don't use invalid debug loc in
 reduction functions.

We have this pattern of code in OMPIRBuilder for many functions that are
used in reduction operations.

 ```
 Function *LtGRFunc = Function::Create
  BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
  Builder.SetInsertPoint(EntryBlock);
```

The insertion point is moved to the new function but the debug location
is not updated. This means that reduction function will use the debug
location that points to another function. This problem gets hidden
because these functions gets inlined but the potential for failure
exists.

This patch resets the debug location when insertion point is moved to new
function. Some `InsertPointGuard` have been added to make sure we restore
the debug location correctly when we are done with the reduction function.

It also clears the debug location for the helper-internal barriers in
emitInterWarpCopyFunction. Those barriers were created with the enclosing
construct's debug location, which updateToLocation leaves set on the
builder, so instructions emitted afterwards in the helper would otherwise
inherit a location scoped to the caller's (kernel) subprogram.

Co-authored-by: Cursor <cursoragent at cursor.com>
---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 30 ++++----
 .../LLVMIR/omptarget-debug-reduc-fn-loc.mlir  | 71 +++++++++++++++++++
 2 files changed, 89 insertions(+), 12 deletions(-)
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index fb1cebf28b95b..57e199bdff09a 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -3526,7 +3526,7 @@ Error OpenMPIRBuilder::emitReductionListCopy(
 Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
     const LocationDescription &Loc, ArrayRef<ReductionInfo> ReductionInfos,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  InsertPointTy SavedIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(), {Builder.getPtrTy(), Builder.getInt32Ty()},
@@ -3540,6 +3540,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   WcFunc->addParamAttr(1, Attribute::NoUndef);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", WcFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // ReduceList: thread local Reduce list.
   // At the stage of the computation when this function is called, partially
@@ -3646,7 +3647,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
 
       // kmpc_barrier.
       InsertPointOrErrorTy BarrierIP1 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3705,7 +3706,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
       // endif
       emitBlock(MergeBB, Builder.GetInsertBlock()->getParent());
       InsertPointOrErrorTy BarrierIP2 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3778,7 +3779,6 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(SavedIP);
 
   return WcFunc;
 }
@@ -3787,6 +3787,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
   LLVMContext &Ctx = M.getContext();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   FunctionType *FuncTy =
       FunctionType::get(Builder.getVoidTy(),
                         {Builder.getPtrTy(), Builder.getInt16Ty(),
@@ -3806,6 +3807,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
   SarFunc->addParamAttr(3, Attribute::SExt);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", SarFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Thread local Reduce list used to host the values of data to be reduced.
   Argument *ReduceListArg = SarFunc->getArg(0);
@@ -4018,7 +4020,7 @@ Expected<Value *> OpenMPIRBuilder::createReductionDescriptorCopy(
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4034,6 +4036,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGCFunc->getArg(0);
@@ -4137,14 +4140,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4160,6 +4162,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGRFunc->getArg(0);
@@ -4245,14 +4248,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {LocalReduceListAddrCast, ReduceList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGRFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4268,6 +4270,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLCFunc->getArg(0);
@@ -4368,14 +4371,13 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   auto *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4391,6 +4393,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLRFunc->getArg(0);
@@ -4476,7 +4479,6 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {ReduceList, ReductionList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLRFunc;
 }
 
@@ -4490,6 +4492,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
     StringRef ReducerName, ArrayRef<ReductionInfo> ReductionInfos,
     ArrayRef<bool> IsByRef, ReductionGenCBKind ReductionGenCBKind,
     AttributeList FuncAttrs) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   auto *FuncTy = FunctionType::get(Builder.getVoidTy(),
                                    {Builder.getPtrTy(), Builder.getPtrTy()},
                                    /* IsVarArg */ false);
@@ -4503,6 +4506,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
   BasicBlock *EntryBB =
       BasicBlock::Create(M.getContext(), "entry", ReductionFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Need to alloca memory here and deal with the pointers before getting
   // LHS/RHS pointers out
@@ -4986,10 +4990,12 @@ static Error populateReductionFunction(
     Function *ReductionFunc,
     ArrayRef<OpenMPIRBuilder::ReductionInfo> ReductionInfos,
     IRBuilder<> &Builder, ArrayRef<bool> IsByRef, bool IsGPU) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   Module *Module = ReductionFunc->getParent();
   BasicBlock *ReductionFuncBlock =
       BasicBlock::Create(Module->getContext(), "", ReductionFunc);
   Builder.SetInsertPoint(ReductionFuncBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
   Value *LHSArrayPtr = nullptr;
   Value *RHSArrayPtr = nullptr;
   if (IsGPU) {
diff --git a/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
new file mode 100644
index 0000000000000..689446eee8de9
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
@@ -0,0 +1,71 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// The device reduction helpers emitted by OpenMPIRBuilder have no debug info of
+// their own. If the builder's current debug location is left set while their
+// bodies are emitted, their instructions get DILocations scoped to the
+// wrong subprogram. Check that none of the GPU reduction helpers carry a
+// !dbg attachment.
+
+#di_file = #llvm.di_file<"repro.f90" in "">
+#di_null_type = #llvm.di_null_type
+#loc1 = loc("repro.f90":1:1)
+#loc2 = loc("repro.f90":7:9)
+#di_compile_unit = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_Fortran95, file = #di_file, producer = "flang", isOptimized = true, emissionKind = Full>
+#di_subroutine_type = #llvm.di_subroutine_type<callingConvention = DW_CC_normal, types = #di_null_type>
+#di_subprogram = #llvm.di_subprogram<id = distinct[1]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "k", linkageName = "k_", file = #di_file, line = 1, scopeLine = 1, subprogramFlags = "Definition|Optimized", type = #di_subroutine_type>
+#di_subprogram1 = #llvm.di_subprogram<id = distinct[2]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "__omp_offloading_k_l7", linkageName = "__omp_offloading_k_l7", file = #di_file, line = 7, scopeLine = 7, subprogramFlags = "LocalToUnit|Definition|Optimized", type = #di_subroutine_type>
+#loc14 = loc(fused<#di_subprogram>[#loc1])
+#loc15 = loc(fused<#di_subprogram1>[#loc2])
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr = dense<64> : vector<4xi64>, !llvm.ptr<1> = dense<64> : vector<4xi64>, !llvm.ptr<5> = dense<32> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, "dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true, omp.version = #omp.version<version = 31>} {
+  omp.declare_reduction @add_reduction_f64 : f64 init {
+  ^bb0(%arg0: f64):
+    %0 = llvm.mlir.constant(0.000000e+00 : f64) : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } combiner {
+  ^bb0(%arg0: f64, %arg1: f64):
+    %0 = llvm.fadd %arg0, %arg1 : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } loc(#loc2)
+  llvm.func @k_(%arg0: !llvm.ptr) {
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, f64) map_clauses(from) capture(ByRef) -> !llvm.ptr {name = "s"} loc(#loc2)
+    omp.target kernel_type(spmd) map_entries(%1 -> %arg1 : !llvm.ptr) {
+      %c1 = llvm.mlir.constant(1 : i32) : i32 loc(#loc2)
+      %cn = llvm.mlir.constant(1024 : i32) : i32 loc(#loc2)
+      omp.teams reduction(@add_reduction_f64 %arg1 -> %arg2 : !llvm.ptr) {
+        omp.parallel {
+          omp.distribute {
+            omp.wsloop reduction(@add_reduction_f64 %arg2 -> %arg3 : !llvm.ptr) {
+              omp.loop_nest (%arg4) : i32 = (%c1) to (%cn) inclusive step (%c1) {
+                %2 = llvm.load %arg3 : !llvm.ptr -> f64 loc(#loc2)
+                %cst = llvm.mlir.constant(1.000000e+00 : f64) : f64 loc(#loc2)
+                %3 = llvm.fadd %2, %cst : f64 loc(#loc2)
+                llvm.store %3, %arg3 : f64, !llvm.ptr loc(#loc2)
+                omp.yield loc(#loc2)
+              } loc(#loc2)
+            } {omp.composite} loc(#loc2)
+          } {omp.composite} loc(#loc2)
+          omp.terminator loc(#loc2)
+        } {omp.composite} loc(#loc2)
+        omp.terminator loc(#loc2)
+      } {omp.combined} loc(#loc2)
+      omp.terminator loc(#loc2)
+    } {omp.combined} loc(#loc15)
+    llvm.return loc(#loc14)
+  } loc(#loc14)
+} loc(#loc1)
+
+// CHECK-LABEL: define internal void @_omp_reduction_shuffle_and_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_inter_warp_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_list_to_global_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }


From llvmlistbot at llvm.org  Thu Jul 23 07:27:35 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Thu, 23 Jul 2026 07:27:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a6224d7.e12b82d4.362e3d.1560@mx.google.com>


FedericoBruzzone wrote:

I really appreciate that. Thanks @banach-space 🫶

https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Thu Jul 23 07:32:51 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Thu, 23 Jul 2026 07:32:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a622613.4f48cfb1.17d461.d143@mx.google.com>


takatodo wrote:

I have read the LLVM AI Tool Use Policy and the other linked policies.
AI assistance is disclosed in the PR description under the Assisted-by note.

https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Thu Jul 23 07:32:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:32:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a622619.2730d81b.215fa6.de1b@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Thu Jul 23 07:32:57 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:32:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a622619.bd23cc60.26c84a.1e53@mx.google.com>


https://github.com/banach-space commented:

Thank you for the updates, this is exactly what I had in mind. The overall picture is much clearer now - the table is super helpful!

> Reject statically-known non-unit strides; a dynamic stride is still
accepted (see linked issue for discussion on why this differs from
vector.load/store). 

I can't find the link. Also, now that this has been simplified, we can discuss whether the special-casing implemented in this PR is worth it. 

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Thu Jul 23 07:32:58 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:32:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a62261a.b3e79337.30043.34ef@mx.google.com>


================
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
----------------
banach-space wrote:

If this method is to check the input memref, why is there also a Vector and an Operaton operand? This is too complex. IMO, you can safely replace this with an inline check.

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Thu Jul 23 07:33:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 07:33:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] dcbf875 - [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (#211004)
Message-ID: <6a62262c.8f540080.38641b.f1d5@mx.google.com>


Author: Federico Bruzzone
Date: 2026-07-23T15:33:10+01:00
New Revision: dcbf87542b26480c365b6388b5accf05d0507626

URL: https://github.com/llvm/llvm-project/commit/dcbf87542b26480c365b6388b5accf05d0507626
DIFF: https://github.com/llvm/llvm-project/commit/dcbf87542b26480c365b6388b5accf05d0507626.diff

LOG: [mlir][sparse] Avoid vectorizing non-contiguous COO coordinate loads (#211004)

`SparseVectorization` assumes direct loop accesses (`a[lo:hi]`) are
contiguous and vectorizes them with `vector.maskedload/maskedstore`.
This is false for `sparse_tensor.coordinates` of a level inside a
trailing AoS COO region, whose buffer is interleaved with other levels:
a silent miscompile.

The true stride is already known from the tensor's encoding, even though
the memref type is still dynamic at this point. Use it to fall back to a
scalar loop when the stride is provably non-unit.

---------

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>

Added: 
    mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir

Modified: 
    mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
index 23436a68535fc..c60ca523d81f3 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseVectorization.cpp
@@ -54,6 +54,50 @@ static bool isInvariantArg(BlockArgument arg, Block *block) {
   return arg.getOwner() != block;
 }
 
+/// Returns true when `mem`'s most minor dimension has a statically known
+/// non-unit stride.
+///
+/// `genVectorLoad/genVectorStore` assume a contiguous
+/// `vector.maskedload/vector.maskedstore` is safe for consecutive loop
+/// indices, which breaks for a strided view extracting one component out
+/// of an interleaved (AoS) COO coordinate buffer.
+///
+/// Example:
+/// A `compressed(nonunique) + singleton` region stores coordinates as
+/// `[row0, col0, row1, col1, ...]`, so a 2-lane masked load of `col[0:2]`
+/// (offset=1) would read physical offsets {1, 2} = `[col0, row1]` instead
+/// of the intended {1, 3} = `[col0, col1]`: a silent miscompile.
+///
+/// NOTE: A stride that can't be proven non-unit by either means is assumed
+/// safe.
+static bool hasKnownNonUnitStride(Value mem) {
+  // sparse_tensor.coordinates isn't lowered to a concrete strided memref
+  // until sparse-tensor-codegen runs, so at this point its type
+  // still has a dynamic stride even when the true stride is already known
+  // from the tensor's encoding -- hence the special case below instead of
+  // trusting the memref type.
+  if (auto toCoords = mem.getDefiningOp<ToCoordinatesOp>()) {
+    SparseTensorType stt = getSparseTensorType(toCoords.getTensor());
+    Level cooStart = stt.getAoSCOOStart();
+    // A single trailing level (lvlRank - cooStart == 1) is not actually
+    // interleaved with anything else, so it degenerates to a contiguous
+    // buffer.
+    if (toCoords.getLevel() >= cooStart)
+      return stt.getLvlRank() - cooStart != 1;
+    return false;
+  }
+
+  auto memTp = dyn_cast<MemRefType>(mem.getType());
+  if (!memTp)
+    return false;
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memTp.getStridesAndOffset(strides, offset)))
+    return false;
+  return !strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+         strides.back() != 1;
+}
+
 /// Constructs vector type for element type.
 static VectorType vectorType(VL vl, Type etp) {
   return VectorType::get(vl.vectorLength, etp, vl.enableVLAVectorization);
@@ -292,6 +336,8 @@ static bool vectorizeSubscripts(PatternRewriter &rewriter, scf::ForOp forOp,
     if (auto load = cast.getDefiningOp<memref::LoadOp>()) {
       if (!innermost)
         return false;
+      if (hasKnownNonUnitStride(load.getMemRef()))
+        return false;
       if (codegen) {
         SmallVector<Value> idxs2(load.getIndices()); // no need to analyze
         Location loc = forOp.getLoc();
@@ -408,6 +454,8 @@ static bool vectorizeExpr(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
   // a[lo:hi] = ind[lo:hi], where 'lo' denotes the current index
   // and 'hi = lo + vl - 1'.
   if (auto load = dyn_cast<memref::LoadOp>(def)) {
+    if (hasKnownNonUnitStride(load.getMemRef()))
+      return false;
     auto subs = load.getIndices();
     SmallVector<Value> idxs;
     if (vectorizeSubscripts(rewriter, forOp, vl, subs, codegen, vmask, idxs)) {
@@ -582,6 +630,8 @@ static bool vectorizeStmt(PatternRewriter &rewriter, scf::ForOp forOp, VL vl,
     }
   } else if (auto store = dyn_cast<memref::StoreOp>(last)) {
     // Analyze/vectorize store operation.
+    if (hasKnownNonUnitStride(store.getMemRef()))
+      return false;
     auto subs = store.getIndices();
     SmallVector<Value> idxs;
     Value rhs = store.getValue();

diff  --git a/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir b/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir
new file mode 100644
index 0000000000000..9b13b15a60e87
--- /dev/null
+++ b/mlir/test/Dialect/SparseTensor/sparse_vector_coo.mlir
@@ -0,0 +1,105 @@
+// RUN: mlir-opt %s --sparse-reinterpret-map -sparsification -cse -sparse-vectorization="vl=2" -cse | FileCheck %s
+
+// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
+
+#SortedCOO = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton)
+}>
+
+#trait_index = {
+  indexing_maps = [
+    affine_map<(i,j) -> (i,j)>,  // A
+    affine_map<(i,j) -> (i,j)>   // X (out)
+  ],
+  iterator_types = ["parallel", "parallel"],
+  doc = "X(i,j) = A(i,j) * j"
+}
+
+// CHECK: #[[$ATTR_0:.+]] = #sparse_tensor.encoding<{ map = (d0, d1) -> (d0 : compressed(nonunique), d1 : singleton) }>
+// CHECK-LABEL:   func.func @sparse_index_2d_coo(
+// CHECK-SAME:      %[[ARG0:.*]]: tensor<8x8xi64, #[[$ATTR_0]]>) -> tensor<8x8xi64> {
+// CHECK:           %[[CONSTANT_0:.*]] = arith.constant true
+// CHECK:           %[[CONSTANT_1:.*]] = arith.constant false
+// CHECK:           %[[CONSTANT_2:.*]] = arith.constant 1 : index
+// CHECK:           %[[CONSTANT_3:.*]] = arith.constant 0 : index
+// CHECK:           %[[CONSTANT_4:.*]] = arith.constant 0 : i64
+// CHECK:           %[[EMPTY_0:.*]] = tensor.empty() : tensor<8x8xi64>
+// CHECK:           %[[VALUES_0:.*]] = sparse_tensor.values %[[ARG0]] : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xi64>
+// CHECK:           %[[TO_BUFFER_0:.*]] = bufferization.to_buffer %[[EMPTY_0]] : tensor<8x8xi64> to memref<8x8xi64>
+// CHECK:           linalg.fill ins(%[[CONSTANT_4]] : i64) outs(%[[TO_BUFFER_0]] : memref<8x8xi64>)
+// CHECK:           %[[POSITIONS_0:.*]] = sparse_tensor.positions %[[ARG0]] {level = 0 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex>
+// CHECK:           %[[COORDINATES_0:.*]] = sparse_tensor.coordinates %[[ARG0]] {level = 0 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK:           %[[COORDINATES_1:.*]] = sparse_tensor.coordinates %[[ARG0]] {level = 1 : index} : tensor<8x8xi64, #[[$ATTR_0]]> to memref<?xindex, strided<[?], offset: ?>>
+// CHECK:           %[[LOAD_0:.*]] = memref.load %[[POSITIONS_0]]{{\[}}%[[CONSTANT_3]]] : memref<?xindex>
+// CHECK:           %[[LOAD_1:.*]] = memref.load %[[POSITIONS_0]]{{\[}}%[[CONSTANT_2]]] : memref<?xindex>
+// CHECK:           %[[WHILE_0:.*]] = scf.while (%[[VAL_0:.*]] = %[[LOAD_0]]) : (index) -> index {
+// CHECK:             %[[CMPI_0:.*]] = arith.cmpi ult, %[[VAL_0]], %[[LOAD_1]] : index
+// CHECK:             %[[IF_0:.*]] = scf.if %[[CMPI_0]] -> (i1) {
+// CHECK:               %[[LOAD_2:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[LOAD_0]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:               %[[LOAD_3:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_0]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:               %[[CMPI_1:.*]] = arith.cmpi eq, %[[LOAD_2]], %[[LOAD_3]] : index
+// CHECK:               scf.yield %[[CMPI_1]] : i1
+// CHECK:             } else {
+// CHECK:               scf.yield %[[CONSTANT_1]] : i1
+// CHECK:             }
+// CHECK:             scf.condition(%[[IF_0]]) %[[VAL_0]] : index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_1:.*]]: index):
+// CHECK:             %[[ADDI_0:.*]] = arith.addi %[[VAL_1]], %[[CONSTANT_2]] : index
+// CHECK:             scf.yield %[[ADDI_0]] : index
+// CHECK:           }
+// CHECK:           %[[WHILE_1:.*]]:2 = scf.while (%[[VAL_2:.*]] = %[[LOAD_0]], %[[VAL_3:.*]] = %[[WHILE_0]]) : (index, index) -> (index, index) {
+// CHECK:             %[[CMPI_2:.*]] = arith.cmpi ult, %[[VAL_2]], %[[LOAD_1]] : index
+// CHECK:             scf.condition(%[[CMPI_2]]) %[[VAL_2]], %[[VAL_3]] : index, index
+// CHECK:           } do {
+// CHECK:           ^bb0(%[[VAL_4:.*]]: index, %[[VAL_5:.*]]: index):
+// CHECK:             %[[LOAD_4:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_4]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:             scf.if %[[CONSTANT_0]] {
+// CHECK:               scf.for %[[VAL_6:.*]] = %[[VAL_4]] to %[[VAL_5]] step %[[CONSTANT_2]] {
+// CHECK:                 %[[LOAD_5:.*]] = memref.load %[[COORDINATES_1]]{{\[}}%[[VAL_6]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                 %[[LOAD_6:.*]] = memref.load %[[VALUES_0]]{{\[}}%[[VAL_6]]] : memref<?xi64>
+// CHECK:                 %[[INDEX_CAST_0:.*]] = arith.index_cast %[[LOAD_5]] : index to i64
+// CHECK:                 %[[MULI_0:.*]] = arith.muli %[[LOAD_6]], %[[INDEX_CAST_0]] : i64
+// CHECK:                 memref.store %[[MULI_0]], %[[TO_BUFFER_0]]{{\[}}%[[LOAD_4]], %[[LOAD_5]]] : memref<8x8xi64>
+// CHECK:               } {"Emitted from" = "linalg.generic"}
+// CHECK:             } else {
+// CHECK:             }
+// CHECK:             %[[IF_1:.*]]:2 = scf.if %[[CONSTANT_0]] -> (index, index) {
+// CHECK:               %[[WHILE_2:.*]] = scf.while (%[[VAL_7:.*]] = %[[VAL_5]]) : (index) -> index {
+// CHECK:                 %[[CMPI_3:.*]] = arith.cmpi ult, %[[VAL_7]], %[[LOAD_1]] : index
+// CHECK:                 %[[IF_2:.*]] = scf.if %[[CMPI_3]] -> (i1) {
+// CHECK:                   %[[LOAD_7:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_5]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                   %[[LOAD_8:.*]] = memref.load %[[COORDINATES_0]]{{\[}}%[[VAL_7]]] : memref<?xindex, strided<[?], offset: ?>>
+// CHECK:                   %[[CMPI_4:.*]] = arith.cmpi eq, %[[LOAD_7]], %[[LOAD_8]] : index
+// CHECK:                   scf.yield %[[CMPI_4]] : i1
+// CHECK:                 } else {
+// CHECK:                   scf.yield %[[CONSTANT_1]] : i1
+// CHECK:                 }
+// CHECK:                 scf.condition(%[[IF_2]]) %[[VAL_7]] : index
+// CHECK:               } do {
+// CHECK:               ^bb0(%[[VAL_8:.*]]: index):
+// CHECK:                 %[[ADDI_1:.*]] = arith.addi %[[VAL_8]], %[[CONSTANT_2]] : index
+// CHECK:                 scf.yield %[[ADDI_1]] : index
+// CHECK:               }
+// CHECK:               scf.yield %[[VAL_5]], %[[WHILE_2]] : index, index
+// CHECK:             } else {
+// CHECK:               scf.yield %[[VAL_4]], %[[VAL_5]] : index, index
+// CHECK:             }
+// CHECK:             scf.yield %[[VAL_9:.*]]#0, %[[VAL_9]]#1 : index, index
+// CHECK:           } attributes {"Emitted from" = "linalg.generic"}
+// CHECK:           %[[TO_TENSOR_0:.*]] = bufferization.to_tensor %[[TO_BUFFER_0]] : memref<8x8xi64> to tensor<8x8xi64>
+// CHECK:           return %[[TO_TENSOR_0]] : tensor<8x8xi64>
+// CHECK:         }
+func.func @sparse_index_2d_coo(%arga: tensor<8x8xi64, #SortedCOO>) -> tensor<8x8xi64> {
+  %init = tensor.empty() : tensor<8x8xi64>
+  %r = linalg.generic #trait_index
+      ins(%arga: tensor<8x8xi64, #SortedCOO>)
+     outs(%init: tensor<8x8xi64>) {
+      ^bb(%a: i64, %x: i64):
+        %j = linalg.index 1 : index
+        %jj = arith.index_cast %j : index to i64
+        %m1 = arith.muli %a, %jj : i64
+        linalg.yield %m1 : i64
+  } -> tensor<8x8xi64>
+  return %r : tensor<8x8xi64>
+}


        

From llvmlistbot at llvm.org  Thu Jul 23 07:33:18 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:33:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Avoid vectorizing
 non-contiguous COO coordinate loads (PR #211004)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211004@github.com>
Message-ID: <6a62262e.cfe63a49.b7b1f.ace4@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/211004

From llvmlistbot at llvm.org  Thu Jul 23 07:35:35 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Thu, 23 Jul 2026 07:35:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a6226b7.54f1ce2e.2541a0.26e8@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH] [mlir][affine] Treat failed dependence checks conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}


From llvmlistbot at llvm.org  Thu Jul 23 07:37:40 2026
From: llvmlistbot at llvm.org (Mircea Trofin)
Date: Thu, 23 Jul 2026 07:37:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add `class-name-format` option
 to `wrap-emitc-func-in-class` pass (PR #211147)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211147@github.com>
Message-ID: <6a622734.4d48cfb1.2e96e7.c67b@mx.google.com>


https://github.com/mtrofin approved this pull request.


https://github.com/llvm/llvm-project/pull/211147

From llvmlistbot at llvm.org  Thu Jul 23 07:38:38 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 23 Jul 2026 07:38:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add more informative error
 messages to block scaled types (PR #211569)
Message-ID: <llvm.org/llvm/llvm-project/pull/211569@github.com>


https://github.com/lhutton1 created https://github.com/llvm/llvm-project/pull/211569

This commit improves block scaled tensor type verification to provide a specific reason about why type verification failed. Previously the error message was a very generic "must be tosa-conformant tensor of number values".

This commit restores previous functionality that was reverted by https://github.com/llvm/llvm-project/pull/207995, but uses the ODS string interpolation mechanism instead to prevent incompatibility with PDLL.

>From 13ca3e97bb22a18e4fa06550dbfd75cf0e86d2be Mon Sep 17 00:00:00 2001
From: Luke Hutton <luke.hutton at arm.com>
Date: Wed, 15 Jul 2026 11:31:41 +0100
Subject: [PATCH] [mlir][tosa] Add more informative error messages to block
 scaled types

This commit improves block scaled tensor type verification to provide
a specific reason about why type verification failed. Previously the
error message was a very generic
"must be tosa-conformant tensor of number values".

This commit resores previous functionality that was reverted by
https://github.com/llvm/llvm-project/pull/207995, but uses the ODS
string interpolation mechanism instead to prevent incompatibility
with PDLL.

Change-Id: I07f74efe8da9b3bdbae46fd14704d47ab96ec174
---
 mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h   | 13 ++++-
 .../mlir/Dialect/Tosa/IR/TosaTypesBase.td     | 22 +++++--
 mlir/lib/Dialect/Tosa/IR/TosaOps.cpp          | 58 ++++++++++++++++---
 mlir/test/Dialect/Tosa/verifier.mlir          | 10 ++--
 4 files changed, 81 insertions(+), 22 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index b1404d9c700ae..2c4d3b7446544 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -25,6 +25,8 @@
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Interfaces/VectorInterfaces.h"
 
+#include <string>
+
 //===----------------------------------------------------------------------===//
 // TOSA dialect and structs includes.
 //===----------------------------------------------------------------------===//
@@ -141,9 +143,14 @@ Type getStorageElementTypeOrSelf(Type type);
 // Returns the storage element type for a given value
 Type getStorageElementTypeOrSelf(Value value);
 
-// Verify a block scaled tensor type is valid
-LogicalResult verifyBlockScaledTensorType(mlir::Type type,
-                                          bool allowScaleValues);
+// Verify that a given type is a valid block scaled tensor type
+LogicalResult verifyBlockScaledTensorType(
+    mlir::Type type,
+    llvm::function_ref<mlir::InFlightDiagnostic()> emitError = nullptr,
+    bool allowScaleValues = false);
+
+// Collect error messages for a given type
+std::string getTosaTensorTypeErrorMessage(mlir::Type type);
 
 } // namespace tosa
 } // namespace mlir
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 2ab135befaa96..9252223cadb3c 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -172,7 +172,19 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::"
+            "verifyBlockScaledTensorType($_self))">;
+
+// Analogous to ShapedContainerType, but with additional 'summary' information.
+// This is used to provide more informative error messages when a type predicate
+// is false.
+class TosaShapedContainerType<list<Type> allowedTypes, Pred containerPred, string descr, string cppType = "::mlir::Type"> :
+    Type<And<[containerPred,
+              Concat<"[](::mlir::Type elementType) { return ",
+                SubstLeaves<"$_self", "elementType",
+                AnyTypeOf<allowedTypes>.predicate>,
+                "; }(getElementTypeOrSelf($_self))">]>,
+         descr # " of " # AnyTypeOf<allowedTypes>.summary # " values" # "{{::mlir::tosa::getTosaTensorTypeErrorMessage($_self)}}", cppType>;
 
 // We include unranked tensors as a supported type for all possible tosa
 // Tensors as unranked does not guarantee invalid. If unranked tensors exist
@@ -181,13 +193,13 @@ def IsValidBlockScaledTensorType
 class TosaTensorOf<list<Type> allowedTypes,
                    list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes,
-               [Or<[
+    : TosaShapedContainerType<allowedTypes,
+               And<[Or<[
                     IsUnrankedTensorTypePred, 
                     And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
                 ]>,
-                IsValidBlockScaledTensorType],
-               summary>;
+                IsValidBlockScaledTensorType]>,
+               summary, "::mlir::TensorType">;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaTensorOf<allowedTypes,
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index 0dfcd23504019..7bfc3de98d08c 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -741,22 +741,31 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
-                                                      bool allowScaleValues) {
+LogicalResult mlir::tosa::verifyBlockScaledTensorType(
+    mlir::Type type, llvm::function_ref<mlir::InFlightDiagnostic()> emitError,
+    bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
-  if (!allowScaleValues && elemType.hasScaleValues())
+  if (!allowScaleValues && elemType.hasScaleValues()) {
+    if (emitError)
+      emitError()
+          << "block scaled tensor type with scale values is not allowed";
     return failure();
+  }
 
   if (!tensorType.hasRank())
     return success();
 
-  if (tensorType.getRank() == 0)
+  if (tensorType.getRank() == 0) {
+    if (emitError)
+      emitError() << "block scaled tensor type must have rank greater than "
+                     "zero";
     return failure();
+  }
 
   const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
@@ -765,19 +774,47 @@ LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
   if (allowScaleValues && elemType.hasScaleValues() &&
       tensorType.hasStaticShape()) {
     const size_t numBlocks = tensorType.getNumElements() / blockSize;
-    if (elemType.getScaleValues().size() != numBlocks)
+    if (elemType.getScaleValues().size() != numBlocks) {
+      if (emitError)
+        emitError() << "block scaled tensor type with scale values must have "
+                       "scale values for each block, expected "
+                    << numBlocks << ", got "
+                    << elemType.getScaleValues().size();
       return failure();
+    }
   }
 
   const int64_t blockedDimension = tensorShape.back();
   if (ShapedType::isDynamic(blockedDimension))
     return success();
-  if (blockedDimension % blockSize != 0)
+
+  if (blockedDimension % blockSize != 0) {
+    if (emitError)
+      emitError() << "last dimension of block scaled tensor type ("
+                  << blockedDimension << ") must be divisible by block size ("
+                  << blockSize << ")";
+
     return failure();
+  }
 
   return success();
 }
 
+std::string mlir::tosa::getTosaTensorTypeErrorMessage(mlir::Type type) {
+  MLIRContext *ctx = type.getContext();
+  std::string message;
+  ScopedDiagnosticHandler handler(
+      ctx, [&](Diagnostic &diag) { message = diag.str(); });
+
+  if (failed(verifyBlockScaledTensorType(
+          type, [ctx] { return emitError(UnknownLoc::get(ctx)); })) &&
+      !message.empty()) {
+    return ": " + message;
+  }
+
+  return "";
+}
+
 static ParseResult parseScaleValues(AsmParser &parser,
                                     SmallVector<Attribute> &scaleValues,
                                     Type scaleType) {
@@ -964,9 +1001,12 @@ LogicalResult tosa::ConstOp::verify() {
       return op.emitOpError(
           "attribute block scaled type must have scale values");
 
-    if (failed(verifyBlockScaledTensorType(attrType, true)))
-      return op.emitOpError("block scaled attribute type is not valid, got ")
-             << attrType;
+    const auto emitAttributeError = [&op]() {
+      return op.emitOpError("attribute block scaled type is invalid: ");
+    };
+
+    if (failed(verifyBlockScaledTensorType(attrType, emitAttributeError, true)))
+      return failure();
 
     const BlockScaledType resultBlockScaledType =
         llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index ba69e4aba632e..f23f822546bb4 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1705,7 +1705,7 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: last dimension of block scaled tensor type (31) must be divisible by block size (32), but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
 }
@@ -1713,7 +1713,7 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: block scaled tensor type must have rank greater than zero, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>
 }
@@ -2346,7 +2346,7 @@ func.func @test_block_scaled_const_scale_value_non_float_explicit_type() -> tens
 !mxint8_scale = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0, 4.0}>
 
 func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid: block scaled tensor type with scale values must have scale values for each block, expected 2, got 3}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!mxint8_scale> : 0 : i8>}> : () -> tensor<2x32x!mxint8>
   return %0 : tensor<2x32x!mxint8>
 }
@@ -2354,7 +2354,7 @@ func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8>
 // -----
 
 func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid: block scaled tensor type with scale values must have scale values for each block, expected 4, got 2}}
   %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0}>> : 0 : i8>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
   return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
@@ -2362,7 +2362,7 @@ func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor
 // -----
 
 func.func @test_block_scaled_const_cast_scale_values_propagate() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> {
-  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values: block scaled tensor type with scale values is not allowed, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
   return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
 }


From llvmlistbot at llvm.org  Thu Jul 23 07:38:42 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:38:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][EmitC] Add `class-name-format` option
 to `wrap-emitc-func-in-class` pass (PR #211147)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211147@github.com>
Message-ID: <6a622772.866f9993.23f309.967c@mx.google.com>


https://github.com/banach-space approved this pull request.


https://github.com/llvm/llvm-project/pull/211147

From llvmlistbot at llvm.org  Thu Jul 23 07:39:10 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 23 Jul 2026 07:39:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa][WIP] More informative error
 messages on block scaled types (PR #209736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209736@github.com>
Message-ID: <6a62278e.3493695c.13acfd.cbb0@mx.google.com>


================
@@ -172,35 +172,42 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::"
+            "verifyBlockScaledTensorType($_self))">;
+
+// Analogous to ShapedContainerType, but with additional 'summary' information.
+// This is used to provide more informative error messages when a type predicate
+// is false.
+class TosaShapedContainerType<list<Type> allowedTypes, Pred containerPred, string descr, string cppType = "::mlir::Type"> :
+    Type<And<[containerPred,
+              Concat<"[](::mlir::Type elementType) { return ",
+                SubstLeaves<"$_self", "elementType",
+                AnyTypeOf<allowedTypes>.predicate>,
+                "; }(::llvm::cast<::mlir::ShapedType>($_self).getElementType())">]>,
----------------
lhutton1 wrote:

Thanks, I've addressed in https://github.com/llvm/llvm-project/pull/211569

https://github.com/llvm/llvm-project/pull/209736

From llvmlistbot at llvm.org  Thu Jul 23 07:39:13 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 23 Jul 2026 07:39:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa][WIP] More informative error
 messages on block scaled types (PR #209736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209736@github.com>
Message-ID: <6a622791.04e40411.17b42c.607f@mx.google.com>


https://github.com/lhutton1 closed https://github.com/llvm/llvm-project/pull/209736

From llvmlistbot at llvm.org  Thu Jul 23 07:40:23 2026
From: llvmlistbot at llvm.org (Stephen Long)
Date: Thu, 23 Jul 2026 07:40:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a6227d7.101bc966.36efd6.c541@mx.google.com>


https://github.com/steplong updated https://github.com/llvm/llvm-project/pull/211354

>From 3c8ac21e769cdb7712ca7317daa492651220bdb0 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 11:04:40 -0700
Subject: [PATCH 1/5] [LinalgBlockPackMatmul] Add support for scalable block
 factors

Assisted by: Claude Sonnet 4.6
---
 mlir/include/mlir/Dialect/Linalg/Passes.td    |  8 +-
 .../Dialect/Linalg/Transforms/Transforms.h    |  5 +
 .../Linalg/Transforms/BlockPackMatmul.cpp     | 57 +++++++++++-
 .../Linalg/block-pack-matmul-scalable.mlir    | 91 +++++++++++++++++++
 4 files changed, 155 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir

diff --git a/mlir/include/mlir/Dialect/Linalg/Passes.td b/mlir/include/mlir/Dialect/Linalg/Passes.td
index 3a43af9ca1855..5573cf8b9feb7 100644
--- a/mlir/include/mlir/Dialect/Linalg/Passes.td
+++ b/mlir/include/mlir/Dialect/Linalg/Passes.td
@@ -215,10 +215,12 @@ def LinalgBlockPackMatmul : Pass<"linalg-block-pack-matmul"> {
       %res = unpack %res_packed : 4D <MBxNBxmbxnb> -> 2D <MxN>
     ```
   }];
-  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect"];
+  let dependentDialects = ["linalg::LinalgDialect", "tensor::TensorDialect",
+                           "arith::ArithDialect",
+                           "vector::VectorDialect"];
   let options = [
-    ListOption<"blockFactors", "block-factors", "int64_t",
-               "Block factors (mb, nb, kb) for relayout">,
+    ListOption<"blockFactors", "block-factors", "std::string",
+               "Block factors (mb, nb, kb) for relayout; use [N] for scalable">,
     Option<"allowPadding", "allow-padding", "bool",
            /*default=*/"true",
            "Allow packing padding">,
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 836682de4c404..9a59fb5ea7931 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -1423,6 +1423,11 @@ struct BlockPackMatmulOptions {
   /// the parallel dimensions and kb is the reduction dimension.
   SmallVector<int64_t, 3> blockFactors;
 
+  /// Scalable flags for block factors. When true, the corresponding block
+  /// factor is multiplied by vector.vscale at runtime (SVE-style scalable).
+  /// Must be empty (all static) or have the same size as blockFactors.
+  SmallVector<bool, 3> scalableBlockFactors;
+
   /// If true, allows packing of dimensions that only partially fit into the
   /// block factors.
   bool allowPadding = true;
diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index 6ea1eb50b13ce..cfc91fe537ccd 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -8,9 +8,11 @@
 
 #include "mlir/Dialect/Linalg/Passes.h"
 
+#include "mlir/Dialect/Arith/IR/Arith.h"
 #include "mlir/Dialect/Linalg/IR/Linalg.h"
 #include "mlir/Dialect/Linalg/Transforms/Transforms.h"
 #include "mlir/Dialect/Linalg/Utils/Utils.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/IR/PatternMatch.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
 #include "llvm/ADT/SmallVector.h"
@@ -161,8 +163,35 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   if (options->blockFactors.size() != 3)
     return rewriter.notifyMatchFailure(linalgOp, "require 3 tile factors");
 
-  SmallVector<OpFoldResult> mnkTiles =
-      getAsOpFoldResult(rewriter.getI64ArrayAttr(options->blockFactors));
+  bool hasScalable = !options->scalableBlockFactors.empty();
+  if (hasScalable && options->scalableBlockFactors.size() != 3)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalableBlockFactors must be empty or have 3 elements");
+
+  // Scalable tile sizes are non-constant at compile time, so they can never
+  // satisfy the full-tile divisibility check. Reject early before creating
+  // any ops to avoid modifying IR before returning notifyMatchFailure.
+  if (!options->allowPadding && hasScalable)
+    return rewriter.notifyMatchFailure(
+        linalgOp, "scalable block factors require allow-padding=true");
+
+  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
+  // arith.constant N : index multiplied by vector.vscale.
+  SmallVector<OpFoldResult> mnkTiles;
+  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
+    bool isScalable = hasScalable && options->scalableBlockFactors[i];
+    if (!isScalable) {
+      mnkTiles.push_back(rewriter.getIndexAttr(factor));
+      continue;
+    }
+    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
+                                               factor);
+    Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
+                                                 rewriter.getIndexType());
+    mnkTiles.push_back(
+        arith::MulIOp::create(rewriter, linalgOp.getLoc(), cst, vscale)
+            .getResult());
+  }
 
   // If padding is disabled, make sure that dimensions can be packed cleanly.
   if (!options->allowPadding &&
@@ -300,7 +329,29 @@ struct LinalgBlockPackMatmul
     ControlBlockPackMatmulFn controlFn =
         [&](linalg::LinalgOp op) -> BlockPackMatmulOptions {
       BlockPackMatmulOptions options;
-      options.blockFactors = SmallVector<int64_t>{*blockFactors};
+
+      // Parse block-factors strings. Each element is either "N" (static) or
+      // "[N]" (scalable, i.e. N * vscale at runtime).
+      for (const std::string &f : *blockFactors) {
+        StringRef s(f);
+        if (s.starts_with("[") && s.ends_with("]")) {
+          int64_t val = 0;
+          s.drop_front().drop_back().getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(true);
+        } else {
+          int64_t val = 0;
+          s.getAsInteger(10, val);
+          options.blockFactors.push_back(val);
+          options.scalableBlockFactors.push_back(false);
+        }
+      }
+      // If all flags are false, clear the vector so blockPackMatmul can take
+      // the cheaper static path.
+      if (llvm::none_of(options.scalableBlockFactors,
+                        [](bool b) { return b; }))
+        options.scalableBlockFactors.clear();
+
       options.allowPadding = allowPadding;
       options.mnkPaddedSizesNextMultipleOf =
           SmallVector<int64_t>{*mnkPaddedSizesNextMultipleOf};
diff --git a/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
new file mode 100644
index 0000000000000..a5829c210d1a8
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/block-pack-matmul-scalable.mlir
@@ -0,0 +1,91 @@
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=[32],[16],[64] allow-padding=0" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=SCALABLE-NOPAD
+
+// RUN: mlir-opt %s -linalg-block-pack-matmul="block-factors=32,16,[64] allow-padding=1" \
+// RUN: -canonicalize -split-input-file | FileCheck %s --check-prefix=MIXED
+
+// -----
+
+// All-scalable block factors, static input shapes, allow-padding=1.
+// Inner tile sizes and outer dimensions are all dynamic (vscale-relative).
+
+func.func @block_matmul_scalable_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// Scalable factors with allow-padding=0: transform does not apply.
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_static(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// All-scalable block factors, dynamic input shapes, allow-padding=1.
+// Both outer tile counts and inner tile sizes are fully dynamic.
+
+func.func @block_matmul_scalable_dynamic(
+    %A: tensor<?x?xf32>, %B: tensor<?x?xf32>, %C: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<?x?xf32>, tensor<?x?xf32>)
+                     outs(%C : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %0 : tensor<?x?xf32>
+}
+
+// SCALABLE-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-DAG: %[[C32:.+]] = arith.constant 32 : index
+// SCALABLE-DAG: %[[C16:.+]] = arith.constant 16 : index
+// SCALABLE-DAG: %[[C64:.+]] = arith.constant 64 : index
+// SCALABLE-DAG: %[[MB:.+]] = arith.muli %{{.+}}, %[[C32]] : index
+// SCALABLE-DAG: %[[NB:.+]] = arith.muli %{{.+}}, %[[C16]] : index
+// SCALABLE-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [%[[NB]], %[[KB]]]
+// SCALABLE: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+// SCALABLE: linalg.generic
+// SCALABLE: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [%[[MB]], %[[NB]]]
+
+// SCALABLE-NOPAD-LABEL: func @block_matmul_scalable_dynamic(
+// SCALABLE-NOPAD-NOT: linalg.pack
+// SCALABLE-NOPAD: linalg.matmul
+// SCALABLE-NOPAD-NOT: linalg.unpack
+
+// -----
+
+// Mixed block factors (mb=32, nb=16 static; kb=[64] scalable), static input.
+// Only the K-dimension inner tile is scalable; mb and nb remain static.
+
+func.func @block_matmul_mixed_static(
+    %A: tensor<128x128xf32>, %B: tensor<128x128xf32>, %C: tensor<128x128xf32>) -> tensor<128x128xf32> {
+  %0 = linalg.matmul ins(%A, %B : tensor<128x128xf32>, tensor<128x128xf32>)
+                     outs(%C : tensor<128x128xf32>) -> tensor<128x128xf32>
+  return %0 : tensor<128x128xf32>
+}
+
+// MIXED-LABEL: func @block_matmul_mixed_static(
+// MIXED-DAG: %[[C64:.+]] = arith.constant 64 : index
+// MIXED-DAG: %[[KB:.+]] = arith.muli %{{.+}}, %[[C64]] : index
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [0, 1] inner_dims_pos = [0, 1] inner_tiles = [32, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} outer_dims_perm = [1, 0] inner_dims_pos = [1, 0] inner_tiles = [16, %[[KB]]]
+// MIXED: linalg.pack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+// MIXED: linalg.generic
+// MIXED: linalg.unpack %{{.+}} inner_dims_pos = [0, 1] inner_tiles = [32, 16]
+

>From 46852e32748c6ffab6973f12de9c06e5c6d6d060 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Wed, 22 Jul 2026 14:34:41 -0700
Subject: [PATCH 2/5] clang-format PR

---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index cfc91fe537ccd..ebc287463bec2 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -184,8 +184,8 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
       mnkTiles.push_back(rewriter.getIndexAttr(factor));
       continue;
     }
-    Value cst = arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(),
-                                               factor);
+    Value cst =
+        arith::ConstantIndexOp::create(rewriter, linalgOp.getLoc(), factor);
     Value vscale = vector::VectorScaleOp::create(rewriter, linalgOp.getLoc(),
                                                  rewriter.getIndexType());
     mnkTiles.push_back(
@@ -348,8 +348,7 @@ struct LinalgBlockPackMatmul
       }
       // If all flags are false, clear the vector so blockPackMatmul can take
       // the cheaper static path.
-      if (llvm::none_of(options.scalableBlockFactors,
-                        [](bool b) { return b; }))
+      if (llvm::none_of(options.scalableBlockFactors, [](bool b) { return b; }))
         options.scalableBlockFactors.clear();
 
       options.allowPadding = allowPadding;

>From c80811912e002c40c448ab690c13a6b97a5f5b4d Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 23 Jul 2026 09:37:08 -0400
Subject: [PATCH 3/5] Update
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp

Co-authored-by: Adam Siemieniuk <adam.siemieniuk at intel.com>
---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index ebc287463bec2..e6147c8814b9d 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -173,7 +173,7 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   // any ops to avoid modifying IR before returning notifyMatchFailure.
   if (!options->allowPadding && hasScalable)
     return rewriter.notifyMatchFailure(
-        linalgOp, "scalable block factors require allow-padding=true");
+        linalgOp, "scalable block factors require padding");
 
   // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
   // arith.constant N : index multiplied by vector.vscale.

>From 3f966c039d99578847d20ae6d06318c6f4ef3bb7 Mon Sep 17 00:00:00 2001
From: Stephen Long <63318318+steplong at users.noreply.github.com>
Date: Thu, 23 Jul 2026 09:37:21 -0400
Subject: [PATCH 4/5] Update
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp

Co-authored-by: Adam Siemieniuk <adam.siemieniuk at intel.com>
---
 mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index e6147c8814b9d..ba64b6b2e4447 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -178,7 +178,7 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
   // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
   // arith.constant N : index multiplied by vector.vscale.
   SmallVector<OpFoldResult> mnkTiles;
-  for (auto [i, factor] : llvm::enumerate(options->blockFactors)) {
+  for (auto [idx, factor] : llvm::enumerate(options->blockFactors)) {
     bool isScalable = hasScalable && options->scalableBlockFactors[i];
     if (!isScalable) {
       mnkTiles.push_back(rewriter.getIndexAttr(factor));

>From f9e0d4f1d982c54023e18c8c99c86c65a8caf014 Mon Sep 17 00:00:00 2001
From: Stephen Long <steplong at qti.qualcomm.com>
Date: Thu, 23 Jul 2026 07:40:02 -0700
Subject: [PATCH 5/5] clean up src files

---
 .../mlir/Dialect/Linalg/Transforms/Transforms.h    |  4 ++--
 .../Dialect/Linalg/Transforms/BlockPackMatmul.cpp  | 14 ++++++--------
 2 files changed, 8 insertions(+), 10 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 9a59fb5ea7931..130f7f6fd2792 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -1424,8 +1424,8 @@ struct BlockPackMatmulOptions {
   SmallVector<int64_t, 3> blockFactors;
 
   /// Scalable flags for block factors. When true, the corresponding block
-  /// factor is multiplied by vector.vscale at runtime (SVE-style scalable).
-  /// Must be empty (all static) or have the same size as blockFactors.
+  /// factor is considered scalable. Must be empty (all static) or have the same
+  /// size as blockFactors.
   SmallVector<bool, 3> scalableBlockFactors;
 
   /// If true, allows packing of dimensions that only partially fit into the
diff --git a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
index ba64b6b2e4447..1f4cda459e738 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/BlockPackMatmul.cpp
@@ -175,11 +175,9 @@ linalg::blockPackMatmul(RewriterBase &rewriter, linalg::LinalgOp linalgOp,
     return rewriter.notifyMatchFailure(
         linalgOp, "scalable block factors require padding");
 
-  // Build OpFoldResult tile sizes. Scalable dimensions are emitted as
-  // arith.constant N : index multiplied by vector.vscale.
   SmallVector<OpFoldResult> mnkTiles;
   for (auto [idx, factor] : llvm::enumerate(options->blockFactors)) {
-    bool isScalable = hasScalable && options->scalableBlockFactors[i];
+    bool isScalable = hasScalable && options->scalableBlockFactors[idx];
     if (!isScalable) {
       mnkTiles.push_back(rewriter.getIndexAttr(factor));
       continue;
@@ -332,16 +330,16 @@ struct LinalgBlockPackMatmul
 
       // Parse block-factors strings. Each element is either "N" (static) or
       // "[N]" (scalable, i.e. N * vscale at runtime).
-      for (const std::string &f : *blockFactors) {
-        StringRef s(f);
-        if (s.starts_with("[") && s.ends_with("]")) {
+      for (const std::string &blockFactor : *blockFactors) {
+        StringRef factor(blockFactor);
+        if (factor.starts_with("[") && factor.ends_with("]")) {
           int64_t val = 0;
-          s.drop_front().drop_back().getAsInteger(10, val);
+          factor.drop_front().drop_back().getAsInteger(10, val);
           options.blockFactors.push_back(val);
           options.scalableBlockFactors.push_back(true);
         } else {
           int64_t val = 0;
-          s.getAsInteger(10, val);
+          factor.getAsInteger(10, val);
           options.blockFactors.push_back(val);
           options.scalableBlockFactors.push_back(false);
         }


From llvmlistbot at llvm.org  Thu Jul 23 07:40:32 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 07:40:32 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 8b690a0 - [mlir][vector] Make CompressstoreOp
 + ExpandloadOp support scalable vectors (#210288)
Message-ID: <6a6227e0.06fa26f6.27dc55.1aff@mx.google.com>


Author: Andrzej Warzyński
Date: 2026-07-23T15:40:26+01:00
New Revision: 8b690a085406337f7a02ab466df494bce5f75f41

URL: https://github.com/llvm/llvm-project/commit/8b690a085406337f7a02ab466df494bce5f75f41
DIFF: https://github.com/llvm/llvm-project/commit/8b690a085406337f7a02ab466df494bce5f75f41.diff

LOG: [mlir][vector] Make CompressstoreOp + ExpandloadOp support scalable vectors (#210288)

Extends `vector.compressstore` + `vector.expandload` to support scalable
vectors and updates relevant tests.

An e2e test for `vector.compressstore` is added. For
`vector.expandload`, we need to wait for QEMU support:
https://github.com/llvm/llvm-project/issues/210942.

Added: 
    mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir

Modified: 
    mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
    mlir/lib/Dialect/Vector/IR/VectorOps.cpp
    mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
    mlir/test/Dialect/Vector/invalid.mlir
    mlir/test/Dialect/Vector/ops.mlir
    mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
    mlir/test/Dialect/Vector/vector-mem-transforms.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
index e3797d7cac5b4..24fdb4f0bb258 100644
--- a/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/IR/VectorOps.td
@@ -2289,7 +2289,7 @@ def Vector_ExpandLoadOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemRead]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$pass_thru,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)>,
     Results<(outs AnyVectorOfNonZeroRank:$result)> {
@@ -2328,8 +2328,6 @@ def Vector_ExpandLoadOp :
     memory at an address aligned to this boundary. Violating this requirement
     triggers immediate undefined behavior.
 
-    Note, at the moment this Op is only available for fixed-width vectors.
-
     Examples:
 
     ```mlir
@@ -2381,7 +2379,7 @@ def Vector_CompressStoreOp :
     ]>,
     Arguments<(ins Arg<AnyMemRef, "", [MemWrite]>:$base,
                Variadic<Index>:$indices,
-               FixedVectorOfNonZeroRankOf<[I1]>:$mask,
+               VectorOfNonZeroRankOf<[I1]>:$mask,
                AnyVectorOfNonZeroRank:$valueToStore,
                OptionalAttr<IntValidAlignment<I64Attr>>: $alignment)> {
 
@@ -2419,8 +2417,6 @@ def Vector_CompressStoreOp :
     memory at an address aligned to this boundary. Violating this requirement
     triggers immediate undefined behavior.
 
-    Note, at the moment this Op is only available for fixed-width vectors.
-
     Examples:
 
     ```mlir

diff  --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index f37083803a2a1..9322c11d401d9 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6655,6 +6655,9 @@ LogicalResult ExpandLoadOp::verify() {
     return emitOpError("requires ") << memType.getRank() << " indices";
   if (resVType.getShape() != maskVType.getShape())
     return emitOpError("expected result shape to match mask shape");
+  if (resVType.getScalableDims() != maskVType.getScalableDims())
+    return emitOpError(
+        "expected result scalable dims to match mask scalable dims");
   if (resVType != passVType)
     return emitOpError("expected pass_thru of same type as result type");
   return success();
@@ -6709,6 +6712,9 @@ LogicalResult CompressStoreOp::verify() {
     return emitOpError("requires ") << memType.getRank() << " indices";
   if (valueVType.getShape() != maskVType.getShape())
     return emitOpError("expected valueToStore shape to match mask shape");
+  if (valueVType.getScalableDims() != maskVType.getScalableDims())
+    return emitOpError(
+        "expected valueToStore scalable dims to match mask scalable dims");
   return success();
 }
 

diff  --git a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
index cc35c6600d12c..4cbea04e8076e 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-to-llvm-interface.mlir
@@ -1944,13 +1944,13 @@ func.func @negative_scatter_on_strided_memref(%arg0: memref<?xf32, strided<[2],
 // vector.expandload
 //===----------------------------------------------------------------------===//
 
-func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
+func.func @expandload(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) -> vector<11xf32> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32> into vector<11xf32>
   return %0 : vector<11xf32>
 }
 
-// CHECK-LABEL: func @expand_load_op
+// CHECK-LABEL: func @expandload
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1959,21 +1959,36 @@ func.func @expand_load_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vec
 
 // -----
 
-func.func @expand_load_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
+func.func @expandload_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) -> vector<[11]xf32> {
+  %c0 = arith.constant 0: index
+  %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32> into vector<[11]xf32>
+  return %0 : vector<[11]xf32>
+}
+
+// CHECK-LABEL: func @expandload_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: %[[E:.*]] = "llvm.intr.masked.expandload"(%[[P]], %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<[11]xi1>, vector<[11]xf32>) -> vector<[11]xf32>
+// CHECK: return %[[E]] : vector<[11]xf32>
+
+// -----
+
+func.func @expandload_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) -> vector<11xindex> {
   %c0 = arith.constant 0: index
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_index
+// CHECK-LABEL: func @expandload_index
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
 
-func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
+func.func @expandload_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) -> vector<11xindex> {
   %0 = vector.expandload %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex> into vector<11xindex>
   return %0 : vector<11xindex>
 }
-// CHECK-LABEL: func @expand_load_op_with_alignment
+// CHECK-LABEL: func @expandload_with_alignment
 // CHECK: %{{.*}} = "llvm.intr.masked.expandload"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{llvm.align = 8 : i64}, {}, {}]}> : (!llvm.ptr, vector<11xi1>, vector<11xi64>) -> vector<11xi64>
 
 // -----
@@ -1982,13 +1997,13 @@ func.func @expand_load_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<1
 // vector.compressstore
 //===----------------------------------------------------------------------===//
 
-func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
+func.func @compressstore(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2: vector<11xf32>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<11xi1>, vector<11xf32>
   return
 }
 
-// CHECK-LABEL: func @compress_store_op
+// CHECK-LABEL: func @compressstore
 // CHECK: %[[CO:.*]] = arith.constant 0 : index
 // CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
 // CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
@@ -1996,21 +2011,35 @@ func.func @compress_store_op(%arg0: memref<?xf32>, %arg1: vector<11xi1>, %arg2:
 
 // -----
 
-func.func @compress_store_op_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
+func.func @compressstore_scalable(%arg0: memref<?xf32>, %arg1: vector<[11]xi1>, %arg2: vector<[11]xf32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xf32>, vector<[11]xi1>, vector<[11]xf32>
+  return
+}
+
+// CHECK-LABEL: func @compressstore_scalable
+// CHECK: %[[CO:.*]] = arith.constant 0 : index
+// CHECK: %[[C:.*]] = builtin.unrealized_conversion_cast %[[CO]] : index to i64
+// CHECK: %[[P:.*]] = llvm.getelementptr %{{.*}}[%[[C]]] : (!llvm.ptr, i64) -> !llvm.ptr, f32
+// CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %[[P]], %{{.*}}) : (vector<[11]xf32>, !llvm.ptr, vector<[11]xi1>) -> ()
+
+// -----
+
+func.func @compressstore_index(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>) {
   %c0 = arith.constant 0: index
   vector.compressstore %arg0[%c0], %arg1, %arg2 : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_index
+// CHECK-LABEL: func @compressstore_index
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----
 
-func.func @compress_store_op_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
+func.func @compressstore_with_alignment(%arg0: memref<?xindex>, %arg1: vector<11xi1>, %arg2: vector<11xindex>, %c0: index) {
   vector.compressstore %arg0[%c0], %arg1, %arg2 { alignment = 8 } : memref<?xindex>, vector<11xi1>, vector<11xindex>
   return
 }
-// CHECK-LABEL: func @compress_store_op_with_alignment
+// CHECK-LABEL: func @compressstore_with_alignment
 // CHECK: "llvm.intr.masked.compressstore"(%{{.*}}, %{{.*}}, %{{.*}}) <{arg_attrs = [{}, {llvm.align = 8 : i64}, {}]}> : (vector<11xi64>, !llvm.ptr, vector<11xi1>) -> ()
 
 // -----

diff  --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index aaa55cface958..0097db39f6ed9 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1688,14 +1688,6 @@ func.func @expand_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>,
 
 // -----
 
-func.func @expand_base_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.expandload' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
-}
-
-// -----
-
 func.func @expand_dim_mask_mismatch(%base: memref<?xf32>, %mask: vector<17xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.expandload' op expected result shape to match mask shape}}
@@ -1742,18 +1734,17 @@ func.func @expand_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
-func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
-  %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
-  vector.compressstore %base[%c0], %mask, %value : memref<?xf64>, vector<16xi1>, vector<16xf32>
+func.func @expand_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16xi1>, %pass_thru: vector<[16]xf32>, %c0: index) {
+  // expected-error at +1 {{expected result scalable dims to match mask scalable dims}}
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<16xi1>, vector<[16]xf32> into vector<[16]xf32>
 }
 
 // -----
 
-func.func @compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %value: vector<[16]xf32>) {
+func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
-  // expected-error at +1 {{'vector.compressstore' op operand #2 must be fixed-length vector of 1-bit signless integer values, but got 'vector<[16]xi1>}}
-  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
+  vector.compressstore %base[%c0], %mask, %value : memref<?xf64>, vector<16xi1>, vector<16xf32>
 }
 
 // -----
@@ -1796,6 +1787,14 @@ func.func @compress_non_power_of_2_alignment(%base: memref<?xf32>, %mask: vector
 
 // -----
 
+func.func @compress_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16xi1>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error at +1 {{expected valueToStore scalable dims to match mask scalable dims}}
+  vector.compressstore %base[%c0], %mask, %value : memref<?xf32>, vector<16xi1>, vector<[16]xf32>
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :

diff  --git a/mlir/test/Dialect/Vector/ops.mlir b/mlir/test/Dialect/Vector/ops.mlir
index e84bd3f1dce17..1b4109a3368ea 100644
--- a/mlir/test/Dialect/Vector/ops.mlir
+++ b/mlir/test/Dialect/Vector/ops.mlir
@@ -885,6 +885,16 @@ func.func @expand_and_compress(%base: memref<?xf32>, %mask: vector<16xi1>, %pass
   return
 }
 
+// CHECK-LABEL: @expand_and_compress_scalable
+func.func @expand_and_compress_scalable(%base: memref<?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}], %{{.*}}, %{{.*}} : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0], %mask, %pass_thru : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}], %{{.*}}, %[[X]] : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0], %mask, %0 : memref<?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @expand_and_compress2d
 func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -895,6 +905,16 @@ func.func @expand_and_compress2d(%base: memref<?x?xf32>, %mask: vector<16xi1>, %
   return
 }
 
+// CHECK-LABEL: @expand_and_compress2d_scalable
+func.func @expand_and_compress2d_scalable(%base: memref<?x?xf32>, %mask: vector<[16]xi1>, %pass_thru: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  // CHECK: %[[X:.*]] = vector.expandload %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %{{.*}} : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  %0 = vector.expandload %base[%c0, %c0], %mask, %pass_thru : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  // CHECK: vector.compressstore %{{.*}}[%{{.*}}, %{{.*}}], %{{.*}}, %[[X]] : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  vector.compressstore %base[%c0, %c0], %mask, %0 : memref<?x?xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL: @multi_reduction
 func.func @multi_reduction(%0: vector<4x8x16x32xf32>, %acc0: vector<4x16xf32>,
                            %acc1: f32) -> f32 {

diff  --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index 41cd2846fd752..ab921b32c05df 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -732,6 +732,19 @@ func.func @cast_away_expandload_leading_one_dims(%base: memref<16xf32>, %i: inde
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_expandload_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[P:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         %[[L:.+]] = vector.expandload %{{.*}}[%{{.*}}], %[[M]], %[[P]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32> into vector<[4]xf32>
+// CHECK:         %[[B:.+]] = vector.broadcast %[[L]] : vector<[4]xf32> to vector<1x[4]xf32>
+// CHECK:         return %[[B]] : vector<1x[4]xf32>
+func.func @cast_away_expandload_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %pass: vector<1x[4]xf32>) -> vector<1x[4]xf32> {
+  %0 = vector.expandload %base[%i], %mask, %pass : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32> into vector<1x[4]xf32>
+  return %0 : vector<1x[4]xf32>
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_gather_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>
@@ -778,6 +791,17 @@ func.func @cast_away_compressstore_leading_one_dims(%base: memref<16xf32>, %i: i
 
 // -----
 
+// CHECK-LABEL: func.func @cast_away_compressstore_leading_one_dims_scalable
+// CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<[4]xi1> from vector<1x[4]xi1>
+// CHECK:         %[[V:.+]] = vector.extract %{{.*}}[0] : vector<[4]xf32> from vector<1x[4]xf32>
+// CHECK:         vector.compressstore %{{.*}}[%{{.*}}], %[[M]], %[[V]] : memref<16xf32>, vector<[4]xi1>, vector<[4]xf32>
+func.func @cast_away_compressstore_leading_one_dims_scalable(%base: memref<16xf32>, %i: index, %mask: vector<1x[4]xi1>, %val: vector<1x[4]xf32>) {
+  vector.compressstore %base[%i], %mask, %val : memref<16xf32>, vector<1x[4]xi1>, vector<1x[4]xf32>
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @cast_away_scatter_leading_one_dims
 // CHECK:         %[[I:.+]] = vector.extract %{{.*}}[0] : vector<4xi32> from vector<1x4xi32>
 // CHECK:         %[[M:.+]] = vector.extract %{{.*}}[0] : vector<4xi1> from vector<1x4xi1>

diff  --git a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
index 2004a47851e2e..cd7ea84cdb2eb 100644
--- a/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-mem-transforms.mlir
@@ -175,6 +175,20 @@ func.func @fold_expandload_all_true(%base: memref<16xf32>, %pass_thru: vector<16
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_true_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-DAG:       %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      %[[T:.*]] = vector.load %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return %[[T]] : vector<[16]xf32>
+func.func @fold_expandload_all_true_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 // CHECK-LABEL:   func @fold_expandload_all_false(
 // CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<16xf32>) -> vector<16xf32> {
@@ -187,6 +201,18 @@ func.func @fold_expandload_all_false(%base: memref<16xf32>, %pass_thru: vector<1
   return %ld : vector<16xf32>
 }
 
+// CHECK-LABEL:   func @fold_expandload_all_false_scalable(
+// CHECK-SAME:                  %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                  %[[PASS_THRU:.*]]: vector<[16]xf32>) -> vector<[16]xf32> {
+// CHECK-NEXT:      return %[[PASS_THRU]] : vector<[16]xf32>
+func.func @fold_expandload_all_false_scalable(%base: memref<16xf32>, %pass_thru: vector<[16]xf32>) -> vector<[16]xf32> {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  %ld = vector.expandload %base[%c0], %mask, %pass_thru
+    : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32> into vector<[16]xf32>
+  return %ld : vector<[16]xf32>
+}
+
 //-----------------------------------------------------------------------------
 // [Pattern: CompressStoreFolder]
 //-----------------------------------------------------------------------------
@@ -204,6 +230,19 @@ func.func @fold_compressstore_all_true(%base: memref<16xf32>, %value: vector<16x
   return
 }
 
+// CHECK-LABEL:   func @fold_compressstore_all_true_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      %[[C:.*]] = arith.constant 0 : index
+// CHECK-NEXT:      vector.store %[[VALUE]], %[[BASE]][%[[C]]] : memref<16xf32>, vector<[16]xf32>
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_true_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [16] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value  : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}
+
 // CHECK-LABEL:   func @fold_compressstore_all_false(
 // CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
 // CHECK-SAME:                    %[[VALUE:.*]]: vector<16xf32>) {
@@ -214,3 +253,14 @@ func.func @fold_compressstore_all_false(%base: memref<16xf32>, %value: vector<16
   vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<16xi1>, vector<16xf32>
   return
 }
+
+// CHECK-LABEL:   func @fold_compressstore_all_false_scalable(
+// CHECK-SAME:                    %[[BASE:.*]]: memref<16xf32>,
+// CHECK-SAME:                    %[[VALUE:.*]]: vector<[16]xf32>) {
+// CHECK-NEXT:      return
+func.func @fold_compressstore_all_false_scalable(%base: memref<16xf32>, %value: vector<[16]xf32>) {
+  %c0 = arith.constant 0 : index
+  %mask = vector.constant_mask [0] : vector<[16]xi1>
+  vector.compressstore %base[%c0], %mask, %value : memref<16xf32>, vector<[16]xi1>, vector<[16]xf32>
+  return
+}

diff  --git a/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
new file mode 100644
index 0000000000000..eddc7c664d3f1
--- /dev/null
+++ b/mlir/test/Integration/Dialect/Vector/CPU/ArmSVE/compress.mlir
@@ -0,0 +1,214 @@
+// REQUIRES: arm-emulator
+
+/// End-to-end test for vector.compressstore for SVE
+
+// In order to demonstrate the impact of using scalable vectors, vscale is set
+// to 2 so that vector<[16]xi32> contains 32 rather than 16 elements at
+// run-time
+//
+// Note that you can also tweak the size of vscale by passing this flag to
+// QEMU:
+//  * -cpu max,sve-max-vq=[1-16]
+// (select the value between 1 and 16).
+
+// DEFINE: %{compile} =  mlir-opt %s -test-lower-to-llvm -o %t
+// DEFINE: %{run} = %mcr_aarch64_cmd %t -e main -entry-point-result=void --march=aarch64 --mattr="+sve"\
+// DEFINE:    -shared-libs=%mlir_runner_utils,%mlir_c_runner_utils,%native_mlir_arm_runner_utils
+
+// RUN: rm -f %t && %{compile} &&  %{run} |  FileCheck %s
+
+//===----------------------------------------------------------------------===//
+// @compress_16
+//
+// The number of inserted elements is 16 x vscale. Insertion index is
+// hard-coded to 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16(%base: memref<?xi32>,
+                 %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c0 = arith.constant 0: index
+  vector.compressstore %base[%c0], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @compress_16_at_8
+//
+// Same as @compress_16, but the insertion index is hard-coded to 8 instead of 0
+//===----------------------------------------------------------------------===//
+func.func @compress_16_at_8(%base: memref<?xi32>,
+                     %mask: vector<[16]xi1>, %value: vector<[16]xi32>) {
+  %c8 = arith.constant 8: index
+  vector.compressstore %base[%c8], %mask, %value
+    : memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @print1DMemRef
+//
+// TODO: Move to an utility file
+//===----------------------------------------------------------------------===//
+func.func @print1DMemRef(%ptr: memref<?xi32>) -> () {
+  %cast = memref.cast %ptr:  memref<?xi32> to memref<*xi32>
+
+  call @printMemrefI32(%cast): (memref<*xi32>) -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @reset_mem_i32
+//
+// Resets the input memory to 0.
+
+// TODO: Create a run-time utility funcion.
+//===----------------------------------------------------------------------===//
+func.func @reset_mem_i32(%ptr: memref<?xi32>, %size: index) {
+  %c0_idx = arith.constant 0: index
+  %c0 = arith.constant 0: i32
+  %step = arith.constant 1: index
+
+  scf.for %i = %c0_idx to %size step %step {
+    memref.store %c0, %ptr[%i] : memref<?xi32>
+  }
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @main
+//
+// The main entry point - sets the value of vscale.
+//===----------------------------------------------------------------------===//
+func.func @main() {
+  // Set vscale to 2 (vector width = 256). This will have identical effect to:
+  //  * qemu-aarch64 -cpu max,sve-max-vq=2 (...)
+  %c256 = arith.constant 256 : i32
+  func.call @setArmVLBits(%c256) : (i32) -> ()
+
+  // Run the tests.
+  func.call @test() : () -> ()
+
+  return
+}
+
+//===----------------------------------------------------------------------===//
+// @test
+//
+// Set-up and run tests.
+//===----------------------------------------------------------------------===//
+func.func @test() {
+  //
+  // Shared constants.
+  //
+  %vs = vector.vscale
+
+  //
+  // Set up memory.
+  //
+  %c16 = arith.constant 16: index
+  %vs_16 = arith.muli %vs, %c16 : index
+  %A = memref.alloc(%vs_16) : memref<?xi32>
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+
+  //
+  // Set the input vector.
+  //
+  %value = vector.step : vector<[16]xi32>
+  vector.print %value : vector<[16]xi32>
+
+  //
+  // Set up masks.
+  //
+
+  %f = arith.constant 0: i1
+  %t = arith.constant 1: i1
+
+  // %none = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %none = vector.constant_mask [0] : vector<[16]xi1>
+
+  // %all = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ..., 1]
+  %all = vector.constant_mask [16] : vector<[16]xi1>
+  vector.print %all : vector<[16]xi1>
+
+  // %first_vscale_4 = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, ..., 0]
+  %c4 = arith.constant 4 : index
+  %vs_4 = arith.muli %vs, %c4 : index
+  %first_vscale_4 = vector.create_mask %vs_4 : vector<[16]xi1>
+  vector.print %first_vscale_4 : vector<[16]xi1>
+
+  // %odd = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, ..., 0]
+  %0 = vector.insert %t, %none[1] : i1 into vector<[16]xi1>
+  %1 = vector.insert %t, %0[3] : i1 into vector<[16]xi1>
+  %2 = vector.insert %t, %1[5] : i1 into vector<[16]xi1>
+  %3 = vector.insert %t, %2[7] : i1 into vector<[16]xi1>
+  %4 = vector.insert %t, %3[9] : i1 into vector<[16]xi1>
+  %5 = vector.insert %t, %4[11] : i1 into vector<[16]xi1>
+  %odd = vector.insert %t, %5[13] : i1 into vector<[16]xi1>
+
+  // %even = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, ..., 0]
+  %6 = vector.insert %t, %none[0] : i1 into vector<[16]xi1>
+  %7 = vector.insert %t, %6[2] : i1 into vector<[16]xi1>
+  %8 = vector.insert %t, %7[4] : i1 into vector<[16]xi1>
+  %9 = vector.insert %t, %8[6] : i1 into vector<[16]xi1>
+  %10 = vector.insert %t, %9[8] : i1 into vector<[16]xi1>
+  %11 = vector.insert %t, %10[10] : i1 into vector<[16]xi1>
+  %even = vector.insert %t, %11[12] : i1 into vector<[16]xi1>
+
+
+  //
+  // Tests.
+  //
+
+  call @compress_16(%A, %none, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 0, 0, 0, 0, 0, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @compress_16(%A, %all, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 
+  /// (...)
+  // CHECK-SAME: 30, 31]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16_at_8(%A, %first_vscale_4, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 0, 0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %odd, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [1,  3,  5,  7,  9,  11,  13,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  call @reset_mem_i32(%A, %vs_16) : (memref<?xi32>, index) -> ()
+  call @compress_16(%A, %even, %value)
+    : (memref<?xi32>, vector<[16]xi1>, vector<[16]xi32>) -> ()
+  call @print1DMemRef(%A) : (memref<?xi32>) -> ()
+  // CHECK: [0,  2,  4,  6,  8,  10,  12,  0,  0,  0,  0,
+  /// (...)
+  // CHECK-SAME: 0, 0]
+
+  memref.dealloc %A : memref<?xi32>
+  return
+}
+
+func.func private @printMemrefI32(%ptr : memref<*xi32>)
+func.func private @setArmVLBits(%bits : i32)


        

From llvmlistbot at llvm.org  Thu Jul 23 07:40:34 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Thu, 23 Jul 2026 07:40:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Make CompressstoreOp +
 ExpandloadOp support scalable vectors (PR #210288)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210288@github.com>
Message-ID: <6a6227e2.91b0c015.208015.c884@mx.google.com>


https://github.com/banach-space closed https://github.com/llvm/llvm-project/pull/210288

From llvmlistbot at llvm.org  Thu Jul 23 07:45:00 2026
From: llvmlistbot at llvm.org (Byeongjee Kang)
Date: Thu, 23 Jul 2026 07:45:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a6228ec.3a12a5a9.26cf77.797a@mx.google.com>


https://github.com/byeongjee updated https://github.com/llvm/llvm-project/pull/196613

>From 4eb1075004eafe159cdb9b436cb5a4fb9a2a33b4 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 00:20:59 -0400
Subject: [PATCH 1/2] [MLIR][WasmSSA] Add global_set op

Adds the `wasmssa.global_set` operation, which writes a value to a
mutable global variable. The op verifies that the referenced symbol is
a `wasmssa.global` or `wasmssa.import_global` marked mutable, and that
the operand type matches the global's declared type.
---
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 19 +++++++
 mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp    | 38 +++++++++++++
 .../Dialect/WasmSSA/custom_parser/global.mlir | 12 ++++
 mlir/test/Dialect/WasmSSA/global-invalid.mlir | 55 +++++++++++++++++++
 4 files changed, 124 insertions(+)

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index bfa0953f50ac8..e2fb642433266 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -381,6 +381,25 @@ def WasmSSA_GlobalGetOp : WasmSSA_Op<"global_get", [DeclareOpInterfaceMethods<Sy
   let assemblyFormat = "$global attr-dict `:` type($global_val)";
 }
 
+def WasmSSA_GlobalSetOp : WasmSSA_Op<"global_set", [
+    DeclareOpInterfaceMethods<SymbolUserOpInterface>]> {
+  let summary = "Sets the value of the mutable global passed as argument.";
+  let description = [{
+     Sets the value of the mutable global referenced by the symbol to the given
+     value.
+
+     Example:
+
+     ```mlir
+     // Sets the value of `@global_0` to `%v`.
+     wasmssa.global_set @global_0 to %v : i32
+     ```
+    }];
+  let arguments = (ins FlatSymbolRefAttr: $global,
+                       WasmSSA_ValType: $value);
+  let assemblyFormat = "$global `to` $value `:` type($value) attr-dict";
+}
+
 def WasmSSA_IfOp : WasmSSA_Op<"if", [Terminator,
     DeclareOpInterfaceMethods<LabelLevelOpInterface>]> {
   let summary = "Execute the if region if condition value is non-zero, the else region otherwise.";
diff --git a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
index 08de887a56a97..eab9c7a9d53a7 100644
--- a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
+++ b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
@@ -331,6 +331,44 @@ GlobalGetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// GlobalSetOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+GlobalSetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+  Operation *symTabOp = SymbolTable::getNearestSymbolTable(*this);
+  StringRef referencedSymbol = getGlobal();
+  Operation *definitionOp = symbolTable.lookupSymbolIn(
+      symTabOp, StringAttr::get(this->getContext(), referencedSymbol));
+  if (!definitionOp)
+    return emitError() << "symbol @" << referencedSymbol << " is undefined";
+
+  Type globalType;
+  bool isMutable = false;
+  if (auto global = dyn_cast<GlobalOp>(definitionOp)) {
+    globalType = global.getType();
+    isMutable = global.getIsMutable();
+  } else if (auto globalImport = dyn_cast<GlobalImportOp>(definitionOp)) {
+    globalType = globalImport.getType();
+    isMutable = globalImport.getIsMutable();
+  } else {
+    return emitError() << "symbol @" << referencedSymbol
+                       << " is not a global symbol";
+  }
+
+  if (!isMutable)
+    return emitError("global.set target must be mutable");
+
+  Type valueType = getValue().getType();
+  if (globalType != valueType)
+    return emitError("global.set value type does not match target global "
+                     "type: expected ")
+           << globalType << " but got " << valueType;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // GlobalImportOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
index a25abbd2a8662..6441731a0a335 100644
--- a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
@@ -19,6 +19,12 @@ module {
     %0 = wasmssa.global_get @global_0 : i32
     wasmssa.return %0 : i32
   }
+
+  wasmssa.func @set_global() {
+    %0 = wasmssa.const 7 : i32
+    wasmssa.global_set @global_2 to %0 : i32
+    wasmssa.return
+  }
 }
 
 // CHECK-LABEL:   wasmssa.import_global "from_js" from "env" as @global_0 : i32
@@ -42,3 +48,9 @@ module {
 // CHECK:           %[[VAL_0:.*]] = wasmssa.global_get @global_0 : i32
 // CHECK:           wasmssa.return %[[VAL_0]] : i32
 // CHECK:         }
+
+// CHECK-LABEL:   wasmssa.func @set_global() {
+// CHECK:           %[[VAL_0:.*]] = wasmssa.const 7 : i32
+// CHECK:           wasmssa.global_set @global_2 to %[[VAL_0]] : i32
+// CHECK:           wasmssa.return
+// CHECK:         }
diff --git a/mlir/test/Dialect/WasmSSA/global-invalid.mlir b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
index c5bc606fd13f3..600b2f0715fdc 100644
--- a/mlir/test/Dialect/WasmSSA/global-invalid.mlir
+++ b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
@@ -40,3 +40,58 @@ module {
     wasmssa.return %0 : i32
   }
 }
+
+// -----
+
+module {
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @missing is undefined}}
+    wasmssa.global_set @missing to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i32 : {
+    %0 = wasmssa.const 0 : i32
+    wasmssa.return %0 : i32
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set target must be mutable}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i64 mutable : {
+    %0 = wasmssa.const 0 : i64
+    wasmssa.return %0 : i64
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set value type does not match target global type: expected 'i64' but got 'i32'}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.func @not_global() {
+    wasmssa.return
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @not_global is not a global symbol}}
+    wasmssa.global_set @not_global to %0 : i32
+    wasmssa.return
+  }
+}

>From c9c103386c84a4756f84bbf3a60cbfb07076b541 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 15:05:35 -0400
Subject: [PATCH 2/2] [MLIR][WasmSSA] Add select op

Adds the `wasmssa.select` operation, a Wasm value-typed ternary that
returns one of two operands based on an i32 condition. The op uses the
`AllTypesMatch` trait to require both branches and the result share the
same Wasm value type, and is marked `Pure` since it has no side effects.

Also adds the `SideEffectInterfaces` td/header includes needed by the
`Pure` trait.
---
 .../include/mlir/Dialect/WasmSSA/IR/WasmSSA.h |  1 +
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 24 ++++++++++++
 .../Dialect/WasmSSA/custom_parser/select.mlir | 37 +++++++++++++++++++
 3 files changed, 62 insertions(+)
 create mode 100644 mlir/test/Dialect/WasmSSA/custom_parser/select.mlir

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
index 64391d807c633..64d480c065b79 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
@@ -38,6 +38,7 @@
 #include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
 
 //===----------------------------------------------------------------------===//
 // WebAssembly Constraints
diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index e2fb642433266..3b3c3e8825fef 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -16,6 +16,7 @@ include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.td"
 
 include "mlir/Interfaces/FunctionInterfaces.td"
 include "mlir/Interfaces/InferTypeOpInterface.td"
+include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/IR/BuiltinAttributeInterfaces.td"
 include "mlir/IR/SymbolInterfaces.td"
 
@@ -628,6 +629,29 @@ def WasmSSA_ReturnOp : WasmSSA_Op<"return", [Terminator]> {
   ];
 }
 
+def WasmSSA_SelectOp : WasmSSA_Op<"select", [
+    AllTypesMatch<["true_value", "false_value", "result"]>,
+    Pure]> {
+  let summary = "Select between two values based on a condition.";
+  let description = [{
+     Returns `true_value` if `condition` is non-zero, `false_value` otherwise.
+     Both operands and the result must share the same Wasm value type.
+
+     Example:
+
+     ```mlir
+     // Returns %a if %cond is non-zero, otherwise %b.
+     %r = wasmssa.select %cond, %a, %b : i32
+     ```
+    }];
+  let arguments = (ins WasmSSA_ValType: $true_value,
+                       WasmSSA_ValType: $false_value,
+                       I32: $condition);
+  let results = (outs WasmSSA_ValType: $result);
+  let assemblyFormat =
+      "$condition `,` $true_value `,` $false_value attr-dict `:` type($result)";
+}
+
 // ---- Numeric ops
 
 class WasmSSA_BinaryNumericalOp<string mnemonic, string summaryStr, string descStr,
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
new file mode 100644
index 0000000000000..80ced11c01bc3
--- /dev/null
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
@@ -0,0 +1,37 @@
+// RUN: mlir-opt %s | FileCheck %s
+
+module {
+  wasmssa.func @select_i32(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to i32>,
+                           %b: !wasmssa<local ref to i32>) -> i32 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to i32
+    %2 = wasmssa.local_get %b : ref to i32
+    %r = wasmssa.select %0, %1, %2 : i32
+    wasmssa.return %r : i32
+  }
+
+  wasmssa.func @select_f64(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to f64>,
+                           %b: !wasmssa<local ref to f64>) -> f64 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to f64
+    %2 = wasmssa.local_get %b : ref to f64
+    %r = wasmssa.select %0, %1, %2 : f64
+    wasmssa.return %r : f64
+  }
+}
+
+// CHECK-LABEL:   wasmssa.func @select_i32(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : i32
+// CHECK:           wasmssa.return %[[R]] : i32
+
+// CHECK-LABEL:   wasmssa.func @select_f64(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : f64
+// CHECK:           wasmssa.return %[[R]] : f64


From llvmlistbot at llvm.org  Thu Jul 23 07:49:46 2026
From: llvmlistbot at llvm.org (Emil Pedersen)
Date: Thu, 23 Jul 2026 07:49:46 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a622a0a.3cb0362c.249480.babf@mx.google.com>


Snowy1803 wrote:

I think as long as the conflict between DIArgList/OP_LLVM_arg and the branching feature causes a verifier failure, this feature is fragile.
What happens if a frontend creates a DIExpression with branching and without an arglist, and then salvageDebugInfo, through an optimization, rewrites the argument to use an arglist when a binary instruction is deleted? Would it just crash?
I think that if an expression can't be lowered due to an implementation detail, the debug value should just be killed. Hopefully support for DIArgList can be added later.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Thu Jul 23 07:51:20 2026
From: llvmlistbot at llvm.org (Spencer Bryngelson)
Date: Thu, 23 Jul 2026 07:51:20 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211566@github.com>
Message-ID: <6a622a68.bd93e731.2ac69c.e582@mx.google.com>


sbryngelson wrote:

Verified this on `d1d3891077f6`. It covers the barrier case that my narrower PR missed, and it applies cleanly.

| suite | result |
|---|---|
| check-flang | 4608 passed, 11 expected failures, 0 failures |
| clang/test/OpenMP | 1573 passed |
| mlir/test/Target/LLVMIR | 407 passed |
| LLVMFrontendTests | 1281 passed |

One thing worth knowing: the end-to-end symptom no longer reproduces at tip. Unpatched, the reproducer from #211385 now builds clean at `-O2 -g` and `-O3 -g`. The bad IR is still there, though; `_omp_reduction_shuffle_and_reduce_func` and `_omp_reduction_inter_warp_copy_func` carry 61 and 48 `!dbg` scoped to the kernel's subprogram. So the failure is latent rather than fixed, and a test at the IR level like yours is the right guard.

I have a flang integration test that checks the same thing through the Fortran driver (`flang/test/Integration/OpenMP/target-reduction-debug-loc.f90`). Your MLIR test covers the bug, so take it only if you want driver-level coverage as well; otherwise ignore this.

I will close #211395 in favor of this.

https://github.com/llvm/llvm-project/pull/211566

From llvmlistbot at llvm.org  Thu Jul 23 07:57:59 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Thu, 23 Jul 2026 07:57:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a622bf7.666420a4.3304c7.cadf@mx.google.com>


https://github.com/bruteforceboy edited https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 08:00:17 2026
From: llvmlistbot at llvm.org (Abid Qadeer)
Date: Thu, 23 Jul 2026 08:00:17 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211566@github.com>
Message-ID: <6a622c81.7270d2fd.1ec500.3bb2@mx.google.com>


abidh wrote:

> Verified this on `d1d3891077f6`. It covers the barrier case that my narrower PR missed, and it applies cleanly.
> 
> suite	result
> check-flang	4608 passed, 11 expected failures, 0 failures
> clang/test/OpenMP	1573 passed
> mlir/test/Target/LLVMIR	407 passed
> LLVMFrontendTests	1281 passed
> One thing worth knowing: the end-to-end symptom no longer reproduces at tip. Unpatched, the reproducer from #211385 now builds clean at `-O2 -g` and `-O3 -g`. The bad IR is still there, though; `_omp_reduction_shuffle_and_reduce_func` and `_omp_reduction_inter_warp_copy_func` carry 61 and 48 `!dbg` scoped to the kernel's subprogram. So the failure is latent rather than fixed, and a test at the IR level like yours is the right guard.
> 
> I have a flang integration test that checks the same thing through the Fortran driver (`flang/test/Integration/OpenMP/target-reduction-debug-loc.f90`). Your MLIR test covers the bug, so take it only if you want driver-level coverage as well; otherwise ignore this.
> 
> I will close #211395 in favor of this.

Thanks for review and testing. We add tests in  flang integration directory only if there is no other good way to test the change. You will see a header in many of the test files there

```
! This directory can be used to add Integration tests involving multiple
! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
! contain executable tests. We should only add tests here sparingly and only
! if there is no other way to test. Repeat this message in each test that is
! added to this directory and sub-directories.

```

https://github.com/llvm/llvm-project/pull/211566

From llvmlistbot at llvm.org  Thu Jul 23 08:00:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:00:31 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211566@github.com>
Message-ID: <6a622c8f.7026a287.a45b0.4324@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Abid Qadeer (abidh)

<details>
<summary>Changes</summary>

This fixes https://github.com/llvm/llvm-project/issues/211385. This was initially landed in https://github.com/llvm/llvm-project/pull/148284.

We have this pattern of code in OMPIRBuilder for many functions that are used in reduction operations.

 ```
 Function *LtGRFunc = Function::Create
  BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
  Builder.SetInsertPoint(EntryBlock);
```

The insertion point is moved to the new function but the debug location
is not updated. This means that reduction function will use the debug
location that points to another function. This problem gets hidden
because these functions gets inlined but the potential for failure
exists.

This patch resets the debug location when insertion point is moved to new
function. Some `InsertPointGuard` have been added to make sure we restore
the debug location correctly when we are done with the reduction function.

It also clears the debug location for the helper-internal barriers in
emitInterWarpCopyFunction.

---
Full diff: https://github.com/llvm/llvm-project/pull/211566.diff


2 Files Affected:

- (modified) llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp (+18-12) 
- (added) mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir (+71) 


``````````diff
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index fb1cebf28b95b..57e199bdff09a 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -3526,7 +3526,7 @@ Error OpenMPIRBuilder::emitReductionListCopy(
 Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
     const LocationDescription &Loc, ArrayRef<ReductionInfo> ReductionInfos,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  InsertPointTy SavedIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(), {Builder.getPtrTy(), Builder.getInt32Ty()},
@@ -3540,6 +3540,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   WcFunc->addParamAttr(1, Attribute::NoUndef);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", WcFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // ReduceList: thread local Reduce list.
   // At the stage of the computation when this function is called, partially
@@ -3646,7 +3647,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
 
       // kmpc_barrier.
       InsertPointOrErrorTy BarrierIP1 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3705,7 +3706,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
       // endif
       emitBlock(MergeBB, Builder.GetInsertBlock()->getParent());
       InsertPointOrErrorTy BarrierIP2 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3778,7 +3779,6 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(SavedIP);
 
   return WcFunc;
 }
@@ -3787,6 +3787,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
   LLVMContext &Ctx = M.getContext();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   FunctionType *FuncTy =
       FunctionType::get(Builder.getVoidTy(),
                         {Builder.getPtrTy(), Builder.getInt16Ty(),
@@ -3806,6 +3807,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
   SarFunc->addParamAttr(3, Attribute::SExt);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", SarFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Thread local Reduce list used to host the values of data to be reduced.
   Argument *ReduceListArg = SarFunc->getArg(0);
@@ -4018,7 +4020,7 @@ Expected<Value *> OpenMPIRBuilder::createReductionDescriptorCopy(
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4034,6 +4036,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGCFunc->getArg(0);
@@ -4137,14 +4140,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4160,6 +4162,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGRFunc->getArg(0);
@@ -4245,14 +4248,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {LocalReduceListAddrCast, ReduceList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGRFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4268,6 +4270,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLCFunc->getArg(0);
@@ -4368,14 +4371,13 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   auto *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4391,6 +4393,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLRFunc->getArg(0);
@@ -4476,7 +4479,6 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {ReduceList, ReductionList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLRFunc;
 }
 
@@ -4490,6 +4492,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
     StringRef ReducerName, ArrayRef<ReductionInfo> ReductionInfos,
     ArrayRef<bool> IsByRef, ReductionGenCBKind ReductionGenCBKind,
     AttributeList FuncAttrs) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   auto *FuncTy = FunctionType::get(Builder.getVoidTy(),
                                    {Builder.getPtrTy(), Builder.getPtrTy()},
                                    /* IsVarArg */ false);
@@ -4503,6 +4506,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
   BasicBlock *EntryBB =
       BasicBlock::Create(M.getContext(), "entry", ReductionFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Need to alloca memory here and deal with the pointers before getting
   // LHS/RHS pointers out
@@ -4986,10 +4990,12 @@ static Error populateReductionFunction(
     Function *ReductionFunc,
     ArrayRef<OpenMPIRBuilder::ReductionInfo> ReductionInfos,
     IRBuilder<> &Builder, ArrayRef<bool> IsByRef, bool IsGPU) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   Module *Module = ReductionFunc->getParent();
   BasicBlock *ReductionFuncBlock =
       BasicBlock::Create(Module->getContext(), "", ReductionFunc);
   Builder.SetInsertPoint(ReductionFuncBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
   Value *LHSArrayPtr = nullptr;
   Value *RHSArrayPtr = nullptr;
   if (IsGPU) {
diff --git a/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
new file mode 100644
index 0000000000000..689446eee8de9
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
@@ -0,0 +1,71 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// The device reduction helpers emitted by OpenMPIRBuilder have no debug info of
+// their own. If the builder's current debug location is left set while their
+// bodies are emitted, their instructions get DILocations scoped to the
+// wrong subprogram. Check that none of the GPU reduction helpers carry a
+// !dbg attachment.
+
+#di_file = #llvm.di_file<"repro.f90" in "">
+#di_null_type = #llvm.di_null_type
+#loc1 = loc("repro.f90":1:1)
+#loc2 = loc("repro.f90":7:9)
+#di_compile_unit = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_Fortran95, file = #di_file, producer = "flang", isOptimized = true, emissionKind = Full>
+#di_subroutine_type = #llvm.di_subroutine_type<callingConvention = DW_CC_normal, types = #di_null_type>
+#di_subprogram = #llvm.di_subprogram<id = distinct[1]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "k", linkageName = "k_", file = #di_file, line = 1, scopeLine = 1, subprogramFlags = "Definition|Optimized", type = #di_subroutine_type>
+#di_subprogram1 = #llvm.di_subprogram<id = distinct[2]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "__omp_offloading_k_l7", linkageName = "__omp_offloading_k_l7", file = #di_file, line = 7, scopeLine = 7, subprogramFlags = "LocalToUnit|Definition|Optimized", type = #di_subroutine_type>
+#loc14 = loc(fused<#di_subprogram>[#loc1])
+#loc15 = loc(fused<#di_subprogram1>[#loc2])
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr = dense<64> : vector<4xi64>, !llvm.ptr<1> = dense<64> : vector<4xi64>, !llvm.ptr<5> = dense<32> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, "dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true, omp.version = #omp.version<version = 31>} {
+  omp.declare_reduction @add_reduction_f64 : f64 init {
+  ^bb0(%arg0: f64):
+    %0 = llvm.mlir.constant(0.000000e+00 : f64) : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } combiner {
+  ^bb0(%arg0: f64, %arg1: f64):
+    %0 = llvm.fadd %arg0, %arg1 : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } loc(#loc2)
+  llvm.func @k_(%arg0: !llvm.ptr) {
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, f64) map_clauses(from) capture(ByRef) -> !llvm.ptr {name = "s"} loc(#loc2)
+    omp.target kernel_type(spmd) map_entries(%1 -> %arg1 : !llvm.ptr) {
+      %c1 = llvm.mlir.constant(1 : i32) : i32 loc(#loc2)
+      %cn = llvm.mlir.constant(1024 : i32) : i32 loc(#loc2)
+      omp.teams reduction(@add_reduction_f64 %arg1 -> %arg2 : !llvm.ptr) {
+        omp.parallel {
+          omp.distribute {
+            omp.wsloop reduction(@add_reduction_f64 %arg2 -> %arg3 : !llvm.ptr) {
+              omp.loop_nest (%arg4) : i32 = (%c1) to (%cn) inclusive step (%c1) {
+                %2 = llvm.load %arg3 : !llvm.ptr -> f64 loc(#loc2)
+                %cst = llvm.mlir.constant(1.000000e+00 : f64) : f64 loc(#loc2)
+                %3 = llvm.fadd %2, %cst : f64 loc(#loc2)
+                llvm.store %3, %arg3 : f64, !llvm.ptr loc(#loc2)
+                omp.yield loc(#loc2)
+              } loc(#loc2)
+            } {omp.composite} loc(#loc2)
+          } {omp.composite} loc(#loc2)
+          omp.terminator loc(#loc2)
+        } {omp.composite} loc(#loc2)
+        omp.terminator loc(#loc2)
+      } {omp.combined} loc(#loc2)
+      omp.terminator loc(#loc2)
+    } {omp.combined} loc(#loc15)
+    llvm.return loc(#loc14)
+  } loc(#loc14)
+} loc(#loc1)
+
+// CHECK-LABEL: define internal void @_omp_reduction_shuffle_and_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_inter_warp_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_list_to_global_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }

``````````

</details>


https://github.com/llvm/llvm-project/pull/211566

From llvmlistbot at llvm.org  Thu Jul 23 08:00:37 2026
From: llvmlistbot at llvm.org (Byeongjee Kang)
Date: Thu, 23 Jul 2026 08:00:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a622c95.5da26cf0.353bc5.2e26@mx.google.com>


byeongjee wrote:

@flemairen6 No problem.

Do you know why global op is raised to `memref<1xT>` instead of `memref<T>`?

https://github.com/llvm/llvm-project/blob/8b690a085406337f7a02ab466df494bce5f75f41/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp#L626

I'll match the convention, but not sure what's the rationale behind it.

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Thu Jul 23 08:01:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:01:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add more informative error
 messages to block scaled types (PR #211569)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211569@github.com>
Message-ID: <6a622cd9.1728eaf3.5e665.692b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Luke Hutton (lhutton1)

<details>
<summary>Changes</summary>

This commit improves block scaled tensor type verification to provide a specific reason about why type verification failed. Previously the error message was a very generic "must be tosa-conformant tensor of number values".

This commit restores previous functionality that was reverted by https://github.com/llvm/llvm-project/pull/207995, but uses the ODS string interpolation mechanism instead to prevent incompatibility with PDLL.

---
Full diff: https://github.com/llvm/llvm-project/pull/211569.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h (+10-3) 
- (modified) mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td (+17-5) 
- (modified) mlir/lib/Dialect/Tosa/IR/TosaOps.cpp (+49-9) 
- (modified) mlir/test/Dialect/Tosa/verifier.mlir (+5-5) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
index b1404d9c700ae..2c4d3b7446544 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaOps.h
@@ -25,6 +25,8 @@
 #include "mlir/Interfaces/SideEffectInterfaces.h"
 #include "mlir/Interfaces/VectorInterfaces.h"
 
+#include <string>
+
 //===----------------------------------------------------------------------===//
 // TOSA dialect and structs includes.
 //===----------------------------------------------------------------------===//
@@ -141,9 +143,14 @@ Type getStorageElementTypeOrSelf(Type type);
 // Returns the storage element type for a given value
 Type getStorageElementTypeOrSelf(Value value);
 
-// Verify a block scaled tensor type is valid
-LogicalResult verifyBlockScaledTensorType(mlir::Type type,
-                                          bool allowScaleValues);
+// Verify that a given type is a valid block scaled tensor type
+LogicalResult verifyBlockScaledTensorType(
+    mlir::Type type,
+    llvm::function_ref<mlir::InFlightDiagnostic()> emitError = nullptr,
+    bool allowScaleValues = false);
+
+// Collect error messages for a given type
+std::string getTosaTensorTypeErrorMessage(mlir::Type type);
 
 } // namespace tosa
 } // namespace mlir
diff --git a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
index 2ab135befaa96..9252223cadb3c 100644
--- a/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
+++ b/mlir/include/mlir/Dialect/Tosa/IR/TosaTypesBase.td
@@ -172,7 +172,19 @@ def AtLeastRankOne : And<[
   CPred<"::llvm::cast<::mlir::RankedTensorType>($_self).getRank() >= 1">]>;
 
 def IsValidBlockScaledTensorType
-    : CPred<"::mlir::succeeded(::mlir::tosa::verifyBlockScaledTensorType($_self, false))">;
+    : CPred<"::mlir::succeeded(::mlir::tosa::"
+            "verifyBlockScaledTensorType($_self))">;
+
+// Analogous to ShapedContainerType, but with additional 'summary' information.
+// This is used to provide more informative error messages when a type predicate
+// is false.
+class TosaShapedContainerType<list<Type> allowedTypes, Pred containerPred, string descr, string cppType = "::mlir::Type"> :
+    Type<And<[containerPred,
+              Concat<"[](::mlir::Type elementType) { return ",
+                SubstLeaves<"$_self", "elementType",
+                AnyTypeOf<allowedTypes>.predicate>,
+                "; }(getElementTypeOrSelf($_self))">]>,
+         descr # " of " # AnyTypeOf<allowedTypes>.summary # " values" # "{{::mlir::tosa::getTosaTensorTypeErrorMessage($_self)}}", cppType>;
 
 // We include unranked tensors as a supported type for all possible tosa
 // Tensors as unranked does not guarantee invalid. If unranked tensors exist
@@ -181,13 +193,13 @@ def IsValidBlockScaledTensorType
 class TosaTensorOf<list<Type> allowedTypes,
                    list<Pred> extraRankedTensorPreds = [],
                    string summary = "tosa-conformant tensor">
-    : TensorOf<allowedTypes,
-               [Or<[
+    : TosaShapedContainerType<allowedTypes,
+               And<[Or<[
                     IsUnrankedTensorTypePred, 
                     And<!listconcat([IsRankedTensorTypePred, HasNo0Dimensions], extraRankedTensorPreds)>
                 ]>,
-                IsValidBlockScaledTensorType],
-               summary>;
+                IsValidBlockScaledTensorType]>,
+               summary, "::mlir::TensorType">;
 
 class TosaTensorRankOf<list<Type> allowedTypes, list<int> ranks>
     : TosaTensorOf<allowedTypes,
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
index 0dfcd23504019..7bfc3de98d08c 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaOps.cpp
@@ -741,22 +741,31 @@ LogicalResult mlir::tosa::mxint8Type::convertFromAttribute(
 // TOSA block scaling utilities.
 //===----------------------------------------------------------------------===//
 
-LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
-                                                      bool allowScaleValues) {
+LogicalResult mlir::tosa::verifyBlockScaledTensorType(
+    mlir::Type type, llvm::function_ref<mlir::InFlightDiagnostic()> emitError,
+    bool allowScaleValues) {
   const auto tensorType = llvm::cast<ShapedType>(type);
   const BlockScaledType elemType =
       llvm::dyn_cast<BlockScaledType>(tensorType.getElementType());
   if (!elemType)
     return success();
 
-  if (!allowScaleValues && elemType.hasScaleValues())
+  if (!allowScaleValues && elemType.hasScaleValues()) {
+    if (emitError)
+      emitError()
+          << "block scaled tensor type with scale values is not allowed";
     return failure();
+  }
 
   if (!tensorType.hasRank())
     return success();
 
-  if (tensorType.getRank() == 0)
+  if (tensorType.getRank() == 0) {
+    if (emitError)
+      emitError() << "block scaled tensor type must have rank greater than "
+                     "zero";
     return failure();
+  }
 
   const ArrayRef<int64_t> tensorShape = tensorType.getShape();
   const uint32_t blockSize =
@@ -765,19 +774,47 @@ LogicalResult mlir::tosa::verifyBlockScaledTensorType(mlir::Type type,
   if (allowScaleValues && elemType.hasScaleValues() &&
       tensorType.hasStaticShape()) {
     const size_t numBlocks = tensorType.getNumElements() / blockSize;
-    if (elemType.getScaleValues().size() != numBlocks)
+    if (elemType.getScaleValues().size() != numBlocks) {
+      if (emitError)
+        emitError() << "block scaled tensor type with scale values must have "
+                       "scale values for each block, expected "
+                    << numBlocks << ", got "
+                    << elemType.getScaleValues().size();
       return failure();
+    }
   }
 
   const int64_t blockedDimension = tensorShape.back();
   if (ShapedType::isDynamic(blockedDimension))
     return success();
-  if (blockedDimension % blockSize != 0)
+
+  if (blockedDimension % blockSize != 0) {
+    if (emitError)
+      emitError() << "last dimension of block scaled tensor type ("
+                  << blockedDimension << ") must be divisible by block size ("
+                  << blockSize << ")";
+
     return failure();
+  }
 
   return success();
 }
 
+std::string mlir::tosa::getTosaTensorTypeErrorMessage(mlir::Type type) {
+  MLIRContext *ctx = type.getContext();
+  std::string message;
+  ScopedDiagnosticHandler handler(
+      ctx, [&](Diagnostic &diag) { message = diag.str(); });
+
+  if (failed(verifyBlockScaledTensorType(
+          type, [ctx] { return emitError(UnknownLoc::get(ctx)); })) &&
+      !message.empty()) {
+    return ": " + message;
+  }
+
+  return "";
+}
+
 static ParseResult parseScaleValues(AsmParser &parser,
                                     SmallVector<Attribute> &scaleValues,
                                     Type scaleType) {
@@ -964,9 +1001,12 @@ LogicalResult tosa::ConstOp::verify() {
       return op.emitOpError(
           "attribute block scaled type must have scale values");
 
-    if (failed(verifyBlockScaledTensorType(attrType, true)))
-      return op.emitOpError("block scaled attribute type is not valid, got ")
-             << attrType;
+    const auto emitAttributeError = [&op]() {
+      return op.emitOpError("attribute block scaled type is invalid: ");
+    };
+
+    if (failed(verifyBlockScaledTensorType(attrType, emitAttributeError, true)))
+      return failure();
 
     const BlockScaledType resultBlockScaledType =
         llvm::dyn_cast<mlir::tosa::BlockScaledType>(resultElemType);
diff --git a/mlir/test/Dialect/Tosa/verifier.mlir b/mlir/test/Dialect/Tosa/verifier.mlir
index ba69e4aba632e..f23f822546bb4 100644
--- a/mlir/test/Dialect/Tosa/verifier.mlir
+++ b/mlir/test/Dialect/Tosa/verifier.mlir
@@ -1705,7 +1705,7 @@ func.func @test_cast_between_block_scaled(%arg0: tensor<4x32x!tosa.block_scaled<
 // -----
 
 func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: last dimension of block scaled tensor type (31) must be divisible by block size (32), but got 'tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<1x16x31x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<1x16x31xf32>
   return %0 : tensor<1x16x31xf32>
 }
@@ -1713,7 +1713,7 @@ func.func @test_block_scaled_cast_invalid_block_shape(%arg0: tensor<1x16x31x!tos
 // -----
 
 func.func @test_block_scaled_cast_scalar(%arg0: tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32> {
-  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
+  // expected-error at +1 {{'tosa.cast' op operand #0 must be tosa-conformant tensor of number values: block scaled tensor type must have rank greater than zero, but got 'tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>'}}
   %0 = tosa.cast %arg0 : (tensor<!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN>>) -> tensor<f32>
   return %0 : tensor<f32>
 }
@@ -2346,7 +2346,7 @@ func.func @test_block_scaled_const_scale_value_non_float_explicit_type() -> tens
 !mxint8_scale = !tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0, 4.0}>
 
 func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid: block scaled tensor type with scale values must have scale values for each block, expected 2, got 3}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!mxint8_scale> : 0 : i8>}> : () -> tensor<2x32x!mxint8>
   return %0 : tensor<2x32x!mxint8>
 }
@@ -2354,7 +2354,7 @@ func.func @test_block_scaled_const_invalid_num_scales() -> tensor<2x32x!mxint8>
 // -----
 
 func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>> {
-  // expected-error at +1 {{'tosa.const' op block scaled attribute type is not valid, got 'tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.000000e+00, 2.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op attribute block scaled type is invalid: block scaled tensor type with scale values must have scale values for each block, expected 4, got 2}}
   %0 = "tosa.const"() <{values = dense<tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8, {1.0, 2.0}>> : 0 : i8>}> : () -> tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
   return %0 : tensor<2x64x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:!tosa.mxint8>>
 }
@@ -2362,7 +2362,7 @@ func.func @test_block_scaled_const_invalid_num_scales_wide_inner_dim() -> tensor
 // -----
 
 func.func @test_block_scaled_const_cast_scale_values_propagate() -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> {
-  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
+  // expected-error at +1 {{'tosa.const' op result #0 must be tosa-conformant tensor of number values: block scaled tensor type with scale values is not allowed, but got 'tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.000000e+00, 4.000000e+00}>>'}}
   %0 = "tosa.const"() <{values = dense<tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>> : 0.0 : f8E4M3FN>}> : () -> tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
   return %0 : tensor<2x32x!tosa.block_scaled<BLOCK_SHAPE_32:f8E8M0FNU:f8E4M3FN, {2.0, 4.0}>>
 }

``````````

</details>


https://github.com/llvm/llvm-project/pull/211569

From llvmlistbot at llvm.org  Thu Jul 23 08:02:13 2026
From: llvmlistbot at llvm.org (Byeongjee Kang)
Date: Thu, 23 Jul 2026 08:02:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a622cf5.d681cab9.33af1e.fbd4@mx.google.com>


byeongjee wrote:

Global Get Op raising seems to be also missing, but I can add it in a separate PR.

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Thu Jul 23 08:09:39 2026
From: llvmlistbot at llvm.org (Chibuoyim Ogbonna)
Date: Thu, 23 Jul 2026 08:09:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a622eb3.1b0626d0.1ff4c3.70f9@mx.google.com>


https://github.com/bruteforceboy edited https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Thu Jul 23 08:12:06 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Thu, 23 Jul 2026 08:12:06 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
Message-ID: <llvm.org/llvm/llvm-project/pull/211577@github.com>


https://github.com/rajatbajpai created https://github.com/llvm/llvm-project/pull/211577

The Rubin architecture extends `tcgen05.commit` operations with two additional features: support for 32-bit CTA multicast masks and the ability to track completion of Matrix A reads from shared memory for all prior `tcgen05.mma` operations.

This change adds support for these features to the `tcgen05.commit` intrinsics.

>From f46648201848a3e1887af7fc3635a096284f736e Mon Sep 17 00:00:00 2001
From: Rajat Bajpai <rbajpai at nvidia.com>
Date: Thu, 23 Jul 2026 10:40:42 +0000
Subject: [PATCH] [NVPTX] Add Rubin extensions to tcgen05.commit

The Rubin architecture extends `tcgen05.commit` operations with two
additional features: support for 32-bit CTA multicast masks and the
ability to track completion of Matrix A reads from shared memory for
all prior `tcgen05.mma` operations.

This change adds support for these features to the `tcgen05.commit`
intrinsics.
---
 llvm/docs/NVPTXUsage.md                       |  16 +-
 llvm/include/llvm/IR/IntrinsicsNVVM.td        |  40 +-
 llvm/lib/Target/NVPTX/NVPTXIntrinsics.td      |  41 +-
 llvm/lib/Target/NVPTX/NVPTXSubtarget.h        |   6 +
 .../Assembler/auto_upgrade_nvvm_intrinsics.ll |  18 +
 .../CodeGen/NVPTX/tcgen05-commit-rubin.ll     | 438 ++++++++++++++++++
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td   |   2 +-
 .../Target/LLVMIR/nvvm/tcgen05-commit.mlir    |   8 +-
 8 files changed, 532 insertions(+), 37 deletions(-)
 create mode 100644 llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll

diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index 923a2b10f6be9..edd3352079df5 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -2634,8 +2634,17 @@ For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parall
 ```llvm
 declare void @llvm.nvvm.tcgen05.commit.{cg1,cg2}(ptr %mbar)
 declare void @llvm.nvvm.tcgen05.commit.shared.{cg1,cg2}(ptr addrspace(3) %mbar)
-declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}(ptr %mbar, i16 %mc)
-declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}(ptr addrspace(3) %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}.i16(ptr %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}.i16(ptr addrspace(3) %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}.i32(ptr %mbar, i32 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}.i32(ptr addrspace(3) %mbar, i32 %mc)
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.{cg1,cg2}(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.{cg1,cg2}(ptr addrspace(3) %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.{cg1,cg2}.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.{cg1,cg2}.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.{cg1,cg2}.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.{cg1,cg2}.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
 ```
 
 ##### Overview:
@@ -2647,7 +2656,8 @@ object (`%mbar`) track the completion of all prior asynchronous tcgen05
 operations. The `.mc` variants allow signaling on the mbarrier objects of
 multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
 variants generate `cta_group::1` and `cta_group::2` flavors of the
-instruction respectively.
+instruction, respectively. The `smem.a.read` variants allow tracking the completion
+of reads of Matrix A from shared memory for all prior `tcgen05.mma` operations.
 
 For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen-async-sync-operations-commit).
 
diff --git a/llvm/include/llvm/IR/IntrinsicsNVVM.td b/llvm/include/llvm/IR/IntrinsicsNVVM.td
index 40f3c40a76bc6..26be050d4b22a 100644
--- a/llvm/include/llvm/IR/IntrinsicsNVVM.td
+++ b/llvm/include/llvm/IR/IntrinsicsNVVM.td
@@ -3085,25 +3085,27 @@ foreach cta_group = ["cg1", "cg2"] in {
   def int_nvvm_tcgen05_relinq_alloc_permit_ # cta_group : Intrinsic<[], [],
     [IntrConvergent, IntrInaccessibleMemOnly]>;
 
-  def int_nvvm_tcgen05_commit_ # cta_group : Intrinsic<[],
-    [llvm_ptr_ty],        // mbar_ptr
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_shared_ # cta_group : Intrinsic<[],
-    [llvm_shared_ptr_ty], // mbar_ptr
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_mc_ # cta_group : Intrinsic<[],
-    [llvm_ptr_ty, llvm_i16_ty], // mbar_ptr, cta_mask
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_mc_shared_ # cta_group : Intrinsic<[],
-    [llvm_shared_ptr_ty, llvm_i16_ty], // mbar_ptr, cta_mask
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
+  foreach smem_a_read = ["", "smem_a_read_"] in {
+    def int_nvvm_tcgen05_commit_ # smem_a_read # cta_group : Intrinsic<[],
+      [llvm_ptr_ty],        // mbar_ptr
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # shared_ # cta_group : Intrinsic<[],
+      [llvm_shared_ptr_ty], // mbar_ptr
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # mc_ # cta_group : Intrinsic<[],
+      [llvm_ptr_ty, llvm_anyint_ty], // mbar_ptr, cta_mask[16 or 32 bits]
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # mc_shared_ # cta_group : Intrinsic<[],
+      [llvm_shared_ptr_ty, llvm_anyint_ty], // mbar_ptr, cta_mask[16 or 32 bits]
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+  }
 
   def int_nvvm_tcgen05_shift_down_ # cta_group : Intrinsic<[],
     [llvm_tmem_ptr_ty],   // tmem_addr
diff --git a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
index 5230e01261fd6..ea2147e720da4 100644
--- a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
+++ b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
@@ -5711,25 +5711,41 @@ def tcgen05_wait_ld: NullaryInst<"tcgen05.wait::ld.sync.aligned", int_nvvm_tcgen
 def tcgen05_wait_st: NullaryInst<"tcgen05.wait::st.sync.aligned", int_nvvm_tcgen05_wait_st>;
 
 multiclass TCGEN05_COMMIT_INTR<string AS, string num> {
-  defvar prefix = "tcgen05.commit.cta_group::" # num #".mbarrier::arrive::one.shared::cluster";
+  defvar prefix = "tcgen05.commit.cta_group::" # num #
+                  ".mbarrier::arrive::one.shared::cluster";
+  defvar prefix_smem_a_read = prefix # ".sync_restrict::shared::read::mma::a";
 
+  defvar intr_prefix = "int_nvvm_tcgen05_commit";
   defvar intr_suffix = !if(!eq(AS, "shared"), "_shared", "") # "_cg" # num;
-  defvar Intr = !cast<Intrinsic>("int_nvvm_tcgen05_commit" # intr_suffix);
-  defvar IntrMC = !cast<Intrinsic>("int_nvvm_tcgen05_commit_mc" # intr_suffix);
+  defvar Intr = !cast<Intrinsic>(intr_prefix # intr_suffix);
+  defvar IntrMC = !cast<Intrinsic>(intr_prefix # "_mc" # intr_suffix);
+  defvar IntrSmemARead =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read" # intr_suffix);
+  defvar IntrSmemAReadMC =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read_mc" # intr_suffix);
 
   def "" : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
              prefix # ".b64",
              [(Intr addr:$mbar)]>;
   def _MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
-                   prefix # ".multicast::cluster.b64",
-                   [(IntrMC addr:$mbar, B16:$mc)]>;
+              prefix # ".multicast::cluster.b64",
+              [(IntrMC addr:$mbar, i16:$mc)]>;
+  let Predicates = [callSubtarget<"hasTcgen05RubinFamilySupport">] in {
+    def _MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                    prefix # ".multicast::cluster::32b.b64",
+                    [(IntrMC addr:$mbar, i32:$mc)]>;
+    def _SMRA : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
+                  prefix_smem_a_read # ".b64",
+                  [(IntrSmemARead addr:$mbar)]>;
+    def _SMRA_MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
+                    prefix_smem_a_read # ".multicast::cluster.b64",
+                    [(IntrSmemAReadMC addr:$mbar, i16:$mc)]>;
+    def _SMRA_MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                        prefix_smem_a_read # ".multicast::cluster::32b.b64",
+                        [(IntrSmemAReadMC addr:$mbar, i32:$mc)]>;
+  }
 }
 
-defm TCGEN05_COMMIT_CG1 : TCGEN05_COMMIT_INTR<"", "1">;
-defm TCGEN05_COMMIT_CG2 : TCGEN05_COMMIT_INTR<"", "2">;
-defm TCGEN05_COMMIT_S64_CG1 : TCGEN05_COMMIT_INTR<"shared", "1">;
-defm TCGEN05_COMMIT_S64_CG2 : TCGEN05_COMMIT_INTR<"shared", "2">;
-
 multiclass TCGEN05_CP_INTR<string shape, string src_fmt, string mc = ""> {
   defvar dst_fmt = !if(!eq(src_fmt, ""), "", ".b8x16");
   defvar fmt_asm = StrJoin<".", [dst_fmt, src_fmt]>.ret;
@@ -5762,6 +5778,11 @@ foreach src_fmt = ["", "b6x16_p32", "b4x16_p64"] in {
 }
 } // Predicates
 
+defm TCGEN05_COMMIT_CG1 : TCGEN05_COMMIT_INTR<"", "1">;
+defm TCGEN05_COMMIT_CG2 : TCGEN05_COMMIT_INTR<"", "2">;
+defm TCGEN05_COMMIT_S64_CG1 : TCGEN05_COMMIT_INTR<"shared", "1">;
+defm TCGEN05_COMMIT_S64_CG2 : TCGEN05_COMMIT_INTR<"shared", "2">;
+
 let Predicates = [callSubtarget<"hasTcgen05ShiftSupport">] in {
 multiclass TCGEN05_SHIFT_INTR<string num, Intrinsic Intr> {
   def "" : BasicNVPTXInst<(outs),
diff --git a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
index 1df5d326f63a6..9561e00c84023 100644
--- a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
+++ b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
@@ -140,6 +140,12 @@ class NVPTXSubtarget : public NVPTXGenSubtargetInfo {
            hasPTXWithAccelSMs(86, {100, 101});
   }
 
+  // Checks Rubin family extensions support.
+  //  - tcgen05.commit
+  bool hasTcgen05RubinFamilySupport() const {
+    return hasPTXWithFamilySMs(94, {107});
+  }
+
   // Checks tcgen05.shift instruction support.
   bool hasTcgen05ShiftSupport() const {
     // sm_101 renamed to sm_110 in PTX 9.0
diff --git a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
index c59c9bda203b8..d3b815a1ef708 100644
--- a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
+++ b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
@@ -115,6 +115,11 @@ declare ptr addrspace(3) @llvm.nvvm.mapa.shared.cluster(ptr addrspace(3), i32)
 declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cluster(ptr addrspace(3), ptr addrspace(3), ptr addrspace(1), i32, i16, i64, i1, i1)
 declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.cluster(ptr addrspace(3), ptr addrspace(3), ptr addrspace(3), i32)
 
+declare void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr, i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr, i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3), i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3), i16)
+
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.2d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i32 %d1, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.3d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i32 %d1, i32 %d2, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
@@ -456,6 +461,19 @@ define void @nvvm_cp_async_bulk_intrinsics(ptr addrspace(3) %dst, ptr addrspace(
   ret void
 }
 
+; CHECK-LABEL: @nvvm_tcgen05_commit_mc_intrinsics
+define void @nvvm_tcgen05_commit_mc_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg1.i16(ptr %bar, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg2.i16(ptr %bar, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i16(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i16(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr %bar, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr %bar, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  ret void
+}
+
 ; CHECK-LABEL: @nvvm_cp_async_bulk_tensor_g2s_im2col
 define void @nvvm_cp_async_bulk_tensor_g2s_im2col(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, i16 %mc, i64 %ch) {
 ; CHECK: call void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.3d(ptr addrspace(7) %1, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i16 0, i64 0, i1 false, i1 false, i32 0)
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll b/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll
new file mode 100644
index 0000000000000..0b54e735ef3a9
--- /dev/null
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll
@@ -0,0 +1,438 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_107f  | FileCheck --check-prefixes=CHECK_PTX64 %s
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_107f  --nvptx-short-ptr | FileCheck --check-prefixes=CHECK_PTX64_SHARED32 %s
+
+; RUN: %if ptxas-sm_107f && ptxas-isa-9.4 %{ llc < %s -march=nvptx64 -mcpu=sm_107f  | %ptxas-verify -arch=sm_107f %}
+; RUN: %if ptxas-sm_107f && ptxas-isa-9.4 %{ llc < %s -march=nvptx64 -mcpu=sm_107f  --nvptx-short-ptr | %ptxas-verify -arch=sm_107f %}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.cg1(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.cg2(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg1(ptr addrspace(3) %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg2(ptr addrspace(3) %bar_addr)
+
+define void @test_tcgen05_commit_smem_a_read_cg1(ptr %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg1_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.cg1(ptr %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_cg2(ptr %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg2_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.cg2(ptr %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_shared_cg1(ptr addrspace(3) %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_shared_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_shared_cg1_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_shared_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_shared_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%r1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg1(ptr addrspace(3) %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_shared_cg2(ptr addrspace(3) %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_shared_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_shared_cg2_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_shared_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_shared_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%r1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg2(ptr addrspace(3) %bar_addr)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg1(ptr %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i16(ptr %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg2(ptr %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i16(ptr %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg1(ptr addrspace(3) %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%r1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg2(ptr addrspace(3) %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%r1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg1_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg2_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+define void @test_tcgen05_commit_mc_cg1_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_cg2_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_shared_cg1_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_shared_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_shared_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_shared_cg2_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_shared_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_shared_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..2bf21db6b6e6f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5331,7 +5331,7 @@ def NVVM_Tcgen05CommitOp : NVVM_Op<"tcgen05.commit", [NVVMRequiresSMf<[100, 101,
     llvm::SmallVector<llvm::Value *> args;
     auto id = NVVM::Tcgen05CommitOp::getIntrinsicIDAndArgs(
       *op, moduleTranslation, args);
-    createIntrinsicCall(builder, id, args);
+    createIntrinsicCall(builder, id, builder.getVoidTy(), args);
   }];
 }
 
diff --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
index 60475bf64ae7a..6997f8a98b2fc 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
@@ -8,10 +8,10 @@ llvm.func @llvm_nvvm_tcgen05_commit_generic(%barrier : !llvm.ptr, %cta_mask : i1
   // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.cg2(ptr %{{.*}})
   nvvm.tcgen05.commit %barrier {group = #nvvm.cta_group<cta_2>} : !llvm.ptr
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg1.i16(ptr %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask : !llvm.ptr, i16
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg2.i16(ptr %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask {group = #nvvm.cta_group<cta_2>} : !llvm.ptr, i16
   llvm.return
 }
@@ -24,10 +24,10 @@ llvm.func @llvm_nvvm_tcgen05_commit_shared(%barrier : !llvm.ptr<3>, %cta_mask :
   // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.shared.cg2(ptr addrspace(3) %{{.*}})
   nvvm.tcgen05.commit %barrier {group = #nvvm.cta_group<cta_2>} : !llvm.ptr<3>
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i16(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask : !llvm.ptr<3>, i16
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i16(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask {group = #nvvm.cta_group<cta_2>} : !llvm.ptr<3>, i16
   llvm.return
 }


From llvmlistbot at llvm.org  Thu Jul 23 08:14:18 2026
From: llvmlistbot at llvm.org (Byeongjee Kang)
Date: Thu, 23 Jul 2026 08:14:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a622fca.b1416da1.20b4cf.a855@mx.google.com>


https://github.com/byeongjee updated https://github.com/llvm/llvm-project/pull/196613

>From 4eb1075004eafe159cdb9b436cb5a4fb9a2a33b4 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 00:20:59 -0400
Subject: [PATCH 1/4] [MLIR][WasmSSA] Add global_set op

Adds the `wasmssa.global_set` operation, which writes a value to a
mutable global variable. The op verifies that the referenced symbol is
a `wasmssa.global` or `wasmssa.import_global` marked mutable, and that
the operand type matches the global's declared type.
---
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 19 +++++++
 mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp    | 38 +++++++++++++
 .../Dialect/WasmSSA/custom_parser/global.mlir | 12 ++++
 mlir/test/Dialect/WasmSSA/global-invalid.mlir | 55 +++++++++++++++++++
 4 files changed, 124 insertions(+)

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index bfa0953f50ac8..e2fb642433266 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -381,6 +381,25 @@ def WasmSSA_GlobalGetOp : WasmSSA_Op<"global_get", [DeclareOpInterfaceMethods<Sy
   let assemblyFormat = "$global attr-dict `:` type($global_val)";
 }
 
+def WasmSSA_GlobalSetOp : WasmSSA_Op<"global_set", [
+    DeclareOpInterfaceMethods<SymbolUserOpInterface>]> {
+  let summary = "Sets the value of the mutable global passed as argument.";
+  let description = [{
+     Sets the value of the mutable global referenced by the symbol to the given
+     value.
+
+     Example:
+
+     ```mlir
+     // Sets the value of `@global_0` to `%v`.
+     wasmssa.global_set @global_0 to %v : i32
+     ```
+    }];
+  let arguments = (ins FlatSymbolRefAttr: $global,
+                       WasmSSA_ValType: $value);
+  let assemblyFormat = "$global `to` $value `:` type($value) attr-dict";
+}
+
 def WasmSSA_IfOp : WasmSSA_Op<"if", [Terminator,
     DeclareOpInterfaceMethods<LabelLevelOpInterface>]> {
   let summary = "Execute the if region if condition value is non-zero, the else region otherwise.";
diff --git a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
index 08de887a56a97..eab9c7a9d53a7 100644
--- a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
+++ b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
@@ -331,6 +331,44 @@ GlobalGetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// GlobalSetOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+GlobalSetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+  Operation *symTabOp = SymbolTable::getNearestSymbolTable(*this);
+  StringRef referencedSymbol = getGlobal();
+  Operation *definitionOp = symbolTable.lookupSymbolIn(
+      symTabOp, StringAttr::get(this->getContext(), referencedSymbol));
+  if (!definitionOp)
+    return emitError() << "symbol @" << referencedSymbol << " is undefined";
+
+  Type globalType;
+  bool isMutable = false;
+  if (auto global = dyn_cast<GlobalOp>(definitionOp)) {
+    globalType = global.getType();
+    isMutable = global.getIsMutable();
+  } else if (auto globalImport = dyn_cast<GlobalImportOp>(definitionOp)) {
+    globalType = globalImport.getType();
+    isMutable = globalImport.getIsMutable();
+  } else {
+    return emitError() << "symbol @" << referencedSymbol
+                       << " is not a global symbol";
+  }
+
+  if (!isMutable)
+    return emitError("global.set target must be mutable");
+
+  Type valueType = getValue().getType();
+  if (globalType != valueType)
+    return emitError("global.set value type does not match target global "
+                     "type: expected ")
+           << globalType << " but got " << valueType;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // GlobalImportOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
index a25abbd2a8662..6441731a0a335 100644
--- a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
@@ -19,6 +19,12 @@ module {
     %0 = wasmssa.global_get @global_0 : i32
     wasmssa.return %0 : i32
   }
+
+  wasmssa.func @set_global() {
+    %0 = wasmssa.const 7 : i32
+    wasmssa.global_set @global_2 to %0 : i32
+    wasmssa.return
+  }
 }
 
 // CHECK-LABEL:   wasmssa.import_global "from_js" from "env" as @global_0 : i32
@@ -42,3 +48,9 @@ module {
 // CHECK:           %[[VAL_0:.*]] = wasmssa.global_get @global_0 : i32
 // CHECK:           wasmssa.return %[[VAL_0]] : i32
 // CHECK:         }
+
+// CHECK-LABEL:   wasmssa.func @set_global() {
+// CHECK:           %[[VAL_0:.*]] = wasmssa.const 7 : i32
+// CHECK:           wasmssa.global_set @global_2 to %[[VAL_0]] : i32
+// CHECK:           wasmssa.return
+// CHECK:         }
diff --git a/mlir/test/Dialect/WasmSSA/global-invalid.mlir b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
index c5bc606fd13f3..600b2f0715fdc 100644
--- a/mlir/test/Dialect/WasmSSA/global-invalid.mlir
+++ b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
@@ -40,3 +40,58 @@ module {
     wasmssa.return %0 : i32
   }
 }
+
+// -----
+
+module {
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @missing is undefined}}
+    wasmssa.global_set @missing to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i32 : {
+    %0 = wasmssa.const 0 : i32
+    wasmssa.return %0 : i32
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set target must be mutable}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i64 mutable : {
+    %0 = wasmssa.const 0 : i64
+    wasmssa.return %0 : i64
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set value type does not match target global type: expected 'i64' but got 'i32'}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.func @not_global() {
+    wasmssa.return
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @not_global is not a global symbol}}
+    wasmssa.global_set @not_global to %0 : i32
+    wasmssa.return
+  }
+}

>From c9c103386c84a4756f84bbf3a60cbfb07076b541 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 15:05:35 -0400
Subject: [PATCH 2/4] [MLIR][WasmSSA] Add select op

Adds the `wasmssa.select` operation, a Wasm value-typed ternary that
returns one of two operands based on an i32 condition. The op uses the
`AllTypesMatch` trait to require both branches and the result share the
same Wasm value type, and is marked `Pure` since it has no side effects.

Also adds the `SideEffectInterfaces` td/header includes needed by the
`Pure` trait.
---
 .../include/mlir/Dialect/WasmSSA/IR/WasmSSA.h |  1 +
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 24 ++++++++++++
 .../Dialect/WasmSSA/custom_parser/select.mlir | 37 +++++++++++++++++++
 3 files changed, 62 insertions(+)
 create mode 100644 mlir/test/Dialect/WasmSSA/custom_parser/select.mlir

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
index 64391d807c633..64d480c065b79 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
@@ -38,6 +38,7 @@
 #include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
 
 //===----------------------------------------------------------------------===//
 // WebAssembly Constraints
diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index e2fb642433266..3b3c3e8825fef 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -16,6 +16,7 @@ include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.td"
 
 include "mlir/Interfaces/FunctionInterfaces.td"
 include "mlir/Interfaces/InferTypeOpInterface.td"
+include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/IR/BuiltinAttributeInterfaces.td"
 include "mlir/IR/SymbolInterfaces.td"
 
@@ -628,6 +629,29 @@ def WasmSSA_ReturnOp : WasmSSA_Op<"return", [Terminator]> {
   ];
 }
 
+def WasmSSA_SelectOp : WasmSSA_Op<"select", [
+    AllTypesMatch<["true_value", "false_value", "result"]>,
+    Pure]> {
+  let summary = "Select between two values based on a condition.";
+  let description = [{
+     Returns `true_value` if `condition` is non-zero, `false_value` otherwise.
+     Both operands and the result must share the same Wasm value type.
+
+     Example:
+
+     ```mlir
+     // Returns %a if %cond is non-zero, otherwise %b.
+     %r = wasmssa.select %cond, %a, %b : i32
+     ```
+    }];
+  let arguments = (ins WasmSSA_ValType: $true_value,
+                       WasmSSA_ValType: $false_value,
+                       I32: $condition);
+  let results = (outs WasmSSA_ValType: $result);
+  let assemblyFormat =
+      "$condition `,` $true_value `,` $false_value attr-dict `:` type($result)";
+}
+
 // ---- Numeric ops
 
 class WasmSSA_BinaryNumericalOp<string mnemonic, string summaryStr, string descStr,
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
new file mode 100644
index 0000000000000..80ced11c01bc3
--- /dev/null
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
@@ -0,0 +1,37 @@
+// RUN: mlir-opt %s | FileCheck %s
+
+module {
+  wasmssa.func @select_i32(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to i32>,
+                           %b: !wasmssa<local ref to i32>) -> i32 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to i32
+    %2 = wasmssa.local_get %b : ref to i32
+    %r = wasmssa.select %0, %1, %2 : i32
+    wasmssa.return %r : i32
+  }
+
+  wasmssa.func @select_f64(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to f64>,
+                           %b: !wasmssa<local ref to f64>) -> f64 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to f64
+    %2 = wasmssa.local_get %b : ref to f64
+    %r = wasmssa.select %0, %1, %2 : f64
+    wasmssa.return %r : f64
+  }
+}
+
+// CHECK-LABEL:   wasmssa.func @select_i32(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : i32
+// CHECK:           wasmssa.return %[[R]] : i32
+
+// CHECK-LABEL:   wasmssa.func @select_f64(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : f64
+// CHECK:           wasmssa.return %[[R]] : f64

>From f8775f7b26b46c32ea6ea09ebb4dbb3bb87b4302 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Thu, 23 Jul 2026 11:07:20 -0400
Subject: [PATCH 3/4] [MLIR][WasmSSA] Add global_set conversion to
 RaiseWasmMLIR

Lowers wasmssa.global_set to memref.get_global + memref.store at index
0, matching the memref<1xT> shape emitted by the existing global
conversions.
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 18 ++++++++++
 .../RaiseWasm/wasm-global-set-to-memref.mlir  | 35 +++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index b4b5006352c11..d719c42459d9d 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -671,6 +671,23 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmGlobalSetOpConversion : OpConversionPattern<GlobalSetOp> {
+  using OpConversionPattern::OpConversionPattern;
+  LogicalResult
+  matchAndRewrite(GlobalSetOp globalSetOp, GlobalSetOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = globalSetOp.getLoc();
+    auto globalPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, adaptor.getValue().getType()),
+        globalSetOp.getGlobal());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    rewriter.replaceOpWithNewOp<memref::StoreOp>(
+        globalSetOp, adaptor.getValue(), globalPtr.getResult(),
+        ValueRange{idx.getResult()});
+    return success();
+  }
+};
+
 struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -876,6 +893,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmGeSIOpConversion,
            WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
+           WasmGlobalSetOpConversion,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
            WasmGtOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
new file mode 100644
index 0000000000000..937f7efd006ba
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
@@ -0,0 +1,35 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+wasmssa.global @global_i32 i32 mutable : {
+  %0 = wasmssa.const 10 : i32
+  wasmssa.return %0 : i32
+}
+
+wasmssa.global @global_f64 f64 mutable : {
+  %0 = wasmssa.const 3.14 : f64
+  wasmssa.return %0 : f64
+}
+
+// CHECK-LABEL:   func.func @set_global_i32() {
+wasmssa.func exported @set_global_i32() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 42 : i32
+  %0 = wasmssa.const 42 : i32
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_i32 : memref<1xi32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xi32>
+  wasmssa.global_set @global_i32 to %0 : i32
+// CHECK:           return
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @set_global_f64() {
+wasmssa.func exported @set_global_f64() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 2.500000e-01 : f64
+  %0 = wasmssa.const 0.25 : f64
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_f64 : memref<1xf64>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xf64>
+  wasmssa.global_set @global_f64 to %0 : f64
+// CHECK:           return
+  wasmssa.return
+}

>From 186be73bc96aa146b2e251e1c94fdacdbb5dc639 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Thu, 23 Jul 2026 11:13:33 -0400
Subject: [PATCH 4/4] [MLIR][WasmSSA] Add select conversion to RaiseWasmMLIR

Lowers wasmssa.select to arith.select, normalizing the i32 condition
to i1 via arith.cmpi ne 0, matching the existing idiom for condition
handling in this pass.
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 20 ++++++
 .../wasm-select-to-arith-select.mlir          | 65 +++++++++++++++++++
 2 files changed, 85 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index d719c42459d9d..fcafbef9d6035 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -813,6 +813,25 @@ struct WasmReturnOpConversion : OpConversionPattern<ReturnOp> {
   }
 };
 
+struct WasmSelectOpConversion : OpConversionPattern<SelectOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SelectOp selectOp, SelectOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = selectOp.getLoc();
+    auto zero = arith::ConstantOp::create(rewriter, loc,
+                                          rewriter.getI32IntegerAttr(0));
+    auto flag =
+        arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::ne,
+                              adaptor.getCondition(), zero.getResult());
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(selectOp, flag.getResult(),
+                                                 adaptor.getTrueValue(),
+                                                 adaptor.getFalseValue());
+    return success();
+  }
+};
+
 struct RaiseWasmMLIRPass : public impl::RaiseWasmMLIRBase<RaiseWasmMLIRPass> {
   void runOnOperation() override {
     ConversionTarget target{getContext()};
@@ -924,6 +943,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmReturnOpConversion,
            WasmRotlOpConversion,
            WasmRotrOpConversion,
+           WasmSelectOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
new file mode 100644
index 0000000000000..ed5d15f87d70f
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
@@ -0,0 +1,65 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @select_i32() -> i32 {
+wasmssa.func @select_i32() -> i32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i32
+  %a = wasmssa.const 12 : i32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i32
+  %b = wasmssa.const 50 : i32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i32
+  %r = wasmssa.select %cond, %a, %b : i32
+// CHECK:           return %[[RES]] : i32
+  wasmssa.return %r : i32
+}
+
+// CHECK-LABEL:   func.func @select_i64() -> i64 {
+wasmssa.func @select_i64() -> i64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i64
+  %a = wasmssa.const 12 : i64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i64
+  %b = wasmssa.const 50 : i64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i64
+  %r = wasmssa.select %cond, %a, %b : i64
+// CHECK:           return %[[RES]] : i64
+  wasmssa.return %r : i64
+}
+
+// CHECK-LABEL:   func.func @select_f32() -> f32 {
+wasmssa.func @select_f32() -> f32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 1.250000e-01 : f32
+  %a = wasmssa.const 0.125 : f32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.500000e-01 : f32
+  %b = wasmssa.const 0.25 : f32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f32
+  %r = wasmssa.select %cond, %a, %b : f32
+// CHECK:           return %[[RES]] : f32
+  wasmssa.return %r : f32
+}
+
+// CHECK-LABEL:   func.func @select_f64() -> f64 {
+wasmssa.func @select_f64() -> f64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 3.140000e+00 : f64
+  %a = wasmssa.const 3.14 : f64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.718000e+00 : f64
+  %b = wasmssa.const 2.718 : f64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f64
+  %r = wasmssa.select %cond, %a, %b : f64
+// CHECK:           return %[[RES]] : f64
+  wasmssa.return %r : f64
+}


From llvmlistbot at llvm.org  Thu Jul 23 08:14:46 2026
From: llvmlistbot at llvm.org (Byeongjee Kang)
Date: Thu, 23 Jul 2026 08:14:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a622fe6.f1c1923d.d9163.36be@mx.google.com>


byeongjee wrote:

@flemairen6 I added the conversion patterns

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Thu Jul 23 08:16:16 2026
From: llvmlistbot at llvm.org (Spencer Bryngelson)
Date: Thu, 23 Jul 2026 08:16:16 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211566@github.com>
Message-ID: <6a623040.bd93e731.2ac69c.65c0@mx.google.com>


sbryngelson wrote:

Understood, that is the right call. Your MLIR test covers it.


https://github.com/llvm/llvm-project/pull/211566

From llvmlistbot at llvm.org  Thu Jul 23 08:17:02 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Thu, 23 Jul 2026 08:17:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Handle null region in
 LoopLikeOpInterface::isDefinedOutsideOfLoop (PR #204521)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/204521@github.com>
Message-ID: <6a62306e.1728eaf3.5e665.f42b@mx.google.com>


https://github.com/makslevental auto_merge_enabled https://github.com/llvm/llvm-project/pull/204521

From llvmlistbot at llvm.org  Thu Jul 23 08:17:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:17:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 4d777c1 - [mlir] Handle null region in
 LoopLikeOpInterface::isDefinedOutsideOfLoop (#204521)
Message-ID: <6a62307b.92ea0e2a.2ed880.ba62@mx.google.com>


Author: Prem C
Date: 2026-07-23T08:17:10-07:00
New Revision: 4d777c139fc23303e8fc5cc9a78069aef94ab0ab

URL: https://github.com/llvm/llvm-project/commit/4d777c139fc23303e8fc5cc9a78069aef94ab0ab
DIFF: https://github.com/llvm/llvm-project/commit/4d777c139fc23303e8fc5cc9a78069aef94ab0ab.diff

LOG: [mlir] Handle null region in LoopLikeOpInterface::isDefinedOutsideOfLoop (#204521)

Fixes #203860 

In LoopLikeOpInterface::isDefinedOutsideOfLoop default implementation,
value.getParentRegion() can return null during signature conversion
rollbacks when blocks/ops are unlinked. Check for null region to avoid a
segmentation fault.

Also, add a regression test for convert-func-to-llvm with
index-bitwidth=32 on functions with affine.for loops.

Added: 
    

Modified: 
    mlir/include/mlir/IR/Region.h
    mlir/include/mlir/Interfaces/LoopLikeInterface.td
    mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/IR/Region.h b/mlir/include/mlir/IR/Region.h
index fa9d8f08d84bb..a61eaaa54d401 100644
--- a/mlir/include/mlir/IR/Region.h
+++ b/mlir/include/mlir/IR/Region.h
@@ -212,6 +212,9 @@ class Region {
   /// Return the parent operation this region is attached to.
   Operation *getParentOp() { return container; }
 
+  /// Return true if this region is attached to an operation.
+  bool isAttached() { return container != nullptr; }
+
   /// Find the first parent operation of the given type, or nullptr if there is
   /// no ancestor operation.
   template <typename ParentT>

diff  --git a/mlir/include/mlir/Interfaces/LoopLikeInterface.td b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
index 5fb897339ffde..0bb82c1e4638e 100644
--- a/mlir/include/mlir/Interfaces/LoopLikeInterface.td
+++ b/mlir/include/mlir/Interfaces/LoopLikeInterface.td
@@ -62,7 +62,8 @@ def LoopLikeOpInterface : OpInterface<"LoopLikeOpInterface"> {
       /*args=*/(ins "::mlir::Value ":$value),
       /*methodBody=*/"",
       /*defaultImplementation=*/[{
-        return !$_op->isAncestor(value.getParentRegion()->getParentOp());
+        ::mlir::Region *region = value.getParentRegion();
+        return !region || !region->isAttached() || !$_op->isAncestor(region->getParentOp());
       }]
     >,
     InterfaceMethod<[{

diff  --git a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
index 94dfceadbc449..6a04960e3513a 100644
--- a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
+++ b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
@@ -583,3 +583,18 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// During signature conversion with index-bitwidth=32, pattern rollback can
+// unlink blocks/ops from their regions. If loop-like operations are folded
+// in this state, querying `isDefinedOutsideOfLoop` on block arguments
+// of unlinked blocks would previously dereference a null parent region pointer
+// and crash.
+// See: https://github.com/llvm/llvm-project/issues/203860
+// CHECK32-LABEL: llvm.func @affine_for_index_bitwidth_32
+func.func @affine_for_index_bitwidth_32(%arg0: index, %arg1: index, %arg2: index, %arg3: index) {
+  %0 = affine.for %arg4 = 0 to 101 iter_args(%arg5 = %arg1) -> (index) {
+    affine.yield %arg3 : index
+  }
+  return
+}
+


        

From llvmlistbot at llvm.org  Thu Jul 23 08:48:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:48:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a6237ba.510f576d.33aa7d.f3ab@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210586

>From ee4b1d4aa4c3c0afaf896ace40531d702551cb77 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 18:07:37 +0800
Subject: [PATCH 1/4] [MLIR][CAPI][Python] Add support for constructing memory
 effect instances

---
 mlir/include/mlir-c/Interfaces.h              |  78 +++++
 .../mlir/Bindings/Python/IRInterfaces.h       |  53 +++-
 mlir/include/mlir/CAPI/Interfaces.h           |   4 +
 mlir/lib/Bindings/Python/DialectTransform.cpp |  20 +-
 mlir/lib/Bindings/Python/IRInterfaces.cpp     | 107 ++++++-
 mlir/lib/CAPI/Interfaces/Interfaces.cpp       |  72 +++++
 mlir/test/CAPI/ir.c                           |  43 +++
 .../dialects/memory_effects_op_interface.py   | 291 ++++++++++++++++++
 8 files changed, 654 insertions(+), 14 deletions(-)
 create mode 100644 mlir/test/python/dialects/memory_effects_op_interface.py

diff --git a/mlir/include/mlir-c/Interfaces.h b/mlir/include/mlir-c/Interfaces.h
index a5251dc78471f..a416a6ab76f87 100644
--- a/mlir/include/mlir-c/Interfaces.h
+++ b/mlir/include/mlir-c/Interfaces.h
@@ -28,7 +28,10 @@ extern "C" {
   };                                                                           \
   typedef struct name name
 
+DEFINE_C_API_STRUCT(MlirMemoryEffect, void);
+DEFINE_C_API_STRUCT(MlirMemoryEffectInstance, void);
 DEFINE_C_API_STRUCT(MlirMemoryEffectInstancesList, void);
+DEFINE_C_API_STRUCT(MlirSideEffectResource, void);
 
 #undef DEFINE_C_API_STRUCT
 
@@ -144,6 +147,81 @@ mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectsOpInterface
 //===---------------------------------------------------------------------===//
 
+/// Returns the borrowed singleton instance of the allocate memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsAllocateGet(void);
+
+/// Returns the borrowed singleton instance of the free memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsFreeGet(void);
+
+/// Returns the borrowed singleton instance of the read memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsReadGet(void);
+
+/// Returns the borrowed singleton instance of the write memory effect.
+MLIR_CAPI_EXPORTED MlirMemoryEffect mlirMemoryEffectsWriteGet(void);
+
+/// Returns the borrowed singleton instance of the default side effect
+/// resource.
+MLIR_CAPI_EXPORTED MlirSideEffectResource
+mlirSideEffectsDefaultResourceGet(void);
+
+/// Creates a memory effect instance without an associated IR entity.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation operand.
+/// `parameters` may be a null attribute. The caller owns the returned instance
+/// and must destroy it with `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpOperand(MlirMemoryEffect effect,
+                                           MlirOpOperand opOperand,
+                                           MlirAttribute parameters, int stage,
+                                           bool effectOnFullRegion,
+                                           MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with an operation result.
+/// `result` must wrap an OpResult. `parameters` may be a null attribute. The
+/// caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForOpResult(MlirMemoryEffect effect,
+                                          MlirValue result,
+                                          MlirAttribute parameters, int stage,
+                                          bool effectOnFullRegion,
+                                          MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a block argument.
+/// `blockArgument` must wrap a BlockArgument. `parameters` may be a null
+/// attribute. The caller owns the returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource);
+
+/// Creates a memory effect instance associated with a symbol. `symbol` must be
+/// a SymbolRefAttr. `parameters` may be a null attribute. The caller owns the
+/// returned instance and must destroy it with
+/// `mlirMemoryEffectInstanceDestroy`.
+MLIR_CAPI_EXPORTED MlirMemoryEffectInstance
+mlirMemoryEffectInstanceCreateForSymbol(MlirMemoryEffect effect,
+                                        MlirAttribute symbol,
+                                        MlirAttribute parameters, int stage,
+                                        bool effectOnFullRegion,
+                                        MlirSideEffectResource resource);
+
+/// Destroys a memory effect instance created by one of the functions above.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance);
+
+/// Appends a copy of `instance` to the given list. This does not take ownership
+/// of `instance`; the caller remains responsible for destroying it.
+MLIR_CAPI_EXPORTED void
+mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                    MlirMemoryEffectInstance instance);
+
 /// Returns the interface TypeID of the MemoryEffectsOpInterface.
 MLIR_CAPI_EXPORTED MlirTypeID mlirMemoryEffectsOpInterfaceTypeID(void);
 
diff --git a/mlir/include/mlir/Bindings/Python/IRInterfaces.h b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
index fb30e030b6c32..1ba995aa82c5e 100644
--- a/mlir/include/mlir/Bindings/Python/IRInterfaces.h
+++ b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
@@ -135,7 +135,58 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-struct PyMemoryEffectsInstanceList {
+/// A borrowed memory effect.
+class PyMemoryEffect {
+public:
+  explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
+
+  MlirMemoryEffect get() const { return effect; }
+
+private:
+  MlirMemoryEffect effect;
+};
+
+/// A borrowed side effect resource.
+class PySideEffectResource {
+public:
+  explicit PySideEffectResource(MlirSideEffectResource resource)
+      : resource(resource) {}
+
+  MlirSideEffectResource get() const { return resource; }
+
+private:
+  MlirSideEffectResource resource;
+};
+
+/// An owning memory effect instance.
+class PyMemoryEffectInstance {
+public:
+  explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
+      : instance(instance) {}
+  PyMemoryEffectInstance(PyMemoryEffectInstance &&other) noexcept
+      : instance(other.instance) {
+    other.instance.ptr = nullptr;
+  }
+  ~PyMemoryEffectInstance() {
+    if (instance.ptr)
+      mlirMemoryEffectInstanceDestroy(instance);
+  }
+
+  MlirMemoryEffectInstance get() const { return instance; }
+
+private:
+  MlirMemoryEffectInstance instance;
+};
+
+/// A callback-scoped view of a list of memory effect instances.
+class PyMemoryEffectsInstanceList {
+public:
+  explicit PyMemoryEffectsInstanceList(MlirMemoryEffectInstancesList effects)
+      : effects(effects) {}
+
+  MlirMemoryEffectInstancesList get() const { return effects; }
+
+private:
   MlirMemoryEffectInstancesList effects;
 };
 
diff --git a/mlir/include/mlir/CAPI/Interfaces.h b/mlir/include/mlir/CAPI/Interfaces.h
index 15afc9fb0f18e..55d850ccd01eb 100644
--- a/mlir/include/mlir/CAPI/Interfaces.h
+++ b/mlir/include/mlir/CAPI/Interfaces.h
@@ -22,5 +22,9 @@
 DEFINE_C_API_PTR_METHODS(
     MlirMemoryEffectInstancesList,
     llvm::SmallVectorImpl<mlir::MemoryEffects::EffectInstance>)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffect, mlir::MemoryEffects::Effect)
+DEFINE_C_API_PTR_METHODS(MlirMemoryEffectInstance,
+                         mlir::MemoryEffects::EffectInstance)
+DEFINE_C_API_PTR_METHODS(MlirSideEffectResource, mlir::SideEffects::Resource)
 
 #endif // MLIR_CAPI_INTERFACES_H
diff --git a/mlir/lib/Bindings/Python/DialectTransform.cpp b/mlir/lib/Bindings/Python/DialectTransform.cpp
index bd72082cea7e8..f84189e3a22ff 100644
--- a/mlir/lib/Bindings/Python/DialectTransform.cpp
+++ b/mlir/lib/Bindings/Python/DialectTransform.cpp
@@ -494,38 +494,38 @@ struct ParamType : PyConcreteType<ParamType> {
 
 namespace {
 void onlyReadsHandle(nb::iterable &operands,
-                     PyMemoryEffectsInstanceList effects) {
+                     const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(),
-                               effects.effects);
+                               effects.get());
 };
 
 void consumesHandle(nb::iterable &operands,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
   mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
 void producesHandle(nb::iterable &results,
-                    PyMemoryEffectsInstanceList effects) {
+                    const PyMemoryEffectsInstanceList &effects) {
   std::vector<MlirValue> resultsVec;
   for (auto result : results)
     resultsVec.push_back(nb::cast<PyOpResult>(result).get());
   mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(),
-                              effects.effects);
+                              effects.get());
 };
 
-void modifiesPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformModifiesPayload(effects.effects);
+void modifiesPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformModifiesPayload(effects.get());
 }
 
-void onlyReadsPayload(PyMemoryEffectsInstanceList effects) {
-  mlirTransformOnlyReadsPayload(effects.effects);
+void onlyReadsPayload(const PyMemoryEffectsInstanceList &effects) {
+  mlirTransformOnlyReadsPayload(effects.get());
 }
 } // namespace
 
diff --git a/mlir/lib/Bindings/Python/IRInterfaces.cpp b/mlir/lib/Bindings/Python/IRInterfaces.cpp
index c4e87246bd68c..4495cb6e8397d 100644
--- a/mlir/lib/Bindings/Python/IRInterfaces.cpp
+++ b/mlir/lib/Bindings/Python/IRInterfaces.cpp
@@ -34,6 +34,70 @@ its return shaped type components. Raises ValueError on failure.)";
 
 namespace {
 
+MlirAttribute unwrapOptionalAttribute(const nb::object &attribute) {
+  if (attribute.is_none())
+    return mlirAttributeGetNull();
+
+  PyAttribute *pyAttribute = nullptr;
+  if (!nb::try_cast<PyAttribute *>(attribute, pyAttribute) || !pyAttribute)
+    throw nb::type_error("parameters must be an Attribute or None");
+  return pyAttribute->get();
+}
+
+void appendMemoryEffectInstance(PyMemoryEffectsInstanceList &effects,
+                                const PyMemoryEffect &effect,
+                                const nb::object &target,
+                                const nb::object &parameters, int stage,
+                                bool effectOnFullRegion,
+                                const PySideEffectResource &resource) {
+  MlirMemoryEffectInstancesList list = effects.get();
+  MlirAttribute unwrappedParameters = unwrapOptionalAttribute(parameters);
+
+  MlirMemoryEffectInstance rawInstance{nullptr};
+  if (target.is_none()) {
+    rawInstance =
+        mlirMemoryEffectInstanceCreate(effect.get(), unwrappedParameters, stage,
+                                       effectOnFullRegion, resource.get());
+  } else {
+    PyOpOperand *opOperand = nullptr;
+    PyValue *value = nullptr;
+    PyAttribute *attribute = nullptr;
+    if (nb::try_cast<PyOpOperand *>(target, opOperand) && opOperand) {
+      rawInstance = mlirMemoryEffectInstanceCreateForOpOperand(
+          effect.get(), *opOperand, unwrappedParameters, stage,
+          effectOnFullRegion, resource.get());
+    } else if (nb::try_cast<PyValue *>(target, value) && value) {
+      MlirValue mlirValue = value->get();
+      if (mlirValueIsAOpResult(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForOpResult(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else if (mlirValueIsABlockArgument(mlirValue)) {
+        rawInstance = mlirMemoryEffectInstanceCreateForBlockArgument(
+            effect.get(), mlirValue, unwrappedParameters, stage,
+            effectOnFullRegion, resource.get());
+      } else {
+        throw nb::type_error(
+            "target Value must be an OpResult or BlockArgument");
+      }
+    } else if (nb::try_cast<PyAttribute *>(target, attribute) && attribute) {
+      MlirAttribute symbol = attribute->get();
+      if (!mlirAttributeIsASymbolRef(symbol))
+        throw nb::type_error("target Attribute must be a SymbolRefAttr");
+      rawInstance = mlirMemoryEffectInstanceCreateForSymbol(
+          effect.get(), symbol, unwrappedParameters, stage, effectOnFullRegion,
+          resource.get());
+    } else {
+      throw nb::type_error(
+          "target must be an OpOperand, OpResult, BlockArgument, "
+          "SymbolRefAttr, or None");
+    }
+  }
+
+  PyMemoryEffectInstance instance(rawInstance);
+  mlirMemoryEffectInstancesListAppend(list, instance.get());
+}
+
 /// Takes in an optional ist of operands and converts them into a std::vector
 /// of MlirVlaues. Returns an empty std::vector if the list is empty.
 std::vector<MlirValue> wrapOperands(std::optional<nb::sequence> operandList) {
@@ -474,9 +538,46 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("Speculatable", MlirSpeculatabilitySpeculatable)
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
-  auto memoryEffectsInstanceListClass =
-      nb::class_<PyMemoryEffectsInstanceList>(m, "MemoryEffectInstancesList");
-  (void)memoryEffectsInstanceListClass;
+  nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
+      .def_prop_ro_static("allocate",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(
+                                mlirMemoryEffectsAllocateGet());
+                          })
+      .def_prop_ro_static("free",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsFreeGet());
+                          })
+      .def_prop_ro_static("read",
+                          [](nb::object & /*class*/) {
+                            return PyMemoryEffect(mlirMemoryEffectsReadGet());
+                          })
+      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+        return PyMemoryEffect(mlirMemoryEffectsWriteGet());
+      });
+
+  nb::class_<PySideEffectResource>(m, "SideEffectResource",
+                                   "A side effect resource.")
+      .def_prop_ro_static("default", [](nb::object & /*class*/) {
+        return PySideEffectResource(mlirSideEffectsDefaultResourceGet());
+      });
+
+  nb::class_<PyMemoryEffectsInstanceList>(
+      m, "MemoryEffectInstancesList",
+      "A memory effect list that is valid only during get_effects.")
+      .def("append", &appendMemoryEffectInstance, nb::arg("effect"),
+           nb::arg("target").none() = nb::none(), nb::kw_only(),
+           nb::arg("parameters").none() = nb::none(), nb::arg("stage") = 0,
+           nb::arg("effect_on_full_region") = false,
+           nb::arg("resource") =
+               PySideEffectResource(mlirSideEffectsDefaultResourceGet()),
+           nb::sig("def append(self, effect: MemoryEffect, target: OpOperand | "
+                   "OpResult | BlockArgument | SymbolRefAttr | None = None, *, "
+                   "parameters: Attribute | None = None, stage: int = 0, "
+                   "effect_on_full_region: bool = False, resource: "
+                   "SideEffectResource = ...) -> None"),
+           "Append a memory effect instance. The target may be an OpOperand, "
+           "OpResult, BlockArgument, SymbolRefAttr, or None.");
 
   PyConditionallySpeculatableOpInterface::bind(m);
   PyInferShapedTypeOpInterface::bind(m);
diff --git a/mlir/lib/CAPI/Interfaces/Interfaces.cpp b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
index 35a2bd562a8a1..1bf8dffa9c431 100644
--- a/mlir/lib/CAPI/Interfaces/Interfaces.cpp
+++ b/mlir/lib/CAPI/Interfaces/Interfaces.cpp
@@ -280,6 +280,78 @@ MlirSpeculatability mlirConditionallySpeculatableOpInterfaceGetSpeculatability(
 // MemoryEffectOpInterface
 //===---------------------------------------------------------------------===//
 
+MlirMemoryEffect mlirMemoryEffectsAllocateGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Allocate::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsFreeGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Free::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsReadGet() {
+  return wrap(static_cast<MemoryEffects::Effect *>(MemoryEffects::Read::get()));
+}
+
+MlirMemoryEffect mlirMemoryEffectsWriteGet() {
+  return wrap(
+      static_cast<MemoryEffects::Effect *>(MemoryEffects::Write::get()));
+}
+
+MlirSideEffectResource mlirSideEffectsDefaultResourceGet() {
+  return wrap(static_cast<SideEffects::Resource *>(
+      SideEffects::DefaultResource::get()));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreate(
+    MlirMemoryEffect effect, MlirAttribute parameters, int stage,
+    bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(parameters), stage, effectOnFullRegion,
+      unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpOperand(
+    MlirMemoryEffect effect, MlirOpOperand opOperand, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), unwrap(opOperand), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForOpResult(
+    MlirMemoryEffect effect, MlirValue result, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<OpResult>(unwrap(result)), unwrap(parameters), stage,
+      effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForBlockArgument(
+    MlirMemoryEffect effect, MlirValue blockArgument, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<BlockArgument>(unwrap(blockArgument)),
+      unwrap(parameters), stage, effectOnFullRegion, unwrap(resource)));
+}
+
+MlirMemoryEffectInstance mlirMemoryEffectInstanceCreateForSymbol(
+    MlirMemoryEffect effect, MlirAttribute symbol, MlirAttribute parameters,
+    int stage, bool effectOnFullRegion, MlirSideEffectResource resource) {
+  return wrap(new MemoryEffects::EffectInstance(
+      unwrap(effect), cast<SymbolRefAttr>(unwrap(symbol)), unwrap(parameters),
+      stage, effectOnFullRegion, unwrap(resource)));
+}
+
+void mlirMemoryEffectInstanceDestroy(MlirMemoryEffectInstance instance) {
+  delete unwrap(instance);
+}
+
+void mlirMemoryEffectInstancesListAppend(MlirMemoryEffectInstancesList list,
+                                         MlirMemoryEffectInstance instance) {
+  unwrap(list)->push_back(*unwrap(instance));
+}
+
 MlirTypeID mlirMemoryEffectsOpInterfaceTypeID() {
   return wrap(MemoryEffectOpInterface::getInterfaceID());
 }
diff --git a/mlir/test/CAPI/ir.c b/mlir/test/CAPI/ir.c
index 57ae8b9a2819b..3c72209ad95c3 100644
--- a/mlir/test/CAPI/ir.c
+++ b/mlir/test/CAPI/ir.c
@@ -2589,6 +2589,8 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: arith.constant speculatability: 1
 
   MlirOperationState storeState = mlirOperationStateGet(storeName, loc);
+  MlirValue constantResult = mlirOperationGetResult(constantOp, 0);
+  mlirOperationStateAddOperands(&storeState, 1, &constantResult);
   MlirOperation storeOp = mlirOperationCreate(&storeState);
   if (mlirOperationImplementsInterface(storeOp, condSpecTypeID)) {
     fprintf(stderr, "ERROR: Expected memref.store instance to not implement "
@@ -2619,6 +2621,47 @@ int testInterfaces(MlirContext ctx) {
   // CHECK: memref.store speculatability: 2
   // CHECK: callback count: 1
 
+  MlirMemoryEffect allocate = mlirMemoryEffectsAllocateGet();
+  MlirMemoryEffect free = mlirMemoryEffectsFreeGet();
+  MlirMemoryEffect read = mlirMemoryEffectsReadGet();
+  MlirMemoryEffect write = mlirMemoryEffectsWriteGet();
+  MlirSideEffectResource defaultResource = mlirSideEffectsDefaultResourceGet();
+  if (!allocate.ptr || !free.ptr || !read.ptr || !write.ptr ||
+      !defaultResource.ptr) {
+    fprintf(stderr, "ERROR: Expected memory effect components\n");
+    return 6;
+  }
+
+  MlirAttribute nullParameters = {NULL};
+  MlirOpOperand opOperand = mlirOperationGetOpOperand(storeOp, 0);
+  MlirBlock block = mlirBlockCreate(1, &i32, &loc);
+  MlirValue blockArgument = mlirBlockGetArgument(block, 0);
+  MlirAttribute symbol = mlirFlatSymbolRefAttrGet(
+      ctx, mlirStringRefCreateFromCString("effect_target"));
+
+  MlirMemoryEffectInstance instances[] = {
+      mlirMemoryEffectInstanceCreate(allocate, nullParameters, 0, false,
+                                     defaultResource),
+      mlirMemoryEffectInstanceCreateForOpOperand(read, opOperand, zero, 1,
+                                                 false, defaultResource),
+      mlirMemoryEffectInstanceCreateForOpResult(
+          write, constantResult, nullParameters, 2, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForBlockArgument(free, blockArgument, zero,
+                                                     3, false, defaultResource),
+      mlirMemoryEffectInstanceCreateForSymbol(read, symbol, zero, 4, true,
+                                              defaultResource),
+  };
+  for (intptr_t i = 0; i < 5; ++i) {
+    if (!instances[i].ptr) {
+      fprintf(stderr, "ERROR: Expected memory effect instance\n");
+      return 7;
+    }
+    mlirMemoryEffectInstanceDestroy(instances[i]);
+  }
+  mlirBlockDestroy(block);
+  fprintf(stderr, "memory effect instances constructed\n");
+  // CHECK: memory effect instances constructed
+
   mlirOperationDestroy(storeOp);
   mlirOperationDestroy(constantOp);
   return 0;
diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
new file mode 100644
index 0000000000000..5689130c584b5
--- /dev/null
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -0,0 +1,291 @@
+# RUN: env PYTHONUNBUFFERED=1 %PYTHON %s 2>&1 | FileCheck %s
+
+from typing import Any
+
+from mlir import ir
+from mlir.dialects import ext, func
+from mlir.passmanager import PassManager
+
+
+class MemoryEffectsTest(ext.Dialect, name="memory_effects_test"):
+    pass
+
+
+class NoEffectModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        pass
+
+
+class ReadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(
+            ir.MemoryEffect.read,
+            op.op_operands[0],
+            parameters=ir.StringAttr.get("read parameter"),
+            stage=1,
+            effect_on_full_region=True,
+            resource=ir.SideEffectResource.default,
+        )
+
+
+class ReadDeadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read)
+
+
+class WriteModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.write)
+
+
+class FreeModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.free)
+
+
+class AllocateModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate)
+
+
+class AllocateResultModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate, op.results[0])
+
+
+class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+
+
+class SymbolTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        try:
+            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+        except TypeError as error:
+            print("invalid symbol target:", error)
+        try:
+            effects.append(ir.MemoryEffect.read, parameters=42)
+        except TypeError as error:
+            print("invalid parameters:", error)
+        effects.append(
+            ir.MemoryEffect.read,
+            ir.FlatSymbolRefAttr.get("global"),
+            parameters=ir.StringAttr.get("symbol parameter"),
+            stage=2,
+            effect_on_full_region=True,
+        )
+
+
+class ReadOp(MemoryEffectsTest.Operation, name="read", traits=[ReadModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteOp(MemoryEffectsTest.Operation, name="write", traits=[WriteModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteBarrierOp(
+    MemoryEffectsTest.Operation, name="write_barrier", traits=[WriteModel]
+):
+    operand: ext.Operand[Any]
+
+
+class NoEffectOp(MemoryEffectsTest.Operation, name="no_effect", traits=[NoEffectModel]):
+    pass
+
+
+class ReadDeadOp(MemoryEffectsTest.Operation, name="read_dead", traits=[ReadDeadModel]):
+    pass
+
+
+class WriteDeadOp(MemoryEffectsTest.Operation, name="write_dead", traits=[WriteModel]):
+    pass
+
+
+class FreeDeadOp(MemoryEffectsTest.Operation, name="free_dead", traits=[FreeModel]):
+    pass
+
+
+class AllocateDeadOp(
+    MemoryEffectsTest.Operation, name="allocate_dead", traits=[AllocateModel]
+):
+    pass
+
+
+class AllocateResultOp(
+    MemoryEffectsTest.Operation,
+    name="allocate_result",
+    traits=[AllocateResultModel],
+):
+    result: ext.Result[Any]
+
+
+class BlockArgumentTargetOp(
+    MemoryEffectsTest.Operation,
+    name="block_argument_target",
+    traits=[ir.NoTerminatorTrait, BlockArgumentTargetModel],
+):
+    body: ext.Region
+
+
+class SymbolTargetOp(
+    MemoryEffectsTest.Operation, name="symbol_target", traits=[SymbolTargetModel]
+):
+    pass
+
+
+def run_pass(source, pipeline):
+    module = ir.Module.parse(source)
+    PassManager.parse(pipeline).run(module.operation)
+    return str(module)
+
+
+with ir.Context(), ir.Location.unknown():
+    MemoryEffectsTest.load()
+
+    # The static properties return wrappers around borrowed, statically-owned
+    # C++ singletons.
+    # CHECK: memory effect properties: True True True True
+    print(
+        "memory effect properties:",
+        isinstance(ir.MemoryEffect.allocate, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.free, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.read, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.write, ir.MemoryEffect),
+    )
+    # CHECK: default resource property: True
+    print(
+        "default resource property:",
+        isinstance(ir.SideEffectResource.default, ir.SideEffectResource),
+    )
+
+    read_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A single Read effect remains CSE-eligible.
+    # CHECK: CSE read count: 1
+    print("CSE read count:", read_cse.count('"memory_effects_test.read"'))
+
+    write_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # Writes cannot be CSE'd.
+    # CHECK: CSE write count: 2
+    print("CSE write count:", write_cse.count('"memory_effects_test.write"'))
+
+    read_across_write = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            "memory_effects_test.write_barrier"(%arg0) : (i32) -> ()
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A potentially-aliasing Write on the default resource blocks Read CSE.
+    # CHECK: CSE read across write count: 2
+    print(
+        "CSE read across write count:",
+        read_across_write.count('"memory_effects_test.read"'),
+    )
+
+    dead_code = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.no_effect"() : () -> ()
+            "memory_effects_test.read_dead"() : () -> ()
+            "memory_effects_test.write_dead"() : () -> ()
+            "memory_effects_test.free_dead"() : () -> ()
+            "memory_effects_test.allocate_dead"() : () -> ()
+            %0 = "memory_effects_test.allocate_result"() : () -> i32
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # Empty and Read-only effect lists are dead. Write, Free and untargeted
+    # Allocate effects are observable. An Allocate targeting its own unused
+    # result is dead.
+    # CHECK: DCE no effect count: 0
+    # CHECK: DCE read count: 0
+    # CHECK: DCE write count: 1
+    # CHECK: DCE free count: 1
+    # CHECK: DCE untargeted allocate count: 1
+    # CHECK: DCE result allocate count: 0
+    print("DCE no effect count:", dead_code.count('"memory_effects_test.no_effect"'))
+    print("DCE read count:", dead_code.count('"memory_effects_test.read_dead"'))
+    print("DCE write count:", dead_code.count('"memory_effects_test.write_dead"'))
+    print("DCE free count:", dead_code.count('"memory_effects_test.free_dead"'))
+    print(
+        "DCE untargeted allocate count:",
+        dead_code.count('"memory_effects_test.allocate_dead"'),
+    )
+    print(
+        "DCE result allocate count:",
+        dead_code.count('"memory_effects_test.allocate_result"'),
+    )
+
+    target_variants = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.block_argument_target"() ({
+            ^bb0(%arg0: i32):
+            }) : () -> ()
+            "memory_effects_test.symbol_target"() : () -> ()
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # These Read effects exercise BlockArgument and SymbolRefAttr targets and
+    # remain removable by trivial-dce.
+    # CHECK: invalid symbol target: target Attribute must be a SymbolRefAttr
+    # CHECK: invalid parameters: parameters must be an Attribute or None
+    # CHECK: DCE block argument target count: 0
+    # CHECK: DCE symbol target count: 0
+    print(
+        "DCE block argument target count:",
+        target_variants.count('"memory_effects_test.block_argument_target"'),
+    )
+    print(
+        "DCE symbol target count:",
+        target_variants.count('"memory_effects_test.symbol_target"'),
+    )

>From ae347ca90903a7a23e8cd26d39019ad5cd75afa9 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 22:56:28 +0800
Subject: [PATCH 2/4] fix

---
 mlir/test/python/dialects/memory_effects_op_interface.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
index 5689130c584b5..cb043cb961bca 100644
--- a/mlir/test/python/dialects/memory_effects_op_interface.py
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -155,8 +155,6 @@ def run_pass(source, pipeline):
 with ir.Context(), ir.Location.unknown():
     MemoryEffectsTest.load()
 
-    # The static properties return wrappers around borrowed, statically-owned
-    # C++ singletons.
     # CHECK: memory effect properties: True True True True
     print(
         "memory effect properties:",

>From 90d24dbc6e65f26f5d0b5824ec0cfe5abecb142a Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Thu, 23 Jul 2026 23:34:23 +0800
Subject: [PATCH 3/4] address comments

---
 .../mlir/Bindings/Python/IRInterfaces.h       |  7 ++--
 mlir/lib/Bindings/Python/DialectTransform.cpp | 13 +++-----
 mlir/lib/Bindings/Python/IRInterfaces.cpp     | 10 +++---
 .../dialects/memory_effects_op_interface.py   | 32 +++++++++----------
 4 files changed, 30 insertions(+), 32 deletions(-)

diff --git a/mlir/include/mlir/Bindings/Python/IRInterfaces.h b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
index 1ba995aa82c5e..c1047c7e20ab7 100644
--- a/mlir/include/mlir/Bindings/Python/IRInterfaces.h
+++ b/mlir/include/mlir/Bindings/Python/IRInterfaces.h
@@ -135,7 +135,7 @@ class PyConcreteOpInterface {
   nanobind::object obj;
 };
 
-/// A borrowed memory effect.
+/// A memory effect.
 class PyMemoryEffect {
 public:
   explicit PyMemoryEffect(MlirMemoryEffect effect) : effect(effect) {}
@@ -146,7 +146,7 @@ class PyMemoryEffect {
   MlirMemoryEffect effect;
 };
 
-/// A borrowed side effect resource.
+/// A side effect resource.
 class PySideEffectResource {
 public:
   explicit PySideEffectResource(MlirSideEffectResource resource)
@@ -158,7 +158,7 @@ class PySideEffectResource {
   MlirSideEffectResource resource;
 };
 
-/// An owning memory effect instance.
+/// A memory effect instance.
 class PyMemoryEffectInstance {
 public:
   explicit PyMemoryEffectInstance(MlirMemoryEffectInstance instance)
@@ -185,6 +185,7 @@ class PyMemoryEffectsInstanceList {
       : effects(effects) {}
 
   MlirMemoryEffectInstancesList get() const { return effects; }
+  operator MlirMemoryEffectInstancesList() const { return effects; }
 
 private:
   MlirMemoryEffectInstancesList effects;
diff --git a/mlir/lib/Bindings/Python/DialectTransform.cpp b/mlir/lib/Bindings/Python/DialectTransform.cpp
index f84189e3a22ff..a4a91b28522cf 100644
--- a/mlir/lib/Bindings/Python/DialectTransform.cpp
+++ b/mlir/lib/Bindings/Python/DialectTransform.cpp
@@ -498,8 +498,7 @@ void onlyReadsHandle(nb::iterable &operands,
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
-  mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(),
-                               effects.get());
+  mlirTransformOnlyReadsHandle(operandsVec.data(), operandsVec.size(), effects);
 };
 
 void consumesHandle(nb::iterable &operands,
@@ -507,8 +506,7 @@ void consumesHandle(nb::iterable &operands,
   std::vector<MlirOpOperand> operandsVec;
   for (auto operand : operands)
     operandsVec.push_back(nb::cast<PyOpOperand>(operand));
-  mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(),
-                              effects.get());
+  mlirTransformConsumesHandle(operandsVec.data(), operandsVec.size(), effects);
 };
 
 void producesHandle(nb::iterable &results,
@@ -516,16 +514,15 @@ void producesHandle(nb::iterable &results,
   std::vector<MlirValue> resultsVec;
   for (auto result : results)
     resultsVec.push_back(nb::cast<PyOpResult>(result).get());
-  mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(),
-                              effects.get());
+  mlirTransformProducesHandle(resultsVec.data(), resultsVec.size(), effects);
 };
 
 void modifiesPayload(const PyMemoryEffectsInstanceList &effects) {
-  mlirTransformModifiesPayload(effects.get());
+  mlirTransformModifiesPayload(effects);
 }
 
 void onlyReadsPayload(const PyMemoryEffectsInstanceList &effects) {
-  mlirTransformOnlyReadsPayload(effects.get());
+  mlirTransformOnlyReadsPayload(effects);
 }
 } // namespace
 
diff --git a/mlir/lib/Bindings/Python/IRInterfaces.cpp b/mlir/lib/Bindings/Python/IRInterfaces.cpp
index 4495cb6e8397d..d16015acaf3dc 100644
--- a/mlir/lib/Bindings/Python/IRInterfaces.cpp
+++ b/mlir/lib/Bindings/Python/IRInterfaces.cpp
@@ -539,26 +539,26 @@ void populateIRInterfaces(nb::module_ &m) {
       .value("RecursivelySpeculatable",
              MlirSpeculatabilityRecursivelySpeculatable);
   nb::class_<PyMemoryEffect>(m, "MemoryEffect", "A memory effect.")
-      .def_prop_ro_static("allocate",
+      .def_prop_ro_static("Allocate",
                           [](nb::object & /*class*/) {
                             return PyMemoryEffect(
                                 mlirMemoryEffectsAllocateGet());
                           })
-      .def_prop_ro_static("free",
+      .def_prop_ro_static("Free",
                           [](nb::object & /*class*/) {
                             return PyMemoryEffect(mlirMemoryEffectsFreeGet());
                           })
-      .def_prop_ro_static("read",
+      .def_prop_ro_static("Read",
                           [](nb::object & /*class*/) {
                             return PyMemoryEffect(mlirMemoryEffectsReadGet());
                           })
-      .def_prop_ro_static("write", [](nb::object & /*class*/) {
+      .def_prop_ro_static("Write", [](nb::object & /*class*/) {
         return PyMemoryEffect(mlirMemoryEffectsWriteGet());
       });
 
   nb::class_<PySideEffectResource>(m, "SideEffectResource",
                                    "A side effect resource.")
-      .def_prop_ro_static("default", [](nb::object & /*class*/) {
+      .def_prop_ro_static("Default", [](nb::object & /*class*/) {
         return PySideEffectResource(mlirSideEffectsDefaultResourceGet());
       });
 
diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
index cb043cb961bca..3e6a1b58c55b0 100644
--- a/mlir/test/python/dialects/memory_effects_op_interface.py
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -21,64 +21,64 @@ class ReadModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
         effects.append(
-            ir.MemoryEffect.read,
+            ir.MemoryEffect.Read,
             op.op_operands[0],
             parameters=ir.StringAttr.get("read parameter"),
             stage=1,
             effect_on_full_region=True,
-            resource=ir.SideEffectResource.default,
+            resource=ir.SideEffectResource.Default,
         )
 
 
 class ReadDeadModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.read)
+        effects.append(ir.MemoryEffect.Read)
 
 
 class WriteModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.write)
+        effects.append(ir.MemoryEffect.Write)
 
 
 class FreeModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.free)
+        effects.append(ir.MemoryEffect.Free)
 
 
 class AllocateModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.allocate)
+        effects.append(ir.MemoryEffect.Allocate)
 
 
 class AllocateResultModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.allocate, op.results[0])
+        effects.append(ir.MemoryEffect.Allocate, op.results[0])
 
 
 class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
-        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+        effects.append(ir.MemoryEffect.Read, op.regions[0].blocks[0].arguments[0])
 
 
 class SymbolTargetModel(ir.MemoryEffectsOpInterface):
     @staticmethod
     def get_effects(op, effects):
         try:
-            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+            effects.append(ir.MemoryEffect.Read, ir.StringAttr.get("not a symbol"))
         except TypeError as error:
             print("invalid symbol target:", error)
         try:
-            effects.append(ir.MemoryEffect.read, parameters=42)
+            effects.append(ir.MemoryEffect.Read, parameters=42)
         except TypeError as error:
             print("invalid parameters:", error)
         effects.append(
-            ir.MemoryEffect.read,
+            ir.MemoryEffect.Read,
             ir.FlatSymbolRefAttr.get("global"),
             parameters=ir.StringAttr.get("symbol parameter"),
             stage=2,
@@ -158,15 +158,15 @@ def run_pass(source, pipeline):
     # CHECK: memory effect properties: True True True True
     print(
         "memory effect properties:",
-        isinstance(ir.MemoryEffect.allocate, ir.MemoryEffect),
-        isinstance(ir.MemoryEffect.free, ir.MemoryEffect),
-        isinstance(ir.MemoryEffect.read, ir.MemoryEffect),
-        isinstance(ir.MemoryEffect.write, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.Allocate, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.Free, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.Read, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.Write, ir.MemoryEffect),
     )
     # CHECK: default resource property: True
     print(
         "default resource property:",
-        isinstance(ir.SideEffectResource.default, ir.SideEffectResource),
+        isinstance(ir.SideEffectResource.Default, ir.SideEffectResource),
     )
 
     read_cse = run_pass(

>From a03c2f97ecc374ecb9563dcbbd62105d8b68c593 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Thu, 23 Jul 2026 23:47:47 +0800
Subject: [PATCH 4/4] address claude comments

---
 mlir/test/python/dialects/memory_effects_op_interface.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/mlir/test/python/dialects/memory_effects_op_interface.py b/mlir/test/python/dialects/memory_effects_op_interface.py
index 3e6a1b58c55b0..dc3b744aae2d7 100644
--- a/mlir/test/python/dialects/memory_effects_op_interface.py
+++ b/mlir/test/python/dialects/memory_effects_op_interface.py
@@ -77,6 +77,10 @@ def get_effects(op, effects):
             effects.append(ir.MemoryEffect.Read, parameters=42)
         except TypeError as error:
             print("invalid parameters:", error)
+        try:
+            effects.append(ir.MemoryEffect.Read, 42)
+        except TypeError as error:
+            print("invalid target:", error)
         effects.append(
             ir.MemoryEffect.Read,
             ir.FlatSymbolRefAttr.get("global"),
@@ -277,6 +281,7 @@ def run_pass(source, pipeline):
     # remain removable by trivial-dce.
     # CHECK: invalid symbol target: target Attribute must be a SymbolRefAttr
     # CHECK: invalid parameters: parameters must be an Attribute or None
+    # CHECK: invalid target: target must be an OpOperand, OpResult, BlockArgument, SymbolRefAttr, or None
     # CHECK: DCE block argument target count: 0
     # CHECK: DCE symbol target count: 0
     print(


From llvmlistbot at llvm.org  Thu Jul 23 08:48:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:48:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a6237d9.c4d3d80a.16dae5.fc5e@mx.google.com>


================
@@ -0,0 +1,289 @@
+# RUN: env PYTHONUNBUFFERED=1 %PYTHON %s 2>&1 | FileCheck %s
+
+from typing import Any
+
+from mlir import ir
+from mlir.dialects import ext, func
+from mlir.passmanager import PassManager
+
+
+class MemoryEffectsTest(ext.Dialect, name="memory_effects_test"):
+    pass
+
+
+class NoEffectModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        pass
+
+
+class ReadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(
+            ir.MemoryEffect.read,
+            op.op_operands[0],
+            parameters=ir.StringAttr.get("read parameter"),
+            stage=1,
+            effect_on_full_region=True,
+            resource=ir.SideEffectResource.default,
+        )
+
+
+class ReadDeadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read)
+
+
+class WriteModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.write)
+
+
+class FreeModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.free)
+
+
+class AllocateModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate)
+
+
+class AllocateResultModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate, op.results[0])
+
+
+class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+
+
+class SymbolTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        try:
+            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+        except TypeError as error:
+            print("invalid symbol target:", error)
+        try:
+            effects.append(ir.MemoryEffect.read, parameters=42)
+        except TypeError as error:
+            print("invalid parameters:", error)
+        effects.append(
+            ir.MemoryEffect.read,
+            ir.FlatSymbolRefAttr.get("global"),
+            parameters=ir.StringAttr.get("symbol parameter"),
+            stage=2,
+            effect_on_full_region=True,
+        )
+
+
+class ReadOp(MemoryEffectsTest.Operation, name="read", traits=[ReadModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteOp(MemoryEffectsTest.Operation, name="write", traits=[WriteModel]):
+    operand: ext.Operand[Any]
+    result: ext.Result[Any]
+
+
+class WriteBarrierOp(
+    MemoryEffectsTest.Operation, name="write_barrier", traits=[WriteModel]
+):
+    operand: ext.Operand[Any]
+
+
+class NoEffectOp(MemoryEffectsTest.Operation, name="no_effect", traits=[NoEffectModel]):
+    pass
+
+
+class ReadDeadOp(MemoryEffectsTest.Operation, name="read_dead", traits=[ReadDeadModel]):
+    pass
+
+
+class WriteDeadOp(MemoryEffectsTest.Operation, name="write_dead", traits=[WriteModel]):
+    pass
+
+
+class FreeDeadOp(MemoryEffectsTest.Operation, name="free_dead", traits=[FreeModel]):
+    pass
+
+
+class AllocateDeadOp(
+    MemoryEffectsTest.Operation, name="allocate_dead", traits=[AllocateModel]
+):
+    pass
+
+
+class AllocateResultOp(
+    MemoryEffectsTest.Operation,
+    name="allocate_result",
+    traits=[AllocateResultModel],
+):
+    result: ext.Result[Any]
+
+
+class BlockArgumentTargetOp(
+    MemoryEffectsTest.Operation,
+    name="block_argument_target",
+    traits=[ir.NoTerminatorTrait, BlockArgumentTargetModel],
+):
+    body: ext.Region
+
+
+class SymbolTargetOp(
+    MemoryEffectsTest.Operation, name="symbol_target", traits=[SymbolTargetModel]
+):
+    pass
+
+
+def run_pass(source, pipeline):
+    module = ir.Module.parse(source)
+    PassManager.parse(pipeline).run(module.operation)
+    return str(module)
+
+
+with ir.Context(), ir.Location.unknown():
+    MemoryEffectsTest.load()
+
+    # CHECK: memory effect properties: True True True True
+    print(
+        "memory effect properties:",
+        isinstance(ir.MemoryEffect.allocate, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.free, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.read, ir.MemoryEffect),
+        isinstance(ir.MemoryEffect.write, ir.MemoryEffect),
+    )
+    # CHECK: default resource property: True
+    print(
+        "default resource property:",
+        isinstance(ir.SideEffectResource.default, ir.SideEffectResource),
+    )
+
+    read_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A single Read effect remains CSE-eligible.
+    # CHECK: CSE read count: 1
+    print("CSE read count:", read_cse.count('"memory_effects_test.read"'))
+
+    write_cse = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            %1 = "memory_effects_test.write"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # Writes cannot be CSE'd.
+    # CHECK: CSE write count: 2
+    print("CSE write count:", write_cse.count('"memory_effects_test.write"'))
+
+    read_across_write = run_pass(
+        """
+        module {
+          func.func @test(%arg0: i32) -> (i32, i32) {
+            %0 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            "memory_effects_test.write_barrier"(%arg0) : (i32) -> ()
+            %1 = "memory_effects_test.read"(%arg0) : (i32) -> i32
+            return %0, %1 : i32, i32
+          }
+        }
+        """,
+        "builtin.module(func.func(cse))",
+    )
+    # A potentially-aliasing Write on the default resource blocks Read CSE.
+    # CHECK: CSE read across write count: 2
+    print(
+        "CSE read across write count:",
+        read_across_write.count('"memory_effects_test.read"'),
+    )
+
+    dead_code = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.no_effect"() : () -> ()
+            "memory_effects_test.read_dead"() : () -> ()
+            "memory_effects_test.write_dead"() : () -> ()
+            "memory_effects_test.free_dead"() : () -> ()
+            "memory_effects_test.allocate_dead"() : () -> ()
+            %0 = "memory_effects_test.allocate_result"() : () -> i32
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # Empty and Read-only effect lists are dead. Write, Free and untargeted
+    # Allocate effects are observable. An Allocate targeting its own unused
+    # result is dead.
+    # CHECK: DCE no effect count: 0
+    # CHECK: DCE read count: 0
+    # CHECK: DCE write count: 1
+    # CHECK: DCE free count: 1
+    # CHECK: DCE untargeted allocate count: 1
+    # CHECK: DCE result allocate count: 0
+    print("DCE no effect count:", dead_code.count('"memory_effects_test.no_effect"'))
+    print("DCE read count:", dead_code.count('"memory_effects_test.read_dead"'))
+    print("DCE write count:", dead_code.count('"memory_effects_test.write_dead"'))
+    print("DCE free count:", dead_code.count('"memory_effects_test.free_dead"'))
+    print(
+        "DCE untargeted allocate count:",
+        dead_code.count('"memory_effects_test.allocate_dead"'),
+    )
+    print(
+        "DCE result allocate count:",
+        dead_code.count('"memory_effects_test.allocate_result"'),
+    )
+
+    target_variants = run_pass(
+        """
+        module {
+          func.func @test() {
+            "memory_effects_test.block_argument_target"() ({
+            ^bb0(%arg0: i32):
+            }) : () -> ()
+            "memory_effects_test.symbol_target"() : () -> ()
+            return
+          }
+        }
+        """,
+        "builtin.module(func.func(trivial-dce))",
+    )
+    # These Read effects exercise BlockArgument and SymbolRefAttr targets and
+    # remain removable by trivial-dce.
+    # CHECK: invalid symbol target: target Attribute must be a SymbolRefAttr
+    # CHECK: invalid parameters: parameters must be an Attribute or None
----------------
PragmaTwice wrote:

Done.

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Thu Jul 23 08:48:48 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:48:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][CAPI][Python] Add support for
 constructing memory effect instances (PR #210586)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210586@github.com>
Message-ID: <6a6237e0.eb165e5c.2e0c9a.b220@mx.google.com>


================
@@ -0,0 +1,289 @@
+# RUN: env PYTHONUNBUFFERED=1 %PYTHON %s 2>&1 | FileCheck %s
+
+from typing import Any
+
+from mlir import ir
+from mlir.dialects import ext, func
+from mlir.passmanager import PassManager
+
+
+class MemoryEffectsTest(ext.Dialect, name="memory_effects_test"):
+    pass
+
+
+class NoEffectModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        pass
+
+
+class ReadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(
+            ir.MemoryEffect.read,
+            op.op_operands[0],
+            parameters=ir.StringAttr.get("read parameter"),
+            stage=1,
+            effect_on_full_region=True,
+            resource=ir.SideEffectResource.default,
+        )
+
+
+class ReadDeadModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read)
+
+
+class WriteModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.write)
+
+
+class FreeModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.free)
+
+
+class AllocateModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate)
+
+
+class AllocateResultModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.allocate, op.results[0])
+
+
+class BlockArgumentTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        effects.append(ir.MemoryEffect.read, op.regions[0].blocks[0].arguments[0])
+
+
+class SymbolTargetModel(ir.MemoryEffectsOpInterface):
+    @staticmethod
+    def get_effects(op, effects):
+        try:
+            effects.append(ir.MemoryEffect.read, ir.StringAttr.get("not a symbol"))
+        except TypeError as error:
+            print("invalid symbol target:", error)
+        try:
+            effects.append(ir.MemoryEffect.read, parameters=42)
+        except TypeError as error:
+            print("invalid parameters:", error)
----------------
PragmaTwice wrote:

Added.

https://github.com/llvm/llvm-project/pull/210586

From llvmlistbot at llvm.org  Thu Jul 23 08:56:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:56:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a623996.666420a4.3304c7.915c@mx.google.com>


https://github.com/PragmaTwice updated https://github.com/llvm/llvm-project/pull/210501

>From cd6ecf25efeece948522106a74e83d711c973027 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 18 Jul 2026 18:37:27 +0800
Subject: [PATCH 1/3] [MLIR][Python] Make Python-defined dialect loading
 context-aware

---
 mlir/include/mlir-c/IR.h                      |  7 ++
 mlir/lib/Bindings/Python/IRCore.cpp           |  8 +++
 mlir/lib/CAPI/IR/IR.cpp                       |  5 ++
 mlir/python/mlir/dialects/ext.py              | 29 ++++----
 mlir/test/python/dialects/ext.py              | 66 +++++++++++++++++++
 .../python/dialects/transform_op_interface.py |  2 +-
 ...ansform_pattern_descriptor_op_interface.py |  2 +-
 7 files changed, 100 insertions(+), 19 deletions(-)

diff --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..98ea0dfc00a1b 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                           MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..9ec6d048dbadf 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
diff --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }
diff --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..b4a392ff5cb36 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -957,18 +957,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a different context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise RuntimeError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +973,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..6791cbe7fde65 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,72 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            print("same context:", e)
+        else:
+            raise AssertionError("loading a dialect twice in one context must fail")
+
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        check_dialect("second context", 2)
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():
diff --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)
diff --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.

>From b96aa343d82a203ecec263e7b291aea0a792c24e Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sun, 19 Jul 2026 14:21:12 +0800
Subject: [PATCH 2/3] address comment

---
 mlir/test/python/dialects/ext.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 6791cbe7fde65..707f86aaac521 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -121,8 +121,6 @@ class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
         attr: ContextLoadAttr
         result: Result[ContextLoadType]
 
-    # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
-
     def check_dialect(context_name, type_value):
         i32 = IntegerType.get_signless(32)
         result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
@@ -139,12 +137,6 @@ def check_dialect(context_name, type_value):
         assert isinstance(op.attr, ContextLoadAttr)
         assert isinstance(op.result.type, ContextLoadType)
 
-        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
-        # CHECK: "first context"
-        # CHECK: 1 : i32
-        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
-        # CHECK: "second context"
-        # CHECK: 2 : i32
         print(
             f"{context_name}: {type(op).__name__}, "
             f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
@@ -160,16 +152,28 @@ def check_dialect(context_name, type_value):
         try:
             ContextLoadDialect.load()
         except RuntimeError as e:
+            # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
             print("same context:", e)
-        else:
-            raise AssertionError("loading a dialect twice in one context must fail")
 
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
         check_dialect("first context", 1)
 
     with second_context, Location.unknown():
         ContextLoadDialect.load()
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
         check_dialect("second context", 2)
 
+    with first_context, Location.unknown():
+        try:
+            ContextLoadDialect.load()
+        except RuntimeError as e:
+            # CHECK: same context again: Dialect 'ext_context_load' has already been loaded in the current context.
+            print("same context again:", e)
+
 
 # CHECK: TEST: testExtDialect
 @run

>From 8a15f38b86b5feadb2b2c924eabaf3bd9ea21090 Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Thu, 23 Jul 2026 23:55:45 +0800
Subject: [PATCH 3/3] remove useless = in lambda capture list

---
 mlir/lib/Bindings/Python/IRCore.cpp | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index 9ec6d048dbadf..cceef55f9b15d 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3397,7 +3397,7 @@ void populateIRCore(nb::module_ &m) {
           "Alias for `dialects`.")
       .def(
           "get_dialect_descriptor",
-          [=](PyMlirContext &self, std::string &name) {
+          [](PyMlirContext &self, std::string &name) {
             MlirDialect dialect = mlirContextGetOrLoadDialect(
                 self.get(), {name.data(), name.size()});
             if (mlirDialectIsNull(dialect)) {
@@ -3410,7 +3410,7 @@ void populateIRCore(nb::module_ &m) {
           "Gets or loads a dialect by name, returning its descriptor object.")
       .def(
           "is_dialect_loaded",
-          [=](PyMlirContext &self, std::string &name) {
+          [](PyMlirContext &self, std::string &name) {
             MlirDialect dialect = mlirContextGetLoadedDialect(
                 self.get(), {name.data(), name.size()});
             return !mlirDialectIsNull(dialect);
@@ -3545,7 +3545,7 @@ void populateIRCore(nb::module_ &m) {
   nb::class_<PyDialects>(m, "Dialects")
       .def(
           "__getitem__",
-          [=](PyDialects &self, std::string keyName) {
+          [](PyDialects &self, std::string keyName) {
             MlirDialect dialect =
                 self.getDialectForKey(keyName, /*attrError=*/false);
             nb::object descriptor =
@@ -3555,7 +3555,7 @@ void populateIRCore(nb::module_ &m) {
           "Gets a dialect by name using subscript notation.")
       .def(
           "__getattr__",
-          [=](PyDialects &self, std::string attrName) {
+          [](PyDialects &self, std::string attrName) {
             MlirDialect dialect =
                 self.getDialectForKey(attrName, /*attrError=*/true);
             nb::object descriptor =


From llvmlistbot at llvm.org  Thu Jul 23 08:56:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 08:56:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a6239a7.492e23e4.292f20.42aa@mx.google.com>


================
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [=](PyMlirContext &self, std::string &name) {
----------------
PragmaTwice wrote:

Done in https://github.com/llvm/llvm-project/pull/210501/commits/8a15f38b86b5feadb2b2c924eabaf3bd9ea21090.

https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Thu Jul 23 08:57:13 2026
From: llvmlistbot at llvm.org (Luke Hutton)
Date: Thu, 23 Jul 2026 08:57:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Add more informative error
 messages to block scaled types (PR #211569)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211569@github.com>
Message-ID: <6a6239d9.d681cab9.33af1e.46bb@mx.google.com>


lhutton1 wrote:

cc @iliyan-georgiev-arm @sahas3 @udaya-ranga 

https://github.com/llvm/llvm-project/pull/211569

From llvmlistbot at llvm.org  Thu Jul 23 11:48:41 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 23 Jul 2026 11:48:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a626209.492e23e4.292f20.7403@mx.google.com>


================

----------------
Jianhui-Li wrote:

inferMaskOffsetLayoutForScatterIO can be removed once the chunksize is away. Mask/offset layout should be same as value now. 

https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Thu Jul 23 11:48:41 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 23 Jul 2026 11:48:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a626209.5a0412d2.325e39.9d02@mx.google.com>


================

----------------
Jianhui-Li wrote:

Only need to use one offset/mask and the first one for one load. 
In case lane_data > 1, the offset must be contiguous, and mask must be constant "true" for the whole vector being load. 

https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Thu Jul 23 11:48:42 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 23 Jul 2026 11:48:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Remove chunk_size attribute;
 infer from types (PR #205122)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/205122@github.com>
Message-ID: <6a62620a.1ba19891.1d5373.c83f@mx.google.com>


================
@@ -237,15 +251,17 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
 
     if (isa<xegpu::StoreScatterOp>(op)) {
       xegpu::StoreScatterOp store(op);
-      int chunkSize = store.getChunkSize().value_or(1);
+      int chunkSize = xegpu::getGatherScatterPayloadChunk(store.getValueType(),
+                                                          store.getMaskType());
       if (layout && idx >= 2 && chunkSize > 1)
         return layout.dropDims(llvm::to_vector(
----------------
Jianhui-Li wrote:

remove these special processing. the mask/offset have the same shape as data now. 

https://github.com/llvm/llvm-project/pull/205122

From llvmlistbot at llvm.org  Thu Jul 23 12:47:52 2026
From: llvmlistbot at llvm.org (Vijay Kandiah)
Date: Thu, 23 Jul 2026 12:47:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Emit multi-IV tile and
 element loops from ACCLoopTiling (PR #211651)
Message-ID: <llvm.org/llvm/llvm-project/pull/211651@github.com>


https://github.com/VijayKandiah created https://github.com/llvm/llvm-project/pull/211651

`ACCLoopTiling` used to lower an N-dimensional `tile()` clause by *uncollapsing* the fused loop into a deep nest of single-IV `acc.loop`s (`tile_1 → … → tile_N → elem_1 → … → elem_N`). This discards the natural grouping of the tile clause: the 2N single-IV loops no longer express "these are the tile iterations" and "these are the in-tile iterations" as collapsible units, and gang/vector end up spread across a deep nest where only the outermost loop of each group carries the attribute. This could potentially lead to poor parallelism assignment.

With this MR:
`tileACCLoops` now rewrites the single fused `acc.loop` in-place into exactly two multi-IV loops, each carrying all the tiled induction variables:
- a **tile-group** loop (gang) whose steps are the original steps scaled by the tile sizes, and
- a nested **element-group** loop (vector) that walks the iterations within one tile, with upper bounds clamped to `min(origUB, tileStart + tileExtent)`.

Keeping each group as a single multi-IV `acc.loop` preserves the tile/element structure as two collapsible units instead of a 2N-deep single-IV nest, which is a simpler canonical form. The gang/worker/vector distribution across the two groups is unchanged from before.

This change also diagnoses `tile` + `collapse` on the same loop because these clauses give conflicting loop-association counts. The pass now emits a NYI diagnostic instead of silently dropping the `collapse` clause. Added a new invalid-input test to verify this.

With the pass no longer uncollapsing, `uncollapseLoops` and the vector-of-loops `tileACCLoops(SmallVector<LoopOp>&, …)` overload (and the now-orphaned `createInnerLoop` helper) are deleted, along with their unit tests. New unit tests are added to cover the surviving fused-loop overload.


>From 3cb4151c7f60f5549dce86110583ed8cbb5f85e0 Mon Sep 17 00:00:00 2001
From: Vijay Kandiah <vkandiah at nvidia.com>
Date: Thu, 23 Jul 2026 12:38:02 -0700
Subject: [PATCH] [mlir][OpenACC] Emit multi-IV tile/element loops from
 ACCLoopTiling

---
 .../mlir/Dialect/OpenACC/OpenACCUtilsTiling.h |  53 ++-
 .../OpenACC/Transforms/ACCLoopTiling.cpp      |  32 +-
 .../OpenACC/Utils/OpenACCUtilsTiling.cpp      | 345 +++++++-----------
 .../OpenACC/acc-loop-tiling-invalid.mlir      |  23 ++
 .../test/Dialect/OpenACC/acc-loop-tiling.mlir |  85 ++---
 .../OpenACC/OpenACCUtilsTilingTest.cpp        | 225 ++----------
 6 files changed, 253 insertions(+), 510 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
index 6fcb706aa3488..05ac37d1106ec 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
@@ -20,23 +20,20 @@
 namespace mlir {
 namespace acc {
 
-/// Uncollapse tile loops with multiple IVs and collapseCount < tileCount.
-/// This is used to prepare loops for tiling when the collapse count is less
-/// than the tile count.
+/// Tile a single fused acc.loop that carries all associated induction
+/// variables (one IV per tile dimension).
 ///
-/// \param origLoop The original loop operation to uncollapse.
-/// \param tileCount The number of tile dimensions.
-/// \param collapseCount The collapse count from the original loop.
-/// \param rewriter The rewriter to use for modifications.
-/// \return A vector of uncollapsed loop operations.
-llvm::SmallVector<mlir::acc::LoopOp>
-uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
-                unsigned collapseCount, mlir::RewriterBase &rewriter);
-
-/// Tile ACC loops according to the given tile sizes.
+/// This produces exactly two multi-IV loops, each carrying all of the tiled
+/// induction variables:
 ///
-/// Tiling a 2-level nested loop will create two 'tile' loops containing two
-/// 'element' loops. The transformation looks like:
+///   - a "tile group" loop that steps over tiles: each step is the
+///     original step multiplied by the tile size. It keeps the original loop's
+///     gang attribute.
+///   - an "element group" loop that walks the iterations inside one tile: it
+///     keeps the original step, its lower bound is the current tile's starting
+///     index, and its upper bound is clamped to min(original upper bound,
+///     tile start + tile extent). It keeps the original loop's vector (or
+///     worker) attribute.
 ///
 /// Before Tiling:
 /// \code
@@ -48,17 +45,15 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 ///  }
 /// \endcode
 ///
-/// After Tiling:
+/// After Tiling (each group is one multi-IV loop over all tiled IVs):
 /// \code
-///  for (i = lb1; i < ub1; i += (step1 * tile_size1)) { // tile loop 1
-///    for (j = lb2; j < ub2; j += (step2 * tile_size2)) { // tile loop 2
-///      for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1) {
-///      // element loop 1
-///        for (jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2)
-///        { // element loop 2
-///          a[ii,jj] = i + j;
-///        }
-///      }
+///  // tile group
+///  for (i = lb1; i < ub1; i += (step1 * tile_size1),
+///       j = lb2; j < ub2; j += (step2 * tile_size2)) {
+///    // element group
+///    for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1,
+///         jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2) {
+///      a[ii,jj] = i + j;
 ///    }
 ///  }
 /// \endcode
@@ -66,13 +61,13 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 /// Unknown tile sizes (represented as -1 in acc dialect for `tile(*)`) are
 /// resolved to the provided default tile size.
 ///
-/// \param tileLoops The loops to tile (outermost first).
-/// \param tileSizes The tile sizes for each dimension. Values of -1 are
+/// \param tileLoop The fused loop to tile.
+/// \param tileSizes The tile sizes for each tiled dimension. Values of -1 are
 ///        treated as unknown and resolved to defaultTileSize.
 /// \param defaultTileSize The default tile size to use for unknown (*) tiles.
 /// \param rewriter The rewriter to use for modifications.
-/// \return The outermost loop after tiling.
-mlir::acc::LoopOp tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+/// \return The tile group loop that is modified in place.
+mlir::acc::LoopOp tileACCLoops(mlir::acc::LoopOp tileLoop,
                                const llvm::SmallVector<mlir::Value> &tileSizes,
                                int32_t defaultTileSize,
                                mlir::RewriterBase &rewriter);
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
index 6bc95ca896f37..d5865bc98727e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
@@ -162,8 +162,16 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
 
     SmallVector<Value> tileSizes(origLoop.getTileValues().begin(),
                                  origLoop.getTileValues().end());
-    unsigned tileCount = tileSizes.size();
-    unsigned collapseCount = origLoop.getCollapseValue().value_or(1);
+
+    // A tile clause and a collapse clause on the same loop are not supported:
+    // they give conflicting descriptions of how many loops the construct
+    // associates.
+    if (origLoop.getCollapseAttr()) {
+      accSupport.emitNYI(origLoop.getLoc(),
+                         "a tile clause combined with a collapse clause on the "
+                         "same loop");
+      return failure();
+    }
 
     // Sanity check tile size types
     if (failed(checkTileSizeTypes(origLoop, tileSizes)))
@@ -182,24 +190,12 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
     origLoop.removeTileOperandsDeviceTypeAttr();
     rewriter.finalizeOpModification(origLoop);
 
-    SmallVector<acc::LoopOp> loopsToTile;
-    if (collapseCount < tileCount) {
-      // Uncollapse tile loops before tiling if necessary
-      loopsToTile =
-          acc::uncollapseLoops(origLoop, tileCount, collapseCount, rewriter);
-      rewriter.replaceOp(origLoop, loopsToTile[0]);
-      LLVM_DEBUG(llvm::dbgs() << "\nAfter uncollapsing:\n"
-                              << *loopsToTile[0] << "\n");
-    } else {
-      loopsToTile.push_back(origLoop);
-    }
-
-    // loopsToTile is a vector of perfectly nested loops. The outermost loop
-    // may have multiple IVs but inner loops can only have one IV.
+    // Tile the single fused loop in place: the tile iterations become one
+    // multi-IV "tile group" loop wrapping one multi-IV "element group" loop.
     // The utility handles unknown tile sizes (*) by using `defaultTileSize`.
-    acc::tileACCLoops(loopsToTile, tileSizes, defaultTileSize, rewriter);
+    acc::tileACCLoops(origLoop, tileSizes, defaultTileSize, rewriter);
 
-    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *loopsToTile[0] << "\n");
+    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *origLoop << "\n");
     return success();
   }
 
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index 09e83af2bc686..32fd3957894fb 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
 #include "mlir/Transforms/RegionUtils.h"
+#include "llvm/ADT/STLExtras.h"
 
 // Resolve unknown tile sizes (represented as -1 for tile(*)) to the default.
 // Returns a value with the same type as targetType.
@@ -48,19 +49,13 @@ static void removeWorkerVectorFromLoop(mlir::acc::LoopOp loop) {
 }
 
 // Create a new ACC loop with new steps, lb, ub from original loop
-static mlir::acc::LoopOp
-createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
-                          mlir::RewriterBase &rewriter, mlir::ValueRange lb,
-                          mlir::ValueRange ub, mlir::ValueRange step,
-                          mlir::DenseBoolArrayAttr inclusiveUBAttr,
-                          mlir::acc::CombinedConstructsTypeAttr combinedAttr,
-                          mlir::Location loc, bool preserveCollapse) {
+static mlir::acc::LoopOp createACCLoopFromOriginal(
+    mlir::acc::LoopOp origLoop, mlir::RewriterBase &rewriter,
+    mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
+    mlir::DenseBoolArrayAttr inclusiveUBAttr,
+    mlir::acc::CombinedConstructsTypeAttr combinedAttr, mlir::Location loc) {
   mlir::ArrayAttr collapseAttr = mlir::ArrayAttr{};
   mlir::ArrayAttr collapseDeviceTypeAttr = mlir::ArrayAttr{};
-  if (preserveCollapse) {
-    collapseAttr = origLoop.getCollapseAttr();
-    collapseDeviceTypeAttr = origLoop.getCollapseDeviceTypeAttr();
-  }
   auto newLoop = mlir::acc::LoopOp::create(
       rewriter, loc, origLoop->getResultTypes(), lb, ub, step, inclusiveUBAttr,
       collapseAttr, collapseDeviceTypeAttr, origLoop.getGangOperands(),
@@ -78,53 +73,6 @@ createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
   return newLoop;
 }
 
-// Create inner loop inside input loop
-static mlir::acc::LoopOp
-createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
-                mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
-                mlir::DenseBoolArrayAttr inclusiveUBAttr, mlir::Location loc) {
-  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
-      inputLoop, rewriter, lb, ub, step, inclusiveUBAttr,
-      mlir::acc::CombinedConstructsTypeAttr{}, loc, /*preserveCollapse*/ false);
-
-  // Remove gang/worker attributes from inner loops
-  rewriter.startOpModification(elementLoop);
-  if (inputLoop.hasGang() ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Num) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Static)) {
-    elementLoop.removeGangAttr();
-    elementLoop.removeGangOperandsArgTypeAttr();
-    elementLoop.removeGangOperandsSegmentsAttr();
-    elementLoop.removeGangOperandsDeviceTypeAttr();
-    // Also drop the operand values themselves so that elementLoop does not
-    // end up with a non-empty gang operand list but no corresponding
-    // device-type/segment/arg-type attributes. Leaving stale operands behind
-    // makes elementLoop look like it still has gang operands to later
-    // queries (e.g. LoopOp::getGangValue), which then dereference the
-    // now-missing device-type attribute.
-    elementLoop.getGangOperandsMutable().clear();
-  }
-  if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
-    elementLoop.removeWorkerAttr();
-    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
-    // As above for gang, also drop the worker operand values so elementLoop
-    // does not keep a dangling worker operand with no device-type attribute.
-    elementLoop.getWorkerNumOperandsMutable().clear();
-  }
-  rewriter.finalizeOpModification(elementLoop);
-
-  // Create empty block in elementLoop and add IV argument
-  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
-                                          elementLoop.getRegion().begin());
-  rewriter.setInsertionPointToEnd(blk);
-  mlir::acc::YieldOp::create(rewriter, loc);
-  elementLoop.getBody().addArgument(
-      inputLoop.getBody().getArgument(0).getType(), loc);
-
-  return elementLoop;
-}
-
 // Move ops from source to target Loop and replace uses of IVs
 static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
                                  mlir::acc::LoopOp targetLoop,
@@ -159,180 +107,145 @@ static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
     rewriter.finalizeOpModification(op);
 }
 
+// Create a single "element group" loop nested in `tileLoop`, carrying
+// `ivTypes.size()` induction variables so the whole element space is one
+// multi-IV loop. The element group carries vector or worker but not gang.
+static mlir::acc::LoopOp
+createElementGroupLoop(mlir::acc::LoopOp tileLoop, mlir::RewriterBase &rewriter,
+                       mlir::ValueRange lbs, mlir::ValueRange ubs,
+                       mlir::ValueRange steps,
+                       mlir::DenseBoolArrayAttr inclusiveUBAttr,
+                       llvm::ArrayRef<mlir::Type> ivTypes, mlir::Location loc) {
+  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
+      tileLoop, rewriter, lbs, ubs, steps, inclusiveUBAttr,
+      mlir::acc::CombinedConstructsTypeAttr{}, loc);
+
+  // Drop gang from the element group, keeping vector/worker. The operand
+  // values must be cleared too, not just the attributes.
+  rewriter.startOpModification(elementLoop);
+  if (tileLoop.hasGang() ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Num) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Static)) {
+    elementLoop.removeGangAttr();
+    elementLoop.removeGangOperandsArgTypeAttr();
+    elementLoop.removeGangOperandsSegmentsAttr();
+    elementLoop.removeGangOperandsDeviceTypeAttr();
+    elementLoop.getGangOperandsMutable().clear();
+  }
+  if (tileLoop.hasVector() || tileLoop.getVectorValue()) {
+    elementLoop.removeWorkerAttr();
+    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    elementLoop.getWorkerNumOperandsMutable().clear();
+  }
+  rewriter.finalizeOpModification(elementLoop);
+
+  // Create the element loop body: one block argument per IV plus a terminator.
+  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
+                                          elementLoop.getRegion().begin());
+  rewriter.setInsertionPointToEnd(blk);
+  mlir::acc::YieldOp::create(rewriter, loc);
+  for (mlir::Type ivType : ivTypes)
+    elementLoop.getBody().addArgument(ivType, loc);
+
+  return elementLoop;
+}
+
 mlir::acc::LoopOp
-mlir::acc::tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+mlir::acc::tileACCLoops(mlir::acc::LoopOp tileLoop,
                         const llvm::SmallVector<mlir::Value> &tileSizes,
                         int32_t defaultTileSize, mlir::RewriterBase &rewriter) {
-  // Tile collapsed and/or nested loops
-  mlir::acc::LoopOp outerLoop = tileLoops[0];
-  const mlir::Location loc = outerLoop.getLoc();
-
-  mlir::acc::LoopOp innerLoop = tileLoops[tileLoops.size() - 1];
-  llvm::SmallVector<mlir::Value, 3> origIVs;
-  llvm::SmallVector<mlir::Value, 3> origSteps;
-  llvm::SmallVector<mlir::Value, 3> origUBs;
-  llvm::SmallVector<mlir::Value, 3> newSteps;
-  llvm::SmallVector<mlir::Value, 3> newUBs;
-  llvm::SmallVector<mlir::Value, 3> newIVs;
-  size_t nOps = innerLoop.getBody().getOperations().size();
-
-  // Extract original inclusiveUBs
+  // Tile a single fused acc.loop that carries all associated induction
+  // variables. This keeps the tile iterations as one multi-IV "tile group"
+  // loop and the in-tile iterations as one multi-IV "element group" loop, each
+  // spanning all of its induction variables.
+  const mlir::Location loc = tileLoop.getLoc();
+  const unsigned tileCount = tileSizes.size();
+
+  llvm::SmallVector<mlir::Value, 3> origIVs(tileLoop.getBody().getArguments());
+  llvm::SmallVector<mlir::Value, 3> origUBs(tileLoop.getUpperbound());
+  llvm::SmallVector<mlir::Value, 3> origSteps(tileLoop.getStep());
+  const unsigned numIVs = origIVs.size();
+  const size_t nOps = tileLoop.getBody().getOperations().size();
+
+  // Original inclusive-UB flags (default false when the attribute is absent).
   llvm::SmallVector<bool> inclusiveUBs;
-  for (auto tileLoop : tileLoops) {
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      // inclusiveUBs are present on the IR from Fortran frontend for DO loops
-      // but might not be present from other frontends (python)
-      // So check if it exists
-      if (tileLoop.getInclusiveUpperboundAttr())
-        inclusiveUBs.push_back(
-            tileLoop.getInclusiveUpperboundAttr().asArrayRef()[j]);
-      else
-        inclusiveUBs.push_back(false);
-    }
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (tileLoop.getInclusiveUpperboundAttr())
+      inclusiveUBs.push_back(
+          tileLoop.getInclusiveUpperboundAttr().asArrayRef()[i]);
+    else
+      inclusiveUBs.push_back(false);
   }
 
-  // Extract original ivs, UBs, steps, and calculate new steps
-  rewriter.setInsertionPoint(outerLoop);
-  for (auto [i, tileLoop] : llvm::enumerate(tileLoops)) {
-    for (auto arg : tileLoop.getBody().getArguments())
-      origIVs.push_back(arg);
-    for (auto ub : tileLoop.getUpperbound())
-      origUBs.push_back(ub);
-
-    llvm::SmallVector<mlir::Value, 3> currentLoopSteps;
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      origSteps.push_back(step);
-      if (i + j >= tileSizes.size()) {
-        currentLoopSteps.push_back(step);
-      } else {
-        mlir::Value tileSize = resolveAndCastTileSize(
-            tileSizes[i + j], defaultTileSize, step.getType(), rewriter, loc);
-        auto newLoopStep =
-            mlir::arith::MulIOp::create(rewriter, loc, step, tileSize);
-        currentLoopSteps.push_back(newLoopStep);
-        newSteps.push_back(newLoopStep);
-      }
+  // Scale each tiled dimension's step by its tile size to form the tile group
+  // loop steps.
+  rewriter.setInsertionPoint(tileLoop);
+  llvm::SmallVector<mlir::Value, 3> scaledSteps;
+  llvm::SmallVector<mlir::Value, 3> tileLoopSteps;
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (i < tileCount) {
+      mlir::Value tileSize = resolveAndCastTileSize(
+          tileSizes[i], defaultTileSize, origSteps[i].getType(), rewriter, loc);
+      mlir::Value scaled =
+          mlir::arith::MulIOp::create(rewriter, loc, origSteps[i], tileSize);
+      scaledSteps.push_back(scaled);
+      tileLoopSteps.push_back(scaled);
+    } else {
+      tileLoopSteps.push_back(origSteps[i]);
     }
-
-    rewriter.startOpModification(tileLoop);
-    tileLoop.getStepMutable().clear();
-    tileLoop.getStepMutable().append(currentLoopSteps);
-    rewriter.finalizeOpModification(tileLoop);
   }
 
-  // Calculate new upper bounds for element loops
-  for (size_t i = 0; i < newSteps.size(); i++) {
-    rewriter.setInsertionPoint(innerLoop.getBody().getTerminator());
-    // UpperBound: min(origUB, origIV+(originalStep*tile_size))
-    auto stepped =
-        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], newSteps[i]);
+  // Compute the element-loop upper bounds min(origUB, origIV + scaledStep).
+  rewriter.setInsertionPoint(tileLoop.getBody().getTerminator());
+  llvm::SmallVector<mlir::Value, 3> elemLBs, elemUBs, elemSteps;
+  llvm::SmallVector<mlir::Type, 3> elemIVTypes;
+  llvm::SmallVector<bool> elemInclusiveUBs;
+  for (unsigned i = 0; i < tileCount; ++i) {
+    mlir::Value stepped =
+        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], scaledSteps[i]);
     mlir::Value newUB = stepped;
     if (inclusiveUBs[i]) {
-      // Handle InclusiveUB
-      // UpperBound: min(origUB, origIV+(originalStep*tile_size - 1))
-      auto c1 = mlir::arith::ConstantOp::create(
-          rewriter, loc, newSteps[i].getType(),
-          rewriter.getIntegerAttr(newSteps[i].getType(), 1));
+      // Inclusive UB: min(origUB, origIV + (scaledStep - 1)).
+      mlir::Value c1 = mlir::arith::ConstantOp::create(
+          rewriter, loc, scaledSteps[i].getType(),
+          rewriter.getIntegerAttr(scaledSteps[i].getType(), 1));
       newUB = mlir::arith::SubIOp::create(rewriter, loc, stepped, c1);
     }
-    newUBs.push_back(
+    elemUBs.push_back(
         mlir::arith::MinSIOp::create(rewriter, loc, origUBs[i], newUB));
+    elemLBs.push_back(origIVs[i]);
+    elemSteps.push_back(origSteps[i]);
+    elemIVTypes.push_back(origIVs[i].getType());
+    elemInclusiveUBs.push_back(inclusiveUBs[i]);
   }
 
-  // Create and insert nested elementLoopOps before terminator of outer loopOp
-  mlir::acc::LoopOp currentLoop = innerLoop;
-  for (size_t i = 0; i < tileSizes.size(); i++) {
-    rewriter.setInsertionPoint(currentLoop.getBody().getTerminator());
-    mlir::DenseBoolArrayAttr inclusiveUBAttr = mlir::DenseBoolArrayAttr{};
-    if (inclusiveUBs[i])
-      inclusiveUBAttr = rewriter.getDenseBoolArrayAttr({true});
-
-    mlir::acc::LoopOp elementLoop =
-        createInnerLoop(innerLoop, rewriter, mlir::ValueRange{origIVs[i]},
-                        mlir::ValueRange{newUBs[i]},
-                        mlir::ValueRange{origSteps[i]}, inclusiveUBAttr, loc);
-
-    // Remove vector/worker attributes from inner element loops except
-    // outermost element loop
-    if (i > 0) {
-      rewriter.startOpModification(elementLoop);
-      removeWorkerVectorFromLoop(elementLoop);
-      rewriter.finalizeOpModification(elementLoop);
-    }
-    newIVs.push_back(elementLoop.getBody().getArgument(0));
-    currentLoop = elementLoop;
-  }
-
-  // Remove vector/worker attributes from outer tile loops
-  for (auto tileLoop : tileLoops) {
-    rewriter.startOpModification(tileLoop);
-    removeWorkerVectorFromLoop(tileLoop);
-    rewriter.finalizeOpModification(tileLoop);
-  }
-
-  // Move ops from inner tile loop to inner element loop and replace IV uses
-  moveOpsAndReplaceIVs(innerLoop, currentLoop, newIVs, origIVs, nOps, rewriter);
-
-  return outerLoop;
-}
-
-llvm::SmallVector<mlir::acc::LoopOp>
-mlir::acc::uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
-                           unsigned collapseCount,
-                           mlir::RewriterBase &rewriter) {
-  llvm::SmallVector<mlir::acc::LoopOp> newLoops;
-  llvm::SmallVector<mlir::Value, 3> newIVs;
-  mlir::Location loc = origLoop.getLoc();
-  llvm::SmallVector<bool> newInclusiveUBs;
-  llvm::SmallVector<mlir::Value, 3> lbs, ubs, steps;
-  for (unsigned i = 0; i < collapseCount; i++) {
-    // inclusiveUpperbound attribute might not be set, default to false
-    bool inclusiveUB = false;
-    if (origLoop.getInclusiveUpperboundAttr())
-      inclusiveUB = origLoop.getInclusiveUpperboundAttr().asArrayRef()[i];
-    newInclusiveUBs.push_back(inclusiveUB);
-    lbs.push_back(origLoop.getLowerbound()[i]);
-    ubs.push_back(origLoop.getUpperbound()[i]);
-    steps.push_back(origLoop.getStep()[i]);
-  }
-  mlir::acc::LoopOp outerLoop = createACCLoopFromOriginal(
-      origLoop, rewriter, lbs, ubs, steps,
-      rewriter.getDenseBoolArrayAttr(newInclusiveUBs),
-      origLoop.getCombinedAttr(), loc, /*preserveCollapse*/ true);
-  mlir::Block *blk = rewriter.createBlock(&outerLoop.getRegion(),
-                                          outerLoop.getRegion().begin());
-  rewriter.setInsertionPointToEnd(blk);
-  mlir::acc::YieldOp::create(rewriter, loc);
-  for (unsigned i = 0; i < collapseCount; i++) {
-    outerLoop.getBody().addArgument(origLoop.getBody().getArgument(i).getType(),
-                                    loc);
-    newIVs.push_back(outerLoop.getBody().getArgument(i));
-  }
-  newLoops.push_back(outerLoop);
-
-  mlir::acc::LoopOp currentLoopOp = outerLoop;
-  for (unsigned i = collapseCount; i < tileCount; i++) {
-    rewriter.setInsertionPoint(currentLoopOp.getBody().getTerminator());
-    bool inclusiveUB = false;
-    if (origLoop.getInclusiveUpperboundAttr())
-      inclusiveUB = origLoop.getInclusiveUpperboundAttr().asArrayRef()[i];
-    mlir::DenseBoolArrayAttr inclusiveUBAttr =
-        rewriter.getDenseBoolArrayAttr({inclusiveUB});
-    mlir::acc::LoopOp innerLoop = createInnerLoop(
-        origLoop, rewriter, mlir::ValueRange{origLoop.getLowerbound()[i]},
-        mlir::ValueRange{origLoop.getUpperbound()[i]},
-        mlir::ValueRange{origLoop.getStep()[i]}, inclusiveUBAttr, loc);
-    newIVs.push_back(innerLoop.getBody().getArgument(0));
-    newLoops.push_back(innerLoop);
-    currentLoopOp = innerLoop;
-  }
-  // Move ops from origLoop to innermost loop and replace uses of IVs
-  size_t nOps = origLoop.getBody().getOperations().size();
-  llvm::SmallVector<mlir::Value, 3> origIVs;
-  for (auto arg : origLoop.getBody().getArguments())
-    origIVs.push_back(arg);
-  moveOpsAndReplaceIVs(origLoop, currentLoopOp, newIVs, origIVs, nOps,
+  // Only attach an inclusiveUpperbound attribute if at least one element
+  // dimension is inclusive.
+  mlir::DenseBoolArrayAttr elemInclAttr = mlir::DenseBoolArrayAttr{};
+  if (llvm::is_contained(elemInclusiveUBs, true))
+    elemInclAttr = rewriter.getDenseBoolArrayAttr(elemInclusiveUBs);
+
+  // Create the element group loop from the unmodified tile loop.
+  mlir::acc::LoopOp elementLoop =
+      createElementGroupLoop(tileLoop, rewriter, elemLBs, elemUBs, elemSteps,
+                             elemInclAttr, elemIVTypes, loc);
+
+  // Move the original body into the element loop and remap the tiled IVs to the
+  // element IVs.
+  llvm::SmallVector<mlir::Value, 3> newIVs(
+      elementLoop.getBody().getArguments());
+  llvm::SmallVector<mlir::Value, 3> tiledOrigIVs(origIVs.begin(),
+                                                 origIVs.begin() + tileCount);
+  moveOpsAndReplaceIVs(tileLoop, elementLoop, newIVs, tiledOrigIVs, nOps,
                        rewriter);
 
-  return newLoops;
+  // Turn the fused loop into the tile group: scaled steps, gang only.
+  rewriter.startOpModification(tileLoop);
+  tileLoop.getStepMutable().clear();
+  tileLoop.getStepMutable().append(tileLoopSteps);
+  removeWorkerVectorFromLoop(tileLoop);
+  rewriter.finalizeOpModification(tileLoop);
+
+  return tileLoop;
 }
diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir
index 6ef1884345f82..53b0467976658 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir
@@ -13,3 +13,26 @@ func.func @tile_wider_than_iv(%arg0: memref<100xf32>) {
   } attributes {independent = [#acc.device_type<none>]}
   return
 }
+
+// -----
+
+// A tile clause combined with a collapse clause on the same loop is not
+// supported: the two clauses describe conflicting loop-association counts. The
+// pass must diagnose it rather than silently drop the collapse clause.
+
+func.func @tile_with_collapse(%arg0: memref<100x50xf32>) {
+  %c0 = arith.constant 0 : index
+  %c100 = arith.constant 100 : index
+  %c50 = arith.constant 50 : index
+  %c1 = arith.constant 1 : index
+  %c4 = arith.constant 4 : index
+  %c8 = arith.constant 8 : index
+  // expected-error @below {{not yet implemented: a tile clause combined with a collapse clause on the same loop}}
+  acc.loop tile({%c4 : index, %c8 : index}) control(%i : index, %j : index) = (%c0, %c0 : index, index) to (%c100, %c50 : index, index) step (%c1, %c1 : index, index) {
+    %val = arith.index_castui %i : index to i32
+    %fval = arith.sitofp %val : i32 to f32
+    memref.store %fval, %arg0[%i, %j] : memref<100x50xf32>
+    acc.yield
+  } attributes {collapse = [2], collapseDeviceType = [#acc.device_type<none>], independent = [#acc.device_type<none>]}
+  return
+}
diff --git a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
index a1f80ad9defdf..724fba48b7595 100644
--- a/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir
@@ -32,7 +32,8 @@ func.func @single_loop_tile(%arg0: memref<10xf32>) {
 }
 
 // Test 2-level nested loop tiling with tile(4, 8)
-// Creates: tile_loop_1 -> tile_loop_2 -> element_loop_1 -> element_loop_2
+// Produces one multi-IV tile-group loop (gang) wrapping one multi-IV
+// element-group loop (vector), each carrying both tiled induction variables.
 
 // CHECK-LABEL: func.func @nested_loop_tile
 // CHECK-DAG:     %[[C0:.*]] = arith.constant 0 : index
@@ -41,22 +42,16 @@ func.func @single_loop_tile(%arg0: memref<10xf32>) {
 // CHECK-DAG:     %[[C1:.*]] = arith.constant 1 : index
 // CHECK-DAG:     %[[C4:.*]] = arith.constant 4 : index
 // CHECK-DAG:     %[[C8:.*]] = arith.constant 8 : index
-// Outer tile loop with gang
-// CHECK:         acc.loop gang control(%[[I:.*]] : index) = (%[[C0]] : index) to (%[[C100]] : index) step (%[[C4]] : index) {
-// Inner tile loop
-// CHECK:           acc.loop control(%[[J:.*]] : index) = (%[[C0]] : index) to (%[[C50]] : index) step (%[[C8]] : index) {
-// Outer element loop with vector
-// CHECK:             acc.loop vector control({{.*}} : index) = (%[[I]] : index) to ({{.*}} : index) step (%[[C1]] : index) {
-// Inner element loop
-// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}} : index) step (%[[C1]] : index) {
-// CHECK:                 acc.yield
-// CHECK:               }
-// CHECK:               acc.yield
-// CHECK:             }
+// Tile-group loop with gang; steps scaled by the tile sizes.
+// CHECK:         acc.loop gang control(%[[I:.*]] : index, %[[J:.*]] : index) = (%[[C0]], %[[C0]] : index, index) to (%[[C100]], %[[C50]] : index, index) step (%[[C4]], %[[C8]] : index, index) {
+// CHECK:           %[[MUB0:.*]] = arith.minsi
+// CHECK:           %[[MUB1:.*]] = arith.minsi
+// Element-group loop with vector.
+// CHECK:           acc.loop vector control(%{{.*}} : index, %{{.*}} : index) = (%[[I]], %[[J]] : index, index) to (%[[MUB0]], %[[MUB1]] : index, index) step (%[[C1]], %[[C1]] : index, index) {
 // CHECK:             acc.yield
-// CHECK:           }
+// CHECK:           } attributes {independent = [#acc.device_type<none>]}
 // CHECK:           acc.yield
-// CHECK:         }
+// CHECK:         } attributes {independent = [#acc.device_type<none>]}
 func.func @nested_loop_tile(%arg0: memref<100x50xf32>) {
   %c0 = arith.constant 0 : index
   %c100 = arith.constant 100 : index
@@ -76,29 +71,19 @@ func.func @nested_loop_tile(%arg0: memref<100x50xf32>) {
 
 // Regression test: a loop with GANG(STATIC: N) combined with a multi-dim
 // TILE clause used to crash with an assertion failure inside
-// LoopOp::getGangValue() because uncollapseLoops() (needed here since the
-// tile count exceeds the implicit collapse count of 1) left one of the
-// generated inner loops with a leftover gang operand but no corresponding
-// gang device-type attribute. Check that the pass completes and that gang
-// is only preserved on the outermost tile loop.
+// LoopOp::getGangValue() when a leftover gang operand was copied onto a
+// generated loop without a corresponding gang device-type attribute. Check
+// that the pass completes and that gang (with its static operand) is only
+// preserved on the tile-group loop, never on the element group.
 
 // CHECK-LABEL: func.func @gang_static_with_multi_dim_tile
-// CHECK:         acc.loop gang({static=%{{.*}} : i32}) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:       gang
-// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK:         acc.loop gang({static=%{{.*}} : i32}) control(%[[I:.*]] : index, %[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK:           acc.loop control(%{{.*}} : index, %{{.*}} : index) = (%[[I]], %[[J]] : index, index) to ({{.*}}) step ({{.*}}) {
 // CHECK-NOT:         gang
-// CHECK:             acc.loop control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:           gang
-// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:             gang
-// CHECK:                 acc.yield
-// CHECK:               }
-// CHECK:               acc.yield
-// CHECK:             }
 // CHECK:             acc.yield
-// CHECK:           }
+// CHECK:           } attributes {independent = [#acc.device_type<none>]}
 // CHECK:           acc.yield
-// CHECK:         }
+// CHECK:         } attributes {independent = [#acc.device_type<none>]}
 func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
   %c0 = arith.constant 0 : index
   %c100 = arith.constant 100 : index
@@ -118,36 +103,22 @@ func.func @gang_static_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
 }
 
 // Regression test: a loop with WORKER(N) combined with VECTOR and a
-// multi-dim TILE clause used to produce invalid IR because createInnerLoop()
-// (via uncollapseLoops()) and removeWorkerVectorFromLoop() removed the
-// worker attributes from generated inner/tile loops without also clearing
-// the worker operand value copied onto them. This left loops with a
-// non-empty worker operand list but no worker device-type attribute,
-// which the verifier rejects ('worker operands count must match worker
-// device_type count'). Check that the pass produces valid IR, with worker
-// only preserved on the outermost tile loop and vector only on the
-// outermost element loop.
+// multi-dim TILE clause used to produce invalid IR because a worker attribute
+// was removed from a generated loop without also clearing the worker operand
+// value copied onto it, leaving a non-empty worker operand list with no worker
+// device-type attribute (rejected by the verifier). Check that the pass
+// produces valid IR, with worker only on the tile-group loop and vector only
+// on the element-group loop.
 
 // CHECK-LABEL: func.func @worker_num_with_multi_dim_tile
-// CHECK:         acc.loop worker(%{{.*}} : i32) control(%[[I:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:       worker
+// CHECK:         acc.loop worker(%{{.*}} : i32) control(%[[I:.*]] : index, %[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
 // CHECK-NOT:       vector
-// CHECK:           acc.loop control(%[[J:.*]] : index) = ({{.*}}) to ({{.*}}) step ({{.*}}) {
+// CHECK:           acc.loop vector control(%{{.*}} : index, %{{.*}} : index) = (%[[I]], %[[J]] : index, index) to ({{.*}}) step ({{.*}}) {
 // CHECK-NOT:         worker
-// CHECK-NOT:         vector
-// CHECK:             acc.loop vector control({{.*}} : index) = (%[[I]] : index) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:           worker
-// CHECK:               acc.loop control({{.*}} : index) = (%[[J]] : index) to ({{.*}}) step ({{.*}}) {
-// CHECK-NOT:             worker
-// CHECK-NOT:             vector
-// CHECK:                 acc.yield
-// CHECK:               }
-// CHECK:               acc.yield
-// CHECK:             }
 // CHECK:             acc.yield
-// CHECK:           }
+// CHECK:           } attributes {independent = [#acc.device_type<none>]}
 // CHECK:           acc.yield
-// CHECK:         }
+// CHECK:         } attributes {independent = [#acc.device_type<none>]}
 func.func @worker_num_with_multi_dim_tile(%arg0: memref<100x50xf32>) {
   %c0 = arith.constant 0 : index
   %c100 = arith.constant 100 : index
diff --git a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp
index 95bc1eab7d3fe..302e60cca168d 100644
--- a/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp
+++ b/mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp
@@ -98,41 +98,36 @@ TEST_F(OpenACCUtilsTilingTest, tileACCLoopsSingleLoop) {
   Value tileSize =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(4));
 
-  // Create the loop
+  // Create the loop (single IV)
   acc::LoopOp loopOp = createLoopOp(b, {lb}, {ub}, {step});
 
   // Tile the loop using IRRewriter
   IRRewriter rewriter(&context);
   rewriter.setInsertionPoint(loopOp);
 
-  SmallVector<acc::LoopOp> loopsToTile = {loopOp};
   SmallVector<Value> tileSizes = {tileSize};
 
-  acc::LoopOp tiledLoop =
-      tileACCLoops(loopsToTile, tileSizes, /*defaultTileSize=*/128, rewriter);
+  acc::LoopOp tileGroup =
+      tileACCLoops(loopOp, tileSizes, /*defaultTileSize=*/128, rewriter);
 
-  // Verify the tiled loop was created
-  EXPECT_TRUE(tiledLoop != nullptr);
-  EXPECT_FALSE(tiledLoop.getBody().empty());
+  // Verify the tile-group loop was created
+  EXPECT_TRUE(tileGroup != nullptr);
+  EXPECT_FALSE(tileGroup.getBody().empty());
 
-  // After tiling a single loop with tile(4), we should have:
-  // - 1 tile loop (the outer loop)
-  // - 1 element loop nested inside
-  // Total: 1 nested loop inside the tile loop
-  EXPECT_EQ(countNestedLoops(tiledLoop), 1u);
+  // A single-IV tile(4) produces one tile-group loop wrapping one element-group
+  // loop: exactly one nested loop.
+  EXPECT_EQ(countNestedLoops(tileGroup), 1u);
 
-  // The tile loop (outer) should have 1 IV
-  EXPECT_EQ(tiledLoop.getBody().getNumArguments(), 1u);
+  // The tile-group loop carries its single IV.
+  EXPECT_EQ(tileGroup.getBody().getNumArguments(), 1u);
 
-  // Collect nested loops and verify
-  auto nestedLoops = collectNestedLoops(tiledLoop);
-  EXPECT_EQ(nestedLoops.size(), 1u);
-  // The element loop should have 1 IV
-  if (!nestedLoops.empty())
-    EXPECT_EQ(nestedLoops[0].getBody().getNumArguments(), 1u);
+  auto nestedLoops = collectNestedLoops(tileGroup);
+  ASSERT_EQ(nestedLoops.size(), 1u);
+  // The element-group loop should carry the single IV as well.
+  EXPECT_EQ(nestedLoops[0].getBody().getNumArguments(), 1u);
 }
 
-TEST_F(OpenACCUtilsTilingTest, tileACCLoopsNestedLoops) {
+TEST_F(OpenACCUtilsTilingTest, tileACCLoopsFusedTwoDim) {
   // Create a module to hold the function
   OwningOpRef<ModuleOp> module = ModuleOp::create(loc);
   Block *moduleBlock = module->getBody();
@@ -148,15 +143,13 @@ TEST_F(OpenACCUtilsTilingTest, tileACCLoopsNestedLoops) {
 
   b.setInsertionPointToStart(funcBlock);
 
-  // Create loop bounds for outer loop
+  // Create bounds for a single fused loop carrying two IVs.
   Value lb1 =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
   Value ub1 =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(100));
   Value step1 =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-
-  // Create loop bounds for inner loop
   Value lb2 =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
   Value ub2 =
@@ -170,179 +163,31 @@ TEST_F(OpenACCUtilsTilingTest, tileACCLoopsNestedLoops) {
   Value tileSize2 =
       arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(8));
 
-  // Create outer loop
-  acc::LoopOp outerLoop = createLoopOp(b, {lb1}, {ub1}, {step1});
-
-  // Create inner loop inside outer loop
-  b.setInsertionPoint(outerLoop.getBody().getTerminator());
-  acc::LoopOp innerLoop = createLoopOp(b, {lb2}, {ub2}, {step2});
-
-  // Tile the loops
-  IRRewriter rewriter(&context);
-  rewriter.setInsertionPoint(outerLoop);
-
-  SmallVector<acc::LoopOp> loopsToTile = {outerLoop, innerLoop};
-  SmallVector<Value> tileSizes = {tileSize1, tileSize2};
-
-  acc::LoopOp tiledLoop =
-      tileACCLoops(loopsToTile, tileSizes, /*defaultTileSize=*/128, rewriter);
-
-  // Verify the tiled loop nest was created
-  EXPECT_TRUE(tiledLoop != nullptr);
-  EXPECT_FALSE(tiledLoop.getBody().empty());
-
-  // After tiling a 2-level nested loop with tile(4,8), we should have:
-  // tile_loop_1 -> tile_loop_2 -> element_loop_1 -> element_loop_2
-  // Total: 3 nested loops inside the outermost tile loop
-  unsigned nestedCount = countNestedLoops(tiledLoop);
-  EXPECT_EQ(nestedCount, 3u);
-
-  // The outermost tile loop should have 1 IV
-  EXPECT_EQ(tiledLoop.getBody().getNumArguments(), 1u);
-
-  // Collect all nested loops and verify each has 1 IV
-  auto nestedLoops = collectNestedLoops(tiledLoop);
-  EXPECT_EQ(nestedLoops.size(), 3u);
-  for (auto loop : nestedLoops)
-    EXPECT_EQ(loop.getBody().getNumArguments(), 1u);
-}
-
-//===----------------------------------------------------------------------===//
-// uncollapseLoops Tests
-//===----------------------------------------------------------------------===//
-
-TEST_F(OpenACCUtilsTilingTest, uncollapseLoopsBasic) {
-  // Create a module to hold the function
-  OwningOpRef<ModuleOp> module = ModuleOp::create(loc);
-  Block *moduleBlock = module->getBody();
-
-  OpBuilder::InsertionGuard guard(b);
-  b.setInsertionPointToStart(moduleBlock);
-
-  // Create a function
-  auto funcType = b.getFunctionType({}, {});
-  OwningOpRef<func::FuncOp> funcOp =
-      func::FuncOp::create(b, loc, "test_func", funcType);
-  Block *funcBlock = funcOp->addEntryBlock();
-
-  b.setInsertionPointToStart(funcBlock);
-
-  // Create loop bounds for a collapsed 2-level loop
-  Value lb1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
-  Value ub1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(10));
-  Value step1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-  Value lb2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
-  Value ub2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(20));
-  Value step2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-
-  // Create a collapsed loop with 2 IVs
-  acc::LoopOp collapsedLoop =
+  // Create a single fused loop with two IVs.
+  acc::LoopOp fusedLoop =
       createLoopOp(b, {lb1, lb2}, {ub1, ub2}, {step1, step2});
 
-  // Set the collapse attribute
-  collapsedLoop.setCollapseForDeviceTypes(&context, {acc::DeviceType::None},
-                                          llvm::APInt(64, 1));
-
-  // Uncollapse the loop: tileCount=2, collapseCount=1
   IRRewriter rewriter(&context);
-  rewriter.setInsertionPoint(collapsedLoop);
-
-  SmallVector<acc::LoopOp> uncollapsedLoops = uncollapseLoops(
-      collapsedLoop, /*tileCount=*/2, /*collapseCount=*/1, rewriter);
-
-  // Should produce 2 loops (one outer with collapse=1, one inner)
-  EXPECT_EQ(uncollapsedLoops.size(), 2u);
-
-  if (uncollapsedLoops.size() >= 2) {
-    // Verify the outer loop has 1 IV (collapseCount=1)
-    acc::LoopOp outerLoop = uncollapsedLoops[0];
-    EXPECT_EQ(outerLoop.getBody().getNumArguments(), 1u);
-    EXPECT_EQ(outerLoop.getLowerbound().size(), 1u);
-    EXPECT_EQ(outerLoop.getUpperbound().size(), 1u);
-    EXPECT_EQ(outerLoop.getStep().size(), 1u);
-
-    // Verify the inner loop has 1 IV
-    acc::LoopOp innerLoop = uncollapsedLoops[1];
-    EXPECT_EQ(innerLoop.getBody().getNumArguments(), 1u);
-    EXPECT_EQ(innerLoop.getLowerbound().size(), 1u);
-    EXPECT_EQ(innerLoop.getUpperbound().size(), 1u);
-    EXPECT_EQ(innerLoop.getStep().size(), 1u);
-
-    // Verify nesting: inner loop should be inside outer loop
-    unsigned nestedCount = countNestedLoops(outerLoop);
-    EXPECT_EQ(nestedCount, 1u);
-  }
-}
-
-TEST_F(OpenACCUtilsTilingTest, uncollapseLoopsThreeLevels) {
-  // Test uncollapsing with 3 levels: collapse(2) with tile(3)
-  OwningOpRef<ModuleOp> module = ModuleOp::create(loc);
-  Block *moduleBlock = module->getBody();
+  rewriter.setInsertionPoint(fusedLoop);
 
-  OpBuilder::InsertionGuard guard(b);
-  b.setInsertionPointToStart(moduleBlock);
-
-  auto funcType = b.getFunctionType({}, {});
-  OwningOpRef<func::FuncOp> funcOp =
-      func::FuncOp::create(b, loc, "test_func", funcType);
-  Block *funcBlock = funcOp->addEntryBlock();
-
-  b.setInsertionPointToStart(funcBlock);
-
-  // Create 3 sets of bounds
-  Value lb1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
-  Value ub1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(10));
-  Value step1 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-  Value lb2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
-  Value ub2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(20));
-  Value step2 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-  Value lb3 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(0));
-  Value ub3 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(30));
-  Value step3 =
-      arith::ConstantOp::create(b, loc, b.getIndexType(), b.getIndexAttr(1));
-
-  // Create a collapsed loop with 3 IVs
-  acc::LoopOp collapsedLoop =
-      createLoopOp(b, {lb1, lb2, lb3}, {ub1, ub2, ub3}, {step1, step2, step3});
-
-  // Set collapse(2)
-  collapsedLoop.setCollapseForDeviceTypes(&context, {acc::DeviceType::None},
-                                          llvm::APInt(64, 2));
+  SmallVector<Value> tileSizes = {tileSize1, tileSize2};
 
-  // Uncollapse: tileCount=3, collapseCount=2
-  // This should create: outer loop with 2 IVs, then 1 inner loop
-  IRRewriter rewriter(&context);
-  rewriter.setInsertionPoint(collapsedLoop);
+  acc::LoopOp tileGroup =
+      tileACCLoops(fusedLoop, tileSizes, /*defaultTileSize=*/128, rewriter);
 
-  SmallVector<acc::LoopOp> uncollapsedLoops = uncollapseLoops(
-      collapsedLoop, /*tileCount=*/3, /*collapseCount=*/2, rewriter);
+  // Verify the tile-group loop nest was created.
+  EXPECT_TRUE(tileGroup != nullptr);
+  EXPECT_FALSE(tileGroup.getBody().empty());
 
-  // Should produce 2 loops
-  EXPECT_EQ(uncollapsedLoops.size(), 2u);
+  // tile(4,8) on a fused 2-IV loop produces exactly two multi-IV loops: a
+  // tile-group loop wrapping a single element-group loop (one nested loop),
+  // rather than a 4-deep nest of single-IV loops.
+  EXPECT_EQ(countNestedLoops(tileGroup), 1u);
 
-  if (uncollapsedLoops.size() >= 2) {
-    // Outer loop should have 2 IVs (from collapse=2)
-    acc::LoopOp outerLoop = uncollapsedLoops[0];
-    EXPECT_EQ(outerLoop.getBody().getNumArguments(), 2u);
-    EXPECT_EQ(outerLoop.getLowerbound().size(), 2u);
+  // Both groups carry all (two) IVs.
+  EXPECT_EQ(tileGroup.getBody().getNumArguments(), 2u);
 
-    // Inner loop should have 1 IV (the 3rd dimension)
-    acc::LoopOp innerLoop = uncollapsedLoops[1];
-    EXPECT_EQ(innerLoop.getBody().getNumArguments(), 1u);
-    EXPECT_EQ(innerLoop.getLowerbound().size(), 1u);
-  }
+  auto nestedLoops = collectNestedLoops(tileGroup);
+  ASSERT_EQ(nestedLoops.size(), 1u);
+  EXPECT_EQ(nestedLoops[0].getBody().getNumArguments(), 2u);
 }


From llvmlistbot at llvm.org  Thu Jul 23 12:48:14 2026
From: llvmlistbot at llvm.org (Vijay Kandiah)
Date: Thu, 23 Jul 2026 12:48:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Emit multi-IV tile and
 element loops from ACCLoopTiling (PR #211651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211651@github.com>
Message-ID: <6a626ffe.1728eaf3.5e665.d48e@mx.google.com>


https://github.com/VijayKandiah edited https://github.com/llvm/llvm-project/pull/211651

From llvmlistbot at llvm.org  Thu Jul 23 12:48:29 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 12:48:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Emit multi-IV tile and
 element loops from ACCLoopTiling (PR #211651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211651@github.com>
Message-ID: <6a62700d.13be6df8.323b61.ec77@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Vijay Kandiah (VijayKandiah)

<details>
<summary>Changes</summary>

`ACCLoopTiling` used to lower an N-dimensional `tile()` clause by *uncollapsing* the fused loop into a deep nest of single-IV `acc.loop`s (`tile_1 → … → tile_N → elem_1 → … → elem_N`). This discards the natural grouping of the tile clause: the 2N single-IV loops no longer express "these are the tile iterations" and "these are the in-tile iterations" as collapsible units, and gang/vector end up spread across a deep nest where only the outermost loop of each group carries the attribute. This could potentially lead to poor parallelism assignment.

With this MR:
`tileACCLoops` now rewrites the single fused `acc.loop` in-place into exactly two multi-IV loops, each carrying all the tiled induction variables:
- a **tile-group** loop whose steps are the original steps scaled by the tile sizes, and
- a nested **element-group** loop that walks the iterations within one tile, with upper bounds clamped to `min(origUB, tileStart + tileExtent)`.

Keeping each group as a single multi-IV `acc.loop` preserves the tile/element structure as two collapsible units instead of a 2N-deep single-IV nest, which is a simpler canonical form. The gang/worker/vector distribution across the two groups is unchanged from before.

This change also diagnoses `tile` + `collapse` on the same loop because these clauses give conflicting loop-association counts. The pass now emits a NYI diagnostic instead of silently dropping the `collapse` clause. Added a new invalid-input test to verify this.

With the pass no longer uncollapsing, `uncollapseLoops` and the vector-of-loops `tileACCLoops(SmallVector&lt;LoopOp&gt;&amp;, …)` overload (and the now-orphaned `createInnerLoop` helper) are deleted, along with their unit tests. New unit tests are added to cover the surviving fused-loop overload.


---

Patch is 44.54 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211651.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h (+24-29) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp (+14-18) 
- (modified) mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp (+129-216) 
- (modified) mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir (+23) 
- (modified) mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir (+28-57) 
- (modified) mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp (+35-190) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
index 6fcb706aa3488..05ac37d1106ec 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
@@ -20,23 +20,20 @@
 namespace mlir {
 namespace acc {
 
-/// Uncollapse tile loops with multiple IVs and collapseCount < tileCount.
-/// This is used to prepare loops for tiling when the collapse count is less
-/// than the tile count.
+/// Tile a single fused acc.loop that carries all associated induction
+/// variables (one IV per tile dimension).
 ///
-/// \param origLoop The original loop operation to uncollapse.
-/// \param tileCount The number of tile dimensions.
-/// \param collapseCount The collapse count from the original loop.
-/// \param rewriter The rewriter to use for modifications.
-/// \return A vector of uncollapsed loop operations.
-llvm::SmallVector<mlir::acc::LoopOp>
-uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
-                unsigned collapseCount, mlir::RewriterBase &rewriter);
-
-/// Tile ACC loops according to the given tile sizes.
+/// This produces exactly two multi-IV loops, each carrying all of the tiled
+/// induction variables:
 ///
-/// Tiling a 2-level nested loop will create two 'tile' loops containing two
-/// 'element' loops. The transformation looks like:
+///   - a "tile group" loop that steps over tiles: each step is the
+///     original step multiplied by the tile size. It keeps the original loop's
+///     gang attribute.
+///   - an "element group" loop that walks the iterations inside one tile: it
+///     keeps the original step, its lower bound is the current tile's starting
+///     index, and its upper bound is clamped to min(original upper bound,
+///     tile start + tile extent). It keeps the original loop's vector (or
+///     worker) attribute.
 ///
 /// Before Tiling:
 /// \code
@@ -48,17 +45,15 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 ///  }
 /// \endcode
 ///
-/// After Tiling:
+/// After Tiling (each group is one multi-IV loop over all tiled IVs):
 /// \code
-///  for (i = lb1; i < ub1; i += (step1 * tile_size1)) { // tile loop 1
-///    for (j = lb2; j < ub2; j += (step2 * tile_size2)) { // tile loop 2
-///      for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1) {
-///      // element loop 1
-///        for (jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2)
-///        { // element loop 2
-///          a[ii,jj] = i + j;
-///        }
-///      }
+///  // tile group
+///  for (i = lb1; i < ub1; i += (step1 * tile_size1),
+///       j = lb2; j < ub2; j += (step2 * tile_size2)) {
+///    // element group
+///    for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1,
+///         jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2) {
+///      a[ii,jj] = i + j;
 ///    }
 ///  }
 /// \endcode
@@ -66,13 +61,13 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 /// Unknown tile sizes (represented as -1 in acc dialect for `tile(*)`) are
 /// resolved to the provided default tile size.
 ///
-/// \param tileLoops The loops to tile (outermost first).
-/// \param tileSizes The tile sizes for each dimension. Values of -1 are
+/// \param tileLoop The fused loop to tile.
+/// \param tileSizes The tile sizes for each tiled dimension. Values of -1 are
 ///        treated as unknown and resolved to defaultTileSize.
 /// \param defaultTileSize The default tile size to use for unknown (*) tiles.
 /// \param rewriter The rewriter to use for modifications.
-/// \return The outermost loop after tiling.
-mlir::acc::LoopOp tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+/// \return The tile group loop that is modified in place.
+mlir::acc::LoopOp tileACCLoops(mlir::acc::LoopOp tileLoop,
                                const llvm::SmallVector<mlir::Value> &tileSizes,
                                int32_t defaultTileSize,
                                mlir::RewriterBase &rewriter);
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
index 6bc95ca896f37..d5865bc98727e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
@@ -162,8 +162,16 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
 
     SmallVector<Value> tileSizes(origLoop.getTileValues().begin(),
                                  origLoop.getTileValues().end());
-    unsigned tileCount = tileSizes.size();
-    unsigned collapseCount = origLoop.getCollapseValue().value_or(1);
+
+    // A tile clause and a collapse clause on the same loop are not supported:
+    // they give conflicting descriptions of how many loops the construct
+    // associates.
+    if (origLoop.getCollapseAttr()) {
+      accSupport.emitNYI(origLoop.getLoc(),
+                         "a tile clause combined with a collapse clause on the "
+                         "same loop");
+      return failure();
+    }
 
     // Sanity check tile size types
     if (failed(checkTileSizeTypes(origLoop, tileSizes)))
@@ -182,24 +190,12 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
     origLoop.removeTileOperandsDeviceTypeAttr();
     rewriter.finalizeOpModification(origLoop);
 
-    SmallVector<acc::LoopOp> loopsToTile;
-    if (collapseCount < tileCount) {
-      // Uncollapse tile loops before tiling if necessary
-      loopsToTile =
-          acc::uncollapseLoops(origLoop, tileCount, collapseCount, rewriter);
-      rewriter.replaceOp(origLoop, loopsToTile[0]);
-      LLVM_DEBUG(llvm::dbgs() << "\nAfter uncollapsing:\n"
-                              << *loopsToTile[0] << "\n");
-    } else {
-      loopsToTile.push_back(origLoop);
-    }
-
-    // loopsToTile is a vector of perfectly nested loops. The outermost loop
-    // may have multiple IVs but inner loops can only have one IV.
+    // Tile the single fused loop in place: the tile iterations become one
+    // multi-IV "tile group" loop wrapping one multi-IV "element group" loop.
     // The utility handles unknown tile sizes (*) by using `defaultTileSize`.
-    acc::tileACCLoops(loopsToTile, tileSizes, defaultTileSize, rewriter);
+    acc::tileACCLoops(origLoop, tileSizes, defaultTileSize, rewriter);
 
-    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *loopsToTile[0] << "\n");
+    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *origLoop << "\n");
     return success();
   }
 
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index 09e83af2bc686..32fd3957894fb 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
 #include "mlir/Transforms/RegionUtils.h"
+#include "llvm/ADT/STLExtras.h"
 
 // Resolve unknown tile sizes (represented as -1 for tile(*)) to the default.
 // Returns a value with the same type as targetType.
@@ -48,19 +49,13 @@ static void removeWorkerVectorFromLoop(mlir::acc::LoopOp loop) {
 }
 
 // Create a new ACC loop with new steps, lb, ub from original loop
-static mlir::acc::LoopOp
-createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
-                          mlir::RewriterBase &rewriter, mlir::ValueRange lb,
-                          mlir::ValueRange ub, mlir::ValueRange step,
-                          mlir::DenseBoolArrayAttr inclusiveUBAttr,
-                          mlir::acc::CombinedConstructsTypeAttr combinedAttr,
-                          mlir::Location loc, bool preserveCollapse) {
+static mlir::acc::LoopOp createACCLoopFromOriginal(
+    mlir::acc::LoopOp origLoop, mlir::RewriterBase &rewriter,
+    mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
+    mlir::DenseBoolArrayAttr inclusiveUBAttr,
+    mlir::acc::CombinedConstructsTypeAttr combinedAttr, mlir::Location loc) {
   mlir::ArrayAttr collapseAttr = mlir::ArrayAttr{};
   mlir::ArrayAttr collapseDeviceTypeAttr = mlir::ArrayAttr{};
-  if (preserveCollapse) {
-    collapseAttr = origLoop.getCollapseAttr();
-    collapseDeviceTypeAttr = origLoop.getCollapseDeviceTypeAttr();
-  }
   auto newLoop = mlir::acc::LoopOp::create(
       rewriter, loc, origLoop->getResultTypes(), lb, ub, step, inclusiveUBAttr,
       collapseAttr, collapseDeviceTypeAttr, origLoop.getGangOperands(),
@@ -78,53 +73,6 @@ createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
   return newLoop;
 }
 
-// Create inner loop inside input loop
-static mlir::acc::LoopOp
-createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
-                mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
-                mlir::DenseBoolArrayAttr inclusiveUBAttr, mlir::Location loc) {
-  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
-      inputLoop, rewriter, lb, ub, step, inclusiveUBAttr,
-      mlir::acc::CombinedConstructsTypeAttr{}, loc, /*preserveCollapse*/ false);
-
-  // Remove gang/worker attributes from inner loops
-  rewriter.startOpModification(elementLoop);
-  if (inputLoop.hasGang() ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Num) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Static)) {
-    elementLoop.removeGangAttr();
-    elementLoop.removeGangOperandsArgTypeAttr();
-    elementLoop.removeGangOperandsSegmentsAttr();
-    elementLoop.removeGangOperandsDeviceTypeAttr();
-    // Also drop the operand values themselves so that elementLoop does not
-    // end up with a non-empty gang operand list but no corresponding
-    // device-type/segment/arg-type attributes. Leaving stale operands behind
-    // makes elementLoop look like it still has gang operands to later
-    // queries (e.g. LoopOp::getGangValue), which then dereference the
-    // now-missing device-type attribute.
-    elementLoop.getGangOperandsMutable().clear();
-  }
-  if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
-    elementLoop.removeWorkerAttr();
-    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
-    // As above for gang, also drop the worker operand values so elementLoop
-    // does not keep a dangling worker operand with no device-type attribute.
-    elementLoop.getWorkerNumOperandsMutable().clear();
-  }
-  rewriter.finalizeOpModification(elementLoop);
-
-  // Create empty block in elementLoop and add IV argument
-  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
-                                          elementLoop.getRegion().begin());
-  rewriter.setInsertionPointToEnd(blk);
-  mlir::acc::YieldOp::create(rewriter, loc);
-  elementLoop.getBody().addArgument(
-      inputLoop.getBody().getArgument(0).getType(), loc);
-
-  return elementLoop;
-}
-
 // Move ops from source to target Loop and replace uses of IVs
 static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
                                  mlir::acc::LoopOp targetLoop,
@@ -159,180 +107,145 @@ static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
     rewriter.finalizeOpModification(op);
 }
 
+// Create a single "element group" loop nested in `tileLoop`, carrying
+// `ivTypes.size()` induction variables so the whole element space is one
+// multi-IV loop. The element group carries vector or worker but not gang.
+static mlir::acc::LoopOp
+createElementGroupLoop(mlir::acc::LoopOp tileLoop, mlir::RewriterBase &rewriter,
+                       mlir::ValueRange lbs, mlir::ValueRange ubs,
+                       mlir::ValueRange steps,
+                       mlir::DenseBoolArrayAttr inclusiveUBAttr,
+                       llvm::ArrayRef<mlir::Type> ivTypes, mlir::Location loc) {
+  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
+      tileLoop, rewriter, lbs, ubs, steps, inclusiveUBAttr,
+      mlir::acc::CombinedConstructsTypeAttr{}, loc);
+
+  // Drop gang from the element group, keeping vector/worker. The operand
+  // values must be cleared too, not just the attributes.
+  rewriter.startOpModification(elementLoop);
+  if (tileLoop.hasGang() ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Num) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Static)) {
+    elementLoop.removeGangAttr();
+    elementLoop.removeGangOperandsArgTypeAttr();
+    elementLoop.removeGangOperandsSegmentsAttr();
+    elementLoop.removeGangOperandsDeviceTypeAttr();
+    elementLoop.getGangOperandsMutable().clear();
+  }
+  if (tileLoop.hasVector() || tileLoop.getVectorValue()) {
+    elementLoop.removeWorkerAttr();
+    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    elementLoop.getWorkerNumOperandsMutable().clear();
+  }
+  rewriter.finalizeOpModification(elementLoop);
+
+  // Create the element loop body: one block argument per IV plus a terminator.
+  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
+                                          elementLoop.getRegion().begin());
+  rewriter.setInsertionPointToEnd(blk);
+  mlir::acc::YieldOp::create(rewriter, loc);
+  for (mlir::Type ivType : ivTypes)
+    elementLoop.getBody().addArgument(ivType, loc);
+
+  return elementLoop;
+}
+
 mlir::acc::LoopOp
-mlir::acc::tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+mlir::acc::tileACCLoops(mlir::acc::LoopOp tileLoop,
                         const llvm::SmallVector<mlir::Value> &tileSizes,
                         int32_t defaultTileSize, mlir::RewriterBase &rewriter) {
-  // Tile collapsed and/or nested loops
-  mlir::acc::LoopOp outerLoop = tileLoops[0];
-  const mlir::Location loc = outerLoop.getLoc();
-
-  mlir::acc::LoopOp innerLoop = tileLoops[tileLoops.size() - 1];
-  llvm::SmallVector<mlir::Value, 3> origIVs;
-  llvm::SmallVector<mlir::Value, 3> origSteps;
-  llvm::SmallVector<mlir::Value, 3> origUBs;
-  llvm::SmallVector<mlir::Value, 3> newSteps;
-  llvm::SmallVector<mlir::Value, 3> newUBs;
-  llvm::SmallVector<mlir::Value, 3> newIVs;
-  size_t nOps = innerLoop.getBody().getOperations().size();
-
-  // Extract original inclusiveUBs
+  // Tile a single fused acc.loop that carries all associated induction
+  // variables. This keeps the tile iterations as one multi-IV "tile group"
+  // loop and the in-tile iterations as one multi-IV "element group" loop, each
+  // spanning all of its induction variables.
+  const mlir::Location loc = tileLoop.getLoc();
+  const unsigned tileCount = tileSizes.size();
+
+  llvm::SmallVector<mlir::Value, 3> origIVs(tileLoop.getBody().getArguments());
+  llvm::SmallVector<mlir::Value, 3> origUBs(tileLoop.getUpperbound());
+  llvm::SmallVector<mlir::Value, 3> origSteps(tileLoop.getStep());
+  const unsigned numIVs = origIVs.size();
+  const size_t nOps = tileLoop.getBody().getOperations().size();
+
+  // Original inclusive-UB flags (default false when the attribute is absent).
   llvm::SmallVector<bool> inclusiveUBs;
-  for (auto tileLoop : tileLoops) {
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      // inclusiveUBs are present on the IR from Fortran frontend for DO loops
-      // but might not be present from other frontends (python)
-      // So check if it exists
-      if (tileLoop.getInclusiveUpperboundAttr())
-        inclusiveUBs.push_back(
-            tileLoop.getInclusiveUpperboundAttr().asArrayRef()[j]);
-      else
-        inclusiveUBs.push_back(false);
-    }
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (tileLoop.getInclusiveUpperboundAttr())
+      inclusiveUBs.push_back(
+          tileLoop.getInclusiveUpperboundAttr().asArrayRef()[i]);
+    else
+      inclusiveUBs.push_back(false);
   }
 
-  // Extract original ivs, UBs, steps, and calculate new steps
-  rewriter.setInsertionPoint(outerLoop);
-  for (auto [i, tileLoop] : llvm::enumerate(tileLoops)) {
-    for (auto arg : tileLoop.getBody().getArguments())
-      origIVs.push_back(arg);
-    for (auto ub : tileLoop.getUpperbound())
-      origUBs.push_back(ub);
-
-    llvm::SmallVector<mlir::Value, 3> currentLoopSteps;
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      origSteps.push_back(step);
-      if (i + j >= tileSizes.size()) {
-        currentLoopSteps.push_back(step);
-      } else {
-        mlir::Value tileSize = resolveAndCastTileSize(
-            tileSizes[i + j], defaultTileSize, step.getType(), rewriter, loc);
-        auto newLoopStep =
-            mlir::arith::MulIOp::create(rewriter, loc, step, tileSize);
-        currentLoopSteps.push_back(newLoopStep);
-        newSteps.push_back(newLoopStep);
-      }
+  // Scale each tiled dimension's step by its tile size to form the tile group
+  // loop steps.
+  rewriter.setInsertionPoint(tileLoop);
+  llvm::SmallVector<mlir::Value, 3> scaledSteps;
+  llvm::SmallVector<mlir::Value, 3> tileLoopSteps;
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (i < tileCount) {
+      mlir::Value tileSize = resolveAndCastTileSize(
+          tileSizes[i], defaultTileSize, origSteps[i].getType(), rewriter, loc);
+      mlir::Value scaled =
+          mlir::arith::MulIOp::create(rewriter, loc, origSteps[i], tileSize);
+      scaledSteps.push_back(scaled);
+      tileLoopSteps.push_back(scaled);
+    } else {
+      tileLoopSteps.push_back(origSteps[i]);
     }
-
-    rewriter.startOpModification(tileLoop);
-    tileLoop.getStepMutable().clear();
-    tileLoop.getStepMutable().append(currentLoopSteps);
-    rewriter.finalizeOpModification(tileLoop);
   }
 
-  // Calculate new upper bounds for element loops
-  for (size_t i = 0; i < newSteps.size(); i++) {
-    rewriter.setInsertionPoint(innerLoop.getBody().getTerminator());
-    // UpperBound: min(origUB, origIV+(originalStep*tile_size))
-    auto stepped =
-        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], newSteps[i]);
+  // Compute the element-loop upper bounds min(origUB, origIV + scaledStep).
+  rewriter.setInsertionPoint(tileLoop.getBody().getTerminator());
+  llvm::SmallVector<mlir::Value, 3> elemLBs, elemUBs, elemSteps;
+  llvm::SmallVector<mlir::Type, 3> elemIVTypes;
+  llvm::SmallVector<bool> elemInclusiveUBs;
+  for (unsigned i = 0; i < tileCount; ++i) {
+    mlir::Value stepped =
+        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], scaledSteps[i]);
     mlir::Value newUB = stepped;
     if (inclusiveUBs[i]) {
-      // Handle InclusiveUB
-      // UpperBound: min(origUB, origIV+(originalStep*tile_size - 1))
-      auto c1 = mlir::arith::ConstantOp::create(
-          rewriter, loc, newSteps[i].getType(),
-          rewriter.getIntegerAttr(newSteps[i].getType(), 1));
+      // Inclusive UB: min(origUB, origIV + (scaledStep - 1)).
+      mlir::Value c1 = mlir::arith::ConstantOp::create(
+          rewriter, loc, scaledSteps[i].getType(),
+          rewriter.getIntegerAttr(scaledSteps[i].getType(), 1));
       newUB = mlir::arith::SubIOp::create(rewriter, loc, stepped, c1);
     }
-    newUBs.push_back(
+    elemUBs.push_back(
         mlir::arith::MinSIOp::create(rewriter, loc, origUBs[i], newUB));
+    elemLBs.push_back(origIVs[i]);
+    elemSteps.push_back(origSteps[i]);
+    elemIVTypes.push_back(origIVs[i].getType());
+    elemInclusiveUBs.push_back(inclusiveUBs[i]);
   }
 
-  // Create and insert nested elementLoopOps before terminator of outer loopOp
-  mlir::acc::LoopOp currentLoop = innerLoop;
-  for (size_t i = 0; i < tileSizes.size(); i++) {
-    rewriter.setInsertionPoint(currentLoop.getBody().getTerminator());
-    mlir::DenseBoolArrayAttr inclusiveUBAttr = mlir::DenseBoolArrayAttr{};
-    if (inclusiveUBs[i])
-      inclusiveUBAttr = rewriter.getDenseBoolArrayAttr({true});
-
-    mlir::acc::LoopOp elementLoop =
-        createInnerLoop(innerLoop, rewriter, mlir::ValueRange{origIVs[i]},
-                        mlir::ValueRange{newUBs[i]},
-                        mlir::ValueRange{origSteps[i]}, inclusiveUBAttr, loc);
-
-    // Remove vector/worker attributes from inner element loops except
-    // outermost element loop
-    if (i > 0) {
-      rewriter.startOpModification(elementLoop);
-      removeWorkerVectorFromLoop(elementLoop);
-      rewriter.finalizeOpModification(elementLoop);
-    }
-    newIVs.push_back(elementLoop.getBody().g...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211651

From llvmlistbot at llvm.org  Thu Jul 23 12:48:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 12:48:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Emit multi-IV tile and
 element loops from ACCLoopTiling (PR #211651)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211651@github.com>
Message-ID: <6a62700e.e5907290.33b622.656e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-openacc

Author: Vijay Kandiah (VijayKandiah)

<details>
<summary>Changes</summary>

`ACCLoopTiling` used to lower an N-dimensional `tile()` clause by *uncollapsing* the fused loop into a deep nest of single-IV `acc.loop`s (`tile_1 → … → tile_N → elem_1 → … → elem_N`). This discards the natural grouping of the tile clause: the 2N single-IV loops no longer express "these are the tile iterations" and "these are the in-tile iterations" as collapsible units, and gang/vector end up spread across a deep nest where only the outermost loop of each group carries the attribute. This could potentially lead to poor parallelism assignment.

With this MR:
`tileACCLoops` now rewrites the single fused `acc.loop` in-place into exactly two multi-IV loops, each carrying all the tiled induction variables:
- a **tile-group** loop whose steps are the original steps scaled by the tile sizes, and
- a nested **element-group** loop that walks the iterations within one tile, with upper bounds clamped to `min(origUB, tileStart + tileExtent)`.

Keeping each group as a single multi-IV `acc.loop` preserves the tile/element structure as two collapsible units instead of a 2N-deep single-IV nest, which is a simpler canonical form. The gang/worker/vector distribution across the two groups is unchanged from before.

This change also diagnoses `tile` + `collapse` on the same loop because these clauses give conflicting loop-association counts. The pass now emits a NYI diagnostic instead of silently dropping the `collapse` clause. Added a new invalid-input test to verify this.

With the pass no longer uncollapsing, `uncollapseLoops` and the vector-of-loops `tileACCLoops(SmallVector&lt;LoopOp&gt;&amp;, …)` overload (and the now-orphaned `createInnerLoop` helper) are deleted, along with their unit tests. New unit tests are added to cover the surviving fused-loop overload.


---

Patch is 44.54 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211651.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h (+24-29) 
- (modified) mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp (+14-18) 
- (modified) mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp (+129-216) 
- (modified) mlir/test/Dialect/OpenACC/acc-loop-tiling-invalid.mlir (+23) 
- (modified) mlir/test/Dialect/OpenACC/acc-loop-tiling.mlir (+28-57) 
- (modified) mlir/unittests/Dialect/OpenACC/OpenACCUtilsTilingTest.cpp (+35-190) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
index 6fcb706aa3488..05ac37d1106ec 100644
--- a/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
+++ b/mlir/include/mlir/Dialect/OpenACC/OpenACCUtilsTiling.h
@@ -20,23 +20,20 @@
 namespace mlir {
 namespace acc {
 
-/// Uncollapse tile loops with multiple IVs and collapseCount < tileCount.
-/// This is used to prepare loops for tiling when the collapse count is less
-/// than the tile count.
+/// Tile a single fused acc.loop that carries all associated induction
+/// variables (one IV per tile dimension).
 ///
-/// \param origLoop The original loop operation to uncollapse.
-/// \param tileCount The number of tile dimensions.
-/// \param collapseCount The collapse count from the original loop.
-/// \param rewriter The rewriter to use for modifications.
-/// \return A vector of uncollapsed loop operations.
-llvm::SmallVector<mlir::acc::LoopOp>
-uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
-                unsigned collapseCount, mlir::RewriterBase &rewriter);
-
-/// Tile ACC loops according to the given tile sizes.
+/// This produces exactly two multi-IV loops, each carrying all of the tiled
+/// induction variables:
 ///
-/// Tiling a 2-level nested loop will create two 'tile' loops containing two
-/// 'element' loops. The transformation looks like:
+///   - a "tile group" loop that steps over tiles: each step is the
+///     original step multiplied by the tile size. It keeps the original loop's
+///     gang attribute.
+///   - an "element group" loop that walks the iterations inside one tile: it
+///     keeps the original step, its lower bound is the current tile's starting
+///     index, and its upper bound is clamped to min(original upper bound,
+///     tile start + tile extent). It keeps the original loop's vector (or
+///     worker) attribute.
 ///
 /// Before Tiling:
 /// \code
@@ -48,17 +45,15 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 ///  }
 /// \endcode
 ///
-/// After Tiling:
+/// After Tiling (each group is one multi-IV loop over all tiled IVs):
 /// \code
-///  for (i = lb1; i < ub1; i += (step1 * tile_size1)) { // tile loop 1
-///    for (j = lb2; j < ub2; j += (step2 * tile_size2)) { // tile loop 2
-///      for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1) {
-///      // element loop 1
-///        for (jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2)
-///        { // element loop 2
-///          a[ii,jj] = i + j;
-///        }
-///      }
+///  // tile group
+///  for (i = lb1; i < ub1; i += (step1 * tile_size1),
+///       j = lb2; j < ub2; j += (step2 * tile_size2)) {
+///    // element group
+///    for (ii = i; ii < min(ub1, (step1 * tile_size1) + i); ii += step1,
+///         jj = j; jj < min(ub2, (step2 * tile_size2) + j); jj += step2) {
+///      a[ii,jj] = i + j;
 ///    }
 ///  }
 /// \endcode
@@ -66,13 +61,13 @@ uncollapseLoops(mlir::acc::LoopOp origLoop, unsigned tileCount,
 /// Unknown tile sizes (represented as -1 in acc dialect for `tile(*)`) are
 /// resolved to the provided default tile size.
 ///
-/// \param tileLoops The loops to tile (outermost first).
-/// \param tileSizes The tile sizes for each dimension. Values of -1 are
+/// \param tileLoop The fused loop to tile.
+/// \param tileSizes The tile sizes for each tiled dimension. Values of -1 are
 ///        treated as unknown and resolved to defaultTileSize.
 /// \param defaultTileSize The default tile size to use for unknown (*) tiles.
 /// \param rewriter The rewriter to use for modifications.
-/// \return The outermost loop after tiling.
-mlir::acc::LoopOp tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+/// \return The tile group loop that is modified in place.
+mlir::acc::LoopOp tileACCLoops(mlir::acc::LoopOp tileLoop,
                                const llvm::SmallVector<mlir::Value> &tileSizes,
                                int32_t defaultTileSize,
                                mlir::RewriterBase &rewriter);
diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
index 6bc95ca896f37..d5865bc98727e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCLoopTiling.cpp
@@ -162,8 +162,16 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
 
     SmallVector<Value> tileSizes(origLoop.getTileValues().begin(),
                                  origLoop.getTileValues().end());
-    unsigned tileCount = tileSizes.size();
-    unsigned collapseCount = origLoop.getCollapseValue().value_or(1);
+
+    // A tile clause and a collapse clause on the same loop are not supported:
+    // they give conflicting descriptions of how many loops the construct
+    // associates.
+    if (origLoop.getCollapseAttr()) {
+      accSupport.emitNYI(origLoop.getLoc(),
+                         "a tile clause combined with a collapse clause on the "
+                         "same loop");
+      return failure();
+    }
 
     // Sanity check tile size types
     if (failed(checkTileSizeTypes(origLoop, tileSizes)))
@@ -182,24 +190,12 @@ struct ACCLoopTilingImpl : public OpRewritePattern<acc::LoopOp> {
     origLoop.removeTileOperandsDeviceTypeAttr();
     rewriter.finalizeOpModification(origLoop);
 
-    SmallVector<acc::LoopOp> loopsToTile;
-    if (collapseCount < tileCount) {
-      // Uncollapse tile loops before tiling if necessary
-      loopsToTile =
-          acc::uncollapseLoops(origLoop, tileCount, collapseCount, rewriter);
-      rewriter.replaceOp(origLoop, loopsToTile[0]);
-      LLVM_DEBUG(llvm::dbgs() << "\nAfter uncollapsing:\n"
-                              << *loopsToTile[0] << "\n");
-    } else {
-      loopsToTile.push_back(origLoop);
-    }
-
-    // loopsToTile is a vector of perfectly nested loops. The outermost loop
-    // may have multiple IVs but inner loops can only have one IV.
+    // Tile the single fused loop in place: the tile iterations become one
+    // multi-IV "tile group" loop wrapping one multi-IV "element group" loop.
     // The utility handles unknown tile sizes (*) by using `defaultTileSize`.
-    acc::tileACCLoops(loopsToTile, tileSizes, defaultTileSize, rewriter);
+    acc::tileACCLoops(origLoop, tileSizes, defaultTileSize, rewriter);
 
-    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *loopsToTile[0] << "\n");
+    LLVM_DEBUG(llvm::dbgs() << "\nAfter tiling:\n " << *origLoop << "\n");
     return success();
   }
 
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
index 09e83af2bc686..32fd3957894fb 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsTiling.cpp
@@ -17,6 +17,7 @@
 #include "mlir/Dialect/OpenACC/OpenACC.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
 #include "mlir/Transforms/RegionUtils.h"
+#include "llvm/ADT/STLExtras.h"
 
 // Resolve unknown tile sizes (represented as -1 for tile(*)) to the default.
 // Returns a value with the same type as targetType.
@@ -48,19 +49,13 @@ static void removeWorkerVectorFromLoop(mlir::acc::LoopOp loop) {
 }
 
 // Create a new ACC loop with new steps, lb, ub from original loop
-static mlir::acc::LoopOp
-createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
-                          mlir::RewriterBase &rewriter, mlir::ValueRange lb,
-                          mlir::ValueRange ub, mlir::ValueRange step,
-                          mlir::DenseBoolArrayAttr inclusiveUBAttr,
-                          mlir::acc::CombinedConstructsTypeAttr combinedAttr,
-                          mlir::Location loc, bool preserveCollapse) {
+static mlir::acc::LoopOp createACCLoopFromOriginal(
+    mlir::acc::LoopOp origLoop, mlir::RewriterBase &rewriter,
+    mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
+    mlir::DenseBoolArrayAttr inclusiveUBAttr,
+    mlir::acc::CombinedConstructsTypeAttr combinedAttr, mlir::Location loc) {
   mlir::ArrayAttr collapseAttr = mlir::ArrayAttr{};
   mlir::ArrayAttr collapseDeviceTypeAttr = mlir::ArrayAttr{};
-  if (preserveCollapse) {
-    collapseAttr = origLoop.getCollapseAttr();
-    collapseDeviceTypeAttr = origLoop.getCollapseDeviceTypeAttr();
-  }
   auto newLoop = mlir::acc::LoopOp::create(
       rewriter, loc, origLoop->getResultTypes(), lb, ub, step, inclusiveUBAttr,
       collapseAttr, collapseDeviceTypeAttr, origLoop.getGangOperands(),
@@ -78,53 +73,6 @@ createACCLoopFromOriginal(mlir::acc::LoopOp origLoop,
   return newLoop;
 }
 
-// Create inner loop inside input loop
-static mlir::acc::LoopOp
-createInnerLoop(mlir::acc::LoopOp inputLoop, mlir::RewriterBase &rewriter,
-                mlir::ValueRange lb, mlir::ValueRange ub, mlir::ValueRange step,
-                mlir::DenseBoolArrayAttr inclusiveUBAttr, mlir::Location loc) {
-  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
-      inputLoop, rewriter, lb, ub, step, inclusiveUBAttr,
-      mlir::acc::CombinedConstructsTypeAttr{}, loc, /*preserveCollapse*/ false);
-
-  // Remove gang/worker attributes from inner loops
-  rewriter.startOpModification(elementLoop);
-  if (inputLoop.hasGang() ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Num) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
-      inputLoop.getGangValue(mlir::acc::GangArgType::Static)) {
-    elementLoop.removeGangAttr();
-    elementLoop.removeGangOperandsArgTypeAttr();
-    elementLoop.removeGangOperandsSegmentsAttr();
-    elementLoop.removeGangOperandsDeviceTypeAttr();
-    // Also drop the operand values themselves so that elementLoop does not
-    // end up with a non-empty gang operand list but no corresponding
-    // device-type/segment/arg-type attributes. Leaving stale operands behind
-    // makes elementLoop look like it still has gang operands to later
-    // queries (e.g. LoopOp::getGangValue), which then dereference the
-    // now-missing device-type attribute.
-    elementLoop.getGangOperandsMutable().clear();
-  }
-  if (inputLoop.hasVector() || inputLoop.getVectorValue()) {
-    elementLoop.removeWorkerAttr();
-    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
-    // As above for gang, also drop the worker operand values so elementLoop
-    // does not keep a dangling worker operand with no device-type attribute.
-    elementLoop.getWorkerNumOperandsMutable().clear();
-  }
-  rewriter.finalizeOpModification(elementLoop);
-
-  // Create empty block in elementLoop and add IV argument
-  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
-                                          elementLoop.getRegion().begin());
-  rewriter.setInsertionPointToEnd(blk);
-  mlir::acc::YieldOp::create(rewriter, loc);
-  elementLoop.getBody().addArgument(
-      inputLoop.getBody().getArgument(0).getType(), loc);
-
-  return elementLoop;
-}
-
 // Move ops from source to target Loop and replace uses of IVs
 static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
                                  mlir::acc::LoopOp targetLoop,
@@ -159,180 +107,145 @@ static void moveOpsAndReplaceIVs(mlir::acc::LoopOp sourceLoop,
     rewriter.finalizeOpModification(op);
 }
 
+// Create a single "element group" loop nested in `tileLoop`, carrying
+// `ivTypes.size()` induction variables so the whole element space is one
+// multi-IV loop. The element group carries vector or worker but not gang.
+static mlir::acc::LoopOp
+createElementGroupLoop(mlir::acc::LoopOp tileLoop, mlir::RewriterBase &rewriter,
+                       mlir::ValueRange lbs, mlir::ValueRange ubs,
+                       mlir::ValueRange steps,
+                       mlir::DenseBoolArrayAttr inclusiveUBAttr,
+                       llvm::ArrayRef<mlir::Type> ivTypes, mlir::Location loc) {
+  mlir::acc::LoopOp elementLoop = createACCLoopFromOriginal(
+      tileLoop, rewriter, lbs, ubs, steps, inclusiveUBAttr,
+      mlir::acc::CombinedConstructsTypeAttr{}, loc);
+
+  // Drop gang from the element group, keeping vector/worker. The operand
+  // values must be cleared too, not just the attributes.
+  rewriter.startOpModification(elementLoop);
+  if (tileLoop.hasGang() ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Num) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Dim) ||
+      tileLoop.getGangValue(mlir::acc::GangArgType::Static)) {
+    elementLoop.removeGangAttr();
+    elementLoop.removeGangOperandsArgTypeAttr();
+    elementLoop.removeGangOperandsSegmentsAttr();
+    elementLoop.removeGangOperandsDeviceTypeAttr();
+    elementLoop.getGangOperandsMutable().clear();
+  }
+  if (tileLoop.hasVector() || tileLoop.getVectorValue()) {
+    elementLoop.removeWorkerAttr();
+    elementLoop.removeWorkerNumOperandsDeviceTypeAttr();
+    elementLoop.getWorkerNumOperandsMutable().clear();
+  }
+  rewriter.finalizeOpModification(elementLoop);
+
+  // Create the element loop body: one block argument per IV plus a terminator.
+  mlir::Block *blk = rewriter.createBlock(&elementLoop.getRegion(),
+                                          elementLoop.getRegion().begin());
+  rewriter.setInsertionPointToEnd(blk);
+  mlir::acc::YieldOp::create(rewriter, loc);
+  for (mlir::Type ivType : ivTypes)
+    elementLoop.getBody().addArgument(ivType, loc);
+
+  return elementLoop;
+}
+
 mlir::acc::LoopOp
-mlir::acc::tileACCLoops(llvm::SmallVector<mlir::acc::LoopOp> &tileLoops,
+mlir::acc::tileACCLoops(mlir::acc::LoopOp tileLoop,
                         const llvm::SmallVector<mlir::Value> &tileSizes,
                         int32_t defaultTileSize, mlir::RewriterBase &rewriter) {
-  // Tile collapsed and/or nested loops
-  mlir::acc::LoopOp outerLoop = tileLoops[0];
-  const mlir::Location loc = outerLoop.getLoc();
-
-  mlir::acc::LoopOp innerLoop = tileLoops[tileLoops.size() - 1];
-  llvm::SmallVector<mlir::Value, 3> origIVs;
-  llvm::SmallVector<mlir::Value, 3> origSteps;
-  llvm::SmallVector<mlir::Value, 3> origUBs;
-  llvm::SmallVector<mlir::Value, 3> newSteps;
-  llvm::SmallVector<mlir::Value, 3> newUBs;
-  llvm::SmallVector<mlir::Value, 3> newIVs;
-  size_t nOps = innerLoop.getBody().getOperations().size();
-
-  // Extract original inclusiveUBs
+  // Tile a single fused acc.loop that carries all associated induction
+  // variables. This keeps the tile iterations as one multi-IV "tile group"
+  // loop and the in-tile iterations as one multi-IV "element group" loop, each
+  // spanning all of its induction variables.
+  const mlir::Location loc = tileLoop.getLoc();
+  const unsigned tileCount = tileSizes.size();
+
+  llvm::SmallVector<mlir::Value, 3> origIVs(tileLoop.getBody().getArguments());
+  llvm::SmallVector<mlir::Value, 3> origUBs(tileLoop.getUpperbound());
+  llvm::SmallVector<mlir::Value, 3> origSteps(tileLoop.getStep());
+  const unsigned numIVs = origIVs.size();
+  const size_t nOps = tileLoop.getBody().getOperations().size();
+
+  // Original inclusive-UB flags (default false when the attribute is absent).
   llvm::SmallVector<bool> inclusiveUBs;
-  for (auto tileLoop : tileLoops) {
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      // inclusiveUBs are present on the IR from Fortran frontend for DO loops
-      // but might not be present from other frontends (python)
-      // So check if it exists
-      if (tileLoop.getInclusiveUpperboundAttr())
-        inclusiveUBs.push_back(
-            tileLoop.getInclusiveUpperboundAttr().asArrayRef()[j]);
-      else
-        inclusiveUBs.push_back(false);
-    }
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (tileLoop.getInclusiveUpperboundAttr())
+      inclusiveUBs.push_back(
+          tileLoop.getInclusiveUpperboundAttr().asArrayRef()[i]);
+    else
+      inclusiveUBs.push_back(false);
   }
 
-  // Extract original ivs, UBs, steps, and calculate new steps
-  rewriter.setInsertionPoint(outerLoop);
-  for (auto [i, tileLoop] : llvm::enumerate(tileLoops)) {
-    for (auto arg : tileLoop.getBody().getArguments())
-      origIVs.push_back(arg);
-    for (auto ub : tileLoop.getUpperbound())
-      origUBs.push_back(ub);
-
-    llvm::SmallVector<mlir::Value, 3> currentLoopSteps;
-    for (auto [j, step] : llvm::enumerate(tileLoop.getStep())) {
-      origSteps.push_back(step);
-      if (i + j >= tileSizes.size()) {
-        currentLoopSteps.push_back(step);
-      } else {
-        mlir::Value tileSize = resolveAndCastTileSize(
-            tileSizes[i + j], defaultTileSize, step.getType(), rewriter, loc);
-        auto newLoopStep =
-            mlir::arith::MulIOp::create(rewriter, loc, step, tileSize);
-        currentLoopSteps.push_back(newLoopStep);
-        newSteps.push_back(newLoopStep);
-      }
+  // Scale each tiled dimension's step by its tile size to form the tile group
+  // loop steps.
+  rewriter.setInsertionPoint(tileLoop);
+  llvm::SmallVector<mlir::Value, 3> scaledSteps;
+  llvm::SmallVector<mlir::Value, 3> tileLoopSteps;
+  for (unsigned i = 0; i < numIVs; ++i) {
+    if (i < tileCount) {
+      mlir::Value tileSize = resolveAndCastTileSize(
+          tileSizes[i], defaultTileSize, origSteps[i].getType(), rewriter, loc);
+      mlir::Value scaled =
+          mlir::arith::MulIOp::create(rewriter, loc, origSteps[i], tileSize);
+      scaledSteps.push_back(scaled);
+      tileLoopSteps.push_back(scaled);
+    } else {
+      tileLoopSteps.push_back(origSteps[i]);
     }
-
-    rewriter.startOpModification(tileLoop);
-    tileLoop.getStepMutable().clear();
-    tileLoop.getStepMutable().append(currentLoopSteps);
-    rewriter.finalizeOpModification(tileLoop);
   }
 
-  // Calculate new upper bounds for element loops
-  for (size_t i = 0; i < newSteps.size(); i++) {
-    rewriter.setInsertionPoint(innerLoop.getBody().getTerminator());
-    // UpperBound: min(origUB, origIV+(originalStep*tile_size))
-    auto stepped =
-        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], newSteps[i]);
+  // Compute the element-loop upper bounds min(origUB, origIV + scaledStep).
+  rewriter.setInsertionPoint(tileLoop.getBody().getTerminator());
+  llvm::SmallVector<mlir::Value, 3> elemLBs, elemUBs, elemSteps;
+  llvm::SmallVector<mlir::Type, 3> elemIVTypes;
+  llvm::SmallVector<bool> elemInclusiveUBs;
+  for (unsigned i = 0; i < tileCount; ++i) {
+    mlir::Value stepped =
+        mlir::arith::AddIOp::create(rewriter, loc, origIVs[i], scaledSteps[i]);
     mlir::Value newUB = stepped;
     if (inclusiveUBs[i]) {
-      // Handle InclusiveUB
-      // UpperBound: min(origUB, origIV+(originalStep*tile_size - 1))
-      auto c1 = mlir::arith::ConstantOp::create(
-          rewriter, loc, newSteps[i].getType(),
-          rewriter.getIntegerAttr(newSteps[i].getType(), 1));
+      // Inclusive UB: min(origUB, origIV + (scaledStep - 1)).
+      mlir::Value c1 = mlir::arith::ConstantOp::create(
+          rewriter, loc, scaledSteps[i].getType(),
+          rewriter.getIntegerAttr(scaledSteps[i].getType(), 1));
       newUB = mlir::arith::SubIOp::create(rewriter, loc, stepped, c1);
     }
-    newUBs.push_back(
+    elemUBs.push_back(
         mlir::arith::MinSIOp::create(rewriter, loc, origUBs[i], newUB));
+    elemLBs.push_back(origIVs[i]);
+    elemSteps.push_back(origSteps[i]);
+    elemIVTypes.push_back(origIVs[i].getType());
+    elemInclusiveUBs.push_back(inclusiveUBs[i]);
   }
 
-  // Create and insert nested elementLoopOps before terminator of outer loopOp
-  mlir::acc::LoopOp currentLoop = innerLoop;
-  for (size_t i = 0; i < tileSizes.size(); i++) {
-    rewriter.setInsertionPoint(currentLoop.getBody().getTerminator());
-    mlir::DenseBoolArrayAttr inclusiveUBAttr = mlir::DenseBoolArrayAttr{};
-    if (inclusiveUBs[i])
-      inclusiveUBAttr = rewriter.getDenseBoolArrayAttr({true});
-
-    mlir::acc::LoopOp elementLoop =
-        createInnerLoop(innerLoop, rewriter, mlir::ValueRange{origIVs[i]},
-                        mlir::ValueRange{newUBs[i]},
-                        mlir::ValueRange{origSteps[i]}, inclusiveUBAttr, loc);
-
-    // Remove vector/worker attributes from inner element loops except
-    // outermost element loop
-    if (i > 0) {
-      rewriter.startOpModification(elementLoop);
-      removeWorkerVectorFromLoop(elementLoop);
-      rewriter.finalizeOpModification(elementLoop);
-    }
-    newIVs.push_back(elementLoop.getBody().g...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211651

From llvmlistbot at llvm.org  Thu Jul 23 12:53:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 12:53:12 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a627128.1b0626d0.1ff4c3.3ff7@mx.google.com>


================
@@ -5096,6 +5424,46 @@ convertOmpLoopNest(Operation &opInst, llvm::IRBuilderBase &builder,
       computeIP = loopInfos.front()->getPreheaderIP();
     }
 
+    // If this loop is the worksharing loop of a scan reduction, generate a
+    // pair of canonical loops (an input loop and a scan loop) instead of a
+    // single loop. The `scan` directive in the body is translated against the
+    // `ScanInfo` recorded here.
+    bool isInScanRegion = false;
+    if (auto wsloopOp = loopOp->getParentOfType<omp::WsloopOp>())
+      isInScanRegion =
+          wsloopOp.getReductionMod() && (wsloopOp.getReductionMod().value() ==
+                                         mlir::omp::ReductionModifier::inscan);
+    if (isInScanRegion) {
+      llvm::Expected<llvm::ScanInfo *> res = ompBuilder->scanInfoInitialize();
+      if (failed(handleError(res, *loopOp)))
+        return failure();
+      llvm::ScanInfo *scanInfo = res.get();
+      moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+          [&](OpenMPLoopInfoStackFrame &frame) {
+            frame.scanInfo = scanInfo;
+            return WalkResult::interrupt();
+          });
+      llvm::Expected<llvm::SmallVector<llvm::CanonicalLoopInfo *>> loopResults =
+          ompBuilder->createCanonicalScanLoops(
----------------
chichunchen wrote:

Could we either reject non-i32 loop types here or make the scan implementation use a consistent index type?
Flang emits i32 loop bounds for default INTEGER and i64 bounds for INTEGER(KIND=8). LoopNestOp::verify() permits both, so an i64 Span can reach this call.
The existing i32 scan test succeeds, while the same valid scan using an i64 induction variable aborts in emitScanBasedDirectiveDeclsIR.

Regression generated by codex and I can verify it triggered assert:

```
  Assertion `C1->getType() == C2->getType() &&
  "Operand types in binary constant expression should match"' failed.
```

```
  program scan_i64_abort
    implicit none
    integer(kind=8) :: i
    integer :: x, result(4)

    x = 0
    !$omp parallel do reduction(inscan, +: x)
    do i = 1_8, 4_8
      x = x + 1
      !$omp scan inclusive(x)
      result(i) = x
    end do
  end program
```

https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Thu Jul 23 12:54:14 2026
From: llvmlistbot at llvm.org (Valentin Clement
 =?utf-8?b?44OQ44Os44Oz44K/44Kk44OzIOOCr+ODrOODoeODsw==?=)
Date: Thu, 23 Jul 2026 12:54:14 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir][acc] Allow for existing
 declare globals in GPU module under unified memory. (PR #211650)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211650@github.com>
Message-ID: <6a627166.4987c784.13f2de.0a48@mx.google.com>


https://github.com/clementval approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/211650

From llvmlistbot at llvm.org  Thu Jul 23 13:49:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 13:49:13 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a627e49.c56fb102.6921f.6f95@mx.google.com>


================
@@ -670,6 +707,97 @@ findCurrentLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
   return loopInfo;
 }
 
+/// Find the scan loop information structure for the scan loop nest being
+/// translated. It will return a `null` value unless called from the
+/// translation function for a loop wrapper operation after successfully
+/// translating its body.
+static llvm::CanonicalLoopInfo *
+findCurrentScanLoopInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::CanonicalLoopInfo *scanLoopInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanLoopInfo = frame.scanloopInfo;
+        return WalkResult::interrupt();
+      });
+  return scanLoopInfo;
+}
+
+/// Find the `ScanInfo` stored on the loop stack frame. Upon encountering an
+/// `inscan` reduction modifier, `scanInfoInitialize` initializes the
+/// `ScanInfo`, which is then used when a `scan` directive is encountered in the
+/// body of the loop nest.
+static llvm::ScanInfo *
+findScanInfo(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::ScanInfo *scanInfo = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        scanInfo = frame.scanInfo;
+        return WalkResult::interrupt();
+      });
+  return scanInfo;
+}
+
+/// The types of reduction variables are used for lowering a `scan` directive
+/// that appears in the body of the loop. The types are stored in the loop frame
+/// when the reduction clause is encountered and used when the `scan` directive
+/// is encountered.
+static llvm::DenseMap<llvm::Value *, llvm::Type *> *
+findReductionVarTypes(LLVM::ModuleTranslation &moduleTranslation) {
+  llvm::DenseMap<llvm::Value *, llvm::Type *> *reductionVarToType = nullptr;
+  moduleTranslation.stackWalk<OpenMPLoopInfoStackFrame>(
+      [&](OpenMPLoopInfoStackFrame &frame) {
+        if (!frame.reductionVarToType)
+          frame.reductionVarToType =
+              std::make_unique<llvm::DenseMap<llvm::Value *, llvm::Type *>>();
+        reductionVarToType = frame.reductionVarToType.get();
+        return WalkResult::interrupt();
+      });
+  return reductionVarToType;
+}
+
+/// Scan reduction requires a shared buffer to be allocated to perform the
+/// reduction. The allocation needs to be done outside the parallel region in
+/// which the scan operation is used.
+static llvm::OpenMPIRBuilder::InsertPointTy
+findParallelAllocaIP(llvm::IRBuilderBase &builder,
----------------
chichunchen wrote:

Orphand worksharing loop seems not handled properly. findParallelAllocaIp() cannot find the enclosing omp.parallel and fallback to the entry block of run_scan.

```
  subroutine run_scan(result, x, n)
    implicit none
    integer :: n, result(n), x, i

    !$omp do reduction(inscan, +: x)
    do i = 1, n
      x = x + 1
      !$omp scan inclusive(x)
      result(i) = x
    end do
    !$omp end do
  end subroutine

  program example
    implicit none
    integer :: x, result(8)

    x = 0

    !$omp parallel shared(x, result)
    call run_scan(result, x, 8)
    !$omp end parallel
  end program
```

https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Thu Jul 23 13:49:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 13:49:51 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a627e6f.0c05e68e.81651.538a@mx.google.com>


https://github.com/chichunchen edited https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Thu Jul 23 13:50:50 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Thu, 23 Jul 2026 13:50:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a627eaa.1ba19891.1d5373.22d8@mx.google.com>


https://github.com/beamandala edited https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Thu Jul 23 13:51:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 13:51:00 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a627eb4.507fc713.3702e2.6943@mx.google.com>


https://github.com/chichunchen requested changes to this pull request.


https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Thu Jul 23 13:52:55 2026
From: llvmlistbot at llvm.org (Leandro Lupori)
Date: Thu, 23 Jul 2026 13:52:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a627f27.8e75e0cb.2de827.e1e9@mx.google.com>


https://github.com/luporl updated https://github.com/llvm/llvm-project/pull/183800

>From 49d7a2054c0b123a6783a49383c71066b7e1d1d9 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Fri, 27 Feb 2026 15:32:58 -0300
Subject: [PATCH 1/5] [mlir][OpenMP] Fix update of linear iteration variables

The final value of a linear iteration variable must be the loop
limit_value + step. Before this patch it was limit_value.

This fixes the second issue reported in #170784.
---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 129 ++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  15 +-
 2 files changed, 103 insertions(+), 41 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index deefa29c157f3..993cd89f74300 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -150,6 +150,37 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearFinalizationBB;
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
+  Value linearLoopIV;
+  Value linearLoopIVStart;
+
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
+                       llvm::Value *var, llvm::Value *varStart,
+                       llvm::Value *step, llvm::Value *iv) {
+    if (!iv->getType()->isIntegerTy())
+      llvm_unreachable("OpenMP loop induction variable must be an integer "
+                       "type");
+
+    if (varType->isIntegerTy()) {
+      // Integer path: normalize all arithmetic to linearVarType
+      iv = builder.CreateSExtOrTrunc(iv, varType);
+      step = builder.CreateSExtOrTrunc(step, varType);
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
+      builder.CreateStore(addInst, var);
+    } else if (varType->isFloatingPointTy()) {
+      // Float path: perform multiply in integer, then convert to float
+      step = builder.CreateSExtOrTrunc(step, iv->getType());
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
+      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
+      builder.CreateStore(addInst, var);
+    } else {
+      llvm_unreachable(
+          "Linear variable must be of integer or floating-point type");
+    }
+  }
 
 public:
   // Register type for the linear variables
@@ -189,46 +220,63 @@ class LinearClauseProcessor {
     }
   }
 
+  // Find linear iteration variable and save it for later updates
+  void initLinearIV(omp::SimdOp simdOp) {
+    auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
+    // NOTE iteration variables can only be linear in non-nested loops.
+    if (loopOp.getIVs().size() != 1)
+      return;
+    // The linear IV is the loop IV's store address.
+    BlockArgument arg = loopOp.getIVs().front();
+    for (const Operation *user : arg.getUsers()) {
+      if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
+        for (Value linearVar : simdOp.getLinearVars()) {
+          if (linearVar == storeOp.getAddr()) {
+            linearLoopIV = linearVar;
+            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
+            break;
+          }
+        }
+      }
+    }
+  }
+
   // Emit IR for updating Linear variables
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
-                       llvm::Value *loopInductionVar) {
+  void updateLinearVars(llvm::IRBuilderBase &builder,
+                        llvm::BasicBlock *loopBody,
+                        llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::Type *linearVarType = linearVarTypes[index];
-      llvm::Value *iv = loopInductionVar;
-      llvm::Value *step = linearSteps[index];
-
-      if (!iv->getType()->isIntegerTy())
-        llvm_unreachable("OpenMP loop induction variable must be an integer "
-                         "type");
-
-      if (linearVarType->isIntegerTy()) {
-        // Integer path: normalize all arithmetic to linearVarType
-        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
-        step = builder.CreateSExtOrTrunc(step, linearVarType);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else if (linearVarType->isFloatingPointTy()) {
-        // Float path: perform multiply in integer, then convert to float
-        step = builder.CreateSExtOrTrunc(step, iv->getType());
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
-        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else {
-        llvm_unreachable(
-            "Linear variable must be of integer or floating-point type");
-      }
+      llvm::LoadInst *linearVarStart = builder.CreateLoad(
+          linearVarTypes[index], linearPreconditionVars[index]);
+      updateLinearVar(builder, linearVarTypes[index],
+                      linearLoopBodyTemps[index], linearVarStart,
+                      linearSteps[index], loopInductionVar);
     }
   }
 
+  // Emit IR for updating linear iteration variables on loop exit
+  void updateLinearIV(llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation,
+                      llvm::Value *loopIV) {
+    if (!linearLoopIV)
+      return;
+    llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
+    llvm::Value *linearIVStart =
+        moduleTranslation.lookupValue(linearLoopIVStart);
+
+    // Find linearIV's index
+    size_t index;
+    for (index = 0; index < linearOrigVal.size(); index++)
+      if (linearIV == linearOrigVal[index])
+        break;
+    if (index == linearOrigVal.size())
+      return;
+
+    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
+                    linearIVStart, linearSteps[index], loopIV);
+  }
+
   // Linear variable finalization is conditional on the last logical iteration.
   // Create BB splits to manage the same.
   void splitLinearFiniBB(llvm::IRBuilderBase &builder,
@@ -3852,8 +3900,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4153,6 +4201,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
 
+  linearClauseProcessor.initLinearIV(simdOp);
+
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
     for (mlir::Attribute linearVarType : linearVarTypes)
@@ -4250,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4261,6 +4311,9 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
+                                       loopInfo->getIndVar());
+
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 38f10320bd3ae..45eb6ac1ef957 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,6 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
+// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -758,8 +759,16 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK: %[[ADD:.*]] = add i32 %[[LOAD]], %[[MUL]]
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
+
+// CHECK: omp.region.cont:
+// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
+// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
+// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
+// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
+// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
       omp.yield
     }
   } {linear_var_types = [i32]}
@@ -784,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -817,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
-// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group

>From 20ac4aef52312cd93944ad17f0fef0b126211102 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 19 May 2026 12:06:23 -0300
Subject: [PATCH 2/5] Fix "instruction does not dominate all uses" issue

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 104 +++++++++---------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++++--
 2 files changed, 90 insertions(+), 62 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 993cd89f74300..aace5283e4362 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -151,36 +151,6 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
   Value linearLoopIV;
-  Value linearLoopIVStart;
-
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
-                       llvm::Value *var, llvm::Value *varStart,
-                       llvm::Value *step, llvm::Value *iv) {
-    if (!iv->getType()->isIntegerTy())
-      llvm_unreachable("OpenMP loop induction variable must be an integer "
-                       "type");
-
-    if (varType->isIntegerTy()) {
-      // Integer path: normalize all arithmetic to linearVarType
-      iv = builder.CreateSExtOrTrunc(iv, varType);
-      step = builder.CreateSExtOrTrunc(step, varType);
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
-      builder.CreateStore(addInst, var);
-    } else if (varType->isFloatingPointTy()) {
-      // Float path: perform multiply in integer, then convert to float
-      step = builder.CreateSExtOrTrunc(step, iv->getType());
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
-      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
-      builder.CreateStore(addInst, var);
-    } else {
-      llvm_unreachable(
-          "Linear variable must be of integer or floating-point type");
-    }
-  }
 
 public:
   // Register type for the linear variables
@@ -233,7 +203,6 @@ class LinearClauseProcessor {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
             linearLoopIV = linearVar;
-            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
             break;
           }
         }
@@ -242,28 +211,51 @@ class LinearClauseProcessor {
   }
 
   // Emit IR for updating Linear variables
-  void updateLinearVars(llvm::IRBuilderBase &builder,
-                        llvm::BasicBlock *loopBody,
-                        llvm::Value *loopInductionVar) {
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
+                       llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::LoadInst *linearVarStart = builder.CreateLoad(
-          linearVarTypes[index], linearPreconditionVars[index]);
-      updateLinearVar(builder, linearVarTypes[index],
-                      linearLoopBodyTemps[index], linearVarStart,
-                      linearSteps[index], loopInductionVar);
+      llvm::Type *linearVarType = linearVarTypes[index];
+      llvm::Value *iv = loopInductionVar;
+      llvm::Value *step = linearSteps[index];
+
+      if (!iv->getType()->isIntegerTy())
+        llvm_unreachable("OpenMP loop induction variable must be an integer "
+                         "type");
+
+      if (linearVarType->isIntegerTy()) {
+        // Integer path: normalize all arithmetic to linearVarType
+        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
+        step = builder.CreateSExtOrTrunc(step, linearVarType);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else if (linearVarType->isFloatingPointTy()) {
+        // Float path: perform multiply in integer, then convert to float
+        step = builder.CreateSExtOrTrunc(step, iv->getType());
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
+        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else {
+        llvm_unreachable(
+            "Linear variable must be of integer or floating-point type");
+      }
     }
   }
 
   // Emit IR for updating linear iteration variables on loop exit
   void updateLinearIV(llvm::IRBuilderBase &builder,
-                      LLVM::ModuleTranslation &moduleTranslation,
-                      llvm::Value *loopIV) {
+                      LLVM::ModuleTranslation &moduleTranslation) {
     if (!linearLoopIV)
       return;
     llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
-    llvm::Value *linearIVStart =
-        moduleTranslation.lookupValue(linearLoopIVStart);
 
     // Find linearIV's index
     size_t index;
@@ -273,8 +265,17 @@ class LinearClauseProcessor {
     if (index == linearOrigVal.size())
       return;
 
-    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
-                    linearIVStart, linearSteps[index], loopIV);
+    // Add one more step to the linear iteration variable
+    llvm::Type *varType = linearVarTypes[index];
+    llvm::Value *var = linearLoopBodyTemps[index];
+    llvm::Value *step = linearSteps[index];
+    if (!varType->isIntegerTy())
+      llvm_unreachable("Linear iteration variable must be of integer type");
+
+    step = builder.CreateSExtOrTrunc(step, varType);
+    llvm::Value *val = builder.CreateLoad(varType, var);
+    llvm::Value *addInst = builder.CreateAdd(val, step);
+    builder.CreateStore(addInst, var);
   }
 
   // Linear variable finalization is conditional on the last logical iteration.
@@ -3900,8 +3901,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4200,7 +4201,6 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-
   linearClauseProcessor.initLinearIV(simdOp);
 
   if (!simdOp.getLinearVars().empty()) {
@@ -4300,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4311,9 +4311,7 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
-  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
-                                       loopInfo->getIndVar());
-
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation);
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 45eb6ac1ef957..51a12e23a0e5e 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,7 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
-// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
+// CHECK-SAME: (i32 %{{.*}}, i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -761,11 +761,11 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
 
 // CHECK: omp.region.cont:
-// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
-// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
-// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
-// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
-// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], %[[STEP]]
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       llvm.store %iv, %x : i32, !llvm.ptr
@@ -793,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
+// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -826,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
-// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group
@@ -841,6 +841,36 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // -----
 
+// Test the update of omp.simd linear iteration variables, when nested inside
+// omp.wsloop.
+llvm.func @wsloop_simd_linear(%x : !llvm.ptr) {
+
+// CHECK-LABEL: @wsloop_simd_linear
+
+// CHECK: omp.wsloop.region:
+// CHECK: %[[LINEAR_VAR:.*]] = alloca i32
+// CHECK: %[[LINEAR_RESULT:.*]] = alloca i32
+
+// CHECK: omp.region.cont2:
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], 25
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+  %lb = llvm.mlir.constant(1 : i32) : i32
+  %ub = llvm.mlir.constant(100 : i32) : i32
+  %step = llvm.mlir.constant(25 : i32) : i32
+  omp.wsloop {
+    omp.simd linear(%x : !llvm.ptr = %step : i32) {
+      omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+        llvm.store %iv, %x : i32, !llvm.ptr
+        omp.yield
+      }
+    } {linear_var_types = [i32], omp.composite}
+  } {omp.composite}
+  llvm.return
+}
+
+// -----
+
 // CHECK-LABEL: @simd_simple_multiple
 llvm.func @simd_simple_multiple(%lb1 : i64, %ub1 : i64, %step1 : i64, %lb2 : i64, %ub2 : i64, %step2 : i64, %arg0: !llvm.ptr, %arg1: !llvm.ptr) {
   omp.simd {

>From 030a957a941fb50d15c99f9d0031e26019b53992 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 09:58:04 -0300
Subject: [PATCH 3/5] Verify that linear variables don't appear in other DSA
 clauses

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp     | 10 ++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir | 14 ++++++++++++++
 mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir | 12 +++++-------
 3 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..b328e15313538 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -3363,6 +3363,16 @@ LogicalResult SimdOp::verify() {
   if (getLinearVars().size() &&
       getLinearVarTypes().value().size() != getLinearVars().size())
     return emitError() << "Ill-formed type attributes for linear variables";
+
+  llvm::DenseSet<Value> privateVars(llvm::from_range, getPrivateVars());
+  llvm::DenseSet<Value> reductionVars(llvm::from_range, getReductionVars());
+  // TODO Check lastprivate vars when their support is added to SimdOp.
+  for (Value var : getLinearVars()) {
+    if (privateVars.contains(var) || reductionVars.contains(var))
+      return emitOpError()
+             << "linear variables cannot appear in other data-sharing clauses";
+  }
+
   return success();
 }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index d9459ace54422..7e2e0061f80be 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -116,6 +116,20 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // -----
 
+omp.private {type = private} @i_private_i32 : i32
+llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr) {
+  // expected-error @below {{linear variables cannot appear in other data-sharing clauses}}
+  omp.simd linear(%i : !llvm.ptr = %step : i32)
+           private(@i_private_i32 %i -> %priv_i : !llvm.ptr) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      omp.yield
+    }
+  } {linear_var_types = [i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
index 8da1fab1f48af..a84c51f08daa7 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @simd_ordered_linear
   llvm.func @simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -38,14 +36,14 @@ module {
     // CHECK: %.linear_var = alloca i32
     // CHECK: %.linear_result = alloca i32
 
-    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c10_i32) inclusive step (%c1_i32) {
         // CHECK: omp.loop_nest.region:
         // CHECK: load i32, ptr %.linear_result
-        llvm.store %iv, %arg0 : i32, !llvm.ptr
+        llvm.store %iv, %i : i32, !llvm.ptr
 
         // Compute a[i] = b[i] * 10
-        %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_val = llvm.load %i : !llvm.ptr -> i32
         %i_idx = llvm.sext %i_val : i32 to i64
         %i_off = llvm.sub %i_idx, %c1_i64 : i64
         %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -58,7 +56,7 @@ module {
         omp.ordered.region par_level_simd {
           // CHECK: omp.ordered.region:
           // CHECK: load i32, ptr %.linear_result
-          %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_ord = llvm.load %i : !llvm.ptr -> i32
           %i_ord_idx = llvm.sext %i_ord : i32 to i64
           %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
           %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -69,7 +67,7 @@ module {
         // Compute c[i] = a[i] * 2 (code after ordered region)
         // CHECK: omp_region.finalize:
         // CHECK: load i32, ptr %.linear_result
-        %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_post = llvm.load %i : !llvm.ptr -> i32
         %i_post_idx = llvm.sext %i_post : i32 to i64
         %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
         %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From 2f8b6c3b6f7ca958505c6e6d7cc41ec53cb0fe0d Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 11:10:29 -0300
Subject: [PATCH 4/5] Update mlir tests

---
 mlir/test/Target/LLVMIR/openmp-simd-guided.mlir      |  3 +--
 .../Target/LLVMIR/openmp-wsloop-simd-ordered.mlir    | 12 +++++-------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
index b1f18b9f85417..496669c5b90d5 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
@@ -1,7 +1,6 @@
 // Ensure that schedule can be used with the guided kind-type and simd construct
 // RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
 
-omp.private {type = private} @_QFEi_private_i32 : i32
 llvm.func @test_simd_guided() {
   %0 = llvm.mlir.constant (1 : i64) : i64
   %c1_i32 = llvm.mlir.constant (1 : i32) : i32
@@ -10,7 +9,7 @@ llvm.func @test_simd_guided() {
   %c4_i32 = llvm.mlir.constant (4 : i32) : i32
   %1 = llvm.alloca %0 x i32 {bindc_name = "i"} : (i64) -> !llvm.ptr
   omp.wsloop schedule(guided = %c4_i32 : i32, simd) {
-    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) private(@_QFEi_private_i32 %1 -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%arg1) : i32 = (%c0_i32) to (%c64_i32) inclusive step (%c1_i32) {
         omp.yield
       }
diff --git a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
index beb300c23651f..8d5b89ecbd00a 100644
--- a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end do simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @wsloop_simd_ordered_linear
   llvm.func @wsloop_simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -39,14 +37,14 @@ module {
     // CHECK: %.linear_result = alloca i32
 
     omp.wsloop ordered(0) {
-      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
         omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c100_i32) inclusive step (%c1_i32) {
           // CHECK: omp.loop_nest.region:
           // CHECK: load i32, ptr %.linear_result
-          llvm.store %iv, %arg0 : i32, !llvm.ptr
+          llvm.store %iv, %i : i32, !llvm.ptr
 
           // Compute a[i] = b[i] * 10
-          %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_val = llvm.load %i : !llvm.ptr -> i32
           %i_idx = llvm.sext %i_val : i32 to i64
           %i_off = llvm.sub %i_idx, %c1_i64 : i64
           %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -59,7 +57,7 @@ module {
           omp.ordered.region par_level_simd {
             // CHECK: omp.ordered.region:
             // CHECK: load i32, ptr %.linear_result
-            %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+            %i_ord = llvm.load %i : !llvm.ptr -> i32
             %i_ord_idx = llvm.sext %i_ord : i32 to i64
             %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
             %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -71,7 +69,7 @@ module {
           // Compute c[i] = a[i] * 2 (code after ordered region)
           // CHECK: omp_region.finalize:
           // CHECK: load i32, ptr %.linear_result
-          %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_post = llvm.load %i : !llvm.ptr -> i32
           %i_post_idx = llvm.sext %i_post : i32 to i64
           %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
           %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From b671d6cf6efa8413abd061ac101b1ea14137ff5f Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Thu, 23 Jul 2026 16:48:58 -0300
Subject: [PATCH 5/5] Report an error if the linear variable can't be
 determined unambiguously

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 20 ++++++++++++++-----
 .../Target/LLVMIR/openmp-llvm-invalid.mlir    | 15 ++++++++++++++
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 2a246ebf98b3c..b938572fe3720 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -191,23 +191,32 @@ class LinearClauseProcessor {
   }
 
   // Find linear iteration variable and save it for later updates
-  void initLinearIV(omp::SimdOp simdOp) {
+  LogicalResult initLinearIV(omp::SimdOp simdOp) {
     auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
     // NOTE iteration variables can only be linear in non-nested loops.
     if (loopOp.getIVs().size() != 1)
-      return;
-    // The linear IV is the loop IV's store address.
+      return success();
+    // Currently, frontends using `omp.simd` always generate a store from the
+    // `omp.loop_nest`'s IV to the corresponding iteration variable.
+    // We leverage this to find the linear iteration variable.
+    //
+    // TODO Add an attribute to `omp.loop_nest` that explicitly lists the
+    //      variables that correspond to the loop induction variables.
     BlockArgument arg = loopOp.getIVs().front();
     for (const Operation *user : arg.getUsers()) {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
+            if (linearLoopIV)
+              return simdOp.emitError(
+                  "Could not determine the linear variable associated with the "
+                  "loop nest induction variable");
             linearLoopIV = linearVar;
-            break;
           }
         }
       }
     }
+    return success();
   }
 
   // Emit IR for updating Linear variables
@@ -4858,7 +4867,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-  linearClauseProcessor.initLinearIV(simdOp);
+  if (linearClauseProcessor.initLinearIV(simdOp).failed())
+    return failure();
 
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index 6eff5accca138..234f404ff5690 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -130,6 +130,21 @@ llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr
 
 // -----
 
+llvm.func @simd_linear_ambiguous_iv(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr, %y : !llvm.ptr) {
+  // expected-error @below {{Could not determine the linear variable associated with the loop nest induction variable}}
+  // expected-error @below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd linear(%x : !llvm.ptr = %step : i32, %y : !llvm.ptr = %step : i32) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
+      llvm.store %iv, %y : i32, !llvm.ptr
+      omp.yield
+    }
+  } {linear_var_types = [i32, i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}


From llvmlistbot at llvm.org  Thu Jul 23 13:54:16 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Thu, 23 Jul 2026 13:54:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a627f78.8e75e0cb.2de827.03b2@mx.google.com>


https://github.com/beamandala updated https://github.com/llvm/llvm-project/pull/211141

>From 4ff9cf5e7abbc47b3f1ab27b44f821eef44763e4 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 16:07:42 -0700
Subject: [PATCH 1/7] Elevate allocs to globals pass

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  |  10 ++
 .../Dialect/MemRef/Transforms/Transforms.h    |   3 +
 .../Dialect/MemRef/Transforms/CMakeLists.txt  |   1 +
 .../Transforms/ElevateAllocsToGlobals.cpp     | 112 ++++++++++++++++++
 .../MemRef/elevate-allocs-to-globals.mlir     |  56 +++++++++
 5 files changed, 182 insertions(+)
 create mode 100644 mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
 create mode 100644 mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index c6be600247696..4e6ead48cbff2 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -340,4 +340,14 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
   ];
 }
 
+def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
+  let summary = "Elevate allocs to globals";
+  let description = [{
+
+  }];
+  let dependentDialects = [
+      "memref::MemRefDialect"
+  ];
+}
+
 #endif // MLIR_DIALECT_MEMREF_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
index 720677455ae5d..6960b8ef00f06 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #ifndef MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 #define MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
 
@@ -165,6 +166,8 @@ void populateExtractAddressComputationsPatterns(RewritePatternSet &patterns);
 /// into one-dimensional memref operations.
 void populateFlattenMemrefsPatterns(RewritePatternSet &patterns);
 
+void populateElevateAllocsToGlobalsPatterns(RewritePatternSet &patterns);
+
 /// Build a new memref::AllocaOp whose dynamic sizes are independent of all
 /// given independencies. If the op is already independent of all
 /// independencies, the same AllocaOp result is returned.
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1a8b03dabcfb7..7d43e66989bc4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -17,6 +17,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   ReifyResultShapes.cpp
   ResolveShapedTypeResultDims.cpp
   RuntimeOpVerification.cpp
+  ElevateAllocsToGlobals.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/MemRef
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
new file mode 100644
index 0000000000000..017b23b810dd3
--- /dev/null
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -0,0 +1,112 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/Interfaces/LoopLikeInterface.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/LogicalResult.h"
+
+namespace mlir {
+namespace memref {
+#define GEN_PASS_DEF_ELEVATEALLOCSTOGLOBALSPASS
+#include "mlir/Dialect/MemRef/Transforms/Passes.h.inc"
+} // namespace memref
+} // namespace mlir
+
+using namespace mlir;
+
+namespace {
+
+// Checks if 'op' is contained inside any branching or looping structure
+static bool isInsideControlFlow(mlir::Operation *op) {
+  if (mlir::getEnclosingRepetitiveRegion(op) != nullptr)
+    return true;
+
+  if (op->getParentOfType<mlir::LoopLikeOpInterface>())
+    return true;
+
+  if (auto regionParent = op->getParentOfType<mlir::RegionBranchOpInterface>())
+    return true;
+
+  return false;
+}
+
+struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
+public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(memref::AllocOp allocOp,
+                                PatternRewriter &rewriter) const final {
+
+    auto memrefType = allocOp.getType();
+    // memref.global requires statically shaped memrefs
+    if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
+      return failure();
+
+    auto loopParent = allocOp->getParentOfType<mlir::LoopLikeOpInterface>();
+    if (loopParent != nullptr || isInsideControlFlow(allocOp))
+      return failure();
+
+    memref::GlobalOp globalOp;
+    {
+      Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
+
+      SymbolTable symbolTable(symbolTableOp);
+
+      OpBuilder builder(rewriter.getContext());
+      StringAttr globalName = rewriter.getStringAttr("global_alloc");
+      globalOp = memref::GlobalOp::create(builder, allocOp.getLoc(), globalName,
+                                          rewriter.getStringAttr("private"),
+                                          memrefType, rewriter.getUnitAttr(),
+                                          false, allocOp.getAlignmentAttr());
+
+      symbolTable.insert(globalOp);
+    }
+
+    SmallVector<Operation *> deallocsToDelete;
+    for (OpOperand &use : allocOp.getResult().getUses()) {
+      Operation *user = use.getOwner();
+      if (isa<memref::DeallocOp>(user))
+        deallocsToDelete.push_back(user);
+    }
+    for (Operation *dealloc : deallocsToDelete)
+      rewriter.eraseOp(dealloc);
+
+    rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
+                                                     globalOp.getName());
+
+    return success();
+  }
+};
+
+struct ElevateAllocsToGlobalsPass
+    : public mlir::memref::impl::ElevateAllocsToGlobalsPassBase<
+          ElevateAllocsToGlobalsPass> {
+  using Base::Base;
+
+  void runOnOperation() override {
+    ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    memref::populateElevateAllocsToGlobalsPatterns(patterns);
+
+    (void)applyPatternsGreedily(moduleOp, std::move(patterns));
+  }
+};
+} // namespace
+
+void mlir::memref::populateElevateAllocsToGlobalsPatterns(
+    RewritePatternSet &patterns) {
+  patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
+}
\ No newline at end of file
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
new file mode 100644
index 0000000000000..dd65cd12989cc
--- /dev/null
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -0,0 +1,56 @@
+// RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
+
+/// Test that a single static memref.alloc is elevated to a memref.global,
+/// replaced with memref.get_global, alignment attribute is preserved, and
+/// associated memref.dealloc is removed.
+
+func.func @single_alloc(%val: f32, %idx: index) {
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
+  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
+  memref.dealloc %0 : memref<10x20xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @single_alloc(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: return
+// CHECK-NOT: memref.dealloc
+// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+
+// -----
+
+/// Test that multiple static memref.alloc ops in the same function are elevated
+/// to global memrefs without symbol name collisions.
+
+func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+  %0 = memref.alloc() : memref<10xf32>
+  %1 = memref.alloc() : memref<20xi32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  return
+}
+
+// CHECK-LABEL: func.func @multiple_allocs(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
+
+// -----
+
+/// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<?xf32>
+  memref.store %val, %0[%sz] : memref<?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>

>From b0f3250cd4e5a865afc304d8a9a04674fe33cd25 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Tue, 21 Jul 2026 17:48:50 -0700
Subject: [PATCH 2/7] Simplify and add tests

---
 .../Transforms/ElevateAllocsToGlobals.cpp     |  2 +-
 .../MemRef/elevate-allocs-to-globals.mlir     | 88 +++++++++++++++----
 2 files changed, 70 insertions(+), 20 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 017b23b810dd3..2dbdeb00c5354 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -109,4 +109,4 @@ struct ElevateAllocsToGlobalsPass
 void mlir::memref::populateElevateAllocsToGlobalsPatterns(
     RewritePatternSet &patterns) {
   patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
-}
\ No newline at end of file
+}
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
index dd65cd12989cc..43477960cf7be 100644
--- a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -1,49 +1,49 @@
 // RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
 
-/// Test that a single static memref.alloc is elevated to a memref.global,
-/// replaced with memref.get_global, alignment attribute is preserved, and
-/// associated memref.dealloc is removed.
+/// Test that a single static memref.alloc is elevated to a memref.global, references
+/// are replaced with memref.get_global, and the associated memref.dealloc is removed.
 
 func.func @single_alloc(%val: f32, %idx: index) {
-  %0 = memref.alloc() {alignment = 64 : i64} : memref<10x20xf32>
-  memref.store %val, %0[%idx, %idx] : memref<10x20xf32>
-  memref.dealloc %0 : memref<10x20xf32>
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+  memref.store %val, %0[%idx] : memref<10xf32>
+  memref.dealloc %0 : memref<10xf32>
   return
 }
 
 // CHECK-LABEL: func.func @single_alloc(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10x20xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x20xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<10xf32>
 // CHECK-NEXT: return
 // CHECK-NOT: memref.dealloc
-// CHECK: memref.global "private" @global_alloc : memref<10x20xf32> = uninitialized {alignment = 64 : i64}
+// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
 
 // -----
 
 /// Test that multiple static memref.alloc ops in the same function are elevated
 /// to global memrefs without symbol name collisions.
 
-func.func @multiple_allocs(%val: f32, %val_i32: i32, %idx: index) {
+func.func @multiple_allocs(%val: f32, %idx: index) {
   %0 = memref.alloc() : memref<10xf32>
-  %1 = memref.alloc() : memref<20xi32>
+  %1 = memref.alloc() : memref<20xf32>
   memref.store %val, %0[%idx] : memref<10xf32>
-  memref.store %val_i32, %1[%idx] : memref<20xi32>
+  memref.store %val, %1[%idx] : memref<20xf32>
   return
 }
 
 // CHECK-LABEL: func.func @multiple_allocs(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: i32, %[[ARG2:.*]]: index) {
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
 // CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
-// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xi32>
-// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG2]]] : memref<10xf32>
-// CHECK: memref.store %[[ARG1]], %[[MEM1]][%[[ARG2]]] : memref<20xi32>
-// CHECK-DAG: memref.global "private" @global_alloc : memref<20xi32> = uninitialized
+// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG1]]] : memref<10xf32>
+// CHECK: memref.store %[[ARG0]], %[[MEM1]][%[[ARG1]]] : memref<20xf32>
+// CHECK-DAG: memref.global "private" @global_alloc : memref<20xf32> = uninitialized
 // CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
 
 // -----
 
 /// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+
 func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
   %0 = memref.alloc(%sz) : memref<?xf32>
   memref.store %val, %0[%sz] : memref<?xf32>
@@ -52,5 +52,55 @@ func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
 
 // CHECK-LABEL: func.func @dynamic_alloc_ignored(
 // CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
-// CHECK: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+
+// -----
+
+/// Test that a partially dynamic memref.alloc is ignored and not elevated to a global.
+
+func.func @partially_dynamic_alloc_ignored(%val: f32, %sz: index) {
+  %0 = memref.alloc(%sz) : memref<10x?xf32>
+  memref.store %val, %0[%sz, %sz] : memref<10x?xf32>
+  return
+}
+
+// CHECK-LABEL: func.func @partially_dynamic_alloc_ignored(
+// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<10x?xf32>
+// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x?xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside a loop is ignored and not elevated to a global.
+
+func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index, %val: f32, %idx: index) {
+  scf.for %i = %lb to %ub step %step {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_loop_ignored(
+// CHECK: scf.for
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+
+// -----
+
+/// Test that a static memref.alloc inside control flow (scf.if) is ignored and not
+/// elevated to a global.
+
+func.func @alloc_in_control_flow_ignored(%cond: i1, %val: f32, %idx: index) {
+  scf.if %cond {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.store %val, %0[%idx] : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_control_flow_ignored(
+// CHECK: scf.if
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]

>From d7f53ae83635215983dd626720eab2f7db5e3593 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:29:11 -0700
Subject: [PATCH 3/7] Pass description

---
 .../mlir/Dialect/MemRef/Transforms/Passes.td  | 24 +++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index 4e6ead48cbff2..a0a4be7318889 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -343,7 +343,31 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
 def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
   let summary = "Elevate allocs to globals";
   let description = [{
+    This pass converts statically-shaped `memref.alloc` operations that are not
+    enclosed within loops or control flow into private `memref.global` operations.
+    `memref.alloc` operations are replaced with `memref.get_global` references and
+    any associated `memref.dealloc` operations are removed.
 
+    Example:
+
+    ```mlir
+    func.func @example() {
+      %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+      memref.dealloc %0 : memref<10xf32>
+      return
+    }
+    ```
+
+    is transformed to
+
+    ```mlir
+    memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
+
+    func.func @example() {
+      %0 = memref.get_global @global_alloc : memref<10xf32>
+      return
+    }
+    ```
   }];
   let dependentDialects = [
       "memref::MemRefDialect"

>From 8b4d7a5e2cccc7b928ce05f986ff40c2e5afb6c7 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:36:18 -0700
Subject: [PATCH 4/7] Clean up helper func

---
 .../Transforms/ElevateAllocsToGlobals.cpp     | 20 ++++++-------------
 1 file changed, 6 insertions(+), 14 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 2dbdeb00c5354..5a079ac8c2b63 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -28,18 +28,11 @@ using namespace mlir;
 
 namespace {
 
-// Checks if 'op' is contained inside any branching or looping structure
-static bool isInsideControlFlow(mlir::Operation *op) {
-  if (mlir::getEnclosingRepetitiveRegion(op) != nullptr)
-    return true;
-
-  if (op->getParentOfType<mlir::LoopLikeOpInterface>())
-    return true;
-
-  if (auto regionParent = op->getParentOfType<mlir::RegionBranchOpInterface>())
-    return true;
-
-  return false;
+// Checks if 'op' is contained inside any branching or looping structure.
+static bool isInsideControlFlow(Operation *op) {
+  return getEnclosingRepetitiveRegion(op) ||
+         op->getParentOfType<LoopLikeOpInterface>() ||
+         op->getParentOfType<RegionBranchOpInterface>();
 }
 
 struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
@@ -54,8 +47,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
     if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
       return failure();
 
-    auto loopParent = allocOp->getParentOfType<mlir::LoopLikeOpInterface>();
-    if (loopParent != nullptr || isInsideControlFlow(allocOp))
+    if (isInsideControlFlow(allocOp))
       return failure();
 
     memref::GlobalOp globalOp;

>From eb611a58fb3a322d0236d71410bc654ffd1af43d Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Wed, 22 Jul 2026 15:48:22 -0700
Subject: [PATCH 5/7] Comments

---
 .../Transforms/ElevateAllocsToGlobals.cpp     | 29 ++++++++++++++++---
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index 5a079ac8c2b63..ec1130e96cb75 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -28,32 +28,51 @@ using namespace mlir;
 
 namespace {
 
-// Checks if 'op' is contained inside any branching or looping structure.
+/// Returns true if `op` is contained inside any branching, region, or looping
+/// structure (such as scf.for, scf.if, or repetitive regions)
 static bool isInsideControlFlow(Operation *op) {
   return getEnclosingRepetitiveRegion(op) ||
          op->getParentOfType<LoopLikeOpInterface>() ||
          op->getParentOfType<RegionBranchOpInterface>();
 }
 
+/// Elevates a static `memref.alloc` operation to a top-level `memref.global` op
+/// if the allocation is not enclosed within any control flow constructs.
+///
+/// Converts:
+/// ```mlir
+/// %0 = memref.alloc() : memref<4x4xf32>
+/// memref.dealloc %0 : memref<4x4xf32>
+/// ```
+/// to:
+/// ```mlir
+/// memref.global "private" @global_alloc : memref<4x4xf32>
+/// ...
+/// %0 = memref.get_global @global_alloc : memref<4x4xf32>
+/// ```
 struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
 public:
   using OpRewritePattern::OpRewritePattern;
 
   LogicalResult matchAndRewrite(memref::AllocOp allocOp,
                                 PatternRewriter &rewriter) const final {
-
     auto memrefType = allocOp.getType();
-    // memref.global requires statically shaped memrefs
+    // `memref.global` requires statically shaped memrefs with no dynamic sizes.
     if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
       return failure();
 
+    // Avoid elevating allocations inside control flow (loops or conditionals),
+    // as converting them to a single static global would make multiple
+    // executions share the same buffer, changing semantics or causing race
+    // conditions.
     if (isInsideControlFlow(allocOp))
       return failure();
 
+    // Create the global variable at the nearest enclosing symbol table (e.g.
+    // module).
     memref::GlobalOp globalOp;
     {
       Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
-
       SymbolTable symbolTable(symbolTableOp);
 
       OpBuilder builder(rewriter.getContext());
@@ -66,6 +85,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
       symbolTable.insert(globalOp);
     }
 
+    // Remove any `memref.dealloc` operations using this allocation
     SmallVector<Operation *> deallocsToDelete;
     for (OpOperand &use : allocOp.getResult().getUses()) {
       Operation *user = use.getOwner();
@@ -75,6 +95,7 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
     for (Operation *dealloc : deallocsToDelete)
       rewriter.eraseOp(dealloc);
 
+    // Replace the original `memref.alloc` with `memref.get_global`.
     rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
                                                      globalOp.getName());
 

>From ca1e6c5c6e3c03923b7af10024ce4b8d10bf2ea9 Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Thu, 23 Jul 2026 13:06:42 -0700
Subject: [PATCH 6/7] restrict MemRef ElevateAllocsToGlobals to ModuleOps and
 add tests

---
 .../Transforms/ElevateAllocsToGlobals.cpp     | 33 +++++++-------
 .../MemRef/elevate-allocs-to-globals.mlir     | 45 +++++++++++++++++++
 2 files changed, 62 insertions(+), 16 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
index ec1130e96cb75..907038b545c62 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -13,8 +13,10 @@
 #include "mlir/IR/BuiltinAttributes.h"
 #include "mlir/IR/BuiltinOps.h"
 #include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
 #include "mlir/Interfaces/LoopLikeInterface.h"
 #include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/Casting.h"
 #include "llvm/Support/LogicalResult.h"
 
 namespace mlir {
@@ -68,22 +70,21 @@ struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
     if (isInsideControlFlow(allocOp))
       return failure();
 
-    // Create the global variable at the nearest enclosing symbol table (e.g.
-    // module).
-    memref::GlobalOp globalOp;
-    {
-      Operation *symbolTableOp = SymbolTable::getNearestSymbolTable(allocOp);
-      SymbolTable symbolTable(symbolTableOp);
-
-      OpBuilder builder(rewriter.getContext());
-      StringAttr globalName = rewriter.getStringAttr("global_alloc");
-      globalOp = memref::GlobalOp::create(builder, allocOp.getLoc(), globalName,
-                                          rewriter.getStringAttr("private"),
-                                          memrefType, rewriter.getUnitAttr(),
-                                          false, allocOp.getAlignmentAttr());
-
-      symbolTable.insert(globalOp);
-    }
+    // Create the global variable at the nearest enclosing symbol table defining
+    // op if it's a ModuleOp.
+    auto moduleOp = llvm::dyn_cast_or_null<ModuleOp>(
+        SymbolTable::getNearestSymbolTable(allocOp));
+    if (!moduleOp)
+      return failure();
+
+    OpBuilder detachedBuilder(rewriter.getContext());
+    StringAttr globalName = rewriter.getStringAttr("global_alloc");
+    memref::GlobalOp globalOp = memref::GlobalOp::create(
+        detachedBuilder, allocOp.getLoc(), globalName,
+        rewriter.getStringAttr("private"), memrefType, rewriter.getUnitAttr(),
+        false, allocOp.getAlignmentAttr());
+
+    SymbolTable(moduleOp).insert(globalOp);
 
     // Remove any `memref.dealloc` operations using this allocation
     SmallVector<Operation *> deallocsToDelete;
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
index 43477960cf7be..b91e5f92c31e1 100644
--- a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -104,3 +104,48 @@ func.func @alloc_in_control_flow_ignored(%cond: i1, %val: f32, %idx: index) {
 // CHECK: scf.if
 // CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
 // CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+
+// -----
+
+/// Test that a static memref.alloc inside a non-ModuleOp symbol table is
+/// ignored and not elevated to a global.
+
+gpu.module @gpu_mod {
+  gpu.func @kernel() {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
+    gpu.return
+  }
+}
+
+// CHECK-LABEL: gpu.module @gpu_mod
+// CHECK: gpu.func @kernel
+// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT: memref.dealloc %[[MEM]] : memref<10xf32>
+// CHECK-NOT: memref.global
+
+// -----
+
+/// Test that in a function with allocs both inside and outside of control flow,
+/// only the alloc outside of control flow is elevated to a global.
+
+func.func @mixed_control_flow_allocs(%cond: i1, %val: f32, %idx: index) {
+  %outside = memref.alloc() : memref<10xf32>
+  memref.store %val, %outside[%idx] : memref<10xf32>
+  scf.if %cond {
+    %inside = memref.alloc() : memref<20xf32>
+    memref.store %val, %inside[%idx] : memref<20xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @mixed_control_flow_allocs(
+// CHECK-SAME: %[[COND:.*]]: i1, %[[VAL:.*]]: f32, %[[IDX:.*]]: index) {
+// CHECK-NEXT: %[[OUTSIDE:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT: memref.store %[[VAL]], %[[OUTSIDE]][%[[IDX]]] : memref<10xf32>
+// CHECK-NEXT: scf.if %[[COND]] {
+// CHECK-NEXT:   %[[INSIDE:.*]] = memref.alloc() : memref<20xf32>
+// CHECK-NEXT:   memref.store %[[VAL]], %[[INSIDE]][%[[IDX]]] : memref<20xf32>
+// CHECK-NEXT: }
+// CHECK-NEXT: return
+// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized

>From 694b3704b165038b349ff3f4700abe27fb9c833b Mon Sep 17 00:00:00 2001
From: Bhavesh Mandalapu <bmandalapu at google.com>
Date: Thu, 23 Jul 2026 13:16:22 -0700
Subject: [PATCH 7/7] Update test cases

---
 .../MemRef/elevate-allocs-to-globals.mlir     | 111 ++++++++----------
 1 file changed, 50 insertions(+), 61 deletions(-)

diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
index b91e5f92c31e1..3ed66024fd57e 100644
--- a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -3,107 +3,98 @@
 /// Test that a single static memref.alloc is elevated to a memref.global, references
 /// are replaced with memref.get_global, and the associated memref.dealloc is removed.
 
-func.func @single_alloc(%val: f32, %idx: index) {
+func.func @single_alloc() -> memref<10xf32> {
   %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
-  memref.store %val, %0[%idx] : memref<10xf32>
   memref.dealloc %0 : memref<10xf32>
-  return
+  return %0 : memref<10xf32>
 }
 
-// CHECK-LABEL: func.func @single_alloc(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<10xf32>
-// CHECK-NEXT: return
-// CHECK-NOT: memref.dealloc
-// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
+// CHECK-LABEL: func.func @single_alloc() -> memref<10xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<10xf32>
+// CHECK-NOT:     memref.dealloc
+// CHECK:       memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
 
 // -----
 
 /// Test that multiple static memref.alloc ops in the same function are elevated
 /// to global memrefs without symbol name collisions.
 
-func.func @multiple_allocs(%val: f32, %idx: index) {
+func.func @multiple_allocs() -> (memref<10xf32>, memref<20xf32>) {
   %0 = memref.alloc() : memref<10xf32>
   %1 = memref.alloc() : memref<20xf32>
-  memref.store %val, %0[%idx] : memref<10xf32>
-  memref.store %val, %1[%idx] : memref<20xf32>
-  return
+  return %0, %1 : memref<10xf32>, memref<20xf32>
 }
 
-// CHECK-LABEL: func.func @multiple_allocs(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-DAG: %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
-// CHECK-DAG: %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
-// CHECK: memref.store %[[ARG0]], %[[MEM0]][%[[ARG1]]] : memref<10xf32>
-// CHECK: memref.store %[[ARG0]], %[[MEM1]][%[[ARG1]]] : memref<20xf32>
-// CHECK-DAG: memref.global "private" @global_alloc : memref<20xf32> = uninitialized
-// CHECK-DAG: memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
+// CHECK-LABEL: func.func @multiple_allocs() -> (memref<10xf32>, memref<20xf32>) {
+// CHECK-DAG:    %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
+// CHECK-DAG:    %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
+// CHECK:        return %[[MEM0]], %[[MEM1]] : memref<10xf32>, memref<20xf32>
+// CHECK-DAG:    memref.global "private" @global_alloc : memref<20xf32> = uninitialized
+// CHECK-DAG:    memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
 
 // -----
 
 /// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
 
-func.func @dynamic_alloc_ignored(%val: f32, %sz: index) {
+func.func @dynamic_alloc_ignored(%sz: index) -> memref<?xf32> {
   %0 = memref.alloc(%sz) : memref<?xf32>
-  memref.store %val, %0[%sz] : memref<?xf32>
-  return
+  return %0 : memref<?xf32>
 }
 
 // CHECK-LABEL: func.func @dynamic_alloc_ignored(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<?xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]]] : memref<?xf32>
+// CHECK-SAME:  %[[SZ:.*]]: index) -> memref<?xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc(%[[SZ]]) : memref<?xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<?xf32>
 
 // -----
 
 /// Test that a partially dynamic memref.alloc is ignored and not elevated to a global.
 
-func.func @partially_dynamic_alloc_ignored(%val: f32, %sz: index) {
+func.func @partially_dynamic_alloc_ignored(%sz: index) -> memref<10x?xf32> {
   %0 = memref.alloc(%sz) : memref<10x?xf32>
-  memref.store %val, %0[%sz, %sz] : memref<10x?xf32>
-  return
+  return %0 : memref<10x?xf32>
 }
 
 // CHECK-LABEL: func.func @partially_dynamic_alloc_ignored(
-// CHECK-SAME: %[[ARG0:.*]]: f32, %[[ARG1:.*]]: index) {
-// CHECK-NEXT: %[[MEM:.*]] = memref.alloc(%[[ARG1]]) : memref<10x?xf32>
-// CHECK-NEXT: memref.store %[[ARG0]], %[[MEM]][%[[ARG1]], %[[ARG1]]] : memref<10x?xf32>
+// CHECK-SAME:  %[[SZ:.*]]: index) -> memref<10x?xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc(%[[SZ]]) : memref<10x?xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<10x?xf32>
 
 // -----
 
 /// Test that a static memref.alloc inside a loop is ignored and not elevated to a global.
 
-func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index, %val: f32, %idx: index) {
+func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index) {
   scf.for %i = %lb to %ub step %step {
     %0 = memref.alloc() : memref<10xf32>
-    memref.store %val, %0[%idx] : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
   }
   return
 }
 
 // CHECK-LABEL: func.func @alloc_in_loop_ignored(
-// CHECK: scf.for
-// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
-// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+// CHECK:       scf.for
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
 
 // -----
 
 /// Test that a static memref.alloc inside control flow (scf.if) is ignored and not
 /// elevated to a global.
 
-func.func @alloc_in_control_flow_ignored(%cond: i1, %val: f32, %idx: index) {
+func.func @alloc_in_control_flow_ignored(%cond: i1) {
   scf.if %cond {
     %0 = memref.alloc() : memref<10xf32>
-    memref.store %val, %0[%idx] : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
   }
   return
 }
 
 // CHECK-LABEL: func.func @alloc_in_control_flow_ignored(
-// CHECK: scf.if
-// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
-// CHECK-NEXT: memref.store %{{.*}}, %[[MEM]]
+// CHECK:       scf.if
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
 
 // -----
 
@@ -119,33 +110,31 @@ gpu.module @gpu_mod {
 }
 
 // CHECK-LABEL: gpu.module @gpu_mod
-// CHECK: gpu.func @kernel
-// CHECK-NEXT: %[[MEM:.*]] = memref.alloc() : memref<10xf32>
-// CHECK-NEXT: memref.dealloc %[[MEM]] : memref<10xf32>
-// CHECK-NOT: memref.global
+// CHECK:       gpu.func @kernel
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
+// CHECK-NOT:   memref.global
 
 // -----
 
 /// Test that in a function with allocs both inside and outside of control flow,
 /// only the alloc outside of control flow is elevated to a global.
 
-func.func @mixed_control_flow_allocs(%cond: i1, %val: f32, %idx: index) {
+func.func @mixed_control_flow_allocs(%cond: i1) -> memref<10xf32> {
   %outside = memref.alloc() : memref<10xf32>
-  memref.store %val, %outside[%idx] : memref<10xf32>
   scf.if %cond {
     %inside = memref.alloc() : memref<20xf32>
-    memref.store %val, %inside[%idx] : memref<20xf32>
+    memref.dealloc %inside : memref<20xf32>
   }
-  return
+  return %outside : memref<10xf32>
 }
 
 // CHECK-LABEL: func.func @mixed_control_flow_allocs(
-// CHECK-SAME: %[[COND:.*]]: i1, %[[VAL:.*]]: f32, %[[IDX:.*]]: index) {
-// CHECK-NEXT: %[[OUTSIDE:.*]] = memref.get_global @global_alloc : memref<10xf32>
-// CHECK-NEXT: memref.store %[[VAL]], %[[OUTSIDE]][%[[IDX]]] : memref<10xf32>
-// CHECK-NEXT: scf.if %[[COND]] {
-// CHECK-NEXT:   %[[INSIDE:.*]] = memref.alloc() : memref<20xf32>
-// CHECK-NEXT:   memref.store %[[VAL]], %[[INSIDE]][%[[IDX]]] : memref<20xf32>
-// CHECK-NEXT: }
-// CHECK-NEXT: return
-// CHECK: memref.global "private" @global_alloc : memref<10xf32> = uninitialized
+// CHECK-SAME:  %[[COND:.*]]: i1) -> memref<10xf32> {
+// CHECK-NEXT:    %[[OUTSIDE:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT:    scf.if %[[COND]] {
+// CHECK-NEXT:      %[[INSIDE:.*]] = memref.alloc() : memref<20xf32>
+// CHECK-NEXT:      memref.dealloc %[[INSIDE]] : memref<20xf32>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    return %[[OUTSIDE]] : memref<10xf32>
+// CHECK:       memref.global "private" @global_alloc : memref<10xf32> = uninitialized


From llvmlistbot at llvm.org  Thu Jul 23 13:54:19 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Thu, 23 Jul 2026 13:54:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a627f7b.3cce8983.247e47.785e@mx.google.com>


https://github.com/beamandala ready_for_review https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Thu Jul 23 13:54:35 2026
From: llvmlistbot at llvm.org (Krzysztof Parzyszek)
Date: Thu, 23 Jul 2026 13:54:35 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a627f8b.507fc713.3702e2.b2c4@mx.google.com>


https://github.com/kparzysz approved this pull request.

I think it's ok to leave the runtime implementation for another PR.

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Thu Jul 23 13:54:44 2026
From: llvmlistbot at llvm.org (Bhavesh M)
Date: Thu, 23 Jul 2026 13:54:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a627f94.92ea0e2a.2ed880.0611@mx.google.com>


beamandala wrote:

@mtrofin @ilovepi @banach-space @ioghiban 

https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Thu Jul 23 13:54:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 13:54:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a627fa3.8e75e0cb.2de827.15a2@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Bhavesh M (beamandala)

<details>
<summary>Changes</summary>

This creates the `elevate-allocs-to-globals` pass which replaces `memref::AllocOp`s with `memref::GlobalOp`s in the closest symbol table defining op if it's a `ModuleOp`. It does this through the following process:
- Check that the `AllocOp` is statically shaped because `GlobalOp` requires static shape.
- Check that the `AllocOp` isn't in control flow such as loops or conditionals because elevating in these scenarios could change program semantics.
- Check that the closest symbol table defining op is a `ModuleOp` because other `GlobalOp`s could be illegal in other symbol table defining ops and to minimize complexity for initial implementation.
- Create the `GlobalOp`, delete any `memref::DeallocOp`s associated with the `AllocOp`
- Replace the `AllocOp` with `memref::GetGlobalOp`

This pass will help reduce/eliminate dynamic memory allocations in frequently executed functions.

---
Full diff: https://github.com/llvm/llvm-project/pull/211141.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td (+34) 
- (modified) mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h (+3) 
- (modified) mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt (+1) 
- (added) mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp (+126) 
- (added) mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir (+140) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
index c6be600247696..a0a4be7318889 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Passes.td
@@ -340,4 +340,38 @@ def FlattenMemrefsPass : Pass<"flatten-memref"> {
   ];
 }
 
+def ElevateAllocsToGlobalsPass : Pass<"elevate-allocs-to-globals", "ModuleOp"> {
+  let summary = "Elevate allocs to globals";
+  let description = [{
+    This pass converts statically-shaped `memref.alloc` operations that are not
+    enclosed within loops or control flow into private `memref.global` operations.
+    `memref.alloc` operations are replaced with `memref.get_global` references and
+    any associated `memref.dealloc` operations are removed.
+
+    Example:
+
+    ```mlir
+    func.func @example() {
+      %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+      memref.dealloc %0 : memref<10xf32>
+      return
+    }
+    ```
+
+    is transformed to
+
+    ```mlir
+    memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
+
+    func.func @example() {
+      %0 = memref.get_global @global_alloc : memref<10xf32>
+      return
+    }
+    ```
+  }];
+  let dependentDialects = [
+      "memref::MemRefDialect"
+  ];
+}
+
 #endif // MLIR_DIALECT_MEMREF_TRANSFORMS_PASSES
diff --git a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
index 720677455ae5d..6960b8ef00f06 100644
--- a/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/MemRef/Transforms/Transforms.h
@@ -14,6 +14,7 @@
 #ifndef MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 #define MLIR_DIALECT_MEMREF_TRANSFORMS_TRANSFORMS_H
 
+#include "mlir/IR/PatternMatch.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/STLFunctionalExtras.h"
 
@@ -165,6 +166,8 @@ void populateExtractAddressComputationsPatterns(RewritePatternSet &patterns);
 /// into one-dimensional memref operations.
 void populateFlattenMemrefsPatterns(RewritePatternSet &patterns);
 
+void populateElevateAllocsToGlobalsPatterns(RewritePatternSet &patterns);
+
 /// Build a new memref::AllocaOp whose dynamic sizes are independent of all
 /// given independencies. If the op is already independent of all
 /// independencies, the same AllocaOp result is returned.
diff --git a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
index 1a8b03dabcfb7..7d43e66989bc4 100644
--- a/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/MemRef/Transforms/CMakeLists.txt
@@ -17,6 +17,7 @@ add_mlir_dialect_library(MLIRMemRefTransforms
   ReifyResultShapes.cpp
   ResolveShapedTypeResultDims.cpp
   RuntimeOpVerification.cpp
+  ElevateAllocsToGlobals.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${MLIR_MAIN_INCLUDE_DIR}/mlir/Dialect/MemRef
diff --git a/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
new file mode 100644
index 0000000000000..907038b545c62
--- /dev/null
+++ b/mlir/lib/Dialect/MemRef/Transforms/ElevateAllocsToGlobals.cpp
@@ -0,0 +1,126 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/MemRef/IR/MemRef.h"
+#include "mlir/Dialect/MemRef/Transforms/Passes.h"
+#include "mlir/Dialect/MemRef/Transforms/Transforms.h"
+#include "mlir/IR/Builders.h"
+#include "mlir/IR/BuiltinAttributes.h"
+#include "mlir/IR/BuiltinOps.h"
+#include "mlir/IR/PatternMatch.h"
+#include "mlir/IR/SymbolTable.h"
+#include "mlir/Interfaces/LoopLikeInterface.h"
+#include "mlir/Transforms/GreedyPatternRewriteDriver.h"
+#include "llvm/Support/Casting.h"
+#include "llvm/Support/LogicalResult.h"
+
+namespace mlir {
+namespace memref {
+#define GEN_PASS_DEF_ELEVATEALLOCSTOGLOBALSPASS
+#include "mlir/Dialect/MemRef/Transforms/Passes.h.inc"
+} // namespace memref
+} // namespace mlir
+
+using namespace mlir;
+
+namespace {
+
+/// Returns true if `op` is contained inside any branching, region, or looping
+/// structure (such as scf.for, scf.if, or repetitive regions)
+static bool isInsideControlFlow(Operation *op) {
+  return getEnclosingRepetitiveRegion(op) ||
+         op->getParentOfType<LoopLikeOpInterface>() ||
+         op->getParentOfType<RegionBranchOpInterface>();
+}
+
+/// Elevates a static `memref.alloc` operation to a top-level `memref.global` op
+/// if the allocation is not enclosed within any control flow constructs.
+///
+/// Converts:
+/// ```mlir
+/// %0 = memref.alloc() : memref<4x4xf32>
+/// memref.dealloc %0 : memref<4x4xf32>
+/// ```
+/// to:
+/// ```mlir
+/// memref.global "private" @global_alloc : memref<4x4xf32>
+/// ...
+/// %0 = memref.get_global @global_alloc : memref<4x4xf32>
+/// ```
+struct ElevateAllocsToGlobals : public OpRewritePattern<memref::AllocOp> {
+public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(memref::AllocOp allocOp,
+                                PatternRewriter &rewriter) const final {
+    auto memrefType = allocOp.getType();
+    // `memref.global` requires statically shaped memrefs with no dynamic sizes.
+    if (!memrefType.hasStaticShape() || !allocOp.getDynamicSizes().empty())
+      return failure();
+
+    // Avoid elevating allocations inside control flow (loops or conditionals),
+    // as converting them to a single static global would make multiple
+    // executions share the same buffer, changing semantics or causing race
+    // conditions.
+    if (isInsideControlFlow(allocOp))
+      return failure();
+
+    // Create the global variable at the nearest enclosing symbol table defining
+    // op if it's a ModuleOp.
+    auto moduleOp = llvm::dyn_cast_or_null<ModuleOp>(
+        SymbolTable::getNearestSymbolTable(allocOp));
+    if (!moduleOp)
+      return failure();
+
+    OpBuilder detachedBuilder(rewriter.getContext());
+    StringAttr globalName = rewriter.getStringAttr("global_alloc");
+    memref::GlobalOp globalOp = memref::GlobalOp::create(
+        detachedBuilder, allocOp.getLoc(), globalName,
+        rewriter.getStringAttr("private"), memrefType, rewriter.getUnitAttr(),
+        false, allocOp.getAlignmentAttr());
+
+    SymbolTable(moduleOp).insert(globalOp);
+
+    // Remove any `memref.dealloc` operations using this allocation
+    SmallVector<Operation *> deallocsToDelete;
+    for (OpOperand &use : allocOp.getResult().getUses()) {
+      Operation *user = use.getOwner();
+      if (isa<memref::DeallocOp>(user))
+        deallocsToDelete.push_back(user);
+    }
+    for (Operation *dealloc : deallocsToDelete)
+      rewriter.eraseOp(dealloc);
+
+    // Replace the original `memref.alloc` with `memref.get_global`.
+    rewriter.replaceOpWithNewOp<memref::GetGlobalOp>(allocOp, memrefType,
+                                                     globalOp.getName());
+
+    return success();
+  }
+};
+
+struct ElevateAllocsToGlobalsPass
+    : public mlir::memref::impl::ElevateAllocsToGlobalsPassBase<
+          ElevateAllocsToGlobalsPass> {
+  using Base::Base;
+
+  void runOnOperation() override {
+    ModuleOp moduleOp = getOperation();
+
+    RewritePatternSet patterns(&getContext());
+    memref::populateElevateAllocsToGlobalsPatterns(patterns);
+
+    (void)applyPatternsGreedily(moduleOp, std::move(patterns));
+  }
+};
+} // namespace
+
+void mlir::memref::populateElevateAllocsToGlobalsPatterns(
+    RewritePatternSet &patterns) {
+  patterns.insert<ElevateAllocsToGlobals>(patterns.getContext());
+}
diff --git a/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
new file mode 100644
index 0000000000000..3ed66024fd57e
--- /dev/null
+++ b/mlir/test/Dialect/MemRef/elevate-allocs-to-globals.mlir
@@ -0,0 +1,140 @@
+// RUN: mlir-opt --elevate-allocs-to-globals --split-input-file %s | FileCheck %s
+
+/// Test that a single static memref.alloc is elevated to a memref.global, references
+/// are replaced with memref.get_global, and the associated memref.dealloc is removed.
+
+func.func @single_alloc() -> memref<10xf32> {
+  %0 = memref.alloc() {alignment = 64 : i64} : memref<10xf32>
+  memref.dealloc %0 : memref<10xf32>
+  return %0 : memref<10xf32>
+}
+
+// CHECK-LABEL: func.func @single_alloc() -> memref<10xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<10xf32>
+// CHECK-NOT:     memref.dealloc
+// CHECK:       memref.global "private" @global_alloc : memref<10xf32> = uninitialized {alignment = 64 : i64}
+
+// -----
+
+/// Test that multiple static memref.alloc ops in the same function are elevated
+/// to global memrefs without symbol name collisions.
+
+func.func @multiple_allocs() -> (memref<10xf32>, memref<20xf32>) {
+  %0 = memref.alloc() : memref<10xf32>
+  %1 = memref.alloc() : memref<20xf32>
+  return %0, %1 : memref<10xf32>, memref<20xf32>
+}
+
+// CHECK-LABEL: func.func @multiple_allocs() -> (memref<10xf32>, memref<20xf32>) {
+// CHECK-DAG:    %[[MEM0:.*]] = memref.get_global @global_alloc_0 : memref<10xf32>
+// CHECK-DAG:    %[[MEM1:.*]] = memref.get_global @global_alloc : memref<20xf32>
+// CHECK:        return %[[MEM0]], %[[MEM1]] : memref<10xf32>, memref<20xf32>
+// CHECK-DAG:    memref.global "private" @global_alloc : memref<20xf32> = uninitialized
+// CHECK-DAG:    memref.global "private" @global_alloc_0 : memref<10xf32> = uninitialized
+
+// -----
+
+/// Test that a dynamically-shaped memref.alloc is ignored and not elevated to a global.
+
+func.func @dynamic_alloc_ignored(%sz: index) -> memref<?xf32> {
+  %0 = memref.alloc(%sz) : memref<?xf32>
+  return %0 : memref<?xf32>
+}
+
+// CHECK-LABEL: func.func @dynamic_alloc_ignored(
+// CHECK-SAME:  %[[SZ:.*]]: index) -> memref<?xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc(%[[SZ]]) : memref<?xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<?xf32>
+
+// -----
+
+/// Test that a partially dynamic memref.alloc is ignored and not elevated to a global.
+
+func.func @partially_dynamic_alloc_ignored(%sz: index) -> memref<10x?xf32> {
+  %0 = memref.alloc(%sz) : memref<10x?xf32>
+  return %0 : memref<10x?xf32>
+}
+
+// CHECK-LABEL: func.func @partially_dynamic_alloc_ignored(
+// CHECK-SAME:  %[[SZ:.*]]: index) -> memref<10x?xf32> {
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc(%[[SZ]]) : memref<10x?xf32>
+// CHECK-NEXT:    return %[[MEM]] : memref<10x?xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside a loop is ignored and not elevated to a global.
+
+func.func @alloc_in_loop_ignored(%lb: index, %ub: index, %step: index) {
+  scf.for %i = %lb to %ub step %step {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_loop_ignored(
+// CHECK:       scf.for
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside control flow (scf.if) is ignored and not
+/// elevated to a global.
+
+func.func @alloc_in_control_flow_ignored(%cond: i1) {
+  scf.if %cond {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
+  }
+  return
+}
+
+// CHECK-LABEL: func.func @alloc_in_control_flow_ignored(
+// CHECK:       scf.if
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
+
+// -----
+
+/// Test that a static memref.alloc inside a non-ModuleOp symbol table is
+/// ignored and not elevated to a global.
+
+gpu.module @gpu_mod {
+  gpu.func @kernel() {
+    %0 = memref.alloc() : memref<10xf32>
+    memref.dealloc %0 : memref<10xf32>
+    gpu.return
+  }
+}
+
+// CHECK-LABEL: gpu.module @gpu_mod
+// CHECK:       gpu.func @kernel
+// CHECK-NEXT:    %[[MEM:.*]] = memref.alloc() : memref<10xf32>
+// CHECK-NEXT:    memref.dealloc %[[MEM]] : memref<10xf32>
+// CHECK-NOT:   memref.global
+
+// -----
+
+/// Test that in a function with allocs both inside and outside of control flow,
+/// only the alloc outside of control flow is elevated to a global.
+
+func.func @mixed_control_flow_allocs(%cond: i1) -> memref<10xf32> {
+  %outside = memref.alloc() : memref<10xf32>
+  scf.if %cond {
+    %inside = memref.alloc() : memref<20xf32>
+    memref.dealloc %inside : memref<20xf32>
+  }
+  return %outside : memref<10xf32>
+}
+
+// CHECK-LABEL: func.func @mixed_control_flow_allocs(
+// CHECK-SAME:  %[[COND:.*]]: i1) -> memref<10xf32> {
+// CHECK-NEXT:    %[[OUTSIDE:.*]] = memref.get_global @global_alloc : memref<10xf32>
+// CHECK-NEXT:    scf.if %[[COND]] {
+// CHECK-NEXT:      %[[INSIDE:.*]] = memref.alloc() : memref<20xf32>
+// CHECK-NEXT:      memref.dealloc %[[INSIDE]] : memref<20xf32>
+// CHECK-NEXT:    }
+// CHECK-NEXT:    return %[[OUTSIDE]] : memref<10xf32>
+// CHECK:       memref.global "private" @global_alloc : memref<10xf32> = uninitialized

``````````

</details>


https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Thu Jul 23 13:59:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 13:59:59 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [openmp] [Flang][OpenMP] Lower
 scan directive and inscan reduction modifier (PR #206747)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206747@github.com>
Message-ID: <6a6280cf.d2cb4adc.23815e.81c7@mx.google.com>


chichunchen wrote:

Please also add 2-dimension or even multiple-dimension test cases to mlir translation and lowering, thanks.

https://github.com/llvm/llvm-project/pull/206747

From llvmlistbot at llvm.org  Thu Jul 23 14:51:38 2026
From: llvmlistbot at llvm.org (Scott Manley)
Date: Thu, 23 Jul 2026 14:51:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] set CollapseCountAttr for
	scf.parallel (PR #211676)
Message-ID: <llvm.org/llvm/llvm-project/pull/211676@github.com>


https://github.com/rscottmanley created https://github.com/llvm/llvm-project/pull/211676

During compute lowering, also set the collapse count attribute when creating scf.parallel ops from acc.loop ops. While this collapse count is implied by the number of IVs, later canonicalizations may use this information to decide whether or not to collapse the loop nest further

>From 57131a55c852fea238e768c13ee0596b35e3e336 Mon Sep 17 00:00:00 2001
From: Scott Manley <scmanley at nvidia.com>
Date: Thu, 23 Jul 2026 14:48:32 -0700
Subject: [PATCH] [OpenACC] set CollapseCountAttr for scf.parallel

During compute lowering, also set the collapse count attribute when
creating scf.parallel ops from acc.loop ops. While this collapse count
is implied by the number of IVs, later canonicalizations may use this
information to decide whether or not to collapse the loop nest further
---
 .../OpenACC/Utils/OpenACCUtilsLoop.cpp        |  1 +
 .../OpenACC/acc-compute-lowering-loop.mlir    | 28 +++++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
index fd46a2800e03d..52f97e0867ee6 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
@@ -320,6 +320,7 @@ scf::ParallelOp convertACCLoopToSCFParallel(LoopOp loopOp,
       normalizeIVUses(rewriter, loc, iv, loopOp.getLowerbound()[idx],
                       loopOp.getStep()[idx]);
 
+  setCollapseCountAttr(parallelOp, parallelOp.getNumLoops());
   return parallelOp;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir b/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
index 0207157383c49..0772e6a2a536d 100644
--- a/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
@@ -83,6 +83,34 @@ func.func @parallel_loop_auto_collapse(%buf: memref<1xi32>, %lb0 : index, %ub0 :
 
 // -----
 
+// CHECK-LABEL: func.func @parallel_loop_collapse
+func.func @parallel_loop_collapse(%buf: memref<1xi32>, %lb0 : index, %ub0 : index, %lb1 : index, %ub1 : index) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+
+  %dev = acc.copyin varPtr(%buf : memref<1xi32>) -> memref<1xi32>
+  // CHECK-NOT: acc.parallel
+  // CHECK: acc.kernel_environment
+  // CHECK-NOT: acc.par_width
+  // CHECK: acc.compute_region
+  // CHECK: scf.parallel (%{{.*}}, %{{.*}})
+  // CHECK-NOT: scf.for
+  // CHECK-NOT: scf.parallel
+  // CHECK: acc.collapse_count = 2 : i64
+  acc.parallel dataOperands(%dev : memref<1xi32>) {
+    acc.loop control(%i : index, %j : index) = (%lb0, %lb1 : index, index) to (%ub0, %ub1 : index, index) step (%c1, %c1 : index, index) {
+      %vi = arith.index_cast %i : index to i32
+      memref.store %vi, %dev[%c0] : memref<1xi32>
+      acc.yield
+    } attributes {independent = [#acc.device_type<none>]}
+    acc.yield
+  }
+  acc.copyout accPtr(%dev : memref<1xi32>) to varPtr(%buf : memref<1xi32>)
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @serial_loop_normalized
 func.func @serial_loop_normalized(%buf: memref<1xi32>) {
   %c0 = arith.constant 0 : index


From llvmlistbot at llvm.org  Thu Jul 23 14:52:58 2026
From: llvmlistbot at llvm.org (Razvan Lupusoru)
Date: Thu, 23 Jul 2026 14:52:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] set CollapseCountAttr for
	scf.parallel (PR #211676)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211676@github.com>
Message-ID: <6a628d3a.1ba19891.1d5373.24b1@mx.google.com>


https://github.com/razvanlupusoru approved this pull request.

Thank you.

https://github.com/llvm/llvm-project/pull/211676

From llvmlistbot at llvm.org  Thu Jul 23 14:54:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 14:54:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] f4b5fb1 - [mlir][OpenACC] Support partial
 ThreadX reduction launches (#211665)
Message-ID: <6a628daf.c4d3d80a.16dae5.d6e6@mx.google.com>


Author: Matsu
Date: 2026-07-23T14:54:50-07:00
New Revision: f4b5fb1569eb944ebe18e52c83e6ad9b0239f53c

URL: https://github.com/llvm/llvm-project/commit/f4b5fb1569eb944ebe18e52c83e6ad9b0239f53c
DIFF: https://github.com/llvm/llvm-project/commit/f4b5fb1569eb944ebe18e52c83e6ad9b0239f53c.diff

LOG: [mlir][OpenACC] Support partial ThreadX reduction launches (#211665)

Example:
```fortran
!$acc parallel loop collapse(2) num_gangs(10) reduction(+:a)
do i = 1, n
  do j = 1, n
    do k = 1, m
      a(k) = a(k) + x(k,i) * y(k,j)
    end do
  end do
end do
```

In this code, ACCCG pads ThreadX to the subgroup size because
`gpu.all_reduce` uses subgroup shuffles. This unnecessarily increases
private reduction storage when only one Y/Z row is active.

Fix: preserve partial ThreadX launches for a single row, while retaining
alignment when Y/Z rows could share a subgroup.

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
    mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 618e45a803146..3bb78cd834b3e 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1032,12 +1032,16 @@ LogicalResult ACCCGToGPULowering::rewrite() {
     // - Subgroup reductions (gpu.all_reduce) require full subgroups
     // - Per-row workgroup barriers require blockDim.x aligned to subgroupSize
     bool isShuffleEnabled = false;
+    bool alignThreadXReduction =
+        getConstantIntValue(launch.getBlockSizeY()) != 1 ||
+        getConstantIntValue(launch.getBlockSizeZ()) != 1;
 
     launch.walk([&](gpu::AllReduceOp allReduce) -> WalkResult {
       ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
           mlir::acc::getParDimsAttr(allReduce).getArray();
       for (auto parDim : parDims) {
-        if (parDim.isThreadX() || parDim.isThreadY()) {
+        if (parDim.isThreadY() ||
+            (alignThreadXReduction && parDim.isThreadX())) {
           // Shuffle are enabled. Need to adjust the ThreadX length.
           isShuffleEnabled = true;
           return WalkResult::interrupt();
@@ -1055,7 +1059,8 @@ LogicalResult ACCCGToGPULowering::rewrite() {
             ArrayRef<mlir::acc::GPUParallelDimAttr> parDims =
                 mlir::acc::getParDimsAttr(allReduce).getArray();
             for (auto parDim : parDims) {
-              if (parDim.isThreadX() || parDim.isThreadY()) {
+              if (parDim.isThreadY() ||
+                  (alignThreadXReduction && parDim.isThreadX())) {
                 isShuffleEnabled = true;
                 return WalkResult::interrupt();
               }
@@ -1073,6 +1078,7 @@ LogicalResult ACCCGToGPULowering::rewrite() {
 
       Value curBlockDimX = launch.getBlockSizeX();
       Value curBlockDimY = launch.getBlockSizeY();
+      Value curBlockDimZ = launch.getBlockSizeZ();
 
       // Emit a report on changing parallelism.
       accSupport.emitRemark(computeRegion, [&]() {
@@ -1099,38 +1105,47 @@ LogicalResult ACCCGToGPULowering::rewrite() {
 
       std::optional<int64_t> constBlockDimX = getConstantIntValue(curBlockDimX);
       std::optional<int64_t> constBlockDimY = getConstantIntValue(curBlockDimY);
+      std::optional<int64_t> constBlockDimZ = getConstantIntValue(curBlockDimZ);
 
       // Skip subgroup alignment only when the total thread count is already
       // below a subgroup (constant blockDim.x in 2..subgroupSize-1 and
-      // constant blockDim.y == 1). If blockDim.y > 1 or is unknown, padding
-      // blockDim.x to a subgroup is still required so subgroups don't cross
-      // row boundaries for row-local shuffle/ThreadY-barrier reductions.
+      // constant blockDim.y/z == 1). If blockDim.y/z > 1 or is unknown,
+      // padding blockDim.x to a subgroup is still required so subgroups don't
+      // cross row boundaries for row-local shuffle/ThreadY-barrier reductions.
       bool skipAlign = false;
-      if (constBlockDimX && constBlockDimY && *constBlockDimX > 1 &&
-          *constBlockDimX < subgroupSize && *constBlockDimY == 1) {
+      if (constBlockDimX && constBlockDimY && constBlockDimZ &&
+          *constBlockDimX > 1 && *constBlockDimX < subgroupSize &&
+          *constBlockDimY == 1 && *constBlockDimZ == 1) {
         skipAlign = true;
       }
 
-      // Update both the ThreadX length and the number of ThreadY.
-      // When the original blockDim.x and blockDim.y are compile-time
+      // Update the ThreadX length and the numbers of ThreadY and ThreadZ.
+      // When the original block dimensions are compile-time
       // constants, compute the adjusted dimensions as constants directly so
       // that the GpuKernelOutliningPass can set `known_block_size` on the
       // outlined gpu.func.
-      Value newBlockDimX, newBlockDimY;
-      if (constBlockDimX && constBlockDimY) {
+      Value newBlockDimX, newBlockDimY, newBlockDimZ;
+      if (constBlockDimX && constBlockDimY && constBlockDimZ) {
         int64_t bdx = *constBlockDimX;
         int64_t bdy = *constBlockDimY;
+        int64_t bdz = *constBlockDimZ;
         int64_t alignedBdx =
             ((bdx + subgroupAlignMask) / subgroupSize) * subgroupSize;
-        int64_t numThreads = bdx * bdy;
-        int64_t newBdy = std::max<int64_t>(1, numThreads / alignedBdx);
+        int64_t numXYThreads = bdx * bdy;
+        int64_t numThreads = numXYThreads * bdz;
+        int64_t newBdy = std::max<int64_t>(1, numXYThreads / alignedBdx);
+        int64_t newBdz =
+            std::max<int64_t>(1, numThreads / (alignedBdx * newBdy));
         newBlockDimX =
             arith::ConstantIndexOp::create(rewriter, loc, alignedBdx);
         newBlockDimY = arith::ConstantIndexOp::create(rewriter, loc, newBdy);
+        newBlockDimZ = arith::ConstantIndexOp::create(rewriter, loc, newBdz);
       } else {
-        // numThreads = blockDim.x * blockDim.y
-        Value numThreads =
+        // numXYThreads = blockDim.x * blockDim.y
+        Value numXYThreads =
             arith::MulIOp::create(rewriter, loc, curBlockDimX, curBlockDimY);
+        Value numThreads =
+            arith::MulIOp::create(rewriter, loc, numXYThreads, curBlockDimZ);
         // blockDim.x = ((blockDim.x + mask) / subgroupSize) * subgroupSize
         Value cstMask =
             arith::ConstantIndexOp::create(rewriter, loc, subgroupAlignMask);
@@ -1142,16 +1157,23 @@ LogicalResult ACCCGToGPULowering::rewrite() {
             arith::DivUIOp::create(rewriter, loc, padded, cstSubgroupSize);
         newBlockDimX = arith::MulIOp::create(rewriter, loc, subgroupsRequired,
                                              cstSubgroupSize);
-        // blockDim.y = max(1, numThreads / blockDim.x)
+        // blockDim.y = max(1, numXYThreads / blockDim.x)
         Value quotient =
-            arith::DivUIOp::create(rewriter, loc, numThreads, newBlockDimX);
+            arith::DivUIOp::create(rewriter, loc, numXYThreads, newBlockDimX);
         Value cst1 = arith::ConstantIndexOp::create(rewriter, loc, 1);
         newBlockDimY = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
+        // blockDim.z = max(1, numThreads / (blockDim.x * blockDim.y))
+        Value newNumXYThreads =
+            arith::MulIOp::create(rewriter, loc, newBlockDimX, newBlockDimY);
+        quotient =
+            arith::DivUIOp::create(rewriter, loc, numThreads, newNumXYThreads);
+        newBlockDimZ = arith::MaxUIOp::create(rewriter, loc, cst1, quotient);
       }
 
       if (!skipAlign) {
         launch.getBlockSizeXMutable().assign(newBlockDimX);
         launch.getBlockSizeYMutable().assign(newBlockDimY);
+        launch.getBlockSizeZMutable().assign(newBlockDimZ);
       }
     }
   }

diff  --git a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
index 422e172e1ec5b..93354321865a7 100644
--- a/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-cg-to-gpu-reduction-array.mlir
@@ -255,3 +255,95 @@ func.func @rank_two_partial_bounds_strided_layout() {
   } {origin = "acc.parallel"}
   return
 }
+
+// CHECK-LABEL: func.func @partial_thread_x_reduction
+// CHECK: %[[C16:.*]] = arith.constant 16 : index
+// CHECK-NOT: arith.constant 31 : index
+// CHECK: gpu.launch
+// CHECK-SAME: threads({{.*}}) in (%{{.*}} = %[[C16]],
+// CHECK: gpu.all_reduce add
+func.func @partial_thread_x_reduction() {
+  %c1 = arith.constant 1 : index
+  %c16 = arith.constant 16 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c16 {par_dim = #acc.par_dim<thread_x>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx) {
+    %c2 = arith.constant 2 : index
+    %local = memref.alloca() : memref<2xi32>
+    %bounds = acc.bounds extent(%c2 : index)
+    acc.reduction_accumulate_array %local bounds(%bounds) <add>
+        : memref<2xi32> {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// CHECK-LABEL: func.func @thread_y_reduction_still_aligned
+// CHECK: %[[C32:.*]] = arith.constant 32 : index
+// CHECK: gpu.launch
+// CHECK-SAME: threads({{.*}}) in (%{{.*}} = %[[C32]],
+func.func @thread_y_reduction_still_aligned() {
+  %c1 = arith.constant 1 : index
+  %c16 = arith.constant 16 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %ty = acc.par_width %c16 {par_dim = #acc.par_dim<thread_y>}
+  acc.compute_region launch(%kbx = %bx, %kty = %ty) {
+    %c0_i32 = arith.constant 0 : i32
+    %local = memref.alloca() : memref<i32>
+    acc.reduction_accumulate %c0_i32 to %local <add>
+        : i32 -> memref<i32>
+        {par_dims = #acc<par_dims[block_x, thread_y]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// A ThreadX-only reduction still needs aligned rows when ThreadY is greater
+// than one, because a physical subgroup must not contain multiple logical rows.
+//
+// CHECK-LABEL: func.func @thread_x_reduction_with_thread_y_width
+// CHECK: %[[C32_ROWS:.*]] = arith.constant 32 : index
+// CHECK: gpu.launch
+// CHECK-SAME: threads({{.*}}) in (%{{.*}} = %[[C32_ROWS]],
+func.func @thread_x_reduction_with_thread_y_width() {
+  %c1 = arith.constant 1 : index
+  %c2 = arith.constant 2 : index
+  %c16 = arith.constant 16 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c16 {par_dim = #acc.par_dim<thread_x>}
+  %ty = acc.par_width %c2 {par_dim = #acc.par_dim<thread_y>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx, %kty = %ty) {
+    %c0_i32 = arith.constant 0 : i32
+    %local = memref.alloca() : memref<i32>
+    acc.reduction_accumulate %c0_i32 to %local <add>
+        : i32 -> memref<i32>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}
+
+// ThreadZ rows have the same subgroup-packing constraint as ThreadY rows.
+//
+// CHECK-LABEL: func.func @thread_x_reduction_with_thread_z_width
+// CHECK: %[[C352_Z_ROWS:.*]] = arith.constant 352 : index
+// CHECK: %[[C2_Z_ROWS:.*]] = arith.constant 2 : index
+// CHECK: gpu.launch
+// CHECK-SAME: threads({{.*}}) in (%{{.*}} = %[[C352_Z_ROWS]], %{{.*}} = %{{.*}}, %{{.*}} = %[[C2_Z_ROWS]])
+func.func @thread_x_reduction_with_thread_z_width() {
+  %c1 = arith.constant 1 : index
+  %c3 = arith.constant 3 : index
+  %c341 = arith.constant 341 : index
+  %bx = acc.par_width %c1 {par_dim = #acc.par_dim<block_x>}
+  %tx = acc.par_width %c341 {par_dim = #acc.par_dim<thread_x>}
+  %tz = acc.par_width %c3 {par_dim = #acc.par_dim<thread_z>}
+  acc.compute_region launch(%kbx = %bx, %ktx = %tx, %ktz = %tz) {
+    %c0_i32 = arith.constant 0 : i32
+    %local = memref.alloca() : memref<i32>
+    acc.reduction_accumulate %c0_i32 to %local <add>
+        : i32 -> memref<i32>
+        {par_dims = #acc<par_dims[block_x, thread_x]>}
+    acc.yield
+  } {origin = "acc.parallel"}
+  return
+}


        

From llvmlistbot at llvm.org  Thu Jul 23 14:54:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 14:54:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Support partial ThreadX
 reduction launches (PR #211665)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211665@github.com>
Message-ID: <6a628db2.1905e68e.25404.14e0@mx.google.com>


https://github.com/khaki3 closed https://github.com/llvm/llvm-project/pull/211665

From llvmlistbot at llvm.org  Thu Jul 23 14:54:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 14:54:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenACC] set CollapseCountAttr for
	scf.parallel (PR #211676)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211676@github.com>
Message-ID: <6a628db2.d76d81af.3b68ca.5f32@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Scott Manley (rscottmanley)

<details>
<summary>Changes</summary>

During compute lowering, also set the collapse count attribute when creating scf.parallel ops from acc.loop ops. While this collapse count is implied by the number of IVs, later canonicalizations may use this information to decide whether or not to collapse the loop nest further

---
Full diff: https://github.com/llvm/llvm-project/pull/211676.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp (+1) 
- (modified) mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir (+28) 


``````````diff
diff --git a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
index fd46a2800e03d..52f97e0867ee6 100644
--- a/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
+++ b/mlir/lib/Dialect/OpenACC/Utils/OpenACCUtilsLoop.cpp
@@ -320,6 +320,7 @@ scf::ParallelOp convertACCLoopToSCFParallel(LoopOp loopOp,
       normalizeIVUses(rewriter, loc, iv, loopOp.getLowerbound()[idx],
                       loopOp.getStep()[idx]);
 
+  setCollapseCountAttr(parallelOp, parallelOp.getNumLoops());
   return parallelOp;
 }
 
diff --git a/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir b/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
index 0207157383c49..0772e6a2a536d 100644
--- a/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
+++ b/mlir/test/Dialect/OpenACC/acc-compute-lowering-loop.mlir
@@ -83,6 +83,34 @@ func.func @parallel_loop_auto_collapse(%buf: memref<1xi32>, %lb0 : index, %ub0 :
 
 // -----
 
+// CHECK-LABEL: func.func @parallel_loop_collapse
+func.func @parallel_loop_collapse(%buf: memref<1xi32>, %lb0 : index, %ub0 : index, %lb1 : index, %ub1 : index) {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+
+  %dev = acc.copyin varPtr(%buf : memref<1xi32>) -> memref<1xi32>
+  // CHECK-NOT: acc.parallel
+  // CHECK: acc.kernel_environment
+  // CHECK-NOT: acc.par_width
+  // CHECK: acc.compute_region
+  // CHECK: scf.parallel (%{{.*}}, %{{.*}})
+  // CHECK-NOT: scf.for
+  // CHECK-NOT: scf.parallel
+  // CHECK: acc.collapse_count = 2 : i64
+  acc.parallel dataOperands(%dev : memref<1xi32>) {
+    acc.loop control(%i : index, %j : index) = (%lb0, %lb1 : index, index) to (%ub0, %ub1 : index, index) step (%c1, %c1 : index, index) {
+      %vi = arith.index_cast %i : index to i32
+      memref.store %vi, %dev[%c0] : memref<1xi32>
+      acc.yield
+    } attributes {independent = [#acc.device_type<none>]}
+    acc.yield
+  }
+  acc.copyout accPtr(%dev : memref<1xi32>) to varPtr(%buf : memref<1xi32>)
+  return
+}
+
+// -----
+
 // CHECK-LABEL: func.func @serial_loop_normalized
 func.func @serial_loop_normalized(%buf: memref<1xi32>) {
   %c0 = arith.constant 0 : index

``````````

</details>


https://github.com/llvm/llvm-project/pull/211676

From llvmlistbot at llvm.org  Thu Jul 23 19:49:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:49:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
Message-ID: <llvm.org/llvm/llvm-project/pull/211713@github.com>


https://github.com/cr-zhao created https://github.com/llvm/llvm-project/pull/211713

### Summary

- Propagate failures from partial Shard-to-MPI conversion and stop the pass before running folding patterns on partially converted IR.
- Propagate failures from greedy folding pattern application.
- Add a regression test for the assertion crash reported in #208331.

Fixes #208331

### Testing

- ShardToMPI invalid conversion regression test
- Existing ShardToMPI conversion test

>From bb851d774439f78fb0600b7df557df96960b7606 Mon Sep 17 00:00:00 2001
From: real-cpu <zhaochenrui757 at gmail.com>
Date: Thu, 23 Jul 2026 19:45:49 -0700
Subject: [PATCH] [mlir][Shard] Propagate failures in ConvertShardToMPI

---
 mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp |  9 +++++--
 .../convert-shard-to-mpi-invalid.mlir         | 24 +++++++++++++++++++
 2 files changed, 31 insertions(+), 2 deletions(-)
 create mode 100644 mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir

diff --git a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
index ea2825bff6bf7..94eb9a3e0f031 100644
--- a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
+++ b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
@@ -1212,13 +1212,18 @@ struct ConvertShardToMPIPass
     populateCallOpTypeConversionPattern(patterns, typeConverter);
     populateReturnOpTypeConversionPattern(patterns, typeConverter);
 
-    (void)applyPartialConversion(getOperation(), target, std::move(patterns));
+    if (failed(applyPartialConversion(getOperation(), target,
+                                      std::move(patterns)))) {
+      signalPassFailure();
+      return;
+    }
 
     // Folding patterns cannot be mixed with conversion patterns -> extra pass.
     patterns.clear();
     SymbolTableCollection symbolTableCollection;
     mlir::shard::populateFoldingPatterns(patterns, symbolTableCollection);
-    (void)applyPatternsGreedily(getOperation(), std::move(patterns));
+    if (failed(applyPatternsGreedily(getOperation(), std::move(patterns))))
+      signalPassFailure();
   }
 };
 
diff --git a/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
new file mode 100644
index 0000000000000..4d93e131e14be
--- /dev/null
+++ b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
@@ -0,0 +1,24 @@
+// RUN: mlir-opt %s -convert-shard-to-mpi -verify-diagnostics
+
+module {
+  func.func @vector_ops(%arg0: memref<4xf32>) {
+    %mask = vector.constant_mask [8] : vector<8xi1>
+    %zero = arith.constant 0.000000e+00 : f32
+    %broadcast = vector.broadcast %zero : f32 to vector<8xf32>
+    %reduction = vector.mask %mask {
+      vector.multi_reduction <add>, %broadcast, %zero [0]
+        : vector<8xf32> to f32
+    } : vector<8xi1> -> f32
+    %result = arith.addf %reduction, %zero : f32
+    return
+  }
+
+  func.func @unsupported_send(%arg0: tensor<4xf32>) {
+    // expected-error at +1 {{failed to legalize operation 'shard.send' that was explicitly marked illegal}}
+    %send = shard.send %arg0 on @grid destination = []
+      : (tensor<4xf32>) -> tensor<4xf32>
+    return
+  }
+
+  shard.grid @grid(shape = 2)
+}


From llvmlistbot at llvm.org  Thu Jul 23 19:50:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:50:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62d2d9.b1416da1.20b4cf.e3fe@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @cr-zhao :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 19:50:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:50:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62d2f9.21a19891.105e40.423f@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: crZhao (cr-zhao)

<details>
<summary>Changes</summary>

### Summary

- Propagate failures from partial Shard-to-MPI conversion and stop the pass before running folding patterns on partially converted IR.
- Propagate failures from greedy folding pattern application.
- Add a regression test for the assertion crash reported in #<!-- -->208331.

Fixes #<!-- -->208331

### Testing

- ShardToMPI invalid conversion regression test
- Existing ShardToMPI conversion test

---
Full diff: https://github.com/llvm/llvm-project/pull/211713.diff


2 Files Affected:

- (modified) mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp (+7-2) 
- (added) mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir (+24) 


``````````diff
diff --git a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
index ea2825bff6bf7..94eb9a3e0f031 100644
--- a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
+++ b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
@@ -1212,13 +1212,18 @@ struct ConvertShardToMPIPass
     populateCallOpTypeConversionPattern(patterns, typeConverter);
     populateReturnOpTypeConversionPattern(patterns, typeConverter);
 
-    (void)applyPartialConversion(getOperation(), target, std::move(patterns));
+    if (failed(applyPartialConversion(getOperation(), target,
+                                      std::move(patterns)))) {
+      signalPassFailure();
+      return;
+    }
 
     // Folding patterns cannot be mixed with conversion patterns -> extra pass.
     patterns.clear();
     SymbolTableCollection symbolTableCollection;
     mlir::shard::populateFoldingPatterns(patterns, symbolTableCollection);
-    (void)applyPatternsGreedily(getOperation(), std::move(patterns));
+    if (failed(applyPatternsGreedily(getOperation(), std::move(patterns))))
+      signalPassFailure();
   }
 };
 
diff --git a/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
new file mode 100644
index 0000000000000..4d93e131e14be
--- /dev/null
+++ b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
@@ -0,0 +1,24 @@
+// RUN: mlir-opt %s -convert-shard-to-mpi -verify-diagnostics
+
+module {
+  func.func @vector_ops(%arg0: memref<4xf32>) {
+    %mask = vector.constant_mask [8] : vector<8xi1>
+    %zero = arith.constant 0.000000e+00 : f32
+    %broadcast = vector.broadcast %zero : f32 to vector<8xf32>
+    %reduction = vector.mask %mask {
+      vector.multi_reduction <add>, %broadcast, %zero [0]
+        : vector<8xf32> to f32
+    } : vector<8xi1> -> f32
+    %result = arith.addf %reduction, %zero : f32
+    return
+  }
+
+  func.func @unsupported_send(%arg0: tensor<4xf32>) {
+    // expected-error at +1 {{failed to legalize operation 'shard.send' that was explicitly marked illegal}}
+    %send = shard.send %arg0 on @grid destination = []
+      : (tensor<4xf32>) -> tensor<4xf32>
+    return
+  }
+
+  shard.grid @grid(shape = 2)
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 19:51:10 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:51:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62d31e.160324b0.17e275.3e01@mx.google.com>


https://github.com/cr-zhao updated https://github.com/llvm/llvm-project/pull/211713

>From bb851d774439f78fb0600b7df557df96960b7606 Mon Sep 17 00:00:00 2001
From: real-cpu <zhaochenrui757 at gmail.com>
Date: Thu, 23 Jul 2026 19:45:49 -0700
Subject: [PATCH] [mlir][Shard] Propagate failures in ConvertShardToMPI

---
 mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp |  9 +++++--
 .../convert-shard-to-mpi-invalid.mlir         | 24 +++++++++++++++++++
 2 files changed, 31 insertions(+), 2 deletions(-)
 create mode 100644 mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir

diff --git a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
index ea2825bff6bf7..94eb9a3e0f031 100644
--- a/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
+++ b/mlir/lib/Conversion/ShardToMPI/ShardToMPI.cpp
@@ -1212,13 +1212,18 @@ struct ConvertShardToMPIPass
     populateCallOpTypeConversionPattern(patterns, typeConverter);
     populateReturnOpTypeConversionPattern(patterns, typeConverter);
 
-    (void)applyPartialConversion(getOperation(), target, std::move(patterns));
+    if (failed(applyPartialConversion(getOperation(), target,
+                                      std::move(patterns)))) {
+      signalPassFailure();
+      return;
+    }
 
     // Folding patterns cannot be mixed with conversion patterns -> extra pass.
     patterns.clear();
     SymbolTableCollection symbolTableCollection;
     mlir::shard::populateFoldingPatterns(patterns, symbolTableCollection);
-    (void)applyPatternsGreedily(getOperation(), std::move(patterns));
+    if (failed(applyPatternsGreedily(getOperation(), std::move(patterns))))
+      signalPassFailure();
   }
 };
 
diff --git a/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
new file mode 100644
index 0000000000000..4d93e131e14be
--- /dev/null
+++ b/mlir/test/Conversion/ShardToMPI/convert-shard-to-mpi-invalid.mlir
@@ -0,0 +1,24 @@
+// RUN: mlir-opt %s -convert-shard-to-mpi -verify-diagnostics
+
+module {
+  func.func @vector_ops(%arg0: memref<4xf32>) {
+    %mask = vector.constant_mask [8] : vector<8xi1>
+    %zero = arith.constant 0.000000e+00 : f32
+    %broadcast = vector.broadcast %zero : f32 to vector<8xf32>
+    %reduction = vector.mask %mask {
+      vector.multi_reduction <add>, %broadcast, %zero [0]
+        : vector<8xf32> to f32
+    } : vector<8xi1> -> f32
+    %result = arith.addf %reduction, %zero : f32
+    return
+  }
+
+  func.func @unsupported_send(%arg0: tensor<4xf32>) {
+    // expected-error at +1 {{failed to legalize operation 'shard.send' that was explicitly marked illegal}}
+    %send = shard.send %arg0 on @grid destination = []
+      : (tensor<4xf32>) -> tensor<4xf32>
+    return
+  }
+
+  shard.grid @grid(shape = 2)
+}


From llvmlistbot at llvm.org  Thu Jul 23 19:53:47 2026
From: llvmlistbot at llvm.org (Hocky Yudhiono)
Date: Thu, 23 Jul 2026 19:53:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve dropped conversion mappings
	(PR #209982)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209982@github.com>
Message-ID: <6a62d3bb.c4d3d80a.16dae5.682d@mx.google.com>


hockyy wrote:

Hi! btw, you can take a look at https://llvm.org/docs/AIToolPolicy.html
Don't forget to add `Assisted-by:` acknowledgment if you are using any LLM assisting tools.

https://github.com/llvm/llvm-project/pull/209982

From llvmlistbot at llvm.org  Thu Jul 23 19:54:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:54:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] fec3679 - [mlir][VectorToLLVM] Fix comment
 still referring to undef instead of poison (NFC) (#211196)
Message-ID: <6a62d3ea.6c5720b3.1f1229.bc2a@mx.google.com>


Author: Bryth
Date: 2026-07-24T02:54:30Z
New Revision: fec36790e2a09168aa85493d029d5fbfde3e11c0

URL: https://github.com/llvm/llvm-project/commit/fec36790e2a09168aa85493d029d5fbfde3e11c0
DIFF: https://github.com/llvm/llvm-project/commit/fec36790e2a09168aa85493d029d5fbfde3e11c0.diff

LOG: [mlir][VectorToLLVM] Fix comment still referring to undef instead of poison (NFC) (#211196)

When updating operators from undef (deprecated) to poison, the
associated comment wasn't updated accordingly. This PR fixes it.

Added: 
    

Modified: 
    mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
index 1720fd955a4e7..54c117bae476b 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
@@ -1821,7 +1821,7 @@ struct VectorBroadcastScalarToNdLowering
       return rewriter.notifyMatchFailure(
           broadcast, "broadcast to 1-d or 0-d handled elsewhere");
 
-    // First insert it into an undef vector so we can shuffle it.
+    // First insert it into a poison vector so we can shuffle it.
     auto loc = broadcast.getLoc();
     auto vectorTypeInfo =
         LLVM::detail::extractNDVectorTypeInfo(resultType, *getTypeConverter());


        

From llvmlistbot at llvm.org  Thu Jul 23 19:54:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:54:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211196@github.com>
Message-ID: <6a62d3ed.1b0626d0.1ff4c3.e325@mx.google.com>


https://github.com/zackc6 closed https://github.com/llvm/llvm-project/pull/211196

From llvmlistbot at llvm.org  Thu Jul 23 19:54:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:54:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][VectorToLLVM] Fix comment still
 referring to undef instead of poison (NFC) (PR #211196)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211196@github.com>
Message-ID: <6a62d3fb.7d702ed2.2f6c05.03fc@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@Brythzz Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/211196

From llvmlistbot at llvm.org  Thu Jul 23 19:54:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 19:54:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62d402.6c8668f2.1842d3.06ce@mx.google.com>


https://github.com/cr-zhao edited https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 20:05:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 20:05:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62d65c.5a0412d2.325e39.1bb6@mx.google.com>


cr-zhao wrote:

> Hello @cr-zhao 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read the linked policies, including the LLVM AI Tool Use Policy.
I reviewed and understand all changes, and the PR description discloses
the AI assistance used.

https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 20:46:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 20:46:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62e031.1b0626d0.1ff4c3.4a98@mx.google.com>


https://github.com/cr-zhao edited https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 20:47:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 20:47:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Shard] Propagate failures in
	ConvertShardToMPI (PR #211713)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211713@github.com>
Message-ID: <6a62e053.bc28b987.1c0bcd.b28f@mx.google.com>


https://github.com/cr-zhao edited https://github.com/llvm/llvm-project/pull/211713

From llvmlistbot at llvm.org  Thu Jul 23 22:09:01 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:09:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Keep ThreadY active for
 inner-combine-fed worker reductions (PR #211696)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211696@github.com>
Message-ID: <6a62f36d.a1accbb4.1ace1c.84d8@mx.google.com>


https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/211696

>From 9c99893bd0cf15ccb8e7e7afb9c21a8b4266822e Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Thu, 23 Jul 2026 16:41:01 -0700
Subject: [PATCH 1/2] [mlir][OpenACC] Keep ThreadY active for inner-combine-fed
 worker reductions

A block_y + thread_y reduction accumulator that is filled by an inner
block-scoped acc.reduction_combine holds one distinct partial per worker
row (each row's shared slot is written by the inner combine). Reducing it
into the destination must therefore run on every worker row. The previous
classification kept ThreadY active only for "proven worker-private"
combines (isThreadYPrivate with hasPrivateDest), which excludes the
block_y+thread_y -> grid/global shape produced by e.g. a CUF
`kernel do(2)` reduction, so it fell back to the row-zero legacy path and
dropped the other workers' partials (result -1 instead of -4 for a 2D SUM).

Add a discriminator: a block_y+thread_y accumulator keeps ThreadY active
when it is fed by an inner block-scoped combine (distinct per-worker
partials). A plain worker accumulate, which lowers to a worker-wide
all_reduce that already broadcasts the total, correctly stays row-zero to
avoid multiplying by the worker count.
---
 .../Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 51db77332c102..ede68126f4cae 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1240,6 +1240,32 @@ static bool hasUnsafeEffectsWhenBroadening(Operation *op) {
   return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
 }
 
+/// True when \p accumulator is the destination of a separate block-scoped
+/// acc.reduction_combine (an inner-level combine that fills its per-worker
+/// shared slot), so it holds one distinct partial per worker row rather than a
+/// worker-wide broadcast.
+static bool isFedByInnerBlockCombine(acc::PrivateLocalOp accumulator,
+                                     Operation *selfCombine) {
+  if (!accumulator)
+    return false;
+  for (Operation *user : accumulator.getResult().getUsers()) {
+    if (user == selfCombine)
+      continue;
+    auto combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp ||
+        unwrapMemRefConversion(combineOp.getDestMemref()).getDefiningOp() !=
+            accumulator.getOperation())
+      continue;
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        }))
+      return true;
+  }
+  return false;
+}
+
 /// Records whether \p combineOp requires ThreadY to remain active.
 static void classifyThreadYCombine(ThreadYBroadeningInfo &info,
                                    Operation *combineOp, Value src, Value dest,
@@ -1263,6 +1289,21 @@ static void classifyThreadYCombine(ThreadYBroadeningInfo &info,
     return;
   }
 
+  // A block_y + thread_y accumulator that is itself fed by an inner
+  // block-scoped combine holds one distinct partial per worker row (each row's
+  // shared slot was filled by the inner combine). Reducing it into the
+  // destination must therefore run on every worker row; predicating it to
+  // ThreadY row zero would drop the other workers' partials. This differs from
+  // a plain worker accumulate, which lowers to a worker-wide all_reduce that
+  // already broadcasts the total, and so must stay row-zero to avoid
+  // multiplying by the worker count.
+  if (hasThreadY && hasBlock &&
+      isThreadYPrivate(srcPrivate, /*allowBlock=*/true, computeRegion) &&
+      isFedByInnerBlockCombine(srcPrivate, combineOp)) {
+    info.hasActiveWorkerCombine = true;
+    return;
+  }
+
   info.hasExplicitInactiveCombine = true;
   if (!info.diagnosticOp)
     info.diagnosticOp = combineOp;

>From 090ae0ed6eea7f170da9ece392e8d28bb718e49c Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Thu, 23 Jul 2026 22:08:49 -0700
Subject: [PATCH 2/2] [mlir][OpenACC] NFC: shorten worker-reduction comment

---
 .../lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp | 15 ++++-----------
 1 file changed, 4 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index ede68126f4cae..969b782a7460c 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1241,9 +1241,7 @@ static bool hasUnsafeEffectsWhenBroadening(Operation *op) {
 }
 
 /// True when \p accumulator is the destination of a separate block-scoped
-/// acc.reduction_combine (an inner-level combine that fills its per-worker
-/// shared slot), so it holds one distinct partial per worker row rather than a
-/// worker-wide broadcast.
+/// combine, so it holds a distinct per-worker partial rather than a broadcast.
 static bool isFedByInnerBlockCombine(acc::PrivateLocalOp accumulator,
                                      Operation *selfCombine) {
   if (!accumulator)
@@ -1289,14 +1287,9 @@ static void classifyThreadYCombine(ThreadYBroadeningInfo &info,
     return;
   }
 
-  // A block_y + thread_y accumulator that is itself fed by an inner
-  // block-scoped combine holds one distinct partial per worker row (each row's
-  // shared slot was filled by the inner combine). Reducing it into the
-  // destination must therefore run on every worker row; predicating it to
-  // ThreadY row zero would drop the other workers' partials. This differs from
-  // a plain worker accumulate, which lowers to a worker-wide all_reduce that
-  // already broadcasts the total, and so must stay row-zero to avoid
-  // multiplying by the worker count.
+  // A block_y+thread_y accumulator fed by an inner block-scoped combine holds
+  // a distinct partial per worker row, so its combine runs on every row; a
+  // plain worker accumulate broadcasts via all_reduce and stays row-zero.
   if (hasThreadY && hasBlock &&
       isThreadYPrivate(srcPrivate, /*allowBlock=*/true, computeRegion) &&
       isFedByInnerBlockCombine(srcPrivate, combineOp)) {


From llvmlistbot at llvm.org  Thu Jul 23 22:29:49 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Thu, 23 Jul 2026 22:29:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Handle size-1 broadcast dim in
 insert_strided_slice layout setup (PR #211725)
Message-ID: <llvm.org/llvm/llvm-project/pull/211725@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/211725

This PR enhance `setupInsertStridedSliceResultLayout` to treat a size-1 source dim as a broadcast dim: keep its lane_data at 1 instead of requiring divisibility. 

Add regression tests for both the layout propagation setup and the subgroup-to-lane distribution of such an op.

Assisted-by-claude

>From 59b316d9c9fbd24e0f0ffb75aa3e9e41e11d416d Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 04:34:32 +0000
Subject: [PATCH] [mlir][xegpu] Handle size-1 broadcast dim in
 insert_strided_slice layout setup

setupInsertStridedSliceResultLayout asserted that every source dimension of
a vector.insert_strided_slice divides the consumer lane layout. This fails
for a size-1 source dim distributed across multiple lanes (e.g. inserting
vector<1x1xf32> into vector<8x1xf32> with lane_layout = [1, 16]), even though
that dim is simply broadcast across the lanes and the slice is inserted along
another dim.

Treat a size-1 source dim as a broadcast dim: keep its lane_data at 1 instead
of requiring divisibility. Add regression tests for both the layout
propagation setup and the subgroup-to-lane distribution of such an op.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../XeGPU/Transforms/XeGPULayoutImpl.cpp      | 13 ++++++++----
 mlir/test/Dialect/XeGPU/propagate-layout.mlir | 17 +++++++++++++++
 .../XeGPU/sg-to-lane-distribute-unit.mlir     | 21 +++++++++++++++++++
 3 files changed, 47 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 48889a8b8febc..ea0846d9d9da4 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -2683,10 +2683,15 @@ xegpu::DistributeLayoutAttr xegpu::setupInsertStridedSliceResultLayout(
                     "insertStridedSlice.");
   } else if (layoutKind == xegpu::LayoutKind::Lane) {
     for (int dim = 0; dim < srcRank; dim++) {
-      assert(srcShape[dim] % consumerLaneLayout[dim] == 0 &&
-             "srcShape must be divisible by laneLayout for all dimensions");
-      laneDataValue = std::min(srcShape[dim] / consumerLaneLayout[dim],
-                               consumerLaneData[dim]);
+      // A size-1 source dim is broadcast across the lanes of that dim.
+      if (srcShape[dim] == 1) {
+        laneDataValue = 1;
+      } else {
+        assert(srcShape[dim] % consumerLaneLayout[dim] == 0 &&
+               "srcShape must be divisible by laneLayout for all dimensions");
+        laneDataValue = std::min(srcShape[dim] / consumerLaneLayout[dim],
+                                 consumerLaneData[dim]);
+      }
       requiredResLayout =
           requiredResLayout.setDimData(dim, -1, -1, laneDataValue);
     }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..d2992904ab495 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -960,6 +960,23 @@ func.func @insert_strided_slice_with_slice_layout(%arg0: memref<8x16xf32>) {
 }
 }
 
+// -----
+gpu.module @test {
+// CHECK-LABEL: func.func @insert_strided_slice_lane_broadcast_dim(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x1xf32>) {
+// CHECK: %[[CST_SMALL:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<1.000000e+00> : vector<1x1xf32>
+// CHECK: %[[CST_LARGE:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0.000000e+00> : vector<8x1xf32>
+// CHECK: %[[INSERT:.*]] = vector.insert_strided_slice %[[CST_SMALL]], %[[CST_LARGE]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+func.func @insert_strided_slice_lane_broadcast_dim(%arg0: memref<8x1xf32>) {
+  %cst_small = arith.constant dense<1.0> : vector<1x1xf32>
+  %cst_large = arith.constant dense<0.0> : vector<8x1xf32>
+  %insert = vector.insert_strided_slice %cst_small, %cst_large {offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+  %tdesc = xegpu.create_nd_tdesc %arg0 : memref<8x1xf32> -> !xegpu.tensor_desc<8x1xf32>
+  xegpu.store_nd %insert, %tdesc[0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x1xf32>, !xegpu.tensor_desc<8x1xf32>
+  return
+}
+}
+
 // -----
 gpu.module @test{
   // CHECK-LABEL: load_store_matrix
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..bdd45a5d5fcea 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -1024,6 +1024,27 @@ gpu.func @vector_insert_strided_slice_different_ranks() {
   gpu.return
 }
 
+// The lane-distributed dim (dim 1) is size 1 in the source and thus broadcast
+// across the lanes of that dim; the insert happens on dim 0 only, so the
+// distributed shapes are unchanged.
+// CHECK-LABEL: gpu.func @vector_insert_strided_slice_lane_broadcast_dim
+// CHECK: %[[ISS:.*]] = vector.insert_strided_slice %{{.*}}, %{{.*}} {offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+gpu.func @vector_insert_strided_slice_lane_broadcast_dim() {
+  %0 = "some_op"()
+    : () -> vector<1x1xf32>
+  %1 = "some_op"()
+    : () -> vector<8x1xf32>
+  %2 = vector.insert_strided_slice %0, %1 { offsets = [0, 0], strides = [1, 1]
+    }
+    : vector<1x1xf32> into vector<8x1xf32>
+  %cl2 = xegpu.convert_layout %2
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
+    }> : vector<8x1xf32>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @convert_layout_removed_when_compatible
 // CHECK-NOT: xegpu.convert_layout
 gpu.func @convert_layout_removed_when_compatible() {


From llvmlistbot at llvm.org  Thu Jul 23 22:30:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:30:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Handle size-1 broadcast dim in
 insert_strided_slice layout setup (PR #211725)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211725@github.com>
Message-ID: <6a62f86d.160324b0.17e275.5df4@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

This PR enhance `setupInsertStridedSliceResultLayout` to treat a size-1 source dim as a broadcast dim: keep its lane_data at 1 instead of requiring divisibility. 

Add regression tests for both the layout propagation setup and the subgroup-to-lane distribution of such an op.

Assisted-by-claude

---
Full diff: https://github.com/llvm/llvm-project/pull/211725.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp (+9-4) 
- (modified) mlir/test/Dialect/XeGPU/propagate-layout.mlir (+17) 
- (modified) mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir (+21) 


``````````diff
diff --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
index 48889a8b8febc..ea0846d9d9da4 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPULayoutImpl.cpp
@@ -2683,10 +2683,15 @@ xegpu::DistributeLayoutAttr xegpu::setupInsertStridedSliceResultLayout(
                     "insertStridedSlice.");
   } else if (layoutKind == xegpu::LayoutKind::Lane) {
     for (int dim = 0; dim < srcRank; dim++) {
-      assert(srcShape[dim] % consumerLaneLayout[dim] == 0 &&
-             "srcShape must be divisible by laneLayout for all dimensions");
-      laneDataValue = std::min(srcShape[dim] / consumerLaneLayout[dim],
-                               consumerLaneData[dim]);
+      // A size-1 source dim is broadcast across the lanes of that dim.
+      if (srcShape[dim] == 1) {
+        laneDataValue = 1;
+      } else {
+        assert(srcShape[dim] % consumerLaneLayout[dim] == 0 &&
+               "srcShape must be divisible by laneLayout for all dimensions");
+        laneDataValue = std::min(srcShape[dim] / consumerLaneLayout[dim],
+                                 consumerLaneData[dim]);
+      }
       requiredResLayout =
           requiredResLayout.setDimData(dim, -1, -1, laneDataValue);
     }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..d2992904ab495 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -960,6 +960,23 @@ func.func @insert_strided_slice_with_slice_layout(%arg0: memref<8x16xf32>) {
 }
 }
 
+// -----
+gpu.module @test {
+// CHECK-LABEL: func.func @insert_strided_slice_lane_broadcast_dim(
+// CHECK-SAME: %[[ARG0:[0-9a-zA-Z]+]]: memref<8x1xf32>) {
+// CHECK: %[[CST_SMALL:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<1.000000e+00> : vector<1x1xf32>
+// CHECK: %[[CST_LARGE:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0.000000e+00> : vector<8x1xf32>
+// CHECK: %[[INSERT:.*]] = vector.insert_strided_slice %[[CST_SMALL]], %[[CST_LARGE]] {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+func.func @insert_strided_slice_lane_broadcast_dim(%arg0: memref<8x1xf32>) {
+  %cst_small = arith.constant dense<1.0> : vector<1x1xf32>
+  %cst_large = arith.constant dense<0.0> : vector<8x1xf32>
+  %insert = vector.insert_strided_slice %cst_small, %cst_large {offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+  %tdesc = xegpu.create_nd_tdesc %arg0 : memref<8x1xf32> -> !xegpu.tensor_desc<8x1xf32>
+  xegpu.store_nd %insert, %tdesc[0, 0] <{layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<8x1xf32>, !xegpu.tensor_desc<8x1xf32>
+  return
+}
+}
+
 // -----
 gpu.module @test{
   // CHECK-LABEL: load_store_matrix
diff --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..bdd45a5d5fcea 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -1024,6 +1024,27 @@ gpu.func @vector_insert_strided_slice_different_ranks() {
   gpu.return
 }
 
+// The lane-distributed dim (dim 1) is size 1 in the source and thus broadcast
+// across the lanes of that dim; the insert happens on dim 0 only, so the
+// distributed shapes are unchanged.
+// CHECK-LABEL: gpu.func @vector_insert_strided_slice_lane_broadcast_dim
+// CHECK: %[[ISS:.*]] = vector.insert_strided_slice %{{.*}}, %{{.*}} {offsets = [0, 0], strides = [1, 1]} : vector<1x1xf32> into vector<8x1xf32>
+gpu.func @vector_insert_strided_slice_lane_broadcast_dim() {
+  %0 = "some_op"()
+    : () -> vector<1x1xf32>
+  %1 = "some_op"()
+    : () -> vector<8x1xf32>
+  %2 = vector.insert_strided_slice %0, %1 { offsets = [0, 0], strides = [1, 1]
+    }
+    : vector<1x1xf32> into vector<8x1xf32>
+  %cl2 = xegpu.convert_layout %2
+    <{
+      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
+      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
+    }> : vector<8x1xf32>
+  gpu.return
+}
+
 // CHECK-LABEL: gpu.func @convert_layout_removed_when_compatible
 // CHECK-NOT: xegpu.convert_layout
 gpu.func @convert_layout_removed_when_compatible() {

``````````

</details>


https://github.com/llvm/llvm-project/pull/211725

From llvmlistbot at llvm.org  Thu Jul 23 22:50:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:50:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add mlirOperationGetParentWithName
	(PR #206533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206533@github.com>
Message-ID: <6a62fd18.f1c1923d.d9163.2b40@mx.google.com>


https://github.com/RattataKing commented:

Code looks good to me, but are there any example use cases that specifically require this new function?
If the main purpose of this mlir-c api is to create a python binding later, maybe it's easier to implement the traversal directly on the binding side instead, see the existing examples: https://github.com/llvm/llvm-project/blob/64bc0065055be6cd9d9008248f7321e886ab73b0/mlir/python/mlir/ir.py#L27-L51
The current implementation in this PR works too.

https://github.com/llvm/llvm-project/pull/206533

From llvmlistbot at llvm.org  Thu Jul 23 22:53:10 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:53:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
Message-ID: <llvm.org/llvm/llvm-project/pull/211730@github.com>


https://github.com/vitalybuka created https://github.com/llvm/llvm-project/pull/211730

After #207274, libc++ fails __assume_aligned assertion in __assume_valid_range.

`operator ArrayRef<T>()` just a cast of
`raw.data()`. `raw` is aligned copy of range from
from BytecodeReader done by
Builtin_DenseArrayRawDataParameter allocator.
However, if range is empty, aligned copying was
omitted, leaving unaligned `ArrayRef<char>`.

The fix is to replace unalined ArrayRef<> with
aligned default constructed.


>From 6fa48ea848628ed240e3e1ecdda5aa9039f4491e Mon Sep 17 00:00:00 2001
From: Vitaly Buka <vitalybuka at google.com>
Date: Thu, 23 Jul 2026 22:52:53 -0700
Subject: [PATCH] =?UTF-8?q?[=F0=9D=98=80=F0=9D=97=BD=F0=9D=97=BF]=20initia?=
 =?UTF-8?q?l=20version?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Created using spr 1.3.7
---
 mlir/include/mlir/IR/BuiltinAttributes.td | 3 +++
 mlir/lib/IR/BuiltinAttributes.cpp         | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/mlir/include/mlir/IR/BuiltinAttributes.td b/mlir/include/mlir/IR/BuiltinAttributes.td
index f238137734d56..ba2c374a9ab48 100644
--- a/mlir/include/mlir/IR/BuiltinAttributes.td
+++ b/mlir/include/mlir/IR/BuiltinAttributes.td
@@ -160,6 +160,9 @@ def Builtin_DenseArrayRawDataParameter : ArrayRefParameter<
           $_allocator.allocate($_self.size(), alignof(uint64_t)));
       llvm::uninitialized_copy($_self, alloc);
       $_dst = ArrayRef<char>(alloc, $_self.size());
+    } else {
+      // Align possibly unaligned empty range.
+      $_dst = {};
     }
   }];
 }
diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 9fda2ef8e5059..888b989785498 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -21,6 +21,7 @@
 #include "llvm/ADT/APSInt.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 
@@ -822,6 +823,7 @@ Attribute DenseArrayAttrImpl<T>::parse(AsmParser &parser, Type odsType) {
 template <typename T>
 DenseArrayAttrImpl<T>::operator ArrayRef<T>() const {
   ArrayRef<char> raw = getRawData();
+  assert(llvm::isAddrAligned(llvm::Align(alignof(T)), raw.data()));
   assert((raw.size() % sizeof(T)) == 0);
   return ArrayRef<T>(reinterpret_cast<const T *>(raw.data()),
                      raw.size() / sizeof(T));


From llvmlistbot at llvm.org  Thu Jul 23 22:53:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:53:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fde9.bc28b987.1c0bcd.cded@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-ods

Author: Vitaly Buka (vitalybuka)

<details>
<summary>Changes</summary>

After #<!-- -->207274, libc++ fails __assume_aligned assertion in __assume_valid_range.

`operator ArrayRef&lt;T&gt;()` just a cast of
`raw.data()`. `raw` is aligned copy of range from
from BytecodeReader done by
Builtin_DenseArrayRawDataParameter allocator.
However, if range is empty, aligned copying was
omitted, leaving unaligned `ArrayRef&lt;char&gt;`.

The fix is to replace unalined ArrayRef&lt;&gt; with
aligned default constructed.


---
Full diff: https://github.com/llvm/llvm-project/pull/211730.diff


2 Files Affected:

- (modified) mlir/include/mlir/IR/BuiltinAttributes.td (+3) 
- (modified) mlir/lib/IR/BuiltinAttributes.cpp (+2) 


``````````diff
diff --git a/mlir/include/mlir/IR/BuiltinAttributes.td b/mlir/include/mlir/IR/BuiltinAttributes.td
index f238137734d56..ba2c374a9ab48 100644
--- a/mlir/include/mlir/IR/BuiltinAttributes.td
+++ b/mlir/include/mlir/IR/BuiltinAttributes.td
@@ -160,6 +160,9 @@ def Builtin_DenseArrayRawDataParameter : ArrayRefParameter<
           $_allocator.allocate($_self.size(), alignof(uint64_t)));
       llvm::uninitialized_copy($_self, alloc);
       $_dst = ArrayRef<char>(alloc, $_self.size());
+    } else {
+      // Align possibly unaligned empty range.
+      $_dst = {};
     }
   }];
 }
diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 9fda2ef8e5059..888b989785498 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -21,6 +21,7 @@
 #include "llvm/ADT/APSInt.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 
@@ -822,6 +823,7 @@ Attribute DenseArrayAttrImpl<T>::parse(AsmParser &parser, Type odsType) {
 template <typename T>
 DenseArrayAttrImpl<T>::operator ArrayRef<T>() const {
   ArrayRef<char> raw = getRawData();
+  assert(llvm::isAddrAligned(llvm::Align(alignof(T)), raw.data()));
   assert((raw.size() % sizeof(T)) == 0);
   return ArrayRef<T>(reinterpret_cast<const T *>(raw.data()),
                      raw.size() / sizeof(T));

``````````

</details>


https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:53:45 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:53:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add mlirOperationGetParentWithName
	(PR #206533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206533@github.com>
Message-ID: <6a62fde9.3cce8983.247e47.ef36@mx.google.com>


https://github.com/PragmaTwice approved this pull request.


https://github.com/llvm/llvm-project/pull/206533

From llvmlistbot at llvm.org  Thu Jul 23 22:53:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:53:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fdea.d76d81af.3b68ca.b674@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Vitaly Buka (vitalybuka)

<details>
<summary>Changes</summary>

After #<!-- -->207274, libc++ fails __assume_aligned assertion in __assume_valid_range.

`operator ArrayRef&lt;T&gt;()` just a cast of
`raw.data()`. `raw` is aligned copy of range from
from BytecodeReader done by
Builtin_DenseArrayRawDataParameter allocator.
However, if range is empty, aligned copying was
omitted, leaving unaligned `ArrayRef&lt;char&gt;`.

The fix is to replace unalined ArrayRef&lt;&gt; with
aligned default constructed.


---
Full diff: https://github.com/llvm/llvm-project/pull/211730.diff


2 Files Affected:

- (modified) mlir/include/mlir/IR/BuiltinAttributes.td (+3) 
- (modified) mlir/lib/IR/BuiltinAttributes.cpp (+2) 


``````````diff
diff --git a/mlir/include/mlir/IR/BuiltinAttributes.td b/mlir/include/mlir/IR/BuiltinAttributes.td
index f238137734d56..ba2c374a9ab48 100644
--- a/mlir/include/mlir/IR/BuiltinAttributes.td
+++ b/mlir/include/mlir/IR/BuiltinAttributes.td
@@ -160,6 +160,9 @@ def Builtin_DenseArrayRawDataParameter : ArrayRefParameter<
           $_allocator.allocate($_self.size(), alignof(uint64_t)));
       llvm::uninitialized_copy($_self, alloc);
       $_dst = ArrayRef<char>(alloc, $_self.size());
+    } else {
+      // Align possibly unaligned empty range.
+      $_dst = {};
     }
   }];
 }
diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 9fda2ef8e5059..888b989785498 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -21,6 +21,7 @@
 #include "llvm/ADT/APSInt.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 
@@ -822,6 +823,7 @@ Attribute DenseArrayAttrImpl<T>::parse(AsmParser &parser, Type odsType) {
 template <typename T>
 DenseArrayAttrImpl<T>::operator ArrayRef<T>() const {
   ArrayRef<char> raw = getRawData();
+  assert(llvm::isAddrAligned(llvm::Align(alignof(T)), raw.data()));
   assert((raw.size() % sizeof(T)) == 0);
   return ArrayRef<T>(reinterpret_cast<const T *>(raw.data()),
                      raw.size() / sizeof(T));

``````````

</details>


https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:54:50 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:54:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fe2a.a1accbb4.1ace1c.658e@mx.google.com>


https://github.com/vitalybuka edited https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:55:00 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:55:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fe34.1905e68e.25404.c6c8@mx.google.com>


github-actions[bot] wrote:

<!--LLVM CODE FORMAT COMMENT: {clang-format}-->


:warning: C/C++ code formatter, clang-format found issues in your code. :warning:

<details>
<summary>
You can test this locally with the following command:
</summary>

``````````bash
git-clang-format --diff origin/main HEAD --extensions cpp -- mlir/lib/IR/BuiltinAttributes.cpp --diff_from_common_commit
``````````

:warning:
The reproduction instructions above might return results for more than one PR
in a stack if you are using a stacked PR workflow. You can limit the results by
changing `origin/main` to the base branch/commit you want to compare against.
:warning:

</details>

<details>
<summary>
View the diff from clang-format here.
</summary>

``````````diff
diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 888b98978..0f84f7a68 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -19,9 +19,9 @@
 #include "mlir/IR/SymbolTable.h"
 #include "mlir/IR/Types.h"
 #include "llvm/ADT/APSInt.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
-#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:55:46 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:55:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fe62.666420a4.3304c7.3853@mx.google.com>


https://github.com/vitalybuka updated https://github.com/llvm/llvm-project/pull/211730

>From 6fa48ea848628ed240e3e1ecdda5aa9039f4491e Mon Sep 17 00:00:00 2001
From: Vitaly Buka <vitalybuka at google.com>
Date: Thu, 23 Jul 2026 22:52:53 -0700
Subject: [PATCH 1/2] =?UTF-8?q?[=F0=9D=98=80=F0=9D=97=BD=F0=9D=97=BF]=20in?=
 =?UTF-8?q?itial=20version?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Created using spr 1.3.7
---
 mlir/include/mlir/IR/BuiltinAttributes.td | 3 +++
 mlir/lib/IR/BuiltinAttributes.cpp         | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/mlir/include/mlir/IR/BuiltinAttributes.td b/mlir/include/mlir/IR/BuiltinAttributes.td
index f238137734d56..ba2c374a9ab48 100644
--- a/mlir/include/mlir/IR/BuiltinAttributes.td
+++ b/mlir/include/mlir/IR/BuiltinAttributes.td
@@ -160,6 +160,9 @@ def Builtin_DenseArrayRawDataParameter : ArrayRefParameter<
           $_allocator.allocate($_self.size(), alignof(uint64_t)));
       llvm::uninitialized_copy($_self, alloc);
       $_dst = ArrayRef<char>(alloc, $_self.size());
+    } else {
+      // Align possibly unaligned empty range.
+      $_dst = {};
     }
   }];
 }
diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 9fda2ef8e5059..888b989785498 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -21,6 +21,7 @@
 #include "llvm/ADT/APSInt.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 
@@ -822,6 +823,7 @@ Attribute DenseArrayAttrImpl<T>::parse(AsmParser &parser, Type odsType) {
 template <typename T>
 DenseArrayAttrImpl<T>::operator ArrayRef<T>() const {
   ArrayRef<char> raw = getRawData();
+  assert(llvm::isAddrAligned(llvm::Align(alignof(T)), raw.data()));
   assert((raw.size() % sizeof(T)) == 0);
   return ArrayRef<T>(reinterpret_cast<const T *>(raw.data()),
                      raw.size() / sizeof(T));

>From ff11bbf480b62962dbae68a7e34f787eeb029e30 Mon Sep 17 00:00:00 2001
From: Vitaly Buka <vitalybuka at google.com>
Date: Thu, 23 Jul 2026 22:55:36 -0700
Subject: [PATCH 2/2] format

Created using spr 1.3.7
---
 mlir/lib/IR/BuiltinAttributes.cpp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 888b989785498..0f84f7a682fcf 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -19,9 +19,9 @@
 #include "mlir/IR/SymbolTable.h"
 #include "mlir/IR/Types.h"
 #include "llvm/ADT/APSInt.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
-#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Endian.h"
 #include <optional>
 


From llvmlistbot at llvm.org  Thu Jul 23 22:56:10 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:56:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fe7a.92d2873c.2cdd42.d50f@mx.google.com>


https://github.com/vitalybuka edited https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:56:21 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:56:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fe85.c56fb102.6921f.840d@mx.google.com>


https://github.com/vitalybuka edited https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:56:50 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 22:56:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62fea2.e5907290.33b622.b81b@mx.google.com>


https://github.com/vitalybuka edited https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 22:57:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 22:57:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add mlirOperationGetParentWithName
	(PR #206533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206533@github.com>
Message-ID: <6a62fed8.4987c784.13f2de.98a8@mx.google.com>


PragmaTwice wrote:

> If the main purpose of this mlir-c api is to create a python binding later, maybe it's easier to implement the traversal directly on the binding side instead, see the existing examples:

Should this function be a method of `_OperationBase`? It looks weird to be a top-level function to me.

https://github.com/llvm/llvm-project/pull/206533

From llvmlistbot at llvm.org  Thu Jul 23 22:58:37 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Thu, 23 Jul 2026 22:58:37 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a62ff0d.92d2873c.2cdd42.4937@mx.google.com>


https://github.com/echristo updated https://github.com/llvm/llvm-project/pull/210850

>From ab4cecf7ddb25e85c1a7a8bbfc38743dc2125ac2 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Wed, 15 Jul 2026 08:51:28 -0700
Subject: [PATCH 1/7] [DebugInfo] Add symbolic branches to DIExpression

Added three pseudo-ops to DIExpression:

- DW_OP_LLVM_label declares a label and emits no bytes.
- DW_OP_LLVM_bra branches to a label when the top of the stack is non-zero.
- DW_OP_LLVM_skip always branches to a label.

DW_OP_bra and DW_OP_skip store a two-byte offset in [-32768, 32767], but
we don't know those offsets until we emit the expression. DIExpression keeps
label IDs, and we fix them up late in CodeGen.

CodeGen records each label's byte offset and leaves a zero placeholder for each
branch. Once it has seen the whole expression, it patches the placeholders in
the existing temporary buffers used for DIE and location-list expressions.
Both paths keep their current encoding and target byte order.

The verifier makes sure every branch has a label in the same expression and
each label ID is declared once. Labels don't need a branch, and branches can go
forward, backward, to themselves, or form cycles. Raw DW_OP_bra and DW_OP_skip
remain invalid in IR because their offsets depend on the final encoding.

Symbolic control flow only works with one location operand. We reject expressions
that combine it with DW_OP_LLVM_arg or DW_OP_LLVM_implicit_pointer, and diagnose
DIArgList records. Existing expression rewrites keep the pseudo-ops and their
IDs in place.

CodeGen reports an error if the final offset is outside [-32768, 32767], or if a
label, branch, or skip splits a deferred conversion. DWARF 5 emits DW_OP_convert
directly, but older versions may defer one conversion until they see the next.

Tests cover IR and MLIR round trips, verifier failures, forward and backward
branches, self-branches and cycles, both byte orders, conversion boundaries,
and out-of-range offsets.
---
 llvm/docs/LangRef.md                          |  16 ++
 llvm/docs/ReleaseNotes.md                     |   6 +
 llvm/docs/SourceLevelDebugging.md             |  61 ++++++-
 llvm/include/llvm/BinaryFormat/Dwarf.h        |   5 +
 llvm/lib/BinaryFormat/Dwarf.cpp               |   9 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  27 ++-
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  83 +++++++++-
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h |  27 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     |  52 +++++-
 llvm/lib/IR/DebugInfoMetadata.cpp             | 103 ++++++++++--
 llvm/lib/IR/Verifier.cpp                      |  13 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../X86/di-expression-symbolic-branches.mir   | 102 ++++++++++++
 .../di-expression-symbolic-branches.ll        |  44 +++++
 ...expression-symbolic-branch-out-of-range.py |  33 ++++
 ...ession-symbolic-branch-convert-boundary.ll |  96 +++++++++++
 ...expression-symbolic-branch-out-of-range.ll |   7 +
 .../X86/di-expression-symbolic-branches.ll    |  84 ++++++++++
 .../di-expression-symbolic-branches.ll        |  93 +++++++++++
 llvm/unittests/BinaryFormat/DwarfTest.cpp     |   7 +
 llvm/unittests/IR/MetadataTest.cpp            | 155 ++++++++++++++++++
 .../Import/debug-info-symbolic-branches.ll    |  23 +++
 .../llvmir-debug-symbolic-branches.mlir       |  19 +++
 24 files changed, 1054 insertions(+), 35 deletions(-)
 create mode 100644 llvm/test/Assembler/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Bitcode/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
 create mode 100644 llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Verifier/di-expression-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..05162290f330a 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7159,6 +7159,22 @@ Some examples of expressions:
 !DIExpression(DW_OP_constu, 42, DW_OP_stack_value)
 ```
 
+`DIExpression` uses three pseudo-ops for local control flow:
+
+```text
+DW_OP_LLVM_label, <label-id>
+DW_OP_LLVM_bra,   <label-id>
+DW_OP_LLVM_skip,  <label-id>
+```
+
+`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
+branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
+branches. Label IDs are local to the expression. Every branch needs a matching
+label, labels without branches are valid, and each ID can only be declared
+once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
+
+See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+
 ##### DIAssignID
 
 `DIAssignID` nodes have no operands and are always distinct. They are used to
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 94208f8e36d19..988740a7e5e39 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -109,6 +109,12 @@ Makes programs 10x faster by doing Special New Thing.
 
 ### Changes to the Debug Info
 
+* Added `DW_OP_LLVM_label`, `DW_OP_LLVM_bra`, and `DW_OP_LLVM_skip` for symbolic
+  branches in `DIExpression`. These operations use label IDs that CodeGen
+  resolves when it emits the expression. `DW_OP_LLVM_convert` can appear before
+  or after them, but when CodeGen can't emit `DW_OP_convert`, it reports an
+  error if a deferred conversion reaches a label, branch, or skip.
+
 ### Changes to the LLVM tools
 
 ### Changes to LLDB
diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 74bb8840a56e4..645e441e6fe2a 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -386,10 +386,9 @@ call void @llvm.dbg.assign(
 
 Debug expressions are represented as {ref}`specialized-metadata`.
 
-Debug expressions are interpreted left-to-right: start by pushing the
-value/address operand of the record onto a stack, then repeatedly push and
-evaluate opcodes from the `DIExpression` until the final variable description
-is produced.
+A debug expression starts with the record's value or address operand on the
+stack, then evaluates operations from left to right unless a symbolic branch
+jumps to a label in the same `DIExpression`.
 
 The opcodes available in these expressions are described in
 {ref}`dwarf-opcodes` and {ref}`internal-opcodes`.
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
+
+- Put labels, branches, and skips before `DW_OP_stack_value` and
+  `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
+- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
+  separately, so they can't be used with symbolic control flow.
+- Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
+  applies to every path.
+
+We don't check reachability, termination, or stack state where paths meet.
+
+`DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
+and we don't match conversions on different paths. When CodeGen can't emit
+`DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
+branch, or skip would split the pair, CodeGen reports an error.
+
+Local expression rewrites stop at labels, branches, and skips; they can still
+add operations to either end, but they don't move, remove, or copy labels.
+
+##### Other Internal Opcodes
+
 - `DW_OP_LLVM_fragment, <offset>, <size>` may appear at most once in an
   expression, and must be the last opcode. It specifies the bit offset and bit
   size of the variable fragment being described by the record or intrinsic
@@ -482,11 +525,15 @@ them.
 - `DW_OP_LLVM_convert, 16, DW_ATE_signed` specifies a bit size and encoding
   (`16` and `DW_ATE_signed` here, respectively) to which the top of the
   expression stack is to be converted. Maps into a `DW_OP_convert` operation
-  that references a base type constructed from the supplied values.
+  that references a base type constructed from the supplied values. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for how conversions work
+  with labels, branches, and skips.
 - `DW_OP_LLVM_tag_offset, tag_offset` specifies that a memory tag should be
   optionally applied to the pointer. The memory tag is derived from the given
-  tag offset in an implementation-defined manner. (This does not affect the
-  semantics of the expression containing it.)
+  tag offset in an implementation-defined manner. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for its placement with
+  labels, branches, and skips.
+  (This does not affect the semantics of the expression containing it.)
 - `DW_OP_LLVM_entry_value, N` evaluates a sub-expression as-if it were
   evaluated upon entry to the current call frame.
 
diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 75eb3a163e06f..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -149,6 +149,11 @@ enum LocationAtom {
   DW_OP_LLVM_arg = 0x1005,               ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_sext = 0x1006, ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_zext = 0x1007, ///< Only used in LLVM metadata.
+  // Labels declare IDs local to each expression, and branches use them as
+  // targets.
+  DW_OP_LLVM_label = 0x1008, ///< Only used in LLVM metadata.
+  DW_OP_LLVM_bra = 0x1009,   ///< Only used in LLVM metadata.
+  DW_OP_LLVM_skip = 0x100a,  ///< Only used in LLVM metadata.
 };
 
 enum LlvmUserLocationAtom {
diff --git a/llvm/lib/BinaryFormat/Dwarf.cpp b/llvm/lib/BinaryFormat/Dwarf.cpp
index 9ede7ebbe20ad..431d01c953768 100644
--- a/llvm/lib/BinaryFormat/Dwarf.cpp
+++ b/llvm/lib/BinaryFormat/Dwarf.cpp
@@ -159,6 +159,12 @@ StringRef llvm::dwarf::OperationEncodingString(unsigned Encoding) {
     return "DW_OP_LLVM_extract_bits_sext";
   case DW_OP_LLVM_extract_bits_zext:
     return "DW_OP_LLVM_extract_bits_zext";
+  case DW_OP_LLVM_label:
+    return "DW_OP_LLVM_label";
+  case DW_OP_LLVM_bra:
+    return "DW_OP_LLVM_bra";
+  case DW_OP_LLVM_skip:
+    return "DW_OP_LLVM_skip";
   }
 }
 
@@ -175,6 +181,9 @@ unsigned llvm::dwarf::getOperationEncoding(StringRef OperationEncodingString) {
       .Case("DW_OP_LLVM_arg", DW_OP_LLVM_arg)
       .Case("DW_OP_LLVM_extract_bits_sext", DW_OP_LLVM_extract_bits_sext)
       .Case("DW_OP_LLVM_extract_bits_zext", DW_OP_LLVM_extract_bits_zext)
+      .Case("DW_OP_LLVM_label", DW_OP_LLVM_label)
+      .Case("DW_OP_LLVM_bra", DW_OP_LLVM_bra)
+      .Case("DW_OP_LLVM_skip", DW_OP_LLVM_skip)
       .Default(0);
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfbb35d516534..45f1f13e999fc 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -191,8 +191,14 @@ void DebugLocDwarfExpression::emitUnsigned(uint64_t Value) {
   getActiveStreamer().emitULEB128(Value, Twine(Value));
 }
 
-void DebugLocDwarfExpression::emitData1(uint8_t Value) {
-  getActiveStreamer().emitInt8(Value, Twine(Value));
+void DebugLocDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    getActiveStreamer().emitInt8(Value >> (Byte * 8), Twine(Value));
+  }
 }
 
 void DebugLocDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -232,6 +238,23 @@ void DebugLocDwarfExpression::commitTemporaryBuffer() {
   TmpBuf->Comments.clear();
 }
 
+void DebugLocDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                         uint64_t Value,
+                                                         unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  assert(TmpBuf && Offset < TmpBuf->Bytes.size() &&
+         Size <= TmpBuf->Bytes.size() - Offset &&
+         "invalid temporary buffer offset");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    TmpBuf->Bytes[Offset + I] = Value >> (Byte * 8);
+    if (Offset + I < TmpBuf->Comments.size())
+      TmpBuf->Comments[Offset + I].clear();
+  }
+}
+
 const DIType *DbgVariable::getType() const {
   return getVariable()->getType();
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 9b676f88a0cff..4054817abb95d 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -13,13 +13,16 @@
 #include "DwarfExpression.h"
 #include "DwarfCompileUnit.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallBitVector.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/CodeGen/Register.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
 #include <algorithm>
 
 using namespace llvm;
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+      });
+
+  SmallVector<LabelOffset, 4> Labels;
+  SmallVector<BranchFixup, 4> Fixups;
+  // Buffer the expression until every label has a byte offset, then patch the
+  // branches.
+  if (HasSymbolicBranches)
+    enableTemporaryBuffer();
 
+  std::optional<DIExpression::ExprOperand> PrevConvertOp;
   while (ExprCursor) {
     auto Op = ExprCursor.take();
     uint64_t OpNum = Op->getOp();
@@ -545,9 +571,27 @@ bool DwarfExpression::addExpression(
     }
 
     switch (OpNum) {
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      if (PrevConvertOp)
+        report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
+                           dwarf::OperationEncodingString(OpNum) +
+                           " without DW_OP_convert support");
+      if (OpNum == dwarf::DW_OP_LLVM_label) {
+        Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
+        break;
+      }
+      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
+                                            : dwarf::DW_OP_skip);
+      Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
+      emitData2(0);
+      break;
     case dwarf::DW_OP_LLVM_arg:
       if (!InsertArg(Op->getArg(0), ExprCursor)) {
         LocationKind = Unknown;
+        if (HasSymbolicBranches)
+          disableTemporaryBuffer();
         return false;
       }
       break;
@@ -579,7 +623,10 @@ bool DwarfExpression::addExpression(
       setSubRegisterPiece(0, 0);
       // Reset the location description kind.
       LocationKind = Unknown;
-      return true;
+      if (!HasSymbolicBranches)
+        return true;
+      // Keep going so we apply the branch fixups before returning.
+      break;
     }
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext: {
@@ -744,9 +791,12 @@ bool DwarfExpression::addExpression(
       break;
     case dwarf::DW_OP_LLVM_implicit_pointer:
       // Handled in DwarfCompileUnit::emitImplicitPointerLocation for
-      // Loc::Single variables. If we reach here, the variable has a
-      // location list or other unsupported path. Drop the
-      // location rather than crashing.
+      // Loc::Single variables. If we reach here, the variable has a location
+      // list or another unsupported path, so stop emitting the expression.
+      // We buffer expressions with symbolic branches, so disable the buffer
+      // before returning.
+      if (HasSymbolicBranches)
+        disableTemporaryBuffer();
       return false;
     default:
       llvm_unreachable("unhandled opcode found in expression");
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
+          static_cast<int64_t>(Label->Offset) -
+          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+      if (!isInt<16>(Displacement))
+        report_fatal_error(Twine("DWARF expression branch offset ") +
+                           Twine(Displacement) + " is outside [-32768, 32767]");
+
+      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                  static_cast<uint16_t>(Displacement));
+    }
+
+    disableTemporaryBuffer();
+    commitTemporaryBuffer();
+  }
   return true;
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
index 2525938c78d86..f5c37140a1ef9 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
@@ -130,7 +130,11 @@ class DwarfExpression {
   /// Emit a raw unsigned value.
   virtual void emitUnsigned(uint64_t Value) = 0;
 
-  virtual void emitData1(uint8_t Value) = 0;
+  /// Emit the low 1, 2, 4, or 8 bytes of a value in the target byte order.
+  virtual void emitData(uint64_t Value, unsigned Size) = 0;
+
+  void emitData1(uint8_t Value) { emitData(Value, 1); }
+  void emitData2(uint16_t Value) { emitData(Value, 2); }
 
   virtual void emitBaseTypeRef(uint64_t Idx) = 0;
 
@@ -150,6 +154,15 @@ class DwarfExpression {
   /// Commit the data stored in the temporary buffer to the main output.
   virtual void commitTemporaryBuffer() = 0;
 
+  /// Replace a 1-, 2-, 4-, or 8-byte zero placeholder at Offset with the low
+  /// bytes of Value in the target byte order.
+  virtual void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                          unsigned Size) = 0;
+
+  void replaceTemporaryBufferData2(unsigned Offset, uint16_t Value) {
+    replaceTemporaryBufferData(Offset, Value, 2);
+  }
+
   /// Emit a normalized unsigned constant.
   void emitConstu(uint64_t Value);
 
@@ -285,10 +298,14 @@ class DwarfExpression {
 
   /// Emit all remaining operations in the DIExpressionCursor. The
   /// cursor must not contain any DW_OP_LLVM_arg operations.
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   void addExpression(DIExpressionCursor &&Expr);
 
   /// Emit all remaining operations in the DIExpressionCursor.
   /// DW_OP_LLVM_arg operations are resolved by calling (\p InsertArg).
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   //
   /// \return false if any call to (\p InsertArg) returns false.
   bool addExpression(
@@ -328,13 +345,15 @@ class DebugLocDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
@@ -358,13 +377,15 @@ class DIEDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index 78c0769e49161..c1c1218a87974 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+}
+
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
                                        DwarfCompileUnit &CU, DIELoc &DIE)
     : DwarfExpression(AP.getDwarfVersion(), CU), AP(AP), OutDIE(DIE) {}
@@ -53,8 +74,8 @@ void DIEDwarfExpression::emitUnsigned(uint64_t Value) {
   CU.addUInt(getActiveDIE(), dwarf::DW_FORM_udata, Value);
 }
 
-void DIEDwarfExpression::emitData1(uint8_t Value) {
-  CU.addUInt(getActiveDIE(), dwarf::DW_FORM_data1, Value);
+void DIEDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  CU.addUInt(getActiveDIE(), getDataForm(Size), getDataValue(Value, Size));
 }
 
 void DIEDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
+}
+
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
                                          llvm::Register MachineReg) {
   return MachineReg == TRI.getFrameRegister(*AP.MF);
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 88f7f2f6240e0..518e4c130e34c 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -1750,6 +1751,9 @@ unsigned DIExpression::ExprOperand::getSize() const {
   case dwarf::DW_OP_LLVM_tag_offset:
   case dwarf::DW_OP_LLVM_entry_value:
   case dwarf::DW_OP_LLVM_arg:
+  case dwarf::DW_OP_LLVM_label:
+  case dwarf::DW_OP_LLVM_bra:
+  case dwarf::DW_OP_LLVM_skip:
   case dwarf::DW_OP_regx:
     return 2;
   default:
@@ -1758,12 +1762,81 @@ unsigned DIExpression::ExprOperand::getSize() const {
 }
 
 bool DIExpression::isValid() const {
+  auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
+    auto FirstOp = expr_op_begin();
+    if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
+      ++FirstOp;
+    return EntryValue.get() == FirstOp->get() && EntryValue.getArg(0) == 1;
+  };
+
+  SmallDenseSet<uint64_t, 4> Labels;
+  SmallVector<uint64_t, 4> LabelReferences;
+  bool HasControlFlow = false;
+  bool HasControlFlowConflict = false;
+  bool HasStackValue = false;
+  bool HasFragment = false;
+  bool HasInvalidControlFlowSuffix = false;
+
+  // Collect labels and branch targets before running the checks below, which
+  // may return early.
   for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
     // Check that there's space for the operand.
     if (I->get() + I->getSize() > E->get())
       return false;
 
     uint64_t Op = I->getOp();
+
+    // Only DW_OP_LLVM_fragment may follow DW_OP_stack_value, and nothing may
+    // follow the fragment.
+    HasInvalidControlFlowSuffix |=
+        HasFragment || (HasStackValue && Op != dwarf::DW_OP_LLVM_fragment);
+
+    switch (Op) {
+    case dwarf::DW_OP_LLVM_label:
+      if (!Labels.insert(I->getArg(0)).second)
+        return false;
+      HasControlFlow = true;
+      break;
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      LabelReferences.push_back(I->getArg(0));
+      HasControlFlow = true;
+      break;
+    // We don't know the DW_OP_bra and DW_OP_skip offsets until CodeGen, so
+    // DIExpression uses the symbolic ops.
+    case dwarf::DW_OP_bra:
+    case dwarf::DW_OP_skip:
+      return false;
+    case dwarf::DW_OP_LLVM_tag_offset:
+      if (HasControlFlow)
+        return false;
+      break;
+    // These ops use lowering paths that don't support symbolic branches.
+    case dwarf::DW_OP_LLVM_arg:
+    case dwarf::DW_OP_LLVM_implicit_pointer:
+      HasControlFlowConflict = true;
+      break;
+    case dwarf::DW_OP_LLVM_entry_value:
+      HasControlFlowConflict |= !IsEntryValueValid(*I);
+      break;
+    default:
+      break;
+    }
+
+    if (Op == dwarf::DW_OP_stack_value)
+      HasStackValue = true;
+    else if (Op == dwarf::DW_OP_LLVM_fragment)
+      HasFragment = true;
+  }
+
+  if (HasControlFlow && (HasControlFlowConflict || HasInvalidControlFlowSuffix))
+    return false;
+  for (uint64_t Label : LabelReferences)
+    if (!Labels.contains(Label))
+      return false;
+
+  for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
+    uint64_t Op = I->getOp();
     if ((Op >= dwarf::DW_OP_reg0 && Op <= dwarf::DW_OP_reg31) ||
         (Op >= dwarf::DW_OP_breg0 && Op <= dwarf::DW_OP_breg31))
       return true;
@@ -1784,7 +1857,7 @@ bool DIExpression::isValid() const {
         return false;
       break;
     }
-    case dwarf::DW_OP_swap: {
+    case dwarf::DW_OP_swap:
       // Must be more than one implicit element on the stack.
 
       // FIXME: A better way to implement this would be to add a local variable
@@ -1795,22 +1868,20 @@ bool DIExpression::isValid() const {
       if (getNumElements() == 1)
         return false;
       break;
-    }
-    case dwarf::DW_OP_LLVM_entry_value: {
+    case dwarf::DW_OP_LLVM_entry_value:
       // An entry value operator must appear at the beginning or immediately
       // following `DW_OP_LLVM_arg 0`, and the number of operations it cover can
       // currently only be 1, because we support only entry values of a simple
       // register location. One reason for this is that we currently can't
       // calculate the size of the resulting DWARF block for other expressions.
-      auto FirstOp = expr_op_begin();
-      if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
-        ++FirstOp;
-      return I->get() == FirstOp->get() && I->getArg(0) == 1;
-    }
+      return IsEntryValueValid(*I);
     case dwarf::DW_OP_LLVM_implicit_pointer:
     case dwarf::DW_OP_LLVM_convert:
     case dwarf::DW_OP_LLVM_arg:
     case dwarf::DW_OP_LLVM_tag_offset:
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext:
     case dwarf::DW_OP_constu:
@@ -2327,13 +2398,15 @@ DIExpression *DIExpression::appendToStack(const DIExpression *Expr,
   // has no DW_OP_stack_value.
   //
   // Match .* DW_OP_stack_value (DW_OP_LLVM_fragment A B)?.
-  std::optional<FragmentInfo> FI = Expr->getFragmentInfo();
-  unsigned DropUntilStackValue = FI ? 3 : 0;
-  ArrayRef<uint64_t> ExprOpsBeforeFragment =
-      Expr->getElements().drop_back(DropUntilStackValue);
-  bool NeedsDeref = (Expr->getNumElements() > DropUntilStackValue) &&
-                    (ExprOpsBeforeFragment.back() != dwarf::DW_OP_stack_value);
-  bool NeedsStackValue = NeedsDeref || ExprOpsBeforeFragment.empty();
+  std::optional<uint64_t> LastOp;
+  // A raw operand can have the same value as an opcode, so use expr_ops().
+  // Labels and branches can't follow DW_OP_stack_value, so LastOp is still
+  // DW_OP_stack_value when the expression has a value suffix.
+  for (auto Op : Expr->expr_ops())
+    if (Op.getOp() != dwarf::DW_OP_LLVM_fragment)
+      LastOp = Op.getOp();
+  bool NeedsDeref = LastOp && *LastOp != dwarf::DW_OP_stack_value;
+  bool NeedsStackValue = NeedsDeref || !LastOp;
 
   // Append a DW_OP_deref after Expr's current op list if needed, then append
   // the new ops, and finally ensure that a single DW_OP_stack_value is present.
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..be14be6a43656 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7182,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        });
+    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
+            &DVR, MD, DVR.getExpression(), BB, F);
+  }
+
   if (DVR.isDbgAssign()) {
     CheckDI(isa_and_nonnull<DIAssignID>(DVR.getRawAssignID()),
             "invalid #dbg_assign DIAssignID", &DVR, DVR.getRawAssignID(), BB,
diff --git a/llvm/test/Assembler/di-expression-symbolic-branches.ll b/llvm/test/Assembler/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..475594c62d444
--- /dev/null
+++ b/llvm/test/Assembler/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Labels use IDs in a DIExpression, so make sure the assembler keeps both
+; forward and backward references.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42)
+!0 = !DIExpression(DW_OP_LLVM_label, 0,
+                   DW_OP_LLVM_bra, 42,
+                   DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 42)
diff --git a/llvm/test/Bitcode/di-expression-symbolic-branches.ll b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..82b3c630360fe
--- /dev/null
+++ b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | llvm-as | llvm-dis | FileCheck %s
+
+; Label IDs are uint64_t values, so use the largest one while checking that the
+; opcodes and IDs survive two bitcode round-trips.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 18446744073709551615, DW_OP_LLVM_bra, 7, DW_OP_LLVM_skip, 18446744073709551615, DW_OP_LLVM_label, 7)
+!0 = !DIExpression(DW_OP_LLVM_label, 18446744073709551615,
+                   DW_OP_LLVM_bra, 7,
+                   DW_OP_LLVM_skip, 18446744073709551615,
+                   DW_OP_LLVM_label, 7)
diff --git a/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
new file mode 100644
index 0000000000000..eda9f210cd378
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
@@ -0,0 +1,102 @@
+# RUN: split-file %s %t
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=asm -o - %t/valid.mir | FileCheck %s
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=ENTRY
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=NATIVE
+# RUN: not --crash llc -disable-verify -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t/missing-label.mir 2>&1 | FileCheck %s --check-prefix=MISSING
+
+# This checks the machine debug-expression path:
+#
+# - DWARF 4 expands conversions before we patch a location-list branch, while
+#   DWARF 5 emits DW_OP_convert.
+# - Entry values keep their branches when we emit an inline expression.
+# - With verification disabled, CodeGen still reports a missing label.
+
+#--- valid.mir
+--- |
+  define void @f() !dbg !5 {
+  entry:
+    ret void, !dbg !12
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9, !10}
+  !9 = !DILocalVariable(name: "before", scope: !5, type: !11)
+  !10 = !DILocalVariable(name: "entry", arg: 1, scope: !5, type: !11)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+  !12 = !DILocation(line: 1, scope: !5)
+...
+---
+name: f
+body: |
+  bb.0.entry:
+    $rcx = MOV64ri 1
+    DBG_VALUE $rdi, $noreg, !10, !DIExpression(DW_OP_LLVM_entry_value, 1, DW_OP_lit0, DW_OP_plus, DW_OP_LLVM_bra, 3, DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 3), debug-location !12
+    DBG_VALUE $rcx, 0, !9, !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_LLVM_convert, 64, DW_ATE_signed, DW_OP_plus_uconst, 1, DW_OP_LLVM_skip, 1), debug-location !12
+    $rcx = MOV64ri 2
+    RET64 debug-location !12
+...
+
+# CHECK: .Ldebug_loc0:
+# CHECK: .byte 114{{.*}}# DW_OP_breg2
+# CHECK-NEXT: .byte 0
+# CHECK-NEXT: .byte 18{{.*}}# DW_OP_dup
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 31
+# CHECK-NEXT: .byte 37{{.*}}# DW_OP_shr
+# CHECK-NEXT: .byte 48{{.*}}# DW_OP_lit0
+# CHECK-NEXT: .byte 32{{.*}}# DW_OP_not
+# CHECK-NEXT: .byte 30{{.*}}# DW_OP_mul
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 32
+# CHECK-NEXT: .byte 36{{.*}}# DW_OP_shl
+# CHECK-NEXT: .byte 33{{.*}}# DW_OP_or
+# CHECK-NEXT: .byte 35{{.*}}# DW_OP_plus_uconst
+# CHECK-NEXT: .byte 1
+# CHECK-NEXT: .byte 47{{.*}}# DW_OP_skip
+# CHECK-NEXT: .byte 240
+# CHECK-NEXT: .byte 255
+
+# ENTRY: DW_AT_location (DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_lit0, DW_OP_plus, DW_OP_bra +2, DW_OP_plus_uconst 0x1)
+
+# NATIVE: DW_OP_breg2 RCX+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_plus_uconst 0x1, DW_OP_skip -15
+
+# MISSING: LLVM ERROR: DWARF expression branch to label 2 has no matching label
+
+#--- missing-label.mir
+--- |
+  define void @missing_label() !dbg !5 {
+  entry:
+    ret void, !dbg !10
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "missing_label", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+  !10 = !DILocation(line: 1, scope: !5)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+...
+---
+name: missing_label
+body: |
+  bb.0.entry:
+    DBG_VALUE 0, $noreg, !9, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_bra, 2), debug-location !10
+    RET64 debug-location !10
+...
diff --git a/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..44159954127be
--- /dev/null
+++ b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
@@ -0,0 +1,44 @@
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -filetype=asm -o - %s | FileCheck %s
+
+; We patch branches in two temporary buffers, so make sure both use the PowerPC
+; byte order: a forward skip in a location list and a backward branch in an
+; inline expression.
+
+define void @f(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_bra, 2), !11)
+  #dbg_value(i64 %x, !10,
+             !DIExpression(DW_OP_LLVM_skip, 3, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_label, 3), !11)
+  call void @clobber(), !dbg !11
+  ret void, !dbg !11
+}
+
+declare void @clobber()
+
+; CHECK: .section .debug_loclists
+; CHECK: .byte 47{{.*}}# DW_OP_skip
+; CHECK-NEXT: .byte 0
+; CHECK-NEXT: .byte 2
+; CHECK: .byte 5{{.*}}# DW_AT_location
+; CHECK-NEXT: .byte 48
+; CHECK-NEXT: .byte 40
+; CHECK-NEXT: .short 65533
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "backward", scope: !5, type: !12)
+!10 = !DILocalVariable(name: "value", arg: 1, scope: !5, type: !12)
+!11 = !DILocation(line: 1, column: 1, scope: !5)
+!12 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
new file mode 100644
index 0000000000000..fc7e0246a6ae2
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
@@ -0,0 +1,33 @@
+import sys
+
+# Put 32768 one-byte DW_OP_dup operations between the branch and its label,
+# which gives us the first offset that won't fit.
+DUPLICATE_OPS = "DW_OP_dup, " * 32768
+
+sys.stdout.write(
+    f"""\
+define void @f() !dbg !5 {{
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, {DUPLICATE_OPS}
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}}
+
+!llvm.dbg.cu = !{{!0}}
+!llvm.module.flags = !{{!3}}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{{i32 2, !"Debug Info Version", i32 3}}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{{null}}
+!9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+!10 = !DILocation(line: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+"""
+)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
new file mode 100644
index 0000000000000..5cb9861502585
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
@@ -0,0 +1,96 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/skip.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=SKIP-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/skip.ll 2>&1 | FileCheck %s --check-prefix=SKIP-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/bra.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=BRA-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/bra.ll 2>&1 | FileCheck %s --check-prefix=BRA-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/label.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=LABEL-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/label.ll 2>&1 | FileCheck %s --check-prefix=LABEL-LEGACY
+
+; DWARF 5 emits DW_OP_convert directly, so labels and branches can appear
+; between conversions. With DWARF 4 we may defer one conversion until the next;
+; if a label, branch, or skip splits the pair, CodeGen reports an error.
+
+; SKIP-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_skip +5, DW_OP_convert {{.*}} "DW_ATE_signed_64")
+; SKIP-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_skip without DW_OP_convert support
+
+; BRA-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_dup, DW_OP_bra +5, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; BRA-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_bra without DW_OP_convert support
+
+; LABEL-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; LABEL-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_label without DW_OP_convert support
+
+;--- skip.ll
+define void @skip(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_skip, 1,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "skip", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "skip", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- label.ll
+define void @label(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 3), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "label", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "label", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- bra.ll
+define void @bra(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_dup,
+                           DW_OP_LLVM_bra, 2,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 2), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "bra", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "bra", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
new file mode 100644
index 0000000000000..677032dc9c7a7
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
@@ -0,0 +1,7 @@
+; RUN: %python %S/Inputs/generate-di-expression-symbolic-branch-out-of-range.py > %t.ll
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t.ll 2>&1 | FileCheck %s
+
+; The first positive offset that doesn't fit is 32768, so make sure CodeGen
+; reports it instead of truncating it.
+
+; CHECK: LLVM ERROR: DWARF expression branch offset 32768 is outside [-32768, 32767]
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..163c3c24c6927
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
@@ -0,0 +1,84 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump -v - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s --check-prefix=LEGACY
+
+; Branch offsets come from the emitted bytes, so check:
+;
+; - zero, forward, backward, and cyclic branches;
+; - ops which grow during lowering;
+; - labels on either side of an emitted op; and
+; - the fragment return path and DWARF 4 convert expansion.
+
+define void @f() !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, DW_OP_LLVM_label, 1), !18)
+  #dbg_value(i64 0, !10,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_skip, 2), !18)
+  #dbg_value(i64 0, !11,
+             !DIExpression(DW_OP_LLVM_label, 3, DW_OP_LLVM_bra, 4,
+                           DW_OP_LLVM_skip, 3, DW_OP_LLVM_label, 4), !18)
+  #dbg_value(i64 0, !12,
+             !DIExpression(DW_OP_LLVM_bra, 5, DW_OP_deref_size, 1,
+                           DW_OP_plus_uconst, 128,
+                           DW_OP_LLVM_extract_bits_sext, 4, 4,
+                           DW_OP_LLVM_label, 5), !18)
+  #dbg_value(i64 0, !13,
+             !DIExpression(DW_OP_LLVM_skip, 6,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 6), !18)
+  #dbg_value(i64 0, !14,
+             !DIExpression(DW_OP_LLVM_bra, 7, DW_OP_LLVM_label, 7,
+                           DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 32), !18)
+  #dbg_value(i64 0, !15,
+             !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 8,
+                           DW_OP_LLVM_skip, 8), !18)
+  #dbg_value(i64 0, !16,
+             !DIExpression(DW_OP_LLVM_label, 9, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_skip, 9), !18)
+  ret void, !dbg !18
+}
+
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"zero"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"backward"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +3, DW_OP_skip -6, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"cycle"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +11, DW_OP_deref_size 0x1, DW_OP_plus_uconst 0x80, DW_OP_constu 0x38, DW_OP_shl, DW_OP_constu 0x3c, DW_OP_shra, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"expanded"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip +10, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"convert"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value, DW_OP_piece 0x4)
+; CHECK: DW_AT_name{{.*}}"fragment"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_after_offset"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -5, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_before_offset"
+
+; LEGACY: DW_AT_location (DW_OP_lit0, DW_OP_skip +11, DW_OP_dup, DW_OP_constu 0x1f, DW_OP_shr, DW_OP_lit0, DW_OP_not, DW_OP_mul, DW_OP_constu 0x20, DW_OP_shl, DW_OP_or, DW_OP_stack_value)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !10, !11, !12, !13, !14, !15, !16}
+!9 = !DILocalVariable(name: "zero", scope: !5, type: !19)
+!10 = !DILocalVariable(name: "backward", scope: !5, type: !19)
+!11 = !DILocalVariable(name: "cycle", scope: !5, type: !19)
+!12 = !DILocalVariable(name: "expanded", scope: !5, type: !19)
+!13 = !DILocalVariable(name: "convert", scope: !5, type: !19)
+!14 = !DILocalVariable(name: "fragment", scope: !5, type: !19)
+!15 = !DILocalVariable(name: "label_after_offset", scope: !5, type: !19)
+!16 = !DILocalVariable(name: "label_before_offset", scope: !5, type: !19)
+!18 = !DILocation(line: 1, scope: !5)
+!19 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/Verifier/di-expression-symbolic-branches.ll b/llvm/test/Verifier/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..db0e0dacdf32f
--- /dev/null
+++ b/llvm/test/Verifier/di-expression-symbolic-branches.ll
@@ -0,0 +1,93 @@
+; RUN: split-file %s %t
+; RUN: opt -passes=verify -disable-output %t/valid.ll
+; RUN: not opt -passes=verify -disable-output %t/arity.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/duplicate.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/missing.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw-after-register.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/incompatible.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/location-arg.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/tag-ordering.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/terminal.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+
+; Check the verifier rules separately:
+;
+; - a label needs an ID;
+; - labels are unique and each branch target exists;
+; - raw branches and incompatible ops are rejected; and
+; - tag_offset stays before control flow, which stays before stack_value.
+
+; DIArgList doesn't support symbolic branches, but normal IR loading drops the
+; bad debug info, so opt still succeeds.
+; RUN: opt -passes=verify -disable-output %t/arg-list.ll 2>&1 | FileCheck %s --check-prefix=ARG-LIST
+
+; INVALID: invalid expression
+; ARG-LIST: DIArgList doesn't support symbolic branches
+; ARG-LIST: warning: ignoring invalid debug info
+
+;--- valid.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 0, DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 0)
+
+;--- arity.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label)
+
+;--- duplicate.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_label, 1)
+
+;--- missing.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 1)
+
+;--- raw.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_bra, 0)
+
+;--- raw-after-register.ll
+; A register normally ends validation, but it must not hide a raw branch later
+; in the expression.
+!named = !{!0}
+!0 = !DIExpression(DW_OP_reg0, DW_OP_skip, 0)
+
+;--- incompatible.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_implicit_pointer, DW_OP_LLVM_label, 1)
+
+;--- location-arg.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_skip, 1,
+                   DW_OP_LLVM_label, 1)
+
+;--- tag-ordering.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_tag_offset, 0)
+
+;--- terminal.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_stack_value, DW_OP_LLVM_label, 1)
+
+;--- arg-list.ll
+; The expression is valid by itself, so this diagnostic comes from DIArgList.
+define void @f(i32 %x) !dbg !4 {
+entry:
+  #dbg_value(!DIArgList(i32 %x), !7,
+             !DIExpression(DW_OP_LLVM_skip, 1, DW_OP_LLVM_label, 1), !8)
+  ret void, !dbg !8
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1, type: !5,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!5 = !DISubroutineType(types: !6)
+!6 = !{null}
+!7 = !DILocalVariable(name: "x", scope: !4)
+!8 = !DILocation(line: 1, column: 1, scope: !4)
diff --git a/llvm/unittests/BinaryFormat/DwarfTest.cpp b/llvm/unittests/BinaryFormat/DwarfTest.cpp
index ba7d59182ea53..6ffb5a921510c 100644
--- a/llvm/unittests/BinaryFormat/DwarfTest.cpp
+++ b/llvm/unittests/BinaryFormat/DwarfTest.cpp
@@ -47,6 +47,13 @@ TEST(DwarfTest, getOperationEncoding) {
   // Some valid ops.
   EXPECT_EQ(DW_OP_deref, getOperationEncoding("DW_OP_deref"));
   EXPECT_EQ(DW_OP_bit_piece, getOperationEncoding("DW_OP_bit_piece"));
+  // These are metadata-only ops, but they still need names in both directions.
+  EXPECT_EQ(DW_OP_LLVM_label, getOperationEncoding("DW_OP_LLVM_label"));
+  EXPECT_EQ(DW_OP_LLVM_bra, getOperationEncoding("DW_OP_LLVM_bra"));
+  EXPECT_EQ(DW_OP_LLVM_skip, getOperationEncoding("DW_OP_LLVM_skip"));
+  EXPECT_EQ("DW_OP_LLVM_label", OperationEncodingString(DW_OP_LLVM_label));
+  EXPECT_EQ("DW_OP_LLVM_bra", OperationEncodingString(DW_OP_LLVM_bra));
+  EXPECT_EQ("DW_OP_LLVM_skip", OperationEncodingString(DW_OP_LLVM_skip));
 
   // Invalid ops.
   EXPECT_EQ(0u, getOperationEncoding("DW_OP_otherthings"));
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 65398b5ca2b1f..9be5384f9647b 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -4251,6 +4251,86 @@ TEST_F(DIExpressionTest, Append) {
   EXPECT_EQ(ResExpr, AppendExpr);
 }
 
+TEST_F(DIExpressionTest, SymbolicBranchRewrites) {
+  // A symbolic op has an ID operand, so fold the math on both sides without
+  // consuming it.
+  for (uint64_t Barrier : {dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                           dwarf::DW_OP_LLVM_skip}) {
+    SmallVector<uint64_t> Ops = {dwarf::DW_OP_constu,
+                                 1,
+                                 dwarf::DW_OP_constu,
+                                 2,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 4,
+                                 Barrier,
+                                 7,
+                                 dwarf::DW_OP_constu,
+                                 5,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 6,
+                                 dwarf::DW_OP_constu,
+                                 7,
+                                 dwarf::DW_OP_plus};
+    SmallVector<uint64_t> Expected = {
+        dwarf::DW_OP_constu,      3, dwarf::DW_OP_constu, 4, Barrier, 7,
+        dwarf::DW_OP_plus_uconst, 5, dwarf::DW_OP_constu, 13};
+    if (Barrier != dwarf::DW_OP_LLVM_label) {
+      Ops.append({dwarf::DW_OP_LLVM_label, 7});
+      Expected.append({dwarf::DW_OP_LLVM_label, 7});
+    }
+    EXPECT_EQ(DIExpression::get(Context, Expected),
+              DIExpression::get(Context, Ops)->foldConstantMath());
+  }
+
+  // Adding ops before or after an expression leaves its labels in place.
+  SmallVector<uint64_t> Ops = {dwarf::DW_OP_LLVM_label, 7,
+                               dwarf::DW_OP_plus_uconst, 1};
+  auto *Expr = DIExpression::get(Context, Ops);
+  auto *Prepended = DIExpression::prepend(Expr, DIExpression::DerefBefore, 0);
+  SmallVector<uint64_t> Expected = {dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_label,
+                                    7, dwarf::DW_OP_plus_uconst, 1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+
+  SmallVector<uint64_t> Prefix = {dwarf::DW_OP_LLVM_tag_offset, 3};
+  Prepended = DIExpression::prependOpcodes(Expr, Prefix);
+  Expected = {dwarf::DW_OP_LLVM_tag_offset, 3, dwarf::DW_OP_LLVM_label, 7,
+              dwarf::DW_OP_plus_uconst,     1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+  EXPECT_TRUE(Prepended->isValid());
+
+  SmallVector<uint64_t> AppendOps = {
+      dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+      dwarf::DW_OP_LLVM_convert, 64, dwarf::DW_ATE_signed};
+  auto *Appended = DIExpression::append(Expr, AppendOps);
+  Expected = {dwarf::DW_OP_LLVM_label,
+              7,
+              dwarf::DW_OP_plus_uconst,
+              1,
+              dwarf::DW_OP_LLVM_convert,
+              32,
+              dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert,
+              64,
+              dwarf::DW_ATE_signed};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+
+  // Label IDs are arbitrary uint64_t values, so appendExt must not mistake one
+  // for DW_OP_stack_value.
+  uint64_t LabelID = dwarf::DW_OP_stack_value;
+  Ops = {dwarf::DW_OP_LLVM_label, LabelID};
+  Appended =
+      DIExpression::appendExt(DIExpression::get(Context, Ops), 32, 64, true);
+  Expected = {dwarf::DW_OP_LLVM_label,   LabelID, dwarf::DW_OP_deref,
+              dwarf::DW_OP_LLVM_convert, 32,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert, 64,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_stack_value};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+}
+
 TEST_F(DIExpressionTest, isValid) {
 #define EXPECT_VALID(...)                                                      \
   do {                                                                         \
@@ -4271,6 +4351,7 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_VALID(dwarf::DW_OP_constu, 6, dwarf::DW_OP_plus);
   EXPECT_VALID(dwarf::DW_OP_constu, 5, dwarf::DW_OP_swap);
   EXPECT_VALID(dwarf::DW_OP_deref);
+  EXPECT_VALID(dwarf::DW_OP_stack_value);
   EXPECT_VALID(dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_plus_uconst, 6, dwarf::DW_OP_deref);
   EXPECT_VALID(dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst, 6);
@@ -4279,12 +4360,31 @@ TEST_F(DIExpressionTest, isValid) {
                dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_VALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_entry_value, 1);
+  // A label may be unused, and branches may refer forward or backward.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 2, dwarf::DW_OP_LLVM_label, 2);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 3, dwarf::DW_OP_LLVM_label, 4);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 4, dwarf::DW_OP_LLVM_skip, 5,
+               dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_LLVM_skip, 4);
+  // Label IDs can use the full uint64_t range.
+  EXPECT_VALID(
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max() - 1,
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max(),
+      dwarf::DW_OP_LLVM_skip, std::numeric_limits<uint64_t>::max() - 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_stack_value);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 6, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_tag_offset, 1, dwarf::DW_OP_LLVM_label, 6,
+               dwarf::DW_OP_LLVM_skip, 6, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
 
   // Invalid constructions.
   EXPECT_INVALID(~0u);
   EXPECT_INVALID(dwarf::DW_OP_plus, 0);
   EXPECT_INVALID(dwarf::DW_OP_plus_uconst);
   EXPECT_INVALID(dwarf::DW_OP_swap);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_deref);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3, 7, dwarf::DW_OP_plus_uconst, 3);
@@ -4294,6 +4394,48 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_plus_uconst, 5,
                  dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_LLVM_entry_value, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip, 1);
+  EXPECT_INVALID(dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_implicit_pointer, dwarf::DW_OP_LLVM_label,
+                 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_tag_offset, 0);
+  // Converts and other stack ops can appear around labels and branches.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_skip, 7, dwarf::DW_OP_LLVM_label, 7);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 1, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_convert, 64,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_dup, dwarf::DW_OP_LLVM_bra, 1,
+               dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 0, 32, dwarf::DW_OP_LLVM_label, 1);
+  // reg0 and entry_value normally end validation early, but they must not hide
+  // invalid control flow later in the expression.
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_LLVM_bra, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_label, 1,
+                 dwarf::DW_OP_LLVM_entry_value, 1);
 
 #undef EXPECT_VALID
 #undef EXPECT_INVALID
@@ -4371,6 +4513,19 @@ TEST_F(DIExpressionTest, createFragmentExpression) {
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
+  // Creating a fragment leaves the branches alone and adds the fragment at the
+  // end.
+  SmallVector<uint64_t> ControlFlowOps = {dwarf::DW_OP_LLVM_label, 1,
+                                          dwarf::DW_OP_LLVM_bra,   1,
+                                          dwarf::DW_OP_LLVM_skip,  1};
+  DIExpression *ControlFlowExpr = DIExpression::get(Context, ControlFlowOps);
+  auto Fragment =
+      DIExpression::createFragmentExpression(ControlFlowExpr, 0, 32);
+  ASSERT_TRUE(Fragment.has_value());
+  ControlFlowOps.append({dwarf::DW_OP_LLVM_fragment, 0, 32});
+  EXPECT_EQ(DIExpression::get(Context, ControlFlowOps), *Fragment);
+  EXPECT_TRUE((*Fragment)->isValid());
+
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
new file mode 100644
index 0000000000000..7ba2b0802fa3c
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
@@ -0,0 +1,23 @@
+; RUN: mlir-translate -import-llvm -mlir-print-debuginfo -emit-expensive-warnings %s 2>&1 | FileCheck %s
+
+; Import a forward branch and a backward skip, and make sure MLIR keeps the op
+; order and label IDs.
+
+; CHECK: llvm.intr.dbg.value {{.*}} #llvm.di_expression<[DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0), DW_OP_LLVM_label(42)]> = {{.*}} : i64
+define void @f(i64 %x) !dbg !4 {
+  #dbg_value(i64 %x, !DILocalVariable(scope: !4),
+             !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42,
+                           DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42),
+             !DILocation(scope: !4))
+  ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1,
+                            type: !DISubroutineType(types: !{null}),
+                            spFlags: DISPFlagDefinition, unit: !0)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
new file mode 100644
index 0000000000000..74a549b952327
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Export a forward branch and a backward skip, and make sure LLVM IR keeps the
+// op order and label IDs.
+
+#file = #llvm.di_file<"test.c" in "/">
+#cu = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_C,
+                            file = #file>
+#sp = #llvm.di_subprogram<compileUnit = #cu, scope = #file, name = "f",
+                          subprogramFlags = "Definition",
+                          type = #llvm.di_subroutine_type<types = #llvm.di_null_type>>
+
+// CHECK: #dbg_value(i64 %{{.*}}, !{{.*}}, !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42), !{{.*}})
+llvm.func @f(%arg: i64) {
+  llvm.intr.dbg.value #llvm.di_local_variable<scope = #sp> #llvm.di_expression<[
+    DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0),
+    DW_OP_LLVM_label(42)]> = %arg : i64
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])

>From 57cf90ad67b0e05c411e79518212f90e3faa39bc Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:53:45 -0700
Subject: [PATCH 2/7] [LLDB] Handle symbolic DIExpression opcodes

These opcodes only appear in LLVM metadata, but adding them to the shared DWARF enum makes LLDB's switch incomplete. List them with the other metadata-only opcodes so a -Wswitch build stays clean.
---
 lldb/source/Expression/DWARFExpression.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lldb/source/Expression/DWARFExpression.cpp b/lldb/source/Expression/DWARFExpression.cpp
index 299dbe97d6176..73bdd18d82cae 100644
--- a/lldb/source/Expression/DWARFExpression.cpp
+++ b/lldb/source/Expression/DWARFExpression.cpp
@@ -205,6 +205,9 @@ GetOpcodeDataSize(const DataExtractor &data, const lldb::offset_t data_offset,
   case DW_OP_LLVM_arg:
   case DW_OP_LLVM_extract_bits_sext:
   case DW_OP_LLVM_extract_bits_zext:
+  case DW_OP_LLVM_label:
+  case DW_OP_LLVM_bra:
+  case DW_OP_LLVM_skip:
     break;
   // Vendor extensions:
   case DW_OP_HP_is_value:

>From e1e1ba8492cb9a32cb924caadb78ff4129cc9581 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:58:04 -0700
Subject: [PATCH 3/7] [DebugInfo] Clarify symbolic branch limitations

Separate the DIExpression validity rules from the combinations we don't handle yet. DIArgList and DW_OP_LLVM_arg need more work in expressions and expression writers, while DW_OP_LLVM_implicit_pointer takes a different emission path. Also document the final branch offset range check.
---
 llvm/docs/SourceLevelDebugging.md | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 645e441e6fe2a..834781444c607 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -492,15 +492,23 @@ Label IDs are local to an expression:
   offset.
 - Branches can go forward, backward, to themselves, or form cycles.
 
-There are a few other restrictions:
+`DIExpression` validation also checks the following:
 
 - Put labels, branches, and skips before `DW_OP_stack_value` and
   `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
-- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
-  separately, so they can't be used with symbolic control flow.
 - Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
   applies to every path.
 
+There are also a couple of cases we don't handle yet:
+
+- `DIArgList` and `DW_OP_LLVM_arg` are currently rejected. CodeGen expands each
+  argument before it resolves the label offsets, so there isn't a representation
+  problem here; it mostly needs work handling it in expressions and expression
+  writers.
+- `DW_OP_LLVM_implicit_pointer` bypasses normal expression emission and only
+  handles a single location today. Supporting branches there is a bit more
+  work, since we'll need to work it back into our normal emission order.
+
 We don't check reachability, termination, or stack state where paths meet.
 
 `DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
@@ -508,6 +516,9 @@ and we don't match conversions on different paths. When CodeGen can't emit
 `DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
 branch, or skip would split the pair, CodeGen reports an error.
 
+CodeGen also reports an error if the final branch offset is outside
+`[-32768, 32767]`.
+
 Local expression rewrites stop at labels, branches, and skips; they can still
 add operations to either end, but they don't move, remove, or copy labels.
 

>From 7d3fcc07c09bcf1d8150f79394720988d8810aae Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 14:14:38 -0700
Subject: [PATCH 4/7] [DebugInfo] Clean up symbolic DIExpression branch
 handling

Pull the symbolic branch checks into two small helpers, since CodeGen only needs branches while the verifier also needs labels.

Use maskTrailingOnes when storing fixed-width DIE values, and make the placeholder walk use a byte cursor with names that describe what we're patching. We still keep the existing form and size so later fixup offsets don't move.

Also make the displacement calculation and DIArgList verifier check a little clearer. There should be no behavior change here.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h        | 10 +++++
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  5 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     | 42 ++++++++++---------
 llvm/lib/IR/Verifier.cpp                      | 16 +++----
 4 files changed, 44 insertions(+), 29 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index c3cff08eccb5c..177c50be7b374 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,6 +1108,16 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
+/// Return true if Op is a symbolic branch to a label.
+inline bool isSymbolicBranchOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
+}
+
+/// Return true if Op is a symbolic label or branch.
+inline bool isSymbolicControlFlowOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
+}
+
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 4054817abb95d..f90edb1e68157 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,8 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        return dwarf::isSymbolicBranchOp(Op.getOp());
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -816,6 +815,8 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("DWARF expression branch to label ") +
                            Twine(Fixup.LabelID) + " has no matching label");
 
+      // DW_OP_bra and DW_OP_skip apply the displacement after reading their
+      // two-byte operand, so use the byte after the placeholder as the base.
       int64_t Displacement =
           static_cast<int64_t>(Label->Offset) -
           static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index c1c1218a87974..ea2a6ddca3c61 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -27,6 +27,7 @@
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/Support/Casting.h"
+#include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include <cassert>
 #include <cstdint>
@@ -53,9 +54,7 @@ static dwarf::Form getDataForm(unsigned Size) {
 }
 
 static uint64_t getDataValue(uint64_t Value, unsigned Size) {
-  if (Size == sizeof(Value))
-    return Value;
-  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+  return Value & maskTrailingOnes<uint64_t>(Size * 8);
 }
 
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
@@ -98,26 +97,31 @@ unsigned DIEDwarfExpression::getTemporaryBufferSize() {
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
-void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
-                                                    uint64_t Value,
-                                                    unsigned Size) {
-  dwarf::Form Form = getDataForm(Size);
-  // Keep the form so replacing the value doesn't move later labels.
-  unsigned CurrentOffset = 0;
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned PlaceholderOffset,
+                                                    uint64_t Replacement,
+                                                    unsigned PlaceholderSize) {
+  // Walk the encoded values until the cursor reaches the placeholder.
+  unsigned ByteCursor = 0;
   for (DIEValue &V : TmpDIE.values()) {
     unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
-    if (Offset < CurrentOffset + ValueSize) {
-      assert(Offset == CurrentOffset && ValueSize == Size &&
-             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
-             V.getDIEInteger().getValue() == 0 &&
-             "symbolic branch fixup does not match its placeholder");
-      V = DIEValue(V.getAttribute(), V.getForm(),
-                   DIEInteger(getDataValue(Value, Size)));
-      return;
+    unsigned ValueEnd = ByteCursor + ValueSize;
+    if (ValueEnd <= PlaceholderOffset) {
+      ByteCursor = ValueEnd;
+      continue;
     }
-    CurrentOffset += ValueSize;
+
+    // Replace the whole zero placeholder and keep its form, since changing its
+    // encoded size would move every later fixup.
+    assert(PlaceholderOffset == ByteCursor && ValueSize == PlaceholderSize &&
+           V.getType() == DIEValue::isInteger &&
+           V.getForm() == getDataForm(PlaceholderSize) &&
+           V.getDIEInteger().getValue() == 0 &&
+           "symbolic branch fixup does not match its placeholder");
+    V = DIEValue(V.getAttribute(), V.getForm(),
+                 DIEInteger(getDataValue(Replacement, PlaceholderSize)));
+    return;
   }
-  llvm_unreachable("invalid temporary DIE offset");
+  llvm_unreachable("temporary DIE placeholder not found");
 }
 
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index be14be6a43656..0dc59163fa864 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7184,15 +7184,15 @@ void Verifier::visit(DbgVariableRecord &DVR) {
 
   // A DIArgList can have a valid branch expression which doesn't use
   // DW_OP_LLVM_arg, so check the record as well.
-  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
-    bool HasControlFlow = llvm::any_of(
-        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+  const DIExpression *Expr = DVR.getExpression();
+  if (DVR.hasArgList() && Expr->isValid()) {
+    bool HasSymbolicControlFlow =
+        llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return dwarf::isSymbolicControlFlowOp(Op.getOp());
         });
-    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
-            &DVR, MD, DVR.getExpression(), BB, F);
+    CheckDI(!HasSymbolicControlFlow,
+            "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,
+            F);
   }
 
   if (DVR.isDbgAssign()) {

>From c173ccf8a312d19de4b574813c9df6f2cda66671 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 18:17:17 -0700
Subject: [PATCH 5/7] [DebugInfo] Add opcode queries to DIExpression operands

Add is and isOneOf to DIExpression::ExprOperand and use them for the
symbolic branch checks. This keeps the checks on the parsed operation and
lets us remove the branch-specific helpers from Dwarf.h.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h          | 10 ----------
 llvm/include/llvm/IR/DebugInfoMetadata.h        |  9 +++++++++
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp |  8 ++++----
 llvm/lib/IR/Verifier.cpp                        |  3 ++-
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 177c50be7b374..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,16 +1108,6 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
-/// Return true if Op is a symbolic branch to a label.
-inline bool isSymbolicBranchOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
-}
-
-/// Return true if Op is a symbolic label or branch.
-inline bool isSymbolicControlFlowOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
-}
-
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 846342bee6071..2d0d6a1e228cb 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -3539,6 +3539,15 @@ class DIExpression : public MDNode {
     /// Get the operand code.
     uint64_t getOp() const { return *Op; }
 
+    /// Return true if this is \p Opcode.
+    bool is(uint64_t Opcode) const { return getOp() == Opcode; }
+
+    /// Return true if this is one of \p Opcodes.
+    template <typename... Ts> bool isOneOf(Ts... Opcodes) const {
+      static_assert(sizeof...(Ts) > 0, "requires at least one opcode");
+      return (is(Opcodes) || ...);
+    }
+
     /// Get an argument to the operand.
     ///
     /// Never returns the operand itself.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index f90edb1e68157..6bf604aae814c 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,7 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return dwarf::isSymbolicBranchOp(Op.getOp());
+        return Op.isOneOf(dwarf::DW_OP_LLVM_bra, dwarf::DW_OP_LLVM_skip);
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -577,12 +577,12 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
                            dwarf::OperationEncodingString(OpNum) +
                            " without DW_OP_convert support");
-      if (OpNum == dwarf::DW_OP_LLVM_label) {
+      if (Op->is(dwarf::DW_OP_LLVM_label)) {
         Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
         break;
       }
-      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
-                                            : dwarf::DW_OP_skip);
+      emitOp(Op->is(dwarf::DW_OP_LLVM_bra) ? dwarf::DW_OP_bra
+                                           : dwarf::DW_OP_skip);
       Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
       emitData2(0);
       break;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 0dc59163fa864..48e7f88d9c07a 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7188,7 +7188,8 @@ void Verifier::visit(DbgVariableRecord &DVR) {
   if (DVR.hasArgList() && Expr->isValid()) {
     bool HasSymbolicControlFlow =
         llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return dwarf::isSymbolicControlFlowOp(Op.getOp());
+          return Op.isOneOf(dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                            dwarf::DW_OP_LLVM_skip);
         });
     CheckDI(!HasSymbolicControlFlow,
             "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,

>From cc43f3efed3c84cc291f8483c7b1b952e151bbe7 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Thu, 23 Jul 2026 18:33:53 -0700
Subject: [PATCH 6/7] [DebugInfo] Add an early return to
 DwarfExpression::addExpression

Add an early return to streamline DwarfExpression::addExpression's handling
of symbolic branches.
---
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    | 47 ++++++++++---------
 1 file changed, 24 insertions(+), 23 deletions(-)

diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 6bf604aae814c..56dd5ff03ebf4 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -806,31 +806,32 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
-  if (HasSymbolicBranches) {
-    for (const BranchFixup &Fixup : Fixups) {
-      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
-        return Candidate.ID == Fixup.LabelID;
-      });
-      if (Label == Labels.end())
-        report_fatal_error(Twine("DWARF expression branch to label ") +
-                           Twine(Fixup.LabelID) + " has no matching label");
-
-      // DW_OP_bra and DW_OP_skip apply the displacement after reading their
-      // two-byte operand, so use the byte after the placeholder as the base.
-      int64_t Displacement =
-          static_cast<int64_t>(Label->Offset) -
-          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
-      if (!isInt<16>(Displacement))
-        report_fatal_error(Twine("DWARF expression branch offset ") +
-                           Twine(Displacement) + " is outside [-32768, 32767]");
-
-      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
-                                  static_cast<uint16_t>(Displacement));
-    }
+  if (!HasSymbolicBranches)
+    return true;
 
-    disableTemporaryBuffer();
-    commitTemporaryBuffer();
+  for (const BranchFixup &Fixup : Fixups) {
+    auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+      return Candidate.ID == Fixup.LabelID;
+    });
+    if (Label == Labels.end())
+      report_fatal_error(Twine("DWARF expression branch to label ") +
+                         Twine(Fixup.LabelID) + " has no matching label");
+
+    // DW_OP_bra and DW_OP_skip apply the displacement after reading their
+    // two-byte operand, so use the byte after the placeholder as the base.
+    int64_t Displacement =
+        static_cast<int64_t>(Label->Offset) -
+        static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+    if (!isInt<16>(Displacement))
+      report_fatal_error(Twine("DWARF expression branch offset ") +
+                         Twine(Displacement) + " is outside [-32768, 32767]");
+
+    replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                static_cast<uint16_t>(Displacement));
   }
+
+  disableTemporaryBuffer();
+  commitTemporaryBuffer();
   return true;
 }
 

>From 109bc0e91e481290ad541872ccced007aa155a91 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Thu, 23 Jul 2026 21:48:42 -0700
Subject: [PATCH 7/7] Update LangRef.md around DWARF branch ops

---
 llvm/docs/LangRef.md | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a95644a1cc006..776e6267c7c24 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7182,13 +7182,19 @@ DW_OP_LLVM_bra,   <label-id>
 DW_OP_LLVM_skip,  <label-id>
 ```
 
-`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
-branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
-branches. Label IDs are local to the expression. Every branch needs a matching
-label, labels without branches are valid, and each ID can only be declared
-once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
-
-See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+`DW_OP_LLVM_bra` and `DW_OP_LLVM_skip` have the same semantics as the standard
+DWARF `DW_OP_bra` and `DW_OP_skip` operations. The LLVM forms use a label ID
+because the byte offset isn't known until emission. CodeGen resolves each
+branch's label ID and replaces the LLVM form with the corresponding standard
+operation and byte offset. `DW_OP_LLVM_label` declares the target for a label
+ID and emits no bytes.
+
+Label IDs are local to the expression. Every branch needs a matching label,
+labels without branches are valid, and each ID can only be declared once. The
+standard `DW_OP_bra` and `DW_OP_skip` operations aren't valid in LLVM IR. See
+[the DWARF standard](https://dwarfstd.org/) for the standard operations and
+{ref}`symbolic control flow <symbolic-control-flow>` for the LLVM-specific
+rules.
 
 ##### DIAssignID
 


From llvmlistbot at llvm.org  Thu Jul 23 23:00:00 2026
From: llvmlistbot at llvm.org (Thurston Dang)
Date: Thu, 23 Jul 2026 23:00:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a62ff60.0c05e68e.81651.4089@mx.google.com>


https://github.com/thurstond approved this pull request.


https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 23:00:20 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Thu, 23 Jul 2026 23:00:20 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a62ff74.b1416da1.20b4cf.d810@mx.google.com>


================
@@ -757,6 +806,31 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
----------------
echristo wrote:

Nice catch :)

Fixed.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Thu Jul 23 23:04:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 23:04:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 906f520 - [mlir] Fix alignment assertion for
 empty DenseArrayAttr (#211730)
Message-ID: <6a63006c.0c05e68e.81651.f39e@mx.google.com>


Author: Vitaly Buka
Date: 2026-07-23T23:04:23-07:00
New Revision: 906f520af01a4c788935f24caea00237b42624e2

URL: https://github.com/llvm/llvm-project/commit/906f520af01a4c788935f24caea00237b42624e2
DIFF: https://github.com/llvm/llvm-project/commit/906f520af01a4c788935f24caea00237b42624e2.diff

LOG: [mlir] Fix alignment assertion for empty DenseArrayAttr (#211730)

After #207274, libc++ fails __assume_aligned assertion in
__assume_valid_range.

https://lab.llvm.org/buildbot/#/builders/25/builds/18994

`DenseArrayAttrImpl<T>::operator ArrayRef<T>()` is just a cast of
`raw.data()`. `raw` is aligned copy of range from
from BytecodeReader done by
Builtin_DenseArrayRawDataParameter allocator.
However, if range is empty, aligned copying was
omitted, leaving unaligned `ArrayRef<char>`.

The fix is to replace unaligned `ArrayRef<>` with
aligned default constructed.

Added: 
    

Modified: 
    mlir/include/mlir/IR/BuiltinAttributes.td
    mlir/lib/IR/BuiltinAttributes.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/IR/BuiltinAttributes.td b/mlir/include/mlir/IR/BuiltinAttributes.td
index f238137734d56..ba2c374a9ab48 100644
--- a/mlir/include/mlir/IR/BuiltinAttributes.td
+++ b/mlir/include/mlir/IR/BuiltinAttributes.td
@@ -160,6 +160,9 @@ def Builtin_DenseArrayRawDataParameter : ArrayRefParameter<
           $_allocator.allocate($_self.size(), alignof(uint64_t)));
       llvm::uninitialized_copy($_self, alloc);
       $_dst = ArrayRef<char>(alloc, $_self.size());
+    } else {
+      // Align possibly unaligned empty range.
+      $_dst = {};
     }
   }];
 }

diff  --git a/mlir/lib/IR/BuiltinAttributes.cpp b/mlir/lib/IR/BuiltinAttributes.cpp
index 9fda2ef8e5059..0f84f7a682fcf 100644
--- a/mlir/lib/IR/BuiltinAttributes.cpp
+++ b/mlir/lib/IR/BuiltinAttributes.cpp
@@ -19,6 +19,7 @@
 #include "mlir/IR/SymbolTable.h"
 #include "mlir/IR/Types.h"
 #include "llvm/ADT/APSInt.h"
+#include "llvm/Support/Alignment.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/DebugLog.h"
 #include "llvm/Support/Endian.h"
@@ -822,6 +823,7 @@ Attribute DenseArrayAttrImpl<T>::parse(AsmParser &parser, Type odsType) {
 template <typename T>
 DenseArrayAttrImpl<T>::operator ArrayRef<T>() const {
   ArrayRef<char> raw = getRawData();
+  assert(llvm::isAddrAligned(llvm::Align(alignof(T)), raw.data()));
   assert((raw.size() % sizeof(T)) == 0);
   return ArrayRef<T>(reinterpret_cast<const T *>(raw.data()),
                      raw.size() / sizeof(T));


        

From llvmlistbot at llvm.org  Thu Jul 23 23:04:30 2026
From: llvmlistbot at llvm.org (Vitaly Buka)
Date: Thu, 23 Jul 2026 23:04:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix alignment assertion for empty
	DenseArrayAttr (PR #211730)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211730@github.com>
Message-ID: <6a63006e.f1c1923d.d9163.6dfb@mx.google.com>


https://github.com/vitalybuka closed https://github.com/llvm/llvm-project/pull/211730

From llvmlistbot at llvm.org  Thu Jul 23 23:14:14 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 23:14:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid verifying live user in
	conversion diagnostic (PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a6302b6.510f576d.33aa7d.81bf@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Thu Jul 23 23:16:03 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 23:16:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Diagnose missing return value
	(PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a630323.6c5720b3.1f1229.ca1c@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/210566

From llvmlistbot at llvm.org  Thu Jul 23 23:21:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 23:21:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Diagnose missing return value
	(PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a63046e.21a19891.105e40.a691@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210566

>From 8d558af4b5334474b52fa60002ce5e521e786b88 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir][SPIRV] Diagnose missing return value

---
 mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp        |  6 +++++-
 .../test-legalize-erased-op-with-uses.mlir    | 21 ++++++++++++++++++-
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index 99bc3913c56c6..6bd91e7dfd8df 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -1226,7 +1226,11 @@ LogicalResult spirv::FuncOp::verifyBody() {
                    "returns 1 value but enclosing function requires ")
                << fnType.getNumResults() << " results";
 
-      auto retOperandType = retOp.getValue().getType();
+      Value retOperand = retOp.getValue();
+      if (!retOperand)
+        return retOp.emitOpError("return value is missing");
+
+      auto retOperandType = retOperand.getType();
       auto fnResultType = fnType.getResult(0);
       if (retOperandType != fnResultType)
         return retOp.emitOpError(" return value's type (")
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..829ff2f183e53 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics can print a live user under an op that is not
+// verifiable while the conversion is in progress.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Thu Jul 23 23:24:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Thu, 23 Jul 2026 23:24:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][SPIRV] Diagnose missing return value
	(PR #210566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210566@github.com>
Message-ID: <6a63052f.8e75e0cb.2de827.2a07@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/210566

>From 6b10020be20510204756ac34199600e578d963f1 Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Sat, 18 Jul 2026 19:10:29 -0700
Subject: [PATCH] [mlir][SPIRV] Diagnose missing return value

---
 mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp        |  6 +++++-
 .../test-legalize-erased-op-with-uses.mlir    | 21 ++++++++++++++++++-
 2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
index 99bc3913c56c6..6bd91e7dfd8df 100644
--- a/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
+++ b/mlir/lib/Dialect/SPIRV/IR/SPIRVOps.cpp
@@ -1226,7 +1226,11 @@ LogicalResult spirv::FuncOp::verifyBody() {
                    "returns 1 value but enclosing function requires ")
                << fnType.getNumResults() << " results";
 
-      auto retOperandType = retOp.getValue().getType();
+      Value retOperand = retOp.getValue();
+      if (!retOperand)
+        return retOp.emitOpError("return value is missing");
+
+      auto retOperandType = retOperand.getType();
       auto fnResultType = fnType.getResult(0);
       if (retOperandType != fnResultType)
         return retOp.emitOpError(" return value's type (")
diff --git a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
index 031442b0ee2da..bf92397a9aa49 100644
--- a/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
+++ b/mlir/test/Transforms/test-legalize-erased-op-with-uses.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -verify-diagnostics
+// RUN: mlir-opt %s -test-legalize-unknown-root-patterns -split-input-file -verify-diagnostics
 
 // Test that an error is emitted when an operation is marked as "erased", but
 // has users that live across the conversion.
@@ -8,3 +8,22 @@ func.func @remove_all_ops(%arg0: i32) -> i32 {
   // expected-note at below {{see existing live user here}}
   return %0 : i32
 }
+
+// -----
+
+// Test that diagnostics for a failed conversion do not crash when printing
+// malformed SPIR-V IR produced during unresolved materialization cleanup.
+module {
+  spirv.func @remove_test_ops() -> i32 "None" {
+    %cst1_i32 = spirv.Constant 1 : i32
+    // expected-error at below {{failed to legalize unresolved materialization from () to ('i32') that remained live after conversion}}
+    %0 = test.with_bounds {smax = 0 : si32, smin = 0 : si32, umax = 0 : i32, umin = 0 : ui32} : i32
+    %1 = builtin.unrealized_conversion_cast %cst1_i32 : i32 to i32
+    // expected-note at below {{see existing live user here}}
+    %2 = spirv.IAdd %0, %1 : i32
+    %3 = builtin.unrealized_conversion_cast %2 : i32 to i32
+    %4 = test.reflect_bounds %3 : i32
+    %5 = builtin.unrealized_conversion_cast %4 : i32 to i32
+    spirv.ReturnValue %5 : i32
+  }
+}


From llvmlistbot at llvm.org  Fri Jul 24 00:00:28 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Fri, 24 Jul 2026 00:00:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a630d8c.cb03ad15.9d9a7.5821@mx.google.com>


https://github.com/adam-smnk approved this pull request.


https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Fri Jul 24 00:05:06 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:05:06 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a630ea2.1905e68e.25404.2104@mx.google.com>


https://github.com/Ritanya-B-Bharadwaj approved this pull request.

Agreed! Looks good to me. Must be merged after the conflicts are resolved. 

https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Fri Jul 24 00:14:20 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:14:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a6310cc.1b0626d0.1ff4c3.97d7@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 3a9b774ca84ccb1395764dac6683a553c6e751e8 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/2] [mlir][memref] Avoid mem2reg crash on element count
 overflow

Avoid crash in test alias analysis operand printing

Fix AffineForOP creash

remove

fix

fix issue
---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;

>From da4eb6dd9bdc89e684ed2f79e1733a8cb698e195 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Fri, 24 Jul 2026 15:10:39 +0800
Subject: [PATCH 2/2] [mlir][affine] Add regression test for fold with detached
 block

---
 mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
index 94dfceadbc449..bbf1c4de63b82 100644
--- a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
+++ b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
@@ -583,3 +583,17 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// Regression test: the affine.for empty loop folder used to crash when fold
+// was attempted during dialect conversion, after the old entry block was
+// detached but the loop still yielded one of its arguments. The folder must
+// bail out instead of hitting the isDefinedOutsideOfLoop assertion.
+
+// CHECK-LABEL: @affine_for_fold_detached_block
+// CHECK: affine.for
+func.func @affine_for_fold_detached_block(%arg0: index) -> index {
+  %0 = affine.for %i = 0 to 10 iter_args(%acc = %arg0) -> (index) {
+    affine.yield %arg0 : index
+  }
+  return %0 : index
+}
\ No newline at end of file


From llvmlistbot at llvm.org  Fri Jul 24 00:14:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:14:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a6310e7.160324b0.17e275.887d@mx.google.com>


https://github.com/LouisLu060211 updated https://github.com/llvm/llvm-project/pull/209111

>From 3a9b774ca84ccb1395764dac6683a553c6e751e8 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Tue, 23 Jun 2026 18:02:12 +0800
Subject: [PATCH 1/3] [mlir][memref] Avoid mem2reg crash on element count
 overflow

Avoid crash in test alias analysis operand printing

Fix AffineForOP creash

remove

fix

fix issue
---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index f095500495f18..544715e6a169d 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2596,6 +2596,9 @@ static SmallVector<OpFoldResult> AffineForEmptyLoopFolder(AffineForOp forOp) {
       return {};
     if (iterArgIt == iterArgs.end()) {
       // `val` is defined outside of the loop.
+      if(!val.getParentRegion()){
+        return{};
+      }
       assert(forOp.isDefinedOutsideOfLoop(val) &&
              "must be defined outside of the loop");
       hasValDefinedOutsideLoop = true;

>From da4eb6dd9bdc89e684ed2f79e1733a8cb698e195 Mon Sep 17 00:00:00 2001
From: LouisLu0602 <yaolu0602 at gmail.com>
Date: Fri, 24 Jul 2026 15:10:39 +0800
Subject: [PATCH 2/3] [mlir][affine] Add regression test for fold with detached
 block

---
 mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
index 94dfceadbc449..bbf1c4de63b82 100644
--- a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
+++ b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
@@ -583,3 +583,17 @@ module attributes {transform.with_named_sequence} {
     transform.yield
   }
 }
+
+// Regression test: the affine.for empty loop folder used to crash when fold
+// was attempted during dialect conversion, after the old entry block was
+// detached but the loop still yielded one of its arguments. The folder must
+// bail out instead of hitting the isDefinedOutsideOfLoop assertion.
+
+// CHECK-LABEL: @affine_for_fold_detached_block
+// CHECK: affine.for
+func.func @affine_for_fold_detached_block(%arg0: index) -> index {
+  %0 = affine.for %i = 0 to 10 iter_args(%acc = %arg0) -> (index) {
+    affine.yield %arg0 : index
+  }
+  return %0 : index
+}
\ No newline at end of file

>From 8f5e3618be3da0dcdd466897590b4bab10e2d11d Mon Sep 17 00:00:00 2001
From: LouisLu060211 <38174270+LouisLu060211 at users.noreply.github.com>
Date: Fri, 24 Jul 2026 00:14:37 -0700
Subject: [PATCH 3/3] Update func-to-llvm.mlir

---
 mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
index bbf1c4de63b82..8229cd964ef1e 100644
--- a/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
+++ b/mlir/test/Conversion/FuncToLLVM/func-to-llvm.mlir
@@ -596,4 +596,4 @@ func.func @affine_for_fold_detached_block(%arg0: index) -> index {
     affine.yield %arg0 : index
   }
   return %0 : index
-}
\ No newline at end of file
+}


From llvmlistbot at llvm.org  Fri Jul 24 00:15:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:15:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a631101.0c05e68e.81651.f3ad@mx.google.com>


LouisLu060211 wrote:

> Add some unit test

updated with a unit test


https://github.com/llvm/llvm-project/pull/209111

From llvmlistbot at llvm.org  Fri Jul 24 00:16:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:16:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Fix affine for fold detached block (PR
	#209111)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209111@github.com>
Message-ID: <6a63113f.83ce629a.2277d7.f321@mx.google.com>


LouisLu060211 wrote:

> Hello @LouisLu060211 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read the LLVM AI Tool Use Policy, the Code-Review Policy, and the Developer Policy. 

https://github.com/llvm/llvm-project/pull/209111

From llvmlistbot at llvm.org  Fri Jul 24 00:20:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:20:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Avoid crash in test alias analysis (PR
	#207670)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207670@github.com>
Message-ID: <6a63123e.5a0412d2.325e39.4350@mx.google.com>


LouisLu060211 wrote:

> Hello @LouisLu060211 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read the polices, and  I confirm that AI tool usage is noted in the PR description. 

https://github.com/llvm/llvm-project/pull/207670

From llvmlistbot at llvm.org  Fri Jul 24 00:25:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:25:19 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a63135f.b4a331f0.2642a.71b5@mx.google.com>


LouisLu060211 wrote:

> Hello @LouisLu060211 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read the policies, and I confirm that AI tool usage is noted in the PR description. 

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 00:27:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:27:19 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Support for "!$omp
	dispatch". (PR #203320)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203320@github.com>
Message-ID: <6a6313d7.9edb5c9d.3c157a.5ca1@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/203320

>From d007fc287982ed638ef3900307ca700f0d529f87 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 10:56:08 -0500
Subject: [PATCH 1/6] Support for "!$omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 33 ++++++-
 flang/test/Integration/OpenMP/dispatch.f90    | 76 ++++++++++++++++
 flang/test/Lower/OpenMP/Todo/dispatch.f90     | 12 ---
 flang/test/Lower/OpenMP/dispatch.f90          | 89 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 15 ++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 ++++
 mlir/test/Dialect/OpenMP/dispatch.mlir        | 77 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  | 81 +++++++++++++++++
 8 files changed, 385 insertions(+), 15 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/dispatch.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90
 create mode 100644 flang/test/Lower/OpenMP/dispatch.f90
 create mode 100644 mlir/test/Dialect/OpenMP/dispatch.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-dispatch.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index be876c563433a..758d8eb8148e5 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2592,6 +2592,18 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -3940,6 +3952,9 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
   case llvm::omp::Directive::OMPD_barrier:
     newOp = genBarrierOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
+  case llvm::omp::Directive::OMPD_dispatch:
+    newOp = genDispatchOp(converter, symTable, semaCtx, eval, loc, queue, item);
+    break;
   case llvm::omp::Directive::OMPD_distribute:
     newOp = genStandaloneDistribute(converter, symTable, stmtCtx, semaCtx, eval,
                                     loc, queue, item);
@@ -5312,9 +5327,21 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
-                   const parser::OpenMPDispatchConstruct &) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPDispatchConstruct");
+                   const parser::OpenMPDispatchConstruct &dispatchConstruct) {
+  const parser::OmpDirectiveSpecification &beginSpec =
+      dispatchConstruct.BeginDir();
+  List<Clause> clauses = makeClauses(beginSpec.Clauses(), semaCtx);
+  if (auto &endSpec = dispatchConstruct.EndDir())
+    clauses.append(makeClauses(endSpec->Clauses(), semaCtx));
+
+  llvm::omp::Directive directive = beginSpec.DirId();
+  mlir::Location currentLocation = converter.genLocation(beginSpec.source);
+
+  ConstructQueue queue{
+      buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
+                          eval, beginSpec.source, directive, clauses)};
+  genOMPDispatch(converter, symTable, semaCtx, eval, currentLocation, queue,
+                 queue.begin());
 }
 
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..c4d16690c8e29
--- /dev/null
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -0,0 +1,76 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
+!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
+!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
+!CHECK: [[IF_TRUE]]:
+!CHECK: call void @_QMfuncsPfoo_variant2()
+!CHECK: [[IF_FALSE]]:
+!CHECK: call void @_QMfuncsPfoo_variant1()
+
+!CHECK-LABEL: define void @_QQmain()
+!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region
+!CHECK: omp.dispatch.region:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont
+!CHECK: omp.region.cont:
+!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region2
+!CHECK: omp.dispatch.region2:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont1
+!CHECK: omp.region.cont1:
+
+module funcs
+  implicit none
+  logical :: foo_sub
+
+contains
+
+  subroutine foo_variant1()
+    print *, "in foo_variant1"
+  end subroutine
+
+  subroutine foo_variant2()
+    print *, "in foo_variant2"
+  end subroutine
+
+  subroutine foo_dispatch()
+    if (foo_sub) then
+      call foo_variant2()
+    else
+      call foo_variant1()
+    end if
+  end subroutine
+
+end module funcs
+
+program dispatch_test
+  use funcs
+  implicit none
+
+  foo_sub = .false.
+  !$omp dispatch
+  call foo_dispatch()
+
+  foo_sub = .true.
+  !$omp dispatch
+  call foo_dispatch()
+
+end program
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
deleted file mode 100644
index 380dfa14eaae1..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ /dev/null
@@ -1,12 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPDispatchConstruct
-program p
-  integer r
-  r = 1
-!$omp dispatch nowait
-  call foo()
-contains
-  subroutine foo
-  end subroutine
-end program p
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..df86503817163
--- /dev/null
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -0,0 +1,89 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
+
+!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
+!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
+!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
+!HLFIR: fir.if %[[CMP_H]] {
+!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!HLFIR: } else {
+!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!HLFIR: }
+
+!FIR-LABEL: func @_QMfuncsPfoo_dispatch
+!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
+!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
+!FIR: fir.if %[[CMP_F]] {
+!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!FIR: } else {
+!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!FIR: }
+
+module funcs
+  implicit none
+
+contains
+
+  subroutine variant1()
+    print *, "in variant1"
+  end subroutine
+
+  subroutine variant2()
+    print *, "in variant2"
+  end subroutine
+
+  subroutine foo_dispatch(x)
+    integer, intent(in) :: x
+    if (x == 1) then
+      call variant1()
+    else
+      call variant2()
+    end if
+  end subroutine
+
+end module funcs
+
+!HLFIR-LABEL: func @_QQmain
+!FIR-LABEL: func @_QQmain
+!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
+!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
+!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+program dispatch_test
+  use funcs
+  implicit none
+  integer :: x
+
+  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
+  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
+  x = 1
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+
+  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
+  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
+  x = 2
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 51e7080db5b29..6170e40a835c6 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2286,6 +2286,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
   ];
 }
 
+//===----------------------------------------------------------------------===//
+// [Spec 5.1] 12.3 dispatch Construct
+//===----------------------------------------------------------------------===//
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+  let summary = "dispatch construct";
+  let description = [{
+    The dispatch construct enables the invocation of a variant of a
+    base procedure. The structured block of a dispatch construct is a
+    single expression statement that contains a function call or a
+    subroutine call.
+  }];
+
+  let assemblyFormat = "$region attr-dict";
+}
+
 //===----------------------------------------------------------------------===//
 // [Spec 5.2] 6.5 allocate Directive
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..12818a5e56ac6 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -761,6 +761,20 @@ static llvm::omp::ProcBindKind getProcBindKind(omp::ClauseProcBindKind kind) {
   llvm_unreachable("Unknown ClauseProcBindKind kind");
 }
 
+/// Convert 'dispatch' operation into LLVM IR.
+static LogicalResult
+convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  auto dispatchOp = cast<omp::DispatchOp>(opInst);
+  auto &region = dispatchOp.getRegion();
+  auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
+                                    moduleTranslation);
+  if (!result)
+    return handleError(result.takeError(), opInst);
+  builder.SetInsertPoint(*result);
+  return success();
+}
+
 /// Converts an OpenMP 'masked' operation into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpMasked(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -8815,6 +8829,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })
+          .Case([&](omp::DispatchOp) {
+            return convertOmpDispatch(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::MaskedOp) {
             return convertOmpMasked(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
new file mode 100644
index 0000000000000..c935f56836185
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: func @foo_dispatch
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @foo_dispatch(%x : memref<i32>) -> () {
+  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
+  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
+  %v = memref.load %x[] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  %cmp = arith.cmpi eq, %v, %c1 : i32
+  cf.cond_br %cmp, ^bb1, ^bb2
+// CHECK: ^[[BB1]]:
+// CHECK: call @variant1() : () -> ()
+^bb1:
+  func.call @variant1() : () -> ()
+  cf.br ^bb3
+// CHECK: ^[[BB2]]:
+// CHECK: call @variant2() : () -> ()
+^bb2:
+  func.call @variant2() : () -> ()
+  cf.br ^bb3
+^bb3:
+  return
+}
+
+// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
+// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// Test multiple dispatch regions with stores selecting different variants.
+// CHECK-LABEL: func @omp_dispatch_multiple
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  memref.store %c1, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
+  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
+  %c2 = arith.constant 2 : i32
+  memref.store %c2, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// CHECK-LABEL: func private @variant1()
+// CHECK-LABEL: func private @variant2()
+func.func private @variant1() -> ()
+func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
new file mode 100644
index 0000000000000..8a3a7a6b5d8e6
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -0,0 +1,81 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.return %0 : i32
+}
+
+llvm.func @variant1() -> ()
+llvm.func @variant2() -> ()
+
+// CHECK-LABEL: define void @foo_dispatch()
+llvm.func @foo_dispatch() {
+  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
+  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
+  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %1 = llvm.load %0 : !llvm.ptr -> i32
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  %cmp = llvm.icmp "eq" %1, %c1 : i32
+  llvm.cond_br %cmp, ^bb1, ^bb2
+// CHECK: [[BB1]]:
+// CHECK: call void @variant1()
+^bb1:
+  llvm.call @variant1() : () -> ()
+  llvm.br ^bb3
+// CHECK: [[BB2]]:
+// CHECK: call void @variant2()
+^bb2:
+  llvm.call @variant2() : () -> ()
+  llvm.br ^bb3
+^bb3:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch()
+llvm.func @test_omp_dispatch() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch_multiple()
+llvm.func @test_omp_dispatch_multiple() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  // CHECK: store i32 2, ptr @x
+  %c2 = llvm.mlir.constant(2 : i32) : i32
+  llvm.store %c2, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region2
+  omp.dispatch {
+    // CHECK: omp.dispatch.region2:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont1
+    omp.terminator
+  }
+  // CHECK: omp.region.cont1:
+  llvm.return
+}

>From b50c11a322615d4e189303f9b4b73e1c6246c7ae Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 12:19:36 -0500
Subject: [PATCH 2/6] Checking for no caluses support with "omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 3 +++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 758d8eb8148e5..71bc392d0e8d9 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -5337,6 +5337,9 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
+  if (!clauses.empty())
+    TODO(currentLocation, "OpenMP Dispatch clauses");
+
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 6170e40a835c6..47c7587c5d08b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2293,9 +2293,10 @@ def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure. The structured block of a dispatch construct is a
-    single expression statement that contains a function call or a
-    subroutine call.
+    base procedure (without any construct's clauses: device, is_device_ptr,
+    nowait, nocontext, novariants). The structured block of a dispatch
+    construct is a single expression statement that contains a function
+    call or a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";

>From 3ca936ebb5aff08e318fcd4abede5951e77fcb07 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 18 Jun 2026 11:50:04 -0500
Subject: [PATCH 3/6] Handling feedbacks suggested by Sergio Afonso.

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 37 ++++++++++++-------
 flang/test/Lower/OpenMP/dispatch.f90          | 26 +------------
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 11 +++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  4 ++
 mlir/test/Dialect/OpenMP/dispatch.mlir        |  1 +
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  |  1 +
 6 files changed, 36 insertions(+), 44 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 33e67ed2c3ab7..fe7d17807727a 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1661,6 +1661,16 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
   cp.processOrder(clauseOps);
 }
 
+static void genDispatchClauses(lower::AbstractConverter &converter,
+                               semantics::SemanticsContext &semaCtx,
+                               const List<Clause> &clauses,
+                               mlir::Location loc) {
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
+                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+      loc, llvm::omp::Directive::OMPD_dispatch);
+}
+
 static void genFlushClauses(lower::AbstractConverter &converter,
                             semantics::SemanticsContext &semaCtx,
                             const ObjectList &objects,
@@ -2154,6 +2164,18 @@ genCriticalOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item, nameAttr);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::FlushOp
 genFlushOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
            semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -2592,18 +2614,6 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
-static mlir::omp::DispatchOp
-genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-              semantics::SemanticsContext &semaCtx,
-              lower::pft::Evaluation &eval, mlir::Location loc,
-              const ConstructQueue &queue,
-              ConstructQueue::const_iterator item) {
-  return genOpWithBody<mlir::omp::DispatchOp>(
-      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
-                        llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
-}
-
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -5294,8 +5304,7 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  if (!clauses.empty())
-    TODO(currentLocation, "OpenMP Dispatch clauses");
+  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
 
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index df86503817163..ecc7b1b31454f 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,5 +1,4 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
-!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
 
 !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
 !HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
@@ -12,16 +11,6 @@
 !HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
 !HLFIR: }
 
-!FIR-LABEL: func @_QMfuncsPfoo_dispatch
-!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
-!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
-!FIR: fir.if %[[CMP_F]] {
-!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!FIR: } else {
-!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!FIR: }
-
 module funcs
   implicit none
 
@@ -35,6 +24,7 @@ subroutine variant2()
     print *, "in variant2"
   end subroutine
 
+  !TODO: replace with declare_variant when the support is merged.
   subroutine foo_dispatch(x)
     integer, intent(in) :: x
     if (x == 1) then
@@ -47,10 +37,6 @@ subroutine foo_dispatch(x)
 end module funcs
 
 !HLFIR-LABEL: func @_QQmain
-!FIR-LABEL: func @_QQmain
-!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
-!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
-!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
 program dispatch_test
   use funcs
   implicit none
@@ -59,31 +45,21 @@ program dispatch_test
   !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
   !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
   !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
   x = 1
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 
   !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
   !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
   x = 2
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 47c7587c5d08b..490bee24799c2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,14 +2289,15 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
+   ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure (without any construct's clauses: device, is_device_ptr,
-    nowait, nocontext, novariants). The structured block of a dispatch
-    construct is a single expression statement that contains a function
-    call or a subroutine call.
+    base procedure. The structured block of a dispatch construct is
+    a single expression statement that contains a function call or
+    a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 12818a5e56ac6..04c36b87d73ad 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -766,6 +766,10 @@ static LogicalResult
 convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
                    LLVM::ModuleTranslation &moduleTranslation) {
   auto dispatchOp = cast<omp::DispatchOp>(opInst);
+
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+
   auto &region = dispatchOp.getRegion();
   auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
                                     moduleTranslation);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c935f56836185..b8d4b93a6c8e3 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -41,6 +41,7 @@ func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
 }
 
 // Test multiple dispatch regions with stores selecting different variants.
+// TODO: Use declare_variant when the support is merged.
 // CHECK-LABEL: func @omp_dispatch_multiple
 // CHECK-SAME: (%[[X:.*]]: memref<i32>)
 func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index 8a3a7a6b5d8e6..d947a871ca30c 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -8,6 +8,7 @@ llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
 llvm.func @variant1() -> ()
 llvm.func @variant2() -> ()
 
+// TODO: Add support for declare_variant, when support to it is merged.
 // CHECK-LABEL: define void @foo_dispatch()
 llvm.func @foo_dispatch() {
   // CHECK: %[[ADDR:.*]] = load i32, ptr @x

>From 6eacbc3cd3d8e278d7b51ea898fe27e009de2228 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 01:09:09 -0500
Subject: [PATCH 4/6] 1) Handling -fc1 -fopenmp-simd 2) Todo testcase for
 clauses not supported in dispatch

---
 flang/lib/Semantics/rewrite-parse-tree.cpp |  5 +++
 flang/test/Lower/OpenMP/Todo/dispatch.f90  | 43 ++++++++++++++++++++++
 flang/test/Semantics/OpenMP/simd-only.f90  | 12 ++++++
 3 files changed, 60 insertions(+)
 create mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90

diff --git a/flang/lib/Semantics/rewrite-parse-tree.cpp b/flang/lib/Semantics/rewrite-parse-tree.cpp
index 4e1c9bae9c153..76a60c75488e3 100644
--- a/flang/lib/Semantics/rewrite-parse-tree.cpp
+++ b/flang/lib/Semantics/rewrite-parse-tree.cpp
@@ -191,6 +191,11 @@ void RewriteMutator::OpenMPSimdOnly(
                        &omp->value().u)}) {
           it = replaceInlineBlock(std::get<parser::Block>(ompBlock->t), it);
           continue;
+        } else if (auto *ompDispatch{
+                       std::get_if<parser::OpenMPDispatchConstruct>(
+                           &omp->value().u)}) {
+          it = replaceInlineBlock(std::get<parser::Block>(ompDispatch->t), it);
+          continue;
         } else if (auto *ompLoop{std::get_if<parser::OpenMPLoopConstruct>(
                        &omp->value().u)}) {
           if (LoopConstructIsSIMD(ompLoop)) {
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
new file mode 100644
index 0000000000000..f100cb40ba3d7
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -0,0 +1,43 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEPEND -o - %s 2>&1 | FileCheck %s --check-prefix=DEPEND
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEVICE -o - %s 2>&1 | FileCheck %s --check-prefix=DEVICE
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
+
+! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
+! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
+! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
+! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
+! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
+! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
+
+subroutine sub()
+#ifdef IS_DEVICE_PTR
+  use iso_c_binding
+  type(c_ptr) :: x
+#endif
+  integer :: r
+#ifdef DEPEND
+!$omp dispatch depend(inout: r)
+#endif
+#ifdef DEVICE
+!$omp dispatch device(0)
+#endif
+#ifdef IS_DEVICE_PTR
+!$omp dispatch is_device_ptr(x)
+#endif
+#ifdef NOCONTEXT
+!$omp dispatch nocontext(.false.)
+#endif
+#ifdef NOVARIANTS
+!$omp dispatch novariants(.true.)
+#endif
+#ifdef NOWAIT
+!$omp dispatch nowait
+#endif
+  call foo()
+contains
+  subroutine foo
+  end subroutine
+end subroutine sub
diff --git a/flang/test/Semantics/OpenMP/simd-only.f90 b/flang/test/Semantics/OpenMP/simd-only.f90
index 01370af0a00a7..860befaf43247 100644
--- a/flang/test/Semantics/OpenMP/simd-only.f90
+++ b/flang/test/Semantics/OpenMP/simd-only.f90
@@ -414,3 +414,15 @@ module test_declare_mapper
   ! CHECK-NOT: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpDeclareMapperDirective
   !$omp declare mapper(myvec_t :: v) map(v, v%data(1:v%len))
 end module
+
+! CHECK-LABEL: Name = 'test_dispatch'
+subroutine test_dispatch()
+  ! CHECK-NOT: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OpenMPDispatchConstruct
+  ! CHECK-NOT: OmpDirectiveName -> llvm::omp::Directive = dispatch
+  ! CHECK: ExecutionPartConstruct -> ExecutableConstruct -> ActionStmt -> CallStmt
+  !$omp dispatch
+  call foo()
+contains
+  subroutine foo()
+  end subroutine
+end subroutine

>From 8f25901612c660e29e53ab9c7e162a21a04ad966 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 07:14:38 -0500
Subject: [PATCH 5/6] Support for "dispatch nowait" except for MLIR to LLVMIR.
 Support in LLVM IR depends on "declare variant".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp                 | 14 ++++++++------
 flang/test/Lower/OpenMP/Todo/dispatch.f90         |  5 -----
 flang/test/Lower/OpenMP/dispatch.f90              |  7 +++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td     | 12 ++++++++----
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp      |  9 +++++++++
 .../Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  |  1 +
 mlir/test/Dialect/OpenMP/dispatch.mlir            | 15 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir          | 11 +++++++++++
 8 files changed, 59 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 4c2a238605f8d..959f7352e2c76 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1774,11 +1774,12 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
 
 static void genDispatchClauses(lower::AbstractConverter &converter,
                                semantics::SemanticsContext &semaCtx,
-                               const List<Clause> &clauses,
-                               mlir::Location loc) {
+                               const List<Clause> &clauses, mlir::Location loc,
+                               mlir::omp::DispatchOperands &clauseOps) {
   ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processNowait(clauseOps);
   cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
-                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+                 clause::Nocontext, clause::Novariants>(
       loc, llvm::omp::Directive::OMPD_dispatch);
 }
 
@@ -2276,10 +2277,13 @@ genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
               lower::pft::Evaluation &eval, mlir::Location loc,
               const ConstructQueue &queue,
               ConstructQueue::const_iterator item) {
+  mlir::omp::DispatchOperands clauseOps;
+  genDispatchClauses(converter, semaCtx, item->clauses, loc, clauseOps);
+
   return genOpWithBody<mlir::omp::DispatchOp>(
       OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
                         llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
+      queue, item, clauseOps);
 }
 
 static mlir::omp::FlushOp
@@ -5426,8 +5430,6 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
-
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
index f100cb40ba3d7..579d47d457def 100644
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -3,14 +3,12 @@
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
 
 ! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
 ! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
 ! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
 ! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
 ! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
-! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
 
 subroutine sub()
 #ifdef IS_DEVICE_PTR
@@ -32,9 +30,6 @@ subroutine sub()
 #endif
 #ifdef NOVARIANTS
 !$omp dispatch novariants(.true.)
-#endif
-#ifdef NOWAIT
-!$omp dispatch nowait
 #endif
   call foo()
 contains
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index ecc7b1b31454f..1b184a18a91af 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -62,4 +62,11 @@ program dispatch_test
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
   !HLFIR: }
+
+  !HLFIR: omp.dispatch nowait {
+  !$omp dispatch nowait
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !HLFIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index f048a9da71063..bb478c52381c5 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,17 +2289,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
 def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
-     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
-   ]> {
+    // TODO: Complete clause list (device, depend, is_device_ptr, nocontext,
+    // novariants).
+    OpenMP_NowaitClause
+  ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
     base procedure. The structured block of a dispatch construct is
     a single expression statement that contains a function call or
     a subroutine call.
-  }];
+  }] # clausesDescription;
 
-  let assemblyFormat = "$region attr-dict";
+  let builders = [
+    OpBuilder<(ins CArg<"const DispatchOperands &">:$clauses)>
+  ];
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index c2956a1cf7b79..6c1d5c29ec95d 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4933,6 +4933,15 @@ void MaskedOp::build(OpBuilder &builder, OperationState &state,
   MaskedOp::build(builder, state, clauses.filteredThreadId);
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Dispatch construct (12.3)
+//===----------------------------------------------------------------------===//
+
+void DispatchOp::build(OpBuilder &builder, OperationState &state,
+                       const DispatchOperands &clauses) {
+  DispatchOp::build(builder, state, clauses.nowait);
+}
+
 //===----------------------------------------------------------------------===//
 // Spec 5.2: Scan construct (5.6)
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e84daaa135748..75c5e707b9933 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -480,6 +480,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkDepend(op, result);
         checkNowait(op, result);
       })
+      .Case([&](omp::DispatchOp op) { checkNowait(op, result); })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
         checkInReductionByref(op, result);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index b8d4b93a6c8e3..c966784b1e22e 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -72,6 +72,21 @@ func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
   return
 }
 
+// Test the nowait clause on omp.dispatch.
+// CHECK-LABEL: func @omp_dispatch_nowait
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch nowait {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  omp.dispatch nowait {
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    omp.terminator
+  }
+  return
+}
+
 // CHECK-LABEL: func private @variant1()
 // CHECK-LABEL: func private @variant2()
 func.func private @variant1() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 377a5bb799be4..1bf9b563f8d6c 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -52,6 +52,17 @@ llvm.func @distribute_order(%lb : i32, %ub : i32, %step : i32) {
 
 // -----
 
+llvm.func @dispatch_nowait() {
+  // expected-error at below {{not yet implemented: Unhandled clause nowait in omp.dispatch operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.dispatch}}
+  omp.dispatch nowait {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @parallel_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.parallel operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.parallel}}

>From 7739eecf4c336629aedfa11911a4bea1cc81a6e8 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 02:20:10 -0500
Subject: [PATCH 6/6] Integrating variants using "declare variant" in the
 tests. Adding dispatch trait to "declare variant" in Utils.cpp Handling
 comments by Chi Chun Chen.

---
 flang/lib/Lower/OpenMP/Utils.cpp             |   3 +
 flang/test/Integration/OpenMP/dispatch.f90   |  42 +-----
 flang/test/Lower/OpenMP/dispatch.f90         |  59 +++-----
 mlir/test/Dialect/OpenMP/dispatch.mlir       | 150 ++++++++-----------
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir |  97 +++---------
 5 files changed, 109 insertions(+), 242 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Utils.cpp b/flang/lib/Lower/OpenMP/Utils.cpp
index 8f57f00d59c58..5cb6a0713462f 100644
--- a/flang/lib/Lower/OpenMP/Utils.cpp
+++ b/flang/lib/Lower/OpenMP/Utils.cpp
@@ -1379,6 +1379,9 @@ void collectEnclosingConstructTraits(
   for (; op; op = op->getParentOp()) {
     if (mlir::isa<mlir::omp::WsloopOp>(op))
       constructTraits.push_back(llvm::omp::TraitProperty::construct_for_for);
+    if (mlir::isa<mlir::omp::DispatchOp>(op))
+      constructTraits.push_back(
+          llvm::omp::TraitProperty::construct_dispatch_dispatch);
     if (mlir::isa<mlir::omp::ParallelOp>(op))
       constructTraits.push_back(
           llvm::omp::TraitProperty::construct_parallel_parallel);
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
index c4d16690c8e29..1ecccab5e90d4 100644
--- a/flang/test/Integration/OpenMP/dispatch.f90
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -8,55 +8,32 @@
 
 !RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
 
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
-!CHECK: call ptr @_FortranAioBeginExternalListOutput
-
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant()
 !CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
-!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
-!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
-!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
-!CHECK: [[IF_TRUE]]:
-!CHECK: call void @_QMfuncsPfoo_variant2()
-!CHECK: [[IF_FALSE]]:
-!CHECK: call void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QQmain()
-!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: call void @_QMfuncsPfoo_dispatch()
 !CHECK: br label %omp.dispatch.region
 !CHECK: omp.dispatch.region:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: call void @_QMfuncsPfoo_variant()
 !CHECK: br label %omp.region.cont
 !CHECK: omp.region.cont:
-!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
-!CHECK: br label %omp.dispatch.region2
-!CHECK: omp.dispatch.region2:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
-!CHECK: br label %omp.region.cont1
-!CHECK: omp.region.cont1:
 
 module funcs
   implicit none
-  logical :: foo_sub
 
 contains
 
-  subroutine foo_variant1()
-    print *, "in foo_variant1"
-  end subroutine
-
-  subroutine foo_variant2()
-    print *, "in foo_variant2"
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
   subroutine foo_dispatch()
-    if (foo_sub) then
-      call foo_variant2()
-    else
-      call foo_variant1()
-    end if
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -65,11 +42,8 @@ program dispatch_test
   use funcs
   implicit none
 
-  foo_sub = .false.
-  !$omp dispatch
   call foo_dispatch()
 
-  foo_sub = .true.
   !$omp dispatch
   call foo_dispatch()
 
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index 1b184a18a91af..4eb9439b7b35b 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,37 +1,23 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
 
-!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
-!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
-!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
-!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
-!HLFIR: fir.if %[[CMP_H]] {
-!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!HLFIR: } else {
-!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!HLFIR: }
+! Variant selection is provided by DECLARE VARIANT with a `construct={dispatch}`
+! match: inside a dispatch region the call to the base procedure `foo_dispatch`
+! is replaced by a call to its variant `foo_variant`.
 
 module funcs
   implicit none
 
 contains
 
-  subroutine variant1()
-    print *, "in variant1"
+  !HLFIR-LABEL: func @_QMfuncsPfoo_variant
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
-  subroutine variant2()
-    print *, "in variant2"
-  end subroutine
-
-  !TODO: replace with declare_variant when the support is merged.
-  subroutine foo_dispatch(x)
-    integer, intent(in) :: x
-    if (x == 1) then
-      call variant1()
-    else
-      call variant2()
-    end if
+  !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+  subroutine foo_dispatch()
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -40,33 +26,22 @@ end module funcs
 program dispatch_test
   use funcs
   implicit none
-  integer :: x
 
-  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
-  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
-  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 1
-  !HLFIR: omp.dispatch {
-  !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
-  !HLFIR:   omp.terminator
-  !HLFIR: }
+  ! A call outside any dispatch region targets the base procedure.
+  !HLFIR: fir.call @_QMfuncsPfoo_dispatch() {{.*}}: () -> ()
+  call foo_dispatch()
 
-  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
-  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 2
   !HLFIR: omp.dispatch {
   !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 
   !HLFIR: omp.dispatch nowait {
   !$omp dispatch nowait
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 end program
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c966784b1e22e..304753ed4bcc0 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -1,93 +1,63 @@
-// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+// RUN: fir-opt %s | FileCheck %s
 
-// CHECK-LABEL: func @foo_dispatch
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @foo_dispatch(%x : memref<i32>) -> () {
-  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
-  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
-  %v = memref.load %x[] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  %cmp = arith.cmpi eq, %v, %c1 : i32
-  cf.cond_br %cmp, ^bb1, ^bb2
-// CHECK: ^[[BB1]]:
-// CHECK: call @variant1() : () -> ()
-^bb1:
-  func.call @variant1() : () -> ()
-  cf.br ^bb3
-// CHECK: ^[[BB2]]:
-// CHECK: call @variant2() : () -> ()
-^bb2:
-  func.call @variant2() : () -> ()
-  cf.br ^bb3
-^bb3:
-  return
-}
-
-// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
-// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test multiple dispatch regions with stores selecting different variants.
-// TODO: Use declare_variant when the support is merged.
-// CHECK-LABEL: func @omp_dispatch_multiple
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  memref.store %c1, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
-  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
-  %c2 = arith.constant 2 : i32
-  memref.store %c2, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test the nowait clause on omp.dispatch.
-// CHECK-LABEL: func @omp_dispatch_nowait
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch nowait {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  omp.dispatch nowait {
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    omp.terminator
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr<270> = dense<32> : vector<4xi64>, !llvm.ptr<271> = dense<32> : vector<4xi64>, !llvm.ptr<272> = dense<64> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, i128 = dense<128> : vector<2xi64>, f80 = dense<128> : vector<2xi64>, !llvm.ptr = dense<64> : vector<4xi64>, i1 = dense<8> : vector<2xi64>, i8 = dense<8> : vector<2xi64>, i16 = dense<16> : vector<2xi64>, i32 = dense<32> : vector<2xi64>, f16 = dense<16> : vector<2xi64>, f64 = dense<64> : vector<2xi64>, f128 = dense<128> : vector<2xi64>, "dlti.endianness" = "little", "dlti.mangling_mode" = "e", "dlti.legal_int_widths" = array<i32: 8, 16, 32, 64>, "dlti.stack_alignment" = 128 : i64>, fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", fir.relocation_model = 1 : i32, llvm.data_layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128", llvm.ident = "flang version 24.0.0 (https://github.com/SunilKuravinakop/llvm-project.git 665c356fed252c9f65f99e7609997822e0e0a908)", llvm.target_triple = "x86_64-unknown-linux-gnu", omp.flags = #omp.flags<openmp_device_version = 31>, omp.is_gpu = false, omp.is_target_device = false, omp.target_triples = [], omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_variant
+  func.func @_QMfuncsPfoo_variant() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c35_i32 = arith.constant 35 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c35_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F76617269616E74) : !fir.ref<!fir.char<1,14>>
+    %c14 = arith.constant 14 : index
+    %5:2 = hlfir.declare %4 typeparams %c14 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F76617269616E74"} : (!fir.ref<!fir.char<1,14>>, index) -> (!fir.ref<!fir.char<1,14>>, !fir.ref<!fir.char<1,14>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,14>>) -> !fir.ref<i8>
+    %7 = fir.convert %c14 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_dispatch
+  func.func @_QMfuncsPfoo_dispatch() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c40_i32 = arith.constant 40 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c40_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F6469737061746368) : !fir.ref<!fir.char<1,15>>
+    %c15 = arith.constant 15 : index
+    %5:2 = hlfir.declare %4 typeparams %c15 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F6469737061746368"} : (!fir.ref<!fir.char<1,15>>, index) -> (!fir.ref<!fir.char<1,15>>, !fir.ref<!fir.char<1,15>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,15>>) -> !fir.ref<i8>
+    %7 = fir.convert %c15 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QQmain
+  func.func @_QQmain() attributes {fir.bindc_name = "DISPATCH_TEST"} {
+    %0 = fir.dummy_scope : !fir.dscope
+    // CHECK: fir.call @_QMfuncsPfoo_dispatch()
+    fir.call @_QMfuncsPfoo_dispatch() fastmath<contract> : () -> ()
+    // CHECK: omp.dispatch {
+    omp.dispatch {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
+  }
+  // CHECK-LABEL: func.func @omp_dispatch_nowait
+  func.func @omp_dispatch_nowait() {
+    // CHECK: omp.dispatch nowait {
+    omp.dispatch nowait {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
   }
-  return
 }
-
-// CHECK-LABEL: func private @variant1()
-// CHECK-LABEL: func private @variant2()
-func.func private @variant1() -> ()
-func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index d947a871ca30c..8a3f07f0ebbb9 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -1,82 +1,27 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
-llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
-  %0 = llvm.mlir.constant(0 : i32) : i32
-  llvm.return %0 : i32
-}
-
-llvm.func @variant1() -> ()
-llvm.func @variant2() -> ()
-
-// TODO: Add support for declare_variant, when support to it is merged.
-// CHECK-LABEL: define void @foo_dispatch()
-llvm.func @foo_dispatch() {
-  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
-  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
-  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %1 = llvm.load %0 : !llvm.ptr -> i32
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  %cmp = llvm.icmp "eq" %1, %c1 : i32
-  llvm.cond_br %cmp, ^bb1, ^bb2
-// CHECK: [[BB1]]:
-// CHECK: call void @variant1()
-^bb1:
-  llvm.call @variant1() : () -> ()
-  llvm.br ^bb3
-// CHECK: [[BB2]]:
-// CHECK: call void @variant2()
-^bb2:
-  llvm.call @variant2() : () -> ()
-  llvm.br ^bb3
-^bb3:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch()
-llvm.func @test_omp_dispatch() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+module attributes {omp.is_target_device = false, omp.is_gpu = false, omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: define void @_QMfuncsPfoo_variant()
+  llvm.func @_QMfuncsPfoo_variant() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch_multiple()
-llvm.func @test_omp_dispatch_multiple() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+  // CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+  llvm.func @_QMfuncsPfoo_dispatch() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  // CHECK: store i32 2, ptr @x
-  %c2 = llvm.mlir.constant(2 : i32) : i32
-  llvm.store %c2, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region2
-  omp.dispatch {
-    // CHECK: omp.dispatch.region2:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont1
-    omp.terminator
+  // CHECK-LABEL: define void @_QQmain()
+  llvm.func @_QQmain() {
+    // CHECK: call void @_QMfuncsPfoo_dispatch()
+    llvm.call @_QMfuncsPfoo_dispatch() : () -> ()
+    // CHECK: br label %omp.dispatch.region
+    // CHECK: omp.dispatch.region:
+    omp.dispatch {
+      // CHECK: call void @_QMfuncsPfoo_variant()
+      llvm.call @_QMfuncsPfoo_variant() : () -> ()
+      // CHECK: br label %omp.region.cont
+      omp.terminator
+    }
+    // CHECK: omp.region.cont:
+    llvm.return
   }
-  // CHECK: omp.region.cont1:
-  llvm.return
 }


From llvmlistbot at llvm.org  Fri Jul 24 00:30:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:30:05 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Support for "!$omp
	dispatch". (PR #203320)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203320@github.com>
Message-ID: <6a63147d.13be6df8.323b61.b36f@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/203320

>From d007fc287982ed638ef3900307ca700f0d529f87 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 10:56:08 -0500
Subject: [PATCH 1/7] Support for "!$omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 33 ++++++-
 flang/test/Integration/OpenMP/dispatch.f90    | 76 ++++++++++++++++
 flang/test/Lower/OpenMP/Todo/dispatch.f90     | 12 ---
 flang/test/Lower/OpenMP/dispatch.f90          | 89 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 15 ++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 ++++
 mlir/test/Dialect/OpenMP/dispatch.mlir        | 77 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  | 81 +++++++++++++++++
 8 files changed, 385 insertions(+), 15 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/dispatch.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90
 create mode 100644 flang/test/Lower/OpenMP/dispatch.f90
 create mode 100644 mlir/test/Dialect/OpenMP/dispatch.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-dispatch.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index be876c563433a..758d8eb8148e5 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2592,6 +2592,18 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -3940,6 +3952,9 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
   case llvm::omp::Directive::OMPD_barrier:
     newOp = genBarrierOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
+  case llvm::omp::Directive::OMPD_dispatch:
+    newOp = genDispatchOp(converter, symTable, semaCtx, eval, loc, queue, item);
+    break;
   case llvm::omp::Directive::OMPD_distribute:
     newOp = genStandaloneDistribute(converter, symTable, stmtCtx, semaCtx, eval,
                                     loc, queue, item);
@@ -5312,9 +5327,21 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
-                   const parser::OpenMPDispatchConstruct &) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPDispatchConstruct");
+                   const parser::OpenMPDispatchConstruct &dispatchConstruct) {
+  const parser::OmpDirectiveSpecification &beginSpec =
+      dispatchConstruct.BeginDir();
+  List<Clause> clauses = makeClauses(beginSpec.Clauses(), semaCtx);
+  if (auto &endSpec = dispatchConstruct.EndDir())
+    clauses.append(makeClauses(endSpec->Clauses(), semaCtx));
+
+  llvm::omp::Directive directive = beginSpec.DirId();
+  mlir::Location currentLocation = converter.genLocation(beginSpec.source);
+
+  ConstructQueue queue{
+      buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
+                          eval, beginSpec.source, directive, clauses)};
+  genOMPDispatch(converter, symTable, semaCtx, eval, currentLocation, queue,
+                 queue.begin());
 }
 
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..c4d16690c8e29
--- /dev/null
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -0,0 +1,76 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
+!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
+!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
+!CHECK: [[IF_TRUE]]:
+!CHECK: call void @_QMfuncsPfoo_variant2()
+!CHECK: [[IF_FALSE]]:
+!CHECK: call void @_QMfuncsPfoo_variant1()
+
+!CHECK-LABEL: define void @_QQmain()
+!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region
+!CHECK: omp.dispatch.region:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont
+!CHECK: omp.region.cont:
+!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region2
+!CHECK: omp.dispatch.region2:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont1
+!CHECK: omp.region.cont1:
+
+module funcs
+  implicit none
+  logical :: foo_sub
+
+contains
+
+  subroutine foo_variant1()
+    print *, "in foo_variant1"
+  end subroutine
+
+  subroutine foo_variant2()
+    print *, "in foo_variant2"
+  end subroutine
+
+  subroutine foo_dispatch()
+    if (foo_sub) then
+      call foo_variant2()
+    else
+      call foo_variant1()
+    end if
+  end subroutine
+
+end module funcs
+
+program dispatch_test
+  use funcs
+  implicit none
+
+  foo_sub = .false.
+  !$omp dispatch
+  call foo_dispatch()
+
+  foo_sub = .true.
+  !$omp dispatch
+  call foo_dispatch()
+
+end program
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
deleted file mode 100644
index 380dfa14eaae1..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ /dev/null
@@ -1,12 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPDispatchConstruct
-program p
-  integer r
-  r = 1
-!$omp dispatch nowait
-  call foo()
-contains
-  subroutine foo
-  end subroutine
-end program p
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..df86503817163
--- /dev/null
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -0,0 +1,89 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
+
+!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
+!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
+!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
+!HLFIR: fir.if %[[CMP_H]] {
+!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!HLFIR: } else {
+!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!HLFIR: }
+
+!FIR-LABEL: func @_QMfuncsPfoo_dispatch
+!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
+!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
+!FIR: fir.if %[[CMP_F]] {
+!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!FIR: } else {
+!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!FIR: }
+
+module funcs
+  implicit none
+
+contains
+
+  subroutine variant1()
+    print *, "in variant1"
+  end subroutine
+
+  subroutine variant2()
+    print *, "in variant2"
+  end subroutine
+
+  subroutine foo_dispatch(x)
+    integer, intent(in) :: x
+    if (x == 1) then
+      call variant1()
+    else
+      call variant2()
+    end if
+  end subroutine
+
+end module funcs
+
+!HLFIR-LABEL: func @_QQmain
+!FIR-LABEL: func @_QQmain
+!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
+!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
+!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+program dispatch_test
+  use funcs
+  implicit none
+  integer :: x
+
+  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
+  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
+  x = 1
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+
+  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
+  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
+  x = 2
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 51e7080db5b29..6170e40a835c6 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2286,6 +2286,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
   ];
 }
 
+//===----------------------------------------------------------------------===//
+// [Spec 5.1] 12.3 dispatch Construct
+//===----------------------------------------------------------------------===//
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+  let summary = "dispatch construct";
+  let description = [{
+    The dispatch construct enables the invocation of a variant of a
+    base procedure. The structured block of a dispatch construct is a
+    single expression statement that contains a function call or a
+    subroutine call.
+  }];
+
+  let assemblyFormat = "$region attr-dict";
+}
+
 //===----------------------------------------------------------------------===//
 // [Spec 5.2] 6.5 allocate Directive
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..12818a5e56ac6 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -761,6 +761,20 @@ static llvm::omp::ProcBindKind getProcBindKind(omp::ClauseProcBindKind kind) {
   llvm_unreachable("Unknown ClauseProcBindKind kind");
 }
 
+/// Convert 'dispatch' operation into LLVM IR.
+static LogicalResult
+convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  auto dispatchOp = cast<omp::DispatchOp>(opInst);
+  auto &region = dispatchOp.getRegion();
+  auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
+                                    moduleTranslation);
+  if (!result)
+    return handleError(result.takeError(), opInst);
+  builder.SetInsertPoint(*result);
+  return success();
+}
+
 /// Converts an OpenMP 'masked' operation into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpMasked(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -8815,6 +8829,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })
+          .Case([&](omp::DispatchOp) {
+            return convertOmpDispatch(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::MaskedOp) {
             return convertOmpMasked(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
new file mode 100644
index 0000000000000..c935f56836185
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: func @foo_dispatch
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @foo_dispatch(%x : memref<i32>) -> () {
+  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
+  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
+  %v = memref.load %x[] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  %cmp = arith.cmpi eq, %v, %c1 : i32
+  cf.cond_br %cmp, ^bb1, ^bb2
+// CHECK: ^[[BB1]]:
+// CHECK: call @variant1() : () -> ()
+^bb1:
+  func.call @variant1() : () -> ()
+  cf.br ^bb3
+// CHECK: ^[[BB2]]:
+// CHECK: call @variant2() : () -> ()
+^bb2:
+  func.call @variant2() : () -> ()
+  cf.br ^bb3
+^bb3:
+  return
+}
+
+// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
+// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// Test multiple dispatch regions with stores selecting different variants.
+// CHECK-LABEL: func @omp_dispatch_multiple
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  memref.store %c1, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
+  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
+  %c2 = arith.constant 2 : i32
+  memref.store %c2, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// CHECK-LABEL: func private @variant1()
+// CHECK-LABEL: func private @variant2()
+func.func private @variant1() -> ()
+func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
new file mode 100644
index 0000000000000..8a3a7a6b5d8e6
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -0,0 +1,81 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.return %0 : i32
+}
+
+llvm.func @variant1() -> ()
+llvm.func @variant2() -> ()
+
+// CHECK-LABEL: define void @foo_dispatch()
+llvm.func @foo_dispatch() {
+  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
+  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
+  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %1 = llvm.load %0 : !llvm.ptr -> i32
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  %cmp = llvm.icmp "eq" %1, %c1 : i32
+  llvm.cond_br %cmp, ^bb1, ^bb2
+// CHECK: [[BB1]]:
+// CHECK: call void @variant1()
+^bb1:
+  llvm.call @variant1() : () -> ()
+  llvm.br ^bb3
+// CHECK: [[BB2]]:
+// CHECK: call void @variant2()
+^bb2:
+  llvm.call @variant2() : () -> ()
+  llvm.br ^bb3
+^bb3:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch()
+llvm.func @test_omp_dispatch() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch_multiple()
+llvm.func @test_omp_dispatch_multiple() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  // CHECK: store i32 2, ptr @x
+  %c2 = llvm.mlir.constant(2 : i32) : i32
+  llvm.store %c2, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region2
+  omp.dispatch {
+    // CHECK: omp.dispatch.region2:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont1
+    omp.terminator
+  }
+  // CHECK: omp.region.cont1:
+  llvm.return
+}

>From b50c11a322615d4e189303f9b4b73e1c6246c7ae Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 12:19:36 -0500
Subject: [PATCH 2/7] Checking for no caluses support with "omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 3 +++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 758d8eb8148e5..71bc392d0e8d9 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -5337,6 +5337,9 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
+  if (!clauses.empty())
+    TODO(currentLocation, "OpenMP Dispatch clauses");
+
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 6170e40a835c6..47c7587c5d08b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2293,9 +2293,10 @@ def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure. The structured block of a dispatch construct is a
-    single expression statement that contains a function call or a
-    subroutine call.
+    base procedure (without any construct's clauses: device, is_device_ptr,
+    nowait, nocontext, novariants). The structured block of a dispatch
+    construct is a single expression statement that contains a function
+    call or a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";

>From 3ca936ebb5aff08e318fcd4abede5951e77fcb07 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 18 Jun 2026 11:50:04 -0500
Subject: [PATCH 3/7] Handling feedbacks suggested by Sergio Afonso.

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 37 ++++++++++++-------
 flang/test/Lower/OpenMP/dispatch.f90          | 26 +------------
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 11 +++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  4 ++
 mlir/test/Dialect/OpenMP/dispatch.mlir        |  1 +
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  |  1 +
 6 files changed, 36 insertions(+), 44 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 33e67ed2c3ab7..fe7d17807727a 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1661,6 +1661,16 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
   cp.processOrder(clauseOps);
 }
 
+static void genDispatchClauses(lower::AbstractConverter &converter,
+                               semantics::SemanticsContext &semaCtx,
+                               const List<Clause> &clauses,
+                               mlir::Location loc) {
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
+                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+      loc, llvm::omp::Directive::OMPD_dispatch);
+}
+
 static void genFlushClauses(lower::AbstractConverter &converter,
                             semantics::SemanticsContext &semaCtx,
                             const ObjectList &objects,
@@ -2154,6 +2164,18 @@ genCriticalOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item, nameAttr);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::FlushOp
 genFlushOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
            semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -2592,18 +2614,6 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
-static mlir::omp::DispatchOp
-genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-              semantics::SemanticsContext &semaCtx,
-              lower::pft::Evaluation &eval, mlir::Location loc,
-              const ConstructQueue &queue,
-              ConstructQueue::const_iterator item) {
-  return genOpWithBody<mlir::omp::DispatchOp>(
-      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
-                        llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
-}
-
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -5294,8 +5304,7 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  if (!clauses.empty())
-    TODO(currentLocation, "OpenMP Dispatch clauses");
+  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
 
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index df86503817163..ecc7b1b31454f 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,5 +1,4 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
-!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
 
 !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
 !HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
@@ -12,16 +11,6 @@
 !HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
 !HLFIR: }
 
-!FIR-LABEL: func @_QMfuncsPfoo_dispatch
-!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
-!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
-!FIR: fir.if %[[CMP_F]] {
-!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!FIR: } else {
-!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!FIR: }
-
 module funcs
   implicit none
 
@@ -35,6 +24,7 @@ subroutine variant2()
     print *, "in variant2"
   end subroutine
 
+  !TODO: replace with declare_variant when the support is merged.
   subroutine foo_dispatch(x)
     integer, intent(in) :: x
     if (x == 1) then
@@ -47,10 +37,6 @@ subroutine foo_dispatch(x)
 end module funcs
 
 !HLFIR-LABEL: func @_QQmain
-!FIR-LABEL: func @_QQmain
-!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
-!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
-!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
 program dispatch_test
   use funcs
   implicit none
@@ -59,31 +45,21 @@ program dispatch_test
   !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
   !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
   !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
   x = 1
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 
   !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
   !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
   x = 2
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 47c7587c5d08b..490bee24799c2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,14 +2289,15 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
+   ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure (without any construct's clauses: device, is_device_ptr,
-    nowait, nocontext, novariants). The structured block of a dispatch
-    construct is a single expression statement that contains a function
-    call or a subroutine call.
+    base procedure. The structured block of a dispatch construct is
+    a single expression statement that contains a function call or
+    a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 12818a5e56ac6..04c36b87d73ad 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -766,6 +766,10 @@ static LogicalResult
 convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
                    LLVM::ModuleTranslation &moduleTranslation) {
   auto dispatchOp = cast<omp::DispatchOp>(opInst);
+
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+
   auto &region = dispatchOp.getRegion();
   auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
                                     moduleTranslation);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c935f56836185..b8d4b93a6c8e3 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -41,6 +41,7 @@ func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
 }
 
 // Test multiple dispatch regions with stores selecting different variants.
+// TODO: Use declare_variant when the support is merged.
 // CHECK-LABEL: func @omp_dispatch_multiple
 // CHECK-SAME: (%[[X:.*]]: memref<i32>)
 func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index 8a3a7a6b5d8e6..d947a871ca30c 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -8,6 +8,7 @@ llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
 llvm.func @variant1() -> ()
 llvm.func @variant2() -> ()
 
+// TODO: Add support for declare_variant, when support to it is merged.
 // CHECK-LABEL: define void @foo_dispatch()
 llvm.func @foo_dispatch() {
   // CHECK: %[[ADDR:.*]] = load i32, ptr @x

>From 6eacbc3cd3d8e278d7b51ea898fe27e009de2228 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 01:09:09 -0500
Subject: [PATCH 4/7] 1) Handling -fc1 -fopenmp-simd 2) Todo testcase for
 clauses not supported in dispatch

---
 flang/lib/Semantics/rewrite-parse-tree.cpp |  5 +++
 flang/test/Lower/OpenMP/Todo/dispatch.f90  | 43 ++++++++++++++++++++++
 flang/test/Semantics/OpenMP/simd-only.f90  | 12 ++++++
 3 files changed, 60 insertions(+)
 create mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90

diff --git a/flang/lib/Semantics/rewrite-parse-tree.cpp b/flang/lib/Semantics/rewrite-parse-tree.cpp
index 4e1c9bae9c153..76a60c75488e3 100644
--- a/flang/lib/Semantics/rewrite-parse-tree.cpp
+++ b/flang/lib/Semantics/rewrite-parse-tree.cpp
@@ -191,6 +191,11 @@ void RewriteMutator::OpenMPSimdOnly(
                        &omp->value().u)}) {
           it = replaceInlineBlock(std::get<parser::Block>(ompBlock->t), it);
           continue;
+        } else if (auto *ompDispatch{
+                       std::get_if<parser::OpenMPDispatchConstruct>(
+                           &omp->value().u)}) {
+          it = replaceInlineBlock(std::get<parser::Block>(ompDispatch->t), it);
+          continue;
         } else if (auto *ompLoop{std::get_if<parser::OpenMPLoopConstruct>(
                        &omp->value().u)}) {
           if (LoopConstructIsSIMD(ompLoop)) {
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
new file mode 100644
index 0000000000000..f100cb40ba3d7
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -0,0 +1,43 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEPEND -o - %s 2>&1 | FileCheck %s --check-prefix=DEPEND
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEVICE -o - %s 2>&1 | FileCheck %s --check-prefix=DEVICE
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
+
+! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
+! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
+! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
+! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
+! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
+! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
+
+subroutine sub()
+#ifdef IS_DEVICE_PTR
+  use iso_c_binding
+  type(c_ptr) :: x
+#endif
+  integer :: r
+#ifdef DEPEND
+!$omp dispatch depend(inout: r)
+#endif
+#ifdef DEVICE
+!$omp dispatch device(0)
+#endif
+#ifdef IS_DEVICE_PTR
+!$omp dispatch is_device_ptr(x)
+#endif
+#ifdef NOCONTEXT
+!$omp dispatch nocontext(.false.)
+#endif
+#ifdef NOVARIANTS
+!$omp dispatch novariants(.true.)
+#endif
+#ifdef NOWAIT
+!$omp dispatch nowait
+#endif
+  call foo()
+contains
+  subroutine foo
+  end subroutine
+end subroutine sub
diff --git a/flang/test/Semantics/OpenMP/simd-only.f90 b/flang/test/Semantics/OpenMP/simd-only.f90
index 01370af0a00a7..860befaf43247 100644
--- a/flang/test/Semantics/OpenMP/simd-only.f90
+++ b/flang/test/Semantics/OpenMP/simd-only.f90
@@ -414,3 +414,15 @@ module test_declare_mapper
   ! CHECK-NOT: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpDeclareMapperDirective
   !$omp declare mapper(myvec_t :: v) map(v, v%data(1:v%len))
 end module
+
+! CHECK-LABEL: Name = 'test_dispatch'
+subroutine test_dispatch()
+  ! CHECK-NOT: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OpenMPDispatchConstruct
+  ! CHECK-NOT: OmpDirectiveName -> llvm::omp::Directive = dispatch
+  ! CHECK: ExecutionPartConstruct -> ExecutableConstruct -> ActionStmt -> CallStmt
+  !$omp dispatch
+  call foo()
+contains
+  subroutine foo()
+  end subroutine
+end subroutine

>From 8f25901612c660e29e53ab9c7e162a21a04ad966 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 07:14:38 -0500
Subject: [PATCH 5/7] Support for "dispatch nowait" except for MLIR to LLVMIR.
 Support in LLVM IR depends on "declare variant".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp                 | 14 ++++++++------
 flang/test/Lower/OpenMP/Todo/dispatch.f90         |  5 -----
 flang/test/Lower/OpenMP/dispatch.f90              |  7 +++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td     | 12 ++++++++----
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp      |  9 +++++++++
 .../Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  |  1 +
 mlir/test/Dialect/OpenMP/dispatch.mlir            | 15 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir          | 11 +++++++++++
 8 files changed, 59 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 4c2a238605f8d..959f7352e2c76 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1774,11 +1774,12 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
 
 static void genDispatchClauses(lower::AbstractConverter &converter,
                                semantics::SemanticsContext &semaCtx,
-                               const List<Clause> &clauses,
-                               mlir::Location loc) {
+                               const List<Clause> &clauses, mlir::Location loc,
+                               mlir::omp::DispatchOperands &clauseOps) {
   ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processNowait(clauseOps);
   cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
-                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+                 clause::Nocontext, clause::Novariants>(
       loc, llvm::omp::Directive::OMPD_dispatch);
 }
 
@@ -2276,10 +2277,13 @@ genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
               lower::pft::Evaluation &eval, mlir::Location loc,
               const ConstructQueue &queue,
               ConstructQueue::const_iterator item) {
+  mlir::omp::DispatchOperands clauseOps;
+  genDispatchClauses(converter, semaCtx, item->clauses, loc, clauseOps);
+
   return genOpWithBody<mlir::omp::DispatchOp>(
       OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
                         llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
+      queue, item, clauseOps);
 }
 
 static mlir::omp::FlushOp
@@ -5426,8 +5430,6 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
-
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
index f100cb40ba3d7..579d47d457def 100644
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -3,14 +3,12 @@
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
 
 ! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
 ! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
 ! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
 ! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
 ! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
-! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
 
 subroutine sub()
 #ifdef IS_DEVICE_PTR
@@ -32,9 +30,6 @@ subroutine sub()
 #endif
 #ifdef NOVARIANTS
 !$omp dispatch novariants(.true.)
-#endif
-#ifdef NOWAIT
-!$omp dispatch nowait
 #endif
   call foo()
 contains
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index ecc7b1b31454f..1b184a18a91af 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -62,4 +62,11 @@ program dispatch_test
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
   !HLFIR: }
+
+  !HLFIR: omp.dispatch nowait {
+  !$omp dispatch nowait
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !HLFIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index f048a9da71063..bb478c52381c5 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,17 +2289,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
 def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
-     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
-   ]> {
+    // TODO: Complete clause list (device, depend, is_device_ptr, nocontext,
+    // novariants).
+    OpenMP_NowaitClause
+  ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
     base procedure. The structured block of a dispatch construct is
     a single expression statement that contains a function call or
     a subroutine call.
-  }];
+  }] # clausesDescription;
 
-  let assemblyFormat = "$region attr-dict";
+  let builders = [
+    OpBuilder<(ins CArg<"const DispatchOperands &">:$clauses)>
+  ];
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index c2956a1cf7b79..6c1d5c29ec95d 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4933,6 +4933,15 @@ void MaskedOp::build(OpBuilder &builder, OperationState &state,
   MaskedOp::build(builder, state, clauses.filteredThreadId);
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Dispatch construct (12.3)
+//===----------------------------------------------------------------------===//
+
+void DispatchOp::build(OpBuilder &builder, OperationState &state,
+                       const DispatchOperands &clauses) {
+  DispatchOp::build(builder, state, clauses.nowait);
+}
+
 //===----------------------------------------------------------------------===//
 // Spec 5.2: Scan construct (5.6)
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e84daaa135748..75c5e707b9933 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -480,6 +480,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkDepend(op, result);
         checkNowait(op, result);
       })
+      .Case([&](omp::DispatchOp op) { checkNowait(op, result); })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
         checkInReductionByref(op, result);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index b8d4b93a6c8e3..c966784b1e22e 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -72,6 +72,21 @@ func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
   return
 }
 
+// Test the nowait clause on omp.dispatch.
+// CHECK-LABEL: func @omp_dispatch_nowait
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch nowait {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  omp.dispatch nowait {
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    omp.terminator
+  }
+  return
+}
+
 // CHECK-LABEL: func private @variant1()
 // CHECK-LABEL: func private @variant2()
 func.func private @variant1() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 377a5bb799be4..1bf9b563f8d6c 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -52,6 +52,17 @@ llvm.func @distribute_order(%lb : i32, %ub : i32, %step : i32) {
 
 // -----
 
+llvm.func @dispatch_nowait() {
+  // expected-error at below {{not yet implemented: Unhandled clause nowait in omp.dispatch operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.dispatch}}
+  omp.dispatch nowait {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @parallel_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.parallel operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.parallel}}

>From 7739eecf4c336629aedfa11911a4bea1cc81a6e8 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 02:20:10 -0500
Subject: [PATCH 6/7] Integrating variants using "declare variant" in the
 tests. Adding dispatch trait to "declare variant" in Utils.cpp Handling
 comments by Chi Chun Chen.

---
 flang/lib/Lower/OpenMP/Utils.cpp             |   3 +
 flang/test/Integration/OpenMP/dispatch.f90   |  42 +-----
 flang/test/Lower/OpenMP/dispatch.f90         |  59 +++-----
 mlir/test/Dialect/OpenMP/dispatch.mlir       | 150 ++++++++-----------
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir |  97 +++---------
 5 files changed, 109 insertions(+), 242 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Utils.cpp b/flang/lib/Lower/OpenMP/Utils.cpp
index 8f57f00d59c58..5cb6a0713462f 100644
--- a/flang/lib/Lower/OpenMP/Utils.cpp
+++ b/flang/lib/Lower/OpenMP/Utils.cpp
@@ -1379,6 +1379,9 @@ void collectEnclosingConstructTraits(
   for (; op; op = op->getParentOp()) {
     if (mlir::isa<mlir::omp::WsloopOp>(op))
       constructTraits.push_back(llvm::omp::TraitProperty::construct_for_for);
+    if (mlir::isa<mlir::omp::DispatchOp>(op))
+      constructTraits.push_back(
+          llvm::omp::TraitProperty::construct_dispatch_dispatch);
     if (mlir::isa<mlir::omp::ParallelOp>(op))
       constructTraits.push_back(
           llvm::omp::TraitProperty::construct_parallel_parallel);
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
index c4d16690c8e29..1ecccab5e90d4 100644
--- a/flang/test/Integration/OpenMP/dispatch.f90
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -8,55 +8,32 @@
 
 !RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
 
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
-!CHECK: call ptr @_FortranAioBeginExternalListOutput
-
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant()
 !CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
-!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
-!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
-!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
-!CHECK: [[IF_TRUE]]:
-!CHECK: call void @_QMfuncsPfoo_variant2()
-!CHECK: [[IF_FALSE]]:
-!CHECK: call void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QQmain()
-!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: call void @_QMfuncsPfoo_dispatch()
 !CHECK: br label %omp.dispatch.region
 !CHECK: omp.dispatch.region:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: call void @_QMfuncsPfoo_variant()
 !CHECK: br label %omp.region.cont
 !CHECK: omp.region.cont:
-!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
-!CHECK: br label %omp.dispatch.region2
-!CHECK: omp.dispatch.region2:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
-!CHECK: br label %omp.region.cont1
-!CHECK: omp.region.cont1:
 
 module funcs
   implicit none
-  logical :: foo_sub
 
 contains
 
-  subroutine foo_variant1()
-    print *, "in foo_variant1"
-  end subroutine
-
-  subroutine foo_variant2()
-    print *, "in foo_variant2"
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
   subroutine foo_dispatch()
-    if (foo_sub) then
-      call foo_variant2()
-    else
-      call foo_variant1()
-    end if
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -65,11 +42,8 @@ program dispatch_test
   use funcs
   implicit none
 
-  foo_sub = .false.
-  !$omp dispatch
   call foo_dispatch()
 
-  foo_sub = .true.
   !$omp dispatch
   call foo_dispatch()
 
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index 1b184a18a91af..4eb9439b7b35b 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,37 +1,23 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
 
-!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
-!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
-!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
-!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
-!HLFIR: fir.if %[[CMP_H]] {
-!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!HLFIR: } else {
-!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!HLFIR: }
+! Variant selection is provided by DECLARE VARIANT with a `construct={dispatch}`
+! match: inside a dispatch region the call to the base procedure `foo_dispatch`
+! is replaced by a call to its variant `foo_variant`.
 
 module funcs
   implicit none
 
 contains
 
-  subroutine variant1()
-    print *, "in variant1"
+  !HLFIR-LABEL: func @_QMfuncsPfoo_variant
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
-  subroutine variant2()
-    print *, "in variant2"
-  end subroutine
-
-  !TODO: replace with declare_variant when the support is merged.
-  subroutine foo_dispatch(x)
-    integer, intent(in) :: x
-    if (x == 1) then
-      call variant1()
-    else
-      call variant2()
-    end if
+  !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+  subroutine foo_dispatch()
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -40,33 +26,22 @@ end module funcs
 program dispatch_test
   use funcs
   implicit none
-  integer :: x
 
-  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
-  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
-  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 1
-  !HLFIR: omp.dispatch {
-  !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
-  !HLFIR:   omp.terminator
-  !HLFIR: }
+  ! A call outside any dispatch region targets the base procedure.
+  !HLFIR: fir.call @_QMfuncsPfoo_dispatch() {{.*}}: () -> ()
+  call foo_dispatch()
 
-  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
-  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 2
   !HLFIR: omp.dispatch {
   !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 
   !HLFIR: omp.dispatch nowait {
   !$omp dispatch nowait
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 end program
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c966784b1e22e..304753ed4bcc0 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -1,93 +1,63 @@
-// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+// RUN: fir-opt %s | FileCheck %s
 
-// CHECK-LABEL: func @foo_dispatch
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @foo_dispatch(%x : memref<i32>) -> () {
-  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
-  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
-  %v = memref.load %x[] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  %cmp = arith.cmpi eq, %v, %c1 : i32
-  cf.cond_br %cmp, ^bb1, ^bb2
-// CHECK: ^[[BB1]]:
-// CHECK: call @variant1() : () -> ()
-^bb1:
-  func.call @variant1() : () -> ()
-  cf.br ^bb3
-// CHECK: ^[[BB2]]:
-// CHECK: call @variant2() : () -> ()
-^bb2:
-  func.call @variant2() : () -> ()
-  cf.br ^bb3
-^bb3:
-  return
-}
-
-// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
-// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test multiple dispatch regions with stores selecting different variants.
-// TODO: Use declare_variant when the support is merged.
-// CHECK-LABEL: func @omp_dispatch_multiple
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  memref.store %c1, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
-  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
-  %c2 = arith.constant 2 : i32
-  memref.store %c2, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test the nowait clause on omp.dispatch.
-// CHECK-LABEL: func @omp_dispatch_nowait
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch nowait {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  omp.dispatch nowait {
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    omp.terminator
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr<270> = dense<32> : vector<4xi64>, !llvm.ptr<271> = dense<32> : vector<4xi64>, !llvm.ptr<272> = dense<64> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, i128 = dense<128> : vector<2xi64>, f80 = dense<128> : vector<2xi64>, !llvm.ptr = dense<64> : vector<4xi64>, i1 = dense<8> : vector<2xi64>, i8 = dense<8> : vector<2xi64>, i16 = dense<16> : vector<2xi64>, i32 = dense<32> : vector<2xi64>, f16 = dense<16> : vector<2xi64>, f64 = dense<64> : vector<2xi64>, f128 = dense<128> : vector<2xi64>, "dlti.endianness" = "little", "dlti.mangling_mode" = "e", "dlti.legal_int_widths" = array<i32: 8, 16, 32, 64>, "dlti.stack_alignment" = 128 : i64>, fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", fir.relocation_model = 1 : i32, llvm.data_layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128", llvm.ident = "flang version 24.0.0 (https://github.com/SunilKuravinakop/llvm-project.git 665c356fed252c9f65f99e7609997822e0e0a908)", llvm.target_triple = "x86_64-unknown-linux-gnu", omp.flags = #omp.flags<openmp_device_version = 31>, omp.is_gpu = false, omp.is_target_device = false, omp.target_triples = [], omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_variant
+  func.func @_QMfuncsPfoo_variant() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c35_i32 = arith.constant 35 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c35_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F76617269616E74) : !fir.ref<!fir.char<1,14>>
+    %c14 = arith.constant 14 : index
+    %5:2 = hlfir.declare %4 typeparams %c14 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F76617269616E74"} : (!fir.ref<!fir.char<1,14>>, index) -> (!fir.ref<!fir.char<1,14>>, !fir.ref<!fir.char<1,14>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,14>>) -> !fir.ref<i8>
+    %7 = fir.convert %c14 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_dispatch
+  func.func @_QMfuncsPfoo_dispatch() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c40_i32 = arith.constant 40 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c40_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F6469737061746368) : !fir.ref<!fir.char<1,15>>
+    %c15 = arith.constant 15 : index
+    %5:2 = hlfir.declare %4 typeparams %c15 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F6469737061746368"} : (!fir.ref<!fir.char<1,15>>, index) -> (!fir.ref<!fir.char<1,15>>, !fir.ref<!fir.char<1,15>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,15>>) -> !fir.ref<i8>
+    %7 = fir.convert %c15 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QQmain
+  func.func @_QQmain() attributes {fir.bindc_name = "DISPATCH_TEST"} {
+    %0 = fir.dummy_scope : !fir.dscope
+    // CHECK: fir.call @_QMfuncsPfoo_dispatch()
+    fir.call @_QMfuncsPfoo_dispatch() fastmath<contract> : () -> ()
+    // CHECK: omp.dispatch {
+    omp.dispatch {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
+  }
+  // CHECK-LABEL: func.func @omp_dispatch_nowait
+  func.func @omp_dispatch_nowait() {
+    // CHECK: omp.dispatch nowait {
+    omp.dispatch nowait {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
   }
-  return
 }
-
-// CHECK-LABEL: func private @variant1()
-// CHECK-LABEL: func private @variant2()
-func.func private @variant1() -> ()
-func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index d947a871ca30c..8a3f07f0ebbb9 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -1,82 +1,27 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
-llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
-  %0 = llvm.mlir.constant(0 : i32) : i32
-  llvm.return %0 : i32
-}
-
-llvm.func @variant1() -> ()
-llvm.func @variant2() -> ()
-
-// TODO: Add support for declare_variant, when support to it is merged.
-// CHECK-LABEL: define void @foo_dispatch()
-llvm.func @foo_dispatch() {
-  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
-  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
-  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %1 = llvm.load %0 : !llvm.ptr -> i32
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  %cmp = llvm.icmp "eq" %1, %c1 : i32
-  llvm.cond_br %cmp, ^bb1, ^bb2
-// CHECK: [[BB1]]:
-// CHECK: call void @variant1()
-^bb1:
-  llvm.call @variant1() : () -> ()
-  llvm.br ^bb3
-// CHECK: [[BB2]]:
-// CHECK: call void @variant2()
-^bb2:
-  llvm.call @variant2() : () -> ()
-  llvm.br ^bb3
-^bb3:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch()
-llvm.func @test_omp_dispatch() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+module attributes {omp.is_target_device = false, omp.is_gpu = false, omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: define void @_QMfuncsPfoo_variant()
+  llvm.func @_QMfuncsPfoo_variant() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch_multiple()
-llvm.func @test_omp_dispatch_multiple() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+  // CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+  llvm.func @_QMfuncsPfoo_dispatch() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  // CHECK: store i32 2, ptr @x
-  %c2 = llvm.mlir.constant(2 : i32) : i32
-  llvm.store %c2, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region2
-  omp.dispatch {
-    // CHECK: omp.dispatch.region2:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont1
-    omp.terminator
+  // CHECK-LABEL: define void @_QQmain()
+  llvm.func @_QQmain() {
+    // CHECK: call void @_QMfuncsPfoo_dispatch()
+    llvm.call @_QMfuncsPfoo_dispatch() : () -> ()
+    // CHECK: br label %omp.dispatch.region
+    // CHECK: omp.dispatch.region:
+    omp.dispatch {
+      // CHECK: call void @_QMfuncsPfoo_variant()
+      llvm.call @_QMfuncsPfoo_variant() : () -> ()
+      // CHECK: br label %omp.region.cont
+      omp.terminator
+    }
+    // CHECK: omp.region.cont:
+    llvm.return
   }
-  // CHECK: omp.region.cont1:
-  llvm.return
 }

>From d515daa0dbde17522438d7cd8b095e553c2306db Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 02:28:33 -0500
Subject: [PATCH 7/7] Handling Chi Chun Chen comments on "NoRegionArguments"

---
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index b8fc6174f9776..93169976e2c8b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2328,11 +2328,13 @@ def MaskedOp : OpenMP_Op<"masked", traits = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+def DispatchOp : OpenMP_Op<"dispatch", traits = [
+    NoRegionArguments
+  ], clauses = [
     // TODO: Complete clause list (device, depend, is_device_ptr, nocontext,
     // novariants).
     OpenMP_NowaitClause
-  ]> {
+  ], singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a


From llvmlistbot at llvm.org  Fri Jul 24 00:32:07 2026
From: llvmlistbot at llvm.org (Felipe de Azevedo Piovezan)
Date: Fri, 24 Jul 2026 00:32:07 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a6314f7.92ea0e2a.2ed880.fd9d@mx.google.com>


felipepiovezan wrote:

> It's also not quite clear to me why RFC is necessary for this particular patch, or is it more general RFC to talk about how overall procedures will be implemented?

Well, there are clear design decisions that were made unilaterally here, and there are valid open questions about future direction. We can keep iterating on those over github, but that's not the best forum.

> What would be the benefit of splitting the patch?

Easier to review, easier to revert, easier to see how things are decoupled. This is the same question of "why are smaller patches better?", no?

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Fri Jul 24 00:33:57 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 00:33:57 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a631565.92ea0e2a.2ed880.3dae@mx.google.com>


LouisLu060211 wrote:

> Hi @LouisLu060211,
> 
> Thank you for the patch! Unfortunately, I do not think Vector should move in this direction.
> 
> From your summary:
> 
> > For example, vector<16xi3> to vector<3xi16> has 48 bits on both sides, but 16 is not divisible by 3.
> 
> It sounds like you are looking for logic to model sub-byte types in Vector. There is already [VectorEmulateNarrowType](https://github.com/llvm/llvm-project/blob/0b5c006208cf7b0fee3709f468152e7de81607ef/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp) for this purpose. While that pass works, it is admittedly somewhat fragile. In my opinion, this is because we never fully settled whether it should support arbitrary integer widths such as `i3`.
> 
> In hindsight, I think `VectorEmulateNarrowType` should not support such exotic types. The underlying logic is very complex, difficult to support, and there does not seem to be enough interest in maintaining it beyond very narrow use cases. In other words, the current support is not very general.
> 
> I also question whether `i3` is a type we should support beyond the most basic examples. I appreciate that some users may want this, but supporting it generally would come at a significant maintenance cost. IMHO, we should instead prioritise the more common cases that have more users and reviewers.
> 
> Is this a major blocker for you? If we start unblocking cases like this, I am worried that we will also need to review many other Vector transformations, such as unrolling and flattening, to make sure they bail out cleanly. That would be a substantial effort, and I would prefer to avoid it at this point.
> 
> If you have broader plans for sub-byte types, I would be happy to hear them.
> 
> Thank you!

@joker-eph After reading your comment. The verifier can actually accept this bitcast because the total bit widths match. So the crash comes from that gap. If the correct fix is to tighten the verifier so cases like vector<16xi3> to vector<3xi16>. Feel free to let me know again, and I can rework the PR.


https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 00:39:23 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 00:39:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6316ab.d1496011.3cfe19.bb37@mx.google.com>


https://github.com/aengelke edited https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 00:39:23 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 00:39:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6316ab.6c8668f2.1842d3.2ca7@mx.google.com>


https://github.com/aengelke commented:

Probably [this c-t-t test](https://llvm-compile-time-tracker.com/compare.php?from=9c52b735ea04f58c04ac36c162ed8e45412287b3&to=5dc86c4710dee1dbd9e576e5846b37081e8aef3a&stat=instructions:u); minor barely significant improvement for a fair amount of complexity. Primary benefit is removing PopulateLoopDFS and reducing the size of deeply nested loops when unmodified.

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 00:39:24 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 00:39:24 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6316ac.666420a4.3304c7.0074@mx.google.com>


================
@@ -62,8 +61,12 @@ template <class BlockT, class LoopT> class LoopBase {
   // Loops contained entirely within this one.
   std::vector<LoopT *> SubLoops;
 
-  // The list of blocks in this loop. First entry is the header node.
-  std::vector<BlockT *> Blocks;
+  // The list of blocks in this loop; first entry is the header. Either borrows
+  // from the owning LoopInfo's BlockLayout with BlockCapacity == BlockLen, or
+  // is a private allocation of BlockCapacity slots from its allocator.
+  BlockT **BlockData = nullptr;
----------------
aengelke wrote:

union instead of reinterpret_cast

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 00:39:24 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 00:39:24 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6316ac.f06f4994.50b1b.feed@mx.google.com>


================
@@ -636,6 +648,49 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
+  /// AllocateLoop for analyze(): stash \p Header (see pendingHeader).
+  /// getHeader() only works once the layout carve has replaced the stash with
+  /// the loop's block list.
+  LoopT *allocateLoop(BlockT *Header) {
+    LoopT *L = AllocateLoop();
+    L->BlockData = reinterpret_cast<BlockT **>(Header);
+    return L;
+  }
+
+  /// The header of a loop under construction, stashed in BlockData until the
+  /// layout carve builds the block list.
+  static BlockT *pendingHeader(const LoopT *L) {
+    return reinterpret_cast<BlockT *>(L->BlockData);
+  }
+
+  void discoverAndMapSubloop(LoopT *L, BlockT *Header,
+                             ArrayRef<BlockT *> Backedges,
+                             const DominatorTreeBase<BlockT, false> &DomTree);
+
+  /// True if \p L's block list is a borrowed slice of BlockLayout rather than
+  /// a private allocation.
+  bool hasBorrowedBlocks(const LoopT &L) const {
+    return L.BlockData >= BlockLayout.get() &&
+           L.BlockData < BlockLayout.get() + BlockLayoutLen;
----------------
aengelke wrote:

This comparison is UB if not borrowed?

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 01:05:33 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 01:05:33 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a631ccd.4a342cbc.1aeb63.944a@mx.google.com>


LouisLu060211 wrote:

> Hi @LouisLu060211,
> 
> Thank you for the patch! Unfortunately, I do not think Vector should move in this direction.
> 
> From your summary:
> 
> > For example, vector<16xi3> to vector<3xi16> has 48 bits on both sides, but 16 is not divisible by 3.
> 
> It sounds like you are looking for logic to model sub-byte types in Vector. There is already [VectorEmulateNarrowType](https://github.com/llvm/llvm-project/blob/0b5c006208cf7b0fee3709f468152e7de81607ef/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp) for this purpose. While that pass works, it is admittedly somewhat fragile. In my opinion, this is because we never fully settled whether it should support arbitrary integer widths such as `i3`.
> 
> In hindsight, I think `VectorEmulateNarrowType` should not support such exotic types. The underlying logic is very complex, difficult to support, and there does not seem to be enough interest in maintaining it beyond very narrow use cases. In other words, the current support is not very general.
> 
> I also question whether `i3` is a type we should support beyond the most basic examples. I appreciate that some users may want this, but supporting it generally would come at a significant maintenance cost. IMHO, we should instead prioritise the more common cases that have more users and reviewers.
> 
> Is this a major blocker for you? If we start unblocking cases like this, I am worried that we will also need to review many other Vector transformations, such as unrolling and flattening, to make sure they bail out cleanly. That would be a substantial effort, and I would prefer to avoid it at this point.
> 
> If you have broader plans for sub-byte types, I would be happy to hear them.
> 
> Thank you!

@banach-space Thank you for the explanation.I found the crash through issue #206090, and I do not have broader plans for sub-byte types or a real use case for i3. I understand the maintenance concern. If the preferred direction is to reject these cases in the verifier instead, as joker-eph suggested, I am happy to rework the PR that way, or close this one if you would rather not go down that path.

> Hi @LouisLu060211,
> 
> Thank you for the patch! Unfortunately, I do not think Vector should move in this direction.
> 
> From your summary:
> 
> > For example, vector<16xi3> to vector<3xi16> has 48 bits on both sides, but 16 is not divisible by 3.
> 
> It sounds like you are looking for logic to model sub-byte types in Vector. There is already [VectorEmulateNarrowType](https://github.com/llvm/llvm-project/blob/0b5c006208cf7b0fee3709f468152e7de81607ef/mlir/lib/Dialect/Vector/Transforms/VectorEmulateNarrowType.cpp) for this purpose. While that pass works, it is admittedly somewhat fragile. In my opinion, this is because we never fully settled whether it should support arbitrary integer widths such as `i3`.
> 
> In hindsight, I think `VectorEmulateNarrowType` should not support such exotic types. The underlying logic is very complex, difficult to support, and there does not seem to be enough interest in maintaining it beyond very narrow use cases. In other words, the current support is not very general.
> 
> I also question whether `i3` is a type we should support beyond the most basic examples. I appreciate that some users may want this, but supporting it generally would come at a significant maintenance cost. IMHO, we should instead prioritise the more common cases that have more users and reviewers.
> 
> Is this a major blocker for you? If we start unblocking cases like this, I am worried that we will also need to review many other Vector transformations, such as unrolling and flattening, to make sure they bail out cleanly. That would be a substantial effort, and I would prefer to avoid it at this point.
> 
> If you have broader plans for sub-byte types, I would be happy to hear them.
> 
> Thank you!

@banach-space Thank you for the detailed explanation, and currently I do not have broader plans for sub-byte types or a real use case for i3. As I take a look into the VectorEmulateNarrowType, I do not think it will affect my fix toward this issue. Let me know if you find other concerns, and I am happy to take a look into it again.

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 01:24:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 01:24:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] "[mlir][linalg] Fix masking for rank-reducing
 contiguous extracts" (PR #206637)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206637@github.com>
Message-ID: <6a632133.507fc713.3702e2.4782@mx.google.com>


LouisLu060211 wrote:

> Hello @LouisLu060211 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read all the policies. I confirm that AI tool usage is noted in the PR description.

https://github.com/llvm/llvm-project/pull/206637

From llvmlistbot at llvm.org  Fri Jul 24 01:50:02 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Fri, 24 Jul 2026 01:50:02 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a63273a.92ea0e2a.2ed880.c825@mx.google.com>


================
@@ -600,14 +546,63 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
     }
     // Perform a backward CFG traversal to discover and map blocks in this loop.
     if (!Backedges.empty()) {
-      LoopT *L = AllocateLoop(Header);
-      discoverAndMapSubloop(L, ArrayRef<BlockT *>(Backedges), this, DomTree);
+      HasLoops = true;
+      LoopT *L = allocateLoop(Header);
+      discoverAndMapSubloop(L, Header, Backedges, DomTree);
+    }
+  }
+  // Most functions have no loops; skip the layout construction.
+  if (!HasLoops)
+    return;
+
+  // Record each in-loop block with its innermost loop in forward CFG postorder,
+  // and build the loop list in PO.
+  SmallVector<std::pair<BlockT *, LoopT *>, 32> PO;
+  SmallVector<LoopT *, 4> LoopsPO;
+  PO.reserve(BBMap.size());
+  for (BlockT *BB : post_order(DomRoot->getBlock())) {
----------------
MaskRay wrote:

`post_order(ParentPtr)` does not compile:
```
error: cannot initialize a variable of type 'llvm::BasicBlock *' with an
       rvalue of type 'reference' (aka 'const BasicBlock *')
```

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 01:59:14 2026
From: llvmlistbot at llvm.org (Abid Qadeer)
Date: Fri, 24 Jul 2026 01:59:14 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] Reland [OMPIRBuilder] Don't use
 invalid debug loc in reduction fn. (PR #211566)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211566@github.com>
Message-ID: <6a632962.c5504292.1bbfc8.40e6@mx.google.com>


https://github.com/abidh updated https://github.com/llvm/llvm-project/pull/211566

>From 2f74146c771de9e925bb0632851c32b195819baa Mon Sep 17 00:00:00 2001
From: Abid Qadeer <haqadeer at amd.com>
Date: Thu, 23 Jul 2026 14:33:11 +0100
Subject: [PATCH] Reland [OMPIRBuilder] Don't use invalid debug loc in
 reduction functions.

We have this pattern of code in OMPIRBuilder for many functions that are
used in reduction operations.

 ```
 Function *LtGRFunc = Function::Create
  BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
  Builder.SetInsertPoint(EntryBlock);
```

The insertion point is moved to the new function but the debug location
is not updated. This means that reduction function will use the debug
location that points to another function. This problem gets hidden
because these functions gets inlined but the potential for failure
exists.

This patch resets the debug location when insertion point is moved to new
function. Some `InsertPointGuard` have been added to make sure we restore
the debug location correctly when we are done with the reduction function.

It also clears the debug location for the helper-internal barriers in
emitInterWarpCopyFunction. Those barriers were created with the enclosing
construct's debug location, which updateToLocation leaves set on the
builder, so instructions emitted afterwards in the helper would otherwise
inherit a location scoped to the caller's (kernel) subprogram.

Co-authored-by: Cursor <cursoragent at cursor.com>
---
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     | 30 ++++----
 .../LLVMIR/omptarget-debug-reduc-fn-loc.mlir  | 71 +++++++++++++++++++
 2 files changed, 89 insertions(+), 12 deletions(-)
 create mode 100644 mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir

diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index fb1cebf28b95b..57e199bdff09a 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -3526,7 +3526,7 @@ Error OpenMPIRBuilder::emitReductionListCopy(
 Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
     const LocationDescription &Loc, ArrayRef<ReductionInfo> ReductionInfos,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  InsertPointTy SavedIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(), {Builder.getPtrTy(), Builder.getInt32Ty()},
@@ -3540,6 +3540,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   WcFunc->addParamAttr(1, Attribute::NoUndef);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", WcFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // ReduceList: thread local Reduce list.
   // At the stage of the computation when this function is called, partially
@@ -3646,7 +3647,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
 
       // kmpc_barrier.
       InsertPointOrErrorTy BarrierIP1 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3705,7 +3706,7 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
       // endif
       emitBlock(MergeBB, Builder.GetInsertBlock()->getParent());
       InsertPointOrErrorTy BarrierIP2 =
-          createBarrier(LocationDescription(Builder.saveIP(), Loc.DL),
+          createBarrier(LocationDescription(Builder.saveIP(), DebugLoc()),
                         omp::Directive::OMPD_unknown,
                         /* ForceSimpleCall */ false,
                         /* CheckCancelFlag */ true);
@@ -3778,7 +3779,6 @@ Expected<Function *> OpenMPIRBuilder::emitInterWarpCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(SavedIP);
 
   return WcFunc;
 }
@@ -3787,6 +3787,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
   LLVMContext &Ctx = M.getContext();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   FunctionType *FuncTy =
       FunctionType::get(Builder.getVoidTy(),
                         {Builder.getPtrTy(), Builder.getInt16Ty(),
@@ -3806,6 +3807,7 @@ Expected<Function *> OpenMPIRBuilder::emitShuffleAndReduceFunction(
   SarFunc->addParamAttr(3, Attribute::SExt);
   BasicBlock *EntryBB = BasicBlock::Create(M.getContext(), "entry", SarFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Thread local Reduce list used to host the values of data to be reduced.
   Argument *ReduceListArg = SarFunc->getArg(0);
@@ -4018,7 +4020,7 @@ Expected<Value *> OpenMPIRBuilder::createReductionDescriptorCopy(
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4034,6 +4036,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGCFunc->getArg(0);
@@ -4137,14 +4140,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4160,6 +4162,7 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", LtGRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = LtGRFunc->getArg(0);
@@ -4245,14 +4248,13 @@ Expected<Function *> OpenMPIRBuilder::emitListToGlobalReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {LocalReduceListAddrCast, ReduceList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return LtGRFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Type *ReductionsBufferTy,
     AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   FunctionType *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4268,6 +4270,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLCFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLCFunc->getArg(0);
@@ -4368,14 +4371,13 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListCopyFunction(
   }
 
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLCFunc;
 }
 
 Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
     ArrayRef<ReductionInfo> ReductionInfos, Function *ReduceFn,
     Type *ReductionsBufferTy, AttributeList FuncAttrs, ArrayRef<bool> IsByRef) {
-  OpenMPIRBuilder::InsertPointTy OldIP = Builder.saveIP();
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   LLVMContext &Ctx = M.getContext();
   auto *FuncTy = FunctionType::get(
       Builder.getVoidTy(),
@@ -4391,6 +4393,7 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
 
   BasicBlock *EntryBlock = BasicBlock::Create(Ctx, "entry", GtLRFunc);
   Builder.SetInsertPoint(EntryBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Buffer: global reduction buffer.
   Argument *BufferArg = GtLRFunc->getArg(0);
@@ -4476,7 +4479,6 @@ Expected<Function *> OpenMPIRBuilder::emitGlobalToListReduceFunction(
   createRuntimeFunctionCall(ReduceFn, {ReduceList, ReductionList})
       ->addFnAttr(Attribute::NoUnwind);
   Builder.CreateRetVoid();
-  Builder.restoreIP(OldIP);
   return GtLRFunc;
 }
 
@@ -4490,6 +4492,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
     StringRef ReducerName, ArrayRef<ReductionInfo> ReductionInfos,
     ArrayRef<bool> IsByRef, ReductionGenCBKind ReductionGenCBKind,
     AttributeList FuncAttrs) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   auto *FuncTy = FunctionType::get(Builder.getVoidTy(),
                                    {Builder.getPtrTy(), Builder.getPtrTy()},
                                    /* IsVarArg */ false);
@@ -4503,6 +4506,7 @@ Expected<Function *> OpenMPIRBuilder::createReductionFunction(
   BasicBlock *EntryBB =
       BasicBlock::Create(M.getContext(), "entry", ReductionFunc);
   Builder.SetInsertPoint(EntryBB);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
 
   // Need to alloca memory here and deal with the pointers before getting
   // LHS/RHS pointers out
@@ -4986,10 +4990,12 @@ static Error populateReductionFunction(
     Function *ReductionFunc,
     ArrayRef<OpenMPIRBuilder::ReductionInfo> ReductionInfos,
     IRBuilder<> &Builder, ArrayRef<bool> IsByRef, bool IsGPU) {
+  IRBuilder<>::InsertPointGuard IPG(Builder);
   Module *Module = ReductionFunc->getParent();
   BasicBlock *ReductionFuncBlock =
       BasicBlock::Create(Module->getContext(), "", ReductionFunc);
   Builder.SetInsertPoint(ReductionFuncBlock);
+  Builder.SetCurrentDebugLocation(llvm::DebugLoc());
   Value *LHSArrayPtr = nullptr;
   Value *RHSArrayPtr = nullptr;
   if (IsGPU) {
diff --git a/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
new file mode 100644
index 0000000000000..689446eee8de9
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/omptarget-debug-reduc-fn-loc.mlir
@@ -0,0 +1,71 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// The device reduction helpers emitted by OpenMPIRBuilder have no debug info of
+// their own. If the builder's current debug location is left set while their
+// bodies are emitted, their instructions get DILocations scoped to the
+// wrong subprogram. Check that none of the GPU reduction helpers carry a
+// !dbg attachment.
+
+#di_file = #llvm.di_file<"repro.f90" in "">
+#di_null_type = #llvm.di_null_type
+#loc1 = loc("repro.f90":1:1)
+#loc2 = loc("repro.f90":7:9)
+#di_compile_unit = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_Fortran95, file = #di_file, producer = "flang", isOptimized = true, emissionKind = Full>
+#di_subroutine_type = #llvm.di_subroutine_type<callingConvention = DW_CC_normal, types = #di_null_type>
+#di_subprogram = #llvm.di_subprogram<id = distinct[1]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "k", linkageName = "k_", file = #di_file, line = 1, scopeLine = 1, subprogramFlags = "Definition|Optimized", type = #di_subroutine_type>
+#di_subprogram1 = #llvm.di_subprogram<id = distinct[2]<>, compileUnit = #di_compile_unit, scope = #di_file, name = "__omp_offloading_k_l7", linkageName = "__omp_offloading_k_l7", file = #di_file, line = 7, scopeLine = 7, subprogramFlags = "LocalToUnit|Definition|Optimized", type = #di_subroutine_type>
+#loc14 = loc(fused<#di_subprogram>[#loc1])
+#loc15 = loc(fused<#di_subprogram1>[#loc2])
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr = dense<64> : vector<4xi64>, !llvm.ptr<1> = dense<64> : vector<4xi64>, !llvm.ptr<5> = dense<32> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, "dlti.alloca_memory_space" = 5 : ui64, "dlti.global_memory_space" = 1 : ui64>, llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_gpu = true, omp.is_target_device = true, omp.version = #omp.version<version = 31>} {
+  omp.declare_reduction @add_reduction_f64 : f64 init {
+  ^bb0(%arg0: f64):
+    %0 = llvm.mlir.constant(0.000000e+00 : f64) : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } combiner {
+  ^bb0(%arg0: f64, %arg1: f64):
+    %0 = llvm.fadd %arg0, %arg1 : f64 loc(#loc2)
+    omp.yield(%0 : f64) loc(#loc2)
+  } loc(#loc2)
+  llvm.func @k_(%arg0: !llvm.ptr) {
+    %1 = omp.map.info var_ptr(%arg0 : !llvm.ptr, f64) map_clauses(from) capture(ByRef) -> !llvm.ptr {name = "s"} loc(#loc2)
+    omp.target kernel_type(spmd) map_entries(%1 -> %arg1 : !llvm.ptr) {
+      %c1 = llvm.mlir.constant(1 : i32) : i32 loc(#loc2)
+      %cn = llvm.mlir.constant(1024 : i32) : i32 loc(#loc2)
+      omp.teams reduction(@add_reduction_f64 %arg1 -> %arg2 : !llvm.ptr) {
+        omp.parallel {
+          omp.distribute {
+            omp.wsloop reduction(@add_reduction_f64 %arg2 -> %arg3 : !llvm.ptr) {
+              omp.loop_nest (%arg4) : i32 = (%c1) to (%cn) inclusive step (%c1) {
+                %2 = llvm.load %arg3 : !llvm.ptr -> f64 loc(#loc2)
+                %cst = llvm.mlir.constant(1.000000e+00 : f64) : f64 loc(#loc2)
+                %3 = llvm.fadd %2, %cst : f64 loc(#loc2)
+                llvm.store %3, %arg3 : f64, !llvm.ptr loc(#loc2)
+                omp.yield loc(#loc2)
+              } loc(#loc2)
+            } {omp.composite} loc(#loc2)
+          } {omp.composite} loc(#loc2)
+          omp.terminator loc(#loc2)
+        } {omp.composite} loc(#loc2)
+        omp.terminator loc(#loc2)
+      } {omp.combined} loc(#loc2)
+      omp.terminator loc(#loc2)
+    } {omp.combined} loc(#loc15)
+    llvm.return loc(#loc14)
+  } loc(#loc14)
+} loc(#loc1)
+
+// CHECK-LABEL: define internal void @_omp_reduction_shuffle_and_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_inter_warp_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_list_to_global_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_copy_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }
+// CHECK-LABEL: define internal void @_omp_reduction_global_to_list_reduce_func(
+// CHECK-NOT:     !dbg
+// CHECK:       }


From llvmlistbot at llvm.org  Fri Jul 24 02:48:23 2026
From: llvmlistbot at llvm.org (Fabrice de Gans)
Date: Fri, 24 Jul 2026 02:48:23 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [polly] [llvm] Forward declare
 VirtualFileSystem (PR #211761)
Message-ID: <llvm.org/llvm/llvm-project/pull/211761@github.com>


https://github.com/Steelskin created https://github.com/llvm/llvm-project/pull/211761

This adds a new header that adds an explicit forward declaration for VirtualFileSystem, as well as a specialization of IntrusiveRefCntPtrInfo for this class. This allows for code that uses
`IntrusiveRefCntPtr<vfs::FileSystem>` to compile when building LLVM as a dylib on Windows, without having to include the full VirtualFileSystem.h header.

The effort to build LLVM as a dylib is tracked in #109483.

>From 2bb7a410bc5a11de3c0c24bb31d2da3129131678 Mon Sep 17 00:00:00 2001
From: Fabrice de Gans <steelskin at gmail.com>
Date: Fri, 24 Jul 2026 11:35:20 +0200
Subject: [PATCH] [llvm] Forward declare VirtualFileSystem

This adds a new header that adds an explicit forward declaration for
VirtualFileSystem, as well as a specialization of IntrusiveRefCntPtrInfo
for this class. This allows for code that uses
`IntrusiveRefCntPtr<vfs::FileSystem>` to compile when building LLVM as a
dylib on Windows, without having to include the full VirtualFileSystem.h
header.

The effort to build LLVM as a dylib is tracked in #109483.
---
 clang/include/clang/Basic/Diagnostic.h        |  4 +--
 .../clang/Basic/SanitizerSpecialCaseList.h    |  8 +----
 clang/include/clang/CIR/LowerToLLVM.h         |  4 +--
 clang/include/clang/CodeGen/BackendUtil.h     |  4 +--
 clang/include/clang/CodeGen/ModuleBuilder.h   |  5 +--
 clang/include/clang/Driver/Driver.h           |  4 +--
 clang/include/clang/Driver/ModulesDriver.h    |  5 +--
 clang/include/clang/Driver/ToolChain.h        |  6 +---
 clang/include/clang/Format/Format.h           |  7 +---
 clang/include/clang/Frontend/ASTUnit.h        |  6 +---
 .../clang/Frontend/CompilerInvocation.h       |  7 +---
 .../clang/Frontend/PrecompiledPreamble.h      |  4 +--
 clang/lib/CodeGen/CodeGenModule.h             |  5 +--
 llvm/include/llvm/CodeGen/MIRSampleProfile.h  |  5 +--
 llvm/include/llvm/CodeGen/Passes.h            |  4 +--
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  5 +--
 .../ProfileData/Coverage/CoverageMapping.h    |  5 +--
 .../llvm/ProfileData/InstrProfReader.h        |  6 +---
 .../llvm/ProfileData/SampleProfReader.h       |  5 +--
 llvm/include/llvm/Support/CommandLine.h       |  5 +--
 llvm/include/llvm/Support/SourceMgr.h         |  5 +--
 llvm/include/llvm/Support/SpecialCaseList.h   |  5 +--
 llvm/include/llvm/Support/VirtualFileSystem.h |  3 +-
 .../llvm/Support/VirtualFileSystemFwd.h       | 36 +++++++++++++++++++
 .../llvm/Transforms/IPO/SampleProfile.h       |  5 +--
 .../Transforms/Instrumentation/MemProfUse.h   |  5 +--
 .../Instrumentation/SanitizerBinaryMetadata.h |  4 +--
 .../Instrumentation/SanitizerCoverage.h       |  4 +--
 .../Utils/SampleProfileLoaderBaseImpl.h       |  5 +--
 llvm/include/llvm/WindowsDriver/MSVCPaths.h   |  5 +--
 llvm/lib/Support/VirtualFileSystem.cpp        | 13 +++++++
 mlir/include/mlir/Target/LLVMIR/Export.h      |  4 +--
 .../mlir/Target/LLVMIR/ModuleTranslation.h    |  4 +--
 polly/include/polly/ScopInliner.h             |  7 +---
 34 files changed, 81 insertions(+), 128 deletions(-)
 create mode 100644 llvm/include/llvm/Support/VirtualFileSystemFwd.h

diff --git a/clang/include/clang/Basic/Diagnostic.h b/clang/include/clang/Basic/Diagnostic.h
index 66e79e3b4300b..ee4a16d241f19 100644
--- a/clang/include/clang/Basic/Diagnostic.h
+++ b/clang/include/clang/Basic/Diagnostic.h
@@ -29,6 +29,7 @@
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ConvertUTF.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cassert>
 #include <cstdint>
 #include <limits>
@@ -46,9 +47,6 @@ namespace llvm {
 class Error;
 class raw_ostream;
 class MemoryBuffer;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Basic/SanitizerSpecialCaseList.h b/clang/include/clang/Basic/SanitizerSpecialCaseList.h
index a05da4c1561d9..2469cf074fc0c 100644
--- a/clang/include/clang/Basic/SanitizerSpecialCaseList.h
+++ b/clang/include/clang/Basic/SanitizerSpecialCaseList.h
@@ -18,16 +18,10 @@
 #include "clang/Basic/Sanitizers.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SpecialCaseList.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <utility>
 #include <vector>
-
-namespace llvm {
-namespace vfs {
-class FileSystem;
-}
-} // namespace llvm
-
 namespace clang {
 
 class SanitizerSpecialCaseList : public llvm::SpecialCaseList {
diff --git a/clang/include/clang/CIR/LowerToLLVM.h b/clang/include/clang/CIR/LowerToLLVM.h
index 305f54017ebe0..d97ac5265c282 100644
--- a/clang/include/clang/CIR/LowerToLLVM.h
+++ b/clang/include/clang/CIR/LowerToLLVM.h
@@ -13,14 +13,12 @@
 #define CLANG_CIR_LOWERTOLLVM_H
 
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 
 namespace llvm {
 class LLVMContext;
 class Module;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace mlir {
diff --git a/clang/include/clang/CodeGen/BackendUtil.h b/clang/include/clang/CodeGen/BackendUtil.h
index 8b0d975a876e6..630dbab00700e 100644
--- a/clang/include/clang/CodeGen/BackendUtil.h
+++ b/clang/include/clang/CodeGen/BackendUtil.h
@@ -11,6 +11,7 @@
 
 #include "clang/Basic/LLVM.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 
 namespace llvm {
@@ -19,9 +20,6 @@ template <typename T> class Expected;
 template <typename T> class IntrusiveRefCntPtr;
 class Module;
 class MemoryBufferRef;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/CodeGen/ModuleBuilder.h b/clang/include/clang/CodeGen/ModuleBuilder.h
index cd93ef7cc654a..99e3bdf013a8e 100644
--- a/clang/include/clang/CodeGen/ModuleBuilder.h
+++ b/clang/include/clang/CodeGen/ModuleBuilder.h
@@ -16,16 +16,13 @@
 #include "clang/AST/ASTConsumer.h"
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace llvm {
   class Constant;
   class LLVMContext;
   class Module;
   class StringRef;
-
-  namespace vfs {
-  class FileSystem;
-  }
 }
 
 // Prefix of the name of the artificial inline frame.
diff --git a/clang/include/clang/Driver/Driver.h b/clang/include/clang/Driver/Driver.h
index eece9ac5293f0..15b6fdcb8a574 100644
--- a/clang/include/clang/Driver/Driver.h
+++ b/clang/include/clang/Driver/Driver.h
@@ -28,6 +28,7 @@
 #include "llvm/Option/Arg.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/StringSaver.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <map>
 #include <set>
@@ -36,9 +37,6 @@
 
 namespace llvm {
 class Triple;
-namespace vfs {
-class FileSystem;
-}
 namespace cl {
 class ExpansionContext;
 }
diff --git a/clang/include/clang/Driver/ModulesDriver.h b/clang/include/clang/Driver/ModulesDriver.h
index 7146d2f6b143f..e67a862f3beff 100644
--- a/clang/include/clang/Driver/ModulesDriver.h
+++ b/clang/include/clang/Driver/ModulesDriver.h
@@ -18,10 +18,7 @@
 #include "clang/Basic/LLVM.h"
 #include "clang/Driver/Types.h"
 #include "llvm/Support/Error.h"
-
-namespace llvm::vfs {
-class FileSystem;
-} // namespace llvm::vfs
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace clang {
 class DiagnosticsEngine;
diff --git a/clang/include/clang/Driver/ToolChain.h b/clang/include/clang/Driver/ToolChain.h
index c5633e10933e8..55856a6e4e1c3 100644
--- a/clang/include/clang/Driver/ToolChain.h
+++ b/clang/include/clang/Driver/ToolChain.h
@@ -26,6 +26,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/VersionTuple.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/TargetParser/Triple.h"
 #include <cassert>
@@ -43,11 +44,6 @@ class ArgList;
 class DerivedArgList;
 
 } // namespace opt
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Format/Format.h b/clang/include/clang/Format/Format.h
index 27b2d8f4a405b..9363222205414 100644
--- a/clang/include/clang/Format/Format.h
+++ b/clang/include/clang/Format/Format.h
@@ -21,15 +21,10 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <optional>
 #include <system_error>
 
-namespace llvm {
-namespace vfs {
-class FileSystem;
-}
-} // namespace llvm
-
 namespace clang {
 namespace format {
 
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h
index b187494e449f2..92d0bd9d03b4d 100644
--- a/clang/include/clang/Frontend/ASTUnit.h
+++ b/clang/include/clang/Frontend/ASTUnit.h
@@ -39,6 +39,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Bitstream/BitstreamWriter.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
@@ -52,11 +53,6 @@ namespace llvm {
 
 class MemoryBuffer;
 
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Frontend/CompilerInvocation.h b/clang/include/clang/Frontend/CompilerInvocation.h
index f71969d515be0..926d77a312c66 100644
--- a/clang/include/clang/Frontend/CompilerInvocation.h
+++ b/clang/include/clang/Frontend/CompilerInvocation.h
@@ -24,6 +24,7 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/ScopeExit.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <memory>
 #include <string>
@@ -38,12 +39,6 @@ class ArgList;
 
 } // namespace opt
 
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
-
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Frontend/PrecompiledPreamble.h b/clang/include/clang/Frontend/PrecompiledPreamble.h
index 565395b4f3986..34839f50d3e86 100644
--- a/clang/include/clang/Frontend/PrecompiledPreamble.h
+++ b/clang/include/clang/Frontend/PrecompiledPreamble.h
@@ -18,6 +18,7 @@
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MD5.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cstddef>
 #include <memory>
 #include <system_error>
@@ -26,9 +27,6 @@
 namespace llvm {
 class MemoryBuffer;
 class MemoryBufferRef;
-namespace vfs {
-class FileSystem;
-}
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/lib/CodeGen/CodeGenModule.h b/clang/lib/CodeGen/CodeGenModule.h
index 54b08b588dfde..4b924e12d7adf 100644
--- a/clang/lib/CodeGen/CodeGenModule.h
+++ b/clang/lib/CodeGen/CodeGenModule.h
@@ -39,6 +39,7 @@
 #include "llvm/IR/Module.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Support/Allocator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/SanitizerStats.h"
 #include <optional>
 
@@ -53,10 +54,6 @@ class FunctionType;
 class LLVMContext;
 class IndexedInstrProfReader;
 
-namespace vfs {
-class FileSystem;
-}
-
 namespace abi {
 class ArgInfo;
 class IRTypeMapper;
diff --git a/llvm/include/llvm/CodeGen/MIRSampleProfile.h b/llvm/include/llvm/CodeGen/MIRSampleProfile.h
index 9ae5472fd83bf..43639c7bb8039 100644
--- a/llvm/include/llvm/CodeGen/MIRSampleProfile.h
+++ b/llvm/include/llvm/CodeGen/MIRSampleProfile.h
@@ -18,6 +18,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/Support/Discriminator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <string>
 
@@ -27,10 +28,6 @@ class MachineBlockFrequencyInfo;
 class MachineFunction;
 class Module;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 using namespace sampleprof;
 
 class MIRProfileLoader;
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 86bf5aea7b23a..d48e1770a4d89 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -18,6 +18,7 @@
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Discriminator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <functional>
 #include <string>
@@ -34,9 +35,6 @@ class raw_ostream;
 enum class RunOutliner;
 
 template <typename T> class IntrusiveRefCntPtr;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 } // namespace llvm
 
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index fee8c712998b5..86252ad20d867 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -27,6 +27,7 @@
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/TargetParser/Triple.h"
 #include <forward_list>
 #include <map>
@@ -43,10 +44,6 @@ class Loop;
 class LoopAnalysis;
 class LoopInfo;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// Move the instruction after an InsertPoint to the beginning of another
 /// BasicBlock.
 ///
diff --git a/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h b/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
index a664b6b5fa560..a74b334c35f06 100644
--- a/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
+++ b/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
@@ -30,6 +30,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
@@ -52,10 +53,6 @@ namespace object {
 class BuildIDFetcher;
 } // namespace object
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 namespace coverage {
 
 class CoverageMappingReader;
diff --git a/llvm/include/llvm/ProfileData/InstrProfReader.h b/llvm/include/llvm/ProfileData/InstrProfReader.h
index 51b97e9f88ee3..2607a2566e39d 100644
--- a/llvm/include/llvm/ProfileData/InstrProfReader.h
+++ b/llvm/include/llvm/ProfileData/InstrProfReader.h
@@ -13,7 +13,6 @@
 
 #ifndef LLVM_PROFILEDATA_INSTRPROFREADER_H
 #define LLVM_PROFILEDATA_INSTRPROFREADER_H
-
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/ProfileSummary.h"
@@ -32,6 +31,7 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
 #include "llvm/Support/SwapByteOrder.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
@@ -45,10 +45,6 @@ namespace llvm {
 
 class InstrProfReader;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// A file format agnostic iterator over profiling data.
 template <class record_type = NamedInstrProfRecord,
           class reader_type = InstrProfReader>
diff --git a/llvm/include/llvm/ProfileData/SampleProfReader.h b/llvm/include/llvm/ProfileData/SampleProfReader.h
index fd7bf8c49374e..830d4086f95e2 100644
--- a/llvm/include/llvm/ProfileData/SampleProfReader.h
+++ b/llvm/include/llvm/ProfileData/SampleProfReader.h
@@ -241,6 +241,7 @@
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <array>
 #include <cstdint>
 #include <list>
@@ -255,10 +256,6 @@ namespace llvm {
 class raw_ostream;
 class Twine;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 namespace sampleprof {
 
 class SampleProfileReader;
diff --git a/llvm/include/llvm/Support/CommandLine.h b/llvm/include/llvm/Support/CommandLine.h
index 662eb44d6d2a8..dd42c4bfb7f49 100644
--- a/llvm/include/llvm/Support/CommandLine.h
+++ b/llvm/include/llvm/Support/CommandLine.h
@@ -29,6 +29,7 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/StringSaver.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <climits>
@@ -41,10 +42,6 @@
 
 namespace llvm {
 
-namespace vfs {
-class FileSystem;
-}
-
 class StringSaver;
 class ElementCount;
 
diff --git a/llvm/include/llvm/Support/SourceMgr.h b/llvm/include/llvm/Support/SourceMgr.h
index 884f15ac482fe..452a9ab28af94 100644
--- a/llvm/include/llvm/Support/SourceMgr.h
+++ b/llvm/include/llvm/Support/SourceMgr.h
@@ -20,14 +20,11 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/SMLoc.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <vector>
 
 namespace llvm {
 
-namespace vfs {
-class FileSystem;
-} // end namespace vfs
-
 class raw_ostream;
 class SMDiagnostic;
 class SMFixIt;
diff --git a/llvm/include/llvm/Support/SpecialCaseList.h b/llvm/include/llvm/Support/SpecialCaseList.h
index 28e264f078d89..3366ee078ebc7 100644
--- a/llvm/include/llvm/Support/SpecialCaseList.h
+++ b/llvm/include/llvm/Support/SpecialCaseList.h
@@ -14,6 +14,7 @@
 
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <string>
 #include <utility>
@@ -23,10 +24,6 @@ namespace llvm {
 class MemoryBuffer;
 class StringRef;
 
-namespace vfs {
-class FileSystem;
-}
-
 /// This is a utility class used to parse user-provided text files with
 /// "special case lists" for code sanitizers. Such files are used to
 /// define an "ABI list" for DataFlowSanitizer and allow/exclusion lists for
diff --git a/llvm/include/llvm/Support/VirtualFileSystem.h b/llvm/include/llvm/Support/VirtualFileSystem.h
index d22c534228331..288f9817976e6 100644
--- a/llvm/include/llvm/Support/VirtualFileSystem.h
+++ b/llvm/include/llvm/Support/VirtualFileSystem.h
@@ -27,6 +27,7 @@
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <atomic>
 #include <cassert>
 #include <cstdint>
@@ -215,8 +216,6 @@ class directory_iterator {
   }
 };
 
-class FileSystem;
-
 namespace detail {
 
 /// Keeps state for the recursive_directory_iterator.
diff --git a/llvm/include/llvm/Support/VirtualFileSystemFwd.h b/llvm/include/llvm/Support/VirtualFileSystemFwd.h
new file mode 100644
index 0000000000000..fc971f7aa9b40
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualFileSystemFwd.h
@@ -0,0 +1,36 @@
+//===- VirtualFileSystemFwd.h - Virtual File System Forward -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Contains the forward declaration for vfs::FileSystem, as well as the
+/// IntrusiveRefCntPtrInfo specialization for it. This allows the
+/// vfs::FileSystem class to be used with IntrusiveRefCntPtr without including
+/// its full definition.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
+#define LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
+
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/Support/Compiler.h"
+
+namespace llvm {
+namespace vfs {
+class FileSystem;
+} // namespace vfs
+
+template <> struct LLVM_ABI IntrusiveRefCntPtrInfo<::llvm::vfs::FileSystem> {
+  static unsigned useCount(const ::llvm::vfs::FileSystem *FS);
+  static void retain(::llvm::vfs::FileSystem *FS);
+  static void release(::llvm::vfs::FileSystem *FS);
+};
+
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
diff --git a/llvm/include/llvm/Transforms/IPO/SampleProfile.h b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
index ab8c754c16fbe..a3496b596dff6 100644
--- a/llvm/include/llvm/Transforms/IPO/SampleProfile.h
+++ b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
@@ -19,6 +19,7 @@
 #include "llvm/Pass.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <string>
 
 namespace llvm {
@@ -32,10 +33,6 @@ LLVM_ABI extern cl::opt<int> ProfileInlineLimitMin;
 LLVM_ABI extern cl::opt<int> ProfileInlineLimitMax;
 LLVM_ABI extern cl::opt<bool> SortProfiledSCC;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// The sample profiler data loader pass.
 class SampleProfileLoaderPass
     : public OptionalPassInfoMixin<SampleProfileLoaderPass> {
diff --git a/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h b/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
index 01104097180e4..3afd88f144859 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
@@ -17,16 +17,13 @@
 #include "llvm/ProfileData/DataAccessProf.h"
 #include "llvm/ProfileData/MemProf.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace llvm {
 class IndexedInstrProfReader;
 class Module;
 class TargetLibraryInfo;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 class MemProfUsePass : public OptionalPassInfoMixin<MemProfUsePass> {
 public:
   LLVM_ABI explicit MemProfUsePass(
diff --git a/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h b/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
index 4cc0253ba0baf..1246c29ca3d46 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
@@ -18,12 +18,10 @@
 #include "llvm/IR/Module.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/Instrumentation.h"
 
 namespace llvm {
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 struct SanitizerBinaryMetadataOptions {
   bool Covered = false;
diff --git a/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h b/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
index 269bcbcd32265..1d8b6fc2441c6 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
@@ -19,13 +19,11 @@
 #include "llvm/IR/PassManager.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/SpecialCaseList.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/Instrumentation.h"
 
 namespace llvm {
 class Module;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 /// This is the ModuleSanitizerCoverage pass used in the new pass manager. The
 /// pass instruments functions for coverage, adds initialization calls to the
diff --git a/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h b/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
index 5346871249cf5..c7262fb35ab95 100644
--- a/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
+++ b/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
@@ -40,6 +40,7 @@
 #include "llvm/ProfileData/SampleProfReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/GenericDomTree.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Utils/SampleProfileInference.h"
 #include "llvm/Transforms/Utils/SampleProfileLoaderBaseUtil.h"
@@ -48,10 +49,6 @@ namespace llvm {
 using namespace sampleprof;
 using namespace sampleprofutil;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 #define DEBUG_TYPE "sample-profile-impl"
 
 namespace afdo_detail {
diff --git a/llvm/include/llvm/WindowsDriver/MSVCPaths.h b/llvm/include/llvm/WindowsDriver/MSVCPaths.h
index 076c2eaf9c3f0..0a5f007d48006 100644
--- a/llvm/include/llvm/WindowsDriver/MSVCPaths.h
+++ b/llvm/include/llvm/WindowsDriver/MSVCPaths.h
@@ -12,16 +12,13 @@
 #include "llvm/ADT/SmallString.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/TargetParser/Triple.h"
 #include <optional>
 #include <string>
 
 namespace llvm {
 
-namespace vfs {
-class FileSystem;
-}
-
 enum class SubDirectoryType {
   Bin,
   Include,
diff --git a/llvm/lib/Support/VirtualFileSystem.cpp b/llvm/lib/Support/VirtualFileSystem.cpp
index 42e8bb4f9958e..7b970bee1adc8 100644
--- a/llvm/lib/Support/VirtualFileSystem.cpp
+++ b/llvm/lib/Support/VirtualFileSystem.cpp
@@ -2996,3 +2996,16 @@ const char OverlayFileSystem::ID = 0;
 const char ProxyFileSystem::ID = 0;
 const char InMemoryFileSystem::ID = 0;
 const char RedirectingFileSystem::ID = 0;
+
+unsigned ::llvm::IntrusiveRefCntPtrInfo<FileSystem>::useCount(
+    const FileSystem *FS) {
+  return FS->UseCount();
+}
+
+void ::llvm::IntrusiveRefCntPtrInfo<FileSystem>::retain(FileSystem *FS) {
+  FS->Retain();
+}
+
+void ::llvm::IntrusiveRefCntPtrInfo<FileSystem>::release(FileSystem *FS) {
+  FS->Release();
+}
diff --git a/mlir/include/mlir/Target/LLVMIR/Export.h b/mlir/include/mlir/Target/LLVMIR/Export.h
index 49cac420c26bf..08f5f92980b20 100644
--- a/mlir/include/mlir/Target/LLVMIR/Export.h
+++ b/mlir/include/mlir/Target/LLVMIR/Export.h
@@ -10,14 +10,12 @@
 #define MLIR_TARGET_LLVMIR_EXPORT_H
 
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 
 namespace llvm {
 class LLVMContext;
 class Module;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace mlir {
diff --git a/mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h b/mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
index c5c4b105ee152..39d2570982ead 100644
--- a/mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
+++ b/mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h
@@ -27,6 +27,7 @@
 #include "llvm/ADT/SetVector.h"
 #include "llvm/IR/FPEnv.h"
 #include "llvm/IR/Module.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace llvm {
 class BasicBlock;
@@ -36,9 +37,6 @@ class Function;
 class IRBuilderBase;
 class OpenMPIRBuilder;
 class Value;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace mlir {
diff --git a/polly/include/polly/ScopInliner.h b/polly/include/polly/ScopInliner.h
index f5cf97833e6a5..940dbf49aebde 100644
--- a/polly/include/polly/ScopInliner.h
+++ b/polly/include/polly/ScopInliner.h
@@ -13,12 +13,7 @@
 #include "llvm/Analysis/CGSCCPassManager.h"
 #include "llvm/Analysis/LazyCallGraph.h"
 #include "llvm/IR/PassManager.h"
-
-namespace llvm {
-namespace vfs {
-class FileSystem;
-}
-} // namespace llvm
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace polly {
 class ScopInlinerPass : public llvm::OptionalPassInfoMixin<ScopInlinerPass> {


From llvmlistbot at llvm.org  Fri Jul 24 02:58:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 02:58:36 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [polly] [llvm] Forward declare
 VirtualFileSystem (PR #211761)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211761@github.com>
Message-ID: <6a63374c.83ce629a.2277d7.7f22@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: Fabrice de Gans (Steelskin)

<details>
<summary>Changes</summary>

This adds a new header that adds an explicit forward declaration for VirtualFileSystem, as well as a specialization of IntrusiveRefCntPtrInfo for this class. This allows for code that uses
`IntrusiveRefCntPtr&lt;vfs::FileSystem&gt;` to compile when building LLVM as a dylib on Windows, without having to include the full VirtualFileSystem.h header.

The effort to build LLVM as a dylib is tracked in #<!-- -->109483.

---

Patch is 23.26 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211761.diff


34 Files Affected:

- (modified) clang/include/clang/Basic/Diagnostic.h (+1-3) 
- (modified) clang/include/clang/Basic/SanitizerSpecialCaseList.h (+1-7) 
- (modified) clang/include/clang/CIR/LowerToLLVM.h (+1-3) 
- (modified) clang/include/clang/CodeGen/BackendUtil.h (+1-3) 
- (modified) clang/include/clang/CodeGen/ModuleBuilder.h (+1-4) 
- (modified) clang/include/clang/Driver/Driver.h (+1-3) 
- (modified) clang/include/clang/Driver/ModulesDriver.h (+1-4) 
- (modified) clang/include/clang/Driver/ToolChain.h (+1-5) 
- (modified) clang/include/clang/Format/Format.h (+1-6) 
- (modified) clang/include/clang/Frontend/ASTUnit.h (+1-5) 
- (modified) clang/include/clang/Frontend/CompilerInvocation.h (+1-6) 
- (modified) clang/include/clang/Frontend/PrecompiledPreamble.h (+1-3) 
- (modified) clang/lib/CodeGen/CodeGenModule.h (+1-4) 
- (modified) llvm/include/llvm/CodeGen/MIRSampleProfile.h (+1-4) 
- (modified) llvm/include/llvm/CodeGen/Passes.h (+1-3) 
- (modified) llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h (+1-4) 
- (modified) llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h (+1-4) 
- (modified) llvm/include/llvm/ProfileData/InstrProfReader.h (+1-5) 
- (modified) llvm/include/llvm/ProfileData/SampleProfReader.h (+1-4) 
- (modified) llvm/include/llvm/Support/CommandLine.h (+1-4) 
- (modified) llvm/include/llvm/Support/SourceMgr.h (+1-4) 
- (modified) llvm/include/llvm/Support/SpecialCaseList.h (+1-4) 
- (modified) llvm/include/llvm/Support/VirtualFileSystem.h (+1-2) 
- (added) llvm/include/llvm/Support/VirtualFileSystemFwd.h (+36) 
- (modified) llvm/include/llvm/Transforms/IPO/SampleProfile.h (+1-4) 
- (modified) llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h (+1-4) 
- (modified) llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h (+1-3) 
- (modified) llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h (+1-3) 
- (modified) llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h (+1-4) 
- (modified) llvm/include/llvm/WindowsDriver/MSVCPaths.h (+1-4) 
- (modified) llvm/lib/Support/VirtualFileSystem.cpp (+13) 
- (modified) mlir/include/mlir/Target/LLVMIR/Export.h (+1-3) 
- (modified) mlir/include/mlir/Target/LLVMIR/ModuleTranslation.h (+1-3) 
- (modified) polly/include/polly/ScopInliner.h (+1-6) 


``````````diff
diff --git a/clang/include/clang/Basic/Diagnostic.h b/clang/include/clang/Basic/Diagnostic.h
index 66e79e3b4300b..ee4a16d241f19 100644
--- a/clang/include/clang/Basic/Diagnostic.h
+++ b/clang/include/clang/Basic/Diagnostic.h
@@ -29,6 +29,7 @@
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ConvertUTF.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cassert>
 #include <cstdint>
 #include <limits>
@@ -46,9 +47,6 @@ namespace llvm {
 class Error;
 class raw_ostream;
 class MemoryBuffer;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Basic/SanitizerSpecialCaseList.h b/clang/include/clang/Basic/SanitizerSpecialCaseList.h
index a05da4c1561d9..2469cf074fc0c 100644
--- a/clang/include/clang/Basic/SanitizerSpecialCaseList.h
+++ b/clang/include/clang/Basic/SanitizerSpecialCaseList.h
@@ -18,16 +18,10 @@
 #include "clang/Basic/Sanitizers.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SpecialCaseList.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <utility>
 #include <vector>
-
-namespace llvm {
-namespace vfs {
-class FileSystem;
-}
-} // namespace llvm
-
 namespace clang {
 
 class SanitizerSpecialCaseList : public llvm::SpecialCaseList {
diff --git a/clang/include/clang/CIR/LowerToLLVM.h b/clang/include/clang/CIR/LowerToLLVM.h
index 305f54017ebe0..d97ac5265c282 100644
--- a/clang/include/clang/CIR/LowerToLLVM.h
+++ b/clang/include/clang/CIR/LowerToLLVM.h
@@ -13,14 +13,12 @@
 #define CLANG_CIR_LOWERTOLLVM_H
 
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 
 namespace llvm {
 class LLVMContext;
 class Module;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace mlir {
diff --git a/clang/include/clang/CodeGen/BackendUtil.h b/clang/include/clang/CodeGen/BackendUtil.h
index 8b0d975a876e6..630dbab00700e 100644
--- a/clang/include/clang/CodeGen/BackendUtil.h
+++ b/clang/include/clang/CodeGen/BackendUtil.h
@@ -11,6 +11,7 @@
 
 #include "clang/Basic/LLVM.h"
 #include "llvm/IR/ModuleSummaryIndex.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 
 namespace llvm {
@@ -19,9 +20,6 @@ template <typename T> class Expected;
 template <typename T> class IntrusiveRefCntPtr;
 class Module;
 class MemoryBufferRef;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/CodeGen/ModuleBuilder.h b/clang/include/clang/CodeGen/ModuleBuilder.h
index cd93ef7cc654a..99e3bdf013a8e 100644
--- a/clang/include/clang/CodeGen/ModuleBuilder.h
+++ b/clang/include/clang/CodeGen/ModuleBuilder.h
@@ -16,16 +16,13 @@
 #include "clang/AST/ASTConsumer.h"
 #include "clang/Basic/LLVM.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace llvm {
   class Constant;
   class LLVMContext;
   class Module;
   class StringRef;
-
-  namespace vfs {
-  class FileSystem;
-  }
 }
 
 // Prefix of the name of the artificial inline frame.
diff --git a/clang/include/clang/Driver/Driver.h b/clang/include/clang/Driver/Driver.h
index eece9ac5293f0..15b6fdcb8a574 100644
--- a/clang/include/clang/Driver/Driver.h
+++ b/clang/include/clang/Driver/Driver.h
@@ -28,6 +28,7 @@
 #include "llvm/Option/Arg.h"
 #include "llvm/Option/ArgList.h"
 #include "llvm/Support/StringSaver.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <map>
 #include <set>
@@ -36,9 +37,6 @@
 
 namespace llvm {
 class Triple;
-namespace vfs {
-class FileSystem;
-}
 namespace cl {
 class ExpansionContext;
 }
diff --git a/clang/include/clang/Driver/ModulesDriver.h b/clang/include/clang/Driver/ModulesDriver.h
index 7146d2f6b143f..e67a862f3beff 100644
--- a/clang/include/clang/Driver/ModulesDriver.h
+++ b/clang/include/clang/Driver/ModulesDriver.h
@@ -18,10 +18,7 @@
 #include "clang/Basic/LLVM.h"
 #include "clang/Driver/Types.h"
 #include "llvm/Support/Error.h"
-
-namespace llvm::vfs {
-class FileSystem;
-} // namespace llvm::vfs
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace clang {
 class DiagnosticsEngine;
diff --git a/clang/include/clang/Driver/ToolChain.h b/clang/include/clang/Driver/ToolChain.h
index c5633e10933e8..55856a6e4e1c3 100644
--- a/clang/include/clang/Driver/ToolChain.h
+++ b/clang/include/clang/Driver/ToolChain.h
@@ -26,6 +26,7 @@
 #include "llvm/MC/MCTargetOptions.h"
 #include "llvm/Option/Option.h"
 #include "llvm/Support/VersionTuple.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Target/TargetOptions.h"
 #include "llvm/TargetParser/Triple.h"
 #include <cassert>
@@ -43,11 +44,6 @@ class ArgList;
 class DerivedArgList;
 
 } // namespace opt
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Format/Format.h b/clang/include/clang/Format/Format.h
index 27b2d8f4a405b..9363222205414 100644
--- a/clang/include/clang/Format/Format.h
+++ b/clang/include/clang/Format/Format.h
@@ -21,15 +21,10 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/Support/Regex.h"
 #include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <optional>
 #include <system_error>
 
-namespace llvm {
-namespace vfs {
-class FileSystem;
-}
-} // namespace llvm
-
 namespace clang {
 namespace format {
 
diff --git a/clang/include/clang/Frontend/ASTUnit.h b/clang/include/clang/Frontend/ASTUnit.h
index b187494e449f2..92d0bd9d03b4d 100644
--- a/clang/include/clang/Frontend/ASTUnit.h
+++ b/clang/include/clang/Frontend/ASTUnit.h
@@ -39,6 +39,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Bitstream/BitstreamWriter.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cassert>
 #include <cstddef>
 #include <cstdint>
@@ -52,11 +53,6 @@ namespace llvm {
 
 class MemoryBuffer;
 
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Frontend/CompilerInvocation.h b/clang/include/clang/Frontend/CompilerInvocation.h
index f71969d515be0..926d77a312c66 100644
--- a/clang/include/clang/Frontend/CompilerInvocation.h
+++ b/clang/include/clang/Frontend/CompilerInvocation.h
@@ -24,6 +24,7 @@
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/ScopeExit.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <memory>
 #include <string>
@@ -38,12 +39,6 @@ class ArgList;
 
 } // namespace opt
 
-namespace vfs {
-
-class FileSystem;
-
-} // namespace vfs
-
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/include/clang/Frontend/PrecompiledPreamble.h b/clang/include/clang/Frontend/PrecompiledPreamble.h
index 565395b4f3986..34839f50d3e86 100644
--- a/clang/include/clang/Frontend/PrecompiledPreamble.h
+++ b/clang/include/clang/Frontend/PrecompiledPreamble.h
@@ -18,6 +18,7 @@
 #include "llvm/ADT/IntrusiveRefCntPtr.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/MD5.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <cstddef>
 #include <memory>
 #include <system_error>
@@ -26,9 +27,6 @@
 namespace llvm {
 class MemoryBuffer;
 class MemoryBufferRef;
-namespace vfs {
-class FileSystem;
-}
 } // namespace llvm
 
 namespace clang {
diff --git a/clang/lib/CodeGen/CodeGenModule.h b/clang/lib/CodeGen/CodeGenModule.h
index 54b08b588dfde..4b924e12d7adf 100644
--- a/clang/lib/CodeGen/CodeGenModule.h
+++ b/clang/lib/CodeGen/CodeGenModule.h
@@ -39,6 +39,7 @@
 #include "llvm/IR/Module.h"
 #include "llvm/IR/ValueHandle.h"
 #include "llvm/Support/Allocator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/SanitizerStats.h"
 #include <optional>
 
@@ -53,10 +54,6 @@ class FunctionType;
 class LLVMContext;
 class IndexedInstrProfReader;
 
-namespace vfs {
-class FileSystem;
-}
-
 namespace abi {
 class ArgInfo;
 class IRTypeMapper;
diff --git a/llvm/include/llvm/CodeGen/MIRSampleProfile.h b/llvm/include/llvm/CodeGen/MIRSampleProfile.h
index 9ae5472fd83bf..43639c7bb8039 100644
--- a/llvm/include/llvm/CodeGen/MIRSampleProfile.h
+++ b/llvm/include/llvm/CodeGen/MIRSampleProfile.h
@@ -18,6 +18,7 @@
 #include "llvm/ADT/StringRef.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/Support/Discriminator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <string>
 
@@ -27,10 +28,6 @@ class MachineBlockFrequencyInfo;
 class MachineFunction;
 class Module;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 using namespace sampleprof;
 
 class MIRProfileLoader;
diff --git a/llvm/include/llvm/CodeGen/Passes.h b/llvm/include/llvm/CodeGen/Passes.h
index 86bf5aea7b23a..d48e1770a4d89 100644
--- a/llvm/include/llvm/CodeGen/Passes.h
+++ b/llvm/include/llvm/CodeGen/Passes.h
@@ -18,6 +18,7 @@
 #include "llvm/Support/CodeGen.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Discriminator.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 #include <functional>
 #include <string>
@@ -34,9 +35,6 @@ class raw_ostream;
 enum class RunOutliner;
 
 template <typename T> class IntrusiveRefCntPtr;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 } // namespace llvm
 
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index fee8c712998b5..86252ad20d867 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -27,6 +27,7 @@
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/TargetParser/Triple.h"
 #include <forward_list>
 #include <map>
@@ -43,10 +44,6 @@ class Loop;
 class LoopAnalysis;
 class LoopInfo;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// Move the instruction after an InsertPoint to the beginning of another
 /// BasicBlock.
 ///
diff --git a/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h b/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
index a664b6b5fa560..a74b334c35f06 100644
--- a/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
+++ b/llvm/include/llvm/ProfileData/Coverage/CoverageMapping.h
@@ -30,6 +30,7 @@
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cassert>
@@ -52,10 +53,6 @@ namespace object {
 class BuildIDFetcher;
 } // namespace object
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 namespace coverage {
 
 class CoverageMappingReader;
diff --git a/llvm/include/llvm/ProfileData/InstrProfReader.h b/llvm/include/llvm/ProfileData/InstrProfReader.h
index 51b97e9f88ee3..2607a2566e39d 100644
--- a/llvm/include/llvm/ProfileData/InstrProfReader.h
+++ b/llvm/include/llvm/ProfileData/InstrProfReader.h
@@ -13,7 +13,6 @@
 
 #ifndef LLVM_PROFILEDATA_INSTRPROFREADER_H
 #define LLVM_PROFILEDATA_INSTRPROFREADER_H
-
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/IR/ProfileSummary.h"
@@ -32,6 +31,7 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
 #include "llvm/Support/SwapByteOrder.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <algorithm>
 #include <cassert>
 #include <cstddef>
@@ -45,10 +45,6 @@ namespace llvm {
 
 class InstrProfReader;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// A file format agnostic iterator over profiling data.
 template <class record_type = NamedInstrProfRecord,
           class reader_type = InstrProfReader>
diff --git a/llvm/include/llvm/ProfileData/SampleProfReader.h b/llvm/include/llvm/ProfileData/SampleProfReader.h
index fd7bf8c49374e..830d4086f95e2 100644
--- a/llvm/include/llvm/ProfileData/SampleProfReader.h
+++ b/llvm/include/llvm/ProfileData/SampleProfReader.h
@@ -241,6 +241,7 @@
 #include "llvm/Support/ErrorOr.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/OnDiskHashTable.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <array>
 #include <cstdint>
 #include <list>
@@ -255,10 +256,6 @@ namespace llvm {
 class raw_ostream;
 class Twine;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 namespace sampleprof {
 
 class SampleProfileReader;
diff --git a/llvm/include/llvm/Support/CommandLine.h b/llvm/include/llvm/Support/CommandLine.h
index 662eb44d6d2a8..dd42c4bfb7f49 100644
--- a/llvm/include/llvm/Support/CommandLine.h
+++ b/llvm/include/llvm/Support/CommandLine.h
@@ -29,6 +29,7 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/StringSaver.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <climits>
@@ -41,10 +42,6 @@
 
 namespace llvm {
 
-namespace vfs {
-class FileSystem;
-}
-
 class StringSaver;
 class ElementCount;
 
diff --git a/llvm/include/llvm/Support/SourceMgr.h b/llvm/include/llvm/Support/SourceMgr.h
index 884f15ac482fe..452a9ab28af94 100644
--- a/llvm/include/llvm/Support/SourceMgr.h
+++ b/llvm/include/llvm/Support/SourceMgr.h
@@ -20,14 +20,11 @@
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/SMLoc.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <vector>
 
 namespace llvm {
 
-namespace vfs {
-class FileSystem;
-} // end namespace vfs
-
 class raw_ostream;
 class SMDiagnostic;
 class SMFixIt;
diff --git a/llvm/include/llvm/Support/SpecialCaseList.h b/llvm/include/llvm/Support/SpecialCaseList.h
index 28e264f078d89..3366ee078ebc7 100644
--- a/llvm/include/llvm/Support/SpecialCaseList.h
+++ b/llvm/include/llvm/Support/SpecialCaseList.h
@@ -14,6 +14,7 @@
 
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <memory>
 #include <string>
 #include <utility>
@@ -23,10 +24,6 @@ namespace llvm {
 class MemoryBuffer;
 class StringRef;
 
-namespace vfs {
-class FileSystem;
-}
-
 /// This is a utility class used to parse user-provided text files with
 /// "special case lists" for code sanitizers. Such files are used to
 /// define an "ABI list" for DataFlowSanitizer and allow/exclusion lists for
diff --git a/llvm/include/llvm/Support/VirtualFileSystem.h b/llvm/include/llvm/Support/VirtualFileSystem.h
index d22c534228331..288f9817976e6 100644
--- a/llvm/include/llvm/Support/VirtualFileSystem.h
+++ b/llvm/include/llvm/Support/VirtualFileSystem.h
@@ -27,6 +27,7 @@
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SourceMgr.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <atomic>
 #include <cassert>
 #include <cstdint>
@@ -215,8 +216,6 @@ class directory_iterator {
   }
 };
 
-class FileSystem;
-
 namespace detail {
 
 /// Keeps state for the recursive_directory_iterator.
diff --git a/llvm/include/llvm/Support/VirtualFileSystemFwd.h b/llvm/include/llvm/Support/VirtualFileSystemFwd.h
new file mode 100644
index 0000000000000..fc971f7aa9b40
--- /dev/null
+++ b/llvm/include/llvm/Support/VirtualFileSystemFwd.h
@@ -0,0 +1,36 @@
+//===- VirtualFileSystemFwd.h - Virtual File System Forward -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// Contains the forward declaration for vfs::FileSystem, as well as the
+/// IntrusiveRefCntPtrInfo specialization for it. This allows the
+/// vfs::FileSystem class to be used with IntrusiveRefCntPtr without including
+/// its full definition.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
+#define LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
+
+#include "llvm/ADT/IntrusiveRefCntPtr.h"
+#include "llvm/Support/Compiler.h"
+
+namespace llvm {
+namespace vfs {
+class FileSystem;
+} // namespace vfs
+
+template <> struct LLVM_ABI IntrusiveRefCntPtrInfo<::llvm::vfs::FileSystem> {
+  static unsigned useCount(const ::llvm::vfs::FileSystem *FS);
+  static void retain(::llvm::vfs::FileSystem *FS);
+  static void release(::llvm::vfs::FileSystem *FS);
+};
+
+} // namespace llvm
+
+#endif // LLVM_SUPPORT_VIRTUALFILESYSTEMFWD_H
diff --git a/llvm/include/llvm/Transforms/IPO/SampleProfile.h b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
index ab8c754c16fbe..a3496b596dff6 100644
--- a/llvm/include/llvm/Transforms/IPO/SampleProfile.h
+++ b/llvm/include/llvm/Transforms/IPO/SampleProfile.h
@@ -19,6 +19,7 @@
 #include "llvm/Pass.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include <string>
 
 namespace llvm {
@@ -32,10 +33,6 @@ LLVM_ABI extern cl::opt<int> ProfileInlineLimitMin;
 LLVM_ABI extern cl::opt<int> ProfileInlineLimitMax;
 LLVM_ABI extern cl::opt<bool> SortProfiledSCC;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 /// The sample profiler data loader pass.
 class SampleProfileLoaderPass
     : public OptionalPassInfoMixin<SampleProfileLoaderPass> {
diff --git a/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h b/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
index 01104097180e4..3afd88f144859 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/MemProfUse.h
@@ -17,16 +17,13 @@
 #include "llvm/ProfileData/DataAccessProf.h"
 #include "llvm/ProfileData/MemProf.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 
 namespace llvm {
 class IndexedInstrProfReader;
 class Module;
 class TargetLibraryInfo;
 
-namespace vfs {
-class FileSystem;
-} // namespace vfs
-
 class MemProfUsePass : public OptionalPassInfoMixin<MemProfUsePass> {
 public:
   LLVM_ABI explicit MemProfUsePass(
diff --git a/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h b/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
index 4cc0253ba0baf..1246c29ca3d46 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/SanitizerBinaryMetadata.h
@@ -18,12 +18,10 @@
 #include "llvm/IR/Module.h"
 #include "llvm/IR/PassManager.h"
 #include "llvm/Support/Compiler.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/Instrumentation.h"
 
 namespace llvm {
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 struct SanitizerBinaryMetadataOptions {
   bool Covered = false;
diff --git a/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h b/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
index 269bcbcd32265..1d8b6fc2441c6 100644
--- a/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
+++ b/llvm/include/llvm/Transforms/Instrumentation/SanitizerCoverage.h
@@ -19,13 +19,11 @@
 #include "llvm/IR/PassManager.h"
 #include "llvm/Support/Compiler.h"
 #include "llvm/Support/SpecialCaseList.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Transforms/Utils/Instrumentation.h"
 
 namespace llvm {
 class Module;
-namespace vfs {
-class FileSystem;
-} // namespace vfs
 
 /// This is the ModuleSanitizerCoverage pass used in the new pass manager. The
 /// pass instruments functions for coverage, adds initialization calls to the
diff --git a/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h b/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
index 5346871249cf5..c7262fb35ab95 100644
--- a/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
+++ b/llvm/include/llvm/Transforms/Utils/SampleProfileLoaderBaseImpl.h
@@ -40,6 +40,7 @@
 #include "llvm/ProfileData/SampleProfReader.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/GenericDomTree.h"
+#include "llvm/Support/VirtualFileSystemFwd.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Transforms/Utils/SampleProfileInference.h"
 #include "llvm/Transforms/Utils/SampleProfileLoaderBaseUtil.h"
@@...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211761

From llvmlistbot at llvm.org  Fri Jul 24 02:59:10 2026
From: llvmlistbot at llvm.org (Ege Beysel)
Date: Fri, 24 Jul 2026 02:59:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a63376e.21a19891.105e40.7fe0@mx.google.com>


egebeysel wrote:

JFYI: I realised that this table has not been updated with this PR, the docs are generated from here:

https://github.com/llvm/llvm-project/blob/383e2662157dfea354edf140bc25477776bc5de9/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td#L3399-L3418

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul 24 03:18:05 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 03:18:05 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [NVVM] Add FP8 (e4m3/e5m2) support to dense
	mma.sync (PR #207307)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/207307@github.com>
Message-ID: <6a633bdd.3cce8983.247e47.3d28@mx.google.com>


weimin023 wrote:

> JFYI: I realised that this table has not been updated with this PR, the docs are generated from here:
> 
> https://github.com/llvm/llvm-project/blob/383e2662157dfea354edf140bc25477776bc5de9/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td#L3399-L3418

Ok, I’ll file a separate ticket to update the documentation ASAP.

https://github.com/llvm/llvm-project/pull/207307

From llvmlistbot at llvm.org  Fri Jul 24 03:45:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 03:45:16 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Support for "!$omp
	dispatch". (PR #203320)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/203320@github.com>
Message-ID: <6a63423c.f3151ff8.f584a.abe1@mx.google.com>


https://github.com/SunilKuravinakop updated https://github.com/llvm/llvm-project/pull/203320

>From d007fc287982ed638ef3900307ca700f0d529f87 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 10:56:08 -0500
Subject: [PATCH 1/8] Support for "!$omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 33 ++++++-
 flang/test/Integration/OpenMP/dispatch.f90    | 76 ++++++++++++++++
 flang/test/Lower/OpenMP/Todo/dispatch.f90     | 12 ---
 flang/test/Lower/OpenMP/dispatch.f90          | 89 +++++++++++++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 15 ++++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 17 ++++
 mlir/test/Dialect/OpenMP/dispatch.mlir        | 77 ++++++++++++++++
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  | 81 +++++++++++++++++
 8 files changed, 385 insertions(+), 15 deletions(-)
 create mode 100644 flang/test/Integration/OpenMP/dispatch.f90
 delete mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90
 create mode 100644 flang/test/Lower/OpenMP/dispatch.f90
 create mode 100644 mlir/test/Dialect/OpenMP/dispatch.mlir
 create mode 100644 mlir/test/Target/LLVMIR/openmp-dispatch.mlir

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index be876c563433a..758d8eb8148e5 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -2592,6 +2592,18 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -3940,6 +3952,9 @@ static void genOMPDispatch(lower::AbstractConverter &converter,
   case llvm::omp::Directive::OMPD_barrier:
     newOp = genBarrierOp(converter, symTable, semaCtx, eval, loc, queue, item);
     break;
+  case llvm::omp::Directive::OMPD_dispatch:
+    newOp = genDispatchOp(converter, symTable, semaCtx, eval, loc, queue, item);
+    break;
   case llvm::omp::Directive::OMPD_distribute:
     newOp = genStandaloneDistribute(converter, symTable, stmtCtx, semaCtx, eval,
                                     loc, queue, item);
@@ -5312,9 +5327,21 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
-                   const parser::OpenMPDispatchConstruct &) {
-  if (!semaCtx.langOptions().OpenMPSimd)
-    TODO(converter.getCurrentLocation(), "OpenMPDispatchConstruct");
+                   const parser::OpenMPDispatchConstruct &dispatchConstruct) {
+  const parser::OmpDirectiveSpecification &beginSpec =
+      dispatchConstruct.BeginDir();
+  List<Clause> clauses = makeClauses(beginSpec.Clauses(), semaCtx);
+  if (auto &endSpec = dispatchConstruct.EndDir())
+    clauses.append(makeClauses(endSpec->Clauses(), semaCtx));
+
+  llvm::omp::Directive directive = beginSpec.DirId();
+  mlir::Location currentLocation = converter.genLocation(beginSpec.source);
+
+  ConstructQueue queue{
+      buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
+                          eval, beginSpec.source, directive, clauses)};
+  genOMPDispatch(converter, symTable, semaCtx, eval, currentLocation, queue,
+                 queue.begin());
 }
 
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..c4d16690c8e29
--- /dev/null
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -0,0 +1,76 @@
+!===----------------------------------------------------------------------===!
+! This directory can be used to add Integration tests involving multiple
+! stages of the compiler (for eg. from Fortran to LLVM IR). It should not
+! contain executable tests. We should only add tests here sparingly and only
+! if there is no other way to test. Repeat this message in each test that is
+! added to this directory and sub-directories.
+!===----------------------------------------------------------------------===!
+
+!RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
+
+!CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
+!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
+!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
+!CHECK: [[IF_TRUE]]:
+!CHECK: call void @_QMfuncsPfoo_variant2()
+!CHECK: [[IF_FALSE]]:
+!CHECK: call void @_QMfuncsPfoo_variant1()
+
+!CHECK-LABEL: define void @_QQmain()
+!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region
+!CHECK: omp.dispatch.region:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont
+!CHECK: omp.region.cont:
+!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
+!CHECK: br label %omp.dispatch.region2
+!CHECK: omp.dispatch.region2:
+!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: br label %omp.region.cont1
+!CHECK: omp.region.cont1:
+
+module funcs
+  implicit none
+  logical :: foo_sub
+
+contains
+
+  subroutine foo_variant1()
+    print *, "in foo_variant1"
+  end subroutine
+
+  subroutine foo_variant2()
+    print *, "in foo_variant2"
+  end subroutine
+
+  subroutine foo_dispatch()
+    if (foo_sub) then
+      call foo_variant2()
+    else
+      call foo_variant1()
+    end if
+  end subroutine
+
+end module funcs
+
+program dispatch_test
+  use funcs
+  implicit none
+
+  foo_sub = .false.
+  !$omp dispatch
+  call foo_dispatch()
+
+  foo_sub = .true.
+  !$omp dispatch
+  call foo_dispatch()
+
+end program
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
deleted file mode 100644
index 380dfa14eaae1..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ /dev/null
@@ -1,12 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OpenMPDispatchConstruct
-program p
-  integer r
-  r = 1
-!$omp dispatch nowait
-  call foo()
-contains
-  subroutine foo
-  end subroutine
-end program p
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
new file mode 100644
index 0000000000000..df86503817163
--- /dev/null
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -0,0 +1,89 @@
+!RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
+!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
+
+!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
+!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
+!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
+!HLFIR: fir.if %[[CMP_H]] {
+!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!HLFIR: } else {
+!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!HLFIR: }
+
+!FIR-LABEL: func @_QMfuncsPfoo_dispatch
+!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
+!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
+!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
+!FIR: fir.if %[[CMP_F]] {
+!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
+!FIR: } else {
+!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
+!FIR: }
+
+module funcs
+  implicit none
+
+contains
+
+  subroutine variant1()
+    print *, "in variant1"
+  end subroutine
+
+  subroutine variant2()
+    print *, "in variant2"
+  end subroutine
+
+  subroutine foo_dispatch(x)
+    integer, intent(in) :: x
+    if (x == 1) then
+      call variant1()
+    else
+      call variant2()
+    end if
+  end subroutine
+
+end module funcs
+
+!HLFIR-LABEL: func @_QQmain
+!FIR-LABEL: func @_QQmain
+!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
+!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
+!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+program dispatch_test
+  use funcs
+  implicit none
+  integer :: x
+
+  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
+  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
+  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
+  x = 1
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+
+  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
+  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
+  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
+  x = 2
+  !HLFIR: omp.dispatch {
+  !FIR: omp.dispatch {
+  !$omp dispatch
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !FIR:   omp.terminator
+  !HLFIR: }
+  !FIR: }
+end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 51e7080db5b29..6170e40a835c6 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2286,6 +2286,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
   ];
 }
 
+//===----------------------------------------------------------------------===//
+// [Spec 5.1] 12.3 dispatch Construct
+//===----------------------------------------------------------------------===//
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+  let summary = "dispatch construct";
+  let description = [{
+    The dispatch construct enables the invocation of a variant of a
+    base procedure. The structured block of a dispatch construct is a
+    single expression statement that contains a function call or a
+    subroutine call.
+  }];
+
+  let assemblyFormat = "$region attr-dict";
+}
+
 //===----------------------------------------------------------------------===//
 // [Spec 5.2] 6.5 allocate Directive
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index d35e8612e158b..12818a5e56ac6 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -761,6 +761,20 @@ static llvm::omp::ProcBindKind getProcBindKind(omp::ClauseProcBindKind kind) {
   llvm_unreachable("Unknown ClauseProcBindKind kind");
 }
 
+/// Convert 'dispatch' operation into LLVM IR.
+static LogicalResult
+convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
+                   LLVM::ModuleTranslation &moduleTranslation) {
+  auto dispatchOp = cast<omp::DispatchOp>(opInst);
+  auto &region = dispatchOp.getRegion();
+  auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
+                                    moduleTranslation);
+  if (!result)
+    return handleError(result.takeError(), opInst);
+  builder.SetInsertPoint(*result);
+  return success();
+}
+
 /// Converts an OpenMP 'masked' operation into LLVM IR using OpenMPIRBuilder.
 static LogicalResult
 convertOmpMasked(Operation &opInst, llvm::IRBuilderBase &builder,
@@ -8815,6 +8829,9 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })
+          .Case([&](omp::DispatchOp) {
+            return convertOmpDispatch(*op, builder, moduleTranslation);
+          })
           .Case([&](omp::MaskedOp) {
             return convertOmpMasked(*op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
new file mode 100644
index 0000000000000..c935f56836185
--- /dev/null
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -0,0 +1,77 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// CHECK-LABEL: func @foo_dispatch
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @foo_dispatch(%x : memref<i32>) -> () {
+  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
+  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
+  %v = memref.load %x[] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  %cmp = arith.cmpi eq, %v, %c1 : i32
+  cf.cond_br %cmp, ^bb1, ^bb2
+// CHECK: ^[[BB1]]:
+// CHECK: call @variant1() : () -> ()
+^bb1:
+  func.call @variant1() : () -> ()
+  cf.br ^bb3
+// CHECK: ^[[BB2]]:
+// CHECK: call @variant2() : () -> ()
+^bb2:
+  func.call @variant2() : () -> ()
+  cf.br ^bb3
+^bb3:
+  return
+}
+
+// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
+// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// Test multiple dispatch regions with stores selecting different variants.
+// CHECK-LABEL: func @omp_dispatch_multiple
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
+  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
+  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
+  %c1 = arith.constant 1 : i32
+  memref.store %c1, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
+  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
+  %c2 = arith.constant 2 : i32
+  memref.store %c2, %x[] : memref<i32>
+  // CHECK: omp.dispatch {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  "omp.dispatch" () ({
+    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
+    "omp.terminator" () : () -> ()
+  }) : () -> ()
+  return
+}
+
+// CHECK-LABEL: func private @variant1()
+// CHECK-LABEL: func private @variant2()
+func.func private @variant1() -> ()
+func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
new file mode 100644
index 0000000000000..8a3a7a6b5d8e6
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -0,0 +1,81 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
+  %0 = llvm.mlir.constant(0 : i32) : i32
+  llvm.return %0 : i32
+}
+
+llvm.func @variant1() -> ()
+llvm.func @variant2() -> ()
+
+// CHECK-LABEL: define void @foo_dispatch()
+llvm.func @foo_dispatch() {
+  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
+  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
+  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %1 = llvm.load %0 : !llvm.ptr -> i32
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  %cmp = llvm.icmp "eq" %1, %c1 : i32
+  llvm.cond_br %cmp, ^bb1, ^bb2
+// CHECK: [[BB1]]:
+// CHECK: call void @variant1()
+^bb1:
+  llvm.call @variant1() : () -> ()
+  llvm.br ^bb3
+// CHECK: [[BB2]]:
+// CHECK: call void @variant2()
+^bb2:
+  llvm.call @variant2() : () -> ()
+  llvm.br ^bb3
+^bb3:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch()
+llvm.func @test_omp_dispatch() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  llvm.return
+}
+
+// CHECK-LABEL: define void @test_omp_dispatch_multiple()
+llvm.func @test_omp_dispatch_multiple() {
+  // CHECK: store i32 1, ptr @x
+  %0 = llvm.mlir.addressof @x : !llvm.ptr
+  %c1 = llvm.mlir.constant(1 : i32) : i32
+  llvm.store %c1, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region
+  omp.dispatch {
+    // CHECK: omp.dispatch.region:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont
+    omp.terminator
+  }
+  // CHECK: omp.region.cont:
+  // CHECK: store i32 2, ptr @x
+  %c2 = llvm.mlir.constant(2 : i32) : i32
+  llvm.store %c2, %0 : i32, !llvm.ptr
+  // CHECK: br label %omp.dispatch.region2
+  omp.dispatch {
+    // CHECK: omp.dispatch.region2:
+    // CHECK-NEXT: call void @foo_dispatch()
+    llvm.call @foo_dispatch() : () -> ()
+    // CHECK-NEXT: br label %omp.region.cont1
+    omp.terminator
+  }
+  // CHECK: omp.region.cont1:
+  llvm.return
+}

>From b50c11a322615d4e189303f9b4b73e1c6246c7ae Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 11 Jun 2026 12:19:36 -0500
Subject: [PATCH 2/8] Checking for no caluses support with "omp dispatch".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 3 +++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 7 ++++---
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 758d8eb8148e5..71bc392d0e8d9 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -5337,6 +5337,9 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
+  if (!clauses.empty())
+    TODO(currentLocation, "OpenMP Dispatch clauses");
+
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 6170e40a835c6..47c7587c5d08b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2293,9 +2293,10 @@ def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure. The structured block of a dispatch construct is a
-    single expression statement that contains a function call or a
-    subroutine call.
+    base procedure (without any construct's clauses: device, is_device_ptr,
+    nowait, nocontext, novariants). The structured block of a dispatch
+    construct is a single expression statement that contains a function
+    call or a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";

>From 3ca936ebb5aff08e318fcd4abede5951e77fcb07 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Thu, 18 Jun 2026 11:50:04 -0500
Subject: [PATCH 3/8] Handling feedbacks suggested by Sergio Afonso.

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 37 ++++++++++++-------
 flang/test/Lower/OpenMP/dispatch.f90          | 26 +------------
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 11 +++---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  4 ++
 mlir/test/Dialect/OpenMP/dispatch.mlir        |  1 +
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir  |  1 +
 6 files changed, 36 insertions(+), 44 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 33e67ed2c3ab7..fe7d17807727a 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1661,6 +1661,16 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
   cp.processOrder(clauseOps);
 }
 
+static void genDispatchClauses(lower::AbstractConverter &converter,
+                               semantics::SemanticsContext &semaCtx,
+                               const List<Clause> &clauses,
+                               mlir::Location loc) {
+  ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
+                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+      loc, llvm::omp::Directive::OMPD_dispatch);
+}
+
 static void genFlushClauses(lower::AbstractConverter &converter,
                             semantics::SemanticsContext &semaCtx,
                             const ObjectList &objects,
@@ -2154,6 +2164,18 @@ genCriticalOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item, nameAttr);
 }
 
+static mlir::omp::DispatchOp
+genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
+              semantics::SemanticsContext &semaCtx,
+              lower::pft::Evaluation &eval, mlir::Location loc,
+              const ConstructQueue &queue,
+              ConstructQueue::const_iterator item) {
+  return genOpWithBody<mlir::omp::DispatchOp>(
+      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
+                        llvm::omp::Directive::OMPD_dispatch),
+      queue, item);
+}
+
 static mlir::omp::FlushOp
 genFlushOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
            semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -2592,18 +2614,6 @@ genMasterOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
       queue, item);
 }
 
-static mlir::omp::DispatchOp
-genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
-              semantics::SemanticsContext &semaCtx,
-              lower::pft::Evaluation &eval, mlir::Location loc,
-              const ConstructQueue &queue,
-              ConstructQueue::const_iterator item) {
-  return genOpWithBody<mlir::omp::DispatchOp>(
-      OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
-                        llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
-}
-
 static mlir::omp::OrderedOp
 genOrderedOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
              semantics::SemanticsContext &semaCtx, lower::pft::Evaluation &eval,
@@ -5294,8 +5304,7 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  if (!clauses.empty())
-    TODO(currentLocation, "OpenMP Dispatch clauses");
+  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
 
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index df86503817163..ecc7b1b31454f 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,5 +1,4 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
-!RUN: %flang_fc1 -emit-mlir -fopenmp %s -o - | FileCheck %s --check-prefix=FIR
 
 !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
 !HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
@@ -12,16 +11,6 @@
 !HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
 !HLFIR: }
 
-!FIR-LABEL: func @_QMfuncsPfoo_dispatch
-!FIR: %[[XD_F:.*]] = fir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!FIR: %[[LOAD_F:.*]] = fir.load %[[XD_F]] : !fir.ref<i32>
-!FIR: %[[CMP_F:.*]] = arith.cmpi eq, %[[LOAD_F]], %{{.*}} : i32
-!FIR: fir.if %[[CMP_F]] {
-!FIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!FIR: } else {
-!FIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!FIR: }
-
 module funcs
   implicit none
 
@@ -35,6 +24,7 @@ subroutine variant2()
     print *, "in variant2"
   end subroutine
 
+  !TODO: replace with declare_variant when the support is merged.
   subroutine foo_dispatch(x)
     integer, intent(in) :: x
     if (x == 1) then
@@ -47,10 +37,6 @@ subroutine foo_dispatch(x)
 end module funcs
 
 !HLFIR-LABEL: func @_QQmain
-!FIR-LABEL: func @_QQmain
-!FIR-DAG: %[[C1:.*]] = arith.constant 1 : i32
-!FIR-DAG: %[[C2:.*]] = arith.constant 2 : i32
-!FIR: %[[X:.*]] = fir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
 program dispatch_test
   use funcs
   implicit none
@@ -59,31 +45,21 @@ program dispatch_test
   !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
   !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
   !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C1]] to %[[X]] : !fir.ref<i32>
   x = 1
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 
   !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
   !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  !FIR: fir.store %[[C2]] to %[[X]] : !fir.ref<i32>
   x = 2
   !HLFIR: omp.dispatch {
-  !FIR: omp.dispatch {
   !$omp dispatch
   !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-  !FIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]) {{.*}}: (!fir.ref<i32>) -> ()
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
-  !FIR:   omp.terminator
   !HLFIR: }
-  !FIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 47c7587c5d08b..490bee24799c2 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,14 +2289,15 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true> {
+def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
+   ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
-    base procedure (without any construct's clauses: device, is_device_ptr,
-    nowait, nocontext, novariants). The structured block of a dispatch
-    construct is a single expression statement that contains a function
-    call or a subroutine call.
+    base procedure. The structured block of a dispatch construct is
+    a single expression statement that contains a function call or
+    a subroutine call.
   }];
 
   let assemblyFormat = "$region attr-dict";
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 12818a5e56ac6..04c36b87d73ad 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -766,6 +766,10 @@ static LogicalResult
 convertOmpDispatch(Operation &opInst, llvm::IRBuilderBase &builder,
                    LLVM::ModuleTranslation &moduleTranslation) {
   auto dispatchOp = cast<omp::DispatchOp>(opInst);
+
+  if (failed(checkImplementationStatus(opInst)))
+    return failure();
+
   auto &region = dispatchOp.getRegion();
   auto result = convertOmpOpRegions(region, "omp.dispatch.region", builder,
                                     moduleTranslation);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c935f56836185..b8d4b93a6c8e3 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -41,6 +41,7 @@ func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
 }
 
 // Test multiple dispatch regions with stores selecting different variants.
+// TODO: Use declare_variant when the support is merged.
 // CHECK-LABEL: func @omp_dispatch_multiple
 // CHECK-SAME: (%[[X:.*]]: memref<i32>)
 func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index 8a3a7a6b5d8e6..d947a871ca30c 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -8,6 +8,7 @@ llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
 llvm.func @variant1() -> ()
 llvm.func @variant2() -> ()
 
+// TODO: Add support for declare_variant, when support to it is merged.
 // CHECK-LABEL: define void @foo_dispatch()
 llvm.func @foo_dispatch() {
   // CHECK: %[[ADDR:.*]] = load i32, ptr @x

>From 6eacbc3cd3d8e278d7b51ea898fe27e009de2228 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 01:09:09 -0500
Subject: [PATCH 4/8] 1) Handling -fc1 -fopenmp-simd 2) Todo testcase for
 clauses not supported in dispatch

---
 flang/lib/Semantics/rewrite-parse-tree.cpp |  5 +++
 flang/test/Lower/OpenMP/Todo/dispatch.f90  | 43 ++++++++++++++++++++++
 flang/test/Semantics/OpenMP/simd-only.f90  | 12 ++++++
 3 files changed, 60 insertions(+)
 create mode 100644 flang/test/Lower/OpenMP/Todo/dispatch.f90

diff --git a/flang/lib/Semantics/rewrite-parse-tree.cpp b/flang/lib/Semantics/rewrite-parse-tree.cpp
index 4e1c9bae9c153..76a60c75488e3 100644
--- a/flang/lib/Semantics/rewrite-parse-tree.cpp
+++ b/flang/lib/Semantics/rewrite-parse-tree.cpp
@@ -191,6 +191,11 @@ void RewriteMutator::OpenMPSimdOnly(
                        &omp->value().u)}) {
           it = replaceInlineBlock(std::get<parser::Block>(ompBlock->t), it);
           continue;
+        } else if (auto *ompDispatch{
+                       std::get_if<parser::OpenMPDispatchConstruct>(
+                           &omp->value().u)}) {
+          it = replaceInlineBlock(std::get<parser::Block>(ompDispatch->t), it);
+          continue;
         } else if (auto *ompLoop{std::get_if<parser::OpenMPLoopConstruct>(
                        &omp->value().u)}) {
           if (LoopConstructIsSIMD(ompLoop)) {
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
new file mode 100644
index 0000000000000..f100cb40ba3d7
--- /dev/null
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -0,0 +1,43 @@
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEPEND -o - %s 2>&1 | FileCheck %s --check-prefix=DEPEND
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DDEVICE -o - %s 2>&1 | FileCheck %s --check-prefix=DEVICE
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
+! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
+
+! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
+! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
+! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
+! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
+! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
+! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
+
+subroutine sub()
+#ifdef IS_DEVICE_PTR
+  use iso_c_binding
+  type(c_ptr) :: x
+#endif
+  integer :: r
+#ifdef DEPEND
+!$omp dispatch depend(inout: r)
+#endif
+#ifdef DEVICE
+!$omp dispatch device(0)
+#endif
+#ifdef IS_DEVICE_PTR
+!$omp dispatch is_device_ptr(x)
+#endif
+#ifdef NOCONTEXT
+!$omp dispatch nocontext(.false.)
+#endif
+#ifdef NOVARIANTS
+!$omp dispatch novariants(.true.)
+#endif
+#ifdef NOWAIT
+!$omp dispatch nowait
+#endif
+  call foo()
+contains
+  subroutine foo
+  end subroutine
+end subroutine sub
diff --git a/flang/test/Semantics/OpenMP/simd-only.f90 b/flang/test/Semantics/OpenMP/simd-only.f90
index 01370af0a00a7..860befaf43247 100644
--- a/flang/test/Semantics/OpenMP/simd-only.f90
+++ b/flang/test/Semantics/OpenMP/simd-only.f90
@@ -414,3 +414,15 @@ module test_declare_mapper
   ! CHECK-NOT: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpDeclareMapperDirective
   !$omp declare mapper(myvec_t :: v) map(v, v%data(1:v%len))
 end module
+
+! CHECK-LABEL: Name = 'test_dispatch'
+subroutine test_dispatch()
+  ! CHECK-NOT: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OpenMPDispatchConstruct
+  ! CHECK-NOT: OmpDirectiveName -> llvm::omp::Directive = dispatch
+  ! CHECK: ExecutionPartConstruct -> ExecutableConstruct -> ActionStmt -> CallStmt
+  !$omp dispatch
+  call foo()
+contains
+  subroutine foo()
+  end subroutine
+end subroutine

>From 8f25901612c660e29e53ab9c7e162a21a04ad966 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 3 Jul 2026 07:14:38 -0500
Subject: [PATCH 5/8] Support for "dispatch nowait" except for MLIR to LLVMIR.
 Support in LLVM IR depends on "declare variant".

---
 flang/lib/Lower/OpenMP/OpenMP.cpp                 | 14 ++++++++------
 flang/test/Lower/OpenMP/Todo/dispatch.f90         |  5 -----
 flang/test/Lower/OpenMP/dispatch.f90              |  7 +++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td     | 12 ++++++++----
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp      |  9 +++++++++
 .../Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  |  1 +
 mlir/test/Dialect/OpenMP/dispatch.mlir            | 15 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir          | 11 +++++++++++
 8 files changed, 59 insertions(+), 15 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 4c2a238605f8d..959f7352e2c76 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -1774,11 +1774,12 @@ static void genDistributeClauses(lower::AbstractConverter &converter,
 
 static void genDispatchClauses(lower::AbstractConverter &converter,
                                semantics::SemanticsContext &semaCtx,
-                               const List<Clause> &clauses,
-                               mlir::Location loc) {
+                               const List<Clause> &clauses, mlir::Location loc,
+                               mlir::omp::DispatchOperands &clauseOps) {
   ClauseProcessor cp(converter, semaCtx, clauses);
+  cp.processNowait(clauseOps);
   cp.processTODO<clause::Depend, clause::Device, clause::IsDevicePtr,
-                 clause::Nocontext, clause::Novariants, clause::Nowait>(
+                 clause::Nocontext, clause::Novariants>(
       loc, llvm::omp::Directive::OMPD_dispatch);
 }
 
@@ -2276,10 +2277,13 @@ genDispatchOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
               lower::pft::Evaluation &eval, mlir::Location loc,
               const ConstructQueue &queue,
               ConstructQueue::const_iterator item) {
+  mlir::omp::DispatchOperands clauseOps;
+  genDispatchClauses(converter, semaCtx, item->clauses, loc, clauseOps);
+
   return genOpWithBody<mlir::omp::DispatchOp>(
       OpWithBodyGenInfo(converter, symTable, semaCtx, loc, eval,
                         llvm::omp::Directive::OMPD_dispatch),
-      queue, item);
+      queue, item, clauseOps);
 }
 
 static mlir::omp::FlushOp
@@ -5426,8 +5430,6 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
   llvm::omp::Directive directive = beginSpec.DirId();
   mlir::Location currentLocation = converter.genLocation(beginSpec.source);
 
-  genDispatchClauses(converter, semaCtx, clauses, currentLocation);
-
   ConstructQueue queue{
       buildConstructQueue(converter.getFirOpBuilder().getModule(), semaCtx,
                           eval, beginSpec.source, directive, clauses)};
diff --git a/flang/test/Lower/OpenMP/Todo/dispatch.f90 b/flang/test/Lower/OpenMP/Todo/dispatch.f90
index f100cb40ba3d7..579d47d457def 100644
--- a/flang/test/Lower/OpenMP/Todo/dispatch.f90
+++ b/flang/test/Lower/OpenMP/Todo/dispatch.f90
@@ -3,14 +3,12 @@
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DIS_DEVICE_PTR -o - %s 2>&1 | FileCheck %s --check-prefix=IS_DEVICE_PTR
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOCONTEXT -o - %s 2>&1 | FileCheck %s --check-prefix=NOCONTEXT
 ! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOVARIANTS -o - %s 2>&1 | FileCheck %s --check-prefix=NOVARIANTS
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=52 -cpp -DNOWAIT -o - %s 2>&1 | FileCheck %s --check-prefix=NOWAIT
 
 ! DEPEND: not yet implemented: Unhandled clause DEPEND in DISPATCH construct
 ! DEVICE: not yet implemented: Unhandled clause DEVICE in DISPATCH construct
 ! IS_DEVICE_PTR: not yet implemented: Unhandled clause IS_DEVICE_PTR in DISPATCH construct
 ! NOCONTEXT: not yet implemented: Unhandled clause NOCONTEXT in DISPATCH construct
 ! NOVARIANTS: not yet implemented: Unhandled clause NOVARIANTS in DISPATCH construct
-! NOWAIT: not yet implemented: Unhandled clause NOWAIT in DISPATCH construct
 
 subroutine sub()
 #ifdef IS_DEVICE_PTR
@@ -32,9 +30,6 @@ subroutine sub()
 #endif
 #ifdef NOVARIANTS
 !$omp dispatch novariants(.true.)
-#endif
-#ifdef NOWAIT
-!$omp dispatch nowait
 #endif
   call foo()
 contains
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index ecc7b1b31454f..1b184a18a91af 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -62,4 +62,11 @@ program dispatch_test
     call foo_dispatch(x)
   !HLFIR:   omp.terminator
   !HLFIR: }
+
+  !HLFIR: omp.dispatch nowait {
+  !$omp dispatch nowait
+  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
+    call foo_dispatch(x)
+  !HLFIR:   omp.terminator
+  !HLFIR: }
 end program
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index f048a9da71063..bb478c52381c5 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2289,17 +2289,21 @@ def MaskedOp : OpenMP_Op<"masked", clauses = [
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
 def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
-     // TODO: Complete clause lists (device, is_device_ptr,nowait, nocontext, novariants).
-   ]> {
+    // TODO: Complete clause list (device, depend, is_device_ptr, nocontext,
+    // novariants).
+    OpenMP_NowaitClause
+  ]> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a
     base procedure. The structured block of a dispatch construct is
     a single expression statement that contains a function call or
     a subroutine call.
-  }];
+  }] # clausesDescription;
 
-  let assemblyFormat = "$region attr-dict";
+  let builders = [
+    OpBuilder<(ins CArg<"const DispatchOperands &">:$clauses)>
+  ];
 }
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index c2956a1cf7b79..6c1d5c29ec95d 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4933,6 +4933,15 @@ void MaskedOp::build(OpBuilder &builder, OperationState &state,
   MaskedOp::build(builder, state, clauses.filteredThreadId);
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Dispatch construct (12.3)
+//===----------------------------------------------------------------------===//
+
+void DispatchOp::build(OpBuilder &builder, OperationState &state,
+                       const DispatchOperands &clauses) {
+  DispatchOp::build(builder, state, clauses.nowait);
+}
+
 //===----------------------------------------------------------------------===//
 // Spec 5.2: Scan construct (5.6)
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index e84daaa135748..75c5e707b9933 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -480,6 +480,7 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkDepend(op, result);
         checkNowait(op, result);
       })
+      .Case([&](omp::DispatchOp op) { checkNowait(op, result); })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
         checkInReductionByref(op, result);
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index b8d4b93a6c8e3..c966784b1e22e 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -72,6 +72,21 @@ func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
   return
 }
 
+// Test the nowait clause on omp.dispatch.
+// CHECK-LABEL: func @omp_dispatch_nowait
+// CHECK-SAME: (%[[X:.*]]: memref<i32>)
+func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
+  // CHECK: omp.dispatch nowait {
+  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
+  // CHECK-NEXT: omp.terminator
+  // CHECK-NEXT: }
+  omp.dispatch nowait {
+    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
+    omp.terminator
+  }
+  return
+}
+
 // CHECK-LABEL: func private @variant1()
 // CHECK-LABEL: func private @variant2()
 func.func private @variant1() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 377a5bb799be4..1bf9b563f8d6c 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -52,6 +52,17 @@ llvm.func @distribute_order(%lb : i32, %ub : i32, %step : i32) {
 
 // -----
 
+llvm.func @dispatch_nowait() {
+  // expected-error at below {{not yet implemented: Unhandled clause nowait in omp.dispatch operation}}
+  // expected-error at below {{LLVM Translation failed for operation: omp.dispatch}}
+  omp.dispatch nowait {
+    omp.terminator
+  }
+  llvm.return
+}
+
+// -----
+
 llvm.func @parallel_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.parallel operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.parallel}}

>From 7739eecf4c336629aedfa11911a4bea1cc81a6e8 Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 02:20:10 -0500
Subject: [PATCH 6/8] Integrating variants using "declare variant" in the
 tests. Adding dispatch trait to "declare variant" in Utils.cpp Handling
 comments by Chi Chun Chen.

---
 flang/lib/Lower/OpenMP/Utils.cpp             |   3 +
 flang/test/Integration/OpenMP/dispatch.f90   |  42 +-----
 flang/test/Lower/OpenMP/dispatch.f90         |  59 +++-----
 mlir/test/Dialect/OpenMP/dispatch.mlir       | 150 ++++++++-----------
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir |  97 +++---------
 5 files changed, 109 insertions(+), 242 deletions(-)

diff --git a/flang/lib/Lower/OpenMP/Utils.cpp b/flang/lib/Lower/OpenMP/Utils.cpp
index 8f57f00d59c58..5cb6a0713462f 100644
--- a/flang/lib/Lower/OpenMP/Utils.cpp
+++ b/flang/lib/Lower/OpenMP/Utils.cpp
@@ -1379,6 +1379,9 @@ void collectEnclosingConstructTraits(
   for (; op; op = op->getParentOp()) {
     if (mlir::isa<mlir::omp::WsloopOp>(op))
       constructTraits.push_back(llvm::omp::TraitProperty::construct_for_for);
+    if (mlir::isa<mlir::omp::DispatchOp>(op))
+      constructTraits.push_back(
+          llvm::omp::TraitProperty::construct_dispatch_dispatch);
     if (mlir::isa<mlir::omp::ParallelOp>(op))
       constructTraits.push_back(
           llvm::omp::TraitProperty::construct_parallel_parallel);
diff --git a/flang/test/Integration/OpenMP/dispatch.f90 b/flang/test/Integration/OpenMP/dispatch.f90
index c4d16690c8e29..1ecccab5e90d4 100644
--- a/flang/test/Integration/OpenMP/dispatch.f90
+++ b/flang/test/Integration/OpenMP/dispatch.f90
@@ -8,55 +8,32 @@
 
 !RUN: %flang_fc1 -emit-llvm -fopenmp %s -o - | FileCheck %s
 
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant1()
-!CHECK: call ptr @_FortranAioBeginExternalListOutput
-
-!CHECK-LABEL: define void @_QMfuncsPfoo_variant2()
+!CHECK-LABEL: define void @_QMfuncsPfoo_variant()
 !CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
-!CHECK: %[[COND:.*]] = load i32, ptr @_QMfuncsEfoo_sub
-!CHECK: %[[CMP:.*]] = icmp ne i32 %[[COND]], 0
-!CHECK: br i1 %[[CMP]], label %[[IF_TRUE:.*]], label %[[IF_FALSE:.*]]
-!CHECK: [[IF_TRUE]]:
-!CHECK: call void @_QMfuncsPfoo_variant2()
-!CHECK: [[IF_FALSE]]:
-!CHECK: call void @_QMfuncsPfoo_variant1()
+!CHECK: call ptr @_FortranAioBeginExternalListOutput
 
 !CHECK-LABEL: define void @_QQmain()
-!CHECK: store i32 0, ptr @_QMfuncsEfoo_sub
+!CHECK: call void @_QMfuncsPfoo_dispatch()
 !CHECK: br label %omp.dispatch.region
 !CHECK: omp.dispatch.region:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
+!CHECK: call void @_QMfuncsPfoo_variant()
 !CHECK: br label %omp.region.cont
 !CHECK: omp.region.cont:
-!CHECK: store i32 1, ptr @_QMfuncsEfoo_sub
-!CHECK: br label %omp.dispatch.region2
-!CHECK: omp.dispatch.region2:
-!CHECK: call void @_QMfuncsPfoo_dispatch()
-!CHECK: br label %omp.region.cont1
-!CHECK: omp.region.cont1:
 
 module funcs
   implicit none
-  logical :: foo_sub
 
 contains
 
-  subroutine foo_variant1()
-    print *, "in foo_variant1"
-  end subroutine
-
-  subroutine foo_variant2()
-    print *, "in foo_variant2"
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
   subroutine foo_dispatch()
-    if (foo_sub) then
-      call foo_variant2()
-    else
-      call foo_variant1()
-    end if
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -65,11 +42,8 @@ program dispatch_test
   use funcs
   implicit none
 
-  foo_sub = .false.
-  !$omp dispatch
   call foo_dispatch()
 
-  foo_sub = .true.
   !$omp dispatch
   call foo_dispatch()
 
diff --git a/flang/test/Lower/OpenMP/dispatch.f90 b/flang/test/Lower/OpenMP/dispatch.f90
index 1b184a18a91af..4eb9439b7b35b 100644
--- a/flang/test/Lower/OpenMP/dispatch.f90
+++ b/flang/test/Lower/OpenMP/dispatch.f90
@@ -1,37 +1,23 @@
 !RUN: %flang_fc1 -emit-hlfir -fopenmp %s -o - | FileCheck %s --check-prefix=HLFIR
 
-!HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
-!HLFIR: %[[XD_H:.*]]:2 = hlfir.declare %{{.*}} {{{.*}}uniq_name = {{.*}}foo_dispatch{{.*}}x{{.*}}
-!HLFIR: %[[LOAD_H:.*]] = fir.load %[[XD_H]]#0 : !fir.ref<i32>
-!HLFIR: %[[C1_H:.*]] = arith.constant 1 : i32
-!HLFIR: %[[CMP_H:.*]] = arith.cmpi eq, %[[LOAD_H]], %[[C1_H]] : i32
-!HLFIR: fir.if %[[CMP_H]] {
-!HLFIR:   fir.call @_QMfuncsPvariant1() {{.*}}: () -> ()
-!HLFIR: } else {
-!HLFIR:   fir.call @_QMfuncsPvariant2() {{.*}}: () -> ()
-!HLFIR: }
+! Variant selection is provided by DECLARE VARIANT with a `construct={dispatch}`
+! match: inside a dispatch region the call to the base procedure `foo_dispatch`
+! is replaced by a call to its variant `foo_variant`.
 
 module funcs
   implicit none
 
 contains
 
-  subroutine variant1()
-    print *, "in variant1"
+  !HLFIR-LABEL: func @_QMfuncsPfoo_variant
+  subroutine foo_variant()
+    print *, "in foo_variant"
   end subroutine
 
-  subroutine variant2()
-    print *, "in variant2"
-  end subroutine
-
-  !TODO: replace with declare_variant when the support is merged.
-  subroutine foo_dispatch(x)
-    integer, intent(in) :: x
-    if (x == 1) then
-      call variant1()
-    else
-      call variant2()
-    end if
+  !HLFIR-LABEL: func @_QMfuncsPfoo_dispatch
+  subroutine foo_dispatch()
+    !$omp declare variant(foo_dispatch:foo_variant) match(construct={dispatch})
+    print *, "in foo_dispatch"
   end subroutine
 
 end module funcs
@@ -40,33 +26,22 @@ end module funcs
 program dispatch_test
   use funcs
   implicit none
-  integer :: x
 
-  !HLFIR: %[[X:.*]]:2 = hlfir.declare %{{.*}} {uniq_name = {{.*}}x{{.*}}
-  !HLFIR: %[[C1:.*]] = arith.constant 1 : i32
-  !HLFIR: hlfir.assign %[[C1]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 1
-  !HLFIR: omp.dispatch {
-  !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
-  !HLFIR:   omp.terminator
-  !HLFIR: }
+  ! A call outside any dispatch region targets the base procedure.
+  !HLFIR: fir.call @_QMfuncsPfoo_dispatch() {{.*}}: () -> ()
+  call foo_dispatch()
 
-  !HLFIR: %[[C2:.*]] = arith.constant 2 : i32
-  !HLFIR: hlfir.assign %[[C2]] to %[[X]]#0 : i32, !fir.ref<i32>
-  x = 2
   !HLFIR: omp.dispatch {
   !$omp dispatch
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 
   !HLFIR: omp.dispatch nowait {
   !$omp dispatch nowait
-  !HLFIR:   fir.call @_QMfuncsPfoo_dispatch(%[[X]]#0) {{.*}}: (!fir.ref<i32>) -> ()
-    call foo_dispatch(x)
+  !HLFIR:   fir.call @_QMfuncsPfoo_variant() {{.*}}: () -> ()
+    call foo_dispatch()
   !HLFIR:   omp.terminator
   !HLFIR: }
 end program
diff --git a/mlir/test/Dialect/OpenMP/dispatch.mlir b/mlir/test/Dialect/OpenMP/dispatch.mlir
index c966784b1e22e..304753ed4bcc0 100644
--- a/mlir/test/Dialect/OpenMP/dispatch.mlir
+++ b/mlir/test/Dialect/OpenMP/dispatch.mlir
@@ -1,93 +1,63 @@
-// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+// RUN: fir-opt %s | FileCheck %s
 
-// CHECK-LABEL: func @foo_dispatch
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @foo_dispatch(%x : memref<i32>) -> () {
-  // CHECK: %[[V:.*]] = memref.load %[[X]][] : memref<i32>
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: %[[CMP:.*]] = arith.cmpi eq, %[[V]], %[[C1]] : i32
-  // CHECK: cf.cond_br %[[CMP]], ^[[BB1:.*]], ^[[BB2:.*]]
-  %v = memref.load %x[] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  %cmp = arith.cmpi eq, %v, %c1 : i32
-  cf.cond_br %cmp, ^bb1, ^bb2
-// CHECK: ^[[BB1]]:
-// CHECK: call @variant1() : () -> ()
-^bb1:
-  func.call @variant1() : () -> ()
-  cf.br ^bb3
-// CHECK: ^[[BB2]]:
-// CHECK: call @variant2() : () -> ()
-^bb2:
-  func.call @variant2() : () -> ()
-  cf.br ^bb3
-^bb3:
-  return
-}
-
-// Test that the generic form of omp.dispatch roundtrips to pretty-printed form.
-// CHECK-LABEL: func @omp_dispatch_generic_to_pretty
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_generic_to_pretty(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test multiple dispatch regions with stores selecting different variants.
-// TODO: Use declare_variant when the support is merged.
-// CHECK-LABEL: func @omp_dispatch_multiple
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_multiple(%x : memref<i32>) -> () {
-  // CHECK: %[[C1:.*]] = arith.constant 1 : i32
-  // CHECK: memref.store %[[C1]], %[[X]][] : memref<i32>
-  %c1 = arith.constant 1 : i32
-  memref.store %c1, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  // CHECK: %[[C2:.*]] = arith.constant 2 : i32
-  // CHECK: memref.store %[[C2]], %[[X]][] : memref<i32>
-  %c2 = arith.constant 2 : i32
-  memref.store %c2, %x[] : memref<i32>
-  // CHECK: omp.dispatch {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  "omp.dispatch" () ({
-    "func.call" (%x) {callee = @foo_dispatch} : (memref<i32>) -> ()
-    "omp.terminator" () : () -> ()
-  }) : () -> ()
-  return
-}
-
-// Test the nowait clause on omp.dispatch.
-// CHECK-LABEL: func @omp_dispatch_nowait
-// CHECK-SAME: (%[[X:.*]]: memref<i32>)
-func.func @omp_dispatch_nowait(%x : memref<i32>) -> () {
-  // CHECK: omp.dispatch nowait {
-  // CHECK-NEXT: func.call @foo_dispatch(%[[X]]) : (memref<i32>) -> ()
-  // CHECK-NEXT: omp.terminator
-  // CHECK-NEXT: }
-  omp.dispatch nowait {
-    func.call @foo_dispatch(%x) : (memref<i32>) -> ()
-    omp.terminator
+module attributes {dlti.dl_spec = #dlti.dl_spec<!llvm.ptr<270> = dense<32> : vector<4xi64>, !llvm.ptr<271> = dense<32> : vector<4xi64>, !llvm.ptr<272> = dense<64> : vector<4xi64>, i64 = dense<64> : vector<2xi64>, i128 = dense<128> : vector<2xi64>, f80 = dense<128> : vector<2xi64>, !llvm.ptr = dense<64> : vector<4xi64>, i1 = dense<8> : vector<2xi64>, i8 = dense<8> : vector<2xi64>, i16 = dense<16> : vector<2xi64>, i32 = dense<32> : vector<2xi64>, f16 = dense<16> : vector<2xi64>, f64 = dense<64> : vector<2xi64>, f128 = dense<128> : vector<2xi64>, "dlti.endianness" = "little", "dlti.mangling_mode" = "e", "dlti.legal_int_widths" = array<i32: 8, 16, 32, 64>, "dlti.stack_alignment" = 128 : i64>, fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", fir.relocation_model = 1 : i32, llvm.data_layout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128", llvm.ident = "flang version 24.0.0 (https://github.com/SunilKuravinakop/llvm-project.git 665c356fed252c9f65f99e7609997822e0e0a908)", llvm.target_triple = "x86_64-unknown-linux-gnu", omp.flags = #omp.flags<openmp_device_version = 31>, omp.is_gpu = false, omp.is_target_device = false, omp.target_triples = [], omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_variant
+  func.func @_QMfuncsPfoo_variant() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c35_i32 = arith.constant 35 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c35_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F76617269616E74) : !fir.ref<!fir.char<1,14>>
+    %c14 = arith.constant 14 : index
+    %5:2 = hlfir.declare %4 typeparams %c14 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F76617269616E74"} : (!fir.ref<!fir.char<1,14>>, index) -> (!fir.ref<!fir.char<1,14>>, !fir.ref<!fir.char<1,14>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,14>>) -> !fir.ref<i8>
+    %7 = fir.convert %c14 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QMfuncsPfoo_dispatch
+  func.func @_QMfuncsPfoo_dispatch() {
+    %0 = fir.dummy_scope : !fir.dscope
+    %c6_i32 = arith.constant 6 : i32
+    %1 = fir.address_of(@_QQclXa0e1d7990b4b8f5533e92e3c2cc80a5d) : !fir.ref<!fir.char<1,89>>
+    %2 = fir.convert %1 : (!fir.ref<!fir.char<1,89>>) -> !fir.ref<i8>
+    %c40_i32 = arith.constant 40 : i32
+    %3 = fir.call @_FortranAioBeginExternalListOutput(%c6_i32, %2, %c40_i32) fastmath<contract> : (i32, !fir.ref<i8>, i32) -> !fir.ref<i8>
+    %4 = fir.address_of(@_QQclX696E20666F6F5F6469737061746368) : !fir.ref<!fir.char<1,15>>
+    %c15 = arith.constant 15 : index
+    %5:2 = hlfir.declare %4 typeparams %c15 {fortran_attrs = #fir.var_attrs<parameter>, uniq_name = "_QQclX696E20666F6F5F6469737061746368"} : (!fir.ref<!fir.char<1,15>>, index) -> (!fir.ref<!fir.char<1,15>>, !fir.ref<!fir.char<1,15>>)
+    %6 = fir.convert %5#0 : (!fir.ref<!fir.char<1,15>>) -> !fir.ref<i8>
+    %7 = fir.convert %c15 : (index) -> i64
+    %8 = fir.call @_FortranAioOutputAscii(%3, %6, %7) fastmath<contract> : (!fir.ref<i8>, !fir.ref<i8>, i64) -> i1
+    %9 = fir.call @_FortranAioEndIoStatement(%3) fastmath<contract> : (!fir.ref<i8>) -> i32
+    return
+  }
+  // CHECK-LABEL: func.func @_QQmain
+  func.func @_QQmain() attributes {fir.bindc_name = "DISPATCH_TEST"} {
+    %0 = fir.dummy_scope : !fir.dscope
+    // CHECK: fir.call @_QMfuncsPfoo_dispatch()
+    fir.call @_QMfuncsPfoo_dispatch() fastmath<contract> : () -> ()
+    // CHECK: omp.dispatch {
+    omp.dispatch {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
+  }
+  // CHECK-LABEL: func.func @omp_dispatch_nowait
+  func.func @omp_dispatch_nowait() {
+    // CHECK: omp.dispatch nowait {
+    omp.dispatch nowait {
+      // CHECK: fir.call @_QMfuncsPfoo_variant()
+      fir.call @_QMfuncsPfoo_variant() fastmath<contract> : () -> ()
+      // CHECK: omp.terminator
+      omp.terminator
+    }
+    return
   }
-  return
 }
-
-// CHECK-LABEL: func private @variant1()
-// CHECK-LABEL: func private @variant2()
-func.func private @variant1() -> ()
-func.func private @variant2() -> ()
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index d947a871ca30c..8a3f07f0ebbb9 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -1,82 +1,27 @@
 // RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
 
-llvm.mlir.global external @x() {addr_space = 0 : i32} : i32 {
-  %0 = llvm.mlir.constant(0 : i32) : i32
-  llvm.return %0 : i32
-}
-
-llvm.func @variant1() -> ()
-llvm.func @variant2() -> ()
-
-// TODO: Add support for declare_variant, when support to it is merged.
-// CHECK-LABEL: define void @foo_dispatch()
-llvm.func @foo_dispatch() {
-  // CHECK: %[[ADDR:.*]] = load i32, ptr @x
-  // CHECK: %[[CMP:.*]] = icmp eq i32 %[[ADDR]], 1
-  // CHECK: br i1 %[[CMP]], label %[[BB1:.*]], label %[[BB2:.*]]
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %1 = llvm.load %0 : !llvm.ptr -> i32
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  %cmp = llvm.icmp "eq" %1, %c1 : i32
-  llvm.cond_br %cmp, ^bb1, ^bb2
-// CHECK: [[BB1]]:
-// CHECK: call void @variant1()
-^bb1:
-  llvm.call @variant1() : () -> ()
-  llvm.br ^bb3
-// CHECK: [[BB2]]:
-// CHECK: call void @variant2()
-^bb2:
-  llvm.call @variant2() : () -> ()
-  llvm.br ^bb3
-^bb3:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch()
-llvm.func @test_omp_dispatch() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+module attributes {omp.is_target_device = false, omp.is_gpu = false, omp.version = #omp.version<version = 31>} {
+  // CHECK-LABEL: define void @_QMfuncsPfoo_variant()
+  llvm.func @_QMfuncsPfoo_variant() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  llvm.return
-}
-
-// CHECK-LABEL: define void @test_omp_dispatch_multiple()
-llvm.func @test_omp_dispatch_multiple() {
-  // CHECK: store i32 1, ptr @x
-  %0 = llvm.mlir.addressof @x : !llvm.ptr
-  %c1 = llvm.mlir.constant(1 : i32) : i32
-  llvm.store %c1, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region
-  omp.dispatch {
-    // CHECK: omp.dispatch.region:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont
-    omp.terminator
+  // CHECK-LABEL: define void @_QMfuncsPfoo_dispatch()
+  llvm.func @_QMfuncsPfoo_dispatch() {
+    llvm.return
   }
-  // CHECK: omp.region.cont:
-  // CHECK: store i32 2, ptr @x
-  %c2 = llvm.mlir.constant(2 : i32) : i32
-  llvm.store %c2, %0 : i32, !llvm.ptr
-  // CHECK: br label %omp.dispatch.region2
-  omp.dispatch {
-    // CHECK: omp.dispatch.region2:
-    // CHECK-NEXT: call void @foo_dispatch()
-    llvm.call @foo_dispatch() : () -> ()
-    // CHECK-NEXT: br label %omp.region.cont1
-    omp.terminator
+  // CHECK-LABEL: define void @_QQmain()
+  llvm.func @_QQmain() {
+    // CHECK: call void @_QMfuncsPfoo_dispatch()
+    llvm.call @_QMfuncsPfoo_dispatch() : () -> ()
+    // CHECK: br label %omp.dispatch.region
+    // CHECK: omp.dispatch.region:
+    omp.dispatch {
+      // CHECK: call void @_QMfuncsPfoo_variant()
+      llvm.call @_QMfuncsPfoo_variant() : () -> ()
+      // CHECK: br label %omp.region.cont
+      omp.terminator
+    }
+    // CHECK: omp.region.cont:
+    llvm.return
   }
-  // CHECK: omp.region.cont1:
-  llvm.return
 }

>From d515daa0dbde17522438d7cd8b095e553c2306db Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 02:28:33 -0500
Subject: [PATCH 7/8] Handling Chi Chun Chen comments on "NoRegionArguments"

---
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index b8fc6174f9776..93169976e2c8b 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -2328,11 +2328,13 @@ def MaskedOp : OpenMP_Op<"masked", traits = [
 //===----------------------------------------------------------------------===//
 // [Spec 5.1] 12.3 dispatch Construct
 //===----------------------------------------------------------------------===//
-def DispatchOp : OpenMP_Op<"dispatch", singleRegion = true, clauses = [
+def DispatchOp : OpenMP_Op<"dispatch", traits = [
+    NoRegionArguments
+  ], clauses = [
     // TODO: Complete clause list (device, depend, is_device_ptr, nocontext,
     // novariants).
     OpenMP_NowaitClause
-  ]> {
+  ], singleRegion = true> {
   let summary = "dispatch construct";
   let description = [{
     The dispatch construct enables the invocation of a variant of a

>From 882d0fe84493d889a0c868869c5424498ccfa27a Mon Sep 17 00:00:00 2001
From: Sunil Kuravinakop <kuravina at pe31.hpc.amslabs.hpecorp.net>
Date: Fri, 24 Jul 2026 05:42:18 -0500
Subject: [PATCH 8/8] Support for dispatch nowait.

---
 .../Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  |  1 -
 mlir/test/Target/LLVMIR/openmp-dispatch.mlir      | 15 +++++++++++++++
 mlir/test/Target/LLVMIR/openmp-todo.mlir          | 11 -----------
 3 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 59a2192e5d855..886f70596ec53 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -512,7 +512,6 @@ static LogicalResult checkImplementationStatus(Operation &op) {
         checkTaskReductionByref(op, result);
       })
       .Case([&](omp::TaskwaitOp op) { checkNowait(op, result); })
-      .Case([&](omp::DispatchOp op) { checkNowait(op, result); })
       .Case([&](omp::TaskloopContextOp op) {
         checkAllocate(op, result);
         checkInReduction(op, result);
diff --git a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
index 8a3f07f0ebbb9..931511b836e8b 100644
--- a/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-dispatch.mlir
@@ -24,4 +24,19 @@ module attributes {omp.is_target_device = false, omp.is_gpu = false, omp.version
     // CHECK: omp.region.cont:
     llvm.return
   }
+  // The nowait clause is accepted; it is a no-op in the current synchronous
+  // inline lowering, producing the same dispatch region.
+  // CHECK-LABEL: define void @test_dispatch_nowait()
+  llvm.func @test_dispatch_nowait() {
+    // CHECK: br label %omp.dispatch.region
+    // CHECK: omp.dispatch.region:
+    omp.dispatch nowait {
+      // CHECK: call void @_QMfuncsPfoo_variant()
+      llvm.call @_QMfuncsPfoo_variant() : () -> ()
+      // CHECK: br label %omp.region.cont
+      omp.terminator
+    }
+    // CHECK: omp.region.cont:
+    llvm.return
+  }
 }
diff --git a/mlir/test/Target/LLVMIR/openmp-todo.mlir b/mlir/test/Target/LLVMIR/openmp-todo.mlir
index 3e78e28be8c67..e29659255f690 100644
--- a/mlir/test/Target/LLVMIR/openmp-todo.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-todo.mlir
@@ -52,17 +52,6 @@ llvm.func @distribute_order(%lb : i32, %ub : i32, %step : i32) {
 
 // -----
 
-llvm.func @dispatch_nowait() {
-  // expected-error at below {{not yet implemented: Unhandled clause nowait in omp.dispatch operation}}
-  // expected-error at below {{LLVM Translation failed for operation: omp.dispatch}}
-  omp.dispatch nowait {
-    omp.terminator
-  }
-  llvm.return
-}
-
-// -----
-
 llvm.func @parallel_allocate(%x : !llvm.ptr) {
   // expected-error at below {{not yet implemented: Unhandled clause allocate in omp.parallel operation}}
   // expected-error at below {{LLVM Translation failed for operation: omp.parallel}}


From llvmlistbot at llvm.org  Fri Jul 24 04:13:01 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:13:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a6348bd.0c05e68e.81651.b342@mx.google.com>


https://github.com/banach-space edited https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:17:11 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:17:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a6349b7.510f576d.33aa7d.3c7a@mx.google.com>


banach-space wrote:

I've updated the title - IMHO it makes the intent clearer.

> Thanks for the contribution!
> 
> I'm not sure what you are trying to achieve. The intent of this pass is to minimize redundant computation on the same data. For your example:
> 
> ```
>   %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
>   %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
>   %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
>   %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
> ```
> 
> I would expect that broadcasts on `%arg0` and `%arg2` are removed but `%arg1` is broadcasted to `<4xf32>`. Then FMA would happen on `<4xf32>`, followed by the broadcast to `<3x4xf32>`.

@dcaballe , your example is a bit complex and from what I can tell, there's no logic to support that. However, the pattern will currently also fail for even basic examples (note, no scalars at all):

```mlir
func.func @fma_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: vector<4xf32>, %arg2: vector<4xf32>) -> vector<3x4xf32> {
  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
  return %3 : vector<3x4xf32>
}
```

IIUC, the intent of this PR is to unblock trivial examples like the one above. I agree with your comment that the following example should also be supported:
```mlir
  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
  %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
```

That said, this PR seems fine to me as is, hence approving.

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:17:39 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:17:39 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a6349d3.eb165e5c.2e0c9a.bbf4@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you!

Lets wait for @dcaballe to also approve before landing. 

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:21:55 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Fri, 24 Jul 2026 04:21:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a634ad3.92ea0e2a.2ed880.1d25@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/210952

>From 73227259bfaa8e5b8866fb38e1b20277a7d08650 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:54 +0200
Subject: [PATCH 1/2] [mlir][vector] Reject non-unit strides on
 masked/expand/compress ops

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 51 ++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir    | 77 ++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 9322c11d401d9..8477ff57d967d 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
+/// statically known non-unit stride; a dynamic (not provably unit) stride
+/// is accepted.
+///
+/// This is more permissive than vector.load/store's stride check: ops such
+/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
+/// also used on memrefs whose most minor dimension is contiguous at runtime
+/// but not provable as such at the type level (e.g., buffers produced by
+/// the sparsifier).
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6326,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6395,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6681,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6705,6 +6747,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index 0097db39f6ed9..d01c2ad4e0c7b 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1741,6 +1759,24 @@ func.func @expand_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16x
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1795,6 +1831,24 @@ func.func @compress_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
@@ -2178,6 +2232,19 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
+// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
+// (unprovable) stride is rejected here too: vector.load/store require a
+// statically known unit stride, with no exception for strides that merely
+// aren't provably non-unit.
+func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
+  %0 = vector.load %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.store
 //===----------------------------------------------------------------------===//
@@ -2214,6 +2281,16 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
+// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
+// unlike vector.maskedstore.
+func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
+  // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
+  vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 func.func @store_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %val: vector<4xf32>) {
   // expected-error @+2 {{'vector.store' op memref strides must be non-negative}}
   %c0 = arith.constant 0 : index

>From 60221acb5a0c7c37a320a68c62df5021b3914b55 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Fri, 24 Jul 2026 13:21:16 +0200
Subject: [PATCH 2/2] Address comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 43 +++++-------------------
 1 file changed, 8 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 8477ff57d967d..07a0aca09e179 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,33 +6190,6 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
-/// Verifies that `memRefTy`'s most minor dimension does not have a
-/// statically known non-unit stride; a dynamic (not provably unit) stride
-/// is accepted.
-///
-/// This is more permissive than vector.load/store's stride check: ops such
-/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
-/// also used on memrefs whose most minor dimension is contiguous at runtime
-/// but not provable as such at the type level (e.g., buffers produced by
-/// the sparsifier).
-static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
-                                                          VectorType vecTy,
-                                                          MemRefType memRefTy) {
-  if (!vecTy.isScalable() &&
-      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
-    return success();
-
-  SmallVector<int64_t> strides;
-  int64_t offset;
-  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
-    return success();
-
-  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
-      strides.back() != 1)
-    return op->emitOpError("most minor memref dim must have unit stride");
-  return success();
-}
-
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6326,8 +6299,8 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6395,8 +6368,8 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6681,8 +6654,8 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.expandload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6747,8 +6720,8 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.compressstore. The lowering
   // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that


From llvmlistbot at llvm.org  Fri Jul 24 04:23:08 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 24 Jul 2026 04:23:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a634b1c.6c5720b3.1f1229.b408@mx.google.com>


https://github.com/dcaballe approved this pull request.

Thanks for clarifying. It make sense!

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:23:08 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 24 Jul 2026 04:23:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a634b1c.21a19891.105e40.1e2a@mx.google.com>


https://github.com/dcaballe edited https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:23:08 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 24 Jul 2026 04:23:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a634b1c.666420a4.3304c7.9172@mx.google.com>


================
@@ -1068,6 +1062,17 @@ struct ReorderElementwiseOpsOnBroadcast final
     Type unbroadcastResultType =
         cloneOrReplace(broadcastSource.getType(), resultElemType);
 
+    // Some ops, e.g. `vector.fma`, only accept vector types. For such ops the
+    // reordering is only possible when the broadcast source is a vector as
+    // well; sinking past a broadcast from a scalar would create an invalid op.
+    // TODO: Support scalar sources for `vector.fma` by rewriting to
+    // `math.fma`.
----------------
dcaballe wrote:

I'm not sure the semantics of `math.fma` match `vector.fma`. This would probably be better supported by promoting the scalar to a single element vector

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:23:08 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 24 Jul 2026 04:23:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a634b1c.b4a331f0.2642a.2c09@mx.google.com>


================
@@ -195,24 +195,91 @@ func.func @source_and_result_mismatch(%arg0 : f32) -> vector<1xi1> {
 
 // -----
 
-// vector.fma only supports vectors - currently it's not possible to replace this with e.g.:
+// vector.fma only supports vectors - when the broadcast source is a scalar,
+// currently it's not possible to replace this with e.g.:
 //    %scalar_res = vector.fma %scalar_1, %scalar2
 //    %vec_res = vector.broadcast %scalar_res
 //
-// TODO: It should be possible to support this case
+// TODO: It should be possible to support this case by rewriting to math.fma
----------------
dcaballe wrote:

Same

https://github.com/llvm/llvm-project/pull/211208

From llvmlistbot at llvm.org  Fri Jul 24 04:23:12 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Fri, 24 Jul 2026 04:23:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a634b20.92d2873c.2cdd42.4507@mx.google.com>


================
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
----------------
FedericoBruzzone wrote:

Thanks, you're right. I simply used `memType.isLastDimUnitStride` :D

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Fri Jul 24 04:24:45 2026
From: llvmlistbot at llvm.org (Ramkumar Ramachandra)
Date: Fri, 24 Jul 2026 04:24:45 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
 strictfp consistent v2 (PR #211769)
Message-ID: <llvm.org/llvm/llvm-project/pull/211769@github.com>


https://github.com/artagnon created https://github.com/llvm/llvm-project/pull/211769

Changes: The patch was originally landed as af0c933e (#209465), but there was an issue with strictfp functions being inlined into non-strictfp functions. The issue has since been resolved by a2b42b10a ([IR] Reject inlining strictfp alwaysinline callee into non-strictfp caller, #210701) and the re-land is identical.

Although the section on constrainedfp in the LangRef clearly states "All function definitions that use constrained floating point intrinsics must have the strictfp attribute", indicating that a function with strictfp calls must be marked with strictfp, the general description of strictfp does not specify this. Refine its semantics and make it so, eliminating the inconsistency.

>From 0696b2be9b4821a46e8442bedffaf301f415bf8c Mon Sep 17 00:00:00 2001
From: Ramkumar Ramachandra <artagnon at tenstorrent.com>
Date: Sat, 18 Jul 2026 14:49:40 +0100
Subject: [PATCH] [IR] Make semantics of strictfp consistent v2

Changes: The patch was originally landed as af0c933e (#209465), but
there was an issue with strictfp functions being inlined into
non-strictfp functions. The issue has since been resolved by a2b42b10a
([IR] Reject inlining strictfp alwaysinline callee into non-strictfp
caller, #210701) and the re-land is identical.

Although the section on constrainedfp in the LangRef clearly states "All
function definitions that use constrained floating point intrinsics must
have the strictfp attribute", indicating that a function with strictfp
calls must be marked with strictfp, the general description of strictfp
does not specify this. Refine its semantics and make it so, eliminating
the inconsistency.
---
 clang/lib/CodeGen/TargetInfo.cpp              |  3 +
 .../cl20-device-side-enqueue-attributes.cl    |  4 +-
 llvm/docs/LangRef.md                          |  6 +-
 llvm/include/llvm/IR/Function.h               |  3 +
 llvm/lib/IR/Verifier.cpp                      |  5 ++
 llvm/test/CodeGen/AArch64/is_fpclass.ll       |  4 +-
 .../sms-loop-carried-fp-exceptions1.mir       |  2 +-
 .../sve-streaming-mode-cvt-fp-to-int.ll       |  4 +-
 .../sve-streaming-mode-cvt-int-to-fp.ll       |  4 +-
 llvm/test/CodeGen/PowerPC/milicode32.ll       |  2 +-
 .../ppcf128-constrained-fp-intrinsics.ll      | 16 +----
 .../CodeGen/RISCV/double-intrinsics-strict.ll |  2 +-
 .../CodeGen/RISCV/float-intrinsics-strict.ll  |  2 +-
 .../RISCV/zfh-half-intrinsics-strict.ll       |  2 +-
 .../RISCV/zfhmin-half-intrinsics-strict.ll    |  2 +-
 .../llvm-intrinsics/constrained-fmuladd.ll    | 12 ++--
 .../EarlyCSE/replace-calls-def-attrs.ll       | 67 ++++++++++---------
 llvm/test/Transforms/InstCombine/erf.ll       | 40 ++++++-----
 llvm/test/Transforms/InstCombine/fdim.ll      |  7 +-
 .../InstSimplify/disable_folding.ll           | 14 ++--
 .../merge-compatible-invokes-of-landingpad.ll | 30 +++++----
 llvm/unittests/IR/VerifierTest.cpp            | 24 +++++++
 .../test/Target/LLVMIR/llvmir-intrinsics.mlir | 50 +++++++-------
 23 files changed, 169 insertions(+), 136 deletions(-)

diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index d6477b61bcb0e..ed46430ae979b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2800,7 +2800,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26632,9 +26633,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c663b1ec440ea..a7682ee246fe0 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3897,6 +3897,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "call site marked strictfp without caller function marked strictfp",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -151,7 +151,7 @@ F:
 define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_preserves_return_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call nonnull ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
@@ -172,8 +172,8 @@ F:
 define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -184,11 +184,12 @@ define i8 @same_parent_combine_diff_attrs_todo(i8 %x, i8 %y) {
 
 }
 
-define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
+define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4:[0-9]+]]
+; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -202,10 +203,10 @@ define i8 @same_parent_combine_diff_attrs_fail(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs_todo(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_todo(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5:[0-9]+]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR6:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -223,13 +224,14 @@ F:
   ret i8 %r2
 }
 
-define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) {
+define i8 @diff_parent_combine_diff_attrs_fail(i1 %c, i8 %x, i8 %y) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs_fail(
-; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR4]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR5]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:       [[F]]:
@@ -247,10 +249,11 @@ F:
   ret i8 %r2
 }
 
-define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
+define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) strictfp {
+; CHECK: Function Attrs: strictfp
 ; CHECK-LABEL: define i32 @commutative_intrinsic_intersection_failure(
-; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) {
-; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR6:[0-9]+]]
+; CHECK-SAME: i32 [[ARG:%.*]], i32 [[ARG1:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG]], i32 [[ARG1]]) #[[ATTR0]]
 ; CHECK-NEXT:    [[CALL2:%.*]] = call i32 @llvm.smin.i32(i32 [[ARG1]], i32 [[ARG]])
 ; CHECK-NEXT:    [[OR:%.*]] = or i32 [[CALL2]], [[CALL]]
 ; CHECK-NEXT:    ret i32 [[OR]]
@@ -262,11 +265,11 @@ define i32 @commutative_intrinsic_intersection_failure(i32 %arg, i32 %arg1) {
 }
 
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
-; CHECK: attributes #[[ATTR1]] = { memory(none) }
-; CHECK: attributes #[[ATTR2]] = { memory(read) }
-; CHECK: attributes #[[ATTR3]] = { alwaysinline memory(none) }
-; CHECK: attributes #[[ATTR4]] = { strictfp memory(none) }
-; CHECK: attributes #[[ATTR5]] = { noinline optnone memory(none) }
-; CHECK: attributes #[[ATTR6]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { nocallback nocreateundeforpoison nofree nosync nounwind speculatable willreturn memory(none) }
+; CHECK: attributes #[[ATTR2]] = { memory(none) }
+; CHECK: attributes #[[ATTR3]] = { memory(read) }
+; CHECK: attributes #[[ATTR4]] = { alwaysinline memory(none) }
+; CHECK: attributes #[[ATTR5]] = { strictfp memory(none) }
+; CHECK: attributes #[[ATTR6]] = { noinline optnone memory(none) }
 ;.
diff --git a/llvm/test/Transforms/InstCombine/erf.ll b/llvm/test/Transforms/InstCombine/erf.ll
index 9f83acabb6fb7..57015fb490a7e 100644
--- a/llvm/test/Transforms/InstCombine/erf.ll
+++ b/llvm/test/Transforms/InstCombine/erf.ll
@@ -75,7 +75,7 @@ define double @erf_inf() {
 
 define float @erff_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +inf) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF0000000000000) readnone
@@ -84,7 +84,7 @@ define float @erff_inf_memory_none() {
 
 define double @erf_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF0000000000000) readnone
@@ -111,7 +111,7 @@ define double @erf_neg_inf() {
 
 define float @erff_neg_inf_memory_none() {
 ; CHECK-LABEL: define float @erff_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0xFFF0000000000000) readnone
@@ -120,7 +120,7 @@ define float @erff_neg_inf_memory_none() {
 
 define double @erf_neg_inf_memory_none() {
 ; CHECK-LABEL: define double @erf_neg_inf_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -inf) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0xFFF0000000000000) readnone
@@ -147,7 +147,7 @@ define double @erf_nan() {
 
 define float @erff_nan_memory_none() {
 ; CHECK-LABEL: define float @erff_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) readnone
@@ -156,7 +156,7 @@ define float @erff_nan_memory_none() {
 
 define double @erf_nan_memory_none() {
 ; CHECK-LABEL: define double @erf_nan_memory_none() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR1]]
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) readnone
@@ -181,36 +181,40 @@ define double @erf_poison() {
   ret double %r
 }
 
-define float @erff_const_strictfp() {
-; CHECK-LABEL: define float @erff_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR2:[0-9]+]]
+define float @erff_const_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 5.000000e-01) strictfp
   ret float %r
 }
 
-define double @erf_const_strictfp() {
-; CHECK-LABEL: define double @erf_const_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR2]]
+define double @erf_const_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_const_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double -5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double -5.000000e-01) strictfp
   ret double %r
 }
 
-define float @erff_nan_strictfp() {
-; CHECK-LABEL: define float @erff_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR2]]
+define float @erff_nan_strictfp() strictfp {
+; CHECK-LABEL: define float @erff_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call float @erff(float +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[R]]
 ;
   %r = call float @erff(float 0x7FF8000000000000) strictfp
   ret float %r
 }
 
-define double @erf_nan_strictfp() {
-; CHECK-LABEL: define double @erf_nan_strictfp() {
-; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR2]]
+define double @erf_nan_strictfp() strictfp {
+; CHECK-LABEL: define double @erf_nan_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[R:%.*]] = call double @erf(double +qnan) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[R]]
 ;
   %r = call double @erf(double 0x7FF8000000000000) strictfp
diff --git a/llvm/test/Transforms/InstCombine/fdim.ll b/llvm/test/Transforms/InstCombine/fdim.ll
index edbdaedc975c0..f167a842642db 100644
--- a/llvm/test/Transforms/InstCombine/fdim.ll
+++ b/llvm/test/Transforms/InstCombine/fdim.ll
@@ -99,9 +99,10 @@ define double @fdim_nzero() {
   ret double %dim
 }
 
-define double @fdim_strictfp() {
-; CHECK-LABEL: define double @fdim_strictfp() {
-; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR1:[0-9]+]]
+define double @fdim_strictfp() strictfp {
+; CHECK-LABEL: define double @fdim_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[DIM:%.*]] = call double @fdim(double 1.000000e+01, double 8.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret double [[DIM]]
 ;
   %dim = call double @fdim(double 10.0, double 8.0) strictfp
diff --git a/llvm/test/Transforms/InstSimplify/disable_folding.ll b/llvm/test/Transforms/InstSimplify/disable_folding.ll
index 69892012629bc..02ec4877e51d9 100644
--- a/llvm/test/Transforms/InstSimplify/disable_folding.ll
+++ b/llvm/test/Transforms/InstSimplify/disable_folding.ll
@@ -31,9 +31,10 @@ define float @test_llvm_sin() {
 }
 
 ; Should not be folded, even when -disable-fp-call-folding is not set, as it is marked as strictfp.
-define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR1:[0-9]+]]
+define float @test_fmax_ftz_nan_f_strictfp() #1 {
+; CHECK-LABEL: define float @test_fmax_ftz_nan_f_strictfp(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.250000e+00, float -2.000000e+00) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.nvvm.fmax.ftz.nan.f(float 1.25, float -2.0) #1
@@ -42,9 +43,10 @@ define float @test_fmax_ftz_nan_f_strictfp() {
 
 ; Check that strictfp disables folding for LLVM math intrinsics like sin.f32
 ; even when -disable-fp-call-folding is not set.
-define float @test_llvm_sin_strictfp() {
-; CHECK-LABEL: define float @test_llvm_sin_strictfp() {
-; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR1]]
+define float @test_llvm_sin_strictfp() #1 {
+; CHECK-LABEL: define float @test_llvm_sin_strictfp(
+; CHECK-SAME: ) #[[ATTR0]] {
+; CHECK-NEXT:    [[RES:%.*]] = call float @llvm.sin.f32(float 5.000000e-01) #[[ATTR0]]
 ; CHECK-NEXT:    ret float [[RES]]
 ;
   %res = call float @llvm.sin.f32(float 0.5) #1
diff --git a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
index 95114abb4ef5f..0479fdb183f4c 100644
--- a/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
+++ b/llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll
@@ -411,7 +411,7 @@ define void @t7_nomerge0() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -454,7 +454,7 @@ define void @t8_nomerge1() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -510,7 +510,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -523,7 +523,7 @@ define void @t9_nomerge2() personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1:.*]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR1]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT2:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[INVOKE_CONT2]]:
 ; CHECK-NEXT:    unreachable
@@ -1075,7 +1075,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect() personality ptr @__gxx_
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1127,7 +1127,7 @@ define void @t17_mismatched_attrs_okay_merge_intersect2() personality ptr @__gxx
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -1166,13 +1166,15 @@ if.end:
 }
 
 
-define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
-; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge() personality ptr @__gxx_personality_v0 {
+define void @t17_mismatched_attrs_prevent_merge() strictfp personality ptr @__gxx_personality_v0 {
+; CHECK: Function Attrs: strictfp
+; CHECK-LABEL: define void @t17_mismatched_attrs_prevent_merge(
+; CHECK-SAME: ) #[[ATTR0:[0-9]+]] personality ptr @__gxx_personality_v0 {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C0]], label %[[IF_THEN0:.*]], label %[[IF_ELSE:.*]]
 ; CHECK:       [[IF_THEN0]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3:[0-9]+]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR0]]
 ; CHECK-NEXT:            to label %[[INVOKE_CONT0:.*]] unwind label %[[LPAD:.*]]
 ; CHECK:       [[INVOKE_CONT0]]:
 ; CHECK-NEXT:    unreachable
@@ -1239,7 +1241,7 @@ define void @t18_attributes_are_preserved() personality ptr @__gxx_personality_v
 ; CHECK-NEXT:    [[C1:%.*]] = call i1 @cond()
 ; CHECK-NEXT:    br i1 [[C1]], label %[[IF_THEN1_INVOKE]], label %[[IF_END:.*]]
 ; CHECK:       [[IF_THEN1_INVOKE]]:
-; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR2]]
+; CHECK-NEXT:    invoke void @simple_throw() #[[ATTR3]]
 ; CHECK-NEXT:            to label %[[IF_THEN1_CONT:.*]] unwind label %[[LPAD]]
 ; CHECK:       [[IF_THEN1_CONT]]:
 ; CHECK-NEXT:    unreachable
@@ -2675,8 +2677,8 @@ declare void @consume(i32)
 
 declare dso_local i32 @__gxx_personality_v0(...)
 ;.
-; CHECK: attributes #[[ATTR0:[0-9]+]] = { noreturn }
-; CHECK: attributes #[[ATTR1]] = { nomerge }
-; CHECK: attributes #[[ATTR2]] = { memory(none) }
-; CHECK: attributes #[[ATTR3]] = { strictfp }
+; CHECK: attributes #[[ATTR0]] = { strictfp }
+; CHECK: attributes #[[ATTR1:[0-9]+]] = { noreturn }
+; CHECK: attributes #[[ATTR2]] = { nomerge }
+; CHECK: attributes #[[ATTR3]] = { memory(none) }
 ;.
diff --git a/llvm/unittests/IR/VerifierTest.cpp b/llvm/unittests/IR/VerifierTest.cpp
index 2d20d66e447df..329f61bb46787 100644
--- a/llvm/unittests/IR/VerifierTest.cpp
+++ b/llvm/unittests/IR/VerifierTest.cpp
@@ -746,4 +746,28 @@ TEST(VerifierTest, IntrinsicRetInvalidStruct) {
   }
 }
 
+TEST(VerifierTest, InvalidStrictFPAttribute) {
+  LLVMContext Ctx;
+  Module M("M", Ctx);
+  FunctionType *FuncTy =
+      FunctionType::get(Type::getVoidTy(Ctx), /*isVarArg=*/false);
+  Function *F =
+      Function::Create(FuncTy, Function::ExternalLinkage, "strictfp_test", M);
+  BasicBlock *Entry = BasicBlock::Create(Ctx, "entry", F);
+  Type *FloatTy = Type::getFloatTy(Ctx);
+
+  Function *CosF =
+      Intrinsic::getOrInsertDeclaration(&M, Intrinsic::cos, FloatTy);
+  CallInst *CI = CallInst::Create(CosF, ConstantFP::getNullValue(FloatTy),
+                                  "strictfp_call", Entry);
+  CI->addFnAttr(Attribute::StrictFP);
+  ReturnInst::Create(Ctx, Entry);
+
+  std::string Error;
+  raw_string_ostream ErrorOS(Error);
+  EXPECT_TRUE(verifyModule(M, &ErrorOS));
+  EXPECT_TRUE(StringRef(Error).starts_with(
+      "call site marked strictfp without caller function marked strictfp"))
+      << Error;
+}
 } // end anonymous namespace
diff --git a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
index 6910a06992b55..7a362ebe2421e 100644
--- a/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
+++ b/mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir
@@ -596,9 +596,9 @@ llvm.func @masked_gather_scatter_intrinsics(%M: vector<7 x !llvm.ptr>, %mask: ve
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics
-llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru)
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough)
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask)
@@ -607,9 +607,9 @@ llvm.func @masked_expand_compress_intrinsics(%ptr: !llvm.ptr, %mask: vector<7xi1
 }
 
 // CHECK-LABEL: @masked_expand_compress_intrinsics_with_alignment
-llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthru: vector<7xf32>) {
+llvm.func @masked_expand_compress_intrinsics_with_alignment(%ptr: !llvm.ptr, %mask: vector<7xi1>, %passthrough: vector<7xf32>) {
   // CHECK: call <7 x float> @llvm.masked.expandload.v7f32.p0(ptr align 8 %{{.*}}, <7 x i1> %{{.*}}, <7 x float> %{{.*}})
-  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthru) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
+  %0 = "llvm.intr.masked.expandload"(%ptr, %mask, %passthrough) {arg_attrs = [{llvm.align = 8 : i32}, {}, {}]}
     : (!llvm.ptr, vector<7xi1>, vector<7xf32>) -> (vector<7xf32>)
   // CHECK: call void @llvm.masked.compressstore.v7f32.p0(<7 x float> %{{.*}}, ptr align 8 %{{.*}}, <7 x i1> %{{.*}})
   "llvm.intr.masked.compressstore"(%0, %ptr, %mask) {arg_attrs = [{}, {llvm.align = 8 : i32}, {}]}
@@ -1202,7 +1202,7 @@ llvm.func @vector_ptrmask(%p: vector<8 x !llvm.ptr>, %mask: vector<8 x i64>) ->
 }
 
 // CHECK-LABEL: @experimental_constrained_fadd
-llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fadd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1215,7 +1215,7 @@ llvm.func @experimental_constrained_fadd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fsub
-llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fsub.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1228,7 +1228,7 @@ llvm.func @experimental_constrained_fsub(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmul
-llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmul.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1241,7 +1241,7 @@ llvm.func @experimental_constrained_fmul(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fdiv
-llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fdiv.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1254,7 +1254,7 @@ llvm.func @experimental_constrained_fdiv(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_frem
-llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.frem.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1267,7 +1267,7 @@ llvm.func @experimental_constrained_frem(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fma
-llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fma.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1280,7 +1280,7 @@ llvm.func @experimental_constrained_fma(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fmuladd
-llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
+llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fmuladd.f32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1293,7 +1293,7 @@ llvm.func @experimental_constrained_fmuladd(%s: f32, %v: vector<4 x f32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_uitofp
-llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.uitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1322,7 +1322,7 @@ llvm.func @experimental_constrained_uitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_sitofp
-llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
+llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.sitofp.f32.i32(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1351,7 +1351,7 @@ llvm.func @experimental_constrained_sitofp(%s: i32, %v: vector<4 x i32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fptrunc
-llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call float @llvm.experimental.constrained.fptrunc.f32.f64(
   // CHECK: metadata !"round.towardzero"
   // CHECK: metadata !"fpexcept.ignore"
@@ -1380,7 +1380,7 @@ llvm.func @experimental_constrained_fptrunc(%s: f64, %v: vector<4xf32>) {
 }
 
 // CHECK-LABEL: @experimental_constrained_fpext
-llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) {
+llvm.func @experimental_constrained_fpext(%s: f32, %v: vector<4xf32>) attributes { passthrough = ["strictfp"] } {
   // CHECK: call double @llvm.experimental.constrained.fpext.f64.f32(
   // CHECK: metadata !"fpexcept.ignore"
   %0 = llvm.intr.experimental.constrained.fpext %s ignore : f32 to f64
@@ -1426,13 +1426,13 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 
 // Check that intrinsics are declared with appropriate types.
 // CHECK-DAG: declare float @llvm.fma.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fma.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare float @llvm.fmuladd.f32(float, float, float)
-// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fmuladd.v8f32(<8 x float>, <8 x float>, <8 x float>)
 // CHECK-DAG: declare void @llvm.prefetch.p0(ptr readonly captures(none), i32 immarg range(i32 0, 2), i32 immarg range(i32 0, 4), i32 immarg range(i32 0, 2))
 // CHECK-DAG: declare i1 @llvm.is.fpclass.f32(float, i32 immarg)
 // CHECK-DAG: declare float @llvm.exp.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.exp.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp2.f32(float)
 // CHECK-DAG: declare <8 x float> @llvm.exp2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.exp10.f32(float)
@@ -1442,19 +1442,19 @@ llvm.func @vector_scmp(%a: vector<4 x i32>, %b: vector<4 x i32>) -> vector<4 x i
 // CHECK-DAG: declare { float, i32 } @llvm.frexp.f32.i32(float)
 // CHECK-DAG: declare { <8 x float>, i32 } @llvm.frexp.v8f32.i32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log10.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log10.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.log2.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.log2.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.fabs.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.fabs.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.sqrt.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.sqrt.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.ceil.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.ceil.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.cos.f32(float)
-// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>) #0
+// CHECK-DAG: declare <8 x float> @llvm.cos.v8f32(<8 x float>)
 // CHECK-DAG: declare { float, float } @llvm.sincos.f32(float)
 // CHECK-DAG: declare { <8 x float>, <8 x float> } @llvm.sincos.v8f32(<8 x float>)
 // CHECK-DAG: declare float @llvm.copysign.f32(float, float)


From llvmlistbot at llvm.org  Fri Jul 24 04:25:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 04:25:18 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
 strictfp consistent v2 (PR #211769)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211769@github.com>
Message-ID: <6a634b9e.f06f4994.50b1b.1646@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-powerpc

Author: Ramkumar Ramachandra (artagnon)

<details>
<summary>Changes</summary>

Changes: The patch was originally landed as af0c933e (#<!-- -->209465), but there was an issue with strictfp functions being inlined into non-strictfp functions. The issue has since been resolved by a2b42b10a ([IR] Reject inlining strictfp alwaysinline callee into non-strictfp caller, #<!-- -->210701) and the re-land is identical.

Although the section on constrainedfp in the LangRef clearly states "All function definitions that use constrained floating point intrinsics must have the strictfp attribute", indicating that a function with strictfp calls must be marked with strictfp, the general description of strictfp does not specify this. Refine its semantics and make it so, eliminating the inconsistency.

---

Patch is 49.67 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211769.diff


23 Files Affected:

- (modified) clang/lib/CodeGen/TargetInfo.cpp (+3) 
- (modified) clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl (+2-2) 
- (modified) llvm/docs/LangRef.md (+2-4) 
- (modified) llvm/include/llvm/IR/Function.h (+3) 
- (modified) llvm/lib/IR/Verifier.cpp (+5) 
- (modified) llvm/test/CodeGen/AArch64/is_fpclass.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir (+1-1) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll (+2-2) 
- (modified) llvm/test/CodeGen/PowerPC/milicode32.ll (+1-1) 
- (modified) llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll (+2-14) 
- (modified) llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll (+6-6) 
- (modified) llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll (+35-32) 
- (modified) llvm/test/Transforms/InstCombine/erf.ll (+22-18) 
- (modified) llvm/test/Transforms/InstCombine/fdim.ll (+4-3) 
- (modified) llvm/test/Transforms/InstSimplify/disable_folding.ll (+8-6) 
- (modified) llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll (+16-14) 
- (modified) llvm/unittests/IR/VerifierTest.cpp (+24) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+25-25) 


``````````diff
diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index d6477b61bcb0e..ed46430ae979b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2800,7 +2800,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26632,9 +26633,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c663b1ec440ea..a7682ee246fe0 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3897,6 +3897,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "call site marked strictfp without caller function marked strictfp",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:     ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211769

From llvmlistbot at llvm.org  Fri Jul 24 04:25:18 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 04:25:18 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
 strictfp consistent v2 (PR #211769)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211769@github.com>
Message-ID: <6a634b9e.bd93e731.2ac69c.1e14@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang-codegen

Author: Ramkumar Ramachandra (artagnon)

<details>
<summary>Changes</summary>

Changes: The patch was originally landed as af0c933e (#<!-- -->209465), but there was an issue with strictfp functions being inlined into non-strictfp functions. The issue has since been resolved by a2b42b10a ([IR] Reject inlining strictfp alwaysinline callee into non-strictfp caller, #<!-- -->210701) and the re-land is identical.

Although the section on constrainedfp in the LangRef clearly states "All function definitions that use constrained floating point intrinsics must have the strictfp attribute", indicating that a function with strictfp calls must be marked with strictfp, the general description of strictfp does not specify this. Refine its semantics and make it so, eliminating the inconsistency.

---

Patch is 49.67 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211769.diff


23 Files Affected:

- (modified) clang/lib/CodeGen/TargetInfo.cpp (+3) 
- (modified) clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl (+2-2) 
- (modified) llvm/docs/LangRef.md (+2-4) 
- (modified) llvm/include/llvm/IR/Function.h (+3) 
- (modified) llvm/lib/IR/Verifier.cpp (+5) 
- (modified) llvm/test/CodeGen/AArch64/is_fpclass.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir (+1-1) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll (+2-2) 
- (modified) llvm/test/CodeGen/PowerPC/milicode32.ll (+1-1) 
- (modified) llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll (+2-14) 
- (modified) llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll (+6-6) 
- (modified) llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll (+35-32) 
- (modified) llvm/test/Transforms/InstCombine/erf.ll (+22-18) 
- (modified) llvm/test/Transforms/InstCombine/fdim.ll (+4-3) 
- (modified) llvm/test/Transforms/InstSimplify/disable_folding.ll (+8-6) 
- (modified) llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll (+16-14) 
- (modified) llvm/unittests/IR/VerifierTest.cpp (+24) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+25-25) 


``````````diff
diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index d6477b61bcb0e..ed46430ae979b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2800,7 +2800,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26632,9 +26633,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c663b1ec440ea..a7682ee246fe0 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3897,6 +3897,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "call site marked strictfp without caller function marked strictfp",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:     ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211769

From llvmlistbot at llvm.org  Fri Jul 24 04:25:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 04:25:19 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
 strictfp consistent v2 (PR #211769)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211769@github.com>
Message-ID: <6a634b9f.824e11a6.322560.6044@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-risc-v

Author: Ramkumar Ramachandra (artagnon)

<details>
<summary>Changes</summary>

Changes: The patch was originally landed as af0c933e (#<!-- -->209465), but there was an issue with strictfp functions being inlined into non-strictfp functions. The issue has since been resolved by a2b42b10a ([IR] Reject inlining strictfp alwaysinline callee into non-strictfp caller, #<!-- -->210701) and the re-land is identical.

Although the section on constrainedfp in the LangRef clearly states "All function definitions that use constrained floating point intrinsics must have the strictfp attribute", indicating that a function with strictfp calls must be marked with strictfp, the general description of strictfp does not specify this. Refine its semantics and make it so, eliminating the inconsistency.

---

Patch is 49.67 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211769.diff


23 Files Affected:

- (modified) clang/lib/CodeGen/TargetInfo.cpp (+3) 
- (modified) clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl (+2-2) 
- (modified) llvm/docs/LangRef.md (+2-4) 
- (modified) llvm/include/llvm/IR/Function.h (+3) 
- (modified) llvm/lib/IR/Verifier.cpp (+5) 
- (modified) llvm/test/CodeGen/AArch64/is_fpclass.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir (+1-1) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll (+2-2) 
- (modified) llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll (+2-2) 
- (modified) llvm/test/CodeGen/PowerPC/milicode32.ll (+1-1) 
- (modified) llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll (+2-14) 
- (modified) llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll (+1-1) 
- (modified) llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll (+6-6) 
- (modified) llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll (+35-32) 
- (modified) llvm/test/Transforms/InstCombine/erf.ll (+22-18) 
- (modified) llvm/test/Transforms/InstCombine/fdim.ll (+4-3) 
- (modified) llvm/test/Transforms/InstSimplify/disable_folding.ll (+8-6) 
- (modified) llvm/test/Transforms/SimplifyCFG/X86/merge-compatible-invokes-of-landingpad.ll (+16-14) 
- (modified) llvm/unittests/IR/VerifierTest.cpp (+24) 
- (modified) mlir/test/Target/LLVMIR/llvmir-intrinsics.mlir (+25-25) 


``````````diff
diff --git a/clang/lib/CodeGen/TargetInfo.cpp b/clang/lib/CodeGen/TargetInfo.cpp
index 641caf6d5cd4b..2f89154c2d2d9 100644
--- a/clang/lib/CodeGen/TargetInfo.cpp
+++ b/clang/lib/CodeGen/TargetInfo.cpp
@@ -195,6 +195,9 @@ llvm::Value *TargetCodeGenInfo::createEnqueuedBlockKernel(
   CGF.CGM.addDefaultFunctionDefinitionAttributes(KernelAttrs);
   F->addFnAttrs(KernelAttrs);
 
+  if (CGF.Builder.getIsFPConstrained())
+    F->addFnAttr(llvm::Attribute::StrictFP);
+
   auto IP = CGF.Builder.saveIP();
   auto *BB = llvm::BasicBlock::Create(C, "entry", F);
   auto &Builder = CGF.Builder;
diff --git a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
index 962ac4afa4991..c93f623016660 100644
--- a/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
+++ b/clang/test/CodeGenOpenCL/cl20-device-side-enqueue-attributes.cl
@@ -188,7 +188,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP-NEXT:    ret void
 //
 //
-// STRICTFP: Function Attrs: convergent nounwind
+// STRICTFP: Function Attrs: convergent nounwind strictfp
 // STRICTFP-LABEL: define spir_kernel void @__device_side_enqueue_block_invoke_kernel(
 // STRICTFP-SAME: ptr addrspace(4) [[TMP0:%.*]]) #[[ATTR4:[0-9]+]] {
 // STRICTFP-NEXT:  [[ENTRY:.*:]]
@@ -207,7 +207,7 @@ kernel void device_side_enqueue(global float *a, global float *b, int i) {
 // STRICTFP: attributes #[[ATTR1:[0-9]+]] = { nocallback nofree nosync nounwind willreturn memory(argmem: readwrite) }
 // STRICTFP: attributes #[[ATTR2]] = { convergent noinline nounwind optnone strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR3:[0-9]+]] = { nocallback nofree nosync nounwind strictfp willreturn memory(inaccessiblemem: readwrite) }
-// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind "stack-protector-buffer-size"="8" }
+// STRICTFP: attributes #[[ATTR4]] = { convergent nounwind strictfp "stack-protector-buffer-size"="8" }
 // STRICTFP: attributes #[[ATTR5]] = { convergent nounwind strictfp }
 // STRICTFP: attributes #[[ATTR6]] = { strictfp }
 //.
diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index d6477b61bcb0e..ed46430ae979b 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -2800,7 +2800,8 @@ fn -> other_fn -> other_fn ; fn is norecurse
     optimizations that require assumptions about the floating-point rounding
     mode or that might alter the state of floating-point status flags that
     might otherwise be set or cleared by calling this function. LLVM will
-    not introduce any new floating-point instructions that may trap.
+    not introduce any new floating-point instructions that may trap. All
+    function definitions that contain strictfp calls must be marked strictfp.
 
 (denormal_fpenv)=
 
@@ -26632,9 +26633,6 @@ point intrinsics must have the `strictfp` attribute either on the
 calling instruction or on the declaration or definition of the function
 being called.
 
-All function *definitions* that use constrained floating point intrinsics
-must have the `strictfp` attribute.
-
 #### '`llvm.experimental.constrained.fadd`' Intrinsic
 
 ##### Syntax:
diff --git a/llvm/include/llvm/IR/Function.h b/llvm/include/llvm/IR/Function.h
index bd28f0d9902da..0238c9b352f5f 100644
--- a/llvm/include/llvm/IR/Function.h
+++ b/llvm/include/llvm/IR/Function.h
@@ -632,6 +632,9 @@ class LLVM_ABI Function : public GlobalObject, public ilist_node<Function> {
     addFnAttr(Attribute::NoRecurse);
   }
 
+  /// Determine if the function has strict floating point sematics.
+  bool isStrictFP() const { return hasFnAttribute(Attribute::StrictFP); }
+
   /// Determine if the function is required to make forward progress.
   bool mustProgress() const {
     return hasFnAttribute(Attribute::MustProgress) ||
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index c663b1ec440ea..a7682ee246fe0 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -3897,6 +3897,11 @@ void Verifier::visitCallBase(CallBase &Call) {
   Check(!Attrs.hasFnAttr(Attribute::DenormalFPEnv),
         "denormal_fpenv attribute may not apply to call sites", Call);
 
+  Check(!Attrs.hasFnAttr(Attribute::StrictFP) ||
+            Call.getFunction()->isStrictFP(),
+        "call site marked strictfp without caller function marked strictfp",
+        Call);
+
   // Verify call attributes.
   verifyFunctionAttrs(FTy, Attrs, &Call, IsIntrinsic, Call.isInlineAsm());
 
diff --git a/llvm/test/CodeGen/AArch64/is_fpclass.ll b/llvm/test/CodeGen/AArch64/is_fpclass.ll
index 71a8f42a7b4db..19ce95641fe98 100644
--- a/llvm/test/CodeGen/AArch64/is_fpclass.ll
+++ b/llvm/test/CodeGen/AArch64/is_fpclass.ll
@@ -249,7 +249,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @isfinite_d_strictfp(double %x) {
+define i1 @isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
@@ -278,7 +278,7 @@ entry:
   ret i1 %0
 }
 
-define i1 @not_isfinite_d_strictfp(double %x) {
+define i1 @not_isfinite_d_strictfp(double %x) strictfp {
 ; CHECK-SD-LABEL: not_isfinite_d_strictfp:
 ; CHECK-SD:       // %bb.0: // %entry
 ; CHECK-SD-NEXT:    fmov x8, d0
diff --git a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
index 878f7b782a9a1..4d3552a056df5 100644
--- a/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
+++ b/llvm/test/CodeGen/AArch64/sms-loop-carried-fp-exceptions1.mir
@@ -23,7 +23,7 @@
 --- |
   @x = dso_local global i32 0, align 4
 
-  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) {
+  define dso_local void @f(ptr nocapture noundef writeonly %a, float noundef %y, i32 noundef %n) #2 {
   entry:
     %cmp6 = icmp sgt i32 %n, 0
     br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
index cfdc1baf8c282..9320908403679 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-fp-to-int.ll
@@ -209,7 +209,7 @@ define i64 @f64_to_u64(double %x) {
   ret i64 %cvt
 }
 
-define i32 @strict_convert_signed(double %x) {
+define i32 @strict_convert_signed(double %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzs w0, d0
@@ -224,7 +224,7 @@ define i32 @strict_convert_signed(double %x) {
   ret i32 %cvt
 }
 
-define i32 @strict_convert_unsigned(float %x) {
+define i32 @strict_convert_unsigned(float %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    fcvtzu w0, s0
diff --git a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
index 83403fc4ad86f..9e96355fd598d 100644
--- a/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
+++ b/llvm/test/CodeGen/AArch64/sve-streaming-mode-cvt-int-to-fp.ll
@@ -209,7 +209,7 @@ entry:
   ret double %cvt
 }
 
-define float @strict_convert_signed(i32 %x) {
+define float @strict_convert_signed(i32 %x) #0 {
 ; CHECK-LABEL: strict_convert_signed:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    scvtf s0, w0
@@ -224,7 +224,7 @@ entry:
   ret float %cvt
 }
 
-define float @strict_convert_unsigned(i64 %x) {
+define float @strict_convert_unsigned(i64 %x) #0 {
 ; CHECK-LABEL: strict_convert_unsigned:
 ; CHECK:       // %bb.0: // %entry
 ; CHECK-NEXT:    ucvtf s0, x0
diff --git a/llvm/test/CodeGen/PowerPC/milicode32.ll b/llvm/test/CodeGen/PowerPC/milicode32.ll
index 2c7e5c9867137..f2caec7e0045f 100644
--- a/llvm/test/CodeGen/PowerPC/milicode32.ll
+++ b/llvm/test/CodeGen/PowerPC/milicode32.ll
@@ -68,7 +68,7 @@ entry:
   ret i32 %call
 }
 
-define i32 @strlen_test_fp_strict(ptr noundef %str) nounwind {
+define i32 @strlen_test_fp_strict(ptr noundef %str) strictfp nounwind {
 ; CHECK-AIX-32-P9-LABEL: strlen_test_fp_strict:
 ; CHECK-AIX-32-P9:       # %bb.0: # %entry
 ; CHECK-AIX-32-P9-NEXT:    mflr r0
diff --git a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
index c1ee436a40c55..756cc6dc9408c 100644
--- a/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
+++ b/llvm/test/CodeGen/PowerPC/ppcf128-constrained-fp-intrinsics.ll
@@ -1972,14 +1972,12 @@ entry:
   ret ppc_fp128 %conv
 }
 
-define i1 @ppcq_to_s1(ppc_fp128 %a) {
+define i1 @ppcq_to_s1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_s1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __gcc_qtou
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -1992,8 +1990,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __gcc_qtou
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2006,8 +2002,6 @@ define i1 @ppcq_to_s1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __gcc_qtou
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
@@ -2019,14 +2013,12 @@ entry:
   ret i1 %conv
 }
 
-define i1 @ppcq_to_u1(ppc_fp128 %a) {
+define i1 @ppcq_to_u1(ppc_fp128 %a) #0 {
 ; PC64LE-LABEL: ppcq_to_u1:
 ; PC64LE:       # %bb.0: # %entry
 ; PC64LE-NEXT:    mflr 0
 ; PC64LE-NEXT:    stdu 1, -32(1)
 ; PC64LE-NEXT:    std 0, 48(1)
-; PC64LE-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE-NEXT:    .cfi_offset lr, 16
 ; PC64LE-NEXT:    bl __fixunstfsi
 ; PC64LE-NEXT:    nop
 ; PC64LE-NEXT:    addi 1, 1, 32
@@ -2039,8 +2031,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64LE9-NEXT:    mflr 0
 ; PC64LE9-NEXT:    stdu 1, -32(1)
 ; PC64LE9-NEXT:    std 0, 48(1)
-; PC64LE9-NEXT:    .cfi_def_cfa_offset 32
-; PC64LE9-NEXT:    .cfi_offset lr, 16
 ; PC64LE9-NEXT:    bl __fixunstfsi
 ; PC64LE9-NEXT:    nop
 ; PC64LE9-NEXT:    addi 1, 1, 32
@@ -2053,8 +2043,6 @@ define i1 @ppcq_to_u1(ppc_fp128 %a) {
 ; PC64-NEXT:    mflr 0
 ; PC64-NEXT:    stdu 1, -112(1)
 ; PC64-NEXT:    std 0, 128(1)
-; PC64-NEXT:    .cfi_def_cfa_offset 112
-; PC64-NEXT:    .cfi_offset lr, 16
 ; PC64-NEXT:    bl __fixunstfsi
 ; PC64-NEXT:    nop
 ; PC64-NEXT:    addi 1, 1, 112
diff --git a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
index 53fcfa19725db..654d99948bd40 100644
--- a/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/double-intrinsics-strict.ll
@@ -1989,7 +1989,7 @@ define i64 @llround_f64(double %a) nounwind strictfp {
   ret i64 %1
 }
 
-define double @ldexp_f64(double %x, i32 signext %y) nounwind {
+define double @ldexp_f64(double %x, i32 signext %y) nounwind strictfp {
 ; RV32IFD-LABEL: ldexp_f64:
 ; RV32IFD:       # %bb.0:
 ; RV32IFD-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
index 3a4acfd8a41ee..b22661976e727 100644
--- a/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/float-intrinsics-strict.ll
@@ -1955,7 +1955,7 @@ define i64 @llround_f32(float %a) nounwind strictfp {
   ret i64 %1
 }
 
-define float @ldexp_f32(float %x, i32 signext %y) nounwind {
+define float @ldexp_f32(float %x, i32 signext %y) nounwind strictfp {
 ; RV32IF-LABEL: ldexp_f32:
 ; RV32IF:       # %bb.0:
 ; RV32IF-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
index eb1848965a9ba..c876b76b8f0d9 100644
--- a/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfh-half-intrinsics-strict.ll
@@ -714,7 +714,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFH-LABEL: ldexp_f16:
 ; RV32IZFH:       # %bb.0:
 ; RV32IZFH-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
index 0529819a4f4e2..891ffa0a0c96b 100644
--- a/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
+++ b/llvm/test/CodeGen/RISCV/zfhmin-half-intrinsics-strict.ll
@@ -744,7 +744,7 @@ define i64 @llround_f16(half %a) nounwind strictfp {
   ret i64 %1
 }
 
-define half @ldexp_f16(half %x, i32 signext %y) nounwind {
+define half @ldexp_f16(half %x, i32 signext %y) nounwind strictfp {
 ; RV32IZFHMIN-LABEL: ldexp_f16:
 ; RV32IZFHMIN:       # %bb.0:
 ; RV32IZFHMIN-NEXT:    addi sp, sp, -16
diff --git a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
index b2d4f570afbd9..67eed241354d2 100644
--- a/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
+++ b/llvm/test/CodeGen/SPIRV/llvm-intrinsics/constrained-fmuladd.ll
@@ -10,7 +10,7 @@
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f32(float %a) {
+define spir_kernel void @test_f32(float %a) strictfp {
 entry:
   %r = tail call float @llvm.experimental.constrained.fmuladd.f32(
               float %a, float %a, float %a,
@@ -18,9 +18,9 @@ entry:
   ret void
 }
 
-; CHECK: OpFMul %[[#]] %[[#]] %[[#]] 
+; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_f64(double %a) {
+define spir_kernel void @test_f64(double %a) strictfp {
 entry:
   %r = tail call double @llvm.experimental.constrained.fmuladd.f64(
               double %a, double %a, double %a,
@@ -30,7 +30,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f32(<2 x float> %a) {
+define spir_kernel void @test_v2f32(<2 x float> %a) strictfp {
 entry:
   %r = tail call <2 x float> @llvm.experimental.constrained.fmuladd.v2f32(
               <2 x float> %a, <2 x float> %a, <2 x float> %a,
@@ -40,7 +40,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v4f32(<4 x float> %a) {
+define spir_kernel void @test_v4f32(<4 x float> %a) strictfp {
 entry:
   %r = tail call <4 x float> @llvm.experimental.constrained.fmuladd.v4f32(
               <4 x float> %a, <4 x float> %a, <4 x float> %a,
@@ -50,7 +50,7 @@ entry:
 
 ; CHECK: OpFMul %[[#]] %[[#]] %[[#]]
 ; CHECK: OpFAdd %[[#]] %[[#]] %[[#]]
-define spir_kernel void @test_v2f64(<2 x double> %a) {
+define spir_kernel void @test_v2f64(<2 x double> %a) strictfp {
 entry:
   %r = tail call <2 x double> @llvm.experimental.constrained.fmuladd.v2f64(
               <2 x double> %a, <2 x double> %a, <2 x double> %a,
diff --git a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
index 1dbffd962a638..b8854e3061dbf 100644
--- a/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
+++ b/llvm/test/Transforms/EarlyCSE/replace-calls-def-attrs.ll
@@ -13,7 +13,7 @@ declare i8 @buz.fp(float, float)
 define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C1]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -27,7 +27,7 @@ define i8 @same_parent_combine_diff_attrs(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -41,7 +41,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2:[0-9]+]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR3:[0-9]+]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -55,7 +55,7 @@ define i8 @same_parent_combine_diff_attrs_fmf(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_fmf2(
 ; CHECK-SAME: float [[X:%.*]], float [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call nnan float @baz.fp(float [[X]], float noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.fp(float [[C1]], float [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -69,7 +69,7 @@ define i8 @same_parent_combine_diff_attrs_fmf2(float %x, float %y) {
 define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_needs_intersect2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -83,7 +83,7 @@ define i8 @same_parent_combine_diff_attrs_needs_intersect2(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_really_needs_intersect(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_really_needs_intersect(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C1:%.*]] = call ptr @baz.ptr(i8 [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz.ptr(ptr [[C1]], ptr noundef [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ;
@@ -112,7 +112,7 @@ define i8 @same_parent_combine_diff_attrs_fail_side_effects(i8 %x, i8 %y) {
 define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(
 ; CHECK-SAME: i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]])
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
@@ -127,10 +127,10 @@ define i8 @same_parent_combine_diff_attrs_quasi_side_effects2(i8 %x, i8 %y) {
 define i8 @diff_parent_combine_diff_attrs(i1 %c, i8 %x, i8 %y) {
 ; CHECK-LABEL: define i8 @diff_parent_combine_diff_attrs(
 ; CHECK-SAME: i1 [[C:%.*]], i8 [[X:%.*]], i8 [[Y:%.*]]) {
-; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR1]]
+; CHECK-NEXT:    [[C1:%.*]] = call i8 @baz(i8 [[X]], i8 noundef [[Y]]) #[[ATTR2]]
 ; CHECK-NEXT:    br i1 [[C]], label %[[T:.*]], label %[[F:.*]]
 ; CHECK:       [[T]]:
-; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR2]]
+; CHECK-NEXT:    [[C0:%.*]] = call i8 @baz(i8 noundef [[X]], i8 noundef [[Y]]) #[[ATTR3]]
 ; CHECK-NEXT:    [[R:%.*]] = call i8 @buz(i8 [[C0]], i8 [[C1]])
 ; CHECK-NEXT:    ret i8 [[R]]
 ; CHECK:     ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211769

From llvmlistbot at llvm.org  Fri Jul 24 04:26:15 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Fri, 24 Jul 2026 04:26:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a634bd7.824e11a6.322560.87b6@mx.google.com>


FedericoBruzzone wrote:

Thanks @banach-space! 
I hope that now it's OK :D

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Fri Jul 24 04:30:20 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:30:20 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a634ccc.92ea0e2a.2ed880.08a3@mx.google.com>


banach-space wrote:

> @banach-space Thank you for the detailed explanation, and currently I do not have broader plans for sub-byte types or a real use case for i3. As I take a look into the VectorEmulateNarrowType, I do not think it will affect my fix toward this issue. Let me know if you find other concerns, and I am happy to take a look into it again.

+1 to updating the verifier instead, thank you!

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 04:32:51 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:32:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Linalg] Fix crash on duplicate
 dimensions in linalg.broadcast (PR #211203)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211203@github.com>
Message-ID: <6a634d63.507fc713.3702e2.c8a6@mx.google.com>


banach-space wrote:

> @banach-space I've updated the summary and applied the review comments. There will be two follow-up PRs after this:
> 
> * Fix for the crash on 0D inputs after flattening, which generated improper code as @nujaa mentioned above.
> * Introducing support for scalar broadcasts.

Nice! I will be away next week, so please expect a delay from my side. There's also plenty of other great reviewers available :)

https://github.com/llvm/llvm-project/pull/211203

From llvmlistbot at llvm.org  Fri Jul 24 04:34:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 04:34:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 98195e0 - [MLIR][Python] Make Python-defined
 dialect loading context-aware (#210501)
Message-ID: <6a634dc6.160324b0.17e275.3ba0@mx.google.com>


Author: Twice
Date: 2026-07-24T19:34:24+08:00
New Revision: 98195e0458dd5ec037785a6f861c6402bec778b3

URL: https://github.com/llvm/llvm-project/commit/98195e0458dd5ec037785a6f861c6402bec778b3
DIFF: https://github.com/llvm/llvm-project/commit/98195e0458dd5ec037785a6f861c6402bec778b3.diff

LOG: [MLIR][Python] Make Python-defined dialect loading context-aware (#210501)

Python-defined dialect loading currently relies on
`Dialect._mlir_module` to infer whether a dialect has already been
loaded. This state belongs to the Python dialect class rather than an
MLIR context.

Consequently, loading the same dialect after switching contexts requires
`reload=True`, while reloading it in a context where it is already
present can hit the operation registration assertion reported in
#210053.

This patch adds `mlirContextGetLoadedDialect` (following
https://github.com/llvm/lighthouse/pull/228#discussion_r3589792891) to
the C API and exposes it as `Context.is_dialect_loaded`.
`Dialect.load()` now queries the active context:
- loading a dialect more than once in the same context raises a
`RuntimeError`;
- loading the same Python-defined dialect in another context succeeds
without a `reload` flag.

The `reload` parameter is semantically wrong (mentioned in
https://github.com/llvm/lighthouse/pull/228#discussion_r3586672116) thus
removed from the API as a small breaking change. Usually, users who are
using `reload` can just remove it and everything will work fine.

Fixes #210053.
Assisted by GPT 5.6 Sol (for writing test cases).

Added: 
    

Modified: 
    mlir/include/mlir-c/IR.h
    mlir/lib/Bindings/Python/IRCore.cpp
    mlir/lib/CAPI/IR/IR.cpp
    mlir/python/mlir/dialects/ext.py
    mlir/test/python/dialects/ext.py
    mlir/test/python/dialects/transform_op_interface.py
    mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir-c/IR.h b/mlir/include/mlir-c/IR.h
index 311234e42df60..98ea0dfc00a1b 100644
--- a/mlir/include/mlir-c/IR.h
+++ b/mlir/include/mlir-c/IR.h
@@ -140,6 +140,13 @@ mlirContextGetNumLoadedDialects(MlirContext context);
 MLIR_CAPI_EXPORTED MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
                                                            MlirStringRef name);
 
+/// Gets the dialect instance owned by the given context using the dialect
+/// namespace to identify it. If the dialect is not loaded by the context,
+/// returns null. Use mlirContextGetOrLoadDialect to load a dialect if it is
+/// registered with the context.
+MLIR_CAPI_EXPORTED MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                                           MlirStringRef name);
+
 /// Set threading mode (must be set to false to mlir-print-ir-after-all).
 MLIR_CAPI_EXPORTED void mlirContextEnableMultithreading(MlirContext context,
                                                         bool enable);

diff  --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index ab75bc20e123e..cceef55f9b15d 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -3397,7 +3397,7 @@ void populateIRCore(nb::module_ &m) {
           "Alias for `dialects`.")
       .def(
           "get_dialect_descriptor",
-          [=](PyMlirContext &self, std::string &name) {
+          [](PyMlirContext &self, std::string &name) {
             MlirDialect dialect = mlirContextGetOrLoadDialect(
                 self.get(), {name.data(), name.size()});
             if (mlirDialectIsNull(dialect)) {
@@ -3408,6 +3408,14 @@ void populateIRCore(nb::module_ &m) {
           },
           "dialect_name"_a,
           "Gets or loads a dialect by name, returning its descriptor object.")
+      .def(
+          "is_dialect_loaded",
+          [](PyMlirContext &self, std::string &name) {
+            MlirDialect dialect = mlirContextGetLoadedDialect(
+                self.get(), {name.data(), name.size()});
+            return !mlirDialectIsNull(dialect);
+          },
+          "dialect_name"_a, "Checks if a dialect is loaded in the context.")
       .def_prop_rw(
           "allow_unregistered_dialects",
           [](PyMlirContext &self) -> bool {
@@ -3537,7 +3545,7 @@ void populateIRCore(nb::module_ &m) {
   nb::class_<PyDialects>(m, "Dialects")
       .def(
           "__getitem__",
-          [=](PyDialects &self, std::string keyName) {
+          [](PyDialects &self, std::string keyName) {
             MlirDialect dialect =
                 self.getDialectForKey(keyName, /*attrError=*/false);
             nb::object descriptor =
@@ -3547,7 +3555,7 @@ void populateIRCore(nb::module_ &m) {
           "Gets a dialect by name using subscript notation.")
       .def(
           "__getattr__",
-          [=](PyDialects &self, std::string attrName) {
+          [](PyDialects &self, std::string attrName) {
             MlirDialect dialect =
                 self.getDialectForKey(attrName, /*attrError=*/true);
             nb::object descriptor =

diff  --git a/mlir/lib/CAPI/IR/IR.cpp b/mlir/lib/CAPI/IR/IR.cpp
index 94442e2be19a4..d5d545a4e1a71 100644
--- a/mlir/lib/CAPI/IR/IR.cpp
+++ b/mlir/lib/CAPI/IR/IR.cpp
@@ -97,6 +97,11 @@ MlirDialect mlirContextGetOrLoadDialect(MlirContext context,
   return wrap(unwrap(context)->getOrLoadDialect(unwrap(name)));
 }
 
+MlirDialect mlirContextGetLoadedDialect(MlirContext context,
+                                        MlirStringRef name) {
+  return wrap(unwrap(context)->getLoadedDialect(unwrap(name)));
+}
+
 bool mlirContextIsRegisteredOperation(MlirContext context, MlirStringRef name) {
   return unwrap(context)->isOperationRegistered(unwrap(name));
 }

diff  --git a/mlir/python/mlir/dialects/ext.py b/mlir/python/mlir/dialects/ext.py
index 10ac002c75d96..4e8d30d82d5f8 100644
--- a/mlir/python/mlir/dialects/ext.py
+++ b/mlir/python/mlir/dialects/ext.py
@@ -29,6 +29,7 @@
 
 __all__ = [
     "Dialect",
+    "DialectAlreadyLoadedError",
     "Operation",
     "Operand",
     "Result",
@@ -47,6 +48,10 @@
 Region = ir.Region
 
 
+class DialectAlreadyLoadedError(RuntimeError):
+    """Raised when a dialect is loaded more than once in the current context."""
+
+
 def construct_instance(origin, args):
     if not issubclass(origin, ir.Type | ir.Attribute):
         raise TypeError(f"unsupported type in constraints: {origin}")
@@ -957,18 +962,11 @@ def _emit_module(cls) -> ir.Module:
         return m
 
     @classmethod
-    def load(
-        cls,
-        *,
-        reload: bool = False,
-    ) -> None:
-        if hasattr(cls, "_mlir_module") and not reload:
-            if cls._mlir_module.context is not ir.Context.current:
-                raise RuntimeError(
-                    "This dialect was loaded in a 
diff erent context. "
-                    "Please set reload=True to reload the dialect in the current context."
-                )
-            return
+    def load(cls) -> None:
+        if ir.Context.current.is_dialect_loaded(cls.DIALECT_NAMESPACE):
+            raise DialectAlreadyLoadedError(
+                f"Dialect '{cls.DIALECT_NAMESPACE}' has already been loaded in the current context."
+            )
 
         cls._mlir_module = cls._emit_module()
         pm = PassManager()
@@ -980,19 +978,21 @@ def load(
         for op in cls.operations:
             op._attach_traits()
 
-        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=reload)
+        _cext.globals._register_dialect_impl(cls.DIALECT_NAMESPACE, cls, replace=True)
 
+        # typeids for dynamic types and attributes are context-dependent,
+        # so we need to register them for every MLIR context.
         for type_ in cls.types:
             typeid = ir.DynamicType.lookup_typeid(type_.type_name)
-            _cext.register_type_caster(typeid, replace=reload)(type_)
+            _cext.register_type_caster(typeid, replace=True)(type_)
 
         for attr in cls.attributes:
             typeid = ir.DynamicAttr.lookup_typeid(attr.attr_name)
-            _cext.register_type_caster(typeid, replace=reload)(attr)
+            _cext.register_type_caster(typeid, replace=True)(attr)
 
         for op in cls.operations:
-            _cext.register_operation(cls, replace=reload)(op)
-            _cext.register_op_adaptor(op, replace=reload)(op.Adaptor)
+            _cext.register_operation(cls, replace=True)(op)
+            _cext.register_op_adaptor(op, replace=True)(op.Adaptor)
 
 
 class Pure:

diff  --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index 2c9ab4fe321a2..dde980df53dba 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -105,6 +105,81 @@ class AddOp(Operation, dialect=MyInt, name="add"):
         print(adaptor1.rhs)
 
 
+# CHECK: TEST: testDialectLoadInMultipleContexts
+ at run
+def testDialectLoadInMultipleContexts():
+    class ContextLoadDialect(Dialect, name="ext_context_load"):
+        pass
+
+    class ContextLoadType(ContextLoadDialect.Type, name="type"):
+        value: IntegerAttr
+
+    class ContextLoadAttr(ContextLoadDialect.Attribute, name="attr"):
+        value: StringAttr
+
+    class ContextLoadOp(ContextLoadDialect.Operation, name="op"):
+        attr: ContextLoadAttr
+        result: Result[ContextLoadType]
+
+    def check_dialect(context_name, type_value):
+        i32 = IntegerType.get_signless(32)
+        result_type = ContextLoadType.get(IntegerAttr.get(i32, type_value))
+        attr = ContextLoadAttr.get(StringAttr.get(context_name))
+
+        module = Module.create()
+        with InsertionPoint(module.body):
+            ContextLoadOp(attr, result_type)
+
+        assert module.operation.verify()
+        module = Module.parse(str(module))
+        op = module.body.operations[0]
+        assert isinstance(op, ContextLoadOp)
+        assert isinstance(op.attr, ContextLoadAttr)
+        assert isinstance(op.result.type, ContextLoadType)
+
+        print(
+            f"{context_name}: {type(op).__name__}, "
+            f"{type(op.attr).__name__}, {type(op.result.type).__name__}"
+        )
+        print(op.attr.value)
+        print(op.result.type.value)
+
+    first_context = Context()
+    second_context = Context()
+
+    with first_context, Location.unknown():
+        ContextLoadDialect.load()
+        try:
+            ContextLoadDialect.load()
+        except DialectAlreadyLoadedError as e:
+            assert isinstance(e, RuntimeError)
+            # CHECK: same context: Dialect 'ext_context_load' has already been loaded in the current context.
+            print("same context:", e)
+        else:
+            raise AssertionError("expected DialectAlreadyLoadedError")
+
+        # CHECK: first context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "first context"
+        # CHECK: 1 : i32
+        check_dialect("first context", 1)
+
+    with second_context, Location.unknown():
+        ContextLoadDialect.load()
+        # CHECK: second context: ContextLoadOp, ContextLoadAttr, ContextLoadType
+        # CHECK: "second context"
+        # CHECK: 2 : i32
+        check_dialect("second context", 2)
+
+    with first_context, Location.unknown():
+        try:
+            ContextLoadDialect.load()
+        except DialectAlreadyLoadedError as e:
+            # CHECK: same context again: Dialect 'ext_context_load' has already been loaded in the current context.
+            print("same context again:", e)
+        else:
+            raise AssertionError("expected DialectAlreadyLoadedError")
+
+
 # CHECK: TEST: testExtDialect
 @run
 def testExtDialect():

diff  --git a/mlir/test/python/dialects/transform_op_interface.py b/mlir/test/python/dialects/transform_op_interface.py
index b0b416530eccc..811dd0a9149fb 100644
--- a/mlir/test/python/dialects/transform_op_interface.py
+++ b/mlir/test/python/dialects/transform_op_interface.py
@@ -25,7 +25,7 @@ def run(emit_schedule):
     with ir.Context() as ctx, ir.Location.unknown():
         payload = emit_payload()
 
-        MyTransform.load(reload=True)
+        MyTransform.load()
 
         GetNamedAttributeOp.attach_interface_impls(ctx)
         PrintParamOp.attach_interface_impls(ctx)

diff  --git a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
index 9cd73331cfdea..e194547fdc33a 100644
--- a/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
+++ b/mlir/test/python/dialects/transform_pattern_descriptor_op_interface.py
@@ -16,7 +16,7 @@ def run(emit_schedule):
     with ir.Context(), ir.Location.unknown():
         payload = emit_payload()
 
-        MyPatternDescriptors.load(reload=True)
+        MyPatternDescriptors.load()
 
         # NB: Pattern descriptor ops have their interfaces attached
         #     in their respective test functions.


        

From llvmlistbot at llvm.org  Fri Jul 24 04:34:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 04:34:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Make Python-defined dialect
 loading context-aware (PR #210501)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210501@github.com>
Message-ID: <6a634dca.d1496011.3cfe19.00be@mx.google.com>


https://github.com/PragmaTwice closed https://github.com/llvm/llvm-project/pull/210501

From llvmlistbot at llvm.org  Fri Jul 24 04:37:52 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 04:37:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a634e90.0318b02e.530b7.85b1@mx.google.com>


banach-space wrote:

Thanks! 

The way this model scalable tile sizes seems consistent with how we do it in other places. As for implementation details, I defer to the expert - @adam-smnk :)

https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Fri Jul 24 04:41:16 2026
From: llvmlistbot at llvm.org (Diego Caballero)
Date: Fri, 24 Jul 2026 04:41:16 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a634f5c.af6ba3ed.1989e7.34d8@mx.google.com>


dcaballe wrote:

I'm not sure I understand what the semantic question or gap is. Vector elements have a packed layout within the vector. What am I missing?

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 04:46:20 2026
From: llvmlistbot at llvm.org (Adam Siemieniuk)
Date: Fri, 24 Jul 2026 04:46:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LinalgBlockPackMatmul] Add support for
 scalable block factors (PR #211354)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211354@github.com>
Message-ID: <6a63508c.66c0679a.9a15c.478e@mx.google.com>


https://github.com/adam-smnk approved this pull request.


https://github.com/llvm/llvm-project/pull/211354

From llvmlistbot at llvm.org  Fri Jul 24 04:47:39 2026
From: llvmlistbot at llvm.org (Jeremy Morse)
Date: Fri, 24 Jul 2026 04:47:39 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a6350db.bc28b987.1c0bcd.0783@mx.google.com>


jmorse wrote:

I tend to agree with @felipepiovezan -- it's not that there's anything wrong with this code, instead it's not clear how we should consider it's composition with other features because nothing generates symbolic-branches yet, and we can't consider the use-cases. This manifests as uncertainty and hedging at later dates.

A prime example is implicit pointers: you consider them in this patch and add carve-outs, however to the best of my knowledge nothing produces implicit-pointers today, and won't until #187642 lands (which may or may not have stalled).   But we can at least know that because there's an overall plan/direction.

I think @snowy1803 has hit the nail on the head with composition with DIArgList: could this lead to two incompatible flavours of DIExpressions? Conceivably this is a legitimate direction because different applications (GPUs etc) have different characteristics, but it's worth considering from the top level down rather than implementation-upwards. Ensuring that "dead" or "invalid" #dbg_values are still retained to kill invalidated variable assignments is a consideration too.

https://github.com/llvm/llvm-project/pull/210850

From llvmlistbot at llvm.org  Fri Jul 24 05:00:42 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 24 Jul 2026 05:00:42 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a6353ea.87185949.2211b7.f0e9@mx.google.com>


================
@@ -2647,7 +2656,8 @@ object (`%mbar`) track the completion of all prior asynchronous tcgen05
 operations. The `.mc` variants allow signaling on the mbarrier objects of
 multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
 variants generate `cta_group::1` and `cta_group::2` flavors of the
-instruction respectively.
+instruction, respectively. The `smem.a.read` variants allow tracking the completion
----------------
durga4github wrote:

nit:
`...variants track the completion of reads of A-matrix...`

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 05:03:07 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 24 Jul 2026 05:03:07 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a63547b.83ce629a.2277d7.f266@mx.google.com>


================
@@ -5711,25 +5711,41 @@ def tcgen05_wait_ld: NullaryInst<"tcgen05.wait::ld.sync.aligned", int_nvvm_tcgen
 def tcgen05_wait_st: NullaryInst<"tcgen05.wait::st.sync.aligned", int_nvvm_tcgen05_wait_st>;
 
 multiclass TCGEN05_COMMIT_INTR<string AS, string num> {
-  defvar prefix = "tcgen05.commit.cta_group::" # num #".mbarrier::arrive::one.shared::cluster";
+  defvar prefix = "tcgen05.commit.cta_group::" # num #
+                  ".mbarrier::arrive::one.shared::cluster";
+  defvar prefix_smem_a_read = prefix # ".sync_restrict::shared::read::mma::a";
 
+  defvar intr_prefix = "int_nvvm_tcgen05_commit";
   defvar intr_suffix = !if(!eq(AS, "shared"), "_shared", "") # "_cg" # num;
-  defvar Intr = !cast<Intrinsic>("int_nvvm_tcgen05_commit" # intr_suffix);
-  defvar IntrMC = !cast<Intrinsic>("int_nvvm_tcgen05_commit_mc" # intr_suffix);
+  defvar Intr = !cast<Intrinsic>(intr_prefix # intr_suffix);
+  defvar IntrMC = !cast<Intrinsic>(intr_prefix # "_mc" # intr_suffix);
+  defvar IntrSmemARead =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read" # intr_suffix);
+  defvar IntrSmemAReadMC =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read_mc" # intr_suffix);
 
   def "" : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
              prefix # ".b64",
              [(Intr addr:$mbar)]>;
   def _MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
-                   prefix # ".multicast::cluster.b64",
-                   [(IntrMC addr:$mbar, B16:$mc)]>;
+              prefix # ".multicast::cluster.b64",
+              [(IntrMC addr:$mbar, i16:$mc)]>;
+  let Predicates = [callSubtarget<"hasTcgen05RubinFamilySupport">] in {
+    def _MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                    prefix # ".multicast::cluster::32b.b64",
+                    [(IntrMC addr:$mbar, i32:$mc)]>;
+    def _SMRA : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
----------------
durga4github wrote:

Can we give it a bit more explicit name?
`SMEM_RA_` or `SMEM_READ` or fully `SMEM_A_READ`..

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 05:05:43 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 24 Jul 2026 05:05:43 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a635517.3cce8983.247e47.2f4d@mx.google.com>


================
@@ -456,6 +461,19 @@ define void @nvvm_cp_async_bulk_intrinsics(ptr addrspace(3) %dst, ptr addrspace(
   ret void
 }
 
+; CHECK-LABEL: @nvvm_tcgen05_commit_mc_intrinsics
+define void @nvvm_tcgen05_commit_mc_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
----------------
durga4github wrote:

do we want the `mc` suffix to be more like "_mc_16b" or "_mc_16bit_"?

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 05:07:39 2026
From: llvmlistbot at llvm.org (Durgadoss R)
Date: Fri, 24 Jul 2026 05:07:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a63558b.e5907290.33b622.0967@mx.google.com>


================
@@ -0,0 +1,438 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
----------------
durga4github wrote:

nit:
On the file naming: I believe our test files are named with arch-number as suffixes instead of arch-names.
Shall we do "tcgen05-commit-sm107" for consistency?

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 05:13:06 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Fri, 24 Jul 2026 05:13:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a6356d2.a1accbb4.1ace1c.4a35@mx.google.com>


FedericoBruzzone wrote:

Hi @beamandala! Thank you for your contribution. I'd be happy to genuinely provide my brief POV, I hope it's well-received.






https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Fri Jul 24 05:14:03 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 05:14:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
	`CastAwayInsertLeadingOneDim` (PR #211777)
Message-ID: <llvm.org/llvm/llvm-project/pull/211777@github.com>


https://github.com/banach-space created https://github.com/llvm/llvm-project/pull/211777

Updates `CastAwayInsertLeadingOneDim` to use vector.shape_cast, rather
than  vector.broadcast, as the canonical form for stripping unit
dimensions.

This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>


>From 10b454e9a44bd7a38fa1ac0ae64c727fbdaa56cb Mon Sep 17 00:00:00 2001
From: Andrzej Warzynski <andrzej.warzynski at arm.com>
Date: Fri, 24 Jul 2026 12:11:48 +0000
Subject: [PATCH] [mlir][vector] Update `CastAwayInsertLeadingOneDim`

Updates `CastAwayInsertLeadingOneDim` to use vector.shape_cast, rather
than  vector.broadcast, as the canonical form for stripping unit
dimensions.

This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>
---
 .../Transforms/VectorDropLeadUnitDim.cpp      | 12 +--
 .../vector-dropleadunitdim-transforms.mlir    | 90 +++++++++----------
 2 files changed, 50 insertions(+), 52 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index d86e9f224bf9d..a5e1c838de60a 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -144,7 +144,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
 };
 
 // Casts away leading one dimensions in vector.insert's vector inputs by
-// inserting vector.broadcast.
+// inserting vector.shape_cast.
 struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
   using Base::Base;
 
@@ -172,11 +172,11 @@ struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
 
     Value newSrcVector = insertOp.getValueToStore();
     if (oldSrcRank != 0) {
-      newSrcVector = vector::ExtractOp::create(
-          rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
+      newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+          loc, cast<VectorType>(newSrcType), insertOp.getValueToStore());
     }
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     // New position rank needs to be computed in two steps: (1) if destination
     // type has leading unit dims, we also trim the position array accordingly,
@@ -193,7 +193,7 @@ struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
     auto newInsertOp = vector::InsertOp::create(rewriter, loc, newSrcVector,
                                                 newDstVector, newPosition);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index ab921b32c05df..bba9f3c855113 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -505,10 +505,10 @@ func.func @cast_away_elementwise_leading_one_dims(
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_scalar
 //  CHECK-SAME: (%[[S:.+]]: f32, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[V]][0, 0] : vector<4xf32> from vector<1x1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[S]], %[[EXTRACT]] [0] : f32 into vector<4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SC:.+]] = vector.shape_cast %[[V]] : vector<1x1x4xf32> to vector<4xf32>
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[S]], %[[SC]] [0] : f32 into vector<4xf32>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_scalar(%s: f32, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -520,10 +520,10 @@ func.func @cast_away_insert_leading_one_dims_scalar(%s: f32, %v: vector<1x1x4xf3
 // CHECK-SAME:    %[[S:.*]]: f32,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
 func.func @cast_away_insert_leading_one_dims_scalar_scalable(%s: f32, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[V]][0, 0] : vector<[4]xf32> from vector<1x1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[EXTRACT]] [0] : f32 into vector<[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[V]] : vector<1x1x[4]xf32> to vector<[4]xf32>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[SC]] [0] : f32 into vector<[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
 }
@@ -534,10 +534,10 @@ func.func @cast_away_insert_leading_one_dims_scalar_scalable(%s: f32, %v: vector
 // CHECK-SAME:    %[[S:.*]]: f32,
 // CHECK-SAME:    %[[V:.*]]: vector<1x[1]x4xf32>) -> vector<1x[1]x4xf32> {
 func.func @cast_away_insert_leading_one_dims_scalar_skip_scalable_dim(%s: f32, %v: vector<1x[1]x4xf32>) -> vector<1x[1]x4xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[V]][0] : vector<[1]x4xf32> from vector<1x[1]x4xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[EXTRACT]] [0, 0] : f32 into vector<[1]x4xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<[1]x4xf32> to vector<1x[1]x4xf32>
-// CHECK:           return %[[BCAST]] : vector<1x[1]x4xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[V]] : vector<1x[1]x4xf32> to  vector<[1]x4xf32>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[SC]] [0, 0] : f32 into vector<[1]x4xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<[1]x4xf32> to vector<1x[1]x4xf32>
+// CHECK:           return %[[RES]] : vector<1x[1]x4xf32>
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x[1]x4xf32>
   return %0: vector<1x[1]x4xf32>
 }
@@ -546,8 +546,8 @@ func.func @cast_away_insert_leading_one_dims_scalar_skip_scalable_dim(%s: f32, %
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank1
 //  CHECK-SAME: (%[[S:.+]]: vector<4xf32>, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[S]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[S]] : vector<4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank1(%s: vector<4xf32>, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0, 0] : vector<4xf32> into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -558,8 +558,8 @@ func.func @cast_away_insert_leading_one_dims_rank1(%s: vector<4xf32>, %v: vector
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank1_scalable(
 // CHECK-SAME:    %[[S:.*]]: vector<[4]xf32>,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[S]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[S]] : vector<[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank1_scalable(%s: vector<[4]xf32>, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
   %0 = vector.insert %s, %v [0, 0] : vector<[4]xf32> into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
@@ -569,9 +569,8 @@ func.func @cast_away_insert_leading_one_dims_rank1_scalable(%s: vector<[4]xf32>,
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank2
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[EXTRACT]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank2(%s: vector<1x4xf32>, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0] : vector<1x4xf32> into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -582,9 +581,8 @@ func.func @cast_away_insert_leading_one_dims_rank2(%s: vector<1x4xf32>, %v: vect
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank2_scalable(
 // CHECK-SAME:    %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[EXTRACT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank2_scalable(%s: vector<1x[4]xf32>, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
   %0 = vector.insert %s, %v [0] : vector<1x[4]xf32> into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
@@ -594,11 +592,11 @@ func.func @cast_away_insert_leading_one_dims_rank2_scalable(%s: vector<1x[4]xf32
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank2_one_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<1x2x1x4xf32>)
-//       CHECK:   %[[EXTRACTS:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[EXTRACTV:.+]] = vector.extract %[[V]][0] : vector<2x1x4xf32> from vector<1x2x1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [1, 0] : vector<4xf32> into vector<2x1x4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<2x1x4xf32> to vector<1x2x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SCS:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to  vector<4xf32>
+//       CHECK:   %[[SCV:.+]] = vector.shape_cast %[[V]] : vector<1x2x1x4xf32> to vector<2x1x4xf32> 
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SCS]], %[[SCV]] [1, 0] : vector<4xf32> into vector<2x1x4xf32>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<2x1x4xf32> to vector<1x2x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank2_one_dest(%s: vector<1x4xf32>, %v: vector<1x2x1x4xf32>) -> vector<1x2x1x4xf32> {
   %0 = vector.insert %s, %v [0, 1] : vector<1x4xf32> into vector<1x2x1x4xf32>
   return %0: vector<1x2x1x4xf32>
@@ -609,11 +607,11 @@ func.func @cast_away_insert_leading_one_dims_rank2_one_dest(%s: vector<1x4xf32>,
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:      %[[V:.*]]: vector<1x2x1x[4]xf32>) -> vector<1x2x1x[4]xf32> {
-// CHECK:           %[[EXTRACTS:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[EXTRACTV:.*]] = vector.extract %[[V]][0] : vector<2x1x[4]xf32> from vector<1x2x1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [1, 0] : vector<[4]xf32> into vector<2x1x[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<2x1x[4]xf32> to vector<1x2x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x2x1x[4]xf32>
+// CHECK:           %[[SCS:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<[4]xf32> 
+// CHECK:           %[[SCV:.*]] = vector.shape_cast %[[V]] : vector<1x2x1x[4]xf32> to vector<2x1x[4]xf32> 
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SCS]], %[[SCV]] [1, 0] : vector<[4]xf32> into vector<2x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<2x1x[4]xf32> to vector<1x2x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x2x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(%s: vector<1x[4]xf32>, %v: vector<1x2x1x[4]xf32>) -> vector<1x2x1x[4]xf32> {
   %0 = vector.insert %s, %v [0, 1] : vector<1x[4]xf32> into vector<1x2x1x[4]xf32>
   return %0: vector<1x2x1x[4]xf32>
@@ -623,8 +621,8 @@ func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(%s: vector<
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_non_one_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<8x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACT]], %[[V]] [5, 0] : vector<4xf32> into vector<8x1x4xf32>
+//       CHECK:   %[[SC:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to vector<4xf32>
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SC]], %[[V]] [5, 0] : vector<4xf32> into vector<8x1x4xf32>
 //       CHECK:   return %[[INSERT]]
 func.func @cast_away_insert_leading_one_dims_non_one_dest(%s: vector<1x4xf32>, %v: vector<8x1x4xf32>) -> vector<8x1x4xf32> {
   %0 = vector.insert %s, %v [5] : vector<1x4xf32> into vector<8x1x4xf32>
@@ -636,8 +634,8 @@ func.func @cast_away_insert_leading_one_dims_non_one_dest(%s: vector<1x4xf32>, %
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:      %[[V:.*]]: vector<8x1x[4]xf32>) -> vector<8x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACT]], %[[V]] [5, 0] : vector<[4]xf32> into vector<8x1x[4]xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<[4]xf32> 
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SC]], %[[V]] [5, 0] : vector<[4]xf32> into vector<8x1x[4]xf32>
 // CHECK:           return %[[INSERT]] : vector<8x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(%s: vector<1x[4]xf32>, %v: vector<8x1x[4]xf32>) -> vector<8x1x[4]xf32> {
   %0 = vector.insert %s, %v [5] : vector<1x[4]xf32> into vector<8x1x[4]xf32>
@@ -648,11 +646,11 @@ func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(%s: vector<1x
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_one_two_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x8xi1>, %[[V:.+]]: vector<1x1x8x1x8xi1>)
-//       CHECK:   %[[EXTRACTS:.+]] = vector.extract %[[S]][0] : vector<8xi1> from vector<1x8xi1>
-//       CHECK:   %[[EXTRACTV:.+]] = vector.extract %[[V]][0, 0] : vector<8x1x8xi1> from vector<1x1x8x1x8xi1>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [7, 0] : vector<8xi1> into vector<8x1x8xi1>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<8x1x8xi1> to vector<1x1x8x1x8xi1>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SCS:.+]] = vector.shape_cast %[[S]] : vector<1x8xi1> to vector<8xi1>
+//       CHECK:   %[[SCV:.+]] = vector.shape_cast %[[V]] : vector<1x1x8x1x8xi1> to vector<8x1x8xi1> 
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SCS]], %[[SCV]] [7, 0] : vector<8xi1> into vector<8x1x8xi1>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<8x1x8xi1> to vector<1x1x8x1x8xi1>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_one_two_dest(%s: vector<1x8xi1>, %v: vector<1x1x8x1x8xi1>) -> vector<1x1x8x1x8xi1> {
   %0 = vector.insert %s, %v [0, 0, 7] : vector<1x8xi1> into vector<1x1x8x1x8xi1>
   return %0: vector<1x1x8x1x8xi1>
@@ -663,11 +661,11 @@ func.func @cast_away_insert_leading_one_dims_one_two_dest(%s: vector<1x8xi1>, %v
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_one_two_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[8]xi1>,
 // CHECK-SAME:      %[[V:.*]]: vector<1x1x8x1x[8]xi1>) -> vector<1x1x8x1x[8]xi1> {
-// CHECK:           %[[EXTRACTS:.*]] = vector.extract %[[S]][0] : vector<[8]xi1> from vector<1x[8]xi1>
-// CHECK:           %[[EXTRACTV:.*]] = vector.extract %[[V]][0, 0] : vector<8x1x[8]xi1> from vector<1x1x8x1x[8]xi1>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [7, 0] : vector<[8]xi1> into vector<8x1x[8]xi1>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<8x1x[8]xi1> to vector<1x1x8x1x[8]xi1>
-// CHECK:           return %[[BCAST]] : vector<1x1x8x1x[8]xi1>
+// CHECK:           %[[SCS:.*]] = vector.shape_cast %[[S]] : vector<1x[8]xi1> to vector<[8]xi1>
+// CHECK:           %[[SCV:.*]] = vector.shape_cast %[[V]] : vector<1x1x8x1x[8]xi1> to vector<8x1x[8]xi1>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SCS]], %[[SCV]] [7, 0] : vector<[8]xi1> into vector<8x1x[8]xi1>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<8x1x[8]xi1> to vector<1x1x8x1x[8]xi1>
+// CHECK:           return %[[RES]] : vector<1x1x8x1x[8]xi1>
 func.func @cast_away_insert_leading_one_dims_one_two_dest_scalable(%s: vector<1x[8]xi1>, %v: vector<1x1x8x1x[8]xi1>) -> vector<1x1x8x1x[8]xi1> {
   %0 = vector.insert %s, %v [0, 0, 7] : vector<1x[8]xi1> into vector<1x1x8x1x[8]xi1>
   return %0: vector<1x1x8x1x[8]xi1>


From llvmlistbot at llvm.org  Fri Jul 24 05:14:38 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 05:14:38 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Update
	`CastAwayInsertLeadingOneDim` (PR #211777)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211777@github.com>
Message-ID: <6a63572e.1905e68e.25404.ba4b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-mlir-vector

@llvm/pr-subscribers-mlir

Author: Andrzej Warzyński (banach-space)

<details>
<summary>Changes</summary>

Updates `CastAwayInsertLeadingOneDim` to use vector.shape_cast, rather
than  vector.broadcast, as the canonical form for stripping unit
dimensions.

This change was originally implemented by @<!-- -->krzysz00 in #<!-- -->196206, but was
subsequently reverted in #<!-- -->199546. This PR intentionally restores only a
subset of #<!-- -->196206, making it easier to identify and triage any potential
regressions.

Co-authored-by: Krzysztof Drewniak &lt;Krzysztof.Drewniak@<!-- -->amd.com&gt;


---
Full diff: https://github.com/llvm/llvm-project/pull/211777.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp (+6-6) 
- (modified) mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir (+44-46) 


``````````diff
diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
index d86e9f224bf9d..a5e1c838de60a 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorDropLeadUnitDim.cpp
@@ -144,7 +144,7 @@ struct CastAwayInsertStridedSliceLeadingOneDim
 };
 
 // Casts away leading one dimensions in vector.insert's vector inputs by
-// inserting vector.broadcast.
+// inserting vector.shape_cast.
 struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
   using Base::Base;
 
@@ -172,11 +172,11 @@ struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
 
     Value newSrcVector = insertOp.getValueToStore();
     if (oldSrcRank != 0) {
-      newSrcVector = vector::ExtractOp::create(
-          rewriter, loc, insertOp.getValueToStore(), splatZero(srcDropCount));
+      newSrcVector = rewriter.createOrFold<vector::ShapeCastOp>(
+          loc, cast<VectorType>(newSrcType), insertOp.getValueToStore());
     }
-    Value newDstVector = vector::ExtractOp::create(
-        rewriter, loc, insertOp.getDest(), splatZero(dstDropCount));
+    Value newDstVector = rewriter.createOrFold<vector::ShapeCastOp>(
+        loc, newDstType, insertOp.getDest());
 
     // New position rank needs to be computed in two steps: (1) if destination
     // type has leading unit dims, we also trim the position array accordingly,
@@ -193,7 +193,7 @@ struct CastAwayInsertLeadingOneDim : public OpRewritePattern<vector::InsertOp> {
     auto newInsertOp = vector::InsertOp::create(rewriter, loc, newSrcVector,
                                                 newDstVector, newPosition);
 
-    rewriter.replaceOpWithNewOp<vector::BroadcastOp>(insertOp, oldDstType,
+    rewriter.replaceOpWithNewOp<vector::ShapeCastOp>(insertOp, oldDstType,
                                                      newInsertOp);
 
     return success();
diff --git a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
index ab921b32c05df..bba9f3c855113 100644
--- a/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
+++ b/mlir/test/Dialect/Vector/vector-dropleadunitdim-transforms.mlir
@@ -505,10 +505,10 @@ func.func @cast_away_elementwise_leading_one_dims(
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_scalar
 //  CHECK-SAME: (%[[S:.+]]: f32, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[V]][0, 0] : vector<4xf32> from vector<1x1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[S]], %[[EXTRACT]] [0] : f32 into vector<4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SC:.+]] = vector.shape_cast %[[V]] : vector<1x1x4xf32> to vector<4xf32>
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[S]], %[[SC]] [0] : f32 into vector<4xf32>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_scalar(%s: f32, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -520,10 +520,10 @@ func.func @cast_away_insert_leading_one_dims_scalar(%s: f32, %v: vector<1x1x4xf3
 // CHECK-SAME:    %[[S:.*]]: f32,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
 func.func @cast_away_insert_leading_one_dims_scalar_scalable(%s: f32, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[V]][0, 0] : vector<[4]xf32> from vector<1x1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[EXTRACT]] [0] : f32 into vector<[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[V]] : vector<1x1x[4]xf32> to vector<[4]xf32>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[SC]] [0] : f32 into vector<[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
 }
@@ -534,10 +534,10 @@ func.func @cast_away_insert_leading_one_dims_scalar_scalable(%s: f32, %v: vector
 // CHECK-SAME:    %[[S:.*]]: f32,
 // CHECK-SAME:    %[[V:.*]]: vector<1x[1]x4xf32>) -> vector<1x[1]x4xf32> {
 func.func @cast_away_insert_leading_one_dims_scalar_skip_scalable_dim(%s: f32, %v: vector<1x[1]x4xf32>) -> vector<1x[1]x4xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[V]][0] : vector<[1]x4xf32> from vector<1x[1]x4xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[EXTRACT]] [0, 0] : f32 into vector<[1]x4xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<[1]x4xf32> to vector<1x[1]x4xf32>
-// CHECK:           return %[[BCAST]] : vector<1x[1]x4xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[V]] : vector<1x[1]x4xf32> to  vector<[1]x4xf32>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[S]], %[[SC]] [0, 0] : f32 into vector<[1]x4xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<[1]x4xf32> to vector<1x[1]x4xf32>
+// CHECK:           return %[[RES]] : vector<1x[1]x4xf32>
   %0 = vector.insert %s, %v [0, 0, 0] : f32 into vector<1x[1]x4xf32>
   return %0: vector<1x[1]x4xf32>
 }
@@ -546,8 +546,8 @@ func.func @cast_away_insert_leading_one_dims_scalar_skip_scalable_dim(%s: f32, %
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank1
 //  CHECK-SAME: (%[[S:.+]]: vector<4xf32>, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[S]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[S]] : vector<4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank1(%s: vector<4xf32>, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0, 0] : vector<4xf32> into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -558,8 +558,8 @@ func.func @cast_away_insert_leading_one_dims_rank1(%s: vector<4xf32>, %v: vector
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank1_scalable(
 // CHECK-SAME:    %[[S:.*]]: vector<[4]xf32>,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[S]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[S]] : vector<[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank1_scalable(%s: vector<[4]xf32>, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
   %0 = vector.insert %s, %v [0, 0] : vector<[4]xf32> into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
@@ -569,9 +569,8 @@ func.func @cast_away_insert_leading_one_dims_rank1_scalable(%s: vector<[4]xf32>,
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank2
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<1x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[EXTRACT]] : vector<4xf32> to vector<1x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to vector<1x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank2(%s: vector<1x4xf32>, %v: vector<1x1x4xf32>) -> vector<1x1x4xf32> {
   %0 = vector.insert %s, %v [0] : vector<1x4xf32> into vector<1x1x4xf32>
   return %0: vector<1x1x4xf32>
@@ -582,9 +581,8 @@ func.func @cast_away_insert_leading_one_dims_rank2(%s: vector<1x4xf32>, %v: vect
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank2_scalable(
 // CHECK-SAME:    %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:    %[[V:.*]]: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[EXTRACT]] : vector<[4]xf32> to vector<1x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<1x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank2_scalable(%s: vector<1x[4]xf32>, %v: vector<1x1x[4]xf32>) -> vector<1x1x[4]xf32> {
   %0 = vector.insert %s, %v [0] : vector<1x[4]xf32> into vector<1x1x[4]xf32>
   return %0: vector<1x1x[4]xf32>
@@ -594,11 +592,11 @@ func.func @cast_away_insert_leading_one_dims_rank2_scalable(%s: vector<1x[4]xf32
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_rank2_one_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<1x2x1x4xf32>)
-//       CHECK:   %[[EXTRACTS:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[EXTRACTV:.+]] = vector.extract %[[V]][0] : vector<2x1x4xf32> from vector<1x2x1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [1, 0] : vector<4xf32> into vector<2x1x4xf32>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<2x1x4xf32> to vector<1x2x1x4xf32>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SCS:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to  vector<4xf32>
+//       CHECK:   %[[SCV:.+]] = vector.shape_cast %[[V]] : vector<1x2x1x4xf32> to vector<2x1x4xf32> 
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SCS]], %[[SCV]] [1, 0] : vector<4xf32> into vector<2x1x4xf32>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<2x1x4xf32> to vector<1x2x1x4xf32>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_rank2_one_dest(%s: vector<1x4xf32>, %v: vector<1x2x1x4xf32>) -> vector<1x2x1x4xf32> {
   %0 = vector.insert %s, %v [0, 1] : vector<1x4xf32> into vector<1x2x1x4xf32>
   return %0: vector<1x2x1x4xf32>
@@ -609,11 +607,11 @@ func.func @cast_away_insert_leading_one_dims_rank2_one_dest(%s: vector<1x4xf32>,
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:      %[[V:.*]]: vector<1x2x1x[4]xf32>) -> vector<1x2x1x[4]xf32> {
-// CHECK:           %[[EXTRACTS:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[EXTRACTV:.*]] = vector.extract %[[V]][0] : vector<2x1x[4]xf32> from vector<1x2x1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [1, 0] : vector<[4]xf32> into vector<2x1x[4]xf32>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<2x1x[4]xf32> to vector<1x2x1x[4]xf32>
-// CHECK:           return %[[BCAST]] : vector<1x2x1x[4]xf32>
+// CHECK:           %[[SCS:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<[4]xf32> 
+// CHECK:           %[[SCV:.*]] = vector.shape_cast %[[V]] : vector<1x2x1x[4]xf32> to vector<2x1x[4]xf32> 
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SCS]], %[[SCV]] [1, 0] : vector<[4]xf32> into vector<2x1x[4]xf32>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<2x1x[4]xf32> to vector<1x2x1x[4]xf32>
+// CHECK:           return %[[RES]] : vector<1x2x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(%s: vector<1x[4]xf32>, %v: vector<1x2x1x[4]xf32>) -> vector<1x2x1x[4]xf32> {
   %0 = vector.insert %s, %v [0, 1] : vector<1x[4]xf32> into vector<1x2x1x[4]xf32>
   return %0: vector<1x2x1x[4]xf32>
@@ -623,8 +621,8 @@ func.func @cast_away_insert_leading_one_dims_rank2_one_dest_scalable(%s: vector<
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_non_one_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x4xf32>, %[[V:.+]]: vector<8x1x4xf32>)
-//       CHECK:   %[[EXTRACT:.+]] = vector.extract %[[S]][0] : vector<4xf32> from vector<1x4xf32>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACT]], %[[V]] [5, 0] : vector<4xf32> into vector<8x1x4xf32>
+//       CHECK:   %[[SC:.+]] = vector.shape_cast %[[S]] : vector<1x4xf32> to vector<4xf32>
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SC]], %[[V]] [5, 0] : vector<4xf32> into vector<8x1x4xf32>
 //       CHECK:   return %[[INSERT]]
 func.func @cast_away_insert_leading_one_dims_non_one_dest(%s: vector<1x4xf32>, %v: vector<8x1x4xf32>) -> vector<8x1x4xf32> {
   %0 = vector.insert %s, %v [5] : vector<1x4xf32> into vector<8x1x4xf32>
@@ -636,8 +634,8 @@ func.func @cast_away_insert_leading_one_dims_non_one_dest(%s: vector<1x4xf32>, %
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[4]xf32>,
 // CHECK-SAME:      %[[V:.*]]: vector<8x1x[4]xf32>) -> vector<8x1x[4]xf32> {
-// CHECK:           %[[EXTRACT:.*]] = vector.extract %[[S]][0] : vector<[4]xf32> from vector<1x[4]xf32>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACT]], %[[V]] [5, 0] : vector<[4]xf32> into vector<8x1x[4]xf32>
+// CHECK:           %[[SC:.*]] = vector.shape_cast %[[S]] : vector<1x[4]xf32> to vector<[4]xf32> 
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SC]], %[[V]] [5, 0] : vector<[4]xf32> into vector<8x1x[4]xf32>
 // CHECK:           return %[[INSERT]] : vector<8x1x[4]xf32>
 func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(%s: vector<1x[4]xf32>, %v: vector<8x1x[4]xf32>) -> vector<8x1x[4]xf32> {
   %0 = vector.insert %s, %v [5] : vector<1x[4]xf32> into vector<8x1x[4]xf32>
@@ -648,11 +646,11 @@ func.func @cast_away_insert_leading_one_dims_non_one_dest_scalable(%s: vector<1x
 
 // CHECK-LABEL: func @cast_away_insert_leading_one_dims_one_two_dest
 //  CHECK-SAME: (%[[S:.+]]: vector<1x8xi1>, %[[V:.+]]: vector<1x1x8x1x8xi1>)
-//       CHECK:   %[[EXTRACTS:.+]] = vector.extract %[[S]][0] : vector<8xi1> from vector<1x8xi1>
-//       CHECK:   %[[EXTRACTV:.+]] = vector.extract %[[V]][0, 0] : vector<8x1x8xi1> from vector<1x1x8x1x8xi1>
-//       CHECK:   %[[INSERT:.+]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [7, 0] : vector<8xi1> into vector<8x1x8xi1>
-//       CHECK:   %[[BCAST:.+]] = vector.broadcast %[[INSERT]] : vector<8x1x8xi1> to vector<1x1x8x1x8xi1>
-//       CHECK:   return %[[BCAST]]
+//       CHECK:   %[[SCS:.+]] = vector.shape_cast %[[S]] : vector<1x8xi1> to vector<8xi1>
+//       CHECK:   %[[SCV:.+]] = vector.shape_cast %[[V]] : vector<1x1x8x1x8xi1> to vector<8x1x8xi1> 
+//       CHECK:   %[[INSERT:.+]] = vector.insert %[[SCS]], %[[SCV]] [7, 0] : vector<8xi1> into vector<8x1x8xi1>
+//       CHECK:   %[[RES:.+]] = vector.shape_cast %[[INSERT]] : vector<8x1x8xi1> to vector<1x1x8x1x8xi1>
+//       CHECK:   return %[[RES]]
 func.func @cast_away_insert_leading_one_dims_one_two_dest(%s: vector<1x8xi1>, %v: vector<1x1x8x1x8xi1>) -> vector<1x1x8x1x8xi1> {
   %0 = vector.insert %s, %v [0, 0, 7] : vector<1x8xi1> into vector<1x1x8x1x8xi1>
   return %0: vector<1x1x8x1x8xi1>
@@ -663,11 +661,11 @@ func.func @cast_away_insert_leading_one_dims_one_two_dest(%s: vector<1x8xi1>, %v
 // CHECK-LABEL:   func.func @cast_away_insert_leading_one_dims_one_two_dest_scalable(
 // CHECK-SAME:      %[[S:.*]]: vector<1x[8]xi1>,
 // CHECK-SAME:      %[[V:.*]]: vector<1x1x8x1x[8]xi1>) -> vector<1x1x8x1x[8]xi1> {
-// CHECK:           %[[EXTRACTS:.*]] = vector.extract %[[S]][0] : vector<[8]xi1> from vector<1x[8]xi1>
-// CHECK:           %[[EXTRACTV:.*]] = vector.extract %[[V]][0, 0] : vector<8x1x[8]xi1> from vector<1x1x8x1x[8]xi1>
-// CHECK:           %[[INSERT:.*]] = vector.insert %[[EXTRACTS]], %[[EXTRACTV]] [7, 0] : vector<[8]xi1> into vector<8x1x[8]xi1>
-// CHECK:           %[[BCAST:.*]] = vector.broadcast %[[INSERT]] : vector<8x1x[8]xi1> to vector<1x1x8x1x[8]xi1>
-// CHECK:           return %[[BCAST]] : vector<1x1x8x1x[8]xi1>
+// CHECK:           %[[SCS:.*]] = vector.shape_cast %[[S]] : vector<1x[8]xi1> to vector<[8]xi1>
+// CHECK:           %[[SCV:.*]] = vector.shape_cast %[[V]] : vector<1x1x8x1x[8]xi1> to vector<8x1x[8]xi1>
+// CHECK:           %[[INSERT:.*]] = vector.insert %[[SCS]], %[[SCV]] [7, 0] : vector<[8]xi1> into vector<8x1x[8]xi1>
+// CHECK:           %[[RES:.*]] = vector.shape_cast %[[INSERT]] : vector<8x1x[8]xi1> to vector<1x1x8x1x[8]xi1>
+// CHECK:           return %[[RES]] : vector<1x1x8x1x[8]xi1>
 func.func @cast_away_insert_leading_one_dims_one_two_dest_scalable(%s: vector<1x[8]xi1>, %v: vector<1x1x8x1x[8]xi1>) -> vector<1x1x8x1x[8]xi1> {
   %0 = vector.insert %s, %v [0, 0, 7] : vector<1x[8]xi1> into vector<1x1x8x1x[8]xi1>
   return %0: vector<1x1x8x1x[8]xi1>

``````````

</details>


https://github.com/llvm/llvm-project/pull/211777

From llvmlistbot at llvm.org  Fri Jul 24 05:17:50 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Fri, 24 Jul 2026 05:17:50 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a6357ee.7d702ed2.2f6c05.bbc1@mx.google.com>


FedericoBruzzone wrote:


I've been struggling with various issues on the LLVM side regarding aspects related to _promote_ allocations :D
And in my experience, these are very delicate aspects that require a great deal of attention.


I'd like to ask a few questions (just to confirm a few points 🫶):
1. `memref.alloc` is a per-call-site allocation. `memref.global` is process-lifetime, i.e., shared. Is correct to affirm that the _promotion_ is correct only when the function is never called concurrently or recursively?
2. A note: "global_alloc" is hardcoded. While the SymbolTable will rename the subsequent insertions, should we want it to behave this way?
3.  The PR removes all `memref.dealloc` uses of the `alloc`. But what if a `dealloc` dominates a second `alloc` of the same type? 🤔
4. I don't want to sound tedious, but, unless I'm missing something (it's possible), we should consider whether the allocator [escapes the current function](https://en.wikipedia.org/wiki/Escape_analysis). Right?
5. MLIR has `alloca_to_global`. Is there a way to eventually reuse this logic?
    https://github.com/llvm/llvm-project/blob/82488f0e7119dd05cd63329458733e8c295f0338/mlir/include/mlir/Dialect/MemRef/TransformOps/MemRefTransformOps.td#L147-L157



https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Fri Jul 24 05:50:47 2026
From: llvmlistbot at llvm.org (Vassil Vassilev)
Date: Fri, 24 Jul 2026 05:50:47 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [polly] [llvm] Forward declare
 VirtualFileSystem (PR #211761)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211761@github.com>
Message-ID: <6a635fa7.bd99a6db.202e82.6e5e@mx.google.com>


vgvassilev wrote:

@compnerd, @Steelskin: This uncovers an annoying limitation where repeating forward declarations will need to be annotated and there is no easy way how to centralize them in a single place because of the different usage patterns. If we merge this as the as a way to handle repetitive patterns we will grow a jungle of fwd.h files -- is that optimal?

https://github.com/llvm/llvm-project/pull/211761

From llvmlistbot at llvm.org  Fri Jul 24 05:51:58 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 05:51:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a635fee.7d702ed2.2f6c05.1221@mx.google.com>


https://github.com/tblah edited https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Fri Jul 24 05:51:58 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 05:51:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a635fee.83ce629a.2277d7.16bb@mx.google.com>


https://github.com/tblah approved this pull request.

LGTM, thanks for the updates

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Fri Jul 24 05:51:58 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 05:51:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a635fee.f3151ff8.f584a.2cc0@mx.google.com>


================
@@ -191,23 +191,32 @@ class LinearClauseProcessor {
   }
 
   // Find linear iteration variable and save it for later updates
-  void initLinearIV(omp::SimdOp simdOp) {
+  LogicalResult initLinearIV(omp::SimdOp simdOp) {
     auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
     // NOTE iteration variables can only be linear in non-nested loops.
     if (loopOp.getIVs().size() != 1)
-      return;
-    // The linear IV is the loop IV's store address.
+      return success();
+    // Currently, frontends using `omp.simd` always generate a store from the
+    // `omp.loop_nest`'s IV to the corresponding iteration variable.
+    // We leverage this to find the linear iteration variable.
+    //
+    // TODO Add an attribute to `omp.loop_nest` that explicitly lists the
+    //      variables that correspond to the loop induction variables.
     BlockArgument arg = loopOp.getIVs().front();
     for (const Operation *user : arg.getUsers()) {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
+            if (linearLoopIV)
----------------
tblah wrote:

nit: Maybe this could be relaxed to allow repeated stores to the same location

https://github.com/llvm/llvm-project/pull/183800

From llvmlistbot at llvm.org  Fri Jul 24 05:59:22 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 05:59:22 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Implement `getRegionInvocationBounds` for
	`scf::ForOp` (PR #211631)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211631@github.com>
Message-ID: <6a6361aa.e5907290.33b622.f509@mx.google.com>


https://github.com/ftynse approved this pull request.


https://github.com/llvm/llvm-project/pull/211631

From llvmlistbot at llvm.org  Fri Jul 24 06:01:33 2026
From: llvmlistbot at llvm.org (Fabrice de Gans)
Date: Fri, 24 Jul 2026 06:01:33 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [polly] [llvm] Forward declare
 VirtualFileSystem (PR #211761)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211761@github.com>
Message-ID: <6a63622d.dcf3bad1.29a5a5.a920@mx.google.com>


Steelskin wrote:

> @compnerd, @Steelskin: This uncovers an annoying limitation where repeating forward declarations will need to be annotated and there is no easy way how to centralize them in a single place because of the different usage patterns. If we merge this as a way to handle repetitive patterns we will grow a jungle of fwd.h files -- is that optimal?

@vgvassilev 
It's not so much the forward declaration that is moved to its header, it's the addition of the template instantiation for `IntrusiveRefCntPtrInfo<::llvm::vfs::FileSystem>`. Note that it is not required to use this header everywhere, I merely thought it was cleaner to do so. I can revert probably 90% of these headers if you'd prefer.

Note that this is what the `IntrusiveRefCntPtr` documentation recommends: https://github.com/llvm/llvm-project/blob/697f0f3b20d371d9e417891d9cdfb6a3613d7828/llvm/include/llvm/ADT/IntrusiveRefCntPtr.h#L157

https://github.com/llvm/llvm-project/pull/211761

From llvmlistbot at llvm.org  Fri Jul 24 06:09:25 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:25 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636405.492e23e4.3ce411.4b38@mx.google.com>


https://github.com/ftynse commented:

Mostly nits, should be good to go with those fixed

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:26 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636406.d2cb4adc.23815e.9cf1@mx.google.com>


https://github.com/ftynse edited https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:26 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636406.e55dc341.dbc09.205f@mx.google.com>


================
@@ -3075,4 +3075,37 @@ def DecomposeWinogradOp : Op<Transform_Dialect,
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+def LinAlgToAffineOp : Op<Transform_Dialect, "structured.linalg_to_affine",
----------------
ftynse wrote:

```suggestion
def LinalgToAffineOp : Op<Transform_Dialect, "structured.to_affine",
```

Nit: we never capitalize the A in linalg

For the purposes of transform dialect, "structured" extension _is_ linalg, no need to duplicate.

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:27 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636407.a1accbb4.1ace1c.6492@mx.google.com>


================
@@ -3075,4 +3075,37 @@ def DecomposeWinogradOp : Op<Transform_Dialect,
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+def LinAlgToAffineOp : Op<Transform_Dialect, "structured.linalg_to_affine",
+    [FunctionalStyleTransformOpTrait,
+     MemoryEffectsOpInterface,
+     TransformOpInterface, TransformEachOpTrait,
+     ReportTrackingListenerFailuresOpTrait]> {
+  let description = [{
+    Convert a linalg operator into affine for loops.
+    The hypothesis are the same than the convert-linalg-to-affine
+    pass: we assume that the linalg operator is already generalized
+    (into linalg.generic operators) and that "memref" are used.
----------------
ftynse wrote:

What happens when this hypothesis is incorrect?

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:27 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636407.7270d2fd.1ec500.1e3f@mx.google.com>


================
@@ -3075,4 +3075,37 @@ def DecomposeWinogradOp : Op<Transform_Dialect,
   }];
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+def LinAlgToAffineOp : Op<Transform_Dialect, "structured.linalg_to_affine",
+    [FunctionalStyleTransformOpTrait,
+     MemoryEffectsOpInterface,
+     TransformOpInterface, TransformEachOpTrait,
+     ReportTrackingListenerFailuresOpTrait]> {
+  let description = [{
+    Convert a linalg operator into affine for loops.
----------------
ftynse wrote:

```suggestion
    Convert a linalg operation into affine for loops.
```
Nit

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:27 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636407.492e23e4.3ce411.4bb3@mx.google.com>


================
@@ -4593,6 +4593,41 @@ DiagnosedSilenceableFailure transform::DecomposeWinogradOp::applyToOne(
   return DiagnosedSilenceableFailure::success();
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure transform::LinAlgToAffineOp::applyToOne(
+    transform::TransformRewriter &rewriter, linalg::LinalgOp target,
+    ApplyToEachResultList &results, TransformState &state) {
+  if (! isa<GenericOp>(target)) {
+    return DiagnosedSilenceableFailure::definiteFailure();
----------------
ftynse wrote:

I'd rather make it a slienceable failure. Definite failures are used when you want to abort the entire transformation process because of irrecoverable IR mutation.

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:27 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636407.1e1ac41d.317291.cb3f@mx.google.com>


================
@@ -4593,6 +4593,41 @@ DiagnosedSilenceableFailure transform::DecomposeWinogradOp::applyToOne(
   return DiagnosedSilenceableFailure::success();
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure transform::LinAlgToAffineOp::applyToOne(
+    transform::TransformRewriter &rewriter, linalg::LinalgOp target,
+    ApplyToEachResultList &results, TransformState &state) {
+  if (! isa<GenericOp>(target)) {
+    return DiagnosedSilenceableFailure::definiteFailure();
+  }
+
+  rewriter.setInsertionPoint(target);
+
+  FailureOr<LinalgLoops> generic = linalgOpToAffineLoops(rewriter, target);
+  if (succeeded(generic)) {
+    assert(! generic->empty() && "expected at least one loop");
----------------
ftynse wrote:

```suggestion
    assert(!generic->empty() && "expected at least one loop");
```

Don't we have 0d generics?

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:28 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636408.1b0626d0.1ff4c3.01b3@mx.google.com>


================
@@ -4593,6 +4593,41 @@ DiagnosedSilenceableFailure transform::DecomposeWinogradOp::applyToOne(
   return DiagnosedSilenceableFailure::success();
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure transform::LinAlgToAffineOp::applyToOne(
+    transform::TransformRewriter &rewriter, linalg::LinalgOp target,
+    ApplyToEachResultList &results, TransformState &state) {
+  if (! isa<GenericOp>(target)) {
+    return DiagnosedSilenceableFailure::definiteFailure();
+  }
+
+  rewriter.setInsertionPoint(target);
+
+  FailureOr<LinalgLoops> generic = linalgOpToAffineLoops(rewriter, target);
+  if (succeeded(generic)) {
----------------
ftynse wrote:

https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:28 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636408.c4d3d80a.16dae5.3840@mx.google.com>


================
@@ -4593,6 +4593,41 @@ DiagnosedSilenceableFailure transform::DecomposeWinogradOp::applyToOne(
   return DiagnosedSilenceableFailure::success();
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure transform::LinAlgToAffineOp::applyToOne(
+    transform::TransformRewriter &rewriter, linalg::LinalgOp target,
+    ApplyToEachResultList &results, TransformState &state) {
+  if (! isa<GenericOp>(target)) {
----------------
ftynse wrote:

```suggestion
  if (!isa<GenericOp>(target)) {
```

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:28 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636408.666420a4.3304c7.23db@mx.google.com>


================
@@ -0,0 +1,94 @@
+// RUN: mlir-opt  %s --transform-interpreter -verify-diagnostics \
+// RUN:     --split-input-file | FileCheck %s
+
+//     CHECK: #[[MAP:.+]] = affine_map<(d0) -> (d0)>
+//     CHECK: func.func @convert_affine_matmul_buffer(%arg0: memref<16x8xf32>, %arg1: memref<8x32xf32>, %arg2: memref<16x32xf32>)
+// CHECK-DAG:   affine.for %[[arg3:.*]] = 0 to 16
+// CHECK-DAG:   affine.for %[[arg4:.*]] = 0 to 32
+// CHECK-DAG:   affine.for %[[arg5:.*]] = 0 to 8
+// CHECK-DAG:   %[[v0:.*]] = affine.apply #[[MAP]](%[[arg3]])
+// CHECK-DAG:   %[[v1:.*]] = affine.apply #[[MAP]](%[[arg5]])
+// CHECK-DAG:   %[[v2:.*]] = affine.load %arg0[%[[v0]], %[[v1]]] : memref<16x8xf32>
+// CHECK-DAG:   %[[v3:.*]] = affine.apply #[[MAP]](%[[arg5]])
+// CHECK-DAG:   %[[v4:.*]] = affine.apply #[[MAP]](%[[arg4]])
+// CHECK-DAG:   %[[v5:.*]] = affine.load %arg1[%[[v3]], %[[v4]]] : memref<8x32xf32>
+// CHECK-DAG:   %[[v6:.*]] = affine.apply #[[MAP]](%[[arg3]])
+// CHECK-DAG:   %[[v7:.*]] = affine.apply #[[MAP]](%[[arg4]])
+// CHECK-DAG:   %[[v8:.*]] = affine.load %arg2[%[[v6]], %[[v7]]] : memref<16x32xf32>
+// CHECK-DAG:   %[[v9:.*]] = affine.apply #[[MAP]](%[[arg3]])
+// CHECK-DAG:   %[[v10:.*]] = affine.apply #[[MAP]](%[[arg4]])
+// CHECK-DAG:   %[[v11:.*]] = arith.mulf %[[v2]], %[[v5]] : f32
+// CHECK-DAG:   %[[v12:.*]] = arith.addf %[[v8]], %[[v11]] : f32
----------------
ftynse wrote:

I don't think this should accept a DAG of lines, order matters here

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:09:28 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:09:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Transform dial] Expose
 convert-linalg-to-affine-loops transf (PR #211308)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211308@github.com>
Message-ID: <6a636408.af4834b4.5fdf.d299@mx.google.com>


================
@@ -4593,6 +4593,41 @@ DiagnosedSilenceableFailure transform::DecomposeWinogradOp::applyToOne(
   return DiagnosedSilenceableFailure::success();
 }
 
+//===----------------------------------------------------------------------===//
+// LinAlgToAffine
+//===----------------------------------------------------------------------===//
+
+DiagnosedSilenceableFailure transform::LinAlgToAffineOp::applyToOne(
+    transform::TransformRewriter &rewriter, linalg::LinalgOp target,
+    ApplyToEachResultList &results, TransformState &state) {
+  if (! isa<GenericOp>(target)) {
+    return DiagnosedSilenceableFailure::definiteFailure();
+  }
+
+  rewriter.setInsertionPoint(target);
+
+  FailureOr<LinalgLoops> generic = linalgOpToAffineLoops(rewriter, target);
+  if (succeeded(generic)) {
+    assert(! generic->empty() && "expected at least one loop");
+
+    // "generic" contains all new "AffineForOp", while we need to topmost one.
+    // Since all linalg operators are perfectly nested loops, these operators
+    //   are totally ordered through the ancestor relation.
+    Operation* opFirst = *(generic->begin());
+    for (auto itop = generic->begin(); itop!=generic->end(); itop++) {
+      if ( (*itop)->isProperAncestor(opFirst)) {
+        opFirst = *itop;
+      }
----------------
ftynse wrote:

Hmm, doesn't `linalgOpToAffineLoops` guarantee a specific order in which loops are returned? I'd expect it would.

https://github.com/llvm/llvm-project/pull/211308

From llvmlistbot at llvm.org  Fri Jul 24 06:11:20 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:11:20 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Fix warning, NFC (PR #211204)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211204@github.com>
Message-ID: <6a636478.1728eaf3.5e665.aefd@mx.google.com>


https://github.com/ftynse approved this pull request.


https://github.com/llvm/llvm-project/pull/211204

From llvmlistbot at llvm.org  Fri Jul 24 06:11:31 2026
From: llvmlistbot at llvm.org (Caroline Newcombe)
Date: Fri, 24 Jul 2026 06:11:31 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a636483.66c0679a.9a15c.6401@mx.google.com>


https://github.com/cenewcombe updated https://github.com/llvm/llvm-project/pull/206175

>From c7412986cd1c73900eeaa44624ee8e7859e7bfc0 Mon Sep 17 00:00:00 2001
From: Caroline Newcombe <caroline.newcombe at hpe.com>
Date: Fri, 26 Jun 2026 13:57:47 -0500
Subject: [PATCH 1/3] [flang][OpenMP] Implement the error directive

---
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 106 +++++++++++++++++-
 flang/lib/Semantics/check-omp-structure.cpp   |  50 +++++++++
 flang/lib/Semantics/check-omp-structure.h     |   3 +
 flang/test/Lower/OpenMP/Todo/error.f90        |   7 --
 flang/test/Lower/OpenMP/error.f90             |  47 ++++++++
 flang/test/Parser/OpenMP/error-unparse.f90    |  13 +--
 flang/test/Semantics/OpenMP/error.f90         |  43 +++++++
 .../llvm/Frontend/OpenMP/OMPIRBuilder.h       |  12 ++
 llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp     |  19 ++++
 .../mlir/Dialect/OpenMP/OpenMPEnums.td        |  20 ++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  31 +++++
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  |  12 ++
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      |  15 +++
 mlir/test/Dialect/OpenMP/invalid.mlir         |   7 ++
 mlir/test/Dialect/OpenMP/ops.mlir             |  25 +++++
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  24 ++++
 16 files changed, 416 insertions(+), 18 deletions(-)
 delete mode 100644 flang/test/Lower/OpenMP/Todo/error.f90
 create mode 100644 flang/test/Lower/OpenMP/error.f90

diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 996599212f5b9..185f28b7d3547 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -34,6 +34,7 @@
 #include "flang/Lower/Support/ReductionProcessor.h"
 #include "flang/Lower/SymbolMap.h"
 #include "flang/Optimizer/Builder/BoxValue.h"
+#include "flang/Optimizer/Builder/Character.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
@@ -6604,6 +6605,105 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                 queue.begin(), critName);
 }
 
+// Copy the character value `str` into fresh stack memory with an appended NUL
+// and return a pointer to its first character. Fortran characters are not
+// NUL-terminated, so the terminator is needed for C runtime entry points (such
+// as `__kmpc_error`) that expect a NUL-terminated string.
+static mlir::Value genNullTerminatedString(fir::FirOpBuilder &builder,
+                                           mlir::Location loc,
+                                           const fir::ExtendedValue &str) {
+  fir::factory::CharacterExprHelper helper(builder, loc);
+  const mlir::Value addr = fir::getBase(str);
+  const mlir::Value len = fir::getLen(str);
+  const auto charTy =
+      mlir::cast<fir::CharacterType>(fir::unwrapRefType(addr.getType()));
+  mlir::MLIRContext *ctx = builder.getContext();
+  const mlir::Type idxTy = builder.getIndexType();
+  const mlir::Value idxLen = builder.createConvert(loc, idxTy, len);
+  const mlir::Value one = builder.createIntegerConstant(loc, idxTy, 1);
+  const mlir::Value lenPlusOne =
+      mlir::arith::AddIOp::create(builder, loc, idxLen, one);
+
+  // Allocate `len + 1` characters and copy the message into them, leaving room
+  // for the terminating NUL.
+  const fir::CharBoxValue temp = helper.createCharacterTemp(charTy, lenPlusOne);
+  helper.createCopy(temp, fir::CharBoxValue{addr, len}, len);
+
+  // Address the buffer as an array of single characters so the terminating NUL
+  // can be stored at index `len`.
+  const auto singleTy = fir::CharacterType::get(
+      ctx, charTy.getFKind(), fir::CharacterType::singleton());
+  const mlir::Type singleRefTy = builder.getRefType(singleTy);
+  const mlir::Type seqRefTy = builder.getRefType(fir::SequenceType::get(
+      {fir::SequenceType::getUnknownExtent()}, singleTy));
+  const mlir::Value seq =
+      builder.createConvert(loc, seqRefTy, temp.getBuffer());
+  const mlir::Value nulAddr =
+      fir::CoordinateOp::create(builder, loc, singleRefTy, seq, idxLen);
+  const mlir::Value zero =
+      builder.createIntegerConstant(loc, builder.getI8Type(), 0);
+  const mlir::Value nul =
+      helper.createSingletonFromCode(zero, charTy.getFKind());
+  fir::StoreOp::create(builder, loc, nul, nulAddr);
+
+  // Pass a pointer to the first character of the buffer.
+  return builder.createConvert(loc, singleRefTy, temp.getBuffer());
+}
+
+// Lower an `!$omp error` directive. The `at(compilation)` form is handled
+// entirely in semantics, so only the `at(execution)` form reaches lowering,
+// where it becomes an `omp.error` operation.
+static void genErrorDirective(lower::AbstractConverter &converter,
+                              semantics::SemanticsContext &semaCtx,
+                              const parser::OmpErrorDirective &errDir) {
+  auto atKind = parser::OmpAtClause::ActionTime::Compilation;
+  auto sevKind = parser::OmpSeverityClause::SevLevel::Fatal;
+  std::optional<std::string> message;
+  MaybeExpr messageExpr;
+  for (const parser::OmpClause &clause : errDir.v.Clauses().v) {
+    if (const auto *at = std::get_if<parser::OmpClause::At>(&clause.u)) {
+      atKind = at->v.v;
+    } else if (const auto *sev =
+                   std::get_if<parser::OmpClause::Severity>(&clause.u)) {
+      sevKind = sev->v.v;
+    } else if (const auto *msg =
+                   std::get_if<parser::OmpClause::Message>(&clause.u)) {
+      if (auto expr = semantics::omp::GetEvaluateExpr(msg->v.v)) {
+        if (auto val = evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr))
+          message = *val;
+        else
+          messageExpr = expr;
+      }
+    }
+  }
+
+  if (atKind != parser::OmpAtClause::ActionTime::Execution ||
+      semaCtx.langOptions().OpenMPSimd)
+    return;
+
+  fir::FirOpBuilder &builder = converter.getFirOpBuilder();
+  const mlir::Location loc = converter.getCurrentLocation();
+  const mlir::omp::ClauseSeverity sev =
+      sevKind == parser::OmpSeverityClause::SevLevel::Warning
+          ? mlir::omp::ClauseSeverity::warning
+          : mlir::omp::ClauseSeverity::fatal;
+
+  // A compile-time-constant message is stored directly on the operation as an
+  // attribute. A non-constant message is lowered to a null-terminated string in
+  // memory and passed as the `message_expr` operand.
+  mlir::StringAttr msgAttr;
+  mlir::Value msgExprVal;
+  if (message) {
+    msgAttr = builder.getStringAttr(*message);
+  } else if (messageExpr) {
+    lower::StatementContext stmtCtx;
+    fir::ExtendedValue str = converter.genExprAddr(loc, *messageExpr, stmtCtx);
+    msgExprVal = genNullTerminatedString(builder, loc, str);
+  }
+
+  mlir::omp::ErrorOp::create(builder, loc, sev, msgAttr, msgExprVal);
+}
+
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
@@ -6612,10 +6712,8 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                     [&](const parser::OmpNothingDirective &) {
                       // nothing-directive is a no-op (OpenMP 5.2 [8.4])
                     },
-                    [&](const parser::OmpErrorDirective &) {
-                      if (!semaCtx.langOptions().OpenMPSimd)
-                        TODO(converter.getCurrentLocation(),
-                             "OmpErrorDirective");
+                    [&](const parser::OmpErrorDirective &errDir) {
+                      genErrorDirective(converter, semaCtx, errDir);
                     },
                 },
                 dir.u);
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 0547baaa4ecba..ec13cce146a96 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2725,6 +2725,38 @@ void OmpStructureChecker::Enter(const parser::OpenMPDispatchConstruct &x) {
   }
 }
 
+void OmpStructureChecker::Enter(const parser::OmpErrorDirective &x) {
+  // Defaults per spec: AT(compilation), SEVERITY(fatal)
+  auto atKind{parser::OmpAtClause::ActionTime::Compilation};
+  auto sevKind{parser::OmpSeverityClause::SevLevel::Fatal};
+  std::optional<std::string> message;
+
+  for (const parser::OmpClause &clause : x.v.Clauses().v) {
+    if (const auto *at{std::get_if<parser::OmpClause::At>(&clause.u)}) {
+      atKind = at->v.v;
+    } else if (const auto *sev{
+                   std::get_if<parser::OmpClause::Severity>(&clause.u)}) {
+      sevKind = sev->v.v;
+    } else if (const auto *msg{
+                   std::get_if<parser::OmpClause::Message>(&clause.u)}) {
+      if (auto expr{GetEvaluateExpr(msg->v.v)}) {
+        if (auto val{
+                evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr)}) {
+          message = *val;
+        }
+      }
+    }
+  }
+
+  if (atKind == parser::OmpAtClause::ActionTime::Compilation) {
+    if (sevKind == parser::OmpSeverityClause::SevLevel::Warning) {
+      context_.Say(x.v.source, "%s"_warn_en_US, message.value_or("WARNING"));
+    } else {
+      context_.Say(x.v.source, "%s"_err_en_US, message.value_or("ERROR"));
+    }
+  }
+}
+
 void OmpStructureChecker::Enter(const parser::OmpClause::At &x) {
   if (GetDirectiveNest(DeclarativeNest) > 0) {
     if (x.v.v == parser::OmpAtClause::ActionTime::Execution) {
@@ -2734,6 +2766,24 @@ void OmpStructureChecker::Enter(const parser::OmpClause::At &x) {
   }
 }
 
+void OmpStructureChecker::Enter(const parser::OmpClause::Message &x) {
+  if (const auto expr{GetEvaluateExpr(x.v.v)}) {
+    const std::optional<evaluate::DynamicType> type{expr->GetType()};
+    if (!type || type->category() != evaluate::TypeCategory::Character) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be of type CHARACTER"_err_en_US);
+    } else if (expr->Rank() != 0) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be scalar"_err_en_US);
+    } else if (type->kind() !=
+        context_.defaultKinds().GetDefaultKind(
+            evaluate::TypeCategory::Character)) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be of default character kind"_err_en_US);
+    }
+  }
+}
+
 void OmpStructureChecker::Enter(const parser::OpenMPAllocatorsConstruct &x) {
   const parser::OmpDirectiveSpecification &beginSpec{x.BeginDir()};
   const parser::OmpDirectiveName &dirName{beginSpec.DirName()};
diff --git a/flang/lib/Semantics/check-omp-structure.h b/flang/lib/Semantics/check-omp-structure.h
index 0f21bbd19189e..22106020bcc3b 100644
--- a/flang/lib/Semantics/check-omp-structure.h
+++ b/flang/lib/Semantics/check-omp-structure.h
@@ -145,6 +145,8 @@ class OmpStructureChecker : public OmpStructureCheckerBase {
   void Enter(const parser::OmpDirectiveSpecification &);
   void Leave(const parser::OmpDirectiveSpecification &);
 
+  void Enter(const parser::OmpErrorDirective &);
+
   void Enter(const parser::OmpMetadirectiveDirective &);
   void Leave(const parser::OmpMetadirectiveDirective &);
 
@@ -202,6 +204,7 @@ class OmpStructureChecker : public OmpStructureCheckerBase {
   void Enter(const parser::OmpClause::Linear &x);
   void Enter(const parser::OmpClause::Looprange &x);
   void Enter(const parser::OmpClause::Map &x);
+  void Enter(const parser::OmpClause::Message &x);
   void Enter(const parser::OmpClause::NumTeams &x);
   void Enter(const parser::OmpClause::NumThreads &x);
   void Enter(const parser::OmpClause::OmpxBare &x);
diff --git a/flang/test/Lower/OpenMP/Todo/error.f90 b/flang/test/Lower/OpenMP/Todo/error.f90
deleted file mode 100644
index c8338278a4267..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/error.f90
+++ /dev/null
@@ -1,7 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OmpErrorDirective
-program p
-  integer, allocatable :: x
-  !$omp error at(compilation) severity(warning) message("an error")
-end program p
diff --git a/flang/test/Lower/OpenMP/error.f90 b/flang/test/Lower/OpenMP/error.f90
new file mode 100644
index 0000000000000..fbca3ec4f45e2
--- /dev/null
+++ b/flang/test/Lower/OpenMP/error.f90
@@ -0,0 +1,47 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+! Lowering of `!$omp error at(execution)` to the `omp.error` operation. The
+! `at(compilation)` form is handled entirely in semantics and produces no IR.
+! Severity is `warning` or `fatal` (default `fatal`); the message is an optional
+! constant string that is omitted when absent. A non-constant message is lowered
+! to a null-terminated runtime string and passed as an operand.
+
+! CHECK-LABEL: func.func @_QQmain()
+program p
+  ! CHECK: omp.error severity(warning) message("a warning")
+  !$omp error at(execution) severity(warning) message("a warning")
+
+  ! CHECK: omp.error severity(fatal) message("fatal error")
+  !$omp error at(execution) severity(fatal) message("fatal error")
+
+  ! No MESSAGE clause; severity defaults to `fatal`.
+  ! CHECK: omp.error severity(fatal)
+  !$omp error at(execution)
+
+  ! An explicit empty MESSAGE string is lowered to an empty constant `message`
+  ! attribute (distinct from an absent clause) at the HLFIR level.
+  ! CHECK: omp.error severity(warning) message("")
+  !$omp error at(execution) severity(warning) message("")
+end program p
+
+! A non-constant MESSAGE is only known at run time, so it is lowered to a
+! null-terminated copy in memory and passed as the `message_expr` operand
+! instead of as a constant attribute.
+! CHECK-LABEL: func.func @_QPf_runtime_message
+subroutine f_runtime_message
+  character(len=16) :: msg
+  msg = "runtime warning"
+  ! CHECK: omp.error severity(warning) message_expr(%{{.*}} : !fir.ref<!fir.char<1>>)
+  !$omp error at(execution) severity(warning) message(msg)
+end subroutine f_runtime_message
+
+! The `at(compilation)` form (and the implicit default, which is also
+! compilation) is resolved entirely in semantics and must not lower to an
+! `omp.error` operation. `severity(warning)` keeps these as non-fatal
+! diagnostics so compilation still succeeds.
+! CHECK-LABEL: func.func @_QPf_compilation
+subroutine f_compilation
+  ! CHECK-NOT: omp.error
+  !$omp error severity(warning) message("implicit compilation")
+  !$omp error at(compilation) severity(warning) message("explicit compilation")
+end subroutine f_compilation
diff --git a/flang/test/Parser/OpenMP/error-unparse.f90 b/flang/test/Parser/OpenMP/error-unparse.f90
index 0902d109fd233..255da1111cbf3 100644
--- a/flang/test/Parser/OpenMP/error-unparse.f90
+++ b/flang/test/Parser/OpenMP/error-unparse.f90
@@ -1,5 +1,11 @@
 ! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-unparse %s 2>&1 | FileCheck %s
 ! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s --check-prefix="PARSE-TREE"
+
+! The AT(COMPILATION) form emits its diagnostic during semantics, so SEVERITY(FATAL)
+! would halt compilation before any unparse/parse-tree output is produced. Only
+! SEVERITY(WARNING) is used with AT(COMPILATION) here; SEVERITY(FATAL) is exercised
+! with AT(EXECUTION), which is not diagnosed until lowering. The compilation-time
+! diagnostics themselves are covered by Semantics/OpenMP/error.f90.
 program main
   character(*), parameter :: message = "This is an error"
   !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(WARNING) MESSAGE("some message here")
@@ -10,13 +16,6 @@ program main
   !PARSE-TREE:  LiteralConstant -> CharLiteralConstant
   !PARSE-TREE:  string = 'some message here'
   !$omp error at(compilation) severity(warning) message("some message here")
-  !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(FATAL) MESSAGE("This is an error")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
-  !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Compilation
-  !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Fatal
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"This is an error"'
-  !PARSE-TREE:  Designator -> DataRef -> Name = 'message'
-  !$omp error at(compilation) severity(fatal) message(message)
   !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(FATAL) MESSAGE("This is an error")
   !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Execution
diff --git a/flang/test/Semantics/OpenMP/error.f90 b/flang/test/Semantics/OpenMP/error.f90
index 067417a8cda3b..97eaa48d9899e 100644
--- a/flang/test/Semantics/OpenMP/error.f90
+++ b/flang/test/Semantics/OpenMP/error.f90
@@ -6,3 +6,46 @@ subroutine f00(x)
   integer :: x
 end
 
+subroutine f01
+!ERROR: an error
+  !$omp error at(compilation) severity(fatal) message("an error")
+end
+
+subroutine f02
+!WARNING: a warning
+  !$omp error at(compilation) severity(warning) message("a warning")
+end
+
+subroutine f03
+!ERROR: ERROR
+  !$omp error
+end
+
+subroutine f04
+!ERROR: ERROR
+  !$omp error at(compilation) severity(fatal)
+end
+
+subroutine f05
+!WARNING: WARNING
+  !$omp error at(compilation) severity(warning)
+end
+
+subroutine f06(n)
+  integer :: n
+!ERROR: The MESSAGE clause expression must be of type CHARACTER
+  !$omp error at(execution) severity(warning) message(n)
+end
+
+subroutine f07(c)
+  character(len=8) :: c(4)
+!ERROR: The MESSAGE clause expression must be scalar
+  !$omp error at(execution) severity(warning) message(c)
+end
+
+subroutine f08
+  character(kind=4, len=8) :: c
+!ERROR: The MESSAGE clause expression must be of default character kind
+  !$omp error at(execution) severity(warning) message(c)
+end
+
diff --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index fee8c712998b5..05307414c94d0 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -1497,6 +1497,18 @@ class OpenMPIRBuilder {
   /// \param Loc The location where the flush directive was encountered
   LLVM_ABI void createFlush(const LocationDescription &Loc);
 
+  /// Generate a call to the runtime to emit the diagnostic of an OpenMP
+  /// `error` directive with `at(execution)`.
+  ///
+  /// \param Loc The location where the error directive was encountered; it is
+  ///        used to build the `ident_t` passed to the runtime.
+  /// \param IsFatal Selects `severity(fatal)` (true) or `severity(warning)`
+  ///        (false).
+  /// \param Message The message string (an `i8*`) to display, or null when no
+  ///        `message` clause is present.
+  LLVM_ABI void createError(const LocationDescription &Loc, bool IsFatal,
+                            Value *Message);
+
   /// Generator for '#omp taskyield'
   ///
   /// \param Loc The location where the taskyield directive was encountered.
diff --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 57e199bdff09a..fc3812502fb2e 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2184,6 +2184,25 @@ void OpenMPIRBuilder::createFlush(const LocationDescription &Loc) {
   emitFlush(Loc);
 }
 
+void OpenMPIRBuilder::createError(const LocationDescription &Loc, bool IsFatal,
+                                  Value *Message) {
+  if (!updateToLocation(Loc))
+    return;
+
+  // Build call void __kmpc_error(ident_t *loc, int severity,
+  //                              const char *message)
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  // Severity: 1 = warning, 2 = fatal.
+  Value *Severity = ConstantInt::get(Int32, IsFatal ? 2 : 1);
+  Value *MessageArg = Message ? Message : ConstantPointerNull::get(Int8Ptr);
+  Value *Args[] = {Ident, Severity, MessageArg};
+
+  createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_error),
+                            Args);
+}
+
 void OpenMPIRBuilder::emitTaskyieldImpl(const LocationDescription &Loc) {
   // Build call __kmpc_omp_taskyield(loc, thread_id, 0);
   uint32_t SrcLocStrSize;
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
index e373d1a52a3fc..c642792f67019 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
@@ -77,6 +77,26 @@ def ClauseDependAttr : OpenMP_EnumAttr<ClauseDepend, "clause_depend"> {
   let assemblyFormat = "`(` $value `)`";
 }
 
+//===----------------------------------------------------------------------===//
+// severity enum.
+//===----------------------------------------------------------------------===//
+
+// The values match the `severity` argument expected by the `__kmpc_error`
+// runtime entry point (1 = warning, 2 = fatal).
+def ClauseSeverityWarning : I32EnumAttrCase<"warning", 1>;
+def ClauseSeverityFatal : I32EnumAttrCase<"fatal", 2>;
+
+def ClauseSeverity : OpenMP_I32EnumAttr<
+    "ClauseSeverity",
+    "severity clause", [
+      ClauseSeverityWarning,
+      ClauseSeverityFatal
+    ]>;
+
+def ClauseSeverityAttr : OpenMP_EnumAttr<ClauseSeverity, "clause_severity"> {
+  let assemblyFormat = "`(` $value `)`";
+}
+
 //===----------------------------------------------------------------------===//
 // clause_requires enum.
 //===----------------------------------------------------------------------===//
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0a5736f46b088..37e98c5c5138e 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1775,6 +1775,37 @@ def BarrierOp : OpenMP_Op<"barrier"> {
   let assemblyFormat = "attr-dict";
 }
 
+//===----------------------------------------------------------------------===//
+// [5.2] 10.4 error directive
+//===----------------------------------------------------------------------===//
+
+def ErrorOp : OpenMP_Op<"error"> {
+  let summary = "error directive";
+  let description = [{
+    The error directive instructs the compiler or runtime to display a message
+    and, when the severity is `fatal`, to terminate execution. The
+    `at(compilation)` form is handled entirely by the frontend; this operation
+    represents only the `at(execution)` form, which lowers to a call to the
+    `__kmpc_error` runtime entry point.
+
+    The `severity` attribute selects `warning` or `fatal` behavior. The message
+    to display is optional and may be provided as a compile-time-constant
+    string in the `message` attribute, or a pointer to a null-terminated string.
+  }];
+
+  let arguments = (ins ClauseSeverityAttr:$severity,
+                       OptionalAttr<StrAttr>:$message,
+                       Optional<OpenMP_PointerLikeType>:$message_expr);
+
+  let assemblyFormat = [{
+    `severity` `(` custom<ClauseAttr>($severity) `)`
+    ( `message` `(` $message^ `)` )?
+    ( `message_expr` `(` $message_expr^ `:` type($message_expr) `)` )? attr-dict
+  }];
+
+  let hasVerifier = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // [5.1] 2.19.9 ordered Construct
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 67bc2bdf35619..95ccb53bce3d4 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4618,6 +4618,18 @@ LogicalResult CriticalOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Error directive (2.5.4)
+//===----------------------------------------------------------------------===//
+
+LogicalResult ErrorOp::verify() {
+  if (getMessage() && getMessageExpr())
+    return emitOpError() << "the message must be provided either as a constant "
+                            "`message` attribute or as a `message_expr` "
+                            "operand, but not both";
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // Ordered construct
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 1c38fd27daae1..97993a80926eb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -9828,6 +9828,21 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             ompBuilder->createFlush(builder);
             return success();
           })
+          .Case([&](omp::ErrorOp op) {
+            if (failed(checkImplementationStatus(*op)))
+              return failure();
+
+            llvm::Value *message = nullptr;
+            if (mlir::Value messageExpr = op.getMessageExpr())
+              message = moduleTranslation.lookupValue(messageExpr);
+            else if (std::optional<StringRef> msg = op.getMessage();
+                     msg && !msg->empty())
+              message = builder.CreateGlobalString(*msg);
+            ompBuilder->createError(
+                llvm::OpenMPIRBuilder::LocationDescription(builder),
+                op.getSeverity() == omp::ClauseSeverity::fatal, message);
+            return success();
+          })
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })
diff --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index cf7be4eb638cc..a568bffe3cf99 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4913,3 +4913,10 @@ func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
   omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
   return
 }
+
+// -----
+func.func @omp_error_message_and_message_expr(%msg : !llvm.ptr) -> () {
+  // expected-error @below {{the message must be provided either as a constant `message` attribute or as a `message_expr` operand, but not both}}
+  omp.error severity(warning) message("a warning") message_expr(%msg : !llvm.ptr)
+  return
+}
diff --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index d08ca5664efe4..ea25a0a75f646 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -60,6 +60,31 @@ func.func @omp_flush(%arg0 : memref<i32>) -> () {
   return
 }
 
+// CHECK-LABEL: func @omp_error
+func.func @omp_error(%msg : !llvm.ptr) -> () {
+  // Warning severity with a message.
+  // CHECK: omp.error severity(warning) message("a warning")
+  omp.error severity(warning) message("a warning")
+
+  // Fatal severity with a message.
+  // CHECK: omp.error severity(fatal) message("fatal error")
+  omp.error severity(fatal) message("fatal error")
+
+  // Fatal severity without a message.
+  // CHECK: omp.error severity(fatal)
+  omp.error severity(fatal)
+
+  // Warning severity without a message.
+  // CHECK: omp.error severity(warning)
+  omp.error severity(warning)
+
+  // Warning severity with a runtime message operand.
+  // CHECK: omp.error severity(warning) message_expr(%{{.*}} : !llvm.ptr)
+  omp.error severity(warning) message_expr(%msg : !llvm.ptr)
+
+  return
+}
+
 func.func @omp_terminator() -> () {
   // CHECK: omp.terminator
   omp.terminator
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index db38c8da30777..d26557d57d0dd 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -41,6 +41,30 @@ llvm.func @test_flush_construct(%arg0: !llvm.ptr) {
   llvm.return
 }
 
+// CHECK-LABEL: define void @test_error_construct
+llvm.func @test_error_construct(%msg : !llvm.ptr) {
+  // Severity is 1 for `warning` and 2 for `fatal`; the message is a pointer to
+  // a constant string, or a null pointer when absent.
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr @{{.*}})
+  omp.error severity(warning) message("a warning")
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 2, ptr @{{.*}})
+  omp.error severity(fatal) message("fatal error")
+
+  // A runtime message is passed through as the supplied pointer operand.
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr %{{.*}})
+  omp.error severity(warning) message_expr(%msg : !llvm.ptr)
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 2, ptr null)
+  omp.error severity(fatal)
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr null)
+  omp.error severity(warning)
+
+  // CHECK-NEXT:    ret void
+  llvm.return
+}
+
 // CHECK-LABEL: define void @test_omp_parallel_1()
 llvm.func @test_omp_parallel_1() -> () {
   // CHECK: call void{{.*}}@__kmpc_fork_call{{.*}}@[[OMP_OUTLINED_FN_1:.*]])

>From 58e52a7e857f8caaa21156a0f34482973ad78db9 Mon Sep 17 00:00:00 2001
From: Caroline Newcombe <caroline.newcombe at hpe.com>
Date: Tue, 30 Jun 2026 09:33:59 -0500
Subject: [PATCH 2/3] [flang][OpenMP] Use no-sema unparse/parse-tree dump in
 ERROR directive test to cover compilation-time fatal cases

---
 flang/test/Parser/OpenMP/error-unparse.f90 | 39 ++++++++++++++--------
 1 file changed, 25 insertions(+), 14 deletions(-)

diff --git a/flang/test/Parser/OpenMP/error-unparse.f90 b/flang/test/Parser/OpenMP/error-unparse.f90
index 255da1111cbf3..5689de8374297 100644
--- a/flang/test/Parser/OpenMP/error-unparse.f90
+++ b/flang/test/Parser/OpenMP/error-unparse.f90
@@ -1,26 +1,37 @@
-! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-unparse %s 2>&1 | FileCheck %s
-! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s --check-prefix="PARSE-TREE"
+! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-unparse-no-sema %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-dump-parse-tree-no-sema %s 2>&1 | FileCheck %s --check-prefix="PARSE-TREE"
 
-! The AT(COMPILATION) form emits its diagnostic during semantics, so SEVERITY(FATAL)
-! would halt compilation before any unparse/parse-tree output is produced. Only
-! SEVERITY(WARNING) is used with AT(COMPILATION) here; SEVERITY(FATAL) is exercised
-! with AT(EXECUTION), which is not diagnosed until lowering. The compilation-time
-! diagnostics themselves are covered by Semantics/OpenMP/error.f90.
+! Compilation-time diagnostics are covered by Semantics/OpenMP/error.f90.
 program main
   character(*), parameter :: message = "This is an error"
   !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(WARNING) MESSAGE("some message here")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Compilation
   !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Warning
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"some message here"'
-  !PARSE-TREE:  LiteralConstant -> CharLiteralConstant
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> LiteralConstant -> CharLiteralConstant
   !PARSE-TREE:  string = 'some message here'
   !$omp error at(compilation) severity(warning) message("some message here")
-  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(FATAL) MESSAGE("This is an error")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
+  !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(FATAL) MESSAGE(message)
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
+  !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Compilation
+  !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Fatal
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> Designator -> DataRef -> Name = 'message'
+  !$omp error at(compilation) severity(fatal) message(message)
+  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(FATAL) MESSAGE(message)
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Execution
   !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Fatal
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"This is an error"'
-  !PARSE-TREE:  Designator ->  DataRef -> Name = 'message'
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> Designator -> DataRef -> Name = 'message'
   !$omp error at(EXECUTION) severity(fatal) message(message)
+  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(WARNING) MESSAGE("a warning")
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
+  !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Execution
+  !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Warning
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> LiteralConstant -> CharLiteralConstant
+  !PARSE-TREE:  string = 'a warning'
+  !$omp error at(execution) severity(warning) message("a warning")
 end program main

>From d42bc53e3db47c3492be41a442e021cc2e0e4404 Mon Sep 17 00:00:00 2001
From: Caroline Newcombe <caroline.newcombe at hpe.com>
Date: Wed, 1 Jul 2026 11:13:03 -0500
Subject: [PATCH 3/3] [flang][OpenMP] Add GetErrorDirectiveArgs helper to share
 error-directive clause parsing

---
 flang/include/flang/Semantics/openmp-utils.h  | 15 ++++++++
 flang/lib/Lower/OpenMP/OpenMP.cpp             | 35 +++++++----------
 flang/lib/Semantics/check-omp-structure.cpp   | 38 ++++++++-----------
 flang/lib/Semantics/openmp-utils.cpp          | 16 ++++++++
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  2 +-
 5 files changed, 61 insertions(+), 45 deletions(-)

diff --git a/flang/include/flang/Semantics/openmp-utils.h b/flang/include/flang/Semantics/openmp-utils.h
index 496cad0b3bd57..031926d4a7cc2 100644
--- a/flang/include/flang/Semantics/openmp-utils.h
+++ b/flang/include/flang/Semantics/openmp-utils.h
@@ -298,6 +298,21 @@ bool HasDataEnvironment(llvm::omp::Directive dir);
 
 bool IsFullUnroll(const parser::OmpDirectiveSpecification &spec);
 
+/// The AT, SEVERITY, and MESSAGE clause values of an `!$omp error` directive.
+/// `at` and `severity` default to AT(compilation)/SEVERITY(fatal) when absent;
+/// `message` is null when there is no MESSAGE clause.
+struct OmpErrorArgs {
+  parser::OmpAtClause::ActionTime at{
+      parser::OmpAtClause::ActionTime::Compilation};
+  parser::OmpSeverityClause::SevLevel severity{
+      parser::OmpSeverityClause::SevLevel::Fatal};
+  const parser::Expr *message{nullptr};
+};
+
+/// Scan the clause list of an `!$omp error` directive for its AT, SEVERITY, and
+/// MESSAGE clause values.
+OmpErrorArgs GetErrorDirectiveArgs(const parser::OmpErrorDirective &errDir);
+
 inline bool IsDoConcurrentLegal(unsigned version) {
   // DO CONCURRENT is allowed (as an alternative to a Canonical Loop Nest)
   // in OpenMP 6.0+.
diff --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 185f28b7d3547..9807023935784 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -6656,35 +6656,28 @@ static mlir::Value genNullTerminatedString(fir::FirOpBuilder &builder,
 static void genErrorDirective(lower::AbstractConverter &converter,
                               semantics::SemanticsContext &semaCtx,
                               const parser::OmpErrorDirective &errDir) {
-  auto atKind = parser::OmpAtClause::ActionTime::Compilation;
-  auto sevKind = parser::OmpSeverityClause::SevLevel::Fatal;
+  const semantics::omp::OmpErrorArgs args{
+      semantics::omp::GetErrorDirectiveArgs(errDir)};
+
+  if (args.at != parser::OmpAtClause::ActionTime::Execution ||
+      semaCtx.langOptions().OpenMPSimd)
+    return;
+
   std::optional<std::string> message;
   MaybeExpr messageExpr;
-  for (const parser::OmpClause &clause : errDir.v.Clauses().v) {
-    if (const auto *at = std::get_if<parser::OmpClause::At>(&clause.u)) {
-      atKind = at->v.v;
-    } else if (const auto *sev =
-                   std::get_if<parser::OmpClause::Severity>(&clause.u)) {
-      sevKind = sev->v.v;
-    } else if (const auto *msg =
-                   std::get_if<parser::OmpClause::Message>(&clause.u)) {
-      if (auto expr = semantics::omp::GetEvaluateExpr(msg->v.v)) {
-        if (auto val = evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr))
-          message = *val;
-        else
-          messageExpr = expr;
-      }
+  if (args.message) {
+    if (auto expr = semantics::omp::GetEvaluateExpr(*args.message)) {
+      if (auto val = evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr))
+        message = *val;
+      else
+        messageExpr = expr;
     }
   }
 
-  if (atKind != parser::OmpAtClause::ActionTime::Execution ||
-      semaCtx.langOptions().OpenMPSimd)
-    return;
-
   fir::FirOpBuilder &builder = converter.getFirOpBuilder();
   const mlir::Location loc = converter.getCurrentLocation();
   const mlir::omp::ClauseSeverity sev =
-      sevKind == parser::OmpSeverityClause::SevLevel::Warning
+      args.severity == parser::OmpSeverityClause::SevLevel::Warning
           ? mlir::omp::ClauseSeverity::warning
           : mlir::omp::ClauseSeverity::fatal;
 
diff --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index ec13cce146a96..f77382c9045fc 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2726,34 +2726,26 @@ void OmpStructureChecker::Enter(const parser::OpenMPDispatchConstruct &x) {
 }
 
 void OmpStructureChecker::Enter(const parser::OmpErrorDirective &x) {
-  // Defaults per spec: AT(compilation), SEVERITY(fatal)
-  auto atKind{parser::OmpAtClause::ActionTime::Compilation};
-  auto sevKind{parser::OmpSeverityClause::SevLevel::Fatal};
-  std::optional<std::string> message;
+  const OmpErrorArgs args{GetErrorDirectiveArgs(x)};
 
-  for (const parser::OmpClause &clause : x.v.Clauses().v) {
-    if (const auto *at{std::get_if<parser::OmpClause::At>(&clause.u)}) {
-      atKind = at->v.v;
-    } else if (const auto *sev{
-                   std::get_if<parser::OmpClause::Severity>(&clause.u)}) {
-      sevKind = sev->v.v;
-    } else if (const auto *msg{
-                   std::get_if<parser::OmpClause::Message>(&clause.u)}) {
-      if (auto expr{GetEvaluateExpr(msg->v.v)}) {
-        if (auto val{
-                evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr)}) {
-          message = *val;
-        }
+  // The AT(execution) form (non-default) is handled at run time.
+  if (args.at != parser::OmpAtClause::ActionTime::Compilation) {
+    return;
+  }
+
+  std::optional<std::string> message;
+  if (args.message) {
+    if (auto expr{GetEvaluateExpr(*args.message)}) {
+      if (auto val{evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr)}) {
+        message = *val;
       }
     }
   }
 
-  if (atKind == parser::OmpAtClause::ActionTime::Compilation) {
-    if (sevKind == parser::OmpSeverityClause::SevLevel::Warning) {
-      context_.Say(x.v.source, "%s"_warn_en_US, message.value_or("WARNING"));
-    } else {
-      context_.Say(x.v.source, "%s"_err_en_US, message.value_or("ERROR"));
-    }
+  if (args.severity == parser::OmpSeverityClause::SevLevel::Warning) {
+    context_.Say(x.v.source, "%s"_warn_en_US, message.value_or("WARNING"));
+  } else {
+    context_.Say(x.v.source, "%s"_err_en_US, message.value_or("ERROR"));
   }
 }
 
diff --git a/flang/lib/Semantics/openmp-utils.cpp b/flang/lib/Semantics/openmp-utils.cpp
index 27d23a9ffde30..731d98f1103a5 100644
--- a/flang/lib/Semantics/openmp-utils.cpp
+++ b/flang/lib/Semantics/openmp-utils.cpp
@@ -920,6 +920,22 @@ bool IsFullUnroll(const parser::OmpDirectiveSpecification &spec) {
   return false;
 }
 
+OmpErrorArgs GetErrorDirectiveArgs(const parser::OmpErrorDirective &errDir) {
+  OmpErrorArgs args;
+  for (const parser::OmpClause &clause : errDir.v.Clauses().v) {
+    if (const auto *at{std::get_if<parser::OmpClause::At>(&clause.u)}) {
+      args.at = at->v.v;
+    } else if (const auto *sev{
+                   std::get_if<parser::OmpClause::Severity>(&clause.u)}) {
+      args.severity = sev->v.v;
+    } else if (const auto *msg{
+                   std::get_if<parser::OmpClause::Message>(&clause.u)}) {
+      args.message = &msg->v.v;
+    }
+  }
+  return args;
+}
+
 static bool IsTransformableLoop(const parser::OmpDirectiveSpecification &spec) {
   return !IsFullUnroll(spec) && IsLoopTransforming(spec.DirId());
 }
diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 37e98c5c5138e..70597b85902c5 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1776,7 +1776,7 @@ def BarrierOp : OpenMP_Op<"barrier"> {
 }
 
 //===----------------------------------------------------------------------===//
-// [5.2] 10.4 error directive
+// [5.1] 2.5.4 error directive
 //===----------------------------------------------------------------------===//
 
 def ErrorOp : OpenMP_Op<"error"> {


From llvmlistbot at llvm.org  Fri Jul 24 06:16:14 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:16:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a63659e.e5907290.33b622.e15d@mx.google.com>


https://github.com/ftynse edited https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Fri Jul 24 06:16:14 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:16:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a63659e.7270d2fd.1ec500.fa17@mx.google.com>


https://github.com/ftynse approved this pull request.

Thanks for the fix

https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Fri Jul 24 06:16:14 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:16:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a63659e.bc28b987.1c0bcd.4753@mx.google.com>


================
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
----------------
ftynse wrote:

This function looks dangerous as it is, can we either (a) rename this to `mustHaveDependence` and introduce a `mayHaveDependence` that conservatively returns true if failed to analyze or (b) make it assert that result != Failure?

https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Fri Jul 24 06:22:34 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:22:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add affine.for verifier and
 move arguments check earlier (PR #206685)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206685@github.com>
Message-ID: <6a63671a.83ce629a.2277d7.74c8@mx.google.com>


https://github.com/ftynse approved this pull request.


https://github.com/llvm/llvm-project/pull/206685

From llvmlistbot at llvm.org  Fri Jul 24 06:22:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 06:22:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] fdbbcba - [mlir][affine] Add affine.for
 verifier and move arguments check earlier (#206685)
Message-ID: <6a636723.5a0412d2.325e39.c0f0@mx.google.com>


Author: Hocky Yudhiono
Date: 2026-07-24T15:22:37+02:00
New Revision: fdbbcba77d625a5c403aea78b5b7fc64547fa558

URL: https://github.com/llvm/llvm-project/commit/fdbbcba77d625a5c403aea78b5b7fc64547fa558
DIFF: https://github.com/llvm/llvm-project/commit/fdbbcba77d625a5c403aea78b5b7fc64547fa558.diff

LOG: [mlir][affine] Add affine.for verifier and move arguments check earlier (#206685)

Fixes #206628 crash by adding an earlier body argument verifier for
`affine.for`. This crash is caused because `LoopLikeOpInterface`
verifier would call `getRegionIterArgs()` and assumed induction var of
`affine.for` exists.

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
    mlir/lib/Dialect/Affine/IR/AffineOps.cpp
    mlir/test/Dialect/Affine/invalid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
index 1e14f9f37288d..57cbbb06b9f65 100644
--- a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
+++ b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
@@ -343,6 +343,7 @@ def AffineForOp : Affine_Op<"for",
 
   let hasCustomAssemblyFormat = 1;
   let hasFolder = 1;
+  let hasVerifier = 1;
   let hasRegionVerifier = 1;
 }
 

diff  --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index defb029865835..98c2be14da5aa 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -2203,19 +2203,21 @@ void AffineForOp::build(OpBuilder &builder, OperationState &result, int64_t lb,
                bodyBuilder);
 }
 
+LogicalResult AffineForOp::verify() {
+  auto *body = getBody();
+  if (body->getNumArguments() == 0 || !getInductionVar().getType().isIndex())
+    return emitOpError("expected body to have an index argument for the "
+                       "induction variable");
+
+  return success();
+}
+
 LogicalResult AffineForOp::verifyRegions() {
   // Step must be a strictly positive integer.
   if (getStepAsInt() <= 0)
     return emitOpError("expected step to be a positive integer, got ")
            << getStepAsInt();
 
-  // Check that the body defines as single block argument for the induction
-  // variable.
-  auto *body = getBody();
-  if (body->getNumArguments() == 0 || !body->getArgument(0).getType().isIndex())
-    return emitOpError("expected body to have a single index argument for the "
-                       "induction variable");
-
   // Verify that the bound operands are valid dimension/symbols.
   /// Lower bound.
   if (getLowerBoundMap().getNumInputs() > 0)

diff  --git a/mlir/test/Dialect/Affine/invalid.mlir b/mlir/test/Dialect/Affine/invalid.mlir
index e31f2f00f96e2..ae209b4092d98 100644
--- a/mlir/test/Dialect/Affine/invalid.mlir
+++ b/mlir/test/Dialect/Affine/invalid.mlir
@@ -632,3 +632,13 @@ func.func @affine_for_zero_step_verifier() {
   }) : () -> ()
   return
 }
+
+// -----
+
+func.func @affine_for_missing_induction_var() {
+  // expected-error at +1 {{'affine.for' op expected body to have an index argument for the induction variable}}
+  "affine.for"() <{lowerBoundMap = affine_map<() -> (0)>, operandSegmentSizes = array<i32: 0, 0, 0>, step = 1 : index, upperBoundMap = affine_map<() -> (2)>}> ({
+    "affine.yield"() : () -> ()
+  }) : () -> ()
+  return
+}


        

From llvmlistbot at llvm.org  Fri Jul 24 06:22:45 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:22:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add affine.for verifier and
 move arguments check earlier (PR #206685)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206685@github.com>
Message-ID: <6a636725.eb165e5c.2e0c9a.d8b0@mx.google.com>


https://github.com/ftynse closed https://github.com/llvm/llvm-project/pull/206685

From llvmlistbot at llvm.org  Fri Jul 24 06:25:45 2026
From: llvmlistbot at llvm.org (Oleksandr Alex Zinenko)
Date: Fri, 24 Jul 2026 06:25:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LowerToLLVM] Pass DataLayoutAnalysis
 to LowerToLLVMOptions in `ArithToLLVM`, `ControlFlowToLLVM`,
 and `VectorToLLVM` passes (PR #206380)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206380@github.com>
Message-ID: <6a6367d9.e55dc341.dbc09.2bf8@mx.google.com>


================

----------------
ftynse wrote:

Can these tests be folded into existing files? It's not like they use different RUN lines

https://github.com/llvm/llvm-project/pull/206380

From llvmlistbot at llvm.org  Fri Jul 24 06:28:59 2026
From: llvmlistbot at llvm.org (Vassil Vassilev)
Date: Fri, 24 Jul 2026 06:28:59 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [polly] [llvm] Forward declare
 VirtualFileSystem (PR #211761)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211761@github.com>
Message-ID: <6a63689b.b4a331f0.2642a.77d8@mx.google.com>


vgvassilev wrote:

> > @compnerd, @Steelskin: This uncovers an annoying limitation where repeating forward declarations will need to be annotated and there is no easy way how to centralize them in a single place because of the different usage patterns. If we merge this as a way to handle repetitive patterns we will grow a jungle of fwd.h files -- is that optimal?
> 
> @vgvassilev It's not so much the forward declaration that is moved to its header, it's the addition of the template instantiation for `IntrusiveRefCntPtrInfo<::llvm::vfs::FileSystem>`. Note that it is not required to use this header everywhere, I merely thought it was cleaner to do so. I can revert probably 90% of these headers if you'd prefer.
> 
> Note that this is what the `IntrusiveRefCntPtr` documentation recommends:
> 
> https://github.com/llvm/llvm-project/blob/697f0f3b20d371d9e417891d9cdfb6a3613d7828/llvm/include/llvm/ADT/IntrusiveRefCntPtr.h#L157

I do not have a strong opinion -- just wanted to raise a concern about creating potential precedent.

https://github.com/llvm/llvm-project/pull/211761

From llvmlistbot at llvm.org  Fri Jul 24 06:37:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 06:37:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 1ada304 - [mlir][OpenACC] Keep ThreadY active
 for inner-combine-fed worker reductions (#211696)
Message-ID: <6a636aa4.79dd0093.f92c.d026@mx.google.com>


Author: Matsu
Date: 2026-07-24T06:37:35-07:00
New Revision: 1ada3044291aa3215b9deb2aeec3e952c13c2813

URL: https://github.com/llvm/llvm-project/commit/1ada3044291aa3215b9deb2aeec3e952c13c2813
DIFF: https://github.com/llvm/llvm-project/commit/1ada3044291aa3215b9deb2aeec3e952c13c2813.diff

LOG: [mlir][OpenACC] Keep ThreadY active for inner-combine-fed worker reductions (#211696)

Example:
```fortran
res = 0
!$cuf kernel do(2) <<< *, (32,8) >>> reduce(+:res)
do j2 = 1, n2
  do j1 = 1, n1
    res = res + a(j1, j2)
  end do
end do
```
In this code the reduction accumulator is per-(block_y, thread_y): each
worker row's shared slot is filled by an inner block-scoped combine, so
the rows hold distinct partials. The final combine into the result was
classified as not "worker-private" (block_y+thread_y into a global
dest), so it fell back to the ThreadY row-zero path and dropped every
worker but row 0 — a 2D SUM returned -1 instead of -4.
Fix: keep ThreadY active for a block_y+thread_y accumulator that is fed
by an inner block-scoped combine (distinct per-worker partials). A plain
worker accumulate still lowers to a worker-wide all_reduce that
broadcasts the total, so it correctly stays row-zero and is not
multiplied by the worker count.

Added: 
    

Modified: 
    mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
index 3bb78cd834b3e..d50d346ed9204 100644
--- a/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
+++ b/mlir/lib/Dialect/OpenACC/Transforms/ACCCGToGPU.cpp
@@ -1276,6 +1276,30 @@ static bool hasUnsafeEffectsWhenBroadening(Operation *op) {
   return !op->hasTrait<OpTrait::HasRecursiveMemoryEffects>();
 }
 
+/// True when \p accumulator is the destination of a separate block-scoped
+/// combine, so it holds a distinct per-worker partial rather than a broadcast.
+static bool isFedByInnerBlockCombine(acc::PrivateLocalOp accumulator,
+                                     Operation *selfCombine) {
+  if (!accumulator)
+    return false;
+  for (Operation *user : accumulator.getResult().getUsers()) {
+    if (user == selfCombine)
+      continue;
+    auto combineOp = dyn_cast<acc::ReductionCombineOp>(user);
+    if (!combineOp ||
+        unwrapMemRefConversion(combineOp.getDestMemref()).getDefiningOp() !=
+            accumulator.getOperation())
+      continue;
+    SmallVector<mlir::acc::GPUParallelDimAttr> parDims =
+        getReductionCombineParDims(combineOp);
+    if (llvm::any_of(parDims, [](mlir::acc::GPUParallelDimAttr d) {
+          return d.isAnyBlock();
+        }))
+      return true;
+  }
+  return false;
+}
+
 /// Records whether \p combineOp requires ThreadY to remain active.
 static void classifyThreadYCombine(ThreadYBroadeningInfo &info,
                                    Operation *combineOp, Value src, Value dest,
@@ -1299,6 +1323,16 @@ static void classifyThreadYCombine(ThreadYBroadeningInfo &info,
     return;
   }
 
+  // A block_y+thread_y accumulator fed by an inner block-scoped combine holds
+  // a distinct partial per worker row, so its combine runs on every row; a
+  // plain worker accumulate broadcasts via all_reduce and stays row-zero.
+  if (hasThreadY && hasBlock &&
+      isThreadYPrivate(srcPrivate, /*allowBlock=*/true, computeRegion) &&
+      isFedByInnerBlockCombine(srcPrivate, combineOp)) {
+    info.hasActiveWorkerCombine = true;
+    return;
+  }
+
   info.hasExplicitInactiveCombine = true;
   if (!info.diagnosticOp)
     info.diagnosticOp = combineOp;


        

From llvmlistbot at llvm.org  Fri Jul 24 06:37:44 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 06:37:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenACC] Keep ThreadY active for
 inner-combine-fed worker reductions (PR #211696)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211696@github.com>
Message-ID: <6a636aa8.5b9fb48b.167c99.6118@mx.google.com>


https://github.com/khaki3 closed https://github.com/llvm/llvm-project/pull/211696

From llvmlistbot at llvm.org  Fri Jul 24 06:58:11 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Fri, 24 Jul 2026 06:58:11 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a636f73.8f4496e3.2cf00e.e789@mx.google.com>


https://github.com/rajatbajpai updated https://github.com/llvm/llvm-project/pull/211577

>From f46648201848a3e1887af7fc3635a096284f736e Mon Sep 17 00:00:00 2001
From: Rajat Bajpai <rbajpai at nvidia.com>
Date: Thu, 23 Jul 2026 10:40:42 +0000
Subject: [PATCH 1/2] [NVPTX] Add Rubin extensions to tcgen05.commit

The Rubin architecture extends `tcgen05.commit` operations with two
additional features: support for 32-bit CTA multicast masks and the
ability to track completion of Matrix A reads from shared memory for
all prior `tcgen05.mma` operations.

This change adds support for these features to the `tcgen05.commit`
intrinsics.
---
 llvm/docs/NVPTXUsage.md                       |  16 +-
 llvm/include/llvm/IR/IntrinsicsNVVM.td        |  40 +-
 llvm/lib/Target/NVPTX/NVPTXIntrinsics.td      |  41 +-
 llvm/lib/Target/NVPTX/NVPTXSubtarget.h        |   6 +
 .../Assembler/auto_upgrade_nvvm_intrinsics.ll |  18 +
 .../CodeGen/NVPTX/tcgen05-commit-rubin.ll     | 438 ++++++++++++++++++
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td   |   2 +-
 .../Target/LLVMIR/nvvm/tcgen05-commit.mlir    |   8 +-
 8 files changed, 532 insertions(+), 37 deletions(-)
 create mode 100644 llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll

diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index 923a2b10f6be9..edd3352079df5 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -2634,8 +2634,17 @@ For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parall
 ```llvm
 declare void @llvm.nvvm.tcgen05.commit.{cg1,cg2}(ptr %mbar)
 declare void @llvm.nvvm.tcgen05.commit.shared.{cg1,cg2}(ptr addrspace(3) %mbar)
-declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}(ptr %mbar, i16 %mc)
-declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}(ptr addrspace(3) %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}.i16(ptr %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}.i16(ptr addrspace(3) %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}.i32(ptr %mbar, i32 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}.i32(ptr addrspace(3) %mbar, i32 %mc)
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.{cg1,cg2}(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.{cg1,cg2}(ptr addrspace(3) %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.{cg1,cg2}.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.{cg1,cg2}.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.{cg1,cg2}.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.{cg1,cg2}.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
 ```
 
 ##### Overview:
@@ -2647,7 +2656,8 @@ object (`%mbar`) track the completion of all prior asynchronous tcgen05
 operations. The `.mc` variants allow signaling on the mbarrier objects of
 multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
 variants generate `cta_group::1` and `cta_group::2` flavors of the
-instruction respectively.
+instruction, respectively. The `smem.a.read` variants allow tracking the completion
+of reads of Matrix A from shared memory for all prior `tcgen05.mma` operations.
 
 For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen-async-sync-operations-commit).
 
diff --git a/llvm/include/llvm/IR/IntrinsicsNVVM.td b/llvm/include/llvm/IR/IntrinsicsNVVM.td
index 40f3c40a76bc6..26be050d4b22a 100644
--- a/llvm/include/llvm/IR/IntrinsicsNVVM.td
+++ b/llvm/include/llvm/IR/IntrinsicsNVVM.td
@@ -3085,25 +3085,27 @@ foreach cta_group = ["cg1", "cg2"] in {
   def int_nvvm_tcgen05_relinq_alloc_permit_ # cta_group : Intrinsic<[], [],
     [IntrConvergent, IntrInaccessibleMemOnly]>;
 
-  def int_nvvm_tcgen05_commit_ # cta_group : Intrinsic<[],
-    [llvm_ptr_ty],        // mbar_ptr
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_shared_ # cta_group : Intrinsic<[],
-    [llvm_shared_ptr_ty], // mbar_ptr
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_mc_ # cta_group : Intrinsic<[],
-    [llvm_ptr_ty, llvm_i16_ty], // mbar_ptr, cta_mask
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
-
-  def int_nvvm_tcgen05_commit_mc_shared_ # cta_group : Intrinsic<[],
-    [llvm_shared_ptr_ty, llvm_i16_ty], // mbar_ptr, cta_mask
-    [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
-     NoCapture<ArgIndex<0>>]>;
+  foreach smem_a_read = ["", "smem_a_read_"] in {
+    def int_nvvm_tcgen05_commit_ # smem_a_read # cta_group : Intrinsic<[],
+      [llvm_ptr_ty],        // mbar_ptr
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # shared_ # cta_group : Intrinsic<[],
+      [llvm_shared_ptr_ty], // mbar_ptr
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # mc_ # cta_group : Intrinsic<[],
+      [llvm_ptr_ty, llvm_anyint_ty], // mbar_ptr, cta_mask[16 or 32 bits]
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+
+    def int_nvvm_tcgen05_commit_ # smem_a_read # mc_shared_ # cta_group : Intrinsic<[],
+      [llvm_shared_ptr_ty, llvm_anyint_ty], // mbar_ptr, cta_mask[16 or 32 bits]
+      [IntrConvergent, IntrInaccessibleMemOrArgMemOnly,
+       NoCapture<ArgIndex<0>>]>;
+  }
 
   def int_nvvm_tcgen05_shift_down_ # cta_group : Intrinsic<[],
     [llvm_tmem_ptr_ty],   // tmem_addr
diff --git a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
index 5230e01261fd6..ea2147e720da4 100644
--- a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
+++ b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
@@ -5711,25 +5711,41 @@ def tcgen05_wait_ld: NullaryInst<"tcgen05.wait::ld.sync.aligned", int_nvvm_tcgen
 def tcgen05_wait_st: NullaryInst<"tcgen05.wait::st.sync.aligned", int_nvvm_tcgen05_wait_st>;
 
 multiclass TCGEN05_COMMIT_INTR<string AS, string num> {
-  defvar prefix = "tcgen05.commit.cta_group::" # num #".mbarrier::arrive::one.shared::cluster";
+  defvar prefix = "tcgen05.commit.cta_group::" # num #
+                  ".mbarrier::arrive::one.shared::cluster";
+  defvar prefix_smem_a_read = prefix # ".sync_restrict::shared::read::mma::a";
 
+  defvar intr_prefix = "int_nvvm_tcgen05_commit";
   defvar intr_suffix = !if(!eq(AS, "shared"), "_shared", "") # "_cg" # num;
-  defvar Intr = !cast<Intrinsic>("int_nvvm_tcgen05_commit" # intr_suffix);
-  defvar IntrMC = !cast<Intrinsic>("int_nvvm_tcgen05_commit_mc" # intr_suffix);
+  defvar Intr = !cast<Intrinsic>(intr_prefix # intr_suffix);
+  defvar IntrMC = !cast<Intrinsic>(intr_prefix # "_mc" # intr_suffix);
+  defvar IntrSmemARead =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read" # intr_suffix);
+  defvar IntrSmemAReadMC =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read_mc" # intr_suffix);
 
   def "" : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
              prefix # ".b64",
              [(Intr addr:$mbar)]>;
   def _MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
-                   prefix # ".multicast::cluster.b64",
-                   [(IntrMC addr:$mbar, B16:$mc)]>;
+              prefix # ".multicast::cluster.b64",
+              [(IntrMC addr:$mbar, i16:$mc)]>;
+  let Predicates = [callSubtarget<"hasTcgen05RubinFamilySupport">] in {
+    def _MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                    prefix # ".multicast::cluster::32b.b64",
+                    [(IntrMC addr:$mbar, i32:$mc)]>;
+    def _SMRA : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
+                  prefix_smem_a_read # ".b64",
+                  [(IntrSmemARead addr:$mbar)]>;
+    def _SMRA_MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
+                    prefix_smem_a_read # ".multicast::cluster.b64",
+                    [(IntrSmemAReadMC addr:$mbar, i16:$mc)]>;
+    def _SMRA_MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                        prefix_smem_a_read # ".multicast::cluster::32b.b64",
+                        [(IntrSmemAReadMC addr:$mbar, i32:$mc)]>;
+  }
 }
 
-defm TCGEN05_COMMIT_CG1 : TCGEN05_COMMIT_INTR<"", "1">;
-defm TCGEN05_COMMIT_CG2 : TCGEN05_COMMIT_INTR<"", "2">;
-defm TCGEN05_COMMIT_S64_CG1 : TCGEN05_COMMIT_INTR<"shared", "1">;
-defm TCGEN05_COMMIT_S64_CG2 : TCGEN05_COMMIT_INTR<"shared", "2">;
-
 multiclass TCGEN05_CP_INTR<string shape, string src_fmt, string mc = ""> {
   defvar dst_fmt = !if(!eq(src_fmt, ""), "", ".b8x16");
   defvar fmt_asm = StrJoin<".", [dst_fmt, src_fmt]>.ret;
@@ -5762,6 +5778,11 @@ foreach src_fmt = ["", "b6x16_p32", "b4x16_p64"] in {
 }
 } // Predicates
 
+defm TCGEN05_COMMIT_CG1 : TCGEN05_COMMIT_INTR<"", "1">;
+defm TCGEN05_COMMIT_CG2 : TCGEN05_COMMIT_INTR<"", "2">;
+defm TCGEN05_COMMIT_S64_CG1 : TCGEN05_COMMIT_INTR<"shared", "1">;
+defm TCGEN05_COMMIT_S64_CG2 : TCGEN05_COMMIT_INTR<"shared", "2">;
+
 let Predicates = [callSubtarget<"hasTcgen05ShiftSupport">] in {
 multiclass TCGEN05_SHIFT_INTR<string num, Intrinsic Intr> {
   def "" : BasicNVPTXInst<(outs),
diff --git a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
index 1df5d326f63a6..9561e00c84023 100644
--- a/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
+++ b/llvm/lib/Target/NVPTX/NVPTXSubtarget.h
@@ -140,6 +140,12 @@ class NVPTXSubtarget : public NVPTXGenSubtargetInfo {
            hasPTXWithAccelSMs(86, {100, 101});
   }
 
+  // Checks Rubin family extensions support.
+  //  - tcgen05.commit
+  bool hasTcgen05RubinFamilySupport() const {
+    return hasPTXWithFamilySMs(94, {107});
+  }
+
   // Checks tcgen05.shift instruction support.
   bool hasTcgen05ShiftSupport() const {
     // sm_101 renamed to sm_110 in PTX 9.0
diff --git a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
index c59c9bda203b8..d3b815a1ef708 100644
--- a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
+++ b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
@@ -115,6 +115,11 @@ declare ptr addrspace(3) @llvm.nvvm.mapa.shared.cluster(ptr addrspace(3), i32)
 declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cluster(ptr addrspace(3), ptr addrspace(3), ptr addrspace(1), i32, i16, i64, i1, i1)
 declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.cluster(ptr addrspace(3), ptr addrspace(3), ptr addrspace(3), i32)
 
+declare void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr, i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr, i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3), i16)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3), i16)
+
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.2d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i32 %d1, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
 declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.3d(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tm, i32 %d0, i32 %d1, i32 %d2, i16 %mc, i64 %ch, i1 %f1, i1 %f2);
@@ -456,6 +461,19 @@ define void @nvvm_cp_async_bulk_intrinsics(ptr addrspace(3) %dst, ptr addrspace(
   ret void
 }
 
+; CHECK-LABEL: @nvvm_tcgen05_commit_mc_intrinsics
+define void @nvvm_tcgen05_commit_mc_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg1.i16(ptr %bar, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg2.i16(ptr %bar, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i16(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i16(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr %bar, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr %bar, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3) %bar_shared, i16 %cta_mask)
+  ret void
+}
+
 ; CHECK-LABEL: @nvvm_cp_async_bulk_tensor_g2s_im2col
 define void @nvvm_cp_async_bulk_tensor_g2s_im2col(ptr addrspace(3) %d, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, i16 %mc, i64 %ch) {
 ; CHECK: call void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.3d(ptr addrspace(7) %1, ptr addrspace(3) %bar, ptr %tmap, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i16 0, i64 0, i1 false, i1 false, i32 0)
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll b/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll
new file mode 100644
index 0000000000000..0b54e735ef3a9
--- /dev/null
+++ b/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll
@@ -0,0 +1,438 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_107f  | FileCheck --check-prefixes=CHECK_PTX64 %s
+; RUN: llc < %s -march=nvptx64 -mcpu=sm_107f  --nvptx-short-ptr | FileCheck --check-prefixes=CHECK_PTX64_SHARED32 %s
+
+; RUN: %if ptxas-sm_107f && ptxas-isa-9.4 %{ llc < %s -march=nvptx64 -mcpu=sm_107f  | %ptxas-verify -arch=sm_107f %}
+; RUN: %if ptxas-sm_107f && ptxas-isa-9.4 %{ llc < %s -march=nvptx64 -mcpu=sm_107f  --nvptx-short-ptr | %ptxas-verify -arch=sm_107f %}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.cg1(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.cg2(ptr %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg1(ptr addrspace(3) %bar_addr)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg2(ptr addrspace(3) %bar_addr)
+
+define void @test_tcgen05_commit_smem_a_read_cg1(ptr %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg1_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.cg1(ptr %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_cg2(ptr %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg2_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.cg2(ptr %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_shared_cg1(ptr addrspace(3) %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_shared_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_shared_cg1_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_shared_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_shared_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%r1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg1(ptr addrspace(3) %bar_addr)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_shared_cg2(ptr addrspace(3) %bar_addr) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_shared_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_shared_cg2_param_0];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%rd1];
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_shared_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_shared_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.b64 [%r1];
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.shared.cg2(ptr addrspace(3) %bar_addr)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i16(ptr %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg1(ptr %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg1_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i16(ptr %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg2(ptr %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_cg2_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i16(ptr %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg1(ptr addrspace(3) %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%r1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg2(ptr addrspace(3) %bar_addr, i16 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%rd1], %rs1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b16 %rs<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b16 %rs1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster.b64 [%r1], %rs1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i16(ptr addrspace(3) %bar_addr, i16 %cta_mask)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg1_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_cg2_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_smem_a_read_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_smem_a_read_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.sync_restrict::shared::read::mma::a.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.smem.a.read.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+declare void @llvm.nvvm.tcgen05.commit.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+define void @test_tcgen05_commit_mc_cg1_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.cg1.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_cg2_i32(ptr %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.cg2.i32(ptr %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_shared_cg1_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_shared_cg1_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_shared_cg1_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg1_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_mc_shared_cg1_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
+
+define void @test_tcgen05_commit_mc_shared_cg2_i32(ptr addrspace(3) %bar_addr, i32 %cta_mask) {
+; CHECK_PTX64-LABEL: test_tcgen05_commit_mc_shared_cg2_i32(
+; CHECK_PTX64:       {
+; CHECK_PTX64-NEXT:    .reg .b32 %r<2>;
+; CHECK_PTX64-NEXT:    .reg .b64 %rd<2>;
+; CHECK_PTX64-EMPTY:
+; CHECK_PTX64-NEXT:  // %bb.0:
+; CHECK_PTX64-NEXT:    ld.param.b64 %rd1, [test_tcgen05_commit_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%rd1], %r1;
+; CHECK_PTX64-NEXT:    ret;
+;
+; CHECK_PTX64_SHARED32-LABEL: test_tcgen05_commit_mc_shared_cg2_i32(
+; CHECK_PTX64_SHARED32:       {
+; CHECK_PTX64_SHARED32-NEXT:    .reg .b32 %r<3>;
+; CHECK_PTX64_SHARED32-EMPTY:
+; CHECK_PTX64_SHARED32-NEXT:  // %bb.0:
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r1, [test_tcgen05_commit_mc_shared_cg2_i32_param_0];
+; CHECK_PTX64_SHARED32-NEXT:    ld.param.b32 %r2, [test_tcgen05_commit_mc_shared_cg2_i32_param_1];
+; CHECK_PTX64_SHARED32-NEXT:    tcgen05.commit.cta_group::2.mbarrier::arrive::one.shared::cluster.multicast::cluster::32b.b64 [%r1], %r2;
+; CHECK_PTX64_SHARED32-NEXT:    ret;
+  call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i32(ptr addrspace(3) %bar_addr, i32 %cta_mask)
+
+  ret void
+}
diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..2bf21db6b6e6f 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -5331,7 +5331,7 @@ def NVVM_Tcgen05CommitOp : NVVM_Op<"tcgen05.commit", [NVVMRequiresSMf<[100, 101,
     llvm::SmallVector<llvm::Value *> args;
     auto id = NVVM::Tcgen05CommitOp::getIntrinsicIDAndArgs(
       *op, moduleTranslation, args);
-    createIntrinsicCall(builder, id, args);
+    createIntrinsicCall(builder, id, builder.getVoidTy(), args);
   }];
 }
 
diff --git a/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir b/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
index 60475bf64ae7a..6997f8a98b2fc 100644
--- a/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
+++ b/mlir/test/Target/LLVMIR/nvvm/tcgen05-commit.mlir
@@ -8,10 +8,10 @@ llvm.func @llvm_nvvm_tcgen05_commit_generic(%barrier : !llvm.ptr, %cta_mask : i1
   // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.cg2(ptr %{{.*}})
   nvvm.tcgen05.commit %barrier {group = #nvvm.cta_group<cta_2>} : !llvm.ptr
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg1(ptr %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg1.i16(ptr %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask : !llvm.ptr, i16
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg2(ptr %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.cg2.i16(ptr %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask {group = #nvvm.cta_group<cta_2>} : !llvm.ptr, i16
   llvm.return
 }
@@ -24,10 +24,10 @@ llvm.func @llvm_nvvm_tcgen05_commit_shared(%barrier : !llvm.ptr<3>, %cta_mask :
   // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.shared.cg2(ptr addrspace(3) %{{.*}})
   nvvm.tcgen05.commit %barrier {group = #nvvm.cta_group<cta_2>} : !llvm.ptr<3>
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i16(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask : !llvm.ptr<3>, i16
 
-  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
+  // CHECK-LLVM: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg2.i16(ptr addrspace(3) %{{.*}}, i16 %{{.*}})
   nvvm.tcgen05.commit %barrier, multicast_mask = %cta_mask {group = #nvvm.cta_group<cta_2>} : !llvm.ptr<3>, i16
   llvm.return
 }

>From 7798fe74c99ae4c9853b063e49ef3b8818be3f77 Mon Sep 17 00:00:00 2001
From: Rajat Bajpai <rbajpai at nvidia.com>
Date: Fri, 24 Jul 2026 13:47:42 +0000
Subject: [PATCH 2/2] Addressed review comments

---
 llvm/docs/NVPTXUsage.md                                     | 4 ++--
 llvm/lib/Target/NVPTX/NVPTXIntrinsics.td                    | 6 +++---
 llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll         | 4 ++--
 .../{tcgen05-commit-rubin.ll => tcgen05-commit-sm107.ll}    | 0
 4 files changed, 7 insertions(+), 7 deletions(-)
 rename llvm/test/CodeGen/NVPTX/{tcgen05-commit-rubin.ll => tcgen05-commit-sm107.ll} (100%)

diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index edd3352079df5..e7f0787e31f2c 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -2656,8 +2656,8 @@ object (`%mbar`) track the completion of all prior asynchronous tcgen05
 operations. The `.mc` variants allow signaling on the mbarrier objects of
 multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
 variants generate `cta_group::1` and `cta_group::2` flavors of the
-instruction, respectively. The `smem.a.read` variants allow tracking the completion
-of reads of Matrix A from shared memory for all prior `tcgen05.mma` operations.
+instruction, respectively. The `smem.a.read` variants track the completion
+of reads of A-matrix from shared memory for all prior `tcgen05.mma` operations.
 
 For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen-async-sync-operations-commit).
 
diff --git a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
index ea2147e720da4..05f9a5592cde4 100644
--- a/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
+++ b/llvm/lib/Target/NVPTX/NVPTXIntrinsics.td
@@ -5734,13 +5734,13 @@ multiclass TCGEN05_COMMIT_INTR<string AS, string num> {
     def _MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
                     prefix # ".multicast::cluster::32b.b64",
                     [(IntrMC addr:$mbar, i32:$mc)]>;
-    def _SMRA : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
+    def _SMEM_A_READ : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
                   prefix_smem_a_read # ".b64",
                   [(IntrSmemARead addr:$mbar)]>;
-    def _SMRA_MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
+    def _SMEM_A_READ_MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
                     prefix_smem_a_read # ".multicast::cluster.b64",
                     [(IntrSmemAReadMC addr:$mbar, i16:$mc)]>;
-    def _SMRA_MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+    def _SMEM_A_READ_MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
                         prefix_smem_a_read # ".multicast::cluster::32b.b64",
                         [(IntrSmemAReadMC addr:$mbar, i32:$mc)]>;
   }
diff --git a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
index d3b815a1ef708..5ffb1be055450 100644
--- a/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
+++ b/llvm/test/Assembler/auto_upgrade_nvvm_intrinsics.ll
@@ -461,8 +461,8 @@ define void @nvvm_cp_async_bulk_intrinsics(ptr addrspace(3) %dst, ptr addrspace(
   ret void
 }
 
-; CHECK-LABEL: @nvvm_tcgen05_commit_mc_intrinsics
-define void @nvvm_tcgen05_commit_mc_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
+; CHECK-LABEL: @nvvm_tcgen05_commit_mc_16b_intrinsics
+define void @nvvm_tcgen05_commit_mc_16b_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
 ; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg1.i16(ptr %bar, i16 %cta_mask)
 ; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.cg2.i16(ptr %bar, i16 %cta_mask)
 ; CHECK: call void @llvm.nvvm.tcgen05.commit.mc.shared.cg1.i16(ptr addrspace(3) %bar_shared, i16 %cta_mask)
diff --git a/llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll b/llvm/test/CodeGen/NVPTX/tcgen05-commit-sm107.ll
similarity index 100%
rename from llvm/test/CodeGen/NVPTX/tcgen05-commit-rubin.ll
rename to llvm/test/CodeGen/NVPTX/tcgen05-commit-sm107.ll


From llvmlistbot at llvm.org  Fri Jul 24 06:58:25 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Fri, 24 Jul 2026 06:58:25 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a636f81.15fbc348.21862a.921c@mx.google.com>


================
@@ -2647,7 +2656,8 @@ object (`%mbar`) track the completion of all prior asynchronous tcgen05
 operations. The `.mc` variants allow signaling on the mbarrier objects of
 multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
 variants generate `cta_group::1` and `cta_group::2` flavors of the
-instruction respectively.
+instruction, respectively. The `smem.a.read` variants allow tracking the completion
----------------
rajatbajpai wrote:

Done

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 06:58:30 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Fri, 24 Jul 2026 06:58:30 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a636f86.f916f9ef.2402a9.e503@mx.google.com>


================
@@ -5711,25 +5711,41 @@ def tcgen05_wait_ld: NullaryInst<"tcgen05.wait::ld.sync.aligned", int_nvvm_tcgen
 def tcgen05_wait_st: NullaryInst<"tcgen05.wait::st.sync.aligned", int_nvvm_tcgen05_wait_st>;
 
 multiclass TCGEN05_COMMIT_INTR<string AS, string num> {
-  defvar prefix = "tcgen05.commit.cta_group::" # num #".mbarrier::arrive::one.shared::cluster";
+  defvar prefix = "tcgen05.commit.cta_group::" # num #
+                  ".mbarrier::arrive::one.shared::cluster";
+  defvar prefix_smem_a_read = prefix # ".sync_restrict::shared::read::mma::a";
 
+  defvar intr_prefix = "int_nvvm_tcgen05_commit";
   defvar intr_suffix = !if(!eq(AS, "shared"), "_shared", "") # "_cg" # num;
-  defvar Intr = !cast<Intrinsic>("int_nvvm_tcgen05_commit" # intr_suffix);
-  defvar IntrMC = !cast<Intrinsic>("int_nvvm_tcgen05_commit_mc" # intr_suffix);
+  defvar Intr = !cast<Intrinsic>(intr_prefix # intr_suffix);
+  defvar IntrMC = !cast<Intrinsic>(intr_prefix # "_mc" # intr_suffix);
+  defvar IntrSmemARead =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read" # intr_suffix);
+  defvar IntrSmemAReadMC =
+      !cast<Intrinsic>(intr_prefix # "_smem_a_read_mc" # intr_suffix);
 
   def "" : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
              prefix # ".b64",
              [(Intr addr:$mbar)]>;
   def _MC : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B16:$mc),
-                   prefix # ".multicast::cluster.b64",
-                   [(IntrMC addr:$mbar, B16:$mc)]>;
+              prefix # ".multicast::cluster.b64",
+              [(IntrMC addr:$mbar, i16:$mc)]>;
+  let Predicates = [callSubtarget<"hasTcgen05RubinFamilySupport">] in {
+    def _MC_32B : BasicNVPTXInst<(outs), (ins ADDR:$mbar, B32:$mc),
+                    prefix # ".multicast::cluster::32b.b64",
+                    [(IntrMC addr:$mbar, i32:$mc)]>;
+    def _SMRA : BasicNVPTXInst<(outs), (ins ADDR:$mbar),
----------------
rajatbajpai wrote:

Sure, done.

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 06:58:34 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Fri, 24 Jul 2026 06:58:34 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a636f8a.8af9f03d.164e93.3241@mx.google.com>


================
@@ -456,6 +461,19 @@ define void @nvvm_cp_async_bulk_intrinsics(ptr addrspace(3) %dst, ptr addrspace(
   ret void
 }
 
+; CHECK-LABEL: @nvvm_tcgen05_commit_mc_intrinsics
+define void @nvvm_tcgen05_commit_mc_intrinsics(ptr %bar, ptr addrspace(3) %bar_shared, i16 %cta_mask) {
----------------
rajatbajpai wrote:

Done

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 06:58:39 2026
From: llvmlistbot at llvm.org (Rajat Bajpai)
Date: Fri, 24 Jul 2026 06:58:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a636f8f.87c5160c.2fe5e9.d82f@mx.google.com>


================
@@ -0,0 +1,438 @@
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 6
----------------
rajatbajpai wrote:

sure, done.

I'm thinking may be we should have separate directory for Rubin extensions.

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 07:00:25 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 07:00:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Lower
 allocator-backed storage for allocate clauses (PR #211621)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211621@github.com>
Message-ID: <6a636ff9.82d06577.3ac103.aa1d@mx.google.com>


================
@@ -3498,6 +3285,59 @@ genParallelOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   assert((!enableDelayedPrivatization || dsp) &&
          "expected valid DataSharingProcessor");
 
+  if (!clauseOps.allocateVars.empty()) {
+    llvm::DenseMap<const semantics::Symbol *, int64_t> privateSlots;
+    for (auto [index, object] : llvm::enumerate(args.priv.objects)) {
+      const semantics::Symbol *symbol = object.sym();
+      if (!symbol)
+        TODO(loc, "ALLOCATE clause private item without a semantic symbol");
+      const semantics::Symbol *ultimate = &symbol->GetUltimate();
+      if (!privateSlots.try_emplace(ultimate, index).second)
+        TODO(loc, "ALLOCATE clause item has multiple private storage slots");
+    }
+
+    llvm::DenseSet<const semantics::Symbol *> allocateSymbols;
+    for (const Clause &clause : item->clauses) {
+      if (clause.id != llvm::omp::Clause::OMPC_allocate)
+        continue;
+      const auto &allocate = std::get<clause::Allocate>(clause.u);
+      const auto &objects = std::get<ObjectList>(allocate.t);
+      for (const Object &object : objects) {
+        const semantics::Symbol *symbol = object.sym();
+        if (!symbol)
+          TODO(loc, "ALLOCATE clause item without a semantic symbol");
+        const semantics::Symbol *ultimate = &symbol->GetUltimate();
+        if (!allocateSymbols.insert(ultimate).second)
+          TODO(loc, "ALLOCATE clause item appears more than once");
+
+        auto privateSlot = privateSlots.find(ultimate);
+        if (privateSlot == privateSlots.end())
+          TODO(loc, "ALLOCATE clause item has no private storage slot");
+
+        auto type = evaluate::DynamicType::From(*ultimate);
+        bool supportedDataSharing =
+            symbol->test(semantics::Symbol::Flag::OmpPrivate) ||
+            symbol->test(semantics::Symbol::Flag::OmpFirstPrivate);
+        bool supportedType =
+            ultimate->Rank() == 0 &&
+            !semantics::IsAllocatableOrPointer(*ultimate) && type &&
+            type->category() != common::TypeCategory::Derived &&
+            !type->RequiresDescriptor() &&
+            !type->HasDeferredOrAssumedTypeParameter();
+        if (!supportedDataSharing || !supportedType)
+          TODO(loc,
+               "ALLOCATE clause currently supports only fixed-size intrinsic "
+               "scalar PRIVATE or FIRSTPRIVATE items");
----------------
tblah wrote:

This looks like a correct usage of TODO

https://github.com/llvm/llvm-project/pull/211621

From llvmlistbot at llvm.org  Fri Jul 24 07:00:25 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 07:00:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Lower
 allocator-backed storage for allocate clauses (PR #211621)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211621@github.com>
Message-ID: <6a636ff9.d170d360.242fd6.9585@mx.google.com>


================
@@ -110,16 +110,14 @@ genAllocateClause(lower::AbstractConverter &converter,
   if (std::get<std::optional<Allocate::AlignModifier>>(clause.t))
     TODO(currentLocation, "OmpAllocateClause ALIGN modifier");
 
-  // Check if allocate clause has allocator specified. If so, add it
-  // to list of allocators, otherwise, add default allocator to
-  // list of allocators.
+  // Use a null handle to select the binding task's default allocator.
   using ComplexModifier = Allocate::AllocatorComplexModifier;
   if (auto &mod = std::get<std::optional<ComplexModifier>>(clause.t)) {
     mlir::Value operand = fir::getBase(converter.genExprValue(mod->v, stmtCtx));
     allocatorOperands.append(objects.size(), operand);
   } else {
     mlir::Value operand = firOpBuilder.createIntegerConstant(
-        currentLocation, firOpBuilder.getI32Type(), 1);
+        currentLocation, firOpBuilder.getI32Type(), 0);
----------------
tblah wrote:

nit: this would be easier to read with some sort of constant defined, probably in the MLIR dialect.

https://github.com/llvm/llvm-project/pull/211621

From llvmlistbot at llvm.org  Fri Jul 24 07:00:25 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 07:00:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Lower
 allocator-backed storage for allocate clauses (PR #211621)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211621@github.com>
Message-ID: <6a636ff9.63d39acb.28e77a.e1ce@mx.google.com>


================
@@ -3498,6 +3285,59 @@ genParallelOp(lower::AbstractConverter &converter, lower::SymMap &symTable,
   assert((!enableDelayedPrivatization || dsp) &&
          "expected valid DataSharingProcessor");
 
+  if (!clauseOps.allocateVars.empty()) {
+    llvm::DenseMap<const semantics::Symbol *, int64_t> privateSlots;
+    for (auto [index, object] : llvm::enumerate(args.priv.objects)) {
+      const semantics::Symbol *symbol = object.sym();
+      if (!symbol)
+        TODO(loc, "ALLOCATE clause private item without a semantic symbol");
----------------
tblah wrote:

Is this something to be implemented later or a compiler bug? If it is just a bug then use a normal fatal error.

And see others further down. You can use `fir::emitFatalError` or assertions for things that are bugs rather than unimplemented features.

In general, the real TODOs are the ones in the test you added in test/Lower/OpenMP/Todo

https://github.com/llvm/llvm-project/pull/211621

From llvmlistbot at llvm.org  Fri Jul 24 07:00:25 2026
From: llvmlistbot at llvm.org (Tom Eccles)
Date: Fri, 24 Jul 2026 07:00:25 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [flang][OpenMP] Lower
 allocator-backed storage for allocate clauses (PR #211621)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211621@github.com>
Message-ID: <6a636ff9.ece797f7.271b0f.c2f9@mx.google.com>


================
@@ -0,0 +1,171 @@
+! REQUIRES: openmp_runtime
+! RUN: %flang_fc1 -emit-hlfir %openmp_flags -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s --check-prefix=HLFIR
+! RUN: %flang_fc1 -emit-llvm %openmp_flags -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s --check-prefix=LLVM
----------------
tblah wrote:

This is an integration test. See https://github.com/llvm/llvm-project/blob/main/flang/test/Integration/README.md. They are more difficult to maintain, and I am not sure if it is needed here because you already added the correct tests in mlir/

https://github.com/llvm/llvm-project/pull/211621

From llvmlistbot at llvm.org  Fri Jul 24 07:03:12 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 07:03:12 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [mlir][vector] Prevent crashing on
 bitcast breakdown (PR #206683)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206683@github.com>
Message-ID: <6a6370a0.79dd0093.f92c.5adc@mx.google.com>


banach-space wrote:

> I'm not sure I understand what the semantic question or gap is. Vector elements have a packed layout within the vector. What am I missing?

You will be more familiar with the "historical" background, but I actually wasn't able to find anything that would document this. 

I read `assert`s as an indication that the underlying semantics have been left as a TODO (or, that assumptions are made in the implementation that expect no such cases ever being encountered). That an my experience with "narrow type emulation" prompted my reservations regarding this change.

Separately, `Vector` semantics are effectively dictated by lowerings to LLVM and SPIR-V - what should we do with e.g. `vector<7xi3>` and operations on such types? IMHO, with no users, we should just restrict the semantics (so that the restrications/lack-of-support are explicit) and leave this as TBD. 

https://github.com/llvm/llvm-project/pull/206683

From llvmlistbot at llvm.org  Fri Jul 24 07:11:08 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Fri, 24 Jul 2026 07:11:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Silence MLIRTargetLLVMTests failures
	pre-ROCm6.4 (PR #147610)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/147610@github.com>
Message-ID: <6a63727c.1728eaf3.12b96d.e087@mx.google.com>


https://github.com/slinder1 closed https://github.com/llvm/llvm-project/pull/147610

From llvmlistbot at llvm.org  Fri Jul 24 07:11:09 2026
From: llvmlistbot at llvm.org (Scott Linder)
Date: Fri, 24 Jul 2026 07:11:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Silence MLIRTargetLLVMTests failures
	pre-ROCm6.4 (PR #147610)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/147610@github.com>
Message-ID: <6a63727d.21a19891.ef87d.957b@mx.google.com>


slinder1 wrote:

I don't have the cycles to pursue this, but if anyone wants to pick it up feel free

https://github.com/llvm/llvm-project/pull/147610

From llvmlistbot at llvm.org  Fri Jul 24 07:18:26 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:18:26 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][DataLayout] Fix ABI alignment
 calculation for ComplexType (PR #209637)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209637@github.com>
Message-ID: <6a637432.9edb5c9d.21e140.3ae1@mx.google.com>


anoopkg6 wrote:

Gentle ping on this PR — would appreciate any feedback when you get a chance. Thanks!

https://github.com/llvm/llvm-project/pull/209637

From llvmlistbot at llvm.org  Fri Jul 24 07:36:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:36:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Vector] Reject invalid inline yields
	(PR #206218)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206218@github.com>
Message-ID: <6a637888.af6ba3ed.1989e7.1d8c@mx.google.com>


================
@@ -794,3 +794,18 @@ llvm.func @caller_cf_br_terminator(%arg0 : i64) -> i64 {
   %0 = llvm.call @callee_cf_br_terminator(%arg0) : (i64) -> i64
   llvm.return %0 : i64
 }
+
+// -----
+// Skip callees with non-LLVM function exits.
+
+llvm.func @callee_vector_yield_terminator(%arg0 : vector<8xf32>) {
+  %0 = math.atanh %arg0 : vector<8xf32>
+  vector.yield %0 : vector<8xf32>
+}
+
----------------
PragmaTwice wrote:

I don't think this is the correct way to solve the problem. I think in the original issue (likely produced by a fuzzer) the input IR is just invalid. There is no real IR that use `vector.yield` as a terminator of a `llvm.func` region.

https://github.com/llvm/llvm-project/pull/206218

From llvmlistbot at llvm.org  Fri Jul 24 07:37:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:37:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Vector] Reject invalid inline yields
	(PR #206218)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206218@github.com>
Message-ID: <6a637898.b1416da1.20b4cf.af7a@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/206218

From llvmlistbot at llvm.org  Fri Jul 24 07:38:21 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:38:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Vector] Reject invalid inline yields
	(PR #206218)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206218@github.com>
Message-ID: <6a6378dd.d681cab9.3d08c3.37c6@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/206218

From llvmlistbot at llvm.org  Fri Jul 24 07:38:27 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:38:27 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Vector] Reject invalid inline yields
	(PR #206218)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206218@github.com>
Message-ID: <6a6378e3.c5504292.1bbfc8.3b7b@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/206218

From llvmlistbot at llvm.org  Fri Jul 24 07:42:08 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 07:42:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [memref] Elevate `AllocOp`s to
	`GlobalOp`s pass (PR #211141)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211141@github.com>
Message-ID: <6a6379c0.f1c1923d.1253fb.2a54@mx.google.com>


banach-space wrote:

> MLIR has alloca_to_global. Is there a way to eventually reuse this logic?

Good catch, looks like this is re-implementing an existing transformation and hence should not be required.

https://github.com/llvm/llvm-project/pull/211141

From llvmlistbot at llvm.org  Fri Jul 24 07:43:13 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:43:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] a2d74bf - [flang][OpenMP] Implement the error
	directive (#206175)
Message-ID: <6a637a01.d1496011.1d7ce5.d2e6@mx.google.com>


Author: Caroline Newcombe
Date: 2026-07-24T10:43:05-04:00
New Revision: a2d74bfe386695ee1a17ba3b60367fb79c3bfdfe

URL: https://github.com/llvm/llvm-project/commit/a2d74bfe386695ee1a17ba3b60367fb79c3bfdfe
DIFF: https://github.com/llvm/llvm-project/commit/a2d74bfe386695ee1a17ba3b60367fb79c3bfdfe.diff

LOG: [flang][OpenMP] Implement the error directive (#206175)

## [flang][OpenMP] Implement the `error` directive

### Summary

Implements the OpenMP `error` directive (OpenMP 5.1, 2.5.4) in Flang.
The directive prints a message and, with `severity(fatal)`, stops.
Handling depends on the `at` clause:

- **`at(compilation)`** (the default): handled in semantics. Emits a
compile-time warning or error and produces no IR.
- **`at(execution)`**: lowered to a new `omp.error` op, which becomes a
call to `__kmpc_error`.

Before this patch the directive was unimplemented and hit a `not yet
implemented: OmpErrorDirective` error during lowering.

Fixes #204240.

### Details

Changes go through the compiler pipeline.

**Flang semantics**
- `Enter(OmpErrorDirective)` emits the compile-time diagnostic for the
`at(compilation)` form, using the directive's severity and the `MESSAGE`
text when present.
- `Enter(OmpClause::Message)` checks that the `MESSAGE` expression is a
scalar, default-kind `CHARACTER`, which is required to pass it to the
runtime as a `const char*`.

**Flang lowering**
- `genErrorDirective` lowers the `at(execution)` form only.
- A constant message is stored in the `message` attribute. A
non-constant message is copied to a null-terminated buffer
(`genNullTerminatedString`) and passed as the `message_expr` operand,
because `__kmpc_error` expects a NUL-terminated string.

**New `omp.error` op (MLIR OpenMP dialect)**
- Adds a `ClauseSeverity` enum (`warning = 1`, `fatal = 2`, matching the
`__kmpc_error` argument) and `ClauseSeverityAttr`.
- `omp.error` takes a required `severity` attribute and an optional
message, given either as:
  - `message` — a constant string attribute, or
  - `message_expr` — a pointer to a null-terminated string.
- A verifier rejects providing both `message` and `message_expr`.

**MLIR → LLVM IR translation**
- Translates `omp.error` to `__kmpc_error(ident_t *, i32 severity, ptr
message)`.

**OMPIRBuilder**
- Adds `OpenMPIRBuilder::createError(...)`, which builds the call, maps
`fatal`→`2` and `warning`→`1`, and passes a null pointer when there is
no message.

### Tests

- **Parser** (`error-unparse.f90`): unparse and parse-tree output.
`at(compilation) severity(fatal)` cases that would stop compilation were
moved to `at(execution)`.
- **Semantics** (`Semantics/OpenMP/error.f90`): compilation
warning/error/default-severity cases, plus the new `MESSAGE` type, rank,
and kind errors.
- **Lowering** (`Lower/OpenMP/error.f90`): constant message, no message,
empty message, runtime `message_expr`, and `at(compilation)` cases that
must not produce `omp.error`. The old `Lower/OpenMP/Todo/error.f90` test
is removed.
- **MLIR** (`ops.mlir`, `invalid.mlir`): round-trip of each `omp.error`
form and the both-messages verifier error.
- **LLVM IR** (`openmp-llvm.mlir`): `__kmpc_error` calls for warning and
fatal, constant and runtime messages, and the null-message cases.

Assisted-by: Github Copilot

Added: 
    flang/test/Lower/OpenMP/error.f90

Modified: 
    flang/include/flang/Semantics/openmp-utils.h
    flang/lib/Lower/OpenMP/OpenMP.cpp
    flang/lib/Semantics/check-omp-structure.cpp
    flang/lib/Semantics/check-omp-structure.h
    flang/lib/Semantics/openmp-utils.cpp
    flang/test/Parser/OpenMP/error-unparse.f90
    flang/test/Semantics/OpenMP/error.f90
    llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
    llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
    mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
    mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
    mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
    mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
    mlir/test/Dialect/OpenMP/invalid.mlir
    mlir/test/Dialect/OpenMP/ops.mlir
    mlir/test/Target/LLVMIR/openmp-llvm.mlir

Removed: 
    flang/test/Lower/OpenMP/Todo/error.f90


################################################################################
diff  --git a/flang/include/flang/Semantics/openmp-utils.h b/flang/include/flang/Semantics/openmp-utils.h
index 496cad0b3bd57..031926d4a7cc2 100644
--- a/flang/include/flang/Semantics/openmp-utils.h
+++ b/flang/include/flang/Semantics/openmp-utils.h
@@ -298,6 +298,21 @@ bool HasDataEnvironment(llvm::omp::Directive dir);
 
 bool IsFullUnroll(const parser::OmpDirectiveSpecification &spec);
 
+/// The AT, SEVERITY, and MESSAGE clause values of an `!$omp error` directive.
+/// `at` and `severity` default to AT(compilation)/SEVERITY(fatal) when absent;
+/// `message` is null when there is no MESSAGE clause.
+struct OmpErrorArgs {
+  parser::OmpAtClause::ActionTime at{
+      parser::OmpAtClause::ActionTime::Compilation};
+  parser::OmpSeverityClause::SevLevel severity{
+      parser::OmpSeverityClause::SevLevel::Fatal};
+  const parser::Expr *message{nullptr};
+};
+
+/// Scan the clause list of an `!$omp error` directive for its AT, SEVERITY, and
+/// MESSAGE clause values.
+OmpErrorArgs GetErrorDirectiveArgs(const parser::OmpErrorDirective &errDir);
+
 inline bool IsDoConcurrentLegal(unsigned version) {
   // DO CONCURRENT is allowed (as an alternative to a Canonical Loop Nest)
   // in OpenMP 6.0+.

diff  --git a/flang/lib/Lower/OpenMP/OpenMP.cpp b/flang/lib/Lower/OpenMP/OpenMP.cpp
index 996599212f5b9..9807023935784 100644
--- a/flang/lib/Lower/OpenMP/OpenMP.cpp
+++ b/flang/lib/Lower/OpenMP/OpenMP.cpp
@@ -34,6 +34,7 @@
 #include "flang/Lower/Support/ReductionProcessor.h"
 #include "flang/Lower/SymbolMap.h"
 #include "flang/Optimizer/Builder/BoxValue.h"
+#include "flang/Optimizer/Builder/Character.h"
 #include "flang/Optimizer/Builder/FIRBuilder.h"
 #include "flang/Optimizer/Builder/Todo.h"
 #include "flang/Optimizer/Dialect/FIROpsSupport.h"
@@ -6604,6 +6605,98 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                 queue.begin(), critName);
 }
 
+// Copy the character value `str` into fresh stack memory with an appended NUL
+// and return a pointer to its first character. Fortran characters are not
+// NUL-terminated, so the terminator is needed for C runtime entry points (such
+// as `__kmpc_error`) that expect a NUL-terminated string.
+static mlir::Value genNullTerminatedString(fir::FirOpBuilder &builder,
+                                           mlir::Location loc,
+                                           const fir::ExtendedValue &str) {
+  fir::factory::CharacterExprHelper helper(builder, loc);
+  const mlir::Value addr = fir::getBase(str);
+  const mlir::Value len = fir::getLen(str);
+  const auto charTy =
+      mlir::cast<fir::CharacterType>(fir::unwrapRefType(addr.getType()));
+  mlir::MLIRContext *ctx = builder.getContext();
+  const mlir::Type idxTy = builder.getIndexType();
+  const mlir::Value idxLen = builder.createConvert(loc, idxTy, len);
+  const mlir::Value one = builder.createIntegerConstant(loc, idxTy, 1);
+  const mlir::Value lenPlusOne =
+      mlir::arith::AddIOp::create(builder, loc, idxLen, one);
+
+  // Allocate `len + 1` characters and copy the message into them, leaving room
+  // for the terminating NUL.
+  const fir::CharBoxValue temp = helper.createCharacterTemp(charTy, lenPlusOne);
+  helper.createCopy(temp, fir::CharBoxValue{addr, len}, len);
+
+  // Address the buffer as an array of single characters so the terminating NUL
+  // can be stored at index `len`.
+  const auto singleTy = fir::CharacterType::get(
+      ctx, charTy.getFKind(), fir::CharacterType::singleton());
+  const mlir::Type singleRefTy = builder.getRefType(singleTy);
+  const mlir::Type seqRefTy = builder.getRefType(fir::SequenceType::get(
+      {fir::SequenceType::getUnknownExtent()}, singleTy));
+  const mlir::Value seq =
+      builder.createConvert(loc, seqRefTy, temp.getBuffer());
+  const mlir::Value nulAddr =
+      fir::CoordinateOp::create(builder, loc, singleRefTy, seq, idxLen);
+  const mlir::Value zero =
+      builder.createIntegerConstant(loc, builder.getI8Type(), 0);
+  const mlir::Value nul =
+      helper.createSingletonFromCode(zero, charTy.getFKind());
+  fir::StoreOp::create(builder, loc, nul, nulAddr);
+
+  // Pass a pointer to the first character of the buffer.
+  return builder.createConvert(loc, singleRefTy, temp.getBuffer());
+}
+
+// Lower an `!$omp error` directive. The `at(compilation)` form is handled
+// entirely in semantics, so only the `at(execution)` form reaches lowering,
+// where it becomes an `omp.error` operation.
+static void genErrorDirective(lower::AbstractConverter &converter,
+                              semantics::SemanticsContext &semaCtx,
+                              const parser::OmpErrorDirective &errDir) {
+  const semantics::omp::OmpErrorArgs args{
+      semantics::omp::GetErrorDirectiveArgs(errDir)};
+
+  if (args.at != parser::OmpAtClause::ActionTime::Execution ||
+      semaCtx.langOptions().OpenMPSimd)
+    return;
+
+  std::optional<std::string> message;
+  MaybeExpr messageExpr;
+  if (args.message) {
+    if (auto expr = semantics::omp::GetEvaluateExpr(*args.message)) {
+      if (auto val = evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr))
+        message = *val;
+      else
+        messageExpr = expr;
+    }
+  }
+
+  fir::FirOpBuilder &builder = converter.getFirOpBuilder();
+  const mlir::Location loc = converter.getCurrentLocation();
+  const mlir::omp::ClauseSeverity sev =
+      args.severity == parser::OmpSeverityClause::SevLevel::Warning
+          ? mlir::omp::ClauseSeverity::warning
+          : mlir::omp::ClauseSeverity::fatal;
+
+  // A compile-time-constant message is stored directly on the operation as an
+  // attribute. A non-constant message is lowered to a null-terminated string in
+  // memory and passed as the `message_expr` operand.
+  mlir::StringAttr msgAttr;
+  mlir::Value msgExprVal;
+  if (message) {
+    msgAttr = builder.getStringAttr(*message);
+  } else if (messageExpr) {
+    lower::StatementContext stmtCtx;
+    fir::ExtendedValue str = converter.genExprAddr(loc, *messageExpr, stmtCtx);
+    msgExprVal = genNullTerminatedString(builder, loc, str);
+  }
+
+  mlir::omp::ErrorOp::create(builder, loc, sev, msgAttr, msgExprVal);
+}
+
 static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                    semantics::SemanticsContext &semaCtx,
                    lower::pft::Evaluation &eval,
@@ -6612,10 +6705,8 @@ static void genOMP(lower::AbstractConverter &converter, lower::SymMap &symTable,
                     [&](const parser::OmpNothingDirective &) {
                       // nothing-directive is a no-op (OpenMP 5.2 [8.4])
                     },
-                    [&](const parser::OmpErrorDirective &) {
-                      if (!semaCtx.langOptions().OpenMPSimd)
-                        TODO(converter.getCurrentLocation(),
-                             "OmpErrorDirective");
+                    [&](const parser::OmpErrorDirective &errDir) {
+                      genErrorDirective(converter, semaCtx, errDir);
                     },
                 },
                 dir.u);

diff  --git a/flang/lib/Semantics/check-omp-structure.cpp b/flang/lib/Semantics/check-omp-structure.cpp
index 0547baaa4ecba..f77382c9045fc 100644
--- a/flang/lib/Semantics/check-omp-structure.cpp
+++ b/flang/lib/Semantics/check-omp-structure.cpp
@@ -2725,6 +2725,30 @@ void OmpStructureChecker::Enter(const parser::OpenMPDispatchConstruct &x) {
   }
 }
 
+void OmpStructureChecker::Enter(const parser::OmpErrorDirective &x) {
+  const OmpErrorArgs args{GetErrorDirectiveArgs(x)};
+
+  // The AT(execution) form (non-default) is handled at run time.
+  if (args.at != parser::OmpAtClause::ActionTime::Compilation) {
+    return;
+  }
+
+  std::optional<std::string> message;
+  if (args.message) {
+    if (auto expr{GetEvaluateExpr(*args.message)}) {
+      if (auto val{evaluate::GetScalarConstantValue<evaluate::Ascii>(*expr)}) {
+        message = *val;
+      }
+    }
+  }
+
+  if (args.severity == parser::OmpSeverityClause::SevLevel::Warning) {
+    context_.Say(x.v.source, "%s"_warn_en_US, message.value_or("WARNING"));
+  } else {
+    context_.Say(x.v.source, "%s"_err_en_US, message.value_or("ERROR"));
+  }
+}
+
 void OmpStructureChecker::Enter(const parser::OmpClause::At &x) {
   if (GetDirectiveNest(DeclarativeNest) > 0) {
     if (x.v.v == parser::OmpAtClause::ActionTime::Execution) {
@@ -2734,6 +2758,24 @@ void OmpStructureChecker::Enter(const parser::OmpClause::At &x) {
   }
 }
 
+void OmpStructureChecker::Enter(const parser::OmpClause::Message &x) {
+  if (const auto expr{GetEvaluateExpr(x.v.v)}) {
+    const std::optional<evaluate::DynamicType> type{expr->GetType()};
+    if (!type || type->category() != evaluate::TypeCategory::Character) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be of type CHARACTER"_err_en_US);
+    } else if (expr->Rank() != 0) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be scalar"_err_en_US);
+    } else if (type->kind() !=
+        context_.defaultKinds().GetDefaultKind(
+            evaluate::TypeCategory::Character)) {
+      context_.Say(GetContext().clauseSource,
+          "The MESSAGE clause expression must be of default character kind"_err_en_US);
+    }
+  }
+}
+
 void OmpStructureChecker::Enter(const parser::OpenMPAllocatorsConstruct &x) {
   const parser::OmpDirectiveSpecification &beginSpec{x.BeginDir()};
   const parser::OmpDirectiveName &dirName{beginSpec.DirName()};

diff  --git a/flang/lib/Semantics/check-omp-structure.h b/flang/lib/Semantics/check-omp-structure.h
index 0f21bbd19189e..22106020bcc3b 100644
--- a/flang/lib/Semantics/check-omp-structure.h
+++ b/flang/lib/Semantics/check-omp-structure.h
@@ -145,6 +145,8 @@ class OmpStructureChecker : public OmpStructureCheckerBase {
   void Enter(const parser::OmpDirectiveSpecification &);
   void Leave(const parser::OmpDirectiveSpecification &);
 
+  void Enter(const parser::OmpErrorDirective &);
+
   void Enter(const parser::OmpMetadirectiveDirective &);
   void Leave(const parser::OmpMetadirectiveDirective &);
 
@@ -202,6 +204,7 @@ class OmpStructureChecker : public OmpStructureCheckerBase {
   void Enter(const parser::OmpClause::Linear &x);
   void Enter(const parser::OmpClause::Looprange &x);
   void Enter(const parser::OmpClause::Map &x);
+  void Enter(const parser::OmpClause::Message &x);
   void Enter(const parser::OmpClause::NumTeams &x);
   void Enter(const parser::OmpClause::NumThreads &x);
   void Enter(const parser::OmpClause::OmpxBare &x);

diff  --git a/flang/lib/Semantics/openmp-utils.cpp b/flang/lib/Semantics/openmp-utils.cpp
index 27d23a9ffde30..731d98f1103a5 100644
--- a/flang/lib/Semantics/openmp-utils.cpp
+++ b/flang/lib/Semantics/openmp-utils.cpp
@@ -920,6 +920,22 @@ bool IsFullUnroll(const parser::OmpDirectiveSpecification &spec) {
   return false;
 }
 
+OmpErrorArgs GetErrorDirectiveArgs(const parser::OmpErrorDirective &errDir) {
+  OmpErrorArgs args;
+  for (const parser::OmpClause &clause : errDir.v.Clauses().v) {
+    if (const auto *at{std::get_if<parser::OmpClause::At>(&clause.u)}) {
+      args.at = at->v.v;
+    } else if (const auto *sev{
+                   std::get_if<parser::OmpClause::Severity>(&clause.u)}) {
+      args.severity = sev->v.v;
+    } else if (const auto *msg{
+                   std::get_if<parser::OmpClause::Message>(&clause.u)}) {
+      args.message = &msg->v.v;
+    }
+  }
+  return args;
+}
+
 static bool IsTransformableLoop(const parser::OmpDirectiveSpecification &spec) {
   return !IsFullUnroll(spec) && IsLoopTransforming(spec.DirId());
 }

diff  --git a/flang/test/Lower/OpenMP/Todo/error.f90 b/flang/test/Lower/OpenMP/Todo/error.f90
deleted file mode 100644
index c8338278a4267..0000000000000
--- a/flang/test/Lower/OpenMP/Todo/error.f90
+++ /dev/null
@@ -1,7 +0,0 @@
-! RUN: %not_todo_cmd %flang_fc1 -emit-fir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
-
-! CHECK: not yet implemented: OmpErrorDirective
-program p
-  integer, allocatable :: x
-  !$omp error at(compilation) severity(warning) message("an error")
-end program p

diff  --git a/flang/test/Lower/OpenMP/error.f90 b/flang/test/Lower/OpenMP/error.f90
new file mode 100644
index 0000000000000..fbca3ec4f45e2
--- /dev/null
+++ b/flang/test/Lower/OpenMP/error.f90
@@ -0,0 +1,47 @@
+! RUN: %flang_fc1 -emit-hlfir -fopenmp -fopenmp-version=51 -o - %s 2>&1 | FileCheck %s
+
+! Lowering of `!$omp error at(execution)` to the `omp.error` operation. The
+! `at(compilation)` form is handled entirely in semantics and produces no IR.
+! Severity is `warning` or `fatal` (default `fatal`); the message is an optional
+! constant string that is omitted when absent. A non-constant message is lowered
+! to a null-terminated runtime string and passed as an operand.
+
+! CHECK-LABEL: func.func @_QQmain()
+program p
+  ! CHECK: omp.error severity(warning) message("a warning")
+  !$omp error at(execution) severity(warning) message("a warning")
+
+  ! CHECK: omp.error severity(fatal) message("fatal error")
+  !$omp error at(execution) severity(fatal) message("fatal error")
+
+  ! No MESSAGE clause; severity defaults to `fatal`.
+  ! CHECK: omp.error severity(fatal)
+  !$omp error at(execution)
+
+  ! An explicit empty MESSAGE string is lowered to an empty constant `message`
+  ! attribute (distinct from an absent clause) at the HLFIR level.
+  ! CHECK: omp.error severity(warning) message("")
+  !$omp error at(execution) severity(warning) message("")
+end program p
+
+! A non-constant MESSAGE is only known at run time, so it is lowered to a
+! null-terminated copy in memory and passed as the `message_expr` operand
+! instead of as a constant attribute.
+! CHECK-LABEL: func.func @_QPf_runtime_message
+subroutine f_runtime_message
+  character(len=16) :: msg
+  msg = "runtime warning"
+  ! CHECK: omp.error severity(warning) message_expr(%{{.*}} : !fir.ref<!fir.char<1>>)
+  !$omp error at(execution) severity(warning) message(msg)
+end subroutine f_runtime_message
+
+! The `at(compilation)` form (and the implicit default, which is also
+! compilation) is resolved entirely in semantics and must not lower to an
+! `omp.error` operation. `severity(warning)` keeps these as non-fatal
+! diagnostics so compilation still succeeds.
+! CHECK-LABEL: func.func @_QPf_compilation
+subroutine f_compilation
+  ! CHECK-NOT: omp.error
+  !$omp error severity(warning) message("implicit compilation")
+  !$omp error at(compilation) severity(warning) message("explicit compilation")
+end subroutine f_compilation

diff  --git a/flang/test/Parser/OpenMP/error-unparse.f90 b/flang/test/Parser/OpenMP/error-unparse.f90
index 0902d109fd233..5689de8374297 100644
--- a/flang/test/Parser/OpenMP/error-unparse.f90
+++ b/flang/test/Parser/OpenMP/error-unparse.f90
@@ -1,27 +1,37 @@
-! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-unparse %s 2>&1 | FileCheck %s
-! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-dump-parse-tree %s 2>&1 | FileCheck %s --check-prefix="PARSE-TREE"
+! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-unparse-no-sema %s 2>&1 | FileCheck %s
+! RUN: %flang_fc1  -fopenmp-version=51 -fopenmp -fdebug-dump-parse-tree-no-sema %s 2>&1 | FileCheck %s --check-prefix="PARSE-TREE"
+
+! Compilation-time diagnostics are covered by Semantics/OpenMP/error.f90.
 program main
   character(*), parameter :: message = "This is an error"
   !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(WARNING) MESSAGE("some message here")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Compilation
   !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Warning
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"some message here"'
-  !PARSE-TREE:  LiteralConstant -> CharLiteralConstant
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> LiteralConstant -> CharLiteralConstant
   !PARSE-TREE:  string = 'some message here'
   !$omp error at(compilation) severity(warning) message("some message here")
-  !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(FATAL) MESSAGE("This is an error")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
+  !CHECK: !$OMP ERROR AT(COMPILATION) SEVERITY(FATAL) MESSAGE(message)
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Compilation
   !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Fatal
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"This is an error"'
-  !PARSE-TREE:  Designator -> DataRef -> Name = 'message'
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> Designator -> DataRef -> Name = 'message'
   !$omp error at(compilation) severity(fatal) message(message)
-  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(FATAL) MESSAGE("This is an error")
-  !PARSE-TREE: ExecutionPartConstruct -> ExecutableConstruct -> OpenMPConstruct -> OmpUtilityDirective -> OmpErrorDirective
+  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(FATAL) MESSAGE(message)
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
   !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Execution
   !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Fatal
-  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr = '"This is an error"'
-  !PARSE-TREE:  Designator ->  DataRef -> Name = 'message'
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> Designator -> DataRef -> Name = 'message'
   !$omp error at(EXECUTION) severity(fatal) message(message)
+  !CHECK: !$OMP ERROR AT(EXECUTION) SEVERITY(WARNING) MESSAGE("a warning")
+  !PARSE-TREE: DeclarationConstruct -> SpecificationConstruct -> OpenMPDeclarativeConstruct -> OmpUtilityDirective -> OmpErrorDirective -> OmpDirectiveSpecification
+  !PARSE-TREE: OmpDirectiveName -> llvm::omp::Directive = error
+  !PARSE-TREE: OmpClauseList -> OmpClause -> At -> OmpAtClause -> ActionTime = Execution
+  !PARSE-TREE: OmpClause -> Severity -> OmpSeverityClause -> SevLevel = Warning
+  !PARSE-TREE:  OmpClause -> Message -> OmpMessageClause -> Expr -> LiteralConstant -> CharLiteralConstant
+  !PARSE-TREE:  string = 'a warning'
+  !$omp error at(execution) severity(warning) message("a warning")
 end program main

diff  --git a/flang/test/Semantics/OpenMP/error.f90 b/flang/test/Semantics/OpenMP/error.f90
index 067417a8cda3b..97eaa48d9899e 100644
--- a/flang/test/Semantics/OpenMP/error.f90
+++ b/flang/test/Semantics/OpenMP/error.f90
@@ -6,3 +6,46 @@ subroutine f00(x)
   integer :: x
 end
 
+subroutine f01
+!ERROR: an error
+  !$omp error at(compilation) severity(fatal) message("an error")
+end
+
+subroutine f02
+!WARNING: a warning
+  !$omp error at(compilation) severity(warning) message("a warning")
+end
+
+subroutine f03
+!ERROR: ERROR
+  !$omp error
+end
+
+subroutine f04
+!ERROR: ERROR
+  !$omp error at(compilation) severity(fatal)
+end
+
+subroutine f05
+!WARNING: WARNING
+  !$omp error at(compilation) severity(warning)
+end
+
+subroutine f06(n)
+  integer :: n
+!ERROR: The MESSAGE clause expression must be of type CHARACTER
+  !$omp error at(execution) severity(warning) message(n)
+end
+
+subroutine f07(c)
+  character(len=8) :: c(4)
+!ERROR: The MESSAGE clause expression must be scalar
+  !$omp error at(execution) severity(warning) message(c)
+end
+
+subroutine f08
+  character(kind=4, len=8) :: c
+!ERROR: The MESSAGE clause expression must be of default character kind
+  !$omp error at(execution) severity(warning) message(c)
+end
+

diff  --git a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
index fee8c712998b5..05307414c94d0 100644
--- a/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
+++ b/llvm/include/llvm/Frontend/OpenMP/OMPIRBuilder.h
@@ -1497,6 +1497,18 @@ class OpenMPIRBuilder {
   /// \param Loc The location where the flush directive was encountered
   LLVM_ABI void createFlush(const LocationDescription &Loc);
 
+  /// Generate a call to the runtime to emit the diagnostic of an OpenMP
+  /// `error` directive with `at(execution)`.
+  ///
+  /// \param Loc The location where the error directive was encountered; it is
+  ///        used to build the `ident_t` passed to the runtime.
+  /// \param IsFatal Selects `severity(fatal)` (true) or `severity(warning)`
+  ///        (false).
+  /// \param Message The message string (an `i8*`) to display, or null when no
+  ///        `message` clause is present.
+  LLVM_ABI void createError(const LocationDescription &Loc, bool IsFatal,
+                            Value *Message);
+
   /// Generator for '#omp taskyield'
   ///
   /// \param Loc The location where the taskyield directive was encountered.

diff  --git a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
index 57e199bdff09a..fc3812502fb2e 100644
--- a/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
+++ b/llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
@@ -2184,6 +2184,25 @@ void OpenMPIRBuilder::createFlush(const LocationDescription &Loc) {
   emitFlush(Loc);
 }
 
+void OpenMPIRBuilder::createError(const LocationDescription &Loc, bool IsFatal,
+                                  Value *Message) {
+  if (!updateToLocation(Loc))
+    return;
+
+  // Build call void __kmpc_error(ident_t *loc, int severity,
+  //                              const char *message)
+  uint32_t SrcLocStrSize;
+  Constant *SrcLocStr = getOrCreateSrcLocStr(Loc, SrcLocStrSize);
+  Value *Ident = getOrCreateIdent(SrcLocStr, SrcLocStrSize);
+  // Severity: 1 = warning, 2 = fatal.
+  Value *Severity = ConstantInt::get(Int32, IsFatal ? 2 : 1);
+  Value *MessageArg = Message ? Message : ConstantPointerNull::get(Int8Ptr);
+  Value *Args[] = {Ident, Severity, MessageArg};
+
+  createRuntimeFunctionCall(getOrCreateRuntimeFunctionPtr(OMPRTL___kmpc_error),
+                            Args);
+}
+
 void OpenMPIRBuilder::emitTaskyieldImpl(const LocationDescription &Loc) {
   // Build call __kmpc_omp_taskyield(loc, thread_id, 0);
   uint32_t SrcLocStrSize;

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
index e373d1a52a3fc..c642792f67019 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPEnums.td
@@ -77,6 +77,26 @@ def ClauseDependAttr : OpenMP_EnumAttr<ClauseDepend, "clause_depend"> {
   let assemblyFormat = "`(` $value `)`";
 }
 
+//===----------------------------------------------------------------------===//
+// severity enum.
+//===----------------------------------------------------------------------===//
+
+// The values match the `severity` argument expected by the `__kmpc_error`
+// runtime entry point (1 = warning, 2 = fatal).
+def ClauseSeverityWarning : I32EnumAttrCase<"warning", 1>;
+def ClauseSeverityFatal : I32EnumAttrCase<"fatal", 2>;
+
+def ClauseSeverity : OpenMP_I32EnumAttr<
+    "ClauseSeverity",
+    "severity clause", [
+      ClauseSeverityWarning,
+      ClauseSeverityFatal
+    ]>;
+
+def ClauseSeverityAttr : OpenMP_EnumAttr<ClauseSeverity, "clause_severity"> {
+  let assemblyFormat = "`(` $value `)`";
+}
+
 //===----------------------------------------------------------------------===//
 // clause_requires enum.
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 0a5736f46b088..70597b85902c5 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -1775,6 +1775,37 @@ def BarrierOp : OpenMP_Op<"barrier"> {
   let assemblyFormat = "attr-dict";
 }
 
+//===----------------------------------------------------------------------===//
+// [5.1] 2.5.4 error directive
+//===----------------------------------------------------------------------===//
+
+def ErrorOp : OpenMP_Op<"error"> {
+  let summary = "error directive";
+  let description = [{
+    The error directive instructs the compiler or runtime to display a message
+    and, when the severity is `fatal`, to terminate execution. The
+    `at(compilation)` form is handled entirely by the frontend; this operation
+    represents only the `at(execution)` form, which lowers to a call to the
+    `__kmpc_error` runtime entry point.
+
+    The `severity` attribute selects `warning` or `fatal` behavior. The message
+    to display is optional and may be provided as a compile-time-constant
+    string in the `message` attribute, or a pointer to a null-terminated string.
+  }];
+
+  let arguments = (ins ClauseSeverityAttr:$severity,
+                       OptionalAttr<StrAttr>:$message,
+                       Optional<OpenMP_PointerLikeType>:$message_expr);
+
+  let assemblyFormat = [{
+    `severity` `(` custom<ClauseAttr>($severity) `)`
+    ( `message` `(` $message^ `)` )?
+    ( `message_expr` `(` $message_expr^ `:` type($message_expr) `)` )? attr-dict
+  }];
+
+  let hasVerifier = 1;
+}
+
 //===----------------------------------------------------------------------===//
 // [5.1] 2.19.9 ordered Construct
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 67bc2bdf35619..95ccb53bce3d4 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -4618,6 +4618,18 @@ LogicalResult CriticalOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// Spec 5.1: Error directive (2.5.4)
+//===----------------------------------------------------------------------===//
+
+LogicalResult ErrorOp::verify() {
+  if (getMessage() && getMessageExpr())
+    return emitOpError() << "the message must be provided either as a constant "
+                            "`message` attribute or as a `message_expr` "
+                            "operand, but not both";
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // Ordered construct
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 1c38fd27daae1..97993a80926eb 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -9828,6 +9828,21 @@ LogicalResult OpenMPDialectLLVMIRTranslationInterface::convertOperation(
             ompBuilder->createFlush(builder);
             return success();
           })
+          .Case([&](omp::ErrorOp op) {
+            if (failed(checkImplementationStatus(*op)))
+              return failure();
+
+            llvm::Value *message = nullptr;
+            if (mlir::Value messageExpr = op.getMessageExpr())
+              message = moduleTranslation.lookupValue(messageExpr);
+            else if (std::optional<StringRef> msg = op.getMessage();
+                     msg && !msg->empty())
+              message = builder.CreateGlobalString(*msg);
+            ompBuilder->createError(
+                llvm::OpenMPIRBuilder::LocationDescription(builder),
+                op.getSeverity() == omp::ClauseSeverity::fatal, message);
+            return success();
+          })
           .Case([&](omp::ParallelOp op) {
             return convertOmpParallel(op, builder, moduleTranslation);
           })

diff  --git a/mlir/test/Dialect/OpenMP/invalid.mlir b/mlir/test/Dialect/OpenMP/invalid.mlir
index cf7be4eb638cc..a568bffe3cf99 100644
--- a/mlir/test/Dialect/OpenMP/invalid.mlir
+++ b/mlir/test/Dialect/OpenMP/invalid.mlir
@@ -4913,3 +4913,10 @@ func.func @omp_unroll_partial_factor_negative(%cli : !omp.cli) -> () {
   omp.unroll_partial(%cli) {unroll_factor = -1 : i64}
   return
 }
+
+// -----
+func.func @omp_error_message_and_message_expr(%msg : !llvm.ptr) -> () {
+  // expected-error @below {{the message must be provided either as a constant `message` attribute or as a `message_expr` operand, but not both}}
+  omp.error severity(warning) message("a warning") message_expr(%msg : !llvm.ptr)
+  return
+}

diff  --git a/mlir/test/Dialect/OpenMP/ops.mlir b/mlir/test/Dialect/OpenMP/ops.mlir
index d08ca5664efe4..ea25a0a75f646 100644
--- a/mlir/test/Dialect/OpenMP/ops.mlir
+++ b/mlir/test/Dialect/OpenMP/ops.mlir
@@ -60,6 +60,31 @@ func.func @omp_flush(%arg0 : memref<i32>) -> () {
   return
 }
 
+// CHECK-LABEL: func @omp_error
+func.func @omp_error(%msg : !llvm.ptr) -> () {
+  // Warning severity with a message.
+  // CHECK: omp.error severity(warning) message("a warning")
+  omp.error severity(warning) message("a warning")
+
+  // Fatal severity with a message.
+  // CHECK: omp.error severity(fatal) message("fatal error")
+  omp.error severity(fatal) message("fatal error")
+
+  // Fatal severity without a message.
+  // CHECK: omp.error severity(fatal)
+  omp.error severity(fatal)
+
+  // Warning severity without a message.
+  // CHECK: omp.error severity(warning)
+  omp.error severity(warning)
+
+  // Warning severity with a runtime message operand.
+  // CHECK: omp.error severity(warning) message_expr(%{{.*}} : !llvm.ptr)
+  omp.error severity(warning) message_expr(%msg : !llvm.ptr)
+
+  return
+}
+
 func.func @omp_terminator() -> () {
   // CHECK: omp.terminator
   omp.terminator

diff  --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index db38c8da30777..d26557d57d0dd 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -41,6 +41,30 @@ llvm.func @test_flush_construct(%arg0: !llvm.ptr) {
   llvm.return
 }
 
+// CHECK-LABEL: define void @test_error_construct
+llvm.func @test_error_construct(%msg : !llvm.ptr) {
+  // Severity is 1 for `warning` and 2 for `fatal`; the message is a pointer to
+  // a constant string, or a null pointer when absent.
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr @{{.*}})
+  omp.error severity(warning) message("a warning")
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 2, ptr @{{.*}})
+  omp.error severity(fatal) message("fatal error")
+
+  // A runtime message is passed through as the supplied pointer operand.
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr %{{.*}})
+  omp.error severity(warning) message_expr(%msg : !llvm.ptr)
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 2, ptr null)
+  omp.error severity(fatal)
+
+  // CHECK: call void @__kmpc_error(ptr @{{[0-9]+}}, i32 1, ptr null)
+  omp.error severity(warning)
+
+  // CHECK-NEXT:    ret void
+  llvm.return
+}
+
 // CHECK-LABEL: define void @test_omp_parallel_1()
 llvm.func @test_omp_parallel_1() -> () {
   // CHECK: call void{{.*}}@__kmpc_fork_call{{.*}}@[[OMP_OUTLINED_FN_1:.*]])


        

From llvmlistbot at llvm.org  Fri Jul 24 07:43:14 2026
From: llvmlistbot at llvm.org (Caroline Newcombe)
Date: Fri, 24 Jul 2026 07:43:14 -0700 (PDT)
Subject: [Mlir-commits] [flang] [llvm] [mlir] [flang][OpenMP] Implement the
 error directive (PR #206175)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206175@github.com>
Message-ID: <6a637a01.5a0412d2.325e39.4e38@mx.google.com>


https://github.com/cenewcombe closed https://github.com/llvm/llvm-project/pull/206175

From llvmlistbot at llvm.org  Fri Jul 24 07:43:59 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:43:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][Vector] Reject invalid inline yields
	(PR #206218)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206218@github.com>
Message-ID: <6a637a2f.eb165e5c.cb4e7.90fc@mx.google.com>


PragmaTwice wrote:

Also could you read https://llvm.org/docs/AIToolPolicy.html and confirm that you are you are following the policy?

https://github.com/llvm/llvm-project/pull/206218

From llvmlistbot at llvm.org  Fri Jul 24 07:46:13 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 07:46:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a637ab5.af4834b4.1d4518.626e@mx.google.com>


banach-space wrote:

> Thanks @banach-space! I hope that now it's OK, it's a way more clearer :D

Much cleaner, thank you!

> A dynamic stride is still accepted, unlike vector.load/vector.store (tests added to document the asymmetry.

Why is this exception required?

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Fri Jul 24 07:47:15 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:47:15 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Reject invalid boundary types (PR
	#208787)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208787@github.com>
Message-ID: <6a637af3.7d702ed2.2f6c05.be4a@mx.google.com>


PragmaTwice wrote:

Hi, could you read https://llvm.org/docs/AIToolPolicy.html and confirm that you are you are following the policy?

https://github.com/llvm/llvm-project/pull/208787

From llvmlistbot at llvm.org  Fri Jul 24 07:47:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:47:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][bufferization] Skip escaping
	allocation users (PR #208979)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208979@github.com>
Message-ID: <6a637b06.af6ba3ed.1989e7.ad1f@mx.google.com>


PragmaTwice wrote:

Hi, could you read https://llvm.org/docs/AIToolPolicy.html and confirm that you are you are following the policy?

https://github.com/llvm/llvm-project/pull/208979

From llvmlistbot at llvm.org  Fri Jul 24 07:47:51 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:47:51 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Coordinate branch argument cleanup (PR
	#208940)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208940@github.com>
Message-ID: <6a637b17.492e23e4.3ce411.cd79@mx.google.com>


PragmaTwice wrote:

Hi, could you read https://llvm.org/docs/AIToolPolicy.html and confirm that you are you are following the policy?

https://github.com/llvm/llvm-project/pull/208940

From llvmlistbot at llvm.org  Fri Jul 24 07:50:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 07:50:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Preserve live branch arguments (PR
	#208742)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208742@github.com>
Message-ID: <6a637b9f.83ce629a.320329.624e@mx.google.com>


PragmaTwice wrote:

Hi, could you read https://llvm.org/docs/AIToolPolicy.html and confirm that you are you are following the policy?

https://github.com/llvm/llvm-project/pull/208742

From llvmlistbot at llvm.org  Fri Jul 24 07:58:57 2026
From: llvmlistbot at llvm.org (Leandro Lupori)
Date: Fri, 24 Jul 2026 07:58:57 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a637db1.d681cab9.3d08c3.913e@mx.google.com>


https://github.com/luporl updated https://github.com/llvm/llvm-project/pull/183800

>From 49d7a2054c0b123a6783a49383c71066b7e1d1d9 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Fri, 27 Feb 2026 15:32:58 -0300
Subject: [PATCH 1/6] [mlir][OpenMP] Fix update of linear iteration variables

The final value of a linear iteration variable must be the loop
limit_value + step. Before this patch it was limit_value.

This fixes the second issue reported in #170784.
---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 129 ++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  15 +-
 2 files changed, 103 insertions(+), 41 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index deefa29c157f3..993cd89f74300 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -150,6 +150,37 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearFinalizationBB;
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
+  Value linearLoopIV;
+  Value linearLoopIVStart;
+
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
+                       llvm::Value *var, llvm::Value *varStart,
+                       llvm::Value *step, llvm::Value *iv) {
+    if (!iv->getType()->isIntegerTy())
+      llvm_unreachable("OpenMP loop induction variable must be an integer "
+                       "type");
+
+    if (varType->isIntegerTy()) {
+      // Integer path: normalize all arithmetic to linearVarType
+      iv = builder.CreateSExtOrTrunc(iv, varType);
+      step = builder.CreateSExtOrTrunc(step, varType);
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
+      builder.CreateStore(addInst, var);
+    } else if (varType->isFloatingPointTy()) {
+      // Float path: perform multiply in integer, then convert to float
+      step = builder.CreateSExtOrTrunc(step, iv->getType());
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
+      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
+      builder.CreateStore(addInst, var);
+    } else {
+      llvm_unreachable(
+          "Linear variable must be of integer or floating-point type");
+    }
+  }
 
 public:
   // Register type for the linear variables
@@ -189,46 +220,63 @@ class LinearClauseProcessor {
     }
   }
 
+  // Find linear iteration variable and save it for later updates
+  void initLinearIV(omp::SimdOp simdOp) {
+    auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
+    // NOTE iteration variables can only be linear in non-nested loops.
+    if (loopOp.getIVs().size() != 1)
+      return;
+    // The linear IV is the loop IV's store address.
+    BlockArgument arg = loopOp.getIVs().front();
+    for (const Operation *user : arg.getUsers()) {
+      if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
+        for (Value linearVar : simdOp.getLinearVars()) {
+          if (linearVar == storeOp.getAddr()) {
+            linearLoopIV = linearVar;
+            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
+            break;
+          }
+        }
+      }
+    }
+  }
+
   // Emit IR for updating Linear variables
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
-                       llvm::Value *loopInductionVar) {
+  void updateLinearVars(llvm::IRBuilderBase &builder,
+                        llvm::BasicBlock *loopBody,
+                        llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::Type *linearVarType = linearVarTypes[index];
-      llvm::Value *iv = loopInductionVar;
-      llvm::Value *step = linearSteps[index];
-
-      if (!iv->getType()->isIntegerTy())
-        llvm_unreachable("OpenMP loop induction variable must be an integer "
-                         "type");
-
-      if (linearVarType->isIntegerTy()) {
-        // Integer path: normalize all arithmetic to linearVarType
-        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
-        step = builder.CreateSExtOrTrunc(step, linearVarType);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else if (linearVarType->isFloatingPointTy()) {
-        // Float path: perform multiply in integer, then convert to float
-        step = builder.CreateSExtOrTrunc(step, iv->getType());
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
-        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else {
-        llvm_unreachable(
-            "Linear variable must be of integer or floating-point type");
-      }
+      llvm::LoadInst *linearVarStart = builder.CreateLoad(
+          linearVarTypes[index], linearPreconditionVars[index]);
+      updateLinearVar(builder, linearVarTypes[index],
+                      linearLoopBodyTemps[index], linearVarStart,
+                      linearSteps[index], loopInductionVar);
     }
   }
 
+  // Emit IR for updating linear iteration variables on loop exit
+  void updateLinearIV(llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation,
+                      llvm::Value *loopIV) {
+    if (!linearLoopIV)
+      return;
+    llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
+    llvm::Value *linearIVStart =
+        moduleTranslation.lookupValue(linearLoopIVStart);
+
+    // Find linearIV's index
+    size_t index;
+    for (index = 0; index < linearOrigVal.size(); index++)
+      if (linearIV == linearOrigVal[index])
+        break;
+    if (index == linearOrigVal.size())
+      return;
+
+    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
+                    linearIVStart, linearSteps[index], loopIV);
+  }
+
   // Linear variable finalization is conditional on the last logical iteration.
   // Create BB splits to manage the same.
   void splitLinearFiniBB(llvm::IRBuilderBase &builder,
@@ -3852,8 +3900,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4153,6 +4201,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
 
+  linearClauseProcessor.initLinearIV(simdOp);
+
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
     for (mlir::Attribute linearVarType : linearVarTypes)
@@ -4250,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4261,6 +4311,9 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
+                                       loopInfo->getIndVar());
+
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 38f10320bd3ae..45eb6ac1ef957 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,6 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
+// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -758,8 +759,16 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK: %[[ADD:.*]] = add i32 %[[LOAD]], %[[MUL]]
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
+
+// CHECK: omp.region.cont:
+// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
+// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
+// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
+// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
+// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
       omp.yield
     }
   } {linear_var_types = [i32]}
@@ -784,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -817,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
-// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group

>From 20ac4aef52312cd93944ad17f0fef0b126211102 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 19 May 2026 12:06:23 -0300
Subject: [PATCH 2/6] Fix "instruction does not dominate all uses" issue

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 104 +++++++++---------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++++--
 2 files changed, 90 insertions(+), 62 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 993cd89f74300..aace5283e4362 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -151,36 +151,6 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
   Value linearLoopIV;
-  Value linearLoopIVStart;
-
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
-                       llvm::Value *var, llvm::Value *varStart,
-                       llvm::Value *step, llvm::Value *iv) {
-    if (!iv->getType()->isIntegerTy())
-      llvm_unreachable("OpenMP loop induction variable must be an integer "
-                       "type");
-
-    if (varType->isIntegerTy()) {
-      // Integer path: normalize all arithmetic to linearVarType
-      iv = builder.CreateSExtOrTrunc(iv, varType);
-      step = builder.CreateSExtOrTrunc(step, varType);
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
-      builder.CreateStore(addInst, var);
-    } else if (varType->isFloatingPointTy()) {
-      // Float path: perform multiply in integer, then convert to float
-      step = builder.CreateSExtOrTrunc(step, iv->getType());
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
-      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
-      builder.CreateStore(addInst, var);
-    } else {
-      llvm_unreachable(
-          "Linear variable must be of integer or floating-point type");
-    }
-  }
 
 public:
   // Register type for the linear variables
@@ -233,7 +203,6 @@ class LinearClauseProcessor {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
             linearLoopIV = linearVar;
-            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
             break;
           }
         }
@@ -242,28 +211,51 @@ class LinearClauseProcessor {
   }
 
   // Emit IR for updating Linear variables
-  void updateLinearVars(llvm::IRBuilderBase &builder,
-                        llvm::BasicBlock *loopBody,
-                        llvm::Value *loopInductionVar) {
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
+                       llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::LoadInst *linearVarStart = builder.CreateLoad(
-          linearVarTypes[index], linearPreconditionVars[index]);
-      updateLinearVar(builder, linearVarTypes[index],
-                      linearLoopBodyTemps[index], linearVarStart,
-                      linearSteps[index], loopInductionVar);
+      llvm::Type *linearVarType = linearVarTypes[index];
+      llvm::Value *iv = loopInductionVar;
+      llvm::Value *step = linearSteps[index];
+
+      if (!iv->getType()->isIntegerTy())
+        llvm_unreachable("OpenMP loop induction variable must be an integer "
+                         "type");
+
+      if (linearVarType->isIntegerTy()) {
+        // Integer path: normalize all arithmetic to linearVarType
+        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
+        step = builder.CreateSExtOrTrunc(step, linearVarType);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else if (linearVarType->isFloatingPointTy()) {
+        // Float path: perform multiply in integer, then convert to float
+        step = builder.CreateSExtOrTrunc(step, iv->getType());
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
+        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else {
+        llvm_unreachable(
+            "Linear variable must be of integer or floating-point type");
+      }
     }
   }
 
   // Emit IR for updating linear iteration variables on loop exit
   void updateLinearIV(llvm::IRBuilderBase &builder,
-                      LLVM::ModuleTranslation &moduleTranslation,
-                      llvm::Value *loopIV) {
+                      LLVM::ModuleTranslation &moduleTranslation) {
     if (!linearLoopIV)
       return;
     llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
-    llvm::Value *linearIVStart =
-        moduleTranslation.lookupValue(linearLoopIVStart);
 
     // Find linearIV's index
     size_t index;
@@ -273,8 +265,17 @@ class LinearClauseProcessor {
     if (index == linearOrigVal.size())
       return;
 
-    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
-                    linearIVStart, linearSteps[index], loopIV);
+    // Add one more step to the linear iteration variable
+    llvm::Type *varType = linearVarTypes[index];
+    llvm::Value *var = linearLoopBodyTemps[index];
+    llvm::Value *step = linearSteps[index];
+    if (!varType->isIntegerTy())
+      llvm_unreachable("Linear iteration variable must be of integer type");
+
+    step = builder.CreateSExtOrTrunc(step, varType);
+    llvm::Value *val = builder.CreateLoad(varType, var);
+    llvm::Value *addInst = builder.CreateAdd(val, step);
+    builder.CreateStore(addInst, var);
   }
 
   // Linear variable finalization is conditional on the last logical iteration.
@@ -3900,8 +3901,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4200,7 +4201,6 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-
   linearClauseProcessor.initLinearIV(simdOp);
 
   if (!simdOp.getLinearVars().empty()) {
@@ -4300,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4311,9 +4311,7 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
-  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
-                                       loopInfo->getIndVar());
-
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation);
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 45eb6ac1ef957..51a12e23a0e5e 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,7 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
-// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
+// CHECK-SAME: (i32 %{{.*}}, i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -761,11 +761,11 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
 
 // CHECK: omp.region.cont:
-// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
-// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
-// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
-// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
-// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], %[[STEP]]
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       llvm.store %iv, %x : i32, !llvm.ptr
@@ -793,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
+// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -826,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
-// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group
@@ -841,6 +841,36 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // -----
 
+// Test the update of omp.simd linear iteration variables, when nested inside
+// omp.wsloop.
+llvm.func @wsloop_simd_linear(%x : !llvm.ptr) {
+
+// CHECK-LABEL: @wsloop_simd_linear
+
+// CHECK: omp.wsloop.region:
+// CHECK: %[[LINEAR_VAR:.*]] = alloca i32
+// CHECK: %[[LINEAR_RESULT:.*]] = alloca i32
+
+// CHECK: omp.region.cont2:
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], 25
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+  %lb = llvm.mlir.constant(1 : i32) : i32
+  %ub = llvm.mlir.constant(100 : i32) : i32
+  %step = llvm.mlir.constant(25 : i32) : i32
+  omp.wsloop {
+    omp.simd linear(%x : !llvm.ptr = %step : i32) {
+      omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+        llvm.store %iv, %x : i32, !llvm.ptr
+        omp.yield
+      }
+    } {linear_var_types = [i32], omp.composite}
+  } {omp.composite}
+  llvm.return
+}
+
+// -----
+
 // CHECK-LABEL: @simd_simple_multiple
 llvm.func @simd_simple_multiple(%lb1 : i64, %ub1 : i64, %step1 : i64, %lb2 : i64, %ub2 : i64, %step2 : i64, %arg0: !llvm.ptr, %arg1: !llvm.ptr) {
   omp.simd {

>From 030a957a941fb50d15c99f9d0031e26019b53992 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 09:58:04 -0300
Subject: [PATCH 3/6] Verify that linear variables don't appear in other DSA
 clauses

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp     | 10 ++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir | 14 ++++++++++++++
 mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir | 12 +++++-------
 3 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..b328e15313538 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -3363,6 +3363,16 @@ LogicalResult SimdOp::verify() {
   if (getLinearVars().size() &&
       getLinearVarTypes().value().size() != getLinearVars().size())
     return emitError() << "Ill-formed type attributes for linear variables";
+
+  llvm::DenseSet<Value> privateVars(llvm::from_range, getPrivateVars());
+  llvm::DenseSet<Value> reductionVars(llvm::from_range, getReductionVars());
+  // TODO Check lastprivate vars when their support is added to SimdOp.
+  for (Value var : getLinearVars()) {
+    if (privateVars.contains(var) || reductionVars.contains(var))
+      return emitOpError()
+             << "linear variables cannot appear in other data-sharing clauses";
+  }
+
   return success();
 }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index d9459ace54422..7e2e0061f80be 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -116,6 +116,20 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // -----
 
+omp.private {type = private} @i_private_i32 : i32
+llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr) {
+  // expected-error @below {{linear variables cannot appear in other data-sharing clauses}}
+  omp.simd linear(%i : !llvm.ptr = %step : i32)
+           private(@i_private_i32 %i -> %priv_i : !llvm.ptr) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      omp.yield
+    }
+  } {linear_var_types = [i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
index 8da1fab1f48af..a84c51f08daa7 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @simd_ordered_linear
   llvm.func @simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -38,14 +36,14 @@ module {
     // CHECK: %.linear_var = alloca i32
     // CHECK: %.linear_result = alloca i32
 
-    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c10_i32) inclusive step (%c1_i32) {
         // CHECK: omp.loop_nest.region:
         // CHECK: load i32, ptr %.linear_result
-        llvm.store %iv, %arg0 : i32, !llvm.ptr
+        llvm.store %iv, %i : i32, !llvm.ptr
 
         // Compute a[i] = b[i] * 10
-        %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_val = llvm.load %i : !llvm.ptr -> i32
         %i_idx = llvm.sext %i_val : i32 to i64
         %i_off = llvm.sub %i_idx, %c1_i64 : i64
         %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -58,7 +56,7 @@ module {
         omp.ordered.region par_level_simd {
           // CHECK: omp.ordered.region:
           // CHECK: load i32, ptr %.linear_result
-          %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_ord = llvm.load %i : !llvm.ptr -> i32
           %i_ord_idx = llvm.sext %i_ord : i32 to i64
           %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
           %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -69,7 +67,7 @@ module {
         // Compute c[i] = a[i] * 2 (code after ordered region)
         // CHECK: omp_region.finalize:
         // CHECK: load i32, ptr %.linear_result
-        %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_post = llvm.load %i : !llvm.ptr -> i32
         %i_post_idx = llvm.sext %i_post : i32 to i64
         %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
         %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From 2f8b6c3b6f7ca958505c6e6d7cc41ec53cb0fe0d Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 11:10:29 -0300
Subject: [PATCH 4/6] Update mlir tests

---
 mlir/test/Target/LLVMIR/openmp-simd-guided.mlir      |  3 +--
 .../Target/LLVMIR/openmp-wsloop-simd-ordered.mlir    | 12 +++++-------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
index b1f18b9f85417..496669c5b90d5 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
@@ -1,7 +1,6 @@
 // Ensure that schedule can be used with the guided kind-type and simd construct
 // RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
 
-omp.private {type = private} @_QFEi_private_i32 : i32
 llvm.func @test_simd_guided() {
   %0 = llvm.mlir.constant (1 : i64) : i64
   %c1_i32 = llvm.mlir.constant (1 : i32) : i32
@@ -10,7 +9,7 @@ llvm.func @test_simd_guided() {
   %c4_i32 = llvm.mlir.constant (4 : i32) : i32
   %1 = llvm.alloca %0 x i32 {bindc_name = "i"} : (i64) -> !llvm.ptr
   omp.wsloop schedule(guided = %c4_i32 : i32, simd) {
-    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) private(@_QFEi_private_i32 %1 -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%arg1) : i32 = (%c0_i32) to (%c64_i32) inclusive step (%c1_i32) {
         omp.yield
       }
diff --git a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
index beb300c23651f..8d5b89ecbd00a 100644
--- a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end do simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @wsloop_simd_ordered_linear
   llvm.func @wsloop_simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -39,14 +37,14 @@ module {
     // CHECK: %.linear_result = alloca i32
 
     omp.wsloop ordered(0) {
-      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
         omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c100_i32) inclusive step (%c1_i32) {
           // CHECK: omp.loop_nest.region:
           // CHECK: load i32, ptr %.linear_result
-          llvm.store %iv, %arg0 : i32, !llvm.ptr
+          llvm.store %iv, %i : i32, !llvm.ptr
 
           // Compute a[i] = b[i] * 10
-          %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_val = llvm.load %i : !llvm.ptr -> i32
           %i_idx = llvm.sext %i_val : i32 to i64
           %i_off = llvm.sub %i_idx, %c1_i64 : i64
           %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -59,7 +57,7 @@ module {
           omp.ordered.region par_level_simd {
             // CHECK: omp.ordered.region:
             // CHECK: load i32, ptr %.linear_result
-            %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+            %i_ord = llvm.load %i : !llvm.ptr -> i32
             %i_ord_idx = llvm.sext %i_ord : i32 to i64
             %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
             %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -71,7 +69,7 @@ module {
           // Compute c[i] = a[i] * 2 (code after ordered region)
           // CHECK: omp_region.finalize:
           // CHECK: load i32, ptr %.linear_result
-          %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_post = llvm.load %i : !llvm.ptr -> i32
           %i_post_idx = llvm.sext %i_post : i32 to i64
           %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
           %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From b671d6cf6efa8413abd061ac101b1ea14137ff5f Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Thu, 23 Jul 2026 16:48:58 -0300
Subject: [PATCH 5/6] Report an error if the linear variable can't be
 determined unambiguously

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 20 ++++++++++++++-----
 .../Target/LLVMIR/openmp-llvm-invalid.mlir    | 15 ++++++++++++++
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 2a246ebf98b3c..b938572fe3720 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -191,23 +191,32 @@ class LinearClauseProcessor {
   }
 
   // Find linear iteration variable and save it for later updates
-  void initLinearIV(omp::SimdOp simdOp) {
+  LogicalResult initLinearIV(omp::SimdOp simdOp) {
     auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
     // NOTE iteration variables can only be linear in non-nested loops.
     if (loopOp.getIVs().size() != 1)
-      return;
-    // The linear IV is the loop IV's store address.
+      return success();
+    // Currently, frontends using `omp.simd` always generate a store from the
+    // `omp.loop_nest`'s IV to the corresponding iteration variable.
+    // We leverage this to find the linear iteration variable.
+    //
+    // TODO Add an attribute to `omp.loop_nest` that explicitly lists the
+    //      variables that correspond to the loop induction variables.
     BlockArgument arg = loopOp.getIVs().front();
     for (const Operation *user : arg.getUsers()) {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
+            if (linearLoopIV)
+              return simdOp.emitError(
+                  "Could not determine the linear variable associated with the "
+                  "loop nest induction variable");
             linearLoopIV = linearVar;
-            break;
           }
         }
       }
     }
+    return success();
   }
 
   // Emit IR for updating Linear variables
@@ -4858,7 +4867,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-  linearClauseProcessor.initLinearIV(simdOp);
+  if (linearClauseProcessor.initLinearIV(simdOp).failed())
+    return failure();
 
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index 6eff5accca138..234f404ff5690 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -130,6 +130,21 @@ llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr
 
 // -----
 
+llvm.func @simd_linear_ambiguous_iv(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr, %y : !llvm.ptr) {
+  // expected-error @below {{Could not determine the linear variable associated with the loop nest induction variable}}
+  // expected-error @below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd linear(%x : !llvm.ptr = %step : i32, %y : !llvm.ptr = %step : i32) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
+      llvm.store %iv, %y : i32, !llvm.ptr
+      omp.yield
+    }
+  } {linear_var_types = [i32, i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}

>From 7d19047506073f01cb6135659cb64561812a3a88 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Fri, 24 Jul 2026 11:57:10 -0300
Subject: [PATCH 6/6] Allow repeated stores to the same location

---
 .../Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  | 2 +-
 mlir/test/Target/LLVMIR/openmp-llvm.mlir                        | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b938572fe3720..ca320ea0b1453 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -207,7 +207,7 @@ class LinearClauseProcessor {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
-            if (linearLoopIV)
+            if (linearLoopIV && linearLoopIV != linearVar)
               return simdOp.emitError(
                   "Could not determine the linear variable associated with the "
                   "loop nest induction variable");
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 01fe0ae690955..de51468efdec1 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -768,6 +768,7 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK-NEXT: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
       llvm.store %iv, %x : i32, !llvm.ptr
       omp.yield
     }


From llvmlistbot at llvm.org  Fri Jul 24 07:59:45 2026
From: llvmlistbot at llvm.org (Leandro Lupori)
Date: Fri, 24 Jul 2026 07:59:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][OpenMP] Fix update of linear iteration
	variables (PR #183800)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/183800@github.com>
Message-ID: <6a637de1.0318b02e.530b7.d13d@mx.google.com>


https://github.com/luporl updated https://github.com/llvm/llvm-project/pull/183800

>From 49d7a2054c0b123a6783a49383c71066b7e1d1d9 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Fri, 27 Feb 2026 15:32:58 -0300
Subject: [PATCH 1/6] [mlir][OpenMP] Fix update of linear iteration variables

The final value of a linear iteration variable must be the loop
limit_value + step. Before this patch it was limit_value.

This fixes the second issue reported in #170784.
---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 129 ++++++++++++------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  15 +-
 2 files changed, 103 insertions(+), 41 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index deefa29c157f3..993cd89f74300 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -150,6 +150,37 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearFinalizationBB;
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
+  Value linearLoopIV;
+  Value linearLoopIVStart;
+
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
+                       llvm::Value *var, llvm::Value *varStart,
+                       llvm::Value *step, llvm::Value *iv) {
+    if (!iv->getType()->isIntegerTy())
+      llvm_unreachable("OpenMP loop induction variable must be an integer "
+                       "type");
+
+    if (varType->isIntegerTy()) {
+      // Integer path: normalize all arithmetic to linearVarType
+      iv = builder.CreateSExtOrTrunc(iv, varType);
+      step = builder.CreateSExtOrTrunc(step, varType);
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
+      builder.CreateStore(addInst, var);
+    } else if (varType->isFloatingPointTy()) {
+      // Float path: perform multiply in integer, then convert to float
+      step = builder.CreateSExtOrTrunc(step, iv->getType());
+
+      llvm::Value *mulInst = builder.CreateMul(iv, step);
+      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
+      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
+      builder.CreateStore(addInst, var);
+    } else {
+      llvm_unreachable(
+          "Linear variable must be of integer or floating-point type");
+    }
+  }
 
 public:
   // Register type for the linear variables
@@ -189,46 +220,63 @@ class LinearClauseProcessor {
     }
   }
 
+  // Find linear iteration variable and save it for later updates
+  void initLinearIV(omp::SimdOp simdOp) {
+    auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
+    // NOTE iteration variables can only be linear in non-nested loops.
+    if (loopOp.getIVs().size() != 1)
+      return;
+    // The linear IV is the loop IV's store address.
+    BlockArgument arg = loopOp.getIVs().front();
+    for (const Operation *user : arg.getUsers()) {
+      if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
+        for (Value linearVar : simdOp.getLinearVars()) {
+          if (linearVar == storeOp.getAddr()) {
+            linearLoopIV = linearVar;
+            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
+            break;
+          }
+        }
+      }
+    }
+  }
+
   // Emit IR for updating Linear variables
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
-                       llvm::Value *loopInductionVar) {
+  void updateLinearVars(llvm::IRBuilderBase &builder,
+                        llvm::BasicBlock *loopBody,
+                        llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::Type *linearVarType = linearVarTypes[index];
-      llvm::Value *iv = loopInductionVar;
-      llvm::Value *step = linearSteps[index];
-
-      if (!iv->getType()->isIntegerTy())
-        llvm_unreachable("OpenMP loop induction variable must be an integer "
-                         "type");
-
-      if (linearVarType->isIntegerTy()) {
-        // Integer path: normalize all arithmetic to linearVarType
-        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
-        step = builder.CreateSExtOrTrunc(step, linearVarType);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else if (linearVarType->isFloatingPointTy()) {
-        // Float path: perform multiply in integer, then convert to float
-        step = builder.CreateSExtOrTrunc(step, iv->getType());
-        llvm::Value *mulInst = builder.CreateMul(iv, step);
-
-        llvm::LoadInst *linearVarStart =
-            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
-        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
-        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
-        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
-      } else {
-        llvm_unreachable(
-            "Linear variable must be of integer or floating-point type");
-      }
+      llvm::LoadInst *linearVarStart = builder.CreateLoad(
+          linearVarTypes[index], linearPreconditionVars[index]);
+      updateLinearVar(builder, linearVarTypes[index],
+                      linearLoopBodyTemps[index], linearVarStart,
+                      linearSteps[index], loopInductionVar);
     }
   }
 
+  // Emit IR for updating linear iteration variables on loop exit
+  void updateLinearIV(llvm::IRBuilderBase &builder,
+                      LLVM::ModuleTranslation &moduleTranslation,
+                      llvm::Value *loopIV) {
+    if (!linearLoopIV)
+      return;
+    llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
+    llvm::Value *linearIVStart =
+        moduleTranslation.lookupValue(linearLoopIVStart);
+
+    // Find linearIV's index
+    size_t index;
+    for (index = 0; index < linearOrigVal.size(); index++)
+      if (linearIV == linearOrigVal[index])
+        break;
+    if (index == linearOrigVal.size())
+      return;
+
+    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
+                    linearIVStart, linearSteps[index], loopIV);
+  }
+
   // Linear variable finalization is conditional on the last logical iteration.
   // Create BB splits to manage the same.
   void splitLinearFiniBB(llvm::IRBuilderBase &builder,
@@ -3852,8 +3900,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4153,6 +4201,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
 
+  linearClauseProcessor.initLinearIV(simdOp);
+
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
     for (mlir::Attribute linearVarType : linearVarTypes)
@@ -4250,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
-                                          loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
+                                           loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4261,6 +4311,9 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
+                                       loopInfo->getIndVar());
+
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 38f10320bd3ae..45eb6ac1ef957 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,6 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
+// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -758,8 +759,16 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK: %[[ADD:.*]] = add i32 %[[LOAD]], %[[MUL]]
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
+
+// CHECK: omp.region.cont:
+// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
+// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
+// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
+// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
+// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
       omp.yield
     }
   } {linear_var_types = [i32]}
@@ -784,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -817,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
-// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
+// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group

>From 20ac4aef52312cd93944ad17f0fef0b126211102 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 19 May 2026 12:06:23 -0300
Subject: [PATCH 2/6] Fix "instruction does not dominate all uses" issue

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 104 +++++++++---------
 mlir/test/Target/LLVMIR/openmp-llvm.mlir      |  48 ++++++--
 2 files changed, 90 insertions(+), 62 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 993cd89f74300..aace5283e4362 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -151,36 +151,6 @@ class LinearClauseProcessor {
   llvm::BasicBlock *linearExitBB;
   llvm::BasicBlock *linearLastIterExitBB;
   Value linearLoopIV;
-  Value linearLoopIVStart;
-
-  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::Type *varType,
-                       llvm::Value *var, llvm::Value *varStart,
-                       llvm::Value *step, llvm::Value *iv) {
-    if (!iv->getType()->isIntegerTy())
-      llvm_unreachable("OpenMP loop induction variable must be an integer "
-                       "type");
-
-    if (varType->isIntegerTy()) {
-      // Integer path: normalize all arithmetic to linearVarType
-      iv = builder.CreateSExtOrTrunc(iv, varType);
-      step = builder.CreateSExtOrTrunc(step, varType);
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *addInst = builder.CreateAdd(varStart, mulInst);
-      builder.CreateStore(addInst, var);
-    } else if (varType->isFloatingPointTy()) {
-      // Float path: perform multiply in integer, then convert to float
-      step = builder.CreateSExtOrTrunc(step, iv->getType());
-
-      llvm::Value *mulInst = builder.CreateMul(iv, step);
-      llvm::Value *mulFp = builder.CreateSIToFP(mulInst, varType);
-      llvm::Value *addInst = builder.CreateFAdd(varStart, mulFp);
-      builder.CreateStore(addInst, var);
-    } else {
-      llvm_unreachable(
-          "Linear variable must be of integer or floating-point type");
-    }
-  }
 
 public:
   // Register type for the linear variables
@@ -233,7 +203,6 @@ class LinearClauseProcessor {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
             linearLoopIV = linearVar;
-            linearLoopIVStart = loopOp.getLoopLowerBounds().front();
             break;
           }
         }
@@ -242,28 +211,51 @@ class LinearClauseProcessor {
   }
 
   // Emit IR for updating Linear variables
-  void updateLinearVars(llvm::IRBuilderBase &builder,
-                        llvm::BasicBlock *loopBody,
-                        llvm::Value *loopInductionVar) {
+  void updateLinearVar(llvm::IRBuilderBase &builder, llvm::BasicBlock *loopBody,
+                       llvm::Value *loopInductionVar) {
     builder.SetInsertPoint(loopBody->getTerminator());
     for (size_t index = 0; index < linearPreconditionVars.size(); index++) {
-      llvm::LoadInst *linearVarStart = builder.CreateLoad(
-          linearVarTypes[index], linearPreconditionVars[index]);
-      updateLinearVar(builder, linearVarTypes[index],
-                      linearLoopBodyTemps[index], linearVarStart,
-                      linearSteps[index], loopInductionVar);
+      llvm::Type *linearVarType = linearVarTypes[index];
+      llvm::Value *iv = loopInductionVar;
+      llvm::Value *step = linearSteps[index];
+
+      if (!iv->getType()->isIntegerTy())
+        llvm_unreachable("OpenMP loop induction variable must be an integer "
+                         "type");
+
+      if (linearVarType->isIntegerTy()) {
+        // Integer path: normalize all arithmetic to linearVarType
+        iv = builder.CreateSExtOrTrunc(iv, linearVarType);
+        step = builder.CreateSExtOrTrunc(step, linearVarType);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+        llvm::Value *addInst = builder.CreateAdd(linearVarStart, mulInst);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else if (linearVarType->isFloatingPointTy()) {
+        // Float path: perform multiply in integer, then convert to float
+        step = builder.CreateSExtOrTrunc(step, iv->getType());
+        llvm::Value *mulInst = builder.CreateMul(iv, step);
+
+        llvm::LoadInst *linearVarStart =
+            builder.CreateLoad(linearVarType, linearPreconditionVars[index]);
+        llvm::Value *mulFp = builder.CreateSIToFP(mulInst, linearVarType);
+        llvm::Value *addInst = builder.CreateFAdd(linearVarStart, mulFp);
+        builder.CreateStore(addInst, linearLoopBodyTemps[index]);
+      } else {
+        llvm_unreachable(
+            "Linear variable must be of integer or floating-point type");
+      }
     }
   }
 
   // Emit IR for updating linear iteration variables on loop exit
   void updateLinearIV(llvm::IRBuilderBase &builder,
-                      LLVM::ModuleTranslation &moduleTranslation,
-                      llvm::Value *loopIV) {
+                      LLVM::ModuleTranslation &moduleTranslation) {
     if (!linearLoopIV)
       return;
     llvm::Value *linearIV = moduleTranslation.lookupValue(linearLoopIV);
-    llvm::Value *linearIVStart =
-        moduleTranslation.lookupValue(linearLoopIVStart);
 
     // Find linearIV's index
     size_t index;
@@ -273,8 +265,17 @@ class LinearClauseProcessor {
     if (index == linearOrigVal.size())
       return;
 
-    updateLinearVar(builder, linearVarTypes[index], linearLoopBodyTemps[index],
-                    linearIVStart, linearSteps[index], loopIV);
+    // Add one more step to the linear iteration variable
+    llvm::Type *varType = linearVarTypes[index];
+    llvm::Value *var = linearLoopBodyTemps[index];
+    llvm::Value *step = linearSteps[index];
+    if (!varType->isIntegerTy())
+      llvm_unreachable("Linear iteration variable must be of integer type");
+
+    step = builder.CreateSExtOrTrunc(step, varType);
+    llvm::Value *val = builder.CreateLoad(varType, var);
+    llvm::Value *addInst = builder.CreateAdd(val, step);
+    builder.CreateStore(addInst, var);
   }
 
   // Linear variable finalization is conditional on the last logical iteration.
@@ -3900,8 +3901,8 @@ convertOmpWsloop(Operation &opInst, llvm::IRBuilderBase &builder,
     if (failed(handleError(afterBarrierIP, *loopOp)))
       return failure();
     builder.restoreIP(*afterBarrierIP);
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
     linearClauseProcessor.splitLinearFiniBB(builder, loopInfo->getExit());
   }
 
@@ -4200,7 +4201,6 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-
   linearClauseProcessor.initLinearIV(simdOp);
 
   if (!simdOp.getLinearVars().empty()) {
@@ -4300,8 +4300,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
     linearClauseProcessor.initLinearVar(builder, moduleTranslation,
                                         loopInfo->getPreheader());
 
-    linearClauseProcessor.updateLinearVars(builder, loopInfo->getBody(),
-                                           loopInfo->getIndVar());
+    linearClauseProcessor.updateLinearVar(builder, loopInfo->getBody(),
+                                          loopInfo->getIndVar());
   }
   builder.SetInsertPoint(*regionBlock, (*regionBlock)->begin());
 
@@ -4311,9 +4311,7 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
                             : nullptr,
                         order, simdlen, safelen);
 
-  linearClauseProcessor.updateLinearIV(builder, moduleTranslation,
-                                       loopInfo->getIndVar());
-
+  linearClauseProcessor.updateLinearIV(builder, moduleTranslation);
   linearClauseProcessor.emitStoresForLinearVar(builder);
 
   // Check if this SIMD loop contains ordered regions
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 45eb6ac1ef957..51a12e23a0e5e 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -743,7 +743,7 @@ llvm.func @simd_simple(%lb : i64, %ub : i64, %step : i64, %arg0: !llvm.ptr) {
 llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // CHECK-LABEL: @simd_linear
-// CHECK-SAME: (i32 %[[LB:.*]], i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
+// CHECK-SAME: (i32 %{{.*}}, i32 %{{.*}}, i32 %[[STEP:.*]], ptr %[[X:.*]])
 
 // CHECK: %[[LINEAR_VAR:.*]] = alloca i32, align 4
 // CHECK: %[[LINEAR_RESULT:.*]] = alloca i32, align 4
@@ -761,11 +761,11 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4, !llvm.access.group !1
 
 // CHECK: omp.region.cont:
-// CHECK: %[[MUL:.*]] = mul i32 %omp_loop.iv, %[[STEP]]
-// CHECK: %[[ADD:.*]] = add i32 %[[LB]], %[[MUL]]
-// CHECK: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]], align 4
-// CHECK: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]], align 4
-// CHECK: store i32 %[[LOAD]], ptr %[[X]], align 4
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], %[[STEP]]
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[LOAD:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
       llvm.store %iv, %x : i32, !llvm.ptr
@@ -793,8 +793,8 @@ llvm.func @simd_linear_i64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // CHECK: omp_loop.body:
 // Verify type conversions: iv (i32) is extended to i64 before multiplication
-// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK: %[[IV_I64:.*]] = sext i32 %omp_loop.iv to i64
+// CHECK: %[[LOAD:.*]] = load i64, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // Verify multiplication and addition use consistent i64 types
 // CHECK: %[[MUL:.*]] = mul i64 %[[IV_I64]], {{.*}}
 // CHECK: %[[ADD:.*]] = add i64 %[[LOAD]], %[[MUL]]
@@ -826,8 +826,8 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 // CHECK: omp_loop.body:
 // Verify integer multiplication, load, and conversion to float
 // CHECK: mul i32 %omp_loop.iv
-// CHECK: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
-// CHECK-NEXT: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK: %[[MUL_INT:.*]] = mul i32 %omp_loop.iv, {{.*}}
+// CHECK-NEXT: %[[LOAD:.*]] = load double, ptr %[[LINEAR_VAR]], {{.*}}!llvm.access.group
 // CHECK-NEXT: %[[MUL_FP:.*]] = sitofp i32 %[[MUL_INT]] to double
 // CHECK-NEXT: %[[ADD:.*]] = fadd double %[[LOAD]], %[[MUL_FP]]
 // CHECK-NEXT: store double %[[ADD]], ptr %[[LINEAR_RESULT]], {{.*}}!llvm.access.group
@@ -841,6 +841,36 @@ llvm.func @simd_linear_f64_var_i32_step(%lb : i32, %ub : i32, %x : !llvm.ptr) {
 
 // -----
 
+// Test the update of omp.simd linear iteration variables, when nested inside
+// omp.wsloop.
+llvm.func @wsloop_simd_linear(%x : !llvm.ptr) {
+
+// CHECK-LABEL: @wsloop_simd_linear
+
+// CHECK: omp.wsloop.region:
+// CHECK: %[[LINEAR_VAR:.*]] = alloca i32
+// CHECK: %[[LINEAR_RESULT:.*]] = alloca i32
+
+// CHECK: omp.region.cont2:
+// CHECK: %[[VAL:.*]] = load i32, ptr %[[LINEAR_RESULT]]
+// CHECK-NEXT: %[[ADD:.*]] = add i32 %[[VAL]], 25
+// CHECK-NEXT: store i32 %[[ADD]], ptr %[[LINEAR_RESULT]]
+  %lb = llvm.mlir.constant(1 : i32) : i32
+  %ub = llvm.mlir.constant(100 : i32) : i32
+  %step = llvm.mlir.constant(25 : i32) : i32
+  omp.wsloop {
+    omp.simd linear(%x : !llvm.ptr = %step : i32) {
+      omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+        llvm.store %iv, %x : i32, !llvm.ptr
+        omp.yield
+      }
+    } {linear_var_types = [i32], omp.composite}
+  } {omp.composite}
+  llvm.return
+}
+
+// -----
+
 // CHECK-LABEL: @simd_simple_multiple
 llvm.func @simd_simple_multiple(%lb1 : i64, %ub1 : i64, %step1 : i64, %lb2 : i64, %ub2 : i64, %step2 : i64, %arg0: !llvm.ptr, %arg1: !llvm.ptr) {
   omp.simd {

>From 030a957a941fb50d15c99f9d0031e26019b53992 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 09:58:04 -0300
Subject: [PATCH 3/6] Verify that linear variables don't appear in other DSA
 clauses

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp     | 10 ++++++++++
 mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir | 14 ++++++++++++++
 mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir | 12 +++++-------
 3 files changed, 29 insertions(+), 7 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index db5fd8f2e3230..b328e15313538 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -3363,6 +3363,16 @@ LogicalResult SimdOp::verify() {
   if (getLinearVars().size() &&
       getLinearVarTypes().value().size() != getLinearVars().size())
     return emitError() << "Ill-formed type attributes for linear variables";
+
+  llvm::DenseSet<Value> privateVars(llvm::from_range, getPrivateVars());
+  llvm::DenseSet<Value> reductionVars(llvm::from_range, getReductionVars());
+  // TODO Check lastprivate vars when their support is added to SimdOp.
+  for (Value var : getLinearVars()) {
+    if (privateVars.contains(var) || reductionVars.contains(var))
+      return emitOpError()
+             << "linear variables cannot appear in other data-sharing clauses";
+  }
+
   return success();
 }
 
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index d9459ace54422..7e2e0061f80be 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -116,6 +116,20 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 
 // -----
 
+omp.private {type = private} @i_private_i32 : i32
+llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr) {
+  // expected-error @below {{linear variables cannot appear in other data-sharing clauses}}
+  omp.simd linear(%i : !llvm.ptr = %step : i32)
+           private(@i_private_i32 %i -> %priv_i : !llvm.ptr) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      omp.yield
+    }
+  } {linear_var_types = [i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}
diff --git a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
index 8da1fab1f48af..a84c51f08daa7 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @simd_ordered_linear
   llvm.func @simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -38,14 +36,14 @@ module {
     // CHECK: %.linear_var = alloca i32
     // CHECK: %.linear_result = alloca i32
 
-    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c10_i32) inclusive step (%c1_i32) {
         // CHECK: omp.loop_nest.region:
         // CHECK: load i32, ptr %.linear_result
-        llvm.store %iv, %arg0 : i32, !llvm.ptr
+        llvm.store %iv, %i : i32, !llvm.ptr
 
         // Compute a[i] = b[i] * 10
-        %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_val = llvm.load %i : !llvm.ptr -> i32
         %i_idx = llvm.sext %i_val : i32 to i64
         %i_off = llvm.sub %i_idx, %c1_i64 : i64
         %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -58,7 +56,7 @@ module {
         omp.ordered.region par_level_simd {
           // CHECK: omp.ordered.region:
           // CHECK: load i32, ptr %.linear_result
-          %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_ord = llvm.load %i : !llvm.ptr -> i32
           %i_ord_idx = llvm.sext %i_ord : i32 to i64
           %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
           %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -69,7 +67,7 @@ module {
         // Compute c[i] = a[i] * 2 (code after ordered region)
         // CHECK: omp_region.finalize:
         // CHECK: load i32, ptr %.linear_result
-        %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+        %i_post = llvm.load %i : !llvm.ptr -> i32
         %i_post_idx = llvm.sext %i_post : i32 to i64
         %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
         %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From 2f8b6c3b6f7ca958505c6e6d7cc41ec53cb0fe0d Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Tue, 16 Jun 2026 11:10:29 -0300
Subject: [PATCH 4/6] Update mlir tests

---
 mlir/test/Target/LLVMIR/openmp-simd-guided.mlir      |  3 +--
 .../Target/LLVMIR/openmp-wsloop-simd-ordered.mlir    | 12 +++++-------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
index b1f18b9f85417..496669c5b90d5 100644
--- a/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-simd-guided.mlir
@@ -1,7 +1,6 @@
 // Ensure that schedule can be used with the guided kind-type and simd construct
 // RUN: mlir-translate -mlir-to-llvmir -split-input-file %s | FileCheck %s
 
-omp.private {type = private} @_QFEi_private_i32 : i32
 llvm.func @test_simd_guided() {
   %0 = llvm.mlir.constant (1 : i64) : i64
   %c1_i32 = llvm.mlir.constant (1 : i32) : i32
@@ -10,7 +9,7 @@ llvm.func @test_simd_guided() {
   %c4_i32 = llvm.mlir.constant (4 : i32) : i32
   %1 = llvm.alloca %0 x i32 {bindc_name = "i"} : (i64) -> !llvm.ptr
   omp.wsloop schedule(guided = %c4_i32 : i32, simd) {
-    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) private(@_QFEi_private_i32 %1 -> %arg0 : !llvm.ptr) {
+    omp.simd linear(%1 : !llvm.ptr = %c1_i32 : i32) {
       omp.loop_nest (%arg1) : i32 = (%c0_i32) to (%c64_i32) inclusive step (%c1_i32) {
         omp.yield
       }
diff --git a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
index beb300c23651f..8d5b89ecbd00a 100644
--- a/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-wsloop-simd-ordered.mlir
@@ -17,8 +17,6 @@
 // !$omp end do simd
 
 module {
-  omp.private {type = private} @i_private_i32 : i32
-
   // CHECK-LABEL: define void @wsloop_simd_ordered_linear
   llvm.func @wsloop_simd_ordered_linear() {
     %c0_i64 = llvm.mlir.constant(0 : i64) : i64
@@ -39,14 +37,14 @@ module {
     // CHECK: %.linear_result = alloca i32
 
     omp.wsloop ordered(0) {
-      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) private(@i_private_i32 %i -> %arg0 : !llvm.ptr) {
+      omp.simd linear(%i : !llvm.ptr = %c1_i32 : i32) {
         omp.loop_nest (%iv) : i32 = (%c1_i32) to (%c100_i32) inclusive step (%c1_i32) {
           // CHECK: omp.loop_nest.region:
           // CHECK: load i32, ptr %.linear_result
-          llvm.store %iv, %arg0 : i32, !llvm.ptr
+          llvm.store %iv, %i : i32, !llvm.ptr
 
           // Compute a[i] = b[i] * 10
-          %i_val = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_val = llvm.load %i : !llvm.ptr -> i32
           %i_idx = llvm.sext %i_val : i32 to i64
           %i_off = llvm.sub %i_idx, %c1_i64 : i64
           %b_ptr = llvm.getelementptr %b[%i_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -59,7 +57,7 @@ module {
           omp.ordered.region par_level_simd {
             // CHECK: omp.ordered.region:
             // CHECK: load i32, ptr %.linear_result
-            %i_ord = llvm.load %arg0 : !llvm.ptr -> i32
+            %i_ord = llvm.load %i : !llvm.ptr -> i32
             %i_ord_idx = llvm.sext %i_ord : i32 to i64
             %i_ord_off = llvm.sub %i_ord_idx, %c1_i64 : i64
             %a_ord_ptr = llvm.getelementptr %a[%i_ord_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32
@@ -71,7 +69,7 @@ module {
           // Compute c[i] = a[i] * 2 (code after ordered region)
           // CHECK: omp_region.finalize:
           // CHECK: load i32, ptr %.linear_result
-          %i_post = llvm.load %arg0 : !llvm.ptr -> i32
+          %i_post = llvm.load %i : !llvm.ptr -> i32
           %i_post_idx = llvm.sext %i_post : i32 to i64
           %i_post_off = llvm.sub %i_post_idx, %c1_i64 : i64
           %a_post_ptr = llvm.getelementptr %a[%i_post_off] : (!llvm.ptr, i64) -> !llvm.ptr, i32

>From b671d6cf6efa8413abd061ac101b1ea14137ff5f Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Thu, 23 Jul 2026 16:48:58 -0300
Subject: [PATCH 5/6] Report an error if the linear variable can't be
 determined unambiguously

---
 .../OpenMP/OpenMPToLLVMIRTranslation.cpp      | 20 ++++++++++++++-----
 .../Target/LLVMIR/openmp-llvm-invalid.mlir    | 15 ++++++++++++++
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index 2a246ebf98b3c..b938572fe3720 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -191,23 +191,32 @@ class LinearClauseProcessor {
   }
 
   // Find linear iteration variable and save it for later updates
-  void initLinearIV(omp::SimdOp simdOp) {
+  LogicalResult initLinearIV(omp::SimdOp simdOp) {
     auto loopOp = cast<omp::LoopNestOp>(simdOp.getWrappedLoop());
     // NOTE iteration variables can only be linear in non-nested loops.
     if (loopOp.getIVs().size() != 1)
-      return;
-    // The linear IV is the loop IV's store address.
+      return success();
+    // Currently, frontends using `omp.simd` always generate a store from the
+    // `omp.loop_nest`'s IV to the corresponding iteration variable.
+    // We leverage this to find the linear iteration variable.
+    //
+    // TODO Add an attribute to `omp.loop_nest` that explicitly lists the
+    //      variables that correspond to the loop induction variables.
     BlockArgument arg = loopOp.getIVs().front();
     for (const Operation *user : arg.getUsers()) {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
+            if (linearLoopIV)
+              return simdOp.emitError(
+                  "Could not determine the linear variable associated with the "
+                  "loop nest induction variable");
             linearLoopIV = linearVar;
-            break;
           }
         }
       }
     }
+    return success();
   }
 
   // Emit IR for updating Linear variables
@@ -4858,7 +4867,8 @@ convertOmpSimd(Operation &opInst, llvm::IRBuilderBase &builder,
 
   // Initialize linear variables and linear step
   LinearClauseProcessor linearClauseProcessor;
-  linearClauseProcessor.initLinearIV(simdOp);
+  if (linearClauseProcessor.initLinearIV(simdOp).failed())
+    return failure();
 
   if (!simdOp.getLinearVars().empty()) {
     auto linearVarTypes = simdOp.getLinearVarTypes().value();
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
index 6eff5accca138..234f404ff5690 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm-invalid.mlir
@@ -130,6 +130,21 @@ llvm.func @simd_linear_private(%lb : i32, %ub : i32, %step : i32, %i : !llvm.ptr
 
 // -----
 
+llvm.func @simd_linear_ambiguous_iv(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr, %y : !llvm.ptr) {
+  // expected-error @below {{Could not determine the linear variable associated with the loop nest induction variable}}
+  // expected-error @below {{LLVM Translation failed for operation: omp.simd}}
+  omp.simd linear(%x : !llvm.ptr = %step : i32, %y : !llvm.ptr = %step : i32) {
+    omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
+      llvm.store %iv, %y : i32, !llvm.ptr
+      omp.yield
+    }
+  } {linear_var_types = [i32, i32]}
+  llvm.return
+}
+
+// -----
+
 module attributes {llvm.target_triple = "amdgcn-amd-amdhsa", omp.is_target_device = true} {
   llvm.func @host_op_in_device(%arg0 : !llvm.ptr) {
     // expected-error @below {{unsupported host op found in device}}

>From 7d19047506073f01cb6135659cb64561812a3a88 Mon Sep 17 00:00:00 2001
From: Leandro Lupori <leandro.lupori at linaro.org>
Date: Fri, 24 Jul 2026 11:57:10 -0300
Subject: [PATCH 6/6] Allow repeated stores to the same location

---
 .../Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp  | 2 +-
 mlir/test/Target/LLVMIR/openmp-llvm.mlir                        | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
index b938572fe3720..ca320ea0b1453 100644
--- a/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
+++ b/mlir/lib/Target/LLVMIR/Dialect/OpenMP/OpenMPToLLVMIRTranslation.cpp
@@ -207,7 +207,7 @@ class LinearClauseProcessor {
       if (auto storeOp = dyn_cast<LLVM::StoreOp>(user)) {
         for (Value linearVar : simdOp.getLinearVars()) {
           if (linearVar == storeOp.getAddr()) {
-            if (linearLoopIV)
+            if (linearLoopIV && linearLoopIV != linearVar)
               return simdOp.emitError(
                   "Could not determine the linear variable associated with the "
                   "loop nest induction variable");
diff --git a/mlir/test/Target/LLVMIR/openmp-llvm.mlir b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
index 01fe0ae690955..de51468efdec1 100644
--- a/mlir/test/Target/LLVMIR/openmp-llvm.mlir
+++ b/mlir/test/Target/LLVMIR/openmp-llvm.mlir
@@ -768,6 +768,7 @@ llvm.func @simd_linear(%lb : i32, %ub : i32, %step : i32, %x : !llvm.ptr) {
 // CHECK-NEXT: store i32 %[[LOAD]], ptr %[[X]], align 4
   omp.simd linear(%x : !llvm.ptr = %step : i32) {
     omp.loop_nest (%iv) : i32 = (%lb) to (%ub) step (%step) {
+      llvm.store %iv, %x : i32, !llvm.ptr
       llvm.store %iv, %x : i32, !llvm.ptr
       omp.yield
     }


From llvmlistbot at llvm.org  Fri Jul 24 08:31:56 2026
From: llvmlistbot at llvm.org (Chuanqi Xu)
Date: Fri, 24 Jul 2026 08:31:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] [vector] Allow element/broadcast
 re-order for FMA if the source is a vector (PR #211208)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211208@github.com>
Message-ID: <6a63856c.e78280df.38cab1.0a60@mx.google.com>


https://github.com/ChuanqiXu9 updated https://github.com/llvm/llvm-project/pull/211208

>From ca4a8c504c93d837cfa053a1b86b1cbd0167a728 Mon Sep 17 00:00:00 2001
From: "yedeng.yd" <yedeng.yd at alibaba-inc.com>
Date: Wed, 22 Jul 2026 16:56:46 +0800
Subject: [PATCH 1/2] [mlir] [vector] Reorder element/broadcast for FMA if the
 source is not scalar

The diagnostic says the transform is not performed as the source may be
a scalar. But we can make it if we can check the source is not a scalar.

AI assisted.
---
 .../Vector/Transforms/VectorTransforms.cpp    | 17 ++--
 mlir/test/Dialect/Vector/vector-sink.mlir     | 81 +++++++++++++++++--
 2 files changed, 85 insertions(+), 13 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index 17e6ad640cada..e1d9bc8d0f4a0 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -1043,12 +1043,6 @@ struct ReorderElementwiseOpsOnBroadcast final
           op, "Op doesn't have ElementwiseMappableTraits");
     if (op->getNumOperands() == 0)
       return failure();
-    if (isa<vector::FMAOp>(op)) {
-      return rewriter.notifyMatchFailure(
-          op,
-          "Op only accepts vector types - not supported as broadcast source "
-          "might be a scalar");
-    }
 
     Type resultElemType = resultType.getElementType();
 
@@ -1068,6 +1062,17 @@ struct ReorderElementwiseOpsOnBroadcast final
     Type unbroadcastResultType =
         cloneOrReplace(broadcastSource.getType(), resultElemType);
 
+    // Some ops, e.g. `vector.fma`, only accept vector types. For such ops the
+    // reordering is only possible when the broadcast source is a vector as
+    // well; sinking past a broadcast from a scalar would create an invalid op.
+    // TODO: Support scalar sources for `vector.fma` by rewriting to
+    // `math.fma`.
+    if (isa<vector::FMAOp>(op) && !isa<VectorType>(unbroadcastResultType)) {
+      return rewriter.notifyMatchFailure(
+          op, "Op only accepts vector types, but the broadcast source is a "
+              "scalar");
+    }
+
     // Make sure that all operands are broadcast from identically-shaped types:
     //  * scalar (`vector.broadcast`), or
     //  * vector (`vector.broadcast`).
diff --git a/mlir/test/Dialect/Vector/vector-sink.mlir b/mlir/test/Dialect/Vector/vector-sink.mlir
index 69fba88a14048..b52828419b10a 100644
--- a/mlir/test/Dialect/Vector/vector-sink.mlir
+++ b/mlir/test/Dialect/Vector/vector-sink.mlir
@@ -195,17 +195,18 @@ func.func @source_and_result_mismatch(%arg0 : f32) -> vector<1xi1> {
 
 // -----
 
-// vector.fma only supports vectors - currently it's not possible to replace this with e.g.:
+// vector.fma only supports vectors - when the broadcast source is a scalar,
+// currently it's not possible to replace this with e.g.:
 //    %scalar_res = vector.fma %scalar_1, %scalar2
 //    %vec_res = vector.broadcast %scalar_res
 //
-// TODO: It should be possible to support this case
+// TODO: It should be possible to support this case by rewriting to math.fma
 
-// CHECK-LABEL: func.func @negative_op_only_supports_vectors
-  //     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
-  //     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
-  //     CHECK:   return %[[RESULT]]
-func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
+// CHECK-LABEL: func.func @negative_fma_scalar_broadcast_source
+//     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
+//     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]
+//     CHECK:   return %[[RESULT]]
+func.func @negative_fma_scalar_broadcast_source(%arg0 : f32) -> vector<1xf32> {
   %0 = vector.broadcast %arg0 : f32 to vector<1xf32>
   %1 = vector.fma %0, %0, %0 : vector<1xf32>
   return %1 : vector<1xf32>
@@ -213,6 +214,72 @@ func.func @negative_op_only_supports_vectors(%arg0 : f32) -> vector<1xf32> {
 
 // -----
 
+// CHECK-LABEL: func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: f32, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[BCAST0:.*]] = vector.broadcast %[[ARG0]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[BCAST1:.*]] = vector.broadcast %[[ARG1]] : f32 to vector<3x4xf32>
+//       CHECK:   %[[BCAST2:.*]] = vector.broadcast %[[ARG2]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[BCAST0]], %[[BCAST1]], %[[BCAST2]] : vector<3x4xf32>
+//       CHECK:   return %[[FMA]] : vector<3x4xf32>
+func.func @negative_fma_mixed_scalar_and_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: f32, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : f32 to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// vector.fma only supports vector operands, hence the broadcast can only be
+// sunk when the broadcast source is a vector as well.
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>, %[[ARG2:.*]]: vector<4xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source(%arg0: vector<4xf32>, %arg1: vector<4xf32>, %arg2: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %2 = vector.broadcast %arg2 : vector<4xf32> to vector<3x4xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x4xf32>
+  return %3 : vector<3x4xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_scalable(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<[4]xf32>, %[[ARG1:.*]]: vector<[4]xf32>, %[[ARG2:.*]]: vector<[4]xf32>)
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[ARG2]] : vector<[4]xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<[4]xf32> to vector<3x[4]xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x[4]xf32>
+func.func @fma_vector_broadcast_source_scalable(%arg0: vector<[4]xf32>, %arg1: vector<[4]xf32>, %arg2: vector<[4]xf32>) -> vector<3x[4]xf32> {
+  %0 = vector.broadcast %arg0 : vector<[4]xf32> to vector<3x[4]xf32>
+  %1 = vector.broadcast %arg1 : vector<[4]xf32> to vector<3x[4]xf32>
+  %2 = vector.broadcast %arg2 : vector<[4]xf32> to vector<3x[4]xf32>
+  %3 = vector.fma %0, %1, %2 : vector<3x[4]xf32>
+  return %3 : vector<3x[4]xf32>
+}
+
+// -----
+
+// CHECK-LABEL: func.func @fma_vector_broadcast_source_and_splat_const(
+//  CHECK-SAME:   %[[ARG0:.*]]: vector<4xf32>, %[[ARG1:.*]]: vector<4xf32>)
+//       CHECK:   %[[NEW_CST:.*]] = arith.constant dense<2.000000e+00> : vector<4xf32>
+//       CHECK:   %[[FMA:.*]] = vector.fma %[[ARG0]], %[[ARG1]], %[[NEW_CST]] : vector<4xf32>
+//       CHECK:   %[[BCAST:.*]] = vector.broadcast %[[FMA]] : vector<4xf32> to vector<3x4xf32>
+//       CHECK:   return %[[BCAST]] : vector<3x4xf32>
+func.func @fma_vector_broadcast_source_and_splat_const(%arg0: vector<4xf32>, %arg1: vector<4xf32>) -> vector<3x4xf32> {
+  %0 = vector.broadcast %arg0 : vector<4xf32> to vector<3x4xf32>
+  %1 = vector.broadcast %arg1 : vector<4xf32> to vector<3x4xf32>
+  %cst = arith.constant dense<2.0> : vector<3x4xf32>
+  %2 = vector.fma %0, %1, %cst : vector<3x4xf32>
+  return %2 : vector<3x4xf32>
+}
+
+// -----
+
 // CHECK-LABEL:   func.func @broadcast_scalar_and_splat_const(
 // CHECK-SAME:     %[[ARG_0:.*]]: index) -> vector<1x4xindex> {
 // CHECK:           %[[NEW_CST:.*]] = arith.constant 2 : index

>From 9c09d4fd7379849ae6d8ed5f0fdf5c97c6cf7f6a Mon Sep 17 00:00:00 2001
From: "yedeng.yd" <yedeng.yd at alibaba-inc.com>
Date: Fri, 24 Jul 2026 23:31:28 +0800
Subject: [PATCH 2/2] Address comments

---
 mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp | 4 ++--
 mlir/test/Dialect/Vector/vector-sink.mlir               | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
index e1d9bc8d0f4a0..d7beb08d3333c 100644
--- a/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/VectorTransforms.cpp
@@ -1065,8 +1065,8 @@ struct ReorderElementwiseOpsOnBroadcast final
     // Some ops, e.g. `vector.fma`, only accept vector types. For such ops the
     // reordering is only possible when the broadcast source is a vector as
     // well; sinking past a broadcast from a scalar would create an invalid op.
-    // TODO: Support scalar sources for `vector.fma` by rewriting to
-    // `math.fma`.
+    // TODO: It may be better to support scalar sources by promoting the scalar
+    // to a single element vector.
     if (isa<vector::FMAOp>(op) && !isa<VectorType>(unbroadcastResultType)) {
       return rewriter.notifyMatchFailure(
           op, "Op only accepts vector types, but the broadcast source is a "
diff --git a/mlir/test/Dialect/Vector/vector-sink.mlir b/mlir/test/Dialect/Vector/vector-sink.mlir
index b52828419b10a..6634f5d20cc85 100644
--- a/mlir/test/Dialect/Vector/vector-sink.mlir
+++ b/mlir/test/Dialect/Vector/vector-sink.mlir
@@ -200,8 +200,8 @@ func.func @source_and_result_mismatch(%arg0 : f32) -> vector<1xi1> {
 //    %scalar_res = vector.fma %scalar_1, %scalar2
 //    %vec_res = vector.broadcast %scalar_res
 //
-// TODO: It should be possible to support this case by rewriting to math.fma
-
+// TODO: It may be better to support this case by promoting the scalar
+// to a single element vector.
 // CHECK-LABEL: func.func @negative_fma_scalar_broadcast_source
 //     CHECK:   %[[BROADCAST:.+]] = vector.broadcast
 //     CHECK:   %[[RESULT:.+]] = vector.fma %[[BROADCAST]]


From llvmlistbot at llvm.org  Fri Jul 24 08:40:55 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 24 Jul 2026 08:40:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
Message-ID: <llvm.org/llvm/llvm-project/pull/211833@github.com>


https://github.com/ioghiban created https://github.com/llvm/llvm-project/pull/211833

This tests are the first step towards monitoring the recently enabled TOSA -> EmitC path that ML optimization models will switch to, for a fully in-tree, framework free alternative to compile them for inference.

For simplicity, yet testing the core functionality, these tests don't use production models, but rather in-tree mock models, generated by these scripts:
[llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py) /
[llvm/lib/Analysis/models/gen-inline-oz-test-model.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/gen-inline-oz-test-model.py)
\+ [llvm/lib/Analysis/models/saved-model-to-tflite.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/saved-model-to-tflite.py) + [tosa-converter-for-tflite](https://www.piwheels.org/project/tosa-converter-for-tflite/)

> Note: `saved-model-to-tflite.py` renames the TFLite converted SavedModel function to `main()`, which causes `clang` errors. Changing this behaviour requires interventions in Tensorflow, which are out of scope. Therefore, the functions were manually renamed.

> Note: The includes passed by argument to `clang` should have been handled by EmitC. This only becomes a problem when compiling these models standalone. Embedding the models in LLVM does not cause issues. Nonetheless, future work will ensure `ConvertToEmitC` includes required libraries for these ops. 

>From b15f5500e791deff24a06b0c9b525bbb4b15b3e8 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 24 Jul 2026 17:21:29 +0200
Subject: [PATCH] Add TOSA -> EmitC tests for MLGO mock models

---
 .../EmitC/inline-oz-test-model-tosa.mlir      | 57 +++++++++++++++++++
 .../regalloc-eviction-test-model-tosa.mlir    | 57 +++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
 create mode 100644 mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir

diff --git a/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
new file mode 100644
index 0000000000000..ff5de17ba06a0
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stdint.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-inline-oz-test-model.py  + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: return type of 'main' is not 'int'
+///   error: too many parameters (40) for 'main': must be 0, 2, or 3
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+///   error: second parameter of 'main' (argument array) must be of type 'char **'
+///   error: third parameter of 'main' (environment) must be of type 'char **'
+
+// CHECK-LABEL: @mock_model
+// CHECK:         emitc.constant
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<i64> {tf_saved_model.index_path = ["callsite_cost"]}, %arg1: tensor<i64> {tf_saved_model.index_path = ["is_multiple_blocks"]}, %arg2: tensor<i64> {tf_saved_model.index_path = ["caller_conditionally_executed_blocks"]}, %arg3: tensor<i64> {tf_saved_model.index_path = ["cold_cc_penalty"]}, %arg4: tensor<i64> {tf_saved_model.index_path = ["callee_conditionally_executed_blocks"]}, %arg5: tensor<i64> {tf_saved_model.index_path = ["callee_users"]}, %arg6: tensor<i64> {tf_saved_model.index_path = ["callee_basic_block_count"]}, %arg7: tensor<i64> {tf_saved_model.index_path = ["nr_ctant_params"]}, %arg8: tensor<i64> {tf_saved_model.index_path = ["load_relative_intrinsic"]}, %arg9: tensor<i64> {tf_saved_model.index_path = ["jump_table_penalty"]}, %arg10: tensor<i64> {tf_saved_model.index_path = ["unsimplified_common_instructions"]}, %arg11: tensor<i64> {tf_saved_model.index_path = ["indirect_call_penalty"]}, %arg12: tensor<i64> {tf_saved_model.index_path = ["load_elimination"]}, %arg13: tensor<i64> {tf_saved_model.index_path = ["is_caller_avail_external"]}, %arg14: tensor<i64> {tf_saved_model.index_path = ["call_penalty"]}, %arg15: tensor<i64> {tf_saved_model.index_path = ["cost_estimate"]}, %arg16: tensor<i64> {tf_saved_model.index_path = ["case_cluster_penalty"]}, %arg17: tensor<i64> {tf_saved_model.index_path = ["node_count"]}, %arg18: tensor<i64> {tf_saved_model.index_path = ["call_argument_setup"]}, %arg19: tensor<i64> {tf_saved_model.index_path = ["sroa_savings"]}, %arg20: tensor<i64> {tf_saved_model.index_path = ["lowered_call_arg_setup"]}, %arg21: tensor<i64> {tf_saved_model.index_path = ["threshold"]}, %arg22: tensor<i64> {tf_saved_model.index_path = ["dead_blocks"]}, %arg23: tensor<i64> {tf_saved_model.index_path = ["constant_args"]}, %arg24: tensor<i64> {tf_saved_model.index_path = ["sroa_losses"]}, %arg25: tensor<i64> {tf_saved_model.index_path = ["simplified_instructions"]}, %arg26: tensor<i64> {tf_saved_model.index_path = ["num_loops"]}, %arg27: tensor<i32> {tf_saved_model.index_path = ["step_type"]}, %arg28: tensor<i64> {tf_saved_model.index_path = ["edge_count"]}, %arg29: tensor<i64> {tf_saved_model.index_path = ["is_callee_avail_external"]}, %arg30: tensor<i64> {tf_saved_model.index_path = ["nested_inlines"]}, %arg31: tensor<i64> {tf_saved_model.index_path = ["caller_basic_block_count"]}, %arg32: tensor<i64> {tf_saved_model.index_path = ["last_call_to_static_bonus"]}, %arg33: tensor<i64> {tf_saved_model.index_path = ["nested_inline_cost_estimate"]}, %arg34: tensor<i64> {tf_saved_model.index_path = ["callsite_height"]}, %arg35: tensor<i64> {tf_saved_model.index_path = ["constant_offset_ptr_args"]}, %arg36: tensor<i64> {tf_saved_model.index_path = ["switch_penalty"]}, %arg37: tensor<f32> {tf_saved_model.index_path = ["discount"]}, %arg38: tensor<i64> {tf_saved_model.index_path = ["caller_users"]}, %arg39: tensor<f32> {tf_saved_model.index_path = ["reward"]}) -> (tensor<i64> {tf_saved_model.index_path = ["inlining_decision"]}) attributes {tf.entry_function = {inputs = "action_callsite_cost:0,action_is_multiple_blocks:0,action_caller_conditionally_executed_blocks:0,action_cold_cc_penalty:0,action_callee_conditionally_executed_blocks:0,action_callee_users:0,action_callee_basic_block_count:0,action_nr_ctant_params:0,action_load_relative_intrinsic:0,action_jump_table_penalty:0,action_unsimplified_common_instructions:0,action_indirect_call_penalty:0,action_load_elimination:0,action_is_caller_avail_external:0,action_call_penalty:0,action_cost_estimate:0,action_case_cluster_penalty:0,action_node_count:0,action_call_argument_setup:0,action_sroa_savings:0,action_lowered_call_arg_setup:0,action_threshold:0,action_dead_blocks:0,action_constant_args:0,action_sroa_losses:0,action_simplified_instructions:0,action_num_loops:0,action_step_type:0,action_edge_count:0,action_is_callee_avail_external:0,action_nested_inlines:0,action_caller_basic_block_count:0,action_last_call_to_static_bonus:0,action_nested_inline_cost_estimate:0,action_callsite_height:0,action_constant_offset_ptr_args:0,action_switch_penalty:0,action_discount:0,action_caller_users:0,action_reward:0", outputs = "PartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = "tosa.const"() <{values = dense<1> : tensor<i64>}> : () -> tensor<i64>
+    return %0 : tensor<i64>
+  }
+}
diff --git a/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
new file mode 100644
index 0000000000000..eafbbe814c96a
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stddef.h -include stdint.h -include stdlib.h\
+// DEFINE:   -include stdbool.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+// CHECK-LABEL: @mock_model
+// CHECK:         for
+// CHECK:         cmp gt
+// CHECK:         conditional
+// CHECK:         assign
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "1.9.0\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<33xi64> {tf_saved_model.index_path = ["mask"]}) -> (tensor<i64> {tf_saved_model.index_path = ["index_to_evict"]}) attributes {tf.entry_function = {inputs = "action_mask:0", outputs = "StatefulPartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = tosa.cast %arg0 : (tensor<33xi64>) -> tensor<33xi32>
+    %1 = tosa.argmax %0 {axis = 0 : i32} : (tensor<33xi32>) -> tensor<i64>
+    return %1 : tensor<i64>
+  }
+}


From llvmlistbot at llvm.org  Fri Jul 24 08:41:30 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 08:41:30 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211833@github.com>
Message-ID: <6a6387aa.458f572a.dff0b.cdcd@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-emitc

Author: ioana ghiban (ioghiban)

<details>
<summary>Changes</summary>

This tests are the first step towards monitoring the recently enabled TOSA -&gt; EmitC path that ML optimization models will switch to, for a fully in-tree, framework free alternative to compile them for inference.

For simplicity, yet testing the core functionality, these tests don't use production models, but rather in-tree mock models, generated by these scripts:
[llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py) /
[llvm/lib/Analysis/models/gen-inline-oz-test-model.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/gen-inline-oz-test-model.py)
\+ [llvm/lib/Analysis/models/saved-model-to-tflite.py](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Analysis/models/saved-model-to-tflite.py) + [tosa-converter-for-tflite](https://www.piwheels.org/project/tosa-converter-for-tflite/)

&gt; Note: `saved-model-to-tflite.py` renames the TFLite converted SavedModel function to `main()`, which causes `clang` errors. Changing this behaviour requires interventions in Tensorflow, which are out of scope. Therefore, the functions were manually renamed.

&gt; Note: The includes passed by argument to `clang` should have been handled by EmitC. This only becomes a problem when compiling these models standalone. Embedding the models in LLVM does not cause issues. Nonetheless, future work will ensure `ConvertToEmitC` includes required libraries for these ops. 

---
Full diff: https://github.com/llvm/llvm-project/pull/211833.diff


2 Files Affected:

- (added) mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir (+57) 
- (added) mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir (+57) 


``````````diff
diff --git a/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
new file mode 100644
index 0000000000000..ff5de17ba06a0
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stdint.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-inline-oz-test-model.py  + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: return type of 'main' is not 'int'
+///   error: too many parameters (40) for 'main': must be 0, 2, or 3
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+///   error: second parameter of 'main' (argument array) must be of type 'char **'
+///   error: third parameter of 'main' (environment) must be of type 'char **'
+
+// CHECK-LABEL: @mock_model
+// CHECK:         emitc.constant
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<i64> {tf_saved_model.index_path = ["callsite_cost"]}, %arg1: tensor<i64> {tf_saved_model.index_path = ["is_multiple_blocks"]}, %arg2: tensor<i64> {tf_saved_model.index_path = ["caller_conditionally_executed_blocks"]}, %arg3: tensor<i64> {tf_saved_model.index_path = ["cold_cc_penalty"]}, %arg4: tensor<i64> {tf_saved_model.index_path = ["callee_conditionally_executed_blocks"]}, %arg5: tensor<i64> {tf_saved_model.index_path = ["callee_users"]}, %arg6: tensor<i64> {tf_saved_model.index_path = ["callee_basic_block_count"]}, %arg7: tensor<i64> {tf_saved_model.index_path = ["nr_ctant_params"]}, %arg8: tensor<i64> {tf_saved_model.index_path = ["load_relative_intrinsic"]}, %arg9: tensor<i64> {tf_saved_model.index_path = ["jump_table_penalty"]}, %arg10: tensor<i64> {tf_saved_model.index_path = ["unsimplified_common_instructions"]}, %arg11: tensor<i64> {tf_saved_model.index_path = ["indirect_call_penalty"]}, %arg12: tensor<i64> {tf_saved_model.index_path = ["load_elimination"]}, %arg13: tensor<i64> {tf_saved_model.index_path = ["is_caller_avail_external"]}, %arg14: tensor<i64> {tf_saved_model.index_path = ["call_penalty"]}, %arg15: tensor<i64> {tf_saved_model.index_path = ["cost_estimate"]}, %arg16: tensor<i64> {tf_saved_model.index_path = ["case_cluster_penalty"]}, %arg17: tensor<i64> {tf_saved_model.index_path = ["node_count"]}, %arg18: tensor<i64> {tf_saved_model.index_path = ["call_argument_setup"]}, %arg19: tensor<i64> {tf_saved_model.index_path = ["sroa_savings"]}, %arg20: tensor<i64> {tf_saved_model.index_path = ["lowered_call_arg_setup"]}, %arg21: tensor<i64> {tf_saved_model.index_path = ["threshold"]}, %arg22: tensor<i64> {tf_saved_model.index_path = ["dead_blocks"]}, %arg23: tensor<i64> {tf_saved_model.index_path = ["constant_args"]}, %arg24: tensor<i64> {tf_saved_model.index_path = ["sroa_losses"]}, %arg25: tensor<i64> {tf_saved_model.index_path = ["simplified_instructions"]}, %arg26: tensor<i64> {tf_saved_model.index_path = ["num_loops"]}, %arg27: tensor<i32> {tf_saved_model.index_path = ["step_type"]}, %arg28: tensor<i64> {tf_saved_model.index_path = ["edge_count"]}, %arg29: tensor<i64> {tf_saved_model.index_path = ["is_callee_avail_external"]}, %arg30: tensor<i64> {tf_saved_model.index_path = ["nested_inlines"]}, %arg31: tensor<i64> {tf_saved_model.index_path = ["caller_basic_block_count"]}, %arg32: tensor<i64> {tf_saved_model.index_path = ["last_call_to_static_bonus"]}, %arg33: tensor<i64> {tf_saved_model.index_path = ["nested_inline_cost_estimate"]}, %arg34: tensor<i64> {tf_saved_model.index_path = ["callsite_height"]}, %arg35: tensor<i64> {tf_saved_model.index_path = ["constant_offset_ptr_args"]}, %arg36: tensor<i64> {tf_saved_model.index_path = ["switch_penalty"]}, %arg37: tensor<f32> {tf_saved_model.index_path = ["discount"]}, %arg38: tensor<i64> {tf_saved_model.index_path = ["caller_users"]}, %arg39: tensor<f32> {tf_saved_model.index_path = ["reward"]}) -> (tensor<i64> {tf_saved_model.index_path = ["inlining_decision"]}) attributes {tf.entry_function = {inputs = "action_callsite_cost:0,action_is_multiple_blocks:0,action_caller_conditionally_executed_blocks:0,action_cold_cc_penalty:0,action_callee_conditionally_executed_blocks:0,action_callee_users:0,action_callee_basic_block_count:0,action_nr_ctant_params:0,action_load_relative_intrinsic:0,action_jump_table_penalty:0,action_unsimplified_common_instructions:0,action_indirect_call_penalty:0,action_load_elimination:0,action_is_caller_avail_external:0,action_call_penalty:0,action_cost_estimate:0,action_case_cluster_penalty:0,action_node_count:0,action_call_argument_setup:0,action_sroa_savings:0,action_lowered_call_arg_setup:0,action_threshold:0,action_dead_blocks:0,action_constant_args:0,action_sroa_losses:0,action_simplified_instructions:0,action_num_loops:0,action_step_type:0,action_edge_count:0,action_is_callee_avail_external:0,action_nested_inlines:0,action_caller_basic_block_count:0,action_last_call_to_static_bonus:0,action_nested_inline_cost_estimate:0,action_callsite_height:0,action_constant_offset_ptr_args:0,action_switch_penalty:0,action_discount:0,action_caller_users:0,action_reward:0", outputs = "PartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = "tosa.const"() <{values = dense<1> : tensor<i64>}> : () -> tensor<i64>
+    return %0 : tensor<i64>
+  }
+}
diff --git a/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
new file mode 100644
index 0000000000000..eafbbe814c96a
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stddef.h -include stdint.h -include stdlib.h\
+// DEFINE:   -include stdbool.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+// CHECK-LABEL: @mock_model
+// CHECK:         for
+// CHECK:         cmp gt
+// CHECK:         conditional
+// CHECK:         assign
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "1.9.0\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<33xi64> {tf_saved_model.index_path = ["mask"]}) -> (tensor<i64> {tf_saved_model.index_path = ["index_to_evict"]}) attributes {tf.entry_function = {inputs = "action_mask:0", outputs = "StatefulPartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = tosa.cast %arg0 : (tensor<33xi64>) -> tensor<33xi32>
+    %1 = tosa.argmax %0 {axis = 0 : i32} : (tensor<33xi32>) -> tensor<i64>
+    return %1 : tensor<i64>
+  }
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/211833

From llvmlistbot at llvm.org  Fri Jul 24 08:42:46 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 24 Jul 2026 08:42:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211833@github.com>
Message-ID: <6a6387f6.3cce8983.19fb00.11f9@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/211833

From llvmlistbot at llvm.org  Fri Jul 24 08:44:08 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 24 Jul 2026 08:44:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211833@github.com>
Message-ID: <6a638848.0c05e68e.272114.ab4f@mx.google.com>


https://github.com/ioghiban edited https://github.com/llvm/llvm-project/pull/211833

From llvmlistbot at llvm.org  Fri Jul 24 08:44:23 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 08:44:23 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a638857.1e1ac41d.8ad93.b0a8@mx.google.com>


================
@@ -600,14 +546,63 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
     }
     // Perform a backward CFG traversal to discover and map blocks in this loop.
     if (!Backedges.empty()) {
-      LoopT *L = AllocateLoop(Header);
-      discoverAndMapSubloop(L, ArrayRef<BlockT *>(Backedges), this, DomTree);
+      HasLoops = true;
+      LoopT *L = allocateLoop(Header);
+      discoverAndMapSubloop(L, Header, Backedges, DomTree);
+    }
+  }
+  // Most functions have no loops; skip the layout construction.
+  if (!HasLoops)
+    return;
+
+  // Record each in-loop block with its innermost loop in forward CFG postorder,
+  // and build the loop list in PO.
+  SmallVector<std::pair<BlockT *, LoopT *>, 32> PO;
+  SmallVector<LoopT *, 4> LoopsPO;
+  PO.reserve(BBMap.size());
+  for (BlockT *BB : post_order(DomRoot->getBlock())) {
----------------
aengelke wrote:

Make ParentT a non const-derived pointer? (Not too important, but I personally hate that we have post_order<BasicBlock *>, post_order<const BasicBlock *>, post_order<Function *>, and post_order<const Function *> -- ending up in the binary doing 99.9% the same thing.)

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 08:44:28 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 24 Jul 2026 08:44:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Allow create_mem_desc from ND
	memref (PR #211836)
Message-ID: <llvm.org/llvm/llvm-project/pull/211836@github.com>


https://github.com/Jianhui-Li created https://github.com/llvm/llvm-project/pull/211836

Relax the create_mem_desc source operand constraint to accept a statically shaped shared-memory memref of any rank, replacing the 1D/2D-only StaticShared{1,2}DMemRefOf classes with a rank-agnostic StaticSharedMemRefOf. 

Add a verifier requiring the source memref to be contiguous row-major, update the op documentation, and add valid/invalid lit tests.

assisted-by-claude

>From cadf585a7a73ede69f27f0507eb2d1cfc5be0edd Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 15:43:20 +0000
Subject: [PATCH] [mlir][xegpu] Allow create_mem_desc from ND memref

Relax the create_mem_desc source operand constraint to accept a
statically shaped shared-memory memref of any rank, replacing the
1D/2D-only StaticShared{1,2}DMemRefOf classes with a rank-agnostic
StaticSharedMemRefOf. This lets the convert-vector-to-xegpu pass build
a mem_desc from an ND (e.g. 3D) SLM memref.

Add a verifier requiring the source memref to be contiguous row-major,
update the op documentation, and add valid/invalid lit tests.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td   |  5 +++--
 mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td | 11 +++--------
 mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt         |  1 +
 mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp           | 11 +++++++++++
 mlir/test/Dialect/XeGPU/invalid.mlir             | 11 ++++++++++-
 mlir/test/Dialect/XeGPU/ops.mlir                 |  9 +++++++++
 6 files changed, 37 insertions(+), 11 deletions(-)

diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..52785fa310aa4 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1277,7 +1277,7 @@ def XeGPU_CreateMemDescOp: XeGPU_Op<"create_mem_desc", [Pure,
     as the underlying shared local memory.
 
     Arguments:
-     - `source` : 1D or 2D statically shaped memref, representing the raw SLM buffer. The provided memref must be contiguous.
+     - `source` : a statically shaped memref of any rank, representing the raw SLM buffer. The provided memref must be contiguous.
 
     Results:
      - `mem_desc` : the memory descriptor (1D or higher).
@@ -1296,9 +1296,10 @@ def XeGPU_CreateMemDescOp: XeGPU_Op<"create_mem_desc", [Pure,
     ```
 
   }];
-  let arguments = (ins AnyTypeOf<[StaticShared1DMemRefOf<[XeGPU_ScalarType]>, StaticShared2DMemRefOf<[XeGPU_ScalarType]>]>:$source);
+  let arguments = (ins StaticSharedMemRefOf<[XeGPU_ScalarType]>:$source);
   let results = (outs XeGPU_MemDesc:$mem_desc);
   let assemblyFormat = "$source prop-dict attr-dict `` `:` type($source) `->` qualified(type($mem_desc))";
+  let hasVerifier = 1;
 }
 
 def XeGPU_LoadMatrixOp: XeGPU_Op<"load_matrix", [MemoryEffects<[MemRead]>,
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
index 0423303c23493..78c29ae9e7f43 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
@@ -40,14 +40,9 @@ class XeGPUTypeDef<string name, string typeMnemonic, list<Trait> traits = [],
 }
 
 def isSharedPred : CPred<"XeGPUDialect::isSharedMemory(llvm::cast<mlir::MemRefType>($_self))">;
-class StaticShared1DMemRefOf<list<Type> allowedTypes> :
-  ConfinedType<MemRefRankOf<allowedTypes, [1]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 1d shaped " # MemRefOf<allowedTypes>.summary # " ",
-     "mlir::MemRefType">;
-
-class StaticShared2DMemRefOf<list<Type> allowedTypes>:
-  ConfinedType<MemRefRankOf<allowedTypes, [2]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 2d shaped " # MemRefOf<allowedTypes>.summary # " ",
+class StaticSharedMemRefOf<list<Type> allowedTypes>:
+  ConfinedType<MemRefOf<allowedTypes>, [HasStaticShapePred, isSharedPred],
+     "reside in share memory and statically shaped " # MemRefOf<allowedTypes>.summary # " ",
      "mlir::MemRefType">;
 
 def XeGPU_TensorDesc: XeGPUTypeDef<"TensorDesc", "tensor_desc",
diff --git a/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt b/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
index 7869a28dfed57..4ea73cda6de95 100644
--- a/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
+++ b/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_dialect_library(MLIRXeGPUDialect
   MLIRArithUtils
   MLIRDialectUtils
   MLIRGPUDialect
+  MLIRMemRefUtils
   MLIRXeVMDialect
   MLIRIR
   MLIRViewLikeInterface
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..bfbb8b08bf173 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/MemRef/Utils/MemRefUtils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
 #include "mlir/Dialect/XeGPU/IR/XeGPU.h"
@@ -195,6 +196,16 @@ IsValidMatrixOpParams(VectorType dataTy, MemDescType mdescTy,
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// XeGPU_CreateMemDescOp
+//===----------------------------------------------------------------------===//
+LogicalResult CreateMemDescOp::verify() {
+  auto srcTy = getSource().getType();
+  if (!memref::isStaticShapeAndContiguousRowMajor(srcTy))
+    return emitOpError("source memref must be contiguous.");
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // XeGPU_CreateNdDescOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..ac87045de7012 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -607,7 +607,7 @@ func.func @slice_attr_repeat_dim() {
 // -----
 func.func @create_mem_desc_non_slm() {
   %m = memref.alloca() {alignment = 1024} : memref<2048xi8, 1>
-  // expected-error at +1 {{operand #0 must be reside in share memory and statically 1d shaped memref }}
+  // expected-error at +1 {{operand #0 must be reside in share memory and statically shaped memref }}
   %mem_desc = xegpu.create_mem_desc %m : memref<2048xi8, 1> -> !xegpu.mem_desc<16x64xf16>
   return
 }
@@ -805,3 +805,12 @@ func.func @contiguity_does_not_divide(%src: i64, %offset: vector<6xindex>, %mask
       : i64, vector<6xindex>, vector<6xi1> -> vector<6xf32>
   return
 }
+
+// -----
+func.func @create_mem_desc_non_contiguous() {
+  %m = memref.alloca() {alignment = 1024} : memref<32x64xf16, 3>
+  %m_sub = memref.subview %m[0, 0][16, 32][1, 1] : memref<32x64xf16, 3> to memref<16x32xf16, strided<[64, 1]>, 3>
+  // expected-error at +1 {{source memref must be contiguous.}}
+  %mem_desc = xegpu.create_mem_desc %m_sub : memref<16x32xf16, strided<[64, 1]>, 3> -> !xegpu.mem_desc<16x32xf16>
+  return
+}
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..5591eea00a2ea 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -576,6 +576,15 @@ gpu.func @create_mem_desc_with_stride_from_2d_memref() {
   gpu.return
 }
 
+// CHECK-LABEL: gpu.func @create_mem_desc_from_3d_memref({{.*}}) {
+gpu.func @create_mem_desc_from_3d_memref() {
+  //CHECK: [[alloc:%.+]] = memref.alloca() {alignment = 1024 : i64} : memref<1x16x64xf16, 3>
+  //CHECK: [[mdesc:%.+]] = xegpu.create_mem_desc [[alloc]] : memref<1x16x64xf16, 3> -> !xegpu.mem_desc<1x16x64xf16>
+  %m = memref.alloca() {alignment = 1024} : memref<1x16x64xf16, 3>
+  %mem_desc = xegpu.create_mem_desc %m : memref<1x16x64xf16, 3> -> !xegpu.mem_desc<1x16x64xf16>
+  gpu.return
+}
+
 // CHECK: gpu.func @load_matrix([[ARG0:%.+]]: !xegpu.mem_desc<16x64xf16>)
 gpu.func @load_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
   // CHECK: xegpu.load_matrix [[ARG0]][8, 8] : !xegpu.mem_desc<16x64xf16> -> vector<8x16xf16>


From llvmlistbot at llvm.org  Fri Jul 24 08:45:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 08:45:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Allow create_mem_desc from ND
	memref (PR #211836)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211836@github.com>
Message-ID: <6a638883.b1416da1.20b4cf.1d74@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

Relax the create_mem_desc source operand constraint to accept a statically shaped shared-memory memref of any rank, replacing the 1D/2D-only StaticShared{1,2}DMemRefOf classes with a rank-agnostic StaticSharedMemRefOf. 

Add a verifier requiring the source memref to be contiguous row-major, update the op documentation, and add valid/invalid lit tests.

assisted-by-claude

---
Full diff: https://github.com/llvm/llvm-project/pull/211836.diff


6 Files Affected:

- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td (+3-2) 
- (modified) mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td (+3-8) 
- (modified) mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt (+1) 
- (modified) mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp (+11) 
- (modified) mlir/test/Dialect/XeGPU/invalid.mlir (+10-1) 
- (modified) mlir/test/Dialect/XeGPU/ops.mlir (+9) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 7f8389a6acc47..52785fa310aa4 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1277,7 +1277,7 @@ def XeGPU_CreateMemDescOp: XeGPU_Op<"create_mem_desc", [Pure,
     as the underlying shared local memory.
 
     Arguments:
-     - `source` : 1D or 2D statically shaped memref, representing the raw SLM buffer. The provided memref must be contiguous.
+     - `source` : a statically shaped memref of any rank, representing the raw SLM buffer. The provided memref must be contiguous.
 
     Results:
      - `mem_desc` : the memory descriptor (1D or higher).
@@ -1296,9 +1296,10 @@ def XeGPU_CreateMemDescOp: XeGPU_Op<"create_mem_desc", [Pure,
     ```
 
   }];
-  let arguments = (ins AnyTypeOf<[StaticShared1DMemRefOf<[XeGPU_ScalarType]>, StaticShared2DMemRefOf<[XeGPU_ScalarType]>]>:$source);
+  let arguments = (ins StaticSharedMemRefOf<[XeGPU_ScalarType]>:$source);
   let results = (outs XeGPU_MemDesc:$mem_desc);
   let assemblyFormat = "$source prop-dict attr-dict `` `:` type($source) `->` qualified(type($mem_desc))";
+  let hasVerifier = 1;
 }
 
 def XeGPU_LoadMatrixOp: XeGPU_Op<"load_matrix", [MemoryEffects<[MemRead]>,
diff --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
index 0423303c23493..78c29ae9e7f43 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUTypes.td
@@ -40,14 +40,9 @@ class XeGPUTypeDef<string name, string typeMnemonic, list<Trait> traits = [],
 }
 
 def isSharedPred : CPred<"XeGPUDialect::isSharedMemory(llvm::cast<mlir::MemRefType>($_self))">;
-class StaticShared1DMemRefOf<list<Type> allowedTypes> :
-  ConfinedType<MemRefRankOf<allowedTypes, [1]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 1d shaped " # MemRefOf<allowedTypes>.summary # " ",
-     "mlir::MemRefType">;
-
-class StaticShared2DMemRefOf<list<Type> allowedTypes>:
-  ConfinedType<MemRefRankOf<allowedTypes, [2]>, [HasStaticShapePred, isSharedPred],
-     "reside in share memory and statically 2d shaped " # MemRefOf<allowedTypes>.summary # " ",
+class StaticSharedMemRefOf<list<Type> allowedTypes>:
+  ConfinedType<MemRefOf<allowedTypes>, [HasStaticShapePred, isSharedPred],
+     "reside in share memory and statically shaped " # MemRefOf<allowedTypes>.summary # " ",
      "mlir::MemRefType">;
 
 def XeGPU_TensorDesc: XeGPUTypeDef<"TensorDesc", "tensor_desc",
diff --git a/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt b/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
index 7869a28dfed57..4ea73cda6de95 100644
--- a/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
+++ b/mlir/lib/Dialect/XeGPU/IR/CMakeLists.txt
@@ -18,6 +18,7 @@ add_mlir_dialect_library(MLIRXeGPUDialect
   MLIRArithUtils
   MLIRDialectUtils
   MLIRGPUDialect
+  MLIRMemRefUtils
   MLIRXeVMDialect
   MLIRIR
   MLIRViewLikeInterface
diff --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index 2ffe883eb0d9a..bfbb8b08bf173 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Arith/Utils/Utils.h"
+#include "mlir/Dialect/MemRef/Utils/MemRefUtils.h"
 #include "mlir/Dialect/Utils/IndexingUtils.h"
 #include "mlir/Dialect/Utils/StaticValueUtils.h"
 #include "mlir/Dialect/XeGPU/IR/XeGPU.h"
@@ -195,6 +196,16 @@ IsValidMatrixOpParams(VectorType dataTy, MemDescType mdescTy,
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// XeGPU_CreateMemDescOp
+//===----------------------------------------------------------------------===//
+LogicalResult CreateMemDescOp::verify() {
+  auto srcTy = getSource().getType();
+  if (!memref::isStaticShapeAndContiguousRowMajor(srcTy))
+    return emitOpError("source memref must be contiguous.");
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // XeGPU_CreateNdDescOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/XeGPU/invalid.mlir b/mlir/test/Dialect/XeGPU/invalid.mlir
index d5d4950fe7d7e..ac87045de7012 100644
--- a/mlir/test/Dialect/XeGPU/invalid.mlir
+++ b/mlir/test/Dialect/XeGPU/invalid.mlir
@@ -607,7 +607,7 @@ func.func @slice_attr_repeat_dim() {
 // -----
 func.func @create_mem_desc_non_slm() {
   %m = memref.alloca() {alignment = 1024} : memref<2048xi8, 1>
-  // expected-error at +1 {{operand #0 must be reside in share memory and statically 1d shaped memref }}
+  // expected-error at +1 {{operand #0 must be reside in share memory and statically shaped memref }}
   %mem_desc = xegpu.create_mem_desc %m : memref<2048xi8, 1> -> !xegpu.mem_desc<16x64xf16>
   return
 }
@@ -805,3 +805,12 @@ func.func @contiguity_does_not_divide(%src: i64, %offset: vector<6xindex>, %mask
       : i64, vector<6xindex>, vector<6xi1> -> vector<6xf32>
   return
 }
+
+// -----
+func.func @create_mem_desc_non_contiguous() {
+  %m = memref.alloca() {alignment = 1024} : memref<32x64xf16, 3>
+  %m_sub = memref.subview %m[0, 0][16, 32][1, 1] : memref<32x64xf16, 3> to memref<16x32xf16, strided<[64, 1]>, 3>
+  // expected-error at +1 {{source memref must be contiguous.}}
+  %mem_desc = xegpu.create_mem_desc %m_sub : memref<16x32xf16, strided<[64, 1]>, 3> -> !xegpu.mem_desc<16x32xf16>
+  return
+}
diff --git a/mlir/test/Dialect/XeGPU/ops.mlir b/mlir/test/Dialect/XeGPU/ops.mlir
index 6cffa3eec369b..5591eea00a2ea 100644
--- a/mlir/test/Dialect/XeGPU/ops.mlir
+++ b/mlir/test/Dialect/XeGPU/ops.mlir
@@ -576,6 +576,15 @@ gpu.func @create_mem_desc_with_stride_from_2d_memref() {
   gpu.return
 }
 
+// CHECK-LABEL: gpu.func @create_mem_desc_from_3d_memref({{.*}}) {
+gpu.func @create_mem_desc_from_3d_memref() {
+  //CHECK: [[alloc:%.+]] = memref.alloca() {alignment = 1024 : i64} : memref<1x16x64xf16, 3>
+  //CHECK: [[mdesc:%.+]] = xegpu.create_mem_desc [[alloc]] : memref<1x16x64xf16, 3> -> !xegpu.mem_desc<1x16x64xf16>
+  %m = memref.alloca() {alignment = 1024} : memref<1x16x64xf16, 3>
+  %mem_desc = xegpu.create_mem_desc %m : memref<1x16x64xf16, 3> -> !xegpu.mem_desc<1x16x64xf16>
+  gpu.return
+}
+
 // CHECK: gpu.func @load_matrix([[ARG0:%.+]]: !xegpu.mem_desc<16x64xf16>)
 gpu.func @load_matrix(%arg0: !xegpu.mem_desc<16x64xf16>) {
   // CHECK: xegpu.load_matrix [[ARG0]][8, 8] : !xegpu.mem_desc<16x64xf16> -> vector<8x16xf16>

``````````

</details>


https://github.com/llvm/llvm-project/pull/211836

From llvmlistbot at llvm.org  Fri Jul 24 08:46:09 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 24 Jul 2026 08:46:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Fix DPAS subgroup layout
 propagation for broadcast K dim (PR #211695)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211695@github.com>
Message-ID: <6a6388c1.8e75e0cb.2de827.c3a2@mx.google.com>


https://github.com/Jianhui-Li edited https://github.com/llvm/llvm-project/pull/211695

From llvmlistbot at llvm.org  Fri Jul 24 08:46:39 2026
From: llvmlistbot at llvm.org (Alexis Engelke)
Date: Fri, 24 Jul 2026 08:46:39 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6388df.a5745248.9066c.8a05@mx.google.com>


https://github.com/aengelke approved this pull request.

LGTM. Not 100% convinced this is worth the complexity, though.

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 08:50:07 2026
From: llvmlistbot at llvm.org (Alex MacLean)
Date: Fri, 24 Jul 2026 08:50:07 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [NVPTX] Add Rubin extensions to
	tcgen05.commit (PR #211577)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211577@github.com>
Message-ID: <6a6389af.adb4cce5.2af4ea.de09@mx.google.com>


================
@@ -2634,8 +2634,17 @@ For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parall
 ```llvm
 declare void @llvm.nvvm.tcgen05.commit.{cg1,cg2}(ptr %mbar)
 declare void @llvm.nvvm.tcgen05.commit.shared.{cg1,cg2}(ptr addrspace(3) %mbar)
-declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}(ptr %mbar, i16 %mc)
-declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}(ptr addrspace(3) %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}.i16(ptr %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}.i16(ptr addrspace(3) %mbar, i16 %mc)
----------------
AlexMaclean wrote:

Can we get rid of the shared/generic variants and use an overloaded intrinsic? This would half the number of intrinsics in this family. 

https://github.com/llvm/llvm-project/pull/211577

From llvmlistbot at llvm.org  Fri Jul 24 08:50:26 2026
From: llvmlistbot at llvm.org (Yingwei Zheng)
Date: Fri, 24 Jul 2026 08:50:26 -0700 (PDT)
Subject: [Mlir-commits] [clang] [llvm] [mlir] [IR] Make semantics of
 strictfp consistent v2 (PR #211769)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211769@github.com>
Message-ID: <6a6389c2.f804b674.76685.4c52@mx.google.com>


https://github.com/dtcxzyw approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/211769

From llvmlistbot at llvm.org  Fri Jul 24 08:54:56 2026
From: llvmlistbot at llvm.org (ioana ghiban)
Date: Fri, 24 Jul 2026 08:54:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211833@github.com>
Message-ID: <6a638ad0.433f7c1d.139069.73d7@mx.google.com>


https://github.com/ioghiban updated https://github.com/llvm/llvm-project/pull/211833

>From b15f5500e791deff24a06b0c9b525bbb4b15b3e8 Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 24 Jul 2026 17:21:29 +0200
Subject: [PATCH 1/2] Add TOSA -> EmitC tests for MLGO mock models

---
 .../EmitC/inline-oz-test-model-tosa.mlir      | 57 +++++++++++++++++++
 .../regalloc-eviction-test-model-tosa.mlir    | 57 +++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
 create mode 100644 mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir

diff --git a/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
new file mode 100644
index 0000000000000..ff5de17ba06a0
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stdint.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-inline-oz-test-model.py  + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: return type of 'main' is not 'int'
+///   error: too many parameters (40) for 'main': must be 0, 2, or 3
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+///   error: second parameter of 'main' (argument array) must be of type 'char **'
+///   error: third parameter of 'main' (environment) must be of type 'char **'
+
+// CHECK-LABEL: @mock_model
+// CHECK:         emitc.constant
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<i64> {tf_saved_model.index_path = ["callsite_cost"]}, %arg1: tensor<i64> {tf_saved_model.index_path = ["is_multiple_blocks"]}, %arg2: tensor<i64> {tf_saved_model.index_path = ["caller_conditionally_executed_blocks"]}, %arg3: tensor<i64> {tf_saved_model.index_path = ["cold_cc_penalty"]}, %arg4: tensor<i64> {tf_saved_model.index_path = ["callee_conditionally_executed_blocks"]}, %arg5: tensor<i64> {tf_saved_model.index_path = ["callee_users"]}, %arg6: tensor<i64> {tf_saved_model.index_path = ["callee_basic_block_count"]}, %arg7: tensor<i64> {tf_saved_model.index_path = ["nr_ctant_params"]}, %arg8: tensor<i64> {tf_saved_model.index_path = ["load_relative_intrinsic"]}, %arg9: tensor<i64> {tf_saved_model.index_path = ["jump_table_penalty"]}, %arg10: tensor<i64> {tf_saved_model.index_path = ["unsimplified_common_instructions"]}, %arg11: tensor<i64> {tf_saved_model.index_path = ["indirect_call_penalty"]}, %arg12: tensor<i64> {tf_saved_model.index_path = ["load_elimination"]}, %arg13: tensor<i64> {tf_saved_model.index_path = ["is_caller_avail_external"]}, %arg14: tensor<i64> {tf_saved_model.index_path = ["call_penalty"]}, %arg15: tensor<i64> {tf_saved_model.index_path = ["cost_estimate"]}, %arg16: tensor<i64> {tf_saved_model.index_path = ["case_cluster_penalty"]}, %arg17: tensor<i64> {tf_saved_model.index_path = ["node_count"]}, %arg18: tensor<i64> {tf_saved_model.index_path = ["call_argument_setup"]}, %arg19: tensor<i64> {tf_saved_model.index_path = ["sroa_savings"]}, %arg20: tensor<i64> {tf_saved_model.index_path = ["lowered_call_arg_setup"]}, %arg21: tensor<i64> {tf_saved_model.index_path = ["threshold"]}, %arg22: tensor<i64> {tf_saved_model.index_path = ["dead_blocks"]}, %arg23: tensor<i64> {tf_saved_model.index_path = ["constant_args"]}, %arg24: tensor<i64> {tf_saved_model.index_path = ["sroa_losses"]}, %arg25: tensor<i64> {tf_saved_model.index_path = ["simplified_instructions"]}, %arg26: tensor<i64> {tf_saved_model.index_path = ["num_loops"]}, %arg27: tensor<i32> {tf_saved_model.index_path = ["step_type"]}, %arg28: tensor<i64> {tf_saved_model.index_path = ["edge_count"]}, %arg29: tensor<i64> {tf_saved_model.index_path = ["is_callee_avail_external"]}, %arg30: tensor<i64> {tf_saved_model.index_path = ["nested_inlines"]}, %arg31: tensor<i64> {tf_saved_model.index_path = ["caller_basic_block_count"]}, %arg32: tensor<i64> {tf_saved_model.index_path = ["last_call_to_static_bonus"]}, %arg33: tensor<i64> {tf_saved_model.index_path = ["nested_inline_cost_estimate"]}, %arg34: tensor<i64> {tf_saved_model.index_path = ["callsite_height"]}, %arg35: tensor<i64> {tf_saved_model.index_path = ["constant_offset_ptr_args"]}, %arg36: tensor<i64> {tf_saved_model.index_path = ["switch_penalty"]}, %arg37: tensor<f32> {tf_saved_model.index_path = ["discount"]}, %arg38: tensor<i64> {tf_saved_model.index_path = ["caller_users"]}, %arg39: tensor<f32> {tf_saved_model.index_path = ["reward"]}) -> (tensor<i64> {tf_saved_model.index_path = ["inlining_decision"]}) attributes {tf.entry_function = {inputs = "action_callsite_cost:0,action_is_multiple_blocks:0,action_caller_conditionally_executed_blocks:0,action_cold_cc_penalty:0,action_callee_conditionally_executed_blocks:0,action_callee_users:0,action_callee_basic_block_count:0,action_nr_ctant_params:0,action_load_relative_intrinsic:0,action_jump_table_penalty:0,action_unsimplified_common_instructions:0,action_indirect_call_penalty:0,action_load_elimination:0,action_is_caller_avail_external:0,action_call_penalty:0,action_cost_estimate:0,action_case_cluster_penalty:0,action_node_count:0,action_call_argument_setup:0,action_sroa_savings:0,action_lowered_call_arg_setup:0,action_threshold:0,action_dead_blocks:0,action_constant_args:0,action_sroa_losses:0,action_simplified_instructions:0,action_num_loops:0,action_step_type:0,action_edge_count:0,action_is_callee_avail_external:0,action_nested_inlines:0,action_caller_basic_block_count:0,action_last_call_to_static_bonus:0,action_nested_inline_cost_estimate:0,action_callsite_height:0,action_constant_offset_ptr_args:0,action_switch_penalty:0,action_discount:0,action_caller_users:0,action_reward:0", outputs = "PartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = "tosa.const"() <{values = dense<1> : tensor<i64>}> : () -> tensor<i64>
+    return %0 : tensor<i64>
+  }
+}
diff --git a/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
new file mode 100644
index 0000000000000..eafbbe814c96a
--- /dev/null
+++ b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
@@ -0,0 +1,57 @@
+// DEFINE: %{pipeline} = "builtin.module(\
+// DEFINE:   func.func(\
+// DEFINE:     tosa-to-linalg-named,\
+// DEFINE:     tosa-to-linalg,\
+// DEFINE:     tosa-to-arith,\
+// DEFINE:     tosa-to-tensor\
+// DEFINE:   ),\
+// DEFINE:   symbol-privatize,\
+// DEFINE:   scalarize-single-element-tensor-return,\
+// DEFINE:   one-shot-bufferize{\
+// DEFINE:     bufferize-function-boundaries=true\
+// DEFINE:     function-boundary-type-conversion=identity-layout-map\
+// DEFINE:     buffer-alignment=0\
+// DEFINE:   },\
+// DEFINE:   buffer-results-to-out-params{\
+// DEFINE:     hoist-static-allocs=true\
+// DEFINE:   },\
+// DEFINE:   buffer-deallocation-pipeline,\
+// DEFINE:   func.func(\
+// DEFINE:     convert-linalg-to-loops\
+// DEFINE:   ),\
+// DEFINE:   expand-strided-metadata,\
+// DEFINE:   canonicalize,\
+// DEFINE:   memref-elide-reinterpret-cast,\
+// DEFINE:   convert-to-emitc,\
+// DEFINE:   math-expand-ops{\
+// DEFINE:     ops=rsqrt\
+// DEFINE:   },\
+// DEFINE:   arith-expand,\
+// DEFINE:   convert-math-to-emitc,\
+// DEFINE:   convert-arith-to-emitc\
+// DEFINE: )"
+
+// DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
+// DEFINE: %{translate} = mlir-translate -mlir-to-cpp %t -o %t.c 
+// DEFINE: %{compile} =  %host_cc -include stddef.h -include stdint.h -include stdlib.h\
+// DEFINE:   -include stdbool.h -fsyntax-only -Wpedantic -Wall -Werror -Wno-unused %t.c
+
+/// Lower via the pipeline defined above
+// RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
+
+/// Generated with llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Renamed "main" to avoid:
+///   error: 'main' should not be declared static
+///   error: first parameter of 'main' (argument count) must be of type 'int'
+// CHECK-LABEL: @mock_model
+// CHECK:         for
+// CHECK:         cmp gt
+// CHECK:         conditional
+// CHECK:         assign
+module attributes {tf_saved_model.semantics, tfl.description = "MLIR Converted.", tfl.metadata = {CONVERSION_METADATA = "\0C\00\00\00\08\00\0E\00\08\00\04\00\08\00\00\00\10\00\00\00$\00\00\00\00\00\06\00\08\00\04\00\06\00\00\00\04\00\00\00\00\00\00\00\00\00\0A\00\10\00\0C\00\08\00\04\00\0A\00\00\00\01\00\00\00\02\00\00\00\04\00\00\00\06\00\00\002.15.1\00\00", min_runtime_version = "1.9.0\00\00\00\00\00\00\00\00\00\00\00"}, tfl.schema_version = 3 : i32} {
+  func.func @mock_model(%arg0: tensor<33xi64> {tf_saved_model.index_path = ["mask"]}) -> (tensor<i64> {tf_saved_model.index_path = ["index_to_evict"]}) attributes {tf.entry_function = {inputs = "action_mask:0", outputs = "StatefulPartitionedCall:0"}, tf_saved_model.exported_names = ["action"]} {
+    %0 = tosa.cast %arg0 : (tensor<33xi64>) -> tensor<33xi32>
+    %1 = tosa.argmax %0 {axis = 0 : i32} : (tensor<33xi32>) -> tensor<i64>
+    return %1 : tensor<i64>
+  }
+}

>From 0197506a62cf6dcbab4b62fee4fa094fec5acead Mon Sep 17 00:00:00 2001
From: Ioana Ghiban <ioana.ghiban at arm.com>
Date: Fri, 24 Jul 2026 17:54:32 +0200
Subject: [PATCH 2/2] Remove unnecessary passes from general pipeline

---
 .../EmitC/inline-oz-test-model-tosa.mlir      | 26 ++-----------------
 .../regalloc-eviction-test-model-tosa.mlir    | 14 +---------
 2 files changed, 3 insertions(+), 37 deletions(-)

diff --git a/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
index ff5de17ba06a0..14750a476c0df 100644
--- a/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
+++ b/mlir/test/Integration/Dialect/EmitC/inline-oz-test-model-tosa.mlir
@@ -1,34 +1,12 @@
 // DEFINE: %{pipeline} = "builtin.module(\
-// DEFINE:   func.func(\
-// DEFINE:     tosa-to-linalg-named,\
-// DEFINE:     tosa-to-linalg,\
-// DEFINE:     tosa-to-arith,\
-// DEFINE:     tosa-to-tensor\
-// DEFINE:   ),\
 // DEFINE:   symbol-privatize,\
 // DEFINE:   scalarize-single-element-tensor-return,\
 // DEFINE:   one-shot-bufferize{\
 // DEFINE:     bufferize-function-boundaries=true\
 // DEFINE:     function-boundary-type-conversion=identity-layout-map\
-// DEFINE:     buffer-alignment=0\
-// DEFINE:   },\
-// DEFINE:   buffer-results-to-out-params{\
-// DEFINE:     hoist-static-allocs=true\
 // DEFINE:   },\
 // DEFINE:   buffer-deallocation-pipeline,\
-// DEFINE:   func.func(\
-// DEFINE:     convert-linalg-to-loops\
-// DEFINE:   ),\
-// DEFINE:   expand-strided-metadata,\
-// DEFINE:   canonicalize,\
-// DEFINE:   memref-elide-reinterpret-cast,\
-// DEFINE:   convert-to-emitc,\
-// DEFINE:   math-expand-ops{\
-// DEFINE:     ops=rsqrt\
-// DEFINE:   },\
-// DEFINE:   arith-expand,\
-// DEFINE:   convert-math-to-emitc,\
-// DEFINE:   convert-arith-to-emitc\
+// DEFINE:   convert-to-emitc\
 // DEFINE: )"
 
 // DEFINE: %{lower_to_emitc} = mlir-opt --pass-pipeline=%{pipeline} %s -o %t
@@ -38,7 +16,7 @@
 /// Lower via the pipeline defined above
 // RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
 
-/// Generated with llvm/lib/Analysis/models/gen-inline-oz-test-model.py  + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Generated by llvm/lib/Analysis/models/gen-inline-oz-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
 /// Renamed "main" to avoid:
 ///   error: 'main' should not be declared static
 ///   error: return type of 'main' is not 'int'
diff --git a/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
index eafbbe814c96a..f777dfc6e329f 100644
--- a/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
+++ b/mlir/test/Integration/Dialect/EmitC/regalloc-eviction-test-model-tosa.mlir
@@ -10,24 +10,12 @@
 // DEFINE:   one-shot-bufferize{\
 // DEFINE:     bufferize-function-boundaries=true\
 // DEFINE:     function-boundary-type-conversion=identity-layout-map\
-// DEFINE:     buffer-alignment=0\
 // DEFINE:   },\
-// DEFINE:   buffer-results-to-out-params{\
-// DEFINE:     hoist-static-allocs=true\
-// DEFINE:   },\
-// DEFINE:   buffer-deallocation-pipeline,\
 // DEFINE:   func.func(\
 // DEFINE:     convert-linalg-to-loops\
 // DEFINE:   ),\
-// DEFINE:   expand-strided-metadata,\
-// DEFINE:   canonicalize,\
-// DEFINE:   memref-elide-reinterpret-cast,\
 // DEFINE:   convert-to-emitc,\
-// DEFINE:   math-expand-ops{\
-// DEFINE:     ops=rsqrt\
-// DEFINE:   },\
 // DEFINE:   arith-expand,\
-// DEFINE:   convert-math-to-emitc,\
 // DEFINE:   convert-arith-to-emitc\
 // DEFINE: )"
 
@@ -39,7 +27,7 @@
 /// Lower via the pipeline defined above
 // RUN: rm -f %t && %{lower_to_emitc} && FileCheck %s --input-file=%t && %{translate} && %{compile}
 
-/// Generated with llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
+/// Generated by llvm/lib/Analysis/models/gen-regalloc-eviction-test-model.py + llvm/lib/Analysis/models/saved-model-to-tflite.py
 /// Renamed "main" to avoid:
 ///   error: 'main' should not be declared static
 ///   error: first parameter of 'main' (argument count) must be of type 'int'


From llvmlistbot at llvm.org  Fri Jul 24 08:58:37 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 08:58:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][doc] Document FP8 e4m3/e5m2 mma.sync
	formats (PR #211841)
Message-ID: <llvm.org/llvm/llvm-project/pull/211841@github.com>


https://github.com/weimin023 created https://github.com/llvm/llvm-project/pull/211841

## Description

Update the documentation for the supported FP8 `mma.sync` formats introduced in #207307

>From cadf5bedd6d3286a073085ed21a66bf538590c16 Mon Sep 17 00:00:00 2001
From: weimin023 <tnwilly at gmail.com>
Date: Fri, 24 Jul 2026 15:49:05 +0000
Subject: [PATCH] [doc] document FP8 mma.sync

Signed-off-by: weimin023 <tnwilly at gmail.com>
---
 mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
index 1f76218936e16..e2ae8ccfb0aef 100644
--- a/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
+++ b/mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td
@@ -3396,6 +3396,8 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     `#nvvm.mma_layout<row>` and `#nvvm.mma_layout<col>` respectively, but other
     combinations are possible for certain layouts according to the table below.
 
+    For FP8 operations, A and B can independently use `e4m3` or `e5m2`.
+
     ```
     | A/B Type | Shape     | ALayout | BLayout | A Type   | B Type   | C/D Type          |
     |----------|-----------|---------|---------|----------|----------|-------------------|
@@ -3415,6 +3417,8 @@ def NVVM_MmaOp : NVVM_Op<"mma.sync", [AttrSizedOperandSegments]> {
     |          | m16n8k64  | row     | col     | 4x i32   | 2x i32   | 4x i32            |
     | b1       | m8n8k128  | row     | col     | 1x i32   | 1x i32   | 2x i32            |
     |          | m16n8k128 | row     | col     | 2x i32   | 1x i32   | 4x i32            |
+    | e4m3/e5m2 | .m16n8k16 | row     | col     | 2x i32   | 1x i32   | 2x f16x2 or 4 f32 |
+    |            | .m16n8k32 | row     | col     | 4x i32   | 2x i32   | 2x f16x2 or 4 f32 |
     ```
 
 


From llvmlistbot at llvm.org  Fri Jul 24 08:58:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 08:58:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][doc] Document FP8 e4m3/e5m2 mma.sync
	formats (PR #211841)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211841@github.com>
Message-ID: <6a638bb6.ca5545e3.76d76.4367@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-llvm

Author: WMC (weimin023)

<details>
<summary>Changes</summary>

## Description

Update the documentation for the supported FP8 `mma.sync` formats introduced in #<!-- -->207307

---

Patch is 53.61 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/211841.diff


1 Files Affected:

- (modified) mlir/include/mlir/Dialect/LLVMIR/NVVMOps.td (+4) 


``````````diff
<!DOCTYPE html>
<!--

Hello future GitHubber! I bet you're here to remove those nasty inline styles,
DRY up these templates and make 'em nice and re-usable, right?

Please, don't. https://github.com/styleguide/templates/2.0

-->
<html>
  <head>
    <title>Unicorn! &middot; GitHub</title>
    <style type="text/css" media="screen">
      body {
        background-color: #f1f1f1;
        margin: 0;
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
      }

      .container { margin: 50px auto 40px auto; width: 600px; text-align: center; }

      a { color: #4183c4; text-decoration: none; }
      a:hover { text-decoration: underline; }

      h1 { letter-spacing: -1px; line-height: 60px; font-size: 60px; font-weight: 100; margin: 0px; text-shadow: 0 1px 0 #fff; }
      p { color: rgba(0, 0, 0, 0.5); margin: 10px 0 10px; font-size: 18px; font-weight: 200; line-height: 1.6em;}

      ul { list-style: none; margin: 25px 0; padding: 0; }
      li { display: table-cell; font-weight: bold; width: 1%; }

      .logo { display: inline-block; margin-top: 35px; }
      .logo-img-2x { display: none; }
      @media
      only screen and (-webkit-min-device-pixel-ratio: 2),
      only screen and (   min--moz-device-pixel-ratio: 2),
      only screen and (     -o-min-device-pixel-ratio: 2/1),
      only screen and (        min-device-pixel-ratio: 2),
      only screen and (                min-resolution: 192dpi),
      only screen and (                min-resolution: 2dppx) {
        .logo-img-1x { display: none; }
        .logo-img-2x { display: inline-block; }
      }

      #suggestions {
        margin-top: 35px;
        color: #ccc;
      }
      #suggestions a {
        color: #666666;
        font-weight: 200;
        font-size: 14px;
        margin: 0 10px;
      }

    </style>
  </head>
  <body>

    <div class="container">
      <p>
        <img width="200" src="data:image/png;base64,
 ...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/211841

From llvmlistbot at llvm.org  Fri Jul 24 09:09:16 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 09:09:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][doc] Document FP8 e4m3/e5m2 mma.sync
	formats (PR #211841)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211841@github.com>
Message-ID: <6a638e2c.ec04e30f.2ba4f0.5c61@mx.google.com>


weimin023 wrote:

Hi @egebeysel, updated. 

https://github.com/llvm/llvm-project/pull/211841

From llvmlistbot at llvm.org  Fri Jul 24 09:14:06 2026
From: llvmlistbot at llvm.org (=?UTF-8?Q?Andrzej_Warzy=C5=84ski?=)
Date: Fri, 24 Jul 2026 09:14:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] Add TOSA -> EmitC tests for MLGO mock models
	(PR #211833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211833@github.com>
Message-ID: <6a638f4e.160324b0.28c9e1.7ece@mx.google.com>


https://github.com/banach-space approved this pull request.

LGTM, thank you!

https://github.com/llvm/llvm-project/pull/211833

From llvmlistbot at llvm.org  Fri Jul 24 09:20:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 09:20:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Support IsIsolatedFromAbove
 trait for python-defined operations (PR #211849)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211849@github.com>
Message-ID: <6a6390b7.bd93e731.32a2a8.6cd4@mx.google.com>


https://github.com/PragmaTwice edited https://github.com/llvm/llvm-project/pull/211849

From llvmlistbot at llvm.org  Fri Jul 24 09:24:04 2026
From: llvmlistbot at llvm.org (Charitha Saumya)
Date: Fri, 24 Jul 2026 09:24:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Make convert_layout
	input_layout optional (PR #210833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210833@github.com>
Message-ID: <6a6391a4.510f576d.23298e.3ce2@mx.google.com>


https://github.com/charithaintc approved this pull request.


https://github.com/llvm/llvm-project/pull/210833

From llvmlistbot at llvm.org  Fri Jul 24 09:24:05 2026
From: llvmlistbot at llvm.org (Eric Christopher)
Date: Fri, 24 Jul 2026 09:24:05 -0700 (PDT)
Subject: [Mlir-commits] [lldb] [llvm] [mlir] [DebugInfo] Add symbolic
 branches to DIExpression (PR #210850)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210850@github.com>
Message-ID: <6a6391a5.1b0626d0.1ff4c3.163f@mx.google.com>


https://github.com/echristo updated https://github.com/llvm/llvm-project/pull/210850

>From ab4cecf7ddb25e85c1a7a8bbfc38743dc2125ac2 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Wed, 15 Jul 2026 08:51:28 -0700
Subject: [PATCH 1/8] [DebugInfo] Add symbolic branches to DIExpression

Added three pseudo-ops to DIExpression:

- DW_OP_LLVM_label declares a label and emits no bytes.
- DW_OP_LLVM_bra branches to a label when the top of the stack is non-zero.
- DW_OP_LLVM_skip always branches to a label.

DW_OP_bra and DW_OP_skip store a two-byte offset in [-32768, 32767], but
we don't know those offsets until we emit the expression. DIExpression keeps
label IDs, and we fix them up late in CodeGen.

CodeGen records each label's byte offset and leaves a zero placeholder for each
branch. Once it has seen the whole expression, it patches the placeholders in
the existing temporary buffers used for DIE and location-list expressions.
Both paths keep their current encoding and target byte order.

The verifier makes sure every branch has a label in the same expression and
each label ID is declared once. Labels don't need a branch, and branches can go
forward, backward, to themselves, or form cycles. Raw DW_OP_bra and DW_OP_skip
remain invalid in IR because their offsets depend on the final encoding.

Symbolic control flow only works with one location operand. We reject expressions
that combine it with DW_OP_LLVM_arg or DW_OP_LLVM_implicit_pointer, and diagnose
DIArgList records. Existing expression rewrites keep the pseudo-ops and their
IDs in place.

CodeGen reports an error if the final offset is outside [-32768, 32767], or if a
label, branch, or skip splits a deferred conversion. DWARF 5 emits DW_OP_convert
directly, but older versions may defer one conversion until they see the next.

Tests cover IR and MLIR round trips, verifier failures, forward and backward
branches, self-branches and cycles, both byte orders, conversion boundaries,
and out-of-range offsets.
---
 llvm/docs/LangRef.md                          |  16 ++
 llvm/docs/ReleaseNotes.md                     |   6 +
 llvm/docs/SourceLevelDebugging.md             |  61 ++++++-
 llvm/include/llvm/BinaryFormat/Dwarf.h        |   5 +
 llvm/lib/BinaryFormat/Dwarf.cpp               |   9 +
 llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp    |  27 ++-
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  83 +++++++++-
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h |  27 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     |  52 +++++-
 llvm/lib/IR/DebugInfoMetadata.cpp             | 103 ++++++++++--
 llvm/lib/IR/Verifier.cpp                      |  13 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../di-expression-symbolic-branches.ll        |  12 ++
 .../X86/di-expression-symbolic-branches.mir   | 102 ++++++++++++
 .../di-expression-symbolic-branches.ll        |  44 +++++
 ...expression-symbolic-branch-out-of-range.py |  33 ++++
 ...ession-symbolic-branch-convert-boundary.ll |  96 +++++++++++
 ...expression-symbolic-branch-out-of-range.ll |   7 +
 .../X86/di-expression-symbolic-branches.ll    |  84 ++++++++++
 .../di-expression-symbolic-branches.ll        |  93 +++++++++++
 llvm/unittests/BinaryFormat/DwarfTest.cpp     |   7 +
 llvm/unittests/IR/MetadataTest.cpp            | 155 ++++++++++++++++++
 .../Import/debug-info-symbolic-branches.ll    |  23 +++
 .../llvmir-debug-symbolic-branches.mlir       |  19 +++
 24 files changed, 1054 insertions(+), 35 deletions(-)
 create mode 100644 llvm/test/Assembler/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Bitcode/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
 create mode 100644 llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
 create mode 100644 llvm/test/Verifier/di-expression-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
 create mode 100644 mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a6b3146cf6749..05162290f330a 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7159,6 +7159,22 @@ Some examples of expressions:
 !DIExpression(DW_OP_constu, 42, DW_OP_stack_value)
 ```
 
+`DIExpression` uses three pseudo-ops for local control flow:
+
+```text
+DW_OP_LLVM_label, <label-id>
+DW_OP_LLVM_bra,   <label-id>
+DW_OP_LLVM_skip,  <label-id>
+```
+
+`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
+branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
+branches. Label IDs are local to the expression. Every branch needs a matching
+label, labels without branches are valid, and each ID can only be declared
+once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
+
+See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+
 ##### DIAssignID
 
 `DIAssignID` nodes have no operands and are always distinct. They are used to
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 94208f8e36d19..988740a7e5e39 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -109,6 +109,12 @@ Makes programs 10x faster by doing Special New Thing.
 
 ### Changes to the Debug Info
 
+* Added `DW_OP_LLVM_label`, `DW_OP_LLVM_bra`, and `DW_OP_LLVM_skip` for symbolic
+  branches in `DIExpression`. These operations use label IDs that CodeGen
+  resolves when it emits the expression. `DW_OP_LLVM_convert` can appear before
+  or after them, but when CodeGen can't emit `DW_OP_convert`, it reports an
+  error if a deferred conversion reaches a label, branch, or skip.
+
 ### Changes to the LLVM tools
 
 ### Changes to LLDB
diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 74bb8840a56e4..645e441e6fe2a 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -386,10 +386,9 @@ call void @llvm.dbg.assign(
 
 Debug expressions are represented as {ref}`specialized-metadata`.
 
-Debug expressions are interpreted left-to-right: start by pushing the
-value/address operand of the record onto a stack, then repeatedly push and
-evaluate opcodes from the `DIExpression` until the final variable description
-is produced.
+A debug expression starts with the record's value or address operand on the
+stack, then evaluates operations from left to right unless a symbolic branch
+jumps to a label in the same `DIExpression`.
 
 The opcodes available in these expressions are described in
 {ref}`dwarf-opcodes` and {ref}`internal-opcodes`.
@@ -470,6 +469,50 @@ Some opcodes do not influence the final DWARF expression directly, instead
 encoding information logically belonging to the debug records which use
 them.
 ```
+
+(symbolic-control-flow)=
+
+##### Symbolic Control Flow
+
+DWARF `DW_OP_bra` and `DW_OP_skip` have a two-byte offset in
+`[-32768, 32767]`, but we don't know that offset until we emit the expression,
+so we use three pseudo-ops with label IDs that CodeGen resolves during
+emission:
+
+- `DW_OP_LLVM_label, ID` marks a destination and emits no bytes.
+- `DW_OP_LLVM_bra, ID` branches to label `ID` when the value on top of the
+  expression stack is non-zero.
+- `DW_OP_LLVM_skip, ID` always branches to label `ID`.
+
+Label IDs are local to an expression:
+
+- Each ID can have at most one label, and every branch needs a matching label
+  in the same expression.
+- Labels don't need branches, and consecutive labels have the same byte
+  offset.
+- Branches can go forward, backward, to themselves, or form cycles.
+
+There are a few other restrictions:
+
+- Put labels, branches, and skips before `DW_OP_stack_value` and
+  `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
+- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
+  separately, so they can't be used with symbolic control flow.
+- Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
+  applies to every path.
+
+We don't check reachability, termination, or stack state where paths meet.
+
+`DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
+and we don't match conversions on different paths. When CodeGen can't emit
+`DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
+branch, or skip would split the pair, CodeGen reports an error.
+
+Local expression rewrites stop at labels, branches, and skips; they can still
+add operations to either end, but they don't move, remove, or copy labels.
+
+##### Other Internal Opcodes
+
 - `DW_OP_LLVM_fragment, <offset>, <size>` may appear at most once in an
   expression, and must be the last opcode. It specifies the bit offset and bit
   size of the variable fragment being described by the record or intrinsic
@@ -482,11 +525,15 @@ them.
 - `DW_OP_LLVM_convert, 16, DW_ATE_signed` specifies a bit size and encoding
   (`16` and `DW_ATE_signed` here, respectively) to which the top of the
   expression stack is to be converted. Maps into a `DW_OP_convert` operation
-  that references a base type constructed from the supplied values.
+  that references a base type constructed from the supplied values. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for how conversions work
+  with labels, branches, and skips.
 - `DW_OP_LLVM_tag_offset, tag_offset` specifies that a memory tag should be
   optionally applied to the pointer. The memory tag is derived from the given
-  tag offset in an implementation-defined manner. (This does not affect the
-  semantics of the expression containing it.)
+  tag offset in an implementation-defined manner. See
+  {ref}`symbolic control flow <symbolic-control-flow>` for its placement with
+  labels, branches, and skips.
+  (This does not affect the semantics of the expression containing it.)
 - `DW_OP_LLVM_entry_value, N` evaluates a sub-expression as-if it were
   evaluated upon entry to the current call frame.
 
diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 75eb3a163e06f..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -149,6 +149,11 @@ enum LocationAtom {
   DW_OP_LLVM_arg = 0x1005,               ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_sext = 0x1006, ///< Only used in LLVM metadata.
   DW_OP_LLVM_extract_bits_zext = 0x1007, ///< Only used in LLVM metadata.
+  // Labels declare IDs local to each expression, and branches use them as
+  // targets.
+  DW_OP_LLVM_label = 0x1008, ///< Only used in LLVM metadata.
+  DW_OP_LLVM_bra = 0x1009,   ///< Only used in LLVM metadata.
+  DW_OP_LLVM_skip = 0x100a,  ///< Only used in LLVM metadata.
 };
 
 enum LlvmUserLocationAtom {
diff --git a/llvm/lib/BinaryFormat/Dwarf.cpp b/llvm/lib/BinaryFormat/Dwarf.cpp
index 9ede7ebbe20ad..431d01c953768 100644
--- a/llvm/lib/BinaryFormat/Dwarf.cpp
+++ b/llvm/lib/BinaryFormat/Dwarf.cpp
@@ -159,6 +159,12 @@ StringRef llvm::dwarf::OperationEncodingString(unsigned Encoding) {
     return "DW_OP_LLVM_extract_bits_sext";
   case DW_OP_LLVM_extract_bits_zext:
     return "DW_OP_LLVM_extract_bits_zext";
+  case DW_OP_LLVM_label:
+    return "DW_OP_LLVM_label";
+  case DW_OP_LLVM_bra:
+    return "DW_OP_LLVM_bra";
+  case DW_OP_LLVM_skip:
+    return "DW_OP_LLVM_skip";
   }
 }
 
@@ -175,6 +181,9 @@ unsigned llvm::dwarf::getOperationEncoding(StringRef OperationEncodingString) {
       .Case("DW_OP_LLVM_arg", DW_OP_LLVM_arg)
       .Case("DW_OP_LLVM_extract_bits_sext", DW_OP_LLVM_extract_bits_sext)
       .Case("DW_OP_LLVM_extract_bits_zext", DW_OP_LLVM_extract_bits_zext)
+      .Case("DW_OP_LLVM_label", DW_OP_LLVM_label)
+      .Case("DW_OP_LLVM_bra", DW_OP_LLVM_bra)
+      .Case("DW_OP_LLVM_skip", DW_OP_LLVM_skip)
       .Default(0);
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
index cfbb35d516534..45f1f13e999fc 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfDebug.cpp
@@ -191,8 +191,14 @@ void DebugLocDwarfExpression::emitUnsigned(uint64_t Value) {
   getActiveStreamer().emitULEB128(Value, Twine(Value));
 }
 
-void DebugLocDwarfExpression::emitData1(uint8_t Value) {
-  getActiveStreamer().emitInt8(Value, Twine(Value));
+void DebugLocDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    getActiveStreamer().emitInt8(Value >> (Byte * 8), Twine(Value));
+  }
 }
 
 void DebugLocDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -232,6 +238,23 @@ void DebugLocDwarfExpression::commitTemporaryBuffer() {
   TmpBuf->Comments.clear();
 }
 
+void DebugLocDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                         uint64_t Value,
+                                                         unsigned Size) {
+  assert((Size == 1 || Size == 2 || Size == 4 || Size == 8) &&
+         "fixed-width data size must be 1, 2, 4, or 8 bytes");
+  assert(TmpBuf && Offset < TmpBuf->Bytes.size() &&
+         Size <= TmpBuf->Bytes.size() - Offset &&
+         "invalid temporary buffer offset");
+  bool IsLittleEndian = CU.getAsmPrinter()->getDataLayout().isLittleEndian();
+  for (unsigned I = 0; I != Size; ++I) {
+    unsigned Byte = IsLittleEndian ? I : Size - I - 1;
+    TmpBuf->Bytes[Offset + I] = Value >> (Byte * 8);
+    if (Offset + I < TmpBuf->Comments.size())
+      TmpBuf->Comments[Offset + I].clear();
+  }
+}
+
 const DIType *DbgVariable::getType() const {
   return getVariable()->getType();
 }
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 9b676f88a0cff..4054817abb95d 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -13,13 +13,16 @@
 #include "DwarfExpression.h"
 #include "DwarfCompileUnit.h"
 #include "llvm/ADT/APInt.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallBitVector.h"
+#include "llvm/ADT/Twine.h"
 #include "llvm/BinaryFormat/Dwarf.h"
 #include "llvm/CodeGen/Register.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/MathExtras.h"
 #include <algorithm>
 
 using namespace llvm;
@@ -530,8 +533,31 @@ bool DwarfExpression::addExpression(
   // and not any other parts of the following DWARF expression.
   assert(!IsEmittingEntryValue && "Can't emit entry value around expression");
 
-  std::optional<DIExpression::ExprOperand> PrevConvertOp;
+  struct LabelOffset {
+    uint64_t ID;
+    uint64_t Offset;
+  };
+  struct BranchFixup {
+    uint64_t LabelID;
+    uint64_t PlaceholderOffset;
+  };
+  constexpr unsigned BranchOffsetByteSize = 2;
+
+  // Iterating over ExprCursor doesn't consume it.
+  bool HasSymbolicBranches =
+      llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
+        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+      });
+
+  SmallVector<LabelOffset, 4> Labels;
+  SmallVector<BranchFixup, 4> Fixups;
+  // Buffer the expression until every label has a byte offset, then patch the
+  // branches.
+  if (HasSymbolicBranches)
+    enableTemporaryBuffer();
 
+  std::optional<DIExpression::ExprOperand> PrevConvertOp;
   while (ExprCursor) {
     auto Op = ExprCursor.take();
     uint64_t OpNum = Op->getOp();
@@ -545,9 +571,27 @@ bool DwarfExpression::addExpression(
     }
 
     switch (OpNum) {
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      if (PrevConvertOp)
+        report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
+                           dwarf::OperationEncodingString(OpNum) +
+                           " without DW_OP_convert support");
+      if (OpNum == dwarf::DW_OP_LLVM_label) {
+        Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
+        break;
+      }
+      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
+                                            : dwarf::DW_OP_skip);
+      Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
+      emitData2(0);
+      break;
     case dwarf::DW_OP_LLVM_arg:
       if (!InsertArg(Op->getArg(0), ExprCursor)) {
         LocationKind = Unknown;
+        if (HasSymbolicBranches)
+          disableTemporaryBuffer();
         return false;
       }
       break;
@@ -579,7 +623,10 @@ bool DwarfExpression::addExpression(
       setSubRegisterPiece(0, 0);
       // Reset the location description kind.
       LocationKind = Unknown;
-      return true;
+      if (!HasSymbolicBranches)
+        return true;
+      // Keep going so we apply the branch fixups before returning.
+      break;
     }
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext: {
@@ -744,9 +791,12 @@ bool DwarfExpression::addExpression(
       break;
     case dwarf::DW_OP_LLVM_implicit_pointer:
       // Handled in DwarfCompileUnit::emitImplicitPointerLocation for
-      // Loc::Single variables. If we reach here, the variable has a
-      // location list or other unsupported path. Drop the
-      // location rather than crashing.
+      // Loc::Single variables. If we reach here, the variable has a location
+      // list or another unsupported path, so stop emitting the expression.
+      // We buffer expressions with symbolic branches, so disable the buffer
+      // before returning.
+      if (HasSymbolicBranches)
+        disableTemporaryBuffer();
       return false;
     default:
       llvm_unreachable("unhandled opcode found in expression");
@@ -757,6 +807,29 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
+  if (HasSymbolicBranches) {
+    for (const BranchFixup &Fixup : Fixups) {
+      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+        return Candidate.ID == Fixup.LabelID;
+      });
+      if (Label == Labels.end())
+        report_fatal_error(Twine("DWARF expression branch to label ") +
+                           Twine(Fixup.LabelID) + " has no matching label");
+
+      int64_t Displacement =
+          static_cast<int64_t>(Label->Offset) -
+          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+      if (!isInt<16>(Displacement))
+        report_fatal_error(Twine("DWARF expression branch offset ") +
+                           Twine(Displacement) + " is outside [-32768, 32767]");
+
+      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                  static_cast<uint16_t>(Displacement));
+    }
+
+    disableTemporaryBuffer();
+    commitTemporaryBuffer();
+  }
   return true;
 }
 
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
index 2525938c78d86..f5c37140a1ef9 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.h
@@ -130,7 +130,11 @@ class DwarfExpression {
   /// Emit a raw unsigned value.
   virtual void emitUnsigned(uint64_t Value) = 0;
 
-  virtual void emitData1(uint8_t Value) = 0;
+  /// Emit the low 1, 2, 4, or 8 bytes of a value in the target byte order.
+  virtual void emitData(uint64_t Value, unsigned Size) = 0;
+
+  void emitData1(uint8_t Value) { emitData(Value, 1); }
+  void emitData2(uint16_t Value) { emitData(Value, 2); }
 
   virtual void emitBaseTypeRef(uint64_t Idx) = 0;
 
@@ -150,6 +154,15 @@ class DwarfExpression {
   /// Commit the data stored in the temporary buffer to the main output.
   virtual void commitTemporaryBuffer() = 0;
 
+  /// Replace a 1-, 2-, 4-, or 8-byte zero placeholder at Offset with the low
+  /// bytes of Value in the target byte order.
+  virtual void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                          unsigned Size) = 0;
+
+  void replaceTemporaryBufferData2(unsigned Offset, uint16_t Value) {
+    replaceTemporaryBufferData(Offset, Value, 2);
+  }
+
   /// Emit a normalized unsigned constant.
   void emitConstu(uint64_t Value);
 
@@ -285,10 +298,14 @@ class DwarfExpression {
 
   /// Emit all remaining operations in the DIExpressionCursor. The
   /// cursor must not contain any DW_OP_LLVM_arg operations.
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   void addExpression(DIExpressionCursor &&Expr);
 
   /// Emit all remaining operations in the DIExpressionCursor.
   /// DW_OP_LLVM_arg operations are resolved by calling (\p InsertArg).
+  /// CodeGen reports an error if a branch offset is outside [-32768, 32767] or
+  /// a deferred DW_OP_LLVM_convert reaches a label, branch, or skip.
   //
   /// \return false if any call to (\p InsertArg) returns false.
   bool addExpression(
@@ -328,13 +345,15 @@ class DebugLocDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
@@ -358,13 +377,15 @@ class DIEDwarfExpression final : public DwarfExpression {
   void emitOp(uint8_t Op, const char *Comment = nullptr) override;
   void emitSigned(int64_t Value) override;
   void emitUnsigned(uint64_t Value) override;
-  void emitData1(uint8_t Value) override;
+  void emitData(uint64_t Value, unsigned Size) override;
   void emitBaseTypeRef(uint64_t Idx) override;
 
   void enableTemporaryBuffer() override;
   void disableTemporaryBuffer() override;
   unsigned getTemporaryBufferSize() override;
   void commitTemporaryBuffer() override;
+  void replaceTemporaryBufferData(unsigned Offset, uint64_t Value,
+                                  unsigned Size) override;
 
   bool isFrameRegister(const TargetRegisterInfo &TRI,
                        llvm::Register MachineReg) override;
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index 78c0769e49161..c1c1218a87974 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -37,6 +37,27 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+static dwarf::Form getDataForm(unsigned Size) {
+  switch (Size) {
+  case 1:
+    return dwarf::DW_FORM_data1;
+  case 2:
+    return dwarf::DW_FORM_data2;
+  case 4:
+    return dwarf::DW_FORM_data4;
+  case 8:
+    return dwarf::DW_FORM_data8;
+  default:
+    llvm_unreachable("fixed-width data size must be 1, 2, 4, or 8 bytes");
+  }
+}
+
+static uint64_t getDataValue(uint64_t Value, unsigned Size) {
+  if (Size == sizeof(Value))
+    return Value;
+  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+}
+
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
                                        DwarfCompileUnit &CU, DIELoc &DIE)
     : DwarfExpression(AP.getDwarfVersion(), CU), AP(AP), OutDIE(DIE) {}
@@ -53,8 +74,8 @@ void DIEDwarfExpression::emitUnsigned(uint64_t Value) {
   CU.addUInt(getActiveDIE(), dwarf::DW_FORM_udata, Value);
 }
 
-void DIEDwarfExpression::emitData1(uint8_t Value) {
-  CU.addUInt(getActiveDIE(), dwarf::DW_FORM_data1, Value);
+void DIEDwarfExpression::emitData(uint64_t Value, unsigned Size) {
+  CU.addUInt(getActiveDIE(), getDataForm(Size), getDataValue(Value, Size));
 }
 
 void DIEDwarfExpression::emitBaseTypeRef(uint64_t Idx) {
@@ -69,11 +90,36 @@ void DIEDwarfExpression::enableTemporaryBuffer() {
 void DIEDwarfExpression::disableTemporaryBuffer() { IsBuffering = false; }
 
 unsigned DIEDwarfExpression::getTemporaryBufferSize() {
-  return TmpDIE.computeSize(AP.getDwarfFormParams());
+  unsigned Size = 0;
+  for (const DIEValue &V : TmpDIE.values())
+    Size += V.sizeOf(AP.getDwarfFormParams());
+  return Size;
 }
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
+                                                    uint64_t Value,
+                                                    unsigned Size) {
+  dwarf::Form Form = getDataForm(Size);
+  // Keep the form so replacing the value doesn't move later labels.
+  unsigned CurrentOffset = 0;
+  for (DIEValue &V : TmpDIE.values()) {
+    unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
+    if (Offset < CurrentOffset + ValueSize) {
+      assert(Offset == CurrentOffset && ValueSize == Size &&
+             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
+             V.getDIEInteger().getValue() == 0 &&
+             "symbolic branch fixup does not match its placeholder");
+      V = DIEValue(V.getAttribute(), V.getForm(),
+                   DIEInteger(getDataValue(Value, Size)));
+      return;
+    }
+    CurrentOffset += ValueSize;
+  }
+  llvm_unreachable("invalid temporary DIE offset");
+}
+
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
                                          llvm::Register MachineReg) {
   return MachineReg == TRI.getFrameRegister(*AP.MF);
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 88f7f2f6240e0..518e4c130e34c 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -13,6 +13,7 @@
 #include "llvm/IR/DebugInfoMetadata.h"
 #include "LLVMContextImpl.h"
 #include "MetadataImpl.h"
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Dwarf.h"
@@ -1750,6 +1751,9 @@ unsigned DIExpression::ExprOperand::getSize() const {
   case dwarf::DW_OP_LLVM_tag_offset:
   case dwarf::DW_OP_LLVM_entry_value:
   case dwarf::DW_OP_LLVM_arg:
+  case dwarf::DW_OP_LLVM_label:
+  case dwarf::DW_OP_LLVM_bra:
+  case dwarf::DW_OP_LLVM_skip:
   case dwarf::DW_OP_regx:
     return 2;
   default:
@@ -1758,12 +1762,81 @@ unsigned DIExpression::ExprOperand::getSize() const {
 }
 
 bool DIExpression::isValid() const {
+  auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
+    auto FirstOp = expr_op_begin();
+    if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
+      ++FirstOp;
+    return EntryValue.get() == FirstOp->get() && EntryValue.getArg(0) == 1;
+  };
+
+  SmallDenseSet<uint64_t, 4> Labels;
+  SmallVector<uint64_t, 4> LabelReferences;
+  bool HasControlFlow = false;
+  bool HasControlFlowConflict = false;
+  bool HasStackValue = false;
+  bool HasFragment = false;
+  bool HasInvalidControlFlowSuffix = false;
+
+  // Collect labels and branch targets before running the checks below, which
+  // may return early.
   for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
     // Check that there's space for the operand.
     if (I->get() + I->getSize() > E->get())
       return false;
 
     uint64_t Op = I->getOp();
+
+    // Only DW_OP_LLVM_fragment may follow DW_OP_stack_value, and nothing may
+    // follow the fragment.
+    HasInvalidControlFlowSuffix |=
+        HasFragment || (HasStackValue && Op != dwarf::DW_OP_LLVM_fragment);
+
+    switch (Op) {
+    case dwarf::DW_OP_LLVM_label:
+      if (!Labels.insert(I->getArg(0)).second)
+        return false;
+      HasControlFlow = true;
+      break;
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
+      LabelReferences.push_back(I->getArg(0));
+      HasControlFlow = true;
+      break;
+    // We don't know the DW_OP_bra and DW_OP_skip offsets until CodeGen, so
+    // DIExpression uses the symbolic ops.
+    case dwarf::DW_OP_bra:
+    case dwarf::DW_OP_skip:
+      return false;
+    case dwarf::DW_OP_LLVM_tag_offset:
+      if (HasControlFlow)
+        return false;
+      break;
+    // These ops use lowering paths that don't support symbolic branches.
+    case dwarf::DW_OP_LLVM_arg:
+    case dwarf::DW_OP_LLVM_implicit_pointer:
+      HasControlFlowConflict = true;
+      break;
+    case dwarf::DW_OP_LLVM_entry_value:
+      HasControlFlowConflict |= !IsEntryValueValid(*I);
+      break;
+    default:
+      break;
+    }
+
+    if (Op == dwarf::DW_OP_stack_value)
+      HasStackValue = true;
+    else if (Op == dwarf::DW_OP_LLVM_fragment)
+      HasFragment = true;
+  }
+
+  if (HasControlFlow && (HasControlFlowConflict || HasInvalidControlFlowSuffix))
+    return false;
+  for (uint64_t Label : LabelReferences)
+    if (!Labels.contains(Label))
+      return false;
+
+  for (auto I = expr_op_begin(), E = expr_op_end(); I != E; ++I) {
+    uint64_t Op = I->getOp();
     if ((Op >= dwarf::DW_OP_reg0 && Op <= dwarf::DW_OP_reg31) ||
         (Op >= dwarf::DW_OP_breg0 && Op <= dwarf::DW_OP_breg31))
       return true;
@@ -1784,7 +1857,7 @@ bool DIExpression::isValid() const {
         return false;
       break;
     }
-    case dwarf::DW_OP_swap: {
+    case dwarf::DW_OP_swap:
       // Must be more than one implicit element on the stack.
 
       // FIXME: A better way to implement this would be to add a local variable
@@ -1795,22 +1868,20 @@ bool DIExpression::isValid() const {
       if (getNumElements() == 1)
         return false;
       break;
-    }
-    case dwarf::DW_OP_LLVM_entry_value: {
+    case dwarf::DW_OP_LLVM_entry_value:
       // An entry value operator must appear at the beginning or immediately
       // following `DW_OP_LLVM_arg 0`, and the number of operations it cover can
       // currently only be 1, because we support only entry values of a simple
       // register location. One reason for this is that we currently can't
       // calculate the size of the resulting DWARF block for other expressions.
-      auto FirstOp = expr_op_begin();
-      if (FirstOp->getOp() == dwarf::DW_OP_LLVM_arg && FirstOp->getArg(0) == 0)
-        ++FirstOp;
-      return I->get() == FirstOp->get() && I->getArg(0) == 1;
-    }
+      return IsEntryValueValid(*I);
     case dwarf::DW_OP_LLVM_implicit_pointer:
     case dwarf::DW_OP_LLVM_convert:
     case dwarf::DW_OP_LLVM_arg:
     case dwarf::DW_OP_LLVM_tag_offset:
+    case dwarf::DW_OP_LLVM_label:
+    case dwarf::DW_OP_LLVM_bra:
+    case dwarf::DW_OP_LLVM_skip:
     case dwarf::DW_OP_LLVM_extract_bits_sext:
     case dwarf::DW_OP_LLVM_extract_bits_zext:
     case dwarf::DW_OP_constu:
@@ -2327,13 +2398,15 @@ DIExpression *DIExpression::appendToStack(const DIExpression *Expr,
   // has no DW_OP_stack_value.
   //
   // Match .* DW_OP_stack_value (DW_OP_LLVM_fragment A B)?.
-  std::optional<FragmentInfo> FI = Expr->getFragmentInfo();
-  unsigned DropUntilStackValue = FI ? 3 : 0;
-  ArrayRef<uint64_t> ExprOpsBeforeFragment =
-      Expr->getElements().drop_back(DropUntilStackValue);
-  bool NeedsDeref = (Expr->getNumElements() > DropUntilStackValue) &&
-                    (ExprOpsBeforeFragment.back() != dwarf::DW_OP_stack_value);
-  bool NeedsStackValue = NeedsDeref || ExprOpsBeforeFragment.empty();
+  std::optional<uint64_t> LastOp;
+  // A raw operand can have the same value as an opcode, so use expr_ops().
+  // Labels and branches can't follow DW_OP_stack_value, so LastOp is still
+  // DW_OP_stack_value when the expression has a value suffix.
+  for (auto Op : Expr->expr_ops())
+    if (Op.getOp() != dwarf::DW_OP_LLVM_fragment)
+      LastOp = Op.getOp();
+  bool NeedsDeref = LastOp && *LastOp != dwarf::DW_OP_stack_value;
+  bool NeedsStackValue = NeedsDeref || !LastOp;
 
   // Append a DW_OP_deref after Expr's current op list if needed, then append
   // the new ops, and finally ensure that a single DW_OP_stack_value is present.
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index b6ee0170904e8..be14be6a43656 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7182,6 +7182,19 @@ void Verifier::visit(DbgVariableRecord &DVR) {
           F);
   visitMDNode(*DVR.getExpression(), AreDebugLocsAllowed::No);
 
+  // A DIArgList can have a valid branch expression which doesn't use
+  // DW_OP_LLVM_arg, so check the record as well.
+  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
+    bool HasControlFlow = llvm::any_of(
+        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
+                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        });
+    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
+            &DVR, MD, DVR.getExpression(), BB, F);
+  }
+
   if (DVR.isDbgAssign()) {
     CheckDI(isa_and_nonnull<DIAssignID>(DVR.getRawAssignID()),
             "invalid #dbg_assign DIAssignID", &DVR, DVR.getRawAssignID(), BB,
diff --git a/llvm/test/Assembler/di-expression-symbolic-branches.ll b/llvm/test/Assembler/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..475594c62d444
--- /dev/null
+++ b/llvm/test/Assembler/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | FileCheck %s
+
+; Labels use IDs in a DIExpression, so make sure the assembler keeps both
+; forward and backward references.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42)
+!0 = !DIExpression(DW_OP_LLVM_label, 0,
+                   DW_OP_LLVM_bra, 42,
+                   DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 42)
diff --git a/llvm/test/Bitcode/di-expression-symbolic-branches.ll b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..82b3c630360fe
--- /dev/null
+++ b/llvm/test/Bitcode/di-expression-symbolic-branches.ll
@@ -0,0 +1,12 @@
+; RUN: llvm-as < %s | llvm-dis | llvm-as | llvm-dis | FileCheck %s
+
+; Label IDs are uint64_t values, so use the largest one while checking that the
+; opcodes and IDs survive two bitcode round-trips.
+
+!named = !{!0}
+
+; CHECK: !DIExpression(DW_OP_LLVM_label, 18446744073709551615, DW_OP_LLVM_bra, 7, DW_OP_LLVM_skip, 18446744073709551615, DW_OP_LLVM_label, 7)
+!0 = !DIExpression(DW_OP_LLVM_label, 18446744073709551615,
+                   DW_OP_LLVM_bra, 7,
+                   DW_OP_LLVM_skip, 18446744073709551615,
+                   DW_OP_LLVM_label, 7)
diff --git a/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
new file mode 100644
index 0000000000000..eda9f210cd378
--- /dev/null
+++ b/llvm/test/DebugInfo/MIR/X86/di-expression-symbolic-branches.mir
@@ -0,0 +1,102 @@
+# RUN: split-file %s %t
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=asm -o - %t/valid.mir | FileCheck %s
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=ENTRY
+# RUN: llc -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/valid.mir | llvm-dwarfdump - | FileCheck %s --check-prefix=NATIVE
+# RUN: not --crash llc -disable-verify -start-after=patchable-function -O0 -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t/missing-label.mir 2>&1 | FileCheck %s --check-prefix=MISSING
+
+# This checks the machine debug-expression path:
+#
+# - DWARF 4 expands conversions before we patch a location-list branch, while
+#   DWARF 5 emits DW_OP_convert.
+# - Entry values keep their branches when we emit an inline expression.
+# - With verification disabled, CodeGen still reports a missing label.
+
+#--- valid.mir
+--- |
+  define void @f() !dbg !5 {
+  entry:
+    ret void, !dbg !12
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9, !10}
+  !9 = !DILocalVariable(name: "before", scope: !5, type: !11)
+  !10 = !DILocalVariable(name: "entry", arg: 1, scope: !5, type: !11)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+  !12 = !DILocation(line: 1, scope: !5)
+...
+---
+name: f
+body: |
+  bb.0.entry:
+    $rcx = MOV64ri 1
+    DBG_VALUE $rdi, $noreg, !10, !DIExpression(DW_OP_LLVM_entry_value, 1, DW_OP_lit0, DW_OP_plus, DW_OP_LLVM_bra, 3, DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 3), debug-location !12
+    DBG_VALUE $rcx, 0, !9, !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_LLVM_convert, 64, DW_ATE_signed, DW_OP_plus_uconst, 1, DW_OP_LLVM_skip, 1), debug-location !12
+    $rcx = MOV64ri 2
+    RET64 debug-location !12
+...
+
+# CHECK: .Ldebug_loc0:
+# CHECK: .byte 114{{.*}}# DW_OP_breg2
+# CHECK-NEXT: .byte 0
+# CHECK-NEXT: .byte 18{{.*}}# DW_OP_dup
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 31
+# CHECK-NEXT: .byte 37{{.*}}# DW_OP_shr
+# CHECK-NEXT: .byte 48{{.*}}# DW_OP_lit0
+# CHECK-NEXT: .byte 32{{.*}}# DW_OP_not
+# CHECK-NEXT: .byte 30{{.*}}# DW_OP_mul
+# CHECK-NEXT: .byte 16{{.*}}# DW_OP_constu
+# CHECK-NEXT: .byte 32
+# CHECK-NEXT: .byte 36{{.*}}# DW_OP_shl
+# CHECK-NEXT: .byte 33{{.*}}# DW_OP_or
+# CHECK-NEXT: .byte 35{{.*}}# DW_OP_plus_uconst
+# CHECK-NEXT: .byte 1
+# CHECK-NEXT: .byte 47{{.*}}# DW_OP_skip
+# CHECK-NEXT: .byte 240
+# CHECK-NEXT: .byte 255
+
+# ENTRY: DW_AT_location (DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_lit0, DW_OP_plus, DW_OP_bra +2, DW_OP_plus_uconst 0x1)
+
+# NATIVE: DW_OP_breg2 RCX+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_plus_uconst 0x1, DW_OP_skip -15
+
+# MISSING: LLVM ERROR: DWARF expression branch to label 2 has no matching label
+
+#--- missing-label.mir
+--- |
+  define void @missing_label() !dbg !5 {
+  entry:
+    ret void, !dbg !10
+  }
+
+  !llvm.dbg.cu = !{!0}
+  !llvm.module.flags = !{!3, !4}
+
+  !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, emissionKind: FullDebug)
+  !1 = !DIFile(filename: "test.c", directory: "/")
+  !3 = !{i32 2, !"Dwarf Version", i32 4}
+  !4 = !{i32 2, !"Debug Info Version", i32 3}
+  !5 = distinct !DISubprogram(name: "missing_label", scope: !1, file: !1, type: !6, spFlags: DISPFlagDefinition, unit: !0, retainedNodes: !8)
+  !6 = !DISubroutineType(types: !7)
+  !7 = !{null}
+  !8 = !{!9}
+  !9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+  !10 = !DILocation(line: 1, scope: !5)
+  !11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+...
+---
+name: missing_label
+body: |
+  bb.0.entry:
+    DBG_VALUE 0, $noreg, !9, !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_bra, 2), debug-location !10
+    RET64 debug-location !10
+...
diff --git a/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..44159954127be
--- /dev/null
+++ b/llvm/test/DebugInfo/PowerPC/di-expression-symbolic-branches.ll
@@ -0,0 +1,44 @@
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -filetype=asm -o - %s | FileCheck %s
+
+; We patch branches in two temporary buffers, so make sure both use the PowerPC
+; byte order: a forward skip in a location list and a backward branch in an
+; inline expression.
+
+define void @f(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_bra, 2), !11)
+  #dbg_value(i64 %x, !10,
+             !DIExpression(DW_OP_LLVM_skip, 3, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_label, 3), !11)
+  call void @clobber(), !dbg !11
+  ret void, !dbg !11
+}
+
+declare void @clobber()
+
+; CHECK: .section .debug_loclists
+; CHECK: .byte 47{{.*}}# DW_OP_skip
+; CHECK-NEXT: .byte 0
+; CHECK-NEXT: .byte 2
+; CHECK: .byte 5{{.*}}# DW_AT_location
+; CHECK-NEXT: .byte 48
+; CHECK-NEXT: .byte 40
+; CHECK-NEXT: .short 65533
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "backward", scope: !5, type: !12)
+!10 = !DILocalVariable(name: "value", arg: 1, scope: !5, type: !12)
+!11 = !DILocation(line: 1, column: 1, scope: !5)
+!12 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
new file mode 100644
index 0000000000000..fc7e0246a6ae2
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/Inputs/generate-di-expression-symbolic-branch-out-of-range.py
@@ -0,0 +1,33 @@
+import sys
+
+# Put 32768 one-byte DW_OP_dup operations between the branch and its label,
+# which gives us the first offset that won't fit.
+DUPLICATE_OPS = "DW_OP_dup, " * 32768
+
+sys.stdout.write(
+    f"""\
+define void @f() !dbg !5 {{
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, {DUPLICATE_OPS}
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}}
+
+!llvm.dbg.cu = !{{!0}}
+!llvm.module.flags = !{{!3}}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{{i32 2, !"Debug Info Version", i32 3}}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{{null}}
+!9 = !DILocalVariable(name: "x", scope: !5, type: !11)
+!10 = !DILocation(line: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+"""
+)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
new file mode 100644
index 0000000000000..5cb9861502585
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-convert-boundary.ll
@@ -0,0 +1,96 @@
+; RUN: split-file %s %t
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/skip.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=SKIP-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/skip.ll 2>&1 | FileCheck %s --check-prefix=SKIP-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/bra.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=BRA-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/bra.ll 2>&1 | FileCheck %s --check-prefix=BRA-LEGACY
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=5 -filetype=obj -o - %t/label.ll | llvm-dwarfdump -v - | FileCheck %s --check-prefix=LABEL-NATIVE
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o /dev/null %t/label.ll 2>&1 | FileCheck %s --check-prefix=LABEL-LEGACY
+
+; DWARF 5 emits DW_OP_convert directly, so labels and branches can appear
+; between conversions. With DWARF 4 we may defer one conversion until the next;
+; if a label, branch, or skip splits the pair, CodeGen reports an error.
+
+; SKIP-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_skip +5, DW_OP_convert {{.*}} "DW_ATE_signed_64")
+; SKIP-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_skip without DW_OP_convert support
+
+; BRA-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_dup, DW_OP_bra +5, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; BRA-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_bra without DW_OP_convert support
+
+; LABEL-NATIVE: DW_AT_location [DW_FORM_exprloc] (DW_OP_breg5 RDI+0, DW_OP_convert {{.*}} "DW_ATE_signed_32")
+; LABEL-LEGACY: LLVM ERROR: cannot lower DW_OP_LLVM_convert across DW_OP_LLVM_label without DW_OP_convert support
+
+;--- skip.ll
+define void @skip(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_skip, 1,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 1), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "skip", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "skip", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- label.ll
+define void @label(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 3), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "label", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "label", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+
+;--- bra.ll
+define void @bra(i64 %x) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9,
+             !DIExpression(DW_OP_LLVM_convert, 32, DW_ATE_signed, DW_OP_dup,
+                           DW_OP_LLVM_bra, 2,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_label, 2), !10)
+  ret void, !dbg !10
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "bra", scope: !1, type: !6,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!9 = !DILocalVariable(name: "bra", arg: 1, scope: !5, type: !11)
+!10 = !DILocation(line: 1, column: 1, scope: !5)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
new file mode 100644
index 0000000000000..677032dc9c7a7
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-out-of-range.ll
@@ -0,0 +1,7 @@
+; RUN: %python %S/Inputs/generate-di-expression-symbolic-branch-out-of-range.py > %t.ll
+; RUN: not --crash llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o /dev/null %t.ll 2>&1 | FileCheck %s
+
+; The first positive offset that doesn't fit is 32768, so make sure CodeGen
+; reports it instead of truncating it.
+
+; CHECK: LLVM ERROR: DWARF expression branch offset 32768 is outside [-32768, 32767]
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..163c3c24c6927
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branches.ll
@@ -0,0 +1,84 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump -v - | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -dwarf-version=4 -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s --check-prefix=LEGACY
+
+; Branch offsets come from the emitted bytes, so check:
+;
+; - zero, forward, backward, and cyclic branches;
+; - ops which grow during lowering;
+; - labels on either side of an emitted op; and
+; - the fragment return path and DWARF 4 convert expansion.
+
+define void @f() !dbg !5 {
+entry:
+  #dbg_value(i64 0, !9,
+             !DIExpression(DW_OP_LLVM_bra, 1, DW_OP_LLVM_label, 1), !18)
+  #dbg_value(i64 0, !10,
+             !DIExpression(DW_OP_LLVM_label, 2, DW_OP_LLVM_skip, 2), !18)
+  #dbg_value(i64 0, !11,
+             !DIExpression(DW_OP_LLVM_label, 3, DW_OP_LLVM_bra, 4,
+                           DW_OP_LLVM_skip, 3, DW_OP_LLVM_label, 4), !18)
+  #dbg_value(i64 0, !12,
+             !DIExpression(DW_OP_LLVM_bra, 5, DW_OP_deref_size, 1,
+                           DW_OP_plus_uconst, 128,
+                           DW_OP_LLVM_extract_bits_sext, 4, 4,
+                           DW_OP_LLVM_label, 5), !18)
+  #dbg_value(i64 0, !13,
+             !DIExpression(DW_OP_LLVM_skip, 6,
+                           DW_OP_LLVM_convert, 32, DW_ATE_signed,
+                           DW_OP_LLVM_convert, 64, DW_ATE_signed,
+                           DW_OP_LLVM_label, 6), !18)
+  #dbg_value(i64 0, !14,
+             !DIExpression(DW_OP_LLVM_bra, 7, DW_OP_LLVM_label, 7,
+                           DW_OP_stack_value, DW_OP_LLVM_fragment, 0, 32), !18)
+  #dbg_value(i64 0, !15,
+             !DIExpression(DW_OP_plus_uconst, 1, DW_OP_LLVM_label, 8,
+                           DW_OP_LLVM_skip, 8), !18)
+  #dbg_value(i64 0, !16,
+             !DIExpression(DW_OP_LLVM_label, 9, DW_OP_plus_uconst, 1,
+                           DW_OP_LLVM_skip, 9), !18)
+  ret void, !dbg !18
+}
+
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"zero"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"backward"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +3, DW_OP_skip -6, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"cycle"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +11, DW_OP_deref_size 0x1, DW_OP_plus_uconst 0x80, DW_OP_constu 0x38, DW_OP_shl, DW_OP_constu 0x3c, DW_OP_shra, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"expanded"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_skip +10, DW_OP_convert {{.*}} "DW_ATE_signed_32", DW_OP_convert {{.*}} "DW_ATE_signed_64", DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"convert"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_bra +0, DW_OP_stack_value, DW_OP_piece 0x4)
+; CHECK: DW_AT_name{{.*}}"fragment"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -3, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_after_offset"
+; CHECK: DW_AT_location [DW_FORM_exprloc] (DW_OP_lit0, DW_OP_plus_uconst 0x1, DW_OP_skip -5, DW_OP_stack_value)
+; CHECK: DW_AT_name{{.*}}"label_before_offset"
+
+; LEGACY: DW_AT_location (DW_OP_lit0, DW_OP_skip +11, DW_OP_dup, DW_OP_constu 0x1f, DW_OP_shr, DW_OP_lit0, DW_OP_not, DW_OP_mul, DW_OP_constu 0x20, DW_OP_shl, DW_OP_or, DW_OP_stack_value)
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3, !4}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Dwarf Version", i32 5}
+!4 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "f", scope: !1, file: !1, type: !6,
+                            spFlags: DISPFlagDefinition,
+                            unit: !0, retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !10, !11, !12, !13, !14, !15, !16}
+!9 = !DILocalVariable(name: "zero", scope: !5, type: !19)
+!10 = !DILocalVariable(name: "backward", scope: !5, type: !19)
+!11 = !DILocalVariable(name: "cycle", scope: !5, type: !19)
+!12 = !DILocalVariable(name: "expanded", scope: !5, type: !19)
+!13 = !DILocalVariable(name: "convert", scope: !5, type: !19)
+!14 = !DILocalVariable(name: "fragment", scope: !5, type: !19)
+!15 = !DILocalVariable(name: "label_after_offset", scope: !5, type: !19)
+!16 = !DILocalVariable(name: "label_before_offset", scope: !5, type: !19)
+!18 = !DILocation(line: 1, scope: !5)
+!19 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
diff --git a/llvm/test/Verifier/di-expression-symbolic-branches.ll b/llvm/test/Verifier/di-expression-symbolic-branches.ll
new file mode 100644
index 0000000000000..db0e0dacdf32f
--- /dev/null
+++ b/llvm/test/Verifier/di-expression-symbolic-branches.ll
@@ -0,0 +1,93 @@
+; RUN: split-file %s %t
+; RUN: opt -passes=verify -disable-output %t/valid.ll
+; RUN: not opt -passes=verify -disable-output %t/arity.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/duplicate.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/missing.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/raw-after-register.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/incompatible.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/location-arg.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/tag-ordering.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+; RUN: not opt -passes=verify -disable-output %t/terminal.ll 2>&1 | FileCheck %s --check-prefix=INVALID --implicit-check-not="invalid expression"
+
+; Check the verifier rules separately:
+;
+; - a label needs an ID;
+; - labels are unique and each branch target exists;
+; - raw branches and incompatible ops are rejected; and
+; - tag_offset stays before control flow, which stays before stack_value.
+
+; DIArgList doesn't support symbolic branches, but normal IR loading drops the
+; bad debug info, so opt still succeeds.
+; RUN: opt -passes=verify -disable-output %t/arg-list.ll 2>&1 | FileCheck %s --check-prefix=ARG-LIST
+
+; INVALID: invalid expression
+; ARG-LIST: DIArgList doesn't support symbolic branches
+; ARG-LIST: warning: ignoring invalid debug info
+
+;--- valid.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 0, DW_OP_LLVM_skip, 0,
+                   DW_OP_LLVM_label, 0)
+
+;--- arity.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label)
+
+;--- duplicate.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_label, 1)
+
+;--- missing.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_bra, 1)
+
+;--- raw.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_bra, 0)
+
+;--- raw-after-register.ll
+; A register normally ends validation, but it must not hide a raw branch later
+; in the expression.
+!named = !{!0}
+!0 = !DIExpression(DW_OP_reg0, DW_OP_skip, 0)
+
+;--- incompatible.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_implicit_pointer, DW_OP_LLVM_label, 1)
+
+;--- location-arg.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_arg, 0, DW_OP_LLVM_skip, 1,
+                   DW_OP_LLVM_label, 1)
+
+;--- tag-ordering.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_LLVM_label, 1, DW_OP_LLVM_tag_offset, 0)
+
+;--- terminal.ll
+!named = !{!0}
+!0 = !DIExpression(DW_OP_stack_value, DW_OP_LLVM_label, 1)
+
+;--- arg-list.ll
+; The expression is valid by itself, so this diagnostic comes from DIArgList.
+define void @f(i32 %x) !dbg !4 {
+entry:
+  #dbg_value(!DIArgList(i32 %x), !7,
+             !DIExpression(DW_OP_LLVM_skip, 1, DW_OP_LLVM_label, 1), !8)
+  ret void, !dbg !8
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1, type: !5,
+                            spFlags: DISPFlagDefinition, unit: !0)
+!5 = !DISubroutineType(types: !6)
+!6 = !{null}
+!7 = !DILocalVariable(name: "x", scope: !4)
+!8 = !DILocation(line: 1, column: 1, scope: !4)
diff --git a/llvm/unittests/BinaryFormat/DwarfTest.cpp b/llvm/unittests/BinaryFormat/DwarfTest.cpp
index ba7d59182ea53..6ffb5a921510c 100644
--- a/llvm/unittests/BinaryFormat/DwarfTest.cpp
+++ b/llvm/unittests/BinaryFormat/DwarfTest.cpp
@@ -47,6 +47,13 @@ TEST(DwarfTest, getOperationEncoding) {
   // Some valid ops.
   EXPECT_EQ(DW_OP_deref, getOperationEncoding("DW_OP_deref"));
   EXPECT_EQ(DW_OP_bit_piece, getOperationEncoding("DW_OP_bit_piece"));
+  // These are metadata-only ops, but they still need names in both directions.
+  EXPECT_EQ(DW_OP_LLVM_label, getOperationEncoding("DW_OP_LLVM_label"));
+  EXPECT_EQ(DW_OP_LLVM_bra, getOperationEncoding("DW_OP_LLVM_bra"));
+  EXPECT_EQ(DW_OP_LLVM_skip, getOperationEncoding("DW_OP_LLVM_skip"));
+  EXPECT_EQ("DW_OP_LLVM_label", OperationEncodingString(DW_OP_LLVM_label));
+  EXPECT_EQ("DW_OP_LLVM_bra", OperationEncodingString(DW_OP_LLVM_bra));
+  EXPECT_EQ("DW_OP_LLVM_skip", OperationEncodingString(DW_OP_LLVM_skip));
 
   // Invalid ops.
   EXPECT_EQ(0u, getOperationEncoding("DW_OP_otherthings"));
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index 65398b5ca2b1f..9be5384f9647b 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -4251,6 +4251,86 @@ TEST_F(DIExpressionTest, Append) {
   EXPECT_EQ(ResExpr, AppendExpr);
 }
 
+TEST_F(DIExpressionTest, SymbolicBranchRewrites) {
+  // A symbolic op has an ID operand, so fold the math on both sides without
+  // consuming it.
+  for (uint64_t Barrier : {dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                           dwarf::DW_OP_LLVM_skip}) {
+    SmallVector<uint64_t> Ops = {dwarf::DW_OP_constu,
+                                 1,
+                                 dwarf::DW_OP_constu,
+                                 2,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 4,
+                                 Barrier,
+                                 7,
+                                 dwarf::DW_OP_constu,
+                                 5,
+                                 dwarf::DW_OP_plus,
+                                 dwarf::DW_OP_constu,
+                                 6,
+                                 dwarf::DW_OP_constu,
+                                 7,
+                                 dwarf::DW_OP_plus};
+    SmallVector<uint64_t> Expected = {
+        dwarf::DW_OP_constu,      3, dwarf::DW_OP_constu, 4, Barrier, 7,
+        dwarf::DW_OP_plus_uconst, 5, dwarf::DW_OP_constu, 13};
+    if (Barrier != dwarf::DW_OP_LLVM_label) {
+      Ops.append({dwarf::DW_OP_LLVM_label, 7});
+      Expected.append({dwarf::DW_OP_LLVM_label, 7});
+    }
+    EXPECT_EQ(DIExpression::get(Context, Expected),
+              DIExpression::get(Context, Ops)->foldConstantMath());
+  }
+
+  // Adding ops before or after an expression leaves its labels in place.
+  SmallVector<uint64_t> Ops = {dwarf::DW_OP_LLVM_label, 7,
+                               dwarf::DW_OP_plus_uconst, 1};
+  auto *Expr = DIExpression::get(Context, Ops);
+  auto *Prepended = DIExpression::prepend(Expr, DIExpression::DerefBefore, 0);
+  SmallVector<uint64_t> Expected = {dwarf::DW_OP_deref, dwarf::DW_OP_LLVM_label,
+                                    7, dwarf::DW_OP_plus_uconst, 1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+
+  SmallVector<uint64_t> Prefix = {dwarf::DW_OP_LLVM_tag_offset, 3};
+  Prepended = DIExpression::prependOpcodes(Expr, Prefix);
+  Expected = {dwarf::DW_OP_LLVM_tag_offset, 3, dwarf::DW_OP_LLVM_label, 7,
+              dwarf::DW_OP_plus_uconst,     1};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Prepended);
+  EXPECT_TRUE(Prepended->isValid());
+
+  SmallVector<uint64_t> AppendOps = {
+      dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+      dwarf::DW_OP_LLVM_convert, 64, dwarf::DW_ATE_signed};
+  auto *Appended = DIExpression::append(Expr, AppendOps);
+  Expected = {dwarf::DW_OP_LLVM_label,
+              7,
+              dwarf::DW_OP_plus_uconst,
+              1,
+              dwarf::DW_OP_LLVM_convert,
+              32,
+              dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert,
+              64,
+              dwarf::DW_ATE_signed};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+
+  // Label IDs are arbitrary uint64_t values, so appendExt must not mistake one
+  // for DW_OP_stack_value.
+  uint64_t LabelID = dwarf::DW_OP_stack_value;
+  Ops = {dwarf::DW_OP_LLVM_label, LabelID};
+  Appended =
+      DIExpression::appendExt(DIExpression::get(Context, Ops), 32, 64, true);
+  Expected = {dwarf::DW_OP_LLVM_label,   LabelID, dwarf::DW_OP_deref,
+              dwarf::DW_OP_LLVM_convert, 32,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_LLVM_convert, 64,      dwarf::DW_ATE_signed,
+              dwarf::DW_OP_stack_value};
+  EXPECT_EQ(DIExpression::get(Context, Expected), Appended);
+  EXPECT_TRUE(Appended->isValid());
+}
+
 TEST_F(DIExpressionTest, isValid) {
 #define EXPECT_VALID(...)                                                      \
   do {                                                                         \
@@ -4271,6 +4351,7 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_VALID(dwarf::DW_OP_constu, 6, dwarf::DW_OP_plus);
   EXPECT_VALID(dwarf::DW_OP_constu, 5, dwarf::DW_OP_swap);
   EXPECT_VALID(dwarf::DW_OP_deref);
+  EXPECT_VALID(dwarf::DW_OP_stack_value);
   EXPECT_VALID(dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_plus_uconst, 6, dwarf::DW_OP_deref);
   EXPECT_VALID(dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst, 6);
@@ -4279,12 +4360,31 @@ TEST_F(DIExpressionTest, isValid) {
                dwarf::DW_OP_LLVM_fragment, 3, 7);
   EXPECT_VALID(dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_VALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_entry_value, 1);
+  // A label may be unused, and branches may refer forward or backward.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 2, dwarf::DW_OP_LLVM_label, 2);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 3, dwarf::DW_OP_LLVM_label, 4);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 4, dwarf::DW_OP_LLVM_skip, 5,
+               dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_LLVM_skip, 4);
+  // Label IDs can use the full uint64_t range.
+  EXPECT_VALID(
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max() - 1,
+      dwarf::DW_OP_LLVM_label, std::numeric_limits<uint64_t>::max(),
+      dwarf::DW_OP_LLVM_skip, std::numeric_limits<uint64_t>::max() - 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 5, dwarf::DW_OP_stack_value);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_label, 6, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_tag_offset, 1, dwarf::DW_OP_LLVM_label, 6,
+               dwarf::DW_OP_LLVM_skip, 6, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_stack_value,
+               dwarf::DW_OP_LLVM_fragment, 0, 32);
 
   // Invalid constructions.
   EXPECT_INVALID(~0u);
   EXPECT_INVALID(dwarf::DW_OP_plus, 0);
   EXPECT_INVALID(dwarf::DW_OP_plus_uconst);
   EXPECT_INVALID(dwarf::DW_OP_swap);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_deref);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 3, 7, dwarf::DW_OP_plus_uconst, 3);
@@ -4294,6 +4394,48 @@ TEST_F(DIExpressionTest, isValid) {
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_plus_uconst, 5,
                  dwarf::DW_OP_LLVM_entry_value, 1);
   EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 1, dwarf::DW_OP_LLVM_entry_value, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_bra, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_skip, 1);
+  EXPECT_INVALID(dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_bra, 0);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_skip, 0);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_arg, 0, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_implicit_pointer, dwarf::DW_OP_LLVM_label,
+                 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_tag_offset, 0);
+  // Converts and other stack ops can appear around labels and branches.
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_skip, 7, dwarf::DW_OP_LLVM_label, 7);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_skip, 1, dwarf::DW_OP_LLVM_convert, 32,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_convert, 64,
+               dwarf::DW_ATE_signed, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_VALID(dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_dup, dwarf::DW_OP_LLVM_bra, 1,
+               dwarf::DW_OP_LLVM_convert, 32, dwarf::DW_ATE_signed,
+               dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_stack_value, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_fragment, 0, 32, dwarf::DW_OP_LLVM_label, 1);
+  // reg0 and entry_value normally end validation early, but they must not hide
+  // invalid control flow later in the expression.
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_LLVM_bra, 1, dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_LLVM_label, 1);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_LLVM_fragment, 0, 32,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_LLVM_entry_value, 1, dwarf::DW_OP_LLVM_bra, 1,
+                 dwarf::DW_OP_LLVM_label, 1, dwarf::DW_OP_stack_value,
+                 dwarf::DW_OP_plus);
+  EXPECT_INVALID(dwarf::DW_OP_reg0, dwarf::DW_OP_LLVM_label, 1,
+                 dwarf::DW_OP_LLVM_entry_value, 1);
 
 #undef EXPECT_VALID
 #undef EXPECT_INVALID
@@ -4371,6 +4513,19 @@ TEST_F(DIExpressionTest, createFragmentExpression) {
   EXPECT_INVALID_FRAGMENT(0, 32, dwarf::DW_OP_deref, dwarf::DW_OP_plus_uconst,
                           2, dwarf::DW_OP_stack_value);
 
+  // Creating a fragment leaves the branches alone and adds the fragment at the
+  // end.
+  SmallVector<uint64_t> ControlFlowOps = {dwarf::DW_OP_LLVM_label, 1,
+                                          dwarf::DW_OP_LLVM_bra,   1,
+                                          dwarf::DW_OP_LLVM_skip,  1};
+  DIExpression *ControlFlowExpr = DIExpression::get(Context, ControlFlowOps);
+  auto Fragment =
+      DIExpression::createFragmentExpression(ControlFlowExpr, 0, 32);
+  ASSERT_TRUE(Fragment.has_value());
+  ControlFlowOps.append({dwarf::DW_OP_LLVM_fragment, 0, 32});
+  EXPECT_EQ(DIExpression::get(Context, ControlFlowOps), *Fragment);
+  EXPECT_TRUE((*Fragment)->isValid());
+
 #undef EXPECT_VALID_FRAGMENT
 #undef EXPECT_INVALID_FRAGMENT
 }
diff --git a/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
new file mode 100644
index 0000000000000..7ba2b0802fa3c
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/Import/debug-info-symbolic-branches.ll
@@ -0,0 +1,23 @@
+; RUN: mlir-translate -import-llvm -mlir-print-debuginfo -emit-expensive-warnings %s 2>&1 | FileCheck %s
+
+; Import a forward branch and a backward skip, and make sure MLIR keeps the op
+; order and label IDs.
+
+; CHECK: llvm.intr.dbg.value {{.*}} #llvm.di_expression<[DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0), DW_OP_LLVM_label(42)]> = {{.*}} : i64
+define void @f(i64 %x) !dbg !4 {
+  #dbg_value(i64 %x, !DILocalVariable(scope: !4),
+             !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42,
+                           DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42),
+             !DILocation(scope: !4))
+  ret void
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!4 = distinct !DISubprogram(name: "f", scope: !1,
+                            type: !DISubroutineType(types: !{null}),
+                            spFlags: DISPFlagDefinition, unit: !0)
diff --git a/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
new file mode 100644
index 0000000000000..74a549b952327
--- /dev/null
+++ b/mlir/test/Target/LLVMIR/llvmir-debug-symbolic-branches.mlir
@@ -0,0 +1,19 @@
+// RUN: mlir-translate -mlir-to-llvmir %s | FileCheck %s
+
+// Export a forward branch and a backward skip, and make sure LLVM IR keeps the
+// op order and label IDs.
+
+#file = #llvm.di_file<"test.c" in "/">
+#cu = #llvm.di_compile_unit<id = distinct[0]<>, sourceLanguage = DW_LANG_C,
+                            file = #file>
+#sp = #llvm.di_subprogram<compileUnit = #cu, scope = #file, name = "f",
+                          subprogramFlags = "Definition",
+                          type = #llvm.di_subroutine_type<types = #llvm.di_null_type>>
+
+// CHECK: #dbg_value(i64 %{{.*}}, !{{.*}}, !DIExpression(DW_OP_LLVM_label, 0, DW_OP_LLVM_bra, 42, DW_OP_LLVM_skip, 0, DW_OP_LLVM_label, 42), !{{.*}})
+llvm.func @f(%arg: i64) {
+  llvm.intr.dbg.value #llvm.di_local_variable<scope = #sp> #llvm.di_expression<[
+    DW_OP_LLVM_label(0), DW_OP_LLVM_bra(42), DW_OP_LLVM_skip(0),
+    DW_OP_LLVM_label(42)]> = %arg : i64
+  llvm.return
+} loc(fused<#sp>["test.c":1:1])

>From 57cf90ad67b0e05c411e79518212f90e3faa39bc Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:53:45 -0700
Subject: [PATCH 2/8] [LLDB] Handle symbolic DIExpression opcodes

These opcodes only appear in LLVM metadata, but adding them to the shared DWARF enum makes LLDB's switch incomplete. List them with the other metadata-only opcodes so a -Wswitch build stays clean.
---
 lldb/source/Expression/DWARFExpression.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lldb/source/Expression/DWARFExpression.cpp b/lldb/source/Expression/DWARFExpression.cpp
index 299dbe97d6176..73bdd18d82cae 100644
--- a/lldb/source/Expression/DWARFExpression.cpp
+++ b/lldb/source/Expression/DWARFExpression.cpp
@@ -205,6 +205,9 @@ GetOpcodeDataSize(const DataExtractor &data, const lldb::offset_t data_offset,
   case DW_OP_LLVM_arg:
   case DW_OP_LLVM_extract_bits_sext:
   case DW_OP_LLVM_extract_bits_zext:
+  case DW_OP_LLVM_label:
+  case DW_OP_LLVM_bra:
+  case DW_OP_LLVM_skip:
     break;
   // Vendor extensions:
   case DW_OP_HP_is_value:

>From e1e1ba8492cb9a32cb924caadb78ff4129cc9581 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 12:58:04 -0700
Subject: [PATCH 3/8] [DebugInfo] Clarify symbolic branch limitations

Separate the DIExpression validity rules from the combinations we don't handle yet. DIArgList and DW_OP_LLVM_arg need more work in expressions and expression writers, while DW_OP_LLVM_implicit_pointer takes a different emission path. Also document the final branch offset range check.
---
 llvm/docs/SourceLevelDebugging.md | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/llvm/docs/SourceLevelDebugging.md b/llvm/docs/SourceLevelDebugging.md
index 645e441e6fe2a..834781444c607 100644
--- a/llvm/docs/SourceLevelDebugging.md
+++ b/llvm/docs/SourceLevelDebugging.md
@@ -492,15 +492,23 @@ Label IDs are local to an expression:
   offset.
 - Branches can go forward, backward, to themselves, or form cycles.
 
-There are a few other restrictions:
+`DIExpression` validation also checks the following:
 
 - Put labels, branches, and skips before `DW_OP_stack_value` and
   `DW_OP_LLVM_fragment`. Only a fragment can follow `DW_OP_stack_value`.
-- `DIArgList`, `DW_OP_LLVM_arg`, and `DW_OP_LLVM_implicit_pointer` are lowered
-  separately, so they can't be used with symbolic control flow.
 - Put `DW_OP_LLVM_tag_offset` before the first label, branch, or skip so it
   applies to every path.
 
+There are also a couple of cases we don't handle yet:
+
+- `DIArgList` and `DW_OP_LLVM_arg` are currently rejected. CodeGen expands each
+  argument before it resolves the label offsets, so there isn't a representation
+  problem here; it mostly needs work handling it in expressions and expression
+  writers.
+- `DW_OP_LLVM_implicit_pointer` bypasses normal expression emission and only
+  handles a single location today. Supporting branches there is a bit more
+  work, since we'll need to work it back into our normal emission order.
+
 We don't check reachability, termination, or stack state where paths meet.
 
 `DW_OP_LLVM_convert` can appear before or after labels, branches, and skips,
@@ -508,6 +516,9 @@ and we don't match conversions on different paths. When CodeGen can't emit
 `DW_OP_convert`, it may defer one conversion until it sees the next; if a label,
 branch, or skip would split the pair, CodeGen reports an error.
 
+CodeGen also reports an error if the final branch offset is outside
+`[-32768, 32767]`.
+
 Local expression rewrites stop at labels, branches, and skips; they can still
 add operations to either end, but they don't move, remove, or copy labels.
 

>From 7d3fcc07c09bcf1d8150f79394720988d8810aae Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 14:14:38 -0700
Subject: [PATCH 4/8] [DebugInfo] Clean up symbolic DIExpression branch
 handling

Pull the symbolic branch checks into two small helpers, since CodeGen only needs branches while the verifier also needs labels.

Use maskTrailingOnes when storing fixed-width DIE values, and make the placeholder walk use a byte cursor with names that describe what we're patching. We still keep the existing form and size so later fixup offsets don't move.

Also make the displacement calculation and DIArgList verifier check a little clearer. There should be no behavior change here.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h        | 10 +++++
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    |  5 ++-
 llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp     | 42 ++++++++++---------
 llvm/lib/IR/Verifier.cpp                      | 16 +++----
 4 files changed, 44 insertions(+), 29 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index c3cff08eccb5c..177c50be7b374 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,6 +1108,16 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
+/// Return true if Op is a symbolic branch to a label.
+inline bool isSymbolicBranchOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
+}
+
+/// Return true if Op is a symbolic label or branch.
+inline bool isSymbolicControlFlowOp(uint64_t Op) {
+  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
+}
+
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 4054817abb95d..f90edb1e68157 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,8 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-               Op.getOp() == dwarf::DW_OP_LLVM_skip;
+        return dwarf::isSymbolicBranchOp(Op.getOp());
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -816,6 +815,8 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("DWARF expression branch to label ") +
                            Twine(Fixup.LabelID) + " has no matching label");
 
+      // DW_OP_bra and DW_OP_skip apply the displacement after reading their
+      // two-byte operand, so use the byte after the placeholder as the base.
       int64_t Displacement =
           static_cast<int64_t>(Label->Offset) -
           static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
index c1c1218a87974..ea2a6ddca3c61 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp
@@ -27,6 +27,7 @@
 #include "llvm/MC/MCSection.h"
 #include "llvm/MC/MCStreamer.h"
 #include "llvm/Support/Casting.h"
+#include "llvm/Support/MathExtras.h"
 #include "llvm/Target/TargetLoweringObjectFile.h"
 #include <cassert>
 #include <cstdint>
@@ -53,9 +54,7 @@ static dwarf::Form getDataForm(unsigned Size) {
 }
 
 static uint64_t getDataValue(uint64_t Value, unsigned Size) {
-  if (Size == sizeof(Value))
-    return Value;
-  return Value & ((uint64_t(1) << (Size * 8)) - 1);
+  return Value & maskTrailingOnes<uint64_t>(Size * 8);
 }
 
 DIEDwarfExpression::DIEDwarfExpression(const AsmPrinter &AP,
@@ -98,26 +97,31 @@ unsigned DIEDwarfExpression::getTemporaryBufferSize() {
 
 void DIEDwarfExpression::commitTemporaryBuffer() { OutDIE.takeValues(TmpDIE); }
 
-void DIEDwarfExpression::replaceTemporaryBufferData(unsigned Offset,
-                                                    uint64_t Value,
-                                                    unsigned Size) {
-  dwarf::Form Form = getDataForm(Size);
-  // Keep the form so replacing the value doesn't move later labels.
-  unsigned CurrentOffset = 0;
+void DIEDwarfExpression::replaceTemporaryBufferData(unsigned PlaceholderOffset,
+                                                    uint64_t Replacement,
+                                                    unsigned PlaceholderSize) {
+  // Walk the encoded values until the cursor reaches the placeholder.
+  unsigned ByteCursor = 0;
   for (DIEValue &V : TmpDIE.values()) {
     unsigned ValueSize = V.sizeOf(AP.getDwarfFormParams());
-    if (Offset < CurrentOffset + ValueSize) {
-      assert(Offset == CurrentOffset && ValueSize == Size &&
-             V.getType() == DIEValue::isInteger && V.getForm() == Form &&
-             V.getDIEInteger().getValue() == 0 &&
-             "symbolic branch fixup does not match its placeholder");
-      V = DIEValue(V.getAttribute(), V.getForm(),
-                   DIEInteger(getDataValue(Value, Size)));
-      return;
+    unsigned ValueEnd = ByteCursor + ValueSize;
+    if (ValueEnd <= PlaceholderOffset) {
+      ByteCursor = ValueEnd;
+      continue;
     }
-    CurrentOffset += ValueSize;
+
+    // Replace the whole zero placeholder and keep its form, since changing its
+    // encoded size would move every later fixup.
+    assert(PlaceholderOffset == ByteCursor && ValueSize == PlaceholderSize &&
+           V.getType() == DIEValue::isInteger &&
+           V.getForm() == getDataForm(PlaceholderSize) &&
+           V.getDIEInteger().getValue() == 0 &&
+           "symbolic branch fixup does not match its placeholder");
+    V = DIEValue(V.getAttribute(), V.getForm(),
+                 DIEInteger(getDataValue(Replacement, PlaceholderSize)));
+    return;
   }
-  llvm_unreachable("invalid temporary DIE offset");
+  llvm_unreachable("temporary DIE placeholder not found");
 }
 
 bool DIEDwarfExpression::isFrameRegister(const TargetRegisterInfo &TRI,
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index be14be6a43656..0dc59163fa864 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7184,15 +7184,15 @@ void Verifier::visit(DbgVariableRecord &DVR) {
 
   // A DIArgList can have a valid branch expression which doesn't use
   // DW_OP_LLVM_arg, so check the record as well.
-  if (DVR.hasArgList() && DVR.getExpression()->isValid()) {
-    bool HasControlFlow = llvm::any_of(
-        DVR.getExpression()->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return Op.getOp() == dwarf::DW_OP_LLVM_label ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_bra ||
-                 Op.getOp() == dwarf::DW_OP_LLVM_skip;
+  const DIExpression *Expr = DVR.getExpression();
+  if (DVR.hasArgList() && Expr->isValid()) {
+    bool HasSymbolicControlFlow =
+        llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
+          return dwarf::isSymbolicControlFlowOp(Op.getOp());
         });
-    CheckDI(!HasControlFlow, "DIArgList doesn't support symbolic branches",
-            &DVR, MD, DVR.getExpression(), BB, F);
+    CheckDI(!HasSymbolicControlFlow,
+            "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,
+            F);
   }
 
   if (DVR.isDbgAssign()) {

>From c173ccf8a312d19de4b574813c9df6f2cda66671 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Tue, 21 Jul 2026 18:17:17 -0700
Subject: [PATCH 5/8] [DebugInfo] Add opcode queries to DIExpression operands

Add is and isOneOf to DIExpression::ExprOperand and use them for the
symbolic branch checks. This keeps the checks on the parsed operation and
lets us remove the branch-specific helpers from Dwarf.h.
---
 llvm/include/llvm/BinaryFormat/Dwarf.h          | 10 ----------
 llvm/include/llvm/IR/DebugInfoMetadata.h        |  9 +++++++++
 llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp |  8 ++++----
 llvm/lib/IR/Verifier.cpp                        |  3 ++-
 4 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/llvm/include/llvm/BinaryFormat/Dwarf.h b/llvm/include/llvm/BinaryFormat/Dwarf.h
index 177c50be7b374..c3cff08eccb5c 100644
--- a/llvm/include/llvm/BinaryFormat/Dwarf.h
+++ b/llvm/include/llvm/BinaryFormat/Dwarf.h
@@ -1108,16 +1108,6 @@ inline bool isTlsAddressOp(uint8_t O) {
   return O == DW_OP_form_tls_address || O == DW_OP_GNU_push_tls_address;
 }
 
-/// Return true if Op is a symbolic branch to a label.
-inline bool isSymbolicBranchOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_bra || Op == DW_OP_LLVM_skip;
-}
-
-/// Return true if Op is a symbolic label or branch.
-inline bool isSymbolicControlFlowOp(uint64_t Op) {
-  return Op == DW_OP_LLVM_label || isSymbolicBranchOp(Op);
-}
-
 LLVM_ABI std::optional<unsigned> LanguageLowerBound(SourceLanguage L);
 
 /// The size of a reference determined by the DWARF 32/64-bit format.
diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 846342bee6071..2d0d6a1e228cb 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -3539,6 +3539,15 @@ class DIExpression : public MDNode {
     /// Get the operand code.
     uint64_t getOp() const { return *Op; }
 
+    /// Return true if this is \p Opcode.
+    bool is(uint64_t Opcode) const { return getOp() == Opcode; }
+
+    /// Return true if this is one of \p Opcodes.
+    template <typename... Ts> bool isOneOf(Ts... Opcodes) const {
+      static_assert(sizeof...(Ts) > 0, "requires at least one opcode");
+      return (is(Opcodes) || ...);
+    }
+
     /// Get an argument to the operand.
     ///
     /// Never returns the operand itself.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index f90edb1e68157..6bf604aae814c 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -546,7 +546,7 @@ bool DwarfExpression::addExpression(
   // Iterating over ExprCursor doesn't consume it.
   bool HasSymbolicBranches =
       llvm::any_of(ExprCursor, [](DIExpression::ExprOperand Op) {
-        return dwarf::isSymbolicBranchOp(Op.getOp());
+        return Op.isOneOf(dwarf::DW_OP_LLVM_bra, dwarf::DW_OP_LLVM_skip);
       });
 
   SmallVector<LabelOffset, 4> Labels;
@@ -577,12 +577,12 @@ bool DwarfExpression::addExpression(
         report_fatal_error(Twine("cannot lower DW_OP_LLVM_convert across ") +
                            dwarf::OperationEncodingString(OpNum) +
                            " without DW_OP_convert support");
-      if (OpNum == dwarf::DW_OP_LLVM_label) {
+      if (Op->is(dwarf::DW_OP_LLVM_label)) {
         Labels.push_back({Op->getArg(0), getTemporaryBufferSize()});
         break;
       }
-      emitOp(OpNum == dwarf::DW_OP_LLVM_bra ? dwarf::DW_OP_bra
-                                            : dwarf::DW_OP_skip);
+      emitOp(Op->is(dwarf::DW_OP_LLVM_bra) ? dwarf::DW_OP_bra
+                                           : dwarf::DW_OP_skip);
       Fixups.push_back({Op->getArg(0), getTemporaryBufferSize()});
       emitData2(0);
       break;
diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp
index 0dc59163fa864..48e7f88d9c07a 100644
--- a/llvm/lib/IR/Verifier.cpp
+++ b/llvm/lib/IR/Verifier.cpp
@@ -7188,7 +7188,8 @@ void Verifier::visit(DbgVariableRecord &DVR) {
   if (DVR.hasArgList() && Expr->isValid()) {
     bool HasSymbolicControlFlow =
         llvm::any_of(Expr->expr_ops(), [](DIExpression::ExprOperand Op) {
-          return dwarf::isSymbolicControlFlowOp(Op.getOp());
+          return Op.isOneOf(dwarf::DW_OP_LLVM_label, dwarf::DW_OP_LLVM_bra,
+                            dwarf::DW_OP_LLVM_skip);
         });
     CheckDI(!HasSymbolicControlFlow,
             "DIArgList doesn't support symbolic branches", &DVR, MD, Expr, BB,

>From cc43f3efed3c84cc291f8483c7b1b952e151bbe7 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Thu, 23 Jul 2026 18:33:53 -0700
Subject: [PATCH 6/8] [DebugInfo] Add an early return to
 DwarfExpression::addExpression

Add an early return to streamline DwarfExpression::addExpression's handling
of symbolic branches.
---
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    | 47 ++++++++++---------
 1 file changed, 24 insertions(+), 23 deletions(-)

diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 6bf604aae814c..56dd5ff03ebf4 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -806,31 +806,32 @@ bool DwarfExpression::addExpression(
     // Turn this into an implicit location description.
     addStackValue();
 
-  if (HasSymbolicBranches) {
-    for (const BranchFixup &Fixup : Fixups) {
-      auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
-        return Candidate.ID == Fixup.LabelID;
-      });
-      if (Label == Labels.end())
-        report_fatal_error(Twine("DWARF expression branch to label ") +
-                           Twine(Fixup.LabelID) + " has no matching label");
-
-      // DW_OP_bra and DW_OP_skip apply the displacement after reading their
-      // two-byte operand, so use the byte after the placeholder as the base.
-      int64_t Displacement =
-          static_cast<int64_t>(Label->Offset) -
-          static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
-      if (!isInt<16>(Displacement))
-        report_fatal_error(Twine("DWARF expression branch offset ") +
-                           Twine(Displacement) + " is outside [-32768, 32767]");
-
-      replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
-                                  static_cast<uint16_t>(Displacement));
-    }
+  if (!HasSymbolicBranches)
+    return true;
 
-    disableTemporaryBuffer();
-    commitTemporaryBuffer();
+  for (const BranchFixup &Fixup : Fixups) {
+    auto Label = llvm::find_if(Labels, [&](const LabelOffset &Candidate) {
+      return Candidate.ID == Fixup.LabelID;
+    });
+    if (Label == Labels.end())
+      report_fatal_error(Twine("DWARF expression branch to label ") +
+                         Twine(Fixup.LabelID) + " has no matching label");
+
+    // DW_OP_bra and DW_OP_skip apply the displacement after reading their
+    // two-byte operand, so use the byte after the placeholder as the base.
+    int64_t Displacement =
+        static_cast<int64_t>(Label->Offset) -
+        static_cast<int64_t>(Fixup.PlaceholderOffset + BranchOffsetByteSize);
+    if (!isInt<16>(Displacement))
+      report_fatal_error(Twine("DWARF expression branch offset ") +
+                         Twine(Displacement) + " is outside [-32768, 32767]");
+
+    replaceTemporaryBufferData2(Fixup.PlaceholderOffset,
+                                static_cast<uint16_t>(Displacement));
   }
+
+  disableTemporaryBuffer();
+  commitTemporaryBuffer();
   return true;
 }
 

>From 109bc0e91e481290ad541872ccced007aa155a91 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Thu, 23 Jul 2026 21:48:42 -0700
Subject: [PATCH 7/8] Update LangRef.md around DWARF branch ops

---
 llvm/docs/LangRef.md | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/llvm/docs/LangRef.md b/llvm/docs/LangRef.md
index a95644a1cc006..776e6267c7c24 100644
--- a/llvm/docs/LangRef.md
+++ b/llvm/docs/LangRef.md
@@ -7182,13 +7182,19 @@ DW_OP_LLVM_bra,   <label-id>
 DW_OP_LLVM_skip,  <label-id>
 ```
 
-`DW_OP_LLVM_label` declares a label ID and emits no bytes. `DW_OP_LLVM_bra`
-branches when the top of the stack is non-zero; `DW_OP_LLVM_skip` always
-branches. Label IDs are local to the expression. Every branch needs a matching
-label, labels without branches are valid, and each ID can only be declared
-once. Raw `DW_OP_bra` and `DW_OP_skip` aren't valid in LLVM IR.
-
-See {ref}`symbolic control flow <symbolic-control-flow>` for the full rules.
+`DW_OP_LLVM_bra` and `DW_OP_LLVM_skip` have the same semantics as the standard
+DWARF `DW_OP_bra` and `DW_OP_skip` operations. The LLVM forms use a label ID
+because the byte offset isn't known until emission. CodeGen resolves each
+branch's label ID and replaces the LLVM form with the corresponding standard
+operation and byte offset. `DW_OP_LLVM_label` declares the target for a label
+ID and emits no bytes.
+
+Label IDs are local to the expression. Every branch needs a matching label,
+labels without branches are valid, and each ID can only be declared once. The
+standard `DW_OP_bra` and `DW_OP_skip` operations aren't valid in LLVM IR. See
+[the DWARF standard](https://dwarfstd.org/) for the standard operations and
+{ref}`symbolic control flow <symbolic-control-flow>` for the LLVM-specific
+rules.
 
 ##### DIAssignID
 

>From fb8ccb37e6c8178cd8f08457c13be36bcc48f5c8 Mon Sep 17 00:00:00 2001
From: Eric Christopher <echristopher at nvidia.com>
Date: Thu, 23 Jul 2026 23:33:22 -0700
Subject: [PATCH 8/8] [DebugInfo] Skip non-emitting ops in register expressions

Labels and tag offsets don't add anything to the DWARF expression, so look
past them when deciding whether a register expression is complex.

This means:
- label-only expressions stay as register locations,
- labels before fragments stay on the simple register path, and
- tag-only expressions stay as register locations.
---
 llvm/include/llvm/IR/DebugInfoMetadata.h      |  4 ++
 .../CodeGen/AsmPrinter/DwarfExpression.cpp    | 24 ++++++----
 llvm/lib/IR/DebugInfoMetadata.cpp             |  4 ++
 .../di-expression-symbolic-branch-register.ll | 46 +++++++++++++++++++
 4 files changed, 69 insertions(+), 9 deletions(-)
 create mode 100644 llvm/test/DebugInfo/X86/di-expression-symbolic-branch-register.ll

diff --git a/llvm/include/llvm/IR/DebugInfoMetadata.h b/llvm/include/llvm/IR/DebugInfoMetadata.h
index 2d0d6a1e228cb..227c29ce3d6d9 100644
--- a/llvm/include/llvm/IR/DebugInfoMetadata.h
+++ b/llvm/include/llvm/IR/DebugInfoMetadata.h
@@ -3548,6 +3548,10 @@ class DIExpression : public MDNode {
       return (is(Opcodes) || ...);
     }
 
+    /// Return true if CodeGen handles this operand without adding bytes to the
+    /// DWARF expression.
+    LLVM_ABI bool isNonEmitting() const;
+
     /// Get an argument to the operand.
     ///
     /// Never returns the operand itself.
diff --git a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
index 56dd5ff03ebf4..a694e7ddc3255 100644
--- a/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
+++ b/llvm/lib/CodeGen/AsmPrinter/DwarfExpression.cpp
@@ -29,6 +29,18 @@ using namespace llvm;
 
 #define DEBUG_TYPE "dwarfdebug"
 
+/// Labels and tag offsets don't add operations to the DWARF expression. Look
+/// past them so an expression that otherwise contains only a fragment still
+/// uses the simple register path.
+static bool hasComplexExpression(const DIExpressionCursor &ExprCursor) {
+  for (DIExpression::ExprOperand Op : ExprCursor) {
+    if (Op.isNonEmitting())
+      continue;
+    return !Op.is(dwarf::DW_OP_LLVM_fragment);
+  }
+  return false;
+}
+
 void DwarfExpression::emitConstu(uint64_t Value) {
   if (Value < 32)
     emitOp(dwarf::DW_OP_lit0 + Value);
@@ -298,10 +310,8 @@ bool DwarfExpression::addMachineRegExpression(const TargetRegisterInfo &TRI,
     return false;
   }
 
-  bool HasComplexExpression = false;
   auto Op = ExprCursor.peek();
-  if (Op && Op->getOp() != dwarf::DW_OP_LLVM_fragment)
-    HasComplexExpression = true;
+  bool HasComplexExpression = hasComplexExpression(ExprCursor);
 
   // If the register can only be described by a complex expression (i.e.,
   // multiple subregisters) it doesn't safely compose with another complex
@@ -351,9 +361,7 @@ bool DwarfExpression::addMachineRegExpression(const TargetRegisterInfo &TRI,
     DwarfRegs.clear();
     // If we need to mask out a subregister, do it now, unless the next
     // operation would emit an OpPiece anyway.
-    auto NextOp = ExprCursor.peek();
-    if (SubRegisterSizeInBits && NextOp &&
-        (NextOp->getOp() != dwarf::DW_OP_LLVM_fragment))
+    if (SubRegisterSizeInBits && hasComplexExpression(ExprCursor))
       maskSubRegister();
     return true;
   }
@@ -418,9 +426,7 @@ bool DwarfExpression::addMachineRegExpression(const TargetRegisterInfo &TRI,
 
   // If we need to mask out a subregister, do it now, unless the next
   // operation would emit an OpPiece anyway.
-  auto NextOp = ExprCursor.peek();
-  if (SubRegisterSizeInBits && NextOp &&
-      (NextOp->getOp() != dwarf::DW_OP_LLVM_fragment))
+  if (SubRegisterSizeInBits && hasComplexExpression(ExprCursor))
     maskSubRegister();
 
   return true;
diff --git a/llvm/lib/IR/DebugInfoMetadata.cpp b/llvm/lib/IR/DebugInfoMetadata.cpp
index 518e4c130e34c..ce105147ad016 100644
--- a/llvm/lib/IR/DebugInfoMetadata.cpp
+++ b/llvm/lib/IR/DebugInfoMetadata.cpp
@@ -1761,6 +1761,10 @@ unsigned DIExpression::ExprOperand::getSize() const {
   }
 }
 
+bool DIExpression::ExprOperand::isNonEmitting() const {
+  return isOneOf(dwarf::DW_OP_LLVM_tag_offset, dwarf::DW_OP_LLVM_label);
+}
+
 bool DIExpression::isValid() const {
   auto IsEntryValueValid = [this](const ExprOperand &EntryValue) {
     auto FirstOp = expr_op_begin();
diff --git a/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-register.ll b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-register.ll
new file mode 100644
index 0000000000000..c347993922a66
--- /dev/null
+++ b/llvm/test/DebugInfo/X86/di-expression-symbolic-branch-register.ll
@@ -0,0 +1,46 @@
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -filetype=obj -o - %s | llvm-dwarfdump - | FileCheck %s
+
+; Labels emit no expression bytes, so they do not turn a simple register
+; location into a memory location or interfere with fragment lookahead.
+
+declare void @clobber()
+
+define void @registers(i64 %x, i32 %simple_subreg, i32 %complex_subreg) !dbg !5 {
+entry:
+  #dbg_value(i64 %x, !9, !DIExpression(DW_OP_LLVM_label, 1), !12)
+  #dbg_value(i32 %simple_subreg, !15,
+             !DIExpression(DW_OP_LLVM_label, 3,
+                           DW_OP_LLVM_fragment, 0, 32), !12)
+  #dbg_value(i32 %complex_subreg, !16,
+             !DIExpression(DW_OP_LLVM_label, 4,
+                           DW_OP_plus_uconst, 1), !12)
+  call void @clobber(), !dbg !12
+  ret void, !dbg !12
+}
+
+; CHECK: DW_OP_reg5 RDI)
+; CHECK: DW_AT_name {{.*}}"label_only"
+; CHECK: DW_OP_reg4 RSI, DW_OP_piece 0x4)
+; CHECK: DW_AT_name {{.*}}"simple_subreg"
+; CHECK: DW_OP_breg1 RDX+0, DW_OP_constu 0xffffffff, DW_OP_and, DW_OP_plus_uconst 0x1)
+; CHECK: DW_AT_name {{.*}}"complex_subreg"
+
+!llvm.dbg.cu = !{!0}
+!llvm.module.flags = !{!3}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1,
+                             emissionKind: FullDebug)
+!1 = !DIFile(filename: "test.c", directory: "/")
+!3 = !{i32 2, !"Debug Info Version", i32 3}
+!5 = distinct !DISubprogram(name: "registers", scope: !1, file: !1,
+                            type: !6, spFlags: DISPFlagDefinition, unit: !0,
+                            retainedNodes: !8)
+!6 = !DISubroutineType(types: !7)
+!7 = !{null}
+!8 = !{!9, !15, !16}
+!9 = !DILocalVariable(name: "label_only", scope: !5, type: !11)
+!11 = !DIBasicType(name: "long", size: 64, encoding: DW_ATE_signed)
+!12 = !DILocation(line: 1, scope: !5)
+!15 = !DILocalVariable(name: "simple_subreg", scope: !5, type: !11)
+!16 = !DILocalVariable(name: "complex_subreg", scope: !5, type: !17)
+!17 = !DIBasicType(name: "int", size: 32, encoding: DW_ATE_signed)


From llvmlistbot at llvm.org  Fri Jul 24 09:24:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 09:24:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Support IsIsolatedFromAbove
 trait for python-defined operations (PR #211849)
Message-ID: <llvm.org/llvm/llvm-project/pull/211849@github.com>


https://github.com/PragmaTwice created https://github.com/llvm/llvm-project/pull/211849

This PR adds dynamic `IsIsolatedFromAbove` trait support and exposes it to Python as `IsIsolatedFromAboveTrait`.

The test verifies that a region with the trait cannot capture an SSA value from above, while a region without the trait can.

Assisted by Codex/GPT 5.6 Sol (for writing test cases).

>From 454f6a61f5032715da011cd59a469146ff18a02f Mon Sep 17 00:00:00 2001
From: PragmaTwice <twice at apache.org>
Date: Sat, 25 Jul 2026 00:15:16 +0800
Subject: [PATCH] [MLIR][Python] Support IsIsolatedFromAbove trait for
 python-defined operations

---
 mlir/include/mlir-c/ExtensibleDialect.h    |  9 ++++
 mlir/include/mlir/Bindings/Python/IRCore.h |  6 +++
 mlir/include/mlir/IR/ExtensibleDialect.h   |  8 +++
 mlir/lib/Bindings/Python/IRCore.cpp        | 23 ++++++++
 mlir/lib/CAPI/IR/ExtensibleDialect.cpp     |  8 +++
 mlir/test/python/dialects/ext.py           | 63 ++++++++++++++++++++++
 6 files changed, 117 insertions(+)

diff --git a/mlir/include/mlir-c/ExtensibleDialect.h b/mlir/include/mlir-c/ExtensibleDialect.h
index c849658c710bb..f8dcb3f6800d3 100644
--- a/mlir/include/mlir-c/ExtensibleDialect.h
+++ b/mlir/include/mlir-c/ExtensibleDialect.h
@@ -54,6 +54,15 @@ mlirDynamicOpTraitIsTerminatorCreate(void);
 /// terminator.
 MLIR_CAPI_EXPORTED MlirTypeID mlirDynamicOpTraitIsTerminatorGetTypeID(void);
 
+/// Get the dynamic op trait that indicates regions are isolated from above.
+MLIR_CAPI_EXPORTED MlirDynamicOpTrait
+mlirDynamicOpTraitIsIsolatedFromAboveCreate(void);
+
+/// Get the type ID of the dynamic op trait that indicates regions are isolated
+/// from above.
+MLIR_CAPI_EXPORTED MlirTypeID
+mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID(void);
+
 /// Get the dynamic op trait that indicates regions have no terminator.
 MLIR_CAPI_EXPORTED MlirDynamicOpTrait
 mlirDynamicOpTraitNoTerminatorCreate(void);
diff --git a/mlir/include/mlir/Bindings/Python/IRCore.h b/mlir/include/mlir/Bindings/Python/IRCore.h
index 43a751bc9cd9f..3314e0b2a8fcf 100644
--- a/mlir/include/mlir/Bindings/Python/IRCore.h
+++ b/mlir/include/mlir/Bindings/Python/IRCore.h
@@ -1991,6 +1991,12 @@ class MLIR_PYTHON_API_EXPORTED NoTerminator : public PyDynamicOpTrait {
   static void bind(nanobind::module_ &m);
 };
 
+class MLIR_PYTHON_API_EXPORTED IsIsolatedFromAbove : public PyDynamicOpTrait {
+public:
+  static bool attach(const nanobind::object &opName, PyMlirContext &context);
+  static void bind(nanobind::module_ &m);
+};
+
 } // namespace PyDynamicOpTraits
 
 MLIR_PYTHON_API_EXPORTED MlirValue getUniqueResult(MlirOperation operation);
diff --git a/mlir/include/mlir/IR/ExtensibleDialect.h b/mlir/include/mlir/IR/ExtensibleDialect.h
index 15904358def11..2219d6fd57977 100644
--- a/mlir/include/mlir/IR/ExtensibleDialect.h
+++ b/mlir/include/mlir/IR/ExtensibleDialect.h
@@ -416,6 +416,14 @@ class IsTerminator : public DynamicOpTraitImpl<OpTrait::IsTerminator> {
 
 class NoTerminator : public DynamicOpTraitImpl<OpTrait::NoTerminator> {};
 
+class IsIsolatedFromAbove
+    : public DynamicOpTraitImpl<OpTrait::IsIsolatedFromAbove> {
+public:
+  LogicalResult verifyRegionTrait(Operation *op) const override {
+    return OpTrait::impl::verifyIsIsolatedFromAbove(op);
+  }
+};
+
 } // namespace DynamicOpTraits
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index cceef55f9b15d..75cfd2a0a1c0b 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -2663,6 +2663,28 @@ void PyDynamicOpTraits::NoTerminator::bind(nb::module_ &m) {
       nb::arg("op_name"), nb::arg("context").none() = nb::none());
 }
 
+bool PyDynamicOpTraits::IsIsolatedFromAbove::attach(const nb::object &opName,
+                                                    PyMlirContext &context) {
+  MlirDynamicOpTrait trait = mlirDynamicOpTraitIsIsolatedFromAboveCreate();
+  return attachOpTrait(opName, trait, context);
+}
+
+void PyDynamicOpTraits::IsIsolatedFromAbove::bind(nb::module_ &m) {
+  nb::class_<PyDynamicOpTraits::IsIsolatedFromAbove, PyDynamicOpTrait> cls(
+      m, "IsIsolatedFromAboveTrait");
+  cls.attr(typeIDAttr) =
+      PyTypeID(mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID());
+  cls.attr("attach") = classmethod(
+      [](const nb::object &cls, const nb::object &opName,
+         DefaultingPyMlirContext context) {
+        return PyDynamicOpTraits::IsIsolatedFromAbove::attach(opName,
+                                                              *context.get());
+      },
+      "Attach IsIsolatedFromAbove trait to the given operation name.",
+      nb::arg("cls"), nb::arg("op_name"),
+      nb::arg("context").none() = nb::none());
+}
+
 } // namespace MLIR_BINDINGS_PYTHON_DOMAIN
 } // namespace python
 } // namespace mlir
@@ -5297,6 +5319,7 @@ void populateIRCore(nb::module_ &m) {
   PyDynamicOpTrait::bind(m);
   PyDynamicOpTraits::IsTerminator::bind(m);
   PyDynamicOpTraits::NoTerminator::bind(m);
+  PyDynamicOpTraits::IsIsolatedFromAbove::bind(m);
 
   // MLIRError exception.
   MLIRError::bind(m);
diff --git a/mlir/lib/CAPI/IR/ExtensibleDialect.cpp b/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
index 5c1f74884f2c5..2b4d5af0c9f13 100644
--- a/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
+++ b/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
@@ -54,6 +54,14 @@ MlirTypeID mlirDynamicOpTraitNoTerminatorGetTypeID() {
   return wrap(DynamicOpTraits::NoTerminator::getStaticTypeID());
 }
 
+MlirDynamicOpTrait mlirDynamicOpTraitIsIsolatedFromAboveCreate() {
+  return wrap(new DynamicOpTraits::IsIsolatedFromAbove());
+}
+
+MlirTypeID mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID() {
+  return wrap(DynamicOpTraits::IsIsolatedFromAbove::getStaticTypeID());
+}
+
 void mlirDynamicOpTraitDestroy(MlirDynamicOpTrait dynamicOpTrait) {
   delete unwrap(dynamicOpTrait);
 }
diff --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index dde980df53dba..6e83da4a4a78a 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -637,6 +637,69 @@ class NoTermOp(TestRegion.Operation, name="no_term", traits=[NoTerminatorTrait])
             print(e)
 
 
+# CHECK: TEST: testIsIsolatedFromAboveTrait
+ at run
+def testIsIsolatedFromAboveTrait():
+    class TestIsolated(Dialect, name="ext_isolated"):
+        pass
+
+    class UseOp(TestIsolated.Operation, name="use"):
+        value: Operand[Any]
+
+    class NotIsolatedOp(
+        TestIsolated.Operation, name="not_isolated", traits=[NoTerminatorTrait]
+    ):
+        body: Region
+
+    class IsolatedOp(
+        TestIsolated.Operation,
+        name="isolated",
+        traits=[NoTerminatorTrait, IsIsolatedFromAboveTrait],
+    ):
+        body: Region
+
+    with Context(), Location.unknown():
+        TestIsolated.load()
+
+        # CHECK: not isolated has trait: False
+        print(
+            "not isolated has trait:",
+            NotIsolatedOp.has_trait(IsIsolatedFromAboveTrait),
+        )
+        # CHECK: isolated has trait: True
+        print("isolated has trait:", IsolatedOp.has_trait(IsIsolatedFromAboveTrait))
+
+        not_isolated_module = Module.create()
+        with InsertionPoint(not_isolated_module.body):
+            i32 = IntegerType.get_signless(32)
+            value = arith.constant(i32, 0)
+            not_isolated = NotIsolatedOp()
+            not_isolated.body.blocks.append()
+            with InsertionPoint(not_isolated.body.blocks[0]):
+                UseOp(value)
+
+        assert not_isolated_module.operation.verify()
+        # CHECK: not isolated: verification succeeded
+        print("not isolated: verification succeeded")
+
+        isolated_module = Module.create()
+        with InsertionPoint(isolated_module.body):
+            value = arith.constant(i32, 0)
+            isolated = IsolatedOp()
+            isolated.body.blocks.append()
+            with InsertionPoint(isolated.body.blocks[0]):
+                UseOp(value)
+
+        try:
+            isolated_module.operation.verify()
+        except MLIRError as e:
+            # CHECK: isolated: Verification failed:
+            # CHECK: using value defined outside the region
+            print("isolated:", e)
+        else:
+            raise AssertionError("expected IsIsolatedFromAbove verification to fail")
+
+
 # CHECK: TEST: testExtDialectWithType
 @run
 def testExtDialectWithType():


From llvmlistbot at llvm.org  Fri Jul 24 10:49:01 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Fri, 24 Jul 2026 10:49:01 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add mlirOperationGetParentWithName
	(PR #206533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206533@github.com>
Message-ID: <6a63a58d.1e1ac41d.8ad93.d38c@mx.google.com>


https://github.com/makslevental closed https://github.com/llvm/llvm-project/pull/206533

From llvmlistbot at llvm.org  Fri Jul 24 10:49:02 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Fri, 24 Jul 2026 10:49:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir-c] Add mlirOperationGetParentWithName
	(PR #206533)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206533@github.com>
Message-ID: <6a63a58e.92d2873c.287016.2dcc@mx.google.com>


makslevental wrote:

Closing because I decided it's better to leave this kind of utility code out of the C bindings themselves.

https://github.com/llvm/llvm-project/pull/206533

From llvmlistbot at llvm.org  Fri Jul 24 10:53:04 2026
From: llvmlistbot at llvm.org (Maksim Levental)
Date: Fri, 24 Jul 2026 10:53:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Support IsIsolatedFromAbove
 trait for python-defined operations (PR #211849)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211849@github.com>
Message-ID: <6a63a680.510f576d.23298e.9fc0@mx.google.com>


https://github.com/makslevental approved this pull request.

LGTM. one question: are these traits bound such that I can query traits on non-dynamic dialects and get these to pop out too? Or does all this work **only** on irdl/dynamic dialects? If the latter, do you think it would be hard to extend to all dialects?

https://github.com/llvm/llvm-project/pull/211849

From llvmlistbot at llvm.org  Fri Jul 24 11:49:42 2026
From: llvmlistbot at llvm.org (Charitha Saumya)
Date: Fri, 24 Jul 2026 11:49:42 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Allow create_mem_desc from ND
	memref (PR #211836)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211836@github.com>
Message-ID: <6a63b3c6.7270d2fd.2df01a.1f7f@mx.google.com>


https://github.com/charithaintc approved this pull request.


https://github.com/llvm/llvm-project/pull/211836

From llvmlistbot at llvm.org  Fri Jul 24 11:52:47 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 11:52:47 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][mem2reg] Promote whole-buffer memref
 to a vector SSA value (PR #211880)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211880@github.com>
Message-ID: <6a63b47f.f06f4994.3876ce.c469@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Jianhui Li (Jianhui-Li)

<details>
<summary>Changes</summary>

  When a vectorized scf.for carries a reduction or accumulation tensor — the C accumulator of a contraction, or a reduction buffer — bufferization materializes that loop-carried tensor into a scratch memref, and promote-buffers-to-stack turns it into a small memref.alloca. The buffer's only accesses are a whole-buffer vector.transfer_write that stores the accumulator and a whole-buffer vector.transfer_read that reads it back, both at index 0, in-bounds, with an identity permutation map and a vector type that exactly matches the memref shape. Such a buffer is effectively an SSA vector spilled to the stack, and it blocks downstream vectorization and register allocation. MLIR's Mem2Reg can already promote single-use memory into SSA values, but today only for scalar slots — memref::AllocaOp::getPromotableSlots offers a slot only for a single-element memref, and memref.load/memref.store are the only load/store ops wired into the pass — so these whole-buffer allocas are left untouched.

  This PR teaches Mem2Reg to promote such a buffer into a single vector SSA value. The machinery is already type-agnostic — MemorySlot carries an arbitrary elemType, and scf.for already threads a slot of any type as an iter_arg/result — so no changes to the pass or to SCF are needed. memref::AllocaOp::getPromotableSlots now also offers a vector-typed slot for a static, multi-element memref, and vector.transfer_read/transfer_write implement PromotableMemOpInterface as external models, gated by an isWholeBufferTransfer predicate (single blocking use on the slot pointer, exact type match, all-zero indices, identity map, in-bounds, no mask). Any access that is not a whole-buffer transfer fails its own canUsesBeRemoved check and aborts promotion of that slot with zero IR mutation. A new lit test covers straight-line and scf.for-threaded promotion, the 2-D case, and six negative cases.
  
  assisted-by-claude


---
Full diff: https://github.com/llvm/llvm-project/pull/211880.diff


6 Files Affected:

- (added) mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h (+21) 
- (modified) mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp (+12-4) 
- (modified) mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt (+2) 
- (added) mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp (+159) 
- (modified) mlir/lib/RegisterAllDialects.cpp (+2) 
- (added) mlir/test/Dialect/Vector/mem2reg.mlir (+172) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h b/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h
new file mode 100644
index 0000000000000..4c2e386728419
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h
@@ -0,0 +1,21 @@
+//===- MemorySlotOpInterfaceImpl.h - Mem2Reg for vector ops -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
+#define MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
+
+namespace mlir {
+
+class DialectRegistry;
+
+namespace vector {
+void registerMemorySlotOpInterfaceExternalModels(DialectRegistry &registry);
+} // namespace vector
+} // namespace mlir
+
+#endif // MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..17b1417550524 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -66,11 +66,19 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   MemRefType type = getType();
   if (!type.hasStaticShape())
     return {};
-  // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
-    return {};
 
-  return {MemorySlot{getResult(), type.getElementType()}};
+  // A single-element memref is promoted to a scalar SSA value.
+  if (type.getNumElements() == 1)
+    return {MemorySlot{getResult(), type.getElementType()}};
+
+  // A multi-element memref can be promoted to a single vector SSA value when it
+  // is only ever accessed as a whole buffer (e.g. through whole-buffer
+  // `vector.transfer_read`/`vector.transfer_write`).
+  if (VectorType::isValidElementType(type.getElementType()))
+    return {MemorySlot{
+        getResult(), VectorType::get(type.getShape(), type.getElementType())}};
+
+  return {};
 }
 
 Value memref::AllocaOp::getDefaultValue(const MemorySlot &slot,
diff --git a/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
index 112a1db6fe93b..dfe873f1a1b8d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRVectorTransforms
   LowerVectorToFromElementsToShuffleTree.cpp
   LowerVectorTransfer.cpp
   LowerVectorTranspose.cpp
+  MemorySlotOpInterfaceImpl.cpp
   SubsetOpInterfaceImpl.cpp
   VectorDistribute.cpp
   VectorDropLeadUnitDim.cpp
@@ -44,6 +45,7 @@ add_mlir_dialect_library(MLIRVectorTransforms
   MLIRGPUUtils
   MLIRIR
   MLIRLinalgDialect
+  MLIRMemorySlotInterfaces
   MLIRMemRefDialect
   MLIRMemRefTransforms
   MLIRMemRefUtils
diff --git a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
new file mode 100644
index 0000000000000..352983ba153fa
--- /dev/null
+++ b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -0,0 +1,159 @@
+//===- MemorySlotOpInterfaceImpl.cpp - Mem2Reg for vector ops -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements Mem2Reg-related interfaces for Vector dialect
+// operations. It allows a memref that is only ever accessed as a whole buffer
+// through `vector.transfer_read`/`vector.transfer_write` to be promoted into a
+// single vector SSA value.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h"
+
+#include "mlir/Dialect/Utils/StaticValueUtils.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Interfaces/MemorySlotInterfaces.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+
+//===----------------------------------------------------------------------===//
+//  Utilities
+//===----------------------------------------------------------------------===//
+
+/// Returns whether `xferOp` accesses exactly the whole contents of `slot`, so
+/// it can act as a plain whole-buffer load/store during Mem2Reg.
+template <typename TransferOpTy>
+static bool
+isWholeBufferTransfer(TransferOpTy xferOp, const MemorySlot &slot,
+                      const SmallPtrSetImpl<OpOperand *> &blockingUses) {
+  // The sole blocking use must be the slot pointer as the transfer's base.
+  if (blockingUses.size() != 1)
+    return false;
+  Value blockingUse = (*blockingUses.begin())->get();
+  if (blockingUse != slot.ptr || xferOp.getBase() != slot.ptr)
+    return false;
+
+  // Reject the tensor form (already implied, since slot pointers are memrefs).
+  if (!isa<MemRefType>(xferOp.getBase().getType()))
+    return false;
+
+  // Exact type match pins rank/extents/element type and rejects scalable
+  // vectors.
+  if (xferOp.getVectorType() != slot.elemType)
+    return false;
+
+  // Access must start at the buffer origin in every dimension.
+  for (Value index : xferOp.getIndices()) {
+    std::optional<int64_t> constIndex = getConstantIntValue(index);
+    if (!constIndex || *constIndex != 0)
+      return false;
+  }
+
+  // Identity map: no broadcast or transpose.
+  if (!xferOp.getPermutationMap().isIdentity())
+    return false;
+
+  // All dimensions in bounds: no out-of-buffer element, no padding.
+  if (xferOp.hasOutOfBoundsDim())
+    return false;
+
+  // A mask would make the access partial.
+  if (xferOp.getMask())
+    return false;
+
+  return true;
+}
+
+//===----------------------------------------------------------------------===//
+//  Interface models
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+struct TransferReadOpMemOpModel
+    : public PromotableMemOpInterface::ExternalModel<TransferReadOpMemOpModel,
+                                                     vector::TransferReadOp> {
+  bool loadsFrom(Operation *op, const MemorySlot &slot) const {
+    return cast<vector::TransferReadOp>(op).getBase() == slot.ptr;
+  }
+
+  bool storesTo(Operation *op, const MemorySlot &slot) const { return false; }
+
+  Value getStored(Operation *op, const MemorySlot &slot, OpBuilder &builder,
+                  Value reachingDef, const DataLayout &dataLayout) const {
+    llvm_unreachable("getStored should not be called on TransferReadOp");
+  }
+
+  bool canUsesBeRemoved(Operation *op, const MemorySlot &slot,
+                        const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                        SmallVectorImpl<OpOperand *> &newBlockingUses,
+                        const DataLayout &dataLayout) const {
+    return isWholeBufferTransfer(cast<vector::TransferReadOp>(op), slot,
+                                 blockingUses);
+  }
+
+  DeletionKind
+  removeBlockingUses(Operation *op, const MemorySlot &slot,
+                     const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                     OpBuilder &builder, Value reachingDefinition,
+                     const DataLayout &dataLayout) const {
+    // Whole-buffer read: replace the loaded vector with the reaching
+    // definition.
+    cast<vector::TransferReadOp>(op).getVector().replaceAllUsesWith(
+        reachingDefinition);
+    return DeletionKind::Delete;
+  }
+};
+
+struct TransferWriteOpMemOpModel
+    : public PromotableMemOpInterface::ExternalModel<TransferWriteOpMemOpModel,
+                                                     vector::TransferWriteOp> {
+  bool loadsFrom(Operation *op, const MemorySlot &slot) const { return false; }
+
+  bool storesTo(Operation *op, const MemorySlot &slot) const {
+    return cast<vector::TransferWriteOp>(op).getBase() == slot.ptr;
+  }
+
+  Value getStored(Operation *op, const MemorySlot &slot, OpBuilder &builder,
+                  Value reachingDef, const DataLayout &dataLayout) const {
+    return cast<vector::TransferWriteOp>(op).getValueToStore();
+  }
+
+  bool canUsesBeRemoved(Operation *op, const MemorySlot &slot,
+                        const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                        SmallVectorImpl<OpOperand *> &newBlockingUses,
+                        const DataLayout &dataLayout) const {
+    // No self-store guard needed: a vector value can never equal a memref slot.
+    return isWholeBufferTransfer(cast<vector::TransferWriteOp>(op), slot,
+                                 blockingUses);
+  }
+
+  DeletionKind
+  removeBlockingUses(Operation *op, const MemorySlot &slot,
+                     const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                     OpBuilder &builder, Value reachingDefinition,
+                     const DataLayout &dataLayout) const {
+    return DeletionKind::Delete;
+  }
+};
+
+} // namespace
+
+//===----------------------------------------------------------------------===//
+//  Register external models
+//===----------------------------------------------------------------------===//
+
+void mlir::vector::registerMemorySlotOpInterfaceExternalModels(
+    DialectRegistry &registry) {
+  registry.addExtension(+[](MLIRContext *ctx, vector::VectorDialect *dialect) {
+    TransferReadOp::attachInterface<TransferReadOpMemOpModel>(*ctx);
+    TransferWriteOp::attachInterface<TransferWriteOpMemOpModel>(*ctx);
+  });
+}
diff --git a/mlir/lib/RegisterAllDialects.cpp b/mlir/lib/RegisterAllDialects.cpp
index 974b5f533860a..62304ae88b43c 100644
--- a/mlir/lib/RegisterAllDialects.cpp
+++ b/mlir/lib/RegisterAllDialects.cpp
@@ -97,6 +97,7 @@
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/BufferizableOpInterfaceImpl.h"
 #include "mlir/Dialect/Vector/Transforms/IndexedAccessOpInterfaceImpl.h"
+#include "mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h"
 #include "mlir/Dialect/Vector/Transforms/SubsetOpInterfaceImpl.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
 #include "mlir/Dialect/X86/X86Dialect.h"
@@ -200,6 +201,7 @@ void mlir::registerAllDialects(DialectRegistry &registry) {
   tosa::registerShardingInterfaceExternalModels(registry);
   vector::registerBufferizableOpInterfaceExternalModels(registry);
   vector::registerIndexedAccessOpInterfaceExternalModels(registry);
+  vector::registerMemorySlotOpInterfaceExternalModels(registry);
   vector::registerSubsetOpInterfaceExternalModels(registry);
   vector::registerValueBoundsOpInterfaceExternalModels(registry);
   NVVM::registerNVVMTargetInterfaceExternalModels(registry);
diff --git a/mlir/test/Dialect/Vector/mem2reg.mlir b/mlir/test/Dialect/Vector/mem2reg.mlir
new file mode 100644
index 0000000000000..ec6e19d0a5c19
--- /dev/null
+++ b/mlir/test/Dialect/Vector/mem2reg.mlir
@@ -0,0 +1,172 @@
+// RUN: mlir-opt %s --pass-pipeline='builtin.module(func.func(mem2reg))' --split-input-file | FileCheck %s
+
+// A memref that is only ever accessed as a whole buffer through
+// vector.transfer_read / vector.transfer_write is promoted to a single vector
+// SSA value.
+
+// CHECK-LABEL: func.func @whole_buffer_write_read
+//   CHECK-SAME:   (%[[PAD:.*]]: f32)
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer_write
+//    CHECK-NOT:   vector.transfer_read
+//        CHECK:   %[[CST:.*]] = arith.constant dense<1.000000e+00> : vector<4xf32>
+//        CHECK:   return %[[CST]] : vector<4xf32>
+func.func @whole_buffer_write_read(%pad: f32) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A whole-buffer slot carried across scf.for is threaded as an iter_arg/result.
+
+// CHECK-LABEL: func.func @whole_buffer_in_loop
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer_write
+//    CHECK-NOT:   vector.transfer_read
+//        CHECK:   %[[RES:.*]] = scf.for {{.*}} iter_args(%[[IT:.*]] = %{{.*}}) -> (vector<4xf32>)
+//        CHECK:     %[[NEXT:.*]] = arith.addf %[[IT]], %[[IT]] : vector<4xf32>
+//        CHECK:     scf.yield %[[NEXT]] : vector<4xf32>
+//        CHECK:   return %[[RES]] : vector<4xf32>
+func.func @whole_buffer_in_loop(%pad: f32, %lb: index, %ub: index, %step: index) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  scf.for %i = %lb to %ub step %step {
+    %v = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+    %n = arith.addf %v, %v : vector<4xf32>
+    vector.transfer_write %n, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  }
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A multi-dimensional whole-buffer memref is promoted to a matching vector.
+
+// CHECK-LABEL: func.func @whole_buffer_2d
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer
+//        CHECK:   return %{{.*}} : vector<2x4xf32>
+func.func @whole_buffer_2d(%pad: f32) -> vector<2x4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<2x4xf32>
+  %a = memref.alloca() : memref<2x4xf32>
+  vector.transfer_write %cst, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
+  %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<2x4xf32>, vector<2x4xf32>
+  return %r : vector<2x4xf32>
+}
+
+// -----
+
+// Non-zero access offset: the transfer does not cover the whole buffer, so the
+// slot must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_nonzero_index
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_nonzero_index(%pad: f32) -> vector<4xf32> {
+  %c1 = arith.constant 1 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<8xf32>
+  vector.transfer_write %cst, %a[%c1] {in_bounds = [true]} : vector<4xf32>, memref<8xf32>
+  %r = vector.transfer_read %a[%c1], %pad {in_bounds = [true]} : memref<8xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A masked transfer only touches part of the buffer: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_masked
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_masked(%pad: f32, %m: vector<4xi1>) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0], %m {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A partial (out-of-bounds) transfer must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_out_of_bounds
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_out_of_bounds(%pad: f32) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<8xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] : vector<8xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad : memref<4xf32>, vector<8xf32>
+  return %r : vector<8xf32>
+}
+
+// -----
+
+// A non-identity (transposing) permutation map is not a whole-buffer identity
+// access: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_transpose_map
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_transpose_map(%pad: f32) -> vector<4x2xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<2x4xf32>
+  %a = memref.alloca() : memref<2x4xf32>
+  vector.transfer_write %cst, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
+  %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true], permutation_map = affine_map<(d0, d1) -> (d1, d0)>} : memref<2x4xf32>, vector<4x2xf32>
+  return %r : vector<4x2xf32>
+}
+
+// -----
+
+// An alloca also accessed through a scalar memref.load cannot be promoted to a
+// vector: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_mixed_scalar_access
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+//        CHECK:   memref.load
+func.func @negative_mixed_scalar_access(%pad: f32) -> (vector<4xf32>, f32) {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  %s = memref.load %a[%c0] : memref<4xf32>
+  return %r, %s : vector<4xf32>, f32
+}
+
+// -----
+
+// A scalable vector never equals the fixed-size slot type: must NOT be
+// promoted.
+
+// CHECK-LABEL: func.func @negative_scalable
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_scalable(%pad: f32) -> vector<[4]xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<[4]xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<[4]xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<[4]xf32>
+  return %r : vector<[4]xf32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/211880

From llvmlistbot at llvm.org  Fri Jul 24 11:58:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 11:58:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Gate min/max expansion in
 arith-expand behind include-min-max (PR #211884)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211884@github.com>
Message-ID: <6a63b5e1.7270d2fd.2df01a.20d8@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-gpu

Author: Md Abdullah Shahneous Bari (mshahneo)

<details>
<summary>Changes</summary>

`arith-expand` expanded `arith.maximumf`/`minimumf`/`maxnumf`/`minnumf` and the signed/unsigned integer max/min ops into `cmpf`/`cmpi` + `select` sequences. These ops also have a direct arith-to-llvm lowering to the `llvm.intr.maximum`/`minimum`/... intrinsics, which are a single hardware instruction on many targets. Pipelines that run arith-to-llvm after arith-expand (e.g. the GPU-to-XeVM pipeline) therefore paid a large, avoidable overhead.

Add an `include-min-max` option (default `true`, preserving existing behavior) that controls whether these min/max ops are expanded. The min/max converters are factored into a new `populateExpandMinMaxPatterns`; the ceil/floor-div and scaling ext/trunc expansions (which have no LLVM lowering) always run. Set `include-min-max=false` in the GPU-to-XeVM pipeline so the intrinsic lowering is used.

---
Full diff: https://github.com/llvm/llvm-project/pull/211884.diff


5 Files Affected:

- (modified) mlir/include/mlir/Dialect/Arith/Transforms/Passes.h (+7) 
- (modified) mlir/include/mlir/Dialect/Arith/Transforms/Passes.td (+8) 
- (modified) mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp (+31-12) 
- (modified) mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp (+7) 
- (added) mlir/test/Dialect/Arith/expand-ops-min-max.mlir (+64) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
index 5a07a01d0928a..4895443a7ba01 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.h
@@ -68,6 +68,13 @@ void populateExpandF8E8M0Patterns(RewritePatternSet &patterns);
 /// Add patterns to expand scaling ExtF/TruncF ops to equivalent arith ops
 void populateExpandScalingExtTruncPatterns(RewritePatternSet &patterns);
 
+/// Add patterns to expand the min/max ops (`arith.maximumf`/`minimumf`/
+/// `maxnumf`/`minnumf` and the signed/unsigned integer `max`/`min`) into
+/// `cmpf`/`cmpi` + `select` sequences. These ops also have a direct
+/// arith-to-llvm lowering, so pipelines that run arith-to-llvm may prefer to
+/// skip this expansion.
+void populateExpandMinMaxPatterns(RewritePatternSet &patterns);
+
 /// Add patterns to expand `arith.flush_denormals` into integer arithmetic
 /// (bitcast + bit masks + compare + select). Only matches IEEE-like
 /// floating-point types.
diff --git a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
index 27e9146ec3606..8fb7439854406 100644
--- a/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Arith/Transforms/Passes.td
@@ -25,6 +25,14 @@ def ArithExpandOpsPass : Pass<"arith-expand"> {
               /*default=*/"false",
               "Enable expansion of `arith.flush_denormals` on IEEE-like "
               "floating-point types">,
+       Option<"includeMinMax", "include-min-max", "bool", /*default=*/"true",
+              "Enable expansion of the min/max ops (maximumf/minimumf/maxnumf/"
+              "minnumf and the signed/unsigned integer max/min). These ops "
+              "also have a direct arith-to-llvm lowering to the "
+              "`llvm.intr.maximum/minimum/maxnum/minnum` (and smax/umax/...) "
+              "intrinsics; disable this so pipelines that run arith-to-llvm "
+              "can use those single-instruction lowerings instead of the "
+              "cmpf/select expansion">,
   ];
 }
 
diff --git a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
index c9217c57a5f25..6e607311300a5 100644
--- a/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
+++ b/mlir/lib/Dialect/Arith/Transforms/ExpandOps.cpp
@@ -823,7 +823,8 @@ struct ArithExpandOpsPass
     RewritePatternSet patterns(&getContext());
     ConversionTarget target(getContext());
 
-    arith::populateArithExpandOpsPatterns(patterns);
+    arith::populateCeilFloorDivExpandOpsPatterns(patterns);
+    arith::populateExpandScalingExtTruncPatterns(patterns);
 
     target.addLegalDialect<arith::ArithDialect>();
     target.addLegalDialect<vector::VectorDialect>();
@@ -833,17 +834,31 @@ struct ArithExpandOpsPass
       arith::CeilDivSIOp,
       arith::CeilDivUIOp,
       arith::FloorDivSIOp,
-      arith::MaxSIOp,
-      arith::MaxUIOp,
-      arith::MinSIOp,
-      arith::MinUIOp,
-      arith::MaximumFOp,
-      arith::MinimumFOp,
-      arith::MaxNumFOp,
-      arith::MinNumFOp,
       arith::ScalingExtFOp,
       arith::ScalingTruncFOp
     >();
+    // clang-format on
+
+    // The min/max ops also have a direct arith-to-llvm lowering to the
+    // `llvm.intr.maximum`/`minimum`/... intrinsics, which are a single hardware
+    // instruction on many targets. Only expand them into cmpf/cmpi + select
+    // when requested, so pipelines that run arith-to-llvm can keep the
+    // intrinsic lowering.
+    if (includeMinMax) {
+      arith::populateExpandMinMaxPatterns(patterns);
+      // clang-format off
+      target.addIllegalOp<
+        arith::MaxSIOp,
+        arith::MaxUIOp,
+        arith::MinSIOp,
+        arith::MinUIOp,
+        arith::MaximumFOp,
+        arith::MinimumFOp,
+        arith::MaxNumFOp,
+        arith::MinNumFOp
+      >();
+      // clang-format on
+    }
 
     if (includeBf16)
       arith::populateExpandBFloat16Patterns(patterns);
@@ -936,9 +951,7 @@ void mlir::arith::populateExpandFlushDenormalsPatterns(
   patterns.add<FlushDenormalsOpConverter>(patterns.getContext());
 }
 
-void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
-  populateCeilFloorDivExpandOpsPatterns(patterns);
-  populateExpandScalingExtTruncPatterns(patterns);
+void mlir::arith::populateExpandMinMaxPatterns(RewritePatternSet &patterns) {
   // clang-format off
   patterns.add<
     MaxMinIOpConverter<MaxSIOp, arith::CmpIPredicate::sgt>,
@@ -952,3 +965,9 @@ void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
    >(patterns.getContext());
   // clang-format on
 }
+
+void mlir::arith::populateArithExpandOpsPatterns(RewritePatternSet &patterns) {
+  populateCeilFloorDivExpandOpsPatterns(patterns);
+  populateExpandScalingExtTruncPatterns(patterns);
+  populateExpandMinMaxPatterns(patterns);
+}
diff --git a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
index 4dc9e2acfe235..7841e34793bc5 100644
--- a/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
+++ b/mlir/lib/Dialect/GPU/Pipelines/GPUToXeVMPipeline.cpp
@@ -111,6 +111,13 @@ void buildGPUPassPipeline(OpPassManager &pm,
   {
     arith::ArithExpandOpsPassOptions arithExpandOptions;
     arithExpandOptions.includeF8E8M0 = true;
+    // Do not expand the min/max ops (arith.maximumf/minimumf/...) into
+    // cmpf + select here: they are lowered directly to the
+    // `llvm.intr.maximum`/`minimum`/... intrinsics (single hardware
+    // instructions) by the later arith-to-llvm conversion. Expanding them
+    // regresses e.g. flash-attention softmax, whose running-max is
+    // arith.maximumf-heavy.
+    arithExpandOptions.includeMinMax = false;
     pm.addNestedPass<gpu::GPUModuleOp>(
         arith::createArithExpandOpsPass(arithExpandOptions));
   }
diff --git a/mlir/test/Dialect/Arith/expand-ops-min-max.mlir b/mlir/test/Dialect/Arith/expand-ops-min-max.mlir
new file mode 100644
index 0000000000000..92d6c8ad773d5
--- /dev/null
+++ b/mlir/test/Dialect/Arith/expand-ops-min-max.mlir
@@ -0,0 +1,64 @@
+// Default (include-min-max=true): the min/max ops are expanded into
+// cmpf/cmpi + select sequences.
+// RUN: mlir-opt %s -arith-expand -split-input-file | FileCheck %s --check-prefix=EXPAND
+
+// include-min-max=false: the min/max ops are left untouched so that a later
+// arith-to-llvm conversion can lower them to the single-instruction
+// llvm.intr.maximum/minimum/... intrinsics.
+// RUN: mlir-opt %s -arith-expand="include-min-max=false" -split-input-file | FileCheck %s --check-prefix=KEEP
+
+// EXPAND-LABEL: func @maximumf
+// KEEP-LABEL:   func @maximumf
+func.func @maximumf(%a: f32, %b: f32) -> f32 {
+  // EXPAND: arith.cmpf ugt
+  // EXPAND: arith.select
+  // EXPAND: arith.cmpf uno
+  // EXPAND: arith.select
+  // EXPAND-NOT: arith.maximumf
+  // KEEP: arith.maximumf
+  // KEEP-NOT: arith.select
+  %result = arith.maximumf %a, %b : f32
+  return %result : f32
+}
+
+// -----
+
+// EXPAND-LABEL: func @minnumf
+// KEEP-LABEL:   func @minnumf
+func.func @minnumf(%a: f32, %b: f32) -> f32 {
+  // EXPAND: arith.cmpf ult
+  // EXPAND: arith.select
+  // EXPAND-NOT: arith.minnumf
+  // KEEP: arith.minnumf
+  // KEEP-NOT: arith.select
+  %result = arith.minnumf %a, %b : f32
+  return %result : f32
+}
+
+// -----
+
+// EXPAND-LABEL: func @maxsi
+// KEEP-LABEL:   func @maxsi
+func.func @maxsi(%a: i32, %b: i32) -> i32 {
+  // EXPAND: arith.cmpi sgt
+  // EXPAND: arith.select
+  // EXPAND-NOT: arith.maxsi
+  // KEEP: arith.maxsi
+  // KEEP-NOT: arith.select
+  %result = arith.maxsi %a, %b : i32
+  return %result : i32
+}
+
+// -----
+
+// Even with min/max expansion disabled, the other expansions (here
+// ceildivsi) still run.
+
+// EXPAND-LABEL: func @ceildivi_still_expands
+// KEEP-LABEL:   func @ceildivi_still_expands
+func.func @ceildivi_still_expands(%a: i32, %b: i32) -> i32 {
+  // EXPAND-NOT: arith.ceildivsi
+  // KEEP-NOT: arith.ceildivsi
+  %result = arith.ceildivsi %a, %b : i32
+  return %result : i32
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/211884

From llvmlistbot at llvm.org  Fri Jul 24 12:56:48 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 24 Jul 2026 12:56:48 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a63c380.c56fb102.11008c.ca2b@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 5971da3896e830c9242b1f73fa7d64a5fa39cb95 Mon Sep 17 00:00:00 2001
From: Victor Perez <victor.pc.upm at gmail.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

A special value that the target type cannot represent is now a parse
error rather than an APFloat abort: the parser rejects infinity, NaN, or
a negative value when the type's semantics lack the matching property
(e.g. `+inf` for a type without infinity, or `-...` for an unsigned
type).

The printer emits the human-readable form by default. The
--mlir-print-float-special-literals-as-hex flag restores the legacy
output, where infinities and NaNs print as a hexadecimal bit pattern.
The parser accepts both forms regardless of the flag. The flag is wired
into every tool that registers the AsmPrinter command-line options
(mlir-opt, mlir-translate, and flang's bbc).

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.

Signed-off-by: Victor Perez <victor.pc.upm at gmail.com>
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  16 +++
 mlir/include/mlir/IR/OperationSupport.h       |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  64 ++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  55 +++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  90 +++++++++++---
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals-hex.mlir          |  50 ++++++++
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  95 ++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 41 files changed, 637 insertions(+), 181 deletions(-)
 create mode 100644 mlir/test/IR/float-literals-hex.mlir
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..2ba81375a6ac4 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,25 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
+The `--mlir-print-float-special-literals-as-hex` flag makes the printer emit infinities
+and NaNs as a `0x` bit pattern instead of the human-readable form. This restores
+the legacy output for backwards compatibility. The parser accepts both forms
+regardless of the flag.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/include/mlir/IR/OperationSupport.h b/mlir/include/mlir/IR/OperationSupport.h
index 942b55e8f7a76..64f9ecfa2f1d7 100644
--- a/mlir/include/mlir/IR/OperationSupport.h
+++ b/mlir/include/mlir/IR/OperationSupport.h
@@ -1239,6 +1239,10 @@ class OpPrintingFlags {
   /// Print SSA IDs using their NameLoc, if provided, as prefix.
   OpPrintingFlags &printNameLocAsPrefix(bool enable = true);
 
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern
+  /// instead of the human-readable form. Restores the legacy output.
+  OpPrintingFlags &printFloatSpecialLiteralsAsHex(bool enable = true);
+
   /// Return if the given ElementsAttr should be elided.
   bool shouldElideElementsAttr(ElementsAttr attr) const;
 
@@ -1282,6 +1286,10 @@ class OpPrintingFlags {
   /// IDs
   bool shouldUseNameLocAsPrefix() const;
 
+  /// Return if special float literals (inf, NaN) should print as a hexadecimal
+  /// bit pattern.
+  bool shouldPrintFloatSpecialLiteralsAsHex() const;
+
 private:
   /// Elide large elements attributes if the number of elements is larger than
   /// the upper limit.
@@ -1318,6 +1326,9 @@ class OpPrintingFlags {
 
   /// Print SSA IDs using NameLocs as prefixes
   bool useNameLocAsPrefix : 1;
+
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern.
+  bool printFloatSpecialLiteralsAsHexFlag : 1;
 };
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..3191b9e2b1d92 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,46 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special float literal (inf/NaN) from `fromSign`, which
+/// starts at the already-consumed sign. On success returns a `floatliteral`
+/// (or `error`) token and advances `curPtr`, else std::nullopt (curPtr intact).
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan(0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Reject `inf`/`qnan` that is only a prefix of a longer identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() && (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+                          rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // `nan(0x..)` / `snan(0x..)`: capture through ')'; APFloat validates later.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1;
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +369,28 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // C-style hex float (`0x1.fp13`): a '.'/'p' distinguishes it from the
+    // bit-pattern form; the binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..835cd892c9a19 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,55 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // The sign is either a preceding '-' token (isNegative) or folded into an
+  // inf/NaN spelling; both at once (e.g. `-+inf`) is invalid. Drop a leading
+  // '+' since convertFromString rejects it on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // Reject values the type cannot represent; APFloat would otherwise abort.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // convertFromString does not accept "qnan".
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build in the target semantics to preserve NaN payloads; overflow/underflow
+  // are tolerated (inf/zero).
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..b1f91a5d6f261 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -201,6 +201,11 @@ struct AsmPrinterOptions {
   llvm::cl::opt<bool> useNameLocAsPrefix{
       "mlir-use-nameloc-as-prefix", llvm::cl::init(false),
       llvm::cl::desc("Print SSA IDs using NameLocs as prefixes")};
+
+  llvm::cl::opt<bool> printFloatSpecialLiteralsAsHexOpt{
+      "mlir-print-float-special-literals-as-hex", llvm::cl::init(false),
+      llvm::cl::desc("Print special float literals (inf, NaN) as a hexadecimal "
+                     "bit pattern instead of the human-readable form")};
 };
 } // namespace
 
@@ -219,7 +224,7 @@ OpPrintingFlags::OpPrintingFlags()
       printGenericOpFormFlag(false), skipRegionsFlag(false),
       assumeVerifiedFlag(false), printLocalScope(false),
       printValueUsersFlag(false), printUniqueSSAIDsFlag(false),
-      useNameLocAsPrefix(false) {
+      useNameLocAsPrefix(false), printFloatSpecialLiteralsAsHexFlag(false) {
   // Initialize based upon command line options, if they are available.
   if (!clOptions.isConstructed())
     return;
@@ -239,6 +244,7 @@ OpPrintingFlags::OpPrintingFlags()
   printValueUsersFlag = clOptions->printValueUsers;
   printUniqueSSAIDsFlag = clOptions->printUniqueSSAIDs;
   useNameLocAsPrefix = clOptions->useNameLocAsPrefix;
+  printFloatSpecialLiteralsAsHexFlag = clOptions->printFloatSpecialLiteralsAsHexOpt;
 }
 
 /// Enable the elision of large elements attributes, by printing a '...'
@@ -331,6 +337,11 @@ OpPrintingFlags &OpPrintingFlags::printNameLocAsPrefix(bool enable) {
   return *this;
 }
 
+OpPrintingFlags &OpPrintingFlags::printFloatSpecialLiteralsAsHex(bool enable) {
+  printFloatSpecialLiteralsAsHexFlag = enable;
+  return *this;
+}
+
 /// Return the size limit for printing large ElementsAttr.
 std::optional<int64_t> OpPrintingFlags::getLargeElementsAttrLimit() const {
   return elementsAttrElementLimit;
@@ -387,6 +398,11 @@ bool OpPrintingFlags::shouldUseNameLocAsPrefix() const {
   return useNameLocAsPrefix;
 }
 
+/// Return if special float literals should print as a hexadecimal bit pattern.
+bool OpPrintingFlags::shouldPrintFloatSpecialLiteralsAsHex() const {
+  return printFloatSpecialLiteralsAsHexFlag;
+}
+
 //===----------------------------------------------------------------------===//
 // NewLineCounter
 //===----------------------------------------------------------------------===//
@@ -418,6 +434,9 @@ class AsmPrinter::Impl {
   /// Returns the output stream of the printer.
   raw_ostream &getStream() { return os; }
 
+  /// Returns the printing flags of the printer.
+  const OpPrintingFlags &getPrinterFlags() const { return printerFlags; }
+
   /// Print a newline and indent the printer to the start of the current
   /// operation/attribute/type.
   /// Note: For attributes and types this method should only be used in
@@ -2256,14 +2275,11 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// Print a floating point value in a way that the parser will be able to
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
-                            bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+                            bool *printedHex = nullptr,
+                            bool printFloatSpecialLiteralsAsHex = false) {
+  // Finite values print as decimal; this is the only form whose type may be
+  // elided.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2311,42 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The rest (inf, NaN, hex bit-pattern) are typeless; keep the type.
   if (printedHex)
     *printedHex = true;
+
+  // The legacy form prints inf/NaN as a hexadecimal bit pattern too.
+  if (!printFloatSpecialLiteralsAsHex && apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (!printFloatSpecialLiteralsAsHex && apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // Preferred quiet NaN = only the quiet (sign-mask) bit set. A zero-width
+    // payload (no mantissa bits, e.g. f8E8M0FNU) has none, so print qnan.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Drop the quiet/signaling bit; trim leading zeros.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern.
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
@@ -2472,7 +2516,8 @@ void AsmPrinter::Impl::printAttributeImpl(Attribute attr,
 
   } else if (auto floatAttr = llvm::dyn_cast<FloatAttr>(attr)) {
     bool printedHex = false;
-    printFloatValue(floatAttr.getValue(), os, &printedHex);
+    printFloatValue(floatAttr.getValue(), os, &printedHex,
+                    printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
 
     // FloatAttr elides the type if F64.
     if (typeElision == AttrTypeElision::May && floatAttr.getType().isF64() &&
@@ -2702,9 +2747,11 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
       printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
         auto complexValue = *(valueIt + index);
         os << "(";
-        printFloatValue(complexValue.real(), os);
+        printFloatValue(complexValue.real(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ",";
-        printFloatValue(complexValue.imag(), os);
+        printFloatValue(complexValue.imag(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ")";
       });
     }
@@ -2717,7 +2764,8 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
     assert(llvm::isa<FloatType>(elementType) && "unexpected element type");
     auto valueIt = attr.value_begin<APFloat>();
     printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
-      printFloatValue(*(valueIt + index), os);
+      printFloatValue(*(valueIt + index), os, /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     });
   }
 }
@@ -2768,7 +2816,8 @@ void AsmPrinter::Impl::printDenseArrayAttr(DenseArrayAttr attr) {
       printDenseIntElement(value, getStream(), type);
     } else {
       APFloat fltVal(llvm::cast<FloatType>(type).getFloatSemantics(), value);
-      printFloatValue(fltVal, getStream());
+      printFloatValue(fltVal, getStream(), /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     }
   };
   llvm::interleaveComma(llvm::seq<unsigned>(0, attr.size()), getStream(),
@@ -3063,7 +3112,8 @@ void AsmPrinter::decreaseIndent() {
 /// Print the given floating point value in a stablized form.
 void AsmPrinter::printFloat(const APFloat &value) {
   assert(impl && "expected AsmPrinter::printFloat to be overriden");
-  printFloatValue(value, impl->getStream());
+  printFloatValue(value, impl->getStream(), /*printedHex=*/nullptr,
+                  impl->getPrinterFlags().shouldPrintFloatSpecialLiteralsAsHex());
 }
 
 void AsmPrinter::printType(Type type) {
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index e559ea5f3c0ab..26ac36f89e1f1 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2019,7 +2019,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2028,7 +2028,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals-hex.mlir b/mlir/test/IR/float-literals-hex.mlir
new file mode 100644
index 0000000000000..601ea62e7ad6e
--- /dev/null
+++ b/mlir/test/IR/float-literals-hex.mlir
@@ -0,0 +1,50 @@
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | FileCheck %s --check-prefix=HEX
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | mlir-opt | FileCheck %s --check-prefix=ROUNDTRIP
+
+// The --mlir-print-float-special-literals-as-hex flag prints infinities and NaNs as a
+// hexadecimal bit pattern (the legacy form). The parser still accepts both the
+// hex and the human-readable spellings, so the hex output round-trips back to
+// the human-readable form.
+
+// HEX-LABEL: @infinities
+// ROUNDTRIP-LABEL: @infinities
+func.func @infinities() {
+  // HEX: arith.constant 0x7F800000 : f32
+  // ROUNDTRIP: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // HEX: arith.constant 0xFF800000 : f32
+  // ROUNDTRIP: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // HEX: arith.constant 0x7FF0000000000000 : f64
+  // ROUNDTRIP: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  return
+}
+
+// HEX-LABEL: @nans
+// ROUNDTRIP-LABEL: @nans
+func.func @nans() {
+  // HEX: arith.constant 0x7FC00000 : f32
+  // ROUNDTRIP: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // HEX: arith.constant 0xFFF8000000000000 : f64
+  // ROUNDTRIP: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // HEX: arith.constant 0x7FC00001 : f32
+  // ROUNDTRIP: arith.constant +nan(0x1) : f32
+  %2 = arith.constant +nan(0x1) : f32
+  // HEX: arith.constant 0x7FF0000000000001 : f64
+  // ROUNDTRIP: arith.constant +snan(0x1) : f64
+  %3 = arith.constant +snan(0x1) : f64
+  return
+}
+
+// Special values also print as hex inside dense elements attributes.
+// HEX-LABEL: @special_values_in_elements
+// ROUNDTRIP-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // HEX: dense<[0x7F800000, 0xFF800000, 0x7FC00000]> : tensor<3xf32>
+  // ROUNDTRIP: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  return
+}
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..03a25f43f09b7 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,16 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The diagnostic must point at the float literal, not the following token
+// (the `}` is on the next line, exposing a stale location).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +685,88 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // Doubly-signed literal: a '-' token before an already-signed inf/NaN.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN has no Inf encoding: reject, do not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN has no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has NaN but no Inf.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: reject negatives, do not crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Fri Jul 24 12:58:36 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 12:58:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Add missing spacing to tensor
 literal closing bracket (PR #195502)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/195502@github.com>
Message-ID: <6a63c3ec.f06f4994.3876ce.5190@mx.google.com>


https://github.com/rdong8 updated https://github.com/llvm/llvm-project/pull/195502

>From 337f08fce5b4bfd1d270c17165908c80f1727a85 Mon Sep 17 00:00:00 2001
From: Richard Dong <66289396+rdong8 at users.noreply.github.com>
Date: Sun, 3 May 2026 06:10:39 +0000
Subject: [PATCH] [mlir][toy] Add missing spacing to tensor literal closing
 bracket

---
 mlir/docs/Tutorials/Toy/Ch-1.md           | 4 ++--
 mlir/examples/toy/Ch1/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch2/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch3/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch4/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch5/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch6/parser/AST.cpp      | 2 +-
 mlir/examples/toy/Ch7/parser/AST.cpp      | 2 +-
 mlir/test/Examples/Toy/Ch1/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch2/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch3/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch4/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch5/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch6/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch7/ast.toy        | 4 ++--
 mlir/test/Examples/Toy/Ch7/struct-ast.toy | 4 ++--
 16 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/mlir/docs/Tutorials/Toy/Ch-1.md b/mlir/docs/Tutorials/Toy/Ch-1.md
index 09cee88db419d..777ea637a932d 100644
--- a/mlir/docs/Tutorials/Toy/Ch-1.md
+++ b/mlir/docs/Tutorials/Toy/Ch-1.md
@@ -91,9 +91,9 @@ Module:
     Params: []
     Block {
       VarDecl a<> @test/Examples/Toy/Ch1/ast.toy:11:3
-        Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @test/Examples/Toy/Ch1/ast.toy:11:11
+        Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @test/Examples/Toy/Ch1/ast.toy:11:11
       VarDecl b<2, 3> @test/Examples/Toy/Ch1/ast.toy:15:3
-        Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @test/Examples/Toy/Ch1/ast.toy:15:17
+        Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @test/Examples/Toy/Ch1/ast.toy:15:17
       VarDecl c<> @test/Examples/Toy/Ch1/ast.toy:19:3
         Call 'multiply_transpose' [ @test/Examples/Toy/Ch1/ast.toy:19:11
           var: a @test/Examples/Toy/Ch1/ast.toy:19:30
diff --git a/mlir/examples/toy/Ch1/parser/AST.cpp b/mlir/examples/toy/Ch1/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch1/parser/AST.cpp
+++ b/mlir/examples/toy/Ch1/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch2/parser/AST.cpp b/mlir/examples/toy/Ch2/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch2/parser/AST.cpp
+++ b/mlir/examples/toy/Ch2/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch3/parser/AST.cpp b/mlir/examples/toy/Ch3/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch3/parser/AST.cpp
+++ b/mlir/examples/toy/Ch3/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch4/parser/AST.cpp b/mlir/examples/toy/Ch4/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch4/parser/AST.cpp
+++ b/mlir/examples/toy/Ch4/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch5/parser/AST.cpp b/mlir/examples/toy/Ch5/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch5/parser/AST.cpp
+++ b/mlir/examples/toy/Ch5/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch6/parser/AST.cpp b/mlir/examples/toy/Ch6/parser/AST.cpp
index 841642488db68..fd9a51aa8b4d2 100644
--- a/mlir/examples/toy/Ch6/parser/AST.cpp
+++ b/mlir/examples/toy/Ch6/parser/AST.cpp
@@ -137,7 +137,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/examples/toy/Ch7/parser/AST.cpp b/mlir/examples/toy/Ch7/parser/AST.cpp
index aa2c7846329c3..809f0552d7f2c 100644
--- a/mlir/examples/toy/Ch7/parser/AST.cpp
+++ b/mlir/examples/toy/Ch7/parser/AST.cpp
@@ -140,7 +140,7 @@ static void printLitHelper(ExprAST *litOrNum) {
   llvm::errs() << "[ ";
   llvm::interleaveComma(literal->getValues(), llvm::errs(),
                         [&](auto &elt) { printLitHelper(elt.get()); });
-  llvm::errs() << "]";
+  llvm::errs() << " ]";
 }
 
 /// Print a literal, see the recursive helper above for the implementation.
diff --git a/mlir/test/Examples/Toy/Ch1/ast.toy b/mlir/test/Examples/Toy/Ch1/ast.toy
index 4af2d25fe1821..0acf9dde09f4b 100644
--- a/mlir/test/Examples/Toy/Ch1/ast.toy
+++ b/mlir/test/Examples/Toy/Ch1/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch2/ast.toy b/mlir/test/Examples/Toy/Ch2/ast.toy
index 48bd4432bf727..674ba30df56c6 100644
--- a/mlir/test/Examples/Toy/Ch2/ast.toy
+++ b/mlir/test/Examples/Toy/Ch2/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch3/ast.toy b/mlir/test/Examples/Toy/Ch3/ast.toy
index 15ac2425390c8..4b86dba5b542d 100644
--- a/mlir/test/Examples/Toy/Ch3/ast.toy
+++ b/mlir/test/Examples/Toy/Ch3/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch4/ast.toy b/mlir/test/Examples/Toy/Ch4/ast.toy
index d665be5e85ab9..1889d9a9e7579 100644
--- a/mlir/test/Examples/Toy/Ch4/ast.toy
+++ b/mlir/test/Examples/Toy/Ch4/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch5/ast.toy b/mlir/test/Examples/Toy/Ch5/ast.toy
index 9840ea2c5663b..10cb1eb5503eb 100644
--- a/mlir/test/Examples/Toy/Ch5/ast.toy
+++ b/mlir/test/Examples/Toy/Ch5/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch6/ast.toy b/mlir/test/Examples/Toy/Ch6/ast.toy
index f5fc2789a2f5c..d7321723b75ec 100644
--- a/mlir/test/Examples/Toy/Ch6/ast.toy
+++ b/mlir/test/Examples/Toy/Ch6/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch7/ast.toy b/mlir/test/Examples/Toy/Ch7/ast.toy
index 878450a6affaa..47c31ac10bab8 100644
--- a/mlir/test/Examples/Toy/Ch7/ast.toy
+++ b/mlir/test/Examples/Toy/Ch7/ast.toy
@@ -48,9 +48,9 @@ def main() {
 # CHECK-NEXT:       Params: []
 # CHECK-NEXT:       Block {
 # CHECK-NEXT:         VarDecl a<> @{{.*}}ast.toy:11:3
-# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}ast.toy:11:11
+# CHECK-NEXT:           Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}ast.toy:11:11
 # CHECK-NEXT:         VarDecl b<2, 3> @{{.*}}ast.toy:15:3
-# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00] @{{.*}}ast.toy:15:17
+# CHECK-NEXT:           Literal: <6>[ 1.000000e+00, 2.000000e+00, 3.000000e+00, 4.000000e+00, 5.000000e+00, 6.000000e+00 ] @{{.*}}ast.toy:15:17
 # CHECK-NEXT:         VarDecl c<> @{{.*}}ast.toy:19:3
 # CHECK-NEXT:           Call 'multiply_transpose' [ @{{.*}}ast.toy:19:11
 # CHECK-NEXT:             var: a @{{.*}}ast.toy:19:30
diff --git a/mlir/test/Examples/Toy/Ch7/struct-ast.toy b/mlir/test/Examples/Toy/Ch7/struct-ast.toy
index d2ccc5accd1cb..80dacfa948ebb 100644
--- a/mlir/test/Examples/Toy/Ch7/struct-ast.toy
+++ b/mlir/test/Examples/Toy/Ch7/struct-ast.toy
@@ -48,8 +48,8 @@ def main() {
 # CHECK-NEXT:  Params: []
 # CHECK-NEXT:  Block {
 # CHECK-NEXT:    VarDecl value<Struct> @{{.*}}struct-ast.toy:16:3
-# CHECK-NEXT:      Struct Literal:             Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}struct-ast.toy:16:19
-# CHECK-NEXT:        Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00]] @{{.*}}struct-ast.toy:16:43
+# CHECK-NEXT:      Struct Literal:             Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}struct-ast.toy:16:19
+# CHECK-NEXT:        Literal: <2, 3>[ <3>[ 1.000000e+00, 2.000000e+00, 3.000000e+00 ], <3>[ 4.000000e+00, 5.000000e+00, 6.000000e+00 ] ] @{{.*}}struct-ast.toy:16:43
 # CHECK-NEXT:       @{{.*}}struct-ast.toy:16:18
 # CHECK-NEXT:    VarDecl c<> @{{.*}}struct-ast.toy:19:3
 # CHECK-NEXT:      Call 'multiply_transpose' [ @{{.*}}struct-ast.toy:19:11


From llvmlistbot at llvm.org  Fri Jul 24 12:59:41 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 24 Jul 2026 12:59:41 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a63c42d.507fc713.3a4929.f3ab@mx.google.com>


https://github.com/victor-eds edited https://github.com/llvm/llvm-project/pull/210422

From llvmlistbot at llvm.org  Fri Jul 24 13:01:01 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Fri, 24 Jul 2026 13:01:01 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a63c47d.d681cab9.3d08c3.5975@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From 8f618007d4517ae1ef7938074c1126574d4663ae Mon Sep 17 00:00:00 2001
From: Victor Perez <victor.pc.upm at gmail.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly. The printer mirrors LLVM's writeAPFloatInternal.

A special value that the target type cannot represent is now a parse
error rather than an APFloat abort: the parser rejects infinity, NaN, or
a negative value when the type's semantics lack the matching property
(e.g. `+inf` for a type without infinity, or `-...` for an unsigned
type).

The printer emits the human-readable form by default. The
--mlir-print-float-special-literals-as-hex flag restores the legacy
output, where infinities and NaNs print as a hexadecimal bit pattern.
The parser accepts both forms regardless of the flag. The flag is wired
into every tool that registers the AsmPrinter command-line options
(mlir-opt, mlir-translate, and flang's bbc).

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.

Signed-off-by: Victor Perez <victor.pc.upm at gmail.com>
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  16 +++
 mlir/include/mlir/IR/OperationSupport.h       |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++--
 mlir/lib/AsmParser/Lexer.cpp                  |  64 ++++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  55 +++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    |  92 +++++++++++---
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 +++----
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +--
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   4 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +--
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +--
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals-hex.mlir          |  50 ++++++++
 mlir/test/IR/float-literals.mlir              | 117 ++++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  |  95 ++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++++-----
 41 files changed, 639 insertions(+), 181 deletions(-)
 create mode 100644 mlir/test/IR/float-literals-hex.mlir
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..2ba81375a6ac4 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,25 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` `(` `0x` hex_digit+ `)`)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. Finite values may also be written as C-style hexadecimal floats
+(`0x1.8p3`). The `0x` bit-pattern form without a fractional part or exponent
+(e.g. `0x7F800000`) is instead interpreted as the raw bit pattern of the target
+type.
+
+The `--mlir-print-float-special-literals-as-hex` flag makes the printer emit infinities
+and NaNs as a `0x` bit pattern instead of the human-readable form. This restores
+the legacy output for backwards compatibility. The parser accepts both forms
+regardless of the flag.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/include/mlir/IR/OperationSupport.h b/mlir/include/mlir/IR/OperationSupport.h
index 942b55e8f7a76..64f9ecfa2f1d7 100644
--- a/mlir/include/mlir/IR/OperationSupport.h
+++ b/mlir/include/mlir/IR/OperationSupport.h
@@ -1239,6 +1239,10 @@ class OpPrintingFlags {
   /// Print SSA IDs using their NameLoc, if provided, as prefix.
   OpPrintingFlags &printNameLocAsPrefix(bool enable = true);
 
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern
+  /// instead of the human-readable form. Restores the legacy output.
+  OpPrintingFlags &printFloatSpecialLiteralsAsHex(bool enable = true);
+
   /// Return if the given ElementsAttr should be elided.
   bool shouldElideElementsAttr(ElementsAttr attr) const;
 
@@ -1282,6 +1286,10 @@ class OpPrintingFlags {
   /// IDs
   bool shouldUseNameLocAsPrefix() const;
 
+  /// Return if special float literals (inf, NaN) should print as a hexadecimal
+  /// bit pattern.
+  bool shouldPrintFloatSpecialLiteralsAsHex() const;
+
 private:
   /// Elide large elements attributes if the number of elements is larger than
   /// the upper limit.
@@ -1318,6 +1326,9 @@ class OpPrintingFlags {
 
   /// Print SSA IDs using NameLocs as prefixes
   bool useNameLocAsPrefix : 1;
+
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern.
+  bool printFloatSpecialLiteralsAsHexFlag : 1;
 };
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..3191b9e2b1d92 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,46 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special float literal (inf/NaN) from `fromSign`, which
+/// starts at the already-consumed sign. On success returns a `floatliteral`
+/// (or `error`) token and advances `curPtr`, else std::nullopt (curPtr intact).
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan(0x` hex_digit+ `)`)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // Reject `inf`/`qnan` that is only a prefix of a longer identifier.
+  if (keyword == "inf" || keyword == "qnan") {
+    if (!rest.empty() && (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+                          rest.front() == '$' || rest.front() == '.'))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // `nan(0x..)` / `snan(0x..)`: capture through ')'; APFloat validates later.
+  if ((keyword == "nan" || keyword == "snan") && rest.consume_front("(")) {
+    StringRef payload = rest.take_until([](char c) { return c == ')'; });
+    if (payload.end() == rest.end()) {
+      curPtr = afterSign.end();
+      return emitError(tokStart, "expected ')' in NaN literal");
+    }
+    curPtr = payload.end() + 1;
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +369,28 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // C-style hex float (`0x1.fp13`): a '.'/'p' distinguishes it from the
+    // bit-pattern form; the binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..835cd892c9a19 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,55 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // The sign is either a preceding '-' token (isNegative) or folded into an
+  // inf/NaN spelling; both at once (e.g. `-+inf`) is invalid. Drop a leading
+  // '+' since convertFromString rejects it on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // Reject values the type cannot represent; APFloat would otherwise abort.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // convertFromString does not accept "qnan".
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build in the target semantics to preserve NaN payloads; overflow/underflow
+  // are tolerated (inf/zero).
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..27a473d3432b0 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -201,6 +201,11 @@ struct AsmPrinterOptions {
   llvm::cl::opt<bool> useNameLocAsPrefix{
       "mlir-use-nameloc-as-prefix", llvm::cl::init(false),
       llvm::cl::desc("Print SSA IDs using NameLocs as prefixes")};
+
+  llvm::cl::opt<bool> printFloatSpecialLiteralsAsHexOpt{
+      "mlir-print-float-special-literals-as-hex", llvm::cl::init(false),
+      llvm::cl::desc("Print special float literals (inf, NaN) as a hexadecimal "
+                     "bit pattern instead of the human-readable form")};
 };
 } // namespace
 
@@ -219,7 +224,7 @@ OpPrintingFlags::OpPrintingFlags()
       printGenericOpFormFlag(false), skipRegionsFlag(false),
       assumeVerifiedFlag(false), printLocalScope(false),
       printValueUsersFlag(false), printUniqueSSAIDsFlag(false),
-      useNameLocAsPrefix(false) {
+      useNameLocAsPrefix(false), printFloatSpecialLiteralsAsHexFlag(false) {
   // Initialize based upon command line options, if they are available.
   if (!clOptions.isConstructed())
     return;
@@ -239,6 +244,8 @@ OpPrintingFlags::OpPrintingFlags()
   printValueUsersFlag = clOptions->printValueUsers;
   printUniqueSSAIDsFlag = clOptions->printUniqueSSAIDs;
   useNameLocAsPrefix = clOptions->useNameLocAsPrefix;
+  printFloatSpecialLiteralsAsHexFlag =
+      clOptions->printFloatSpecialLiteralsAsHexOpt;
 }
 
 /// Enable the elision of large elements attributes, by printing a '...'
@@ -331,6 +338,11 @@ OpPrintingFlags &OpPrintingFlags::printNameLocAsPrefix(bool enable) {
   return *this;
 }
 
+OpPrintingFlags &OpPrintingFlags::printFloatSpecialLiteralsAsHex(bool enable) {
+  printFloatSpecialLiteralsAsHexFlag = enable;
+  return *this;
+}
+
 /// Return the size limit for printing large ElementsAttr.
 std::optional<int64_t> OpPrintingFlags::getLargeElementsAttrLimit() const {
   return elementsAttrElementLimit;
@@ -387,6 +399,11 @@ bool OpPrintingFlags::shouldUseNameLocAsPrefix() const {
   return useNameLocAsPrefix;
 }
 
+/// Return if special float literals should print as a hexadecimal bit pattern.
+bool OpPrintingFlags::shouldPrintFloatSpecialLiteralsAsHex() const {
+  return printFloatSpecialLiteralsAsHexFlag;
+}
+
 //===----------------------------------------------------------------------===//
 // NewLineCounter
 //===----------------------------------------------------------------------===//
@@ -418,6 +435,9 @@ class AsmPrinter::Impl {
   /// Returns the output stream of the printer.
   raw_ostream &getStream() { return os; }
 
+  /// Returns the printing flags of the printer.
+  const OpPrintingFlags &getPrinterFlags() const { return printerFlags; }
+
   /// Print a newline and indent the printer to the start of the current
   /// operation/attribute/type.
   /// Note: For attributes and types this method should only be used in
@@ -2256,14 +2276,11 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// Print a floating point value in a way that the parser will be able to
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
-                            bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+                            bool *printedHex = nullptr,
+                            bool printFloatSpecialLiteralsAsHex = false) {
+  // Finite values print as decimal; this is the only form whose type may be
+  // elided.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2312,42 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The rest (inf, NaN, hex bit-pattern) are typeless; keep the type.
   if (printedHex)
     *printedHex = true;
+
+  // The legacy form prints inf/NaN as a hexadecimal bit pattern too.
+  if (!printFloatSpecialLiteralsAsHex && apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (!printFloatSpecialLiteralsAsHex && apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    APInt payload = apValue.getNaNPayload();
+    // Preferred quiet NaN = only the quiet (sign-mask) bit set. A zero-width
+    // payload (no mantissa bits, e.g. f8E8M0FNU) has none, so print qnan.
+    if (payload.getBitWidth() == 0 || payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Drop the quiet/signaling bit; trim leading zeros.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern.
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
@@ -2472,7 +2517,8 @@ void AsmPrinter::Impl::printAttributeImpl(Attribute attr,
 
   } else if (auto floatAttr = llvm::dyn_cast<FloatAttr>(attr)) {
     bool printedHex = false;
-    printFloatValue(floatAttr.getValue(), os, &printedHex);
+    printFloatValue(floatAttr.getValue(), os, &printedHex,
+                    printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
 
     // FloatAttr elides the type if F64.
     if (typeElision == AttrTypeElision::May && floatAttr.getType().isF64() &&
@@ -2702,9 +2748,11 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
       printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
         auto complexValue = *(valueIt + index);
         os << "(";
-        printFloatValue(complexValue.real(), os);
+        printFloatValue(complexValue.real(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ",";
-        printFloatValue(complexValue.imag(), os);
+        printFloatValue(complexValue.imag(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ")";
       });
     }
@@ -2717,7 +2765,8 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
     assert(llvm::isa<FloatType>(elementType) && "unexpected element type");
     auto valueIt = attr.value_begin<APFloat>();
     printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
-      printFloatValue(*(valueIt + index), os);
+      printFloatValue(*(valueIt + index), os, /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     });
   }
 }
@@ -2768,7 +2817,8 @@ void AsmPrinter::Impl::printDenseArrayAttr(DenseArrayAttr attr) {
       printDenseIntElement(value, getStream(), type);
     } else {
       APFloat fltVal(llvm::cast<FloatType>(type).getFloatSemantics(), value);
-      printFloatValue(fltVal, getStream());
+      printFloatValue(fltVal, getStream(), /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     }
   };
   llvm::interleaveComma(llvm::seq<unsigned>(0, attr.size()), getStream(),
@@ -3063,7 +3113,9 @@ void AsmPrinter::decreaseIndent() {
 /// Print the given floating point value in a stablized form.
 void AsmPrinter::printFloat(const APFloat &value) {
   assert(impl && "expected AsmPrinter::printFloat to be overriden");
-  printFloatValue(value, impl->getStream());
+  printFloatValue(
+      value, impl->getStream(), /*printedHex=*/nullptr,
+      impl->getPrinterFlags().shouldPrintFloatSpecialLiteralsAsHex());
 }
 
 void AsmPrinter::printType(Type type) {
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index 3459164c5c0a7..6ca81b47a0675 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index e559ea5f3c0ab..26ac36f89e1f1 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2019,7 +2019,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2028,7 +2028,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals-hex.mlir b/mlir/test/IR/float-literals-hex.mlir
new file mode 100644
index 0000000000000..601ea62e7ad6e
--- /dev/null
+++ b/mlir/test/IR/float-literals-hex.mlir
@@ -0,0 +1,50 @@
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | FileCheck %s --check-prefix=HEX
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | mlir-opt | FileCheck %s --check-prefix=ROUNDTRIP
+
+// The --mlir-print-float-special-literals-as-hex flag prints infinities and NaNs as a
+// hexadecimal bit pattern (the legacy form). The parser still accepts both the
+// hex and the human-readable spellings, so the hex output round-trips back to
+// the human-readable form.
+
+// HEX-LABEL: @infinities
+// ROUNDTRIP-LABEL: @infinities
+func.func @infinities() {
+  // HEX: arith.constant 0x7F800000 : f32
+  // ROUNDTRIP: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // HEX: arith.constant 0xFF800000 : f32
+  // ROUNDTRIP: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // HEX: arith.constant 0x7FF0000000000000 : f64
+  // ROUNDTRIP: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  return
+}
+
+// HEX-LABEL: @nans
+// ROUNDTRIP-LABEL: @nans
+func.func @nans() {
+  // HEX: arith.constant 0x7FC00000 : f32
+  // ROUNDTRIP: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // HEX: arith.constant 0xFFF8000000000000 : f64
+  // ROUNDTRIP: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // HEX: arith.constant 0x7FC00001 : f32
+  // ROUNDTRIP: arith.constant +nan(0x1) : f32
+  %2 = arith.constant +nan(0x1) : f32
+  // HEX: arith.constant 0x7FF0000000000001 : f64
+  // ROUNDTRIP: arith.constant +snan(0x1) : f64
+  %3 = arith.constant +snan(0x1) : f64
+  return
+}
+
+// Special values also print as hex inside dense elements attributes.
+// HEX-LABEL: @special_values_in_elements
+// ROUNDTRIP-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // HEX: dense<[0x7F800000, 0xFF800000, 0x7FC00000]> : tensor<3xf32>
+  // ROUNDTRIP: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  return
+}
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..d26ad638eeb13
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,117 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU has zero mantissa bits: its NaN prints as the canonical qnan.
+  // CHECK: arith.constant +qnan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // Payload wider than the mantissa is masked (f8E5M2 keeps 1 payload bit).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %1 = arith.constant +nan(0x7) : f8E5M2
+  // FNUZ formats have a single NaN encoding; a signaling request collapses to it.
+  // CHECK: arith.constant -nan(0x0) : f8E4M3FNUZ
+  %2 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..03a25f43f09b7 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,16 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The diagnostic must point at the float literal, not the following token
+// (the `}` is on the next line, exposing a stale location).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +685,88 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // Doubly-signed literal: a '-' token before an already-signed inf/NaN.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN has no Inf encoding: reject, do not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN has no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has NaN but no Inf.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: reject negatives, do not crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Fri Jul 24 18:57:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 18:57:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 571a1de - [LoopInfo] Store blocks using Euler
 tour representation (#211485)
Message-ID: <6a6417f3.1728eaf3.12b96d.4e25@mx.google.com>


Author: Fangrui Song
Date: 2026-07-25T01:57:01Z
New Revision: 571a1deb31125780ffb86fbae027b03efd0dcdd8

URL: https://github.com/llvm/llvm-project/commit/571a1deb31125780ffb86fbae027b03efd0dcdd8
DIFF: https://github.com/llvm/llvm-project/commit/571a1deb31125780ffb86fbae027b03efd0dcdd8.diff

LOG: [LoopInfo] Store blocks using Euler tour representation (#211485)

Block list construction appends each block to all enclosing loops
(O(depth) per block), and each loop owns a separately allocated vector.

Switch to an Euler tour representation: one BlockLayout array per
LoopInfo holding the in-loop blocks in a loop-contiguous reverse
postorder, each loop's list a [begin, end) slice of it, subloop slices
nested inside their parent's. Headers remain first;
`SubLoops`/`TopLevelLoops` orders are unchanged. A few tests observe the
order and are updated. (Previously a loop's list was the function's
reverse postorder restricted to its members, so a subloop's blocks were
interleaved with the parent's own blocks at their RPO positions.)

Mutations first copy a borrowed slice into private storage from
LoopInfo's allocator. `contains(BlockT *)` remains map-based: a
materialized loop's slice is not a membership test. This copy is rare:
across default<O2> over dagcombiner and sqlite3, only 12--14% of loops
built by analyze() are ever mutated; the rest keep the borrowed slice.

AllocateLoop() no longer forwards constructor arguments; analyze() uses
allocateLoop(Header), removing the header-taking loop constructors.

Aided by Claude Fable 5

Added: 
    

Modified: 
    bolt/include/bolt/Core/BinaryLoop.h
    bolt/test/X86/loop-nest.test
    llvm/include/llvm/Analysis/LoopInfo.h
    llvm/include/llvm/CodeGen/MachineLoopInfo.h
    llvm/include/llvm/Support/GenericLoopInfo.h
    llvm/include/llvm/Support/GenericLoopInfoImpl.h
    llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
    llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
    llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
    mlir/include/mlir/Analysis/CFGLoopInfo.h
    mlir/lib/Analysis/CFGLoopInfo.cpp

Removed: 
    


################################################################################
diff  --git a/bolt/include/bolt/Core/BinaryLoop.h b/bolt/include/bolt/Core/BinaryLoop.h
index b425c75715d8b..1c52a051a84cc 100644
--- a/bolt/include/bolt/Core/BinaryLoop.h
+++ b/bolt/include/bolt/Core/BinaryLoop.h
@@ -36,11 +36,6 @@ class BinaryLoop : public LoopBase<BinaryBasicBlock, BinaryLoop> {
   uint64_t ExitCount{0};
 
   // Most of the public interface is provided by LoopBase.
-
-protected:
-  friend class LoopInfoBase<BinaryBasicBlock, BinaryLoop>;
-  explicit BinaryLoop(BinaryBasicBlock *BB)
-      : LoopBase<BinaryBasicBlock, BinaryLoop>(BB) {}
 };
 
 class BinaryLoopInfo : public LoopInfoBase<BinaryBasicBlock, BinaryLoop> {

diff  --git a/bolt/test/X86/loop-nest.test b/bolt/test/X86/loop-nest.test
index 51c8fcdb32eaa..208cc9ebdfaaf 100644
--- a/bolt/test/X86/loop-nest.test
+++ b/bolt/test/X86/loop-nest.test
@@ -18,6 +18,6 @@ CHECK-NEXT: Loop basic blocks: .Ltmp[[#MAIN_OUTER_HDR]], .Ltmp[[#]]
 CHECK:      Binary Function "foo" after building cfg
 CHECK:      Loop Info for Function "foo"
 CHECK:      Outer loop header: .Ltmp[[#FOO_OUTER_HDR:]]
-CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_BB:]]
+CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_OUTER_HDR]], .Ltmp[[#]], .LFT[[#]], .Ltmp[[#]], .Ltmp[[#FOO_INNER_HDR:]], .Ltmp[[#FOO_INNER_BB:]], .Ltmp[[#]]
 CHECK:      Nested loop header: .Ltmp[[#FOO_INNER_HDR]]
 CHECK-NEXT: Loop basic blocks: .Ltmp[[#FOO_INNER_HDR]], .Ltmp[[#FOO_INNER_BB]]

diff  --git a/llvm/include/llvm/Analysis/LoopInfo.h b/llvm/include/llvm/Analysis/LoopInfo.h
index 13b404505b5e2..15c22e43c44bf 100644
--- a/llvm/include/llvm/Analysis/LoopInfo.h
+++ b/llvm/include/llvm/Analysis/LoopInfo.h
@@ -417,7 +417,6 @@ class LLVM_ABI Loop : public LoopBase<BasicBlock, Loop> {
 
   friend class LoopInfoBase<BasicBlock, Loop>;
   friend class LoopBase<BasicBlock, Loop>;
-  explicit Loop(BasicBlock *BB) : LoopBase<BasicBlock, Loop>(BB) {}
   ~Loop() = default;
 };
 

diff  --git a/llvm/include/llvm/CodeGen/MachineLoopInfo.h b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
index 60b3f6c6a3ba5..7893f70cd353c 100644
--- a/llvm/include/llvm/CodeGen/MachineLoopInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
@@ -96,9 +96,6 @@ class MachineLoop : public LoopBase<MachineBasicBlock, MachineLoop> {
   /// Returns true if the given physreg has no defs inside the loop.
   bool isLoopInvariantImplicitPhysReg(Register Reg) const;
 
-  explicit MachineLoop(MachineBasicBlock *MBB)
-    : LoopBase<MachineBasicBlock, MachineLoop>(MBB) {}
-
   MachineLoop() = default;
 };
 

diff  --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index 0bdfce7dc889e..e2819a21fe0ce 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -51,7 +51,6 @@ namespace llvm {
 
 template <class N, class M> class LoopInfoBase;
 template <class N, class M> class LoopBase;
-template <class N, class M> class PopulateLoopsDFS;
 
 //===----------------------------------------------------------------------===//
 /// Instances of this class are used to represent loops that are detected in the
@@ -62,8 +61,21 @@ template <class BlockT, class LoopT> class LoopBase {
   // Loops contained entirely within this one.
   std::vector<LoopT *> SubLoops;
 
-  // The list of blocks in this loop. First entry is the header node.
-  std::vector<BlockT *> Blocks;
+  // The list of blocks in this loop; first entry is the header. Either borrows
+  // a slice of the owning LoopInfo's BlockLayout, marked by the
+  // BorrowedCapacity sentinel, or is a private allocation of BlockCapacity
+  // slots from its allocator.
+  //
+  // Until analyze()'s layout carve runs, PendingHeader stashes the loop header
+  // (see pendingHeader()).
+  union {
+    BlockT *PendingHeader;
+    BlockT **BlockData = nullptr;
+  };
+  unsigned BlockLen = 0;
+  unsigned BlockCapacity = 0;
+
+  static constexpr unsigned BorrowedCapacity = -1u;
 
   // The LoopInfo that owns this loop. Used to answer contains(BlockT *) from
   // the central block-to-loop map.
@@ -186,7 +198,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Get a list of the basic blocks which make up this loop.
   ArrayRef<BlockT *> getBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks;
+    return ArrayRef<BlockT *>(BlockData, BlockLen);
   }
   using block_iterator = typename ArrayRef<BlockT *>::const_iterator;
   block_iterator block_begin() const { return getBlocks().begin(); }
@@ -200,7 +212,7 @@ template <class BlockT, class LoopT> class LoopBase {
   /// Invalidate the loop, indicating that it is no longer a loop.
   unsigned getNumBlocks() const {
     assert(!isInvalid() && "Loop not in a valid state!");
-    return Blocks.size();
+    return BlockLen;
   }
 
   /// Return true if this loop is no longer valid.  The only valid use of this
@@ -400,19 +412,19 @@ template <class BlockT, class LoopT> class LoopBase {
   /// transformations should use addBasicBlockToLoop.
   void addBlockEntry(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.push_back(BB);
-  }
-
-  /// interface to reverse Blocks[from, end of loop] in this loop
-  void reverseBlock(unsigned from) {
-    assert(!isInvalid() && "Loop not in a valid state!");
-    std::reverse(Blocks.begin() + from, Blocks.end());
+    // A borrowed slice or a full private allocation grows into fresh private
+    // storage before appending.
+    if (BlockCapacity == BorrowedCapacity || BlockLen == BlockCapacity)
+      LI->reallocBlocks(*static_cast<LoopT *>(this),
+                        std::max(2 * BlockLen, 4u));
+    BlockData[BlockLen++] = BB;
   }
 
   /// interface to do reserve() for Blocks
-  void reserveBlocks(unsigned size) {
+  void reserveBlocks(unsigned Size) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    Blocks.reserve(size);
+    if (BlockCapacity < Size)
+      LI->reallocBlocks(*static_cast<LoopT *>(this), Size);
   }
 
   /// interface to do reserve() for SubLoops
@@ -421,21 +433,18 @@ template <class BlockT, class LoopT> class LoopBase {
     SubLoops.reserve(Size);
   }
 
-  /// Capacity of the block list; input to an enclosing loop's reserveBlocks()
-  /// during construction, when this loop's list is not yet fully populated.
-  unsigned getBlocksCapacity() const { return Blocks.capacity(); }
-
   /// This method is used to move BB (which must be part of this loop) to be the
   /// loop header of the loop (the block that dominates all others).
   void moveToHeader(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    if (Blocks[0] == BB)
+    if (BlockData[0] == BB)
       return;
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
     for (unsigned i = 0;; ++i) {
-      assert(i != Blocks.size() && "Loop does not contain BB!");
-      if (Blocks[i] == BB) {
-        Blocks[i] = Blocks[0];
-        Blocks[0] = BB;
+      assert(i != BlockLen && "Loop does not contain BB!");
+      if (BlockData[i] == BB) {
+        BlockData[i] = BlockData[0];
+        BlockData[0] = BB;
         return;
       }
     }
@@ -446,9 +455,12 @@ template <class BlockT, class LoopT> class LoopBase {
   /// class.
   void removeBlockFromLoop(BlockT *BB) {
     assert(!isInvalid() && "Loop not in a valid state!");
-    auto I = find(Blocks, BB);
+    LI->materializeBlocks(*static_cast<LoopT *>(this));
+    MutableArrayRef<BlockT *> Blocks(BlockData, BlockLen);
+    auto *I = llvm::find(Blocks, BB);
     assert(I != Blocks.end() && "N is not in this list!");
-    Blocks.erase(I);
+    std::move(I + 1, Blocks.end(), I);
+    --BlockLen;
   }
 
   /// Verify loop structure
@@ -469,15 +481,10 @@ template <class BlockT, class LoopT> class LoopBase {
 
 protected:
   friend class LoopInfoBase<BlockT, LoopT>;
-  friend class PopulateLoopsDFS<BlockT, LoopT>;
 
   /// This creates an empty loop.
   LoopBase() : ParentLoop(nullptr) {}
 
-  explicit LoopBase(BlockT *BB) : ParentLoop(nullptr) {
-    Blocks.push_back(BB);
-  }
-
   // Since loop passes like SCEV are allowed to key analysis results off of
   // `Loop` pointers, we cannot re-use pointers within a loop pass manager.
   // This means loop passes should not be `delete` ing `Loop` objects directly
@@ -495,7 +502,10 @@ template <class BlockT, class LoopT> class LoopBase {
     IsInvalid = true;
 #endif
     SubLoops.clear();
-    Blocks.clear();
+    // The block storage is reclaimed by the owning LoopInfo.
+    BlockData = nullptr;
+    BlockLen = 0;
+    BlockCapacity = 0;
     ParentLoop = nullptr;
   }
 };
@@ -520,11 +530,16 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   // occurs in (or null).
   SmallVector<LoopT *> BBMap;
 
-  using ParentT = decltype(std::declval<const BlockT *>()->getParent());
+  using ParentT = decltype(std::declval<BlockT *>()->getParent());
   ParentT ParentPtr = nullptr;
   unsigned BlockNumberEpoch;
 
   std::vector<LoopT *> TopLevelLoops;
+
+  // Shared reverse postorder layout of the in-loop blocks. Each initial loop is
+  // a slice of this array, subloop slices nested inside their parent's.
+  std::unique_ptr<BlockT *[]> BlockLayout;
+
   BumpPtrAllocator LoopAllocator;
 
   friend class LoopBase<BlockT, LoopT>;
@@ -540,6 +555,7 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   LoopInfoBase(LoopInfoBase &&Arg)
       : BBMap(std::move(Arg.BBMap)),
         TopLevelLoops(std::move(Arg.TopLevelLoops)),
+        BlockLayout(std::move(Arg.BlockLayout)),
         LoopAllocator(std::move(Arg.LoopAllocator)) {
     ParentPtr = Arg.ParentPtr;
     BlockNumberEpoch = Arg.BlockNumberEpoch;
@@ -556,6 +572,7 @@ template <class BlockT, class LoopT> class LoopInfoBase {
       L->~LoopT();
 
     TopLevelLoops = std::move(RHS.TopLevelLoops);
+    BlockLayout = std::move(RHS.BlockLayout);
     LoopAllocator = std::move(RHS.LoopAllocator);
     resetLoopInfoOwners();
     RHS.TopLevelLoops.clear();
@@ -568,12 +585,13 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     for (auto *L : TopLevelLoops)
       L->~LoopT();
     TopLevelLoops.clear();
+    BlockLayout.reset();
     LoopAllocator.Reset();
   }
 
-  template <typename... ArgsTy> LoopT *AllocateLoop(ArgsTy &&...Args) {
+  LoopT *AllocateLoop() {
     LoopT *Storage = LoopAllocator.Allocate<LoopT>();
-    LoopT *L = new (Storage) LoopT(std::forward<ArgsTy>(Args)...);
+    LoopT *L = new (Storage) LoopT();
     L->LI = this;
     return L;
   }
@@ -621,7 +639,8 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   }
 
   /// Verify that used block numbers are still valid.
-  void verifyBlockNumberEpoch(ParentT BBParent) const {
+  void
+  verifyBlockNumberEpoch(const std::remove_pointer_t<ParentT> *BBParent) const {
     assert(ParentPtr == BBParent &&
            "loop info queried with block of other function");
     assert(BlockNumberEpoch ==
@@ -636,6 +655,45 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     return Number < BBMap.size() ? BBMap[Number] : nullptr;
   }
 
+  /// AllocateLoop for analyze(): stash \p Header (see pendingHeader).
+  /// getHeader() only works once the layout carve has replaced the stash with
+  /// the loop's block list.
+  LoopT *allocateLoop(BlockT *Header) {
+    LoopT *L = AllocateLoop();
+    L->PendingHeader = Header;
+    return L;
+  }
+
+  /// The header of a loop under construction, stashed until the layout carve
+  /// builds the block list.
+  static BlockT *pendingHeader(const LoopT *L) { return L->PendingHeader; }
+
+  void discoverAndMapSubloop(LoopT *L, BlockT *Header,
+                             ArrayRef<BlockT *> Backedges,
+                             const DominatorTreeBase<BlockT, false> &DomTree);
+
+  /// True if \p L borrows its block list from BlockLayout.
+  static bool hasBorrowedBlocks(const LoopT &L) {
+    return L.BlockCapacity == LoopT::BorrowedCapacity;
+  }
+
+  /// Replace \p L's block list with a private allocation of NewCapacity
+  /// slots. The old storage is abandoned in place so slices sharing it stay
+  /// intact; it is reclaimed when this LoopInfo is cleared.
+  void reallocBlocks(LoopT &L, unsigned NewCapacity) {
+    assert(NewCapacity >= L.BlockLen && "capacity below size");
+    BlockT **New = LoopAllocator.Allocate<BlockT *>(NewCapacity);
+    llvm::copy(L.getBlocks(), New);
+    L.BlockData = New;
+    L.BlockCapacity = NewCapacity;
+  }
+
+  /// Copy \p L's borrowed block list into private storage before a mutation.
+  void materializeBlocks(LoopT &L) {
+    if (hasBorrowedBlocks(L))
+      reallocBlocks(L, L.BlockLen);
+  }
+
 public:
   /// Return the inner most loop that BB lives in. If a basic block is in no
   /// loop (for example the entry node), null is returned.
@@ -672,7 +730,10 @@ template <class BlockT, class LoopT> class LoopInfoBase {
   /// ancestors or descendants, and not the block-to-loop mapping.
   template <typename PredicateT>
   void removeBlocksIf(LoopT &L, PredicateT Pred) {
-    llvm::erase_if(L.Blocks, Pred);
+    materializeBlocks(L);
+    L.BlockLen = llvm::remove_if(
+                     MutableArrayRef<BlockT *>(L.BlockData, L.BlockLen), Pred) -
+                 L.BlockData;
   }
 
   /// Remove every block satisfying \p Pred from \p Start and each of its

diff  --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index 170f05e9d3a49..5ea9f440dc28b 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -282,7 +282,7 @@ void LoopBase<BlockT, LoopT>::addBasicBlockToLoop(
     BlockT *NewBB, LoopInfoBase<BlockT, LoopT> &LIB) {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  if (!Blocks.empty()) {
+  if (!getBlocks().empty()) {
     auto SameHeader = LIB[getHeader()];
     assert(contains(SameHeader) && getHeader() == SameHeader->getHeader() &&
            "Incorrect LI specified for this loop!");
@@ -325,7 +325,7 @@ template <class BlockT, class LoopT>
 void LoopBase<BlockT, LoopT>::verifyLoop() const {
   assert(!isInvalid() && "Loop not in a valid state!");
 #ifndef NDEBUG
-  assert(!Blocks.empty() && "Loop header is missing");
+  assert(!getBlocks().empty() && "Loop header is missing");
 
   // Setup for using a depth-first iterator to visit every block in the loop.
   SmallVector<BlockT *, 8> ExitBBs;
@@ -371,7 +371,7 @@ void LoopBase<BlockT, LoopT>::verifyLoop() const {
 
   if (VisitedBBs.size() != getNumBlocks()) {
     dbgs() << "The following blocks are unreachable in the loop: ";
-    for (auto *BB : Blocks) {
+    for (auto *BB : getBlocks()) {
       if (!VisitedBBs.count(BB)) {
         dbgs() << *BB << "\n";
       }
@@ -455,12 +455,11 @@ void LoopBase<BlockT, LoopT>::print(raw_ostream &OS, bool Verbose,
 /// this loop are mapped to this loop or a subloop. And all subloops within this
 /// loop have their parent loop set to this loop or a subloop.
 template <class BlockT, class LoopT>
-static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
-                                  LoopInfoBase<BlockT, LoopT> *LI,
-                                  const DomTreeBase<BlockT> &DomTree) {
+void LoopInfoBase<BlockT, LoopT>::discoverAndMapSubloop(
+    LoopT *L, BlockT *Header, ArrayRef<BlockT *> Backedges,
+    const DominatorTreeBase<BlockT, false> &DomTree) {
   using InvBlockTraits = GraphTraits<Inverse<BlockT *>>;
 
-  unsigned NumBlocks = 0;
   unsigned NumSubloops = 0;
 
   // Perform a backward CFG traversal using a worklist.
@@ -469,15 +468,14 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
     BlockT *PredBB = ReverseCFGWorklist.back();
     ReverseCFGWorklist.pop_back();
 
-    LoopT *Subloop = LI->getLoopFor(PredBB);
+    LoopT *Subloop = getLoopFor(PredBB);
     if (!Subloop) {
       if (!DomTree.isReachableFromEntry(PredBB))
         continue;
 
       // This is an undiscovered block. Map it to the current loop.
-      LI->changeLoopFor(PredBB, L);
-      ++NumBlocks;
-      if (PredBB == L->getHeader())
+      changeLoopFor(PredBB, L);
+      if (PredBB == Header)
         continue;
       // Push all block predecessors on the worklist.
       ReverseCFGWorklist.insert(ReverseCFGWorklist.end(),
@@ -494,84 +492,29 @@ static void discoverAndMapSubloop(LoopT *L, ArrayRef<BlockT *> Backedges,
       // Discover a subloop of this loop.
       Subloop->setParentLoop(L);
       ++NumSubloops;
-      NumBlocks += Subloop->getBlocksCapacity();
-      PredBB = Subloop->getHeader();
+      PredBB = pendingHeader(Subloop);
       // Continue traversal along predecessors that are not loop-back edges from
       // within this subloop tree itself. Note that a predecessor may directly
       // reach another subloop that is not yet discovered to be a subloop of
       // this loop, which we must traverse.
       for (const auto Pred : inverse_children<BlockT *>(PredBB)) {
-        if (LI->getLoopFor(Pred) != Subloop)
+        if (getLoopFor(Pred) != Subloop)
           ReverseCFGWorklist.push_back(Pred);
       }
     }
   }
   L->reserveSubLoops(NumSubloops);
-  L->reserveBlocks(NumBlocks);
-}
-
-/// Populate all loop data in a stable order during a single forward DFS.
-template <class BlockT, class LoopT> class PopulateLoopsDFS {
-  using BlockTraits = GraphTraits<BlockT *>;
-  using SuccIterTy = typename BlockTraits::ChildIteratorType;
-
-  LoopInfoBase<BlockT, LoopT> *LI;
-
-public:
-  PopulateLoopsDFS(LoopInfoBase<BlockT, LoopT> *li) : LI(li) {}
-
-  void traverse(BlockT *EntryBlock);
-
-protected:
-  void insertIntoLoop(BlockT *Block);
-};
-
-/// Top-level driver for the forward DFS within the loop.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::traverse(BlockT *EntryBlock) {
-  for (BlockT *BB : post_order(EntryBlock))
-    insertIntoLoop(BB);
-}
-
-/// Add a single Block to its ancestor loops in PostOrder. If the block is a
-/// subloop header, add the subloop to its parent in PostOrder, then reverse the
-/// Block and Subloop vectors of the now complete subloop to achieve RPO.
-template <class BlockT, class LoopT>
-void PopulateLoopsDFS<BlockT, LoopT>::insertIntoLoop(BlockT *Block) {
-  LoopT *Subloop = LI->getLoopFor(Block);
-  if (Subloop && Block == Subloop->getHeader()) {
-    // We reach this point once per subloop after processing all the blocks in
-    // the subloop.
-    if (!Subloop->isOutermost())
-      Subloop->getParentLoop()->SubLoops.push_back(Subloop);
-    else
-      LI->addTopLevelLoop(Subloop);
-
-    // For convenience, Blocks and Subloops are inserted in postorder. Reverse
-    // the lists, except for the loop header, which is always at the beginning.
-    Subloop->reverseBlock(1);
-    std::reverse(Subloop->SubLoops.begin(), Subloop->SubLoops.end());
-
-    Subloop = Subloop->getParentLoop();
-  }
-  for (; Subloop; Subloop = Subloop->getParentLoop())
-    Subloop->addBlockEntry(Block);
 }
 
 /// Analyze LoopInfo discovers loops during a reverse preorder DominatorTree
 /// traversal interleaved with backward CFG traversals within each subloop
 /// (discoverAndMapSubloop). The backward traversal skips inner subloops, so
-/// this part of the algorithm is linear in the number of CFG edges. Subloop and
-/// Block vectors are then populated during a single forward CFG traversal
-/// (PopulateLoopDFS).
+/// this part of the algorithm is linear in the number of CFG edges.
 ///
-/// During the two CFG traversals each block is seen three times:
-/// 1) Discovered and mapped by a reverse CFG traversal.
-/// 2) Visited during a forward DFS CFG traversal.
-/// 3) Reverse-inserted in the loop in postorder following forward DFS.
-///
-/// The Block vectors are inclusive, so step 3 requires loop-depth number of
-/// insertions per block.
+/// Then build a loop-contiguous reverse postorder for in-loops blocks. Lists
+/// are header-first with each subloop's blocks contiguous, ordered by first
+/// appearance in RPO; SubLoops keep program order, TopLevelLoops reverse
+/// program order.
 template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   const DomTreeNodeBase<BlockT> *DomRoot = DomTree.getRootNode();
@@ -587,6 +530,7 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
   for (const DomTreeNodeBase<BlockT> *Node : DomTree.nodes())
     PreorderNodes[Node->getDFSNumIn()] = Node;
 
+  bool HasLoops = false;
   for (const DomTreeNodeBase<BlockT> *DomNode : llvm::reverse(PreorderNodes)) {
     BlockT *Header = DomNode->getBlock();
     SmallVector<BlockT *, 4> Backedges;
@@ -600,14 +544,64 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
     }
     // Perform a backward CFG traversal to discover and map blocks in this loop.
     if (!Backedges.empty()) {
-      LoopT *L = AllocateLoop(Header);
-      discoverAndMapSubloop(L, ArrayRef<BlockT *>(Backedges), this, DomTree);
+      HasLoops = true;
+      LoopT *L = allocateLoop(Header);
+      discoverAndMapSubloop(L, Header, Backedges, DomTree);
     }
   }
-  // Perform a single forward CFG traversal to populate block and subloop
-  // vectors for all loops.
-  PopulateLoopsDFS<BlockT, LoopT> DFS(this);
-  DFS.traverse(DomRoot->getBlock());
+  // Most functions have no loops; skip the layout construction.
+  if (!HasLoops)
+    return;
+
+  // Record each in-loop block with its innermost loop in forward CFG postorder,
+  // and build the loop list in PO.
+  SmallVector<std::pair<BlockT *, LoopT *>, 32> PO;
+  SmallVector<LoopT *, 4> LoopsPO;
+  PO.reserve(BBMap.size());
+  for (BlockT *BB : post_order(ParentPtr)) {
+    LoopT *L = lookupLoopFor(BB);
+    if (!L)
+      continue;
+    PO.emplace_back(BB, L);
+    ++L->BlockLen;
+    if (BB != pendingHeader(L))
+      continue;
+    LoopsPO.push_back(L);
+    if (LoopT *Parent = L->getParentLoop())
+      Parent->BlockLen += L->BlockLen;
+    else
+      TopLevelLoops.push_back(L);
+  }
+  // Headers are dominator-tree nodes, hence reachable and in the postorder.
+  assert(!LoopsPO.empty() && "discovered loops but found no header");
+
+  BlockLayout.reset(new BlockT *[PO.size()]);
+  BlockT **RootCursor = BlockLayout.get();
+  for (auto &[BB, L] : llvm::reverse(PO)) {
+    if (L->BlockCapacity == 0) {
+      // The first block of a L is its the header. Carve its slice from the
+      // parent (already visited)'s cursor.
+      if (LoopT *Parent = L->getParentLoop()) {
+        assert(Parent->BlockCapacity != 0 &&
+               "parent slice not carved before child");
+        L->BlockData = Parent->BlockData + Parent->BlockCapacity;
+        Parent->BlockCapacity += L->BlockLen;
+        Parent->SubLoops.push_back(L);
+      } else {
+        L->BlockData = RootCursor;
+        RootCursor += L->BlockLen;
+      }
+    }
+    // Each block lands once, at its innermost loop's cursor.
+    L->BlockData[L->BlockCapacity++] = BB;
+  }
+
+  // Mark every slice as borrowed from BlockLayout; a later mutation copies it
+  // into private storage (see materializeBlocks).
+  for (LoopT *L : LoopsPO) {
+    assert(L->BlockCapacity == L->BlockLen && "layout slice not fully used");
+    L->BlockCapacity = LoopT::BorrowedCapacity;
+  }
 }
 
 template <class BlockT, class LoopT>

diff  --git a/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll b/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
index 69ef9f3e530ad..aec401ece0b3a 100644
--- a/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
+++ b/llvm/test/Transforms/LoopVectorize/early_exit_with_outer_loop.ll
@@ -46,7 +46,7 @@ loop.inner.end:
 ; loops at depths 1 and 2, respectively.
 define void @early_exit_in_outer_loop2() {
 ; CHECK-LABEL: Loop info for function 'early_exit_in_outer_loop2':
-; CHECK: Loop at depth 1 containing: %loop.outer<header>,%loop.middle,%loop.inner.found,%loop.inner.end,%loop.middle.end,%loop.outer.latch<latch>,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit
+; CHECK: Loop at depth 1 containing: %loop.outer<header>,%loop.middle,%loop.inner.end,%loop.inner.found,%loop.middle.end,%loop.outer.latch<latch>,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit
 ; CHECK:    Loop at depth 2 containing: %loop.middle<header>,%loop.inner.end<latch><exiting>,%vector.ph,%vector.body<exiting>,%vector.body.interim,%middle.block
 ; CHECK:        Loop at depth 3 containing: %vector.body<header><exiting>,%vector.body.interim<latch><exiting>
 entry:
@@ -268,7 +268,7 @@ early.exit.leave:
 ; (one going to a middle loop, one going to the outer loop).
 define i32 @multi_early_exit_two_
diff erent_loops(i1 %c, ptr dereferenceable(1024) %src) nofree {
 ; CHECK-LABEL: Loop info for function 'multi_early_exit_two_
diff erent_loops':
-; CHECK-NEXT: Loop at depth 1 containing: %outer.header<header>,%middle.header,%early.exit.outer,%early.exit.middle,%middle.latch,%outer.latch<latch>,%middle.latch.loopexit,%outer.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check,%vector.early.exit.1,%vector.early.exit.0
+; CHECK-NEXT: Loop at depth 1 containing: %outer.header<header>,%middle.header,%early.exit.middle,%middle.latch,%early.exit.outer,%outer.latch<latch>,%middle.latch.loopexit,%outer.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check,%vector.early.exit.1,%vector.early.exit.0
 ; CHECK-NEXT:     Loop at depth 2 containing: %middle.header<header>,%early.exit.middle,%middle.latch<latch><exiting>,%middle.latch.loopexit,%vector.ph,%vector.body,%vector.body.interim,%middle.block,%vector.early.exit.check<exiting>,%vector.early.exit.0
 ; CHECK-NEXT:         Loop at depth 3 containing: %vector.body<header><exiting>,%vector.body.interim<latch><exiting>
 entry:

diff  --git a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
index f55ec1bedc0b8..925b919c5a397 100644
--- a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
+++ b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch-freeze.ll
@@ -1134,10 +1134,10 @@ define i32 @test13a(ptr %ptr, i1 %cond, ptr %a.ptr, ptr %b.ptr) {
 ; CHECK:       loop_b_inner_body.us:
 ; CHECK-NEXT:    [[V4_US:%.*]] = load i1, ptr [[PTR]], align 1
 ; CHECK-NEXT:    br i1 [[V4_US]], label [[LOOP_B_INNER_LATCH_US]], label [[LOOP_B_INNER_EXIT_US:%.*]]
-; CHECK:       loop_b_inner_exit.us:
-; CHECK-NEXT:    br label [[LOOP_LATCH_US:%.*]]
 ; CHECK:       loop_b_inner_latch.us:
 ; CHECK-NEXT:    br label [[LOOP_B_INNER_HEADER_US]]
+; CHECK:       loop_b_inner_exit.us:
+; CHECK-NEXT:    br label [[LOOP_LATCH_US:%.*]]
 ; CHECK:       loop_a.us:
 ; CHECK-NEXT:    [[V2_US:%.*]] = load i1, ptr [[PTR]], align 1
 ; CHECK-NEXT:    br i1 [[V2_US]], label [[LOOP_EXIT_SPLIT_US:%.*]], label [[LOOP_LATCH_US]]
@@ -1426,10 +1426,10 @@ define i32 @test29(i32 %arg) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARG_FR:%.*]] = freeze i32 [[ARG:%.*]]
 ; CHECK-NEXT:    switch i32 [[ARG_FR]], label [[ENTRY_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 0, label [[ENTRY_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 1, label [[ENTRY_SPLIT_US]]
-; CHECK-NEXT:    i32 2, label [[ENTRY_SPLIT_US1:%.*]]
-; CHECK-NEXT:    i32 3, label [[ENTRY_SPLIT]]
+; CHECK-NEXT:      i32 0, label [[ENTRY_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 1, label [[ENTRY_SPLIT_US]]
+; CHECK-NEXT:      i32 2, label [[ENTRY_SPLIT_US1:%.*]]
+; CHECK-NEXT:      i32 3, label [[ENTRY_SPLIT]]
 ; CHECK-NEXT:    ]
 ; CHECK:       entry.split.us:
 ; CHECK-NEXT:    br label [[HEADER_US:%.*]]
@@ -1587,10 +1587,10 @@ define i32 @test30(i32 %arg) {
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARG_FR:%.*]] = freeze i32 [[ARG:%.*]]
 ; CHECK-NEXT:    switch i32 [[ARG_FR]], label [[ENTRY_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 -1, label [[ENTRY_SPLIT]]
-; CHECK-NEXT:    i32 0, label [[ENTRY_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 1, label [[ENTRY_SPLIT_US1:%.*]]
-; CHECK-NEXT:    i32 2, label [[ENTRY_SPLIT_US1]]
+; CHECK-NEXT:      i32 -1, label [[ENTRY_SPLIT]]
+; CHECK-NEXT:      i32 0, label [[ENTRY_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 1, label [[ENTRY_SPLIT_US1:%.*]]
+; CHECK-NEXT:      i32 2, label [[ENTRY_SPLIT_US1]]
 ; CHECK-NEXT:    ]
 ; CHECK:       entry.split.us:
 ; CHECK-NEXT:    br label [[HEADER_US:%.*]]
@@ -2259,9 +2259,9 @@ define void @hoist_inner_loop_switch(ptr %ptr) {
 ; CHECK-NEXT:    [[V1:%.*]] = call i32 @cond.i32()
 ; CHECK-NEXT:    [[V1_FR:%.*]] = freeze i32 [[V1]]
 ; CHECK-NEXT:    switch i32 [[V1_FR]], label [[B_HEADER_SPLIT:%.*]] [
-; CHECK-NEXT:    i32 1, label [[B_HEADER_SPLIT_US:%.*]]
-; CHECK-NEXT:    i32 2, label [[B_HEADER_SPLIT_US]]
-; CHECK-NEXT:    i32 3, label [[B_HEADER_SPLIT_US]]
+; CHECK-NEXT:      i32 1, label [[B_HEADER_SPLIT_US:%.*]]
+; CHECK-NEXT:      i32 2, label [[B_HEADER_SPLIT_US]]
+; CHECK-NEXT:      i32 3, label [[B_HEADER_SPLIT_US]]
 ; CHECK-NEXT:    ]
 ; CHECK:       b.header.split.us:
 ; CHECK-NEXT:    br label [[C_HEADER_US:%.*]]

diff  --git a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
index 0cb0e0d1cca2b..8fddaa1839576 100644
--- a/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
+++ b/llvm/test/Transforms/SimpleLoopUnswitch/nontrivial-unswitch.ll
@@ -2116,12 +2116,12 @@ loop_latch:
 ; CHECK-NEXT:    %[[V:.*]] = load i1, ptr %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_b_inner_latch.us, label %loop_b_inner_exit.us
 ;
-; CHECK:       loop_b_inner_exit.us:
-; CHECK-NEXT:    br label %loop_latch.us
-;
 ; CHECK:       loop_b_inner_latch.us:
 ; CHECK-NEXT:    br label %loop_b_inner_header.us
 ;
+; CHECK:       loop_b_inner_exit.us:
+; CHECK-NEXT:    br label %loop_latch.us
+;
 ; CHECK:       loop_a.us:
 ; CHECK-NEXT:    %[[V:.*]] = load i1, ptr %ptr
 ; CHECK-NEXT:    br i1 %[[V]], label %loop_exit.split.us, label %loop_latch.us

diff  --git a/mlir/include/mlir/Analysis/CFGLoopInfo.h b/mlir/include/mlir/Analysis/CFGLoopInfo.h
index 15ac0d71cb94d..7060a9df7de2d 100644
--- a/mlir/include/mlir/Analysis/CFGLoopInfo.h
+++ b/mlir/include/mlir/Analysis/CFGLoopInfo.h
@@ -36,7 +36,7 @@ namespace mlir {
 /// class provides accessors to the loop analysis.
 class CFGLoop : public llvm::LoopBase<mlir::Block, mlir::CFGLoop> {
 private:
-  explicit CFGLoop(mlir::Block *block);
+  CFGLoop() = default;
 
   friend class llvm::LoopBase<mlir::Block, CFGLoop>;
   friend class llvm::LoopInfoBase<mlir::Block, CFGLoop>;

diff  --git a/mlir/lib/Analysis/CFGLoopInfo.cpp b/mlir/lib/Analysis/CFGLoopInfo.cpp
index 9fe220344dcea..e03b664f9c72e 100644
--- a/mlir/lib/Analysis/CFGLoopInfo.cpp
+++ b/mlir/lib/Analysis/CFGLoopInfo.cpp
@@ -16,9 +16,6 @@ template class llvm::LoopInfoBase<mlir::Block, mlir::CFGLoop>;
 
 using namespace mlir;
 
-CFGLoop::CFGLoop(mlir::Block *block)
-    : llvm::LoopBase<mlir::Block, CFGLoop>(block) {}
-
 CFGLoopInfo::CFGLoopInfo(
     const llvm::DominatorTreeBase<mlir::Block, false> &domTree) {
   analyze(domTree);


        

From llvmlistbot at llvm.org  Fri Jul 24 18:57:10 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Fri, 24 Jul 2026 18:57:10 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6417f6.666420a4.3252ee.114c@mx.google.com>


https://github.com/MaskRay closed https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 19:45:53 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 24 Jul 2026 19:45:53 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][mem2reg] Promote whole-buffer memref
 to a vector SSA value (PR #211880)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211880@github.com>
Message-ID: <6a642361.f3151ff8.2f3401.7ae7@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/211880

>From 7b298c473292e8ea53c2b398df384713b908bafb Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 16:54:30 +0000
Subject: [PATCH 1/4] [mlir][mem2reg] Promote whole-buffer memref to vector via
 transfer ops

Mem2Reg previously only promoted memref allocas holding a single element
(to a scalar SSA value). The pass machinery itself is type-agnostic: a
MemorySlot carries an arbitrary elemType, the reaching definition is
threaded at that type, and scf.for already threads any slot type as an
iter_arg/result. The scalar-only behavior lived entirely in
memref.alloca's getPromotableSlots and the load/store type checks.

This teaches Mem2Reg to promote a statically-shaped multi-element memref
whose only accesses are whole-buffer vector.transfer_read/transfer_write
into a single vector SSA value:

- memref::AllocaOp::getPromotableSlots now additionally offers a
  VectorType slot (shape+element type of the memref) for static
  multi-element memrefs with a valid vector element type. This is purely
  additive and self-guarding: any non-whole-buffer access fails its
  canUsesBeRemoved check and promotion of the slot safely aborts.

- vector.transfer_read/transfer_write implement PromotableMemOpInterface
  via an out-of-line ExternalModel (matching how the Vector dialect
  attaches its other interfaces). A transfer is treated as a whole-buffer
  load/store only when it targets the slot pointer with a matching vector
  type, all-zero constant indices, an identity permutation map, all
  in-bounds dims, and no mask -- which also rejects scalable vectors.

No changes to the Mem2Reg pass or SCF are required.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Transforms/MemorySlotOpInterfaceImpl.h    |  21 +++
 .../Dialect/MemRef/IR/MemRefMemorySlot.cpp    |  19 +-
 .../Dialect/Vector/Transforms/CMakeLists.txt  |   2 +
 .../Transforms/MemorySlotOpInterfaceImpl.cpp  | 165 +++++++++++++++++
 mlir/lib/RegisterAllDialects.cpp              |   2 +
 mlir/test/Dialect/Vector/mem2reg.mlir         | 172 ++++++++++++++++++
 6 files changed, 377 insertions(+), 4 deletions(-)
 create mode 100644 mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h
 create mode 100644 mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
 create mode 100644 mlir/test/Dialect/Vector/mem2reg.mlir

diff --git a/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h b/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h
new file mode 100644
index 0000000000000..4c2e386728419
--- /dev/null
+++ b/mlir/include/mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h
@@ -0,0 +1,21 @@
+//===- MemorySlotOpInterfaceImpl.h - Mem2Reg for vector ops -----*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
+#define MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
+
+namespace mlir {
+
+class DialectRegistry;
+
+namespace vector {
+void registerMemorySlotOpInterfaceExternalModels(DialectRegistry &registry);
+} // namespace vector
+} // namespace mlir
+
+#endif // MLIR_DIALECT_VECTOR_TRANSFORMS_MEMORYSLOTOPINTERFACEIMPL_H
diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 6748e2cf71804..999abc7354106 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -66,11 +66,22 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   MemRefType type = getType();
   if (!type.hasStaticShape())
     return {};
-  // Make sure the memref contains only a single element.
-  if (type.getNumElements() != 1)
-    return {};
 
-  return {MemorySlot{getResult(), type.getElementType()}};
+  // A single-element memref is promoted to a scalar SSA value.
+  if (type.getNumElements() == 1)
+    return {MemorySlot{getResult(), type.getElementType()}};
+
+  // A multi-element memref can be promoted to a single vector SSA value when it
+  // is only ever accessed as a whole buffer (e.g. through whole-buffer
+  // `vector.transfer_read`/`vector.transfer_write`). Offering the slot is safe
+  // and purely additive: any access that is not a whole-buffer transfer will
+  // fail its `canUsesBeRemoved` check and abort promotion of this slot.
+  if (VectorType::isValidElementType(type.getElementType()))
+    return {MemorySlot{
+        getResult(),
+        VectorType::get(type.getShape(), type.getElementType())}};
+
+  return {};
 }
 
 Value memref::AllocaOp::getDefaultValue(const MemorySlot &slot,
diff --git a/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
index 112a1db6fe93b..dfe873f1a1b8d 100644
--- a/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Vector/Transforms/CMakeLists.txt
@@ -15,6 +15,7 @@ add_mlir_dialect_library(MLIRVectorTransforms
   LowerVectorToFromElementsToShuffleTree.cpp
   LowerVectorTransfer.cpp
   LowerVectorTranspose.cpp
+  MemorySlotOpInterfaceImpl.cpp
   SubsetOpInterfaceImpl.cpp
   VectorDistribute.cpp
   VectorDropLeadUnitDim.cpp
@@ -44,6 +45,7 @@ add_mlir_dialect_library(MLIRVectorTransforms
   MLIRGPUUtils
   MLIRIR
   MLIRLinalgDialect
+  MLIRMemorySlotInterfaces
   MLIRMemRefDialect
   MLIRMemRefTransforms
   MLIRMemRefUtils
diff --git a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
new file mode 100644
index 0000000000000..e911c38542e7a
--- /dev/null
+++ b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -0,0 +1,165 @@
+//===- MemorySlotOpInterfaceImpl.cpp - Mem2Reg for vector ops -------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements Mem2Reg-related interfaces for Vector dialect
+// operations. It allows a memref that is only ever accessed as a whole buffer
+// through `vector.transfer_read`/`vector.transfer_write` to be promoted into a
+// single vector SSA value.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h"
+
+#include "mlir/Dialect/Utils/StaticValueUtils.h"
+#include "mlir/Dialect/Vector/IR/VectorOps.h"
+#include "mlir/IR/BuiltinTypes.h"
+#include "mlir/Interfaces/MemorySlotInterfaces.h"
+
+using namespace mlir;
+using namespace mlir::vector;
+
+//===----------------------------------------------------------------------===//
+//  Utilities
+//===----------------------------------------------------------------------===//
+
+/// Returns whether the transfer operation `xferOp` accesses exactly the whole
+/// contents of the memory slot `slot`, so that it can be treated as a plain
+/// whole-buffer load or store during Mem2Reg. `blockingUses` are the uses of
+/// the slot pointer that this operation must stop using for promotion.
+template <typename TransferOpTy>
+static bool
+isWholeBufferTransfer(TransferOpTy xferOp, const MemorySlot &slot,
+                      const SmallPtrSetImpl<OpOperand *> &blockingUses) {
+  // The only blocking use must be the slot pointer itself.
+  if (blockingUses.size() != 1)
+    return false;
+  Value blockingUse = (*blockingUses.begin())->get();
+  if (blockingUse != slot.ptr || xferOp.getBase() != slot.ptr)
+    return false;
+
+  // Only the memref form can access a memref slot. This is already implied by
+  // `getBase() == slot.ptr` above (slot pointers are always memrefs), but guard
+  // defensively against the tensor form.
+  if (!isa<MemRefType>(xferOp.getBase().getType()))
+    return false;
+
+  // The transferred vector must match the slot type exactly. This pins the
+  // rank, per-dimension extent and element type, and rejects scalable vectors
+  // (which never equal the fixed slot type).
+  if (xferOp.getVectorType() != slot.elemType)
+    return false;
+
+  // All indices must be constant zero so the access starts at the buffer
+  // origin in every dimension.
+  for (Value index : xferOp.getIndices()) {
+    std::optional<int64_t> constIndex = getConstantIntValue(index);
+    if (!constIndex || *constIndex != 0)
+      return false;
+  }
+
+  // The permutation map must be the identity: no broadcast, no transpose.
+  if (!xferOp.getPermutationMap().isIdentity())
+    return false;
+
+  // Every dimension must be in bounds so no element lies outside the buffer and
+  // no padding takes effect.
+  if (xferOp.hasOutOfBoundsDim())
+    return false;
+
+  // A mask could disable some elements, making the access partial.
+  if (xferOp.getMask())
+    return false;
+
+  return true;
+}
+
+//===----------------------------------------------------------------------===//
+//  Interface models
+//===----------------------------------------------------------------------===//
+
+namespace {
+
+struct TransferReadOpMemOpModel
+    : public PromotableMemOpInterface::ExternalModel<TransferReadOpMemOpModel,
+                                                     vector::TransferReadOp> {
+  bool loadsFrom(Operation *op, const MemorySlot &slot) const {
+    return cast<vector::TransferReadOp>(op).getBase() == slot.ptr;
+  }
+
+  bool storesTo(Operation *op, const MemorySlot &slot) const { return false; }
+
+  Value getStored(Operation *op, const MemorySlot &slot, OpBuilder &builder,
+                  Value reachingDef, const DataLayout &dataLayout) const {
+    llvm_unreachable("getStored should not be called on TransferReadOp");
+  }
+
+  bool canUsesBeRemoved(Operation *op, const MemorySlot &slot,
+                        const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                        SmallVectorImpl<OpOperand *> &newBlockingUses,
+                        const DataLayout &dataLayout) const {
+    return isWholeBufferTransfer(cast<vector::TransferReadOp>(op), slot,
+                                 blockingUses);
+  }
+
+  DeletionKind removeBlockingUses(
+      Operation *op, const MemorySlot &slot,
+      const SmallPtrSetImpl<OpOperand *> &blockingUses, OpBuilder &builder,
+      Value reachingDefinition, const DataLayout &dataLayout) const {
+    // `canUsesBeRemoved` guaranteed a whole-buffer read of the slot.
+    cast<vector::TransferReadOp>(op).getVector().replaceAllUsesWith(
+        reachingDefinition);
+    return DeletionKind::Delete;
+  }
+};
+
+struct TransferWriteOpMemOpModel
+    : public PromotableMemOpInterface::ExternalModel<TransferWriteOpMemOpModel,
+                                                     vector::TransferWriteOp> {
+  bool loadsFrom(Operation *op, const MemorySlot &slot) const { return false; }
+
+  bool storesTo(Operation *op, const MemorySlot &slot) const {
+    return cast<vector::TransferWriteOp>(op).getBase() == slot.ptr;
+  }
+
+  Value getStored(Operation *op, const MemorySlot &slot, OpBuilder &builder,
+                  Value reachingDef, const DataLayout &dataLayout) const {
+    return cast<vector::TransferWriteOp>(op).getValueToStore();
+  }
+
+  bool canUsesBeRemoved(Operation *op, const MemorySlot &slot,
+                        const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                        SmallVectorImpl<OpOperand *> &newBlockingUses,
+                        const DataLayout &dataLayout) const {
+    auto xferOp = cast<vector::TransferWriteOp>(op);
+    // The stored value must not be the slot pointer itself.
+    if (xferOp.getValueToStore() == slot.ptr)
+      return false;
+    return isWholeBufferTransfer(xferOp, slot, blockingUses);
+  }
+
+  DeletionKind removeBlockingUses(
+      Operation *op, const MemorySlot &slot,
+      const SmallPtrSetImpl<OpOperand *> &blockingUses, OpBuilder &builder,
+      Value reachingDefinition, const DataLayout &dataLayout) const {
+    return DeletionKind::Delete;
+  }
+};
+
+} // namespace
+
+//===----------------------------------------------------------------------===//
+//  Register external models
+//===----------------------------------------------------------------------===//
+
+void mlir::vector::registerMemorySlotOpInterfaceExternalModels(
+    DialectRegistry &registry) {
+  registry.addExtension(+[](MLIRContext *ctx, vector::VectorDialect *dialect) {
+    TransferReadOp::attachInterface<TransferReadOpMemOpModel>(*ctx);
+    TransferWriteOp::attachInterface<TransferWriteOpMemOpModel>(*ctx);
+  });
+}
diff --git a/mlir/lib/RegisterAllDialects.cpp b/mlir/lib/RegisterAllDialects.cpp
index 974b5f533860a..62304ae88b43c 100644
--- a/mlir/lib/RegisterAllDialects.cpp
+++ b/mlir/lib/RegisterAllDialects.cpp
@@ -97,6 +97,7 @@
 #include "mlir/Dialect/Vector/IR/VectorOps.h"
 #include "mlir/Dialect/Vector/Transforms/BufferizableOpInterfaceImpl.h"
 #include "mlir/Dialect/Vector/Transforms/IndexedAccessOpInterfaceImpl.h"
+#include "mlir/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.h"
 #include "mlir/Dialect/Vector/Transforms/SubsetOpInterfaceImpl.h"
 #include "mlir/Dialect/WasmSSA/IR/WasmSSA.h"
 #include "mlir/Dialect/X86/X86Dialect.h"
@@ -200,6 +201,7 @@ void mlir::registerAllDialects(DialectRegistry &registry) {
   tosa::registerShardingInterfaceExternalModels(registry);
   vector::registerBufferizableOpInterfaceExternalModels(registry);
   vector::registerIndexedAccessOpInterfaceExternalModels(registry);
+  vector::registerMemorySlotOpInterfaceExternalModels(registry);
   vector::registerSubsetOpInterfaceExternalModels(registry);
   vector::registerValueBoundsOpInterfaceExternalModels(registry);
   NVVM::registerNVVMTargetInterfaceExternalModels(registry);
diff --git a/mlir/test/Dialect/Vector/mem2reg.mlir b/mlir/test/Dialect/Vector/mem2reg.mlir
new file mode 100644
index 0000000000000..ec6e19d0a5c19
--- /dev/null
+++ b/mlir/test/Dialect/Vector/mem2reg.mlir
@@ -0,0 +1,172 @@
+// RUN: mlir-opt %s --pass-pipeline='builtin.module(func.func(mem2reg))' --split-input-file | FileCheck %s
+
+// A memref that is only ever accessed as a whole buffer through
+// vector.transfer_read / vector.transfer_write is promoted to a single vector
+// SSA value.
+
+// CHECK-LABEL: func.func @whole_buffer_write_read
+//   CHECK-SAME:   (%[[PAD:.*]]: f32)
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer_write
+//    CHECK-NOT:   vector.transfer_read
+//        CHECK:   %[[CST:.*]] = arith.constant dense<1.000000e+00> : vector<4xf32>
+//        CHECK:   return %[[CST]] : vector<4xf32>
+func.func @whole_buffer_write_read(%pad: f32) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A whole-buffer slot carried across scf.for is threaded as an iter_arg/result.
+
+// CHECK-LABEL: func.func @whole_buffer_in_loop
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer_write
+//    CHECK-NOT:   vector.transfer_read
+//        CHECK:   %[[RES:.*]] = scf.for {{.*}} iter_args(%[[IT:.*]] = %{{.*}}) -> (vector<4xf32>)
+//        CHECK:     %[[NEXT:.*]] = arith.addf %[[IT]], %[[IT]] : vector<4xf32>
+//        CHECK:     scf.yield %[[NEXT]] : vector<4xf32>
+//        CHECK:   return %[[RES]] : vector<4xf32>
+func.func @whole_buffer_in_loop(%pad: f32, %lb: index, %ub: index, %step: index) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  scf.for %i = %lb to %ub step %step {
+    %v = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+    %n = arith.addf %v, %v : vector<4xf32>
+    vector.transfer_write %n, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  }
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A multi-dimensional whole-buffer memref is promoted to a matching vector.
+
+// CHECK-LABEL: func.func @whole_buffer_2d
+//    CHECK-NOT:   memref.alloca
+//    CHECK-NOT:   vector.transfer
+//        CHECK:   return %{{.*}} : vector<2x4xf32>
+func.func @whole_buffer_2d(%pad: f32) -> vector<2x4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<2x4xf32>
+  %a = memref.alloca() : memref<2x4xf32>
+  vector.transfer_write %cst, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
+  %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true]} : memref<2x4xf32>, vector<2x4xf32>
+  return %r : vector<2x4xf32>
+}
+
+// -----
+
+// Non-zero access offset: the transfer does not cover the whole buffer, so the
+// slot must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_nonzero_index
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_nonzero_index(%pad: f32) -> vector<4xf32> {
+  %c1 = arith.constant 1 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<8xf32>
+  vector.transfer_write %cst, %a[%c1] {in_bounds = [true]} : vector<4xf32>, memref<8xf32>
+  %r = vector.transfer_read %a[%c1], %pad {in_bounds = [true]} : memref<8xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A masked transfer only touches part of the buffer: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_masked
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_masked(%pad: f32, %m: vector<4xi1>) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0], %m {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  return %r : vector<4xf32>
+}
+
+// -----
+
+// A partial (out-of-bounds) transfer must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_out_of_bounds
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_out_of_bounds(%pad: f32) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<8xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] : vector<8xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad : memref<4xf32>, vector<8xf32>
+  return %r : vector<8xf32>
+}
+
+// -----
+
+// A non-identity (transposing) permutation map is not a whole-buffer identity
+// access: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_transpose_map
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_transpose_map(%pad: f32) -> vector<4x2xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<2x4xf32>
+  %a = memref.alloca() : memref<2x4xf32>
+  vector.transfer_write %cst, %a[%c0, %c0] {in_bounds = [true, true]} : vector<2x4xf32>, memref<2x4xf32>
+  %r = vector.transfer_read %a[%c0, %c0], %pad {in_bounds = [true, true], permutation_map = affine_map<(d0, d1) -> (d1, d0)>} : memref<2x4xf32>, vector<4x2xf32>
+  return %r : vector<4x2xf32>
+}
+
+// -----
+
+// An alloca also accessed through a scalar memref.load cannot be promoted to a
+// vector: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_mixed_scalar_access
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+//        CHECK:   memref.load
+func.func @negative_mixed_scalar_access(%pad: f32) -> (vector<4xf32>, f32) {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<4xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<4xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<4xf32>
+  %s = memref.load %a[%c0] : memref<4xf32>
+  return %r, %s : vector<4xf32>, f32
+}
+
+// -----
+
+// A scalable vector never equals the fixed-size slot type: must NOT be
+// promoted.
+
+// CHECK-LABEL: func.func @negative_scalable
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   vector.transfer_read
+func.func @negative_scalable(%pad: f32) -> vector<[4]xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<[4]xf32>
+  %a = memref.alloca() : memref<4xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<[4]xf32>, memref<4xf32>
+  %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<[4]xf32>
+  return %r : vector<[4]xf32>
+}

>From 24c11489338115c200a48d648a8787901c4bb604 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 17:40:45 +0000
Subject: [PATCH 2/4] [mlir] Simplify comments and drop dead self-store guard
 in vector mem2reg

Trim verbose comments in the whole-buffer transfer promotion logic and
remove the unreachable self-store check in TransferWriteOp's
canUsesBeRemoved: a transfer_write's stored value is always a vector and
the slot pointer always a memref, so they can never be the same Value.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Dialect/MemRef/IR/MemRefMemorySlot.cpp    |  4 +--
 .../Transforms/MemorySlotOpInterfaceImpl.cpp  | 36 +++++++------------
 2 files changed, 14 insertions(+), 26 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 999abc7354106..59798a65f72e8 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -73,9 +73,7 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
 
   // A multi-element memref can be promoted to a single vector SSA value when it
   // is only ever accessed as a whole buffer (e.g. through whole-buffer
-  // `vector.transfer_read`/`vector.transfer_write`). Offering the slot is safe
-  // and purely additive: any access that is not a whole-buffer transfer will
-  // fail its `canUsesBeRemoved` check and abort promotion of this slot.
+  // `vector.transfer_read`/`vector.transfer_write`).
   if (VectorType::isValidElementType(type.getElementType()))
     return {MemorySlot{
         getResult(),
diff --git a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
index e911c38542e7a..1789c59da0720 100644
--- a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -27,51 +27,43 @@ using namespace mlir::vector;
 //  Utilities
 //===----------------------------------------------------------------------===//
 
-/// Returns whether the transfer operation `xferOp` accesses exactly the whole
-/// contents of the memory slot `slot`, so that it can be treated as a plain
-/// whole-buffer load or store during Mem2Reg. `blockingUses` are the uses of
-/// the slot pointer that this operation must stop using for promotion.
+/// Returns whether `xferOp` accesses exactly the whole contents of `slot`, so
+/// it can act as a plain whole-buffer load/store during Mem2Reg.
 template <typename TransferOpTy>
 static bool
 isWholeBufferTransfer(TransferOpTy xferOp, const MemorySlot &slot,
                       const SmallPtrSetImpl<OpOperand *> &blockingUses) {
-  // The only blocking use must be the slot pointer itself.
+  // The sole blocking use must be the slot pointer as the transfer's base.
   if (blockingUses.size() != 1)
     return false;
   Value blockingUse = (*blockingUses.begin())->get();
   if (blockingUse != slot.ptr || xferOp.getBase() != slot.ptr)
     return false;
 
-  // Only the memref form can access a memref slot. This is already implied by
-  // `getBase() == slot.ptr` above (slot pointers are always memrefs), but guard
-  // defensively against the tensor form.
+  // Reject the tensor form (already implied, since slot pointers are memrefs).
   if (!isa<MemRefType>(xferOp.getBase().getType()))
     return false;
 
-  // The transferred vector must match the slot type exactly. This pins the
-  // rank, per-dimension extent and element type, and rejects scalable vectors
-  // (which never equal the fixed slot type).
+  // Exact type match pins rank/extents/element type and rejects scalable vectors.
   if (xferOp.getVectorType() != slot.elemType)
     return false;
 
-  // All indices must be constant zero so the access starts at the buffer
-  // origin in every dimension.
+  // Access must start at the buffer origin in every dimension.
   for (Value index : xferOp.getIndices()) {
     std::optional<int64_t> constIndex = getConstantIntValue(index);
     if (!constIndex || *constIndex != 0)
       return false;
   }
 
-  // The permutation map must be the identity: no broadcast, no transpose.
+  // Identity map: no broadcast or transpose.
   if (!xferOp.getPermutationMap().isIdentity())
     return false;
 
-  // Every dimension must be in bounds so no element lies outside the buffer and
-  // no padding takes effect.
+  // All dimensions in bounds: no out-of-buffer element, no padding.
   if (xferOp.hasOutOfBoundsDim())
     return false;
 
-  // A mask could disable some elements, making the access partial.
+  // A mask would make the access partial.
   if (xferOp.getMask())
     return false;
 
@@ -110,7 +102,7 @@ struct TransferReadOpMemOpModel
       Operation *op, const MemorySlot &slot,
       const SmallPtrSetImpl<OpOperand *> &blockingUses, OpBuilder &builder,
       Value reachingDefinition, const DataLayout &dataLayout) const {
-    // `canUsesBeRemoved` guaranteed a whole-buffer read of the slot.
+    // Whole-buffer read: replace the loaded vector with the reaching definition.
     cast<vector::TransferReadOp>(op).getVector().replaceAllUsesWith(
         reachingDefinition);
     return DeletionKind::Delete;
@@ -135,11 +127,9 @@ struct TransferWriteOpMemOpModel
                         const SmallPtrSetImpl<OpOperand *> &blockingUses,
                         SmallVectorImpl<OpOperand *> &newBlockingUses,
                         const DataLayout &dataLayout) const {
-    auto xferOp = cast<vector::TransferWriteOp>(op);
-    // The stored value must not be the slot pointer itself.
-    if (xferOp.getValueToStore() == slot.ptr)
-      return false;
-    return isWholeBufferTransfer(xferOp, slot, blockingUses);
+    // No self-store guard needed: a vector value can never equal a memref slot.
+    return isWholeBufferTransfer(cast<vector::TransferWriteOp>(op), slot,
+                                 blockingUses);
   }
 
   DeletionKind removeBlockingUses(

>From 1075f1a57a3eff4effb6071b00574a1f9980efc4 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 17:44:13 +0000
Subject: [PATCH 3/4] [mlir] clang-format vector mem2reg files

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../Dialect/MemRef/IR/MemRefMemorySlot.cpp    |  3 +--
 .../Transforms/MemorySlotOpInterfaceImpl.cpp  | 24 +++++++++++--------
 2 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
index 59798a65f72e8..17b1417550524 100644
--- a/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
+++ b/mlir/lib/Dialect/MemRef/IR/MemRefMemorySlot.cpp
@@ -76,8 +76,7 @@ SmallVector<MemorySlot> memref::AllocaOp::getPromotableSlots() {
   // `vector.transfer_read`/`vector.transfer_write`).
   if (VectorType::isValidElementType(type.getElementType()))
     return {MemorySlot{
-        getResult(),
-        VectorType::get(type.getShape(), type.getElementType())}};
+        getResult(), VectorType::get(type.getShape(), type.getElementType())}};
 
   return {};
 }
diff --git a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
index 1789c59da0720..352983ba153fa 100644
--- a/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
+++ b/mlir/lib/Dialect/Vector/Transforms/MemorySlotOpInterfaceImpl.cpp
@@ -44,7 +44,8 @@ isWholeBufferTransfer(TransferOpTy xferOp, const MemorySlot &slot,
   if (!isa<MemRefType>(xferOp.getBase().getType()))
     return false;
 
-  // Exact type match pins rank/extents/element type and rejects scalable vectors.
+  // Exact type match pins rank/extents/element type and rejects scalable
+  // vectors.
   if (xferOp.getVectorType() != slot.elemType)
     return false;
 
@@ -98,11 +99,13 @@ struct TransferReadOpMemOpModel
                                  blockingUses);
   }
 
-  DeletionKind removeBlockingUses(
-      Operation *op, const MemorySlot &slot,
-      const SmallPtrSetImpl<OpOperand *> &blockingUses, OpBuilder &builder,
-      Value reachingDefinition, const DataLayout &dataLayout) const {
-    // Whole-buffer read: replace the loaded vector with the reaching definition.
+  DeletionKind
+  removeBlockingUses(Operation *op, const MemorySlot &slot,
+                     const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                     OpBuilder &builder, Value reachingDefinition,
+                     const DataLayout &dataLayout) const {
+    // Whole-buffer read: replace the loaded vector with the reaching
+    // definition.
     cast<vector::TransferReadOp>(op).getVector().replaceAllUsesWith(
         reachingDefinition);
     return DeletionKind::Delete;
@@ -132,10 +135,11 @@ struct TransferWriteOpMemOpModel
                                  blockingUses);
   }
 
-  DeletionKind removeBlockingUses(
-      Operation *op, const MemorySlot &slot,
-      const SmallPtrSetImpl<OpOperand *> &blockingUses, OpBuilder &builder,
-      Value reachingDefinition, const DataLayout &dataLayout) const {
+  DeletionKind
+  removeBlockingUses(Operation *op, const MemorySlot &slot,
+                     const SmallPtrSetImpl<OpOperand *> &blockingUses,
+                     OpBuilder &builder, Value reachingDefinition,
+                     const DataLayout &dataLayout) const {
     return DeletionKind::Delete;
   }
 };

>From 9699ca7b9767d5a28cb79aa6aa4439bca9a86895 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 25 Jul 2026 02:45:38 +0000
Subject: [PATCH 4/4] add test

---
 mlir/test/Dialect/Vector/mem2reg.mlir | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/mlir/test/Dialect/Vector/mem2reg.mlir b/mlir/test/Dialect/Vector/mem2reg.mlir
index ec6e19d0a5c19..7b2f0bd75ec31 100644
--- a/mlir/test/Dialect/Vector/mem2reg.mlir
+++ b/mlir/test/Dialect/Vector/mem2reg.mlir
@@ -170,3 +170,23 @@ func.func @negative_scalable(%pad: f32) -> vector<[4]xf32> {
   %r = vector.transfer_read %a[%c0], %pad {in_bounds = [true]} : memref<4xf32>, vector<[4]xf32>
   return %r : vector<[4]xf32>
 }
+
+// -----
+
+// The buffer has a use (subview) that mem2reg does not recognize as a
+// removable load or store, so the slot is left untouched: must NOT be promoted.
+
+// CHECK-LABEL: func.func @negative_subview
+//        CHECK:   memref.alloca
+//        CHECK:   vector.transfer_write
+//        CHECK:   memref.subview
+//        CHECK:   vector.transfer_read
+func.func @negative_subview(%pad: f32) -> vector<4xf32> {
+  %c0 = arith.constant 0 : index
+  %cst = arith.constant dense<1.0> : vector<8xf32>
+  %a = memref.alloca() : memref<8xf32>
+  vector.transfer_write %cst, %a[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32>
+  %sv = memref.subview %a[0] [4] [1] : memref<8xf32> to memref<4xf32, strided<[1]>>
+  %r = vector.transfer_read %sv[%c0], %pad {in_bounds = [true]} : memref<4xf32, strided<[1]>>, vector<4xf32>
+  return %r : vector<4xf32>
+}


From llvmlistbot at llvm.org  Fri Jul 24 19:51:03 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 24 Jul 2026 19:51:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Support batched matmul in
 VectorToXeGPU ContractionLowering (PR #211947)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211947@github.com>
Message-ID: <6a642497.b1416da1.20b4cf.e5b1@mx.google.com>


https://github.com/Jianhui-Li updated https://github.com/llvm/llvm-project/pull/211947

>From d93be98f0671d505e7c28a55aefcc1b82ae5b2a9 Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Fri, 24 Jul 2026 22:20:54 +0000
Subject: [PATCH 1/2] [mlir][xegpu] Support batched matmul in VectorToXeGPU
 ContractionLowering

Generalize the row-major matmul check in ContractionLowering so that
(batched) N-D vector.contract ops lower to xegpu.dpas, where any leading
dimensions are batch dimensions shared by lhs, rhs, and acc and the
innermost two dimensions are the core matmul dims (M, K), (K, N), (M, N).

Contractions that do not match this pattern (e.g. an N-D contract whose
leading dim is not a shared batch dim) get a soft match failure with a
descriptive reason instead of silently hitting the old "Expects lhs and
rhs 2D vectors" message.

Co-Authored-By: Claude Opus 4.8 <noreply at anthropic.com>
---
 .../VectorToXeGPU/VectorToXeGPU.cpp           | 81 ++++++++++++++++---
 .../VectorToXeGPU/contract-to-xegpu.mlir      | 48 +++++++++++
 2 files changed, 118 insertions(+), 11 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 727c98aef6619..850fd80793736 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -924,6 +924,63 @@ struct StoreLowering : public OpRewritePattern<vector::StoreOp> {
   }
 };
 
+// If `indexingMaps` describe a (batched) row-major matmul
+//   lhs[b..., m, k], rhs[b..., k, n], acc[b..., m, n]
+// return the number of leading batch dims (0 for a plain 2D matmul);
+// otherwise return std::nullopt.
+static std::optional<int64_t>
+getRowMajorMatmulBatchRank(ArrayAttr indexingMaps) {
+  if (indexingMaps.size() != 3)
+    return std::nullopt;
+
+  AffineMap mapA = cast<AffineMapAttr>(indexingMaps[0]).getValue();
+  AffineMap mapB = cast<AffineMapAttr>(indexingMaps[1]).getValue();
+  AffineMap mapC = cast<AffineMapAttr>(indexingMaps[2]).getValue();
+
+  // The result map exposes the batch dims followed by the core (m, n) dims.
+  if (mapC.getNumResults() < 2)
+    return std::nullopt;
+  int64_t batchRank = mapC.getNumResults() - 2;
+
+  // A single `k` reduction gives batchRank + 3 iteration dims; each operand
+  // map exposes batchRank + 2 dims (batch dims + 2 core dims).
+  unsigned numDims = static_cast<unsigned>(batchRank) + 3;
+  unsigned numOperandResults = static_cast<unsigned>(batchRank) + 2;
+  if (mapA.getNumInputs() != numDims || mapB.getNumInputs() != numDims ||
+      mapC.getNumInputs() != numDims)
+    return std::nullopt;
+  if (mapA.getNumResults() != numOperandResults ||
+      mapB.getNumResults() != numOperandResults)
+    return std::nullopt;
+
+  // Reconstruct the canonical maps from the batch/m/n dims of the result and
+  // the k dim of lhs, then compare against the actual maps.
+  MLIRContext *context = indexingMaps.getContext();
+  ArrayRef<AffineExpr> batchDims = mapC.getResults().take_front(batchRank);
+  AffineExpr m = mapC.getResult(batchRank);
+  AffineExpr n = mapC.getResult(batchRank + 1);
+  AffineExpr k = mapA.getResult(batchRank + 1);
+
+  SmallVector<AffineExpr> aDims = llvm::to_vector(batchDims);
+  aDims.push_back(m);
+  aDims.push_back(k);
+  SmallVector<AffineExpr> bDims = llvm::to_vector(batchDims);
+  bDims.push_back(k);
+  bDims.push_back(n);
+  SmallVector<AffineExpr> cDims = llvm::to_vector(batchDims);
+  cDims.push_back(m);
+  cDims.push_back(n);
+
+  auto expected = ArrayAttr::get(
+      context,
+      {AffineMapAttr::get(AffineMap::get(numDims, 0, aDims, context)),
+       AffineMapAttr::get(AffineMap::get(numDims, 0, bDims, context)),
+       AffineMapAttr::get(AffineMap::get(numDims, 0, cDims, context))});
+  if (indexingMaps != expected)
+    return std::nullopt;
+  return batchRank;
+}
+
 struct ContractionLowering : public OpRewritePattern<vector::ContractionOp> {
   using Base::Base;
 
@@ -935,21 +992,23 @@ struct ContractionLowering : public OpRewritePattern<vector::ContractionOp> {
       return rewriter.notifyMatchFailure(contractOp,
                                          "Expects add combining kind");
 
+    TypedValue<VectorType> lhs = contractOp.getLhs();
+    TypedValue<VectorType> rhs = contractOp.getRhs();
     TypedValue<Type> acc = contractOp.getAcc();
     VectorType accType = dyn_cast<VectorType>(acc.getType());
-    if (!accType || accType.getRank() != 2)
-      return rewriter.notifyMatchFailure(contractOp, "Expects acc 2D vector");
+    if (!accType)
+      return rewriter.notifyMatchFailure(contractOp, "Expects vector acc");
 
-    // Accept only plain 2D data layout.
-    // VNNI packing is applied to DPAS as a separate lowering step.
-    TypedValue<VectorType> lhs = contractOp.getLhs();
-    TypedValue<VectorType> rhs = contractOp.getRhs();
-    if (lhs.getType().getRank() != 2 || rhs.getType().getRank() != 2)
-      return rewriter.notifyMatchFailure(contractOp,
-                                         "Expects lhs and rhs 2D vectors");
+    if (!getRowMajorMatmulBatchRank(contractOp.getIndexingMapsAttr()))
+      return rewriter.notifyMatchFailure(
+          contractOp, "Expects a (batched) row-major matmul: leading dims must "
+                      "be batch dims shared by lhs, rhs, and acc; innermost two "
+                      "dims must be (M, K), (K, N), and (M, N)");
 
-    if (!isRowMajorMatmul(contractOp.getIndexingMapsAttr()))
-      return rewriter.notifyMatchFailure(contractOp, "Invalid indexing maps");
+    // xegpu.dpas operands are limited to rank 4 (2 batch + 2 core dims).
+    if (accType.getRank() > 4)
+      return rewriter.notifyMatchFailure(contractOp,
+                                         "Expects operands of rank 4 or less");
 
     auto dpasOp = xegpu::DpasOp::create(rewriter, loc,
                                         TypeRange{contractOp.getResultType()},
diff --git a/mlir/test/Conversion/VectorToXeGPU/contract-to-xegpu.mlir b/mlir/test/Conversion/VectorToXeGPU/contract-to-xegpu.mlir
index 292e4ff882000..7a5bc8f95f3c7 100644
--- a/mlir/test/Conversion/VectorToXeGPU/contract-to-xegpu.mlir
+++ b/mlir/test/Conversion/VectorToXeGPU/contract-to-xegpu.mlir
@@ -180,3 +180,51 @@ func.func @negative_accumulator_shape(%lhs: vector<8x16xf16>, %rhs: vector<16x16
 
 // CHECK-LABEL: @negative_accumulator_shape(
 // CHECK:       vector.contract
+
+// -----
+
+// A batched matmul whose leading dimension is a batch dimension shared across
+// lhs, rhs, and acc lowers to a batched xegpu.dpas.
+
+#map = affine_map<(d0, d1, d2, d3) -> (d0, d1, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d0, d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d0, d1, d2)>
+func.func @dpas_batched_gemm(%lhs: vector<4x8x16xf16>, %rhs: vector<4x16x16xf16>,
+    %acc: vector<4x8x16xf32>) -> vector<4x8x16xf32> {
+  %3 = vector.contract
+    {indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["parallel", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>} %lhs, %rhs, %acc
+    : vector<4x8x16xf16>, vector<4x16x16xf16> into vector<4x8x16xf32>
+  return %3 : vector<4x8x16xf32>
+}
+
+// CHECK-LABEL: @dpas_batched_gemm(
+// CHECK-SAME:  %[[LHS:.+]]: vector<4x8x16xf16>,
+// CHECK-SAME:  %[[RHS:.+]]: vector<4x16x16xf16>,
+// CHECK-SAME:  %[[ACC:.+]]: vector<4x8x16xf32>
+// CHECK:       %[[DPAS:.+]] = xegpu.dpas
+// CHECK-SAME:    %[[LHS]], %[[RHS]], %[[ACC]]
+// CHECK-SAME:    {{.*}}-> vector<4x8x16xf32>
+// CHECK:       return %[[DPAS]]
+
+// -----
+
+// An N-D contraction whose leading dimension is not a batch dimension shared
+// across lhs, rhs, and acc does not map to a (batched) row-major matmul.
+
+#map = affine_map<(d0, d1, d2, d3) -> (d1, d0, d3)>
+#map1 = affine_map<(d0, d1, d2, d3) -> (d3, d2)>
+#map2 = affine_map<(d0, d1, d2, d3) -> (d1, d2)>
+func.func @negative_non_batched_nd(%lhs: vector<128x64x64xf16>, %rhs: vector<64x64xf16>,
+    %acc: vector<128x64xf16>) -> vector<128x64xf16> {
+  %3 = vector.contract
+    {indexing_maps = [#map, #map1, #map2],
+    iterator_types = ["parallel", "parallel", "parallel", "reduction"],
+    kind = #vector.kind<add>} %lhs, %rhs, %acc
+    : vector<128x64x64xf16>, vector<64x64xf16> into vector<128x64xf16>
+  return %3 : vector<128x64xf16>
+}
+
+// CHECK-LABEL: @negative_non_batched_nd(
+// CHECK:       vector.contract

>From 6a2f80e02ff9dedf7220f0381050c05e25b045fc Mon Sep 17 00:00:00 2001
From: Jianhui Li <jian.hui.li at intel.com>
Date: Sat, 25 Jul 2026 02:50:46 +0000
Subject: [PATCH 2/2] fix format

---
 mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
index 850fd80793736..29f5fb3a1d9b2 100644
--- a/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
+++ b/mlir/lib/Conversion/VectorToXeGPU/VectorToXeGPU.cpp
@@ -1001,9 +1001,10 @@ struct ContractionLowering : public OpRewritePattern<vector::ContractionOp> {
 
     if (!getRowMajorMatmulBatchRank(contractOp.getIndexingMapsAttr()))
       return rewriter.notifyMatchFailure(
-          contractOp, "Expects a (batched) row-major matmul: leading dims must "
-                      "be batch dims shared by lhs, rhs, and acc; innermost two "
-                      "dims must be (M, K), (K, N), and (M, N)");
+          contractOp,
+          "Expects a (batched) row-major matmul: leading dims must "
+          "be batch dims shared by lhs, rhs, and acc; innermost two "
+          "dims must be (M, K), (K, N), and (M, N)");
 
     // xegpu.dpas operands are limited to rank 4 (2 batch + 2 core dims).
     if (accType.getRank() > 4)


From llvmlistbot at llvm.org  Fri Jul 24 20:08:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 20:08:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] cf95452 - [mlir][xegpu] Make convert_layout
 input_layout optional (#210833)
Message-ID: <6a6428ca.f06f4994.3876ce.c4eb@mx.google.com>


Author: Jianhui Li
Date: 2026-07-24T20:08:53-07:00
New Revision: cf9545250838bcb35e18ef872a89b790e7e893c3

URL: https://github.com/llvm/llvm-project/commit/cf9545250838bcb35e18ef872a89b790e7e893c3
DIFF: https://github.com/llvm/llvm-project/commit/cf9545250838bcb35e18ef872a89b790e7e893c3.diff

LOG: [mlir][xegpu] Make convert_layout input_layout optional (#210833)

Make the `input_layout` attribute of `xegpu.convert_layout` optional.
When it is not set, the effective input layout defaults to
`target_layout`, i.e. the source is assumed to already be in the target
distribution.

assisted-by-Claude

Co-authored-by: Claude Opus 4.8 <noreply at anthropic.com>

Added: 
    

Modified: 
    mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
    mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
    mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
    mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
    mlir/test/Dialect/XeGPU/peephole-optimize.mlir
    mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
    mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
    mlir/test/Dialect/XeGPU/propagate-layout.mlir
    mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
    mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
    mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
    mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
    mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
    mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
    mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
index 52785fa310aa4..65c046bb37ac4 100644
--- a/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
+++ b/mlir/include/mlir/Dialect/XeGPU/IR/XeGPUOps.td
@@ -1227,8 +1227,9 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
       Arguments:
       - `source`: The input vector whose data is to be redistributed. The source and
       result types must match.
-      - `input_layout`: The layout attribute describing the current distribution of `source`
-      across subgroups and/or lanes.
+      - `input_layout`: [optional] The layout attribute describing the current distribution
+      of `source` across subgroups and/or lanes. When omitted, it defaults to `target_layout`,
+      i.e. the source is assumed to already be in the target distribution.
       - `target_layout`: The layout attribute describing the desired distribution of the result
       across subgroups and/or lanes.
 
@@ -1241,7 +1242,7 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
         ```
     }];
     let arguments = (ins XeGPU_VectorOrScalarType: $source,
-                         DistributeLayoutAttr: $input_layout,
+                         OptionalAttr<DistributeLayoutAttr>: $input_layout,
                          DistributeLayoutAttr: $target_layout);
     let results = (outs XeGPU_VectorOrScalarType: $result);
     let assemblyFormat = [{
@@ -1256,6 +1257,13 @@ def XeGPU_ConvertLayoutOp: XeGPU_Op<"convert_layout", [AllTypesMatch<["source",
         setTargetLayoutAttr(anchorLayout);
       }
 
+      /// Returns `input_layout` if set, otherwise `target_layout`.
+      xegpu::DistributeLayoutAttr getEffectiveInputLayout() {
+        if (auto layout = getInputLayoutAttr())
+          return layout;
+        return getTargetLayoutAttr();
+      }
+
     }];
 
     let hasVerifier = 1;

diff  --git a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
index bfbb8b08bf173..060da62302867 100644
--- a/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
+++ b/mlir/lib/Dialect/XeGPU/IR/XeGPUOps.cpp
@@ -854,12 +854,10 @@ LogicalResult DpasOp::verify() {
 // XeGPU_ConvertLayoutOp
 //===----------------------------------------------------------------------===//
 LogicalResult ConvertLayoutOp::verify() {
-  auto srcLayout = getInputLayout();
   auto resLayout = getTargetLayout();
-  if (!srcLayout)
-    return emitOpError("expected input layout.");
   if (!resLayout)
     return emitOpError("expected target layout.");
+  auto srcLayout = getEffectiveInputLayout();
 
   // both input and target layouts should be WgLayout or SgLayout at the same
   // time.

diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
index 57b8ac73f41ac..9d1686403a431 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUBlocking.cpp
@@ -109,7 +109,7 @@ XeGPUBlockingPass::getTileShape(Operation *op) const {
 
   if (auto convertLayoutOp = dyn_cast<xegpu::ConvertLayoutOp>(op)) {
     auto inputInstData =
-        convertLayoutOp.getInputLayout().getEffectiveInstDataAsInt();
+        convertLayoutOp.getEffectiveInputLayout().getEffectiveInstDataAsInt();
     auto targetInstData =
         convertLayoutOp.getTargetLayout().getEffectiveInstDataAsInt();
     // return the one with larger size

diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
index 23338fe423cbb..c4a103fad6c56 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUPropagateLayout.cpp
@@ -1016,8 +1016,9 @@ void LayoutInfoPropagation::visitConvertLayoutOp(
   // TODO: fix if one of the layouts is a slice layout
   auto targetLayoutAttr =
       dyn_cast<xegpu::LayoutAttr>(convert.getTargetLayoutAttr());
+  // input_layout is optional, so it may be null.
   auto inputLayoutAttr =
-      dyn_cast<xegpu::LayoutAttr>(convert.getInputLayoutAttr());
+      dyn_cast_if_present<xegpu::LayoutAttr>(convert.getInputLayoutAttr());
 
   // The result's propagated layout is authoritative for the converted value.
   // Fill the lane_layout / lane_data / order parameters the target_layout is
@@ -1053,7 +1054,7 @@ void LayoutInfoPropagation::visitConvertLayoutOp(
     }
   }
 
-  xegpu::DistributeLayoutAttr anchorLayout = convert.getInputLayoutAttr();
+  xegpu::DistributeLayoutAttr anchorLayout = convert.getEffectiveInputLayout();
   LayoutInfo convertLayout = makeLayoutInfo(anchorLayout);
   // Propagate the new layout to the tensor descriptor operand.
   propagateIfChanged(operands[0], operands[0]->meet(convertLayout));
@@ -1670,6 +1671,10 @@ ResolveLayoutConflicts::resolveVectorConsumer(OpOperand &operand) {
   if (auto producerConvert =
           vectorValue.getDefiningOp<xegpu::ConvertLayoutOp>();
       producerConvert && vectorValue.hasOneUse()) {
+    // Pin the effective input before retargeting target, else an omitted
+    // input_layout would follow target and make the conversion a no-op.
+    producerConvert.setInputLayoutAttr(
+        producerConvert.getEffectiveInputLayout());
     producerConvert.setTargetLayoutAttr(consumerLayout);
     return success();
   }

diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
index 874487da10b30..a5a998f9b4427 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUSgToLaneDistribute.cpp
@@ -1673,7 +1673,7 @@ struct SgToLaneConvertLayout
   LogicalResult
   matchAndRewrite(xegpu::ConvertLayoutOp op, OpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
-    auto inputLayout = op.getInputLayoutAttr();
+    auto inputLayout = op.getEffectiveInputLayout();
     auto targetLayout = op.getTargetLayoutAttr();
     Type valType = op.getResult().getType();
 

diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
index bcb9946373298..14061ab24ffe0 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUUnroll.cpp
@@ -1048,7 +1048,7 @@ struct UnrollConvertLayoutOp : public UnrollPattern<xegpu::ConvertLayoutOp> {
     Location loc = op.getLoc();
     Type valType = op.getType();
 
-    xegpu::DistributeLayoutAttr inputLayout = op.getInputLayoutAttr();
+    xegpu::DistributeLayoutAttr inputLayout = op.getEffectiveInputLayout();
     xegpu::DistributeLayoutAttr targetLayout = op.getTargetLayoutAttr();
     if (!inputLayout || !targetLayout)
       return rewriter.notifyMatchFailure(op, "missing layout attributes.");

diff  --git a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
index 9ca6b3c2b0272..afea358dafe29 100644
--- a/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
+++ b/mlir/lib/Dialect/XeGPU/Transforms/XeGPUWgToSgDistribute.cpp
@@ -530,7 +530,7 @@ struct WgToSgConvertLayoutOp
   matchAndRewrite(xegpu::ConvertLayoutOp op, OneToNOpAdaptor adaptor,
                   ConversionPatternRewriter &rewriter) const override {
     Location loc = op.getLoc();
-    auto inputLayout = op.getInputLayout();
+    auto inputLayout = op.getEffectiveInputLayout();
     auto targetLayout = op.getTargetLayout();
 
     if (!inputLayout || !targetLayout || !inputLayout.isForWorkgroup() ||
@@ -1661,7 +1661,8 @@ void XeGPUWgToSgDistributePass::runOnOperation() {
 
   target.addDynamicallyLegalOp<xegpu::ConvertLayoutOp>(
       [=](xegpu::ConvertLayoutOp op) -> bool {
-        return isLegal(op.getInputLayout()) && isLegal(op.getTargetLayout());
+        return isLegal(op.getEffectiveInputLayout()) &&
+               isLegal(op.getTargetLayout());
       });
 
   target.addDynamicallyLegalDialect<math::MathDialect, arith::ArithDialect>(

diff  --git a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
index 9620e21f9bfdf..07dcb438ca553 100644
--- a/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
+++ b/mlir/lib/Dialect/XeGPU/Utils/XeGPUUtils.cpp
@@ -223,7 +223,7 @@ xegpu::getDistributeLayoutAttr(const OpOperand &opr) {
       return nullptr;
     }
     if (auto convertOp = dyn_cast<xegpu::ConvertLayoutOp>(op)) {
-      return convertOp.getInputLayoutAttr();
+      return convertOp.getEffectiveInputLayout();
     }
     auto layout = anchorOp.getAnchorLayout();
 

diff  --git a/mlir/test/Dialect/XeGPU/peephole-optimize.mlir b/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
index 79a6e248b0d83..fa5ff9ca56d34 100644
--- a/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
+++ b/mlir/test/Dialect/XeGPU/peephole-optimize.mlir
@@ -433,7 +433,7 @@ gpu.module @xevm_test {
 // CHECK:      %[[REDUCE_1:.*]] = vector.multi_reduction <add>, %[[LOADED]], %[[ACC_VEC]] [0] : vector<4x16xf32> to vector<16xf32>
 // CHECK:      %[[REDUCE_2:.*]] = vector.multi_reduction <add>, %[[REDUCE_1]], %[[ACC_SCALAR]] [0] : vector<16xf32> to f32
 // CHECK:      %[[BRIDGE:.*]] = xegpu.convert_layout %[[REDUCE_2]] <{input_layout = #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims = [0]>, target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
-// CHECK:      %[[CVT:.*]] = xegpu.convert_layout %[[BRIDGE]] <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>, target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
+// CHECK:      %[[CVT:.*]] = xegpu.convert_layout %[[BRIDGE]] <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}> : f32
 // CHECK:      %[[BCAST:.*]] = vector.broadcast %[[CVT]] : f32 to vector<16xf32>
 // CHECK:      xegpu.store %[[BCAST]], %[[ARG1]]
 gpu.module @xevm_test {
@@ -448,8 +448,7 @@ gpu.module @xevm_test {
      {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}
      [0, 1] : vector<4x16xf32> to f32
     %cvt = xegpu.convert_layout %reduce
-     <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>,
-       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}>
+     <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0, 1]>}>
      : f32
     %reduce_bcast = vector.broadcast %cvt
      {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>}

diff  --git a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
index a9182a01958ae..2ab9d7e4f3f66 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-inst-data.mlir
@@ -359,15 +359,14 @@ gpu.module @test {
 // CHECK-SAME: !xegpu.tensor_desc<256x32xui8, #xegpu.layout<inst_data = [32, 16], lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<256x32xui8>
 // CHECK: %[[BC:.*]] = vector.bitcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>} : vector<256x32xui8> to vector<256x64xf4E2M1FN>
 // CHECK: xegpu.convert_layout %[[BC]]
-// CHECK-SAME: <{input_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>, target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
+// CHECK-SAME: <{target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
 // CHECK-SAME: : vector<256x64xf4E2M1FN>
 func.func @bitcast_ui8_to_f4(%arg0: memref<256x32xui8>) {
   %0 = xegpu.create_nd_tdesc %arg0 : memref<256x32xui8> -> !xegpu.tensor_desc<256x32xui8>
   %1 = xegpu.load_nd %0[0, 0] : !xegpu.tensor_desc<256x32xui8> -> vector<256x32xui8>
   %2 = vector.bitcast %1 : vector<256x32xui8> to vector<256x64xf4E2M1FN>
   %3 = xegpu.convert_layout %2
-     <{input_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>,
-      target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
+     <{target_layout = #xegpu.layout<inst_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 2]>}>
      : vector<256x64xf4E2M1FN>
   return
 }
@@ -382,15 +381,14 @@ gpu.module @test {
 // CHECK-SAME: !xegpu.tensor_desc<256x16xui16, #xegpu.layout<inst_data = [32, 16], lane_layout = [1, 16], lane_data = [1, 1]>> -> vector<256x16xui16>
 // CHECK: %[[BC:.*]] = vector.bitcast %[[LOAD]] {layout_result_0 = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>} : vector<256x16xui16> to vector<256x64xf4E2M1FN>
 // CHECK: xegpu.convert_layout %[[BC]]
-// CHECK-SAME: <{input_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>, target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
+// CHECK-SAME: <{target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
 // CHECK-SAME: : vector<256x64xf4E2M1FN>
 func.func @bitcast_ui16_to_f4(%arg0: memref<256x16xui16>) {
   %0 = xegpu.create_nd_tdesc %arg0 : memref<256x16xui16> -> !xegpu.tensor_desc<256x16xui16>
   %1 = xegpu.load_nd %0[0, 0] : !xegpu.tensor_desc<256x16xui16> -> vector<256x16xui16>
   %2 = vector.bitcast %1 : vector<256x16xui16> to vector<256x64xf4E2M1FN>
   %3 = xegpu.convert_layout %2
-     <{input_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>,
-      target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
+     <{target_layout = #xegpu.layout<inst_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 4]>}>
      : vector<256x64xf4E2M1FN>
   return
 }

diff  --git a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
index 5c29695bafa31..70e13b39783bc 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout-subgroup.mlir
@@ -373,8 +373,7 @@ gpu.module @test {
     // CHECK-SAME: layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>
     %1 = xegpu.load_matrix %arg0[%c0, %c0] : !xegpu.mem_desc<32x128xf32>, index, index -> vector<32x128xf32>
     %2 = xegpu.convert_layout %1
-       <{input_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>,
-        target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
+       <{target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
        : vector<32x128xf32>
     gpu.return
   }

diff  --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..d48cb7993bfab 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -128,8 +128,7 @@ func.func @extf_truncf(%arg0: !xegpu.tensor_desc<8x16xf16>, %arg1: !xegpu.tensor
   %3 = arith.truncf %2 : vector<16x16xf32> to vector<16x16xf16>
   %4 = xegpu.dpas %0, %3 : vector<8x16xf16>, vector<16x16xf16> -> vector<8x16xf32>
     %5 = xegpu.convert_layout %4
-     <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
-      target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}>
+     <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}>
      : vector<8x16xf32>
   return %4 : vector<8x16xf32>
 }

diff  --git a/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir b/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
index 55d0e64bb2c65..6a625ff515029 100644
--- a/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
+++ b/mlir/test/Dialect/XeGPU/resolve-layout-conflicts.mlir
@@ -261,8 +261,7 @@ func.func @convert_layout() {
       dense<0.000000e+00>
      : vector<32x128xf32>
   %src0_cvt = xegpu.convert_layout %src0
-    <{input_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>,
-     target_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>}>
+    <{target_layout = #xegpu.layout<sg_layout=[8, 4], sg_data=[4, 32]>}>
     : vector<32x128xf32>
   %src1 = arith.constant
     {layout_result_0 = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>}
@@ -272,8 +271,7 @@ func.func @convert_layout() {
     {layout_result_0 = #xegpu.layout<sg_layout = [4, 8], sg_data = [8, 16]>}
     : vector<32x128xf32>
   %desc_cvt = xegpu.convert_layout %dest
-     <{input_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>,
-      target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
+     <{target_layout = #xegpu.layout<sg_layout=[4, 8], sg_data=[8, 16]>}>
      : vector<32x128xf32>
   return
 }

diff  --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
index fe356e6af35c1..a338ed5de7efc 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute-unit.mlir
@@ -118,7 +118,6 @@ gpu.func @dpas() {
     : vector<8x16xf16>, vector<16x16xf16>, vector<8x16xf32>  -> vector<8x16xf32>
   %anchor = xegpu.convert_layout %4
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -141,7 +140,6 @@ gpu.func @elementwise() {
     : vector<16x16xf32>
   %cl3 = xegpu.convert_layout %3
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -154,7 +152,6 @@ gpu.func @arith_constant() {
   %0 = arith.constant dense<1.0> : vector<16x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -176,7 +173,6 @@ gpu.func @arith_constant_non_splat() {
     dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]> : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xindex>
   gpu.return
@@ -200,7 +196,6 @@ gpu.func @arith_constant_non_splat_lane_data() {
     dense<[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]> : vector<32xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 2]>, dims = [0]>
     }> : vector<32xindex>
   gpu.return
@@ -239,7 +234,6 @@ gpu.func @arith_constant_non_splat_2d_vertical_lanedata() {
            [96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127]]> : vector<4x32xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [2, 1]>
     }> : vector<4x32xindex>
   gpu.return
@@ -377,7 +371,6 @@ gpu.func @vector_reduction() {
   %2 = vector.reduction <add>, %0, %acc : vector<32xf32> into f32
   %anchor = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>
     }> : f32
   gpu.return
@@ -440,7 +433,6 @@ gpu.func @vector_multi_reduction_dim1_distributed_dim1_reduction(%laneid: index)
       [1] : vector<2x16xf32> to vector<2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>
     }> : vector<2xf32>
   gpu.return
@@ -503,7 +495,6 @@ gpu.func @vector_multi_reduction_dim0_distributed_dim0_reduction(%laneid: index)
       [0] : vector<16x2xf32> to vector<2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [0]>
     }> : vector<2xf32>
   gpu.return
@@ -526,7 +517,6 @@ gpu.func @vector_multi_reduction_dim1_distributed_dim0_reduction(%laneid: index)
       [0] : vector<4x16xf32> to vector<16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xf32>
   gpu.return
@@ -549,7 +539,6 @@ gpu.func @vector_multi_reduction_dim0_distributed_dim1_reduction(%laneid: index)
       [1] : vector<16x12xf32> to vector<16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>, dims = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -567,7 +556,6 @@ gpu.func @vector_transpose() {
     : vector<16x2xf32> to vector<2x16xf32>
   %transpose2 = xegpu.convert_layout %transpose
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<2x16xf32>
   gpu.return
@@ -583,7 +571,6 @@ gpu.func @vector_bitcast() {
   %bitcast = vector.bitcast %cst : vector<4x32xi8> to vector<4x16xi16>
   %anchor = xegpu.convert_layout %bitcast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<4x16xi16>
   gpu.return
@@ -602,7 +589,6 @@ gpu.func @create_mask_1d(%m0: index) {
     : vector<16xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xi1>
   gpu.return
@@ -621,7 +607,6 @@ gpu.func @constant_mask_1d() {
     : vector<16xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xi1>
   gpu.return
@@ -646,7 +631,6 @@ gpu.func @create_mask_2d(%m0: index, %m1: index) {
     : vector<8x4xi1>
   %mask_cl = xegpu.convert_layout %mask
     <{
-      input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
     }> : vector<8x4xi1>
   gpu.return
@@ -672,7 +656,6 @@ gpu.func @constant_mask_2d() {
     : vector<8x4xi1>
       %mask_cl = xegpu.convert_layout %mask
         <{
-          input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
           target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
         }> : vector<8x4xi1>
       gpu.return
@@ -699,7 +682,6 @@ gpu.func @vector_multi_reduction_3d_leading_unit_dim_lane_local() {
       [1] : vector<1x16x32xf32> to vector<1x32xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>, dims = [1]>
     }> : vector<1x32xf32>
   gpu.return
@@ -730,7 +712,6 @@ gpu.func @vector_multi_reduction_3d_leading_unit_dim_cross_lane() {
       [1] : vector<1x16x2xf32> to vector<1x2xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16, 1], lane_data = [1, 1, 1]>, dims = [1]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16, 1], lane_data = [1, 1, 1]>, dims = [1]>
     }> : vector<1x2xf32>
   gpu.return
@@ -744,7 +725,6 @@ gpu.func @vector_extract_from_2d() {
     : vector<16xf32> from vector<4x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -759,7 +739,6 @@ gpu.func @vector_extract_from_2d_offset2() {
     : vector<16xf32> from vector<8x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<16xf32>
   gpu.return
@@ -776,7 +755,6 @@ gpu.func @vector_insert_into_2d() {
     : vector<16xf32> into vector<4x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<4x16xf32>
   gpu.return
@@ -793,7 +771,6 @@ gpu.func @vector_insert_into_2d_offset2() {
     : vector<16xf32> into vector<8x16xf32>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -809,7 +786,6 @@ gpu.func @vector_extract_strided_slice_distributed_dim_fully_extracted() {
     : vector<24x16xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -825,7 +801,6 @@ gpu.func @vector_extract_strided_slice_inner_distributed() {
     : vector<24x64xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -841,7 +816,6 @@ gpu.func @vector_extract_strided_slice_outer_distributed() {
     : vector<32x16xf32> to vector<16x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>
     }> : vector<16x16xf32>
   gpu.return
@@ -857,7 +831,6 @@ gpu.func @vector_extract_strided_slice_1d() {
     : vector<64xf32> to vector<32xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<32xf32>
   gpu.return
@@ -873,7 +846,6 @@ gpu.func @vector_extract_strided_slice_partial_offsets() {
     : vector<24x16xf32> to vector<8x16xf32>
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return
@@ -913,22 +885,18 @@ gpu.func @convert_layout_repack_lane_data() {
     : vector<32x16xi8> to vector<8x16xi8>
   %a0 = xegpu.convert_layout %s0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a1 = xegpu.convert_layout %s1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a2 = xegpu.convert_layout %s2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   %a3 = xegpu.convert_layout %s3
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1], order = [1, 0]>
     }> : vector<8x16xi8>
   gpu.return
@@ -946,7 +914,6 @@ gpu.func @vector_insert_strided_slice_distributed_dim_fully_inserted() {
     : vector<16x16xf32> into vector<64x16xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x16xf32>
   gpu.return
@@ -964,7 +931,6 @@ gpu.func @vector_insert_strided_slice_inner_distributed() {
     : vector<16x16xf32> into vector<64x32xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x32xf32>
   gpu.return
@@ -982,7 +948,6 @@ gpu.func @vector_insert_strided_slice_outer_distributed() {
     : vector<16x16xf32> into vector<48x32xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1]>
     }> : vector<48x32xf32>
   gpu.return
@@ -1000,7 +965,6 @@ gpu.func @vector_insert_strided_slice_1d() {
     : vector<16xf32> into vector<48xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>,
       target_layout = #xegpu.layout<lane_layout = [16], lane_data = [1]>
     }> : vector<48xf32>
   gpu.return
@@ -1018,7 +982,6 @@ gpu.func @vector_insert_strided_slice_
diff erent_ranks() {
     : vector<16xf32> into vector<64x16xf32>
   %cl2 = xegpu.convert_layout %2
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<64x16xf32>
   gpu.return
@@ -1054,8 +1017,7 @@ gpu.module @xevm_module {
 gpu.func @convert_layout_scalar() {
   %0 = "some_op"() : () -> f32
   %1 = xegpu.convert_layout %0
-    <{input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
-    target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>}>
+    <{target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>}>
     : f32
   "some_use"(%1) : (f32) -> ()
   gpu.return
@@ -1139,7 +1101,6 @@ gpu.func @elementwise_wrap_around_dim() {
     : vector<16x1xf16>
    %cl1 = xegpu.convert_layout %1
      <{
-       input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
        target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
      }> : vector<16x1xf16>
    gpu.return
@@ -1158,7 +1119,6 @@ gpu.func @vector_step_slice() {
   %0 = vector.step : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 2]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 2]>
     }> : vector<16xindex>
   gpu.return
@@ -1173,7 +1133,6 @@ gpu.func @vector_step_slice_unit() {
   %0 = vector.step : vector<1xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 3]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 1, 1, 16], lane_data = [1, 1, 1, 1]>, dims = [0, 1, 3]>
     }> : vector<1xindex>
   gpu.return
@@ -1195,7 +1154,6 @@ gpu.func @vector_step_slice_multi_dist() {
   %0 = vector.step : vector<16xindex>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [2, 4, 2], lane_data = [1, 2, 1]>, dims = [0, 2]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [2, 4, 2], lane_data = [1, 2, 1]>, dims = [0, 2]>
     }> : vector<16xindex>
   gpu.return
@@ -1213,7 +1171,6 @@ gpu.func @vector_shapecast_rank_increasing() {
     : vector<16xf32> to vector<1x16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<1x16xf32>
   gpu.return
@@ -1231,7 +1188,6 @@ gpu.func @vector_shapecast_rank_reducing() {
     : vector<1x16xf32> to vector<16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>
     }> : vector<16xf32>
   gpu.return
@@ -1249,7 +1205,6 @@ gpu.func @vector_shapecast_rank_increasing_without_slicing_layout() {
     : vector<16xf32> to vector<1x16xf32>
   %cast_cl = xegpu.convert_layout %cast
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<1x16xf32>
   gpu.return
@@ -1267,7 +1222,6 @@ gpu.func @vector_broadcast_1d_to_2d(%laneid: index) {
   %1 = vector.broadcast %0 : vector<16xf16> to vector<16x16xf16>
   %anchor = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x16xf16>
   gpu.return
@@ -1283,7 +1237,6 @@ gpu.func @constant_wrap_around_dim() {
   %0 = arith.constant dense<1.0> : vector<16x1xf16>
   %cl0 = xegpu.convert_layout %0
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<16x1xf16>
   gpu.return
@@ -1301,7 +1254,6 @@ gpu.func @vector_broadcast_2d_to_3d(%laneid: index) {
   %1 = vector.broadcast %0 : vector<16x16xf16> to vector<1x16x16xf16>
   %2 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 1, 16], lane_data = [1, 1, 1]>
     }> : vector<1x16x16xf16>
   "some_use"(%2) : (vector<1x16x16xf16>) -> ()
@@ -1317,7 +1269,7 @@ gpu.module @xevm_module {
 gpu.func @vector_broadcast_2d_to_2d_noop(%laneid: index) {
   %0 = "some_op"() : () -> vector<16x1xf16>
   %1 = vector.broadcast %0 : vector<16x1xf16> to vector<16x16xf16>
-  %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
+  %2 = xegpu.convert_layout %1 <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
   "some_use"(%2) : (vector<16x16xf16>) -> ()
   gpu.return
 }
@@ -1332,7 +1284,7 @@ gpu.module @xevm_module {
 gpu.func @vector_broadcast_scalar_to_vector(%laneid: index) {
   %0 = "some_op"() : () -> f16
   %1 = vector.broadcast %0 : f16 to vector<16x16xf16>
-  %2 = xegpu.convert_layout %1 <{input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
+  %2 = xegpu.convert_layout %1 <{target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>}> : vector<16x16xf16>
   "some_use"(%2) : (vector<16x16xf16>) -> ()
   gpu.return
 }
@@ -1379,7 +1331,6 @@ gpu.func @vector_multi_reduction_1d_to_scalar() {
       [0] : vector<32xf32> to f32
   %cl1 = xegpu.convert_layout %1
     <{
-      input_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>,
       target_layout = #xegpu.slice<#xegpu.layout<lane_layout = [16], lane_data = [1]>, dims = [0]>
     }> : f32
   gpu.return
@@ -1400,7 +1351,6 @@ gpu.func @vector_interleave() {
 
   %cl1 = xegpu.convert_layout %2
   <{
-    input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 2]>,
     target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 2]>
   }> : vector<8x4xf32>
 
@@ -1419,7 +1369,6 @@ gpu.func @vector_deinterleave() {
 
   %cl1 = xegpu.convert_layout %even
   <{
-    input_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>,
     target_layout = #xegpu.layout<lane_layout = [8, 2], lane_data = [1, 1]>
   }> : vector<8x2xf32>
 
@@ -1455,7 +1404,6 @@ gpu.func @xegpu_dpas_mx(%arg0: !xegpu.mem_desc<8x8xf16>, %arg1: !xegpu.mem_desc<
 
   %anchor = xegpu.convert_layout %4
     <{
-      input_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>,
       target_layout = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>
     }> : vector<8x16xf32>
   gpu.return

diff  --git a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
index fa9897770a08e..9eed7160c0baa 100644
--- a/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
+++ b/mlir/test/Dialect/XeGPU/sg-to-lane-distribute.mlir
@@ -484,7 +484,6 @@ gpu.module @xevm_module {
       %4 = vector.reduction <add>, %3 : vector<16xf32> into f32
       %anchor = xegpu.convert_layout %4
         <{
-          input_layout = #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims=[0]>,
           target_layout =  #xegpu.slice<#xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [0]>, dims=[0]>
         }>
         : f32

diff  --git a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
index bc50bad2b2302..2dc177cf8aa69 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-blocking.mlir
@@ -424,7 +424,7 @@ gpu.module @test_kernel {
     %a_tdesc = xegpu.create_nd_tdesc %arg0 : memref<16x16xf16> -> !xegpu.tensor_desc<16x16xf16, #a>
     %a = xegpu.load_nd %a_tdesc[0, 0] {layout = #a}: !xegpu.tensor_desc<16x16xf16, #a> -> vector<16x16xf16>
     %a_reduce = vector.multi_reduction <add>, %a, %acc [0, 1] : vector<16x16xf16> to f16
-    %13 = xegpu.convert_layout %a_reduce <{input_layout = #xegpu.slice<#a, dims = [0, 1]>, target_layout = #xegpu.slice<#a, dims = [0, 1]>}> : f16
+    %13 = xegpu.convert_layout %a_reduce <{target_layout = #xegpu.slice<#a, dims = [0, 1]>}> : f16
     memref.store %13, %arg1[%c0] : memref<4xf16>
     gpu.return
   }
@@ -483,7 +483,7 @@ gpu.module @test_kernel {
   gpu.func @convert_layout(%B: vector<8x32x2xf16>) -> vector<8x32x2xf16> {
     %b = xegpu.convert_layout %B <{input_layout = #lb, target_layout = #b}> : vector<8x32x2xf16>
     %e = math.exp %b : vector<8x32x2xf16>
-    %anchor = xegpu.convert_layout %e <{input_layout = #b, target_layout = #b}> : vector<8x32x2xf16>
+    %anchor = xegpu.convert_layout %e <{target_layout = #b}> : vector<8x32x2xf16>
     gpu.return %e : vector<8x32x2xf16>
   }
 }

diff  --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
index 3e8d183242a91..918d587f12db5 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-elemwise.mlir
@@ -15,7 +15,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %negf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8]>
       }> : vector<24x32xf32>
     gpu.return
@@ -34,7 +33,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %negf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -54,7 +52,6 @@ gpu.module @test_elementwise_ops {
     %powf = math.powf %addf, %load_b : vector<24x32xf32>
     %anchor = xegpu.convert_layout %powf
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -79,7 +76,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %fma
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return
@@ -101,7 +97,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf16> to vector<24x32xi16>
     %anchor = xegpu.convert_layout %bitcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xi16>
     gpu.return
@@ -130,7 +125,6 @@ gpu.module @test_elementwise_ops {
     %res = arith.select %cmpi, %cmpi, %cmpf : vector<24x32xi1>, vector<24x32xi1>
     %anchor = xegpu.convert_layout %res
       <{
-        input_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [2, 4], sg_data = [12, 8], lane_layout = [2, 8], lane_data = [1, 1]>
       }> : vector<24x32xi1>
     gpu.return
@@ -155,7 +149,6 @@ gpu.module @test_elementwise_ops {
       : vector<24x32xf32>
     %anchor = xegpu.convert_layout %powf
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2], lane_layout = [2, 2], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2], lane_layout = [2, 2], lane_data = [1, 1]>
       }> : vector<24x32xf32>
     gpu.return

diff  --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
index 819937d3ec86c..2969e551d93cb 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg-rr.mlir
@@ -93,7 +93,6 @@ gpu.module @test_distribution {
       : vector<256x64xf32> to vector<256xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 1], sg_data = [16, 64]>, dims = [1]>
       }> : vector<256xf32>
     gpu.return
@@ -123,7 +122,6 @@ gpu.module @test_distribution {
     %cst_2 = arith.constant dense<[[0], [16], [32], [48], [64], [80], [96], [112], [128], [144], [160], [176], [192], [208], [224], [240], [256], [272], [288], [304], [320], [336], [352], [368], [384], [400], [416], [432], [448], [464], [480], [496]]> : vector<32x1xindex>
     %anchor = xegpu.convert_layout %cst_2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [2, 1]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [2, 1]>
       }> : vector<32x1xindex>    
     gpu.return
@@ -142,7 +140,6 @@ gpu.module @test_distribution {
     : vector<256x128xf32> to vector<128x256xf32>
     %anchor = xegpu.convert_layout %trans
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [16, 32], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 8], sg_data = [16, 32], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<128x256xf32>
       gpu.return
@@ -155,7 +152,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [16, 16] : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>
       }> : vector<256x128xi1>
     gpu.return
@@ -168,7 +164,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst16, %cst16 : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [16, 16]>
       }> : vector<256x128xi1>
     gpu.return
@@ -241,7 +236,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %val, %acc [1] : vector<8x256xf32> to vector<8xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 16], sg_data = [4, 16]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 16], sg_data = [4, 16]>, dims = [1]>
       }> : vector<8xf32>
     gpu.return
@@ -253,7 +247,6 @@ gpu.module @test_distribution {
     %cst_2 = arith.constant dense<0> : vector<8xindex>
     %anchor = xegpu.convert_layout %cst_2
       <{
-        input_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4], order = [0, 1]>, dims = [0]>,
         target_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4], order = [0, 1]>, dims = [0]>
       }> : vector<8xindex>
     gpu.return
@@ -277,7 +270,6 @@ gpu.module @test_distribution {
     %bcast = vector.broadcast %2 : vector<8xindex> to vector<256x8xindex>
     %anchor = xegpu.convert_layout %bcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4]>,
         target_layout = #xegpu.layout<sg_layout = [16, 1], sg_data = [16, 4]>
       }> : vector<256x8xindex>
     gpu.return
@@ -306,7 +298,6 @@ gpu.module @test_distribution {
       : vector<128x1xf32> to vector<128x64xf32>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 1], sg_data = [16, 32], lane_layout = [8, 1], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 1], sg_data = [16, 32], lane_layout = [8, 1], lane_data = [1, 1]>
       }> : vector<128x64xf32>
     gpu.return

diff  --git a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
index 8c5dd6c55b99d..cdc9392f02e04 100644
--- a/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
+++ b/mlir/test/Dialect/XeGPU/xegpu-wg-to-sg.mlir
@@ -164,8 +164,7 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %load
       : vector<256x1xf32> to vector<256x32xf32>
     %anchor = xegpu.convert_layout %broadcast
-      <{input_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>,
-        target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>}>
+      <{target_layout = #xegpu.layout<sg_layout = [8, 1], sg_data = [32, 32], lane_layout = [8, 1], lane_data = [1, 1]>}>
       : vector<256x32xf32>
     gpu.return
   }
@@ -183,7 +182,6 @@ gpu.module @test_distribution {
       : vector<1x128xf32> to vector<32x128xf32>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [1, 4], sg_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [1, 4], sg_data = [32, 32], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<32x128xf32>
     gpu.return
@@ -270,7 +268,6 @@ gpu.module @test_distribution {
       %exp = math.exp %load : vector<128x64xf32>
       %anchor = xegpu.convert_layout %exp
         <{
-          input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>,
           target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>
         }> : vector<128x64xf32>
     }{sg_id_range = #xegpu.range<[2, 18]>}
@@ -304,7 +301,6 @@ gpu.module @test_distribution {
         %exp = math.exp %ld : vector<128x64xf32>
         %anchor = xegpu.convert_layout %exp
           <{
-            input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>,
             target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [32, 16], lane_layout = [8, 4], lane_data = [1, 1]>
           }> : vector<128x64xf32>
     }
@@ -419,8 +415,7 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [0]
       : vector<4x128xf32> to vector<128xf32>
     %anchor = xegpu.convert_layout %reduce
-      <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>,
-      target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>}>
+      <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32], sg_data = [4, 4]>, dims = [0]>}>
       : vector<128xf32>
     gpu.return
   }
@@ -437,8 +432,7 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [1]
       : vector<256x64xf32> to vector<256xf32>
     %anchor = xegpu.convert_layout %reduce
-      <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>,
-      target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>}>
+      <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [16, 1], sg_data = [16, 64]>, dims = [1]>}>
       : vector<256xf32>
     gpu.return
   }
@@ -454,7 +448,6 @@ gpu.module @test_distribution {
       : vector<4x2x6x32xf16> to vector<4x2x6xf16>
       %anchor = xegpu.convert_layout %reduce
         <{
-          input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>, dims = [3]>,
           target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>, dims = [3]>
         }> : vector<4x2x6xf16>
       gpu.return
@@ -483,7 +476,6 @@ gpu.module @test_distribution {
       : vector<32x32xf32> to f32
     %anchor = xegpu.convert_layout %reduce
         <{
-          input_layout =  #xegpu.slice<#xegpu.layout<sg_layout = [4, 4], sg_data = [8, 8]>, dims = [0, 1]>,
           target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 4], sg_data = [8, 8]>, dims = [0, 1]>
         }> : f32
     gpu.return
@@ -507,7 +499,6 @@ gpu.module @test_distribution {
     %step = vector.step : vector<128xindex>
     %anchor = xegpu.convert_layout %step
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 8], sg_data = [32, 32]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 8], sg_data = [32, 32]>, dims = [1]>
       }> : vector<128xindex>
     gpu.return
@@ -527,7 +518,6 @@ gpu.module @test_distribution {
     %step = vector.step : vector<128xindex>
     %anchor = xegpu.convert_layout %step
       <{
-        input_layout = #xegpu.layout<sg_layout = [16], sg_data = [8]>,
         target_layout = #xegpu.layout<sg_layout = [16], sg_data = [8]>
       }> : vector<128xindex>
     gpu.return
@@ -539,7 +529,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<10> : vector<4xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 1]>, dims = [1, 2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 1]>, dims = [1, 2, 3]>
       }> : vector<4xindex>
     gpu.return
@@ -554,7 +543,6 @@ gpu.module @test_distribution {
     %shape_cast = vector.shape_cast %muli : vector<128xindex> to vector<1x1x1x128xindex>
     %anchor = xegpu.convert_layout %shape_cast
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1, 1, 4], sg_data = [1, 1, 1, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1, 1, 4], sg_data = [1, 1, 1, 32]>
       }> : vector<1x1x1x128xindex>
     gpu.return
@@ -567,7 +555,6 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %muli : index to vector<4x2x6x32xindex>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>,
         target_layout = #xegpu.layout<sg_layout = [4, 2, 6, 1], sg_data = [1, 1, 1, 32]>
       }> : vector<4x2x6x32xindex>
     gpu.return
@@ -585,7 +572,6 @@ gpu.module @test_distribution {
       : vector<256x32xf32> to vector<32x256xf32>
     %anchor = xegpu.convert_layout %trans
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64], lane_layout = [1, 16], lane_data = [1, 1]>
       }> : vector<32x256xf32>
       gpu.return
@@ -606,7 +592,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<[[0], [16], [32], [48], [64], [80], [96], [112], [128], [144], [160], [176], [192], [208], [224], [240], [256], [272], [288], [304], [320], [336], [352], [368], [384], [400], [416], [432], [448], [464], [480], [496]]> : vector<32x1xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 1]>
       }> : vector<32x1xindex>
     gpu.return
@@ -641,7 +626,6 @@ gpu.module @test_distribution {
       ]> : vector<8x8xindex>
     %anchor = xegpu.convert_layout %cst_8x8
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2]>,
         target_layout = #xegpu.layout<sg_layout = [4, 4], sg_data = [2, 2]>
       }> : vector<8x8xindex>
       gpu.return
@@ -660,14 +644,12 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<[0, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496]> : vector<32xindex>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [32], sg_data = [1]>,
         target_layout = #xegpu.layout<sg_layout = [32], sg_data = [1]>
       }> : vector<32xindex>
     // CHECK: arith.constant dense<{{\[}}{{\[}}0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15{{\]}}{{\]}}> : vector<1x16xindex>
     %cst_1 = arith.constant dense<[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]> : vector<1x16xindex>
     %anchor_1 = xegpu.convert_layout %cst_1
       <{
-        input_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 16]>,
         target_layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 16]>
       }> : vector<1x16xindex>
     gpu.return
@@ -679,7 +661,6 @@ gpu.module @test_distribution {
     %broadcast = vector.broadcast %arg0 : index to vector<4x1x1xindex>
     %anchor = xegpu.convert_layout %broadcast
       <{
-        input_layout = #xegpu.layout<sg_layout = [4, 8, 1], sg_data = [1, 1, 1]>,
         target_layout = #xegpu.layout<sg_layout = [4, 8, 1], sg_data = [1, 1, 1]>
       }> : vector<4x1x1xindex>
     gpu.return
@@ -698,7 +679,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [8] : vector<32xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>,
         target_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>
       }> : vector<32xi1>
     gpu.return
@@ -724,7 +704,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.constant_mask [16, 16] : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi1>
     gpu.return
@@ -744,7 +723,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst8 : vector<32xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>,
         target_layout = #xegpu.layout<sg_layout = [2], sg_data = [16]>
       }> : vector<32xi1>
     gpu.return
@@ -771,7 +749,6 @@ gpu.module @test_distribution {
     %constant_mask = vector.create_mask %cst16, %cst16 : vector<256x128xi1>
     %anchor = xegpu.convert_layout %constant_mask
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi1>
     gpu.return
@@ -791,7 +768,6 @@ gpu.module @test_distribution {
     %4 = vector.broadcast %3 : vector<256xf32> to vector<256x256xf32>
     %anchor = xegpu.convert_layout %4
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [32, 32], inst_data = [8, 16]>
       }> : vector<256x256xf32>
     gpu.return
@@ -824,7 +800,6 @@ gpu.module @test_distribution {
     %15 = vector.multi_reduction <add>, %14, %cst_3 [1] : vector<1x32x32xf32> to vector<1x32xf32>
     %anchor = xegpu.convert_layout %15
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>, dims = [1]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [1, 32, 1], sg_data = [1, 1, 32]>, dims = [1]>
       }> : vector<1x32xf32>
     gpu.return
@@ -866,7 +841,6 @@ gpu.module @test_distribution {
       : vector<256x128xf32> to vector<128xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>, dims = [0]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>, dims = [0]>
       }> : vector<128xf32>
     gpu.return
@@ -899,7 +873,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<2x2x128x128xf32> to vector<2x2xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>, dims = [2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [1, 1, 32, 32]>, dims = [2, 3]>
       }> : vector<2x2xf32>
     gpu.return
@@ -932,7 +905,6 @@ gpu.module @test_distribution {
     %reduce = vector.multi_reduction <add>, %load, %cst [2, 3] : vector<32x32x128x128xf32> to vector<32x32xf32>
     %anchor = xegpu.convert_layout %reduce
       <{
-        input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>, dims = [2, 3]>,
         target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [2, 2, 4, 4], sg_data = [16, 16, 32, 32]>, dims = [2, 3]>
       }> : vector<32x32xf32>
     gpu.return
@@ -1017,7 +989,6 @@ gpu.module @test_distribution {
                                    target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>}> : vector<128x256xf32>
     %anchor = xegpu.convert_layout %2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8], sg_data = [16, 32], inst_data = [16, 16]>
       }> : vector<128x256xf32>
     gpu.return
@@ -1060,7 +1031,6 @@ gpu.module @test_distribution {
                                    target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>}> : vector<8x128x256xf32>
     %anchor = xegpu.convert_layout %2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 8, 8], sg_data = [1, 16, 32], inst_data = [1, 16, 16]>
       }> : vector<8x128x256xf32>
     gpu.return
@@ -1076,7 +1046,7 @@ gpu.module @test_distribution {
     %11 = xegpu.load %10[%offset], %mask <{layout = #xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>}> : i64, vector<32x32xindex>, vector<32x32xi1> -> vector<32x32xf32>
     %12 = vector.multi_reduction <add>, %11, %cst_0 [0, 1] : vector<32x32xf32> to f32
     // CHECK-NOT: xegpu.convert_layout
-    %13 = xegpu.convert_layout %12 <{input_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>, target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>}> : f32
+    %13 = xegpu.convert_layout %12 <{target_layout = #xegpu.slice<#xegpu.layout<sg_layout = [32, 1], sg_data = [1, 32]>, dims = [0, 1]>}> : f32
     gpu.return
   }
 
@@ -1111,7 +1081,6 @@ gpu.module @test_distribution {
     %bcast2 = vector.broadcast %scast2 : vector<256x16x1x256x16x16xf32> to vector<256x16x16x256x16x16xf32>
     %anchor = xegpu.convert_layout %bcast2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 1, 1, 8, 1, 1], sg_data = [32, 16, 16, 32, 16, 16]>,
         target_layout = #xegpu.layout<sg_layout = [8, 1, 1, 8, 1, 1], sg_data = [32, 16, 16, 32, 16, 16]>
       }> : vector<256x16x16x256x16x16xf32>
     gpu.return
@@ -1296,7 +1265,6 @@ gpu.module @test_distribution {
     %cst = arith.constant dense<1.0> : vector<256x128xf32>
     %anchor = xegpu.convert_layout %cst
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xf32>
     gpu.return
@@ -1340,7 +1308,6 @@ gpu.module @test_distribution {
     %bitcast2 = vector.bitcast %add : vector<256x256xi16> to vector<256x128xi32>
     %anchor = xegpu.convert_layout %bitcast2
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xi32>
     gpu.return
@@ -1361,7 +1328,6 @@ gpu.module @test_distribution {
       : vector<256x128xf32> -> vector<256x256xf32>
     %anchor = xegpu.convert_layout %interleave
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 64]>
       }> : vector<256x256xf32>
     gpu.return
@@ -1375,7 +1341,6 @@ gpu.module @test_distribution {
     %deinterleave:2 = vector.deinterleave %load : vector<256x256xf32> -> vector<256x128xf32>
     %anchor = xegpu.convert_layout %deinterleave#0
       <{
-        input_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>,
         target_layout = #xegpu.layout<sg_layout = [8, 4], sg_data = [32, 32]>
       }> : vector<256x128xf32>
     gpu.return


        

From llvmlistbot at llvm.org  Fri Jul 24 20:09:00 2026
From: llvmlistbot at llvm.org (Jianhui Li)
Date: Fri, 24 Jul 2026 20:09:00 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][xegpu] Make convert_layout
	input_layout optional (PR #210833)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210833@github.com>
Message-ID: <6a6428cc.bc28b987.1c0bcd.2cc4@mx.google.com>


https://github.com/Jianhui-Li closed https://github.com/llvm/llvm-project/pull/210833

From llvmlistbot at llvm.org  Fri Jul 24 20:36:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 20:36:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Support IsIsolatedFromAbove
 trait for python-defined operations (PR #211849)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211849@github.com>
Message-ID: <6a642f2b.1905e68e.117dfd.9612@mx.google.com>


PragmaTwice wrote:

> are these traits bound such that I can query traits on non-dynamic dialects and get these to pop out too?

Yes.

In the Python side, `op.has_trait(..)` supports all dialects (C++ defined and Python defined), e.g. `vector_yield_op.has_trait(IsTerminatorTrait)` returns `True`.

In the C++ side, `op->hasTrait<T>()` also supports all dialects (C++ defined and Python defined), e.g. for `class MyOp(.., traits=[IsTerminatorTrait])` in Python, if the op is handled in a C++ pass, `op->hasTrait<IsTerminator>()` will return `true`.

https://github.com/llvm/llvm-project/pull/211849

From llvmlistbot at llvm.org  Fri Jul 24 20:42:07 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 20:42:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 339792f - [MLIR][Python] Support
 IsIsolatedFromAbove trait for python-defined operations (#211849)
Message-ID: <6a64308f.a1accbb4.18b583.3fc9@mx.google.com>


Author: Twice
Date: 2026-07-25T11:42:02+08:00
New Revision: 339792f456abaae623c22789df6d1e1c59288839

URL: https://github.com/llvm/llvm-project/commit/339792f456abaae623c22789df6d1e1c59288839
DIFF: https://github.com/llvm/llvm-project/commit/339792f456abaae623c22789df6d1e1c59288839.diff

LOG: [MLIR][Python] Support IsIsolatedFromAbove trait for python-defined operations (#211849)

This PR adds dynamic `IsIsolatedFromAbove` trait support and exposes it
to Python as `IsIsolatedFromAboveTrait`.

The test verifies that a region with the trait cannot capture an SSA
value from above, while a region without the trait can.

Assisted by Codex/GPT 5.6 Sol (for writing test cases).

Added: 
    

Modified: 
    mlir/include/mlir-c/ExtensibleDialect.h
    mlir/include/mlir/Bindings/Python/IRCore.h
    mlir/include/mlir/IR/ExtensibleDialect.h
    mlir/lib/Bindings/Python/IRCore.cpp
    mlir/lib/CAPI/IR/ExtensibleDialect.cpp
    mlir/test/python/dialects/ext.py

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir-c/ExtensibleDialect.h b/mlir/include/mlir-c/ExtensibleDialect.h
index c849658c710bb..f8dcb3f6800d3 100644
--- a/mlir/include/mlir-c/ExtensibleDialect.h
+++ b/mlir/include/mlir-c/ExtensibleDialect.h
@@ -54,6 +54,15 @@ mlirDynamicOpTraitIsTerminatorCreate(void);
 /// terminator.
 MLIR_CAPI_EXPORTED MlirTypeID mlirDynamicOpTraitIsTerminatorGetTypeID(void);
 
+/// Get the dynamic op trait that indicates regions are isolated from above.
+MLIR_CAPI_EXPORTED MlirDynamicOpTrait
+mlirDynamicOpTraitIsIsolatedFromAboveCreate(void);
+
+/// Get the type ID of the dynamic op trait that indicates regions are isolated
+/// from above.
+MLIR_CAPI_EXPORTED MlirTypeID
+mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID(void);
+
 /// Get the dynamic op trait that indicates regions have no terminator.
 MLIR_CAPI_EXPORTED MlirDynamicOpTrait
 mlirDynamicOpTraitNoTerminatorCreate(void);

diff  --git a/mlir/include/mlir/Bindings/Python/IRCore.h b/mlir/include/mlir/Bindings/Python/IRCore.h
index 43a751bc9cd9f..3314e0b2a8fcf 100644
--- a/mlir/include/mlir/Bindings/Python/IRCore.h
+++ b/mlir/include/mlir/Bindings/Python/IRCore.h
@@ -1991,6 +1991,12 @@ class MLIR_PYTHON_API_EXPORTED NoTerminator : public PyDynamicOpTrait {
   static void bind(nanobind::module_ &m);
 };
 
+class MLIR_PYTHON_API_EXPORTED IsIsolatedFromAbove : public PyDynamicOpTrait {
+public:
+  static bool attach(const nanobind::object &opName, PyMlirContext &context);
+  static void bind(nanobind::module_ &m);
+};
+
 } // namespace PyDynamicOpTraits
 
 MLIR_PYTHON_API_EXPORTED MlirValue getUniqueResult(MlirOperation operation);

diff  --git a/mlir/include/mlir/IR/ExtensibleDialect.h b/mlir/include/mlir/IR/ExtensibleDialect.h
index 15904358def11..2219d6fd57977 100644
--- a/mlir/include/mlir/IR/ExtensibleDialect.h
+++ b/mlir/include/mlir/IR/ExtensibleDialect.h
@@ -416,6 +416,14 @@ class IsTerminator : public DynamicOpTraitImpl<OpTrait::IsTerminator> {
 
 class NoTerminator : public DynamicOpTraitImpl<OpTrait::NoTerminator> {};
 
+class IsIsolatedFromAbove
+    : public DynamicOpTraitImpl<OpTrait::IsIsolatedFromAbove> {
+public:
+  LogicalResult verifyRegionTrait(Operation *op) const override {
+    return OpTrait::impl::verifyIsIsolatedFromAbove(op);
+  }
+};
+
 } // namespace DynamicOpTraits
 
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/lib/Bindings/Python/IRCore.cpp b/mlir/lib/Bindings/Python/IRCore.cpp
index cceef55f9b15d..75cfd2a0a1c0b 100644
--- a/mlir/lib/Bindings/Python/IRCore.cpp
+++ b/mlir/lib/Bindings/Python/IRCore.cpp
@@ -2663,6 +2663,28 @@ void PyDynamicOpTraits::NoTerminator::bind(nb::module_ &m) {
       nb::arg("op_name"), nb::arg("context").none() = nb::none());
 }
 
+bool PyDynamicOpTraits::IsIsolatedFromAbove::attach(const nb::object &opName,
+                                                    PyMlirContext &context) {
+  MlirDynamicOpTrait trait = mlirDynamicOpTraitIsIsolatedFromAboveCreate();
+  return attachOpTrait(opName, trait, context);
+}
+
+void PyDynamicOpTraits::IsIsolatedFromAbove::bind(nb::module_ &m) {
+  nb::class_<PyDynamicOpTraits::IsIsolatedFromAbove, PyDynamicOpTrait> cls(
+      m, "IsIsolatedFromAboveTrait");
+  cls.attr(typeIDAttr) =
+      PyTypeID(mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID());
+  cls.attr("attach") = classmethod(
+      [](const nb::object &cls, const nb::object &opName,
+         DefaultingPyMlirContext context) {
+        return PyDynamicOpTraits::IsIsolatedFromAbove::attach(opName,
+                                                              *context.get());
+      },
+      "Attach IsIsolatedFromAbove trait to the given operation name.",
+      nb::arg("cls"), nb::arg("op_name"),
+      nb::arg("context").none() = nb::none());
+}
+
 } // namespace MLIR_BINDINGS_PYTHON_DOMAIN
 } // namespace python
 } // namespace mlir
@@ -5297,6 +5319,7 @@ void populateIRCore(nb::module_ &m) {
   PyDynamicOpTrait::bind(m);
   PyDynamicOpTraits::IsTerminator::bind(m);
   PyDynamicOpTraits::NoTerminator::bind(m);
+  PyDynamicOpTraits::IsIsolatedFromAbove::bind(m);
 
   // MLIRError exception.
   MLIRError::bind(m);

diff  --git a/mlir/lib/CAPI/IR/ExtensibleDialect.cpp b/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
index 5c1f74884f2c5..2b4d5af0c9f13 100644
--- a/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
+++ b/mlir/lib/CAPI/IR/ExtensibleDialect.cpp
@@ -54,6 +54,14 @@ MlirTypeID mlirDynamicOpTraitNoTerminatorGetTypeID() {
   return wrap(DynamicOpTraits::NoTerminator::getStaticTypeID());
 }
 
+MlirDynamicOpTrait mlirDynamicOpTraitIsIsolatedFromAboveCreate() {
+  return wrap(new DynamicOpTraits::IsIsolatedFromAbove());
+}
+
+MlirTypeID mlirDynamicOpTraitIsIsolatedFromAboveGetTypeID() {
+  return wrap(DynamicOpTraits::IsIsolatedFromAbove::getStaticTypeID());
+}
+
 void mlirDynamicOpTraitDestroy(MlirDynamicOpTrait dynamicOpTrait) {
   delete unwrap(dynamicOpTrait);
 }

diff  --git a/mlir/test/python/dialects/ext.py b/mlir/test/python/dialects/ext.py
index dde980df53dba..6e83da4a4a78a 100644
--- a/mlir/test/python/dialects/ext.py
+++ b/mlir/test/python/dialects/ext.py
@@ -637,6 +637,69 @@ class NoTermOp(TestRegion.Operation, name="no_term", traits=[NoTerminatorTrait])
             print(e)
 
 
+# CHECK: TEST: testIsIsolatedFromAboveTrait
+ at run
+def testIsIsolatedFromAboveTrait():
+    class TestIsolated(Dialect, name="ext_isolated"):
+        pass
+
+    class UseOp(TestIsolated.Operation, name="use"):
+        value: Operand[Any]
+
+    class NotIsolatedOp(
+        TestIsolated.Operation, name="not_isolated", traits=[NoTerminatorTrait]
+    ):
+        body: Region
+
+    class IsolatedOp(
+        TestIsolated.Operation,
+        name="isolated",
+        traits=[NoTerminatorTrait, IsIsolatedFromAboveTrait],
+    ):
+        body: Region
+
+    with Context(), Location.unknown():
+        TestIsolated.load()
+
+        # CHECK: not isolated has trait: False
+        print(
+            "not isolated has trait:",
+            NotIsolatedOp.has_trait(IsIsolatedFromAboveTrait),
+        )
+        # CHECK: isolated has trait: True
+        print("isolated has trait:", IsolatedOp.has_trait(IsIsolatedFromAboveTrait))
+
+        not_isolated_module = Module.create()
+        with InsertionPoint(not_isolated_module.body):
+            i32 = IntegerType.get_signless(32)
+            value = arith.constant(i32, 0)
+            not_isolated = NotIsolatedOp()
+            not_isolated.body.blocks.append()
+            with InsertionPoint(not_isolated.body.blocks[0]):
+                UseOp(value)
+
+        assert not_isolated_module.operation.verify()
+        # CHECK: not isolated: verification succeeded
+        print("not isolated: verification succeeded")
+
+        isolated_module = Module.create()
+        with InsertionPoint(isolated_module.body):
+            value = arith.constant(i32, 0)
+            isolated = IsolatedOp()
+            isolated.body.blocks.append()
+            with InsertionPoint(isolated.body.blocks[0]):
+                UseOp(value)
+
+        try:
+            isolated_module.operation.verify()
+        except MLIRError as e:
+            # CHECK: isolated: Verification failed:
+            # CHECK: using value defined outside the region
+            print("isolated:", e)
+        else:
+            raise AssertionError("expected IsIsolatedFromAbove verification to fail")
+
+
 # CHECK: TEST: testExtDialectWithType
 @run
 def testExtDialectWithType():


        

From llvmlistbot at llvm.org  Fri Jul 24 20:42:09 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 20:42:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Python] Support IsIsolatedFromAbove
 trait for python-defined operations (PR #211849)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211849@github.com>
Message-ID: <6a643091.510f576d.23298e.f817@mx.google.com>


https://github.com/PragmaTwice closed https://github.com/llvm/llvm-project/pull/211849

From llvmlistbot at llvm.org  Fri Jul 24 21:07:28 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 21:07:28 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Collapse dense and compressed
 iteration ranges (PR #211154)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211154@github.com>
Message-ID: <6a643680.492e23e4.3ce411.67d6@mx.google.com>


https://github.com/qyingwu updated https://github.com/llvm/llvm-project/pull/211154

>From e8482a923674bdb6bdb92c65a2e1ba0a1ba3bc2c Mon Sep 17 00:00:00 2001
From: qyingwu <qiyingwu at utexas.edu>
Date: Tue, 21 Jul 2026 18:35:09 -0700
Subject: [PATCH] [mlir][sparse] Diagnose unsupported collapsed iteration
 spaces

---
 .../Transforms/SparseTensorPasses.cpp         | 27 +++++++++++++++++++
 .../Transforms/Utils/SparseTensorIterator.cpp |  7 +++++
 .../SparseTensor/sparse_iteration_to_scf.mlir | 15 +++++++++++
 .../sparse_iteration_to_scf_invalid.mlir      | 15 +++++++++++
 4 files changed, 64 insertions(+)
 create mode 100644 mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf_invalid.mlir

diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
index b660e22154688..df8d696d75fd5 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/SparseTensorPasses.cpp
@@ -48,6 +48,28 @@ namespace {
 // Passes implementation.
 //===----------------------------------------------------------------------===//
 
+static bool supportsCollapsedRangeBetween(LevelType lt) {
+  return isDenseLT(lt) || isSingletonLT(lt);
+}
+
+static LogicalResult verifyLowerableCollapsedIterSpaces(Operation *op) {
+  WalkResult result = op->walk([&](ExtractIterSpaceOp op) {
+    auto [lvlLo, lvlHi] = op.getLvlRange();
+    SparseTensorEncodingAttr enc = cast<SparseTensorEncodingAttr>(
+        getRankedTensorType(op.getTensor()).getEncoding());
+    for (Level lvl = lvlLo + 1; lvl < lvlHi; ++lvl) {
+      LevelType lt = enc.getLvlType(lvl);
+      if (!supportsCollapsedRangeBetween(lt)) {
+        op.emitOpError() << "cannot lower collapsed iteration space with "
+                         << toMLIRString(lt) << " level after the first level";
+        return WalkResult::interrupt();
+      }
+    }
+    return WalkResult::advance();
+  });
+  return failure(result.wasInterrupted());
+}
+
 struct SparseAssembler : public impl::SparseAssemblerBase<SparseAssembler> {
   SparseAssembler() = default;
   SparseAssembler(const SparseAssembler &pass) = default;
@@ -167,6 +189,11 @@ struct LowerSparseIterationToSCFPass
       default;
 
   void runOnOperation() override {
+    if (failed(verifyLowerableCollapsedIterSpaces(getOperation()))) {
+      signalPassFailure();
+      return;
+    }
+
     auto *ctx = &getContext();
     RewritePatternSet patterns(ctx);
     SparseIterationTypeConverter converter;
diff --git a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/SparseTensorIterator.cpp b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/SparseTensorIterator.cpp
index b25e095d4c418..aa16eb837df3c 100644
--- a/mlir/lib/Dialect/SparseTensor/Transforms/Utils/SparseTensorIterator.cpp
+++ b/mlir/lib/Dialect/SparseTensor/Transforms/Utils/SparseTensorIterator.cpp
@@ -102,6 +102,12 @@ class DenseLevel : public SparseTensorLevel {
     Value posLo = MULI(p, lvlSize);
     return {posLo, lvlSize};
   }
+
+  ValuePair collapseRangeBetween(OpBuilder &b, Location l, ValueRange,
+                                 ValuePair parentRange) const override {
+    return {MULI(parentRange.first, lvlSize),
+            MULI(parentRange.second, lvlSize)};
+  }
 };
 
 class BatchLevel : public SparseTensorLevel {
@@ -168,6 +174,7 @@ class CompressedLevel : public SparseLevel</*hasPosBuf=*/true> {
     ValueRange posRange = posRangeIf.getResults();
     return {posRange.front(), posRange.back()};
   }
+
 }; // namespace
 
 class LooseCompressedLevel : public SparseLevel</*hasPosBuf=*/true> {
diff --git a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
index 855f1e99e7396..a649338bf02cc 100644
--- a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
+++ b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf.mlir
@@ -55,6 +55,10 @@ func.func @sparse_iteration_to_scf(%sp : tensor<4x8xf32, #COO>) -> index {
   map = (d0, d1) -> (d0 : dense, d1 : compressed)
 }>
 
+#DenseDense = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : dense, d1 : dense)
+}>
+
 // CHECK-LABEL:   @sparse_iteration_dense_level
 // CHECK:           scf.for
 func.func @sparse_iteration_dense_level(%sp: tensor<?x?xf64, #DenseCompressed>) {
@@ -65,3 +69,14 @@ func.func @sparse_iteration_dense_level(%sp: tensor<?x?xf64, #DenseCompressed>)
   }
   return
 }
+
+// CHECK-LABEL:   @sparse_iteration_dense_dense_space
+// CHECK:           scf.for
+func.func @sparse_iteration_dense_dense_space(%sp: tensor<?x?xf64, #DenseDense>) {
+  %0 = sparse_tensor.extract_iteration_space %sp lvls = 0 to 2
+      : tensor<?x?xf64, #DenseDense> -> !sparse_tensor.iter_space<#DenseDense, lvls = 0 to 2>
+  sparse_tensor.iterate %it in %0 at(%i, %j)
+      : !sparse_tensor.iter_space<#DenseDense, lvls = 0 to 2> {
+  }
+  return
+}
diff --git a/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf_invalid.mlir b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf_invalid.mlir
new file mode 100644
index 0000000000000..316488c108eb1
--- /dev/null
+++ b/mlir/test/Dialect/SparseTensor/sparse_iteration_to_scf_invalid.mlir
@@ -0,0 +1,15 @@
+// RUN: mlir-opt %s --lower-sparse-iteration-to-scf -verify-diagnostics
+
+#DenseCompressed = #sparse_tensor.encoding<{
+  map = (d0, d1) -> (d0 : dense, d1 : compressed)
+}>
+
+func.func @unsupported_dense_compressed_space(%sp: tensor<?x?xf64, #DenseCompressed>) {
+  // expected-error at below {{cannot lower collapsed iteration space with compressed level after the first level}}
+  %0 = sparse_tensor.extract_iteration_space %sp lvls = 0 to 2
+      : tensor<?x?xf64, #DenseCompressed> -> !sparse_tensor.iter_space<#DenseCompressed, lvls = 0 to 2>
+  sparse_tensor.iterate %it in %0 at(%i, %j)
+      : !sparse_tensor.iter_space<#DenseCompressed, lvls = 0 to 2> {
+  }
+  return
+}


From llvmlistbot at llvm.org  Fri Jul 24 21:17:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 21:17:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][sparse] Collapse dense and compressed
 iteration ranges (PR #211154)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211154@github.com>
Message-ID: <6a6438de.507fc713.3a4929.c97d@mx.google.com>


https://github.com/qyingwu edited https://github.com/llvm/llvm-project/pull/211154

From llvmlistbot at llvm.org  Fri Jul 24 22:21:15 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Fri, 24 Jul 2026 22:21:15 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Store blocks using Euler
 tour representation (PR #211485)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211485@github.com>
Message-ID: <6a6447cb.9edb5c9d.21e140.7cce@mx.google.com>


MaskRay wrote:

> > LGTM. Not 100% convinced this is worth the complexity, though.
> 
> Thanks for the review! Agreed that instructions:u delta is marginal. The value is two structural points you mentioned: removing `PopulateLoopDFS` and shrinking deeply-nested-loop storage.
> 
> Removing `PopulateLoopDFS` may unlock further optimizations.

Building `LoopInfo` is only ~1~2% of an `opt -O2` run, so the return is small.

https://github.com/llvm/llvm-project/pull/211485

From llvmlistbot at llvm.org  Fri Jul 24 22:48:43 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 22:48:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][SPIRV] Lower static workgroup
 attributions as global instead of argument (PR #210910)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210910@github.com>
Message-ID: <6a644e3b.af4834b4.1d4518.5388@mx.google.com>


Zustin wrote:

> @Zustin, i don't have Intel GPU platform, so I can't run mlir-runner to verify result. I just analyisis input.mlir and host mlir, It's most likely a problem of inconsistent parameters.

Hi @woruyu, I can confirm that the bug has been fixed by your PR (version: bb91a890c89697fe2736c16b81090f3b657f0081).

https://github.com/llvm/llvm-project/pull/210910

From llvmlistbot at llvm.org  Fri Jul 24 23:03:41 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Fri, 24 Jul 2026 23:03:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add
 affine-loop-constantize-bounds transfrom pass to affine dialect (PR
 #211989)
Message-ID: <llvm.org/llvm/llvm-project/pull/211989@github.com>


https://github.com/linuxlonelyeagle created https://github.com/llvm/llvm-project/pull/211989

This PR introduces the `affine-loop-constantize-bounds` pass, which leverages `ValueBoundsConstraintSet` analysis to refine dynamic/symbolic affine.for loop bounds into compile-time constant bounds.This transformation effectively bridges the gap between constrained symbolic bounds and the rigid constant requirements of the existing affine infrastructure. RFC: https://discourse.llvm.org/t/rfc-mlir-enable-dynamic-and-tighter-affine-unrolling-via-valueboundsconstraintset/91055/2

>From 559a1cebe5d06c1b21e84ea5f40cfbebca7a219d Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Sat, 25 Jul 2026 05:56:24 +0000
Subject: [PATCH] add affine-loop-constantize-bounds pass.

---
 .../mlir/Dialect/Affine/Transforms/Passes.td  |  13 +++
 .../AffineLoopConstantizeBounds.cpp           | 104 ++++++++++++++++++
 .../Dialect/Affine/Transforms/CMakeLists.txt  |   1 +
 .../affine-loop-constantize-bounds.mlir       |  45 ++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp
 create mode 100644 mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td b/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
index 7cecbcac5b932..44cb733c525b9 100644
--- a/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
@@ -465,4 +465,17 @@ def AffineFoldMemRefAliasOps : Pass<"affine-fold-memref-alias-ops"> {
   let dependentDialects = ["memref::MemRefDialect"];
 }
 
+def AffineLoopConstantizeBounds : Pass<"affine-loop-constantize-bounds"> {
+  let summary = "Constantize affine loop bounds using range analysis";
+  let description = [{
+    The pass leverages value range analysis (e.g., via the Presburger library)
+    to transform affine loops with dynamic symbolic bounds into a static-trip-count
+    main loop and a residual tail loop.
+
+    By rewriting the main loop to have strict compile-time constant bounds
+    (typically normalized to `0` to `TC_safe` with step `1`), it exposes the loop
+    body to downstream affine optimization pipelines.
+  }];
+}
+
 #endif // MLIR_DIALECT_AFFINE_PASSES
diff --git a/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp b/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp
new file mode 100644
index 0000000000000..529d4043f5693
--- /dev/null
+++ b/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp
@@ -0,0 +1,104 @@
+//===- AffineLoopConstantizeBounds.cpp - Constantize loop bounds pass ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a pass to transform affine loops with symbolic bounds
+// into a static-trip-count main loop and a residual tail loop using range
+// analysis.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Affine/Transforms/Passes.h"
+#include "mlir/Dialect/Affine/Utils.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
+#include <cstdint>
+
+namespace mlir {
+namespace affine {
+#define GEN_PASS_DEF_AFFINELOOPCONSTANTIZEBOUNDS
+#include "mlir/Dialect/Affine/Transforms/Passes.h.inc"
+} // namespace affine
+} // namespace mlir
+
+#define DEBUG_TYPE "affine-loop-constantize-bounds"
+
+using namespace mlir;
+using namespace mlir::affine;
+
+namespace {
+
+/// Computes the constant upper or lower bound for a given affine map expression
+/// \p map and its operands \p operands, constrained by the specified \p type.
+static FailureOr<int64_t> computeConstantBound(AffineMap map,
+                                               ValueRange operands,
+                                               presburger::BoundType type) {
+  ValueBoundsConstraintSet::Variable var(map, operands);
+  return ValueBoundsConstraintSet::computeConstantBound(type, var, nullptr,
+                                                        {true, true});
+}
+
+static LogicalResult inferAffineLoopUpperConstantBound(AffineForOp forOp) {
+  // Ensure the loop is normalized (lower bound is strictly 0 and step is 1).
+  if (!forOp.hasConstantLowerBound() || forOp.getConstantLowerBound() != 0)
+    return failure();
+  if (forOp.getStepAsInt() != 1)
+    return failure();
+
+  // Infer the range [upperMin, upperMax] for the upper bound. We require a
+  // strictly positive minimum bound (upperMin > 0) to guarantee a safe,
+  // non-empty static trip count for the main loop.
+  FailureOr<int64_t> upperMin = computeConstantBound(
+      forOp.getUpperBoundMap(), forOp.getUpperBoundOperands(),
+      presburger::BoundType::LB);
+  FailureOr<int64_t> upperMax = computeConstantBound(
+      forOp.getUpperBoundMap(), forOp.getUpperBoundOperands(),
+      presburger::BoundType::UB);
+  if (failed(upperMin) || *upperMin <= 0)
+    return failure();
+
+  IRRewriter b(forOp->getContext());
+
+  // The upper bound is dynamic within [upperMin, upperMax]. Split the loop into
+  // a static main loop (0 to upperMin) and a residual tail loop (upperMin to
+  // dynamic bound).
+  if (failed(upperMax) || *upperMax > *upperMin) {
+    b.setInsertionPoint(forOp);
+    AffineForOp clonedForOp = cast<AffineForOp>(b.clone(*forOp));
+    clonedForOp.setConstantUpperBound(*upperMin);
+    forOp.setConstantLowerBound(*upperMin);
+    forOp.getInitsMutable().assign(clonedForOp->getResults());
+    return success();
+  }
+
+  // If upperMin == upperMax. The upper bound is proven to be a strict constant
+  // at compile time. Directly constantize the bound without peeling a tail
+  // loop.
+  forOp.setConstantUpperBound(*upperMin);
+  return success();
+}
+
+struct AffineLoopConstantizeBounds
+    : public affine::impl::AffineLoopConstantizeBoundsBase<
+          AffineLoopConstantizeBounds> {
+  void runOnOperation() override;
+};
+} // namespace
+
+void AffineLoopConstantizeBounds::runOnOperation() {
+  SmallVector<AffineForOp, 4> loops;
+  getOperation()->walk([&](AffineForOp forOp) {
+    // First collect eligible loops and normalize them so that their lower bound
+    // is locked to 0 and step to 1.
+    if (succeeded(normalizeAffineFor(forOp, false)))
+      loops.push_back(forOp);
+  });
+  // Infer and rewrite the upper bound into a compile-time constant for each
+  // loop.
+  for (AffineForOp loop : loops)
+    (void)inferAffineLoopUpperConstantBound(loop);
+}
diff --git a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
index 9d912139810b2..3a854e1c7dc36 100644
--- a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
@@ -2,6 +2,7 @@ add_mlir_dialect_library(MLIRAffineTransforms
   AffineDataCopyGeneration.cpp
   AffineExpandIndexOps.cpp
   AffineExpandIndexOpsAsAffine.cpp
+  AffineLoopConstantizeBounds.cpp
   AffineLoopInvariantCodeMotion.cpp
   AffineLoopNormalize.cpp
   AffineParallelize.cpp
diff --git a/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir b/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir
new file mode 100644
index 0000000000000..6bb9aed223e83
--- /dev/null
+++ b/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir
@@ -0,0 +1,45 @@
+// RUN: mlir-opt %s -affine-loop-constantize-bounds -split-input-file | FileCheck %s
+
+// CHECK-DAG: #[[$MAP_APPLY:.+]] = affine_map<(d0)[s0] -> (d0 * 2 + s0)>
+// CHECK-DAG: #[[$MAP_UB:.+]] = affine_map<()[s0] -> ((-s0 + 9) ceildiv 2)>
+
+// CHECK-LABEL: func @peeling_main_loop
+
+func.func @peeling_main_loop() {
+  %c0 = arith.constant 0 :index
+  %bound = test.value_with_bounds { min = 0 : index, max = 1 : index}
+  affine.for %iv = %bound to 9 step 2 iter_args(%arg = %c0) -> index {
+    %sum = arith.addi %arg, %bound : index
+    affine.yield %sum : index
+  }
+  return
+}
+
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[BOUND:.*]] = test.value_with_bounds {max = 1 : index, min = 0 : index}
+// CHECK: %[[MAIN_RES:.*]] = affine.for %[[IV_MAIN:.*]] = 0 to 4 iter_args(%[[ARG_MAIN:.*]] = %[[C0]]) -> (index) {
+// CHECK:   %{{.*}} = affine.apply #map(%[[IV_MAIN]])[%[[BOUND]]]
+// CHECK: }
+// CHECK: %[[TAIL_RES:.*]] = affine.for %[[IV_TAIL:.*]] = 4 to #map1()[%[[BOUND]]] iter_args(%[[ARG_TAIL:.*]] = %[[MAIN_RES]]) -> (index) {
+// CHECK:   %{{.*}} = affine.apply #map(%[[IV_TAIL]])[%[[BOUND]]]
+// CHECK: }
+
+// -----
+// CHECK: #map = affine_map<(d0)[s0] -> (d0 * 2 + s0)>
+// CHECK-LABEL: func @fully_constantized_no_peeling
+
+func.func @fully_constantized_no_peeling() {
+  %c0 = arith.constant 0 :index
+  %bound = test.value_with_bounds { min = 0 : index, max = 1 : index}
+  affine.for %iv = %bound to 6 step 2 iter_args(%arg = %c0) -> index {
+    %sum = arith.addi %arg, %bound : index
+    affine.yield %sum : index
+  }
+  return
+}
+
+// CHECK:   %[[C0:.*]] = arith.constant 0 : index
+// CHECK:   %[[BOUND:.*]] = test.value_with_bounds {max = 1 : index, min = 0 : index}
+// CHECK:   %{{.*}} = affine.for %[[IV:.*]] = 0 to 3 iter_args(%{{.*}} = %[[C0]]) -> (index) {
+// CHECK:     %{{.*}} = affine.apply #map(%[[IV]])[%[[BOUND]]]
+// CHECK:   }


From llvmlistbot at llvm.org  Fri Jul 24 23:04:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Fri, 24 Jul 2026 23:04:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add
 affine-loop-constantize-bounds transfrom pass to affine dialect (PR
 #211989)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211989@github.com>
Message-ID: <6a6451e3.507fc713.3a4929.057c@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

This PR introduces the `affine-loop-constantize-bounds` pass, which leverages `ValueBoundsConstraintSet` analysis to refine dynamic/symbolic affine.for loop bounds into compile-time constant bounds.This transformation effectively bridges the gap between constrained symbolic bounds and the rigid constant requirements of the existing affine infrastructure. RFC: https://discourse.llvm.org/t/rfc-mlir-enable-dynamic-and-tighter-affine-unrolling-via-valueboundsconstraintset/91055/2

---
Full diff: https://github.com/llvm/llvm-project/pull/211989.diff


4 Files Affected:

- (modified) mlir/include/mlir/Dialect/Affine/Transforms/Passes.td (+13) 
- (added) mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp (+104) 
- (modified) mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt (+1) 
- (added) mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir (+45) 


``````````diff
diff --git a/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td b/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
index 7cecbcac5b932..44cb733c525b9 100644
--- a/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
+++ b/mlir/include/mlir/Dialect/Affine/Transforms/Passes.td
@@ -465,4 +465,17 @@ def AffineFoldMemRefAliasOps : Pass<"affine-fold-memref-alias-ops"> {
   let dependentDialects = ["memref::MemRefDialect"];
 }
 
+def AffineLoopConstantizeBounds : Pass<"affine-loop-constantize-bounds"> {
+  let summary = "Constantize affine loop bounds using range analysis";
+  let description = [{
+    The pass leverages value range analysis (e.g., via the Presburger library)
+    to transform affine loops with dynamic symbolic bounds into a static-trip-count
+    main loop and a residual tail loop.
+
+    By rewriting the main loop to have strict compile-time constant bounds
+    (typically normalized to `0` to `TC_safe` with step `1`), it exposes the loop
+    body to downstream affine optimization pipelines.
+  }];
+}
+
 #endif // MLIR_DIALECT_AFFINE_PASSES
diff --git a/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp b/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp
new file mode 100644
index 0000000000000..529d4043f5693
--- /dev/null
+++ b/mlir/lib/Dialect/Affine/Transforms/AffineLoopConstantizeBounds.cpp
@@ -0,0 +1,104 @@
+//===- AffineLoopConstantizeBounds.cpp - Constantize loop bounds pass ---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file implements a pass to transform affine loops with symbolic bounds
+// into a static-trip-count main loop and a residual tail loop using range
+// analysis.
+//
+//===----------------------------------------------------------------------===//
+
+#include "mlir/Dialect/Affine/IR/AffineOps.h"
+#include "mlir/Dialect/Affine/Transforms/Passes.h"
+#include "mlir/Dialect/Affine/Utils.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
+#include <cstdint>
+
+namespace mlir {
+namespace affine {
+#define GEN_PASS_DEF_AFFINELOOPCONSTANTIZEBOUNDS
+#include "mlir/Dialect/Affine/Transforms/Passes.h.inc"
+} // namespace affine
+} // namespace mlir
+
+#define DEBUG_TYPE "affine-loop-constantize-bounds"
+
+using namespace mlir;
+using namespace mlir::affine;
+
+namespace {
+
+/// Computes the constant upper or lower bound for a given affine map expression
+/// \p map and its operands \p operands, constrained by the specified \p type.
+static FailureOr<int64_t> computeConstantBound(AffineMap map,
+                                               ValueRange operands,
+                                               presburger::BoundType type) {
+  ValueBoundsConstraintSet::Variable var(map, operands);
+  return ValueBoundsConstraintSet::computeConstantBound(type, var, nullptr,
+                                                        {true, true});
+}
+
+static LogicalResult inferAffineLoopUpperConstantBound(AffineForOp forOp) {
+  // Ensure the loop is normalized (lower bound is strictly 0 and step is 1).
+  if (!forOp.hasConstantLowerBound() || forOp.getConstantLowerBound() != 0)
+    return failure();
+  if (forOp.getStepAsInt() != 1)
+    return failure();
+
+  // Infer the range [upperMin, upperMax] for the upper bound. We require a
+  // strictly positive minimum bound (upperMin > 0) to guarantee a safe,
+  // non-empty static trip count for the main loop.
+  FailureOr<int64_t> upperMin = computeConstantBound(
+      forOp.getUpperBoundMap(), forOp.getUpperBoundOperands(),
+      presburger::BoundType::LB);
+  FailureOr<int64_t> upperMax = computeConstantBound(
+      forOp.getUpperBoundMap(), forOp.getUpperBoundOperands(),
+      presburger::BoundType::UB);
+  if (failed(upperMin) || *upperMin <= 0)
+    return failure();
+
+  IRRewriter b(forOp->getContext());
+
+  // The upper bound is dynamic within [upperMin, upperMax]. Split the loop into
+  // a static main loop (0 to upperMin) and a residual tail loop (upperMin to
+  // dynamic bound).
+  if (failed(upperMax) || *upperMax > *upperMin) {
+    b.setInsertionPoint(forOp);
+    AffineForOp clonedForOp = cast<AffineForOp>(b.clone(*forOp));
+    clonedForOp.setConstantUpperBound(*upperMin);
+    forOp.setConstantLowerBound(*upperMin);
+    forOp.getInitsMutable().assign(clonedForOp->getResults());
+    return success();
+  }
+
+  // If upperMin == upperMax. The upper bound is proven to be a strict constant
+  // at compile time. Directly constantize the bound without peeling a tail
+  // loop.
+  forOp.setConstantUpperBound(*upperMin);
+  return success();
+}
+
+struct AffineLoopConstantizeBounds
+    : public affine::impl::AffineLoopConstantizeBoundsBase<
+          AffineLoopConstantizeBounds> {
+  void runOnOperation() override;
+};
+} // namespace
+
+void AffineLoopConstantizeBounds::runOnOperation() {
+  SmallVector<AffineForOp, 4> loops;
+  getOperation()->walk([&](AffineForOp forOp) {
+    // First collect eligible loops and normalize them so that their lower bound
+    // is locked to 0 and step to 1.
+    if (succeeded(normalizeAffineFor(forOp, false)))
+      loops.push_back(forOp);
+  });
+  // Infer and rewrite the upper bound into a compile-time constant for each
+  // loop.
+  for (AffineForOp loop : loops)
+    (void)inferAffineLoopUpperConstantBound(loop);
+}
diff --git a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
index 9d912139810b2..3a854e1c7dc36 100644
--- a/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
+++ b/mlir/lib/Dialect/Affine/Transforms/CMakeLists.txt
@@ -2,6 +2,7 @@ add_mlir_dialect_library(MLIRAffineTransforms
   AffineDataCopyGeneration.cpp
   AffineExpandIndexOps.cpp
   AffineExpandIndexOpsAsAffine.cpp
+  AffineLoopConstantizeBounds.cpp
   AffineLoopInvariantCodeMotion.cpp
   AffineLoopNormalize.cpp
   AffineParallelize.cpp
diff --git a/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir b/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir
new file mode 100644
index 0000000000000..6bb9aed223e83
--- /dev/null
+++ b/mlir/test/Dialect/Affine/affine-loop-constantize-bounds.mlir
@@ -0,0 +1,45 @@
+// RUN: mlir-opt %s -affine-loop-constantize-bounds -split-input-file | FileCheck %s
+
+// CHECK-DAG: #[[$MAP_APPLY:.+]] = affine_map<(d0)[s0] -> (d0 * 2 + s0)>
+// CHECK-DAG: #[[$MAP_UB:.+]] = affine_map<()[s0] -> ((-s0 + 9) ceildiv 2)>
+
+// CHECK-LABEL: func @peeling_main_loop
+
+func.func @peeling_main_loop() {
+  %c0 = arith.constant 0 :index
+  %bound = test.value_with_bounds { min = 0 : index, max = 1 : index}
+  affine.for %iv = %bound to 9 step 2 iter_args(%arg = %c0) -> index {
+    %sum = arith.addi %arg, %bound : index
+    affine.yield %sum : index
+  }
+  return
+}
+
+// CHECK: %[[C0:.*]] = arith.constant 0 : index
+// CHECK: %[[BOUND:.*]] = test.value_with_bounds {max = 1 : index, min = 0 : index}
+// CHECK: %[[MAIN_RES:.*]] = affine.for %[[IV_MAIN:.*]] = 0 to 4 iter_args(%[[ARG_MAIN:.*]] = %[[C0]]) -> (index) {
+// CHECK:   %{{.*}} = affine.apply #map(%[[IV_MAIN]])[%[[BOUND]]]
+// CHECK: }
+// CHECK: %[[TAIL_RES:.*]] = affine.for %[[IV_TAIL:.*]] = 4 to #map1()[%[[BOUND]]] iter_args(%[[ARG_TAIL:.*]] = %[[MAIN_RES]]) -> (index) {
+// CHECK:   %{{.*}} = affine.apply #map(%[[IV_TAIL]])[%[[BOUND]]]
+// CHECK: }
+
+// -----
+// CHECK: #map = affine_map<(d0)[s0] -> (d0 * 2 + s0)>
+// CHECK-LABEL: func @fully_constantized_no_peeling
+
+func.func @fully_constantized_no_peeling() {
+  %c0 = arith.constant 0 :index
+  %bound = test.value_with_bounds { min = 0 : index, max = 1 : index}
+  affine.for %iv = %bound to 6 step 2 iter_args(%arg = %c0) -> index {
+    %sum = arith.addi %arg, %bound : index
+    affine.yield %sum : index
+  }
+  return
+}
+
+// CHECK:   %[[C0:.*]] = arith.constant 0 : index
+// CHECK:   %[[BOUND:.*]] = test.value_with_bounds {max = 1 : index, min = 0 : index}
+// CHECK:   %{{.*}} = affine.for %[[IV:.*]] = 0 to 3 iter_args(%{{.*}} = %[[C0]]) -> (index) {
+// CHECK:     %{{.*}} = affine.apply #map(%[[IV]])[%[[BOUND]]]
+// CHECK:   }

``````````

</details>


https://github.com/llvm/llvm-project/pull/211989

From llvmlistbot at llvm.org  Sat Jul 25 02:55:36 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 25 Jul 2026 02:55:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LowerToLLVM] Pass DataLayoutAnalysis
 to LowerToLLVMOptions in `ArithToLLVM`, `ControlFlowToLLVM`,
 and `VectorToLLVM` passes (PR #206380)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206380@github.com>
Message-ID: <6a648818.6c5720b3.2e9b28.fdb0@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/206380

>From 305f638bb0911c2e2926a76e262710bac00e1746 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Mon, 29 Jun 2026 01:54:40 +0200
Subject: [PATCH 1/2] [mlir][conversion] Pass DataLayoutAnalysis to
 LowerToLLVMOptions in `ArithToLLVM`, `ControlFlowToLLVM`, and `VectorToLLVM`
 passes

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../Conversion/ArithToLLVM/ArithToLLVM.cpp    |  7 +-
 .../lib/Conversion/ArithToLLVM/CMakeLists.txt |  2 +
 .../ControlFlowToLLVM/CMakeLists.txt          |  1 +
 .../ControlFlowToLLVM/ControlFlowToLLVM.cpp   |  7 +-
 .../Conversion/VectorToLLVM/CMakeLists.txt    |  2 +
 .../VectorToLLVM/ConvertVectorToLLVMPass.cpp  | 10 ++-
 .../arith-to-llvm-32bit-index.mlir            | 74 ++++++++++++++++++
 .../ControlFlowToLLVM/branch-32bit-index.mlir | 65 ++++++++++++++++
 ...vector-load-store-32bit-index-to-llvm.mlir | 76 +++++++++++++++++++
 9 files changed, 238 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
 create mode 100644 mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
 create mode 100644 mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir

diff --git a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
index 6bf0fe85bb62f..64f6358dd938d 100644
--- a/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
+++ b/mlir/lib/Conversion/ArithToLLVM/ArithToLLVM.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Conversion/ArithToLLVM/ArithToLLVM.h"
 
+#include "mlir/Analysis/DataLayoutAnalysis.h"
 #include "mlir/Conversion/ArithCommon/AttrToLLVMConverter.h"
 #include "mlir/Conversion/ConvertToLLVM/ToLLVMInterface.h"
 #include "mlir/Conversion/LLVMCommon/ConversionTarget.h"
@@ -710,11 +711,13 @@ struct ArithToLLVMConversionPass
     LLVMConversionTarget target(getContext());
     RewritePatternSet patterns(&getContext());
 
-    LowerToLLVMOptions options(&getContext());
+    const auto &dataLayoutAnalysis = getAnalysis<DataLayoutAnalysis>();
+    LowerToLLVMOptions options(&getContext(),
+                               dataLayoutAnalysis.getAtOrAbove(getOperation()));
     if (indexBitwidth != kDeriveIndexBitwidthFromDataLayout)
       options.overrideIndexBitwidth(indexBitwidth);
 
-    LLVMTypeConverter converter(&getContext(), options);
+    LLVMTypeConverter converter(&getContext(), options, &dataLayoutAnalysis);
     arith::populateCeilFloorDivExpandOpsPatterns(patterns);
     arith::populateArithToLLVMConversionPatterns(converter, patterns);
 
diff --git a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
index 0a0e25e18b47a..8fdfa991a35d3 100644
--- a/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ArithToLLVM/CMakeLists.txt
@@ -14,6 +14,8 @@ add_mlir_conversion_library(MLIRArithToLLVM
   MLIRArithAttrToLLVMConversion
   MLIRArithDialect
   MLIRArithTransforms
+  MLIRAnalysis
+  MLIRDataLayoutInterfaces
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
   )
diff --git a/mlir/lib/Conversion/ControlFlowToLLVM/CMakeLists.txt b/mlir/lib/Conversion/ControlFlowToLLVM/CMakeLists.txt
index dcc78c8367a43..b36c85165d234 100644
--- a/mlir/lib/Conversion/ControlFlowToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/ControlFlowToLLVM/CMakeLists.txt
@@ -13,6 +13,7 @@ add_mlir_conversion_library(MLIRControlFlowToLLVM
 
   LINK_LIBS PUBLIC
   MLIRAnalysis
+  MLIRDataLayoutInterfaces
   MLIRControlFlowDialect
   MLIRLLVMCommonConversion
   MLIRLLVMDialect
diff --git a/mlir/lib/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.cpp b/mlir/lib/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.cpp
index fef78a46d69fc..e3d26cb68859b 100644
--- a/mlir/lib/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.cpp
+++ b/mlir/lib/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.cpp
@@ -13,6 +13,7 @@
 
 #include "mlir/Conversion/ControlFlowToLLVM/ControlFlowToLLVM.h"
 
+#include "mlir/Analysis/DataLayoutAnalysis.h"
 #include "mlir/Conversion/ConvertToLLVM/ToLLVMInterface.h"
 #include "mlir/Conversion/LLVMCommon/ConversionTarget.h"
 #include "mlir/Conversion/LLVMCommon/Pattern.h"
@@ -273,11 +274,13 @@ struct ConvertControlFlowToLLVM
              ctx->getLoadedDialect<cf::ControlFlowDialect>();
     });
 
-    LowerToLLVMOptions options(ctx);
+    const auto &dataLayoutAnalysis = getAnalysis<DataLayoutAnalysis>();
+    LowerToLLVMOptions options(ctx,
+                               dataLayoutAnalysis.getAtOrAbove(getOperation()));
     if (indexBitwidth != kDeriveIndexBitwidthFromDataLayout)
       options.overrideIndexBitwidth(indexBitwidth);
 
-    LLVMTypeConverter converter(ctx, options);
+    LLVMTypeConverter converter(ctx, options, &dataLayoutAnalysis);
     RewritePatternSet patterns(ctx);
     mlir::cf::populateControlFlowToLLVMConversionPatterns(converter, patterns);
     mlir::cf::populateAssertToLLVMConversionPattern(converter, patterns);
diff --git a/mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt b/mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt
index 0d700ea65eb4e..ea1a1de75f9bf 100644
--- a/mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt
+++ b/mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt
@@ -38,6 +38,8 @@ add_mlir_conversion_library(MLIRVectorToLLVMPass
   MLIRArmNeonTransforms
   MLIRArmSVEDialect
   MLIRArmSVETransforms
+  MLIRAnalysis
+  MLIRDataLayoutInterfaces
   MLIRX86Dialect
   MLIRX86Transforms
 )
diff --git a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
index 2b358d312dcfe..d80769ec4dc0b 100644
--- a/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
+++ b/mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
@@ -8,6 +8,7 @@
 
 #include "mlir/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.h"
 
+#include "mlir/Analysis/DataLayoutAnalysis.h"
 #include "mlir/Conversion/LLVMCommon/ConversionTarget.h"
 #include "mlir/Conversion/LLVMCommon/TypeConverter.h"
 #include "mlir/Dialect/Arith/IR/Arith.h"
@@ -108,8 +109,13 @@ void ConvertVectorToLLVMPass::runOnOperation() {
   }
 
   // Convert to the LLVM IR dialect.
-  LowerToLLVMOptions options(&getContext());
-  LLVMTypeConverter converter(&getContext(), options);
+  // Use the module's data layout so that the index bitwidth is derived from the
+  // target (e.g. i32 on 32-bit targets) rather than hard-wired to i64, which
+  // mirrors what FinalizeMemRefToLLVMConversionPass does.
+  const auto &dataLayoutAnalysis = getAnalysis<DataLayoutAnalysis>();
+  LowerToLLVMOptions options(&getContext(),
+                             dataLayoutAnalysis.getAtOrAbove(getOperation()));
+  LLVMTypeConverter converter(&getContext(), options, &dataLayoutAnalysis);
   RewritePatternSet patterns(&getContext());
   populateVectorTransferLoweringPatterns(patterns);
   populateVectorToLLVMConversionPatterns(
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
new file mode 100644
index 0000000000000..85b0324388b8a
--- /dev/null
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
@@ -0,0 +1,74 @@
+// RUN: mlir-opt %s --convert-arith-to-llvm -split-input-file | FileCheck %s
+
+// Verify that ArithToLLVMConversionPass respects the module's data layout
+// when deriving the index type.  When the module declares a 32-bit index via
+// dlti.dl_spec, index constants and index arithmetic ops must be emitted as
+// i32 instead of the default i64.
+
+// -----
+
+// 32-bit data layout: arith.constant with index type -> i32 constant.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @constant_index_32bit() -> index {
+  %c0 = arith.constant 0 : index
+  return %c0 : index
+}
+
+}
+
+// CHECK-LABEL: func @constant_index_32bit
+// CHECK: llvm.mlir.constant(0 : index) : i32
+
+// -----
+
+// 32-bit data layout: arith.cmpi on index type -> icmp on i32.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cmpi_index_32bit(%a: index, %b: index) -> i1 {
+  %cmp = arith.cmpi slt, %a, %b : index
+  return %cmp : i1
+}
+
+}
+
+// CHECK-LABEL: func @cmpi_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.icmp "slt" %{{.*}}, %{{.*}} : i32
+
+// -----
+
+// 32-bit data layout: arith.addi on index type -> add on i32.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @addi_index_32bit(%a: index, %b: index) -> index {
+  %add = arith.addi %a, %b : index
+  return %add : index
+}
+
+}
+
+// CHECK-LABEL: func @addi_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.add %{{.*}}, %{{.*}} : i32
+
+// -----
+
+// Without dlti.dl_spec the default index width (i64) is preserved.
+
+module {
+
+func.func @constant_index_default() -> index {
+  %c0 = arith.constant 0 : index
+  return %c0 : index
+}
+
+}
+
+// CHECK-LABEL: func @constant_index_default
+// CHECK: llvm.mlir.constant(0 : index) : i64
diff --git a/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir b/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
new file mode 100644
index 0000000000000..fcb32bc86a69d
--- /dev/null
+++ b/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
@@ -0,0 +1,65 @@
+// RUN: mlir-opt %s --convert-cf-to-llvm -split-input-file | FileCheck %s
+
+// Verify that ConvertControlFlowToLLVMPass respects the module's data layout
+// when deriving the index type for block arguments.  When the module declares
+// a 32-bit index via dlti.dl_spec, cf.br and cf.cond_br must pass index-typed
+// block arguments as i32 instead of the default i64.
+
+// -----
+
+// 32-bit data layout: cf.br with index block argument -> i32 branch arg.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cf_br_index_32bit(%arg0: index) -> index {
+  cf.br ^bb1(%arg0 : index)
+^bb1(%a: index):
+  return %a : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_br_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.br ^{{.*}}(%{{.*}} : i32)
+// CHECK: ^{{.*}}(%{{.*}}: i32):
+
+// -----
+
+// 32-bit data layout: cf.cond_br with index block arguments -> i32 branch args.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cf_cond_br_index_32bit(%cond: i1, %a: index, %b: index) -> index {
+  cf.cond_br %cond, ^bb1(%a : index), ^bb2(%b : index)
+^bb1(%x: index):
+  return %x : index
+^bb2(%y: index):
+  return %y : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_cond_br_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.cond_br %{{.*}}, ^{{.*}}(%{{.*}} : i32), ^{{.*}}(%{{.*}} : i32)
+
+// -----
+
+// Without dlti.dl_spec the default index width (i64) is preserved.
+
+module {
+
+func.func @cf_br_index_default(%arg0: index) -> index {
+  cf.br ^bb1(%arg0 : index)
+^bb1(%a: index):
+  return %a : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_br_index_default
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// CHECK: llvm.br ^{{.*}}(%{{.*}} : i64)
+// CHECK: ^{{.*}}(%{{.*}}: i64):
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir
new file mode 100644
index 0000000000000..29496be37322f
--- /dev/null
+++ b/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir
@@ -0,0 +1,76 @@
+// RUN: mlir-opt %s -convert-vector-to-llvm -split-input-file | FileCheck %s --check-prefixes=ALL,DEFAULT
+// RUN: mlir-opt %s -convert-vector-to-llvm='enable-gep-inbounds-nuw=1' -split-input-file | FileCheck %s --check-prefixes=ALL,INBOUNDS
+
+// Verify that ConvertVectorToLLVMPass respects the module's data layout when
+// deriving the index type. When the module declares a 32-bit index (via
+// dlti.dl_spec), GEP arithmetic should be emitted in i32 instead of the
+// default i64.  This also makes enable-gep-inbounds-nuw meaningful on 32-bit
+// targets: the flag emits nsw/nuw on the *narrow* i32 multiply, allowing SCEV
+// to form a clean 32-bit AddRec and enabling LSR to use direct pointer
+// induction instead of per-iteration sign/zero extension.
+
+// -----
+
+// 32-bit data layout: index -> i32 in GEP arithmetic.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @load_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+}
+
+// ALL-LABEL: func @load_32bit_index
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<8xf32>
+
+// -----
+
+// With enable-gep-inbounds-nuw, the narrow i32 multiply and add carry
+// nsw/nuw flags, enabling SCEV to form a clean 32-bit AddRec.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @load_32bit_index_nuw_mul(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+}
+
+// DEFAULT-LABEL: func @load_32bit_index_nuw_mul
+// DEFAULT: llvm.mul %{{.*}}, %{{.*}} : i32
+// DEFAULT: llvm.add %{{.*}}, %{{.*}} : i32
+// DEFAULT: llvm.getelementptr %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
+
+// INBOUNDS-LABEL: func @load_32bit_index_nuw_mul
+// INBOUNDS: llvm.mul %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
+// INBOUNDS: llvm.add %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
+// INBOUNDS: llvm.getelementptr inbounds|nuw %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
+
+// -----
+
+// Same test for store.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @store_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index, %val : vector<8xf32>) {
+  vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return
+}
+
+}
+
+// ALL-LABEL: func @store_32bit_index
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<8xf32>, !llvm.ptr

>From 0b60d5342adf276f64ae763ad1baf3d5a2ce3e64 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Sat, 25 Jul 2026 11:55:16 +0200
Subject: [PATCH 2/2] Address comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 .../arith-to-llvm-32bit-index.mlir            | 74 -----------------
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir | 83 +++++++++++++++++--
 .../ControlFlowToLLVM/branch-32bit-index.mlir | 65 ---------------
 .../Conversion/ControlFlowToLLVM/branch.mlir  | 64 ++++++++++++++
 ...vector-load-store-32bit-index-to-llvm.mlir | 76 -----------------
 .../vector-load-store-to-llvm.mlir            | 70 ++++++++++++++++
 6 files changed, 212 insertions(+), 220 deletions(-)
 delete mode 100644 mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
 delete mode 100644 mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
 delete mode 100644 mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir

diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
deleted file mode 100644
index 85b0324388b8a..0000000000000
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm-32bit-index.mlir
+++ /dev/null
@@ -1,74 +0,0 @@
-// RUN: mlir-opt %s --convert-arith-to-llvm -split-input-file | FileCheck %s
-
-// Verify that ArithToLLVMConversionPass respects the module's data layout
-// when deriving the index type.  When the module declares a 32-bit index via
-// dlti.dl_spec, index constants and index arithmetic ops must be emitted as
-// i32 instead of the default i64.
-
-// -----
-
-// 32-bit data layout: arith.constant with index type -> i32 constant.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @constant_index_32bit() -> index {
-  %c0 = arith.constant 0 : index
-  return %c0 : index
-}
-
-}
-
-// CHECK-LABEL: func @constant_index_32bit
-// CHECK: llvm.mlir.constant(0 : index) : i32
-
-// -----
-
-// 32-bit data layout: arith.cmpi on index type -> icmp on i32.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @cmpi_index_32bit(%a: index, %b: index) -> i1 {
-  %cmp = arith.cmpi slt, %a, %b : index
-  return %cmp : i1
-}
-
-}
-
-// CHECK-LABEL: func @cmpi_index_32bit
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: llvm.icmp "slt" %{{.*}}, %{{.*}} : i32
-
-// -----
-
-// 32-bit data layout: arith.addi on index type -> add on i32.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @addi_index_32bit(%a: index, %b: index) -> index {
-  %add = arith.addi %a, %b : index
-  return %add : index
-}
-
-}
-
-// CHECK-LABEL: func @addi_index_32bit
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: llvm.add %{{.*}}, %{{.*}} : i32
-
-// -----
-
-// Without dlti.dl_spec the default index width (i64) is preserved.
-
-module {
-
-func.func @constant_index_default() -> index {
-  %c0 = arith.constant 0 : index
-  return %c0 : index
-}
-
-}
-
-// CHECK-LABEL: func @constant_index_default
-// CHECK: llvm.mlir.constant(0 : index) : i64
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..db4a2b6f0cee7 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -1,4 +1,4 @@
-// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s
+// RUN: mlir-opt -pass-pipeline="builtin.module(func.func(convert-arith-to-llvm))" %s -split-input-file | FileCheck %s --check-prefixes=CHECK,CHECK-DERIVE
 
 // Same below, but using the `ConvertToLLVMPatternInterface` entry point
 // and the generic `convert-to-llvm` pass.
@@ -969,11 +969,11 @@ func.func @unsupported_fp_type(%arg0: f4E2M1FN, %arg1: vector<4xf4E2M1FN>, %arg2
 
 // -----
 
-//   CHECK-LABEL: func @supported_fp_type
-//         CHECK:   llvm.fadd {{.*}} : f32
-//         CHECK:   llvm.fadd {{.*}} : vector<4xf32>
+// CHECK-LABEL: func @supported_fp_type
+//       CHECK:   llvm.fadd {{.*}} : f32
+//       CHECK:   llvm.fadd {{.*}} : vector<4xf32>
 // CHECK-COUNT-4:   llvm.fadd {{.*}} : vector<8xf32>
-//         CHECK:   llvm.fcmp {{.*}} : f32
+//       CHECK:   llvm.fcmp {{.*}} : f32
 func.func @supported_fp_type(%arg0: f32, %arg1: vector<4xf32>, %arg2: vector<4x8xf32>, %arg3: f32) {
   %0 = arith.addf %arg0, %arg0 : f32
   %1 = arith.addf %arg1, %arg1 : vector<4xf32>
@@ -982,3 +982,76 @@ func.func @supported_fp_type(%arg0: f32, %arg1: vector<4xf32>, %arg2: vector<4x8
   return
 }
 
+// -----
+
+// Verify that the pass respects the module's data layout when deriving the
+// index type.  When the module declares a 32-bit index via dlti.dl_spec,
+// index constants and index arithmetic ops must be emitted as i32 instead of
+// the default i64.
+
+// 32-bit data layout: arith.constant with index type -> i32 constant.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @constant_index_32bit() -> index {
+  %c0 = arith.constant 0 : index
+  return %c0 : index
+}
+
+}
+
+// CHECK-DERIVE-LABEL: func @constant_index_32bit
+// CHECK-DERIVE: llvm.mlir.constant(0 : index) : i32
+
+// -----
+
+// 32-bit data layout: arith.cmpi on index type -> icmp on i32.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cmpi_index_32bit(%a: index, %b: index) -> i1 {
+  %cmp = arith.cmpi slt, %a, %b : index
+  return %cmp : i1
+}
+
+}
+
+// CHECK-DERIVE-LABEL: func @cmpi_index_32bit
+// CHECK-DERIVE: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK-DERIVE: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK-DERIVE: llvm.icmp "slt" %{{.*}}, %{{.*}} : i32
+
+// -----
+
+// 32-bit data layout: arith.addi on index type -> add on i32.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @addi_index_32bit(%a: index, %b: index) -> index {
+  %add = arith.addi %a, %b : index
+  return %add : index
+}
+
+}
+
+// CHECK-DERIVE-LABEL: func @addi_index_32bit
+// CHECK-DERIVE: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK-DERIVE: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK-DERIVE: llvm.add %{{.*}}, %{{.*}} : i32
+
+// -----
+
+// Without dlti.dl_spec the default index width (i64) is preserved.
+
+module {
+
+func.func @constant_index_default() -> index {
+  %c0 = arith.constant 0 : index
+  return %c0 : index
+}
+
+}
+
+// CHECK-DERIVE-LABEL: func @constant_index_default
+// CHECK-DERIVE: llvm.mlir.constant(0 : index) : i64
+
diff --git a/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir b/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
deleted file mode 100644
index fcb32bc86a69d..0000000000000
--- a/mlir/test/Conversion/ControlFlowToLLVM/branch-32bit-index.mlir
+++ /dev/null
@@ -1,65 +0,0 @@
-// RUN: mlir-opt %s --convert-cf-to-llvm -split-input-file | FileCheck %s
-
-// Verify that ConvertControlFlowToLLVMPass respects the module's data layout
-// when deriving the index type for block arguments.  When the module declares
-// a 32-bit index via dlti.dl_spec, cf.br and cf.cond_br must pass index-typed
-// block arguments as i32 instead of the default i64.
-
-// -----
-
-// 32-bit data layout: cf.br with index block argument -> i32 branch arg.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @cf_br_index_32bit(%arg0: index) -> index {
-  cf.br ^bb1(%arg0 : index)
-^bb1(%a: index):
-  return %a : index
-}
-
-}
-
-// CHECK-LABEL: func @cf_br_index_32bit
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: llvm.br ^{{.*}}(%{{.*}} : i32)
-// CHECK: ^{{.*}}(%{{.*}}: i32):
-
-// -----
-
-// 32-bit data layout: cf.cond_br with index block arguments -> i32 branch args.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @cf_cond_br_index_32bit(%cond: i1, %a: index, %b: index) -> index {
-  cf.cond_br %cond, ^bb1(%a : index), ^bb2(%b : index)
-^bb1(%x: index):
-  return %x : index
-^bb2(%y: index):
-  return %y : index
-}
-
-}
-
-// CHECK-LABEL: func @cf_cond_br_index_32bit
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
-// CHECK: llvm.cond_br %{{.*}}, ^{{.*}}(%{{.*}} : i32), ^{{.*}}(%{{.*}} : i32)
-
-// -----
-
-// Without dlti.dl_spec the default index width (i64) is preserved.
-
-module {
-
-func.func @cf_br_index_default(%arg0: index) -> index {
-  cf.br ^bb1(%arg0 : index)
-^bb1(%a: index):
-  return %a : index
-}
-
-}
-
-// CHECK-LABEL: func @cf_br_index_default
-// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i64
-// CHECK: llvm.br ^{{.*}}(%{{.*}} : i64)
-// CHECK: ^{{.*}}(%{{.*}}: i64):
diff --git a/mlir/test/Conversion/ControlFlowToLLVM/branch.mlir b/mlir/test/Conversion/ControlFlowToLLVM/branch.mlir
index 7c78211d59010..7d2aa0bc5d86e 100644
--- a/mlir/test/Conversion/ControlFlowToLLVM/branch.mlir
+++ b/mlir/test/Conversion/ControlFlowToLLVM/branch.mlir
@@ -81,3 +81,67 @@ func.func @cf_cond_br_with_weights(%cond: i1, %a: index, %b: index) -> index {
 ^bb2(%arg2: index):
   return %arg2 : index
 }
+
+// -----
+
+// Verify that the pass respects the module's data layout when deriving the
+// index type for block arguments.  When the module declares a 32-bit index
+// via dlti.dl_spec, cf.br and cf.cond_br must pass index-typed block
+// arguments as i32 instead of the default i64.
+
+// 32-bit data layout: cf.br with index block argument -> i32 branch arg.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cf_br_index_32bit(%arg0: index) -> index {
+  cf.br ^bb1(%arg0 : index)
+^bb1(%a: index):
+  return %a : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_br_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.br ^{{.*}}(%{{.*}} : i32)
+// CHECK: ^{{.*}}(%{{.*}}: i32):
+
+// -----
+
+// 32-bit data layout: cf.cond_br with index block arguments -> i32 branch args.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @cf_cond_br_index_32bit(%cond: i1, %a: index, %b: index) -> index {
+  cf.cond_br %cond, ^bb1(%a : index), ^bb2(%b : index)
+^bb1(%x: index):
+  return %x : index
+^bb2(%y: index):
+  return %y : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_cond_br_index_32bit
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i32
+// CHECK: llvm.cond_br %{{.*}}, ^{{.*}}(%{{.*}} : i32), ^{{.*}}(%{{.*}} : i32)
+
+// -----
+
+// Without dlti.dl_spec the default index width (i64) is preserved.
+
+module {
+
+func.func @cf_br_index_default(%arg0: index) -> index {
+  cf.br ^bb1(%arg0 : index)
+^bb1(%a: index):
+  return %a : index
+}
+
+}
+
+// CHECK-LABEL: func @cf_br_index_default
+// CHECK: builtin.unrealized_conversion_cast %{{.*}} : index to i64
+// CHECK: llvm.br ^{{.*}}(%{{.*}} : i64)
+// CHECK: ^{{.*}}(%{{.*}}: i64):
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir
deleted file mode 100644
index 29496be37322f..0000000000000
--- a/mlir/test/Conversion/VectorToLLVM/vector-load-store-32bit-index-to-llvm.mlir
+++ /dev/null
@@ -1,76 +0,0 @@
-// RUN: mlir-opt %s -convert-vector-to-llvm -split-input-file | FileCheck %s --check-prefixes=ALL,DEFAULT
-// RUN: mlir-opt %s -convert-vector-to-llvm='enable-gep-inbounds-nuw=1' -split-input-file | FileCheck %s --check-prefixes=ALL,INBOUNDS
-
-// Verify that ConvertVectorToLLVMPass respects the module's data layout when
-// deriving the index type. When the module declares a 32-bit index (via
-// dlti.dl_spec), GEP arithmetic should be emitted in i32 instead of the
-// default i64.  This also makes enable-gep-inbounds-nuw meaningful on 32-bit
-// targets: the flag emits nsw/nuw on the *narrow* i32 multiply, allowing SCEV
-// to form a clean 32-bit AddRec and enabling LSR to use direct pointer
-// induction instead of per-iteration sign/zero extension.
-
-// -----
-
-// 32-bit data layout: index -> i32 in GEP arithmetic.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @load_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
-  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
-  return %0 : vector<8xf32>
-}
-
-}
-
-// ALL-LABEL: func @load_32bit_index
-// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
-// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
-// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
-// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
-// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
-// ALL: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<8xf32>
-
-// -----
-
-// With enable-gep-inbounds-nuw, the narrow i32 multiply and add carry
-// nsw/nuw flags, enabling SCEV to form a clean 32-bit AddRec.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @load_32bit_index_nuw_mul(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
-  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
-  return %0 : vector<8xf32>
-}
-
-}
-
-// DEFAULT-LABEL: func @load_32bit_index_nuw_mul
-// DEFAULT: llvm.mul %{{.*}}, %{{.*}} : i32
-// DEFAULT: llvm.add %{{.*}}, %{{.*}} : i32
-// DEFAULT: llvm.getelementptr %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
-
-// INBOUNDS-LABEL: func @load_32bit_index_nuw_mul
-// INBOUNDS: llvm.mul %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
-// INBOUNDS: llvm.add %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
-// INBOUNDS: llvm.getelementptr inbounds|nuw %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
-
-// -----
-
-// Same test for store.
-
-module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
-
-func.func @store_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index, %val : vector<8xf32>) {
-  vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
-  return
-}
-
-}
-
-// ALL-LABEL: func @store_32bit_index
-// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
-// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
-// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
-// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
-// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
-// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<8xf32>, !llvm.ptr
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
index cab462efee2b5..3cddf0acaa89d 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-load-store-to-llvm.mlir
@@ -236,3 +236,73 @@ func.func @store_with_alignment(%memref : memref<200x100xf32>, %i : index, %j :
 
 // ALL-LABEL: func @store_with_alignment
 // ALL: llvm.store %{{.*}} {alignment = 8 : i64} :  vector<4xf32>, !llvm.ptr
+
+// -----
+
+// Verify that the pass respects the module's data layout when deriving the
+// index type. When the module declares a 32-bit index (via dlti.dl_spec),
+// GEP arithmetic should be emitted in i32 instead of the default i64.
+
+// 32-bit data layout: index -> i32 in GEP arithmetic.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @load_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+}
+
+// ALL-LABEL: func @load_32bit_index
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// ALL: llvm.load %[[GEP]] {alignment = 4 : i64} : !llvm.ptr -> vector<8xf32>
+
+// -----
+
+// With enable-gep-inbounds-nuw, the narrow i32 multiply and add carry
+// nsw/nuw flags, enabling SCEV to form a clean 32-bit AddRec.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @load_32bit_index_nuw_mul(%memref : memref<200x100xf32>, %i : index, %j : index) -> vector<8xf32> {
+  %0 = vector.load %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+}
+
+// DEFAULT-LABEL: func @load_32bit_index_nuw_mul
+// DEFAULT: llvm.mul %{{.*}}, %{{.*}} : i32
+// DEFAULT: llvm.add %{{.*}}, %{{.*}} : i32
+// DEFAULT: llvm.getelementptr %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
+
+// INBOUNDS-LABEL: func @load_32bit_index_nuw_mul
+// INBOUNDS: llvm.mul %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
+// INBOUNDS: llvm.add %{{.*}}, %{{.*}} overflow<nsw, nuw> : i32
+// INBOUNDS: llvm.getelementptr inbounds|nuw %{{.*}} : (!llvm.ptr, i32) -> !llvm.ptr, f32
+
+// -----
+
+// Same test for store.
+
+module attributes { dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<index, 32>> } {
+
+func.func @store_32bit_index(%memref : memref<200x100xf32>, %i : index, %j : index, %val : vector<8xf32>) {
+  vector.store %val, %memref[%i, %j] : memref<200x100xf32>, vector<8xf32>
+  return
+}
+
+}
+
+// ALL-LABEL: func @store_32bit_index
+// ALL: %[[C100:.*]] = llvm.mlir.constant(100 : index) : i32
+// ALL: %[[MUL:.*]] = llvm.mul %{{.*}}, %[[C100]]
+// ALL: %[[ADD:.*]] = llvm.add %[[MUL]], %{{.*}}
+// DEFAULT: %[[GEP:.*]] = llvm.getelementptr %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// INBOUNDS: %[[GEP:.*]] = llvm.getelementptr inbounds|nuw %{{.*}}[%[[ADD]]] : (!llvm.ptr, i32) -> !llvm.ptr, f32
+// ALL: llvm.store %{{.*}}, %[[GEP]] {alignment = 4 : i64} : vector<8xf32>, !llvm.ptr


From llvmlistbot at llvm.org  Sat Jul 25 02:56:45 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sat, 25 Jul 2026 02:56:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][LowerToLLVM] Pass DataLayoutAnalysis
 to LowerToLLVMOptions in `ArithToLLVM`, `ControlFlowToLLVM`,
 and `VectorToLLVM` passes (PR #206380)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/206380@github.com>
Message-ID: <6a64885d.c4d3d80a.348a49.16a2@mx.google.com>


================

----------------
FedericoBruzzone wrote:

Hi @ftynse! Thanks for your suggestion :D 

https://github.com/llvm/llvm-project/pull/206380

From llvmlistbot at llvm.org  Sat Jul 25 06:24:10 2026
From: llvmlistbot at llvm.org (Akimasa Watanuki)
Date: Sat, 25 Jul 2026 06:24:10 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Harden affine loop folding (PR #208736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208736@github.com>
Message-ID: <6a64b8fa.6c8668f2.1674c5.9b2c@mx.google.com>


https://github.com/Men-cotton closed https://github.com/llvm/llvm-project/pull/208736

From llvmlistbot at llvm.org  Sat Jul 25 06:24:11 2026
From: llvmlistbot at llvm.org (Akimasa Watanuki)
Date: Sat, 25 Jul 2026 06:24:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir] Harden affine loop folding (PR #208736)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/208736@github.com>
Message-ID: <6a64b8fb.f3151ff8.2f3401.31b5@mx.google.com>


Men-cotton wrote:

Closing as superseded by #204521. 

https://github.com/llvm/llvm-project/pull/208736

From llvmlistbot at llvm.org  Sat Jul 25 10:34:35 2026
From: llvmlistbot at llvm.org (Mingfei Guo)
Date: Sat, 25 Jul 2026 10:34:35 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve WAR dependences in
	loop fusion (PR #212045)
Message-ID: <llvm.org/llvm/llvm-project/pull/212045@github.com>


https://github.com/guoriyue created https://github.com/llvm/llvm-project/pull/212045

## Summary

Preserve affine write-after-read dependences when fusing a producer slice into a consumer loop.

The legality check composes the fused schedule (T -> S) with the original dependence (S -> D) and rejects fusion when the producer would execute lexicographically after its destination (T > D). Unsupported relation shapes are conservatively rejected.

Fixes #212032.

## Testing

- ninja -C build-212032 -j8 mlir-opt FileCheck count not
- llvm-lit: loop-fusion-3.mlir (1 passed)
- llvm-lit: full Affine loop-fusion filter (10 passed)
- Same-build A/B with the new guard disabled: 5 negative checks fail without the fix

>From 2cb8501fb4ef979931043d866e86baaa168b172d Mon Sep 17 00:00:00 2001
From: Mingfei Guo <1800012773 at pku.edu.cn>
Date: Sat, 25 Jul 2026 10:29:20 -0700
Subject: [PATCH] [MLIR][Affine] Preserve WAR dependences in loop fusion

---
 .../Dialect/Affine/Utils/LoopFusionUtils.cpp  | 148 +++++++++++++
 mlir/test/Dialect/Affine/loop-fusion-3.mlir   | 198 ++++++++++++++++++
 2 files changed, 346 insertions(+)

diff --git a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
index 68296ea3368a1..2a44ef3faaac2 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
@@ -11,6 +11,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "mlir/Dialect/Affine/LoopFusionUtils.h"
+#include "mlir/Analysis/Presburger/IntegerRelation.h"
 #include "mlir/Analysis/SliceAnalysis.h"
 #include "mlir/Analysis/TopologicalSortUtils.h"
 #include "mlir/Dialect/Affine/Analysis/AffineAnalysis.h"
@@ -242,6 +243,146 @@ static unsigned getMaxLoopDepth(ArrayRef<Operation *> srcOps,
   return loopDepth;
 }
 
+/// Return whether `dependenceConstraints` contains a dependence that becomes
+/// lexicographically backward after executing the source slice in the
+/// destination loop nest.
+static std::optional<bool> hasBackwardDependenceAfterFusion(
+    const FlatAffineValueConstraints &dependenceConstraints,
+    const FlatAffineValueConstraints &sliceConstraints,
+    ArrayRef<Value> srcIVs, ArrayRef<Value> dstIVs, unsigned dstLoopDepth,
+    unsigned numCommonLoops) {
+  if (dstLoopDepth < numCommonLoops || dstIVs.size() < dstLoopDepth)
+    return std::nullopt;
+
+  unsigned numSrcIVs = srcIVs.size();
+  if (sliceConstraints.getNumDimVars() != numSrcIVs ||
+      dependenceConstraints.getNumDimVars() !=
+          numSrcIVs + dstIVs.size())
+    return std::nullopt;
+  for (auto [index, iv] : llvm::enumerate(srcIVs)) {
+    if (!sliceConstraints.hasValue(index) ||
+        sliceConstraints.getValue(index) != iv ||
+        !dependenceConstraints.hasValue(index) ||
+        dependenceConstraints.getValue(index) != iv)
+      return std::nullopt;
+  }
+  for (auto [index, iv] : llvm::enumerate(dstIVs)) {
+    unsigned pos = numSrcIVs + index;
+    if (!dependenceConstraints.hasValue(pos) ||
+        dependenceConstraints.getValue(pos) != iv)
+      return std::nullopt;
+  }
+
+  // View the slice as a relation from its destination insertion point to the
+  // source iteration, and the dependence as source-to-destination. Their
+  // composition directly relates the transformed source schedule to the
+  // original dependence destination.
+  unsigned firstScheduleSymbol = 0;
+  unsigned numScheduleDims = dstLoopDepth - numCommonLoops;
+  for (unsigned depth = numCommonLoops; depth < dstLoopDepth; ++depth) {
+    unsigned pos;
+    if (!sliceConstraints.findVar(dstIVs[depth], &pos,
+                                  sliceConstraints.getNumDimVars()))
+      return std::nullopt;
+    unsigned symbolPos = pos - sliceConstraints.getNumDimVars();
+    if (depth == numCommonLoops)
+      firstScheduleSymbol = symbolPos;
+    else if (symbolPos != firstScheduleSymbol + depth - numCommonLoops)
+      return std::nullopt;
+  }
+
+  presburger::IntegerRelation sliceRelation(sliceConstraints);
+  sliceRelation.convertVarKind(presburger::VarKind::Symbol,
+                               firstScheduleSymbol,
+                               firstScheduleSymbol + numScheduleDims,
+                               presburger::VarKind::Domain);
+  presburger::IntegerRelation fusedDependence(dependenceConstraints);
+  fusedDependence.convertVarKind(presburger::VarKind::Range, 0, numSrcIVs,
+                                 presburger::VarKind::Domain);
+  fusedDependence.mergeAndCompose(sliceRelation);
+  if (fusedDependence.getNumDomainVars() != numScheduleDims ||
+      fusedDependence.getNumRangeVars() != dstIVs.size())
+    return std::nullopt;
+
+  unsigned dstOffset =
+      fusedDependence.getVarKindOffset(presburger::VarKind::Range);
+  for (unsigned depth = 0; depth < numScheduleDims; ++depth) {
+    presburger::IntegerRelation backwardConstraints(fusedDependence);
+    for (unsigned outerDepth = 0; outerDepth < depth; ++outerDepth) {
+      SmallVector<int64_t> equality(backwardConstraints.getNumCols(), 0);
+      equality[outerDepth] = 1;
+      equality[dstOffset + numCommonLoops + outerDepth] = -1;
+      backwardConstraints.addEquality(equality);
+    }
+
+    SmallVector<int64_t> inequality(backwardConstraints.getNumCols(), 0);
+    inequality[depth] = 1;
+    inequality[dstOffset + numCommonLoops + depth] = -1;
+    inequality.back() = -1;
+    backwardConstraints.addInequality(inequality);
+    if (!backwardConstraints.isIntegerEmpty())
+      return true;
+  }
+  return false;
+}
+
+/// Check affine WAR dependences between the source and destination that did
+/// not define the producer-consumer slice.
+static bool preservesAdditionalAffineWARDependences(
+    ArrayRef<Operation *> srcOps, ArrayRef<Operation *> dstOps,
+    unsigned dstLoopDepth, unsigned numCommonLoops,
+    const ComputationSliceState &srcSlice) {
+  FlatAffineValueConstraints sliceConstraints;
+  if (failed(srcSlice.getAsConstraints(&sliceConstraints)))
+    return false;
+
+  for (Operation *srcOp : srcOps) {
+    MemRefAccess srcAccess(srcOp);
+    for (Operation *dstOp : dstOps) {
+      MemRefAccess dstAccess(dstOp);
+      if (srcAccess.memref != dstAccess.memref)
+        continue;
+      // Source stores already participate in the producer-consumer slice.
+      // Only source-read/destination-write dependences are additional to it.
+      if (srcAccess.isStore() || !dstAccess.isStore())
+        continue;
+
+      FlatAffineValueConstraints dependenceConstraints;
+      DependenceResult result = checkMemrefAccessDependence(
+          srcAccess, dstAccess, numCommonLoops + 1, &dependenceConstraints);
+      if (result.value == DependenceResult::NoDependence)
+        continue;
+      if (result.value == DependenceResult::Failure)
+        return false;
+
+      SmallVector<AffineForOp> dstLoops;
+      getAffineForIVs(*dstOp, &dstLoops);
+      SmallVector<AffineForOp> srcLoops;
+      getAffineForIVs(*srcOp, &srcLoops);
+      SmallVector<Value> srcIVs;
+      for (AffineForOp loop : srcLoops)
+        srcIVs.push_back(loop.getInductionVar());
+      SmallVector<Value> dstIVs;
+      for (AffineForOp loop : dstLoops)
+        dstIVs.push_back(loop.getInductionVar());
+      std::optional<bool> hasBackward = hasBackwardDependenceAfterFusion(
+          dependenceConstraints, sliceConstraints, srcIVs, dstIVs,
+          dstLoopDepth, numCommonLoops);
+      if (!hasBackward) {
+        LDBG() << "Could not compare the affine dependence with the fused "
+                  "schedule";
+        return false;
+      }
+      if (*hasBackward) {
+        LDBG() << "Affine WAR dependence becomes backward between " << *srcOp
+               << " and " << *dstOp;
+        return false;
+      }
+    }
+  }
+  return true;
+}
+
 // TODO: This pass performs some computation that is the same for all the depths
 // (e.g., getMaxLoopDepth). Implement a version of this utility that processes
 // all the depths at once or only the legal maximal depth for maximal fusion.
@@ -349,6 +490,13 @@ FusionResult mlir::affine::canFuseLoops(AffineForOp srcForOp,
     return FusionResult::FailIncorrectSlice;
   }
 
+  if (fusionStrategy.getStrategy() == FusionStrategy::ProducerConsumer &&
+      !preservesAdditionalAffineWARDependences(
+          opsA, opsB, dstLoopDepth, numCommonLoops, *srcSlice)) {
+    LDBG() << "Fusion would reverse an additional affine WAR dependence";
+    return FusionResult::FailFusionDependence;
+  }
+
   return FusionResult::Success;
 }
 
diff --git a/mlir/test/Dialect/Affine/loop-fusion-3.mlir b/mlir/test/Dialect/Affine/loop-fusion-3.mlir
index 70d6c82105543..edd802e0a6b44 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-3.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-3.mlir
@@ -1,5 +1,6 @@
 // RUN: mlir-opt -allow-unregistered-dialect %s -pass-pipeline='builtin.module(func.func(affine-loop-fusion))' -split-input-file | FileCheck %s
 // RUN: mlir-opt -allow-unregistered-dialect %s -pass-pipeline='builtin.module(func.func(affine-loop-fusion{maximal}))' -split-input-file | FileCheck %s --check-prefix=MAXIMAL
+// RUN: mlir-opt -allow-unregistered-dialect %s -pass-pipeline='builtin.module(func.func(affine-loop-fusion{mode=producer maximal}))' -split-input-file | FileCheck %s --check-prefix=PC-MAXIMAL
 
 // Part I of fusion tests in  mlir/test/Transforms/loop-fusion.mlir.
 // Part II of fusion tests in mlir/test/Transforms/loop-fusion-2.mlir
@@ -1294,5 +1295,202 @@ func.func @unknown_memref_def_op() {
 }
 func.func private @bar() -> memref<10xf32>
 
+// -----
+
+// A producer-read/consumer-write dependence becomes backward if the producer
+// is sliced pointwise: source iteration 2 would read %a[1] after destination
+// iteration 1 writes it.
+
+// CHECK-LABEL: func.func @producer_read_consumer_write_backward
+// MAXIMAL-LABEL: func.func @producer_read_consumer_write_backward
+// PC-MAXIMAL-LABEL: func.func @producer_read_consumer_write_backward
+func.func @producer_read_consumer_write_backward(
+    %a: memref<3xf32>, %scratch: memref<3xf32>) {
+  affine.for %s = 1 to 3 {
+    %value = affine.load %a[1] : memref<3xf32>
+    affine.store %value, %scratch[%s] : memref<3xf32>
+  }
+  // PC-MAXIMAL: affine.for %[[S:.*]] = 1 to 3
+  // PC-MAXIMAL:   affine.load %{{.*}}[1]
+  // PC-MAXIMAL: affine.for %[[D:.*]] = 1 to 3
+  // PC-MAXIMAL:   affine.load %{{.*}}[%[[D]]]
+  affine.for %d = 1 to 3 {
+    %value = affine.load %scratch[%d] : memref<3xf32>
+    affine.store %value, %a[%d] : memref<3xf32>
+  }
+  return
+}
+
+// -----
+
+// The producer iteration s is scheduled at destination iteration s + 1. The
+// additional dependence also targets s + 1, so it remains forward.
+
+// CHECK-LABEL: func.func @offset_slice_forward_war
+// MAXIMAL-LABEL: func.func @offset_slice_forward_war
+// PC-MAXIMAL-LABEL: func.func @offset_slice_forward_war
+// PC-MAXIMAL-SAME: %[[A:[a-zA-Z0-9_]+]]: memref<10xf32>
+func.func @offset_slice_forward_war(
+    %a: memref<10xf32>, %scratch: memref<10xf32>) {
+  affine.for %s = 0 to 8 {
+    %value = affine.load %a[%s + 1] : memref<10xf32>
+    affine.store %value, %scratch[%s + 1] : memref<10xf32>
+  }
+  // The source slice is cloned into the destination loop.
+  // PC-MAXIMAL: affine.for %{{.*}} = 0 to 8
+  // PC-MAXIMAL: affine.for %[[D:.*]] = 1 to 9
+  // PC-MAXIMAL:   affine.load %[[A]][%{{.*}} + 1]
+  // PC-MAXIMAL:   affine.store %{{.*}}, %[[A]][%[[D]]]
+  affine.for %d = 1 to 9 {
+    %value = affine.load %scratch[%d] : memref<10xf32>
+    affine.store %value, %a[%d] : memref<10xf32>
+  }
+  return
+}
+
+// -----
+
+// Here the same source iteration is scheduled at s + 1, but its read must
+// precede destination iteration s. Fusion would reverse that dependence.
+
+// CHECK-LABEL: func.func @offset_slice_backward_war
+// MAXIMAL-LABEL: func.func @offset_slice_backward_war
+// PC-MAXIMAL-LABEL: func.func @offset_slice_backward_war
+// PC-MAXIMAL-SAME: %[[A:[a-zA-Z0-9_]+]]: memref<10xf32>, %[[SCRATCH:[a-zA-Z0-9_]+]]: memref<10xf32>
+func.func @offset_slice_backward_war(
+    %a: memref<10xf32>, %scratch: memref<10xf32>) {
+  affine.for %s = 0 to 8 {
+    %value = affine.load %a[%s] : memref<10xf32>
+    affine.store %value, %scratch[%s + 1] : memref<10xf32>
+  }
+  // PC-MAXIMAL: affine.for %{{.*}} = 0 to 8
+  // PC-MAXIMAL: affine.for %[[D:.*]] = 1 to 9
+  // PC-MAXIMAL-NOT: affine.load %[[A]]
+  // PC-MAXIMAL:   affine.load %[[SCRATCH]][%[[D]]]
+  // PC-MAXIMAL:   affine.store %{{.*}}, %[[A]][%[[D]]]
+  affine.for %d = 1 to 9 {
+    %value = affine.load %scratch[%d] : memref<10xf32>
+    affine.store %value, %a[%d] : memref<10xf32>
+  }
+  return
+}
+
+// -----
+
+// Verify that common surrounding loops remain parameters of the composed
+// schedule relation and do not prevent a safe pointwise fusion.
+
+// CHECK-LABEL: func.func @common_outer_pointwise_war
+// MAXIMAL-LABEL: func.func @common_outer_pointwise_war
+// PC-MAXIMAL-LABEL: func.func @common_outer_pointwise_war
+// PC-MAXIMAL-SAME: %[[A:[a-zA-Z0-9_]+]]: memref<4x10xf32>
+func.func @common_outer_pointwise_war(
+    %a: memref<4x10xf32>, %scratch: memref<4x10xf32>) {
+  affine.for %outer = 0 to 4 {
+    affine.for %s = 1 to 9 {
+      %value = affine.load %a[%outer, %s] : memref<4x10xf32>
+      affine.store %value, %scratch[%outer, %s] : memref<4x10xf32>
+    }
+    // The source slice is cloned into the destination loop.
+    // PC-MAXIMAL: affine.for %[[OUTER:.*]] = 0 to 4
+    // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 9
+    // PC-MAXIMAL:   affine.for %[[D:.*]] = 1 to 9
+    // PC-MAXIMAL:     affine.load %[[A]][%[[OUTER]], %[[D]]]
+    affine.for %d = 1 to 9 {
+      %value = affine.load %scratch[%outer, %d] : memref<4x10xf32>
+      affine.store %value, %a[%outer, %d] : memref<4x10xf32>
+    }
+  }
+  return
+}
+
+// -----
+
+// CHECK-LABEL: func.func @common_outer_backward_war
+// MAXIMAL-LABEL: func.func @common_outer_backward_war
+// PC-MAXIMAL-LABEL: func.func @common_outer_backward_war
+// PC-MAXIMAL-SAME: %[[A:[a-zA-Z0-9_]+]]: memref<4x10xf32>, %[[SCRATCH:[a-zA-Z0-9_]+]]: memref<4x10xf32>
+func.func @common_outer_backward_war(
+    %a: memref<4x10xf32>, %scratch: memref<4x10xf32>) {
+  affine.for %outer = 0 to 4 {
+    affine.for %s = 1 to 9 {
+      %value = affine.load %a[%outer, 1] : memref<4x10xf32>
+      affine.store %value, %scratch[%outer, %s] : memref<4x10xf32>
+    }
+    // PC-MAXIMAL: affine.for %[[OUTER:.*]] = 0 to 4
+    // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 9
+    // PC-MAXIMAL:   affine.for %[[D:.*]] = 1 to 9
+    // PC-MAXIMAL-NOT: affine.load %[[A]]
+    // PC-MAXIMAL:     affine.load %[[SCRATCH]][%[[OUTER]], %[[D]]]
+    // PC-MAXIMAL:     affine.store %{{.*}}, %[[A]][%[[OUTER]], %[[D]]]
+    affine.for %d = 1 to 9 {
+      %value = affine.load %scratch[%outer, %d] : memref<4x10xf32>
+      affine.store %value, %a[%outer, %d] : memref<4x10xf32>
+    }
+  }
+  return
+}
+
+// -----
+
+// An inner-dimension backward dependence prevents depth-2 fusion, while
+// depth-1 fusion remains legal because each full source inner loop still
+// executes before the corresponding destination inner loop.
+
+// CHECK-LABEL: func.func @two_dimensional_inner_backward_war
+// MAXIMAL-LABEL: func.func @two_dimensional_inner_backward_war
+// PC-MAXIMAL-LABEL: func.func @two_dimensional_inner_backward_war
+func.func @two_dimensional_inner_backward_war(
+    %a: memref<6x6xf32>, %scratch: memref<6x6xf32>) {
+  affine.for %s0 = 1 to 5 {
+    affine.for %s1 = 1 to 5 {
+      %value = affine.load %a[%s0, %s1 - 1] : memref<6x6xf32>
+      affine.store %value, %scratch[%s0, %s1] : memref<6x6xf32>
+    }
+  }
+  // PC-MAXIMAL: affine.for %[[OUTER:.*]] = 1 to 5
+  // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL-NOT: affine.for
+  // PC-MAXIMAL: return
+  affine.for %d0 = 1 to 5 {
+    affine.for %d1 = 1 to 5 {
+      %value = affine.load %scratch[%d0, %d1] : memref<6x6xf32>
+      affine.store %value, %a[%d0, %d1] : memref<6x6xf32>
+    }
+  }
+  return
+}
+
+// -----
+
+// A backward dependence in the outer dimension prevents fusion at every
+// depth, irrespective of the inner-dimension direction.
+
+// CHECK-LABEL: func.func @two_dimensional_outer_backward_war
+// MAXIMAL-LABEL: func.func @two_dimensional_outer_backward_war
+// PC-MAXIMAL-LABEL: func.func @two_dimensional_outer_backward_war
+func.func @two_dimensional_outer_backward_war(
+    %a: memref<6x6xf32>, %scratch: memref<6x6xf32>) {
+  affine.for %s0 = 1 to 5 {
+    affine.for %s1 = 1 to 5 {
+      %value = affine.load %a[%s0 - 1, %s1 + 1] : memref<6x6xf32>
+      affine.store %value, %scratch[%s0, %s1] : memref<6x6xf32>
+    }
+  }
+  // PC-MAXIMAL: affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL: affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL:   affine.for %{{.*}} = 1 to 5
+  // PC-MAXIMAL-NOT: affine.for
+  // PC-MAXIMAL: return
+  affine.for %d0 = 1 to 5 {
+    affine.for %d1 = 1 to 5 {
+      %value = affine.load %scratch[%d0, %d1] : memref<6x6xf32>
+      affine.store %value, %a[%d0, %d1] : memref<6x6xf32>
+    }
+  }
+  return
+}
 
 // Add further tests in mlir/test/Transforms/loop-fusion-4.mlir


From llvmlistbot at llvm.org  Sat Jul 25 10:34:55 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 10:34:55 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve WAR dependences in
	loop fusion (PR #212045)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212045@github.com>
Message-ID: <6a64f3bf.92d2873c.287016.56fe@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @guoriyue :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/212045

From llvmlistbot at llvm.org  Sat Jul 25 10:49:08 2026
From: llvmlistbot at llvm.org (lonely eagle)
Date: Sat, 25 Jul 2026 10:49:08 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add early exit to
 getLowerBound/getUpperBound function (NFC) (PR #212047)
Message-ID: <llvm.org/llvm/llvm-project/pull/212047@github.com>


https://github.com/linuxlonelyeagle created https://github.com/llvm/llvm-project/pull/212047

When an AffineExpr is an AffineConstantExpr, its bound can be directly returned  without computing the lower/upper bounds for all operand values. This avoids  redundant bound evaluations on operands and speeds up static bound computation.

>From 728fbec02a34b0019242fb9f731ea41d05df4971 Mon Sep 17 00:00:00 2001
From: linuxlonelyeagle <2020382038 at qq.com>
Date: Sat, 25 Jul 2026 17:35:07 +0000
Subject: [PATCH] add early exit to getUpperBound/getLowerBound function.

---
 mlir/lib/Dialect/Affine/IR/AffineOps.cpp | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)

diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index defb029865835..fa5266affbd20 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -775,6 +775,9 @@ static std::optional<int64_t> getUpperBound(Value iv) {
 static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -784,9 +787,6 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
-    return constExpr.getValue();
-
   return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
                                constUpperBounds,
                                /*isUpper=*/true);
@@ -798,6 +798,9 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
 static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -807,15 +810,9 @@ static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  std::optional<int64_t> lowerBound;
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr)) {
-    lowerBound = constExpr.getValue();
-  } else {
-    lowerBound = getBoundForAffineExpr(expr, numDims, numSymbols,
-                                       constLowerBounds, constUpperBounds,
-                                       /*isUpper=*/false);
-  }
-  return lowerBound;
+  return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
+                               constUpperBounds,
+                               /*isUpper=*/false);
 }
 
 /// Simplify `expr` while exploiting information from the values in `operands`.


From llvmlistbot at llvm.org  Sat Jul 25 10:49:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 10:49:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add early exit to
 getLowerBound/getUpperBound function (NFC) (PR #212047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212047@github.com>
Message-ID: <6a64f734.510f576d.23298e.4dac@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-affine

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

When an AffineExpr is an AffineConstantExpr, its bound can be directly returned  without computing the lower/upper bounds for all operand values. This avoids  redundant bound evaluations on operands and speeds up static bound computation.

---
Full diff: https://github.com/llvm/llvm-project/pull/212047.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+9-12) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index defb029865835..fa5266affbd20 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -775,6 +775,9 @@ static std::optional<int64_t> getUpperBound(Value iv) {
 static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -784,9 +787,6 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
-    return constExpr.getValue();
-
   return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
                                constUpperBounds,
                                /*isUpper=*/true);
@@ -798,6 +798,9 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
 static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -807,15 +810,9 @@ static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  std::optional<int64_t> lowerBound;
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr)) {
-    lowerBound = constExpr.getValue();
-  } else {
-    lowerBound = getBoundForAffineExpr(expr, numDims, numSymbols,
-                                       constLowerBounds, constUpperBounds,
-                                       /*isUpper=*/false);
-  }
-  return lowerBound;
+  return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
+                               constUpperBounds,
+                               /*isUpper=*/false);
 }
 
 /// Simplify `expr` while exploiting information from the values in `operands`.

``````````

</details>


https://github.com/llvm/llvm-project/pull/212047

From llvmlistbot at llvm.org  Sat Jul 25 10:49:41 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 10:49:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Add early exit to
 getLowerBound/getUpperBound function (NFC) (PR #212047)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212047@github.com>
Message-ID: <6a64f735.f06f4994.3876ce.f665@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: lonely eagle (linuxlonelyeagle)

<details>
<summary>Changes</summary>

When an AffineExpr is an AffineConstantExpr, its bound can be directly returned  without computing the lower/upper bounds for all operand values. This avoids  redundant bound evaluations on operands and speeds up static bound computation.

---
Full diff: https://github.com/llvm/llvm-project/pull/212047.diff


1 Files Affected:

- (modified) mlir/lib/Dialect/Affine/IR/AffineOps.cpp (+9-12) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
index defb029865835..fa5266affbd20 100644
--- a/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
+++ b/mlir/lib/Dialect/Affine/IR/AffineOps.cpp
@@ -775,6 +775,9 @@ static std::optional<int64_t> getUpperBound(Value iv) {
 static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -784,9 +787,6 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
-    return constExpr.getValue();
-
   return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
                                constUpperBounds,
                                /*isUpper=*/true);
@@ -798,6 +798,9 @@ static std::optional<int64_t> getUpperBound(AffineExpr expr, unsigned numDims,
 static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
                                             unsigned numSymbols,
                                             ArrayRef<Value> operands) {
+  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr))
+    return constExpr.getValue();
+
   // Get the constant lower or upper bounds on the operands.
   SmallVector<std::optional<int64_t>> constLowerBounds, constUpperBounds;
   constLowerBounds.reserve(operands.size());
@@ -807,15 +810,9 @@ static std::optional<int64_t> getLowerBound(AffineExpr expr, unsigned numDims,
     constUpperBounds.push_back(getUpperBound(operand));
   }
 
-  std::optional<int64_t> lowerBound;
-  if (auto constExpr = dyn_cast<AffineConstantExpr>(expr)) {
-    lowerBound = constExpr.getValue();
-  } else {
-    lowerBound = getBoundForAffineExpr(expr, numDims, numSymbols,
-                                       constLowerBounds, constUpperBounds,
-                                       /*isUpper=*/false);
-  }
-  return lowerBound;
+  return getBoundForAffineExpr(expr, numDims, numSymbols, constLowerBounds,
+                               constUpperBounds,
+                               /*isUpper=*/false);
 }
 
 /// Simplify `expr` while exploiting information from the values in `operands`.

``````````

</details>


https://github.com/llvm/llvm-project/pull/212047

From llvmlistbot at llvm.org  Sat Jul 25 12:31:29 2026
From: llvmlistbot at llvm.org (Mingfei Guo)
Date: Sat, 25 Jul 2026 12:31:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Preserve WAR dependences in
	loop fusion (PR #212045)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212045@github.com>
Message-ID: <6a650f11.c56fb102.11008c.9182@mx.google.com>


https://github.com/guoriyue edited https://github.com/llvm/llvm-project/pull/212045

From llvmlistbot at llvm.org  Sat Jul 25 16:01:45 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:01:45 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fix folds crashing on
 dynamic-shaped tensors (PR #212072)
Message-ID: <llvm.org/llvm/llvm-project/pull/212072@github.com>


https://github.com/victor-eds created https://github.com/llvm/llvm-project/pull/212072

`Builder::getZeroAttr` asserts when asked to build a constant of a ranked tensor with a dynamic shape (`DenseElementsAttr` requires a static shape). Three self-identity folds/patterns reached it without a guard and aborted on dynamic-shaped operands:

```mlir
arith.xori %x, %x : tensor<?xi32>
arith.subui_extended %x, %x : tensor<?xi32>, tensor<?xi1>
arith.subi (arith.subi %a, %b), %a : tensor<?xi32>   (SubISubILHSRHSLHS)
```

Guard each with a static-shape check (as `arith.subi(x,x)` already does) so they leave the op unfolded instead of crashing. The TableGen pattern gets a new `HasStaticOrScalarShape` constraint.

>From 909d76d0a3bfce57bd1e8c88b0cf590b0af1935e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Sat, 25 Jul 2026 15:52:26 -0700
Subject: [PATCH] [mlir][arith] Fix folds crashing on dynamic-shaped tensors

Builder::getZeroAttr asserts when asked to build a constant of a
ranked tensor with a dynamic shape (DenseElementsAttr requires a static
shape). Three self-identity folds/patterns reached it without a guard and
aborted on dynamic-shaped operands:

  arith.xori %x, %x : tensor<?xi32>
  arith.subui_extended %x, %x : tensor<?xi32>, tensor<?xi1>
  arith.subi (arith.subi %a, %b), %a : tensor<?xi32>   (SubISubILHSRHSLHS)

Guard each with a static-shape check (as arith.subi(x,x) already does) so
they leave the op unfolded instead of crashing. The TableGen pattern gets
a new HasStaticOrScalarShape constraint.
---
 .../Dialect/Arith/IR/ArithCanonicalization.td | 12 ++++++-
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        | 14 ++++++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 34 +++++++++++++++++++
 3 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index bfb8e5cc34218..1a10744e7fbe1 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -15,6 +15,15 @@ include "mlir/Dialect/Arith/IR/ArithOps.td"
 // Create zero attribute of type matching the argument's type.
 def GetZeroAttr : NativeCodeCall<"$_builder.getZeroAttr($0.getType())">;
 
+// A value whose type is a scalar or has a static shape, i.e. one for which a
+// splat constant can be built. Guards patterns that materialize a constant via
+// GetZeroAttr, which asserts on a dynamically-shaped type.
+def HasStaticOrScalarShape :
+    Constraint<CPred<"!::llvm::isa<::mlir::ShapedType>($0.getType()) || "
+                     "::llvm::cast<::mlir::ShapedType>($0.getType())"
+                     ".hasStaticShape()">,
+               "type is a scalar or has a static shape">;
+
 // Add two integer attributes and create a new one with the result.
 def AddIntAttrs : NativeCodeCall<"addIntegerAttrs($_builder, $0, $1, $2)">;
 
@@ -193,7 +202,8 @@ def SubILHSSubConstantLHS :
 def SubISubILHSRHSLHS :
     Pat<(Arith_SubIOp:$res (Arith_SubIOp $x, $y, $ovf1), $x, $ovf2),
         (Arith_SubIOp (Arith_ConstantOp (GetZeroAttr $y)), $y,
-            (MergeOverflow $ovf1, $ovf2))>;
+            (MergeOverflow $ovf1, $ovf2)),
+        [(HasStaticOrScalarShape $y)]>;
 
 //===----------------------------------------------------------------------===//
 // MulSIExtendedOp
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..7f138b30eb8dd 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -539,6 +539,11 @@ arith::SubUIExtendedOp::fold(FoldAdaptor adaptor,
 
   // subui_extended(x, x) -> 0, false
   if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getDiff().getType());
+    if (shapedType && !shapedType.hasStaticShape())
+      return failure();
     Builder builder(getContext());
     auto zeroDiff = builder.getZeroAttr(getDiff().getType());
     auto falseValue = builder.getZeroAttr(borrowTy);
@@ -1153,8 +1158,13 @@ OpFoldResult arith::XOrIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_Zero()))
     return getLhs();
   /// xor(x, x) -> 0
-  if (getLhs() == getRhs())
-    return Builder(getContext()).getZeroAttr(getType());
+  if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getType());
+    if (!shapedType || shapedType.hasStaticShape())
+      return Builder(getContext()).getZeroAttr(getType());
+  }
   /// xor(xor(x, a), a) -> x
   /// xor(xor(a, x), a) -> x
   if (arith::XOrIOp prev = getLhs().getDefiningOp<arith::XOrIOp>()) {
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..d48939bd3ab32 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -4263,3 +4263,37 @@ func.func @convertf_fold_f8() -> f8E5M2 {
   return %result : f8E5M2
 }
 
+
+// -----
+
+// Self-identity folds and patterns must not build a constant of a dynamic
+// shape (which would assert); they must leave the op unfolded.
+// CHECK-LABEL: func @xori_self_dynamic
+//       CHECK:   arith.xori
+func.func @xori_self_dynamic(%arg0 : tensor<?xi32>) -> tensor<?xi32> {
+  %0 = arith.xori %arg0, %arg0 : tensor<?xi32>
+  return %0 : tensor<?xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func @subui_extended_self_dynamic
+//       CHECK:   arith.subui_extended
+func.func @subui_extended_self_dynamic(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi1>) {
+  %low, %bo = arith.subui_extended %arg0, %arg0
+      : tensor<?xi32>, tensor<?xi1>
+  return %low, %bo : tensor<?xi32>, tensor<?xi1>
+}
+
+// -----
+
+// CHECK-LABEL: func @subi_subi_lhs_rhs_lhs_dynamic
+//       CHECK:   arith.subi
+//       CHECK:   arith.subi
+func.func @subi_subi_lhs_rhs_lhs_dynamic(%a : tensor<?xi32>, %b : tensor<?xi32>)
+    -> tensor<?xi32> {
+  %0 = arith.subi %a, %b : tensor<?xi32>
+  %1 = arith.subi %0, %a : tensor<?xi32>
+  return %1 : tensor<?xi32>
+}


From llvmlistbot at llvm.org  Sat Jul 25 16:02:24 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 16:02:24 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fix folds crashing on
 dynamic-shaped tensors (PR #212072)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212072@github.com>
Message-ID: <6a654080.6c5720b3.2e9b28.fece@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Victor Perez (victor-eds)

<details>
<summary>Changes</summary>

`Builder::getZeroAttr` asserts when asked to build a constant of a ranked tensor with a dynamic shape (`DenseElementsAttr` requires a static shape). Three self-identity folds/patterns reached it without a guard and aborted on dynamic-shaped operands:

```mlir
arith.xori %x, %x : tensor&lt;?xi32&gt;
arith.subui_extended %x, %x : tensor&lt;?xi32&gt;, tensor&lt;?xi1&gt;
arith.subi (arith.subi %a, %b), %a : tensor&lt;?xi32&gt;   (SubISubILHSRHSLHS)
```

Guard each with a static-shape check (as `arith.subi(x,x)` already does) so they leave the op unfolded instead of crashing. The TableGen pattern gets a new `HasStaticOrScalarShape` constraint.

---
Full diff: https://github.com/llvm/llvm-project/pull/212072.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td (+11-1) 
- (modified) mlir/lib/Dialect/Arith/IR/ArithOps.cpp (+12-2) 
- (modified) mlir/test/Dialect/Arith/canonicalize.mlir (+34) 


``````````diff
diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index bfb8e5cc34218..1a10744e7fbe1 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -15,6 +15,15 @@ include "mlir/Dialect/Arith/IR/ArithOps.td"
 // Create zero attribute of type matching the argument's type.
 def GetZeroAttr : NativeCodeCall<"$_builder.getZeroAttr($0.getType())">;
 
+// A value whose type is a scalar or has a static shape, i.e. one for which a
+// splat constant can be built. Guards patterns that materialize a constant via
+// GetZeroAttr, which asserts on a dynamically-shaped type.
+def HasStaticOrScalarShape :
+    Constraint<CPred<"!::llvm::isa<::mlir::ShapedType>($0.getType()) || "
+                     "::llvm::cast<::mlir::ShapedType>($0.getType())"
+                     ".hasStaticShape()">,
+               "type is a scalar or has a static shape">;
+
 // Add two integer attributes and create a new one with the result.
 def AddIntAttrs : NativeCodeCall<"addIntegerAttrs($_builder, $0, $1, $2)">;
 
@@ -193,7 +202,8 @@ def SubILHSSubConstantLHS :
 def SubISubILHSRHSLHS :
     Pat<(Arith_SubIOp:$res (Arith_SubIOp $x, $y, $ovf1), $x, $ovf2),
         (Arith_SubIOp (Arith_ConstantOp (GetZeroAttr $y)), $y,
-            (MergeOverflow $ovf1, $ovf2))>;
+            (MergeOverflow $ovf1, $ovf2)),
+        [(HasStaticOrScalarShape $y)]>;
 
 //===----------------------------------------------------------------------===//
 // MulSIExtendedOp
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..7f138b30eb8dd 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -539,6 +539,11 @@ arith::SubUIExtendedOp::fold(FoldAdaptor adaptor,
 
   // subui_extended(x, x) -> 0, false
   if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getDiff().getType());
+    if (shapedType && !shapedType.hasStaticShape())
+      return failure();
     Builder builder(getContext());
     auto zeroDiff = builder.getZeroAttr(getDiff().getType());
     auto falseValue = builder.getZeroAttr(borrowTy);
@@ -1153,8 +1158,13 @@ OpFoldResult arith::XOrIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_Zero()))
     return getLhs();
   /// xor(x, x) -> 0
-  if (getLhs() == getRhs())
-    return Builder(getContext()).getZeroAttr(getType());
+  if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getType());
+    if (!shapedType || shapedType.hasStaticShape())
+      return Builder(getContext()).getZeroAttr(getType());
+  }
   /// xor(xor(x, a), a) -> x
   /// xor(xor(a, x), a) -> x
   if (arith::XOrIOp prev = getLhs().getDefiningOp<arith::XOrIOp>()) {
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..d48939bd3ab32 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -4263,3 +4263,37 @@ func.func @convertf_fold_f8() -> f8E5M2 {
   return %result : f8E5M2
 }
 
+
+// -----
+
+// Self-identity folds and patterns must not build a constant of a dynamic
+// shape (which would assert); they must leave the op unfolded.
+// CHECK-LABEL: func @xori_self_dynamic
+//       CHECK:   arith.xori
+func.func @xori_self_dynamic(%arg0 : tensor<?xi32>) -> tensor<?xi32> {
+  %0 = arith.xori %arg0, %arg0 : tensor<?xi32>
+  return %0 : tensor<?xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func @subui_extended_self_dynamic
+//       CHECK:   arith.subui_extended
+func.func @subui_extended_self_dynamic(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi1>) {
+  %low, %bo = arith.subui_extended %arg0, %arg0
+      : tensor<?xi32>, tensor<?xi1>
+  return %low, %bo : tensor<?xi32>, tensor<?xi1>
+}
+
+// -----
+
+// CHECK-LABEL: func @subi_subi_lhs_rhs_lhs_dynamic
+//       CHECK:   arith.subi
+//       CHECK:   arith.subi
+func.func @subi_subi_lhs_rhs_lhs_dynamic(%a : tensor<?xi32>, %b : tensor<?xi32>)
+    -> tensor<?xi32> {
+  %0 = arith.subi %a, %b : tensor<?xi32>
+  %1 = arith.subi %0, %a : tensor<?xi32>
+  return %1 : tensor<?xi32>
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/212072

From llvmlistbot at llvm.org  Sat Jul 25 16:02:59 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:02:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
Message-ID: <llvm.org/llvm/llvm-project/pull/212073@github.com>


https://github.com/victor-eds created https://github.com/llvm/llvm-project/pull/212073

`tosa.mul` and `tosa.intdiv` fold constant splat operands by building a `DenseElementsAttr` of the result type, which asserts when that type has a dynamic shape. When the operands are constants but the result type is dynamic, folding aborted. Guard `mulBinaryFolder` and the `IntDivOp` splat path on a static result shape, matching the other fold paths in these ops.

>From bad184c9383cf81929249f7f69cc569130a87603 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?= <victorperez at fb.com>
Date: Sat, 25 Jul 2026 15:52:27 -0700
Subject: [PATCH] [mlir][tosa] Fix mul/intdiv folds crashing on dynamic-shaped
 results

tosa.mul and tosa.intdiv fold constant splat operands by building a
DenseElementsAttr of the result type, which asserts when that type has a
dynamic shape. When the operands are constants but the result type is
dynamic, folding aborted. Guard mulBinaryFolder and the IntDivOp splat
path on a static result shape, matching the other fold paths in these ops.
---
 .../Dialect/Tosa/IR/TosaCanonicalizations.cpp |  5 +++-
 mlir/test/Dialect/Tosa/canonicalize.mlir      | 25 +++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 19f35757bdd83..8937875df7fc5 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1796,7 +1796,7 @@ OpFoldResult IntDivOp::fold(FoldAdaptor adaptor) {
   }
 
   if (rhsAttr && lhsAttr && rhsAttr.isSplat() && lhsAttr.isSplat() &&
-      llvm::isa<IntegerType>(resultETy)) {
+      llvm::isa<IntegerType>(resultETy) && resultTy.hasStaticShape()) {
     APInt l = lhsAttr.getSplatValue<APInt>();
     APInt r = rhsAttr.getSplatValue<APInt>();
     if (!r.isZero()) {
@@ -1841,6 +1841,9 @@ std::optional<APInt> mulInt(APInt lhs, APInt rhs, int32_t shift,
 
 DenseElementsAttr mulBinaryFolder(DenseElementsAttr lhs, DenseElementsAttr rhs,
                                   RankedTensorType ty, int32_t shift) {
+  // A constant result can only be built for a statically-shaped type.
+  if (!ty.hasStaticShape())
+    return {};
   if (rhs && lhs && rhs.isSplat() && lhs.isSplat()) {
     if (llvm::isa<IntegerType>(ty.getElementType())) {
       APInt l = lhs.getSplatValue<APInt>();
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 585589ac1405a..7c9dd260ca497 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1441,6 +1441,31 @@ func.func @no_fold_mul_result_exceeds_i32() -> tensor<i32> {
 
 // -----
 
+// A dynamically-shaped result cannot be a constant; folding must bail instead
+// of building a DenseElementsAttr of a non-static shape (which would assert).
+// CHECK-LABEL: @no_fold_mul_dynamic_result
+// CHECK: tosa.mul
+func.func @no_fold_mul_dynamic_result() -> tensor<?xf32> {
+    %0 = "tosa.const"() <{values = dense<2.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %1 = "tosa.const"() <{values = dense<3.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %2 = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
+    %3 = tosa.mul %0, %1, %2 : (tensor<4xf32>, tensor<4xf32>, tensor<1xi8>) -> tensor<?xf32>
+    return %3 : tensor<?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @no_fold_intdiv_dynamic_result
+// CHECK: tosa.intdiv
+func.func @no_fold_intdiv_dynamic_result() -> tensor<?xi32> {
+    %0 = "tosa.const"() <{values = dense<6> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %1 = "tosa.const"() <{values = dense<2> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %2 = tosa.intdiv %0, %1 : (tensor<4xi32>, tensor<4xi32>) -> tensor<?xi32>
+    return %2 : tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_fold_i1_to_i32_cast
 // CHECK: %[[OUT:.*]] = "tosa.const"() <{values = dense<1> : tensor<i32>}> : () -> tensor<i32>
 // CHECK: return %[[OUT]] : tensor<i32>


From llvmlistbot at llvm.org  Sat Jul 25 16:03:34 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 16:03:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212073@github.com>
Message-ID: <6a6540c6.e5907290.788a7.dec6@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-tosa

Author: Victor Perez (victor-eds)

<details>
<summary>Changes</summary>

`tosa.mul` and `tosa.intdiv` fold constant splat operands by building a `DenseElementsAttr` of the result type, which asserts when that type has a dynamic shape. When the operands are constants but the result type is dynamic, folding aborted. Guard `mulBinaryFolder` and the `IntDivOp` splat path on a static result shape, matching the other fold paths in these ops.

---
Full diff: https://github.com/llvm/llvm-project/pull/212073.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp (+4-1) 
- (modified) mlir/test/Dialect/Tosa/canonicalize.mlir (+25) 


``````````diff
diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 19f35757bdd83..8937875df7fc5 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1796,7 +1796,7 @@ OpFoldResult IntDivOp::fold(FoldAdaptor adaptor) {
   }
 
   if (rhsAttr && lhsAttr && rhsAttr.isSplat() && lhsAttr.isSplat() &&
-      llvm::isa<IntegerType>(resultETy)) {
+      llvm::isa<IntegerType>(resultETy) && resultTy.hasStaticShape()) {
     APInt l = lhsAttr.getSplatValue<APInt>();
     APInt r = rhsAttr.getSplatValue<APInt>();
     if (!r.isZero()) {
@@ -1841,6 +1841,9 @@ std::optional<APInt> mulInt(APInt lhs, APInt rhs, int32_t shift,
 
 DenseElementsAttr mulBinaryFolder(DenseElementsAttr lhs, DenseElementsAttr rhs,
                                   RankedTensorType ty, int32_t shift) {
+  // A constant result can only be built for a statically-shaped type.
+  if (!ty.hasStaticShape())
+    return {};
   if (rhs && lhs && rhs.isSplat() && lhs.isSplat()) {
     if (llvm::isa<IntegerType>(ty.getElementType())) {
       APInt l = lhs.getSplatValue<APInt>();
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 585589ac1405a..7c9dd260ca497 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1441,6 +1441,31 @@ func.func @no_fold_mul_result_exceeds_i32() -> tensor<i32> {
 
 // -----
 
+// A dynamically-shaped result cannot be a constant; folding must bail instead
+// of building a DenseElementsAttr of a non-static shape (which would assert).
+// CHECK-LABEL: @no_fold_mul_dynamic_result
+// CHECK: tosa.mul
+func.func @no_fold_mul_dynamic_result() -> tensor<?xf32> {
+    %0 = "tosa.const"() <{values = dense<2.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %1 = "tosa.const"() <{values = dense<3.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %2 = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
+    %3 = tosa.mul %0, %1, %2 : (tensor<4xf32>, tensor<4xf32>, tensor<1xi8>) -> tensor<?xf32>
+    return %3 : tensor<?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @no_fold_intdiv_dynamic_result
+// CHECK: tosa.intdiv
+func.func @no_fold_intdiv_dynamic_result() -> tensor<?xi32> {
+    %0 = "tosa.const"() <{values = dense<6> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %1 = "tosa.const"() <{values = dense<2> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %2 = tosa.intdiv %0, %1 : (tensor<4xi32>, tensor<4xi32>) -> tensor<?xi32>
+    return %2 : tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_fold_i1_to_i32_cast
 // CHECK: %[[OUT:.*]] = "tosa.const"() <{values = dense<1> : tensor<i32>}> : () -> tensor<i32>
 // CHECK: return %[[OUT]] : tensor<i32>

``````````

</details>


https://github.com/llvm/llvm-project/pull/212073

From llvmlistbot at llvm.org  Sat Jul 25 16:20:13 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:20:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212073@github.com>
Message-ID: <6a6544ad.af4834b4.1d4518.7463@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212073

>From 4a9a471ce1920a0ce8f231d7301a22569c672580 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:53 -0700
Subject: [PATCH] [mlir][tosa] Fix mul/intdiv folds crashing on dynamic-shaped
 results
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

tosa.mul and tosa.intdiv fold constant splat operands by building a
DenseElementsAttr of the result type, which asserts when that type has a
dynamic shape. When the operands are constants but the result type is
dynamic, folding aborted. Guard mulBinaryFolder and the IntDivOp splat
path on a static result shape, matching the other fold paths in these ops.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 .../Dialect/Tosa/IR/TosaCanonicalizations.cpp |  5 +++-
 mlir/test/Dialect/Tosa/canonicalize.mlir      | 25 +++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 19f35757bdd83..8937875df7fc5 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1796,7 +1796,7 @@ OpFoldResult IntDivOp::fold(FoldAdaptor adaptor) {
   }
 
   if (rhsAttr && lhsAttr && rhsAttr.isSplat() && lhsAttr.isSplat() &&
-      llvm::isa<IntegerType>(resultETy)) {
+      llvm::isa<IntegerType>(resultETy) && resultTy.hasStaticShape()) {
     APInt l = lhsAttr.getSplatValue<APInt>();
     APInt r = rhsAttr.getSplatValue<APInt>();
     if (!r.isZero()) {
@@ -1841,6 +1841,9 @@ std::optional<APInt> mulInt(APInt lhs, APInt rhs, int32_t shift,
 
 DenseElementsAttr mulBinaryFolder(DenseElementsAttr lhs, DenseElementsAttr rhs,
                                   RankedTensorType ty, int32_t shift) {
+  // A constant result can only be built for a statically-shaped type.
+  if (!ty.hasStaticShape())
+    return {};
   if (rhs && lhs && rhs.isSplat() && lhs.isSplat()) {
     if (llvm::isa<IntegerType>(ty.getElementType())) {
       APInt l = lhs.getSplatValue<APInt>();
diff --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 585589ac1405a..7c9dd260ca497 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1441,6 +1441,31 @@ func.func @no_fold_mul_result_exceeds_i32() -> tensor<i32> {
 
 // -----
 
+// A dynamically-shaped result cannot be a constant; folding must bail instead
+// of building a DenseElementsAttr of a non-static shape (which would assert).
+// CHECK-LABEL: @no_fold_mul_dynamic_result
+// CHECK: tosa.mul
+func.func @no_fold_mul_dynamic_result() -> tensor<?xf32> {
+    %0 = "tosa.const"() <{values = dense<2.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %1 = "tosa.const"() <{values = dense<3.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %2 = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
+    %3 = tosa.mul %0, %1, %2 : (tensor<4xf32>, tensor<4xf32>, tensor<1xi8>) -> tensor<?xf32>
+    return %3 : tensor<?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @no_fold_intdiv_dynamic_result
+// CHECK: tosa.intdiv
+func.func @no_fold_intdiv_dynamic_result() -> tensor<?xi32> {
+    %0 = "tosa.const"() <{values = dense<6> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %1 = "tosa.const"() <{values = dense<2> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %2 = tosa.intdiv %0, %1 : (tensor<4xi32>, tensor<4xi32>) -> tensor<?xi32>
+    return %2 : tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_fold_i1_to_i32_cast
 // CHECK: %[[OUT:.*]] = "tosa.const"() <{values = dense<1> : tensor<i32>}> : () -> tensor<i32>
 // CHECK: return %[[OUT]] : tensor<i32>


From llvmlistbot at llvm.org  Sat Jul 25 16:20:13 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:20:13 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fix folds crashing on
 dynamic-shaped tensors (PR #212072)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212072@github.com>
Message-ID: <6a6544ad.1e1ac41d.8ad93.86b3@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212072

>From 8d14817e0c194fcba07a8c9ce5a1f5d7aa58dca8 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:51 -0700
Subject: [PATCH] [mlir][arith] Fix folds crashing on dynamic-shaped tensors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Builder::getZeroAttr asserts when asked to build a constant of a
ranked tensor with a dynamic shape (DenseElementsAttr requires a static
shape). Three self-identity folds/patterns reached it without a guard and
aborted on dynamic-shaped operands:

  arith.xori %x, %x : tensor<?xi32>
  arith.subui_extended %x, %x : tensor<?xi32>, tensor<?xi1>
  arith.subi (arith.subi %a, %b), %a : tensor<?xi32>   (SubISubILHSRHSLHS)

Guard each with a static-shape check (as arith.subi(x,x) already does) so
they leave the op unfolded instead of crashing. The TableGen pattern gets
a new HasStaticOrScalarShape constraint.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 .../Dialect/Arith/IR/ArithCanonicalization.td | 12 ++++++-
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        | 14 ++++++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 34 +++++++++++++++++++
 3 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index bfb8e5cc34218..1a10744e7fbe1 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -15,6 +15,15 @@ include "mlir/Dialect/Arith/IR/ArithOps.td"
 // Create zero attribute of type matching the argument's type.
 def GetZeroAttr : NativeCodeCall<"$_builder.getZeroAttr($0.getType())">;
 
+// A value whose type is a scalar or has a static shape, i.e. one for which a
+// splat constant can be built. Guards patterns that materialize a constant via
+// GetZeroAttr, which asserts on a dynamically-shaped type.
+def HasStaticOrScalarShape :
+    Constraint<CPred<"!::llvm::isa<::mlir::ShapedType>($0.getType()) || "
+                     "::llvm::cast<::mlir::ShapedType>($0.getType())"
+                     ".hasStaticShape()">,
+               "type is a scalar or has a static shape">;
+
 // Add two integer attributes and create a new one with the result.
 def AddIntAttrs : NativeCodeCall<"addIntegerAttrs($_builder, $0, $1, $2)">;
 
@@ -193,7 +202,8 @@ def SubILHSSubConstantLHS :
 def SubISubILHSRHSLHS :
     Pat<(Arith_SubIOp:$res (Arith_SubIOp $x, $y, $ovf1), $x, $ovf2),
         (Arith_SubIOp (Arith_ConstantOp (GetZeroAttr $y)), $y,
-            (MergeOverflow $ovf1, $ovf2))>;
+            (MergeOverflow $ovf1, $ovf2)),
+        [(HasStaticOrScalarShape $y)]>;
 
 //===----------------------------------------------------------------------===//
 // MulSIExtendedOp
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..7f138b30eb8dd 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -539,6 +539,11 @@ arith::SubUIExtendedOp::fold(FoldAdaptor adaptor,
 
   // subui_extended(x, x) -> 0, false
   if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getDiff().getType());
+    if (shapedType && !shapedType.hasStaticShape())
+      return failure();
     Builder builder(getContext());
     auto zeroDiff = builder.getZeroAttr(getDiff().getType());
     auto falseValue = builder.getZeroAttr(borrowTy);
@@ -1153,8 +1158,13 @@ OpFoldResult arith::XOrIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_Zero()))
     return getLhs();
   /// xor(x, x) -> 0
-  if (getLhs() == getRhs())
-    return Builder(getContext()).getZeroAttr(getType());
+  if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getType());
+    if (!shapedType || shapedType.hasStaticShape())
+      return Builder(getContext()).getZeroAttr(getType());
+  }
   /// xor(xor(x, a), a) -> x
   /// xor(xor(a, x), a) -> x
   if (arith::XOrIOp prev = getLhs().getDefiningOp<arith::XOrIOp>()) {
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..d48939bd3ab32 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -4263,3 +4263,37 @@ func.func @convertf_fold_f8() -> f8E5M2 {
   return %result : f8E5M2
 }
 
+
+// -----
+
+// Self-identity folds and patterns must not build a constant of a dynamic
+// shape (which would assert); they must leave the op unfolded.
+// CHECK-LABEL: func @xori_self_dynamic
+//       CHECK:   arith.xori
+func.func @xori_self_dynamic(%arg0 : tensor<?xi32>) -> tensor<?xi32> {
+  %0 = arith.xori %arg0, %arg0 : tensor<?xi32>
+  return %0 : tensor<?xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func @subui_extended_self_dynamic
+//       CHECK:   arith.subui_extended
+func.func @subui_extended_self_dynamic(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi1>) {
+  %low, %bo = arith.subui_extended %arg0, %arg0
+      : tensor<?xi32>, tensor<?xi1>
+  return %low, %bo : tensor<?xi32>, tensor<?xi1>
+}
+
+// -----
+
+// CHECK-LABEL: func @subi_subi_lhs_rhs_lhs_dynamic
+//       CHECK:   arith.subi
+//       CHECK:   arith.subi
+func.func @subi_subi_lhs_rhs_lhs_dynamic(%a : tensor<?xi32>, %b : tensor<?xi32>)
+    -> tensor<?xi32> {
+  %0 = arith.subi %a, %b : tensor<?xi32>
+  %1 = arith.subi %0, %a : tensor<?xi32>
+  return %1 : tensor<?xi32>
+}


From llvmlistbot at llvm.org  Sat Jul 25 16:27:18 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:27:18 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
Message-ID: <llvm.org/llvm/llvm-project/pull/212074@github.com>


https://github.com/victor-eds created https://github.com/llvm/llvm-project/pull/212074

Add value-preserving folds mirroring LLVM's InstructionSimplify for the integer division and remainder ops:

```
divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
remui/remsi(0, x) -> 0
remui/remsi(x, x) -> 0
```

Division or remainder by zero is undefined behaviour, so refining the self and zero-dividend cases to a constant is always valid; no overflow flags are required. The folds return a scalar or splat constant and bail out on shaped types with a dynamic shape.

----

Code generated by Claude Code.

>From e219872f40fc5fbc1ed12582365e49ef778627fb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:43 -0700
Subject: [PATCH] [mlir][arith] Fold trivial integer division and remainder
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add value-preserving folds mirroring LLVM's InstructionSimplify for the
integer division and remainder ops:

  divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
  divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
  remui/remsi(0, x) -> 0
  remui/remsi(x, x) -> 0

Division or remainder by zero is undefined behaviour, so refining the
self and zero-dividend cases to a constant is always valid; no overflow
flags are required. The folds return a scalar or splat constant and bail
out on shaped types with a dynamic shape.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp    |  67 +++++++++++-
 mlir/test/Dialect/Arith/canonicalize.mlir | 119 ++++++++++++++++++++++
 2 files changed, 184 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..d6420c3a222fa 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -795,6 +808,14 @@ OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -835,6 +856,14 @@ OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -891,6 +920,14 @@ OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -921,6 +958,14 @@ OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -990,6 +1035,14 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1011,7 +1064,12 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1038,7 +1096,12 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true


From llvmlistbot at llvm.org  Sat Jul 25 16:27:56 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 16:27:56 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a65467c.b4a331f0.1c45b5.e321@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-arith

Author: Victor Perez (victor-eds)

<details>
<summary>Changes</summary>

Add value-preserving folds mirroring LLVM's InstructionSimplify for the integer division and remainder ops:

```
divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -&gt; 0
divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -&gt; 1
remui/remsi(0, x) -&gt; 0
remui/remsi(x, x) -&gt; 0
```

Division or remainder by zero is undefined behaviour, so refining the self and zero-dividend cases to a constant is always valid; no overflow flags are required. The folds return a scalar or splat constant and bail out on shaped types with a dynamic shape.

----

Code generated by Claude Code.

---
Full diff: https://github.com/llvm/llvm-project/pull/212074.diff


2 Files Affected:

- (modified) mlir/lib/Dialect/Arith/IR/ArithOps.cpp (+65-2) 
- (modified) mlir/test/Dialect/Arith/canonicalize.mlir (+119) 


``````````diff
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..d6420c3a222fa 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -795,6 +808,14 @@ OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -835,6 +856,14 @@ OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -891,6 +920,14 @@ OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -921,6 +958,14 @@ OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -990,6 +1035,14 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1011,7 +1064,12 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1038,7 +1096,12 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true

``````````

</details>


https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 16:48:33 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:48:33 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a654b51.4987c784.13f2de.b154@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212074

>From 0523c1048ff114bf923b4113bf7cc227b26ff449 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:43 -0700
Subject: [PATCH] [mlir][arith] Fold trivial integer division and remainder
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add value-preserving folds mirroring LLVM's InstructionSimplify for the
integer division and remainder ops:

  divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
  divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
  remui/remsi(0, x) -> 0
  remui/remsi(x, x) -> 0

Division or remainder by zero is undefined behaviour, so refining the
self and zero-dividend cases to a constant is always valid; no overflow
flags are required. The folds return a scalar or splat constant and bail
out on shaped types with a dynamic shape.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        |  67 +++++++++-
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir |  24 ++--
 .../arith-to-spirv-unsupported.mlir           |   4 +-
 .../ArithToSPIRV/arith-to-spirv.mlir          |  22 ++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 119 ++++++++++++++++++
 5 files changed, 209 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..d6420c3a222fa 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -795,6 +808,14 @@ OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -835,6 +856,14 @@ OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -891,6 +920,14 @@ OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -921,6 +958,14 @@ OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -990,6 +1035,14 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1011,7 +1064,12 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1038,7 +1096,12 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d22677fde774a 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -11,14 +11,14 @@ func.func @vector_ops(%arg0: vector<4xf32>, %arg1: vector<4xi1>, %arg2: vector<4
   %0 = arith.constant dense<42.> : vector<4xf32>
 // CHECK-NEXT:  %1 = llvm.fadd %arg0, %0 : vector<4xf32>
   %1 = arith.addf %arg0, %0 : vector<4xf32>
-// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg2 : vector<4xi64>
-  %3 = arith.divsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg2 : vector<4xi64>
-  %4 = arith.divui %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg2 : vector<4xi64>
-  %5 = arith.remsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg2 : vector<4xi64>
-  %6 = arith.remui %arg2, %arg2 : vector<4xi64>
+// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg3 : vector<4xi64>
+  %3 = arith.divsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg3 : vector<4xi64>
+  %4 = arith.divui %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg3 : vector<4xi64>
+  %5 = arith.remsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg3 : vector<4xi64>
+  %6 = arith.remui %arg2, %arg3 : vector<4xi64>
 // CHECK-NEXT:  %6 = llvm.fdiv %arg0, %0 : vector<4xf32>
   %7 = arith.divf %arg0, %0 : vector<4xf32>
 // CHECK-NEXT:  %7 = llvm.frem %arg0, %0 : vector<4xf32>
@@ -780,16 +780,16 @@ func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
 }
 
 // CHECK-LABEL: @ceildivui
-// CHECK-SAME: %[[ARG0:.*]]: i32) -> i32
-func.func @ceildivui(%arg0 : i32) -> i32 {
+// CHECK-SAME: %[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32) -> i32
+func.func @ceildivui(%arg0 : i32, %arg1 : i32) -> i32 {
 // CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
 // CHECK: %[[CMP0:.*]] = llvm.icmp "eq" %[[ARG0]], %[[CST0]] : i32
 // CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
 // CHECK: %[[SUB0:.*]] = llvm.sub %[[ARG0]], %[[CST1]] : i32
-// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG0]] : i32
+// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG1]] : i32
 // CHECK: %[[ADD0:.*]] = llvm.add %[[DIV0]], %[[CST1]] : i32
 // CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[CST0]], %[[ADD0]] : i1, i32
-  %0 = arith.ceildivui %arg0, %arg0 : i32
+  %0 = arith.ceildivui %arg0, %arg1 : i32
   return %0: i32
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
index 92b587d5ed1e4..9ecebb502630c 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
@@ -53,10 +53,10 @@ func.func @unsupported_2x2elem_vector(%arg0: vector<2x2xi32>) {
 
 // -----
 
-func.func @int_vector4_invalid(%arg0: vector<2xi16>) {
+func.func @int_vector4_invalid(%arg0: vector<2xi16>, %arg1: vector<2xi16>) {
   // expected-error @+2 {{failed to legalize operation 'arith.divui'}}
   // expected-error @+1 {{bitwidth emulation is not implemented yet on unsigned op}}
-  %0 = arith.divui %arg0, %arg0: vector<2xi16>
+  %0 = arith.divui %arg0, %arg1: vector<2xi16>
   return
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index 5c9966055b540..4cef06992727b 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -198,18 +198,18 @@ func.func @float32_binary_scalar(%lhs: f32, %rhs: f32) {
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
 // CHECK-LABEL: @index_vector
-func.func @index_vector(%arg0: vector<4xindex>) {
+func.func @index_vector(%arg0: vector<4xindex>, %arg1: vector<4xindex>) {
   // CHECK: spirv.UMod %{{.*}}, %{{.*}}: vector<4xi32>
-  %0 = arith.remui %arg0, %arg0: vector<4xindex>
+  %0 = arith.remui %arg0, %arg1: vector<4xindex>
   return
 }
 
@@ -1522,11 +1522,11 @@ func.func @float32_maxnumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>)
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
@@ -1570,11 +1570,11 @@ module attributes {
 } {
 
 // CHECK-LABEL: @int_vector23
-func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<3xi16>) {
+func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<3xi16>, %arg3: vector<3xi16>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi32>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<3xi32>
-  %1 = arith.divsi %arg1, %arg1: vector<3xi16>
+  %1 = arith.divsi %arg2, %arg3: vector<3xi16>
   return
 }
 
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true


From llvmlistbot at llvm.org  Sat Jul 25 16:50:16 2026
From: llvmlistbot at llvm.org (Ivan Butygin)
Date: Sat, 25 Jul 2026 16:50:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a654bb8.507fc713.3a4929.ab2a@mx.google.com>


Hardcode84 wrote:

Folding `div(x, 0) -> poison` may be more profitable, but then we have an ordering problem between `div(x, x)` and `div(x, 0)` foldings. Can you check alive2 and llvm proper if 1. poison folding is actually allowed, 2. how they handle ordering.

https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 16:53:37 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Sat, 25 Jul 2026 16:53:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a654c81.f1c1923d.1253fb.50e1@mx.google.com>


kuhar wrote:

+1, this needs alive2 proofs

https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 16:55:43 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:55:43 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a654cff.492e23e4.3ce411.3857@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/210422

>From ce60603f9ea6a2f7960467fb3fae5f4147f3bde4 Mon Sep 17 00:00:00 2001
From: Victor Perez <victor.pc.upm at gmail.com>
Date: Fri, 17 Jul 2026 11:52:35 -0700
Subject: [PATCH] [mlir] Support human-readable float literals in the textual
 IR

Add LLVM-style textual floating-point literals to the MLIR assembly
format, both parsed and printed:

  +inf / -inf
  +qnan / -qnan
  +nan(0x<payload>) / +snan(0x<payload>)   (and their negatives)
  0x1.8p3                                   (C-style hexadecimal float)

The sign is mandatory for inf/NaN, so bare `inf`/`nan` remain ordinary
identifiers. C-style hex floats are distinguished from the existing
bit-pattern form by a fractional part and/or a binary exponent; the plain
`0x..` form is still parsed as a bit pattern.

Float literal values are now built directly in the target type's
semantics: Token::getFloatingPointValue takes the target fltSemantics
(and sign) and returns std::optional<APFloat> via
APFloat::convertFromString, instead of first converting to double. This
preserves NaN payloads and fixes a precision bug for types wider than
double: e.g. `0.1 : f128` was previously stored as the double
approximation of 0.1 widened to f128 rather than the correctly-rounded
f128 value. convertFromString accepts inf/nan/snan spellings but not
"qnan" (built via APFloat::getQNaN) nor a leading '+', so the sign is
handled explicitly.

The NaN text form follows the type's NaN encoding, not the type family.
Types with the IEEE encoding keep the full form (payload and
quiet/signaling bit). A type whose NaN carries no payload prints
`+nan`/`-nan`, and a type whose only NaN is negative prints only `-nan`.
Today these are the NanOnly fp8 formats (e.g. f8E4M3FN and the *FNUZ
types). The lexer therefore accepts `nan` and `snan` with an optional
payload, so the bare `+nan`/`-nan` round-trips. This differs from LLVM's
writeAPFloatInternal, which prints the IEEE form for every type; that is
wrong for these encodings, which today exist only in MLIR. APFloat is
not changed.

A special value that the target type cannot represent is now a parse
error rather than an APFloat abort. The parser rejects infinity, NaN,
or a negative value when the type lacks the matching property (e.g.
`+inf` for a type without infinity, or `-...` for an unsigned type).
For the NanOnly types it also rejects a payload, a signaling NaN, or a
positive NaN on a type whose only NaN is negative, instead of silently
re-encoding the value.

The printer emits the human-readable form by default. The
--mlir-print-float-special-literals-as-hex flag restores the legacy
output, where infinities and NaNs print as a hexadecimal bit pattern.
The parser accepts both forms regardless of the flag. The flag is wired
into every tool that registers the AsmPrinter command-line options
(mlir-opt, mlir-translate, and flang's bbc).

Also thread the float literal's location into the FloatAttr parse
diagnostics so "not valid for specified type" points at the constant
rather than the following token.

Signed-off-by: Victor Perez <victor.pc.upm at gmail.com>
---
 flang/test/Transforms/simplifyintrinsics.fir  |   4 +-
 mlir/docs/LangRef.md                          |  21 +++
 mlir/include/mlir/IR/OperationSupport.h       |  11 ++
 mlir/lib/AsmParser/AttributeParser.cpp        |  21 ++-
 mlir/lib/AsmParser/Lexer.cpp                  |  77 ++++++++
 mlir/lib/AsmParser/Lexer.h                    |   1 +
 mlir/lib/AsmParser/Parser.cpp                 |  11 +-
 mlir/lib/AsmParser/Token.cpp                  |  77 +++++++-
 mlir/lib/AsmParser/Token.h                    |  11 +-
 mlir/lib/IR/AsmPrinter.cpp                    | 101 ++++++++--
 .../8-bit-float-saturation-ocp.mlir           |   8 +-
 .../ArithToAMDGPU/8-bit-float-saturation.mlir |   8 +-
 .../ComplexToLLVM/complex-range-option.mlir   |   4 +-
 .../ComplexToSPIRV/complex-to-spirv.mlir      |   2 +-
 .../complex-range-option.mlir                 |   4 +-
 .../convert-to-standard.mlir                  |  44 ++---
 .../GPUToNVVM/wmma-ops-to-nvvm.mlir           |   8 +-
 .../TosaToLinalg/tosa-to-linalg.mlir          |  16 +-
 .../vector-reduction-to-llvm.mlir             |   6 +-
 .../SuperVectorize/vectorize_reduction.mlir   |  12 +-
 mlir/test/Dialect/Arith/emulate-wide-int.mlir |   2 +-
 mlir/test/Dialect/Builtin/Bytecode/attrs.mlir |  10 +-
 .../Builtin/Bytecode/builtin_fixed.mlir       |   4 +-
 .../Builtin/Bytecode/builtin_fixed_0.mlirbc   | Bin 5470 -> 5470 bytes
 mlir/test/Dialect/Complex/canonicalize.mlir   |   6 +-
 .../Linalg/transform-op-decompose.mlir        |   2 +-
 .../Linalg/transform-op-split-reduction.mlir  |   8 +-
 .../Linalg/transform-tile-reduction.mlir      |   2 +-
 mlir/test/Dialect/Math/canonicalize.mlir      |   6 +-
 mlir/test/Dialect/Math/expand-math.mlir       |   8 +-
 .../Math/polynomial-approximation.mlir        |  18 +-
 mlir/test/Dialect/SPIRV/IR/structure-ops.mlir |   2 +-
 .../Tosa/constant-reciprocal-fold.mlir        |  14 +-
 mlir/test/Dialect/Tosa/invalid.mlir           |   4 +-
 mlir/test/Dialect/XeGPU/propagate-layout.mlir |   2 +-
 mlir/test/IR/array-of-attr.mlir               |   2 +-
 mlir/test/IR/custom-float-attr-roundtrip.mlir |   6 +-
 mlir/test/IR/float-literals-hex.mlir          |  61 ++++++
 mlir/test/IR/float-literals.mlir              | 159 ++++++++++++++++
 mlir/test/IR/invalid-builtin-attributes.mlir  | 176 ++++++++++++++++++
 mlir/test/IR/parser.mlir                      |  66 +++----
 41 files changed, 823 insertions(+), 182 deletions(-)
 create mode 100644 mlir/test/IR/float-literals-hex.mlir
 create mode 100644 mlir/test/IR/float-literals.mlir

diff --git a/flang/test/Transforms/simplifyintrinsics.fir b/flang/test/Transforms/simplifyintrinsics.fir
index 1cfda08e39694..585083df690c5 100644
--- a/flang/test/Transforms/simplifyintrinsics.fir
+++ b/flang/test/Transforms/simplifyintrinsics.fir
@@ -2005,7 +2005,7 @@ func.func @_QPtestminloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0x7FF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant +inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
@@ -2581,7 +2581,7 @@ func.func @_QPtestmaxloc_works1d_scalarmask_f64(%arg0: !fir.ref<!fir.array<10xf6
 // CHECK:             %[[FLAG_SET:.*]] = arith.constant 1 : i32
 // CHECK:             %[[FLAG_EMPTY:.*]] = arith.constant 0 : i32
 // CHECK:             fir.store %[[FLAG_EMPTY]] to %[[FLAG_ALLOC]] : !fir.ref<i32>
-// CHECK:             %[[MAX:.*]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK:             %[[MAX:.*]] = arith.constant -inf : f64
 // CHECK:             %[[C_INDEX1:.*]] = arith.constant 1 : index
 // CHECK:             %[[DIM_INDEX:.*]] = arith.constant 0 : index
 // CHECK:             %[[DIMS:.*]]:3 = fir.box_dims %[[BOX_INARR]], %[[DIM_INDEX]] : (!fir.box<!fir.array<?xf64>>, index) -> (index, index, index)
diff --git a/mlir/docs/LangRef.md b/mlir/docs/LangRef.md
index fea34c093418d..741e0c6745de2 100644
--- a/mlir/docs/LangRef.md
+++ b/mlir/docs/LangRef.md
@@ -174,9 +174,30 @@ integer-literal ::= decimal-literal | hexadecimal-literal
 decimal-literal ::= digit+
 hexadecimal-literal ::= `0x` hex_digit+
 float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+                | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
+                | float-special-literal
+float-special-literal ::= [-+] (`inf` | `qnan` | `s`? `nan` (`(` `0x` hex_digit+ `)`)?)
 string-literal  ::= `"` [^"\n\f\v\r]* `"`   TODO: define escaping rules
 ```
 
+Special floating point values are spelled with a mandatory sign: `+inf`/`-inf`
+for infinities, `+qnan`/`-qnan` for the preferred quiet NaN, and
+`+nan(0x..)`/`+snan(0x..)` for quiet/signaling NaNs with an explicit hexadecimal
+payload. A payload and the quiet/signaling distinction are only accepted for
+types whose NaN encoding supports them (the standard IEEE encoding). A type
+whose NaN carries no payload spells it as a plain `+nan`/`-nan`, and a type
+whose only NaN is negative accepts only `-nan`. The parser rejects a payload,
+an `snan`, or a positive NaN that the type's NaN encoding cannot represent,
+rather than silently re-encoding the value. Finite values may also be written
+as C-style hexadecimal floats (`0x1.8p3`). The `0x` bit-pattern form without a
+fractional part or exponent (e.g. `0x7F800000`) is instead interpreted as the
+raw bit pattern of the target type.
+
+The `--mlir-print-float-special-literals-as-hex` flag makes the printer emit infinities
+and NaNs as a `0x` bit pattern instead of the human-readable form. This restores
+the legacy output for backwards compatibility. The parser accepts both forms
+regardless of the flag.
+
 Not listed here, but MLIR does support comments. They use standard BCPL syntax,
 starting with a `//` and going until the end of the line.
 
diff --git a/mlir/include/mlir/IR/OperationSupport.h b/mlir/include/mlir/IR/OperationSupport.h
index 942b55e8f7a76..64f9ecfa2f1d7 100644
--- a/mlir/include/mlir/IR/OperationSupport.h
+++ b/mlir/include/mlir/IR/OperationSupport.h
@@ -1239,6 +1239,10 @@ class OpPrintingFlags {
   /// Print SSA IDs using their NameLoc, if provided, as prefix.
   OpPrintingFlags &printNameLocAsPrefix(bool enable = true);
 
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern
+  /// instead of the human-readable form. Restores the legacy output.
+  OpPrintingFlags &printFloatSpecialLiteralsAsHex(bool enable = true);
+
   /// Return if the given ElementsAttr should be elided.
   bool shouldElideElementsAttr(ElementsAttr attr) const;
 
@@ -1282,6 +1286,10 @@ class OpPrintingFlags {
   /// IDs
   bool shouldUseNameLocAsPrefix() const;
 
+  /// Return if special float literals (inf, NaN) should print as a hexadecimal
+  /// bit pattern.
+  bool shouldPrintFloatSpecialLiteralsAsHex() const;
+
 private:
   /// Elide large elements attributes if the number of elements is larger than
   /// the upper limit.
@@ -1318,6 +1326,9 @@ class OpPrintingFlags {
 
   /// Print SSA IDs using NameLocs as prefixes
   bool useNameLocAsPrefix : 1;
+
+  /// Print special float literals (inf, NaN) as a hexadecimal bit pattern.
+  bool printFloatSpecialLiteralsAsHexFlag : 1;
 };
 
 //===----------------------------------------------------------------------===//
diff --git a/mlir/lib/AsmParser/AttributeParser.cpp b/mlir/lib/AsmParser/AttributeParser.cpp
index ca8e4ae2cecbc..5f7ae4aa8bdc6 100644
--- a/mlir/lib/AsmParser/AttributeParser.cpp
+++ b/mlir/lib/AsmParser/AttributeParser.cpp
@@ -339,9 +339,8 @@ ParseResult Parser::parseAttributeDict(NamedAttrList &attributes) {
 
 /// Parse a float attribute.
 Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
-  auto val = getToken().getFloatingPointValue();
-  if (!val)
-    return (emitError("floating point value too large for attribute"), nullptr);
+  Token tok = getToken();
+  SMLoc loc = tok.getLoc();
   consumeToken(Token::floatliteral);
   if (!type) {
     // Default to F64 when no type is specified.
@@ -350,10 +349,18 @@ Attribute Parser::parseFloatAttr(Type type, bool isNegative) {
     else if (!(type = parseType()))
       return nullptr;
   }
-  if (!isa<FloatType>(type))
-    return (emitError("floating point value not valid for specified type"),
-            nullptr);
-  return FloatAttr::get(type, isNegative ? -*val : *val);
+  auto floatType = dyn_cast<FloatType>(type);
+  if (!floatType) {
+    emitError(loc, "floating point value not valid for specified type");
+    return nullptr;
+  }
+
+  std::optional<APFloat> result =
+      tok.getFloatingPointValue(isNegative, floatType.getFloatSemantics(),
+                                [&] { return emitError(loc); });
+  if (!result)
+    return nullptr;
+  return FloatAttr::get(floatType, *result);
 }
 
 /// Construct an APint from a parsed value, a known attribute type and
diff --git a/mlir/lib/AsmParser/Lexer.cpp b/mlir/lib/AsmParser/Lexer.cpp
index 161dcb4ca27c0..d499c8babc696 100644
--- a/mlir/lib/AsmParser/Lexer.cpp
+++ b/mlir/lib/AsmParser/Lexer.cpp
@@ -147,6 +147,9 @@ Token Lexer::lexToken() {
       return formToken(Token::equal, tokStart);
 
     case '+':
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::plus, tokStart);
     case '*':
       return formToken(Token::star, tokStart);
@@ -155,6 +158,9 @@ Token Lexer::lexToken() {
         ++curPtr;
         return formToken(Token::arrow, tokStart);
       }
+      if (std::optional<Token> tok = lexSpecialFloatLiteral(
+              StringRef(tokStart, curBuffer.end() - tokStart)))
+        return *tok;
       return formToken(Token::minus, tokStart);
 
     case '?':
@@ -308,10 +314,59 @@ Token Lexer::lexEllipsis(const char *tokStart) {
   return formToken(Token::ellipsis, tokStart);
 }
 
+/// Try to lex a signed special float literal (inf/NaN) from `fromSign`, which
+/// starts at the already-consumed sign. On success returns a `floatliteral`
+/// (or `error`) token and advances `curPtr`, else std::nullopt (curPtr intact).
+///
+///   float-special ::= [-+] (`inf` | `qnan` | `s`? `nan` (`(0x` hex_digit+
+///   `)`)?)
+///
+std::optional<Token> Lexer::lexSpecialFloatLiteral(StringRef fromSign) {
+  const char *tokStart = fromSign.data();
+  StringRef afterSign = fromSign.drop_front();
+  StringRef keyword = afterSign.take_while(llvm::isAlpha);
+  StringRef rest = afterSign.drop_front(keyword.size());
+
+  // True when `rest` continues the keyword into a longer identifier, so the
+  // keyword is only a prefix (e.g. `infinity`, `nano`) and not a special float.
+  auto isIdentifierContinuation = [](StringRef rest) {
+    return !rest.empty() &&
+           (llvm::isAlnum(rest.front()) || rest.front() == '_' ||
+            rest.front() == '$' || rest.front() == '.');
+  };
+
+  if (keyword == "inf" || keyword == "qnan") {
+    if (isIdentifierContinuation(rest))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  // `nan` / `snan`, with an optional `(0x..)` payload; APFloat validates later.
+  if (keyword == "nan" || keyword == "snan") {
+    if (rest.consume_front("(")) {
+      StringRef payload = rest.take_until([](char c) { return c == ')'; });
+      if (payload.end() == rest.end()) {
+        curPtr = afterSign.end();
+        return emitError(tokStart, "expected ')' in NaN literal");
+      }
+      curPtr = payload.end() + 1;
+      return formToken(Token::floatliteral, tokStart);
+    }
+    if (isIdentifierContinuation(rest))
+      return std::nullopt;
+    curPtr = rest.data();
+    return formToken(Token::floatliteral, tokStart);
+  }
+
+  return std::nullopt;
+}
+
 /// Lex a number literal.
 ///
 ///   integer-literal ::= digit+ | `0x` hex_digit+
 ///   float-literal ::= [-+]?[0-9]+[.][0-9]*([eE][-+]?[0-9]+)?
+///                    | `0x` hex_digit+ (`.` hex_digit*)? [pP] [-+]? digit+
 ///
 Token Lexer::lexNumber(const char *tokStart) {
   assert(isdigit(curPtr[-1]));
@@ -327,6 +382,28 @@ Token Lexer::lexNumber(const char *tokStart) {
     while (isxdigit(*curPtr))
       ++curPtr;
 
+    // C-style hex float (`0x1.fp13`): a '.'/'p' distinguishes it from the
+    // bit-pattern form; the binary exponent is mandatory.
+    if (*curPtr == '.' || *curPtr == 'p' || *curPtr == 'P') {
+      if (*curPtr == '.') {
+        ++curPtr;
+        while (isxdigit(*curPtr))
+          ++curPtr;
+      }
+      if (*curPtr != 'p' && *curPtr != 'P')
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      ++curPtr;
+      if (*curPtr == '-' || *curPtr == '+')
+        ++curPtr;
+      if (!isdigit(static_cast<unsigned char>(*curPtr)))
+        return emitError(tokStart, "expected binary exponent in hexadecimal "
+                                   "floating point literal");
+      while (isdigit(*curPtr))
+        ++curPtr;
+      return formToken(Token::floatliteral, tokStart);
+    }
+
     return formToken(Token::integer, tokStart);
   }
 
diff --git a/mlir/lib/AsmParser/Lexer.h b/mlir/lib/AsmParser/Lexer.h
index 670444eb1f5b4..802ebcb7a8393 100644
--- a/mlir/lib/AsmParser/Lexer.h
+++ b/mlir/lib/AsmParser/Lexer.h
@@ -60,6 +60,7 @@ class Lexer {
   Token lexBareIdentifierOrKeyword(const char *tokStart);
   Token lexEllipsis(const char *tokStart);
   Token lexNumber(const char *tokStart);
+  std::optional<Token> lexSpecialFloatLiteral(StringRef fromSign);
   Token lexPrefixedIdentifier(const char *tokStart);
   Token lexString(const char *tokStart);
 
diff --git a/mlir/lib/AsmParser/Parser.cpp b/mlir/lib/AsmParser/Parser.cpp
index 952d7e460c6e2..d9173a5308eee 100644
--- a/mlir/lib/AsmParser/Parser.cpp
+++ b/mlir/lib/AsmParser/Parser.cpp
@@ -401,14 +401,9 @@ ParseResult Parser::parseFloatFromLiteral(std::optional<APFloat> &result,
                                           const llvm::fltSemantics &semantics) {
   // Check for a floating point value.
   if (tok.is(Token::floatliteral)) {
-    auto val = tok.getFloatingPointValue();
-    if (!val)
-      return emitError(tok.getLoc()) << "floating point value too large";
-
-    result.emplace(isNegative ? -*val : *val);
-    bool unused;
-    result->convert(semantics, APFloat::rmNearestTiesToEven, &unused);
-    return success();
+    result = tok.getFloatingPointValue(isNegative, semantics,
+                                       [&] { return emitError(tok.getLoc()); });
+    return failure(!result);
   }
 
   // Check for a hexadecimal float value.
diff --git a/mlir/lib/AsmParser/Token.cpp b/mlir/lib/AsmParser/Token.cpp
index 5bf25bac6a5c6..bad2325a8d822 100644
--- a/mlir/lib/AsmParser/Token.cpp
+++ b/mlir/lib/AsmParser/Token.cpp
@@ -11,8 +11,10 @@
 //===----------------------------------------------------------------------===//
 
 #include "Token.h"
+#include "mlir/IR/Diagnostics.h"
 #include "mlir/Support/LLVM.h"
 #include "llvm/ADT/StringExtras.h"
+#include "llvm/Support/Error.h"
 #include "llvm/Support/ErrorHandling.h"
 #include <cassert>
 #include <cstdint>
@@ -51,12 +53,77 @@ std::optional<uint64_t> Token::getUInt64IntegerValue(StringRef spelling) {
   return result;
 }
 
-/// For a floatliteral, return its value as a double. Return std::nullopt if the
-/// value underflows or overflows.
-std::optional<double> Token::getFloatingPointValue() const {
-  double result = 0;
-  if (spelling.getAsDouble(result))
+/// For a floatliteral token, build its value in `semantics`, combining any sign
+/// folded into the spelling with `isNegative`. On failure, emits a diagnostic
+/// through `emitError` and returns std::nullopt.
+std::optional<APFloat> Token::getFloatingPointValue(
+    bool isNegative, const llvm::fltSemantics &semantics,
+    function_ref<InFlightDiagnostic()> emitError) const {
+  // The sign is either a preceding '-' token (isNegative) or folded into an
+  // inf/NaN spelling; both at once (e.g. `-+inf`) is invalid. Drop a leading
+  // '+' since convertFromString rejects it on NaN forms.
+  StringRef str = spelling;
+  if (isNegative && (str.starts_with("+") || str.starts_with("-"))) {
+    emitError() << "floating point literal has more than one sign";
     return std::nullopt;
+  }
+  bool isNeg = isNegative || str.consume_front("-");
+  str.consume_front("+");
+
+  // Reject values the type cannot represent; APFloat would otherwise abort.
+  if (isNeg && !APFloat::semanticsHasSignedRepr(semantics)) {
+    emitError() << "floating point type does not support negative values";
+    return std::nullopt;
+  }
+  if (str == "inf" && !APFloat::semanticsHasInf(semantics)) {
+    emitError() << "floating point type does not support infinity";
+    return std::nullopt;
+  }
+  bool wantsNaN =
+      str == "qnan" || str.starts_with("nan") || str.starts_with("snan");
+  if (wantsNaN && !APFloat::semanticsHasNaN(semantics)) {
+    emitError() << "floating point type does not support NaN";
+    return std::nullopt;
+  }
+
+  // NanOnly types have a single NaN per representable sign, with no payload and
+  // no quiet/signaling distinction, and the negative-zero encoding has only a
+  // negative NaN. Reject spellings the type cannot represent instead of
+  // silently re-encoding them.
+  if (wantsNaN &&
+      semantics.nonFiniteBehavior == llvm::fltNonfiniteBehavior::NanOnly) {
+    if (str.starts_with("snan")) {
+      emitError() << "floating point type does not support signaling NaN";
+      return std::nullopt;
+    }
+    if (str.contains('(')) {
+      emitError() << "floating point type does not support NaN payload";
+      return std::nullopt;
+    }
+    // The negative-zero encoding has only a negative NaN; a positive one is not
+    // representable and would silently yield the negative bit pattern.
+    if (!isNeg && semantics.nanEncoding == llvm::fltNanEncoding::NegativeZero) {
+      emitError() << "floating point type only supports negative NaN";
+      return std::nullopt;
+    }
+  }
+
+  // convertFromString does not accept "qnan".
+  if (str == "qnan")
+    return APFloat::getQNaN(semantics, isNeg);
+
+  // Build in the target semantics to preserve NaN payloads; overflow/underflow
+  // are tolerated (inf/zero).
+  APFloat result(semantics);
+  llvm::Expected<APFloat::opStatus> status =
+      result.convertFromString(str, APFloat::rmNearestTiesToEven);
+  if (!status) {
+    llvm::consumeError(status.takeError());
+    emitError() << "invalid floating point literal";
+    return std::nullopt;
+  }
+  if (isNeg)
+    result.changeSign();
   return result;
 }
 
diff --git a/mlir/lib/AsmParser/Token.h b/mlir/lib/AsmParser/Token.h
index 9cae80fd51da0..9193465af9419 100644
--- a/mlir/lib/AsmParser/Token.h
+++ b/mlir/lib/AsmParser/Token.h
@@ -10,11 +10,13 @@
 #define MLIR_LIB_PARSER_TOKEN_H
 
 #include "mlir/Support/LLVM.h"
+#include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/SMLoc.h"
 #include <optional>
 
 namespace mlir {
+class InFlightDiagnostic;
 
 /// This represents a token in the MLIR syntax.
 class Token {
@@ -84,9 +86,12 @@ class Token {
     return getUInt64IntegerValue(getSpelling());
   }
 
-  /// For a floatliteral token, return its value as a double. Returns
-  /// std::nullopt in the case of underflow or overflow.
-  std::optional<double> getFloatingPointValue() const;
+  /// For a floatliteral token, build its value in `semantics`, combining any
+  /// sign folded into the spelling with `isNegative`. On failure, emits a
+  /// diagnostic through `emitError` and returns std::nullopt.
+  std::optional<APFloat>
+  getFloatingPointValue(bool isNegative, const llvm::fltSemantics &semantics,
+                        function_ref<InFlightDiagnostic()> emitError) const;
 
   /// For an inttype token, return its bitwidth.
   std::optional<unsigned> getIntTypeBitwidth() const;
diff --git a/mlir/lib/IR/AsmPrinter.cpp b/mlir/lib/IR/AsmPrinter.cpp
index ca5c2d2a88ee5..1cb040f3b2fac 100644
--- a/mlir/lib/IR/AsmPrinter.cpp
+++ b/mlir/lib/IR/AsmPrinter.cpp
@@ -201,6 +201,11 @@ struct AsmPrinterOptions {
   llvm::cl::opt<bool> useNameLocAsPrefix{
       "mlir-use-nameloc-as-prefix", llvm::cl::init(false),
       llvm::cl::desc("Print SSA IDs using NameLocs as prefixes")};
+
+  llvm::cl::opt<bool> printFloatSpecialLiteralsAsHexOpt{
+      "mlir-print-float-special-literals-as-hex", llvm::cl::init(false),
+      llvm::cl::desc("Print special float literals (inf, NaN) as a hexadecimal "
+                     "bit pattern instead of the human-readable form")};
 };
 } // namespace
 
@@ -219,7 +224,7 @@ OpPrintingFlags::OpPrintingFlags()
       printGenericOpFormFlag(false), skipRegionsFlag(false),
       assumeVerifiedFlag(false), printLocalScope(false),
       printValueUsersFlag(false), printUniqueSSAIDsFlag(false),
-      useNameLocAsPrefix(false) {
+      useNameLocAsPrefix(false), printFloatSpecialLiteralsAsHexFlag(false) {
   // Initialize based upon command line options, if they are available.
   if (!clOptions.isConstructed())
     return;
@@ -239,6 +244,8 @@ OpPrintingFlags::OpPrintingFlags()
   printValueUsersFlag = clOptions->printValueUsers;
   printUniqueSSAIDsFlag = clOptions->printUniqueSSAIDs;
   useNameLocAsPrefix = clOptions->useNameLocAsPrefix;
+  printFloatSpecialLiteralsAsHexFlag =
+      clOptions->printFloatSpecialLiteralsAsHexOpt;
 }
 
 /// Enable the elision of large elements attributes, by printing a '...'
@@ -331,6 +338,11 @@ OpPrintingFlags &OpPrintingFlags::printNameLocAsPrefix(bool enable) {
   return *this;
 }
 
+OpPrintingFlags &OpPrintingFlags::printFloatSpecialLiteralsAsHex(bool enable) {
+  printFloatSpecialLiteralsAsHexFlag = enable;
+  return *this;
+}
+
 /// Return the size limit for printing large ElementsAttr.
 std::optional<int64_t> OpPrintingFlags::getLargeElementsAttrLimit() const {
   return elementsAttrElementLimit;
@@ -387,6 +399,11 @@ bool OpPrintingFlags::shouldUseNameLocAsPrefix() const {
   return useNameLocAsPrefix;
 }
 
+/// Return if special float literals should print as a hexadecimal bit pattern.
+bool OpPrintingFlags::shouldPrintFloatSpecialLiteralsAsHex() const {
+  return printFloatSpecialLiteralsAsHexFlag;
+}
+
 //===----------------------------------------------------------------------===//
 // NewLineCounter
 //===----------------------------------------------------------------------===//
@@ -418,6 +435,9 @@ class AsmPrinter::Impl {
   /// Returns the output stream of the printer.
   raw_ostream &getStream() { return os; }
 
+  /// Returns the printing flags of the printer.
+  const OpPrintingFlags &getPrinterFlags() const { return printerFlags; }
+
   /// Print a newline and indent the printer to the start of the current
   /// operation/attribute/type.
   /// Note: For attributes and types this method should only be used in
@@ -2256,14 +2276,11 @@ void AsmPrinter::Impl::printLocationInternal(LocationAttr loc, bool pretty,
 /// Print a floating point value in a way that the parser will be able to
 /// round-trip losslessly.
 static void printFloatValue(const APFloat &apValue, raw_ostream &os,
-                            bool *printedHex = nullptr) {
-  // We would like to output the FP constant value in exponential notation,
-  // but we cannot do this if doing so will lose precision.  Check here to
-  // make sure that we only output it in exponential format if we can parse
-  // the value back and get the same value.
-  bool isInf = apValue.isInfinity();
-  bool isNaN = apValue.isNaN();
-  if (!isInf && !isNaN) {
+                            bool *printedHex = nullptr,
+                            bool printFloatSpecialLiteralsAsHex = false) {
+  // Finite values print as decimal; this is the only form whose type may be
+  // elided.
+  if (!apValue.isInfinity() && !apValue.isNaN()) {
     SmallString<128> strValue;
     apValue.toString(strValue, /*FormatPrecision=*/6, /*FormatMaxPadding=*/0,
                      /*TruncateZero=*/false);
@@ -2295,14 +2312,51 @@ static void printFloatValue(const APFloat &apValue, raw_ostream &os,
     }
   }
 
-  // Print special values in hexadecimal format. The sign bit should be included
-  // in the literal.
+  // The rest (inf, NaN, hex bit-pattern) are typeless; keep the type.
   if (printedHex)
     *printedHex = true;
+
+  // The legacy form prints inf/NaN as a hexadecimal bit pattern too.
+  if (!printFloatSpecialLiteralsAsHex && apValue.isInfinity()) {
+    os << (apValue.isNegative() ? "-inf" : "+inf");
+    return;
+  }
+
+  if (!printFloatSpecialLiteralsAsHex && apValue.isNaN()) {
+    os << (apValue.isNegative() ? '-' : '+');
+    // NanOnly types have a single NaN per representable sign, with no payload
+    // and no quiet/signaling bit; their text form is just `nan`. Only the IEEE
+    // model carries a payload and a quiet/signaling distinction.
+    if (apValue.getSemantics().nonFiniteBehavior ==
+        llvm::fltNonfiniteBehavior::NanOnly) {
+      os << "nan";
+      return;
+    }
+    // Only IEEE-encoded types reach here (NanOnly types returned above), so the
+    // precision is >= 3 and the payload below is at least one bit wide.
+    APInt payload = apValue.getNaNPayload();
+    // Preferred quiet NaN = only the quiet (sign-mask) bit set.
+    if (payload.isSignMask()) {
+      os << "qnan";
+      return;
+    }
+    if (apValue.isSignaling())
+      os << 's';
+    os << "nan(";
+    // Drop the quiet/signaling bit; trim leading zeros.
+    payload.clearBit(payload.getBitWidth() - 1);
+    SmallVector<char, 16> str;
+    payload.trunc(std::max(payload.getActiveBits(), 1u))
+        .toString(str, /*Radix=*/16, /*Signed=*/false,
+                  /*formatAsCLiteral=*/true);
+    os << str << ')';
+    return;
+  }
+
+  // Hexadecimal bit pattern.
   SmallVector<char, 16> str;
-  APInt apInt = apValue.bitcastToAPInt();
-  apInt.toString(str, /*Radix=*/16, /*Signed=*/false,
-                 /*formatAsCLiteral=*/true);
+  apValue.bitcastToAPInt().toString(str, /*Radix=*/16, /*Signed=*/false,
+                                    /*formatAsCLiteral=*/true);
   os << str;
 }
 
@@ -2472,7 +2526,8 @@ void AsmPrinter::Impl::printAttributeImpl(Attribute attr,
 
   } else if (auto floatAttr = llvm::dyn_cast<FloatAttr>(attr)) {
     bool printedHex = false;
-    printFloatValue(floatAttr.getValue(), os, &printedHex);
+    printFloatValue(floatAttr.getValue(), os, &printedHex,
+                    printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
 
     // FloatAttr elides the type if F64.
     if (typeElision == AttrTypeElision::May && floatAttr.getType().isF64() &&
@@ -2702,9 +2757,11 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
       printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
         auto complexValue = *(valueIt + index);
         os << "(";
-        printFloatValue(complexValue.real(), os);
+        printFloatValue(complexValue.real(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ",";
-        printFloatValue(complexValue.imag(), os);
+        printFloatValue(complexValue.imag(), os, /*printedHex=*/nullptr,
+                        printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
         os << ")";
       });
     }
@@ -2717,7 +2774,8 @@ void AsmPrinter::Impl::printDenseTypedElementsAttr(DenseTypedElementsAttr attr,
     assert(llvm::isa<FloatType>(elementType) && "unexpected element type");
     auto valueIt = attr.value_begin<APFloat>();
     printDenseElementsAttrImpl(attr.isSplat(), type, os, [&](unsigned index) {
-      printFloatValue(*(valueIt + index), os);
+      printFloatValue(*(valueIt + index), os, /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     });
   }
 }
@@ -2768,7 +2826,8 @@ void AsmPrinter::Impl::printDenseArrayAttr(DenseArrayAttr attr) {
       printDenseIntElement(value, getStream(), type);
     } else {
       APFloat fltVal(llvm::cast<FloatType>(type).getFloatSemantics(), value);
-      printFloatValue(fltVal, getStream());
+      printFloatValue(fltVal, getStream(), /*printedHex=*/nullptr,
+                      printerFlags.shouldPrintFloatSpecialLiteralsAsHex());
     }
   };
   llvm::interleaveComma(llvm::seq<unsigned>(0, attr.size()), getStream(),
@@ -3063,7 +3122,9 @@ void AsmPrinter::decreaseIndent() {
 /// Print the given floating point value in a stablized form.
 void AsmPrinter::printFloat(const APFloat &value) {
   assert(impl && "expected AsmPrinter::printFloat to be overriden");
-  printFloatValue(value, impl->getStream());
+  printFloatValue(
+      value, impl->getStream(), /*printedHex=*/nullptr,
+      impl->getPrinterFlags().shouldPrintFloatSpecialLiteralsAsHex());
 }
 
 void AsmPrinter::printType(Type type) {
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
index 2df5f2fa1965f..f66a7194d2afb 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation-ocp.mlir
@@ -10,8 +10,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -37,8 +37,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2 {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FN> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-4.480000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<4.480000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
index 07a428566d488..f36b35b913c9b 100644
--- a/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
+++ b/mlir/test/Conversion/ArithToAMDGPU/8-bit-float-saturation.mlir
@@ -6,8 +6,8 @@
 // CHECK-SAME: ([[V:%.+]]: f16)
 // CHECK-DAG: [[CMin:%.+]] = arith.constant -5.734400e+04 : f16
 // CHECK-DAG: [[CMax:%.+]] = arith.constant 5.734400e+04 : f16
-// CHECK-DAG: [[CInf:%.+]] = arith.constant 0x7C00 : f16
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant 0xFC00 : f16
+// CHECK-DAG: [[CInf:%.+]] = arith.constant +inf : f16
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant -inf : f16
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
@@ -33,8 +33,8 @@ func.func @scalar_trunc(%v: f16) -> f8E5M2FNUZ {
 // CHECK-SAME: ([[V:%.+]]: vector<2xf32>) -> vector<2xf8E4M3FNUZ> {
 // CHECK-DAG: [[CMin:%.+]] = arith.constant dense<-2.400000e+02> : vector<2xf32>
 // CHECK-DAG: [[CMax:%.+]] = arith.constant dense<2.400000e+02> : vector<2xf32>
-// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<0x7F800000> : vector<2xf32>
-// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<0xFF800000> : vector<2xf32>
+// CHECK-DAG: [[CInf:%.+]] = arith.constant dense<+inf> : vector<2xf32>
+// CHECK-DAG: [[CNegInf:%.+]] = arith.constant dense<-inf> : vector<2xf32>
 // CHECK: [[ISINF:%.+]] = arith.cmpf oeq, [[V]], [[CInf]]
 // CHECK: [[ISNEGINF:%.+]] = arith.cmpf oeq, [[V]], [[CNegInf]]
 // CHECK: [[ISNAN:%.+]] = arith.cmpf uno, [[V]], [[V]]
diff --git a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
index 78e8db795788a..cef9ec850387e 100644
--- a/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToLLVM/complex-range-option.mlir
@@ -50,7 +50,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -200,7 +200,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = llvm.or %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = llvm.and %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = llvm.and %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(0x7F800000 : f32) : f32
+// DIV-SMITH: %[[INF:.*]] = llvm.mlir.constant(+inf : f32) : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = llvm.intr.copysign(%[[INF]], %[[RHS_REAL]]) : (f32, f32) -> f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = llvm.fmul %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] {fastmathFlags = #llvm.fastmath<nsz, arcp>} : f32
diff --git a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
index deb4eb6d9d08c..9e1c540d8cd5b 100644
--- a/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
+++ b/mlir/test/Conversion/ComplexToSPIRV/complex-to-spirv.mlir
@@ -46,7 +46,7 @@ func.func @complex_const() -> complex<f32> {
 }
 
 // CHECK-LABEL: func.func @complex_const()
-//       CHECK:   spirv.Constant dense<[0x7FC00000, 0.000000e+00]> : vector<2xf32>
+//       CHECK:   spirv.Constant dense<[+qnan, 0.000000e+00]> : vector<2xf32>
 
 // -----
 
diff --git a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
index 97f37d8ebe77e..ce588fa5d20dc 100644
--- a/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/complex-range-option.mlir
@@ -46,7 +46,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -183,7 +183,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // DIV-SMITH: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // DIV-SMITH: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // DIV-SMITH: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// DIV-SMITH: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// DIV-SMITH: %[[INF:.*]] = arith.constant +inf : f32
 // DIV-SMITH: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // DIV-SMITH: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nsz,arcp> : f32
 // DIV-SMITH: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nsz,arcp> : f32
diff --git a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
index 1242b1b66ff7b..bef9163077b15 100644
--- a/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
+++ b/mlir/test/Conversion/ComplexToStandard/convert-to-standard.mlir
@@ -117,7 +117,7 @@ func.func @complex_div(%lhs: complex<f32>, %rhs: complex<f32>) -> complex<f32> {
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] : f32
@@ -213,7 +213,7 @@ func.func @complex_exp(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] : f32
@@ -486,7 +486,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %[[NEG_ONE]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -508,7 +508,7 @@ func.func @complex_tan(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -534,7 +534,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] : f32
@@ -556,7 +556,7 @@ func.func @complex_tanh(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] : f32
 // CHECK: %true = arith.constant true
@@ -605,9 +605,9 @@ func.func @complex_sqrt(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] : f32
@@ -847,7 +847,7 @@ func.func @complex_exp_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK-DAG: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG: %[[HALF:.*]] = arith.constant 5.000000e-01 : f32
-// CHECK-DAG: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK-DAG: %[[EXP_REAL:.*]] = math.exp %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[REAL_HALF:.*]] = arith.mulf %[[REAL]], %[[HALF]] fastmath<nnan,contract> : f32
 // CHECK-DAG: %[[EXP_HALF:.*]] = math.exp %[[REAL_HALF]] fastmath<nnan,contract> : f32
@@ -1011,9 +1011,9 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1084,7 +1084,7 @@ func.func @complex_atan2_with_fmf(%lhs: complex<f32>,
 // CHECK: %[[VAR355:.*]] = arith.ori %[[VAR353]], %[[VAR354]] : i1
 // CHECK: %[[VAR356:.*]] = arith.andi %[[VAR350]], %[[VAR352]] : i1
 // CHECK: %[[VAR357:.*]] = arith.andi %[[VAR355]], %[[VAR356]] : i1
-// CHECK: %[[CST_17:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[CST_17:.*]] = arith.constant +inf : f32
 // CHECK: %[[VAR358:.*]] = math.copysign %[[CST_17]], %[[VAR329]] : f32
 // CHECK: %[[VAR359:.*]] = arith.mulf %[[VAR358]], %[[VAR327]] fastmath<nnan,contract> : f32
 // CHECK: %[[VAR360:.*]] = arith.mulf %[[VAR358]], %[[VAR328]] fastmath<nnan,contract> : f32
@@ -1221,7 +1221,7 @@ func.func @complex_div_with_fmf(%lhs: complex<f32>, %rhs: complex<f32>) -> compl
 // CHECK: %[[LHS_CONTAINS_NOT_NAN_VALUE:.*]] = arith.ori %[[LHS_REAL_IS_NOT_NAN]], %[[LHS_IMAG_IS_NOT_NAN]] : i1
 // CHECK: %[[RHS_IS_ZERO:.*]] = arith.andi %[[RHS_REAL_ABS_IS_ZERO]], %[[RHS_IMAG_ABS_IS_ZERO]] : i1
 // CHECK: %[[RESULT_IS_INFINITY:.*]] = arith.andi %[[LHS_CONTAINS_NOT_NAN_VALUE]], %[[RHS_IS_ZERO]] : i1
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[INF_WITH_SIGN_OF_RHS_REAL:.*]] = math.copysign %[[INF]], %[[RHS_REAL]] : f32
 // CHECK: %[[INFINITY_RESULT_REAL:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[INFINITY_RESULT_IMAG:.*]] = arith.mulf %[[INF_WITH_SIGN_OF_RHS_REAL]], %[[LHS_IMAG]] fastmath<nnan,contract> : f32
@@ -1323,9 +1323,9 @@ func.func @complex_sqrt_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_RE:.*]] = arith.mulf %[[SQRT_ABS]], %[[COS]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM:.*]] = arith.mulf %[[SQRT_ABS]], %[[SIN]] fastmath<nnan,contract> : f32
 // CHECK: %[[RESULT_IM2:.*]] = arith.select %[[SIN_ZERO]], %[[ZERO]], %[[RESULT_IM]] : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
-// CHECK: %[[NINF:.*]] = arith.constant 0xFF800000 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
+// CHECK: %[[NINF:.*]] = arith.constant -inf : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABSIM:.*]] = math.absf %[[IM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMINF:.*]] = arith.cmpf oeq, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABSIMNOTINF:.*]] = arith.cmpf one, %[[ABSIM]], %[[INF]] fastmath<nnan,contract> : f32
@@ -1440,7 +1440,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[V0:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
 // CHECK: %[[REAL:.*]] = arith.mulf %[[V0]], %cst fastmath<nnan,contract> : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1462,7 +1462,7 @@ func.func @complex_tan_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
@@ -1489,7 +1489,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[REAL:.*]] = complex.re %[[ARG]] : complex<f32>
 // CHECK: %[[IMAG:.*]] = complex.im %[[ARG]] : complex<f32>
 // CHECK: %[[NEG_ONE:.*]] = arith.constant -1.000000e+00 : f32
-// CHECK: %[[INF:.*]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[INF:.*]] = arith.constant +inf : f32
 // CHECK: %[[FOUR:.*]] = arith.constant 4.000000e+00 : f32
 // CHECK: %[[TWO_REAL:.*]] = arith.addf %[[REAL]], %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[NEG_TWO_REAL:.*]] = arith.mulf %[[NEG_ONE]], %[[TWO_REAL]] fastmath<nnan,contract> : f32
@@ -1511,7 +1511,7 @@ func.func @complex_tanh_with_fmf(%arg: complex<f32>) -> complex<f32> {
 // CHECK: %[[RESULT_IMAG:.*]] = arith.divf %[[IMAG_NUM]], %[[DENOM]] fastmath<nnan,contract> : f32
 // CHECK: %[[ABS_REAL:.*]] = math.absf %[[REAL]] fastmath<nnan,contract> : f32
 // CHECK: %[[ZERO:.*]] = arith.constant 0.000000e+00 : f32
-// CHECK: %[[NAN:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK: %[[NAN:.*]] = arith.constant +qnan : f32
 // CHECK: %[[ABS_REAL_INF:.*]] = arith.cmpf oeq, %[[ABS_REAL]], %[[INF]] fastmath<nnan,contract> : f32
 // CHECK: %[[IMAG_ZERO:.*]] = arith.cmpf oeq, %[[IMAG]], %[[ZERO]] fastmath<nnan,contract> : f32
 // CHECK: %true = arith.constant true
diff --git a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
index a0801443057ea..eaacce2f91e7f 100644
--- a/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
+++ b/mlir/test/Conversion/GPUToNVVM/wmma-ops-to-nvvm.mlir
@@ -335,7 +335,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP0:.*]] = llvm.fcmp "ogt" %[[A0]], %[[B0]] : vector<2xf16>
 //       CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[A0]], %[[B0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP1:.*]] = llvm.fcmp "uno" %[[A0]], %[[B0]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C0:.*]] = llvm.select %[[CMP1]], %[[NAN]], %[[SEL0]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M1:.*]] = llvm.insertvalue %[[C0]], %[[M0]][0] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A1:.*]] = llvm.extractvalue %{{.*}}[1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -343,7 +343,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP2:.*]] = llvm.fcmp "ogt" %[[A1]], %[[B1]] : vector<2xf16>
 //       CHECK: %[[SEL1:.*]] = llvm.select %[[CMP2]], %[[A1]], %[[B1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP3:.*]] = llvm.fcmp "uno" %[[A1]], %[[B1]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C1:.*]] = llvm.select %[[CMP3]], %[[NAN]], %[[SEL1]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M2:.*]] = llvm.insertvalue %[[C1]], %[[M1]][1] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A2:.*]] = llvm.extractvalue %{{.*}}[2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -351,7 +351,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP4:.*]] = llvm.fcmp "ogt" %[[A2]], %[[B2]] : vector<2xf16>
 //       CHECK: %[[SEL2:.*]] = llvm.select %[[CMP4]], %[[A2]], %[[B2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP5:.*]] = llvm.fcmp "uno" %[[A2]], %[[B2]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C2:.*]] = llvm.select %[[CMP5]], %[[NAN]], %[[SEL2]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M3:.*]] = llvm.insertvalue %[[C2]], %[[M2]][2] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 //       CHECK: %[[A3:.*]] = llvm.extractvalue %{{.*}}[3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
@@ -359,7 +359,7 @@ gpu.module @test_module {
 //       CHECK: %[[CMP6:.*]] = llvm.fcmp "ogt" %[[A3]], %[[B3]] : vector<2xf16>
 //       CHECK: %[[SEL3:.*]] = llvm.select %[[CMP6]], %[[A3]], %[[B3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[CMP7:.*]] = llvm.fcmp "uno" %[[A3]], %[[B3]] : vector<2xf16>
-//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(0x7E00 : f16) : vector<2xf16>
+//       CHECK: %[[NAN:.*]] = llvm.mlir.constant(+qnan : f16) : vector<2xf16>
 //       CHECK: %[[C3:.*]] = llvm.select %[[CMP7]], %[[NAN]], %[[SEL3]] : vector<2xi1>, vector<2xf16>
 //       CHECK: %[[M5:.*]] = llvm.insertvalue %[[C3]], %[[M3]][3] : !llvm.struct<(vector<2xf16>, vector<2xf16>, vector<2xf16>, vector<2xf16>)>
 
diff --git a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
index e6bd800a0cf0a..938426c271f82 100644
--- a/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
+++ b/mlir/test/Conversion/TosaToLinalg/tosa-to-linalg.mlir
@@ -601,8 +601,8 @@ func.func @test_simple_f16(%arg0: tensor<1xf16>) -> () {
   // CHECK: linalg.generic
   // CHECK: [[ROUND:%.+]] = math.roundeven {{%[a-z0-9_]+}} : f16
   // CHECK: [[CONV:%.+]] = arith.fptosi [[ROUND]] : f16 to i32
-  // CHECK: [[POSINF:%.+]] = arith.constant 0x7C00 : f16
-  // CHECK: [[NEGINF:%.+]] = arith.constant 0xFC00 : f16
+  // CHECK: [[POSINF:%.+]] = arith.constant +inf : f16
+  // CHECK: [[NEGINF:%.+]] = arith.constant -inf : f16
   // CHECK: [[OVERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[POSINF]] : f16
   // CHECK: [[UNDERFLOW:%.+]] = arith.cmpf ueq, [[ROUND]], [[NEGINF]] : f16
   // CHECK: [[MININT:%.+]] = arith.constant -2147483648 : i32
@@ -2334,7 +2334,7 @@ func.func @reduce_min_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2353,7 +2353,7 @@ func.func @reduce_max_nan_propagate(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf3
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2372,7 +2372,7 @@ func.func @reduce_min_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2391,7 +2391,7 @@ func.func @reduce_max_nan_ignore_int(%arg0: tensor<5x4xi8>, %arg1: tensor<5x4xi8
   // CHECK-NOT: arith.cmpf uno
   // CHECK-NOT: arith.select
   // CHECK: linalg.yield
-  // CHECK-NOT: arith.constant 0x7FC00000
+  // CHECK-NOT: arith.constant +qnan
   // CHECK-NOT: tensor.empty()
   // CHECK-NOT: linalg.fill
   // CHECK-NOT: tensor.empty()
@@ -2410,7 +2410,7 @@ func.func @reduce_min_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
@@ -2428,7 +2428,7 @@ func.func @reduce_max_nan_ignore(%arg0: tensor<5x4xf32>, %arg1: tensor<5x4xf32>)
   // CHECK: arith.cmpf uno
   // CHECK: arith.select
   // CHECK: linalg.yield
-  // CHECK: arith.constant 0x7FC00000
+  // CHECK: arith.constant +qnan
   // CHECK: tensor.empty()
   // CHECK: linalg.fill
   // CHECK: tensor.empty()
diff --git a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
index c7d9e22fb2423..4a536c003abb3 100644
--- a/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
+++ b/mlir/test/Conversion/VectorToLLVM/vector-reduction-to-llvm.mlir
@@ -123,7 +123,7 @@ func.func @masked_reduce_minf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmin"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
@@ -137,7 +137,7 @@ func.func @masked_reduce_minf_f32_scalable(%arg0: vector<[16]xf32>, %mask : vect
 // CHECK-LABEL:   func.func @masked_reduce_minf_f32_scalable(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<[16]xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<[16]xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0xFFC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(-qnan : f32) : f32
 // CHECK:           %[[VL_BASE:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           %[[VSCALE:.*]] = "llvm.intr.vscale"() : () -> i64
 // CHECK:           %[[CAST_IDX:.*]] = builtin.unrealized_conversion_cast %[[VSCALE]] : i64 to index
@@ -155,7 +155,7 @@ func.func @masked_reduce_maxf_f32(%arg0: vector<16xf32>, %mask : vector<16xi1>)
 // CHECK-LABEL:   func.func @masked_reduce_maxf_f32(
 // CHECK-SAME:                                      %[[INPUT:.*]]: vector<16xf32>,
 // CHECK-SAME:                                      %[[MASK:.*]]: vector<16xi1>) -> f32 {
-// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(0x7FC00000 : f32) : f32
+// CHECK:           %[[NEUTRAL:.*]] = llvm.mlir.constant(+qnan : f32) : f32
 // CHECK:           %[[VL:.*]] = llvm.mlir.constant(16 : i32) : i32
 // CHECK:           "llvm.intr.vp.reduce.fmax"(%[[NEUTRAL]], %[[INPUT]], %[[MASK]], %[[VL]]) : (f32, vector<16xf32>, vector<16xi1>, i32) -> f32
 
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
index 9dacc316f1072..31dffa96adace 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_reduction.mlir
@@ -44,7 +44,7 @@ func.func @vecdim_reduction_minf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_minf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmax:.*]] = arith.constant dense<0x7F800000> : vector<128xf32>
+// CHECK:         %[[vmax:.*]] = arith.constant dense<+inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmax]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[min:.*]] = arith.minimumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -71,7 +71,7 @@ func.func @vecdim_reduction_maxf(%in: memref<256x512xf32>, %out: memref<256xf32>
 
 // CHECK-LABEL: @vecdim_reduction_maxf
 // CHECK:       affine.for %{{.*}} = 0 to 256 {
-// CHECK:         %[[vmin:.*]] = arith.constant dense<0xFF800000> : vector<128xf32>
+// CHECK:         %[[vmin:.*]] = arith.constant dense<-inf> : vector<128xf32>
 // CHECK:         %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vmin]]) -> (vector<128xf32>) {
 // CHECK:           %[[ld:.*]] = vector.transfer_read %{{.*}} : memref<256x512xf32>, vector<128xf32>
 // CHECK:           %[[max:.*]] = arith.maximumf %[[red_iter]], %[[ld]] : vector<128xf32>
@@ -294,9 +294,9 @@ func.func @vecdim_reduction_minnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_minnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0x7FC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<+qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
@@ -328,9 +328,9 @@ func.func @vecdim_reduction_maxnumf(%in: memref<256x512xf32>, %out: memref<256xf
 // CHECK-LABEL:   func.func @vecdim_reduction_maxnumf(
 // CHECK-SAME:      %[[input:.*]]: memref<256x512xf32>,
 // CHECK-SAME:      %[[output:.*]]: memref<256xf32>) {
-// CHECK:           %[[cst:.*]] = arith.constant 0xFF800000 : f32
+// CHECK:           %[[cst:.*]] = arith.constant -inf : f32
 // CHECK:           affine.for %{{.*}} = 0 to 256 {
-// CHECK:             %[[vzero:.*]] = arith.constant dense<0xFFC00000> : vector<128xf32>
+// CHECK:             %[[vzero:.*]] = arith.constant dense<-qnan> : vector<128xf32>
 // CHECK:             %[[vred:.*]] = affine.for %{{.*}} = 0 to 512 step 128 iter_args(%[[red_iter:.*]] = %[[vzero]]) -> (vector<128xf32>) {
 // CHECK:               %[[poison:.*]] = ub.poison : f32
 // CHECK:               %[[ld:.*]] = vector.transfer_read %[[input]]{{\[}}%{{.*}}, %{{.*}}], %[[poison]] {in_bounds = [true]} : memref<256x512xf32>, vector<128xf32>
diff --git a/mlir/test/Dialect/Arith/emulate-wide-int.mlir b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
index 695d8fd453fd7..b23121cf101cc 100644
--- a/mlir/test/Dialect/Arith/emulate-wide-int.mlir
+++ b/mlir/test/Dialect/Arith/emulate-wide-int.mlir
@@ -991,7 +991,7 @@ func.func @uitofp_i64_f64_vector(%a : vector<3xi64>) -> vector<3xf64> {
 // CHECK-NEXT:    [[HIEQ0:%.+]]  = arith.cmpi eq, [[HI]], [[CST0]] : i32
 // CHECK-NEXT:    [[LOWFP:%.+]]  = arith.uitofp [[LOW]] : i32 to f16
 // CHECK-NEXT:    [[HIFP:%.+]]   = arith.uitofp [[HI]] : i32 to f16
-// CHECK-NEXT:    [[POW:%.+]]    = arith.constant 0x7C00 : f16
+// CHECK-NEXT:    [[POW:%.+]]    = arith.constant +inf : f16
 // CHECK-NEXT:    [[RESHI:%.+]]  = arith.mulf [[HIFP]], [[POW]] : f16
 // CHECK-NEXT:    [[RES:%.+]]    = arith.addf [[LOWFP]], [[RESHI]] : f16
 // CHECK-NEXT:    [[SEL:%.+]]    = arith.select [[HIEQ0]], [[LOWFP]], [[RES]] : f16
diff --git a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
index 7dedfbdff10a7..978c29de3fc85 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/attrs.mlir
@@ -97,12 +97,12 @@ module @TestDenseStringElementsAttr attributes {
 // CHECK-LABEL: @TestFloat
 module @TestFloat attributes {
   // CHECK: bytecode.float = 1.000000e+01 : f64
-  // CHECK: bytecode.float1 = 0.10000{{.*}} : f80
-  // CHECK: bytecode.float2 = 0.10000{{.*}} : f128
+  // CHECK: bytecode.float1 = 1.000000e-01 : f80
+  // CHECK: bytecode.float2 = 1.000000e-01 : f128
   // CHECK: bytecode.float3 = -5.000000e-01 : bf16
-  // CHECK: bytecode.inf = 0x7FF0000000000000 : f64
-  // CHECK: bytecode.nan = 0x7FF8000000000000 : f64
-  // CHECK: bytecode.ninf = 0xFFF0000000000000 : f64
+  // CHECK: bytecode.inf = +inf : f64
+  // CHECK: bytecode.nan = +qnan : f64
+  // CHECK: bytecode.ninf = -inf : f64
   bytecode.float = 10.0 : f64,
   bytecode.float1 = 0.1 : f80,
   bytecode.float2 = 0.1 : f128,
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
index d427035435277..ebb198456b3c1 100644
--- a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
+++ b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed.mlir
@@ -125,8 +125,8 @@ module @TestFloatAttr attributes {
   // CHECK-DAG: bytecode.f16 = 1.500000e+00 : f16
   // CHECK-DAG: bytecode.f32 = 3.140000e+00 : f32
   // CHECK-DAG: bytecode.f64 = 1.000000e+01 : f64
-  // CHECK-DAG: bytecode.f80 = 0.1{{.*}} : f80
-  // CHECK-DAG: bytecode.f128 = 0.1{{.*}} : f128
+  // CHECK-DAG: bytecode.f80 = 1.000000e-01 : f80
+  // CHECK-DAG: bytecode.f128 = 1.000000e-01 : f128
   // CHECK-DAG: bytecode.tf32 = 1.000000e+00 : tf32
   // CHECK-DAG: bytecode.f8E5M2 = 1.000000e+00 : f8E5M2
   // CHECK-DAG: bytecode.f8E4M3 = 1.000000e+00 : f8E4M3
diff --git a/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc b/mlir/test/Dialect/Builtin/Bytecode/builtin_fixed_0.mlirbc
index 95be4de8fd1ab202b51658ce69f8037276dd74e8..4f661b5a1d233336116b303958bc724529f88b5a 100644
GIT binary patch
delta 27
ecmcbobx&)<4<?S&XCPn_vo2F=+Gc%beIWqP@(kAi

delta 27
ecmcbobx&)<4<?TP5U_s|vo6#B_|5vv`a%Hd#SX&&

diff --git a/mlir/test/Dialect/Complex/canonicalize.mlir b/mlir/test/Dialect/Complex/canonicalize.mlir
index 1c5216c82e5c3..065e1a9cdb4c9 100644
--- a/mlir/test/Dialect/Complex/canonicalize.mlir
+++ b/mlir/test/Dialect/Complex/canonicalize.mlir
@@ -332,7 +332,7 @@ func.func @div_op_with_rhs_has_nan_real() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 1.0 : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -342,7 +342,7 @@ func.func @div_op_with_rhs_has_nan_imag() -> complex<f32> {
   %a = complex.constant [1.0 : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
@@ -352,7 +352,7 @@ func.func @div_op_with_rhs_has_nan_real_imag() -> complex<f32> {
   %a = complex.constant [0x7fffffff : f32, 0x7fffffff : f32]: complex<f32>
   %b = complex.constant [1.0: f32, 0.0 : f32]: complex<f32>
   %div = complex.div %a, %b : complex<f32>
-  // CHECK: %[[DIV:.*]] = complex.constant [0x7FFFFFFF : f32, 0x7FFFFFFF : f32] : complex<f32>
+  // CHECK: %[[DIV:.*]] = complex.constant [+nan(0x3FFFFF) : f32, +nan(0x3FFFFF) : f32] : complex<f32>
   // CHECK: return %[[DIV]] : complex<f32>
   return %div : complex<f32>
 }
diff --git a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
index 3897f8502bb04..deb59e8fc3ab0 100644
--- a/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-decompose.mlir
@@ -369,7 +369,7 @@ func.func @softmax(%arg0: tensor<2x16x32xf32>, %dst: tensor<2x16x32xf32>) -> ten
 // CHECK-LABEL:      func.func @softmax(
 // CHECK-SAME:           %[[ARG0:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>, %[[DST:[a-zA-Z0-9_]+]]: tensor<2x16x32xf32>) -> tensor<2x16x32xf32> {
 // CHECK-DAG:        %[[D1:.+]] = tensor.empty() : tensor<2x16xf32>
-// CHECK-DAG:        %[[CST:.+]] = arith.constant 0xFFC00000 : f32
+// CHECK-DAG:        %[[CST:.+]] = arith.constant -qnan : f32
 // CHECK:        %[[D2:.+]] = linalg.fill ins(%[[CST]] : f32) outs(%[[D1]] : tensor<2x16xf32>) -> tensor<2x16xf32>
 // CHECK:        %[[D3:.+]] = linalg.generic {indexing_maps = [#[[$MAP]], #[[$MAP1]]], iterator_types = ["parallel",
 // CHECK-SAME:     "parallel", "reduction"]} ins(%[[ARG0]] : tensor<2x16x32xf32>) outs(%[[D2]] : tensor<2x16xf32>) {
diff --git a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
index 9849f36285b16..a97f0a3327acb 100644
--- a/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-op-split-reduction.mlir
@@ -118,7 +118,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFF800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [4, 8, 2] : tensor<32x2xf32> into tensor<4x8x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 4, 8] : tensor<5x32xf32> into tensor<5x4x8xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -323,7 +323,7 @@ func.func @generic_split_3d(%input: tensor<32x2xf32>, %input_2: tensor<5x32xf32>
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
 //  CHECK-DAG: #[[$MAP4:.*]] = affine_map<(d0, d1, d2) -> (d0, d1)>
 // CHECK-LABEL:  func @generic_split_3d
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7F800000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +inf : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1], [2]] output_shape [8, 4, 2] : tensor<32x2xf32> into tensor<8x4x2xf32>
 //  CHECK-DAG: %[[I2:.*]] = tensor.expand_shape %{{.*}}[0], [1, 2]] output_shape [5, 8, 4] : tensor<5x32xf32> into tensor<5x8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<5x2x4xf32>
@@ -429,7 +429,7 @@ func.func @generic_split_maxnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_maxnumf
 //  The float value 0xFFC00000 that is filled into the init tensor represents negative NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0xFFC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant -qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
@@ -475,7 +475,7 @@ func.func @generic_split_minnumf(%in: tensor<32xf32>, %out: tensor<f32>) -> tens
 //  CHECK-DAG: #[[$MAP3:.*]] = affine_map<(d0) -> ()>
 // CHECK-LABEL:  func @generic_split_minnumf
 //  The float value 0x7FC00000 that is filled into the init tensor represents positive NaN.
-//  CHECK-DAG: %[[ID:.*]] = arith.constant 0x7FC00000 : f32
+//  CHECK-DAG: %[[ID:.*]] = arith.constant +qnan : f32
 //  CHECK-DAG: %[[I1:.*]] = tensor.expand_shape %{{.*}}[0, 1]] output_shape [8, 4] : tensor<32xf32> into tensor<8x4xf32>
 //  CHECK-DAG: %[[INI:.*]] = tensor.empty() : tensor<4xf32>
 //      CHECK: %[[F:.*]] = linalg.fill ins(%[[ID]] : f32) outs(%[[INI]] : tensor<4xf32>) -> tensor<4xf32>
diff --git a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
index 25af6796d1f63..daa80a98c9af0 100644
--- a/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
+++ b/mlir/test/Dialect/Linalg/transform-tile-reduction.mlir
@@ -317,7 +317,7 @@ module attributes {transform.with_named_sequence} {
 
 // CHECK: func @reduction_tile_multiple_results
 // CHECK-DAG:   %[[SUM_ID:.+]] = arith.constant 0.000000e+00 : f32
-// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant 0xFF800000 : f32
+// CHECK-DAG:   %[[MAX_ID:.+]] = arith.constant -inf : f32
 // CHECK-DAG:   %[[SUM_INIT:.+]] = linalg.fill ins(%[[SUM_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK-DAG:   %[[MAX_INIT:.+]] = linalg.fill ins(%[[MAX_ID]] : f32) outs(%{{.*}} : tensor<?x5xf32>) -> tensor<?x5xf32>
 // CHECK:       %[[OUT:.+]]:2 = scf.for
diff --git a/mlir/test/Dialect/Math/canonicalize.mlir b/mlir/test/Dialect/Math/canonicalize.mlir
index a73a15f34e577..b0e34449690da 100644
--- a/mlir/test/Dialect/Math/canonicalize.mlir
+++ b/mlir/test/Dialect/Math/canonicalize.mlir
@@ -28,7 +28,7 @@ func.func @log2_fold() -> f32 {
 }
 
 // CHECK-LABEL: @log2_fold2
-// CHECK: %[[cst:.+]] = arith.constant 0xFF800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant -inf : f32
   // CHECK: return %[[cst]]
 func.func @log2_fold2() -> f32 {
   %c = arith.constant 0.0 : f32
@@ -56,7 +56,7 @@ func.func @log2_fold_64() -> f64 {
 }
 
 // CHECK-LABEL: @log2_fold2_64
-// CHECK: %[[cst:.+]] = arith.constant 0xFFF0000000000000 : f64
+// CHECK: %[[cst:.+]] = arith.constant -inf : f64
   // CHECK: return %[[cst]]
 func.func @log2_fold2_64() -> f64 {
   %c = arith.constant 0.0 : f64
@@ -651,7 +651,7 @@ func.func @fpowi_fold_const() -> f32 {
 // The fold no longer requires the exponent to be exactly representable as
 // f32: it is used as an integer, so this folds and overflows to +inf.
 // CHECK-LABEL: @fpowi_fold_overflow
-// CHECK: %[[cst:.+]] = arith.constant 0x7F800000 : f32
+// CHECK: %[[cst:.+]] = arith.constant +inf : f32
 // CHECK: return %[[cst]]
 func.func @fpowi_fold_overflow() -> f32 {
   %cst = arith.constant 2.000000e+00 : f32
diff --git a/mlir/test/Dialect/Math/expand-math.mlir b/mlir/test/Dialect/Math/expand-math.mlir
index 84b0bb0aa314a..973eab6348d66 100644
--- a/mlir/test/Dialect/Math/expand-math.mlir
+++ b/mlir/test/Dialect/Math/expand-math.mlir
@@ -634,8 +634,8 @@ func.func @math_fpowi_neg_odd_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CUBE:.*]] = arith.mulf %[[SQ]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[CUBE]], %[[CST0]] : tensor<8xf32>
@@ -657,8 +657,8 @@ func.func @math_fpowi_neg_even_power(%0 : tensor<8xf32>) -> tensor<8xf32> {
 //  CHECK-DAG:    %[[CST1:.*]] = arith.constant dense<1.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CST0:.*]] = arith.constant dense<0.000000e+00> : tensor<8xf32>
 //  CHECK-DAG:    %[[CSTNEG0:.*]] = arith.constant dense<-0.000000e+00> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<0x7F800000> : tensor<8xf32>
-//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<0xFF800000> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTINF:.*]] = arith.constant dense<+inf> : tensor<8xf32>
+//  CHECK-DAG:    %[[CSTNEGINF:.*]] = arith.constant dense<-inf> : tensor<8xf32>
 //  CHECK:        %[[SQ:.*]] = arith.mulf %[[ARG0]], %[[ARG0]] : tensor<8xf32>
 //  CHECK:        %[[PW4:.*]] = arith.mulf %[[SQ]], %[[SQ]] : tensor<8xf32>
 //  CHECK:        %[[CMP0:.*]] = arith.cmpf oeq, %[[PW4]], %[[CST0]] : tensor<8xf32>
diff --git a/mlir/test/Dialect/Math/polynomial-approximation.mlir b/mlir/test/Dialect/Math/polynomial-approximation.mlir
index c14ed47853411..9f447ecd5f497 100644
--- a/mlir/test/Dialect/Math/polynomial-approximation.mlir
+++ b/mlir/test/Dialect/Math/polynomial-approximation.mlir
@@ -114,7 +114,7 @@ func.func @erf_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:     %[[cst_24:.*]] = arith.constant -4.000000e+00 : f32
 // CHECK-DAG:     %[[cst_25:.*]] = arith.constant -2.000000e+00 : f32
 // CHECK-DAG:     %[[cst_26:.*]] = arith.constant 2.000000e+00 : f32
-// CHECK-DAG:     %[[cst_27:.*]] = arith.constant 0x7F800000 : f32
+// CHECK-DAG:     %[[cst_27:.*]] = arith.constant +inf : f32
 // CHECK-DAG:     %[[cst_28:.*]] = arith.constant 10.0546875 : f32
 // CHECK:         %[[val_2:.*]] = math.absf %[[val_arg0]] : f32
 // CHECK-NEXT:    %[[val_3:.*]] = arith.addf %[[val_2]], %[[cst_26]] : f32
@@ -308,9 +308,9 @@ func.func @exp_scalable_vector(%arg0: vector<[8]xf32>) -> vector<[8]xf32> {
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_21:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_22:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_23:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_24:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_25:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_26:.*]] = arith.constant -0.115146101 : f32
@@ -434,9 +434,9 @@ func.func @expm1_scalable_vector(%arg0: vector<8x[8]xf32>) -> vector<8x[8]xf32>
 // CHECK-DAG:       %[[VAL_2:.*]] = arith.constant 1.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_3:.*]] = arith.constant -5.000000e-01 : f32
 // CHECK-DAG:       %[[VAL_4:.*]] = arith.constant 1.17549435E-38 : f32
-// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant 0xFF800000 : f32
-// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant 0x7F800000 : f32
-// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_5:.*]] = arith.constant -inf : f32
+// CHECK-DAG:       %[[VAL_6:.*]] = arith.constant +inf : f32
+// CHECK-DAG:       %[[VAL_7:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_8:.*]] = arith.constant 0.707106769 : f32
 // CHECK-DAG:       %[[VAL_9:.*]] = arith.constant 0.0703768358 : f32
 // CHECK-DAG:       %[[VAL_10:.*]] = arith.constant -0.115146101 : f32
@@ -683,7 +683,7 @@ func.func @rsqrt_scalar(%arg0: f32) -> f32 {
 // CHECK:           math.rsqrt
 // AVX2-LABEL:    func @rsqrt_vector_8xf32(
 // AVX2-SAME:       %[[VAL_0:.*]]: vector<8xf32>) -> vector<8xf32> {
-// AVX2:   %[[VAL_1:.*]] = arith.constant dense<0x7F800000> : vector<8xf32>
+// AVX2:   %[[VAL_1:.*]] = arith.constant dense<+inf> : vector<8xf32>
 // AVX2:   %[[VAL_2:.*]] = arith.constant dense<1.500000e+00> : vector<8xf32>
 // AVX2:   %[[VAL_3:.*]] = arith.constant dense<-5.000000e-01> : vector<8xf32>
 // AVX2:   %[[VAL_4:.*]] = arith.constant dense<1.17549435E-38> : vector<8xf32>
@@ -857,7 +857,7 @@ func.func @atan_scalar(%arg0: f32) -> f32 {
 // CHECK-DAG:       %[[VAL_17:.*]] = arith.constant 0.000000e+00 : f32
 // CHECK-DAG:       %[[VAL_18:.*]] = arith.constant 3.14159274 : f32
 // CHECK-DAG:       %[[VAL_19:.*]] = arith.constant -1.57079637 : f32
-// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant 0x7FC00000 : f32
+// CHECK-DAG:       %[[VAL_20:.*]] = arith.constant +qnan : f32
 // CHECK-DAG:       %[[VAL_21:.*]] = arith.extf %[[VAL_0]] : f16 to f32
 // CHECK-DAG:       %[[VAL_22:.*]] = arith.extf %[[VAL_1]] : f16 to f32
 // CHECK-DAG:       %[[VAL_23:.*]] = arith.divf %[[VAL_21]], %[[VAL_22]] : f32
diff --git a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
index 37a9e97106de9..134ea627cf165 100644
--- a/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
+++ b/mlir/test/Dialect/SPIRV/IR/structure-ops.mlir
@@ -177,7 +177,7 @@ func.func @struct_constant_identified() -> () {
 // -----
 
 func.func @coop_matrix_const_non_dense() -> () {
-    // expected-error @+2 {{floating point value not valid for specified type}}
+    // expected-error @below {{floating point value not valid for specified type}}
     %0 = spirv.Constant 0.000000e+00 : !spirv.coopmatrix<16x16xf32, Subgroup, MatrixAcc>
     return
 }
diff --git a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
index 12a3aee428ac6..e17e1b09e9e2e 100644
--- a/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
+++ b/mlir/test/Dialect/Tosa/constant-reciprocal-fold.mlir
@@ -22,8 +22,8 @@ func.func @reciprocal_fold_splat() -> tensor<12x7xf32> {
 
 // CHECK-LABEL: @reciprocal_div_zero
 func.func @reciprocal_div_zero() -> tensor<f32> {
-  // 0x7F800000 is the value for +infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7F800000
+  // +inf is the value for +infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0.0> : tensor<f32>} : () -> tensor<f32>
@@ -33,8 +33,8 @@ func.func @reciprocal_div_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_neg_zero
 func.func @reciprocal_div_neg_zero() -> tensor<f32> {
-  // 0xFF800000 is the value for -infinity
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0xFF800000
+  // -inf is the value for -infinity
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}-inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<-0.0> : tensor<f32>} : () -> tensor<f32>
@@ -44,8 +44,8 @@ func.func @reciprocal_div_neg_zero() -> tensor<f32> {
 
 // CHECK-LABEL: @reciprocal_div_nan
 func.func @reciprocal_div_nan() -> tensor<f32> {
-  // 0x7FC00000 is the value for NAN
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7FC00000
+  // +qnan is the value for NAN
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+qnan
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<0x7FC00000> : tensor<f32>} : () -> tensor<f32>
@@ -85,7 +85,7 @@ func.func @reciprocal_div_underflow() -> tensor<2xf16> {
 
 // CHECK-LABEL: @reciprocal_div_overflow
 func.func @reciprocal_div_overflow() -> tensor<2xf16> {
-  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}0x7C00, 0xFC00
+  // CHECK: [[RES:]] ={{.*}}tosa.const{{.*}}+inf, -inf
   // CHECK-NOT: tosa.reciprocal
   // CHECK: return [[RES]]
   %0 = "tosa.const"() {values = dense<[0.0000001, -0.0000001]> : tensor<2xf16>} : () -> tensor<2xf16>
diff --git a/mlir/test/Dialect/Tosa/invalid.mlir b/mlir/test/Dialect/Tosa/invalid.mlir
index a1a9f6d59b0ac..63ecd06dd207a 100644
--- a/mlir/test/Dialect/Tosa/invalid.mlir
+++ b/mlir/test/Dialect/Tosa/invalid.mlir
@@ -2019,7 +2019,7 @@ func.func @test_mul_out_i16(%arg0: tensor<13x21x3xi8>, %arg1: tensor<13x1x3xi8>,
 
 // CHECK-LABEL: test_clamp_nan_min_val
 func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=0xFFFFFFFF : f32, max_val=1.000000e+00 : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=-nan(0x3FFFFF) : f32, max_val=1.000000e+00 : f32}}
   %0 = tosa.clamp %arg0 {min_val = 0xFFFFFFFF : f32, max_val = 1.0: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
@@ -2028,7 +2028,7 @@ func.func @test_clamp_nan_min_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3x
 
 // CHECK-LABEL: test_clamp_nan_max_val
 func.func @test_clamp_nan_max_val(%arg0: tensor<13x21x3xf32>) -> tensor<13x21x3xf32> {
-  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=0x7FFFFFFF : f32}}
+  // expected-error at +1 {{'tosa.clamp' op min/max attributes should not be 'NaN', got min_val=2.300000e+00 : f32, max_val=+nan(0x3FFFFF) : f32}}
   %0 = tosa.clamp %arg0 {min_val = 2.3 : f32, max_val = 0x7FFFFFFF: f32} : (tensor<13x21x3xf32>) -> tensor<13x21x3xf32>
   return %0 : tensor<13x21x3xf32>
 }
diff --git a/mlir/test/Dialect/XeGPU/propagate-layout.mlir b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
index 79a5d229263c5..ceb8123329cdb 100644
--- a/mlir/test/Dialect/XeGPU/propagate-layout.mlir
+++ b/mlir/test/Dialect/XeGPU/propagate-layout.mlir
@@ -776,7 +776,7 @@ func.func @vector_2d_reduction_with_fractional_subgroup_size_1x4(%arg0: memref<1
 gpu.module @test {
 // CHECK: func.func @vector_reduction_broadcast_transpose(%[[ARG0:.*]]: memref<1024x64xf32>, %[[ARG1:.*]]: memref<1024x64xf32>)
 // CHECK: %[[CST:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} dense<0.000000e+00> : vector<1xf32>
-// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<0xFF800000> : vector<1x16xf32>
+// CHECK: %[[CST_0:.*]] = arith.constant {layout_result_0 = #xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>} dense<-inf> : vector<1x16xf32>
 // CHECK: %[[CST_1:.*]] = arith.constant {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [16, 1], lane_data = [1, 1], order = [0, 1]>, dims = [0]>} dense<0.000000e+00> : vector<8xf32>
 // CHECK: %[[C0:.*]] = arith.constant 0 : index
 // CHECK: %[[RED:.*]] = vector.multi_reduction <add>, %[[CST_0]], %[[CST]] {layout_result_0 = #xegpu.slice<#xegpu.layout<lane_layout = [1, 16], lane_data = [1, 1]>, dims = [1]>} [1] : vector<1x16xf32> to vector<1xf32>
diff --git a/mlir/test/IR/array-of-attr.mlir b/mlir/test/IR/array-of-attr.mlir
index 2c7b5009f74a2..96f7207170394 100644
--- a/mlir/test/IR/array-of-attr.mlir
+++ b/mlir/test/IR/array-of-attr.mlir
@@ -14,5 +14,5 @@ test.array_of_attr_op
 test.array_of_attr_op a = [], b = [], c = []
 
 // CHECK: "test.test_array_float"
-// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, 0x7FF0000000000000 : f64
+// CHECK-SAME: 1.000000e+00 : f32, 1.000000e+00, +inf : f64
 "test.test_array_float"() {test.float_arr = [1.0 : f32, 1.0 : f64, 0x7FF0000000000000 : f64]} : () -> ()
diff --git a/mlir/test/IR/custom-float-attr-roundtrip.mlir b/mlir/test/IR/custom-float-attr-roundtrip.mlir
index a8da89ba7372d..5cf82f493a7f8 100644
--- a/mlir/test/IR/custom-float-attr-roundtrip.mlir
+++ b/mlir/test/IR/custom-float-attr-roundtrip.mlir
@@ -8,11 +8,11 @@ func.func @test_enum_attr_roundtrip() -> () {
   "test.op"() {attr = #test.custom_float<"double" : 2.>} : () -> ()
    // CHECK: attr = #test.custom_float<"fp80" : 2.000000e+00>
   "test.op"() {attr = #test.custom_float<"fp80" : 2.>} : () -> ()
-  // CHECK: attr = #test.custom_float<"float" : 0x7FC00000>
+  // CHECK: attr = #test.custom_float<"float" : +qnan>
   "test.op"() {attr = #test.custom_float<"float" : 0x7FC00000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"double" : 0x7FF0000001000000>
+  // CHECK: attr = #test.custom_float<"double" : +snan(0x1000000)>
   "test.op"() {attr = #test.custom_float<"double" : 0x7FF0000001000000>} : () -> ()
-  // CHECK: attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>
+  // CHECK: attr = #test.custom_float<"fp80" : +nan(0x100000)>
   "test.op"() {attr = #test.custom_float<"fp80" : 0x7FFFC000000000100000>} : () -> ()
   return
 }
diff --git a/mlir/test/IR/float-literals-hex.mlir b/mlir/test/IR/float-literals-hex.mlir
new file mode 100644
index 0000000000000..cb843f6c4bb93
--- /dev/null
+++ b/mlir/test/IR/float-literals-hex.mlir
@@ -0,0 +1,61 @@
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | FileCheck %s --check-prefix=HEX
+// RUN: mlir-opt %s --mlir-print-float-special-literals-as-hex | mlir-opt | FileCheck %s --check-prefix=ROUNDTRIP
+
+// The --mlir-print-float-special-literals-as-hex flag prints infinities and NaNs as a
+// hexadecimal bit pattern (the legacy form). The parser still accepts both the
+// hex and the human-readable spellings, so the hex output round-trips back to
+// the human-readable form.
+
+// HEX-LABEL: @infinities
+// ROUNDTRIP-LABEL: @infinities
+func.func @infinities() {
+  // HEX: arith.constant 0x7F800000 : f32
+  // ROUNDTRIP: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // HEX: arith.constant 0xFF800000 : f32
+  // ROUNDTRIP: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // HEX: arith.constant 0x7FF0000000000000 : f64
+  // ROUNDTRIP: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  return
+}
+
+// HEX-LABEL: @nans
+// ROUNDTRIP-LABEL: @nans
+func.func @nans() {
+  // HEX: arith.constant 0x7FC00000 : f32
+  // ROUNDTRIP: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // HEX: arith.constant 0xFFF8000000000000 : f64
+  // ROUNDTRIP: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // HEX: arith.constant 0x7FC00001 : f32
+  // ROUNDTRIP: arith.constant +nan(0x1) : f32
+  %2 = arith.constant +nan(0x1) : f32
+  // HEX: arith.constant 0x7FF0000000000001 : f64
+  // ROUNDTRIP: arith.constant +snan(0x1) : f64
+  %3 = arith.constant +snan(0x1) : f64
+  // NanOnly types also print their bit pattern in the legacy form and round-trip
+  // to the human-readable `nan`.
+  // HEX: arith.constant 0x7F : f8E4M3FN
+  // ROUNDTRIP: arith.constant +nan : f8E4M3FN
+  %4 = arith.constant +nan : f8E4M3FN
+  // HEX: arith.constant 0xFF : f8E4M3FN
+  // ROUNDTRIP: arith.constant -nan : f8E4M3FN
+  %5 = arith.constant -nan : f8E4M3FN
+  // HEX: arith.constant 0x80 : f8E4M3FNUZ
+  // ROUNDTRIP: arith.constant -nan : f8E4M3FNUZ
+  %6 = arith.constant -nan : f8E4M3FNUZ
+  return
+}
+
+// Special values also print as hex inside dense elements attributes.
+// HEX-LABEL: @special_values_in_elements
+// ROUNDTRIP-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // HEX: dense<[0x7F800000, 0xFF800000, 0x7FC00000]> : tensor<3xf32>
+  // ROUNDTRIP: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  return
+}
diff --git a/mlir/test/IR/float-literals.mlir b/mlir/test/IR/float-literals.mlir
new file mode 100644
index 0000000000000..35eebc6f351f1
--- /dev/null
+++ b/mlir/test/IR/float-literals.mlir
@@ -0,0 +1,159 @@
+// RUN: mlir-opt %s | mlir-opt | FileCheck %s
+
+// Human-readable floating point special values and C-style hexadecimal floats,
+// as accepted by the parser and produced by the printer.
+
+// CHECK-LABEL: @infinities
+func.func @infinities() {
+  // CHECK: arith.constant +inf : f32
+  %0 = arith.constant +inf : f32
+  // CHECK: arith.constant -inf : f32
+  %1 = arith.constant -inf : f32
+  // CHECK: arith.constant +inf : f64
+  %2 = arith.constant +inf : f64
+  // CHECK: arith.constant -inf : f16
+  %3 = arith.constant -inf : f16
+  // CHECK: arith.constant +inf : bf16
+  %4 = arith.constant +inf : bf16
+  return
+}
+
+// CHECK-LABEL: @quiet_nans
+func.func @quiet_nans() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +qnan : f32
+  // CHECK: arith.constant -qnan : f64
+  %1 = arith.constant -qnan : f64
+  // CHECK: arith.constant +qnan : f16
+  %2 = arith.constant +qnan : f16
+  // CHECK: arith.constant -qnan : bf16
+  %3 = arith.constant -qnan : bf16
+  return
+}
+
+// CHECK-LABEL: @nan_payloads
+func.func @nan_payloads() {
+  // CHECK: arith.constant +nan(0x1) : f32
+  %0 = arith.constant +nan(0x1) : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
+  %1 = arith.constant -nan(0x3FFFFF) : f32
+  // CHECK: arith.constant +snan(0x1) : f64
+  %2 = arith.constant +snan(0x1) : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
+  %3 = arith.constant -snan(0x1000000) : f64
+  return
+}
+
+// Bare `nan`/`snan` (no payload) are accepted on IEEE types: `nan` normalizes to
+// the canonical quiet NaN, `snan` to a signaling NaN with the default payload.
+// CHECK-LABEL: @bare_nan_snan
+func.func @bare_nan_snan() {
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan : f32
+  // CHECK: arith.constant -qnan : f32
+  %1 = arith.constant -nan : f32
+  // CHECK: arith.constant +snan(0x200000) : f32
+  %2 = arith.constant +snan : f32
+  // CHECK: arith.constant -snan(0x4000000000000) : f64
+  %3 = arith.constant -snan : f64
+  return
+}
+
+// CHECK-LABEL: @hex_floats
+func.func @hex_floats() {
+  // 0x1.8p3 == 1.5 * 2^3 == 12.0; the exponent sign and 'p'/'P' case may vary.
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %0 = arith.constant 0x1.8p3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %1 = arith.constant 0x1.8p+3 : f64
+  // CHECK: arith.constant 1.200000e+01 : f64
+  %2 = arith.constant 0x1.8P3 : f64
+  // CHECK: arith.constant -1.200000e+01 : f64
+  %3 = arith.constant -0x1.8p3 : f64
+  // 0x1.8p-3 == 1.5 * 2^-3 == 0.1875
+  // CHECK: arith.constant 1.875000e-01 : f64
+  %4 = arith.constant 0x1.8p-3 : f64
+  // No fractional part.
+  // CHECK: arith.constant 1.600000e+01 : f32
+  %5 = arith.constant 0x1p4 : f32
+  // CHECK: arith.constant 0.000000e+00 : f32
+  %6 = arith.constant 0x0p0 : f32
+  // Built directly in the (wider than double) target semantics.
+  // CHECK: arith.constant 1.200000e+01 : f80
+  %7 = arith.constant 0x1.8p3 : f80
+  // CHECK: arith.constant 1.200000e+01 : f128
+  %8 = arith.constant 0x1.8p3 : f128
+  // A value that needs full precision to round-trip.
+  // CHECK: arith.constant 3.1415926535897931 : f64
+  %9 = arith.constant 0x1.921fb54442d18p1 : f64
+  return
+}
+
+// CHECK-LABEL: @special_values_in_elements
+func.func @special_values_in_elements() {
+  // CHECK: dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  %0 = arith.constant dense<[+inf, -inf, +qnan]> : tensor<3xf32>
+  // A splat built from a single special value.
+  // CHECK: dense<-inf> : tensor<4xf64>
+  %1 = arith.constant dense<-inf> : tensor<4xf64>
+  return
+}
+
+// Low-precision and other builtin float types on the special-value path.
+// CHECK-LABEL: @low_precision_and_other_types
+func.func @low_precision_and_other_types() {
+  // f8E8M0FNU is a NanOnly (AllOnes) unsigned type: a single NaN, spelled `nan`;
+  // the `qnan` alias normalizes to it.
+  // CHECK: arith.constant +nan : f8E8M0FNU
+  %0 = arith.constant +qnan : f8E8M0FNU
+  // A direct `+nan` round-trips (not only via the `qnan` alias).
+  // CHECK: arith.constant +nan : f8E8M0FNU
+  %1 = arith.constant +nan : f8E8M0FNU
+  // f8E5M2 is IEEE-encoded, so it keeps a payload (masked to 1 bit here).
+  // CHECK: arith.constant +nan(0x1) : f8E5M2
+  %2 = arith.constant +nan(0x7) : f8E5M2
+  // CHECK: arith.constant +inf : tf32
+  %3 = arith.constant +inf : tf32
+  return
+}
+
+// Payload spelling normalizations and overflow tolerance.
+// CHECK-LABEL: @nan_payload_and_overflow
+func.func @nan_payload_and_overflow() {
+  // A zero payload normalizes to the canonical quiet NaN.
+  // CHECK: arith.constant +qnan : f32
+  %0 = arith.constant +nan(0x0) : f32
+  // An unprefixed payload is decimal; it prints back as hex.
+  // CHECK: arith.constant +nan(0xA) : f32
+  %1 = arith.constant +nan(10) : f32
+  // Decimal overflow saturates to infinity rather than erroring.
+  // CHECK: arith.constant +inf : f16
+  %2 = arith.constant 1.0e40 : f16
+  return
+}
+
+// AllOnes NaN encoding (f8E4M3FN): a single NaN per sign, spelled `nan`, with no
+// payload and no quiet/signaling bit. The `qnan` alias normalizes to `nan`.
+// CHECK-LABEL: @nan_all_ones
+func.func @nan_all_ones() {
+  // CHECK: arith.constant +nan : f8E4M3FN
+  %0 = arith.constant +nan : f8E4M3FN
+  // CHECK: arith.constant -nan : f8E4M3FN
+  %1 = arith.constant -nan : f8E4M3FN
+  // CHECK: arith.constant +nan : f8E4M3FN
+  %2 = arith.constant +qnan : f8E4M3FN
+  // CHECK: arith.constant -nan : f8E4M3FN
+  %3 = arith.constant -qnan : f8E4M3FN
+  return
+}
+
+// NegativeZero NaN encoding (f8E4M3FNUZ): a single, always-negative NaN, spelled
+// `-nan`. The `-qnan` alias normalizes to it.
+// CHECK-LABEL: @nan_negative_zero
+func.func @nan_negative_zero() {
+  // CHECK: arith.constant -nan : f8E4M3FNUZ
+  %0 = arith.constant -nan : f8E4M3FNUZ
+  // CHECK: arith.constant -nan : f8E4M3FNUZ
+  %1 = arith.constant -qnan : f8E4M3FNUZ
+  return
+}
diff --git a/mlir/test/IR/invalid-builtin-attributes.mlir b/mlir/test/IR/invalid-builtin-attributes.mlir
index 0d00b3141fbc6..1f3e41d6ff639 100644
--- a/mlir/test/IR/invalid-builtin-attributes.mlir
+++ b/mlir/test/IR/invalid-builtin-attributes.mlir
@@ -52,6 +52,16 @@ func.func @elementsattr_floattype2() -> () {
 
 // -----
 
+// The diagnostic must point at the float literal, not the following token
+// (the `}` is on the next line, exposing a stale location).
+func.func @float_attr_non_float_type() -> () {
+  // expected-error at below {{floating point value not valid for specified type}}
+  "foo"(){bar = 1.0 : i32
+  } : () -> ()
+}
+
+// -----
+
 func.func @elementsattr_toolarge1() -> () {
   "foo"(){bar = dense<[777]> : tensor<1xi8>} : () -> () // expected-error {{integer constant out of range}}
 }
@@ -675,3 +685,169 @@ func.func @expect_to_parse_literal() {
   %0 = arith.constant dense<[23]> : tensor<1x!unknown<>>
   return
 }
+
+// -----
+
+func.func @hex_float_without_exponent() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.8 : f64
+  return
+}
+
+// -----
+
+func.func @hex_float_missing_exponent_digits() {
+  // expected-error at below {{expected binary exponent in hexadecimal floating point literal}}
+  %0 = arith.constant 0x1.0p : f64
+  return
+}
+
+// -----
+
+func.func @unclosed_nan_literal() {
+  // expected-error at below {{expected ')' in NaN literal}}
+  %0 = arith.constant +nan(0x1 : f32
+  return
+}
+
+// -----
+
+func.func @invalid_nan_payload() {
+  // expected-error at below {{invalid floating point literal}}
+  %0 = arith.constant +nan(0xZZ) : f32
+  return
+}
+
+// -----
+
+func.func @inf_on_integer_type() {
+  // expected-error at below {{floating point value not valid for specified type}}
+  %0 = arith.constant +inf : i32
+  return
+}
+
+// -----
+
+func.func @doubly_signed_inf() {
+  // Doubly-signed literal: a '-' token before an already-signed inf/NaN.
+  // expected-error at below {{floating point literal has more than one sign}}
+  %0 = arith.constant -+inf : f64
+  return
+}
+
+// -----
+
+func.func @inf_on_type_without_inf() {
+  // f4E2M1FN has no Inf encoding: reject, do not crash APFloat.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f4E2M1FN
+  return
+}
+
+// -----
+
+func.func @nan_on_type_without_nan() {
+  // f6E2M3FN has no NaN encoding.
+  // expected-error at below {{floating point type does not support NaN}}
+  %0 = arith.constant +qnan : f6E2M3FN
+  return
+}
+
+// -----
+
+func.func @inf_on_nan_only_type() {
+  // f8E4M3FN has NaN but no Inf.
+  // expected-error at below {{floating point type does not support infinity}}
+  %0 = arith.constant +inf : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @negative_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: reject negatives, do not crash the printer.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -1.0 : f8E8M0FNU
+  return
+}
+
+// -----
+
+func.func @signaling_nan_on_all_ones_type() {
+  // f8E4M3FN (AllOnes) has a single NaN with no signaling bit.
+  // expected-error at below {{floating point type does not support signaling NaN}}
+  %0 = arith.constant +snan(0x1) : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @payload_nan_on_all_ones_type() {
+  // f8E4M3FN (AllOnes) has a single NaN with no payload.
+  // expected-error at below {{floating point type does not support NaN payload}}
+  %0 = arith.constant +nan(0x1) : f8E4M3FN
+  return
+}
+
+// -----
+
+func.func @signaling_nan_on_negative_zero_type() {
+  // f8E4M3FNUZ (NegativeZero) has a single NaN with no signaling bit.
+  // expected-error at below {{floating point type does not support signaling NaN}}
+  %0 = arith.constant +snan(0x1) : f8E4M3FNUZ
+  return
+}
+
+// -----
+
+func.func @payload_nan_on_negative_zero_type() {
+  // f8E4M3FNUZ (NegativeZero) has a single NaN with no payload.
+  // expected-error at below {{floating point type does not support NaN payload}}
+  %0 = arith.constant -nan(0x1) : f8E4M3FNUZ
+  return
+}
+
+// -----
+
+func.func @positive_qnan_on_negative_zero_type() {
+  // f8E4M3FNUZ (NegativeZero) has only a negative NaN.
+  // expected-error at below {{floating point type only supports negative NaN}}
+  %0 = arith.constant +qnan : f8E4M3FNUZ
+  return
+}
+
+// -----
+
+func.func @positive_nan_on_negative_zero_type() {
+  // f8E4M3FNUZ (NegativeZero) has only a negative NaN.
+  // expected-error at below {{floating point type only supports negative NaN}}
+  %0 = arith.constant +nan : f8E4M3FNUZ
+  return
+}
+
+// -----
+
+func.func @negative_nan_on_unsigned_type() {
+  // f8E8M0FNU is unsigned: a negative NaN is rejected like any negative value.
+  // expected-error at below {{floating point type does not support negative values}}
+  %0 = arith.constant -nan : f8E8M0FNU
+  return
+}
+
+// -----
+
+func.func @signaling_nan_on_unsigned_type() {
+  // f8E8M0FNU (AllOnes) has a single NaN with no signaling bit.
+  // expected-error at below {{floating point type does not support signaling NaN}}
+  %0 = arith.constant +snan : f8E8M0FNU
+  return
+}
+
+// -----
+
+func.func @payload_nan_on_unsigned_type() {
+  // f8E8M0FNU (AllOnes) has a single NaN with no payload.
+  // expected-error at below {{floating point type does not support NaN payload}}
+  %0 = arith.constant +nan(0x1) : f8E8M0FNU
+  return
+}
diff --git a/mlir/test/IR/parser.mlir b/mlir/test/IR/parser.mlir
index e10dbf36e2797..1375c4edb11d0 100644
--- a/mlir/test/IR/parser.mlir
+++ b/mlir/test/IR/parser.mlir
@@ -1022,18 +1022,18 @@ func.func @dialect_attribute_with_type() {
 // CHECK-LABEL: @f16_special_values
 func.func @f16_special_values() {
   // F16 NaNs.
-  // CHECK: arith.constant 0x7C01 : f16
+  // CHECK: arith.constant +snan(0x1) : f16
   %0 = arith.constant 0x7C01 : f16
-  // CHECK: arith.constant 0x7FFF : f16
+  // CHECK: arith.constant +nan(0x1FF) : f16
   %1 = arith.constant 0x7FFF : f16
-  // CHECK: arith.constant 0xFFFF : f16
+  // CHECK: arith.constant -nan(0x1FF) : f16
   %2 = arith.constant 0xFFFF : f16
 
   // F16 positive infinity.
-  // CHECK: arith.constant 0x7C00 : f16
+  // CHECK: arith.constant +inf : f16
   %3 = arith.constant 0x7C00 : f16
   // F16 negative infinity.
-  // CHECK: arith.constant 0xFC00 : f16
+  // CHECK: arith.constant -inf : f16
   %4 = arith.constant 0xFC00 : f16
 
   return
@@ -1042,22 +1042,22 @@ func.func @f16_special_values() {
 // CHECK-LABEL: @f32_special_values
 func.func @f32_special_values() {
   // F32 signaling NaNs.
-  // CHECK: arith.constant 0x7F800001 : f32
+  // CHECK: arith.constant +snan(0x1) : f32
   %0 = arith.constant 0x7F800001 : f32
-  // CHECK: arith.constant 0x7FBFFFFF : f32
+  // CHECK: arith.constant +snan(0x3FFFFF) : f32
   %1 = arith.constant 0x7FBFFFFF : f32
 
   // F32 quiet NaNs.
-  // CHECK: arith.constant 0x7FC00000 : f32
+  // CHECK: arith.constant +qnan : f32
   %2 = arith.constant 0x7FC00000 : f32
-  // CHECK: arith.constant 0xFFFFFFFF : f32
+  // CHECK: arith.constant -nan(0x3FFFFF) : f32
   %3 = arith.constant 0xFFFFFFFF : f32
 
   // F32 positive infinity.
-  // CHECK: arith.constant 0x7F800000 : f32
+  // CHECK: arith.constant +inf : f32
   %4 = arith.constant 0x7F800000 : f32
   // F32 negative infinity.
-  // CHECK: arith.constant 0xFF800000 : f32
+  // CHECK: arith.constant -inf : f32
   %5 = arith.constant 0xFF800000 : f32
 
   return
@@ -1066,22 +1066,22 @@ func.func @f32_special_values() {
 // CHECK-LABEL: @f64_special_values
 func.func @f64_special_values() {
   // F64 signaling NaNs.
-  // CHECK: arith.constant 0x7FF0000000000001 : f64
+  // CHECK: arith.constant +snan(0x1) : f64
   %0 = arith.constant 0x7FF0000000000001 : f64
-  // CHECK: arith.constant 0x7FF8000000000000 : f64
+  // CHECK: arith.constant +qnan : f64
   %1 = arith.constant 0x7FF8000000000000 : f64
 
   // F64 quiet NaNs.
-  // CHECK: arith.constant 0x7FF0000001000000 : f64
+  // CHECK: arith.constant +snan(0x1000000) : f64
   %2 = arith.constant 0x7FF0000001000000 : f64
-  // CHECK: arith.constant 0xFFF0000001000000 : f64
+  // CHECK: arith.constant -snan(0x1000000) : f64
   %3 = arith.constant 0xFFF0000001000000 : f64
 
   // F64 positive infinity.
-  // CHECK: arith.constant 0x7FF0000000000000 : f64
+  // CHECK: arith.constant +inf : f64
   %4 = arith.constant 0x7FF0000000000000 : f64
   // F64 negative infinity.
-  // CHECK: arith.constant 0xFFF0000000000000 : f64
+  // CHECK: arith.constant -inf : f64
   %5 = arith.constant 0xFFF0000000000000 : f64
 
   // Check that values that can't be represented with the default format, use
@@ -1095,22 +1095,22 @@ func.func @f64_special_values() {
 // CHECK-LABEL: @bfloat16_special_values
 func.func @bfloat16_special_values() {
   // bfloat16 signaling NaNs.
-  // CHECK: arith.constant 0x7F81 : bf16
+  // CHECK: arith.constant +snan(0x1) : bf16
   %0 = arith.constant 0x7F81 : bf16
-  // CHECK: arith.constant 0xFF81 : bf16
+  // CHECK: arith.constant -snan(0x1) : bf16
   %1 = arith.constant 0xFF81 : bf16
 
   // bfloat16 quiet NaNs.
-  // CHECK: arith.constant 0x7FC0 : bf16
+  // CHECK: arith.constant +qnan : bf16
   %2 = arith.constant 0x7FC0 : bf16
-  // CHECK: arith.constant 0xFFC0 : bf16
+  // CHECK: arith.constant -qnan : bf16
   %3 = arith.constant 0xFFC0 : bf16
 
   // bfloat16 positive infinity.
-  // CHECK: arith.constant 0x7F80 : bf16
+  // CHECK: arith.constant +inf : bf16
   %4 = arith.constant 0x7F80 : bf16
   // bfloat16 negative infinity.
-  // CHECK: arith.constant 0xFF80 : bf16
+  // CHECK: arith.constant -inf : bf16
   %5 = arith.constant 0xFF80 : bf16
 
   return
@@ -1119,22 +1119,22 @@ func.func @bfloat16_special_values() {
 // CHECK-LABEL: @f80_special_values
 func.func @f80_special_values() {
   // F80 signaling NaNs.
-  // CHECK: arith.constant 0x7FFFE000000000000001 : f80
+  // CHECK: arith.constant +nan(0x2000000000000001) : f80
   %0 = arith.constant 0x7FFFE000000000000001 : f80
-  // CHECK: arith.constant 0x7FFFB000000000000011 : f80
+  // CHECK: arith.constant +snan(0x3000000000000011) : f80
   %1 = arith.constant 0x7FFFB000000000000011 : f80
 
   // F80 quiet NaNs.
-  // CHECK: arith.constant 0x7FFFC000000000100000 : f80
+  // CHECK: arith.constant +nan(0x100000) : f80
   %2 = arith.constant 0x7FFFC000000000100000 : f80
-  // CHECK: arith.constant 0x7FFFE000000001000000 : f80
+  // CHECK: arith.constant +nan(0x2000000001000000) : f80
   %3 = arith.constant 0x7FFFE000000001000000 : f80
 
   // F80 positive infinity.
-  // CHECK: arith.constant 0x7FFF8000000000000000 : f80
+  // CHECK: arith.constant +inf : f80
   %4 = arith.constant 0x7FFF8000000000000000 : f80
   // F80 negative infinity.
-  // CHECK: arith.constant 0xFFFF8000000000000000 : f80
+  // CHECK: arith.constant -inf : f80
   %5 = arith.constant 0xFFFF8000000000000000 : f80
 
   return
@@ -1152,14 +1152,14 @@ func.func @f32_potential_precision_loss() {
 
 // CHECK-LABEL: @special_float_values_in_tensors
 func.func @special_float_values_in_tensors() {
-  // CHECK: dense<0xFFFFFFFF> : tensor<4x4xf32>
+  // CHECK: dense<-nan(0x3FFFFF)> : tensor<4x4xf32>
   "foo"(){bar = dense<0xFFFFFFFF> : tensor<4x4xf32>} : () -> ()
-  // CHECK: dense<[{{\[}}0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>
+  // CHECK: dense<[{{\[}}-nan(0x3FFFFF), +inf], [+snan(0x3FFFFF), +snan(0x1)]]> : tensor<2x2xf32>
   "foo"(){bar = dense<[[0xFFFFFFFF, 0x7F800000], [0x7FBFFFFF, 0x7F800001]]> : tensor<2x2xf32>} : () -> ()
-  // CHECK: dense<[0xFFFFFFFF, 0.000000e+00]> : tensor<2xf32>
+  // CHECK: dense<[-nan(0x3FFFFF), 0.000000e+00]> : tensor<2xf32>
   "foo"(){bar = dense<[0xFFFFFFFF, 0.0]> : tensor<2xf32>} : () -> ()
 
-  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [0xFFFFFFFF, 0x7F800001]>
+  // CHECK: sparse<[{{\[}}1, 1, 0], [0, 1, 1]], [-nan(0x3FFFFF), +snan(0x1)]>
   "foo"(){bar = sparse<[[1,1,0],[0,1,1]], [0xFFFFFFFF, 0x7F800001]> : tensor<2x2x2xf32>} : () -> ()
 }
 


From llvmlistbot at llvm.org  Sat Jul 25 16:57:37 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Sat, 25 Jul 2026 16:57:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212073@github.com>
Message-ID: <6a654d71.0c05e68e.272114.4e49@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/212073

From llvmlistbot at llvm.org  Sat Jul 25 16:59:12 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 16:59:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] 7af5c42 - [mlir][tosa] Fix mul/intdiv folds
 crashing on dynamic-shaped results (#212073)
Message-ID: <6a654dd0.c4d3d80a.348a49.3659@mx.google.com>


Author: Victor Perez
Date: 2026-07-25T19:59:07-04:00
New Revision: 7af5c42beb1d76a5335ef5095c6bd58331a882b8

URL: https://github.com/llvm/llvm-project/commit/7af5c42beb1d76a5335ef5095c6bd58331a882b8
DIFF: https://github.com/llvm/llvm-project/commit/7af5c42beb1d76a5335ef5095c6bd58331a882b8.diff

LOG: [mlir][tosa] Fix mul/intdiv folds crashing on dynamic-shaped results (#212073)

`tosa.mul` and `tosa.intdiv` fold constant splat operands by building a
`DenseElementsAttr` of the result type, which asserts when that type has
a dynamic shape. When the operands are constants but the result type is
dynamic, folding aborted. Guard `mulBinaryFolder` and the `IntDivOp`
splat path on a static result shape, matching the other fold paths in
these ops.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>

Added: 
    

Modified: 
    mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
    mlir/test/Dialect/Tosa/canonicalize.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
index 19f35757bdd83..8937875df7fc5 100644
--- a/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
+++ b/mlir/lib/Dialect/Tosa/IR/TosaCanonicalizations.cpp
@@ -1796,7 +1796,7 @@ OpFoldResult IntDivOp::fold(FoldAdaptor adaptor) {
   }
 
   if (rhsAttr && lhsAttr && rhsAttr.isSplat() && lhsAttr.isSplat() &&
-      llvm::isa<IntegerType>(resultETy)) {
+      llvm::isa<IntegerType>(resultETy) && resultTy.hasStaticShape()) {
     APInt l = lhsAttr.getSplatValue<APInt>();
     APInt r = rhsAttr.getSplatValue<APInt>();
     if (!r.isZero()) {
@@ -1841,6 +1841,9 @@ std::optional<APInt> mulInt(APInt lhs, APInt rhs, int32_t shift,
 
 DenseElementsAttr mulBinaryFolder(DenseElementsAttr lhs, DenseElementsAttr rhs,
                                   RankedTensorType ty, int32_t shift) {
+  // A constant result can only be built for a statically-shaped type.
+  if (!ty.hasStaticShape())
+    return {};
   if (rhs && lhs && rhs.isSplat() && lhs.isSplat()) {
     if (llvm::isa<IntegerType>(ty.getElementType())) {
       APInt l = lhs.getSplatValue<APInt>();

diff  --git a/mlir/test/Dialect/Tosa/canonicalize.mlir b/mlir/test/Dialect/Tosa/canonicalize.mlir
index 585589ac1405a..7c9dd260ca497 100644
--- a/mlir/test/Dialect/Tosa/canonicalize.mlir
+++ b/mlir/test/Dialect/Tosa/canonicalize.mlir
@@ -1441,6 +1441,31 @@ func.func @no_fold_mul_result_exceeds_i32() -> tensor<i32> {
 
 // -----
 
+// A dynamically-shaped result cannot be a constant; folding must bail instead
+// of building a DenseElementsAttr of a non-static shape (which would assert).
+// CHECK-LABEL: @no_fold_mul_dynamic_result
+// CHECK: tosa.mul
+func.func @no_fold_mul_dynamic_result() -> tensor<?xf32> {
+    %0 = "tosa.const"() <{values = dense<2.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %1 = "tosa.const"() <{values = dense<3.0> : tensor<4xf32>}> : () -> tensor<4xf32>
+    %2 = "tosa.const"() <{values = dense<0> : tensor<1xi8>}> : () -> tensor<1xi8>
+    %3 = tosa.mul %0, %1, %2 : (tensor<4xf32>, tensor<4xf32>, tensor<1xi8>) -> tensor<?xf32>
+    return %3 : tensor<?xf32>
+}
+
+// -----
+
+// CHECK-LABEL: @no_fold_intdiv_dynamic_result
+// CHECK: tosa.intdiv
+func.func @no_fold_intdiv_dynamic_result() -> tensor<?xi32> {
+    %0 = "tosa.const"() <{values = dense<6> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %1 = "tosa.const"() <{values = dense<2> : tensor<4xi32>}> : () -> tensor<4xi32>
+    %2 = tosa.intdiv %0, %1 : (tensor<4xi32>, tensor<4xi32>) -> tensor<?xi32>
+    return %2 : tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_fold_i1_to_i32_cast
 // CHECK: %[[OUT:.*]] = "tosa.const"() <{values = dense<1> : tensor<i32>}> : () -> tensor<i32>
 // CHECK: return %[[OUT]] : tensor<i32>


        

From llvmlistbot at llvm.org  Sat Jul 25 16:59:14 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 16:59:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212073@github.com>
Message-ID: <6a654dd2.510f576d.23298e.3930@mx.google.com>


https://github.com/victor-eds closed https://github.com/llvm/llvm-project/pull/212073

From llvmlistbot at llvm.org  Sat Jul 25 17:04:46 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 17:04:46 -0700 (PDT)
Subject: [Mlir-commits] [flang] [mlir] [mlir] Support human-readable float
 literals in the textual IR (PR #210422)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210422@github.com>
Message-ID: <6a654f1e.b4a331f0.1c45b5.6723@mx.google.com>


https://github.com/victor-eds edited https://github.com/llvm/llvm-project/pull/210422

From llvmlistbot at llvm.org  Sat Jul 25 17:36:34 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sat, 25 Jul 2026 17:36:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a655692.2915925c.20d2c4.837f@mx.google.com>


victor-eds wrote:

@Hardcode84 @kuhar 

> Folding `div(x, 0) -> poison` may be more profitable, but then we have an ordering problem between `div(x, x)` and `div(x, 0)` foldings. Can you check alive2 and llvm proper if 1. poison folding is actually allowed, 2. how they handle ordering.

Good point. [Here's the LLVM equivalent patterns](https://godbolt.org/z/dzTjYjs4e). LLVM *does* apply the `div(x, 0) -> poison` transformation you mention, and it in fact applies it with higher priority than the ones I am proposing here. Note I didn't originally add `div(x, 0) -> poison` as that would require adding a dependency with the UB dialect. I can do it if we agree that's reasonable.

Also, as requested: Alive2 verification (all report "Transformation seems to be correct!"):

| Fold | `udiv` | `sdiv` | `urem` | `srem` |
| :--- | :---: | :---: | :---: | :---: |
| `X / 0` -> `poison` | [Correct](https://alive2.llvm.org/ce/z/AyT9Rz) | [Correct](https://alive2.llvm.org/ce/z/-AevcK) | [Correct](https://alive2.llvm.org/ce/z/VLDE5N) | [Correct](https://alive2.llvm.org/ce/z/APvEKa) |
| `X / X` -> `1` / `0` | [Correct](https://alive2.llvm.org/ce/z/YjZCEA) | [Correct](https://alive2.llvm.org/ce/z/BSWQrm) | [Correct](https://alive2.llvm.org/ce/z/gx5sRH) | [Correct](https://alive2.llvm.org/ce/z/VUYkr4) |
| `0 / X` -> `0` | [Correct](https://alive2.llvm.org/ce/z/keqeCZ) | [Correct](https://alive2.llvm.org/ce/z/jyf5wm) | [Correct](https://alive2.llvm.org/ce/z/fZa6ND) | [Correct](https://alive2.llvm.org/ce/z/fo6ss8) |

And the ordering overlap, [udiv 0, 0 -> poison](https://alive2.llvm.org/ce/z/AEW4ms).

I'll add these tests to the commit message/PR when we agree on the extent of the changes.


https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 17:38:43 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Sat, 25 Jul 2026 17:38:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a655713.af6ba3ed.1989e7.13e1@mx.google.com>


kuhar wrote:

I think 0 is fine for now

https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 17:38:44 2026
From: llvmlistbot at llvm.org (Jakub Kuderski)
Date: Sat, 25 Jul 2026 17:38:44 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a655714.507fc713.3a4929.32c6@mx.google.com>


https://github.com/kuhar approved this pull request.


https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 17:48:03 2026
From: llvmlistbot at llvm.org (Ivan Butygin)
Date: Sat, 25 Jul 2026 17:48:03 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a655943.3cce8983.19fb00.0032@mx.google.com>


https://github.com/Hardcode84 approved this pull request.

Yeah, 0 is fine for now, can you add a comment/TODO to code about the poison.

https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sat Jul 25 18:16:49 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Sat, 25 Jul 2026 18:16:49 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a656001.4a342cbc.e86ca.93a9@mx.google.com>


flemairen6 wrote:

> Do you know why global op is raised to memref<1xT> instead of memref<T>? 

That's a good question, I don't remember and that doesn't sound good to me. I'll have a look over the week but thanks for matching the existing behaviour, I'll open a PR to change it later.

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Sat Jul 25 18:16:59 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Sat, 25 Jul 2026 18:16:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a65600b.824e11a6.2fe22.5678@mx.google.com>


https://github.com/flemairen6 approved this pull request.

LGTM

https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Sat Jul 25 18:17:14 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Sat, 25 Jul 2026 18:17:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a65601a.160324b0.28c9e1.3407@mx.google.com>


https://github.com/flemairen6 updated https://github.com/llvm/llvm-project/pull/196613

>From 4eb1075004eafe159cdb9b436cb5a4fb9a2a33b4 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 00:20:59 -0400
Subject: [PATCH 1/4] [MLIR][WasmSSA] Add global_set op

Adds the `wasmssa.global_set` operation, which writes a value to a
mutable global variable. The op verifies that the referenced symbol is
a `wasmssa.global` or `wasmssa.import_global` marked mutable, and that
the operand type matches the global's declared type.
---
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 19 +++++++
 mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp    | 38 +++++++++++++
 .../Dialect/WasmSSA/custom_parser/global.mlir | 12 ++++
 mlir/test/Dialect/WasmSSA/global-invalid.mlir | 55 +++++++++++++++++++
 4 files changed, 124 insertions(+)

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index bfa0953f50ac8..e2fb642433266 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -381,6 +381,25 @@ def WasmSSA_GlobalGetOp : WasmSSA_Op<"global_get", [DeclareOpInterfaceMethods<Sy
   let assemblyFormat = "$global attr-dict `:` type($global_val)";
 }
 
+def WasmSSA_GlobalSetOp : WasmSSA_Op<"global_set", [
+    DeclareOpInterfaceMethods<SymbolUserOpInterface>]> {
+  let summary = "Sets the value of the mutable global passed as argument.";
+  let description = [{
+     Sets the value of the mutable global referenced by the symbol to the given
+     value.
+
+     Example:
+
+     ```mlir
+     // Sets the value of `@global_0` to `%v`.
+     wasmssa.global_set @global_0 to %v : i32
+     ```
+    }];
+  let arguments = (ins FlatSymbolRefAttr: $global,
+                       WasmSSA_ValType: $value);
+  let assemblyFormat = "$global `to` $value `:` type($value) attr-dict";
+}
+
 def WasmSSA_IfOp : WasmSSA_Op<"if", [Terminator,
     DeclareOpInterfaceMethods<LabelLevelOpInterface>]> {
   let summary = "Execute the if region if condition value is non-zero, the else region otherwise.";
diff --git a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
index 08de887a56a97..eab9c7a9d53a7 100644
--- a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
+++ b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
@@ -331,6 +331,44 @@ GlobalGetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// GlobalSetOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+GlobalSetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+  Operation *symTabOp = SymbolTable::getNearestSymbolTable(*this);
+  StringRef referencedSymbol = getGlobal();
+  Operation *definitionOp = symbolTable.lookupSymbolIn(
+      symTabOp, StringAttr::get(this->getContext(), referencedSymbol));
+  if (!definitionOp)
+    return emitError() << "symbol @" << referencedSymbol << " is undefined";
+
+  Type globalType;
+  bool isMutable = false;
+  if (auto global = dyn_cast<GlobalOp>(definitionOp)) {
+    globalType = global.getType();
+    isMutable = global.getIsMutable();
+  } else if (auto globalImport = dyn_cast<GlobalImportOp>(definitionOp)) {
+    globalType = globalImport.getType();
+    isMutable = globalImport.getIsMutable();
+  } else {
+    return emitError() << "symbol @" << referencedSymbol
+                       << " is not a global symbol";
+  }
+
+  if (!isMutable)
+    return emitError("global.set target must be mutable");
+
+  Type valueType = getValue().getType();
+  if (globalType != valueType)
+    return emitError("global.set value type does not match target global "
+                     "type: expected ")
+           << globalType << " but got " << valueType;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // GlobalImportOp
 //===----------------------------------------------------------------------===//
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
index a25abbd2a8662..6441731a0a335 100644
--- a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
@@ -19,6 +19,12 @@ module {
     %0 = wasmssa.global_get @global_0 : i32
     wasmssa.return %0 : i32
   }
+
+  wasmssa.func @set_global() {
+    %0 = wasmssa.const 7 : i32
+    wasmssa.global_set @global_2 to %0 : i32
+    wasmssa.return
+  }
 }
 
 // CHECK-LABEL:   wasmssa.import_global "from_js" from "env" as @global_0 : i32
@@ -42,3 +48,9 @@ module {
 // CHECK:           %[[VAL_0:.*]] = wasmssa.global_get @global_0 : i32
 // CHECK:           wasmssa.return %[[VAL_0]] : i32
 // CHECK:         }
+
+// CHECK-LABEL:   wasmssa.func @set_global() {
+// CHECK:           %[[VAL_0:.*]] = wasmssa.const 7 : i32
+// CHECK:           wasmssa.global_set @global_2 to %[[VAL_0]] : i32
+// CHECK:           wasmssa.return
+// CHECK:         }
diff --git a/mlir/test/Dialect/WasmSSA/global-invalid.mlir b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
index c5bc606fd13f3..600b2f0715fdc 100644
--- a/mlir/test/Dialect/WasmSSA/global-invalid.mlir
+++ b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
@@ -40,3 +40,58 @@ module {
     wasmssa.return %0 : i32
   }
 }
+
+// -----
+
+module {
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @missing is undefined}}
+    wasmssa.global_set @missing to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i32 : {
+    %0 = wasmssa.const 0 : i32
+    wasmssa.return %0 : i32
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set target must be mutable}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i64 mutable : {
+    %0 = wasmssa.const 0 : i64
+    wasmssa.return %0 : i64
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set value type does not match target global type: expected 'i64' but got 'i32'}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.func @not_global() {
+    wasmssa.return
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @not_global is not a global symbol}}
+    wasmssa.global_set @not_global to %0 : i32
+    wasmssa.return
+  }
+}

>From c9c103386c84a4756f84bbf3a60cbfb07076b541 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Fri, 8 May 2026 15:05:35 -0400
Subject: [PATCH 2/4] [MLIR][WasmSSA] Add select op

Adds the `wasmssa.select` operation, a Wasm value-typed ternary that
returns one of two operands based on an i32 condition. The op uses the
`AllTypesMatch` trait to require both branches and the result share the
same Wasm value type, and is marked `Pure` since it has no side effects.

Also adds the `SideEffectInterfaces` td/header includes needed by the
`Pure` trait.
---
 .../include/mlir/Dialect/WasmSSA/IR/WasmSSA.h |  1 +
 .../mlir/Dialect/WasmSSA/IR/WasmSSAOps.td     | 24 ++++++++++++
 .../Dialect/WasmSSA/custom_parser/select.mlir | 37 +++++++++++++++++++
 3 files changed, 62 insertions(+)
 create mode 100644 mlir/test/Dialect/WasmSSA/custom_parser/select.mlir

diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
index 64391d807c633..64d480c065b79 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
@@ -38,6 +38,7 @@
 #include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
 
 //===----------------------------------------------------------------------===//
 // WebAssembly Constraints
diff --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index e2fb642433266..3b3c3e8825fef 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -16,6 +16,7 @@ include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.td"
 
 include "mlir/Interfaces/FunctionInterfaces.td"
 include "mlir/Interfaces/InferTypeOpInterface.td"
+include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/IR/BuiltinAttributeInterfaces.td"
 include "mlir/IR/SymbolInterfaces.td"
 
@@ -628,6 +629,29 @@ def WasmSSA_ReturnOp : WasmSSA_Op<"return", [Terminator]> {
   ];
 }
 
+def WasmSSA_SelectOp : WasmSSA_Op<"select", [
+    AllTypesMatch<["true_value", "false_value", "result"]>,
+    Pure]> {
+  let summary = "Select between two values based on a condition.";
+  let description = [{
+     Returns `true_value` if `condition` is non-zero, `false_value` otherwise.
+     Both operands and the result must share the same Wasm value type.
+
+     Example:
+
+     ```mlir
+     // Returns %a if %cond is non-zero, otherwise %b.
+     %r = wasmssa.select %cond, %a, %b : i32
+     ```
+    }];
+  let arguments = (ins WasmSSA_ValType: $true_value,
+                       WasmSSA_ValType: $false_value,
+                       I32: $condition);
+  let results = (outs WasmSSA_ValType: $result);
+  let assemblyFormat =
+      "$condition `,` $true_value `,` $false_value attr-dict `:` type($result)";
+}
+
 // ---- Numeric ops
 
 class WasmSSA_BinaryNumericalOp<string mnemonic, string summaryStr, string descStr,
diff --git a/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
new file mode 100644
index 0000000000000..80ced11c01bc3
--- /dev/null
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
@@ -0,0 +1,37 @@
+// RUN: mlir-opt %s | FileCheck %s
+
+module {
+  wasmssa.func @select_i32(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to i32>,
+                           %b: !wasmssa<local ref to i32>) -> i32 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to i32
+    %2 = wasmssa.local_get %b : ref to i32
+    %r = wasmssa.select %0, %1, %2 : i32
+    wasmssa.return %r : i32
+  }
+
+  wasmssa.func @select_f64(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to f64>,
+                           %b: !wasmssa<local ref to f64>) -> f64 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to f64
+    %2 = wasmssa.local_get %b : ref to f64
+    %r = wasmssa.select %0, %1, %2 : f64
+    wasmssa.return %r : f64
+  }
+}
+
+// CHECK-LABEL:   wasmssa.func @select_i32(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : i32
+// CHECK:           wasmssa.return %[[R]] : i32
+
+// CHECK-LABEL:   wasmssa.func @select_f64(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : f64
+// CHECK:           wasmssa.return %[[R]] : f64

>From f8775f7b26b46c32ea6ea09ebb4dbb3bb87b4302 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Thu, 23 Jul 2026 11:07:20 -0400
Subject: [PATCH 3/4] [MLIR][WasmSSA] Add global_set conversion to
 RaiseWasmMLIR

Lowers wasmssa.global_set to memref.get_global + memref.store at index
0, matching the memref<1xT> shape emitted by the existing global
conversions.
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 18 ++++++++++
 .../RaiseWasm/wasm-global-set-to-memref.mlir  | 35 +++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index b4b5006352c11..d719c42459d9d 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -671,6 +671,23 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmGlobalSetOpConversion : OpConversionPattern<GlobalSetOp> {
+  using OpConversionPattern::OpConversionPattern;
+  LogicalResult
+  matchAndRewrite(GlobalSetOp globalSetOp, GlobalSetOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = globalSetOp.getLoc();
+    auto globalPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, adaptor.getValue().getType()),
+        globalSetOp.getGlobal());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    rewriter.replaceOpWithNewOp<memref::StoreOp>(
+        globalSetOp, adaptor.getValue(), globalPtr.getResult(),
+        ValueRange{idx.getResult()});
+    return success();
+  }
+};
+
 struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -876,6 +893,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmGeSIOpConversion,
            WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
+           WasmGlobalSetOpConversion,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
            WasmGtOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
new file mode 100644
index 0000000000000..937f7efd006ba
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
@@ -0,0 +1,35 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+wasmssa.global @global_i32 i32 mutable : {
+  %0 = wasmssa.const 10 : i32
+  wasmssa.return %0 : i32
+}
+
+wasmssa.global @global_f64 f64 mutable : {
+  %0 = wasmssa.const 3.14 : f64
+  wasmssa.return %0 : f64
+}
+
+// CHECK-LABEL:   func.func @set_global_i32() {
+wasmssa.func exported @set_global_i32() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 42 : i32
+  %0 = wasmssa.const 42 : i32
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_i32 : memref<1xi32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xi32>
+  wasmssa.global_set @global_i32 to %0 : i32
+// CHECK:           return
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @set_global_f64() {
+wasmssa.func exported @set_global_f64() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 2.500000e-01 : f64
+  %0 = wasmssa.const 0.25 : f64
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_f64 : memref<1xf64>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xf64>
+  wasmssa.global_set @global_f64 to %0 : f64
+// CHECK:           return
+  wasmssa.return
+}

>From c4f7eb5c4035e1c23572f895f164adcd156cb2f8 Mon Sep 17 00:00:00 2001
From: Byeongjee Kang <byeongjee.kang at gmail.com>
Date: Thu, 23 Jul 2026 11:13:33 -0400
Subject: [PATCH 4/4] [MLIR][WasmSSA] Add select conversion to RaiseWasmMLIR

Lowers wasmssa.select to arith.select, normalizing the i32 condition
to i1 via arith.cmpi ne 0, matching the existing idiom for condition
handling in this pass.
---
 .../Conversion/RaiseWasm/RaiseWasmMLIR.cpp    | 19 ++++++
 .../wasm-select-to-arith-select.mlir          | 65 +++++++++++++++++++
 2 files changed, 84 insertions(+)
 create mode 100644 mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir

diff --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index d719c42459d9d..c26a3b038e918 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -813,6 +813,24 @@ struct WasmReturnOpConversion : OpConversionPattern<ReturnOp> {
   }
 };
 
+struct WasmSelectOpConversion : OpConversionPattern<SelectOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SelectOp selectOp, SelectOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = selectOp.getLoc();
+    auto zero =
+        arith::ConstantOp::create(rewriter, loc, rewriter.getI32IntegerAttr(0));
+    auto flag = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::ne,
+                                      adaptor.getCondition(), zero.getResult());
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(selectOp, flag.getResult(),
+                                                 adaptor.getTrueValue(),
+                                                 adaptor.getFalseValue());
+    return success();
+  }
+};
+
 struct RaiseWasmMLIRPass : public impl::RaiseWasmMLIRBase<RaiseWasmMLIRPass> {
   void runOnOperation() override {
     ConversionTarget target{getContext()};
@@ -924,6 +942,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmReturnOpConversion,
            WasmRotlOpConversion,
            WasmRotrOpConversion,
+           WasmSelectOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,
diff --git a/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
new file mode 100644
index 0000000000000..ed5d15f87d70f
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
@@ -0,0 +1,65 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @select_i32() -> i32 {
+wasmssa.func @select_i32() -> i32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i32
+  %a = wasmssa.const 12 : i32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i32
+  %b = wasmssa.const 50 : i32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i32
+  %r = wasmssa.select %cond, %a, %b : i32
+// CHECK:           return %[[RES]] : i32
+  wasmssa.return %r : i32
+}
+
+// CHECK-LABEL:   func.func @select_i64() -> i64 {
+wasmssa.func @select_i64() -> i64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i64
+  %a = wasmssa.const 12 : i64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i64
+  %b = wasmssa.const 50 : i64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i64
+  %r = wasmssa.select %cond, %a, %b : i64
+// CHECK:           return %[[RES]] : i64
+  wasmssa.return %r : i64
+}
+
+// CHECK-LABEL:   func.func @select_f32() -> f32 {
+wasmssa.func @select_f32() -> f32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 1.250000e-01 : f32
+  %a = wasmssa.const 0.125 : f32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.500000e-01 : f32
+  %b = wasmssa.const 0.25 : f32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f32
+  %r = wasmssa.select %cond, %a, %b : f32
+// CHECK:           return %[[RES]] : f32
+  wasmssa.return %r : f32
+}
+
+// CHECK-LABEL:   func.func @select_f64() -> f64 {
+wasmssa.func @select_f64() -> f64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 3.140000e+00 : f64
+  %a = wasmssa.const 3.14 : f64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.718000e+00 : f64
+  %b = wasmssa.const 2.718 : f64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f64
+  %r = wasmssa.select %cond, %a, %b : f64
+// CHECK:           return %[[RES]] : f64
+  wasmssa.return %r : f64
+}


From llvmlistbot at llvm.org  Sat Jul 25 18:17:48 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Sat, 25 Jul 2026 18:17:48 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a65603c.0318b02e.530b7.adb8@mx.google.com>


https://github.com/flemairen6 auto_merge_enabled https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Sat Jul 25 18:27:23 2026
From: llvmlistbot at llvm.org (LLVM Continuous Integration)
Date: Sat, 25 Jul 2026 18:27:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][tosa] Fix mul/intdiv folds crashing on
 dynamic-shaped results (PR #212073)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212073@github.com>
Message-ID: <6a65627b.0f490953.196d5c.aabf@mx.google.com>


llvm-ci wrote:

LLVM Buildbot has detected a new failure on builder `clang-aarch64-sve-vla-2stage` running on `linaro-g3-04` while building `mlir` at step 12 "ninja check 2".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/41/builds/12326

<details>
<summary>Here is the relevant piece of the build log for the reference</summary>

```
Step 12 (ninja check 2) failure: stage 2 checked (failure)
******************** TEST 'Clang :: ClangScanDeps/logging-two-threads.c' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 5
rm -rf /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: rm -rf /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# RUN: at line 6
split-file /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# executed command: split-file /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp
# RUN: at line 7
sed -e "s|DIR|/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g" /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template > /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json
# executed command: sed -e 's|DIR|/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp|g' /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json.template
# RUN: at line 9
/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/clang-scan-deps -compilation-database /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json    -format experimental-full -log-path=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/clang-scan-deps -compilation-database /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/cdb.json -format experimental-full -log-path=/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log -j 2 -o /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/deps.json
# RUN: at line 13
/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1 < /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU1
# RUN: at line 14
/home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2 < /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TU2
# RUN: at line 24
grep -c "pcm_compile:" /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# executed command: grep -c pcm_compile: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=COMPILES
# RUN: at line 25
grep -c "pcm_write:" /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=PCMWRITES
# executed command: grep -c pcm_write: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=PCMWRITES
# RUN: at line 26
grep -c "timestamp_write:" /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log | /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TSWRITES
# executed command: grep -c timestamp_write: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/tools/clang/test/ClangScanDeps/Output/logging-two-threads.c.tmp/scan.log
# executed command: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c --check-prefix=TSWRITES
# .---command stderr------------
# | /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c:29:14: error: TSWRITES: expected string not found in input
# | // TSWRITES: {{^2$}}
# |              ^
# | <stdin>:1:1: note: scanning from here
# | 3
# | ^
# | 
# | Input file: <stdin>
# | Check file: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla-2stage/llvm/clang/test/ClangScanDeps/logging-two-threads.c
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
...

```

</details>

https://github.com/llvm/llvm-project/pull/212073

From llvmlistbot at llvm.org  Sat Jul 25 18:32:04 2026
From: llvmlistbot at llvm.org (Mingfei Guo)
Date: Sat, 25 Jul 2026 18:32:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Check access coverage before
 fusion privatization (PR #212080)
Message-ID: <llvm.org/llvm/llvm-project/pull/212080@github.com>


https://github.com/guoriyue created https://github.com/llvm/llvm-project/pull/212080

This fixes an affine loop fusion bug where privatization could create a buffer that is too small for consumer accesses.

The pass now checks that producer writes cover all consumer reads. If not, fusion continues but unsafe privatization is skipped.

>From 8191701fd1941593b29d79f30fe7df1cd9b3c724 Mon Sep 17 00:00:00 2001
From: Mingfei Guo <1800012773 at pku.edu.cn>
Date: Sat, 25 Jul 2026 13:03:44 -0700
Subject: [PATCH] [MLIR][Affine] Check access coverage before fusion
 privatization

---
 .../Dialect/Affine/Transforms/LoopFusion.cpp  | 214 +++++++++++++-----
 mlir/test/Dialect/Affine/loop-fusion-4.mlir   | 126 +++++++++++
 mlir/test/mlir-runner/affine-loop-fusion.mlir |  69 ++++++
 3 files changed, 356 insertions(+), 53 deletions(-)
 create mode 100644 mlir/test/mlir-runner/affine-loop-fusion.mlir

diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..d84fac437faca 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -12,6 +12,7 @@
 
 #include "mlir/Dialect/Affine/Transforms/Passes.h"
 
+#include "mlir/Dialect/Affine/Analysis/AffineAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/AffineStructures.h"
 #include "mlir/Dialect/Affine/Analysis/LoopAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/Utils.h"
@@ -272,6 +273,77 @@ getDominanceFilterForPrivateMemRefRepl(Block *sliceInsertionBlock,
   return firstAncestor;
 }
 
+static std::optional<presburger::IntegerRelation>
+getAccessRelationForPrivateBuffer(Operation *op, unsigned loopDepth) {
+  if (!isa<AffineLoadOp, AffineStoreOp>(op))
+    return std::nullopt;
+
+  // Reject domains that getAccessRelation cannot represent exactly.
+  Operation *ancestor = op->getParentOp();
+  for (; ancestor && !ancestor->hasTrait<OpTrait::AffineScope>();
+       ancestor = ancestor->getParentOp()) {
+    if (auto forOp = dyn_cast<AffineForOp>(ancestor)) {
+      if (forOp.getStepAsInt() != 1 && !forOp.hasConstantLowerBound())
+        return std::nullopt;
+    } else if (auto parallelOp = dyn_cast<AffineParallelOp>(ancestor)) {
+      if (!llvm::all_of(parallelOp.getSteps(),
+                        [](int64_t step) { return step == 1; }))
+        return std::nullopt;
+    } else {
+      return std::nullopt;
+    }
+  }
+  if (!ancestor)
+    return std::nullopt;
+
+  presburger::IntegerRelation relation(
+      presburger::PresburgerSpace::getRelationSpace());
+  if (failed(MemRefAccess(op).getAccessRelation(relation)))
+    return std::nullopt;
+
+  if (getNestingDepth(op) != relation.getNumDomainVars() ||
+      loopDepth > relation.getNumDomainVars())
+    return std::nullopt;
+
+  relation.convertToLocal(presburger::VarKind::Domain, loopDepth,
+                          relation.getNumDomainVars());
+  return relation;
+}
+
+/// Returns true if every load is covered by `store` in each private-buffer
+/// instance.
+static bool areLoadsCoveredByStore(Operation *store,
+                                   ArrayRef<Operation *> loads,
+                                   unsigned loopDepth) {
+  assert(store && "expected producer store");
+  assert(!loads.empty() && "expected consumer load");
+
+  auto write = getAccessRelationForPrivateBuffer(store, loopDepth);
+  if (!write)
+    return false;
+  for (Operation *load : loads) {
+    auto read = getAccessRelationForPrivateBuffer(load, loopDepth);
+    if (!read ||
+        !write->getSpace().isAligned(read->getSpace(),
+                                     presburger::VarKind::Domain) ||
+        write->getNumRangeVars() != read->getNumRangeVars())
+      return false;
+
+    write->mergeAndAlignSymbols(*read);
+    if (!read->isSubsetOf(*write))
+      return false;
+  }
+  return true;
+}
+
+static bool haveSameAccessFunction(ArrayRef<Operation *> stores) {
+  assert(!stores.empty() && "expected producer store");
+  MemRefAccess first(cast<AffineWriteOpInterface>(stores.front()));
+  return llvm::all_of(llvm::drop_begin(stores), [&](Operation *store) {
+    return first == MemRefAccess(cast<AffineWriteOpInterface>(store));
+  });
+}
+
 /// Returns the amount of additional (redundant) computation that will be done
 /// as a fraction of the total computation if `srcForOp` is fused into
 /// `dstForOp` at depth `depth`. The method returns the compute cost of the
@@ -327,41 +399,60 @@ static std::optional<double> getAdditionalComputeFraction(
 // Creates and returns a private (single-user) memref for fused loop rooted at
 // 'forOp', with (potentially reduced) memref size based on the memref region
 // written to by `storeOps` at depth 'dstLoopDepth'. 'sliceInsertionBlock'
-// specifies the block in which the slice was/will be inserted. The method
-// expects that all stores ops to the memref have the same access function.
-// Returns nullptr if the creation failed.
+// specifies the block in which the slice was/will be inserted. Returns nullptr
+// unless the fused producer covers every consumer access that would be
+// redirected to the private memref.
 static Value createPrivateMemRef(AffineForOp forOp,
                                  ArrayRef<Operation *> storeOps,
+                                 Operation *fusedProducerStore,
                                  unsigned dstLoopDepth,
                                  std::optional<unsigned> fastMemorySpace,
                                  Block *sliceInsertionBlock,
                                  uint64_t localBufSizeThreshold) {
   assert(!storeOps.empty() && "no source stores supplied");
 
-  // Check if all stores have the same access function; we only support this
-  // case.
-  // TODO: Use union of memref write regions to compute private memref footprint
-  // for store ops with different access functions.
-  if (storeOps.size() > 1 &&
-      !std::equal(std::next(storeOps.begin()), storeOps.end(), storeOps.begin(),
-                  [](Operation *a, Operation *b) {
-                    MemRefAccess aM(cast<AffineWriteOpInterface>(a));
-                    MemRefAccess bM(cast<AffineWriteOpInterface>(b));
-                    return aM == bM;
-                  })) {
+  // TODO: Use a union of write regions to support stores with different access
+  // functions.
+  if (!fusedProducerStore)
+    return nullptr;
+  if (!haveSameAccessFunction(storeOps)) {
     LDBG() << "Private memref creation unsupported for multiple producer "
-           << "stores with different access functions.";
+              "stores with different access functions.";
     return nullptr;
   }
 
-  Operation *srcStoreOp = storeOps[0];
+  Operation *srcStoreOp = storeOps.front();
+  Value oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
+  if (cast<AffineWriteOpInterface>(fusedProducerStore).getMemRef() != oldMemRef)
+    return nullptr;
+
+  // Match the exact set of users that replaceAllMemRefUsesWith will rewrite.
+  // Privatization is only legal when the fused producer writes every element
+  // read from each private-buffer instance.
+  Operation *domFilter =
+      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
+  DominanceInfo domInfo(domFilter->getParentOfType<FunctionOpInterface>());
+  SmallVector<Operation *, 4> loads;
+  for (Operation *user : oldMemRef.getUsers()) {
+    if (!domInfo.dominates(domFilter, user) ||
+        hasSingleEffect<MemoryEffects::Free>(user, oldMemRef))
+      continue;
+    if (isa<AffineLoadOp>(user)) {
+      loads.push_back(user);
+      continue;
+    }
+    if (!isa<AffineStoreOp>(user) || !llvm::is_contained(storeOps, user))
+      return nullptr;
+  }
+  if (loads.empty() ||
+      !areLoadsCoveredByStore(fusedProducerStore, loads, dstLoopDepth))
+    return nullptr;
 
   // Create builder to insert alloc op just before 'forOp'.
   OpBuilder b(forOp);
   // Builder to create constants at the top level.
   OpBuilder top(forOp->getParentRegion());
   // Create new memref type based on slice bounds.
-  auto oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
   auto oldMemRefType = cast<MemRefType>(oldMemRef.getType());
   unsigned rank = oldMemRefType.getRank();
 
@@ -441,12 +532,8 @@ static Value createPrivateMemRef(AffineForOp forOp,
       AffineMap::get(outerIVs.size() + rank, 0, remapExprs, forOp.getContext());
 
   // Replace all users of 'oldMemRef' with 'newMemRef'.
-  Operation *domFilter =
-      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
   auto userFilterFn = [&](Operation *user) {
-    auto domInfo = std::make_unique<DominanceInfo>(
-        domFilter->getParentOfType<FunctionOpInterface>());
-    return domInfo->dominates(domFilter, user);
+    return domInfo.dominates(domFilter, user);
   };
   LogicalResult res = replaceAllMemRefUsesWith(
       oldMemRef, newMemRef, /*extraIndices=*/{}, indexRemap,
@@ -1075,17 +1162,22 @@ struct GreedyFusion {
             srcId, dstId, bestSlice, fusedLoopInsPoint, srcEscapingMemRefs,
             *mdg);
 
-        DenseSet<Value> privateMemrefs;
+        DenseSet<Value> privateMemRefCandidates;
         for (Value memref : producerConsumerMemrefs) {
           if (canCreatePrivateMemRef(memref, srcEscapingMemRefs, srcId, dstId,
                                      removeSrcNode)) {
-            // Create a private version of this memref.
-            LDBG() << "Creating private memref for " << memref;
-            // Create a private version of this memref.
-            privateMemrefs.insert(memref);
+            LDBG() << "Considering private memref for " << memref;
+            privateMemRefCandidates.insert(memref);
           }
         }
 
+        DenseSet<Operation *> preexistingDstStores;
+        if (!privateMemRefCandidates.empty()) {
+          dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
+            preexistingDstStores.insert(storeOp.getOperation());
+          });
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1098,35 +1190,51 @@ struct GreedyFusion {
         if (fusedLoopInsPoint != dstAffineForOp)
           dstAffineForOp->moveBefore(fusedLoopInsPoint);
 
-        // Update edges between 'srcNode' and 'dstNode'.
-        mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
-                         removeSrcNode);
-
-        // Create private memrefs.
-        if (!privateMemrefs.empty()) {
-          // Note the block into which fusion was performed. This can be used to
-          // place `alloc`s that create private memrefs.
-          Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
-
-          // Gather stores for all the private-to-be memrefs.
-          DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        // Privatization rewrites the consumer accesses to a buffer sized from
+        // the fused producer stores. Only do so when the producer's exact
+        // access set covers every rewritten consumer load. Otherwise, retain
+        // the legal fusion while keeping accesses on the original memref.
+        unsigned privateMemRefLoopDepth =
+            getNestingDepth(dstAffineForOp) + bestDstLoopDepth;
+        Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
+        DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        DenseMap<Value, Operation *> privateMemRefToFusedProducerStore;
+        if (!privateMemRefCandidates.empty()) {
           dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
-            Value storeMemRef = storeOp.getMemRef();
-            if (privateMemrefs.count(storeMemRef) > 0)
-              privateMemRefToStores[storeMemRef].push_back(storeOp);
+            Value memref = storeOp.getMemRef();
+            if (!privateMemRefCandidates.contains(memref))
+              return;
+            privateMemRefToStores[memref].push_back(storeOp);
+            if (!preexistingDstStores.contains(storeOp))
+              privateMemRefToFusedProducerStore.try_emplace(memref, storeOp);
           });
+        }
+
+        // Create only private memrefs whose producer covers every rewritten
+        // load, and update the dependence graph with exactly those memrefs.
+        DenseSet<Value> privateMemrefs;
+        SmallVector<Value, 4> newPrivateMemrefs;
+        for (auto &[oldMemRef, stores] : privateMemRefToStores) {
+          Value newMemRef = createPrivateMemRef(
+              dstAffineForOp, stores,
+              privateMemRefToFusedProducerStore.lookup(oldMemRef),
+              privateMemRefLoopDepth, fastMemorySpace, sliceInsertionBlock,
+              localBufSizeThreshold);
+          if (!newMemRef) {
+            LDBG() << "Skipping private memref with unsupported or uncovered "
+                      "consumer accesses: "
+                   << oldMemRef;
+            continue;
+          }
+          privateMemrefs.insert(oldMemRef);
+          newPrivateMemrefs.push_back(newMemRef);
+        }
+
+        // Update edges between 'srcNode' and 'dstNode'.
+        mdg->updateEdges(srcId, dstId, privateMemrefs, removeSrcNode);
 
-          // Replace original memrefs with private memrefs. Note that all the
-          // loads and stores on these memrefs will be replaced with a new
-          // loads and stores. Any reference to the original ones becomes
-          // invalid after this point.
-          for (auto &memrefToStoresPair : privateMemRefToStores) {
-            ArrayRef<Operation *> storesForMemref = memrefToStoresPair.second;
-            Value newMemRef = createPrivateMemRef(
-                dstAffineForOp, storesForMemref, bestDstLoopDepth,
-                fastMemorySpace, sliceInsertionBlock, localBufSizeThreshold);
-            if (!newMemRef)
-              continue;
+        if (!newPrivateMemrefs.empty()) {
+          for (Value newMemRef : newPrivateMemrefs) {
             // Create new node in dependence graph for 'newMemRef' alloc op.
             unsigned newMemRefNodeId = mdg->addNode(newMemRef.getDefiningOp());
             // Add edge from 'newMemRef' node to dstNode.
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..c216fc14b9590 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -884,3 +884,129 @@ func.func @high_trip_count(%arg0: memref<1024x4096xf32>, %arg1: memref<8192x4096
   }
   return %alloc : memref<1024x8192xf32>
 }
+
+// -----
+
+// Regression test for https://github.com/llvm/llvm-project/issues/212039.
+// Do not privatize a producer buffer when its exact, strided write set does
+// not cover the consumer read set. Fusion itself remains legal.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_holes(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_strided_holes(
+    %in: memref<32xf64>, %comm: memref<32xf64>,
+    %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<32xf64>
+  }
+  return
+}
+
+// -----
+
+// Preserve privatization when the strided producer covers every read.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_coverage(
+// PRODUCER-CONSUMER-MAXIMAL: memref.alloc() : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL-NOT: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}}[0] : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}}[0] : memref<1xf64>
+func.func @private_buffer_strided_coverage(
+    %in: memref<16xf64>, %out: memref<16xf64>) {
+  %comm = memref.alloc() : memref<32xf64>
+  affine.for %i = 0 to 16 {
+    %a = affine.load %in[%i] : memref<16xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[2 * %j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<16xf64>
+  }
+  return
+}
+
+// -----
+
+// Although the second read's union across all outer iterations equals the
+// producer write union, it is not covered in the same private-buffer instance.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_outer_iv_mismatch(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for
+// PRODUCER-CONSUMER-MAXIMAL:     memref.alloc() : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:     affine.for %[[J:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL-NEXT: affine.for %[[I:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL:       affine.store {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_outer_iv_mismatch(
+    %in: memref<2x4x16xf64>, %out: memref<2x4x16xf64>) {
+  affine.for %instance = 0 to 2 {
+    affine.for %batch = 0 to 4 {
+      %comm = memref.alloc() : memref<64xf64>
+      affine.for %i = 0 to 16 {
+        %a = affine.load %in[%instance, %batch, %i] : memref<2x4x16xf64>
+        affine.store %a, %comm[16 * %batch + %i] : memref<64xf64>
+      }
+      affine.for %j = 0 to 16 {
+        %a = affine.load %comm[16 * %batch + %j] : memref<64xf64>
+        %b = affine.load %comm[-16 * %batch + %j + 48] : memref<64xf64>
+        %sum = arith.addf %a, %b : f64
+        affine.store %sum, %out[%instance, %batch, %j]
+            : memref<2x4x16xf64>
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// Access relations conservatively approximate a non-unit affine.for with a
+// dynamic lower bound. Do not use that approximation to prove coverage.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_dynamic_lb_stride(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_dynamic_lb_stride(
+    %lb: index, %in: memref<4xf64>, %comm: memref<4x?xf64>,
+    %out: memref<4xf64>) {
+  affine.for %batch = 0 to 4 {
+    affine.for %k = %lb to affine_map<(d0) -> (d0 + 8)>(%lb) step 2 {
+      %a = affine.load %in[%batch] : memref<4xf64>
+      affine.store %a, %comm[%batch, %k] : memref<4x?xf64>
+    }
+  }
+  affine.for %batch = 0 to 4 {
+    %a = affine.load %comm[%batch, %lb] : memref<4x?xf64>
+    %b = affine.load %comm[%batch, %lb + 1] : memref<4x?xf64>
+    %c = affine.load %comm[%batch, %lb + 2] : memref<4x?xf64>
+    %d = affine.load %comm[%batch, %lb + 3] : memref<4x?xf64>
+    %ab = arith.addf %a, %b : f64
+    %cd = arith.addf %c, %d : f64
+    %sum = arith.addf %ab, %cd : f64
+    affine.store %sum, %out[%batch] : memref<4xf64>
+  }
+  return
+}
diff --git a/mlir/test/mlir-runner/affine-loop-fusion.mlir b/mlir/test/mlir-runner/affine-loop-fusion.mlir
new file mode 100644
index 0000000000000..ecf1858d5c456
--- /dev/null
+++ b/mlir/test/mlir-runner/affine-loop-fusion.mlir
@@ -0,0 +1,69 @@
+// RUN: mlir-opt %s \
+// RUN:   -pass-pipeline='builtin.module(func.func(affine-loop-fusion{mode=producer maximal}),func.func(lower-affine),generate-runtime-verification,convert-scf-to-cf,convert-to-llvm)' | \
+// RUN: mlir-runner -e main -entry-point-result=i64 \
+// RUN:   %if target={{s390x-.*}} %{ -argext-abi-check=false %} | FileCheck %s
+// XFAIL: system-aix
+
+// CHECK: {{^0$}}
+
+// The producer overwrites the even elements; the odd elements keep their
+// initial values. Runtime verification makes an undersized private buffer fail
+// deterministically instead of relying on undefined out-of-bounds behavior.
+memref.global "private" constant @expected : memref<16xf64> =
+    dense<[1000.0, 101.0, 1001.0, 103.0, 1002.0, 105.0, 1003.0, 107.0,
+           1004.0, 109.0, 1005.0, 111.0, 1006.0, 113.0, 1007.0, 115.0]>
+
+func.func @kernel(%in: memref<32xf64>, %comm: memref<32xf64>,
+                  %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<32xf64>
+  }
+  return
+}
+
+func.func @main() -> i64 {
+  %in = memref.alloc() : memref<32xf64>
+  %comm = memref.alloc() : memref<32xf64>
+  %out = memref.alloc() : memref<32xf64>
+  %expected = memref.get_global @expected : memref<16xf64>
+  %c0 = arith.constant 0 : i64
+  %c1 = arith.constant 1 : i64
+  %c100 = arith.constant 100.0 : f64
+  %c1000 = arith.constant 1000.0 : f64
+  %c0Index = arith.constant 0 : index
+  %c1Index = arith.constant 1 : index
+  %c16 = arith.constant 16 : index
+  %c32 = arith.constant 32 : index
+
+  scf.for %i = %c0Index to %c32 step %c1Index {
+    %i64 = arith.index_cast %i : index to i64
+    %f64 = arith.sitofp %i64 : i64 to f64
+    %inValue = arith.addf %c1000, %f64 : f64
+    %commValue = arith.addf %c100, %f64 : f64
+    memref.store %inValue, %in[%i] : memref<32xf64>
+    memref.store %commValue, %comm[%i] : memref<32xf64>
+  }
+
+  call @kernel(%in, %comm, %out)
+      : (memref<32xf64>, memref<32xf64>, memref<32xf64>) -> ()
+
+  %errors = scf.for %j = %c0Index to %c16 step %c1Index
+      iter_args(%errorsIn = %c0) -> (i64) {
+    %actual = memref.load %out[%j] : memref<32xf64>
+    %expectedValue = memref.load %expected[%j] : memref<16xf64>
+    %equal = arith.cmpf oeq, %actual, %expectedValue : f64
+    %error = arith.select %equal, %c0, %c1 : i64
+    %errorsOut = arith.addi %errorsIn, %error : i64
+    scf.yield %errorsOut : i64
+  }
+
+  memref.dealloc %in : memref<32xf64>
+  memref.dealloc %comm : memref<32xf64>
+  memref.dealloc %out : memref<32xf64>
+  return %errors : i64
+}


From llvmlistbot at llvm.org  Sat Jul 25 18:32:23 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 18:32:23 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Check access coverage before
 fusion privatization (PR #212080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212080@github.com>
Message-ID: <6a6563a7.9edb5c9d.21e140.e9e8@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @guoriyue :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/212080

From llvmlistbot at llvm.org  Sat Jul 25 18:33:02 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 18:33:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Check access coverage before
 fusion privatization (PR #212080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212080@github.com>
Message-ID: <6a6563ce.d2cb4adc.23815e.592b@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Mingfei Guo (guoriyue)

<details>
<summary>Changes</summary>

This fixes an affine loop fusion bug where privatization could create a buffer that is too small for consumer accesses.

The pass now checks that producer writes cover all consumer reads. If not, fusion continues but unsafe privatization is skipped.

---

Patch is 21.58 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/212080.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp (+161-53) 
- (modified) mlir/test/Dialect/Affine/loop-fusion-4.mlir (+126) 
- (added) mlir/test/mlir-runner/affine-loop-fusion.mlir (+69) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..d84fac437faca 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -12,6 +12,7 @@
 
 #include "mlir/Dialect/Affine/Transforms/Passes.h"
 
+#include "mlir/Dialect/Affine/Analysis/AffineAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/AffineStructures.h"
 #include "mlir/Dialect/Affine/Analysis/LoopAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/Utils.h"
@@ -272,6 +273,77 @@ getDominanceFilterForPrivateMemRefRepl(Block *sliceInsertionBlock,
   return firstAncestor;
 }
 
+static std::optional<presburger::IntegerRelation>
+getAccessRelationForPrivateBuffer(Operation *op, unsigned loopDepth) {
+  if (!isa<AffineLoadOp, AffineStoreOp>(op))
+    return std::nullopt;
+
+  // Reject domains that getAccessRelation cannot represent exactly.
+  Operation *ancestor = op->getParentOp();
+  for (; ancestor && !ancestor->hasTrait<OpTrait::AffineScope>();
+       ancestor = ancestor->getParentOp()) {
+    if (auto forOp = dyn_cast<AffineForOp>(ancestor)) {
+      if (forOp.getStepAsInt() != 1 && !forOp.hasConstantLowerBound())
+        return std::nullopt;
+    } else if (auto parallelOp = dyn_cast<AffineParallelOp>(ancestor)) {
+      if (!llvm::all_of(parallelOp.getSteps(),
+                        [](int64_t step) { return step == 1; }))
+        return std::nullopt;
+    } else {
+      return std::nullopt;
+    }
+  }
+  if (!ancestor)
+    return std::nullopt;
+
+  presburger::IntegerRelation relation(
+      presburger::PresburgerSpace::getRelationSpace());
+  if (failed(MemRefAccess(op).getAccessRelation(relation)))
+    return std::nullopt;
+
+  if (getNestingDepth(op) != relation.getNumDomainVars() ||
+      loopDepth > relation.getNumDomainVars())
+    return std::nullopt;
+
+  relation.convertToLocal(presburger::VarKind::Domain, loopDepth,
+                          relation.getNumDomainVars());
+  return relation;
+}
+
+/// Returns true if every load is covered by `store` in each private-buffer
+/// instance.
+static bool areLoadsCoveredByStore(Operation *store,
+                                   ArrayRef<Operation *> loads,
+                                   unsigned loopDepth) {
+  assert(store && "expected producer store");
+  assert(!loads.empty() && "expected consumer load");
+
+  auto write = getAccessRelationForPrivateBuffer(store, loopDepth);
+  if (!write)
+    return false;
+  for (Operation *load : loads) {
+    auto read = getAccessRelationForPrivateBuffer(load, loopDepth);
+    if (!read ||
+        !write->getSpace().isAligned(read->getSpace(),
+                                     presburger::VarKind::Domain) ||
+        write->getNumRangeVars() != read->getNumRangeVars())
+      return false;
+
+    write->mergeAndAlignSymbols(*read);
+    if (!read->isSubsetOf(*write))
+      return false;
+  }
+  return true;
+}
+
+static bool haveSameAccessFunction(ArrayRef<Operation *> stores) {
+  assert(!stores.empty() && "expected producer store");
+  MemRefAccess first(cast<AffineWriteOpInterface>(stores.front()));
+  return llvm::all_of(llvm::drop_begin(stores), [&](Operation *store) {
+    return first == MemRefAccess(cast<AffineWriteOpInterface>(store));
+  });
+}
+
 /// Returns the amount of additional (redundant) computation that will be done
 /// as a fraction of the total computation if `srcForOp` is fused into
 /// `dstForOp` at depth `depth`. The method returns the compute cost of the
@@ -327,41 +399,60 @@ static std::optional<double> getAdditionalComputeFraction(
 // Creates and returns a private (single-user) memref for fused loop rooted at
 // 'forOp', with (potentially reduced) memref size based on the memref region
 // written to by `storeOps` at depth 'dstLoopDepth'. 'sliceInsertionBlock'
-// specifies the block in which the slice was/will be inserted. The method
-// expects that all stores ops to the memref have the same access function.
-// Returns nullptr if the creation failed.
+// specifies the block in which the slice was/will be inserted. Returns nullptr
+// unless the fused producer covers every consumer access that would be
+// redirected to the private memref.
 static Value createPrivateMemRef(AffineForOp forOp,
                                  ArrayRef<Operation *> storeOps,
+                                 Operation *fusedProducerStore,
                                  unsigned dstLoopDepth,
                                  std::optional<unsigned> fastMemorySpace,
                                  Block *sliceInsertionBlock,
                                  uint64_t localBufSizeThreshold) {
   assert(!storeOps.empty() && "no source stores supplied");
 
-  // Check if all stores have the same access function; we only support this
-  // case.
-  // TODO: Use union of memref write regions to compute private memref footprint
-  // for store ops with different access functions.
-  if (storeOps.size() > 1 &&
-      !std::equal(std::next(storeOps.begin()), storeOps.end(), storeOps.begin(),
-                  [](Operation *a, Operation *b) {
-                    MemRefAccess aM(cast<AffineWriteOpInterface>(a));
-                    MemRefAccess bM(cast<AffineWriteOpInterface>(b));
-                    return aM == bM;
-                  })) {
+  // TODO: Use a union of write regions to support stores with different access
+  // functions.
+  if (!fusedProducerStore)
+    return nullptr;
+  if (!haveSameAccessFunction(storeOps)) {
     LDBG() << "Private memref creation unsupported for multiple producer "
-           << "stores with different access functions.";
+              "stores with different access functions.";
     return nullptr;
   }
 
-  Operation *srcStoreOp = storeOps[0];
+  Operation *srcStoreOp = storeOps.front();
+  Value oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
+  if (cast<AffineWriteOpInterface>(fusedProducerStore).getMemRef() != oldMemRef)
+    return nullptr;
+
+  // Match the exact set of users that replaceAllMemRefUsesWith will rewrite.
+  // Privatization is only legal when the fused producer writes every element
+  // read from each private-buffer instance.
+  Operation *domFilter =
+      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
+  DominanceInfo domInfo(domFilter->getParentOfType<FunctionOpInterface>());
+  SmallVector<Operation *, 4> loads;
+  for (Operation *user : oldMemRef.getUsers()) {
+    if (!domInfo.dominates(domFilter, user) ||
+        hasSingleEffect<MemoryEffects::Free>(user, oldMemRef))
+      continue;
+    if (isa<AffineLoadOp>(user)) {
+      loads.push_back(user);
+      continue;
+    }
+    if (!isa<AffineStoreOp>(user) || !llvm::is_contained(storeOps, user))
+      return nullptr;
+  }
+  if (loads.empty() ||
+      !areLoadsCoveredByStore(fusedProducerStore, loads, dstLoopDepth))
+    return nullptr;
 
   // Create builder to insert alloc op just before 'forOp'.
   OpBuilder b(forOp);
   // Builder to create constants at the top level.
   OpBuilder top(forOp->getParentRegion());
   // Create new memref type based on slice bounds.
-  auto oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
   auto oldMemRefType = cast<MemRefType>(oldMemRef.getType());
   unsigned rank = oldMemRefType.getRank();
 
@@ -441,12 +532,8 @@ static Value createPrivateMemRef(AffineForOp forOp,
       AffineMap::get(outerIVs.size() + rank, 0, remapExprs, forOp.getContext());
 
   // Replace all users of 'oldMemRef' with 'newMemRef'.
-  Operation *domFilter =
-      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
   auto userFilterFn = [&](Operation *user) {
-    auto domInfo = std::make_unique<DominanceInfo>(
-        domFilter->getParentOfType<FunctionOpInterface>());
-    return domInfo->dominates(domFilter, user);
+    return domInfo.dominates(domFilter, user);
   };
   LogicalResult res = replaceAllMemRefUsesWith(
       oldMemRef, newMemRef, /*extraIndices=*/{}, indexRemap,
@@ -1075,17 +1162,22 @@ struct GreedyFusion {
             srcId, dstId, bestSlice, fusedLoopInsPoint, srcEscapingMemRefs,
             *mdg);
 
-        DenseSet<Value> privateMemrefs;
+        DenseSet<Value> privateMemRefCandidates;
         for (Value memref : producerConsumerMemrefs) {
           if (canCreatePrivateMemRef(memref, srcEscapingMemRefs, srcId, dstId,
                                      removeSrcNode)) {
-            // Create a private version of this memref.
-            LDBG() << "Creating private memref for " << memref;
-            // Create a private version of this memref.
-            privateMemrefs.insert(memref);
+            LDBG() << "Considering private memref for " << memref;
+            privateMemRefCandidates.insert(memref);
           }
         }
 
+        DenseSet<Operation *> preexistingDstStores;
+        if (!privateMemRefCandidates.empty()) {
+          dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
+            preexistingDstStores.insert(storeOp.getOperation());
+          });
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1098,35 +1190,51 @@ struct GreedyFusion {
         if (fusedLoopInsPoint != dstAffineForOp)
           dstAffineForOp->moveBefore(fusedLoopInsPoint);
 
-        // Update edges between 'srcNode' and 'dstNode'.
-        mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
-                         removeSrcNode);
-
-        // Create private memrefs.
-        if (!privateMemrefs.empty()) {
-          // Note the block into which fusion was performed. This can be used to
-          // place `alloc`s that create private memrefs.
-          Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
-
-          // Gather stores for all the private-to-be memrefs.
-          DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        // Privatization rewrites the consumer accesses to a buffer sized from
+        // the fused producer stores. Only do so when the producer's exact
+        // access set covers every rewritten consumer load. Otherwise, retain
+        // the legal fusion while keeping accesses on the original memref.
+        unsigned privateMemRefLoopDepth =
+            getNestingDepth(dstAffineForOp) + bestDstLoopDepth;
+        Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
+        DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        DenseMap<Value, Operation *> privateMemRefToFusedProducerStore;
+        if (!privateMemRefCandidates.empty()) {
           dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
-            Value storeMemRef = storeOp.getMemRef();
-            if (privateMemrefs.count(storeMemRef) > 0)
-              privateMemRefToStores[storeMemRef].push_back(storeOp);
+            Value memref = storeOp.getMemRef();
+            if (!privateMemRefCandidates.contains(memref))
+              return;
+            privateMemRefToStores[memref].push_back(storeOp);
+            if (!preexistingDstStores.contains(storeOp))
+              privateMemRefToFusedProducerStore.try_emplace(memref, storeOp);
           });
+        }
+
+        // Create only private memrefs whose producer covers every rewritten
+        // load, and update the dependence graph with exactly those memrefs.
+        DenseSet<Value> privateMemrefs;
+        SmallVector<Value, 4> newPrivateMemrefs;
+        for (auto &[oldMemRef, stores] : privateMemRefToStores) {
+          Value newMemRef = createPrivateMemRef(
+              dstAffineForOp, stores,
+              privateMemRefToFusedProducerStore.lookup(oldMemRef),
+              privateMemRefLoopDepth, fastMemorySpace, sliceInsertionBlock,
+              localBufSizeThreshold);
+          if (!newMemRef) {
+            LDBG() << "Skipping private memref with unsupported or uncovered "
+                      "consumer accesses: "
+                   << oldMemRef;
+            continue;
+          }
+          privateMemrefs.insert(oldMemRef);
+          newPrivateMemrefs.push_back(newMemRef);
+        }
+
+        // Update edges between 'srcNode' and 'dstNode'.
+        mdg->updateEdges(srcId, dstId, privateMemrefs, removeSrcNode);
 
-          // Replace original memrefs with private memrefs. Note that all the
-          // loads and stores on these memrefs will be replaced with a new
-          // loads and stores. Any reference to the original ones becomes
-          // invalid after this point.
-          for (auto &memrefToStoresPair : privateMemRefToStores) {
-            ArrayRef<Operation *> storesForMemref = memrefToStoresPair.second;
-            Value newMemRef = createPrivateMemRef(
-                dstAffineForOp, storesForMemref, bestDstLoopDepth,
-                fastMemorySpace, sliceInsertionBlock, localBufSizeThreshold);
-            if (!newMemRef)
-              continue;
+        if (!newPrivateMemrefs.empty()) {
+          for (Value newMemRef : newPrivateMemrefs) {
             // Create new node in dependence graph for 'newMemRef' alloc op.
             unsigned newMemRefNodeId = mdg->addNode(newMemRef.getDefiningOp());
             // Add edge from 'newMemRef' node to dstNode.
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..c216fc14b9590 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -884,3 +884,129 @@ func.func @high_trip_count(%arg0: memref<1024x4096xf32>, %arg1: memref<8192x4096
   }
   return %alloc : memref<1024x8192xf32>
 }
+
+// -----
+
+// Regression test for https://github.com/llvm/llvm-project/issues/212039.
+// Do not privatize a producer buffer when its exact, strided write set does
+// not cover the consumer read set. Fusion itself remains legal.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_holes(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_strided_holes(
+    %in: memref<32xf64>, %comm: memref<32xf64>,
+    %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<32xf64>
+  }
+  return
+}
+
+// -----
+
+// Preserve privatization when the strided producer covers every read.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_coverage(
+// PRODUCER-CONSUMER-MAXIMAL: memref.alloc() : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL-NOT: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}}[0] : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}}[0] : memref<1xf64>
+func.func @private_buffer_strided_coverage(
+    %in: memref<16xf64>, %out: memref<16xf64>) {
+  %comm = memref.alloc() : memref<32xf64>
+  affine.for %i = 0 to 16 {
+    %a = affine.load %in[%i] : memref<16xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[2 * %j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<16xf64>
+  }
+  return
+}
+
+// -----
+
+// Although the second read's union across all outer iterations equals the
+// producer write union, it is not covered in the same private-buffer instance.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_outer_iv_mismatch(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for
+// PRODUCER-CONSUMER-MAXIMAL:     memref.alloc() : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:     affine.for %[[J:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL-NEXT: affine.for %[[I:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL:       affine.store {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_outer_iv_mismatch(
+    %in: memref<2x4x16xf64>, %out: memref<2x4x16xf64>) {
+  affine.for %instance = 0 to 2 {
+    affine.for %batch = 0 to 4 {
+      %comm = memref.alloc() : memref<64xf64>
+      affine.for %i = 0 to 16 {
+        %a = affine.load %in[%instance, %batch, %i] : memref<2x4x16xf64>
+        affine.store %a, %comm[16 * %batch + %i] : memref<64xf64>
+      }
+      affine.for %j = 0 to 16 {
+        %a = affine.load %comm[16 * %batch + %j] : memref<64xf64>
+        %b = affine.load %comm[-16 * %batch + %j + 48] : memref<64xf64>
+        %sum = arith.addf %a, %b : f64
+        affine.store %sum, %out[%instance, %batch, %j]
+            : memref<2x4x16xf64>
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// Access relations conservatively approximate a non-unit affine.for with a
+// dynamic lower bound. Do not use that approximation to prove coverage.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_dynamic_lb_stride(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_dynamic_lb_stride(
+    %lb: index, %in: memref<4xf64>, %comm: memref<4x?xf64>,
+    %out: memref<4xf64>) {
+  affine.for %batch = 0 to 4 {
+    affine.for %k = %lb to affine_map<(d0) -> (d0 + 8)>(%lb) step 2 {
+      %a = affine.load %in[%batch] : memref<4xf64>
+      affine.store %a, %comm[%batch, %k] : memref<4x?xf64>
+    }
+  }
+  affine.for %batch = 0 to 4 {
+    %a = affine.load %comm[%batch, %lb] : memref<4x?xf64>
+    %b = affine.load %comm[%batch, %lb + 1] : memref<4x?xf64>
+    %c = affine.load %comm[%batch, %lb + 2] : memref<4x?xf64>
+    %d = affine.load %comm[%batch, %lb + 3] : memref<4x?xf64>
+    %ab = arith.addf %a, %b : f64
+    %cd = arith.addf %c, %d : f64
+    %sum = arith.addf %ab, %cd : f64
+    affine.store %sum, %out[%batch] : memref<4xf64>
+  }
+  return
+}
diff --git a/mlir/test/mlir-runner/affine-loop-fusion.mlir b/mlir/test/mlir-runner/affine-loop-fusion.mlir
new file mode 100644
index 0000000000000..ecf1858d5c456
--- /dev/null
+++ b/mlir/test/mlir-runner/affine-loop-fusion.mlir
@@ -0,0 +1,69 @@
+// RUN: mlir-opt %s \
+// RUN:   -pass-pipeline='builtin.module(func.func(affine-loop-fusion{mode=producer maximal}),func.func(lower-affine),generate-runtime-verification,convert-scf-to-cf,convert-to-llvm)' | \
+// RUN: mlir-runner -e main -entry-point-result=i64 \
+// RUN:   %if target={{s390x-.*}} %{ -argext-abi-check=false %} | FileCheck %s
+// XFAIL: system-aix
+
+// CHECK: {{^0$}}
+
+// The producer overwrites the even elements; the odd elements keep their
+// initial values. Runtime verification makes an undersized private buffer fail
+// deterministically instead of relying on undefined out-of-bounds behavior.
+memref.global "private" constant @expected : memref<16xf64> =
+    dense<[1000.0, 101.0, 1001.0, 103.0, 1002.0, 105.0, 1003.0, 107.0,
+           1004.0, 109.0, 1005.0, 111.0, 1006.0, 113.0, 1007.0, 115.0]>
+
+func.func @kernel(%in: memref<32xf64>, %comm: memref<32xf64>,
+                  %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/212080

From llvmlistbot at llvm.org  Sat Jul 25 18:33:04 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 18:33:04 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Check access coverage before
 fusion privatization (PR #212080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212080@github.com>
Message-ID: <6a6563d0.6c5720b3.2e9b28.51a1@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-affine

Author: Mingfei Guo (guoriyue)

<details>
<summary>Changes</summary>

This fixes an affine loop fusion bug where privatization could create a buffer that is too small for consumer accesses.

The pass now checks that producer writes cover all consumer reads. If not, fusion continues but unsafe privatization is skipped.

---

Patch is 21.58 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/212080.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp (+161-53) 
- (modified) mlir/test/Dialect/Affine/loop-fusion-4.mlir (+126) 
- (added) mlir/test/mlir-runner/affine-loop-fusion.mlir (+69) 


``````````diff
diff --git a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
index 1ec5fbfef50c3..d84fac437faca 100644
--- a/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
+++ b/mlir/lib/Dialect/Affine/Transforms/LoopFusion.cpp
@@ -12,6 +12,7 @@
 
 #include "mlir/Dialect/Affine/Transforms/Passes.h"
 
+#include "mlir/Dialect/Affine/Analysis/AffineAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/AffineStructures.h"
 #include "mlir/Dialect/Affine/Analysis/LoopAnalysis.h"
 #include "mlir/Dialect/Affine/Analysis/Utils.h"
@@ -272,6 +273,77 @@ getDominanceFilterForPrivateMemRefRepl(Block *sliceInsertionBlock,
   return firstAncestor;
 }
 
+static std::optional<presburger::IntegerRelation>
+getAccessRelationForPrivateBuffer(Operation *op, unsigned loopDepth) {
+  if (!isa<AffineLoadOp, AffineStoreOp>(op))
+    return std::nullopt;
+
+  // Reject domains that getAccessRelation cannot represent exactly.
+  Operation *ancestor = op->getParentOp();
+  for (; ancestor && !ancestor->hasTrait<OpTrait::AffineScope>();
+       ancestor = ancestor->getParentOp()) {
+    if (auto forOp = dyn_cast<AffineForOp>(ancestor)) {
+      if (forOp.getStepAsInt() != 1 && !forOp.hasConstantLowerBound())
+        return std::nullopt;
+    } else if (auto parallelOp = dyn_cast<AffineParallelOp>(ancestor)) {
+      if (!llvm::all_of(parallelOp.getSteps(),
+                        [](int64_t step) { return step == 1; }))
+        return std::nullopt;
+    } else {
+      return std::nullopt;
+    }
+  }
+  if (!ancestor)
+    return std::nullopt;
+
+  presburger::IntegerRelation relation(
+      presburger::PresburgerSpace::getRelationSpace());
+  if (failed(MemRefAccess(op).getAccessRelation(relation)))
+    return std::nullopt;
+
+  if (getNestingDepth(op) != relation.getNumDomainVars() ||
+      loopDepth > relation.getNumDomainVars())
+    return std::nullopt;
+
+  relation.convertToLocal(presburger::VarKind::Domain, loopDepth,
+                          relation.getNumDomainVars());
+  return relation;
+}
+
+/// Returns true if every load is covered by `store` in each private-buffer
+/// instance.
+static bool areLoadsCoveredByStore(Operation *store,
+                                   ArrayRef<Operation *> loads,
+                                   unsigned loopDepth) {
+  assert(store && "expected producer store");
+  assert(!loads.empty() && "expected consumer load");
+
+  auto write = getAccessRelationForPrivateBuffer(store, loopDepth);
+  if (!write)
+    return false;
+  for (Operation *load : loads) {
+    auto read = getAccessRelationForPrivateBuffer(load, loopDepth);
+    if (!read ||
+        !write->getSpace().isAligned(read->getSpace(),
+                                     presburger::VarKind::Domain) ||
+        write->getNumRangeVars() != read->getNumRangeVars())
+      return false;
+
+    write->mergeAndAlignSymbols(*read);
+    if (!read->isSubsetOf(*write))
+      return false;
+  }
+  return true;
+}
+
+static bool haveSameAccessFunction(ArrayRef<Operation *> stores) {
+  assert(!stores.empty() && "expected producer store");
+  MemRefAccess first(cast<AffineWriteOpInterface>(stores.front()));
+  return llvm::all_of(llvm::drop_begin(stores), [&](Operation *store) {
+    return first == MemRefAccess(cast<AffineWriteOpInterface>(store));
+  });
+}
+
 /// Returns the amount of additional (redundant) computation that will be done
 /// as a fraction of the total computation if `srcForOp` is fused into
 /// `dstForOp` at depth `depth`. The method returns the compute cost of the
@@ -327,41 +399,60 @@ static std::optional<double> getAdditionalComputeFraction(
 // Creates and returns a private (single-user) memref for fused loop rooted at
 // 'forOp', with (potentially reduced) memref size based on the memref region
 // written to by `storeOps` at depth 'dstLoopDepth'. 'sliceInsertionBlock'
-// specifies the block in which the slice was/will be inserted. The method
-// expects that all stores ops to the memref have the same access function.
-// Returns nullptr if the creation failed.
+// specifies the block in which the slice was/will be inserted. Returns nullptr
+// unless the fused producer covers every consumer access that would be
+// redirected to the private memref.
 static Value createPrivateMemRef(AffineForOp forOp,
                                  ArrayRef<Operation *> storeOps,
+                                 Operation *fusedProducerStore,
                                  unsigned dstLoopDepth,
                                  std::optional<unsigned> fastMemorySpace,
                                  Block *sliceInsertionBlock,
                                  uint64_t localBufSizeThreshold) {
   assert(!storeOps.empty() && "no source stores supplied");
 
-  // Check if all stores have the same access function; we only support this
-  // case.
-  // TODO: Use union of memref write regions to compute private memref footprint
-  // for store ops with different access functions.
-  if (storeOps.size() > 1 &&
-      !std::equal(std::next(storeOps.begin()), storeOps.end(), storeOps.begin(),
-                  [](Operation *a, Operation *b) {
-                    MemRefAccess aM(cast<AffineWriteOpInterface>(a));
-                    MemRefAccess bM(cast<AffineWriteOpInterface>(b));
-                    return aM == bM;
-                  })) {
+  // TODO: Use a union of write regions to support stores with different access
+  // functions.
+  if (!fusedProducerStore)
+    return nullptr;
+  if (!haveSameAccessFunction(storeOps)) {
     LDBG() << "Private memref creation unsupported for multiple producer "
-           << "stores with different access functions.";
+              "stores with different access functions.";
     return nullptr;
   }
 
-  Operation *srcStoreOp = storeOps[0];
+  Operation *srcStoreOp = storeOps.front();
+  Value oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
+  if (cast<AffineWriteOpInterface>(fusedProducerStore).getMemRef() != oldMemRef)
+    return nullptr;
+
+  // Match the exact set of users that replaceAllMemRefUsesWith will rewrite.
+  // Privatization is only legal when the fused producer writes every element
+  // read from each private-buffer instance.
+  Operation *domFilter =
+      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
+  DominanceInfo domInfo(domFilter->getParentOfType<FunctionOpInterface>());
+  SmallVector<Operation *, 4> loads;
+  for (Operation *user : oldMemRef.getUsers()) {
+    if (!domInfo.dominates(domFilter, user) ||
+        hasSingleEffect<MemoryEffects::Free>(user, oldMemRef))
+      continue;
+    if (isa<AffineLoadOp>(user)) {
+      loads.push_back(user);
+      continue;
+    }
+    if (!isa<AffineStoreOp>(user) || !llvm::is_contained(storeOps, user))
+      return nullptr;
+  }
+  if (loads.empty() ||
+      !areLoadsCoveredByStore(fusedProducerStore, loads, dstLoopDepth))
+    return nullptr;
 
   // Create builder to insert alloc op just before 'forOp'.
   OpBuilder b(forOp);
   // Builder to create constants at the top level.
   OpBuilder top(forOp->getParentRegion());
   // Create new memref type based on slice bounds.
-  auto oldMemRef = cast<AffineWriteOpInterface>(srcStoreOp).getMemRef();
   auto oldMemRefType = cast<MemRefType>(oldMemRef.getType());
   unsigned rank = oldMemRefType.getRank();
 
@@ -441,12 +532,8 @@ static Value createPrivateMemRef(AffineForOp forOp,
       AffineMap::get(outerIVs.size() + rank, 0, remapExprs, forOp.getContext());
 
   // Replace all users of 'oldMemRef' with 'newMemRef'.
-  Operation *domFilter =
-      getDominanceFilterForPrivateMemRefRepl(sliceInsertionBlock, storeOps);
   auto userFilterFn = [&](Operation *user) {
-    auto domInfo = std::make_unique<DominanceInfo>(
-        domFilter->getParentOfType<FunctionOpInterface>());
-    return domInfo->dominates(domFilter, user);
+    return domInfo.dominates(domFilter, user);
   };
   LogicalResult res = replaceAllMemRefUsesWith(
       oldMemRef, newMemRef, /*extraIndices=*/{}, indexRemap,
@@ -1075,17 +1162,22 @@ struct GreedyFusion {
             srcId, dstId, bestSlice, fusedLoopInsPoint, srcEscapingMemRefs,
             *mdg);
 
-        DenseSet<Value> privateMemrefs;
+        DenseSet<Value> privateMemRefCandidates;
         for (Value memref : producerConsumerMemrefs) {
           if (canCreatePrivateMemRef(memref, srcEscapingMemRefs, srcId, dstId,
                                      removeSrcNode)) {
-            // Create a private version of this memref.
-            LDBG() << "Creating private memref for " << memref;
-            // Create a private version of this memref.
-            privateMemrefs.insert(memref);
+            LDBG() << "Considering private memref for " << memref;
+            privateMemRefCandidates.insert(memref);
           }
         }
 
+        DenseSet<Operation *> preexistingDstStores;
+        if (!privateMemRefCandidates.empty()) {
+          dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
+            preexistingDstStores.insert(storeOp.getOperation());
+          });
+        }
+
         // Fuse computation slice of 'srcLoopNest' into 'dstLoopNest'.
         fuseLoops(srcAffineForOp, dstAffineForOp, bestSlice);
         dstNodeChanged = true;
@@ -1098,35 +1190,51 @@ struct GreedyFusion {
         if (fusedLoopInsPoint != dstAffineForOp)
           dstAffineForOp->moveBefore(fusedLoopInsPoint);
 
-        // Update edges between 'srcNode' and 'dstNode'.
-        mdg->updateEdges(srcNode->id, dstNode->id, privateMemrefs,
-                         removeSrcNode);
-
-        // Create private memrefs.
-        if (!privateMemrefs.empty()) {
-          // Note the block into which fusion was performed. This can be used to
-          // place `alloc`s that create private memrefs.
-          Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
-
-          // Gather stores for all the private-to-be memrefs.
-          DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        // Privatization rewrites the consumer accesses to a buffer sized from
+        // the fused producer stores. Only do so when the producer's exact
+        // access set covers every rewritten consumer load. Otherwise, retain
+        // the legal fusion while keeping accesses on the original memref.
+        unsigned privateMemRefLoopDepth =
+            getNestingDepth(dstAffineForOp) + bestDstLoopDepth;
+        Block *sliceInsertionBlock = bestSlice.insertPoint->getBlock();
+        DenseMap<Value, SmallVector<Operation *, 4>> privateMemRefToStores;
+        DenseMap<Value, Operation *> privateMemRefToFusedProducerStore;
+        if (!privateMemRefCandidates.empty()) {
           dstAffineForOp.walk([&](AffineWriteOpInterface storeOp) {
-            Value storeMemRef = storeOp.getMemRef();
-            if (privateMemrefs.count(storeMemRef) > 0)
-              privateMemRefToStores[storeMemRef].push_back(storeOp);
+            Value memref = storeOp.getMemRef();
+            if (!privateMemRefCandidates.contains(memref))
+              return;
+            privateMemRefToStores[memref].push_back(storeOp);
+            if (!preexistingDstStores.contains(storeOp))
+              privateMemRefToFusedProducerStore.try_emplace(memref, storeOp);
           });
+        }
+
+        // Create only private memrefs whose producer covers every rewritten
+        // load, and update the dependence graph with exactly those memrefs.
+        DenseSet<Value> privateMemrefs;
+        SmallVector<Value, 4> newPrivateMemrefs;
+        for (auto &[oldMemRef, stores] : privateMemRefToStores) {
+          Value newMemRef = createPrivateMemRef(
+              dstAffineForOp, stores,
+              privateMemRefToFusedProducerStore.lookup(oldMemRef),
+              privateMemRefLoopDepth, fastMemorySpace, sliceInsertionBlock,
+              localBufSizeThreshold);
+          if (!newMemRef) {
+            LDBG() << "Skipping private memref with unsupported or uncovered "
+                      "consumer accesses: "
+                   << oldMemRef;
+            continue;
+          }
+          privateMemrefs.insert(oldMemRef);
+          newPrivateMemrefs.push_back(newMemRef);
+        }
+
+        // Update edges between 'srcNode' and 'dstNode'.
+        mdg->updateEdges(srcId, dstId, privateMemrefs, removeSrcNode);
 
-          // Replace original memrefs with private memrefs. Note that all the
-          // loads and stores on these memrefs will be replaced with a new
-          // loads and stores. Any reference to the original ones becomes
-          // invalid after this point.
-          for (auto &memrefToStoresPair : privateMemRefToStores) {
-            ArrayRef<Operation *> storesForMemref = memrefToStoresPair.second;
-            Value newMemRef = createPrivateMemRef(
-                dstAffineForOp, storesForMemref, bestDstLoopDepth,
-                fastMemorySpace, sliceInsertionBlock, localBufSizeThreshold);
-            if (!newMemRef)
-              continue;
+        if (!newPrivateMemrefs.empty()) {
+          for (Value newMemRef : newPrivateMemrefs) {
             // Create new node in dependence graph for 'newMemRef' alloc op.
             unsigned newMemRefNodeId = mdg->addNode(newMemRef.getDefiningOp());
             // Add edge from 'newMemRef' node to dstNode.
diff --git a/mlir/test/Dialect/Affine/loop-fusion-4.mlir b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
index cf530016c201a..c216fc14b9590 100644
--- a/mlir/test/Dialect/Affine/loop-fusion-4.mlir
+++ b/mlir/test/Dialect/Affine/loop-fusion-4.mlir
@@ -884,3 +884,129 @@ func.func @high_trip_count(%arg0: memref<1024x4096xf32>, %arg1: memref<8192x4096
   }
   return %alloc : memref<1024x8192xf32>
 }
+
+// -----
+
+// Regression test for https://github.com/llvm/llvm-project/issues/212039.
+// Do not privatize a producer buffer when its exact, strided write set does
+// not cover the consumer read set. Fusion itself remains legal.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_holes(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<32xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_strided_holes(
+    %in: memref<32xf64>, %comm: memref<32xf64>,
+    %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<32xf64>
+  }
+  return
+}
+
+// -----
+
+// Preserve privatization when the strided producer covers every read.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_strided_coverage(
+// PRODUCER-CONSUMER-MAXIMAL: memref.alloc() : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL-NOT: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.store {{.*}}[0] : memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}}[0] : memref<1xf64>
+func.func @private_buffer_strided_coverage(
+    %in: memref<16xf64>, %out: memref<16xf64>) {
+  %comm = memref.alloc() : memref<32xf64>
+  affine.for %i = 0 to 16 {
+    %a = affine.load %in[%i] : memref<16xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[2 * %j] : memref<32xf64>
+    affine.store %b, %out[%j] : memref<16xf64>
+  }
+  return
+}
+
+// -----
+
+// Although the second read's union across all outer iterations equals the
+// producer write union, it is not covered in the same private-buffer instance.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_outer_iv_mismatch(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for
+// PRODUCER-CONSUMER-MAXIMAL:     memref.alloc() : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL:     affine.for %[[J:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL-NEXT: affine.for %[[I:.*]] = 0 to 16 {
+// PRODUCER-CONSUMER-MAXIMAL:       affine.store {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL:       affine.load {{.*}} : memref<64xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref<1xf64>
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_outer_iv_mismatch(
+    %in: memref<2x4x16xf64>, %out: memref<2x4x16xf64>) {
+  affine.for %instance = 0 to 2 {
+    affine.for %batch = 0 to 4 {
+      %comm = memref.alloc() : memref<64xf64>
+      affine.for %i = 0 to 16 {
+        %a = affine.load %in[%instance, %batch, %i] : memref<2x4x16xf64>
+        affine.store %a, %comm[16 * %batch + %i] : memref<64xf64>
+      }
+      affine.for %j = 0 to 16 {
+        %a = affine.load %comm[16 * %batch + %j] : memref<64xf64>
+        %b = affine.load %comm[-16 * %batch + %j + 48] : memref<64xf64>
+        %sum = arith.addf %a, %b : f64
+        affine.store %sum, %out[%instance, %batch, %j]
+            : memref<2x4x16xf64>
+      }
+    }
+  }
+  return
+}
+
+// -----
+
+// Access relations conservatively approximate a non-unit affine.for with a
+// dynamic lower bound. Do not use that approximation to prove coverage.
+// PRODUCER-CONSUMER-MAXIMAL-LABEL: func.func @private_buffer_dynamic_lb_stride(
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL: affine.for
+// PRODUCER-CONSUMER-MAXIMAL:   affine.for {{.*}} step 2
+// PRODUCER-CONSUMER-MAXIMAL:     affine.store {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL:   affine.load {{.*}} : memref<4x?xf64>
+// PRODUCER-CONSUMER-MAXIMAL-NOT: memref.alloc
+// PRODUCER-CONSUMER-MAXIMAL: return
+func.func @private_buffer_dynamic_lb_stride(
+    %lb: index, %in: memref<4xf64>, %comm: memref<4x?xf64>,
+    %out: memref<4xf64>) {
+  affine.for %batch = 0 to 4 {
+    affine.for %k = %lb to affine_map<(d0) -> (d0 + 8)>(%lb) step 2 {
+      %a = affine.load %in[%batch] : memref<4xf64>
+      affine.store %a, %comm[%batch, %k] : memref<4x?xf64>
+    }
+  }
+  affine.for %batch = 0 to 4 {
+    %a = affine.load %comm[%batch, %lb] : memref<4x?xf64>
+    %b = affine.load %comm[%batch, %lb + 1] : memref<4x?xf64>
+    %c = affine.load %comm[%batch, %lb + 2] : memref<4x?xf64>
+    %d = affine.load %comm[%batch, %lb + 3] : memref<4x?xf64>
+    %ab = arith.addf %a, %b : f64
+    %cd = arith.addf %c, %d : f64
+    %sum = arith.addf %ab, %cd : f64
+    affine.store %sum, %out[%batch] : memref<4xf64>
+  }
+  return
+}
diff --git a/mlir/test/mlir-runner/affine-loop-fusion.mlir b/mlir/test/mlir-runner/affine-loop-fusion.mlir
new file mode 100644
index 0000000000000..ecf1858d5c456
--- /dev/null
+++ b/mlir/test/mlir-runner/affine-loop-fusion.mlir
@@ -0,0 +1,69 @@
+// RUN: mlir-opt %s \
+// RUN:   -pass-pipeline='builtin.module(func.func(affine-loop-fusion{mode=producer maximal}),func.func(lower-affine),generate-runtime-verification,convert-scf-to-cf,convert-to-llvm)' | \
+// RUN: mlir-runner -e main -entry-point-result=i64 \
+// RUN:   %if target={{s390x-.*}} %{ -argext-abi-check=false %} | FileCheck %s
+// XFAIL: system-aix
+
+// CHECK: {{^0$}}
+
+// The producer overwrites the even elements; the odd elements keep their
+// initial values. Runtime verification makes an undersized private buffer fail
+// deterministically instead of relying on undefined out-of-bounds behavior.
+memref.global "private" constant @expected : memref<16xf64> =
+    dense<[1000.0, 101.0, 1001.0, 103.0, 1002.0, 105.0, 1003.0, 107.0,
+           1004.0, 109.0, 1005.0, 111.0, 1006.0, 113.0, 1007.0, 115.0]>
+
+func.func @kernel(%in: memref<32xf64>, %comm: memref<32xf64>,
+                  %out: memref<32xf64>) {
+  affine.for %i = 0 to 8 {
+    %a = affine.load %in[%i] : memref<32xf64>
+    affine.store %a, %comm[2 * %i] : memref<32xf64>
+  }
+  affine.for %j = 0 to 16 {
+    %b = affine.load %comm[%...
[truncated]

``````````

</details>


https://github.com/llvm/llvm-project/pull/212080

From llvmlistbot at llvm.org  Sat Jul 25 18:35:58 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 18:35:58 -0700 (PDT)
Subject: [Mlir-commits] [mlir] abd6e74 - [MLIR][WasmSSA] Add global_set and
	select ops (#196613)
Message-ID: <6a65647e.9edb5c9d.21e140.3b7d@mx.google.com>


Author: Byeongjee Kang
Date: 2026-07-26T01:35:53Z
New Revision: abd6e745e22e83af7986cdca4c1ba2d798e00156

URL: https://github.com/llvm/llvm-project/commit/abd6e745e22e83af7986cdca4c1ba2d798e00156
DIFF: https://github.com/llvm/llvm-project/commit/abd6e745e22e83af7986cdca4c1ba2d798e00156.diff

LOG: [MLIR][WasmSSA] Add global_set and select ops (#196613)

Adds two missing ops to the WasmSSA dialect.

- `wasmssa.global_set`
- `wasmssa.select`

---------

Co-authored-by: Ferdinand Lemaire <flscminecraft at gmail.com>

Added: 
    mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
    mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
    mlir/test/Dialect/WasmSSA/custom_parser/select.mlir

Modified: 
    mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
    mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
    mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
    mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
    mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
    mlir/test/Dialect/WasmSSA/global-invalid.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
index 64391d807c633..64d480c065b79 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSA.h
@@ -38,6 +38,7 @@
 #include "mlir/Interfaces/CallInterfaces.h"
 #include "mlir/Interfaces/FunctionInterfaces.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
+#include "mlir/Interfaces/SideEffectInterfaces.h"
 
 //===----------------------------------------------------------------------===//
 // WebAssembly Constraints

diff  --git a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
index bfa0953f50ac8..3b3c3e8825fef 100644
--- a/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
+++ b/mlir/include/mlir/Dialect/WasmSSA/IR/WasmSSAOps.td
@@ -16,6 +16,7 @@ include "mlir/Dialect/WasmSSA/IR/WasmSSAInterfaces.td"
 
 include "mlir/Interfaces/FunctionInterfaces.td"
 include "mlir/Interfaces/InferTypeOpInterface.td"
+include "mlir/Interfaces/SideEffectInterfaces.td"
 include "mlir/IR/BuiltinAttributeInterfaces.td"
 include "mlir/IR/SymbolInterfaces.td"
 
@@ -381,6 +382,25 @@ def WasmSSA_GlobalGetOp : WasmSSA_Op<"global_get", [DeclareOpInterfaceMethods<Sy
   let assemblyFormat = "$global attr-dict `:` type($global_val)";
 }
 
+def WasmSSA_GlobalSetOp : WasmSSA_Op<"global_set", [
+    DeclareOpInterfaceMethods<SymbolUserOpInterface>]> {
+  let summary = "Sets the value of the mutable global passed as argument.";
+  let description = [{
+     Sets the value of the mutable global referenced by the symbol to the given
+     value.
+
+     Example:
+
+     ```mlir
+     // Sets the value of `@global_0` to `%v`.
+     wasmssa.global_set @global_0 to %v : i32
+     ```
+    }];
+  let arguments = (ins FlatSymbolRefAttr: $global,
+                       WasmSSA_ValType: $value);
+  let assemblyFormat = "$global `to` $value `:` type($value) attr-dict";
+}
+
 def WasmSSA_IfOp : WasmSSA_Op<"if", [Terminator,
     DeclareOpInterfaceMethods<LabelLevelOpInterface>]> {
   let summary = "Execute the if region if condition value is non-zero, the else region otherwise.";
@@ -609,6 +629,29 @@ def WasmSSA_ReturnOp : WasmSSA_Op<"return", [Terminator]> {
   ];
 }
 
+def WasmSSA_SelectOp : WasmSSA_Op<"select", [
+    AllTypesMatch<["true_value", "false_value", "result"]>,
+    Pure]> {
+  let summary = "Select between two values based on a condition.";
+  let description = [{
+     Returns `true_value` if `condition` is non-zero, `false_value` otherwise.
+     Both operands and the result must share the same Wasm value type.
+
+     Example:
+
+     ```mlir
+     // Returns %a if %cond is non-zero, otherwise %b.
+     %r = wasmssa.select %cond, %a, %b : i32
+     ```
+    }];
+  let arguments = (ins WasmSSA_ValType: $true_value,
+                       WasmSSA_ValType: $false_value,
+                       I32: $condition);
+  let results = (outs WasmSSA_ValType: $result);
+  let assemblyFormat =
+      "$condition `,` $true_value `,` $false_value attr-dict `:` type($result)";
+}
+
 // ---- Numeric ops
 
 class WasmSSA_BinaryNumericalOp<string mnemonic, string summaryStr, string descStr,

diff  --git a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
index b4b5006352c11..c26a3b038e918 100644
--- a/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
+++ b/mlir/lib/Conversion/RaiseWasm/RaiseWasmMLIR.cpp
@@ -671,6 +671,23 @@ struct WasmGlobalWithGetGlobalInitConversion
   }
 };
 
+struct WasmGlobalSetOpConversion : OpConversionPattern<GlobalSetOp> {
+  using OpConversionPattern::OpConversionPattern;
+  LogicalResult
+  matchAndRewrite(GlobalSetOp globalSetOp, GlobalSetOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = globalSetOp.getLoc();
+    auto globalPtr = memref::GetGlobalOp::create(
+        rewriter, loc, MemRefType::get({1}, adaptor.getValue().getType()),
+        globalSetOp.getGlobal());
+    auto idx = arith::ConstantIndexOp::create(rewriter, loc, 0);
+    rewriter.replaceOpWithNewOp<memref::StoreOp>(
+        globalSetOp, adaptor.getValue(), globalPtr.getResult(),
+        ValueRange{idx.getResult()});
+    return success();
+  }
+};
+
 struct WasmMemoryOpConversion : OpConversionPattern<MemOp> {
   using OpConversionPattern::OpConversionPattern;
 
@@ -796,6 +813,24 @@ struct WasmReturnOpConversion : OpConversionPattern<ReturnOp> {
   }
 };
 
+struct WasmSelectOpConversion : OpConversionPattern<SelectOp> {
+  using OpConversionPattern::OpConversionPattern;
+
+  LogicalResult
+  matchAndRewrite(SelectOp selectOp, SelectOp::Adaptor adaptor,
+                  ConversionPatternRewriter &rewriter) const override {
+    auto loc = selectOp.getLoc();
+    auto zero =
+        arith::ConstantOp::create(rewriter, loc, rewriter.getI32IntegerAttr(0));
+    auto flag = arith::CmpIOp::create(rewriter, loc, arith::CmpIPredicate::ne,
+                                      adaptor.getCondition(), zero.getResult());
+    rewriter.replaceOpWithNewOp<arith::SelectOp>(selectOp, flag.getResult(),
+                                                 adaptor.getTrueValue(),
+                                                 adaptor.getFalseValue());
+    return success();
+  }
+};
+
 struct RaiseWasmMLIRPass : public impl::RaiseWasmMLIRBase<RaiseWasmMLIRPass> {
   void runOnOperation() override {
     ConversionTarget target{getContext()};
@@ -876,6 +911,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmGeSIOpConversion,
            WasmGeUIOpConversion,
            WasmGlobalImportOpConverter,
+           WasmGlobalSetOpConversion,
            WasmGlobalWithConstInitConversion,
            WasmGlobalWithGetGlobalInitConversion,
            WasmGtOpConversion,
@@ -906,6 +942,7 @@ void mlir::populateRaiseWasmMLIRConversionPatterns(
            WasmReturnOpConversion,
            WasmRotlOpConversion,
            WasmRotrOpConversion,
+           WasmSelectOpConversion,
            WasmShLOpConversion,
            WasmShRSOpConversion,
            WasmShRUOpConversion,

diff  --git a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
index 08de887a56a97..eab9c7a9d53a7 100644
--- a/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
+++ b/mlir/lib/Dialect/WasmSSA/IR/WasmSSAOps.cpp
@@ -331,6 +331,44 @@ GlobalGetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// GlobalSetOp
+//===----------------------------------------------------------------------===//
+
+LogicalResult
+GlobalSetOp::verifySymbolUses(SymbolTableCollection &symbolTable) {
+  Operation *symTabOp = SymbolTable::getNearestSymbolTable(*this);
+  StringRef referencedSymbol = getGlobal();
+  Operation *definitionOp = symbolTable.lookupSymbolIn(
+      symTabOp, StringAttr::get(this->getContext(), referencedSymbol));
+  if (!definitionOp)
+    return emitError() << "symbol @" << referencedSymbol << " is undefined";
+
+  Type globalType;
+  bool isMutable = false;
+  if (auto global = dyn_cast<GlobalOp>(definitionOp)) {
+    globalType = global.getType();
+    isMutable = global.getIsMutable();
+  } else if (auto globalImport = dyn_cast<GlobalImportOp>(definitionOp)) {
+    globalType = globalImport.getType();
+    isMutable = globalImport.getIsMutable();
+  } else {
+    return emitError() << "symbol @" << referencedSymbol
+                       << " is not a global symbol";
+  }
+
+  if (!isMutable)
+    return emitError("global.set target must be mutable");
+
+  Type valueType = getValue().getType();
+  if (globalType != valueType)
+    return emitError("global.set value type does not match target global "
+                     "type: expected ")
+           << globalType << " but got " << valueType;
+
+  return success();
+}
+
 //===----------------------------------------------------------------------===//
 // GlobalImportOp
 //===----------------------------------------------------------------------===//

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
new file mode 100644
index 0000000000000..937f7efd006ba
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-global-set-to-memref.mlir
@@ -0,0 +1,35 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+wasmssa.global @global_i32 i32 mutable : {
+  %0 = wasmssa.const 10 : i32
+  wasmssa.return %0 : i32
+}
+
+wasmssa.global @global_f64 f64 mutable : {
+  %0 = wasmssa.const 3.14 : f64
+  wasmssa.return %0 : f64
+}
+
+// CHECK-LABEL:   func.func @set_global_i32() {
+wasmssa.func exported @set_global_i32() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 42 : i32
+  %0 = wasmssa.const 42 : i32
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_i32 : memref<1xi32>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xi32>
+  wasmssa.global_set @global_i32 to %0 : i32
+// CHECK:           return
+  wasmssa.return
+}
+
+// CHECK-LABEL:   func.func @set_global_f64() {
+wasmssa.func exported @set_global_f64() {
+// CHECK:           %[[VAL_0:.*]] = arith.constant 2.500000e-01 : f64
+  %0 = wasmssa.const 0.25 : f64
+// CHECK:           %[[VAL_1:.*]] = memref.get_global @global_f64 : memref<1xf64>
+// CHECK:           %[[VAL_2:.*]] = arith.constant 0 : index
+// CHECK:           memref.store %[[VAL_0]], %[[VAL_1]]{{\[}}%[[VAL_2]]] : memref<1xf64>
+  wasmssa.global_set @global_f64 to %0 : f64
+// CHECK:           return
+  wasmssa.return
+}

diff  --git a/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
new file mode 100644
index 0000000000000..ed5d15f87d70f
--- /dev/null
+++ b/mlir/test/Conversion/RaiseWasm/wasm-select-to-arith-select.mlir
@@ -0,0 +1,65 @@
+// RUN: mlir-opt %s --raise-wasm-mlir | FileCheck %s
+
+// CHECK-LABEL:   func.func @select_i32() -> i32 {
+wasmssa.func @select_i32() -> i32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i32
+  %a = wasmssa.const 12 : i32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i32
+  %b = wasmssa.const 50 : i32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i32
+  %r = wasmssa.select %cond, %a, %b : i32
+// CHECK:           return %[[RES]] : i32
+  wasmssa.return %r : i32
+}
+
+// CHECK-LABEL:   func.func @select_i64() -> i64 {
+wasmssa.func @select_i64() -> i64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 12 : i64
+  %a = wasmssa.const 12 : i64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 50 : i64
+  %b = wasmssa.const 50 : i64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : i64
+  %r = wasmssa.select %cond, %a, %b : i64
+// CHECK:           return %[[RES]] : i64
+  wasmssa.return %r : i64
+}
+
+// CHECK-LABEL:   func.func @select_f32() -> f32 {
+wasmssa.func @select_f32() -> f32 {
+// CHECK:           %[[COND:.*]] = arith.constant 1 : i32
+  %cond = wasmssa.const 1 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 1.250000e-01 : f32
+  %a = wasmssa.const 0.125 : f32
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.500000e-01 : f32
+  %b = wasmssa.const 0.25 : f32
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f32
+  %r = wasmssa.select %cond, %a, %b : f32
+// CHECK:           return %[[RES]] : f32
+  wasmssa.return %r : f32
+}
+
+// CHECK-LABEL:   func.func @select_f64() -> f64 {
+wasmssa.func @select_f64() -> f64 {
+// CHECK:           %[[COND:.*]] = arith.constant 0 : i32
+  %cond = wasmssa.const 0 : i32
+// CHECK:           %[[VAL_A:.*]] = arith.constant 3.140000e+00 : f64
+  %a = wasmssa.const 3.14 : f64
+// CHECK:           %[[VAL_B:.*]] = arith.constant 2.718000e+00 : f64
+  %b = wasmssa.const 2.718 : f64
+// CHECK:           %[[ZERO:.*]] = arith.constant 0 : i32
+// CHECK:           %[[FLAG:.*]] = arith.cmpi ne, %[[COND]], %[[ZERO]] : i32
+// CHECK:           %[[RES:.*]] = arith.select %[[FLAG]], %[[VAL_A]], %[[VAL_B]] : f64
+  %r = wasmssa.select %cond, %a, %b : f64
+// CHECK:           return %[[RES]] : f64
+  wasmssa.return %r : f64
+}

diff  --git a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
index a25abbd2a8662..6441731a0a335 100644
--- a/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/global.mlir
@@ -19,6 +19,12 @@ module {
     %0 = wasmssa.global_get @global_0 : i32
     wasmssa.return %0 : i32
   }
+
+  wasmssa.func @set_global() {
+    %0 = wasmssa.const 7 : i32
+    wasmssa.global_set @global_2 to %0 : i32
+    wasmssa.return
+  }
 }
 
 // CHECK-LABEL:   wasmssa.import_global "from_js" from "env" as @global_0 : i32
@@ -42,3 +48,9 @@ module {
 // CHECK:           %[[VAL_0:.*]] = wasmssa.global_get @global_0 : i32
 // CHECK:           wasmssa.return %[[VAL_0]] : i32
 // CHECK:         }
+
+// CHECK-LABEL:   wasmssa.func @set_global() {
+// CHECK:           %[[VAL_0:.*]] = wasmssa.const 7 : i32
+// CHECK:           wasmssa.global_set @global_2 to %[[VAL_0]] : i32
+// CHECK:           wasmssa.return
+// CHECK:         }

diff  --git a/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
new file mode 100644
index 0000000000000..80ced11c01bc3
--- /dev/null
+++ b/mlir/test/Dialect/WasmSSA/custom_parser/select.mlir
@@ -0,0 +1,37 @@
+// RUN: mlir-opt %s | FileCheck %s
+
+module {
+  wasmssa.func @select_i32(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to i32>,
+                           %b: !wasmssa<local ref to i32>) -> i32 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to i32
+    %2 = wasmssa.local_get %b : ref to i32
+    %r = wasmssa.select %0, %1, %2 : i32
+    wasmssa.return %r : i32
+  }
+
+  wasmssa.func @select_f64(%cond: !wasmssa<local ref to i32>,
+                           %a: !wasmssa<local ref to f64>,
+                           %b: !wasmssa<local ref to f64>) -> f64 {
+    %0 = wasmssa.local_get %cond : ref to i32
+    %1 = wasmssa.local_get %a : ref to f64
+    %2 = wasmssa.local_get %b : ref to f64
+    %r = wasmssa.select %0, %1, %2 : f64
+    wasmssa.return %r : f64
+  }
+}
+
+// CHECK-LABEL:   wasmssa.func @select_i32(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : i32
+// CHECK:           wasmssa.return %[[R]] : i32
+
+// CHECK-LABEL:   wasmssa.func @select_f64(
+// CHECK:           %[[COND:.*]] = wasmssa.local_get
+// CHECK:           %[[A:.*]] = wasmssa.local_get
+// CHECK:           %[[B:.*]] = wasmssa.local_get
+// CHECK:           %[[R:.*]] = wasmssa.select %[[COND]], %[[A]], %[[B]] : f64
+// CHECK:           wasmssa.return %[[R]] : f64

diff  --git a/mlir/test/Dialect/WasmSSA/global-invalid.mlir b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
index c5bc606fd13f3..600b2f0715fdc 100644
--- a/mlir/test/Dialect/WasmSSA/global-invalid.mlir
+++ b/mlir/test/Dialect/WasmSSA/global-invalid.mlir
@@ -40,3 +40,58 @@ module {
     wasmssa.return %0 : i32
   }
 }
+
+// -----
+
+module {
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @missing is undefined}}
+    wasmssa.global_set @missing to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i32 : {
+    %0 = wasmssa.const 0 : i32
+    wasmssa.return %0 : i32
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set target must be mutable}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.global @global_0 i64 mutable : {
+    %0 = wasmssa.const 0 : i64
+    wasmssa.return %0 : i64
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{global.set value type does not match target global type: expected 'i64' but got 'i32'}}
+    wasmssa.global_set @global_0 to %0 : i32
+    wasmssa.return
+  }
+}
+
+// -----
+
+module {
+  wasmssa.func @not_global() {
+    wasmssa.return
+  }
+  wasmssa.func @f() {
+    %0 = wasmssa.const 1 : i32
+    // expected-error at +1 {{symbol @not_global is not a global symbol}}
+    wasmssa.global_set @not_global to %0 : i32
+    wasmssa.return
+  }
+}


        

From llvmlistbot at llvm.org  Sat Jul 25 18:36:02 2026
From: llvmlistbot at llvm.org (Ferdinand Lemaire)
Date: Sat, 25 Jul 2026 18:36:02 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a656482.8e75e0cb.2de827.eb94@mx.google.com>


https://github.com/flemairen6 closed https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Sat Jul 25 18:36:19 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sat, 25 Jul 2026 18:36:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][WasmSSA] Add global_set and select ops
	(PR #196613)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/196613@github.com>
Message-ID: <6a656493.92d2873c.287016.50a4@mx.google.com>


github-actions[bot] wrote:

<!--LLVM BUILDBOT INFORMATION COMMENT-->

@byeongjee Congratulations on having your first Pull Request (PR) merged into the LLVM Project!

Your changes will be combined with recent changes from other authors, then tested by our [build bots](https://lab.llvm.org/buildbot/). If there is a problem with a build, you may receive a report in an email or a comment on this PR.

Please check whether problems have been caused by your change specifically, as the builds can include changes from many authors. It is not uncommon for your change to be included in a build that fails due to someone else's changes, or infrastructure issues.

How to do this, and the rest of the post-merge process, is covered in detail [here](https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr).

If your change does cause a problem, it may be reverted, or you can revert it yourself. This is a normal part of [LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy). You can fix your changes and open a new PR to merge them again.

If you don't get any reports, no action is required from you. Your changes are working as expected, well done!


https://github.com/llvm/llvm-project/pull/196613

From llvmlistbot at llvm.org  Sat Jul 25 19:26:31 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sat, 25 Jul 2026 19:26:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a657057.21a19891.ef87d.ed32@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH 1/2] [mlir][affine] Treat failed dependence checks
 conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}

>From 5789f7fdc90c045761a5d96d643633394bc1e6e3 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 26 Jul 2026 11:26:07 +0900
Subject: [PATCH 2/2] [mlir][affine] Distinguish possible from proven
 dependences

---
 .../mlir/Dialect/Affine/Analysis/AffineAnalysis.h    | 12 +++++++++---
 mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp  |  2 +-
 mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp    |  2 +-
 mlir/lib/Dialect/Affine/Utils/Utils.cpp              |  2 +-
 mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp |  2 +-
 5 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 6bfa887e95fd1..a001be9c42346 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -176,12 +176,18 @@ DependenceResult checkMemrefAccessDependence(
     SmallVector<DependenceComponent, 2> *dependenceComponents = nullptr,
     bool allowRAR = false);
 
-/// Utility function that returns true if the provided DependenceResult
-/// corresponds to a dependence result.
-inline bool hasDependence(DependenceResult result) {
+/// Returns true if the provided DependenceResult proves that a dependence
+/// exists.
+inline bool mustHaveDependence(DependenceResult result) {
   return result.value == DependenceResult::HasDependence;
 }
 
+/// Returns true unless the provided DependenceResult proves that no dependence
+/// exists.
+inline bool mayHaveDependence(DependenceResult result) {
+  return result.value != DependenceResult::NoDependence;
+}
+
 /// Returns true if the provided DependenceResult corresponds to the absence of
 /// a dependence.
 inline bool noDependence(DependenceResult result) {
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index aef5ec85858a7..f86c5e0e8421e 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -721,7 +721,7 @@ LogicalResult mlir::affine::getDependenceComponents(
             &depComps);
         if (result.value == DependenceResult::Failure)
           return failure();
-        if (hasDependence(result))
+        if (mustHaveDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
index 68296ea3368a1..984015943251e 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
@@ -229,7 +229,7 @@ static unsigned getMaxLoopDepth(ArrayRef<Operation *> srcOps,
         // TODO: Cache dependence analysis results, check cache here.
         DependenceResult result =
             checkMemrefAccessDependence(srcAccess, dstAccess, d);
-        if (hasDependence(result)) {
+        if (mayHaveDependence(result)) {
           // Store minimum loop depth and break because we want the min 'd' at
           // which there is a dependence.
           loopDepth = std::min(loopDepth, d - 1);
diff --git a/mlir/lib/Dialect/Affine/Utils/Utils.cpp b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
index 7043083298615..ec69aba8619f5 100644
--- a/mlir/lib/Dialect/Affine/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
@@ -646,7 +646,7 @@ static bool mustReachAtInnermost(const MemRefAccess &srcAccess,
       getNumCommonSurroundingLoops(*srcAccess.opInst, *destAccess.opInst);
   DependenceResult result =
       checkMemrefAccessDependence(srcAccess, destAccess, nsLoops + 1);
-  return hasDependence(result);
+  return mustHaveDependence(result);
 }
 
 /// Returns true if `srcMemOp` may have an effect on `destMemOp` within the
diff --git a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
index b3b9a590773b0..b08e1e7cd1df0 100644
--- a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
+++ b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
@@ -89,7 +89,7 @@ static void checkDependences(ArrayRef<Operation *> loadsAndStores) {
         if (result.value == DependenceResult::Failure) {
           srcOpInst->emitError("dependence check failed");
         } else {
-          bool ret = hasDependence(result);
+          bool ret = mustHaveDependence(result);
           // TODO: Print dependence type (i.e. RAW, etc) and print
           // distance vectors as: ([2, 3], [0, 10]). Also, shorten distance
           // vectors from ([1, 1], [3, 3]) to (1, 3).


From llvmlistbot at llvm.org  Sat Jul 25 20:33:36 2026
From: llvmlistbot at llvm.org (Mingfei Guo)
Date: Sat, 25 Jul 2026 20:33:36 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [MLIR][Affine] Check access coverage before
 fusion privatization (PR #212080)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212080@github.com>
Message-ID: <6a658010.b4a331f0.1c45b5.3857@mx.google.com>


https://github.com/guoriyue edited https://github.com/llvm/llvm-project/pull/212080

From llvmlistbot at llvm.org  Sat Jul 25 21:10:06 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sat, 25 Jul 2026 21:10:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a65889e.bd93e731.32a2a8.d21e@mx.google.com>


takatodo wrote:

@ftynse 
Implemented option (a), thanks. hasDependence is now mustHaveDependence, and mayHaveDependence conservatively returns true for Failure. 
I also audited the callers: loop-fusion legality uses mayHaveDependence, while callers requiring a proven dependence use mustHaveDependence.

https://github.com/llvm/llvm-project/pull/211014

From llvmlistbot at llvm.org  Sat Jul 25 21:10:12 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sat, 25 Jul 2026 21:10:12 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][affine] Treat failed dependence checks
 conservatively (PR #211014)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/211014@github.com>
Message-ID: <6a6588a4.1728eaf3.12b96d.95c4@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/211014

>From 58adfdcb1eb0542803ada861317124ecbf9fbaa8 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 19 Jul 2026 17:57:36 +0900
Subject: [PATCH 1/2] [mlir][affine] Treat failed dependence checks
 conservatively

Reject tiling and loop permutation when Affine dependence analysis cannot construct a complete relation, and preserve the original loop nest when sequential-loop sinking encounters the same failure. Add regressions for a semi-affine same-memref dependence.

Fixes #210585

Assisted-by: OpenAI Codex
---
 .../Dialect/Affine/Analysis/AffineAnalysis.h  |  4 +--
 .../Affine/Analysis/AffineAnalysis.cpp        |  5 +++-
 .../Dialect/Affine/Analysis/LoopAnalysis.cpp  |  5 +++-
 mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp   |  6 +++--
 .../Affine/loop-transformation-validity.mlir  | 25 +++++++++++++++++++
 5 files changed, 39 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Affine/loop-transformation-validity.mlir

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 3e4b8648061ff..6bfa887e95fd1 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -190,8 +190,8 @@ inline bool noDependence(DependenceResult result) {
 
 /// Returns in 'depCompsVec', dependence components for dependences between all
 /// load and store ops in loop nest rooted at 'forOp', at loop depths in range
-/// [1, maxLoopDepth].
-void getDependenceComponents(
+/// [1, maxLoopDepth]. Returns failure if any dependence cannot be analyzed.
+LogicalResult getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec);
 
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index 3d1a73417d1ea..aef5ec85858a7 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -694,7 +694,7 @@ DependenceResult mlir::affine::checkMemrefAccessDependence(
 
 /// Gathers dependence components for dependences between all ops in loop nest
 /// rooted at 'forOp' at loop depths in range [1, maxLoopDepth].
-void mlir::affine::getDependenceComponents(
+LogicalResult mlir::affine::getDependenceComponents(
     AffineForOp forOp, unsigned maxLoopDepth,
     std::vector<SmallVector<DependenceComponent, 2>> *depCompsVec) {
   // Collect all load and store ops in loop nest rooted at 'forOp'.
@@ -719,9 +719,12 @@ void mlir::affine::getDependenceComponents(
         DependenceResult result = checkMemrefAccessDependence(
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
+        if (result.value == DependenceResult::Failure)
+          return failure();
         if (hasDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
   }
+  return success();
 }
diff --git a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
index 40802cc6e85e5..2dbc1baa02236 100644
--- a/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/LoopAnalysis.cpp
@@ -524,8 +524,11 @@ bool mlir::affine::isTilingValid(ArrayRef<AffineForOp> loops) {
             srcAccess, dstAccess, d, /*dependenceConstraints=*/nullptr,
             &depComps);
 
+        if (result.value == DependenceResult::Failure)
+          return false;
+
         // Skip if there is no dependence in this case.
-        if (!hasDependence(result))
+        if (noDependence(result))
           continue;
 
         // Check whether there is any negative direction vector in the
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
index 90bc57e950cf1..bb002229d7b50 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopUtils.cpp
@@ -1357,7 +1357,8 @@ bool mlir::affine::isValidLoopInterchangePermutation(
   // Gather dependence components for dependences between all ops in loop nest
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return false;
   return checkLoopInterchangeDependences(depCompsVec, loops, loopPermMap);
 }
 
@@ -1466,7 +1467,8 @@ AffineForOp mlir::affine::sinkSequentialLoops(AffineForOp forOp) {
   // rooted at 'loops[0]', at loop depths in range [1, maxLoopDepth].
   unsigned maxLoopDepth = loops.size();
   std::vector<SmallVector<DependenceComponent, 2>> depCompsVec;
-  getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec);
+  if (failed(getDependenceComponents(loops[0], maxLoopDepth, &depCompsVec)))
+    return forOp;
 
   // Mark loops as either parallel or sequential.
   SmallVector<bool, 8> isParallelLoop(maxLoopDepth, true);
diff --git a/mlir/test/Dialect/Affine/loop-transformation-validity.mlir b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
new file mode 100644
index 0000000000000..b57e9ddcc357b
--- /dev/null
+++ b/mlir/test/Dialect/Affine/loop-transformation-validity.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -test-loop-permutation="permutation-map=1,0 check-validity=1" | FileCheck %s
+// RUN: mlir-opt %s -affine-loop-tile="tile-size=4" | FileCheck %s
+
+#dynamic_index = affine_map<()[s0, s1] -> (s0 * s1)>
+
+// Dependence analysis cannot represent the common semi-affine index. The
+// remaining indices carry a (1, -1) dependence, so both transforms must fail.
+// CHECK-LABEL: func.func @unknown_dependence
+func.func @unknown_dependence(
+    %A: memref<?x9x9xi32>, %B: memref<9x9xi32>,
+    %p: index, %q: index, %value: i32) {
+  // CHECK:      affine.for %[[I:.*]] = 1 to 8 {
+  // CHECK-NEXT:   affine.for %[[J:.*]] = 1 to 8 {
+  affine.for %i = 1 to 8 {
+    affine.for %j = 1 to 8 {
+      %z = affine.apply #dynamic_index()[%p, %q]
+      // CHECK: affine.store %{{.*}}, %{{.*}}[%{{.*}}, %[[I]], %[[J]]]
+      affine.store %value, %A[%z, %i, %j] : memref<?x9x9xi32>
+      %loaded = affine.load %A[%z, %i - 1, %j + 1]
+          : memref<?x9x9xi32>
+      affine.store %loaded, %B[%i, %j] : memref<9x9xi32>
+    }
+  }
+  return
+}

>From 5789f7fdc90c045761a5d96d643633394bc1e6e3 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 26 Jul 2026 11:26:07 +0900
Subject: [PATCH 2/2] [mlir][affine] Distinguish possible from proven
 dependences

---
 .../mlir/Dialect/Affine/Analysis/AffineAnalysis.h    | 12 +++++++++---
 mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp  |  2 +-
 mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp    |  2 +-
 mlir/lib/Dialect/Affine/Utils/Utils.cpp              |  2 +-
 mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp |  2 +-
 5 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
index 6bfa887e95fd1..a001be9c42346 100644
--- a/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
+++ b/mlir/include/mlir/Dialect/Affine/Analysis/AffineAnalysis.h
@@ -176,12 +176,18 @@ DependenceResult checkMemrefAccessDependence(
     SmallVector<DependenceComponent, 2> *dependenceComponents = nullptr,
     bool allowRAR = false);
 
-/// Utility function that returns true if the provided DependenceResult
-/// corresponds to a dependence result.
-inline bool hasDependence(DependenceResult result) {
+/// Returns true if the provided DependenceResult proves that a dependence
+/// exists.
+inline bool mustHaveDependence(DependenceResult result) {
   return result.value == DependenceResult::HasDependence;
 }
 
+/// Returns true unless the provided DependenceResult proves that no dependence
+/// exists.
+inline bool mayHaveDependence(DependenceResult result) {
+  return result.value != DependenceResult::NoDependence;
+}
+
 /// Returns true if the provided DependenceResult corresponds to the absence of
 /// a dependence.
 inline bool noDependence(DependenceResult result) {
diff --git a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
index aef5ec85858a7..f86c5e0e8421e 100644
--- a/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
+++ b/mlir/lib/Dialect/Affine/Analysis/AffineAnalysis.cpp
@@ -721,7 +721,7 @@ LogicalResult mlir::affine::getDependenceComponents(
             &depComps);
         if (result.value == DependenceResult::Failure)
           return failure();
-        if (hasDependence(result))
+        if (mustHaveDependence(result))
           depCompsVec->push_back(depComps);
       }
     }
diff --git a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
index 68296ea3368a1..984015943251e 100644
--- a/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/LoopFusionUtils.cpp
@@ -229,7 +229,7 @@ static unsigned getMaxLoopDepth(ArrayRef<Operation *> srcOps,
         // TODO: Cache dependence analysis results, check cache here.
         DependenceResult result =
             checkMemrefAccessDependence(srcAccess, dstAccess, d);
-        if (hasDependence(result)) {
+        if (mayHaveDependence(result)) {
           // Store minimum loop depth and break because we want the min 'd' at
           // which there is a dependence.
           loopDepth = std::min(loopDepth, d - 1);
diff --git a/mlir/lib/Dialect/Affine/Utils/Utils.cpp b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
index 7043083298615..ec69aba8619f5 100644
--- a/mlir/lib/Dialect/Affine/Utils/Utils.cpp
+++ b/mlir/lib/Dialect/Affine/Utils/Utils.cpp
@@ -646,7 +646,7 @@ static bool mustReachAtInnermost(const MemRefAccess &srcAccess,
       getNumCommonSurroundingLoops(*srcAccess.opInst, *destAccess.opInst);
   DependenceResult result =
       checkMemrefAccessDependence(srcAccess, destAccess, nsLoops + 1);
-  return hasDependence(result);
+  return mustHaveDependence(result);
 }
 
 /// Returns true if `srcMemOp` may have an effect on `destMemOp` within the
diff --git a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
index b3b9a590773b0..b08e1e7cd1df0 100644
--- a/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
+++ b/mlir/test/lib/Analysis/TestMemRefDependenceCheck.cpp
@@ -89,7 +89,7 @@ static void checkDependences(ArrayRef<Operation *> loadsAndStores) {
         if (result.value == DependenceResult::Failure) {
           srcOpInst->emitError("dependence check failed");
         } else {
-          bool ret = hasDependence(result);
+          bool ret = mustHaveDependence(result);
           // TODO: Print dependence type (i.e. RAW, etc) and print
           // distance vectors as: ([2, 3], [0, 10]). Also, shorten distance
           // vectors from ([1, 1], [3, 3]) to (1, 3).


From llvmlistbot at llvm.org  Sun Jul 26 00:56:34 2026
From: llvmlistbot at llvm.org (Muyao Xiao)
Date: Sun, 26 Jul 2026 00:56:34 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [OpenMP][MLIR] RegionBranchOpInterface for
 TargetOp, TargetDataOp (PR #209710)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/209710@github.com>
Message-ID: <6a65bdb2.510f576d.23298e.992f@mx.google.com>


https://github.com/mooxiu updated https://github.com/llvm/llvm-project/pull/209710

>From d1f2d200b78e343af54901de5aa5506fb45572e7 Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 18:19:00 +0900
Subject: [PATCH 1/3] implement regional branch interface for omp::TargetOp,
 omp::TargetDataOp, and omp::terminatorOp

---
 mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td |  7 +++---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp  | 22 +++++++++++++++++++
 2 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
index 70597b85902c5..4cc322849591c 100644
--- a/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
+++ b/mlir/include/mlir/Dialect/OpenMP/OpenMPOps.td
@@ -222,7 +222,7 @@ def ParallelOp : OpenMP_Op<"parallel", traits = [
   let hasRegionVerifier = 1;
 }
 
-def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure]> {
+def TerminatorOp : OpenMP_Op<"terminator", [Terminator, Pure, RegionBranchTerminatorOpInterface]> {
   let summary = "terminator for OpenMP regions";
   let description = [{
     A terminator operation for regions that appear in the body of OpenMP
@@ -1457,7 +1457,8 @@ def MapInfoOp : OpenMP_Op<"map.info", [AttrSizedOperandSegments]> {
 //===---------------------------------------------------------------------===//
 
 def TargetDataOp: OpenMP_Op<"target_data", traits = [
-    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>
+    AttrSizedOperandSegments, DeclareOpInterfaceMethods<ComposableOpInterface>,
+    DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     OpenMP_DeviceClause, OpenMP_IfClause, OpenMP_MapClause,
     OpenMP_UseDeviceAddrClause, OpenMP_UseDevicePtrClause
@@ -1621,7 +1622,7 @@ def TargetUpdateOp: OpenMP_Op<"target_update", traits = [
 def TargetOp : OpenMP_Op<"target", traits = [
     AttrSizedOperandSegments, BlockArgOpenMPOpInterface,
     DeclareOpInterfaceMethods<ComposableOpInterface>, IsolatedFromAbove,
-    OutlineableOpenMPOpInterface
+    OutlineableOpenMPOpInterface, DeclareOpInterfaceMethods<RegionBranchOpInterface>
   ], clauses = [
     // TODO: Complete clause list (defaultmap, uses_allocators).
     OpenMP_AllocateClause, OpenMP_DependClause, OpenMP_DeviceClause,
diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 95ccb53bce3d4..2d73a073205bd 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2571,6 +2571,17 @@ LogicalResult TargetDataOp::verify() {
   return verifyMapClause(*this, getMapVars(), getMapIterated());
 }
 
+// Adapted from fir.if implementation.
+void TargetDataOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // TargetEnterDataOp
 //===----------------------------------------------------------------------===//
@@ -2853,6 +2864,17 @@ LogicalResult TargetOp::verifyRegions() {
   return success();
 }
 
+// Copy from `TargetDataOp::getSuccessorRegions`
+void TargetOp::getSuccessorRegions(
+  mlir::RegionBranchPoint point, 
+  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+  if (!point.isParent()) {
+    regions.push_back(mlir::RegionSuccessor::parent());
+    return;
+  }
+  regions.push_back(mlir::RegionSuccessor(&getRegion()));
+}
+
 //===----------------------------------------------------------------------===//
 // ParallelOp
 //===----------------------------------------------------------------------===//

>From 8ab20817c4c87cba0aa81b1cd851810e017460bb Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 19:22:52 +0900
Subject: [PATCH 2/3] update to latest API: no parent() anymore

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 2d73a073205bd..6290aeef0bca1 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2576,7 +2576,7 @@ void TargetDataOp::getSuccessorRegions(
   mlir::RegionBranchPoint point, 
   llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
   if (!point.isParent()) {
-    regions.push_back(mlir::RegionSuccessor::parent());
+    regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;
   }
   regions.push_back(mlir::RegionSuccessor(&getRegion()));
@@ -2869,7 +2869,7 @@ void TargetOp::getSuccessorRegions(
   mlir::RegionBranchPoint point, 
   llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
   if (!point.isParent()) {
-    regions.push_back(mlir::RegionSuccessor::parent());
+    regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;
   }
   regions.push_back(mlir::RegionSuccessor(&getRegion()));

>From 9e094fb66903a25d52e7258b2251a9f286256c6f Mon Sep 17 00:00:00 2001
From: Muyao Xiao <monetshaw at outlook.com>
Date: Wed, 15 Jul 2026 19:44:22 +0900
Subject: [PATCH 3/3] fix format

---
 mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
index 6290aeef0bca1..b3bbf441e5e75 100644
--- a/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
+++ b/mlir/lib/Dialect/OpenMP/IR/OpenMPDialect.cpp
@@ -2573,8 +2573,8 @@ LogicalResult TargetDataOp::verify() {
 
 // Adapted from fir.if implementation.
 void TargetDataOp::getSuccessorRegions(
-  mlir::RegionBranchPoint point, 
-  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+    mlir::RegionBranchPoint point,
+    llvm::SmallVectorImpl<::mlir::RegionSuccessor> &regions) {
   if (!point.isParent()) {
     regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;
@@ -2866,8 +2866,8 @@ LogicalResult TargetOp::verifyRegions() {
 
 // Copy from `TargetDataOp::getSuccessorRegions`
 void TargetOp::getSuccessorRegions(
-  mlir::RegionBranchPoint point, 
-  llvm::SmallVectorImpl<::mlir::RegionSuccessor>& regions) {
+    mlir::RegionBranchPoint point,
+    llvm::SmallVectorImpl<::mlir::RegionSuccessor> &regions) {
   if (!point.isParent()) {
     regions.push_back(mlir::RegionSuccessor(getOperation()));
     return;


From llvmlistbot at llvm.org  Sun Jul 26 02:25:59 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 26 Jul 2026 02:25:59 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a65d2a7.d1496011.1d7ce5.155d@mx.google.com>


FedericoBruzzone wrote:


> > A dynamic stride is still accepted, unlike vector.load/vector.store (tests added to document the asymmetry.
> 
> Why is this exception required?

Thanks for the comment @banach-space!  It gave me the chance to double-check.

The issue I had encountered was with the Sparse Vectorizer. That problem no longer arises.
This means we've managed to align the table (I’ve updated it, as well as the comment in the PR).
There's good code reuse now :D

https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Sun Jul 26 02:27:19 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 26 Jul 2026 02:27:19 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a65d2f7.7d702ed2.2f6c05.b5ed@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/210952

>From 73227259bfaa8e5b8866fb38e1b20277a7d08650 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:54 +0200
Subject: [PATCH 1/3] [mlir][vector] Reject non-unit strides on
 masked/expand/compress ops

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 51 ++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir    | 77 ++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 9322c11d401d9..8477ff57d967d 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
+/// statically known non-unit stride; a dynamic (not provably unit) stride
+/// is accepted.
+///
+/// This is more permissive than vector.load/store's stride check: ops such
+/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
+/// also used on memrefs whose most minor dimension is contiguous at runtime
+/// but not provable as such at the type level (e.g., buffers produced by
+/// the sparsifier).
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6326,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6395,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6681,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6705,6 +6747,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index 0097db39f6ed9..d01c2ad4e0c7b 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1741,6 +1759,24 @@ func.func @expand_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16x
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1795,6 +1831,24 @@ func.func @compress_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
@@ -2178,6 +2232,19 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
+// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
+// (unprovable) stride is rejected here too: vector.load/store require a
+// statically known unit stride, with no exception for strides that merely
+// aren't provably non-unit.
+func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
+  %0 = vector.load %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.store
 //===----------------------------------------------------------------------===//
@@ -2214,6 +2281,16 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
+// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
+// unlike vector.maskedstore.
+func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
+  // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
+  vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 func.func @store_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %val: vector<4xf32>) {
   // expected-error @+2 {{'vector.store' op memref strides must be non-negative}}
   %c0 = arith.constant 0 : index

>From 60221acb5a0c7c37a320a68c62df5021b3914b55 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Fri, 24 Jul 2026 13:21:16 +0200
Subject: [PATCH 2/3] Address comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 43 +++++-------------------
 1 file changed, 8 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 8477ff57d967d..07a0aca09e179 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,33 +6190,6 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
-/// Verifies that `memRefTy`'s most minor dimension does not have a
-/// statically known non-unit stride; a dynamic (not provably unit) stride
-/// is accepted.
-///
-/// This is more permissive than vector.load/store's stride check: ops such
-/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
-/// also used on memrefs whose most minor dimension is contiguous at runtime
-/// but not provable as such at the type level (e.g., buffers produced by
-/// the sparsifier).
-static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
-                                                          VectorType vecTy,
-                                                          MemRefType memRefTy) {
-  if (!vecTy.isScalable() &&
-      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
-    return success();
-
-  SmallVector<int64_t> strides;
-  int64_t offset;
-  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
-    return success();
-
-  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
-      strides.back() != 1)
-    return op->emitOpError("most minor memref dim must have unit stride");
-  return success();
-}
-
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6326,8 +6299,8 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6395,8 +6368,8 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6681,8 +6654,8 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.expandload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6747,8 +6720,8 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.compressstore. The lowering
   // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that

>From 7b563d37fe1777e4c7335c76e3eba228a60793c2 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Sun, 26 Jul 2026 11:26:42 +0200
Subject: [PATCH 3/3] Align to load/store

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 16 +++++------
 mlir/test/Dialect/Vector/invalid.mlir    | 36 ++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 07a0aca09e179..e169ca4fe0c82 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6299,8 +6299,8 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, resVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6368,8 +6368,8 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, valueVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6654,8 +6654,8 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, resVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.expandload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6720,8 +6720,8 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, valueVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.compressstore. The lowering
   // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index d01c2ad4e0c7b..4d91c37ae02b4 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1431,6 +1431,15 @@ func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>
 
 // -----
 
+func.func @maskedload_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1493,6 +1502,15 @@ func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?
 
 // -----
 
+func.func @maskedstore_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1768,6 +1786,15 @@ func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>
 
 // -----
 
+func.func @expandload_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
   %c0 = arith.constant 0 : index
   // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
@@ -1840,6 +1867,15 @@ func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset:
 
 // -----
 
+func.func @compressstore_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}


From llvmlistbot at llvm.org  Sun Jul 26 02:30:41 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 26 Jul 2026 02:30:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a65d3c1.21a19891.ef87d.649b@mx.google.com>


https://github.com/FedericoBruzzone updated https://github.com/llvm/llvm-project/pull/210952

>From 73227259bfaa8e5b8866fb38e1b20277a7d08650 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Tue, 21 Jul 2026 15:41:54 +0200
Subject: [PATCH 1/4] [mlir][vector] Reject non-unit strides on
 masked/expand/compress ops

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 51 ++++++++++++++++
 mlir/test/Dialect/Vector/invalid.mlir    | 77 ++++++++++++++++++++++++
 2 files changed, 128 insertions(+)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 9322c11d401d9..8477ff57d967d 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,6 +6190,33 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
+/// Verifies that `memRefTy`'s most minor dimension does not have a
+/// statically known non-unit stride; a dynamic (not provably unit) stride
+/// is accepted.
+///
+/// This is more permissive than vector.load/store's stride check: ops such
+/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
+/// also used on memrefs whose most minor dimension is contiguous at runtime
+/// but not provable as such at the type level (e.g., buffers produced by
+/// the sparsifier).
+static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
+                                                          VectorType vecTy,
+                                                          MemRefType memRefTy) {
+  if (!vecTy.isScalable() &&
+      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
+    return success();
+
+  SmallVector<int64_t> strides;
+  int64_t offset;
+  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
+    return success();
+
+  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
+      strides.back() != 1)
+    return op->emitOpError("most minor memref dim must have unit stride");
+  return success();
+}
+
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6299,6 +6326,9 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6365,6 +6395,9 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
   // assume non-negative strides to avoid undefined behavior.
@@ -6648,6 +6681,15 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.expandload. The lowering to
+  // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(
           verifyElementTypesMatch(*this, memType, resVType, "base", "result")))
     return failure();
@@ -6705,6 +6747,15 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
+  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
+    return failure();
+
+  // Negative strides are not supported on vector.compressstore. The lowering
+  // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
+  // assume non-negative strides to avoid undefined behavior.
+  if (memref::hasNegativeStaticStride(memType))
+    return emitOpError("memref strides must be non-negative");
+
   if (failed(verifyElementTypesMatch(*this, memType, valueVType, "base",
                                      "valueToStore")))
     return failure();
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index 0097db39f6ed9..d01c2ad4e0c7b 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1422,6 +1422,15 @@ func.func @maskedload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1
 
 // -----
 
+func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1475,6 +1484,15 @@ func.func @maskedstore_negative_stride(%src: memref<100x100xf32, strided<[-100,
 
 // -----
 
+func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1741,6 +1759,24 @@ func.func @expand_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<16x
 
 // -----
 
+func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
+func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
+  %0 = vector.expandload %src[%c0, %c0], %mask, %pass_thru : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32> into vector<8xf32>
+  return %0 : vector<8xf32>
+}
+
+// -----
+
 func.func @compress_base_type_mismatch(%base: memref<?xf64>, %mask: vector<16xi1>, %value: vector<16xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error at +1 {{'vector.compressstore' op base element type ('f64') does not match valueToStore element type ('f32')}}
@@ -1795,6 +1831,24 @@ func.func @compress_scalable_dims_mismatch(%base: memref<?xf32>, %mask: vector<1
 
 // -----
 
+func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[2], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
+func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}
+  vector.compressstore %src[%c0, %c0], %mask, %value : memref<100x100xf32, strided<[-100, 1]>>, vector<8xi1>, vector<8xf32>
+  return
+}
+
+// -----
+
 func.func @scan_reduction_dim_constraint(%arg0: vector<2x3xi32>, %arg1: vector<3xi32>) -> vector<3xi32> {
   // expected-error at +1 {{'vector.scan' op reduction dimension 5 has to be less than 2}}
   %0:2 = vector.scan <add>, %arg0, %arg1 {inclusive = true, reduction_dim = 5} :
@@ -2178,6 +2232,19 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
+// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
+// (unprovable) stride is rejected here too: vector.load/store require a
+// statically known unit stride, with no exception for strides that merely
+// aren't provably non-unit.
+func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
+  %0 = vector.load %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.store
 //===----------------------------------------------------------------------===//
@@ -2214,6 +2281,16 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
+// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
+// unlike vector.maskedstore.
+func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
+  // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
+  vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>
+  return
+}
+
+// -----
+
 func.func @store_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %val: vector<4xf32>) {
   // expected-error @+2 {{'vector.store' op memref strides must be non-negative}}
   %c0 = arith.constant 0 : index

>From 60221acb5a0c7c37a320a68c62df5021b3914b55 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Fri, 24 Jul 2026 13:21:16 +0200
Subject: [PATCH 2/4] Address comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 43 +++++-------------------
 1 file changed, 8 insertions(+), 35 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 8477ff57d967d..07a0aca09e179 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6190,33 +6190,6 @@ static LogicalResult verifyLoadStoreMemRefLayout(Operation *op,
   return success();
 }
 
-/// Verifies that `memRefTy`'s most minor dimension does not have a
-/// statically known non-unit stride; a dynamic (not provably unit) stride
-/// is accepted.
-///
-/// This is more permissive than vector.load/store's stride check: ops such
-/// as vector.maskedload/maskedstore and vector.expandload/compressstore are
-/// also used on memrefs whose most minor dimension is contiguous at runtime
-/// but not provable as such at the type level (e.g., buffers produced by
-/// the sparsifier).
-static LogicalResult verifyNonStaticNonUnitStrideRejected(Operation *op,
-                                                          VectorType vecTy,
-                                                          MemRefType memRefTy) {
-  if (!vecTy.isScalable() &&
-      (vecTy.getRank() == 0 || vecTy.getNumElements() == 1))
-    return success();
-
-  SmallVector<int64_t> strides;
-  int64_t offset;
-  if (failed(memRefTy.getStridesAndOffset(strides, offset)))
-    return success();
-
-  if (!strides.empty() && !ShapedType::isDynamic(strides.back()) &&
-      strides.back() != 1)
-    return op->emitOpError("most minor memref dim must have unit stride");
-  return success();
-}
-
 LogicalResult vector::LoadOp::verify() {
   VectorType resVecTy = getVectorType();
   MemRefType memRefTy = getMemRefType();
@@ -6326,8 +6299,8 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6395,8 +6368,8 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6681,8 +6654,8 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, resVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.expandload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6747,8 +6720,8 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (failed(verifyNonStaticNonUnitStrideRejected(*this, valueVType, memType)))
-    return failure();
+  if (!memType.isLastDimUnitStride())
+    return emitOpError("most minor memref dim must have unit stride");
 
   // Negative strides are not supported on vector.compressstore. The lowering
   // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that

>From 7b563d37fe1777e4c7335c76e3eba228a60793c2 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Sun, 26 Jul 2026 11:26:42 +0200
Subject: [PATCH 3/4] Align to load/store

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/lib/Dialect/Vector/IR/VectorOps.cpp | 16 +++++------
 mlir/test/Dialect/Vector/invalid.mlir    | 36 ++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 8 deletions(-)

diff --git a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
index 07a0aca09e179..e169ca4fe0c82 100644
--- a/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/IR/VectorOps.cpp
@@ -6299,8 +6299,8 @@ LogicalResult MaskedLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, resVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.maskedload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6368,8 +6368,8 @@ LogicalResult MaskedStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, valueVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.maskedstore. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6654,8 +6654,8 @@ LogicalResult ExpandLoadOp::verify() {
   VectorType resVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, resVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.expandload. The lowering to
   // LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
@@ -6720,8 +6720,8 @@ LogicalResult CompressStoreOp::verify() {
   VectorType valueVType = getVectorType();
   MemRefType memType = getMemRefType();
 
-  if (!memType.isLastDimUnitStride())
-    return emitOpError("most minor memref dim must have unit stride");
+  if (failed(verifyLoadStoreMemRefLayout(*this, valueVType, memType)))
+    return failure();
 
   // Negative strides are not supported on vector.compressstore. The lowering
   // to LLVM emits arithmetic operations (e.g., GEP, mul) with nuw flags that
diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index d01c2ad4e0c7b..4d91c37ae02b4 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -1431,6 +1431,15 @@ func.func @maskedload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>
 
 // -----
 
+func.func @maskedload_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %pass: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedload' op most minor memref dim must have unit stride}}
+  %0 = vector.maskedload %src[%c0], %mask, %pass : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 //===----------------------------------------------------------------------===//
 // vector.maskedstore
 //===----------------------------------------------------------------------===//
@@ -1493,6 +1502,15 @@ func.func @maskedstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?
 
 // -----
 
+func.func @maskedstore_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.maskedstore' op most minor memref dim must have unit stride}}
+  vector.maskedstore %src[%c0], %mask, %value : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @gather_from_vector(%base: vector<16xf32>, %indices: vector<16xi32>,
                                 %mask: vector<16xi1>, %pass_thru: vector<16xf32>) {
   %c0 = arith.constant 0 : index
@@ -1768,6 +1786,15 @@ func.func @expandload_non_unit_stride(%src: memref<?xi8, strided<[2], offset: ?>
 
 // -----
 
+func.func @expandload_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %pass_thru: vector<8xi8>) -> vector<8xi8> {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.expandload' op most minor memref dim must have unit stride}}
+  %0 = vector.expandload %src[%c0], %mask, %pass_thru : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8> into vector<8xi8>
+  return %0 : vector<8xi8>
+}
+
+// -----
+
 func.func @expandload_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %pass_thru: vector<8xf32>) -> vector<8xf32> {
   %c0 = arith.constant 0 : index
   // expected-error @+1 {{'vector.expandload' op memref strides must be non-negative}}
@@ -1840,6 +1867,15 @@ func.func @compressstore_non_unit_stride(%src: memref<?xi8, strided<[2], offset:
 
 // -----
 
+func.func @compressstore_dynamic_stride(%src: memref<?xi8, strided<[?], offset: ?>>, %mask: vector<8xi1>, %value: vector<8xi8>) {
+  %c0 = arith.constant 0 : index
+  // expected-error @+1 {{'vector.compressstore' op most minor memref dim must have unit stride}}
+  vector.compressstore %src[%c0], %mask, %value : memref<?xi8, strided<[?], offset: ?>>, vector<8xi1>, vector<8xi8>
+  return
+}
+
+// -----
+
 func.func @compressstore_negative_stride(%src: memref<100x100xf32, strided<[-100, 1]>>, %mask: vector<8xi1>, %value: vector<8xf32>) {
   %c0 = arith.constant 0 : index
   // expected-error @+1 {{'vector.compressstore' op memref strides must be non-negative}}

>From 7e1f210b20d7bf70eac1a5db81d11c15becc8d15 Mon Sep 17 00:00:00 2001
From: Federico Bruzzone <federico.bruzzone.i at gmail.com>
Date: Sun, 26 Jul 2026 11:30:19 +0200
Subject: [PATCH 4/4] Remove useless comments

Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
---
 mlir/test/Dialect/Vector/invalid.mlir | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/mlir/test/Dialect/Vector/invalid.mlir b/mlir/test/Dialect/Vector/invalid.mlir
index 4d91c37ae02b4..4e5788dc28899 100644
--- a/mlir/test/Dialect/Vector/invalid.mlir
+++ b/mlir/test/Dialect/Vector/invalid.mlir
@@ -2268,10 +2268,6 @@ func.func @load_non_unit_stride(%src : memref<?xi8, strided<[2], offset: ?>>) {
 
 // -----
 
-// Unlike vector.maskedload/maskedstore/expandload/compressstore, a dynamic
-// (unprovable) stride is rejected here too: vector.load/store require a
-// statically known unit stride, with no exception for strides that merely
-// aren't provably non-unit.
 func.func @load_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>) {
   %c0 = arith.constant 0 : index
   // expected-error @+1 {{'vector.load' op most minor memref dim must have unit stride}}
@@ -2317,8 +2313,6 @@ func.func @store_non_unit_stride(%src : memref<?xi8, strided<[2], offset:?>>,%va
 
 // -----
 
-// See load_dynamic_stride above: vector.store also rejects a dynamic stride,
-// unlike vector.maskedstore.
 func.func @store_dynamic_stride(%src : memref<?xi8, strided<[?], offset: ?>>, %val : vector<16xi8>, %c0: index) {
   // expected-error @below {{'vector.store' op most minor memref dim must have unit stride}}
   vector.store %val, %src[%c0] : memref<?xi8, strided<[?], offset: ?>>, vector<16xi8>


From llvmlistbot at llvm.org  Sun Jul 26 02:31:37 2026
From: llvmlistbot at llvm.org (Federico Bruzzone)
Date: Sun, 26 Jul 2026 02:31:37 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][vector] Verify non-unit strides on
 `masked/expand/compress` ops (PR #210952)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/210952@github.com>
Message-ID: <6a65d3f9.d681cab9.3d08c3.aa1a@mx.google.com>


https://github.com/FedericoBruzzone edited https://github.com/llvm/llvm-project/pull/210952

From llvmlistbot at llvm.org  Sun Jul 26 06:04:09 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 26 Jul 2026 06:04:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reject partial softmax
	reduction tiling (PR #212117)
Message-ID: <llvm.org/llvm/llvm-project/pull/212117@github.com>


https://github.com/takatodo created https://github.com/llvm/llvm-project/pull/212117

@qcolombet @harsh-nod This fixes a correctness issue in the original
SoftmaxOp TilingInterface implementation.

Partial tiling of the reduction dimension cloned an independent softmax for each slice. 
For example, tiling [0, 0, 0, 0] by 2 produced [0.5, 0.5, 0.5, 0.5] instead of [0.25, 0.25, 0.25, 0.25].

The change requires the reduction-dimension tile to start at offset zero and uses ValueBoundsConstraintSet to prove that its size covers the full reduction dimension. Dynamically shaped cases are accepted when this equality can be proven.

Assisted-by: OpenAI Codex

>From c189ff32f8ec41eb0103cfcf710f4b5fb78ce81d Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 26 Jul 2026 21:20:03 +0900
Subject: [PATCH] [mlir][linalg] Reject partial softmax reduction tiling

Assisted-by: OpenAI Codex
---
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      | 19 ++++++++
 .../Dialect/Linalg/tile-softmax-invalid.mlir  | 25 ++++++++++
 mlir/test/Dialect/Linalg/tile-softmax.mlir    | 46 ++++++++++++++++---
 3 files changed, 84 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir

diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..7a6b9b0d0f96f 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -37,6 +37,7 @@
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
@@ -2940,6 +2941,18 @@ SmallVector<utils::IteratorType> SoftmaxOp::getLoopIteratorTypes() {
   return iteratorTypes;
 }
 
+static bool isFullSoftmaxDimensionTile(OpFoldResult offset, OpFoldResult size,
+                                       Value source, int64_t dimension) {
+  // Softmax cannot be computed independently on parts of its reduction
+  // dimension without combining their maxima and normalization factors.
+  if (!isZeroInteger(offset))
+    return false;
+  FailureOr<bool> isFullDimension = ValueBoundsConstraintSet::areEqual(
+      ValueBoundsConstraintSet::Variable(size),
+      ValueBoundsConstraintSet::Variable(source, dimension));
+  return succeeded(isFullDimension) && *isFullDimension;
+}
+
 /// The inner tile alignment hint is only used by `linalg.pack` and
 /// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
 /// overload.
@@ -2953,6 +2966,12 @@ FailureOr<TilingResult>
 SoftmaxOp::getTiledImplementation(OpBuilder &builder,
                                   ArrayRef<OpFoldResult> offsets,
                                   ArrayRef<OpFoldResult> sizes) {
+  int64_t reductionDimension = getDimension();
+  if (!isFullSoftmaxDimensionTile(offsets[reductionDimension],
+                                  sizes[reductionDimension], getInput(),
+                                  reductionDimension))
+    return failure();
+
   int64_t rank = getInputOperandRank();
   auto oneAttr = builder.getI64IntegerAttr(1);
   SmallVector<OpFoldResult> strides(rank, oneAttr);
diff --git a/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
new file mode 100644
index 0000000000000..aefb3d0b3de35
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -transform-interpreter -verify-diagnostics
+
+func.func @do_not_tile_softmax_reduction_dimension(
+    %input: tensor<4xf32>) -> tensor<4xf32> {
+  %init = tensor.empty() : tensor<4xf32>
+  // expected-error @below {{failed to tile operation}}
+  // expected-error @below {{failed to generate tiling loops}}
+  %result = linalg.softmax dimension(0)
+      ins(%input : tensor<4xf32>)
+      outs(%init : tensor<4xf32>) -> tensor<4xf32>
+  return %result : tensor<4xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/tile-softmax.mlir b/mlir/test/Dialect/Linalg/tile-softmax.mlir
index 7d201b58a8c3d..df16a6f82abcf 100644
--- a/mlir/test/Dialect/Linalg/tile-softmax.mlir
+++ b/mlir/test/Dialect/Linalg/tile-softmax.mlir
@@ -21,7 +21,7 @@
 // CHECK:               %[[VAL_13:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_11]])
 // CHECK:               %[[VAL_14:.*]] = tensor.extract_slice %[[VAL_0]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
 // CHECK:               %[[VAL_15:.*]] = tensor.extract_slice %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
-// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(1) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
+// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(2) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
 // CHECK:               %[[VAL_17:.*]] = tensor.insert_slice %[[VAL_16]] into %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<2x?x256xf32> into tensor<16x64x256xf32>
 // CHECK:               scf.yield %[[VAL_17]] : tensor<16x64x256xf32>
 // CHECK:             }
@@ -32,7 +32,7 @@
 func.func @softmax(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256xf32> {
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -46,6 +46,40 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
+// Check that an untiled dynamic reduction dimension is recognized as the full
+// reduction dimension.
+
+// CHECK-LABEL: func.func @softmax_dynamic(
+// CHECK:         scf.for
+// CHECK:           tensor.extract_slice
+// CHECK:           linalg.softmax dimension(1)
+func.func @softmax_dynamic(%arg0: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %dim0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %dim1 = tensor.dim %arg0, %c1 : tensor<?x?xf32>
+  %empty = tensor.empty(%dim0, %dim1) : tensor<?x?xf32>
+  %result = linalg.softmax dimension(1)
+      ins(%arg0 : tensor<?x?xf32>)
+      outs(%empty : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %result : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
 // Test the softmax tiling interface with the tile_using_forall transform and
 // check that it composes properly with the fuse transform.
 // This should sink the linalg.generic inside the scf.forall and run that
@@ -73,7 +107,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               linalg.yield %[[VAL_19]] : f32
 // CHECK:             } -> tensor<2x4x256xf32>
 // CHECK:             %[[VAL_20:.*]] = tensor.extract_slice %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x4x256xf32>
-// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(1) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
+// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(2) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
 // CHECK:             scf.forall.in_parallel {
 // CHECK:               tensor.parallel_insert_slice %[[VAL_21]] into %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<2x4x256xf32> into tensor<16x64x256xf32>
 // CHECK:             }
@@ -98,7 +132,7 @@ func.func @softmax_tile_n_fuse(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256
 
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -135,14 +169,14 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               %[[VAL_9:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_8]])
 // CHECK:               %[[VAL_10:.*]] = memref.subview %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
 // CHECK:               %[[VAL_11:.*]] = memref.subview %[[VAL_1]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
-// CHECK:               linalg.softmax dimension(1) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
+// CHECK:               linalg.softmax dimension(2) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
 // CHECK:             }
 // CHECK:           }
 // CHECK:           return
 // CHECK:         }
 func.func @softmax_memref(%arg0: memref<16x64x256xf32>, %arg1: memref<16x64x256xf32>) {
   linalg.softmax
-    dimension(1) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
+    dimension(2) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
   return
 }
 


From llvmlistbot at llvm.org  Sun Jul 26 06:04:21 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 26 Jul 2026 06:04:21 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reject partial softmax
	reduction tiling (PR #212117)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212117@github.com>
Message-ID: <6a6605d5.7d702ed2.2f6c05.a408@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/212117

>From c189ff32f8ec41eb0103cfcf710f4b5fb78ce81d Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 26 Jul 2026 21:20:03 +0900
Subject: [PATCH] [mlir][linalg] Reject partial softmax reduction tiling

Assisted-by: OpenAI Codex
---
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      | 19 ++++++++
 .../Dialect/Linalg/tile-softmax-invalid.mlir  | 25 ++++++++++
 mlir/test/Dialect/Linalg/tile-softmax.mlir    | 46 ++++++++++++++++---
 3 files changed, 84 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir

diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..7a6b9b0d0f96f 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -37,6 +37,7 @@
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
@@ -2940,6 +2941,18 @@ SmallVector<utils::IteratorType> SoftmaxOp::getLoopIteratorTypes() {
   return iteratorTypes;
 }
 
+static bool isFullSoftmaxDimensionTile(OpFoldResult offset, OpFoldResult size,
+                                       Value source, int64_t dimension) {
+  // Softmax cannot be computed independently on parts of its reduction
+  // dimension without combining their maxima and normalization factors.
+  if (!isZeroInteger(offset))
+    return false;
+  FailureOr<bool> isFullDimension = ValueBoundsConstraintSet::areEqual(
+      ValueBoundsConstraintSet::Variable(size),
+      ValueBoundsConstraintSet::Variable(source, dimension));
+  return succeeded(isFullDimension) && *isFullDimension;
+}
+
 /// The inner tile alignment hint is only used by `linalg.pack` and
 /// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
 /// overload.
@@ -2953,6 +2966,12 @@ FailureOr<TilingResult>
 SoftmaxOp::getTiledImplementation(OpBuilder &builder,
                                   ArrayRef<OpFoldResult> offsets,
                                   ArrayRef<OpFoldResult> sizes) {
+  int64_t reductionDimension = getDimension();
+  if (!isFullSoftmaxDimensionTile(offsets[reductionDimension],
+                                  sizes[reductionDimension], getInput(),
+                                  reductionDimension))
+    return failure();
+
   int64_t rank = getInputOperandRank();
   auto oneAttr = builder.getI64IntegerAttr(1);
   SmallVector<OpFoldResult> strides(rank, oneAttr);
diff --git a/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
new file mode 100644
index 0000000000000..aefb3d0b3de35
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -transform-interpreter -verify-diagnostics
+
+func.func @do_not_tile_softmax_reduction_dimension(
+    %input: tensor<4xf32>) -> tensor<4xf32> {
+  %init = tensor.empty() : tensor<4xf32>
+  // expected-error @below {{failed to tile operation}}
+  // expected-error @below {{failed to generate tiling loops}}
+  %result = linalg.softmax dimension(0)
+      ins(%input : tensor<4xf32>)
+      outs(%init : tensor<4xf32>) -> tensor<4xf32>
+  return %result : tensor<4xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/tile-softmax.mlir b/mlir/test/Dialect/Linalg/tile-softmax.mlir
index 7d201b58a8c3d..df16a6f82abcf 100644
--- a/mlir/test/Dialect/Linalg/tile-softmax.mlir
+++ b/mlir/test/Dialect/Linalg/tile-softmax.mlir
@@ -21,7 +21,7 @@
 // CHECK:               %[[VAL_13:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_11]])
 // CHECK:               %[[VAL_14:.*]] = tensor.extract_slice %[[VAL_0]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
 // CHECK:               %[[VAL_15:.*]] = tensor.extract_slice %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
-// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(1) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
+// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(2) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
 // CHECK:               %[[VAL_17:.*]] = tensor.insert_slice %[[VAL_16]] into %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<2x?x256xf32> into tensor<16x64x256xf32>
 // CHECK:               scf.yield %[[VAL_17]] : tensor<16x64x256xf32>
 // CHECK:             }
@@ -32,7 +32,7 @@
 func.func @softmax(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256xf32> {
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -46,6 +46,40 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
+// Check that an untiled dynamic reduction dimension is recognized as the full
+// reduction dimension.
+
+// CHECK-LABEL: func.func @softmax_dynamic(
+// CHECK:         scf.for
+// CHECK:           tensor.extract_slice
+// CHECK:           linalg.softmax dimension(1)
+func.func @softmax_dynamic(%arg0: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %dim0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %dim1 = tensor.dim %arg0, %c1 : tensor<?x?xf32>
+  %empty = tensor.empty(%dim0, %dim1) : tensor<?x?xf32>
+  %result = linalg.softmax dimension(1)
+      ins(%arg0 : tensor<?x?xf32>)
+      outs(%empty : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %result : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
 // Test the softmax tiling interface with the tile_using_forall transform and
 // check that it composes properly with the fuse transform.
 // This should sink the linalg.generic inside the scf.forall and run that
@@ -73,7 +107,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               linalg.yield %[[VAL_19]] : f32
 // CHECK:             } -> tensor<2x4x256xf32>
 // CHECK:             %[[VAL_20:.*]] = tensor.extract_slice %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x4x256xf32>
-// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(1) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
+// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(2) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
 // CHECK:             scf.forall.in_parallel {
 // CHECK:               tensor.parallel_insert_slice %[[VAL_21]] into %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<2x4x256xf32> into tensor<16x64x256xf32>
 // CHECK:             }
@@ -98,7 +132,7 @@ func.func @softmax_tile_n_fuse(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256
 
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -135,14 +169,14 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               %[[VAL_9:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_8]])
 // CHECK:               %[[VAL_10:.*]] = memref.subview %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
 // CHECK:               %[[VAL_11:.*]] = memref.subview %[[VAL_1]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
-// CHECK:               linalg.softmax dimension(1) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
+// CHECK:               linalg.softmax dimension(2) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
 // CHECK:             }
 // CHECK:           }
 // CHECK:           return
 // CHECK:         }
 func.func @softmax_memref(%arg0: memref<16x64x256xf32>, %arg1: memref<16x64x256xf32>) {
   linalg.softmax
-    dimension(1) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
+    dimension(2) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
   return
 }
 


From llvmlistbot at llvm.org  Sun Jul 26 06:04:46 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 26 Jul 2026 06:04:46 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reject partial softmax
	reduction tiling (PR #212117)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212117@github.com>
Message-ID: <6a6605ee.2915925c.20d2c4.982e@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir-linalg

Author: Takayuki Todokoro (takatodo)

<details>
<summary>Changes</summary>

@<!-- -->qcolombet @<!-- -->harsh-nod This fixes a correctness issue in the original
SoftmaxOp TilingInterface implementation.

Partial tiling of the reduction dimension cloned an independent softmax for each slice. 
For example, tiling [0, 0, 0, 0] by 2 produced [0.5, 0.5, 0.5, 0.5] instead of [0.25, 0.25, 0.25, 0.25].

The change requires the reduction-dimension tile to start at offset zero and uses ValueBoundsConstraintSet to prove that its size covers the full reduction dimension. Dynamically shaped cases are accepted when this equality can be proven.

Assisted-by: OpenAI Codex

---
Full diff: https://github.com/llvm/llvm-project/pull/212117.diff


3 Files Affected:

- (modified) mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp (+19) 
- (added) mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir (+25) 
- (modified) mlir/test/Dialect/Linalg/tile-softmax.mlir (+40-6) 


``````````diff
diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..7a6b9b0d0f96f 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -37,6 +37,7 @@
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
@@ -2940,6 +2941,18 @@ SmallVector<utils::IteratorType> SoftmaxOp::getLoopIteratorTypes() {
   return iteratorTypes;
 }
 
+static bool isFullSoftmaxDimensionTile(OpFoldResult offset, OpFoldResult size,
+                                       Value source, int64_t dimension) {
+  // Softmax cannot be computed independently on parts of its reduction
+  // dimension without combining their maxima and normalization factors.
+  if (!isZeroInteger(offset))
+    return false;
+  FailureOr<bool> isFullDimension = ValueBoundsConstraintSet::areEqual(
+      ValueBoundsConstraintSet::Variable(size),
+      ValueBoundsConstraintSet::Variable(source, dimension));
+  return succeeded(isFullDimension) && *isFullDimension;
+}
+
 /// The inner tile alignment hint is only used by `linalg.pack` and
 /// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
 /// overload.
@@ -2953,6 +2966,12 @@ FailureOr<TilingResult>
 SoftmaxOp::getTiledImplementation(OpBuilder &builder,
                                   ArrayRef<OpFoldResult> offsets,
                                   ArrayRef<OpFoldResult> sizes) {
+  int64_t reductionDimension = getDimension();
+  if (!isFullSoftmaxDimensionTile(offsets[reductionDimension],
+                                  sizes[reductionDimension], getInput(),
+                                  reductionDimension))
+    return failure();
+
   int64_t rank = getInputOperandRank();
   auto oneAttr = builder.getI64IntegerAttr(1);
   SmallVector<OpFoldResult> strides(rank, oneAttr);
diff --git a/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
new file mode 100644
index 0000000000000..aefb3d0b3de35
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -transform-interpreter -verify-diagnostics
+
+func.func @do_not_tile_softmax_reduction_dimension(
+    %input: tensor<4xf32>) -> tensor<4xf32> {
+  %init = tensor.empty() : tensor<4xf32>
+  // expected-error @below {{failed to tile operation}}
+  // expected-error @below {{failed to generate tiling loops}}
+  %result = linalg.softmax dimension(0)
+      ins(%input : tensor<4xf32>)
+      outs(%init : tensor<4xf32>) -> tensor<4xf32>
+  return %result : tensor<4xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/tile-softmax.mlir b/mlir/test/Dialect/Linalg/tile-softmax.mlir
index 7d201b58a8c3d..df16a6f82abcf 100644
--- a/mlir/test/Dialect/Linalg/tile-softmax.mlir
+++ b/mlir/test/Dialect/Linalg/tile-softmax.mlir
@@ -21,7 +21,7 @@
 // CHECK:               %[[VAL_13:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_11]])
 // CHECK:               %[[VAL_14:.*]] = tensor.extract_slice %[[VAL_0]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
 // CHECK:               %[[VAL_15:.*]] = tensor.extract_slice %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
-// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(1) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
+// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(2) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
 // CHECK:               %[[VAL_17:.*]] = tensor.insert_slice %[[VAL_16]] into %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<2x?x256xf32> into tensor<16x64x256xf32>
 // CHECK:               scf.yield %[[VAL_17]] : tensor<16x64x256xf32>
 // CHECK:             }
@@ -32,7 +32,7 @@
 func.func @softmax(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256xf32> {
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -46,6 +46,40 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
+// Check that an untiled dynamic reduction dimension is recognized as the full
+// reduction dimension.
+
+// CHECK-LABEL: func.func @softmax_dynamic(
+// CHECK:         scf.for
+// CHECK:           tensor.extract_slice
+// CHECK:           linalg.softmax dimension(1)
+func.func @softmax_dynamic(%arg0: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %dim0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %dim1 = tensor.dim %arg0, %c1 : tensor<?x?xf32>
+  %empty = tensor.empty(%dim0, %dim1) : tensor<?x?xf32>
+  %result = linalg.softmax dimension(1)
+      ins(%arg0 : tensor<?x?xf32>)
+      outs(%empty : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %result : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
 // Test the softmax tiling interface with the tile_using_forall transform and
 // check that it composes properly with the fuse transform.
 // This should sink the linalg.generic inside the scf.forall and run that
@@ -73,7 +107,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               linalg.yield %[[VAL_19]] : f32
 // CHECK:             } -> tensor<2x4x256xf32>
 // CHECK:             %[[VAL_20:.*]] = tensor.extract_slice %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x4x256xf32>
-// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(1) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
+// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(2) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
 // CHECK:             scf.forall.in_parallel {
 // CHECK:               tensor.parallel_insert_slice %[[VAL_21]] into %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<2x4x256xf32> into tensor<16x64x256xf32>
 // CHECK:             }
@@ -98,7 +132,7 @@ func.func @softmax_tile_n_fuse(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256
 
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -135,14 +169,14 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               %[[VAL_9:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_8]])
 // CHECK:               %[[VAL_10:.*]] = memref.subview %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
 // CHECK:               %[[VAL_11:.*]] = memref.subview %[[VAL_1]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
-// CHECK:               linalg.softmax dimension(1) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
+// CHECK:               linalg.softmax dimension(2) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
 // CHECK:             }
 // CHECK:           }
 // CHECK:           return
 // CHECK:         }
 func.func @softmax_memref(%arg0: memref<16x64x256xf32>, %arg1: memref<16x64x256xf32>) {
   linalg.softmax
-    dimension(1) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
+    dimension(2) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
   return
 }
 

``````````

</details>


https://github.com/llvm/llvm-project/pull/212117

From llvmlistbot at llvm.org  Sun Jul 26 06:08:07 2026
From: llvmlistbot at llvm.org (Takayuki Todokoro)
Date: Sun, 26 Jul 2026 06:08:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][linalg] Reject partial softmax
	reduction tiling (PR #212117)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212117@github.com>
Message-ID: <6a6606b7.0975a56b.fbb08.76a3@mx.google.com>


https://github.com/takatodo updated https://github.com/llvm/llvm-project/pull/212117

>From 96dd7f067d65840c2c7659321ad56aa1643cacb2 Mon Sep 17 00:00:00 2001
From: takatodo <takatodo1227 at gmail.com>
Date: Sun, 26 Jul 2026 21:20:03 +0900
Subject: [PATCH] [mlir][linalg] Reject partial softmax reduction tiling

Assisted-by: OpenAI Codex
---
 mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp      | 19 ++++++++
 .../Dialect/Linalg/tile-softmax-invalid.mlir  | 25 ++++++++++
 mlir/test/Dialect/Linalg/tile-softmax.mlir    | 46 ++++++++++++++++---
 3 files changed, 84 insertions(+), 6 deletions(-)
 create mode 100644 mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir

diff --git a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
index 8b2f5064e78bd..7a6b9b0d0f96f 100644
--- a/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
+++ b/mlir/lib/Dialect/Linalg/IR/LinalgOps.cpp
@@ -37,6 +37,7 @@
 #include "mlir/IR/TypeUtilities.h"
 #include "mlir/Interfaces/InferTypeOpInterface.h"
 #include "mlir/Interfaces/SideEffectInterfaces.h"
+#include "mlir/Interfaces/ValueBoundsOpInterface.h"
 
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
@@ -2940,6 +2941,18 @@ SmallVector<utils::IteratorType> SoftmaxOp::getLoopIteratorTypes() {
   return iteratorTypes;
 }
 
+static bool isFullSoftmaxDimensionTile(OpFoldResult offset, OpFoldResult size,
+                                       Value source, int64_t dimension) {
+  // Softmax cannot be computed independently on parts of its reduction
+  // dimension without combining their maxima and normalization factors.
+  if (!isZeroInteger(offset))
+    return false;
+  FailureOr<bool> isFullDimension = ValueBoundsConstraintSet::areEqual(
+      ValueBoundsConstraintSet::Variable(size),
+      ValueBoundsConstraintSet::Variable(source, dimension));
+  return succeeded(isFullDimension) && *isFullDimension;
+}
+
 /// The inner tile alignment hint is only used by `linalg.pack` and
 /// `linalg.unpack` operations. Therefore, this is forwarded to the hint-less
 /// overload.
@@ -2953,6 +2966,12 @@ FailureOr<TilingResult>
 SoftmaxOp::getTiledImplementation(OpBuilder &builder,
                                   ArrayRef<OpFoldResult> offsets,
                                   ArrayRef<OpFoldResult> sizes) {
+  int64_t reductionDimension = getDimension();
+  if (!isFullSoftmaxDimensionTile(offsets[reductionDimension],
+                                  sizes[reductionDimension], getInput(),
+                                  reductionDimension))
+    return failure();
+
   int64_t rank = getInputOperandRank();
   auto oneAttr = builder.getI64IntegerAttr(1);
   SmallVector<OpFoldResult> strides(rank, oneAttr);
diff --git a/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
new file mode 100644
index 0000000000000..aefb3d0b3de35
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/tile-softmax-invalid.mlir
@@ -0,0 +1,25 @@
+// RUN: mlir-opt %s -transform-interpreter -verify-diagnostics
+
+func.func @do_not_tile_softmax_reduction_dimension(
+    %input: tensor<4xf32>) -> tensor<4xf32> {
+  %init = tensor.empty() : tensor<4xf32>
+  // expected-error @below {{failed to tile operation}}
+  // expected-error @below {{failed to generate tiling loops}}
+  %result = linalg.softmax dimension(0)
+      ins(%input : tensor<4xf32>)
+      outs(%init : tensor<4xf32>) -> tensor<4xf32>
+  return %result : tensor<4xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
diff --git a/mlir/test/Dialect/Linalg/tile-softmax.mlir b/mlir/test/Dialect/Linalg/tile-softmax.mlir
index 7d201b58a8c3d..df16a6f82abcf 100644
--- a/mlir/test/Dialect/Linalg/tile-softmax.mlir
+++ b/mlir/test/Dialect/Linalg/tile-softmax.mlir
@@ -21,7 +21,7 @@
 // CHECK:               %[[VAL_13:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_11]])
 // CHECK:               %[[VAL_14:.*]] = tensor.extract_slice %[[VAL_0]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
 // CHECK:               %[[VAL_15:.*]] = tensor.extract_slice %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x?x256xf32>
-// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(1) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
+// CHECK:               %[[VAL_16:.*]] = linalg.softmax dimension(2) ins(%[[VAL_14]] : tensor<2x?x256xf32>) outs(%[[VAL_15]] : tensor<2x?x256xf32>) -> tensor<2x?x256xf32>
 // CHECK:               %[[VAL_17:.*]] = tensor.insert_slice %[[VAL_16]] into %[[VAL_12]]{{\[}}%[[VAL_8]], %[[VAL_11]], 0] [2, %[[VAL_13]], 256] [1, 1, 1] : tensor<2x?x256xf32> into tensor<16x64x256xf32>
 // CHECK:               scf.yield %[[VAL_17]] : tensor<16x64x256xf32>
 // CHECK:             }
@@ -32,7 +32,7 @@
 func.func @softmax(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256xf32> {
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%arg0 : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -46,6 +46,40 @@ module attributes {transform.with_named_sequence} {
 
 // -----
 
+// Check that an untiled dynamic reduction dimension is recognized as the full
+// reduction dimension.
+
+// CHECK-LABEL: func.func @softmax_dynamic(
+// CHECK:         scf.for
+// CHECK:           tensor.extract_slice
+// CHECK:           linalg.softmax dimension(1)
+func.func @softmax_dynamic(%arg0: tensor<?x?xf32>) -> tensor<?x?xf32> {
+  %c0 = arith.constant 0 : index
+  %c1 = arith.constant 1 : index
+  %dim0 = tensor.dim %arg0, %c0 : tensor<?x?xf32>
+  %dim1 = tensor.dim %arg0, %c1 : tensor<?x?xf32>
+  %empty = tensor.empty(%dim0, %dim1) : tensor<?x?xf32>
+  %result = linalg.softmax dimension(1)
+      ins(%arg0 : tensor<?x?xf32>)
+      outs(%empty : tensor<?x?xf32>) -> tensor<?x?xf32>
+  return %result : tensor<?x?xf32>
+}
+
+module attributes {transform.with_named_sequence} {
+  transform.named_sequence @__transform_main(
+      %root: !transform.any_op {transform.readonly}) {
+    %softmax = transform.structured.match
+        ops{["linalg.softmax"]} in %root
+        : (!transform.any_op) -> !transform.any_op
+    %tiled, %loop = transform.structured.tile_using_for %softmax
+        tile_sizes [2]
+        : (!transform.any_op) -> (!transform.any_op, !transform.any_op)
+    transform.yield
+  }
+}
+
+// -----
+
 // Test the softmax tiling interface with the tile_using_forall transform and
 // check that it composes properly with the fuse transform.
 // This should sink the linalg.generic inside the scf.forall and run that
@@ -73,7 +107,7 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               linalg.yield %[[VAL_19]] : f32
 // CHECK:             } -> tensor<2x4x256xf32>
 // CHECK:             %[[VAL_20:.*]] = tensor.extract_slice %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<16x64x256xf32> to tensor<2x4x256xf32>
-// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(1) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
+// CHECK:             %[[VAL_21:.*]] = linalg.softmax dimension(2) ins(%[[VAL_22:.*]] : tensor<2x4x256xf32>) outs(%[[VAL_20]] : tensor<2x4x256xf32>) -> tensor<2x4x256xf32>
 // CHECK:             scf.forall.in_parallel {
 // CHECK:               tensor.parallel_insert_slice %[[VAL_21]] into %[[VAL_7]]{{\[}}%[[VAL_8]], %[[VAL_9]], 0] [2, 4, 256] [1, 1, 1] : tensor<2x4x256xf32> into tensor<16x64x256xf32>
 // CHECK:             }
@@ -98,7 +132,7 @@ func.func @softmax_tile_n_fuse(%arg0: tensor<16x64x256xf32>) -> tensor<16x64x256
 
   %0 = tensor.empty() : tensor<16x64x256xf32>
   %1 = linalg.softmax
-         dimension(1) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
+         dimension(2) ins(%eltwise : tensor<16x64x256xf32>) outs(%0 : tensor<16x64x256xf32>) -> tensor<16x64x256xf32>
   return %1 : tensor<16x64x256xf32>
 }
 
@@ -135,14 +169,14 @@ module attributes {transform.with_named_sequence} {
 // CHECK:               %[[VAL_9:.*]] = affine.min #[[$MIN_MAP]](%[[VAL_8]])
 // CHECK:               %[[VAL_10:.*]] = memref.subview %[[VAL_0]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
 // CHECK:               %[[VAL_11:.*]] = memref.subview %[[VAL_1]]{{\[}}%[[VAL_7]], %[[VAL_8]], 0] [2, %[[VAL_9]], 256] [1, 1, 1] : memref<16x64x256xf32> to memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>
-// CHECK:               linalg.softmax dimension(1) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
+// CHECK:               linalg.softmax dimension(2) ins(%[[VAL_10]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>) outs(%[[VAL_11]] : memref<2x?x256xf32, strided<[16384, 256, 1], offset: ?>>)
 // CHECK:             }
 // CHECK:           }
 // CHECK:           return
 // CHECK:         }
 func.func @softmax_memref(%arg0: memref<16x64x256xf32>, %arg1: memref<16x64x256xf32>) {
   linalg.softmax
-    dimension(1) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
+    dimension(2) ins(%arg0 : memref<16x64x256xf32>) outs(%arg1 : memref<16x64x256xf32>)
   return
 }
 


From llvmlistbot at llvm.org  Sun Jul 26 06:47:14 2026
From: llvmlistbot at llvm.org (Felix Ye)
Date: Sun, 26 Jul 2026 06:47:14 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests
	(PR #212118)
Message-ID: <llvm.org/llvm/llvm-project/pull/212118@github.com>


https://github.com/yanjs created https://github.com/llvm/llvm-project/pull/212118

Add missing `| FileCheck %s` to `RUN` lines so `CHECK` lines can be verified. Also correct the expected result from `30` to `36`. The result should be `6*6=36`.

>From 8586bddd0ced5ccdc5ccfb38f08afbeb4d09f920 Mon Sep 17 00:00:00 2001
From: Felix Ye <felixyjs at gmail.com>
Date: Sun, 26 Jul 2026 21:10:10 +0800
Subject: [PATCH] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests

Add missing `| FileCheck %s` to `RUN` lines so `CHECK` lines can be verified.
Also correct the expected result from `30` to `36`.

Signed-off-by: Felix Ye <felixyjs at gmail.com>
---
 mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir | 4 ++--
 mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir b/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
index 37ad2bd808a53..8e2f82cf15c5c 100644
--- a/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
+++ b/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
@@ -1,4 +1,4 @@
-// RUN: toyc-ch6 %s -emit=llvm -opt
+// RUN: toyc-ch6 %s -emit=llvm -opt 2>&1 | FileCheck %s
 
 toy.func @main() {
   %0 = toy.constant dense<[[1.000000e+00, 2.000000e+00, 3.000000e+00], [4.000000e+00, 5.000000e+00, 6.000000e+00]]> : tensor<2x3xf64>
@@ -20,4 +20,4 @@ toy.func @main() {
 // CHECK: @printf
 // CHECK-SAME: 9.000000e+00
 // CHECK: @printf
-// CHECK-SAME: 3.000000e+01
+// CHECK-SAME: 3.600000e+01
diff --git a/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir b/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
index fc4c8d5019355..b4d25393b4ef0 100644
--- a/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
+++ b/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
@@ -1,4 +1,4 @@
-// RUN: toyc-ch7 %s -emit=llvm -opt
+// RUN: toyc-ch7 %s -emit=llvm -opt 2>&1 | FileCheck %s
 
 toy.func @main() {
   %0 = toy.constant dense<[[1.000000e+00, 2.000000e+00, 3.000000e+00], [4.000000e+00, 5.000000e+00, 6.000000e+00]]> : tensor<2x3xf64>
@@ -20,4 +20,4 @@ toy.func @main() {
 // CHECK: @printf
 // CHECK-SAME: 9.000000e+00
 // CHECK: @printf
-// CHECK-SAME: 3.000000e+01
+// CHECK-SAME: 3.600000e+01


From llvmlistbot at llvm.org  Sun Jul 26 06:47:31 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 26 Jul 2026 06:47:31 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests
	(PR #212118)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212118@github.com>
Message-ID: <6a660ff3.6c5720b3.2e9b28.fc4d@mx.google.com>


github-actions[bot] wrote:

<!--LLVM NEW CONTRIBUTOR COMMENT-->

Hello @yanjs :wave:

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

* All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
* The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
* Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

---

### Frequently asked questions

**How do I add reviewers?**

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.

**What if there are no comments?**

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

**Are any special GitHub settings required to contribute to LLVM?**

We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.

---

If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).

Thank you,
The LLVM Community

https://github.com/llvm/llvm-project/pull/212118

From llvmlistbot at llvm.org  Sun Jul 26 06:48:11 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 26 Jul 2026 06:48:11 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests
	(PR #212118)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212118@github.com>
Message-ID: <6a66101b.9edb5c9d.21e140.db19@mx.google.com>


llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-mlir

Author: Felix Ye (yanjs)

<details>
<summary>Changes</summary>

Add missing `| FileCheck %s` to `RUN` lines so `CHECK` lines can be verified. Also correct the expected result from `30` to `36`. The result should be `6*6=36`.

---
Full diff: https://github.com/llvm/llvm-project/pull/212118.diff


2 Files Affected:

- (modified) mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir (+2-2) 
- (modified) mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir (+2-2) 


``````````diff
diff --git a/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir b/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
index 37ad2bd808a53..8e2f82cf15c5c 100644
--- a/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
+++ b/mlir/test/Examples/Toy/Ch6/llvm-lowering.mlir
@@ -1,4 +1,4 @@
-// RUN: toyc-ch6 %s -emit=llvm -opt
+// RUN: toyc-ch6 %s -emit=llvm -opt 2>&1 | FileCheck %s
 
 toy.func @main() {
   %0 = toy.constant dense<[[1.000000e+00, 2.000000e+00, 3.000000e+00], [4.000000e+00, 5.000000e+00, 6.000000e+00]]> : tensor<2x3xf64>
@@ -20,4 +20,4 @@ toy.func @main() {
 // CHECK: @printf
 // CHECK-SAME: 9.000000e+00
 // CHECK: @printf
-// CHECK-SAME: 3.000000e+01
+// CHECK-SAME: 3.600000e+01
diff --git a/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir b/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
index fc4c8d5019355..b4d25393b4ef0 100644
--- a/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
+++ b/mlir/test/Examples/Toy/Ch7/llvm-lowering.mlir
@@ -1,4 +1,4 @@
-// RUN: toyc-ch7 %s -emit=llvm -opt
+// RUN: toyc-ch7 %s -emit=llvm -opt 2>&1 | FileCheck %s
 
 toy.func @main() {
   %0 = toy.constant dense<[[1.000000e+00, 2.000000e+00, 3.000000e+00], [4.000000e+00, 5.000000e+00, 6.000000e+00]]> : tensor<2x3xf64>
@@ -20,4 +20,4 @@ toy.func @main() {
 // CHECK: @printf
 // CHECK-SAME: 9.000000e+00
 // CHECK: @printf
-// CHECK-SAME: 3.000000e+01
+// CHECK-SAME: 3.600000e+01

``````````

</details>


https://github.com/llvm/llvm-project/pull/212118

From llvmlistbot at llvm.org  Sun Jul 26 06:53:52 2026
From: llvmlistbot at llvm.org (Felix Ye)
Date: Sun, 26 Jul 2026 06:53:52 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests
	(PR #212118)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212118@github.com>
Message-ID: <6a661170.1728eaf3.12b96d.8e9a@mx.google.com>


yanjs wrote:

> Hello @yanjs 👋
> 
> Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
> 
> * All contributions to LLVM must follow our [LLVM AI Tool Use Policy](https://llvm.org/docs/AIToolPolicy.html). In particular, if you used AI while working on this PR, remember to add a note to the PR description.
> * The [LLVM Code-Review Policy and Practices](https://llvm.org/docs/CodeReview.html) document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
> * Our [LLVM Developer Policy](https://llvm.org/docs/DeveloperPolicy.html) describes our expectations for code quality, commit summaries and contains notes on our CI system.
> 
> Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.
> 
> ### Frequently asked questions
> **How do I add reviewers?**
> 
> This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.
> 
> You can also add reviewers manually using the **Reviewers** section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using `@` followed by their GitHub username.
> 
> **What if there are no comments?**
> 
> If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.
> 
> **Are any special GitHub settings required to contribute to LLVM?**
> 
> We only require contributors to have a public email address associated with their GitHub commits, see this [section](https://llvm.org/docs/DeveloperPolicy.html#email-addresses) of LLVM Developer Policy for details.
> 
> If you have questions, feel free to leave a comment on this PR, or ask on [LLVM Discord](https://discord.com/invite/xS7Z362) or [LLVM Discourse](https://discourse.llvm.org/).
> 
> Thank you, The LLVM Community

I have read these policies. No AI tools were used in the code.

https://github.com/llvm/llvm-project/pull/212118

From llvmlistbot at llvm.org  Sun Jul 26 07:19:16 2026
From: llvmlistbot at llvm.org (Felix Ye)
Date: Sun, 26 Jul 2026 07:19:16 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][toy] Fix Ch6/Ch7 llvm-lowering tests
	(PR #212118)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212118@github.com>
Message-ID: <6a661764.666420a4.3252ee.5915@mx.google.com>


https://github.com/yanjs edited https://github.com/llvm/llvm-project/pull/212118

From llvmlistbot at llvm.org  Sun Jul 26 09:07:07 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:07:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a6630ab.7270d2fd.2df01a.b29d@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212074

>From e5e0e140224c85bfd2163ebf25f2e32d41323db0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:43 -0700
Subject: [PATCH] [mlir][arith] Fold trivial integer division and remainder
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add value-preserving folds mirroring LLVM's InstructionSimplify for the
integer division and remainder ops:

  divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
  divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
  remui/remsi(0, x) -> 0
  remui/remsi(x, x) -> 0

The self and zero-dividend cases are valid refinements because division
or remainder by zero is undefined behaviour; no overflow flags are
required. The folds return a scalar or splat constant and bail out on
shaped types with a dynamic shape.

Folding `x / 0` and `x % 0` to poison is left as a TODO: it would make the
arith dialect depend on the ub dialect to materialize `ub.poison`.

LLVM performs the same simplifications: https://godbolt.org/z/dzTjYjs4e

Verified with Alive2:

| Fold | udiv | sdiv | urem | srem |
| :--- | :---: | :---: | :---: | :---: |
| `x / x -> 1`, `x % x -> 0` | [✓](https://alive2.llvm.org/ce/z/YjZCEA) | [✓](https://alive2.llvm.org/ce/z/BSWQrm) | [✓](https://alive2.llvm.org/ce/z/gx5sRH) | [✓](https://alive2.llvm.org/ce/z/VUYkr4) |
| `0 / x -> 0`, `0 % x -> 0` | [✓](https://alive2.llvm.org/ce/z/keqeCZ) | [✓](https://alive2.llvm.org/ce/z/jyf5wm) | [✓](https://alive2.llvm.org/ce/z/fZa6ND) | [✓](https://alive2.llvm.org/ce/z/fo6ss8) |

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        |  95 +++++++++++++-
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir |  24 ++--
 .../arith-to-spirv-unsupported.mlir           |   4 +-
 .../ArithToSPIRV/arith-to-spirv.mlir          |  22 ++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 119 ++++++++++++++++++
 5 files changed, 237 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..e260ca02509de 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -791,10 +804,22 @@ static Value foldDivMul(Value lhs, Value rhs,
 }
 
 OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divui (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -831,10 +856,22 @@ Speculation::Speculatability arith::DivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divsi (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -887,10 +924,22 @@ static APInt signedCeilNonnegInputs(const APInt &a, const APInt &b,
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivui (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -917,10 +966,22 @@ Speculation::Speculatability arith::CeilDivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -986,10 +1047,22 @@ Speculation::Speculatability arith::CeilDivSIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: floordivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // floordivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1009,9 +1082,18 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remui (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1036,9 +1118,18 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remsi (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d22677fde774a 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -11,14 +11,14 @@ func.func @vector_ops(%arg0: vector<4xf32>, %arg1: vector<4xi1>, %arg2: vector<4
   %0 = arith.constant dense<42.> : vector<4xf32>
 // CHECK-NEXT:  %1 = llvm.fadd %arg0, %0 : vector<4xf32>
   %1 = arith.addf %arg0, %0 : vector<4xf32>
-// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg2 : vector<4xi64>
-  %3 = arith.divsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg2 : vector<4xi64>
-  %4 = arith.divui %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg2 : vector<4xi64>
-  %5 = arith.remsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg2 : vector<4xi64>
-  %6 = arith.remui %arg2, %arg2 : vector<4xi64>
+// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg3 : vector<4xi64>
+  %3 = arith.divsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg3 : vector<4xi64>
+  %4 = arith.divui %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg3 : vector<4xi64>
+  %5 = arith.remsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg3 : vector<4xi64>
+  %6 = arith.remui %arg2, %arg3 : vector<4xi64>
 // CHECK-NEXT:  %6 = llvm.fdiv %arg0, %0 : vector<4xf32>
   %7 = arith.divf %arg0, %0 : vector<4xf32>
 // CHECK-NEXT:  %7 = llvm.frem %arg0, %0 : vector<4xf32>
@@ -780,16 +780,16 @@ func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
 }
 
 // CHECK-LABEL: @ceildivui
-// CHECK-SAME: %[[ARG0:.*]]: i32) -> i32
-func.func @ceildivui(%arg0 : i32) -> i32 {
+// CHECK-SAME: %[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32) -> i32
+func.func @ceildivui(%arg0 : i32, %arg1 : i32) -> i32 {
 // CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
 // CHECK: %[[CMP0:.*]] = llvm.icmp "eq" %[[ARG0]], %[[CST0]] : i32
 // CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
 // CHECK: %[[SUB0:.*]] = llvm.sub %[[ARG0]], %[[CST1]] : i32
-// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG0]] : i32
+// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG1]] : i32
 // CHECK: %[[ADD0:.*]] = llvm.add %[[DIV0]], %[[CST1]] : i32
 // CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[CST0]], %[[ADD0]] : i1, i32
-  %0 = arith.ceildivui %arg0, %arg0 : i32
+  %0 = arith.ceildivui %arg0, %arg1 : i32
   return %0: i32
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
index 92b587d5ed1e4..9ecebb502630c 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
@@ -53,10 +53,10 @@ func.func @unsupported_2x2elem_vector(%arg0: vector<2x2xi32>) {
 
 // -----
 
-func.func @int_vector4_invalid(%arg0: vector<2xi16>) {
+func.func @int_vector4_invalid(%arg0: vector<2xi16>, %arg1: vector<2xi16>) {
   // expected-error @+2 {{failed to legalize operation 'arith.divui'}}
   // expected-error @+1 {{bitwidth emulation is not implemented yet on unsigned op}}
-  %0 = arith.divui %arg0, %arg0: vector<2xi16>
+  %0 = arith.divui %arg0, %arg1: vector<2xi16>
   return
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index 5c9966055b540..4cef06992727b 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -198,18 +198,18 @@ func.func @float32_binary_scalar(%lhs: f32, %rhs: f32) {
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
 // CHECK-LABEL: @index_vector
-func.func @index_vector(%arg0: vector<4xindex>) {
+func.func @index_vector(%arg0: vector<4xindex>, %arg1: vector<4xindex>) {
   // CHECK: spirv.UMod %{{.*}}, %{{.*}}: vector<4xi32>
-  %0 = arith.remui %arg0, %arg0: vector<4xindex>
+  %0 = arith.remui %arg0, %arg1: vector<4xindex>
   return
 }
 
@@ -1522,11 +1522,11 @@ func.func @float32_maxnumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>)
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
@@ -1570,11 +1570,11 @@ module attributes {
 } {
 
 // CHECK-LABEL: @int_vector23
-func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<3xi16>) {
+func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<3xi16>, %arg3: vector<3xi16>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi32>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<3xi32>
-  %1 = arith.divsi %arg1, %arg1: vector<3xi16>
+  %1 = arith.divsi %arg2, %arg3: vector<3xi16>
   return
 }
 
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true


From llvmlistbot at llvm.org  Sun Jul 26 09:08:07 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:08:07 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a6630e7.666420a4.3252ee.5f96@mx.google.com>


https://github.com/victor-eds edited https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sun Jul 26 09:08:43 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:08:43 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a66310b.abe55a58.5923c.25d6@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212074

>From 34f5d2e0a6069c78a76fd8d4ec1c173947a83dd1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sat, 25 Jul 2026 16:06:43 -0700
Subject: [PATCH] [mlir][arith] Fold trivial integer division and remainder
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add value-preserving folds mirroring LLVM's InstructionSimplify for the
integer division and remainder ops:

  divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
  divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
  remui/remsi(0, x) -> 0
  remui/remsi(x, x) -> 0

The self and zero-dividend cases are valid refinements because division
or remainder by zero is undefined behaviour; no overflow flags are
required. The folds return a scalar or splat constant and bail out on
shaped types with a dynamic shape.

Folding `x / 0` and `x % 0` to poison is left as a TODO: it would make the
arith dialect depend on the ub dialect to materialize `ub.poison`.

LLVM performs the same simplifications: https://godbolt.org/z/dzTjYjs4e

Verified with Alive2:

| Fold | udiv | sdiv | urem | srem |
| :--- | :---: | :---: | :---: | :---: |
| `x / x -> 1`, `x % x -> 0` | [✓](https://alive2.llvm.org/ce/z/YjZCEA) | [✓](https://alive2.llvm.org/ce/z/BSWQrm) | [✓](https://alive2.llvm.org/ce/z/gx5sRH) | [✓](https://alive2.llvm.org/ce/z/VUYkr4) |
| `0 / x -> 0`, `0 % x -> 0` | [✓](https://alive2.llvm.org/ce/z/keqeCZ) | [✓](https://alive2.llvm.org/ce/z/jyf5wm) | [✓](https://alive2.llvm.org/ce/z/fZa6ND) | [✓](https://alive2.llvm.org/ce/z/fo6ss8) |

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        |  95 +++++++++++++-
 .../Conversion/ArithToLLVM/arith-to-llvm.mlir |  24 ++--
 .../arith-to-spirv-unsupported.mlir           |   4 +-
 .../ArithToSPIRV/arith-to-spirv.mlir          |  22 ++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 119 ++++++++++++++++++
 5 files changed, 237 insertions(+), 27 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..e260ca02509de 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -791,10 +804,22 @@ static Value foldDivMul(Value lhs, Value rhs,
 }
 
 OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divui (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -831,10 +856,22 @@ Speculation::Speculatability arith::DivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divsi (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -887,10 +924,22 @@ static APInt signedCeilNonnegInputs(const APInt &a, const APInt &b,
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivui (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -917,10 +966,22 @@ Speculation::Speculatability arith::CeilDivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -986,10 +1047,22 @@ Speculation::Speculatability arith::CeilDivSIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: floordivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // floordivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1009,9 +1082,18 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remui (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1036,9 +1118,18 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remsi (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
diff --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d22677fde774a 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -11,14 +11,14 @@ func.func @vector_ops(%arg0: vector<4xf32>, %arg1: vector<4xi1>, %arg2: vector<4
   %0 = arith.constant dense<42.> : vector<4xf32>
 // CHECK-NEXT:  %1 = llvm.fadd %arg0, %0 : vector<4xf32>
   %1 = arith.addf %arg0, %0 : vector<4xf32>
-// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg2 : vector<4xi64>
-  %3 = arith.divsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg2 : vector<4xi64>
-  %4 = arith.divui %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg2 : vector<4xi64>
-  %5 = arith.remsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg2 : vector<4xi64>
-  %6 = arith.remui %arg2, %arg2 : vector<4xi64>
+// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg3 : vector<4xi64>
+  %3 = arith.divsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg3 : vector<4xi64>
+  %4 = arith.divui %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg3 : vector<4xi64>
+  %5 = arith.remsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg3 : vector<4xi64>
+  %6 = arith.remui %arg2, %arg3 : vector<4xi64>
 // CHECK-NEXT:  %6 = llvm.fdiv %arg0, %0 : vector<4xf32>
   %7 = arith.divf %arg0, %0 : vector<4xf32>
 // CHECK-NEXT:  %7 = llvm.frem %arg0, %0 : vector<4xf32>
@@ -780,16 +780,16 @@ func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
 }
 
 // CHECK-LABEL: @ceildivui
-// CHECK-SAME: %[[ARG0:.*]]: i32) -> i32
-func.func @ceildivui(%arg0 : i32) -> i32 {
+// CHECK-SAME: %[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32) -> i32
+func.func @ceildivui(%arg0 : i32, %arg1 : i32) -> i32 {
 // CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
 // CHECK: %[[CMP0:.*]] = llvm.icmp "eq" %[[ARG0]], %[[CST0]] : i32
 // CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
 // CHECK: %[[SUB0:.*]] = llvm.sub %[[ARG0]], %[[CST1]] : i32
-// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG0]] : i32
+// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG1]] : i32
 // CHECK: %[[ADD0:.*]] = llvm.add %[[DIV0]], %[[CST1]] : i32
 // CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[CST0]], %[[ADD0]] : i1, i32
-  %0 = arith.ceildivui %arg0, %arg0 : i32
+  %0 = arith.ceildivui %arg0, %arg1 : i32
   return %0: i32
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
index 92b587d5ed1e4..9ecebb502630c 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
@@ -53,10 +53,10 @@ func.func @unsupported_2x2elem_vector(%arg0: vector<2x2xi32>) {
 
 // -----
 
-func.func @int_vector4_invalid(%arg0: vector<2xi16>) {
+func.func @int_vector4_invalid(%arg0: vector<2xi16>, %arg1: vector<2xi16>) {
   // expected-error @+2 {{failed to legalize operation 'arith.divui'}}
   // expected-error @+1 {{bitwidth emulation is not implemented yet on unsigned op}}
-  %0 = arith.divui %arg0, %arg0: vector<2xi16>
+  %0 = arith.divui %arg0, %arg1: vector<2xi16>
   return
 }
 
diff --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index 5c9966055b540..4cef06992727b 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -198,18 +198,18 @@ func.func @float32_binary_scalar(%lhs: f32, %rhs: f32) {
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
 // CHECK-LABEL: @index_vector
-func.func @index_vector(%arg0: vector<4xindex>) {
+func.func @index_vector(%arg0: vector<4xindex>, %arg1: vector<4xindex>) {
   // CHECK: spirv.UMod %{{.*}}, %{{.*}}: vector<4xi32>
-  %0 = arith.remui %arg0, %arg0: vector<4xindex>
+  %0 = arith.remui %arg0, %arg1: vector<4xindex>
   return
 }
 
@@ -1522,11 +1522,11 @@ func.func @float32_maxnumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>)
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
@@ -1570,11 +1570,11 @@ module attributes {
 } {
 
 // CHECK-LABEL: @int_vector23
-func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<3xi16>) {
+func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<3xi16>, %arg3: vector<3xi16>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi32>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<3xi32>
-  %1 = arith.divsi %arg1, %arg1: vector<3xi16>
+  %1 = arith.divsi %arg2, %arg3: vector<3xi16>
   return
 }
 
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true


From llvmlistbot at llvm.org  Sun Jul 26 09:09:09 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:09:09 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a663125.8e75e0cb.2de827.98cd@mx.google.com>


https://github.com/victor-eds auto_merge_enabled https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sun Jul 26 09:12:29 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:12:29 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fix folds crashing on
 dynamic-shaped tensors (PR #212072)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212072@github.com>
Message-ID: <6a6631ed.af4834b4.1d4518.70fc@mx.google.com>


https://github.com/victor-eds updated https://github.com/llvm/llvm-project/pull/212072

>From c7c2738cf82566478c10742c1aeb009c394ee6fb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?V=C3=ADctor=20P=C3=A9rez=20Carrasco?=
 <victor.pc.upm at gmail.com>
Date: Sun, 26 Jul 2026 09:11:22 -0700
Subject: [PATCH] [mlir][arith] Fix folds crashing on dynamic-shaped tensors
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Builder::getZeroAttr asserts when asked to build a constant of a
ranked tensor with a dynamic shape (DenseElementsAttr requires a static
shape). Three self-identity folds/patterns reached it without a guard and
aborted on dynamic-shaped operands:

  arith.xori %x, %x : tensor<?xi32>
  arith.subui_extended %x, %x : tensor<?xi32>, tensor<?xi1>
  arith.subi (arith.subi %a, %b), %a : tensor<?xi32>   (SubISubILHSRHSLHS)

Guard each with a static-shape check (as arith.subi(x,x) already does) so
they leave the op unfolded instead of crashing. The TableGen pattern gets
a new HasStaticOrScalarShape constraint.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>
---
 .../Dialect/Arith/IR/ArithCanonicalization.td | 12 ++++++-
 mlir/lib/Dialect/Arith/IR/ArithOps.cpp        | 14 ++++++--
 mlir/test/Dialect/Arith/canonicalize.mlir     | 34 +++++++++++++++++++
 3 files changed, 57 insertions(+), 3 deletions(-)

diff --git a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
index bfb8e5cc34218..2615bcca23e35 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
+++ b/mlir/lib/Dialect/Arith/IR/ArithCanonicalization.td
@@ -15,6 +15,15 @@ include "mlir/Dialect/Arith/IR/ArithOps.td"
 // Create zero attribute of type matching the argument's type.
 def GetZeroAttr : NativeCodeCall<"$_builder.getZeroAttr($0.getType())">;
 
+// A value whose type is a scalar or has a static shape, i.e. one for which a
+// splat constant can be built. Guards patterns that materialize a constant via
+// GetZeroAttr, which asserts on a dynamically-shaped type.
+def IsScalarOrHasStaticShape :
+    Constraint<CPred<"!::llvm::isa<::mlir::ShapedType>($0.getType()) || "
+                     "::llvm::cast<::mlir::ShapedType>($0.getType())"
+                     ".hasStaticShape()">,
+               "type is a scalar or has a static shape">;
+
 // Add two integer attributes and create a new one with the result.
 def AddIntAttrs : NativeCodeCall<"addIntegerAttrs($_builder, $0, $1, $2)">;
 
@@ -193,7 +202,8 @@ def SubILHSSubConstantLHS :
 def SubISubILHSRHSLHS :
     Pat<(Arith_SubIOp:$res (Arith_SubIOp $x, $y, $ovf1), $x, $ovf2),
         (Arith_SubIOp (Arith_ConstantOp (GetZeroAttr $y)), $y,
-            (MergeOverflow $ovf1, $ovf2))>;
+            (MergeOverflow $ovf1, $ovf2)),
+        [(IsScalarOrHasStaticShape $y)]>;
 
 //===----------------------------------------------------------------------===//
 // MulSIExtendedOp
diff --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..7f138b30eb8dd 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -539,6 +539,11 @@ arith::SubUIExtendedOp::fold(FoldAdaptor adaptor,
 
   // subui_extended(x, x) -> 0, false
   if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getDiff().getType());
+    if (shapedType && !shapedType.hasStaticShape())
+      return failure();
     Builder builder(getContext());
     auto zeroDiff = builder.getZeroAttr(getDiff().getType());
     auto falseValue = builder.getZeroAttr(borrowTy);
@@ -1153,8 +1158,13 @@ OpFoldResult arith::XOrIOp::fold(FoldAdaptor adaptor) {
   if (matchPattern(adaptor.getRhs(), m_Zero()))
     return getLhs();
   /// xor(x, x) -> 0
-  if (getLhs() == getRhs())
-    return Builder(getContext()).getZeroAttr(getType());
+  if (getLhs() == getRhs()) {
+    // A dynamically-shaped result cannot be a constant; bail before
+    // getZeroAttr, which would assert on a non-static shape.
+    auto shapedType = dyn_cast<ShapedType>(getType());
+    if (!shapedType || shapedType.hasStaticShape())
+      return Builder(getContext()).getZeroAttr(getType());
+  }
   /// xor(xor(x, a), a) -> x
   /// xor(xor(a, x), a) -> x
   if (arith::XOrIOp prev = getLhs().getDefiningOp<arith::XOrIOp>()) {
diff --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..d48939bd3ab32 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -4263,3 +4263,37 @@ func.func @convertf_fold_f8() -> f8E5M2 {
   return %result : f8E5M2
 }
 
+
+// -----
+
+// Self-identity folds and patterns must not build a constant of a dynamic
+// shape (which would assert); they must leave the op unfolded.
+// CHECK-LABEL: func @xori_self_dynamic
+//       CHECK:   arith.xori
+func.func @xori_self_dynamic(%arg0 : tensor<?xi32>) -> tensor<?xi32> {
+  %0 = arith.xori %arg0, %arg0 : tensor<?xi32>
+  return %0 : tensor<?xi32>
+}
+
+// -----
+
+// CHECK-LABEL: func @subui_extended_self_dynamic
+//       CHECK:   arith.subui_extended
+func.func @subui_extended_self_dynamic(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi1>) {
+  %low, %bo = arith.subui_extended %arg0, %arg0
+      : tensor<?xi32>, tensor<?xi1>
+  return %low, %bo : tensor<?xi32>, tensor<?xi1>
+}
+
+// -----
+
+// CHECK-LABEL: func @subi_subi_lhs_rhs_lhs_dynamic
+//       CHECK:   arith.subi
+//       CHECK:   arith.subi
+func.func @subi_subi_lhs_rhs_lhs_dynamic(%a : tensor<?xi32>, %b : tensor<?xi32>)
+    -> tensor<?xi32> {
+  %0 = arith.subi %a, %b : tensor<?xi32>
+  %1 = arith.subi %0, %a : tensor<?xi32>
+  return %1 : tensor<?xi32>
+}


From llvmlistbot at llvm.org  Sun Jul 26 09:13:06 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:13:06 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fix folds crashing on
 dynamic-shaped tensors (PR #212072)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212072@github.com>
Message-ID: <6a663212.7270d2fd.2df01a.b926@mx.google.com>


https://github.com/victor-eds edited https://github.com/llvm/llvm-project/pull/212072

From llvmlistbot at llvm.org  Sun Jul 26 09:25:40 2026
From: llvmlistbot at llvm.org (llvmlistbot at llvm.org)
Date: Sun, 26 Jul 2026 09:25:40 -0700 (PDT)
Subject: [Mlir-commits] [mlir] bdd8c64 - [mlir][arith] Fold trivial integer
 division and remainder (#212074)
Message-ID: <6a663504.f3151ff8.2f3401.e132@mx.google.com>


Author: Victor Perez
Date: 2026-07-26T16:25:34Z
New Revision: bdd8c64a2ae89cc9d860bd3b37b1bb782ccde077

URL: https://github.com/llvm/llvm-project/commit/bdd8c64a2ae89cc9d860bd3b37b1bb782ccde077
DIFF: https://github.com/llvm/llvm-project/commit/bdd8c64a2ae89cc9d860bd3b37b1bb782ccde077.diff

LOG: [mlir][arith] Fold trivial integer division and remainder (#212074)

Add value-preserving folds mirroring LLVM's InstructionSimplify for the
integer division and remainder ops:

```
divui/divsi/ceildivui/ceildivsi/floordivsi(0, x) -> 0
divui/divsi/ceildivui/ceildivsi/floordivsi(x, x) -> 1
remui/remsi(0, x) -> 0
remui/remsi(x, x) -> 0
```

The self and zero-dividend cases are valid refinements because division
or remainder by zero is undefined behaviour; no overflow flags are
required. The folds return a scalar or splat constant and bail out on
shaped types with a dynamic shape.

Folding `x / 0` and `x % 0` to poison is left as a TODO: it would make
the arith dialect depend on the ub dialect to materialize `ub.poison`.

LLVM performs the same simplifications: https://godbolt.org/z/dzTjYjs4e

Verified with Alive2:

| Fold | udiv | sdiv | urem | srem |
| :--- | :---: | :---: | :---: | :---: |
| `x / x -> 1`, `x % x -> 0` | [✓](https://alive2.llvm.org/ce/z/YjZCEA)
| [✓](https://alive2.llvm.org/ce/z/BSWQrm) |
[✓](https://alive2.llvm.org/ce/z/gx5sRH) |
[✓](https://alive2.llvm.org/ce/z/VUYkr4) |
| `0 / x -> 0`, `0 % x -> 0` | [✓](https://alive2.llvm.org/ce/z/keqeCZ)
| [✓](https://alive2.llvm.org/ce/z/jyf5wm) |
[✓](https://alive2.llvm.org/ce/z/fZa6ND) |
[✓](https://alive2.llvm.org/ce/z/fo6ss8) |

----

Code generated by Claude Code.

Signed-off-by: Víctor Pérez Carrasco <victor.pc.upm at gmail.com>

Added: 
    

Modified: 
    mlir/lib/Dialect/Arith/IR/ArithOps.cpp
    mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
    mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
    mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
    mlir/test/Dialect/Arith/canonicalize.mlir

Removed: 
    


################################################################################
diff  --git a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
index 674577e5764d8..e260ca02509de 100644
--- a/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
+++ b/mlir/lib/Dialect/Arith/IR/ArithOps.cpp
@@ -178,6 +178,19 @@ static Attribute getBoolAttribute(Type type, bool value) {
   return DenseElementsAttr::get(shapedType, boolAttr);
 }
 
+/// Return a scalar or splat integer attribute of `type` (an integer/index type
+/// or a shaped type thereof) holding `value`. Returns a null attribute for
+/// shaped types with a dynamic shape, so callers can bail out of folding.
+static Attribute getIntegerAttrOfType(Type type, int64_t value) {
+  auto scalarAttr = IntegerAttr::get(getElementTypeOrSelf(type), value);
+  ShapedType shapedType = dyn_cast<ShapedType>(type);
+  if (!shapedType)
+    return scalarAttr;
+  if (!shapedType.hasStaticShape())
+    return {};
+  return DenseElementsAttr::get(shapedType, scalarAttr);
+}
+
 //===----------------------------------------------------------------------===//
 // TableGen'd canonicalization patterns
 //===----------------------------------------------------------------------===//
@@ -791,10 +804,22 @@ static Value foldDivMul(Value lhs, Value rhs,
 }
 
 OpFoldResult arith::DivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divui (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nuw))
     return val;
@@ -831,10 +856,22 @@ Speculation::Speculatability arith::DivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::DivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: divsi (x, 0) -> poison. Division by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // divsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // divsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // divsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // (a * b) / b -> a
   if (Value val = foldDivMul(getLhs(), getRhs(), IntegerOverflowFlags::nsw))
     return val;
@@ -887,10 +924,22 @@ static APInt signedCeilNonnegInputs(const APInt &a, const APInt &b,
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivui (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivui (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivui (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivui (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   bool overflowOrDiv0 = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
       adaptor.getOperands(), [&](APInt a, const APInt &b) {
@@ -917,10 +966,22 @@ Speculation::Speculatability arith::CeilDivUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::CeilDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: ceildivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // ceildivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // ceildivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // ceildivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   // TODO: This hook won't fold operations where a = MININT, because
   // negating MININT overflows. This can be improved.
@@ -986,10 +1047,22 @@ Speculation::Speculatability arith::CeilDivSIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: floordivsi (x, 0) -> poison. Division by zero is undefined behaviour
+  // and could fold to poison, but that would make the arith dialect depend on
+  // the ub dialect to materialize ub.poison; left out for now.
+
   // floordivsi (x, 1) -> x.
   if (matchPattern(adaptor.getRhs(), m_One()))
     return getLhs();
 
+  // floordivsi (0, x) -> 0. Division by zero is UB, so refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()))
+    return getLhs();
+
+  // floordivsi (x, x) -> 1.
+  if (getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 1);
+
   // Don't fold if it would overflow or if it requires a division by zero.
   bool overflowOrDiv = false;
   auto result = constFoldBinaryOp<IntegerAttr>(
@@ -1009,9 +1082,18 @@ OpFoldResult arith::FloorDivSIOp::fold(FoldAdaptor adaptor) {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemUIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remui (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remui (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remui (0, x) -> 0 and remui (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;
@@ -1036,9 +1118,18 @@ Speculation::Speculatability arith::RemUIOp::getSpeculatability() {
 //===----------------------------------------------------------------------===//
 
 OpFoldResult arith::RemSIOp::fold(FoldAdaptor adaptor) {
+  // TODO: remsi (x, 0) -> poison. Remainder by zero is undefined behaviour and
+  // could fold to poison, but that would make the arith dialect depend on the
+  // ub dialect to materialize ub.poison; left out for now.
+
   // remsi (x, 1) -> 0.
   if (matchPattern(adaptor.getRhs(), m_One()))
-    return Builder(getContext()).getZeroAttr(getType());
+    return getIntegerAttrOfType(getType(), 0);
+
+  // remsi (0, x) -> 0 and remsi (x, x) -> 0. Division by zero is UB, so
+  // refining to 0 is valid.
+  if (matchPattern(adaptor.getLhs(), m_Zero()) || getLhs() == getRhs())
+    return getIntegerAttrOfType(getType(), 0);
 
   // Don't fold if it would require a division by zero.
   bool div0 = false;

diff  --git a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
index 43f21561e6544..d22677fde774a 100644
--- a/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
+++ b/mlir/test/Conversion/ArithToLLVM/arith-to-llvm.mlir
@@ -11,14 +11,14 @@ func.func @vector_ops(%arg0: vector<4xf32>, %arg1: vector<4xi1>, %arg2: vector<4
   %0 = arith.constant dense<42.> : vector<4xf32>
 // CHECK-NEXT:  %1 = llvm.fadd %arg0, %0 : vector<4xf32>
   %1 = arith.addf %arg0, %0 : vector<4xf32>
-// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg2 : vector<4xi64>
-  %3 = arith.divsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg2 : vector<4xi64>
-  %4 = arith.divui %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg2 : vector<4xi64>
-  %5 = arith.remsi %arg2, %arg2 : vector<4xi64>
-// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg2 : vector<4xi64>
-  %6 = arith.remui %arg2, %arg2 : vector<4xi64>
+// CHECK-NEXT:  %2 = llvm.sdiv %arg2, %arg3 : vector<4xi64>
+  %3 = arith.divsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %3 = llvm.udiv %arg2, %arg3 : vector<4xi64>
+  %4 = arith.divui %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %4 = llvm.srem %arg2, %arg3 : vector<4xi64>
+  %5 = arith.remsi %arg2, %arg3 : vector<4xi64>
+// CHECK-NEXT:  %5 = llvm.urem %arg2, %arg3 : vector<4xi64>
+  %6 = arith.remui %arg2, %arg3 : vector<4xi64>
 // CHECK-NEXT:  %6 = llvm.fdiv %arg0, %0 : vector<4xf32>
   %7 = arith.divf %arg0, %0 : vector<4xf32>
 // CHECK-NEXT:  %7 = llvm.frem %arg0, %0 : vector<4xf32>
@@ -780,16 +780,16 @@ func.func @ceildivsi(%arg0 : i64, %arg1 : i64) -> i64 {
 }
 
 // CHECK-LABEL: @ceildivui
-// CHECK-SAME: %[[ARG0:.*]]: i32) -> i32
-func.func @ceildivui(%arg0 : i32) -> i32 {
+// CHECK-SAME: %[[ARG0:.*]]: i32, %[[ARG1:.*]]: i32) -> i32
+func.func @ceildivui(%arg0 : i32, %arg1 : i32) -> i32 {
 // CHECK: %[[CST0:.*]] = llvm.mlir.constant(0 : i32) : i32
 // CHECK: %[[CMP0:.*]] = llvm.icmp "eq" %[[ARG0]], %[[CST0]] : i32
 // CHECK: %[[CST1:.*]] = llvm.mlir.constant(1 : i32) : i32
 // CHECK: %[[SUB0:.*]] = llvm.sub %[[ARG0]], %[[CST1]] : i32
-// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG0]] : i32
+// CHECK: %[[DIV0:.*]] = llvm.udiv %[[SUB0]], %[[ARG1]] : i32
 // CHECK: %[[ADD0:.*]] = llvm.add %[[DIV0]], %[[CST1]] : i32
 // CHECK: %[[SEL0:.*]] = llvm.select %[[CMP0]], %[[CST0]], %[[ADD0]] : i1, i32
-  %0 = arith.ceildivui %arg0, %arg0 : i32
+  %0 = arith.ceildivui %arg0, %arg1 : i32
   return %0: i32
 }
 

diff  --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
index 92b587d5ed1e4..9ecebb502630c 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv-unsupported.mlir
@@ -53,10 +53,10 @@ func.func @unsupported_2x2elem_vector(%arg0: vector<2x2xi32>) {
 
 // -----
 
-func.func @int_vector4_invalid(%arg0: vector<2xi16>) {
+func.func @int_vector4_invalid(%arg0: vector<2xi16>, %arg1: vector<2xi16>) {
   // expected-error @+2 {{failed to legalize operation 'arith.divui'}}
   // expected-error @+1 {{bitwidth emulation is not implemented yet on unsigned op}}
-  %0 = arith.divui %arg0, %arg0: vector<2xi16>
+  %0 = arith.divui %arg0, %arg1: vector<2xi16>
   return
 }
 

diff  --git a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
index 5c9966055b540..4cef06992727b 100644
--- a/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
+++ b/mlir/test/Conversion/ArithToSPIRV/arith-to-spirv.mlir
@@ -198,18 +198,18 @@ func.func @float32_binary_scalar(%lhs: f32, %rhs: f32) {
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
 // CHECK-LABEL: @index_vector
-func.func @index_vector(%arg0: vector<4xindex>) {
+func.func @index_vector(%arg0: vector<4xindex>, %arg1: vector<4xindex>) {
   // CHECK: spirv.UMod %{{.*}}, %{{.*}}: vector<4xi32>
-  %0 = arith.remui %arg0, %arg0: vector<4xindex>
+  %0 = arith.remui %arg0, %arg1: vector<4xindex>
   return
 }
 
@@ -1522,11 +1522,11 @@ func.func @float32_maxnumf_scalar(%arg0 : vector<2xf32>, %arg1 : vector<2xf32>)
 
 // Check int vector types.
 // CHECK-LABEL: @int_vector234
-func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<4xi64>) {
+func.func @int_vector234(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<4xi64>, %arg3: vector<4xi64>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi8>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.UDiv %{{.*}}, %{{.*}}: vector<4xi64>
-  %1 = arith.divui %arg1, %arg1: vector<4xi64>
+  %1 = arith.divui %arg2, %arg3: vector<4xi64>
   return
 }
 
@@ -1570,11 +1570,11 @@ module attributes {
 } {
 
 // CHECK-LABEL: @int_vector23
-func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<3xi16>) {
+func.func @int_vector23(%arg0: vector<2xi8>, %arg1: vector<2xi8>, %arg2: vector<3xi16>, %arg3: vector<3xi16>) {
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<2xi32>
-  %0 = arith.divsi %arg0, %arg0: vector<2xi8>
+  %0 = arith.divsi %arg0, %arg1: vector<2xi8>
   // CHECK: spirv.SDiv %{{.*}}, %{{.*}}: vector<3xi32>
-  %1 = arith.divsi %arg1, %arg1: vector<3xi16>
+  %1 = arith.divsi %arg2, %arg3: vector<3xi16>
   return
 }
 

diff  --git a/mlir/test/Dialect/Arith/canonicalize.mlir b/mlir/test/Dialect/Arith/canonicalize.mlir
index d11b00e1c1e24..5352f777109de 100644
--- a/mlir/test/Dialect/Arith/canonicalize.mlir
+++ b/mlir/test/Dialect/Arith/canonicalize.mlir
@@ -3187,6 +3187,125 @@ func.func @no_fold_divsi_of_muli(%arg0 : index, %arg1 : index) -> index {
 
 // -----
 
+// CHECK-LABEL: func @divui_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divui_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divui %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]]
+func.func @divsi_zero_dividend(%arg0 : i32) -> i32 {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.divsi %c0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divui_self(%arg0 : i32) -> i32 {
+  %0 = arith.divui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @divsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @divsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivui_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivui_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivui %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @ceildivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @ceildivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.ceildivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @floordivsi_self
+//       CHECK:   %[[C1:.+]] = arith.constant 1 : i32
+//       CHECK:   return %[[C1]]
+func.func @floordivsi_self(%arg0 : i32) -> i32 {
+  %0 = arith.floordivsi %arg0, %arg0 : i32
+  return %0 : i32
+}
+
+// CHECK-LABEL: func @remui_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remui_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remui %arg0, %arg0 : i32
+  %1 = arith.remui %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @remsi_self_and_zero
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]]
+func.func @remsi_self_and_zero(%arg0 : i32) -> (i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.remsi %arg0, %arg0 : i32
+  %1 = arith.remsi %c0, %arg0 : i32
+  return %0, %1 : i32, i32
+}
+
+// CHECK-LABEL: func @divui_self_vector
+//       CHECK:   %[[C1:.+]] = arith.constant dense<1> : vector<4xi32>
+//       CHECK:   return %[[C1]]
+func.func @divui_self_vector(%arg0 : vector<4xi32>) -> vector<4xi32> {
+  %0 = arith.divui %arg0, %arg0 : vector<4xi32>
+  return %0 : vector<4xi32>
+}
+
+// CHECK-LABEL: func @ceildivui_ceildivsi_floordivsi_zero_dividend
+//       CHECK:   %[[C0:.+]] = arith.constant 0 : i32
+//       CHECK:   return %[[C0]], %[[C0]], %[[C0]]
+func.func @ceildivui_ceildivsi_floordivsi_zero_dividend(%arg0 : i32)
+    -> (i32, i32, i32) {
+  %c0 = arith.constant 0 : i32
+  %0 = arith.ceildivui %c0, %arg0 : i32
+  %1 = arith.ceildivsi %c0, %arg0 : i32
+  %2 = arith.floordivsi %c0, %arg0 : i32
+  return %0, %1, %2 : i32, i32, i32
+}
+
+// Distinct operands must not fold to the self identity.
+// CHECK-LABEL: func @divsi_distinct_no_fold
+//       CHECK:   arith.divsi
+func.func @divsi_distinct_no_fold(%arg0 : i32, %arg1 : i32) -> i32 {
+  %0 = arith.divsi %arg0, %arg1 : i32
+  return %0 : i32
+}
+
+// A dynamic shape must bail out of the self/zero-dividend constant folds.
+// CHECK-LABEL: func @div_rem_self_dynamic_no_fold
+//       CHECK:   arith.divui
+//       CHECK:   arith.remui
+func.func @div_rem_self_dynamic_no_fold(%arg0 : tensor<?xi32>)
+    -> (tensor<?xi32>, tensor<?xi32>) {
+  %0 = arith.divui %arg0, %arg0 : tensor<?xi32>
+  %1 = arith.remui %arg0, %arg0 : tensor<?xi32>
+  return %0, %1 : tensor<?xi32>, tensor<?xi32>
+}
+
+// -----
+
 // CHECK-LABEL: @test_cmpf(
 func.func @test_cmpf(%arg0 : f32) -> (i1, i1, i1, i1) {
 //   CHECK-DAG:   %[[T:.*]] = arith.constant true


        

From llvmlistbot at llvm.org  Sun Jul 26 09:25:41 2026
From: llvmlistbot at llvm.org (Victor Perez)
Date: Sun, 26 Jul 2026 09:25:41 -0700 (PDT)
Subject: [Mlir-commits] [mlir] [mlir][arith] Fold trivial integer division
	and remainder (PR #212074)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212074@github.com>
Message-ID: <6a663505.f06f4994.3876ce.0b10@mx.google.com>


https://github.com/victor-eds closed https://github.com/llvm/llvm-project/pull/212074

From llvmlistbot at llvm.org  Sun Jul 26 11:06:38 2026
From: llvmlistbot at llvm.org (Fangrui Song)
Date: Sun, 26 Jul 2026 11:06:38 -0700 (PDT)
Subject: [Mlir-commits] [llvm] [mlir] [LoopInfo] Build dominator tree only
 for irreducible CFG (PR #212098)
In-Reply-To: <llvm.org/llvm/llvm-project/pull/212098@github.com>
Message-ID: <6a664cae.f06f4994.3876ce.756c@mx.google.com>


https://github.com/MaskRay updated https://github.com/llvm/llvm-project/pull/212098

>From 2b3ce5ac2897a8614e7a5953896e859616d5d352 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sat, 25 Jul 2026 11:08:14 -0700
Subject: [PATCH 1/2] [LoopInfo] Build dominator tree only for irreducible CFG

analyze() requires a dominator tree, so LoopAnalysis and MachineLoopAnalysis
request one for every function, though only an irreducible CFG queries it.
Clients that build their own, from InlineCost to XRayInstrumentation, need it
for nothing else.

Take the function and a callback returning the tree instead, and call it when
an edge re-enters a loop. Add an analyze(F) overload for a client that holds
no tree.

The number of dominator tree builds does not change in an -O2 pipeline
building sqlite3.bc, where SROA and InstCombine cache one before
LoopAnalysis runs.

Tests that observed the tree through LoopAnalysis now require it explicitly.

MachineLoopInfoWrapperPass keeps requiring one: the legacy pass manager cannot
provide it on demand.

Aided by Claude Opus 5
---
 llvm/include/llvm/CodeGen/MachineLoopInfo.h   |  6 ++++++
 llvm/include/llvm/Support/GenericLoopInfo.h   | 13 +++++++++++-
 .../llvm/Support/GenericLoopInfoImpl.h        | 21 ++++++++++++++++++-
 llvm/lib/Analysis/InlineCost.cpp              | 11 +++++-----
 llvm/lib/Analysis/LoopInfo.cpp                |  5 ++++-
 llvm/lib/CodeGen/MachineLoopInfo.cpp          | 14 ++++++++++++-
 llvm/lib/CodeGen/XRayInstrumentation.cpp      | 17 ++++++++-------
 llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp |  5 ++---
 llvm/lib/Transforms/IPO/PartialInlining.cpp   |  4 ++--
 .../Instrumentation/InstrProfiling.cpp        |  5 ++---
 .../Instrumentation/PGOInstrumentation.cpp    | 11 +++++-----
 ...ate-branch-lowering-preserve-loop-info.mir |  2 +-
 ...i-pre-emit-peephole-preserve-loop-info.mir |  4 ++--
 .../HLSLControlFlowHint-pass-check.ll         |  2 +-
 .../test/Transforms/SCCP/preserve-analysis.ll |  5 +++--
 15 files changed, 88 insertions(+), 37 deletions(-)

diff --git a/llvm/include/llvm/CodeGen/MachineLoopInfo.h b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
index 7893f70cd353c..8ca01a54532d9 100644
--- a/llvm/include/llvm/CodeGen/MachineLoopInfo.h
+++ b/llvm/include/llvm/CodeGen/MachineLoopInfo.h
@@ -131,6 +131,12 @@ class MachineLoopInfo : public LoopInfoBase<MachineBasicBlock, MachineLoop> {
 
   /// Calculate the natural loop information.
   LLVM_ABI void calculate(MachineDominatorTree &MDT);
+
+  /// Rebuild the loop forest. \p GetDomTree is called only for an irreducible
+  /// CFG.
+  LLVM_ABI void
+  calculate(MachineFunction &MF,
+            function_ref<const DomTreeBase<MachineBasicBlock> &()> GetDomTree);
 };
 
 /// Analysis pass that exposes the \c MachineLoopInfo for a machine function.
diff --git a/llvm/include/llvm/Support/GenericLoopInfo.h b/llvm/include/llvm/Support/GenericLoopInfo.h
index f3ebef97c9e8b..19f725f29d4cc 100644
--- a/llvm/include/llvm/Support/GenericLoopInfo.h
+++ b/llvm/include/llvm/Support/GenericLoopInfo.h
@@ -840,8 +840,19 @@ template <class BlockT, class LoopT> class LoopInfoBase {
     return isNotAlreadyContainedIn(SubLoop->getParentLoop(), ParentLoop);
   }
 
-  /// Create the loop forest using a stable algorithm.
+  /// Create the loop forest for a function. A dominator tree is needed only for
+  /// an irreducible CFG, where dominance reduces a loop that an edge re-enters
+  /// to the natural loop of its header's backedges.
+  ///@{
+  /// Build a dominator tree if one is needed.
+  void analyze(ParentT F);
+  /// Call \p GetDomTree if a dominator tree is needed.
+  void
+  analyze(ParentT F,
+          function_ref<const DominatorTreeBase<BlockT, false> &()> GetDomTree);
+  /// Analyze the function \p DomTree describes.
   void analyze(const DominatorTreeBase<BlockT, false> &DomTree);
+  ///@}
 
   // Debugging
   void print(raw_ostream &OS) const;
diff --git a/llvm/include/llvm/Support/GenericLoopInfoImpl.h b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
index c89bdb7c0f566..4b0cc1b460681 100644
--- a/llvm/include/llvm/Support/GenericLoopInfoImpl.h
+++ b/llvm/include/llvm/Support/GenericLoopInfoImpl.h
@@ -461,12 +461,28 @@ void LoopBase<BlockT, LoopT>::print(raw_ostream &OS, bool Verbose,
 /// program order.
 template <class BlockT, class LoopT>
 void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
+  analyze(DomTree.getRootNode()->getBlock()->getParent(),
+          [&]() -> const DomTreeBase<BlockT> & { return DomTree; });
+}
+
+template <class BlockT, class LoopT>
+void LoopInfoBase<BlockT, LoopT>::analyze(ParentT F) {
+  DomTreeBase<BlockT> DomTree;
+  analyze(F, [&]() -> const DomTreeBase<BlockT> & {
+    DomTree.recalculate(*F);
+    return DomTree;
+  });
+}
+
+template <class BlockT, class LoopT>
+void LoopInfoBase<BlockT, LoopT>::analyze(
+    ParentT F, function_ref<const DomTreeBase<BlockT> &()> GetDomTree) {
   using BlockTraits = GraphTraits<BlockT *>;
   auto num = [](const BlockT *BB) {
     return GraphTraits<const BlockT *>::getNumber(BB);
   };
 
-  ParentPtr = DomTree.getRootNode()->getBlock()->getParent();
+  ParentPtr = F;
   BlockNumberEpoch = GraphTraits<ParentT>::getNumberEpoch(ParentPtr);
   unsigned MaxNumber = GraphTraits<ParentT>::getMaxNumber(ParentPtr);
 
@@ -593,6 +609,9 @@ void LoopInfoBase<BlockT, LoopT>::analyze(const DomTreeBase<BlockT> &DomTree) {
     // splice the header out of the chain of every other block.
     for (unsigned H : Reentries)
       Info[H].Pos = IsReentered;
+    const DomTreeBase<BlockT> &DomTree = GetDomTree();
+    assert(DomTree.getRootNode()->getBlock() ==
+           GraphTraits<ParentT>::getEntryNode(ParentPtr));
     DomTree.updateDFSNumbers();
     SmallVector<unsigned, 0> Mark(MaxNumber, NoBlock);
     SmallVector<BlockT *, 8> Worklist;
diff --git a/llvm/lib/Analysis/InlineCost.cpp b/llvm/lib/Analysis/InlineCost.cpp
index 67fd5f6495c5a..6cd84376d8a98 100644
--- a/llvm/lib/Analysis/InlineCost.cpp
+++ b/llvm/lib/Analysis/InlineCost.cpp
@@ -34,7 +34,6 @@
 #include "llvm/IR/AssemblyAnnotationWriter.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/DataLayout.h"
-#include "llvm/IR/Dominators.h"
 #include "llvm/IR/GetElementPtrTypeIterator.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/InlineAsm.h"
@@ -1083,11 +1082,11 @@ class InlineCostCallAnalyzer final : public CallAnalyzer {
     // movement, require a certain amount of setup, etc. So when optimising for
     // size, we penalise any call sites that perform loops. We do this after all
     // other costs here, so will likely only be dealing with relatively small
-    // functions (and hence DT and LI will hopefully be cheap).
+    // functions (and hence LI will hopefully be cheap).
     auto *Caller = CandidateCall.getFunction();
     if (Caller->hasMinSize()) {
-      DominatorTree DT(F);
-      LoopInfo LI(DT);
+      LoopInfo LI;
+      LI.analyze(&F);
       int NumLoops = 0;
       for (Loop *L : LI) {
         // Ignore loops that will not be executed
@@ -1395,8 +1394,8 @@ class InlineCostFeaturesAnalyzer final : public CallAnalyzer {
   InlineResult finalizeAnalysis() override {
     auto *Caller = CandidateCall.getFunction();
     if (Caller->hasMinSize()) {
-      DominatorTree DT(F);
-      LoopInfo LI(DT);
+      LoopInfo LI;
+      LI.analyze(&F);
       for (Loop *L : LI) {
         // Ignore loops that will not be executed
         if (DeadBlocks.count(L->getHeader()))
diff --git a/llvm/lib/Analysis/LoopInfo.cpp b/llvm/lib/Analysis/LoopInfo.cpp
index f7f577784f762..96faf5b242426 100644
--- a/llvm/lib/Analysis/LoopInfo.cpp
+++ b/llvm/lib/Analysis/LoopInfo.cpp
@@ -1011,7 +1011,10 @@ LoopInfo LoopAnalysis::run(Function &F, FunctionAnalysisManager &AM) {
   // objects. I don't want to add that kind of complexity until the scope of
   // the problem is better understood.
   LoopInfo LI;
-  LI.analyze(AM.getResult<DominatorTreeAnalysis>(F));
+  // The dominator tree is needed only for an irreducible CFG.
+  LI.analyze(&F, [&]() -> const DominatorTree & {
+    return AM.getResult<DominatorTreeAnalysis>(F);
+  });
   return LI;
 }
 
diff --git a/llvm/lib/CodeGen/MachineLoopInfo.cpp b/llvm/lib/CodeGen/MachineLoopInfo.cpp
index 9479b41f57b17..3fb0d043b01a1 100644
--- a/llvm/lib/CodeGen/MachineLoopInfo.cpp
+++ b/llvm/lib/CodeGen/MachineLoopInfo.cpp
@@ -37,7 +37,12 @@ AnalysisKey MachineLoopAnalysis::Key;
 MachineLoopAnalysis::Result
 MachineLoopAnalysis::run(MachineFunction &MF,
                          MachineFunctionAnalysisManager &MFAM) {
-  return MachineLoopInfo(MFAM.getResult<MachineDominatorTreeAnalysis>(MF));
+  MachineLoopInfo LI;
+  // The dominator tree is needed only for an irreducible CFG.
+  LI.calculate(MF, [&]() -> const MachineDominatorTree & {
+    return MFAM.getResult<MachineDominatorTreeAnalysis>(MF);
+  });
+  return LI;
 }
 
 PreservedAnalyses
@@ -80,6 +85,13 @@ void MachineLoopInfo::calculate(MachineDominatorTree &MDT) {
   analyze(MDT);
 }
 
+void MachineLoopInfo::calculate(
+    MachineFunction &MF,
+    function_ref<const DomTreeBase<MachineBasicBlock> &()> GetDomTree) {
+  releaseMemory();
+  analyze(&MF, GetDomTree);
+}
+
 void MachineLoopInfoWrapperPass::getAnalysisUsage(AnalysisUsage &AU) const {
   AU.setPreservesAll();
   AU.addRequired<MachineDominatorTreeWrapperPass>();
diff --git a/llvm/lib/CodeGen/XRayInstrumentation.cpp b/llvm/lib/CodeGen/XRayInstrumentation.cpp
index ce13abbc2145f..b9cc60820be8c 100644
--- a/llvm/lib/CodeGen/XRayInstrumentation.cpp
+++ b/llvm/lib/CodeGen/XRayInstrumentation.cpp
@@ -228,17 +228,18 @@ bool XRayInstrumentation::run(MachineFunction &MF) {
     bool TooFewInstrs = MICount < XRayThreshold;
 
     if (!IgnoreLoops) {
-      // Get MachineDominatorTree or compute it on the fly if it's unavailable
+      // Get MachineLoopInfo or compute it on the fly if it's unavailable,
+      // which needs a MachineDominatorTree only for an irreducible CFG.
       MachineDominatorTree ComputedMDT;
-      if (!MDT) {
-        ComputedMDT.recalculate(MF);
-        MDT = &ComputedMDT;
-      }
-
-      // Get MachineLoopInfo or compute it on the fly if it's unavailable
       MachineLoopInfo ComputedMLI;
       if (!MLI) {
-        ComputedMLI.analyze(*MDT);
+        ComputedMLI.calculate(MF, [&]() -> const MachineDominatorTree & {
+          if (!MDT) {
+            ComputedMDT.recalculate(MF);
+            MDT = &ComputedMDT;
+          }
+          return *MDT;
+        });
         MLI = &ComputedMLI;
       }
 
diff --git a/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp b/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
index e5afaa5c63e4b..793c48ceb0f84 100644
--- a/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
+++ b/llvm/lib/Target/SPIRV/SPIRVEmitIntrinsics.cpp
@@ -22,7 +22,6 @@
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringSet.h"
 #include "llvm/Analysis/LoopInfo.h"
-#include "llvm/IR/Dominators.h"
 #include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/InstIterator.h"
 #include "llvm/IR/InstVisitor.h"
@@ -3590,8 +3589,8 @@ void SPIRVEmitIntrinsicsImpl::emitUnstructuredLoopControls(Function &F,
 
   // For non-shader targets without the Intel extension, emit OpLoopMerge
   // using spv_loop_merge intrinsics, mirroring the structurizer approach.
-  DominatorTree DT(F);
-  LoopInfo LI(DT);
+  LoopInfo LI;
+  LI.analyze(&F);
   if (LI.empty())
     return;
 
diff --git a/llvm/lib/Transforms/IPO/PartialInlining.cpp b/llvm/lib/Transforms/IPO/PartialInlining.cpp
index e01bb9f78fca3..2dfb7411413db 100644
--- a/llvm/lib/Transforms/IPO/PartialInlining.cpp
+++ b/llvm/lib/Transforms/IPO/PartialInlining.cpp
@@ -905,10 +905,10 @@ void PartialInlinerImpl::computeCallsiteToProfCountMap(
   auto ComputeCurrBFI = [&,this](Function *Caller) {
       // For the old pass manager:
       if (!GetBFI) {
-        DominatorTree DT(*Caller);
         CycleInfo CI;
         CI.compute(*Caller);
-        LoopInfo LI(DT);
+        LoopInfo LI;
+        LI.analyze(Caller);
         BranchProbabilityInfo BPI(*Caller, CI);
         TempBFI.reset(new BlockFrequencyInfo(*Caller, BPI, LI));
         CurrentCallerBFI = TempBFI.get();
diff --git a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
index ef97dac7a5f0a..fa46d10277ab4 100644
--- a/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
+++ b/llvm/lib/Transforms/Instrumentation/InstrProfiling.cpp
@@ -33,7 +33,6 @@
 #include "llvm/IR/DIBuilder.h"
 #include "llvm/IR/DerivedTypes.h"
 #include "llvm/IR/DiagnosticInfo.h"
-#include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalAlias.h"
 #include "llvm/IR/GlobalValue.h"
@@ -973,10 +972,10 @@ void InstrLowerer::promoteCounterLoadStores(Function *F) {
   if (!isCounterPromotionEnabled())
     return;
 
-  DominatorTree DT(*F);
   CycleInfo CI;
   CI.compute(*F);
-  LoopInfo LI(DT);
+  LoopInfo LI;
+  LI.analyze(F);
   DenseMap<Loop *, SmallVector<LoadStorePair, 8>> LoopPromotionCandidates;
 
   std::unique_ptr<BlockFrequencyInfo> BFI;
diff --git a/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp b/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
index 47f98d98a50e9..92cd9b33f2ef1 100644
--- a/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
+++ b/llvm/lib/Transforms/Instrumentation/PGOInstrumentation.cpp
@@ -74,7 +74,6 @@
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/CycleInfo.h"
 #include "llvm/IR/DiagnosticInfo.h"
-#include "llvm/IR/Dominators.h"
 #include "llvm/IR/EHPersonalities.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/GlobalAlias.h"
@@ -1582,10 +1581,10 @@ void PGOUseFunc::populateCoverage() {
   }
 
   unsigned NumCorruptCoverage = 0;
-  DominatorTree DT(F);
   CycleInfo CI;
   CI.compute(F);
-  LoopInfo LI(DT);
+  LoopInfo LI;
+  LI.analyze(&F);
   BranchProbabilityInfo BPI(F, CI);
   BlockFrequencyInfo BFI(F, BPI, LI);
   auto IsBlockDead = [&](const BasicBlock &BB) -> std::optional<bool> {
@@ -2342,7 +2341,8 @@ static bool annotateAllFunctions(
     if (PGOViewCounts != PGOVCT_None &&
         (ViewBlockFreqFuncName.empty() ||
          F.getName() == ViewBlockFreqFuncName)) {
-      LoopInfo LI{DominatorTree(F)};
+      LoopInfo LI;
+      LI.analyze(&F);
       CycleInfo CI;
       CI.compute(F);
       std::unique_ptr<BranchProbabilityInfo> NewBPI =
@@ -2373,7 +2373,8 @@ static bool annotateAllFunctions(
     if (PGOVerifyBFI || PGOVerifyHotBFI || PGOFixEntryCount) {
       CycleInfo CI;
       CI.compute(F);
-      LoopInfo LI{DominatorTree(F)};
+      LoopInfo LI;
+      LI.analyze(&F);
       BranchProbabilityInfo NBPI(F, CI);
 
       // Fix func entry count.
diff --git a/llvm/test/CodeGen/AMDGPU/si-late-branch-lowering-preserve-loop-info.mir b/llvm/test/CodeGen/AMDGPU/si-late-branch-lowering-preserve-loop-info.mir
index 87925e41c2e71..fa7852e0b93b5 100644
--- a/llvm/test/CodeGen/AMDGPU/si-late-branch-lowering-preserve-loop-info.mir
+++ b/llvm/test/CodeGen/AMDGPU/si-late-branch-lowering-preserve-loop-info.mir
@@ -4,8 +4,8 @@
 # due to early termination handling.
 
 # CHECK: Running analysis: MachineLoopAnalysis on early_term_in_loop
-# CHECK-NEXT: Running analysis: MachineDominatorTreeAnalysis on early_term_in_loop
 # CHECK-NEXT: Running pass: SILateBranchLoweringPass on early_term_in_loop
+# CHECK-NEXT: Running analysis: MachineDominatorTreeAnalysis on early_term_in_loop
 # CHECK-NEXT: Running pass: MachineLoopPrinterPass on early_term_in_loop
 # CHECK-NEXT: Machine loop info for machine function 'early_term_in_loop':
 # CHECK-NOT: Running analysis: MachineLoopAnalysis on early_term_in_loop
diff --git a/llvm/test/CodeGen/AMDGPU/si-pre-emit-peephole-preserve-loop-info.mir b/llvm/test/CodeGen/AMDGPU/si-pre-emit-peephole-preserve-loop-info.mir
index 12eff8da0e477..89424af721277 100644
--- a/llvm/test/CodeGen/AMDGPU/si-pre-emit-peephole-preserve-loop-info.mir
+++ b/llvm/test/CodeGen/AMDGPU/si-pre-emit-peephole-preserve-loop-info.mir
@@ -1,7 +1,7 @@
-# RUN: llc -mtriple=amdgpu9.00-amd-amdhsa -passes="require<machine-loops>,si-pre-emit-peephole,print<machine-loops>" -debug-pass-manager -filetype=null %s 2>&1 | FileCheck %s
+# RUN: llc -mtriple=amdgpu9.00-amd-amdhsa -passes="require<machine-dom-tree>,require<machine-loops>,si-pre-emit-peephole,print<machine-loops>" -debug-pass-manager -filetype=null %s 2>&1 | FileCheck %s
 
+# CHECK: Running analysis: MachineDominatorTreeAnalysis on vcc_and_removal_preserves_mli
 # CHECK: Running analysis: MachineLoopAnalysis on vcc_and_removal_preserves_mli
-# CHECK-NEXT: Running analysis: MachineDominatorTreeAnalysis on vcc_and_removal_preserves_mli
 # CHECK-NEXT: Running pass: SIPreEmitPeepholePass on vcc_and_removal_preserves_mli
 # CHECK-NEXT: Invalidating analysis: MachineDominatorTreeAnalysis on vcc_and_removal_preserves_mli
 # CHECK-NEXT: Running pass: MachineLoopPrinterPass on vcc_and_removal_preserves_mli
diff --git a/llvm/test/CodeGen/SPIRV/structurizer/HLSLControlFlowHint-pass-check.ll b/llvm/test/CodeGen/SPIRV/structurizer/HLSLControlFlowHint-pass-check.ll
index 07a31b6c26f52..78c5d6e263087 100644
--- a/llvm/test/CodeGen/SPIRV/structurizer/HLSLControlFlowHint-pass-check.ll
+++ b/llvm/test/CodeGen/SPIRV/structurizer/HLSLControlFlowHint-pass-check.ll
@@ -5,8 +5,8 @@
 ; RUN: opt -passes='spirv-structurizer' -disable-output -debug-pass-manager \
 ; RUN:   -mtriple=spirv-unknown-unknown %s 2>&1 | FileCheck %s --check-prefix=INVALIDATE
 ; INVALIDATE: Running pass: SPIRVStructurizerWrapper on test_branch
-; INVALIDATE: Invalidating analysis: DominatorTreeAnalysis on test_branch
 ; INVALIDATE: Invalidating analysis: LoopAnalysis on test_branch
+; INVALIDATE: Invalidating analysis: DominatorTreeAnalysis on test_branch
 ; INVALIDATE: Invalidating analysis: SPIRVConvergenceRegionAnalysis on test_branch
 
 ; CHECK-LABEL: define spir_func noundef i32 @test_branch
diff --git a/llvm/test/Transforms/SCCP/preserve-analysis.ll b/llvm/test/Transforms/SCCP/preserve-analysis.l